Danja Brandt, Martin Tschaikner, Teodor Chiaburu, Henning Schmidt, Ilona
Schrimpf, Alexandra Stadel, Ingeborg Beckers, Frank Haußer 11institutetext:
Berliner Hochschule fuer Technik,http://bht-berlin.de
11email<EMAIL_ADDRESS>22institutetext: UBZ Listhof e.V.,
Reutlingen, http://www.listhof-reutlingen.de.

# Low Cost Machine Vision for Insect Classification

Danja Brandt and Martin Tschaikner 1111
Teodor Chiaburu 11 Henning Schmidt 11 Ilona Schrimpf 22 Alexandra Stadel and
Ingeborg E. Beckers 2211 Frank Haußer 11

###### Abstract

Preserving the number and diversity of insects is one of our society’s most
important goals in the area of environmental sustainability. A prerequisite
for this is a systematic and up-scaled monitoring in order to detect
correlations and identify countermeasures. Therefore, automatized monitoring
using live traps is important, but so far there is no system that provides
image data of sufficient detailed information for entomological
classification.

In this work, we present an imaging method as part of a multisensor system
developed as a low-cost, scalable, open-source system that is adaptable to
classical trap types. The image quality meets the requirements needed for
classification in the taxonomic tree. Therefore, illumination and resolution
have been optimized and motion artefacts have been suppressed. The system is
evaluated exemplarily on a dataset consisting of 16 insect species of the same
as well as different genus, family and order. We demonstrate that standard
CNN-architectures like ResNet50 (pretrained on iNaturalist data) or MobileNet
perform very well for the prediction task after re-training. Smaller custom
made CNNs also lead to promising results. Classification accuracy of $>96\%$
has been achieved. Moreover, it was proved that image cropping of insects is
necessary for classification of species with high inter-class similarity.

###### keywords:

computer vision, image classification, insect monitoring, biodiversity

##  1 Introduction

With over 1 million species worldwide, insects are the largest group of
animals on our planet. Many ecological and economic interactions of insects
with their environment are still unexplored. The importance of insects for
humans is becoming increasingly clear. One example is their pollination
service, which is estimated at 3.8 billion € for Germany alone[1]. All over
the world the number of insect species and individuals is declining sharply
[2]. In Germany, the Krefeld study reports a 75% decrease in insect biomass
from 1989 to 2015 [3]. Both in the short term and in the long term, knowledge
about the occurrence, distribution and population trends of insects in
correlation with environmental influences is of great importance. Still most
classical monitoring systems consist of dead traps. Biomass is determined and
only a few species are identified. In order to avoid blind spots in monitoring
and to multiply expert knowledge, research in the field of automated
classification systems is becoming more and more important. In this context,
three partly overlapping use cases can be identified:

  1. 1.

Mobile applications (Picture Insect, Insect Identifier, a.o. ) and Open Source
Citizen Science applications, which contribute to raising public awareness
through gamification approaches. Large amounts of data are generated this way.
In particular, the iNaturalist platform serves as a global database and had an
enormous impact in scientific research questions [4]. It also serves as a
networking tool. Large data sets of images are labeled by the community in a
collaborative manner (whenever more than two thirds of the annotators agree on
the label, the sample is awarded a Research Grade as a seal of quality).
However, these randomly collected cell phone or camera images cannot properly
contribute to scientific monitoring which rather requires standardized data
collected systematically over a period of time at fixed locations.

  2. 2.

Monitoring for pest control and identification of beneficial insects in
agriculture [5]. In this use case, multilabel classification machine learning
algorithms are used to identify certain insects species, which are of special
interest [6, 7].

  3. 3.

Biodiversity monitoring [8] Systematic monitoring of insect diversity requires
suitable sensor systems and multilabel classification algorithms for fine-
grained images or multisensor data. Moreover, the data sets are highly
unbalanced due to rare insect species.

Automated monitoring uses different technologies for insect species detection
and differentiation [8]. The data acquisition for monitoring is mainly based
on wingbeat frequency measurements and image data. For wing beat frequencies
acquisition alone a wide variety of different methods is used, including
acoustic sensors [9], radar [10, 11], multispectral analysis of reflected
light [12, 13], as well as capacitance change [14] and optoacoustic sensors
[15, 16, 17, 18, 19]. There are also different approaches for imaging. One
requirement is that insects must be in relative rest for a camera to take
sharp images. [20, 21] use images from museums and other collections of dead
insects for training of automated systems. [22, 23, 24] use yellow traps on
which the insects stick. For monitoring living insects, yellow fields [25] are
used - suitable for pollinators – or light traps [26, 27, 28] – suitable for
nocturnal insects. A recent survey of machine learning methods for insect
classification is given in [29].

We developed an automated low cost open source insect multisensor monitoring
system where insects are counted and classified according to the taxonomic
system of GBIF [30] (order, family, genus, species). Citizen scientists are
involved in developing and measuring. The sensor system includes an infrared
optoacustic wingbeat frequency detector and a computer vision unit. For a
robust classification, local environmental data are collected additionally
i.e. time, temperature, humidity as well as spectral irradiance. While based
on standard insect traps from classical entomology, our multisensor [31]
discards the container with the lethal liquid and leaves the insects unharmed.

Hence, our approach allows to deal with all three use cases as discussed
above. It is designed for scientific biodiversity monitoring but due to its
adaptability to various types of traps, it may as well be used to monitor
specific insects e.g. by training the neural network in a one-versus-All
classification.

In this paper we focus on the imaging system: the hardware setup, image post-
processing as well as evaluation by solving a classification task with machine
learning. A particular effort has been made to optimize and standardize image
acquisition.

The paper is structured as follows: First, the requirements for the imaging
system are summarized (2.1). Structures on which experts distinguish species
need to be resolved on images of living insects in motion. According to these
requirements the camera system is developed (chapter 2.2). The present dataset
shows one of the biggest challenges, which stands as representative for insect
monitoring: The unbalanced distribution of insect data (chapter 2.3). The
overall system was evaluated for a data set of 16 species, some differing at
the family level others only at the species level (chapter 2.4). Standard
machine learning models like U-Net for semantic segmentation and ResNet-50
[32], MobileNet [33] and a small self designed CNN demonstrate convincing
prediction power for the insect species (Chapter 3). Finally, results are
presented and discussed in the context of currently used methods.

##  2 Materials and Methods

###  2.1 Imaging Requirements

Classifying insects from images requires taking various features into
consideration such as color, contrast, patterning, size, and shape
relationships. Special features for entomological differentiation such as
pubescence and wing vascular pattern may also be important for certain
species, see Fig. 1.

Figure 1: Image of a European paper wasp taken with the Imaging Unit with
zoomed-in characteristic details of body pattern, tabia-tarsus junction and
wing details.

For example, Vespula vulgaris and Vespula germanica, both of the Vespidae
family, look similar but are distinguished by a particular pattern on the hind
limb. On the other hand, Polistes dominula, also from the Vespidae family, has
a distinctive shape relationship (Fig. 2).

Figure 2: Comparison of 3 wasp species. (Top) Images acquired from the imaging
unit. (Bottom) Schematic representation according to Ripberger [34]. From left
to right: Vespula germanica, Vespa vulgaris, Polistes dominula

The developed system is standardized according to homogeneous illumination,
suppression of motion blur, resolution and depth-of-field, what is critical
for an accurate classification. A homogeneous, diffuse illumination is
essential as well as a resolution that matches insect structural sizes.
Furthermore, shadows on the background make automatic segmentation and
detection difficult. To address these requirements, the acquisition unit has
diffusing panels on the side walls and the bottom. A trade-off between maximal
resolving power and large depth-of-field has been figured out. Characteristic
structures like the hairs or the wing veins are in the range of 20 - 100
microns. However, the distance between the veins is at least one order of
magnitude greater. The choice of an effective aperture (f$/$8) leads to a
resolution of 10 micrometers, which makes it possible to differentiate vein
sequences on the object side. The related depth of field is 15 mm for a rear
distance of 150 mm (Fig. 3).

###  2.2 Hardware Setup

Behind the insect trap, the multisensor system [31] with the imaging unit
(Fig. 3 (left)) replaces the container filled with poisonous agent. This
imaging unit is described below.

0

Figure 3: Image acquisition unit (left) and principle of the insect arena
(right). The red region indicates the light barrier in the center of the arena
that starts the image recording.

The area for imaging as shown in Fig. 3 (right) is confined to $60\times
45\times 20\,\mathrm{mm}^{3}$. The field of view (FoV) is optimized for
$\mathrm{FoV}=60\times 45\,\mathrm{mm}^{2}$. The adapters at the input and
output are 3D printed from PLA. The arena itself is cut from Plexiglas by a
laser cutter. The top side is transparent, while the side walls and bottom are
made of diffusing PMMA with a light transmission of $62\,\%$. Two
photoelectric sensors (IS471FE OPIC optical integrated circuit), one at the
bottom and one at the top, facing each other in the center of the arena,
trigger the photo detection. Insects passing by trigger the strobe flash
light, consisting of three 12V LED strips with three units each (nine in
total), which homogeneously illuminate the insects over the entire height.
Moreover, the LED strips provide a neutral gray background by allowing light
to enter the floor from the side facets. The IS471FE chip as receiver of the
light barrier provides a modulated signal to an external IR LED (transmitter
with $940\ \textrm{nm}$), so that interfering influences, such as ambient
light, are filtered. To reduce stray light, an additional $0.75\ \textrm{mm}$
slit aperture is applied. This increases the sensitivity of the photoelectric
light barrier sensor especially for small insects such as fruit flies and
hover flies.

Custom electronic circuit boards are designed and developed to be plugged into
the Raspberry Pi minicomputer. For imaging, a Raspberry Pi HQ camera (Sony
CMOS sensor IMX477R, 12.3 megapixels) is used in combination with a 10MP 16mm
telephoto lens with variable aperture. The entire system is located in a dark
case of size $296\times 127\times 115\,\mathrm{mm}^{3}$. The length is chosen
according to an optimal trade-off for resolution, field-of-view and depth of
field. As a result, an additional $2\,\mathrm{mm}$ spacer ring was 3D-printed
to extend the image distance, resulting in greater magnification and shorter
object distance.

Flashing in a dark scene minimizes motion blur caused by CMOS rolling shutter
technology. The FSTROBE pin of the camera chip sends a PWM signal synchronized
with the frame rate. The combination of a short flash of 500 microseconds and
an exposure time of 23.5 milliseconds, which is related to the readout time
and the reset time of a single line, results in the simulation of a global
shutter that minimizes distortion effects. Ants, for example, crawl very fast
with a speed of $50\,\textrm{cm/s}$. During one single flash, this corresponds
to a path of $0.25\,\mathrm{mm}$. At 10x reduction in magnitude, this is
equivalent to 25 micrometers on the chip (13 pixels on the Sony IMX477R sensor
chip). This results in motion blur 2.5 times worse than the blur due to
diffraction-limited resolving power at f/8. However, most insects are slower;
besides, they also slow down in the arena. It is evident in the acquired
images that the wings, whose wingbeat frequency can reach values in the kHz
range, as well as individual limbs, exhibit motion blur, while the trunk is
detected free of motion blur, see Fig. 2. This has been demonstrated for
various insects including ants.

To calibrate the camera system a Bokeh application (Python) has been
implemented, which is running on the Raspberry Pi as a local server. The
automated measurement is processed by a Python script, too. It controls the
individual sensors, processes the incoming data and stores the information on
SD card. The camera provides an in-memory H264 video stream based on a ring
buffer to limit RAM memory usage. The size of the ring buffer corresponds to
an H264 video of about 1.5 seconds in length at a resolution of $1440\times
1080$ pixels. A sudden increase in frame brightness triggers the extraction of
the first image frame from the video, followed by two others, saved as png-
images.

###  2.3 Dataset and Preprocessing

To evaluate the quality of our insect images, we will solve a classification
task using a dataset of 1154 annotated images with 16 insect species classes,
which are spread over the taxonomic tree - order, family, genus, species. The
dataset is extremely unbalanced, as can be seen in Fig. 4.

Figure 4: Distribution of the species in the total data set used for the
experiments. Notice the long tail of the histogram which is typical for insect
monitoring data.

This is a realistic assumption for real world data, since certain insect
species will pass through the sensor system more often than others e.g. rare
species. Thus, the data set represents the reality within insect monitoring
very well. The detailed list of the distribution of species in the data set
used in this work is given in Table 1.

The original images are of size $1440\times 1080$ pixels. We used Label Studio
[35] to manually crop them to a square region containing the insect. Section
2.5 discusses how cropping may be done automatically and very reliably using
semantic segmentation.

order | family | genus | species
---|---|---|---
Hymenoptera | Apidae | Apis | mellifica
Bombus | lapidarius
terrestris
Xylocopa | violacea
Vespidae | Vespa | crabro
Vespula | germanica
Polistes | dominula
Coleoptera | Chrysomelidae | Leptinotarsa | decemlineata
Cetoniidae | Cetonia | aurata
Mecoptera | Panorpidae | Panorpa | communis
Diptera | Tachinidae | Sarcophaga | carnaria
Syrphidae | Scaeva | pyrastri
Eristalis | tenax
Episyrphus | balteatus
Syrphus | ribesii
Heteroptera | Pentatomidae | Graphosoma | lineatum
Table 1: The dataset consists of 1154 images of 16 insect species from various
insect orders (5), families (8) and genera (15).

###  2.4 Machine Learning, Insect Classification

This work shows that the high quality image data provided by the presented
imaging system gives very good results for automatic insect classification on
minicomputers like Raspberry Pis. We decided to use two standard convolutional
neural networks (CNN) of different sizes and a custom made rather small CNN.
In all three cases, the input image size is $224\times 224$.

As already stated, insect classification is a fine-grained image analysis
task. Moreover, data from monitoring will always be extremly unbalanced. An
additional constraint on the monitoring system is that predictions must be
performed on a small devices of low power consumption, which precludes massive
neural network models.

As a benchmark model, ResNet-50 [32] was used, which is a rather large CNN
($\approx 25$M parameters). Pretrained on the iNaturalist dataset 2021 with
2.7M images from 10k different species [36] the ResNet converged within a
small number of epochs. No layers have been frozen. Secondly, a smaller
MobileNet ($\approx 3.5$M parameters, default depth parameter $\alpha=1$)
pretrained on ImageNet was retrained, also with no frozen layers. Finally, a
small custom made CNN with 5 layers and $\approx 1.3M$ parameters was trained
from scratch, see Fig 8 in the Appendix.

The performance of the three CNNs is compared for two distinct cases: full
images of the $6\,\mathrm{cm}\times 4,\mathrm{cm}$ arena, see Fig 3 and
cropped images of the insects. In the first case, downscaling from $1440\times
1080$ pixels to the NN input size of $224\times 224$ results in a serious loss
of morphological details of the insects. Top-1 accuracies as well as confusion
matrices are discussed in Section 3.1.

The models and experiments have been implemented in Python using Keras and
Tensorflow 2.8.

###  2.5 Semantic Segmentation

For a precise and robust object localisation, we evaluated a small sized U-Net
[37] as provided by [38]. The images were resized from $1440\times 1080$ down
to $256\times 256$. Label Studio [35] was used to segment insects on 301
images by hand. Based on this ground truth, the U-Net was trained to predict
masks. Preliminary results are presented in Section 3.2. The predicted masks
can be used to define a minimal bounding box containing the insect and, thus,
perform an automatic cropping.

##  3 Results and Discussion

###  3.1 Classification

The data set was split in fixed training, validation and test sets of ratios
60/20/20. Top-1 accuracies on the test set are reported in Table 2 111As a
side note, we did not fine-tune class weights or over-/under-sampling
techniques to deal with the underrepresented classes. Nonetheless, the models
performed very well on the cropped data..

Model | Accuracy (full) | Accuracy (cropped)
---|---|---
ResNet-50 | 0.9685 | 0.9604
MobileNet | 0.8776 | 0.9780
Custom CNN | 0.7058 | 0.8722
Table 2: Comparison of insect classification performance (Top-1 accuracy) on
the test set using full images and cropped images.

It is quite evident that all models perform very well on cropped images. Also
worth emphasizing is that the MobileNet and the Custom CNN reach noticeably
higher scores when trained on the cropped samples as opposed to the uncropped
ones. This is, in contrast, not the case with the ResNet, which seems to
perform equally well on the raw and zoomed-in images. At first sight, this
appears counterintuitive, since downsampling full images to 224 $\times$ 224
pixels causes substantial information loss and results in rather coarse/blurry
images. This indicates, that up to a certain classification accuracy within
the taxonomic tree, the features expressed by high spatial frequencies (fine
grained structures) are not relevant. Moreover, a reasonable explanation for
the still very high scores may lie in the favorable initialization with iNat21
weights. Hence, the ResNet starts the retraining already with prior knowledge
about insects, which a MobileNet, for instance, did not inherit from the
ImageNet weights. And indeed, the size of the ResNet (around 6 times more
parameters than MobileNet and 20 times more than the custom CNN) is also not
to be neglected.

Since the dataset is highly unbalanced, the test accuracy alone does not
provide a complete picture of the performance of the models in the individual
classes. The confusion matrices in Fig. 7 and Fig. 6 from the Appendix reveal
that especially the rare species are considerably difficult to distinguish in
full images. In particular the two rarest species Polistes dominula and
Episyrphus balteatus (Fig. 4) are never predicted correctly in the full
images, see Fig. 7.

A key parameter to further improve the accuracy of the models and to handle a
large number of different species is to increase the amount of data.
Evidently, the more data provided from different species, the more species can
be classified. For this reason, the contribution of citizen scientists in data
collection is central to the project.

There are insect species for which it is very difficult or even impossible to
determine the correct taxon based on multisensor data. Here, dissection or
genetic tests would be necessary. Therefore, the classifier should not always
predict the insect species but rather the highest possible level in the
taxonomic hierarchy, e.g. species or genus, or family. To achieve this, a
hierarchical classification scheme may be implemented, see Gitlab repository
[39].

###  3.2 Bounding Boxes via Segmentation

Figure 5: Example of inferring a bounding box: From left to right: original
image, manually annotated segmentation mask (used as label for training the
U-Net in a supervised context), U-Net predicted mask, bounding box inferred
from the predicted mask.

The results discussed in detail in Section 3.1 show that zooming in on the
insect produces better results, although size information is lost when
cropping. Therefore, we implement automated cropping in our classification
workflow, see Fig. 5.

The automated cropping algorithm based on segmentation and described in
Section 2.5, reaches an IoU of 0.7658 for the predicted masks using 301
labeled images as input. This method is more robust than commonly used
thresholding techniques. Variations in brightness, white balance and reflects
can occur in every image, making it impossible to set a global threshold value
suitable for every image. Thresholding also leads to problems if several
insects of different colours are present in one image or if certain parts of
the insect have similar colours compared to the background, such as
transparent wings. In addition, we observed that the arena gets dirty quickly,
making it necessary to filter out non-specific objects such as dust or insect
dirt from the bounding box evaluation. Prediction of insect masks overcomes
these problems.

In order to improve the preprocessing pipeline, one-stage models such as YOLO
[40] or other single-shot detectors will also be tested to obtain the best
bounding box evaluator suitable for our data.

##  4 Conclusion and Outlook

The goal of this project is to develop a complete and autonomous system for
insect species classification for use in the field that can be replicated and
operated by citizen scientists.

To achieve this, a low-cost system based on a minicomputer with various
sensors and a Raspberry Pi HQ camera has been developed for standardized and
AI-based automated monitoring of insects. The presented work focuses on the
image acquisition unit including image processing. Both have been optimized to
meet the requirements for an automated species-level classification on a
minicomputer that can serve as edge device. The image processing includes an
automatic trigger to capture a video clip, an adjusted frame selection,
identification of insects in each frame and cropping, and classification of
the insect. The image processing system is evaluated based on the correlation
between the preprocessing (cropping) and the classification accuracy for
different CNNs (ResNet-50, MobileNet, and a Custom-defined CNN). The ResNet-50
pretrained on the iNaturalist data of insects shows the best results for full
images with up to 96$\%$ accuracy, whereas the MobilNet and ResNet both
achieved 96 $\%$ accuracy for cropped images. However, a custom designed
network proves to be a promising approach for an energy-efficient application,
in particular inference on the Raspberry Pi as edge device. In addition, we
found that the accuracy of the smaller models is significantly higher trained
on cropped images.

As a next step, we will integrate multisensor data such as wing beat
frequencies, environmental data and metadata such as insect size, which is
known by its relationship to fixed FoV. Especially, the classification of even
very small insects is sensitive to characteristic wingbeat frequencies. Prior
probabilities depending on size, temperature, season and other factors, which
are also used by experts, should similarly support the classification in
automated monitoring. To overcome the problem of rare species, achieving high
accuracy even with few input samples, class weighting and oversampling
techniques will be investigated as well as methods for few shot learning such
as contrastive learning. Extensive data collection with citizen scientists
will be conducted during this year’s flight season.

## References

  * [1] Lippert, C., Feuerbacher, A., Narjes, M.: Revisiting the economic valuation of agricultural losses due to large-scale changes in pollinator populations.  Ecological Economics 180, 106,860 (2021).  https://doi.org/10.1016/j.ecolecon.2020.106860.  URL https://www.sciencedirect.com/science/article/pii/S0921800920300793
  * [2] Sánchez-Bayo, F., Wyckhuys, K.A.: Worldwide decline of the entomofauna: A review of its drivers.  Biological conservation 232, 8–27 (2019)
  * [3] Hallmann, C.A., Sorg, M., Jongejans, E., Siepel, H., Hofland, N., Schwan, H., Stenmans, W., Müller, A., Sumser, H., Hörren, T., Goulson, D., Kroon, H.D.: More than 75 percent decline over 27 years in total flying insect biomass in protected areas.  PLoS ONE 12 (2017).  10.1371/journal.pone.0185809
  * [4] Van Horn, G., Mac Aodha, O., Song, Y., Cui, Y., Sun, C., Shepard, A., Adam, H., Perona, P., Belongie, S.: The inaturalist species classification and detection dataset.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8769–8778 (2018)
  * [5] Lima, M.C.F., de Almeida Leandro, M.E.D., Valero, C., Coronel, L.C.P., Bazzo, C.O.G.: Automatic detection and monitoring of insect pests—a review.  Agriculture 10(5), 161 (2020)
  * [6] Wu, X., Zhan, C., Lai, Y.K., Cheng, M.M., Yang, J.: Ip102: A large-scale benchmark dataset for insect pest recognition.  In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8787–8796 (2019)
  * [7] Kirkeby, C., Rydhmer, K., Cook, S.M., Strand, A., Torrance, M.T., Swain, J.L., Prangsma, J., Johnen, A., Jensen, M., Brydegaard, M., Græsbøll, K.: Advances in automatic identification of flying insects using optical sensors and machine learning.  Scientific Reports 11 (2021).  10.1038/s41598-021-81005-0
  * [8] van Klink, R., August, T., Bas, Y., Bodesheim, P., Bonn, A., Fossøy, F., Høye, T.T., Jongejans, E., Menz, M.H., Miraldo, A., Roslin, T., Roy, H.E., Ruczyński, I., Schigel, D., Schäffler, L., Sheard, J.K., Svenningsen, C., Tschan, G.F., Wäldchen, J., Zizka, V.M., Åström, J., Bowler, D.E.: Emerging technologies revolutionise insect ecology and monitoring.  Trends in Ecology and Evolution 37, 872–885 (2022).  10.1016/j.tree.2022.06.001
  * [9] Mankin, R., Hagstrum, D., Guo, M., Eliopoulos, P., Njoroge, A.: Automated applications of acoustics for stored product insect detection, monitoring, and management.  Insects 12 (2021).  10.3390/insects12030259
  * [10] Rhodes, M.W., Bennie, J.J., Spalding, A., ffrench Constant, R.H., Maclean, I.M.: Recent advances in the remote sensing of insects.  Biological Reviews 97(1), 343–360 (2022)
  * [11] Wang, R., Hu, C., Fu, X., Long, T., Zeng, T.: Micro-doppler measurement of insect wing-beat frequencies with w-band coherent radar.  Scientific Reports 7 (2017).  10.1038/s41598-017-01616-4
  * [12] Rigakis, I., Potamitis, I., Tatlas, N.A., Livadaras, I., Ntalampiras, S.: A multispectral backscattered light recorder of insects’ wingbeats.  Electronics (Switzerland) 8 (2019).  10.3390/electronics8030277
  * [13] Rydhmer, K., Bick, E., Still, L., Strand, A., Luciano, R., Helmreich, S., Beck, B.D., Grønne, C., Malmros, L., Poulsen, K., et al.: Automating insect monitoring using unsupervised near-infrared sensors.  Scientific Reports 12(1), 1–11 (2022)
  * [14] Khoo, B., T, B.T.K., P, R.R., Kitt, W.W.: Capacitive mosquito wing-beat sensor: A novel sensor for intelligent traps ijera journal capacitive mosquito wing-beat sensor: A novel sensor for intelligent traps.  International Journal of Engineering Research and Applications www.ijera.com 10, 18–27 (2020).  10.9790/9622-1007071827.  URL www.ijera.com
  * [15] degli studi Roma tre, U., for Signal Processing, E.A., Society, I.S.P., of Electrical, I., Engineers, E.: EUSIPCO 2018 : 26th European Signal Processing Conference : Rome, Italy, September 3 - 7, 2018 (2018)
  * [16] Wang, J., Zhu, S., Lin, Y., Svanberg, S., Zhao, G.: Mosquito counting system based on optical sensing.  Applied Physics B: Lasers and Optics 126 (2020).  10.1007/s00340-019-7361-2
  * [17] Balla, E., Flórián, N., Gergócs, V., Gránicz, L., Tóth, F., Németh, T., Dombos, M.: An opto-electronic sensor-ring to detect arthropods of significantly different body sizes.  Sensors (Switzerland) 20 (2020).  10.3390/s20040982
  * [18] Potamitis, I., Rigakis, I., Vidakis, N., Petousis, M., Weber, M.: Affordable bimodal optical sensors to spread the use of automated insect monitoring.  Journal of Sensors 2018 (2018).  10.1155/2018/3949415
  * [19] Batista, G., Keogh, E., Neto, A.M., Rowton, E.: Sigkdd demo: Sensors and software to allow computational entomology, an emerging application of data mining.  pp. 761–764 (2011).  10.1145/2020408.2020530
  * [20] Amrani, A., Sohel, F., Diepeveen, D., Murray, D., Jones, M.G.K.: Insect detection from imagery using yolov3-based adaptive feature fusion convolution network.  Crop and Pasture Science (2022).  10.1071/cp21710
  * [21] Wang, Q.J., Zhang, S.Y., Dong, S.F., Zhang, G.C., Yang, J., Li, R., Wang, H.Q.: Pest24: A large-scale very small object data set of agricultural pests for multi-target detection.  Computers and Electronics in Agriculture 175, 105,585 (2020)
  * [22] Ramalingam, B., Mohan, R.E., Pookkuttath, S., Gómez, B.F., Borusu, C.S.C.S., Teng, T.W., Tamilselvam, Y.K.: Remote insects trap monitoring system using deep learning framework and iot.  Sensors (Switzerland) 20, 1–17 (2020).  10.3390/s20185280
  * [23] Gerovichev, A., Sadeh, A., Winter, V., Bar-Massada, A., Keasar, T., Keasar, C.: High throughput data acquisition and deep learning for insect ecoinformatics.  Frontiers in Ecology and Evolution 9, 600,931 (2021)
  * [24] Sütő, J.: Embedded system‐based sticky paper trap with deep learning‐based insect‐counting algorithm.  Electronics (Switzerland) 10 (2021).  10.3390/electronics10151754
  * [25] M. Sittinger: Insect detect.  https://maxsitt.github.io/insect-detect-docs.  Online; accessed 30 January 2023
  * [26] Yao, Q., Feng, J., Tang, J., gen Xu, W., hua Zhu, X., jun Yang, B., Lü, J., ze Xie, Y., Yao, B., zhen Wu, S., yang Kuai, N., jun Wang, L.: Development of an automatic monitoring system for rice light-trap pests based on machine vision.  Journal of Integrative Agriculture 19, 2500–2513 (2020).  10.1016/S2095-3119(20)63168-9
  * [27] Hogeweg, L., Zeegers, T., Katramados, I., Jongejans, E.: Smart insect cameras.  Biodiversity Information Science and Standards 3, e39,241 (2019)
  * [28] Høye, T.T., Ärje, J., Bjerge, K., Hansen, O.L.P., Iosifidis, A., Leese, F., Mann, H.M.R., Meissner, K., Melvad, C., Raitoharju, J.: Deep learning and computer vision will transform entomology.  PNAS 118, 2002545,117 (2021).  10.1073/pnas.2002545117/-/DCSupplemental
  * [29] Teixeira, A.C., Ribeiro, J., Morais, R., Sousa, J.J., Cunha, A.: A systematic review on automatic insect detection using deep learning.  Agriculture 13(3) (2023).  10.3390/agriculture13030713.  URL https://www.mdpi.com/2077-0472/13/3/713
  * [30] Gbif.  https://www.gbif.org
  * [31] A. Stadel, I. Beckers, D. Brandt, F. Haußer, M. Tschaikner, I. Schrimpf, H. Treutlein: Kinsecta project.  http://www.kinsecta.org/project.  Online; accessed 30 January 2023
  * [32] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition (2015).  10.48550/ARXIV.1512.03385.  URL https://arxiv.org/abs/1512.03385
  * [33] Howard, A.G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., Adam, H.: Mobilenets: Efficient convolutional neural networks for mobile vision applications (2017).  10.48550/ARXIV.1704.04861.  URL https://arxiv.org/abs/1704.04861
  * [34] Ripberger, R., Hutter, C., Koch, F.: Schützt die hornissen.–1992. 119 s. hard cover. k. thienemanns verlag gmbh & co., stuttgart. preis: 36,-dm. isbn 3-522-30450-0 (1993)
  * [35] Label studio.  https://github.com/heartexlabs/label-studio
  * [36] Van Horn, G., Cole, E., Beery, S., Wilber, K., Belongie, S., Mac Aodha, O.: Benchmarking representation learning for natural world image collections (2021).  10.48550/ARXIV.2103.16483.  URL https://arxiv.org/abs/2103.16483
  * [37] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation (2015).  10.48550/ARXIV.1505.04597.  URL https://arxiv.org/abs/1505.04597
  * [38] Bhattiprolu, S.: (2021).  URL https://github.com/bnsreenu/python˙for˙microscopists/blob/master/216˙simple˙unet˙model.py
  * [39] Chiaburu, T.: (2021).  URL https://gitlab.com/kinsecta/ml/wingbeats
  * [40] Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection.  In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 779–788 (2016).  10.1109/CVPR.2016.91

## Appendix

### A. Confusion Matrices

Figure 6: Confusion Matrix MobileNet, test set with cropped images Figure 7:
Confusion Matrix for MobileNet, test set with full images. Although the Top-1
Accuracy (0.8776) is rather high also for full images, the classifier does not
perform very well on some rare species, e.g. Polistes dominula and Episyrphus
balteatus.

### B. Custom CNN Model

Figure 8: Custom CNN architecture - 1.270.992 parameters


# Knowledge-Based Support for Adhesive Selection: Will it
Stick?11footnotemark: 1

SIMON VANDEVELDE<EMAIL_ADDRESS>JEROEN JORDENS
<EMAIL_ADDRESS>BART VAN DONINCK
<EMAIL_ADDRESS>MAARTEN WITTERS
<EMAIL_ADDRESS>JOOST VENNEKENS<EMAIL_ADDRESS>
###### Abstract

As the popularity of adhesive joints in industry increases, so does the need
for tools to support the process of selecting a suitable adhesive. While some
such tools already exist, they are either too limited in scope, or offer too
little flexibility in use. This work presents a more advanced tool, that was
developed together with a team of adhesive experts. We first extract the
experts’ knowledge about this domain and formalize it in a Knowledge Base
(KB). The IDP-Z3 reasoning system can then be used to derive the necessary
functionality from this KB. Together with a user-friendly interactive
interface, this creates an easy-to-use tool capable of assisting the adhesive
experts. To validate our approach, we performed user testing in the form of
qualitative interviews. The experts are very positive about the tool, stating
that, among others, it will help save time and find more suitable adhesives.

###### keywords:

Knowledge Base, First Order Logic, Decision Support, Adhesive Selection

##  1 Introduction

The Flanders Make Joining & Materials Lab (FM JML) is specialized in adhesive
bonding. They support companies in selecting the most appropriate adhesive for
a specific use case, by accounting for characteristics such as strength,
temperature resistances, adhesive durability, and more. Currently, this is
done manually by one of the adhesive experts that work at the lab. Selecting a
suitable adhesive is a time-consuming and labor intensive task, due to the
large number of adhesives available on the market, each with extensive data
sheets. Currently, the experts do not use any supporting tools to help them
perform the selection, because the current generation of tools does not meet
their requirements.

This paper describes our work on a logic-based tool which supports the experts
in the selection process. It is structured as follows. We start by describing
the process of selecting an adhesive and the state-of-the-art tools in Section
2, and elaborate on the logical system used in this work in Section 3. Next,
we present our Adhesive Selector Tool in Section 4, where we discuss the
process of Knowledge Acquisition, how the system handles unknown parameter
values, and how the experts interface with the knowledge. We share the results
of our preliminary three-fold validation in Section 5 and the results of our
comprehensive user study in Section 6. Finally, we describe our lessons
learned in Section 7 and conclude in Section 8.

This paper is an extended version of Vandevelde et al., (2022), as presented
at the LPNMR22 conference. Its main addition is a user-study with the target
users of the JML lab, in the form of semi-structured interviews. Additionally,
we elaborated on some sections and updated the text with various minor
improvements.

##  2 Adhesive Selection and Current Tools

As there is no universally applicable adhesive, the selection of an adhesive
is an important process. There are many factors that influence the choice of
an adhesive: structural requirements such as bonding strength and maximum
elongation, environmental factors such as temperature and humidity, economic
factors, and more. Due to the complexity of the problem, there is quite a
potential for tools that support this selection process. Yet, Ewen J.C.,
(2010) concludes that “there is a severe shortage of selection software, which
is perplexing especially when the task of adhesive selection is so important.”

Currently, when tasked with a use case, the experts work in two steps. First,
they try to identify requirements, such as temperature ranges or values for
parameters like minimum strength. Based on this list of requirements, they
perform an initial selection by manually looking through various data sheets
while keeping track of which adhesives are suitable. In the second step, these
inititally chosen adhesives are put to the test by performing real-life
experiments in FM’s lab, to ensure suitability. However, this testing step is
costly and time-consuming, so it is important that the initial selection is as
precise as possible. While there are tools available for this process, the FM
experts do not use them because they are either too simplistic, or not
sufficiently flexible.

The most straightforward selection tools are websites offering simple
interfaces222such as www.adhesivestoolkit.com and www.adhesives.org da Silva
et al., (2018). Based on a series of questions, they provide advice to support
selection. However, they still require the expert to look up and process the
information themselves.

There are also a number of expert systems to be found in the literature Kannan
and Prabu, (2004); RB and HH, (1995); Su et al., (1993); Moseley and
Cartwright, (1992); Meyler and Brescia, (1993); Lees and Selby, (1993); Lammel
and Dilger, (2002). Here, domain knowledge is captured and formalized in the
form of rules, which can be used for adhesive selection by forward chaining
and often also for generating explanations by backward chaining. However,
these systems have a number of downsides: they are low in both
interpretability and maintainability by the expert, often not all required
knowledge can be expressed, and they generally only contain a low number of
adhesives or substrates. Finally, forward and backward chaining are not
capable of providing all the functionality the expert needs. For instance, a
situation might arise in which an adhesive is already pre-defined (e.g., left-
over from a previous gluing operation), and the selection of a second
substrate is required. While this selection requires the same knowledge, the
expert tools are not capable of performing this operation.

##  3 Knowledge Base Paradigm

The core idea in the Knowledge Base Paradigm (KBP) Denecker and Vennekens,
(2008) is to represent knowledge in a purely declarative way, independently of
how it is to be applied. Knowledge is stored in a Knowledge Base (KB) and can
be put to use via a multitude of inference tasks. In this way, the approach
stimulates knowledge reuse, as multiple inference tasks can be used to solve
multiple problems with the same knowledge.

###  3.1 IDP

The IDP (Imperative Declarative Programming) system De Cat et al., (2018) is
an implementation of the KBP. The knowledge in the KB is represented in a rich
extension of First Order Logic (FOL), called FO($\cdot$) (pronounced FO-dot).
It extends FOL with types, aggregates, inductive definitions and more.
FO($\cdot$) is an expressive and flexible knowledge representation language,
capable of modeling complex domains. The knowledge in a KB is structured in
three kinds of blocks: vocabularies, structures and theories.

A vocabulary specifies a set of symbols. A symbol is either a type, predicate,
or a function. A type is either a standard type such as the set of real
numbers $\mathbb{R}$, or the name of an application-specific type, such as
$Adhesive$. A predicate symbol either expresses a boolean or a relation on one
or more types, such as BondSealing() or Available(Adhesive). A function symbol
represents a function from the Cartesian product $T_{1}\times\ldots\times
T_{n}$ of a number of types to a type $T_{n+1}$. For example, the function
$\mathit{BondStrength}:\mathit{Adhesive}\rightarrow\mathbb{R}$ maps each
adhesive on its bond strength.

A (partial) structure specifies an interpretation for some of the symbols of a
given vocabulary. A structure is total if it specifies an interpretation for
each symbol of the vocabulary.

A theory contains a set of logical formulae in FO($\cdot$).

By itself, the KB cannot be executed: it is merely a “bag of knowledge”,
without information on how it should be used. The latest version of the IDP
system, IDP-Z3 Carbonnelle et al., (2022), supports many different inference
tasks that can be applied to this knowledge. We will briefly go over the
inference tasks that are relevant: propagation, model expansion, optimization
and explanation. Given a partial interpretation $\mathcal{I}$ for the
vocabulary of a theory $\mathit{T}$, propagation derives the consequences of
$\mathcal{I}$ according to $\mathit{T}$, resulting in a more precise partial
interpretation $\mathcal{I^{\prime}}$. Model expansion extends a partial
structure $\mathcal{I}$ to a complete interpretation $I$ that satisfies the
theory T ($I\models T$). Optimization is similar to model expansion, but looks
for the model with the lowest/highest value for a given term. Finally,
explanation will, given a structure $\mathcal{I}$ which does not satisfy the
theory T ($\mathcal{I}\not\models T$), find minimal subsets of the
interpretations in $\mathcal{I}$ which together explain why $\mathcal{I}$ does
not satisfy the theory.

###  3.2 Interactive Consultant

The Interactive Consultant Carbonnelle et al., (2019) is a graphical user
interface for IDP-Z3, which aims at facilitating interaction between a user
and the system. It is a generic interface, in the sense that it is capable of
generating a view for any syntactically correct KB. In short, each symbol of
the KB is represented using a symbol tile, which allows users to set or
inspect that symbol’s value. In this way, the GUI represents a partial
structure to which a user can add and remove values. Each time a value is
added, removed or modified, IDP’s propagation is performed and the interface
is updated: symbols for which the value was propagated are updated
accordingly, and for the other symbols the values that are no longer possible
are removed. In this way, a user is guided towards a correct solution: they
cannot enter a value that would make the partial structure represented by the
current state of the GUI inconsistent with the theory.

At any point in time the user can ask for an explanation of a value that was
derived by the system, e.g., when the user does not understand it or agree
with it. The system will then respond with the relevant formulas and user-made
assignments that lead to the derived value. In this sense, the tool is
explainable, leading to more trust in the system.

A similar functionality is in place for the rare cases in which a user manages
to reach an inconsistent state, i.e., a set of assignments that can no longer
be extended to a solution. While the IC removes values that have become
impossible, it cannot do so for variables belonging to unbounded integer or
real domains. For example, it is possible to input “Min Temperature = 20”
followed by “Max Temperature = 10”, as both of these variables can have any
value between $-\infty$ to $+\infty$. In the case of an inconsistency, the
interface alerts the user and explains why no solutions are possible by
showing the relevant design choices and laws.

The Interactive Consultant interface has already successfully been used in
multiple applications in the past Aerts et al., (2022); Deryck et al., (2019).

##  4 Adhesive Selector Tool

This section outlines the creation and usage of the tool, and the main
challenges that were faced in that process.

###  4.1 Knowledge Acquisition

The creation of the knowledge base is an important element in the development
process of knowledge-based tools. It requires performing knowledge
acquisition, which is traditionally the most difficult step, as the knowledge
about the problem domain needs to be extracted from the domain expert to be
formalized by the knowledge engineer. While knowledge acquisition comes in
many shapes and forms, we applied the Knowledge Articulation method Deryck and
Vennekens, (2022). The central principle of this method is to formalize
knowledge in a common notation for both domain expert and knowledge engineer,
so that both sides actively participate in the formalization process.

We started by organizing three knowledge articulation workshops, each lasting
between three and four hours. Each of these workshops was held with a group of
domain experts. While typically a single domain expert would suffice for
knowledge extraction, having a group present can help as an initial form of
knowledge validation, as the experts discuss their personal way of working
amongst themselves, before coming to a consensus. For the common notation we
used Constraint Decision Model and Notation (cDMN) Vandevelde et al., (2021),
an extension of the Decision Model and Notation (DMN) standard Object
Modelling Group, (2021). DMN is a user-friendly, intuitive notation for simple
decision logic. Its main component is the decision table, which allows to
define the value of a number of “output variables” in terms of a number of
“input variables”. Decision tables are structured together in a Decision
Requirements Diagram (DRD), which is a graph that provides an overview of the
total decision model by showing the connections between input variables
(ovals) and decisions (rectangles). cDMN aims to increase the expressiveness
of DMN (e.g., by adding constraints and quantification) while maintaining this
user-friendliness.

Figure 1: Snippet of created DRD.

The first workshop consisted of identifying all relevant adhesive selection
parameters and using them to create an initial DRD, of which a fragment is
shown in Fig. 1. It is structured in a bottom-to-top way, similar to how the
experts would reason: they start by calculating the thermal expansions, and
then work their way up to the calculation of the maximum stress.

During subsequent workshops, the rest of the model was fleshed out. This
consists of decision tables and constraint tables. An example of such a
decision table can be found in Fig. 2(a). In such a table, the “inputs” (in
green, left) define the “outputs” (in light blue, right). Each row represents
a decision rule, which fires if the values of the input variables match the
values listed in the row. If a row fires, the value of the output is set
accordingly. E.g., if $\mathit{Support}=\mathit{fixed}$, then the
MinElongation is calculated as $\mathit{deltaLength}/\mathit{BondThickness}$.
The (U)nique hit policy of this table, indicated in the top left, means that
the different rows must be mutually exclusive.

(a) Calculation of MinElongation

(b) BondStrength constraint table.

Figure 2: Example cDMN tables

Fig. 2(b) shows a constraint table (denoted by the E* in the top-left corner).
In such a table, the output specifies a constraint that must hold if the input
is satisfied. In other words, this table states that if the bond strength is
known, then Max Stress should be higher than a minimum value. This differs
from decision tables in that it does not define a specific value, but rather
constrains its possible values.

After these three initial workshops, the cDMN model was converted into an
FO($\cdot$) KB using the cDMN conversion tool. Since then, multiple one-on-one
workshops were held between the knowledge engineer (first author) and the
primary domain expert (second author) to further fine-tune the KB. Among other
things, this included adding a list of adhesives, substrates, and their
relevant parameter values, and further validating the knowledge. In total, the
current version of the KB contains information on 55 adhesives and 31
substrates. For the adhesives, the KB contains 21 adhesive parameters, such as
temperature resistances, strength and maximum elongation. Similarly, it
contains 11 parameters for the substrates, such as their water absorption and
their solvent resistance. These parameters are a mix of discrete and
continuous: in total, 15 are continuous, and 17 are discrete.

###  4.2 Unknown adhesive parameters

One of the main challenges in the formalization of the KB was handling unknown
adhesive data. Indeed, often an adhesive’s data sheet does not list all of its
properties. This raises the question of how the tool should deal with this:
should the adhesive be excluded, or should it simply ignore the constraints
that mention unknown properties? Together with the experts we agreed on a
third approach, in which we first look at the adhesive’s family. Each adhesive
belongs to one of 18 families, for which often some indicative parameter
values are known. Whenever an adhesive’s parameter is unknown, we use its
family’s value as an approximation. If the family’s value is also unknown,
then the constraint is ignored. This best corresponds to how the experts
typically work.

This way of reasoning is formalized in the KB. For example, the constraint
that an adhesive should have a minimum required bonding strength is written as
follows:

$\displaystyle\forall
p\in\mathit{param}:\mathit{Known}(\mathit{p})\Leftrightarrow$
$\displaystyle(\mathit{KnownAdhesive}(\mathit{p})\lor\mathit{KnownFamily}(\mathit{p}))$
$\displaystyle\mathit{KnownAdhesive}(\mathit{strength})\Rightarrow\mathit{BondStrength}=$
$\displaystyle\mathit{StrengthAdhesive}(\mathit{Adhesive}).$
$\displaystyle\neg\mathit{KnownAdhesive}(\mathit{strength})\Rightarrow\mathit{BondStrength}=$
$\displaystyle\mathit{StrengthFamily}(\mathit{Family(Adhesive)}).$
$\displaystyle\mathit{Known}(\mathit{strength})\Rightarrow\mathit{BondStrength}\geq$
$\displaystyle\mathit{MinBondStrength}.$

with StrengthAdhesive and StrengthFamily representing respectively the
specific adhesive’s and its family’s bonding strength. This approach is used
for all 21 adhesive parameters.

One caveat to this approach is that IDP-Z3 currently does not support partial
functions, i.e., all functions must be totally defined. To overcome this, we
assign the value -1000 to unknown parameter values, and define that the value
is only known if it is different from this number. We chose -1000 as there is
no adhesive parameter for which it is a realistic value.

$\displaystyle\forall
p\in\mathit{param}:\mathit{KnownAdh(p)}\Leftrightarrow\mathit{StrengthAdhesive(Adhesive)}\neq-1000.$
$\displaystyle\forall
p\in\mathit{param}:\mathit{KnownFam(p)}\Leftrightarrow\mathit{StrengthFamily(Adhesive)}\neq-1000.$

(a) Some of the symbol tiles available in the interface, divided into two
categories.

(b) List of remaining suitable adhesives during selection.

(c) Example of explanation.

(d) Inconsistency window.

Figure 3: Screenshots of the interface.

###  4.3 Interface

A crucial requirement of this application is the ability to interactively
explore the search space. To this end, our tool integrates the Interactive
Consultant to facilitate interaction with the KB. This interface makes use of
several functionalities of the IDP system to make interactive exploration
possible: the propagation inference algorithm is used to show the consequences
of each choice, the explain inference is used to help the user understand why
certain propagations were made, the optimize inference is used to compute the
best adhesive that matches all of the choices made so far.

When using the interface, the user fills in symbol tiles, each representing a
different symbol of the KB, and the system each time computes the
consequences. For example, Fig. 3(a) shows a segment of the interface in which
a user set a maximum application temperature of $38^{\circ}C$ as a
requirement. To make it easier to navigate the symbol tiles, they are all
divided in five categories: Performance, Production, Bond, Substrate A and
Substrate B. In the top-right of the interface, the number of adhesives that
remain feasible is shown: e.g., after setting the temperature constraint, that
drops from 55 to 12, as shown in Fig. 3(b).

The tool is also capable of generating two types of explanations. Firstly, if
the user does not understand why a certain value was propagated, they can
click on that value to receive a clarification, as demonstrated in Fig. 3(c).
Secondly, if the user manages to reach an inconsistent state, the tool will
try to help resolving the issue by listing what is causing it. For example,
Fig. 3(d) shows an inconsistency in which a substrate is selected that cannot
handle the required operating temperature.

Besides generating a list of all the adhesives that meet certain requirements,
the tool can also find the optimal adhesive according to a specific criterion,
such as lowest price or highest strength.

##  5 Preliminary Validation

Initially, we performed three types of validation for this tool: a benchmark
to measure the efficiency, a survey to measure the opinion of the adhesive
experts and a discussion with the Flanders Make AI project lead.

##### Benchmark

In an initial benchmark, an adhesive expert was tasked with finding a suitable
adhesive for an industrial use case which the company received. In total, it
took the expert about three hours to find such an adhesive, after delving
through multiple data sheets. We then used our tool for the same use case, and
were able to find the exact same adhesive within three minutes. Interestingly,
the reasoning of the tool closely mimicked that of the expert: for example,
they both excluded specific families for the same reasons.

##### Survey

After a demonstration of the tool to four adhesive experts, we asked them to
fill out a short quantitative survey to better gauge their opinion. Their
answers can be summarized as follows.

  * •

The experts find the tool most useful for finding an initial list of adhesives
to start performance testing with.

  * •

The tool will be most useful for newer, less knowledgeable members of the lab.
They can use the tool to learn more about the specifics of adhesive selection,
or to verify if their result is correct.

  * •

However, it is also useful for senior experts as they can discover adhesives
which they have not yet used before.

The main criticism of the tool given by the experts is that more adhesives
should be added, to make the selection more complete.

##### Project Lead Discussion

As part of a discussion with Flanders Make’s project lead, who oversees
multiple AI-related projects, they outlined their perception of our tool. They
see many advantages. Firstly, as there is not much data available on the
process of adhesive selection (e.g., previous use cases and the selected
adhesives), and data generation is quite expensive, data-based approaches are
not feasible. Therefore, building a tool based on a formalization of the
knowledge they already have is very interesting. Secondly, by “storing” the
expert knowledge formally in a KB they can retain this information, even when
experts leave the company. Thirdly, having a formal representation also makes
the selection process more uniform across different experts, who typically use
different heuristics or rules-of-thumb. Lastly, they indicated that there is
trust in the system, because the knowledge it contains is tangible. This makes
it more likely that the experts will agree with the outcome of the tool.

The project lead also expressed that there is potential to maintain and extend
this tool themselves, which would be a significant advantage compared to their
other AI systems. However, we currently have not yet focused on this aspect.

##  6 User Study

On top of the preliminary validation presented in Section 5, we performed a
user study with members of the Flanders Make JML group. This study
distinguishes itself from the preliminary validation in two ways: (1) all
experts actually got to use the tool themselves and (2) it is a qualitative
study instead of quantitative w.r.t. the input of the experts. Our main
motivation for this study is to thoroughly validate the Adhesive Selector,
with the following goals in mind:

  1. 1.

Gauge the tool’s effectiveness in a real-life setting.

  2. 2.

Observe how new users interact with the tool.

  3. 3.

Get feedback on the different aspects of the tool (interactivity, interface,
explanations, …).

For this validation, we asked the FM JML members to work out two real-life use
cases, after which we performed one-on-one semi-structured interviews (SSI).
In the following subsections we first elaborate on our methodology, then
discuss the results of our interviews followed by describing the limitations
of our approach.

###  6.1 Methodology

In our study, we held interviews with four members of the FM JML, who each
possess knowledge on adhesives but have varying degrees of involvement in
adhesive selection. Two interviewees are adhesive experts that often perform
adhesive selection. The other two do not perform adhesive selection as part of
their job, but are knowledgeable on glues in general. We included such “non-
experts” to explore whether the tool is capable of making adhesive selection
more accessible. We held an online one-on-one session with each of the
interviewees. First the interviewee was asked to perform adhesive selection
using the tool, and then we conducted a semi-structured interview to gather
their opinions.

##### Adhesive selection.

Two real-life bonding cases were selected by the second author to be used as
test cases for the study: both are cases that the JML lab has received from
companies in the past. In the first one, a plastic door needs to be glued to
the body of an industrial harvester. Originally, the process of finding the
correct glue took weeks, as the requirements are fairly tight. Moreover, the
specification contains an inconsistency in which a higher temperature is
required than allowed by the substrate, which is the same inconsistency as
shown in Fig. 3(d). The second case details a join between a plastic component
and an aluminum body, and is less challenging.

Both cases were presented to the participants as a short description of the
gluing operation, together with a table containing the actual requirements.
For the first example, the table lists requirements such as “Material A =
Virgin ABS”, “Gap filling of min 1 mm”, “Application between $15^{\circ}C$ to
$35^{\circ}C$”, etc. We have taken care here to specify requirements in the
same terminology as the tool. In total, the use cases consist of nine and
seven requirements respectively.

Before letting the experts work on the cases, we also gave a brief
introduction on how to use the tool. We explained how to enter information,
where they could find the list of possible adhesives and how to see that the
tool was performing calculations.

We observed each tester during the selection process, to gather information on
how they interacted with the tool. To better understand their thought process,
the interviewees were encouraged to talk to themselves about their process as
a way to elicitate their thoughts. We did not intervene while they were
working out these use cases, even when errors were made, and only made
suggestions on what to do if they got stuck or an unexpected bug popped up.

##### Interviews.

Right after the selection, we held an interview with each participant. Because
our goal is to explore the opinions of the JML members, we opted for a semi-
structured interview set-up, for which we prepared four main questions to
serve as a general guide:

  1. 1.

Do you see a role for the tool in your job?

  2. 2.

Do you feel that you understand what the tool does when you use it?

  3. 3.

What was your experience working with the tool?

  4. 4.

How would you compare our tool to the ones that you are used to working with?

In addition to these main questions, we asked additional questions to zoom in
on specific aspects of the answers given by the participants. The interviews
were led by the first author, with additional support from the second author.
They were audio-recorded and transcribed, so that they could be analysed
thoroughly.

After transcribing the recordings, we followed the guidelines of Richards and
Hemphill, (2018) and performed open coding followed by axial coding Corbin and
Strauss, (1990). Here, the goal is to identify various codes that pop up
during the interviews, and then further group them into several main
categories. These two steps were also performed separately by an external
member of the research group, after which the results were compared and
adapted to mitigate biases (consensus coding).

###  6.2 Results

Based on the interview transcripts, we identified 26 codes in total. Table 1
shows an overview of the interview statistics. The last column of this table
shows the Code distribution, calculated as the cumulative percentage of codes
discovered after each interview. This is an important parameter that indicates
whether data saturation is reached, i.e., the point at which the same themes
keep recurring and new interviews would not yield new results. According to
Guest et al., (2020), this point is reached when the difference in code
distribution between the current and previous interview is $\leq 5\%$ (i.e.,
less than $5\%$ of new information was found). As the information threshold
between the third and fourth interview is 4%, we conclude that we have reached
data saturation. A more detailed table showing the codes per interview is
included in the Appendix.

Table 1: Interview statistics |  |  |  |
---|---|---|---|---
Use Case Duration (min) | Interview Duration (min) | Words | Total codes | Code distribution
31 | 26 | 2092 | 15 | 58%
55 | 39 | 2107 | 16 | 88%
66 | 47 | 3497 | 19 | 96%
41 | 49 | 3680 | 13 | 100%
|  |  |  |
|  |  |  |

We sub-divided the codes into five main themes: Knowledge, Interactivity,
Expert, Interface and Explainability. These themes, together with their codes,
are visualised in the graph in Fig. 4. We will now briefly go over each theme
and highlight the most important findings.

Figure 4: Graph showing connections between the interview themes and their
codes.

####  6.2.1 Knowledge

A major advantage of our tool is that it reasons on the knowledge of the
adhesive experts “like they themselves would”. This is affirmed by the
participants, who particularly liked its level of detail in two ways. Firstly,
instead of being limited to selecting adhesive families, as is the case in
other selector systems, our tool can help them find specific adhesives.
Secondly, the number of parameters available in the tool is unparalleled,
allowing for a more fine-grained search.

“[…] the good thing that I like about this tool is that it’s quite detailed.
It’s really one or multiple steps further, multiple steps deeper than these
other tools. […] So there it’s super helpful, definitely.”

Similarly, the tool can also efficiently reason on a broader number of
specific adhesives than an expert. As the latter typically knows some
approximate parameter values by heart for a handful of glues and families,
they tend to look at these adhesives first before widening their search to
others when needed. Here, the Adhesive Selector can help them to find
additional adhesives that they would not have considered without the tool,
while also saving them from having to manually go through their data sheets.

“You might encounter situations in which you find an adhesive that you hadn’t
thought about before. That quite increases your search scope, I think. It
could also help in mitigating bias.”

“If you want to understand the glue that you want to use you have to read a
lot of datasheets, while having this tool I think optimized the time in a way
that is crazy.”

The main criticism expressed by the participants is that the number of
specific adhesives in the knowledge base is still rather limited. While 55
adhesives is a good start, adding more adhesives will definitely make the tool
more effective. Similarly, some participants remarked that the knowledge base
should be with more expert knowledge such as the environmental impact and re-
usability of adhesives.

####  6.2.2 Interactivity

Throughout the interviews, interactivity was an often recurring topic. For
example, one aspect that the participants all appreciated was that they could
immediately see the effects of entering requirements in the interface.

“Interactively choosing the glue now feels like online shopping – I can select
more options, and see the total number of adhesives go down until I have
entered all requirements.”

Besides making the number of adhesives go down, each time a new requirement is
entered the interface also greys out parameters that have become irrelevant,
and removes parameter values that are no longer possible. This immediate
feedback helps preventing mistakes in the selection.

“I think it’s easier to spot if there are some problems, like the ones that
popped up.”

“It’s safer, you avoid wasting materials and time.”

Moreover, the immediate feedback also allows the experts to “play around” with
the knowledge in the tool. In this way, they can get a feel for the effect of
certain parameter values on the suitable adhesives.

“Us engineers typically want to play with things. They want to see what
happens when they change something, thereby implicitly performing a
sensitivity analysis.”

However, some participants felt that always having to enter the requirements
one-by-one is too inefficient. While they all agreed that the tool is
sufficiently fast, they stated that they would like to “bulk update” choices
to be more efficient in cases where they already know that they are correct.

####  6.2.3 Expert

As our tool is designed to support JML’s experts based on their own knowledge,
they have played a big role in the creation of the tool and will continue to
play a big role in its further existence. It is necessary to always have an
expert in the loop: the tool cannot be used by laymen, who lack the specific
knowledge required to extract requirements from a description of a use case
and who cannot understand the technical jargon.

Besides supporting the experts in making suitable selections, the participants
described two other ways in which the Adhesive Selector can help them.
Firstly, the tool covers sufficient knowledge to also assist in the design of
the entire joint. This is different from adhesive selection in that many
“environmental parameters” are still left open, e.g., substrates might need to
be picked, a decision on joining method needs to be made, etc.

“This tool can already help me to list all these requirements [required for
the dimensioning of the joint] so that I have a bird’s eye view of the whole
design.”

The second additional use of the tool identified by the participants is its
potential to be used as a teaching tool. For instance, newer members of the
lab could use it to gain their footing when starting out.

“You can use it to teach people ‘If you select these requirements, these are
the consequences, which means you can no longer use these adhesives’.”

“[The tool] can also give some confidence, if they say ‘I would select this’
and the tool confirms it, you would feel more certain in your selection”

####  6.2.4 Interface

Having the participants talk out loud while working the use cases proved to be
a valuable source of information, allowing us to gain insight in how someone
without prior experience interacts with the tool. Some of these insights are
fairly minor, e.g., that we should order the values of the drop down lists
alphabetically and support folding in/out entire categories to make the
interface easier to navigate. Other insights are more major, such as how much
difficulty the participants experienced with the structure of the interface –
due to the large number of parameter tiles (78 in total), it typically took
them around 10 seconds to find the right one. Moreover, the participants often
lost overview of which choices they had already entered in the system, as
these are spread all over the interface.

One positive note that the experts really liked was the automatically updating
counter showing the remaining number of suitable adhesives:

“I can look at the number of adhesives and see that we are converging,
converging, converging. So it’s fun, to say it like that.”

This ties in nicely with the idea of the Adhesive Selector as a didactical
tool: if a choice rules out many adhesives at once, the user can assume it is
more “important” than a choice that only removes a handful. Moreover, another
participant suggested using diagrams to annotate symbols with their intended
meaning, e.g., a diagram showing two bonded substrates with arrows pointing to
the “bond line”, the “bonding surface”, etc. This would make the tool more
self-explanatory to people less familiar with these terms, such as the newer
members of the team.

####  6.2.5 Explainability

While explainability is one of the focuses of our knowledge-based approach,
the experts were not yet fully convinced of this functionality. When prompted
with the “inconsistency” window (Fig. 3(d)), none of the participants knew
what to do. Only two of them quickly understood what the cause of the
inconsistency was, but none were able to resolve it by themselves. As one
participant later remarked in an interview:

“There were multiple sentences below each other, I didn’t know if it was three
remarks or a single one. […] I was confused, and could not see the information
I needed”

In other words, they had some difficulties navigating the inconsistency
window: partly due to its layout, but also due to the complexity of the
knowledge. However, they did appreciate the potential that this feature holds,
for example to assist in experimenting with the knowledge.

The difficult explanations also did not have an impact on the expert’s trust
in the system. Indeed, it seems that it is more important to know that its
behaviour is derived from the lab’s own knowledge, than to actually 100%
understand the explanations.

“I trust it, because it contains our knowledge. So, well, I do trust it, but
only because I know it’s ours.”

##### Other Results

A participant with a slightly more business background pointed out an
interesting result that could not be codified under the other themes. They see
the tool as a “uniform way of collecting data relevant to adhesive selection”,
which could help drive the team’s decision making. Examples of relevant
information are the use cases performed by experts, the substrates that were
used, which adhesives are typically picked, what constraints were present,
etc. This data could be useful for:

  * •

finding trainings specifically for the most commonly picked adhesives

  * •

identifying target industries that could also be helped by JML

  * •

deciding which adhesives to keep in stock, and what equipment should be
purchased

This is a unique take that we had not yet considered for our tool.

###  6.3 Limitations of the Study

As with any study, ours is not without its limitations. The main limitation of
this study is our low number of interviewees. However, as pointed out in
Section 6.2, we do reach data saturation when looking at the code distribution
of our interviews. Therefore, we feel that the low number of interviewees does
not have a major impact.

Another limitation is that all interviewees are from the same organization and
would therefore have a “common” approach to selecting glues, while external
experts might have a different focus that could result in additional feedback.
However, as the tool has specifically been developed for use within Flanders
Make JML, we believe that this feedback would be less relevant.

During the testing of the tool, we continually observed the interviewees to
study their interaction with the tool. Because of this, the users might have
felt pressured to work in a more “efficient” way, as a form of the Hawthorne
effect McCarney et al., (2007). In fact, one participant said so explicitly,
when talking about their experience with the inconsistency window:

“I think that if I were alone, in the lab, I would have taken more time to
read the pop-up. I wanted to be a bit quick.”

Because we were observing, we might have inhibited the users to truly “play
around” with the tool and test it to their heart’s content.

##  7 Lessons Learned

Typically, knowledge acquisition is a time-consuming and difficult process. We
have found that the use of cDMN as a common notation can help facilitate this
process. The use of a formal representation that the experts can also
understand helps to keep them in the loop and allows them to actively
participate in the formalization process. This way of working is less error-
prone, as it functions as a preliminary validation of the knowledge.

After our three initial workshops, we mainly held one-on-one meetings with one
of the experts to add information on the adhesives, and to further fine-tune
the knowledge. This resulted in a tight feedback loop, which turned out to be
a key element in our formalization. Indeed, thanks to thorough examinations of
the tool by the expert, we were able to discover additional bugs in our KB.
Here, the Interactive Consultant was of paramount importance: each time the KB
was modified, the expert could immediately play around with it using the
generic interface. In this way, the knowledge validation of the tool could
happen immediately after the modifications, allowing for a swifter detection
of any errors.

Having knowledge in a declarative format, independent of how it will be used,
has multiple advantages. To begin with, it allows using the knowledge for
multiple purposes, even when this initially might not seem useful.
Furthermore, it increases the experts’ trust in the system, as it reasons on
the same knowledge as they do, and is interpretable.

The main advantage of using IDP-Z3 does not lie in any one of its inference
algorithms, but rather in the fact that it allows all of the functionalities
that are required for interactive exploration of the search space to be
performed by applying different inference algorithms to a single KB.

The validation of the tool by the actual end-users proved to be a source of
valuable feedback. Through our observations, we have gained insights on how
the users interact with the tool. By means of semi-structured interviews, we
gathered their opinions, thoughts and suggestions. These two sources of input
combined will help shape further development of our tool.

##  8 Conclusions and Future Work

This paper presents the Adhesive Selector, a tool to support adhesive
selection using a knowledge-based approach. The Knowledge Base was constructed
by conducting several workshops and one-on-one meetings, using a combination
of DMN and cDMN. Our current iteration of the tool contains sufficient
knowledge to assist an expert in finding an initial list of adhesives.
Compared to the state of the art, it is declarative, more explainable, and
more extensive. The KB is also not limited to just adhesive selection, but can
also be used to perform other related tasks.

In future work, we plan on converting the entire FO($\cdot$) KB into cDMN, and
evaluating its readability and maintainability from the perspective of the
domain experts. Besides this, we intend to test the tool using more real-life
use cases, to quantify the gain in efficiency. Additionally, we are also
collaborating with an external research group to develop an AI-based tool
capable of extracting adhesive parameter values from data sheets, to
efficiently add more adhesives to our KB. Flanders Make is planning on using
the tool in production soon, and expanding the list of available adhesives is
the only remaining bottle-neck. Note that such information is only a part of
the knowledge that is required for selecting a suitable adhesive, and other
forms of expert knowledge will continue to be added throughout the lifetime of
the tool through the methods described in the text.

Competing interests: The authors declare none.

## Acknowledgements

This research received funding from the Flemish Government under the
“Onderzoeksprogramma Artificiële Intelligentie (AI) Vlaanderen” programme and
from Flanders Make vzw. We would also like to thank Wouter Groeneveld for his
suggestions on the interview structure, Benjamin Callewaert for cross-
validating the codes and themes, and the members of the Joining and Materials
Lab for participating in our study.

##  9 Appendix

Table 2 shows an overview of each code and the interviews in which it
appeared, and Table 3 contains the “code book” explaining each of the codes.

While we cannot release the knowledge base itself, as it contains sensitive
data from our external partner, the tools and techniques used in this paper
are open-source and available online. We refer anyone interested in trying out
IDP-Z3333https://www.IDP-Z3.be and https://gitlab.com/krr/IDP-Z3/, the
Interactive Consultant444https://interactive-consultant.idp-z3.be/ or
cDMN555https://www.cdmn.be and https://gitlab.com/EAVISE/cdmn/cdmn-solver to
their online resources.

Table 2: Discovered codes per participant, and the data saturation. |  |  |  |
---|---|---|---|---
Code | 1 | 2 | 3 | 4
Cluttered Interface | ✓ | ✓ | ✓ | ✓
Interactive Counter | ✓ | ✓ | ✓ |
Time Gain | ✓ | ✓ | ✓ | ✓
Avoiding Mistakes | ✓ |  |  |
Playing Around | ✓ |  | ✓ |
More Confident | ✓ |  | ✓ | ✓
Understandable | ✓ |  |  |
Trust in the System | ✓ | ✓ | ✓ | ✓
Verification of the Result | ✓ | ✓ | ✓ | ✓
Adequate Performance | ✓ | ✓ | ✓ | ✓
Broader Scope | ✓ |  | ✓ | ✓
Understanding Knowledge | ✓ |  |  | ✓
Teaching Tool | ✓ |  | ✓ | ✓
Explainability | ✓ | ✓ | ✓ |
Difficult Explanations | ✓ | ✓ | ✓ | ✓
Extended DB Required |  | ✓ | ✓ |
Select Specific Adhesives |  | ✓ | ✓ |
Help in Design |  | ✓ |  |
Expert in the Loop |  | ✓ | ✓ | ✓
Interactivity |  | ✓ | ✓ |
Relevancy |  | ✓ |  |
More Detail |  | ✓ | ✓ | ✓
User-added Knowledge |  | ✓ |  |
Flexibility |  |  | ✓ |
Immediate Feedback |  |  | ✓ |
Central Place for Knowledge |  |  |  | ✓
Total codes | 15 | 16 | 19 | 13
Data saturation | 58% | 88% | 96% | 100%
|  |  |  |
|  |  |  |
|
---|---
Code | Explanation
Cluttered Interface | The interface’s structure seems chaotic.
Interactive Counter | The counter denoting the number of suitable adhesives.
Time Gain | Experts can more efficiently select an adhesive.
Avoiding Mistakes | The tool prevents making mistakes.
Playing Around |  Interacting with the knowledge allows the expert to “play around” with it.
More Confident | The experts are more sure of their selection using the tool.
Understandable |  The interface presents information in a sufficiently understandable way
Trust in the System | The experts trust the choices of the system.
Verification of the Result |  The resulting short-list of adhesives will always need to be verified.
Adequate Performance | The loading time of the tool is within reason.
Broader Scope | The experts discover more adhesives.
Understanding Knowledge | The tool helps understand the knowledge.
Teaching Tool | The tool is well-suited for teaching newer members of the lab.
Explainability | Explainability is an important factor.
Difficult Explanations | The automatically generated explanations are difficult.
Extended DB Required | More adhesives should be added to the system.
Select Specific Adhesives |  The AS allows selecting specific adhesives, whereas the other tools only select families.
Help in Design | The tool can also play a role in joint design.
Expert in the Loop | An expert is always required to stay in the loop.
Interactivity | Interactivity is a crucial aspect of the tool.
Relevancy |  Detecting which parameters are still relevant aids in selecting adhesives.
More Detail |  The tool is more detailed than any other in terms of possible parameters.
User-added Knowledge |  The experts would like to add their own knowledge to the system.
Flexibility | The tool adapts well to the expert’s way of working.
Immediate Feedback |  The experts appreciate that the interface updates automatically.
Central Place for Knowledge |  All knowledge for adhesive selection is stored in one location.
|
|
Table 3: A “code book” elaborating on the meaning of each code.

## References

  * Aerts et al., (2022) Aerts, B., Deryck, M., and Vennekens, J. 2022\.  Knowledge-based decision support for machine component design: A case study, 187, 115869.
  * Carbonnelle et al., (2019) Carbonnelle, P., Aerts, B., Deryck, M., Vennekens, J., and Denecker, M. An Interactive Consultant.  In Proceedings of the 31st Benelux Conference on Artificial Intelligence (BNAIC 2019) and the 28th Belgian Dutch Conference on Machine Learning (BENELEARN 2019), Brussels, Belgium, November 6-8, 2019 2019, volume 2491 of CEUR workshop proceedings. CEUR-WS.org.
  * Carbonnelle et al., (2022) Carbonnelle, P., Vandevelde, S., Vennekens, J., and Denecker, M. 2022\.  IDP-Z3: a reasoning engine for FO(.).
  * Corbin and Strauss, (1990) Corbin, J. M. and Strauss, A. 1990\.  Grounded theory research: Procedures, canons, and evaluative criteria, 13, 1, 3–21.
  * da Silva et al., (2018) da Silva, L. F. M., Öchsner, A., and Adams, R. D. 2018\.  Handbook of adhesion technology, volume 1.  Springer.
  * De Cat et al., (2018) De Cat, B., Bogaerts, B., Bruynooghe, M., Janssens, G., and Denecker, M. Predicate logic as a modeling language: the IDP system.  In Kifer, M. and Liu, Y. A., editors, Declarative Logic Programming: Theory, Systems, and Applications 2018, pp. 279–323. ACM.
  * Denecker and Vennekens, (2008) Denecker, M. and Vennekens, J. Building a Knowledge Base System for an Integration of Logic Programming and Classical Logic.  In Garcia de la Banda, M. and Pontelli, E., editors, Logic Programming 2008, volume 5366, pp. 71–76. Springer Berlin Heidelberg, Berlin, Heidelberg.  Series Title: Lecture Notes in Computer Science.
  * Deryck et al., (2019) Deryck, M., Devriendt, J., Marynissen, S., and Vennekens, J. Legislation in the knowledge base paradigm: interactive decision enactment for registration duties.  In Proceedings of the 13th IEEE Conference on Semantic Computing 2019, pp. 174–177. IEEE.
  * Deryck and Vennekens, (2022) Deryck, M. and Vennekens, J. An integrated method for knowledge management in product configuration projects.  In Andersen, A.-L., Andersen, R., Brunoe, T. D., Larsen, M. S. S., Nielsen, K., Napoleone, A., and Kjeldgaard, S., editors, Towards sustainable customization: Bridging smart products and manufacturing systems 2022, pp. 860–868, Cham. Springer International Publishing.
  * Ewen J.C., (2010) Ewen J.C., K. Key issues in selecting the right adhesive.  In Dillard, D. A., editor, Advances in structural adhesive bonding 2010, Woodhead publishing in materials, pp. 3–19. Woodhead Publishing.
  * Guest et al., (2020) Guest, G., Namey, E., and Chen, M. 2020\.  A simple method to assess and report thematic saturation in qualitative research, 15, 5, e0232076.
  * Kannan and Prabu, (2004) Kannan, T. and Prabu, S. S. Expert system for selection of adhesives.  In Proceedings of the Recent Developments in Materials Processing Conference 2004\.
  * Lammel and Dilger, (2002) Lammel, C. and Dilger, K. 2002\.  Software for a rule-based adhesive-selection system., 9, 5, 42–43.  Publisher: Business News Publishing Company.
  * Lees and Selby, (1993) Lees, W. and Selby, P. 1993\.  The PAL program mark II, 13, 2, 120–125.  Publisher: Elsevier.
  * McCarney et al., (2007) McCarney, R., Warner, J., Iliffe, S., Van Haselen, R., Griffin, M., and Fisher, P. 2007\.  The Hawthorne Effect: a randomised, controlled trial, 7, 1, 1–8.
  * Meyler and Brescia, (1993) Meyler, K. L. and Brescia, J. A. 1993\.  Design of a computer expert system for adhesive selection using artificial intelligence techniques.  Technical report, Army armament research development and engineering center Picatinny Arsenal.
  * Moseley and Cartwright, (1992) Moseley, L. and Cartwright, M. 1992\.  The development of an expert system for operational use in the selection of industrial adhesives, 5, 4, 319–328.  Publisher: Elsevier.
  * Object Modelling Group, (2021) Object Modelling Group 2021\.  Decision model and notation v1.3.
  * RB and HH, (1995) RB, A. and HH, V. 1995\.  Expert-system selects adhesives for composite-material joints, 38, 7, 16–19.  Publisher: Intertec Publishing Corp. PRIMEDIA Co., Chicago, USA.
  * Richards and Hemphill, (2018) Richards, K. A. R. and Hemphill, M. A. 2018\.  A Practical Guide to Collaborative Qualitative Data Analysis, 37.
  * Su et al., (1993) Su, Y., Srihari, K., and Adriance, J. 1993\.  A knowledge update mechanism for an adhesive advisor, 25, 1-4, 111–114.  Publisher: Elsevier.
  * Vandevelde et al., (2021) Vandevelde, S., Aerts, B., and Vennekens, J. 2021\.  Tackling the DM challenges with cDMN: A tight integration of DMN and constraint reasoning.  Publisher: Cambridge University Press (CUP).
  * Vandevelde et al., (2022) Vandevelde, S., Jordens, J., Van Doninck, B., Witters, M., and Vennekens, J. Knowledge-based support for adhesive selection.  In Gottlob, G., Inclezan, D., and Maratea, M., editors, Logic Programming and Nonmonotonic Reasoning 2022, pp. 445–455, Cham. Springer International Publishing.