# Convolutional Neural Networks:
Basic Concepts and Applications in Manufacturing

Shengli Jiang†, Shiyi Qin†, Joshua L. Pulsipher‡, and Victor M. Zavala†
†Department of Chemical and Biological Engineering
University of Wisconsin-Madison, 1415 Engineering Dr, Madison, WI 53706, USA
‡Department of Chemical Engineering
Carnegie Mellon University, 5000 Forbes Ave, Pittsburgh, PA 15213, USA
Corresponding Author<EMAIL_ADDRESS>
###### Abstract

We discuss basic concepts of convolutional neural networks (CNNs) and outline
uses in manufacturing. We begin by discussing how different types of data
objects commonly encountered in manufacturing (e.g., time series, images,
micrographs, videos, spectra, molecular structures) can be represented in a
flexible manner using tensors and graphs. We then discuss how CNNs use
convolution operations to extract informative features (e.g., geometric
patterns and textures) from the such representations to predict emergent
properties and phenomena and/or to identify anomalies. We also discuss how
CNNs can exploit color as a key source of information, which enables the use
of modern computer vision hardware (e.g., infrared, thermal, and hyperspectral
cameras). We illustrate the concepts using diverse case studies arising in
spectral analysis, molecule design, sensor design, image-based control, and
multivariate process monitoring.

Keywords: computer vision, convolutional neural networks, manufacturing,
images, graphs.

##  1 Introduction

Manufacturing is seeing an increasing use of real-time sensing and
instrumentation technologies that generate data in the form of images/video
(e.g., infrared and thermal), vibration/audio, and other complex data forms
such as chemical spectra and geometrical structures (e.g., 3D printed objects,
synthesized molecules, crystals). Manufacturing is also seeing the increasing
use of automation systems that aim to exploit such data to make decisions
(e.g., optimize production and detect anomalies). Moreover, modern automation
systems are being designed to take instructions/targets in the form of complex
data objects (e.g., voice, text, chemical spectra, and molecular structures).

Modern automation systems used in manufacturing embed highly sophisticated
computing workflows that use tools from data science and machine learning (ML)
to extract and interpret actionable information from complex data streams.
Such workflows resemble those used in other advanced technologies such as
autonomous vehicles (e.g., aerial, terrestrial, and aquatic) and robotics.
Moreover, such workflows begin to resemble human systems in which visual,
auditive, tactile, and olfactory signals (data) are routinely used to make
decisions. For instance, the human olfactory system generates signals when
exposed to specific chemical structures and such signals are processed and
interpreted by the brain to detect anomalies. Similarly, the human visual
system generates interpretable signals when exposed to objects with specific
geometrical and color features and our auditory system generates interpretable
signals when exposed with specific frequencies. As such, from a conceptual
stand-point, we can see that sensing and data science technologies are
enabling increasing convergence between industrial (artificial) and human
(natural) perception and decision-making. This opens new and exciting
opportunities to synergize human and artificial intelligence with the ultimate
goal of making manufacturing more efficient, safe, sustainable, and reliable.

In this chapter, we focus on ML technologies that enable information
extraction from complex data sources commonly encountered in manufacturing.
Specifically, we review basic concepts of convolutional neural networks (CNNs)
and outline how these tools can be used to conduct diverse decision-making
tasks of interest in manufacturing. At their core, CNNs use a powerful and
flexible mathematical operation known as convolution to extract information
from data objects that are represented in the form of regular grids (1D
vectors, 2D matrices, and high-dimensional tensors) and irregular grids (2D
graphs and high-dimensional hypergraphs). These data representations are
flexible and can be used to encode a wide range of data objects such as audio
signals, chemical spectra, molecules, images, and videos. Moreover, such
representations can encode multi-channel data, which allows capturing color
and multi-variate inputs (e.g., multi-variate time series and molecular
graphs). CNNs use convolution operations to identify features that can be
extracted from the data that best explain an output; such features are
identified by identifying optimal convolution filters or operators, which are
the parameters that are learned by the CNN. The learning process of the
operators requires sophisticated optimization algorithms and can be a
computationally expensive process. CNNs is a class of models of an emergent ML
field known as geometric deep learning, which leverages tools from geometry
and topology to represent and process data.

The earliest version of a CNN was proposed in 1980 by Kunihiko Fukushima [21]
and was used for pattern recognition. In the late 1980s, the LeNet model
proposed by LeCun et al. introduced the concept of backward propagation, which
streamlined learning computations using optimization techniques [45]. Although
the LeNet model had a simple architecture, it was capable of recognizing hand-
written digits with high accuracy. In 1998, Rowley et al. proposed a CNN model
capable of performing face recognition tasks (this work revolutionized object
classification and detection) [75]. The complexity of CNN models (and their
predictive power) has dramatically expanded with the advent of parallel
computing architectures such as graphics processing units [65]. Modern CNN
models for image recognition include SuperVision [44], GoogLeNet [86], VGG
[82], and ResNet [29]. New models are currently being developed to perform
diverse computer vision tasks such as object detection [69], semantic
segmentation [49], action recognition [81], and 3D analysis [37]. Nowadays,
CNNs are routinely used in smartphones (unlock feature based on face
recognition) [64].

While CNNs were originally developed for computer vision, the grid data
representation used by CNNs is flexible and can be used to process datasets
arising in many different applications. For instance, in the field of
chemistry, Hirohara and co-workers proposed a matrix representations of SMILES
strings (which encodes molecular topology) by using a technique known as one-
hot encoding [30]. The authors used this representation to train a CNN that
could predict the toxicity of chemicals; it was shown that the CNN
outperformed traditional models based on fingerprints (an alternative
molecular representation). In biology, Xie and co-workers applied CNNs to
count and detect cells from micrographs [92]. More recently, CNNs have been
expanded to process graph data representations, which has greatly expanded
their application scope. These types of CNNs (known as graph neural networks)
have been widely used in the context of molecular property predictions [17,
26].

Manufacturing covers a broad space of important products and processes that is
virtually impossible to enumerate; in this chapter, we focus our attention on
applications of CNNs to examples of potential relevance to chemical and
biological manufacturing (which cover domains such as pharmaceuticals,
agricultural products, food products, consumer products, petrochemicals, and
materials). We also highlight that these manufacturing sectors are seeing an
emergent use of autonomous platforms that enable flexible and high-throughput
experimentation and/or on-demand production production; as such, the concepts
discussed can be applicable in such context. We provide specific case studies
that we believe provide representative examples on how CNNs can be used to
facilitate decision-making in manufacturing. Specifically, we show how to use
CNNs to i) decode multivariate time series data; ii) decode complex signals
generated from microscopy and flow cytometry to detect contaminants in air and
solution; iii) decode real-time ATR-FTIR spectra to characterize plastic waste
streams; iv) predict surfactant properties directly from their molecular
structures, and v) map image data into signals for feedback control.

##  2 Data Objects and Mathematical Representations

A wide range of datasets encountered in manufacturing can be represented in
the form of a couple of fundamental mathematical objects: tensors and graphs.
Such representations are so general that, in fact, it is difficult to imagine
a dataset that cannot be represented in this way. The key distinction between
a tensor and a graph is that a tensor is a regular object (e.g., a regular
mesh) while a graph is not (e.g., an irregular mesh). Moreover, tensors
implicitly encode positional context, while graphs might or might not encode
positional context. Differences between tensors and graph representations play
a key role in the way that CNN architectures are designed to extract
information from data. Unfortunately, it is often not obvious what data
representation is most suitable for a particular application and sometimes the
representation might not naturally emerge from the data. In fact, one can
think of CNNs as a tool for representation learning, in the sense that it aims
to learn the best way to represent the data to make a prediction.

###  2.1 Tensor Representations

Data objects are often attached to a grid; the most common example of this is
a grayscale image, which can be represented as a 2D grid object. Here, every
spatial grid point is a pixel and the data entry in such pixel is the
intensity of light. A grayscale video is simply a sequence of grayscale images
and can be represented as a 3D grid object (a couple of spatial dimensions
plus time); here, every space-time grid point is a voxel and the data entry is
the intensity of light. A common misconception of grid data is that it can
only be used to represent images and videos but its scope is much broader. For
instance, 3D density fields of chemicals or velocities (as those generated
using molecular and fluid dynamics simulations) can be represented as a 3D
grid; moreover, the thermal field of a surface can be represented as a 2D
grid.

Tensors are mathematical objects used to represent grid data. A tensor is a
generalization of vectors (1D tensors) and matrices (2D tensors) to high
dimensions [27]. A key property of a tensor is that it implicitly encodes
positional context and order (it lives in a Euclidean space); specifically,
every entry of a tensor has an associated set of coordinates/indexes that
uniquely specify the location of an entry in the tensor. Due to its positional
context, the nature of the tensor can altered by rotations; for instance,
rotating an image (or transposing its associated matrix) distorts its
properties.

Figure 1: Representation of a grayscale image (left) as a 2D grid object
(middle). The grid is a matrix in which each entry represents a pixel and the
numerical value in the entry is the intensity of light. Representation of the
grayscale image as a manifold (right); this reveals geometrical patterns of
the image.

Tensors are flexible objects that can also be used to represent multi-
attribute/multi-channel grid data. For example, color images can be
represented as a superposition of three grids (red, green, and blue channels).
Here, each channel is a 2D tensor (a matrix) and the stacking of these three
channels is a 3D tensor. Channels can also be used to represent multi-variate
data in each entry of a grid. For instance, an audio or sensor signal is a
time series that can be represented as a one-channel vector, while a
multivariate time series (e.g., as such obtained from a collection of sensors
in a manufacturing facility) can be represented as multi-channel vector.

It is important to highlight that there is an inherent duality between images
(reality as perceived by human vision or an optical device) and tensors (its
mathematical representation). Specifically, images are optical fields that our
visual sensing system capture and process to navigate the world and make
decisions, while tensors are artificial mathematical representations used for
computer processing. Making this distinction explicit is important, because
humans typically excel at extracting information from visual data (without
having any knowledge of mathematics) compared to numerical data (e.g., number
sequences). As such, this begs the questions: Why do automation systems
present data to human operators as numbers? What are the best visual
representations that humans can use to interpret and analyze data more easily?
These questions are at the core of human-computer interaction and highlight
the relevance of data visualization and processing techniques.

It is also important to highlight that the human vision system and the brain
have inherent limitations in sensing and interpreting optical fields. For
instance, the human vision and auditory system cannot capture all frequencies
present in an optical field or an audio signal; as such, we need
instrumentation (e.g., microscopes and nocturnal vision systems) that
reveal/highlight information that cannot be captured with our limited senes.
Moreover, the human brain often gets “confused” by distortions of optical
fields and audio signals (e.g., rotations and deformations) and by noise
(e.g., fog and white noise). These limitations can be overcome with the use of
artificial intelligence tools such as CNNs. Here, sensing signals such as
images and audio signals are represented mathematically as grid data to
extract information. Unfortunately, grid data representations are limited in
that they inherently represent regular objects and are susceptible to rotation
and deformations.

###  2.2 Graph Representations

Graphs provide another flexible and powerful mathematical data representation.
A graph is a topological object that consists of a set of nodes and a set of
edges; each node is a point in a graph object and each edge connects a pair of
nodes. The connectivity (topology) of a graph is represented as an adjacency
matrix.

Figure 2: Representation of a grayscale image (left) as graph (middle). Each
node in the graph represents a pixel and the weight in the node encodes the
intensity of light. The graph is a topologically invariant object that is not
affected by deformations (right).

A graph defines an irregular grid and one can attach multi-channel data
(features) to nodes and edges in such grid. For instance, when representing a
molecule, one can attach multiple features to each atom (e.g., identity and
charge). Here, one can think of each feature as a the channel of the graph;
graphs with multiple features in nodes and edges are also known as multi-
attribute graphs. An example on how multi-attribute graphs are used to
represent molecules is presented in Figure 3.

Figure 3: Representation of a molecule as a multi-channel/multi-attribute
graph. The molecule topology is encoded in the adjacency matrix, while
features of the atoms are encoded as vectors that are stacked in a feature
matrix. Reproduced from [67] with permission from the American Chemical
Society.

A fundamental property of a graph is that it does not directly encode
positional context (it does not live in a Euclidean space). As such, the graph
can be stretched and rotated or the nodes can be shuffled around in space
without affecting the underlying topology of the graph. In other words, the
topology of the graph is fully dictated by the node-edge connectivity. This
gives graph representations many useful properties (such as rotational
invariance), which can be highly beneficial when representing certain systems.
For instance, in molecular simulations, molecules might move around randomly
in space, but their connectivity (e.g., hydrogen bonds) might not be affected
[36]. Another key property of graphs is that, because these are irregular grid
objects, one can develop CNNs that learn features of graphs of different sizes
(e.g., polymer and peptide sequences).

It is worth highlighting that, while graphs are inherently 2D objects with no
positional context, it is possible to encode 3D (or higher dimensional)
position to each node in a graph as features [93]. For instance, in a graph
that represents the time evolution of a supply chain network, node features
can contain the specific time point and the geographical coordinates.
Moreover, it is possible to generalize graph representations to hypergraphs,
which allow representing objects in which an edge connects multiple nodes.
These more advanced geometric representations give rise to an emerging field
of machine learning known as geometric deep learning [7] and to topological
data analysis [85, 83]. These approaches enable the analysis of complex 3D
(and higher-dimensional) objects.

###  2.3 Color Representations

The human vision system uses color as a key source of information to make
decisions. For computer processing, color is typically represented using a
red-green-blue (RGB) color space (see Figure 4). The RGB color space is an
additive space in which the red, green, and blue channels are mixed together
to produce a wide range of colors [31].

Before the RGB space became the default in computer electronic systems, it
already had a solid theory based on human perception of color. The choice of
these three primary colors is related to the physiology of the human eye;
specifically, the three photoreceptor cells (cone cells) in the human eye have
peak sensitivity at three wavelengths associated red, green, and blue. The RGB
color space was designed to maximize the difference in response of cone cells
to different wavelengths of light [32]. However, there are many other color
spaces that can be useful in different situations. For example, the CIEL*A*B
(or LAB) color space is a powerful representation that can be used for
quantitative color comparisons.

Figure 4: Representation of a color image (left) as a superposition of red,
green, and blue channels (middle). Each channel is a 2D grid object (a matrix)
and stacking the channels generates a 3D tensor.

The LAB space was defined by the International Commission on Illumination
(CIE) to be a perceptually uniform color space, meaning that a group of colors
separated by the same distance in LAB color space will look equally different.
The three coordinates of the LAB color space represent the brightness of the
color ($L*=0$ for black, $L*=100$ for white), its position between red and
green ($a$, negative values indicate green, positive values indicate red), and
its position between yellow and blue ($b$, negative values indicate blue,
positive values indicate yellow). One can convert the RGB color space to a LAB
color space using a nontrivial, nonlinear transformation. The LAB space can be
used to highlight image features that might not be directly obvious to the
human eye; this is a principle that can be exploited by CNNs to extract hidden
information from images.

Unlike RGB and LAB imaging, which captures only three wavelength bands in the
visible spectrum, spectral imaging can use multiple bands across the
electromagnetic spectrum [11]. Spectral imaging can use infrared, visible
spectrum, ultraviolet, X-rays, or some combination of the above. Multispectral
and hyperspectral cameras can capture hundreds of bands for each pixel in an
image, which can be interpreted as a complete spectrum. Due to the wealth of
information encoded in spectral images, this technique has been widely used in
agriculture [50], healthcare [19]. and non-destructive analysis of materials
[53].

##  3 CNN Architectures

CNNs denote a broad class of ML models that utilize specialized convolutional
blocks to extract feature information from data objects. CNNs were originally
designed to extract data from grid objects but they recently have been adapted
to extract information from graphs. The key mathematical operation behind CNNs
is convolution; loosely speaking, this operation transforms the data in a
domain of the data object by applying a weighted sum of the entries in such
domain. The weights of such sum are defined by a convolution filter or
operator. The goal of the CNN is to identify optimal filters or operators that
maximize information extracted from a collection of input data objects; here,
“maximum information” means that such information can predict outcomes
(labels) associated with the input data objects. For instance, one can train a
CNN to predict the toxicity of a molecule directly from its molecular
structure; this is done by training a CNN with a collection of molecules and
associated toxicity levels.

Figure 5: High-level view of a CNN architecture. The input data object (e.g.,
grid or graph) is propagated through a sequence of convolution, activation,
and pooling operations to make an output prediction. The trainable parameters
of the operations are learned by minimizing a loss function. The
derivative/gradient of the loss function is computed using backpropagation.
Reproduced from [41] with permission from Wiley publishing.

A CNN architecture is simply a forward propagation of the input data object
through a sequence of convolution, activation, and pooling operations that
contain parameters that are learned by minimizing a loss function. A backward
propagation (backpropagation) scheme is used to compute the gradient of the
loss function with respect to the parameters.

In this section, we present a basic discussion on the mathematics of
convolution and computational procedures behind the training of CNNs. With
this, we aim to highlight the inner workings of CNNs and associated
bottlenecks and challenges.

###  3.1 Convolution Operations

We will first consider CNN networks $f_{\text{cnn}}$ that map an input tensor
$V$ to output predictions $\hat{y}$. A particular convolutional layer takes an
input object $V$ and _convolves_ it by applying a convolutional operator $U$
which employs a set of convolutional filters or operators to produce a feature
map $\Psi$. The convolution operation is denoted as $\Psi=U*V$ and is simply a
weighted summation between the entries of the convolution operator and the
input object. Here, we also notice that the operator is typically much smaller
than the input object and thus the operator is applied over a moving window
(neighborhood) that covers the entire input object. This operation is not
well-defined by the boundaries of $V$, as some indexing will violate the input
domain, but this is typically resolved by adding padding of zero-valued
entries around the image (a technique called zero-padding). We can also
express this operation in the compact form $\Psi=f_{c}(V;U)$. As shown in
Figure 6, different operators highlight different features of the input
object. Moreover, we note that one can propose a wide range of operators to
extract information from the input data. In a CNN architecture, these
operators are learned in a way that they extract information that best
explains an output of interest.

Figure 6: Convolution of a 3-channel grid (a color image). Each channel is
processed using a different convolution operator. Each operator
highlights/extracts different features (e.g., geometrical features) from the
image. Reproduced from [41] with permission from Wiley publishing.

Following Figure 7, a common and intuitive interpretation of applying
convolution operators is in terms of pattern recognition. The filters that
comprise $U$ each embed a particular pattern that, when convolved with a
particular neighborhood of $V$ (as depicted in Figure 6), gives a score of how
well the pattern is matched (where larger values denote greater matching).
Hence, the feature map $\Psi$ can be interpreted as a scoresheet of how
certain patterns (as encoded by the filters in $U$) are manifested in $V$.

Figure 7: Convolution as a pattern matching technique. The convolution
operator highlights regions in the grid in which the patterns of the image and
of the operator matched. Reproduced from [41] with permission from Wiley
publishing.

In a graph CNN architecture [95], the input graph is propagated based on its
topology; here, convolution operations are performed analogously to grid data
objects, by using weighted summations of the features of a node and of its
neighboring nodes. A message passing framework [26] is typically used as a
generalized approach to build GNN architectures that incorporate both node and
edge features.

###  3.2 Activation Functions

The feature map output of a convolutional layer is typically mapped element-
wise through an activation function $\alpha$ to yield the activated object
$A$. This operation can be written as $A=h_{c}(\Psi)$ and helps the CNN encode
nonlinear behavior. Common choices for the activation function include:

$\begin{gathered}\alpha_{\text{sig}}(z)=\frac{1}{1+e^{-z}}\\\
\alpha_{\text{tanh}}(z)=\text{tanh}(z)\\\
\alpha_{\text{ReLU}}(z)=\text{max}(0,z).\end{gathered}$

Here, the Rectified Linear Unit (ReLU) function $\alpha_{\text{ReLU}}(\cdot)$
is highly popular since it generally exhibits greater sensitivity to changes
in the input [63]. Moreover, this function can be easily represented using
piecewise linear functions and mixed-integer programming formulations.

###  3.3 Pooling

Another key component of CNNs are pooling layers; pooling operations are
dimension reduction mappings $f_{p}$ that seek to summarize/reduce the
activation $A$ by collapsing certain sub-regions of smaller dimension
(referred to as pooling). Pooling helps to make the learned representation
more invariant to small length-scale perturbations [62]. Common choices
include max-pooling and average-pooling where the maximum or average of a sub-
region is used to scalarize, respectively.

###  3.4 Convolution Blocks

Convolution blocks in CNNs denote the combination of convolution, activation,
and pooling of a given input. In other words, a convolution block employs the
mapping $f_{\text{cb}}$ such that:

$P=f_{\text{cb}}(V;U)=f_{p}(h_{c}(f_{c}(V;U))).$ (3.1)

Blocks can employ more complex operation nesting (e.g., multiple convolutional
layers), but we consider blocks that follow (3.1) for simplicity in
presentation. Moreover, these can take pooled outputs $P$ as input and thus
facilitate the use of multiple convolutional blocks in succession via
recursively calling $f_{\text{cb}}$. These blocks essentially act as a feature
extractors via convolutional filters they employ. Moreover, this feature
extraction becomes more specialized for blocks that are located deeper in the
CNN. In the context of computer vision, this means that the first blocks
extract simple features (e.g., edges or colors) and the deeper blocks can
extract more complex patterns (e.g., particular shapes). Thus, we can think of
performing multiple convolution operations as a multi-stage distillation
process where the successive feature spaces capture patterns of increased
length-scale and complexity (i.e., distill the image data into an increasingly
purified features) [14].

###  3.5 Feedforward Neural Networks

The output $P$ of the last convolution block is typically flattened
(vectorized); this can be represented as the mapping $f_{f}$ and yields the
feature vector $v$:

$v=f_{f}(P).$ (3.2)

The feature vector is then fed into a feedforward neural network model $f_{d}$
which predicts the desired state space vector $\hat{y}$. Figure 5 illustrates
a typical image CNN model that implements the components described above. It
employs a couple of convolution blocks and can be described in the functional
form:

$\hat{y}=f_{\text{cnn}}(V),$ (3.3)

where the mapping $f_{\text{cnn}}$ is a nested set of convolution, activation,
pooling, and flattening operations. This emphasizes that convolution blocks
are feature information extractors and the dense layers act as the predictors
whose feature space is the flattened output $v$ of the final block.

###  3.6 Data Augmentation

Data augmentation denotes a class of techniques that aim to artificially
expand the size of a training dataset by applying varied
perturbations/transformations to the input data. In the context of computer
vision, augmentation is often used to expand the size of the training image
set in an effort to decrease the likelihood of the CNN encountering novel
images and in an effort to avoid rotation invariance issues associated with
grid data objects. Image augmentation generally denotes perturbing the
training images such that the CNN sensor can be robust to those types of
visual disturbance. Common perturbations include rotation, translation,
cropping, blurring, brightness changing, splattering, and more. There are many
software tools available that implement these transformations, which include
TensorFlow and ImgAug [24, 42]. Figure 8 shows how a training image is
augmented via a variety of perturbations. This methodology helps mitigate the
risk of CNNs encountering novel images, but it is not typically possible to
account for all the disturbance a process might encounter.

(a) Original

(b) Splattered

(c) Fogged

(d) Shifted

Figure 8: Examples of image perturbation methods used in data augmentation
techniques. Reproduced from [66] with permission from Elsevier.

###  3.7 Training and Testing Procedures

Training CNN procedures seek optimal model parameters (i.e., convolution
operators and dense network weights) that minimize the error incurred by the
output CNN predictions relative to the outputs of the training dataset. Here,
we consider a training set $|\mathcal{K}|$ that contains input-output pairs.
The prediction error minimized in the training procedure is called the loss
function $L$. For example, regression models typically use a sum-of-squared-
error (SSE) loss function:

$L(\hat{y})=||\hat{y}-y||_{2}^{2}.$ (3.4)

Thus, by grouping all the CNN model parameters into $\theta$ we can express
model training as a standard optimization problem:

$\displaystyle\min_{\theta}$
$\displaystyle\sum_{k\in\mathcal{K}}L(\hat{y}^{(k)})$ (3.5) s.t.
$\displaystyle\hat{y}^{(k)}=f_{\text{cnn}}(V^{(k)};\theta),$ $\displaystyle
k\in\mathcal{K}.$

This can readily be expressed as an unconstrained optimization problem by
inserting the constraint equations directly into the objective. Stochastic
Gradient Descent (SGD) is typically used to solve this problem due to the
large amount of training data, the high number of model parameters, and the
model complexity. Moreover, forward and backward propagation techniques are
used to evaluate the objective and derivative values required by each
iteration of the SGD algorithm (see Figure 5). For classification models, one
typically uses an entropy loss function.

###  3.8 CNN Architecture Optimization

Neural architecture search (NAS) has been designed to automatically search for
the best CNN architecture for a given dataset. Specifically, it uses a search
method (e.g., reinforcement learning, evolutionary algorithms, or stochastic
gradient descent) to explore the user-defined search space and chooses the
best architecture based on the performance of the generated model (e.g.,
validation accuracy) on a given task. The search space contains all possible
architectures. The architectures discovered by NAS have been shown to
outperform manually designed architectures in various tasks such as image
classification [68], image segmentation [47], natural language processing
[18], and time series prediction [56].

###  3.9 Transfer Learning

The training process of a CNN can be highly computationally intensive because
this requires performing a large number of convolution operations (e.g., at
each SGD iteration and for a large training set). However, it is possible to
use the filters/operators of existing, trained CNNs to extract information for
a new dataset and such information can be used for different tasks (e.g.,
develop a different ML model such as a support vector machine). This approach
is known as transfer learning and is based on the observation that, while the
filters used are not optimal for the new dataset at hand, they can still
extract valuable information. For instance, in the context of human learning,
one can often detect features for objects that we have not seen before.

##  4 Case Studies

In this section we present a set of case studies that highlight potential uses
of CNNs in a manufacturing context. Our goal with this is not to provide an
exhaustive list of applications, but rather to highlight the capabilities of
CNNs as well as to show how data representations can be used in creative ways
to tackle diverse problems.

###  4.1 CNNs for Sensor Design

####  4.1.1 Detecting Contaminants in Solution

This case study is based on the results presented in [39]. Endotoxins are
lipopolysaccharides (LPS) present in the outer membrane of bacteria [4].
Recently, it has been observed that micrometer-sized liquid crystal (LC)
droplets dispersed in an aqueous solution can be used as a sensing method to
detect and measure the concentration of endotoxins of different bacterial
organisms. After exposure to endotoxins, LC droplets undergo transitions that
yield distinct optical signatures that can be quantified using flow cytometry
(Figure 9).

Figure 9: Overview of the interaction between an endotoxin and LC emulsions.
(a) Generation of FSC/SSC scatter fields. The endotoxin-LC emulsions are
pumped into the flow cytometer in the direction of the sheath flow. Laser
light is scattered from the LC droplets and collected at two angles (FSC and
SSC). By combining the FSC and SSC data points for 10,000 LC droplets, we
generate an FSC/SSC field. (b) Scatter field generated by LC droplets exposed
to 100 pg/mL of endotoxin. Marginal probability densities of FSC (top) and SSC
(right) light in log scale are generated with 50 bins. (c) 2D grid of the
scatter fields by binning and counting the number of events in a bin.
Reproduced from [39] with permission from the Royal Society of Chemistry.

Flow cytometry produces complex data objects in the form of scatter point
clouds of forward and side scattering (FSC/SSC); here, each point represents a
scattering event of a given droplet. The key observation is that a point could
be converted into a 2D grid data object via binning; this is done by
discretizing the FSC/SSC domain and by counting the number of points in a bin.
The 2D grid object obtained is a matrix that we can visualize as a grayscale
image; here, each pixel is a bin and the intensity is the number of
events/droplets in the bin (bins with more droplets appear darker). This
visualization is a 2D projection of a 3D histogram (the third dimension
corresponds to the number of events, also known as the frequency). In other
words, the 2D grid object captures the geometry (shape) of the joint
probability density of the FSC/SSC. These geometrical features can be
extracted automatically using a CNN.

Figure 10: Effect of endotoxin concentration on the FSC/SSC scatter field
(represented as 2D grid objects). Scatter fields obtained using LC droplets
exposed to different concentrations of endotoxin. As the endotoxin
concentration increases, the LC droplet population shifts from a bipolar to a
radial configuration. Reproduced from [39] with permission from the Royal
Society of Chemistry.

The effect of endotoxin concentration on the FSC/SSC scatter fields (after
binning) is shown in Figure 10; clear differences in the patterns can be
observed at concentrations that are far apart but differences are subtle at
close concentrations. We trained a 2D CNN that can automatically detect these
changes and predict concentrations from the scatter fields. We used the
following procedure to obtain the 2D grid data objects (samples) that were fed
to the CNN. For each sample, we generated bins for a given scatter field by
partitioning the FSC and SSC dimensions into 50 segments (the grid has
$50\times 50=2,500$ pixels). For each sample, we were also given reference
scatter fields that represented limiting behavior: bipolar control (negative)
and radial control (positive). Each sample is given by a 3-channel object
$\mathcal{V}$ where the first channel is the negative reference matrix
$\mathcal{V}_{(1)}$, the second channel is the target matrix
$\mathcal{V}_{(2)}$, and the third channel is a positive reference matrix
$\mathcal{V}_{(3)}$ (each channel contains a $50\times 50$ matrix). This
procedure is illustrated in Figure 11. This multi-channel data representation
approach magnifies differences in the target matrix from the references (we
will see that neglecting the negative/positive references does not give
accurate predictions).

Figure 11: EndoNet architecture. The input to EndoNet is a 3-channel object
$\mathcal{V}\in\mathbb{R}^{50\times 50\times 3}$; the channels correspond to
the target, the negative reference, and the positive reference (each is a
matrix of dimension $50\times 50$). EndoNet includes a convolutional block
with 64, $3\times 3$ operators and a max-pooling block with $2\times 2$
operators. The feature map $\mathcal{A}$ generated by the convolution and
activation blocks is a tensor $\mathbb{R}^{48\times 48\times 64}$. The max-
pooling block generates a feature map $\mathcal{P}\in\mathbb{R}^{24\times
24\times 64}$ that is flattened into a long vector $v\in\mathbb{R}^{36863}$.
This vector is passed to two dense layers (each having 32 hidden units). The
predicted endotoxin concentration $\hat{y}$ is the output from the dense layer
activated by a linear function. Reproduced from [39] with permission from the
Royal Society of Chemistry.

The 3-channel data object was fed to a CNN, which we call EndoNet. EndoNet has
an architecture of Conv(64)-MaxPool-Flatten-Dense(32)-Dense(32)-Dense(1). The
output block generates a scalar prediction $\hat{y}$ (corresponding to the
endotoxin concentration). In other words, the CNN seeks to predict the
endotoxin concentration from the input flow cytometry fields. The architecture
of EndoNet is shown in Figure 11. The regression results for EndoNet are
presented in Figure 12. EndoNet extracts pattern information within and
between each channel of the input image. Capturing differences between
channels provides context for the CNN and has the effect of highlighting the
domains in the scatter field that contain the most information. To validate
this hypothesis, we conducted predictions for EndoNet using the 1-channel
representation as input (we ignored the positive and negative channels). For
the 1-channel representation, we obtained an RMSE of 0.97; for the 3-channel
representation we obtained and RMSE of 0.78. It is particularly remarkable
that EndoNet can accurately predict concentrations that span eight orders of
magnitude.

Figure 12: Predicted and actual concentrations at different concentrations
using 1-channel and 3-channel representations. Reproduced from [39] with
permission from the Royal Society of Chemistry.

####  4.1.2 Detecting Contaminants in Air

This case study is based on work presented in [84]. LCs also provide a
versatile platform for sensing of air contaminants [80, 61] and for sensing of
heat transfer and shear stress (mechanical sensing) [34]. In the context of
air chemical sensing, an LC sensor can be prepared by supporting a thin LC
film on a chemically functionalized surface. The molecules within the LC film
(the mesogen) bind to the surface and assume a homeotropic orientation that
provides an initial optical signal. Subsequent exposure of the LC film to an
analyte leads to diffusive transport of the analyte through the LC phase and
displacement of the mesogen at the surface, triggering rich space-time optical
responses (see Figure 13).

Figure 13: Working design principles of a liquid crystal chemical sensor. A
functionalized LC film selectively responds to the presence of an air
contaminant and triggers an optical response. Reproduced from [84] with
permission from the American Chemical Society.

A challenge in the development of LC sensors is their sensitivity to
interfering species. For instance, LC sensors designed for detection of
dimethyl methylphosphonate (DMMP), CH3PO(OCH3)2, might exhibit similar optical
responses when exposed to humid nitrogen [94]. These issues are illustrated in
the experimental responses shown in Figure 14. Our goal here is to determine
whether or not one can unravel hidden patterns in the optical responses that
can help discern between chemical species.

Figure 14: Optical responses of liquid crystals under gaseous
$\textrm{N}_{2}$-water (30% relative humidity) and $\textrm{N}_{2}$-DMMP (10
PPM) environments. LCs were deposited into microwells with a diameter of 3mm
to enable high-throughput data collection. Reproduced from [84] with
permission from the American Chemical Society.

We designed a CNN for automatically detect contaminant presence from optical
LC images (micrographs). In summary, the approach us based on transfer
learning; specifically, we used VGG16, which is a pre-trained CNN for
extracting features from the optical responses and these features were fed
into a simple support vector machine (SVM) that predicts if the contaminant is
present. The results indicate that features extracted from the first and
second convolutional layers of VGG16 allow for a perfect classification
accuracy. The results also indicate that complex spatial color patterns are
developed within seconds in the LC response, which leads us to hypothesize
that fluctuations in LC tilt orientation (angle) play a key role in sensor
selectivity. Moreover, the analysis reveals that color is a key source of
information that the CNN is looking for.

The dataset analyzed in this study was obtained from six videos that show the
response of LCs to a gaseous stream of $\textrm{N}_{2}$ containing 10 ppm DMMP
and six videos that show the response of LCs to a gaseous stream of
$\textrm{N}_{2}$ containing 30% relative humidity (both at room temperature).
Each video tracks the dynamic evolution of multiple independent microwells
(the total number of microwells recorded was 391). We split each frame into
several images, each containing a single microwell at a specific time. The
total number of microwell snapshots generated was 75,081.

Figure 15:  Schematic of pre-trained VGG16 architecture used for feature
extraction of LC micrographs. Reproduced from [84] with permission from the
American Chemical Society. Figure 16:  Schematic of ML framework including
feature extraction and classification based on SVM. Reproduced from [84] with
permission from the American Chemical Society.

The trained VGG16 network is freely available through the Keras software and
are what is used during feature extraction [25]. This highlights the fact that
highly sophisticated CNNs are openly available nowadays for conducting feature
extraction. A simplified representation of the VGG16 architecture is shown in
Figure 15. The VGG16 network has been pre-trained to classify highly complex
images from the internet (not related to our particular application, such as
cat and dogs) and the deepest layers have been carefully tuned to
differentiate such images. The early layers of the network are the most
general and are easier to interpret; accordingly, we use the outputs of the
first and second convolutional blocks to inform features for LSVM
classification. A visual representation of this process for the first and
second convolutional blocks is provided in Figure 16.

The framework using VGG16 features and LSVM was able to classify water and
DMMP micrographs with 100% accuracy. This result was achieved when using all
of the 128 features of the second convolutional layer. An accuracy of 98% is
obtained when we use the 64 features of the first convolutional layer. These
results indicate that LC features that develop early in the sensor response
are highly informative and sufficient to discriminate among chemical
environments.

###  4.2 Molecule Design

ML techniques have been recently used for property predictions of molecules
such as water solubility [33, 52, 3], toxicity [57, 1, 38], and lipophilicity
[79, 87]. A fundamental step in the use of ML algorithms is the pre-definition
or pre-calculation of molecular descriptors [58, 73, 43, 59]; such descriptors
are used as input data to develop quantitative structure-property relationship
models [48]. Among the various ML techniques, graph CNNs (GNNs) [17, 26] have
gained special popularity because they can directly incorporate molecular
graph representations, thus retaining key structural information on molecules
while avoiding the need to pre-calculate/pre-define molecular descriptors for
which density functional theory (DFT) or molecular dynamics (MD) simulations
may be required. Overall, GNNs have shown strong predictive power in molecular
property predictions, and have great potential to be applied to other fields
for more accurate model development as well as enabling high-throughput
screening of materials for manufacturing.

We show how to use GNNs for predicting critical micelle concentrations (CMCs)
of surfactants. This study is based on the work presented in [67]. When
dissolved in water, surfactant monomers will undergo a cooperative aggregation
process, called self-assembly, to form spherical micelles or related aggregate
structures [35]. The formation of micelles in a solution can induce
significant changes in key solution properties including the electrical
conductivity, surface tension, light scattering, and reactivity [74, 35].
Consequently, predicting conditions under which surfactants self-assemble is
important for surfactant selection and design [13]. A critical parameter that
characterizes surfactant self-assembly behavior is the CMC, which is the
minimum surfactant concentration at which self-assembly occurs.

Conventionally, the CMCs are obtained experimentally by tensiometry, but it is
laborious and expensive [23, 78, 20]. Here, we show that GNNs can predict CMC
values directly from the molecular graph of a surfactant monomer. We gathered
experimental CMC data measured at room temperature in water for 202
surfactants [74, 23, 22, 60] covering all surfactant classes. The proposed GNN
architecture consists of two graph convolutional layers, one average pooling
layer, two fully connected hidden layers, and one final output layer. A graph
convolution layer updates each atom by aggregating the features of itself and
its neighbors and maps the updated features into a hidden layer with 256
hidden features. The GCN model contains a total of 216,833 trainable
parameters.

Figure 17: Overview of surfactant molecular structures and self-assembly
process in micelles. (a–d) Sample structures of four classes of surfactants.
Surfactants are categorized by the properties of their head groups as nonionic
(a), cationic (b), anionic (c), or zwitterionic (d). (e) Surfactant monomers
aggregate into spherical micelles in water with hydrophilic head groups facing
toward the solvent and hydrophobic tail groups sequestered inside the micelle
core. (f) Snapshots of a surfactant micelle from a representative MD
simulation with water shown in blue. Reproduced from [67] with permission from
the American Chemical Society. Figure 18: GNN predictions for all classes of
surfactants. Left: low-dimensional distribution of surfactant fingerprints
using t-SNE. The test samples (red crosses) are widespread, and most of the
designed surfactants (green points) fall outside of the clusters of the
existing data set. Right: parity plot between the predicted and experimental
log CMC values (training data in blue and test data in red). The best-fit
slope of the test data is 0.91 (R2 = 0.92), and the test RMSE is 0.30.
Molecular structures are shown for the selected extreme points. Structure (a)
is an anionic surfactant (minor outlier) with a high log CMC value. Structure
(b) is a cationic surfactant (minor outlier) with a high log CMC value.
Structure (c) is a zwitterionic surfactant (major outlier). Structure (d) is a
nonionic surfactant with a low log CMC value. Reproduced from [67] with
permission from the American Chemical Society.

As illustrated in Figure refcmcparity, the cross-validation (CV) RMSE on all
classes of surfactants has a mean value of 0.39 with no significant outliers.
We tested the model performance on a test data set, which contains samples
from each surfactant class. We verified the distribution of the test samples
using t-distributed stochastic neighbor embedding (t-SNE) [88], a nonlinear
dimension reduction technique to visualize high-dimensional data, on the
molecular fingerprints [73] of the surfactants. Figure 18 illustrates that the
test samples are widespread, indicating the inclusion of unlike surfactant
structures and classes in the test data set, which cover a much more diverse
spectrum of surfactants than the data sets used in previous QSPR models.

Figure 18 also shows a parity plot between the experimental and predicted log
CMC values for the training and testing sets. Cationic surfactants have the
lowest test RMSE (0.07) followed by nonionic (0.18) and anionic (0.32)
surfactants, and the model performs worst for zwitterionic surfactants (0.76).
The parity plot also suggested a slightly lower accuracy for surfactants with
relatively large log CMC values (¿4.5). The overall predictability of the GCN
model outperforms that of a prior QSPR model reported in the literature [23].
The differences in the molecular structures found in our dataset highlights
the wide variety of surfactants that the GCN model can capture.

###  4.3 Decoding of Spectra

We now show how to use CNNs to decode real-time spectra; specifically, we show
how to characterize plastic components using real-time ATR-FTIR spectra. This
case study also aims to illustrate how innovative (nonintuitive) data
representations can be used to obtain more information from spectra. This
study is based on the work reported in [40].

Plastics are used in a wide range of applications such as food packaging,
construction, transportation, health care, and electronics. Notably, only 20%
of all plastics produced were recycled [71]; this recycling rate is notably
low compared to that of other materials (e.g., aluminum has a recycling rate
of nearly 100%). A key obstacle that arises here is the characterization of
plastic components in mixed-platic-waste (MPW) streams. ATR-FTIR (attenuated
total reflection-Fourier transform infrared spectroscopy) is an
instrumentation technique that can be used for analyzing plastic components
found in MPW in real-time; as such, one can envision the development of fast,
online ML techniques that can analyze ATR-FTIR spectra to characterize MPW
streams. Here, we show that CNNs can characterize plastic components of MPW by
decoding ATR-FTIR spectra.

Figure 19: Normalized infrared spectral intensities of various plastic
materials. Each spectrum is a vector of length 4150. The resulting spectra
contain significant noise and systematic errors. Reproduced from [40] with
permission from Elsevier.

Experimental data was obtained by preparing small sheets of plastics of
different shapes and used ATR-FTIR to scan sheets for 10 different types; this
data collection approach mimics how rigid waste plastics are found in online
processing of MPW streams. The spectra collected can be represented as 1D
vectors and analyzed by using 1D CNNs [12]. The 1D CNN extracts features of a
spectrum by convolving it with different filters; however, the 1D
representation might fail to capture correlations across frequencies. Gramian
Angular fields (GAF) have been recently used to encode 1D series data into
matrices that capture correlation structures and that are processed using 2D
CNNs; this data transformation approach has been shown to improve
classification accuracy for time series data [90]. A GAF represents vectors in
a polar coordinate system and converts these angles into symmetric matrices
using various operations. There are a couple of GAF types: Gramian Angular
Summation fields (GASF) and Gramian Angular Difference fields (GADF). The
conversion of spectra to GASF and GADF matrices is illustrated in Figure 20;
here, the GAF matrices are represented as grayscale images.

Figure 20: Conversion from 1D signal to GASF and GADF matrices. The 1D signal
is first mapped to the polar coordinate system and finally converted to GASF
and GADF matrices. Encoding the 1D signal into GAF matrices captures the
relationship between the signal intensity at different wavenumbers. Reproduced
from [40] with permission from Elsevier. Figure 21: Architectures of (a)
PlasticNet (1D) and (b) PlasticNet (2D). PlasticNet (1D) inputs a vector of
4150 and outputs the predicted plastic type. It contains 4 1D convolutional
layers (each has 64 filters of dim 3), 1D max-pooling layers (each has a
pooling window size of 2), a flatten layer, and 3 fully-connected layers (each
has 64 units and a dropout ratio of 0.2). The activation functions between the
layers are ReLUs. The final output activation function is softmax. PlasticNet
(2D) inputs a GASF and a GADF matrix. The input size varies from $50\times
50\times 2$ to $250\times 250\times 2$. It has four 2D convolutional layers
(each has 64 filters of $3\times 3$), two 2D max-pooling layers (each has a
pooling window size of $2\times 2$). The flatten, fully-connected layers and
activation function setups are the same as the ones of PlasticNet (1D).
Reproduced from [40] with permission from Elsevier.

The comparison of architectures of PlasticNet (1D) and (2D) is shown in Figure
21. Classification results for PlasticNet (1D) and (2D) are presented in
Figure 22. PlasticNet (2D) has a higher accuracy when the input size is larger
than 100×100, compared to PlasticNet (1D) on raw IR spectra (77.7%).
Specifically, PlasticNet (2D) increases the accuracy of the PlasticNet (1D) by
12.4%; this confirms that correlation information in spectra is important for
classification. To validate the effectiveness of the proposed CNN models, we
compared them with four commonly used ML classifiers, including Radial Basis
Function (RBF) based Support Vector Machine (RBF-SVM), Random Forest (RF),
k-Nearest Neighbors (kNN), Gaussian Process Classifier (GPC). The accuracy of
PlasticNet (2D) is slightly higher ( 1%) than that of RBF-SVM when the input
size is larger than $200\times 200$. This indicates that RBF-SVM is comparable
to CNN-based methods; however, SVMs offer limited flexibility to capture
different representations for IR data. The accuracy of all methods saturates
at 87%, which suggests that the dataset itself contains significant errors
that neither the CNN-based nor the SVM methods can explain. In subsequent
work, we have shown that decoding fast MIR (mid-infrared spectroscopy) spectra
using CNNs can achieve a nearly perfect plastic classification accuracy [96].

Figure 22: Comparison of the accuracy of CNN-based methods and other ML
algorithms. PlasticNet (2D) with an input size of $200\times 200\times 2$ has
the highest accuracy of 87.29%. SVM with RBF kernels has a comparable accuracy
of 86.14%. The accuracy of PlasticNet (2D) is higher than that of PlasticNet
(1D), indicating that the conversion from the original 1D signal to 2D GAF
matrices captures more information. The accuracy of PlasticNet (2D) increases
as the input matrix increases, indicating that a larger input matrix contains
more information. Reproduced from [40] with permission from Elsevier. Figure
23: Confusion matrix for PlasticNet (2D) with an input size 200×200×2. The
overall accuracy is 87.3%. Each column represents a true plastic species, and
each row represents a model predicted plastic species. The entries along the
diagonal are where the plastic species are correctly classified. Many diagonal
entries are close to one, indicating that the PlasticNet (2D) has excellent
classification accuracy. However, some plastic types cannot be classified with
high accuracy (e.g., PC and AC). Reproduced from [40] with permission from
Elsevier.

###  4.4 CNNs for Multivariate Process Monitoring

Multivariate process monitoring is a common task performed in manufacturing to
identify abnormal/faulty behavior. Here, the idea is to collect multivariate
time series (for different process variables) under different modes of
operation (each mode is induced by a specific fault). The goal is to identify
features (signatures) in the time series to determine if the process is a
given mode. The case study presented here uses benchmark data for the
Tennessee-Eastman (TE) process [16]. This study was based on the work
presented in [91, 41].

Figure 24: Representation of a multivariate time series as a 2D image. (a) 52
process variables collected 60 times over a 3-hour period. The variables are
normalized with a zero mean and a unit variance. (b) An ${52\times 60}$ matrix
(visualized as an image). The red line in (a) and red row in (b) indicate the
same data. Each row of the image represents one of the time series in (a). The
fault number is 7 for (a) and (b), 9 for (c), and 15 for (d). We can see that
(c) and (d) are visually similar but belong to different fault groups.
Reproduced from [41] with permission from Wiley publishing.

The TE process units include a reactor, condenser, compressor, separator and
stripper. The TE process produces two products (G and H) and a byproduct (F)
from four reactants (A, C, D and E). Component B is an inert compound. In
total, the TE process contains a total of 52 measured variables; 41 of them
are process variables and 11 are manipulated variables. This process exhibits
20 different types of faults related to changes in feed temperatures,
compositions, reaction kinetics, and so on.

Figure 25: Confusion matrix from the CNN prediction. Each column represents a
true fault type, and each row represents a CNN predicted fault type. The
entries along the diagonal are where the fault types are correctly classified.
Most of the diagonal entries are close to 1, indicating that the CNN has good
classification accuracy. Reproduced from [41] with permission from Wiley
publishing.

The TE process data is obtained from Harvard Dataverse [70]. The 52 process
variables are sampled every 3 minutes; the transformation of multivariate
signal data to matrices is shown in Figure 24. We construct an input data
sample by using 52 signal vectors (each vector contains 60 time points) that
are combined into ${52\times 60}$ matrix ($V$). We have a total of 6947 input
samples and the training:validation:testing data ratio used is 11:4:5. Figure
25 is the confusion matrix obtained for the CNN; the overall classification
accuracy was 0.7561. With the exception of faults 3, 4, 5, 9, and 15, most
faults can be identified accurately. Fault 3, 9 and 15 are especially
difficult to detect because the mean, variance, and higher-order variances do
not vary significantly.

###  4.5 CNNs for Image-Based Feedback Control

In this study, we consider techniques for incorporating CNN sensors (CNNs that
map image signals to controllable state signals) in feedback control systems.
We place a particular focus on the need for real-time novelty/anomaly
detection approaches that provide robustness in effectively mitigating the
consequences of visual disturbances. The concepts discussed here are a summary
of the work presented in [66]. This work in turn draws upon existing emergent
applications of image data in control, as those presented in [89, 54, 72, 51].

Figure 26: Feedback control system that incorporates a CNN sensor to convert
image data into a controllable measurement signal that can be used for
feedback control. Reproduced from [66] with permission from Elsevier. Figure
27: A summary of the SAFE-OCC novelty detection framework that operates on a
feature map to produce a novelty signal. Reproduced from [66] with permission
from Elsevier.

We consider leveraging a CNN sensor to autonomously map image data to a
controllable state variable such that we achieve closed-loop control. With
this, we remove the human operator from the control loop in the sense that
they will no longer need to actively interpret and act upon visual process
data. Such a control system is depicted in Figure 26. Here, the camera and the
CNN work together to form a computer vision sensor that is able to measure
states $y_{\text{vis}}$ that otherwise would not be available using
traditional process measurement devices. Hence, following this new paradigm we
obtain a fully automatic control system that can exhibit improved setpoint
tracking performance which promotes increased consistency and performance.

The paradigm shift from an operator-centric control system to the CNN-aided
system of Figure 26 introduces a significant vulnerability: poor prediction
accuracy of $y_{\text{vis}}$ when the image $V$ is novel relative to the
training data used to prepare the CNN (i.e., the CNN sensor makes a highly
inaccurate prediction because it is extrapolating). Injecting erroneous
measurement data into a feedback control architecture can have severe
consequences. Thus, we require an appropriate novelty detection approach
(depicted in Figure 26) to automatically recognize in real-time when the
visual data $V$ is novel relative to the CNN sensor being used. Novelty
detection denotes a set of unsupervised learning methods that differentiate
between novel and normal data [76]. A couple of paradigms are reconstruction
models and one-class classification. One-Class Classification (OCC) denotes an
area of methods that learn a single class of normal instances from unlabeled
training data (typically assumed to be comprised of normal instances). These
then identify novel data instances by determining if they lie outside the
learned class. Here, we discuss the Sensor Activated Feature Extraction Once-
Class Classification (SAFE-OCC) novelty detection framework [66]. The SAFE-OCC
framework leverages the native feature space of a CNN sensor to achieve
novelty detection that is complimentary. The SAFE-OCC novelty detector
involves three steps: feature extraction via the feature maps of a CNN sensor,
feature refinement, and novelty detection via OCC (see Figure 27).

(a) Simulation 1

(b) Simulation 2

(c) Simulation 3

(d) Simulation 4

Figure 28: Representative snapshots from simulations used in the cart-pole
case study. Reproduced from [66] with permission from Elsevier.

(a) Simulation 4 Control Response

(b) Simulation 2 Control Response

(c) Simulation 4 Novelty Response

(d) Simulation 2 Novelty Response

Figure 29: The control response trajectories of Simulations 2 (foggy
disturbance) and 4 (blockage disturbance). The vertical dotted line at time-
step 150 indicates when the fog disturbance is introduced in Simulation 2 and
the blockage disturbance is introduced in Simulation 4. Effective control is
maintained in Simulation 2 because these are normal images for the CNN.
Ineffective control, however, is obtained in Simulation 4 because the images
are novel to the CNN. The SAFE-OCC framework detects this novel images to
prevent this type of behavior. Reproduced from [66] with permission from
Elsevier.

We illustrate the application of SAFE-OCC to control the CartPole-v1
environment from OpenAI-Gym[6], which corresponds to the classic cart-pole
control problem introduced in [2]. Here, we seek to balance a pendulum above a
cart which we can move either right or left at a mixed rate. We consider the
angle of the pendulum (measured in degrees relative to vertical alignment) as
the state variable $y$ (ignoring the position of the cart), and we take the
cart movement direction to be the control variable
$z\in\\{0,1\\}\subset\mathbb{Z}$ (where 0 is left and 1 is right). Thus, we
have a single-input single-output (SISO) process to control. With this
simplification, we implement a PID controller with a derivative filter.

We conducted four simulations: a base case that uses unperturbed images and
three others that invoke a particular simulated visual disturbance after 150
time-steps. The three disturbance types are produced via ImgAug using the Fog,
Spatter, and Cutout methods which correspond to fog, splattering, and square
blockages, respectively. Figure 28 shows representative images of these
simulations. Figure 29 shows the responses exhibited in Simulations 4
(blockage) and 2 (fog). In Figures 29(b), we observe that effective control in
tracking the set-point is achieved for both simulations. This behavior can be
attributed to the CNN sensor being trained with clear and fogged images which
means that its predictions $\hat{y}$ incur a low error relative to $y$ as
shown in Figure 29(d). Figure 29(c) shows the response in the fourth
simulation; here, the blockage disturbance is novel relative to the CNN sensor
and thus, significant prediction error is incurred in each case once the
sensor is subjected to the disturbance. SAFE-OCC accurately identifies the
novel images once they are injected into the CNN sensor and can avoid
catastrophic control failure.

##  5 Conclusion

This chapter has reviewed the use of convolutional neural networks (CNNs) for
extracting information from complex data sources that are commonly encountered
in manufacturing. We have used a set of selected case studies to demonstrate
how these machine learning (ML) techniques can be used to conduct a variety of
tasks such as classification, anomaly detection, and prediction of properties.

The discussion outlined in this chapter is heavily biased by research
conducted by our research group and is not meant to provide an exhaustive
review. The field of ML learning is quickly evolving, with many different
applications and techniques being developed. For instance, transformer models
are nowadays being developed for computer vision and provide an alternative to
address limitations of CNNs [9]. Moreover, CNNs are actively being used to
tackle a wide range of challenges arising in catalysis, materials, healthcare,
and biology [28, 8, 10].

An area that is still in it infancy is the use of computer vision techniques
to enable feedback control; specifically, closing the loop between image data
and feedback control is a technical challenge. This is because computer vision
signals are inherently infinite-dimensional objects (e.g., fields) that cannot
be controlled directly by control systems. As such, it is necessary to extract
key descriptors that effectively summarize these objects and it is necessary
to develop techniques for building dynamical models directly from computer
vision signals. Techniques such as autoencoders, recurrent neural networks,
and dynamic mode decomposition provide some alternatives [51, 55]. In this
context, it is also important to re-think how to design control architectures
that can properly act on image data, given that most manufacturing systems
have limited actuation. This is particularly relevant in 3D-printing and
additive manufacturing applications, in which it is necessary to achieve high
level of precision in shaping 3D objects. Moreover, it is important to think
about how to leverage these image data sources in the development of physical
models [46]. The use of hyperspectral imaging in manufacturing is also an
exciting direction [5]; this type of data can reveal properties of systems,
materials, and products that provide rich information about quality and
health. A fundamental challenge in dealing with hyperspectral imaging,
however, is it inherit high dimensionality. Specifically, hyperspectral images
contain many color channels and are challenging to process using CNNs.

Another exciting area of current research is speech recognition [15]; one can
envision a future in which instructions are provided to an automation system
in the form of voice or in which the automation system summarizes the behavior
of the system in narrative form or explains the reasoning behind a decision.
Along these lines, the use of audio signals (e.g., as those obtained from
vibration sensors) are currently being investigated to detect faults [77].

## Acknowledgments

We acknowledge funding from the U.S. National Science Foundation under BIGDATA
grant IIS-1837812 and support from the members of the Texas-Wisconsin-
California Control Consortium (TWCCC).

## References

  * [1] P. Banerjee, A. O. Eckert, A. K. Schrey, and R. Preissner.  Protox-ii: a webserver for the prediction of toxicity of chemicals.  Nucleic acids research, 46(W1):W257–W263, 2018.
  * [2] A. G. Barto, R. S. Sutton, and C. W. Anderson.  Neuronlike adaptive elements that can solve difficult learning control problems.  IEEE transactions on systems, man, and cybernetics, (5):834–846, 1983.
  * [3] S. Boobier, D. R. Hose, A. J. Blacker, and B. N. Nguyen.  Machine learning with physicochemical relationships: solubility prediction in organic solvents and water.  Nature communications, 11(1):1–10, 2020.
  * [4] F. Borek.  Handbook of endotoxin, vol. 1, chemistry of endotoxin.  Journal of Immunological Methods, 82(2), 1985.
  * [5] J. Bøtker, J. X. Wu, and J. Rantanen.  Hyperspectral imaging as a part of pharmaceutical product design.  In Data Handling in Science and Technology, volume 32, pages 567–581. Elsevier, 2020.
  * [6] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba.  Openai gym, 2016.
  * [7] M. M. Bronstein, J. Bruna, Y. LeCun, A. Szlam, and P. Vandergheynst.  Geometric deep learning: going beyond euclidean data.  IEEE Signal Processing Magazine, 34(4):18–42, 2017.
  * [8] A. E. Bruno, P. Charbonneau, J. Newman, E. H. Snell, D. R. So, V. Vanhoucke, C. J. Watkins, S. Williams, and J. Wilson.  Classification of crystallization outcomes using deep convolutional neural networks.  PLOS one, 13(6):e0198883, 2018.
  * [9] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko.  End-to-end object detection with transformers.  In European conference on computer vision, pages 213–229. Springer, 2020.
  * [10] A. Cecen, H. Dai, Y. C. Yabansu, S. R. Kalidindi, and L. Song.  Material structure-property linkages using three-dimensional convolutional neural networks.  Acta Materialia, 146:76–84, 2018.
  * [11] C.-I. Chang.  Hyperspectral imaging: techniques for spectral detection and classification, volume 1.  Springer Science & Business Media, 2003.
  * [12] X. Chen, Q. Chai, N. Lin, X. Li, and W. Wang.  1d convolutional neural network for the discrimination of aristolochic acids and their analogues based on near-infrared spectroscopy.  Analytical Methods, 11(40):5118–5125, 2019.
  * [13] K. C. Cheng, Z. S. Khoo, N. W. Lo, W. J. Tan, and N. G. Chemmangattuvalappil.  Design and performance optimisation of detergent product containing binary mixture of anionic-nonionic surfactants.  Heliyon, 6(5):e03861, 2020.
  * [14] F. Chollet.  Deep learning with Python.  Simon and Schuster, 2017.
  * [15] T. Dimitrov, C. Kreisbeck, J. S. Becker, A. Aspuru-Guzik, and S. K. Saikin.  Autonomous molecular design: then and now.  ACS applied materials & interfaces, 11(28):24825–24836, 2019.
  * [16] J. J. Downs and E. F. Vogel.  A plant-wide industrial process control problem.  Computers & chemical engineering, 17(3):245–255, 1993.
  * [17] D. K. Duvenaud, D. Maclaurin, J. Iparraguirre, R. Bombarell, T. Hirzel, A. Aspuru-Guzik, and R. P. Adams.  Convolutional networks on graphs for learning molecular fingerprints.  Advances in neural information processing systems, 28, 2015.
  * [18] Y. Fan, F. Tian, Y. Xia, T. Qin, X.-Y. Li, and T.-Y. Liu.  Searching better architectures for neural machine translation.  IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2020.
  * [19] B. Fei.  Hyperspectral imaging in medical applications.  In Data Handling in Science and Technology, volume 32, pages 523–565. Elsevier, 2020.
  * [20] A. Fluksman and O. Benny.  A robust method for critical micelle concentration determination using coumarin-6 as a fluorescent probe.  Analytical Methods, 11(30):3810–3818, 2019.
  * [21] K. Fukushima.  Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position.  Biological cybernetics, 36(4):193–202, 1980.
  * [22] C. G. Gahan, S. J. Patel, M. E. Boursier, K. E. Nyffeler, J. Jennings, N. L. Abbott, H. E. Blackwell, R. C. Van Lehn, and D. M. Lynn.  Bacterial quorum sensing signals self-assemble in aqueous media to form micelles and vesicles: An integrated experimental and molecular dynamics study.  The Journal of Physical Chemistry B, 124(18):3616–3628, 2020.
  * [23] T. Gaudin, P. Rotureau, I. Pezron, and G. Fayet.  New qspr models to predict the critical micelle concentration of sugar-based surfactants.  Industrial & Engineering Chemistry Research, 55(45):11716–11726, 2016.
  * [24] A. Géron.  Hands-on machine learning with Scikit-Learn, Keras, and TensorFlow: Concepts, tools, and techniques to build intelligent systems.  O’Reilly Media, 2019.
  * [25] A. Géron.  Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems.  O’Reilly Media, 2019.
  * [26] J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl.  Neural message passing for quantum chemistry.  In International Conference on Machine Learning, pages 1263–1272. PMLR, 2017.
  * [27] I. Goodfellow, Y. Bengio, and A. Courville.  Deep learning.  MIT press, 2016.
  * [28] H. A. Haenssle, C. Fink, R. Schneiderbauer, F. Toberer, T. Buhl, A. Blum, A. Kalloo, A. B. H. Hassen, L. Thomas, A. Enk, et al.  Man against machine: diagnostic performance of a deep learning convolutional neural network for dermoscopic melanoma recognition in comparison to 58 dermatologists.  Annals of oncology, 29(8):1836–1842, 2018.
  * [29] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition, 2015.
  * [30] M. Hirohara, Y. Saito, Y. Koda, K. Sato, and Y. Sakakibara.  Convolutional neural network based on smiles representation of compounds for detecting chemical motif.  BMC bioinformatics, 19(19):526, 2018.
  * [31] R. Hirsch.  Exploring colour photography: a complete guide.  Laurence King Publishing, 2004.
  * [32] R. W. G. Hunt.  The reproduction of colour.  John Wiley & Sons, 2005.
  * [33] J. Huuskonen, M. Salo, and J. Taskinen.  Aqueous solubility prediction of drugs based on molecular topology and neural network modeling.  Journal of chemical information and computer sciences, 38(3):450–456, 1998.
  * [34] P. Ireland and T. Jones.  Liquid crystal measurements of heat transfer and surface shear stress.  Measurement Science and Technology, 11(7):969, 2000.
  * [35] J. N. Israelachvili.  Intermolecular and surface forces.  Academic press, 2011.
  * [36] L. Je, G. W. Huber, R. C. Van Lehn, and V. M. Zavala.  On the integration of molecular dynamics, data science, and experiments for studying solvent effects on catalysis.  Current Opinion in Chemical Engineering, 36:100796, 2022.
  * [37] S. Ji, W. Xu, M. Yang, and K. Yu.  3d convolutional neural networks for human action recognition.  IEEE transactions on pattern analysis and machine intelligence, 35(1):221–231, 2012.
  * [38] J. Jiang, R. Wang, and G.-W. Wei.  Ggl-tox: Geometric graph learning for toxicity prediction.  Journal of chemical information and modeling, 61(4):1691–1700, 2021\.
  * [39] S. Jiang, J. Noh, C. Park, A. D. Smith, N. L. Abbott, and V. M. Zavala.  Using machine learning and liquid crystal droplets to identify and quantify endotoxins from different bacterial species.  Analyst, 146(4):1224–1233, 2021.
  * [40] S. Jiang, Z. Xu, M. Kamran, S. Zinchik, S. Paheding, A. G. McDonald, E. Bar-Ziv, and V. M. Zavala.  Using atr-ftir spectra and convolutional neural networks for characterizing mixed plastic waste.  Computers & Chemical Engineering, 155:107547, 2021.
  * [41] S. Jiang and V. M. Zavala.  Convolutional neural nets in chemical engineering: Foundations, computations, and applications.  AIChE Journal, 67(9):e17282, 2021.
  * [42] A. B. Jung, K. Wada, J. Crall, S. Tanaka, J. Graving, C. Reinders, S. Yadav, J. Banerjee, G. Vecsei, A. Kraft, Z. Rui, J. Borovec, C. Vallentin, S. Zhydenko, K. Pfeiffer, B. Cook, I. Fernández, F.-M. De Rainville, C.-H. Weng, A. Ayala-Acevedo, R. Meudec, M. Laporte, et al.  imgaug.  https://github.com/aleju/imgaug, 2020.  Online; accessed 23-July-2021.
  * [43] M. Karelson, V. S. Lobanov, and A. R. Katritzky.  Quantum-chemical descriptors in qsar/qspr studies.  Chemical reviews, 96(3):1027–1044, 1996.
  * [44] A. Krizhevsky, I. Sutskever, and G. E. Hinton.  Imagenet classification with deep convolutional neural networks.  In Advances in neural information processing systems, pages 1097–1105, 2012.
  * [45] Y. Le Cun, L. D. Jackel, B. Boser, J. S. Denker, H. P. Graf, I. Guyon, D. Henderson, R. E. Howard, and W. Hubbard.  Handwritten digit recognition: Applications of neural network chips and automatic learning.  IEEE Communications Magazine, 27(11):41–46, 1989.
  * [46] J. Li, R. Jin, and Z. Y. Hang.  Integration of physically-based and data-driven approaches for thermal field prediction in additive manufacturing.  Materials & Design, 139:473–485, 2018.
  * [47] C. Liu, L.-C. Chen, F. Schroff, H. Adam, W. Hua, A. L. Yuille, and L. Fei-Fei.  Auto-deeplab: Hierarchical neural architecture search for semantic image segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 82–92, 2019.
  * [48] Y.-C. Lo, S. E. Rensi, W. Torng, and R. B. Altman.  Machine learning in chemoinformatics and drug discovery.  Drug discovery today, 23(8):1538–1546, 2018.
  * [49] J. Long, E. Shelhamer, and T. Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431–3440, 2015.
  * [50] B. Lu, P. D. Dao, J. Liu, Y. He, and J. Shang.  Recent advances of hyperspectral imaging technology and applications in agriculture.  Remote Sensing, 12(16):2659, 2020.
  * [51] Q. Lu and V. M. Zavala.  Image-based model predictive control via dynamic mode decomposition.  Journal of Process Control, 104:146–157, 2021.
  * [52] A. Lusci, G. Pollastri, and P. Baldi.  Deep architectures and deep learning in chemoinformatics: the prediction of aqueous solubility for drug-like molecules.  Journal of chemical information and modeling, 53(7):1563–1575, 2013\.
  * [53] M. Manley.  Near-infrared spectroscopy and hyperspectral imaging: non-destructive analysis of biological materials.  Chemical Society Reviews, 43(24):8200–8214, 2014.
  * [54] A. Martynenko.  Computer vision for real-time control in drying.  Food Engineering Reviews, 9(2):91–111, 2017.
  * [55] D. Masti and A. Bemporad.  Learning nonlinear state–space models using autoencoders.  Automatica, 129:109666, 2021.
  * [56] R. Maulik, R. Egele, B. Lusch, and P. Balaprakash.  Recurrent neural network architecture search for geophysical emulation.  arXiv preprint arXiv:2004.10928, 2020.
  * [57] A. Mayr, G. Klambauer, T. Unterthiner, and S. Hochreiter.  Deeptox: toxicity prediction using deep learning.  Frontiers in Environmental Science, 3:80, 2016.
  * [58] H. L. Morgan.  The generation of a unique machine description for chemical structures-a technique developed at chemical abstracts service.  Journal of chemical documentation, 5(2):107–113, 1965.
  * [59] H. Moriwaki, Y.-S. Tian, N. Kawashita, and T. Takagi.  Mordred: a molecular descriptor calculator.  Journal of cheminformatics, 10(1):1–14, 2018.
  * [60] P. Mukerjee and K. J. Mysels.  Critical micelle concentrations of aqueous surfactant systems.  Technical report, National Standard reference data system, 1971.
  * [61] D. Mulder, A. Schenning, and C. Bastiaansen.  Chiral-nematic liquid crystals as one dimensional photonic materials in optical sensors.  Journal of Materials Chemistry C, 2(33):6695–6705, 2014.
  * [62] J. Nagi, F. Ducatelle, G. A. Di Caro, D. Cireşan, U. Meier, A. Giusti, F. Nagi, J. Schmidhuber, and L. M. Gambardella.  Max-pooling convolutional neural networks for vision-based hand gesture recognition.  In 2011 IEEE International Conference on Signal and Image Processing Applications (ICSIPA), pages 342–347. IEEE, 2011.
  * [63] V. Nair and G. E. Hinton.  Rectified linear units improve restricted boltzmann machines.  In Proceedings of the 27th international conference on machine learning (ICML-10), pages 807–814, 2010.
  * [64] A. O.-d. D. N. Network.  for face detection, 2017.
  * [65] J. Nickolls, I. Buck, M. Garland, and K. Skadron.  Scalable parallel programming with cuda.  Queue, 6(2):40–53, 2008.
  * [66] J. L. Pulsipher, L. D. Coutinho, T. A. Soderstrom, and V. M. Zavala.  Safe-occ: A novelty detection framework for convolutional neural network sensors and its application in process control.  Journal of Process Control, 117:78–97, 2022.
  * [67] S. Qin, T. Jin, R. C. Van Lehn, and V. M. Zavala.  Predicting critical micelle concentrations for surfactants using graph convolutional neural networks.  The Journal of Physical Chemistry B, 125(37):10610–10620, 2021\.
  * [68] E. Real, S. Moore, A. Selle, S. Saxena, Y. L. Suematsu, J. Tan, Q. Le, and A. Kurakin.  Large-scale evolution of image classifiers.  arXiv preprint arXiv:1703.01041, 2017.
  * [69] S. Ren, K. He, R. Girshick, and J. Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  In Advances in neural information processing systems, pages 91–99, 2015.
  * [70] C. A. Rieth, B. D. Amsel, R. Tran, and M. B. Cook.  Additional Tennessee Eastman Process Simulation Data for Anomaly Detection Evaluation, 2017.
  * [71] H. Ritchie and M. Roser.  Plastic pollution.  Our World in Data, 2018.  https://ourworldindata.org/plastic-pollution.
  * [72] B. A. Rizkin, K. Popovich, and R. L. Hartman.  Artificial neural network control of thermoelectrically-cooled microfluidics using computer vision based on ir thermography.  Computers & chemical engineering, 121:584–593, 2019.
  * [73] D. Rogers and M. Hahn.  Extended-connectivity fingerprints.  Journal of chemical information and modeling, 50(5):742–754, 2010\.
  * [74] M. J. Rosen and J. T. Kunjappu.  Surfactants and interfacial phenomena.  John Wiley & Sons, 2012.
  * [75] H. A. Rowley, S. Baluja, and T. Kanade.  Neural network-based face detection.  IEEE Transactions on pattern analysis and machine intelligence, 20(1):23–38, 1998.
  * [76] L. Ruff, J. R. Kauffmann, R. A. Vandermeulen, G. Montavon, W. Samek, M. Kloft, T. G. Dietterich, and K.-R. Müller.  A unifying review of deep and shallow anomaly detection.  Proceedings of the IEEE, 2021.
  * [77] R. M. Scheffel, A. A. Fröhlich, and M. Silvestri.  Automated fault detection for additive manufacturing using vibration sensors.  International Journal of Computer Integrated Manufacturing, 34(5):500–514, 2021.
  * [78] N. Scholz, T. Behnke, and U. Resch-Genger.  Determination of the critical micelle concentration of neutral and ionic surfactants with fluorometry, conductometry, and surface tension—a method comparison.  Journal of Fluorescence, 28(1):465–476, 2018.
  * [79] T. Schroeter, A. Schwaighofer, S. Mika, A. Ter Laak, D. Suelzle, U. Ganzer, N. Heinrich, and K.-R. Müller.  Machine learning models for lipophilicity and their domain of applicability.  Molecular pharmaceutics, 4(4):524–538, 2007.
  * [80] R. R. Shah and N. L. Abbott.  Principles for measurement of chemical exposure based on recognition-driven anchoring transitions in liquid crystals.  Science, 293(5533):1296–1299, 2001.
  * [81] K. Simonyan and A. Zisserman.  Two-stream convolutional networks for action recognition in videos.  In Advances in neural information processing systems, pages 568–576, 2014.
  * [82] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [83] A. Smith and V. M. Zavala.  The euler characteristic: A general topological descriptor for complex data.  Computers & Chemical Engineering, 154:107463, 2021.
  * [84] A. D. Smith, N. Abbott, and V. M. Zavala.  Convolutional network analysis of optical micrographs for liquid crystal sensors.  The Journal of Physical Chemistry C, 124(28):15152–15161, 2020\.
  * [85] A. D. Smith, P. Dłotko, and V. M. Zavala.  Topological data analysis: concepts, computation, and applications in chemical engineering.  Computers & Chemical Engineering, 146:107202, 2021.
  * [86] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich.  Going deeper with convolutions, 2014.
  * [87] B. Tang, S. T. Kramer, M. Fang, Y. Qiu, Z. Wu, and D. Xu.  A self-attention based message passing neural network for predicting molecular lipophilicity and aqueous solubility.  Journal of cheminformatics, 12(1):1–9, 2020.
  * [88] L. Van der Maaten and G. Hinton.  Visualizing data using t-sne.  Journal of machine learning research, 9(11), 2008.
  * [89] J. Villalba-Diez, D. Schmidt, R. Gevers, J. Ordieres-Meré, M. Buchwitz, and W. Wellbrock.  Deep learning for industrial computer vision quality control in the printing industry 4.0.  Sensors, 19(18):3987, 2019.
  * [90] Z. Wang and T. Oates.  Encoding time series as images for visual inspection and classification using tiled convolutional neural networks.  In Workshops at the twenty-ninth AAAI conference on artificial intelligence, 2015.
  * [91] H. Wu and J. Zhao.  Deep convolutional neural network model based chemical process fault diagnosis.  Computers & chemical engineering, 115:185–197, 2018.
  * [92] W. Xie, J. A. Noble, and A. Zisserman.  Microscopy cell counting and detection with fully convolutional regression networks.  Computer methods in biomechanics and biomedical engineering: Imaging & Visualization, 6(3):283–292, 2018.
  * [93] K. Yang, K. Swanson, W. Jin, C. Coley, P. Eiden, H. Gao, A. Guzman-Perez, T. Hopper, B. Kelley, M. Mathea, et al.  Analyzing learned molecular representations for property prediction.  Journal of chemical information and modeling, 59(8):3370–3388, 2019\.
  * [94] K.-L. Yang, K. Cadwell, and N. L. Abbott.  Use of self-assembled monolayers, metal ions and smectic liquid crystals to detect organophosphonates.  Sensors and Actuators B: Chemical, 104(1):50–56, 2005.
  * [95] J. Zhou, G. Cui, S. Hu, Z. Zhang, C. Yang, Z. Liu, L. Wang, C. Li, and M. Sun.  Graph neural networks: A review of methods and applications.  AI Open, 1:57–81, 2020.
  * [96] S. Zinchik, S. Jiang, S. Friis, F. Long, L. Høgstedt, V. M. Zavala, and E. Bar-Ziv.  Accurate characterization of mixed plastic waste using machine learning and fast infrared spectroscopy.  ACS Sustainable Chemistry & Engineering, 9(42):14143–14151, 2021\.