# Learning Deep Morphological Networks with Neural Architecture Search

Yufei Hu
U2IS, ENSTA Paris, Institut Polytechnique de Paris
<EMAIL_ADDRESS>
Nacim Belkhir
Safrantech, Safran Group
<EMAIL_ADDRESS>
Jesus Angulo
CMM, Mines ParisTech, PSL Research University
<EMAIL_ADDRESS>
Angela Yao
School of Computing National University of Singapore
<EMAIL_ADDRESS>
Gianni Franchi
U2IS, ENSTA Paris, Institut Polytechnique de Paris
<EMAIL_ADDRESS>

###### Abstract

Deep Neural Networks (DNNs) are generated by sequentially performing linear
and non-linear processes. Using a combination of linear and non-linear
procedures is critical for generating a sufficiently deep feature space. The
majority of non-linear operators are derivations of activation functions or
pooling functions. Mathematical morphology is a branch of mathematics that
provides non-linear operators for a variety of image processing problems. We
investigate the utility of integrating these operations in an end-to-end deep
learning framework in this paper. DNNs are designed to acquire a realistic
representation for a particular job. Morphological operators give topological
descriptors that convey salient information about the shapes of objects
depicted in images. We propose a method based on meta-learning to incorporate
morphological operators into DNNs. The learned architecture demonstrates how
our novel morphological operations significantly increase DNN performance on
various tasks, including picture classification and edge detection.

_Keywords_ Mathematical Morphology; deep learning; architecture search.

##  1 Introduction

Over the last decade, deep learning has made several breakthroughs and
demonstrated successful applications in various fields (_e.g_. in computer
vision Krizhevsky et al. (2012); Simonyan and Zisserman (2014a); He et al.
(2016a); Huang et al. (2017), object detection Redmon et al. (2016), or NLP
Dai et al. (2019); Radford et al. (2019)). This success is mainly due to its
automation of the feature engineering process. This success is mainly
attributable to the fact that it automates the feature engineering process.
Rather than manually designed features, features are learned in an end-to-end
process from data. The need for improved architecture has swiftly followed the
advent of deep learning. Experts now place a premium on architecture
engineering _in lieu of_ features engineering.

Architecture engineering is concerned with determining the most appropriate
operations for the network, their hyper-parameters (_e.g_. the number of
neurons for fully connected layers, or the number of filters or kernel size
for convolutional layers), and the connectivity of all the operations.
Generally, practitioners propose novel operations to validate various
architectures and tasks in order to improve performance on specific tasks. As
a result, developing a novel operation remains a time-consuming and costly
process. It necessitates a manual search for the optimal configuration. As a
result, it is prone to failure when practitioners lack computational
resources. An alternative approach focus on automatically finding the network
architecture design using Neural architecture search (NAS methods) Liu et al.
(2019a); Luo et al. (2019); Liu et al. (2018); Pham et al. (2018); Weng et al.
(2019); Wistuba et al. (2019); Elsken et al. (2019) in lieu of a manual
design. Given a set of data and a performance metric for a learning task, a
NAS algorithm attempts to find the optimal architecture concerning a search
strategy. It can be viewed as an optimization problem in the space of an
architecture network defined by a collection of operations and their possible
combinations. Recently, it was demonstrated empirically on several
applications that architectures discovered using NAS outperform those
discovered manually.

Recently, we have observed a steady increase in interest in mathematical
morphology amongst the deep learning community. Indeed, the intrinsic features
of mathematical morphology operators that enable them to extract information
from topological structures make them excellent candidates. Morphological
operators have been shown to capture image edges Rivest et al. (1993),
granulometry Serra (1988); Thibault et al. (2013) and distances to object
borders Franchi and Angulo (2014). Two methodologies are employed in the
literature to evaluate the utility of morphological operators. Some analyses
Cavallaro et al. (2017); Velasco-Forero and Angulo (2013); Franchi and Angulo
(2016) directly extract descriptors from unlearned morphological layers, while
others Franchi et al. (2020); Valle (2020); Mondal et al. (2020) propose
learning the structural element of the morphological operators. Despite their
superior performance in various application situations, these methodologies
are prone to failure if the deep network architecture is misdesigned, which
may dissuade researchers from pursuing this research path.

This paper proposes a novel methodology based on neural architecture search to
assess the usefulness of newly developed architecture layers and, in
particular, morphological layers. The following paper empirically investigates
morphological layers applied to deep networks on CIFAR10/CIFAR100 Krizhevsky
(2009) and an edge detection task using BSD500 Martin et al. (2001), to
determine the optimal design for morphological layers. We compare our results
to the best architecture discovered using conventional convolutional layers.

Our contributions can be summarized as follows:

  * •

First, we propose novel procedures based on sub-pixel convolutions and
mathematical morphology to construct pseudo-morphological operations using
standard convolution layers.

  * •

We integrate these procedures into deep networks using morphological layers
and NAS algorithms. We demonstrate empirically that our architecture tailored
to morphological layers can outperform conventional convolutional layers.

  * •

We outline some current issues in NAS and introduce the problem of choosing
the backbone,_i.e_. the higher-level architecture design on which the search
will be performed. We offer novel network space descriptions suitable for the
edge identification job.

  * •

We are the first to examine architectural search mixed with morphological
procedures for edge detection. Our new specialized architecture achieves
state-of-the-art performance for edge detection.

##  2 Related work

###  2.1 Mathematical Morphology

Mathematical morphology has been extensively used to denoise raw images Serra
(1983); it has also been used to characterize and analyze microscopic images
Boizeau et al. (2013); Franchi et al. (2018); Drouyer et al. (2017), and
remote sensing Franchi and Angulo (2016, 2014); Cavallaro (2016); Cavallaro et
al. (2017); Fauvel et al. (2007); Benediktsson et al. (2003); Velasco-Forero
and Angulo (2013). Additionally, these operators have been used to generate
medical images Dufour et al. (2013); Zhang et al. (2012). However, all of
these morphological paper operators were used as filters to derive feature
descriptors from the classifier’s input data. We decided to integrate them
into a Deep Neural network in this case.

###  2.2 Morphological neural network

Masci et al. Masci et al. (2013) pioneered the use of morphological operators
in neural networks. They began researching pseudo harmonic morphological
operators in conjunction with DNNs. First morphological Perceptrons has been
investigated in Saeedan et al. (2018); Zhang et al. (2019); Mondal et al.
(2019); Valle (2020); Charisopoulos and Maragos (2017). Some investigations
attempted to integrate morphological layers and DNN architecture Mellouli et
al. (2017); Mondal et al. (2020); Franchi et al. (2020); Nogueira et al.
(2019). The issue with these works is that one could argue that morphological
procedures are architecture-dependent. As a result, we propose in our work to
link the search for architecture using morphological layers. Let us disregard
the work of Blusseau et al. Blusseau et al. (01 Jan. 2020), who used auto-
encoder networks to approximate morphological operators.

###  2.3 Neural Architecture search

Experts in the machine learning field typically create deep neural networks by
hand and pick hyper-parameters through a trial-and-error process. As a result,
the process becomes tiring and tedious, as well as prone to errors. A
different perspective sees model design as a decision-making process that can
be improved. We can automatically find the best combination of algorithms to
maximize the performance of a task. With the growing interest in deep
learning, AutoML Zöller and Huber (2019); Zoller and Huber (2020) and Neural
architecture search Wistuba et al. (2019) emerged where the entire DL pipeline
could be automated, with the expectation to reduce the overall development
cost and approach experts performances. There has been several efforts Zoph
and Le (2016); Floreano et al. (2008); Ren et al. (2020); Pham et al. (2018),
tracing back to the 90’s Kitano (1990); Miller et al. (1989), formulating NAS
as an optimization in the space of network architectures and solved using
either reinforcement learning algorithms Pham et al. (2018); Liu et al.
(2019b), gradient-based optimization Liu et al. (2019a) or sequential model-
based optimizationLiu et al. (2019b); Camero et al. (2020).

##  3 Morphological Architecture Search (MAS)

We first outline some preliminaries on the convolution operation and its
relationship to mathematical morphology in Sec. 3.1. We then outline our
proposed pseudo-morphological dilation operation (Sec. 3.2) and the variants
in (Sec. 3.3). We then explain the architecture search algorithm that we use
to integrate the operators into a neural network (Sec. 3.4). Finally, describe
the proposed architecture backbone.

###  3.1 Preliminaries

Consider a discrete RGB image $f$, where $f[i,j,0],f[i,j,1],f[i,j,2]$ denotes
the red, green, and blue values at position $(i,j)$. We further denote as $g$
the feature map resulting from a DNN’s convolution layer of $f$ with the
filter $\omega$ without bias. The feature map $g$ can be expressed as

$\displaystyle
g[n_{1},n_{2},c]=\sum_{k=0}^{2}\sum_{(i,j)\in\mathcal{N}}f[n_{1}+i,n_{2}+j,k]\omega[i,j,k]+b[c]\mbox{
, with }c\in[1,C_{\mbox{out}}]$ (1)

where $\mathcal{N}$ is a square kernel that defines the spatial size of the
convolution kernel, $c$ is the index of the channel, and $C_{\mbox{out}}$ is
the number of channels output of the layer, and $b$ is the bias equal to zeros
if we consider the unbiased convolution. In this case, we consider the
convolution with a bias $b$, then $g$ can be expressed as

By analogy, mathematical morphology Serra (1983) operators are non-linear
image operators based on the spatial structure of the image. First, these
operators were proposed for binary images; now, they have been extended for
grayscale images. Let $f$ be a grayscale image representing a function, with
the intensity at position $x$, denoted as $f(x)$. The two basic operations in
morphology are performed at the grey level, such that we define the erosion
and dilation operations on their discrete version respectively as:
$\varepsilon_{b}(f)[n_{1},n_{2}]=\min_{(i,j)\in\mathcal{N}}f[n_{1}+i,n_{2}+j]-b[i,j]$
and
$\varepsilon_{b}(f)[n_{1},n_{2}]=\max_{(i,j)\in\mathcal{N}}f[n_{1}-i,n_{2}-j]+b[i,j]$
where $b=\left[b_{-n},\ldots,b_{n}\right]$ is a structuring element (SE). In
Figure 1, we observe that the dilation increases the bright areas based on the
shape of the SE leading to a brighter image. Erosion is the morphological dual
to dilation and will decrease the bright areas. We can see a direct link
between the dilation erosion and convolution where the dilation is a
convolution in the max + algebra as explained in Angulo (2017).

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

Figure 1: Dilation and erosion transformation where the structuring element is
disk(3) applied on a sample image from the BSD500 dataset Arbelaez et al.
(2011).

These operations are increasing hence:
$f<g\Rightarrow\varepsilon_{b}(f)(x)<\varepsilon_{b}(f)(x)$
or$f<g\Rightarrow\delta_{b}(f)(x)<\delta_{b}(f)(x)$. In addition, the erosion
is anti-extensive, while the dilation is extensive
hence:$f>=\varepsilon_{b}(f)(x)$ and $\delta_{b}(f)(x)>=f$.

By combining these two basic operations, we can build new ones such as the
opening and closing. The opening of image $f$ by structuring element $b$ is
given by applying an erosion on $f$ with the structuring element $b$ and then
applying a dilation on the previous results with the same SE. The closing
operation is the morphological dual to the opening.

$\displaystyle\gamma_{b}(f)=\delta_{b}\left(\varepsilon_{b}(f)\right)\mbox{ ,
and }\varphi_{b}(f)=\varepsilon_{b}\left(\delta_{b}(f)\right).$ (2)

By combining these two basic operations, we can also build the internal
gradient, the external gradient denoted respectively $G_{i}$ and $G_{e}$ and
defined as:

$\displaystyle G_{e}(f)=\delta_{b}(f)-f\mbox{ , and
}G_{i}(f)=f-\varepsilon_{b}(f)$ (3)

In addition, we can also build the morphological gradient which is the
difference between the dilation and the erosion with the same SE applied on
the same image. Hence it is defined as:
$G_{b}(f)=\delta_{b}(f)-\varepsilon_{b}(f)$ The structuring element impacts
morphological operations by both the geometry of its support and its weights.
Hence, by combining morphological operators, we can build new operators. One
could wonder how to combine these operators to have the best performances for
a given task. For a while, these operators were combined based on expert
knowledge. In this work, we propose to combine them, thanks to an architecture
search algorithm.

###  3.2 Our pseudo-morphological dilation

We propose the bfpseudo morphological dilation operation. This operation is
composed of the following step:

  1. 1.

We apply a traditional convolution operation to transform the feature map form
$(C,H,W)$ into $(C\times r\times r,H,W)$ where $r$ represents the size of the
convolution kernel, $H$ and $W$ represents respectively the height and width
of the image. Hence, we increase the number of feature map channels. Each
feature map represents a neighborhood map that will be used in the step. For
example, in Figure 2(a), we have four feature maps. We denote this step the
Projection convolution step. Let us denote $f$ the input image and $\omega$
the convolution kernel of spatial size $1\times 1$ and $b$ the bias of the
projection convolution. Let us consider that the convolutional layer has
$C_{\mbox{in}}$ input channels and $C_{\mbox{out}}$ output channels. Hence,
the resulting feature map at pixel $n_{1},n_{2}$ is equal to:

$\displaystyle
g[n_{1},n_{2},c]=\sum_{k=0}^{C_{\mbox{in}}}f[n_{1},n_{2},k]\omega[0,0,k]+b[c]\mbox{
, with }k_{2}\in[1,C_{\mbox{out}}]$ (4)

  2. 2.

We then apply the Pixel Shuffle transformation to the feature maps
representing coordinates as illustrated in Figure 2(b). Pixel Shuffle
transformation, also known as sub-pixel convolution, was introduced Shi et al.
(2016) for the super-resolution task. The Pixel Shuffle transformation
reorganizes the low-resolution image channels to obtain a bigger image with
fewer channels. Specifically, it increases the spatial size of the feature map
by reducing the number of channels. Hence it rearranges the input tensor
elements expressed as $(C\times r\times r,H,W)$ to form a scaled $(C,H\times
r,W\times r)$. This operation is interesting since it is stable, compatible
with deep learning back propagation, and does not add any artifacts. We denote
this step the Pixel Shuffle step. More formally, each channel $c$ of the
previous step represents a neighborhood as illustrated in Figure 2. Let us
decompose equation 4 in two terms such that for all $c$ we have
$g[n_{1},n_{2},c]=\tilde{f}[n_{1},n_{2},c]+b[c]$ where $\tilde{f}$ is the
results of the cross correlation of the convolution layer. Note that
$C_{\mbox{out}}=r^{2}C_{\mbox{in}}$. The output of pixel shuffle is :

$\displaystyle h[n_{1},n_{2},c]=g[\lfloor n_{1}/r\rfloor,\lfloor
n_{2}/r\rfloor,c^{\prime}]$ (5)

where $\lfloor n_{1}/r\rfloor$ is the floor fraction: it takes as input a real
number $n_{1}/r$ and outputs the greatest integer value. The channel
$c^{\prime}$ is equal to
$c^{\prime}=c+n_{1}\mbox{.mod}(r)+n_{2}\mbox{.mod}(r)+r\mathbf{1}_{n_{2}\mbox{.mod}(r)>0}$,
where :$n_{1}{\mbox{.mod}(r)}=n_{1}-(\lfloor n_{1}/r\rfloor\times r)$.

  3. 3.

On the output of the Pixel Shuffle step we apply a max-pooling of stride $r$.
We denote this step the Max-pooling step. We apply the max pooling into $h$
with kernel of spatial size of $r\times r$ and a stride $r$, let us denote
$h^{\prime}$ the result. Then $h^{\prime}$ is equal to :

$\displaystyle
h^{\prime}[n_{1},n_{2},c]=\max_{(i,j)\in\mathcal{N}}\tilde{f}[\lfloor(n_{1}+i)/r\rfloor,\lfloor(n_{2}+j)/r\rfloor,c^{\prime}[i,j]]+b[c^{\prime}[i,j]]$
(6)

where
$c^{\prime}[i,j]=c+(n_{1}+i)\mbox{.mod}(r)+(n_{2}+j)\mbox{.mod}(r)+r\mathbf{1}_{(n_{2}+j)\mbox{.mod}(r)>0}$.
So we ended having a dilation on $\tilde{f}$ where the bias $b$ is the
structuring element, with a connectivity depending on the channel.

Classical morphological operators can be hard to train due to the non
linearity, as pointed out in Franchi et al. (2020), where the authors proposed
to clip the gradients of these layers and apply specific learning on these
layers. Hence integrating them into a NAS framework can be challenging, this
is due to the fact that NAS framework will try to learn the best architecture
with the set of operations. So we cannot build a hand-designed architecture
that will stabilize the new layers. Hence the new layers must be stable to any
architecture. So we proposed a stable version that we are going to introduce
in this section.

(a)

(b)

Figure 2: A simple example to explain the projection convolution and pixel
shuffle step. We start with a feature map composed of 4 pixels of value
(1,2,3,4), then we apply the projection convolution have four feature maps of
value (12,8,11,2),(13,10,4,0),(14,1,7,9) and (5,3,6,15). Then by applying the
pixel shuffle, we have just one feature map.

The full layer is represented in the diagram of Figure 3. We simulate the
structuring element’s shape by working with the parameter $r$ of the max
pooling and the pixel shuffle layer. We notice that by adding a Batch
Normalisation Ioffe and Szegedy (2015) before the Projection convolution step,
the results were more stable. Hence, we used it on all our layers. We denote
this layer as a pseudo dilation because it is not an increasing function nor
an extensive function between the input and the output. Still, it checks these
properties between the input and output of the max pooling.

Figure 3: The overview of pseudo morphological dilation, pool and erosion. The
transformation are composed of 4 steps mandatory (a),(b),(c),(d) and one
optional step (e). (a) is a Batch normalisation to transform the input feature
maps.(b) Traditional convolution to transform the feature map from (C, H W)
into (C×r×r, H, W). (c) is a Sub-Pixel convolution that makes the length and
weight of feature map expanded by r times. (d) max-pooling or a min-poling if
we perform a dilation or an erosion respectively that makes the output feature
map dimension consistent with the input feature map. (e) is an optional max-
pooling operation with a stride of 2.

###  3.3 Our morphological layers

Based on the description of our pseudo morphological dilation, which is our
base operators, we will describe four more operators in this sub-section.
Namely, these operators are: the pseudo morphological erosion, the pseudo
morphological pooling, the pseudo morphological upsampling, and the pseudo
morphological gradient.

The pseudo morphological erosion is constructed with the same exam step as the
pseudo morphological dilation except that instead of doing a max-pooling after
the pixel shuffle step, we apply a min-pooling. We proposed this operation to
check the usefulness of such an operation based on the minimum. We illustrated
this layer in as shown in Figure 3.

Similarly to convolution layers with stride, we proposed Pseudo morphological
pooling. This operation consists of applying first a pseudo morphological
dilation operation followed by an extra max-pooling as shown in Figure 3. In
other words, we achieved downsampling by this approach of the input image.

We propose for edge detection task pseudo morphological upsampling which is
similar to a deconvolution layer. This operation is implemented by
transforming the feature map from $(C,H,W)$ to $(r^{2}\times s^{2}\times
C,H,W)$ through the Projection convolution step. Then we apply the Pixel
Shuffle step to form feature maps with the size of $(C,r\times s\times
H,r\times s\times W)$. Subsequently we apply the Max-pooling step with stride
$r$.

We propose a new layer called the pseudo morphological gradient. For this
layer, we achieve the same Projection convolution step, and Pixel Shuffle
step. Then we get pseudo morphological dilation feature maps using max-
pooling, and we get the final gradient feature map by performing vector
subtraction with the input image. Contrary to the morphological gradient which
is positive for all pixels, this one is not necessarily positive. This is also
happening with the morphological Laplacian Serra (1988) that is interesting to
denoise images. Also, the skip connection provided by this gradient layer can
help to avoid vanishing gradients. This layer is illustrated in Figure 4.

Figure 4: The overview of pseudo morphological gradient The process of
(a)(b)(c)(d) is same with pseudo morphological dilation. (e) the vector
subtraction to get the external pseudo gradient.

###  3.4 Neural Architecture Search (NAS)

Neural Architecture Search (NAS) Liu et al. (2019a); Luo et al. (2019); Liu et
al. (2018); Pham et al. (2018); Weng et al. (2019); Wistuba et al. (2019);
Elsken et al. (2019) has recently identified neural network architectures that
exceed human-designed ones on large-scale image classification Liu et al.
(2019a); Luo et al. (2019); Liu et al. (2018); Pham et al. (2018). As
illustrated in Figure 5, NAS need three key elements :

Figure 5: Components of Neural Architecture Search(NAS) model

  1. 1.

Search Space: which defines a set of operations (e.g. convolution, fully-
connected, pooling) and how we want them to be connected to build valid
network architectures. In some sence, the search space defines the space of
admissible solution.

  2. 2.

Search algorithm: which is the algorithm used to optmise the architecture

  3. 3.

Criterion: which defines the measure to estimate or predict the performance of
an architecture.

The criterion used for all the different tasks is the accuracy criterion
applied to the validation; we want to optimize for the given task. On the rest
of this section, we will describe our search space and our search algorithm.

####  3.4.1 Search space- Cell search

In the NAS community, the search space represents the space where we search
the DNN’s architecture. Thanks to a set of operations and a backbone
explaining how operations can be connected to construct valid network
architectures, it is defined thanks to a set of operations and a backbone. The
search space can be classified into two categories. The first one is the
Global Search Space where the algorithm has to learn all the DNN architecture.
The second one is the Cell-Based Search Space where the DNN architecture lies
in a backbone composed of basic components called cells, and the goal is to
learn what is inside the cells. Inspired by DNN architecture such as ResNet He
et al. (2016b), VGG Simonyan and Zisserman (2014b), and Inception Szegedy et
al. (2017), the author of NASNET Zoph et al. (2018), who initially proposed
cell-based search space, noticed that a DNNs are composed of blocks with the
same kind of operations that are repeated multiple times. In Figure 6, we
represented the backbone of DNN model for CIFAR10 Krizhevsky (2009).

Figure 6: The overall backbone of Cell-Based CNN model for CIFAE 10.

The Cell-Based Search Space appears to be a more popular alternative than
Global Search Space because the newly discovered neural architecture based on
Cell-Based Search Space can be easily transferred between datasets. Two sorts
of cells are commonly employed for classification. The first is the standard
cell, which preserves the feature map’s spatial size, and the second is the
reduction cell, which shrinks the feature map’s spatial size.

Cell search space for classification

We tested three cell search spaces for the classification task, two with the
morphological layer and one without. The different search spaces are
illustrated in Table 1.

Cell search space | Cell search space | Cell search space
---|---|---
without morphological layer | with dilation | with erosion
| separable conv $3\times 3$ |
| separable conv $5\times 5$ |
| average pooling |
| maximum pooling |
$\oslash$ | pseudo morphological dilation $3\times 3$ | pseudo morphological erosion $3\times 3$

Table 1: Cell search space for the classification task. separable conv
$3\times 3$ and separable conv $5\times 5$ are the separable convolutions
introduced in Chollet (2017) with a kernel size of $3\times 3$ and $5\times
5$.

Cell search space for edge detection

For the edge detection task, we proposed three cell search spaces composed of
6 operations. To study pseudo morphological operation’s influence, all the
search spaces have the same number of operations. The different search spaces
are illustrated in Table 2.

Cell search space | Cell search space | Cell search space
---|---|---
without morphological layer | with dilation | with erosion
| cweight $3\times 3$ |
| separable conv $3\times 3$ |
| conv $3\times 3$ |
| average pooling |
| maximum pooling |
separable conv $5\times 5$ | pseudo morphological dilation $3\times 3$ | pseudo morphological gradient $3\times 3$

Table 2: Cell search space for the edge detection task. cweight $3\times 3$ is
a the Squeeze-and-Excitation Networks(SENet) introduced in Luo et al. (2019)
with a kernel size of $3\times 3$.

####  3.4.2 Search space of Architecture

As illustrated in Figure 6, the search architecture space is a backbone
designed by repeating multiple modules composed of reduction cells and normal
cells for classification. For the task classification, we did not change the
search architecture space. However, we proposed search architecture space for
segmentations where the DNNs play with multiple resolutions, skip connections,
and deconvolution.

U-Net Architecture search space The U-Net architecture Ronneberger et al.
(2015) inspired our first search architecture space, that we denote U-Net
search space. We started with this search space since U-Net has state-of-the-
art for medical images.

The U-Net architecture is a fully convolutional network that reinjects the
decoder feature map from the encoder information. Hence the spatial
information might be more precise.

Our U-Net search space backbone is similar to the U-Net DNN backbone. However,
it is composed of two types of cell: downsampling segmentation cell and
upsampling segmentation cell, which we denote DownSC and UpSC. The upsampling
segmentation cell is composed of the following operations:

  * •

separable conv $3\times 3$

  * •

separable conv $5\times 5$

  * •

average pooling

  * •

maximum pooling

  * •

pseudo mophological gradient $3\times 3$

  * •

transpose convolution

The overall U-Net search backbone is shown as the Figure 7 following.

Figure 7: The overall backbone of Cell-Based U-netRonneberger et al.
(2015)Weng et al. (2019), where C equals 8, B represents the number of
intermediate nodes, H0 equals 0.25H, H1 equals 0.5H0 and the rest can be done
in the same manner. Similarly, the value of W is derived in the same way. In
addition, the aux_dropout is a operation sequential.

Multi-scale decoder Architecture search space

Inspired by Deeplab V3+ Chen et al. (2018), PSPNet Zhao et al. (2017), and
RCFLiu et al. (2019b), we notice that a good architecture for general images
relies on an encoder pretrained on ImageNetRussakovsky et al. (2015) and a
decoder that takes as input multiple resolutions and associate them to build
the output. Based on that we propose our new network search space that we
denote Multi-scale decoder search network space.

Figure 8: ResNet101_Decoder

Our new network search space illustrated in Figure 8 can be summarized as
follows. We use a ResNet architecture He et al. (2016b) pretrained on
ImageNetRussakovsky et al. (2015) as an encoder network similarly to Chen et
al. (2018); Zhao et al. (2017); Liu et al. (2019b). ResNet is composed of 4
blocs. We connected each block’s output to two layers. The first one is a
preprocessing layer that we denote $S_{0}$. This layer is composed of a
$3\times 3$ convolution and output 42 feature maps. This operation allows us
to control the depth map that will enter the cells. The second preprocessing
applied on the output of $S_{0}$, and denoted $S_{1}$, is composed of a
$3\times 3$ convolution and outputs 42 feature maps. The input of the cells
are the output of $S_{0}$ and $S_{1}$, so that the cell learns if it wants to
use $S_{0}$ or/and $S_{1}$. After each block, We use a $1\times 1$ convolution
on the cells’ output to reduce the channel’s number to one. This convolution
is followed by an upsampling to resize all the feature maps to input size and
we concatenate all upsampled results. Finally, we use a $1\times 1$ to produce
the final edge detection map.

####  3.4.3 Neural Architecture Optimization (NAO)

NAO Luo et al. (2019) is an optimization algorithm that searches the best
architectures based on the following principle. First, NAO is a Cell-Based
Search Space algorithm like most modern NAS. Hence we just need to learn the
adjacency matrix that represents the cells. Secondly, NAO is a two-step
algorithm that first searches the best architecture in step 1. Then, in step 2
with this architecture, NAO optimizes the DNN’s weights to searches for the
best model. Thirdly, NAO does not directly optimize the cell parameters, and
it performs its optimization on a latent space of cell parameters.

NAO process is illustrated in Figure 9. In detail, The NAO algorithm consists
of an encoder, a predictor, and a decoder networkLuo et al. (2019). The
encoder of NAOLuo et al. (2019) takes an architecture sequence randomly
generated describing an architecture as input, and then maps it into a
continuous space C. Specifically, the encoder is denoted as
$\mathrm{E}:\mathrm{X}\rightarrow\mathrm{C}$. Let us write $e_{x}=E(x)$ the
latent representation of the DNN architecture.

The performance predictorLuo et al. (2019)
$\mathrm{P}:\mathrm{C}\rightarrow\mathrm{R}$ maps the latent representation of
an architecture $x$ into its performance $s_{x}$. For an architecture $x$ and
its performance $s_{x}$ as training data, the optimization of $P$ aims at
minimizing the least-square regression $\mbox{ loss
}=\left(s_{x}-P(E(x))\right)^{2}$.

The decoder of NAOLuo et al. (2019), which is similar to the decoder in the
DNN model, the decoder in NAO is responsible for decoding out the string
tokens in $x$, taking $e_{x}$ as input. Mathematically the decoder is denoted
as function $\mathrm{D}:\mathrm{C}\rightarrow\mathrm{X}$ which decodes the
input taking $e_{x}$. The training process consist of optimizing the following
loss : $\sum_{x\in X}\log P_{D}(x\mid E(x))$.

The Encode-Decoder will learn to build a latent space that can represent the
space of architecture. The performance predictor learns to map this space to
its performance for the given task. Finally, they generate new architecture by
trying to find the one that has the best performance.

Figure 9: Overview of NAO algorithm. The encoder maps architecture $x$ into a
continuous space represented by $e_{x}$. Afterwards, the predictor optimizes
$e_{x}$ by maximizing the output of performance using gradient descent. And
the output of predictor is represented as $e_{x^{\prime}}$. The decoder
transforms $e_{x^{\prime}}$ into a new discrete architecture $x^{\prime}$Luo
et al. (2019)

##  4 Experiments

In this section, we will explain the experiments confirming our morphological
layers’ utility. We looked at two different types of experiments:
classification and edge detection.

###  4.1 Classification task

We evaluated our layers on two datasets : CIFAR10Krizhevsky (2009) and
CIFAR100Krizhevsky (2009). CIFAR10 and CIFAR100 are two data set composed of
50 000 training RGB images and 10 000 RGB test images. The Images are of size
$32\times 32$. CIFAR10 has 10 classes, while CIFAR100 has 100 classes. To
train the DNN, we used the cross-entropy loss and reported the classification
error. To train the DNN, we use the same experimental protocol as NAO use for
CIFAR 10. During the architecture search, we use a small network with $B=5$
(number of nodes), $N=3$ (number of normal cells), $F=8$ (number of channels)
and search the architecture for four iterations; each of them is composed of
50 epochs. After the best cell architectures are found, we increase the
architecture with $B=5$, $N=6$, $F=36$ and optimize the weight of the DNNs for
600 epochs. The batch size for the two-step is 128. The results in Table 3 are
the mean of 3 seeds. The mean error of our architecture with the morphological
layer is 2.65%, which is below 2.93% of NAOLuo et al. (2019). We tried to
replace the morphological dilation with erosion and saw that the result
decreased. Hence we noticed on CIFAR 10 that the morphological dilation layer
improves the performances of the DNN. We also noticed from Figure 10 that NAO
used this operation since it is present in 3 edges out of 15 for the Normal
Cell and 4 out of 15 for the Reduction Cell. Morphological operations seem to
bring information not present by traditional convolutional that can help to
improve performances.

We have illustrated in Figure 10 the normal and reduction cell structure that
we learned for the CIFAR10 Krizhevsky (2009) classification task. As one can
the cells are represented by a graph, which can be characterized by an
adjacency matrix that we are drawn in Figure 11.

(a)

(b)

Figure 10: Cell structure for the classification of CIFAR10, where $c_{k-1}$
represents the previous output. $c_{k-2}$ represents the output of the cell
just before $c_{k-1}$. The blue number 0-4 represent the intermediate nodes
and every edge represents different operations. In addition, $c_{k}$
represents the output of the current cell which concatenates the results of
intermediate node 0-4. Figure 11: The adjacency matrix for normal cell and
reduction cell. From left to right: the adjacency matrix of normal cell, the
adjacency matrix of reduction cell. Among them, the number 1,2,3,4,5,6, and 7
represent respectively the following layers : separable conv $3\times 3$,
separable conv $5\times 5$, avg pool, max pool, identity, pseudo morphological
dilation $3\times 3$, and concatenate operation respectively.

We also trained an architecture search on CIFAR100 and checked that we could
confirm our previous result. We can see in Table 4 our results for CIFAR100.
We notice that the morphological layer seems to improve the performances of
the classification task. We also noticed that training from scratch on the
architecture search on CIFAR100 bring worse results than transferring the
architecture learned on CIFAR10.

Method | B | Error(%)
---|---|---
ResNet with stochastic depthHuang et al. (2016) |  | 5.25
Wide ResNetZagoruyko and Komodakis (2017) |  | 4.00
ENAS+CutoutPham et al. (2018) | $5$ | 3.54
Block-QNN-S more filtersZhong et al. (2018) |  | 3.54
DenseNet-BCHuang et al. (2018) |  | 3.46
PNAS+CutoutLiu et al. (2018) | $5$ | 3.41
DARTS+CutoutLiu et al. (2019a) | $5$ | $2.83$
NAONet-WSLuo et al. (2019) | $5$ | $3.53$
NAONet-WS+CutoutLuo et al. (2019) | $5$ | $2.93$
NAONet-WS+Cutout+ |  |
pseudo morphological dilation | $5$ | 2.65
NAONet-WS+Cutout+ |  |
pseudo morphological erosion | $5$ | $3.20$

Table 3: CIFAR10 Performance of NAONet with pseudo morphological operations,
where B represents the number of nodes within the cell. Error represents the
accuracy error.

Method | B | Error(%)
---|---|---
ResNet with stochastic depthHuang et al. (2016) |  | 24.98
Wide ResNetZagoruyko and Komodakis (2017) |  | 19.25
Block-QNN-S more filtersZhong et al. (2018) |  | 18.06
PNAS+Cutout∗Liu et al. (2018) | $5$ | $17.44$
DenseNet-BCHuang et al. (2018) |  | 17.18
ENAS+Cutout∗Pham et al. (2018) | $5$ | $16.44$
NAONet-WS+Cutout∗Luo et al. (2019) | $5$ | 15.67
NAONet-WS+Cutout+ |  |
without pseudo morphological dilation | $5$ | 16.9
NAONet-WS+Cutout+ |  |
pseudo morphological dilation | $5$ | 16.23

Table 4: CIFAR100 Performance of NAONet with pseudo morphological operations,
where B represents the number of nodes within the cell. Error represents the
accuracy error. Please note that the results of the techniques with a star ∗
trained the architecture search on CIFAR10 and learn the model on CIFAR 100.

###  4.2 Edge detection task

We offer a method for extracting image edges to highlight the power of
morphological operators in deep learning frameworks. We recommended two
backbones for this task. Both the U-Net search backbone and the Multi-scale
decoder Architecture search space are detailed in the section 3.4.2.

We trained our DNNs on BSDS500 Arbelaez et al. (2011), which comprises 200
training, 100 validation, and 200 test images. Up to 9 annotators labeled each
image. Like previous works Liu et al. (2019b); Liu and Lew (2016); Yang et al.
(2016); Kokkinos (2015), we used the training set and validation set for
tuning the DNN and test set for evaluation and mixed the augmented training
data of BSDS500 with flippedVOC Context dataset Mottaghi et al. (2014).

On medical images, U-Net backbone has state-of-the-art performance for
semantic segmentation, yet as illustrated in Table 5 this backbone does not
have good results for edge detection. Then we proposed a Multi-scale decoder
Architecture search space that we denote NAO-Multi-scale. This backbone is
inspired by Traditional algorithmsWen et al. (2018) that perform well on this
task. This algorithm learns a decoder using an encoder pre-trained on
ImageNetRussakovsky et al. (2015).

To evaluate our novel algorithm, we use the F1-score. The F1-score is the
harmonic average of the precision and recall and ranges between 0 and 1, with
higher values being better. We evaluate the F1 score for each image of the
test set of BSDS500Arbelaez et al. (2011) at different thresholds for the edge
prediction. We apply different thresholds since our results are edge
probabilities with values between zero and one. The closest to 1 the edge
value is, the more probable this edge is correct. Yet this metric does not
provide a result for all the dataset. Hence similarly to papers Liu et al.
(2019b); Liu and Lew (2016); Yang et al. (2016); Kokkinos (2015) the Optimal
Dataset Scale (ODS) and Optimal Image Scale (OIS) that provide a metric for
the whole dataset.

The Optimal Dataset Scale (ODS), where one chooses the optimal threshold for
the entire dataset before applying the F1 score, the Optimal Image Scale
(OIS), where one chooses the optimal threshold per-image before using the F1
score, are two metrics to evaluate the quality of edge detection algorithm on
the whole dataset. The OIS is always a bit better than the ODS, since it
considers the best scale for each image. The OIS corresponds to the optimistic
situation where we have the optimal threshold for each image of the dataset.
For more information about these classical measures for edge detection, we
refer to Arbelaez et al. (2011)Liu et al. (2019).

We can see in Figure 12 examples where RCF has missed to detect edges, and in
Figure 13 examples where RCF detect edges while there are no edges. Finally,
in Figure 14 we can see some examples where our algorithm fails.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

Figure 12:  Overview of results of different methods on BSD500Arbelaez et al.
(2011).From left to right: original image, ground truth, RCFLiu et al. (2019b)
predict edge map, NAO_multi_scale predicted edge map. We can see in these
examples that RCF has more False Negative than NAO_multi_scale.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

Figure 13:  Overview of results of different methods on BSD500Arbelaez et al.
(2011).From left to right: original image, ground truth, RCFLiu et al. (2019b)
predict edge map, NAO_multi_scale predicted edge map. We can see in these
examples that RCF has more False Positive than NAO_multi_scale.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

Figure 14:  Overview of results of different methods on BSD500Arbelaez et al.
(2011).From left to right: original image, ground truth, RCFLiu et al. (2019b)
predict edge map, NAO_multi_scale predicted edge map. We can see in these
examples that NAO_multi_scale has more False Negatif than RCF.

Method | ODS | OIS | AP | R50
---|---|---|---|---
DeepEdge Bertasius et al. (2015a) | 0.753 | 0.772 | 0.807 |
$N^{4}$-Fields Ganin and Lempitsky (2014) | 0.753 | 0.767 | 0.780 |
HFL Bertasius et al. (2015b) | 0.767 | 0.788 | 0.800 |
HED Xie and Tu (2015a) | 0.782 | 0.804 | 0.833 |
RDS Liu and Lew (2016) | 0.792 | 0.810 |  |
CEDN Yang et al. (2016) | 0.788 | 0.804 |  |
AMH-Net(fusion) Xu et al. (2018) | 0.798 | 0.829 | 0.869 |
CED Wang et al. (2019) | 0.803 | 0.820 | 0.871 |
MIL+G-DSN+VOC+MS+NCuts Kokkinos (2015) | 0.813 | 0.831 |  |
RCF_ResNet101 Liu et al. (2019b) | 0.812 | 0.829 |  |
NAO-U-NET(ours) | 0.788 | 0.808 | 0.814 | 0.899
with dilation search space |  |  |  |
NAO-Multi-scale(ours) | 0.812 | 0.830 | 0.827 | 0.903
without morphological search space | $\pm 0.001$ | $\pm 0.001$ | $\pm 0.014$ | $\pm 0.00471$
NAO-Multi-scale(ours) | 0.809 | 0.829 | 0.825 | 0.900
with dilation search space | $\pm 0.001$ | $\pm 0.001$ | $\pm 0.0149$ | $\pm 0.0048$
NAO-Multi-scale(ours) | 0.814 | 0.831 | 0.850 | 0.908
with gradient search space | $\pm 0.001$ | $\pm 0.001$ | $\pm 0.002$ | $\pm 0.005$

Table 5: Performance of NAO_ResNet and comparison with other competitors on
BSD500 test set, measured using the methodXie and Tu (2015b). For our
experiments, the results are presented as mean $\pm std-dev$ computed from 3
runs.

In Table 5, one can see our results NAO-Multi-scale outperforms state-of-the-
art results when we use the gradient search space. This comes from the fact
that gradient can be used to detect edge detection; hence, using this layer
helps the DNN to estimate the edge detection.

###  4.3 Discussions

We demonstrated in prior experiments that morphological layers boost the DNN’s
performance for a specific architecture search methodology. However, the
improvement is not consistent across all levels. As seen in Table 3 erosion
layers perform worse than DNN without morphological layers. However, dilation
layers increase performance. This improvement is due to the fact that erosion
is associated with a min-pooling process that extracts the less noticeable
region. Similarly, as seen in Table 5, the DNN learned with dilation layers
degrades performance, whereas the gradient-based DNN improves it. This brings
up one point: if the morphological layers are chosen correctly, they can
enhance the representational power of a DNN. Another intriguing element of the
edge detection studies is that the search space of the architecture matters,
as we observe that the U-Net search space produces lower results than the
multi-scale decoder search space that we suggest. As illustrated in Figure 12
and 13 multi-resolution aids in obtaining a better edge since some items may
be more easily spotted at specific resolutions than others. Finally, we note
that the results obtained using the multi-scale decoder search space with the
gradient layer are state-of-the-art. As a result, this demonstrates the use of
these layers.

##  5 Conclusion

This paper introduces a new layer for Deep Neural Networks that is based on
mathematical morphology. We offer a novel equitable technique for determining
the utility of a new layer. This method is applied to our newly created
layers. We conclude that our layer has the potential to be extremely useful
for image categorization and edge detection. This evaluation methodology
appears to be more equitable than the standard approach, which entails
proposing a new layer and handcrafting an architecture to increase
performance. Everything is optimized using an algorithm in this case.

Additionally, we suggest a new backbone architecture for architecture search.
Both the input and output are images, which we refer to as NAO-Multi-scale. We
compared it to the Unet architecture search and established that our search
architecture outperforms the Unet architecture search. Finally, we achieved
state-of-the-art edge detection performance using NAO-Multi-scale and gradient
operations.

In future research, we will examine how to employ such layers to do semantic
segmentation based on these encouraging results. Semantic segmentation is a
hot topic, and it will be interesting to watch how it interacts with these
morphological levels. Additionally, it will be fascinating to deal with more
specialized information, such as remote sensing data containing several
photographs of buildings and roads with unique geometric shapes. As such, it
may be instructive to observe how these new layers act in this scenario.

## Acknowledgement

We would like to thank CNRS Jean-Zay supercomputer.

## References

  * Krizhevsky et al. [2012] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  Imagenet classification with deep convolutional neural networks.  In _International Conference on Neural Information Processing Systems_ , pages 1097–1105, 2012.
  * Simonyan and Zisserman [2014a] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014a.
  * He et al. [2016a] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016a.
  * Huang et al. [2017] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 4700–4708, 2017.
  * Redmon et al. [2016] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 779–788, 2016.
  * Dai et al. [2019] Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov.  Transformer-xl: Attentive language models beyond a fixed-length context.  _arXiv preprint arXiv:1901.02860_ , 2019.
  * Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  _OpenAI Blog_ , 1:8, 2019.
  * Liu et al. [2019a] Hanxiao Liu, Karen Simonyan, and Yiming Yang.  Darts: Differentiable architecture search, 2019a.
  * Luo et al. [2019] Renqian Luo, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu.  Neural architecture optimization, 2019.
  * Liu et al. [2018] Chenxi Liu, Barret Zoph, Maxim Neumann, Jonathon Shlens, Wei Hua, Li-Jia Li, Li Fei-Fei, Alan Yuille, Jonathan Huang, and Kevin Murphy.  Progressive neural architecture search.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 19–34, 2018.
  * Pham et al. [2018] Hieu Pham, Melody Guan, Barret Zoph, Quoc Le, and Jeff Dean.  Efficient neural architecture search via parameters sharing.  In _International Conference on Machine Learning_ , pages 4095–4104. PMLR, 2018.
  * Weng et al. [2019] Y. Weng, T. Zhou, Y. Li, and X. Qiu.  Nas-unet: Neural architecture search for medical image segmentation.  _IEEE Access_ , 7:44247–44257, 2019.  ISSN 2169-3536.  doi:10.1109/ACCESS.2019.2908991.
  * Wistuba et al. [2019] Martin Wistuba, Ambrish Rawat, and Tejaswini Pedapati.  A survey on neural architecture search, 2019.
  * Elsken et al. [2019] Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter.  Neural architecture search: A survey, 2019.
  * Rivest et al. [1993] Jean-Francois Rivest, Pierre Soille, and Serge Beucher.  Morphological gradients.  _Journal of Electronic Imaging_ , 2(4):326–336, 1993.
  * Serra [1988] Jean Serra.  Image analysis and mathematical morphology, v. 2.  1988\.
  * Thibault et al. [2013] Guillaume Thibault, Jesus Angulo, and Fernand Meyer.  Advanced statistical matrices for texture characterization: application to cell classification.  _IEEE Transactions on Biomedical Engineering_ , 61(3):630–637, 2013.
  * Franchi and Angulo [2014] Gianni Franchi and Jesus Angulo.  Comparative study on morphological principal component analysis of hyperspectral images.  In _2014 6th Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing (WHISPERS)_ , pages 1–4. IEEE, 2014.
  * Cavallaro et al. [2017] Gabriele Cavallaro, Nicola Falco, Mauro Dalla Mura, and Jón Atli Benediktsson.  Automatic attribute profiles.  _IEEE Transactions on Image Processing_ , 26(4):1859–1872, 2017.
  * Velasco-Forero and Angulo [2013] Santiago Velasco-Forero and Jesus Angulo.  Classification of hyperspectral images by tensor modeling and additive morphological decomposition.  _Pattern Recognition_ , 46(2):566–577, 2013.
  * Franchi and Angulo [2016] Gianni Franchi and Jesus Angulo.  Morphological principal component analysis for hyperspectral image analysis.  _ISPRS International Journal of Geo-Information_ , 5(6):83, 2016.
  * Franchi et al. [2020] Gianni Franchi, Amin Fehri, and Angela Yao.  Deep morphological networks.  _Pattern Recognition_ , 102:107246, 2020.  ISSN 0031-3203.
  * Valle [2020] Marcos Eduardo Valle.  Reduced dilation-erosion perceptron for binary classification.  _Mathematics_ , 8(4):512, Apr 2020.  ISSN 2227-7390.
  * Mondal et al. [2020] Ranjan Mondal, Moni Shankar Dey, and Bhabatosh Chanda.  Image restoration by learning morphological opening-closing network.  _Mathematical Morphology-Theory and Applications_ , 4(1):87–107, 2020.
  * Krizhevsky [2009] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  Technical report, 2009.
  * Martin et al. [2001] D. Martin, C. Fowlkes, D. Tal, and J. Malik.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics.  In _Proc. 8th Int’l Conf. Computer Vision_ , volume 2, pages 416–423, July 2001.
  * Serra [1983] Jean Serra.  _Image Analysis and Mathematical Morphology_.  Academic Press, Inc., Orlando, FL, USA, 1983.  ISBN 0126372403.
  * Boizeau et al. [2013] Marie-Laure Boizeau, Pierre Fons, Lionel Cousseins, Josette Desjobert, David Sibrac, Corinne Michaux, Anne-Laure Nestor, Bruno Gautret, Keith Neil, Corentin Herbert, et al.  Automated image analysis of in vitro angiogenesis assay.  _Journal of laboratory automation_ , 18(5):411–415, 2013.
  * Franchi et al. [2018] Gianni Franchi, Jesus Angulo, Maxime Moreaud, and Loïc Sorbier.  Enhanced edx images by fusion of multimodal sem images using pansharpening techniques.  _Journal of microscopy_ , 269(1):94–112, 2018\.
  * Drouyer et al. [2017] Sébastien Drouyer, Serge Beucher, Michel Bilodeau, Maxime Moreaud, and Loïc Sorbier.  Sparse stereo disparity map densification using hierarchical image segmentation.  In _International symposium on mathematical morphology and its applications to signal and image processing_ , pages 172–184. Springer, 2017.
  * Cavallaro [2016] Gabriele Cavallaro.  _Spectral-Spatial Classification of Remote Sensing Optical Data with Morphological Attribute Profiles using Parallel and Scalable Methods_.  PhD thesis, University of Iceland, 2016.
  * Fauvel et al. [2007] Mathieu Fauvel, Jocelyn Chanussot, Jon Atli Benediktsson, and Johannes R Sveinsson.  Spectral and spatial classification of hyperspectral data using svms and morphological profiles.  In _2007 IEEE International Geoscience and Remote Sensing Symposium_ , pages 4834–4837. IEEE, 2007.
  * Benediktsson et al. [2003] Jon Atli Benediktsson, Martino Pesaresi, and Kolbeinn Amason.  Classification and feature extraction for remote sensing images from urban areas based on morphological transformations.  _IEEE Transactions on Geoscience and Remote Sensing_ , 41(9):1940–1949, 2003.
  * Dufour et al. [2013] Alice Dufour, Christian Ronse, Joseph Baruthio, Olena Tankyevych, Hugues Talbot, and Nicolas Passat.  Morphology-based cerebrovascular atlas.  In _2013 IEEE 10th International Symposium on Biomedical Imaging_ , pages 1210–1214. IEEE, 2013.
  * Zhang et al. [2012] Xiwei Zhang, Guillaume Thibault, Etienne Decencière, Gwénolé Quellec, Guy Cazuguel, Ali Erginay, Pascale Massin, and Agnès Chabouis.  Spatial normalization of eye fundus images.  In _ISBI 2012: 9th IEEE International Symposium on Biomedical Imaging_. IEEE, 2012.
  * Masci et al. [2013] Jonathan Masci, Jesús Angulo, and Jürgen Schmidhuber.  A learning framework for morphological operators using counter–harmonic mean.  In _International Symposium on Mathematical Morphology and Its Applications to Signal and Image Processing_ , pages 329–340. Springer, 2013.
  * Saeedan et al. [2018] Faraz Saeedan, Nicolas Weber, Michael Goesele, and Stefan Roth.  Detail-preserving pooling in deep networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 9108–9116, 2018.
  * Zhang et al. [2019] Yunxiang Zhang, Samy Blusseau, Santiago Velasco-Forero, Isabelle Bloch, and Jesus Angulo.  Max-plus operators applied to filter selection and model pruning in neural networks.  _arXiv preprint arXiv:1903.08072_ , 2019.
  * Mondal et al. [2019] Ranjan Mondal, Sanchayan Santra, and Bhabatosh Chanda.  Dense morphological network: An universal function approximator.  _arXiv preprint arXiv:1901.00109_ , 2019.
  * Charisopoulos and Maragos [2017] Vasileios Charisopoulos and Petros Maragos.  Morphological perceptrons: geometry and training algorithms.  In _International Symposium on Mathematical Morphology and Its Applications to Signal and Image Processing_ , pages 3–15. Springer, 2017.
  * Mellouli et al. [2017] Dorra Mellouli, Tarek M Hamdani, Mounir Ben Ayed, and Adel M Alimi.  Morph-cnn: A morphological convolutional neural network for image classification.  In _International Conference on Neural Information Processing_ , pages 110–117. Springer, 2017.
  * Nogueira et al. [2019] Keiller Nogueira, Jocelyn Chanussot, Mauro Dalla Mura, William Robson Schwartz, and Jefersson A. dos Santos.  An introduction to deep morphological networks, 2019.
  * Blusseau et al. [01 Jan. 2020] Samy Blusseau, Bastien Ponchon, Santiago Velasco-Forero, Jesús Angulo, and Isabelle Bloch.  Approximating morphological operators with part-based representations learned by asymmetric auto-encoders.  _Mathematical Morphology - Theory and Applications_ , 4(1):64 – 86, 01 Jan. 2020.
  * Zöller and Huber [2019] Marc-André Zöller and Marco F Huber.  Survey on automated machine learning.  _arXiv preprint arXiv:1904.12054_ , 2019.
  * Zoller and Huber [2020] Marcandre Zoller and Marco F Huber.  Benchmark and survey of automated machine learning frameworks.  _arXiv: Learning_ , 2020.
  * Zoph and Le [2016] Barret Zoph and Quoc V Le.  Neural architecture search with reinforcement learning.  _arXiv preprint arXiv:1611.01578_ , 2016.
  * Floreano et al. [2008] Dario Floreano, Peter Dürr, and Claudio Mattiussi.  Neuroevolution: from architectures to learning.  _Evolutionary intelligence_ , 1(1):47–62, 2008\.
  * Ren et al. [2020] Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Xiaojiang Chen, and Xin Wang.  A comprehensive survey of neural architecture search: Challenges and solutions, 2020.
  * Kitano [1990] Hiroaki Kitano.  Designing neural networks using genetic algorithms with graph generation system.  _Complex systems_ , 4(4):461–476, 1990.
  * Miller et al. [1989] Geoffrey F Miller, Peter M Todd, and Shailesh U Hegde.  Designing neural networks using genetic algorithms.  In _ICGA_ , volume 89, pages 379–384, 1989.
  * Liu et al. [2019b] Yun Liu, Ming-Ming Cheng, Xiaowei Hu, Jia-Wang Bian, Le Zhang, Xiang Bai, and Jinhui Tang.  Richer convolutional features for edge detection.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 41(8):1939–1946, Aug 2019b.  ISSN 1939-3539.
  * Camero et al. [2020] Andrés Camero, Hao Wang, Enrique Alba, and Thomas Bäck.  Bayesian neural architecture search using a training-free performance metric.  _arXiv preprint arXiv:2001.10726_ , 2020.
  * Angulo [2017] Jesus Angulo.  Convolution in (max,min)-algebra and its role in mathematical morphology.  In _Advances in imaging and electron physics_ , volume 203, pages 1–66. Elsevier, 2017.
  * Arbelaez et al. [2011] Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik.  Contour detection and hierarchical image segmentation.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 33(5):898–916, May 2011.  ISSN 0162-8828.
  * Shi et al. [2016] Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang.  Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network, 2016.
  * Ioffe and Szegedy [2015] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In _International conference on machine learning_ , pages 448–456. PMLR, 2015.
  * He et al. [2016b] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016b.
  * Simonyan and Zisserman [2014b] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  _arXiv preprint arXiv:1409.1556_ , 2014b.
  * Szegedy et al. [2017] Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander Alemi.  Inception-v4, inception-resnet and the impact of residual connections on learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 31, 2017.
  * Zoph et al. [2018] Barret Zoph, Vijay Vasudevan, Jonathon Shlens, and Quoc V Le.  Learning transferable architectures for scalable image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 8697–8710, 2018.
  * Chollet [2017] François Chollet.  Xception: Deep learning with depthwise separable convolutions.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 1251–1258, 2017.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation, 2015\.
  * Chen et al. [2018] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In _Proceedings of the European conference on computer vision (ECCV)_ , pages 801–818, 2018.
  * Zhao et al. [2017] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 2881–2890, 2017.
  * Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  Imagenet large scale visual recognition challenge, 2015.
  * Huang et al. [2016] Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger.  Deep networks with stochastic depth, 2016.
  * Zagoruyko and Komodakis [2017] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks, 2017.
  * Zhong et al. [2018] Zhao Zhong, Junjie Yan, Wei Wu, Jing Shao, and Cheng-Lin Liu.  Practical block-wise neural network architecture generation, 2018.
  * Huang et al. [2018] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger.  Densely connected convolutional networks, 2018.
  * Liu and Lew [2016] Yu Liu and Michael S Lew.  Learning relaxed deep supervision for better edge detection.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 231–240, 2016.
  * Yang et al. [2016] Jimei Yang, Brian Price, Scott Cohen, Honglak Lee, and Ming-Hsuan Yang.  Object contour detection with a fully convolutional encoder-decoder network.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 193–202, 2016.
  * Kokkinos [2015] Iasonas Kokkinos.  Pushing the boundaries of boundary detection using deep learning.  _arXiv preprint arXiv:1511.07386_ , 2015.
  * Mottaghi et al. [2014] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille.  The role of context for object detection and semantic segmentation in the wild.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 891–898, 2014.
  * Wen et al. [2018] Changbao Wen, Pengli Liu, Wenbo Ma, Zhirong Jian, Changheng Lv, Jitong Hong, and Xiaowen Shi.  Edge detection with feature re-extraction deep convolutional neural network.  _Journal of Visual Communication and Image Representation_ , 57:84 – 90, 2018.  ISSN 1047-3203.
  * Liu et al. [2019] Y. Liu, M. Cheng, X. Hu, J. Bian, L. Zhang, X. Bai, and J. Tang.  Richer convolutional features for edge detection.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 41(8):1939–1946, 2019.  doi:10.1109/TPAMI.2018.2878849.
  * Bertasius et al. [2015a] Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani.  Deepedge: A multi-scale bifurcated deep network for top-down contour detection.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 4380–4389, 2015a.
  * Ganin and Lempitsky [2014] Yaroslav Ganin and Victor Lempitsky.  $n^{4}$-fields: Neural network nearest neighbor fields for image transforms, 2014.
  * Bertasius et al. [2015b] Gedas Bertasius, Jianbo Shi, and Lorenzo Torresani.  High-for-low and low-for-high: Efficient boundary detection from deep object features and its applications to high-level vision, 2015b.
  * Xie and Tu [2015a] Saining Xie and Zhuowen Tu.  Holistically-nested edge detection, 2015a.
  * Xu et al. [2018] Dan Xu, Wanli Ouyang, Xavier Alameda-Pineda, Elisa Ricci, Xiaogang Wang, and Nicu Sebe.  Learning deep structured multi-scale features using attention-gated crfs for contour prediction, 2018.
  * Wang et al. [2019] Yupei Wang, Xin Zhao, Yin Li, and Kaiqi Huang.  Deep crisp boundaries: From boundaries to higher-level tasks.  _IEEE Transactions on Image Processing_ , 28(3):1285–1298, Mar 2019.  ISSN 1941-0042.
  * Xie and Tu [2015b] Saining Xie and Zhuowen Tu.  Holistically-nested edge detection.  In _Proceedings of the IEEE international conference on computer vision_ , pages 1395–1403, 2015b.