# Group-Wise Semantic Mining for Weakly Supervised Semantic Segmentation

Xueyi Li1, Tianfei Zhou2, Jianwu Li1, Yi Zhou3, Zhaoxiang Zhang4
Corresponding author: Tianfei Zhou

###### Abstract

Acquiring sufficient ground-truth supervision to train deep visual models has
been a bottleneck over the years due to the data-hungry nature of deep
learning. This is exacerbated in some structured prediction tasks, such as
semantic segmentation, which requires pixel-level annotations. This work
addresses weakly supervised semantic segmentation (WSSS), with the goal of
bridging the gap between image-level annotations and pixel-level segmentation.
We formulate WSSS as a novel group-wise learning task that explicitly models
semantic dependencies in a group of images to estimate more reliable pseudo
ground-truths, which can be used for training more accurate segmentation
models. In particular, we devise a graph neural network (GNN) for group-wise
semantic mining, wherein input images are represented as graph nodes, and the
underlying relations between a pair of images are characterized by an
efficient co-attention mechanism. Moreover, in order to prevent the model from
paying excessive attention to common semantics only, we further propose a
graph dropout layer, encouraging the model to learn more accurate and complete
object responses. The whole network is end-to-end trainable by iterative
message passing, which propagates interaction cues over the images to
progressively improve the performance. We conduct experiments on the popular
PASCAL VOC 2012 and COCO benchmarks, and our model yields state-of-the-art
performance. Our code is available at: https://github.com/Lixy1997/Group-WSSS.

## Introduction

Figure 1: Architecture comparison of existing frameworks vs. Ours. (a) Single-
image models feed each image one by one into the network for training, which
bears high similarity with standard classifiers (e.g., VGG). (b) Pair-wise
methods extract features from a pair of images using a Siamese network, and
make predictions using a pair-wise classifier which has learned the
correlation between the two images. (c) We propose a group-wise method that
accepts an arbitrary number of images as input. The input images are
iteratively processed by a GNN to enable substantial information to exchange,
and a group-wise classifier is then adopted for prediction.

Semantic segmentation is a fundamental task in computer vision, aiming to
assign a semantic category to each pixel in an image. It can benefit a wide
variety of applications including autonomous driving, image editing and
medical diagnosis. With the recent renaissance of deep neural networks,
semantic segmentation has achieved tremendous progress. However, most of the
leading approaches (Long, Shelhamer, and Darrell 2015; Wang et al. 2019b; Zhou
et al. 2020a, b) are fully supervised, requiring massive amounts of pixel-
level annotated training data, which are extremely expensive and time-
consuming to obtain. In contrast, the weak supervision alternatives, e.g.,
image-level tags (Pathak, Krahenbuhl, and Darrell 2015; Kolesnikov and Lampert
2016; Qi et al. 2016; Wei et al. 2016; Chaudhry, Dokania, and Torr 2017; Ahn
and Kwak 2018; Fan et al. 2018), scribbles (Lin et al. 2016; Vernaza and
Chandraker 2017) or bounding-box annotations (Dai, He, and Sun 2015; Khoreva
et al. 2017; Song et al. 2019), are less costly. Thus, it is of interest to
explore the potential of these weak supervision cues in providing a data-
efficient solution for semantic segmentation. In this paper, we aim to address
weakly supervised semantic segmentation (WSSS) under the supervision of image-
level tags, which can be obtained effortlessly.

WSSS based on image tags is extremely challenging because fine-grained pixel-
level annotations, which are typically required for semantic segmentation, are
difficult to obtain from class labels. The pioneering work, (Zhou et al.
2016), proposes an efficient and straightforward way to solve this by
recognizing the discriminative regions specific to a given category using
class activation maps (CAMs), which are then refined to obtain pseudo ground-
truths for supervising a semantic segmentation network. Along this line, a
number of approaches have been proposed to improve the estimation of CAMs so
that they cover the full extent of objects rather than only the most
representative parts. For example, some approaches (Wei et al. 2017;
Kolesnikov and Lampert 2016; Choe and Shim 2019) manipulate internal feature
maps to guide the network to perceive easily ignored but essential parts,
while others (Hou et al. 2018; Chang et al. 2020; Fan et al. 2020a; Wang et
al. 2020c) adopt self-ensembling or self-supervision to improve localization.

However, the mainstream methods above are merely based on single images
(Figure 1 (a)), ignoring the valuable semantic context existing in a group of
images. The very recent studies (Fan et al. 2020b; Sun et al. 2020) utilized
Siamese networks to model the relations between a pair of images, leading to a
pair-wise solution (Figure 1 (b)). These approaches have proven effective in
locating more accurate object regions. However, seeking relations between two
images at a time is still limited in capturing substantial semantic context.
Accordingly, we introduce a more promising, and fundamentally different group-
wise solution (Figure 1 (c)) which comprehensively mines richer semantics from
a group of images. Our main motivation is that the availability of group
images containing instances of the same semantic classes can make up for the
absence of detailed supervisory information. From this perspective, we
hypothesize that it is desirable to take advantage of all available
information for WSSS, including not only individual image properties, but also
group-level synergetic relationships.

Based on the above analysis, we propose a novel deep learning model for WSSS.
Unlike previous pair-wise approaches, our model is targeted at group-wise
semantic mining to capture more comprehensive relations among input images.
Specifically, we develop an efficient, end-to-end trainable graph neural
network (GNN), and conduct recursive reasoning for group-wise semantic
understanding. In our graph, the nodes represent a group of input images, and
edges describe pair-wise relations between two connected images. We consider
two images as connected only if they share common semantic objects with each
other, and their relation is then characterized by an elaborately designed co-
attention mechanism. Through iterative message passing, the information from
individual elements can be efficiently integrated and broadcasted over the
graph structure. In this way, our model is capable of leveraging explicit
semantic dependencies among images to obtain better node representations.
However, this graph reasoning strategy mainly focuses on co-occurring
semantics in a group of images, ignoring isolated objects. To address this, we
further introduce a graph dropout layer, which can be seamlessly integrated
into the GNN for iterative inference. The graph dropout layer selectively
suppresses the most salient objects, forcing the network to be biased toward
their counterparts.

Our method has two appealing characteristics over single-image and pair-wise
methods. First, it is capable of learning semantic relations from an arbitrary
number of images using a flexible GNN framework. The GNN empowers our model to
inherit the complementary strengths of neural networks in learning capability
and graphical models in structure representations. Second, our model adopts
multi-step, iterative inference to progressively improve image
representations. This is more favorable than directly producing
representations by one-step inference in previous approaches

In summary, our main contributions are three-fold: First, we demonstrate the
advantages of group-wise semantic mining for WSSS, and proffer a graph-aware
solution for effective inference. Second, we develop a graph dropout layer to
promote the missing categories, leading to more accurate localization. Third,
we evaluate the proposed approach on two large-scale benchmarks, i.e., PASCAL
VOC 2012 (Everingham et al. 2010) and COCO (Lin et al. 2014), and the results
demonstrate its superiority.

## Related Work

Weakly Supervised Semantic Segmentation. Recent years have seen a surge of
interest in semantic segmentation under weak supervision (e.g., image-level
labels, scribbles, bounding boxes), greatly reducing human efforts in manual
labeling. In particular, methods operating with image-level labels have
attracted the most attention since they require minimal annotation efforts.
Most of these methods follow a popular pipeline that trains an image
classifier using image-level labels, and exploits CAMs to highlight the most
discriminative object regions for a particular semantic category as its pseudo
ground-truth. However, CAMs are weak in revealing complete object regions,
resulting in poor segmentation performance. Some pioneering efforts address
this difficulty by learning pixel affinities (Ahn and Kwak 2018), erasing the
most discriminative parts (Wei et al. 2017; Choe and Shim 2019; Lee et al.
2019), optimizing intra-class discrimination (Fan et al. 2020a), or applying
region growing (Kolesnikov and Lampert 2016; Wei et al. 2018; Huang et al.
2018) to capture the full extent of objects. However, these methods are
confined to using only limited image-level information. More recent approaches
thus follow the self-supervised paradigm to acquire additional supervisions
(Shimoda and Yanai 2019; Wang et al. 2020c), or rely on Siamese networks to
capture semantic relations between a pair of images (Fan et al. 2020b; Sun et
al. 2020).

In this paper, we take a further step toward discovering higher-order
relations among images. A graph model is designed to encode such
relationships. Through graph reasoning, our model iteratively refines object
representations by accepting informative knowledge from other images.

$\bm{l}_{4}^{m}$$\bm{l}_{3}^{m}$$\bm{l}_{2}^{m}$$\bm{l}_{1}^{m}$$\bm{l}_{1}^{g}$$\bm{l}_{2}^{g}$$\bm{l}_{3}^{g}$$\bm{l}_{4}^{g}$$\bm{l}_{1}$$\bm{l}_{2}$$\bm{l}_{3}$$\bm{l}_{4}$$\mathcal{G}=(\mathcal{V},\mathcal{E})$

Figure 2: Overview of the proposed group-wise semantic mining network during
the training phase. Given a group of images (i.e.,
$\\{\bm{I}_{i}\\}_{i=1}^{4}$), our model uses VGG16 to extract convolutional
features (i.e., $\bm{h}_{i}^{0}\\}_{i=1}^{4}$), which are used as the initial
embeddings for graph construction. Next, our model conducts $T$-step graph
reasoning to iteratively refine the features by message passing (Eq. (8)),
message aggregation (Eq. (2)), and graph dropout (Eq. (10)). The final
features (i.e., $\\{\hat{\bm{h}}{}^{T}_{i}\\}_{i=1}^{4}$) are fed into a
readout function (Eq. (9)) to get the predictions (i.e.,
$\\{\bm{l}_{i}^{g}\\}_{i=1}^{4}$).

Graph Neural Networks. Graph neural networks were proposed in (Scarselli et
al. 2008), and have since gained widespread attention due to their superiority
in dealing with flexible graph-structured data. GNNs typically model the graph
elements (e.g., nodes, edges) and approximation inference as learnable neural
networks, and conduct iterative reasoning to explicitly discover the relations
among nodes. They have achieved wide success in a variety of fields, including
molecular biology (Gilmer et al. 2017), computer vision (Qi et al. 2017; Lu et
al. 2020; Marino, Salakhutdinov, and Gupta 2017; Wang et al. 2019a; Santoro et
al. 2017; Wang et al. 2020a), and machine learning (Veličković et al. 2018;
Qu, Bengio, and Tang 2019). Inspired by these efforts, we build an image-level
graph network to model their semantic relations for the WSSS task. Assisted by
a graph dropout layer, our model can generate more accurate pseudo ground-
truths for semantic segmentation.

## Methodology

In this section, we elaborate on the proposed model for WSSS. Given training
images with only image-level labels, current efforts operate on two sub-tasks
to achieve pixel-wise predictions. The first one is pseudo ground-truth
generation, which relies on an image classification network to localize
discriminative regions. The other one is semantic segmentation, which conducts
dense predictions using a fully convolutional network (FCN) under the
supervision of pseudo labels. Our approach also follows this pipeline.
However, unlike previous approaches that treat each single image
independently, our model aims to mine common semantic patterns from multiple
images by graph inference. In this way, our model can alleviate the
incomplete-annotation problem in WSSS and produce more accurate pseudo labels.

### Preliminary: Graph Neural Networks

We start by revisiting the basic concept of GNNs. We define a graph
$\mathcal{G}\\!=\\!(\mathcal{V},\mathcal{E})$ by its node set
$\mathcal{V}\\!=\\!\\{v_{1},\ldots,v_{n}\\}$ and edge set
$\mathcal{E}\\!=\\!\\{e_{i,j}\\!=\\!(v_{i},v_{j})|v_{i},v_{j}\in\mathcal{V}\\}$.
We assume that each node $v_{i}$ is associated with a feature embedding vector
$\bm{h}_{i}$, and each edge $e_{i,j}$ has an edge representation
$\bm{e}_{i,j}$. During inference, GNNs iteratively improve the feature
representations at a node by aggregating its neighborhood features.
Specifically, a GNN maps the graph $\mathcal{G}$ to the node outputs through
two phases: a message passing phase and a readout phase. The message passing
phase is defined in terms of a message function $\mathcal{F}_{M}$, whose input
is a node’s features and output is a message, and an aggregation function
$\mathcal{F}_{A}$, whose input is a set of messages and output is the updated
features. Suppose we conduct $T$ rounds of message passing; the $t$-th round
for a node $v_{i}$ can be described as:

$\displaystyle\text{message
passing:}~{}~{}~{}\bm{m}_{i}^{t}=\sum_{v_{j}\\!\in\\!\mathcal{N}_{i}}\mathcal{F}_{M}^{t}(\bm{h}_{i}^{t-1},\bm{h}_{j}^{t-1},\bm{e}_{i,j}),$
(1) $\displaystyle\text{message
aggregation:}~{}~{}~{}\bm{h}_{i}^{t}=\mathcal{F}_{A}(\bm{h}_{i}^{t-1},\bm{m}_{i}^{t}),$
(2)

where for $v_{i}$, the message function firstly summarizes the information
(i.e., $\bm{m}_{i}^{t}$) from its neighbors $\mathcal{N}_{i}$, and then uses
it to update the node state. Then, in the readout phase, a task-specific
readout function $\mathcal{F}_{R}$ operates on the final node representation
$\bm{h}_{i}^{T}$ to produce a node output:

$\text{readout phase:}\quad o_{i}=\mathcal{F}_{R}(\bm{h}_{i}^{T}).$ (3)

Next, we will present the details of the proposed graph-based semantic mining
model for pseudo ground-truth generation in WSSS.

### Group-Wise Semantic Mining Network

Problem Definition: Given a collection of training samples, our first goal is
to generate corresponding pseudo ground-truths, which will later be used to
supervise semantic segmentation networks. To achieve this, we formulate the
problem as graph-based semantic co-mining among multiple images. Formally, we
denote $\mathcal{I}\\!=\\!\\{(\bm{I}_{i},\bm{l}_{i})\\}_{i=1}^{N}$ as the
training data, where
$\bm{I}_{i}\\!\in\\!\mathbb{R}^{w\\!\times\\!h\\!\times\\!3}$ is an image and
$\bm{l}_{i}\\!\in\\!\\{0,1\\}^{L}$ is the corresponding image-level ground-
truth with $L$ possible semantic categories. During training, we selectively
sample $K$ images $\\{\bm{I}_{i}\\}_{i=1}^{K}$ as a mini-batch, and model
their relations as a directed graph
$\mathcal{G}\\!=\\!(\mathcal{V},\mathcal{E})$, where the image $\bm{I}_{i}$ is
denoted as node $v_{i}\\!\in\\!\mathcal{V}$, and the relation between $v_{i}$
and $v_{j}$ is represented by edge $e_{i,j}\\!\in\\!\mathcal{E}$. To better
capture more comprehensive common semantics, we consider two nodes $v_{i}$ and
$v_{j}$ to be linked only if there is at least one semantic category shared
between them. Besides, we assume that every node has a self-edge, e.g.,
$e_{i,i}$ for $v_{i}$.

Given the above definitions, our network aims to conduct pseudo ground-truth
generation in a graph learning scheme, under the full supervision of image-
level labels as well as the implicit semantic relations among different
images. In this manner, our model can capture richer semantic information and
obtain more accurate pseudo labels. Next, we describe the details of each
component in our model.

Node Embedding: As an initial step, we abstract a high-level feature
representation for each input image. Formally, given $\bm{I}_{i}$, we extract
features $\bm{h}_{i}\\!\in\\!\mathbb{R}^{W\\!\times\\!H\\!\times\\!C}$ from
the convolutional stages of a standard classification network (e.g., VGG
(Simonyan and Zisserman 2015)). The embedding of node $v_{i}$ is then
initialized by $\bm{h}_{i}$, which is a $(W,H,C)$-dimensional tensor
preserving full spatial details for more effective pixel-level matching during
graph reasoning.

Edge Embedding: For each edge $e_{i,j}$ connecting $v_{i}$ to $v_{j}$, we aim
to learn an edge embedding $\bm{e}_{i,j}^{t}$ at each iteration $t$ to find
the correct semantic correspondence between the two nodes. This is achieved by
dense matching over node embeddings using the following bilinear model:

$\bm{e}_{i,j}^{t}=\bm{h}_{i}^{t}\bm{W}\bm{h}_{j}^{t\top}\in\mathbb{R}^{WH\\!\times\\!WH},$
(4)

where $\bm{h}_{i}^{t}\\!\in\\!\mathbb{R}^{WH\\!\times\\!C}$ and
$\bm{h}_{j}^{t}\\!\in\\!\mathbb{R}^{WH\\!\times\\!C}$ are flattened into
matrix representations for computational convenience.
$\bm{W}\in\mathbb{R}^{C\\!\times\\!C}$ is a trainable weight matrix. In Eq.
(4), $\bm{e}_{i,j}^{t}$ encodes the similarity between $\bm{h}_{i}^{t}$ and
$\bm{h}_{j}^{t}$ for all pairs of spatial locations. For the edge $e_{j,i}$,
its embedding at iteration $t$ is simply calculated as
$\bm{e}_{j,i}^{t}=\bm{e}_{i,j}^{t\top}$.

It should be noted that Eq. (4) introduces a large number of parameters,
increasing the computational cost. To alleviate this, $\bm{W}$ is
approximately factorized into two low-rank matrices
$\bm{P}\\!\in\\!\mathbb{R}^{C\\!\times\\!\frac{C}{d}}$ and
$\bm{Q}\\!\in\\!\mathbb{R}^{C\\!\times\\!\frac{C}{d}}$, where
$d~{}(d\\!>\\!1)$ is a reduction ratio. Then, Eq. (4) can be rewritten as:

$\bm{e}_{i,j}^{t}=\bm{h}_{i}^{t}\bm{P}\bm{Q}^{\top}\\!\bm{h}_{j}^{t\top}\in\mathbb{R}^{WH\\!\times\\!WH}.$
(5)

Eq. (5) has significant advantages over Eq. (4) in both model parameters and
computational cost: 1) it reduces the number of parameters by $2/d$ times; 2)
it only requires $(2WHC^{2}\\!+\\!W^{2}H^{2}C)/d$ multiplication operations,
instead of the $WHC^{2}+W^{2}H^{2}C$ in Eq. (4).

In addition, for each self-edge $e_{i,i}$, its embedding $\bm{e}_{i,i}$
captures the self-relation over the node representation $\bm{h}_{i}$. We
compute $\bm{e}_{i,i}^{t}$ at iteration $t$ by self-attention (Vaswani et al.
2017; Wang et al. 2018a), which can effectively capture long-range semantic
dependencies. In particular, the self-attention calculates the response at a
position by attending to all the positions within the same node embedding:

$\bm{e}_{i,i}^{t}=\text{softmax}(\phi_{f}(\bm{h}_{i}^{t})\phi_{g}^{\top}(\bm{h}_{i}^{t}))\phi_{h}(\bm{h}_{i}^{t})+\bm{h}_{i}^{t}\\!\in\\!\mathbb{R}^{W\\!\times\\!H\\!\times\\!C},$
(6)

where $\phi_{\\{f,g,h\\}}$ are $1\\!\times\\!1$ convolutional operators. As
seen, we also consider it to be a residual layer in Eq. (6), which can
effectively preserve information in the original feature map.

Message Passing: Given the node and edge embeddings, our model iteratively
updates the hidden states of graph nodes by applying message functions to
collect information from their neighboring nodes. More specifically, for a
node $v_{i}$, it absorbs knowledge along two types of edges: 1) a self-edge
$e_{i,i}$ that encodes rich context-aware knowledge in $v_{i}$; and 2) other
edges $\\{e_{j,i}\\}_{j}$ that connect $v_{j}$ to $v_{i}$. For the former, our
message function directly reads the message from $\bm{e}_{i,i}$, i.e.,
$\bm{m}_{i,i}^{t}\\!=\\!\bm{e}_{i,i}^{t-1}$; while for the latter, the
messages are summarized as:

$\bm{m}_{j,i}^{t}\\!=\\!\text{softmax}_{r}(\bm{e}_{i,j}^{t-1})\bm{h}_{j}^{t-1}\\!\in\\!\mathbb{R}^{WH\\!\times\\!C},$
(7)

where $\text{softmax}_{r}$ denotes the row-wise softmax operation. In Eq.
(7), we accumulate knowledge from $\bm{h}_{j}^{t-1}$, which is weighted based
on the similarity between $\bm{h}_{i}^{t-1}$ and $\bm{h}_{j}^{t-1}$.
$\bm{m}_{j,i}^{t}$ is then reshaped to a $(W,H,C)$-dimensional tensor. Then,
we can easily summarize the message for $v_{i}$ at the $t$-th iteration as:

$\bm{m}_{i}^{t}=\sum_{v_{j}\\!\in\\!\mathcal{N}_{i}}\bm{m}_{j,i}^{t-1}+\bm{m}_{i,i}^{t-1}.$
(8)

Next, the aggregation function $A$ updates the hidden states of nodes, as
given in Eq. (2). In our method, $A$ is instantiated by a ConvGRU network
(Ballas et al. 2016), which is an extension of the GRU update function used in
(Gilmer et al. 2017). In this way, the message passing algorithm runs for $T$
steps before convergence, iteratively collecting messages and updating node
embeddings.

Readout Phase: Having repeated the above process for $T$ time steps, we obtain
the final node embedding
$\bm{h}_{i}^{T}\\!\in\\!\mathbb{R}^{W\\!\times\\!H\\!\times\\!C}$ for $v_{i}$.
Then, the readout function $R$ is applied to the features $\bm{h}_{i}^{T}$ for
image classification:

$\bm{l}_{i}^{g}=\mathcal{F}_{R}(\bm{h}_{i}^{T})=\text{GAP}(\phi_{r}(\bm{h}_{i}^{T}))\in\mathbb{R}^{L},$
(9)

where $\phi_{r}$ is a class-aware convolutional layer with kernel size
$1\\!\times\\!1$ that obtains a feature map with $L$ channels, and GAP denotes
a global average pooling layer which produces the final classification
outputs.

Pseudo Ground-Truth Generation by Self-Ensembling: Once the classification
results are obtained (Eq. (9)), we discover the discriminative image regions
for a particular category following (Jiang et al. 2019). These regions are
further thresholded to generate pseudo ground-truths.

Besides, as shown in Figure 2, for each input image, our network produces two
outputs based on raw convolutional features $\bm{h}_{i}^{0}$ as well as
enriched features $\bm{h}_{i}^{T}$. This not only introduces additional deeply
supervised constraints (Lee et al. 2015) which could benefit the performance,
but also enables the results to be further improved by ensembling the CAMs of
multiple outputs. We found that the pseudo ground-truths from different
outputs are well complementary with each other, and self-ensembling them by
averaging can further improve the performance (see Table Performance on COCO).

### Graph Dropout Layer

The above graph reasoning scheme enables our model to discover common
semantics present in different images (Eq. (5)). The features of these
semantics can be accordingly enriched by summarizing all the information from
other images (Eq. (8)). However, standalone categories, which may exist only
in a single image, are almost ignored in this procedure. To address this, we
introduce a graph dropout layer to force the network to pay more attention to
these categories. Formally, given the feature map
$\bm{h}_{i}^{t}\\!\in\\!\mathbb{R}^{W\\!\times\\!H\\!\times\\!C}$ at the
$t$-th iteration, we average it along the channel dimension to obtain
$\bm{o}_{i}^{t}\\!\in\\!\mathbb{R}^{W\\!\times\\!H}$. Then, we generate a mask
$\bm{s}^{t}_{i}\\!\in\mathbb{R}^{W\\!\times\\!H}$ as follows:

$\bm{s}^{t}_{i}\\!=\\!\begin{cases}\text{sigmoid}(\bm{o}_{i}^{t}),~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}\text{if}~{}~{}r<\delta_{r};\\\
\bm{o}_{i}^{t}\mathbbm{1}(\bm{o}_{i}^{t}<\text{max}(\bm{o}_{i}^{t})*\delta_{d}),~{}~{}~{}\text{otherwise}.\end{cases}$
(10)

Here, the parameter $\delta_{r}$ is a drop rate threshold, determining whether
to carry out the dropout operation or not. The parameter $r$ is a scalar
generated from a random generator. If $r\\!<\\!\delta_{r}$, $\bm{s}^{t}_{i}$
is an importance map which supports the activations in $\bm{h}_{i}^{t}$;
otherwise, the layer drops the highly activated semantic regions to emphasize
standalone semantics. $\mathbbm{1}(x)$ is a matrix indicator function which
returns $1$ for the true elements in $x$, and $0$ otherwise. The
$\text{max}(\cdot)$ operation calculates the maximum value for a 2D tensor.
$\delta_{d}$ is a threshold controlling the dropout. Finally, we enhance the
feature maps by:

$\hat{\bm{h}}{}^{t}_{i}=\bm{h}_{i}^{t}\otimes\bm{s}^{t}_{i},$ (11)

where $\otimes$ denotes spatial-wise multiplication. Note that
$\hat{\bm{h}}{}^{t}_{i}$ is then used to replace original features
$\bm{h}_{i}^{t}$ in the next iteration.

Table 1: Quantitative comparison of different methods on PASCAL VOC 2012 val and test in terms of mIoU. ∗: VGG backbone. †: ResNet backbone. 2.5 Methods | Pub. | Val | Test
---|---|---|---
∗MEFF (Ge, Yang, and Yu 2018) | CVPR18 | - | 55.6%
∗GAIN (Li et al. 2018) | CVPR18 | 55.3% | 56.8%
∗MDC (Wei et al. 2018) | CVPR18 | 60.4% | 60.8%
∗RRM (Zhang et al. 2020) | AAAI20 | 60.7% | 61.0%
†MCOF (Wang et al. 2018b) | CVPR18 | 60.3% | 61.2%
†SeeNet (Hou et al. 2018) | NIPS18 | 63.1% | 62.8%
†DSRG (Huang et al. 2018) | CVPR18 | 61.4% | 63.2%
†AffinityNet (Ahn and Kwak 2018) | CVPR18 | 61.7% | 63.7%
†SS-WSSS (Araslanov and Roth 2020) | CVPR20 | 62.7% | 64.3%
†SSNet (Zeng et al. 2019) | ICCV19 | 63.3% | 64.3%
†IRNet (Ahn, Cho, and Kwak 2019) | CVPR19 | 63.5% | 64.8%
†CIAN (Fan et al. 2020b) | AAAI20 | 64.3% | 65.3%
†FickleNet (Lee et al. 2019) | CVPR19 | 64.9% | 65.3%
†IAL (Wang et al. 2020b) | IJCV20 | 64.3% | 65.4%
†SSDD (Shimoda and Yanai 2019) | ICCV19 | 64.9% | 65.5%
†SEAM (Wang et al. 2020c) | CVPR20 | 64.5% | 65.7%
†SubCat (Chang et al. 2020) | CVPR20 | 66.1% | 65.9%
†OAA+ (Jiang et al. 2019) | ICCV19 | 65.2% | 66.4%
†RRM (Zhang et al. 2020) | AAAI20 | 66.3% | 66.5%
†BES (Chen et al. 2020) | ECCV20 | 65.7% | 66.6%
†EME (Fan, Zhang, and Tan 2020) | ECCV20 | 67.2% | 66.7%
†MCIS (Sun et al. 2020) | ECCV20 | 66.2% | 66.9%
†ICD (Fan et al. 2020a) | CVPR20 | 67.8% | 68.0%
∗Ours (VGG16) | – | 63.3% | 63.6%
†Ours (ResNet101) | – | 68.2% | 68.5%

### Detailed Network Architecture

Our model is comprised of two sub-networks: a classification network for
group-wise pseudo ground-truth generation and a segmentation network for
semantic segmentation.

Classification Network. We choose VGG16 (Simonyan and Zisserman 2015) as the
backbone, which is pre-trained on ImageNet (Deng et al. 2009). We replace the
last convolutional layer in VGG16 by dilated convolutions with a rate of 2,
and the feature maps from this layer are taken as the initial node
representations for the GNN. For each image $\bm{I}_{i}$, our network has two
outputs: an intermediate output $\bm{l}^{m}_{i}$ which is directly obtained
from the backbone (Figure 2), and a final output $\bm{l}^{g}_{i}$ after graph
reasoning (Figure 2). Then, the loss function of the classification network
for image $i$ is:

$\mathcal{L}=\mathcal{L}_{\text{CE}}\left(\bm{l}_{i}^{g},\bm{l}_{i}\right)+\lambda\mathcal{L}_{\text{CE}}\left(\bm{l}_{i}^{m},\bm{l}_{i}\right),$
(12)

where $\mathcal{L}_{\text{CE}}$ indicates the standard sigmoid cross entropy
loss, and $\lambda$ balances the two losses.

After training, we obtain the CAMs for each training image from the two
classification layers mentioned earlier, and combine them to obtain foreground
object seeds. Besides, we also follow conventional practices (Jiang et al.
2019; Fan et al. 2020a) to estimate background seeds using an off-the-shelf
salient object detection model (Hou et al. 2017). The final pseudo labels are
generated by combining the foreground and background seeds.

Segmentation Network. Following (Chang et al. 2020; Fan et al. 2020b), we
choose DeepLab-v2 (Chen et al. 2017) as the segmentation network due to its
superior performance in fully supervised semantic segmentation tasks.

Table 2: Quantitative comparison of different methods on COCO val in terms of mIoU. All methods use VGG16 as the backbone. 2.5 Methods | Pub. | Val
---|---|---
BFBP (Saleh et al. 2016) | ECCV16 | 20.4%
SEC (Kolesnikov and Lampert 2016) | ECCV16 | 22.4%
DSRG (Huang et al. 2018) | CVPR18 | 26.0%
IAL (Wang et al. 2020b) | IJCV20 | 27.7%
Ours | – | 28.4%

## Experiments

### Experimental Setup

Datasets: We conduct our experiments on two datasets: PASCAL VOC 2012
(Everingham et al. 2010) and COCO (Lin et al. 2014). 1) PASCAL VOC 2012 is
currently the most popular benchmark for WSSS. The dataset contains 20
semantic categories (e.g., person, bicycle, cow) and one background category.
Following standard protocol (Huang et al. 2018; Lee et al. 2019; Wang et al.
2020c), extra data from SBD (Hariharan et al. 2011) is also used for training,
leading to a total of 10,582 training images. We evaluate our model on the
standard validation and test sets, which have 1,449 and 1,456 images,
respectively. 2) COCO is a more challenging benchmark with 80 semantic
classes. Since more complex contextual relations exist among these categories,
it is interesting to examine the performance of our model in this dataset.
Following (Wang et al. 2020b), we use the default train/val splits (80k images
for training and 40k for validation) in the experiment.

Evaluation Metric: For fair comparison, we utilize a widely used metric (Wang
et al. 2018b; Choe and Shim 2019; Sun et al. 2020), mean Intersection-over-
Union (mIoU), for evaluation. The scores on the test set of PASCAL VOC are
obtained from the official evaluation server.

Training Details: 1) Greedy Mini-Batch Sampling. During training, we design a
heuristic, greedy strategy to sample $K$ training images in each mini-batch.
Starting from a randomly sampled image $\bm{I}_{i}$, we further find another
$K\\!-\\!1$ images, each of which shares as many common semantic objects with
$\bm{I}_{i}$ as possible. These $K$ images are then used to build a $K$-node
GNN. This sampling strategy enables our model to better explore rich
relationships among groups of images and improve the results. 2) Training
Settings. For the classification network, the number of nodes $K$ and message
passing steps $T$ in the GNN are separately set to 4 and 3 by default. The
input image size is $224\\!\times\\!224$. The entire network is trained using
the SGD optimizer with initial learning rates of 1e-3 for the backbone and
1e-2 for the GNN, which are reduced by 0.1 every five epochs. The total number
of epochs, momentum and weight decay are set to 15, 0.9, and 5e-4,
respectively. The $\lambda$ in Eq. (12) is empirically set to 0.4 and the $d$
in Eq. (5) is set to 4. For the segmentation network, we follow the training
setting in (Chen et al. 2017), but use the generated pseudo ground-truths as
the supervision.

Reproducibility: Our network is implemented in PyTorch and trained on four
NVIDIA RTX 2080Ti GPUs with 11GB memory per card. The testing is conducted on
the same machine with one GPU card.

### Performance on PASCAL VOC 2012

We evaluate the proposed approach on PASCAL VOC 2012 against current top-
performing WSSS methods that only operate with image-level labels. Following
conventions, we evaluate the performance of our model using VGG16 (Simonyan
and Zisserman 2015) and ResNet101 (He et al. 2016) as the backbones,
respectively. As reported in Table 1, our model with ResNet101 achieves the
best performance, scoring 68.2% and 68.5% on the val and test sets,
respectively. It significantly outperforms the current leading approach, i.e.,
ICD (Fan et al. 2020a), by +0.4% and +0.5% on the two evaluation sets.

In addition, Table 1 also shows that the proposed approach outperforms both
pair-wise models (i.e., CIAN (Fan et al. 2020b) and MCIS (Sun et al. 2020)),
and all single-image based models (e.g., RRM (Zhang et al. 2020), OAA+ (Jiang
et al. 2019)), by a large margin. The reason lies in that existing methods
exploit limited context in image collection, while our approach can learn more
effective inter-image representations with GNNs.

In Figure 3, we also provide sample results for representative images in
PASCAL VOC 2012 val. The images cover various challenging factors in WSSS,
such as multiple objects, different semantic categories, small objects, and
cluttered background. We see that our model can deal with these difficulties
well, resulting in appealing segmentation results.

### Performance on COCO

We further examine the performance of our model on COCO. As reported in Table
2, our model achieves the best mIoU score (i.e., 28.4%) on the validation set,
outperforming the second-best result, i.e., IAL (Wang et al. 2020b), by 0.7%.
This further proves the superiority of our model.

Table 3: Diagnostic experiments of our model on PASCAL VOC 2012 val in terms
of mIoU. For all variants, we use ResNet101 as the backbone.

2.5 Aspect |  Variant | mIoU
---|---|---
Full Model |  $T=3$, $K=4$ |  68.2%
$\delta_{r}=0.8$, $\delta_{d}=0.7$
|  Node Number | $K=3$ | 68.1%
| $K=5$ | 67.8%
| $K=6$ | 67.6%
Graph Reasoning |  Message Passing | $T=2$ | 67.8%
$T=4$ | 68.0%
$T=5$ | 68.0%
|  Graph Dropout |  $\delta_{r}=0.8$ | $\delta_{d}=0.9$ | 68.0%
| $\delta_{d}=0.5$ | 67.7%
| $\delta_{r}=0.6$ |  $\delta_{d}=0.7$ | 66.8%
| $\delta_{r}=0.4$ | 63.6%
|  w/o dropout | 67.7%
Self-Ensembling | intermediate output | 64.1%
graph output | 67.8%
self-ensembling | 68.2%

Figure 3: Qualitative results on PASCAL VOC 2012 val. From top to bottom:
input images, ground-truths, and our segmentation results. Figure 4: Visual
comparisons of CAMs generated w/ or w/o the graph dropout layer.

Figure 5: Visual comparisons of CAMs. Here we provide the results of two
groups of images. For each group, we show the input images, CAMs from the
intermediate readout layer and CAMs from the graph readout layer (from left to
right). Our model clearly provides more accurate CAMs after group-wise graph
reasoning.

### Diagnostic Experiments

We further conduct diagnostic analysis on PASCAL VOC 2012 val set to verify
the effectiveness of the essential modules in our approach. We use ResNet101
as the default backbone for all the studies. The performance of our full model
with default parameters is given in the first row of Table Performance on
COCO.

Number of Nodes $K$: We first investigate the effect of the node number $K$
used in the GNN, which indicates the number of images in a group. As shown in
Table Performance on COCO, the model achieves comparably high performance with
three or four nodes. However, when more nodes are added, the performance
decreases significantly. This can be attributed to the trade-off between
meaningful semantic relations and noise brought by group images. When
$K\\!=\\!3$ or $4$, the semantic relations can be fully exploited to improve
the integral regions of objects. However, when more images are further
considered, meaningful semantic cues reach a bottleneck and noise, introduced
by imperfect localization of the classifier, dominates, thus leading to
performance degradation.

Number of Message Passing Steps $T$: We further evaluate the impact of the
message passing steps by comparing the performance with different $T$ ranging
from 2 to 5. From Table Performance on COCO, we observe that the mIoU score is
significantly improved when $T$ varies from 2 to 3. The performance decreases
slightly when considering more steps. Therefore, we set $T\\!=\\!3$ as default
for message passing.

Graph Dropout Layer: To verify the effectiveness of the proposed graph dropout
layer, we design multiple experiments to search the optimal configuration of
parameters drop-rate and drop-th. We observe that both parameters have great
influences on the performance. As observed in Table Performance on COCO, our
model reaches the best performance at $\delta_{r}\\!=\\!0.8$ and
$\delta_{d}\\!=\\!0.7$. If $\delta_{d}$ is higher (e.g., 0.9), most
discriminative regions will be kept, and thus ignored regions will remain
unactivated. In contrast, if the $\delta_{d}$ is lower, the regions with high
responses will be excessively dropped, leading to degraded classification
accuracy.

In addition, the parameter $\delta_{r}$ controls whether to drop the responses
or not during training. As shown in Table Performance on COCO, a $\delta_{r}$
of $0.8$ helps to achieve the best mIoU score. Such a setting not only
maintains the classification ability of the network by keeping discriminative
regions with a high probability, but also drives the network to mildly attend
to other regions. We can also see that by setting $\delta_{r}$ to smaller
values (e.g., $0.6$ or $0.4$), the performance encounters a significant
decrease.

Moreover, we examine the performance of our model without the graph dropout
layer. As seen, without the dropout layer, the performance of our model
decreases by $0.5\%$ in terms of mIoU, which reveals its importance.

Finally, we illustrate some examples of the final CAMs generated with or
without the graph dropout layer. As shown in Figure 4, without the dropout
layer, the network only focuses on the most discriminative parts (e.g., heads
of the cat and the horse). This is improved with our dropout layer, which
helps to highlight non-discriminative object regions.

Self-Ensembling: In addition to the supervision on the final outputs, we also
introduce deep supervision signals on the intermediate features. Such multi-
level supervision has proven effective for improving the performance of
various vision tasks. Besides, this enables us to combine the multiple outputs
with low cost to further boost the performance. Here, we examine the self-
ensembling strategy by building three network variants, i.e., intermediate
output, graph output and self-ensembling, in which the final CAMs are
separately extracted from the intermediate readout layer, graph-aware readout
layer, and their ensemble, respectively. As shown in Table Performance on
COCO, the intermediate output only obtains an mIoU score of $64.1\%$, greatly
lagging behind the $67.8\%$ obtained by the graph output. This demonstrates
that through iterative graph reasoning, our model can improve the image
representations by integrating information from group images, leading to huge
performance gains. Furthermore, the self-ensembling strategy boosts the
performance to $68.2\%$.

In Figure 5, we illustrate two groups of images with their CAMs from the
intermediate readout layer and graph readout layer. As seen, in both groups,
the CAMs are well-refined to cover more complete foreground regions after
graph reasoning. Besides, in many cases, the CAMs from two output layers
complement with each other well, enabling better results to be obtained by
self-ensembling.

## Conclusion

In this paper, we have introduced a group-wise learning framework for weakly
supervised semantic segmentation (WSSS). We formulate the task within a graph
neural network (GNN), which operates on a group of images and explores their
semantic relations for representation learning. By iterative graph reasoning,
our model provides better pseudo ground-truths, which further lead to
significant performance improvement for the semantic segmentation results. We
also devise a graph dropout layer to facilitate the discovery of complete
object regions. We conduct extensive experiments on PASCAL VOC 2012 and COCO
benchmarks, and the results demonstrate that the proposed approach performs
favorably against the state-of-the-art methods.

## References

  * Ahn, Cho, and Kwak (2019) Ahn, J.; Cho, S.; and Kwak, S. 2019.  Weakly supervised learning of instance segmentation with inter-pixel relations.  In _CVPR_ , 2209–2218.
  * Ahn and Kwak (2018) Ahn, J.; and Kwak, S. 2018.  Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation.  In _CVPR_ , 4981–4990.
  * Araslanov and Roth (2020) Araslanov, N.; and Roth, S. 2020.  Single-Stage Semantic Segmentation from Image Labels.  In _CVPR_ , 4253–4262.
  * Ballas et al. (2016) Ballas, N.; Yao, L.; Pal, C.; and Courville, A. 2016.  Delving deeper into convolutional networks for learning video representations.  In _ICLR_.
  * Chang et al. (2020) Chang, Y.-T.; Wang, Q.; Hung, W.-C.; Piramuthu, R.; Tsai, Y.-H.; and Yang, M.-H. 2020.  Weakly-Supervised Semantic Segmentation via Sub-Category Exploration.  In _CVPR_ , 8991–9000.
  * Chaudhry, Dokania, and Torr (2017) Chaudhry, A.; Dokania, P. K.; and Torr, P. H. 2017.  Discovering class-specific pixels for weakly-supervised semantic segmentation.  _arXiv preprint arXiv:1707.05821_ .
  * Chen et al. (2020) Chen, L.; Wu, W.; Fu, C.; Han, X.; and Zhang, Y. 2020.  Weakly Supervised Semantic Segmentation with Boundary Exploration.  In _ECCV_.
  * Chen et al. (2017) Chen, L.-C.; Papandreou, G.; Kokkinos, I.; Murphy, K.; and Yuille, A. L. 2017.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  _IEEE TPAMI_ 40(4): 834–848.
  * Choe and Shim (2019) Choe, J.; and Shim, H. 2019.  Attention-based dropout layer for weakly supervised object localization.  In _CVPR_ , 2219–2228.
  * Dai, He, and Sun (2015) Dai, J.; He, K.; and Sun, J. 2015.  Boxsup: Exploiting bounding boxes to supervise convolutional networks for semantic segmentation.  In _ICCV_ , 1635–1643.
  * Deng et al. (2009) Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009.  Imagenet: A large-scale hierarchical image database.  In _CVPR_ , 248–255.
  * Everingham et al. (2010) Everingham, M.; Van Gool, L.; Williams, C. K.; Winn, J.; and Zisserman, A. 2010\.  The pascal visual object classes (voc) challenge.  _IJCV_ 88(2): 303–338.
  * Fan et al. (2020a) Fan, J.; Zhang, Z.; Song, C.; and Tan, T. 2020a.  Learning Integral Objects With Intra-Class Discriminator for Weakly-Supervised Semantic Segmentation.  In _CVPR_ , 4283–4292.
  * Fan, Zhang, and Tan (2020) Fan, J.; Zhang, Z.; and Tan, T. 2020.  Employing Multi-Estimations for Weakly-Supervised Semantic Segmentation.  In _ECCV_.
  * Fan et al. (2020b) Fan, J.; Zhang, Z.; Tan, T.; Song, C.; and Xiao, J. 2020b.  CIAN: Cross-Image Affinity Net for Weakly Supervised Semantic Segmentation.  In _AAAI_ , 10762–10769.
  * Fan et al. (2018) Fan, R.; Hou, Q.; Cheng, M.-M.; Yu, G.; Martin, R. R.; and Hu, S.-M. 2018.  Associating inter-image salient instances for weakly supervised semantic segmentation.  In _ECCV_ , 367–383.
  * Ge, Yang, and Yu (2018) Ge, W.; Yang, S.; and Yu, Y. 2018.  Multi-evidence filtering and fusion for multi-label classification, object detection and semantic segmentation based on weakly supervised learning.  In _CVPR_ , 1277–1286.
  * Gilmer et al. (2017) Gilmer, J.; Schoenholz, S. S.; Riley, P. F.; Vinyals, O.; and Dahl, G. E. 2017.  Neural message passing for quantum chemistry.  In _ICML_.
  * Hariharan et al. (2011) Hariharan, B.; Arbeláez, P.; Bourdev, L.; Maji, S.; and Malik, J. 2011.  Semantic contours from inverse detectors.  In _ICCV_ , 991–998.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Identity mappings in deep residual networks.  In _ECCV_ , 630–645.
  * Hou et al. (2017) Hou, Q.; Cheng, M.-M.; Hu, X.; Borji, A.; Tu, Z.; and Torr, P. H. 2017.  Deeply supervised salient object detection with short connections.  In _ICCV_ , 3203–3212.
  * Hou et al. (2018) Hou, Q.; Jiang, P.; Wei, Y.; and Cheng, M.-M. 2018.  Self-erasing network for integral object attention.  In _NeurIPS_ , 549–559.
  * Huang et al. (2018) Huang, Z.; Wang, X.; Wang, J.; Liu, W.; and Wang, J. 2018.  Weakly-supervised semantic segmentation network with deep seeded region growing.  In _CVPR_ , 7014–7023.
  * Jiang et al. (2019) Jiang, P.-T.; Hou, Q.; Cao, Y.; Cheng, M.-M.; Wei, Y.; and Xiong, H.-K. 2019.  Integral object mining via online attention accumulation.  In _ICCV_ , 2070–2079.
  * Khoreva et al. (2017) Khoreva, A.; Benenson, R.; Hosang, J.; Hein, M.; and Schiele, B. 2017.  Simple does it: Weakly supervised instance and semantic segmentation.  In _CVPR_ , 876–885.
  * Kolesnikov and Lampert (2016) Kolesnikov, A.; and Lampert, C. H. 2016.  Seed, expand and constrain: Three principles for weakly-supervised image segmentation.  In _ECCV_ , 695–711.
  * Lee et al. (2015) Lee, C.-Y.; Xie, S.; Gallagher, P.; Zhang, Z.; and Tu, Z. 2015.  Deeply-supervised nets.  In _Artificial intelligence and statistics_ , 562–570.
  * Lee et al. (2019) Lee, J.; Kim, E.; Lee, S.; Lee, J.; and Yoon, S. 2019.  Ficklenet: Weakly and semi-supervised semantic image segmentation using stochastic inference.  In _CVPR_ , 5267–5276.
  * Li et al. (2018) Li, K.; Wu, Z.; Peng, K.-C.; Ernst, J.; and Fu, Y. 2018.  Tell me where to look: Guided attention inference network.  In _CVPR_ , 9215–9223.
  * Lin et al. (2016) Lin, D.; Dai, J.; Jia, J.; He, K.; and Sun, J. 2016.  Scribblesup: Scribble-supervised convolutional networks for semantic segmentation.  In _CVPR_ , 3159–3167.
  * Lin et al. (2014) Lin, T.-Y.; Maire, M.; Belongie, S.; Hays, J.; Perona, P.; Ramanan, D.; Dollár, P.; and Zitnick, C. L. 2014.  Microsoft coco: Common objects in context.  In _ECCV_ , 740–755.
  * Long, Shelhamer, and Darrell (2015) Long, J.; Shelhamer, E.; and Darrell, T. 2015.  Fully convolutional networks for semantic segmentation.  In _CVPR_ , 3431–3440.
  * Lu et al. (2020) Lu, X.; Wang, W.; Danelljan, M.; Zhou, T.; Shen, J.; and Van Gool, L. 2020.  Video object segmentation with episodic graph memory networks.  In _ECCV_.
  * Marino, Salakhutdinov, and Gupta (2017) Marino, K.; Salakhutdinov, R.; and Gupta, A. 2017.  The more you know: Using knowledge graphs for image classification.  In _CVPR_ , 20–28.
  * Pathak, Krahenbuhl, and Darrell (2015) Pathak, D.; Krahenbuhl, P.; and Darrell, T. 2015.  Constrained convolutional neural networks for weakly supervised segmentation.  In _ICCV_ , 1796–1804.
  * Qi et al. (2017) Qi, X.; Liao, R.; Jia, J.; Fidler, S.; and Urtasun, R. 2017.  3D graph neural networks for rgbd semantic segmentation.  In _ICCV_ , 5199–5208.
  * Qi et al. (2016) Qi, X.; Liu, Z.; Shi, J.; Zhao, H.; and Jia, J. 2016.  Augmented feedback in semantic segmentation under image level supervision.  In _ECCV_ , 90–105.
  * Qu, Bengio, and Tang (2019) Qu, M.; Bengio, Y.; and Tang, J. 2019.  GMNN: Graph Markov Neural Networks.  In _ICML_ , 5241–5250.
  * Saleh et al. (2016) Saleh, F.; Aliakbarian, M. S.; Salzmann, M.; Petersson, L.; Gould, S.; and Alvarez, J. M. 2016.  Built-in foreground/background prior for weakly-supervised semantic segmentation.  In _ECCV_ , 413–432.
  * Santoro et al. (2017) Santoro, A.; Raposo, D.; Barrett, D. G.; Malinowski, M.; Pascanu, R.; Battaglia, P.; and Lillicrap, T. 2017.  A simple neural network module for relational reasoning.  In _NeurIPS_ , 4967–4976.
  * Scarselli et al. (2008) Scarselli, F.; Gori, M.; Tsoi, A. C.; Hagenbuchner, M.; and Monfardini, G. 2008\.  The graph neural network model.  _IEEE TNN_ 20(1): 61–80.
  * Shimoda and Yanai (2019) Shimoda, W.; and Yanai, K. 2019.  Self-supervised difference detection for weakly-supervised semantic segmentation.  In _ICCV_ , 5208–5217.
  * Simonyan and Zisserman (2015) Simonyan, K.; and Zisserman, A. 2015.  Very deep convolutional networks for large-scale image recognition.  In _ICLR_.
  * Song et al. (2019) Song, C.; Huang, Y.; Ouyang, W.; and Wang, L. 2019.  Box-driven class-wise region masking and filling rate guided loss for weakly supervised semantic segmentation.  In _CVPR_ , 3136–3145.
  * Sun et al. (2020) Sun, G.; Wang, W.; Dai, J.; and Van Gool, L. 2020.  Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentation.  In _ECCV_.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A. N.; Kaiser, Ł.; and Polosukhin, I. 2017.  Attention is all you need.  In _NeurIPS_ , 5998–6008.
  * Veličković et al. (2018) Veličković, P.; Cucurull, G.; Casanova, A.; Romero, A.; Lio, P.; and Bengio, Y. 2018.  Graph attention networks.  In _ICLR_.
  * Vernaza and Chandraker (2017) Vernaza, P.; and Chandraker, M. 2017.  Learning random-walk label propagation for weakly-supervised semantic segmentation.  In _CVPR_ , 7158–7166.
  * Wang et al. (2019a) Wang, W.; Lu, X.; Shen, J.; Crandall, D. J.; and Shao, L. 2019a.  Zero-shot video object segmentation via attentive graph neural networks.  In _ICCV_ , 9236–9245.
  * Wang et al. (2019b) Wang, W.; Zhang, Z.; Qi, S.; Shen, J.; Pang, Y.; and Shao, L. 2019b.  Learning compositional neural information fusion for human parsing.  In _ICCV_ , 5703–5713.
  * Wang et al. (2020a) Wang, W.; Zhu, H.; Dai, J.; Pang, Y.; Shen, J.; and Shao, L. 2020a.  Hierarchical human parsing with typed part-relation reasoning.  In _CVPR_ , 8929–8939.
  * Wang et al. (2018a) Wang, X.; Girshick, R.; Gupta, A.; and He, K. 2018a.  Non-local neural networks.  In _CVPR_ , 7794–7803.
  * Wang et al. (2020b) Wang, X.; Liu, S.; Ma, H.; and Yang, M.-H. 2020b.  Weakly-Supervised Semantic Segmentation by Iterative Affinity Learning.  _IJCV_ 1–14.
  * Wang et al. (2018b) Wang, X.; You, S.; Li, X.; and Ma, H. 2018b.  Weakly-supervised semantic segmentation by iteratively mining common object features.  In _CVPR_ , 1354–1362.
  * Wang et al. (2020c) Wang, Y.; Zhang, J.; Kan, M.; Shan, S.; and Chen, X. 2020c.  Self-supervised Equivariant Attention Mechanism for Weakly Supervised Semantic Segmentation.  In _CVPR_ , 12275–12284.
  * Wei et al. (2017) Wei, Y.; Feng, J.; Liang, X.; Cheng, M.-M.; Zhao, Y.; and Yan, S. 2017.  Object region mining with adversarial erasing: A simple classification to semantic segmentation approach.  In _CVPR_ , 1568–1576.
  * Wei et al. (2016) Wei, Y.; Liang, X.; Chen, Y.; Shen, X.; Cheng, M.-M.; Feng, J.; Zhao, Y.; and Yan, S. 2016.  Stc: A simple to complex framework for weakly-supervised semantic segmentation.  _IEEE TPAMI_ 39(11): 2314–2320.
  * Wei et al. (2018) Wei, Y.; Xiao, H.; Shi, H.; Jie, Z.; Feng, J.; and Huang, T. S. 2018.  Revisiting dilated convolution: A simple approach for weakly-and semi-supervised semantic segmentation.  In _CVPR_ , 7268–7277.
  * Zeng et al. (2019) Zeng, Y.; Zhuge, Y.; Lu, H.; and Zhang, L. 2019.  Joint learning of saliency detection and weakly supervised semantic segmentation.  In _ICCV_ , 7223–7233.
  * Zhang et al. (2020) Zhang, B.; Xiao, J.; Wei, Y.; Sun, M.; and Huang, K. 2020.  Reliability Does Matter: An End-to-End Weakly Supervised Semantic Segmentation Approach.  In _AAAI_ , 12765–12772.
  * Zhou et al. (2016) Zhou, B.; Khosla, A.; Lapedriza, A.; Oliva, A.; and Torralba, A. 2016.  Learning deep features for discriminative localization.  In _CVPR_ , 2921–2929.
  * Zhou et al. (2020a) Zhou, T.; Li, J.; Wang, S.; Tao, R.; and Shen, J. 2020a.  MATNet: Motion-Attentive Transition Network for Zero-Shot Video Object Segmentation.  _IEEE TIP_ 29: 8326–8338.
  * Zhou et al. (2020b) Zhou, T.; Wang, S.; Zhou, Y.; Yao, Y.; Li, J.; and Shao, L. 2020b.  Motion-Attentive Transition for Zero-Shot Video Object Segmentation.  _arXiv preprint arXiv:2003.04253_ .


Zhiping Xu∗

# A Data-Driven Approach to Morphogenesis under Structural Instability

Yingjie Zhao<EMAIL_ADDRESS>Applied Mechanics Laboratory, Department of
Engineering Mechanics, Tsinghua University, Beijing 100084, China

###### Abstract

Morphological development into evolutionary patterns under structural
instability is ubiquitous in living systems and often of vital importance for
engineering structures. Here we propose a data-driven approach to understand
and predict their spatiotemporal complexities. A machine-learning framework is
proposed based on the physical modeling of morphogenesis triggered by internal
or external forcing. Digital libraries of structural patterns are constructed
from the simulation data, which are then used to recognize the abnormalities,
predict their development, and assist in risk assessment and prognosis. The
capabilities to identify the key bifurcation characteristics and predict the
history-dependent development from the global and local features are
demonstrated by examples of brain growth and aerospace structural design,
which offer guidelines for disease diagnosis/prognosis and instability-
tolerant design.

## Introduction

Morphological evolution and pattern development are key processes of growing
living organisms and can be detrimental to engineering structures in service
(Fig. 1a) [1, 2, 3]. Brain growth from neural tubes into complex cortical
folds sees prominent structural evolution. Cerebral palsy and seizures are
highly correlated with abnormal brain morphologies [4, 5, 6]. In engineering,
skins of thin-walled aerospace structures buckle under residual compressive
stress, resulting in wrinkled morphology and risks of catastrophic structural
failure due to abrupt changes in the morphology and load redistribution in the
structures [7]. Predicting the loss of stability and subsequent morphological
development are of significant importance in disease diagnosis and engineering
design. However, the processes can be complicated by the underlying, often
nonlinear processes spanning over multiple lengths and time scales.

The construction of predictive models with simultaneously high accuracy and
high efficacy is challenging. Global features and local characteristic
patterns of the brain are inherited from the genetic, molecular, cellular, and
mechanical factors in brain development [8, 9]. The nucleation and
amplification of wrinkling or buckling patterns in the structural components
are relevant to both residual stresses and external loading conditions [10].
In practice, domain knowledge is often harnessed to make decisions and
predictions based on the development of morphological patterns. Medical
experts diagnose brain abnormalities through visual judgments of neuroimaging
features and clinical phenotyping [4]. Safety factors are defined as the ratio
between the ultimate and limit loads to prevent structural failure in
aerospace engineering design [11].

Massive data collected in past decades from experimental tests and high-
fidelity computer simulations make it possible to construct databases to
assist in disease diagnosis and the design of aerospace structures. For brain
abnormalities, computed tomography (CT) and magnetic resonance imaging (MRI)
provide experimental data of the organ morphologies, which is limited by the
resolution of techniques though [12]. High-fidelity physics-based simulation
results contain full-scale information on brain morphology, which are,
however, constrained by the experimental facts [13, 14, 15, 16]. The emergence
of morphological features such as sulci and gyri indicate large deformation
that often leads to geometric, material, and contact nonlinearities in the
problem [17, 18, 19]. The transition from smooth to corrugated surfaces is
history-dependent, which introduces additional difficulties in the modeling
(Supplementary Fig. 1a). Recently, machine-learning techniques show the
potential to address these issues. Specially designed network architectures
such as the long short-term memory (LSTM) network could tackle the
nonlinearities and capture the historical dependency, which is used to learn
the process of morphological development and construct the phase diagrams of
bifurcation [20, 21, 22].

In aerospace engineering design, thin-walled structures are often used to
reduce weight, fuel consumption, and the costs of manufacturing and operating
[7]. Plates and cylindrical shells are endangered by buckling withstanding
compressive loading [23, 24]. The morphological development features local,
tiny wrinkles after the first buckling, and then intense global crumples that
may lead to collapse [25]. In conservative design, the ultimate load is
commonly set below the first buckling load to prevent instability [7].
Numerically accurate and experimentally validated simulations of post-buckling
and structural collapse can elevate the ultimate load beyond the first-
buckling load while below the collapse load [26, 27]. Data-driven methods
adequately characterize the phase space of structural stability and the
bifurcation diagram can forecast the post-buckling and failure behaviors, and
allows _instability-tolerant_ design reminiscent of the ‘damage-tolerance’
design for structural integrity (Fig. 1b). The procedure can be made highly
efficient and integrated into the ‘digital twin’ technology provided with
high-fidelity databases prepared in advance.

In this work, a machine-learning framework is developed by combining finite
element analysis (FEA), the PointNet algorithm, and the LSTM network to
resolve the morphological and developmental complexities in the presence of
structural instability (Fig. 1c). Digital libraries of simulated brain and
aerospace structures are constructed using FEA [15, 28, 16, 14, 25, 7]. We use
PointNet for feature extraction and dimensionality reduction of the
morphological features, and LSTM to capture the historical dependency. Key
indicators such as the gyrification index (GI), mean curvatures, localization
factor ($L_{\rm F}$), and dynamic eigenvalues (DEVs) are analyzed. The
information is then fed into tasks including morphological recognition,
bifurcation identification, developmental prediction, and coarse-graining of
the processes.

## Results

### Digital libraries

We construct digital libraries of morphological patterns that are then used to
identify abnormal patterns, predict their development, and assist in diagnosis
and risk reduction (Fig. 1b,c). To model brain morphologies, a representative
core-shell geometry is used for its simplicity [28, 16, 29, 3], as implemented
to explore the mechanical instability in cortical folding [15, 30, 13, 31, 32,
14]. The outer spherical shell represents the cerebral cortex (gray matter),
and the inner core for the white matter. Following the experimental evidences
[33, 34, 35, 36], the cortical thickness ranges from $0.03-1.63$ $\rm{mm}$
according to the abnormal and normal human cerebral cortex measurements and
the scale factor [33, 34], and the relative shear modulus ($\mu_{\rm
grey}/\mu_{\rm white}$) ranges from $0.65-1$ [35, 36]. The tangential growth
(TG) model is used to simulate the cellular mechanisms that create the growth
stresses and lead to the pattern evolution (see Methods for details) [28, 15,
9].

In aerospace engineering design, thin plates and cylindrical shells are
commonly used in primary structures of aircraft and rockets for structural
efficiency [23, 24]. The Johnson-Cook model of plasticity is used for the
materials [37, 38]. Axial compression is applied to trigger buckling of the
thin-walled structures [39].

### Morphological recognition

To resolve the morphological complexity and for diagnosis purposes, the first
step is to recognize the geometrical features during brain development, such
as the GI and mean curvatures. We discretize the brain surfaces into a
tetrahedral mesh with a density of $10^{6}$ tetrahedra/cm3 [28, 15, 16]. The
convergence of calculated descriptors is confirmed. The value of GI measures
the depth and width of the cortical folding [40] and is defined as the areal
ratio of the cortical surface to the smooth convex hull (Fig. 2a),

${\rm GI}=\frac{S_{\rm cortex}}{S_{\rm convex}}.$ (1)

The mean curvatures are defined as the average of the two principal curvatures
($k_{\rm max}$ and $k_{\rm min}$) (Supplementary Fig. 2a),

$\kappa=\frac{k_{\rm max}+k_{\rm min}}{2},$ (2)

which quantify the local corrugation of cortical folding. The average value of
absolute curvature, $\lvert\kappa\rvert$, is defined as a key geometrical
indicator [16]. The variance in $\kappa$ increases as the cortical convolution
becomes stronger. In addition to the geometrical measures, topological
features are also crucial for neuronal message-passing through chemical and
electrical signals across the gyri-to-gyri contact, and vital for brain
morphologies and functions [41]. A localization factor ($L_{\rm F}$) is thus
defined as the mean distance of contact mapped into the reference
configuration to characterize the topology of contact in cortical convolution
[42], that is

$L_{\rm F}=\frac{\sum_{i}\alpha_{i}}{N},$ (3)
$\alpha_{i}=\frac{\sum_{j}D_{ij}}{M_{i}},$ (4)

where $D_{ij}$ is the distance between the nodes $i$ and $j$ in the mesh grid,
$M_{i}$ is the number of contacts for the node $i$, $j$ presents all nodes in
contact with $i$, $\alpha_{i}$ represents the average contact distance of the
node $i$, and $N$ is the number of all nodes that are in contact with others
(Fig. 2b).

The combined set of parameters (GI, $\kappa$, $L_{\rm F}$) is used to
characterize the brain morphology. Unsupervised clustering of simulated brain
morphologies in the space of GI and $\kappa$ is summarized in Fig. 2d. Four
distinct types are identified and labeled as ‘convolution-free’, ‘small
folds’, ‘thick convolutions’, and ‘normal convolutions’, which aligns well
with the experimental evidence of lissencephaly, polymicrogyria, pachygyria,
and normal brain, respectively (Fig. 2d) [4]. The labels are then fed to
supervised learning using PointNet. The confusion matrix of the well-trained
classifier which is summarized in Fig. 2e,f, Supplementary Fig. 3 shows that
the accuracy of morphological identification exceeds $95\%$.

Interestingly, we find that the complexity of pattern development can be
coarse-grained according to low-dimensional features such as the network of
grooves and ridges in the brain morphology [43]. Compared to the original
point-cloud representation, the reduced models constructed from the boundaries
between grooves and ridges can be used to develop evolutionary dynamics models
to understand the underlying biophysical principles. Local patterns such as
the nodes are identified in the groove-and-ridge network (Supplementary Fig.
1b-d), which are nucleated, grow, and merge, showing distinct bifurcation
behaviors.

### Pattern development

To efficiently represent the pattern development, we adopt the PointNet
autoencoder by considering rotation and permutation invariance. The learned
representations are then fed to the LSTM network to include the historical
dependency in the latent space (Fig. 1c). The predicted morphologies at
specific stages of development are shown in (Supplementary Fig. 2e,f) for two
representative samples with different cortical thicknesses. The model
predictions are evaluated by the Chamfer distance, which measures the
discrepancy between predicted morphologies at different values of observation
time $t_{\rm obs}$ and the ground truth [44] (Fig. 2h, Supplementary Fig. 2b,
c). As $t_{\rm obs}$ increases, more information is extracted and the model
corrects earlier predictions, e.g., from cortical convolutions to smooth
appearance which forecasts the risk of lissencephaly (Supplementary Fig.
2b,e). The models also modify the local patterns continuously, demonstrating
the capability to resolve the fine morphological features during the evolution
(Fig. 2g,h, Supplementary Fig. 2f).

Pattern development from smooth to highly curled surfaces involving buckling
of the thin shells, which is related to the malformation of the brain
(Supplementary Fig. 1b) resulting from migration and growth of neurons as
simulated by FEA using the TG model. Identifying the bifurcation behaviors in
the morphological changes (e.g., the cortical convolutions) is of vital
importance for prognosis purposes (Fig. 2i, Supplementary Fig. 2d). Our
results show that bifurcation behaviors are identified from the evolution of
the mean value of $\lvert\kappa\rvert$ (Supplementary Fig. 2a). Their features
are further analyzed through the descriptors such as GI and local $L_{\rm F}$,
as well as the DEVs, which characterize the occurrence and nature of
bifurcation (Fig. 2a-c, Supplementary Fig. 4,5, see Methods for details) [45].
Primary and secondary bifurcation are identified in Supplementary Fig. 5. It
should be noted that the mapping between the simulation time of brain
development and the gestational age is non-linear [16]. The time series of the
global indicator, GI, and DEVs analysis indicate a primary fold bifurcation at
$t=0.725$ (Fig. 2a, Supplementary Fig. 5a,b, see Methods for details). On the
other hand, the time series of the local indicator, $L_{\rm F}$, and DEVs
analysis indicate a secondary fold bifurcation at $t=0.937$ (Fig. 2b,
Supplementary Fig. 5c,d). Impressively, we find that our framework forecasts
the bifurcation behaviors at $t=0.428$ through the Chamfer distance between
the prediction and the ground truth (Fig. 2h, Supplementary Fig. 2b,c).

Moreover, similar to the coarse-grained representation of morphologies in the
configurational space, the complexity of developmental patterns in the time
domain can be addressed by dimensionality reduction according to the
occurrence of bifurcation. Determining the nature of branching could advance
our understanding of the physics that governs the spatiotemporal evolution of
complex systems (e.g., for brain development).

### Diagnosis of brain malformations

The well-trained models (recognizer and predictor) based on the digital
libraries of the brain can be applied to clinical diagnosis of malformations
and abnormal development (Fig. 1c, Supplementary Fig. 3a,b, Supplementary Fig.
6a). The point clouds reconstructed from MRI can be fed to the models to
assess the current state and predict its future evolution (Fig. 1c,
Supplementary Fig. 6a). Once transition into a critical branch in the
bifurcation diagram is predicted (such as normal brain to lissencephaly,
polymicrogyria, or pachygyria), medical intervention may need to be imposed in
conjunction with diagnosis by medical experts, forecasting the event and type
of bifurcation is thus highly valuable for the diagnosis purposes (Fig. 4a,c).

### Instability-tolerant design for aerospace structures

Predicting post-buckling behaviors allows to develop instability-tolerant
design of aerospace structures (Fig. 3, Supplementary Fig. 6b). Once the load
increases beyond the buckling limit, plates and cylindrical shells lose their
stability through the primary bifurcation, and weak wrinkles form on their
surfaces (Fig. 3a). As the load continues to increase, a secondary bifurcation
takes place before the collapse. More significant and irregular crumples form
and the structures can no longer withstand the loads (Fig. 3b, Supplementary
Fig. 6b). In conservative design, limit and ultimate loads are set to be lower
than the first buckling load to avoid structural failure [7]. In practice, one
can alleviate the design constraint by permitting post-buckling, which reduces
the weight and prolong the lifespan of structural components. Based on
accurate and efficient simulations, the European project Improved POstbuckling
SImulation for Design of Fibre COmposite Stiffened Fuselage Structures
(POSICOSS) lift the limit and ultimate loads above the first buckling load for
the fuselage structures, which were validated by comprehensive experimental
databases and derived design guidelines [26]. The follow-up project Improved
MATerial Exploitation at Safe Design of COmposite Airframe Structures by
Accurate Simulation of COllapse (COCOMAT) further accounts for the collapse of
structures by considering material degradation [27]. However, these strategies
are enforced in the design stage. An instability-tolerant design proposed in
this work can allow for bifurcations with weak effects on the load-bearing
capacity (Supplementary Fig. 6b), and the tolerance of subcritical wrinkles in
service reduces the time and cost of maintenance (Fig. 3a).

## Discussion

### Physics of the model

In addition to the morphological evolution, internal stress also arises in the
structures as driven by mechanical instability. A recent work studying 2D
macromolecular membranes shows that including the lattice strain and
topological contact in the machine-learning process significantly improves the
performance of morphological classification and identification [42]. In
contrast, the outstanding performance ($>$ $95$%) reported here using the
geometric information only (Fig. 2e and Supplementary Fig. 3c) suggests that
there is an intrinsic relation between morphology and internal stress, while
the morphology of 2D macromolecules is sensitive to thermal fluctuation, which
is externally perturbed [46].

### The nature of bifurcations

The capability to identify bifurcation behaviors helps to construct phase
diagrams of structural stability. Morphological development in complex
structures can switch from one branch to another, which is prohibited in
traditional path-following methods (Fig. 4c) [47]. Fold bifurcation leads to a
catastrophic shift, period-doubling results in fluctuation, and Hopf
bifurcation drives the system to an oscillatory state (Fig. 4a, Supplementary
Fig. 6c). Identifying the nature of bifurcation based on dimensionality
reduction and LSTM can thus guide the diagnosis and structural health
management (SHM) to minimize potential losses [48]. The phase diagram of
structural stability explored in the current model is limited by the
constructed digital libraries and can be extended by using generative models
[49, 50].

### Rare external events

Pattern development under external forcing often experiences uncertain or rare
events. For example, the initial geometry and intrinsic wrinkles may vary from
sample to sample. The cortical thickness may change as a result of neuronal
cell migration. Genes or genomic mutations also modulate the molecular and
cellular processes and brain morphology [51, 52]. For aerospace engineering,
bird strikes, conflicts between airplanes, collisions between space debris and
satellites, and extreme weather conditions such as lighting strikes or wake
vortex circulations also result in uncertain factors that may lead to
unexpected morphological pattern evolution and structural failures [53]. These
rare events (Fig. 4b) can be handled in the framework by enriching the
datasets, using active learning to improve the efficiency of sampling, or
reweighting the samples in calculating the loss function [54]. However, the
relatively smooth predictions from data-driven models can underestimate the
impact of rare events [55].

## Conclusion

Pattern development due to mechanical instability in living systems and
engineering structures usually shows nonlinear dynamics and bifurcation
behaviors. A machine-learning framework is proposed here to resolve
progressive morphogenesis. The performance of our model is demonstrated
through two representative problems of brain growth and post-buckling of
aerospace thin-walled structures. The validated approach could assist in
clinical diagnosis and prognosis and prevent structural failure in
instability-tolerant design based on pre-computed digital libraries.

## Methods

### Modeling morphological development

Brain development is regulated by genetic, molecular, cellular, and mechanical
factors across multiple spatiotemporal scales [8, 9], and the differential
tangential growth hypothesis is commonly used [15, 8, 9]. Finite element
analysis (FEA) can model morphological evolution during brain growth at the
continuum level [15, 28, 13, 31, 16]. In the TG model, the tangential growth
of the outer gray matter is faster than the inner white matter [15].
Compression resulting from the mismatch in deformation may then leads to
mechanical instability of the brain surface, forming characteristic sulci and
gyri structures [28, 15, 30, 13, 16, 31, 32].

In continuum modeling, the reference configuration can be mapped to the
current one through the deformation gradient tensor as

$\textbf{F}=\textbf{F}^{\rm e}\cdot\textbf{G},$ (5)

where $\textbf{F}^{\rm e}$ is the elastic deformation gradient and G is the
growth term. In the TG model, the growth tensor G is

$\textbf{G}=g\textbf{I}+(1-g)\hat{\textbf{n}}\otimes\hat{\textbf{n}},$ (6)

where $\hat{\textbf{n}}$ is the surface normal of the reference configuration,
I is the unit tensor, and

$g=1+\frac{\alpha_{t}}{1+e^{10(\frac{y}{T}-1)}},$ (7)

is the growth coefficient, where $\alpha_{t}$ controls the magnitude of local
cortical expansion. There is a smooth transition from the surface of the gray
matter layer to the white matter layer with a gradually decreasing growth
coefficient. $y$ is the distance to the surface, and $T$ is the thickness of
the cortex. The brain is modeled as a nonlinear neo-Hookean hyperelastic
material, where the strain energy density is

$W=\frac{\mu}{2}[\rm{Tr}(\textbf{F}^{\rm e}\textbf{F}^{\rm
eT})\emph{J}^{-2/3}-3]+\frac{\emph{K}}{2}(\emph{J}-1)^{2},$ (8)

where $\mu$ is the shear modulus, $J$ is the determinant of Jacobian matrix,
$K$ is the bulk modulus. The interaction between surfaces is modeled with an
energy penalty via vertex-triangle contact, which prevents the nodes from
penetrating the faces of elements [56]. The explicit solver is used to
minimize the total (elastic and contact) energy of the quasi-static system.
The time step $\Delta t=0.05a\sqrt{\rho/\emph{K}}$ is set to ensure the
convergence, where $a$ is mesh size and $\rho$ is mass density [57].

### Machine learning

The morphological data obtained from FEA are stored in point clouds, which
possess high-dimensional characteristics and redundant information.
Experimentally, point-cloud data can be collected from 3D MRI images [58].
Feature extraction and dimensionality reduction are carried out to improve the
performance of downstream tasks (Fig. 1c, Supplementary Fig. 7). PointNet is
used to extract key features and train models for morphological phase
classification and recognition [59]. The K-means algorithm is used for
clustering, and the labels are fed into PointNet (Fig. 1c). The historical
dependency of data is addressed by using an autoencoder design based on LSTM
(Fig. 1c, Supplementary Fig. 8) [20, 60].

### Bifurcation Characterization

DEVs analysis is performed to characterize the bifurcation behaviors based on
Refs. [45, 61, 62], which makes use of principal eigenvalues (PEs) of the
Jacobian matrix in the reconstructed state space of a univariate time series
[45]. A dynamical system approaches the bifurcation point as the Euclidean
norm of PEs approaches $1$ (Fig. 4a, Supplementary Fig. 6c) [45, 63].
According to the characteristics of PEs in the complex plane, the type of
bifurcation can be determined (Fig. 4a, Supplementary Fig. 6c) [45, 63].
Specifically, a fold bifurcation has a real part of $1$ and an imaginary part
of $0$. The period-doubling bifurcation can be recognized by real and
imaginary parts of $-1$ and $0$, respectively. The Hopf bifurcation has a non-
zero imaginary part and the Euclidean norm of PEs approaches $1$ [45, 63].

Figure 1:  The machine-learning framework for pattern development. a,
Morphogenesis, digital models, and pattern development. b, Disease diagnosis
and instability-tolerant design in the brain growth and aerospace structural
design. c, The neural-network framework to address the spatiotemporal
complexities of morphogenesis.

Figure 2:  Identification and prediction of morphological development. a-c,
Sequences of structural features involving global feature gyrification index
(GI) (a), local feature localization factor ($L_{\rm F}$) (b), and the
absolute values of dynamical eigenvalues (DEVs) (c). d, Unsupervised
clustering of morphologies in the space spanned by GI and curvature. e, The
confusion matrix of recognized brain morphologies. f, The representative
morphological samples and corresponding probabilities predicted by the models,
in which a high probability value indicates the ability of models to identify
the morphology of the simulated brain. g, Predicted map of brain growth. h,
The Chamfer distance calculated between predicted final morphology and the
ground truth as $t_{\rm obs}$ increases. i, The branching behaviors can be
accurately characterized during brain growth.

Figure 3:  Instability-tolerant design for aerospace structures. a, Real-time
predictions of pattern evolution for aerospace structures. b, Instability-
tolerant design enlarges the permissible regions in design in comparison with
traditional design.

Figure 4:  Bifurcation behaviors and rare events. a, The bifurcation nature of
pattern development for brain growth and aerospace structural design is
identified as fold bifurcation by the analysis of DEVs. b, Rare events
generate complex morphological patterns and structural failures. c, The
bifurcation diagram covered by the constructed digital libraries can be fully
explored, in stark contrast with the single-path nature in path-following
methods [47].

## Data availability

All data needed to evaluate the conclusions are present in the paper and/or
the Supplementary Materials. Additional data related to this paper may be
requested from the authors.

## Code availability

The code used for this study is available at
https://github.com/zhaoyj21/instability_tolerance and the Zenodo repository
(https://zenodo.org/badge/latestdoi/657911680).

## References

  *   * Chen and Gu [2023] Chen, C.-T., Gu, G.X.: Physics-informed deep-learning for elasticity: Forward, inverse, and mixed problems. Adv. Sci. 10, 2300439 (2023)
  * Liu et al. [2020] Liu, F., Jiang, X., Wang, X., Wang, L.: Machine learning-based design and optimization of curved beams for multistable structures and metamaterials. Extreme Mech. Lett. 41, 101002 (2020)
  * Yin et al. [2008] Yin, J., Cao, Z., Li, C., Sheinman, I., Chen, X.: Stress-driven buckling patterns in spheroidal core/shell structures. Proc. Natl. Acad. Sci. USA 105(49), 19132–19135 (2008)
  * Severino et al. [2020] Severino, M., Geraldo, A.F., Utz, N., Tortora, D., Pogledic, I., Klonowski, W., Triulzi, F., Arrigoni, F., Mankad, K., Leventer, R.J., et al.: Definitions and classification of malformations of cortical development: practical guidelines. Brain 143(10), 2874–2894 (2020)
  * Besson et al. [2008] Besson, P., Andermann, F., Dubeau, F., Bernasconi, A.: Small focal cortical dysplasia lesions are located at the bottom of a deep sulcus. Brain 131(12), 3246–3255 (2008)
  * Borne et al. [2021] Borne, L., Rivière, D., Cachia, A., Roca, P., Mellerio, C., Oppenheim, C., Mangin, J.-F.: Automatic recognition of specific local cortical folding patterns. NeuroImage 238, 118208 (2021)
  * Degenhardt et al. [2014] Degenhardt, R., Castro, S.G., Arbelo, M.A., Zimmerman, R., Khakimova, R., Kling, A.: Future structural stability design for composite space and airframe structures. Thin-Walled Struct. 81, 29–38 (2014)
  * Klingler et al. [2021] Klingler, E., Francis, F., Jabaudon, D., Cappello, S.: Mapping the molecular and cellular complexity of cortical malformations. Science 371(6527), 4517 (2021)
  * Llinares-Benadero and Borrell [2019] Llinares-Benadero, C., Borrell, V.: Deconstructing cortical folding: Genetic, cellular and mechanical determinants. Nat. Rev. Neurosci. 20(3), 161–176 (2019)
  * Zimmermann and Wang [2020] Zimmermann, N., Wang, P.H.: A review of failure modes and fracture analysis of aircraft composite materials. Eng. Fail. Anal. 115, 104692 (2020)
  * Bristow and Irving [2007] Bristow, J.W., Irving, P.: Safety factors in civil aircraft design requirements. Eng. Fail. Anal. 14(3), 459–470 (2007)
  * Volkow et al. [1997] Volkow, N.D., Rosen, B., Farde, L.: Imaging the living human brain: Magnetic resonance imaging and positron emission tomography. Proc. Natl. Acad. Sci. USA 94(7), 2787–2788 (1997)
  * Darayi et al. [2021] Darayi, M., Hoffman, M.E., Sayut, J., Wang, S., Demirci, N., Consolini, J., Holland, M.A.: Computational models of cortical folding: A review of common approaches. J. Biomech., 110851 (2021)
  * Alenyà et al. [2022] Alenyà, M., Wang, X., Lefèvre, J., Auzias, G., Fouquet, B., Eixarch, E., Rousseau, F., Camara, O.: Computational pipeline for the generation and validation of patient-specific mechanical models of brain development. Brain Multiphys. 3, 100045 (2022)
  * Tallinen et al. [2016] Tallinen, T., Chung, J.Y., Rousseau, F., Girard, N., Lefèvre, J., Mahadevan, L.: On the growth and form of cortical convolutions. Nat. Phys. 12(6), 588–593 (2016)
  * Wang et al. [2021] Wang, X., Lefèvre, J., Bohi, A., Harrach, M.A., Dinomais, M., Rousseau, F.: The influence of biophysical parameters in a biomechanical model of cortical folding patterns. Sci. Rep. 11, 7686 (2021)
  * Hohlfeld and Mahadevan [2011] Hohlfeld, E., Mahadevan, L.: Unfolding the sulcus. Phys. Rev. Lett. 106(10), 105702 (2011)
  * Holland et al. [2018] Holland, M., Budday, S., Goriely, A., Kuhl, E.: Symmetry breaking in wrinkling patterns: Gyri are universally thicker than sulci. Phys. Rev. Lett. 121(22), 228002 (2018)
  * Gandikota and Schwarz [2021] Gandikota, M., Schwarz, J.: Buckling without bending morphogenesis: Nonlinearities, spatial confinement, and branching hierarchies. New J. Phys. 23(6), 063060 (2021)
  * Hochreiter and Schmidhuber [1997] Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997)
  * Hesthaven and Ubbiali [2018] Hesthaven, J.S., Ubbiali, S.: Non-intrusive reduced order modeling of nonlinear problems using neural networks. J. Comput. Phys. 363, 55–78 (2018)
  * Pichi et al. [2021] Pichi, F., Ballarin, F., Rozza, G., Hesthaven, J.S.: Artificial neural network for bifurcating phenomena modelled by nonlinear parametrized pdes. PAMM 20(S1), 202000350 (2021)
  * Quinn et al. [2009] Quinn, D., Murphy, A., McEwan, W., Lemaitre, F.: Stiffened panel stability behaviour and performance gains with plate prismatic sub-stiffening. Thin-Walled Struct. 47(12), 1457–1468 (2009)
  * Franzoni et al. [2019] Franzoni, F., Odermann, F., Wilckens, D., Skuķis, E., Kalniņš, K., Arbelo, M.A., Degenhardt, R.: Assessing the axial buckling load of a pressurized orthotropic cylindrical shell through vibration correlation technique. Thin-Walled Struct. 137, 353–366 (2019)
  * Degenhardt et al. [2008] Degenhardt, R., Kling, A., Rohwer, K., Orifici, A., Thomson, R.: Design and analysis of stiffened composite panels including post-buckling and collapse. Comput. Struct. 86(9), 919–929 (2008)
  * Zimmermann and Rolfes [2006] Zimmermann, R., Rolfes, R.: POSICOSS: Improved postbuckling simulation for design of fibre composite stiffened fuselage structures. Compos. Struct. 73(2), 171–174 (2006)
  * Degenhardt et al. [2006] Degenhardt, R., Rolfes, R., Zimmermann, R., Rohwer, K.: COCOMAT: Improved material exploitation of composite airframe structures by accurate simulation of postbuckling and collapse. Compos. Struct. 73(2), 175–178 (2006)
  * Tallinen et al. [2014] Tallinen, T., Chung, J.Y., Biggins, J.S., Mahadevan, L.: Gyrification from constrained cortical expansion. Proc. Natl. Acad. Sci. USA 111(35), 12667–12672 (2014)
  * Xu et al. [2022] Xu, F., Huang, Y., Zhao, S., Feng, X.-Q.: Chiral topographic instability in shrinking spheres. Nat. Comput. Sci. 2(10), 632–640 (2022)
  * Striedter et al. [2015] Striedter, G.F., Srinivasan, S., Monuki, E.S.: Cortical folding: When, where, how, and why? Annu. Rev. Neurosci. 38, 291–307 (2015)
  * Budday and Steinmann [2018] Budday, S., Steinmann, P.: On the influence of inhomogeneous stiffness and growth on mechanical instabilities in the developing brain. Int. J. Solids Struct. 132, 31–41 (2018)
  * da Costa Campos et al. [2021] Costa Campos, L., Hornung, R., Gompper, G., Elgeti, J., Caspers, S.: The role of thickness inhomogeneities in hierarchical cortical folding. NeuroImage 231, 117779 (2021)
  * Fischl and Dale [2000] Fischl, B., Dale, A.M.: Measuring the thickness of the human cerebral cortex from magnetic resonance images. Proc. Natl. Acad. Sci. USA 97(20), 11050–11055 (2000)
  * Chang et al. [2007] Chang, B.S., Duzcan, F., Kim, S., Cinbis, M., Aggarwal, A., Apse, K.A., Ozdel, O., Atmaca, M., Zencir, S., Bagci, H., et al.: The role of reln in lissencephaly and neuropsychiatric disease. Am. J. Med. Genet. B Neuropsychiatr. Genet. 144, 58–63 (2007)
  * Xu et al. [2010] Xu, G., Knutsen, A.K., Dikranian, K., Kroenke, C.D., Bayly, P.V., Taber, L.A.: Axons pull on the brain, but tension does not drive cortical folding. J. Biomech. Eng. 132(7) (2010)
  * Dervaux and Amar [2008] Dervaux, J., Amar, M.B.: Morphogenesis of growing soft tissues. Phys. Rev. Lett. 101(6), 068101 (2008)
  * Johnson [1983] Johnson, G.R.: A constitutive model and data for materials subjected to large strains, high strain rates, and high temperatures. Proc. 7th Inf. Sympo. Ballistics, 541–547 (1983)
  * Seidt and Gilat [2013] Seidt, J., Gilat, A.: Plastic deformation of 2024-T351 aluminum plate over a wide range of loading conditions. Int. J. Solids Struct. 50(10), 1781–1790 (2013)
  * Ko and Fleischer [2013] Ko, W.L., Fleischer, V.T.: Method for estimating operational loads on aerospace structures using span-wisely distributed surface strains. Technical report (2013)
  * Clouchoux et al. [2012] Clouchoux, C., Kudelski, D., Gholipour, A., Warfield, S.K., Viseur, S., Bouyssi-Kobar, M., Mari, J.-L., Evans, A.C., Du Plessis, A.J., Limperopoulos, C.: Quantitative in vivo MRI measurement of cortical development in the fetus. Brain Struct. Funct. 217, 127–139 (2012)
  * Park and Friston [2013] Park, H.-J., Friston, K.: Structural and functional brain networks: from connections to cognition. Science 342(6158), 1238411 (2013)
  * Zhao et al. [2022] Zhao, Y., Qin, J., Wang, S., Xu, Z.: Unraveling the morphological complexity of two-dimensional macromolecules. Patterns 3, 100497 (2022)
  * Boucher et al. [2009] Boucher, M., Whitesides, S., Evans, A.: Depth potential function for folding pattern representation, registration and analysis. Med. Image Anal. 13(2), 203–214 (2009)
  * Fan et al. [2017] Fan, H., Su, H., Guibas, L.J.: A point set generation network for 3D object reconstruction from a single image. In: Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., pp. 605–613 (2017)
  * Grziwotz et al. [2023] Grziwotz, F., Chang, C.-W., Dakos, V., Nes, E.H., Schwarzländer, M., Kamps, O., Heßler, M., Tokuda, I.T., Telschow, A., Hsieh, C.-H.: Anticipating the occurrence and type of critical transitions. Sci. Adv. 9(1), 4558 (2023)
  * Wang et al. [2020] Wang, Y., Wang, S., Li, P., Rajendran, S., Xu, Z., Liu, S., Guo, F., He, Y., Li, Z., Xu, Z., et al.: Conformational phase map of two-dimensional macromolecular graphene oxide in solution. Matter 3(1), 230–245 (2020)
  * Keller [1987] Keller, H.B.: Lectures on numerical methods in bifurcation problems. Appl. Math. 217, 50 (1987)
  * Bury et al. [2021] Bury, T.M., Sujith, R., Pavithran, I., Scheffer, M., Lenton, T.M., Anand, M., Bauch, C.T.: Deep learning for early warning signals of tipping points. Proc. Natl. Acad. Sci. USA 118(39), 2106140118 (2021)
  * Ni et al. [2023] Ni, B., Kaplan, D.L., Buehler, M.J.: Generative design of de novo proteins based on secondary-structure constraints using an attention-based diffusion model. Chem 9, 1828–1849 (2023)
  * Luu et al. [2023] Luu, R.K., Wysokowski, M., Buehler, M.J.: Generative discovery of de novo chemical designs using diffusion modeling and transformer deep neural networks with application to deep eutectic solvents. Appl. Phys. Lett. 122, 234103 (2023)
  * Barak et al. [2011] Barak, T., Kwan, K.Y., Louvi, A., Demirbilek, V., Saygı, S., Tüysüz, B., Choi, M., Boyacı, H., Doerschner, K., Zhu, Y., et al.: Recessive LAMC3 mutations cause malformations of occipital cortical development. Nat. Genet. 43(6), 590–594 (2011)
  * Poirier et al. [2013] Poirier, K., Lebrun, N., Broix, L., Tian, G., Saillour, Y., Boscheron, C., Parrini, E., Valence, S., Pierre, B.S., Oger, M., et al.: Mutations in TUBG1, DYNC1H1, KIF5C and KIF2A cause malformations of cortical development and microcephaly. Nat. Genet. 45(6), 639–647 (2013)
  * Morio and Balesdent [2015] Morio, J., Balesdent, M.: Estimation of Rare Event Probabilities in Complex Aerospace and Other Systems: A Practical Approach. Woodhead Publishing, Amsterdam (2015)
  * Pickering et al. [2022] Pickering, E., Guth, S., Karniadakis, G.E., Sapsis, T.P.: Discovering and forecasting extreme events via active learning in neural operators. Nat. Comput. Sci. 2(12), 823–833 (2022)
  * Bi et al. [2023] Bi, K., Xie, L., Zhang, H., Chen, X., Gu, X., Tian, Q.: Accurate medium-range global weather forecasting with 3D neural networks. Nature 619, 533–538 (2023)
  * Ericson [2004] Ericson, C.: Real-time Collision Detection. CRC Press, Florida (2004)
  * Belytschko et al. [2014] Belytschko, T., Liu, W.K., Moran, B., Elkhodary, K.: Nonlinear Finite Elements for Continua and Structures. John Wiley & Sons, New Jersey (2014)
  * Fan et al. [2017] Fan, B., Rao, Y., Liu, W., Wang, Q., Wen, H.: Region-based growing algorithm for 3D reconstruction from MRI images. In: Int. Conf. Image Vis. Comput., pp. 521–525 (2017)
  * Qi et al. [2017] Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning on point sets for 3D classification and segmentation. In: Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., pp. 652–660 (2017)
  * Achlioptas et al. [2018] Achlioptas, P., Diamanti, O., Mitliagkas, I., Guibas, L.: Learning representations and generative models for 3D point clouds. In: Int. Conf. Mach. Learn., pp. 40–49 (2018)
  * Ushio et al. [2018] Ushio, M., Hsieh, C.-h., Masuda, R., Deyle, E.R., Ye, H., Chang, C.-W., Sugihara, G., Kondoh, M.: Fluctuating interaction network and time-varying stability of a natural fish community. Nature 554(7692), 360–363 (2018)
  * Cenci and Saavedra [2019] Cenci, S., Saavedra, S.: Non-parametric estimation of the structural stability of non-equilibrium community dynamics. Nat. Ecol. Evol. 3(6), 912–918 (2019)
  * Kuznetsov et al. [1998] Kuznetsov, Y.A., Kuznetsov, I.A., Kuznetsov, Y.: Elements of Applied Bifurcation Theory vol. 112. Springer, New York (1998)

## Acknowledgments

This study was supported by the National Natural Science Foundation of China
through grants 11825203, 11832010, 11921002, and 52090032. The computation was
performed on the Explorer 100 cluster system of the Tsinghua National
Laboratory for Information Science and Technology.

## Author contributions

Z.X. conceived and supervised the research. Y.Z. performed the simulations and
analysis. Both authors wrote the manuscript.

## Competing interests

The authors declare that they have no competing financial interests.

## Additional information

Supplementary information The online version contains supplementary material
available.
Correspondence and requests for materials should be addressed to Z.X.
(xuzp@tsinghua.edu.cn).