# Crystal Twins: Self-supervised Learning for Crystalline Material Property
Prediction

Rishikesh Magar  Yuyang Wang  Amir Barati Farimani  [<EMAIL_ADDRESS>
###### Abstract

Machine learning (ML) models have been widely successful in the prediction of
material properties. However, large labeled datasets required for training
accurate ML models are elusive and computationally expensive to generate.
Recent advances in Self-Supervised Learning (SSL) frameworks capable of
training ML models on unlabeled data have mitigated this problem and
demonstrated superior performance in computer vision and natural language
processing tasks. Drawing inspiration from the developments in SSL, we
introduce Crystal Twins (CT): an SSL method for crystalline materials property
prediction. Using a large unlabeled dataset, we pre-train a Graph Neural
Network (GNN) by applying the redundancy reduction principle to the graph
latent embeddings of augmented instances obtained from the same crystalline
system. By sharing the pre-trained weights when fine-tuning the GNN for
regression tasks, we significantly improve the performance for 7 challenging
material property prediction benchmarks.

###### keywords:

American Chemical Society, LaTeX

Carnegie Mellon University] Department of Mechanical Engineering, Carnegie
Mellon University, Pittsburgh, PA 15213 IR,NMR,UV

##  1 Introduction

Machine Learning (ML) based predictive models have made rapid strides in
computational chemistry due to their efficiency and performance. Characterized
by their computational efficiency and accuracy, these methods are capable of
faster high-throughput screening compared to classical physics models 1, 2.
This capability has roots in both novel learning algorithms and improved
hardware. Even though ML models can offer faster predictions, the accuracy of
these models is highly correlated with the availability of clean labeled data.
3. In general, it is difficult to develop accurate and robust ML models
without sufficiently large labeled data 4. Moreover, the acquisition of
labeled data is expensive as it involves performing Density Functional Theory
(DFT) simulations or experiments to characterize materials 5, 6. On the other
hand, gigantic databases containing structures and compositions of materials
without labels (property) are available. These databases can not be used in
supervised learning tasks due to the lack of labels. Given the availability of
large unlabeled datasets, some interesting questions can be asked 1.) can we
develop more efficient ML models that are capable of learning the underlying
structural chemistry from unlabeled data and 2.) can these models be used to
make the supervised learning tasks more accurate?

In this work, we aim to address these questions by leveraging Self-Supervised
Learning (SSL) for material property prediction. Unlike supervised learning
which uses labels for supervision, SSL makes use of the large unlabeled data
for supervision to learn robust and generalizable representations that can be
used for various tasks. Recently, SSL frameworks such as SimCLR7 , Barlow
Twins 8 , BYOL9 , SwAV 10 , momentum contrastive learning 11 , SimSiam 12 ,
Albert 13 , and self-supervised dialogue learning 14 have been successfully
applied to computer vision and natural language processing tasks. The success
of these SSL methods has inspired many works in molecular ML, leading to the
development of highly accurate frameworks such as MolCLR 15 , dual view
molecule pre-training 16 , 3D Infomax 17 , and numerous other popular works18,
19, 20, 21, 22, 23, 24, 25.

Figure 1: Overview of the Crystal Twins (CT) framework. A.) The Crystal Twins
framework takes the structural file (CIF) as the input and then augments the
structure to create two different augmented instances. Each instance is passed
to the CGCNN graph encoder followed by a projector to generate embedding. The
pre-training objective aims to maximize the cross-correlation between the two
embeddings. B.) To create augmented instances, three augmentation techniques
are used in this work: random perturbations, atom masking, and edge masking.
C.) In the pre-training stage we trained using SSL. In the fine-tuning
stage,the pre-trained weights are shared with the encoder(CGCNN) which is
connected MLP head to predict the material property.

Most of the promising works developed for material property prediction tasks
are using graph neural networks (GNN). GNNs consider non-Euclidean topology to
construct a graph representation that can be learned and modified according to
the task 26, 27, 28. GNNs developed for material property prediction include
CGCNN29 , OGCNN30 , SchNet31 , MegNet32, and other models 33, 34, 35, 36, 37,
38, 39, 40, 41, 42, 43. Developments have also been made in tasks such as
material structure generation and prediction 44, 45, 46, 47, 48, 49 as well as
identifying new materials with specific properties50. Despite progress being
made in developing self-supervised ML architectures in the molecular ML, there
is a noticeable lack of research works implementing such techniques for the
periodic crystalline systems property prediction. In this work, we introduce
Crystal Twins (CT): an SSL framework for crystalline material property
prediction with GNNs (Figure 1). In pre-training, the CT framework does not
make use of any labeled data to learn crystalline representations, instead, it
trains ML models in a self-supervised manner. In the CT framework, we use the
CGCNN29 as the encoder to learn expressive representations of crystalline
systems (Figure 1a). The GNN encoder (i.e., CGCNN) generates representations
of two augmented instances from the same crystal and the objective of CT pre-
training is to make the cross-correlation matrix of the two embeddings as
close as possible to the identity matrix (Figure 1a). To create augmented
instances, we introduce the combination of three different augmentation
techniques: random perturbations, atom masking, and edge masking (Figure 1b).
The representations learned by the encoder are later used for downstream
material property prediction tasks in the fine-tuning stage (Figure 1c). In
the pre-training stage, the CT model learns representations without any
labeled data. Using the pre-trained weights to initialize the GNN encoder for
fine-tuning, CT demonstrates superior prediction performances on 7 challenging
datasets. We also compare the performance of the CT model with other
competitive supervised learning baselines.

##  2 Methods

In this section, we describe the components of the CT framework (Figure 1). In
general, SSL frameworks employ correlations in the input itself to learn
robust and generalizable representations from unlabeled data. 51. In the CT
framework, the goal during pre-training is to force the empirical cross-
correlation matrix created from the encoder embeddings two different
augmentations generated by the same crystal towards the identity matrix. All
the elements in the cross-correlation matrix lie between -1 and 1, with 1
representing maximum correlation. Intuitively, since the embeddings are
generated from augmentations of the same crystalline system, the cross-
correlation matrix must be close to the identity matrix. Using such an
objective during pre-training allows the graph encoder to learn robust
representations. To create the augmented instances, we use augmentation
techniques, including atom masking, edge masking, and random perturbation. The
embeddings for the augmented instances of the crystalline system are generated
via the CGCNN graph encoder. To pre-train the CGCNN in a self-supervised
manner, we use the Barlow Twins loss function. The weights of the pre-trained
self-supervised model are used to initialize the graph encoder model during
the fine-tuning stage for material property prediction.

Most of the successful deep learning approaches for crystalline material
property prediction are based on GNNs because of their ability to capture
structural geometry and chemistry. In a crystal graph ($G$), we consider the
atoms as the nodes ($V$), and interactions between them are modeled via edges
($E$). In general, GNNs aggregate information from the neighborhood of the
node to construct embeddings that are updated iteratively. The update for the
GNN can be described as in Equation 1

$\displaystyle\begin{split}\boldsymbol{h}_{v}^{(k)}&=\text{COMBINE}^{(k)}\bigg{(}\boldsymbol{h}_{v}^{(k-1)},\text{AGGREGATE}^{(k)}\Big{(}\\{\boldsymbol{h}_{u}^{(k-1)}|u\in\mathcal{N}(v)\\}\Big{)}\bigg{)},\end{split}$
(1)

where $\boldsymbol{h}_{v}^{(k)}$ is the feature of the node $v$ at the $k$-th
layer and $\boldsymbol{h}_{v}^{(0)}$ is initialized by node feature
$\boldsymbol{x}_{v}$. $\mathcal{N}(v)$ denotes the set of all the neighbors of
node $v$. $\boldsymbol{a}_{v}^{(k)}$ is the output from the aggregation
operation at the $k^{th}$ layer. The aggregation operation collects the
features of neighboring nodes and the combination operation combines the
original node feature with the aggregated features. To extract the feature of
the entire crystal system, $\boldsymbol{h}_{G}$, readout operation integrates
all the node features among the graph $G$ as given in Equation 2:

$\boldsymbol{h}_{G}=\text{READOUT}\Big{(}\\{\boldsymbol{h}_{v}^{(k)}|v\in
G\\}\Big{)}.$ (2)

The readout operations such as summation, averaging, and max pooling are most
commonly used52.

In this work, we implement the CGCNN29 architecture as the GNN encoder. We
choose CGCNN because of its competitive performance and computational
efficiency when compared to other GNN baselines. To encode crystal features
and obtain an embedding, we use mean pooling to generate a latent
representation with the dimension of 64. To expand the latent embedding and
have a better combination of encoded features, we use a projector with 2 MLP
layers that generates a final embedding of size 128. We use this embedding to
generate the cross-correlation matrix for applying the Barlow Twins loss.

In the pre-training stage, we use the Barlow Twins loss function to learn
graph representations from crystals. This loss is based on the redundancy
reduction principle proposed by neuroscientist H. Barlow 53, 54 and was
introduced to SSL by Zbontar et al.8. We use the Barlow Twins loss function in
CT because of its high performance and ease of implementation. The Barlow
Twins loss function is applied to the cross-correlation matrix created from
encoder-generated embeddings of the two different augmentations of the same
crystalline system. The Barlow Twins loss function is represented by Equation
3,

$L_{BT}\overset{\Delta}{=}\sum_{i}(1-C_{ii})^{2}+\lambda\sum_{i}\sum_{j\neq
i}C_{ij}^{2},$ (3)

where $C$ is the cross-correlation matrix of embeddings from two augmented
instances, the cross correlation matrix is given by Equation 4. The $\lambda$
used in this work is $0.0051$ same as the original paper.

$C_{ij}\overset{\Delta}{=}\frac{\sum_{b}Z_{b,i}^{A}Z_{b,j}^{B}}{\sqrt{(Z_{b,i}^{A})^{2}}{\sqrt{(Z_{b,j}^{B})^{2}}}}$
(4)

where $b$ is the index of the batch and $i,j$ index the vector dimensions of
the projector output ($Z^{A}$ and $Z^{B}$), for both the augmented instances
$A$ and $B$ from the same crystalline material.

To generate self-supervised learning representations, we need to construct
different augmentations of the crystalline system. Inspired by AugLiChem,55 we
devise three different augmentation techniques (Figure 1b), namely, random
perturbation, atom masking and edge masking. The random perturbation
augmentation perturbs all the atoms in the crystalline system by a distance of
0.05 Å. For atom masking, we mask 10% of the atoms in the crystal randomly,
similarly for edge masking we randomly mask 10% of the edge features between
two neighboring atoms. More details on atom masking and edge masking are
provided in the supplementary information (Figure S1). These augmentations are
applied to the crystalline systems and two augmented instances are generated
for pre-training.

In the pre-training stage, the embedding dimension of the CGCNN encoder is set
to 128. We use the Adam optimizer56 with a learning rate of $0.00001$ and a
batch size of $64$ and pre-train the model for $15$ epochs. The other
hyperparameters for the CGCNN model are kept the same as in the original
paper. The train/validation ratio for pre-training data is 95%/5%. For pre-
training, we combine the datasets from the Matminer database 57 and the
hypothetical Metal-Organic Framework dataset58, aggregating a total of 428,275
samples. The labels in the datasets are not used during CT pre-training. In
the fine-tuning stage, we add a randomly initialized MLP head with two fully
connected layers to generate the final property prediction. The model in the
fine-tuning stage is trained for 200 epochs. Similarly, the results for
supervised CGCNN are also reported after training the model for 200 epochs.
For fine-tuning we use the Adam optimizer with a learning rate of $0.001$ and
batch size 128.

##  3 Results

To comprehensively evaluate the performance of the CT framework, we test its
performance on seven challenging regression benchmark datasets. The properties
we evaluate include formation energy, band gap, Fermi energy, and shear
modulus for different materials. An overview of the dataset used for
benchmarking the performance of the CT framework is shown in Table 1 . A
detailed description of these datasets is available in the supplementary
information.

Dataset | # Crystals | Property | Unit
---|---|---|---
HOIP | 1333 | Band gap | eV
Lanthanides | 4166 | Formation energy | eV/atom
GVRH | 10987 | Shear Modulus | $\log_{10}VRH$
Perovskites | 18928 | Formation Energy | eV/atom
Fermi Energy | 26447 | Fermi energy | eV
Formation Energy (FE) | 26078 | Formation energy | eV/atom
Band Gap (BG) | 26709 | Band gap | eV

Table 1: Overview of the datasets used for benchmarking the performance of the
CT framework

As we pre-trained the model with CGCNN encoder, the comparison with the
supervised CGCNN model is the most direct and fair, and it offers insights
into how self-supervised learning methods can help in predicting the
crystalline material properties with a high degree of accuracy. We also
compare the performance of the CT framework with other popular supervised GNN
models, i.e., GATGNN33, GIN20, and OGCNN30. All the models used for comparison
are trained with the same hyperparameters as suggested in their publicly
available codes. The train/validation/test split for the datasets is set to
$0.6/0.2/0.2$ following previous standard benchmarking protocols. The test
Mean Absolute Errors (MAEs) for the supervised training baselines and the CT
framework are shown in Table 2. The detailed hyperparameters used for
supervised models are listed in the supplementary information (Table S2).

Dataset |  HOIP59 |  Lanthanides60 |  GVRH57, 61 |  Perovskites62 |  Fermi Energy63 |  FE63 |  BG63
---|---|---|---|---|---|---|---
# Crystals | 1333 | 4166 | 10987 | 18928 | 26447 | 26078 | 26709
GIN 20 |  0.666$\pm$0.123 |  0.197$\pm$0.038 |  0.133$\pm$0.007 |  0.380$\pm$0.008 |  0.605$\pm$0.015 |  0.109$\pm$0.007 |  0.601$\pm$0.038
CGCNN 29 |  0.170$\pm$0.013 |  0.080$\pm$0.003 |  0.092$\pm$0.001 |  0.057$\pm$0.002 | 0.400$\pm$0.003 |  0.040$\pm$0.001 |  0.369$\pm$0.003
GATGNN 33 |  0.169$\pm$0.016 |  0.097$\pm$0.004 |  0.091$\pm$0.000 |  0.071$\pm$0.002 |  0.428$\pm$0.007 |  0.050$\pm$0.004 |  0.376$\pm$0.007
OGCNN 30 |  0.164$\pm$0.013 |  0.072$\pm$0.002 |  0.089$\pm$0.001 |  0.056$\pm$0.001 |  0.446$\pm$0.018 |  0.035$\pm$0.001 |  0.353$\pm$0.008
CT | 0.153$\pm$0.003 | 0.058$\pm$0.001 | 0.087$\pm$0.001 | 0.046$\pm$0.001 | 0.399$\pm$0.004 | 0.025$\pm$0.001 | 0.328$\pm$0.002

Table 2: Mean and standard deviation of test MAE of Crystal Twins (CT) in
comparison to the supervised baselines on regression benchmarks.

It is observed that the CT model outperforms all supervised learning baselines
on all the 7 regression tasks. We would like to note that the performance
improvements (Table S1) achieved by the CT over the baseline supervised CGCNN
model is non-trivial. We observed an average improvement of 15.55% improvement
for CT over supervised-learning CGCNN. The results in Table 2 clearly
demonstrate the merit of using self-supervised learning frameworks for
periodic crystal property prediction.

To compare the effectiveness of the different augmentations techniques, we
pre-train three CT models, 1) using only random perturbation augmentations
(RP), 2) using only atom masking and edge masking augmentations (AM+EM), 3)
using all three random perturbation, atom masking, and edge masking
augmentations (RP+AM+EM). We report the MAE of the model on different fine-
tuning tasks to determine the effective augmentation techniques (Figure 2).

Figure 2: Ablation study of three augmentation techniques, random perturbation
(RP), atom masking (AM), and edge masking (EM), for CT framework. A.)
Evaluating the effect of different augmentation techniques in Band Gap and
HOIP dataset where the label is band gap. B.) Evaluating the effect of
augmentation techniques on the FE, Lanthanides, and Perovskites datasets for
which the label is formation energy. C.) Evaluating the effect of different
augmentation strategies on the Fermi energy and log10 VRH - shear modulus of
the structures prediction.

The performance of AM+EM augmentation is better than RP for perovskites, BG
and GVRH datasets , whereas RP augmentation has better performance than AM+EM
for Fermi energy, lanthanides, and HOIP datasets. For FE dataset the
performance of both RP and AM+EM augmentation techniques is the same. It must
be noted that the performance of models trained with different augmentation
techniques is almost identical, making it difficult to conclusively ascertain
which augmentation technique is better. Moreover, we also observe that the
effectiveness of the augmentation techniques is dataset dependent. We would
also like to note that the standard deviation of MAE is always lower when
using the pre-trained model with all augmentation techniques. Therefore, using
a combination of all three augmentation techniques is most effective.

To understand the CT representations, we visualize the representations from
the pre-trained and fine-tuned CT framework in comparison to the supervised
CGCNN in 2D using t-SNE64. The t-SNE representation maps the embedding based
on the similarity in the 2D space. The comparison between the representations
of the supervised CGCNN model and the CT model for the perovskites dataset is
shown in Figure 3. Each point is colored by the formation energy of
perovskites which is the label that the model is trained on in the fine-tuning
stage.

Figure 3: Visualizing the embeddings space for the Perovskites dataset using
t-SNE. Every point on the t-SNE plot is colored corresponding to the formation
energy of the crystalline system. A.) The t-SNE plot for the embedding was
generated from the supervised CGCNN model B.) The t-SNE plot for the embedding
was generated from the graph encoder of CT model after fine-tuning.

We observe that the t-SNE projection from the CT model has a better
separation, the crystalline materials with higher formation energy are
clustered at the top left of the t-SNE projection plot (Figure 3B) when
compared to the supervised CGCNN (Figure 3A). Similarly the materials with
lower formation energy are clustered at the bottom of the graph(Figure 3B).
For example, perovskites $InOsO_{3}$ and $LaReO_{3}$ with relatively lower
formation energies of -0.58 and -0.64 $eV/atom$ respectively are clustered
closely together in t-SNE projection from CT than from supervised CGCNN. This
demonstrates the generalizability of the representations learned by the CT
model when compared to supervised learning. Such representation learnt from
the CT framework can also be used to characterize and understand the large
chemical space of materials.

##  4 Conclusion

In this work, we develop Crystal Twins (CT), an SSL framework for crystalline
material property prediction. The CT model achieves a superior performance
compared to other competitive supervised learning baselines. The CT framework
demonstrates high generalizability and robustness by learning representations
that can be used to predict a variety of properties like formation energy,
band gap, and Fermi energy of different crystalline materials. The CT
framework has been trained on significantly less data when compared SSL models
in other domains like molecular machine learning, computer vision and natural
language processing. In general, SSL models are known to demonstrate a better
performance with larger unlabeled data as it allows them to learn more
generalizable representations. We expect the CT model to demonstrate a
superior performance with larger training data when compared to our current
results. The representations learned by the CT model are of great promise and
can open up avenues for exciting research in understanding the chemical space
and designing materials with desired properties.

The work has been supported by the start-up fund provided by Department of
Mechanical Engineering at Carnegie Mellon University. The authors would like
to thank Prakarsh Yadav and Alison Bartsch for their comments on the
manuscript.

## Data and Code Availability

All data used in this work is publicly available. Please contact the
corresponding author for more details about code and data.

## References

  * Schmidt et al. 2019 Schmidt, J.; Marques, M. R.; Botti, S.; Marques, M. A. Recent advances and applications of machine learning in solid-state materials science. _npj Computational Materials_ 2019, _5_ , 1–36
  * Keith et al. 2021 Keith, J. A.; Vassilev-Galindo, V.; Cheng, B.; Chmiela, S.; Gastegger, M.; Müller, K.-R.; Tkatchenko, A. Combining machine learning and computational chemistry for predictive insights into chemical systems. _arXiv preprint arXiv:2102.06321_ 2021,
  * Najafabadi et al. 2015 Najafabadi, M. M.; Villanustre, F.; Khoshgoftaar, T. M.; Seliya, N.; Wald, R.; Muharemagic, E. Deep learning applications and challenges in big data analytics. _Journal of big data_ 2015, _2_ , 1–21
  * Bengio et al. 2021 Bengio, Y.; Lecun, Y.; Hinton, G. Deep learning for AI. _Communications of the ACM_ 2021, _64_ , 58–65
  * Schleder et al. 2019 Schleder, G. R.; Padilha, A. C.; Acosta, C. M.; Costa, M.; Fazzio, A. From DFT to machine learning: recent approaches to materials science–a review. _Journal of Physics: Materials_ 2019, _2_ , 032001
  * Chen et al. 2020 Chen, A.; Zhang, X.; Zhou, Z. Machine learning: accelerating materials development for energy storage and conversion. _InfoMat_ 2020, _2_ , 553–576
  * Chen et al. 2020 Chen, T.; Kornblith, S.; Norouzi, M.; Hinton, G. A simple framework for contrastive learning of visual representations. International conference on machine learning. 2020; pp 1597–1607
  * Zbontar et al. 2021 Zbontar, J.; Jing, L.; Misra, I.; LeCun, Y.; Deny, S. Barlow twins: Self-supervised learning via redundancy reduction. International Conference on Machine Learning. 2021; pp 12310–12320
  * Grill et al. 2020 Grill, J.-B.; Strub, F.; Altché, F.; Tallec, C.; Richemond, P.; Buchatskaya, E.; Doersch, C.; Avila Pires, B.; Guo, Z.; Gheshlaghi Azar, M., et al. Bootstrap your own latent-a new approach to self-supervised learning. _Advances in Neural Information Processing Systems_ 2020, _33_ , 21271–21284
  * Caron et al. 2020 Caron, M.; Misra, I.; Mairal, J.; Goyal, P.; Bojanowski, P.; Joulin, A. Unsupervised learning of visual features by contrasting cluster assignments. _Advances in Neural Information Processing Systems_ 2020, _33_ , 9912–9924
  * He et al. 2020 He, K.; Fan, H.; Wu, Y.; Xie, S.; Girshick, R. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020; pp 9729–9738
  * Chen and He 2021 Chen, X.; He, K. Exploring simple siamese representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021; pp 15750–15758
  * Lan et al. 2019 Lan, Z.; Chen, M.; Goodman, S.; Gimpel, K.; Sharma, P.; Soricut, R. Albert: A lite bert for self-supervised learning of language representations. _arXiv preprint arXiv:1909.11942_ 2019,
  * Wu et al. 2019 Wu, J.; Wang, X.; Wang, W. Y. Self-supervised dialogue learning. _arXiv preprint arXiv:1907.00448_ 2019,
  * Wang et al. 2022 Wang, Y.; Wang, J.; Cao, Z.; Barati Farimani, A. Molecular contrastive learning of representations via graph neural networks. _Nature Machine Intelligence_ 2022, 1–9
  * Zhu et al. 2021 Zhu, J.; Xia, Y.; Qin, T.; Zhou, W.; Li, H.; Liu, T.-Y. Dual-view Molecule Pre-training. 2021
  * Stärk et al. 2021 Stärk, H.; Beaini, D.; Corso, G.; Tossou, P.; Dallago, C.; Günnemann, S.; Liò, P. 3D Infomax improves GNNs for Molecular Property Prediction. 2021
  * Liu et al. 2019 Liu, S.; Demirel, M. F.; Liang, Y. N-gram graph: Simple unsupervised representation for graphs, with applications to molecules. _Advances in neural information processing systems_ 2019, _32_
  * Rong et al. 2020 Rong, Y.; Bian, Y.; Xu, T.; Xie, W.; Wei, Y.; Huang, W.; Huang, J. Self-supervised graph transformer on large-scale molecular data. _Advances in Neural Information Processing Systems_ 2020, _33_ , 12559–12571
  * Hu et al. 2019 Hu, W.; Liu, B.; Gomes, J.; Zitnik, M.; Liang, P.; Pande, V.; Leskovec, J. Strategies for pre-training graph neural networks. _arXiv preprint arXiv:1905.12265_ 2019,
  * Li et al. 2021 Li, P.; Wang, J.; Qiao, Y.; Chen, H.; Yu, Y.; Yao, X.; Gao, P.; Xie, G.; Song, S. An effective self-supervised framework for learning expressive molecular global representations to drug discovery. _Briefings in Bioinformatics_ 2021, _22_ , bbab109
  * Chithrananda et al. 2020 Chithrananda, S.; Grand, G.; Ramsundar, B. Chemberta: Large-scale self-supervised pretraining for molecular property prediction. _arXiv preprint arXiv:2010.09885_ 2020,
  * Rong et al. 2020 Rong, Y.; Bian, Y.; Xu, T.; Xie, W.; Wei, Y.; Huang, W.; Huang, J. Grover: Self-supervised message passing transformer on large-scale molecular data. _Advances in Neural Information Processing Systems_ 2020,
  * Zhang et al. 2021 Zhang, Z.; Liu, Q.; Wang, H.; Lu, C.; Lee, C.-K. Motif-based Graph Self-Supervised Learning for Molecular Property Prediction. _Advances in Neural Information Processing Systems_ 2021, _34_
  * Wang et al. 2022 Wang, Y.; Magar, R.; Liang, C.; Farimani, A. B. Improving Molecular Contrastive Learning via Faulty Negative Mitigation and Decomposed Fragment Contrast. _arXiv preprint arXiv:2202.09346_ 2022,
  * LeCun et al. 2015 LeCun, Y.; Bengio, Y.; Hinton, G. Deep learning. _nature_ 2015, _521_ , 436–444
  * Wu et al. 2020 Wu, Z.; Pan, S.; Chen, F.; Long, G.; Zhang, C.; Philip, S. Y. A comprehensive survey on graph neural networks. _IEEE transactions on neural networks and learning systems_ 2020, _32_ , 4–24
  * Kipf and Welling 2017 Kipf, T. N.; Welling, M. Semi-Supervised Classification with Graph Convolutional Networks. 2017
  * Xie and Grossman 2018 Xie, T.; Grossman, J. C. Crystal graph convolutional neural networks for an accurate and interpretable prediction of material properties. _Physical review letters_ 2018, _120_ , 145301
  * Karamad et al. 2020 Karamad, M.; Magar, R.; Shi, Y.; Siahrostami, S.; Gates, I. D.; Farimani, A. B. Orbital graph convolutional neural network for material property prediction. _Physical Review Materials_ 2020, _4_ , 093801
  * Schütt et al. 2018 Schütt, K. T.; Sauceda, H. E.; Kindermans, P.-J.; Tkatchenko, A.; Müller, K.-R. Schnet–a deep learning architecture for molecules and materials. _The Journal of Chemical Physics_ 2018, _148_ , 241722
  * Chen et al. 2019 Chen, C.; Ye, W.; Zuo, Y.; Zheng, C.; Ong, S. P. Graph networks as a universal machine learning framework for molecules and crystals. _Chemistry of Materials_ 2019, _31_ , 3564–3572
  * Louis et al. 2020 Louis, S.-Y.; Zhao, Y.; Nasiri, A.; Wang, X.; Song, Y.; Liu, F.; Hu, J. Graph convolutional neural networks with global attention for improved materials property prediction. _Physical Chemistry Chemical Physics_ 2020, _22_ , 18141–18148
  * Klicpera et al. 2020 Klicpera, J.; Groß, J.; Günnemann, S. Directional message passing for molecular graphs. _arXiv preprint arXiv:2003.03123_ 2020,
  * Klicpera et al. 2020 Klicpera, J.; Giri, S.; Margraf, J. T.; Günnemann, S. Fast and uncertainty-aware directional message passing for non-equilibrium molecules. _arXiv preprint arXiv:2011.14115_ 2020,
  * Palizhati et al. 2019 Palizhati, A.; Zhong, W.; Tran, K.; Back, S.; Ulissi, Z. W. Toward predicting intermetallics surface properties with high-throughput DFT and convolutional neural networks. _Journal of chemical information and modeling_ 2019, _59_ , 4742–4749
  * Back et al. 2019 Back, S.; Yoon, J.; Tian, N.; Zhong, W.; Tran, K.; Ulissi, Z. W. Convolutional neural network of atomic surface structures to predict binding energies for high-throughput screening of catalysts. _The journal of physical chemistry letters_ 2019, _10_ , 4401–4408
  * Gilmer et al. 2017 Gilmer, J.; Schoenholz, S. S.; Riley, P. F.; Vinyals, O.; Dahl, G. E. Neural message passing for quantum chemistry. International conference on machine learning. 2017; pp 1263–1272
  * Unke and Meuwly 2019 Unke, O. T.; Meuwly, M. PhysNet: A neural network for predicting energies, forces, dipole moments, and partial charges. _Journal of chemical theory and computation_ 2019, _15_ , 3678–3693
  * Gu et al. 2020 Gu, G. H.; Noh, J.; Kim, S.; Back, S.; Ulissi, Z.; Jung, Y. Practical deep-learning representation for fast heterogeneous catalyst screening. _The Journal of Physical Chemistry Letters_ 2020, _11_ , 3185–3191
  * Jha et al. 2018 Jha, D.; Ward, L.; Paul, A.; Liao, W.-k.; Choudhary, A.; Wolverton, C.; Agrawal, A. Elemnet: Deep learning the chemistry of materials from only elemental composition. _Scientific reports_ 2018, _8_ , 1–13
  * Dunn et al. 2020 Dunn, A.; Wang, Q.; Ganose, A.; Dopp, D.; Jain, A. Benchmarking materials property prediction methods: the Matbench test set and Automatminer reference algorithm. _npj Computational Materials_ 2020, _6_ , 1–10
  * Choudhary and DeCost 2021 Choudhary, K.; DeCost, B. Atomistic Line Graph Neural Network for improved materials property predictions. _npj Computational Materials_ 2021, _7_ , 1–8
  * Moosavi et al. 2020 Moosavi, S. M.; Jablonka, K. M.; Smit, B. The role of machine learning in the understanding and design of materials. _Journal of the American Chemical Society_ 2020, _142_ , 20273–20287
  * Ryan et al. 2018 Ryan, K.; Lengyel, J.; Shatruk, M. Crystal structure prediction via deep learning. _Journal of the American Chemical Society_ 2018, _140_ , 10158–10168
  * Liang et al. 2020 Liang, H.; Stanev, V.; Kusne, A. G.; Takeuchi, I. CRYSPNet: Crystal structure predictions via neural networks. _Physical Review Materials_ 2020, _4_ , 123802
  * Long et al. 2021 Long, T.; Fortunato, N. M.; Opahle, I.; Zhang, Y.; Samathrakis, I.; Shen, C.; Gutfleisch, O.; Zhang, H. Constrained crystals deep convolutional generative adversarial network for the inverse design of crystal structures. _npj Computational Materials_ 2021, _7_ , 1–7
  * Kim et al. 2020 Kim, S.; Noh, J.; Gu, G. H.; Aspuru-Guzik, A.; Jung, Y. Generative adversarial networks for crystal structure prediction. _ACS central science_ 2020, _6_ , 1412–1420
  * Xie et al. 2021 Xie, T.; Fu, X.; Ganea, O.-E.; Barzilay, R.; Jaakkola, T. Crystal Diffusion Variational Autoencoder for Periodic Material Generation. _arXiv preprint arXiv:2110.06197_ 2021,
  * Yao et al. 2021 Yao, Z.; Sánchez-Lengeling, B.; Bobbitt, N. S.; Bucior, B. J.; Kumar, S. G. H.; Collins, S. P.; Burns, T.; Woo, T. K.; Farha, O. K.; Snurr, R. Q., et al. Inverse design of nanoporous crystalline reticular materials with deep generative models. _Nature Machine Intelligence_ 2021, _3_ , 76–86
  * Hadsell et al. 2006 Hadsell, R.; Chopra, S.; LeCun, Y. Dimensionality reduction by learning an invariant mapping. 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06). 2006; pp 1735–1742
  * Xu et al. 2018 Xu, K.; Hu, W.; Leskovec, J.; Jegelka, S. How Powerful are Graph Neural Networks? 2018,
  * Barlow 2001 Barlow, H. Redundancy reduction revisited. _Network: computation in neural systems_ 2001, _12_ , 241
  * Barlow et al. 1961 Barlow, H. B., et al. Possible principles underlying the transformation of sensory messages. _Sensory communication_ 1961, _1_
  * Magar et al. 2021 Magar, R.; Wang, Y.; Lorsung, C.; Liang, C.; Ramasubramanian, H.; Li, P.; Farimani, A. B. AugLiChem: Data Augmentation Library of Chemical Structures for Machine Learning. 2021
  * Kingma and Ba 2014 Kingma, D. P.; Ba, J. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_ 2014,
  * Ward et al. 2018 Ward, L.; Dunn, A.; Faghaninia, A.; Zimmermann, N. E.; Bajaj, S.; Wang, Q.; Montoya, J.; Chen, J.; Bystrom, K.; Dylla, M., et al. Matminer: An open source toolkit for materials data mining. _Computational Materials Science_ 2018, _152_ , 60–69
  * Wilmer et al. 2012 Wilmer, C. E.; Leaf, M.; Lee, C. Y.; Farha, O. K.; Hauser, B. G.; Hupp, J. T.; Snurr, R. Q. Large-scale screening of hypothetical metal–organic frameworks. _Nature chemistry_ 2012, _4_ , 83–89
  * Kim et al. 2017 Kim, C.; Huan, T. D.; Krishnan, S.; Ramprasad, R. A hybrid organic-inorganic perovskite dataset. _Scientific Data_ 2017, _4_ , 1–11
  * Pham et al. 2017 Pham, T. L.; Kino, H.; Terakura, K.; Miyake, T.; Tsuda, K.; Takigawa, I.; Dam, H. C. Machine learning reveals orbital interaction in materials. _Science and Technology of Advanced Materials_ 2017, _18_ , 756–765
  * de Jong et al. 2015 de Jong, M.; Chen, W.; Angsten, T.; Jain, A.; Notestine, R.; Gamst, A.; Sluiter, M.; Krishna Ande, C.; van der Zwaag, S.; Plata, J. J.; Toher, C.; Curtarolo, S.; Ceder, G.; Persson, K. A.; Asta, M. Charting the complete elastic properties of inorganic crystalline compounds. _Scientific Data_ 2015, _2_ , 150009
  * Castelli et al. 2012 Castelli, I. E.; Olsen, T.; Datta, S.; Landis, D. D.; Dahl, S.; Thygesen, K. S.; Jacobsen, K. W. Computational screening of perovskite metal oxides for optimal solar light capture. _Energy Environ. Sci._ 2012, _5_ , 5814–5819
  * Jain et al. 2013 Jain, A.; Ong, S. P.; Hautier, G.; Chen, W.; Richards, W. D.; Dacek, S.; Cholia, S.; Gunter, D.; Skinner, D.; Ceder, G.; Persson, K. A. Commentary: The Materials Project: A materials genome approach to accelerating materials innovation. _APL Materials_ 2013, _1_ , 011002
  * Van der Maaten and Hinton 2008 Van der Maaten, L.; Hinton, G. Visualizing data using t-SNE. _Journal of machine learning research_ 2008, _9_


# Unsupervised Spiking Instance Segmentation on Event Data using STDP

Paul Kirkland, Davide L. Manna, Alex Vicente-Sola, and Gaetano Di Caterina
Neuromorphic Sensor Signal Processing Lab
Centre for Image and Signal Processing
Electrical and Electronic Engineering
University of Strathclyde
Glasgow, United Kingdom.
<EMAIL_ADDRESS>

###### Abstract

Spiking Neural Networks (SNN) and the field of Neuromorphic Engineering has
brought about a paradigm shift in how to approach Machine Learning (ML) and
Computer Vision (CV) problem. This paradigm shift comes from the adaption of
event-based sensing and processing. An event-based vision sensor allows for
sparse and asynchronous events to be produced that are dynamically related to
the scene. Allowing not only the spatial information but a high-fidelity of
temporal information to be captured. Meanwhile avoiding the extra overhead and
redundancy of conventional high frame rate approaches. However, with this
change in paradigm, many techniques from traditional CV and ML are not
applicable to these event-based spatial-temporal visual streams. As such a
limited number of recognition, detection and segmentation approaches exist. In
this paper, we present a novel approach that can perform instance segmentation
using just the weights of a Spike Time Dependent Plasticity trained Spiking
Convolutional Neural Network that was trained for object recognition. This
exploits the spatial and temporal aspects of the network’s internal feature
representations adding this new discriminative capability. We highlight the
new capability by successfully transforming a single class unsupervised
network for face detection into a multi-person face recognition and instance
segmentation network.

_Keywords_ Unsupervised Learning $\cdot$ STDP $\cdot$ Instance Segmentation
$\cdot$ Continual Learning

##  1 Introduction

Human vision has the innate ability to detect, localise, and differentiate
objects of interest, even in the face of multiple background and foreground
distractors. Meanwhile, in Computer Vision (CV), this ability known as
instance segmentation is a complex and challenging task. Instance segmentation
is the detecting and pixel-wise delineation of each distinct object of
interest appearing in an image. In essence instance segmentation is the hybrid
of two key computer vision tasks, object detection and semantic segmentation.
Object detection is the process of detecting instances of objects belonging to
a certain class, while also identifying their spatial location typically with
a bounding box. While semantic segmentation is the task of clustering parts of
an image together that belong to the same object class, resulting in a much
more detailed pixel-wise localisation.

The main component of an object detector for segmentation is a good internal
feature representation [1, 2]. Traditionally, considerable effort was put into
designing handcrafted local descriptors: SIFT [3, 4], HOG [5] and SURF [6].
These low-level features are then utilised to create high-level image
representations through Bag of Words [7] and Fisher Vectors [8]. Which in
turn, are then used for classification. The downside to these feature
representation methods is that often the internal feature representation was
based on the level of domain expertise. Lately, with the advent of Deep
Learning (DL) [9], beyond state-of-the-art performance [10] was now reachable
without any domain expertise and handcrafted features for a multitude of
tasks: image recognition (AlexNet and ResNet) [11, 12], object detection
(Faster R-CNN and RFCN) [13, 14], object tracking (FCNT and MD Net)[15, 16]
and semantic and instance segmentation (FCN, SegNet, U-Net and Mask R-CNN)
[17, 18, 19, 20]. This improvement is mainly due to the combination of feature
extraction and classification now being governed by supervised techniques
using backpropagation. Consequently, this shifted the problem of internal
feature representation to the development of bigger and better neural
networks, where the goal of better segmentation accuracy often transcends
other factors, such as segmentation efficiency. This leads to a stark increase
in the real-time computational and memory overhead of the system [21, 22]. Not
to mention the burden of large volumes of labelled pixel-wise segmentation
images for training. This is especially true for deployable systems, where the
tight constraints on computation are most prevalent [23], while also requiring
an accurate, real-time application. This often leads to an impasse in terms of
efficiency and accuracy, with the traditional methods being more efficiency
focused, while the DL methods being more accuracy focused. Coupled with these,
other desirable features of deployable systems are adaptability and continual
learning [24], allowing the system to better deal with changes in the
environment and unprecedented situations.

The expectation of a modern deployable system is threefold [25]: low latency,
in its ability to perform accurate real-time calculations; low power, in its
efficiency in implementing the computations; continual learning, in having the
ability to learn new features or objects on the fly. Given this, one might
expect a complex method is required to achieve all these objectives in one
system. However, Neuromorphic Engineering [26, 27], exploiting its paradigm
shift from utilising Spiking Neural Networks (SNN) [28] and Event-Based Vision
Sensors (EBVS) [29], has demonstrated that such a system can be attainable.
Furthermore, we demonstrate that from a simple unsupervised Spike Time
Dependent Plasticity (STDP) [30] trained objection recognition Spiking
Convolutional Neural Network (SCNN) [31, 32], our method can extend this to
object detection and instance segmentation. Additionally, this occurs without
any further learning, so in essence, can be strapped onto any STDP trained
network.

Our proposed method, called Hierarchical Unravelling of Linked Kernels and
Similarity Matching through Active Spike Hashing (HULK SMASH), presents a
technique of utilising the internal featural-temporal representation to gain
this instance segmentation ability, that is using the changing internal
feature representation over time to further discriminate (i.e. sub-cluster)
the intra-class representations. HULK SMASH utilises the inherent saliency of
the STDP spatial-temporal features captured by the original network for each
layer. Meaning it no longer just classifies from the upmost hierarchical layer
of the SCNN. Instead utilising the temporal occurrence of internal feature
representations to take an unsupervised network trained to recognise one
class, faces, and extend it to identify and localise individuals. This idea
builds on the efficient use of low-level features of traditional CV methods,
while using networks and techniques from DL to build better hierarchical
features and achieve better accuracy. SNNs provide the ideal platform for this
hybrid approach, while further extending the feature set of the implemented
network.

The HULK SMASH algorithm is not only able to exploit all the useful features
of Neuromorphic Engineering, but adds a further level of interpretability.
This is in part due to the use of sparsity of the visual features making them
easy to realise, while also exploiting the saliency mapping of the SCNN to
help visualise which pixels and features played a part in the classification
process. This allows the overall interpretability of the network to increase,
with the decisions made by the network to become viewable and assessable, so
removing the black box stigma of neural networks. This work makes use of
previous research that showed how an SCNN trained with STDP could first
perform image classification [33], and recently how it can be transformed into
an encoder-decoder network, allowing semantic segmentation to be realised [34,
35]. However, this previous research could not distinguish instances within
these semantic segmentations. Therefore, the main contributions of this paper
are:

  * •

The instance-wise mapping of semantic labels through the decoder network,
HULK. This allows the individual spiking instances of the deepest
convolutional layer (the pseudo classification layer) of a fully convolution
encoder-decoder to be tracked back to the pixel domain. This tracking
illustrates that each semantic label is often built up from multiple spiking
instances from within the pseudo classification layer.

  * •

These semantic instances are then passed onto SMASH, which looks at the
network’s internal featural-temporal representation and uses this to compare
instances. Allowing a metric in which to form an evaluation of similarity
between instances. This is then coupled with a locality checking algorithm
which then reports a SMASH score for each instance. Where similar instances
can combine to form larger representations for objects. Failing that, creating
a new object for any dissimilar instances.

The formulation of the algorithms HULK and SMASH is described in Section 2,
illustrating how to extrapolate the internal feature representation from a
network. Section 3 details the experimental results used to gauge the ability
of the network to now perform object detection and instance segmentation. This
is also extended into situations where partial and full occlusions of objects
happen, with the final experiment showing how the network performs on
occlusion recovery. Section 4 presents details on the interpretability of the
network. Discussing how visualising the internal feature representation of the
network can help to explain how the original object recognition network was
extended for object detection and instance segmentation task. Section 5 then
concludes the paper.

##  2 Temporal Spike Matching

The proposed temporal spike matching algorithm HULK SMASH makes use of an
image classification SNN trained using STDP, similar to that seen in [32, 31,
33]. Prior developments have shown that extending an image classification SNN
to a semantic segmentation network was possible [35, 34]. These two prior
works also utilised the N-Caltech dataset [36] making use of the event driven
nature of the input sequences. This work presents how semantic segmentation
can also be further extended into an efficient method for object detection and
instance segmentation, where instead of using a regression process to indicate
which classification instance are connected, as seen with DL models [17, 18,
19, 20], a proximity and temporal-featural (time- and feature-based)
similarity metric is used. This allows the temporal occurrence of the internal
feature representation, brought about by the temporal dynamic of the input
event stream, to further discriminate intra-class instances. As illustrated in
Figure 1, this method of temporal spike matching for instance segmentation can
be broken down into two main parts, the _Intra-_ and _Inter-Sequence
Processing_ : in other words, what is happening internally and externally
throughout the process. The intra-sequence is where most of the processing
happens, while the inter-sequence allows the processing to link to other
instance of the process running, e.g. to compare object instance to see if
they are the same object. This can be seen in a block diagram of the proposed
method in Figure 1, within the green and red dot-dashed boxes.

Figure 1: Block Diagram of the HULK SMASH system and where it intersects with
the SpikeSEG Network

Figure 1 highlights that the _intra-sequence Process_ starts at the
intersection of the encoder decoder network from SpikeSEG [34]. The first step
of the process is to individually process each of the spiking instances from
the classification layer of the SpikeSEG network. This is in contrast to the
original network, which just grouped all instance based on class. The process
of decoding each instance individually is what is referred to as the HULK
process. Once each classification spike has been decoded back into the
original pixel space, this can then be feed to the SMASH process. This in turn
calculates a bounding box of each instance $S_{BB}$ in the pixel space, by
taking the max and min value in the x and y coordinates. This bounding box is
then it used as a _Proximity Score_ check with every other instance’s bounding
box $S_{BB^{\prime}}$ within the _intra-sequence_. This _Proximity Score_ is
calculated with the Jaccard Index $J(S_{BB},S_{BB^{\prime}})$ [37]. Meanwhile,
the other parallel process performs the _Active Spike Hashing (ASH)_ part of
the SMASH process, which stores 2D featural-temporal data $S_{ci}$ from the 4D
spatial-featural-temporal decoding class instance processing (X, Y, Feature
Map and Time). This then passes onto the _Similarity Matching (SM)_ phase,
where a similarity score is given to the featural-temporal data compared with
every other instance $S_{ci^{\prime}}$ within the _intra-sequence_ , again
with the Jaccard index $J(S_{ci},S_{ci^{\prime}})$. The combination of both
the Similarity and Proximity Score gives the SMASH score $SMASH_{ci,BB}$. This
indicates if the $S_{ci}$ and $S_{BB}$ of that particular class instance match
any other $S_{ci^{\prime}}$ and $S_{BB^{\prime}}$, resulting in an outcome of
whether the instance is part of the same _Object_ $SMASH_{co}$ or a different
one $SMASH_{co^{\prime}}$. This intra-sequence process is run in parallel once
all the instances have their ASH and bounding box process complete. This is
run for every input sequence of data. Which in this case is 10ms of NVS
N-Caltech101 [36] event data, which is internally treated as an asynchronous
stream based on the timestamps of the events. This allows both object
detection and instance segmentation to be performed on this spiking input
sequence.

###  2.1 Hierarchical Unravelling of Linked Kernels

The Hierarchical Unravelling of Linked Kernels (HULK) process permits spiking
activity from the classification convolution layer, to be tracked as it
propagates through the decoding layers of the network. This tracked
propagation allows a record of each subsequent layer’s spatial (x, y),
featural (m) and temporal (t) spiking activity. This process runs in parallel
with the transposed convolution process, which itself is mapping each active
spike in a layer to the subsequent layer. However, after each layer transform,
a record of each spike is stored. This allows a causal hierarchical map to be
created, tracing each classification spike back to the original pixel space.
This process is illustrated within Figure 2, with a decoded sequence shown
entitled ‘Accumulated Decoded Spiking Activity’ and the individual class spike
breakdown from this are shown within Instance A through D. Each instance, in
this case, is representing a single spiking pixel from the classification
layer. Whereas in semantic segmentation, all the instance belonging to one
class are treated as the same entity and decoded. Figure 2 highlights how the
class instance is broken down into its individual instances, where each
instance often provides enough information to recreate the face in the output
pixel space. Although, it is clear that some of the instances favour certain
features over others. It is also apparent that there is much repetition and
accumulation of features even with this sparse spiking domain. It is through
this process of unravelling the classification spiking activity that permits
the ensuing similarity matching process.

Figure 2: Illustration of the individual class spikes being unravelled,
indicating activity caused by these class spikes

###  2.2 Similarity Matching through Active Spike Hashing

The Active Spike Hashing (ASH) is the process of taking the recorded spiking
activity and implementing an efficient and effective way to store the sparse 4
dimensional spatial (x, y), featural (m) and temporal (t) values. This is done
by realising that the convolution structure of the SCNN SpikeSEG is already
dealing with the translational invariance and spatial dimension, together with
the bounding box proximity score. It can also be noted that position within
the scene is not a useful evaluation metric of the similarity between two
objects. The ASH process then results in a 2D featural-temporal hashing of the
spiking activity. The featural data then has the same total number of features
as the network, which in this case, is 41. This value comes from the use of a
network with a similar structure as the Face-Motorbike Network used Kirkland
et al. [35], except with only one classification layer. The other 40 features
come from the 36 from Trans-Conv2 and 4 from Trans-Conv1.

A further memory reduction to the ASH process is permitted by storing the
values as binary terms. As the number of spikes recorded within each map per
timestep is more a measure of the total spiking activity, rather than the
featural-temporal characteristics we evaluate with the similarity measure.

An illustration of the ASH process where the spiking activity is assigned its
feature map and time-stamp index (f,t) can be seen in Figure 3. This builds
upon the HULK process previously seen in Figure 2 with the indices of the
spiking activity now visible in Figure 3 for each convolutional layer. Each
active neuron is being assigned an f and t index on a class-instance basis.
Allowing a 2D matrix of each instance to be realised where from Figure 3,
Instance A would have 1s in the first column (time) forth row (feature map)
for the classification layer activity. Meanwhile, the green coloured features
of Instance A’s Trans-Conv 2, are stored with row 19 and column 3 and 5. The
ASH process typically reduces the size of the tensor by 98%, reducing the
memory overhead considerably. The result of the ASH process leaves the
internal feature representation similar to that of a conventional spike train.

Figure 3: Further explanation of the Active Spike Hashing mechanism from the
HULK process, feature map and time given to each active neurons

Once the spiking activity is hashed, it is ready for the Similarity Matching
section of SMASH. The Jaccard similarity coefficient was once again utilised,
within the intersection over union used to calculate the segmentation bounding
box overlap. However, within this instance, the measure is used to see how
much similarity there is from each instance in the intra-sequence process,
comparing the featural-temporal similarity of each instances spiking activity,
with respect to the overall amount of spiking activity [37].

$J(S_{Ci},S_{Ci^{\prime}})=\frac{|S_{Ci}\cap S_{Ci^{\prime}}|}{|S_{Ci}\cup
S_{Ci^{\prime}}|}$ (1)

where $S_{Ci}$ is the spiking activity of that class instance, and
$S_{Ci^{\prime}}$ represents the spiking activity of any other class instance
for that sequence. However, due to the ASH process storing binary values of
the activities, this equation can be simplified down to a logical calculation
performed with only ORs and ANDs. This allows the quick comparison of the
number of spikes that feature in both the current instance _and_ the
comparison, divided by the number of spikes that feature in the current
instance _or_ the comparison.

$J(S_{Ci},S_{Ci^{\prime}})=\frac{S_{Ci}\land S_{Ci^{\prime}}}{S_{Ci}\lor
S_{Ci^{\prime}}}$ (2)

To complete the intra-sequence of the SMASH process, the bounding box IoU
score must be calculated for each class instance bounding box $Ci_{BB}$
against the other instances bounding boxes $Ci_{BB^{\prime}}$, instead of for
each class in total

$J(Ci_{BB},Ci_{BB^{\prime}})=\frac{|Ci_{BB}\cap
Ci_{BB^{\prime}}|}{|Ci_{BB}\cup Ci_{BB^{\prime}}|}$ (3)

Multiplication of the similarity score, with the IoU, results in the novel
proposed SMASH score for each instance

$\text{SMASH}(Ci,Ci^{\prime})=J(S_{Ci},S_{Ci^{\prime}})\times
J(Ci_{BB},Ci_{BB^{\prime}})$ (4)

Once the SMASH score is calculated for each instance, the maximum of each
instance is assigned to a class object $Co$

$S_{Co}=\operatorname*{arg\,max}_{Ci}(\text{SMASH}(Ci,Ci^{\prime}))$ (5)

This maximum SMASH score is based on the pairing $Ci,Ci^{\prime}$, where each
object $S_{Co}$ is compiled from any overlapping parings e.g. for 5 instances
the argmax presents the following sets of $Ci,Ci^{\prime}$ pairs, (1-3),
(2-4), (3-5), (4-2), (5-3). The values 2 and 4 are assigned to one object,
while 1, 3 and 5, as 3 and 5 are matched pairs and 1 is associated with 3, so
becomes part of that object. It is these class objects $S_{Co}$, that are used
within the inter-sequence processing. This permits class objects from
preceding sequences $\tilde{S_{Co}}$ to be compared with current ones, again
looking for similarity with the binary Jaccard coefficient

$J(S_{Co},\tilde{S_{Co}})=\operatorname*{arg\,max}_{S_{Co}}\frac{S_{Co}\land\tilde{S_{Co}}}{S_{Co}\lor\tilde{S_{Co}}}$
(6)

These class objects within the inter-sequence then allow sequence to sequence
continuity, thus allowing tracking of objects permitted that they maintain a
level of feature similarity.

As the original SpikeSEG network outputs a semantic segmentation output, the
HULK and SMASH processes are supplementary to this process in transforming the
semantic regions into instance-based objects. Based on the block diagram seen
in Figure 1, the pseudo-code for the SMASH process is provided within
Algorithm 1. This pseudo-code allows further insight to the method for
comparison both intra- and inter-sequence and serves to compliment the block
diagram seen in Figure 1, but with the addition on the internal functions used
to calculate the values.

Algorithm 1 SMASH

1:procedure intra-sequence($a,b$)

2: for each input sequence do

3: for each spiking instance in the classification layer do

4: perform HULK for each instance

5: compute max and min, x and y values

6: compute the [$X,Y,W,H$] bounding boxes

7: perform ASH to create $S_{Ci}$

8: return $S_{Ci}$

9: end for

10: for each hashed instance, $S_{Ci}$ do

11: compute Similarity score against other instances, $S_{Ci^{\prime}}$ as in
(2)

12: compute Bounding Box IoU score against other instances $S_{Ci^{\prime}}$
as in (3)

13: compute the SMASH score as in (4)

14: assign max SMASH score instance pair to object $S_{Co}$ as in (5)

15: return $ListofS_{Co}$

16: end for

17: end for

18:end procedure

19:procedure inter-sequence($a,b$)

20: for each class object $S_{Co}$ do

21: compute Similarity score against previous class objects $\tilde{S_{Co}}$
as in (6)

22: return $max$($J(S_{Co},\tilde{S_{Co}})$)

23: end for

24:end procedure

From the pseudo-code, it can be determined that the SMASH method has two
conditions before it concludes that instances are of the same object:

  * •

That the two instances must have overlapping bounding boxes

  * •

That the two instances must share a featural-temporal data

Without either of these, the multiplication of the similarity score and
intersection over union will result in 0, and it will determine that the two
instances are different objects. All instances that combine into the same
object are then stored as one set of features and used at the inter-sequence
stage to track objects over multiple input sequences.

The HULK and SMASH processes are able to build upon the semantic segmentation
abilities of the semantic segmentation network. Giving object detection and
instance segmentation capability, without the use of a regression sequence.
Instead, this method opts for locality and similarity of the most salient
features, utilising the information already available from the SCNN process.

##  3 Experimental Results

In the evaluation of the performance of the novel HULK SMASH algorithm
presented in this paper, a number of tests have been carried out. The tests
include:

  * •

Semantic to instance segmentation with object detection

  * •

Sequence to sequence tracking of objects with object occlusion and recovery

  * •

Similarity matching for intra-class grouping (feature detection)

These tests are all carried out with the N-Caltech Dataset [36], mostly
utilising the Face category. This Face subset provides a subtle yet distinct
amount of intra-class variance, allowing the successful extraction of multiple
diverse face life features within the second convolution layer, which then
provides a robust and general face detection within the classification layer.
The training was carried out using 10 of the different people within the
subset, with testing being carried out on the same 10 people, however with
different input sequences used for each. A total of 3 input sequences are used
in training, each containing 300ms of spiking activity, so 30 buffered inputs
(3 of each of the 10 people), for a total of 900 input sequences. Testing was
carried out using 2 input sequences, so 600 buffered inputs. The network
parameters are set to be the same as within [34], with the only difference
being 36 features available for the one class present. This was to limit the
number of external factors and focus the testing on the intra-classification
abilities rather than inter classification. To further validate the SMASH
method the 5 and 10 class networks where also utilised from [35]. These tests
helps to validate the SMASH method in a more complex feature similarity
environment, where each class has less representation in layer Conv 2, as
these networks utilised 16 features times the number of classes, 80 and 160
for the 5 and 10 classes respectively.

###  3.1 Semantic to Instance Segmentation with Object Detection

A series of multiple input streams are given as input to the SpikeSEG network,
illustrated in Figure 4. The network then creates both the semantic
segmentation output and via the HULK-SMASH process, an object
detection/instance segmentation output. The input sequence is shown in Figure
4 (a), with 5 input stream presented at the same times, 3 with Faces and 2
without. Figure 4 (b) highlights the semantic segmentation output of the
SpikeSEG network, while Figure 4 (c) demonstrates the HULK process extracting
each instance from the semantic representation. Figure 4 (d) illustrates how
the SMASH process groups instances that score above 0 into class instance
objects. Lastly Figure 4 (e) presents the final instance segmentation output,
thanks to the object detection separating the semantic representation.

Figure 4: Processing from Semantic Segmentation to Instance Segmentation: (a)
the accumulated input sequence, (b) the semantic segmentation output, (c) the
class instances from the semantic segmentation, (d) the grouped class objects
and (e) the instance segmentation output

Within this scenario shown in Figure 4, the separate input streams are not
overlapping, consequently meaning the bounding boxes between input stream also
do not overlap. This means the SMASH score between the input streams will
always be 0 regardless of the similarity score. This is partially the
intention of the SMASH process, allowing objects that do not share spatial
locality to lead to false object grouping. As such the performance of the
object detection to identify that the individual class instances belonging to
one object is being evaluated, similar to the process depicted in Figure 2 and
3. This means that even though the network is classifying all the Faces as
just that, Faces, the SMASH process is able to identify that these do no share
the same featural content. The object detection accuracy to identify three
separate distinct faces is 100% over the complete input sequence, with
numerical results tabulated within Table 1. However, over each input buffer,
this drops to 97.33% as there are a few cases in which the similarity is 0 due
to a low input spiking rate producing the minimal number of spikes for
activation. This is typical of the inputs from the start and end of the
saccade movement within the N-Caltech dataset [36].

Table 1: Results from the experimental testing split into 4 sections covering
the Object detection, detection with occlusion, occlusion recovery and object
self matching. *MS - Multistream

| Object Detection Accuracy (%)
---|---
| per Input | per Sequence |
MS N-CalTech (Face) | 97.33 | 100 |
MS N-CalTech (5 Class) | 95.17 | 100 |
MS N-CalTech (10 Class) | 91.67 | 100 |
| Detection Accuracy with rate of Occlusion (%)
| 5% | 25% | 50%
MS N-CalTech (Face) | 97.33 | 62 | 28
| Occlusion Recovery Rate (%)
| without noise | with noise |
MS N-CalTech (Face) | 100 | 100 |
| Object Self Matching
| Top 1 | Top 3 | Top 10
MS N-CalTech (Face) | 95 | 100 | 100

###  3.2 Object Detection and Segmentation in a Multi-Class Environment

To further evaluate the SMASH method, the multi-class semantic segmentation
networks were tested. This allowed testing of the object detection and
instance segmentation from the 5 and 10 class semantic segmentation
environment. Together with the increased number of classes present in each
segment, there was also a reduction in the overall number of features
available within the second convolution layer, Conv 2. This presents a
scenario where the features are more generalised and therefore present less
opportunities to learn features that help to diversify. Despite this, the
SMASH process was still able to deliver high accuracy in determining how many
object instances of a class there were, with 95.17% within the 5 class testing
and 91.67% for the 10 class test, collated in Table 1. To help visualise the
results from this experiment, two examples of successful instance
segmentations outputs from the 10 class test are shown in Figure 5, which
shows for both (a) and (b), the input overlaid with the classification layer
feature map and the semantic segmentation output respectively. Figure 5 also
indicates how the class instances are assigned in the ‘Instance Map’, where
the numerical assignment is displayed to help illustrate the SMASH score also
seen within the image. The Pairs are shown in numerical order with associated
SMASH score, and how that is grouped into objects.

Figure 5: Illustrating the process of going from semantic segmentation to
instance wise object construction, displaying the instance mapping and pairing
with SMASH scores that help to construct the class instance objects for two
examples (a) and (b). Respectively showing the process for the classes Face
and Stop Sign, and Watch and In-line Skate.

The classes other than Face presented a different problem to the Face class,
as most of the other classes (excluding Stop Sign) had a higher feature count
in the second convolution layer Conv 2. This presented a problem in which the
timing of the occurrences of the features, was more important than its
occurrence. Which during the Face only testing, was not an issue due to the
sparsity at which those features appeared. This testing highlights the
importance of the temporal component of the similarity matching, as simple
feature-wise similarity matching alone is unable to determine class instances
from one another. This extra temporal dimension can be seen a being able to
leverage the saliency (timing) of features within individual class instances.

###  3.3 Object Occlusion

To help test the HULK-SMASHs ability to be able to perform sequence to
sequence tracking of objects, further evaluation of the object detection
capability is tested within three more challenging scenarios, where the class
instance will now overlap. Taking the same multi-stream approach as the
previous test, however, now the central stream of the image will be positioned
such that occlusion of 5%, 25% and 50% will occur. Partial occlusions test the
feature extraction and similarity matching performance when there is a
bounding box overlap, meaning the SMASH score is now the active measure of the
combined similarity and locality. The three occlusion states are illustrated
in Figure 6, where (a), (b) and (c) depicted the occlusion covering for 5 %,
25% and 50% respectively as the stream of data in the centre is repositioned
to create the occlusion.

Figure 6: Input occlusion example illustrating: (a) 5% occlusion of center
image, (b) 25% occlusion of center image, and (c) 50% occlusion of center
image

With the 5% occlusion, the performance was unchanged at 97.33% also shown in
Table 1, which was to be expected considering the example shown in Figure 6
(a) shows no overlapping of the facial features, with all images in the
dataset having the faces centred in the image. While the 25% occlusion reduced
the performance down to 62% and with 50% occlusion getting 28% also both in
Table 1 when testing over each input buffer. A more detailed example from the
25% occlusion testing is shown in Figure 7. The figure highlights the process
from the input in (a), the class latent space neurons that were active in (b),
the HULK end process of the active class pixels represented back in the pixel
space with (c), and finally, the instance segmentation output (d), with three
successful instances of each face detected. The only concern with this
approach is the overlapped section between the bottom two images has a region
that is assigned to two objects. This results in an area of uncertainty and
even with a process where the two objects could compete for the pixels, the
misclassification appears to exist in the feature detection domain, not the
HULK-SMASH process. However, it must be noted that the failures within this
testing phase are exclusively from the failure to find enough features to give
a positive classification, not the incorrect assignment of instances. This
element is highlighted in Figure 8, where the classification process has
failed to result in classification for the centre image; however, the bottom
left and top right faces are correctly classified and segmented without
overlap onto the unclassified region.

Figure 7: Input 25% occlusion example illustrating: (a) The input sequence,
(b) The latent classification space, (c) The class instance breakdown and (d)
The instance segmentation output Figure 8: Segmentation mapped over the input
sequence, with a failure to identify the central image, thus failure to
include in instance segmentation, however successful segmentation of the other
two faces

###  3.4 Object Occlusion Recovery

As shown in the previous section, the object detection and instance
segmentation method can identify features belonging to a particular instance
of that class on an input by input basis, known as intra-sequence detection.
The experimental set up for this section allows testing of object detection
across inputs multiple sequences known as inter-sequence detection. This test
then allows sequence to sequence tracking, then partial and full occlusion and
subsequent recovery. The inputs are the full 300 ms sequence of the N-Caltech
dataset [36] broken into the usual 10 ms steps as seen in previous sections.
This places two inputs at the top right and bottom left, then has then move
diagonally across towards each other for each buffered input. The results of
the test from a few selected instances are shown in Figure 9. Figure 9 (a)
depicts the start position of the two different sequences and their respective
instance segmentation and bounding boxes (red and black). Figure 9 (b-e) show
the transition of the respective segmentations diagonally across, with (c) and
(d) showing the occlusion due to overlap where the black object completely
occludes the red object. Figure 9 (e) then highlights the detection picking up
the red object again, from finding a high similarity with the red object that
existed prior to the occlusion.

Figure 9: Segmentation mapped over the input sequence, with a failure to
identify the central image, thus failure to include in instance segmentation,
however successful segmentation of the other two faces

The experimental results show the inter-sequence similarity matching is able
to recover the occluded input sequence, such that across all the test
sequences the similarity check managed to recover the occluded objects 100% of
the time, as seen in Table 1. This feat is more impressive due to the occluded
object being a dynamic sequence itself, meaning the post occlusion object is
not just a replica of pre occlusion one. This occlusion recovery was also
tested again with a noisy input, that is added noise to both the input
sequence and to the general background. This noisy sequence is shown in Figure
10, where Figure 10 (a) and (e) shown the pre and post occlusion states,
presenting that the recovery works correctly, while Figure 10 (b), (c) and (d)
show some of the issues the occlusion can cause. Figure 10 (b) highlights the
failure of the object detection in finding extra objects that do not exist.
Figure 10 (c) correctly illustrates the full occlusion state where only the
female face object is present. Figure 10 (d) displays the point at which
partial occlusion is still causing only one object to be present, due to the
similarity of class instances being close across the 9 instances in this case.
Some of the 9 instances include multiple features of both objects, similar to
what is shown in Figure 10 (b). However, in this case, there was at least one
instance that matched another within both objects. The object in Figure 10 (d)
is also blue as it scored the highest similarity with that object on the
inter-sequence process. Despite some of the mentioned inaccuracies associated
with the added noise, the system was able to re-identify occluded objects 100%
of the time, as shown in Figure 10 with this result recorded with the others
in Table 1.

Figure 10: Occlusion recovery with noisy input. (a) Pre Occlusion state, (b)
Occlusion causing extra objects to appear, (c) Full occlusion with only one
object present, (d) Partial occlusion grouping of objects and (e) Post
occlusion state

##  4 Intuition and Understanding through Visualisation

The principle behind how object detection and instance segmentation are
resolved can be better understood through visualisation. Through
visualisations of the weights as they are mapped back into the pixel space
through the subsequent layers are shown in Figure 11. Figure 11 (a) showing
the features of Conv 1, which are the unlearned pre-determined edge detection
features, (b) showing the features of layer Conv 2 (Trans-Conv 2) and (c)
highlighting the classification layer feature which depicts a Face. Figure 11
(b) contains the bulk of the information used for the similarity matches
process, thus contains the key to interpreting how the network can
differentiate between different people in the dataset.

Figure 11: The set of features from the trained network, (a) Conv 1, (b) Conv
2 and (c) Conv3

Mapping the features of the network from Conv 2 layer onto some original
images from the Caltech dataset presents an insight into how this layer is
differentiating between the different people in the dataset, as each person
has unique and distinctive enough features to be learned through repeat
occurrence. This ability to learn sub-classification feature clustering stems
from the ability to classify up to 10 different objects due to the variance
between the classes (inter-class) being higher than the variance of one
particular class (intra-class). However, in this case, it is now the intra
sub-class variance that allows the unique featural-temporal identifiers of
each face to be captured. This is highlighted in Figure 12 where 4 distinctive
features that appear to represent a particular person from the dataset are
overlaid onto the non-spiking version of the original image (for easier
visualisation) it matches closest. In which the features of Figure 12 appear
to pick out the typical facial features of eyes nose mouth and hair.

Figure 12: Overlaid Features from Conv 2 onto Images from original Caltech
Dataset, displaying how the learned features represent different faces

To test just how well the features are allowing the individuals within the
dataset to be recognised, an experiment matching each sequence with its top 1,
top 3 and top 10 matches was carried out. This permits an insight into whether
or not the HULK-SMASH process was able to differentiate between the people
within the dataset, purely on re-occurrence of similar salient features, or in
other words distinctive facial features in a featural-temporal manner. The
test processes each buffered input, within each sequence of the testing set,
and compares it with the full test set, recording the top, top 3 and top 10
most similar spatial-temporal patterns to its own. The results show that over
95% of the test inputs for an individual match the highest with themselves.
While 100% match with themselves if extended into the top 3 and top 10,
displayed in table 1. This result is illustrated in Figure 13, where the top
10 similarity matches for a selection of inputs are shown in a coloured
column-wise manner, with the left of the column showing the input sequence and
the right showing the top 10 matches in descending order.

Figure 13: Columns of top 10 matches, left on each column is the test input
repeated 10 times with each of the 10 matches to the right, ranked in
descending order. Each colour represents a new column for easier visual
separation

Figure 13 helps to build interpretability to the similarity matching process,
allowing an understanding as to how and why the accuracy of the classification
and object detection is as such. The sparse feature sets of a winner takes all
approach to STDP allowing easier visualisation. This in turn helped to
construct a method of matching these spatial-temporal features which are a
result of the neuromorphic input to the spiking network both maintaining the
important temporal causality of salient features.

##  5 Conclusion

In this paper a new method for spiking object detection and instance
segmentation, HULK-SMASH was presented. The system utilised STDP learned
features that previously gave a semantic segmentation output. The
classification layer was subsequently processed by unravelling each individual
latent spiking neuron in the layer back to the pixel space, seen as HULK. This
allows each instance of the classification layer to be treated as a class
instance. It is then through comparing the class instances that objects can be
identified through the SMASH score looking at the similarity and locality of
instances. Permitting object detection and instance segmentation to be carried
out on each buffered input. Then through similarity matching of the objects in
sequences, sequence to sequence object detection and segmentation was
achieved. This allowed object occlusion and reappearance to be realised, with
the ability to deal with subtle changes to the object pre and post occlusion.
The same mechanisms for SMASH could be used for continual online learning in
which new objects are created so long as they trigger a classification neuron.
Visualisation of experimental results gave a better understanding and
intuition as to where the network was working well and why some failures
occur. The HULK-SMASH processes were shown to be robust to multiple network
instances with single and multi-class experiments with successful results
while aiding in neural network interpretability.

## References

  * [1] R. Girshick, J. Donahue, T. Darrell, and J. Malik, “Rich feature hierarchies for accurate object detection and semantic segmentation,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580–587, 2014.
  * [2] S. J. Dickinson, A. Leonardis, B. Schiele, and M. J. Tarr, Object categorization: computer and human vision perspectives.  Cambridge University Press, 2009.
  * [3] D. G. Lowe, “Object recognition from local scale-invariant features,” in Proceedings of the seventh IEEE international conference on computer vision, vol. 2, pp. 1150–1157, Ieee, 1999.
  * [4] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” International journal of computer vision, vol. 60, no. 2, pp. 91–110, 2004.
  * [5] N. Dalal and B. Triggs, “Histograms of oriented gradients for human detection,” in 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05), vol. 1, pp. 886–893, Ieee, 2005.
  * [6] H. Bay, T. Tuytelaars, and L. Van Gool, “Surf: Speeded up robust features,” in European conference on computer vision, pp. 404–417, Springer, 2006\.
  * [7] J. Sivic and A. Zisserman, “Video google: A text retrieval approach to object matching in videos,” in Computer Vision, IEEE International Conference on, vol. 3, pp. 1470–1470, IEEE Computer Society, 2003.
  * [8] F. Perronnin, J. Sánchez, and T. Mensink, “Improving the fisher kernel for large-scale image classification,” in European conference on computer vision, pp. 143–156, Springer, 2010.
  * [9] Y. LeCun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015.
  * [10] A. Voulodimos, N. Doulamis, A. Doulamis, and E. Protopapadakis, “Deep learning for computer vision: A brief review,” Computational intelligence and neuroscience, vol. 2018, 2018.
  * [11] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in neural information processing systems, pp. 1097–1105, 2012.
  * [12] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  * [13] S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real-time object detection with region proposal networks,” in Advances in neural information processing systems, pp. 91–99, 2015.
  * [14] J. Dai, Y. Li, K. He, and J. Sun, “R-fcn: Object detection via region-based fully convolutional networks,” in Advances in neural information processing systems, pp. 379–387, 2016.
  * [15] L. Wang, W. Ouyang, X. Wang, and H. Lu, “Visual tracking with fully convolutional networks,” in Proceedings of the IEEE international conference on computer vision, pp. 3119–3127, 2015.
  * [16] H. Nam and B. Han, “Learning multi-domain convolutional neural networks for visual tracking,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4293–4302, 2016.
  * [17] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3431–3440, IEEE, jun 2015.
  * [18] V. Badrinarayanan, A. Kendall, and R. Cipolla, “SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 39, pp. 2481–2495, dec 2017.
  * [19] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in International Conference on Medical image computing and computer-assisted intervention, pp. 234–241, Springer, 2015\.
  * [20] K. He, G. Gkioxari, P. Dollár, and R. Girshick, “Mask r-cnn,” in Proceedings of the IEEE international conference on computer vision, pp. 2961–2969, 2017.
  * [21] G. Marcus, “Deep learning: A critical appraisal,” arXiv preprint arXiv:1801.00631, 2018.
  * [22] N. C. Thompson, K. Greenewald, K. Lee, and G. F. Manso, “The computational limits of deep learning,” arXiv preprint arXiv:2007.05558, 2020.
  * [23] E. García-Martín, C. F. Rodrigues, G. Riley, and H. Grahn, “Estimation of energy consumption in machine learning,” Journal of Parallel and Distributed Computing, vol. 134, pp. 75–88, 2019.
  * [24] F. Zenke, B. Poole, and S. Ganguli, “Continual learning through synaptic intelligence,” in International Conference on Machine Learning, pp. 3987–3995, PMLR, 2017.
  * [25] J. Chen and X. Ran, “Deep learning with edge computing: A review.,” Proc. IEEE, vol. 107, no. 8, pp. 1655–1674, 2019.
  * [26] G. Indiveri and T. K. Horiuchi, “Frontiers in neuromorphic engineering,” Frontiers in neuroscience, vol. 5, p. 118, 2011.
  * [27] C. Mead, “How we created neuromorphic engineering,” Nature Electronics, vol. 3, no. 7, pp. 434–435, 2020.
  * [28] S. Ghosh-Dastidar and H. Adeli, “Spiking neural networks,” International journal of neural systems, vol. 19, no. 04, pp. 295–308, 2009.
  * [29] G. Gallego, T. Delbruck, G. Orchard, C. Bartolozzi, B. Taba, A. Censi, A. J. Leutenegger, S. Davison, J. Conradt, K. Daniilidis, and D. Scaramuzza, “Event-based vision: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
  * [30] G.-q. Bi and M.-m. Poo, “Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type,” Journal of neuroscience, vol. 18, no. 24, pp. 10464–10472, 1998\.
  * [31] S. R. Kheradpisheh, M. Ganjtabesh, S. J. Thorpe, and T. Masquelier, “STDP-based spiking deep convolutional neural networks for object recognition,” Neural Networks, vol. 99, pp. 56–67, mar 2018.
  * [32] T. Masquelier and S. J. Thorpe, “Unsupervised learning of visual features through spike timing dependent plasticity,” PLoS computational biology, vol. 3, no. 2, 2007.
  * [33] P. Kirkland, G. Di Caterina, J. Soraghan, Y. Andreopoulos, and G. Matich, “Uav detection: a stdp trained deep convolutional spiking neural network retina-neuromorphic approach,” in International Conference on Artificial Neural Networks, pp. 724–736, Springer, 2019.
  * [34] P. Kirkland, G. Di Caterina, J. Soraghan, and G. Matich, “Spikeseg: Spiking segmentation via stdp saliency mapping,” in 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1–8, IEEE, 2020.
  * [35] P. Kirkland, G. Di Caterina, J. Soraghan, and G. Matich, “Perception understanding action: adding understanding to the perception action cycle with spiking segmentation,” Frontiers in Neurorobotics, vol. 14, 2020.
  * [36] G. Orchard, A. Jayawant, G. K. Cohen, and N. Thakor, “Converting static image datasets to spiking neuromorphic datasets using saccades,” Frontiers in neuroscience, vol. 9, p. 437, 2015.
  * [37] P. Jaccard, “Distribution de la flore alpine dans le bassin des dranses et dans quelques régions voisines,” Bull Soc Vaudoise Sci Nat, vol. 37, pp. 241–272, 1901.