SHOARI NEJAD et al

*Amin Shoari Nejad,

# SERT: A Transfomer Based Model for Spatio-Temporal Sensor Data with Missing
Values for Environmental Monitoring

Amin Shoari Nejad  Rocío Alaiz-Rodríguez  Gerard D. McCarthy  Brian Kelleher
Anthony Grey  Andrew Parnell  Hamilton Institute, Insight Centre for Data
Analytics, Maynooth University, Kildare, Ireland  Department of Electrical,
Systems and Automation, University of León, Spain  ICARUS, Department of
Geography, Maynooth University, Maynooth, Ireland  Organic Geochemical
Research Laboratory, Dublin City University, Glasnevin Campus, Dublin 9,
Ireland<EMAIL_ADDRESS>
###### Abstract

[Abstract]Environmental monitoring is crucial to our understanding of climate
change, biodiversity loss and pollution. The availability of large-scale
spatio-temporal data from sources such as sensors and satellites allows us to
develop sophisticated models for forecasting and understanding key drivers.
However, the data collected from sensors often contain missing values due to
faulty equipment or maintenance issues. The missing values rarely occur
simultaneously leading to data that are multivariate misaligned sparse time
series. We propose two models that are capable of performing multivariate
spatio-temporal forecasting while handling missing data naturally without the
need for imputation. The first model is a transformer-based model, which we
name SERT (Spatio-temporal Encoder Representations from Transformers). The
second is a simpler model named SST-ANN (Sparse Spatio-Temporal Artificial
Neural Network) which is capable of providing interpretable results. We
conduct extensive experiments on two different datasets for multivariate
spatio-temporal forecasting and show that our models have competitive or
superior performance to those at the state-of-the-art.

###### keywords:

Spatio-temporal, Deep learning, Transformers, Environmental monitoring

##  1 Introduction

The importance of spatio-temporal forecasting has increased significantly in
recent years due to the availability of large-scale spatio-temporal data from
various sources such as sensors and satellites 12. Spatio-temporal forecasting
involves predicting how data vary over space and time, which is critical for a
wide range of applications such as water quality forecasting 5. A common
approach to modelling spatio-temporal data is to use a multivariate time
series structure, where each time series is associated with a variable at a
specific location 27.

Spatio-temporal data often contain missing values which is a common problem in
environmental monitoring, and can be caused by sensor failure, malfunction or
communication problems (for example see Figure 1). A common remedy for
forecasting with missing data is to impute the missing values using a variety
of methods 25, 21. However, these methods are not always effective, can reduce
the signal in the data, and thus make the modelling or forecasting task more
challenging and unreliable. Therefore it is important to design models that
can handle missing data naturally whilst simultaneously learning the
underlying patterns in the data without resorting to imputation.

Figure 1: An example of our environmental monitoring dataset; a multivariate
environmental time-series with missing values.

We propose a new model that is capable of performing multivariate spatio-
temporal forecasting named SERT (Spatio-temporal Encoder Representations from
Transformers). Our model is an extension of the well-known transformer
architecture that has shown remarkable success in natural language processing
and also image analysis 6, 8. SERT is designed to capture the complex joint
temporal and spatial dependencies among the input variables. An important
feature of our proposed model which differentiates it from the many other
available methods is its ability to handle missing data more naturally without
requiring any missing value imputation.

In addition to the SERT model, we introduce an interpretable simplified
version that provides insights into the underlying factors that drive the
predicted values, which can assist in decision and policy-making. Our proposed
simplified model is named SST-ANN (Sparse Spatio-Temporal Artificial Neural
Network) and removes the transformer layers from the SERT structure. Despite
being less accurate than SERT, it is capable of providing insightful results
with faster computation time while similarly being able to handle missing
values. Depending on the complexity of the problem, the required accuracy and
the available computational resources, the user can fit both SERT and SST-ANN,
using the former to provide more accurate forecasts and the latter to forecast
and gain insights about how the results were obtained.

To evaluate the performance of our proposed models, we conducted extensive
experiments on two different datasets for multivariate spatio-temporal
forecasting. We fitted our models to a simulated dataset to assess their
ability to function under different levels of sparsity. We also evaluated the
performance of the models on a real-world dataset, including missing values,
of environmental variables in Dublin Bay for 7 hour ahead forecasting. Our
experimental results show that our models are competitive with state-of-the-
art models for multivariate spatio-temporal forecasting.

Our paper is organized as follows. In Section 2, we provide a brief overview
of related work on models developed for analysing sequential data in general
and spatio-temporal forecasting applied to environmental monitoring in
particular. In Section 3, we describe the proposed SERT and SST-ANN models in
detail. In Section 4, we present the experimental results and analysis.
Finally, in Section 5, we conclude the paper and discuss future directions of
research.

##  2 Related Work

In this section we provide a brief overview of the recent developments in deep
learning models for sequential data analysis and spatio-temporal models for
environmental monitoring, and also methods for handling missing data and
adding interpretability to deep learning models applied to time series data.

###  2.1 Deep Learning Models for Sequential Data

Recurrent Neural Networks (RNNs) have been one of the most popular deep
learning models for sequential data 15. However, RNNs suffer from the
vanishing gradient problem which makes them unable to learn long-term
dependencies in the data. To address this problem, Long Short-Term Memory
(LSTM) networks 13 and Gated Recurrent Unit (GRU) networks 3 were introduced.
These models have been applied to various tasks such as machine translation 1,
speech recognition 10, and time series forecasting 16.

More recently a new type of deep learning model named transformers was
introduced 26. Transformers are based on the attention mechanism which enables
them to learn the dependencies between the input and output sequences.
Transformers are comprised of an encoder and a decoder network. The encoder
network is responsible for learning the representation of the input sequence,
while the decoder network is responsible for generating the output sequence
based on the learned representation. Models developed on the transformer
architecture include BERT 6, which uses only the encoder part, and GPT 19
which uses only the decoder part. These models have been applied to various
tasks in natural language processing such as question answering 6, text
classification 22, and text summarization 17. Overall, transformers have
proven themselves to be more effective than recurrent based models in many
applications, especially in natural language processing.

###  2.2 Deep Learning Models for Spatio-Temporal data

Spatio-temporal forecasting is often framed as multivariate time series
forecasting where each time series is associated with a variable at a specific
location. The application of deep learning models for spatio-temporal
forecasting is not new. For example, 30 used an LSTM to forecast daily land
surface temperature, and 18 developed an ensemble quadratic echo state network
for forecasting Pacific sea surface temperature. However, the application of
transformers to spatio-temporal forecasting is relatively new and challenging
because transformers have been mainly developed in the field of natural
language processing (NLP). Nonetheless researchers were inspired by the
success of transformers in NLP and started adapting them to spatio-temporal
forecasting which can be formulated as a sequence-to-sequence problem, where
the input is a sequence of historical observations of multiple variables at
different locations, and the output is a sequence of future predictions of the
same variables at the same locations. A common approach for sequence-to-
sequence modeling is to use an encoder-decoder architecture, where an encoder
network maps the input sequence into a latent representation, and a decoder
network generates the output sequence from the latent representation 23. 11
used this idea to develop a new model called Spacetimeformer and applied it to
traffic prediction and weather forecasting. However, to the best of our
knowledge, the application of transformers to environmental monitoring is
limited to the recent work by 28 who used a transformer-based model for hourly
$PM_{2.5}$ forecasting in Los Angeles.

###  2.3 Addressing Missing Values in Modelling

As mentioned in the introduction, a major challenge in spatio-temporal
forecasting in the environmental monitoring context is dealing with missing
values. A common approach for dealing with the missing values is imputation 25
before conducting any analysis. In time series modelling care needs to be
taken to avoid introducing bias; last observation carried forward is a common
approach. An alternative used in the literature is that of a Bayesian
framework which enables defining a prior distribution over the missing values
so that they can be inferred with the other unobserved parameters when fitting
the models. For example, 20 proposed a Bayesian model (called VARICH) for
spatio-temporal modelling of turbidity data with many missing values. However,
this approach is computationally expensive and requires a large number of
samples from the posterior distribution to obtain acceptable results and thus
is not suitable for large spatio-temporal datasets.

To address the missing data problem in time series, 14 proposed a novel
approach to encode multivariate time series using set functions and introduced
a new model called SeFT for classifying time series with irregularly sampled
clinical data. More recently, 24 proposed a new transformer based model called
STraTS that represents each observation as a triplet of the form (time,
variable name, value). As opposed to SeFT, STraTS uses a learnable positional
encoding and a Continuous Value Embedding (CVE) scheme that is a one-to-many
feed-forward network. STraTS was developed to perform multivariate time series
forecasting during the pre-training phase and classification as the final task
on irregularly sampled clinical data. It has been shown to have higher
accuracy than its predecessor, the SeFT model, when applied to the
classification of clinical time series. Both SeFT and STraTS can handle
missing values without requiring any imputation. To the best of our knowledge,
none of these novel methods have been applied to environmental monitoring
challenges; we adapt STraTS to a spatio-temporal setting.

###  2.4 Interpretability of Deep Learning Models

Deep learning models are often considered as black-box models because they are
hard to interpret 2. However, in many applications, it is important to
understand the model’s decision making process 9. Some authors have proposed
methods to help interpret the results of deep learning models applied to time
series data. For example, SeFT uses the attention mechanism in its
architecture and the authors of the work showed that the attention weights can
be used to gain insights into the importance of input data, including multiple
variables. 24, inspired by 4 and 29, proposed an interpretable version of the
STraTS model, called STraTS-I, which uses an almost identical structure to
their STraTS model but instead uses encoded inputs directly to the output
layer as opposed to STraTS that uses the contextualized inputs to the output
layer. Their approach allows for the calculation of a contribution score for
each input observation towards the prediction, achieved by multiplying the
encoded input, attention weights, and output layer weights. This modification
aims to compensate accuracy for interpretability while both models have
similar computational complexity. We follow a similar simplification routine
in the creation of our SST-ANN approach explained in Section 3.3.

##  3 Proposed Methods

In this section we first define the problem followed by the details of the
general model architectures that we use to build SERT and SST-ANN to address
the problem. We then introduce a modification for encoding location
information in the models’ input data. Finally, we describe the masked loss
function that we use for training the models.

###  3.1 Problem Definition

We have a dataset $D=\\{(T_{j},Y_{j},M_{j})\\}_{j=1}^{J}$ where $T_{j}$ is a
multivariate time series consisting of triples (time, variable, value) written
as $\\{(t_{i},f_{i},v_{i}),i=1,\ldots,N\\}$. $Y_{j}$ is the values (and
associated variables) for a future time horizon for which we want the model to
forecast. $M_{j}$ is a binary vector indicating whether each of element in
$Y_{j}$ is observed for sample $j$ in the dataset. $M_{j}$ is used in the loss
function (see Section 3.5 for details) for masking the unobserved values in
the forecast window. A schematic of a sample from the dataset is shown in
Figure 2. The goal is to learn a model $G$ that maps $T_{j}$ to $Y_{j}$, i.e.,
$G(T_{j})=Y_{j}$, without imputing the missing values in $T_{j}$ or aligning
the time series.

Figure 2: An example of a sample of a spatiotemporal dataset to be used for
training our proposed models where $h$ is the desired forecast horizon.

###  3.2 SERT

We first describe the data encoding scheme and then the model architecture
including an encoder network and a linear layer. The schematic diagram of the
model is shown in Figure 3. We will slightly modify this structure to show an
alternative approach for encoding the location information in Section 3.4.

Figure 3: Schematic diagram of the SERT model.

Data Encoding Scheme

The input data to our model is a multivariate time series dataset $T$ that
consists of $N$ time series, each of which is associated with a time series
variable $f$ which is a sequence of observation values $v$’s. Accordingly, an
individual data point $i$ is represented as a triplet $(t_{i},f_{i},v_{i})$
where $t_{i}$ is time, $f_{i}$ is the variable name and $v_{i}$ is the
measured value of the data point. To use the triplet in the model, we encode
each component into an embedding and then add the embeddings together. Let
$e_{i}^{f}\in R^{d}$ be the embedding of the variable name $f_{i}$ which can
be encoded similar to words using a lookup table, $e_{i}^{t}\in R^{d}$ be the
embedding of the time index $t$ which can be encoded using a continuous value
embedding (CVE) scheme which is a one-to-many feed forward neural network 24
and $e_{i}^{v}\in R^{d}$ be the embedding of the value $v_{i}$ which can also
be encoded using the CVE. The embedding of the triplet $i$ is then defined as
$e_{i}=e_{i}^{f}+e_{i}^{t}+e_{i}^{v}$. The size of the embedding vector $d$ is
a hyperparameter of the model.

Encoder Network

Similar to the well-known BERT model 7, the main component of our model is the
encoder part of the transformer model introduced by 26. Since transformers
have become very common, we omit the details of the architecture and refer the
reader to 26 for the full description. Intuitively, we can think of the
encoder network as layers that take the triplet embeddings of the input data
and transform them into contextualized embeddings that capture the long-range
dependencies within a time series as well as cross dependencies between
different time series.

Linear Layer

After obtaining the contextualised embeddings of the input data using the
encoder network, we then flatten the embeddings and apply a linear layer to
them to generate the predictions. The linear layer is a feed-forward network
with a single hidden layer and a ReLU activation function.

###  3.3 SST-ANN

The SST-ANN model is a simplified version of the SERT model that consists of
only the triplet encoding and a linear layer to the output with no transformer
structure in between. SST-ANN first encodes the input data using the triplet
encoding scheme and then uses the embeddings as the input to single layer feed
forward network to generate the predictions. Since there is no transformer
structure in between, the SST-ANN model is much faster than the SERT model
and, using the embeddings and the weights of the linear layer, we can compute
a contribution score for each observation to the final prediction. This is
useful for interpretability and variable importance analysis. More formally
the output of the model can be expressed as follows:

$\hat{y}_{k}=\sum_{i=1}^{N}c_{i}+b,\mbox{ with }c_{i}=W_{ik}^{T}\cdot e_{i},$
(1)

where $\hat{y}_{k}$ is the prediction of variable $k$, $c_{i}$ is the
contribution of the triplet $(t_{i},f_{i},v_{i})$ to the prediction, $N$ is
the number of observations in the input sample, $b$ is the bias term, $e_{i}$
are the embeddings of the triplet and $W_{ik}$ is the vector of output weights
associated with the embedding $e_{i}$ and the target variable $k$.

Using the contribution scores, we can define a variable importance index. We
first calculate the average contribution value of all observations belonging
to the same variable as the average contribution of that variable. This
calculation can be performed for a single sample to gain insights into the
importance of the predictor variable for a specific target prediction, or for
multiple samples used in multiple predictions to obtain an overall
understanding of the predictor variable’s importance in general. Next, we
compute the importance of each variable by normalizing the absolute value of
the average contribution values for the variables. More formally, we can
express this as:

$I_{k}=\frac{|\bar{c}_{k}|}{\sum_{k=1}^{K}|\bar{c}_{k}|}\times 100$ (2)

where $I_{k}$ is the importance (in percentage) and $\bar{c}_{k}$ is the
average contribution value of the variable $k$.

###  3.4 Location Encoding

We consider two different approaches to encode the location information in the
input data. The first approach is to encode the location information together
with the variable name $f_{i}$ in the triplet encoding scheme. For example,
our naming scheme for the variables can be
$f_{i}=\\{location\\}_{i}\cdot\\{variable\\}_{i}$ where $\\{location\\}_{i}$
is the location of the time series $i$ and $\\{variable\\}_{i}$ is the
variable name of the time series $i$ (e.g. Tolka.Turbidity). This approach
uses the exact same architecture explained in Section 3.2. The second approach
is to encode the location information separately from the variable name. This
way the input time series are all assumed to arise from the same location and
the location embedding is concatenated to the contextualized embeddings before
the linear layer is used for prediction. In this approach, we need to
structure the dataset such that all the time series from the same location are
grouped together. Formally, we can define the grouped dataset as
$D^{\prime}=\\{(T_{j}^{L},Y_{j}^{L},M_{j}^{L})\\}_{j=1}^{J^{\prime}}$ where
$L\in S$ is the location and $S$ is the set of all locations. This approach
needs a minimal modification to the previously described architecture and its
schematic diagram is shown in Figure 4.

Figure 4: Schematic diagram of the SERT model with a separated location
embedding layer.

The first approach is similar to how the Spacetimeformer model 11 encodes the
location information while the second approach is similar to how the STraTS
model 24 encodes the non-temporal information (patient demographics in their
work).

###  3.5 Masked Loss Function

We use a masked Mean Squared Error (MSE) loss function to train our models.
Masking in the loss function is used to handle the missing values in the
output data. The masked MSE loss function is defined as follows:

$\mathcal{L}=\frac{1}{\left|J\right|}\sum_{j=1}^{J}\sum_{k=1}^{K}m_{k}^{j}\left(\tilde{\mathbf{y}}_{k}^{j}-\mathbf{y}_{k}^{j}\right)^{2}$
(3)

where $\mathbf{y}_{k}^{j}$ is the ground truth, $\tilde{\mathbf{y}}_{k}^{j}$
is the predicted value, and $m_{k}^{j}$ is the mask value of the target
variable $k$ in sample $j$.

##  4 Experiments

In this section, we will describe the experiments we conducted to evaluate the
performance of our proposed models. We evaluated our models using both a
simulated dataset and a real-world dataset. The primary objective of the
simulation experiment is to investigate how the models perform under different
sparsity levels. The real-world experiment aimed to serve as a proof of
concept for the models’ ability to conduct spatiotemporal multi-step ahead
forecasting in real-world scenarios.

To assess the effectiveness of our models, we compared them with a baseline
Naive forecaster model, which simply uses the present hour’s observation as
the next hour forecast. We also compare our models against the LSTM model and
the STraTS model. Since the Naive forecaster and LSTM model cannot handle
missing values, we first imputed these values using a forward filling method
25 p. 16. The source code for our experiments is available at
https://github.com/Aminsn/SERT2023.

###  4.1 Sparsity Analysis

We first simulated a dataset that consists of 16 time series each with 40,000
observations. We denote $Y_{t}\in R^{16}$ as the vector of observations at
time $t$ generated from the following process:

$Y_{t}=2+0.4Y_{t-1}+X_{t}+s_{t},\;s_{t}\sim MVN(0,\Sigma)$ (4)

where $s_{t}$ is spatial random effect with mean zero and variance-covariance
matrix $\Sigma$ that is generated with:

$\Sigma=U\cdot U^{T},\ U\in R^{16\times 16},\;U_{i,j}\sim Uniform(-1,1),$ (5)

and $X_{t}\in R^{16}$ is the vector of temporal effects generated as:

$\displaystyle X_{t}^{T}$ $\displaystyle=$
$\displaystyle\left[10\sin(p_{1}t),\cos(p_{2}t),p_{3}t,-p_{3}t+10\sin(p_{1}t),5\sin(p_{2}t),12\cos(p_{2}t),7\sin(p_{2}t),8\cos(p_{2}t),\right.$
(6)
$\displaystyle\left.2\sin(p_{2}t),3\cos(p_{2}t),12\sin(p_{2}t),18\cos(p_{2}t),4\sin(p_{2}t),15\cos(p_{2}t),11\sin(p_{2}t),10\cos(p_{2}t)\right]$

where $p_{1}=0.005$, $p_{2}=0.0005$ and $p_{3}=0.002$.

We use the first 37,000 time steps to train the models and the remaining 3,000
time steps to evaluate the performance of them. We used the model structure
shown in Figure 3 to train our proposed models. We trained all models for 1
step ahead forecasting using the previous 10 hours of observations. We
consider five different sparsity levels to fit the models. Accordingly, we
remove $n\%$ of the observations randomly for
$n=\\{0\%,20\%,40\%.60\%,80\%\\}$. We use the root mean squared error (RMSE)
as the evaluation metric. The results are presented in Figure 5.

Figure 5: a) Performance of all the models used on the simulated dataset. b)
Comparison of SST-ANN, STraTS, and SERT zoomed in.

###  4.2 Real Dataset; Environmental Monitoring in Dublin bay, Ireland

The dataset includes hourly measurements from 2017-01-01 to 2021-12-31. An
example of the data is shown in Figure 1. The locations of the data are shown
in Figure 6. We use the data from the first four years to train the models and
the last year to evaluate their performance. We train the models using the
previous 10 hours of observations as the input and forecasting seven hours
ahead (forecasting only 1 or 2 hours ahead is a relatively trivial task, and
forecasting >12 hours ahead reduces the performance of models that cannot
account for non-seasonality). We use the same evaluation metric as in the
simulated dataset. We tried both location encoding approaches (explained in
Section 3.4) with our proposed models on the real-world dataset and found that
the second approach, depicted in Figure 4, performed better and here we only
report the results of the superior approach. The results are presented in
Table 1.

Figure 6: Buoys measuring various environmental variables in Dublin Bay. The 4 main environmental variables (Turbidity, Salinity, Dissolved Oxygen, Temperature) are available at the number buoys. The weather measurements (Rainfall, Wind Speed) come from Dublin Airport. The water level variable comes from Howth Harbour. Model | Dissolved Oxygen | Precipitation | Salinity | Temperature | Turbidity | Water Level | Wind Speed
---|---|---|---|---|---|---|---
Naive | 0.64 | 1.19 | 1.22 | 0.21 | 0.93 | 1.87 | 0.8
LSTM | 0.65 | 0.89 | 0.8 | 0.41 | 0.81 | 0.5 | 0.82
STraTS | 0.53 | 0.88 | 0.7 | 0.18 | 0.72 | 0.38 | 0.76
SERT (ours) | 0.49 | 0.88 | 0.67 | 0.18 | 0.72 | 0.4 | 0.73
SST-ANN (ours) | 0.51 | 0.88 | 0.73 | 0.45 | 0.88 | 0.63 | 0.79
Table 1: RMSE of the models for 7 hour ahead forecasting of the 7
environmental variables in Dublin bay.

We utilized the same computational resources (a single NVIDIA P100 GPU) to
train the models. The specifications and speed performance details of the
models are reported in the Table 2.

Model | Specifications | Sec/epoch
---|---|---
Naive | — | —
LSTM | units = 60 | 2
STraTS | d = 60, # of heads = 6, k = 6 | 230
SERT | d = 60, # of heads = 6, k = 6 | 40
SST-ANN | d = 60 | 9
Table 2: Computational specifications of the fitted models.

###  4.3 Interpretability of the SST-ANN model

The results of the variable importance for the real-world dataset experiment
are presented in Figure 7. The sign of the average contribution score
$\bar{c}_{j}$ (explained in Section 3.3) is multiplied by the importance index
$I_{j}$ to give an insight into the direction of the average contribution of
the predictor variable to the prediction of the target variable. In this
example, we only consider the contributions of water level, temperature, wind
speed and precipitation to predictions of turbidity, disolved oxygen and
salinity, since we know that the former variables could affect the latter
variables but not vice versa. According to the results, temperature followed
by precipitation are the most important variables in predicting the target
variables.

Figure 7: Variable importance of the selected variables in predicting the
target variables.

##  5 Conclusion

In this paper, we proposed two novel models for spatio-temporal forecasting
called SERT and SST-ANN. SERT is a transformer based model while SST-ANN is a
simple ANN model combined with triplet encoding of the input data.
Furthermore, we showed that STraTS, a model originally developed for sparse
and irregularly sampled clinical time series classification, can be used for
spatio-temporal forecasting, especially when missing values are present in the
data. The proposed approaches do not require aggregation or missing value
imputation techniques, and avoid the problems introduced by such methods. We
evaluated the performance of the proposed models on a simulated dataset with
varying levels of sparsity and showed that in general increasing sparsity has
a negative effect in the performance of all the models, but SERT followed by
STraTS and SST-ANN are more robust to the increase in sparsity. We also
evaluated the performance of the proposed models on a real-world dataset of
environmental variables in Dublin Bay, Ireland. The results indicate that SERT
outperformed the other models in 7-hour ahead forecasting for 6 out of the 7
variables, with 3 of them being on par with STraTS while demonstrating
significantly faster performance. We then showed how SST-ANN can be used to
interpret the predictions of the model by calculating and using the
contribution score of the input data to develop an importance index using the
average contribution scores.

We introduced two different methods to encode the location information in our
proposed models, including encoding the time series variable name with the
location name simultaneously and encoding the location name separately.
However, neither of these methods takes into account the distance between the
locations, which is a limitation of our work. We believe future research
should focus on incorporating this information into the models, as it has the
potential to improve forecasting performance and be utilized for
spatiotemporal interpolation tasks.

## Acknowledgement

We would like to express our sincere gratitude to Dublin Port Company for
providing us with the real dataset. This work was supported by an SFI
Investigator award (16/IA/4520).

## References

  * Bahdanau . 2014 bahdanau2014Bahdanau, D., Cho, K. Bengio, Y.  2014\.  Neural machine translation by jointly learning to align and translate Neural machine translation by jointly learning to align and translate.  arXiv preprint arXiv:1409.0473.
  * Buhrmester . 2021 buhrmester2021analysisBuhrmester, V., Münch, D. Arens, M.  2021\.  Analysis of explainers of black box deep neural networks for computer vision: A survey Analysis of explainers of black box deep neural networks for computer vision: A survey.  Machine Learning and Knowledge Extraction34966–989.
  * Cho . 2014 cho2014Cho, K., Van Merriënboer, B., Bahdanau, D. Bengio, Y.  2014\.  On the properties of neural machine translation: Encoder-decoder approaches On the properties of neural machine translation: Encoder-decoder approaches.  arXiv preprint arXiv:1409.1259.
  * Choi . 2016 choi2016Choi, E., Bahadori, MT., Sun, J., Kulas, J., Schuetz, A. Stewart, W.  2016\.  Retain: An interpretable predictive model for healthcare using reverse time attention mechanism Retain: An interpretable predictive model for healthcare using reverse time attention mechanism.  Advances in neural information processing systems29.
  * Deng . 2022 deng2022spatiotemporalDeng, L., Chen, K., Liu, Z., Wu, B., Chen, Z. He, S.  2022\.  Spatiotemporal variation evaluation of water quality in middle and lower Han River, China Spatiotemporal variation evaluation of water quality in middle and lower han river, china.  Scientific Reports1211–14.
  * Devlin . 20181 devlin2018Devlin, J., Chang, MW., Lee, K. Toutanova, K.  20181\.  Bert: Pre-training of deep bidirectional transformers for language understanding Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805.
  * Devlin . 20182 Devlin2019bertDevlin, J., Chang, MW., Lee, K. Toutanova, K.  20182\.  Bert: Pre-training of deep bidirectional transformers for language understanding Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805.
  * Dosovitskiy . 2020 dosovitskiy2020imageDosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T.others  2020\.  An image is worth 16x16 words: Transformers for image recognition at scale An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929.
  * Du . 2019 du2019techniquesDu, M., Liu, N. Hu, X.  2019\.  Techniques for interpretable machine learning Techniques for interpretable machine learning.  Communications of the ACM63168–77.
  * Graves . 2013 graves2013Graves, A., Mohamed, Ar. Hinton, G.  2013\.  Speech recognition with deep recurrent neural networks Speech recognition with deep recurrent neural networks.  2013 IEEE international conference on acoustics, speech and signal processing 2013 ieee international conference on acoustics, speech and signal processing ( 6645–6649).
  * Grigsby . 2021 grigsby2019spatiotemporalGrigsby, J., Wang, Z. Qi, Y.  2021\.  Long-range transformers for dynamic spatiotemporal forecasting Long-range transformers for dynamic spatiotemporal forecasting.  arXiv preprint arXiv:2109.12218.
  * Hamdi . 2022 hamdi2022spatiotemporalHamdi, A., Shaban, K., Erradi, A., Mohamed, A., Rumi, SK. Salim, FD.  2022\.  Spatiotemporal data mining: a survey on challenges and open problems Spatiotemporal data mining: a survey on challenges and open problems.  Artificial Intelligence Review1–48.
  * Hochreiter Schmidhuber 1997 hochreiter1997Hochreiter, S. Schmidhuber, J.  1997\.  Long short-term memory Long short-term memory.  Neural computation981735–1780.
  * Horn . 2020 horn2020SeFTHorn, M., Moor, M., Bock, C., Rieck, B. Borgwardt, K.  2020\.  Set functions for time series Set functions for time series.  International Conference on Machine Learning International conference on machine learning ( 4353–4363).
  * Li . 2018 li2018independentlyLi, S., Li, W., Cook, C., Zhu, C. Gao, Y.  2018\.  Independently recurrent neural network (indrnn): Building a longer and deeper rnn Independently recurrent neural network (indrnn): Building a longer and deeper rnn.  Proceedings of the IEEE conference on computer vision and pattern recognition Proceedings of the ieee conference on computer vision and pattern recognition ( 5457–5466).
  * Lim Zohren 2021 lim2021timeLim, B. Zohren, S.  2021\.  Time-series forecasting with deep learning: a survey Time-series forecasting with deep learning: a survey.  Philosophical Transactions of the Royal Society A379219420200209.
  * Liu Lapata 2019 liu2019Liu, Y. Lapata, M.  2019\.  Text summarization with pretrained encoders Text summarization with pretrained encoders.  arXiv preprint arXiv:1908.08345.
  * McDermott Wikle 2017 mcdermott2017McDermott, PL. Wikle, CK.  2017\.  An ensemble quadratic echo state network for non-linear spatio-temporal forecasting An ensemble quadratic echo state network for non-linear spatio-temporal forecasting.  Stat61315–330.
  * Radford . 2018 radford2018Radford, A., Narasimhan, K., Salimans, T., Sutskever, I. .  2018\.  Improving language understanding by generative pre-training Improving language understanding by generative pre-training.
  * Shoari Nejad . 2022 shoarinejad2022Shoari Nejad, A., McCarthy, GD., Kelleher, B., Grey, A. Parnell, A.  2022\.  Vector Time Series Modelling of Turbidity in Dublin Bay Vector time series modelling of turbidity in dublin bay.  arXiv preprint arXiv:2209.06880.
  * Stekhoven Bühlmann 2012 stekhoven2012missforestStekhoven, DJ. Bühlmann, P.  2012\.  MissForest—non-parametric missing value imputation for mixed-type data Missforest—non-parametric missing value imputation for mixed-type data.  Bioinformatics281112–118.
  * Sun . 2019 sun2019Sun, C., Qiu, X., Xu, Y. Huang, X.  2019\.  How to fine-tune bert for text classification? How to fine-tune bert for text classification?  Chinese Computational Linguistics: 18th China National Conference, CCL 2019, Kunming, China, October 18–20, 2019, Proceedings 18 Chinese computational linguistics: 18th china national conference, ccl 2019, kunming, china, october 18–20, 2019, proceedings 18 ( 194–206).
  * Sutskever . 2014 sutskever2014sequenceSutskever, I., Vinyals, O. Le, QV.  2014\.  Sequence to sequence learning with neural networks Sequence to sequence learning with neural networks.  Advances in neural information processing systems27.
  * Tipirneni Reddy 2022 Tipirneni2022STraTSTipirneni, S. Reddy, CK.  2022\.  Self-supervised transformer for sparse and irregularly sampled multivariate clinical time-series Self-supervised transformer for sparse and irregularly sampled multivariate clinical time-series.  ACM Transactions on Knowledge Discovery from Data (TKDD)1661–17.
  * Van Buuren 2018 van2018flexibleVan Buuren, S.  2018\.  Flexible imputation of missing data Flexible imputation of missing data.  CRC press.
  * Vaswani . 2017 vaswani2017Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, AN.Polosukhin, I.  2017\.  Attention is all you need Attention is all you need.  Advances in neural information processing systems30.
  * Wikle 2015 wikle2015Wikle, CK.  2015\.  Modern perspectives on statistics for spatio-temporal data Modern perspectives on statistics for spatio-temporal data.  Wiley Interdisciplinary Reviews: Computational Statistics7186–98.
  * Yu . 2023 yu2023Yu, M., Masrur, A. Blaszczak-Boxe, C.  2023\.  Predicting hourly PM2. 5 concentrations in wildfire-prone areas using a SpatioTemporal Transformer model Predicting hourly pm2. 5 concentrations in wildfire-prone areas using a spatiotemporal transformer model.  Science of The Total Environment860160446.
  * Zhang . 2020 zhang2020Zhang, X., Qian, B., Cao, S., Li, Y., Chen, H., Zheng, Y. Davidson, I.  2020\.  INPREM: An interpretable and trustworthy predictive model for healthcare Inprem: An interpretable and trustworthy predictive model for healthcare.  Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining Proceedings of the 26th acm sigkdd international conference on knowledge discovery & data mining ( 450–460).
  * Zhang . 2018 Zhang2018Zhang, X., Zhang, Q., Zhang, G., Nie, Z., Gui, Z. Que, H.  2018\.  A novel hybrid data-driven model for daily land surface temperature forecasting using long short-term memory neural network based on ensemble empirical mode decomposition A novel hybrid data-driven model for daily land surface temperature forecasting using long short-term memory neural network based on ensemble empirical mode decomposition.  International journal of environmental research and public health1551032.


(Yenugunti and Yau, 2020) Custom Trust Score;PBFT Private Permissioned 2020 A
Contract Based User-Centric Computational Tr… (Hu et al., 2020) Hyperledger
Fabric Not specified Private Permissioned 2020 A Framework of Blockchain-Based
Collaborative … (Li et al., 2020b) Not specified Not specified Private
Permissioned 2020 Blockchain-enabled Collaborative Intrusion Det… (Fan et al.,
2020) Ethereum Proof of Authority;PBFT Private Permissioned 2020 Distributed
Intrusion Detection System using B… (Kumar and Singh, 2020) Not specified Not
specified Not specified Not specified 2020 LegIoT: Ledgered Trust Management
Platform for… (Neureither et al., 2020) Hyperledger Sawtooth Not specified
Private Permissioned 2020 SOChain: A Privacy-Preserving DDoS Data Exchan… (Yeh
et al., 2020) Ethereum Proof of Work Private Permissioned 2020 TRIDEnT:
Towards a Decentralized Threat Indica… (Alexopoulos et al., 2020) Ethereum
Proof of Burn;Proof of Work Public Permissionless 2020 Blockchain-based
Federated Learning for Device… (Zhang et al., 2020) Ethereum Proof of Work
Private Permissionless 2020 Application of Blockchain within Aviation Cybe…
(Adhikari and Davis, 2020) Not specified Not specified Not specified Not
specified 2020 Blockchain as a solution to Drone Cybersecurity (Ossamah, 2020)
Not specified Not specified Private Permissioned 2020 Blockchain technology in
the future of busines… (Demirkan et al., 2020) Not specified Not specified Not
specified Not specified 2020 Suitability of Blockchain for Collaborative In…
(Dawit et al., 2020) Not specified Independent Not specified Not specified
2021 A Distributed Collaborative Entrance Defense F… (Guo et al., 2022) Not
specified Not specified Private Permissioned 2021 A Deep Blockchain Framework-
Enabled Collaborat… (Alkadi et al., 2020) Ethereum Proof of Work Private
Permissioned 2021 A Comprehensive Study of Anomaly Detection Sch… (Diro et
al., 2021) Independent Independent Not specified Not specified 2021
Distributed Watchdogs Based on Blockchain for … (Lee and Kwon, 2021) Ethereum
Proof of Authority Private Permissioned 2021 Toward a blockchain-based
framework for challe… (Li et al., 2021) Custom Proof of Identity Private
Permissioned 2021 SeDID: An SGX-enabled decentralized intrusion … (Liu et al.,
2021a) Custom Custom Private Permissioned 2021 Designing consensus algorithm
for collaborativ… (Winanto et al., 2021) Not specified Proof of Work;Proof of
Stake;Custom Not specified Not specified 2021 LDBT: A Lightweight DDoS Attack
Tracing Scheme… (Guo et al., 2021) Hyperledger Fabric PBFT Private
Permissioned 2021 Blockchain and Federated Learning for Collabor… (Liu et al.,
2021b) Ethereum Proof of Work;Proof of Accuracy Private Permissioned 2021
Blockchained Federated Learning for Threat Def… (Demertzis, 2021) Not
specified Not specified Not specified Not specified 2021 Cybersecurity of
Renewable Energy Data and App… (Cali et al., 2021) Not specified Not specified
Private Permissioned 2021 Secure exchange of cyber threat intelligence u…
(Pahlevan et al., 2021) Not specified Not specified Private Permissioned 2022
FolketID: A Decentralized Blockchain-Based Nem… (Chiu et al., 2022) Ethereum
Proof of Authority Private Permissioned 2022 Blockchain-Based Decentralized
Replay Attack D… (Ramanan et al., 2021) Ethereum Proof of Authority Private
Permissioned 2022 BlockCSDN: Towards Blockchain-Based Collaborat… (Li et al.,
2022) Not specified Not specified Private Permissioned 2022 CIDS:
Collaborative Intrusion Detection System… (Gurung et al., 2022) Hyperledger
Fabric Not specified Private Permissioned 2022 IDS-Chain: A Collaborative
Intrusion Detection… (Aljuhani, 2022) Not specified Not specified Private
Permissioned 2022 Design of Blockchain enabled intrusion detecti… (Saveetha
and Maragatham, 2022) Ethereum Not specified Private Permissioned 2022
Blockchain-Enabled Intrusion Detection and Pre… (Alevizos et al., 2022)
Hyperledger Fabric Not specified Private Permissioned 2022 FIDChain: Federated
Intrusion Detection System… (Ashraf et al., 2022) Custom Round-robin Private
Permissioned 2022 Privacy-preserved Cyberattack Detection in Ind… (Abdel-
Basset et al., 2022) Algorand Pure Proof of Stake Private Permissioned 2022
CGAN-Based Collaborative Intrusion Detection f… (He et al., 2022) Not
specified Proof of Authority Private Permissioned 2022 A Framework for
Verifiable and Auditable Feder… (Santin et al., 2022) Hyperledger Fabric Not
specified Private Permissioned 2022 InFEDge: A Blockchain-based Incentive
Mechanis… (Wang et al., 2022) FISCO BCOS Not specified Private Permissioned
2022 An Intelligent Platform for Threat Assessment … (Kolokotronis et al.,
2022) Not specified Not specified Not specified Not specified 2022 Blockchain-
based Trusted Traffic Offloading in… (Tang et al., 2022) Not specified
Enhanced PBFT Private Permissioned 2022 A Distributed Intrusion Detection
System to De… (Kumar et al., 2022) Not specified Not specified Not specified
Not specified 2023 FusionFedBlock: Fusion of Blockchain and Feder… (Singh et
al., 2023) Ethereum Proof of Authority Private Permissioned

###  A.2. Research groups and venues

Figure 16 shows the number of papers per research group per year. There does
not seem to be any particular trend when it comes to the research groups, and
the figure indicates a shift in research groups over the years.

Figure 17 represents the number of papers per venue per year. As with the
research groups, there is a shift in venues those works are published in over
the years. Yet another bit of evidence that the field is still new and
fragmented.

Figure 16. Number of papers per research group per year. Figure 17. Number of
papers per venue per year.