# Explicit Context Integrated Recurrent Neural Network for Sensor Data
Applications

Rashmi Dutta Baruah
Department of Telematic Engineering
Universidad Carloss III de Madrid
Avda. de la Universidad, 30
Laganès, Madrid, 28911, Spain
<EMAIL_ADDRESS>
& Mario Muñoz Organero
Department of Telematic Engineering
Universidad Carloss III de Madrid
Avda. de la Universidad, 30
Laganès, Madrid, 28911, Spain
<EMAIL_ADDRESS>
Department of Computer Science and Engineering, IIT Guwahati, Guwahati 781039,
Assam, India

###### Abstract

The development and progress in sensor, communication and computing
technologies have led to data rich environments. In such environments, data
can easily be acquired not only from the monitored entities but also from the
surroundings where the entity is operating. The additional data that are
available from the problem domain, which cannot be used independently for
learning models, constitute explicit context. Such context, if taken into
account while learning, can potentially improve the performance of predictive
models. Typically, the data from various sensors are present in the form of
time series. Recurrent Neural Networks (RNNs) are preferred for such data as
it can inherently handle temporal context. However, the conventional RNN
models such as Elman RNN, Long Short-Term Memory (LSTM) and Gated Recurrent
Unit (GRU) in their present form do not provide any mechanism to integrate
explicit contexts. In this paper, we propose a Context Integrated RNN (CiRNN)
that enables integrating explicit contexts represented in the form of
contextual features. In CiRNN, the network weights are influenced by
contextual features in such a way that the primary input features which are
more relevant to a given context are given more importance. To show the
efficacy of CiRNN, we selected an application domain, engine health
prognostics, which captures data from various sensors and where contextual
information is available. We used the NASA Turbofan Engine Degradation
Simulation dataset for estimating Remaining Useful Life (RUL) as it provides
contextual information. We compared CiRNN with baseline models as well as the
state-of-the-art methods. The experimental results show an improvement of 39%
and 87% respectively, over state-of-the art models, when performance is
measured with RMSE and score from an asymmetric scoring function. The latter
measure is specific to the task of RUL estimation.

_Keywords_ Context dependent recurrent neural network $\cdot$ Context
integrated recurrent neural network $\cdot$ Contextual Gated Recurrent Unit
$\cdot$ Context sensitive recurrent neural network $\cdot$ Machine prognostics
and health management $\cdot$ Predictive maintenance $\cdot$ Remaining useful
life estimation

##  1 Introduction

The Internet of Things (IoT) has led to many ’smart’ innovations such as smart
homes, smart cities, smart factories, smart agriculture, to name a few. The
primary contributors to IoT enabled smart creations are advanced sensor,
communication, and computing technologies. The sensors play a crucial role as
IoT devices depend on them to get the data from the monitored entities. The
monitored entity can range from a person (for example, to get health
parameters), environment (to get air, water, or soil parameters), to a large
industrial setup (to get parameters associated to various processes or
equipment). However, acquiring and storing data is not sufficient, the data
needs to be analysed and interpreted to extract knowledge, which can further
be used for decision making and automation at various levels. Machine Learning
(ML) offers a way to achieve this through predictive analytics. Particularly,
Deep Learning (DL) has gained a lot of attention due to huge amount of data
generated by smart environments (homes, cities, industries etc.) (Chen and
Lin, 2014), (Najafabadi et al., 2015). Typically, the data received through
sensors from such domains are time series data. Recurrent Neural Networks
(RNNs) can be a preferable approach as they inherently capture temporal
context available in the time series or sequential data Karim et al. (2019),
Gers et al. (2001), Malhotra et al. (2015).

One of the early and widely known RNNs, Elman network (Elman, 1990),
introduced the notion of context neurons that function as network’s memory and
allow implicitly representing the temporal context of the input data. The
input layer is augmented with a context layer. The context units are
initialized to a specific value. At a particular time step $t$, the hidden
units are activated by both input layer units and context layer units. The
hidden units with feed forward connections, in turn, activates the output
layer units. The hidden units also activate the context unit through feed-back
connections which have unit weight. Thus, in the next time step $t+1$, the
context units have the hidden unit values of previous time step $t$. The
hidden unit patterns is the context that is saved in the context units. The
hidden units map both the input and the previous state to desired output.
Thus, the internal representations developed in the process are sensitive to
the temporal context (Elman, 1990). In this paper, we treat the notion of
context in a different way, and the focus is not on the temporal context or
the contexts that are generated within the network from input and/or output
signals as we already have various forms of RNNs that elegantly capture them.

The notions of context, context-aware, context-dependent, and context-
sensitive are widely used across various domains. Several definitions of these
terms, particularly for the term context, have appeared in the literature.
During the early 90’s these concepts gained huge popularity within the
research community of pervasive and ubiquitous computing, which evolved from
mobile and distributed computing. As defined by Abowd et al.(Abowd et al.,
1999), context is any information that can be used to characterise the
situation of an entity. An entity is a person, place, or object that is
considered relevant to the interaction between a user and an application,
including the user and applications themselves. From machine learning
perspective, we are concerned with context that is available as data from the
problem domain but not necessarily independently used for learning. The
context data may influence the decision by improving the model but may not be
involved directly in learning. For example, in medical diagnosis, the
patient’s gender, age, and weight are often available, which can be considered
as contextual data. For clarity, we adopt the distinction between primary
features and contextual features as described in (Turney, 1996). Primary
features are useful for machine learning task (classification/regression) when
considered in isolation, irrespective of other features. Contextual features
are not useful in isolation, but can be useful when combined with primary
features. For example, for a gas turbine engine diagnosis problem, thrust,
temperature, and pressure are primary features, whereas the weather conditions
such as ambient temperature and humidity can be considered as contextual
features. We also distinguish between explicit and implicit context. The
context that can be captured from the environment where primary data is
acquired is considered as explicit. The above-mentioned examples of medical
diagnosis and gas turbine engine diagnosis have explicit contexts represented
by contextual features. On the other hand, implicit context is present in the
primary data itself. For example, for image classification, many approaches
consider the neighbouring pixels as context while processing a particular
pixel. Similarly, in Natural Language Processing (NLP) task, the neighbouring
words can be considered as context. Finally, the temporal sequence of the
instances in a data set is another example of implicit context. For
simplicity, in rest of the paper, we will be referring ’explicit context’ as
’context’.

In several problem domains, data from operating environment is often captured
along with the data from monitored system (or entity), in other words, the
contextual information is available. Yet, during learning, the focus is on the
internal state data of the monitored entity and the external data from the
operating environment is not fully considered or even ignored. As mentioned
earlier RNNs (including Long Short Term Memory-LSTM and Gated Recurrent Unit-
GRU) can handle the temporal context very well. However, there is no explicit
way of leveraging the contextual information. Usually, the feature space
comprising of primary features is expanded with contextual features, and the
models treat them in the same manner as primary features.

In this paper, we propose an approach that integrates explicit contexts to RNN
and enhances its potentials. We refer to the the resulting RNN as Context
Integrated RNN (CiRNN) which takes into account implicit temporal context as
well as explicit context (Dutta Baruah and Muñoz Organero, 2023, accepted). In
the proposed approach, the contextual features are used to weight the primary
features such that features that are more relevant in a given context are
assigned more importance. This is also termed as contextual weighting (Turney,
1996). This work draws inspiration from the work of Ciskowski and Rafajlowicz
(Ciskowski and Rafajlowicz, 2004) where they introduced Context-Dependent
Feed-Forward Neural Networks. We demonstrate the efficacy of CiRNN by applying
it to the domain of predictive maintenance where contextual information is
available through various sensors.

The rest of the paper is organised as follows. In the next section we briefly
discuss methods where context is added to RNN in various ways. In section 3,
we discuss the architecture and learning of the proposed context integrated
RNN with GRU. Section 4, discusses the experiments and results achieved.
Finally, section 5 concludes the paper.

##  2 Related Work

In the past decade, many researchers have considered integrating context to
RNNs. These methods largely appear in the literature of NLP and recommender
systems. In this section, we briefly discuss various RNN architectures that
leverage context information for better performance in a given task.

Mikolov and Zweig (Mikolov and Zweig, 2012) proposed a context dependent
recurrent neural network language model (CDRNNLM) that extends the basic
recurrent neural network language model (RNNLM) by introducing context.
Context is a real valued vector that is computed based on the sentence history
using Latent Dirichlet Allocation (LDA). The architecture introduces a feature
layer in addition to the input, output, and hidden layer. The feature layer
with associated weights is connected to both hidden and output layers. It
provides the context i.e. complementary information to the input word vector,
for example, features representing topic information. The conventional
approach is to partition the data and to develop multiple topic specific
models. However, through CDRNNLM a topic-conditioned RNNLM is attained that
avoids the data partitioning. They demonstrated that CDRNNLM improves the
performance over state-of-the art methods using benchmark dataset.

In (Wang and Cho, 2016), a method is proposed that incorporates corpus-level
discourse information into language modelling and the model is referred to as
larger-context language model. The effect of context is modeled by a
conditional probability where the probability of the next words in a given
sentence is determined using previous words in the same sentence, and a
context sentence that consists of one or more sentences of arbitrary length.
To realize this, a conditional LSTM is proposed which is implemented in two
ways, early fusion and late fusion. In early fusion, the context vector
representation of preceding sentences is simply considered as an input at
every time step. Late fusion tries to maintain separately the dependencies
within the sentence being modelled (intra-sentence dependencies) and those
between the preceding sentences and the current sentence (inter-sentence
dependencies). The existing LSTM memory cell $c_{t}$ models the intra-sentence
dependency. The inter-sentence dependency is modeled through the interaction
between $c_{t}$ and the context vector. First, a controlled context vector
$r_{t}$ is determined using $c_{t}$ and the context vector. The vector $r_{t}$
represents the amount of influence of context or preceding sentences. Finally,
the controlled context vector is used to compute the output of the LSTM layer.
The experimental results showed that later fusion outperformed early fusion.

A Language Model that uses a modified version of the standard bidirectional
LSTM called contextual bidirectional LSTM (cBLSTM) is proposed in (Mousa and
Schuller, 2017) for sentiment analysis. cBLSTM predicts a word given the full
left and right context while excluding the predicted word itself from the
conditional dependence. This is achieved by introducing a forward and a
backward sub-layer. The encoded input consists of sentence start symbol and
all the words before the sentence end symbol. It is given to the forward sub-
layer and the forward hidden states predict the words between start and the
end symbol. The backward sub-layer does the reverse operation. When the two
sub-layers are interleaved it results in prediction of any target word using
the full left and right contexts. The obtained results on benchmark dataset
show that cBLSTM language model outperforms both LSTM and BLSTM based models.
In (Smirnova and Vasile, 2017), a family of Contextual RNNs (CRNNs) is
presented that leverages contextual information for item sequence modeling for
next item prediction in a recommender system. Apart from order of items, the
model considers other available information about user item interaction such
as type of the interaction, the time gaps between events and time of
interaction as context. The two proposed CRNN architectures are compared with
non-contextual models and the results show that CRNNs significantly
outperforms all of the other methods on two e-commerce datasets.

Tran et al. (Tran et al., 2018) proposed a Context-dependent Additive
Recurrent Neural Network (CARNN) for the problem of sequence mapping in NLP.
The learning process considers the primary source of information as the given
text, for example, the history of utterance in a dialog system or the sequence
of words in language modeling. The previous utterance in dialog system or the
discourse information in language modeling is taken as context. The CARNN
units consist of two gates (update gate and reset gate) that regulate the
information from the input. The gates are functions that depend on global
context, word embedding (input), and previous hidden vector. In (Wenke and
Fleming, 2019), a RNN architecture named as Contextual RNN is discussed where
contextual information is used to parameterize the initial hidden state.
Typically, the hidden state of RNN is initialised to a default constant, most
commonly to zero. In this paper, the initial hidden state is conditioned on
contextual information. The contextual information can be acquired from the
input sequence or from a coarse image with multiple objects related to the
task. The authors show that Contextual RNN initialised with contextual
information from input sequence improves the performance for an associative
retrieval task compared to the baseline with zero initialisation.

Next, we consider recent literature in various other domains where sensory
data is used as a context for RNN models. In (Batbaatar et al., 2018), a LSTM
model is used to predict appliances energy. Energy usage of home appliances is
highly dependent on weather conditions. In this work, two different datasets
are combined with the aim of introducing contextual features to the LSTM
model. The contextual features are obtained from the first dataset that
consists of house temperature and humidity measures for a period of 4.5
months. The second dataset provides the measurements of individual household
electric power consumption over a period of 47 months. The work focused on
identifying the best predictors for the prediction task and did not attempt to
adapt the model for incorporating contextual information. Lee and Hauskrecht
(Lee and Hauskrecht, 2019), proposed a LSTM-based model for predicting event
occurrence in clinical event time series with events such as medication
orders, lab tests and their results, or physiological signals. The proposed
model considers the usual abstracted information from the past through the
hidden states, and also recent event occurrence information. The recent event
at the current time step is represented as a multi-hot vector which is
linearly transformed and added to the LSTM model. The output from the model
depends on both the hidden state and the current event information. The
authors demonstrated through experimental results with MIMIC-III clinical
event data Johnson et al. (2016) that by combining recent event information
and the summarised past events through LSTM hidden states improves the
prediction over models that do not incorporate the former information.

In (Belhadi et al., 2020), the task of long-term traffic flow (traffic flow
over long interval) prediction is performed using a RNN that considers inputs
from multiple data sources. The RNN uses, flow information, temporal
information, and contextual information. The type of the day represents the
contextual feature. For example, weekend or regular and event or no-event day
(celebration days). Binary values are used for representation, 0 is used for
weekend and event day and 1 is used for regular and non-event days. In this
work also no emphasis is given on contextual learning. It simply extends the
input feature vector to accommodate the contextual features. Ma et al. (Ma et
al., 2021) also proposed a contextual convolutional RNN for long-term (daily)
traffic flow forecasting. The traffic flow data is represented in a form such
that it is similar to a sequence of square images which is given as input to
the CNN. The CNN is used to extract the inter-and intra-day traffic pattern.
The output of CNN is given as input to a LSTM network. Here, four contextual
features are used, day of the week, season, weather, and holiday. The
contextual features are represented as one-hot encoding. Finally, contextual
features and the output from the LSTM are concatenated and given as input to a
linear dense layer for the prediction of traffic flow. The performance of the
model was evaluated using real data of Seattle city. The results show that the
proposed model provides better prediction particularly during high demand
periods. In (Qu et al., 2021), a Feature injected RNNs (Fi-RNN) is presented
to predict short-term traffic speed. The model consists of an RNN, an
autoencoder, and a feedforward neural network. The historical traffic flow
data is given as input to the RNN (LSTM) to learn the sequential pattern in
terms of sequential features. The contextual data is used by a sparse
autoencoder to obtain an expanded set of contextual features. The expanded
contextual features are restricted to [0,1] using a softmax function and this
value is used to weight the features from LSTM (the hidden states of LSTM).
Finally, these weighted feature vectors are used as input to the feed-forward
neural network to learn traffic patterns and predict the speed. The
experimental results from two real datasets show that Fi-RNN that incorporates
contextual features achieves improved accuracy.

To summarize, all the RNN models that are discussed in this section obtained
improved results by adding additional contextual information to the model. In
these approaches, the context information has been added primarily in three
ways to the RNN model. Let us denote the primary feature vector at time $t$ as
$\mathbf{x}_{t}\in\mathbb{R}^{(n_{x}\times 1)}$ and contextual feature vector
as $\mathbf{z}_{t}\in\mathbb{R}^{(n_{z}\times 1)}$. Firstly, most of the
approaches, integrate context with the primary input to get an extended input
feature vector. The extended vector can be represented as
$\mathbf{x}\;^{\prime}=[\mathbf{x}_{t},\mathbf{z}_{t}]$ where
$\mathbf{x}\;^{\prime}_{t}\in\mathbb{R}^{(n_{x}+n_{z})\times 1}$. Sometimes,
the new input vector is obtained multiplicatively as
$\mathbf{x}\;^{\prime}=[\mathbf{x}_{t}\odot\mathbf{z}_{t}]$. Note that in this
case, either the context feature or both primary and context features are
transformed such that they are of the same dimension. In the second manner,
context is introduced additively and provided as a separate input to the
hidden or output layer or both. The hidden state is computed as
$f(\mathbf{W}\mathbf{x}_{t}+\mathbf{U}\mathbf{h}_{t-1}+\mathbf{V}\mathbf{z}_{t})$
and when context is added to the output, it is computed as
$g(\mathbf{U}\mathbf{h}_{t}+\mathbf{V}\mathbf{z}_{t})$. Here,
$\mathbf{W},\mathbf{U},\mathbf{V}$ are the associated weights and $f,g$ are
the activation functions. Finally, the context is integrated multiplicatively
in the hidden unit as as
$f(\mathbf{W}\mathbf{x}_{t}\odot\mathbf{V}\mathbf{z}_{t}+\mathbf{U}\mathbf{h}_{t-1}\odot\mathbf{V}\mathbf{z}_{t})$
and the output as $g(\mathbf{U}\mathbf{h}_{t}\odot\mathbf{V}\mathbf{z}_{t})$.

As mentioned briefly in section 1, in contrast to the above approaches, in
this paper we integrate context such that the weights associated with the
primary input vector is dependent on the context. Accordingly, the hidden
state is given as,
$f(\mathbf{W}(\mathbf{z}_{t})\mathbf{x}_{t}+\mathbf{U}\mathbf{h}_{t-1})$ and
the output unit computation remains same as RNN. This will be discussed in the
next section in more detail.

##  3 Proposed Approach

In this section, we describe the proposed Context integrated Recurrent Neural
Network (CiRNN) model which uses GRUs as the basic unit of the network. We
first give the notations, and then briefly discuss the GRU to relate it to
CiRNN. Finally, we present the details of CiRNN.

###  3.1 Notations

The following notations are used to describe CiRNN.

$\begin{split}&n_{x}:\text{primary input dimension}\\\ &n_{h}:\text{number of
hidden units}\\\ &n_{y}:\text{output dimension}\\\ &n_{z}:\text{context input
dimension}\\\ &T_{x}:\text{input sequence length}\\\ &T_{y}:\text{output
sequence length}\\\ &(\text{for many-to-one RNN architecture $T_{y}=1$})\\\
&\mathbf{x}_{t}\in\Re^{(n_{x}\times 1)}:\text{input vector at time step
$t$}\\\ &\mathbf{z}_{t}\in\Re^{(n_{z}\times 1)}:\text{context vector at time
step $t$}\\\ &\mathbf{y}_{t}\in\Re^{(n_{y}\times 1)}:\text{output at time step
$t$}\\\ &\hat{\mathbf{y}}_{t}\in\Re^{(n_{y}\times 1)}:\text{estimated output
at time step $t$}\\\ &\mathbf{h}_{t}\in\Re^{(n_{h}\times 1)}:\text{hidden unit
activation at time step $t$}\\\ &\tilde{\mathbf{h}}_{t}\in\Re^{(n_{h}\times
1)}:\text{candidate activation at time step $t$}\\\
&\mathbf{s}_{t}\in\Re^{(n_{h}\times 1)}:\text{set/update gate at time step
$t$}\\\ &\mathbf{r}_{t}\in\Re^{(n_{h}\times 1)}:\text{reset gate at time step
$t$}\\\ &\mathbf{W}^{s}\in\Re^{(n_{h}\times n_{x})}:\text{weights (set gate to
input)}\\\ &\mathbf{W}^{h}\in\Re^{(n_{h}\times n_{x})}:\text{weights (hidden
to input)}\\\ &\mathbf{W}^{r}\in\Re^{(n_{h}\times n_{x})}:\text{weights (reset
gate to input)}\\\ &\mathbf{U}^{s}\in\Re^{(n_{h}\times n_{h})}:\text{weights
(set gate to hidden)}\\\ &\mathbf{U}^{h}\in\Re^{(n_{h}\times
n_{h})}:\text{weights (hidden to hidden)}\\\
&\mathbf{U}^{r}\in\Re^{(n_{h}\times n_{h})}:\text{weights (reset gate to
hidden)}\\\ &\mathbf{V}\in\Re^{(n_{y}\times n_{h})}:\text{weights (output to
hidden)}\\\ &\mathbf{b}_{y}\in\Re^{(n_{y}\times 1)}:\text{bias of output
unit}\\\ \end{split}$ (1)

###  3.2 Gated Recurrent Units

A GRU (Cho et al., 2014) is a unit or cell, which is used as a hidden unit in
RNNs. It can adaptively remember and forget the information in the network
during the learning phase. The primary motivation of GRU is same as LSTM,
which is to model long-term sequences while avoiding the vanishing gradient
problem of RNN. However, compared to LSTM, GRU does this with less number of
parameters and less number of operations. GRU comprises of two gates, reset
and update gate that controls the flow of information. For time step t, the
GRU computes the output $\hat{\mathbf{y}}_{t}$ using input $\mathbf{x}_{t}$
and the hidden state as follows:

$\begin{split}&\hat{\mathbf{y}_{t}}=f(\mathbf{V}\mathbf{h}_{t}+\mathbf{b}_{y})\\\
&\mathbf{h}_{t}=\mathbf{s}_{t}\odot\mathbf{h}_{t-1}+(1-\mathbf{s}_{t})\odot\tilde{\mathbf{h}}_{t}\\\
&\tilde{\mathbf{h}}_{t}=tanh(\mathbf{W}^{h}\mathbf{x}_{t}+\mathbf{U}^{h}(\mathbf{r}_{t}\odot\mathbf{h}_{t-1}))\\\
&\mathbf{s}_{t}=\sigma(\mathbf{W}^{s}\mathbf{x}_{t}+\mathbf{U}^{s}\mathbf{h}_{t-1})\\\
&\mathbf{r}_{t}=\sigma(\mathbf{W}^{r}\mathbf{x}_{t}+\mathbf{U}^{r}\mathbf{h}_{t-1})\end{split}$
(2)

The reset gate functions very similar to the forget gate of the LSTM cell. It
is a linear combination of input at time step $t$ and hidden state at previous
time step $t-1$. The reset gate resets the hidden state with current input and
ignores the information from previous hidden state when its value is close to
0. It controls how much information from the previous hidden state will be
removed. On the other hand, the update gate controls how much information from
the previous hidden state will be remembered.

###  3.3 Context Integrated RNN

Context Integrated RNN is a RNN with GRU as hidden units. For clarity of
presentation, in Figure 1, the basic architecture of a RNN is compared with
CiRNN. The main difference between the two architectures is that in CiRNN, the
input to hidden unit weights are dependent on the context variables. When each
of the hidden units $\mathbf{h}_{1},\mathbf{h}_{2},\dots\mathbf{h}_{nh}$ in
Figure 1 is a GRU then the parameters
$\mathbf{W}^{s},\mathbf{W}^{r},\mathbf{W}^{h}$ are dependent on the context
variables $\mathbf{z}$.

Figure 1: Basic architecture of RNN vs. CiRNN. The connection weights between
various layers is represented by $\mathbf{W}$. In CiRNN, the input to hidden
layer connection weights ($\mathbf{W}_{hx}$) are dependent on the context
$\mathbf{z}$ (Dutta Baruah and Muñoz Organero, 2023, accepted)

.

The output computation in a single unit of CiRNN is same as in GRU. However,
the computation of candidate hidden state, set/update gate, and reset gate
values are computed differently as it involves weights that are dependent on
context $\mathbf{z}_{t}$ as shown below:

$\begin{split}&\hat{\mathbf{y}_{t}}=f(\mathbf{V}\mathbf{h}_{t}+\mathbf{b}_{y})\\\
&\mathbf{h}_{t}=\mathbf{s}_{t}\odot\mathbf{h}_{t-1}+(1-\mathbf{s}_{t})\odot\tilde{\mathbf{h}}_{t}\\\
&\tilde{\mathbf{h}}_{t}=tanh(\mathbf{W}^{h}(\mathbf{z}_{t})\mathbf{x}_{t}+\mathbf{U}^{h}(\mathbf{r}_{t}\odot\mathbf{h}_{t-1}))\\\
&\mathbf{s}_{t}=\sigma(\mathbf{W}^{s}(\mathbf{z}_{t})\mathbf{x}_{t}+\mathbf{U}^{s}\mathbf{h}_{t-1})\\\
&\mathbf{r}_{t}=\sigma(\mathbf{W}^{r}(\mathbf{z}_{t})\mathbf{x}_{t}+\mathbf{U}^{r}\mathbf{h}_{t-1})\end{split}$
(3)

In equation 3, the weights associated with the input $(\mathbf{x}_{t})$ are
dependent on the vector of contextual variables $(\mathbf{z}_{t})$. Let us
consider one of the such parameters, $\mathbf{W}^{s}$. Note that
$\mathbf{W}^{h}(\mathbf{z}_{t})$ and $\mathbf{W}^{r}(\mathbf{z}_{t})$ can be
expressed in similar way.

The matrix $\mathbf{W}^{s}(\mathbf{z}_{t})$ is of dimension $n_{h}\times
n_{x}$ and each of the components can be given as:

$\mathbf{W}^{s}(\mathbf{z}_{t})=\begin{bmatrix}w_{11}^{s}(\mathbf{z}_{t})&w_{12}^{s}(\mathbf{z}_{t})&\cdots&w_{1n_{x}}^{s}(\mathbf{z}_{t})\\\
w_{21}^{s}(\mathbf{z}_{t})&w_{22}^{s}(\mathbf{z}_{t})&\cdots&w_{2n_{x}}^{s}(\mathbf{z}_{t})\\\
\vdots&\vdots&\cdots&\vdots\\\
w_{n_{h}1}^{s}(\mathbf{z}_{t})&w_{n_{h}2}^{s}(\mathbf{z}_{t})&\cdots&w_{n_{h}n_{x}}^{s}(\mathbf{z}_{t})\\\
\end{bmatrix}$ (4)

where each element of the matrix can be expressed as:

$\begin{split}&w_{ki}^{s}=\mathbf{A}_{ki}^{s}\mathbf{G}(\mathbf{z}_{t}),\quad
k=1,..,n_{h},\quad i=1,..,n_{x}\\\
&\mathbf{A}_{ki}^{s}=[a_{ki1}^{s},a_{ki2}^{s},\cdots,a_{kim}^{s}]\end{split}$
(5)

In (5),
$\mathbf{G}(\mathbf{z}_{t})=[g_{1}(\mathbf{z}_{t}),g_{2}(\mathbf{z}_{t}),...,g_{m}(\mathbf{z}_{t})]^{T}$
is a vector of basis functions that can be chosen at the time of design and
$\mathbf{A}_{ki}^{s}$ is a vector of coefficients that specify the dependence
of weights on context variables (Ciskowski and Rafajlowicz, 2004). We can
define a matrix $\mathbf{A}^{s}$ where each row $\mathbf{A}^{s}_{k}$ can be
formed by concatenating coefficient vectors $\mathbf{A}_{ki}^{s}$ as shown
below. Therefore, $\mathbf{A}^{s}$ is of dimension $(n_{h}\times n_{x}m)$.

$\mathbf{A}_{k}^{s}=[\mathbf{A}_{k1}^{s},\mathbf{A}_{k2}^{s},\cdots,\mathbf{A}_{kn_{x}}^{s}],\quad
k=1,2,...,n_{h}$ (6)

Using $\mathbf{A}^{s}$ and similarly $\mathbf{A}^{h}$ and $\mathbf{A}^{r}$,
the candidate hidden state $\tilde{\mathbf{h}}_{t}$, the update (set) gate
$\mathbf{s}_{t}$, and reset gate $\mathbf{r}_{t}$ in equation (3) can be
expressed as:

$\begin{split}&\tilde{\mathbf{h}_{t}}=tanh[\mathbf{A}^{h}(\mathbf{x}_{t}\otimes\mathbf{G}(\mathbf{z}_{t}))+\mathbf{U}^{h}(\mathbf{r}_{t}\odot\mathbf{h}_{t-1})]\\\
&\mathbf{s}_{t}=\sigma[\mathbf{A}^{s}(\mathbf{x}_{t}\otimes\mathbf{G}(\mathbf{z}_{t}))+\mathbf{U}^{s}\mathbf{h}_{t-1}]\\\
&\mathbf{r}_{t}=\sigma[\mathbf{A}^{r}(\mathbf{x}_{t}\otimes\mathbf{G}(\mathbf{z}_{t}))+\mathbf{U}^{r}\mathbf{h}_{t-1}]\end{split}$
(7)

In (7), $\otimes$ denotes Kronecker product of matrices. Learning of each of
the weights $w_{ki}^{s}$ will require learning of the vector of coefficients
$\mathbf{A}_{ki}^{s}$ with $m$ elements. The parameter learning process in
CiRNN is similar to RNNs which requires defining a loss function and
optimization of parameters using any suitable optimization algorithm such as
stochastic gradient descent (SGD) or Adam optimization algorithm (Kingma and
Ba, 2014) with Back Propagation Through Time (BPTT or Truncated BPTT). (The
gradient calculation of L2 Loss function is provided in the Appendix A.)

##  4 Experiments and Results

In this section, we first describe the evaluation task and the dataset, and
then discuss the experiments and the results achieved with the proposed model.
The results of CiRNN are compared with baseline models and also with the
state-of-art methods from the existing literature.

###  4.1 Predictive Maintenance

For evaluating the efficacy of CiRNN, we considered the task of predictive
maintenance. Predictive maintenance (PdM) also referred to as Prognostic and
Health Management (PHM) has become more relevant with the arrival of Industry
4.0 within the context of smart manufacturing and industrial big data. Through
PdM, various industries aim to achieve near-zero failures, downtime,
accidents, and unscheduled maintenance thereby saving considerable amount of
cost and also improve operational safety (Zhang et al., 2019).

PdM primarily comprises of diagnostics, and prognostics mechanisms. The former
deals with fault detection, isolation and identification and the latter
attends to degradation monitoring and failure prediction. Compared to
diagnostics, prognostics is more efficient to achieve zero-downtime
performance (Shin and Jun, 2015). Prediction of Remaining Useful Life (RUL) is
one of the crucial tasks in prognostics. RUL, also called remaining service
life, residual life or remnant life, refers to the time left before observing
a failure given the current machine age and condition, and the past operation
profile (Jardine et al., 2006). A reliable estimation of RUL enables the
stakeholders to attain the benefits of PdM.

In recent years, industrial cyber-physical system has emerged as one of the
key technologies for reliable data acquisition in real-time. Thanks to such
systems that made huge amount of data available which in turn led to a growing
interest among researchers to apply machine learning, in particular deep
learning, approaches to industrial applications. A review of deep neural
networks applied to PHM is available in (Rezaeianjouybari and Shang, 2020) and
(Zhang et al., 2019). In (Wang et al., 2020) the state-of-the-art deep
learning approaches for RUL prediction is discussed. Here, we briefly provide
an overview of some of the recent approaches for prediction of RUL that employ
C-MAPSS dataset as our work is evaluated with the same dataset. More details
on the dataset is provided in the next section. In (Heimes, 2008), one of the
early works that used a basic RNN for estimation of RUL of turbofan engines is
presented. It is based on a RNN trained with back-propagation through time
using Extended Kalman Filter. This work was placed second in the IEEE 2008
Prognostics and Health Management conference challenge problem (Jia et al.,
2018).

In (Yuan et al., 2016) and (Wu et al., 2018), a vanilla LSTM is used for
prediction of RUL. They showed that LSTM could perform better compared to
basic RNNs, GRUs and Adaboost-LSTM. In (Zheng et al., 2017), deep LSTM with a
feedforward newtwork is used to estimate the RUL. The six operating conditions
of the engine are encoded as one hot vector and are used while training the
model. The results showed that deep LSTM model with feedforwarded network
performed better compared to Multi-layer Perceptron, Support Vector
Regression, Relevance Vector Regression, and Deep Convolutional Neural Network
(CNN). Li et al. (Li et al., 2018) applied 1D convolutions in sequence without
pooling operations for useful feature extraction in RUL prediction. The model
achieved good results without incurring high training time compared to
recurrent models. Listou Ellefsen et al., (Listou Ellefsen et al., 2019), used
a Restricted Boltzmann Machine (RBM) to investigate the effect of unsupervised
pre-training in RUL predictions. Initially, RBM is used to get the initial
weights from unlabeled data and later fine tuning is performed in an LSTM
network with labeled data. Further, a Genetic Algorithm (GA) approach is
applied for tuning the hyper-parameters during the training. The model
proposed in (Zhao et al., 2019), emphasizes on trend features. The trend
features are extracted using Complete Ensemble Empirical Mode Decomposition
(CEEMD) approach, followed by reconstruction procedure. These features are
used to train a LSTM for RUL prediction. A global attention mechanism is used
with LSTM network in (da Costa et al., 2019). It learns the input to RUL
mapping directly from the raw sensor data and do not require feature
engineering or unsupervised pre-training. Further, the attention mechanism
allows visualising the learned attention weights at each step of prediction of
RUL.

The existing approaches discussed here consider the C-MAPSS dataset. The
dataset consists of three operational setting parameters that provide
information about different flight operating conditions which is contextual
information. The existing approaches do not explicitly utilize this
information during the learning process. They either excluded this information
and used only the sensor data during training (Zhao et al., 2019) or included
them as primary features (Yuan et al., 2016), (da Costa et al., 2019).

###  4.2 Dataset Description

NASA Turbofan Engine Degradation Simulation Data Set (Saxena et al., 2008) is
a widely used benchmark for RUL prediction. The dataset is generated using
Commercial Modular Aero-Propulsion System Simulation (C-MAPSS) tool. It
consists of four distinct datasets (FD001, FD002, FD003, and FD004) that
contain information from 21 sensors (such as Total temperature at fan inlet,
Total temperature at Low Pressure Compressor outlet), 3 operational settings
(flight altitude, Mach number, and throttle resolver angle), engine
identification number, and cycles of each engine. The degradation in engine
performance is captured by the sensor data. The engine performance is also
substantially influenced by the three operational settings. FD001 has one
operating condition and one failure mode. FD002 has six operating conditions
and one failure mode. FD003 has one operating condition and two failure modes
and FD004 has six operating conditions and two failure modes. The details are
provided in Table 1. For our approach, both FD002 and FD004 are suitable as
the six operating conditions provide the contextual information. However, for
experiments we have also considered FD001 and FD003 to evaluate the
performance where only one context is available (single operating condition).

The engines start with a different degree of initial wear and manufacturing
variation which is not known to the user and are not considered as a fault
condition. Each of the datasets consist of separate training and test sets. In
the training set, the sensor data is captured until the system fails whereas
in the test set it is captured up to a certain time prior to the failure. The
test sets also have the true Remaining Useful Life (RUL) values.

The objective is to predict the number of remaining operational cycles before
failure as provided in the test set.

Table 1: Specification of C-MAPSS datsets Specification | FD001 | FD002 | FD003 | FD004
---|---|---|---|---
Engine units (Training) | 100 | 260 | 100 | 249
Engine units (Testing) | 100 | 259 | 100 | 248
Operating Conditions | 1 | 6 | 1 | 6
Fault Modes | 1 | 1 | 2 | 2

###  4.3 Data Preprocessing

For each of the dataset, univariate and bivariate analysis of data from the 21
sensors is performed. Each of the sensor data series is analyzed to see the
trend with respect to the time cycles. Further, for each dataset, sensors are
selected based on the scatter plot observations and correlation analysis. The
selected sensors and operational settings is provided in Table 2. The sensor
values are used as primary features and the operational settings are used as
contextual features while learning the models.

Table 2: Selected sensors and operational settings Dataset |  Sensors |  Operational Settings
---|---|---
FD001 |  s2, s3, s4, s7, s8, s9, s11, s12, s13, s15, s17, s20, s21 |  OS1, OS2
FD002 |  s1, s2, s8, s13, s14, s19 |  OS1, OS2, OS3
FD003 |  s2, s3, s4, s7, s8, s9, s11, s15, s17, s20, s21 1 |  OS1,OS2
FD004 |  s2, s8, s14, s16 |  OS1, OS2, OS3
OS1- Flight altitude, OS2- Mach number, OS3- Throttle resolver

The RUL for training is required to be identified as the true RUL values are
not provided in the training data. We adopted a piece-wise linear degradation
model to get the RUL values for the training data as given in (Listou Ellefsen
et al., 2019) and (da Costa et al., 2019) . The degradation model assumes that
after an initial period with constant RUL values (initially when the engine
operates in normal conditions) (Heimes, 2008) the RUL decrease linearly as
there is progress in the number of time cycles. For our experiments, the
initial constant RUL is considered to be 125 cycles so that results can be
compared with existing works. Figure 2 shows the RUL for engine unit 2 of
FD002 training data.

Figure 2: RUL of engine unit 1 (FD002 training data)

For FD001 and FD003 (with only one operational condition), we performed min-
max normalisation on both primary and contextual inputs, and also on the
target. For FD002 and FD004 (with six operational conditions) we performed
contextual normalisation (Turney, 1996) in addition to min-max normalisation
(excluding the target). For contextual normalisation, the data is clustered
according to 6 operational regimes using k-means clustering and then the data
is normalised using the cluster statistics (mean and range). Finally, data
smoothing is performed using moving averages with window size of 3.

###  4.4 Performance Metrics

Two metrics are used to measure the performance of the proposed model. The
first metric is the RMSE, and the second metric is a score that is computed
from an asymmetric scoring function as proposed by Saxena et al. (Saxena et
al., 2008), which is given as,

$s=\left\\{\begin{array}[]{l l}\sum_{i=1}^{n}e^{-\frac{D_{i}}{a_{1}}}-1,\
\text{if}\ D_{i}<0\\\ \sum_{i=1}^{n}e^{\frac{D_{i}}{a_{2}}}-1,\ \text{if}\
D_{i}\geq 0\end{array}\right.$ (8)

where $a_{1}=10$, $a_{2}=13$, and $D_{i}=\hat{RUL}_{i}-RUL_{i}$ is the
difference between predicted RUL and actual RUL values, $n$ is the number of
samples in the test data. The scoring metric penalizes the late predictions
(positive errors) more compared to early predictions (negative errors). In
either case, the penalty increases exponentially with error.

###  4.5 Training and Validation

To train the models, the training and validation data is prepared in the
following way. From each of the dataset, each engine unit is considered and
last $k$ samples are selected for validation and remaining are used for
training. It is to be noted that $k$ is required to be multiple of sequence
length used while training the models. As a result, the validation set
consists of samples from each engine unit as in the test set. Table 3 shows
various hyperparameters that are used for tuning the models. The number of
layers is fixed to one, due to computational overhead of custom built CiRNN
code. For the contextual inputs, polynomial basis functions of degree 2 are
used.

Table 3: Hyperparameter values and optimizer used for tuning Hyperparameter | Value
---|---
Number of hidden units | {10, 15, 20, 25, 30 }
Sequence length | {10, 15, 20}
Batch size | {64, 128, 256 }
Learning rate | loguniform(1e-5, 1e-3)
Optimizer | SGD, Adam, RMSProp

We considered GRU-RNN as a baseline where the RNN consits of GRU but without
contextual weighting. Both baseline and CiRRN are trained with various
configurations. For example, GRU-RNN with contextual normalisation and with
context features where context features are treated as primary features. The
later is also referred to as contextual expansion (Turney, 1996). Table 4
shows various models and Table 5 presents the best hyperparameter values for
each model that are achieved after tuning them. In the next subsection, we
discuss the results obtained with these models.

Table 4: Model configurations Dataset | Model | Model Acronym |  Contextual Normalisation |  Context Features
---|---|---|---|---
FD001 | CiRNN | CiRNN_D1 |  No |  Yes
(1 operational condition) | GRU | GRU_D1_CxF |  No |  Yes
| GRU | GRU_D1 |  No |  No
FD002 | CiRNN | CiRNN_D2 |  Yes |  Yes
(6 operational conditions) | CiRNN | CiRNN_D2_CxF |  No |  Yes
| GRU | GRU_D2 |  No |  No
| GRU | GRU_D2_CxF |  No |  Yes
| GRU | GRU_D2_CxN |  Yes |  No
| GRU | GRU_D2_CxN_CxF |  Yes |  Yes
FD003 | CiRNN | CiRNN_D3 |  No |  Yes
(1 operational condition) | GRU | GRU_D3_CxF |  No |  Yes
| GRU | GRU_D3 |  No |  No
FD004 | CiRNN | CiRNN_D4 |  Yes |  Yes
(6 operational conditions) | CiRNN | CiRNN_D4_CxF |  No |  Yes
| GRU | GRU_D4 |  No |  No
| GRU | GRU_D4_CxF |  No |  Yes
| GRU | GRU_D4_CxN |  Yes |  No
| GRU | GRU_D4_CxN_CxF |  Yes |  Yes
Table 5: Model configurations and Hyperparameters Model Acronym |  Hyperparameter values | Optimizer
---|---|---
CiRNN_D1 |  $15,20,64,1\times 10^{-2}$ | Adam
GRU_D1_CxF |  $25,20,128,5\times 10^{-3}$ | Adam
GRU_D1 |  $10,15,64,9\times 10^{-3}$ | Adam
CiRNN_D2 |  $20,15,64,5\times 10^{-3},$ | RMSProp
CiRNN_D2_CxF |  $15,20,128,5\times 10^{-3}$ | RMSProp
GRU_D2 |  $25,15,128,1\times 10^{-2}$ | Adam
GRU_D2_CxF |  $30,15,128,5\times 10^{-3}$ | Adam
GRU_D2_CxN |  $20,10,64,8\times 10^{-3}$ | RMSProp
GRU_D2_CxN_CxF |  $15,10,64,9\times 10^{-3}$ | RMSProp
CiRNN_D3 |  $30,10,64,8\times 10^{-3}$ | RMSProp
GRU_D3_CxF |  $15,10,64,2\times 10^{-3}$ | RMSProp
GRU_D3 |  $20,10,64,5\times 10^{-3}$ | RMSProp
CiRNN_D4 |  $15,10,128,8\times 10^{-3}$ | RMSProp
CiRNN_D4_CxF |  $25,20,128,9\times 10^{-3}$ | Adam
GRU_D4 |  $25,20,128,8\times 10^{-3}$ | Adam
GRU_D4_CxF |  $25,15,256,5\times 10^{-3}$ | Adam
GRU_D4_CxN |  $20,15,256,1\times 10^{-2}$ | RMSProp
GRU_D4_CxN_CxF |  $15,15,256,9\times 10^{-3}$ | Adam
Hyperparameter values: hidden units, sequence length, batch size, learning
rate

###  4.6 Results

Table 6 summarises the results obtained from CiRNN and the baseline models
with the test dataset. The testing is performed for each engine unit
separately and the average RMSE and average score with standard deviation is
reported. For FD001 dataset, which has only one operational condition, RNN
with GRU model where context features are added and treated as primary
features (GRU_D1_CxF) performed better than other two models. The percentage
increase in RMSE for CiRNN_D1 in comparison to GRU_D1_CxF is around 7%. For
all the remaining datasets (FD002, FD003, FD004) CiRNN performed better or at
par with the baseline models. It can be observed that contextual normalisation
along with contextual weighting through CiRNN, particularly in datasets FD002
and FD004, have noticeably improved the performance. Further, it is to be
noted that the distribution of the score metric is skewed. Here, we have
provided distribution of RMSE and scores for FD002 and FD004 with CiRNN in
Figure 3. It is apparent from the figures that the count of engines having
high scores (more than 1000) is very low.

Table 6: Comparison of CiRNN performance with baseline models Dataset | Model | RMSE | Score $(s)$
---|---|---|---
|  | (mean, std) | (mean, std)
FD001 | CiRNN_D1 | 14.53, 7.07 | 745.21, 2588.64
| GRU_D1 | 14.52, 7.64 | 658.55, 1490.477
| GRU_D1_CxF | 13.57, 7.19 | 450.69, 733.00
FD002 | CiRNN_D2 | 11.97, 4.94 | 363.03, 710.70
| CiRNN_D2_CxF | 25.74, 10.11 | 4123.71, 8035.64
| GRU_D2 | 26.57, 7.78 | 3156.17, 5158.81
| GRU_D2_CxF | 25.83, 10.04 | 4122.89, 7878.20
| GRU_D2_CxN | 12.49, 4.91 | 417.26, 841.84
| GRU_D2_CxN_CxF | 12.61, 5.09 | 470.85, 1006.85
FD003 | CiRNN_D3 | 12.87, 10.77 | 949.39, 2437.71
| GRU_D3 | 13.23, 11.26 | 1633.70, 7559.84
| GRU_D3_CxF | 12.90, 3.39 | 902.63, 2339.36
FD004 | CiRNN_D4 | 12.35, 3.06 | 377.94, 353.19
| CiRNN_D4_CxF | 23.11, 8.91 | 3730.99, 15989.09
| GRU_D4 | 22.87, 10.92 | 4604.87, 13398.62
| GRU_D4_CxF | 21.92, 9.55 | 3539.59, 12060.03
| GRU_D4_CxN | 16.91, 4.70 | 797.56, 1360.32
| GRU_D4_CxN_CxF | 17.75, 4.29 | 788.20, 678.51

(a)

(b)

Figure 3: Distribution of RMSE and Score values obtained from (a) CiRNN_D2
with FD002 test set (b) CiRNN_D4 with FD004 test set

Figure 4 shows the predicted RUL values versus actual RUL values for the first
1200 samples of the FD004 test data and in Figure 5 the predicted and actual
RUL values of two randomly selected engine units from FD001 and FD004 test
data is shown. It can be noticed from Figure 4 that for longer running times,
the model prediction is better compared to shorter running times in FD004. The
late predictions in case of shorter running times also contribute to higher
score from scoring function. If we observe the predictions of engine units
closely (Figure 5), late predictions are more in FD001 compared to FD004. This
is also observed in Table 6, where CiRNN predictions have much higher scores
in datasets with one operating condition due to late predictions.

A comparison of results achieved from CiRNN and results from state-of-the art
deep learning models applied to the same dataset is presented in Table 7
(FD001 and FD003) and Table 8 (FD002 and FD004). The models that are selected
for comparison are sequential models based on LSTM (Zheng et al., 2017),
(Listou Ellefsen et al., 2019), (da Costa et al., 2019), (Zhao et al., 2019)
and additionally CNN (Li et al., 2018) is considered. It is worth mentioning
here that there are several works reported in the literature that use C-MAPSS
dataset, however, most of the works are based on only FD001 dataset and are
not tested on all the four datasets. As it can be seen from the Table 8, CiRNN
performed better in both the datasets FD002 and FD004 where contextual
information in terms of operational settings is available. Our model resulted
in 32% (FD002) and 39% (FD004) relative improvement over the best reported
RMSE values (LSTM+Attention). The proposed model also achieved much lower
scores for both FD002 and FD004 datasets with an improvement of 82% and 87%
respectively over the previous best results. Thus, CiRNN is able to reduce the
number of late RUL predictions. For FD003 dataset (Table 8), the RMSE is at
par with other models, however, the score is comparable to only one model
(LSTM+FNN) and much higher than the remaining models. In FD001, the CiRNN
model performed better than two models, LSTM+FNN and CEEMD+LSTM in terms of
RMSE. However, there is a 13% decrease in performance than the best model
(RBM+LSTM). Further, CiRNN score value is higher in comparison to other
models.

From the experimental results, it is apparent that CiRNN models’ performance
is significantly higher than other models in presence of contexts and
comparable to other models when context is not explicitly available (data with
one operating condition). Also, in comparison to other models CiRNN achieved
the given performance with relatively simpler model with 1 layer and 15 hidden
neurons. Feature selection and contextual normalisation also have an influence
on the performance of CiRNN. Further, each of the approaches consider the
operating conditions (contextual features) in a different way. For example,
Zheng et al. (Zheng et al., 2017), in their approach, clustered the operating
conditions and use one-hot encoding for their representation. It is then
include as a primary feature. Li et al. (Li et al., 2018) used only the sensor
data for modeling. In (Listou Ellefsen et al., 2019) and (da Costa et al.,
2019), all the three operational settings and all the sensor measurements are
used. However, compared to these approaches, contextual weighting in CiRNN
performs better in all the datasets with explicit contexts.

Figure 4: Actual versus Predicted RUL with CiRNN using FD004 test data

(a)

(b)

(c)

(d)

Figure 5: Actual versus predicted RUL for selected engines (a) and (b) CiRNN with FD001 test data, (c) and (d) CiRNN with FD004 test data Table 7: Comparison of CiRNN performance with existing approaches reported in the literature (FD001 and FD003) Method | FD001 | FD003
---|---|---
RMSE | Score (s) | RMSE | Score (s)
LSTM+ FNN | 16.14 | 338 | 16.18 | 852
(Zheng et al., 2017) |  |  |  |
CNN + FNN | 12.61 | 274 | 12.64 | 284
(Li et al., 2018) |  |  |  |
RBM + LSTM | 12.56 | 231 | 12.10 | 251
(Listou Ellefsen et al., 2019) |  |  |  |
LSTM + Attention | 13.95 | 320 | 12.72 | 223
(da Costa et al., 2019) |  |  |  |
CEEMD + LSTM | 14.72 | 262 | 17.72 | 452
(Zhao et al., 2019) |  |  |  |
CiRNN (Proposed) | 14.53 | 451 | 12.87 | 949
Table 8: Comparison of CiRNN performance with existing approaches reported in the literature (FD002 and FD004) Method | FD002 | FD004
---|---|---
RMSE | Score (s) | RMSE | Score (s)
LSTM+ FNN | 24.49 | 4,450 | 28.17 | 5,550
(Zheng et al., 2017) |  |  |  |
CNN + FNN | 22.36 | 10,412 | 23.31 | 12,466
(Li et al., 2018) |  |  |  |
RBM + LSTM | 22.73 | 3,366 | 22.66 | 2,840
(Listou Ellefsen et al., 2019) |  |  |  |
LSTM + Attention | 17.65 | 2.102 | 20.21 | 3,100
(da Costa et al., 2019) |  |  |  |
CEEMD + LSTM | 29.00 | 6953 | 33.43 | 15069
(Zhao et al., 2019) |  |  |  |
CiRNN (Proposed) | 11.97 | 363 | 12.35 | 378

Finally, in Figure 6, the contextual weights $\mathbf{A}^{s}$ are shown for
CiRNN with FD002 dataset (CiRNN_FD002). Note that this model has 15 hidden
units, 6 primary inputs, and 3 contextual inputs. The primary inputs are,
$x_{1}=s_{1}$ (total temperature at fan inlet), $x_{2}=s_{2}$ (total
temperature at LPC outlet), $x_{3}=s_{8}$ (physical fan speed), $x_{4}=s_{13}$
(corrected fan speed), $x_{5}=s_{14}$ (corrected core speed), and
$x_{6}=s_{19}$ (demanded corrected fan speed). With 9 polynomial basis
functions of degree 2, the size of $\mathbf{A}^{s}$ is $(15\times(6\times 9))$
i.e. $(15\times 54)$. The entire weight matrix is represented through 6
smaller heatmaps of size $15\times 9$. The first heatmap represents the
weights associated with $x_{1t}\times G(\mathbf{Z}_{t})$ and the hidden units.
Similarly, second heat map represents the weights associated with
$x_{2t}\times G(\mathbf{Z}_{t})$ and the hidden units, and so on. It can be
observed from Figure 6 that weights associated with sensor 1 data (total
temperature at fan inlet) , and sensor 2 (total temperature at LPC outlet) are
mostly negative. However, hidden unit 10 has connections with positive
weights. The connections from sensor 8 data (physical fan speed) has more
positive weights compared to remaining 5 sensors. This indicates that physical
fan speed has more impact on update gate output and in turn RUL prediction.

Figure 6: Input-to-hidden weights $\mathbf{A}^{s}$

##  5 Conclusion

In this paper, we proposed a novel RNN architecture, named CiRNN, that enables
integrating explicit contexts available from the problem domain. In CiRNN, the
input features referred to as primary features are weighted by context which
is represented by contextual features. Thus, external factors influences the
change in network parameters. To demonstrate the pertinence of CiRNN to
applications where contextual information is accessible through sensors, we
applied CiRNN to engine health prognostics. In particular, we used CiRNN to
estimate the RUL using widely known benchmark dataset (C-MAPSS dataset). The
dataset consists of information on operational environment of the flight
engines through three settings parameters. This provides the required
contextual information for CiRNN. We performed several experiments using
datsets with contexts as well as datasets without only one context is
available without any variations. The experimental results show that
integrating context to RNN (with GRU cell) significantly improves the
performance of the resulting CiRNN model in comparison to the baseline models
and also the existing approaches with relatively simple network configuration,
particularly when context is present (FD002 and FD004). For datasets that do
not exhibit context (FD001 and FD003), CiRNN’s performance is comparable to
the baseline and existing models.

As our environment is becoming data rich, CiRNN can potentially be applied to
many real life application, such as in the domain of smart healthcare, smart
manufacturing, and smart agriculture, where contextual information can be
acquired and used naturally. In future, we intend to explore such areas.

## Acknowledgements

This work is supported by CONEX-Plus programme funded by Universidad Carlos
III de Madrid and the European Union’s Horizon 2020 research and innovation
programme under the Marie Sklodowska-Curie grant agreement No. 801538.

##  Appendix A Gradient computation of L2 Loss function with respect to CiRNN
parameters

Let $L_{t}$ be the loss at time step $t$ and $L$ be the total loss over the
given time sequence, which are given as,

$\begin{split}L_{t}=\frac{1}{2}(\mathbf{y}_{t}-\hat{\mathbf{y}}_{t})^{2}\end{split}$
(9)

$\begin{split}L=\frac{1}{2}\sum_{t}(\mathbf{y}_{t}-\hat{\mathbf{y}}_{t})^{2}\end{split}$
(10)

To train CiRNN, we need the values of all the parameters
$(\mathbf{A}^{h},\mathbf{A}^{s},\mathbf{A}^{r})$,
$(\mathbf{U}^{h},\mathbf{U}^{s},\mathbf{U}^{r})$, and
$(\mathbf{V},\mathbf{b}_{y})$ that minimize the loss $L$. For gradient-based
optimization, the derivative of $L$ w.r.t. each of the parameters is required.

Let us first consider ${\partial L}/{\partial\mathbf{A}^{s}}$. The calculation
for the other parameters would be similar. Since $L=1/2\sum_{t}L_{t}$ and the
parameters remain same in each step, we also have ${\partial
L}/{\partial\mathbf{A}^{s}}=\sum_{t}(\partial L_{t}/\partial\mathbf{A}^{s})$.
So, we can calculate $(\partial L_{t}/\partial\mathbf{A}^{s})$ independently
and finally sum them up.

$\begin{split}\frac{\partial L_{t}}{\partial\mathbf{A}^{s}}&=\frac{\partial
L_{t}}{\partial{\mathbf{h}}_{t}}\frac{\partial\mathbf{h}_{t}}{\partial\mathbf{A}^{s}}\\\
\end{split}$ (11)

Let us consider each of the derivatives on the R.H.S. of the equation (11).

$\begin{split}\frac{\partial L_{t}}{\partial\mathbf{h}_{t}}&=\frac{\partial
L_{t}}{\partial\hat{\mathbf{y}}_{t}}\frac{\partial\hat{\mathbf{y}}_{t}}{\partial\mathbf{h}_{t}}=(\hat{\mathbf{y}}_{t}-\mathbf{y}_{t})f^{\prime}(u_{t})\mathbf{V}\\\
\end{split}$ (12)

where $u_{t}=\mathbf{V}\mathbf{h}_{t}+\mathbf{b}_{y}$

In the second term of R.H.S. of equation (11), $\mathbf{h}_{t}$ also depends
on $\mathbf{A}^{s}$ through $\mathbf{s_{t}}$ and also depends on
$\mathbf{h}_{t-1}$ that in turn depends $\mathbf{A}^{s}$. So, $\mathbf{h}_{t}$
depends on $\mathbf{A}^{s}$ explicitly and also implicitly through
$\mathbf{h}_{t-1}$, and we can write
$\partial\mathbf{h}_{t}/\partial\mathbf{A}^{s}$ as,

$\begin{split}\frac{\partial\mathbf{h}_{t}}{\partial\mathbf{A}^{s}}={\frac{\partial\mathbf{h}_{t}}{\partial\mathbf{A}^{s}}}^{*}+\frac{\partial\mathbf{h}_{t}}{\partial\mathbf{h}_{t-1}}\frac{\partial\mathbf{h}_{t-1}}{\partial\mathbf{A}^{s}}\end{split}$
(13)

The explicit derivative is denoted by a ’$*$’ in the above equation where
$\mathbf{h}_{t-1}$ is considered as constant. Again, the last term in equation
(13) can be expanded and written in terms of implicit and explicit derivatives
as given below:

$\begin{split}\frac{\partial\mathbf{h}_{t}}{\partial\mathbf{A}^{s}}={\frac{\partial\mathbf{h}_{t}}{\partial\mathbf{A}^{s}}}^{*}+\frac{\partial\mathbf{h}_{t}}{\partial\mathbf{h}_{t-1}}{\frac{\partial\mathbf{h}_{t-1}}{\partial\mathbf{A}^{s}}}^{*}+\frac{\partial\mathbf{h}_{t}}{\partial\mathbf{h}_{t-1}}\frac{\partial\mathbf{h}_{t-1}}{\partial\mathbf{h}_{t-2}}\frac{\partial\mathbf{h}_{t-2}}{\partial\mathbf{A}^{s}}\end{split}$

This process continues until $\partial\mathbf{h}_{1}/\partial\mathbf{A}^{s}$
is reached. $\mathbf{h}_{1}$ is implicitly dependent on $\mathbf{h}_{0}$,
however, $\mathbf{h}_{0}$ is constant and initialized to a vector of zeros.
Therefore, equation (13) can be given as,

$\begin{split}\frac{\partial\mathbf{h}_{t}}{\partial\mathbf{A}^{s}}&=\sum_{k=1}^{t}\frac{\partial\mathbf{h}_{t}}{\partial\mathbf{h}_{k}}{\frac{\partial\mathbf{h}_{k}}{\partial\mathbf{A}^{s}}}^{*}\\\
\end{split}$ (14)

where $\frac{\partial\mathbf{h}_{t}}{\partial\mathbf{h}_{k}}$ is a chain rule
in itself, for example,
$\frac{\partial\mathbf{h}_{3}}{\partial\mathbf{h}_{1}}=\frac{\partial\mathbf{h}_{3}}{\partial\mathbf{h}_{2}}\
\frac{\partial\mathbf{h}_{2}}{\partial\mathbf{h}_{1}}$. Further, equation (14)
can be given as,

$\begin{split}\frac{\partial\mathbf{h}_{t}}{\partial\mathbf{A}^{s}}&=\sum_{k=1}^{t}\left(\left(\prod_{k=1}^{t-1}\frac{\partial\mathbf{h}_{k+1}}{\partial\mathbf{h}_{k}}\right){\frac{\partial\mathbf{h}_{k}}{\partial\mathbf{A}^{s}}}^{*}\right)\\\
\end{split}$ (15)

Using equations (11), (12), and (15), ${\partial
L_{t}}/{\partial\mathbf{A}^{s}}$ can be given as,

$\frac{\partial
L_{t}}{\partial\mathbf{A}^{s}}=(\hat{\mathbf{y}}_{t}-\mathbf{y}_{t})f^{\prime}(u_{t})\mathbf{V}\sum_{k=1}^{t}\left(\left(\prod_{k=1}^{t-1}\frac{\partial\mathbf{h}_{k+1}}{\partial\mathbf{h}_{k}}\right){\frac{\partial\mathbf{h}_{k}}{\partial\mathbf{A}^{s}}}^{*}\right)\\\
$ (16)

The terms within the summation are derived as follows:

$\begin{split}{\frac{\partial\mathbf{h}_{t}}{\partial\mathbf{A}^{s}}}^{*}&=\left((\mathbf{h}_{t-1}-\tilde{\mathbf{h}}_{t})\odot\mathbf{s}_{t}\odot(1-\mathbf{s}_{t})\right)(\mathbf{x}_{t}\otimes\mathbf{G}(t))^{T}\\\
\end{split}$ (17)

$\begin{split}{\frac{\partial\mathbf{h}_{t}}{\partial\mathbf{h}_{t-1}}}&={\frac{\partial\mathbf{h}_{t}}{\partial\tilde{\mathbf{h}}_{t}}}{\frac{\partial\tilde{\mathbf{h}}_{t}}{\partial\mathbf{h}_{t-1}}}+{\frac{\partial\mathbf{h}_{t}}{\partial\mathbf{s}_{t}}}{\frac{\partial\mathbf{s}_{t}}{\partial\mathbf{h}_{t-1}}}+{\frac{\partial\mathbf{h}_{t}}{\partial\mathbf{h}_{t-1}}}^{*}\\\
&={\frac{\partial\mathbf{h}_{t}}{\partial\tilde{\mathbf{h}}_{t}}}\left({\frac{\partial\tilde{\mathbf{h}}_{t}}{\partial\mathbf{r}_{t}}}{\frac{\partial\mathbf{r}_{t}}{\partial\mathbf{h}_{t-1}}}+{\frac{\partial\tilde{\mathbf{h}}_{t}}{\partial\mathbf{h}_{t-1}}}^{*}\right)\\\
&+{\frac{\partial\mathbf{h}_{t}}{\partial\mathbf{s}_{t}}}{\frac{\partial\mathbf{s}_{t}}{\partial\mathbf{h}_{t-1}}}+{\frac{\partial\mathbf{h}_{t}}{\partial\mathbf{h}_{t-1}}}^{*}\end{split}$
(18)

$\begin{split}{\frac{\partial\mathbf{h}_{t}}{\partial\mathbf{h}_{t-1}}}&=(1-\mathbf{s}_{t})T_{1}+T_{2}+\mathbf{s}_{t}\\\
T_{1}&=(\mathbf{U}^{rT}((\mathbf{U}^{hT}(1-\tilde{\mathbf{h}}_{t}\odot\tilde{\mathbf{h}}_{t}))\odot\mathbf{h}_{t-1}\odot\mathbf{r}_{t}\\\
&\odot(1-\mathbf{r}_{t}))+((\mathbf{U}^{hT}(1-\tilde{\mathbf{h}}_{t}\odot\tilde{\mathbf{h}}_{t}))\odot\mathbf{r}_{t})\\\
T_{2}&=\mathbf{U}^{sT}((\mathbf{h}_{t-1}-\tilde{\mathbf{h}}_{t})\odot\mathbf{s}_{t}\odot(1-\mathbf{s}_{t}))\\\
\end{split}$ (19)

So far, we have determined all the components required for $\partial
L_{t}/\partial\mathbf{A}^{s}$. The gradient of $L_{t}$ with respect to other
parameters are similar.

## References

  * Chen and Lin [2014] Xue-Wen Chen and Xiaotong Lin.  Big data deep learning: Challenges and perspectives.  _IEEE Access_ , 2:514–525, 2014.  doi:10.1109/ACCESS.2014.2325029.
  * Najafabadi et al. [2015] Maryam M. Najafabadi, Flavio Villanustre, Taghi M. Khoshgoftaar, Naeem Seliya, Randall Wald, and Edin Muharemagic.  Deep learning applications and challenges in big data analytics.  _Journal of Big Data_ , 2(1), 2015.  doi:10.1186/s40537-014-0007-7.
  * Karim et al. [2019] Fazle Karim, Somshubra Majumdar, Houshang Darabi, and Samuel Harford.  Multivariate lstm-fcns for time series classification.  _Neural Networks_ , 116:237–245, 2019.  ISSN 0893-6080.
  * Gers et al. [2001] Felix A. Gers, Douglas Eck, and Jürgen Schmidhuber.  Applying lstm to time series predictable through time-window approaches.  In Georg Dorffner, Horst Bischof, and Kurt Hornik, editors, _Artificial Neural Networks — ICANN 2001_ , Berlin, Heidelberg, 2001. Springer Berlin Heidelberg.
  * Malhotra et al. [2015] Pankaj Malhotra, Lovekesh Vig, Gautam M. Shroff, and Puneet Agarwal.  Long short term memory networks for anomaly detection in time series.  In _The European Symposium on Artificial Neural Networks_ , 2015.
  * Elman [1990] Jeffrey L. Elman.  Finding structure in time.  _Cognitive Science_ , 14(2):179–211, 1990.  ISSN 0364-0213.  doi:https://doi.org/10.1016/0364-0213(90)90002-E.
  * Abowd et al. [1999] Gregory D. Abowd, Anind K. Dey, Peter J. Brown, Nigel Davies, Mark Smith, and Pete Steggles.  Towards a better understanding of context and context-awareness.  In Hans-W. Gellersen, editor, _Handheld and Ubiquitous Computing_ , pages 304–307, Berlin, Heidelberg, 1999. Springer Berlin Heidelberg.
  * Turney [1996] Peter D. Turney.  The management of context-sensitive features: A review of strategies.  In _Workshop on Learning in Context-Sensitive Domains,at the 13th International Conference on Machine Learning (ICML-96)_ , pages 60–66, 1996\.
  * Dutta Baruah and Muñoz Organero [2023, accepted] Rashmi Dutta Baruah and Mario Muñoz Organero.  Integrating explicit contexts with recurrent neural networks for improving prognostic models.  In _IEEE Aerospace Conference_ , 2023, accepted.
  * Ciskowski and Rafajlowicz [2004] P. Ciskowski and E. Rafajlowicz.  Context-dependent neural nets-structures and learning.  _IEEE Transactions on Neural Networks_ , 15(6):1367–1377, 2004.  doi:10.1109/TNN.2004.837839.
  * Mikolov and Zweig [2012] Tomas Mikolov and Geoffrey Zweig.  Context dependent recurrent neural network language model.  In _2012 IEEE Spoken Language Technology Workshop (SLT)_ , pages 234–239, 2012.  doi:10.1109/SLT.2012.6424228.
  * Wang and Cho [2016] Tian Wang and Kyunghyun Cho.  Larger-context language modelling with recurrent neural network.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1319–1329, Berlin, Germany, August 2016. Association for Computational Linguistics.  doi:10.18653/v1/P16-1125.  URL https://aclanthology.org/P16-1125.
  * Mousa and Schuller [2017] Amr Mousa and Björn Schuller.  Contextual bidirectional long short-term memory recurrent neural network language models: A generative approach to sentiment analysis.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers_ , pages 1023–1032, Valencia, Spain, April 2017. Association for Computational Linguistics.  URL https://aclanthology.org/E17-1096.
  * Smirnova and Vasile [2017] Elena Smirnova and Flavian Vasile.  Contextual sequence modeling for recommendation with recurrent neural networks.  In _Proceedings of the 2nd Workshop on Deep Learning for Recommender Systems_ , DLRS 2017, page 2–9, New York, NY, USA, 2017. Association for Computing Machinery.  ISBN 9781450353533.  doi:10.1145/3125486.3125488.  URL https://doi.org/10.1145/3125486.3125488.
  * Tran et al. [2018] Quan Hung Tran, Tuan Lai, Gholamreza Haffari, Ingrid Zukerman, Trung Bui, and Hung Bui.  The context-dependent additive recurrent neural net.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1274–1283, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.  doi:10.18653/v1/N18-1115.  URL https://aclanthology.org/N18-1115.
  * Wenke and Fleming [2019] Sam Wenke and Jim Fleming.  Contextual recurrent neural networks, 2019.  URL https://arxiv.org/abs/1902.03455.
  * Batbaatar et al. [2018] Erdenebileg Batbaatar, Hyun Woo Park, Dingkun Li, Meijing Li, and Keun Ho Ryu.  Deepenergy: Prediction of appliances energy with long-short term memory recurrent neural network.  In Ngoc Thanh Nguyen, Duong Hung Hoang, Tzung-Pei Hong, Hoang Pham, and Bogdan Trawiński, editors, _Intelligent Information and Database Systems_ , pages 224–234, Cham, 2018. Springer International Publishing.  ISBN 978-3-319-75420-8.
  * Lee and Hauskrecht [2019] Jeong Min Lee and Milos Hauskrecht.  Recent context-aware lstm for clinical event time-series prediction.  _Artificial intelligence in medicine. Conference on Artificial Intelligence in Medicine (2005- )_ , 11526(01):13–23, 2019\.  doi:10.1609/aaai.v33i01.33014983.
  * Johnson et al. [2016] Alistair E.W. Johnson, Tom J. Pollard, Lu Shen, Li wei H. Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G. Mark.  Mimic-iii, a freely accessible critical care database.  _Scientific Data_ , 3:160035, 2016.
  * Belhadi et al. [2020] Asma Belhadi, Youcef Djenouri, Djamel Djenouri, and Jerry Chun-Wei Lin.  A recurrent neural network for urban long-term traffic flow forecasting.  _Applied Intelligence_ , 50(10):3252–3265, 2020\.  doi:10.1007/s10489-020-01716-1.
  * Ma et al. [2021] Dongfang Ma, Xiang Song, and Pu Li.  Daily traffic flow forecasting through a contextual convolutional recurrent neural network modeling inter- and intra-day traffic patterns.  _IEEE Transactions on Intelligent Transportation Systems_ , 22(5):2627–2636, 2021.  doi:10.1109/TITS.2020.2973279.
  * Qu et al. [2021] Licheng Qu, Jiao Lyu, Wei Li, Dongfang Ma, and Haiwei Fan.  Features injected recurrent neural networks for short-term traffic speed prediction.  _Neurocomputing_ , 451:290–304, 2021.  ISSN 0925-2312.  doi:https://doi.org/10.1016/j.neucom.2021.03.054.  URL https://www.sciencedirect.com/science/article/pii/S0925231221004380.
  * Cho et al. [2014] Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  Learning phrase representations using RNN encoder–decoder for statistical machine translation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1724–1734, Doha, Qatar, October 2014. Association for Computational Linguistics.  doi:10.3115/v1/D14-1179.
  * Kingma and Ba [2014] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization, 2014.  URL https://arxiv.org/abs/1412.6980.
  * Zhang et al. [2019] Weiting Zhang, Dong Yang, and Hongchao Wang.  Data-driven methods for predictive maintenance of industrial equipment: A survey.  _IEEE Systems Journal_ , 13(3):2213–2227, 2019\.  doi:10.1109/JSYST.2019.2905565.
  * Shin and Jun [2015] Jong-Ho Shin and Hong-Bae Jun.  On condition based maintenance policy.  _Journal of Computational Design and Engineering_ , 2(2):119–127, 2015.  ISSN 2288-4300.  doi:https://doi.org/10.1016/j.jcde.2014.12.006.
  * Jardine et al. [2006] Andrew K.S. Jardine, Daming Lin, and Dragan Banjevic.  A review on machinery diagnostics and prognostics implementing condition-based maintenance.  _Mechanical Systems and Signal Processing_ , 20(7):1483–1510, 2006.  ISSN 0888-3270.  doi:https://doi.org/10.1016/j.ymssp.2005.09.012.
  * Rezaeianjouybari and Shang [2020] Behnoush Rezaeianjouybari and Yi Shang.  Deep learning for prognostics and health management: State of the art, challenges, and opportunities.  _Measurement_ , 163:107929, 2020.  ISSN 0263-2241.  doi:https://doi.org/10.1016/j.measurement.2020.107929.
  * Wang et al. [2020] Youdao Wang, Yifan Zhao, and Sri Addepalli.  Remaining useful life prediction using deep learning approaches: A review.  _Procedia Manufacturing_ , 49:81–88, 2020.  ISSN 2351-9789.  doi:https://doi.org/10.1016/j.promfg.2020.06.015.  Proceedings of the 8th International Conference on Through-Life Engineering Services – TESConf 2019.
  * Heimes [2008] Felix O. Heimes.  Recurrent neural networks for remaining useful life estimation.  In _2008 International Conference on Prognostics and Health Management_ , pages 1–6, 2008.  doi:10.1109/PHM.2008.4711422.
  * Jia et al. [2018] Xiaodong Jia, Bin Huang, Jianshe Feng, Haoshu Cai, and J. Lee.  A review of phm data competitions from 2008 to 2017: Methodologies and analytics.  In _2018 Annual Conference of the PHM Society_ , volume 10, pages 1–10, 2018.
  * Yuan et al. [2016] Mei Yuan, Yuting Wu, and Li Lin.  Fault diagnosis and remaining useful life estimation of aero engine using lstm neural network.  In _2016 IEEE International Conference on Aircraft Utility Systems (AUS)_ , pages 135–140, 2016.  doi:10.1109/AUS.2016.7748035.
  * Wu et al. [2018] Yuting Wu, Mei Yuan, Shaopeng Dong, Li Lin, and Yingqi Liu.  Remaining useful life estimation of engineered systems using vanilla lstm neural networks.  _Neurocomputing_ , 275:167–179, 2018.  ISSN 0925-2312.  doi:https://doi.org/10.1016/j.neucom.2017.05.063.
  * Zheng et al. [2017] Shuai Zheng, Kosta Ristovski, Ahmed Farahat, and Chetan Gupta.  Long short-term memory network for remaining useful life estimation.  In _2017 IEEE International Conference on Prognostics and Health Management (ICPHM)_ , pages 88–95, 2017.  doi:10.1109/ICPHM.2017.7998311.
  * Li et al. [2018] Xiang Li, Qian Ding, and Jian-Qiao Sun.  Remaining useful life estimation in prognostics using deep convolution neural networks.  _Reliability Engineering and System Safety_ , 172:1–11, 2018.  ISSN 0951-8320.  doi:https://doi.org/10.1016/j.ress.2017.11.021.
  * Listou Ellefsen et al. [2019] André Listou Ellefsen, Emil Bjørlykhaug, Vilmar Æsøy, Sergey Ushakov, and Houxiang Zhang.  Remaining useful life predictions for turbofan engine degradation using semi-supervised deep architecture.  _Reliability Engineering and System Safety_ , 183:240–251, 2019.  ISSN 0951-8320.  doi:https://doi.org/10.1016/j.ress.2018.11.027.
  * Zhao et al. [2019] Sen Zhao, Yong Zhang, Shang Wang, Beitong Zhou, and Cheng Cheng.  A recurrent neural network approach for remaining useful life prediction utilizing a novel trend features construction method.  _Measurement_ , 146:279–288, 2019.  ISSN 0263-2241.  doi:https://doi.org/10.1016/j.measurement.2019.06.004.
  * da Costa et al. [2019] Paulo da Costa, Alp Eren Akçay, Yingqian Zhang, and Uzay Kaymak.  Attention and long short-term memory network for remaining useful lifetime predictions of turbofan engine degradation.  _IJPHM Special Issue on PHM Applications of Deep Learning and Emerging Analytics_ , 10(4):1–12, 2019.
  * Saxena et al. [2008] Abhinav Saxena, Kai Goebel, Don Simon, and Neil Eklund.  Damage propagation modeling for aircraft engine run-to-failure simulation.  In _2008 International Conference on Prognostics and Health Management_ , pages 1–9, 2008.  doi:10.1109/PHM.2008.4711414.