# Enhancing Functional Data Analysis with Sequential Neural Networks:
Advantages and Comparative Study

Jianxiang Zhao
Intelligent Information Processing Laboratory,
Hangzhou Dianzi University, Hangzhou
Jingjing Li
Department of Mathematics and Statistics,
Wake Forest University, Winston-Salem
Minghan Chen
Department of Computer Science,
Wake Forest University, Winston-Salem
and
Sneha Jadhav
Department of Mathematics and Statistics,
Wake Forest University, Winston-Salem  Corresponding author

###### Abstract

Functional Data Analysis (FDA) is a statistical domain developed to handle
functional data characterized by high dimensionality and complex data
structures. Sequential Neural Networks (SNNs) are specialized neural networks
capable of processing sequence data, a fundamental aspect of functional data.
Despite their great flexibility in modeling functional data, SNNs have been
inadequately employed in the FDA community. One notable advantage of SNNs is
the ease of implementation, making them accessible to a broad audience beyond
academia. Conversely, FDA-based methodologies present challenges, particularly
for practitioners outside the field, due to their intricate complexity. In
light of this, we propose utilizing SNNs in FDA applications and demonstrate
their effectiveness through comparative analyses against popular FDA
regression models based on numerical experiments and real-world data analysis.
SNN architectures allow us to surpass the limitations of traditional FDA
methods, offering scalability, flexibility, and improved analytical
performance. Our findings highlight the potential of SNN-based methodologies
as powerful tools for data applications involving functional data.

Keywords: Neural Networks, Functional Data Analysis, Deep Learning

##  1 Introduction

Advancing technology has enabled the collection of substantial data that are
ordered in time, space, and other domains, such as wearable devices tracking
physical activity, blood oxygen level, heart rate at regular time intervals,
and house prices organized by region (space). Multivariate methods although
useful do not take into consideration the main feature of data, i.e., its
ordered/sequential nature. Moreover, correlations in the data pose additional
challenges to multivariate methods. Two approaches, Functional Data Analysis
(FDA) and Sequential Neural Networks (SNNs), have been developed for such
data, each with its own advantages and drawbacks in both theoretical and
practical aspects. The two approaches are currently studied independently as
far as we know. Researchers tend to use one or the other method based on their
backgrounds. For example, in the FDA community, SNNs are rarely considered as
alternative methods for benchmarking and comparison. Given that such ordered
and correlated data arise in crucial applications such as health, we bring
attention to both approaches as well as compare and contrast them. In the
process, we also provide comprehensive and high-level knowledge of both
approaches, which will allow a wider audience to access and leverage their
potential benefits.

Often, plots of sequential data reveal underlying patterns that can be
interpreted as realizations of functions or curves. FDA has evolved over the
past two decades to take advantage of functional characteristics such as
smoothness, shapes, and derivatives among others (Wang et al., 2016). FDA-
based methods treat the collection of observations at different time points as
a single function. This provides a more natural, informative, and holistic
approach to such data. Consider the Berkely growth data containing heights of
children at different ages in Figure 1. Treating the entire data from a child
as a realization of a continuous function allows us to study the rate of
growth, acceleration, the onset of puberty, etc. There has been an enormous
growth of FDA-based methods, offering a plethora of insights that might not be
attainable through multivariate techniques. Of these, functional regression
models have gained considerable popularity. Regression models can be used to
make predictions or determine relationships between variables in diverse
fields, such as health and finance. Though theoretical complexities make
functional regression obscure to practitioners outside of the statistical or
mathematical domains, many popular FDA methods can be applied with the help of
several easy-to-use software packages or codes provided by authors (Morris,
2015). However, even with these aids, unless a practitioner possesses a
profound understanding of FDA and can precisely identify the most suitable
model from the array of available options, they often find themselves
experimenting with various approaches. This need for extensive knowledge in
FDA poses a formidable barrier when working with ordered data sets.

Figure 1: Berkeley Age Study: children’s heights at different ages.

Sequential Neural Networks that use recurrent connections such as Recurrent
Neural Network (RNN) (Rumelhart et al., 1985), Long Short Term Memory (LSTM)
(Hochreiter and Schmidhuber, 1997), and Gated Recurrent Unit (GRU) (Goodfellow
et al., 2016), are classical go-to approaches in machine learning for
analyzing time-series, image, text, and other types of datasets that have
inherent sequential structures. More recently, attention models (Bahdanau et
al., 2014; Niu et al., 2021; Vaswani et al., 2017) have become popular given
their improved performance in handling sequential data. These models possess
the capability to dynamically regulate the attention or weight assigned to
inputs at different times according to their importance and relevance. Massive
attention models with millions of parameters have revolutionized natural
language processing. Functional data that is clearly sequential in nature
typically has a much simpler structure than text data. We explore the
viability of using relatively simpler SNN models such as RNN or LSTM. Though
neural network based models are not known to provide interpretable results,
they are known for their prediction power, particularly in the presence of
large data sets. Despite having complexities on both the theoretical and
computational front, it may be relatively easier to understand the basics of
SNN architecture for a meaningful implementation compared to the FDA-based
regression methods. The network architecture of neural networks may vary
according to the task, but the core components (weights, biases, forward
propagation, optimization) remain largely unchanged making them readily
adaptable for various tasks; while in the domain of statistics and machine
learning, new methods along with their appropriate models, estimations, and
implementation details must be developed from scratch. On the computational
side, the immense advances in deep learning packages facilitate the easy
implementation of different neural network architectures. The same cannot be
said for FDA methodologies as they often require different packages, software,
input/output formats, and most importantly a reasonable understanding of the
methodology. The implementation and interpretation of SNNs are
straightforward, which motivates the exploration of SNNs for functional data
analysis and comparison with FDA-based approaches. We provide brief
explanations and develop simulation studies for a comparison of the two
approaches. We demonstrate that SNNs not only exhibit comparable or superior
performance, but also provide a highly versatile platform to fit different
types of linear, non-linear models with varying dimensions of the input/output
unlike FDA-based methods.

The rest of this paper is organized as follows. Sections 2 and 3 introduce FDA
regression methods and sequential neural networks, respectively. We compare
their performance in various simulation settings in Section 4, and then apply
them to real functional data in Section 5. In Section 5, we conclude with some
remarks and future directions. Our focus is on the basic and common practices
of both approaches, to examine if SNNs can be employed for functional data.

##  2 Functional Data Analysis

In Functional Data Analysis (FDA), data transformation and representation play
a crucial role as they enable the conversion of raw discrete data points into
smooth function curves or finite-dimensional coefficient vectors. This
facilitates subsequent data analysis and modeling processes. Data analysis and
modeling constitute the core steps of FDA, wherein appropriate functional
regression models can be selected to describe the relationship between the
output and input variables. Additionally, methods such as Principal Component
Analysis, Functional Analysis of Variance, and others can be utilized for data
dimensionality reduction and feature extraction, thereby revealing the
underlying structures and patterns within functional data.

###  2.1 Transformation from Discrete Data Points to Smooth Functions

FDA offers methods for analyzing data containing functions. However, obtaining
a complete function curve is not feasible; only discrete data points can be
observed. Many approaches utilize these discrete observations to recover the
underlying function, based on the assumption that the observed values exhibit
typical smoothness in their discrete implementation. Nevertheless, the
observed data is often subject to measurement errors, leading to deviations
from the underlying function curve. Consequently, smoothing plays a crucial
role in FDA by transforming the raw discrete data points into a smoothly
varying function curve, with the aim of minimizing short-term deviations
caused by observation errors such as measurement errors or system noise, while
highlighting patterns within the data.

Suppose, we observe data $Y_{1},...,Y_{T}$, at time points $t_{1},...,t_{T}$.
Let $Y(\cdot)$ denote the underlying function, i.e., $Y(t_{j})=Y_{j},\forall
j.$ Basis expansion postulates that
$Y(t)=\sum_{j=1}^{\infty}\theta_{j}\phi_{j}(t),$ where $\phi_{j}(t),j\geq 1$
are basis functions. In practice, functions can be well approximated using a
small or finite number of basis functions, i.e.,
$Y(t)=\sum_{j=1}^{q}\theta_{j}\phi_{j}(t).$ Thus, for a selected value of $q$
and basis functions, we obtain the model
$Y_{j}=\sum_{i=1}^{q}\theta_{i}\phi_{j}(t_{j}),j=1,...,T.$ This is a
regression model with observed data as response and basis function as
predictors, allowing us to readily obtain estimates
$\widehat{\theta}_{1},...,\widehat{\theta}_{q}$. The function underlying the
observation is approximated as
$\sum_{j=1}^{q}\widehat{\theta}_{j}\phi_{j}(t).$ This is referred to as a
smoothing technique. More advanced versions of the technique using
penalization can be found in Ramsay and Silverman (2008). Several examples of
FDA can be found in Ramsay and Silverman (2008). The function passing through
the data in Figure 1 can be obtained via such a smoothing technique. Many
methodologies in FDA preprocess data to obtain underlying functions before
conducting further analysis.

###  2.2 Selection and Challenges of Functional Regression Models

Functional regression models, similar to their multivariate counterparts, are
determined by factors such as: i) The nature of the dependent (output)
variable, which can be discrete or continuous. ii) The input (independent
variable) output relationship, which can be linear, non-linear, parametric,
non-parametric, or semi-parametric. Regularization and correlated samples add
another dimension that we do not consider here. Most of these methods can be
easily implemented using packages in software such as R, SAS, etc. For
functional regression models, in addition to these factors, we also need to
consider the following factors.

(1) Functional Variables–A functional linear model with both input and output
functions differs from a model with only an input or a function. (2)
Density/Quantity of Discrete Observations–Dense data can be easily recovered
from underlying functions, while sparse data presents challenges. (3) Temporal
Dependency between Input and Output. For example, to study the relationship
between step count and activity intensity, we can examine the relationship
between step count at time $t$ and intensity at the same time. To model the
relationship between daily activity and BMI, we can consider the entire
activity function as the input. Additionally, historical models can be
considered, where the output at time $t$ depends on the entire input history
up to that time.

Even after determining all these components, the treatment of functional data
can vary. Functions exist in an infinite-dimensional space, which means that
functional regression involves estimating parameters that are also functions.
Consequently, functional regression becomes an ill-posed regression problem.
To address this issue, a commonly employed approach is a combination of
spectral decomposition and truncation. Previous works have utilized
regularization techniques to tackle this challenge (Hall and Horowitz, 2007;
Chakraborty and Panaretos, 2016). We have only highlighted a few factors that
need to be considered when selecting a functional regression model. In some
cases, multiple models exist for the same problem, each with its own set of
assumptions and estimation methods. While this plethora of choices increases
the likelihood of finding a model that meets the data requirements, it also
necessitates familiarity with functional data literature. Although certain
methods are accompanied by user-friendly software, this is not always the
case, making it challenging to explore various options for analyzing a
dataset. This is where neural networks possess an advantage, as they provide a
unified platform for implementing multiple neural network architectures. In
the following section, we will provide a brief overview of some widely used
function regression architectures.

###  2.3 Functional Predictor Regression Models

These models have a scalar response $Y$ and a functional regressor
$X(t),t\in\mathcal{I}$. The linear model is written as

$\displaystyle Y=\int_{\mathcal{I}}X(t)\beta(t)+e,$ (1)

where $\beta(\cdot)$ is the model parameter function and $e$ is the model
error. Clearly, this model can be viewed as a generalization of the
multivariate regression model, where the sum is replaced by the integral.
Asymptotics and estimation procedures for this model have been developed
extensively under varying frameworks (Wang et al., 2016; Cardot et al., 2003;
Cai and Yuan, 2012). Model (1) is a functional version of the linear
regression model. However, when dealing with binary data, more sophisticated
models are necessary, such as the semi-parametric model defined as:

$\displaystyle Y=g\left(\int_{\mathcal{I}}X(t)\beta(t)\right)+e.$ (2)

There have been developments in the semi-parametric with known link functions
(James, 2002; Cardot and Sarda, 2005; Dou et al., 2012) as well as unknown
link functions (Chen et al., 2011; Müller and Stadtmüller, 2005). Furthermore,
there is the non-parametric model, formulated as

$\displaystyle E(Y|X)=r(X),$ (3)

where $r$ is an operator estimated using functional kernel (Ferraty and Vieu,
2006). Other types of models are also prevalent, such as additive, single
index ones (McLean et al., 2014; Müller and Yao, 2008; Hall and Horowitz,
2007; Chen et al., 2011). Regularization is another crucial factor influencing
model considerations (Chakraborty and Panaretos, 2016). In the initial stages
of data analysis, it is often challenging to determine which model is best
suited for the data. As a result, exploring several different models becomes
essential, whenever possible. In Table 1, we provide a list of packages
available in R that can be utilized to implement scalar functional regression
models. Additionally, we outline the general categories of models associated
with these packages. We have also listed the broad category of their model.
Note that even if the model type is the same (e.g., linear), the actual model,
estimation methods, or other details may differ between different methods
represented by different functions. Further resources regarding software can
be found in Morris (2015).

Table 1: List of R packages used for function regression models. R Package | Function name | Model Type
---|---|---
fda | fRegress | linear model
fda.usc | fregre.basis, fregre.bootstrap, fregre.lm | linear model
fda.usc | fregre.gkam, fregre.glm, fregre.gsam | linear model
fda.usc | fregre.np | non-parametric model
fdapace | FLM | linear model
fdapace | FAM | additive model
refund | fgam | additive model
refund | fpcr | generalized linear model
fdamixed | fdaLm | linear mixed-effects model

###  2.4 Function on Function Regression Models

In this scenario, both the response denoted as $Y(t),t\in\mathcal{I}$ and the
regressor $X(t),t\in\mathcal{I}$ are functions. There are several types of
models here dictated by the time aspect relationship of the input to the
output. The concurrent model uses the input at time $t$ to predict the output
at the same time $t$. Let $e(\cdot)$ denote error function and
$\alpha(\cdot),~{}\beta(\cdot)$ denote parameters functions. The concurrent
linear model is given as

$\displaystyle Y(t)=\alpha(t)+\beta(t)X(t)+e(t).$ (4)

This is a basic model structure with multiple methods available to estimate
the model parameters (Ramsay and Silverman, 2008; Manrique et al., 2018; Wu et
al., 1998; Şentürk and Müller, 2010). There are models that allow more general
non-linear relationships with varying degrees of complexity and flexibility.
We give three models as examples (Maity and Staicu, 2016; Scheipl et al.,
2015, 2016), shown below.

$\displaystyle Y(t)=\alpha(t)+f(X(t))+e(t),$ (5) $\displaystyle
Y(t)=\alpha(t)+F\\{X(t),t\\}+e(t),$ (6) $\displaystyle
g(E(Y(t)|X(t),t))=F\\{X(t),t\\},$ (7)

where $g(\cdot)$ is a known link function while the remaining functions are
unknown. A large number of regressor functions might call for variable
selection (Goldsmith and Schwartz, 2017). The non-concurrent model uses the
entire input or in some cases a part of it to predict $Y(t)$. The most common
non-concurrent model is the following linear model.

$\displaystyle Y(t)=\alpha(t)+\int_{s\in\mathcal{I}}\beta(t,s)X(s)ds+e(t).$
(8)

Parameter estimation of this model has been addressed in many works, some of
which are Ramsay and Dalzell (1991); Wu and Müller (2011); Besse and Cardot
(1996); Cuevas et al. (2002); Bernardi et al. (2021). The method in Yao et al.
(2005) suggests estimation for the case when data is sparsely observed. A
model based on Tikhonov regularization is proposed in Benatia et al. (2017).
While the literature primarily focuses on linear models, there are also works
that consider non-linear cases. One such model from Scheipl et al. (2016) is

$\displaystyle Y(t)=\int f(X(s),s,t)ds+e(t).$ (9)

Some other non-linear models can be found in Müller and Yao (2008); Ma et al.
(2016). A wide range of models is accessible when both input and output are
functions (Morris, 2015). There are also historical models where past values
of the data are used to predict the current value (Malfait and Ramsay, 2003;
Morris, 2015). For implementing the non-concurrent linear model, several
options are available. One can use fregre.basis.fr function from R package
fda.usc, FLM function from library FDApace, or pffr from library refund. The
latter two offer easier implementations as they mainly require data input from
the user. The pffr function can be used easily to fit concurrent, non-
concurrent, linear, and non-linear models. It also offers options to
incorporate random effects.

###  2.5 Functional Response Regression Models

This framework is relatively less explored compared to the previous ones. In
this case, the only functional variable is the response. A typical model is

$\displaystyle Y(t)=\alpha(t)+X\beta(t)+e(t).$ (10)

To estimate the effect $\beta(t)$ of $X$ on $Y(t)$, a simple and commonly used
strategy suggested in Ramsay and Silverman (2008) is to use the basis
representation, i.e., $\beta(t)=\sum_{j=1}^{q}b_{j}\phi_{j}(t)$. Least squares
along with penalization are used to obtain regularized estimates. Mixed model
approaches in Morris and Carroll (2006); Antoniadis and Sapatinas (2007) treat
the individual curves as random. More options for estimating the parameters
can be found in Reiss et al. (2010); Chiou et al. (2004).

##  3 Sequential Neural Networks

Neural networks are much like lego blocks, different arrangements of the same
blocks yield different architectures that can capture different data
representations. This flexibility is very convenient for data applications.
Basic neural network blocks mainly consist of nodes, layers, and activation
functions. Figure 2 represents a basic neural network with an input layer, two
hidden layers, and an output layer. The nodes in a layer (circles) represent
mathematical operations and the arrows represent the inputs for these
operations. Each node in a layer takes a weighted linear combination of the
input or the output of the previous layer, adds a bias term, and passes the
result ($Z_{j}^{[i]}$) to an activation function ($g_{i}$) that is typically
non-linear. The number of outputs of each layer is determined by its number of
nodes. The output of a neural network is obtained using compositions of
several functions. A typical feed forward neural network with input $x\in
R^{d}$ and $L$ layers is represented as follows

$\displaystyle f(x,\theta)=g_{L}\Bigl{[}W_{L}g_{L-1}\Bigl{(}W_{L-1}\ldots
g_{2}\Bigl{(}W_{2}\bigl{(}g_{1}(W_{1}x+b_{1})+b_{2}\bigr{)}\Bigr{)}+b_{L}\Bigr{)}\Bigr{]}.$
(11)

where the $\theta=(W_{1},...,W_{L},b_{1},...b_{L}),W_{L}\in R^{d_{l}\times
d_{l-1}},d_{0}=d,g_{l}:R^{d_{l}}\to R^{d_{l}}$ are known or predetermined
activation functions (Bartlett et al., 2021). Each layer $i$, transforms the
output of the previous layer as $a_{i}=g_{i}(W_{i}a_{i-1}+b_{i})$ and then
feeds it to the next layer. Hence, the name “feed forward neural network”. The
function $g_{i}(\cdot)$ is implemented component wise. The activation function
determines the nature of the outputs of each layer. Thus, we can use a sigmoid
activation function for a binary classification problem or a non-linear
activation for the regression prediction problem. Thus, neural networks can be
easily adapted to any type of output including multiple outputs with the help
of activation functions and the number of nodes in the output layer. The
weights ($W_{i}$) and biases ($b_{i}$) are obtained to minimize a loss
function that is determined by the problem at hand. For example, mean squared
error loss is used for a continuous output or cross-entropy for binary output.
With enough data, neural networks have been found to work well in obtaining
predictions for many complex data sets (Bartlett et al., 2021). It is common
for neural networks to involve a large number of parameters that may cause an
overfitting issue. Though there are methods to avoid overfitting (Goodfellow
et al., 2016) neural networks typically involve much larger models compared to
statistical methods. Even though neural networks use gradient descent based
methods to solve highly non-convex problems they have a very high predictive
accuracy. In Bartlett et al. (2021), this over parametrization is referred to
as benign overfitting.

Figure 2: A feed forward neural network with two hidden layers.

These feed forward network models are not appropriate for data that has a
sequential structure such as time series, functional data, or even text data
as they do not have a mechanism to consider the order of the data. Sequential
neural networks are a special type of neural networks designed for sequential
data. Their main feature is that they propagate information through ‘time’
with units that have common weights (parameters). Equipped with self
connections, multiple layers, and units, SNNs are able to capture long term
dependencies in the data. Figure 3 shows a basic SNN model, namely an RNN that
adapts to the sequential data structure. The notations along the arrows denote
the weight matrices. For example, $W_{xh}$ is the weight matrix involved
between input and the hidden layer. Note that the weights do not change over
different time points, thus optimal weights capture the time changing relation
between the inputs and outputs. Moreover, SNN can undergo different
morphological changes according to the different task requirements. These
include one-to-many, many-to-one and many-to-many. There are other types of
SNN such as LSTM (Hochreiter and Schmidhuber, 1997), GRU (Cho et al., 2014),
etc. While RNN suffers from short term memory (Bengio et al., 1994) and does
not work when the length of the input sequence is large, LSTM and GRU overcome
this issue to an extent. Given their advantages, neural networks have been
used to model functional data (Conan-Guez et al., 2005; Wang et al., 2020;
Conan-Guez and Rossi, 2002). These works propose a feed forward type of
functional architecture, where the input is a function, and the rest of the
architecture resembles the ‘usual’ feed forward networks. Since it is not
practical to input a function directly, these methods use a finite-dimensional
representation of functions. This approach considers the entire functional
input as a single entity, unlike an SNN that considers the functional data as
a series of inputs observed sequentially. Moreover, these are suitable when
the input is a function whereas the output is not a function. In this work, we
examine the use of SNNs for prediction involving functional data input,
output, or both. In the process, the flexibility of SNNs to model different
types of input/output relationships becomes evident.

Figure 3: Top left: A simple recurrent neural network (RNN) with common weight
at the different time steps. Top right: A different representation of the same
RNN using recurrent connection. Bottom: Different options for inputs/outputs
in RNNs.

Importantly, we are not constrained by restrictive models as in functional
regression. It is easy to change network architecture to model different
relationships. We can fit fairly complex models using high-level but easy-to-
use libraries such as Keras, Tensorflow, etc. For example, Table 2 shows a
code snippet of fitting the SNN model using Keras. First, we establish a
sequential model. Then, we introduce an LSTM layer into the model to
facilitate the processing of sequential data. Next, we add a dense layer to
the model for predicting output results. After adding the layers, we continue
to compile the model, specifying the optimizer, loss function, and evaluation
metrics, among other parameters. Finally, we start the model training process,
in which we input the training data (including input and output) into the
model and perform backpropagation optimization based on the specified
optimizer and loss function. This iterative process aims to gradually converge
the model and obtain suitable parameter configurations.

Table 2: Code snippet of SNN using Keras. Step | Code | Explain
---|---|---
1 | model = Sequential() | Create a Sequential model
2 | model.add(LSTM) | Add a LSTM layer to the model
3 | model.add(Dense) | Add a Dense layer to the model
4 | model.compile(loss, optimizer, metrics) | Compile the model
5 | model.fit(input, label) | Train the model

It can be seen that one advantage of using SNNs for functional data analysis
is that they are easy to implement with existing libraries and frameworks.
These libraries provide high-level APIs that allow users to define, train and
evaluate neural network models with minimal coding. Moreover, these libraries
support multiple input/output scenarios, such as scalar-to-function, function-
to-scalar, function-to-function etc. This means that users can easily adapt
their models to different types of functional data analysis problems, without
having to change the underlying architecture or algorithm. Deep learning
frameworks are difficult to customize, but they have a clear computational
advantage in dealing with large-scale data and complex models, compared to
traditional statistical methods. For example, to customize an LSTM layer in
the Keras framework, one not only needs to master the principles and
mathematical formulas of LSTM, but also needs to use TensorFlow’s low-level
API to implement them, which undoubtedly increases the complexity and time
consumption, and may reduce the performance and compatibility. However, with
the continuous development of artificial intelligence, deep learning
frameworks also provide a rich and diverse set of APIs, which enable us to
easily build the desired models without too much customization.

##  4 Numerical Simulation

We conducted a comparative study of sequential neural networks and functional
data analysis methods for different input functions and output scalars or
functions. By generating functional data with linear and non-linear
relationships, we employed various neural network models and functional
regression models for prediction. These experiments were kept simple to
control the number of factors that might influence the output, which helped
avoid ambiguity and enhance clarity.

###  4.1 Scalar Response and Functional Predictor

####  4.1.1 Continuous Scalar Response

For the experimental setup, the input functional data is generated as
$X_{ij}=X_{i}(t_{j})=\Sigma_{k=1}^{q}x_{ik}\phi_{k}(t_{j}),x_{ij}\sim
N(0,0.5),i=1,...,n,t_{j}\in[0,1],j=1,...,100,$ and $\phi_{k}(\cdot)$
represents Fourier basis functions. Two cases are adopted to generate the
response:

  * •

Linear Case: $Y_{i}=\int X_{i}(t)\beta(t)dt+e_{i},$

  * •

Non-linear Case: $Y_{i}=sin(\int X_{i}(t)\beta(t)dt)+e_{i},$

where $e_{i}\sim N(0,0.5),\beta(t_{j})=\Sigma_{k=1}^{q}(1/k)\phi_{k}(t_{j})$.

We generate a training set of size $n=200$ and a test set of size $n=50$.
Neural networks are typically used for complex data sets and accordingly large
sample sizes are required to train them. We deliberately choose a smaller
sample size to examine its “worst-case performance”. Neural networks with
recurrence such as RNNs and even LSTMs have a memory problem, i.e., they
forget earlier data as the input sequence gets large causing their performance
to deteriorate. With this in mind, we explore their performance when the input
length is considerably large at 100. The number of basis functions $q$
determines the complexity of functions with larger values resulting in more
complex functions. We vary data complexity to investigate its effect. The
following two neural network models are considered.

Sequential Neural Network (SNN): This paper uses the original LSTM layer
(Figure 4a) as the core component of the SNN, which can effectively solve the
long-term dependency problem. Through the Many-to-Many structure, LSTM can not
only extract patterns and rules from past historical information, but also
predict future trends. This paper constructs a two-layer SNN model, the first
layer is the original LSTM layer, containing 32 units, used to capture all the
information of the sequence; the second layer is a hidden layer, used to
produce the required format of output. The SNN model proposed in this paper is
relatively small, but has high prediction accuracy.

Feed forward Neural Network (FNN): This model treats all observations at
different time points as separate variables, which are then fed into a simple
feed forward neural network (Figure 4b). We use two dense hidden layers, each
with 32 units, and then use another dense layer to generate the appropriate
output.

We consider a variety of FDA approaches that are readily available in R. We
examine functional linear models (1), denoted as FLM1 (Goldsmith et al., 2011)
and FLM2 (Yao et al., 2005). FLM1 is implemented using pfr function from
refund library, while FLM2 is implemented using FLM function from R package
fdapace. FNLM denotes functional non-linear model (3) that uses kernel
estimation from Ferraty and Vieu (2006). This is implemented using fregre.np
function from fda.usc library. FAM is the functional additive model from
Scheipl et al. (2016) executed using pfr function from refund library.

The mean squared error for all the methods based on 500 replicates is reported
in Table 3. In the case of simple data (where $q$ represents the complexity of
the data), the functional regression model achieves the minimum error. FNN has
larger errors than SNN, thus making a case for the importance of accounting
for the sequential nature of the data. Overall, SNN demonstrated its
superiority in handling complex data, achieving or surpassing the predictive
performance of the FDA-based model. We observed that as the data complexity
increased ($q$ increased), the performance of the FDA-based method
deteriorated drastically, while the performance of SNN declined more mildly.
Particularly in high-complexity scenarios, SNN outperformed the FDA-based
model significantly. The drop is not as much for SNN. Moreover, the SNN model
we use is quite small and relatively arbitrary so we do expect its performance
to drop with increasing data complexity. However, its performance is easy to
improve with architecture modifications. While the performance of FDA methods
can be improved, it is less straightforward and simple compared to neural
networks.

Table 3: Prediction errors (mean squared errors) for continuous scalar response. $q$ | type | SNN | FNN | FLM1 | FLM2 | FNLM | FAM
---|---|---|---|---|---|---|---
5 | linear | 1.39E-02 | 1.91E-02 | 4.70E-01 | 1.90E-02 | 9.47E-03 | 9.45E-03
9 | linear | 1.61E-02 | 1.97E-02 | 4.53E-01 | 2.00E-02 | 2.66E-02 | 2.62E-02
21 | linear | 2.69E-02 | 2.84E-02 | 4.53E-01 | 2.92E-02 | 6.74E-02 | 6.63E-02
5 | sin | 1.42E-02 | 1.89E-02 | 4.52E-01 | 1.91E-02 | 9.62E-03 | 9.60E-03
9 | sin | 1.57E-02 | 1.94E-02 | 4.37E-01 | 1.96E-02 | 2.33E-02 | 2.28E-02
21 | sin | 2.18E-02 | 2.43E-02 | 4.40E-01 | 2.47E-02 | 6.73E-02 | 6.55E-02

####  4.1.2 Binary Scalar Response

The functional inputs are generated exactly as in Section 3.1.1. We generate
the binary output using the following two cases.

  * •

Linear Case: $P(Y=1)=1/(1+e^{-p}),p=\int X_{i}(t)\beta(t)dt$

  * •

Non-linear Case: $P(Y=1)=1/(1+e^{-p}),p=sin(\int X_{i}(t)\beta(t)dt)$

The simulation settings are the same as the continuous response case. We use
the same neural network architecture as before, i.e., the SNN and FNN models
but with a softmax activation function in the output layer. FLM is functional
linear logistic regression implemented using fregre.glm function from fda.usc
library. Table 4 below shows that SNN-based models perform overall better in
both linear and non-linear cases, demonstrating the advantages of neural
networks. Note that the SNN model can be modified easily in many ways to
improve performance.

Table 4: Prediction errors (mean squared errors) for binary scalar response. $q$ | type | SNN | FNN | FLM
---|---|---|---|---
5 | linear | 0.069 | 0.075 | 0.011
9 | linear | 0.083 | 0.135 | 0.148
21 | linear | 0.105 | 0.361 | 0.207
5 | sin | 0.072 | 0.067 | 0.013
9 | sin | 0.082 | 0.140 | 0.151
21 | sin | 0.170 | 0.405 | 0.230

###  4.2 Functional Response and Functional Predictor

The functional input is the same as before. To facilitate a comparison between
a variety of FDA-based models and neural network based models, we generate the
functional output using a combination of concurrent, non-concurrent, linear,
and non-linear models.

  * •

Concurrent, Linear Case (CL): $Y(t)=\beta(t)X(t)+e(t)$, where function
$\beta(\cdot)$ is generated similar to the previous section and $e(\cdot)$ is
a centred gaussian process.

  * •

Concurrent, Non-linear Case (CNL): $Y(t)=sin(\beta(t)X(t))+e(t)$, where
functions $\beta(\cdot),~{}e(\cdot)$ are generated similar to the previous
section.

  * •

Non-concurrent, Linear Case (NCL): $Y(t)=\int\alpha(t,s)X(s)ds+e(t)$, where
functions $\alpha(t,s)=\beta_{1}(t)\beta_{2}(s),$ and
$\beta_{1}(t)=\Sigma_{k=1}^{q}(k/q)\phi_{k}(t_{j}),~{}\beta_{2}(t)=\Sigma_{k=1}^{q}(q-k+1)/(q)\phi_{k}(t_{j})$
and $\phi_{k}(\cdot)$ represents Fourier basis functions

  * •

Non-concurrent, Non-linear Case (NCNL):
$Y(t)=sin(\int\alpha(t,s)X(s)ds)+e(t)$.

We consider the following two neural network models.

Distributed Response (DR) Model: This model outputs the predicted response
$\widehat{Y}_{j}$ at $j^{th}$ unit (or time step) of the sequential network as
it processes the input (Figure 4c). We use one bi-directional LSTM layer with
32 units followed by a time distributed dense layer (a dense layer that is
repeated at all time points) used with linear activation to obtain the output.

Combined Response (CR) Model: This model employs a bidirectional LSTM to model
the entire input data, and then generates the entire response based on that
information. (Figure 4d). The model adopts a many-to-many scheme, taking the
output sequence of the bidirectional LSTM as the input of the decoder, and
producing the response sequence step by step. The model can fully exploit the
bidirectional contextual information of the input data, and ensure the
temporal consistency of the response sequence.

For FDA-based methods, we use a functional linear model based method FLM1 (8)
from Yao et al. (2005) that is implemented using fdapace library in R. We also
implement FLM2, another functional linear model based method using pffr
function from refund library in R (Scheipl et al., 2016, 2015). The pffr
function allows implementations of several other functional models.
Specifically, we use it to fit a functional non-linear model FNLM1, expressed
as $Y(t)=\int_{s}f(X(s),s,t)+e(t)$. We also use pffr function to implement a
concurrent model with a smooth varying index effect, referred to as FNLM2 and
expressed as $Y(t)=f(X(t),t)+e(t)$, a linear index-varying effect model FLM3,
with the formulation $Y(t)=\alpha(t)+\beta(t)X(t)+e(t)$, and a non-linear
effect model FNLM3 with the expression $Y(t)=f(X(t))+e(t)$.

Table 5 reports the mean squared error of all methods based on 500 replicates.
We find that the SNN-based methods DR and CR outperform most of the FDA-based
models in terms of prediction performance. In particular, CR significantly
surpasses all the FDA-based methods. This advantage is more pronounced in the
long sequence experiment with non-concurrent cases. It is worth noting that DR
and CR are the most common types in SNN, and their performance can be further
enhanced by adjusting the architecture.

(a) SNN model

(b) FNN model

(c) DR model

(d) CR model

Figure 4: SNN model: Consisting of an original LSTM layer and a hidden layer,
using a many-to-many structure to process sequential data. FNN
model:Consisting of two fully connected layers. DR model: Consisting of a
bidirectional LSTM layer and a time-distributed layer, using a many-to-one
structure to process sequential data. CR model: Consisting of a bidirectional
LSTM layer and a hidden layer, using a many-to-many structure to process
sequential data. Table 5: Prediction errors (mean squared errors) for
functional response.

T | 100 | 100 | 100 | 100 | 20 | 20 | 20 | 20
---|---|---|---|---|---|---|---|---
Case | CL | CNL | NCL | NCNL | CL | CNL | NCL | NCNL
DR | $6.23\times 10^{-3}$ | $5.13\times 10^{-3}$ | $2.39\times 10^{-3}$ | $2.32\times 10^{-3}$ | $2.62\times 10^{-2}$ | $1.75\times 10^{-2}$ | $1.83\times 10^{-2}$ | $2.11\times 10^{-2}$
CR | $\mathbf{5.88\times 10^{-4}}$ | $\mathbf{4.67\times 10^{-4}}$ | $\mathbf{2.16\times 10^{-4}}$ | $\mathbf{2.91\times 10^{-4}}$ | $\mathbf{1.52\times 10^{-4}}$ | $\mathbf{1.48\times 10^{-4}}$ | $\mathbf{1.37\times 10^{-4}}$ | $\mathbf{1.43\times 10^{-4}}$
FNN | $4.24\times 10^{-3}$ | $2.77\times 10^{-3}$ | $7.28\times 10^{-4}$ | $4.33\times 10^{-4}$ | $1.06\times 10^{-2}$ | $1.06\times 10^{-2}$ | $1.06\times 10^{-2}$ | $1.06\times 10^{-2}$
FLM1 | $8.89\times 10^{-1}$ | $6.11\times 10^{-1}$ | $2.29\times 10^{-1}$ | $1.94\times 10^{-1}$ | $2.01\times 10^{-1}$ | $2.00\times 10^{-1}$ | $2.01\times 10^{-1}$ | $1.88\times 10^{-1}$
FLM2 | $4.60\times 10^{-3}$ | $2.99\times 10^{-3}$ | $6.61\times 10^{-5}$ | $1.18\times 10^{-4}$ | $3.13\times 10^{-3}$ | $2.14\times 10^{-3}$ | $2.59\times 10^{-3}$ | $2.00\times 10^{-3}$
FLM3 | $1.07\times 10^{-1}$ | $6.78\times 10^{-2}$ | $7.13\times 10^{-3}$ | $4.11\times 10^{-3}$ | $4.21\times 10^{-3}$ | $3.67\times 10^{-3}$ | $3.87\times 10^{-3}$ | $3.52\times 10^{-3}$
FNLM1 | $5.35\times 10^{-1}$ | $3.05\times 10^{-1}$ | $2.48\times 10^{-1}$ | $1.66\times 10^{-1}$ | $1.11\times 10^{-1}$ | $1.09\times 10^{-1}$ | $1.08\times 10^{-1}$ | $1.06\times 10^{-1}$
FNLM2 | $5.35\times 10^{-1}$ | $3.05\times 10^{-1}$ | $2.36\times 10^{-1}$ | $1.64\times 10^{-1}$ | $1.16\times 10^{-1}$ | $1.12\times 10^{-1}$ | $1.12\times 10^{-1}$ | $1.08\times 10^{-1}$
FNLM3 | $5.35\times 10^{-1}$ | $3.05\times 10^{-1}$ | $2.24\times 10^{-1}$ | $1.57\times 10^{-1}$ | $1.12\times 10^{-1}$ | $1.39\times 10^{-1}$ | $1.39\times 10^{-1}$ | $1.35\times 10^{-1}$

##  5 Real-World Application

To validate the conclusions we obtained from the simulation experiments, we
conducted data analyses on two real data sets. We compared the prediction
performance of FDA and neural network methods in realistic scenarios. The
first data analysis utilized the Tecator dataset(Manuel Febrero-Bande, 2022a),
which consists of a scalar response variable and a functional predictor
variable. The second data analysis utilized the Aemet dataset(Manuel Febrero-
Bande, 2022b), which comprises both predictor variables and response variables
related to weather conditions. We compared the prediction errors of different
methods.

###  5.1 Tecator Data: Scalar on Function Regression Dataset

The Tecator dataset is a commonly used example for functional regression with
scalar responses. It encompasses absorbance data across 100 channels from 215
samples of minced meat, along with the proportions of fat, moisture, and
protein, determined through professional chemical analysis. The input data of
absorbance captures discrete points within a certain wavelength range and is
organized in a matrix. Our objective is to utilize near-infrared absorbance
spectra to predict the fat content in minced meat samples, which is a scalar
value.

To evaluate the predictive performance of different methods on functional
regression problems with scalar responses, we conducted the same analysis as
the simulation experiment on the Tecator dataset. We divided the dataset into
a training set of size 200 and a test set of size 15, and repeated the
experiment ten times, taking the mean of the results. Figure 5a shows the
prediction errors of various methods. Among them, SNN and FNN methods are
consistent with the methods introduced in Section 4.1.1. It can be seen that
the FLM1 method based on FDA performs the worst, while the SNN and FNN methods
based on neural networks are significantly better than all methods based on
FDA. Among all methods, SNN method has the lowest prediction error. This
indicates that SNN method can capture the features and changes in complex
function data more effectively than FDA method, indicating that neural network
methods have stronger generalization ability when dealing with complex
function data. Compared with FNN method, SNN method can better utilize the
temporal features in function data, thereby improving prediction accuracy.

###  5.2 Aemet Data: Function on Function Regression Dataset

The AEMET dataset refers to the AEMET OpenData dataset obtained from the
Spanish Meteorological Agency. This dataset is accessible through the R
package. It includes geographical information for each weather station and the
average daily values of temperature, precipitation, and wind speed spanning
from 1980 to 2009. Specifically, we utilize temperature data to predict
logarithmic precipitation data. Both the input and output data are presented
in a matrix format with 73 rows and 365 columns.

We adopted the same analysis method as the simulation experiment in Section
4.2 to conduct a real data experiment on functional regression problems with
functional responses, using the Aemet dataset, which contains function
variables and response variables related to weather conditions. We divided the
dataset into a training set of size 65 and a test set of size 8, and repeated
the experiment ten times, taking the mean of the results.

Figure 5b shows the prediction errors of various methods. Consistent with the
results in Section 5.1, the FLM1 method based on FDA performs the worst. In
contrast, the DR, CR and FNN methods based on neural networks are
significantly better than all methods based on FDA in terms of prediction
performance. This indicates that FDA methods have a significant increase in
prediction error when dealing with high-complexity function data, and may not
be able to effectively capture the intrinsic structure and dynamics of
function data. Among the methods based on neural networks, the CR model using
a many-to-many scheme achieves the best prediction effect, because the many-
to-many scheme can better handle the alignment relationship between input and
output sequences. That is, it can take into account the relevance between
elements in the same position of input and output sequences, rather than
predicting the entire output sequence as a whole based on the entire input
sequence. In this way, it can more accurately capture the changes and patterns
in function data, thereby improving prediction performance. In addition, the
CR model can adapt to input and output sequences of different lengths or
misalignments, while the DR model using a many-to-one scheme requires fixing
or padding input sequences, that is, padding all sequences with 0 to be as
long as the longest sequence.

(a) Tecator

(b) Aemet

Figure 5: Comparison of prediction errors for different methods on the Tecator
and Aemet Datasets.

##  6 Conclusion

This paper systematically investigates the application of sequential neural
networks in the field of function data analysis and comprehensively compares
them with traditional function data analysis methods. Through rigorous
numerical experiments and real-world data applications, the study effectively
demonstrates the advantages of sequential neural networks in handling high-
dimensional and complex data structures in the context of function data, as
well as their efficiency in tasks such as prediction and classification.
Compared to popular function data analysis regression models, the architecture
of sequential neural networks provides greater flexibility, scalability, and
effectiveness. It is worth emphasizing that the sequential neural network
model proposed in this paper can easily replace the original LSTM with
bidirectional LSTM and other variants, and flexibly adjust the activation
function types, the number of neural nodes, and the depth of the network to
meet the specific requirements of different scenarios.

This study offers novel insights and directions for future research in
function data analysis. The sequential neural network demonstrates powerful
and flexible characteristics, making it a potent tool for efficient function
data processing with broad application potential. We hope this research
inspires scholars to explore and innovate in the application of sequential
neural networks for function data analysis, further advancing the field and
achieving remarkable results in both academia and practical applications.

## References

  * Antoniadis and Sapatinas (2007) Anestis Antoniadis and Theofanis Sapatinas.  Estimation and inference in functional mixed-effects models.  _Computational statistics & data analysis_, 51(10):4793–4813, 2007.
  * Bahdanau et al. (2014) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  _arXiv preprint arXiv:1409.0473_ , 2014.
  * Bartlett et al. (2021) Peter L Bartlett, Andrea Montanari, and Alexander Rakhlin.  Deep learning: a statistical viewpoint.  _Acta numerica_ , 30:87–201, 2021.
  * Benatia et al. (2017) David Benatia, Marine Carrasco, and Jean-Pierre Florens.  Functional linear regression with functional response.  _Journal of econometrics_ , 201(2):269–291, 2017\.
  * Bengio et al. (1994) Yoshua Bengio, Patrice Simard, and Paolo Frasconi.  Learning long-term dependencies with gradient descent is difficult.  _IEEE transactions on neural networks_ , 5(2):157–166, 1994.
  * Bernardi et al. (2021) Mauro Bernardi, Antonio Canale, and Marco Stefanucci.  Locally sparse function on function regression.  _arXiv preprint arXiv:2109.13042_ , 2021.
  * Besse and Cardot (1996) Philippe C Besse and Hervé Cardot.  Approximation spline de la prévision d’un processus fonctionnel autorégressif d’ordre 1.  _Canadian Journal of Statistics_ , 24(4):467–487, 1996.
  * Cai and Yuan (2012) T Tony Cai and Ming Yuan.  Minimax and adaptive prediction for functional linear regression.  _Journal of the American Statistical Association_ , 107(499):1201–1216, 2012.
  * Cardot and Sarda (2005) Hervé Cardot and Pacal Sarda.  Estimation in generalized linear models for functional data via penalized likelihood.  _Journal of Multivariate Analysis_ , 92(1):24–41, 2005.
  * Cardot et al. (2003) Hervé Cardot, Frédéric Ferraty, and Pascal Sarda.  Spline estimators for the functional linear model.  _Statistica Sinica_ , pages 571–591, 2003.
  * Chakraborty and Panaretos (2016) Anirvan Chakraborty and Victor M Panaretos.  Hybrid regularisation of functional linear models.  _arXiv preprint arXiv:1610.00951_ , 2016.
  * Chen et al. (2011) Dong Chen, Peter Hall, and Hans-Georg Müller.  Single and multiple index functional regression models with nonparametric link.  _The Annals of Statistics_ , 39(3):1720–1747, 2011.
  * Chiou et al. (2004) Jeng-Min Chiou, Hans-Georg Müller, and Jane-Ling Wang.  Functional response models.  _Statistica Sinica_ , pages 675–693, 2004.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  Learning phrase representations using rnn encoder-decoder for statistical machine translation, 2014.
  * Conan-Guez and Rossi (2002) Brieuc Conan-Guez and Fabrice Rossi.  Multi-layer perceptrons for functional data analysis: a projection based approach.  In _International Conference on Artificial Neural Networks_ , pages 667–672. Springer, 2002.
  * Conan-Guez et al. (2005) Brieuc Conan-Guez, Nicolas Delannay, Fabrice Rossi, and Michel Verleysen.  Representation of functional data in neural networks.  2005\.
  * Cuevas et al. (2002) Antonio Cuevas, Manuel Febrero, and Ricardo Fraiman.  Linear functional regression: the case of fixed design and functional response.  _Canadian Journal of Statistics_ , 30(2):285–300, 2002.
  * Dou et al. (2012) Winston Wei Dou, David Pollard, and Harrison H Zhou.  Estimation in functional regression for general exponential families.  _The Annals of Statistics_ , 40(5):2421–2451, 2012.
  * Ferraty and Vieu (2006) Frédéric Ferraty and Philippe Vieu.  _Nonparametric functional data analysis: theory and practice_ , volume 76.  Springer, 2006.
  * Goldsmith and Schwartz (2017) Jeff Goldsmith and Joseph E Schwartz.  Variable selection in the functional linear concurrent model.  _Statistics in medicine_ , 36(14):2237–2250, 2017\.
  * Goldsmith et al. (2011) Jeff Goldsmith, Jennifer Bobb, Ciprian M Crainiceanu, Brian Caffo, and Daniel Reich.  Penalized functional regression.  _Journal of computational and graphical statistics_ , 20(4):830–851, 2011.
  * Goodfellow et al. (2016) Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  _Deep learning_.  MIT press, 2016.
  * Hall and Horowitz (2007) Peter Hall and Joel L Horowitz.  Methodology and convergence rates for functional linear regression.  _The Annals of Statistics_ , 35(1):70–91, 2007\.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780, 1997.
  * James (2002) Gareth M James.  Generalized linear models with functional predictors.  _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ , 64(3):411–432, 2002.
  * Ma et al. (2016) HaiQiang Ma, Yang Bai, and ZhongYi Zhu.  Dynamic single-index model for functional data.  _Science China Mathematics_ , 59(12):2561–2584, 2016.
  * Maity and Staicu (2016) Arnab Maity and Ana-Maria Staicu.  General functional concurrent model.  Technical report, North Carolina State University. Dept. of Statistics, 2016.
  * Malfait and Ramsay (2003) Nicole Malfait and James O Ramsay.  The historical functional linear model.  _Canadian Journal of Statistics_ , 31(2):115–128, 2003.
  * Manrique et al. (2018) Tito Manrique, Christophe Crambes, and Nadine Hilgert.  Ridge regression for the functional concurrent model.  _Electronic Journal of Statistics_ , 12(1):985–1018, 2018.
  * Manuel Febrero-Bande (2022a) Manuel Oviedo Manuel Febrero-Bande.  tecator: tecator data.  Online, October 17 2022a.  URL https://rdrr.io/cran/fda.usc/man/tecator.html.
  * Manuel Febrero-Bande (2022b) Manuel Oviedo Manuel Febrero-Bande.  aemet: aemet data.  Online, October 17 2022b.  URL https://rdrr.io/cran/fda.usc/man/tecator.html.
  * McLean et al. (2014) Mathew W McLean, Giles Hooker, Ana-Maria Staicu, Fabian Scheipl, and David Ruppert.  Functional generalized additive models.  _Journal of Computational and Graphical Statistics_ , 23(1):249–269, 2014.
  * Morris (2015) Jeffrey S Morris.  Functional regression.  _Annual Review of Statistics and Its Application_ , 2:321–359, 2015.
  * Morris and Carroll (2006) Jeffrey S Morris and Raymond J Carroll.  Wavelet-based functional mixed models.  _Journal of the Royal Statistical Society: Series B (Statistical Methodology)_ , 68(2):179–199, 2006.
  * Müller and Stadtmüller (2005) Hans-Georg Müller and Ulrich Stadtmüller.  Generalized functional linear models.  _the Annals of Statistics_ , 33(2):774–805, 2005\.
  * Müller and Yao (2008) Hans-Georg Müller and Fang Yao.  Functional additive models.  _Journal of the American Statistical Association_ , 103(484):1534–1544, 2008.
  * Niu et al. (2021) Zhaoyang Niu, Guoqiang Zhong, and Hui Yu.  A review on the attention mechanism of deep learning.  _Neurocomputing_ , 452:48–62, 2021.
  * Ramsay and Dalzell (1991) James O Ramsay and CJ1125714 Dalzell.  Some tools for functional data analysis.  _Journal of the Royal Statistical Society: Series B (Methodological)_ , 53(3):539–561, 1991.
  * Ramsay and Silverman (2008) James O Ramsay and Bernhard W Silverman.  Functional data analysis.  _İnternet Adresi: http_ , 2008.
  * Reiss et al. (2010) Philip T Reiss, Lei Huang, and Maarten Mennes.  Fast function-on-scalar regression with penalized basis expansions.  _The international journal of biostatistics_ , 6(1), 2010\.
  * Rumelhart et al. (1985) David E Rumelhart, Geoffrey E Hinton, Ronald J Williams, et al.  Learning internal representations by error propagation, 1985.
  * Scheipl et al. (2015) Fabian Scheipl, Ana-Maria Staicu, and Sonja Greven.  Functional additive mixed models.  _Journal of Computational and Graphical Statistics_ , 24(2):477–501, 2015.
  * Scheipl et al. (2016) Fabian Scheipl, Jan Gertheiss, and Sonja Greven.  Generalized functional additive mixed models.  _Electronic Journal of Statistics_ , 10(1):1455–1492, 2016.
  * Şentürk and Müller (2010) Damla Şentürk and Hans-Georg Müller.  Functional varying coefficient models for longitudinal data.  _Journal of the American Statistical Association_ , 105(491):1256–1264, 2010.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Wang et al. (2016) Jane-Ling Wang, Jeng-Min Chiou, and Hans-Georg Müller.  Functional data analysis.  _Annual Review of Statistics and Its Application_ , 3:257–295, 2016.
  * Wang et al. (2020) Shuoyang Wang, Guanqun Cao, and Zuofeng Shang.  Estimation of the mean function of functional data via deep neural networks.  _arXiv preprint arXiv:2012.04573_ , 2020.
  * Wu et al. (1998) Colin O Wu, Chin-Tsang Chiang, and Donald R Hoover.  Asymptotic confidence regions for kernel smoothing of a varying-coefficient model with longitudinal data.  _Journal of the American statistical Association_ , 93(444):1388–1402, 1998.
  * Wu and Müller (2011) Shuang Wu and Hans-Georg Müller.  Response-adaptive regression for longitudinal data.  _Biometrics_ , 67(3):852–860, 2011.
  * Yao et al. (2005) Fang Yao, Hans-Georg Müller, and Jane-Ling Wang.  Functional linear regression analysis for longitudinal data.  _The Annals of Statistics_ , 33(6):2873–2903, 2005.