#  LongChecker: Improving scientific claim verification by modeling full-
abstract context

David Wadden† Kyle Lo‡ Lucy Lu Wang‡
Arman Cohan‡ Iz Beltagy‡ Hannaneh Hajishirzi†‡
† University of Washington, Seattle, WA, USA
‡ Allen Institute for Artificial Intelligence, Seattle, WA, USA
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

We introduce the LongChecker system for scientific claim verification. Given a
scientific claim and an evidence-containing research abstract, LongChecker
predicts a veracity label and identifies supporting rationales in a multitask
fashion based on a shared encoding of the claim and abstract. We perform
experiments on the SciFact dataset, and find that LongChecker achieves state-
of-the-art performance. We conduct analysis to understand the source of this
improvement, and find that identifying the relationship between a claim and a
rationale reporting a scientific finding often requires understanding the
_context_ in which the rationale appears. By making labeling decisions based
on all available context, LongChecker achieves better performance on cases
requiring this type of understanding. In addition, we show that LongChecker is
able to leverage weakly-supervised in-domain data to facilitate few-shot
domain adaptation for scientific claim verification.111Our code and model
checkpoints are available at https://github.com/dwadden/longchecker.

##  1 Introduction

The task of scientific claim verification requires a system to assess the
veracity of a scientific claim against a corpus of documents. The
proliferation of mis- and dis-information on the web – particularly as it
relates the COVID-19 pandemic – has spurred the development of new datasets
and modeling approaches for this task.

In this work, we focus on the SciFact task and dataset (Wadden et al., 2020).
Substantial modeling progress has been made on SciFact since its release,
facilitated by techniques like neural re-ranking for document retrieval
Pradeep et al. (2021), and methods that incorporate document-level context for
rationale selection Li et al. (2021); Zhang et al. (2021). One important
commonality among existing models is that they make label predictions by first
selecting a small number of rationales (i.e. sentences reporting results that
entail or contradict the claim) from a larger document context, and then
predict a veracity label based on the selected rationales. In contrast, the
model we present in this work predicts fact-checking labels based on an
encoding of the full title and abstract context.

Our approach offers two advantages. First, scientific writing makes heavy use
of acronyms, qualifiers, coreference, and anaphora Nye et al. (2020); Barnett
and Doubleday (2020). As a result, the relationship between a claim and a
sentence reporting a scientific finding is often only clear when the finding
is interpreted in its original context; Figure 1 provides an example. A manual
analysis of a collection of claims from the SciFact dataset indicates that
this situation occurs in roughly two-thirds of claim / evidence pairs in
SciFact.

Figure 1: A claim from the SciFact dev set, supported by a research abstract.
The sentence in green reports a finding that Supports the claim, but this
finding cannot be interpreted properly without the context in blue.
LongChecker incorporates the full context of the evidence-containing title and
abstract when predicting fact-checking labels.

Second, while fact-checking annotations in the scientific domain are expensive
and require expert annotators, unlabeled scientific papers are extremely
plentiful. In addition, it is possible to devise simple heuristics to weakly
label documents that Support or Refute a given claim – for instance, a paper
abstract is likely to Support a claim made in its title. LongChecker can be
trained directly on these weakly-supervised claim / abstract pairs, whereas
models which first select rationales as inputs for label prediction would
require some additional form of supervision to identify the rationales.

We make the following contributions:

  1. 1.

We introduce the LongChecker model. LongChecker verifies claims against
scientific research abstracts by encoding the full claim, title, and abstract
together, and then predicts a veracity level and identifies rationales in a
multitask fashion.

  2. 2.

We find that LongChecker establishes a new state-of-the-art on the SciFact
task.

  3. 3.

We perform analysis to understand the source of this improvement, and find
that it is due primarily to LongChecker’s better performance on instances
requiring abstract-level context.

  4. 4.

We demonstrate that LongChecker can leverage weakly-labeled in-domain data
during training, facilitating zero-shot and few-shot domain adaptation for
scientific fact-checking.

  5. 5.

We find that both LongChecker—as well as two other strong baselines on
SciFact—achieve performance close to the “upper bound” set by human agreement,
suggesting the need for more challenging datasets for scientific claim
verification.

##  2 Related work

We review literature on fact-checking and claim verification in the scientific
domain.

###  2.1 Datasets for scientific claim verification

Since 2020, a number of datasets have been released to facilitate the
development of models for scientific claim verification. PubHealth Kotonya and
Toni (2020) collects public health claims from fact-checking websites like
Snopes, and verifies them against information in news articles and fact-
checking websites.

COVID-Fact Saakyan et al. (2021) collects claims about COVID-19 scraped from a
COVID-19 subreddit, and verifies them against linked scientific papers, as
well as documents retrieved via Google search. HealthVer Sarrouti et al.
(2021) also focuses on COVID-related claims, which are obtained by re-writing
answers to questions from TREC-COVID (Voorhees et al., 2020), and verifies
them against documents retrieved from the CORD-19 corpus Wang et al. (2020).
Both COVID-Fact and HealthVer are formatted as entailment-style tasks, pairing
claims with the rationales that entail or contradict them.

The SciFact dataset collects claims by re-writing citation sentences appearing
in scientific articles, and verifies them against the abstracts of the
articles mentioned in each citation (Wadden et al., 2020). The SciFact dataset
includes the full abstracts which Support and Refute each claim, annotated
with the rationales reporting the findings which entail each label (in the
context of the abstract). In this work, we focus on SciFact because it (1)
focuses on verifying claims about scientific findings against primary research
literature, and (2) includes the full context associated with each fact-
checking label.

###  2.2 Models for the SciFact task

Substantial modeling progress has been made on the SciFact task since its
introduction, driven in part by a shared task (called SciVer) and leaderboard
Wadden and Lo (2021). The two strongest systems on the shared task were
Vert5Erini Pradeep et al. (2021) and ParagraphJoint Li et al. (2021), which we
describe in detail in §5.1. More recently, ARSJoint Zhang et al. (2021)
achieved performance competitive with these top two systems.

These models all employ a three-stage approach consisting of (1) abstract
retrieval, (2) rationale selection, and (3) label prediction using the
selected rationales. They differ in how they select rationales. While
Vert5Erini makes rationale selection decisions for each sentence
independently, ParagraphJoint and ARSJoint encode all sentences in the source
abstract (truncating as necessary to fit within model limits), and use pooling
and self-attention to produce sentence representations used for rationale
selection.

Our proposed system, LongChecker, uses a document-level encoding approach like
ParagraphJoint and ARSJoint. Unlike those systems, it uses this encoding to
make label predictions directly, rather than using it to select evidence to
serve as downstream inputs for label prediction. It also uses a long-document
transformer, Longformer Beltagy et al. (2020), to avoid the need for abstract
truncation. Long-document transformers have been used for general-domain fact-
checking on the Fever dataset Thorne et al. (2018): Stammbach (2021) uses Big
Bird Zaheer et al. (2020) to encode Wikipedia articles for token-level
evidence selection.

##  3 Scientific claim verification

We briefly review the scientific claim verification task as defined in
SciFact. For more details, see Wadden et al. (2020).

#### Task definition

Given a claim $c$ and a corpus of scientific research abstracts
${\mathcal{A}}$, the scientific claim verification task consists of
identifying all _evidence abstracts_ ${\mathcal{E}}\in{\mathcal{A}}$ relevant
to $c$, and for each abstract $a\in{\mathcal{E}}$ predicting a _label_
$y(c,a)$ which indicates the relationship (either Supports or Refutes) of $a$
to $c$. All abstracts $a^{\prime}\in{\mathcal{E}}$ that do not contain
relevant evidence are labeled $y(c,a^{\prime})$ = NEI (Not Enough Info). The
system must also identify _rationales_ which report the findings sufficient to
justify each label, in the context of the abstract.

#### Evaluation metrics

We perform two styles of evaluation. _Abstract-level_ evaluation is similar to
the Fever score Thorne et al. (2018); it measures a model’s ability to
identify evidence-containing abstracts, and label their relationship (Supports
or Refutes) relative to a claim. The _label-only_ variant of this evaluation
only requires a model to predict the correct veracity label, whereas the
_label+rationale_ variant requires the model to also select supporting
rationales; this latter metric rewards high recall in selecting rationales.

_Sentence-level_ evaluation, on the other hand, rewards high precision in
selecting rationales. The _selection-only_ variant measures the model’s
ability to identify rationales, while the _selection+label_ variant requires
that the model also predict the correct veracity label.

#### Retrieval settings

We evaluate models in two settings: in the _oracle abstract_ setting, models
are provided with “gold” evidence-containing abstracts. In the _open_ setting,
models must first retrieve relevant documents from the corpus ${\mathcal{A}}$,
which consists of 5,183 abstracts.

#### What counts as a rationale?

Gold rationales include all sentences containing a statement of a _result_
that entails or contradicts the claim. However, they may not contain all
_context_ necessary to interpret the result; for instance, context outside the
rationales may be required to resolve coreferential expressions or acronyms,
or may contain qualifiers specifying experimental context or study population.
This convention is consistent with related tasks in rationalized NLP for
biomedical literature, such as Lehman et al. (2019); DeYoung et al. (2020). We
will explore this issue further in §6.

##  4 LongChecker: Modeling full abstracts

We develop a model for full-abstract claim verification, which consists of two
stages: (1) retrieval of relevant abstracts, and (2) multitask prediction of
labels and rationales.

###  4.1 Retrieval

In the open retrieval setting, models must first retrieve documents likely to
contain evidence. To accomplish this, we use the retrieval model from the
Vert5Erini Pradeep et al. (2021) system, the top-performing retrieval system
on SciVer. This model first retrieves abstracts using BM25 Robertson and
Zaragoza (2009), then refines the predictions using a neural re-ranker based
on Nogueira et al. (2020), which is trained on the MS MARCO passage dataset
Campos et al. (2016).222We performed some exploratory experiments using DPR
trained on Fever Karpukhin et al. (2020), but found that the two-stage model
enjoyed better performance. This is consistent with the findings from Thakur
et al. (2021).

###  4.2 Multi-task label prediction and rationale selection

Prior work on related tasks involving long-document context – such as question
answering over NLP papers Dasigi et al. (2021) and multi-hop QA Beltagy et al.
(2020) – has demonstrated that models can benefit from shared representations
learned on the tasks of (1) predicting a final, document-level label or answer
and (2) identifying the relevant information necessary to arrive at the final
label. Motivated by this observation, we develop LongChecker, which cross-
encodes a given claim and candidate document (both title and abstract), and
uses this representation to make label and rationale predictions in a
multitask fashion.

#### Long-document encoding

Since many abstracts in our corpus exceed the 512-token length limit common to
many transformer-based language models like BERT (Devlin et al., 2019) and
RoBERTa (Liu et al., 2019), we leverage the Longformer model Beltagy et al.
(2020) to cross-encode the claim, title, and abstract.333We use a Longformer-
large checkpoint pretrained on the S2ORC corpus Lo et al. (2020).

Given a claim $c$, title $t$, and abstract $a$ consisting of sentences
$s_{1},\dots,s_{n}$, we concatenate the inputs separated by </s> tokens. The
</s> token following each sentence $s_{i}$ is notated
as$\>\texttt{</s>}_{i}\>$:

$\>\texttt{<s>}\>c\>\texttt{</s>}\>t\>\texttt{</s>}\>s_{1}\>\texttt{</s>}_{1}\>\dots
s_{n}\>\texttt{</s>}_{n}\>$

We encode this sequence using Longformer, assigning global attention to the
<s> token, as well as all tokens in the claim $c$ and all </s> tokens.

#### Multi-task prediction and loss

We predict a label
$\widehat{y}(c,a)\in\\{\textsc{Supports},\textsc{Refutes},\textsc{NEI}\\}$ by
adding a classification head over the encoding of the <s> token, consisting of
two feedforward layers followed by a three-way softmax. We predict whether
sentence $s_{i}$ is a rationale by adding a similar two-way classification
head on top of$\>\texttt{</s>}_{i}\>$.

During training, we compute the cross-entropy losses for the label and
rationale predictions, and train to minimize the multitask loss:

$L=L_{\textrm{label}}+\lambda_{\textrm{rationale}}L_{\textrm{rationale}},$

where $\lambda_{\textrm{rationale}}$ is tuned on the dev set.

###  4.3 Two-stage finetuning for domain adaptation

Collecting fact-checking annotations in scientific domains like biomedicine or
public health is challenging and time-consuming, even for expert annotators
with domain expertise. Given the availability of large general-domain fact-
checking datasets like Fever Thorne et al. (2018), domain adaptation
techniques to shift these models to new domains with minimal in-domain
annotations could facilitate the development of fact-checking models for new
subfields of science, or other specialized domains.

With this motivation in mind, we develop a two-stage finetuning procedure. In
Stage 1, we finetune Longformer on a combination of one human-labeled, out-of-
domain fact verification dataset, and two weakly-labeled in-domain datasets.
In Stage 2, we continue finetuning on the SciFact dataset, using additional
“hard” negatives also obtained from S2ORC. In §5, we conduct ablations to
assess the benefits of incorporating weakly-labeled in-domain data into Stage
1 finetuning.

#### Stage 1: Finetuning on out-of-domain and weakly-labeled in-domain data

We finetune LongChecker on a combination of 136,564 supervised, out-of-domain
claims from the Fever fact-checking dataset, together with two sources of
weakly-supervised in-domain data derived from the EvidenceInference and
PubmedQA datasets.

The EvidenceInference dataset Lehman et al. (2019); DeYoung et al. (2020) was
released to facilitate the development of systems capable of extracting
results from clinical trial reports. Clinical trial reports are a specific
type of research paper, which examines the effect of an _intervention_ on an
_outcome_ , relative to a _comparator_ (or baseline). This dataset contains
annotations consisting of “ICO” (intervention / comparator / outcome)
_prompts_ paired with (1) labels indicating whether the outcome _increased_ or
_decreased_ as a result of the intervention, and (2) rationales justifying
each label. We use rule-based heuristics to convert these prompts into claims
(see Figure 2 for an example). This process yields 7,395 weakly-labeled
claims.

Figure 2: An example showing how an evidence inference prompt (top) can be
converted into a claim (bottom) using templates. A refuted claim could be
generated by substituting “increases” for “decreases” in the prompt text.

We also use the PQA-A subset of PubmedQA Jin et al. (2019), a large collection
of biomedical abstracts with “claim-like” titles – for instance, “Vitamin B6
supplementation increases immune responses in critically ill patients.” We
treat the paper titles as claims and the corresponding abstracts as the
evidence sources. Since abstracts presumably contain evidence supporting their
titles, we generate additional contradictory claims by identifying titles
featuring a negation and removing the negation words (e.g. replacing “does
not” with “does”). In total, PQA-A provides us with an additional 58,375
claims. Since these instances are not annotated with rationales, existing
models for scientific claim verification cannot utilize them for training
without first generating weakly-labeled rationales. Using a multitask approach
eliminates this requirement; we simply set $\lambda_{\textrm{rationale}}=0$ in
our multitask loss function when training on these instances.

#### Stage 2: Finetuning on SciFact

We continue finetuning on the claim / evidence pairs from the SciFact dataset.
Based on the findings from Li et al. (2021), we train on additional negative
examples – i.e. claim / abstract pairs that do not have any relationship. We
create a search index from 500K abstracts randomly selected from the
biomedical subset of the S2ORC corpus. For each claim, we obtain “hard”
negative abstracts by using the retrieval system from §4.1 to retrieve the
top-1000 most-similar abstracts from this index, removing abstracts that are
annotated as containing evidence, and randomly sampling 20 abstracts to be
used as negatives during training.

##  5 Experiments

We conduct the following experiments: (1) We evaluate the performance of
LongChecker on the SciFact test set, relative to two state-of-the-art
baselines on SciFact. (2) We assess the performance of LongChecker and the
baselines relative to the “upper bound” imposed by human agreement. (3) We
examine the “zero-shot” and “few-shot” generalization performance of
LongChecker when trained on Fever alone, compared to its performance when also
trained on weakly-labeled in-domain data as described in §4.3.

###  5.1 Baseline systems

We compare LongChecker to the two models which achieved the best performance
on the SciVer shared task.

####  Vert5Erini

Pradeep et al. (2021) uses BM25 followed by neural re-ranking step for
retrieval. For rationale selection, it makes a separate decision for each
sentence, conditioned on the claim $c$ but not the surrounding abstract
context. The selected rationales are used as inputs to make a final label
decision. As T5-3B is used as both the rationale selection and label
prediction models, this system has roughly 10x as many parameters as
LongChecker.

####  ParagraphJoint

Li et al. (2021) performs retrieval using representations derived from
BioSentVec Chen et al. (2019). Then, it uses RoBERTa-large to encode the title
and abstract, truncating sentences as necessary to fit the 512-token limit.
Rationale labels are predicted for each sentence based on the sentence’s
encoding in the context of the abstract. As with Vert5Erini, the selected
rationales are given as input to another RoBERTa-large model which makes the
final label decision. This model has roughly the same number of parameters as
LongChecker.

|  | Sentence-Level | Abstract-Level
---|---|---|---
|  | Selection-Only | Selection+Label | Label-Only | Label+Rationale
|  | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1
Retrieval | Model |  |  |  |  |  |  |  |  |  |  |  |
Oracle abstract | Vert5Erini | 87.9 | 68.6 | 77.1 | 79.6 | 62.2 | 69.8 | 90.7 | 74.3 | 81.7 | 87.4 | 71.6 | 78.7
ParagraphJoint | 88.7 | 63.8 | 74.2 | 76.7 | 55.1 | 64.1 | 87.2 | 64.4 | 74.1 | 84.8 | 62.6 | 72.0
LongChecker | 90.1 | 78.6 | 84.0 | 80.5 | 70.3 | 75.0 | 87.4 | 75.2 | 80.9 | 85.9 | 73.9 | 79.4
Human | 70.7 | 70.7 | 70.7 | 67.4 | 67.4 | 67.4 | 94.8 | 84.1 | 89.1 | 90.3 | 80.1 | 84.9
Open | Vert5Erini | 63.0 | 69.2 | 66.0 | 60.6 | 66.5 | 63.4 | 64.0 | 73.0 | 68.2 | 62.8 | 71.6 | 67.0
ParagraphJoint | 79.9 | 63.2 | 70.6 | 68.9 | 54.6 | 60.9 | 75.8 | 63.5 | 69.1 | 73.7 | 61.7 | 67.2
LongChecker | 74.5 | 74.1 | 74.2 | 67.4 | 67.0 | 67.2 | 73.8 | 71.2 | 72.5 | 72.4 | 69.8 | 71.1
(a) LongChecker establishes a new state-of-the-art on the SciFact task. In the
“oracle abstract” setting where gold abstracts are provided, state-of-the-art
models surpass human performance as measured by sentence-level evaluation, and
approach human performance for abstract-level evaluation. For human agreement,
sentence-level precision and recall are (coincidentally) identical.

|  | Sentence-Level | Abstract-Level
---|---|---|---
|  | Selection-Only | Selection+Label | Label-Only | Label+Rationale
|  | P | R | F1 | P | R | F1 | P | R | F1 | P | R | F1
Setting | Data |  |  |  |  |  |  |  |  |  |  |  |
Zero-shot | Fever | 77.4 | 6.5 | 12.0 | 74.2 | 6.2 | 11.5 | 83.9 | 11.7 | 20.5 | 74.2 | 10.4 | 18.2
Fever \+ Adapt | 81.0 | 13.8 | 23.6 | 68.2 | 11.6 | 19.9 | 74.6 | 19.8 | 31.3 | 69.5 | 18.5 | 29.2
Few-shot | Fever | 46.3 | 38.9 | 42.3 | 44.0 | 37.0 | 40.2 | 62.3 | 53.6 | 57.6 | 52.9 | 45.5 | 48.9
Fever \+ Adapt | 48.1 | 46.8 | 47.4 | 44.2 | 43.0 | 43.6 | 61.5 | 61.3 | 61.4 | 52.9 | 52.7 | 52.8
(b) Zero-shot and few-shot experiments on SciFact. We present results in the
“Open retrieval” setting; the results in the “Oracle” setting are
qualitatively similar. Models are pre-trained either on Fever, or Fever
combined with EvidenceInference and PubmedQA for domain adaptation (indicated
as “Fever \+ Adapt”). In the “zero-shot” setting, models are not finetuned on
any SciFact examples, while in the “few-shot” setting, they are finetuned on
45 examples from SciFact.

Table 1: Performance on the SciFact task.

###  5.2 Results

####  LongChecker establishes a new state-of-the-art on SciFact

Table 1(a) shows the performance of LongChecker and the two baselines on the
original SciFact task. We present results in both the “oracle” setting where
gold abstracts are provided as inputs, and the original SciFact setting, where
evidence must be retrieved from a corpus of 5K abstracts. In both retrieval
settings, LongChecker outperforms previous systems (including Vert5Erini,
which is an order of magnitude larger) as measured by both “sentence-level”
and “abstract-level” evaluation. We explore the source of this improvement in
Section 6.

####  LongChecker and baselines achieve near human-level performance

During the collection of the SciFact dataset, 151 claim-evidence pairs were
independently annotated by two different SciFact annotators. We obtain an
estimate of “human-level” performance by treating the first annotator’s
results as “gold”, and the second annotator’s results as predictions. The
results in Table 1(a) show that, as measured by sentence-level evaluation,
existing systems surpass human agreement when “oracle” abstracts are provided,
and match it with retrieved abstracts. For abstract-level evaluation, human
agreement still exceeds the best systems, but the gap is only 5.5 F1 in the
“oracle abstract” setting. It is understandable that human agreement is lower
on sentence-level evaluation, since the task of selecting rationales is
inherently more subjective than that of assigning an overall label.
Collectively, these results suggest that existing models are capable of very
strong performance on the fact verification sub-task of predicting whether an
abstract (or collection of rationales) Supports or Refutes a claim, given that
the abstract is known to contain relevant information (i.e. given oracle
retrieval). Saakyan et al. (2021) and Sarrouti et al. (2021) report similarly
strong performance (roughly 80 F1) on the HealthVer and COVID-Fact tasks,
suggesting that scientific fact-checking research stands to benefit from new
tasks and datasets designed to challenge the abilities of existing models.

Category | Example
---|---
Context (Acronym) |  Claim: |  Hematopoietic stem cells segregate their chromosomes randomly.
|  Context: |  _we tested these hypotheses in hematopoietic stem cells (HSCs)…_
|  Evidence: |  _…indicated that all HSCs segregate their chromosomes randomly._
|  Explanation: |  HSCs is an acronym for Hematopoietic stem cells.
Context (Coreference) |  Claim: |  Errors in peripheral IV drug administration are most common during bolus administration
|  Context: |  _OBJECTIVES: To determine the incidence of errors in the administration of intravenous drugs …_
|  Evidence: |  _…Most errors occurred when giving bolus doses_
|  Explanation: |  The evidentiary sentence reporting the finding does not specify the type of error.
Background |  Claim: |  Silencing of Bcl2 slows the progression of tumors.
|  Evidence: |  _Eliminating BCL-2 yielded rapid loss of leukemic cells_
|  Explanation: |  Leukemic cells contribute to tumor progression.
Numerical |  Claim: |  1/2000 in UK have abnormal PrP positivity.
|  Evidence: |  _Of the 32,441 appendix samples, 16 were positive for abnormal PrP_
|  Explanation: |  Verifying the claim requires recognizing that 16 / 32,441 is roughly 1 / 2000.
Table 2: Example claim-evidence pairs demonstrating a type of modeling capability. |  | Abstract Label+Rationale |
---|---|---|---
|  | Count | ParagraphJoint | Vert5Erini | LongChecker | $\Delta_{\textsc{LongChecker}}$
Category | Difficulty |  |  |  |  |
Context | No | 43 | 85.0 | 85.4 | 78.0 | -7.1
Yes | 85 | 71.7 | 72.5 | 77.1 | 5.0
Background | No | 106 | 78.1 | 75.4 | 80.0 | 3.3
Yes | 22 | 68.4 | 85.0 | 65.0 | -11.7
Numerical | No | 106 | 74.9 | 76.4 | 77.1 | 1.4
Yes | 22 | 84.2 | 80.0 | 79.1 | -3.0
All | All | 128 | 76.4 | 77.1 | 77.4 | 0.7
Table 3: Breakdown of performance for claims / evidence pairs requiring
different modeling capabilities. $\Delta_{\textsc{LongChecker}}$ indicates the
difference between the average of the two LongChecker models, and the two
baseline systems; higher values indicate better relative performance of
LongChecker.

#### Weakly-labeled in-domain training data facilitates domain adaptation

We examine the effectiveness of our strategy to leverage weakly-labeled in-
domain data to improve few-shot generalization. In particular, we compare two
variants of Stage 1 finetuning from §4.3: one in which we finetune only on
Fever, and one in which we finetune on Fever, along with weakly-supervised in-
domain data from EvidenceInference and PQA-A. We evaluate these models in two
settings. In the “zero-shot” setting, we do not finetune on any SciFact data
(i.e. we skip Stage 2 finetuning from §4.3), while in the “few-shot” setting
we finetune on 45 SciFact examples.

The results are shown in Table 1(b). We find that pre-training on weakly-
supervised in-domain data improves performance in both the zero-shot and few-
shot settings. This improvement is particularly substantial in the zero-shot
setting, leading to relative performance improvements of 73% and 60% on
sentence-level and abstract-level F1, respectively. One important factor
contributing to this improvement is that classification thresholds learned on
Fever tend to be poorly-calibrated for scientific fact-checking, yielding high
precision at the expense of very low recall. Pretraining on in-domain data
(even lacking any gold labels) appears to improve calibration. Finally, we
observe that the performance gap between the few-shot and complete-data
settings remains substantial, indicating that there is ample room for
improvement on domain adaptation for scientific claim verification.

##  6 Analysis

To determine the source of LongChecker’s performance improvements over
previous systems, as well as to identify potential areas for further model
development, we annotate 128 claim / evidence pairs from the SciFact test set
to understand the specific linguistic and reasoning challenges associated with
verifying scientific claims.

#### Verifying scientific claims poses distinct modeling challenges

We identify three major challenges. The first, and most frequent, is
_context_. As mentioned in §3, SciFact follows the convention that sentences
from an abstract $a$ are annotated as rationales if they report a result that
entails or contradicts the claim, _in the context of the abstract_. We find
that these rationales would entail the claim _in isolation_ in only 43 / 128
cases. In the other 85 cases, context is required to clarify the meaning of an
acronym, resolve a coreference link, or otherwise specify qualifiers or
conditions necessary for the rationale to entail the claim. Examples of this
phenomenon are provided in Table 2. This challenge has previously been
observed in the context of understanding clinical trial reports. For instance,
Nye et al. (2020) observe that the conditions of an experiment (e.g. the
population being studied, intervention, etc) are often reported early in the
abstract, while the results are often reported at the end.

The two remaining challenges—reasoning over background knowledge not
explicitly expressed in the abstract, and recognizing or reasoning over
numerical expressions—are less common; both occur in 22 / 128 of our annotated
instances. Examples are provided in Table 2.

####  LongChecker performs strongly on examples requiring abstract-level
context

Table 3 shows the performance of LongChecker and the two baselines on each of
the modeling challenges just described. The results indicate that LongChecker
improves over previous baselines because of its superior performance on
instances where context outside the rationales is required to make a correct
label prediction. In particular, LongChecker exhibits essentially equal
performance on claims that do and do not require additional context, while
ParagraphJoint and Vert5Erini suffer performance drops of 13.3 F1 and 12.9 F1,
respectively. Interestingly, LongChecker does the worst on instances where no
additional context is required – presumably because making a classification
decision is more challenging when given unnecessary context as input.

On claims requiring background knowledge, Vert5Erini strongly outperforms the
other two systems, consistent with observations indicating that larger
language models are able to store more knowledge in their parameters Raffel et
al. (2020).

Interestingly, none of the systems seem to struggle with instances involving
numerical or statistical statements. Due to the small number of examples, it
is difficult to determine whether the systems are actually interpreting
numerical statements correctly, or using other signals in the evidence to
produce verdicts.

##  7 Conclusion and future work

In this work, we developed a new model for scientific claim verification that
incorporates all available context when predicting fact-checking labels. We
found that this modeling approach improved performance on the SciFact task,
and facilitated domain adaptation via finetuning on weakly-labeled in-domain
data.

Our work points toward a number of promising future directions. The
observation that existing systems achieve roughly human-level performance on
the entailment portion of scientific fact-checking suggests the need for
newer, and more realistic, versions of the scientific fact-checking task. This
could involve predicing more fine-grained labels than a simple Supports /
Refutes, or requiring models to verify claims against much larger research
corpora. Domain adaptation represents another area with substantial room for
improvement, as the few-shot models in this work still exhibit a large
performance gap relative to their fully-supervised counterparts. We hope that
our models and findings will facilitate progress on these research challenges.

## References

  * Barnett and Doubleday (2020) Adrian P. A. Barnett and Zoe Doubleday. 2020.  The growth of acronyms in the scientific literature.  _eLife_.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _ArXiv_ , abs/2004.05150.
  * Campos et al. (2016) Daniel Fernando Campos, T. Nguyen, M. Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, L. Deng, and Bhaskar Mitra. 2016.  Ms marco: A human generated machine reading comprehension dataset.  _NeurIPS_.
  * Chen et al. (2019) Qingyu Chen, Yifan Peng, and Zhiyong Lu. 2019.  Biosentvec: creating sentence embeddings for biomedical texts.  In _IEEE International Conference on Healthcare Informatics (ICHI)_.
  * Dasigi et al. (2021) Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021.  A dataset of information-seeking questions and answers anchored in research papers.  In _NAACL_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _NAACL_.
  * DeYoung et al. (2020) Jay DeYoung, E. Lehman, B. Nye, I. Marshall, and Byron C. Wallace. 2020.  Evidence inference 2.0: More data, better models.  In _Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing_.
  * Jin et al. (2019) Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. 2019\.  Pubmedqa: A dataset for biomedical research question answering.  In _EMNLP_.
  * Karpukhin et al. (2020) V. Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Yu Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _EMNLP_.
  * Kotonya and Toni (2020) Neema Kotonya and F. Toni. 2020.  Explainable automated fact-checking for public health claims.  In _EMNLP_.
  * Lehman et al. (2019) Eric P. Lehman, Jay DeYoung, R. Barzilay, and Byron C. Wallace. 2019.  Inferring which medical treatments work from reports of clinical trials.  In _NAACL_.
  * Li et al. (2021) Xiangci Li, G. Burns, and Nanyun Peng. 2021.  A paragraph-level multi-task learning model for scientific fact-verification.  In _Workshop on Scientific Document Understanding @ AAAI_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _ArXiv_ , abs/1907.11692.
  * Lo et al. (2020) Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Michael Kinney, and Daniel S. Weld. 2020\.  S2orc: The semantic scholar open research corpus.  In _ACL_.
  * Nogueira et al. (2020) Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. 2020.  Document ranking with a pretrained sequence-to-sequence model.  In _EMNLP_.
  * Nye et al. (2020) Benjamin E. Nye, Jay DeYoung, E. Lehman, A. Nenkova, I. Marshall, and Byron C. Wallace. 2020.  Understanding clinical trial reports: Extracting medical entities and their relations.  _ArXiv_ , abs/2010.03550.
  * Pradeep et al. (2021) Ronak Pradeep, Xueguang Ma, Rodrigo Nogueira, and Jimmy Lin. 2021.  Scientific claim verification with vert5erini.  In _Proceedings of the 12th International Workshop on Health Text Mining and Information Analysis @EACL_.
  * Raffel et al. (2020) Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, W. Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _JMLR_.
  * Robertson and Zaragoza (2009) S. Robertson and H. Zaragoza. 2009.  The probabilistic relevance framework: Bm25 and beyond.  _Foundations and Trends in Information Retrieval_.
  * Saakyan et al. (2021) Arkadiy Saakyan, Tuhin Chakrabarty, and Smaranda Muresan. 2021.  Covid-fact: Fact extraction and verification of real-world claims on covid-19 pandemic.  In _ACL_.
  * Sarrouti et al. (2021) Mourad Sarrouti, Asma Ben Abacha, Yassine Mrabet, and Dina Demner-Fushman. 2021\.  Evidence-based fact-checking of health-related claims.  In _EMNLP_.
  * Stammbach (2021) Dominik Stammbach. 2021.  Evidence selection as a token-level prediction task.  In _Proceedings of the Fourth Workshop on Fact Extraction and VERification (FEVER) @ EMNLP_.
  * Thakur et al. (2021) Nandan Thakur, N. Reimers, Andreas Ruckl’e, Abhishek Srivastava, and Iryna Gurevych. 2021.  Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models.  In _NeurIPS_.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018\.  Fever: a large-scale dataset for fact extraction and verification.  In _NAACL_.
  * Voorhees et al. (2020) Ellen M. Voorhees, Tasmeer Alam, Steven Bedrick, Dina Demner-Fushman, William R. Hersh, Kyle Lo, Kirk Roberts, Ian Soboroff, and Lucy Lu Wang. 2020\.  Trec-covid: Constructing a pandemic information retrieval test collection.  In _SIGIR_.
  * Wadden and Lo (2021) David Wadden and Kyle Lo. 2021.  Overview and insights from the sciver shared task on scientific claim verification.  In _Proceedings of the Second Workshop on Scholarly Document Processing @ NAACL_.
  * Wadden et al. (2020) David Wadden, Kyle Lo, Lucy Lu Wang, Shanchuan Lin, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020.  Fact or fiction: Verifying scientific claims.  In _EMNLP_.
  * Wang et al. (2020) Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Darrin Eide, Kathryn Funk, Rodney Michael Kinney, Ziyang Liu, William Cooper Merrill, Paul Mooney, Dewey A. Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex D Wade, Kuansan Wang, Christopher Wilhelm, Boya Xie, Douglas A. Raymond, Daniel S. Weld, Oren Etzioni, and Sebastian Kohlmeier. 2020.  Cord-19: The covid-19 open research dataset.  _ArXiv_.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020.  Big bird: Transformers for longer sequences.  In _NeurIPS_.
  * Zhang et al. (2021) Zhiwei Zhang, Jiyi Li, Fumiyo Fukumoto, and Yanming Ye. 2021.  Abstract, rationale, stance: A joint model for scientific claim verification.  In _EMNLP_.


# A Non-linear Function-on-Function Model for Regression with Time Series Data

Qiyao Wang1, Haiyan Wang1, Chetan Gupta1, Aniruddha Rajendra Rao2, Hamed
Khorasgani1  1Industrial AI Lab, Hitachi America, Ltd. R$\&$D, Santa Clara, CA
{Qiyao.Wang, Haiyan.Wang, Chetan.Gupta<EMAIL_ADDRESS>2Department of Statistics, Penn State University, University Park, PA
<EMAIL_ADDRESS>
###### Abstract

In the last few decades, building regression models for non-scalar variables,
including time series, text, image, and video, has attracted increasing
interests of researchers from the data analytic community. In this paper, we
focus on a multivariate time series regression problem. Specifically, we aim
to learn mathematical mappings from multiple chronologically measured
numerical variables within a certain time interval $\mathcal{S}$ to multiple
numerical variables of interest over time interval $\mathcal{T}$. Prior arts,
including the multivariate regression model, the Seq2Seq model, and the
functional linear models, suffer from several limitations. The first two types
of models can only handle regularly observed time series. Besides, the
conventional multivariate regression models tend to be biased and inefficient,
as they are incapable of encoding the temporal dependencies among observations
from the same time series. The sequential learning models explicitly use the
same set of parameters along time, which has negative impacts on accuracy. The
function-on-function linear model in functional data analysis (a branch of
statistics) is insufficient to capture complex correlations among the
considered time series and suffer from underfitting easily. In this paper, we
propose a general functional mapping that embraces the function-on-function
linear model as a special case. We then propose a non-linear function-on-
function model using the fully connected neural network to learn the mapping
from data, which addresses the aforementioned concerns in the existing
approaches. For the proposed model, we describe in detail the corresponding
numerical implementation procedures. The effectiveness of the proposed model
is demonstrated through the application to two real-world problems.

###### Index Terms:

Regression, Time series, Multivariate data analysis, Non-linear model,
Functional data analysis

11footnotetext: ©2020 IEEE. Personal use of this material is permitted.
Permission from IEEE must be obtained for all other uses, in any current or
future media, including reprinting/republishing this material for advertising
or promotional purposes, creating new collective works, for resale or
redistribution to servers or lists, or reuse of any copyrighted component of
this work in other works.

##  I Introduction

In data analytics, one of the most important types of analysis is regression.
The objective of regression is to mathematically estimate the relationship
between one or more dependent variables (i.e., outcome variables being
studied) and a set of independent variables (i.e., variables that have impacts
on the outcome variables) [1]. The learned mathematical mapping plays an
effective role in not only sorting out which predictors/covariates, and how
they interact with each other, to impact the dependent variables, but as well
as predicting the outcomes for new samples. Due to its explanatory and
predictive power, regression is an active area of research. Particularly, in
the last few decades, building regression models for non-scalar variables,
including time series, text, image, and video, has attracted increasing
interests of researchers from the data analytic community. These modern
regression models are beneficial to various domains in today’s world where
non-scalar types of data have become prevalent [2, 3, 4, 5].

In this paper, we consider the problem of regression with time series data
that occur ubiquitously in many industrial and scientific fields.
Specifically, we focus on building mathematical mappings from multiple
chronologically measured numerical variables within a certain time interval
$\mathcal{S}$ to multiple numerical variables of interest over time interval
$\mathcal{T}$. It is noteworthy that depending on the data collecting
mechanism, the time series can be either regular (i.e., the spacing of
observation time is constant) or irregular (i.e., the spacing is not
constant). Both types of time series are frequently encountered in real-world
applications. Regular time series data is common in domains such as economics
and meteorology. Examples include the daily temperature and the monthly
interest rate. Irregularly spaced time series also naturally occur in many
fields. For example, in the Internet of things, sensors often collect and
transmit data only when the operational setting or the state of equipment
changes, to reduce data storage and communication costs. In this paper, we
consider the most general setting where the input and output time series could
be either regular or irregular in the regression model. Furthermore, the data
collection time could vary across variables and data instances.

When data is irregular, a common practice is to preliminarily transform the
time series into observations at a common and equally spaced time grid for all
the subjects, so that the input and output data can be written as vectors
consisting of temporally ordered scalar variables. Then the conventional
multivariate regression models [6] or the sequence to sequence learning models
[2, 7] can be used to construct the mapping between the vectors of covariates
and responses. A significant drawback of this approach is that the data
manipulating step can introduce unquantifiable biases. Therefore, it is clear
that regression models that directly use the unevenly spaced time series in
their unaltered form would be useful.

Besides, even for regular time series that don’t necessitate pre-processing
data into equally spaced time series, the existing approaches are known to
suffer from several limitations. Despite their power in dealing with multiple
variables observed at a single timestamp, the traditional multivariate
regression models are inefficient in capturing temporal patterns and
therefore, they are not good choices for solving regression problems with time
series data. In particular, these models ignore the crucial fact that the
covariates and responses consist of random observations of the same variables
at different timestamps and solely rely on the regression models to account
for the intricate temporal correlations. The sequential deep learning models
are specifically designed to encode the sequential information in data with
orders [2, 8, 9, 10]. These models have been widely deployed in difficult
learning tasks in neural language processing [2, 8] and they have started to
play an increasingly important role in time series data analysis in recent
years [9, 10]. However, the sequential learning models are built upon
iterating the same transformation on the hidden states (i.e., up-to-present
memory) and the present covariates along time. This is a significant
limitation, since the correlation between the responses and the covariates
often varies over time.

In the statistical field, function-on-function linear models (FFLM) are
standard approaches for building regression models with time series covariates
and responses [11, 4, 12]. Under the central assumption that the time series
are smooth realizations of underlying continuous stochastic processes, FFLM
considers the entire time series as individual samples of the corresponding
random processes and attempts to learn the unknown bivariate parameter
function $\beta(s,t)$ that quantifies the correlation of covariates at any
time $s\in\mathcal{S}$ with the response at any time $t\in\mathcal{T}$. This
new perspective of modeling addresses the above-mentioned concerns in the
previous approaches, since both regular and irregular time series can be
analyzed as long as they contain a sufficient amount of information regarding
the underlying continuous random process [13, 14, 12, 15]. More critically,
unlike the sequential learning models, the flexible functional setting (i.e.,
$\beta(s,t)$) allows the correlation between covariates and responses to
change within the considered time domains. However, these models are linear
and therefore suffer from underfitting when the underlying mapping is complex.

In this paper, we also formulate the problem from the functional data analysis
perspective. We innovatively identify a general mathematical mapping between
the functional input and output data, based on which a non-linear model is
proposed. The contributions of this paper are summarized as follows:

  1. 1.

We propose a general functional mapping from multivariate temporal covariates
to responses that embraces the function-on-function linear model as a special
case.

  2. 2.

We propose a new model to address the considered regression problem. For
scenarios where the underlying process that generates the observed time series
is smooth, the proposed model possesses several advantages, including its
ability to handle versatile format of time series data, capture timely varying
correlations among variables, and build complex mappings.

  3. 3.

We describe in detail how to implement the proposed model from the beginning
to the end and point to the existing packages that can be used in each step.

  4. 4.

We demonstrate the effectiveness of the proposed approach through numerical
experiments and apply it to solve two real-world challenges.

##  II Preliminaries

###  II-A Notations and Prior Art

The goal of the considered multivariate time series regression problem is to
build a mapping from multiple time series covariates to several temporally
measured responses, leveraging the temporal dependencies within and between
the involved variables.

Suppose that we have access to data from $N$ independent subjects. For each
subject $i\in\\{1,2,...,N\\}$, $R$ covariates are continuously recorded within
a compact time interval $\mathcal{S}\subseteq\mathbb{R}$. Note that subject
and variable indexes are included in the following notations to reflect the
fact that the measuring timestamps can vary across different variables and
different subjects. In particular, the measuring timestamps of the $r$-th
feature for subject $i$ are stored in a $M^{(i,r)}_{s}$-dimensional vector
$\mathbf{S}^{(i,r)}=[S^{(i,r)}_{1},...,S^{(i,r)}_{j},...,S^{(i,r)}_{M^{(i,r)}_{s}}]^{T}$,
with $M^{(i,r)}_{s}$ representing the number of observations in the time
series and $S^{(i,r)}_{j}\in\mathcal{S}$ for
$i=1,...,n;r=1,...,R;j=1,...,M^{(i,r)}_{s}$. The corresponding temporal
observations are denoted as
$\mathbf{X}^{(i,r)}=[X^{(i,r)}_{1},...,X^{(i,r)}_{j},...,X^{(i,r)}_{M^{(i,r)}_{s}}]^{T}$.
Likewise, for a given subject $i$, there are $D$ responses being continuously
measured within a compact time interval $\mathcal{T}\subseteq\mathbb{R}$. The
$d$-th response is evaluated at $M^{(i,d)}_{t}$ timestamps. The measuring
times and the observations are respectively represented by
$\mathbf{T}^{(i,d)}=[T^{(i,d)}_{1},...,T^{(i,d)}_{j},...,T^{(i,d)}_{M^{(i,d)}_{t}}]^{T}$
and
$\mathbf{Y}^{(i,d)}=[Y^{(i,d)}_{1},...,Y^{(i,d)}_{j},...,Y^{(i,d)}_{M^{(i,d)}_{t}}]^{T}$,
for $d=1,...,D$. In summary, the observed data is
$\\{\mathbf{X}^{(i,1)},...,\mathbf{X}^{(i,R)},\mathbf{Y}^{(i,1)},...,\mathbf{Y}^{(i,D)}\\}_{i=1}^{N}$.
Intuitively, to effectively correlate the multivariate covariates over
$\mathcal{S}$ to the responses over $\mathcal{T}$, it is required that, for
any temporal covariate/response, there exist data from some subjects at
timestamps across the period $\mathcal{S}$/$\mathcal{T}$, so that the overall
temporal pattern can be estimated given data from the $N$ samples. Theoretical
arguments that specify handleable irregularities for most of the data
analytics models are provided in [14, 16].

In the prior art, the time series need to be first transformed into regular
time series evaluated at common time grids for all subjects. Let $M_{s}$ be
the number of observations in the transformed covariates and
$\mathbf{\tilde{X}}^{(i,r)}$ be the $M_{s}$-dimensional vector that stores the
processed data for the $r$-th covariate of subject $i$. Similarly, let $M_{t}$
be the number of observations in the transformed responses and
$\mathbf{\tilde{Y}}^{(i,d)}$ represents the $M_{t}$ observations from the
$d$-th response of subject $i$. The conventional approaches then concatenate
the $R$ temporal covariates and the $D$ time series-type responses, obtaining
$\mathbf{\tilde{X}}^{(i)}=[\mathbf{\tilde{X}}^{{(i,1)}^{T}},...,\mathbf{\tilde{X}}^{{(i,R)}^{T}}]^{T}$
and
$\mathbf{\tilde{Y}}^{(i)}=[\mathbf{\tilde{Y}}^{{(i,1)}^{T}},...,\mathbf{\tilde{Y}}^{{(i,D)}^{T}}]^{T}$.
Given samples
$\\{\mathbf{\tilde{X}}^{(i)},\mathbf{\tilde{Y}}^{(i)}\\}_{i=1}^{N}$, the
multivariate regression or sequential learning models are then utilized to
learn the mapping

$\mathbf{\tilde{Y}}^{(i)}=F(\mathbf{\tilde{X}}^{(i)}).$ (1)

The disadvantages of the above approaches are twofold. On one hand,
immensurable biases may be introduced when conducting data pre-processing so
that the learned mapping tends to deviate from the ground truth. On the other
hand, these widely used models have their own limitations in solving
regression problems with time series inputs and outputs. The multivariate
regression models are incapable of encoding the temporal dependencies among
$\mathbf{\tilde{X}}^{(i,r)}$ and $\mathbf{\tilde{Y}}^{(i,d)}$, $r=1,...,R$ and
$d=1,...,D$. Accordingly, the sequential learning models explicitly apply the
same mathematical operations (i.e., use the same set of parameters) on the
$R$-dimensional inputs and the hidden states to obtain the $D$-dimensional
outputs for all the timestamps.

###  II-B Functional Data Analysis and a New Formulation

In this section, we describe an alternative problem formulation from the
functional data analysis (FDA) point of view. Functional data analysis refers
to the analysis of data samples consisting of dynamically varying data over a
continuum. It is a key methodology for the analysis of data that can be viewed
as realizations of random functions or surfaces, such as time series, image,
and tracking data (e.g., handwriting and driving path) [13]. When modeling
time series data, FDA methods uniquely deal with the continuous underlying
curves $X^{(i,r)}(s),s\in\mathcal{S}$ that generate the observed discrete time
series $\mathbf{X}^{(i,r)}$. The input and output data for functional
regression models are
$\\{X^{(i,1)}(s),...,X^{(i,R)}(s),s\in\mathcal{S};Y^{(i,1)}(t),...,Y^{(i,D)}(t),t\in\mathcal{T}\\}_{i=1}^{N}$.
Denote the sequential covariates and responses as vectors of random functions,
i.e., $\mathbf{X}^{(i)}(s)=[X^{(i,1)}(s),...,X^{(i,R)}(s)]^{T}$ and
$\mathbf{Y}^{(i)}(t)=[Y^{(i,1)}(t),...,Y^{(i,D)}(t)]^{T}$. Functional
regression models aim to learn the mapping

$\mathbf{Y}^{(i)}(t)=F(\mathbf{X}^{(i)}(s)).$ (2)

For instance, function-on-function linear models [11, 4, 12] focus on learning
the bivariate parameter functions in a $D\times R$ matrix
${\mbox{\boldmath${\beta}$}}(s,t)=[\beta_{r,d}(s,t)]^{T}_{r=1,...,R;d=1,...,D}$

$\mathbf{Y}^{(i)}(t)={\mbox{\boldmath${\mu}$}}(t)+\int_{s}{\mbox{\boldmath${\beta}$}}(s,t)\mathbf{X}^{(i)}(s)ds,$
(3)

where ${\mbox{\boldmath${\mu}$}}(t)=[\mu_{1}(t),...,\mu_{D}(t)]^{T}$ consists
of the mean function for the $D$ responses over $\mathcal{T}$.

In FDA, although smoothness of underlying random functions
$\\{\mathbf{X}^{(i)}(s),s\in\mathcal{S};\mathbf{Y}^{(i)}(t),t\in\mathcal{T}\\}_{i=1}^{N}$,
such as existence of continuous second derivatives, is often imposed for
regularization, FDA techniques often accommodate moderate random errors in the
actual discrete observations [14]. Therefore, they are applicable in analyzing
a wide range of time series data. Unlike the conventional models in Section
II-A, functional models can directly analyze the raw time series, which
greatly enhances the flexibility in applications. Furthermore, in contrast to
the sequential learning models that keep the parameters the same over time,
functional models abandon this restrictive assumption and explicitly allow the
covariate effects to change along $\mathcal{S}$ for different timestamps in
the response time interval $\mathcal{T}$.

Most studies on regression models with both functional covariates and
functional responses have focused on linear models [11, 17]. However, the
linear structures are inadequate and make the models suffer from underfitting
easily [18, 19, 20, 21]. In the next section, we propose a non-linear
function-on-function regression model leveraging the power of fully connected
Neural Networks.

##  III Proposed Non-linear Function-on-function Regression Model

###  III-A Multivariate Functional principal Component Analysis

In the section, we briefly summarize the multivariate functional principal
component analysis (multivariate FPCA), an useful dimension reduction tool
that frequently serves as a key component in many functional models [22, 17].
This section focuses on describing the theory. The specific estimation
procedures using the actual observations are included in Section III-C.

The basic objects in multivariate FPCA is a set of real-valued random
functions on a common compact interval, such as the multivariate functional
covariates and responses introduced in Section II-B. Let’s take the
$D$-dimensional functional responses
$\mathbf{Y}^{(i)}(t)=[Y^{(i,1)}(t),...,Y^{(i,D)}(t)]^{T}$ as an example. Each
element in $\mathbf{Y}^{(i)}(t)$ is typically assumed to follow a stochastic
process with unknown mean function $\mu_{Y}^{(d)}(t)$ and covariance function
$G_{Y}^{(dd)}(t,t^{\prime})$, for $d=1,...,D$. Also, the variables are cross-
correlated, with the covariance function between the $d$-th and the
$d^{\prime}$-th functional variable being
$G_{Y}^{(dd^{\prime})}(t,t^{\prime})$, for $d,d^{\prime}=1,....,D$ and
$t,t^{\prime}\in\mathcal{T}$.

To take the possibly uneven extent of variations among the $D$ random
processes into account, we follow the proposal in [17] to normalize data
through a point-wise Z-score standarization, i.e.,
$Y_{z}^{(i,d)}(t)=v_{Y}^{(d)}(t)^{-1/2}(Y^{(i,d)}(t)-\mu_{Y}^{(d)}(t))$, with
$v_{Y}^{(d)}(t)=G_{Y}^{(dd)}(t,t)$ being the variance among observations at
time $t$. Let’s denote the normalized random functions as
$\mathbf{Y}_{z}^{(i)}(t)=[Y_{z}^{(i,1)}(t),...,Y_{z}^{(i,D)}(t)]^{T}$, whose
mean is a $D$-dimensional function, which takes a value $0$ over $\mathcal{T}$
and the matrix of covariance functions is
$\mathbf{G}_{Y_{z}}(t,t^{\prime})=[G_{Y_{z}}^{(dd^{\prime})}(t,t^{\prime})]_{d,d^{\prime}=1,...,D}$,
$t,t^{\prime}\in\mathcal{T}$.

Under certain regularity requirements on $\mathbf{G}_{Y_{z}}(t,t^{\prime})$,
it has been shown that there exists an orthonormal basis of eigenfunctions
${\mbox{\boldmath${\phi}$}}_{p}(t)=[\phi_{p}^{(1)}(t),...,\phi_{p}^{(D)}(t)]^{T}$
such that

$\int\mathbf{G}_{Y_{z}}(t,t^{\prime}){\mbox{\boldmath${\phi}$}}_{p}(t^{\prime})dt^{\prime}=\lambda_{p}{\mbox{\boldmath${\phi}$}}_{p}(t),\text{with
}\lim_{p\rightarrow\infty}\lambda_{p}=0,$ (4)

where $\lambda_{p}\in\mathbb{R}$ is the eigenvalue corresponding to the
$D$-dimensional eigenfunction ${\mbox{\boldmath${\phi}$}}_{p}(t)$. Similar to
the conventional PCA, $\lambda_{p}$ quantifies the amount of variability in
$\mathbf{Y}_{z}^{(i)}(t)$ being captured by
${\mbox{\boldmath${\phi}$}}_{p}(t)$. The output of
$\int\mathbf{G}_{Y_{z}}(t,t^{\prime}){\mbox{\boldmath${\phi}$}}_{p}(t^{\prime})dt^{\prime}$
is a $D$-dimensional function over $\mathcal{T}$, with the $d$-th element
$(\int\mathbf{G}_{Y_{z}}(t,t^{\prime}){\mbox{\boldmath${\phi}$}}_{p}(t^{\prime})dt^{\prime})^{(d)}$
being

$\left(\int\mathbf{G}_{Y_{z}}(t,t^{\prime}){\mbox{\boldmath${\phi}$}}_{p}(t^{\prime})dt^{\prime}\right)^{(d)}=\sum_{d^{\prime}=1}^{D}\int
G_{Y_{z}}^{(dd^{\prime})}(t,t^{\prime})\phi_{p}^{(d^{\prime})}(t^{\prime})dt^{\prime}.$
(5)

Given Eq (4), it has been shown that the multivariate random function
$\mathbf{Y}_{z}^{(i)}(t)$ can be represented as a linear combination of the
multivariate eigenfunctions with real-valued random coefficients
$\xi_{p}^{(i)}=\sum_{d=1}^{D}\int\phi_{p}^{(d)}(t)Y_{z}^{(i,d)}(t)dt$,
$p=1,...,P$. Mathematically, it can be seen as

$\mathbf{Y}_{z}^{(i)}(t)=\sum_{p=1}^{\infty}\xi_{p}^{(i)}{\mbox{\boldmath${\phi}$}}_{p}(t).$
(6)

Due to the smoothness of each functions in $\mathbf{Y}_{z}^{(i)}(t)$, it has
been proved that the eigenvalues decay to 0 at a fast rate such that the
information in $\mathbf{Y}_{z}^{(i)}(t)$ is well captured by a finite number
of the random coefficients. That is Eq (6) becomes

$\mathbf{Y}_{z}^{(i)}(t)\approx\sum_{p=1}^{P}\xi_{p}^{(i)}{\mbox{\boldmath${\phi}$}}_{p}(t).$
(7)

The number of truncation $P$ is typically determined by methods such as
fraction of variance explained, cross-validation, or some AIC and BIC-based
approaches [14].

In the next section, we describe in detail the proposed non-linear function-
on-function regression model based on multivariate FPCA, followed by
discussions on numerical implementations in Section III-C.

###  III-B Proposed Model

Following the convention in the literature [4, 17], let’s first use the point-
wise Z-score normalization to make the magnitude of variation comparable among
the covariates and responses. Let’s denote the standardized data as
$\mathbf{X}_{z}^{(i)}(s)=[X_{z}^{(i,1)}(s),...,X_{z}^{(i,R)}(s)]^{T}$ and
$\mathbf{Y}_{z}^{(i)}(t)=[Y_{z}^{(i,1)}(t),...,Y_{z}^{(i,D)}(t)]^{T}$. To
learn the mapping between $\mathbf{X}_{z}^{(i)}(s)$ and
$\mathbf{Y}_{z}^{(i)}(t)$, some existing models assume certain functional
forms and then estimate the parameter functions associated with the functional
covariates. For instance, the linear model being considered in literature is

$\mathbf{Y}_{z}^{(i)}(t)=\int_{s}{\mbox{\boldmath${\beta}$}}(s,t)\mathbf{X}_{z}^{(i)}(s)ds.$
(8)

Eq (7) implies that $\mathbf{Y}_{z}^{(i)}(t)$ can be effectively approximated
by a linear combination of the eigenfunctions
${\mbox{\boldmath${\phi}$}}_{p}(t)$, with the major modes of variations among
$\mathbf{Y}_{z}^{(i)}(t)$ captured. Under the regression setting, to learn the
impact of different values of $\mathbf{X}_{z}^{(i)}(s)$ on the variation of
$\mathbf{Y}_{z}^{(i)}(t)$, we propose to set the coefficients $\xi_{p}^{(i)}$
in Eq (7) as a function of $\mathbf{X}_{z}^{(i)}(s)$. Mathematically, let
${\Phi}$ be a $D\times P$ matrix, with the $(d,p)$-th element being
$\phi_{p}^{(d)}(t)$. Let $\mathbf{f}(\cdot)$ be a mapping from $R$-dimensional
$L_{2}(\mathcal{T})$ to $\mathbb{R}^{P}$. Then we define a general mapping
from $\mathbf{X}_{z}^{(i)}(s)$ to $\mathbf{Y}_{z}^{(i)}(t)$ as

$\mathbf{Y}_{z}^{(i)}(t)={\mbox{\boldmath${\Phi}$}}\mathbf{f}(\mathbf{X}_{z}^{(i)}(s)).$
(9)

The goal of the regression problem is then to learn the mapping
$\mathbf{f}(\cdot)$ in Eq (9). It is worth noting that the function-on-
function linear model in Eq (8) is a special case of the general mapping in Eq
(9). The linear models set the unknown parameter function in Eq (8) as
${\mbox{\boldmath${\beta}$}}(s,t)={\mbox{\boldmath${\Phi}$}}\mathbf{B}{\mbox{\boldmath${\Psi}$}}^{T}$,
where ${\Psi}$ is a $R\times L$ matrix that stores the first $L$ multivariate
eigenfunctions of $\mathbf{X}_{z}^{(i)}(s)$ and $\mathbf{B}$ is a $P\times L$
matrix holds real-valued unknown parameters that defines the parameter
function ${\mbox{\boldmath${\beta}$}}(s,t)$ [4, 13]. This is equivalent to say
that these models consider $\mathbf{f}(\cdot)$ in Eq (9) as
$\mathbf{B}{\mbox{\boldmath${\Psi}$}}^{T}\mathbf{X_{z}^{(i)}}(s)$.

To capture non-linear relationships, in this paper, we propose to replace the
linear structure by a fully connected Neural Network with $W$ layers and the
number of neurons being $O_{w}$, for $w=1,...,W$. Specifically, the model is

$\mathbf{Y}_{z}^{(i)}(t)={\mbox{\boldmath${\Phi}$}}\textbf{NN}_{\\{O_{w}\\}_{w=1}^{W}}({\mbox{\boldmath${\Psi}$}}^{T}\mathbf{X}_{z}^{(i)}(s)).$
(10)

Leveraging the power of Neural Networks, the proposed mapping in Eq (10) is
able to learn an intricate mapping from the multivariate covariates
$\mathbf{X}_{z}^{(i)}(s)$ to the response $\mathbf{Y}_{z}^{(i)}(t)$. This
extension significantly enhances the applicability of functional regression
models. Note that $W,O_{1},...,O_{W}$ are hyperparameters that can be
carefully tuned to obtain the best model performance. The graphical
representation of the proposed mode is provided in Fig. 1.

Figure 1: A graphical representation of the proposed model.

###  III-C Numerical Implementation

In this section, we consider the numerical implementation involved in the
training and application phase of the proposed model.

####  III-C1 Training phase

In this subsection, we present in details how the model in Eq (10) can be
trained using the finitely observed data
$\\{\mathbf{X}^{(i,1)},...,\mathbf{X}^{(i,R)},\mathbf{Y}^{(i,1)},...,\mathbf{Y}^{(i,D)}\\}_{i=1}^{N}$.

  * •

Data standardization: We begin by summarizing the estimation procedure for the
mean and covariance functions for each of the considered variables. Let’s use
the $d$-th response as an example. Note that the same procedure will be
iterated on the other variables. Given data
$\\{\mathbf{Y}^{(i,d)}\\}_{i=1}^{N}$ and the corresponding observation time
$\\{\mathbf{T}^{(i,d)}\\}_{i=1}^{N}$, for any $t\in\mathcal{T}$, let’s define
a smoothing kernel $K_{Y,1}^{(d)}(\cdot)$ with a tunable bandwidth
$h_{\mu_{Y}^{(d)}}$ and minimize

$\sum_{i=1}^{N}\sum_{j=1}^{M_{t}^{(i,d)}}K_{Y,1}^{(d)}\left(\frac{T^{(i,d)}_{j}-t}{h_{\mu_{Y}^{(d)}}}\right)\left[Y^{(i,d)}_{j}-\beta^{(d)}_{Y,0}-\beta^{(d)}_{Y,1}(t-T^{(i,d)}_{j})\right]^{2}$

(11)

with respect to $\beta^{(d)}_{Y,0}$ and $\beta^{(d)}_{Y,1}$, leading
$\hat{\mu}_{Y}^{(d)}(t)=\hat{\beta}^{(d)}_{Y,0}(t)$. For $i=1,...,N$, let
$U_{Y}^{(i,d)}(T^{(i,d)}_{j_{1}},T^{(i,d)}_{j_{2}})=(Y^{(i,d)}_{j_{1}}-\hat{\mu}_{Y}^{(d)}(T^{(i,d)}_{j_{1}}))(Y^{(i,d)}_{j_{2}}-\hat{\mu}_{Y}^{(d)}(T^{(i,d)}_{j_{2}}))$.
Also, let’s define a smoothing kernel $K_{Y,2}^{(d)}(\cdot,\cdot)$ with a
bandwidth $h_{G_{Y}^{(d)}}$. For any given $t,t^{\prime}\in\mathcal{T}$,
minimize

$\leavevmode\resizebox{399.2946pt}{}{$\sum_{i=1}^{N}\sum_{1\leq j_{1}\neq
j_{2}\leq
M_{t}^{(i,d)}}K_{Y,2}^{(d)}\left(\frac{T^{(i,d)}_{j_{1}}-t}{h_{G_{Y}^{(d)}}},\frac{T^{(i,d)}_{j_{2}}-t^{\prime}}{h_{G_{Y}^{(d)}}}\right)\times$}\\\
\leavevmode\resizebox{469.75499pt}{}{$\left[U_{Y}^{(i,d)}(T^{(i,d)}_{j_{1}},T^{(i,d)}_{j_{2}})-\gamma^{(d)}_{Y,0}-\gamma^{(d)}_{Y,1}(t-T^{(i,d)}_{j_{1}})-\gamma^{(d)}_{Y,2}(t^{\prime}-T^{(i,d)}_{j_{2}})\right]^{2}$}$

with respect to $\gamma^{(d)}_{Y,0}$, $\gamma^{(d)}_{Y,1}$ and
$\gamma^{(d)}_{Y,2}$, leading
$\hat{G}_{Y}^{(d)}(t,t^{\prime})=\hat{\gamma}^{(d)}_{Y,0}(t,t^{\prime})$ and
accordingly, $\hat{v}_{Y}^{(d)}(t)=\hat{\gamma}^{(d)}_{Y,0}(t,t)$, for
$t\in\mathcal{T}$. The estimation procedure can be implemented by R packages
including ‘fdapace’ [23] and ‘fpca’ [24]. Given the estimated mean and
covariance function, we obtain the standardized data
$\\{\mathbf{Y}_{z}^{(i,d)}\\}_{i=1}^{N}$, with the $j$-th element of
$\mathbf{Y}_{z}^{(i,d)}$ being

$Y_{z,j}^{(i,d)}=\hat{v}_{Y}^{(d)}\left(T^{(i,d)}_{j}\right)^{-1/2}\left(Y^{(i,d)}_{j}-\hat{\mu}_{Y}^{(d)}\left(T^{(i,d)}_{j}\right)\right).$
(12)

After conducting the above calculation for all the variables, we obtain the
standardized data
$\\{\mathbf{X}_{z}^{(i,1)},...,\mathbf{X}_{z}^{(i,R)},\mathbf{Y}_{z}^{(i,1)},...,\mathbf{Y}_{z}^{(i,D)}\\}_{i=1}^{N}$,
with observation times being
$\\{\mathbf{S}^{(i,1)},...,\mathbf{S}^{(i,R)},\mathbf{T}^{(i,1)},...,\mathbf{T}^{(i,D)}\\}_{i=1}^{N}$
as defined in Section II-A.

  * •

Multivariate FPCA: The objective is to estimate the multivariate functional
principal components ${\Phi}$ and ${\Psi}$ in the proposed model in Eq (10).
The core theoretical result in [22] implies that the multivariate principal
components ${\Phi}$ and ${\Psi}$ can be represented by the less involved
univariate functional principal components. In particular, for $d=1,...,D$, we
start with using data $\\{\mathbf{Y}_{z}^{(i,d)}\\}_{i=1}^{N}$ to obtain the
estimated univariate eigenfunction $\\{\hat{\phi}_{p}^{*}{}^{(d)}(t)\\}$
through the restricted maximum likelihood method [25] or the local linear
smoothing based approach [14], both of which are included in R packages
‘fdapace’ and ‘fpca’. Next, we estimate $\int
Y_{z}^{(i,d)}(t)\phi_{p}^{*}{}^{(d)}(t)dt$ based on $\mathbf{Y}_{z}^{(i,d)}$
and $\hat{\phi}_{p}^{*}{}^{(d)}(t)$ through numerical integration [26]. Let’s
denote the achieved value as $\hat{\xi}_{p}^{*}{}^{(i,d)}$, for
$i=1,...,N;d=1,...,D;p=1,...,P_{d}$. Let $P_{+}=\sum_{d=1}^{D}P_{d}$ and
${\Xi}$ is a $P_{+}\times P_{+}$ consisting of blocks
${\mbox{\boldmath${\Xi^{(dd^{\prime})}}$}}\in\mathbb{R}^{P_{d}\times
P_{d^{\prime}}}$ with the $(p,p^{\prime})$-th entry being

$\begin{split}\Xi_{pp^{\prime}}^{(dd^{\prime})}&=\text{Cov}(\hat{\xi}_{p}^{*}{}^{(i,d)},\hat{\xi}_{p^{\prime}}^{*}{}^{(i,d^{\prime})})\\\
&=\frac{1}{N-1}\sum_{i=1}^{N}(\hat{\xi}_{p}^{*}{}^{(i,d)}-\delta_{p}^{*}{}^{(d)})(\hat{\xi}_{p^{\prime}}^{*}{}^{(i,d^{\prime})}-\delta_{p^{\prime}}^{*}{}^{(d^{\prime})}),\end{split}$
(13)

where
$\delta_{p}^{*}{}^{(d)}$=$\frac{1}{N}\sum_{i=1}^{N}\hat{\xi}_{p}^{*}{}^{(i,d)}$
and
$\delta_{p^{\prime}}^{*}{}^{(d^{\prime})}=\frac{1}{N}\sum_{i=1}^{N}\hat{\xi}_{p^{\prime}}^{*}{}^{(i,d^{\prime})}$
are the corresponding sample means. Let’s conduct eigen decomposition on
matrix ${\Xi}$ and denote the $p$-th eigenvector as $\mathbf{c}_{p}$. Note
that $\mathbf{c}_{p}$ can be considered as a vector consisting of $D$ blocks,
with the $d$-th block being denoted as
$[\mathbf{c}_{p}]^{(d)}\in\mathbb{R}^{P_{d}}$. According to the proposition in
[22], we can estimate the $(d,p)$-th element of ${\Phi}$ (i.e., the matrix of
the multivariate functional principal components) in Section III-B by

$\hat{\phi}_{p}^{(d)}(t)=\sum_{m=1}^{P_{d}}[\mathbf{c}_{p}]_{m}^{(d)}\hat{\phi}_{m}^{*}{}^{(d)}(t).$
(14)

Likewise, we can estimate the multivariate functional principal components of
the $R$-dimensional covariates. Let’s denote the achieved estimate as
${\hat{\Phi}}$ and ${\hat{\Psi}}$.

  * •

Train the proposed model: Given the orthnormality of eigenfunctions, Eq (10)
is equivalent to

${\mbox{\boldmath${\Phi}$}}^{T}\mathbf{Y}_{z}^{(i)}(t)=\textbf{NN}_{\\{O_{w}\\}_{w=1}^{W}}({\mbox{\boldmath${\Psi}$}}^{T}\mathbf{X}_{z}^{(i)}(s)).$
(15)

Given this observation, to train the proposed model, we first use numerical
integration to estimate the $P$-dimensional output vector
${\mbox{\boldmath${\Phi}$}}^{T}\mathbf{Y}_{z}^{(i)}(t)$ from ${\hat{\Phi}}$
and $\\{\mathbf{Y}_{z}^{(i,1)},...,\mathbf{Y}_{z}^{(i,D)}\\}$. Similarly, we
estimate the $L$-dimensional input vector
${\mbox{\boldmath${\Psi}$}}^{T}\mathbf{X}_{z}^{(i)}(s)$. Next, a fully
connected neural network is trained based on the estimated scalar projections
of ${\mbox{\boldmath${\Phi}$}}^{T}\mathbf{Y}_{z}^{(i)}(t)$ and
${\mbox{\boldmath${\Psi}$}}^{T}\mathbf{X}_{z}^{(i)}(s)$.

####  III-C2 Application phase

In this subsection, we describe the application of the trained model to a new
subject $\\{\mathbf{X}^{(new,1)},...,\mathbf{X}^{(new,R)}\\}$, with
$\mathbf{X}^{(new,r)}$ consisting of discretized observations of the $r$-th
feature within time interval $\mathcal{S}$. The observation times are denoted
as $\mathbf{S}^{(new,r)}$, for $r=1,...,R$.

  * •

Data standardization: To deploy the learned model on the new data, we first
conduct data standardization for each of the covariates. For $r=1,....,R$, let
the estimated mean and variance function from the input data in training be
$\hat{\mu}_{X}^{(r)}(s)$ and $\hat{v}_{X}^{(r)}(s)$, the $j$-th element of the
standardized data $\mathbf{X}_{z}^{(new,r)}$ is

$\leavevmode\resizebox{446.2658pt}{}{$X_{z,j}^{(new,r)}=\hat{v}_{X}^{(r)}\left(S^{(new,r)}_{j}\right)^{-1/2}\left(X^{(new,r)}_{j}-\hat{\mu}_{X}^{(r)}\left(S^{(new,r)}_{j}\right)\right)$}.$
(16)

The normalized data is
$\\{\mathbf{X}_{z}^{(new,1)},...,\mathbf{X}_{z}^{(new,R)}\\}$.

  * •

Make predictions: Given the normalized data in Eq (16), ${\hat{\Psi}}$, and
the multivariate FPC learned in the training phase, we can compute estimations
for the $L$-dimensional vector
${\mbox{\boldmath${\Psi}$}}^{T}\mathbf{X}_{z}^{(new)}(s)$, where
$\mathbf{X}_{z}^{(new)}(s)$ are the underlying continuous functions that
render the discrete observations
$\\{\mathbf{X}_{z}^{(new,1)},...,\mathbf{X}_{z}^{(new,R)}\\}$. Let’s denote
the achieved estimation as ${\mbox{\boldmath${\eta}$}}^{(new)}$, then we have

$\mathbf{\hat{Y}}_{z}^{(new)}(t)={\mbox{\boldmath${\hat{\Phi}}$}}\hat{\textbf{NN}}_{\\{O_{w}\\}_{w=1}^{W}}({\mbox{\boldmath${\eta}$}}^{(new)}).$
(17)

Note that $\mathbf{\hat{Y}}_{z}^{(new)}(t)$ is a $D$-dimensional vector of
functions, with the $d$-th element corresponding to the prediction of the
$d$-th response variable given
$\\{\mathbf{X}_{z}^{(new,1)},...,\mathbf{X}_{z}^{(new,R)}\\}$.

  * •

Convert the prediction to its original scale: In this section, for each
prediction in $\mathbf{\hat{Y}}_{z}^{(new)}(t)$, we revert the calculation in
the point-wise Z-score standardization to scale it back to its original
extent. Mathematically, the final estimate $\hat{Y}^{(new,d)}(t)$ is

$\hat{Y}^{(new,d)}(t)=\hat{Y}^{(new,d)}(t)\hat{v}_{Y}^{(d)}(t)^{1/2}+\hat{\mu}_{Y}^{(d)}(t),$
(18)

where $\hat{\mu}_{Y}^{(d)}(t)$ and $\hat{v}_{Y}^{(d)}(t)$ are the mean and
variance function estimates from the training data.

##  IV Numerical Experiments

In this section, we apply the proposed model to solve two real-world
challenges. The first problem is about understanding the association between
electricity demand and temperature, which plays a key role in electricity
supply management. The second problem attempts to perform short-term traffic
prediction for facilitating driving decision making and improving the overall
transportation efficiency. We compare the performance of the proposed non-
linear functional model with several state-of-the-art approaches, including
the multivariate regression model (specifically, the multivariate linear
regression), the widely used Seq2Seq model with LSTM-based encoder and decoder
[7], and the function-on-function linear regression model [4]. As shown by the
experimental results, the proposed model can serve as an effective solution
for regressions with multivariate temporal covariates and responses. For the
considered problems, the proposed model outperforms the above-mentioned
alternative methods.

###  IV-A An Application to Electricity Demand Analysis

In the energy field, due to the high costs associated with the storage of
electricity, it is of great significance to understand the variability of
electricity demand within a region over time so that the authorities and
suppliers can make informative operational decisions. Temperature is one of
the most essential factors that contribute to the variability in electricity
demand, as the weather condition affects the usage of heating and cooling
appliances. In the literature, several efforts have been made to investigate
the association between temperature and electricity demand [27, 28]. In this
experiment, following the formulation in [27], we apply the proposed model to
build a mathematical mapping from the daily temperature trajectory for the 7
days of a week to the daily electricity demand trajectory for the 7 days of
the same week.

We use the temperature and electricity demand records of Adelaide, a city in
the state of South Australia, between 7/6/1997 and 3/31/2007. For a given day,
we have access to half-hourly data for both temperature and electricity
demand, i.e. the observed covariates and responses are regular time series of
length 48 within the 24 hours time period. There are 508 weeks within the
considered period, i.e., sample size $N=508$. In Fig. 2, we plot the
multivariate temporal covariates and responses for all 508 samples.

(a) 7-dimensional covariates in the sample.

(b) 7-dimensional responses in the sample.

Figure 2: Observed within-a-day trajectories of temperature and electricity
demand for the 508 weeks in the experiment.

We randomly select 400 samples to build the regression model and evaluate the
model performance on the remaining 108 weeks of data. Let the number of
subjects in the testing set be $N_{\text{test}}$. The ground truth and
estimation for the half-hourly electricity demand be
$\mathbf{Y}^{(i,d)}=[Y^{(i,d)}_{1},...,Y^{(i,d)}_{j},...,Y^{(i,d)}_{48}]$ and
$\mathbf{\hat{Y}}^{(i,d)}=[\hat{Y}^{(i,d)}_{1},...,\hat{Y}^{(i,d)}_{j},...,\hat{Y}^{(i,d)}_{48}]$
for $i=1,...,N_{\text{test}};d=1,...,48$. For a given response, we quantify
the accuracy by the root mean squared error (RMSE)

$\text{RMSE}\left(\mathbf{Y}^{(i,d)},\mathbf{\hat{Y}}^{(i,d)}\right)=\frac{1}{48N_{\text{test}}}\sum_{i=1}^{N_{\text{test}}}\sum_{j=1}^{48}\left(Y^{(i,d)}_{j}-\hat{Y}^{(i,d)}_{j}\right)^{2}$,

(19)

and the relative mean squared prediction error (RMSPE)

$\text{RMSPE}\left(\mathbf{Y}^{(i,d)},\mathbf{\hat{Y}}^{(i,d)}\right)=\frac{1}{N_{\text{test}}}\sum_{i=1}^{N_{\text{test}}}\frac{\sum_{j=1}^{48}\left(Y^{(i,d)}_{j}-\hat{Y}^{(i,d)}_{j}\right)^{2}}{\sum_{j=1}^{48}\left(Y^{(i,d)}_{j}\right)^{2}}.$

(20)

The implementations of the considered models are summarized as follows. Note
that the time-wise Z-score standardization described in Section III is
utilized in all models. For the multivariate linear regression (‘Multi LR’),
the half-hourly temperature measurements in a week are treated as individual
features (i.e., the dimension of input is 336), which are jointly mapped to
half-hourly electricity demands throughout the week (i.e., the dimension of
output is 336) through $336$ linear functions. In this non-linear formulation,
the temporal information within each day is not efficiently captured. The
number of unknown parameters is 112,896 (i.e., $48\times 7\times 48\times 7$).
For the LSTM-based sequential learning model (‘Seq2Seq LSTM’), we use a LSTM
layer with the input shape being $(48,7)$ and the output is a
$336$-dimensional vector. This layer is followed by a fully connected layer of
$336$ neurons. The output is re-shaped into $(48,7)$ to create the outputs. As
for the functional models, first, the multivariate functional principal
component analysis is numerically implemented for both the functional
covariates and functional responses, based on the procedure in Section III-C.
We choose the number of functional principal components using the fraction of
variance explained approach, with the cutoff being $99\%$. The selected values
are $\hat{L}$=$11$ and $\hat{P}$=$10$. It means that, for any given subject,
the complete information among the 7 correlated temporal covariates are well
preserved in the $11$-dimensional vector that holds the projection of
covariates onto the principal components. Analogously, the 7 correlated
responses are well represented by the projections. The function-on-function
linear model uses linear models to learn the unknown parameters in the
$10\times 11$ transformation matrix. To build the proposed model in Eq (10),
we build a neural network with the following architecture: 11 nodes in the
input layer, 16 neurons in the first hidden layer with ‘elu’ (i.e.,
Exponential Linear Unit) being the activation function, and 10 nodes in the
output layer with linear activation functions.

The results including the number of parameters and the accuracy metrics are
given in Table II. It can be seen that the proposed model outperforms the
other models. Note that the observed time series are regular in this
experiment. Therefore, the error of the non-functional models reported in the
table does not include the possible biases introduced when transforming the
raw data into regular time series.

###  IV-B An Application to Short-term Traffic Predictions

Making short-term predictions for traffic is a substantial problem in the
transportation field. Accurate predictions of traffic not only help the
drivers make smarter choices that can save them time and fuel but also assist
authorities in managing the transportation system. Characteristic metrics for
the traffic situation at a given location often include speed (i.e., average
vehicle travel distance per unit time), flow (i.e., number of vehicles passed
per unit time), and occupancy (i.e., percentage of time in which a unit length
of roadway is occupied by vehicles) [4]. In this experiment, we use the
proposed model to simultaneously predict the trajectories of the three traffic
metrics in the remaining time of the day, based on the partially observed
trajectories up to a certain time.

We crawled data from the Caltrans Performance Measurement System (PeMS), a
publicly accessible system that contains real-time traffic data from over
39,000 individual detectors spanned over California’s highway system.
Specifically, we obtain the historical speed, flow, and occupancy data from
detector ‘400017’, which is located on the southbound of CA-85 (near Bascom
Avenue in Los Gatos). Totally, we include 739 non-holiday workdays between
1/1/2017 and 12/31/2019. All the three traffic variables were recorded over
the day in 5-min intervals. The problem is to predict the speed, flow and
occupancy over $\mathcal{T}=[10:00,24:00]$ using the corresponding
trajectories within $\mathcal{S}=[00:00,10:00]$.

We randomly assign $80\%$ of the samples into the training set and the
remaining $20\%$ into the testing set. The implementation of the models are
similar to Section IV-A. There are $120$ observations in period $\mathcal{T}$
and $168$ measurements over $\mathcal{S}$ for each traffic variable. So the
number of unknown parameters for the multivariate linear regression models is
$181,440$ (i.e., $120\times 3\times 168\times 3$). The selected number of
principal components are $\hat{L}$=$27$ and $\hat{P}$=$30$, based on the
$99\%$ variance of explained rule. For the proposed model, there is one hidden
layer with 16 neurons. As shown by Table II, the proposed model yields the
best performance in terms of both RMSE and RMSPE. The inputs, the actual
responses, and the predictions from our model for two randomly picked subjects
in the testing set are visualized in Fig. 3. Compare to the result in Section
IV-A, the achieved improvement over the linear function-on-function model is
smaller. Our explanation is that the underlying relationship is close to a
linear mapping in this study, while the mapping in Section IV-A is complex.

(a) One example in the testing set.

(b) Another example in the testing set.

Figure 3: Two illustrative examples from the testing set. Trajectories before
the blue vertical line are the inputs. The black trajectories after the blue
vertical line are the ground truth. The red trajectories are the predicted
trajectories from the proposed mode.  TABLE I: RMSE and RMSPE comparisons for
the electricity analysis task in Section IV-A.

Model |  $\\#$ of parameters | RMSE for each response | RMSPE for each response
---|---|---|---
Multi LR | 112,896 | (442.9, 383.5, 339.5, 323.8, 372.7, 343.2, 363.2) | (0.29, 0.25, 0.22, 0.21, 0.25, 0.12, 0.13)
Seq2Seq LSTM | 575,568 | (176.2, 152.8, 153.2, 156.7, 168.6, 157.5, 158.6) | (0.12, 0.10, 0.10, 0.11, 0.12, 0.11, 0.11)
FFLM | 110 | (210.9, 182.1, 184.5, 194.9, 198.3, 170.1. 171.0) | (0.14, 0.12, 0.12, 0.13, 0.14, 0.12, 0.13)
Proposed model | 362 | (156.2, 133.7, 131.3, 139.7, 154.2, 122.1, 132.1) | (0.11, 0.09, 0.09, 0.10, 0.11, 0.09, 0.10)

TABLE II: RMSE and RMSPE comparisons for the short-term traffic prediction
task in Section IV-B.

Model |  $\\#$ of parameters | RMSE for each response | RMSPE for each response
---|---|---|---
Multi LR | 181,440 | (108.86, 10.65, 16.94) | (0.344, 0.714, 0.322)
Seq2Seq LSTM | 1,278,648 | (40.21, 4.62, 8.13) | (0.154, 0.387, 0.169)
FFLM | 810 | (36.34, 4.42, 7.54) | (0.131, 0.337, 0.140)
Proposed model | 958 | (34.62, 3.98, 6.63) | (0.125, 0.307, 0.122)

##  V Conclusions

In this paper, we proposed a novel model for the multivariate time series
regression problem, a frequently encountered topic in a wide range of fields.
Compared to the existing approaches, the proposed model possesses several
advantages, including its ability to handle versatile format of time series
data, capture timely varying correlations among variables, and build complex
mappings. To enhance the understanding, we described the numerical
implementation of the proposed model step-by-step. We applied the proposed
model to study the association between daily temperature and electricity
demand in a week and to tackle short-term traffic prediction problem, in
comparison with several common practices in the prior art. The proposed non-
linear functional model produced smaller estimation and prediction errors than
the state-of-the-art approaches. We expect the proposed model to be widely
applicable to diverse real-world problems where the goal is to study the
correlation among several time series.

## References

  * [1] M. H. Kutner, C. J. Nachtsheim, J. Neter, W. Li _et al._ , _Applied linear statistical models_. McGraw-Hill Irwin New York, 2005, vol. 5.
  * [2] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [3] K. Fukushima and S. Miyake, “Neocognitron: A self-organizing neural network model for a mechanism of visual pattern recognition,” in _Competition and cooperation in neural nets_. Springer, 1982, pp. 267–285.
  * [4] J.-M. Chiou, Y.-F. Yang, and Y.-T. Chen, “Multivariate functional linear regression and prediction,” _Journal of Multivariate Analysis_ , vol. 146, pp. 301–312, 2016.
  * [5] S. Xingjian, Z. Chen, H. Wang, D.-Y. Yeung, W.-K. Wong, and W.-c. Woo, “Convolutional lstm network: A machine learning approach for precipitation nowcasting,” in _Advances in neural information processing systems_ , 2015, pp. 802–810.
  * [6] J. F. Hair, W. C. Black, B. J. Babin, R. E. Anderson, R. L. Tatham _et al._ , _Multivariate data analysis_. Prentice hall Upper Saddle River, NJ, 1998, vol. 5, no. 3.
  * [7] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in _Advances in neural information processing systems_ , 2014, pp. 3104–3112.
  * [8] M. I. Jordan, “Attractor dynamics and parallelism in a connectionist sequential machine,” in _Artificial neural networks: concept learning_ , 1990, pp. 112–127.
  * [9] J. T. Connor, R. D. Martin, and L. E. Atlas, “Recurrent neural networks and robust time series prediction,” _IEEE transactions on neural networks_ , vol. 5, no. 2, pp. 240–254, 1994.
  * [10] G. Dorffner, “Neural networks for time series processing,” in _Neural network world_. Citeseer, 1996.
  * [11] J. J. Faraway, “Regression analysis for a functional response,” _Technometrics_ , vol. 39, no. 3, pp. 254–261, 1997.
  * [12] P. Kokoszka and M. Reimherr, _Introduction to Functional Data Analysis_. New York: Chapman and Hall/CRC, 2018. [Online]. Available: https://doi.org/10.1201/9781315117416
  * [13] J. O. Ramsay, _Functional data analysis_. Wiley Online Library, 2006.
  * [14] F. Yao, H.-G. Müller, and J.-L. Wang, “Functional data analysis for sparse longitudinal data,” _Journal of the American Statistical Association_ , vol. 100, no. 470, pp. 577–590, 2005.
  * [15] Q. Wang, “Two-sample inference and change point detection for sparse functional data,” Ph.D. dissertation, University of Pittsburgh, 2017.
  * [16] D. Paul, J. Peng _et al._ , “Consistency of restricted maximum likelihood estimators of principal components,” _The Annals of Statistics_ , vol. 37, no. 3, pp. 1229–1271, 2009.
  * [17] J.-M. Chiou, Y.-T. Chen, and Y.-F. Yang, “Multivariate functional principal component analysis: A normalization approach,” _Statistica Sinica_ , pp. 1571–1596, 2014.
  * [18] F. Rossi, B. Conan-Guez, and F. Fleuret, “Functional data analysis with multi layer perceptrons,” in _Proceedings of IJCNN_. Citeseer, 2002, pp. 2843–2848.
  * [19] Q. Wang, S. Zheng, A. Farahat, S. Serita, T. Saeki, and C. Gupta, “Multilayer perceptron for sparse functional data,” in _2019 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2019, pp. 1–10.
  * [20] Q. Wang, S. Zheng, A. Farahat, S. Serita, and C. Gupta, “Remaining useful life estimation using functional data analysis,” _arXiv preprint arXiv:1904.06442_ , 2019.
  * [21] A. R. Rao, Q. Wang, H. Wang, H. Khorasgani, and C. Gupta, “Spatio-temporal functional neural networks,” _arXiv preprint arXiv:2009.05665_ , 2020.
  * [22] C. Happ and S. Greven, “Multivariate functional principal component analysis for data observed on different (dimensional) domains,” _Journal of the American Statistical Association_ , vol. 113, no. 522, pp. 649–659, 2018.
  * [23] Y. Chen, C. Carroll, X. Dai, J. Fan, P. Z. Hadjipantelis, K. Han, H. Ji, H.-G. Mueller, and J.-L. Wang, _fdapace: Functional Data Analysis and Empirical Dynamics_ , 2019, r package version 0.5.0. [Online]. Available: https://CRAN.R-project.org/package=fdapace
  * [24] J. Peng, _fpca: Restricted MLE for Functional Principal Components Analysis_ , 2011, r package version 0.2.1. [Online]. Available: https://CRAN.R-project.org/package=fpca
  * [25] J. Peng and D. Paul, “A geometric approach to maximum likelihood estimation of the functional principal components from sparse longitudinal data,” _Journal of Computational and Graphical Statistics_ , vol. 18, no. 4, pp. 995–1015, 2009.
  * [26] P. J. Davis and P. Rabinowitz, _Methods of numerical integration_. Courier Corporation, 2007.
  * [27] X. Liu, S. Ma, and K. Chen, “Multivariate functional regression via nested reduced-rank regularization,” _arXiv preprint arXiv:2003.04786_ , 2020.
  * [28] L. Magnano, “Mathematical models for temperature and electricity demand,” Ph.D. dissertation, 2007.