# Financial Time Series Data Augmentation with Generative Adversarial Networks
and Extended Intertemporal Return Plots

Justin Hellermann
School of Business and Economics
Humboldt University of Berlin
Berlin
<EMAIL_ADDRESS>
&Qinzhuan Qian
IT Engineer ING
ING Bank N.V.
Amsterdam
<EMAIL_ADDRESS>
&Ankit Shah
Senior IT Chapter Lead ING
ING Bank N.V.
Amsterdam
<EMAIL_ADDRESS>

###### Abstract

Data augmentation is a key regularization method to support the forecast and
classification performance of highly parameterized models in computer vision.
In the time series domain however, regularization in terms of augmentation is
not equally common even though these methods have proven to mitigate effects
from small sample size or non-stationarity. In this paper we apply state-of-
the art image-based generative models for the task of data augmentation and
introduce the extended intertemporal return plot (XIRP), a new image
representation for time series. Multiple tests are conducted to assess the
quality of the augmentation technique regarding its ability to synthesize time
series effectively and improve forecast results on a subset of the M4
competition. We further investigate the relationship between data set
characteristics and sampling results via Shapley values for feature
attribution on the performance metrics and the optimal ratio of augmented
data. Over all data sets, our approach proves to be effective in reducing the
return forecast error by 7% on 79% of the financial data sets with varying
statistical properties and frequencies.

_K_ eywords Data Augmentation $\cdot$ Generative Adversarial Networks $\cdot$
Time Series $\cdot$ Extended Intertemporal Return Plots $\cdot$ XIRP $\cdot$
Intertemporal Return Plots $\cdot$ IRP $\cdot$ Shapley Values

##  1 Introduction

Augmenting image data has become a fundamental part of many machine learning
pipelines and is a widely applied regularization technique to improve model
robustness and performance (Geron, 2017, p. 309). In the computer vision
community these techniques magnify the number of training examples by scaling,
rotating and shifting images. This allows for running multiple training epochs
on variations of the same underlying image.
Even though time series also face the challenge of limited training examples,
the application of data augmentation methods is not equally common. In the
classic, parametric setting, data augmentation is not per se required, since
the models have few parameters to fit. Nevertheless, this changes quickly,
once more complex neural network based models have to be trained. But data
augmentation has more far-reaching benefits than solely allowing to train
highly parameterized models. Expanding the amount of train data can improve
the performance of regression and classification tasks, analogously to the
improved estimates via bootstrapping techniques. In physics and system
engineering the benefit of time series augmentation is established and applied
successfully to enlarge existing train data (Theiler et al., 1992). This
partly results from the characteristics of time series in the context of
system engineering which often exhibit less randomness compared to economic
and financial data sets.
For decades, data augmentation therefore remained in the area of physics
before moving into finance (Le Guennec et al., 2016; Ziyin et al., 2021). Just
recently, researchers began to apply data augmentation methods to financial
time series trying to mitigate the effects of random and noisy features,
sudden jumps and nonstationary. Since many nonparametric time series models
need increased amounts of data to generalize well, augmenting existing time
series is promising to boost performance, as shown by Yuan et al. (2020); Lee
and Kim (2020). In this paper we consider different financial data sets and
introduce an extended version of the IRP to create image representations
encoding both the time series as well as intertemporal returns. Combining the
new image representation with advanced generative models facilitates the
augmentation of time series data. The new approach is tested on a large number
of financial datasets with varying frequencies spanning from daily to yearly
data. A series of test is conducted in and among the different data
frequencies to answer the following research questions: i) Can forecasting
models benefit from augmented data and if so to what extent? ii) What is the
optimal ratio between real and augmented data? iii) Which statistical
properties contribute to a successful augmentation?

##  2 Related Work

Since the proposed approach combines generative models, image representations
of time series and data augmentation, we discuss related work at different
topical intersections. The related work mainly focuses on time series data
augmentation methods and its the benefits as well as an introduction to other
time series image representations.
Data augmentation for the purpose of classification has been applied by
Forestier et al. (2017), who use dynamic time warping barycentric averaging to
produce new samples. A more complex approach is chosen by Le Guennec et al.
(2016), who use a deep convolutional neural network architecture to generate
synthetic data for time series classification. For time series forecasting
many models rely on transformation and bootstrapping methods Iftikhar et al.
(2017); Bergmeir et al. (2016); Bandara et al. (2021). Bergmeir et al. (2016)
use repeated sampling techniques to improve forecasts by aggregation (bagging)
of exponential smoothing methods. By separating the time series into the
trend, seasonal part, and remainder bagging using Box–Cox transformation and
STL decomposition, their approach returned promising results for monthly
economic data. The related concept of IAAFT by Schreiber and Schmitz (2000)
combines the bootstrap approach with shuffling time series which have been
generated by Fourier transform to simulate new time series. From the area of
parametric simulations Denaxas et al. (2015); Papaefthymiou and Klockl (2008)
apply Markov Chain Monte Carlo methods to simulate and thereby augment time
series data. Another recent approach named GRATIS by Kang et al. (2020), use
mixture autoregressive models to generate time series with diverse and
controllable features. According to the authors, the main benefit lies in a
more reliable model selection due to the possibility to test on an extended
amount of data. Apart from mixture autoregressive models which require
assumptions on the underlying process, the non-parametric class of GAN
architectures, especially the one proposed by Karras et al. (2020), have
received significant attention, as a result of their outstanding successes in
generating new images. The successes of the adversarial methods were also
partly transferred to the domain of time series to synthetic data (Fu et al.,
2019; Yoon et al., 2019; Esteban et al., 2017; Zhang et al., 2015)
Studies regarding the augmentation benefit for forecasting are sparsely
covered. Bandara et al. (2021) evaluated the GRATIS and Moving Block
Bootstrapping, Dynamic Time Warping Barycentric Averaging techniques to
improve the accuracy of global forecasting models. Smyl and Kuber (2016)
examine multiple time series augmentation methods (Markov Chain Monte Carlo,
local-global-trend algorithm, exponential smoothing) for multiple short time
series forecasting with recurrent neural networks.
Related image representations are Gramian Angular Summation and Gramian
Angular Difference Fields (Wang and Oates, 2014), which rely on a polar
encodings their inverse trigonometric function to the sum or the difference of
the encoding. Also Waveforms are often used to encode the shape of a time
series graph as a function of time. Gramian Angular Fields, Waveforms and
recurrence plots rely on a deterministic encodings of a time series. A
different approach is chosen by Markov transition fields as applied by Wang
and Oates (2014) to bin the time series in different classes and then
calculate transition probabilities and represent them in a two dimensional
space. Also spectogramms rely on a probablistic encoding and have been applied
successfully in processing audio signals and visualizing the spectrum of
frequencies of a signal as it varies with time (Wyse, 2017) while each plot is
characteristic for a certain sound or a spoken word. Depending on the approach
of encoding, some of the upper methods are limited in their applications and
can either recover the initial time series only partially or approximately.
This paper complements the existing studies by two novel aspects. First, the
paper introduces a new image representation named XIRP and then trains an
image-based GAN to augment the existing data. Further, we apply Shapley values
for model explainability to get an improved insight into what factors
contribute to each of the performance metrics. The ensemble of Shapley values
and image-based generative models for time series is, to the best of our
knowledge, not yet covered in the literature.

##  3 Methodology

In this paper we use a Wasserstein GAN which is an extension of the vanilla
GAN by Goodfellow et al. (2014). The vanilla GAN consists a generator network
$G$ and a discriminator network $D$ with opposing objective functions. In the
course of training, $G$ generates a fake distribution $\tilde{x}$ and proposes
it to $D$, which assesses the likelihood of $\tilde{x}$ originating from the
true data distribution. The adversarial game between $G$ and $D$ can be
described by the objective function

$\min_{G}\quad\max_{D}\quad\mathop{\mathbb{E}}_{x\sim\mathop{\mathbb{P}_{r}}}[log(D(x))]+\mathop{\mathbb{E}}_{\tilde{x}\sim\mathop{\mathbb{P}_{g}}}[log(1-D(\tilde{x}))]$
(1)

where $D$ and $G$ try to maximize and minimize (3), respectively. $D(x)$ is
the estimate of the discriminator that $x$ is real while $\mathbb{P}_{r}$
denotes the real data distribution. The generated data is denoted by
$\tilde{x}$ with $\tilde{x}=G(z)$ and $z\sim p(z)$ where $p$ denotes some
distribution, such as uniform or Gaussian. $\mathbb{P}_{g}$ is the
distribution of the generated data, or, put differently, the model
distribution. Having the opposing objectives, both networks play an iterative
game and train each other in the process. The above vanilla GAN suffers from
stability problems and is difficult to train (Goodfellow et al., 2014).
Arjovsky et al. (2017) address this problem by proposing a Wasserstein-GAN
(WGAN) with a more stable training behavior including correlation between the
value of the objective function and sample quality. An intuitive explanation
for the Wasserstein distance as a loss function can be obtained from its
synonym Earth Mover Distance. It can be related to the minimum cost of
transporting mass in order to transform the distribution $q$ into the
distribution $p$, whereby the cost is mass times transport distance. The
Wasserstein metric changes the upper objective function to

$\displaystyle\min_{G}\quad\max_{D\epsilon\mathcal{D}}\quad\mathop{\mathbb{E}}_{x\sim\mathop{\mathbb{P}_{r}}}[D(x)]-\mathop{\mathbb{E}}_{\tilde{x}\sim\mathop{\mathbb{P}_{g}}}[D(\tilde{x})]$
(2)

where $\mathcal{D}$ is defined as a set of 1-Lipschitz functions and $\hat{x}$
is a randomly sampled subset. Instead of classifying generated images as real
or fake, the WGAN replaces the discriminator model by a critic that scores the
realness or fakeness of an image. Gulrajani et al. (2017) further improve
training in their WGAN-GP, which incorporates gradient penalty terms for the
discriminative network. This changes the objective function to

$\displaystyle\min_{G}\quad\max_{D\epsilon\mathcal{D}}\quad\mathop{\mathbb{E}}_{x\sim\mathop{\mathbb{P}_{r}}}[D(x)]-\mathop{\mathbb{E}}_{\tilde{x}\sim\mathop{\mathbb{P}_{g}}}[D(\tilde{x})]+\lambda\mathop{\mathbb{E}}_{\tilde{x}\sim\mathop{\mathbb{P}}_{\hat{x}}}[({\|\nabla_{\hat{x}}D(\hat{x})\|}_{2}-1)^{2}]$
(3)

The gradient penalty term $[({\|\nabla_{\hat{x}}D(\hat{x})\|}_{2}-1)^{2}]$
opposes vanishing and exploding gradients. We refer to Gulrajani et al. (2017)
for a more detailed explanation of all underlying concepts and considerations.
In this paper, $G$ is trained on an extended version of IRPs recently proposed
by Hellermann and Lessmann (2021). The concept of IRPs mixes insights from
quantitative finance with the idea recurrence plots by Eckmann et al. (1987).
It calculates returns between all points in a time series returning an $NxN$
representation of a time series with length $N$. The IRP with indexes $i$ and
$j$ is obtained by calculating the pairwise logarithmic return

$\displaystyle R^{C}_{i,j}=log\left(\frac{x_{i}}{x_{j}}\right)\quad\forall
i,j\in\\{1,2,\ldots,S\\}$ (4)

Once the start value at $t=0$ is known, any time series can be recovered from
the IRP regardless whether it is a real or simulated representation. Note,
that $x\in\mathbb{R}_{+}$ must be ensured by scaling since returns must be
calculated on positive values.
Since we want to compare augmentation techniques that do not sample returns
but a scaled time series an extension is required. A careful look at the IRP
reveals that in case $i=j\implies R^{C}_{i,j}=0$ which leads to a diagonal
filled with zeros. This motivates the formulation to the extended IRP (XIRP)
which complements the intertemporal return structure by adding extra
information to the encoding. Thus, equation (4) can be written as:

$R^{X}_{i,j}=\begin{cases}log\left(\frac{x_{i}}{x_{j}}\right)\quad&\forall
i,j\in\\{1,2,\ldots,S\\}\wedge i\neq j\\\ x_{i}\quad&\forall
i,j\in\\{1,2,\ldots,S\\}\wedge i=j\end{cases}$ (5)

The advantages of the approach are manifold. First, the diagonal of the IRP
does not contain any information and is filled with additional feature in the
XIRP. Second, we can directly compare the results to other sampling techniques
without rescaling and calculating returns for the object of comparison. Since
the diagonal of the XIRP contains an additional feature, a joint scaling would
suggest a single feature and affect results negatively. As a result, the
diagonal and the off-diagonal have to be scaled independently. Linking the
concepts of GAN and XIRP we can now sample and extract the time series from
the diagonal while the returns are saved in the off-diagonal. There are
multiple ways of recovering a time series from an XIRP. Since the initial
values are stored in the diagonal, these can be directly extracted, ignoring
all off-diagonal information encoded in the data. However, this would neglect
parts of the encoded information thus we propose to extract the diagonal in a
first step and then use the diagonal as a baseline time series to calculate
variants. This involves replacing the diagonal of the initial XIRP with
$d^{XIRP}_{ij}=1\quad\forall i,j\in\\{1,2,\ldots,S\\}$. Afterwards, the
extracted diagonal can be multiplied column-wise with the $XIRP_{d=1}$, so
that $S$ time series are obtained which consider the dynamics encoded in the
off-diagonal. An additional option is to average over the $S$ time series or
randomly pick one of them. Randomly choosing a single instance or averaging
over all time series are valid methods to obtain a single time series. Thus,
the approach can be used as a scale-invariant encoding and also as an encoding
for the raw time series.
In order to identify the contributing factors for a successful augmentation we
make use of the concept of Shapley Values. Invented as a solution mechanism
for cooperative behavior in game theory by Shapley (1951), it has become
popular for explaining the contribution of features to a prediction and is now
frequently used to enhance model understanding (Ghorbani and Zou, 2019;
Merrick and Taly, 2020; Antipov and Pokryshevskaya, 2020). To quantify the
feature importance, these are successively added in a random order and the
contribution to the prediction is calculated. The Shapley value of a feature
value is the average change in the prediction that results from the new
feature having joint. Its ability to explain the predictions of nonlinear
models along with its desirable properties make it a competitive alternative
to more complex models such as Layer-Wise Relevance Propagation or Locally
Interpretable Model-Agnostic Explanations (LIME). Please refer to Shapley
(1951) for a detailed introduction to the concept, its variants and
considerations.

##  4 Data and Test Design

For the upcoming series of tests we use financial data with varying
frequencies from the M4 forecasting competition of Makridakis et al. (2018).
In order to reduce the computation volume related to the extensive backtests,
we use $N=100$ data sets of each $n=20$ daily, weekly, monthly, quarterly and
yearly data wile the number of elements is limited to $n_{d}=1000$ daily,
$n_{w}=500$ weekly, $n_{m}=250$ monthly, $n_{q}=100$ quarterly and $n_{y}=75$
yearly data points. For a complete list of variables see Table 1 in the
Appendix.
To test the quality of the synthetic data generated by the different
augmentation methods, we conduct a series of experiments. The first three
experiments test for the similarity of synthetic and real instances via
different approaches. The first test uses a dimensionality reduction technique
to visualize synthetic and real data sequences. The second test trains a
recurrent neural network on synthetic data to predict real data while the
third test trains a classifier to distinguish between real and synthetic
sequences. A final fourth test evaluates the benefit of the augmentation
methods by training a model on varying levels of generated data.
The first test applies the the Uniform Manifold Approximation and Projection
(UMAP) algorithm (McInnes et al., 2020) for clustering. A similar approach has
been applied by Yoon et al. (2019); Ali et al. (2019) and treats each of the
$d$ time steps as an individual feature. The UMAP algorithm receives sequences
of length $D_{seq}=28$ and reduces the dimension to $D_{umap}=2$. Elements
located closely in the 2D space are likely to have similar characteristics.
Then, we plot the representation in $d=2$ and randomly choose up to $n=500$
synthetic and real sequences, depending on the size of the initial data set.
The individual points are colored to indicate which are original or synthetic.
In case, real and synthetic time series are clearly different, we observe two
distinct clusters with small distance inside each cluster and a large
difference to the other cluster(s). Else, we observe one or more mixed
clusters with heterogeneous distribution of real and synthetic elements inside
each cluster.

(a) UMAP plot Weekly Data (W77)

(b) UMAP plot for Weekly Data (W85)

Figure 1: Selected UMAP Results

Figure 1 (a) contains the clustering result of a weekly data set (W77). The
original data (green) and the generated data (red) separate with few
exceptions in two distinguishable clusters indicating deficient sample
quality. Here, the samples are significantly different such that the UMAP
algorithm can easily differentiate between them. An increased homogeneity in
the samples is displayed Figure 1 (b). It shows real and synthetic instances
of weekly data (W85), but displays a single large cluster mixed with synthetic
and real instances. Except for a few outliers, synthetic and real elements mix
well, while the synthetic instances have a slightly more wide-spread cluster.
This is an example of a successful augmentation where the UMAP algorithm
cannot distinguish between original and synthetic elements. Due to the high
number of data sets we only show two exemplary UMAP plots at this point.
After a first qualitative assessment, the next test performs a predictive
assessment on the synthetic data. In order to get more detailed information
whether the augmentation is useful for predicting the real data, we train a
recurrent neural network (RNN) solely on synthetic instances before predicting
the real instances. We choose a small LSTM network of $n_{hl}=3$ hidden layers
with $n_{ne}=7$ neurons each. In case the augmentation method works well, a
model trained on synthetic data predicts real data accurately. To assess the
quality we perform a 80/20 split for train and test set. The training process
is repeated $k=10$ times and uses early stopping after $c=5$ periods on the
mean absolute error of the test set to counter imperfect initialization and
overfitting. The predictive score $s_{p}$ for all data sets can be obtained
from Table 1 in the Appendix section.
The third, discriminative test, acts a quantitative expansion to the first
qualitative test and assesses whether a generated time series is reasonably
similar to the original time series. Again, a RNN is trained on sequences of
length $D=28$ but now with the goal to distinguish real and synthetic
instances with the same split for train and test data as well as early
stopping on the binary crossentropy error. Again, we choose a small LSTM
network with $n_{hl}=3$ hidden layers with $n_{ne}=8$ neurons each. The
discriminative score $s_{d}$ can be obtained from Table 1 in the Appendix
section.
A fourth experiment examines the augmentation benefit for the WGAN-GP trained
on the XIRPs for each of the data sets. Therefore an RNN is trained stepwise
with an increasing fraction of synthetic data ranging from 0% to 50%. At each
step a train/test split shuffles the sequences and randomly assigns them to
one of the sets. The RNN is evaluated analogously to the predictive test using
the same early stopping mechanism. Note, that the test and validation set
always consist of real data instances to keep synthetic elements in the
training set and thus avoid corrupting reliability of the performance measure.
A train/test split of 80/20 is kept throughout the test. For each fraction of
synthetic data $\alpha$ we repeat the test $k=10$ times to improve the
stability of the results. As a reference, we choose $RMSE_{\alpha=0}$ and then
use the percentage improvement compared to the optimal level
$RMSE_{\alpha^{\star}}$.

Figure 2: Relation between Predictive, Discriminative and Augmentation Score

Figure 2 plots the three scores $s_{p}$, $s_{d}$ and $s_{a}$ and their
relationship.
In case of perfect alignment, we could observe a clear relationship, be it
linear, quadratic or kubic and see a gradular color change along either axis.
A look at the relationships between the scores via the Spearman rank
correlation coefficient $\rho_{r}$ displayed in Table 2 supports the
impression of the plot, that there is a negative relationship between $s_{d}$
and $s_{p}$, namely $\rho_{r}(s_{d},s_{p})=-0.4469$.

$\rho$ | $s_{a}$ | $s_{p}$ | $s_{d}$
---|---|---|---
$s_{a}$ | 1 | -0.1359 | 0.2837
$s_{p}$ | 0.1359 | 1 | 0.4469
$s_{d}$ | -0.2837 | -0.4469 | 1
Table 1: Spearman rank correlation between the test scores

The negative sign originates from the construction of the scores, since a
small predictive, high discriminative and high augmentation scores are
optimal. The relationship of the augmentation benefit towards predictive and
discriminative score is less distinct. Its interaction with the predictive
score $\rho_{r}(s_{a},s_{p})=-0.1359$ is a slightly negative one, while
$\rho_{r}(s_{a},s_{d})=0.2837$ is slightly positive. Since the measures
account for different aspects of the generative model, they are designed to be
slightly correlated but not perfectly aligned, which would indicate
redundancy.
An important aspect when augmenting data, is to what extend we can reduce the
forecast error. In a first approach, inspect the augmentation score in terms
of percentage decrease in RMSE on the frequency domain. The scores aggregated
over all frequencies and plotted in Figure 3 (a) shows that some data sets
benefit from augmentation resulting in a decrease the error by 30% while for
other data sets, augmentation is not successful resulting in larger
forecasting errors and thus a negative $s_{a}$. On average, the RMSE decreases
by $\mu=0.07$ or 7%. Taking a closer look at the daily data in Figure 3(b)
shows that for a daily frequency the benefit diverges to either small
positive/negative benefits or large benefits. For weekly data $\mu$ stays
roughly unchanged and the data almost resembles a uniform distribution from
$s_{a}=-0.05$ to $s_{a}=0.15$. For monthly data $\mu=0.04$ is smaller but
differs little regarding the variance. Regarding the quarterly and yearly data
in Figures 3(e) and 3(f) we see a slight shift in the mean towards a higher
augmentation benefit.
In all six frequencies we see only little differences in terms of $\mu$, which
is surprising, since the observations and their data generating process are
different in nature. Nevertheless, the WGAN-GP approach with XIRPs is able to
improve 85% of the daily data, 75% of the weekly, 70% of the monthly, 90% of
the quarterly data and 75% of the yearly data sets.

(a) $s_{a}$ for all data sets

(b) $s_{a}$ for daily data

(c) $s_{a}$ for weekly data

(d) $s_{a}$ for monthly data

(e) $s_{a}$ for quarterly data

(f) $s_{a}$ for yearly data

Figure 3: Augmentation Score Distribution

(a) $\alpha$ for all data sets

(b) $\alpha$ for daily data

(c) $\alpha$ for weekly data

(d) $\alpha$ for monthly data

(e) $\alpha$ for quarterly data

(f) $s_{a}$ for yearly data

Figure 4: Augmentation Level Distribution

In the following step, we explore the optimal amount of augmented data
$\alpha_{\star}$ and therefore consider the distribution of $\alpha$. The
results are displayed in Figure 4 where we see relatively huge bars for
$\alpha=0$, which corresponds to an unsuccessful augmentation. In this case
the best is to omit generated data. The results look similar for all
frequencies with varying $\mu$ except of the quarterly data. The
$\alpha_{\star}$ for quarterly data exhibits a large proportion of high
augmentation levels, where 5 out of 20 data sets can be augmented up to a rate
of 50% of generated data. This is interesting since compared to the next lower
frequency of yearly data, this is only the case for 1 in 20 data sets. Over
all financial data sets, we can see that on average $\alpha_{\star}=.2$ or a
level of roughly 20% of generated data lowers the forecast error in the most
effective way.
In Figures 3 and 4 we can see that $s_{\alpha}$ and $\alpha_{\star}$ vary in
and across the different frequencies. In order to know, which factors drive
not only $s_{a}$ but also $s_{p}$ and $s_{d}$, we apply the concept of Shapley
values to a set of moments (mean $\mu$, variance. $\sigma^{2}$, skewness
$\mu_{3}$ and kurtosis $\mu_{4}$) contained in Table 1 as well as the results
from the previous discriminative and predictive tests. Besides the descriptive
statistics, we add the value of Ljung-Box test statistic to quantify whether a
group of autocorrelations differ from zero Ljung and Box (1978); Box and
Pierce (1970). With $H_{0}$ the test assumes the data is independently
distributed and thus any correlations must be of random nature. The number of
lags being tested over is denoted by $h$ and we can reject $H_{0}$ if the test
statistic111The test statistic is calculated via
$Q=n(n+2)\sum_{k=1}^{h}{\frac{{\hat{\rho}}_{k}^{2}}{n-k}}$ with sample size
$n$ and sample autocorrelation $\hat{\rho}$ at lag $k$
$Q>\chi_{1-\alpha,h}^{2}$. To incorporate the commonly present behavior of
volatility clustering, the test is conducted once on the raw returns $Q_{r}$
and on the absolute returns $Q_{\lvert r\rvert}$ to capture large sequential
movements with both signs.

(a) SHAP values for $s_{p}$

(b) SHAP values for $s_{d}$

(c) SHAP values for $s_{a}$

(d) SHAP values for $\alpha^{\star}$

Figure 5: Shapley Values for $s_{p}$, $s_{d}$, $s_{a}$ and $\alpha^{\star}$

In order to make a statement on the influence of features regarding the
output, we regress the calculated statistics sequentially on $s_{p}$, $s_{d}$,
$s_{a}$ and $\alpha^{\star}$. We train a neural network regression model with
$n_{hl}=2$ hidden layers of $n_{ne}=10$ neurons each and use early stopping as
regularization. Figure 5 shows the Shapley values for the different
characteristics of the data set where each represented by a dot. The
corresponding plots sort the features regarding their importance from top to
bottom and plot the Shapely value of each instance against its feature value
indicated by the coloring from blue (low) to cyan (high). In Figure 5(a) we
can see that the high first order autocorrelation of returns, denoted by
$Q_{r}$, leads to a smaller $s_{p}$. Please remind at this point, that the
smaller $s_{p}$, the better the generated data. This appears reasonable, since
highly correlated time series are easier to predict than time series with
little autocorrelation. For $Q_{r}$ multiple dots with high feature values
(cyan) are visible. For the variance we see an opposite behavior, of high
variance leading to a higher and thus worse predictive score. In addition,
datasets with higher $\mu_{4}$ and $\mu$ result in a higher $s_{p}$, while
$\mu_{3}$ has the tendency to improve $s_{p}$. What we can see here ist that
the features mainly have either small positive or small negative effect and
contribute largely only in case of extreme feature values. For the
discriminative score in Figure 5(b), we observe the same order in the feature
importance. Interestingly, only the results of $Q_{r}$ reverse their effects,
since one would expect the opposite effects consistently for all variables.
But as mentioned above, the discriminative score measures a different aspect
of the generation process. For the discriminative score, a higher variance
leads to a higher discriminative score, which is reasonable, since a data set
with $\sigma^{2}=0$ would be easily distinguishable from generated data, which
naturally comes with some variance.
The Shapley value for the augmentation benefit $s_{a}$ are of particular
interest for this paper. For $Q_{r}$ we can observe multiple interesting
effects. the plot proposes that an average feature value increases the
augmentation score. On the one hand, instances with a medium low level of
$Q_{r}$ seem to improve the score. On the other hand elements with very low
and very high feature values both increase the benefit of augmentation.
Regarding $s_{a}$, this behavior is not observed for any other feature.
Strikingly, compared to $Q_{r}$, opposite effects can be observer for
$Q_{\lvert r\rvert}$. It has the opposing effect not only in Figure 5(c) but
also in the previous ones. Return distributions with high $\mu_{3}$ seem to
harm $s_{a}$ while excess $\mu_{4}$ improves the augmentation benefit.
However, the second and third moment are of relatively small importance
compared to $Q_{r}$ and $\sigma$, unless they have high feature values.
Regarding the choice of $\alpha^{\star}$, we see in Figure 5(d) that smaller
values of $Q_{r}$ have a small positive influence, but the higher values the
more they reduce $\alpha^{\star}$. This means for data sets with high
autocrrelation there exists the tendency to reduce the optimal level of
$\alpha^{\star}$. The opposite applies for $\sigma^{2}$ and $\mu$ which have
the tendency to favor higher fractions of generated data for higher features.
It is clearly observable that all features except $Q_{r}$ have little effect
on $\alpha^{\star}$ unless they take high values. Throughout the evaluation of
$s_{p}$, $s_{d}$, $s_{a}$, $\alpha^{\star}$ we see that the test statistic of
the Ljung Box test provides most information regarding the Shapley values. The
other features become only relevant once their values are high. However, we
cannot make a deterministic statement which level of $\alpha$ is the best, but
we can surely provide some guidance on whether an augmentation is likely to be
successful and limit the search space for an exhaustive search of $\alpha$.

##  5 Conclusion

This paper evaluates the image-based financial time series augmentation method
for the first time. The new extended IRP encodes the initial time series as
well as its intertemporal dynamics in a single image. In a sequence of tests
the data augmentation increases the performance of return forecasting models
substantially. This is valid for all considered data frequencies (daily,
weekly, monthly, quarterly and yearly) where the model decreased the average
forecasting error by 7%. In a series of tests and multiple repetitions we
observe an optimal amount of augmented data ranging from 17-25% and the
forecast error was reduced by 79% of the datasets. Further, Shapley values
identify important features such as low variance and high autocorrelation to
have positive effects on the quality of the simulated returns indicated by the
performance metrics. In addition, the distribution of optimal augmentation
levels proposes a fraction of 19% of synthetic data over all data frequencies.
In a more far reaching study, the number of predictors in the analysis could
be extended to account for drifts, seasonality and jump frequency. Further, a
cross model comparison with other time series augmentation methods could
provide useful benchmarks regarding the efficiency of the WGAN-GP trained on
XIRP.

##  6 Appendix

See pages - of Data_Aug_M4_data_set_characteristics.pdf See pages - of
Data_Aug_M4_aug_benefit.pdf

## References

  * Ali et al. (2019) Mohammed Ali, Mark W. Jones, Xianghua Xie, and Mark Williams.  TimeCluster: dimension reduction applied to temporal data for visual analytics.  _The Visual Computer_ , 35(6):1013–1026, June 2019.  ISSN 1432-2315.  doi: 10.1007/s00371-019-01673-y.  URL https://doi.org/10.1007/s00371-019-01673-y.
  * Antipov and Pokryshevskaya (2020) Evgeny A Antipov and Elena B Pokryshevskaya.  Interpretable machine learning for demand modeling with high-dimensional data using gradient boosting machines and shapley values.  _Journal of Revenue and Pricing Management_ , 19(5):355–364, 2020.
  * Arjovsky et al. (2017) Martin Arjovsky, Soumith Chintala, and Léon Bottou.  Wasserstein GAN.  _arXiv:1701.07875 [cs, stat]_ , December 2017.  URL http://arxiv.org/abs/1701.07875.  arXiv: 1701.07875.
  * Bandara et al. (2021) Kasun Bandara, Hansika Hewamalage, Yuan-Hao Liu, Yanfei Kang, and Christoph Bergmeir.  Improving the accuracy of global forecasting models using time series data augmentation.  _Pattern Recognition_ , 120:108148, 2021.  ISSN 0031-3203.  doi: https://doi.org/10.1016/j.patcog.2021.108148.  URL https://www.sciencedirect.com/science/article/pii/S0031320321003356.
  * Bergmeir et al. (2016) Christoph Bergmeir, Rob J Hyndman, and José M Benítez.  Bagging exponential smoothing methods using stl decomposition and box–cox transformation.  _International journal of forecasting_ , 32(2):303–312, 2016.
  * Box and Pierce (1970) G. E. P. Box and David A. Pierce.  Distribution of Residual Autocorrelations in Autoregressive-Integrated Moving Average Time Series Models.  _Journal of the American Statistical Association_ , 65(332):1509–1526, December 1970.  ISSN 0162-1459.  doi: 10.1080/01621459.1970.10481180.  URL https://www.tandfonline.com/doi/abs/10.1080/01621459.1970.10481180.
  * Denaxas et al. (2015) Evangelos A Denaxas, Rubenka Bandyopadhyay, Dalia Patiño-Echeverri, and Nikos Pitsianis.  Syntise: A modified multi-regime mcmc approach for generation of wind power synthetic time series.  In _2015 Annual IEEE Systems Conference (SysCon) Proceedings_ , pages 668–674. IEEE, 2015.
  * Eckmann et al. (1987) J.-P Eckmann, S. Oliffson Kamphorst, and D Ruelle.  Recurrence Plots of Dynamical Systems.  _Europhysics Letters (EPL)_ , 4(9):973–977, November 1987.  ISSN 0295-5075, 1286-4854.  doi: 10.1209/0295-5075/4/9/004.
  * Esteban et al. (2017) Cristóbal Esteban, Stephanie L. Hyland, and Gunnar Rätsch.  Real-valued (medical) time series generation with recurrent conditional gans, 2017.
  * Forestier et al. (2017) Germain Forestier, François Petitjean, Hoang Anh Dau, Geoffrey I Webb, and Eamonn Keogh.  Generating synthetic time series to augment sparse datasets.  In _2017 IEEE international conference on data mining (ICDM)_ , pages 865–870. IEEE, 2017.
  * Fu et al. (2019) Rao Fu, Jie Chen, Shutian Zeng, Yiping Zhuang, and Agus Sudjianto.  Time series simulation by conditional generative adversarial net, 2019\.
  * Geron (2017) Aurelien Geron.  _Hands-on machine learning with Scikit-Learn and TensorFlow: concepts, tools, and techniques to build intelligent systems_.  O’Reilly Media, Beijing ; Boston, first edition edition, 2017.  ISBN 9781491962299.  OCLC: ocn953432302.
  * Ghorbani and Zou (2019) Amirata Ghorbani and James Zou.  Data shapley: Equitable valuation of data for machine learning.  In _International Conference on Machine Learning_ , pages 2242–2251. PMLR, 2019.
  * Goodfellow et al. (2014) Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative Adversarial Networks.  _arXiv:1406.2661 [cs, stat]_ , June 2014.  URL http://arxiv.org/abs/1406.2661.  arXiv: 1406.2661.
  * Gulrajani et al. (2017) Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville.  Improved Training of Wasserstein GANs.  _arXiv:1704.00028 [cs, stat]_ , December 2017.  URL http://arxiv.org/abs/1704.00028.  arXiv: 1704.00028.
  * Hellermann and Lessmann (2021) Justin Hellermann and Stefan Lessmann.  Leveraging image-based generative adversarial networks for time series generation, 2021.  URL https://arxiv.org/abs/2112.08060.
  * Iftikhar et al. (2017) Nadeem Iftikhar, Xiufeng Liu, Sergiu Danalachi, Finn Ebertsen Nordbjerg, and Jens Henrik Vollesen.  A scalable smart meter data generator using spark.  In _OTM Confederated International Conferences" On the Move to Meaningful Internet Systems"_ , pages 21–36. Springer, 2017.
  * Kang et al. (2020) Yanfei Kang, Rob J Hyndman, and Feng Li.  Gratis: Generating time series with diverse and controllable characteristics.  _Statistical Analysis and Data Mining: The ASA Data Science Journal_ , 13(4):354–376, 2020.
  * Karras et al. (2020) Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Analyzing and improving the image quality of stylegan, 2020.
  * Le Guennec et al. (2016) Arthur Le Guennec, Simon Malinowski, and Romain Tavenard.  Data augmentation for time series classification using convolutional neural networks.  In _ECML/PKDD workshop on advanced analytics and learning on temporal data_ , 2016.
  * Lee and Kim (2020) Si Woon Lee and Ha Young Kim.  Stock market forecasting with super-high dimensional time-series data using convlstm, trend sampling, and specialized data augmentation.  _Expert Systems with Applications_ , 161:113704, 2020.
  * Ljung and Box (1978) G. M. Ljung and G. E. P. Box.  On a measure of lack of fit in time series models.  _Biometrika_ , 65(2):297–303, 08 1978.  ISSN 0006-3444.  doi: 10.1093/biomet/65.2.297.  URL https://doi.org/10.1093/biomet/65.2.297.
  * Makridakis et al. (2018) Spyros Makridakis, Evangelos Spiliotis, and Vassilios Assimakopoulos.  The m4 competition: Results, findings, conclusion and way forward.  _International Journal of Forecasting_ , 34(4):802–808, 2018.
  * McInnes et al. (2020) Leland McInnes, John Healy, and James Melville.  UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction.  _arXiv:1802.03426 [cs, stat]_ , September 2020.  URL http://arxiv.org/abs/1802.03426.  arXiv: 1802.03426.
  * Merrick and Taly (2020) Luke Merrick and Ankur Taly.  The explanation game: Explaining machine learning models using shapley values.  In _International Cross-Domain Conference for Machine Learning and Knowledge Extraction_ , pages 17–38. Springer, 2020.
  * Papaefthymiou and Klockl (2008) George Papaefthymiou and Bernd Klockl.  Mcmc for wind power simulation.  _IEEE transactions on energy conversion_ , 23(1):234–240, 2008.
  * Schreiber and Schmitz (2000) Thomas Schreiber and Andreas Schmitz.  Surrogate time series.  _Physica D: Nonlinear Phenomena_ , 142(3):346–382, 2000.  ISSN 0167-2789.  doi: https://doi.org/10.1016/S0167-2789(00)00043-9.  URL https://www.sciencedirect.com/science/article/pii/S0167278900000439.
  * Shapley (1951) Lloyd S. Shapley.  Notes on the N-Person Game — II: The Value of an N-Person Game.  Technical report, RAND Corporation, August 1951.  URL https://www.rand.org/pubs/research_memoranda/RM0670.html.
  * Smyl and Kuber (2016) Slawek Smyl and Karthik Kuber.  Data preprocessing and augmentation for multiple short time series forecasting with recurrent neural networks.  In _36th International Symposium on Forecasting_ , 2016.
  * Theiler et al. (1992) James Theiler, Stephen Eubank, André Longtin, Bryan Galdrikian, and J Doyne Farmer.  Testing for nonlinearity in time series: the method of surrogate data.  _Physica D: Nonlinear Phenomena_ , 58(1-4):77–94, 1992.
  * Wang and Oates (2014) Zhiguang Wang and Tim Oates.  Encoding time series as images for visual inspection and classification using tiled convolutional neural networks.  2014\.
  * Wyse (2017) L. Wyse.  Audio Spectrogram Representations for Processing with Convolutional Neural Networks.  _arXiv:1706.09559 [cs]_ , June 2017.  URL http://arxiv.org/abs/1706.09559.  arXiv: 1706.09559.
  * Yoon et al. (2019) Jinsung Yoon, Daniel Jarrett, and Mihaela van der Schaar.  Time-series generative adversarial networks.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper/2019/file/c9efe5f26cd17ba6216bbe2a7d26d490-Paper.pdf.
  * Yuan et al. (2020) Yuyu Yuan, Wen Wen, and Jincui Yang.  Using data augmentation based reinforcement learning for daily stock trading.  _Electronics_ , 9(9):1384, 2020.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun.  Character-level convolutional networks for text classification.  _Advances in neural information processing systems_ , 28:649–657, 2015.
  * Ziyin et al. (2021) Liu Ziyin, Kentaro Minami, and Kentaro Imajo.  What data augmentation do we need for deep-learning-based finance?  _arXiv preprint arXiv:2106.04114_ , 2021.


# Paragraph-based Transformer Pre-training for Multi-Sentence Inference

Luca Di Liello1 , Siddhant Garg2, Luca Soldaini3 , Alessandro Moschitti2
1University of Trento , 2Amazon Alexa AI, 3Allen Institute for AI
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
Work done as an intern at Amazon Alexa AI Work completed at Amazon Alexa AI

###### Abstract

Inference tasks such as answer sentence selection (AS2) or fact verification
are typically solved by fine-tuning transformer-based models as individual
sentence-pair classifiers. Recent studies show that these tasks benefit from
modeling dependencies across multiple candidate sentences jointly. In this
paper, we first show that popular pre-trained transformers perform _poorly_
when used for fine-tuning on multi-candidate inference tasks. We then propose
a new pre-training objective that models the paragraph-level semantics across
multiple input sentences. Our evaluation on three AS2 and one fact
verification datasets demonstrates the superiority of our pre-training
technique over the traditional ones for transformers used as joint models for
multi-candidate inference tasks, as well as when used as cross-encoders for
sentence-pair formulations of these tasks.

##  1 Introduction

Pre-trained transformers Devlin et al. (2019); Liu et al. (2019); Clark et al.
(2020) have become the de facto standard for several NLP applications, by
means of fine-tuning on downstream data. The most popular architecture uses
self-attention mechanisms for modeling long range dependencies between
compounds in the text, to produce deep contextualized representations of the
input. There are several downstream NLP applications that require reasoning
across multiple inputs candidates jointly towards prediction. Some popular
examples include (i) Answer Sentence Selection (AS2) Garg et al. (2020), which
is a Question Answering (QA) task that requires selecting the best answer from
a set of candidates for a question; and (ii) Fact Verification Thorne et al.
(2018), which reasons whether a claim is supported/refuted by multiple
evidences. Inherently, these tasks can utilize information from multiple
candidates (answers/evidences) to support the prediction of a particular
candidate.

Pre-trained transformers such as BERT are used for these tasks as cross-
encoders by setting them as sentence-pair classification problems, i.e,
aggregating inferences independently over each candidate. Recent studies Zhang
et al. (2021); Tymoshenko and Moschitti (2021) have shown that these tasks
benefit from encoding multiple candidates together, e.g., encoding five answer
candidates per question in the transformer, so that the cross-attention can
model dependencies between them. However, Zhang et al. only improved over the
pairwise cross-encoder by aggregating multiple pairwise cross-encoders
together (one for each candidate), and not by jointly encoding all candidates
together in a single model.

In this paper, we first show that popular pre-trained transformers such as
RoBERTa perform _poorly_ when used for jointly modeling inference tasks (e.g.,
AS2) using multi-candidates. We show that this is due to a shortcoming of
their pre-training objectives, being unable to capture meaningful dependencies
among multiple candidates for the fine-tuning task. To improve this aspect, we
propose a new pre-training objective for ‘joint’ transformer models, which
captures paragraph-level semantics across multiple input sentences.
Specifically, given a target sentence $s$ and multiple sentences (from the
same/different paragraph/document), the model needs to recognize which
sentences belong to the same paragraph as $s$ in the document used.

Joint inference over multiple-candidates entails modeling interrelated
information between multiple _short_ sentences, possibly from different
paragraphs or documents. This differs from related works Beltagy et al.
(2020); Zaheer et al. (2020); Xiao et al. (2021) that reduce the asymptotic
complexity of transformer attention to model long contiguous inputs
(documents) to get longer context for tasks such as machine reading and
summarization.

We evaluate our pre-trained multiple-candidate based joint models by (i)
performing AS2 on ASNQ Garg et al. (2020), WikiQA Yang et al. (2015), TREC-QA
Wang et al. (2007) datasets; and (ii) Fact Verification on the FEVER Thorne et
al. (2018) dataset. We show that our pre-trained joint models substantially
improve over the performance of transformers such as RoBERTa being used as
joint models for multi-candidate inference tasks, as well as when being used
as cross-encoders for sentence-pair formulations of these tasks.

##  2 Related Work

Multi-Sentence Inference: Inference over a set of multiple candidates has been
studied in the past Bian et al. (2017); Ai et al. (2018). The most relevant
for AS2 are the works of Bonadiman and Moschitti (2020) and Zhang et al.
(2021), the former improving over older neural networks but failing to beat
the performance of transformers; the latter using task-specific models (answer
support classifiers) on top of the transformer for performance improvements.
For fact verification, Tymoshenko and Moschitti (2021) propose jointly
embedding multiple evidence with the claim towards improving the performance
of baseline pairwise cross-encoder transformers.

Transformer pre-training Objectives: Masked Language Modeling (MLM) is a
popular transformer pre-training objective Devlin et al. (2019); Liu et al.
(2019). Other models are trained using token-level Clark et al. (2020); Joshi
et al. (2020); Yang et al. (2019); Liello et al. (2021) and/or sentence-level
Devlin et al. (2019); Lan et al. (2020); Wang et al. (2020) objectives. REALM
Guu et al. (2020) uses a differentiable neural retriever over Wikipedia to
improve MLM pre-training. This differs from our pre-training setting as it
uses additional knowledge to improve the pre-trained LM. DeCLUTR Giorgi et al.
(2021) uses a contrastive learning objective for cross-encoding two sentences
coming from the same/different documents in a transformer. DeCLUTR is
evaluated for sentence-pair classification tasks and embeds the two inputs
independently without any cross-attention, which differs from our setting of
embedding multiple candidates jointly for inference.

Modeling Longer Sequences: Beltagy et al. (2020); Zaheer et al. (2020) reduce
the asymptotic complexity of transformer attention to model very long inputs
for longer context. For tasks with short sequence lengths, LongFormer works on
par or slightly worse than RoBERTa (attributed to reduced attention
computation). These works encode a single contiguous long piece of text, which
differs from our setting of having multiple _short_ candidates, for a
topic/query, possibly from different paragraphs and documents. DCS Ginzburg et
al. (2021) proposes a cross-encoder for the task of document-pair matching.
DCS is related to our work as it uses a contrastive pre-training objective
over two sentences extracted from the same paragraph, however different from
our joint encoding of multiple sentences, DCS individually encodes the two
sentences and then uses the InfoNCE loss over the embeddings. CDLM Caciularu
et al. (2021) specializes the Longformer for document-pair matching and cross-
document coreference resolution. While the pre-training objective in CDLM
exploits information from multiple documents, it differs from our setting of
joint inference over multiple short sentences.

##  3 Multi-Sentence Transformers Models

###  3.1 Multi-sentence Inference Tasks

AS2: We denote the question by $q$, and the set of answer candidates by
$C{=}\\{c_{1},\dots c_{n}\\}$. The objective is to re-rank $C$ and find the
best answer $A$ for q. AS2 is typically treated as a binary classification
task: first, a model $f$ is trained to predict the correctness/incorrectness
of each $c_{i}$; then, the candidate with the highest likelihood of being
correct is selected as an answer, i.e., $A{=}\text{argmax}^{n}_{i=1}\text{
}f(c_{i})$. Intuitively, modeling interrelated information between multiple
$c_{i}$’s can help in selecting the best answer candidate Zhang et al. (2021).

Fact Verification: We denote the claim by $F$, and the set of _evidences_ by
$C{=}\\{c_{1}\dots c_{n}\\}$ that are retrieved using DocIR. The objective is
to predict whether $F$ is supported/refuted/neither using $C$ (at least one
evidence $c_{i}$ is required for supporting/refuting $F$). Tymoshenko and
Moschitti (2021) jointly model evidences for supporting/refuting a claim as
they can complement each other.

Figure 1: Multi-sentence ‘Joint’ transformer model. $E_{i}$ refers to
embedding for the question/each candidate.

###  3.2 Joint Encoder Architecture

For jointly modeling multi-sentence inference tasks, we use a monolithic
transformer cross-encoder to encode multiple sentences using self-attention as
shown in Fig 1. To perform joint inference over $k$ sentences for question $q$
or claim $F$, the model receives concatenated sentences $[s_{0}\dots s_{k}]$
as input, where the first sentence is either the question or the claim
($s_{0}{=}q$ or $s_{0}{=}F$), and the remainder are $k$ candidates
$s_{i}{=}c_{i}\ ,i{=}\\{1\dots k\\}$. We pad (or truncate) each sentence
$s_{i}$ to the same fixed length $L$ (total input length $L{\times}(k+1)$),
and use the embedding for the [CLS] / [SEP] token in front of each sentence
$s_{i}$ as its embedding (denoted by $E_{i}$). Similar to Devlin et al., we
create positional embeddings of tokens using integers $0$ to $L(k{+}1){-}1$,
and extend the token type ids from $\\{0,1\\}$ to $\\{0\dots k\\}$
corresponding to $(k+1)$ input sentences.

###  3.3 Inference using Joint Transformer Model

We use the output embeddings $[E_{0}\dots E_{k}]$ of sentences for performing
prediction as following:

Predicting a single label: We use two separate classification heads to predict
a single label for the input to the joint model $[s_{0}\dots s_{k}]$: (i) IE1:
a linear layer on the output embedding $E_{0}$ of $s_{0}$ (similar to BERT)
referred to as the Individual Evidence (IE1) inference head, and (ii) AE1: a
linear layer on the average of the output embeddings
$[E_{0},E_{1},\dots,E_{k}]$ to explicitly factor in information from all
candidates, referred to as the Aggregated Evidence (AE1) inference head. For
Fact Verification, we use prediction heads IE1 and AE1.

Predicting Multiple Labels: We use two separate classification heads to
predict $k$ labels, one label each for every input $[s_{1}\dots s_{k}]$
specific to $s_{0}$: (i) IEk: a shared linear layer applied to the output
embedding $E_{i}$ of each candidate $s_{i}\ ,\ i\in\\{1\dots k\\}$ referred to
as $k$-candidate Individual Evidence (IEk) inference head, and (ii) AEk: a
shared linear layer applied to the concatenation of output embedding $E_{0}$
of input $s_{0}$ and the output embedding $E_{i}$ of each candidate $s_{i}\ ,\
i\in\\{1\dots k\\}$ referred to as $k$-candidate Aggregated Evidence (AEk)
inference head. For AS2, we use prediction heads IEk and AEk. Prediction heads
are illustrated in Figure 2.

Figure 2: Inference heads for joint transformer model. $E_{i}$ refers to
embedding for the question/each candidate.

###  3.4 Pre-training with Paragraph-level Signals

Long documents are typically organized into paragraphs to address the
document’s general topic from different viewpoints. The majority of
transformer pre-training strategies have not exploited this rich source of
information, which can possibly provide some weak supervision to the otherwise
unsupervised pre-training phase. To enable joint transformer models to
effectively capture dependencies across multiple sentences, we design a new
pre-training task where the model is (i) provided with $(k+1)$ sentences
$\\{s_{0}\dots s_{k}\\}$, and (ii) tasked to predict which sentences from
$\\{s_{1}\dots s_{k}\\}$ belong to the same paragraph $P$ as $s_{0}$ in the
document $D$. We call this pre-training task Multi-Sentences in Paragraph
Prediction (MSPP). We use the IEk and AEk prediction heads, defined above, on
top of the joint model to make $k$ predictions $p_{i}$ corresponding to
whether each sentence $s_{i},\ i{\in}\\{1\dots k\\}$ lies in the same
paragraph $P\in D$ as $s_{0}$. More formally:

$p_{i}=\begin{cases}1\ \ \text{if}\ s_{0},s_{i}\in P\ \text{in}\ D\\\ 0\ \
\text{otherwise}\\\ \end{cases}\forall i{=}\\{1,\dots,k\\}$

We randomly sample a sentence from a paragraph $P$ in a document $D$ to be
used as $s_{0}$, and then (i) randomly sample $k_{1}$ sentences (other than
$s_{0}$) from $P$ as positives, (ii) randomly sample $k_{2}$ sentences from
paragraphs other than $P$ in the same document $D$ as hard negatives, and
(iii) randomly sample $k_{3}$ sentences from documents other than $D$ as easy
negatives (note that $k_{1}{+}k_{2}{+}k_{3}{=k}$).

##  4 Experiments

We evaluate our joint transformers on three AS2 and one Fact Verification
datasets 111Code and pre-trained model checkpoints: https://github.com/amazon-
research/wqa-multi-sentence-inference. Common LM benchmarks, such as GLUE Wang
et al. (2018), are not suitable for our study as they only involve sentence
pair classification.

Model | ASNQ |  | WikiQA |  | TREC-QA
---|---|---|---|---|---
P@1 | MAP | MRR |  | P@1 | MAP | MRR |  | P@1 | MAP | MRR
Pairwise RoBERTa-Base | 61.8 (0.2) | 66.9 (0.1) | 73.1 (0.1) |  | 77.1 (2.1) | 85.3 (0.9) | 86.5 (1.0) |  | 87.9 (2.2) | 89.3 (0.9) | 93.1 (1.0)
Joint RoBERTa-Base ${\rightarrow}$ FT IEk | 3.4 (2.3) | 8.0 (1.9) | 10.0 (2.4) |  | 19.7 (1.9) | 39.4 (1.6) | 40.3 (1.8) |  | 30.9 (5.4) | 41.9 (2.4) | 50.8 (3.9)
Joint RoBERTa-Base ${\rightarrow}$ FT AEk | 3.6 (2.7) | 8.0 (2.2) | 10.2 (2.8) |  | 18.7 (3.9) | 39.0 (2.8) | 39.7 (2.9) |  | 29.7 (6.9) | 42.3 (3.2) | 49.2 (5.0)
(Ours) Joint MSPP IEk ${\rightarrow}$ FT IEk |  63.0 (0.3) | 67.2 (0.2) |  73.7 (0.2) |  |  82.7 (2.2) |  88.5 (1.5) |  89.0 (1.5) |  |  91.7 (2.2) |  91.1 (0.5) |  95.2 (1.3)
(Ours) Joint MSPP AEk ${\rightarrow}$ FT AEk |  63.0 (0.3) |  67.3 (0.2) |  73.7 (0.2) |  |  81.9 (2.6) |  87.9 (1.4) |  89.0 (1.5) |  | 88.7 (0.8) | 90.1 (1.0) | 93.6 (0.6)

Table 1: Results (std. dev. in parenthesis) on AS2 datasets. MSPP, FT refer to
our pre-training task and fine-tuning respectively. We indicate the prediction
head (IEk/AEk) used for both pre-training and fine-tuning. We underline
statistically significant gains over the baseline (Student t-test with $95\%$
confidence level).

###  4.1 Datasets

Pre-training:  To eliminate any improvements stemming from usage of more data,
we perform pre-training on the same corpora as RoBERTa: English Wikipedia, the
BookCorpus, OpenWebText and CC-News. For our proposed pre-training, we
randomly sample sentences from paragraphs as $s_{0}$, and choose
$k_{1}{=}1,k_{2}{=}2,k_{3}{=}2$ as the specific values for creating positive
and negative candidates for $s_{0}$. For complete details refer to Appendix A.

Fine-tuning: For AS2, we compare performance with MAP, MRR and Precision of
top ranked answer (P@1). For fact verification, we measure Label Accuracy
(LA). Brief description of datasets is presented below (details in Appendix
A):

  * •

ASNQ: A large AS2 dataset Garg et al. (2020) derived from NQ (Kwiatkowski et
al., 2019), where the candidate answers are from Wikipedia pages and the
questions are from search queries of the Google search engine. We use the dev.
and test splits released by Soldaini and Moschitti.

  * •

WikiQA: An AS2 dataset Yang et al. (2015) where the questions are derived from
query logs of the Bing search engine, and the answer candidate are extracted
from Wikipedia. We use the most popular _clean_ setting (questions having at
least one positive and one negative answer).

  * •

TREC-QA: A popular AS2 dataset Wang et al. (2007) containing factoid
questions. We only retain questions with at least one positive and one
negative answer in the development and test sets.

  * •

FEVER: A dataset for fact extraction and verification Thorne et al. (2018) to
retrieve evidences given a claim and identify if the evidences support/refute
the claim. As we are interested in the fact verification sub-task, we use
evidences retrieved by Liu et al. using a BERT-based DocIR.

###  4.2 Experimental Details and Baselines

We use $k{=}5$ for our experiments (following Zhang et al. (2021) and
Tymoshenko and Moschitti (2021)), and perform continued pre-training starting
from RoBERTa-Base using a combination of MLM and our MSPP pre-training for
100k steps with a batch size of 4,096. We use two different prediction heads,
IEk and AEk, for pre-training. For evaluation, we fine-tune all models on the
downstream AS2 and FEVER datasets using the corresponding IEk and AEk
prediction heads. We consider the pairwise RoBERTa-Base cross-encoder and
RoBERTa-Base LM used as a joint model with IEk and AEk prediction heads as the
baseline for AS2 tasks. For FEVER, we use several baselines: GEAR Zhou et al.
(2019), KGAT Liu et al. (2020), Transformer-XH Zhao et al. (2020), and three
models from Tymoshenko and Moschitti (2021): (i) Joint RoBERTa-Base with IE1
prediction head, (ii) Pairwise RoBERTa-Base with max-pooling, and (iii)
weighted-sum heads. For complete experimental details, refer to Appendix B.

###  4.3 Results

Model | ASNQ | WikiQA | TREC-QA
---|---|---|---
Pairwise RoBERTa-Base | 61.8 (0.2) | 77.1 (2.1) | 87.9 (2.2)
Joint RoBERTa-Base ${\rightarrow}$ FT IEk | 25.2 (3.1) | 24.6 (3.1) | 57.6 (4.8)
Joint RoBERTa-Base ${\rightarrow}$ FT AEk | 25.4 (3.3) | 26.4 (2.2) | 60.9 (4.9)
(Ours) Joint MSPP IEk ${\rightarrow}$ FT IEk |  63.9 (0.8) |  82.7 (3.0) |  92.2 (0.8)
(Ours) Joint MSPP AEk ${\rightarrow}$ FT AEk |  64.3 (1.1) |  82.1 (1.1) | 91.2 (2.9)

Table 2: P@1 of joint models for AS2 when re-ranking answers ranked in top-5
by pairwise RoBERTa-Base. Statistically significant results (Student t-test
$95\%$) are underlined. Complete results in Appendix C.

Answer Sentence Selection: The results for AS2 tasks are presented in Table 1,
averaged across five independent runs. From the table, we can see that the
RoBERTa-Base when used as a joint model for multi-candidate inference using
either the IEk or AEk prediction heads performs inferior to RoBERTa-Base used
as a pairwise cross-encoder. Across five experimental runs, we observe that
fine-tuning RoBERTa-Base as a joint model faces convergence issues (across
various hyper-parameters) indicating that the MLM pre-training task is not
sufficient to learn text semantics which can be exploited for multi-sentence
inference.

Our MSPP pre-trained joint models (with both IEk, AEk heads) get significant
improvements over the pairwise cross-encoder baseline and very large
improvements over the RoBERTa-Base joint model. The former highlights modeling
improvements stemming from joint inference over multiple-candidates, while the
latter highlights improvements stemming from our MSPP pre-training strategy.
Across all three AS2 datasets, our joint models are able to get the highest
P@1 scores while also improving the MAP and MRR metrics.

To demonstrate that our joint models can effectively use information from
multiple candidates towards prediction, we perform a study in Table 2 where
the joint models are used to re-rank the top-$k$ candidates ranked by the
pairwise RoBERTa-Base cross-encoder. Our joint models can significantly
improve the P@1 over the baseline for all datasets. The performance gap stems
from questions for which the pairwise RoBERTa model was unable to rank the
correct answer at the top position, but support from other candidates in the
top-k helped the joint model rank it in the top position.

Model | Dev | Test
---|---|---
GEAR | 70.69 | 71.60
KGAT with RoBERTa-Base | 78.29 | 74.07
Transformer-XH | 78.05 | 72.39
Pairwise RoBERTa-Base + MaxPool | 79.82 | -
Pairwise RoBERTa-Base + WgtSum | 80.01 | -
Joint RoBERTa-Base + FT IE1 | 79.25 | 73.56
(Ours) Joint Pre IEk \+ FT IE1 |  81.21 (0.24) | 74.39
(Ours) Joint Pre IEk \+ FT AE1 |  81.10 (0.15) | 74.25
(Ours) Joint Pre AEk \+ FT IE1 |  81.18 (0.14) | 73.77
(Ours) Joint Pre AEk \+ FT AE1 |  81.21 (0.16) | 74.13

Table 3: Results on FEVER dev and test sets. For our method, prediction heads
(IE1/AE1) are only used for fine-tuning (FT), while for pre-training (Pre) we
use (IEk/AEk) heads. ‘-’ denotes models not released publicly, and results not
reported in the paper. Statistically significant results (Student t-test
$95\%$) are underlined.

Fact Verification: The results for the FEVER task are presented in Table 3 and
show that our joint models (pre-trained with both the IEk and AEk heads and
fine-tuned with the IE1 and AE1 heads) outperform all previous baselines
considered, including the RoBERTa-Base joint model directly applied for multi-
sentence inference.

ASNQ
---
Q: Who invented the submarine during the civil war?
A1: H.L. Hunley , often referred to as Hunley , was a submarine of the
Confedera
A2: Hunley , McClintock , and Baxter Watson first built Pioneer , which was
tested
in February 1862 in the Mississippi River and was later towed to Lake
Pontchartrain
for additional trials .
A3: She was named for her inventor, Horace Lawson Hunley , shortly after she
was
taken into government service under the control of the Confederate States Army
at Charleston , South Carolina.
A4: 1864 painting of H.L. Hunley by Conrad Wise Chapman History Confederate
States
Name : H.L. Hunley Namesake : Horace Lawson Hunley Builder : James McClintock
Laid down : Early 1863 Launched : July 1863 Acquired : August 1863 In service:
Feb-
ruary 17 , 1864 Out of service : February 17, 1864 Status : Awaiting
conservation General
characteristics Displacement : 7.5 short tons ( 6.8 metric tons ) Length :
39.5 ft
A5: Johan F. Carlsen was born in Ærøskøbing April 9, 1841.
WikiQA
Q: What is the erb/heart?
A1: Heart valves are labeled with "B", "T", "A", and "P".First heart sound:
caused by
atrioventricular valves - Bicuspid/Mitral (B) and Tricuspid (T).
A2: Second heart sound caused by semilunar valves – Aortic (A) and Pulmonary/
Pulmonic (P).
A3: Front of thorax , showing surface relations of bones , lungs (purple),
pleura (blue),
and heart (red outline).
A4: In cardiology, Erb’s point refers to the third intercostal space on the
left sternal
border where sS2 is best auscultated .
A5: It is essentially the same location as what is referred to with left lower
sternal
border (LLSB).
TREC-QA
Q: When was the Khmer Rouge removed from power ?
A1: Sihanouk was named head of state after the Khmer Rouge seized power in
1975,
but was locked in his palace by the communists as they embarked on their
brutal
attempt to create an agrarian utopia .
A2: When a Vietnamese invasion drove the Khmer Rouge from power in 1979,
Duch fled with other Khmer Rouge leaders into the jungles.
A3: Religious practices were revived after the Khmer Rouge were driven from
power
by a Vietnamese invasion in 1979
A4: Moreover, 20 years after the Khmers Rouges were ousted from power,
Cambodia
still struggles on the brink of chaos , ruled by the gun , not by law .
A5: Sihanouk resigned in 1976 , but the Khmer Rouge kept him under house
arrest
until they were driven from power by an invading Vietnamese army in 1979 .

Table 4: Examples from AS2 datasets where the pairwise RoBERTa-Base model is
unable to rank a  correct answer for the question at the top position, but our
joint model (Joint MSPP IEk ${\rightarrow}$ FT IEk) can. We present answers
$\\{A1,\dots,A5\\}$ in their ranked order by the pairwise RoBERTa-Base model.
For all these examples we highlight the top ranked answer by the pairwise
RoBERTa-Base model in  red since it is incorrect.

Compute Overhead: We present a _simplified_ latency analysis for AS2 (assuming
sentence length $L$) as follows: a pairwise cross-encoder uses $k$ transformer
steps with input length $2L$, while our model uses $1$ step with input length
$(k{+}1){\times}L$. Since transformer attention scales quadratic on input
length, our model should take $\frac{(k{+}1)^{2}}{4k}$ times the inference
time of the cross-encoder, which is $1.8$ when $k{=}5$. However, when we fine-
tune for WikiQA on one A100-GPU, we only observe latency increasing from
$71s{\rightarrow}81s$ (only $14.1\%$ increase). The input embeddings and
feedforward layers vary linearly with input length, reducing overheads of
self-attention. Refer to Appendix C.3 for details.

Qualitative Examples: We present some qualitative examples from the three AS2
datasets highlighting cases where the pairwise RoBERTa-Base model is unable to
rank the correct answer on the top position, but our pre-trained joint model
(Joint MSPP IEk ${\rightarrow}$ FT IEk) can do this using supporting
information from other candidates in Table 4.

##  5 Conclusions

In this paper we have presented a multi-sentence cross-encoder for performing
inference jointly on multiple sentences for tasks like answer sentence
selection and fact verification. We have proposed a novel pre-training task to
capture paragraph-level semantics. Our experiments on three answer selection
and one fact verification datasets show that our pre-trained joint models can
outperform pairwise cross-encoders and pre-trained LMs when directly used as
joint models.

## References

  * Ai et al. (2018) Qingyao Ai, Keping Bi, Jiafeng Guo, and W. Bruce Croft. 2018.  Learning a deep listwise context model for ranking refinement.  _The 41st International ACM SIGIR Conference on Research and Development in Information Retrieval_.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv:2004.05150_.
  * Bian et al. (2017) Weijie Bian, Si Li, Zhao Yang, Guang Chen, and Zhiqing Lin. 2017.  A compare-aggregate model with dynamic-clip attention for answer selection.  In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_ , CIKM ’17, page 1987–1990, New York, NY, USA. Association for Computing Machinery.
  * Bonadiman and Moschitti (2020) Daniele Bonadiman and Alessandro Moschitti. 2020.  A study on efficiency, accuracy and document structure for answer sentence selection.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 5211–5222, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Caciularu et al. (2021) Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew Peters, Arie Cattan, and Ido Dagan. 2021.  CDLM: Cross-document language modeling.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 2648–2662, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020.  Electra: Pre-training text encoders as discriminators rather than generators.  In _International Conference on Learning Representations_.
  * Detlefsen et al. (2022) Nicki Skafte Detlefsen, Jiri Borovec, Justus Schock, Ananya Harsh Jha, Teddy Koker, Luca Di Liello, Daniel Stancl, Changsheng Quan, Maxim Grechkin, and William Falcon. 2022.  Torchmetrics - measuring reproducibility in pytorch.  _Journal of Open Source Software_ , 7(70):4101.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Garg and Moschitti (2021) Siddhant Garg and Alessandro Moschitti. 2021.  Will this question be answered? question filtering via answer model distillation for efficient question answering.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7329–7346, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Garg et al. (2020) Siddhant Garg, Thuy Vu, and Alessandro Moschitti. 2020.  Tanda: Transfer and adapt pre-trained transformer models for answer sentence selection.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 34(05):7780–7788.
  * Ginzburg et al. (2021) Dvir Ginzburg, Itzik Malkiel, Oren Barkan, Avi Caciularu, and Noam Koenigstein. 2021\.  Self-supervised document similarity ranking via contextualized language models and hierarchical inference.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 3088–3098, Online. Association for Computational Linguistics.
  * Giorgi et al. (2021) John Giorgi, Osvald Nitski, Bo Wang, and Gary Bader. 2021.  DeCLUTR: Deep contrastive learning for unsupervised textual representations.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 879–895, Online. Association for Computational Linguistics.
  * Gokaslan and Cohen (2019) Aaron Gokaslan and Vanya Cohen. 2019.  Openwebtext corpus.  http://Skylion007.github.io/OpenWebTextCorpus.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020.  REALM: Retrieval-augmented language model pre-training.  _arXiv preprint arXiv:2002.08909_.
  * Joshi et al. (2020) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020.  SpanBERT: Improving pre-training by representing and predicting spans.  _Transactions of the Association for Computational Linguistics_ , 8:64–77.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020.  Albert: A lite bert for self-supervised learning of language representations.  In _International Conference on Learning Representations_.
  * Liello et al. (2021) Luca Di Liello, Matteo Gabburo, and Alessandro Moschitti. 2021.  Efficient pre-training objectives for transformers.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Liu et al. (2020) Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. 2020.  Fine-grained fact verification with kernel graph attention network.  In _Proceedings of ACL_.
  * Shen et al. (2017) Gehui Shen, Yunlun Yang, and Zhi-Hong Deng. 2017.  Inter-weighted alignment network for sentence pair modeling.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 1179–1189, Copenhagen, Denmark. Association for Computational Linguistics.
  * Soldaini and Moschitti (2020) Luca Soldaini and Alessandro Moschitti. 2020.  The cascade transformer: an application for efficient answer sentence selection.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5697–5708, Online. Association for Computational Linguistics.
  * Stammbach and Ash (2020) Dominik Stammbach and Elliott Ash. 2020.  e-fever: Explanations and summaries forautomated fact checking.  In _TTO_.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018\.  FEVER: a large-scale dataset for fact extraction and VERification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics.
  * Tymoshenko and Moschitti (2021) Kateryna Tymoshenko and Alessandro Moschitti. 2021.  Strong and light baseline models for fact-checking joint inference.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 4824–4830, Online. Association for Computational Linguistics.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Wang et al. (2007) Mengqiu Wang, Noah A. Smith, and Teruko Mitamura. 2007.  What is the Jeopardy model? a quasi-synchronous grammar for QA.  In _Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL)_ , pages 22–32, Prague, Czech Republic. Association for Computational Linguistics.
  * Wang et al. (2020) Wei Wang, Bin Bi, Ming Yan, Chen Wu, Jiangnan Xia, Zuyi Bao, Liwei Peng, and Luo Si. 2020.  Structbert: Incorporating language structures into pre-training for deep language understanding.  In _International Conference on Learning Representations_.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Xiao et al. (2021) Wen Xiao, Iz Beltagy, Giuseppe Carenini, and Arman Cohan. 2021.  PRIMER: pyramid-based masked sentence pre-training for multi-document summarization.  _CoRR_ , abs/2110.08499.
  * Yang et al. (2015) Yi Yang, Scott Wen-tau Yih, and Chris Meek. 2015.  Wikiqa: A challenge dataset for open-domain question answering.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_. ACL - Association for Computational Linguistics.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. 2020.  Big bird: Transformers for longer sequences.  _Advances in Neural Information Processing Systems_ , 33.
  * Zhang et al. (2021) Zeyu Zhang, Thuy Vu, and Alessandro Moschitti. 2021.  Joint models for answer verification in question answering systems.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3252–3262, Online. Association for Computational Linguistics.
  * Zhao et al. (2020) Chen Zhao, Chenyan Xiong, Corby Rosset, Xia Song, Paul Bennett, and Saurabh Tiwary. 2020.  Transformer-xh: Multi-evidence reasoning with extra hop attention.  In _The Eighth International Conference on Learning Representations (ICLR 2020)_.
  * Zhong et al. (2020) Wanjun Zhong, Jingjing Xu, Duyu Tang, Zenan Xu, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. 2020.  Reasoning over semantic-level graph for fact checking.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6170–6180, Online. Association for Computational Linguistics.
  * Zhou et al. (2019) Jie Zhou, Xu Han, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. 2019.  GEAR: Graph-based evidence aggregating and reasoning for fact verification.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 892–901, Florence, Italy. Association for Computational Linguistics.

## Appendix

##  Appendix A Datasets

We present the complete details for all the datasets used in this paper along
with links to download them for reproducibility of results.

###  A.1 Pre-training Datasets

We use the Wikipedia222https://dumps.wikimedia.org/enwiki/20211101/,
BookCorpus333https://huggingface.co/datasets/bookcorpusopen, OpenWebText
(Gokaslan and Cohen, 2019) and CC-News444https://commoncrawl.org/2016/10/news-
dataset-available/ datasets for performing pre-training of our joint
transformer models. We do not use the STORIES dataset as it is no longer
available for research use
555https://github.com/tensorflow/models/tree/archive/research/lm_commonsense#1-download-
data-files. After decompression and cleaning we obtained 6GB, 11GB, 38GB and
394GB of raw text respectively from the BookCorpus, Wikipedia, OpenWebText and
CC-News.

###  A.2 Finetuning Datasets

We evaluate our joint transformers on three AS2 and one Fact Verification
datasets. The latter differs from the former in not selecting the best
candidate, but rather explicitly using all candidates to predict the target
label. Here are the details of the finetuning datasets that we use for our
experiments along with data statistics for each dataset:

Dataset | Split | # Questions | # Candidates | Avg. # C/Q
---|---|---|---|---
ASNQ | Train | 57,242 | 20,377,568 | 356.0
Dev | 1,336 | 463,914 | 347.2
Test | 1,336 | 466,148 | 348.9
WikiQA | Train | 2,118 | 20,360 | 9.6
Dev | 122 | 1,126 | 9.2
Test | 237 | 2,341 | 9.9
TREC-QA | Train | 1,226 | 53,417 | 43.6
Dev | 69 | 1,343 | 19.5
Test | 68 | 1,442 | 21.2

Table 5: Statistics for ASNQ, WikiQA and TREC-QA datasets.

  * •

ASNQ: A large-scale AS2 dataset Garg et al.
(2020)666https://github.com/alexa/wqa_tanda where the candidate answers are
from Wikipedia pages and the questions are from search queries of the Google
search engine. ASNQ is a modified version of the Natural Questions (NQ)
Kwiatkowski et al. (2019) dataset by converting it from a machine reading to
an AS2 dataset. This is done by labelling sentences from the long answers
which contain the short answer string as positive correct answer candidates
and all other answer candidates as negatives. We use the dev. and test splits
released by Soldaini and Moschitti777https://github.com/alexa/wqa-cascade-
transformers.

  * •

WikiQA: An AS2 dataset released by Yang et al.888http://aka.ms/WikiQA where
the questions are derived from query logs of the Bing search engine, and the
answer candidate are extracted from Wikipedia. This dataset has a subset of
questions having no correct answers (all-) or having only correct answers
(all+). We remove both the all- and all+ questions for our experiments
(“clean" setting).

  * •

TREC-QA: A popular AS2 dataset released by Wang et al.. For our experiments,
we trained on the train-all split, which contains more noise but also more
question-answer pairs. Regarding the dev. and test sets we removed the
questions without answers, or those having only correct or only incorrect
answer sentence candidates. This setting refers to the “clean" setting Shen et
al. (2017), which is a TREC-QA standard.

  * •

FEVER: A popular benchmark for fact extraction and verification released by
Thorne et al. The aim is to retrieve evidences given a claim, and then
identify whether the retrieved evidences support or refute the claim or if
there is not enough information to make a choice. For supporting/refuting a
claim, at least one of the retrieved evidences must support/retrieve the
claim. Note that the performance on FEVER depends crucially on the retrieval
system and the candidates retrieved. For our experiments, we are interested
only in the fact verification sub-task and thus we exploit the evidences
retrieved by Liu et al. using a BERT-based
DocIR999https://github.com/thunlp/KernelGAT/tree/master/data.

Split | # Claims | # Evidences | Avg. # E/C
---|---|---|---
Train | 145,406 | 722,473 | 4.97
Dev | 19,998 | 98,915 | 4.95
Test | 19,998 | 98,839 | 4.94
Table 6: Statistics for the FEVER dataset where evidences has been retrieved
using (Liu et al., 2020).

##  Appendix B Experimental Setup

###  B.1 Complete Experimental Details

Following standard practice, the token ids, positional ids and token type ids
are embedded using separate embedding layers, and their sum is fed as the
input to the transformer layers. We use $k{=}5$ for our experiments (following
Zhang et al.; Tymoshenko and Moschitti), and perform continuous pre-training
starting from the RoBERTa-Base checkpoint using a combination of MLM and our
MSPP pre-training objective for 100,000 steps with a batch size of 4096. We
use a triangular learning rate with 10,000 warmup steps and a peak value of
$5*10^{-5}$. We use Adam optimizer with $\beta_{1}=0.9$, $\beta_{2}=0.999$ and
$\epsilon=10^{-8}$. We apply a weight decay of $0.01$ and gradient clipping
when values are higher than $1.0$. We set the dropout ratio to $0.1$ and we
use two different prediction heads for pre-training: IEk and AEk. We follow
the strategy of Devlin et al. (2019); Lan et al. (2020), and equally weight
the the two pre-training loss objectives: MLM and MSPP.

For evaluation, we fine-tune all models on the downstream AS2 and FEVER
datasets: using the same IEk and AEk prediction heads exploited in pre-
training for AS2 and using either IE1 or AE1 prediction heads for FEVER. We
finetune every model with the same maximum sequence length equal to
$64*(k+1)=384$ tokens. For ASNQ we train for up to 6 epochs with a batch size
of 512 and a learning rate of $10^{-5}$ with the same Adam optimizer described
above but warming up for only 5000 steps. We do early stopping on the MAP of
the development set. For WikiQA and TREC-QA, we created batches of 32 examples
and we used a learning equal to $2*10^{-6}$ and 1000 warm up steps. We train
for up to 40 epochs again with early stopping on the MAP of the development
set. On FEVER, we use a batch size of 64, a learning rate of $10^{-5}$, 1000
warm up steps and we do early stopping checking the Accuracy over the
development set. We implemented our code based on HuggingFace’s Transformers
library Wolf et al. (2020).

###  B.2 Baselines

For AS2, we consider two baselines: (i) pairwise RoBERTa-Base model when used
as a cross-encoder for AS2, and (ii) RoBERTa-Base LM when used as a joint
model with IEk and AEk prediction heads independently for AS2 tasks.

For FEVER, we use several recent baselines from Tymoshenko and Moschitti: (i)
GEAR Zhou et al. (2019), (ii) KGAT Liu et al. (2020), (iii) Transformer-XH
Zhao et al. (2020), (iv) joint RoBERTa-Base with IE1 prediction head
Tymoshenko and Moschitti (2021), (v) pairwise RoBERTa-Base when used as a
cross-encoder with max-pooling head Tymoshenko and Moschitti (2021), (vi)
pairwise RoBERTa-Base when used as a cross-encoder with weighted-sum head
Tymoshenko and Moschitti (2021).

We used metrics from Torchmetrics Detlefsen et al. (2022) to compute MAP, MRR,
Precision@1 and Accuracy.

###  B.3 Metrics

The performance of AS2 systems in practical applications is typically Garg and
Moschitti (2021) measured using the Accuracy in providing correct answers for
the questions (the percentage of correct responses provided by the system),
also called the Precision-at-1 (P@1). In addition to P@1, we use Mean Average
Precision (MAP) and Mean Reciprocal Recall (MRR) to evaluate the ranking
produced of the set of candidates by the model.

For FEVER, we measure the performance using Label Accuracy (LA), a standard
metric for this dataset, that measures the accuracy of predicting
support/refute/neither for a claim using a set of evidences.

##  Appendix C Complete Results and Discussion

Model | ASNQ |  | WikiQA |  | TREC-QA
---|---|---|---|---|---
P@1 | MAP | MRR |  | P@1 | MAP | MRR |  | P@1 | MAP | MRR
Pairwise RoBERTa-Base | 61.8 | 66.9 | 73.1 |  | 77.1 | 85.3 | 86.5 |  | 87.9 | 89.3 | 93.1
Joint RoBERTa-Base ${\rightarrow}$ FT IEk | 25.2 | 44.0 | 45.6 |  | 24.6 | 49.3 | 49.7 |  | 57.6 | 73.7 | 74.6
Joint RoBERTa-Base ${\rightarrow}$ FT AEk | 25.4 | 44.8 | 46.2 |  | 26.4 | 50.6 | 51.1 |  | 60.9 | 74.6 | 76.7
(Ours) Joint MSPP IEk ${\rightarrow}$ FT IEk | 63.9 | 71.3 | 73.1 |  | 82.7 | 88.5 | 89.0 |  | 92.2 | 93.5 | 95.4
(Ours) Joint MSPP AEk ${\rightarrow}$ FT AEk | 64.3 | 71.5 | 73.4 |  | 82.1 | 87.9 | 88.7 |  | 91.2 | 93.5 | 94.9

Table 7: Complete results of our joint models for AS2 datasets when re-ranking
the answer candidates ranked in top-k by Pairwise RoBERTa-Base. MSPP, FT refer
to our pre-training task and finetuning respectively. We indicate the
prediction head (IEk/AEk) used for both pre-training and finetuning.

Model | Dev | Test
---|---|---
GEAR | 70.69 | 71.60
KGAT with RoBERTa-Base | 78.29 | 74.07
Transformer-XH | 78.05 | 72.39
Pairwise BERT-Base | 73.30 | 69.75
Pairwise RoBERTa-Base + MaxPool | 79.82 | -
Pairwise RoBERTa-Base + WgtSum | 80.01 | -
Joint BERT-Base | 73.67 | 71.01
Joint RoBERTa-Base + FT IE1 | 79.25 | 73.56
(Ours) Joint Pre IEk \+ FT IE1 |  81.21 (0.24) | 74.39
(Ours) Joint Pre IEk \+ FT AE1 |  81.10 (0.15) | 74.25
(Ours) Joint Pre AEk \+ FT IE1 |  81.18 (0.14) | 73.77
(Ours) Joint Pre AEk \+ FT AE1 |  81.21 (0.16) | 74.13
Methods with larger models and/or sophisticated retrieval
DOMLIN++ | 77.48 | 76.60
DREAM | 79.16 | 76.85

Table 8: Complete Results on FEVER dev and test sets. For our method,
prediction heads (IE1/AE1) are only used for finetuning (FT), while for pre-
training (Pre) we use the (IEk/AEk) heads. ’-’ denotes models that are not
publicly released and have no reported results on the test split in their
published paper. Statistically significant results (T-Test $95\%$) are
underlined.

###  C.1 Results on AS2 with cascaded pairwise and Joint re-ranker

Below we present results of evaluating our joint models to re-rank the top-$k$
candidates ranked by the pairwise RoBERTa-Base cross-encoder. Our joint models
can significantly improve the P@1, MAP and MRR over the baseline for all
datasets. The performance gap stems from questions for which the pairwise
RoBERTa model was unable to rank the correct answer at the top position, but
support from other candidates in the top-k helped the joint model rank it in
the top position.

###  C.2 Results on FEVER

Here we present complete results on the FEVER dataset in Table 8, by also
presenting some additional baselines such as: (i) pairwise BERT-Base cross-
encoder Tymoshenko and Moschitti (2021), (ii) joint BERT-Base cross-encoder
with IE1 prediction head, (iii) DOMLIN++ Stammbach and Ash (2020) which uses
additional DocIR components and data (MNLI Williams et al. (2018)) for fine-
tuning, (iv) DREAM Zhong et al. (2020) that uses the XL-Net model. Note that
comparing our joint models with (iii) and (iv) is unfair since they use
additional retrieval components, datasets and larger models. We just include
these results here for the sake for completeness. Interestingly, our joint
models outperform DREAM and DOMLIN++ on the dev set without using additional
retrieval and larger models.

###  C.3 Compute Overhead of Joint Models

Change in Number of Model Parameters: The transformer block of our joint
inference model is identical to pre-trained models such as RoBERTa, and
contains the exact same number of parameters. Classification heads
$IE_{1},IE_{k}$ and $AE_{1}$ all operate on the embedding of a single token,
and are identical to the classification head of RoBERTa ($AE_{k}$ operates on
the concatenation of two token embeddings, and contains double the number of
parameters as the RoBERTa). The maximum sequence length allowed for both the
models is the same (512). The exact number of parameters of our joint model
with $AE_{k}$ and the RoBERTa model are $124,062,720$ and $124,055,040$
respectively.

Change in Inference Latency: While our joint model provides a longer input
sequence to the transformer, it also reduces the number of forward passes that
need to be done by a pairwise cross-encoder. A _simplified_ latency analysis
for AS2 (assuming each sentence has a length $L$): pairwise cross-encoder will
need to make $k$ forward passes of the transformer with a sequence of length
$2L$ ($q$ with each candidate $c_{i}$), while our joint model will only need
to make $1$ forward pass of the transformer with input length
$(k{+}1){\times}L$ ($q$ with $k$ candidates). Transformer self-attention is
quadratic in input sequence length, so this should lead to the inference time
of out joint model being $\frac{(k+1)^{2}}{4k}$ times the inference time of
the cross-encoder. However, the input embedding layer and the feedforward
layers are linear in input sequence length, so this should lead to a reduction
in the inference time of our joint model by $\frac{(k+1)}{2k}$ times the
inference time of the cross-encoder. Empirically, when we fine-tune for WikiQA
on one A100-GPU, we only observe latency increasing from $71s{\rightarrow}81s$
(increase of only $14.1\%$).