# W-Transformers : A Wavelet-based Transformer Framework for Univariate Time
Series Forecasting

Lena Sasal  Sorbonne Center for Artificial Intelligence
Sorbonne University Abu Dhabi
Abu Dhabi, UAE
<EMAIL_ADDRESS>Tanujit Chakraborty  Dept. of Science and Engineering
Sorbonne University Abu Dhabi
Abu Dhabi, UAE
<EMAIL_ADDRESS>Abdenour Hadid  Sorbonne Center for
Artificial Intelligence
Sorbonne University Abu Dhabi
Abu Dhabi, UAE
<EMAIL_ADDRESS>
###### Abstract

Deep learning utilizing transformers has recently achieved a lot of success in
many vital areas such as natural language processing, computer vision, anomaly
detection, and recommendation systems, among many others. Among several merits
of transformers, the ability to capture long-range temporal dependencies and
interactions is desirable for time series forecasting, leading to its progress
in various time series applications. In this paper, we build a transformer
model for non-stationary time series. The problem is challenging yet crucially
important. We present a novel framework for univariate time series
representation learning based on the wavelet-based transformer encoder
architecture and call it W-Transformer. The proposed W-Transformers utilize a
maximal overlap discrete wavelet transformation (MODWT) to the time series
data and build local transformers on the decomposed datasets to vividly
capture the nonstationarity and long-range nonlinear dependencies in the time
series. Evaluating our framework on several publicly available benchmark time
series datasets from various domains and with diverse characteristics, we
demonstrate that it performs, on average, significantly better than the
baseline forecasters for short-term and long-term forecasting, even for
datasets that consist of only a few hundred training samples.

###### Index Terms:

Transformers, time series forecasting, deep learning, wavelet decomposition.

##  I Introduction

Forecasting the future movement and value of time series is a key component of
formulating effective strategies in most business, industrial fields, and
medical domains, among many others [1, 2]. Specific applications include
forecasting epidemiological cases (including Covid-19) [3, 4], stock price
prediction [5], weather forecasts [6], web traffic forecasting [7] and others
applied fields [8, 9]. Over the last decades, many techniques have evolved in
time series forecasting literature. Statistics models (such as autoregressive
integrated moving average (ARIMA) [10], state-space exponential smoothing
(ETS) [2], and self-excited threshold autoregressive (SETAR) [11] models) have
been intensively used in several applied domains due to their easy
interpretability. However, they tend to fail when highly complex situations
arise (for example, nonlinear and nonstationary time series) and require
expertise in manually selecting trends, seasonality, and other components. On
the other hand, machine learning methods such as LightGBM [12] and
autoregressive neural networks (ARNN) [13] can leverage the ground truth data
to learn the trends and patterns in an automated manner while sometimes
failing when there is long-term dependence on the observed time series data.

Current progress in deep learning has brought us deep autoregressive (DeepAR)
[14] models, temporal convolutional networks (TCN) [15] and many others [16,
17] which can deal with nonlinear and non stationary time series. Time series
data have also been modeled using Recurrent Neural Network (RNN) [18] in an
autoregressive way that can tackle the irregularity of the time series.
However, due to the gradient vanishing and exploding problem, RNNs are
challenging to train. The problems continue to exist despite the introduction
of numerous versions, such as long short-term memory (LSTM) [19] and gated
recurrent unit (GRU) [20] that often outperform standard deep models. However,
practical forecasting applications frequently contain both long and short-term
recurrent patterns, and RNN-based methods sometimes struggle to capture long-
term dependencies.

The introduction of Transformers in deep learning [21] has brought great
interests for researchers in natural language processing [22], computer vision
[23], and other disciplines [24]. Transformers have shown great success in
modeling and extrapolating the long-range dependencies and interactions in
temporal data; hence they are pertinent in time series forecasting [25]. Among
various applications of Transformers for time series forecasting, some seminal
papers are: probabilistic Transformer for multivariate time series
representation learning [26, 27], deep transformer models for epidemic
forecasting [28] and many others [24]. The multi-headed attention mechanism of
Transformers makes them particularly suitable for time series data analysis:
(a) they concurrently represent each input sequence element by considering its
context, (b) while multiple attention heads can consider different
representation subspaces, i.e., multiple aspects of relevance between input
elements – for time series, this, for example, may correspond to multiple
periodicities in the signal [26]. Other applications of Transformers include
spatio-temporal forecasting in which, introducing a temporal Transformer block
to capture temporal dependencies, it is also essential to design a spatial
Transformer block [24]. Transformers are also proven effective in various time
series classification tasks due to their superior capability in capturing
long-term dependency [29]. Compared to RNN and other neural network
structures, the advantage of Transformers is that they easily enable parallel
processing and reduce computational time since they do not involve recurrent
behavior. A remaining challenge within Transformers is how to effectively
model long-range and short-range temporal dependency and capture
nonstationarity and nonlinearity simultaneously for forecasting time series
data.

So far, many sophisticated decomposition techniques have shown strong
competitiveness in time series pre-processing tasks, such as Fourier
transformation [30], fast Fourier transformation [31], and Wavelet
decomposition [32] to cite a few. These methods of decomposing the complex
time series into simpler parts have often led to satisfactory results. Among
them, Wavelet decomposition, a mathematical tool that can reveal information
within the signals in both the time and scale (frequency) domains, overcomes
the basic drawback of Fourier analysis [3]. Most of the previous applications
of wavelets have used discrete wavelet transform (DWT) and during the task of
forecasting, it has been combined with statistical or machine learning
algorithm [33, 34]. The limitation of DWT on the signal length made us use the
Maximal Overlap discrete transform (MODWT) algorithm which counters this
limitation but has the same abilities [35]. In fact, MODWT preserves the
number of coefficients of the original time series for all the levels in the
decomposition. Wavelet-ARIMA (WARIMA) [36] and Wavelet neural network [4] are
some of the hybrid algorithms that combine MODWT algorithm and statistical or
machine learning models. Those hybrid approaches have been used in several
practical applications, for example, epidemics [4], hydrology [37], traffic
volumes [34], and wind speed [33] forecasting problems.

Motivated by the above observations, we propose a novel framework combining
the MODWT algorithm with Transformers to effectively model the long-range and
short-range temporal dependency and capture nonstationarity and nonlinearity
simultaneously of the time series. Incorporating wavelets within Transformers
will enable the model to capture all the irregularities of the time series by
having a more detailed decomposition of low and high-frequency coefficients to
separate the signal from the noise. Furthermore, scaling the wavelet used on
the signal will reveal more about the frequency or the time. In fact,
stretching the wavelet will give more information about the frequency while
losing some time localization. Similarly, squeezing the wavelet will result in
losing a part of the frequency domain while better preserving time location.
Given those observations, our proposed W-Transformers is expected to generate
more accurate and stable out-of-sample forecasts for long-term horizons for
nonstationary time series data. Our main contributions to this paper can be
described as follows :

  * •

We propose a novel model (namely W-Transformers) for time series forecasting
elegantly combining wavelet decomposition and Transformers in the ensemble
approach.

  * •

The proposed W-Transformers possess the capacity to handle long-range
dependencies and nonlinear behavior (with the help of Transformers) to capture
the nonstationarity of the time series (with the use of Wavelet decomposition)
in its ensemble framework. Thus, it is expected to be more reliable and
accurate than conventional transformers and other deep learning architectures.

  * •

We perform extensive experiments with multiple univariate time series
datasets, demonstrating superior performance compared with the current state-
of-the-art forecast methods for short-term and long-term predictions.

##  II Mathematical Preliminaries

This section discusses the mathematical foundation of the wavelet
decomposition to be used in our proposed W-Transformers model. For more
details, one may refer to [32].

###  II-A Wavelet Decomposition

Wavelet is a wave-like oscillation localized in time. It can capture the time
and the frequency of a signal, contrary to Fourier Transform, which only
captures the frequency of the signal. There are two different kinds of
wavelets: (a) Continuous wavelet transforms (CWT), which applies an infinite
number of wavelet (i.e., every possible wavelet) and (b) Discrete wavelet
transform (DWT), where a finite number of wavelets in specific time and
location are applied. In our work, we focus on DWT that represents a signal
using an orthonormal basis representation, widely used in hydrology [37],
epidemics [4], and geophysics [38] to name a few. This decomposition will give
us a set of time series where each time series has a coefficient describing
the evolution in time of the signal in a specific frequency band. The DWT uses
a discrete dyadic (octave) grid for scale parameter $i$ and shift parameter
$k$, and the equation for forward DWT is as follows [32]:

$c(i,k)=\sum_{t}f(t)\Psi_{i,k}^{*}(t),\text{ where
}\Psi_{i,k}^{*}(t)=2^{\frac{i}{2}}\Psi\left(2^{i}t-k\right).$

The mathematical equation for inverse DWT is as follows:

$f(t)=\sum_{k}\sum_{i}c(i,k)\Psi_{i,k}(t),$

where $f(\cdot)$ is a function and $\Psi$ the mother wavelet [32].

###  II-B MODWT algorithm

Since DWT restricts the sample size to be exactly a power of $2$, a modified
version of the DWT, namely maximal overlapping discrete wavelet
transformations (MODWT), is used for decomposing arbitrary time series [35].
Both MODWT and DWT can handle multi-resolution analysis, which is a scaled-
based additive decomposition, but in the case of the MODWT, it also has the
capacity to catch the circular shift of the signal. The shift-invariance
property of MODWT is an added advantage over DWT in its time series
applications. Below, we define the detail coefficients $(\tilde{s}_{j,l})$ and
scaling coefficients $(\tilde{s}_{j,l})$ that are produced by an MODWT
algorithm with Haar filter (most commonly used wavelet filter in time series
literature) [39].

$\tilde{d}_{j,l}={d_{j,l}}/2^{\frac{j}{2}}\;\;\text{and}\;\;{\tilde{s}_{j,l}}={s_{j,l}}/{2^{\frac{j}{2}}},$

where $d_{j,l}$ and $s_{j,l}$ are wavelet and scaling filters of DWT
algorithm. The coefficient of level $j$ $(l=0,1,\ldots,L-1)$ then convolve the
original time series $\\{Y_{t}:t=0,1,\ldots,N-1\\}$ and the MODWT pyramid
algorithm generates the MODWT detail coefficients $(\tilde{D}_{j,t})$ and
scaling coefficients $(\tilde{S}_{j,t})$ as follows [32]:

$\tilde{D}_{j,t}=\sum_{l=0}^{L_{j}-1}\tilde{d}_{j,l}Y_{(t-l)modN}\;\;\text{and}\;\;\tilde{S}_{j,t}=\sum_{l=0}^{L_{j}-1}\tilde{s}_{j,l}Y_{(t-l)modN},$

where $L_{j}=(2^{j}-1)(L-1)+1$. All the levels and the smooth series will have
the same length as the original one.

##  III Proposed Method: W-Transformers

In our proposed approach, we use the MODWT algorithm (discussed in Section
II-B) as a data pre-processing method to capture more explicit information
about the observed time series. The MODWT algorithm will provide us with $J$
detail coefficients (higher frequency bands) and a smooth (trend component) of
the available time series $Y_{t}\;(t=1,2,\ldots,N)$, where $N$ is the number
of past observations. We show a pictorial view of the MODWT decomposition
using a Haar filter for the publicly available website traffic data in Fig. 1.
Wavelet levels are chosen as an integer value that specifies the number of
time series decompositions into smooth details. We set it as
$J+1=\lfloor\log_{e}{N}\rfloor$ based on the recommendation in [32]. These
high and low-frequency decomposed series are termed wavelet (details), and
scaling (smooth) coefficients that can track the original series as:

$Y_{t}=\sum_{j=1}^{J}D_{j,t}+S_{j,t},$

where $D_{j,t}$ is the $j^{th}$ level details and $S_{j,t}$ is the smooth of
the decomposed time series. The MRA-based MODWT decomposed series can handle
nonstationarity and seasonality in the time series datasets and capture the
overall trend of the series. Each level of the time series and the smooth
component are then passed into several independent Transformers for the
prediction task. Time series datasets are passed through two parallel networks
starting both an embedding layer. The encoder embedding takes the input of the
batch train data and the decoder takes the output of the batch train data.
Further, the output of both the encoder and decoder will enter into a second
decoder. The encoder is composed of multi-head attention, a normalization
layer, and a feedforward layer. The first decoder has a masked multi-head
attention layer and normalization, and the second decoder has the same
structure as the encoder. We apply a linear and a softmax layer to the output
of the second decoder, which gives us the weight parameters of our model. Once
all the epochs are done, we have our final model ready for making predictions
and one-step ahead forecasts. In the end, we predict a time series for each
level of decomposition and the smooth size of the forecasting horizon with
their respective transformer model and reverse the process of MODWT with
inverse MODWT as discussed in Sec. II-B. The result of the inverse MODWT will
be our final prediction that could be compared with the test time series.
Multi-step ahead forecasts are generated using an iterative method applied to
the fitted W-Transformers model. To generate $h$-step ahead forecasts
$(\hat{Y}_{N+h})$, we use the following equation:

$\hat{Y}_{N+h}=\sum_{j=1}^{J}\hat{D}_{j,N+h}+\hat{S}_{J,N+h},$

where

$\displaystyle\hat{D}_{j,N+h}$
$\displaystyle=f(D_{j,1},D_{j,2},\ldots,D_{j,N});\;j=1,2,\ldots,J,$
$\displaystyle\hat{S}_{J,N+h}$
$\displaystyle=f(S_{J,1},S_{J,2},\ldots,S_{J,N}),$

and $f$ is the Transformer model. We describe the mathematical formulation of
the Transformer model that we use in our ensemble W-Transformers framework to
keep the current work self-contained, following [21]. Once the nonstationarity
and seasonality are handled by the MODWT algorithm, then Transformers model
the long-range dependencies and interactions in sequential decomposed
datasets. Mathematically, the self-attention mechanism of Transformers is
defined as follows:

$Attn(Q,K,V)=\text{softmax}\left(\frac{QK^{T}}{\sqrt{d}}\right)V,$

where $Q$ is the query matrix, $K$ is the key matrix, $V$ is the value matrix,
and $d$ is the dimension of $Q$, $K$, and $V$. Next, the softmax function is
used to obtain the weights on the values to be put into a matrix $Q$. Now, the
keys, queries, and values are then linearly projected, and these projections
perform attention function in parallel yielding the output values.
Theoretically, we define these heads as follows:

$head_{p}=Attn(QW^{Q}_{p},KW^{K}_{p},VW^{V}_{p}),$

where the projections are parameter matrices, denoted by
$W^{Q}_{p},W^{K}_{p},W^{V}_{p}$. Finally, the multi-head attention is simply a
concatenation of the $head$ for $m$-times, where $m$ is an hyper-parameter
that is usually set to a pre-specified integer value $8$ (as in [21]) and $d$
the dimension of $Q$ and $V$ defined by $d=\frac{d_{model}}{m}$, where
$d_{model}$ is the dimension of the output of the embedding layer. To jointly
attend the information from various subspaces, we use multi-head attention
defined as follows:

$\text{MultiHead}(Q,K,V)=\text{Concat}(head_{1},...,head_{m})W^{O},$

where $W^{O}$ is the projection matrix of the output layer.

Algorithm 1 W-Transformers

The original Time Series (training set): $Y_{t}$ (t=1,2,…,N)

The prediction of the testing series: $\hat{Y}_{N+h}$

$NForecast\leftarrow\text{Number of data point of testing set}\;(h);$

$J+1\;\text{-decomposition}\leftarrow\text{MODWT}(Y_{t});$

for $i$ in $J+1$ do

$train\leftarrow(J+1)$-$Levels[i][1:N];$

$test\leftarrow(J+1)$-$Levels[i][N+1:N+h];$

for each $epochs$ do

$embedIn\leftarrow\text{Embedding1}(train);$

$embedOut\leftarrow\text{Embedding2}(train);$

$encodOut\leftarrow\text{Encoder}(embedIn);$

$decod1Out\leftarrow\text{Decoder1}(embedOut);$

$decod2Out\leftarrow\text{Decoder2}(encodOut,decod1Out);$

$Model\leftarrow softmax(Linear(decod2Out));$

end for

$transformerResults[i]\leftarrow prediction(Model,test);$

end for

$\hat{Y}_{N+h}\leftarrow$InverseMODWT$(TransformerResult);$

A detailed explanation of the W-Transformers is given in Algorithm 1 and a
schematic diagram is also presented in Fig. 2.

Figure 1: MODWT decomposition of the website traffic dataset. The first curve
represents the original time series; the last represents the smooth series,
i.e., the scaling coefficient of the time series after computing the MODWT
algorithm. The other seven curves (in between) represent the different level
coefficient calculated by the MODWT algorithm. The filter used for this
decomposition is the Haar filter. Figure 2: Diagram representing the proposed
W-Transformers architecture; $P_{i}\;(i:0,...,j)$ represent the prediction.

##  IV Experimental Analysis

In this section, we present the used datasets, the performance measures, the
detailed description of the implementation scheme of the proposed
W-Transformers, and the baseline methods for comparisons.

###  IV-A Data

We considered seven publicly available datasets (see Table I) from various
applied fields for our experiments. For example, one dataset is from the stock
market, three are from epidemiological fields, two are related to internet
traffic, and the last one is about the sunspot data.

  * •

Netflix: This dataset is the Netflix closing stock price from June 2021 to
June 2022. The data are coming from Yahoo Finance and are daily data.

  * •

Website Traffic: This dataset is the number of first-time visitors to a
website. This dataset is publicly available on Kaggle and contains five years
of daily time points.

  * •

Sunspot: This standard time series dataset contains daily total sunspot number
derived by the formula: $R=Ns+10Ng$, with $Ns$ the number of spots and $Ng$
the number of groups counted over the entire solar disk [25].

  * •

Colombia Dengue: This dataset represents the weekly number of infected people
with dengue in Colombia [40].

  * •

Japan Flu: This dataset reports weekly influenza-like cases in Japan [4].

  * •

Bangkok Dengue: This dataset is the monthly number of infected people with
Dengue in Bangkok [40].

  * •

Network Analytics: This dataset represents the percentage of network outbound
utilization every 5-min and is available on Kaggle.

On these seven univariate datasets, we applied KPSS Test for stationarity,
Terasvirta test for linearity, and Hurst exponent check for long-term
dependencies [41, 1]. Results of these tests on different characteristics are
shown in Table I. Other than the Japan Flu dataset, all other datasets exhibit
strong nonstationarity and long-term dependency. Only NFLX and Sunspot
datasets show linear structure in the time series.

TABLE I: Data Characteristics of seven time series datasets Data Name | Frequency | Observations | Test Size | Min-Max Value | Behaviour
---|---|---|---|---|---
NFLX | Daily | 254 | 30/90 | 166.37 - 691.69 | Non-stationary, Linear, Long-term dependant
Website Traffic | Daily | 2167 | 30/90 | 2 - 5541 | Non-stationary, Non-linear, Long-term dependant
Sunspot | Daily | 2663 | 30/90 | 0 - 172 | Non-stationary, Linear, Long-term dependant
Colombia Dengue | Weekly | 626 | 26/52 | 164 - 3354 | Non-stationary, Non-linear, Long-term dependant
Japan Flu | Weekly | 964 | 26/52 | 0 - 1656 | Stationary, Non-linear, Long-term dependant
Bangkok Dengue | Monthly | 180 | 12/24 | 29 - 8337 | Non-stationary, Non-linear, Long-term dependant
Network Analytics | 5-min | 25631 | 576/1152 | 4.82e-05 - 99.9955 | Non-stationary, Non-linear, Long-term dependant

###  IV-B Performance Measures

To comprehensive analysis and evaluation of different methods, we considered
four commonly used performance metrics including Root Mean Squared Error
(RMSE), Mean Absolute Error (MAE), Symmetric Mean Absolute Percentage Error
(sMAPE), and Mean Absolute Scaled Error (MASE) [1]. The lower values of these
measures indicate better forecasting results.

###  IV-C Protocol

To perform MODWT and inverse MODWT, we used the python package ’Pywavelet’
[42]. To use MODWT for time series analysis, we carefully chose the number of
levels for our decomposition to get the optimal performance in the wavelet
domain as discussed in Sec. III. Once our proposed model is implemented, we
computed the different metrics for all the baseline models for different
forecasting horizons on each dataset. The different forecasting horizons are
defined as short-term forecasting and long-term forecasting, as shown in the
test size column in Table I. The implementation of Transformers on the
decomposed datasets is done using ‘darts’ [43] Python implementation tools by
setting the number of epochs to be 200 for all the series. We tested several
configurations for the input/output length, and regarding the result, we
normalized the value to 12/1 for every dataset. The number of heads $m$ is set
to $8$ and the dimension of the model, $d$, is 16, and the batch size is set
to 32. We have used two encoder layers as well as two decoder layers. The
dropout is set to 0.1, and the activation used in the W-Transformers model is
‘relu’ [26]. Our proposed model was trained on GPU (graphics processing unit)
but can also be computed using a CPU (central processing unit).

###  IV-D Results

For a comprehensive comparison, we computed the results of our proposed
W-Transformers and ten baseline models on seven datasets with two different
forecast horizons. The experimental results for short-term and long-term
forecasting are shown in Table II and Table III, respectively. The experiments
show that the performance of the models mainly depends on the length of the
forecast horizons. W-Transformers is, in fact, more efficient for long-term
forecasting. This comes from the MODWT decomposition; wavelets can handle
nonstationarity and Transformers work mostly well for long-term forecasting.
To demonstrate the statistical robustness of the obtained results, we applied
multiple comparisons with the best (MCB) test by combining all the results
based on four performance metrics and reporting the results in Figures 4 and
5. The overall performance of our W-Transformers is superior compared to all
deep learning models and statistical forecasters considered in this study. For
small sample-sized datasets, the performance of statistical methods is
comparatively better than that of W-Transformers. However, the proposal
performed better than all the baseline models for large temporal datasets (for
example, the Network Analytics dataset, etc.). All the data and codes are
available at https://github.com/CapWidow/W-Transformer for public use to
support the principle of reproducible research.

TABLE II: Short-Term Comparison of proposed W-Transformers (W-Trans.) with statistical, machine learning, and deep learning models in terms of RMSE, MAE, sMAPE, and MASE ((Best results in Bold and second best in _italic_). Data | Metrics | Auto | WARIMA | ETS | SETAR | LightGBM | ARNN | RNN | TCN | DeepAR | Transformer | W-Trans.
---|---|---|---|---|---|---|---|---|---|---|---|---
|  |  ARIMA [44] | [36] | [2] | [45] | [46] | [13] | [47] | [48] | [14] | [21] | (Proposed)
NFLX | RMSE | 89.06 | 87.70 | 13.09 | 116.54 | 81.31 | 153.90 | 172.62 | 90.02 | 149.27 | 203.54 | _45.71_
| MAE | 75.12 | 77.15 | 10.21 | 95.24 | 79.75 | 148.82 | 172.34 | 89.24 | 148.95 | 202.66 | _40.00_
Stock | sMAPE | 54.41 | 54.77 | 5.37 | 78.63 | 34.77 | 136.82 | 166.41 | 38.13 | 129.14 | 198.97 | _24.26_
| MASE | 8.32 | 12.00 | 1.59 | 14.81 | 8.84 | 23.15 | 19.10 | 9.89 | 16.51 | 22.46 | _4.43_
Website | RMSE | 812.65 | 1110.80 | 849.02 | _786.99_ | 899.36 | 989.77 | 2227.65 | 869.51 | 1628.14 | 1584.98 | 728.70
| MAE | 435.80 | 762.14 | 579.08 | 633.82 | 786.15 | 877.43 | 2080.15 | 542.88 | 1536.25 | 1257.2 | _473.82_
Traffic | sMAPE | 25.32 | 36.05 | 31.11 | 35.05 | 45.36 | 55.25 | 158.91 | 30.95 | 98.13 | 79.10 | _27.35_
| MASE | 0.48 | 0.97 | 0.73 | 0.80 | 0.87 | 1.11 | 2.29 | 0.60 | 1.69 | 1.38 | _0.52_
Sunspot | RMSE | 51.97 | 50.72 | 73.25 | 85.53 | 84.32 | _39.22_ | 82.35 | 4448.45 | 79.30 | 59.37 | 34.05
| MAE | 42.46 | 40.74 | 62.87 | 76.05 | 75.86 | _33.55_ | 70.90 | 2832.47 | 70.61 | 48.72 | 30.82
| sMAPE | 47.79 | 45.37 | 84.16 | 116.78 | 119.15 | _37.80_ | 104.36 | 179.82 | 111.39 | 57.52 | 37.23
| MASE | 7.13 | _2.91_ | 4.49 | 5.43 | 12.73 | 2.40 | 11.90 | 475.42 | 11.85 | 8.18 | 5.17
Colombia | RMSE | 949.82 | 840.50 | 917.22 | 809.30 | 702.48 | 1140.47 | 1263.69 | 1283.29 | 1153.45 | _635.18_ | 634.41
| MAE | 877.08 | 793.21 | 843.16 | 741.10 | 642.27 | 1058.74 | 1207.63 | 1224.61 | 1091.73 | 537.16 | _541.77_
Dengue | sMAPE | 54.79 | 51.26 | 53.34 | _48.81_ | 44.10 | 61.92 | 184.26 | 183.90 | 150.59 | 50.21 | 51.03
| MASE | 9.07 | 8.20 | 8.72 | 7.66 | 7.40 | 10.95 | 13.92 | 14.12 | 12.59 | 6.19 | _6.25_
Japan | RMSE | 114.51 | 114.87 | 6.84 | 239.40 | 29.78 | 283.89 | _10.71_ | 11.09 | 151.34 | 20.58 | 94.14
| MAE | 108.39 | 103.27 | 5.69 | 210.15 | 13.43 | 228.04 | 8.06 | _7.53_ | 114.48 | 18.93 | 90.51
Flu | sMAPE | 171.98 | 167.0 | _73.38_ | 176.98 | 66.38 | 163.75 | 169.00 | 90.78 | 157.19 | 189.51 | 170.81
| MASE | 2.12 | 18.18 | 1.00 | 37.00 | 0.26 | 40.15 | _0.16_ | 0.15 | 2.24 | 0.37 | 1.77
Bangkok | RMSE | 493.09 | 779.49 | 424.68 | 415.42 | _424.15_ | 518.08 | 886.48 | 998.86 | 841.74 | 827.06 | 815.52
| MAE | 441.83 | 743.68 | _362.42_ | 375.64 | 359.32 | 457.00 | 754.74 | 917.99 | 701.68 | 684.24 | 681.20
Dengue | sMAPE | 65.43 | 76.93 | _53.95_ | 56.23 | 53.13 | 67.98 | 188.83 | 189.04 | 152.32 | 142.41 | 150.26
| MASE | 1.63 | 3.26 | _1.59_ | 1.65 | 1.33 | 2.00 | 2.79 | 3.39 | 2.59 | 2.53 | 2.52
Network | RMSE | 25.8 | _25.08_ | 52.43 | 44.63 | 26.76 | 32.57 | 31.68 |  1.28$E^{95}$ | 53.70 | 37.58 | 12.09
| MAE | 23.02 | 19.69 | 48.49 | 41.10 | _18.13_ | 29.73 | 28.98 |  7.76$E^{93}$ | 49.73 | 34.39 | 10.91
Analytics | sMAPE | 88.35 | 83.29 | 114.08 | 108.25 | _77.20_ | 97.14 | 96.13 | 199.76 | 114.96 | 101.79 | 60.09
| MASE | 3.94 | 3.43 | 8.45 | 7.16 | _3.08_ | 5.18 | 8.85 |  1.32$E^{9}3$ | 8.45 | 5.98 | 1.72
TABLE III: Long-Term Comparison of proposed W-Transformers (W-Trans.) with statistical, machine learning, and deep learning models in terms of RMSE, MAE, sMAPE, and MASE ((Best results in Bold and second best in _italic_). Data | Metrics | Auto | WARIMA | ETS | SETAR | LightGBM | ARNN | RNN | TCN | DeepAR | Transformer | W-Trans.
---|---|---|---|---|---|---|---|---|---|---|---|---
|  |  ARIMA [44] | [36] | [2] | [45] | [46] | [13] | [47] | [48] | [14] | [21] | (Proposed)
NFLX | RMSE | 269.52 | 696.59 | 276.92 | 270.18 | _210.78_ | 160.85 | 311.52 | 254.39 | 300.90 | 328.69 | 237.67
| MAE | 245.91 | 615.44 | 252.77 | 247.12 | _189.15_ | 125.40 | 297.47 | 241.01 | 286.36 | 312.38 | 219.75
Stock | sMAPE | 127.19 | 165.81 | 128.90 | 60.33 | _50.70_ | 37.80 | 187.98 | 149.49 | 173.50 | 198.81 | 107.29
| MASE | 30.93 | 62.97 | 25.86 | 25.28 | _23.79_ | 12.83 | 37.41 | 30.31 | 36.02 | 39.29 | 27.64
Website | RMSE | _959.73_ | 1281.64 | 1192.66 | 1082.51 | 1399.02 | 1356.29 | 2593.36 | 1166.51 | 2010.79 | 2638.05 | 847.41
| MAE | 624.22 | 975.38 | 864.14 | 921.82 | 1261.93 | 1065.48 | 2413.45 | 782.65 | 1875.34 | 2470.93 | _634.74_
Traffic | sMAPE | 29.49 | 39.48 | 36.31 | 43.89 | 64.28 | 41.23 | 164.07 | 33.91 | 107.14 | 180.14 | _31.02_
| MASE | 0.69 | 1.10 | 0.98 | 1.04 | 1.39 | 1.21 | 2.66 | 0.86 | 2.07 | 2.73 | _0.70_
Sunspot | RMSE | 44.61 | 41.48 | _37.46_ | 57.06 | 66.99 | 71.83 | 74.16 | 41900.15 | 52.50 | 40.63 | 30.07
| MAE | 34.93 | 33.05 | _30.72_ | 45.67 | 53.09 | 56.93 | 63.75 | 15410.47 | 41.78 | 32.36 | 22.63
| sMAPE | 44.15 | 41.48 | _38.21_ | 62.91 | 84.14 | 97.60 | 108.69 | 166.08 | 65.21 | 40.40 | 30.09
| MASE | 5.98 | _2.80_ | 2.60 | 3.87 | 9.08 | 4.82 | 10.91 | 2636.93 | 7.15 | 5.54 | 3.87
Colombia | RMSE | 998.32 | 863.05 | 997.25 | 707.28 | 1183.99 | _839.24_ | 2114.89 | 2126.41 | 2016.42 | 1549.02 | 1452.04
| MAE | 802.61 | 736.40 | 802.10 | 640.13 | 946.09 | _683.25_ | 1949.85 | 1966.53 | 1842.58 | 1324.07 | 1332.04
Dengue | sMAPE | 40.99 | 38.92 | 40.97 | 35.15 | 45.43 | _36.62_ | 188.75 | 193.33 | 165.68 | 88.37 | 98.81
| MASE | 4.86 | 4.46 | 4.86 | 3.88 | 11.83 | _4.14_ | 24.38 | 24.59 | 23.04 | 16.55 | 16.65
Japan | RMSE | _164.41_ | 196.65 | 186.15 | 297.30 | 192.54 | 239.31 | 171.51 | 194.94 | 179.61 | 326.55 | 76.21
| MAE | 145.25 | 174.17 | 171.63 | 281.93 | 126.51 | 199.93 | 114.01 | _106.89_ | 163.67 | 276.56 | 58.98
Flu | sMAPE | 130.81 | 136.76 | 134.94 | 142.31 | _106.52_ | 126.77 | 130.00 | 131.07 | 133.18 | 131.81 | 103.19
| MASE | 2.89 | 4.83 | 3.95 | 6.49 | 2.52 | 4.60 | 2.27 | _2.13_ | 3.26 | 5.51 | 1.17
Bangkok | RMSE | 4152.04 | 1889.92 | 3454.05 | 2153.80 | 1492.16 | 819.90 | 824.70 | 1032.96 | 786.21 | _767.52_ | 735.00
| MAE | 4131.17 | 1756.66 | 3423.33 | 1486.24 | 1418.78 | 678.36 | 681.73 | 877.07 | 634.59 | _611.18_ | 608.30
Dengue | sMAPE | 152.16 | 119.20 | 145.50 | 114.83 | _108.60_ | 76.91 | 187.26 | 157.79 | 151.00 | 136.43 | 154.62
| MASE | 15.51 | 7.57 | 14.75 | 6.40 | 5.33 | 2.92 | 2.56 | 3.29 | 2.38 | _2.29_ | 2.28
Network | RMSE | 27.26 | 43.94 | 23.65 | 40.58 | 28.56 | 24.71 | 43.00 | $\infty$ | _22.51_ | 29.21 | 19.00
| MAE | 24.08 | 39.06 | _18.31_ | 35.97 | 25.19 | 21.99 | 37.98 |  9.66$E^{182}$ | 19.09 | 25.80 | 15.96
Analytics | sMAPE | 78.28 | 94.56 | _70.46_ | 91.69 | 80.02 | 75.80 | 93.34 | 199.80 | 71.52 | 80.64 | 60.31
| MASE | 4.10 | 6.49 | _3.04_ | 5.97 | 4.29 | 3.66 | 6.46 |  1.64$E^{182}$ | 3.25 | 4.39 | 2.71
Figure 3: The predicted curve of W-Transformers (red), ETS (green), and ARNN
(blue) were superposed over the test data (violet) for long-term forecasting.
Figure 4: Average model ranking regarding the four evaluation metrics for
short-term forecasting using multiple comparisons with best(MCB). The number
after the model name represents the average ranking. Figure 5: Average model
ranking regarding the four evaluation metrics for long-term forecasting using
multiple comparisons with best (MCB). The number after the model name
represents the average ranking.

##  V Conclusions and Discussions

We proposed a novel approach called W-Transformers for time series
forecasting. Our model uses wavelet decomposition within the Transformers
framework. The proposed model achieves much better results on average than
several existing models. Moreover, our proposed approach consistently
outperforms all conventional Transformers-like methods on the considered
datasets. It is worth noting, nevertheless, that our proposed approach works
better with large-size datasets. Statistical models can easily outperform deep
learning-based approaches with too small data sizes. Another important finding
from W-Transformer compared with the benchmarks is that the proposal is
superior for long-term forecasting. In future work, extending the proposed
method to work better with small data sizes will be of interest. Improving the
results for short-term forecasting is also a key challenge and can be
considered a future scope of the study. Applying the proposed W-Transformers
approach to multivariate time series is under investigation.

## Acknowledgment

The support of TotalEnergies is fully acknowledged. Lena Sasal (PhD Student)
and Abdenour Hadid (Professor, Industry Chair at SCAI Center of Abu Dhabi) are
funded by TotalEnergies collaboration agreement with Sorbonne University Abu
Dhabi.

## References

  * [1] R. J. Hyndman and G. Athanasopoulos, _Forecasting: principles and practice_. OTexts, 2018.
  * [2] R. Hyndman, A. B. Koehler, J. K. Ord, and R. D. Snyder, _Forecasting with exponential smoothing: the state space approach_. Springer Science & Business Media, 2008.
  * [3] T. Chakraborty and I. Ghosh, “Real-time forecasts and risk assessment of novel coronavirus (covid-19) cases: A data-driven analysis,” _Chaos, Solitons & Fractals_, vol. 135, p. 109850, 2020.
  * [4] M. Panja, T. Chakraborty, U. Kumar, and N. Liu, “Epicasting: An ensemble wavelet neural network (ewnet) for forecasting epidemics,” _arXiv preprint arXiv:2206.10696_ , 2022.
  * [5] S. Mehtab and J. Sen, “Time series analysis-based stock price prediction framework using machine learning and deep learning models,” _Dept. Data Sci. and Artif. Intell._ , 2020.
  * [6] X. Ren, X. Li, K. Ren, J. Song, Z. Xu, K. Deng, and X. Wang, “Deep learning-based weather prediction: a survey,” _Big Data Research_ , vol. 23, p. 100178, 2021.
  * [7] R. Casado-Vara, A. Martin del Rey, D. Pérez-Palau, L. de-la Fuente-Valentín, and J. M. Corchado, “Web traffic time series forecasting using lstm neural networks with distributed asynchronous training,” _Mathematics_ , vol. 9, no. 4, p. 421, 2021.
  * [8] R. Vautard, M. Beekmann, J. Roux, and D. Gombert, “Validation of a hybrid forecasting system for the ozone concentrations over the paris area,” _Atmospheric Environment_ , vol. 35, no. 14, pp. 2449–2461, 2001.
  * [9] Y. Qin, K. Li, Z. Liang, B. Lee, F. Zhang, Y. Gu, L. Zhang, F. Wu, and D. Rodriguez, “Hybrid forecasting model based on long short term memory network and deep learning neural network for wind signal,” _Applied energy_ , vol. 236, pp. 262–272, 2019.
  * [10] G. E. Box, G. M. Jenkins, G. C. Reinsel, and G. M. Ljung, _Time series analysis: forecasting and control_. John Wiley & Sons, 1970.
  * [11] H. Tong and K. S. Lim, “Threshold autoregression, limit cycles and cyclical data,” in _Exploration Of A Nonlinear World: An Appreciation of Howell Tong’s Contributions to Statistics_. World Scientific, 2009, pp. 9–56.
  * [12] G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu, “Lightgbm: A highly efficient gradient boosting decision tree,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [13] J. Faraway and C. Chatfield, “Time series forecasting with neural networks: a comparative study using the air line data,” _Journal of the Royal Statistical Society: Series C (Applied Statistics)_ , vol. 47, no. 2, pp. 231–250, 1998.
  * [14] D. Salinas, V. Flunkert, J. Gasthaus, and T. Januschowski, “Deepar: Probabilistic forecasting with autoregressive recurrent networks,” _International Journal of Forecasting_ , vol. 36, no. 3, pp. 1181–1191, 2020\.
  * [15] S. Bai, J. Z. Kolter, and V. Koltun, “An empirical evaluation of generic convolutional and recurrent networks for sequence modeling,” _arXiv preprint arXiv:1803.01271_ , 2018.
  * [16] F. Petropoulos, D. Apiletti, V. Assimakopoulos, M. Z. Babai, D. K. Barrow, S. B. Taieb, C. Bergmeir, R. J. Bessa, J. Bijak, J. E. Boylan _et al._ , “Forecasting: theory and practice,” _International Journal of Forecasting_ , 2022.
  * [17] B. Lim and S. Zohren, “Time-series forecasting with deep learning: a survey,” _Philosophical Transactions of the Royal Society A_ , vol. 379, no. 2194, p. 20200209, 2021.
  * [18] H. Hewamalage, C. Bergmeir, and K. Bandara, “Recurrent neural networks for time series forecasting: Current status and future directions,” _International Journal of Forecasting_ , vol. 37, no. 1, pp. 388–427, 2021\.
  * [19] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural Computation_ , vol. 9, no. 8, p. 1735–1780, nov 1997.
  * [20] K. Cho, B. van Merrienboer, D. Bahdanau, and Y. Bengio, “On the properties of neural machine translation: Encoder-decoder approaches,” _CoRR_ , vol. abs/1409.1259, 2014.
  * [21] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” _CoRR_ , vol. abs/1706.03762, 2017.
  * [22] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz _et al._ , “Transformers: State-of-the-art natural language processing,” in _Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations_ , 2020, pp. 38–45.
  * [23] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” _arXiv preprint arXiv:2010.11929_ , 2020.
  * [24] Q. Wen, T. Zhou, C. Zhang, W. Chen, Z. Ma, J. Yan, and L. Sun, “Transformers in time series: A survey,” _arXiv preprint arXiv:2202.07125_ , 2022.
  * [25] R. Godahewa, C. Bergmeir, G. I. Webb, R. J. Hyndman, and P. Montero-Manso, “Monash time series forecasting archive,” _arXiv preprint arXiv:2105.06643_ , 2021.
  * [26] G. Zerveas, S. Jayaraman, D. Patel, A. Bhamidipaty, and C. Eickhoff, “A transformer-based framework for multivariate time series representation learning,” in _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, 2021, pp. 2114–2124.
  * [27] B. Tang and D. S. Matteson, “Probabilistic transformer for time series analysis,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 23 592–23 608, 2021.
  * [28] N. Wu, B. Green, X. Ben, and S. O’Banion, “Deep transformer models for time series forecasting: The influenza prevalence case,” _arXiv preprint arXiv:2001.08317_ , 2020.
  * [29] S. Feng, K. Xu, J. Wu, P. Wu, F. Lin, and P. Zhao, “Multi-scale attention flow for probabilistic time series forecasting,” _arXiv preprint arXiv:2205.07493_ , 2022.
  * [30] I. N. Sneddon, _Fourier transforms_. Courier Corporation, 1995.
  * [31] W. T. Cochran, J. W. Cooley, D. L. Favin, H. D. Helms, R. A. Kaenel, W. W. Lang, G. C. Maling, D. E. Nelson, C. M. Rader, and P. D. Welch, “What is the fast fourier transform?” _Proceedings of the IEEE_ , vol. 55, no. 10, pp. 1664–1674, 1967.
  * [32] D. B. Percival and A. T. Walden, _Wavelet methods for time series analysis_. Cambridge university press, 2000, vol. 4.
  * [33] H. Liu, H.-q. Tian, D.-f. Pan, and Y.-f. Li, “Forecasting models for wind speed using wavelet, wavelet packet, time series and artificial neural networks,” _Applied Energy_ , vol. 107, pp. 191–208, 2013.
  * [34] D. Boto-Giralda, F. J. Díaz-Pernas, D. González-Ortega, J. F. Díez-Higuera, M. Antón-Rodríguez, M. Martínez-Zarzuela, and I. Torre-Díez, “Wavelet-based denoising for traffic volume time series forecasting with self-organizing neural networks,” _Computer-Aided Civil and Infrastructure Engineering_ , vol. 25, no. 7, pp. 530–545, 2010.
  * [35] D. B. Percival and H. O. Mofjeld, “Analysis of subtidal coastal sea level fluctuations using wavelets,” _Journal of the American Statistical Association_ , vol. 92, no. 439, pp. 868–880, 1997.
  * [36] M. Aminghafari and J.-M. Poggi, “Forecasting time series using wavelets,” _International Journal of Wavelets, Multiresolution and Information Processing_ , vol. 5, no. 05, pp. 709–724, 2007.
  * [37] Y.-F. Sang, “A review on the applications of wavelet transform in hydrology time series analysis,” _Atmospheric research_ , vol. 122, pp. 8–15, 2013\.
  * [38] A. Grinsted, J. C. Moore, and S. Jevrejeva, “Application of the cross wavelet transform and wavelet coherence to geophysical time series,” _Nonlinear processes in geophysics_ , vol. 11, no. 5/6, pp. 561–566, 2004.
  * [39] L. Zhu, Y. Wang, and Q. Fan, “Modwt-arma model for time series prediction,” _Applied Mathematical Modelling_ , vol. 38, no. 5-6, pp. 1859–1865, 2014\.
  * [40] T. Chakraborty, S. Chattopadhyay, and I. Ghosh, “Forecasting dengue epidemics using a hybrid methodology,” _Physica A: Statistical Mechanics and its Applications_ , vol. 527, p. 121266, 2019.
  * [41] T. Chakraborty, I. Ghosh, T. Mahajan, and T. Arora, “Nowcasting of covid-19 confirmed cases: Foundations, trends, and challenges,” _Modeling, Control and Drug Development for COVID-19 Outbreak Prevention_ , pp. 1023–1064, 2022.
  * [42] G. Lee, R. Gommers, F. Waselewski, K. Wohlfahrt, and A. O’Leary, “Pywavelets: A python package for wavelet analysis,” _Journal of Open Source Software_ , vol. 4, no. 36, p. 1237, 2019.
  * [43] J. Herzen, F. Lässig, S. G. Piazzetta, T. Neuer, L. Tafti, G. Raille, T. Van Pottelbergh, M. Pasieka, A. Skrodzki, N. Huguenin _et al._ , “Darts: User-friendly modern machine learning for time series,” _Journal of Machine Learning Research_ , vol. 23, no. 124, pp. 1–6, 2022\.
  * [44] R. J. Hyndman and Y. Khandakar, “Automatic time series forecasting: the forecast package for r,” _Journal of statistical software_ , vol. 27, pp. 1–22, 2008.
  * [45] H. Tong, _Non-linear time series: a dynamical system approach_. Oxford University Press, 1990.
  * [46] G. Ke, Q. Meng, T. Finley, T. Wang, W. Chen, W. Ma, Q. Ye, and T.-Y. Liu, “Lightgbm: A highly efficient gradient boosting decision tree,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [47] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [48] Y. Chen, Y. Kang, Y. Chen, and Z. Wang, “Probabilistic forecasting with temporal convolutional neural network,” _Neurocomputing_ , vol. 399, pp. 491–501, 2020.


# Mining multi-modal communication patterns in interaction with explainable
and non-explainable robots

Suna Bensch1, Amanda Eriksson1 1Department of Computing Science, Umeå
University, 901 87 Umeå, Sweden<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

We investigate interaction patterns for humans interacting with explainable
and non-explainable robots. Non-explainable robots are here robots that do not
explain their actions or non-actions, neither do they give any other feedback
during interaction, in contrast to explainable robots. We video recorded and
analyzed human behavior during a board game, where 20 humans verbally
instructed either an explainable or non-explainable Pepper robot to move
objects on the board. The transcriptions and annotations of the videos were
transformed into transactions for association rule mining. Association rules
discovered communication patterns in the interaction between the robots and
the humans, and the most interesting rules were also tested with regular chi-
square tests. Some statistically significant results are that there is a
strong correlation between men and non-explainable robots and women and
explainable robots, and that humans mirror some of the robot’s modality. Our
results also show that it is important to contextualize human interaction
patterns, and that this can be easily done using association rules as an
investigative tool. The presented results are important when designing robots
that should adapt their behavior to become understandable for the interacting
humans.

##  I INTRODUCTION

There is an increasing interest in explainability for AI in general [1]. In
robotics research, explainability goes also under several other names, for
example understandability [2], readability [3], anticipation [4], legibility
[5, 6], and predictability [7]. The specific focus also varies a lot and
includes how robots should be designed such that humans understand their
actions [8] or explanations [9], motions [10, 11, 12], emotions [13, 14],
goals [15], failures [16], and various types of social behavior [17, 18, 19].
The benefits with explainable robots has also been investigated in several
studies. In [20], explainability was shown to support situational awareness in
human-machine teams. In [21], robots acting without communicating their
intentions to involved humans created anxiety. In cooperative tasks, the
efficiency has been shown to be negatively affected if the human cannot
correctly predict the actions of the robot [4, 22]. Despite this acknowledged
importance of, and active research in, explainable robots, there is still
little knowledge about human communication patterns in interaction with
explainable and non-explainable robots. Such knowledge about communication
patterns or behavior is important when designing explainable robots that
should adapt their behavior in order to increase explainability for the
interacting human.

In this paper, we investigate human communication patterns when interacting
with explainable and non-explainable robots using association rule mining.

Our results show that humans mirrored the robot’s choice of modality, and that
there was a strong correlation between male users and non-explainable robots
and female users and explainable robots. Male user tended to engage verbally
and with gestures when interacting with non-explainable robots, whereas female
users engaged verbally and with gestures when interacting with explainable
robots. Typical examples for such interactions are shown in Table I.

TABLE I: Example dialogues for men reacting to non-understandable robots and women reaction to understandable robots. Agent | Utterance
---|---
Man: | Can you move the yellow triangle to square 11?
Robot: | No.
Man: | No? OK.
Woman | What about moving it to Cell 9?
Robot | I cannot reach Cell 9. You have to find another way.
Woman | Oho, right.

Note here that the variable _gender_ emerged as a significant variable through
the generated association rules (it is not a controlled independent variable).

We annotated and analyzed 20 videos (amounting to around 200 minutes) of
human-robot interactions, where the participants played a game with either an
explainable or non-explainable robot, instructing the robot to move certain
pieces on a virtual game board displayed on the chest screen of a Pepper
robot. The annotations were translated into 295  _transactions_ that were used
to generate association rules describing relations between several observed
variables (e.g. both human and robot verbal utterances, gestures, no
feedback). This approach allowed us to investigate multi-modal communication
patterns in interaction with explainable and non-explainable robots, and how
the observed variables co-varied, and possibly affected each other. The usage
of association rules as an investigative tool demonstrated the importance of
contextualizing human behavior and feedback. For example, we found that
regular statistical tests of general questions, such as _Is there a difference
in how humans behave when interacting with understandable versus non-
understandable robots?_ , lead to the conclusion that no difference can be
shown. In terms of association rules, the Lift for these rules was approx.
1.0, meaning that _explainability_ and _human behavior_ were independent
events. However, when more context was taken into account, which corresponds
to more items in the antecedent (explained in detail below), differences in
robot explainability and gender emerged. Such contextualization is important
for HRI research, in particular for design of explainable robots, that should
interpret human behavior as accurately as possible. For example, our results
indicate also that absence of human feedback not necessarily is a signal of
non-understanding.

##  II METHODS

###  II-A Experimental and robot setup

We target a scenario where a human plays a board game with a Pepper robot,
verbally instructing the robot to move pieces on the board with the goal to
move a certain piece to a specified cell (see Fig. 1).

Figure 1: To the left is the initial configuration of the game board, which
was displayed on the robot’s chest-mounted tablet. The game was won when the
blue circle was moved to Cell 4. Cells 9 and 3 were not reachable and the
triangles were not movable as illustrated to the right.

Some pieces on the board were not movable and some cells were not reachable,
however the human had no knowledge of which pieces or cells these were but had
to find out in interaction with the robot. The human was either interacting
with an explainable or non-explainable robot. An explainable robot, for
example, explained why it could not execute an invalid instruction given by
the human (e.g _Human: Move the green square to Cell 3. Robot: I cannot reach
Cell 3_), whereas a non-explainable robot did not execute an invalid
instruction without giving the human an explanation or gesture feedback.

Twenty users, thereof seven identifying as female and thirteen identifying as
male, interacted with either an explainable or non-explainable robot. The age
range was between 20 and 45, sixteen of the users were students studying
computer science, interaction design, physics, or engineering. The goal for
each user was to verbally guide the robot to move the blue circle from Cell 1
to Cell 4 (see Fig. 1) by giving the robot a sequence of instructions on
objects to move. The users were informed that some of the objects were not
movable and some cells were not reachable for the robot and they had to find
out which objects and cells these were by interacting with the robot and
exploring possible paths to reach the goal.

The robot was remote-controlled by a Wizard who sat in an adjacent but
separate room using the WoZ4U interface [23]. The behavior of the explainable
robot included explaining when and why it could not execute the human’s
instruction, for example, _I cannot reach Cell 3, you have to find another
way_ or _I cannot move the triangle_. The behavior of the non-explainable
robot was less verbal or expressive in gestures, even though the robot
sometimes uttered _yes_ or _no_ , and nodded or shook its head to signal, for
example, invalid instructions or auditory problems. In most cases, the role of
the non-explainable robot was to not react, to give no feedback at all. The
robot would, for example, often do nothing at all when invalid instructions
were given by the human, or when the robot (the Wizard) had auditory problems
and could not hear what the participant said.

###  II-B Video annotation and analysis

Each participant interacted with the robot for, in average, around 10 minutes.
The interactions with the understandable robot were in average 9.7 minutes
long, whereas the interactions with the non-understandable robot were in
average 11.6 minutes. The verbal and non-verbal interactions between the
humans and the robot were manually transcribed and annotated with the video
annotation software ELAN [24] by one annotator (see Figure 2).

Figure 2: Sample excerpt of a transcription and initial annotation in ELAN of
an interaction between a non-explainable robot and a user.

In addition, we defined the notion of _events_. An event starts with a
participant’s instruction and ends just before the participant’s next
instruction begins. An event consists of all human and robot verbal utterances
and behaviors (i.e. gestures or no reaction), information on whether the human
instruction was valid or not, and whether the instruction was executed or
not).

###  II-C Association rules and dataset

Association rule mining introduced in [25] is today an umbrella term for
algorithms that identify associations for co-occurring instances in a dataset.
The terminology necessary for this paper is described in the following [25].

Let ${\cal{I}}=\\{I_{1},I_{2},\ldots,I_{n}\\}$ be a set of binary attributes
called _items_. Let $T=\\{t_{1},t_{2},\ldots,t_{m}\\}$ be a database of
transactions. Each transaction $t$ is represented as a binary vector where
each element is true or false depending on whether the corresponding item
belongs to the transaction or not. An association rule is written as:
$\alpha\to\beta$ where $\alpha,\beta\subseteq{\cal{I}}$ and $\alpha\neq\beta$.
Both $\alpha$ and $\beta$ are referred to as item sets. We refer to the left
hand side $\alpha$ as _antecedent_ and to the right hand side $\beta$ as
_consequent_. The interpretation of an association rule is “the observation of
the antecedent $\alpha$ is correlated with the observation of the consequent
$\beta$”.

Given a dataset of transactions $T$, association rules are generated
satisfying certain conditions:

####  II-C1 Syntactic constraints

are specifications of the exact items that should appear in the antecedent and
consequent.

####  II-C2 Support

is the number of transactions containing $\alpha$ and $\beta$, divided by the
total number of transactions in $T$. Support is a measure of how frequently a
rule’s item sets occur in a data set.

####  II-C3 Confidence

is defined as the support of $\alpha$ and $\beta$ divided by the support of
$\alpha$. Confidence is a measure of how _strong_ a rule is, in the sense how
strongly the occurrence of the antecedent is correlated with the the
occurrence of the consequent.

####  II-C4 Lift

is defined as the support of union of item sets in the antecedent and
consequent divided by support of the antecedent times the support of the
consequent [26]. If Lift equals 1, the antecedent and consequent are
independent. If Lift is $>1$, there is a positive correlation between the
antecedent and consequent and they appear more often together than they would
if they were independent. If Lift is $<1$, there is a negative correlation
between the antecedent and consequent and they appear less often together than
they would if they were independent. Hence, while Confidence for a rule may be
high simply because the consequent in general is very common in data, Lift
compensates for this.

All events were transformed into 295 _transactions_ for association rule
mining as shown in Figure 3 and Table II shows item names and the possible
values, where T stands for TRUE and F for FALSE.

Figure 3: Excerpt from the in total 295 transactions. Each row is a compressed representation of a transaction. For compact representation, the items are not only binary but take discrete values. TABLE II: Items used for generation of association rules Item name | Values
---|---
Execution | {T, F}
Instruction | {T, F}
RobotFeedback | {verbal, gesture, none}
HumanFeedback | {VERBAL, GESTURE, NONE}
Understandable | {T, F}
Female | {T, F}

To each transaction we added the information whether the robot was explainable
or not (i.e. Understandable=T or F) and whether the user was female or male
(i.e. Female=T or F) (see Fig. 3). Thus, a transaction contains information on
whether or not an instruction was valid, whether or not the instruction was
executed, the type of robot feedback, the type of human feedback, whether or
not the robot was understandable, and whether or not the participant
identified as female. The data from one male participant, interacting with a
non-understandable robot, was excluded since they decided to leave the
experiment. The 295 transactions were input to the Orange association rules
mining tool [27], which implements FP-growth frequent pattern mining algorithm
[28] with bucket optimization [29]. The minimum Support was set to 3%, the
minimum Minimum confidence level was set to 20%, and the minimum Lift was set
to $>1.0$. These settings were a trade-off between the number of transactions
(monitored through the Support value) for each generated rule, and the total
number of interesting generated rules (monitored through the Lift parameter).

Furthermore, we controlled the generation of association rules by specifying
syntactic constraints for the antecedent and consequent, i.e. which items to
include in the rules. The consequent was constrained to be
HumanFeedback=VERBAL, HumanFeedback=GESTURE, or HumanFeedback=NONE. By not
specifying any items in the antecedent, all items were allowed.

##  III RESULTS

In this section we present some of the most interesting rules, that are both
statistically significant and interpretable. The rules are described in
Figures 4-7, showing the attributes in the antecedent as well as the
consequent side. The Support, Confidence, and Lift values are also listed. To
assess the statistical significance level of the dependence between antecedent
and consequent [30], p-values for conducted chi-square tests are also provided
(for rules with $p<0.015$). Figure 4 shows two rules indicating gender
effects. The first rule, with Lift 2.846, indicates that male users who
interacted with a non-explainable robot that gave verbal feedback, provided
verbal feedback (rather than other types of feedback). The second rule, with
Lift 1.941, indicates that female users who interacted with an explainable
robot that gave verbal feedback, provided verbal feedback.

Figure 4: Generated association rules with Lift $\geq 1.9$, indicating a
gender effect with strong correlation between male users and non-explainable
robots and female users and explainable robots, both using verbal utterances.

The rules in Figure 5 indicate other gender effects, for cases where the robot
did not execute an instruction given by the user. Here, female users who
interacted with explainable robots, as well as male users interacting with
non-understandable robots, tended to give gesture feedback.

Figure 5: Generated association rules with Lift $\geq 1.9$, indicating a
gender effect when explainable and non-explainable robots do not execute a
human instruction.

Figure 6 shows association rules suggesting that humans mimic the robot’s
choice of modality. The non-shaded rules have Lift $<<1$. This means that
there is a negative correlation between the antecedent and consequent in the
transactions.

Figure 6: Generated association rules indicating that humans mimic the robot’s
modality. The third association rule with Lift 1.434 and Support 0.058 had no
statistical significance.

The association rules in Figure 7 have the consequent HumanFeedback=NONE,
which stands for no human feedback. The statistically significant second rule
shows that humans provide no feedback to an explainable robot that executes an
instruction. The first rule suggests that humans provide no feedback to a non-
explainable robot that does not execute an instruction and provides no
feedback itself.

Figure 7: The first rule, even though not statistically significant, suggests
that humans do not provide feedback when interacting with a non-explainable
robot that does not execute a human instruction and does not give any robot
feedback. The second rule shows that humans do not provide any feedback in
interaction with an explainable robot executing the human instruction.

Our results show several statistically significant multi-modal communication
patterns in humans’ interaction with explainable and non-explainable robots:

  1. 1.

Men react verbally to non-explainable verbal robots, whereas women react
verbally to explainable verbal robots.

  2. 2.

In cases where robots do not execute the human’s instruction, men provide
gesture feedback to gesturing non-explainable robots, whereas women provide
gesture feedback to explainable robots.

  3. 3.

Humans provide no feedback when interacting with an explainable robot that
executes an instruction.

  4. 4.

Humans give verbal feedback if the robot does the same, and give no feedback
if the robot does not give any feedback.

##  IV DISCUSSION

Figure 8 lists the number of transactions for male and female users
interacting with explainable and non-explainable robots. The number of users
is given in parenthesis. The anonymized data can be provided by the first
author upon request.

Figure 8: Number of transactions for male and female users interacting with
explainable and non-explainable robots. The number of participants $\\#$ is
given in parenthesis.

Association rule mining provides several advantages for data analysis in HRI.
One advantage is that interaction patterns can be investigated without
specific hypotheses being formulated beforehand. Another advantage is that
association rules allow for analysis of contextualized human behaviors by
means of implication rules if $\alpha$ then $\beta$ for which further
syntactic constraints can be given. For example, even if there were no
indications of general differences between explainable and non-explainable
robots and human behavior, statistically significant association rules could
be found when more contextual factors were taken into account.

## ACKNOWLEDGMENT

The authors thank Thomas Hellström for suggesting the chi-square tests to
assess the statistical significance of the rules.

## References

  * [1] D. Calvaresi, A. Najjar, M. Winikoff, and K. Främling, Eds., _Explainable and Transparent AI and Multi-Agent Systems - 4th International Workshop, EXTRAAMAS 2022, Virtual Event, May 9-10, 2022, Revised Selected Papers_ , ser. Lecture Notes in Computer Science, vol. 13283\. Springer, 2022. [Online]. Available: https://doi.org/10.1007/978-3-031-15565-9
  * [2] T. Hellström and S. Bensch, “Understandable robots-what, why, and how,” _Paladyn, Journal of Behavioral Robotics_ , vol. 9, no. 1, pp. 110–123, 2018\.
  * [3] L. Takayama, D. Dooley, and W. Ju, “Expressing thought: Improving robot readability with animation principles,” in _2011 6th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_ , March 2011, pp. 69–76.
  * [4] M. J. Gielniak and A. L. Thomaz, “Generating anticipation in robot motion,” in _2011 RO-MAN_ , July 2011, pp. 449–454.
  * [5] C. Lichtenthäler, “Legibility of robot behavior investigating legibility of robot navigation in human-robot path crossing scenarios,” Ph.D. dissertation, Technische Universität München, 2014.
  * [6] C. Lichtenthäler, T. Lorenzy, and A. Kirsch, “Influence of legibility on perceived safety in a virtual human-robot path crossing task,” in _2012 IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication_ , Sept 2012, pp. 676–681.
  * [7] K. Dautenhahn, S. Woods, C. Kaouri, M. Walters, K. L. Koay, and I. Werry, “What is a robot companion - friend, assistant or butler?” in _Proc. IEEE IRS/RSJ Int. Conference on Intelligent Robots and Systems_ , Edmonton, Alberta, Canada,, 2005, pp. 1488–1493.
  * [8] A. K. Singh, N. Baranwal, K.-F. Richter, T. Hellström, and S. Bensch, “Verbal explanations by collaborating robot teams,” _Paladyn, Journal of Behavioral Robotics_ , vol. 12, no. 1, pp. 47–57, 2021. [Online]. Available: https://doi.org/10.1515/pjbr-2021-0001
  * [9] Y. Mualla, I. Tchappi, T. Kampik, A. Najjar, D. Calvaresi, A. Abbas-Turki, S. Galland, and C. Nicolle, “The quest of parsimonious xai: A human-agent architecture for explanation formulation,” _Artificial Intelligence_ , vol. 302, p. 103573, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0004370221001247
  * [10] A. Zhou, D. Hadfield-Menell, A. Nagabandi, and A. D. Dragan, “Expressive robot motion timing,” in _Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction, HRI 2017, Vienna, Austria, March 6-9, 2017_ , 2017, pp. 22–31. [Online]. Available: http://dl.acm.org/citation.cfm?id=3020221
  * [11] A. D. Dragan, K. C. Lee, and S. S. Srinivasa, “Legibility and predictability of robot motion,” in _Proceedings of the 8th ACM/IEEE International Conference on Human-robot Interaction_ , ser. HRI ’13. Piscataway, NJ, USA: IEEE Press, 2013, pp. 301–308. [Online]. Available: http://dl.acm.org/citation.cfm?id=2447556.2447672
  * [12] R. Chadalavada, H. Andreasson, R. Krug, and A. J. Lilienthal, “That’s on my mind! robot to human intention communication through on-board projection on shared floor space,” in _Proceedings of European Conference on Mobile Robots_ , 2015.
  * [13] K. Kobayashi and S. Yamada, “Making a mobile robot to express its mind by motion overlap,” in _Advances in Human-Robot Interaction,_ , V. A. Kulyukin, Ed. InTech, 2009.
  * [14] B. Kühnlenz, S. Sosnowski, M. Buß, D. Wollherr, K. Kühnlenz, and M. Buss, “Increasing helpfulness towards a robot by emotional adaption to the user,” _Int J Soc Robot_ , vol. 5, no. 4, pp. 457–476, 2013.
  * [15] H. Karvonen and I. Aaltonen, “Intent communication of highly autonomous robots,” in _The 12th ACM/IEEE International Conference on Human-Robot Interaction (HRI 2017)_ , 2017.
  * [16] V. Raman and H. Kress-Gazit, “Explaining impossible high-level robot behaviors,” _IEEE Transactions on Robotics_ , vol. 29, no. 1, pp. 94–104, 2013.
  * [17] M. Matarese, F. Rea, and A. Sciutti, “A user-centred framework for explainable artificial intelligence in human-robot interaction,” _CoRR_ , vol. abs/2109.12912, 2021. [Online]. Available: https://arxiv.org/abs/2109.12912
  * [18] C. Breazeal, “Towards sociable robots,” _ROBOTICS AND AUTONOMOUS SYSTEMS_ , vol. 42, 2002.
  * [19] K. Dautenhahn, “The art of designing socially intelligent agents: Science, fiction, and the human in the loop,” _Applied Artificial Intelligence_ , vol. 12, no. 7-8, pp. 573–617, 1998. [Online]. Available: https://doi.org/10.1080/088395198117550
  * [20] R. Paleja, M. Ghuy, N. Ranawaka Arachchige, R. Jensen, and M. Gombolay, “The utility of explainable ai in ad hoc human-machine teaming,” in _Advances in Neural Information Processing Systems_ , M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 610–623. [Online]. Available: https://proceedings.neurips.cc/paper˙files/paper/2021/file/05d74c48b5b30514d8e9bd60320fc8f6-Paper.pdf
  * [21] T. Nomugaudra and K. Kawakami, “Relationships between robot’s self- disclosures and human’s anxiety toward robots,” in _Proceedings of the 2011 IEEE/WIC/ACM International Conferences on Web Intelligence and Intelligent Agent Technology_. IEEE Computer Society, 2011, pp. 66–69.
  * [22] G. Baud-Bovy, P. Morasso, F. Nori, G. Sandini, and A. Sciutti, “Human machine interaction and communication in cooperative actions,” in _Bioinspired Approaches for Human-Centric Technologies_ , S. I. Publishing, Ed., 2014, pp. 241–268.
  * [23] F. Rietz, A. Sutherland, S. Bensch, S. Wermter, and T. Hellström, “WoZ4U: An open-source wizard-of-oz interface for easy, efficient and robust hri experiments,” _Frontiers in Robotics and AI_ , vol. 8, 2021.
  * [24] P. Wittenburg, H. Brugman, A. Russel, A. Klassmann, and H. Sloetjes, “ELAN: a professional framework for multimodality research,,” in _5th International Conference on Language Resources and Evaluation (LREC 2006)_ , 2006, pp. 1556–1559.
  * [25] R. Agrawal, T. Imieliński, and A. Swami, “Mining association rules between sets of items in large databases,” _SIGMOD Rec._ , vol. 22, no. 2, p. 207–216, 1993. [Online]. Available: https://doi.org/10.1145/170036.170072
  * [26] P.-N. Tan, M. Steinbach, and V. Kumar, _Introduction to Data Mining_ , us ed. Addison Wesley, May 2005. [Online]. Available: http://www.amazon.com/exec/obidos/redirect?tag=citeulike07-20&path=ASIN/0321321367
  * [27] J. Demsar, T. Curk, A. Erjavec, C. Gorup, T. Hocevar, M. Milutinovic, M. Mozina, M. Polajnar, M. Toplak, A. Staric, M. Stajdohar, L. Umek, L. Zagar, J. Zbontar, M. Zitnik, and B. Zupan, “Orange: Data mining toolbox in python,” _Journal of Machine Learning Research_ , vol. 14, pp. 2349–2353, 2013.
  * [28] J. Han, J. Pei, Y. Yin, and R. Mao, “Mining frequent patterns without candidate generation: A frequent-pattern tree approach,” _Data Mining and Knowledge Discovery_ , vol. 8, pp. 53–87, 2004.
  * [29] C. R. Agrawal, C. C. Aggarwal, and V. V. V. Prasad, “Depth first generation of long patterns,” in _Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining (KDD ’00)_ , 2000, pp. 108–118.
  * [30] P. Baxter, J. Kennedy, E. Senft, S. Lemaignan, and T. Belpaeme, “From characterising three years of hri to methodology and reporting recommendations,” in _The Eleventh ACM/IEEE International Conference on Human Robot Interaction_ , ser. HRI ’16. IEEE Press, 2016, p. 391–398.