[a,b,c]Zhi-zhong Xing

# Towards identifying the minimal flavor symmetry behind neutrino oscillations

###### Abstract

Current neutrino oscillation data indicate that the $3\times 3$ Pontecorvo-
Maki-Nakagawa-Sakata matrix $U$ exhibits a $\mu$-$\tau$ flavor interchange
symmetry $|U_{\mu i}|=|U_{\tau i}|$ (for $i=1,2,3$) as a good approximation.
In particular, the T2K measurement implies that the maximal neutrino mixing
angle $\theta_{23}$ and the CP-violating phase $\delta$ should be close to
$\pi/4$ and $-\pi/2$, respectively. Behind these observations lies a minimal
flavor symmetry — the effective Majorana neutrino mass term keeps invariant
under the transformations $\nu_{e\rm L}\to(\nu_{e\rm L})^{c}$, $\nu_{\mu\rm
L}\to(\nu_{\tau\rm L})^{c}$, $\nu_{\tau\rm L}\to(\nu_{\mu\rm L})^{c}$.
Extending this flavor symmetry to the canonical seesaw mechanism, we find that
the $R$-matrix describing the strength of weak charged-current interactions of
heavy Majorana neutrinos satisfies $|R_{\mu i}|=|R_{\tau i}|$ as a consequence
of $|U_{\mu i}|=|U_{\tau i}|$. This result can be used to set a new upper
bound, which is about three orders of magnitude more stringent than before, on
the flavor mixing factor associated with the charged-lepton-flavor-violating
decay mode $\tau^{-}\to e^{-}+\gamma$.

##  1 Introduction

Current neutrino oscillation data [1] allow us to see two salient features of
the $3\times 3$ Pontecorvo-Maki-Nakagawa-Sakata (PMNS) lepton flavor mixing
matrix $U$. On the one hand, the observed pattern of $U$ exhibits a
$\mu$-$\tau$ interchange symmetry $|U_{\mu i}|=|U_{\tau i}|$ (for $i=1,2,3$)
as a very good approximation. On the other hand, the best-fit values of the
maximal neutrino mixing angle $\theta_{23}$ and the CP-violating phase
$\delta$ in the standard parametrization of $U$ are close respectively to
$\pi/4$ and $-\pi/2$, as illustrated in Fig. 1 according to the recent T2K
measurement [2].

Figure 1: An illustration of the recent T2K data on the octant of
$\theta_{23}$ and the quadrant of $\delta$ [2], together with their slight
deviations from the $\mu$-$\tau$ reflection symmetry predictions
$\theta_{23}=\pi/4$ and $\delta=-\pi/2$.

These two observations motivate us to conjecture that behind $U$ is most
likely a kind of non-Abelian discrete flavor symmetry group, whose Clebsch-
Gordan coefficients may help determine some elements of $U$ [3]. So far a lot
of flavor symmetries have been proposed and studied. At this stage one is
allowed to make use of Occam’s razor to shave away those complicated models
and focus on the minimal flavor symmetry in the neutrino sector. Such a
minimal flavor symmetry should belong to a residual flavor symmetry group
which is as simple as possible, and its predictions should be as close as
possible to the experimental data. We argue that the $\mu$-$\tau$ reflection
symmetry is the most appropriate example in this regard [4, 5, 6]: the
effective Majorana neutrino mass term

$\displaystyle-{\cal L}_{\rm mass}=\frac{1}{2}\hskip
1.42271pt\overline{\nu_{\rm L}}\hskip 1.42271ptM_{\nu}(\nu_{\rm L})^{c}+{\rm
h.c.}\;$ (1)

keeps invariant under the transformations $\nu_{e\rm L}\to(\nu_{e\rm L})^{c}$,
$\nu_{\mu\rm L}\to(\nu_{\tau\rm L})^{c}$ and $\nu_{\tau\rm L}\to(\nu_{\mu\rm
L})^{c}$. The flavor texture of symmetric $M_{\nu}$ can therefore be
constrained by $M_{\nu}={\cal P}M^{*}_{\nu}{\cal P}$, where

$\displaystyle{\cal P}={\cal P}^{T}={\cal
P}^{\dagger}=\left(\begin{matrix}1&0&0\cr 0&0&1\cr
0&1&0\cr\end{matrix}\right)\;.$ (2)

Then both $\theta_{23}=\pi/4$ and $\delta=\pm\pi/2$ can easily be achieved for
the PMNS matrix $U$ through the diagonalization
$U^{\dagger}M_{\nu}U^{*}=D_{\nu}\equiv{\rm Diag}\\{m_{1},m_{2},m_{3}\\}$ with
$m_{i}$ being the light neutrino masses.

Here we are going to identify the same $\mu$-$\tau$ reflection symmetry
associated with three species of active and sterile neutrinos in the canonical
seesaw mechanism by starting from the very possibility of $\big{|}U_{\mu
i}\big{|}=\big{|}U_{\tau i}\big{|}$ (for $i=1,2,3$) that is favored by current
neutrino oscillation data. Considering that $U$ is not exactly unitary and it
is intrinsically correlated with another $3\times 3$ flavor mixing matrix $R$
describing the strength of weak charged-current interactions of heavy Majorana
neutrinos, we show that $\big{|}U_{\mu i}\big{|}=\big{|}U_{\tau i}\big{|}$
gives rise to a novel prediction $\big{|}R_{\mu i}\big{|}=\big{|}R_{\tau
i}\big{|}$ (for $i=1,2,3$). We prove that behind these two sets of equalities
and the preliminary experimental evidence for leptonic CP violation [2] lies
an expected minimal flavor symmetry; namely, the overall neutrino mass term
keeps invariant when the left-handed neutrino fields transform as $\nu_{e\rm
L}\to(\nu_{e\rm L})^{c}$, $\nu_{\mu\rm L}\to(\nu_{\tau\rm L})^{c}$,
$\nu_{\tau\rm L}\to(\nu_{\mu\rm L})^{c}$ and the right-handed neutrino fields
undergo an arbitrary unitary transformation. We find that this novel result
can be used to set a new upper limit, which is about three orders of magnitude
more stringent than before, on the flavor mixing factor associated with the
charged-lepton-flavor-violating decay mode $\tau^{-}\to e^{-}+\gamma$.

##  2 The canonical seesaw mechanism

To explain why the masses of three active neutrinos are tiny, a minimal
extension of the standard model (SM) of particle physics is to add three
right-handed neutrino fields $N_{\alpha\rm R}$ (for $\alpha=e,\mu,\tau$) and
allow for lepton number violation. In this case the neutrino mass terms can be
written as

$\displaystyle-{\cal L}_{\nu}=\overline{\ell_{\rm L}}\hskip
1.42271ptY_{\nu}\widetilde{H}N_{\rm R}+\frac{1}{2}\hskip
1.42271pt\overline{(N_{\rm R})^{c}}\hskip 1.42271ptM_{\rm R}N_{\rm R}+{\rm
h.c.}\;,$ (3)

where $\ell_{\rm L}$ denotes the $\rm SU(2)_{\rm L}$ doublet of the left-
handed lepton fields, $\widetilde{H}\equiv{\rm i}\sigma_{2}H^{*}$ with $H$
being the Higgs doublet and $\sigma_{2}$ being the second Pauli matrix,
$N_{\rm R}=(N_{e\rm R},N_{\mu\rm R},N_{\tau\rm R})^{T}$ is the column vector
of three right-handed neutrino fields which are the $\rm SU(2)_{\rm L}$
singlets, $(N_{\rm R})^{c}\equiv{\cal C}\overline{N_{\rm R}}^{T}$ denotes the
charge conjugation of $N_{\rm R}$. After spontaneous gauge symmetry breaking,
Eq. (3) becomes

$\displaystyle-{\cal L}^{\prime}_{\nu}=\frac{1}{2}\hskip
1.42271pt\overline{\left[\nu_{\rm L}\hskip 5.69046pt(N_{\rm
R})^{c}\right]}\left(\begin{matrix}{\bf 0}&M_{\rm D}\cr M^{T}_{\rm D}&M_{\rm
R}\end{matrix}\right)\left[\begin{matrix}(\nu_{\rm L})^{c}\cr N_{\rm
R}\end{matrix}\right]+{\rm h.c.}\;,$ (4)

where $\nu_{\rm L}=(\nu_{e\rm L},\nu_{\mu\rm L},\nu_{\tau\rm L})^{T}$ denotes
the column vector of three left-handed neutrino fields, $M_{\rm D}\equiv
Y_{\nu}\langle H\rangle$ with $\langle H\rangle$ being the vacuum expectation
value of the Higgs field. The overall symmetric $6\times 6$ neutrino mass
matrix in Eq. (4) can be diagonalized by the unitary transformation

$\displaystyle\left(\begin{matrix}U&R\cr
S&Q\cr\end{matrix}\right)^{\dagger}\left(\begin{matrix}{\bf 0}&M_{\rm D}\cr
M^{T}_{\rm D}&M_{\rm R}\cr\end{matrix}\right)\left(\begin{matrix}U&R\cr
S&Q\cr\end{matrix}\right)^{*}=\left(\begin{matrix}D_{\nu}&{\bf 0}\cr{\bf
0}&D_{N}\cr\end{matrix}\right)\;,$ (5)

where $D_{N}\equiv{\rm Diag}\big{\\{}M_{1},M_{2},M_{3}\big{\\}}$ with $M_{i}$
being the heavy Majorana neutrino masses. The $3\times 3$ submatrices $U$,
$R$, $S$ and $Q$ satisfy the unitarity conditions:

$\displaystyle UU^{\dagger}+RR^{\dagger}=SS^{\dagger}+QQ^{\dagger}$
$\displaystyle=$ $\displaystyle I\;,$ $\displaystyle
U^{\dagger}U+S^{\dagger}S=R^{\dagger}R+Q^{\dagger}Q$ $\displaystyle=$
$\displaystyle I\;,$ $\displaystyle
US^{\dagger}+RQ^{\dagger}=U^{\dagger}R+S^{\dagger}Q$ $\displaystyle=$
$\displaystyle{\bf 0}\;;\hskip 14.22636pt$ (6)

and the exact seesaw formula that characterizes a kind of balance between the
light and heavy neutrino sectors can also be obtained from Eq. (5):

$\displaystyle UD_{\nu}U^{T}+RD_{N}R^{T}={\bf 0}\;.$ (7)

The smallness of $m_{i}$ is therefore ascribed to the highly suppressed
magnitude of $R$ which signifies the largeness of $M_{i}$ with respect to the
electroweak scale. In this seesaw framework $\nu_{\alpha}$ (for
$\alpha=e,\mu,\tau$) can be expressed as a linear combination of the mass
eigenstates of three active (light) neutrinos and three sterile (heavy)
neutrinos (i.e., $\nu_{i}=(\nu_{i})^{c}$ and $N_{i}=(N_{i})^{c}$ for
$i=1,2,3$). The standard weak charged-current interactions of these six
Majorana neutrinos is given by

$\displaystyle-{\cal L}_{\rm cc}=\frac{g}{\sqrt{2}}\hskip
2.84544pt\overline{\left(e\hskip 5.69046pt\mu\hskip 5.69046pt\tau\right)_{\rm
L}}\hskip
2.84544pt\gamma^{\mu}\left[U\left(\begin{matrix}\nu_{1}\cr\nu_{2}\cr\nu_{3}\cr\end{matrix}\right)_{\rm
L}+R\left(\begin{matrix}N_{1}\cr N_{2}\cr N_{3}\cr\end{matrix}\right)_{\rm
L}\right]W^{-}_{\mu}+{\rm h.c.}\;.$ (8)

So $U$ describes flavor mixing and CP violation of three active neutrinos in
neutrino oscillations, and $R$ measures the strength of weak charged-current
interactions of three heavy neutrinos in precision collider physics and
thermal leptogenesis. Note that $U$ must be non-unitary, but its deviation
from unitarity is at most of ${\cal O}(1\%)$ and hence is insensitive to all
the present experiments.

##  3 The minimal flavor symmetry

As pointed out in Ref. [7], the possibilities of $\big{|}U_{\mu
i}\big{|}=\big{|}U_{\tau i}\big{|}$ and $\delta=\pm\pi/2$ inspire us to
conjecture how $U_{\mu i}$ is directly related to $U^{*}_{\tau i}$ under a
simple flavor symmetry. In the same flavor symmetry $\big{|}U_{ei}\big{|}$
should keep unchanged. We find that $U={\cal P}U^{*}\zeta$ with $\zeta={\rm
Diag}\\{\eta_{1},\eta_{2},\eta_{3}\\}$ and $\eta_{i}=\pm 1$, where $\cal P$
has been given in Eq. (2), satisfies our requirements and allows the PMNS
matrix elements $U_{\alpha i}$ (for $\alpha=e,\mu,\tau$) to transform
together. Inserting $U={\cal P}U^{*}\zeta$ into Eq. (7) and taking the complex
conjugate for the whole equation, we arrive at

$\displaystyle UD_{\nu}U^{T}+{\cal P}R^{*}D_{N}({\cal P}R^{*})^{T}={\bf 0}\;.$
(9)

Comparing this equation with Eq. (7), we immediately get $R={\cal
P}R^{*}\zeta^{\prime}$ with $\zeta^{\prime}={\rm
Diag}\\{\eta^{\prime}_{1},\eta^{\prime}_{2},\eta^{\prime}_{3}\\}$ and
$\eta^{\prime}_{i}=\pm 1$. This result leads us to a novel and rephasing-
invariant prediction $\big{|}R_{\mu i}\big{|}=\big{|}R_{\tau i}\big{|}$, which
is actually a natural consequence of $\big{|}U_{\mu i}\big{|}=\big{|}U_{\tau
i}\big{|}$ in the canonical seesaw mechanism.

Substituting $U={\cal P}U^{*}\zeta$ and $R={\cal P}R^{*}\zeta^{\prime}$ into
Eq. (6), we find $S={\cal T}S^{*}\zeta$ and $Q={\cal T}Q^{*}\zeta^{\prime}$
with $\cal T$ being an arbitrary unitary matrix. Let us proceed to insert all
these four relations into Eq. (5) and then take the complex conjugate for the
whole equation. We are therefore left with

$\displaystyle\left(\begin{matrix}U&R\cr
S&Q\cr\end{matrix}\right)^{\dagger}\left(\begin{matrix}{\bf 0}&{\cal
P}M^{*}_{\rm D}{\cal T}\cr{\cal T}^{T}M^{\dagger}_{\rm D}{\cal P}&{\cal
T}^{T}M^{*}_{\rm R}{\cal T}\cr\end{matrix}\right)\left(\begin{matrix}U&R\cr
S&Q\cr\end{matrix}\right)^{*}=\left(\begin{matrix}D_{\nu}&{\bf 0}\cr{\bf
0}&D_{N}\cr\end{matrix}\right)\;.$ (10)

A comparison between Eqs. (5) and (10) yields

$\displaystyle M_{\rm D}={\cal P}M^{*}_{\rm D}{\cal T}\;,\quad M_{\rm R}={\cal
T}^{T}M^{*}_{\rm R}{\cal T}\;.$ (11)

Inserting Eq. (11) into Eq. (4), the overall neutrino mass term ${\cal
L}^{\prime}_{\nu}$ reads as

$\displaystyle-{\cal L}^{\prime\prime}_{\nu}=\frac{1}{2}\hskip
1.42271pt\overline{\left[{\cal P}(\nu_{\rm L})^{c}\hskip 10.52737pt{\cal
T}N_{\rm R}\right]}\left(\begin{matrix}{\bf 0}&M_{\rm D}\cr M^{T}_{\rm
D}&M_{\rm R}\end{matrix}\right)\left[\begin{matrix}{\cal P}\nu_{\rm L}\cr{\cal
T}^{*}(N_{\rm R})^{c}\end{matrix}\right]+{\rm h.c.}\;.$ (12)

Comparing between Eqs. (4) and (12), we find that ${\cal
L}^{\prime\prime}_{\nu}={\cal L}^{\prime}_{\nu}$ holds under the
transformations [7]

$\displaystyle\nu_{\rm L}\to{\cal P}(\nu_{\rm L})^{c}\;,\quad N_{\rm
R}\to{\cal T}^{*}(N_{\rm R})^{c}\;,$ (13)

no matter what specific form of $\cal T$ is taken.

If the heavy degrees of freedom in the canonical seesaw mechanism are
integrated out, one may obtain the effective Majorana neutrino mass term in
Eq. (1). The corresponding effective mass matrix $M_{\nu}$ can be given by the
approximate seesaw relation $M_{\nu}\simeq-M_{\rm D}M^{-1}_{\rm R}M^{T}_{\rm
D}$. Inserting Eq. (11) into this formula and take the complex conjugate, we
arrive at $M_{\nu}={\cal P}M^{*}_{\nu}{\cal P}$ (namely, $M_{\nu}$ respects
the $\mu$-$\tau$ reflection symmetry although it originates from the seesaw
mechanism).

Figure 2: The charged-lepton-flavor-violating $\beta^{-}\to\alpha^{-}+\gamma$
decays mediated by the light Majorana neutrinos $\nu_{i}$ and the heavy
Majorana neutrinos $N_{i}$ (for $i=1,2,3$), where $\alpha$ and $\beta$ run
over the $e$, $\mu$ and $\tau$ flavors and $m_{\beta}>m_{\alpha}$ holds. The
related unitarity hexagons, reduced as the effective $U$-dominated unitarity
triangles corrected by the effective $R$-dominated vertices, are also shown.

##  4 Charged lepton flavor violation

Now we show that the $\mu$-$\tau$ reflection symmetry in the canonical seesaw
framework can help constrain the unitarity of $U$ through the charged-lepton-
flavor-violating decay modes $\beta^{-}\to\alpha^{-}+\gamma$ (for
$\alpha,\beta=e,\mu,\tau$ and $m_{\beta}>m_{\alpha}$) as illustrated in Fig.
2, which are mediated by both the light Majorana neutrino $\nu_{i}$ and the
heavy Majorana neutrino $N_{i}$ (for $i=1,2,3$). For this purpose, we are more
interested in the following ratios in the natural case of $m_{i}\ll M_{W}\ll
M_{i}$ [8, 9]:

$\displaystyle\xi_{\alpha\beta}$ $\displaystyle\equiv$
$\displaystyle\frac{\Gamma\big{(}\beta^{-}\to\alpha^{-}+\gamma\big{)}}{\Gamma\big{(}\beta^{-}\to\alpha^{-}+\overline{\nu}_{\alpha}+\nu_{\beta}\big{)}}\;$
(14) $\displaystyle\simeq$ $\displaystyle\frac{3\alpha_{\rm
em}}{2\pi}\left|\sum^{3}_{i=1}U_{\alpha i}U^{\ast}_{\beta
i}\left(-\frac{5}{6}+\frac{1}{4}\cdot\frac{m^{2}_{i}}{M^{2}_{W}}\right)-\frac{1}{3}\sum^{3}_{i=1}R_{\alpha
i}R^{\ast}_{\beta i}\right|^{2}$ $\displaystyle\simeq$
$\displaystyle\frac{3\alpha_{\rm em}}{8\pi}\left|\sum^{3}_{i=1}U_{\alpha
i}U^{\ast}_{\beta i}\right|^{2}=\frac{3\alpha_{\rm
em}}{8\pi}\left|\sum^{3}_{i=1}R_{\alpha i}R^{\ast}_{\beta i}\right|^{2}\;,$

where $\alpha_{\rm em}$ is the fine structure constant of electromagnetic
interactions, and $UU^{\dagger}+RR^{\dagger}=I$ has been used. Then we obtain

$\displaystyle\left|\sum^{3}_{i=1}R_{\alpha i}R^{\ast}_{\beta
i}\right|=\left|\sum^{3}_{i=1}U_{\alpha i}U^{\ast}_{\beta
i}\right|=\sqrt{\frac{8\pi}{3\alpha_{\rm em}}\xi_{\alpha\beta}}\simeq
33.88\sqrt{\xi_{\alpha\beta}}\;,$ (15)

where $\alpha_{\rm em}\simeq 1/137$ has been input. Taking account of current
experimental upper bounds on the branching ratios of
$\beta^{-}\to\alpha^{-}+\gamma$ [1], we have $\xi_{e\mu}<4.20\times 10^{-13}$,
$\xi_{e\tau}<1.85\times 10^{-7}$ and $\xi_{\mu\tau}<2.42\times 10^{-7}$. So
Eq. (15) give the constraints

$\displaystyle\left|\sum^{3}_{i=1}R_{ei}\hskip 0.71114ptR^{\ast}_{\mu
i}\right|<2.20\times 10^{-5}\;,\quad\left|\sum^{3}_{i=1}R_{ei}\hskip
1.28032ptR^{\ast}_{\tau i}\right|<1.46\times
10^{-2}\;,\quad\left|\sum^{3}_{i=1}R_{\mu i}R^{\ast}_{\tau
i}\right|<1.66\times 10^{-2}\;.$ (16)

These results mean that the effective $R$-dominated vertices of the effective
$U$-dominated unitarity triangles in Fig. 2 are really small, at most at the
${\cal O}(1\%)$ level.

As we have shown in section 3, $R$ respects the $\mu$-$\tau$ reflection
symmetry as $U$ does. In this case we can achieve a new upper bound on the
flavor mixing factor associated with $\tau^{-}\to e^{-}+\gamma$, which is
about three orders of magnitude more stringent than that obtained in Eq. (16):

$\displaystyle\left|\sum^{3}_{i=1}R_{ei}R^{\ast}_{\tau
i}\right|=\left|\sum^{3}_{i=1}R_{ei}R^{\ast}_{\mu i}\right|<2.20\times
10^{-5}\;.$ (17)

Some other important applications of the $\mu$-$\tau$ reflection symmetry in
neutrino phenomenology have recently been reviewed in Refs. [5, 6]. A further
study of its implications is well worth the wait.

This work was supported in part by the NSFC under grants No. 12075254 and No.
11835013.

## References

  * [1] R. L. Workman et al. [Particle Data Group], “Review of Particle Physics,” PTEP 2022 (2022), 083C01
  * [2] K. Abe et al. [T2K], “Constraint on the matter–antimatter symmetry-violating phase in neutrino oscillations,” Nature 580 (2020) no.7803, 339-344 [erratum: Nature 583 (2020) no.7814, E16] [arXiv:1910.03887 [hep-ex]].
  * [3] Z. z. Xing, “Flavor structures of charged fermions and massive neutrinos,” Phys. Rept. 854 (2020), 1-147 [arXiv:1909.09610 [hep-ph]].
  * [4] P. F. Harrison and W. G. Scott, “$\mu$-$\tau$ reflection symmetry in lepton mixing and neutrino oscillations,” Phys. Lett. B 547 (2002), 219-228 [arXiv:hep-ph/0210197 [hep-ph]].
  * [5] Z. z. Xing and Z. h. Zhao, “A review of $\mu$-$\tau$ flavor symmetry in neutrino physics,” Rept. Prog. Phys. 79 (2016) no.7, 076201 [arXiv:1512.04207 [hep-ph]].
  * [6] Z. z. Xing, “The $\mu$-$\tau$ reflection symmetry of Majorana neutrinos,” [arXiv:2210.11922 [hep-ph]].
  * [7] Z. z. Xing, “Identifying a minimal flavor symmetry of the seesaw mechanism behind neutrino oscillations,” JHEP 06 (2022), 034 [arXiv:2203.14185 [hep-ph]].
  * [8] Z. z. Xing and D. Zhang, “Radiative decays of charged leptons as constraints of unitarity polygons for active-sterile neutrino mixing and CP violation,” Eur. Phys. J. C 80 (2020) no.12, 1134 [arXiv:2009.09717 [hep-ph]].
  * [9] D. Zhang and S. Zhou, “Radiative decays of charged leptons in the seesaw effective field theory with one-loop matching,” Phys. Lett. B 819 (2021), 136463 [arXiv:2102.04954 [hep-ph]].


# ShapeFormer: Shapelet Transformer for Multivariate Time Series
Classification

Xuan-May Le<EMAIL_ADDRESS>The University of
MelbourneMelbourneVictoriaAustralia3010 ,  Ling Luo<EMAIL_ADDRESS>The University of MelbourneMelbourneVictoriaAustralia3010 ,  Uwe Aickelin
<EMAIL_ADDRESS>The University of
MelbourneMelbourneVictoriaAustralia3010 and  Minh-Tuan Tran
<EMAIL_ADDRESS>Monash UniversityMelbourneVictoriaAustralia3800

(2024)

###### Abstract.

Multivariate time series classification (MTSC) has attracted significant
research attention due to its diverse real-world applications. Recently,
exploiting transformers for MTSC has achieved state-of-the-art performance.
However, existing methods focus on generic features, providing a comprehensive
understanding of data, but they ignore class-specific features crucial for
learning the representative characteristics of each class. This leads to poor
performance in the case of imbalanced datasets or datasets with similar
overall patterns but differing in minor class-specific details. In this paper,
we propose a novel Shapelet Transformer (ShapeFormer), which comprises class-
specific and generic transformer modules to capture both of these features. In
the class-specific module, we introduce the discovery method to extract the
discriminative subsequences of each class (i.e. shapelets) from the training
set. We then propose a Shapelet Filter to learn the difference features
between these shapelets and the input time series. We found that the
difference feature for each shapelet contains important class-specific
features, as it shows a significant distinction between its class and others.
In the generic module, convolution filters are used to extract generic
features that contain information to distinguish among all classes. For each
module, we employ the transformer encoder to capture the correlation between
their features. As a result, the combination of two transformer modules allows
our model to exploit the power of both types of features, thereby enhancing
the classification performance. Our experiments on 30 UEA MTSC datasets
demonstrate that ShapeFormer has achieved the highest accuracy ranking
compared to state-of-the-art methods. The code is available at
https://github.com/xuanmay2701/shapeformer.

time series; shapelet; transformer; attention; classification

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX††conference:
30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining; Aug 25 -
29, 2024; Barcelona, Spain††isbn: 978-1-4503-XXXX-X/18/06††ccs: Information
systems Data mining††ccs: Computing methodologies Machine learning

##  1\. Introduction

Figure 1. The illustration depicts the shapelet in the Atrial Fibrillation
dataset. The best-fit subsequence is the subsequence with the sortest distance
to the shapelet in the time series. It is clear that the shapelet can
discriminate between classes by utilising their distance to the best-fit
subsequences.

A multivariate time series (MTS) is a collection of data points where each
point is composed of multiple variables that have been observed or measured
over time. This data structure is prevalent in various fields, such as
economics (Patton, 2012), weather prediction (McGovern et al., 2011),
education (Do et al., 2017), and healthcare (Stevner et al., 2019). Time
series classification stands out as a fundamental and crucial aspect within
the domain of time series analysis (Ruiz et al., 2021a). However, there are
still many challenges in the research on MTS classification (MTSC) (Ruiz et
al., 2021a), especially in capturing the correlations among variables.

Over the past few decades, various approaches have been introduced to enhance
the performance of MTSC (Zhang et al., 2020; Schäfer and Leser, 2017; Li et
al., 2021; Wang et al., 2023; Gao et al., 2022; Zuo et al., 2023). Among
these, shapelets, which are class-specific time series subsequences, have
demonstrated their effectiveness in (Le et al., 2022; Li et al., 2021; Ye and
Keogh, 2009; Grabocka et al., 2016). This success comes from the fact that
each shapelet contains class-specific information representative of its class.
It is evident that the distance between the shapelet and the time series of
its class is far smaller than the time series of other classes (see Figure 1).
Hence, there has been an increased focus on harnessing the capabilities of
shapelets in the field of MTSC.

In 2017, Vaswani et al. (Vaswani et al., 2017) introduced the breakthrough
Transformer architecture, initially designed for Natural Language Processing
but later demonstrating success in Computer Vision tasks (Dosovitskiy et al.,
2020). Following these successes, Transformer-based models have been
effectively applied to MTSC. GTN (Liu et al., 2021) employs a two-tower multi-
headed attention approach to extract distinctive information from input
series, SVP-T (Zuo et al., 2023) captures short- and long-term dependencies
among subseries using clustering and employing them as inputs for the
Transformer, and ConvTran (Foumani et al., 2023) integrates absolute and
relative position encoding for improved position embedding in the Transformer
model.

Obviously, Transformers utilised in MTSC have demonstrated state-of-the-art
(SOTA) performances (Zerveas et al., 2021; Zuo et al., 2023; Foumani et al.,
2023). Existing methods only discover the generic features from timestamps
(Zerveas et al., 2021; Liu et al., 2021; Foumani et al., 2023) or common
subsequences (Zuo et al., 2023) in time series as inputs for the Transformer
model to capture the correlation among them. These features merely contain
generic characteristics of time series, offering a broad understanding of the
data. Nevertheless, they overlook the essential class-specific features
necessary to allow the model to capture the representative characteristics of
each class. As a result, the model exhibits poor performance in two cases: 1)
the dataset has instances that are very similar in overall patterns, differing
only in minor class-specific patterns, effective classification cannot be
achieved using solely generic features; 2) the imbalanced dataset, where
generic features only focus on classifying the majority classes and ignore
those of minority. As can be seen in Figure 2, the hyperplane created using
the generic feature (Figure 2a) attempts to classify the majority classes
(orange triangles and blue circles) and ignores the minority (green squares),
while the class-specific feature (Figure 2b) tries to separate each class from
the others.

Figure 2. The separating hyperplane using (a) the generic feature has a higher
overall accuracy, while the hyperplane using (b) the class-specific feature is
better in classifying a single class.

To address the aforementioned problem, we propose a novel method called
Shapelet Transformer (ShapeFormer), which comprises class-specific and generic
transformer modules to capture both of these features. In the class-specific
module, we initially introduce Offline Shapelet Discovery, inspired by (Le et
al., 2022), to MTS. Based on this, we extract a small number of high-quality
shapelets from the training set. Subsequently, we propose a Shapelet Filter
that leverages the precomputed shapelets to discover the best-fit subsequences
in the input time series. Following this, the Shapelet Filter learns the
difference between the embedding of these shapelets and their most fitting
subsequences derived from the input time series. As shown in Figure 1, the
distance of shapelets to the time series in the same class is far smaller than
the time series of other classes. Similar to the distance, our difference
feature also highlights the substantial distinctions among classes.
Additionally, rather than using the original shapelets extracted from the
dataset, we propose considering these shapelets as the initialisation and then
dynamically optimising shapelets during training to effectively represent the
distinguishing information. In the generic module, we utilise convolution
filters for the extraction of features over all classes. For each module, we
employ the transformer encoder to capture the dependencies between their
features. Through the integration of these two modules, our ShapeFormer excels
in capturing not only class-specific features but also generic characteristics
from time series data. This dual capability contributes to an enhancement in
the overall performance of classification tasks.

Our contributions can be summarised as follows:

  * •

We introduce ShapeFormer, which effectively captures both class-specific and
generic discriminative features in time series.

  * •

We propose the Offline Shapelet Discovery for MTS to effectively and
efficiently extract shapelets from training set.

  * •

We propose the Shapelet Filter, which learns the difference between shapelets
and input time series, which contain important class-specific features. The
shapelets are also dynamically optimised during training to effectively
represent the class distinguishing information.

  * •

We conduct experiments on all 30 UEA MTS datasets and demonstrate that
ShapeFormer has achieved the highest accuracy ranking compared to SOTA
methods.

To the best of our knowledge, our ShapeFormer is a pioneering transformer-
based approach that leverages the power of shapelets for MTSC.

##  2\. Relative Works

###  2.1. Multivariate Time Series Classification

We categorise the MTSC methods into two main categories: non-deep learning,
and deep learning.

Non-deep learning methods. They primarily utilise distance measures (Tran et
al., 2017, 2019), such as Euclidean Distance (Kim et al., 2004), Dynamic Time
Warping, and its diverse variants (Berndt and Clifford, 1994; Keogh and
Ratanamahatana, 2005), to calculate the similarity between time series.
Otherwise, they leverage special features, such as bag of patterns (Lin et
al., 2012), Symbolic Aggregate approXimation (Le et al., 2020), bag of SFA
symbols (Schäfer, 2015), and convolution kernel features (Dempster et al.,
2021; Tan et al., 2022) for classification. (Ruiz et al., 2021b) gives a
comprehensive survey of the conventional methods mentioned.

Deep learning methods. Various neural network methods were proposed for MTSC
(Ismail Fawaz et al., 2019). Specifically, the LSTM-FCN (Karim et al., 2019a)
model features an LSTM layer and stacked CNN layers which directly extract
features from time series. These features are subsequently fed into a softmax
layer to produce class probabilities. However, it has a limitation in
capturing long dependencies among different variables. To address this, Hao et
al (Hao and Cao, 2020). proposed to use of two cross-attention modules to
enhance their CNN-based model. TapNet (Zhang et al., 2020) constructs an
attentional prototype network that incorporates LSTM, and CNN to learn multi-
dimensional interaction features. RLPAM (Gao et al., 2022) adopts a unique
approach by transforming MTS into a univariate cluster sequence and
subsequently employs reinforcement learning for pattern selection. WHEN (Wang
et al., 2023) was proposed to learn heterogeneity by utilising a hybrid
attention network, incorporating both DTW attention and wavelet attention.

###  2.2. Transformer-Based Time Series Classifiers

In 2017, Vaswani et al. introduced the Transformer architecture, achieving a
breakthrough in Natural Language Processing (Vaswani et al., 2017) and
demonstrating notable success in Computer Vision tasks (Dosovitskiy et al.,
2020). Recently, it has proven effective in time series classification tasks.
Specifically, GTN (Liu et al., 2021) utilises a two-tower multi-headed
attention approach for extracting distinctive information from the input
series. The integration of the output from the two towers is achieved through
gating, implemented by a learnable matrix. ConvTran (Foumani et al., 2023) was
proposed to enhance the position embedding by leveraging both absolute and
relative position encoding. SVP-T (Zuo et al., 2023) uses clustering to
identify time series subsequences and employs them as inputs for the
Transformer, enabling the capture of long- and short-term dependencies among
subseries. Recently, the application of pretrained transformer-based self-
supervised learning models like BERT (Devlin et al., 2018) has achieved
significant success not only in the field of NLP but also in other areas (Yu
et al., 2020; Tran et al., 2023; Yang and Han, 2023; Tran et al., 2024).
Inspired by these successes, many models attempt to adopt a similar structure
for time series classification (Zerveas et al., 2021; Yue et al., 2022). It is
noteworthy that most previous transformer-based methods effectively exploit
the generic information of time series.

###  2.3. Shapelet Discovery for Time Series

Shapelets refer to short subsequences within time series that contain class-
specific information by exhibiting a small distance to the time series of the
target class and a larger distance to other classes (see Figure 1).
Additionally, each shapelet can encompass crucial subsequences located at
different positions and variables within a time series. This coverage enables
them to effectively represent the time series. In the last decade, the
effectiveness of shapelets for time series has been proven by many related
studies (Le et al., 2022; Ye and Keogh, 2009; Lines et al., 2012; Grabocka et
al., 2014; Li et al., 2021). The original shapelet discovery method (Ye and
Keogh, 2009) extracts all possible subsequences in the training set and
considers the subsequences as shapelets when they have the highest information
gain ratio. It requires excessive computing time and is hard to apply to MTSC.
Other methods use random shapelets that lack position and variable information
(Grabocka et al., 2016), or employ the common subsequences as shapelets, which
unfortunately have limited discriminative features (Li et al., 2021).
Recently, (Le et al., 2022) proposed the hyperfast Offline Shapelet Discovery
(OSD), which utilises important points to extract a small number of high-
quality shapelets from the original time series data. It has been demonstrated
to be a SOTA method for univariate time series classification.

##  3\. Preliminaries

Multivariate Time Series Classification. We represent MTS as
$\mathbf{X}\in\mathbb{R}^{V\times T}$, where $V$ denotes the number of
variables and $T$ represents the length of the time series. Here,
$\mathbf{X}={X^{1},\ldots,X^{V}}$, and each $X^{v}$ corresponds to a time
series for variable $v$. Specifically, $X^{v}={x^{v}_{1},\ldots,x^{v}_{T}}$,
where $x^{v}_{1}$ signifies a value for variable $v$ at timestamp $t$ within
$\mathbf{X}$. Consider a training dataset
$\mathcal{D}=\\{({\bm{X}}_{i},{\bm{Y}}_{i})\\}_{i=1}^{M}$, where $M$ is the
number of time series instances and the pair $({\bm{X}}_{i},{\bm{Y}}_{i})$
represents a training sample and its corresponding label, respectively. The
objective of MTSC is to train a classifier $f(X)$ to predict a class label for
a multivariate time series with an unknown label.

Time Series Subsequence. Given a time series $X$ of length $T$, a time series
subsequence $X[p_{s}:p_{e}]=x_{p_{s}},...,x_{p_{e}}$ is a consecutive
subsequence of time series $X$, where $p_{s}$ is a start index and $p_{e}$ is
an end index.

Perceptual Subsequence Distance (PSD). Given a time series $X$ of length $T$,
and a subsequence $S=s_{1},...,s_{l}$ of length $l$, with $l\leq T$, the PSD
(Le et al., 2022) of $X$ and $S$ is determined as:

(1) $PSD(X,S)=\min_{j=1}^{T-l+1}\left(\text{CID}(T[j:j+l-1],S)\right)\;,$

where CID is the complexity-invariant distance, which is commonly used in time
series mining, in general, (Batista et al., 2011) and shapelet discovery in
particular (Le et al., 2022).

Figure 3. The general architecture of ShapeFormer.

##  4\. Shapelet Transformer Model

We propose ShapeFormer, a transformer-based method that leverages the strength
of both class-specific and generic features in time series. In contrast to
existing transformer-based MTSC methods (Liu et al., 2021; Zerveas et al.,
2021; Zuo et al., 2023), our approach first extracts shapelets from the
training datasets (Section 4.1). Subsequently, these extracted shapelets are
used to discover discriminative features in time series through the use of a
class-specific transformer module (Section 4.2). Additionally, we introduce
the use of convolution layers with a generic transformer module to extract
generic features in time series (Section 4.3). Finally, the overall
architecture of ShapeFormer is summarised in Section 4.4 and Figure 3.

###  4.1. Shapelet Discovery

This section introduces the Offline Shapelet Discovery (OSD) method, inspired
by (Le et al., 2022), to multivariate time series. In contrast with other
methods, our OSD employs Perceptually Important Points (PIPs) (Chung et al.,
2001), condensing time series data by choosing points that closely resemble
the original, to efficiently select high-quality shapelets. The selection
process is based on the reconstruction distance, with the highest index
continuously chosen. We define the reconstruction distance as the
perpendicular distance between a target point and a line reconstructed by the
two nearest selected important points(Le et al., 2022; Chung et al., 2001).
The process of our OSD is illustrated in Figure 4 and the pseudo-code is
presented in Algorithm 1. Given the dataset
$\mathcal{D}=\\{({\bm{X}}_{i},{\bm{Y}}_{i})\\}_{i=1}^{M}$, our method contains
two main phases, including shapelet extraction and shapelet selection.

In the first phase, our OSD initially extracts shapelet candidates by
identifying PIPs. Specifically, the first and last indices are added to the
PIPs set. Subsequently, the index with the highest reconstruction distance is
continuously added to the PIPs set. Each time a new PIP is added, we extract
new shapelet candidates with three consecutive PIPs points. This means that,
with each new PIP, a maximum of three shapelet candidates can be added to the
set. In this paper, we set the number of PIPs as $npip=0.2\times T$, where $T$
represents the time series length. Our method aims to select a maximum of
$3\times npip$ candidates, therefore, we only extract an average of 5900
candidates for each dataset. This count is significantly smaller than the 45
million candidates typically extracted through classic shapelet discovery
methods (Ye and Keogh, 2009; Lines et al., 2012), thereby significantly
speeding up the process. We then store four types of information for each
shapelet, including the value vector of shapelets, its start index, end index,
and variables.

In the second phase, our method selects an equal number of shapelets for each
class. Given the shapelet candidate $S_{i}$ of class $Y_{i}$, we first compute
its PSD with all instances in the training datasets (Eq. 1). After that, their
distance will be used to find optimal information gain. This implies that the
optimal information gain is the highest ratio achievable by the shapelet
$S_{i}$ (Le et al., 2022). Finally, the top $g$ candidates with the highest
information gain are chosen as the shapelets and stored in the shapelet pool
${\bm{S}}$.

Figure 4. The process of Offline Shapelet Discovery.

Input: $\mathcal{D}=\\{({\bm{X}}_{i},{\bm{Y}}_{i})\\}_{i=1}^{M}$: dataset;
time series length: $T$; number of variables: $V$, number of PIPs: $k$; number
of shapelets: $g$; set of classes: $\mathcal{Y}$; $|Y|$ is the number of
classes.

/* Shapelet Extracting */

1 $\mathcal{C}$ = [];

2 foreach  _$X\sim{\bm{D}}$_ do

3 for  _$v$ = 1 to $V$_ do

4 ${\bm{P}}$ = [1, $T$] # Add the first and last index into PIPs set: $P$;

5 for  _$j$ = 1 to k -2_ do

6 Find index $p$ from 1 to $T$ with maximum reconstruction distance;

7 ${\bm{P}}$.append($p$).sorted() # Add a new index $p$ into ${\bm{P}}$;

8 $idx$ = ${\bm{P}}$.index($p$) for  _$z$ = 0 to 2_ do

9 # Validating newly generated candidates;

10 if  _$idx+2\leq|{\bm{I}}|$ and $idx-z\geq 1$_ then

11 $idx_{s}={\bm{P}}[idx-z]$;

12 $idx_{e}={\bm{P}}[idx+2-z]$;

13 $C=X[idx_{s}:idx_{e}]$;

14 Add new candidates $C$, its start index $idx_{s}$, end index $idx_{e}$, and
its variables $v$ into $\mathcal{C}$.

15

16

17

18

/* Shapelet Selecting */

19 foreach  _$S_{i}\sim\mathcal{C}$_ do

20 $D$ = [];

21 foreach  _$X\sim{\bm{D}}$_ do

22 $d=\text{PSD}(X,S_{i})$ (Eq. 1);

23 $D$.append($d$);

24

25 Compute the optimal information gain of $S_{i}$ using $D$;

26

27$\mathcal{S}$ = [];

28 foreach  _$Y^{i}\sim\mathcal{Y}$_ do

29 Select the top $g/|Y|$ shapelets ranked by information gain in class
$Y^{i}$ from $\mathcal{C}$; Add them to set $\mathcal{S}$;

return $\mathcal{S}$

Algorithm 1 Offline Shapelet Discovery

###  4.2. Class-Specific Transformer

Figure 5. The illustrations for: (a) best-fit subsequence finding method; (b)
difference feature calculation method.

To utilise the class-specific characteristics of shapelets, we first propose
the Shapelet Filter which is used to effectively discover input tokens for the
transformer model.

Shapelet Filter. Given a shapelet pool $\mathcal{S}$ (as discussed in Section
4.1), an input time series $X$ and its label $Y$, we first select the best-fit
subsequence for each shapelet in $\mathcal{S}$ (refer to Figure 5a).
Specifically, with each shapelet $S_{i}\in\mathcal{S}$, its length $l$, start
index $p^{i}_{s}$, end index $p^{i}_{e}$ and variables $v^{i}$, we calculate
the distance CID of them with all subsequences in time series $X$ (Le et al.,
2022). After that, the subsequence with the shortest distance will be selected
as an important subsequence $I_{i}$ of $S_{i}$.

(2) index $\displaystyle=\text{argmin}_{j=0}^{T-l+1}{CID(X[j:j+l],S_{i})}\;,$
(3) $\displaystyle I_{i}$ $\displaystyle=X[\text{index}:\text{index}+l]\;.$

To reduce computing time and effectively utilise the position information of
the shapelet, we propose limiting the search for the best-fit subsequence to a
neighbouring area within the hyperparameter window size $w$ on both the left
and right sides of the actual position of the shapelet. This means that one
shapelet only calculates the distance with maximum $2w+1$ subsequences in $X$.

(4) index
$\displaystyle=\text{argmin}_{j=p_{s}-w}^{p_{s}+w+1}{CID(X[j:j+l],S_{i})}\;,$
(5) $\displaystyle I_{i}$ $\displaystyle=X[\text{index}:\text{index}+l]\;.$

Subsequently, we compute the difference features $U_{i}\in\mathbb{R}^{1\times
d_{spe}}$ between the embedding of each shapelet and their most fitting
subsequences derived from the input time series (see Figure 5b).

(6) $\displaystyle U_{i}=\mathcal{P}_{I}(I_{i})-\mathcal{P}_{S}(S_{i})\;,$

where $\mathcal{P}$ is the linear projector of $\mathbb{R}^{l\times d_{spe}}$
with $l$ is length of shapelet and $d_{spe}$ is the embedding size of
difference features.

Similar to the distance between shapelet and time series, our difference
feature also highlights the substantial distinctions among classes.
Furthermore, by directly incorporating the shapelets in computing the
difference features (Eq. 6), the shapelets are now considered as the learnable
parameters of the Shapelet Filter component. Therefore, rather than using
fixed shapelets, we can use them as the initial parameters of the Shapelet
Filter, which will be optimised during training.

Position Embedding. The difference features $U_{i}$ are then integrated with
position embeddings to capture their order. To better indicate the position
information of shapelets, the embeddings of three types of positions are
considered, including the start index, end index, and variables. Specifically,
we propose to use a one-hot vector representation for these indices and then
employ a linear projector to learn their embedding.

(7) $\displaystyle\text{PE}(p)=\texttt{Linear}(\text{one-hot}(p))\;,$ (8)
$\displaystyle
U_{i}=U_{i}+\text{PE}(p_{s}^{i})+\text{PE}(p_{e}^{i})+\text{PE}(v^{i})\;.$

We also observed that the performance is enhanced when we only use the
position of shapelets instead of the position of best-fit subsequences. This
improvement can be attributed to the fact that the fixed position is easier to
learn than the unstable position of best-fit subsequences.

Transformer Encoder. The class-specific difference features, along with their
corresponding position embeddings, are then input into a transformer encoder
to learn their correlation. Specifically, we employ the multi-head attention
mechanism (MHA) (Vaswani et al., 2017) for this purpose. Given an input
series, ${\bm{U}}=U_{1},\ldots,U_{g}$ and the projections $W_{q}$, $W_{k}$,
$W_{v}\in\mathbb{R}^{d_{spe}\times d_{spe}}$ represent query, key, and value
matrices, respectively. These matrices, $W_{q}$, $W_{k}$, $W_{v}$, undergo
reshaping into $\mathbb{R}^{h\times d_{spe}\times(d_{spe}/h)}$ to signify the
$h$ attention heads and are subsequently concatenated into standard dimensions
after computation. Each attention head within this set is capable of capturing
distinct relationships of the features. Finally, these matrices are used to
compute an output
$\mathbf{Z}^{\text{spe}}=Z_{1}^{\text{spe}},...,Z_{g}^{\text{spe}}$ where
$Z_{i}^{\text{spe}}\in\mathbb{R}^{d_{spe}}$:

(9) $\displaystyle Z_{i}^{\text{spe}}=\sum_{j=1}^{g}{a_{i,j}(U_{j}*W_{v})}\;,$

where $a_{i,j}$ is an attention score which is calculated as:

(10) $\displaystyle a_{i,j}$
$\displaystyle=\texttt{softmax}(\frac{(U_{i}*W_{q})(U_{j}*W_{k})}{\sqrt{d_{spe}}})\;,$

Thanks to the class-represented characteristics of these features, the
attention score for features within the same class is boosted compared to
features in different classes. This enhancement helps the model better
distinguish between different classes. Additionally, owing to the nature of
shapelets, the difference features possess the ability to identify significant
subsequences across different temporal locations and variables within the time
series. This capability enables the module to effectively capture temporal and
variable dependencies in time series data.

Class Token. Existing transformer-based methods apply averaging pooling to
$\mathbf{Z}^{\text{spe}}$, to obtain the final token for classification
(Foumani et al., 2023; Liu et al., 2021). However, our class-specific
transformer module utilises difference features that capture the distinctive
characteristics of each shapelet. Applying average pooling may diminish these
properties, potentially limiting performance. To address this, we propose
using only the first difference feature of the highest information gain
shapelet $Z^{\text{spe}}_{1}$ as the class token $Z^{\text{spe}}_{*}$ for
final classification. The reason for this is that when averaging all tokens,
there is a loss of information regarding distinct features $U_{i}$. Moreover,
the first token $Z_{1}^{\text{spe}}$, which carries the highest information
gain, harbors the most crucial features for effectively classifying time
series.

###  4.3. Generic Transformer

Besides leveraging the power of class-specific features, in this section, we
introduce the generic transformer module, utilising convolution filters to
extract generic features in the time series. Specifically, we employ two CNN
components (Foumani et al., 2023, 2021), each comprising Conv1D, BatchNorm,
and GELU, to effectively discover generic features. The first block is
designed to capture the temporal patterns in time series by using the Conv1D
filter $\in\mathbb{R}^{1\times d_{c}}$. On the other hand, the second block
uses the Conv1D filter $\in\mathbb{R}^{V\times 1}$ to capture the correlation
between variables in time series. In this context, $V$ represents the number
of variables, and $d_{c}$ is the kernel size of the convolution filter, which
is fixed at 8 in all experiments. From that, the output generic feature
$V_{i}\in\mathbb{R}^{1\times d_{gen}}$ is calculated as follows:

(11) ConvBlock(X)
$\displaystyle=\texttt{GELU}(\texttt{BatchNorm}(\texttt{Conv1D}(X)))\;,$ (12)
$\displaystyle{\bm{V}}$
$\displaystyle=\texttt{ConvBlock}(\texttt{ConvBlock}(X))\;.$

Afterward, these features will be fed to the $h$ multi-attention heads to
learn the correlation. Each attention head has the capacity to capture
distinct patterns within time series data.

(13) $\displaystyle{\bm{Z}}^{\text{gen}}=\text{MHA}({\bm{V}}+{\bm{P}})\;,$

as the position of each element $V_{i}\in{\bm{V}}$ lacks inherent meaning, we
utilise the learnable position embedding ${\bm{P}}$ for representing them.
Furthermore, since the module takes classic features as input tokens, we
employ averaging pooling to derive the final class token.

(14)
$\displaystyle{\bm{Z}}^{\text{gen}}_{*}=\texttt{AvgPooling}({\bm{Z}}^{\text{gen}})\;.$

###  4.4. Overall Architecture of ShapeFormer

To enhance clarity, we present the overall architecture of ShapeFormer in
Figure 3. Our method initiates by extracting shapelets from the training
datasets. Subsequently, for a given input time series ${\bm{X}}$, it is
processed through dual transformer modules, comprising the class-specific
shapelet transformer and the generic convolution transformer. The outputs from
these two modules are then concatenated and fed into the final classification
head.

(15) $\displaystyle{\bm{Z}}$
$\displaystyle=\texttt{concat}({\bm{Z}}^{\text{spe}}_{*},{\bm{Z}}^{\text{gen}}_{*})\;,$
(16) $\displaystyle\hat{y}$
$\displaystyle=\texttt{argmax}(\texttt{softmax}(\texttt{Linear}({\bm{Z}})))\;.$

The predictions $\hat{y}$ are used to optimise the model parameters based on
the following objective function:

(17) $\displaystyle\mathcal{L}$
$\displaystyle=\mathcal{L}_{\text{CE}}(\hat{y},Y)\;.$

where, CE is the Cross-Entropy Loss, which can be calculated as
$\mathcal{L}_{\text{CE}}(\hat{y},Y)=\sum_{i}^{|Y|}{y_{i}log(\hat{y_{i}})}$.

Table 1. Accuracies of our proposed method ShapeFormer and 12 compared methods
on all datasets of the UEA archive (Bagnall et al., 2018).

EDI DTWD WEASEL +MUSE MiniRocket LCEM MLSTM -FCNs Tapnet Shapenet WHEN TST
ConvTran SVPT Our ArticularyWordRecognition 0.970 0.987 0.990 0.992 0.993
0.973 0.987 0.987 0.993 0.983 0.983 0.993 0.993 AtrialFibrillation 0.267 0.220
0.333 0.133 0.467 0.267 0.333 0.400 0.467 0.200 0.400 0.400 0.660 BasicMotions
0.676 0.975 1.000 1.000 1.000 0.950 1.000 1.000 1.000 0.975 1.000 1.000 1.000
CharacterTrajectories 0.964 0.989 0.990 0.993 0.979 0.985 0.997 0.980 0.996
0.000 0.992 0.990 0.996 Cricket 0.944 1.000 1.000 0.986 0.986 0.917 0.958
0.986 1.000 0.958 1.000 1.000 1.000 DuckDuckGeese 0.275 0.600 0.575 0.650
0.375 0.675 0.575 0.725 0.700 0.480 0.620 0.700 0.725 ERing 0.133 0.929 0.133
0.981 0.200 0.133 0.133 0.133 0.959 0.933 0.963 0.937 0.966 EigenWorms 0.549
0.618 0.890 0.962 0.527 0.504 0.489 0.878 0.893 N/A 0.593 0.925 0.925 Epilepsy
0.666 0.964 1.000 1.000 0.986 0.761 0.971 0.987 0.993 0.920 0.986 0.986 0.993
EthanolConcentration 0.293 0.323 0.430 0.468 0.372 0.373 0.323 0.312 0.422
0.337 0.361 0.331 0.378 FaceDetection 0.519 0.529 0.545 0.620 0.614 0.545
0.556 0.602 0.658 0.681 0.672 0.512 0.658 FingerMovements 0.550 0.530 0.490
0.550 0.590 0.580 0.530 0.580 0.660 0.776 0.560 0.600 0.700
HandMovementDirection 0.278 0.231 0.365 0.392 0.649 0.365 0.378 0.338 0.554
0.608 0.405 0.392 0.486 Handwriting 0.200 0.286 0.605 0.507 0.287 0.286 0.357
0.452 0.561 0.305 0.375 0.433 0.507 Heartbeat 0.619 0.717 0.727 0.771 0.761
0.663 0.751 0.756 0.780 0.712 0.785 0.790 0.800 InsectWingbeat 0.128 N/A N/A
0.595 0.228 0.167 0.208 0.250 0.657 0.684 0.713 0.184 0.314 JapaneseVowels
0.924 0.949 0.973 0.989 0.978 0.976 0.965 0.984 0.995 0.994 0.989 0.978 0.997
LSST 0.456 0.551 0.590 0.643 0.652 0.373 0.568 0.590 0.663 0.381 0.616 0.666
0.700 Libras 0.833 0.870 0.878 0.922 0.772 0.856 0.850 0.856 0.933 0.844 0.928
0.883 0.961 MotorImagery 0.510 0.500 0.500 0.550 0.600 0.510 0.590 0.610 0.630
N/A 0.560 0.650 0.670 NATOPS 0.850 0.883 0.870 0.928 0.916 0.889 0.939 0.883
0.978 0.900 0.944 0.906 0.989 PEMS-SF 0.973 0.711 N/A 0.522 0.942 0.699 0.751
0.751 0.925 0.919 0.828 0.867 0.925 PenDigits 0.705 0.977 0.948 N/A 0.977
0.978 0.980 0.977 0.987 0.974 0.987 0.983 0.990 PhonemeSpectra 0.104 0.151
0.190 0.292 0.288 0.110 0.175 0.298 0.293 0.088 0.306 0.176 0.293 RacketSports
0.868 0.803 0.934 0.868 0.941 0.803 0.868 0.882 0.934 0.829 0.862 0.842 0.895
SelfRegulationSCP1 0.771 0.775 0.710 0.925 0.839 0.874 0.652 0.782 0.908 0.925
0.918 0.884 0.911 SelfRegulationSCP2 0.483 0.539 0.460 0.522 0.550 0.472 0.550
0.578 0.589 0.589 0.583 0.600 0.633 SpokenArabicDigits 0.967 0.963 0.982 0.620
0.973 0.990 0.983 0.975 0.997 0.993 N/A 0.986 0.997 StandWalkJump 0.200 0.200
0.333 0.333 0.400 0.067 0.400 0.533 0.533 0.267 0.333 0.467 0.600
UWaveGestureLibrary 0.881 0.903 0.916 0.938 0.897 0.891 0.894 0.906 0.919
0.903 0.891 0.941 0.922 Average rank 11.200 9.783 7.933 5.900 6.600 9.833
8.233 6.850 3.117 8.117 5.650 5.283 2.500 Number of top-1 1 1 4 6 4 0 2 2 4 3
4 5 15 Wins 29 29 24 21 25 30 28 27 16 25 24 24 - Draws 0 1 2 2 2 0 1 2 9 0 2
5 - Loses 1 0 4 7 3 0 1 1 5 5 4 1 - P-value 0.000 0.000 0.001 0.014 0.003
0.000 0.000 0.002 0.475 0.005 0.024 0.000 -

##  5\. Experiments

###  5.1. Experimental Setting

Dataset. We assess our approach using the UEA archive, a well-known benchmark
made up of 30 distinct datasets for MTSC (Bagnall et al., 2018). It covers
various domains, including Human Activity Recognition, Motion classification,
ECG classification, EEG/MEG classification, Audio Spectra classification, and
more. The sample sizes of datasets in the UEA archive range from 27 to 50,000,
the time series lengths spanning 8 to 17,984, and dimensions varying from 2 to
1,345.

Metrics. We use classification accuracy to evaluate model performance and
compare methods based on their average ranks and win/draw/loss counts on all
datasets. Finally, we evaluate the statistical significance of performance
differences using the p-value of Friedman and Wilcoxon signed-rank test
(Bagnall et al., 2018).

Implementation Details. Our model was trained using the RAdam optimiser with
an initial learning rate set as 0.01, a momentum of 0.9, and a weight decay of
5e-4. The training process involved a batch size of 16 for a total of 200
epochs. We configured the number of attention heads to be 16 and followed the
protocol outlined in (Zuo et al., 2023; Zerveas et al., 2021). This protocol
involves splitting the training set into 80% for training and 20% for
validation, allowing us to fine-tune hyperparameters. Once the hyperparameters
were finalised, we conducted model training on the entire training set and
subsequently evaluated its performance on the designated official test set.

Environment. All the experiments are conducted on a machine with one Intel(R)
Xeon(R) Silver 4214 CPU @ 2.20GHz and one NVIDIA Tesla V100 SXM2.

###  5.2. Baselines

We have selected 12 baseline methods for the comparative experiments,
comprising two distance-based methods: EDI, $DTW_{D}$ (Bagnall et al., 2018);
a pattern-based algorithm: WEASEL+MUSE (Schäfer and Leser, 2017); a feature-
based algorithm: MiniRocket (Dempster et al., 2021); an ensemble method: LCEM
(Fauvel et al., 2020); three deep learning models: MLSTM-FCNs (Karim et al.,
2019b), Tapnet (Zhang et al., 2020), Shapenet (Li et al., 2021); an attention-
based model: WHEN (Wang et al., 2023); and three transformer-based models: TST
(Zerveas et al., 2021), ConvTran (Foumani et al., 2023), SVP-T (Zuo et al.,
2023). They all attained the SOTA performance described in the most recent
research. The details of 12 baseline methods are shown in Appendix A.

###  5.3. Performance Evaluation

Table 1 illustrates the experimental results of our method with 12 other
competitors on the UEA multivariate time series classification archive
(Bagnall et al., 2018). The accuracy of 12 baseline methods are taken from
(Zuo et al., 2023), except the results of WHEN, and ConvTran are taken from
their original papers (Wang et al., 2023; Foumani et al., 2023). The best
result on each dataset is indicated in bold, and the summarised information is
provided in the last six lines of the table.

The results show that among all methods, ShapeFormer achieves the best
performance in both the highest average rank (2.5) and the largest number of
top-1 (the best in 15 out of 30 datasets). This indicates that ShapeFormer can
be taken as a SOTA for MTSC. The rank index signifies that, even on some
datasets where our model does not exhibit the highest performance, its results
remain highly competitive. Specifically, the average rank of our method is
slightly higher compared to that of the runner-up, WHEN, a difference of
0.617. Meanwhile, the gap in average rank between ShaperFormer and three
Transformer-based methods (TST, ConvTran, SVP-T) is large, with 5.617, 3.15,
and 2.783 respectively. The p-value is $\leq 0.05$, which confirms there ranks
have statistically significant differences. Specifically, the p-values for
ShapeFormer in comparison to all methods are below 0.05, which indicates the
results are statistically significant except for WHEN. However, regarding the
number of top-1, our ShapeFormer attained SOTA results in 15 datasets compared
to WHEN, only 4 datasets.

###  5.4. Ablation Study and Model Design

Effectiveness of Using Shapelets. In Figure 6, we compare the performance when
using random subsequences, common subsequences as mentioned in (Zuo et al.,
2023), and shapelets in our methods. The results demonstrate that the
shapelets outperform the other two methods in terms of accuracy across all
five datasets. This highlights the benefit of highly discriminative shapelet
features in increasing the performance of the transformer-based model, thereby
indicating the contribution of our work.

Figure 6. Accuracies of using shapelets and two other types of subsequences.

Component Evaluation. We begin by evaluating the impact of two key modules in
our ShapeFormer: the Class-specific Transformer (Section 4.2) and the Generic
Transformer (Section 4.3), in comparison with the baseline method, SVP-T (Zuo
et al., 2023) on the first 10 datasets of UEA archive (Bagnall et al., 2018).
In this process, individual components are incrementally incorporated to
assess their impact on the ultimate accuracy. As depicted in Figure 7,
applying the generic transformer alone exhibits a lower accuracy compared to
the baseline. In contrast, utilising only the class-specific module results in
significantly improved performance over the baselines, emphasising the
effectiveness of class-specific features in the transformer-based time series
model. Furthermore, the combination of class-specific and generic components
shows a positive impact on the enhancement of classification accuracy. This
combination harnesses the power of both features, significantly boosting
overall performance.

Figure 7. Average ranks for 3 variations of ShapeFormer and the baseline
(SVP-T (Zuo et al., 2023) \- the current SOTA transformer-based method).

Choosing between the Position of Shapelets and Best-fit Subsequences. Our
ShapeFormer leverages shapelets to find the best-fit subsequences and employ
the difference features $U_{i}$ calculated by them as the inputs for the
Transformer encoder. Then there is a question ”Should we choose the positions
of the shapelets or the best-fit subsequences for position embedding?”. Figure
8 illustrates a comparison between the accuracies achieved by employing the
position of the best-fit subsequences and shapelets, as indicated in Eq. 8,
for position embedding in the Transformer encoder. The outcomes indicate that
our approach exhibits superior performance when utilising the position of
shapelets across all five datasets under consideration. This enhancement can
be ascribed to the fact that learning from the fixed position of shapelets is
easier compared to the unstable position of the best-fit subsequences.

Figure 8. Accuracies of using the position of best-fit subsequences and
shapelets.

Comparison with Various Methods for Calculating Difference Features. The
critical difference diagram in Figure 9 displays the performance of using
different methods for calculating difference features in Eq. 6, including
Manhattan Distance, Euclidean Distance, and the subtraction between
$\mathcal{P}_{I}(I_{i})$ and $\mathcal{P}_{S}(S_{i})$. The results demonstrate
that: 1) All calculation methods for difference features yield better results
compared to the SVP-T baseline; 2) Using subtraction exhibits the highest
performance. Although the subtraction is simple, its superiority lies in
effectively capturing relative changes by considering both the magnitude and
direction of changes between embedding vectors $\mathcal{P}_{I}(I_{i})$ and
$\mathcal{P}_{S}(S_{i})$.

Figure 9. Average accuracy ranks of various calculation methods for difference
features.

Different Class Token Designs. The output of the class-specific transformer
consists of a series of tokens $Z_{1}^{\text{spe}},\ldots,Z_{g}^{\text{spe}}$.
The question at hand is, ”Are there any effective ways to design class tokens
before feeding them to the classification head?”. In Figure 10, we analyse the
impact of different class token designs on the performance of ShapeFormer. The
results indicate that: 1) Our ShapeFormer outperforms the baseline with all
types of class token designs, demonstrating the advantage of our method; 2)
Utilising the first token $Z_{1}^{\text{spe}}$ as the final class token
$Z_{*}^{\text{spe}}$ for ShapeFormer yields the best performance. This is due
to the fact that learning or averaging all tokens results in a loss of
information on difference features $U_{i}$. Furthermore, the first token,
containing the highest information gain, possesses the most discriminative
features for classifying time series.

Figure 10. Average accuracy ranks of different class token designs.

###  5.5. Hyperparameter Sensitivity

Tuning Window Size and Number of Shapelets. In our method, there are two main
parameters related to shapelet discovery that need tuning, including the
window size when calculating the distances between shapelets and subsequences
and the number of shapelets. Regarding the window size, we propose to tune it
exclusively during the shapelet discovery phase. For each dataset, we will
select a window size from the set [10, 20, 50, 100, 200], aiming to provide
the top 100 shapelets with the highest information gain. This tuning technique
can significantly reduce training time since it only operates during the
shapelet discovery phase. As for the number of shapelets, considering the
diverse characteristics of different datasets, we choose this number from [1,
3, 10, 30, 100]. The details of our tuned parameters are shown in Appendix B.

Number of PIPs. As shown in the following Table 2, the model accuracy
increases as we increase the number of PIPs (npips) from 0.05T to 0.2T.
Afterward, accuracy remains stable even with further increases in npips.
Therefore, we set npips at 0.2 for all of our experiments.

Table 2. The average accuracy for various numbers of PIPs.

Npips ($\times T$) 0.05 0.1 0.15 0.2 0.25 0.3 0.4 0.5 Accuracy 0.832 0.848
0.856 0.864 0.864 0.864 0.864 0.864

The Scale Factors $d_{\text{spe}}$ and $d_{\text{gen}}$. In Figure 11a, we
compare the impact of different scale factors of the class-specific and
generic embedding sizes on the classification accuracy of ShapeFormer. The
results show that: 1) The pair of $d_{\text{spe}}=128$ and $d_{\text{gen}}=32$
has achieved the highest accuracy; 2) In general, a larger class-specific
embedding size has achieved better performance, indicating the benefit of
using shapelets in a transformer-based time series model.

Dropout Ratios. In Figure 11b, we analyse the impact of different dropout
ratios of ShapeFormer. It is evident that our methods work well and achieve
high performance with small dropout ratios, with the ratio of 0.4 yielding the
highest performance.

Figure 11. Effectiveness of (a) class-specific and generic scale factors and
(b) different dropout ratios.

###  5.6. Improving Performance in Specific Datasets by Optimizing Scale
Factor

In MTSC, it is crucial to develop models that generalise well across a
majority of datasets rather than models tailored to specific datasets. For
example, in terms of InsectWingbeat dataset, we observed that setting
$d_{\text{gen}}$ (embedding size of generic feature) to 256 leads to
significantly better performance (0.704) compared to $d_{\text{gen}}$ at 32
(0.314) (our chosen parameter). However, this improvement comes at the cost of
decreased performance on other datasets (from 0.864 to 0.831). Therefore, we
recommend tuning this hyperparameter to achieve better performance on specific
datasets if needed.

Table 3. Accuracy for InsectWingbeat dataset and the first 10 UEA datasets
with various $d_{\text{gen}}$ factors.

$d_{\text{gen}}$ 32 (our choice) 64 128 256 InsectWingbeat 0.314 0.500 0.634
0.704 First 10 UEA datasets 0.864 0.852 0.844 0.831

###  5.7. A Case Study of LSST (Imbalanced Datasets)

To illustrate the effectiveness of combining both class-specific and generic
features transformer modules to classify imbalanced data, we conducted
experiments on the LSST dataset. The LSST dataset comprises 16 classes, and we
randomly selected 4 classes to be represented by the colors blue, orange,
green, and red, with 35, 270, 382, and 63 instances, respectively. It is clear
that the sizes of blue and red classes are significantly smaller compared to
the sizes of the green and orange classes. Figure 12a shows that the generic
transformer prioritises majority classes (green and orange), but neglects
minority ones (blue and red). However, in Figure 12b, the combination of
class-specific and generic transformers effectively distinguishes all four
classes.

Figure 12. The t-SNE visualisation in 4 classes of LSST dataset using (a) our
generic transformer and (b) both class-specific and generic transformers. Each
point indicates an instance and the colors of the points signify the true
labels.

###  5.8. A Case Study of BasicMotions

To interpret ShapeFormer results, we use the BasicMotions dataset from the UEA
archive (Bagnall et al., 2018), focusing on human activity recognition with 4
classes (playing badminton, standing, walking, and running). Each class is
associated with 6 variables, and 10 shapelets are set for analysis. Randomly
selecting a ’walking’ instance from the training set. Figure 13a showcases the
top three shapelets for this class and three from others, highlighting
ShapeFormer’s ability to identify crucial subsequences across diverse
locations and variables in the time series. Moreover, shapelets within the
same ’walking’ class tend to share greater similarity with best-fit
subsequences than those from other classes.

In Figure 13b, the attention heat map for all 40 shapelets across 4 classes
reveals that shapelets within the same class generally attain higher attention
scores. For instance, $S_{20}$ and $S_{23}$ belonging to the ’walking’ class
show a small difference feature (Eq. 6), resulting in higher attention scores.
This enhanced attention allows the model to focus more on the correlation
between shapelets within the same classes, thereby improving overall
performance.

Figure 13. (a) The green box depicts the top three shapelets, and the orange
box displays three random shapelets from other classes, extracted in one
random input time series of the ’walking’ class in the BasicMotions dataset.
(b) The attention heat map for all shapelets.

##  6\. Conclusion

In this paper, we propose a novel Shapelet Transformer (ShapeFormer) for
multivariate time series classification. It consists of dual transformer
modules aimed at identifying class-specific and generic features within time
series data. In particular, the first module discovers class-specific features
by utilising discriminative subsequences (shapelets) extracted from the entire
dataset. Meanwhile, the second transformer module employs convolution filters
to extract generic features across all classes. The experimental results show
that by combining both modules, our ShapeFormer has achieved the highest rank
in terms of classification accuracy when compared to the SOTA methods. In
future work, we intend to utilise the power of shapelets in many different
time series analysis tasks such as forecasting or anomaly detection.

## References

  * (1)
  * Bagnall et al. (2018) Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. 2018.  The UEA multivariate time series classification archive, 2018.  _arXiv preprint arXiv:1811.00075_ (2018).
  * Batista et al. (2011) Gustavo EAPA Batista, Xiaoyue Wang, and Eamonn J Keogh. 2011.  A complexity-invariant distance measure for time series. In _Proceedings of the 2011 SIAM international conference on data mining_. SIAM, 699–710.
  * Berndt and Clifford (1994) Donald J Berndt and James Clifford. 1994.  Using dynamic time warping to find patterns in time series. In _Proceedings of the 3rd international conference on knowledge discovery and data mining_. 359–370.
  * Chung et al. (2001) Fu Lai Korris Chung, Tak-Chung Fu, Wing Pong Robert Luk, and Vincent To Yee Ng. 2001.  Flexible time series pattern matching based on perceptually important points. In _Workshop on Learning from Temporal and Spatial Data in International Joint Conference on Artificial Intelligence_.
  * Dempster et al. (2021) Angus Dempster, Daniel F Schmidt, and Geoffrey I Webb. 2021.  Minirocket: A very fast (almost) deterministic transform for time series classification. In _Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining_. 248–257.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Do et al. (2017) Thuc-Doan Do, Tuan Minh Tran, Xuan-May Thi Le, and Thuy-Van T Duong. 2017.  Detecting special lecturers using information theory-based outlier detection method. In _Proceedings of the International Conference on Compute and Data Analysis_. 240–244.
  * Dosovitskiy et al. (2020) Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. 2020\.  An image is worth 16x16 words: Transformers for image recognition at scale.  _arXiv preprint arXiv:2010.11929_ (2020).
  * Fauvel et al. (2020) Kevin Fauvel, Élisa Fromont, Véronique Masson, Philippe Faverdin, and Alexandre Termier. 2020.  Local cascade ensemble for multivariate data classification.  _arXiv preprint arXiv:2005.03645_ (2020).
  * Foumani et al. (2023) Navid Mohammadi Foumani, Chang Wei Tan, Geoffrey I Webb, and Mahsa Salehi. 2023.  Improving Position Encoding of Transformers for Multivariate Time Series Classification.  _arXiv preprint arXiv:2305.16642_ (2023).
  * Foumani et al. (2021) Seyed Navid Mohammadi Foumani, Chang Wei Tan, and Mahsa Salehi. 2021.  Disjoint-cnn for multivariate time series classification. In _2021 International Conference on Data Mining Workshops (ICDMW)_. IEEE, 760–769.
  * Gao et al. (2022) Ge Gao, Qitong Gao, Xi Yang, Miroslav Pajic, and Min Chi. 2022.  A reinforcement learning-informed pattern mining framework for multivariate time series classification. In _In the Proceeding of 31th International Joint Conference on Artificial Intelligence (IJCAI-22)_.
  * Grabocka et al. (2014) Josif Grabocka, Nicolas Schilling, Martin Wistuba, and Lars Schmidt-Thieme. 2014.  Learning time-series shapelets. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_. 392–401.
  * Grabocka et al. (2016) Josif Grabocka, Martin Wistuba, and Lars Schmidt-Thieme. 2016.  Fast classification of univariate and multivariate time series through shapelet discovery.  _Knowledge and information systems_ 49 (2016), 429–454.
  * Hao and Cao (2020) Yifan Hao and Huiping Cao. 2020.  A new attention mechanism to classify multivariate time series. In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence_.
  * Ismail Fawaz et al. (2019) Hassan Ismail Fawaz, Germain Forestier, Jonathan Weber, Lhassane Idoumghar, and Pierre-Alain Muller. 2019.  Deep learning for time series classification: a review.  _Data mining and knowledge discovery_ 33, 4 (2019), 917–963.
  * Karim et al. (2019a) Fazle Karim, Somshubra Majumdar, Houshang Darabi, and Samuel Harford. 2019a.  Multivariate LSTM-FCNs for time series classification.  _Neural networks_ 116 (2019), 237–245.
  * Karim et al. (2019b) Fazle Karim, Somshubra Majumdar, Houshang Darabi, and Samuel Harford. 2019b.  Multivariate LSTM-FCNs for time series classification.  _Neural networks_ 116 (2019), 237–245.
  * Keogh and Ratanamahatana (2005) Eamonn Keogh and Chotirat Ann Ratanamahatana. 2005.  Exact indexing of dynamic time warping.  _Knowledge and information systems_ 7 (2005), 358–386.
  * Kim et al. (2004) Sang-Wook Kim, Dae-Hyun Park, and Heon-Gil Lee. 2004.  Efficient processing of subsequence matching with the Euclidean metric in time-series databases.  _Information processing letters_ 90, 5 (2004), 253–260.
  * Le et al. (2022) Xuan-May Le, Minh-Tuan Tran, and Van-Nam Huynh. 2022.  Learning Perceptual Position-Aware Shapelets for Time Series Classification. In _Joint European Conference on Machine Learning and Knowledge Discovery in Databases_. Springer, 53–69.
  * Le et al. (2020) Xuan-May Thi Le, Tuan Minh Tran, and Hien T Nguyen. 2020.  An improvement of SAX representation for time series by using complexity invariance.  _Intelligent Data Analysis_ 24, 3 (2020), 625–641.
  * Li et al. (2021) Guozhong Li, Byron Choi, Jianliang Xu, Sourav S Bhowmick, Kwok-Pan Chun, and Grace Lai-Hung Wong. 2021.  Shapenet: A shapelet-neural network approach for multivariate time series classification. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 35. 8375–8383.
  * Lin et al. (2012) Jessica Lin, Rohan Khade, and Yuan Li. 2012.  Rotation-invariant similarity in time series using bag-of-patterns representation.  _Journal of Intelligent Information Systems_ 39 (2012), 287–315.
  * Lines et al. (2012) Jason Lines, Luke M Davis, Jon Hills, and Anthony Bagnall. 2012.  A shapelet transform for time series classification. In _Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining_. 289–297.
  * Liu et al. (2021) Minghao Liu, Shengqi Ren, Siyuan Ma, Jiahui Jiao, Yizhou Chen, Zhiguang Wang, and Wei Song. 2021.  Gated transformer networks for multivariate time series classification.  _arXiv preprint arXiv:2103.14438_ (2021).
  * McGovern et al. (2011) Amy McGovern, Derek H Rosendahl, Rodger A Brown, and Kelvin K Droegemeier. 2011.  Identifying predictive multi-dimensional time series motifs: an application to severe weather prediction.  _Data Mining and Knowledge Discovery_ 22 (2011), 232–258.
  * Nie et al. (2022) Yuqi Nie, Nam H Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. 2022.  A time series is worth 64 words: Long-term forecasting with transformers.  _arXiv preprint arXiv:2211.14730_ (2022).
  * Patton (2012) Andrew J Patton. 2012.  A review of copula models for economic time series.  _Journal of Multivariate Analysis_ 110 (2012), 4–18.
  * Ruiz et al. (2021a) Alejandro Pasos Ruiz, Michael Flynn, James Large, Matthew Middlehurst, and Anthony Bagnall. 2021a.  The great multivariate time series classification bake off: a review and experimental evaluation of recent algorithmic advances.  _Data Mining and Knowledge Discovery_ 35, 2 (2021), 401–449.
  * Ruiz et al. (2021b) Alejandro Pasos Ruiz, Michael Flynn, James Large, Matthew Middlehurst, and Anthony Bagnall. 2021b.  The great multivariate time series classification bake off: a review and experimental evaluation of recent algorithmic advances.  _Data Mining and Knowledge Discovery_ 35, 2 (2021), 401–449.
  * Schäfer (2015) Patrick Schäfer. 2015.  The BOSS is concerned with time series classification in the presence of noise.  _Data Mining and Knowledge Discovery_ 29 (2015), 1505–1530.
  * Schäfer and Leser (2017) Patrick Schäfer and Ulf Leser. 2017.  Multivariate time series classification with WEASEL+ MUSE.  _arXiv preprint arXiv:1711.11343_ (2017).
  * Stevner et al. (2019) ABA Stevner, Diego Vidaurre, Joana Cabral, K Rapuano, SørenFønsVind Nielsen, Enzo Tagliazucchi, Helmut Laufs, Peter Vuust, Gustavo Deco, Mark W Woolrich, et al. 2019\.  Discovery of key whole-brain transitions and dynamics during human wakefulness and non-REM sleep.  _Nature communications_ 10, 1 (2019), 1035.
  * Tan et al. (2022) Chang Wei Tan, Angus Dempster, Christoph Bergmeir, and Geoffrey I Webb. 2022.  MultiRocket: multiple pooling operators and transformations for fast and effective time series classification.  _Data Mining and Knowledge Discovery_ 36, 5 (2022), 1623–1646.
  * Tran et al. (2024) Minh-Tuan Tran, Trung Le, Xuan-May Le, Mehrtash Harandi, and Dinh Phung. 2024.  Text-Enhanced Data-free Approach for Federated Class-Incremental Learning.  _arXiv preprint arXiv:2403.14101_ (2024).
  * Tran et al. (2023) Minh-Tuan Tran, Trung Le, Xuan-May Le, Mehrtash Harandi, Quan Hung Tran, and Dinh Phung. 2023.  NAYER: Noisy Layer Data Generation for Efficient and Effective Data-free Knowledge Distillation.  _arXiv preprint arXiv:2310.00258_ (2023).
  * Tran et al. (2019) Tuan Minh Tran, Xuan-May Thi Le, Hien T Nguyen, and Van-Nam Huynh. 2019.  A novel non-parametric method for time series classification based on k-Nearest Neighbors and Dynamic Time Warping Barycenter Averaging.  _Engineering Applications of Artificial Intelligence_ 78 (2019), 173–185.
  * Tran et al. (2017) Tuan Minh Tran, Xuan-May Thi Le, Vo Thanh Vinh, Hien T Nguyen, and Tuan M Nguyen. 2017.  A weighted local mean-based k-nearest neighbors classifier for time series. In _Proceedings of the 9th International Conference on Machine Learning and Computing_. 157–161.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ 30 (2017).
  * Wang et al. (2023) Jingyuan Wang, Chen Yang, Xiaohan Jiang, and Junjie Wu. 2023.  WHEN: A Wavelet-DTW Hybrid Attention Network for Heterogeneous Time Series Analysis. In _Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_. 2361–2373.
  * Wu et al. (2022) Haixu Wu, Tengge Hu, Yong Liu, Hang Zhou, Jianmin Wang, and Mingsheng Long. 2022.  Timesnet: Temporal 2d-variation modeling for general time series analysis. In _The eleventh international conference on learning representations_.
  * Yang and Han (2023) Carl Yang and Jiawei Han. 2023.  Revisiting citation prediction with cluster-aware text-enhanced heterogeneous graph neural networks. In _2023 IEEE 39th International Conference on Data Engineering (ICDE)_. IEEE, 682–695.
  * Ye and Keogh (2009) Lexiang Ye and Eamonn Keogh. 2009.  Time series shapelets: a new primitive for data mining. In _Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining_. 947–956.
  * Yu et al. (2020) Wenhui Yu, Xiao Lin, Junfeng Ge, Wenwu Ou, and Zheng Qin. 2020.  Semi-supervised collaborative filtering by text-enhanced domain adaptation. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 2136–2144.
  * Yue et al. (2022) Zhihan Yue, Yujing Wang, Juanyong Duan, Tianmeng Yang, Congrui Huang, Yunhai Tong, and Bixiong Xu. 2022.  Ts2vec: Towards universal representation of time series. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 36. 8980–8987.
  * Zerveas et al. (2021) George Zerveas, Srideepika Jayaraman, Dhaval Patel, Anuradha Bhamidipaty, and Carsten Eickhoff. 2021.  A transformer-based framework for multivariate time series representation learning. In _Proceedings of the 27th ACM SIGKDD conference on knowledge discovery & data mining_. 2114–2124.
  * Zhang et al. (2020) Xuchao Zhang, Yifeng Gao, Jessica Lin, and Chang-Tien Lu. 2020.  Tapnet: Multivariate time series classification with attentional prototypical network. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 6845–6852.
  * Zhou et al. (2023) Tian Zhou, Peisong Niu, Liang Sun, Rong Jin, et al. 2023\.  One fits all: Power general time series analysis by pretrained lm.  _Advances in neural information processing systems_ 36 (2023), 43322–43355.
  * Zuo et al. (2023) Rundong Zuo, Guozhong Li, Byron Choi, Sourav S Bhowmick, Daphne Ngar-yin Mah, and Grace LH Wong. 2023.  SVP-T: a shape-level variable-position transformer for multivariate time series classification. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 37. 11497–11505.

##  Appendix A Baselines

12 baseline methods are utilised in the comparative experiments, comprising
two distance-based methods, a pattern-based algorithm, a feature-based
algorithm, an ensemble method, three deep learning models, an attention-based
model, and three transformer-based models. They all attained the SOTA
performance described in the most recent research.

  * •

EDI and $DTW_{D}$ (Bagnall et al., 2018): The two benchmark classifiers based
on Euclidean Distance and dimension-dependent dynamic time warping.

  * •

WEASEL+MUSE (Schäfer and Leser, 2017): A classifier based on a bag-of-pattern
approach demonstrated SOTA performance when compared with similar competitors
for MTSC. We choose this algorithm as the representative baseline among
pattern-based methods.

  * •

MiniRocket (Dempster et al., 2021): A feature-based method utilizes random
convolutional kernels to discover features. It performs well in both
univariate and multivariate time series classification.

  * •

LCEM (Fauvel et al., 2020): A hybrid ensemble method that integrates boosting,
bagging, divide-and-conquer, and decision tree components. LCEM demonstrated
superior performance when compared to other random forest methods. We choose
it as a representative illustration of ensemble methods.

  * •

MLSTM-FCNs (Karim et al., 2019b): A deep learning method for MTSC which
utilizes LSTM layer and stacked CNN layers to discover features.

  * •

Tapnet (Zhang et al., 2020): A classifier constructs an attentional prototype
network. Tapnet incorporates LSTM, and CNN to learn multi dimensional
interaction features. We opt for it as another representative of the deep
learning method.

  * •

Shapenet (Li et al., 2021): Shapenet aims to learn representations of
different shapelet candidates in a unified space and selects final shapelets
by training a dilated causal CNN module followed by standard classification.
This model can capture dependencies among variables. We choose it as a
representative of the shapelet-based method.

  * •

WHEN (Wang et al., 2023): An attention-based method that learns heterogeneity
by utilising a hybrid attention network, incorporating both DTW attention and
wavelet attention. It achieved SOTA performance for MTSC on the UEA datasets.

  * •

TST (Zerveas et al., 2021): A transformer-based framework for MTS
representation learning. TST is considered as baseline method that takes the
values at each timestamp as the input for the Transformer model. It gains
great performance for many sequential tasks, such as regression, and
classification.

  * •

ConvTran (Foumani et al., 2023): A transformer-based method for MTSC that
proposed to improve the position embedding in the Transformer model by
leveraging both absolute and relative position encoding.

  * •

SVP-T (Zuo et al., 2023): A method uses clustering to identify time series
subsequences and employs them as inputs for the Transformer, enabling the
capture of long- and short-term dependencies among subseries. It achieved SOTA
performance for MTSC. We choose it as another representative of the
transformer-based method.

##  Appendix B Selected Hyperparameters

In this section, we follow the setting mentioned in Section 5.5 to tune the
hyperparameter of window size and number of shapelets per class. In Table 4,
we provide the selected window size and number of shapelets for each class on
30 UEA MTSC datasets.

Table 4. The selected window size and number of shapelets for each UEA MTSC
dataset.

Dataset Window size Number of shapelets (per class) ArticularyWordRecognition
50 10 AtrialFibrillation 100 3 BasicMotions 100 10 CharacterTrajectories 50 3
Cricket 200 30 DuckDuckGeese 10 100 ERing 50 100 EigenWorms 10 10 Epilepsy 20
30 EthanolConcentration 200 100 FaceDetection 10 10 FingerMovements 20 30
HandMovementDirection 200 100 Handwriting 20 30 Heartbeat 200 100
InsectWingbeat 10 30 JapaneseVowels 10 1 LSST 20 10 Libras 10 30 MotorImagery
100 30 NATOPS 20 1 PEMS-SF 50 10 PenDigits 4 10 PhonemeSpectra 20 30
RacketSports 10 10 SelfRegulationSCP1 100 100 SelfRegulationSCP2 100 100
SpokenArabicDigits 10 100 StandWalkJump 10 100 UWaveGestureLibrary 10 10

##  Appendix C Comparison with Time Series Representation Methods

We conducted an experiment to compare the average accuracy of our ShapeFormer
against SOTA representation methods, specifically TST (Zerveas et al., 2021),
PatchTST (Nie et al., 2022), TimesNet (Wu et al., 2022), and GPT2 (Zhou et
al., 2023) (as shown in Table 5). By adhering to the GPT2 protocol across 10
datasets, our method outperforms the others on all datasets, achieving an
average accuracy of 0.773.

Table 5. Comparison between our proposed ShapeFormer and SOTA time series
representation methods.

TST GPT2 TimesNet PatchTST Ours Averaging Accuracy 0.736 0.740 0.736 0.679
0.773