# A Survey of Time Series Foundation Models: Generalizing Time Series
Representation with Large Language Model

Jiexia YE<EMAIL_ADDRESS>Hong Kong University of Science and
Technology (Guangzhou)China ,  Weiqi Zhang<EMAIL_ADDRESS>Hong Kong
University of Science and TechnologyChina ,  Ke Yi<EMAIL_ADDRESS>South China University of TechnologyChina ,  Yongzi Yu<EMAIL_ADDRESS>Hong
Kong University of Science and Technology (Guangzhou)China ,  Ziyue Li
<EMAIL_ADDRESS>University of CologneGermany ,  Jia Li<EMAIL_ADDRESS>Hong Kong University of Science and TechnologyChina and  Fugee Tsung
<EMAIL_ADDRESS>Hong Kong University of Science and TechnologyChina

###### Abstract.

Time series data are ubiquitous across various domains, making time series
analysis critically important. Traditional time series models are task-
specific, featuring singular functionality and limited generalization
capacity. Recently, large language foundation models have unveiled their
remarkable capabilities for cross-task transferability, zero-shot/few-shot
learning, and decision-making explainability. This success has sparked
interest in the exploration of foundation models to solve multiple time series
challenges simultaneously. There are two main research lines, namely pre-
training foundation models from scratch for time series and adapting large
language foundation models for time series. They both contribute to the
development of a unified model that is highly generalizable, versatile, and
comprehensible for time series analysis. This survey offers a 3E analytical
framework for comprehensive examination of related research. Specifically, we
examine existing works from three dimensions, namely Effectiveness, Efficiency
and Explainability. In each dimension, we focus on discussing how related
works devise tailored solution by considering unique challenges in the realm
of time series.Furthermore, we provide a domain taxonomy to help followers
keep up with the domain-specific advancements. In addition, we introduce
extensive resources to facilitate the field’s development, including datasets,
open-source, time series libraries. A GitHub repository is also maintained for
resource updates (https://github.com/start2020/Awesome-TimeSeries-LLM-FM).

##  1\. Introduction

Time series data refers to a sequence of data points recorded at successive
time intervals. Time series analysis has a longstanding research history
closely tied to the real-world applications (Esling and Agon, 2012). The
earliest time series mining can be traced back to ancient Egypt, where
observations of the Nile River’s fluctuations were analyzed to guide
agricultural production (Conway and Hulme, 1993). In the early days, research
on time series focused on areas such as business and economic activities
(Frank and Stengos, 1988), meteorology and population statistics where the
collected data was relatively small in simple structure (e.g., univariate
sequence). At that time, statistics was the dominant methodology, leading to
the development of various classical models including ARIMA, ARCH (Engle,
1982) and Markov transition model (Hamilton, 1989). However, the advent of
large-scale industrial systems, spanning sectors such as transportation (Zhu
et al., 2018), healthcare (Li et al., 2018), Internet of Things (IoT) (Ge et
al., 2018) and E-commerce (Akter and Wamba, 2016), has led to the generation
of vast and intricate time series data. Beyond time series data, some systems
also generate data in varied modalities including text (Jin et al., 2023a),
images (Shin et al., 2011), and graphs (Li et al., 2019). The data explosion
has fueled the emergence of novel time series applications with increasingly
sophisticated patterns. Examples include traffic congestion detection (Akhtar
and Moridpour, 2021), electrocardiograms (ECGs) classification (Houssein et
al., 2017), e-commerce sales demand prediction (Bandara et al., 2019).
Statistical methods fall short in managing such massive and heterogeneous
datasets, and the reliance on pre-defined pattern assumptions restricts their
utility in tackling those applications with dynamic and complex patterns.

Over the past few decades, machine learning and deep learning have made
remarkable strides across various domains, notably in computer vision (CV) and
natural language processing (NLP) (Young et al., 2018). Unlike statistical
methods, these approaches can handle larger, more diverse datasets in a more
automated manner, requiring less human effort and expertise. These
technologies have introduced sophisticated architectures capable of detecting
more intricate patterns, sparking significant interest of time series
community (Tang et al., 2022; Masini et al., 2023; Ismail Fawaz et al., 2019;
Lim and Zohren, 2021). Consequently, a diverse array of effective
architectures tailored for time series modeling have emerged with different
backbones, including RNNs (Lindemann et al., 2021), CNNs (Zhao et al., 2017;
Chen et al., 2016; Liu et al., 2019), GNNs (Chen and Eldardiry, 2023; Cheng et
al., 2022), Transformers (Wen et al., 2022), Diffusion model(Lin et al.,
2023).

Although these powerful architectures have propelled time series analysis to a
new level, there are still unsolved challenges in this field.

  * •

The first challenge is about knowledge transferability (Shi et al., 2022).
Time series often exhibit seasonality (regular fluctuations at specific
intervals) (Findley et al., 2017) and trends (long-term direction of the data)
(Mudelsee, 2019). Alongside with these identifiable patterns, time series data
also manifests a degree of randomness or noise, often attributed to unknown
factors or patterns. These characteristics can differ greatly across various
domains or even within the same domain over time, due to distribution shifts
(Kim et al., 2021), making it challenging to transfer model or time series
representations learned from one specific task to others. For instance, time
series models (Xie et al., 2023a) trained on stock market data learn patterns
influenced by highly volatile factors like economic indicators, investor
sentiment. While climate models (Mudelsee, 2010) focus on long-term patterns,
seasonal cycles, governed by physical laws rather than human behavior. Owing
to the fundamentally distinct data nature, the knowledge transferability
between different domains remains challenging.

  * •

The second is related to data sparseness. In many traditional time series
scenarios (Sun et al., 2019; Ekambaram et al., 2020), the data is gathered
daily, monthly, or annually (e.g., economic indicators (Basha et al., 2023)),
resulting in inherently sparse datasets. Alternatively, acquiring and
annotating data may have privacy restrictions. For example, classifying
electrocardiograms (ECGs) (Pyakillya et al., 2017) necessitates clinical
diagnoses but they are expensive and the data availability is constrained by
patient privacy. This data scarcity hampers the effective training of deep
learning models. Indeed, in most cases, the available datasets are still not
enough for learning high-quality models (Liu et al., 2023d).

  * •

The third is about the multimodal learning (Baltrušaitis et al., 2018). In the
context of multimodal time series analysis, leveraging the complementary
insights across different modalities can enhance explainability and boost
model performance. For example, in stock movement prediction, news and
comments from social media can directly affect trading activities and
integrating them into models can result in more precise forecasts (Wang et
al., 2020b; Xie et al., 2023b). However, aligning multimodal data collected at
various frequencies or intervals to accurately reflect temporal relationships
between different modality is challenging. Moreover, different modalities may
require distinct techniques to capture information effectively and integrating
these information seamlessly into a cohesive model can be complex.

  * •

Finally, explainability is also highly needed (Zhao et al., 2023a). Detailed
explanations of how models generate predictions or identify patterns can
significantly enhance the utility and acceptability of time series. One case
is that if a utility company uses an energy demand forecasting model (Islam et
al., 2020) to plan electricity generation or set prices, it needs to justify
these decisions to regulators and consumers, showing that the model decisions
are based on reasonable and understandable factors. However, most of the
existing time series models are black-box in nature and lack of explanation
about the model behavior or prediction.

Figure 1. Foundation models have been employed across various tasks and
domains in the context of time series analysis. They have two classes, namely
foundation models pre-trained from scratch for time series and adapted large
language foundation models (i.e. LLM) for time series. Currently, researches
focus on improving the generalization capability of these models by
effectively integrating time series properties or fusing multi-modality data.
The explainability and efficiency are also concerned topics in this field.

Some efforts have been made to tackle the aforementioned challenges, like
transfer learning for time series (Weber et al., 2021; Yan et al., 2024; Ma et
al., 2023; Ismail Fawaz et al., 2018), time series data augmentation (Wen et
al., 2021), multimodal time series analysis (Ding et al., 2023; Chambon et
al., 2018) and explainable artificial intelligence for time series (Rojat et
al., 2021). However, these works mostly concentrate on individual challenges.
The time series community anticipates a multifaceted model capable of
addressing multiple challenges simultaneously. An ideal model would possess
robust generalization abilities for unseen time series tasks during training
and those with scarce data. Additionally, it should also demonstrate the
capability to integrate data from different modalities seamlessly and provide
understandable explanations for its decision-making processes.

Over the past few years, to facilitate knowledge transfer, a novel learning
paradigm that combines transfer learning and self-supervision has emerged,
namely pre-training and fine-tuning paradigm (Han et al., 2021). It first pre-
trains a model on a source domain with broad data and then fine-tunes it on
the target task relevant to the source domain (Deng et al., 2009). BERT
(Devlin et al., 2019) is a language model pre-trained on a large-scale
corpora. Researchers find that it can be adapted to a wide range of downstream
NLP tasks and largely raise their performance bar. This study has inspired a
large number of follow-up works in both NLP (Lewis et al., 2020; Radford et
al., 2019; Zhou et al., 2023a) and CV (Radford et al., 2021; Awais et al.,
2023). Such class of models is termed foundation model (FM) (Bommasani and
Hudson, 2021). They demonstrate strong generalization capabilities on various
downstream tasks. When NLP researchers scale up foundation models by
increasing data or model size, they observe that these larger foundation
models acquire surprising abilities which are not present in smaller
counterparts. Such unexpected capacities are termed emergent abilities (Wei et
al., 2022a), including in-context learning (Brown et al., 2020), instruction
following (Hendrycks et al., 2020), chain-of-thought (CoT) (Miao et al.,
2021). They shift the language foundation models from a transferable NLP task
solver to a general-purpose task solver across domains, now widely known as
large language models (LLMs). The development of LLMs has been rapid and
vigorous, giving rise to many powerful LLMs, like GPT series (Radford et al.,
2019; Brown et al., 2020).

Figure 2. Four research questions and their corresponding contents and
sections.

Inspired by the remarkable success of large language foundation models in NLP,
time series community is increasingly interested in the potential of
foundation models for time-series analysis (Liu et al., 2023a; Cao et al., [n.
d.]; Jin et al., 2023a). One research line is to pre-train a foundation model
from scratch with time series data, mirroring language foundation models.
Pioneering efforts, such as TimesFM (Das et al., 2023) and TimeGPT (Garza and
Mergenthaler-Canseco, 2023), have initiated the pre-training of foundation
models in time series domain. However, data in time series field is relatively
small in scale compared to the vast corpus available in NLP, making it
challenging to produce foundation models with emergent abilities as LLM.
Additionally, foundation models pre-trained on time series data lack the
language generation capabilities, hindering their ability to generate human-
readable explanations. Attracted by the strong generalization ability of large
language foundation models in various downstream tasks, another research line
focuses on adapting large language foundation models (namely LLM) for time
series tasks. Large language foundation models’ strengths in cross-task
generalization, zero-shot/few-shot learning, and reasoning can address
challenges such as knowledge transfer, data scarcity and explainability in the
context of time-series analysis. Broadly speaking, there are two adaption
paradigms of LLM towards time series tasks, namely embedding-visible LLM
adaption and text-visible LLM adaption (Xue and D.Salim, 2022; Xue et al.,
2022; Liu et al., 2023c). They are different in LLM utilization, using fine-
tuning for prompting strategy to adapt LLM towards time series task. They both
face challenges including temporal and LLM space alignment, time series
properties and patterns recognition, multimodal data fusion. Although these
two research lines explore foundation models pre-trained on datasets with
different structures (i.e. time series or text corpus), they all contribute to
a ultimate goal, that is to solve multiple time series challenges in a unified
and comprehensible architecture with powerful generalization capacity.

This survey provides an in-depth analysis on the advancement of foundation
models for time series. The review is guided by four research questions in
Figure 2, covering three analytical dimensions (i.e. effectiveness,
efficiency, explainability) and one taxonomy (i.e. domain taxonomy). (1) How
to adapt foundation models effectively in the context of time series analysis?
We classify the related works into two research lines: pre-training foundation
models from scratch for time series and adapting large language foundation
models (namely LLMs) for time series. For the first line, we discuss
effectiveness through two key phases: data collection and alignment,
architectural design. Regarding the second line, we identify two adaption
paradigms, i.e. embedding visible LLM adaption and textual visible LLM
adaption. Under each adaption paradigm, we discuss the LLM utilization, time
series extraction and multi-modal data fusion. The time series extraction
includes challenges like obtaining appropriate time series representation,
aligning temporal space and LLM space, identifying time series properties and
patterns. Additionally, we examine diverse roles of LLMs that further increase
the effectiveness of LLM adaption. (2) How to pre-train or fine-tune
foundation models efficiently for time series tasks? Given that this area is
emerging, current efficient techniques are adapted from NLP. Therefore, we
first provide a brief overview of cutting-edge NLP efficient methods that are
transferable to this context. We then discuss the efficiency under different
tuning paradigms and summarize efficient methods already in use. (3) How to
obtain explainability of foundation models’ behaviors or decisions in time
series applications? The practical deployment of models necessitates
explainability. We start by exploring the concept of explainability in AI,
highlighting both global and local explanations. We then proceed to review and
distill the advancements in explainability within existing research.  (4) What
is the advancement of foundation models in each time series application
domain? To answer this question, we introduce a domain taxonomy. This taxonomy
allows us to compare the objectives, contributions, and limitations of
existing studies within each domain. Furthermore, we offer a wealth of
resources, such as code, benchmark datasets, time series libraries, and tools
for accelerating LLMs, to support future research endeavors. Figure 4 provides
a comprehensive overview of works based on the four research questions.

Paper Organization The remainder of the survey is arranged as follows: Section
2 introduces relevant survey on foundation models and time series analysis,
directing readers to more studies in each area. Section 3 equips readers with
basic knowledge on foundation models and time series tasks. Section 4 delves
into the critical phases of pre-training foundation models for time series.
Section 5 examines LLM adaption towards time series tasks. Section 6 discusses
efficiency of model fine-tuning and inference. Section 7 summarizes research
on explaining model behavior or decisions. Section 8 introduces the
advancement within each domain. Lastly, Section 9 provides resources including
benchmark datasets, code, and time series libraries and LLM tools.

##  2\. Related Surveys

Survey | Effectiveness | Efficiency | Explainability | Domain
---|---|---|---|---
Foundation Model Pre-trained from Scratch for Time Series | LLM Adaption for Time Series |  Efficient Tuning |  Local Explanation |  Global Explanation |  Specific or General
Adaption to Time Series | Alignment |  Time Series Characteristics | Multimodal
(Jin et al., 2024) | ✗ | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | Specific
(Jin et al., 2023b) | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | Both
(Su et al., 2024) | ✗ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | General
(Jiang et al., 2024) | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | Both
(Zhang et al., 2024a) | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | Both
Ours | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | Both

Table 1. Comparative overview of related surveys, spanning four aspects:
Effectiveness, Efficiency, Explainability, and Domain.

In recent years, traditional algorithms for time series analysis have
undergone significant development and have been thoroughly reviewed in the
literature (Benidis et al., 2022; Hahn et al., 2023). Furthermore, studies by
(Ma et al., 2023; Miller et al., 2024) have delved into the methodologies of
pre-training and universal representation in this field. Recent trends in time
series research have increasingly shifted towards leveraging Large Language
Models (LLMs), motivated by their demonstrated versatility and transferability
across various domains. (Jiang et al., 2024) summarizes the general pipeline
for LLM-based time series analysis and highlights the key techniques for
existing methods (i.e., direct query, prompt design, fine-tuning). (Zhang et
al., 2024a) explores strategies for transferring and distilling knowledge from
LLMs to numerical time series analysis. Various methodologies (i.e., direct
prompting, time series quantization, alignment) are detailed in this survey.
(Su et al., 2024) analyzes the applications of LLMs to time-series forecasting
and anomaly detection. (Jin et al., 2023b) comprehensively reviews large
models tailored for analyzing time series and spatiotemporal data. (Jin et
al., 2024) explore and debate possible directions towards LLM-centric time
series analysis where LLMs play roles of augmenters, predictors, or agents.
While these surveys have investigated the application of Large Language Models
in the domain of time series analysis, they have not systematically summarized
large models for time series —not only the LLMs but also the foundational time
series models trained from scratch. This survey aims to address this gap by
providing a comprehensive review. Furthermore, we extend beyond merely
summarizing existing literature by analyzing the explainability and efficiency
of LLM-based time series models. More detailed comparisons of this survey with
related work are shown in Table 1. Our goal is to provide answers to the
previously outlined research questions and suggest future research directions.

##  3\. Preliminary

###  3.1. Foundation Models

Foundation models are built on the pre-training and fine-tuning paradigm. A
large amount of data is involved in model pre-training, whose target is to
train a general model that could be fine-tuned easily in various downstream
applications and tasks. The development of foundation models has driven the
advancement of AI algorithms in terms of homogenization and emergence
(Bommasani and Hudson, 2021; Wei et al., 2022a). In detail, foundation models
accelerate the consolidation of methodologies and have demonstrated great
potential in learning feature representations automatically and implicitly.

Unlike LLMs, which mainly focus on natural language related data, foundation
models have been widely developed in many different domains, for example,
computer vision (Dosovitskiy et al., 2020; He et al., 2022a; Kirillov et al.,
2023), natural language processing (Devlin et al., 2019; Lewis et al., 2020;
Achiam et al., 2023; Touvron et al., 2023), and graph learning (Cheng et al.,
2023; Zhu et al., 2021; Xia et al., 2022; Hou et al., 2022; Sun et al.,
2023a). Given that data exhibits multiple modalities in the real world,
numerous discussions have emerged on how to develop multimodal foundation
models. For example, CLIP (Radford et al., 2021) learns joint visual and
textual representations. DALL-E (Ramesh et al., 2021) focuses on efficient
image generation conditioned on open-ended text descriptions/prompts. VALL-E
(Wang et al., 2023b) shows the in-context learning capabilities for text to
speech synthesis. However, how to incorporate time series data as one of the
inputs for multimodal foundation models remains a topic worth discussing.

###  3.2. Time Series Analysis

Time series data, regarding the number of channels, could be divided into
univariate time series and multivariate time series. For a $T$-length
univariate time series sample
$\bm{X}=\left\\{x_{t}\right\\}_{t\in\left\\{1,...T\right\\}}\in\mathbb{R}^{T}$,
$x_{t}$ is the temporal signal collected at timestamp $t$. A multivariate time
series sample with $N(N>1)$ channels
$\bm{X}=\left\\{\bm{x_{t}}\right\\}_{t\in\left\\{1,...T\right\\}}\in\mathbb{R}^{N\times
T}$ consisted of an ordered set of $N$-dimensional vectors, where each record
$\bm{x_{t}}$ is the signal at a specific timestamp $t$. For simplicity, we
discuss the multivariate time series in the remaining sections. The univariate
time series could be treated in a similar way.

To tackle different channels of multivariate time series, there are two
popular strategies (Liu et al., 2023a): channel-mixing and channel-
independence. For the channel-mixing strategy, the time series model will
first project multiple channels into the hidden space for channel fusion. For
channel-independence, the model will process different channels individually,
which indicates that all the channels share the same model. Compared with
channel-mixing, whose projector depends on the number of multivariate time
series channels, channel-independence configuration could easily handle the
time series with varying channels. In this vein, channel-independence is
adopted widely for time series analysis with domain transfer or model pre-
training.

In general, the mainstream time series analysis tasks include time series
classification, time series forecasting, time series imputation, and time
series anomaly detection (as shown in Fig. 3).

####  3.2.1. Time series classification

Given a multivariate time series data $\bm{X}$, the time series classification
model $f:\mathbb{R}^{N\times T}\rightarrow\mathbb{R}^{C}$ aims to distinguish
the time series data of $C$ different categories by

$I^{c}=f(\bm{X}),$

and return the class index $I^{c}$.

####  3.2.2. Time series imputation

The goal of time series imputation is to recover the value of missing
observations precisely. To describe the data missingness of time series
$\bm{X}$, we denote $\bm{M}\in\\{0,1\\}^{N\times T}$ as the mask matrix, where
$\bm{X}_{i,j}$ can be observed only if $\bm{M}_{i,j}=1$.

Given the observed time series data $\bm{X}^{\star}$, mask matrix $\bm{M}$,
the imputation algorithm aims to recover the missingness of data matrix by

$\bm{\hat{X}}=f\left(\bm{X}^{\star}\right),$

where $\bm{X}^{\star}_{i,j}=\begin{cases}\mathrm{NA},&\text{if
}\bm{M}_{i,j}=0\\\ \bm{X}_{i,j},&\text{ otherwise }\end{cases}$, $\mathrm{NA}$
represents a missing value, and $f:(\mathbb{R}\cup\\{\mathrm{NA}\\})^{N\times
T}\rightarrow\mathbb{R}^{N\times T}$ denotes a learnable imputation function.

####  3.2.3. Time series anomaly detection

At timestamp $t$, the anomaly detection model is developed to find out whether
there exist anomalies in the past time series subsequence. Given a
multivariate time series subsequence
$\left\\{\bm{x_{t-w+1}},...,\bm{x_{t}}\right\\}$ with window size $w$, the
multivariate time series anomaly detection model $f:\mathbb{R}^{N\times
W}\rightarrow\left\\{0,1\right\\}$ aims to return an anomaly indicator
$I^{a}_{t}\in\left\\{0,1\right\\}$ by

$I^{a}_{t}=f\left(\left\\{\bm{x_{t-w+1}},...,\bm{x_{t}}\right\\}\right),$

where there is indeed an anomaly in subsequence if $I^{a}_{t}=1$, the
subsequence has no anomaly otherwise.

####  3.2.4. Time series forecasting

Time series forecasting aims at predicting the future value with horizon $H$
based on the input time series with lookback window $L$. Given the input time
series data $\bm{X}_{L}=\left\\{\bm{x_{1}},...,\bm{x_{L}}\right\\}$, the
forecasting model provide the predicted value for future $H$ timestamps by

$\left\\{\bm{\hat{x}_{L+1}},...,\bm{\hat{x}_{L+H}}\right\\}=f(\bm{X}_{L}),$

where $f:\mathbb{R}^{N\times L}\rightarrow\mathbb{R}^{N\times H}$ is a
learnable forecasting function. Meanwhile, considering the length of the
forecasting horizon, the task could be further divided into long-term/short-
term time series forecasting. Based on the quantity of training data, few-
shot/zero-shot forecasting pipeline could also be established.

Figure 3. Illustration for typical time series analysis tasks: Time series
classification, imputation, anomaly detection, and forecasting.

###  3.3. Time Series Properties

####  3.3.1. Temporal Dependency.

Time series data, gathered at various timestamps, inherently exhibits temporal
dependencies. Past observations serve as indicators for predicting future
values. Consequently, time series analysis models typically necessitate a
subsequence of the time series as input to effectively discern these latent
temporal relationships (Leung et al., 2023; Zhang et al., 2023d; Zeng et al.,
2023; Liang et al., 2023; Zhang et al., 2023c; Deng et al., 2022).

For instance, in various practical contexts, time series data exhibit
characteristics like nonstationarity and seasonality. Nonstationarity refers
to the statistical properties of the series, such as its distribution, evolve
over time. This variability introduces complexity to forecasting since past
trends may not persist, given changes in the data’s generative process (Liu et
al., 2022, 2023b, 2024; Shen et al., 2023).

Seasonality, another key aspect of temporal dependency, involves consistent
and predictable fluctuations occurring at regular intervals—daily, weekly,
monthly, or yearly. Identifying and understanding seasonality enhances the
modeling process by allowing for the incorporation of these cyclic patterns
and some timestamp information into predictions, thereby improving the
comprehension and analysis of the data (Cao et al., [n. d.]; Wu et al., 2022;
Zhang et al., 2023c; Woo et al., 2021).

####  3.3.2. Spatial Dependency.

For multivariate time series data, individual time series could represent
different entities of complex systems. The inter-series relationship, referred
to as spatial dependency, is vital for holistic system modeling. On the one
hand, certain variables within a multivariate time series might fall short of
providing ample information for model construction. On the other hand,
effective modeling of spatial dependency is crucial to uncover the concealed
patterns within the time series data (Zhang et al., 2022b; Ye et al., 2022;
Shao et al., 2022; Wang et al., 2023a).

####  3.3.3. Semantics Diversity.

Unlike image and text data, where consistent semantics can often be found
across different domains (with each word or visual patch representing similar
meanings in various sentences or images), time series data lacks this
uniformity. Identical subsequences or shapelets in time series datasets can
represent entirely different concepts depending on the context. This
variability complicates the process of learning representations and
transferring models in time series analysis, presenting unique challenges to
accurately interpret and utilize the data (Trirat et al., 2024).

0-1=dj0-1= Generalizing Time Series Representation with Foundation Model
---
0-1= Effectiveness 0-1= Pre-training FM from Scratch for Time Series 0-1=
Encoder-decoder 0-1= TimeGPT (Garza and Mergenthaler-Canseco, 2023) 0-1=
Encoder-only 0-1= ForecastPFN (Dooley et al., 2024), TimeCLR (Yeh et al.,
2023), GTT (Feng et al., 2024) 0-1= Decoder-only 0-1= Lag-Llama (Rasul et al.,
2023), TimesFM (Das et al., 2023) 0-1= Adapting LLM for Time Series 0-1= Text-
Visible LLM Adaption 0-1= TWSN (Xie et al., 2023b), TDML (Yu et al., 2023),
UMEF (Xue and Salim, 2023), LLMST (Zhang et al., 2023a), LLM-Mob (Wang et al.,
2023c), LLMTIME (Gruver et al., 2023), PromptCast (Xue and D.Salim, 2022),
LLMFS (Liu et al., 2023c), AuxMobLCast (Xue et al., 2022) 0-1= Embedding-
Visible LLM Adaption 0-1= FPT (Zhou et al., 2023b), UniTime (Liu et al.,
2023a), TEST (Sun et al., 2023b), TEMPO (Cao et al., [n. d.]), LLM4TS (Chang
et al., 2023), TimeLLM (Jin et al., 2023a) 0-1= Efficiency 0-1= Adapting LLM
for Time Series 0-1= Tuning Free 0-1= LLMTIME (Gruver et al., 2023), TWSN (Xie
et al., 2023b), TDML (Yu et al., 2023), LLMST (Zhang et al., 2023a), LLM-Mob
(Wang et al., 2023c), LLMFS (Liu et al., 2023c) 0-1= Full Fine-tuning 0-1=
PromptCast (Xue and D.Salim, 2022), UniTime (Liu et al., 2023a), UMEF (Xue and
Salim, 2023), AuxMobLCast (Xue et al., 2022) 0-1= Parameter-efficient Fine-
tuning 0-1= TEMPO (Cao et al., [n. d.]), TEST (Sun et al., 2023b), FPT (Zhou
et al., 2023b), LLM4TS (Chang et al., 2023), TimeLLM (Jin et al., 2023a) 0-1=
Explainability 0-1= Adapting LLM for Time Series 0-1= Local Explanation 0-1=
TWSN (Xie et al., 2023b), TDML (Yu et al., 2023), TEMPO (Cao et al., [n. d.]),
LLMST (Zhang et al., 2023a), LLM-Mob (Wang et al., 2023c) 0-1= Global
Explanation 0-1= FPT (Zhou et al., 2023b) 0-1= Domain 0-1= Pre-training FM
from Scratch for Time Series 0-1= General Domain 0-1= ForecastPFN (Dooley et
al., 2024), TimeCLR (Yeh et al., 2023), GTT (Feng et al., 2024), Lag-Llama
(Rasul et al., 2023), TimesFM (Das et al., 2023),TimeGPT (Garza and
Mergenthaler-Canseco, 2023) 0-1= Adapting LLM for Time Series 0-1= General
Domain 0-1= TEMPO (Cao et al., [n. d.]), LLMTIME (Gruver et al., 2023),
PromptCast (Xue and D.Salim, 2022), UniTime (Liu et al., 2023a), TEST (Sun et
al., 2023b), FPT (Zhou et al., 2023b), LLM4TS (Chang et al., 2023), TimeLLM
(Jin et al., 2023a) 0-1= Traffic 0-1= LLMST (Zhang et al., 2023a), LLM-Mob
(Wang et al., 2023c), AuxMobLCast (Xue et al., 2022) 0-1= Finance 0-1= TWSN
(Xie et al., 2023b), TDML (Yu et al., 2023) 0-1= Healthcare 0-1= LLMFS (Liu et
al., 2023c)

Figure 4. A taxonomy of investigated works under four research themes, namely
effectiveness, efficiency, explainability and domain. FM refers to Foundation
Model.

##  4\. Pre-training Foundation Models from Scratch for Time Series

Model |  Parameter Size |  Transformer Mode |  Channel Setting |  Task Type |  Pre-trained Dataset |  Data Size
---|---|---|---|---|---|---
ForecastPFN (Dooley et al., 2024) | - | Encoder-only | Uni. | Fore. | Synthetic Data | -
TimeGPT (Garza and Mergenthaler-Canseco, 2023) | - | Encoder-decoder | Uni. | Fore. | - | 100 B time points
TimesFM (Das et al., 2023) | 225M | Decoder-only | Uni. | Fore. |  Google Trends (Goo, 2023) Wiki Pageviews (Wik, 2023) Synthetic Data | 101B time points
Lag-Llama (Rasul et al., 2023) | - | Decoder-only | Uni. | Fore. | Monash (Tan et al., 2020) | 0.3B time points
TimeCLR (Yeh et al., 2023) | - | Encoder-only | Uni. | Class. | UCR (Dau et al., 2019) | -
GTT (Feng et al., 2024) | 57M | Encoder-only | Multi. | Fore. | - | 2.4B time points

Table 2. A comparable analysis of general purposes foundation models. The
channel setting has two types, i.e. univariate (Univ.) and multivariate
(Mult.). ”Fore.” and ”Class.” mean forecasting and classification tasks,
respectively.

The term ”foundation model” was coined in (Bommasani and Hudson, 2021) to
describe a broad category of models initially trained on large and diverse
datasets, subsequently adapted for various downstream tasks through methods
such as fine-tuning. Originating from deep learning, incorporating self-
supervision and transfer learning, foundation models are not entirely new. As
these models grow in scale, they demonstrate impressive zero-shot and few-shot
learning capabilities for a range of new tasks, alongside unexpected emergent
abilities, such as chain-of-thought reasoning, frequently surpassing task-
specific models. This success is notably evident in NLP and CV. On the other
hand, many traditional time series scenarios often grapple with data
limitations, struggling to accumulate enough data to train complex models
effectively. This drawback has led to growing anticipation within the time
series community for a foundational model akin to large language models,
expected to excel in zero-shot/few-shot learning with limited time series
data.

Compared to the exponential growth of foundation models in NLP and CV, the
foundation model pre-trained on time series data has been comparatively
limited. This limitation is primarily attributed to the relatively modest
scale of available time series datasets. However, there have been some
pioneering efforts recently, including ForecastPFN (Dooley et al., 2024),
TimeGPT (Garza and Mergenthaler-Canseco, 2023), Lag-Llama (Rasul et al.,
2023), GTT (Feng et al., 2024), TimeCLR (Yeh et al., 2023), TimesFM (Das et
al., 2023). We provide a comprehensive comparison of the current foundation
models in Table 2. In this section, we examine the crucial stages in pre-
training a foundation model that influences its effectiveness, including data
processing and architecture design.

###  4.1. Data Processing

####  4.1.1. Data Collection

The remarkable generalization capability of Large Language Models (LLMs) stems
from their pre-training on large-scale and high-quality textual corpora.
Similarly, for time series analysis, access to extensive and high-quality data
is essential for pre-training an effective foundation model. We review the
data collection practices of existing studies, focusing on aspects such as
data split, data source and scale, data augmentation, and data quality.

Data Split. In constructing a foundation model with standardized protocols,
datasets for pre-training are divided into training and validation sets.
During the fine-tuning phase, the model is exposed to target datasets not seen
during pre-training, with each dataset being further split into training,
validation, and test sets.

Data Source and Scale. Among the existing foundation models for time series
forecasting, Lag-Llama (Rasul et al., 2023) is pre-trained mainly on the
Monash time series repository with 305,443 individual time series. The unseen
M4 Weekly and traffic datasets are reserved as test sets. GTT (Feng et al.,
2024) collects a repository from both internal and public sources, consisting
of 180,000 time series with a total of 2.4 billion time points. It generates
about 200M high-quality training samples and 24M validation samples for pre-
training data. Following PatchTST (Nie et al., 2022), GTT (Feng et al., 2024)
uses 8 popular datasets as the target datasets. To build a large pre-training
corpus, TimesFM (Das et al., 2023) mainly chooses three data sources, i.e.,
Google trends, Wiki Pageview statistics, and synthetic time series with
approximately 101B data points. They utilize three common archives as target
datasets, namely Darts (Herzen and et.al, 2022), Monash (Tan et al., 2020),
and Informer datasets (Zhou et al., 2022b). TimeGPT (Garza and Mergenthaler-
Canseco, 2023) claims that it has built up the largest time series repository
with over 100 billion data points from public sources. However, it does not
public its repository and reveal the data details. TimeCLR (Yeh et al., 2023)
builds a foundation model for time series classification with the UCR (Dau et
al., 2019) archive. A pre-training set is created by randomly sampling 50% of
each dataset in UCR but the label information is not available. The fine-
tuning set is the remainder of each UCR dataset with a mutually exclusive
3:1:1 ratio as training/validation/test sets.

Data Augmentation. To enlarge the pre-training datasets, various data
augmentation techniques are utilized in the existing works. Lag-Llama (Rasul
et al., 2023) employs Freq-Mix and Freq-Mask (Chen et al., 2023a) to generate
more training samples to prevent overfitting. TimeCLR (Yeh et al., 2023)
adopts the data augmentation techniques (e.g., jittering, time warping,
cropping) to generate more data, enabling the model to be invariant to
warping, different noise types, and so on. It also utilizes a single
augmentation function to generate positive pairs for contrastive learning.
Instead of utilizing real-world data, ForecastPFN (Dooley et al., 2024) is
pre-trained on purely synthetic data distribution. It assumes that the real-
world datasets are derived from a prior distribution of time series.

Data Quality. Data quality is essential for ensuring the effectiveness of
models. Poor-quality pre-training data has been shown to impair the
performance of Large Language Models (Lee et al., 2022). Challenges such as
missing values, noise, and outliers are common in time series data. For
example, UCR archive contains series with missing values (Dau et al., 2019).
To remove the diverse outliers that might cause exploding gradients,
ForecastPFN (Dooley et al., 2024) first masks out the missing value and then
clips all 3-sigma outliers. Similarly, GTT eliminates data points normalized
beyond a value of 9 to remove extreme outliers (Feng et al., 2024).

In summary, the time series data used in current studies are predominantly
public, with the exception of GTT’s internal sources. For more information on
these public datasets, please refer to section 9. When it comes to data
volume, TimeGPT has assembled the largest repository of time series data to
date, though it remains unavailable for public access.

####  4.1.2. Data Alignment

Different from a specific model trained on one dataset, the foundation model
is pre-trained on multiple datasets from diverse domains. These datasets are
different in data scale, numerical magnitude, channel number, input and output
length. It is essential to align and balance these datasets to ensure the
generalizability of model.

Unlike modalities such as images, audio and text, a unique challenge in
handling time series data lies in the variability of value ranges across
datasets. Standard scaling methods, such as Z-score normalization and min-max
scaling, are often ineffective because the absolute range differs greatly
across datasets. GTT (Feng et al., 2024) normalizes each time series sample on
a channel-wise basis while Lag-Llama (Rasul et al., 2023) uses the scaling
heuristic from DeepAR (Salinas et al., 2020) for normalization within each
univariate input window. These sample-specific normalization techniques
greatly improve the convenience of model usage during inference.

To address the variation in channel numbers across datasets, most studies
convert multichannel data into single-channel inputs. But GTT (Feng et al.,
2024) is the only work to build a multivariate foundation model by setting the
channel count to 32 and using zero-padding for datasets with fewer than 32
channels. As for the context length, GTT (Feng et al., 2024) fixes it to 1024
and uses zero-masking to create samples of varied lengths within the training
dataset.

Given the significant variance in dataset sizes, it is crucial to adopt a
balanced approach to guarantee that the model learns patterns from a wide
range of datasets instead of overfitting specific ones. TimesFM (Das et al.,
2023) employs variable context lengths for datasets of different granularities
(e.g., weekly, monthly) to achieve data balance across all levels of
granularity.

In summary, critical measures for model alignment include value scaling,
handling variable input and output lengths, managing multichannel data, and
implementing balanced sampling. These strategies are essential for stabilizing
the training process and preventing performance degradation.

###  4.2. Architecture Design

In this subsection, we delve into the factors that shape the effective
architecture of foundation models, covering aspects like the choice of
backbone models, transformer variants, and strategies for input tokenization.
First, we summarize how existing works choose their backbone models.
Interestingly, all the works ultimately select transformers as the backbone
model but with different variants. We then discuss the pros and cons of these
variants for time series analysis. Given that transformers are designed for
token-like inputs, the discussion will extend to how current studies handle
channel settings of time series data.

####  4.2.1. Backbone for Foundation Model

A deep learning model can serve as the basis for a foundation model, provided
that its size can be scaled up. Scaling is indeed crucial for developing
remarkably successful LLMs (Chowdhery et al., 2023; Kaplan et al., 2020). Due
to transformer architecture’s outstanding capacity for parallelization, it
allows scaling to a massive number of parameters, making it the preferred
backbone for LLMs.

In the context of time series analysis, TimeCLR (Yeh et al., 2023) compares
several backbone models, including GRU, LSTM, ResNet, and transformer, and
finds that the transformer outperforms the alternatives. TimesFM (Das et al.,
2023) suggests that the transformer’s adaptability to varying context lengths
makes it suitable for processing time series data of diverse lengths. It also
replaces the transformer with different architectures, namely PatchTST and
N-BEATS, and finds that the transformer has a better or similar performance
than its counterparts in most benchmark datasets. Lag-Llama (Rasul et al.,
2023) holds that the attention mechanism can extract the diverse past events
and predict the potential future distributions correctly, which is suitable
for probabilistic time series forecasting. Ultimately, all the existing
foundation models choose transformers as their backbone models. The key
differences among these transformer-based foundation models lie in the
transformer mode, input tokenization, and predictive objects. We will continue
the discussion and comparisons in the following. Note that additional
potential architectures have been suggested in (Miller et al., 2024),
including Transformer++ and State-Space Models (Lu et al., 2024).

####  4.2.2. Transformer Mode

The first transformer (Vaswani et al., 2017) encompasses two stacks of blocks
as the encoder and decoder. Each block is composed of multiple layers with the
multi-head self-attention mechanism. The encoder embeds the input sequence
into latent representations, and then the decoder processes them to generate
the output sequence. It has two modes, namely, the encode-only and the
decoder-only transformer. At the early stage, the encoder-only LLMs are more
popular, and the most representative one is BERT (Devlin et al., 2019). Since
the introduction of GPT-3, decoder-only LLMs have seen a significant rise in
popularity, e.g., GPT series (Radford et al., 2019), LLaMA (Touvron et al.,
2023), OPT (Zhang et al., 2022a), PaLM (Chowdhery et al., 2023), Bloom
(Workshop et al., 2022). The encoder-decoder LLM includes BART (Lewis et al.,
2020) and T5 (Raffel et al., 2020), Flan-T5 (Kanakarajan and Sankarasubbu,
2023). Experiments show that the decoder-only mode has better performance in
zero-shot and few-shot learning than other architectures (Wang et al., 2022b).
GPT-3 (Brown et al., 2020) serves as the most convincing proof of being an
effective few-shot learner. Also, the scaling law has been widely observed in
decoder-only LLMs to improve the performance substantially, while efforts to
investigate the large-scale encoder-decoder models are still lacking.

These modes differ in properties and suitability for time series tasks. The
encoder-only architecture handles the entire input sequence at once and
predicts based on the understanding of the full context. It is ideal for tasks
that require interpreting the whole input data (such as classification,
sentiment analysis (Das and Singh, 2023), or named entity recognition (Ehrmann
et al., 2023)). TimeCLR (Yeh et al., 2023) builds an encoder-only foundation
model for time series classification. Although the encoder-only model lacks
auto-regressive nature, GTT (Feng et al., 2024) and ForecastPFN (Dooley et
al., 2024) still develop encoder-only foundation models for time series
forecasting. GTT (Feng et al., 2024) clarifies that it adopts the encoder-only
mode for ensuring the normalization of predicted values within the entire
context window, while the decoder-only mode can not assure this due to its
causal attention mechanism. The decoder-only models generate tokens
autoregressively, where each output token is generated based on all the
previous tokens. It is suitable for sequential generation tasks, such as text
generation, machine translation, or summarization. TimesFM (Das et al., 2023)
and Lag-Llama (Rasul et al., 2023) choose decoder-only architecture for time
series forecasting. The encoder-decoder mode is more complex in architecture
than its counterparts. But its distinct separation of input processing and
output generation phases gives it a competitive edge in tasks where the
relationship between input and output is complicated. TimeGPT (Garza and
Mergenthaler-Canseco, 2023) chooses encoder-decoder mode perhaps because it
needs to extract varied and intricate patterns from over 100 billion data
points.

####  4.2.3. Channel Setting

Another issue relevant to the architecture design of time series foundation
models is the channel setting, specifically channel-independence and channel-
mixing. Channel-independence refers to accept univariate sequence input while
channel-mixing involves the utilization of multivariate sequence input. These
different channel settings result in varied tokenization approaches for time
sequences and necessitate distinct model designs.

Channel-Independence. Designing univariate foundational models is simpler than
multivariate models because the channel numbers of multivariate data may
differ across datasets (Rasul et al., 2023). Moreover, since multivariate
series can be converted into univariate series, most researches prefer the
channel-independence setting for model design. To construct a univariate
transformer-based foundation model, the initial step involves converting the
univariate input sequence into a vector sequence, similar to the token
sequence. There are different techniques to achieve this goal. Inspired by the
success of time series patching strategy in PatchTST (Nie et al., 2022),
TimesFM (Das et al., 2023) cuts the univariate sequence into patches. By
aggregating adjacent time steps to form a single patch, the patching strategy
can extract local temporal information of the input sequence. Lag-Llama (Rasul
et al., 2023) argues that the overlapping patch strategy may mix causality, as
each patch vector includes values from both preceding and subsequent vectors.
To address this, they recommend tailoring vectorization to the time series
data’s frequency, constructing lag vectors by applying lag operations to
capture features at various time intervals—quarterly, monthly, weekly, daily,
and hourly. In this way, each lag vector contains values from only the
previous vectors. TimeCLR (Yeh et al., 2023) utilizes 1-D convolution to
vectorize the univariate sequence into a 2D embedding. ForecastPFN (Dooley et
al., 2024) constructs a token to include the time value and its temporal
attributes, like year, month, and day.

Channel-Mixing. The decomposition of multivariate sequences into multiple
univariate sequences may overlook the relationships and interactions between
different channels/variables. GTT (Feng et al., 2024) builds a multivariate
foundation model for time series forecasting. They reshape the channel
variables into the batch size to get a univariate sequence and slice it into
patches. During the inference stage, the model can accept varying channel
numbers since the channel variable has been fused into the batch size.

##  5\. Adapting Large Language Models for Time Series

Figure 5. There are two LLM adaption paradigms for time series: (a) Embedding
visible LLM adaption, which repurposes LLM to fit time series task with
redesigning LLM to perceive time series embedding directly. (b) Text visible
LLM adaption, which adapt time series task to fit LLM with formulating time
series input-output in a sentence-to-sentence manner.

After pre-training, LLMs acquire general abilities for various downstream
tasks, sparking increased interest in their application to time series
analysis. Effectiveness, efficiency, and explainability are all concerned
topics for LLM adaption toward time series. This section will concentrate on
examining the effectiveness of adapting LLMs to time series, with subsequent
sections dedicated to efficiency and explainability.

In NLP, “pre-train and fine-tune” paradigm used to be prevalent since 2017 .
This paradigm adapts LLM to downstream tasks by reprogramming pre-trained LLM
via objective engineering (Liu et al., 2023d). However, since 2021, ”pre-
train, prompt, and predict” paradigm has gained prominence (Liu et al., 2023d)
for the rapid expansion of Large Language Models (LLMs). This paradigm adapts
downstream tasks to LLM by reformulating their input-output in sentence-to-
sentence manner to activate LLM’s capacities, minicing NLP tasks. Inspired by
these two paradigms, we classify LLM adaption in the context of time series
analysis into two categories, namely embedding-visible LLM adaption and
textual-visible LLM adaption. These two paradigms mainly differ in input-
output manner, LLM utilization and the way to extract time series information
or multi-modal data fusion. There is a general comparison of these two
adaption paradigms in Figure 5.

The two paradigms above have contains an implicit assumption that LLM is
utilized as a forecaster to solve time series problem. However, the
functionality of LLMs extends beyond forecasting; they also serve as
enhancers, data generators and explainers. These roles greatly expand LLMs’
versatility across various applications.

In this section, we first introduce some key concepts of LLM adaptation to
facilitate understanding. Then we analyze each paradigm from multiple critical
stages, including time series extraction (namely time series representation,
semantic spaces alignment between time series and LLM, time series properties
and patterns recognition), multi-modal data fusion and LLM utilization through
fine-tuning or prompting. Finally, we discuss various roles of LLMs beyond
forecaster in solving time series problems.

###  5.1. Terminology

Here, we introduce two learning paradigms and some classical fine-tuning and
prompting techniques to equip readers with essential concepts for
understanding this section.

####  5.1.1. Zero-Shot and Few-Shot Learning

Both zero-shot and few-shot learning are promising learning paradigms, which
mimic human reasoning to quickly learn new tasks from past experiences with
few or even no samples. They can alleviate the heavy reliance on current AI
methods for large-scale data. Zero-shot learning, first proposed in CV
(Lampert et al., 2009; Palatucci et al., 2009), requires models to perform
tasks unseen during training (Wang et al., 2019b). Few-shot learning
introduced earlier (Fei-Fei et al., 2006), focuses on learning and
generalization from a limited number of examples (Wang et al., 2019a).

The zero-shot learning ability of LLMs was discovered in GPT-1 and further
developed in GPT-2 (Radford et al., 2019), albeit with moderate success. To
address this issue, GPT-3 (Brown et al., 2020) shifted focus towards few-shot
learning. With its massive scale and extensive pre-training data, GPT-3
demonstrates impressive few-shot performance in diverse NLP tasks without
specific fine-tuning. Subsequently, GPT-4 elevates the capabilities of zero-
shot/few-shot learning to a new level. Inspired by this success, the time
series community has also begun to explore and evaluate the zero-shot/few-shot
performance of LLMs in time series analysis.

####  5.1.2. In-Context Learning

In-context learning (ICL), introduced by GPT-3, is a widely used prompt method
in LLM applications. It combines task descriptions, test instances, and
demonstrations in natural language, allowing LLMs (particularly GPT series) to
execute tasks using only the provided context without further fine-tuning or
training. ICL exemplifies zero-shot/few-shot learning in LLMs. In essence, it
leverages the model’s existing knowledge and capabilities to perform new tasks
by carefully crafting prompts (Dong et al., 2022). Several works have
demonstrated that the effectiveness of ICL heavily relies on the instruction
design and demonstration designs (i.e. demonstration selection, order, and
format) (Zhao et al., 2021; Min et al., 2022). In addition, (Brown et al.,
2020) indicates that a model’s in-context learning capabilities improve with
increasing model size (Brown et al., 2020).

####  5.1.3. Chain-of-Thought

Chain-of-thought (CoT) is an advanced prompt strategy to enable the LLMs to
solve complex reasoning tasks (e.g. arithmetic reasoning (Miao et al., 2021),
symbolic reasoning (Wei et al., 2022b)). CoT can be adopted with in-context
learning under both few-shot and zero-shot settings. Few-shot CoT, introduced
in (Wei et al., 2022b), incorporates intermediate reasoning steps into ICL
prompt, transforming each demonstration from ¡input, output¿ to ¡input, CoT,
output¿ (Zhao et al., 2023b). Zero-shot CoT, proposed in (Kojima et al.,
2022), adds the phrase ”Let’s think step by step” to the end of ICL prompt.
CoT significantly improves large-scale models’ performance, marking an
emergent capability while being less effective for smaller models. Moreover,
CoT boosts model explainability by detailing the reasoning process, helping
users comprehend the model decision.

####  5.1.4. Instruction Tuning

Instruction tuning is an approach to fine-tune pre-trained language models by
performing tasks described by instructions (Wei et al., 2021; Lou et al.,
2023). It is a combination of the pretrain–finetune and prompting paradigms
(Liu et al., 2023d). This method enhances LLM’s instruction-following ability
through supervised fine-tuning, boosting zero-shot performance on unseen
tasks. Typically, an instruction-based prompt in the zero-shot setting
includes a task description (i.e., instruction) and an input-output pair,
while in the few-shot setting, it also incorporates additional exemplars (Zhao
et al., 2023b). In addition, (Wei et al., 2021) concludes that the design of
instruction, model scale, and variety of datasets influence the effectiveness
of instruction tuning.

In summary, both instruction tuning and in-context learning utilize natural
language to format the task. However, instruction tuning involves fine-tuning
LLMs for adaptation while ICL simply prompts LLMs for utilization (Zhao et
al., 2023b).

Model | Domain |  Task Type |  LLM Adaption |  Time Series Properties
---|---|---|---|---
TWSN (Xie et al., 2023b) | Finance | Class. | Text-Visible | Multivariate Dependency
TDML (Yu et al., 2023) | Finance | Class. | Text-Visible | Cross-sequence Dependency
LLMST (Zhang et al., 2023a) | Transportation | Anom. | Text-Visible | Cross-sequence Dependency
LLM-Mob (Wang et al., 2023c) | Transportation | Fore. | Text-Visible | Temporal Patterns
AuxMobLCast (Xue et al., 2022) | Transportation | Fore. | Text-Visible | Temporal Patterns
LLMFS (Liu et al., 2023c) | Healthcare | Class. | Text-Visible | -
LLMTIME (Gruver et al., 2023) | General | Fore. | Text-Visible | -
PromptCast (Xue and D.Salim, 2022) | General | Fore. | Text-Visible | -
FPT (Zhou et al., 2023b) | General | Fore. | Embedding-Visible | -
TEMPO (Cao et al., [n. d.]) | General | Fore. | Embedding-Visible | Temporal Patterns
UniTime (Liu et al., 2023a) | General | Fore. | Embedding-Visible | -
TEST (Sun et al., 2023b) | General | Fore./Class. | Embedding-Visible | Multivariate Dependency
LLM4TS (Chang et al., 2023) | General | Fore. | Embedding-Visible | -
TimeLLM (Jin et al., 2023a) | General | Fore. | Embedding-Visible | -
UMEF (Xue and Salim, 2023) | Other | Fore. | Text-Visible | -

Table 3. LLM Adaption for Time Series Analysis. There are three kinds of task,
i.e., Forecasting (Fore.), Classification (Class.), and Anomaly Detection
(Anom.)

###  5.2. Embedding-Visible LLM Adaption

The embedding-visible LLM adaption utilizes traditional ”pre-train and fine-
tune” paradigm to redesign LLM and fine-tune it towards downstream time series
tasks. Under this paradigm, LLM is redesigned to perceive time series
embedding directly instead of the traditional textual input.

####  5.2.1. Vectorized Time Series Representation

The first step to utilize LLM in the context of time series analysis is to
transform time series input into representations that can be perceived by LLM.
LLMs are based on transformer and require vector sequence as input. In NLP
tasks, each word (also called token) has its own vector embedding and sentence
composed of multiple words is a vector sequence to be fed into LLM. Under the
embedding-visible LLM adaption, time series input is transformed to vectorized
time series representation and the LLM is also redesigned to perceive the time
series embedding directly.

Inspired by the success of the patching strategy in transformer-based models
(Nie et al., 2022), many works slice the univariate time series into patches
and feed them into LLM as tokens (Cao et al., [n. d.]; Chang et al., 2023;
Zhou et al., 2023b; Jin et al., 2023a). Each single patch is a sub-sequence of
a time series, and it can retain meaningful local semantic information. The
patch length is fixed for any input length which can broaden the input scope.
As for the multivariate time series, many works decompose them into multiple
univariate time series and then adopt the patching tokenization technique. One
reason to support channel independence setting is that every channel acts
differently, and combining them all could reduce overall performance. However,
TEST (Sun et al., 2023b) vectorizes time series data under channel-mixing
settings with the argument that the cross-variable dependency should not be
neglected.

####  5.2.2. Semantic Spaces Alignment between Time Series and LLM

After transformation, time series can be represented as vector sequence to be
fed into LLM. However, these representations are likely to deviate a lot from
the LLM’s cognitive embedding space as there is modality gap between temporal
space and natural language space.

For those transform time series into temporal embedding sequence, they first
need to consider the dimension alignment. For example, GPT-2 of different
sizes has a unique text space with different embedding dimensions (Radford et
al., 2019). To align these two dimension space and enable LLM to understand
the time series embedding, existing research works usually redesign the input
embedding layer and fine-tuning it on downstream time series datasets. The new
embedding layer projects time series tokens to the required dimension of the
specific LLMs (Cao et al., [n. d.]; Zhou et al., 2023b; Chang et al., 2023).
Besides the modification of the input layer, some works explore the mapping of
time series features to the corresponding linguistic elements within LLMs.
Time-LLM (Jin et al., 2023a) considers that a frozen LLM without fine-tuning
can not directly comprehend patch embedding. They propose to reprogram the
embedding with text prototypes to align the source and target modalities.
Specifically, they utilize a small collection of pre-trained word embedding to
learn connecting language cues (e.g. ”short up”, ”long steady”) for
characterizing each patch and fuse them by trainable multi-head attention
layers. TEST (Sun et al., 2023b) also adopts a frozen LLM as backbone and
utilizes the text prototypes to constrain the embedding space of time series
based on contrastive learning.

####  5.2.3. Time Series Properties and Patterns Recognition

Besides filling the semantic gap of temporal modality and natural language
modality, another issue is to identify time series unique characteristics.
Time series data fundamentally differs from natural language in patterns and
properties, posing challenges in applying LLMs to the time series tasks. Time
series unique characteristics include multivariate and cross-sequence
dependencies, distribution shifts, and complex temporal patterns such as
seasonality, trends, and randomness. These elements are crucial for time
series modeling.

While transformer-based architectures have reached SOTA performance in certain
time series tasks, largely due to their capacity for modeling long-range
dependencies via self-attention, they often fail to account for the unique
properties of time series data. For instance, TEMPO (Cao et al., [n. d.])
highlights that transformers cannot inherently detect trends and seasonality.
Some recent findings demonstrate that time series transformers do not match
the robustness of their counterparts in CV and may even be outperformed by
linear models in specific benchmarks (Zeng et al., 2023). Furthermore, LLMs
are not intrinsically equipped with the knowledge and reasoning capacities to
identify time series properties and patterns during their pre-training phase.
From this perspective, it is worth our attention to consider these intrinsic
characteristics when activating LLMs for time series analysis.

Temporal Patterns. TEMPO (Cao et al., [n. d.]) decomposes each univariate
sequence into trend, seasonal, and residual components. They prove that such
decomposition significantly simplifies the prediction of LLMs by identifying
abnormal observations of seasonality or trends. In addition, they design a
learnable prompt pool to encode shared temporal patterns across different time
periods. The decomposed components are normalized, patched, and embedded
individually and then concatenated with the retrieved prompts before being fed
into a GPT module. In the experiments, they design variant models to validate
the effectiveness of the decomposition method and prompt pool.

AuxMobLCast (Xue et al., 2022) observes that the POI category is associated
closely with passenger patterns in human mobility forecasting. They integrate
an auxiliary POI classification module into the encoder-decoder architecture
to help it better identify different visiting patterns correlated with
different POI categories. The ablation study shows that the auxiliary module
achieves a substantial improvement with BERT encoder.

Multivariate Dependency. Many time series data are multivariate, e.g., stock
prices dataset and ECG dataset, while the text is univariate. Many studies
adapt LLMs towards time series partition multivariate time series into
multiple univariate sequences and process them individually (Zhou et al.,
2023b; Liu et al., 2023a). However, TEST (Sun et al., 2023b) argues that these
channel-independence works overlook the multivariate dependency in time
series. Indeed, variables of time series can be highly correlated or have
causal relationships, and such inter-dependencies are beneficial for modeling.
TEST perceives multivariate time sequence as input and slices it into tokens
at random lengths, which are then fed into an encoder to produce temporal
embedding.

Timestamp Information. Many studies have validated the benefits of
incorporating the timestamp information in transformer-based models (Wen et
al., 2022; Zhou et al., 2022b; Wu et al., 2021; Zhou et al., 2022a),
especially in those scenarios with explicit temporal patterns, e.g. disease
forecasting, traffic flow forecasting. LLM4TS (Chang et al., 2023) designates
the initial timestamp to each time series patch. They encode each time
attribute and fuse them into one temporal embedding, which is then summed with
token and positional embedding to produce the final embedding.

Distribution Shift. FPT (Zhou et al., 2023b) and TEMPO (Cao et al., [n. d.])
incorporate a normalization block to normalize the univariate input sequence
with reverse instance norm (RevIN (Kim et al., 2021)) before patching to
mitigate the distribution shift and further facilitate knowledge transfer.
LLM4TS (Chang et al., 2023) considers that trainable affine transformation of
RevIN is not suitable for autoregressive models like GPT-2. They adopt
standard instance normalization in the supervised fine-tuning stage.

####  5.2.4. Multi-modal Data Fusion

Multi-modal learning has been a well-established topic, especially in NLP and
CV, such as visual question answering (Li et al., 2023a), image-text
generation (Zhang et al., 2023b; Ramesh et al., 2021), and audio-text
generation (Wang et al., 2023b).Time series analysis also encounters multi-
modal scenarios. For instance, financial datasets can be enriched with textual
news and reports. Medical datasets may include relevant textual or image-based
supplementary information. Time series data is inherently numeric and
abstract, and similar temporal segments could have varying meanings in
different contexts, posing significant challenges in semantic mining and
knowledge transfer. Supplementing time series data with other multi-modal
information (e.g.textual descriptions) can assist the model in learning
complex temporal patterns, enhancing the representation ability,
generalization ability, and explainability for time series analysis.

Recently, inspired by the success of multi-modal LLMs in jointly leveraging
multiple knowledge bases, some works utilize the advanced pre-trained LLMs to
design multi-modal models for time series analysis. They can be divided into
two groups based on the granularity of multi-modal signals.

Sample-Level Multi-modal Fusion One research line utilizes multi-modal signals
to enrich details and complement internal knowledge to time series samples. To
be specific, the granularity of such signals is usually sample-level. METS (Li
et al., 2023c) feeds the ECG signals and its paired clinical report into a
multi-modal comparative learning framework. The report is processed by a
large, pre-trained medical language model, namely ClinicalBert, to guide the
training of the ECG encoder. In other words, ClinicalBert, acting like a
medical expert, summarizes the prior diagnostic knowledge from the ECG report
text and distills it into the ECG encoder to enrich the information of the ECG
signal. TEMPO (Cao et al., [n. d.]) incorporates contextual information like
quarterly news and reports for predicting quarterly financial indicators with
temporal embedding. They also train a soft prompt to extract the summary
information.

Task-Level Multi-modal Fusion Another research line leverages multi-modal
knowledge to improve the model’s generalization ability and facilitate
knowledge transfer across datasets. Typically, such knowledge pertains to the
task level or domain level. UniTime (Liu et al., 2023a) argues that there is
domain confusion when training a model on datasets from various domains. Since
the temporal patterns or distributions in different domains might show
significant differences, the model may have difficulties in discerning and
generalizing different datasets. They provide domain instructions to help the
model identify the data source and adopt the prediction strategy
correspondingly. The domain instructions are hand-crafted in sentence
descriptions to contain domain knowledge, which goes through a tokenizer and
an encoder, and finally fused with the temporal embedding. To enhance model’s
understanding of time series concepts, Time-LLM (Jin et al., 2023a) proposes
Prompt-as-Prefix, a technique to augment time series representation with a
prompt containing domain knowledge, task instruction, and data statistics. The
prompt is fed into a frozen LLM and then added at the top of the temporal
embedding, facilitating reasoning and pattern recognition of LLM.

####  5.2.5. Fine-tuning

Fine-tuning a pre-trained Large Language Model (LLM) is crucial for harnessing
its potential for specific downstream tasks. In embedding-visible LLM
adaption, the input/output layers and the objective function of the LLM are
reconfigured to suit the target task. This process involves updating all or
partial LLM parameters using thousands of supervised training samples. Once
fine-tuned, the model typically demonstrates enhanced performance on the
target tasks.

Among the time series works we investigate, UniTime (Liu et al., 2023a) adopts
the traditional fine-tune strategy on GPT-2 to learn a unified model for
cross-domain time series forecasting and finds that the fully tuning yields
the best performance compared with baselines. However, updating all the
parameters may disrupt the internal knowledge of LLMs obtained from pre-
training stage. The catastrophic forgetting may occur, where LLM loses its
previous capacities. Another drawback is the requirement of large training
datasets and heavy computational resources. For downstream tasks with small
datasets, fully fine-tuning LLM may have unstable performance or over-fit.

To alleviate these defects, some works leverage efficient tuning, aiming to
reduce the number of trainable parameters while retaining a good performance
as possible. FPT (Zhou et al., 2023b), TEMPO (Cao et al., [n. d.]) and LLM4TS
(Chang et al., 2023) all freeze the majority of the LLM parameters and only
update the minority ones in general time series forecasting. This method can
retain the major knowledge and capacities of LLM and require less data for
training. TEST (Sun et al., 2023b) and TimeLLM (Jin et al., 2023a) further
freeze all the parameters of LLMs and add learnable “soft prompt” parameters
to the input for fine-tuning LLMs to forecast time series. For those works
that utilize a frozen LLM, they need to align the time series embedding to the
textual space of LLM to make it understand time series representations. To
further enhance the reasoning capacities of LLM on time series data, some
works also re-purpose LLM by explicitly identifying the temporal patterns (Cao
et al., [n. d.]) or incorporating supplementary information from other
modalities (Jin et al., 2023a). The details will be discussed later in
subsections LABEL:subsec-ext and 5.2.4.

###  5.3. Text-Visible LLM Adaption

The text-visible LLM adaption follows the ”pre-train, prompt, and predict”
paradigm to redesign time series tasks and utilize prompting techniques to
activate the LLM capacities. Under this paradigm, the input-output pair of
time series tasks is reformulated into textual prompts.

####  5.3.1. Textual Time Series Representation

Under the text-visible LLM adaption, numerical time series data is transformed
into a string so that they can be seamlessly integrated into prompt as natural
language inputs (Jiang et al., 2024). Previous works that utilize LLMs to
infer directly on downstream tasks without any fine-tuning always format the
task in a sentence-to-sentence manner. Numerical values are transferred and
described as natural language sentences and integrated into the prompt along
with other contextual information. Some works fine-tune LLMs towards specific
downstream tasks and formulate tasks in natural language, for instance,
AuxMobLCast (Xue et al., 2022) for human mobility forecasting, LLMFS (Liu et
al., 2023c) for health tasks, PromptCast (Xue and D.Salim, 2022) for weather,
energy consumption, and customer flow forecasting.

####  5.3.2. Semantic Spaces Alignment between Time Series and LLM

For those transform time series into sentence, LLM tokenizes the strings into
words to comprehend them. However, (Spathis and Kawsar, [n. d.]) points out
that tokenization methods of LLMs are originally designed for words instead of
numerical values. They may separate the consecutive values into several tokens
and disregard the temporal meaning, which can significantly complicate
arithmetic (Liu and Low, [n. d.]). For example, GPT-3 (Brown et al., 2020)
tokenizer slices the number 42235630 into [422, 35, 630] tokens while LLaMA
(Touvron et al., 2023) tokenizes the number into individual digits by default
(Gruver et al., 2023). LLMTIME (Gruver et al., 2023) proposes to carefully
pre-process time series before they are tokenized, for example, adding space
technique for GPT series. (Spathis and Kawsar, [n. d.]) proposes that prompt
tuning could be a potential solution to bridge the gap between numerical data
and text. Prompt tuning involves adding a trainable embedding to the input,
which can be optimized to incorporate knowledge and aim at guiding the LLM to
comprehend time series information that may not be present in the original
pre-trained LLM.

####  5.3.3. Time Series Properties and Patterns Recognition

Different from time series characteristics extraction under embedding-visible
LLM adaption, text-visible LLMs identify time series unique properties and
patterns by integrating relevant information into prompt.

Temporal Patterns. LLM-Mob (Wang et al., 2023c) argues that it is difficult
for LLMs to capture useful information from complex raw stays directly for
human mobility prediction. They propose to decompose the mobility data into
historical sequence and context sequence to help LLMs better understand the
underlying long-term and short-term mobility patterns of passengers, which are
later utilized to form the prompt.

AuxMobLCast (Xue et al., 2022) observes that the POI category is associated
closely with passenger patterns in human mobility forecasting. They integrate
an auxiliary POI classification module into the encoder-decoder architecture
to help it better identify different visiting patterns correlated with
different POI categories. The ablation study shows that the auxiliary module
achieves a substantial improvement with BERT encoder.

Cross-sequence Dependency. To solve the cross-sequence dependency challenge in
stock movement forecasting, To address the challenge of cross-sequence
dependencies in stock movement forecasting, TDML (Yu et al., 2023) offers
numerous examples from stocks similar to the target stock when creating in-
context learning prompts. This approach illustrates that LLMs can effectively
incorporate cross-sequence information from related stocks. LLMST (Zhang et
al., 2023a) puts all the trajectories in a single prompt to observe whether
the model can consider the interaction between different trajectories for
human mobility detection. They conclude that the presentation of all the
trajectories in a single prompt may result in better performance.

Multivariate Dependency. TWSN (Xie et al., 2023b) leverages multiple
historical stock features like open, close, high, and low prices for stock
movement prediction. They transform these multivariate price features into a
tabular format string and integrate it into the textual prompt.

Timestamp Information. UMEF (Xue and Salim, 2023) incorporates the timestamp
into a pre-defined template prompt for energy consumption prediction, namely
”The electric load at {Time} is {Usage}.” LLM-Mob (Wang et al., 2023c)
considers the time information of the target stays in the prompt for human
mobility prediction. They also guide LLMs towards analytical thinking by
incorporating facts into the prompt that highlight how people’s mobility
patterns vary according to different times and days. AuxMobLCast (Xue et al.,
2022) also integrates the date information in its mobility prompting. It finds
performance reduction when removing the temporal date information from the
prompt. They explain that the timestamp is beneficial for the LLMs to capture
the temporal patterns from the numerical tokens.

####  5.3.4. Multi-modal Data Fusion

Compared to the embedding-visible LLM setting, there is relatively less
research on multi-modal learning in the text-visible LLM setting. As to sample
level multi-modal data fusion, TWSN (Xie et al., 2023b) analyzes ChatGPT’s
capabilities for stock movement prediction by constructing a multi-modal
prompt to combine historical stock price features with tweets from the same
days. When it comes to task-level multi-modal data fusion, the task-level
information can be added into the prompt as supplementary information to
augment the model performance.

####  5.3.5. Prompting

Motivated by LLMs’ powerful generalization across NLP tasks, many works
utilize prompting to activate LLM’s capacities on downstream time series
tasks. They integrate time series input into textual prompts to guide a pre-
trained LLM to generate desired output in natural language. It can be divided
into tuning free prompting and fine-tuning based prompting. The former
directly evaluates LLM performance on time series tasks under zero-shot/few-
shot settings without fine-tuning while the latter fine-tunes the LLM model
and update all or some of its parameters. We summarize each branch and discuss
its pros and cons.

(1) Tuning Free Prompting This branch relies only on prompts to utilize the
internal knowledge of LLM without any parameter update, and prompt engineering
is pivotal to improving model performance. Some prompts are designed as
instruction, and the instruction following capacity also influences the model
performance (Zhang et al., 2024b).

TDML (Yu et al., 2023) designs an instruction-based prompt to experiment with
GPT-4 in both zero-shot and few-shot inference. The prompt’s structure is
optimized empirically by splitting the instruction into two parts placed at
the start and end of the prompt. They also augment the prompt with the zero-
shot CoT technique by pretending, ”Can you reason step by step before
finalizing the output?” Surprisingly, they found that a zero-shot CoT prompt
can improve model performance notably.

TWSN (Xie et al., 2023b) evaluates the strengths and limitations of ChatGPT in
multi-modal stock price prediction with instruction prompt in zero-shot
inference. Unlike TDML, TWSN adds tweet information to its prompt. However,
the experiments show that ChatGPT underperforms the state-of-the-art methods.
Besides vanilla prompting, they also experiment with the CoT enhanced prompt.
They conclude that even the CoT technique can improve the model performance to
some extent but it is still worse than other specialized approaches.

LLMF (Liu et al., 2023c) tests the zero-shot performance of PaLM-24B on four
arithmetic health problems. They also construct an ICL prompt with three
demonstrations to evaluate the few-shot model performance. The findings
indicate that PaLM is not capable of zero-shot inferences and struggles to
numerical tasks, perhaps due to the inadequate representation of the target
health tasks in the PaLM training set. The 3-shot performance is slightly
better than zero-shot performance on most tasks, but PaLM is unable to
generate output for about 50% of test data in two health tasks under the
3-shot setting.

LLMST (Zhang et al., 2023a) constructs different prompts to evaluate the
effectiveness of several LLMs (i.e. GPT-3.5, GPT-4 and Claude-2) for human
mobility anomaly detection. They form the prompt by incorporating trajectory
sequence along with task instruction. They then supplement the prompt with
indicative clues about potential anomalies to test their effectiveness. Their
conclusion is that LLM demonstrates reasonable performance on detecting
anomaly mobility behavior. They also prepend ”Give your analysis” in the
prompt to improve the transparency of model decisions.

LLM-Mob (Wang et al., 2023c) leverages the textual understanding and reasoning
capabilities of GPT-3.5 for human mobility data analysis by carefully designed
prompt engineering. They first manually decompose the raw mobility data into
two separated sequences and incorporate them with particular instructions in
the prompt to enable LLM better identify mobility patterns. They also give
clear instructions and provide prior knowledge to inform LLM of the activity
patterns. They believe that these operations can guide the model to think
logically and serve as CoT function.

LLMFS (Liu et al., 2023c) examines the current LLM’s performance on health
tasks under zero-shot and few-shot in-context learning. They demonstrate that
the PaLM with 24 billion parameters can digest time-series health data under
few-shot learning setting and make meaningful inferences. But the performance
falls significantly in zero-shot ICL, which proves that LLM struggles with
arithmetic tasks without any contextual information.

All the works above require careful design of the context-inclusive prompts,
which integrate numerical time sequence with supplementary contextual
information (e.g., task descriptions). However, LLMTIME (Gruver et al., 2023)
shows that LLM can be directly utilized as a zero-shot forecaster with a
numerical-only prompt, which purely contains the numerical data without any
additional textual information. They argue that careful pre-processing of the
time sequence is the key to fitting the tokenization strategy of LLM,
influencing LLM to learn from meaningful patterns within the tokenized
sequences.

(2) Fine-tuning based Prompting The tuning free prompting is inflexible for it
can only leverage the inherent knowledge of LLM which might be limited for
specific time series downstream task. Some works combine traditional fine-
tuning and prompting to update parameters of LLM to make it more suitable for
time series tasks.

PromptCast (Xue and D.Salim, 2022) is the first to adopt instruction tuning
for general time series forecasting. They design a zero-shot instruction-based
prompt in a question-answer format for three forecasting tasks, i.e., weather
temperature, energy consumption forecasting, and customer flow forecasting.
The prompt has a template that contains a context part providing historical
time series sequence and timestamps, a question part for queries about the
future steps, and an answer part providing ground truth to respond to the
question. UMEF (Xue and Salim, 2023) adopts the instructionless tuning for
energy consumption forecasting. A template is pre-defined to convert energy
consumption data into descriptive sentences, like ”The electric load at {Time}
is {Usage}.” They evaluate the effectiveness of three LLMs, i.e., BART,
Bigbird, and Pegasus, for energy load forecasting. The experiments demonstrate
that LLM outperforms traditional numerical forecasting in energy consumption
prediction. AuxMobLCast (Xue et al., 2022) also utilizes instructionless
tuning for human mobility forecasting. They design a mobility prompt to
combine the mobility data, timestamp, and POI information in the input
sentence and integrate the target visitors’ number in the answer sentence.
They fine-tune a LLM-based encoder-decoder architecture. The encoder (e.g.,
BERT) extracted interaction between mobility data and other clues to produce
prompting embedding for the decoder. GPT-2 is utilized as the decoder for
final prediction. They also design different prompts to find out the most
effective combination of the input elements. LLMFS (Liu et al., 2023c) designs
a question-answer based prompt for various health tasks. Different from the
above works updating all the parameters of LLM, LLMFS freezes the whole LLM
and adds a soft learnable prompt embedding as a prefix for each downstream
task to enable LLM to comprehend time series data from different tasks.

In summary, the key contribution of studies utilizing prompting lies in their
intricate prompt designs tailored to specific time series scenarios. To
enhance prompt effectiveness, some studies integrate additional information,
such as time series characteristics (Wang et al., 2023c), alternative data
modalities (Xie et al., 2023b), and expert knowledge (Zhang et al., 2023a).
Techniques like chain-of-thought (CoT) have been applied in several works (Yu
et al., 2023; Xie et al., 2023b; Wang et al., 2023c; Zhang et al., 2023a),
showing promise in boosting model performance.

###  5.4. Further Discussion: Functionality of LLMs

Model | Predictor | Enhancer | Data Generator | Explainer
---|---|---|---|---
TEMPO (Cao et al., [n. d.]) | GPT2 | - | GPT4 | No
PromptCast (Xue and D.Salim, 2022) | 10 LLMs | - | - | No
LLMTIME (Gruver et al., 2023) | GPT3/LLaMA2-70B | - | - | No
TEST (Sun et al., 2023b) | GPT2/BERT/ChatGLM/LLaMa2 | - | - | No
FPT (Zhou et al., 2023b) | GPT2 | - | - | No
LLM4TS (Chang et al., 2023) | GPT2 | - | - | No
TimeLLM (Jin et al., 2023a) | LLaMA-7B | LLaMA | - | No
UniTime (Liu et al., 2023a) | GPT2 | - | - | No
CIGN (Chen et al., 2023b) | - | - | ChatGPT | No
LLMFS (Liu et al., 2023c) | PaLM | - | - | No
METS (Li et al., 2023c) | - | ClinicalBert | - | No
UMEF (Xue and Salim, 2023) | BART/Bigbird/Pegasu | - | - | No
AuxMobLCast (Xue et al., 2022) | GPT2 | - | - | No
TWSN (Xie et al., 2023b) | ChatGPT | - | - | CoT
TDML (Yu et al., 2023) | GPT4/LLaMA-13B | - | GPT4 | CoT
LLMST (Zhang et al., 2023a) | GPT3.5/GPT4/Claude2 | - | - | Yes
LLM-Mob (Wang et al., 2023c) | GPT3.5 | - | - | Yes
Table 4. Functionality and backbones of LLMs in investigated works.

Most research utilizes LLM as a core module for prediction in the context of
time series analysis. However, the functionality of LLM extends beyond
predictor. It varies from processing other modality data, producing raw data
for specific tasks, and providing a reasoning process or explanation for their
predictions. The diverse applications of LLMs enhance solutions for time
series tasks and unlock their potential from multiple perspectives. In this
section, we discuss the diverse roles of LLMs in current research (as shown in
Table 4).

####  5.4.1. Predictor

Specifically, if an LLM mainly takes the representations of temporal data as
input and produces the output, it is treated as a predictor. All the works
analyzed in subsection 5 adopt LLM as a predictor, i.e., the core module for
time series tasks. The differences among these works lie in the tunability of
LLM, the time series tokenization, the prompting techniques, and some enhanced
model designs for different time series tasks. More details have been
introduced in the previous subsections.

####  5.4.2. Enhancer

If an LLM is in charge of extracting high-level information only from other
modality data without time series data and assisting the core module to
predict, we identify it as an enhancer. As analyzed in subsection 5.2.4, the
multi-modal signal can be classified into sample-level and task-level,
providing supplementary information in different granularities to enrich the
representation of time series data.

METS (Li et al., 2023c) utilizes ClinicalBert to process ECG-paired medical
reports for ECG classification. TWSN (Xie et al., 2023b) adopts ChatGPT to
incorporate stock-level tweets with daily stock price for stock movement
prediction. TEMPO (Cao et al., [n. d.]) provides quarterly news and reports
for forecasting quarterly financial indicators by feeding it into GPT2.
TimeLLM (Jin et al., 2023a) and UniTime (Liu et al., 2023a) leverage domain
knowledge described in sentences to help LLMs identify sources of time series
data, facilitating the recognition of different temporal patterns in different
domains.

Note that all the multi-modal signals in the existing works are textual
information currently, like reports, news, and domain knowledge in short
sentences. Different from the predictor LLMs, the enhancer LLMs are usually
frozen and do not require training or fine-tuning. Furthermore, the enhancer
may consist of a specialized LLM, such as ClinicalBert, or a general LLM, such
as GPT2. Intuitively, a specialized LLM is a better choice for domain-specific
information extraction if it is available.

####  5.4.3. Data Generator

Drawing inspiration from the vast knowledge possessed by LLMs and their robust
capabilities in language generation and contextual understanding, some studies
employ LLMs to generate additional data needed to complement the input.

CIGN (Chen et al., 2023b) considers that ChatGPT has learned massive
information about financial entities (e.g., stocks, companies, events) during
training, and it is an ideal tool to extract the inter-dependencies among
stock equities. They feed financial news headlines into ChatGPT to produce
affected company sets. Then, they construct a graph of these companies. The
company graph is then utilized as model input for stock movement prediction.
TDML (Yu et al., 2023) utilizes GPT-4 to produce company descriptions and
general positive/negative factors for each stock. This textual information is
utilized with stock prices for prediction. They also use GPT-4 to extract a
condensed summary and keywords from crawled news articles of each stock, which
is utilized as the ground truth to evaluate the quality of the explanations
generated by GPT-4. TEMPO (Cao et al., [n. d.]) adopts ChatGPT to collect news
and reports for each company in a specified duration and these data will be
treated as contextual information for financial indicator prediction.

In summary, large language models (LLMs) are able to generate the desired data
through a well-designed prompt. They can also be utilized to preprocess or
refine the raw data. Generating data with LLMs can be more efficient and cost-
effective than some traditional methods of data collection or annotation.
Furthermore, the LLMs utilized as data generators in existing studies are
high-quality and advanced models, such as ChatGPT, all of which are employed
to generate textual information in the finance domain. However, it is worth
noting that the data generated by LLMs is not always reliable or accurate for
the hallucination challenge.

####  5.4.4. Explainer

If an LLM generates an explanation for its output, it is regarded as an
explainer. The explainability of model behavior or prediction is important in
time series analysis, especially in those high-stakes scenarios.

While LLMs are often considered black boxes with limited explainability, their
proficiency in text generation allows them to serve as natural explainers,
generating human-readable explanations for their decisions. Furthermore,
augmented with prompting techniques like Chain-of-Thoughts (CoT), LLMs can
reveal their reasoning process step by step and provide detailed explanations
for the predictions.

LLMST (Zhang et al., 2023a) utilizes ChatGPT to support anomalous human
mobility detection. LLM-Mob (Wang et al., 2023c) also explores the
explainability of how GPT-3.5 predicts human mobility. TWSN (Xie et al.,
2023b) designs a CoT enhanced prompt to enable ChatGPT to provide cogent
explanations for results. TDML (Yu et al., 2023) also investigates the GPT-4’s
potential for explainable stock price forecasting. More details about model
explainability can be found in Section 7.

##  6\. Efficiency

In deep learning, efficiency (Menghani, 2023) refers to the effectiveness of
computational resources utilization. It involves multiple aspects, e.g.
computation time, processor utilization (CPU or GPU), memory requirements, and
energy consumption. Both pre-trained time series foundation models and LLM-
centered models for time series have exhibited powerful generalization
ability, surpassing many task-specific models. However, these advanced
capabilities are accompanied by intensive resource consumption. For example,
when using a large language model (such as GPT-3) for time series data, the
training and inference processes can be very time-consuming due to the
complexity and large number of parameters of the model. This may lead to the
need to consider efficiency issues when using large language models for time
series analysis in real-time applications. Given that time series foundation
models are still in the early stages of development, the efficient methods
employed in time series domain are derived from NLP. we first summarize the
efficiency techniques for LLMs that can be transferred to time series
analysis, followed by efficiency discussion on existing works with different
tuning paradigms.

###  6.1. Efficient Techniques in Large Language Models

large language models (LLMs) have expansive parameter size (e.g.billions or
trillions), leading to significant computational and memory demand (Wan et
al., 2023). Efficiency is increasingly crucial for large language models
(LLMs) and can be improved through the entire lifecycle of LLM, including
input handling, architecture design, model compression, pre-training, fine-
tuning, and inference (Wan et al., 2023).

Since fine-tuning the whole LLM for downstream tasks is inefficient as it
always updates all the LLM parameters, requiring huge computation resources.
Many works utilize parameter-efficient fine-tuning (PEFT) (also named light-
weight fine-tuning). PEFT solves the infeasibility and impracticality of fully
fine-tuning LLM. It freezes the majority of LLM parameters and only trains a
small set of parameters which might be a subset of existing model parameters
or newly added parameters (Lialin et al., 2023). We introduce some popular
PEFT techniques that have been applied or can be potentially applied in LLM-
centered time series research, including adapters, prompt tuning, prefix-
tuning, Low-Rank Adaptation (LoRA).

Adapters. The idea of adapters was initially proposed in CV (Rebuffi et al.,
2018) and then adapted to NLP (Houlsby et al., 2019). It injects small-scale
neural networks after attention and feedforward neural network (FNN) layers in
the transformer. Adapters might have only less than 4% of the model parameters
but they show competitive performance to full fine-tuning (Houlsby et al.,
2019). There are various adapters, like SparseAdapter (He et al., 2022b), MAM
Adapter (He et al., 2021), Parallel Adapter (He et al., 2021).

Prompt Tuning. Unlike the adapters that add extra FFN layers, prompt tuning
(Lester et al., 2021) wraps a trainable tensor to the model input embedding,
commonly referred as ”soft prompt”. Prompt tuning becomes more effective with
larger model size, and its efficiency improvement accelerates at a rate faster
than the growth of model size (Su et al., 2021).

Prefix-Tuning. To directly optimize the soft prompt in prompt tuning is
observed to be unstable in (Li and Liang, 2021). Different from prompt tuning
prepending soft prompt to the input layer, prefix-tuning (Li and Liang, 2021)
adds a trainable vector (namely prefix token) to each intermediate layer. It
is shown to have a close performance to full fine-tuning by only updating 0.1%
parameters of BART (Li and Liang, 2021).

Low-Rank Adaptation (LoRA). Inspired by the idea of Intrinsic SAID (Aghajanyan
et al., 2021) to adopt low-rank fine-tuning, LoRA (Hu et al., 2021) decomposes
the weight matrix of LLMs into a product of two low-rank matrices. During the
fine-tuning process, the original matrix is frozen while the two low-rank
matrices are updated. Although LoRA is effective, it necessitates updating the
low-rank matrices for all layers of the LLMs in every iteration. Note that
While LoRA is a reparameterization-based method, the remainder are additive
methods (Ding et al., 2022).

###  6.2. Efficiency in LLM Adaption

In this subsection, we discuss the efficiency issues when adapting LLM towards
time series analysis. We first provide a tuning paradigm taxonomy for existing
works based on the updated parameter amount of their LLM-centered
architectures. The taxonomy has three branches, i.e. tuning free paradigm
without any updated parameters, fully fine-tuning paradigm which updating all
the parameters of LLM backbone and parameter-efficient fine-tuning which
updating only a small set of model parameters.

####  6.2.1. Tuning Free Paradigm

In the development of LLM-centered time series solutions, the most efficient
strategy is to directly call API interfaces (Gruver et al., 2023; Zhang et
al., 2023a; Wang et al., 2023c), without any updating of model parameters,
i.e. tuning free paradigm. For instance, TWSN (Xie et al., 2023b) and TDML (Yu
et al., 2023) explore the zero-shot inference of ChatGPT/GPT-4 in multimodal
stock movement prediction. No parameter is updated under this strategy. This
approach does not alter the model’s structure. Instead, it heavily depends on
the inherent pattern recognition and reasoning capabilities of LLMs, along
with prompt engineering, to enhance model performance. However, recent
research shows that this strategy may lead to poorer performance of the model
on some tasks. For instance, TDML (Yu et al., 2023) observes that LLM
underperforms not only SOTA methods but also some traditional methods like
linear regression. In addition, it is also expensive to call ChatGPT API for a
large amount of input.

####  6.2.2. Full Fine-Tuning Paradigm

Due to the inferior performance of tuning free paradigm, some works choose to
fine-tune LLM to adapt it for downstream time series tasks. A fully fine-
tuning strategy usually redesigns the input layer to encode time series data
and the output layer to adapt to downstream tasks. This strategy can better
guide LLMs to perform on a specific time series task, like AuxMobLCast for
human mobility prediction (Xue et al., 2022), UniTime (Liu et al., 2023a) and
TEMPO (Cao et al., [n. d.]) for general forecasting. However, it is usually
computationally intensive and requires a longer training time, as it involves
updating all the parameters of a large language model.

####  6.2.3. Parameter-Efficient Fine-Tuning Paradigm

To strike a good balance between effectiveness and efficiency, more works
adopt a parameter-efficient fine-tuning (PEFT) strategy.

FPT (Zhou et al., 2023b) retains a significant portion of the pre-trained
knowledge by freezing the major parameters of GPT-2, specifically the self-
attention and feedforward layers. It redesigns the input layer and retrains it
with the positional embedding layers and normalization layers on varied time
series datasets to activate LLM’s capacity for downstream tasks. Similar to
FPT (Zhou et al., 2023b), TEMPO (Cao et al., [n. d.]) also freezes and updates
the same layers in the GPT block. Furthermore, it utilizes LoRA (Hu et al.,
2021) within the multi-head attention block for model compression, thereby
facilitating efficient adaptation to diverse time series distributions.
Drawing inspiration from supervised fine-tuning in the chatbot domain, LLM4TS
(Chang et al., 2023) introduces a dual-phase fine-tuning approach. The initial
phase employs techniques similar to those used in TEMPO (Cao et al., [n. d.]),
such as partial freezing and LoRA. The distinction lies in the subsequent
phase, which allocates the first half of the epochs to linear probing and the
remainder to comprehensive fine-tuning.

All the works above only freeze the majority of LLM parameters and leave the
remainder tunable. To further alleviate the challenges of insufficient data
and limited computational resources, some works choose to employ a completely
frozen LLM (Sun et al., 2023b; Jin et al., 2023a; Liu et al., 2023c) and
introduce a small number of new parameters for fine-tuning. TEST (Sun et al.,
2023b) designs a soft prompt (i.e., trainable task-specific embedding vectors)
concatenated with time series embedding to instruct LLM on subsequent time
series tasks efficiently. TimeLLM (Jin et al., 2023a) reprograms the input
layer by aligning the trainable patch embedding with text prototypes. LLMFS
(Liu et al., 2023c) attaches a soft learnable embedding as a prefix to each
sample in the prompt to make LLM comprehend the domain knowledge in time
series data.

In conclusion, LoRA has been utilized in TEMPO (Cao et al., [n. d.]) and
LLM4TS (Chang et al., 2023). Similarly, prompt tuning is employed in TEST (Sun
et al., 2023b), TimeLLM (Jin et al., 2023a), LLMFS (Liu et al., 2023c).
However, specific strategies for efficient architecture design and model
compression have yet to be explored in LLM-centered time series analysis.

##  7\. Explainability

Explainability or explainability refers to how well humans can understand the
behaviors or predictions of a model, which has always been an important
problem for both AI and time series communities (Rojat et al., 2021). Making
the model transparent and explainable can enable users to understand,
appropriately trust and effectively manage the model deployment in real-world
scenarios (Du et al., 2019). In some contexts where the competent model
performance can meet most requirements (e.g. text generation), the lack of
explainability might not be a major issue. However, for those critical
decision-making domains where user trust, safety, fairness, and privacy are of
utmost importance, like automatic driving, healthcare and finance (Bracke et
al., 2019), the absence of explainability will hamper the practical model
applications. In fact, the benefits of explainability extend far to knowledge
discovery (Du et al., 2019), model debugging (Du et al., 2019), and solving
disagreements between AI systems and human experts (Ferreira and Monteiro,
2020).

In this section, we aim to clarify what to explain and how to explain in the
context of large model based time series analysis. Although the complexity of
large models greatly exceeds general AI methods, they are essentially black-
box AI models. The explanatory methods and principles applicable to other AI
models can be adapted to these large models. Therefore, we start with a brief
introduction to the research background in AI explainability. We then proceed
to summarize and analyze the contributions of existing research in enhancing
the explainability of large models for time series analysis.

###  7.1. Explainability of AI Models

While certain models, such as decision trees, inherently offer explainability
due to their straightforward structures, most models, especially those within
deep learning, are black-box in nature, obscuring their inner working
mechanism from users (Zhao et al., 2023a). To address this challenge, the
field of Explainable AI (XAI) (Arrieta et al., 2020) focuses on developing
methods that make the internal mechanisms or outcomes of deep learning models
comprehensible to humans. Research in XAI can be broadly categorized into two
approaches: local and global explanations (Du et al., 2019). Local
explanations (Plumb et al., 2018) aim to clarify the reasoning behind the
model’s decision for a specific instance, seeking to reveal the causal
relationships between a given input and its outcome. In contrast, global
explanations (Ibrahim et al., 2019) strive to shed light on the model’s
overall internal mechanisms, examining its structure and parameters across all
instances. The explanation techniques can be also divided into ante-hoc and
post-hoc methods (Rojat et al., 2021). Ante-hoc method (Wang et al., 2020a;
Rudin, 2019) directly incorporates the explainability of the model structure
while the post-hoc method (Guidotti et al., 2018) focuses on explaining the
model behaviors without changing its underlying structure.

###  7.2. Explainability of LLM-centered Time Series Models

Here we delve into the explainability of time series analysis with LLM,
involving both local and global explanation.

####  7.2.1. Global Explanation

The global explanation refers to understanding what LLMs have learned and how
they can effectively interpret numerical time series. On one hand, LLM lacks
of built-in explainability, and its internal mechanisms are still unclear
(Zhao et al., 2024), posing challenges to explain its adaption to time series.
On the other hand, the current modification of LLM structure towards time
series data concerns about performance improvement instead of explainability.
Fortunately, there is one work providing explanation on the adaption of LLMs
to time series anlaysis. FPT (Zhou et al., 2023b) proves that the behaviors of
the self-attention module in LLMs is similar to principle component analysis
(PCA). It can perform task-agnostic operations as a general computation
calculator on time series tasks.

####  7.2.2. Local Explanation

Compared with other black-box deep learning models, LLMs can generate
reasonable natural language explanations for its output guided by the chain-
of-thought prompt. Several studies have examined LLM’s ability to produce
local explanations.

When activating LLMs for detecting anomalous human mobility, LLMST (Zhang et
al., 2023a) prompts GPT-3.5 and GPT-4 to provide supporting analysis for their
detections by including an instruction in the prompt:”Give your analysis…”.
They find that LLMs are capable of providing persuasive explanation for their
decisions. Mob (Wang et al., 2023c) also structures its prompts to request
explanations, using an instruction ”Please organize your answer around the key
reason (a concise explanation supporting your prediction).” They argue that
this approach not only improves the explainability of the results but also
enhances the model’s reasoning capabilities.

Explainable predictions also play a vital role in numerous financial
applications, enhancing decision-making by ensuring reliability and
transparency. In stock movement prediction, TWSN (Xie et al., 2023b) utilizes
a chain of thought enhanced zero-shot prompt to enable ChatGPT to deliver
coherent explanations for its predictions. Guiding by the zero-shot CoT
”explain your predictions step by step”,” ChatGPT can provide detailed
insights into its stock movement forecasts. However, it’s noted that ChatGPT’s
explanations are constrained by its concentration on a narrow range of input
factors, whereas the stock market’s dynamics are much more intricate. TDML (Yu
et al., 2023) also explores the potential of LLM for explainable stock price
forecasting. They prompt GPT-4 to predict the summary and keywords for the
following week, which subsequently serve as explanations for the predicted
stock returns during that period. Additionally, they experiment with the CoT
technique by appending an instruction at the end of the prompt: ”Can you
provide step-by-step reasoning before finalizing the output?” They found that
this method enhances GPT-4’s ability to generate detailed and coherent
reasoning steps, leading to improved model performance in most experiments.

All these works above utilize GPT-3.5 and GPT-4 to generate explanations. No
work has yet used smaller LLMs like GPT-2 (Radford et al., 2019) for
explanation generation, possibly because their language generation and
reasoning capabilities are still lacking. For those embedding-visible LLMs,
the CoT prompt can not be utilized to generate an explanation in natural
language. TEMPO (Cao et al., [n. d.]) is the first to introduce a post-hoc
method for local explanation of an embedding-visible LLM (GPT-2), namely. They
propose to build a generalized additive model to measure the contributions of
different components (i.e. trend, season, and residual) based on GPT’s output.

##  8\. Domain Taxonomy

Time series analysis has a broad spectrum of applications across various
domains. This section provides a domain taxonomy to classify research into
general, traffic, healthcare, finance, and other domain. By examining the
motivations, contributions, and limitations of studies within each domain, we
intend to provide a clear road map of the advancement of foundation models in
each domain.

###  8.1. General Domain

While specialized methods still dominate in time series analysis, general-
purpose models like ChatGPT and CLIP (Radford et al., 2021) have achieved
considerable success in a wide range of downstream NLP and CV tasks. These
universal task solvers markedly diminish the need for extensive data
collection, intensive model design, and trivial task-specific training, thus
greatly attracting attention from time series community.

(1) Foundation Model Pre-trained for Time Series. To develop a universal model
for time series analysis, one research line focuses on pre-training a
foundational model with time series data which serves as a domain-agnostic
solver. ForecastPFN (Dooley et al., 2024) is the first transformer-based
network trained purely on synthetic time series data for zero-shot
forecasting. TimeGPT (Garza and Mergenthaler-Canseco, 2023) is the largest
transformer-based foundation model pre-trained on more than 100B data points.
Lag-Llama (Rasul et al., 2023) is a decoder-only foundation model trained on
Monash repository for univariate probabilistic time-series forecasting. GTT
(Feng et al., 2024) is an encoder-only foundation model pre-trained on 200M
samples for zero-shot multivariate time series forecasting. TimesFM (Das et
al., 2023) is a patched-decoder pre-trained on Google Trends for univariate
time series forecasting. All these foundation models have a certain degree of
zero-shot capacity on unseen tasks. However, due to the lack of large-scale
time series data, they are still limited in emergent abilities compared with
LLMs.

(2) LLM Adaption for Time Series. Another research line concentrates on
customizing LLMs for time series tasks. FPT (Zhou et al., 2023b) utilizes
parameter-efficient fine-tuning on GPT-2 and evalutes it on all major types of
time series tasks including forecasting, classification and anomaly detection.
They also offer a theoretical rationale for the successful application of LLMs
to time series data. They contribute such success to the self-attention module
behaviors of LLM similar to principle component analysis (PCA). LLM4TS (Chang
et al., 2023) employs a lightweight, two-stage fine-tuning approach for GPT-2,
drawing inspiration from the methods used in supervised fine-tuning within
chatbot applications. The initial stage involves autoregressive supervised
fine-tuning to acquaint the LLM with the general patterns of time series
forecasting. The second stage implements a downstream fine-tuning to guide LLM
towards specific application. Different from fine-tuning an embedding-visible
LLM in (Zhou et al., 2023b; Chang et al., 2023), PromptCast (Xue and D.Salim,
2022) is the first to fully fine-tune a text-visible LLM with carefully
designed prompt.

TimeLLM (Jin et al., 2023a) and TEST (Sun et al., 2023b) consider that the
successful transferability of LLM for time-series forecasting lies in
effectively aligning modalities of time series data and text. They both focus
on reprogramming the input layer of frozen LLM. More specifically, TimeLLM
(Jin et al., 2023a) reprograms the input time series with text prototypes to
directly bootstrap LLM for understanding time series. It also adds a prompt as
prefix of input to enhance the LLM’s adaptability to downstream tasks. TEST
(Sun et al., 2023b) utilizes contrastive learning to produce similarity-based,
instance-wise, feature-wise, and text-prototype-aligned embeddings as input
for LLM to understand. A soft prompt is then designed to instruct LLM on
subsequent time series tasks. Furthermore, LLMTIME (Gruver et al., 2023) finds
that LLM under frozen setting can show impressive performance with purely
numerical sequence input after carefully pre-processing. This success is
attributed to the LLM’s affinity for simple or repetitive sequences, aligning
well with the inherent structure of time series data. The study also reveals
that the method of number tokenization in LLMs significantly impacts their
forecasting effectiveness.

UniTime (Liu et al., 2023a) identifies the challenges to learn a unified model
for cross-domain time series forecasting while handling varying domain-
dependent characteristics of time series data, e.g. different temporal
patterns or distributions, diverse convergence rates. They utilize GPT-2 as
the backbone to align time series data and domain instruction to solve the
domain confusion. In addition, they adopt masking to mitigate the domain
convergence speed imbalance. TEMPO (Cao et al., [n. d.]) focuses on activating
GPT-2 to fully capture the temporal pattern evolution and interrelated
dynamics of time series data. They first model specific time series patterns
by trend, seasonality, and residual decomposition and then encode more
universal temporal patterns from historical data in a shared prompt pool.
These inputs enable more effective fine-tuning of LLM towards time series
modeling.

###  8.2. Finance

The textual and time series data are two major modalities in the financial
realm. Large language models have demonstrated great potential in financial
text mining, e.g. pre-trained financial foundation models, financial sentiment
analysis (Lan et al., 2023), financial text mining (Liu et al., 2021),
investment portfolio selection (Romanko et al., 2023). However, their
application in financial time series data remains in the early stages of
development.

The stock movement prediction is one of the multimodal financial applications
that relied on historical prices along with textual information, where some
researchers tried to unlock LLMs’ capacities from different perspectives. CIGN
(Chen et al., 2023b) contributes ChatGPT’s success to its extensive knowledge
of entities (e.g. companies, events, individuals) and their relationships.
They offer financial news headlines to ChatGPT to generate relationships among
the listed companies of DOW 30, which are subsequently utilized to construct
company graphs for stock movement prediction. They argue that such
automatically latent relationship extraction may be more efficient than
handcrafted features.

TWSN (Xie et al., 2023b) provides historical stock prices and tweets data to
ChatGPT to test its zero-shot learning capacities in stock movement
prediction. They find that ChatGPT shows inferior performance compared with
both deep learning and traditional linear regression. They also observe its
limitations of explainability and stability. Compared with TWSN (Xie et al.,
2023b), Yu et al. (2023) draw a nearly opposite conclusion about LLM’s
performance. They focus on NASDAQ-100 stocks with price time series and
utilize GPT-4 to produce company metadata and preprocess financial news as
supplementary input. They test GPT-4 under zero-shot, few-shot with/without
chain-of-thought (CoT) settings and find it outperforms the widely applied
classic ARMA-GARCH model. They also find that CoT can enhance LLM’s
performance and provide human-readable reasoning over input. Moreover, they
fine-tune LLaMA to demonstrate the feasibility of open source LLMs for this
task.

###  8.3. Traffic

Time series sequences are the primary data type for numerous traffic-related
applications, including traffic flow prediction and human trajectory
detection. While some applications also utilize tabular data, textual
information is rarely encountered in traffic-related tasks. The current works
exploring LLMs’ potential in traffic field concentrated on human mobility
forecasting or detection. In Mob, (Wang et al., 2023c) implements an in-
context learning process to harness the textual understanding and reasoning
capabilities of GPT-3.5 for human mobility data analysis. They decompose the
raw stay sequence into historical stays and context stays and integrate them
into instruction based prompt to help LLM understand the short-term and long-
term dependencies in human movement. LLMST (Zhang et al., 2023a) also employ
GPT-3.5 and GPT-4 to detect anomalous behaviors from mobility data. They
develop various prompts to evaluate the impact of specific indicators of
anomalies and the interactions among distinct trajectories. AuxMobLCast (Xue
et al., 2022) also designs different prompts to find out the most effective
combination of mobility data, timestamp, and POI information. But different
from the frozen LLM configuration in (Wang et al., 2023c; Zhang et al.,
2023a), AuxMobLCast (Xue et al., 2022) is the first work to fine-tune a LLM
based encoder-decoder architecture. The encoder (e.g. BERT) extracted
interaction between mobility data and other clues to produce prompting
embedding for decoder. GPT-2 is utilized as the decoder for final prediction.
They also integrate an auxiliary POI classification module in the architecture
to provide LLM with traveling pattern information for better volume
prediction.

###  8.4. Healthcare

Some applications in the health domain can collect time series data from
wearable and medical sensor recordings, like activity recognition and disease
diagnosis. LLMF (Liu et al., 2023c) examines the current LLM’s performance on
health tasks with a limited number of training examples. They demonstrated
that the PaLM with 24 billion parameters can digest time-series health data
under few-shot learning setting and make meaningful inferences. METS (Li et
al., 2023c) is the first work to use LLM as an enhancer for ECG multimodal
self-supervised learning. Specifically, they utilized a large clinical
language model, named ClinicalBert, to analyze the ECG reports and direct the
training of the ECG encoder via contrastive learning. During the inference
phase, they expanded the ECG labels into medical diagnostic sentences and
input these into ClinicalBert to acquire embeddings for ECG classification.

###  8.5. Other Domains

Motivated by the LLM’s ability to learn rich representations of textual data,
UMEF (Xue and Salim, 2023) formulates the energy load forecasting in a
sentence-to-sentence form. A template is pre-defined to convert energy
consumption data into descriptive sentences, like ”The electric load at {Time}
is {Usage}”. LLM is then fully fine-tuned using prompting and an
autoregressive prediction mechanism focused on the hourly energy consumption
of buildings. The experiments demonstrate that LLM outperforms traditional
numerical forecasting in energy consumption prediction.

##  9\. Resources

In this section, we introduce the common time series datasets and archives
utilized in investigated works. We also provide some practical time series
libraries for the baselines comparison of time series analysis. Moreover, we
introduce some LLM deployment resources to the fine-tuning of LLM towards time
series or the pre-training of time series foundation models.

###  9.1. Datasets

Time series analysis is a fundamental problem studied in various fields and
the research popularity of real-world applications largely depends on the
accessible time series datasets. In addition, some datasets not only include
time series data but also other data modalities, enabling the implementation
of multimodal learning. Except for those task-specific datasets, there are
some time series archives encompassing multiple datasets cross-application and
cross-domain for general time series analysis. In the following, we briefly
introduce several widely used datasets and archives to reveal information
about data size, sampling frequency, features, processing tools, download
links, and more.

####  9.1.1. Stock Datasets

The stock dataset typically comprises time series data representing market
information for each stock, sampled at a daily frequency. This includes
various price metrics (open, close, high, low), daily ask and bid prices, and
trading volume. A stock dataset usually contains multiple chosen stocks and
the commonly used ones include NASDAQ100 (Yu et al., 2023) and the Dow
Industrial Average 30 Companies (DOW 30) (Chen et al., 2023b). As for
obtaining datasets, we can get them from public channels, such as obtaining
NASDAQ-100 from yfinance package. We can also obtain them from non-public
channels, such as accessing Dow30 from CRSP databases (Chen et al., 2023b).

In addition, some stock datasets may also include data from other modalities,
mostly text data. These textual data associated with the stocks includes news
headlines (Chen et al., 2023b), company profile data (Yu et al., 2023), tweets
(Xie et al., 2023b). However, most of these data are not public. Some research
works utilize LLMs to be data generator, providing a new approach to accessing
textual data. For example, (Yu et al., 2023) utilizes GPT-4 to generate
company descriptions, and general positive/negative factors.

####  9.1.2. ECG Datasets

The electrocardiogram (ECG) is the most popular diagnostic tool to assist in
the clinical diagnosis of heart diseases and computational ECG analysis has
been studied for over 60 years (Macfarlane and Kennedy, 2021). The ECG dataset
contains recordings of ECG signals capturing the electrical activity of the
heart over a specific period from individuals. There are many publicly
available databases containing ECG datasets with different recording settings,
like duration, leads, and sampling frequency (Merdjanovska and Rashkovska,
2023). Usually, ECG datasets with 12 or 15 leads are obtained from the
standard resting ECG monitor, while those with a few leads are produced by the
Holter monitor. The recording duration usually ranges from 6s to 30 min and
the sampling frequencies of each recording generally vary from 250HZ to
1000HZ. The majority of available ECG datasets can be accessed through public
repositories such as PhysioNet, Figshare, Zenodo, and IEEE Data Port
(Merdjanovska and Rashkovska, 2023). In recent years, there has been a notable
increase in the publication of large-scale arrhythmia datasets, like PTB-XL
(Wagner et al., 2020), MIMIC-III (Johnson et al., 2016).

Some ECG datasets also have textual data as complementary descriptions of the
ECG signals. Each recording in PTB-XL (Wagner et al., 2020) is paired with the
corresponding ECG reports. The reports are generated by the machine and only
describe the ECG without final diagnosis. These time series texts can be
utilized for multimodal learning akin to image-text pairs in CLIP (Radford et
al., 2021).

####  9.1.3. ETT Datasets

ETT (Electricity Transformer Temperature) dataset was first introduced by
(Zhou et al., 2022b) for long sequence time-series forecasting in 2021. It is
a collection of time series data closely related to the temperature of
electricity transformers, serving as a crucial indicator for long-term
electric power deployment. This dataset was gathered between July 2016 and
July 2018 from two distinct counties in China. ETT involves four subsets with
different temporal granularity, ETTm1 and ETTm2 are recorded at 15-minute
intervals, while ETTh1 and ETTh2 are recorded hourly. Each data point
comprises the target variable ”oil temperature” and six power load features.
ETT is very popular in the works we investigate (Zhou et al., 2023b; Jin et
al., 2023a; Liu et al., 2023a; Cao et al., [n. d.]).

####  9.1.4. Other Popular Benchmarks

There are three widely used benchmarks in the works for general time series
analysis. (1) Weather: It is recorded every 10 minutes for the whole year in
2020 in Germany. It is used for meteorologic monitoring and contains 52696
instances with 21 meteorological features, such as air temperature, humidity,
etc. (2) Traffic: It contains the occupation rate of the freeway system across
the State of California. It is recorded hourly with 17544 length and each
record has 862 variables. (3) Electricity: ECL (Electricity Consuming Load)
consists of the hourly power consumption of 321 clients from 2012 to 2014 and
it has 26304 instances.

####  9.1.5. General Archives

To pre-train a foundation model for time series or to test the generalization
ability of LLMs on various time series tasks, we need various time series
datasets from different applications and domains. There are several popular
benchmarking archives that have motivated the development of SOTA time series
algorithms, including UCR (Dau et al., 2019), UEA (Bagnall et al., 2018), TSER
(Tan et al., 2020), M4 (Makridakis et al., 2018), Monash (Godahewa et al.,
2021), UCI (Asuncion, 2007).

Time Series Classification: Time series classification has greatly benefited
from the University of California Riverside (UCR) and University of East
Anglia (UEA) time series archives. UCR archive was first introduced in 2002
with 16 datasets, then grew to 85 datasets in 2015, and finally expanded to
128 datasets in 2018. The latest UCR archive contains a wide range of
problems, including variable length series, but it still only contains
univariate time series.To remedy this defect, also in 2018, UEA researchers
released the first official multivariate time series classification archive
encompassing 30 datasets of equal length with no missing values. Furthermore,
TSER was released in 2020 as the first time series extrinsic regression
archive containing 19 multi-dimensional time series datasets.

Time Series Forecasting: M-competition series (Makridakis et al., 2018, 1982)
is the most popular one with the latest version M5 (Makridakis et al., 2022)
in 2020. Others well-known competitions include NN3 and NN5 (Ben Taieb et al.,
2012), CIF 2016 (Stepnicka and Burda, 2017), Wikipedia web traffic (Wang et
al., 2022a), and Rossmann sales competitions. The M4 benchmark includes a
collection of 100,000 time series data from diverse fields typically
encountered in business, finance, and economic forecasting. Among these
competitions, there was a paradigm shift from single-series to cross-series
learning in competition models. The previous archives with mostly unrelated
single time series are of limited use to evaluate the cross-series learning
models. The Monash forecasting archive was proposed in 2021 for multivariate
time series forecasting and the latest version contains 30 publicly available
datasets with over 400,000 individual time series. The University of
California Irvine (UCI) repository currently contains 507 datasets from
various domains and is the most well-known benchmark archive in general
machine learning. It also contains multiple time series datasets for both
forecasting and classification.

While these archives have made notable contributions to the advancement of
general time series mining, their utility for pre-training foundation model is
still hindered by the sample sizes and generality. The time series community
still lacks large-scale, generic datasets comparable to ImageNet, which are
crucial for facilitating research on large models.

###  9.2. Time series Libraries

Library | Institute |  Release Time |  Channel Setting | Task Type | Model Type
---|---|---|---|---|---
Prophet (Taylor and Letham, 2018) | Facebook | 2017 | Uni. | Fore. | Statistical
HyperTS |  Polytechnic University of Catalonia | 2019 | Mult. | Fore., Class. | Statistical, Deep learning
GluonTS (Alexandrov et al., 2020, 2019) | Amazon | 2019 | Multi. | Fore., Anom. | Deep learning
Tslearn (Tavenard et al., 2020) | University of Rennes | 2020 | Uni. | Fore., Class., Clust. | Machine learning