# NuwaTS: Mending Every Incomplete Time Series

David S. Hippocampus
Department of Computer Science
Cranberry-Lemon University
Pittsburgh Use footnote for providing further information about author
(webpage, alternative address)— _not_ for acknowledging funding agencies.  PA
15213
<EMAIL_ADDRESS>

###### Abstract

Time series imputation plays a crucial role in various real-world systems and
has been extensively explored. Models for time series imputation often require
specialization, necessitating distinct designs for different domains and
missing patterns. In this study, we introduce NuwaTS, a framework to repurpose
Pre-trained Language Model (PLM) for general time series imputation. Once
trained, this model can be applied to imputation tasks on incomplete time
series from any domain with any missing patterns. We begin by devising
specific embeddings for each sub-series patch of the incomplete time series.
These embeddings encapsulate information about the patch itself, the missing
data patterns within the patch, and the patch’s statistical characteristics.
To enhance the model’s adaptability to different missing patterns, we propose
a contrastive learning approach to make representations of the same patch more
similar across different missing patterns. By combining this contrastive loss
with the missing data imputation task, we train PLMs to obtain a one-for-all
imputation model. Furthermore, we utilize a plug-and-play layer-wise fine-
tuning approach to train domain-specific models. Experimental results
demonstrate that leveraging a dataset of over seventeen million time series
from diverse domains, we obtain a one-for-all imputation model which
outperforms existing domain-specific models across various datasets and
missing patterns. Additionally, we find that NuwaTS can be generalized to
other time series tasks such as forecasting.

##  1 Introduction

Time series data are pervasive across numerous fields, including
transportation (li2018diffusion), economics, healthcare
(tonekaboni2020unsupervised), and meteorology (bi2023accurate). However, real-
world time series data often suffer from missing values. Incomplete data
complicate various time series applications such as forecasting and
classification, ultimately degrading the quality of data analysis
(ma2020adversarial). This makes time series imputation especially critical.
Traditionally, time series imputation methods have leaned heavily on
statistical techniques like mean imputation and interpolation van2011mice.
Yet, with the rise of deep learning, there has been a notable shift towards
deep learning models for imputation tasks (cao2018brits; du2023saits). Despite
significant advancements in deep learning-based imputation methods, they often
encounter limitations as they are generally trained and utilized within a
single domain-specific dataset. For instance, a model trained on time series
data from the transportation sector may not be effective for imputing missing
data in other domains such as weather and electricity (Figure 1(a)).

In contrast, foundation models from the fields of Computer Vision (CV)
(kirillov2023segment) and Natural Language Processing (NLP)
(brown2020language; peng2023instruction), pre-trained on vast and diverse
datasets from various domains, have shown remarkable versatility. These models
can be adapted to a wide array of applications across different domains
through methods such as fine-tuning (he2022towards; transferlearningsurvey) or
prompting (jia2022visual; kirillov2023segment). Inspired by these
achievements, we aim to develop a versatile one-for-all model that can impute
incomplete time series data from any domain and accommodate any pattern of
missing data. (Figure 1(b)).

((a)) Specialized models are separately trained on time series domains with
specific missing data patterns and are only applicable to their specialized
domains and missing data patterns.

((b)) One-for-all models (Ours): learned from relatively complete time series
across various domains. Once trained, they can be applied to impute any
incomplete time series from any domain.

Figure 1: Comparison between one-for-all models and specialized models for
missing data imputation on time series.

However, effectively learning a one-for-all imputation model for diverse
domains is technically challenging. The model must possess strong adaptability
to various domains and missing data patterns. Additionally, in some
applications, it needs the capability to quickly specialize to a specific
domain with few-shot learning while retaining its generalizability. To tackle
these challenges, we have developed NuwaTS, a one-for-all model for incomplete
time series, distinguished by several strategies: First, recognizing that
patches based on sub-series offer richer semantic information than individual
time points, we have designed specific embeddings for each sub-series patch.
These tokens include information for the patch itself, the missing data
patterns within the patch, and the patch’s statistical information, serving as
inputs to the PLMs. Second, we introduced a contrastive learning module that
encourages the model to produce more similar representations of the same patch
under varying missing data patterns. The contrastive loss and the
reconstruction loss together constitute the loss function we use to train the
PLM parameters. Third, when a specialized model tailored to a specific domain
is required, we have designed a domain-specific prefix embedding and developed
a plug-and-play fine-tuning mechanism. This mechanism introduces modules that
insert well-designed continuous prompts into each layer of the frozen pre-
trained one-for-all model without altering any of its weights. Once these
modules are removed, the model reverts to its original state as a versatile,
one-for-all model.

Our contributions are as follows:

  1. 1.

We introduce a novel solution, NuwaTS, capable of performing missing data
imputation tasks on any incomplete time series unseen during the training
phase. To the best of our knowledge, this represents the pioneering attempt in
this field.

  2. 2.

We present a “plug-and-play” fine-tuning technique that seamlessly transforms
a one-for-all model into a domain-specific model with minimal data and
computational resources, all without modifying the model’s original weights.

  3. 3.

The one-for-all NuwaTS consistently outperforms domain-specific state-of-the-
art methods in imputation tasks across nearly all missing rates. Moreover,
fine-tuned NuwaTS can be extended to time series forecasting, where its
forecasting results are comparable to or even better than existing domain-
specific time series forecasting models.

##  2 Related Works

###  2.1 Incomplete Time Series Imputation

Many time series imputation models are tailored to specific missing data
patterns and domains, such as randomly missing traffic time series. For
instance, matrix factorization models are designed for multivariate time
series imputation (yu2016temporal). For a dataset from a specific domain with
a particular missing rate, it often requires optimizing a specialized model
using a low-rank prior as the optimization target for imputation. Recent
advancements in deep learning techniques have shown promising results in
addressing missing data imputation. Generative models such as Generative
Adversarial Networks (GANs)(ijcai2019p429; tashiro2021csdi) and diffusion
models are used to learn the underlying distribution of the incomplete time
series. Several architectures, such as Recurrent Neural Networks
(RNNs)(cao2018brits; ma2019end; liu2019naomi; tang2020joint) and attention
mechanisms (du2023saits), are proposed to capture temporal dependencies within
incomplete time series. While achieving good performance on narrow domains and
missing data patterns, these models lack versatility and generalizability to
diverse domains and missing data patterns.

###  2.2 Foundation Model for Time Series

In recent years, foundational models for time series analysis have made
notable strides, primarily leveraging pre-trained backbones from NLP and
aligning modalities to extend their reasoning capabilities to time series
tasks. For example, Gruver et al. (gruver2024large) discovered that by
encoding time series as strings of numerical digits, LLMs can perform time
series forecasting with zero-shot capabilities. GPT4TS (GPT4TS) trains time
series models using pre-trained GPT weights. UniTime (Liu2023UniTimeAL)
integrates domain-specific instructions into LLMs, enhancing their
generalization across domains. TIME-LLM (timellm) employs LLMs’ reasoning by
framing time series data and statistical information in textual prompts,
aligning time patches with word embeddings via cross-attention mechanisms for
improved zero-shot learning. Autotimes (liu2024autotimes) utilizes precomputed
text embedding as positional embeddings and an autoregressive approach for
long-term forecasting. TEST (Sun2023TESTTP) uses text-prototype-aligned
embeddings to enhance LLMs’ reasoning in time series data. $S^{2}$IP-LLM
(Pan2024S2IPLLMSS) applies seasonal-trend decomposition to time series and
uses semantic space-informed prompting to retrieve appropriate prompts from
word token embeddings as prefixes. LLM4TS (Bian2024MultiPatchPA) adapts LLMs
for time series representation learning by directly converting individual
patches into time series sequences. Chronos (ansari2024chronos) trains
language models from scratch on a large collection of time series data, using
scaling and quantization for tokenization. This model demonstrates strong
capabilities in zero-shot probabilistic forecasting. These models primarily
focus on time series forecasting and do not specifically address incomplete
time series issues or the design of embeddings for missing data patterns.

##  3 Methodology

###  3.1 Preliminaries and Problem Statement

###### Definition 1

Incomplete Time Series: We define an incomplete time series
$\mathbf{x}=\\{x_{1},x_{2},\ldots,x_{T}\\}\in\mathbb{R}^{T}$ from domain
$\mathcal{S}$ as a sequence of $T$ observations. Each observation $x_{t}$ is
associated with a timestamp $s_{t}$. In practice, an observation $x_{t}$ may
not be observable due to various reasons. To represent the missing values in
$\mathbf{x}$, we introduce a masking vector $\mathbf{m}\in\mathbb{R}^{T}$
defined by:

$m_{t}=\begin{cases}1&\text{if }x_{t}\text{ is observed}\\\
0&\text{otherwise}\end{cases}$ (1)

Suppose we have acquired $N$ relatively complete time series datasets
$\mathcal{D}=\\{\mathbf{x}^{1},\mathbf{x}^{2},\ldots,\mathbf{x}^{N}\\}$ from a
diverse set of domains
$\\{\mathcal{S}^{1},\mathcal{S}^{2},\ldots,\mathcal{S}^{K}\\}$, where
$\mathbf{x}^{n}\in\mathbb{R}^{T_{n}}$ represents the $n^{th}$ time series with
length $T_{n}$. Here, a “relatively complete time series” implies that
$\sum{{m}_{t}^{n}}\ll T_{n}$, where $\mathbf{m}^{n}$ denotes the missing data
pattern in the $n$-th series. Our objective is to utilize $\mathcal{D}$ to
train an imputation model, denoted as $f_{\mathbf{\Phi}}$, characterized by
parameters $\mathbf{\Phi}$. For any incomplete time series
$\hat{\mathbf{x}}\in\mathbb{R}^{\hat{T}}$ accompanied by any missing pattern
$\hat{\mathbf{m}}\in\mathbb{R}^{\hat{T}}$ from any domain $\hat{\mathcal{S}}$,
the model $f_{\mathbf{\Phi}}$ can impute the missing values in
$\hat{\mathbf{x}}$ as accurately as possible.

###  3.2 Model Architecture

In this work, we initialize model parameters $\mathbf{\Phi}$ with weights from
PLMs. Given the constraints of computational resources, we limit our use to
the parameters of the first six layers, thereby making NuwaTS more accessible
to applications with limited computational resources.

Figure 2: Overview of NuwaTS. To fully leverage the semantic information of
time series and their missing patterns, NuwaTS introduces the tokenization of
time series in patches. It utilizes the missing data patterns, statistical
information for each pattern and the entire series, and a domain-specific
embedding, trained through imputation and contrastive learning tasks.

To train PLMs to adapt to various domains and missing data patterns, we have
implemented several key design features for training. These include Instance
Normalization & Patching, Statistical Embedding, Missing Embedding, a Domain-
Specific Embedding for fine-tuning and Contrastive Learning with Variable
Missing Patterns. We visualize the model architecture in Figure 2.

####  3.2.1 Instance Normalization & Patching

Time series from different domains may exhibit variations in magnitude and
distribution. To mitigate these discrepancies, we apply reversible instance
normalization (kim2021reversible) to each variable
$\mathbf{x}_{i}\in\mathbb{R}^{L}$ (where $\mathbf{x}_{i}$ is a segment of
length $L$ randomly extracted from a time series in the dataset $\mathcal{D}$)
before inputting it into the model. For the missing values, we assign them a
value of $0$. Recognizing that individual time points lack sufficient semantic
meaning and hinder the model’s ability to recognize domain information, we
utilize patching techniques. We define the patch length as $P$ and the non-
overlapping region between two consecutive patches as $S$. The patching
process then generates a sequence of patches
$\mathbf{X}_{i,(p)}\in\mathbb{R}^{P\times N}$, where $N$ is the number of
patches, calculated as $N=\left\lfloor\frac{L-P}{S}\right\rfloor+2$.
Additionally, we pad the sequence with $S$ repeated values of the last data
point $\mathbf{x}_{i}$ before initiating the patching process. We then use a
shared and learnable linear projection to embed each patch into a hidden space
$\mathbf{Z}_{i,(p)}\in\mathbb{R}^{D\times N}$, where $D$ is set to match the
dimension used in the subsequent model.

####  3.2.2 Statistical Embedding

Previous work primarily utilized hard textual prompts to translate dataset
descriptions and statistical information (Liu2023UniTimeAL; liu2024autotimes).
Due to the fact that time series patches and textual prompts come from two
different sources, aligning them has proven to be a complex task (timellm). In
this work, we move away from using hard textual prompts to describe time
series statistics. Instead, we generate crucial statistical information
separately for both the entire segment and each non-overlapping patch,
including the minimum, median, and maximum values (after normalization), as
well as trends (calculated as the last observation value of a patch minus its
first). This information is represented as
$\mathbf{V}_{i}\in\mathbb{R}^{4\times(N+1)}$. Similarly, we employ a shared
and learnable linear projection to embed $\mathbf{V}_{i}$ into a hidden space,
denoted by $\mathbf{Z}_{i,(v)}\in\mathbb{R}^{D\times(N+1)}$, where
$\mathbf{Z}_{i,(v)}=[\mathbf{z}_{i,(v_{g})},\mathbf{Z}_{i,(v_{p})}]$, within
this embedding, the first embedding $\mathbf{z}_{i,(v_{g})}\in\mathbb{R}^{D}$
represents the statistical information for the entire time series. The
remaining $\mathbf{Z}_{i,(v_{p})}\in\mathbb{R}^{D\times N}$ pertains to each
individual patch.

####  3.2.3 Missing Embedding

Adapting to the missing data pattern is also a crucial aspect. The missing
rate of each patch accurately reflects the distribution of missing data across
the target time series. Therefore, we have also designed a Missing Embedding
$\mathbf{z}_{i,(m)}\in\mathbb{R}^{D\times 1}$. It is a randomly-initialized
and learnable parameter. We multiply it with the corresponding patch’s mask
ratio $\mathbf{r}_{i}\in\mathbb{R}^{1\times N}$ and add it to the final
embedding of the patching information:
$\mathbf{E}_{i,(p)}=\mathbf{Z}_{i,(p)}+\mathbf{Z}_{i,(v_{p})}+\mathbf{z}_{i,(m)}\times\mathbf{r}_{i}$.

####  3.2.4 Domain-Specific Embedding

In some instances, we may need our one-for-all model to function within a
specific domain for which we have training data, and we also want to preserve
its capability for cross-domain generalization. To achieve this, we explicitly
introduce a new embedding, $\mathbf{k}\in\mathbb{R}^{D}$. We anticipate that
this embedding will encapsulate unique information from the training data
throughout the training process and serve as a repository of domain-specific
knowledge. We insert this embedding before the patch embeddings
$\mathbf{E}_{i,(p)}$. By integrating the domain-specific embedding and the
global statistical embedding with patch embeddings $\mathbf{E}_{i,(p)}$, the
final embeddings for input $\mathbf{x}_{i}$ is expressed as
$\mathbf{E}_{i}=[\mathbf{k},\mathbf{z}_{i,(v_{g})},\mathbf{E}_{i,(p)}]$, where
$\mathbf{E}_{i}\in\mathbb{R}^{(N+2)\times D}$.

Figure 3: Illustration of contrastive learning: Representations of each patch
under different masks are treated as positive samples for each other, while
representations from other time series and different patches from the same
sequence under varying masks are considered negative samples.

####  3.2.5 Output Layer

After passing through the PLM backbone, we discard the segments corresponding
to the two added prefixes: the domain-specific embedding and the variable-
specific embedding. This means that we use only the representations of the $N$
patches corresponding to the time series as inputs for the model’s final
layer. Although the first two embeddings also participate in the attention
calculations, they are not included in the final input. We then flatten the
final representation to obtain
$\mathbf{h}^{(\text{Layer})}_{i}\in\mathbb{R}^{N\times D}$. Finally, we
linearly map these representations back to their original dimensions,
obtaining the model outputs $\mathbf{o}_{i}\in\mathbb{R}^{L}$.

####  3.2.6 One-For-All Model Training

To further enhance our model’s adaptability to different missing patterns, we
have incorporated a Contrastive Learning module into our training process. The
principle behind this module is straightforward: it ensures that the model can
learn relatively similar representations of the same patch, when subjected to
various missing rates and patterns. During training, for each input
$\mathbf{x}_{i}\in\mathbb{R}^{L}$, we generate two random masks,
$\mathbf{m}_{i,1}$ and $\mathbf{m}_{i,2}\in\mathbb{R}^{L}$, to mask the
inputs. These two masked time series are then simultaneously used as inputs
for the PLM. Each patch outputs a representation and an output head is
utilized for contrastive learning. As illustrated in Figure 3, we treat the
representations of each patch under different masks as positive samples for
each other, while representations from other time series and other patches
from the same series under different masks are treated as negative samples. To
filter out some similar samples, we do not use representations from other
patches of the same time series under the same masks as negative samples. We
use the InfoNCE loss (infonce) combined with the Mean Squared Error (MSE) loss
between the model outputs and the original time series as the loss function
for the one-for-all model (see Appendix B.3 for details). For training
parameters, we freeze the self-attention layers of the PLM and only train the
layer normalization, word positional embeddings, and feed-forward neural
networks within the PLM.

###  3.3 Domain-Specific Fine-Tuning

To achieve a domain-specific model, we borrow the idea of P-tuningv2 strategy
(P-tuningv2). Unlike pre-training phase, where the domain-specific embedding
$\mathbf{k}$ is only added to the input embedding. We incorporate this domain-
specific information into every layer of the PLM. Initially, we employ a
domain-transfer layer—a two-layer MLP network—to map
$\mathbf{k}\in\mathbb{R}^{D}$ to
$\boldsymbol{\mathcal{\hat{K}}}\in\mathbb{R}^{2\times\text{Layer}\times D}$.
Following the P-tuningv2 approach, we also randomly initialize a continuous
prompt $\boldsymbol{\mathcal{P}}\in\mathbb{R}^{2\times\text{Layer}\times D}$
for each layer. We then combine the randomly initialized
$\boldsymbol{\mathcal{P}}$ with the domain-specific information
$\boldsymbol{\mathcal{\hat{K}}}$ to serve as the domain-specific prefix Key
and Value in the PLM’s hidden layers. Thus, we have
$[\mathbf{Key}_{p},\mathbf{Value}_{p}]=\boldsymbol{\mathcal{P}}+\beta\boldsymbol{\mathcal{\hat{K}}}$,
where $\beta=0.01$.

Figure 4: Illustration of domain-specific fine-tuning.

Here, $\mathbf{Key}_{p}$ and
$\mathbf{Value}_{p}\in\mathbb{R}^{\text{Layer}\times D}$ contain the domain-
specific prefix key and value for each layer and participate in the subsequent
attention computation. During fine-tuning on time series data from the target
domain, we freeze all parameters except for the randomly initialized
$\boldsymbol{\mathcal{P}}$ and the domain-transfer layer. The fine-tuning
process is illustrated in Figure 4. If these two trainable modules are
removed, our domain-specific model reverts to the original one-for-all
foundation model. Therefore, the fine-tuning strategy we propose is
essentially “plug-and-play”.

##  4 Experiment

We conducted comprehensive experiments on NuwaTS across ten commonly used
datasets from various domains. To facilitate comparison, we provided three
versions of the model based on the GPT2 architecture: a specific model trained
on a single domain, an one-for-all model trained on a general fused dataset
(17.6M collected samples from various domains), and a fine-tuned model for a
specific domain (fine-tuned based on the one-for-all model). Moreover, we
evaluated BERT (devlin2018bert) and LLaMA2 (touvron2023llama) as backbones for
NuwaTS (details in Appendix D.3). To assess its zero-shot capability, we
trained another GPT2-based model on a transportation domain dataset, LargeST
(liu2024largest) (comprising 100.1 million collected samples) and evaluate it
on time series from other domains.

Baselines. Following the configuration outlined in (du2023saits), we evaluated
two naive imputation methods: Median, where missing values are filled with the
median value, and Last, where missing values are filled with the last previous
observations. To ensure a fair comparison, all methods followed a unified
pipeline111https://github.com/thuml/Time-Series-Library. We assessed two
classic deep learning-based imputation models, BRITS(cao2018brits) and
SAITS(du2023saits). We also compared our model with other foundation models
for time series such as DLinear(dlinear), PatchTST(patchtst),
iTransformer(liu2023itransformer), TimesNet(wu2022timesnet),
Autoformer(wu2021autoformer), Fedformer(zhou2022fedformer), and
GPT4TS(GPT4TS). We trained GPT4TS separately on single datasets with the
number of layers set to 6 to align with NuwaTS. We trained all these models
using a domain-specific approach. Since PatchTST is a transformer-based model
with bi-directional attention, we also trained and tested it on the general
fused dataset.

Setups.  We partitioned the dataset along the sensor (variable) dimension into
training, validation, and test sets in a 1:1:1 ratio. This division simulates
the process of collecting relatively complete time series data from a few
sensors or variables with lower missing rates, training a model with this
data, and then using it to impute data from other sensors or variables that
have higher missing rates. The input time series length $L$ is 96, and we
trained all baselines under random missing rates (sampled from 0.1 to 0.9) and
tested them separately under 9 missing rates: 0.1, 0.2,…, 0.9. We used a total
of ten datasets for domain-specific training and mixed them into a general
fused dataset for one-for-all training. Appendix A.1 shows the details of
datasets. For zero-shot training, we chose
LargeST222https://github.com/liuxu77/LargeST, a large-scale traffic dataset
including a total of 8,600 time series and over 100 million samples, to train
NuwaTS and conduct zero-shot experiments on other domains such as ETTs,
weather and electricity.

Table 1: Imputation results are presented for ETTs, ECL, and weather datasets
with nine test missing rates: 0.1, 0.2, …, 0.9. All results are averaged
across the nine different missing rates. A lower MSE or MAE signifies better
imputation performance.  Red highlights the best results, while  Blue
indicates the second-best.

[b]  Model ETTh1 ETTh2 ETTm1 ETTm2 ECL Weather MSE MAE MSE MAE MSE MAE MSE MAE
MSE MAE MSE MAE Median 0.723 0.611 0.728 0.472 0.699 0.584 0.744 0.464 1.010
0.834 0.998 0.497 Last 0.432 0.476 0.089 0.149 0.336 0.413 0.059 0.122 0.965
0.826 0.731 0.349 Autoformer(2021) 0.552 0.547 0.472 0.420 0.346 0.409 0.211
0.312 0.137 0.262 0.610 0.450 Fedformer(2022) 0.354 0.436 0.538 0.429 0.076
0.185 0.055 0.156 0.129 0.258 0.237 0.211 Dlinear(2023) 0.356 0.414 0.245
0.301 0.274 0.357 0.266 0.351 0.317 0.419 0.355 0.309 iTransformer(2024) 0.639
0.572 0.369 0.376 0.352 0.413 0.356 0.416 0.092 0.200 0.515 0.412 BRITS(2018)1
0.213 0.313 0.117 0.172 0.096 0.183 0.071 0.123 0.317 0.427 0.793 0.474
PatchTST(2023) 0.185 0.298 0.022 0.093 0.080 0.183 0.011 0.066 0.136 0.266
0.271 0.122 TimesNet(2022) 0.166 0.280 0.021 0.091 0.066 0.155 0.011 0.063
0.362 0.450 0.681 0.280 SAITS(2023)1 0.196 0.289 0.215 0.190 0.087 0.163 0.135
0.139 0.445 0.512 0.932 0.484 GPT4TS(2024) 0.196 0.290 0.025 0.092 0.078 0.161
0.012 0.063 0.296 0.401 0.939 0.321 NuwaTS(specific) 0.182 0.293 0.020 0.091
0.070 0.164 0.011 0.064 0.086 0.191 0.307 0.127 PatchTST(one-for-all) 0.178
0.288 0.019 0.088 0.075 0.172 0.011 0.065 0.121 0.243 0.230 0.116 NuwaTS(one-
for-all) 0.164 0.263 0.018 0.084 0.064 0.147 0.010 0.060 0.085 0.186 0.206
0.088 NuwaTS(fine-tuned) 0.156 0.255 0.017 0.082 0.060 0.142 0.010 0.058 0.081
0.183 0.207 0.086

  * 1

We replace the MAE loss function in SAITS (du2023saits) and BRITS
(cao2018brits) with MSE loss function for fair comparison.

Table 2: Imputation results on four PEMS datasets with same setting as Table
1.

Model | PEMS03 | PEMS04 | PEMS07 | PEMS08
---|---|---|---|---
MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE
Median | 0.691 | 0.609 | 0.742 | 0.645 | 0.755 | 0.646 | 0.734 | 0.653
Last | 0.474 | 0.506 | 0.487 | 0.517 | 0.507 | 0.517 | 0.446 | 0.495
Autoformer(2021) | 0.792 | 0.658 | 0.404 | 0.495 | 0.406 | 0.492 | 1.068 | 0.750
Fedformer(2022) | 0.263 | 0.378 | 0.350 | 0.444 | 0.318 | 0.423 | 0.274 | 0.374
Dlinear(2023) | 0.262 | 0.390 | 0.266 | 0.392 | 0.259 | 0.387 | 0.268 | 0.393
iTransformer(2024) | 0.108 | 0.237 | 0.131 | 0.259 | 0.099 | 0.225 | 0.162 | 0.291
BRITS(2018) | 0.143 | 0.267 | 0.259 | 0.370 | 0.228 | 0.353 | 0.225 | 0.344
PatchTST(2023) | 0.059 | 0.170 | 0.074 | 0.190 | 0.052 | 0.159 | 0.067 | 0.180
TimesNet(2022) | 0.102 | 0.221 | 0.152 | 0.268 | 0.132 | 0.252 | 0.114 | 0.231
SAITS(2023) | 0.157 | 0.286 | 0.271 | 0.380 | 0.228 | 0.349 | 0.250 | 0.364
GPT4TS(2024) | 0.101 | 0.220 | 0.158 | 0.271 | 0.131 | 0.250 | 0.110 | 0.227
NuwaTS(specific) | 0.049 | 0.149 | 0.058 | 0.159 | 0.040 | 0.129 | 0.057 | 0.156
PatchTST(one-for-all) | 0.056 | 0.167 | 0.066 | 0.179 | 0.050 | 0.157 | 0.060 | 0.167
NuwaTS(one-for-all) | 0.047 | 0.146 | 0.058 | 0.159 | 0.040 | 0.129 | 0.052 | 0.146
NuwaTS(fine-tuned) | 0.047 | 0.146 | 0.058 | 0.159 | 0.040 | 0.129 | 0.052 | 0.146

Figure 5: Main results across ten datasets under different missing rates.

###  4.1 Main Results

We present the average MAE and MSE for specific, one-for-all, and fine-tuned
NuwaTS models, along with other baselines, across 10 different datasets in
Table 1 and Table 5. Furthermore, we visualize the average MSE across all
datasets for the NuwaTS models and several baselines in Figure 5. On nearly
all datasets, the specific NuwaTS model outperforms other domain-specific
models. The comparison with GPT4TS GPT4TS is particularly noteworthy. GPT4TS
also involves training on a PLM backbone. However, it not only underperforms
compared to NuwaTS, but it is also worse than PatchTST. This underscores the
importance of the patchifying technique333GPT4TS employs a channel-dependent
approach on imputation task. in helping the model understand missing data
patterns. Moreover, we observed that the generalization ability of NuwaTS
(one-for-all) was further augmented after training on the general fused
dataset, thereby affirming the existence of a scaling law (kaplan2020scaling)
within the realm of time series modeling. We visualized the imputation results
in Appendix C.1.

###  4.2 Zero-shot Capability

We validated the zero-shot capability of our model by directly evaluating a
trained model on target data without further training. TimesNet and GPT4TS,
which use a channel-dependent approach, can only perform zero-shot across
datasets with the same number of variables, specifically the four ETT datasets
in our study. In contrast, NuwaTS and PatchTST, as channel-independent
approach, support inference on datasets with different variable dimension. We
trained them on LargeST and then evaluated them on ECL and Weather datasets.
The results in Table 3 shows NuwaTS exhibits superior zero-shot capability,
achieving better performance on all cases. The reason may lie in the use of
specialized missing data embedding and statistical embedding. Additionally,
the language knowledge embedded in the PLM has strong generalization
capabilities, whereas the PatchTST model trained on LargeST does not possess
such strong capabilities.

Table 3: Zero shot evaluation results. All methods were trained on one dataset and zero shot to the other. A lower MSE or MAE indicates a better imputation performance.  Red: the best. /: model failed to work. Metrics | NuwaTS | PatchTST | GPT4TS | TimesNet
---|---|---|---|---
MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE
|  $ETTh1$ $\Rightarrow$ $ETTh2$ | 0.023 | 0.091 | 0.024 | 0.092 | 0.029 | 0.095 | 0.026 | 0.095
|  $ETTh1$ $\Rightarrow$ $ETTm2$ | 0.013 | 0.072 | 0.013 | 0.073 | 0.015 | 0.073 | 0.014 | 0.072
|  $ETTm1$ $\Rightarrow$ $ETTh2$ | 0.021 | 0.094 | 0.022 | 0.096 | 0.027 | 0.093 | 0.024 | 0.096
|  $ETTm1$ $\Rightarrow$ $ETTm2$ | 0.011 | 0.063 | 0.011 | 0.067 | 0.013 | 0.064 | 0.013 | 0.066
|  $LargeST$ $\Rightarrow$ $ECL$ | 0.338 | 0.366 | 0.385 | 0.433 | / | / | / | /
|  $LargeST$$\Rightarrow$ $Weather$ | 0.217 | 0.087 | 0.236 | 0.121 | / | / | / | /

###  4.3 Few-shot Domain-Specific Fine-Tuning

The ability to generalize to the target domain with a small amount of data is
an important criterion for evaluating the generality of a model. We use $10\%$
and $1\%$ of the data for fine-tuning the NuwaTS model trained on LargeST. The
results shown in Table 4 indicate that domain-specific fine-tuning is
particularly effective for fields with limited data. On the ETT datasets, we
achieved the same results using only 10% of the data as we did using 100% of
the data.

Table 4: Finetuned results from the model pre-trained on LargeST (liu2024largest). A lower MSE or MAE indicates a better imputation performance.  Red: the best,  Blue: the second best, bold${\downarrow}$: fine-tuning works. Methods | ECL | ETTh1 | ETTh2 | ETTm1 | ETTm2
---|---|---|---|---|---
MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE
Zero-Shot | 0.338 | 0.366 | 0.212 | 0.306 | 0.021 | 0.089 | 0.066 | 0.145 | 0.011 | 0.059
Fine-tuning with $1\%$ data | 0.284${\downarrow}$ | 0.338${\downarrow}$ | 0.205${\downarrow}$ | 0.305${\downarrow}$ | 0.021${\downarrow}$ | 0.090 | 0.063${\downarrow}$ | 0.147 | 0.010${\downarrow}$ | 0.059
Fine-tuning with $10\%$ data |  0.190 ${\downarrow}$ |  0.292${\downarrow}$ |  0.180 ${\downarrow}$ |  0.282${\downarrow}$ |  0.019${\downarrow}$ |  0.086${\downarrow}$ | 0.060 ${\downarrow}$ | 0.142${\downarrow}$ |  0.010 ${\downarrow}$ |  0.058 ${\downarrow}$
Fine-tuning with $100\%$ data |  0.143 ${\downarrow}$ |  0.253 ${\downarrow}$ |  0.180 ${\downarrow}$ |  0.280 ${\downarrow}$ | 0.019${\downarrow}$ | 0.085${\downarrow}$ |  0.061${\downarrow}$ |  0.143${\downarrow}$ |  0.010 ${\downarrow}$ |  0.058${\downarrow}$

###  4.4 Ablation Study

Table 5: Ablation results of the model training on LargeST and zero-shot on ECL and Weather. A lower MSE or MAE indicates a better imputation performance.  Red: the best. Model | ECL | Weather
---|---|---
MSE | MAE | MSE | MAE
| Default | 0.338 | 0.366 | 0.217 | 0.087
| w/o-StatisticEmbedding | 0.371 | 0.383 | 0.224 | 0.093
| w/o-ContrastiveLearning | 0.348 | 0.366 | 0.218 | 0.087
| w/o-MissingEmbedding | 0.338 | 0.368 | 0.221 | 0.092
| w/o-GPT2weight | 0.339 | 0.368 | 0.218 | 0.089
| FrozenBackbone | 0.533 | 0.560 | 0.290 | 0.131

We conducted ablation experiments via zero-shot evaluation. We trained the
default model, several ablated models on LargeST. We then performed zero-shot
evaluation on ECL and Weather datasets for out-of-domain evaluation. As shown
in Table 5, after ablation of the Statistic Embedding, Missing Embedding, and
contrastive learning module, the model’s zero-shot ability decreases
significantly, proving the necessity of these key components. Freezing the PLM
backbone leads to poor performance. Finally, we discovered that when training
the model from scratch without using weights pre-trained on NLP tasks, its
zero-shot generalization ability on other datasets deteriorates significantly.
This indicates that training the model on NLP tasks benefits its performance
on time series tasks, demonstrating that cross-modality training is meaningful
(irrelevant_modility).

We also verified the design of domain-specific fine-tuning strategy of NuwaTS
in Table 6. When the randomly initialized prefix key&value and the knowledge-
transfer layer were ablated, the model’s fine-tuning performance showed a
significant decline which indicates that domain-transfer layer retains the
learned knowledge while newly added prefix key&value provide flexibility in
transferring to specific domain. Additionally, we find that merely appending
the prefix to the input layer and fine-tuning it significantly reduces the
effectiveness of fine-tuning. Inserting the prefix into each layer of the PLM
enhances the prefix’s representation capacity during domain transfer
(P-tuningv2).

Table 6: Domain-specific fine-tuning ablation study on ETTs based on the model pre-trained on general fused dataset. A lower MSE or MAE indicates a better imputation performance.  Red: the best. Model | ETTh1 | ETTh2 | ETTm1 | ETTm2
---|---|---|---|---
MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE
Domain-Specific Fine-Tuning | 0.156 | 0.255 | 0.017 | 0.082 | 0.060 | 0.142 | 0.010 | 0.058
w/o random initialized prefix key&value | 0.160 | 0.260 | 0.019 | 0.085 | 0.063 | 0.149 | 0.010 | 0.060
w/o domain-transfer layer | 0.158 | 0.256 | 0.018 | 0.083 | 0.060 | 0.142 | 0.010 | 0.058
only apply to input embedding | 0.163 | 0.263 | 0.018 | 0.084 | 0.062 | 0.145 | 0.010 | 0.060
no fine-tuning | 0.164 | 0.263 | 0.018 | 0.084 | 0.064 | 0.147 | 0.010 | 0.060

###  4.5 Enhancing Forecasting on Incomplete Time Series

Table 7:  We trained and evaluated TimesNet (wu2022timesnet) on incomplete time series with original missing rates of 0.2 and 0.5, comparing its forecasting performance on data imputed by PatchTST and NuwaTS. We set the length of input sequence and forecasting both to 96. A lower MSE or MAE indicates a better forecasting performance.  Red: the best.  Methods | ETTh1 | ETTh2 | ETTm1 | ETTm2
---|---|---|---|---
MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE
Complete Data | 0.384 | 0.402 | 0.340 | 0.374 | 0.338 | 0.375 | 0.187 | 0.267
Missing Rate: 0.2 | Default | 0.457 | 0.450 | 0.353 | 0.389 | 0.389 | 0.404 | 0.200 | 0.284
+PatchTST | 0.451 | 0.451 | 0.353 | 0.387 | 0.369 | 0.398 | 0.190 | 0.274
+NuwaTS | 0.440 | 0.444 | 0.345 | 0.380 | 0.356 | 0.389 | 0.192 | 0.271
Missing Rate: 0.5 | Default | 0.585 | 0.515 | 0.370 | 0.403 | 0.572 | 0.496 | 0.238 | 0.318
+PatchTST | 0.490 | 0.474 | 0.392 | 0.409 | 0.449 | 0.443 | 0.200 | 0.284
+NuwaTS | 0.471 | 0.463 | 0.378 | 0.400 | 0.425 | 0.428 | 0.194 | 0.278

In practical applications, forecasting models often have to train on
incomplete data, which can significantly impair their performance. To address
this issue, we applied pre-trained imputation models to impute the incomplete
data before training the forecasting model. We chose TimesNet (wu2022timesnet)
as the forecasting model and used four ETT datasets for training. We randomly
discarded 20% and 50% of the data in the training sets of four ETT datasets.

To prevent data leakage during imputation, we used NuwaTS and PatchTST pre-
trained on LargeST to impute the training data. Finally, we trained TimesNet
wu2022timesnet on the imputed time series and tested it on the complete time
series.

As shown in Table 7, the data imputed by NuwaTS improved the forecasting
model’s performance across most metrics. Additionally, the imputed data from
NuwaTS was of higher quality for downstream tasks compared to PatchTST,
demonstrating that NuwaTS can effectively address the issue of incomplete time
series data in practical applications.

###  4.6 Fine-tuning Imputation Model to Forecasting Model

Table 8: Forecasting results. We set the length of input sequence and
forecasting both to 96. Forecasting results from other baselines come from
wu2022timesnet; liu2023itransformer; cai2024msgnet. We fine-tuned the model
pre-trained on LargeST in order to avoid data leakage. A lower MSE or MAE
indicates a better performance.  Red: the best,  Blue: the second best.

Model | ETTh1 | ETTh2 | ETTm1 | ETTm2 | ECL | Weather
---|---|---|---|---|---|---
MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE
Dlinear(2023) | 0.397 | 0.412 | 0.333 | 0.387 | 0.345 | 0.372 | 0.193 | 0.292 | 0.197 | 0.282 | 0.196 | 0.255
Autoformer(2021) | 0.449 | 0.459 | 0.346 | 0.388 | 0.505 | 0.475 | 0.255 | 0.339 | 0.201 | 0.317 | 0.266 | 0.336
Fedformer(2022) | 0.395 | 0.424 | 0.358 | 0.397 | 0.379 | 0.419 | 0.203 | 0.287 | 0.193 | 0.308 | 0.217 | 0.296
MSGNet(2024) | 0.390 | 0.411 | 0.328 | 0.371 | 0.319 | 0.366 | 0.177 | 0.262 | 0.165 | 0.274 | 0.163 | 0.212
iTransformer(2024) | 0.386 | 0.405 | 0.297 | 0.349 | 0.334 | 0.368 | 0.180 | 0.264 | 0.148 | 0.240 | 0.174 | 0.214
PatchTST(2023) | 0.460 | 0.447 | 0.308 | 0.355 | 0.352 | 0.374 | 0.183 | 0.270 | 0.190 | 0.296 | 0.186 | 0.227
TimesNet(2022) | 0.384 | 0.402 | 0.340 | 0.374 | 0.338 | 0.375 | 0.187 | 0.267 | 0.168 | 0.272 | 0.172 | 0.220
NuwaTS(fine-tuned without output layer) | 0.398 | 0.419 | 0.319 | 0.369 | 0.316 | 0.357 | 0.191 | 0.275 | 0.197 | 0.284 | 0.176 | 0.219
NuwaTS(fine-tuned with output layer) | 0.375 | 0.404 | 0.292 | 0.348 | 0.319 | 0.360 | 0.185 | 0.268 | 0.183 | 0.267 | 0.172 | 0.215

By directly inserting the masked padding tokens $\mathbf{p}$ where
$\mathbf{p}\in\mathbb{R}^{M\times D}$ after the original input embeddings
$\mathbf{E}_{i}\in\mathbb{R}^{(N+2)\times D}$, we get
$\mathbf{E}_{i}^{f}=[\mathbf{k},\mathbf{z}_{i,(v_{g})},\mathbf{E}_{i,(p)},\mathbf{p}]$
where $\mathbf{E}_{i}^{f}\in\mathbb{R}^{(N+2+M)\times D}$. The model
automatically imputes the future patches, effectively transforming into a
forecasting model. We discard the prefixal part and obtain the final
forecasting represenation
$\mathbf{h}^{(\text{Layer})}_{i}\in\mathbb{R}^{M\times D}$. Then
$\mathbf{h}^{(\text{Layer})}_{i}$ pass through the output layer and we get the
final forecasting results. In order to further help NuwaTS adapt to the
forecasting task, we applied domain-specific fine-tuning on the one-for-all
model and trained its domain-transfer layer, LayerNorm layers as well as
position embeddings, which account for only 9.35% of the model’s total
parameters (details in Appendix A.3). Notably, even without training the
output layer, the model still achieves remarkable performance on the long-term
forecasting task, as shown in Table 8. We visualized the forecasting results
in Appendix C.2

##  5 Conclusion

We propose NuwaTS, a time series imputation model equipped with a PLM backbone
and a contrastive learning module that explicitly incorporates patch-wise
statistical information and missing patterns. Experimental results demonstrate
that our imputation model significantly outperforms existing methods and can
adapt to diverse domains and missing rates in a zero-shot manner.
Additionally, we developed an efficient domain-specific fine-tuning technique
that, with minimal cost, can adapt the model from an all-in-one to a domain-
specific one, and also transform it from an imputation model to a forecasting
model. To the best of our knowledge, this is the first attempt to design a
foundational time series imputation model that can be applied across different
domains. Our proposed method serves as a valuable reference for this field

##  Appendix A Experimental Details

###  A.1 Dataset Details

Our research encompass experimental evaluations utilizing a selection of ten
popular multivariate datasets, including (1)
ETT444https://github.com/zhouhaoyi/ETDataset reports seven factors of
electricity transformers, encompassing four subsets (ETTm1, ETTm2, ETTh1,
ETTh2). These are divided into two categories based on temporal granularity,
with data recorded at intervals of 15 minutes (m) and 1 hour (h),
respectively. (2)
ECL555https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014
records the hourly electricity consumption of 321 customers. (3)
Weather666https://www.bgc-jena.mpg.de/wetter/ comprises 21 meteorological
factors, recorded at ten-minute intervals throughout the year 2020, including
variables such as temperature, humidity. (4) PEMS encompasses four datasets
(03, 04, 07 and 08), each of which collects data on California’s public
traffic network at a frequency of every five minutes. For the LargeST
[liu2024largest], which consists of 8,600 sensors and spans from 2017 to 2021,
we only selected the 2019 data for pre-training NuwaTS in this paper.

For the eleven datasets mentioned above, we split the datasets from the
variable dimension rather than the time dimension in a 1:1:1 ratio. This is
because some baseline models use a channel-dependent approach, requiring fixed
input dimensions for time series. This ensures proper training, validation,
and testing.

For the general fused datasets, we processed each time variable separately. We
sliced time variables from ten datasets (ETT,Weather, ECL, PEMS) of varying
lengths with a stride of 1, creating a combined dataset with 17.6 million time
segments of fixed length. In this study, we set it to 96. These segments were
then randomly masked and used for model training.

Using the same approach, we sliced LargeST into 100.1 million segments to
train NuwaTS and PatchTST. This was done for zero-shot evaluation and fine-
tuning for forecasting tasks on ETT, weather and ECL, ensuring no data
leakage.

###  A.2 Implementation Details

Regarding the training details of our model compared to others, we mainly
followed the imputation task configuration including optimizer, learning rate
and early stop strategy in the https://github.com/thuml/Time-Series-Library
for fair comparisons. We trained three different PLM backbones: GPT2 [gpt2],
BERT [devlin2018bert], and LLaMA [touvron2023llama]. All the comparison
models, as well as GPT2 and BERT, were trained on NVIDIA RTX 3090-24G GPUs.
LLaMA-version (the first six layers) was trained on an NVIDIA A6000-48G GPU.

###  A.3 Different parameter-efficient training strategy

We trained NuwaTS using three different PLM backbones. Table.9 presents the
detailed training specifics for each experiment and backbone, including fine-
tuned network layers, memory usage, and the number of parameters.

Table 9: Different parameter-efficient training strategy on GPT2, BERT and
LLaMA2.

[b]  Task InputSize Backbone(number of layers) TrainingModule
MemoryOccupation(GB) Training/TotalParameter(M) Imputation (512,96,1) GPT2(3)
LN, WPE, IN&OUT, FFN 2.7 20.3/66.0 Imputation (512,96,1) GPT2(6) LN, WPE,
IN&OUT, FFN 4.6 38.0/90.8 Imputation (512,96,1) GPT2(9) LN, WPE, IN&OUT, FFN
6.4 55.8/115.6 Imputation (512,96,1) GPT2(12) LN, WPE, IN&OUT, FFN 8.3
73.5/140.4 Imputation (512,96,1) LLaMA2(6) LN, IN&OUT 19.5 6.3/1351.7
Imputation (128,96,1) LLaMA2(6) LN, IN&OUT, FFN 20.3 818.0/1351.7 Imputation
(512,96,1) BERT(6) LN, WPE, IN&OUT, FFN 3.1 34.1/68.2 Imputation(fine-tuned)
(512,96,1) GPT2(6) DTL 2.4 7.7/90.8 Forecasting(w/o-OutputLayer) (512,96,1)
GPT2(6) DTL 3.8 8.5/90.8 Forecasting(w-OutputLayer) (512,96,1) GPT2(6) DTL,
OUT 3.8 8.9/90.8

  * •

LN: LayerNorm Layer, WPE: Word Position Encoding, IN&OUT:
EmbedLayer&OutputLayer

  * •

FFN: Feed-Foward Network, DTL: Domain-Transfer Layer&prefix

##  Appendix B Training Algorithm and Mathematical formula

###  B.1 Algorithm

Algorithm 1 One-for-all Model Training

1: Given the relatively complete time series datasets
$\mathcal{D}=\\{\mathbf{x}^{1},\mathbf{x}^{2},\ldots,\mathbf{x}^{N}\\}$ from
diverse set of domains.

2: for $\mathbf{x}_{i}\in\mathbb{R}^{L}$ $\leftarrow$ $\mathcal{D}$ do

3: Generate two random mask $\mathbf{m}_{i,1}\in\mathbb{R}^{L}$,
$\mathbf{m}_{i,2}\in\mathbb{R}^{L}$

4:
$\hat{\mathbf{x}}_{i,1},\hat{\mathbf{x}}_{i,2}\leftarrow\mathbf{x}_{i}\times\mathbf{m}_{i,1},\mathbf{x}_{i}\times\mathbf{m}_{i,2}$

5: $\mathbf{Z}_{i,(p)}\leftarrow\text{Embed}(\mathbf{\hat{x}}_{i})$

6:
$\mathbf{E}_{i,(p)}\leftarrow\mathbf{Z}_{i,(p)}+\mathbf{Z}_{i,(v_{p})}+\mathbf{z}_{i,(m)}\times\mathbf{r}_{i}$

7:
$\mathbf{E}_{i}\leftarrow[\mathbf{k},\mathbf{z}_{i,(v_{g})},\mathbf{E}_{i,(p)}]$

8: Obtain the last hidden states
$\mathbf{h}_{i}^{(\text{Layer})}\leftarrow\text{PLM}(\mathbf{E}_{i})$

9: Obatin the imputed series
$\mathbf{o}_{i}\leftarrow\text{OutputLayer}\left(\mathbf{h}_{i}^{(\text{Layer})}\right)$

10: Update $\mathbf{\Phi}$ by gradients for
$\mathcal{L}_{mse}\left(\mathbf{o}_{i,1},\mathbf{x}_{i}\right)+\mathcal{L}_{mse}\left(\mathbf{o}_{i,2},\mathbf{x}_{i}\right)+\alpha\mathcal{L}_{cl}\left(\mathbf{h}^{(\text{Layer})}_{i,1},\mathbf{h}^{(\text{Layer})}_{i,2}\right)$

11: end for

Algorithm 2 Domain Specific Fine-tuning

1: Initialize continuous prefix
$\boldsymbol{\mathcal{P}}\in\mathbb{R}^{2\times\text{Layer}\times d}$

2: Obtain
$\boldsymbol{\mathcal{\hat{K}}}\in\mathbb{R}^{2\times\text{Layer}\times
d}\leftarrow\text{DomainTrans}\left(\mathbf{k}\right)$

3: WITH NO GRAD:

4: for $n$ $\leftarrow$ 0 to Layer do

5: Obtain hidden state $\mathbf{h}^{(n-1)}$ from $\text{PLMLayer}^{(n-1)}$

6: Obtain
$[\mathbf{Key}_{p},\mathbf{Value}_{p}]\leftarrow\boldsymbol{\mathcal{P}}^{(n)}+\beta\boldsymbol{\mathcal{\hat{K}}}^{(n)}$
where $\boldsymbol{\mathcal{P}}^{(n)}\in\mathbb{R}^{2\times d}$

7: Obtain $\mathbf{Key}$ $\leftarrow$
$\text{Concat}\left(\mathbf{Key}_{p};\mathbf{Key}^{(n)}\right)$

8: Obtain $\mathbf{Value}$ $\leftarrow$
$\text{Concat}\left(\mathbf{Value}_{p};\mathbf{Value}^{(n)}\right)$

9: Obtain $\mathbf{h}^{(n)}\leftarrow$
$\text{PLMLayer}^{(n)}$$\left(\mathbf{h}^{(n-1)},\mathbf{Key},\mathbf{Value}\right)$

10: end for

11: Obtain $\mathbf{h}^{(\text{Layer})}$

###  B.2 Instance Norm

We applied reversible instance normalization (RevIN) [kim2021reversible] to
individual series before splitting them into patches. Each series was
normalized to have zero mean and unit standard deviation, with the original
mean and standard deviation stored for later reversal, thereby reducing the
domain distribution shift caused by non-stationary information. The details
are as follows:

$\begin{split}&\mathbb{E}_{t}\left[\mathbf{x}_{i}\right]=\frac{1}{L}\sum_{j=1}^{L}\mathbf{x}_{i,j}\\\
&\operatorname{Var}\left[\mathbf{x}_{i}\right]=\frac{1}{L}\sum_{j=1}^{L}\left(\mathbf{x}_{i,j}-\mathbb{E}_{t}\left[\mathbf{x}_{i}\right]\right)^{2}\\\
&\hat{\mathbf{x}}_{i}=\left(\frac{\mathbf{x}_{i}-\mathbb{E}_{t}\left[\mathbf{x}_{i}\right]}{\sqrt{\operatorname{Var}\left[\mathbf{x}_{i}\right]+\epsilon}}\right)\\\
\end{split}$ (2)

###  B.3 Contrastive Learning Loss

Given a time patch represenation as a query $q$, and a set of keys
$\mathbb{K}=\left\\{k_{0},k_{1},\ldots\right\\}$, including a positive target
$k_{+}$. We employ the InfoNCE [infonce] as a auxiliary loss function to
optimize the model’s performance:

$\mathcal{L}_{cl}=-\log\frac{\exp\left(q^{T}Wk_{+}\right)}{\exp\left(q^{T}Wk_{+}\right)+\sum_{i=0}^{K-1}\exp\left(q^{T}Wk_{i}\right)},$
(3)

where we employ bi-linear inner-product and $W$ is a learnable matrix.

##  Appendix C Visualization

###  C.1 Imputation Visualization

We visualized the imputation results of NuwaTS in Figure 6,Figure 7,Figure 8.

((a)) NuwaTS ((b)) PatchTST ((c)) GPT4TS ((d)) TimesNet

((e)) Brits ((f)) NuwaTS ((g)) PatchTST ((h)) GPT4TS

((i)) TimesNet ((j)) Brits

Figure 6: Case visualization when missing rate set to 0.1.

((a)) NuwaTS ((b)) PatchTST ((c)) GPT4TS ((d)) TimesNet

((e)) Brits ((f)) NuwaTS ((g)) PatchTST ((h)) GPT4TS

((i)) TimesNet ((j)) Brits

Figure 7: Case visualization when missing rate set to 0.5.

((a)) NuwaTS ((b)) PatchTST ((c)) GPT4TS ((d)) TimesNet

((e)) Brits ((f)) NuwaTS ((g)) PatchTST ((h)) GPT4TS

((i)) TimesNet ((j)) Brits

Figure 8: Case visualization when missing rate set to 0.9.

###  C.2 Forecasting Visualization

We visualized the forecasting results of fine-tuned NuwaTS whose backbone had
not undergone any forecasting training in Figure 9.

((a)) ECL ((b)) ECL ((c)) ECL ((d)) ETTh1

((e)) ETTh1 ((f)) ETTh1 ((g)) ETTh2 ((h)) ETTm1

((i)) ETTm2 ((j)) Weather

Figure 9: Case visualization when sequence length set to 96 and forecasting
length set to 96.

###  C.3 Domain Recognition

We also visualized the domain recognition ability of NuwaTS in Figure 10. By
directly appending domain-specific embeddings $\mathbf{k}$ again to the end of
the input embedding $\mathbf{E}_{i}\in\mathbb{R}^{(N+2)\times D}$, we get
$\hat{\mathbf{E}_{i}}=[\mathbf{k},\mathbf{z}_{i,(v_{g})},\mathbf{E}_{i,(p)},\mathbf{k}]$,
where $\hat{\mathbf{E}_{i}}\in\mathbb{R}^{D\times(N+3)}$. We feed the
$\hat{\mathbf{E}_{i}}$ into the pre-trained model, and get the final
representation
$\hat{\mathbf{h}}^{(\text{Layer})}_{i}\in\mathbb{R}^{(N+3)\times D}$. Then We
extract the last embeddings $\mathbf{k^{\prime}}\in\mathbb{R}^{D}$ from
$\hat{\mathbf{h}}^{(\text{Layer})}_{i}$ which is mixed with the patch
embeddings in front of the input embeddings, thus exhibiting distinct domain
characteristics. We collected $N$ time seires from different domain and
extracted $\mathbf{k^{\prime}}$. After applying the t-SNE [van2008visualizing]
method to reduce the dimension to 2, we obtained scattered points
$\mathbf{k^{\prime}}^{p}\in\mathbb{R}^{N\times 2}$ and visualized them. The
scattered points corresponding to time series from the same domain tend to
cluster together which indicates that NuwaTS could recognize domain
information.

Figure 10: Illustration of domain recognition.

##  Appendix D Extra Experiments

###  D.1 Learning From Incomplete Training data

Table 10: When model training on incomplete data with 0.2 and 0.5 original
missing rate. The comparison between NuwaTS and PatchTST. A lower MSE or MAE
indicates a better imputation performance.

Methods | ETTh1 | ETTh2 | ETTm1 | ETTm2 | ECL | Weather
---|---|---|---|---|---|---
MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE
Default | 0.164 | 0.263 | 0.018 | 0.084 | 0.064 | 0.147 | 0.010 | 0.060 | 0.085 | 0.186 | 0.206 | 0.088
NuwaTS-Origin Missing Rate 0.2 | 0.169 | 0.276 | 0.018 | 0.085 | 0.070 | 0.160 | 0.010 | 0.061 | 0.130 | 0.233 | 0.214 | 0.099
NuwaTS-Origin Missing Rate 0.5 | 0.170 | 0.274 | 0.020 | 0.086 | 0.077 | 0.167 | 0.011 | 0.062 | 0.160 | 0.258 | 0.228 | 0.114
PatchTST | 0.178 | 0.288 | 0.019 | 0.088 | 0.075 | 0.172 | 0.011 | 0.065 | 0.121 | 0.243 | 0.230 | 0.116
PatchTST-Origin Missing Rate 0.2 | 0.185 | 0.295 | 0.019 | 0.089 | 0.079 | 0.179 | 0.011 | 0.066 | 0.150 | 0.264 | 0.239 | 0.118
PatchTST-Origin Missing Rate 0.5 | 0.202 | 0.316 | 0.021 | 0.093 | 0.079 | 0.181 | 0.011 | 0.067 | 0.293 | 0.387 | 0.244 | 0.124

We simulated the training of the model on incomplete time series by randomly
omitting 20% and 50% of the original training data and then tested its ability
of resisting disturbance. The experimental results indicate that NuwaTS has
strong resilience and can achieve good generalization performance when trained
on data with a high missing rate (details shown in Table 10).

###  D.2 The effect of the number of the GPT2 layer and Patch Size.

Through experiments, we tested the impact of the number of GPT-2 layers and
patch size on the final results in Figure 11. The experimental results showed
that using 6 GPT-2 layers yielded the best performance. Additionally, we found
that smaller patch sizes led to better model performance, indicating that for
imputation tasks, smaller patch sizes allow the model to capture finer time
series details. However, we chose a patch size of 16 to balance model
performance and computational resource requirements.

((a)) The effect of the number of GPT2 layers on Imputation Performance.

((b)) The effect of Patch Size on Imputation Performance.

Figure 11: The effect of the number of the GPT2 layer and Patch Size.

###  D.3 Effect of the Type of PLM Backbone

Table 11: Effect of different PLM backbone. A lower MSE or MAE indicates a
better imputation performance.  Red: the best.

Model | ETTh1 | ETTh2 | ETTm1 | ETTm2 | ECL | Weather
---|---|---|---|---|---|---
MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE
GPT2 | 0.164 | 0.263 | 0.018 | 0.084 | 0.064 | 0.147 | 0.010 | 0.060 | 0.085 | 0.186 | 0.206 | 0.088
BERT | 0.169 | 0.267 | 0.018 | 0.084 | 0.065 | 0.148 | 0.010 | 0.060 | 0.091 | 0.193 | 0.217 | 0.091
LLaMA2 | 0.170 | 0.272 | 0.018 | 0.084 | 0.067 | 0.152 | 0.010 | 0.060 | 0.096 | 0.200 | 0.210 | 0.090

We experimented with three different backbones in Table 11: GPT2 [gpt2], BERT
[devlin2018bert], and LLaMA [touvron2023llama], uniformly using the first six
layers of each. Among them, BERT is a transformer with bi-directional
attention, while the other two, GPT2 and LLaMA, employ causal attention.

Using GPT-2 as the backbone outperformed the much larger LLaMA2, likely due to
the relatively small amount of training data compared to LLaMA2’s massive
parameter count.

Additionally, LLaMA2’s large embedding dimensions produced sparse features and
the slow speed of LLaMA2 during training and evaluation makes deployment on
edge devices challenging.

BERT, with the same dimension (768) as GPT-2, mainly differs in its
bidirectional attention. It slightly lags behind GPT-2 due to the causal
nature of time series data. However, BERT’s bidirectional attention allows the
domain-specific embedding to capture domain information from the following
embed patches. Figure 12 visualizes this, showing that our NuwaTS framework
can recognize data domains and distribution, making it potential for time
series classification and anomaly detection.

Figure 12: Domain information visualization. We take the model’s first output
embedding as the [CLS] token, which carries domain information from following
sequences. We then use the t-SNE [van2008visualizing] method to visualize the
domain information from different datasets.

###  D.4 Embed with simple linear layer VS. Embed with text-alignment.

For tokenization, we employ a simple linear layer to embed patches. We
acknowledge that recent research [timellm] has proposed utilizing Patch
Reprogramming mechanism to align time patches with PLM’s pre-trained word
embeddings, thereby activating the model’s time series understanding and
reasoning capabilities. However, we contend that, due to the high proportion
of missing values in the input masked patches and the varying locations of
these missing values, modality alignment is not effective in representing such
complex masked time series. Consequently, we opt for employing a simple linear
mapping layer on patches, which proves sufficient for our purposes in Table
12.

Table 12: Embedding methods comparision.

Model | ETTh1 | ETTh2 | ETTm1 | ETTm2 | ECL | Weather
---|---|---|---|---|---|---
MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE | MSE | MAE
Simple linear layer | 0.164 | 0.263 | 0.018 | 0.084 | 0.064 | 0.147 | 0.010 | 0.060 | 0.085 | 0.186 | 0.206 | 0.088
Embed with text-alignment [timellm] | 0.250 | 0.357 | 0.024 | 0.100 | 0.128 | 0.242 | 0.014 | 0.014 | 0.205 | 0.311 | 0.323 | 0.153

##  Appendix E Main Results in Detailed

We reported detailed results for various baselines and different variants of
our model across ten datasets, with missing rates ranging from 0.1 in Table 13
to 0.9 in Table 21.

Table 13: Detailed Imputation results on ETTs, ECL and weather with missing
rate set to 0.1. We set the input length to 96. A lower MSE or MAE indicates a
better imputation performance.

[b]  Model ETTh1 ETTh2 ETTm1 ETTm2 ECL Weather PEMS03 PEMS04 PEMS07 PEMS08 MSE
MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE
Median 0.723 0.609 0.725 0.472 0.698 0.582 0.746 0.464 0.992 0.825 0.991 0.494
0.667 0.597 0.716 0.633 0.728 0.634 0.708 0.641 Last 0.399 0.460 0.068 0.135
0.310 0.399 0.039 0.109 0.912 0.811 0.692 0.399 0.448 0.489 0.460 0.501 0.480
0.500 0.420 0.479 Autoformer(2021) 0.487 0.529 0.629 0.509 0.518 0.477 0.231
0.319 0.074 0.194 0.435 0.370 2.698 1.348 0.532 0.583 0.677 0.670 3.714 1.575
Fedformer(2022) 0.276 0.393 1.278 0.645 0.030 0.120 0.023 0.106 0.072 0.193
0.142 0.126 0.151 0.281 0.455 0.503 0.361 0.463 0.252 0.334 Dlinear(2023)
0.284 0.383 0.235 0.330 0.262 0.383 0.399 0.472 0.259 0.403 0.355 0.353 0.204
0.377 0.219 0.388 0.283 0.445 0.315 0.468 iTransformer(2024) 0.447 0.490 0.056
0.147 0.129 0.254 0.074 0.196 0.053 0.153 0.308 0.243 0.075 0.192 0.084 0.204
0.062 0.175 0.097 0.220 BRITS(2018) 0.119 0.226 0.063 0.127 0.046 0.122 0.032
0.084 0.233 0.367 0.798 0.476 0.140 0.263 0.252 0.362 0.227 0.353 0.210 0.331
PatchTST(2023) 0.123 0.246 0.017 0.086 0.059 0.167 0.009 0.061 0.078 0.216
0.193 0.105 0.051 0.166 0.070 0.192 0.044 0.151 0.064 0.184 TimesNet(2022)
0.109 0.230 0.014 0.080 0.034 0.118 0.008 0.056 0.319 0.422 0.617 0.271 0.086
0.200 0.139 0.253 0.119 0.236 0.100 0.213 SAITS(2023) 0.124 0.231 0.188 0.176
0.064 0.130 0.135 0.139 0.392 0.480 0.930 0.492 0.154 0.282 0.264 0.374 0.223
0.346 0.245 0.359 GPT4TS(2024) 0.095 0.207 0.017 0.079 0.028 0.103 0.008 0.053
0.223 0.348 0.968 0.317 0.087 0.205 0.143 0.256 0.116 0.234 0.096 0.215
NuwaTS(specific) 0.123 0.244 0.017 0.086 0.052 0.152 0.009 0.062 0.047 0.147
0.237 0.112 0.040 0.138 0.050 0.148 0.033 0.119 0.054 0.156 PatchTST(one-for-
all) 0.116 0.237 0.014 0.080 0.044 0.138 0.009 0.060 0.072 0.199 0.175 0.100
0.047 0.156 0.057 0.169 0.041 0.146 0.051 0.158 NuwaTS(one-for-all) 0.101
0.209 0.014 0.075 0.034 0.113 0.008 0.055 0.046 0.142 0.141 0.075 0.040 0.138
0.052 0.152 0.033 0.121 0.046 0.139 NuwaTS(finetuned) 0.100 0.208 0.014 0.076
0.035 0.114 0.008 0.054 0.045 0.141 0.138 0.072 0.041 0.139 0.053 0.155 0.034
0.123 0.046 0.140

Table 14: Detailed Imputation results on ETTs, ECL and weather with missing
rate set to 0.2. We set the input length to 96. A lower MSE or MAE indicates a
better imputation performance.

[b]  Model ETTh1 ETTh2 ETTm1 ETTm2 ECL Weather PEMS03 PEMS04 PEMS07 PEMS08 MSE
MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE
Median 0.713 0.606 0.725 0.471 0.692 0.580 0.741 0.462 0.994 0.825 0.997 0.492
0.679 0.603 0.727 0.638 0.740 0.639 0.719 0.646 Last 0.402 0.461 0.071 0.136
0.312 0.400 0.041 0.110 0.917 0.813 0.703 0.340 0.450 0.490 0.463 0.502 0.482
0.502 0.422 0.480 Autoformer(2021) 0.425 0.493 0.532 0.472 0.345 0.407 0.139
0.263 0.082 0.205 0.423 0.342 1.168 0.880 0.218 0.356 0.250 0.384 1.725 1.061
Fedformer(2022) 0.270 0.388 0.806 0.536 0.043 0.145 0.026 0.113 0.080 0.204
0.162 0.138 0.151 0.285 0.357 0.445 0.293 0.410 0.223 0.323 Dlinear(2023)
0.215 0.331 0.120 0.227 0.153 0.284 0.219 0.345 0.179 0.323 0.263 0.253 0.114
0.266 0.126 0.280 0.156 0.319 0.178 0.341 iTransformer(2024) 0.480 0.507 0.101
0.199 0.154 0.279 0.104 0.235 0.058 0.160 0.344 0.277 0.080 0.199 0.091 0.216
0.068 0.186 0.107 0.235 BRITS(2018) 0.131 0.241 0.069 0.133 0.050 0.128 0.036
0.089 0.243 0.375 0.779 0.474 0.140 0.263 0.252 0.363 0.226 0.352 0.212 0.332
PatchTST(2023) 0.131 0.253 0.018 0.087 0.059 0.164 0.009 0.061 0.082 0.219
0.198 0.107 0.050 0.162 0.068 0.187 0.043 0.150 0.062 0.177 TimesNet(2022)
0.114 0.235 0.015 0.081 0.036 0.120 0.008 0.057 0.323 0.425 0.635 0.272 0.088
0.203 0.140 0.255 0.120 0.238 0.102 0.215 SAITS(2023) 0.131 0.238 0.177 0.169
0.063 0.131 0.132 0.125 0.405 0.488 0.935 0.492 0.154 0.283 0.264 0.375 0.224
0.346 0.245 0.360 GPT4TS(2024) 0.111 0.221 0.018 0.081 0.032 0.110 0.009 0.055
0.233 0.356 0.973 0.317 0.088 0.206 0.144 0.258 0.118 0.236 0.096 0.213
NuwaTS(specific) 0.129 0.250 0.017 0.086 0.049 0.144 0.009 0.060 0.049 0.149
0.241 0.114 0.040 0.137 0.049 0.147 0.032 0.118 0.051 0.150 PatchTST(one-for-
all) 0.120 0.240 0.015 0.080 0.045 0.139 0.009 0.060 0.075 0.202 0.177 0.100
0.047 0.157 0.057 0.169 0.041 0.146 0.051 0.157 NuwaTS(one-for-all) 0.103
0.209 0.014 0.075 0.035 0.111 0.008 0.054 0.049 0.145 0.148 0.072 0.038 0.135
0.049 0.148 0.032 0.118 0.044 0.136 NuwaTS(finetuned) 0.101 0.207 0.014 0.075
0.035 0.111 0.008 0.053 0.047 0.144 0.140 0.071 0.039 0.135 0.050 0.149 0.032
0.118 0.044 0.136

Table 15: Detailed Imputation results on ETTs, ECL and weather with missing
rate set to 0.3. We set the input length to 96. A lower MSE or MAE indicates a
better imputation performance.

[b]  Model ETTh1 ETTh2 ETTm1 ETTm2 ECL Weather PEMS03 PEMS04 PEMS07 PEMS08 MSE
MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE
Median 0.707 0.605 0.717 0.469 0.689 0.579 0.740 0.462 0.988 0.826 0.991 0.492
0.681 0.604 0.730 0.639 0.742 0.640 0.722 0.647 Last 0.406 0.463 0.072 0.138
0.314 0.401 0.043 0.111 0.922 0.814 0.703 0.341 0.452 0.492 0.465 0.504 0.485
0.504 0.425 0.482 Autoformer(2021) 0.409 0.481 0.423 0.418 0.279 0.373 0.120
0.248 0.091 0.217 0.444 0.359 0.664 0.644 0.190 0.336 0.188 0.334 1.012 0.790
Fedformer(2022) 0.271 0.388 0.574 0.468 0.038 0.136 0.029 0.121 0.088 0.216
0.162 0.149 0.157 0.294 0.296 0.405 0.248 0.374 0.206 0.319 Dlinear(2023)
0.187 0.307 0.061 0.159 0.098 0.221 0.112 0.243 0.143 0.279 0.200 0.181 0.074
0.202 0.083 0.214 0.088 0.227 0.103 0.246 iTransformer(2024) 0.517 0.523 0.155
0.250 0.186 0.307 0.147 0.281 0.062 0.167 0.366 0.313 0.086 0.209 0.100 0.227
0.075 0.196 0.120 0.250 BRITS(2018) 0.143 0.254 0.076 0.138 0.055 0.135 0.040
0.094 0.254 0.384 0.783 0.474 0.140 0.263 0.253 0.363 0.225 0.351 0.214 0.334
PatchTST(2023) 0.138 0.259 0.018 0.088 0.059 0.163 0.009 0.062 0.086 0.222
0.213 0.108 0.050 0.160 0.066 0.183 0.043 0.149 0.060 0.173 TimesNet(2022)
0.120 0.241 0.017 0.083 0.038 0.123 0.009 0.057 0.329 0.428 0.652 0.273 0.090
0.206 0.142 0.257 0.121 0.239 0.103 0.217 SAITS(2023) 0.139 0.246 0.180 0.169
0.064 0.135 0.132 0.124 0.417 0.495 0.947 0.490 0.155 0.283 0.266 0.376 0.224
0.347 0.246 0.360 GPT4TS(2024) 0.124 0.235 0.019 0.083 0.038 0.118 0.009 0.056
0.244 0.364 0.953 0.317 0.089 0.207 0.146 0.260 0.119 0.238 0.097 0.212
NuwaTS(specific) 0.136 0.257 0.017 0.086 0.048 0.140 0.009 0.059 0.052 0.153
0.251 0.115 0.040 0.138 0.049 0.148 0.032 0.118 0.049 0.146 PatchTST(one-for-
all) 0.126 0.245 0.015 0.080 0.047 0.142 0.009 0.060 0.079 0.206 0.174 0.101
0.047 0.157 0.057 0.170 0.042 0.147 0.052 0.158 NuwaTS(one-for-all) 0.109
0.214 0.014 0.076 0.036 0.113 0.008 0.054 0.052 0.150 0.152 0.073 0.038 0.134
0.049 0.148 0.032 0.118 0.044 0.135 NuwaTS(finetuned) 0.106 0.211 0.014 0.075
0.036 0.113 0.008 0.053 0.050 0.148 0.146 0.071 0.038 0.134 0.049 0.148 0.032
0.118 0.044 0.135

Table 16: Detailed Imputation results on ETTs, ECL and weather with missing
rate set to 0.4. We set the input length to 96. A lower MSE or MAE indicates a
better imputation performance.

[b]  Model ETTh1 ETTh2 ETTm1 ETTm2 ECL Weather PEMS03 PEMS04 PEMS07 PEMS08 MSE
MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE
Median 0.710 0.606 0.716 0.469 0.687 0.579 0.736 0.461 0.988 0.827 0.984 0.493
0.688 0.607 0.736 0.642 0.749 0.643 0.728 0.650 Last 0.411 0.466 0.073 0.139
0.318 0.403 0.045 0.113 0.930 0.816 0.701 0.343 0.455 0.494 0.468 0.506 0.488
0.506 0.428 0.484 Autoformer(2021) 0.426 0.487 0.347 0.367 0.238 0.349 0.113
0.239 0.103 0.232 0.501 0.393 0.419 0.500 0.231 0.374 0.213 0.356 0.632 0.608
Fedformer(2022) 0.280 0.395 0.447 0.421 0.053 0.164 0.033 0.129 0.100 0.231
0.174 0.163 0.171 0.310 0.255 0.378 0.217 0.350 0.199 0.321 Dlinear(2023)
0.194 0.311 0.046 0.138 0.084 0.200 0.055 0.167 0.144 0.275 0.182 0.149 0.073
0.200 0.080 0.207 0.064 0.185 0.073 0.195 iTransformer(2024) 0.565 0.542 0.230
0.309 0.235 0.345 0.213 0.341 0.069 0.175 0.412 0.357 0.094 0.221 0.110 0.240
0.083 0.208 0.135 0.266 BRITS(2018) 0.162 0.273 0.086 0.147 0.062 0.145 0.047
0.101 0.270 0.395 0.781 0.472 0.140 0.264 0.253 0.364 0.224 0.350 0.217 0.336
PatchTST(2023) 0.148 0.269 0.019 0.089 0.061 0.163 0.009 0.062 0.093 0.227
0.218 0.11 0.050 0.160 0.064 0.180 0.044 0.148 0.058 0.169 TimesNet(2022)
0.128 0.249 0.017 0.085 0.041 0.128 0.009 0.059 0.336 0.433 0.664 0.274 0.093
0.210 0.144 0.259 0.123 0.242 0.106 0.221 SAITS(2023) 0.150 0.257 0.191 0.174
0.066 0.139 0.132 0.126 0.429 0.503 0.936 0.486 0.155 0.284 0.267 0.377 0.226
0.347 0.246 0.361 GPT4TS(2024) 0.143 0.252 0.021 0.085 0.045 0.128 0.010 0.058
0.258 0.375 0.945 0.317 0.090 0.208 0.149 0.262 0.122 0.240 0.099 0.214
NuwaTS(specific) 0.145 0.265 0.017 0.086 0.050 0.140 0.009 0.059 0.058 0.160
0.264 0.117 0.041 0.139 0.050 0.149 0.033 0.119 0.049 0.145 PatchTST(one-for-
all) 0.136 0.254 0.016 0.082 0.051 0.147 0.009 0.060 0.085 0.212 0.187 0.104
0.048 0.159 0.058 0.171 0.043 0.148 0.052 0.159 NuwaTS(one-for-all) 0.119
0.225 0.015 0.077 0.040 0.118 0.009 0.055 0.057 0.157 0.162 0.075 0.039 0.135
0.050 0.148 0.033 0.119 0.044 0.136 NuwaTS(finetuned) 0.115 0.220 0.014 0.076
0.040 0.117 0.008 0.054 0.055 0.155 0.168 0.074 0.039 0.135 0.050 0.148 0.033
0.119 0.044 0.136

Table 17: Detailed Imputation results on ETTs, ECL and weather with missing
rate set to 0.5. We set the input length to 96. A lower MSE or MAE indicates a
better imputation performance.

[b]  Model ETTh1 ETTh2 ETTm1 ETTm2 ECL Weather PEMS03 PEMS04 PEMS07 PEMS08 MSE
MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE
Median 0.708 0.605 0.721 0.470 0.686 0.578 0.735 0.461 0.994 0.830 0.991 0.493
0.669 0.597 0.721 0.634 0.731 0.634 0.712 0.642 Last 0.416 0.469 0.078 0.142
0.323 0.406 0.049 0.115 0.941 0.820 0.716 0.345 0.460 0.498 0.474 0.509 0.493
0.509 0.433 0.487 Autoformer(2021) 0.469 0.507 0.320 0.336 0.223 0.339 0.116
0.237 0.119 0.249 0.571 0.432 0.329 0.441 0.301 0.432 0.276 0.408 0.461 0.515
Fedformer(2022) 0.298 0.407 0.389 0.394 0.061 0.176 0.038 0.139 0.114 0.248
0.192 0.182 0.197 0.336 0.241 0.371 0.210 0.346 0.203 0.331 Dlinear(2023)
0.239 0.345 0.084 0.183 0.122 0.240 0.062 0.177 0.189 0.319 0.212 0.186 0.120
0.268 0.125 0.272 0.094 0.233 0.099 0.234 iTransformer(2024) 0.620 0.564 0.321
0.372 0.297 0.390 0.299 0.405 0.077 0.187 0.473 0.405 0.104 0.234 0.122 0.253
0.093 0.220 0.151 0.283 BRITS(2018) 0.184 0.294 0.097 0.157 0.072 0.158 0.055
0.109 0.290 0.410 0.790 0.471 0.141 0.264 0.255 0.366 0.223 0.349 0.221 0.339
PatchTST(2023) 0.162 0.281 0.020 0.090 0.065 0.166 0.010 0.063 0.102 0.236
0.239 0.113 0.051 0.161 0.064 0.178 0.045 0.149 0.059 0.168 TimesNet(2022)
0.140 0.260 0.019 0.087 0.046 0.135 0.010 0.060 0.345 0.439 0.683 0.276 0.096
0.214 0.146 0.262 0.126 0.245 0.109 0.225 SAITS(2023) 0.166 0.271 0.203 0.180
0.070 0.146 0.134 0.131 0.441 0.510 0.935 0.483 0.156 0.284 0.269 0.379 0.227
0.348 0.248 0.362 GPT4TS(2024) 0.169 0.273 0.022 0.088 0.056 0.141 0.011 0.060
0.275 0.387 0.939 0.317 0.093 0.211 0.152 0.265 0.124 0.243 0.102 0.217
NuwaTS(specific) 0.158 0.277 0.018 0.087 0.054 0.144 0.009 0.060 0.065 0.171
0.280 0.120 0.043 0.141 0.052 0.151 0.035 0.122 0.049 0.146 PatchTST(one-for-
all) 0.150 0.267 0.017 0.084 0.057 0.155 0.010 0.062 0.093 0.221 0.195 0.107
0.050 0.161 0.060 0.172 0.044 0.150 0.054 0.161 NuwaTS(one-for-all) 0.135
0.241 0.016 0.080 0.045 0.126 0.009 0.056 0.064 0.167 0.180 0.079 0.041 0.138
0.051 0.150 0.034 0.121 0.046 0.138 NuwaTS(finetuned) 0.128 0.233 0.015 0.078
0.045 0.124 0.009 0.055 0.062 0.164 0.188 0.077 0.041 0.138 0.051 0.150 0.034
0.121 0.046 0.138

Table 18: Detailed Imputation results on ETTs, ECL and weather with missing
rate set to 0.6. We set the input length to 96. A lower MSE or MAE indicates a
better imputation performance.

[b]  Model ETTh1 ETTh2 ETTm1 ETTm2 ECL Weather PEMS03 PEMS04 PEMS07 PEMS08 MSE
MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE
Median 0.710 0.607 0.718 0.470 0.689 0.580 0.736 0.461 1.005 0.829 0.990 0.495
0.686 0.606 0.735 0.642 0.747 0.642 0.728 0.650 Last 0.425 0.473 0.083 0.145
0.329 0.409 0.054 0.119 0.955 0.824 0.725 0.347 0.467 0.502 0.480 0.513 0.499
0.513 0.439 0.491 Autoformer(2021) 0.526 0.535 0.335 0.335 0.234 0.347 0.136
0.254 0.137 0.269 0.640 0.469 0.322 0.439 0.378 0.489 0.349 0.463 0.408 0.487
Fedformer(2022) 0.323 0.423 0.348 0.370 0.060 0.172 0.045 0.151 0.131 0.266
0.216 0.204 0.235 0.369 0.254 0.383 0.230 0.362 0.218 0.346 Dlinear(2023)
0.313 0.397 0.163 0.261 0.198 0.313 0.125 0.254 0.265 0.390 0.279 0.262 0.203
0.364 0.206 0.365 0.169 0.327 0.170 0.324 iTransformer(2024) 0.674 0.585 0.418
0.430 0.367 0.437 0.394 0.467 0.087 0.198 0.539 0.450 0.113 0.245 0.135 0.267
0.103 0.232 0.166 0.297 BRITS(2018) 0.210 0.318 0.112 0.169 0.085 0.175 0.066
0.119 0.314 0.427 0.791 0.471 0.141 0.265 0.257 0.369 0.224 0.348 0.225 0.343
PatchTST(2023) 0.180 0.297 0.021 0.092 0.071 0.173 0.010 0.065 0.115 0.249
0.259 0.118 0.053 0.164 0.066 0.180 0.047 0.152 0.060 0.170 TimesNet(2022)
0.155 0.273 0.020 0.090 0.053 0.144 0.010 0.062 0.356 0.446 0.696 0.277 0.100
0.219 0.149 0.266 0.129 0.249 0.112 0.230 SAITS(2023) 0.185 0.287 0.217 0.188
0.076 0.156 0.135 0.138 0.453 0.517 0.934 0.480 0.156 0.285 0.271 0.380 0.228
0.349 0.249 0.364 GPT4TS(2024) 0.197 0.295 0.025 0.092 0.069 0.157 0.012 0.063
0.293 0.401 0.924 0.318 0.096 0.215 0.155 0.269 0.128 0.247 0.106 0.222
NuwaTS(specific) 0.175 0.291 0.019 0.088 0.060 0.151 0.010 0.062 0.075 0.183
0.297 0.123 0.045 0.145 0.054 0.154 0.037 0.125 0.051 0.148 PatchTST(one-for-
all) 0.169 0.284 0.018 0.086 0.065 0.165 0.010 0.063 0.104 0.232 0.209 0.111
0.052 0.163 0.062 0.175 0.046 0.152 0.056 0.163 NuwaTS(one-for-all) 0.155
0.260 0.017 0.083 0.053 0.137 0.010 0.058 0.073 0.179 0.200 0.084 0.043 0.141
0.054 0.154 0.036 0.124 0.048 0.142 NuwaTS(finetuned) 0.146 0.249 0.016 0.080
0.051 0.134 0.010 0.057 0.070 0.176 0.202 0.082 0.043 0.141 0.053 0.153 0.036
0.124 0.048 0.141

Table 19: Detailed Imputation results on ETTs, ECL and weather with missing
rate set to 0.7. We set the input length to 96. A lower MSE or MAE indicates a
better imputation performance.

[b]  Model ETTh1 ETTh2 ETTm1 ETTm2 ECL Weather PEMS03 PEMS04 PEMS07 PEMS08 MSE
MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE
Median 0.717 0.610 0.718 0.470 0.694 0.583 0.744 0.463 1.015 0.833 1.002 0.499
0.710 0.623 0.758 0.657 0.775 0.658 0.750 0.665 Last 0.438 0.480 0.093 0.151
0.341 0.416 0.062 0.125 0.980 0.830 0.740 0.351 0.478 0.509 0.491 0.520 0.511
0.520 0.450 0.498 Autoformer(2021) 0.612 0.576 0.402 0.368 0.284 0.382 0.201
0.311 0.165 0.295 0.725 0.514 0.376 0.476 0.477 0.556 0.446 0.531 0.429 0.502
Fedformer(2022) 0.369 0.449 0.288 0.332 0.077 0.196 0.057 0.171 0.155 0.289
0.256 0.238 0.303 0.421 0.305 0.420 0.292 0.407 0.261 0.379 Dlinear(2023)
0.432 0.470 0.301 0.366 0.331 0.417 0.256 0.370 0.391 0.491 0.397 0.363 0.339
0.487 0.341 0.484 0.303 0.454 0.301 0.449 iTransformer(2024) 0.742 0.613 0.544
0.497 0.466 0.499 0.518 0.537 0.102 0.216 0.627 0.502 0.122 0.256 0.152 0.283
0.117 0.247 0.184 0.313 BRITS(2018) 0.251 0.353 0.135 0.189 0.107 0.203 0.082
0.136 0.351 0.452 0.793 0.471 0.143 0.267 0.262 0.373 0.226 0.350 0.232 0.349
PatchTST(2023) 0.208 0.320 0.023 0.095 0.082 0.186 0.011 0.067 0.139 0.273
0.292 0.126 0.058 0.170 0.070 0.185 0.051 0.159 0.065 0.176 TimesNet(2022)
0.180 0.296 0.023 0.094 0.066 0.160 0.011 0.065 0.374 0.458 0.710 0.280 0.105
0.227 0.155 0.273 0.135 0.257 0.118 0.237 SAITS(2023) 0.217 0.312 0.233 0.199
0.089 0.172 0.137 0.146 0.467 0.525 0.925 0.477 0.158 0.286 0.274 0.383 0.230
0.350 0.251 0.366 GPT4TS(2024) 0.239 0.328 0.028 0.097 0.092 0.182 0.013 0.067
0.320 0.420 0.918 0.320 0.102 0.223 0.161 0.275 0.134 0.254 0.112 0.229
NuwaTS(specific) 0.202 0.312 0.021 0.092 0.072 0.166 0.011 0.064 0.091 0.202
0.338 0.130 0.049 0.151 0.058 0.160 0.040 0.131 0.055 0.154 PatchTST(one-for-
all) 0.199 0.308 0.020 0.091 0.079 0.182 0.011 0.066 0.122 0.249 0.232 0.118
0.056 0.168 0.066 0.179 0.049 0.157 0.060 0.168 NuwaTS(one-for-all) 0.188
0.289 0.019 0.087 0.067 0.157 0.011 0.061 0.090 0.198 0.229 0.092 0.047 0.147
0.058 0.159 0.040 0.130 0.052 0.147 NuwaTS(finetuned) 0.175 0.275 0.018 0.084
0.064 0.150 0.011 0.060 0.086 0.195 0.231 0.089 0.047 0.147 0.057 0.159 0.040
0.130 0.052 0.147

Table 20: Detailed Imputation results on ETTs, ECL and weather with missing
rate set to 0.8. We set the input length to 96. A lower MSE or MAE indicates a
better imputation performance.

[b]  Model ETTh1 ETTh2 ETTm1 ETTm2 ECL Weather PEMS03 PEMS04 PEMS07 PEMS08 MSE
MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE
Median 0.734 0.616 0.740 0.475 0.709 0.588 0.749 0.465 1.023 0.844 1.005 0.502
0.708 0.617 0.762 0.653 0.774 0.654 0.753 0.662 Last 0.463 0.491 0.110 0.162
0.362 0.427 0.077 0.134 1.019 0.840 0.766 0.358 0.498 0.521 0.511 0.532 0.531
0.532 0.469 0.510 Autoformer(2021) 0.724 0.624 0.523 0.432 0.381 0.443 0.311
0.398 0.200 0.324 0.811 0.557 0.477 0.540 0.581 0.620 0.551 0.598 0.514 0.552
Fedformer(2022) 0.448 0.491 0.264 0.304 0.123 0.248 0.078 0.198 0.185 0.315
0.327 0.295 0.403 0.490 0.398 0.483 0.399 0.481 0.348 0.441 Dlinear(2023)
0.574 0.544 0.480 0.469 0.499 0.521 0.465 0.506 0.536 0.586 0.557 0.469 0.502
0.605 0.502 0.598 0.470 0.576 0.468 0.571 iTransformer(2024) 0.810 0.642 0.671
0.557 0.581 0.562 0.647 0.602 0.127 0.242 0.720 0.552 0.128 0.262 0.170 0.300
0.132 0.263 0.208 0.334 BRITS(2018) 0.305 0.394 0.170 0.216 0.144 0.244 0.108
0.160 0.400 0.484 0.799 0.473 0.146 0.271 0.269 0.379 0.232 0.355 0.241 0.357
PatchTST(2023) 0.248 0.351 0.027 0.101 0.102 0.208 0.013 0.071 0.186 0.316
0.341 0.137 0.067 0.180 0.080 0.196 0.059 0.170 0.073 0.185 TimesNet(2022)
0.221 0.329 0.028 0.101 0.093 0.191 0.013 0.070 0.401 0.474 0.724 0.286 0.113
0.237 0.162 0.281 0.143 0.267 0.125 0.246 SAITS(2023) 0.269 0.347 0.253 0.214
0.111 0.198 0.139 0.155 0.484 0.534 0.924 0.477 0.160 0.289 0.277 0.385 0.232
0.352 0.255 0.369 GPT4TS(2024) 0.293 0.367 0.032 0.104 0.127 0.218 0.015 0.072
0.358 0.446 0.911 0.325 0.112 0.235 0.169 0.284 0.143 0.265 0.123 0.242
NuwaTS(specific) 0.242 0.343 0.024 0.097 0.092 0.189 0.013 0.068 0.119 0.231
0.372 0.139 0.057 0.160 0.066 0.169 0.047 0.141 0.062 0.164 PatchTST(one-for-
all) 0.243 0.342 0.023 0.098 0.104 0.208 0.013 0.072 0.153 0.275 0.284 0.132
0.064 0.176 0.073 0.188 0.056 0.165 0.067 0.176 NuwaTS(one-for-all) 0.236
0.327 0.023 0.094 0.093 0.187 0.012 0.066 0.116 0.226 0.267 0.104 0.054 0.157
0.065 0.169 0.046 0.140 0.059 0.157 NuwaTS(finetuned) 0.218 0.310 0.021 0.090
0.085 0.175 0.012 0.064 0.112 0.223 0.269 0.100 0.054 0.156 0.064 0.168 0.046
0.139 0.058 0.156

Table 21: Detailed Imputation results on ETTs, ECL and weather with missing
rate set to 0.9. We set the input length to 96. A lower MSE or MAE indicates a
better imputation performance.

[b]  Model ETTh1 ETTh2 ETTm1 ETTm2 ECL Weather PEMS03 PEMS04 PEMS07 PEMS08 MSE
MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE MSE MAE
Median 0.786 0.635 0.768 0.483 0.748 0.603 0.772 0.472 1.086 0.870 1.027 0.513
0.733 0.626 0.795 0.665 0.805 0.667 0.788 0.676 Last 0.526 0.521 0.156 0.191
0.419 0.456 0.121 0.164 1.112 0.863 0.834 0.377 0.555 0.556 0.569 0.566 0.590
0.566 0.526 0.543 Autoformer(2021) 0.889 0.690 0.735 0.544 0.612 0.563 0.535
0.541 0.268 0.372 0.937 0.616 0.676 0.654 0.731 0.705 0.704 0.688 0.718 0.657
Fedformer(2022) 0.650 0.592 0.445 0.393 0.194 0.307 0.168 0.277 0.239 0.355
0.500 0.402 0.596 0.615 0.592 0.607 0.609 0.618 0.557 0.575 Dlinear(2023)
0.768 0.634 0.711 0.579 0.721 0.635 0.699 0.625 0.748 0.707 0.749 0.567 0.728
0.738 0.712 0.720 0.707 0.715 0.706 0.709 iTransformer(2024) 0.895 0.679 0.826
0.625 0.754 0.648 0.810 0.675 0.196 0.304 0.842 0.610 0.168 0.310 0.215 0.344
0.160 0.294 0.292 0.417 BRITS(2018) 0.415 0.464 0.244 0.277 0.245 0.336 0.171
0.218 0.502 0.548 0.825 0.483 0.155 0.282 0.281 0.389 0.246 0.369 0.258 0.371
PatchTST(2023) 0.328 0.404 0.038 0.114 0.159 0.260 0.018 0.082 0.344 0.436
0.482 0.172 0.098 0.211 0.116 0.230 0.094 0.205 0.104 0.216 TimesNet(2022)
0.328 0.407 0.039 0.115 0.191 0.278 0.018 0.081 0.478 0.520 0.750 0.307 0.145
0.269 0.188 0.306 0.170 0.293 0.152 0.274 SAITS(2023) 0.386 0.416 0.293 0.241
0.176 0.263 0.140 0.170 0.516 0.554 0.926 0.482 0.166 0.295 0.283 0.390 0.236
0.356 0.262 0.374 GPT4TS(2024) 0.390 0.431 0.043 0.118 0.213 0.294 0.020 0.083
0.460 0.511 0.925 0.338 0.148 0.271 0.200 0.313 0.176 0.296 0.158 0.278
NuwaTS(specific) 0.329 0.402 0.034 0.110 0.153 0.248 0.017 0.078 0.219 0.320
0.466 0.170 0.085 0.192 0.095 0.201 0.072 0.172 0.090 0.196 PatchTST(one-for-
all) 0.344 0.410 0.034 0.113 0.179 0.275 0.019 0.086 0.308 0.389 0.440 0.173
0.093 0.206 0.105 0.218 0.085 0.196 0.094 0.204 NuwaTS(one-for-all) 0.331
0.394 0.032 0.109 0.172 0.264 0.018 0.080 0.219 0.309 0.377 0.141 0.083 0.189
0.095 0.201 0.074 0.174 0.086 0.188 NuwaTS(finetuned) 0.312 0.379 0.030 0.105
0.149 0.239 0.016 0.076 0.201 0.301 0.379 0.136 0.080 0.186 0.092 0.198 0.071
0.171 0.083 0.185

##  Appendix F Limitation

The model was trained on time series segments fixed at a length of 96,
imputing the masked series based on the unmasked portions. Although our model
can adapt to different domains and missing patterns, showing strong
adaptability after domain-transfer fine-tuning, in practical applications, the
model may require further fine-tuning when dealing with segments longer than
96 as well as the segments that entirely missed. Additionally, NuwaTS discards
multivariate correlations, relying on temporal information and identified
domain information, which may degrade the imputation performance. For
instance, if a sensor loses a large amount of data, the data from adjacent
sensors can be used as supplementary information. Future work will aim to
improve NuwaTS’s capability in handling longer missing segments and
incorporate multi-dimensional correlations.

## NeurIPS Paper Checklist

  1. 1.

Claims

  2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

  3. Answer: [Yes] .

  4. Justification: This paper aims to introduce a one-for-all imputation method for incomplete time series, applicable to any domain and any missing patterns.

  5. Guidelines:

     * •

The answer NA means that the abstract and introduction do not include the
claims made in the paper.

     * •

The abstract and/or introduction should clearly state the claims made,
including the contributions made in the paper and important assumptions and
limitations. A No or NA answer to this question will not be perceived well by
the reviewers.

     * •

The claims made should match theoretical and experimental results, and reflect
how much the results can be expected to generalize to other settings.

     * •

It is fine to include aspirational goals as motivation as long as it is clear
that these goals are not attained by the paper.

  6. 2.

Limitations

  7. Question: Does the paper discuss the limitations of the work performed by the authors?

  8. Answer: [Yes] .

  9. Justification: In the Appendix F, we discuss the limitation of NuwaTS.

  10. Guidelines:

     * •

The answer NA means that the paper has no limitation while the answer No means
that the paper has limitations, but those are not discussed in the paper.

     * •

The authors are encouraged to create a separate "Limitations" section in their
paper.

     * •

The paper should point out any strong assumptions and how robust the results
are to violations of these assumptions (e.g., independence assumptions,
noiseless settings, model well-specification, asymptotic approximations only
holding locally). The authors should reflect on how these assumptions might be
violated in practice and what the implications would be.

     * •

The authors should reflect on the scope of the claims made, e.g., if the
approach was only tested on a few datasets or with a few runs. In general,
empirical results often depend on implicit assumptions, which should be
articulated.

     * •

The authors should reflect on the factors that influence the performance of
the approach. For example, a facial recognition algorithm may perform poorly
when image resolution is low or images are taken in low lighting. Or a speech-
to-text system might not be used reliably to provide closed captions for
online lectures because it fails to handle technical jargon.

     * •

The authors should discuss the computational efficiency of the proposed
algorithms and how they scale with dataset size.

     * •

If applicable, the authors should discuss possible limitations of their
approach to address problems of privacy and fairness.

     * •

While the authors might fear that complete honesty about limitations might be
used by reviewers as grounds for rejection, a worse outcome might be that
reviewers discover limitations that aren’t acknowledged in the paper. The
authors should use their best judgment and recognize that individual actions
in favor of transparency play an important role in developing norms that
preserve the integrity of the community. Reviewers will be specifically
instructed to not penalize honesty concerning limitations.

  11. 3.

Theory Assumptions and Proofs

  12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

  13. Answer: [Yes] .

  14. Justification: The detailed assumptions, proofs and formulas can be found in Appendix B.

  15. Guidelines:

     * •

The answer NA means that the paper does not include theoretical results.

     * •

All the theorems, formulas, and proofs in the paper should be numbered and
cross-referenced.

     * •

All assumptions should be clearly stated or referenced in the statement of any
theorems.

     * •

The proofs can either appear in the main paper or the supplemental material,
but if they appear in the supplemental material, the authors are encouraged to
provide a short proof sketch to provide intuition.

     * •

Inversely, any informal proof provided in the core of the paper should be
complemented by formal proofs provided in appendix or supplemental material.

     * •

Theorems and Lemmas that the proof relies upon should be properly referenced.

  16. 4.

Experimental Result Reproducibility

  17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

  18. Answer: [Yes] .

  19. Justification: The code was uploaded in https://anonymous.4open.science/r/NuwaTS-85FB and the detailed implementation settings are listed in Appendix A.

  20. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

If the paper includes experiments, a No answer to this question will not be
perceived well by the reviewers: Making the paper reproducible is important,
regardless of whether the code and data are provided or not.

     * •

If the contribution is a dataset and/or model, the authors should describe the
steps taken to make their results reproducible or verifiable.

     * •

Depending on the contribution, reproducibility can be accomplished in various
ways. For example, if the contribution is a novel architecture, describing the
architecture fully might suffice, or if the contribution is a specific model
and empirical evaluation, it may be necessary to either make it possible for
others to replicate the model with the same dataset, or provide access to the
model. In general. releasing code and data is often one good way to accomplish
this, but reproducibility can also be provided via detailed instructions for
how to replicate the results, access to a hosted model (e.g., in the case of a
large language model), releasing of a model checkpoint, or other means that
are appropriate to the research performed.

     * •

While NeurIPS does not require releasing code, the conference does require all
submissions to provide some reasonable avenue for reproducibility, which may
depend on the nature of the contribution. For example

       1. (a)

If the contribution is primarily a new algorithm, the paper should make it
clear how to reproduce that algorithm.

       2. (b)

If the contribution is primarily a new model architecture, the paper should
describe the architecture clearly and fully.

       3. (c)

If the contribution is a new model (e.g., a large language model), then there
should either be a way to access this model for reproducing the results or a
way to reproduce the model (e.g., with an open-source dataset or instructions
for how to construct the dataset).

       4. (d)

We recognize that reproducibility may be tricky in some cases, in which case
authors are welcome to describe the particular way they provide for
reproducibility. In the case of closed-source models, it may be that access to
the model is limited in some way (e.g., to registered users), but it should be
possible for other researchers to have some path to reproducing or verifying
the results.

  21. 5.

Open access to data and code

  22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

  23. Answer: [Yes] .

  24. Justification: The code was uploaded in https://anonymous.4open.science/r/NuwaTS-85FB and the datasets come from open source repository https://github.com/thuml/Time-Series-Library and https://github.com/liuxu77/LargeST.

  25. Guidelines:

     * •

The answer NA means that paper does not include experiments requiring code.

     * •

Please see the NeurIPS code and data submission guidelines
(https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.

     * •

While we encourage the release of code and data, we understand that this might
not be possible, so “No” is an acceptable answer. Papers cannot be rejected
simply for not including code, unless this is central to the contribution
(e.g., for a new open-source benchmark).

     * •

The instructions should contain the exact command and environment needed to
run to reproduce the results. See the NeurIPS code and data submission
guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more
details.

     * •

The authors should provide instructions on data access and preparation,
including how to access the raw data, preprocessed data, intermediate data,
and generated data, etc.

     * •

The authors should provide scripts to reproduce all experimental results for
the new proposed method and baselines. If only a subset of experiments are
reproducible, they should state which ones are omitted from the script and
why.

     * •

At submission time, to preserve anonymity, the authors should release
anonymized versions (if applicable).

     * •

Providing as much information as possible in supplemental material (appended
to the paper) is recommended, but including URLs to data and code is
permitted.

  26. 6.

Experimental Setting/Details

  27. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

  28. Answer: [Yes] .

  29. Justification: We provided detailed experimental settings in Section 4 and Appendix A.

  30. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The experimental setting should be presented in the core of the paper to a
level of detail that is necessary to appreciate the results and make sense of
them.

     * •

The full details can be provided either with the code, in appendix, or as
supplemental material.

  31. 7.

Experiment Statistical Significance

  32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

  33. Answer: [No] .

  34. Justification: The common practice of time series imputation task is reporting the average of multiple experiments, rather than providing error bars.

  35. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The authors should answer "Yes" if the results are accompanied by error bars,
confidence intervals, or statistical significance tests, at least for the
experiments that support the main claims of the paper.

     * •

The factors of variability that the error bars are capturing should be clearly
stated (for example, train/test split, initialization, random drawing of some
parameter, or overall run with given experimental conditions).

     * •

The method for calculating the error bars should be explained (closed form
formula, call to a library function, bootstrap, etc.)

     * •

The assumptions made should be given (e.g., Normally distributed errors).

     * •

It should be clear whether the error bar is the standard deviation or the
standard error of the mean.

     * •

It is OK to report 1-sigma error bars, but one should state it. The authors
should preferably report a 2-sigma error bar than state that they have a 96%
CI, if the hypothesis of Normality of errors is not verified.

     * •

For asymmetric distributions, the authors should be careful not to show in
tables or figures symmetric error bars that would yield results that are out
of range (e.g. negative error rates).

     * •

If error bars are reported in tables or plots, The authors should explain in
the text how they were calculated and reference the corresponding figures or
tables in the text.

  36. 8.

Experiments Compute Resources

  37. Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

  38. Answer: [Yes] .

  39. Justification: We provide details of compute workers in Appendix A.2 and memory occupation in Appendix A.3.

  40. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The paper should indicate the type of compute workers CPU or GPU, internal
cluster, or cloud provider, including relevant memory and storage.

     * •

The paper should provide the amount of compute required for each of the
individual experimental runs as well as estimate the total compute.

     * •

The paper should disclose whether the full research project required more
compute than the experiments reported in the paper (e.g., preliminary or
failed experiments that didn’t make it into the paper).

  41. 9.

Code Of Ethics

  42. Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

  43. Answer: [Yes] .

  44. Justification: We follow the NeurIPS Code of Ethics in this paper.

  45. Guidelines:

     * •

The answer NA means that the authors have not reviewed the NeurIPS Code of
Ethics.

     * •

If the authors answer No, they should explain the special circumstances that
require a deviation from the Code of Ethics.

     * •

The authors should make sure to preserve anonymity (e.g., if there is a
special consideration due to laws or regulations in their jurisdiction).

  46. 10.

Broader Impacts

  47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

  48. Answer: [Yes] .

  49. Justification: We believe that our Nuwa model not only plays an important role in the time series domain but also that its one-for-all training paradigm and domain-specific fine-tuning method will inspire broader applications in the field of data mining.

  50. Guidelines:

     * •

The answer NA means that there is no societal impact of the work performed.

     * •

If the authors answer NA or No, they should explain why their work has no
societal impact or why the paper does not address societal impact.

     * •

Examples of negative societal impacts include potential malicious or
unintended uses (e.g., disinformation, generating fake profiles,
surveillance), fairness considerations (e.g., deployment of technologies that
could make decisions that unfairly impact specific groups), privacy
considerations, and security considerations.

     * •

The conference expects that many papers will be foundational research and not
tied to particular applications, let alone deployments. However, if there is a
direct path to any negative applications, the authors should point it out. For
example, it is legitimate to point out that an improvement in the quality of
generative models could be used to generate deepfakes for disinformation. On
the other hand, it is not needed to point out that a generic algorithm for
optimizing neural networks could enable people to train models that generate
Deepfakes faster.

     * •

The authors should consider possible harms that could arise when the
technology is being used as intended and functioning correctly, harms that
could arise when the technology is being used as intended but gives incorrect
results, and harms following from (intentional or unintentional) misuse of the
technology.

     * •

If there are negative societal impacts, the authors could also discuss
possible mitigation strategies (e.g., gated release of models, providing
defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms
to monitor how a system learns from feedback over time, improving the
efficiency and accessibility of ML).

  51. 11.

Safeguards

  52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

  53. Answer: [N/A] .

  54. Justification: This paper poses no such risks. The datasets are commonly used benchmark in time series imputation task.

  55. Guidelines:

     * •

The answer NA means that the paper poses no such risks.

     * •

Released models that have a high risk for misuse or dual-use should be
released with necessary safeguards to allow for controlled use of the model,
for example by requiring that users adhere to usage guidelines or restrictions
to access the model or implementing safety filters.

     * •

Datasets that have been scraped from the Internet could pose safety risks. The
authors should describe how they avoided releasing unsafe images.

     * •

We recognize that providing effective safeguards is challenging, and many
papers do not require this, but we encourage authors to take this into account
and make a best faith effort.

  56. 12.

Licenses for existing assets

  57. Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

  58. Answer: [Yes] .

  59. Justification: Yes, we have.

  60. Guidelines:

     * •

The answer NA means that the paper does not use existing assets.

     * •

The authors should cite the original paper that produced the code package or
dataset.

     * •

The authors should state which version of the asset is used and, if possible,
include a URL.

     * •

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

     * •

For scraped data from a particular source (e.g., website), the copyright and
terms of service of that source should be provided.

     * •

If assets are released, the license, copyright information, and terms of use
in the package should be provided. For popular datasets,
paperswithcode.com/datasets has curated licenses for some datasets. Their
licensing guide can help determine the license of a dataset.

     * •

For existing datasets that are re-packaged, both the original license and the
license of the derived asset (if it has changed) should be provided.

     * •

If this information is not available online, the authors are encouraged to
reach out to the asset’s creators.

  61. 13.

New Assets

  62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

  63. Answer: [Yes] .

  64. Justification: This paper follows CC 4.0 and the code is in an anonymous URL https://anonymous.4open.science/r/NuwaTS-85FB.

  65. Guidelines:

     * •

The answer NA means that the paper does not release new assets.

     * •

Researchers should communicate the details of the dataset/code/model as part
of their submissions via structured templates. This includes details about
training, license, limitations, etc.

     * •

The paper should discuss whether and how consent was obtained from people
whose asset is used.

     * •

At submission time, remember to anonymize your assets (if applicable). You can
either create an anonymized URL or include an anonymized zip file.

  66. 14.

Crowdsourcing and Research with Human Subjects

  67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

  68. Answer: [N/A] .

  69. Justification: This paper does not involve crowdsourcing nor research with human subjects.

  70. Guidelines:

     * •

The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.

     * •

Including this information in the supplemental material is fine, but if the
main contribution of the paper involves human subjects, then as much detail as
possible should be included in the main paper.

     * •

According to the NeurIPS Code of Ethics, workers involved in data collection,
curation, or other labor should be paid at least the minimum wage in the
country of the data collector.

  71. 15.

Institutional Review Board (IRB) Approvals or Equivalent for Research with
Human Subjects

  72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

  73. Answer: [N/A] .

  74. Justification: This paper does not involve crowdsourcing nor research with human subjects.

  75. Guidelines:

     * •

The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.

     * •

Depending on the country in which research is conducted, IRB approval (or
equivalent) may be required for any human subjects research. If you obtained
IRB approval, you should clearly state this in the paper.

     * •

We recognize that the procedures for this may vary significantly between
institutions and locations, and we expect authors to adhere to the NeurIPS
Code of Ethics and the guidelines for their institution.

     * •

For initial submissions, do not include any information that would break
anonymity (if applicable), such as the institution conducting the review.