11institutetext: Rossum.ai
11email: {matyas.skalicky, stepan.simsa, michal.uricar<EMAIL_ADDRESS>
# Business Document Information Extraction: Towards Practical Benchmarks

Matyáš Skalický  0000-0002-0197-7134 Štěpán Šimsa  0000-0001-6687-1210 Michal
Uřičář  0000-0002-2606-4470 Milan Šulc  0000-0002-6321-0131

###### Abstract

Information extraction from semi-structured documents is crucial for
frictionless business-to-business (B2B) communication. While machine learning
problems related to Document Information Extraction (IE) have been studied for
decades, many common problem definitions and benchmarks do not reflect domain-
specific aspects and practical needs for automating B2B document
communication. We review the landscape of Document IE problems, datasets and
benchmarks. We highlight the practical aspects missing in the common
definitions and define the Key Information Localization and Extraction (KILE)
and Line Item Recognition (LIR) problems. There is a lack of relevant datasets
and benchmarks for Document IE on semi-structured business documents as their
content is typically legally protected or sensitive. We discuss potential
sources of available documents including synthetic data.

###### Keywords:

Document Understanding Survey Benchmarks Datasets

##  1 Introduction

The majority of B2B communication takes place through the exchange of semi-
structured111The term semi-structured documents is commonly used in different
meanings: Some use it for text files containing semi-structured data [94],
such as XML files. We use the term to refer to visually rich documents without
a fixed layout [66]. business documents such as invoices, purchase orders and
delivery notes. Automating information extraction from such documents has a
considerable potential to reduce repetitive manual work and to streamline
business communication. There have been efforts to provide standards for
electronic data interchange of business document metadata [5, 52, 7]. Despite,
e.g., electronic invoices taking place rapidly [16], the standards did not get
globally adopted, none of them prevails, and most are not inter-operable [21].

Machine learning (ML), natural language processing (NLP), and computer vision
problems related to Document Understanding and Document IE have been studied
for decades. Despite the major potential of IE from semi-structured business
documents, published research on Document IE often focuses on other domains
[100, 73, 13, 14, 39, 38], and many of the defined tasks and benchmarks do not
reflect domain-specific ML aspects and pitfalls of IE from semi-structured
business documents. Publications dealing with business documents typically use
private datasets [40, 19, 34, 59, 60, 47, 97, 69], hindering reproducibility
and cross-evaluation. This is caused by the absence of a large public dataset
of semi-structured business documents, noted by several authors [60, 77, 20,
41].

The contributions of this position paper are threefold: first, we provide a
review of IE problems, datasets and benchmarks relevant to semi-structured
business documents. Second, we identify unaddressed aspects of the tasks and
formulate new definitions for Key Information Localization and Extraction and
Line Item Recognition. Third, we stress the lack of a large-scale dataset of
semi-structured business documents and we discuss potential sources of
documents for such dataset.

##  2 Document Information Extraction Problems

###  2.1 Key Information Extraction (KIE) and Localization (KILE)

Most formulations of KIE come from NLP, where it is usually defined as the
extraction of values of a fixed set of entities/classes from an unstructured
text source into a structured form [43, 96, 56, 35, 96]. Based on the document
representation, Garncarek et al. [25] categorize KIE into three groups: (i)
sequence-based (working with serialized text [37]), (ii) graph-based (modeling
each doc./page as a graph with nodes corresponding to textual segments and
their features [17, 41, 32]), and (iii) grid-based (treating documents as a 2D
grid of token embeddings [40, 19]). Sequence-based KIE is closely related to
Named Entity Recognition (NER) [43] — a sub-task of KIE [96, 48] dealing with
sequence tagging problems. Borchmann et al. [6] say that (end-to-end) KIE,
unlike NER, does not assume that token-level annotations are available. The
task is also referred to as Slot Filling [60], meaning that a pre-defined slot
is filled with the extracted text.

The common definitions of KIE, as well as some of the datasets [73, 6], do not
require the location of the extracted information within the document. While
the localization is typically not crucial w.r.t. to the downstream task, it
plays a vital role in applications that require human validation. We extend
the definition by explicitly including the localization:

######  Definition 1 (KILE).

Given a document, the goal of Key Information Localization and Extraction
(KILE) is to localize (e.g., by a bounding box) fields of each pre-defined
category (key), to read out their values, and to aggregate the values to
extract the key information of each category.

Compared to Semantic Entity Recognition, as defined by Xu et al. [93],
bounding boxes in KILE are not limited to individual words (tokens).

###  2.2 Table Extraction and Line Items

Figure 1: Example of a table structure where field type is not uniquely
determined by its column. Source: https://rossum.ai/help/article/extracting-
nested-values-line-items.

Table Understanding [33] and Table Extraction (TE) [26, 98] are problems where
the tabular structure is crucial for IE. Unlike KIE, which outputs individual
fields independently, TE typically deals with a list of (line) items [19, 33,
60, 48, 4], each consisting of a tuple of fields (e.g., goods and price).

In simple tables, columns determine the field type and rows determine which
item the value belongs to. The table can therefore be represented as a grid
[68, 78]. A bottom-up approach [62, 98] can handle more complex tables as in
Fig. 1, without relying on a row or column detection. Detected cells or fields
can be converted to table structure (determining the line items and columns)
in a post-processing step, e.g., spatial clustering [98]. Other works [99, 46]
tackle the table extraction by directly solving an image-to-markup (e.g., XML
or TeX) problem.

We argue that the problem definition should not rely on the structure but
rather reflect the information to be extracted and stored. This is close to
the problem of detecting the area belonging to a single line item [19]. We
define Line Item (LI) and the task of Line Item Recognition (LIR) as follows:

######  Definition 2 (LI).

A Line Item is a tuple of fields describing a single object instance to be
extracted, e.g., a row in a table.

######  Definition 3 (LIR).

Given an image of a document page or of a table, the goal of Line Item
Recognition is to detect all LI present in the section, classify them into a
fixed set of classes (e.g., ordered item, discount, …) and for every detected
LI, localize and extract key information (as in Definition 1) related to it.

Note that this definition of LIR allows: (i) detection of several tables with
different item types, as well as different item types within a single table;
(ii) a single field (e.g., a date) to belong to several line items.

###  2.3 One-Shot Learning for Information Extraction

Layouts of business documents vary greatly, even within a single document
type. There are thousands of invoice templates available, and vendors often
further adjust them to their needs. Systems without the ability of fast re-
training are at risk of degraded performance when faced with a shift in the
incoming data distribution [29], such as when presented with previously unseen
layouts.

Improving IE with each processed document is known as a one-shot [20] / online
[74] template matching, case-based reasoning [29], or configuration-free IE
[69]. This includes systems that reuse annotations of similar documents [20,
69] or iteratively build and refine a representation of a document class [29,
53, 67, 17]. Annotations of documents’ templates are not part of any public IE
dataset of sufficient size.

###  2.4 Other Document IE Problems and Tasks

Optical Character Recognition (OCR) [72], handwritten OCR [31], scene text
recognition [95, 2], including (sub)word or text-line level predictions, are
standard problems with reviews and comparisons available [28, 54, 58, 36].
While highly relevant to the document IE, this paper aims at the “higher-
level“ document IE problems, often assuming text extracted from PDF or OCR is
available.

Document Layout Analysis (DLA) is typically posed as an object detection
problem: given a document page, find the minimum bounding boxes (or other area
representation [1, 15]) of layout elements such as Paragraph, Heading, Table,
Figure, or Caption. Most DLA datasets [23, 1, 100, 15] contain such layout
annotations for scientific / technical publications and magazines.

Extraction of Key-Value Pairs (KVP) refers to recognizing pairs of linked data
items where the key is used as a unique identifier for the value. This task
usually consists of semantic labeling and semantic linking [92, 93]. Contrary
to KIE, KVP extraction does not require the set of keys to be fixed. It also
assumes that both key and value are present in the document. This may be
useful, e.g., to extract data from unknown forms. However, in semi-structured
business documents, it is pretty standard that the keys of interests (known in
advance) are not explicitly present in the document.

Question Answering (QA), also known as Machine Reading Comprehension, is a
common problem in information retrieval and NLP. The goal is to automatically
answer questions formulated in natural language. Many NLP tasks can be
reformulated as QA [51, 42]. Similar to KIE, QA can be extended to incorporate
visual information to Visual Question Answering (VQA) [50]. VQA system may
also interpret and extract content from the figures, diagrams, and other non-
textual elements.

KIE can be formulated as an instance of VQA. However, we typically know which
classes of key information should be extracted, rendering the natural language
interface unnecessary.

##  3 Semi-Structured Business Document Datasets

Publications on business document IE are often based on private datasets [40,
19, 34, 59, 60, 47, 97, 69]. Due to the documents’ sensitive content, authors
are typically not allowed to share the experimental data. Large third-party
sources like common crawl are publicly available; however, re-publishing such
data may pose legal issues. For example, a large common crawl dataset of PDF
documents by Xu et al. [93] was not published, while pre-training on it was
crucial for the proposed method, and the C4 dataset [64] is shared in the form
of code that extracts it directly from Common Crawl.

Publicly available datasets for KI(L)E from business documents are summarized
in Table 1. However, most of them are relatively small and only contain a few
annotated field types. The two largest datasets consist entirely of receipts.
Table 1 does not include datasets without KIE annotations — RVL-CDIP [30]
(classification), FUNSD [27] and XFUND [93] (no fieldtypes), NIST [84] (forms
identification) and DocVQA [50] (QA) — and datasets we were not able to
download222For some only the annotations are available, without the original
PDFs/images. [8, 65, 101, 3].

Borchmann et al. [6] recently joined and re-formulated several existing
document IE datasets to build the DUE benchmark for several document
understanding tasks on different document domains. DeepForm [75] and Kleister
Charity [73] are the only subsets of DUE with business document KIE
annotations.

Table 1: Datasets related to KI(L)E from semi-structured business documents.

name | document type | docs | fieldtypes | source | multipage | lang. | type
---|---|---|---|---|---|---|---
WildReceipt [76] | receipts | $1740$ | $25$ | photo | no | en | KILE
Ghega [53] | patents/datasheets | $246$ |  $11$/$8$ | scan | yes | en | KILE
EPHOIE [79] | chinese forms | $1494$ | $10$ | scan | no | zh | KILE
CORD [61] | receipts | $11000$ | $42$ | photo | no | ind | KILE
DeepForm [75] | invoices, orders | $1000$ | $6$ | scan | yes | en | KILE
Kleister Charity [73] | financial reports | $2788$ | $8$ | scan | yes | en | KIE
Kleister NDA [73] | NDA documents | $540$ | $4$ | scan | yes | en | KIE
SROIE [35] | receipts | $973$ | $4$ | scan | no | en | KIE

While there are many datasets for Table Detection and LIR [22, 100, 71, 26,
24, 15, 70, 90, 87, 18, 63, 99, 98, 57], some of them are not accessible
anymore [26, 24, 70, 18]. We find only FinTabNet [98] and SynthTabNet [57] to
be relevant to us by covering complex financial tables.

##  4 Where to Get More Documents

#### Publicly Available Documents.

Business documents are typically not shared publicly due to their private
content, often including confidential and personal information. There are
exceptions to this rule — e.g., institutions such as governments or charities
have to make certain financial documents publicly available for transparency
reasons. Databases of such documents have already been used to create public
datasets for document IE: Several datasets333Some of the datasets are subsets:
FUNSD [27] $\subset$ RVL-CDIP [30] $\subset$ IIT-CDIP [44]. — IIT-CDIP [44],
RVL-CDIP [30], FUNSD [27], and DocVQA [50] — were built from documents from
the UCSF Industry Documents Library444A large proportion of the UCSF Industry
Document Library are old documents, often written on a typewriter, which
presents a domain shift w.r.t. today’s documents. [83]. Annual Reports of the
S&P 500 companies [88] were used to create FinTabNet [98]. Non-disclosure
agreements from the EDGAR555Automated crawling of the site not allowed:
https://www.sec.gov/os/accessing-edgar-data database [82], collected for the
U.S. Securities and Exchange Commission, were used for the Kleister-NDA [73]
dataset. The DeepForm dataset [75] consists of documents related to broadcast
stations from the FCC Public Inspection Files [86]. Financial records from the
Charity Commission [81] were used to create the Kleister-Charity dataset [73].
Several QA datasets [9, 55, 101] were also collected from open data sources
[91, 85, 80]. Other datasets were build via web search [49, 76, 10], from
Common Crawl666CC-MAIN-2022-05 contains almost 3 billion documents out of
which 0.84% are PDFs [89] – however, most of them are not semi-structured
business documents. [93, 64], Wikipedia [11, 12, 13], or platforms for sharing
scientific papers [39, 100, 38].

#### Synthetic documents.

Manual annotation is expensive, and the collection of data from public sources
may be limited by the presence of personal data or intellectual property. This
reasoning calls for leveraging synthetic datasets. Xu et al. [93] manually
replaced the content of publicly available documents with synthetic data.
Bensch et al. [4] generate synthetic invoice documents automatically. However,
we observe that the generated invoices have a plain style and do not resemble
the distribution of visual layouts of real business documents. Nassar et al.
[57] synthesized a table dataset of four appearance styles based on existing
datasets [99, 98, 46].

We consider three ways to define layouts to be filled with synthetic data: (i)
manual design of layouts which can be used to generate a high number of
documents with different semantically-matching values, but costly at scale
with increasing num. of layouts, (ii) extraction from public documents
followed by sensitive anonymization like in [93], and (iii) using a generative
model, e.g., to generate realistic layouts dissimilar to those already present
in the dataset. We consider such a problem statement an interesting open
research problem.

##  5 Discussion and Future Work

We argue that the problems of KILE and LIR, as defined in Sec. 2, are crucial
for automating B2B document communication where key information must be
extracted from localized fields and line items. The review of public datasets
in Sec. 3 shows that — except for receipts [76, 61, 35] — semi-structured
business documents like invoices, orders, and delivery notes are
underrepresented in document IE. Based on manual inspection of selected
documents from publicly available sources in Sec. 4, we noticed the
distribution of documents differs significantly among different sources. An
ideal dataset should cover a large variety of visual styles and layouts and
provide diagnostic subsets [6] to differentiate errors in various special
cases. Due to high annotation costs and possibly legally protected content of
business documents, synthetic data are a potentially affordable alternative
for building a large-scale dataset. While synthetic data have been proven
successful for OCR [45], the potential of data synthesis for business document
IE has not yet been fulfilled: existing attempts either target other tasks and
document types [93] or do not reflect the rich visual distribution of semi-
structured business documents [4]. An advantage of generating synthetic
documents of a given layout is the known layout annotation for benchmarking
one-shot information extraction.

To enable benchmarking of information extraction on data and tasks highly
relevant to real-world application scenarios, in our future work, we are
preparing a large-scale public dataset of semi-structured business documents,
following the observations and points made in this paper.

## References

  * [1] Antonacopoulos, A., Bridson, D., Papadopoulos, C., Pletschacher, S.: A realistic dataset for performance evaluation of document layout analysis. In: Proceedings of ICDAR. pp. 296–300. IEEE (2009)
  * [2] Baek, Y., Lee, B., Han, D., Yun, S., Lee, H.: Character region awareness for text detection. In: Proceedings of the IEEE/CVF CVPR. pp. 9365–9374 (2019)
  * [3] Baviskar, D., Ahirrao, S., Kotecha, K.: Multi-layout invoice document dataset (MIDD): A dataset for named entity recognition. Data (2021). https://doi.org/10.3390/data6070078
  * [4] Bensch, O., Popa, M., Spille, C.: Key information extraction from documents: Evaluation and generator. In: Abbès, S.B., Hantach, R., Calvez, P., Buscaldi, D., Dessì, D., Dragoni, M., Recupero, D.R., Sack, H. (eds.) Proceedings of DeepOntoNLP and X-SENTIMENT. CEUR Workshop Proceedings, vol. 2918, pp. 47–53. CEUR-WS.org (2021)
  * [5] Berge, J.: The EDIFACT standards. Blackwell Publishers, Inc. (1994)
  * [6] Borchmann, Ł., Pietruszka, M., Stanislawek, T., Jurkiewicz, D., Turski, M., Szyndler, K., Graliński, F.: DUE: End-to-end document understanding benchmark. In: Proceeedings of NeurIPS (2021)
  * [7] Bosak, J., McGrath, T., Holman, G.K.: Universal business language v2. 0. Organization for the Advancement of Structured Information Standards (OASIS), Standard (2006)
  * [8] Cesarini, F., Francesconi, E., Gori, M., Soda, G.: Analysis and understanding of multi-class invoices. Document Analysis and Recognition 6(2), 102–114 (2003)
  * [9] Chaudhry, R., Shekhar, S., Gupta, U., Maneriker, P., Bansal, P., Joshi, A.: LEAF-QA: locate, encode & attend for figure question answering. In: Proceeedings of WACV. pp. 3501–3510. IEEE (2020). https://doi.org/10.1109/WACV45572.2020.9093269
  * [10] Chen, L., Chen, X., Zhao, Z., Zhang, D., Ji, J., Luo, A., Xiong, Y., Yu, K.: Websrc: A dataset for web-based structural reading comprehension. CoRR (2021)
  * [11] Chen, W., Chang, M., Schlinger, E., Wang, W.Y., Cohen, W.W.: Open question answering over tables and text. In: Proceedings of ICLR (2021)
  * [12] Chen, W., Wang, H., Chen, J., Zhang, Y., Wang, H., Li, S., Zhou, X., Wang, W.Y.: TabFact: A large-scale dataset for table-based fact verification. In: Proceedings of ICLR (2020)
  * [13] Chen, W., Zha, H., Chen, Z., Xiong, W., Wang, H., Wang, W.Y.: Hybridqa: A dataset of multi-hop question answering over tabular and textual data. In: Cohn, T., He, Y., Liu, Y. (eds.) Findings of the Association for Computational Linguistics: EMNLP. Findings of ACL, vol. EMNLP 2020, pp. 1026–1036. Association for Computational Linguistics (2020). https://doi.org/10.18653/v1/2020.findings-emnlp.91
  * [14] Cho, M., Amplayo, R.K., Hwang, S., Park, J.: Adversarial TableQA: Attention Supervision for Question Answering on Tables. In: Zhu, J., Takeuchi, I. (eds.) Proceedings of ACML. Proceedings of Machine Learning Research, vol. 95, pp. 391–406 (2018)
  * [15] Clausner, C., Antonacopoulos, A., Pletschacher, S.: ICDAR 2019 competition on recognition of documents with complex layouts-rdcl2019. In: Proceedings of ICDAR. pp. 1521–1526. IEEE (2019)
  * [16] Cristani, M., Bertolaso, A., Scannapieco, S., Tomazzoli, C.: Future paradigms of automated processing of business documents. International Journal of Information Management 40, 67–75 (2018)
  * [17] d’Andecy, V.P., Hartmann, E., Rusinol, M.: Field extraction by hybrid incremental and a-priori structural templates. In: 2018 13th IAPR International Workshop on Document Analysis Systems (DAS). pp. 251–256. IEEE (2018)
  * [18] Deng, Y., Rosenberg, D.S., Mann, G.: Challenges in end-to-end neural scientific table recognition. In: Proceedings of ICDAR. pp. 894–901. IEEE (2019). https://doi.org/10.1109/ICDAR.2019.00148
  * [19] Denk, T.I., Reisswig, C.: Bertgrid: Contextualized embedding for 2d document representation and understanding. arXiv preprint arXiv:1909.04948 (2019)
  * [20] Dhakal, P., Munikar, M., Dahal, B.: One-shot template matching for automatic document data capture. In: Proceeedings of Artificial Intelligence for Transforming Business and Society (AITB). vol. 1, pp. 1–6. IEEE (2019)
  * [21] Directive 2014/55/eu of the european parliament and of the council on electronic invoicing in public procurement (Apr 2014), https://eur-lex.europa.eu/eli/dir/2014/55/oj
  * [22] Fang, J., Tao, X., Tang, Z., Qiu, R., Liu, Y.: Dataset, ground-truth and performance metrics for table detection evaluation. In: Blumenstein, M., Pal, U., Uchida, S. (eds.) Proceedings of IAPR International Workshop on Document Analysis Systems, DAS. pp. 445–449. IEEE (2012). https://doi.org/10.1109/DAS.2012.29
  * [23] Ford, G., Thoma, G.R.: Ground truth data for document image analysis. In: Symposium on Document Image Understanding and Technology. pp. 199–205. Citeseer (2003)
  * [24] Gao, L., Yi, X., Jiang, Z., Hao, L., Tang, Z.: ICDAR2017 competition on page object detection. In: Proceedings of ICDAR. pp. 1417–1422 (2017). https://doi.org/10.1109/ICDAR.2017.231
  * [25] Garncarek, L., Powalski, R., Stanislawek, T., Topolski, B., Halama, P., Turski, M., Gralinski, F.: LAMBERT: layout-aware language modeling for information extraction. In: Lladós, J., Lopresti, D., Uchida, S. (eds.) Proceedings of ICDAR. vol. 12821, pp. 532–547. Springer (2021). https://doi.org/10.1007/978-3-030-86549-8_34
  * [26] Göbel, M.C., Hassan, T., Oro, E., Orsi, G.: ICDAR 2013 table competition. In: Proceedings of ICDAR. pp. 1449–1453. IEEE Computer Society (2013). https://doi.org/10.1109/ICDAR.2013.292
  * [27] Guillaume Jaume, Hazim Kemal Ekenel, J.P.T.: Funsd: A dataset for form understanding in noisy scanned documents. In: Accepted to ICDAR-OST (2019)
  * [28] Hamad, K.A., Mehmet, K.: A detailed analysis of optical character recognition technology. International Journal of Applied Mathematics Electronics and Computers 1(Special Issue-1), 244–249 (2016)
  * [29] Hamza, H., Belaïd, Y., Belaïd, A.: Case-based reasoning for invoice analysis and recognition. In: International conference on case-based reasoning. pp. 404–418. Springer (2007)
  * [30] Harley, A.W., Ufkes, A., Derpanis, K.G.: Evaluation of deep convolutional nets for document image classification and retrieval. In: International Conference on Document Analysis and Recognition (ICDAR) (2015)
  * [31] He, S., Schomaker, L.: Beyond ocr: Multi-faceted understanding of handwritten document characteristics. Pattern Recognition 63, 321–333 (2017)
  * [32] Holeček, M.: Learning from similarity and information extraction from structured documents. International Journal on Document Analysis and Recognition (IJDAR) pp. 1–17 (2021)
  * [33] Holeček, M., Hoskovec, A., Baudiš, P., Klinger, P.: Table understanding in structured documents. In: 2019 International Conference on Document Analysis and Recognition Workshops (ICDARW). vol. 5, pp. 158–164. IEEE (2019)
  * [34] Holt, X., Chisholm, A.: Extracting structured data from invoices. In: Proceedings of the Australasian Language Technology Association Workshop 2018\. pp. 53–59 (2018)
  * [35] Huang, Z., Chen, K., He, J., Bai, X., Karatzas, D., Lu, S., Jawahar, C.V.: ICDAR2019 competition on scanned receipt OCR and information extraction. In: Proceedings of ICDAR. pp. 1516–1520. IEEE (2019). https://doi.org/10.1109/ICDAR.2019.00244
  * [36] Islam, N., Islam, Z., Noor, N.: A survey on optical character recognition system. arXiv preprint arXiv:1710.05703 (2017)
  * [37] Jiang, J.: Information extraction from text. In: Mining text data, pp. 11–41. Springer (2012)
  * [38] Jobin, K.V., Mondal, A., Jawahar, C.V.: Docfigure: A dataset for scientific document figure classification. In: 13th IAPR International Workshop on Graphics Recognition, GREC@ICDAR. pp. 74–79. IEEE (2019). https://doi.org/10.1109/ICDARW.2019.00018
  * [39] Kardas, M., Czapla, P., Stenetorp, P., Ruder, S., Riedel, S., Taylor, R., Stojnic, R.: Axcell: Automatic extraction of results from machine learning papers. arXiv preprint arXiv:2004.14356 (2020)
  * [40] Katti, A.R., Reisswig, C., Guder, C., Brarda, S., Bickel, S., Höhne, J., Faddoul, J.B.: Chargrid: Towards understanding 2d documents. arXiv preprint arXiv:1809.08799 (2018)
  * [41] Krieger, F., Drews, P., Funk, B., Wobbe, T.: Information extraction from invoices: A graph neural network approach for datasets with high layout variety. In: International Conference on Wirtschaftsinformatik. pp. 5–20. Springer (2021)
  * [42] Kumar, A., Irsoy, O., Ondruska, P., Iyyer, M., Bradbury, J., Gulrajani, I., Zhong, V., Paulus, R., Socher, R.: Ask me anything: Dynamic memory networks for natural language processing. In: Balcan, M., Weinberger, K.Q. (eds.) Proceedings of ICML. vol. 48, pp. 1378–1387. JMLR.org (2016)
  * [43] Lample, G., Ballesteros, M., Subramanian, S., Kawakami, K., Dyer, C.: Neural architectures for named entity recognition. In: Knight, K., Nenkova, A., Rambow, O. (eds.) Proceedings of NAACL HLT. pp. 260–270 (2016). https://doi.org/10.18653/v1/n16-1030
  * [44] Lewis, D., Agam, G., Argamon, S., Frieder, O., Grossman, D., Heard, J.: Building a test collection for complex document information processing. In: Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval. pp. 665–666 (2006)
  * [45] Li, J., Wang, S., Wang, Y., Tang, Z.: Synthesizing data for text recognition with style transfer. Multimedia Tools and Applications 78(20), 29183–29196 (2019)
  * [46] Li, M., Cui, L., Huang, S., Wei, F., Zhou, M., Li, Z.: Tablebank: Table benchmark for image-based table detection and recognition. In: Calzolari, N., Béchet, F., Blache, P., Choukri, K., Cieri, C., Declerck, T., Goggi, S., Isahara, H., Maegaard, B., Mariani, J., Mazo, H., Moreno, A., Odijk, J., Piperidis, S. (eds.) Proceedings of The 12th Language Resources and Evaluation Conference, LREC. pp. 1918–1925 (2020)
  * [47] Liu, W., Zhang, Y., Wan, B.: Unstructured document recognition on business invoice. Mach. Learn., Stanford iTunes Univ., Stanford, CA, USA, Tech. Rep (2016)
  * [48] Majumder, B.P., Potti, N., Tata, S., Wendt, J.B., Zhao, Q., Najork, M.: Representation learning for information extraction from form-like documents. In: Jurafsky, D., Chai, J., Schluter, N., Tetreault, J.R. (eds.) Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL. pp. 6495–6504 (2020). https://doi.org/10.18653/v1/2020.acl-main.580
  * [49] Mathew, M., Bagal, V., Tito, R., Karatzas, D., Valveny, E., Jawahar, C.: Infographicvqa. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 1697–1706 (2022)
  * [50] Mathew, M., Karatzas, D., Jawahar, C.V.: Docvqa: A dataset for VQA on document images. In: Proceedings of WACV. pp. 2199–2208. IEEE (2021). https://doi.org/10.1109/WACV48630.2021.00225
  * [51] McCann, B., Keskar, N.S., Xiong, C., Socher, R.: The natural language decathlon: Multitask learning as question answering. CoRR (2018)
  * [52] Meadows, B., Seaburg, L.: Universal business language 1.0. Organization for the Advancement of Structured Information Standards (OASIS) (2004)
  * [53] Medvet, E., Bartoli, A., Davanzo, G.: A probabilistic approach to printed document understanding. Int. J. Document Anal. Recognit. 14(4), 335–347 (2011). https://doi.org/10.1007/s10032-010-0137-1
  * [54] Memon, J., Sami, M., Khan, R.A., Uddin, M.: Handwritten optical character recognition (ocr): A comprehensive systematic literature review (slr). IEEE Access 8, 142642–142668 (2020)
  * [55] Methani, N., Ganguly, P., Khapra, M.M., Kumar, P.: Plotqa: Reasoning over scientific plots. In: Proceedings of WACV. pp. 1516–1525 (2020). https://doi.org/10.1109/WACV45572.2020.9093523
  * [56] Nadeau, D., Sekine, S.: A survey of named entity recognition and classification. Lingvisticæ Investigationes pp. 3–26 (2007). https://doi.org/https://doi.org/10.1075/li.30.1.03nad
  * [57] Nassar, A., Livathinos, N., Lysak, M., Staar, P.W.J.: Tableformer: Table structure understanding with transformers. CoRR abs/2203.01017 (2022). https://doi.org/10.48550/arXiv.2203.01017
  * [58] Nayef, N., Patel, Y., Busta, M., Chowdhury, P.N., Karatzas, D., Khlif, W., Matas, J., Pal, U., Burie, J.C., Liu, C.l., et al.: ICDAR 2019 robust reading challenge on multi-lingual scene text detection and recognition—rrc-mlt-2019. In: Proceedings of ICDAR. pp. 1582–1587. IEEE (2019)
  * [59] Palm, R.B., Laws, F., Winther, O.: Attend, copy, parse end-to-end information extraction from documents. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 329–336. IEEE (2019)
  * [60] Palm, R.B., Winther, O., Laws, F.: Cloudscan - A configuration-free invoice analysis system using recurrent neural networks. In: Proceedings of ICDAR. pp. 406–413. IEEE (2017). https://doi.org/10.1109/ICDAR.2017.74
  * [61] Park, S., Shin, S., Lee, B., Lee, J., Surh, J., Seo, M., Lee, H.: Cord: A consolidated receipt dataset for post-ocr parsing. In: Workshop on Document Intelligence at NeurIPS 2019 (2019)
  * [62] Prasad, D., Gadpal, A., Kapadni, K., Visave, M., Sultanpure, K.: Cascadetabnet: An approach for end to end table detection and structure recognition from image-based documents. In: Proceedings of CVPRw. pp. 2439–2447 (2020). https://doi.org/10.1109/CVPRW50498.2020.00294
  * [63] Qasim, S.R., Mahmood, H., Shafait, F.: Rethinking table recognition using graph neural networks. In: Proceedings of ICDAR. pp. 142–147. IEEE (2019). https://doi.org/10.1109/ICDAR.2019.00031
  * [64] Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint arXiv:1910.10683 (2019)
  * [65] Rastogi, M., Ali, S.A., Rawat, M., Vig, L., Agarwal, P., Shroff, G., Srinivasan, A.: Information extraction from document images via FCA based template detection and knowledge graph rule induction. In: Proceedings of CVPRw. pp. 2377–2385 (2020). https://doi.org/10.1109/CVPRW50498.2020.00287
  * [66] Riba, P., Dutta, A., Goldmann, L., Fornés, A., Ramos, O., Lladós, J.: Table detection in invoice documents by graph neural networks. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 122–127. IEEE (2019)
  * [67] Rusinol, M., Benkhelfallah, T., Poulain dAndecy, V.: Field extraction from administrative documents by incremental structural templates. In: 2013 12th International Conference on Document Analysis and Recognition. pp. 1100–1104. IEEE (2013)
  * [68] Schreiber, S., Agne, S., Wolf, I., Dengel, A., Ahmed, S.: DeepDeSRT: Deep Learning for Detection and Structure Recognition of Tables in Document Images. In: Proceedings of ICDAR. pp. 1162–1167 (2017). https://doi.org/10.1109/ICDAR.2017.192
  * [69] Schuster, D., Muthmann, K., Esser, D., Schill, A., Berger, M., Weidling, C., Aliyev, K., Hofmeier, A.: Intellix–end-user trained information extraction for document archiving. In: 2013 12th International Conference on Document Analysis and Recognition. pp. 101–105. IEEE (2013)
  * [70] Shahab, A., Shafait, F., Kieninger, T., Dengel, A.: An open approach towards the benchmarking of table structure recognition systems. In: Doermann, D.S., Govindaraju, V., Lopresti, D.P., Natarajan, P. (eds.) The Ninth IAPR International Workshop on Document Analysis Systems, DAS. pp. 113–120 (2010). https://doi.org/10.1145/1815330.1815345
  * [71] Siegel, N., Lourie, N., Power, R., Ammar, W.: Extracting scientific figures with distantly supervised neural networks. In: Chen, J., Gonçalves, M.A., Allen, J.M., Fox, E.A., Kan, M., Petras, V. (eds.) Proceedings of the 18th ACM/IEEE on Joint Conference on Digital Libraries, JCDL. pp. 223–232 (2018). https://doi.org/10.1145/3197026.3197040
  * [72] Smith, R.: An overview of the tesseract ocr engine. In: Ninth international conference on document analysis and recognition (ICDAR 2007). vol. 2, pp. 629–633. IEEE (2007)
  * [73] Stanisławek, T., Graliński, F., Wróblewska, A., Lipiński, D., Kaliska, A., Rosalska, P., Topolski, B., Biecek, P.: Kleister: key information extraction datasets involving long documents with complex layouts. In: International Conference on Document Analysis and Recognition. pp. 564–579. Springer (2021)
  * [74] Stockerl, M., Ringlstetter, C., Schubert, M., Ntoutsi, E., Kriegel, H.P.: Online template matching over a stream of digitized documents. In: Proceedings of the 27th International Conference on Scientific and Statistical Database Management. pp. 1–12 (2015)
  * [75] Stray, J., Svetlichnaya, S.: Deepform: Extract information from documents (2020), https://wandb.ai/deepform/political-ad-extraction, benchmark
  * [76] Sun, H., Kuang, Z., Yue, X., Lin, C., Zhang, W.: Spatial dual-modality graph reasoning for key information extraction. arXiv preprint arXiv:2103.14470 (2021)
  * [77] Sunder, V., Srinivasan, A., Vig, L., Shroff, G., Rahul, R.: One-shot information extraction from document images using neuro-deductive program synthesis. arXiv preprint arXiv:1906.02427 (2019)
  * [78] Tensmeyer, C., Morariu, V.I., Price, B., Cohen, S., Martinez, T.: Deep splitting and merging for table structure decomposition. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). pp. 114–121. IEEE (2019)
  * [79] Wang, J., Liu, C., Jin, L., Tang, G., Zhang, J., Zhang, S., Wang, Q., Wu, Y., Cai, M.: Towards robust visual information extraction in real world: New dataset and novel solution. In: Proceedings of the AAAI Conference on Artificial Intelligence (2021)
  * [80] Web: Annual reports. https://www.annualreports.com/, accessed: 2022-04-28
  * [81] Web: Charity Commission for England and Wales. https://apps.charitycommission.gov.uk/showcharity/registerofcharities/RegisterHomePage.aspx, accessed: 2022-04-22
  * [82] Web: EDGAR. https://www.sec.gov/edgar.shtml, accessed: 2022-04-22
  * [83] Web: Industry Documents Library. https://www.industrydocuments.ucsf.edu/, accessed: 2022-04-22
  * [84] Web: NIST Special Database 2. https://www.nist.gov/srd/nist-special-database-2, accessed: 2022-04-25
  * [85] Web: Open Government Data (OGD) Platform India. https://visualize.data.gov.in/, accessed: 2022-04-22
  * [86] Web: Public Inspection Files. https://publicfiles.fcc.gov/, accessed: 2022-04-22
  * [87] Web: Scitsr. https://github.com/Academic-Hammer/SciTSR, accessed: 2022-04-26
  * [88] Web: S&P 500 Companies with Financial Information. https://www.spglobal.com/spdji/en/indices/equity/sp-500/#data, accessed: 2022-04-25
  * [89] Web: Statistics of Common Crawl Monthly Archives — MIME Types. https://commoncrawl.github.io/cc-crawl-statistics/plots/mimetypes, accessed: 2022-04-22
  * [90] Web: Tablebank. https://github.com/doc-analysis/TableBank, accessed: 2022-04-26
  * [91] Web: World Bank Open Data. https://data.worldbank.org/, accessed: 2022-04-22
  * [92] Xu, Y., Li, M., Cui, L., Huang, S., Wei, F., Zhou, M.: Layoutlm: Pre-training of text and layout for document image understanding. In: Gupta, R., Liu, Y., Tang, J., Prakash, B.A. (eds.) Proceedings on KDD. pp. 1192–1200 (2020). https://doi.org/10.1145/3394486.3403172
  * [93] Xu, Y., Lv, T., Cui, L., Wang, G., Lu, Y., Florêncio, D., Zhang, C., Wei, F.: LayoutXLM: Multimodal Pre-training for Multilingual Visually-rich Document Understanding. CoRR (2021)
  * [94] Yi, J., Sundaresan, N.: A classifier for semi-structured documents. In: Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining. pp. 340–344 (2000)
  * [95] Yu, D., Li, X., Zhang, C., Liu, T., Han, J., Liu, J., Ding, E.: Towards accurate scene text recognition with semantic reasoning networks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12113–12122 (2020)
  * [96] Yu, W., Lu, N., Qi, X., Gong, P., Xiao, R.: PICK: processing key information extraction from documents using improved graph learning-convolutional networks. In: Proceedings of ICPR. pp. 4363–4370. IEEE (2020). https://doi.org/10.1109/ICPR48806.2021.9412927
  * [97] Zhao, X., Wu, Z., Wang, X.: CUTIE: learning to understand documents with convolutional universal text information extractor. CoRR abs/1903.12363 (2019), http://arxiv.org/abs/1903.12363
  * [98] Zheng, X., Burdick, D., Popa, L., Zhong, X., Wang, N.X.R.: Global table extractor (GTE): A framework for joint table identification and cell structure recognition using visual context. In: Proceedings of WACV. pp. 697–706. IEEE (2021). https://doi.org/10.1109/WACV48630.2021.00074
  * [99] Zhong, X., ShafieiBavani, E., Jimeno-Yepes, A.: Image-based table recognition: Data, model, and evaluation. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J. (eds.) Proceedings of ECCV. pp. 564–580. Springer (2020). https://doi.org/10.1007/978-3-030-58589-1_34
  * [100] Zhong, X., Tang, J., Jimeno-Yepes, A.: Publaynet: Largest dataset ever for document layout analysis. In: Proceedings of ICDAR. pp. 1015–1022. IEEE (Sep 2019). https://doi.org/10.1109/ICDAR.2019.00166
  * [101] Zhu, F., Lei, W., Huang, Y., Wang, C., Zhang, S., Lv, J., Feng, F., Chua, T.: TAT-QA: A question answering benchmark on a hybrid of tabular and textual content in finance. In: Zong, C., Xia, F., Li, W., Navigli, R. (eds.) Proceedings International Joint Conference on Natural Language Processing. pp. 3277–3287 (2021). https://doi.org/10.18653/v1/2021.acl-long.254


# Review of Data-centric Time Series Analysis from Sample, Feature, and Period

Chenxi Sun1,2,3 Hongyan Li1,2,3,4 Yaliang Li5 Shenda Hong6,7
1National Key Laboratory of General Artificial Intelligence, Peking University
2Key Laboratory of Machine Perception (Ministry of Education), Peking
University 3School of Intelligence Science and Technology, Peking University
4PKU-WUHAN Institute for Artificial Intelligence 5Alibaba Group 6National
Institute of Health Data Science, Peking University 7Institute of Medical
Technology, Health Science Center of Peking University
{sun_chenxi<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Data is essential to performing time series analysis utilizing machine
learning approaches, whether for classic models or today’s large language
models. A good time-series dataset is advantageous for the model’s accuracy,
robustness, and convergence, as well as task outcomes and costs. The emergence
of data-centric AI represents a shift in the landscape from model refinement
to prioritizing data quality. Even though time-series data processing methods
frequently come up in a wide range of research fields, it hasn’t been well
investigated as a specific topic. To fill the gap, in this paper, we
systematically review different data-centric methods in time series analysis,
covering a wide range of research topics. Based on the time-series data
characteristics at sample, feature, and period, we propose a taxonomy for the
reviewed data selection methods. In addition to discussing and summarizing
their characteristics, benefits, and drawbacks targeting time-series data, we
also introduce the challenges and opportunities by proposing recommendations,
open problems, and possible research topics.

##  1 Introduction

Time-Series (TS) data widely exists in real-world applications, such as
industry, healthcare, finance, meteorology, etc. The growing interest in TS
analysis, encompassing tasks like forecasting, classification, anomaly
detection, and causality analysis, has garnered significant attention.

TS analysis has predominantly taken model-centric approaches, placing a
primary emphasis on refining model designs to enhance performance using fixed
datasets. The relevant models have evolved from traditional statistical
approaches, including Auto-Regressive models (AR), Moving Average models (MA),
and others, to sophisticated deep learning approaches like Recurrent Neural
Networks (RNNs), Convolutional Neural Networks (CNNs), TS Transformers (TSTs),
and their various adaptations.

However, this model-centric approach often overlooks issues related to data
quality, creating ambiguity regarding whether the enhancements in the model
genuinely reflect its potential or if they are simply a result of overfitting
to the dataset. Specifically, it is essential to note that when researching TS
applications, tasks, or models, specific datasets must be provided, and the
quality of the data directly influences the outcomes. The importance of data
selection has persisted from classical models to contemporary Large Models
(LM). It plays a crucial role in boosting the automation of classical models,
contributing to the reduction of complexity and costs, and facilitating the
adaptation of pre-trained Large Language Models (LLM) to TS data Sun et al.
(2023d).

Consequently, some emerging advocates have shifted their focus from model
refinement to prioritizing data quality. But currently, the data-centric TS
analysis has not been systematically investigated as a specialized field, even
though it appears regularly across numerous study topics such as clustering,
causality analysis Assaad et al. (2023), few-shot learning Yang et al.
(2022b), imbalanced classification Chou et al. (2020), representation learning
Meng et al. (2023), curriculum learning Sun et al. (2023c), and Dimension
reduction Ashraf et al. (2023), etc.

Figure 1: Data-Centric Time Series Analysis: A case of time series selection
for data quantity reduction from the perspectives of sample (blue), feature
(green), and period (yellow).

In this paper, to fill the aforementioned gaps, we review the utilization of
data-related methods in TS analysis. We aim to analyze, summarize, and
categorize various data selection methods based on the characteristics of TS
data from three perspectives: sample (Section 3), feature (Section 4), and
period (Section 5). We discuss the benefits and drawbacks of related work
(Table 2), provide conclusions and recommendations (Section 6), and present
the challenges and opportunities by proposing open problems and future work
(Table 3).

We assert that data-centric approaches constitute a nascent frontier that
complements existing efforts, and play an increasingly important role, not
solely confined to TS analysis but also instrumental in diverse AI domains.

##  2 Time-Series Data Characteristics

######  Definition 1 (Sample).

A TS sample is noted as $x=\\{x^{d}_{i}\\}_{i=1,d=1}^{T,D}$. It has $D$
variables (horizontal), $T$ time points (longitudinal) with value $x_{i}$ and
possible timestamp record $t_{i}$, and the possible label information $c$.

######  Definition 2 (Feature).

A multivariate TS $x=\\{x^{d}_{i}\\}_{i=1,d=1}^{T,D}$ has two types of
features: 1) The intuitive features (original space) are reflected in its
values in $D$ variables and its time information of records $t$. 2) The
implied features (new space) are reflected in the relation between $D$
variables and new embeddings mapped in other spaces.

######  Definition 3 (Period).

It represents a certain length on TS data. A full-length TS
$x=\\{x_{i}\\}_{i=1}^{T}$ is represented as $T$ time steps or a time interval
from $t_{1}$ to $t_{T}$. Its partial-length sample can be noted as some
segmentations $x^{\prime}=\\{x_{j}\\}_{j\geq 1}^{\leq T}$.

As shown in Figure 1, in typical scenarios, data preprocessing is imperative
after dataset collection, involving denoising and removing missing data. We
can filter the data if the dataset is massive, and augment the data if it is
too small. Then, when utilizing TS models, we can select appropriate features,
windows, and learning orders to improve the model’s convergence, accuracy, and
robustness.

Meanwhile, TS data in non-random systems often exhibits one or more of the
following properties in two domains:

######  Property 1 (Time domain properties).

1) Decomposability: $x=s+t+r$, $s$ is the seasonal term, $t$ is the trend
term, and $r$ is the residual term; 2) Autocorrelation:
$x_{i}=f(\\{x_{j}\\}_{j<i})$; 3) Variable correlation:
$x^{d}=f(\\{x^{j}\\}_{j\neq d})$.

######  Property 2 (Frequency domain properties).

1) Discrete Fourier Transform (DFT): The frequency spectrum is
$F(\omega_{k})=\frac{1}{T}\sum_{i=1}^{T}x_{i}e^{-j\omega_{k}i}$. It can be
decomposed into amplitude $A$ and phase spectra $\theta$ by
$F(\omega_{k})=A(\omega_{k})\exp[j\theta(\omega_{k})]$ 2) Discrete Wavelet
Transform (DWT): The approximation coefficients and detail coefficients of
one-level DWT are $a_{n}=\sum_{k}h_{k}x_{2n-k}$ and
$d_{n}=\sum_{k}g_{k}x_{2n-k}$. $h,g$ are the low-pass and high-pass filter
coefficients.

##  3 Sample Selection

This section covers the effects and requirements related to dataset size,
quality, and learning order in various TS models, tasks, and scenarios. In
summary, the primary research focus of the sample selection work outlined here
is their concentration on existing models and their adaptation to the domain
of TS analysis based on TS properties.

Category Topic Task Model Dataset111https://github.com/SCXsunchenxi/ISMTS-
Review Sample Few-shot learning Imbalanced learning Transfer learning Meta-
learning Federated learning Curriculum learning Continual learning Clustering
Generation Forecasting Classification One-class Multi-task SVM CNN RNN TST SNN
GAN DIFF Healthcare Industry Meteorology Finance Traffic Synthetic Archive
Feature Representation learning Contrastive learning Dimension reduction
Causal learning Graph learning Classification Forecasting Imputation Anomaly
detection Causal inference Tree GNN TCN AE Attent. Healthcare Electricity
Chaotic sys Archive Period Representation learning Dimension reduction
Continual learning Online learning TST Design Segmentation Staging, subtyping
Anomaly detection Object localization Continuous classification PCA DTW CNN
TST Activity Healthcare Archive

Table 1: Research Topics, Models, and Datasets

###  3.1 Data Filtering

To ensure the quality of a large and noisy dataset, it becomes essential to
extract high-quality data and perform data filtering. Most of the time,
preprocessing steps, such as filtering out missing and noisy data, are
employed after obtaining the TS dataset. While this method is straightforward,
there is a risk of either insufficiently or excessively filtering the data.

Due to the varied evaluation criteria in different applications, the
determination of what constitutes useful data varies. Consequently, most data
filtering or sample reduction methods are tailored to specific tasks or
fields. For instance, some methods focus on farthest boundary point estimation
for support vector data description Alam (2020), while others concentrate on
common prototype generation for anomaly detection Li (2023). There are also
methods designed for one-class classification such as distance-based
representation Mauceri (2020) and local cluster balance Hayashi (2024).

###  3.2 Data Augmentation

The success of Deep Learning (DL) heavily relies on a large number of training
data. This necessity becomes particularly crucial in the prevailing trend of
LM. But in real-world applications, data scarcity is a common challenge,
primarily manifesting in three situations: 1) The total samples are limited.
That is a research topic in small sample learning, few-shot learning, zero-
shot learning, meta-learning, etc. 2) The labeled samples is limited. That is
a research topic in un-/self-supervised learning, positive unlabelled
learning, active learning, etc. 3) The dataset is imbalanced. That is a
research topic in imbalanced classification and regression, etc.

The problems in the above fields can be alleviated by increasing the number of
total samples or samples in some specific classes. The main methods for
accomplishing this involve data augmentation, generation, and synthesis. It is
worth noting that their techniques are interconnected.

Unlike the static data, TS data possesses unique and intrinsic properties
related to temporal dependency and time-frequency transformation. This
complexity increases when modeling multivariate TS, where intricate dynamics
among variables exist. Thus, the straightforward application of data
augmentation methods from image and speech processing is not effective Wen et
al. (2021). Here, we summarize current data augmentation and expansion methods
specifically for TS data into the following three categories:

  * •

Basic methods in time domain and frequency domain.

Transformations in the time domain are relatively straightforward. Techniques
involving noise injection directly manipulate the original TS data, including
the addition of Gaussian noise, spikes, step-like, and slope-like trend
patterns Wen (2019). Time window-based methods, unique to TS, encompass window
warping, flipping, cropping, slicing, jittering, perturbation, and their
combinations.

Transformation in the frequency domain has inspired fewer augmentation
methods. TS frequency can be decomposed into $A$ and $\theta$ as described in
Property 2. For randomly selected segments, $A$ and perturbations of $\theta$
are replaced with Gaussian noise with the original mean and variance, added by
an extra zero-mean Gaussian noise, respectively Gao et al. (2020).
Additionally, the random phase shuffle can also be utilized after the
amplitude-adjusted Fourier transform, which can approximately preserve the
temporal correlation, amplitude distribution, and power spectra. Moreover, as
TS has both time and frequency features, few methods use the time-frequency
analysis S.Park (2019).

  * •

Generative models.

Statistical generative models intuitively describe TS dynamics, such as the
mixture of Gaussian trees and the mixture autoregressive Kang et al. (2020).
These models rely on the autocorrelation in Property 1, such conditional
assumption will generate new sequences if initial values are perturbed.

Deep generative models, such as Generative Adversarial Networks (GANs) and
Diffusion models (DIFF), have received significant attention in Computer
Vision (CV). Their design needs to be adjusted for TS characteristics Lin et
al. (2023). For example, TimeGAN Yoon et al. (2019) generates TS in various
domains by a learned embedding space; TimeDiff Shen and Kwok (2023) models TS
by a diffusion model with future mixup and autoregressive initialization.

  * •

Decomposition and synthesis.

Synthesis TS dataset can be obtained through some empirical formulas, such as
Property 1 decomposability, Property 2 DFT, and time lag
$x_{i}=\sum_{j<i}\alpha_{j}x_{j}$. For example, Mackey-Glass system is
$x_{i}=-bx_{i}+\frac{ax_{i-\tau}}{1+(x_{i-\tau})^{c}}$, where $\tau$ is the
lag.

###  3.3 Learning Order Arrangement

Training models in a meaningful order can improve model performance over the
standard approach based on random order. Curriculum Learning strategies (CL)
train models from easier data to harder data, imitating the human curricula.
We can summarize the easy-to-hard CL methods into predefined CL, where the
difficulty measurer and the training scheduler are provided before training,
and automatic CL, where the evaluation and scheduling dynamically change with
training by self-paced learning, transfer teacher, or reinforcement learning
strategies Portelas et al. (2020).

Although such an easy-to-use plug-in has demonstrated its power in improving
the generalization capacity and convergence rate of various models in CV field
Wang et al. (2022), few of them are specifically designed for TS data. Koh
(2024) uses CL to smooth the noise during training for the control tasks over
highly stochastic TS data. Sun et al. (2023c) proposes a curricular and
cyclical loss to train TS models based on the TS distribution changing
dynamics.

##  4 Feature Selection

TS data has many features, some of which are listed in Definition 2. This
section covers dimension reduction and feature augmentation methods. In
summary, apart from conventional feature engineering techniques, most recent
work has concentrated on deep features and representation learning.

###  4.1 Feature Augmentation

Expanding features on existing ones can provide a comprehensive description.
We summarize work into two categories:

  * •

Add static features.

Including extra descriptions and statistical features in the input is a
typical strategy for improving the performance of DL models. For instance,
incorporating demographic information such as patient age and gender into
medical TS data has been shown to improve diagnostic accuracy Sun (2021).
Descriptive statistics for TS encompass measures like mean, variance,
skewness, and kurtosis, while trends involve moving averages, seasonality
includes indices, and frequency involves histograms. Other characteristics,
such as those related to stationarity testing, correlation, and
autocorrelation analysis, can also be considered. These additional static
features can be added to the input, concatenated to the hidden layer vectors,
or integrated into the output Yan et al. (2020).

  * •

Add dynamic features.

TS has dynamics and dependencies over time development. The relations among
variants may be changing. For this, graph-based methods have been utilized for
TS analysis like GNN4TS Jin (2023). They can provide the interactive features
Deng (2021), hypergraph-based local features Cai et al. (2022), and spectral
graph features Yu (2022); The time information in TS data is likewise dynamic.
For example, irregularly sampled TS has varying time intervals and lacks
temporal alignment across dimensions. Such uneven time stamps can be encoded
as the new dynamic features for DL models Sun et al. (2021).

  * •

Utilize representation learning.

The approach regarding the variant representation space of features can be
applied in this context. Recently, universal representation learning for TS
analysis has been popular. Most of them are unsupervised and learn
discriminative feature representations from unlabeled data. This route also
provides a foundation for implementing the general TS model. We summarize them
into three categories:

Deep clustering methods allow the clustering-oriented iterative optimization,
where clustering assignments can serve as self-supervised signals, thereby
facilitating the self-learning process, such as CRLI Ma et al. (2021) for
incomplete TS, DTCR Ma et al. (2019) for temporal representations, IDFD Tao et
al. (2021) for spectral clustering, etc. They mainly focus on higher-level
semantic information.

Reconstruction methods minimize the difference between the reconstructed
output and the raw input, ignoring insignificant features like noise. The
related methods generally utilize the encoder-decoder architecture, such as
RNN-based TimeNet and Transformer-based CRT Zhang et al. (2023b).

Self-supervised learning methods design diverse pretext tasks that
automatically generate useful supervised signals from the original data. Meng
et al. (2023) classifies the TS self-supervised learning methods into three
categories: adversarial methods, predictive methods, and contrastive methods.
Among them, contrastive learning has been the subject of extensive research
lately. They first generate positive and negative views of the anchor data and
then employ the pretext task to bring similar pairs closer while pushing
dissimilar pairs apart. Most related research is conducted in CV. For TS data,
some efforts have been made to implement instance-level contrast Woo et al.
(2022), temporal-level contrast Franceschi (2019), and clustering-level
contrast. Meanwhile, the reconstruction methods introduced above can also be
utilized as a pretext task for self-supervised learning to provide contextual
information.

###  4.2 Dimension Reduction

Many real-world TS is characterized by high dimensionality. The dimensionality
curse largely causes issues for learning approaches that can fail to capture
the temporal dependencies. To address this, selecting a subset of features
while preserving intrinsic properties becomes crucial. This not only reduces
computational complexity but also prevents lower learning by eliminating
irrelevant and redundant features. Here, we summarize the existing methods
into two categories:

  * •

Invariant representation space of features.

The approaches select $m$ from $n$ variables/dimensions ($m\leq n$) while
maintaining the original feature distribution and numerical space. The
straightforward method is the similarity measurement, which chooses one or
more features from a set of similar features. The classic methods include
Mahalanobis distance, DTW, and similarity-based clustering Ma et al. (2019).
But they may be sensitive to noise and have high complexity. Some recent
methods consider the dynamic correlation among multiple variables, like
according to the decomposition Zhang et al. (2023a) and attention Su (2023).

  * •

Variant representation space of features.

Most algorithms change the original feature spaces, such as the well-known
PCA, SVD, LE. For TS data, there are frequency PCA Sundararajan (2021), 2D SVD
Ge (2023), temporal extension LE Huang (2019), etc. Additionally, the popular
representation learning methods summarized in Section 4.1 can also be used for
dimension reduction purposes.

Category | Subcategory | Purpose | Method | Technology | Pro | Con | Paper
---|---|---|---|---|---|---|---
Sample |  Data filtering |  Select high- quality data for training efficiency and model stability |  Filter missing and low-quality samples |  Preprocessing: remove missing, noisy, etc |  Necessary data processing steps, simple, intuitive |  Need predefinition. Possible incomplete or excessive filtering |  Filtering Sampling Denoising
Find important samples |  Estimate farthest boundary points |  Enhance the classification boundary |  Only targeting specific anomaly detection tasks |  Alam (2020) Mauceri (2020)
Evaluate suitable samples |  Clustering Prototyping |  Denoise and maintain data distribution |  Make models less robust and prone to overfitting |  Hayashi (2024) Li (2023)
Data augmenation |  Increase the number of samples to reduce bias or distortion |  Expand small datasets |  Time domain |  Simple, robust |  Noises occur without prior knowledge |  Window level Guo et al. (2023); Add noise Wen (2019)
Frequency domain |  Simple, robust |  Noises occur without prior knowledge |  Decompose Gao et al. (2020)
Generate |  Need less prior knowledge |  Possible lack of temporal features |  GAN-based Yoon et al. (2019); Diffusion models Shen and Kwok (2023)
Synthetise |  Controllable, flexible |  Need empirical formula |  Decompose expression Dooley et al. (2023); Correlation expression
Expand imbalanced -class datasets |  Oversampling |  Simple, intuitive |  Overfitting risk |  Zhao et al. (2022)
Loss |  Improved minority class prediction |  Sensitivity to Hyperparameters |  Ircio (2023)
Order arrangement |  Make models learn data in order to help convergence and generaliz- ation |  Easy-to-hard curriculum |  Predefined |  No additional training complexity |  Sensitivity to manual design, potential bias |  Wang et al. (2022)
Automatic |  Adaptive learning process |  Complexity overhead, training instability |  Self-paced learning; Transfer teacher; Reinforcement learning Portelas et al. (2020)
Other curriculum |  Cyclical |  Targeting time- series data |  Only for specific downstream tasks |  Sun et al. (2023c)
Feature |  Feature augmentation |  Enrich the information of original data |  Add static information |  Add statistic, description |  More comprehensive |  Need prior knowledge |  Yan et al. (2020) Hong et al. (2017)
Add dynamic information |  Add time features |  Model temporal irregular relations |  Need additional timestamp records |  Sun et al. (2021) Sun et al. (2020)
Add correlation features |  Establish relations between variants |  Possible redundancy and complexity |  Yu (2022) Jin (2023)
Representation learning |  Encode to high- dimensional space |  Deep feature discovery |  Interpretability challenges |  Meng et al. (2023)
Dimension reduction |  Reduce redundancy of high- dimensional data |  Invariant feature domain |  Similarity-based |  Simple and explainable |  Noise sensitivity, complexity |  Clustering; Distance Zhang et al. (2023a)
Attention-based |  Perceived sequent -ial features |  Still need to input all variables |  End-to-end; Two-step Su (2023)
Variant feature domain |  Time domain component analysis |  multicollinearity mitigation |  Outliers impact |  Sundararajan (2021) Ge (2023)
Frequency domain transformation |  Global description |  Lost time information |  Li et al. (2022)
Shape domain extraction |  Extract and show patterns |  Complexity, hard for high-dimensionality |  Yamaguchi et al. (2023)
Period |  Window size setting |  Select the window size of models |  Towards classi- cal models |  Warping window |  More flexible and automated |  Computational complexity |  González (2021)
Towards neural networks |  Input window size |  Model-agnostic Task-agnostic |  Need regressive hypothesis |  Imani et al. (2021)
Patching |  Adapt Transformers to time-series data |  Need experience, not automatic |  Nie et al. (2023)
Data segmentation |  Find meaningful segment sequences |  Unsupervised |  Similarity-based |  Intuitive grouping, no prior assumption |  Sensitive to distance metrics and outliers |  Bai et al. (2020)
Model-based |  Model interpretability |  Dependency on assumption |  ARMA; GMM;HMM
Supervised |  Classification |  Handle complex patterns |  Need task objectives or additional labels |  Ermshaus et al. (2023)
Length reduction |  Extract key sub-length sequences |  Longitudinal Dimension reduction |  Classical dimension reduction methods |  Can utilize different methods |  Only applicable to univariable data |  Sonbhadra (2023)
Fragment extraction |  Temporal pattern recognition |  Interpretable, handle multivariable data |  Need downstream task objective |  Ermshaus et al. (2022) Hong et al. (2022)

Table 2: Summary of Related Work on Data-centric Methods in Time Series
Analysis

##  5 Period Selection

TS data has the special length property as we described in Definition 3. TS
often exhibit temporally repeated states. For example, human activity data is
a series of repeated actions such as standing, walking, running, etc.
Partitioning TS into a sequence of states can simplify and interpret the
complex TS. The majority of TS period operations is dedicated to data
segmentation: The unsupervised segmentation methods are usually distance-based
or model-based Bai et al. (2020); The supervised methods are supported by
downstream objectives Ermshaus et al. (2023). However, because segmentation is
a specific task rather than the data processing and selection that we are
interested in, we will not go into it in depth below.

###  5.1 Window Size Setting

Determining an appropriate window size for TS models can reduce model
complexity and learn multi-span features.

  * •

Dynamic window size for classical models.

In DTW, obtaining the best performance requires setting its only parameter,
the maximum amount of warping $w$; In model-based methods such as the Gaussian
process, appropriate time-training intervals can extract unambiguous
structural information from kernel hyperparameters. “The larger/smaller the
better” needs to be permitted by the computational resources. Thus, in
addition to reliable feature representation, many works focus on the
complexity, such as utilizing semi-supervised dynamic optimization González
(2021).

  * •

Select the window size for neural networks.

Usually, except for some RNNs, when inputting TS data into a neural network,
it is not a single point, but a subsequence, e.g. the sliding window for RNNs,
the convolutional kernel for CNNs, and the patching for Transformers Wen et
al. (2023). These methods can be seen as the inherent structure of a model
such as Temporal Convolutional Networks (TCNs), or as additional mechanisms
before the model such as the external encoder Sun (2021).

In addition to the purpose of solving the online long term problem and end
effect in online learning Guo et al. (2023), most studies start with a
thorough modeling of the data from various levels, spans, and granularities,
especially for RNN- or CNN-based models Zhou et al. (2019). Currently, TS
Transformers (TSTs) are the most popular model used for TS forecasting.
PatchTST Nie et al. (2023) emphasizes the importance of TS
patching/tokenizing. Then self-attention focuses on the relations between
local patterns. However, there is currently no agreement on the size of the
patch. And the model-agnostic methods have not yet been implemented.

Meanwhile, the window size setting may be related to specific datasets, a
domain-agnostic method uses the moving average to set Imani et al. (2021), but
it can only be considered semi-automatic as the requirements of a candidate
set.

Category | Key Technology and Domain | Scenario | Potential Topic
---|---|---|---
Sample |  How to improve sample quality? Evaluation, data cleaning, large models |  Large model and massive datasets, noisy dataset |  $\bullet$ How to implement a data Juicer for TS-based large models?
How to increase the number of samples? Few-shot learning, imbalanced classification, federated learning |  Data collection difficulty, privacy. Small sample, skew dataset, cold start |  $\bullet$ How to balance skewed datasets? $\bullet$ How to create a larger synthetic dataset? $\bullet$ How to generate finer TS data from different granularities? E.g. can we design Diffusion models combined with the progressive network?
How to reduce the number of samples? Clustering, transfer learning |  Noisy datasets, representative samples, explanations |  $\bullet$ How to find representative samples in one-class? $\bullet$ What is the law between data quantity and model quantity of TS-based large models?
How to arrange the learning order? Curriculum learning, measurer, scheduler |  High training costs, mode convergence difficulty, improve training benefits |  $\bullet$ How to achieve the efficient teacher-student for TS large models? $\bullet$ How to solve the catastrophic forgetting problem in continuous learning by the learning order?
Feature |  How to select representative features? Correlation measurement, causal analysis |  Explain, denoise, reduce complexity, improve robustness |  $\bullet$ How to combine Granger causality analysis with dynamic hypergraphs?
How to embed and represent features? Representation learning, embedding, unsupervised/self-supervised learning, contrastive learning |  Unlabeled/partial-labeled data, the internal information is richer than annotations. Subtyping, clustering |  $\bullet$ How to achieve the TS universal representation as the foundation of the TS general model? $\bullet$ How to achieve a flexible attention that spans horizontally and longitudinally?
Period |  How to choose the appropriate window size for TS model? |  Use models from other fields, such as Transformers in NLP. Segmentation, understanding |  $\bullet$ How to design a flexible patching method for TS Transformers with dynamic window and spans? $\bullet$ How to design a general window selection method for different TS models?
|  How to extract key TS segments? Dimension reduction, pattern recognition, representation learning |  Interpretation, semantic segmentation, turning point recognition, staging |  $\bullet$ How to find key segments in long sequences that can interpret the downstream labels (object detection on TS data)?

Table 3: Potential Research Topics

###  5.2 Subsequence Extraction

The important insights from the TS data can be drawn by inspecting local
substructures, and not the recordings as a whole Ermshaus et al. (2022). For
example, ECG are medical TS that is repeated by normal or anomalous heartbeats
within a longer sequence. As such, many SOTA methods characterize TS by
inspecting local substructures Hong et al. (2022). And the method also needs
to pay attention to the noise caused by non-stationary characteristics.
Meanwhile, time pattern recognition is important for subtyping, staging, and
interpretation. The time pattern is made up of key points from TS. For
example, in medical mining tasks, it can discover disease stages, risk, and
biomarkers Sun et al. (2024).

TS length or span also can be treated as the dimension. The large sample
length increases the computing complexities together with its innate problem
of dimensionality curse, which can be solved through existing Dimension
reduction methods introduced in Section 4.2. But, towards the length
reduction, to the best of our knowledge, only one publication has been
reported Sonbhadra (2023).

##  6 Discussion

We have summarized the data-related work across diverse fields (Table 1), We
also distilled potential challenges and envisioned technological pathways
(Table 3), where we encapsulates the pivotal issues tailored to various
scenarios, alongside the topics we put forward.

In this section, we first propose some open problems and share our reflections
(Section 6.1), then discuss some potential research topics requiring further
exploration (Section 6.2). We hope that our newly proposed topics can give
research inspiration to someone involved.

###  6.1 Open Problems

  * •

More data or less data?

The exceptional performance of deep neural networks hinges significantly on
the abundance of training data. However, TS data is more specialized,
particularly in fields like medicine, where privacy and security are paramount
concerns. The largest open dataset is typically less than 10GB, significantly
smaller than those in Natural Language Processing (NLP) field Zhou et al.
(2023). Hence, many effort is dedicated to enhancing the dataset.

However, in our engineering experience within healthcare Hong and Zhao (2024)
and aerospace Sun et al. (2023b), what’s surprising is that the challenge
doesn’t arise due to data scarcity; quite the opposite, data is often
abundant. The challenge comes from a lack of well-managed methodologies for
data accumulation, processing, and management, particularly when there’s a
shortage of individuals with a background in data science. Therefore, data
organization, processing, and selection become increasingly crucial. Of
course, in scenarios where data collection is notably challenging, such as
with rare medical diseases, employing techniques like few-shot learning and
imbalanced learning becomes justifiable.

On the other hand, TS is often derived from human-made processes which adhere
to natural laws. E.g., the Earth’s revolution and rotation create seasonal and
diurnal patterns. It influences the rhythm and routines of human life and
production, resulting in a consistent timing pattern observed across TS data
in fields such as electricity, traffic, healthcare, and more, encompassing the
timings of hours, days, weeks, months, and years. Because of this potential
regularity or certain properties outlined in Section 2, there might not be a
necessity for an excessively large volume of data; it can be condensed.
Especially in the current trend of LLM, once an adequate amount of data is
acquired, researchers are contemplating methods to select high-quality data,
optimize data utilization, and enhance overall efficiency Chen et al. (2023).

Therefore, for TS data selection, we posit that a gradual process—initiating
with expansion and subsequently transitioning to compaction—holds more
validity. For data expansion, we can focus more on collection and management;
For data compaction, we focus more on filtering and evaluation.

  * •

General large model or specialized model?

LMs stand as the current trend, and there have been initial endeavors in
developing LM for TS data Jin et al. (2023). Although they has demonstrated
SOTA capabilities and, in some cases, exhibit few-shot or even zero-shot
forecasting abilities, we can’t confidently claim that they are genuinely
general. Their models are typically trained on TS datasets specific to
particular fields, where a forecasting dataset archive Wu et al. (2023) is
commonly used. We won’t delve into the extent of this archive’s
representativeness at the moment, but it’s worth noting that its validity can
be easily questioned. In a general sense, the true general model would involve
the capability to train on data from diverse fields concurrently and then
deploy it across various domains. Retraining LLM-inspired TS models in a
specific TS field can only produce a specific type of TST model, not a general
one.

Meanwhile, from a data-centric perspective, unlike NLP’s subjective text data
that humans can perceive, TS is more objective, consolidating and uniformly
describing them across various fields is challenging. These manifest in
differences in sampling rates at sample level, the variability in variable
number and correlations at feature level, and the potential for pattern
alterations at period level. Although many TS data exhibit seasonality and
trends, there’s still a substantial amount of data from chaotic systems that
are random or non-stationary. Moreover, most data selection methods are
tailored for specific models and tasks, complicating efforts to standardize or
unify them. For example, data augmentation methods applicable for
classification may not be valid for anomaly detection Wen et al. (2021).

Hence, in the current technological landscape, it’s more feasible to build
distinct TS-LMs tailored to individual fields.

###  6.2 Potential Topics

We provide many potential research topics as listed in the last column of
Table 3. Here we discuss some of them in detail:

  * •

Skewed, imbalanced, or long-tailed distributed data issues.

In most practical applications where data collection is feasible, skewed data
issues tend to be more severe than small samples. In electronic medical
records, typical cases make up a significant portion, whereas rare or orphan
diseases, despite receiving heightened attention, constitute a very small
fraction. Atrial fibrillation records account for less than 0.5% in ECG Chou
et al. (2020). In specific tasks, beyond the prevalent challenges of
imbalanced classification problems, it’s also crucial to note the significance
of imbalanced or long-tailed regression Stocksieker et al. (2023).

  * •

The scaling law between the number of TS data and that of TS model parameters.

To attain improved accuracy or and utilize continuously collected data, it
might be necessary to expand both the training dataset and the models. In NLP,
studies have investigated the scaling relationship between the amount of text
data and the parameters within LLMs, drawing conclusions that align with power
law Kaplan et al. (2020). But this conclusion might not be entirely applicable
to the TS data analysis. Hence, investigating the TS scaling law and crafting
dedicated data selection methods like data-juicer Chen et al. (2023) is
worthwhile for future research.

  * •

Address multiple problems uniformly through the sample-level data-augmented
generation.

Viewing forecasting, classification, imputation, and similar tasks as
challenges associated with generation can offer unified strategies.
Considering the entire sample can simplify the need to account for sequential
dependencies, feature correlations, irregular time, and other intricate
details. Diffusion models gradually climbed the ranks among SOTA methods for
TS data Yang et al. (2022a). Despite all this, there’s still a scarcity of
well-established research on frequency domain diffusion and progressive models
within TS analysis.

  * •

Approach data processing and selection by considering relationships
dynamically.

TS data evolves dynamically, with key features and segments undergoing
continual changes at various intervals. For instance, the concept of dynamic
Granger causality highlights this evolving nature Sun et al. (2023b). Thus,
akin to online learning and continual learning paradigms Sun et al. (2023a),
data selection at both the feature and length levels could be more dynamic,
immediate, and real-time-oriented. This suggests that a unified structure
could be established by integrating an encoder as a selector positioned ahead
of the TS model, e.g. it can complete dynamic patching for TST.

  * •

Utilize other data structures.

TS data can be roughly conceptualized with three sampling rate categories:
low, medium, and high. For low-frequency data, like diagnosis events, they can
be modeled in a tabular data format; For medium-frequency data, like vital
signs, they can be structured using graphs and hypergraphs that emphasize
feature correlation; For high-frequency data, like ECG physiological signals,
they can be converted into images. This approach broadens the scope of
available data selection methods, allowing for the utilization of techniques
from diverse fields, not confined solely to temporal structures.

##  7 Conclusion

This paper is the first systematic review to investigate the data-centric
methods for TS analysis as a specialized topic. We underscore the significance
of data management and organization within the context of current
technological trends. We acknowledge the significance of unified general
models for TS data but emphasize that currently, the domain-specific general
models and corresponding data selection methods are more practical and
effective. We recommend to select TS data from all perspectives of sample,
feature, and period. In the future, it is also necessary to study TS data-
centric methods for data labeling, model interpretability and fairness, prompt
engineering, and proposing new datasets or benchmarks.

## Acknowledgments

This work is supported by National Natural Science Foundation of China
(No.62172018, No.62102008) and Wuhan East Lake High-Tech Development Zone
National Comprehensive Experimental Base for Governance of Intelligent
Society.

## References

  * Alam [2020] Shamshe Alam.  Sample reduction using farthest boundary point estimation for support vector data description.  Pattern Recognit. Lett., 131:268–276, 2020.
  * Ashraf et al. [2023] Mohsena Ashraf, Farzana Anowar, and et al.  A survey on dimensionality reduction techniques for time-series data.  IEEE Access, 11:42909–42923, 2023.
  * Assaad et al. [2023] Charles K. Assaad, Emilie Devijver, and et al.  Survey and evaluation of causal discovery methods for time series.  In IJCAI, pages 6839–6844, 2023.
  * Bai et al. [2020] Yue Bai, Lichen Wang, Yunyu Liu, and et al.  Dual-side auto-encoder for high-dimensional time series segmentation.  In ICDM, pages 918–923, 2020.
  * Cai et al. [2022] Derun Cai, Moxian Song, Chenxi Sun, and et al.  Hypergraph structure learning for hypergraph neural networks.  In IJCAI, pages 1923–1929, 2022.
  * Chen et al. [2023] Daoyuan Chen, Yilun Huang, and et al.  Data-juicer: A one-stop data processing system for large language models.  CoRR, abs/2309.0203, 2023.
  * Chou et al. [2020] Yen-hsiu Chou, Shenda Hong, and et al.  An interpretable deep model for classifying imbalanced ecg data.  Neurocomputing, 417:64–73, 2020.
  * Deng [2021] Liwei Deng.  HIFI: anomaly detection for multivariate time series with high-order feature interactions.  In DASFAA, volume 12681, pages 641–649, 2021.
  * Dooley et al. [2023] Samuel Dooley, Gurnoor Singh Khurana, and et al.  Forecastpfn: Synthetically-trained zero-shot forecasting.  In NueurIPS, 2023.
  * Ermshaus et al. [2022] Arik Ermshaus, Patrick Schäfer, and Ulf Leser.  Window size selection in unsupervised time series analytics: A review and benchmark.  In PKDD, 2022.
  * Ermshaus et al. [2023] Arik Ermshaus, Patrick Schäfer, and Ulf Leser.  Clasp: parameter-free time series segmentation.  Data Min. Knowl. Discov., 37(3):1262–1300, 2023.
  * Franceschi [2019] Jean-Yves Franceschi.  Unsupervised scalable representation learning for multivariate time series.  In NeurIPS, pages 4652–4663, 2019.
  * Gao et al. [2020] Jingkun Gao, Xiaomin Song, and et al.  Robust time series anomaly detection via decomposition and convolutional neural networks.  In KDD MileTS, 2020.
  * Ge [2023] NingZhen Ge.  Unsupervised anomaly detection via two-dimensional svd and subspace reconstruction for multivariate time series.  Appl. Intell., 2023.
  * González [2021] Juan Luis Gómez González.  Window size optimization for gaussian processes in large time series forecasting.  In HAIS, pages 137–148, 2021.
  * Guo et al. [2023] Xin Guo, Wen-jing Li, and Jun-Fei Qiao.  A self-organizing modular neural network based on empirical mode decomposition with sliding window for time series prediction.  Appl. Soft Comput., 145:110559, 2023.
  * Hayashi [2024] Toshitaka Hayashi.  Distance-based one-class time-series classification approach using local cluster balance.  Expert Syst. Appl., 235:121201, 2024.
  * Hong and Zhao [2024] Shenda Hong and Qinghao Zhao.  Expanding electrocardiogram abilities for postoperative mortality prediction with deep learning.  The Lancet Digital Health, 6(1):e4–e5, 2024.
  * Hong et al. [2017] Shenda Hong, Meng Wu, and et al.  ENCASE: an ensemble classifier for ECG classification using expert features and deep neural networks.  In CinC, 2017.
  * Hong et al. [2022] Shenda Hong, Wenrui Zhang, and Chenxi Sun.  Practical lessons on 12-lead ecg classification: Meta-analysis of methods from physionet/computing in cardiology challenge 2020.  Front. in Physiol., 2022.
  * Huang [2019] Xuan Huang.  A review on dimensionality reduction techniques.  Int. J. Pattern Recognit. Artif. Intell., 33(10):1950017:1–1950017:25, 2019.
  * Imani et al. [2021] Shima Imani, Alireza Abdoli, and et al.  Multi-window-finder: Domain agnostic window size for time series data.  In KDD MileTS, 2021.
  * Ircio [2023] Josu Ircio.  Minimum recall-based loss function for imbalanced time series classification.  IEEE Trans. Knowl. Data Eng., 35(10):10024–10034, 2023.
  * Jin et al. [2023] Ming Jin, Qingsong Wen, and et al.  Large models for time series and spatio-temporal data: A survey and outlook.  CoRR, abs/2310.10196, 2023.
  * Jin [2023] Ming Jin.  A survey on graph neural networks for time series: Forecasting, classification,imputation, and anomaly detection.  CoRR, abs/2307.03759, 2023.
  * Kang et al. [2020] Yanfei Kang, Rob J. Hyndman, and et al.  Generating time series with diverse and controllable characteristics.  Stat. Anal. Data Min., 13(4):354–376, 2020.
  * Kaplan et al. [2020] Jared Kaplan, Sam McC., and et al.  Scaling laws for neural language models.  CoRR, 2020.
  * Koh [2024] Woosung Koh.  Curriculum learning and imitation learning for model-free control on financial time-series.  In AAAI AI4TS, pages 1–7, 2024.
  * Li et al. [2022] Xuegui Li, Shuo Feng, and et al.  Applications of kalman filtering in time series prediction.  In ICIRA, volume 13457, pages 520–531, 2022.
  * Li [2023] Yuxin Li.  Prototype-oriented unsupervised anomaly detection for multivariate time series.  In ICML, volume 202, pages 19407–19424, 2023.
  * Lin et al. [2023] Lequan Lin, Zhengkun Li, Ruikun Li, and et al.  Diffusion models for time series applications: A survey.  CoRR, abs/2305.00624, 2023.
  * Ma et al. [2019] Qianli Ma, Jiawei Zheng, Sen Li, and et al.  Learning representations for time series clustering.  In NeurIPS, pages 3776–3786, 2019.
  * Ma et al. [2021] Qianli Ma, Chuxin Chen, Sen Li, and et al.  Learning representations for incomplete time series clustering.  In AAAI, pages 8837–8846, 2021.
  * Mauceri [2020] Stefano Mauceri.  Dissimilarity-based representations for one-class classification on time series.  Pattern Recognit., 100:107122, 2020.
  * Meng et al. [2023] Qianwen Meng, Hangwei Qian, Yong Liu, and et al.  Unsupervised representation learning for time series: A review.  CoRR, abs/2308.01578, 2023.
  * Nie et al. [2023] Yuqi Nie, Nam H. Nguyen, and et al.  A time series is worth 64 words: Long-term forecasting with transformers.  In ICLR, 2023.
  * Portelas et al. [2020] Rémy Portelas, Cédric Colas, Lilian Weng, and et al.  Automatic curriculum learning for deep RL: A short survey.  In IJCAI, pages 4819–4825, 2020.
  * Shen and Kwok [2023] Lifeng Shen and James T. Kwok.  Non-autoregressive conditional diffusion models for time series prediction.  In ICML, pages 31016–31029, 2023.
  * Sonbhadra [2023] Sanjay Sonbhadra.  Target-class guided sample length reduction and training set selection of univariate time-series.  Appl. Intell., 53(6):7056–7073, 2023.
  * S.Park [2019] Daniel S.Park.  Specaugment: A simple data augmentation method for automatic speech recognition.  In ISCA, pages 2613–2617, 2019.
  * Stocksieker et al. [2023] Samuel Stocksieker, Denys Pommeret, and Arthur Charpentier.  Data augmentation for imbalanced regression.  In AISTATS, pages 7774–7799, 2023.
  * Su [2023] Qiqi Su.  Modular neural networks for time series forecasting: Interpretability and feature selection using attention.  CoRR, abs/2311.16834, 2023.
  * Sun et al. [2020] Chenxi Sun, Shenda Hong, and et al.  A review of deep learning methods for irregularly sampled medical time series data.  CoRR, abs/2010.12493, 2020.
  * Sun et al. [2021] Chenxi Sun, Shenda Hong, and et al.  Time encoding echo state network for prediction based on irregularly sampled time series data.  In IJCAI, 2021.
  * Sun et al. [2023a] Chenxi Sun, Hongyan Li, and et al.  Continuous diagnosis and prognosis by controlling the update process of deep neural networks.  Patterns, 4(2), 2023.
  * Sun et al. [2023b] Chenxi Sun, Hongyan Li, and et al.  A multi-model architecture based on deep learning for aircraft load prediction.  Nat Commun Eng, 2(47):1–12, 2023.
  * Sun et al. [2023c] Chenxi Sun, Hongyan Li, Moxian Song, and et al.  Curricular and cyclical loss for time series learning strategy.  CoRR, abs/2312.15853, 2023.
  * Sun et al. [2023d] Chenxi Sun, Yaliang Li, Hongyan Li, and Shenda Hong.  TEST: text prototype aligned embedding to activate llm’s ability for time series.  CoRR, 2023.
  * Sun et al. [2024] Chenxi Sun, Hongyan Li, and et al.  Time pattern reconstruction for classification of irregularly sampled time series.  Pattern Recognition, 147:110075, 2024.
  * Sun [2021] Chenxi Sun.  Interpretable time-aware and co-occurrence-aware network for medical prediction.  BMC Med Inform Decis Mak, 21(1):305, 2021.
  * Sundararajan [2021] R R. Sundararajan.  Principal component analysis using frequency components of multivariate time series.  Comput. Stat. Data Anal., 157:107164, 2021.
  * Tao et al. [2021] Yaling Tao, Kentaro Takagi, and et al.  Clustering-friendly representation learning via instance discrimination and feature decorrelation.  In ICLR, 2021.
  * Wang et al. [2022] Xin Wang, Yudong Chen, and Wenwu Zhu.  A survey on curriculum learning.  IEEE Trans. Pattern Anal. Mach. Intell., 44(9):4555–4576, 2022.
  * Wen et al. [2021] Qingsong Wen, Liang Sun, Fan Yang, and et al.  Time series data augmentation for deep learning: A survey.  In IJCAI, pages 4653–4660, 2021.
  * Wen et al. [2023] Qingsong Wen, Tian Zhou, Chaoli Zhang, and et al.  Transformers in time series: A survey.  In IJCAI, pages 6778–6786, 2023.
  * Wen [2019] Tailai Wen.  Time series anomaly detection using convolutional neural networks and transfer learning.  In IJCAI AI4IoT, pages 266–275, 2019.
  * Woo et al. [2022] Gerald Woo, Chenghao Liu, and et al.  Cost: Contrastive learning of disentangled seasonal-trend representations for time series forecasting.  In ICLR, 2022.
  * Wu et al. [2023] Haixu Wu, Tengge Hu, and et al.  Timesnet: Temporal 2d-variation modeling for general time series analysis.  In ICLR, 2023.
  * Yamaguchi et al. [2023] Akihiro Yamaguchi, Ken Ueno, and et al.  Time-series shapelets with learnable lengths.  In CIKM, pages 2866–2876, 2023.
  * Yan et al. [2020] Li Yan, Hai-Tao Zhang, and et al.  An interpretable mortality prediction model for covid-19 patients.  Nat Mach Intell, 2:283–288, 2020.
  * Yang et al. [2022a] Ling Yang, Shenda Hong, and et al.  Diffusion models: A comprehensive survey of methods and applications.  ACM Comput. Surv., 56(105):1–9, 2022.
  * Yang et al. [2022b] Ling Yang, Shenda Hong, and Luxia Zhang.  Spectral propagation graph network for few-shot time series classification.  CoRR, abs/2202.04769, 2022.
  * Yoon et al. [2019] Jinsung Yoon, Daniel Jarrett, and Mihaela van der Schaar.  Time-series generative adversarial networks.  In NeurIPS, pages 5509–5519, 2019.
  * Yu [2022] Hongyuan Yu.  Regularized graph structure learning with semantic knowledge for multi-variates time-series forecasting.  In IJCAI, pages 2362–2368, 2022.
  * Zhang et al. [2023a] Qingzhen Zhang, Chaoqi Zhang, Langfu Cui, and et al.  A method for measuring similarity of time series based on series decomposition and dynamic time warping.  Appl. Intell., 53(6):6448–6463, 2023.
  * Zhang et al. [2023b] Wenrui Zhang, Shenda Hong, and et al.  Cross reconstruction transformer for self-supervised time series representation learning.  TNNLS, 2023.
  * Zhao et al. [2022] Pu Zhao, Chuan Luo, and et al.  T-SMOTE: temporal-oriented synthetic minority oversampling technique for imbalanced time series classification.  In IJCAI, pages 2406–2412, 2022.
  * Zhou et al. [2019] Yuxi Zhou, Shenda Hong, and et al.  K-margin-based residual-convolution-recurrent neural network for atrial fibrillation detection.  In IJCAI, 2019.
  * Zhou et al. [2023] Tian Zhou, PeiSong Niu, Xue Wang, Liang Sun, and Rong Jin.  One fits all:power general time series analysis by pretrained lm.  In NeurIPS, 2023.