11institutetext:  Université de Lille, CNRS, Inria, Centrale Lille, UMR 9189 –
CRIStAL, France
11email<EMAIL_ADDRESS>
22institutetext: Université de Lille, Inria, CNRS, Centrale Lille, UMR 9189 –
CRIStAL, France
33institutetext: CIM, Lille, France
44institutetext: Berger-Levrault, Montpellier, France

# Modular Moose: A new generation
software reverse engineering environment

Nicolas Anquetil 11 0000-0003-1486-8399 Anne Etien 11 0000-0003-3034-873X
Mahugnon H. Houekpetodji 1133 Benoit Verhaeghe 1144 Stéphane Ducasse 22
0000-0001-6070-6599 Clotilde Toullec 22 Fatiha Djareddir 33 Jerôme Sudich 33
Moustapha Derras 44

###### Abstract

Advanced reverse engineering tools are required to cope with the complexity of
software systems and the specific requirements of numerous different tasks
(re-architecturing, migration, evolution). Consequently, reverse engineering
tools should adapt to a wide range of situations. Yet, because they require a
large infrastructure investment, being able to reuse these tools is key. Moose
is a reverse engineering environment answering these requirements. While Moose
started as a research project 20 years ago, it is also used in industrial
projects, exposing itself to all these difficulties. In this paper we present
ModMoose, the new version of Moose. ModMoose revolves around a new meta-model,
modular and extensible; a new toolset of generic tools (query module,
visualization engine, …); and an open architecture supporting the
synchronization and interaction of tools per task. With ModMoose, tool
developers can develop specific meta-models by reusing existing elementary
concepts, and dedicated reverse engineering tools that can interact with the
existing ones.

##  1 Introduction

As software technologies evolve, old systems experience a growing need to
follow this evolution. From the end-user point of view, they need to offer
functionalities entirely unforeseen when they were first conceived. From the
developer point of view, they need to adapt to the new technologies that would
allow one to implement these functionalities [8].

Given the size of these systems and lack of resources in the industry, such
evolution can only happen with the help of automated tooling [3, 15].
Concurrently, because such tooling requires a large infrastructure investment,
it must be generic and reusable across technologies and reverse engineering
tasks. This tooling needs to cope with the following problems:

  * •

_Diversity of languages and analyses._ Many programming languages and versions
of such languages are used in the industry. Meta-modeling was proposed to cope
with that diversity, but it does not solve all problems. Software reverse
engineering requires to represent source code with a high degree of details
that are specific to each programming language and to the reverse engineering
tasks themselves. How to model the different needs for details while remaining
generic is an issue meta-models have not tackled yet.

  * •

_Sheer amount of data._ Tools are particularly useful for large systems
(millions of lines of code). The size of the analyzed systems stresses the
modeling capabilities of the tools and the efficiency of the analyses.

  * •

_Reverse engineering task diversity._ The evolution needs are numerous, from
restructuring a system towards a micro component architecture [5], to
migrating its graphical user interface [24], evolving its database [7], or
cleaning the code [2]. This calls for various tools (query module,
visualizations, metrics, navigation, analyses) that must integrate together to
answer each need still acting as a coherent tool.

  * •

_Specific tasks require specific tools._ Some tools can be useful in various
situations, but specialized tools are also needed111According to Bruneliere
[6], the plurality of reengineering projects requires adaptable/portable
reverse engineering tools.. If a tool is too specific to a technology, its
advantages are lost when working with others. It is important that they can be
easily added and integrated into a reverse engineering environment.

We report here our experience with Moose, a software analysis platform
answering to the basic needs of reverse engineering, reusable in different
situations, extensible and adaptable to new needs [21]. Moose was initiated as
a research project and still fulfills this role. But it is also used in a
number of industrial projects [2, 7, 24].

Our redesign of Moose has the following goals: (1) the ability to develop
specific meta-models by reusing existing elementary concepts, (2) the ability
to develop dedicated reverse engineering tools by configuring and composing
existing tools or developing new ones, (3) the ability to seamlessly integrate
new tools in the existing reverse engineering environment. ModMoose, the new
version of Moose described in this article, revolves around a modular
extensible meta-model, a new toolset of generic tools (_e.g.,_ query module,
visualization engine) and an open architecture supporting the synchronization
and interaction of tools.

The contributions of this article are (1) the description of ModMoose new
generation reverse engineering environment, (2) FamixNG supporting the
definition of new meta-models based on the composition of elementary concepts
implemented using stateful traits, and (3) a bus architecture to support the
communication between independent and reusable tools.

In the following sections, we discuss the difficulties of having a generic,
multi-language, software meta-model (Section 2) and how we solved them with
FamixNG (Section 3). Then, we briefly present the new generic tools and how
the new open architecture of ModMoose supports the collaboration of group of
tools and their synchronization (Section 4). We conclude in Section 5.

##  2 Software Representation and Reverse Engineering Tool Challenges

To analyze code, tools need to represent it. Such a representation should be
_generic_ (programming languages are all build around similar concepts that
should be reusable), support _multi-language_ , and _detailed_ (a proper
analysis must pay attention to little, meaningful, details).

Each one of these challenges may be tackled by meta-models. For example,
existing IDEs (Integrated Development Environments) like Eclipse, or software
quality tools (_e.g.,_ CAST, SonarQube) offer tools based on an internal
representation of the source code. But in Eclipse, the model is not the same
for version management and source code highlighting. In that way, they can
represent with enough details the analyzed systems. The Lightweight
Multilingual Software Analysis approach proposed by Lyons _et al.,_ uses
different parsers (one for each programming language analyzed) to populate
their _different_ models: “variable assignment locations”, or “reverse control
flow”. The challenge is still how to support the development of multiple
dedicated and specific meta-models for different languages in a tractable way.

###  2.1 Problem: Handling the Diversity of Languages

Many publications highlight the need, for modern software analysis tools, to
deal with multi-language systems (_e.g.,_ [9, 13, 18, 19, 20]). The generality
and reusability of the analysis tools depend on their ability to work with all
these languages.

Meta-models were supposed to solve this issue. Naively, one could hope that a
single “object-oriented meta-model” would be able to represent programs in all
OO programming languages and another “procedural meta-model” would represent
all procedural programming languages.

Thus, for example, in the Dagstuhl Middle Meta-model [18] “the notions of
routine, function and subroutine are all treated the same.” But in practice,
one soon realizes that “various programming languages have minor semantic
differences regarding how they implement these concepts” ([18] again.) To be
meaningful and allow precise analyses, the meta-models must represent these
little differences and the tools must “understand” their specific semantics.
As a consequence, Washizaki _et al.,_ [25] report that “even if a meta-model
is stated to be language independent, we often found that it actually only
supports a very limited number of languages.” A good meta-model must represent
all details to allow meaningful analyses. Software is not linear, small
details may have huge impacts. Reverse engineering is about abstracting from
the source code, but also sticking tightly to it because one does not know in
advance what “details” are making a difference for different tasks.

Visibility rules are a concrete example of the complexity to represent varying
details for apparently universal concepts: Abstracting the API of a class
requires understanding the visibility rules of classes, methods, and
attributes. They are not exactly the same for Java (public, protected,
private, default package, and now export) or C++ (public, protected, private,
and friend). And the tools need to be aware of each different semantics to
make the proper inferences.

Other problems are raised by the containment relationship (_i.e.,_ ownership).
Many meta-models assume a hierarchical containment tree: all entities (except
the root entity) are owned by exactly one parent entity. However, in C#, class
definitions may be scattered over several files (_partial_ definitions), and
in Objective-C or Smalltalk a package may add methods to a class defined in
another package (extension methods). In these contexts, the notion of single
owner is less consensual.

Our experience designing and using a programming language meta-model, Famix,
for more than 15 years showed that we need dedicated meta-models for each
language.

###  2.2 Single inheritance: Tyranny of the dominant decomposition

Moose was based on a single meta-model (Famix [10, 11, 21]) extensible with
plugins. It did allow to model many different programming languages222Some
languages are only partially supported (Ada, C, C++, FORTRAN,
4D333http://en.wikipedia.org/wiki/4th_Dimension_(software), Java, JavaScript,
Mantis444http://www.lookupmainframesoftware.com/soft_detail/dispsoft/339,
Pharo, PHP, PostgresSQL/PgPlSQL,
PowerBuilder555http://en.wikipedia.org/wiki/PowerBuilder), and even other
sources of information (_e.g.,_ SVN logs). But extending Famix was an awkward
process since it relied on a single inheritance tree of concepts.

To address language variety, the choice was to have core entities representing
“typical features” of programming languages (_e.g.,_ function, methods,
package, class) and extend this core for language specific entities (_e.g.,_
the macros in C). The Dagstuhl Middle Meta-model [18] made the same choice.

However this was only possible through a careful definition of the meta-model
and the hierarchy of modeled entities. In practice, Famix core was tailored
towards Java and Smalltalk, its initial focus. Several extensions covered
other OOP languages with varying degrees of success. For example, Enum and
Class were both Types (see Figure 1, right). In Java, both accept an
implements relationship to Interface (another type). Therefore this
relationship is defined at the level of the Type concept. But then, a
PrimitiveType (also a Type) inherits this same property which does not make
sense in Java, and also an Interface is allowed to implements another
Interface, which again is not correct in Java.

Other generic meta-models have the same issue. In the Dagstuhl Middle Meta-
model [18] (see Figure 1, left), all relationships (_e.g.,_ IsMethodOf,
Includes, Invokes) may occur between ModelElements (_e.g.,_ Type, Routine,
Field). As a result many entities (_e.g.,_ Field) accept relationships
(_e.g.,_ IsMethodOf) that they don’t use.

Figure 1: Examples of the Dominant Decomposition problem in the Dagstuhl
Middle Meta-model (left) and in Famix (right)

A generic single inheritance meta-model thus turns out to be too permissive
with entities owning properties that they don’t use. Incidentally, this has
consequences on the size of the models since these properties occupy uselessly
memory that multiplied by the number of instances (tens or hundreds of
thousands) adds up to significant memory loss.

EMF (Eclipse Modeling Framework) attempts to solve the problems raised by
single inheritance inheritance tree. EMF implementation uses inheritance and
interfaces, thus mimicking multiple inheritance. But this still imposes to
choose one dominant decomposition for inheritance and the other decompositions
have to implement interfaces’ API.

###  2.3 ModMoose’s Goals

To design a new generation and modular reverse engineering environment we set
goals at two different levels: meta-modeling and tooling.

##### Meta-modeling goals.

We are looking for generic modeling that leaves open a wealth of analyses
(metrics, dead-code, design patterns, anti-patterns, dependency analysis,
etc). So we needed a unified representation that:

  1. 1.

Supports the precise representation of various programming language features
(identified in [22]);

  2. 2.

Is compatible with many analysis tools (the _processing environment_
identified in [25]);

  3. 3.

Is wary of memory consumption (identified in [17]);

  4. 4.

Is extensible (allow for easy addition of new languages).

##### Tooling goals.

Tools are needed to [3, 15]:

  * •

Automate tedious tasks to bring tremendous speed-up with less errors.666A
source code model computes in seconds a method call graph that takes weeks to
recover by hand. We had the case in two different companies.

  * •

Handle large quantity of data and in our case all the details of multi-
millions lines of code. If information size can become an issue even for
automated tools, this is several orders of magnitude above what a human can
handle.

  * •

Summarize data to allow a human abstracting a big picture understanding.

  * •

Quickly verify hypotheses so that there is little or no cost attached to wrong
hypotheses, and several concurrent hypotheses can easily be compared.

##  3 A Composable Meta-model of Programming Languages

FamixNG is a redesign of Famix around the idea of composing entities from
language properties represented as traits [23]. With FamixNG (See Section
3.1), one defines a new meta-model out of reusable elements describing
elementary concepts (See Section 3.2).

###  3.1 FamixNG

To support the reuse of elementary language concept, FamixNG relies on
_stateful traits_ [23].

##### What is a trait?

“A trait is a set of methods, divorced from any class hierarchy. Traits can be
composed in arbitrary order. The composite entity (class or trait) has
complete control over the composition and can resolve conflicts explicitly”.
In their original form, traits were stateless, _i.e.,_ pure groups of methods
without any attributes. Stateful traits extend stateless traits by introducing
a single attribute access operator to give clients of the trait control over
the visibility of attributes [23]. A class or another trait is composed from
traits and it retains full control of the composition being able to ignore or
rename some methods of the traits it uses.

In FamixNG, all properties that were previously defined in classes are now
defined as independent traits: For example, the trait TNamedEntity only
defines a name property (a string) and may belong to trait TNamespaceEntity.
Therefore any entity using this trait will have a name and can be part of a
namespace entity. Similarly, entities (presumably typed variables or
functions) composed with the trait TTypedEntity have a declaredType pointing
to an entity using the trait TType.

Four types of traits can be used to compose a new meta-model:

Associations

represent usage of entities in the source code. This includes the four former
associations of Famix: inheritance, invocation (of a function or a method),
access (to a variable), and reference (to a type). In FamixNG, we also found a
need for three more specialized associations: DereferencedInvocation (call of
a pointer to a function in C), FileInclude (also in C), and TraitUsage.

Technical

traits do not model programming language entities but are used to implement
Moose functionality. Currently, this includes several types of TSourceAnchors,
associated to TSourcedEntity to allow recovering their source code (a typical
TSourceAnchor contains a filename, and start and end positions in this file.)
Other _Technical traits_ implement software engineering metric computation, or
are used to implement the generic query engine of Moose (see Section 4.2).
There are 16 _Technical traits_ currently in FamixNG.

Core

traits model composable properties that source code entities may possess. This
includes TNamedEntity and TTypedEntity (see above), or a number of entities
modeling ownership: TWithGlobalVariables (entities that can own
TGlobalVariables), TWithFunctions (entities that can own TFunctions),… There
are 46 _Core traits_ currently in FamixNG including 38 traits modeling
ownership of various possible kind of entities.

Terminal

traits model entities that can be found in the source code such as Functions,
Classes, Exceptions, …These entities are often defined as a composition of
some of the _Core traits_. For example, TClass is composed of:
TInvocationsReceiver (class can be receiver of static messages), TPackageable,
TType (classes can be used to type other entities), TWithAttributes,
TWithComments, TWithInheritances, TWithMethods. The name _Terminal trait_
refers to the fact that they can be used directly to create a programming
language concept (a class, a package), whereas _Core traits_ are typically
composed with other traits to make a meaningful programming language concept.
There are 38 such _Terminal traits_ currently in FamixNG.

Contrary to the old Famix, the _Terminal traits_ are rather minimal
definitions (intersection of languages) than maximal ones (union of
languages). They are intended to have only the properties found in any
programming language. For example TClass does not use the TWithModifiers trait
(for attaching visibility) since not all languages explicitly define
visibility of classes. For any given meta-model, additional properties may be
added to these “ _Terminal traits_ ”, for example JavaClass uses the traits
TClass and TWithModifiers since Java classes can be declared public, private,
…More specialized languages (_e.g.,_ SQL) can always compose new entities from
the set of _Core traits_ offered by FamixNG.

We validated this new approach by defining different meta-models (_e.g.,_ GUI
meta-model [24], Java, OpenStack cloud infrastructure, Pharo, PowerBuilder,
SQL/PLPGSQL).

###  3.2 Creating a new Meta-model with FamixNG

To create a new meta-model, the developer may extend an existing meta-model or
start from scratch. This later case can be relevant if the language to model
is quite different from what ModMoose currently handles, such as a specific
domain or SQL. For example, while being different from procedural languages,
stored procedures in SQL bear resemblance to functions and can be composed
from the same FamixNG traits.

Figure 2 shows the definition of a toy meta-model (left) with the meta-model
itself (right). There are four entities: the root Entity, and File, Commit and
Author. In grey, the TNamedEntity trait, part of FamixNG. The embedded DSL
needs first the entity creation, then definition of inheritance relationships,
properties of each entity, and finally the relationships between entities. To
improve readability, entity descriptions are stored in variables having the
same name (_e.g.,_ author variable for Author entity description). One can see
an example of using a predefined trait, TNamedEntity that adds a “name :
String” property to the classes using it. The same syntax, “\--|>”, is used
for class inheritance and traits usage. From this, the builder generates all
the corresponding classes and methods.

⬇

entity := builder newClassNamed: ’Entity’.

file := builder newClassNamed: ’File’.

commit := builder newClassNamed: ’Commit’.

author := builder newClassNamed: ’Author’.

"inheritance"

file \--|> entity.

file \--|> TNamedEntity.

commit \--|> entity.

author \--|> entity.

author \--|> TNamedEntity.

"properties"

commit property: ’revision’ type: #Number.

commit property: ’date’ type: #Object.

commit property: ’message’ type: #String.

"associations"

file *-* commit.

commit *- author.

Figure 2: A simple File/Commit/Author meta-model (left: building script;
right: UML view; gray: predefined FamixNG trait)

##### Meta-model reuse and extension.

The extension of existing meta-model is done by specifying which meta-model to
extend (similarly to import instruction in Java), then the existing traits can
be reused at will. It is also possible to manage multi-language systems by
composing several meta-models. We cannot illustrate these two points for lack
of space.

##  4 ModMoose: A Reverse Engineering Environment

Program comprehension is still primarily a manual task. It requires knowledge
on computer science, the application domain, the system, the organization
using it, etc [1]. Yet the sheer size of the current software systems
(millions of lines of code, hundred of thousands of entities modeled)
precludes any software development team to fully understand these systems. Any
significant program comprehension activity imposes the use of specialized
tools.

ModMoose lets software engineers design specialized reverse engineering tools
by (1) using infrastructure tools (see Section 4.2); (2) taking advantage of a
bus architecture supporting smart interactions between tools (see Section
4.3); and, (3) reuse a set of generic and configurable tools (see Section
4.4).

###  4.1 ModMoose Architecture

Figure 3: ModMoose architecture.

ModMoose is architectured on the following principles:

  * •

Tools are part of the ModMoose environment which acts as a master and
centralizes data;

  * •

Tools communicate through buses, they “read” model entities on their bus(es)
and “write” entities back on their bus(es) (see below);

  * •

Tools are focusing on a single task: _e.g.,_ the Query Browser works on a set
of model entities and produces another set of entities.

These simple principles ensure that tools can be easily added to ModMoose and
collaborate between themselves in a flexible manner.

###  4.2 Infrastructure Tools

We identified three important requirement to analyze software systems: (i)
query and navigate a model to find entities of interest; (ii) visualise the
software to abstract information; and (iii) annotate entities to represent
meaning and reach a higher level of abstraction.

####  4.2.1 Moose Query

is an API to programmatically navigate FamixNG models. For any FamixNG meta-
model, Moose Query777https://moosequery.ferlicot.fr/ computes the answer to
generic queries:

  * •

containment: parent or children entities of a given type from a current
entity.

  * •

dependencies: following incoming or outgoing associations from or to a given
entity. Dependencies are also deduced from parent/children relationships,
_e.g.,_ dependencies between methods can be uplifted to dependencies between
their parent classes or parent packages.

  * •

description: all properties of a given entity.

In a FamixNG meta-model, relationships denoting a containment are declared as
such in the association part of the meta-model construction (see Figure 2),
_e.g.,_ method *-<> class or class <>-* method. Containment queries may go up
and down the containment tree based on the expected entity type: _e.g.,_ from
a Method one can ask to go up to any entity(ies) of type Class owning it.

Navigating dependencies is based on containment and association relationships.
It is possible to navigate through a specific association (_e.g.,_ “all
invoked methods”) or all types of association (_e.g.,_ “all dependencies”) to
other elements. Direction of navigation (incoming, outgoing) must be
specified. Dependencies between children of entities can be automatically
abstracted at the level of their parents.

##### Package level communication example.

A frequent query is to find how packages interact with each other via method
calls. This can be done by iterating over all the methods of all the classes
of a package, collecting what other methods they invoke. From this one finds
the parent classes and parent packages of the invoked methods.

With Moose Query, such a query is simply expressed:
(aPackage queryOutgoing: FamixInvocation) atScope: FamixPackage (_i.e.,_ find
all invocations stemming from aPackage, and raise the result at the level of
the receiving package).

##### Stored procedures referencing a column example.

In SQL, one may want to know all the stored procedures888functions directly
defined inside the database management system accessing a given column of a
database table. These stored procedures can directly reference columns, in the
case of triggers, or contain SQL queries that reference the columns. SQL
queries contain clauses that themselves can contain other queries. So, the
answer can be computed from a given column of a table, by collecting the
references targeting this column, and recursively analyze them to identify the
ones that are inside a stored procedure. Due to the possible nesting of SQL
queries, this has to be a recursive process.

With Moose Query, this query is simply expressed as follows:
aColumn queryIncomingDependencies atScope: FamixSQLStoredProcedure (_i.e.,_
find all incoming dependencies to aColumn and raise the result at the level of
the stored procedures. Incoming dependencies not stemming from a stored
procedures are simply dropped here).

##### Analysis.

These examples show that MooseQuery API is independent from the meta-model
used. Obviously, each query depends on the meta-model of the model it is
applied (one cannot ask for methods in a SQL model). In the first example
(Java), there are different kind of dependencies in the model (invocation,
inheritance, access, reference), therefore the kind one is interested in must
be specified (queryOutgoing: FamixInvocation). In the second example there
exist only one kind of dependencies to a column. Consequently, the query can
be simpler: queryIncomingDependencies.

In the second example, Moose Query implicitly filters out all queries that do
not stem from a stored procedure. In the first example, no such filtering
occurs since all methods belong to a package in Java.

####  4.2.2 Visualisation Engines:

ModMoose uses Roassal [4], a generic visualization engine, to script
interactive graphs or charts. Roassal is primarily used to display software
entities in different forms or colors and their relationships. Possible
examples are to display the classes of a package as a UML class diagram, or as
a Dependency Structure
Matrix999https://en.wikipedia.org/wiki/Design_structure_matrix. Roassal
visualization are interactive.

ModMoose also uses the Telescope, more abstract, visualization engine that
eases building new visualizations in terms of entities, their relationships,
their forms and colors, positioning, etc [16]. Telescope offers abstractions
such as predefined visualizations and interaction, and relies on Roassal to do
the actual drawing. For example, Telescope offers a predefined “Butterfly”
visualization, centered on an entity and showing to the left all its incoming
dependencies and to the right all its outgoing dependencies.

####  4.2.3 Tags

are labels attached by the user to any entities either interactively or as
result of queries [14]. Tags enrich models with extra information. They have
many different uses:

  * •

to decorate entities with a virtual property _e.g.,_ tagging all methods
directly accessing a database.

  * •

to represent entities that are not directly expressed in the programming
language constructs: _e.g.,_ representing architectural elements or tagging
subsets of a god class’ members as belonging to different virtual classes [2].

  * •

to mark to do and done entities in a repetitive process.

An important property of ModMoose tags is that they are first class entities.
Tags are not only an easy way to search entities, they can also be manipulated
as any other model entity: a query may collect all dependencies from or to a
tag, the visualization engines may display tags as container entities with
actual entities within them, one can compute software engineering metrics
(_e.g.,_ cohesion/coupling) on tags considered as virtual packages.

###  4.3 Smart Tool Interactions through Buses

The power of ModMoose comes from the generic interaction of specialized tools.
In particular, it is key that: (i) different tools display the _same entities_
from different points of view, or (ii) different instances of the same tool
may be used to compare different entities from the same point of view.
ModMoose supports such scenarios using an open architecture based on buses and
tool behavior controls.

##### Communication Buses.

Tools communicate through buses. They read entities produced by other tools on
the buses and write entities on the same buses for other tools to consume.
Several buses can be created, and groups of tools can be set on different
buses. This allows one to explore different parts of the model or different
courses of actions: _e.g.,_ imagine two buses each one attached to a Query
browser and a Dependency Graph browser (described in Section 4.4). Each Query
browser selects different entities from the model, the Dependency Graph
browsers display them, and the two buses ensure interaction within each pair
of browsers.

Tools can also be detached from all available buses to keep their current
result and presumably allow one to compare it with other results from other
instances of the same tools.

Since tools can be attached to more than one bus, a tool can be set as a
bridge between all buses: It listens to all buses and forwards activity on
them. A natural candidate for this is the Logger (described in Section 4.4).
By selecting an interesting set of entities in the Logger, one can propagate
these entities to all buses, thus synchronizing them.

##### Tool Behavior Controls.

On top of the bus architecture, three behavior controls (_frozen_ ,
_highlighting_ , _following_) fine-tune the tool reaction to bus events:

  * •

Following is the default state where a tool reads entities that pass on its
bus(es) and writes entities to the same bus(es). In such mode, a tool reacts
to read entities.

  * •

Frozen is a state where the tool does not listen to the bus(es) for incoming
entities and therefore keeps its state and display independent of entities
written on the bus(es). The tool is not detached from bus(es) and can still
write entities on them. A reverse engineer can interact with a frozen tool and
other tools will be informed of this.

  * •

Highlighting is a sort of intermediary state, where the current entities on
which the tool works remain the same (similar to Frozen), but when new
entities pass on the bus(es), the tool looks for them in its “frozen” display
and highlight them if they are present (they are ignored otherwise). This is
useful to highlight a new result in the context of a preceding result that was
already displayed in the tool.

###  4.4 Specialized Reusable Tools

On top of the modular meta-model (Section 3) and infrastructure tools (Section
4.2), ModMoose offers some specialized tools that answer different recurring
needs in software analysis.

There is a number of reusable tools already implemented, among which:

Model Browser:

Imports or creates new models of given software systems, selects a model, and
browses the model entities. It outputs the selected model. This is the entry
point for working with the environment.

Entity Inspector:

Lists all properties of a selected entity and their values (such as metrics,
tags and others). If the input is a group of entities, it lists all their
common properties. It allows navigating the model when the value of a property
is another entity (_e.g.,_ the methods property of a Class). It outputs any
entity selected for navigation.

Query Browser:

Offers a graphical interface to the Query module (Section 4.2). It works on
the entities found on the bus(es) and can filter and/or navigate from these
entities. It outputs the result of the queries.

Dependency Graph Browser:

Shows as a graph all direct incoming and outgoing dependencies of a group of
entities (see Figure 5, right). Entities are selectable and are written on the
bus for other tools to read them allowing interactive navigation.

Duplication Browser:

Computes and displays duplication in the source code of a group of entities.
In Figure 4, left, the big boxes are software entities (typically methods),
and the small squares are the duplication fragments found in them. Each
duplication fragment has a color to identify it in all its occurrences (in the
various software entities). Entities are selectable and can be output on the
bus(es).

Source Code:

Displays the source code of an input entity. If there are several entities in
input or an entity with no source code (a model for example), does not do
anything. Currently produces no output.

Logger:

Records all entities (individually or in a group) that pass on a bus. It lets
the user come back to a previous stage by selecting the entities that were
produced at that stage. It can export entities in files (txt, csv). It outputs
any selected entity or group of entities.

Figure 4: ModMoose example of specialized tools: Duplication browser. Figure
5: ModMoose example of specialized tools: Call graph browser.

###  4.5 Creating a new Tool

New tools can be added and are planned, for example, a metric dashboard with
standard metrics on the set of entities given in input, or new software maps
such a Distribution Map visualisation [12].

Since tools are specialized, they are easy to develop, particularly with the
help of the visualization engines and the query module. The open architecture
with the buses makes the integration of new tools smooth and easy.

##  5 Conclusion

From literature and our industrial experience, we identified key aspects that
a program comprehension environment must fulfill: a meta-model easily
extensible and adaptable to represent new programming languages or sources of
information; interoperating tools that can be adapted to the comprehension
task at hand.

In this paper, we presented the new architecture of ModMoose, our reverse
engineering environment. It is first based on FamixNG a new way to express
meta-models by composing new entities from a set of traits, each describing
individual properties that are generally encountered in programming languages.
We developed _infrastructure_ tools to manipulate models and an architecture
with specialized end-user tools that interact through information buses.

This is not the end of the road. We will complete our tool suite to respond to
other aspects of reverse engineering and re-engineering. Two research
directions are drawing our attention: (1) how to more easily develop importers
for new programming languages; (2) how to generate new code from the models in
any programming language for which we have a meta-model.

## References

  * [1] N. Anquetil, K. M. de Oliveira, K. D. de Sousa, and M. G. Batista Dias.  Software maintenance seen as a knowledge management issue.  Information Software Technology, 49(5):515–529, 2007.
  * [2] N. Anquetil, A. Etien, G. Andreo, and S. Ducasse.  Decomposing God Classes at Siemens.  In International Conference on Software Maintenance and Evolution (ICSME), 2019.
  * [3] B. Bellay and H. Gall.  An evaluation of reverse engineering tools.  Journal of Software Maintenance: Research and Practice, 1998.
  * [4] A. Bergel.  Agile Visualization.  LULU Press, 2016.
  * [5] S. Bragagnolo, N. Anquetil, S. Ducasse, S. Abderrahmane, and M. Derras.  Analysing microsoft access projects: Building a model in a partially observable domain.  In International Conference on Software and Systems Reuse, ICSR2020, dec 2020.  in submission.
  * [6] H. Bruneliere, J. Cabot, G. Dupé, and F. Madiot.  Modisco: A model driven reverse engineering framework.  Information and Software Technology, 56(8):1012–1032, 2014.
  * [7] J. Delplanque, A. Etien, N. Anquetil, and S. Ducasse.  Recommendations for evolving relational databases.  In International Conference on Advanced Information Systems Engineering (CAiSE), volume 12127 of LNCS, pages 498–514. Springer, 2020\.
  * [8] S. Demeyer, S. Ducasse, and O. Nierstrasz.  Object-Oriented Reengineering Patterns.  Morgan Kaufmann, 2002.
  * [9] S. Demeyer, S. Ducasse, and S. Tichelaar.  Why unified is not universal. UML shortcomings for coping with round-trip engineering.  In B. Rumpe, editor, Proceedings of the International Conference on The Unified Modeling Language (UML’99), volume 1723 of LNCS, pages 630–644, Kaiserslautern, Germany, Oct. 1999. Springer-Verlag.
  * [10] S. Demeyer, S. Ducasse, and S. Tichelaar.  Why unified is not universal. UML shortcomings for coping with round-trip engineering.  In International Conference on The Unified Modeling Language (UML’99), volume 1723, pages 630–644. Springer, 1999.
  * [11] S. Demeyer, S. Tichelaar, and S. Ducasse.  FAMIX 2.1 — The FAMOOS Information Exchange Model.  Technical report, University of Bern, 2001.
  * [12] S. Ducasse, T. Gîrba, and A. Kuhn.  Distribution map.  In Proceedings of 22nd IEEE International Conference on Software Maintenance, ICSM’06, pages 203–212, Los Alamitos CA, 2006. IEEE Computer Society.
  * [13] A. Egyed and N. Medvidovic.  A formal approach to heterogeneous software modeling.  In Fundamental Approaches to Software Engineering, Lecture Notes in Computer Science, pages 178–192. Springer, Berlin, Heidelberg, Mar. 2000.
  * [14] B. Govin, N. Anquetil, A. Etien, S. Ducasse, and A. Monegier Du Sorbier.  Managing an Industrial Software Rearchitecting Project With Source Code Labelling.  In Complex Systems Design & Management conference (CSD&M), Paris, France, Dec. 2017.
  * [15] H. M. Kienle and H. A. Müller.  The tools perspective on software reverse engineering: Requirements, construction, and evaluation.  In Advanced in Computers, volume 79, pages 189–290. Elsevier, 2010\.
  * [16] G. Larcheveque, U. Bhatti, N. Anquetil, and S. Ducasse.  Telescope: A high-level model to build dynamic visualizations.  In International Workshop on Smalltalk Technologies (IWST’15), 2015\.
  * [17] J. Laval, S. Denier, S. Ducasse, and J.-R. Falleri.  Supporting simultaneous versions for software evolution assessment.  Journal of Science of Computer Programming (SCP), 76(12):1177–1193, May 2011.
  * [18] T. Lethbridge, S. Tichelaar, and E. Plödereder.  The dagstuhl middle metamodel: A schema for reverse engineering.  In Electronic Notes in Theoretical Computer Science, volume 94, pages 7–18, 2004.
  * [19] D. M. Lyons, A. M. Bogar, and D. Baird.  Lightweight Multilingual Software Analysis.  Challenges and Opportunities in ICT Research Projects, 2018.
  * [20] P. Mayer.  A taxonomy of cross-language linking mechanisms in open source frameworks.  Computing, 99(7):701–724, July 2017.
  * [21] O. Nierstrasz, S. Ducasse, and T. Gîrba.  The story of Moose: an agile reengineering environment.  In M. Wermelinger and H. Gall, editors, Proceedings of the European Software Engineering Conference, ESEC/FSE’05, pages 1–10, New York NY, 2005. ACM Press.  Invited paper.
  * [22] A. Shatnawi, H. Mili, M. Abdellatif, Y.-G. Guéhéneuc, N. Moha, G. Hecht, G. E. Boussaidi, and J. Privat.  Static Code Analysis of Multilanguage Software Systems, June 2019.  arXiv: 1906.00815.
  * [23] P. Tesone, S. Ducasse, G. Polito, L. Fabresse, and N. Bouraqadi.  A new modular implementation for stateful traits.  Science of Computer Programming, 2020.
  * [24] B. Verhaeghe, A. Etien, N. Anquetil, A. Seriai, L. Deruelle, S. Ducasse, and M. Derras.  GUI migration using MDE from GWT to Angular 6: An industrial case.  In 2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER), 2019.
  * [25] H. Washizaki, Y.-G. Gueheneuc, and F. Khomh.  A Taxonomy for Program Metamodels in Program Reverse Engineering.  In 2016 IEEE International Conference on Software Maintenance and Evolution (ICSME), pages 44–55, Raleigh, NC, USA, Oct. 2016. IEEE.


11institutetext: Leibniz University Hannover, Hannover, Germany
11email<EMAIL_ADDRESS>22institutetext: Ludwig-Maximilian
University, Munich, Germany 33institutetext: Fraunhofer Institute for
Integrated Circuits (IIS), Erlangen, Germany 44institutetext: University of
Freiburg, Freiburg, Germany 55institutetext: Bosch Center for Artificial
Intelligence, Renningen, Germany

Difan Deng

# Efficient Automated Deep Learning for
Time Series Forecasting

Difan Deng 11 Florian Karl 2233
Frank Hutter 4455 Bernd Bischl 2233 Marius Lindauer 11

###### Abstract

Recent years have witnessed tremendously improved efficiency of Automated
Machine Learning (AutoML), especially Automated Deep Learning (AutoDL)
systems, but recent work focuses on tabular, image, or NLP tasks. So far,
little attention has been paid to general AutoDL frameworks for time series
forecasting, despite the enormous success in applying different novel
architectures to such tasks. In this paper, we propose an efficient approach
for the joint optimization of neural architecture and hyperparameters of the
entire data processing pipeline for time series forecasting. In contrast to
common NAS search spaces, we designed a novel neural architecture search space
covering various state-of-the-art architectures, allowing for an efficient
macro-search over different DL approaches. To efficiently search in such a
large configuration space, we use Bayesian optimization with multi-fidelity
optimization. We empirically study several different budget types enabling
efficient multi-fidelity optimization on different forecasting datasets.
Furthermore, we compared our resulting system, dubbed Auto-PyTorch-TS, against
several established baselines and show that it significantly outperforms all
of them across several datasets.

###### Keywords:

AutoML Time Series Forecasting Neural Architecture Search

##  1 Introduction

Time series (TS) forecasting plays a key role in many business and industrial
problems, because an accurate forecasting model is a crucial part of a data-
driven decision-making system. Previous forecasting approaches mainly consider
each individual time series as one task and create a local model [3, 7, 26].
In recent years, with growing dataset size and the ascent of Deep Learning
(DL), research interests have shifted to global forecasting models that are
able to learn information across all time series in a dataset collected from
similar sources [20, 41]. Given the strong ability of DL models to learn
complex feature representations from a large amount of data, there is a
growing trend of applying new DL models to forecasting tasks [38, 46, 50, 57].

Automated machine learning (AutoML) addresses the need of choosing the
architecture and its hyperparameters depending on the task at hand to achieve
peak predictive performance. The former is formalized as neural architecture
search (NAS) [14] and the latter as hyperparameter optimization (HPO) [17].
Several techniques from the fields of NAS and HPO have been successfully
applied to tabular and image benchmarks [15, 18, 33, 62]. Recent works have
also shown that jointly optimizing both problems provides superior models that
better capture the underlying structure of the target task [61, 62].

Although the principle idea of applying AutoML to time series forecasting
models is very natural, there are only few prior approaches addressing this
[32, 37, 43, 52]. In fact, combining state-of-the-art AutoML methods, such as
Bayesian Optimization with multi-fidelity optimization [16, 30, 34, 36], with
state-of-the-art time series forecasting models leads to several challenges we
address in this paper. First, recent approaches for NAS mainly cover cell
search spaces, allowing only for a very limited design space, that does not
support different macro designs [12, 60]. Our goal is to search over a large
variety of different architectures covering state-of-the-art ideas. Second,
evaluating DL models for time series forecasting is fairly expensive and a
machine learning practicioner may not be able to afford many model
evaluations. Multi-fidelity optimization, e.g. [36], was proposed to alleviate
this problem by only allocating a fraction of the resources to evaluated
configurations and promoting the most promising configurations to give them
additional resources. Third, as a consequence of applying multi-fidelity
optimization, we have to choose how different fidelities are defined, i.e.
what kind of budget is used. Examples for such _budget types_ are number of
epochs, dataset size or time series length. Depending on the correlation
between lower and highest fidelity, multi-fidelity optimization can boost the
efficiency of AutoML greatly or even slow it down in the worst case. Since we
are the first to consider multi-fidelity optimization for AutoML on time
series forecasting, we studied the efficiency of different budget types across
many datasets. Fourth, all of these need to be put together; to that effect,
we propose a new open-source package for Automated Deep Learning (AutoDL) for
time series forecasting, dubbed Auto-PyTorch-TS.111The code is available under
https://github.com/automl/Auto-PyTorch. Specifically, our contributions are as
follows:

  1. 1.

We propose the AutoDL framework Auto-PyTorch-TS that is able to jointly
optimize the architecture and the corresponding hyperparameters for a given
dataset for time series forecasting.

  2. 2.

We present a unified architecture configuration space that contains several
state-of-the-art forecasting architectures, allowing for a flexible and
powerful macro-search.

  3. 3.

We provide insights into the configuration space of Auto-PyTorch-TS by
studying the most important design decisions and show that different
architectures are reasonable for different datasets.

  4. 4.

We show that Auto-PyTorch-TS is able to outperform a set of well-known
traditional statistical models and modern deep learning models with an average
relative error reduction of $19\%$ against the best baseline across many
forecasting datasets.

##  2 Related Work

We start by discussing the most closely related work in DL for time series
forecasting, AutoDL, and AutoML for time series forecasting.

###  2.1 Deep Learning based Forecasting

Early work on forecasting focused on building a local model for each
individual series to predict future values, ignoring the correlation between
different series. In contrast, global forecasting models are able to capture
information of multiple time series in a dataset and use this at prediction
time [31]. With growing dataset size and availability of multiple time series
from similar sources, this becomes increasingly appealing over local models.
We will in the following briefly introduce some popular forecasting DL models.

Simple feed-forward MLPs have been used for time series forecasting and
extended to more complex models. For example, the N-BEATS framework [46] is
composed of multiple stacks, each consisting of several blocks. This
architectural choice aligns with the main principle of modern architecture
design: Networks should be designed in a block-wise manner instead of layer-
wise [63].

Additionally, RNNs [9, 23] were proposed to process sequential data and thus
they are directly applicable to time series forecasting [22, 57]. A typical
RNN-based model is the Seq2Seq network [9] that contains an RNN encoder and
decoder. Wen et al. [57] further replaced the Seq2Seq’s RNN decoder with a
multi-head MLP. Flunkert et al. [50] proposed DeepAR that wraps an RNN encoder
as an auto-regressive model and uses it to iteratively generate new sample
points based on sampled trajectories from the last time step.

In contrast, CNNs can extract local, spatially-invariant relationships.
Similarly, time series data may have time-invariant relationships, which makes
CNN-based models suitable for time series tasks, e.g. WaveNet [6, 45] and
Temporal Convolution Networks (TCN) [4]. Similar to RNNs, CNNs could also be
wrapped by an auto-regressive model to recursively forecast future targets [6,
45].

Last but not least, attention mechanisms and transformers have shown superior
performance over RNNs on natural language processing tasks [56] and over CNNs
on computer vision tasks [13]. Transformers and RNNs can also be combined;
e.g. Lim et al. [38] proposed temporal fusion transformers (TFT) that stack a
transformer layer on top of an RNN to combine the best of two worlds.

###  2.2 Automated Deep Learning (AutoDL)

State-of-the-art AutoML approaches include Bayesian Optimization (BO) [18],
Evolutionary Algorithms (EA) [44], reinforcement learning [63] or ensembles
[15]. Most of them consider AutoML system as a black-box optimization problem
that aims at finding the most promising machine learning models and their
optimal corresponding hyperparameters. Neural Architecture Search (NAS), on
the other hand, only contains one search space: its architecture. NAS aims at
finding the optimal architecture for the given task with a fixed set of
hyperparameters. Similar to the traditional approach, the architecture could
be optimized with BO [33, 62], EA [49] or Reinforcement Learning [63] among
others, but there also exist many NAS-specific speedup techniques, such as
one-shot models [59] and zero-cost proxies [1]. In this work we follow the
state-of-the-art approach from Auto-PyTorch [62] and search for both the
optimal architecture and its hyperparameters with BO.

Training a deep neural network requires lots of computational resources.
Multi-fidelity optimization [16, 30, 36] is a common approach to accelerate
AutoML and AutoDL. It prevents the optimizer from investing too many resources
on the poorly performing configurations and allows for spending more on the
most promising ones. However, the correlation between different fidelities
might be weak [60] for DL models, in which case the result on a lower fidelity
will provide little information for those on higher fidelities. Thus, it is an
open question how to properly select the budget type for a given target task,
and researchers often revert to application-specific decisions.

###  2.3 AutoML for Time Series Forecasting

While automatic forecasting has been of interest in the research community in
the past [28], dedicated AutoML approaches for time series forecasting
problems have only been explored recently [21, 32, 35, 42, 51]. Optimization
methods such as random search [55], genetic algorithms [10], monte carlo tree
search and algorithms akin to multi-fidelity optimization [51] have been used
among others. Paldino et al. [47] showed that AutoML frameworks not intended
for time series forecasting originally - in combination with feature
engineering - were not able to significantly outperform simple forecasting
strategies; a similar approach is presented in [10]. As part of a review of
AutoML for forecasting pipelines, Meisenbacher et al. [42] concluded that
there is a need for optimizing the entire pipeline as existing works tend to
only focus on certain parts. We took all of these into account by proposing
Auto-PyTorch-TS as a framework that is specifically designed to optimize over
a flexible and powerful configuration space of forecasting pipelines.

##  3 AutoPyTorch Forecasting

For designing an AutoML system, we need to consider the following components:
optimization targets, configuration space and optimization algorithm. The
high-level workflow of our Auto-PyTorch-TS framework is shown in Figure 1; in
many ways it functions similar to existing state-of-the-art AutoML frameworks
[17, 62]. To better be able to explain unique design choice for time series
forecasting, we first present a formal statement of the forecasting problem
and discuss challenges in evaluating forecasting pipelines before describing
the components in detail.

Training DatasetTask Setup Hyperparameter Optimizer Pipeline Evaluation
EnsembleForecastingTest SeriesConfi-gurationValidationLoss Figure 1: An
overview of Auto-PyTorch-TS. Given a dataset, Auto-PyTorch-TS automatically
prepares the data to fit the requirement of a forecasting pipeline. The AutoML
optimizer will then use the selected budget type to search for desirable
neural architectures and hyperparameters from the pipeline configuration
space. Finally, we create an ensemble out of the most promising pipelines to
do the final forecasting on the test sets.

###  3.1 Problem Definition

A multi-series forecasting task is defined as follows: given a dataset that
contains $N$ series:
$\mathcal{D}=\\{\mathcal{D}_{\mathit{i}}\\}_{\mathit{i}=1}^{N}$ and
$\mathcal{D}_{\mathit{i}}$ represents one series in the dataset:
$\mathcal{D}_{\mathit{i}}=\\{\mathbf{y}_{\mathit{i},1:T_{\mathit{i}}},\mathbf{x}^{(p)}_{\mathit{i},1:T_{\mathit{i}}},\mathbf{x}^{(f)}_{\mathit{i},T_{\mathit{i}}+1:T_{\mathit{i}}+H}\\}$222For
the sake of brevity, we omit the sequence index $i$ in the following part of
this paper unless stated otherwise., where $T$ is the number of time steps
until forecasting starts; $H$ is the forecasting horizon that the model is
required to predict; $\mathbf{y}_{1:T}$, $\mathbf{x}^{(p)}_{1:T}$ and
$\mathbf{x}^{(f)}_{T_{\mathit{i}}+1:T_{\mathit{i}}+H}$ are the sets of
observed past targets, past features and known future features values,
respectively. The task of time series forecasting is to predict the possible
future values with a model trained on $\mathcal{D}$:

$\hat{\mathbf{y}}_{T+1:T+H}=f(\mathbf{y}_{1:T},\mathbf{x}_{1:T+H};\boldsymbol{\theta})$
(1)

where
$\mathbf{x}_{1:T+H}:=[\mathbf{x}^{(p)}_{1:T},\mathbf{x}^{(f)}_{T+1:T+H}]$,
$\boldsymbol{\theta}$ are the model parameters that are optimized with
training losses $\mathcal{L}_{train}$, and $\hat{\mathbf{y}}_{T+1:T+H}$ are
the predicted future target values. Depending on the model type,
$\hat{\mathbf{y}}_{T+1:T+H}$ can be distributions [50] or scalar values [46].
Finally, the forecasting quality is measured by the discrepancy between the
predicted targets $\hat{\mathbf{y}}_{T+1:T+H}$ and the ground truth future
targets $\mathbf{y}_{T+1:T+H}$ according to a defined loss function
$\mathcal{L}$. The most commonly applied metrics include mean absolute scaled
error (MASE), mean absolute percentage error (MAPE), symmetric mean absolute
percentage error (sMAPE) and mean absolute error (MAE) [19, 29, 46].

###  3.2 Evaluating Forecasting Pipelines

We split each sequence into three parts to obtain: a training set
$\mathcal{D}_{\text{train}}=\\{\mathbf{y}_{1:T-H},\mathbf{x}_{1:T+H}\\}$, a
validation set
$\mathcal{D}_{\text{val}}=\\{\mathbf{y}_{T-H+1:T},\mathbf{x}_{T-H+1:T}\\}$ and
a test set
$\mathcal{D}_{\text{test}}=\\{\mathbf{y}_{T+1:T+H},\mathbf{x}_{T+1:T+H}\\}$,
i.e., the tails of each sequences are reserved as $\mathcal{D}_{\text{val}}$.
At each iteration, our AutoML optimizer suggests a new hyperparameter and
architecture configuration $\boldsymbol{\lambda}$, trains it on
$\mathcal{D}_{\text{train}}$ and evaluates it on $\mathcal{D}_{\text{val}}$.

Both in AutoML frameworks [18, 62] and in forecasting frameworks [46],
ensembling of models is a common approach. We combine these two worlds in
Auto-PyTorch-TS by using ensemble selection [8] to construct a weighted
ensemble that is composed of the best $k$ forecasting models from the
previously evaluated configurations $\mathcal{D}_{\text{hist}}$. Finally, we
retrain all ensemble members on
$\mathcal{D}_{\text{val}}\cup\mathcal{D}_{\text{train}}$ before evaluating on
$\mathcal{D}_{\text{test}}$.

###  3.3 Forecasting Pipeline Configuration Space

EncoderDecoderVariable SelectionPast InputKnown Future InputTemporal
FusionForecasting HeadOutput

(a) An overview on the components of our architecture; the dashed lines and
blocks are optional components

Encoder$\mathbf{y}_{1:T}$$\mathbf{x}^{(p)}_{1:T}$Decoder$\mathbf{hx}_{T}$$\mathbf{x}^{(f)}_{T+1:T+H}$$\hat{\mathbf{y}}_{T+1:T+H}$

(b) Non-Auto-Regressive Model

Encoder$\mathbf{y}_{1:T}$$\mathbf{x}^{(p)}_{1:T}$Decoder$\mathbf{hx}_{T}$$\mathbf{y}_{T}$$\mathbf{x}^{(f)}_{T+1}$$\hat{\mathbf{y}}_{T+1}$Decoder$\hat{\mathbf{y}}_{T:T+1}$$\mathbf{x}^{(f)}_{T+1:T+2}$$\hat{\mathbf{y}}_{T+2}$Decoder…$\hat{\mathbf{y}}_{T:T+H-1}$$\mathbf{x}^{(f)}_{T+1:T+H}$$\hat{\mathbf{y}}_{T+H}$

(c) Auto-Regressive Model with Sequential Decoder

EncoderEncoderEncoderEncoder$\mathbf{y}_{1:T{}}$$\mathbf{x}^{(p)}_{1:T+1}$Decoder$\mathbf{hx}_{T}$$\hat{\mathbf{y}}_{T+1}$$\mathbf{y}_{1:T{+1}}$$\mathbf{x}^{(p)}_{1:T+2}$Decoder$\mathbf{hx}_{T+1}$$\hat{\mathbf{y}}_{T+2}$$\mathbf{y}_{1:T{+2}}$$\mathbf{x}^{(p)}_{1:T+3}$Decoder$\mathbf{hx}_{T+2}$$\hat{\mathbf{y}}_{T+3}$$\mathbf{y}_{1:T{+H-1}}$$\mathbf{x}^{(p)}_{1:T+H}$Decoder$\mathbf{hx}_{T+H-1}$$\hat{\mathbf{y}}_{T+H}$…

(d) Auto-Regressive Model with Flat Decoder

Figure 2: Overview of the architectures that can be built by our framework.
(a) shows the main components of our architecture space. (b)-(d) are specific
instances of (a) and its data flow given different architecture properties.

Existing DL packages for time series forecasting [2, 5] follow the typical
structure of traditional machine learning libraries: models are built
individually with their own hyperparameters. Similar to other established
AutoML tools [15, 18, 44], we designed the configuration space of Auto-
PyTorch-TS as a combined algorithm selection and hyperparameter (CASH) problem
[53], i.e., the optimizer first selects the most promising algorithms and then
optimizes for their optimal hyperparameter configurations, with a hierarchy of
design decisions. Deep neural networks, however, are built with stacked blocks
[63] that can be disentangled to fit different requirements [58]. For
instance, Seq2Seq [9], MQ-RNN [57] and DeepAR [50] all contain an RNN as their
encoders. These models naturally share common aspects and cannot be simply
treated as completely different models. To fully utilize the relationships of
different models, we propose a configuration space that includes all the
possible components in a forecasting network.

As shown in Figure 2(b), most existing forecasting architectures can be
decomposed into 3 parts: encoder, decoder and forecasting heads: the encoder
receives the past target values and embeds them into the latent space. The
latent embedding, together with the known future features (if applicable), are
fed to the decoder network; the output of the decoder network is finally
passed to the forecasting head to generate a sequence of scalar values or
distributions, depending on the type of forecasting head. Additionally, the
variable selection, temporal fusion and skip connection layers introduced by
TFT [38] can be seamlessly integrated into our networks and are treated as
optional components.

Encoder | Decoder | auto-regressive | Architecture Class
---|---|---|---
Flat Encoder | MLP | MLP | No | Feed Forward Network
N-BEATS | N-BEATS | No | N-BEATS [46]
Seq. Encoder | RNN/Transformer | RNN/Transformer | Yes | Seq2Seq [9]
No | TFT [38]
MLP | Yes | DeepAR [50]
No | MQ-RNN [57]
TCN | MLP | Yes | DeepAR [50]/WaveNet [45]
No | MQ-CNN [57]

Table 1: An overview of the possible combinations and design decisions of the
models that exists in our configuration space. Only the TFT Network contains
the optional components presented in Figure 2(b).

Table 1 lists all possible choices of encoders, decoders, and their
corresponding architectures in our configuration space. Specifically, we
define two types of network components: sequential encoder (Seq. Encoder) and
flat encoder (Flat Encoder). The former (e.g., RNN, Transformer and TCN)
directly processes sequential data and output a new sequence; the latter
(e.g., MLP and N-BEATS) needs to flatten the sequential data into a 2D matrix
to fuse the information from different time steps. Through this configuration
space, Auto-PyTorch-TS is able to encompass the “convex hull” of several
state-of-the-art global forecasting models and tune them.

As shown in Figure 2, given the properties of encoders, decoders, and models
themselves, we construct three types of architectures that forecast the future
targets in different ways. Non-Auto-Regressive models (Figure 2(b)), including
MLP, MQ-RNN, MQ-CNN, N-BEATS and TFT, forecast the multi-horizontal
predictions within one single step. In contrast, Auto-Regressive models do
only one-step forecasting within each forward pass. The generated forecasting
values are then iteratively fed to the network to forecast the value at the
next time step. All the auto-regressive models are trained with teacher
forcing [22]. Only sequential networks could serve as an encoder in auto-
regressive models, however, we could select both sequential and flat decoders
for auto-regressive models. Sequential decoders are capable of independently
receiving the newly generated predictions. We consider this class of
architectures as a Seq2Seq [9] model: we first feed the past input values to
the encoder to generate its output $\mathbf{hx}$ and then pass $\mathbf{hx}$
to the decoder, as shown in Figure 2(d). Having acquired $\mathbf{hx}$, the
decoder then generates a sequence of predictions with the generated
predictions and known future values by itself. Finally, Auto-Regressive Models
with flat decoders are classified as the family of DeepAR models [50]. As the
decoder could not collect more information as the number of generated samples
increases, we need to feed the generated samples back to the encoder, as shown
in Figure 2(d).

Besides its architectures, hyperparemters also play an important role on the
performance of a deep neural network [61], for the details of other
hyperparameters in our configuration space, we refer to the Appendix.

###  3.4 Hyperparameter Optimization

We optimize the loss on the validation set
$\mathcal{L}_{\mathcal{D}_{\text{val}}}$ with BO [17]. It is known for its
sample efficiency, making it a good approach for expensive black-box
optimization tasks, such as AutoDL for expensive global forecasting DL models.
Specifically, we optimize the hyperparameters with SMAC [25]333We used SMAC3
[39] from https://github.com/automl/SMAC3 that constructs a random forest to
model the loss distribution over the configuration space.

Similar to other AutoML tools [18, 62] for supervised classification, we
utilize multi-fidelity optimization to achieve better any-time performance.
Multi-fidelity optimizers start with the lowest budget and gradually assign
higher budgets to well-performing configurations. Thereby, the choice of what
budget type to use is essential for the efficiency of a multi-fidelity
optimizer. The most popular choices of budget type in DL tasks are the number
of epochs and dataset size. For time series forecasting, we propose the
following four different types of budget:

  * •

Number of Epochs ($\\#$Epochs)

  * •

Series Resolution (Resolution)

  * •

Number of Series ($\\#$Series)

  * •

Number of Samples in each Series ($\\#$SMPs per Ser.)

A higher Resolution indicates an extended sample interval. The sample interval
is computed by the inverse of the fidelity value, e.g., a resolution fidelity
of $0.1$ indicates for each series we take every tenth point: we shrink the
size of the sliding window accordingly to ensure that the lower fidelity
optimizer does not receive more information than the higher fidelity
optimizer. $\\#$Series means that we only sample a fraction of sequences to
train our model. Finally, $\\#$SMPs per Ser. indicates that we decrease the
expected value of the number of samples within each sequence; see Section 3.2
for sample-generation method. Next to these multi-fidelity variants, we also
consider vanilla Bayesian optimization (Vanilla BO) using the maximum of all
these fidelities.

###  3.5 Proxy-Evaluation on Many Time Series

All trained models must query every series to evaluate $\mathcal{L}_{val}$.
However, the number of series could be quite large. Additionally, many
forecasting models (e.g., DeepAR) are cheap to be trained but expensive during
inference time. As a result, rather than training time, inference time is more
likely to become a bottleneck to optimize the hyperparameters on a large
dataset (for instance, with $10$k series or more), where configuration with
lower fidelities would no longer provide the desirable speed-up when using the
full validation set. Thereby, we consider a different evaluation strategy on
large datasets (with more than $1$k series) and lower budgets: we ask the
model to only evaluate a fraction of the validation set (we call this fraction
“proxy validation set”) while the other series are predicted by a dummy
forecaster (which simply repeats the last target value in the training series,
i.e., $\mathbf{y}_{T}$, $H$ times). The size of the proxy validation set is
proportional to the budget allocated to the configuration: maximal budget
indicates that the model needs to evaluate the entire validation set. We set
the minimal number of series in the proxy set to be $1$k to ensure that it
contains enough information from the validation set. The proxy validation set
is generated with a grid to ensure that all the configurations under the same
fidelity are evaluated on the same proxy set.

##  4 Experiments

We evaluate Auto-PyTorch-TS on the established benchmarks of the Monash Time
Series Forecasting Repository [20]444https://forecastingdata.org/. This
repository contains various datasets that come from different domains, which
allows us to assess the robustness of our framework against different data
distributions. Additionally, it records the performance of several models,
including local models [3, 7, 11, 26, 27], global traditional machine learning
models [48, 54], and global DL models [2, 6, 46, 50, 56] on
$\mathcal{D}_{\text{test}}$, see [20] for details. For evaluating Auto-
PyTorch-TS, we will follow the exact same protocol and dataset splits. We
focus our comparison of Auto-PyTorch-TS against two types of baselines: (i)
the overall single best baseline from [20], assuming a user would have the
required expert knowledge and (ii) the best dataset-specific baseline. We note
that the latter is a very strong baseline and a priori it is not known which
baseline would be best for a given dataset; thus we call it the theoretical
oracle baseline. Since the Monash Time Series Forecasting Repository does not
record the standard deviation of each method, we reran those baselines on our
cluster for 5 times. Compared to the repository, our configuration space
includes one more strong class of algorithms, TFT [38], which we added to our
set of baselines to ensure a fair and even harder comparison.

We set up our task following the method described in Section 3.2: HPO is only
executed on $\mathcal{D}_{train/val}$ while $H$ is given by the original
repository. As described in Section 3.2, we create an ensemble with size $20$
that collects multiple models during the course of optimization. When the
search finishes, we refit the ensemble to the union of
$\mathcal{D}_{train/val}$ and evaluate the refitted model on
$\mathcal{D}_{\text{test}}$. Both $\mathcal{L}_{val}$ and $\mathcal{L}_{test}$
are measured with the mean value of MASE [29] across all the series in the
dataset. To leverage available expert knowledge, Auto-PyTorch-TS runs an
initial design with the default configurations of each model in Table 1.
Please note that this initial design will be evaluated on the smallest
available fidelity. All multi-fidelity variants of Auto-PyTorch-TS start with
the cheapest fidelity of $1/9$, use then $1/3$ and end with the highest
fidelity ($1.0$). The runs of Auto-PyTorch-TS are repeated $5$ times with
different random seeds.

We ran all the datasets on a cluster node equipped with 8 Intel Xeon Gold
6254@ 3.10GHz CPU cores and one NVIDIA GTX 2080TI GPU equipped with PyTorch
1.10 and Cuda 11.6. The hyperparameters were optimized with SMAC3 v1.0.1 for
$10$ hours, and then we refit the ensemble on $\mathcal{D}_{train/val}$ and
evaluate it on the test set. All the jobs were finished within $12$ hours.

###  4.1 Time Series Forecasting

| Auto-PyTorch-TS |  |
---|---|---|---
|  $\\#$ Epochs | Resolution |  $\\#$ Series |  $\\#$ SMPs per Ser. | Vanilla BO |  Best dataset- specific baseline |  Overall single best baseline
M3 Yearly | 2.73(0.10) | 2.66(0.05) | 2.76(0.09) |  2.64(0.09) | 2.68(0.08) | 2.77(0.00) | 3.13(0.00)
M3 Quarterly |  1.08(0.01) | 1.10(0.01) | 1.10(0.01) | 1.09(0.02) | 1.12(0.03) | 1.12(0.00) | 1.26(0.00)
M3 Monthly |  0.85(0.01) | 0.89(0.02) | 0.86(0.01) | 0.87(0.04) | 0.86(0.02) | 0.86(0.00) | 0.86(0.00)
M3 Other | 1.90(0.07) | 1.82(0.03) | 1.98(0.13) | 1.92(0.05) | 1.95(0.15) |  1.81(0.00) | 1.85(0.00)
M4 Quarterly | 1.15(0.01) | 1.13(0.01) |  1.13(0.01) | 1.15(0.01) | 1.15(0.02) | 1.16(0.00) | 1.19(0.00)
M4 Monthly | 0.93(0.02) | 0.93(0.02) | 0.93(0.02) |  0.93(0.02) | 0.96(0.02) | 0.95(0.00) | 1.05(0.00)
M4 Weekly | 0.44(0.01) | 0.45(0.02) |  0.43(0.02) | 0.44(0.02) | 0.45(0.01) | 0.48(0.00) | 0.50(0.00)
M4 Daily | 1.14(0.01) | 1.18(0.07) | 1.16(0.06) | 1.14(0.04) | 1.38(0.41) |  1.13(0.02) | 1.16(0.00)
M4 Hourly | 0.86(0.12) | 0.95(0.11) |  0.78(0.07) | 0.85(0.07) | 0.85(0.06) | 1.66(0.00) | 2.66(0.00)
M4 Yearly |  3.05(0.03) | 3.08(0.04) | 3.05(0.01) | 3.09(0.04) | 3.10(0.02) | 3.38(0.00) | 3.44(0.00)
Tourism Quarterly | 1.61(0.03) | 1.57(0.05) | 1.59(0.05) | 1.59(0.02) | 1.55(0.03) |  1.50(0.01) | 1.83(0.00)
Tourism Monthly |  1.42(0.03) | 1.44(0.03) | 1.45(0.04) | 1.47(0.02) | 1.42(0.02) | 1.44(0.02) | 1.75(0.00)
Dominick | 0.51(0.04) | 0.49(0.00) |  0.49(0.01) | 0.49(0.01) | 0.49(0.01) | 0.51(0.00) | 0.72(0.00)
Kdd Cup | 1.20(0.02) | 1.18(0.02) | 1.18(0.03) | 1.18(0.03) | 1.20(0.03) |  1.17(0.01) | 1.39(0.00)
Weather | 0.63(0.08) | 0.58(0.04) | 0.59(0.02) | 0.59(0.06) |  0.57(0.00) | 0.64(0.01) | 0.69(0.00)
NN5 Daily | 0.79(0.01) | 0.80(0.01) | 0.81(0.04) |  0.78(0.01) | 0.79(0.01) | 0.86(0.00) | 0.86(0.00)
NN5 Weekly | 0.76(0.01) | 0.76(0.03) | 0.76(0.01) | 0.77(0.01) |  0.76(0.01) | 0.77(0.01) | 0.87(0.00)
Hospital | 0.76(0.01) | 0.76(0.00) | 0.76(0.00) |  0.75(0.01) | 0.75(0.01) | 0.76(0.00) | 0.77(0.00)
Traffic Weekly | 1.04(0.07) | 1.10(0.03) | 1.04(0.05) | 1.08(0.09) | 1.03(0.07) |  0.99(0.03) | 1.15(0.00)
Electricity Weekly | 0.78(0.04) | 1.06(0.13) | 0.80(0.04) |  0.74(0.07) | 0.85(0.11) | 0.76(0.01) | 0.79(0.00)
Electricity Hourly | 1.52(0.05) | 1.54(0.00) | 1.58(0.08) | 1.54(0.06) |  1.51(0.05) | 1.60(0.02) | 3.69(0.00)
Kaggle Web Traffic Weekly | 0.56(0.01) | 0.56(0.01) |  0.55(0.00) | 0.57(0.01) | 0.59(0.01) | 0.61(0.02) | 0.62(0.00)
Covid Deaths | 5.11(1.60) | 4.54(0.05) |  4.43(0.13) | 4.58(0.30) | 4.53(0.24) | 5.16(0.04) | 5.72(0.00)
Temperature Rain | 0.76(0.05) | 0.75(0.01) | 0.73(0.02) | 0.73(0.03) | 0.71(0.04) |  0.71(0.03) | 1.23(0.00)
$\varnothing$ Rel. Impr Best | 0.82 | 0.83 | 0.81 | 0.81 | 0.82 | 0.86 | 1.0
$\varnothing$ Rel. Impr Oracle | 0.96 | 0.97 | 0.95 | 0.95 | 0.96 | 1.0 | 1.17

Table 2: We compare variants of Auto-PyTorch-TS against the single best
baseline (TBATS) and a theoretically optimal oracle of choosing the correct
baseline for each dataset wrt mean MASE errors on the test sets. We show the
mean and standard deviation for each dataset. The best results are highlighted
in bold-face. We computed the relative improvement wrt the Oracle Baseline on
each dataset and used the geometric average for aggregation over the datasets.

Table 2 shows how different variants of Auto-PyTorch-TS perform against the
two types of baselines across multiple datasets. Even using the theoretical
oracle baseline for comparison, Auto-PyTorch-TS is able to outperform it on 18
out of 24 datasets. On the other 6 datasets, it achieved nearly the same
performance as the baselines. On average, we were able to reduce the MASE by
up to $5\%$ against the oracle and by up to $19\%$ against the single best
baseline, establishing a new robust state-of-the-art overall.

Surprisingly, the forecasting-specific budget types did not perform
significantly better than the number of epochs (the common budget type in
classification). Nevertheless, the optimal choice of budget type varies across
datasets, which aligns with our intuition that on a given dataset the
correlation between lower and higher fidelities may be stronger for certain
budget types than for other types. If we were to construct a theoretically
optimal budget-type selector, which utilizes the best-performing budget type
for a given dataset, we would reduce the relative error by $2\%$ over the
single best (i.e., $\\#$ SMPs per Ser.).

###  4.2 Hyperparameter Importance

Although HPO is often considered as a black-box optimization problem [17], it
is important to shed light on the importance of different hyperparameters to
provide insights into the design choice of DL models and to indicate how to
design the next generation of AutoDL systems.

Here we evaluate the importance of the hyperparameters with a global analysis
based on fANOVA [24], which measures the importance of hyperparameters by the
variance caused by changing one single hyperparameter while marginalizing over
the effect of all other hyperparameters. Results on individual datasets can be
found in appendix.

Figure 3: Hyperparameter importance with fANOVA across all datasets of Table 2

For each of the 10 most important hyperparameters in our configuration space
(of more than 200 dimensions), Figure 3 shows a box plot of the importance
across our datasets. The most important hyperparameters are closely associated
with the training procedure: 3 of them control the optimizer of the neural
network and its learning rate. Additionally, 4 hyperparameters (window_size,
num_batches_per_epoch, batch_size, target_scaler) contribute to the sampler
and data preprocessing, showing the importance of the data fed to the network.
Finally, the fact that two hyperparameters controlling the data distribution
are amongst the most important ones indicates that identifying the correct
potential data distribution might be beneficial to the performance of the
model.

###  4.3 Ablation Study

Figure 4: Validation losses over time with different multi-fidelity
approaches. We compute the area under the curve (AUC) of our approach (PE) and
naive multi-fidelity optimizer (FE) and list them in the figures.

In Section 3.5, we propose to partially evaluate the validation set on larger
datasets to further accelerate the optimization process. To study the
efficiency gain of this approach, we compare evaluation on the full validation
set vs the proxy-evaluation on parts of the validation set. We ran this
ablation study on the largest dataset, namely "Dominick" ($115\,704$ series).

Figure 4 shows the results. It takes much less time for our optimizer (blue)
to finish the first configuration evaluations on the lowest fidelity,
improving efficiency early on and showing the need of efficient validation and
not only training. We note that the final performance does not change
substantially between the different methods. Overall, Auto-PyTorch-TS achieves
the best any-time performance. We note that Auto-PyTorch-TS has not converged
after 10h and will most likely achieve even better performance if provided
with more compute resources. The results on the other datasets show a similar
trend and can be found in the appendix.

##  5 Conclusion and Future Work

In this work, we introduced Auto-PyTorch-TS, an AutoDL framework for the joint
optimization of architecture and hyperparameters of DL models for time series
forecasting tasks. To this end, we propose a new flexible configuration space
encompassing several state-of-the-art forecasting DL models by identifying key
concepts in different model classes and combining them into a single
framework.

Given the flexibility of our configuration space, new developers can easily
adapt their architectures to our framework under the assumption that they can
be formulated as an encoder-decoder-head architecture. Despite recent advances
and competitive results, DL methods have until now not been considered the
undisputed best approach in time series forecasting tasks: Traditional machine
learning approaches and statistical methods have remained quite competitive
[20, 40]. By conducting a large benchmark, we demonstrated, that our proposed
Auto-PyTorch-TS framework is able to outperform current state-of-the-art
methods on a variety of forecasting datasets from different domains and even
improves over a theoretically optimal oracle comprised of the best possible
baseline model for each dataset.

While we were able to show superior performance over existing methods, our
results suggest, that a combination of DL approaches with traditional machine
learning and statistical methods could further improve performance. The
optimal setup for such a framework and how to best utilize these model classes
side by side poses an interesting direction for further research. Our
framework makes use of BO and utilizes multi-fidelity optimization in order to
alleviate the costs incurred by the expensive training of DL models. Our
experiments empirically demonstrate, that the choice of budget type can have
an influence on the quality of the optimization and ultimately performance.

To the best of our knowledge there is currently no research concerning the
choice of fidelity when utilizing multi-fidelity optimization for architecture
search and HPO of DL models; not only for time series forecasting, but other
tasks as well. This provides a great opportunity for future research and could
further improve current state-of-the-art methods already utilizing multi-
fidelity optimization. Additionally, we used our extensive experiments to
examine the importance of hyperparameters in our configuration space and were
able to identify some of the critical choices for the configuration of DL
architectures for time series forecasting. Finally, in contrast to previous
AutoML systems, to the best of our knowledge, time series forecasting is the
first task, where not only efficient training is important but also efficient
validation. Although we showed empirical evidence for the problem and took a
first step in the direction of efficient validation, it remains an open
challenge for future work. Auto-PyTorch-TS can automatically optimize the
hyperparameter configuration for a given task and can be viewed as a benchmark
tool that isolates the influence of hyperparameter configurations of the
model. This makes our framework an asset to the research community as it
enables researchers to conveniently compare their methods to existing DL
models.

## Acknowledgements

Difan Deng and Marius Lindauer acknowledge financial support by the Federal
Ministry for Economic Affairs and Energy of Germany in the project CoyPu under
Grant No. 01MK21007L. Bernd Bischl acknowledges funding by the German Federal
Ministry of Education and Research (BMBF) under Grant No. 01IS18036A. Florian
Karl acknowledges support by the Bavarian Ministry of Economic Affairs,
Regional Development and Energy through the Center for Analytics – Data –
Applications (ADACenter) within the framework of BAYERN DIGITAL II
(20-3410-2-9-8). Frank Hutter acknowledges support by European Research
Council (ERC) Consolidator Grant “Deep Learning 2.0” (grant no. 101045765).

## References

  * [1] Abdelfattah, M.S., Mehrotra, A., Dudziak, Ł., Lane, N.D.: Zero-cost proxies for lightweight NAS. In: ICLR (2021)
  * [2] Alexandrov, A., et al.: Gluonts: Probabilistic and neural time series modeling in python. Journal of Machine Learning Research 21 (2020)
  * [3] Assimakopoulos, V., Nikolopoulos, K.: The theta model: a decomposition approach to forecasting. International journal of forecasting 16(4) (2000)
  * [4] Bai, S., Kolter, J.Z., Koltun, V.: An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv:1803.01271 (2018)
  * [5] Beitner, J.: PyTorch Forecasting: Time series forecasting with PyTorch (2020)
  * [6] Borovykh, A., Bohte, S., Oosterlee, C.W.: Conditional time series forecasting with convolutional neural networks. arXiv:1703.04691 (2017)
  * [7] Box, G.E., Jenkins, G.M., Reinsel, G.C., Ljung, G.M.: Time series analysis: forecasting and control (2015)
  * [8] Caruana, R., Niculescu-Mizil, A., Crew, G., Ksikes, A.: Ensemble selection from libraries of models. In: ICML. (2004)
  * [9] Cho, K., et al.: Learning phrase representations using RNN encoder-decoder for statistical machine translation (2014)
  * [10] Dahl, S.M.J.: TSPO: an autoML approach to time series forecasting. Ph.D. thesis, Universidade NOVA de Lisboa (2020)
  * [11] De Livera, A.M., Hyndman, R.J., Snyder, R.D.: Forecasting time series with complex seasonal patterns using exponential smoothing. Journal of the American statistical association 106(496) (2011)
  * [12] Dong, X., Yang, Y.: NAS-Bench-201: Extending the scope of reproducible neural architecture search. In: ICLR) (2020)
  * [13] Dosovitskiy, A., et al: An image is worth 16x16 words: Transformers for image recognition at scale. In: ICLR (2021)
  * [14] Elsken, T., Metzen, J.H., Hutter, F.: Neural architecture search. In: Automatic Machine Learning: Methods, Systems, Challenges (2019)
  * [15] Erickson, N., et al.: Autogluon-tabular: Robust and accurate automl for structured data. arXiv:2003.06505 (2020)
  * [16] Falkner, S., Klein, A., Hutter, F.: BOHB: robust and efficient hyperparameter optimization at scale. In: ICML (2018)
  * [17] Feurer, M., Hutter, F.: Hyperparameter optimization. In: Automatic Machine Learning: Methods, Systems, Challenges (2019)
  * [18] Feurer, M., Klein, A., Eggensperger, K., Springenberg, J.T., Blum, M., Hutter, F.: Efficient and robust automated machine learning. In: NeurIPS (2015)
  * [19] Flores, B.E.: A pragmatic view of accuracy measurement in forecasting. Omega (1986)
  * [20] Godahewa, R., Bergmeir, C., Webb, G.I., Hyndman, R.J., Montero-Manso, P.: Monash time series forecasting archive. In: NeurIPS Track on Datasets and Benchmarks (2021)
  * [21] Halvari, T., Nurminen, J.K., Mikkonen, T.: Robustness of automl for time series forecasting in sensor networks. In: IFIP Networking Conference (2021)
  * [22] Hewamalage, H., Bergmeir, C., Bandara, K.: Recurrent neural networks for time series forecasting: Current status and future directions. International Journal of Forecasting (2021)
  * [23] Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. (1997)
  * [24] Hutter, F., Hoos, H., Leyton-Brown, K.: An efficient approach for assessing hyperparameter importance. In: ICML (2014)
  * [25] Hutter, F., Hoos, H.H., Leyton-Brown, K.: Sequential model-based optimization for general algorithm configuration. In: Learning and Intelligent Optimization (2011)
  * [26] Hyndman, R., Koehler, A.B., Ord, J.K., Snyder, R.D.: Forecasting with exponential smoothing: the state space approach (2008)
  * [27] Hyndman, R.J., Athanasopoulos, G.: Forecasting: principles and practice (2021)
  * [28] Hyndman, R.J., Khandakar, Y.: Automatic time series forecasting: the forecast package for r. Journal of statistical software 27 (2008)
  * [29] Hyndman, R.J., Koehler, A.B.: Another look at measures of forecast accuracy. International Journal of Forecasting (2006)
  * [30] Jamieson, K.G., Talwalkar, A.: Non-stochastic best arm identification and hyperparameter optimization. In: AISTA (2016)
  * [31] Januschowski, T., et al.: Criteria for classifying forecasting methods. International Journal of Forecasting 36(1) (2020)
  * [32] Javeri, I.Y., Toutiaee, M., Arpinar, I.B., Miller, J.A., Miller, T.W.: Improving neural networks for time-series forecasting using data augmentation and automl. In: BigDataService (2021)
  * [33] Jin, H., Song, Q., Hu, X.: Auto-keras: An efficient neural architecture search system. In: SIGKDD (2019)
  * [34] Klein, A., Tiao, L., Lienart, T., Archambeau, C., Seeger, M.: Model-based asynchronous hyperparameter and neural architecture search (2020)
  * [35] Kurian, J.J., Dix, M., Amihai, I., Ceusters, G., Prabhune, A.: BOAT: A bayesian optimization automl time-series framework for industrial applications. In: BigDataService (2021)
  * [36] Li, L., Jamieson, K.G., DeSalvo, G., Rostamizadeh, A., Talwalkar, A.: Hyperband: A novel bandit-based approach to hyperparameter optimization. J. Mach. Learn. Res. (2017)
  * [37] Li, T., Zhang, J., Bao, K., Liang, Y., Li, Y., Zheng, Y.: Autost: Efficient neural architecture search for spatio-temporal prediction. In: SIGKDD (2020)
  * [38] Lim, B., Arık, S.Ö., Loeff, N., Pfister, T.: Temporal fusion transformers for interpretable multi-horizon time series forecasting. International Journal of Forecasting (2021)
  * [39] Lindauer, M., et al.: SMAC3: A versatile bayesian optimization package for hyperparameter optimization. Journal of Machine Learning Research (2022)
  * [40] Makridakis, S., Spiliotis, E., Assimakopoulos, V.: The m4 competition: Results, findings, conclusion and way forward. International Journal of Forecasting (2018)
  * [41] Makridakis, S., Spiliotis, E., Assimakopoulos, V.: The m4 competition: 100,000 time series and 61 forecasting methods. International Journal of Forecasting (2020)
  * [42] Meisenbacher, S., et al: Review of automated time series forecasting pipelines. arXiv: 2202.01712 (2022)
  * [43] Montero-Manso, P., Athanasopoulos, G., Hyndman, R.J., Talagala, T.S.: Fforma: Feature-based forecast model averaging. International Journal of Forecasting 36(1) (2020)
  * [44] Olson, R.S., Bartley, N., Urbanowicz, R.J., Moore, J.H.: Evaluation of a tree-based pipeline optimization tool for automating data science. In: GECCO (2016)
  * [45] van den Oord, A. et al: Wavenet: A generative model for raw audio. In: ISCA Speech Synthesis Workshop (2016)
  * [46] Oreshkin, B.N., Carpov, D., Chapados, N., Bengio, Y.: N-BEATS: neural basis expansion analysis for interpretable time series forecasting. In: ICLR (2020)
  * [47] Paldino, G.M., De Stefani, J., De Caro, F., Bontempi, G.: Does automl outperform naive forecasting? In: Engineering Proceedings. vol. 5 (2021)
  * [48] Prokhorenkova, L., Gusev, G., Vorobev, A., Dorogush, A.V., Gulin, A.: Catboost: Unbiased boosting with categorical features. In: NeurIPS (2018)
  * [49] Real, E., Aggarwal, A., Huang, Y., Le, Q.V.: Regularized evolution for image classifier architecture search. In: AAAI (2019)
  * [50] Salinas, D., Flunkert, V., Gasthaus, J., Januschowski, T.: Deepar: Probabilistic forecasting with autoregressive recurrent networks. International Journal of Forecasting 36(3) (2020)
  * [51] Shah, S.Y., et al.: AutoAI-ts: Autoai for time series forecasting. In: SIGMOD (2021)
  * [52] Talagala, T.S., Hyndman, R.J., Athanasopoulos, G., et al.: Meta-learning how to forecast time series. Monash Econometrics and Business Statistics Working Papers 6 (2018)
  * [53] Thornton, C., Hutter, F., Hoos, H.H., Leyton-Brown, K.: Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms. In: SIGKDD (2013)
  * [54] Trapero, J.R., Kourentzes, N., Fildes, R.: On the identification of sales forecasting models in the presence of promotions. Journal of the Operational Research Society (2015)
  * [55] van Kuppevelt, D., Meijer, C., Huber, F., van der Ploeg, A., Georgievska, S., van Hees, V.: Mcfly: Automated deep learning on time series. SoftwareX 12 (2020)
  * [56] Vaswani, A., et al.: Attention is all you need. In: NeurIPS (2017)
  * [57] Wen, R., Torkkola, K., Narayanaswamy, B., Madeka, D.: A multi-horizon quantile recurrent forecaster. 31st Conference on NeurIPS, Time Series Workshop (2017)
  * [58] Wu, B., et al.: FBNetV5: Neural architecture search for multiple tasks in one run. arXiv:2111.10007 (2021)
  * [59] Xiao, Y., Qiu, Y., Li, X.: A survey on one-shot neural architecture search. In: IOP Conference Series: Materials Science and Engineering. vol. 750. IOP Publishing (2020)
  * [60] Ying, C., Klein, A., Christiansen, E., Real, E., Murphy, K., Hutter, F.: NAS-bench-101: Towards reproducible neural architecture search. In: ICML (2019)
  * [61] Zela, A., Klein, A., Falkner, S., Hutter, F.: Towards automated deep learning: Efficient joint neural architecture and hyperparameter search. In: ICML 2018 AutoML Workshop (2018)
  * [62] Zimmer, L., Lindauer, M., Hutter, F.: Auto-pytorch tabular: Multi-fidelity metalearning for efficient and robust autodl. IEEE Transactions on Pattern Analysis and Machine Intelligence (2021)
  * [63] Zoph, B., Vasudevan, V., Shlens, J., Le, Q.V.: Learning transferable architectures for scalable image recognition. In: CVPR (2018)