# On Realization of Intelligent Decision-Making in the Real World: A
Foundation Decision Model Perspective

Ying Wen1<EMAIL_ADDRESS>
Ziyu<EMAIL_ADDRESS>
Ming Zhou1<EMAIL_ADDRESS>
Shufang Hou2<EMAIL_ADDRESS>
Zhe Cao1<EMAIL_ADDRESS>
Chenyang Le1<EMAIL_ADDRESS>
Jingxiao Chen1<EMAIL_ADDRESS>
Zheng Tian3<EMAIL_ADDRESS>
Weinan Zhang1<EMAIL_ADDRESS>
Jun Wang2,4<EMAIL_ADDRESS>
1Shanghai Jiao Tong University
2Digital Brain Laboratory
3ShanghaiTech University
4University College London

###### Abstract

Our situated environment is full of uncertainty and highly dynamic, thus
hindering the widespread adoption of machine-led Intelligent Decision-Making
(IDM) in real world scenarios. This means IDM should have the capability of
continuously learning new skills and efficiently generalizing across wider
applications. IDM benefits from any new approaches and theoretical
breakthroughs that exhibit Artificial General Intelligence (AGI) breaking the
barriers between tasks and applications. Recent research has well-examined
neural architecture, Transformer, as a backbone foundation model and its
generalization to various tasks, including computer vision, natural language
processing, and reinforcement learning. We therefore argue that a foundation
decision model (FDM) can be established by formulating various decision-making
tasks as a sequence decoding task using the Transformer architecture; this
would be a promising solution to advance the applications of IDM in more
complex real world tasks. In this paper, we elaborate on how a foundation
decision model improves the efficiency and generalization of IDM. We also
discuss potential applications of a FDM in multi-agent game AI, production
scheduling, and robotics tasks. Finally, through a case study, we demonstrate
our realization of the FDM, DigitalBrain (DB1) with 1.2 billion parameters,
which achieves human-level performance over 453 tasks, including text
generation, images caption, video games playing, robotic control, and
traveling salesman problems. As a foundation decision model, DB1 would be a
baby step towards more autonomous and efficient real world IDM applications.
††Work completed when Z. Wan, M. Zhou, Z. Cao, C. Le, J. Chen are interns at
Digital Brain Laboratory. Correspondence to: Weinan Zhang
<EMAIL_ADDRESS>
Keywords: artificial intelligence, intelligent decision-making, transformer,
foundation decision model

##  1 Introduction

Intelligent decision-making (IDM) is the application of artificial
intelligence (AI) and related technologies to address real world decision-
making tasks. From operating a machine to governing a country, all production
activities of human beings rely on a series of decisions to achieve desired
goals. Humans make decisions depending on knowledge, experience, and intuition
Khatri and Ng (2000). Nevertheless, the capacity of a human mind is limited,
restricting the decision bandwidth for many complex and counter-intuitive
problems. With the advent of digital technologies and information processing
techniques, the complexity and uncertainty of decision problems are increasing
dramatically Peres et al. (2020). These complex decision problems are often
beyond humans’ decision capacity. AI facilitates data analysis with a lens
that surpasses human capacity through its ability to process data and
information, addressing the issue of ‘bounded rationality’ Simon (1990). Based
on rules, data analysis, control theory Kirk (2004), and operations research
(OR) Winston and Goldberg (2004), traditional IDM models improve decision
efficiency by automating part of the decision process in various domains Eom
and Kim (2006), e.g. production scheduling Herrmann (2006), air traffic
control Nolan (2011), disease diagnosis Barnett et al. (1987), enterprise
management Ranjan (2009). However, traditional IDM models are still limited to
solving specific tasks due to low model complexity and requirements for manual
feature design and model selection.

Most of the existing IDM models can only address narrowly defined decision-
making tasks Eom and Kim (2006) to support human decision-making. Given a
fixed, small-scale, white-box scheduling problem, OR algorithms can generate a
logistics scheduling strategy Gudehus and Kotzab (2012). However, OR
algorithms must re-perform the optimization for new scheduling problems
without reusing historical experience. Therefore, it is hard for existing
decision-making models to give an optimal strategy for much more extensive,
dynamic, and complex tasks, introducing higher uncertainty and exponentially
growing solution space, such as urban traffic light control Jensen et al.
(2016) and state grid dispatch. Such complex decision problems are common in
real world applications beyond traditional IDM models’ capabilities, which
indicates applying IDM to help production activities significantly improve
their operating capabilities and value is still challenging.

Figure 1: Paradigm shifts of IDM: from traditional data analysis, control
theory and OR methods, deep learning based perception methods to foundation
model based methods.

Therefore, in the booming era of AI, humans place higher expectations on IDM
to further improve efficiency, reduce costs, and have wider applications in
complex real world production activities Phillips-Wren and Ichalkaranje
(2008). AI has already released its potential to solve prediction tasks to
accelerate automated decisions in business Gupta and Farahat (2020). For
example, an AI algorithm can predict whether you like the product and give an
estimated bidding price for the potential advertising slot Wang et al. (2017).
Based on deep learning LeCun et al. (2015), such IDM is more concerned with
perception problems and very good at recognizing data patterns. Unlike
traditional IDM models, deep perception based IDM models enjoy end-to-end
representation learning and high model complexity, but require large amounts
of data. Understanding data patterns can help eliminate some uncertainties
during the decision process. However, it is not the destination of IDM. A
complete decision cycle includes the perception of the environment and
adjusting the behaviors based on the feedback. Aligned with IDM’s long-term
vision to make successful decisions in the real world, AI aims to study the
intelligent agents that receive perception from the environment and learn to
make adaptive decisions based on feedback to maximize expected utility Russell
(2010). Learning and feedback mechanisms are key features of AI, making it
possible to implement the closed-loop IDM models in the next stage, which
constantly adapt to more dynamic and complex environments, such as robotics
Murphy (2019) and autonomous unmanned systems Chen et al. (2022) in the real
world.

The focus of IDM is now shifting from intelligent perception to higher-level
autonomous decisions, capable of interacting with the real world to learn.
Recent advances in deep reinforcement learning Mnih et al. (2013); Sutton and
Barto (2018) show that IDM surpasses human-level performance in multi-step
complex decision-making tasks with well-defined problem settings, e.g. Go
Silver et al. (2017), StarCraft 2 Vinyals et al. (2019b). But IDM models are
still in the early stage of getting a wide application in the real world
Sutton (2019), as they can only solve some well-defined problems. When
applying IDM models to interact with real world environments, a set of unique
challenges still have not been well addressed, especially efficiency and
generalization issues. The real world is essentially an open environment
composed of various tasks, goals, and agents. In the process of constant
dynamic changes, countless new tasks and goals are continuously generated
Peres et al. (2020). Therefore, in IDM applications, there are many realistic
issues, such as limited data, unbalanced distribution classes, multi-modal
data indexed over space and time, a need to carefully compose different models
in multiple decision stages, and high-cost trial and error for decisions Gupta
and Farahat (2020); Qin et al. (2021). Before we can tackle challenges and
improve the existing IDM models, two questions need to be answered: (1) can we
standardize data inputs/outputs with a unified interface? (2) how to map the
infinite number of real world problems to tractable computational problems?

In the past few years, transformer-based foundation models have been
increasingly used as a unified solution for various downstream tasks,
including natural language processing (NLP) Sutskever et al. (2014) and
computer vision (CV) Krizhevsky et al. (2012). The typical work includes BERT
Kenton and Toutanova (2019), GPT-3 Brown et al. (2020) and Swin transformer
Liu et al. (2021b). Thanks to extremely high model complexity and low labeled
data demand, foundation models have good generalization ability for new tasks
and become the new backbone network for voice, video, and related cross-
modality tasks. Although foundation models are originally designed for
prediction tasks, it is natural to ask if they can serve as an effective
neural architecture for decision-making problems. Following this direction,
recently, foundation models are also emerging in reinforcement learning (RL)
Sutton and Barto (2018) with noticeable effectiveness and generalization
ability, such as decision transformer (DT) Chen et al. (2021) and trajectory
transformer (TT) Janner et al. (2021) and video pre-training (VPT) Baker et
al. (2022), which are important attempts at general decision-making models.

In this wave of foundation model revolution, as illustrated in Figure 1, we
cannot help but wonder if the IDM models can culminate in a single
transformer, the foundation decision model (FDM), which takes the various
decision problems as a sequence decoding tasks using transformer architecture,
to rule all decision-making tasks. FDM uses standard transformer-based
autoregressive decoder for decoding the decisions, which supports multi-
modality, multi-task, large-scale, and general-purpose decision-making tasks.
FDM differs from DT Chen et al. (2021) and TT Janner et al. (2021), which use
well-designed transformer architectures for reinforcement learning problems,
while losing the extension ability for other decision-making tasks. The most
closely related work is DeepMind’s Gato Reed et al. (2022). Gato is a 1.2
billion parameter FDM, in which all data was serialized into a sequence of
tokens. Gato shows that a single transformer can directly handle hundreds of
tasks in robotics, single-agent games, and vision and language. Gato is the
first attempt at FDM, but it is still far from revealing all possibilities of
FDM. In this paper, we discuss the opportunity for using FDM as an IDM
solution in real world applications, which eases the poor sample efficiency
and generalization issues of IDM models to a large extent.

FDM generalizes to more decision problems by strengthening the capability to
encode implicit knowledge and process multi-task Han et al. (2020), multi-
modal Xu et al. (2022); Tsai et al. (2019), and space-time data Wen et al.
(2022b),. With the help of transformers, FDM incorporates data across
modalities, including time series, text, voice, image, video, categorical or
numerical data Bommasani et al. (2021); Bao et al. (2021); Wang et al. (2022),
which enables broad multi-agent applications, such as video games, robotics,
and vehicle routing tasks. Besides, for modeling series and sequence patterns,
FDM has better training and inference efficiency than recurrent neural network
based models, thanks to their parallel computation in training or inference
Kenton and Toutanova (2019). Furthermore, the self-attention mechanism in FDM
can learn the interactions between input tokens, which are inherently
permutation invariant Lin et al. (2017) for processing a sequence of states
and actions, e.g., well-suited for multi-agent tasks Wen et al. (2022a).
Overall, FDM represents the inputs as token sequences followed by self-
attention interactions, which are intrinsically compatible with more
modalities and tasks with good generalization potential.

FDM solves online decision problems in feasible time and cost domains more
efficiently. Firstly, thanks to the large-scale parameters, multi-head
operators, and global aggregation Kenton and Toutanova (2019); Brown et al.
(2020), FDM naturally owns extremely high model complexity and learns cross-
domain knowledge via effective pre-training on large-scale, multi-modal, and
multi-task datasets. Secondly, FDM intrinsically shares the same learning
objective across the offline and online training stages, reducing the
difficulty of continuous learning. Finally, FDM adopts the tuning modes from
the transformer, such as fine-tuning and prompt learning, which further
enhances the learning efficiency. In summary, FDM uses collected offline data
to perform pre-training, which can vastly improve the data efficiency and
speed up the calculating speed of online decision-making tasks.

We believe FDM has great potential as a unified solution by converting all
types of tasks into sequence modeling problems, which is one of the promising
directions toward realizing IDM in the real world by trading off efficiency
and generalization. We also discuss some possibilities of FDM in real world
applications, including game AI, production scheduling, and robotics. To
verify our hypothesis, we develop DigitalBrain (DB1) with around 1.2 billion
parameters. DB1 achieves human-level performance over 453 tasks, including
text, text-image, video games, robotics, and path planning tasks. DB1 is a new
attempt at FDM and has extensions in parameter size, sequence length, and
supported tasks compared to Gato. However, DB1 is still a partial
implementation of general FDM. We indicate the gaps between DB1 and general
FDM in Table 1. We believe DB1 is a further step towards general FDM, which
points out the further improvement directions of FDM.

Table 1: Comparisons between DB1 and general FDM models. As a partial implementation of general FDM, DB1 has 1.2B parameters and 2048 input sequence length, which supports text, image, category, numerical inputs and dialogue, image captioning, video games, robotics, and path planning tasks. DB1 uses offline training and supports fine-tuning for new tasks. | DB1 | General FDM
---|---|---
Task support | 5 | 5+
Data Modality | 4 | 4+
Model parameter | 1.2B | 10B+
Sequence Length | 2048 | 2048+
Training paradigm | Offline | Offline + Online
Task Planning | N/A | Yes
Tuning model | Fine-tuning | Fine-tuning + Prompt

In the rest of this paper, Section 2 focuses on the generalization possibility
of multi-modal, multi-task decision-making tasks via a unified model. Section
3 discusses the efficiency issue of IDM algorithms to solve different types of
decision problems, Section 4 introduces some typical applications, Section 6
demonstrates the potential of the FDM via a case study of the DB1 model,and
Section 6 concludes the paper with a summary.

##  2 FDM Makes Generalizable Decisions

In the real world, it is rare to encounter the exact same situation
repeatedly, making generalization crucial for real world decision-making. FDM
learns generalizable representations for multi-modality and multi-task
decision-making problems, which is the fundamental requirement for IDM models
to adapt to new tasks continuously Seger and Peterson (2013); Kirk et al.
(2021). Furthermore, FDM can perform online adaption after offline pre-
training, further strengthening its generalization ability to implement life-
long decision learning Poquet and De Laat (2021).

###  2.1 Multi-modal Decisions

Initially designed for NLP Kenton and Toutanova (2019), the transformer has
been widely extended to other modality tasks Xu et al. (2022), enabling FDM to
operate across highly distinct domains and modalities. The real world is a
multi-modal environment, which means both observations and behaviors for IDM
applications are multi-modal. For example, a robot needs multi-modal sensors
to perceive the real world environment, e.g., cameras, voice, radar, and maps.
Many deep-learning-based methods have been proposed to improve the multi-modal
perception ability of AI models. MultiModel Kaiser et al. (2017) is trained
jointly on eight distinct speech, image, and text processing tasks, in which
modality-specific encoders were used to process text, images, audio, and
categorical data, while the rest of the network parameters are shared across
tasks. In 2018, the Google AI team proposed ‘one big net for everything’
Schmidhuber (2018), describing a method for the incremental training of an
increasingly general problem solver.

With the development of pre-trained foundation models, FDM are further
equipped with the ability to interact with multi-modal environment. Recently,
research on autonomous IDM in multi-modal virtual environments Gan et al.
(2020) has gradually emerged, especially in visual navigation and object
manipulation. In April 2022, the Google Robotics team released SayCan Ahn et
al. (2022). SayCan robots generate sub-task instructions for a given task
using a natural language pre-trained model and then performs corresponding
actions according to the sub-task to complete the task goal. In such a way,
SayCan robots are able to solve tasks given natural language descriptions
through visual perception in real environments. Gato Reed et al. (2022),
launched by DeepMind in June 2022, unifies multi-step decision-making tasks,
multi-round dialogues, and image-text generation tasks into a transformer-
based autoregressive problem, and has achieved good results in over 600 tasks
, which preliminarily verifies the potential of multi-modal pre-trained
foundation models on agent learning tasks. In July 2022, based on the open-
world game Minecraft Perez-Liebana et al. (2019), NVIDIA’s newly released
MineDojo Fan et al. (2022) provides multi-modal data such as game videos,
Wikipedia, and forum pages, and creates thousands of single-agent tasks at the
same time. Based on the CLIP pre-training model Radford et al. (2021),
MineDojo conducts comparative learning on the video-text pair dataset and
trains a zero-shot prediction model of the correlation between natural
language targets and environmental visual states.

###  2.2 Multi-task Decisions

FDM is an end-to-end multi-task learning transformer framework that
simultaneously learns multiple language, vision, and decision-making tasks.
Multi-task learning (MTL) Vandenhende et al. (2021) aims to leverage useful
information in multiple tasks to help improve the generalization performance
of all the tasks. Existing MTL studies mainly focus on supervised learning
tasks, but very few other tasks such as self-supervised and reinforcement
learning tasks Zhang and Yang (2021). FDM jointly learns multiple tasks, such
as reinforcement learning and OR tasks. Also, FDM combines the self-
supervised, supervised and policy optimization objectives into a single
sequence modeling objective, which largely improves the generalization
performance of FDM on all type of the tasks.

It needs to be emphasized that FDM uses one single multi-task network
architecture, other than a single network with part of shared hidden layers
for all tasks. One single multi-task network means all tasks share the same
input and outputs structure, and use the same learning objectives, which
provides better cross-domain multi-task generalization ability. In previous
work, it is common to use the same policy architecture across tasks with
different learning parameters for each task, which leads to a loss of
generality. To further improve the generalization ability, researchers
attempted to use the exact same policy on a single domain, and have achieved
good performance on Atari57 Badia et al. (2020) and DMLab Espeholt et al.
(2018). Such methods has good generalization ability for similar tasks in the
same domain, but is vulnerable to the tasks from other domains. To tackle the
cross-domain multi-task problems, FDM learns a single network with the same
weights across a diverse set of tasks. Similar choice has been adopted by TD
and TT, which has been proved that large sequence foundation model can deal
with such cross-domain decision-making tasks.

###  2.3 Sim-to-Real Decisions

Many IDM models are trained in simulation environments, but real world
applications additionally require physical-world interactions. Simulations in
virtual environments allow us to train decision models with faster and safer
procedures. However, after an IDM model is trained, it has to been transferred
to the real world, which is known as sim-to-real transfer Zhao et al. (2020).
When the virtual environments have sufficiently simulation accurate, the
transferred policy can work in the real world Salvato et al. (2021).
Unfortunately, the high fidelity simulations needs heavier the computational
cost. In practice, less accurate simulators is often preferred, which brings
gap between training virtual environment and real world applications. It
raises an additional requirement for IDM models to tolerate environmental
changes from the simulator to the physical world.

Thanks to the foundation models’ high learning capacity, FDM can learn as much
from various data as we can, which dramatically improves the transfer ability
to real world environments. Despite several attempts found in the literature
to make sim-to-real transferable controllers, many of which associate the idea
of robustness with that of sim-to-real transferability, the lack of uniformity
of the considered tasks does not allow to determine which solution is more
appropriate in terms of the sim-to-real gap. Previous sim-to-real methods
usually require two successive training phases, which may exhibit low
efficiency Salvato et al. (2021). Besides, if these variations and
disturbances do not correctly represent the simulator’s inaccuracies to
reality, it might result in a sim-to-real transfer failure. In FDM, given a
distribution over tasks, the model learns an adaptive policy that maximizes
the expected reward for a new task from the distribution. Therefore, FDM is a
promising and robust solution for quickly adapting the experience gained in
the real world simulations.

##  3 FDM Learns to Make Decisions Efficiently

With pre-training on offline datasets, FDM avoids data inefficiency issues
during learning and keeps low-latency features during inference. RL and OR are
two mainstream IDM methods for solving different decision-making tasks. On one
hand, deep reinforcement learning (DRL) has achieved many successes in video
games Mnih et al. (2013). Nevertheless, DRL still suffers from the sample
efficiency issue due to the requirement for large-scale, high-quality, cheap
data, which can be easily sampled via simulators online. However, the real
world is not a simulator; large-scale, high-quality data does not come cheap.
Therefore, it is difficult for DRL to learn a good decision policy outside of
the simulator. On the other hand, OR algorithms are widely used for scheduling
and planning decisions. As OR models are inherently sensitive to the
complexity of the problem, it is hard to implement dynamic scheduling and
planning in many real world tasks.

This section discusses how FDM solves the data-inefficiency problems for
reinforcement learning and multi-agent learning tasks. Furthermore, we
illustrate how FDM learns the OR optimization via learning OR models and
converting them from long-cycle decision services to instant decision tools.

###  3.1 Offline Learning from Demonstrations

The FDM is usually trained offline using demonstration data, which is a
typical imitation learning (IL) manner with high sample efficiency. Imitation
learning aims to learn from experts’ demonstrations to replicate optimal
behaviors. IL methods, such as behavior cloning (BC) and inverse RL (IRL),
have been examined for their effectiveness in video games, robotic
simulations, and object manipulation Zheng et al. (2021); Ho and Ermon (2016).
BC replicates expert behavior and works well in environments with fixed
environmental parameters. IRL combines behavior cloning and prediction to
achieve higher data efficiency, achieving good performance in some dynamic
environments. Despite the above successful examples, it is still challenging
for ILs to learn diverse behaviors and leverage sub-optimal demonstrations or
even follow language/voice instructions to perform the imitation Hua et al.
(2021). Apart from the advantages inherited from IL, FDM can naturally handle
multi-modal data and multi-task learning, and combine behavior cloning and
prediction to address these issues. Besides, FDM bridges offline imitation and
online learning, making it possible to learn new tasks consistently and
efficiently.

###  3.2 Muli-agent Decisions

FDM uses a sequential update scheme for multi-agent decisions, effectively
transforming multi-agent joint policy optimization into a sequential modeling
problem. Dealing with non-stationary movement is one of the most critical
issues in multi-agent decision learning research, which makes it problematic
to apply single-agent decision models directly in the multi-agent context. FDM
avoids such problems by converting the agents’ simultaneous decision-making to
an agent-by-agent sequential decision-making process Wen et al. (2022a). In
MARL, unlike single-agent cases, due to the existence of agents’ interactions,
the result of these actions is not standalone but usually depends on others’
choices Gronauer and Diepold (2022). To illustrate the effects of multi-agent
interactions, let us consider a motivating example of controlling an adaptive
robot car to go through an intersection. At each step, the robot car can move
around by steering, throttling, or braking. The goal is to pass through the
intersection safely and reach the destination. In addition to detecting
environments, such as positions, traffic lights, and lane markings, the robot
car also needs to be aware of other cars, such as human cars or other adaptive
cars. As there are multiple adaptive agents in a shared environment, they
improve their policies simultaneously. Therefore, from a single agent’s
perspective, the environment it perceives is non-stationary. Sequential update
scheme has been used in multi-agent decision transformer (MADT) Meng et al.
(2021) and multi-agent transformer (MAT) Wen et al. (2022a), ensuring the
efficient and stable improvement during multi-agent learning. Based on multi-
agent advantage decomposition theorem Kuba et al. (2021), FDM owns the joint
performance improvement guarantee and learns an optimal joint policy
efficiently.

###  3.3 Learning to Optimization

FDM treats a series of white-box OR optimization problems as sequence modeling
problems, which can make highly efficient inferences once it learns a good
policy. The OR optimization algorithms obtain the optimal solution under
certain constraints based on an accurate description and characterization of
the specific problems Winston and Goldberg (2004). OR algorithms do not
require a large amount of data, and the results have good interpretability.
Therefore, OR algorithms have been widely used in planning, scheduling, and
coordination problems. However, the computation complexity of traditional OR
algorithms grows exponentially with the problem size. While real world
applications usually have many uncertainties and large problem sizes, it is
often too complex for traditional discrete optimization algorithms to give an
optimal solution in a feasible time. FDM is a data-driven approach that can
address the efficiency problem of OR algorithms. Once it learns a good
approximate policy for OR problems, FDM can generate the decisions with one-
pass forward inference. It means that for any learned discrete optimization
problem, FDM can eliminate the curses of computation complexity and enjoy
linear time complexity.

##  4 Real-world Applications

We discussed the advantages of FDM in previous sections. To further clarify
the effectiveness of FDM in real world IDM applications, in this section, we
list four decision problems suitable for DBM to solve.

###  4.1 Game AI

With good multi-modal and multi-task performance, FDM can improve the
generalization ability of game AI. The deep RL (DRL) methods have made
significant progress in game AI, ranging from the DQN agent Hausknecht and
Stone (2015), AlphaGo Silver et al. (2016), Libratus Brown et al. (2017), and
OpenAI Five Berner et al. (2019) to AlphaStar Vinyals et al. (2019a), in which
learning DRL agents defeat professional human players. It is a big step for
IDM, indicating that DRL can solve very complex games Yin et al. (2021).
However, these DRL agents are usually designed for specific games. For
example, AlphaGo Zero Silver et al. (2017), which utilizes Monte Carlo tree
search, self-play, and deep learning to defeat dozens of professional go
players, represents powerful techniques for larger perfect information games.
Using self-play, deep reinforcement learning, and continuous transfer via
surgery, OpenAI Five became the first AI to beat the world champions at an
esports game, displaying practical techniques for games of complex, imperfect
information. On the contrary, the real world is full of different games, and
it is almost impossible for DRL agents to find a universe policy purely via
self-play, deep learning, and reinforcement learning. We argue that previous
training frameworks, especially self-play with distributed learning, cannot
handle these kinds of situations. This is because a two-player asymmetric game
has very different strategies for each side, and self-play-based mechanisms
may not work well. Via offline training, FDM can distill the experts’
knowledge across various single-task expert agents and provide a unified
solution for game AIs.

###  4.2 Production Scheduling

FDM significantly improves the problem-solving efficiency for production
scheduling, making it possible to output a feasible solution for real world
tasks instantly. Scheduling is a

Figure 2: Training pipeline of DB1. We distribute DB1 to multiple GPU nodes
and devices to perform data-parallel training. A shared data storage is
equipped with caching mechanism to speed up the data loading, and a
distributed strategy for data saving. The sampled training data will be
discretized with 4 tokenizers before feeding to the model.

highly complex factor in the management and control of production systems,
which is crucial to improving the rapid response and global optimization
capabilities of intelligent manufacturing systems. Although numerous studies
have investigated and even proposed various scheduling models and methods,
some challenges concerning research in the field remain a question for the
future development trend. At present, information and intelligent technologies
are bringing about changes and innovations in products and production
organizations, which increases the complexity of manufacturing system
management and control Jiang et al. (2022). As the production data accumulates
exponentially, data-driven IDM models provide service scheduling in a
sustainable manufacturing environment that synergistically optimizes
production efficiency and energy consumption Waubert de Puiseau et al. (2022).

###  4.3 Robotics

Due to the high complexity of the model, FDM can learn from diverse datasets,
minimizing the sim-to-real gap and improving the robustness of the robotics in
real world applications. Having robots with the full range of physical
capabilities that a human has is a long, long way off-arguably even more
remote than having AI systems with the full range of intellectual
capabilities. In the physical world, an embodied agent can face many changing
factors, such as physical parameters, action spaces, tasks, visual appearances
of the scenes, geometry, topology of the objects, etc. Moreover, many critical
real world tasks need generalizable policy learning, e.g., visual navigation,
object manipulation, and autonomous driving Bonin-Font et al. (2008).
Therefore, learning generalizable policies is crucial to developing
intelligent embodied agents in the real world. Most current AI robotics models
have no real world experience due to the sample efficiency issue and sim-to-
real gaps, which restricts the capabilities of IDM models. Generalization is
also essential when robots interact with the physical world. The key stumbling
block is collecting the correct data and effectively learning from the data
Hua et al. (2021). We argue that this is why we need an IDM model that can
adapt to new embodiments and learn new tasks with fewer data Singh et al.
(2021). FDM can meet the generalization requirements of both embodiments and
the physical world. FDM learns generalizable policies in the physical world
based on synergistic efforts across vision, learning, and robotics fields.

##  5 Case Study: the DigitalBrain Model

As discussed in the above sections, FDM is a promising solution for real world
IDM applications. Gato is the first attempt of FDM, which demonstrates the
potential of FDM for text, image-text, video games, robotics tasks. We trained
the DigitalBrain model, noted as DB1, a reproduction of Gato with minor
modifications and additional operation research tasks. In the following
sections, we introduce the details of DB1, including the model design, dataset
collection and processing, training schemes, and empirical results.

###  5.1 Model Design

Model Structure. As shown in Figure 2, we train a FDM model, DigitalBrain
(DB1) with around 1.2B parameters. DB1 uses TransfomerXL Dai et al. (2019) as
backbone which is a decoder-only model contains masked multi-head attention
and feed-forward network with residual connections. Compared to GPT Brown et
al. (2020); Radford et al. (2019), TransformerXL, located on the far left of
Figure 2, uses a relative position encoding scheme instead of an absolute one,
and it introduces a memory mechanism which improves computation efficiency
during inference and theoretically increases DB1’s sequence length by caching
and propagating each layer’s hidden state. Although DB1 only uses the memory
mechanism during the evaluation phase, we find it to be more efficient for
sampling long sequences.

Loss Function. We use a classical language modeling loss in the setting of
teacher forcing Williams and Zipser (1989), i.e., we only predict the next
token during training. For a batch of $N$ instances of sequence length $L$,
the loss is computed as the following equation with a given model parameter
$\theta$:

$\mathcal{L}(\theta)=-\sum\limits_{b=0}^{N-1}\sum\limits_{l=0}^{L-1}m(b,l)\log
p_{\theta}(s_{l}^{(b)}|s_{0}^{(b)},s_{1}^{(b)},\ldots,s_{l-1}^{(b)}),$ (1)

where $s_{l}^{(b)}$ denotes the $l$-th token in the $b$-th sequence of a
batch, $m(b,l)$ denotes a binary loss mask on each token. We set $m(b,l)=1$
when $s_{l}^{(b)}$ is a token of an action or text, namely we compute loss for
all text tokens and action tokens in RL tasks, which is equivalent to Gato.

###  5.2 Dataset Collection and Processing

We introduce three kinds of tasks in our case study. Specifically, they are
simulated control, vision & language and traveling salesman problem (TSP)
tasks. For each category, multiple heterogeneous sub-task sets are considered
to satisfy the requirements of multi-modality learning. As for the data
collection, we should ensure that the dataset should be expert, as the loss
function implies the training scheme of DB1 is supervised learning. To obtain
high-quality datasets, the experts on the simulated control tasks are
implemented as well-trained reinforcement learning algorithms; the experts on
vision and language tasks are humans; and the experts on TSP tasks are well-
designed solvers. As for data processing, we follow the same principle as
Gato, i.e., quantify and discretize raw data into integers. We provide more
details as follows.

Table 2: Datasets of simulated control and TSP task suites. For each task suite, we summarize the number of tasks, the number of collected trajectories, and the number of tokens. To avoid overfitting to a large dataset, the sample weights of task suites are well-tuned instead of being proportional to the dataset size. Task Suite | Tasks | Trajectories | Tokens | Sample weights
---|---|---|---|---
DeepMind Lab | 411 | 3M | 171B | 13%
ALE Atari | 52 | 32K | 1.4B | 13%
Sokoban | 1 | 910 | 2.8M | 1.7%
BabyAI | 55 | 5.5M | 33.5B | 13%
DM Control Suite | 25 | 675K | 20.2B | 5.2%
Meta-World | 42 | 88.2K | 1.9B | 11.7%
Procgen Benchmark | 16 | 1.6M | 7.2B | 6.5%
Modular RL | 38 | 843.6K | 13.9B | 10.4%
TSP | 230 | 2.3M | 108.5B | 11.7%

####  5.2.1 Simulated Control Tasks

The simulated control tasks consist of various video games and robotics
control task sets, which have been treated as common benchmarking environments
for reinforcement learning. The multi-modality property in these tasks is not
only reflected in the task types but also in the state/action space
definition.

  * •

DeepMind Lab Beattie et al. (2016) is a 3D learning environment developed for
the research and development of artificial general intelligence and machine
learning systems. It provides a suite of challenging 3D navigation and puzzle-
solving tasks for learning agents. We split the existing levels in DeepMind
Lab into 6 task sets and train a v-trace agent Schmitt et al. (2020) on each
task set for data collection, 65,000 episodes per level, for a total of 3M
episodes.

  * •

Procgen Cobbe et al. (2019) consists of 16 procedurally generated game-like
environments for evaluating sample efficiency and generalization in
reinforcement learning. The data is collected with a v-trace or r2d2 agent
Schmitt et al. (2020) in each environment. All the distribution modes are set
to hard, except for mazes and heists which are easy. We collect 100,000
episodes for each environment, for a total of 1.6M episodes.

  * •

Sokoban Racanière et al. (2017) is a planning problem in which the agent has
to push boxes to target locations. Since some of the moves are irreversible,
mistakes can make the puzzle unsolvable. Planning ahead of time is therefore
necessary to succeed on this puzzle. We take advantage of the reversed play
processes when generating the maps. We use a breadth-first search in the
reversed play processes and extract the solutions from it as the expert data.
So this expert data is optimal in most cases.

  * •

BabyAI Chevalier-Boisvert et al. (2018) is a grid-world environment whose
levels consist of instruction-following tasks described by a synthetic
language. We generate data for these levels with the built-in BabyAI bot. The
bot has access to extra information for executing optimal solutions. See
Appendix C of Chevalier et al. Chevalier-Boisvert et al. (2018) for more
details about the bot. We collect 100,000 episodes for each level, for a total
of 5.5M episodes.

  * •

Modular RL Huang et al. (2020) is a collection of MuJoCo Todorov et al. (2012)
based continuous control environments, composed of variants of the OpenAI Gym
Brockman et al. (2016). Each variant is a morphological modification of the
original body: the set of morphologies is generated by enumerating all
possible subsets of limbs and keeping only those sets that a) contain the
torso and b) still form a connected graph. This results in a set of variants
with different input and output sizes, as well as different dynamics from the
original morphologies. We use the qpos and qval as observations, just like in
the OpenAI Gym. And we train a D4PG agent Barth-Maron et al. (2018) on each
variant. 843.6K episodes are collected for this task suite.

  * •

Meta-World Yu et al. (2020) is a suite of environments for benchmarking meta-
reinforcement learning and multi-task learning. We collect data from all
training and test tasks in the MT1 mode by training an APPO agent Berner et
al. (2019) from sample-factory repository Petrenko et al. (2020). For this
task suite, we collect 88.2K episodes in total.

  * •

The DeepMind Control Suite Tunyasuvunakool et al. (2020) is a set of physics-
based environments. For each task in the control suite, we collect two
disjoint sets of data, one using only state features and the other using only
pixels. We use a D4PG agent Barth-Maron et al. (2018) to collect data from
tasks with state features and an APPO agent Berner et al. (2019) to collect
data using pixels. For this task suite, we collect 675K episodes in total.

####  5.2.2 Vision and Language Tasks

DB1 is trained on a large collection of English language resources from web
pages. Moreover, we include vision-language tasks such as Microsoft COCO
CaptionChen et al. (2015).

####  5.2.3 TSP Tasks

To investigate DB1 with capabilities of handling the optimization in
operations research, we select the traveling salesman problem (TSP) Gavish and
Graves (1978), one of the best-known combinatorial optimization problems as a
representative task. Generally, it is modeled as a sequential problem (i.e.,
the coordination and neighbor information as states, and node choice as
decision making), and many reinforcement learning methods have been developed
as its solutions. We construct this dataset with 2.3M trajectories from 200
tasks of different scales. Specifically, there are 200 randomly generated
tasks in which 100 or 200 nodes are uniformly sampled from a 2D plane.
Additionally, there are 30 city-walking tasks, each consisting of 200 Chinese
cities. The expert on data collection in TSP is from LKH Lin and Kernighan
(1973), a heuristic algorithm. As the original action space is large in the
hundreds, we further narrow the action space of DB1 by using the expert’s
closest neighbor in each decision. For the state representation, we encode the
neighbor information with a pre-trained graph convolutional network (GCN) Kipf
and Welling (2017).

We list the sample weights for the TSP tasks and the simulated control tasks
on Table 2, which sum up to 87% in total. For the rest of the data, we sample
8.7% from the language corpus and 5.1% from COCO Caption Chen et al. (2015).

####  5.2.4 Tokenization for Multi-modal Data

As so many multi-modal/heterogeneous datasets and tasks are introduced in
training, we need to convert these different data types into a unified
representation. Thus, we implemented four kinds of tokenizers for different
data types, which are introduced as follows.

  * •

Text. We first train a tokenizer on the collected corpora with a fixed
vocabulary size of 32k. For the OpenWebText dataset, we tokenize them
beforehand and then build a sample index based on the sequence length during
training.

  * •

Image. We first divide the image input into a series of 16-by-16 patches, and
then we do normalization inside each patch. Then, without tokenization, each
patch will be mapped into the embedding space of our model. We utilize a
ResNet-v2 block, similar to Gato, to extract visual attributes. Additionally,
we substitute GroupNorm with 32 groups for BatchNorm. We also employ the patch
positional encoding specified in Gato to notify the model about the global
position of a patch within the picture from which it was extracted. By
normalizing the patch’s pixel intervals by the picture resolution, the
relative row and column intervals of the patch are first determined. After
being quantized into a vocabulary size of 128, the row and column normalized
intervals are used to index a row and column table of learnable position
encodings. Whether we are training or evaluating the model determines how the
quantized row and column intervals are transformed into indices: when
training, a random index is picked uniformly from the quantized interval,
while during evaluation, we deterministically take the (rounded) mean of the
interval. After retrieving row and column position encoding from the embedding
table, it is added to the vision embedding created by a ResNet-v2 block.

  * •

Discrete data. For discrete values, we tokenize them into [0,1024) overlapping
our text vocabs.

  * •

Continuous data. For continuous values, we tokenize them into [32000, 33024).
And we also introduce a special token $\varphi$ as a split between each
$s_{t}$ and $a_{t}$ with token ID 33204 as $(s_{t},\varphi,a_{t})$ .

For an arbitrary environment, we will categorize its types into text, image,
and discrete and continuous values. For a nested structure, we will categorize
every underlying attribute. Then we concatenate each observation within a
timestep according to a fixed order. First, we put values of different types
in the order of text, vision and tensors. Vision patch tokens are in raster
order. Tensor tokens are in row-major order. For nested structured input with
each input type, we put them lexicographically by their keys. In order to tell
the model which part is observation and which part is action, we additionally
use a specific reinforcement learning local positional encoding, as described
in Gato. We employ an embedding specifically designed for actions of each
action token. The positional encoding of the observation series is determined
by their corresponding sequence order within a timestep.

####  5.2.5 Dataset Building and Cache

The preprocessing process of reinforcement data is similar to the one used in
Trajectory Transformer Janner et al. (2021). We operate on separate
trajectories split by environmental terminal signals, then episodic returns,
path lengths, and other statistical properties are calculated. We build our
index by enumerating a 3-tuple of path index, start index, end index on a
sliding window that contains the minimal timestep we can get tokens of at
least the sequence length set for the model.

It is usually impossible to load all of the data into memory at once because
there is so much of it. In order to load the indices and processed data as
needed, we save them on disk for each dataset.

Table 3: Hyperparameters of DB1. Hyperparameter | Value/Choice
---|---
Transformer blocks | 24
Attention heads | 16
Layer width | 2048
Shared embedding | True
Layer Normalization | PostNorm
FeedForward size before activation | 8192
Activation Function | GeGLU
Dropout Probability | 0.1

###  5.3 Training Scheme

Hyperparameters. As shown in Table 3, DB1 consists of 16 transformer blocks
with 16 heads, 2048 layer width and shared embedding. We use dropout
probability of $0.1$, PreNorm Xiong et al. (2020) for layer normalization and
GeGLU Shazeer (2020) as activation function.

Figure 3: Selected results of vision and language tasks. Figure 4: Performance
of DB1 on simulated control tasks. The performance of a single task is plotted
in each row organized by domain. Numbers on the $x-$axis reflect a certain
percentage of expert score, with $0$ denoting the performance of a random
agent. Each dot on the figure represents a certain task. The diamond means the
mean score of each domain.

Optimizer. We use the AdamW optimizer Loshchilov and Hutter (2017) with a
linear warm-up and cosine schedule decay. The linear warmup lasts for 15000
steps, starting with zero learning rate and ending at a maximum learning rate
of 5e-5. This learning rate is then cosine decayed by a factor 20x over
240,000 steps. The AdamW optimizer has parameters
$\beta_{1}=0.9,\beta_{2}=0.95$, $\epsilon=$1e-8. We use a batch size of 512
and a sequence length of 1024 tokens.

Prompt Learning in Control Tasks. DB1 is trained to complete multiple control
tasks by modeling them as different sequences. However, for some tasks that
share the same observation and action specifications, more information is
needed for the model to distinguish them. Compared to one-hot identifiers, DB1
is trained with prompt conditions Liu et al. (2021a). During training, we
sample a batch of 512 sequences from different task domains with a fixed
sample weight defined before training. We use a prompt scheme similar to the
one mentioned in Gato Reed et al. (2022): 25% of samples in a batch will be
prepended with a prompt sequence. For each prompt sequence, half of which come
from the end of an episode for goal conditioning, and the other is a
subsequence uniformly sampled from an episode. We use a fixed length of prompt
length, set to about half the length of the model’s input sequence. During
evaluation, we select a successful demonstration as prompt, i.e., episodes
with top 10% returns, and feed the first $k$ timesteps whose length after
tokenization is just equal to or more than the model’s input sequence.

###  5.4 Evaluation Results

We evaluate DB1’s performance after 250K training iterations. 111Code is
available at https://github.com/Shanghai-Digital-Brain-Laboratory/BDM-DB1 For
each simulated control and TSP task, we normalize its demonstration results to
the range from 0 to 1. Formally, we normalize the results with a score of
random policy $R_{\min}$ and expert score $R_{E}$ and the rule is formulated
as: $score=\frac{1}{N}\sum^{N}_{i=1}(r_{i}-R_{\min})/(R_{E}-R_{\min})$. To
eliminate the influence of randomness, we repeat $N=10$ evaluations for each
task ($r_{i}$ in the equation the result of $i$-th evaluation) and report the
average normalized score as DB1’s performance.

The results of the control and TSP tasks are provided in Figure 4, which
presents the performance distribution of each task set. Statistically, DB1
outperforms 50% of the expert scores on 666 tasks out of 870 tasks. In BabyAI
Chevalier-Boisvert et al. (2018), DB1 has an average performance of about 85%
of the expert score on all 55 tasks. DB1 achieves about 96% of the expert
score on all TSP tasks, which we think may be because for combinatorial
optimization problems like TSP, it is relatively easier to achieve a fairly
good result than an near-optimal one. The performance of DM Control Suite
Tunyasuvunakool et al. (2020) and Atari Bellemare et al. (2013) is not as good
as the others, probably because the collected data is not good enough for
supervised learning only.

For the evaluation of image-caption, we randomly sample images from COCO
Caption Dataset Chen et al. (2015) and show the results without cherry-
picking. Figure 3 presents the selected results of text-generation and image-
captioning tasks. As we can learn from the results, DB1 has learned knowledge
well from the training corpus and learns the connections between vision and
language modalities. The caption sentences show that DB1 can figure out what
is the main element in an image. Although there are some mistakes in the
details of numbers and colors, we believe a much larger dataset can resolve
them.

##  6 Conclusion

FDM is a basic but effective model for various multi-modal, multi-task
decision-making problems, including text, vision, reinforcement learning, and
OR tasks. Inheriting the characters from foundation models, FDM is born with
good generalization and efficient learning abilities. According to the scaling
law, the performance of FDM will increase with the scale of parameters and
data, further strengthening the cross-task learning and generalization
abilities of FDM. For future work, we will build a more autonomous decision-
making model and integrate modern IDM models into real-world environments by
continuously training, fine-tuning, and scaling up the FDM.

## References

  * Ahn et al. (2022) Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, et al.  Do as i can, not as i say: Grounding language in robotic affordances.  _arXiv preprint arXiv:2204.01691_ , 2022.
  * Badia et al. (2020) Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell.  Agent57: Outperforming the atari human benchmark.  In _International Conference on Machine Learning_ , pages 507–517. PMLR, 2020.
  * Baker et al. (2022) Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune.  Video pretraining (vpt): Learning to act by watching unlabeled online videos.  _arXiv preprint arXiv:2206.11795_ , 2022.
  * Bao et al. (2021) Hangbo Bao, Li Dong, and Furu Wei.  Beit: Bert pre-training of image transformers.  _arXiv preprint arXiv:2106.08254_ , 2021.
  * Barnett et al. (1987) G Octo Barnett, James J Cimino, Jon A Hupp, and Edward P Hoffer.  Dxplain: an evolving diagnostic decision-support system.  _Jama_ , 258(1):67–74, 1987.
  * Barth-Maron et al. (2018) Gabriel Barth-Maron, Matthew W Hoffman, David Budden, Will Dabney, Dan Horgan, Dhruva Tb, Alistair Muldal, Nicolas Heess, and Timothy Lillicrap.  Distributed distributional deterministic policy gradients.  _arXiv preprint arXiv:1804.08617_ , 2018.
  * Beattie et al. (2016) Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Víctor Valdés, Amir Sadik, et al.  Deepmind lab.  _arXiv preprint arXiv:1612.03801_ , 2016.
  * Bellemare et al. (2013) Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling.  The arcade learning environment: An evaluation platform for general agents.  _Journal of Artificial Intelligence Research_ , 47:253–279, 2013.
  * Berner et al. (2019) Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al.  Dota 2 with large scale deep reinforcement learning.  _arXiv preprint arXiv:1912.06680_ , 2019.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ , 2021.
  * Bonin-Font et al. (2008) Francisco Bonin-Font, Alberto Ortiz, and Gabriel Oliver.  Visual navigation for mobile robots: A survey.  _Journal of intelligent and robotic systems_ , 53(3):263–296, 2008.
  * Brockman et al. (2016) Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba.  Openai gym.  _arXiv preprint arXiv:1606.01540_ , 2016.
  * Brown et al. (2017) Noam Brown, Tuomas Sandholm, and Strategic Machine.  Libratus: The superhuman ai for no-limit poker.  In _IJCAI_ , pages 5226–5228, 2017.
  * Brown et al. (2020) Tom Brown et al.  Language models are few-shot learners.  _NeurIPS_ , 2020.
  * Chen et al. (2022) Jie Chen, Jian Sun, and Gang Wang.  From unmanned systems to autonomous intelligent systems.  _Engineering_ , 12:16=19, 2022.
  * Chen et al. (2021) Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch.  Decision transformer: Reinforcement learning via sequence modeling.  _Advances in neural information processing systems_ , 34:15084–15097, 2021.
  * Chen et al. (2015) Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco captions: Data collection and evaluation server.  _arXiv preprint arXiv:1504.00325_ , 2015.
  * Chevalier-Boisvert et al. (2018) Maxime Chevalier-Boisvert, Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio.  Babyai: A platform to study the sample efficiency of grounded language learning.  _arXiv preprint arXiv:1810.08272_ , 2018.
  * Cobbe et al. (2019) Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman.  Leveraging procedural generation to benchmark reinforcement learning.  _arXiv preprint arXiv:1912.01588_ , 2019.
  * Dai et al. (2019) Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov.  Transformer-xl: Attentive language models beyond a fixed-length context.  _arXiv preprint arXiv:1901.02860_ , 2019.
  * Eom and Kim (2006) Sean Eom and E Kim.  A survey of decision support system applications (1995–2001).  _Journal of the Operational Research Society_ , 57(11):1264–1278, 2006.
  * Espeholt et al. (2018) Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al.  Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures.  In _International conference on machine learning_ , pages 1407–1416. PMLR, 2018.
  * Fan et al. (2022) Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, and Anima Anandkumar.  Minedojo: Building open-ended embodied agents with internet-scale knowledge.  _arXiv preprint arXiv:2206.08853_ , 2022.
  * Gan et al. (2020) Chuang Gan, Yiwei Zhang, Jiajun Wu, Boqing Gong, and Joshua B Tenenbaum.  Look, listen, and act: Towards audio-visual embodied navigation.  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 9701–9707. IEEE, 2020.
  * Gavish and Graves (1978) Bezalel Gavish and Stephen C Graves.  The travelling salesman problem and related problems.  1978\.
  * Gronauer and Diepold (2022) Sven Gronauer and Klaus Diepold.  Multi-agent deep reinforcement learning: a survey.  _Artificial Intelligence Review_ , 55(2):895–943, 2022.
  * Gudehus and Kotzab (2012) Timm Gudehus and Herbert Kotzab.  _Comprehensive logistics_.  Springer Science & Business Media, 2012.
  * Gupta and Farahat (2020) Chetan Gupta and Ahmed Farahat.  Deep learning for industrial ai: Challenges, new methods and best practices.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining_ , pages 3571–3572, 2020.
  * Han et al. (2020) Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu, et al.  A survey on visual transformer.  _arXiv preprint arXiv:2012.12556_ , 2(4), 2020.
  * Hausknecht and Stone (2015) Matthew Hausknecht and Peter Stone.  Deep recurrent q-learning for partially observable mdps.  _2015 AAAI fall symposium series_ , 2015.
  * Herrmann (2006) Jeffrey W Herrmann.  _Handbook of production scheduling_ , volume 89.  Springer Science & Business Media, 2006.
  * Ho and Ermon (2016) Jonathan Ho and Stefano Ermon.  Generative adversarial imitation learning.  _NeurIPS_ , 2016.
  * Hua et al. (2021) Jiang Hua, Liangcai Zeng, Gongfa Li, and Zhaojie Ju.  Learning for a robot: Deep reinforcement learning, imitation learning, transfer learning.  _Sensors_ , 21(4):1278, 2021.
  * Huang et al. (2020) Wenlong Huang, Igor Mordatch, and Deepak Pathak.  One policy to control them all: Shared modular policies for agent-agnostic control.  In _International Conference on Machine Learning_ , pages 4455–4464. PMLR, 2020.
  * Janner et al. (2021) Michael Janner, Qiyang Li, and Sergey Levine.  Offline reinforcement learning as one big sequence modeling problem.  _Advances in neural information processing systems_ , 34:1273–1286, 2021.
  * Jensen et al. (2016) Morten Bornø Jensen, Mark Philip Philipsen, Andreas Møgelmose, Thomas Baltzer Moeslund, and Mohan Manubhai Trivedi.  Vision for looking at traffic lights: Issues, survey, and perspectives.  _IEEE Transactions on Intelligent Transportation Systems_ , 17(7):1800–1815, 2016.
  * Jiang et al. (2022) Zengqiang Jiang, Shuai Yuan, Jing Ma, and Qiang Wang.  The evolution of production scheduling from industry 3.0 through industry 4.0.  _International Journal of Production Research_ , 60(11):3534–3554, 2022.
  * Kaiser et al. (2017) Lukasz Kaiser, Aidan N Gomez, Noam Shazeer, Ashish Vaswani, Niki Parmar, Llion Jones, and Jakob Uszkoreit.  One model to learn them all.  _arXiv preprint arXiv:1706.05137_ , 2017.
  * Kenton and Toutanova (2019) Jacob Devlin Ming-Wei Chang Kenton and Lee Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _Proceedings of NAACL-HLT_ , 2019.
  * Khatri and Ng (2000) Naresh Khatri and H Alvin Ng.  The role of intuition in strategic decision making.  _Human relations_ , 53(1):57–86, 2000.
  * Kipf and Welling (2017) Thomas N. Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  In _International Conference on Learning Representations_ , 2017.  URL https://openreview.net/forum?id=SJU4ayYgl.
  * Kirk (2004) Donald E Kirk.  _Optimal control theory: an introduction_.  Courier Corporation, 2004.
  * Kirk et al. (2021) Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel.  A survey of generalisation in deep reinforcement learning.  _arXiv preprint arXiv:2111.09794_ , 2021.
  * Krizhevsky et al. (2012) Alex Krizhevsky et al.  Imagenet classification with deep convolutional neural networks.  _NeurIPS_ , 2012.
  * Kuba et al. (2021) Jakub Grudzien Kuba, Ruiqing Chen, Munning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang.  Trust region policy optimisation in multi-agent reinforcement learning.  _arXiv preprint arXiv:2109.11251_ , 2021.
  * LeCun et al. (2015) Yann LeCun, Yoshua Bengio, and Geoffrey Hinton.  Deep learning.  _nature_ , 521(7553):436–444, 2015.
  * Lin and Kernighan (1973) Shen Lin and Brian W Kernighan.  An effective heuristic algorithm for the traveling-salesman problem.  _Operations research_ , 21(2):498–516, 1973.
  * Lin et al. (2017) Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio.  A structured self-attentive sentence embedding.  _arXiv preprint arXiv:1703.03130_ , 2017.
  * Liu et al. (2021a) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _arXiv preprint arXiv:2107.13586_ , 2021a.
  * Liu et al. (2021b) Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  pages 10012–10022, 2021b.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ , 2017.
  * Meng et al. (2021) Linghui Meng, Muning Wen, Yaodong Yang, Chenyang Le, Xiyun Li, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, and Bo Xu.  Offline pre-trained multi-agent decision transformer: One big sequence model conquers all starcraftii tasks.  _arXiv preprint arXiv:2112.02845_ , 2021.
  * Mnih et al. (2013) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_ , 2013.
  * Murphy (2019) Robin R Murphy.  Introduction to ai robotics.  2019\.
  * Nolan (2011) Michael S Nolan.  _Fundamentals of air traffic control_.  Cengage learning, 2011.
  * Peres et al. (2020) Ricardo Silva Peres, Xiaodong Jia, Jay Lee, Keyi Sun, Armando Walter Colombo, and Jose Barata.  Industrial artificial intelligence in industry 4.0-systematic review, challenges and outlook.  _IEEE Access_ , 8:220121–220139, 2020.
  * Perez-Liebana et al. (2019) Diego Perez-Liebana, Katja Hofmann, Sharada Prasanna Mohanty, Noburu Kuno, Andre Kramer, Sam Devlin, Raluca D Gaina, and Daniel Ionita.  The multi-agent reinforcement learning in malm$\backslash$" o (marl$\backslash$" o) competition.  _arXiv preprint arXiv:1901.08129_ , 2019.
  * Petrenko et al. (2020) Aleksei Petrenko, Zhehui Huang, Tushar Kumar, Gaurav Sukhatme, and Vladlen Koltun.  Sample factory: Egocentric 3d control from pixels at 100000 fps with asynchronous reinforcement learning.  In _International Conference on Machine Learning_ , pages 7652–7662. PMLR, 2020.
  * Phillips-Wren and Ichalkaranje (2008) Gloria Phillips-Wren and Nikhil Ichalkaranje.  _Intelligent decision making: An AI-based approach_ , volume 97.  Springer Science & Business Media, 2008.
  * Poquet and De Laat (2021) Oleksandra Poquet and Maarten De Laat.  Developing capabilities: Lifelong learning in the age of ai.  _British Journal of Educational Technology_ , 52(4):1695–1708, 2021.
  * Qin et al. (2021) Rongjun Qin, Songyi Gao, Xingyuan Zhang, Zhen Xu, Shengkai Huang, Zewen Li, Weinan Zhang, and Yang Yu.  Neorl: A near real-world benchmark for offline reinforcement learning.  _arXiv preprint arXiv:2102.00714_ , 2021.
  * Racanière et al. (2017) Sébastien Racanière, Théophane Weber, David Reichert, Lars Buesing, Arthur Guez, Danilo Jimenez Rezende, Adrià Puigdomènech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, et al.  Imagination-augmented agents for deep reinforcement learning.  _Advances in neural information processing systems_ , 30, 2017.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9, 2019.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , pages 8748–8763. PMLR, 2021.
  * Ranjan (2009) Jayanthi Ranjan.  Business intelligence: Concepts, components, techniques and benefits.  _Journal of theoretical and applied information technology_ , 9(1):60–70, 2009.
  * Reed et al. (2022) Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al.  A generalist agent.  _arXiv preprint arXiv:2205.06175_ , 2022.
  * Russell (2010) Stuart J Russell.  _Artificial intelligence a modern approach_.  Pearson Education, Inc., 2010.
  * Salvato et al. (2021) Erica Salvato, Gianfranco Fenu, Eric Medvet, and Felice Andrea Pellegrino.  Crossing the reality gap: a survey on sim-to-real transferability of robot controllers in reinforcement learning.  _IEEE Access_ , 2021.
  * Schmidhuber (2018) Juergen Schmidhuber.  One big net for everything.  _arXiv preprint arXiv:1802.08864_ , 2018.
  * Schmitt et al. (2020) Simon Schmitt, Matteo Hessel, and Karen Simonyan.  Off-policy actor-critic with shared experience replay.  In _International Conference on Machine Learning_ , pages 8545–8554. PMLR, 2020.
  * Seger and Peterson (2013) Carol A Seger and Erik J Peterson.  Categorization= decision making+ generalization.  _Neuroscience & Biobehavioral Reviews_, 37(7):1187–1200, 2013.
  * Shazeer (2020) Noam Shazeer.  Glu variants improve transformer.  _arXiv preprint arXiv:2002.05202_ , 2020.
  * Silver et al. (2016) David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al.  Mastering the game of go with deep neural networks and tree search.  _nature_ , 529(7587):484–489, 2016.
  * Silver et al. (2017) David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al.  Mastering the game of go without human knowledge.  _nature_ , 550(7676):354–359, 2017.
  * Simon (1990) Herbert A Simon.  Bounded rationality.  In _Utility and probability_ , pages 15–18. Springer, 1990.
  * Singh et al. (2021) Bharat Singh, Rajesh Kumar, and Vinay Pratap Singh.  Reinforcement learning in robotic applications: a comprehensive survey.  _Artificial Intelligence Review_ , pages 1–46, 2021.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le.  Sequence to sequence learning with neural networks.  _Advances in neural information processing systems_ , 27, 2014.
  * Sutton (2019) Richard Sutton.  The bitter lesson.  _Incomplete Ideas (blog)_ , 13:12, 2019.
  * Sutton and Barto (2018) Richard S. Sutton and Andrew G. Barto.  _Reinforcement Learning: An Introduction_.  A Bradford Book, Cambridge, MA, USA, 2018.  ISBN 0262039249.
  * Todorov et al. (2012) Emanuel Todorov, Tom Erez, and Yuval Tassa.  Mujoco: A physics engine for model-based control.  In _2012 IEEE/RSJ international conference on intelligent robots and systems_ , pages 5026–5033. IEEE, 2012.
  * Tsai et al. (2019) Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov.  Multimodal transformer for unaligned multimodal language sequences.  In _Proceedings of the conference. Association for Computational Linguistics. Meeting_ , volume 2019, page 6558. NIH Public Access, 2019.
  * Tunyasuvunakool et al. (2020) Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa.  dm control: Software and tasks for continuous control.  _Software Impacts_ , 6:100022, 2020.  ISSN 2665-9638.  doi: https://doi.org/10.1016/j.simpa.2020.100022.  URL https://www.sciencedirect.com/science/article/pii/S2665963820300099.
  * Vandenhende et al. (2021) Simon Vandenhende, Stamatios Georgoulis, Wouter Van Gansbeke, Marc Proesmans, Dengxin Dai, and Luc Van Gool.  Multi-task learning for dense prediction tasks: A survey.  _IEEE transactions on pattern analysis and machine intelligence_ , 2021.
  * Vinyals et al. (2019a) Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al.  Grandmaster level in starcraft ii using multi-agent reinforcement learning.  _Nature_ , 575(7782):350–354, 2019a.
  * Vinyals et al. (2019b) Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al.  Grandmaster level in starcraft ii using multi-agent reinforcement learning.  _Nature_ , 575(7782):350–354, 2019b.
  * Wang et al. (2017) Jun Wang, Weinan Zhang, Shuai Yuan, et al.  Display advertising with real-time bidding (rtb) and behavioural targeting.  _Foundations and Trends® in Information Retrieval_ , 11(4-5):297–435, 2017.
  * Wang et al. (2022) Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al.  Image as a foreign language: Beit pretraining for all vision and vision-language tasks.  _arXiv preprint arXiv:2208.10442_ , 2022.
  * Waubert de Puiseau et al. (2022) Constantin Waubert de Puiseau, Richard Meyes, and Tobias Meisen.  On reliability of reinforcement learning based production scheduling systems: a comparative survey.  _Journal of Intelligent Manufacturing_ , 33(4):911–927, 2022.
  * Wen et al. (2022a) Muning Wen, Jakub Grudzien Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang.  Multi-agent reinforcement learning is a sequence modeling problem.  _arXiv preprint arXiv:2205.14953_ , 2022a.
  * Wen et al. (2022b) Qingsong Wen, Tian Zhou, Chaoli Zhang, Weiqi Chen, Ziqing Ma, Junchi Yan, and Liang Sun.  Transformers in time series: A survey.  _arXiv preprint arXiv:2202.07125_ , 2022b.
  * Williams and Zipser (1989) Ronald J Williams and David Zipser.  A learning algorithm for continually running fully recurrent neural networks.  _Neural computation_ , 1(2):270–280, 1989.
  * Winston and Goldberg (2004) Wayne L Winston and Jeffrey B Goldberg.  _Operations research: applications and algorithms_ , volume 3.  Thomson Brooks/Cole Belmont, 2004.
  * Xiong et al. (2020) Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu.  On layer normalization in the transformer architecture.  In _International Conference on Machine Learning_ , pages 10524–10533. PMLR, 2020.
  * Xu et al. (2022) Peng Xu, Xiatian Zhu, and David A Clifton.  Multimodal learning with transformers: A survey.  _arXiv preprint arXiv:2206.06488_ , 2022.
  * Yin et al. (2021) Qiyue Yin, Jun Yang, Wancheng Ni, Bin Liang, and Kaiqi Huang.  Ai in games: Techniques, challenges and opportunities.  _arXiv preprint arXiv:2111.07631_ , 2021.
  * Yu et al. (2020) Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine.  Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.  In _Conference on robot learning_ , pages 1094–1100. PMLR, 2020.
  * Zhang and Yang (2021) Yu Zhang and Qiang Yang.  A survey on multi-task learning.  _IEEE Transactions on Knowledge and Data Engineering_ , 2021.
  * Zhao et al. (2020) Wenshuai Zhao, Jorge Peña Queralta, and Tomi Westerlund.  Sim-to-real transfer in deep reinforcement learning for robotics: a survey.  In _2020 IEEE Symposium Series on Computational Intelligence (SSCI)_ , pages 737–744. IEEE, 2020.
  * Zheng et al. (2021) Boyuan Zheng, Sunny Verma, Jianlong Zhou, Ivor Tsang, and Fang Chen.  Imitation learning: Progress, taxonomies and opportunities.  _arXiv preprint arXiv:2106.12177_ , 2021.