# Karolos: An Open-Source Reinforcement Learning Framework for Robot-Task
Environments

Christian Bitter  Timo Thun  Tobias Meisen

###### Abstract

In Reinforcement Learning (RL) research, simulations enable benchmarks between
algorithms, as well as prototyping and hyper-parameter tuning of agents. In
order to promote RL both in research and real-world applications, frameworks
are required which are on the one hand efficient in terms of running
experiments as fast as possible. On the other hand, they must be flexible
enough to allow the integration of newly developed optimization techniques,
e.g. new RL algorithms, which are continuously put forward by an active
research community. In this paper, we introduce _Karolos_ , a RL framework
developed for robotic applications, with a particular focus on transfer
scenarios with varying robot-task combinations reflected in a modular
environment architecture. In addition, we provide implementations of state-of-
the-art RL algorithms along with common learning-facilitating enhancements, as
well as an architecture to parallelize environments across multiple processes
to significantly speed up experiments. The code is open source and published
on GitHub with the aim of promoting research of RL applications in robotics.

Reinforcement Learning, Industrial Robotics

RL

    Reinforcement Learning
HER

    Hindsight Experience Replay
DR

    Domain Randomization
PER

    Prioritized Experience Replay
TD

    Temporal Difference
MC

    Monte Carlo

##  1 Introduction

Reinforcement Learning (RL) aims at learning to solve tasks by an agent
exploring an environment in a trial-and-error fashion. An agent observes the
state of an environment and decides for an action according to its policy.
Performing the action results in a state change of the environment. By
receiving rewards that reflect the status of a certain task and gaining
sufficient experience, an agent can derive which actions are useful with
respect to solving the task. Recent successes of reinforcement learning
include learning strategies for games such as Go (Silver et al., 2017) and
StarCraft (Vinyals et al., 2019), to industrial settings such as production
scheduling (Kemmerling et al., 2021) and robotic manipulation tasks (Levine et
al., 2016; Meyes et al., 2018).

In real-world robotics scenarios, the highly iterative, data-driven nature of
reinforcement learning has led to the use of simulations to pre-train agents
and subsequently perform a sim2real transfer to reduce real-world training
efforts (Tobin et al., 2017). On top of easing access to environments, thus
facilitating benchmarking, simulations run faster and can be parallelized,
which significantly speeds up experiments. Also, actions in simulations are
consequence-free, meaning that performing sub-optimal actions does not bear
the risk of hardware damage and associated costs.

As RL is an active field of research with first real-world applications, two
major domains can be identified. On the one hand, research on new algorithms
and techniques to make learning more efficient is mostly focused on improving
agents. Hereby, to effectively compare different approaches, it is essential
that the benchmark environments, such as OpenAI Gym (Brockman et al., 2016),
exist and can be utilized. However, research by its very nature also
necessitates a higher degree of flexibility and modifiability of software to
model particular aspects not relevant to most other use cases. On the other
hand, applications of RL are focused on framing a certain use-case as an
environment and adapting existing and tested algorithms by matching interfaces
and setting hyper-parameters to the problem at hand. While both domains,
research and application of RL, focus on different aspects, they naturally
depend on each other and benefit from similar optimization techniques.

In this work, we introduce our software framework _Karolos_ aimed at research
of RL in the robotics domain with a distinct focus on cross-robot transfer
learning. Transfer learning in general is concerned with the reuse of
knowledge acquired in one domain towards facilitating learning in another
domain. Cross-robot transfer learning as a subfield of research is aimed at
transferring knowledge about solving a common task between different robotic
systems, which may differ in both morphology and/or abilities. Our
contribution for this field of research is a RL framework which is

  * •

modular: In order to accommodate a wide spectrum of application scenarios, a
generalized framework must allow for a high degree of flexibility regarding
the combination as well as parametrization of modular components, such as RL
algorithms and environments. Additionally, as RL is an active field of
research, it is important to keep frameworks flexible enough to incorporate
future optimization techniques developed by the community. In order to
facilitate research into cross-robot transfer learning, we provide a modular
infrastructure which allows the composition of environments by mix-and-
matching robot models with tasks. However, it is also possible to quickly
integrate other (non-robotic) RL environments, such as the gym benchmark
(Brockman et al., 2016).

  * •

fast: RL experiments are usually very data-intensive due to their trial-and-
error nature. As applications become more complex, the need for speeding up
experiments to keep costs feasible will only increase. Besides improving the
efficiency of learning algorithms, it is also possible to use multiple
environments in parallel to generate more interaction experience for an agent
to learn from. In _Karolos_ , we provide common optimization techniques, such
as Prioritized Experience Replay (PER) (Schaul et al., 2015), Hindsight
Experience Replay (HER) (Andrychowicz et al., 2017) and Domain Randomization
(DR) (Tobin et al., 2017). In addition, our architecture allows massive
parallelization of environments.

  * •

open-source: In order to facilitate research, it is important to utilize
software packages accessible to everyone. The framework is built using open-
source software components in order to ensure access, regular updates and
long-term support (Paulson et al., 2004). To allow operation without the need
to acquire additional software licenses, _Karolos_ is based on the open-source
physics engine Pybullet (Coumans & Bai, 2016–2019) and open-source tensor
computation library Pytorch (Paszke et al., 2019).

The framework is available at https://github.com/tmdt-buw/karolos.

##  2 Software description

In the following we describe the software architecture of our framework
Karolos and motivate the various design choices.

###  2.1 Software Architecture

The architecture of the _Karolos_ framework is depicted in Figure 1. When
launching an experiment, the Experiment serves as the entry point. The class
Experiment coordinates the interactions between an RL agent and one or
multiple environments. Its main functions are to keep track of the episode
count and monitor agent performance by testing at regular intervals, meaning
that the agent is executed without exploration for a specified amount of
epochs to calculate performance metrics. An experiment can be monitored in
real-time, as all parameters are logged using tensorboard (Abadi et al.,
2016).

Figure 1: The software architecture of Karolos consists of Environments, which
are combinations of Robots and Tasks, and Agents used to learn a respective
task. The training is controlled and monitored by the Manager.

The collection of experience is one major bottleneck when prototyping RL
agents. For this purpose, the class Orchestrator is utilized to parallelize
multiple copies of an environment. Parallelization is hereby possible by
multiprocessing, multithreading or a combination of both. The individual
environments are addressed via unique IDs, which are used by an instance of
the Experiment class to match asynchronously collected corresponding states
and actions into valid experience trajectories.

An instance of the interface Environment represents an individual environment
and provides an interface which allows resetting and executing actions. In
addition, each environment must explicitly define a reward function and
success criterion. The information required to determine reward and success is
not included in the state, but separated in a goal variable. These design
choices are required for data augmentation methods such as HER (Andrychowicz
et al., 2017), which retroactively adjust experience based on hypothetical
goals.

As _Karolos_ is developed with a focus on research into cross-robot transfer
learning (Devin et al., 2017; Gupta et al., 2017; Fickinger et al., 2021), a
composable environment EnvironmentRobotTask is provided. The environment is
further structured into a Robot and a Task, which can be combined arbitrarily
as demonstrated in Figure 2. The Robot represents the physical unit which can
be manipulated by an agent. It’s step function therefore takes an action and
translates it into movement. The Task contains all elements which constitute
the environment surrounding the robot, such as manipulable objects and
obstacles. It also provides the reward function and success criterion, which
formalize the task at hand. Thus We utilize Pybullet as the physics engine to
simulate the interactions in the environment (Coumans & Bai, 2016–2019).

Figure 2: The separation of an environment into robots and tasks allows for
arbitrary combination into a multitude of robotic scenarios.

To date, three robotic arms are available: a Franka Emika Panda
111www.franka.de, a Universal Robots UR5 222www.universal-robots.com and a
KUKA LBR iiwa 333www.kuka.com, whereby the latter two are equipped with a
Robotiq 2F-85 Gripper 444www.robotiq.com. The robots were chosen, as they are
in general attractive both for research and industrial applications due to
them being safer and cheaper than traditional industrial robots (Galin et al.,
2020). As the robots are self-contained modules, adding new robots and using
them interchangeably is straightforward.

Complementary, three tasks are implemented:

  * •

Reach: A robot’s end effector must be moved to a specified target position.
This task arguably poses the simplest robotic task of reaching a desired
configuration, without interaction with any objects or obstacles.

  * •

Pick_Place: An object must be picked up and moved to a desired target
location. This task poses a more realistic challenge, as pick-and-place
scenarios are common applications for industrial robots (Dillmann, 2020).

  * •

Throw: An object is spawned in the robot’s gripper. The robot must be moved
and gripper opened so that the released object reaches a target location
outside the robot’s reach. This scenario introduces a delayed reward, as the
agent must learn that it’s success only depend on actions performed prior to
the release of the object.

Analogous to robots, tasks are stand-alone modules which facilitates the
implementation of new tasks and allows for arbitrary robot-task combinations.

Finally, an instance of the Agent interface contains the actual RL algorithm.
It hosts the models required for training and making predictions, and uses a
Replay Buffer to store and sample collected experience. To date, four deep RL
architectures are implemented in _Karolos_ : soft actor-critic (SAC) (Haarnoja
et al., 2018), deep deterministic policy gradient (DDPG) (Lillicrap et al.,
2015), deep Q-learning (DQN) (Hasselt et al., 2016) and Proximal Policy
Optimization (PPO) (Schulman et al., 2017).

###  2.2 Additional Optimization Techniques

In this section, we highlight three popular concepts which are implemented as
extensions in _Karolos_ : PER (2.2.1) and HER (2.2.2) which facilitate off-
policy algorithmic convergence and DR (2.2.3) to bridge the simulation-to-
reality gap for transferring algorithms pre-trained in simulation to real-
world setups. As before, modularity was emphasized and led to numerous
important design choices, which will be explained in the following.

####  2.2.1 Prioritized Experience Replay

During training, not all experience is equally valuable with regard to
improving an agent. It is much more efficient to show the agent samples of
sparse events rather than samples which occur frequently in order to maximize
the insights to be gained. One option to achieve this is by using a PER buffer
(Schaul et al., 2015). Instead of sampling data from the pool of collected
experience uniformly at every learning step, samples are weighted according to
the magnitude of the Temporal Difference (TD) error. The weights of the
individual samples can hereby be determined by a probability distribution,
which formalizes the deviation of the state-action valuation by the agent to
the actual valuation under consideration of the observed reward. By doing so,
the agent effectively optimizes on samples more frequently where it’s value
predictions are still off. In _Karolos_ PER is implemented in the class
ReplayBufferPriority. By modularizing the replay buffer functionality it is
possible to switch to other sampling strategies, e.g. an first-in-first-out
sampling without replacement strategy (ReplayBufferOnpolicy) used in on-policy
algorithms such as PPO.

####  2.2.2 Hindsight Experience Replay

Finding a dense reward function which guides the agent into the direction of a
solution can be a hard endeavour. Instead, many complex tasks can only be
described by a sparse reward function. This, however, impedes learning as the
chances of an agent receiving a positive reward signal requires it to solve
the task during exploration, the probability of which diminishes exponentially
with the amount of steps required for solving a task. HER is a strategy to
mitigate the implications of sparse reward functions on policy optimization
(Andrychowicz et al., 2017). The intuition behind HER is to learn as much from
a failed attempt as from a successful one. This is achieved by modifying an
experience trajectory after completion. In essence, the states in the
experiences are modified in such a way that the goal is no longer the original
one, but instead corresponds to the state the agent ended up in. This data
augmentation provides the agent with experience where the goal was achieved,
even if it was missed in the actual trajectory. In order to achieve HER in
_Karolos_ , experience is buffered by the instance of Experiment until an
episode is terminated. The complete trajectory is then handed over to the
agent, which generates new samples according to HER and finally stores the
original and the new HER experience samples in its memory buffer.

####  2.2.3 Domain Randomization

The final goal of robotics research is to deploy agents to the real world,
with simulations allowing for a fast and economic pretraining. However, the
inherent complexity resulting from unknown or complicated causalities in the
real world makes exact simulations infeasible. DR is an approach to overcome
this gap between simulation and reality (Tobin et al., 2017). By intentionally
introducing parameter modifications into training, an agent is exposed to many
environment variations. Doing so increases the robustness of an agent to such
variations, which increases the chances of a successful transfer to the real
world. In _Karolos_ , parameters such as gravity or joint frictions are
sampled from parametrizable normal distributions at each reset of an
environment. The randomization is thereby delegated to individual modules,
i.e. the randomization of robot parameters is handled by the robot instance.

##  3 Illustrative Examples

Launching an experiment with _Karolos_ can be achieved with a few lines of
code, as depected in Listing 1. Each experiment is specified by a dictionary,
which is stored in the results directory for documentation. After launching
the experiment, the training progress is logged into the results directory and
can be monitored in real time using tensorboard.

Listing 1: Minimal example of launching an experiment in Karolos

⬇

from karolos.experiment import Experiment

Experiment({

”total_timesteps”: 5_000_000,

”test_interval”: 500_000,

”agent_config”: {

”name”: ”sac”,

},

”env_config”: {

”robot_config”: {

”name”: ”panda”,

},

”task_config”: {

”name”: ”reach”,

},

}

}).run(results_dir=”./results”)

###  3.1 Agent Training

To demonstrate general functionality of _Karolos_ , we train two agents, SAC
and PPO on solving the task TaskReach with the robot RobotPanda. The reach
task serves as good benchmark for general functionality, as it is the simplest
robotics task. The resulting success ratios are displayed in Figures 4 and 4.
The success ratio measures the percentage of cases in which the
success_criterion of the task was met during a certain number, in our case
100, test episodes with random starting conditions using the policy of the
agent. Each algorithm is executed seven times with different random seeds, and
a kernel density estimation (KDE) technique using Simpson’s rule is applied to
approximate the underlying probability density function of each algorithms’
performance. The results clearly show, that the reach task can be learned with
the implementations in _Karolos_.

Figure 3: Test success ratio of solving the task TaskReach with the robot
RobotPanda using the agent AgentSAC.

Figure 4: Test success ratio of solving the task TaskReach with the robot
RobotPanda using the agent AgentPPO.

####  3.1.1 Parallelization

As described above, one central entity of _Karolos_ is the Orchestrator, which
is used to spawn environments in parallel in multiple processes and threads.
To showcase the speedup achieved by parallelization, we perform two
benchmarking experiments. In the first experiment, we initialize Orchestrators
to use an increasing amount of CPUs. For each configuration, environments are
reset 1.000 times and the wall clock time is recorded. Each configuration is
run 10 times and the results are displayed in Figure 6. As expected, the
durations significantly decrease with increasing parallelization through
multiprocessing. While not as extensive, similar observations can be made when
increasing the number of environments even further with multithreading, as
depicted in Figure 6. Here, we chose the maximum number of CPUs available
(96), as this setting yielded the best performance in the previous experiment.
The benchmarking code is available in the examples folder and may be used to
determine the optimal orchestrator parameters for a given system.

Figure 5: Benchmarking experiment of wall clock speed increase by spawning
environments on multiple CPUs. Each CPU holds one environment.

Figure 6: Further speed improvement by spawning multiple environments per CPU.
A total of 96 CPUs was used.

###  3.2 Industrial Applications

In addition to its application in academic research and education of
reinforcement learning for robotics, _Karolos_ was already successfully
utilized in two industrial research settings, which we will describe in the
following.

####  3.2.1 Use-Case 1: Robotic Assembly in Airplane Manufacturing

In the aerospace industry, many assembly tasks are still performed manually
due to two reasons: On the one hand, product quantities are relatively low
compared to other industries, such as automotive. Thus, amortization periods
are much longer. On the other hand, structures are usually very big and
difficult to move, which means that a robotic solution needs to be mobile,
which brings additional challenges, such as the compensation of offsets
resulting from movement. To address these challenges, a research project was
conducted to investigate the possibilities for employing RL as means to reduce
manual programming efforts and thus make an automation solution more viable.
In addition, the cross-robot transfer necessary to enable an
interchangeability of different cobots was investigated 7. _Karolos_ was
chosen as backbone for the RL architecture. Effectively, we were able to reuse
the entire training pipeline as-is. The only major modification lay in the
creation of a suitable environment on the basis of process specifications and
proprietary virtual models, which were integrated into a simulation based on
Pybullet. The simulation environment is depicted in Figure 7. The experiments,
and the success of the project in direct dependence, significantly benefited
from the ability of parallelization and the resulting reduction of time-to-
insights.

Figure 7: The simulation environment developed for research into the assembly
of aircraft shells with different collaborative robots.

####  3.2.2 Use-Case 2: Order-based Job Shop Scheduling

In an array of industries such as lumber (Faaland & Briggs, 1984), glass
(Durak & Tuzun Aksu, 2017) or textile (Ozdamar, 2000), the task lies in
cutting a big piece of material into smaller chunks. Specifically, the yield
is to be increased by cutting around defects in a way which produces chunks to
certain quality specifications. To investigate the possibility to employ RL as
means to learn this optimization problem under consideration of boundary
conditions, such as different order sizes and quality requirements, an
industrial research project was conducted. Again, _Karolos_ was chosen as RL
framework. In this instance, the availability of a pre-existing process
simulation required the implementation of a suitable environment wrapper,
after which the rest of the learning architecture was reusable to conduct
necessary experiments. While not surprising due to the domain-agnostic nature
of RL, the ability to significantly reduce the ramp-up to the necessary
experiments hints towards a usability of _Karolos_ for use-cases in domains
different from our initial focus on robotics.

##  4 Related Work

In recent years numerous software packages emerged in the RL domain, which
focus on different aspects of research - from building environments suitable
for benchmarking purposes to implementing the algorithms themselves in order
to facilitate application of RL.

The most similar projects to _Karolos_ in terms of the robotic environment are
panda-gym (Gallouédec et al., 2021) and robosuite (Zhu et al., 2020), which
both share similar design decisions regarding the separation of robots and
tasks. The major differences to _Karolos_ are, that panda-gym only provides an
implementation of the Panda robot and robosuite is based on the Mujoco physics
engine (Todorov et al., 2012). Other projects, such as rl-reach (Aumjaud et
al., 2021), Gymperium (Ellenberger, 2018–2019), Gym (Brockman et al., 2016) or
Metaworld (Yu et al., 2020) do not provide the flexibility in composing
scenarios by selecting both the robot and the task.

One important sidenote regarding projects based on Mujoco (Todorov et al.,
2012) is the circumstance that the python package mujoco-py
555https://github.com/openai/mujoco-py was deprecated for Windows. While
Mujoco was aquired by the company Deepmind and the new Python package mujoco
supports Windows, to date many projects have not been updated accordingly.
Being built upon Pybullet, _Karolos_ can be run on Windows, Linux and Mac
machines.

Due to the generalized and straight-forward formalization of RL, the
implementation of our training pipeline shares many similarities, and is
partly inspired by, other projects (Raffin et al., 2019; Castro et al., 2018;
Gauci et al., 2018; Liang et al., 2018), with rlpyt (Stooke & Abbeel, 2019)
being the most similar to our implementation in _Karolos_ regarding the
architectural structure. Two aspects which set _Karolos_ apart are on the one
hand its utilization of multithreading for further parallelization of
environments. While the most significant speedup is achieved through
multiprocessing, the minor improvement by multithreading scales, as the tasks
RL is applied to become more and more complex. On the other hand, the modular
architecture allows an easy way to implement or customize agents. Some
projects either prioritize a lean code base, such as stable-baselines (Raffin
et al., 2019), which is aimed at benchmarking different RL algorithms and
exhibits a pronounced inheritance structure. Other projects, such as rlpyt
(Stooke & Abbeel, 2019) provide a degree of modularity, however in some cases
not sufficiently. One example of this are is PER, which is implemented for the
deep-q agent (DQN) and thus not available for the other agents without
reimplementation.

##  5 Conclusions and Future Work

This paper presents our reinforcement-learning framework _Karolos_ , with
which we provide environments of robotic and common RL benchmarking tasks as
well as RL algorithms. We deliberately chose open-source components for
_Karolos_ , as we see a necessity to provide open-source training frameworks
to the research community. In addition, we emphasized modularity in the
software architecture, which facilitates the integration of new components and
concepts into the framework.

While to date the framework is functional and can be used to train agents
successfully, the project is under active development, which is driven by our
applications in both academic and industrial research projects. Our focus in
the near future will be set on the expansion of the available simulations to
create a benchmark for robot-to-robot transfer learning (Devin et al., 2017;
Gupta et al., 2017) as a complement to research into task-to-task transfer
learning (Yu et al., 2020). We also plan on using the foundations laid in this
work to to continuously expand _Karolos_ by new robots, tasks and agents.

## References

  * Abadi et al. (2016) Abadi, M., Agarwal, A., Barham, P., Brevdo, E., Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., et al.  Tensorflow: Large-scale machine learning on heterogeneous distributed systems.  _arXiv preprint arXiv:1603.04467_ , 2016.
  * Andrychowicz et al. (2017) Andrychowicz, M., Wolski, F., Ray, A., Schneider, J., Fong, R., Welinder, P., McGrew, B., Tobin, J., Abbeel, P., and Zaremba, W.  Hindsight experience replay.  _arXiv preprint arXiv:1707.01495_ , 2017.
  * Aumjaud et al. (2021) Aumjaud, P., McAuliffe, D., Lera, F. J. R., and Cardiff, P.  rl_reach: Reproducible reinforcement learning experiments for robotic reaching tasks.  _Software Impacts_ , 8:100061, 2021.
  * Brockman et al. (2016) Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W.  Openai gym.  _arXiv preprint arXiv:1606.01540_ , 2016.
  * Castro et al. (2018) Castro, P. S., Moitra, S., Gelada, C., Kumar, S., and Bellemare, M. G.  Dopamine: A research framework for deep reinforcement learning.  _arXiv preprint arXiv:1812.06110_ , 2018.
  * Coumans & Bai (2016–2019) Coumans, E. and Bai, Y.  Pybullet, a python module for physics simulation for games, robotics and machine learning.  https://github.com/bulletphysics/bullet3, 2016–2019.
  * Devin et al. (2017) Devin, C., Gupta, A., Darrell, T., Abbeel, P., and Levine, S.  Learning modular neural network policies for multi-task and multi-robot transfer.  In _2017 IEEE international conference on robotics and automation (ICRA)_ , pp. 2169–2176. IEEE, 2017.
  * Dillmann (2020) Dillmann, R.  Types of robots and their integration into computer-integrated manufacturing systems rüdiger dillmann university of karlsruhe, karlsruhe, federal republic of germany.  _Robot Technology and Applications_ , pp. 1, 2020.
  * Durak & Tuzun Aksu (2017) Durak, B. and Tuzun Aksu, D.  Dynamic programming and mixed integer programming based algorithms for the online glass cutting problem with defects and production targets.  _International Journal of Production Research_ , 55(24):7398–7411, 2017.
  * Ellenberger (2018–2019) Ellenberger, B.  Pybullet gymperium.  https://github.com/benelot/pybullet-gym, 2018–2019.
  * Faaland & Briggs (1984) Faaland, B. and Briggs, D.  Log bucking and lumber manufacturing using dynamic programming.  _Management science_ , 30(2):245–257, 1984.
  * Fickinger et al. (2021) Fickinger, A., Cohen, S., Russell, S., and Amos, B.  Cross-domain imitation learning via optimal transport, 2021.  URL http://arxiv.org/pdf/2110.03684v2.
  * Galin et al. (2020) Galin, R., Meshcheryakov, R., Kamesheva, S., and Samoshina, A.  Cobots and the benefits of their implementation in intelligent manufacturing.  In _IOP Conference Series: Materials Science and Engineering_ , volume 862, pp. 032075. IOP Publishing, 2020.
  * Gallouédec et al. (2021) Gallouédec, Q., Cazin, N., Dellandréa, E., and Chen, L.  panda-gym: Open-Source Goal-Conditioned Environments for Robotic Learning.  _4th Robot Learning Workshop: Self-Supervised and Lifelong Learning at NeurIPS_ , 2021.
  * Gauci et al. (2018) Gauci, J., Conti, E., Liang, Y., Virochsiri, K., Chen, Z., He, Y., Kaden, Z., Narayanan, V., and Ye, X.  Horizon: Facebook’s open source applied reinforcement learning platform.  _arXiv preprint arXiv:1811.00260_ , 2018.
  * Gupta et al. (2017) Gupta, A., Devin, C., Liu, Y., Abbeel, P., and Levine, S.  Learning invariant feature spaces to transfer skills with reinforcement learning, 2017.
  * Haarnoja et al. (2018) Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P., and Levine, S.  Soft actor-critic algorithms and applications, 2018.
  * Hasselt et al. (2016) Hasselt, H. V., Guez, A., and Silver, D.  Deep reinforcement learning with double q-learning.  _arXiv preprint arXiv:1509.06461_ , 2016.
  * Kemmerling et al. (2021) Kemmerling, M., Samsonov, V., Janke, T., Lütticke, D., Gützlaff, A., Schmidhuber, M., Meisen, T., and Schuh, G.  Towards productionready reinforcement learning scheduling agents: A hybrid two-step training approach based on discrete-event simulations.  _Simulation in Produktion und Logistik_ , pp. 325–336, 2021.
  * Levine et al. (2016) Levine, S., Finn, C., Darrell, T., and Abbeel, P.  End-to-end training of deep visuomotor policies.  _The Journal of Machine Learning Research_ , 17(1):1334–1373, 2016.
  * Liang et al. (2018) Liang, E., Liaw, R., Nishihara, R., Moritz, P., Fox, R., Goldberg, K., Gonzalez, J., Jordan, M., and Stoica, I.  Rllib: Abstractions for distributed reinforcement learning.  In _International Conference on Machine Learning_ , pp. 3053–3062. PMLR, 2018.
  * Lillicrap et al. (2015) Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D., and Wierstra, D.  Continuous control with deep reinforcement learning.  _arXiv preprint arXiv:1509.02971v6_ , 2015.
  * Meyes et al. (2018) Meyes, R., Scheiderer, C., and Meisen, T.  Continuous motion planning for industrial robots based on direct sensory input.  _Procedia CIRP_ , 72:291–296, 2018.
  * Ozdamar (2000) Ozdamar, L.  The cutting-wrapping problem in the textile industry: optimal overlap of fabric lengths and defects for maximizing return based on quality.  _International Journal of Production Research_ , 38(6):1287–1309, 2000.
  * Paszke et al. (2019) Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.  Pytorch: An imperative style, high-performance deep learning library.  _Advances in neural information processing systems_ , 32:8026–8037, 2019.
  * Paulson et al. (2004) Paulson, J. W., Succi, G., and Eberlein, A.  An empirical study of open-source and closed-source software products.  _IEEE transactions on software engineering_ , 30(4):246–256, 2004.
  * Raffin et al. (2019) Raffin, A., Hill, A., Ernestus, M., Gleave, A., Kanervisto, A., and Dormann, N.  Stable baselines3.  _GitHub repository_ , 2019.
  * Schaul et al. (2015) Schaul, T., Quan, J., Antonoglou, I., and Silver, D.  Prioritized experience replay.  _arXiv preprint arXiv:1511.05952_ , 2015.
  * Schulman et al. (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Silver et al. (2017) Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al.  Mastering the game of go without human knowledge.  _nature_ , 550(7676):354–359, 2017.
  * Stooke & Abbeel (2019) Stooke, A. and Abbeel, P.  rlpyt: A research code base for deep reinforcement learning in pytorch.  https://github.com/astooke/rlpyt, 2019.
  * Tobin et al. (2017) Tobin, J., Fong, R., Ray, A., Schneider, J., Zaremba, W., and Abbeel, P.  Domain randomization for transferring deep neural networks from simulation to the real world.  In _2017 IEEE/RSJ international conference on intelligent robots and systems (IROS)_ , pp. 23–30. IEEE, 2017.
  * Todorov et al. (2012) Todorov, E., Erez, T., and Tassa, Y.  Mujoco: A physics engine for model-based control.  In _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pp. 5026–5033. IEEE, 2012.
  * Vinyals et al. (2019) Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., et al.  Grandmaster level in starcraft ii using multi-agent reinforcement learning.  _Nature_ , 575(7782):350–354, 2019.
  * Yu et al. (2020) Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S.  Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.  In _Conference on Robot Learning_ , pp. 1094–1100. PMLR, 2020.
  * Zhu et al. (2020) Zhu, Y., Wong, J., Mandlekar, A., and Martín-Martín, R.  robosuite: A modular simulation framework and benchmark for robot learning.  In _arXiv preprint arXiv:2009.12293_ , 2020.

  *[RL]: Reinforcement Learning
  *[PER]: Prioritized Experience Replay
  *[HER]: Hindsight Experience Replay
  *[DR]: Domain Randomization
  *[TD]: Temporal Difference