#  Position: Automatic Environment Shaping is the Next Frontier in RL

Younghyo Park  Gabriel B. Margolis  Pulkit Agrawal

###### Abstract

Many roboticists dream of presenting a robot with a task in the evening and
returning the next morning to find the robot capable of solving the task. What
is preventing us from achieving this? Sim-to-real reinforcement learning (RL)
has achieved impressive performance on challenging robotics tasks, but
requires substantial human effort to set up the task in a way that is amenable
to RL. It’s our position that algorithmic improvements in policy optimization
and other ideas should be guided towards resolving the primary bottleneck of
shaping the training environment, i.e., designing observations, actions,
rewards and simulation dynamics. Most practitioners don’t tune the RL
algorithm, but other environment parameters to obtain a desirable controller.
We posit that scaling RL to diverse robotic tasks will only be achieved if the
community focuses on automating environment shaping procedures.

Machine Learning, ICML

##  1 Introduction

Figure 1: Flowchart of a typical behavior generation pipeline using
reinforcement learning with simulation, illustrating four distinct subtasks of
sample environment modeling, environment shaping, RL training, and outer
feedback loop with behavior evaluation and reflection. We highlight the
manual, task-driven environment shaping as a key, yet often overlooked,
bottleneck in generalizing the success of RL. We thus advocate for automating
the environment shaping process to broaden RL’s applicability.

The advent of foundation models for speech, vision, and language processing
has revolutionized Artificial Intelligence. It is widely believed that
robotics is the next frontier, and the race to develop a foundation model for
robotics is ongoing. The critical challenge in this pursuit is the limited
availability of robotic data: trajectories of observations and robot actions.
This starkly contrasts with computer vision and natural language processing,
where large amounts of readily available internet data can be used. One way to
gather robotic data is to deploy robots for many tasks in diverse
environments. However, such deployment is only feasible if robots create
value, i.e., successfully solve the task often enough. The result is a
chicken-and-the-egg situation – robots need to be useful to be deployed, but
for them to be useful requires collecting enough data to train a controller
that creates value. The real question, therefore, is bootstrapping data
collection.

A natural way to collect data is by teleoperating a robot to perform diverse
tasks. However, this paradigm is challenging to scale as the human effort
grows linearly with the amount of data that needs to be gathered. To ease data
collection, recent works have made teleoperation more capable and easier
(ALOHA; MobileALOHA), but it doesn’t change the linear scaling. At some point
in the future, it is plausible that we will have enough data to train a large
model that will reduce the number of demonstrations required to learn a new
task and make the human effort sub-linear. However, we are far from that
point. The primary reason is that policies obtained via supervised learning on
a small dataset of demonstrations (i.e., learning from demonstration) have
limited robustness and generalization and, therefore, cannot be used to
collect data autonomously in more diverse settings.

In theory, given a reward function or by inferring a reward function from the
demonstrations, autonomous data collection is possible using reinforcement
learning (RL). It has been hard to realize this promise of RL because training
in the real world often requires babysitting the robot to ensure safe
operation, ensuring the reward function is not hacked, performing resets, and
the data inefficiency of RL algorithms means they run for a long time before
useful behaviors are discovered. Real-world RL training is an active area of
research (luo2024serl), and recent work has shown the plausibility of learning
locomotion behaviors from real-world RL training (wu2023daydreamer;
lei2023uni). However, real-world RL is yet to achieve state-of-the-art robotic
controllers, which means the data it generates is sub-optimal.

A related line of work bypasses the difficulty of training in the real world
by training with RL in simulation and then successfully deploying policies in
reality (i.e., sim-to-real RL). Such training has achieved state-of-the-art
behaviors across many robot morphologies and complex tasks involving legged
locomotion (miki2022learning; ji2023dribble; hoeller2023anymal;
zhuang2023robot; cheng2023extreme; jenelten2024dtc), dexterous manipulation
(andrychowicz2020learning; chen2023visual; handa2023dextreme;
yin2023rotating), drone racing (song2023reaching; kaufmann2023champion) and
others. A single general-purpose RL algorithm, PPO (schulman2017proximal), has
powered these successes. Consequently, one might believe that a recipe for
scaling to diverse tasks and collecting a sizeable high-quality dataset of
trajectories exists. However, the reality is that immense manual task-specific
modeling and engineering are required – something we call environment shaping
(a generalization of the term reward shaping to include other environment
choices to ease optimization) – to make things work, which is the primary
bottleneck in our opinion. Although the environment shaping bottleneck is
highly relevant to robotics, it exists in any domain where reinforcement
learning is applied to real-world problems, including transportation systems
(wei2018intellilight), autonomous driving (wu2017emergent), finance
(finrl_meta_2021), and power management (vazquez2019citylearn).

We expand the previous usage of the term environment shaping
(co2020ecological) to include all choices such as designing reward function,
curriculum, observation/action spaces, initial state distribution, reset
functions performed manually to make training possible. These issues have been
individually studied for a long time (selfridge1985training; ng1999policy;
singh2009rewards; esser2023guided), but a holistic and critical analysis of
the human effort required to make these choices even with the latest
algorithmic advances has not been made.

Using examples from recent applications of RL to robotics, this position paper
argues that the primary bottleneck for scaling up reinforcement learning is
its need for manual environment shaping. Specifically, this is a call to
action for the RL research community:

  * •

Distinguish modeling from shaping design choices in RL environments. Many
works describe the final shaped environment but not a repeatable procedure
that can transfer the same shaping to a new robot or task.

  * •

Prioritize research into automatic environment shaping as a pathway to
generalize domain-specific successes in RL.

  * •

Prioritize benchmarks that measure the total expense of applying reinforcement
learning to real-world robotics tasks, by including environments with
‘unshaped’ versions and corresponding human-shaped baselines. The point being
that existing benchmarks have already performed environment shaping (i.e.,
hide the true problem) and, therefore do not serve as good candidates for
further research into better algorithms. 111 https://auto-env-
shaping.github.io/

##  2 Robotic Behavior Generation with RL

To support a precise definition of environment shaping, we first describe a
typical workflow for generating robotic behaviors using reinforcement learning
(RL) in simulation (Figure 1).

We decompose behavior generation into four subtasks; sample environment
generation (Sec 2.1), environment shaping (Sec 2.2), RL training (Sec 2.3),
and the outer feedback loop with behavior evaluation and reflection (Sec 2.4).
In delineating these substages, we pinpoint typical human efforts involved in
the process.

###  2.1 Modeling Sample Environments

Consider $\hat{p}(e)$ as the oracle distribution of the target environment we
want to deploy our robots in. Our goal is to generate behavior that is
performant (with respect to objective $J$) and robust under $\hat{p}(e)$,

$\max_{\pi}\mathbb{E}_{\hat{e}\sim\hat{p}(e)}J(\pi;\hat{e})$

The target environment can either be a specific real-world environment that
already exists (e.g. kitchen at a specific location) or a generic concept
(e.g. typical household kitchen).

Unfortunately, it’s extremely difficult to model this oracle distribution
either way; it requires comprehensive knowledge of all possible environmental
variables and conditions, which is often infeasible due to the complexity,
variability, and limited observability in the real world. Innate vagueness of
generic concepts is often an issue as well. Just imagine modeling a true
oracle distribution of a chaotic real-world dishwasher in simulation! (Figure
2)

In contrast, modeling a single sample environment, $\hat{e}$, a specific
instance drawn from oracle environment distribution,

$\hat{e}\sim\hat{p}(e),$

and importing that to a simulation is much more feasible. This is why robotics
practitioners typically start the behavior generation process by first
designing a single sample environment: (a) modeling and importing robots and
necessary assets in simulation, and (b) manually placing them in their default
poses. We often generate a set of those sample environments to kick things
off. This is what practitioners do for the first blue box in Figure 1.

Such a set of sample environments actually serves a purpose: it is a useful
representative testbed environment that can be used to estimate the behavior
performance under the true oracle distribution $\hat{p}(e)$. Combined with any
form of task specification $r$ (task-specification-problem), we define a
simulated testbed $\mathcal{E}^{\text{test}}$ where the trained behaviors can
be evaluated.

######  Definition 2.1 (Test Environment).

Let $\\{\hat{e}_{1},\cdots,\hat{e}_{n}\\}$ be a set of $n$ sample environments
each independently drawn from oracle environment distribution $\hat{p}(e)$.
The simulated counterparts are denoted as $\hat{e}_{\text{sim},i}$. Given a
task specification $r$, a test environment ${\mathcal{E}}^{\text{test}}$ is
defined as a set of tuples:

$\mathcal{E}^{\text{test}}=\\{\left<\hat{e}_{\text{sim},i},r\right>\\}_{i=1,\cdots,n},$

where the generated behavior $\pi$ will be evaluated in.

Figure 2: Example of environment complexity: an overloaded and disorganized
real-world dishwasher.

Meanwhile, to prevent the behavior from overfitting to a few sample
environments within $\mathcal{E}^{\text{test}}$, one can maintain two sets of
sample environments; the test environment can be strictly held out from the
rest of the behavior generation pipeline, letting it serve the sole purpose of
behavior evaluation. A distinct set of sample environments then can be used to
effectively guide the remaining design choices. We call that a reference
environment, $\mathcal{E}^{\text{ref}}$, as illustrated in Figure 1.

######  Definition 2.2 (Reference Environment).

A Reference Environment $\mathcal{E}^{\text{ref}}$ is a distinct set of sample
environments that provides useful context to the shaping algorithm. Trained
behaviors will not be evaluated here to avoid overfitting.

For instance, when designing a robotic behavior for unloading a dishwasher,
sample environments would include multiple instances of dishwashers loaded
with varying configurations of dishes and utensils. Sampled configurations
then could be split into a set of reference environments for guiding the
shaping and a set of held-out test environments for evaluation. This diversity
in configurations provides the shaping algorithm with a broad context,
incentivizing it to infer the underlying distribution of object placements.
The goal is for the reinforcement learning (RL) trainer to sample from this
inferred distribution during training, ensuring that the generated behaviors
are robust and adaptable to various real-world scenarios without overfitting
to a specific set of environments.

###  2.2 Shaping Reference Environments

A straightforward subsequent step of behavior generation might be to directly
use the reference environments $\mathcal{E}^{\text{ref}}$ as an RL
environment, with the expectation that algorithms like Proximal Policy
Optimization (PPO) (schulman2017proximal) will find performant and
generalizable behaviors. However, this approach often falls short due to the
inherent sparsity of these environments.

Reference environments often present a challenging optimization landscape for
RL algorithms due to their sparse nature. For example, it might be rare to
obtain nonzero rewards or the observation space might be dominated by spurious
features that encourage poor local minima. To mitigate these challenges, human
engineers typically go through a process of shaping the elements of
$\mathcal{E}^{\text{ref}}$. This modification, aimed at enhancing the
learnability of the environment, includes introducing denser learning signals
and additional modifications encouraging effective exploration. The resulting
shaped environment (co2020ecological) then used as a training ground for the
subsequent optimal control solver.

######  Definition 2.3 (Shaped Environment).

A Shaped Environment ${\mathcal{E}}^{\text{shaped}}$ is a modification of a
reference environment, i.e.,
${\mathcal{E}^{\text{shaped}}}=f(\mathcal{E}^{\text{ref}})$. The
transformation $f$ incorporates design choices specifically optimized for
learning performance, smoothing the optimization landscape enabling the solver
to find better solutions with maximal performance in
$\mathcal{E}^{\text{test}}$. The transformation function

$f:\mathcal{E}^{\text{ref}}\rightarrow\mathcal{E}^{\text{shaped}}$ (1)

is defined as shaping function.

The shaping process for a robotics environment usually involves manually
explored design choices, including shaping the reward (ng1999policy;
singh2009rewards), modifying the action space (peng2017learning;
aljalbout2023role), designing curricula on the environment dynamics, initial
state distributions, and goal distributions (selfridge1985training;
matiisen2019teacher; portelas2020automatic; lee2020learning), crafting the
state space (yu2023identifying), and shaping the right failure conditions for
early termination (co2020ecological). We describe details of common shaping
operations in Sec 3.1.

###  2.3 RL Training

Once the shaped environment is obtained, the next step of behavior generation
is to use RL algorithms, i.e., PPO (schulman2017proximal), to find a behavior
$\pi$ that best performs on the shaped environment
$\mathcal{E}^{\text{shaped}}$. Formally, the algorithm aims to find an optimal
behavior $\pi$ for the following optimization problem:

$\displaystyle\max_{\pi}\mathbb{E}_{\tau\sim\pi}\left[\sum_{t=0}^{T}\gamma^{t}r_{t}(s_{t},a_{t}))\right]$
(2) $\displaystyle\leavevmode\nobreak\ \leavevmode\nobreak\ \text{s.t.
}\leavevmode\nobreak\ \leavevmode\nobreak\ s_{t+1}\sim
p(s_{t},a_{t};\mathcal{E}^{\text{shaped}}).$

While the RL training process also requires a range of design decisions, such
as algorithmic choices and hyperparameter adjustments, these areas are
relatively well-researched and documented (parker2022automated;
kiran2022hyperparameter).

However, we note that in a pratical context of robotic behavior generation,
tuning the RL setting (e.g. neural architecture search for policy or
hyperparameter tuning) is often underprioritized compared to the effort put
into environment shaping. In IsaacGymEnvs (makoviychuk2021isaac), for
instance, simple Multilayer Perceptron (MLP) networks are employed, and off-
the-shelf RL algorithm implementations are utilized with their default
configurations. This shows that algorithms like PPO and their default settings
are capable enough when paired with ideally shaped environment.

###  2.4 Optimizing Environment Shaping via Iterative Behavior Evaluation and
Reflection

Once an optimal behavior $\pi^{\star}$ is obtained via RL training, the
behavior is evaluated on the test environment $\mathcal{E}^{\text{test}}$ and
reflected by human engineers. Denoting the reflection process as $\mathcal{H}$
of analyzing the generated behavior $\pi^{\star}$ in test environment
$\mathcal{E}^{\text{test}}$ and coming up with a better environment shaping
$f$,

$\mathcal{H}:f_{k}\times
J(\pi_{k}^{\star};\mathcal{E}^{\text{test}})\rightarrow f_{k+1},$ (3)

robotic behavior generation process can be formally defined as an iterative
optimization process over the environment shaping function $f$,

$\displaystyle
f_{k+1}=\mathcal{H}\left(f_{k},J(\pi^{\star}_{k};\mathcal{E}^{\text{test}})\right)$
(4) $\displaystyle\text{where}\leavevmode\nobreak\ \leavevmode\nobreak\
\pi^{\star}_{k}=\underset{\pi}{\text{argmax}}\
J(\pi;\mathcal{E}^{\text{shaped}}_{k}),$ $\displaystyle\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\
\mathcal{E}^{\text{shaped}}_{k}=f_{k}(\mathcal{E}^{\text{ref}}),\leavevmode\nobreak\
\leavevmode\nobreak\ f_{0}=\mathbf{I}_{\text{identity}}$

which aims to find an optimal shaping function $f\in\mathcal{F}$ for the
following bi-level optimization problem:

$\displaystyle{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}\max_{f\in\mathcal{F}}}\leavevmode\nobreak\
J(\pi^{\star};\mathcal{E}^{\text{test}})$ (5)
$\displaystyle\leavevmode\nobreak\ \text{s.t.}\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\
\pi^{\star}\in\arg\max_{\pi}J({\pi};{\mathcal{E}^{\text{shaped}}}),\leavevmode\nobreak\
\leavevmode\nobreak\
{\mathcal{E}^{\text{shaped}}}={\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}f}(\mathcal{E}^{\text{ref}}).$

After shaping is applied, the new environment may not reflect the original
task; therefore, note that the outer level of the bilevel optimization
maximizes $J(\pi^{*};\mathcal{E}^{\mathrm{test}})$ which is the return
evaluated in the original test environments without any shaping. If the inner
level optimizes a shaped environment well, but with poor correspondence to the
original task, it will be dispreferred by the outer loop.

##  3 The Current State of Environment Shaping

Having established the role of environment shaping operations in successful
behavior generation, we now probe deeper into the unique challenges that the
problem of environment shaping and its optimization procedure presents.
Specifically, we make the following arguments with supporting experiments and
analysis:

  * •

Popular RL benchmark environments are artificially made easy for RL with task-
specific environment shaping. We should benchmark our algorithms in unshaped
environments if we want them to solve new problems without task-specific
environment shaping step. (Section 3.1)

  * •

Shaping multiple attributes of the environment at once (reward, observation
space, action space, etc.) is a tricky, non-convex optimization problem
(Section LABEL:sec:localoptima).

  * •

Reward shaping is not the only problem. Existing automation efforts focus too
narrowly on rewards (Section LABEL:sec:eval-eureka).

###  3.1 RL Benchmarks for Robotics are Artificially Easy

Benchmark environments for robot reinforcement learning include task-specific
environment shaping to make it feasible to help baseline RL algorithms perform
reasonably well. However, these modified environments might not fully assess
how RL algorithms progress towards solving various control problems
independently, treated as a black-box, without needing task-specific
adjustments.

In this section, we outline the common task-specific design choices associated
with different aspects of environment shaping, while formally defining what an
unshaped counterpart — one with minimal or no human engineering required —
would look like. We consider a case study of the IsaacGymEnvs task suite
(makoviychuk2021isaac).

AllegroHand | Reward | Change
---|---|---
all shaped | $38777$ | –
sparse reward | $0$ | $\downarrow 38777$
unshaped action space | $21530$ | $\downarrow 17247$
unshaped observation space | $2114$ | $\downarrow 36663$
no early termination | $0$ | $\downarrow 38777$
single initial state | $0$ | $\downarrow 38777$
single goal state | $141155$ | $\uparrow 102378$

Humanoid | Reward | Change
---|---|---
all shaped | $7554$ | –
sparse reward | $5237$ | $\downarrow 2317$
unshaped action space | $67$ | $\downarrow 7487$
unshaped observation space | $0$ | $\downarrow 7554$
no early termination | $705$ | $\downarrow 6849$
single initial state | $5735$ | $\downarrow 1819$

Anymal | Reward | Change
---|---|---
all shaped | $-45$ | –
sparse reward | $-2789$ | $\downarrow 2744$
unshaped action space | $-2499$ | $\downarrow 2454$
unshaped observation space | $-2656$ | $\downarrow 2611$
no early termination | $-43$ | $\uparrow 2$
single initial state | $-17$ | $\uparrow 28$
single goal state | $-2516$ | $\downarrow 2470$

Table 1: Impact of environment shaping on policy optimization. Removing task-
specific design choices in the reward, action space, state space, early
termination, or initialization incurs performance reductions. Top row:
environment with original shaped design choices. Each subsequent row shows
performance after training with a corresponding unshaped design choice. The
performance of all policies is evaluated in a fully unshaped environment.

⬇

1 def shaped_action_space(self):

2 # scale the targets by the joint limits

3 self.cur_targets = scale(self.actions,

4 self.shadow_hand_dof_lower_limits,

5 self.shadow_hand_dof_upper_limits)

6

7 # compute the moving average of targets

8 self.cur_targets = self.alpha * self.cur_targets +

9 (1 - self.alpha) * self.prev_targets

10 self.cur_targets = tensor_clamp(self.cur_targets,

11 self.shadow_hand_dof_lower_limits,

12 self.shadow_hand_dof_upper_limits)

13 self.prev_targets = self.cur_targets[:]

14

15 # compute the torques according to PD control law

16 torque = 3 * (self.cur_targets - self.shadow_hand_dof_pos) - 0.1 *
self.shadow_hand_dof_vel

17

18 # apply torques

19 self.gym.set_dof_actuation_force_tensor(self.sim,
gymtorch.unwrap_tensor(torque))\end{lstlisting}

20

21 \begin{lstlisting}[language=Python]

22 def unshaped_action_space(self):

23 # directly apply the prediction action as torques

24 self.gym.set_dof_actuation_force_tensor(self.sim,
gymtorch.unwrap_tensor(self.actions))\end{lstlisting}

25 % \end{codebox}

26 \vspace{-0.2cm}

27 \caption{Action space shaping: (Top) Original shaped action space with
task-specific features. (Bottom) Unshaped action space consisting of joint
torque commands. Some shaped code has been slightly modified from the source
to increase brevity and clarity while preserving the original logic.}

28 \label{fig:action_shaping_code}

29\end{figure}

30

31

32

33

34\begin{figure}[t] % ’h’ for ’here’, can be changed to ’t’, ’b’, etc.

35 % \begin{codebox}

36 \begin{lstlisting}

37 def shaped_observation_space(self):

38 root_states = rigid_bodies[:, self.root_handle]

39 base_quat = root_states[:, 3:7]

40 # base linear velocity (from local frame)

41 base_lin_vel = quat_rotate_inverse(base_quat, root_states[:, 7:10]) *
lin_vel_scale

42 # base angular velocity (from local frame)

43 base_ang_vel = quat_rotate_inverse(base_quat, root_states[:, 10:13]) *
ang_vel_scale

44 # transformed base orientation

45 projected_grav = quat_rotate(base_quat, gravity_vec)

46

47 # scaling/normalizing values

48 commands_scaled = scale(commands,

49 [lin_vel_scale, ang_vel_scale])

50 dof_pos_scaled = dof_pos_scale *

51 (dof_pos - default_dof_pos)

52 dof_vel_scaled = dof_vel_scale * dof_vel

53

54 # concatenate the relevant features

55 obs = torch.cat([base_lin_vel, base_ang_vel,

56 projected_grav, commands_scaled,

57 dof_pos_scaled, dof_vel_scaled,

58 actions], dim=-1)\end{lstlisting}

59 \begin{lstlisting}

60 def unshaped_observation_space(self):

61 # include entire unprocessed simulator state

62 obs = torch.cat([dof_pos,

63 dof_vel, torques,

64 rigid_bodies.reshape(num_envs, -1),

65 rigid_body_force.reshape(num_envs, -1),

66 commands, actions], dim = -1)\end{lstlisting}

67 % \end{codebox}

68 \vspace{-0.2cm}

69 \caption{State space shaping: (Top) Original shaped state space with task-
specific features. (Bottom) Unshaped state space contains the entire raw
simulator state.}

70 \label{fig:state_shaping_code}

71\end{figure}

72

73

74\textbf{Action Space. }~How would an \textit{unshaped} action space look,
and how \textit{shaped} is the action space in the case environments?

75

76For a typical robot in rigid multibody simulation, the unshaped action space
would be the motor torques: passing in the policy output $a$ directly to the
motor as a torque $\tau$ \--- no scaling or transforming the outputs, no gains
to be tuned.

77

78Designing a shaped action space for a robot is thus equivalent to the
problem of choosing a low-level controller (and its corresponding parameters)
that converts a policy output $\mathbf{a}$ with different physical meanings
into an executable motor torque $\tau$ that can drive the actuators. The low-
level controller can implement a prior like whether to resist or comply to
external forces. Examples include proportional-derivative control,
differential inverse kinematics controller, operational space control
\cite{khatib1985operational}, or impedance control \cite{hogan1984impedance}.
These are projections of the policy outputs onto a strict subset of the space
of possible torque commands. The choice of action space can significantly
influence the performance of learning algorithms in the
environment~\cite{peng2017learning}.

79

80In IsaacGymEnvs, there are multiple distinct cases of shaped action spaces.
\texttt{AllegroHand} uses joint position targets with moving average smoothing
as its action space (See Figure \ref{fig:action_shaping_code} for example
code). For \texttt{Anymal}, the relative joint position target is used as an
action space with proportional-derivative controller. For \texttt{Humanoid},
scaled joint torques are used. Table \ref{table:shaping-matters} empirically
shows that removing these \textit{shaping} operations on action space largely
impacts the training performance.

81

82\textbf{Observation Space.}~ What is an \textit{unshaped} observation space,
and how did IsaacGymEnvs \textit{shape} the observation?

83

84Crafting a \textit{shaped} observation space involves strategic selection
and transformation of variables from an unshaped observation \--- the entire
state in simulation that’s been exposed to the user. The process, integral to
optimizing policy learning performance, includes (a) useful transformation of
the unshaped variables (i.e., features) and (b) discarding unnecessary
variables. Figure \ref{fig:state_shaping_code} shows the details of the shaped
state space in the IsaacGymEnvs \texttt{Anymal} environment.

85

86Different choices of observation space shaping can substantially impact
robot performance. In the setting of quadrupedal locomotion,
\citet{yu2023identifying} found that different aspects of the task
(robustness, performance) are sensitive to different features included in the
observation space. Table \ref{table:shaping-matters} shows our findings
agreeing that observation shaping has a significant impact on training
performance. For \texttt{Humanoid}, for instance, using an unshaped
observation space makes the task completely unlearnable.

87

88\textbf{Reward. }~The \textit{unshaped} reward represents the true objective
function we actually wish to optimize, often defined as {extrinsic} reward
\cite{singh2010intrinsically, singh2009rewards} or task-fitness function
\cite{niekum2010genetic}.

89

90Shaping the \textit{extrinsic} reward into a more informative
\textit{intrinsic} reward to make it more conducive to RL algorithms is a
well-studied topic in the community \cite{ng1999policy,
singh2010intrinsically, gupta2022unpacking, eschmann2021reward}. The shaping
procedure is usally designed to provide \textit{denser} learning signals and
to prevent the policy from overly exploiting the innate vagueness of extrinsic
reward, i.e., reward hacking. It typically involves a few commonly used
strategies. One popular strategy is to reward \textit{progress} towards the
goal by adding distance metrics measuring how close the agent is to reaching
it. If some attributes of successful behavior are known beforehand, those
terms can be added to the shaped reward \cite{chen2020learning,
margolis2023walk}. If a trajectory of a successful behavior is known, one can
also compute a \textit{similarity} between the generated behavior and the
trajectory and try to maximize the similarity \cite{peng2021amp}. Offering
bonuses for exploring new states \cite{pathak2017curiosity} is also known to
facilitate RL training. Table \ref{table:shaping-matters} shows that
\textit{shaping} reward has significant impact on the training performance.

91

92It’s worth noting, however, that defining the extrinsic reward itself poses
its own set of challenges, which is well addressed in \cite{task-
specification-problem}. The task fundamentally requires translating human
intentions and goals into numerical values that an algorithm can optimize,
often posing significant challenges. In the scope of this position paper, we
presume the task has been defined and consider the challenge of learning a
policy to perform it.

93

94\textbf{Initial/Goal State.} Let’s consider a concrete example: how would
you \textit{shape} an initial state of a dishwasher to make the behavior
robust under the chaotic randomness typically exhibited in real-world (Figure
\ref{dishwasher})? Where would you even start shaping things from? How would
an \textit{unshaped} counterpart look like for initial/goal state shaping?

95

96A reasonable \textit{unshaped} intial/goal state to start with are the
nominal states defined in \textit{reference} environment \-- manually designed
sample environment with every actors (robots and assets) staying in its
nominal pose. We often assume such nominal pose to be a mean of its underlying
distribution, commonly assumed as Gaussian or Uniform distribution. This is
indeed the most simple yet common technique of designing initial state
distribution for many robotics tasks \-- we just randomly perturb robot joints
and assets around its nominal (reference) pose!

97

98When the task gets more complex, however, this simple approach starts to
break quickly. Imagine randomly perturbing a single nominal state of a
dishwasher, or randomly perturbing the nominal pose of a quadruped standing on
a rough terrain; dishes and ladles, feets and terrain will be in penetration
most of the time. Doing rejection sampling can be a stopgap solution, but it
might end up rejecting most of the samples, making the approach nearly
unusable.

99

100In practice, robotics engineers thus take a clever, but heavily heurstic,
task-dependent approach to shape initial states. To randomly initialize a
quadruped on a rough terrain, for instance, people make the robot walk off a
small region of flat ground ~\cite{rudin2022learning}, letting the simulation
engine figure out the phsyics constraints. To generate random initial states
for cluttered bin-picking tasks, we often drop objects from height in random
order. To obtain initial states to train a fall-recovery policy for humanoid,
we drop the humanoid from height \cite{peng2021amp}. For in-hand manipulation
tasks, to obtain a diverse set of downward-facing initial grasps to start
with, we first train a grasping policy and execute it to generate diverse
initial starting configurations that does not drop the object immediately
\cite{chen2022system}. The task-specific nature of such strategies poses
challenges in automating this shaping operation.

101

102Moreover, designing \textit{how} we sample from the shaped initial/goal
distribution can naturally set up a learning curriculum that progresses from
simpler to more challenging tasks. Take the example of training a quadruped to
run at high-speed. Training only with high-speed commands could hinder the
learning process. Instead, shaping a goal distribution to be a wider
distribution then the actual target speed, and designing the sampling strategy
to begin with slower ones and gradually increasing, can facilitate more
effective learning of fast running behaviors~\cite{margolis2022rapid}.

103

104

105\textbf{Terminal Condition.} ~A strict definition of \textit{unshaped}
terminal condition might be to never terminate, resembling how the real world
never terminates and never gets a chance to reset from the beginning. However,
since this poses quite challenging problem for most learning algorithms,

106one more common choice of \textit{unshaped} terminal condition that is
fairly environment agnostic is to set a predefined episode time limit.

107

108\textit{Shaping} the terminal condition thus corresponds to deploying
strategies of \textit{early termination}, which is known to significantly
improve training performance~\cite{co2020ecological}. In {IsaacGymEnvs},
\texttt{Anymal} terminates when non-foot bodies contact the ground,
\texttt{Humanoid} terminates when the torso falls below a certain height, and

109\texttt{AllegroHand} terminates when the object falls off from the hand.

110Table \ref{table:shaping-matters} shows that early termination
significantly improves the training performance in all three tasks.

111

112\subsection{Shaping the Entire Environment is Harder than Shaping One
Component}

113\label{sec:localoptima}

114

115In the previous section, we saw that policy learning can be highly
sensitive to each individual aspect of environment shaping (Table
\ref{table:shaping-matters}) and that the shaping choices in IsaacGymEnvs vary
qualitatively depending on the task. This motivates that, to promote the
development of truly automatic behavior generation, we should consider
benchmarking against a suite of unshaped environments. If an algorithm can
learn policies in unshaped environments, it should be applicable to newly
defined tasks and environments without requiring additional manual shaping on
those environments. The unshaped environments are an appropriate benchmark for
methods that couple automatic environment shaping with RL, or to directly
attempt to solve by improving RL algorithms.

116

117What kinds of methods might we try using to optimize an entirely unshaped
environment? If we aim to accomplish this by introducing shaping, we may have
to search over not just one aspect but all aspects of shaping. To understand
the optimization landscape, suppose we have access to an oracle that proposes
the human-designed environment shaping as a candidate, and we use this to
perform a hill-climbing search in one aspect of the environment at a time:
first, optimize the observation space, then the action space, then the reward
function, and repeat until convergence.

118Figure \ref{fig:local_optima} illustrates the consequence of this strategy
in three environments from {IsaacGymEnvs}. Each node represents a shaped
version of the environment. The node’s color indicates the trained policy’s
performance on the shared test environment (unshaped). Edges indicate
environments that differ by one type of change (state space, reward, etc.),
and the bold arrows indicate the path taken by decoupled hill-climbing on one
change at a time. We found that each environment has multiple local maxima
where the hill climbing would get stuck in a suboptimal configuration. These
local maxima differ in at least two (as many as four!) types of shaping. This
suggests that the environment shaping design space is non-convex in the
different types of shaping. Therefore, we should aim to develop techniques
that consider shaping all parts of the environment jointly.

119

120\begin{figure}[t!]

121\centering

122

123\includegraphics[width=0.95\linewidth]{images/ConfigGraphBothOverlayAll.pdf}

124\vspace{-0.5cm}

125

126\caption{\textbf{Local optima in environment shaping problems.}

127Each node represents a shaped training environment. Edges connect
environments that are separated by modifying one type of shaping (action
space, state space, reward function, initial state, goal, or terminal
condition). Bold arrows represent optimal choices for hill climbing. Each
environment is shown to have multiple local optima corresponding to the top
row of nodes.

128}

129\label{fig:local_optima}

130\end{figure}

131

132

133\subsection{Existing Automation Focuses Narrowly on Rewards}

134\label{sec:eval-eureka}

135

136

137There have been prior attempts to partially automate the process of
environment shaping.

138However, most of the efforts have focused on the subtask of reward shaping.

139The concept of reward shaping was formalized by~\cite{ng1999policy} and the
idea of automated reward shaping through evolutionary search was advanced
by~\citet{singh2009rewards, singh2010intrinsically}. Recent works have
formulated the problem of reward shaping as bilevel optimization with the
environment design at the top level and policy learning at the bottom level
and used LLMs~\cite{goyal2019using, ma2023eureka, xie2023text2reward} or
gradient-based methods~\cite{hu2020learning} to generate candidates.

140

141LLM-based methods can benefit from prior knowledge about coding and
robotics derived from internet training data to act as an efficient sampler
for generating candidate shaped rewards expressed as code. Since other aspects
of the environment shaping can also be expressed as code, it is
straightforward to test how these methods extend. We evaluated the LLM-based
reward shaping algorithm Eureka~\cite{ma2023eureka} at the task of designing
the observation and action space for the \texttt{Anymal} IsaacGymEnvs
environment. As in the original application to reward shaping, we generated
five generations of 16 parallel candidate shaping functions using GPT-4,
including the best performing candidate from each generation in the prompt for
the next generation. Eureka succeeded in designing effective observation and
action spaces for this environment (Table \ref{table:eureka-eval}) while all
other aspects of the environment were pre-shaped.

142

143Motivated by Section \ref{sec:localoptima}, we also tested whether Eureka
can jointly optimize multiple aspects of the environment. We found that
performance drastically drops when Eureka is tasked to jointly optimize
multiple environment aspects, i.e., shaping both the reward and observation,
even though it could optimize each individually. This suggests that shaping
multiple environment aspects jointly is yet an open problem.

144

145\begin{table*}[]

146\footnotesize

147\centering

148\begin{tabular}{@{}rc|c|c@{}}

149\toprule

150shaping component & \begin{tabular}[c]{@{}c@{}}Eureka\\\
\cite{ma2023eureka}\end{tabular} & \begin{tabular}[c]{@{}c@{}}Human Design\\\
\cite{makoviychuk2021isaac}\end{tabular} &
\begin{tabular}[c]{@{}l@{}}Automation\\\ Performance\end{tabular} \\\ \midrule

151$^\star$reward & $0.986$ & $0.973$ & $\uparrow 0.013$ \\\

152$^\dag$observation & $0.967$ & $0.973$ & $\downarrow 0.006$ \\\

153$^\dag$action & $0.982$ & $0.973$ & $\uparrow 0.009$ \\\

154$^\circ$reward $\times$ observation & $0.196$ & $0.973$ & $\downarrow
0.777$ \\\

155$^\circ$reward $\times$ action & $0.536$ & $0.973$ & $\downarrow 0.437$ \\\

156% $^\circ$reward $\rightarrow$ action & & $0.973$ & \\\

157% $^\circ$action $\rightarrow$ reward & & $0.973$ & \\\

158

159$^\circ$reward $\times$ observation $\times$ action & N/A & $0.973$ & N/A
\\\

160\bottomrule

161\end{tabular}

162\caption{Evaluating \cite{ma2023eureka} for $^\star$reward shaping,
$^\dag$shaping different components, and $^\circ$coupled shaping. {\small
\texttt{Anymal}} task \cite{makoviychuk2021isaac} is used as an environment.
Automation performance is measured by the relative gain obtained by automating
the shaping process, compared to design choices made by humans
\cite{makoviychuk2021isaac}. Behavior is evaluated with the following task
specification: $r = \exp\left ( {\text{negative\\_distance\\_to\\_command}}
\right )$. 5 iterations of outer loop for Eureka. Used latest GPT-4 model.
Note that GPT-4 failed to generate working code for results labeled N/A. }

163\label{table:eureka-eval}

164\end{table*}

165

166\label{sec:existingtools}

167

168

169\section{Paths Forward to Automated Environment Shaping}

170

171Today, no algorithm can solve diverse unshaped tasks, although we know them
to be solvable with environment shaping by human designers. How can we fix
this and generalize successes in RL for robotics? There are a few
possibilities:

172

173 \textbf{Scale up computation.} ~Existing bi-level search techniques like
\citet{ma2023eureka} can be extended to design environment shaping and run
with more compute resources to search over a greater number of candidate
shaping designs. Massively parallel simulation has led to realistic robotics
environments that can train quickly~\cite{makoviychuk2021isaac,
rudin2022learning}; However, some state-of-the-art sim-to-real methods take
weeks to train a single policy due to high data requirements or expensive
subroutines~\cite{chen2023visual, jenelten2024dtc}. This would make performing
much more outer-loop search impractical.

174

175

176 \textbf{Improve priors.} ~If we can’t search over more candidates, a
better way is to generate higher-quality candidates more quickly. One
possibility is that improved foundation models will zero-shot generate better
candidate shaped environments~\cite{xian2023towards, yang2023holodeck}.
However, it’s hard to predict how much this will help. Another good idea is to
\textit{mechanistically understand the strategies of human designers}. By what
mechanism does the choice of observation space or curriculum improve
performance? \citet{co2020ecological} proposed a holistic view of environment
shaping (‘Ecological Reinforcement Learning’) and studied the impact of
stochasticity, goal distribution, and early termination design on
learnability. \citet{yu2023identifying} examined the impact of observation
space on learning quadruped locomotion and offered an explanation of how
different observations can work better for different subtasks.
\citet{esser2023guided} surveyed a number of works across applied
reinforcement learning, comparing practitioners’ design choices.
\citet{peng2017learning} analyzed a set of common action spaces for robot
control and \citet{aljalbout2023role} developed metrics to explain the
performance gap. \citet{kim2023not} found that constraints require less tuning
than rewards to transfer across diverse robots. If more understanding is
documented in these areas, improved biases could be implemented as a prior,
e.g., in an LLM’s context.

177

178

179 \textbf{Shape online.} ~Instead of iteratively improving on the
environment shaping $f$ across training runs, can we improve it dynamically
within a RL training loop? If, for example, multi-objective reinforcement
learning or bilevel optimization can trade off the coefficients of multiple
reward terms~\cite{singh2010intrinsically, zheng2018learning,
chen2022redeeming}, then perhaps the LLM search can be performed only over the
form of the different reward terms, and their relative weighting can be
optimized at runtime. In other aspects of the environment, an analogous
approach would be to generate a parameterized shaping operator and
automatically tune its parameters. For example, the scale of the observation
or the termination height could be assigned as parameters for online
optimization.

180

181

182\textbf{A Robotics Benchmark for Environment Shaping.}

183Instead of evaluating RL algorithms on environments pre-shaped to work with
PPO, the community should evaluate them on unshaped environments.

184These environments will be \textit{too hard} to solve with existing RL
algorithms. Thus, it will be necessary to (a) study how to modify aspects of
the environment to make it efficiently solvable with RL and / or (b) develop
better RL algorithms that can handle such challenging sparse environments.

185

186We modified the environments in IsaacGymEnvs to use unshaped design choices
to provide a good proxy for the task-agnostic output from procedural
environment generators. To facilitate environment shaping research, our code
exposes an API for modifying the environment code, which allows the optimizer
to transform the reward, observation space, action space, etc. by editing
Python functions at runtime. The API is designed so that any language models
can be easily integrated to perform such transformations. Our implementation
also facilitates faster evaluation of multiple environment shaping choices by
training multiple policies in a single process, leveraging parallel
simulation.

187

188

189\section{Conclusion}

190

191Reinforcement learning has long promised to solve decision-making problems
in a task-agnostic manner. It has found great success in solving challenging
but narrowly-scoped tasks in robotics. In this position paper, we argued that
the key bottleneck for scalability of RL is a limited mechanistic
understanding of task-specific engineering (\textit{environment shaping}) that
transforms environments to be solved more easily and is universal across
domains and benchmarks. We proposed a formal definition of environment shaping
as an optimization problem and identified instances of shaping in robotic
tasks. Finally, we identified key steps forward such as developing
computationally efficient search over environment shaping; improving our tools
for implementing such shaping and understanding its impact on learning
dynamics; and defining benchmarks for this problem. We hope this will motivate
an increased focus in RL research on communicating and evaluating environment-
shaping measures that impact performance rather than solely emphasizing the
impact of the learning algorithm or policy architecture.

192

193\section*{Acknowledgements}

194

195We thank the members of the Improbable AI lab for the helpful discussions
and feedback on the paper. We are grateful to MIT Supercloud and the Lincoln
Laboratory Supercomputing Center for providing HPC resources. This research
was partly supported by Hyundai Motor Company, DARPA Machine Common Sense
Program, the MIT-IBM Watson AI Lab, and the National Science Foundation under
Cooperative Agreement PHY-2019786 (The NSF AI Institute for Artificial
Intelligence and Fundamental Interactions, http://iaifi.org/). We acknowledge
support from ONR MURI under grant number N00014-22-1-2740. Research was
sponsored by the Army Research Office and was accomplished under Grant Number
W911NF-21-1-0328. The views and conclusions contained in this document are
those of the authors and should not be interpreted as representing the
official policies, either expressed or implied, of the Army Research Office or
the U.S. Government. The U.S. Government is authorized to reproduce and
distribute reprints for Government purposes notwithstanding any copyright
notation herein.

196

197\section*{Impact Statement}

198

199This paper presents work whose goal is to advance the field of

200Machine Learning. There are many potential societal consequences

201of our work, none which we feel must be specifically highlighted here.

202

203\bibliography{example_paper}

204\bibliographystyle{icml2024}

205

206

207\end{document}