# Unveiling the Significance of Toddler-Inspired Reward Transition
in Goal-Oriented Reinforcement Learning

Junseok Park1, Yoonsung Kim1111Equal Contributions., Hee Bin Yoo1*, Min Whoo
Lee1, Kibeom Kim1, Won-Seok Choi1, Minsu Lee1,2222Corresponding Authors.,
Byoung-Tak Zhang1,2,†

###### Abstract

Toddlers evolve from free exploration with sparse feedback to exploiting prior
experiences for goal-directed learning with denser rewards. Drawing
inspiration from this Toddler-Inspired Reward Transition, we set out to
explore the implications of varying reward transitions when incorporated into
Reinforcement Learning (RL) tasks. Central to our inquiry is the transition
from sparse to potential-based dense rewards, which share optimal strategies
regardless of reward changes. Through various experiments, including those in
egocentric navigation and robotic arm manipulation tasks, we found that proper
reward transitions significantly influence sample efficiency and success
rates. Of particular note is the efficacy of the toddler-inspired Sparse-to-
Dense (S2D) transition. Beyond these performance metrics, using Cross-Density
Visualizer technique, we observed that transitions, especially the S2D, smooth
the policy loss landscape, promoting wide minima that enhance generalization
in RL models.

## Introduction

In early years, toddlers behave much like exploratory agents. Throughout their
development, they interact with their surroundings without much prior
knowledge, akin to someone embarking on new experiences without expecting
immediate rewards (Oudeyer and Smith 2016). As they grow, toddlers transition
from free exploration to more goal-directed learning, aiming for specific
goals, resembling someone working towards known rewards for their efforts
(Gopnik, Meltzoff, and Kuhl 1999; Gibson 1988; Piaget, Cook et al. 1952;
Gopnik et al. 2017) as illustrated in Figure 1.

This learning pattern in toddlers can be incorporated in Reinforcement
Learning (RL), as illustrated in Figure 1-(a). In RL, agents learn by
interacting with their environment and receiving feedback, much like how
toddlers learn from their interactions. Similar to toddlers, agents must
navigate towards positive feedback they receive, which can be infrequent
(sparse) or detailed (dense). Sparse feedback might mean that the agent
requires more attempts to figure out the desired behavior due to limited
guidelines (Andrychowicz et al. 2020; Knox et al. 2023). Meanwhile, dense
feedback can guide the agent faster but might inadvertently focus them on
immediate outcomes, missing out on the bigger picture or long-term strategies
(Laud 2004).

Given these intricacies, simply sticking to one type of feedback might not
capture the essence of learning. Drawing inspiration from toddler
developmental stages, blending both feedback types could provide richer
insights. The transition toddlers make from free exploration – akin to sparse
feedback – to specific, goal-driven learning – similar to dense feedback in RL
– offers a unique perspective, as shown in Figure 1-(a). With this
perspective, our paper addresses the following question: “How does reward
transition that proceeds in a sparse-to-dense manner, inspired by toddler
learning, affect the learning of agents?” Through a series of experiments,
including egocentric navigation and robotic arm manipulation tasks, we aim to
explore the Toddler-Inspired Sparse to Potential-based Dense (S2D) Reward
Transition. Our goal is not only to explore its efficacy but also to delve
into the underpinning reasons by analyzing its comparative advantages against
other rewards or prevalent strategies in the field of RL.

Figure 1: Parallel learning trajectories: toddlers and agents. (a) The figure
compares a toddler’s learning journey with an agent’s. On the left, a toddler
freely explores the apple, symbolizing sparse reward learning. As we
transition right, the toddler’s focus on specific tasks reflects goal-guided
learning. Similarly, the agent’s progression from sparse to potential-based
dense rewards is charted above, highlighting parallels in learning evolution.
(b) As reward transitions occur, the depth of local minima reduces, leading to
a wide minima via the smoothing effect, thereby enhancing more generalization.

Taking the concept of “reward transition in learning” further, we consider
visualizing the learning parameters as a topographical map. One type of such
landscape representation, a policy loss landscape provides an intuitive
visualization of learning dynamics in RL (Li et al. 2018). On this map, each
point corresponds to a set of learning parameters, and its altitude signifies
the loss value. As in any landscape, some areas are rugged, featuring steep
mountains or deep valleys, indicating challenging learning regions.
Understanding the notion of smoothness of loss landscape is vital in neural
network-based learning. Smooth terrains in this landscape, devoid of abrupt
pitfalls, enable quicker and more reliable convergence through gradient
descent.

Critically, smoother landscapes often promote wide minima, which are
associated with better generalization of learned policies to novel situations
in dynamic environments (Keskar et al. 2017). Empirically, we observed that
employing the Sparse-to-Dense (S2D) Reward Transition made this terrain
smoother by reducing the depth of local minima, as illustrated in Figure
1-(b).

Our study contributes to a deeper understanding of the intricate balance
between exploration and exploitation and provides insight into designing
reward structures in RL. By emulating the learning processes observed in
toddlers, we hope to bridge the gap between biological and artificial learning
mechanisms. Based on this, we have offered a novel perspective in applying and
addressing this synthesis for more robust, adaptable, and efficient RL
systems.

The main contributions of this paper can be summarized as follows: (1) We
observed that the Toddler-Inspired Reward Transition enhances success rates,
sample efficiency, and generalization within goal-conditioned RL. (2) Our
experimental analyses support that such transition has smoothing effects on
the policy loss landscape and promotes wide minima, corroborating the
performance improvements. (3) Our findings highlight the potential of
biologically inspired approaches in providing clues for exploration-
exploitation tradeoff and reward shaping challenges.

## Related Work

#### Toddler-inspired learning.

Drawing insights from toddler developmental stages has provided a fresh
perspective in advancing deep learning. By harnessing the innate exploratory
tendencies and distinctive learning mechanisms of toddlers, researchers have
refined both supervised and reinforcement learning techniques. For instance,
classifiers trained on datasets of toddlers’ perspectives on objects
outperformed those using adults’ perspectives (Bambach et al. 2018),
underscoring the potential of leveraging toddlers’ exploration mechanism.
Similarly, critical learning phases in toddlers have counterparts in RL (Park
et al. 2021; De Kleijn, Sen, and Kachergis 2022) and deep networks (Achille,
Rovere, and Soatto 2018). Such toddler-inspired methodologies emphasize the
alignment between toddler growth and AI model evolution, underscoring the
potential of biological insights in driving AI forward.

#### Exploration-exploitation in deep RL.

Balancing exploration with exploitation is an inherent challenge of RL (Ladosz
et al. 2022). Moreover, deep RL intensifies this complexity with high-
dimensional spaces, like raw image inputs, where pixels are unique dimensions.
To mitigate this, many algorithms favor exploration, utilizing tools like
intrinsic motivation (Badia et al. 2020; Aubret, Matignon, and Hassas 2019;
Pathak et al. 2017). Drawing from toddlers’ learning, we observe the
transition from free exploration to specific exploitation based on collected
previous experience. Our approach offers a fresh take on RL’s exploration-
exploitation dilemma without introducing algorithmic complexities.

#### Curriculum learning.

Curriculum Learning (CL), inspired by curricula in human’s education, has been
known to boost performance, training speed (Hacohen and Weinshall 2019), and
safety (Turchetta et al. 2020) in machine learning. CL’s progression from
simple to complex tasks promotes generalization and convergence (Bengio et al.
2009; Weinshall, Cohen, and Amir 2018) in both supervised and reinforcement
learning (Florensa et al. 2018; Graves et al. 2017; Narvekar and Stone 2020).
Unlike the studies that initially restrict the diversity to easy tasks
(Kalantidis et al. 2020; Du et al. 2021; Dong, Gong, and Zhu 2017), several
studies (Zhang 1994; MacKay 1992) advocate a general-to-specific approach,
where agent initially collects varied learning experiences and then exploits
these experiences later in the curriculum. Adapting the S2D transition
observed in toddlers into RL, we embrace this philosophy in reward transition
for goal-oriented tasks.

#### Potential-based reward shaping (PBRS).

In RL, maximizing cumulative rewards guides agent behavior. However, due to
the inherent challenges in designing optimal reward functions for various
tasks, it often necessitates a process known as reward engineering. Reward
Shaping (RS) is one such technique that enhances training by supplementing the
environment’s feedback (Taylor and Stone 2009). Particularly, in environments
where the reward changes, an additional reward from a potential function is
employed to ensure the agent’s optimal strategy remains unaffected (Ng,
Harada, and Russell 1999b). Commonly, such shaped rewards are consistently
applied throughout training. Unlike this, we explore the Toddler-Inspired
Reward Transition, focusing on the effects of changing the density of rewards.

Figure 2: Overview of the baseline rewards. The S2D presents reward inspired
by toddler learning. In sparse rewards, agents are rewarded upon reaching the
target. For potential-based dense rewards, they get an extra reward determined
by the distance to a specific unit from the object.

## Preliminaries

#### Reinforcement learning.

RL is a field of machine learning in which the agent learns through trial and
error, similar to how humans acquire skills. It is applied to various tasks
that involve sequential decision making. A widely used formulation of RL
problem, Markov Decision Process (MDP) is defined as
$\langle\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},\gamma\rangle$, where
$\mathcal{S}$ is a set of environment states, $\mathcal{A}$ is a set of
possible actions,
$\mathcal{P}:\mathcal{S}\times\mathcal{A}\rightarrow\Delta(\mathcal{S})$ is a
transition probability distribution,
$\mathcal{R}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ is a reward
function and $\gamma$ is a discount factor. At every time step
$t\in\mathbb{N}$, the agent in the current state $s_{t}\in\mathcal{S}$
performs the action $a_{t}\in\mathcal{A}$ according to a policy
$\pi(\cdot|s_{t})$, and receives the next state
$s_{t+1}\sim\mathcal{P}(\cdot|s_{t},a_{t})$ and reward
$\mathcal{R}(s_{t},a_{t})$. RL algorithms aim to obtain an optimal policy
$\pi^{*}\in\Pi^{*}$ that maximizes the expected cumulative rewards
$R=\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}\mathcal{R}\left(s_{t},a_{t}\right)\right]$
with $\gamma$ applied, where $\Pi^{*}$ is the set of optimal policies.

#### Curriculum learning.

Curriculum Learning (CL) is a strategy to train an ML model using tasks that
gradually increase in difficulty. In RL, CL can be formulated as a learning
framework where the agent is trained on a sequentially changing series of MDPs
$\mathcal{M}_{i}=\langle\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R}_{i},\gamma\rangle$.

######  Definition 1 (Curriculum)

Let $\mathcal{M}_{1},\mathcal{M}_{2},\cdots,\mathcal{M}_{N}$ be a sequence of
MDPs
$\mathcal{M}_{i}=\langle\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R}_{i},\gamma\rangle$,
and $\mathcal{T}=(T_{1},T_{2},\cdots,T_{N-1})\in\mathbb{N}^{N-1}$. A
curriculum is a tuple
$\mathscr{C}=(\\{\mathcal{M}_{i}\\}_{i=1}^{N},\mathcal{T})$ where the agent is
trained on $\mathcal{M}_{I(t;\mathcal{T})}$ at training step $t$. Stage
indicator $I(t;\mathcal{T})$ is defined as:

$I(t;\mathcal{T}):=i,\quad t\in\left[T_{i-1},T_{i}\right)$

for each stage $i\in\\{1,\cdots,N\\}$, where $T_{0}:=0$ and $T_{N}:=\infty$.
We call $\mathcal{T}=(T_{1},T_{2},\cdots,T_{N-1})$ the stage transitions.

CL also boosts the training by arranging the tasks as “general-to-specific,”
where the agent is provided rewards with monotonically increasing densities
over training. Formally, we say an environment is sparse when only a small
portion of the state space is included in $\text{supp}(\mathcal{R})$. That is,

$\displaystyle|\text{supp}(\mathcal{R})|\ll|\mathcal{S}|,\,\,\,\text{where}$
$\displaystyle\text{supp}(\mathcal{R})$
$\displaystyle=\\{s\in\mathcal{S}\mid\exists
a\in\mathcal{A}\,\,\,s.t.\,\,\,\mathcal{R}(s,a)\neq 0\\}.$

$\text{supp}(\mathcal{R})$ means the region supported by reward function
$\mathcal{R}$ which has non-zero reward for some actions.

#### Wide minima phenomenon and loss landscape.

Deep neural networks traverse a high-dimensional loss landscape, with altitude
indicating the loss for specific parameters (Li et al. 2018). The aim is to
find the minima. In wide minima, due to broad gradients, gradient descent is
more likely to converge smoothly to global minima. This fosters robustness and
superior generalization to new data (Keskar et al. 2016). Conversely, in sharp
minima, steep gradients can trap models in local minima, resulting in
overfitting and poor generalization across diverse data distributions
(Goodfellow, Vinyals, and Saxe 2014). Empirically, models within wide minima
demonstrate better performance and generalization than those in sharp minima
(Keskar et al. 2017; Jastrzębski et al. 2018). In deep RL as well, where the
distribution of agent’s experiences may slightly vary every time step,
policies in wide minima could improve in generalization.

##  Toddler-Inspired Reward Transition

To conduct our experiments, we need to formulate and design a toddler-inspired
reward transition in RL, emulating the toddler reward transition paradigm.
Furthermore, we analyze the influence of this reward transition on the
learning behavior of agents, focusing on the policy loss landscape and the
wide minima phenomenon.

### Toddler-Inspired Sparse to Potential-based Dense Reward Curriculum

In this paper, we harness curriculum learning from the perspective of
encouraging exploration-to-exploitation as a Sparse to potential-based Dense
(S2D) reward transition curriculum and explain how this learning mechanism can
benefit RL. We define
$\mathscr{C}=(\\{\mathcal{M}_{i}\\}_{i=1}^{N},\mathcal{T})$ as an
S2D-curriculum if reward functions of MDPs
$(\mathcal{M}_{1},\mathcal{M}_{2},\cdots,\mathcal{M}_{N})$ become
progressively denser while preserving optimality.

######  Definition 2 (Toddler-inspired S2D-curriculum)

A curriculum $\mathscr{C}=(\\{\mathcal{M}_{i}\\}_{i=1}^{N},\mathcal{T})$ with
MDPs $\\{\mathcal{M}_{i}\\}_{i=1}^{N}$ is an S2D-curriculum if following
conditions are satisfied:

$\mathrm{supp}(\mathcal{R}_{1})\subseteq\mathrm{supp}(\mathcal{R}_{2})\subseteq\cdots\subseteq\mathrm{supp}(\mathcal{R}_{N})$
(1) $\Pi^{*}_{1}\supseteq\Pi^{*}_{2}\supseteq\cdots\supseteq\Pi^{*}_{N},$ (2)

$\Pi_{i}^{*}$ is a set of optimal policies with MDP $\mathcal{M}_{i}$. Here,
we call $\\{\mathcal{R}_{i}\\}_{i=1}^{N}$ a guidance of the curriculum
$\mathscr{C}$.

The first condition in Equation 1 denotes that the reward function becomes
denser, _i.e._ , the guidance becomes more explicit. The second condition in
Equation 2 constrains the optimality to be preserved during the transition of
the MDPs, _i.e._ , the optimal policies of $\mathcal{M}_{i}$ are also optimal
in $\mathcal{M}_{i+1}$. At a high level, the sequence of MDPs in the above
definition is S2D-curricular in the sense that the reward functions are
arranged in a “sparse-to-dense” order.

### Visualizing Post-Transition 3D Policy Loss Landscape: Cross-Density
Visualizer

As seen in Figure 6 and Appendix B, our study delves into the effect of the
S2D transition on policy loss landscape, reflecting toddlers’ cognitive
evolution (Gopnik et al. 2017; Piaget, Cook et al. 1952). Following (Li et al.
2018), we visualize the policy loss landscapes using grids of parameters
$\tilde{\theta}=\theta+\alpha\mathbf{x}+\beta\mathbf{y}$. Here, $\theta$
signifies current parameters and $\alpha$, $\beta$ are normalized coordinates.
Vectors $\mathbf{x}$ and $\mathbf{y}$ that form the axes arise from two
specific perturbations in the network’s parameter space. $\mathbf{x}$ and
$\mathbf{y}$ are made unit vectors by a normalized filter for consistent
scaling and clarity. The Z-axis captures policy loss, averaged over a batch of
transitions from the replay buffer. We are not concerned with the altitude and
which landscape is above or below another, because the two landscapes
correspond to separate network parameters, each of them having its own loss
range according to its current learning progress.

Noting the lack of visualization techniques in prior studies for policy loss
landscapes based on reward transitions, we design the Cross-Density
Visualizer. This method portrays the 3D policy loss landscape during
transitions from purely sparse or dense rewards to mixed-reward settings.
Thus, one set includes Sparse-to-Dense (S2D) and Sparse-to-Sparse (Only
Sparse), while the other contains Dense-to-Sparse (D2S) and Dense-to-Dense
(Only Dense). Our representations, displayed in Figure 6 and expanded upon in
Appendix B, highlight comparable smoothing effects particularly in the S2D
model.

### Exploring Minima Sharpness After Reward Transitions

Observing a reduction in the depth of local minima due to smoothing effects
led us to hypothesize that the S2D transition promotes escape from local
minima and enhances generalization in wide minima. Wide minima in neural
networks can serve as a measure indicating robust and adaptable models (Keskar
et al. 2017; Jastrzębski et al. 2018). By exploring minima with this
transition, we aspire for both performance and a deeper grasp of agent
adaptability in diverse scenarios. To check how much the policy resides in a
wide minima, we measure the end-of-training convergence of S2D’s neural
network to wide minima using the sharpness metric of Equation 3 and compare
with those of baselines in the same way as proposed in (Foret et al. 2021),
which is a specific form of sharpness measure proposed in (Keskar et al.
2017).

$\max_{||\epsilon||_{2}\leq\rho}L_{\pi}(\theta+\epsilon)-L_{\pi}(\theta)$ (3)

Here, $\theta$ is the current parameter in the policy loss landscape. The
maximizer $\epsilon$ can be estimated from
$\hat{\epsilon}=\rho\,\text{sgn}(\nabla_{\theta}L_{\pi}(\theta))\cdot\nicefrac{{|\nabla_{\theta}L_{\pi}(\theta)|^{q-1}}}{{\big{(}||\nabla_{\theta}L_{\pi}(\theta)||^{q}_{q}\big{)}^{\frac{1}{p}}}},$
where $1/p+1/q=1$, $\text{sgn}(\cdot)$ is element-wise sign function (Foret et
al. 2021). We use $\rho=0.02,p=2$ in our experiments.

(a)

(b)

(c)

(d)

(e)

Figure 3: Experimental environments. In particular, (a) and (b) are
environments for evaluating generalization.

(a)

(b)

(c)

Figure 4: Performance of the agent with various reward types in multiple goal-
oriented tasks. Notably, in (c) LunarLander, the accumulated reward from
intrinsic rewards was significantly below zero, indicated by a dashed line.

(a)

(b)

Figure 5:  Generalization performance of the ViZDoom agent with various types
of rewards.

Task Metric S2D($\mathscr{C}_{1}$) S2D($\mathscr{C}_{2}$)
S2D($\mathscr{C}_{3}$) Only Sparse Only dense D2S($\mathscr{C}_{1}$)
D2S($\mathscr{C}_{2}$) D2S($\mathscr{C}_{3}$) Lunar Lander Perf.
138.71$\pm$3.71 63.40$\pm$160.55 168.88$\pm$23.66 142.50$\pm$4.25
139.68$\pm$14.90 140.75$\pm$7.46 130.63$\pm$19.69 142.373$\pm$15.62 Sharp.
27.06$\pm$36.31 1231.93$\pm$2424.61 7.46$\pm$3.37 8.97$\pm$2.83 8.71$\pm$4.43
8.95$\pm$2.89 8.99$\pm$2.97 11.32$\pm$3.72 CartPole Perf. 3.18$\pm$4.0
14.61$\pm$10.96 5.29$\pm$7.472 0.14$\pm$0.25 3.88$\pm$4.63 1.55$\pm$0.29
0.38$\pm$0.07 0.97$\pm$0.19 Sharp. 0.12$\pm$0.24 0.01$\pm$0.15 0.01$\pm$0.24
0.08$\pm$0.57 0.19$\pm$0.03 0.16$\pm$0.09 0.05$\pm$0.21 0.02$\pm$0.17 UR5
Perf. 65.54$\pm$10.86 65.69$\pm$17.32 94.15$\pm$4.28 0.00$\pm$0.00
64.23$\pm$13.03 0.00$\pm$0.00 0.00$\pm$0.00 0.00$\pm$0.00 Sharp. 0.67$\pm$0.01
0.62$\pm$0.11 0.61$\pm$0.04 0.09$\pm$0.52 0.67$\pm$0.01 0.52$\pm$0.24
0.56$\pm$0.28 0.47$\pm$0.20

Table 1: Performance and sharpness metric measured for more than 5 different
random seeds in each environment. We highlight the best performance and its
sharpness values in bold, confirming that the top-performing S2D also resides
in the widest minima.

## Experiments

In our experimental section, we delve deeply into the dynamics of the S2D
reward transition compared to multiple reward-driven methods. We unveil its
substantial effects within multiple challenging environments, which are
illustrated in Appendix A. We particularly explored the implications of
applying the reward transition to RL by probing three critical questions:

• Performance Enhancement: How does the Toddler-inspired S2D reward transition
compare to other diverse reward settings? • Post-Transition 3D Policy Loss
Landscape: What are the effects of the S2D transition on the policy loss
landscape? • Correlation Between Wide Minima and Toddler-Inspired Reward
Transition: Does the S2D transition foster convergence to wide minima?

Our understanding is further supported by extensive supplementary experiments
in Appendix C.

### Reward Setting Details

#### Design of sparse and dense reward.

In the sparse reward setting, the agent receives a reward only upon success,
i.e., when reaching the goal. In the dense reward setting, the agent receives
a potential-based reward based on its proximity to the goal. This is expressed
as $\psi(s)=\operatorname{diam}(\mathcal{S})-||s-g||_{2}$, where $\mathcal{S}$
is the set of states, $g\in\mathcal{S}$ is the goal state, and
$\operatorname{diam}(\cdot)$ denotes the diameter of the given set.

#### Reward-driven baselines for comparison.

Aiming to investigate the relation between exploration-exploitation tradeoff
and reward transitions, we explore the Sparse-to-Dense (S2D) approach to
mirror toddlers’ developmental progression. We also evaluate its counterpart,
Dense-to-Sparse (D2S), and solely sparse or dense reward schemes for a
comprehensive assessment of reward strategies. Given the prominence of
intrinsic motivation in tackling exploration-exploitation, we adopted NGU
(Badia et al. 2020)—designed for discrete environments like ViZDoom and
LunarLander—and RND (Burda et al. 2018)—tailored for continuous action such as
CartPole and UR5—as additional baselines. Both methods incentivize agents to
explore by providing additional intrinsic rewards for discovering novel
states.

#### Design for reward transition.

We also explored the hyperparameter for reward transition through ablation
studies as seen in Table 1. Recognizing the importance of the temporal aspects
of initial cognitive and motor interactions in early developmental stages
(Piaget, Cook et al. 1952; Shonkoff and Phillips 2000), we divide the first
quarter and segmented this period into three points for reward transition. We
set the transition timings at $t\in\\{1N,2N,3N\\}$, where $N$ corresponds to
roughly a third of the first quarter of the entire training duration. The
specific value of $N$, adjusted for each environment’s episode length, is
detailed in the Appendix A. We denote these phases as $\mathscr{C}_{1}$,
$\mathscr{C}_{2}$, and $\mathscr{C}_{3}$, signifying the S2D or D2S reward
transition points.

Figure 6: The 3D visualization illustrates the policy loss landscape following
a reward transition, which begins with either a sparse or dense reward. It
includes two sets of transitions: one transitioning from sparse-to-dense (S2D)
and to sparse (Only Sparse), and the other from dense-to-dense (Only Dense)
and to sparse (D2S). Notably, the S2D transitions often exhibited more
distinct smoothing effects locally compared to others. These effects were
noticeable after the transition at T=50 and T=2000 in LunarLander, and at
T=3500 in Cartpole. Detailed 3D visualizations for environments are available
in Appendix B.

Task Reward Transition Number of updates after reward transition phase
$\mathcal{P}_{1}$ $\Delta\mathcal{P}_{1}$ phase $\mathcal{P}_{2}$
$\Delta\mathcal{P}_{2}$ phase $\mathcal{P}_{3}$ $\Delta\mathcal{P}_{3}$ phase
$\mathcal{P}_{4}$ $\Delta\mathcal{P}_{4}$ phase $\mathcal{P}_{5}$ Lunar Lander
S2D($\mathscr{C}_{3}$) 0.031$\pm$0.01 -0.004 0.027$\pm$0.01 +0.003
0.030$\pm$0.01 -0.008 0.022$\pm$0.00 -0.001 0.021$\pm$0.00
D2S($\mathscr{C}_{3}$) 0.043$\pm$0.02 -0.008 0.035$\pm$0.00 +0.004
0.039$\pm$0.01 +0.004 0.043$\pm$0.02 -0.009 0.034$\pm$0.01 only sparse
0.029$\pm$0.01 0.000 0.029$\pm$0.00 +0.005 0.034$\pm$0.01 -0.004
0.030$\pm$0.00 +0.007 0.037$\pm$0.01 only dense 0.039$\pm$0.01 -0.007
0.032$\pm$0.01 0.000 0.032$\pm$0.01 +0.005 0.037$\pm$0.01 -0.002
0.035$\pm$0.01 CartPole S2D($\mathscr{C}_{2}$) 0.028$\pm$0.00 +0.003
0.031$\pm$0.01 0.000 0.031$\pm$0.00 +0.005 0.036$\pm$0.00 -0.013
0.023$\pm$0.00 D2S($\mathscr{C}_{2}$) 0.033$\pm$0.01 +0.014 0.047$\pm$0.04
+0.001 0.048$\pm$0.01 -0.002 0.046$\pm$0.02 +0.001 0.047$\pm$0.02 only sparse
0.027$\pm$0.01 +0.016 0.043$\pm$0.01 -0.009 0.034$\pm$0.01 +0.007
0.041$\pm$0.02 -0.014 0.027$\pm$0.00 only dense 0.033$\pm$0.01 +0.021
0.054$\pm$0.02 -0.013 0.041$\pm$0.02 +0.004 0.045$\pm$0.02 -0.001
0.044$\pm$0.01 UR5 S2D($\mathscr{C}_{3}$) 0.084$\pm$0.01 +0.016 0.100$\pm$0.02
-0.033 0.067$\pm$0.01 +0.028 0.095$\pm$0.02 -0.062 0.033$\pm$0.01
D2S($\mathscr{C}_{3}$) 0.060$\pm$0.03 -0.007 0.053$\pm$0.02 -0.005
0.048$\pm$0.02 +0.011 0.059$\pm$0.02 -0.003 0.056$\pm$0.01 only sparse
0.059$\pm$0.02 -0.013 0.046$\pm$0.00 0.000 0.046$\pm$0.01 +0.031
0.077$\pm$0.00 -0.031 0.046$\pm$0.01 only dense 0.079$\pm$0.02 -0.012
0.067$\pm$0.02 -0.009 0.058$\pm$0.02 -0.005 0.053$\pm$0.01 +0.005
0.058$\pm$0.01

Table 2: Comparison of the depth of local minima in policy loss following
reward transitions. A lower depth of local minima indicates a smoother
terrain. Phase $\mathcal{P}_{i}$ indicates the number of updates, which are
$(T=50,400,800,1200,1600)$ for LunarLander, and $(T=50,1000,2000,3000,4000)$
for others.

### Environment Details

We assess the efficacy of reward dynamics across diverse conditions,
encompassing state and visual observations as well as both discrete and
continuous action domains, as seen in Appendix A. We evaluate the reward
settings, including S2D reward transition, across a variety of goal-based
tasks in well-established benchmark environments. As seen in Figure 3, these
include LunarLander (Brockman et al. 2016), CartPole, and UR5 (Todorov, Erez,
and Tassa 2012). In Appendix A, we particularly detail challenging dynamics
for UR5 and CartPole, featuring randomized placements of the agent, goal, and
obstacles, termed the ’reacher’ version. All agents have full access to the
current state and are tested using the SAC (Haarnoja et al. 2018) algorithm.
We also adjusted the reward structure for both sparse and dense settings, with
details in Appendix A.

#### Environment for generalization.

To measure the improvement in generalization, we design a challenging
egocentric navigation task using the ViZDoom environment (Kempka et al. 2016),
depicted in Figure 3. In the Seen environment (Appendix Figure 9-(a)), object
locations are random, and walls have one of three textures. The Unseen
environment (Appendix Figure 9-(b)) requires generalization to three new wall
textures, distinct from the Seen environment. Here, We employed A3C (Mnih et
al. 2016).

## Results

### Performance Enhancement

#### Sample efficiency and success rate.

Experiments are conducted in diverse environments with sparse rewards, and the
results are shown in Figure 4, 5 and Table 1. The agents in LunarLander,
CartPole-Reacher, ViZDoom-Seen, and ViZDoom-Unseen environments achieve the
lowest performance with default sparse rewards. S2D consistently outperforms
all other baselines in these settings, achieving better sample efficiency.
Even in UR5-Reacher, which is notably more challenging with only sparse
rewards than in other environments, the S2D strategy consistently outperforms
other baselines. While algorithms based on intrinsic motivation often
prioritized exploration over goal attainment in goal-oriented RL tasks, our
approach exhibited exceptional results, exhibiting better exploration-
exploitation tradeoff while being simple and universally applicable.
Importantly, the outcomes associated with D2S consistently fall below those of
S2D across all environments, underlining the efficacy of the S2D transition as
a curriculum.

#### Generalization performance.

The S2D reward transition outperforms other agents across all dynamic
environments requiring generalization as seen in Figure 5-(a), (b).
Particularly in ViZDoom-Unseen, where agents encounter drastic visual changes
with the emergence of three previously unseen walls, the S2D transition
achieves robust generalization and superior performance compared to other
baselines.

### Post-Transition 3D Policy Loss Landscape

We note a marked smoothing effect at various update points after the reward
transition, attributed to the reduction in the depth of local minima, as shown
in Table 2 and Figure 6. This effect is predominantly observed under the S2D
reward. Such smoothing could aid in overcoming local minima, possibly leading
to wide minima. 3D policy loss landscapes of other reward baselines are
visualized using the Cross-Density Visualizer in Appendix B. To further verify
the smoothing quantitatively, we calculate the depth of local minima of
$\pi_{\theta}$ function, which measures the average of differences between the
local maximum and minimum values for a number of updates after the reward
transition. The results, as shown in Table 2, demonstrate that the depth
primarily decreased for the S2D reward transition model.

### Results of Wide Minima

We assess the end-of-training convergence of the neural networks guided by S2D
using sharpness metrics and contrast this with the baselines. Areas of lower
sharpness correspond to wide minima, which can enhance generalization
performance. As demonstrated in Table 1, only the agents guided by S2D reward
transition that converge to the widest minima exhibit superior performance in
challenging environments.

## Discussion and Analyses

In the following analyses, we clearly address the three pivotal questions
raised in our experimental framework:

#### Performance enhancement.

As shown in Figure 4, Figure 5, and Table 1, S2D surpassed other reward
baselines. In our experiments comparing intrinsic motivation methods, we
observed that agents driven by such algorithms tend to prioritize state
coverage over achieving specific goals in goal-oriented RL tasks. This
suggests that while they may excel in exploration, the S2D transition approach
more effectively balances exploration and exploitation, thus ensuring proper
goal acquisition. Additionally, we explored the optimal timing for reward
transition through ablation studies. While this timing is unique for each
environment, it was, in all cases, near a quarter of the total training time.
Particularly challenging tasks like UR5-Reacher required longer periods of
free exploration compared to relatively simpler ones, such as LunarLander.
This echoes the critical early learning phases observed in infants.

#### Post-transition 3D policy loss landscape.

Our 3D visualizations reveal that S2D predominantly smooths the landscape.
Although our main experiments are based on SAC, we also tested other
algorithms such as PPO (Schulman et al. 2017) and DQN (Mnih et al. 2013) for
more comprehensive analyses. This smoothing effect was also uniquely seen with
the S2D reward transition in additional gridworld experiments, as discussed in
Appendix C.

In UR5, minima depth for S2D decreased significantly from 0.095 to 0.033 in
phase $\mathcal{P}_{5}$. This may relate to its higher performance than only
dense rewards particularly at a later stage, as in Figure 4-(b). Conversely,
in CartPole and LunarLander, performance quickly improved after the reward
transition, reflecting S2D’s overall low local minima.

#### Correlation between wide minima and toddler-inspired reward transition.

In Table 1, S2D-guided agents converged to the widest minima with the highest
performance in the LunarLander and CartPole-Reacher, where agents receiving
only sparse rewards achieved nonzero performance. Conversely, in the
UR5-Reacher, agents with only sparse rewards showed zero performance. This
implies that in sparse reward situations, premature convergence into wide
minima can lead to gradient stagnation and retain low sharpness, with high
variance. Yet, S2D, compared to only dense rewards, still posts the highest
performance with the lowest sharpness, suggesting its alignment within wide
minima.

## Conclusion

Inspired by toddler developmental learning, our research pioneers a shift from
static, single-density to dynamic reward transitions in goal-oriented RL. This
toddler-inspired approach demonstrates notable effects of transitions on
learning dynamics in RL. We examine its implications across various scenarios,
focusing on its efficiency. Using the Cross-Density Visualizer, we observe the
primary smoothing effects on the policy loss landscape during the S2D
transition. Sharpness metrics further confirm the smoothing effects of S2D,
guiding agents towards wider minima to improve generalization. This blend of
biological and artificial paradigms may lead to robust, high-performance
learning systems.

Limitations. While our study focuses on understanding the implications of S2D
reward transition, we haven’t provided an automatic method for finding an
optimal transition yet. Nonetheless, our preliminary research on criteria sets
the stage for future development of automated methods.

## Acknowledgments

The authors would like to express their sincere gratitude to Inwoo Hwang,
Changhoon Jeong, Moonhoen Lee, and Dong-Sig Han for their insightful
discussions and valuable suggestions on the early drafts of this paper. This
work was partly supported by the IITP (2021-0-02068-AIHub/15%,
2021-0-01343-GSAI/10%, 2022-0-00951-LBA/15%, 2022-0-00953-PICA/25%) and NRF
(RS-2023-00274280/10%, 2021R1A2C1010970/25%) grant funded by the Korean
government.

## References

  * Achille, Rovere, and Soatto (2018) Achille, A.; Rovere, M.; and Soatto, S. 2018.  Critical learning periods in deep networks.  In _International Conference on Learning Representations_.
  * Andrychowicz et al. (2020) Andrychowicz, O. M.; Baker, B.; Chociej, M.; Jozefowicz, R.; McGrew, B.; Pachocki, J.; Petron, A.; Plappert, M.; Powell, G.; Ray, A.; et al. 2020.  Learning dexterous in-hand manipulation.  _The International Journal of Robotics Research_ , 39(1): 3–20.
  * Aubret, Matignon, and Hassas (2019) Aubret, A.; Matignon, L.; and Hassas, S. 2019.  A survey on intrinsic motivation in reinforcement learning.  _arXiv preprint arXiv:1908.06976_.
  * Badia et al. (2020) Badia, A. P.; Sprechmann, P.; Vitvitskyi, A.; Guo, D.; Piot, B.; Kapturowski, S.; Tieleman, O.; Arjovsky, M.; Pritzel, A.; Bolt, A.; et al. 2020.  Never give up: Learning directed exploration strategies.  _arXiv preprint arXiv:2002.06038_.
  * Bambach et al. (2018) Bambach, S.; Crandall, D.; Smith, L.; and Yu, C. 2018.  Toddler-inspired visual object learning.  _Advances in neural information processing systems_ , 31.
  * Bengio et al. (2009) Bengio, Y.; Louradour, J.; Collobert, R.; and Weston, J. 2009.  Curriculum learning.  In _ICML ’09_.
  * Brockman et al. (2016) Brockman, G.; Cheung, V.; Pettersson, L.; Schneider, J.; Schulman, J.; Tang, J.; and Zaremba, W. 2016.  Openai gym.  _arXiv preprint arXiv:1606.01540_.
  * Burda et al. (2018) Burda, Y.; Edwards, H.; Storkey, A.; and Klimov, O. 2018.  Exploration by random network distillation.  _arXiv preprint arXiv:1810.12894_.
  * De Kleijn, Sen, and Kachergis (2022) De Kleijn, R.; Sen, D.; and Kachergis, G. 2022.  A Critical Period for Robust Curriculum-Based Deep Reinforcement Learning of Sequential Action in a Robot Arm.  _Topics in Cognitive Science_ , 2(2): 311–326.
  * Dong, Gong, and Zhu (2017) Dong, Q.; Gong, S.; and Zhu, X. 2017.  Class rectification hard mining for imbalanced deep learning.  In _Proceedings of the IEEE International Conference on Computer Vision_ , 1851–1860.
  * Du et al. (2021) Du, B.; Gao, X.; Hu, W.; and Li, X. 2021.  Self-Contrastive Learning with Hard Negative Sampling for Self-Supervised Point Cloud Learning.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , MM ’21, 3133–3142. New York, NY, USA: Association for Computing Machinery.  ISBN 9781450386517.
  * Florensa et al. (2018) Florensa, C.; Held, D.; Geng, X.; and Abbeel, P. 2018.  Automatic goal generation for reinforcement learning agents.  In _International conference on machine learning_ , 1515–1528. PMLR.
  * Foret et al. (2021) Foret, P.; Kleiner, A.; Mobahi, H.; and Neyshabur, B. 2021.  Sharpness-aware Minimization for Efficiently Improving Generalization.  In _International Conference on Learning Representations_.
  * Gibson (1988) Gibson, E. J. 1988.  Exploratory behavior in the development of perceiving, acting, and the acquiring of knowledge.  _Annual review of psychology_ , 39(1): 1–42.
  * Goodfellow, Vinyals, and Saxe (2014) Goodfellow, I. J.; Vinyals, O.; and Saxe, A. M. 2014.  Qualitatively characterizing neural network optimization problems.  _arXiv preprint arXiv:1412.6544_.
  * Gopnik, Meltzoff, and Kuhl (1999) Gopnik, A.; Meltzoff, A. N.; and Kuhl, P. K. 1999.  _The scientist in the crib: Minds, brains, and how children learn._ William Morrow & Co.
  * Gopnik et al. (2017) Gopnik, A.; O’Grady, S.; Lucas, C. G.; Griffiths, T. L.; Wente, A.; Bridgers, S.; Aboody, R.; Fung, H.; and Dahl, R. E. 2017.  Changes in cognitive flexibility and hypothesis search across human life history from childhood to adolescence to adulthood.  _Proceedings of the National Academy of Sciences_ , 114(30): 7892–7899.
  * Graves et al. (2017) Graves, A.; Bellemare, M. G.; Menick, J.; Munos, R.; and Kavukcuoglu, K. 2017.  Automated curriculum learning for neural networks.  In _international conference on machine learning_ , 1311–1320. PMLR.
  * Haarnoja et al. (2018) Haarnoja, T.; Zhou, A.; Abbeel, P.; and Levine, S. 2018.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In _International conference on machine learning_ , 1861–1870. PMLR.
  * Hacohen and Weinshall (2019) Hacohen, G.; and Weinshall, D. 2019.  On The Power of Curriculum Learning in Training Deep Networks.  _ArXiv_ , 2.
  * Harutyunyan et al. (2015) Harutyunyan, A.; Devlin, S.; Vrancx, P.; and Nowé, A. 2015.  Expressing arbitrary reward functions as potential-based advice.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 29.
  * Jastrzębski et al. (2018) Jastrzębski, S.; Kenton, Z.; Arpit, D.; Ballas, N.; Fischer, A.; Bengio, Y.; and Storkey, A. 2018.  Finding Flatter Minima with SGD.
  * Kalantidis et al. (2020) Kalantidis, Y.; Sariyildiz, M. B.; Pion, N.; Weinzaepfel, P.; and Larlus, D. 2020.  Hard Negative Mixing for Contrastive Learning.  In Larochelle, H.; Ranzato, M.; Hadsell, R.; Balcan, M.; and Lin, H., eds., _Advances in Neural Information Processing Systems_ , volume 33, 21798–21809. Curran Associates, Inc.
  * Kempka et al. (2016) Kempka, M.; Wydmuch, M.; Runc, G.; Toczek, J.; and Jaśkowski, W. 2016.  Vizdoom: A doom-based ai research platform for visual reinforcement learning.  In _2016 IEEE conference on computational intelligence and games (CIG)_ , 1–8. IEEE.
  * Keskar et al. (2016) Keskar, N. S.; Mudigere, D.; Nocedal, J.; Smelyanskiy, M.; and Tang, P. T. P. 2016.  On large-batch training for deep learning: Generalization gap and sharp minima.  _arXiv preprint arXiv:1609.04836_.
  * Keskar et al. (2017) Keskar, N. S.; Mudigere, D.; Nocedal, J.; Smelyanskiy, M.; and Tang, P. T. P. 2017.  On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima.  In _International Conference on Learning Representations_.
  * Kim et al. (2023a) Kim, K.; Lee, H.; Lee, M. W.; Lee, M.; Lee, M.; and Zhang, B.-T. 2023a.  L-SA: Learning Under-Explored Targets in Multi-Target Reinforcement Learning.  _arXiv preprint arXiv:2305.13741_.
  * Kim et al. (2021) Kim, K.; Lee, M. W.; Kim, Y.; Ryu, J.; Lee, M.; and Zhang, B.-T. 2021.  Goal-aware cross-entropy for multi-target reinforcement learning.  _Advances in Neural Information Processing Systems_ , 34: 2783–2795.
  * Kim et al. (2023b) Kim, K.; Shin, K.; Lee, M. W.; Lee, M.; Lee, M.; and Zhang, B.-T. 2023b.  Visual Hindsight Self-Imitation Learning for Interactive Navigation.  _arXiv preprint arXiv:2312.03446_.
  * Knox et al. (2023) Knox, W. B.; Allievi, A.; Banzhaf, H.; Schmitt, F.; and Stone, P. 2023.  Reward (mis) design for autonomous driving.  _Artificial Intelligence_ , 316: 103829.
  * Ladosz et al. (2022) Ladosz, P.; Weng, L.; Kim, M.; and Oh, H. 2022.  Exploration in deep reinforcement learning: A survey.  _Information Fusion_ , 85: 1–22.
  * Laud (2004) Laud, A. D. 2004.  _Theory and application of reward shaping in reinforcement learning_.  University of Illinois at Urbana-Champaign.
  * Li et al. (2018) Li, H.; Xu, Z.; Taylor, G.; Studer, C.; and Goldstein, T. 2018.  Visualizing the loss landscape of neural nets.  _Advances in neural information processing systems_ , 31.
  * MacKay (1992) MacKay, D. J. 1992.  Information-based objective functions for active data selection.  _Neural computation_ , 4(4): 590–604.
  * Mnih et al. (2016) Mnih, V.; Badia, A. P.; Mirza, M.; Graves, A.; Lillicrap, T.; Harley, T.; Silver, D.; and Kavukcuoglu, K. 2016.  Asynchronous methods for deep reinforcement learning.  In _International conference on machine learning_ , 1928–1937. PMLR.
  * Mnih et al. (2013) Mnih, V.; Kavukcuoglu, K.; Silver, D.; Graves, A.; Antonoglou, I.; Wierstra, D.; and Riedmiller, M. 2013.  Playing atari with deep reinforcement learning.  _arXiv preprint arXiv:1312.5602_.
  * Mnih et al. (2015) Mnih, V.; Kavukcuoglu, K.; Silver, D.; Rusu, A. A.; Veness, J.; Bellemare, M. G.; Graves, A.; Riedmiller, M.; Fidjeland, A. K.; Ostrovski, G.; Petersen, S.; Beattie, C.; Sadik, A.; Antonoglou, I.; King, H.; Kumaran, D.; Wierstra, D.; Legg, S.; and Hassabis, D. 2015.  Human-level control through deep reinforcement learning.  _Nature_ , 518(7540): 529–533.
  * Narvekar and Stone (2020) Narvekar, S.; and Stone, P. 2020.  Generalizing curricula for reinforcement learning.  In _4th Lifelong Machine Learning Workshop at ICML 2020_.
  * Ng, Harada, and Russell (1999a) Ng, A.; Harada, D.; and Russell, S. J. 1999a.  Policy Invariance Under Reward Transformations: Theory and Application to Reward Shaping.  In _International Conference on Machine Learning_.
  * Ng, Harada, and Russell (1999b) Ng, A. Y.; Harada, D.; and Russell, S. 1999b.  Policy invariance under reward transformations: Theory and application to reward shaping.  In _Icml_ , volume 99, 278–287. Citeseer.
  * Oudeyer and Smith (2016) Oudeyer, P.-Y.; and Smith, L. B. 2016.  How evolution may work through curiosity-driven developmental process.  _Topics in Cognitive Science_ , 8(2): 492–502.
  * Papoudakis et al. (2021) Papoudakis, G.; Christianos, F.; Schäfer, L.; and Albrecht, S. V. 2021.  Benchmarking Multi-Agent Deep Reinforcement Learning Algorithms in Cooperative Tasks.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)_.
  * Park et al. (2021) Park, J.; Park, K.; Oh, H.; Lee, G.; Lee, M.; Lee, Y.; and Zhang, B.-T. 2021.  Toddler-Guidance Learning: Impacts of Critical Period on Multimodal AI Agents.  In _Proceedings of the 2021 International Conference on Multimodal Interaction_ , 212–220.
  * Pathak et al. (2017) Pathak, D.; Agrawal, P.; Efros, A. A.; and Darrell, T. 2017.  Curiosity-driven exploration by self-supervised prediction.  In _International conference on machine learning_ , 2778–2787. PMLR.
  * Piaget, Cook et al. (1952) Piaget, J.; Cook, M.; et al. 1952.  _The origins of intelligence in children_ , volume 8.  International Universities Press New York.
  * Qin, Yan, and Tang (2022) Qin, L.; Yan, R.; and Tang, H. 2022.  A Low Latency Adaptive Coding Spiking Framework for Deep Reinforcement Learning.  _arXiv preprint arXiv:2211.11760_.
  * Schulman et al. (2017) Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; and Klimov, O. 2017.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_.
  * Shonkoff and Phillips (2000) Shonkoff, J.; and Phillips, D. 2000.  From Neurons to Neighborhoods: The Science of Early Childhood Development. eric. ed. gov.  _National Academy of Sciences Press: Washington DC. Accessed on May_ , 8: 2015.
  * Taylor and Stone (2009) Taylor, M. E.; and Stone, P. 2009.  Transfer learning for reinforcement learning domains: A survey.  _Journal of Machine Learning Research_ , 10(7).
  * Todorov, Erez, and Tassa (2012) Todorov, E.; Erez, T.; and Tassa, Y. 2012.  Mujoco: A physics engine for model-based control.  In _2012 IEEE/RSJ international conference on intelligent robots and systems_ , 5026–5033. IEEE.
  * Turchetta et al. (2020) Turchetta, M.; Kolobov, A.; Shah, S.; Krause, A.; and Agarwal, A. 2020.  Safe reinforcement learning via curriculum induction.  _Advances in Neural Information Processing Systems_ , 33: 12151–12162.
  * Weinshall, Cohen, and Amir (2018) Weinshall, D.; Cohen, G.; and Amir, D. 2018.  Curriculum learning by transfer learning: Theory and experiments with deep networks.  In _International Conference on Machine Learning_ , 5238–5246. PMLR.
  * Zhang (1994) Zhang, B.-T. 1994.  Selecting a Critical Subset of Given Examples during Learning.  In _International Conference on Artificial Neural Networks_ , 517–520. Springer.
  * Zhang et al. (2022a) Zhang, D.; Zhang, T.; Jia, S.; Wang, Q.; and Xu, B. 2022a.  Tuning Synaptic Connections instead of Weights by Genetic Algorithm in Spiking Policy Network.  _arXiv preprint arXiv:2301.10292_.
  * Zhang et al. (2022b) Zhang, D.; Zhang, T.; Jia, S.; and Xu, B. 2022b.  Multi-sacle dynamic coding improved spiking actor network for reinforcement learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, 59–67.

Appendix.
Unveiling the Significance of Toddler-Inspired Reward Transition in Goal-
Oriented Reinforcement Learning

• Section A: This section provides further details about the experiments
conducted in the main text and the appendices. • Section B: This section
presents comprehensive results of visualizing the 3D policy loss landscape
after the stage-transition for Toddler-inspired S2D Reward Transition and
various baselines, which is related to the section: Visualizing Post-
Transition 3D Policy Loss Landscape: Cross-Density Visualizer in the main
text. • Section C: This section features additional experiments and analyses,
as well as further visualizations of the 3D policy loss landscape for various
algorithms in the gridworld.

##  Appendix A Section A: Experimental Details

### A.1 Comparison of Overall Experimental Setup

Table 3: An overall comparison of experimental environments. Those above the
double line relate to environments from the main text, while those below
correspond to those in the appendices. Each environment and reward was
modified to adapt to our experiments for Toddler-inspired S2D reward
transition, with details provided in the respective environment setup
descriptions.

Environment Task Difficulty Settings Environments Type Input # of Stages Point
of View Action Space Observation Types [HTML]EEEEEE LunarLander-V2 - 2D
Coordinate & Velocity & Angle & Boolean flag value [HTML]EEEEEE [HTML]EEEEEE
OpenAI Gym(Todorov, Erez, and Tassa 2012) 2-stage Allocentric View Continous
State-based RL MuJoCo(Todorov, Erez, and Tassa 2012) CartPole-Reacher - 3D
Joint Value & Goal Position 2-stage Allocentric View Continuous State-based RL
[HTML]EEEEEE UR5-Reacher - 3D Joint Value & Goal Position [HTML]EEEEEE
[HTML]EEEEEE MuJoCo(Todorov, Erez, and Tassa 2012) 2-stage Allocentric View
Continuous State-based RL Seen & Unseen - 3D RGB-D ViZDoom(Kempka et al. 2016)
2-stage Egocentric View Discrete Visual RL [HTML]EEEEEE Shelf-delivery Level3
2D Internal state of the surrounding tiles [HTML]EEEEEE RWARE(Papoudakis et
al. 2021) Non-humanoid Allocentric View Discrete 3-stage & State-based RL
[HTML]FFFFFF Navigation - 2D Position-based value [HTML]FFFFFF [HTML]FFFFFF
Gridworld 2-stage Allocentric View Discrete State-based RL

Table 4: Detailed settings for hyperparameter $N$. $N$ is the number of frames
of training for the environment after which the stage transition occurs for
the respective transition setting ($\mathscr{C}_{1}$, $\mathscr{C}_{2}$, and
$\mathscr{C}_{3}$). In case of ViZDoom experiments, $N$ indicates the number
of updates. For Gridworld-DQN, $\mathscr{C}_{1}$=100, $\mathscr{C}_{2}$=200,
and $\mathscr{C}_{3}$=300 episodes.

Environment Total # of Train 1$N$(${\mathscr{C}_{1}}$)
2$N$(${\mathscr{C}_{2}}$) 3$N$(${\mathscr{C}_{3}}$) [HTML]EEEEEE
LunarLander-V2 1M frames 100k 200k 400k ViZDoom-Seen & Unseen 1M frames 50k
100k 250k [HTML]EEEEEE CartPole-Reacher 12k episodes 1k 2k 3k UR5-Reacher 25k
episodes 1k 2k 3k [HTML]EEEEEE RWARE 7M frames 1M 2M 3M [HTML]FFFFFF Gridworld
25k episodes 3k 5k 7k

### A.2 Reward Transition Hyperparameters

In Table 4 and Figure 7, we have documented the specific point at which the
agent transitioned to dense reward stages for each environment with a number
of stages.

Figure 7: Visualization of the overall setup, including the number of stages
and the transition times, in Toddler-inspired S2D experiments across all
environments.

### A.3 Model Hyperparameters

The Table 5 provides information on the hyperparameters for each environment
used in our study.

Table 5: Hyperparameters for our various experiments for Figure 4 and
Appendices. For visualizing the policy loss landscape in LunarLander, we
employed discount factors $\gamma$ of both 1 and 0.99.

Hyperparameters LunarLander CartPole-Reacher UR5-Reacher ViZDoom-S&U RWARE
Gridworld [HTML]EEEEEE RL algorithms SAC SAC SAC A3C PPO PPO Learning Rate
3e-4 0.0007 0.0007 7e-5 5e-4 5e-4 [HTML]EEEEEE Value Function Coefficient 3e-4
- - 0.5 - 5e-4 Discount Factor 0.99 0.99 0.99 1.0 0.99 0.99 [HTML]EEEEEE Batch
Size 128 128 128 - 10 128 Optimizer Adam Adam Adam Adam Adam Adam [HTML]EEEEEE
Maximum # of Steps 500 200 500 50 150 50 Entropy Coefficient 0.2 Auto Auto 0.1
0.01 0.03

### A.4 Environment Details

All reward-driven baselines and S2D reward were run with more than 4 random
seeds for each environment. Our experiments were conducted using four NVIDIA
GeForce RTX 3090 GPUs, or two NVIDIA GeForce RTX 2080ti GPUs, with the CPU
being an AMD Ryzen Threadripper 3960X 24-Core Processor, and utilizing 188GB
of RAM.

#### Design of sparse to potential-based dense reward.

In a sparse-to-dense curriculum, an agent is eventually trained in a dense
environment. Here, if a dense reward is not properly defined, the optimal
agent in the dense environment may not be optimal in the base (sparse)
environment. Therefore, an additional reward in the dense environment must be
defined properly so that policy optimality is preserved. In order for
optimality to be preserved, we utilize potential-based reward shaping (PBRS)
(Ng, Harada, and Russell 1999a; Harutyunyan et al. 2015). In PBRS, an
additional reward is given as follows:333While $\Phi$ is the notation commonly
used for potential function, we have used $\psi$ throughout our paper.

$F_{i}(s,a)=\mathbb{E}_{s^{\prime}\sim\mathcal{P}(s,a)}[\gamma\Phi_{i}(s^{\prime})-\Phi_{i}(s)],$
(4)

where $\Phi_{i}:\mathcal{S}\rightarrow\mathbb{R}$ is an arbitrary function
called a _potential function_ at stage $i$.

By utilizing the additional reward function $F_{i}$, optimality of the policy
is preserved, _i.e._ , optimal policy $\pi^{*}$ with respect to reward
function $\mathcal{R}_{i}+F_{i}$ is still optimal with respect to
$\mathcal{R}_{i}$, as indicated below (Harutyunyan et al. 2015). We note that
$R_{i}+F_{i}$ gives a denser reward than $R_{i}$.

$\displaystyle Q(s,a)$
$\displaystyle=\mathbb{E}_{\mathcal{P},\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}\left(\mathcal{R}^{t}_{i}+F^{t}_{i}\right)\mid
s_{0}=s\right]$
$\displaystyle=\mathbb{E}_{\mathcal{P},\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}\left(\mathcal{R}^{t}_{i}+\gamma\Phi_{i}\left(s_{t+1}\right)-\Phi_{i}(s_{t})\right)\mid
s_{0}=s\right]=\mathbb{E}_{\mathcal{P},\pi}\left[\sum_{t=0}^{\infty}\gamma^{t}\mathcal{R}^{t}_{i}-\Phi_{i}(s_{0})\right].$

In the sparse reward setting ($\mathcal{M}_{1}$), the agent only receives a
reward when it successfully reaches the goal within a specific distance
threshold. In the dense reward setting ($\mathcal{M}_{2}$, $\mathcal{M}_{3}$),
the agent receives a potential-based dense reward in addition to the goal-
based reward. The dense reward $\psi(\cdot)$, shown in Equation 5, is
determined by the agent’s proximity to the goal, which is measured using a
positive value based on the L2-distance between the agent’s current position
$s\in\mathcal{S}$ and the goal $g\in\mathcal{G}$:

$\psi(s)=\operatorname{diam}(\mathcal{S})-||s-g||_{2},$ (5)

where $\mathcal{S}$ is set of states, $\mathcal{G}\subseteq\mathcal{S}$ is set
of goal states, and $\operatorname{diam}(\cdot)$ denotes the diameter of a
set.

From a potential-based reward shaping perspective, Table 6 showcases the
precise formulations for both sparse and dense rewards across various
experimental environments, utilizing distance as the pivotal potential
function in the reward design.

Table 6:  Reward setting formulation in the environments used for the
experiments. Each cell indicates the condition at which a respective sparse or
dense reward is provided to the agent. The specific reward values are outlined
in the text for each task description.

[HTML]EEEEEE  Reward LunarLander-V2 CartPole-Reacher UR5 ViZDoom-Seen ViZDoom-
Unseen Sparse $||s-g||_{2}<1$ $||s-g||_{2}<0.02$ $||s-g||_{2}<0.02$
$||s-g||_{2}<0.0075$ $||s-g||_{2}<0.0075$ Dense
$\gamma\psi(s_{t+1})-\psi(s_{t})<0.3$ $\gamma\psi(s_{t+1})-\psi(s_{t})<1$
$\gamma\psi(s_{t+1})-\psi(s_{t})<0.14$

#### OpenAI Gym: LunarLander-V2.

The lander begins each episode in mid-air, equipped with a random initial
velocity and orientation. The primary objective for the agent is to skillfully
maneuver the lander’s engines, guiding it to touch down between the two flags.
Ideally, this landing should be at the center of the pad, as vertically
aligned as possible, and with minimal speed to ensure a gentle touchdown.
Rewards are granted for skillful approaches: transitioning from the screen’s
top to the landing pad, achieving a gentle landing with minimal speed, and for
each leg that makes ground contact. Conversely, penalties apply for excessive
main engine use, encouraging fuel conservation, and significant penalties are
incurred for crashes or landing far from the intended pad. For our
experiments, we employed the environment’s default rewards as the sparse
reward structure and incorporated distance-based, potential-driven dense
rewards as part of our Toddler-inspired S2D reward transition, detailed in the
above design of reward transition.

Moreover, the potential-based reward shaping we introduced is rooted in a
distance-based potential function. Unlike other goal-conditioned environments,
in the LunarLander scenario, there is a possibility of receiving rewards for
landing at any point on the ground. As such, when the lander approaches within
a distance of 0.3 from the ground (where 1.0 indicates the entire screen
height), as seen in Table 6, an additional potential-based reward per frame is
granted.

#### MuJoCo: CartPole-Reacher & UR5-Reacher tasks.

Figure 8: Examples of environments where goals randomly spawn. (a)
UR5-Reacher. (b) CartPole-Reacher.  Figure 9: Egocentric views of a ViZDoom
agent in environments with various walls and objects. (a) Three walls in
ViZDoom-Seen. (b) Three walls in ViZDoom-Unseen.

To evaluate and compare the effectiveness of Toddler-inspired S2D reward
transition in a variety of tasks, we used MuJoCo (Todorov, Erez, and Tassa
2012) engine, a widely used physics engine to simulate manipulation in virtual
physical environments. Experiments were conducted in challenging continuous
control tasks where a goal is randomly given each episode, and the agent must
reach the goal.

CartPole-Reacher is a task where an agent moves the cart along the horizontal
line to make the connected pole stand for as long time as possible. Since the
agent easily reaches the highest performance in the default task, we made it
more challenging by setting a goal that the end of the pole must be within a
radius less than a specific threshold. The goal is sampled in the upper side
of the horizontal line where it is possible to reach the pole as seen in
Figure 8-(b).

UR5-Reacher is a six-degree-of-freedom arm with six joints that each allow
movement along one degree of freedom. This flexibility enables the arm to
reach a large number of positions and orientations but also makes controlling
the arm a challenging task. The UR5-Reacher is often simulated as an
environment where an agent must learn to control the arm. In our task, the
agent needs to learn how to move the arm to a specific position. Our designed
goal is randomly spawned for every episode, as seen in Figure 8-(a).

For both UR5-Reacher and CartPole-Reacher, from the very beginning, potential-
based dense reward is given based on the distance between the current state of
the agent and the goal state as seen in Table 6. Also, a constant time penalty
is provided to encourage the agents to solve the tasks quickly.

#### ViZDoom-Seen & Unseen.

ViZDoom (Kempka et al. 2016) is a simulator, based on the first-person shooter
game Doom, that was developed to accelerate the reinforcement learning
research. In particular, we adopt and modify the egocentric navigation task
designed by (Kim et al. 2021), where the agent is placed in one corner of the
square room at the start of the episode and must navigate to the correct
object out of the two objects spawned in the room. The correct “goal” object
is specified every episode uniformly at random, which is known to the agent.
The two objects spawned are Card and Skull, each of them appearing as one of
three color variants (Red, Blue, Yellow). This prevents the agent from merely
memorizing the objects’ appearances according to their colors. In addition,
three wall textures are used in each of Seen and Unseen versions, and these
two sets of wall textures are mutually exclusive.

The agent receives a concatenation of 4 frames of RGB-D vision input. Each
vision channel has width and height of 42 and 42. The agent can choose among 3
discrete actions: turning clockwise, turning counterclockwise, and moving
forward. A single chosen action is repeated for 4 in-game frames. Our
manuscript refers to a single “step” within this environment as these 4
consecutive steps.

As a sparse reward function, the agent receives a reward of 10 for arriving at
the goal object, and a reward of -1 for arriving at the incorrect object.
Reaching either of these objects terminates the episode, which otherwise ends
after 50 steps and grants a negative reward of -0.1. For accelerating the
training, a negative reward of -0.01 is provided every time step. To render
this task visually complex, as well as to evaluate generalization of the
trained agent, we use Seen and Unseen versions of the map as shown in Figure 9
that differ in the set of wall textures used. Dense reward is designed as an
additional potential reward of 5.0 provided when the agent arrives within
distance of 100 from the goal object. Note that the map size is 700 by 700.

As the baseline, we use the A3C algorithm (Mnih et al. 2016) and the
architecture implemented by (Kim et al.
2021)444https://github.com/kibeomKim/GACE-GDAN and (Kim et al. 2023a, b). We
also note that all ViZDoom experiments were conducted on the two hardware
settings. Experiments on unified hardware settings will be further provided.

##  Appendix B Section B: 3D Visualization of the Policy Loss Landscape After
Stage Transition

This study visualizes the 3D policy loss landscape after reward transitions
from initially sparse or dense reward settings to two versions: one where
sparse reward is provided and another where dense reward is provided. This
visualization is done within the same parameter space using Cross-Density
Visualizer. As hyperparameters for Cross-Density Visualizer, the values for
$\alpha$ and $\beta$ in
$\tilde{\theta}=\theta+\alpha\mathbf{x}+\beta\mathbf{y}$ range from -10 to 10.
Consequently, Sparse-to-Dense (S2D) and Sparse-to-Sparse (Only Sparse)
constitute one set, while Dense-to-Sparse (D2S) and Dense-to-Dense (Only
Dense) make up another set. We observed a more numerous and distinct smoothing
effect, especially under the reward transition of Toddler-inspired S2D. This
smoothing effect could facilitate the surmounting of local minima, potentially
leading to wider minima.

#### Results.

In the Toddler-inspired S2D reward transition, a greater number of noticeable
smoothing effects, such as the reduction in the depth of local minima, are
observed. This is evident in the blue landscapes of Figures 10, 11, and 12,
particularly in the sections below showing sparse-to-sparse and sparse-to-
dense visualizations, in contrast to the D2S, Only Dense, and Only Sparse
methods.

Based on our observation of reduced local minima depth, we hypothesize that
this facilitates escape from local minima, improving generalization
performance on wider minima. To test this, we measure the end-of-training
convergence of Toddler-inspired S2D’s neural network to wider minima using
sharpness metrics and compare it with those of baselines. As shown in Table 1,
Toddler-inspired S2D-guided agents converging to wider minima perform better
in dynamic environments. This suggests a correlation between the smoothing
effect of the local loss landscape and the likelihood of escaping local
minima.

### B.1 Additional Insights: Visualizing Policy Loss Landscape After Reward
Transition

To gain a deeper understanding of how reward transitions influence agent
behavior, we visualized the policy loss landscape after transitions. This
examination provides a nuanced view into the nature of the model’s
optimization landscape, revealing distinct challenges or advantages that can
shape continuous learning.

#### Distinguishing features of LunarLander-V2’s landscape.

What sets LunarLander-V2 as seen in Figure 10 apart from other environments is
its unique reward distribution. In this environment, actions such as moving
from the screen’s top to the landing pad or achieving a resting state provide
rewards ranging between 100 and 140 points. Adversely, actions like deviating
from the landing pad or causing a crash lead to deductions. We speculate that
such a plethora of reward avenues is what renders the agent’s policy loss
landscape rounder and more gently sloped, and less peak-ridden compared to the
landscapes in other environments.

#### Pronounced peaks in CartPole-Reacher and UR5-Reacher.

In contrast, the CartPole-Reacher and UR5-Reacher environments exhibit a more
singular reward paradigm. Rewards are focused and localized, leading to their
policy loss landscapes showcasing distinct peaks, as shown in Figure 11 and 12
for CartPole-Reacher, and UR5-Reacher. This is indicative of the few specific
scenarios where agents earn positive feedback.

Through these visualizations, we not only discern the distinct reward
mechanisms inherent to different environments but also appreciate how such
structures sculpt the policy loss landscapes, influencing agent learning
trajectories post-transition.

### B.2 Detailed 3D Visualizations for All Baselines

### LunarLander-V2: Dense-to-Sparse (D2S) & Dense-to-Dense / Sparse-to-Dense
(S2D) & Sparse-to-Sparse

Figure 10:  Visualization of the 3D policy loss landscape after reward
transitions: The ’low’ on the left side illustrates the landscape immediately
following the transition, while the ’low’ on the right side depicts the
landscape around t = 2000 after the transition. The first two lines represent
dense-to-sparse (D2S, $\mathscr{C}_{1}$, red) and dense-to-dense (Only Dense,
blue) transitions. In contrast, the second set of lines represent sparse-to-
dense (Toddler-inspired S2D, $\mathscr{C}_{2}$, blue) and sparse-to-sparse
(Only Sparse, red) transitions. Notably, the depth of local minima reduction
(smoothing effects) is more pronounced under the Toddler-inspired S2D,
observed across multiple update points after the reward transition.

### CartPole-Reacher: Dense-to-Sparse (D2S) & Dense-to-Dense / Sparse-to-Dense
(S2D) & Sparse-to-Sparse

Figure 11: In the visualization of the 3D policy loss landscape after reward
transitions, significant smoothing effects were more frequently observed when
applying the Toddler-inspired S2D (sparse-to-dense, $\mathscr{C}_{2}$, blue)
reward transition.

### UR5-Reacher: : Dense-to-Sparse (D2S) & Dense-to-Dense / Sparse-to-Dense
(S2D) & Sparse-to-Sparse

Figure 12: Visualization of the 3D policy loss landscape after stage-
transitions in sparse-to-dense (Toddler-inspired S2D, $\mathscr{C}_{3}$, blue)
and sparse-to-sparse (Only Sparse, red). Here, in the UR5-reacher, the
improvement in performance of S2D over only dense becomes apparent later, and
until that point, the performance of the two is generally not significantly
different. Perhaps because of that, the Only Dense model also often showed a
lower depth of local minima depending on the seed immediately after the reward
transition. Yet, we observed even more smoothing effects due to the decrease
in the depth of local minima under Toddler-inspired S2D reward transition,
even compared to other baselines.

##  Appendix C Section C: Additional Experiments of Toddler-Inspired S2D
Reward Transition in Various Environments

### C.1 Shelf Delivery Tasks in RWARE: Experiments on a Suboptimal Reward
Structure with Three-Stage Guidance

#### Task settings.

As seen in Figure 13 and 14, We conduct additional experiments on grid-world
RWARE environments with discrete state-action space. Modified from RWARE
(Papoudakis et al. 2021) to be a single-agent setting, each environment
contains a mobile agent and several rows of shelves (blue), some of which are
randomly sampled and “requested” (red) to be delivered. The agent has an
available action space of {MoveForward, TurnLeft, TurnRight, Load/Unload,
Noop}. The agent observes only the information of tiles within 3-by-3 square
area centered at the agent. The agent must carry and deliver requested shelves
to the goal location (green). This can be deemed as a task with subgoals,
where the agent must first reach the tile with the requested shelf, pick up
the shelf and carry it to the goal, and then return the shelf back to its
original position.

We designate separate environmental settings for three difficulty settings:
Level1 contains 8 shelves, 3 of which are requested; Level2 contains 10
shelves, 3 of which are requested, leading to sparser reward; Level3 is a vast
environment with a total of 32 shelves, 16 of which are requested. The space
to explore is magnified as the difficulty increases. These environments are
illustrated in Figure 14.

#### Design of guidance.

Reward functions for three different stages of guidance are naturally designed
according to this sequence of subtasks: Stage-1 guidance provides +1.0 reward
when the agent successfully carries a requested shelf to a goal location;
Stage-2 guidance provides reward for the delivery and returning the shelf to
its original location; Stage-3 guidance provides an additional reward for
picking up a requested shelf.

Figure 13: Performance of the agents in RWARE tasks. Vertical axis indicates
the average number of requested shelves successfully delivered to the goal per
episode. Horizontal axis indicates the number of time steps. The solid lines
show the performances of Toddler-inspired S2D reward transition agents
$(\mathscr{C}_{1})$, $(\mathscr{C}_{2})$, and $(\mathscr{C}_{3})$, while the
dotted lines show the performances of only dense reward agents
$(\mathscr{C}_{4})$. The mean and standard deviation of each agent in each
level across five trials are shown as lines and shaded regions respectively.
$(\mathscr{C}_{3})$ agent shows one of highest-performing learning curves in
the three environments, even when the Stage-3 guidance may be unhelpful as in
Level 3. Both the Toddler-inspired S2D reward transition agents and the only
dense reward agents show substantial standard deviation, which we claim is due
to the difficulty of these RWARE tasks, which involve subgoals.

#### Results on RWARE environments.

The purpose of this study is to examine the effect of different Toddler-
inspired S2D reward transition on learning the tasks in RWARE that involve a
sequence of subgoals in order to clearly observe the sub-optimal situation
following the Toddler-inspired S2D models near the transition times. The
experiments investigated the Toddler-inspired S2D reward transition models at
the timing of three different stage transitions ( $\mathscr{C}_{1}$,
$\mathscr{C}_{2}$, and $\mathscr{C}_{3}$), as well as only dense reward
applied to learn with only Stage-2 ($\mathscr{C}_{4}$) or Stage-3
($\mathscr{C}_{5}$) guidance provided. The transition times are the same as
those specified in Figure 7, with $N$ being set to 1 million (1M) time steps.
PPO (Schulman et al. 2017) was used as the RL algorithm, and the means and
standard deviations were recorded across five trials. We analyze the results
below:

  * •

Level1. (Figure 13-(a)). Among the Toddler-inspired S2D reward transition
agents, the $\mathscr{C}_{3}$ agent performs the best, while the
$\mathscr{C}_{1}$ agent fails to deliver any shelves after the 4M time step.
While the Only 2 ($\mathscr{C}_{4}$) and Only 3 ($\mathscr{C}_{5}$) agents
achieve remarkable performances, $\mathscr{C}_{3}$ agent performs on par with,
if not better than, these only dense reward agents.

  * •

Level2. (Figure 13-(b)). Similarly to the results in Level-1, the
$\mathscr{C}_{3}$ agent performs the best, followed by $\mathscr{C}_{2}$
agent. The Only 2 ($\mathscr{C}_{4}$) and Only 3 ($\mathscr{C}_{5}$) agents
achieve performances similar to that of $\mathscr{C}_{3}$ agent in this
problem as well.

  * •

Level3. (Figure 13-(c)). The $\mathscr{C}_{3}$ agent achieves the top
performance in this most difficult setting. It should be noted that, despite
our original intent, the Stage-3 guidance was not beneficial in learning to
solve the task, as indicated by the performance of Only 3 agent
($\mathscr{C}_{4}$). Nevertheless, $\mathscr{C}_{3}$ agent showed the greatest
robustness to such detrimental reward.

#### Overall analysis on RWARE environments.

In Figure 14, we illustrate the policy trajectories of various agents in
RWARE-Level1. The goal, requested shelves, and non-requested shelves are
marked as green, red, and blue squares respectively. Here, the starting point
of the agent is the same as the goal position (green). The $\mathscr{C}_{1}$
agent picks incorrect shelves twice, and then halts exploration. The
$\mathscr{C}_{2}$ agent initially picks the correct shelf and delivers it, but
inefficiently explores the destination area in subsequent attempts. Unlike
these two agents, the $\mathscr{C}_{3}$ agent delivers more shelves
efficiently and experiences fewer subgoal failures. This highlights the impact
of reward transition times on task performance.

Figure 14: (a) Visualization of the trajectory for each agent policy in RWARE-
Level1. (b) Three levels of RWARE environments.

#### Importance of early free exploration.

We also observe that the $\mathscr{C}_{1}$ and $\mathscr{C}_{2}$ agents
perform poorly when the first stage transition is given early, before the
point of 3M training steps, as seen in Figure 13. Therefore, we believe that a
proper amount of exploration stage (Stage-1) given sparse reward is essential
to successful learning. For Only 3 agent ($\mathscr{C}_{5}$), even with the
richest reward, the agent fails completely in Level3. However, we found that
our $\mathscr{C}_{3}$ agent is even robust to the unbeneficial reward through
first stage transition in Toddler-inspired S2D setting.

#### Limitations.

In the given context, the primary objective of this study was to investigate
the influence of reward transition timings on performance metrics in tasks
involving subgoals. The research documented the nuanced effects that the
timing of these transitions exerts on overall task performance. However, it
must be acknowledged that the establishment of an unequivocally optimal reward
shaping strategy for tasks embedded with such subgoals remains elusive.
Consequently, this delineates a clear avenue for future research,
necessitating a comprehensive exploration of optimized reward shaping
techniques. This includes, but is not limited to, a detailed examination of
potential-based methods within this specialized environment. Such
investigative efforts are projected to significantly improve the efficacy of
toddler-inspired reward transition guidance mechanisms, ultimately leading to
enhanced performance outcomes.

### C.2 Gridworld: Add-on Algorithms Experiments on 3D Loss Landscape

Figure 15: Gridworld-navigation task. Left: Potential-based dense reward
environment of $10\times 10$ with PPO. Center: sparse reward environment of
$10\times 10$ with PPO. Right-Top: sparse reward environment of $4\times 4$
with DQN. Right-Bottom: Potential-based dense reward environment of $4\times
4$ with DQN.

We conducted 3D visualization of the policy landscape and experiments on
performance in the Gridworld environment, which we selected due to its
relative simplicity and minimal environmental factors. This setup provided a
more straightforward context in which to observe the results according to both
the S2D reward transition and the baseline, thereby offering clearer insights
into the implications of our work. In the main text, we focused solely on
visualizing the policy loss landscape using the SAC algorithm (Haarnoja et al.
2018). To verify whether the smoothing effects still persist in other various
algorithms, we conduced additional experiments using the DQN (Mnih et al.
2015) and PPO (Schulman et al. 2017).

#### Environment setup.

The experimental environment is a gridworld, as shown in Figure 15. The agent
can perform actions of moving in four directions: up, down, left, and right.
It receives a living penalty of $-0.1$ to encourage exploration. The
experiment is conducted under two environment settings: (1) a fixed goal
$4\times 4$ environment with DQN (Mnih et al. 2015) and (2) a random goal
$10\times 10$ environment with PPO (Schulman et al. 2017). Initially, we found
an appropriate stage transition: $T=200$ for a fixed goal within a total of
1000 steps, and $T=5000$ for a random goal within a total of 100,000 steps.
The neural network used for learning (Mnih et al. 2015; Schulman et al. 2017)
consists of three fully connected layers with ReLU activation. The batch size
is set to 128. In the case of PPO, we updated every 2 episodes. We employed
the Cross-Density Visualizer strategy.

#### The result of loss landscape in DQN and PPO algorithms.

We visualized PPO for the policy loss landscape and DQN for the Q-function
loss landscape. This is because, while DQN (Deep Q-Network) primarily aims to
learn a Q-value function, observing the Q-value loss landscape can offer
valuable insights into the network’s learning dynamics and its adaptability to
varied reward schemes. For both Gridworld-DQN and PPO, only the Toddler-
inspired S2D reward transition exhibits a notable smoothing effect compared to
other baselines, as depicted in Figures 16, 17, and 18(b), achieving the
highest performance. However, as seen in Figures 17 and 18-(a), we observe
little to no difference between the dense and sparse loss landscapes as the
number of updates increases. Therefore, we conclude that proceeding in a
sparse-to-dense manner, i.e., using Toddler-inspired S2D reward transition, is
an effective method for reducing the depth of local minima. It proves to be
significantly more effective than the dense-to-sparse (D2S) or Only Dense
approach.

### C.4 Detailed 3D Loss Landscape Visualizations for Gridworld-PPO and DQN
Algorithm

### Gridworld-PPO: Sparse-to-Dense (S2D) & Sparse-to-Sparse (Only Sparse)

Figure 16: Visualization of the 3D policy loss landscape after stage-
transitions in sparse-to-dense (Toddler-inspired S2D, $\mathscr{C}_{3}$, blue)
and sparse-to-sparse (Only Sparse, red). Here, we also observed notable
smoothing effects (reduction in the depth of local minima) under Toddler-
inspired S2D reward transition in PPO algorithm.

### Gridworld-PPO: Dense-to-Sparse (D2S) & Dense-to-Dense (Only Dense)

Figure 17: Visualization of the 3D policy loss landscape after stage-
transitions in dense-to-sparse (D2S, $\mathscr{C}_{3}$, red) and dense-to-
dense (Only Dense, blue). Similar to other D2S and Only Dense results in
Appendix Section B, we also cannot observe notable smoothing effects
consistent with the number of updates within Gridworld-PPO.

### Gridworld-DQN: S2D, Only Sparse & D2S, Only Dense

Figure 18: Visualization of the 3D Q-value loss landscape after reward
transitions in Toddler-inspired S2D and other baselines. A smoothing effect is
not apparent for the baselines (D2S, Only Dense, Only Sparse). However, under
Toddler-inspired S2D reward transition in (b) exclusively, a clear smoothing
effect can be observed after transitioning to potential-based dense rewards by
reducing the depth of local minima.

## Additional Future Works

The approach examined in our research is inspired by the modeling of bio-
inspired systems, particularly those based on toddler behaviors. Within this
framework, our future research endeavors will focus on a deeper exploration of
bio-plausible networks or models. We propose the following areas for future
investigation, which we believe are pivotal for advancing our understanding at
two critical junctures.

  1. 1.

Reward Transition Patterns in SNN Based RL: One promising avenue is the
utilization of spiking neural networks (SNNs), which offer a more biologically
realistic simulation of neural activity. Building upon bio-plausible SNN based
RL research (Qin, Yan, and Tang 2022; Zhang et al. 2022b, a), we intend to
investigate how SNNs can model and potentially enhance our understanding of
reward transition dynamics in RL scenarios. By integrating toddler-inspired
approaches into SNN research, we could provide novel insights into how rewards
are processed and learned over time in SNN.

  2. 2.

Exploring the Critical Period: Optimal Reward Transition Timing: Another
intriguing aspect is the initial period of free exploration in RL, which is
crucial for learning effective strategies in our toddler-inspired reward
transition. Through empirical experimentation, we were able to observe that
the timing of the reward transition with proper guidance significantly affects
subsequent performance and is a crucial parameter. This phenomenon can also be
considered similar to the concept of the ’critical period’ observed in
infants. In human infants, appropriate training or proper guidance in the
early stages of learning significantly impacts future learning. Moreover, the
importance of suitable early education is evident from research on RL models
(Park et al. 2021; De Kleijn, Sen, and Kachergis 2022) and neural networks
(Achille, Rovere, and Soatto 2018) regarding this critical period. To advance
this research area, our analysis, which includes the smoothing effect and low
sharpness metrics of the S2D model, suggests that exploring the optimal timing
for reward transition during the initial learning phase presents a promising
avenue for future studies.

We hope that our research will contribute to a broader understanding of neural
network architectures and their learning mechanisms, bridging the gap between
artificial intelligence and biological or cognitive modeling systems.

##  Appendix D Pytorch-like Code for Loss Landscape Rendering

Algorithm 1 Perpendicular Random Directions Generation

1:$\text{rand\\_direction\\_list1}\leftarrow[]$

2:$\text{rand\\_direction\\_list2}\leftarrow[]$

3:for each $param$ in $network.parameters()$ do

4: $r\leftarrow\text{torch.randn\\_like}(param)$

5: $r2\leftarrow\text{torch.randn\\_like}(param)$

6: $r\leftarrow
r-\left(\frac{\text{torch.dot}(r.\text{reshape}(-1),r2.\text{reshape}(-1))}{\|r2.\text{reshape}(-1)\|^{2}}\right)*r2$

7: $r\leftarrow\left(\frac{r}{\|r\|}\right)*\|param.\text{detach}()\|$

8: $r2\leftarrow\left(\frac{r2}{\|r2\|}\right)*\|param.\text{detach}()\|$

9:
$\text{rand\\_direction\\_list1}.\text{append}(\text{torch.nan\\_to\\_num}(r))$

10:
$\text{rand\\_direction\\_list2}.\text{append}(\text{torch.nan\\_to\\_num}(r2))$

11:end for

Algorithm 2 Deterministic Sampling

1:class ReplayBuffer:

2: def __init__(self, memory_size):

3: self.memory = deque(maxlen=memory_size)

4: self.experience = namedtuple("Experience",

5: field_names=["state", "action", "reward", "next_state", "done", "episode"])

6: # The rest is the same as a conventional replaybuffer.

7: def deterministic_sample(self, size):

8: idx = min(size-1, len(self.memory)-1)

9: experiences = [self.memory[i] for i in range(len(self.memory)-1-idx,
len(self.memory)-1)]

10:

11: states = np.vstack([e.state for e in experiences if e is not None])

12: actions = np.vstack([e.action for e in experiences if e is not None])

13: rewards = np.vstack([e.reward for e in experiences if e is not None])

14: next_states = np.vstack([e.next_state for e in experiences if e is not
None])

15: dones = np.vstack([e.done for e in experiences if e is not
None]).astype(np.bool_)

16:

17: min_epi = self.memory[0].episode

18: max_epi = self.memory[idx - 1].episode

19:

20: return states, actions,

21: np.squeeze(rewards, axis=1), next_states,

22: np.squeeze(dones, axis=1)

Algorithm 3 Network Base

1:class NetworkBase:

2: def __init__(self):

3: self.value_net = ValueNetwork(state_dim, hidden_dim)

4: self.target_value_net = ValueNetwork(state_dim, hidden_dim)

5:

6: self.soft_q_net1 = SoftQNetwork(state_dim, action_dim, hidden_dim)

7: self.soft_q_net2 = SoftQNetwork(state_dim, action_dim, hidden_dim)

8: self.policy_net = PolicyNetwork(state_dim, action_dim, hidden_dim)

Algorithm 4 Network Copy

1:function replica(self)

2: $result\leftarrow\text{NetworkBase}(\text{self.device})$

3: for each $target\\_param,param$ in
$\text{zip}(result.value\\_net.parameters(),\text{self.value\\_net.parameters()}):$
do

4: $target\\_param.data.copy\\_(param.data)$ $\triangleright$ Repeated for all
networks.

5: end for

6: return $result$

7:end function

Algorithm 5 Loss Landscape Calculation for Policy

1:$state,action,reward,next\\_state,done\leftarrow\text{replay\\_buffer.deterministic\\_sample()}$

2:$xs\leftarrow\text{torch.linspace}(-10,10,\text{steps}=50)$

3:$ys\leftarrow\text{torch.linspace}(-10,10,\text{steps}=50)$

4:$xs,ys\leftarrow\text{torch.meshgrid}(xs,ys,\text{indexing}=^{\prime}xy^{\prime})$

5:$zs\leftarrow\text{torch.zeros\\_like}(xs)$

6:for $i$ in $\text{range}(xs.\text{shape}[0])$ do

7: for $j$ in $\text{range}(ys.\text{shape}[1])$ do

8: $new\\_model\leftarrow\text{self.replica()}$

9: for $param,new\\_param,rand\\_dir1,rand\\_dir2$ in

10: zip$($self.policy_net.parameters(), new_model.policy_net.parameters(),

11: self.rand_direction_list1, self.rand_direction_list2$)$ do

12: $transformed\leftarrow param+xs[i][j]*rand\\_dir1+ys[i][j]*rand\\_dir2$

13: $new\\_param.\text{copy\\_}(transformed)$

14: end for

15:
$new\\_a,\text{log\\_prob},\epsilon,\text{mean},\text{log\\_std}\leftarrow\text{new\\_model.policy\\_net.evaluate}(state)$

16: $predicted\\_new\\_q\\_value\leftarrow\text{torch.min}($

17: $\text{new\\_model.soft\\_q\\_net1}(state,new\\_a),$

18: $\text{new\\_model.soft\\_q\\_net2}(state,new\\_a))$

19:
$zs[i][j]\leftarrow(\text{log\\_prob}-\text{predicted\\_new\\_q\\_value}).\text{mean}().\text{item}()$

20: end for

21:end for

22:return $xs.\text{numpy}(),ys.\text{numpy}(),zs.\text{numpy}()$

Algorithm 6 Cross-Density Visualizer

1:$dense\\_agent\leftarrow\text{NetworkBase}(\text{device}=device)$

2:$dense\\_replay\\_buffer\leftarrow\text{ReplayBuffer}(\text{replay\\_buffer\\_size})$

3:for $episode$ in $\text{trange}(transition\\_episode):$ do

4: $state,\\_\leftarrow\text{env.reset}(\text{seed}=seed)$

5: for $step$ in $\text{range}(\text{max\\_step}):$ do

6: $action\leftarrow\text{env.action\\_space.sample}()$

7: $next\\_state,reward,done,\\_,\\_$

8: $=\text{env.step}(action)$

9: $reward\leftarrow\text{dense\\_reward}(next\\_state,reward)$

10:
$dense\\_replay\\_buffer.\text{push}(state,action,reward,next\\_state,done,episode)$

11: $state\leftarrow next\\_state$

12: $dense\\_agent.\text{update}(dense\\_replay\\_buffer)$

13: if $done$ then

14: break

15: end if

16: end for

17: $\text{draw\\_landscape}()$

18:end for

19:$\triangleright$ Cross-Density Running

20:$dense\\_to\\_sparse\\_agent\leftarrow dense\\_agent.\text{replica}()$

21:$dense\\_to\\_sparse\\_replay\\_buffer\leftarrow\text{copy.deepcopy}(dense\\_replay\\_buffer)$

22:for $episode$ in
$\text{trange}(transition\\_episode,\text{max\\_episodes}):$ do

23: $state,\\_\leftarrow\text{env.reset}(\text{seed}=seed)$

24: for $step$ in $\text{range}(\text{max\\_step}):$ do

25: $action\leftarrow
dense\\_agent.policy\\_net.\text{get\\_action}(state).\text{detach}()$

26: $next\\_state,reward,done,\\_,\\_$

27: $=\text{env.step}(action.\text{numpy}())$

28: $reward\leftarrow\text{dense\\_reward}(next\\_state,reward)$

29:
$dense\\_replay\\_buffer.\text{push}(state,action,reward,next\\_state,done,episode)$

30: $state\leftarrow next\\_state$

31: $\\_\leftarrow dense\\_agent.\text{update}(dense\\_replay\\_buffer)$

32: if $done$ then

33: break

34: end if

35: end for

36:

37: $state,\\_\leftarrow\text{env.reset}(\text{seed}=seed)$

38: for $step$ in $\text{range}(\text{max\\_step}):$ do

39: $action\leftarrow
dense\\_to\\_sparse\\_agent.policy\\_net.\text{get\\_action}(state).\text{detach}()$

40: $next\\_state,reward,done,\\_,\\_$

41: $=\text{env.step}(action.\text{numpy}())$

42:
$dense\\_to\\_sparse\\_replay\\_buffer.\text{push}(state,action,reward,next\\_state,done,episode)$

43: $state\leftarrow next\\_state$

44: $\\_\leftarrow
dense\\_to\\_sparse\\_agent.\text{update}(dense\\_to\\_sparse\\_replay\\_buffer)$

45: if $done$ then

46: break

47: end if

48: end for

49:

50: $\text{draw\\_landscape}()$

51:end for