# Generalization, Mayhems and Limits in Recurrent Proximal Policy Optimization

Marco Pleines  Matthias Pallasch  Frank Zimmer  Mike Preuss

# Supplementary Material

Marco Pleines  Matthias Pallasch  Frank Zimmer  Mike Preuss

###### Abstract

At first sight it may seem straightforward to use recurrent layers in Deep
Reinforcement Learning algorithms to enable agents to make use of memory in
the setting of partially observable environments. Starting from widely used
Proximal Policy Optimization (PPO), we highlight vital details that one must
get right when adding recurrence to achieve a correct and efficient
implementation, namely: properly shaping the neural net’s forward pass,
arranging the training data, correspondingly selecting hidden states for
sequence beginnings and masking paddings for loss computation. We further
explore the limitations of recurrent PPO by benchmarking the contributed novel
environments Mortar Mayhem and Searing Spotlights that challenge the agent’s
memory beyond solely capacity and distraction tasks. Remarkably, we can
demonstrate a transition to strong generalization in Mortar Mayhem when
scaling the number of training seeds, while the agent does not succeed on
Searing Spotlights, which seems to be a tough challenge for memory-based
agents.

Deep Reinforcement Learning, Proximal Policy Optimization, Truncated
Backpropagation Through Time, Recurrence, GRU, LSTM, Recurrent Policy Gradient

##  1 Introduction

Leveraging recurrent neural networks in Deep Reinforcement Learning (DRL)
enabled agents to possess memory that can provide information on the history
of the current task that is not present in the agent’s contemporary
observation. Exploiting such memory is necessary to solve many Partially
Observable Markov Decision Processes (POMDP). This led to many emergent and
compelling agent behaviors that play complex games like Capture the flag
(Jaderberg et al., 2018), StarCraft ii@ (Vinyals et al., 2019), DotA 2 (Berner
et al., 2019) and Hide and Seek (Baker et al., 2020). The utilization of
recurrent layers, such as the widely used long short-term memory (LSTM)
(Hochreiter & Schmidhuber, 1997) and gated recurrent unit (GRU) (Cho et al.,
2014), notably increases the complexity and thus the error-proneness of DRL
algorithms.

Figure 1: CartPole can be turned into a POMDP by removing the velocity from
the observation space. For instance, if paddings are not masked during loss
computation, somewhat good results might mislead to false conclusions. Each
experiment was trained 5 times and evaluated on 10 novel seeds 3 times.

Adding recurrence to DRL algorithms, as PPO, seems to be largely considered
straightforward, which leaves a lot of details to engineering that however
plays a vital role in obtaining a correctly working and efficient
implementation111Bugs that hindered progress are described in appendix A.. We
may consider four major issues that must be resolved:

  * •

efficiently organizing the model’s forward pass

  * •

data handling concerning unequal sequence lengths

  * •

correctly sampling hidden states for sequence starts

  * •

mask paddings while computing the loss

Correctly setting up the way training data is structured, propagated through
the neural net and used for optimization is quite an effort and prone to
errors. There is also the risk of misleading results on low-scale problems to
verify an implementation’s correctness, as portrayed by figure 1. It may be
for this reason that how to do it has not been thoroughly documented in
literature according to our knowledge. This shortcoming entails possible
differences in research works that may actually lead to quite diverse results.

We believe this work is the first to precisely describe how to successfully
add a recurrent layer to the on-policy gradient algorithm Proximal Policy
Optimization (PPO) (Schulman et al., 2017), focusing especially on pixel input
(visual observation) and discrete action spaces. Additionally, a densely
commented and documented baseline
implementation222https://github.com/MarcoMeter/recurrent-ppo-truncated-bptt,
supporting truncated backpropagation through time, is contributed.

Next to these algorithmic adaptations, we also contribute two novel
environments challenging the agent’s memory, called Mortar Mayhem and Searing
Spotlights. These environments enrich the landscape of assessing memory-based
agents, because these are not solely limited to capacity and distraction
problems, as most well-established environments in this field. Examining the
capacity and the robustness to distractions is not unique to DRL as it can be
applied to a much broader context employing a recurrent, transformer or
similar architecture. Both environments require the agent not just to maintain
initial goal cues in its memory for many steps, but also to continuously
modify its memory to memorize its past behavior to solve the task at hand.

We show that Mortar Mayhem’s second task cannot be trained without memory.
Strong generalization is presented on its complete task, which is notably
related to scaling the number of training seeds. However, this effect does not
apply to the ballet environment, leaving us with weak generalization as
already shown by Lampinen et al. (2021). Searing Spotlights pushes recurrent
PPO to its limits. Only simplified versions of this environment are solvable
to some extent. Further, refreshing stale hidden states and advantages does
not provide a gain in performance that satisfies the gain in computational
cost.

##  2 Related Work

###  2.1 Proximal Policy Optimization and Recurrence

Revisiting literature on PPO helps to understand significant design and
implementation decisions (Henderson et al., 2018; Engstrom et al., 2020; Hsu
et al., 2020; Andrychowicz et al., 2021). Related work, in the domain of PPO
using recurrence on control tasks (MountainCar, Ant, HalfCheetha, ect.) with
continuous action spaces and state-based inputs (i.e. vector observation),
examines generalization (Packer et al., 2018), meta reinforcement learning (Yu
et al., 2019) and robust reinforcement learning (Zhang et al., 2021). None of
the aforementioned publications describe the specifics of getting recurrent
PPO to work, which is actually a non-trivial process. In contrast to those,
all our experiments are run on environments that feature discrete action
spaces and pixel input. Our work also details all the necessary components to
hook up a recurrent layer.

###  2.2 Applications of Recurrent Layers in DRL

Next to PPO, other algorithms were published that employ recurrence.
Hausknecht & Stone (2015) extended the Deep Q-Network (DQN) algorithm and
examined the way how episodes and their hidden states are sampled from the
experience replay buffer. More literature is available on adding recurrence to
variants of Deep Deterministic Policy Gradient (DDPG) algorithms (Heess et
al., 2015; Meng et al., 2021). Especially Ni et al. (2021) proposed a
recurrent Twin-Delayed DDPG implementation and shared how their replay buffer
is made more memory-efficient. The authors of the off-policy algorithm R2D2
(Recurrent Replay Distributed DQN) provide approaches to sample sequences from
the replay buffer, to initialize hidden states and to mitigate their staleness
(Kapturowski et al., 2019). Ndousse (2020) showed potential to mitigate such
staleness in PPO by refreshing the hidden states every few minibatch updates.
Contrary to their findings, our experiments show no notable improvement when
refreshing hidden states during optimization.

###  2.3 Partially Observable Environments as Benchmarks

Typical POMDPs for assessing recurrent DRL algorithms are the Morris water
maze (D’Hooge & De Deyn, 2001) and the T-maze (Wierstra et al., 2007). The
T-maze asks the agent to initially memorize a goal cue, then to traverse a
long alley and once the end is reached it has to choose the correct exit.
Inside the water maze, the agent has to find an invisible goal platform and
memorize its location. Once found, the agent is set to a random position and
has to find the goal again. The Memory Task Suite (Fortunato et al., 2019)
comprises 13 tasks to assess memory in the broader context of machine
learning. Some variants build on top of the idea of the water maze (Goal
Navigation), whereas others extend the mechanics of the T-maze (Spot the
Difference). A few environments, like dancing the ballet, are contributed by
Lampinen et al. (2021). Control tasks are normally fully observable. These are
turned into POMDPs by adding noise to the observations or by removing
information, such as the velocity, from the observation space. Atari games are
usually trained using a frame stack of past observations. Removing that stack
and adding flickering frames transforms these tasks to POMDPs as well
(Hausknecht & Stone, 2015). Our work contributes two novel environments
providing unique challenges to memory-based DRL agents.

Figure 2: The data preprocessing starts out by sampling trajectories across
$W$ workers for $T$ steps. Next, $E$ episodes of varying length $ET$ are
extracted from the trajectories. Those can be further split into $\\#$
sequences of varying length $\\#T$. At last, zero padding is used to retrieve
sequences of fixed length $max(\\#T)$.

##  3 Recurrent Proximal Policy Optimization

The agent is trained using Proximal Policy Optimization (PPO) using the
clipped surrogate objective (Schulman et al., 2017). Due to the usage of a
recurrent layer (e.g. LSTM), the to be selected action $a_{t}$ of the policy
$\pi_{\theta}$ depends on the current observation $o_{t}$ and hidden state
$h_{t}$ of the recurrent layer. $\hat{A}_{t}$ denotes advantage estimates
based on generalized advantage estimation (GAE), $\theta$ the parameters of a
neural net and $\epsilon$ the clip range.

$L^{C}_{t}(\theta)=\hat{\mathbb{E}}_{t}[min(q_{t}(\theta)\hat{A}_{t},clip(q_{t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{t})]$
(1) $\textnormal{with
ratio}~{}q_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|o_{t},h_{t})}{\pi_{\theta\text{old}}(a_{t}|o_{t},h_{t})}$

The value function is optimized using the squared-error loss
$L_{t}^{V}(\theta)$. $\mathcal{H}[\pi_{\theta}](o_{t})$ denotes an entropy
bonus encouraging exploration (Schulman et al., 2017). Both are weighted using
the coefficients $c_{1}$ and $c_{2}$ and are added to $L_{t}^{C}$ to complete
the loss:

$L^{CVH}_{t}(\theta)=\hat{\mathbb{E}}_{t}[L^{C}_{t}(\theta)-c_{1}L^{V}_{t}(\theta)+c_{2}\mathcal{H}[\pi_{\theta}](o_{t},h_{t})]$
(2)
$\textnormal{with}~{}L^{V}_{t}(\theta)=(V_{\theta}(o_{t},h_{t})-V_{t}^{targ})^{2}$

Figure 3: Recurrent PPO utilizes a feed-forward convolutional recurrent neural
network. First, visual and vector observations are encoded as an entire batch
by either convolutional or fully connected layers. $o_{t}$, $a_{t-1}$ and
$r_{t-1}$ denote the current observation, the last action and the last reward.
The encoded features are concatenated and then reshaped to sequences before
feeding them to the recurrent layer. Its output has to be reshaped into the
original batch shape. Further, the forward pass is divided into two streams
relating to the value function and the policy. The number of policy heads is
equal to the number of action dimensions given by a multi-discrete action
space.

###  3.1 Fundamental Implementation Details

Adding a recurrent layer to a model that is trained with PPO is not just plug-
and-play. To ensure an efficient and correct implementation, three major
components have to be touched: the processing of the sampled training data
(Figure 2), the forward pass of the model (Figure 3) and the loss function
(Equation 3).

The training data is sampled by a fixed number of workers for a fixed amount
of steps and is stored as a tensor. Each collected trajectory may contain
multiple episodes that might have been truncated. After sampling, the data has
to be split into episodes. Optionally, these can be further split into smaller
sequences of fixed length. Otherwise, the actual sequence length is equal to
the length of the longest episode. Episodes or sequences that are shorter than
the desired length are padded using zeros. As the data is structured into
fragments of episodes, the hidden states of the recurrent layer have to be
selected correctly. The output hidden state of the previous sequence is the
input hidden state of its consecutive one. This approach is also known as
truncated backpropagation through time (truncated bptt). Finally, minibatches
sample multiple sequences from the just processed data.

Concerning the forward pass of the model, it is more efficient to feed every
non-recurrent layer the entire batch of the data (i.e.
$batch~{}size=workers\times steps$) and not each sequence one by one. Whenever
the batch is going to be fed to a recurrent layer during optimization, the
batch has to be reshaped to the dimensions: number of sequences and sequence
length. After passing the sequences to the recurrent layer, the data has to be
reshaped again to the overall batch size. Note that the forward pass for
sampling trajectories operates on a sequence length of one. In this case, the
data keeps its shape throughout the entire forward pass.

Once the loss function is being computed, the padded values of the sequences
have to be masked so that these do not affect the gradients. $L^{mask}$ is the
average over all losses not affected by the paddings.

$L^{mask}(\theta)=\frac{\sum_{t}^{T}\left[mask_{t}\times
L^{CVH}_{t}(\theta)\right]}{\sum_{t}^{T}[mask_{t}]}$ (3)

$\textnormal{with}~{}mask_{t}=\begin{cases}0&\text{where padding is used}\\\
1&\text{where no padding is used}\\\ \end{cases}$

###  3.2 Refreshing Stale Advantages and Hidden States

Once the parameters of the model are updated using at least one minibatch, the
consecutive minibatch updates of the current iteration operate to some extent
on stale data. Andrychowicz et al. (2021) empirically identified that
recomputing advantage estimates before each epoch can be beneficial to policy
gradient algorithms given the on-policy setting. This measure can also be
applied to hidden states of a recurrent layer, as recurrent staleness was also
observed by Kapturowski et al. 2019. According to our knowledge, refreshing
stale hidden states were only briefly examined by Ndousse (2020) on the on-
policy algorithm PPO. The contributed recurrent PPO algorithm encompasses the
possibility to refresh advantages and hidden states before each epoch to
mitigate their staleness. Finally, algorithm 1 describes our realization of
recurrent PPO.

Algorithm 1 Recurrent PPO using Truncated BPTT

for iteration=$1,2,\ldots$ do

for worker=$1,2,\ldots,W$ do

Run policy $\pi_{\theta old}$ in environment for $T$ timesteps

end for

Compute advantage estimates $\hat{A}_{1,1},\ldots,\hat{A}_{T,W}$

Split trajectories into episodes

if fixed sequence length then

Split episodes into sequences of fixed length

else

sequences = episodes

end if

Zero pad sequences that are too short

Select initial hidden states of each sequence

for epoch do

for minibatch=$1,2,\ldots MB$ with size $M\leq WT$ do

Optimize $L^{mask}(\theta)$

end for

if epoch $>0$ then

Recompute hidden states and advantages w.r.t. $\theta$

end if

$\theta_{old}\leftarrow\theta$

end for

end for

##  4 Novel Environments

Many partially observable environments exist that challenge the agent’s
memory. The majority of them, such as MiniGrid Memory (Chevalier-Boisvert et
al., 2018), Spot the Difference (Fortunato et al., 2019) and Remembering the
Ballet (Lampinen et al., 2021) demand the agent’s memory to solely memorize
the to be observed goal cues during the very beginning of the episode. Once
this cue is memorized, there is no need to further manipulate the agent’s
memory. Simply maintaining it is enough to solve the task. It can be
hypothesized that the extracted features from observing the goal cues are
sufficient to solve the ballet environment. To show this, the ballet
environment can be made fully observable (markovization) by feeding the entire
sequence of cues to a recurrent encoder that extracts features, which are
utilized by the agent’s policy. Consequently, the policy itself does not need
to maintain a memory in this case. The task at hand gets much easier, because
the agent does not need to make obsolete decisions while observing the
sequence of cues as its position is frozen. To investigate a higher level of
complexity, the environments Mortar Mayhem and Searing Spotlights are
introduced next. These require the agent to properly manipulate its memory
throughout an entire episode, because it will be relevant to the agent to
memorize not just only initial observations (e.g. cues), but especially also
the past decisions that it has made. All possible environment configurations
and exemplary episodes can be found in Appendix B and C.

(a) Clue Task

(b) Act Task

(c) Act Only Task

Figure 4: In Mortar Mayhem, the agent has to memorize a sequence of commands
first (a) and then has to execute them one by one. Between commands, the agent
has 5 steps to move to the commanded tile. Once the time is up, lava tiles (b)
verify the correct position of the agent. The environment can also be run
given all commands at once during each step (c).

###  4.1 Mortar Mayhem

The Mortar Mayhem environment (Fig. 4) is inspired by the commercial video
game Pummel Party333http://rebuiltgames.com/ and is implemented using the
minimalistic gridworld environment (Chevalier-Boisvert et al., 2018). Two
tasks make up the dynamics of the environment. At first, the agent has to
memorize a sequence of commands (Clue Task) and afterwards it has to execute
each command in the observed order (Act Only Task). One command orders the
agent to move to one adjacent floor tile or to stay at the current one. If the
agent fails, the episode terminates, while receiving no reward. Upon
successfully executing one command, a reward of $+0.1$ is signaled. Mortar
Mayhem’s observation space comprises 490 values that are retrieved by encoding
the underlying grid world one-hot. Alternatively, the observation space can be
represented by $84\times 84\times 3$ pixels. The action space is discrete and
provides the actions move forward, rotate left, rotate right and stay. In
contrast to environments as the aforementioned ones, the agent has to
continuously modify its memory to solve the second task. It has to memorize
which orders were already taken to succeed on the consecutive command.

(a) Initial state

(b) In progress

Figure 5: In Searing Spotlights, the environment is fully observable for a few
initial steps (a) and then dims the light until off. The agent is the red
circle, while coins are illustrated by yellow ellipses. Over time, more and
more harmful spotlights move across the environment (b). The agent has to
collect all coins, while not being unveiled by a spotlight.

###  4.2 Searing Spotlights

Searing Spotlights (Figure 5) is inspired by Pummel Party as well and is
implemented using the ml-agents toolkit (Juliani et al., 2018). The agent’s
primary goal in this environment is to avoid being unveiled by spotlights.
Initially, fewer spotlights traverse the environment, but over time these are
more frequently spawned. Thus, the environment is less hostile early on, while
providing little information. As time progresses, more information is
available to the agent, but the environment is more dangerous. After all, the
agent is especially challenged to memorize its past decisions to estimate its
current location. To avoid numb policies, a coin collection task is added to
the environment. This ensures that the agent has to move across the
environment and therefore putting itself more at risk. Searing spotlights
features an input of $84\times 84\times 3$ pixels. The agent’s action space is
multi-discrete comprising two discrete dimensions. Moving left, right and not
moving horizontally are part of the first dimension. The second dimension
enables the agent to move up, down or not move vertically at all. Concerning
the reward function, the agent is punished for being visible inside a
spotlight by a reward of $-0.01$ for each step, while it is being rewarded by
$+1.0$ for collecting a coin. Terminal states are reached once the agent loses
all of his 100 hit points (spotlights damage the agent by $0.01$ every step)
or once all coins are collected.

Table 1: Parameters used for training. Vector observations combine the current observation, the last action (one-hot) and reward. The learning rate $\alpha$ and the entropy bonus coefficient $c_{2}$ decay linearly. The parameters on the bottom part are the same for each environment. Parameter | Mortar | Searing | Ballet
---|---|---|---
Seeds | 1000 | 1000 | 500
Visual Observation | 84$\times$84$\times$3 | 84$\times$84$\times$3 | 99$\times$99$\times$3
Vector Observation | 490+4+1 | 6+1 | 13+8+1
Action Space | 4 | 3, 3 | 8
Vector Encoder Size | 256 | 128 | 128
Number of Workers | 32 | 16 | 32
Batch Size | 16384 | 8192 | 16384
Epochs | 4 | 3 | 3
Initial $\alpha$ | 0.0003 | 0.0003 | 0.0003
Min $\alpha$ | 0.00003 | 0.0003 | 0.0003
Initial $c_{2}$ | 0.1 | 0.0001 | 0.001
Min $c_{2}$ | 0.0001 | 0.0001 | 0.0005
Sequence Length | Max | 128 | Max
Discount Factor | 0.99
Lamda (GAE) | 0.95
Worker Steps | 512
Minibatches | 8
$c_{1}$ | 0.25
Max Gradient Norm | 0.5
Clip Range | 0.2
Hidden State Size | 512
Recurrent Layer | GRU
Activations | ReLU
Optimizer | AdamW

##  5 Experiments

Every provided sample efficiency curve shows the interquartile mean (IQM) and
confidence interval (95%) as recommended by the evaluation library rliable
(Agarwal et al., 2021). Each experiment trained 5 agents. The result data is
retrieved from evaluating 10 training and 10 novel environment seeds, which
were repeated 3 times. Thus, one data point aggregates 150 episodes with
respect to training and novel seeds. Table 1 enumerates the used training
parameters. Based on grid search (Appendix D), the hidden state size, the
sequence length and the recurrent layer type were selected. GRU turned out to
be slightly more effective than LSTM in the trained environments. The source
code is available online444https://github.com/MarcoMeter/neroRL.

###  5.1 Act Only Task in Mortar Mayhem

Figure 6: Training performance of the agent with and without recurrence on the
Act Only Task in Mortar Mayhem.

To verify the second task of Mortar Mayhem, the agent is asked to complete 10
commands with and without recurrence, while all commands are visible during
each step throughout one episode. Figure 6 presents the training performance
where the observation space of the agent is varied by providing several
permutations of using visual or vector observations and whether the agent may
perceive its last action and last reward. Recurrence is essential to solve the
Act Task, while visual observations are slightly better than their vector
counterparts if no memory is present. Under these circumstances, the agent
slightly profits from sensing its last action, but this small impact is not
observed for its last reward. It can be argued that more model parameters
(e.g. due to the convolutional layers) and the info of the last action aid
overfitting. None of these training runs surpassed an IQM return of $0.1$
given novel seeds.

Figure 7: Performance on novel and training seeds in the ballet environment
assessed on either 2, 4, or 8 dances.

###  5.2 Generalization

Figure 8: Generalization performance on training either 200, 500 or 1000 seeds
in Mortar Mayhem.

Generalization is examined on Mortar Mayhem and the ballet environment. The
agent generalizes well in the former one when trained on 10 commands (Figure
8). However, 1000 seeds are required to achieve this performance, while 500
only reach an IQM return of less than $0.6$. The scale of 200 seeds is by far
inferior. Concerning the ballet environment (Figure 7), the agent has no
problems solving training seeds, while its performance on novel seeds may be
worse than chance555Chance depends on the number of dances.. Our results
pretty much confirm the ones reported by Lampinen et al. (2021). However,
there are a few apparent differences in the training setup. We trained the
agent only for about 25 million steps (not 5 billion). The dance to be
identified is one-hot encoded as part of the vector observation space, whereas
Lampinen et al. (2021) use a recurrent language encoder. During training, the
number of dances varied between 2, 4 and 8 dances, while the delay between
dances was held constant at 16. Scaling up the number of training seeds to
1000 does not yield any improvement over 500 ones.

###  5.3 Refreshing Stale Hidden States and Advantages

Figure 9: Generalization performance on novel seeds when stale hidden states
and advantages are refreshed in Mortar Mayhem.

Figure 10: Training performance on 8 dances when stale hidden states and
advantages are refreshed in the ballet environment.

Refreshing stale hidden states and advantages does not yield promising
results. Applying the refresh every other epoch slightly improves the
performance in Mortar Mayhem on novel seeds (Figure 9) and ballet on training
seeds (Figure 10). This minor improvement is likely not worth the cost of
another forward pass on the entire data, especially when the observation space
and the model are complex. Exploiting less stale data, using more epochs,
severely degrades performance in both environments.

Figure 11: Generalization performance on training either 10, 20 or 30 commands
in Mortar Mayhem.

###  5.4 Observed Limits

Reaching the limits of the underlying recurrent PPO implementation is not
easily captured using Mortar Mayhem (Figure 11). When trained on 20 or 30
commands, the highest possible return is $2.0$ or $3.0$. Both variants do not
fulfill this amplitude, but it can be considered that its solution is rather a
question of model scale and computation time. Concerning the ballet
environment, its limitation to generalize is already apparent (Figure 7). If
trained on a constant number of dances and constant delay, the agent is able
to generalize to novel seeds. A completely different impression is obtained on
Searing Spotlights. The agent completely fails at the entire task, which is
discussed in the next section.

##  6 Discussion

###  6.1 Scaling the Difficulty of Mortar Mayhem

As shown before, the agent needs memory to succeed in Mortar Mayhem. The
challenge of this environment is not just limited to the number of commands to
be executed by the agent. Applying modifications to Mortar Mayhem during
training, like varying the number of commands and the delays between showing
and executing each command, leads to a more complex distribution of training
scenarios similar to the one seen in dancing the ballet. Also, the reward
signal can be made sparse, by only providing a reward upon successfully
completing the entire command sequence. Such alterations may further add value
to challenging memory-based agents.

###  6.2 Challenges of Searing Spotlights

In Searing Spotlights, the agent is asked to collect all coins, while not
being unveiled by any of the moving spotlights. The agent’s observation
seldomly encompasses information on the whereabouts of the coins and itself,
because these are only visible during the beginning of the episode and while
being spotted by a light. Therefore, this important information has to be
inferred from past observations, which should be taken care of by the agent’s
memory. Recurrent PPO does not seem to find any policy to solve this
environment. This can be due to the agent not being able to directly impact
its observations for the most part of the episode, even though it perceives
its last action. So how should the agent be able to assign credit to its
performed decisions? The spotlights may worsen this dilemma by causing noise
and distractions due to their stochastic behavior (random spawn positions and
stochastic movement). To gain more insights, the environment can be ablated
using these simplifications:

  * •

removing spotlight penalties (i.e. damage and reward)

  * •

limiting the number of coins to one

  * •

increasing the scale of the agent and the coin

  * •

making the spotlights stationary

  * •

making the agent or the coin permanently visible

Applying the first two measures did not lead to successful training. Scaling
up the size of the agent and the coin by an order of two or three failed as
well. Once spotlights are stationary, the agent easily learns to collect the
coin, but only when its scale is set to three666More figures, showing the
scale of the agent and stationary spotlights, can be found in Appendix C.2..
Lowering the scale to two does not always work out like before. The original
scale did not solve the task under stationary spotlights. If the agent is
scaled up and always visible, but the spotlights move, the task remains
unsolved. We did not explore all permutations of available simplifications
yet, but it seems that moving spotlights are the toughest challenge in Searing
Spotlights. To overcome their threat of distraction, a completely different
approach to recurrent PPO shall be considered. We believe that a transformer
architecture as episodic memory, like the Hierarchical Chunk Attention Memory
(Lampinen et al., 2021), is more capable and robust to distractions. Doing so,
the agent could recall every single extracted feature from past observations
to filter out spotlights and to better trace itself. This, however, is left to
future work.

###  6.3 Staleness of Hidden States and Advantages

Our observed results differ from the ones reported by Ndousse (2020) and
Andrychowicz et al. (2021). The monitored gradient norms and the
Kullback–Leibler divergence between $\theta_{old}$ and $\theta$ do not yield
further insights (Appendix E). It seems that the issue of staleness in DRL
needs further examination. On-policy data might not suffer as much as off-
policy ones from being stale. This issue might be amplified in distributed
training setups. Maybe there is an impact on how advantage estimates are
normalized. Hsu et al. 2020 provide supplementary material where some agents
performed worse on control tasks using normalization. Ndousse 2020 normalized
advantages across single episodes, while most implementations normalize those
across minibatches or the entire batch of data. It shall be investigated
whether refreshing hidden states and advantages mitigate a higher variance
when smaller chunks of data are used for normalization.

###  6.4 Open Threads concerned with Recurrence

More algorithmic details are left to future work. One concerns the naive
initialization of hidden states to zero. Making the initial states learnable
parameters as part of $\theta$ is non-trivial, because gradients are truncated
and therefore cannot be backpropagated to the hidden states’ origin.
Kapturowski et al. 2019 employed a ’burn-in’ approach that produces a start
hidden state by feeding the model a fragment of a sequence, while the
remaining items of the sequence are used for optimization. However, it remains
unclear how an agent shall initialize its hidden state when acting without the
experience replay buffer used during training. Moreover, future work can
examine the utility of augmenting the training data by overlapping sequences,
which was also touched by Kapturowski et al. 2019. To break impairing
correlations (Mnih et al., 2015), which were reintroduced due to maintaining
sequences, a stochastically positioned sliding window could be used to sample
sequences.

##  7 Conclusion

In this paper, we demonstrated how to resolve the problems that originate from
adding recurrence to the widely used PPO algorithm. In more detail, these are
concerned with the proper arrangement of the training data, the efficient
organization of the neural net’s forward pass, the correct sampling of hidden
states related to sequence beginnings and the masking of paddings during loss
computation. Getting those right is essential to achieve an efficient and
working implementation.

We further contributed the environments Mortar Mayhem and Searing Spotlights
that challenge memory-based agents beyond solely capacity and distraction
tasks. These and the ballet environment are used to assess generalization and
limitations of recurrent PPO. Remarkably, when scaling up the number of
training seeds in Mortar Mayhem, the agent is capable of generalizing well to
novel seeds. However, this is not observed in the ballet environment where the
performance on training seeds is great, but worse than chance on novel ones.
Refreshing potentially stale hidden states and advantages did not improve
results in our experiments.

At last, clear limitations of recurrent PPO are unveiled by trying to solve
Searing Spotlights. Only a heavily simplified version of this environment is
solvable by the taken approach. It seems that randomly moving spotlights cause
too much noise and distraction to the agent. To remedy the problems posed by
Searing Spotlights, we see the potential for future work on adding an episodic
memory to the agent via transformer architectures.

## References

  * Agarwal et al. (2021) Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., and Bellemare, M. G.  Deep reinforcement learning at the edge of the statistical precipice.  In _Thirty-Fifth Conference on Neural Information Processing Systems_ , 2021.
  * Andrychowicz et al. (2021) Andrychowicz, M., Raichuk, A., Stańczyk, P., Orsini, M., Girgin, S., Marinier, R., Hussenot, L., Geist, M., Pietquin, O., Michalski, M., Gelly, S., and Bachem, O.  What matters for on-policy deep actor-critic methods? a large-scale study.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=nIAxjsniDzg.
  * Baker et al. (2020) Baker, B., Kanitscheider, I., Markov, T. M., Wu, Y., Powell, G., McGrew, B., and Mordatch, I.  Emergent tool use from multi-agent autocurricula.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020.  URL https://openreview.net/forum?id=SkxpxJBKwS.
  * Berner et al. (2019) Berner, C., Brockman, G., Chan, B., Cheung, V., Debiak, P., Dennison, C., Farhi, D., Fischer, Q., Hashme, S., Hesse, C., Józefowicz, R., Gray, S., Olsson, C., Pachocki, J., Petrov, M., de Oliveira Pinto, H. P., Raiman, J., Salimans, T., Schlatter, J., Schneider, J., Sidor, S., Sutskever, I., Tang, J., Wolski, F., and Zhang, S.  Dota 2 with large scale deep reinforcement learning.  _CoRR_ , abs/1912.06680, 2019.  URL http://arxiv.org/abs/1912.06680.
  * Chevalier-Boisvert et al. (2018) Chevalier-Boisvert, M., Willems, L., and Pal, S.  Minimalistic gridworld environment for openai gym.  https://github.com/maximecb/gym-minigrid, 2018.
  * Cho et al. (2014) Cho, K., van Merrienboer, B., Gülçehre, Ç., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y.  Learning phrase representations using RNN encoder-decoder for statistical machine translation.  In Moschitti, A., Pang, B., and Daelemans, W. (eds.), _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL_ , pp. 1724–1734. ACL, 2014.  doi: 10.3115/v1/d14-1179.  URL https://doi.org/10.3115/v1/d14-1179.
  * D’Hooge & De Deyn (2001) D’Hooge, R. and De Deyn, P. P.  Applications of the morris water maze in the study of learning and memory.  _Brain research reviews_ , 36(1):60–90, 2001\.
  * Engstrom et al. (2020) Engstrom, L., Ilyas, A., Santurkar, S., Tsipras, D., Janoos, F., Rudolph, L., and Madry, A.  Implementation matters in deep rl: A case study on ppo and trpo.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=r1etN1rtPB.
  * Fortunato et al. (2019) Fortunato, M., Tan, M., Faulkner, R., Hansen, S., Badia, A. P., Buttimore, G., Deck, C., Leibo, J. Z., and Blundell, C.  Generalization of reinforcement learners with working and episodic memory.  In _Advances in Neural Information Processing Systems_ , pp. 12448–12457, 2019.
  * Hausknecht & Stone (2015) Hausknecht, M. J. and Stone, P.  Deep recurrent q-learning for partially observable mdps.  In _2015 AAAI Fall Symposia, Arlington, Virginia, USA, November 12-14, 2015_ , pp. 29–37. AAAI Press, 2015.  URL http://www.aaai.org/ocs/index.php/FSS/FSS15/paper/view/11673.
  * Heess et al. (2015) Heess, N., Hunt, J. J., Lillicrap, T. P., and Silver, D.  Memory-based control with recurrent neural networks.  _CoRR_ , abs/1512.04455, 2015.  URL http://arxiv.org/abs/1512.04455.
  * Henderson et al. (2018) Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger, D.  Deep reinforcement learning that matters.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 32(1), Apr. 2018.  URL https://ojs.aaai.org/index.php/AAAI/article/view/11694.
  * Hochreiter & Schmidhuber (1997) Hochreiter, S. and Schmidhuber, J.  Long Short-Term Memory.  _Neural Computation_ , 9(8):1735–1780, 11 1997\.  ISSN 0899-7667.  doi: 10.1162/neco.1997.9.8.1735.  URL https://doi.org/10.1162/neco.1997.9.8.1735.
  * Hsu et al. (2020) Hsu, C. C., Mendler-Dünner, C., and Hardt, M.  Revisiting design choices in proximal policy optimization.  _CoRR_ , abs/2009.10897, 2020.  URL https://arxiv.org/abs/2009.10897.
  * Jaderberg et al. (2018) Jaderberg, M., Czarnecki, W. M., Dunning, I., Marris, L., Lever, G., Castañeda, A. G., Beattie, C., Rabinowitz, N. C., Morcos, A. S., Ruderman, A., Sonnerat, N., Green, T., Deason, L., Leibo, J. Z., Silver, D., Hassabis, D., Kavukcuoglu, K., and Graepel, T.  Human-level performance in first-person multiplayer games with population-based deep reinforcement learning.  _CoRR_ , abs/1807.01281, 2018.  URL http://arxiv.org/abs/1807.01281.
  * Juliani et al. (2018) Juliani, A., Berges, V., Vckay, E., Gao, Y., Henry, H., Mattar, M., and Lange, D.  Unity: A general platform for intelligent agents.  _CoRR_ , abs/1809.02627, 2018.  URL http://arxiv.org/abs/1809.02627.
  * Kapturowski et al. (2019) Kapturowski, S., Ostrovski, G., Dabney, W., Quan, J., and Munos, R.  Recurrent experience replay in distributed reinforcement learning.  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=r1lyTjAqYX.
  * Lampinen et al. (2021) Lampinen, A. K., Chan, S. C. Y., Banino, A., and Hill, F.  Towards mental time travel: a hierarchical memory for reinforcement learning agents.  _CoRR_ , abs/2105.14039, 2021.  URL https://arxiv.org/abs/2105.14039.
  * Meng et al. (2021) Meng, L., Gorbet, R., and Kulic, D.  Memory-based deep reinforcement learning for pomdps.  In _IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2021, Prague, Czech Republic, September 27 - Oct. 1, 2021_ , pp. 5619–5626. IEEE, 2021.  doi: 10.1109/IROS51168.2021.9636140.  URL https://doi.org/10.1109/IROS51168.2021.9636140.
  * Mnih et al. (2015) Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., and Hassabis, D.  Human-level control through deep reinforcement learning.  _Nature_ , 518(7540):529–533, 02 2015.  ISSN 0028-0836.  URL http://dx.doi.org/10.1038/nature14236.  Letter.
  * Ndousse (2020) Ndousse, K.  Stale hidden states in ppo-lstm, 2020.  URL https://kam.al/blog/ppo_stale_states/.
  * Ni et al. (2021) Ni, T., Eysenbach, B., and Salakhutdinov, R.  Recurrent model-free RL is a strong baseline for many pomdps.  _CoRR_ , abs/2110.05038, 2021.  URL https://arxiv.org/abs/2110.05038.
  * Packer et al. (2018) Packer, C., Gao, K., Kos, J., Krähenbühl, P., Koltun, V., and Song, D.  Assessing generalization in deep reinforcement learning.  _CoRR_ , abs/1810.12282, 2018.  URL http://arxiv.org/abs/1810.12282.
  * Schulman et al. (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O.  Proximal policy optimization algorithms.  _arXiv:1707.06347_ , 2017.
  * Vinyals et al. (2019) Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., Choi, D. H., Powell, R., Ewalds, T., Georgiev, P., Oh, J., Horgan, D., Kroiss, M., Danihelka, I., Huang, A., Sifre, L., Cai, T., Agapiou, J. P., Jaderberg, M., Vezhnevets, A. S., Leblond, R., Pohlen, T., Dalibard, V., Budden, D., Sulsky, Y., Molloy, J., Paine, T. L., Gülçehre, Ç., Wang, Z., Pfaff, T., Wu, Y., Ring, R., Yogatama, D., Wünsch, D., McKinney, K., Smith, O., Schaul, T., Lillicrap, T. P., Kavukcuoglu, K., Hassabis, D., Apps, C., and Silver, D.  Grandmaster level in starcraft II using multi-agent reinforcement learning.  _Nat._ , 575(7782):350–354, 2019.  doi: 10.1038/s41586-019-1724-z.  URL https://doi.org/10.1038/s41586-019-1724-z.
  * Wierstra et al. (2007) Wierstra, D., Förster, A., Peters, J., and Schmidhuber, J.  Solving deep memory pomdps with recurrent policy gradients.  In de Sá, J. M., Alexandre, L. A., Duch, W., and Mandic, D. P. (eds.), _Artificial Neural Networks - ICANN 2007, 17th International Conference, Porto, Portugal, September 9-13, 2007, Proceedings, Part I_ , volume 4668 of _Lecture Notes in Computer Science_ , pp. 697–706. Springer, 2007.  doi: 10.1007/978-3-540-74690-4“˙71.  URL https://doi.org/10.1007/978-3-540-74690-4_71.
  * Yu et al. (2019) Yu, T., Quillen, D., He, Z., Julian, R., Hausman, K., Finn, C., and Levine, S.  Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning.  In Kaelbling, L. P., Kragic, D., and Sugiura, K. (eds.), _3rd Annual Conference on Robot Learning, CoRL 2019, Osaka, Japan, October 30 - November 1, 2019, Proceedings_ , volume 100 of _Proceedings of Machine Learning Research_ , pp. 1094–1100. PMLR, 2019.  URL http://proceedings.mlr.press/v100/yu20a.html.
  * Zhang et al. (2021) Zhang, H., Chen, H., Boning, D. S., and Hsieh, C.  Robust reinforcement learning on state observations with learned optimal adversary.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net, 2021.  URL https://openreview.net/forum?id=sCZbhBvqQaU.

##  Appendix A Inhibiting Factors

Deep Reinforcement Learning algorithms in general are prone to bugs, that are
difficult to trace. Adding recurrent layers to the PPO model notably increases
the complexity of the entire implementation. While working on the baseline
implementation, several bugs severely hindered progress. Two of those are
related to the utilized machine learning framework PyTorch. Unintentionally
feeding $None$ as hidden state to the recurrent layer causes the network to
initialize hidden states as zero. This bug causes gradients to explode.
Further, when feeding a recurrent layer, PyTorch expects the sequence length
to be the first dimension of the fed data. Naively reshaping the data causes a
catastrophic shift. This bug may be circumvented by setting the batch first
argument of the recurrent layer’s forward function to true. Also, one has to
note that the output of an LSTM or GRU cell is already activated. Adding
another activation, like ReLU, on top might hurt performance.

##  Appendix B Mortar Mayhem

###  B.1 Configuration

Mortar Mayhem can be configured differently using the following 5 options. To
further increase the challenge to the agent, one could vary those options
during training as it has been done for the ballet environment.

Number of commands to execute (default: 10): The count of commands that the
agent has to execute one by one after observing them.
Available commands (default: first 5 choices): The command sequence can be
composed of 9 different commands: Up, Down, Left, Right, Stay, Up Left, Up
Right, Down Left and Down Right.
Command show duration (default: 1 step): This determines for how long one
command is presented to the agent.
Command show delay (default: 1 step): The duration of not showing a command
between commands.
Command execution delay (default: 5 steps): This delay determines the number
of steps the agent has to correctly execute the command.

Figure 12: The Unity ML-Agents version of Mortar Mayhem. a) shows how commands
are presented to the agent, whereas b) presents the alternative visualization.
The correct command execution is visualized using particle explosions (c)).

Also, an alternative version of the environment is implemented using the Unity
ML-Agents toolkit (Juliani et al., 2018) as shown by Figure 12. In this
environment, the agent’s actions lead to much smaller changes in position.
Hence, the agent needs more steps to position itself on an adjacent tile. This
environment features visual observations only. Its action space is multi-
discrete (3, 3) like how it is done in Searing Spotlights. The first action
dimension concerns the horizontal movement, whereas the other does the
vertical movement. Additionally, an alternative command visualization is
available. Instead of showing arrows, the target tile is highlighted using a
particle effect. The minimalistic gridworld version of this environment is
used in this paper, because the Unity one cannot be easily scaled up to 32
environment instances. Also, the simulation speed is faster.

###  B.2 Example Episode

Figure 13: One complete episode of Mortar Mayhem using 3 commands. The first 6
frames demand the agent to memorize the commands stay, right and up. Once the
7th frame is reached, the agent’s position and rotation is not frozen anymore.
Its task is now to execute each command one by one. Success is signaled by a
reward of $0.1$. Also, a visual feedback based on the lave tiles is provided.
The final return of this episode is $0.3$.

Figure 13 shows the frames from left to right of an agent playing one episode
of Mortar Mayhem given 3 commands. During the first 6 frames, the agent has to
observe the green commands, while not being able to move. Then, the agent has
5 frames per command to correctly execute each one of them. Given the command
sequence from Figure 13, the agent has to stay first, then to move right and
finally to move up.

##  Appendix C Searing Spotlights

###  C.1 Configuration

Searing Spotlights can be configured differently using the following 18
options.

Number of Spotlight Spawns (default: 50). The number of spotlights that are
spawned during one episode.
Initial Spawn Interval (default: 12 steps). How many steps to wait for the
next spotlight to be spawned.
Interval Threshold (default: 4 steps). The spawn interval decreases over time
until it reaches its lover threshold.
Min Spotlight Speed (default: 20). The minimum of the to be sampled spotlight
speed.
Max Spotlight Speed (default: 45). The maximum of the to be sampled spotlight
speed.
Min Spotlight Radios (default: 10). The minimum of the to be sampled spotlight
radius.
Max Spotlight Radios (default: 15). The maximum of the to be sampled spotlight
radius.
Spotlight Damage (default: 1). The amount of damage that one spotlight
inflicts to the agent each step.
Light On Duration (default: 3 steps). How many steps the global light is
turned on during the beginning of the episode.
Light Dim Duration (default: 10 steps). How many steps the global light is
dimmed until off.
Number of Coins (default: 1 coin). The number of coins that the agent has to
collect.
Coin Scale (default: 1). The scale of coins.
Coin Always Visible (default: False). Whether the coin is never hidden in the
dark.
Use Exit (default: False). Whether the agent has to find an exit to positively
terminate the environment. If coins are available, the agent has to collect
those first.
Exit Scale (default: 1). The scale of the exit.
Agent Speed (default: 35). The speed of the agent to move around.
Agent Health (default: 100). The amount of the agent’s health, which is
reduced by being visible inside spotlights. Once 0 is reached, the episode
terminates.
Agent Always Visible (default: False). Whether the agent is never hidden in
the dark.

###  C.2 Simplifications

Like how it was mentioned in the main body of this paper, the following
measures can be applied to simplify Searing Spotlights.

  * •

removing spotlight penalties (i.e. damage and reward)

  * •

limiting the number of coins to one

  * •

increasing the scale of the agent and the coin

  * •

making the spotlights stationary

  * •

making the agent or the coin permanently visible

Figure 14 describes the appearance of the environment upon scaling the coin
and the agent. When scaled up, the agent needs fewer steps to reach the coin.
Future work shall explore all possible permutations to analyze the exact level
of solvability of Searing Spotlights.

(a) Scale 1

(b) Scale 2

(c) Scale 3

Figure 14: The appearance of the agent (red dot) and the coin (yellow
ellipsis) when scaled by 1, 2 or 3. These images (i.e. initial observations)
feature 6 stationary spotlights.

###  C.3 Example Episode

One episode segment of Searing Spotlights is illustrated by Figure 15. The
agent’s observations are read from left to right. The coin and the agent are
at scale 2.

Figure 15: An episode segment of a random agent playing Searing Spotlights.
About 6 spotlights stochastically move across the environment at this
(advanced) point of time. Spotlights are spawned randomly on the outside of
the environment. They move to a stochastically positioned and moving target on
the opposite side.

###  C.4 Potential Strategies

One strategy could be to ease memorizing the agent’s position by moving
continuously towards environment boundaries, because moving for long enough in
one direction raises the confidence of being at the min or max position on one
or both axes. Another valid approach could be to trade-off between the
spotlight’s penalty and directly observing the agent’s position. The agent
might intentionally risk being visible to reveal its position.

##  Appendix D Grid Search Results

Grid search was conducted across various hyperparameters concerning recurrence
on Mortar Mayhem (Table 2) and the ballet environment (Table 3). The green
shaded area corresponds to the recurrent layer type, the blue one to the
sequence length and the red one to the hidden state size. All possible
permutations were trained three times and evaluated on 5 training and 5 novel
seeds. Each data point accumulates the episodes of the last three checkpoints.
The results show that longer sequences and greater hidden state sizes are
favorable. GRU is usually superior to LSTM.

Table 2: Mortar Mayhem Grid Search Results. Each column shows the percentage of succeeding at least on the specified number of commands. Note: 200 seeds were only present during training, which limit the agent’s ability to generalize. The training lasted for about 27 million steps. | Training Seeds | Novel Seeds
---|---|---
# Com. | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 1 | 2 | 3 | 4 | 5 | 6 | 7
GRU | 0.92 | 0.82 | 0.71 | 0.64 | 0.55 | 0.5 | 0.4 | 0.29 | 0.24 | 0.51 | 0.22 | 0.11 | 0.06 | 0.003 | 0.0 | 0.0
LSTM | 0.92 | 0.8 | 0.66 | 0.61 | 0.51 | 0.45 | 0.38 | 0.27 | 0.21 | 0.42 | 0.1 | 0.05 | 0.03 | 0.001 | 0.0 | 0.0
1 | 0.77 | 0.46 | 0.29 | 0.23 | 0.17 | 0.1 | 0.03 | 0.0 | 0.0 | 0.23 | 0.04 | 0.01 | 0.01 | 0.0 | 0.0 | 0.0
8 | 0.93 | 0.82 | 0.61 | 0.53 | 0.36 | 0.26 | 0.14 | 0.03 | 0.02 | 0.28 | 0.06 | 0.01 | 0.003 | 0.0 | 0.0 | 0.0
16 | 0.87 | 0.78 | 0.61 | 0.55 | 0.45 | 0.39 | 0.27 | 0.14 | 0.11 | 0.3 | 0.06 | 0.03 | 0.02 | 0.001 | 0.0 | 0.0
32 | 0.99 | 0.92 | 0.84 | 0.78 | 0.67 | 0.65 | 0.57 | 0.4 | 0.29 | 0.69 | 0.3 | 0.15 | 0.08 | 0.001 | 0.0 | 0.0
64 | 0.99 | 0.95 | 0.89 | 0.84 | 0.77 | 0.74 | 0.68 | 0.54 | 0.45 | 0.63 | 0.23 | 0.13 | 0.09 | 0.004 | 0.0 | 0.0
Max | 0.99 | 0.92 | 0.87 | 0.82 | 0.76 | 0.72 | 0.67 | 0.55 | 0.49 | 0.65 | 0.27 | 0.16 | 0.09 | 0.006 | 0.0 | 0.0
64 | 0.9 | 0.7 | 0.5 | 0.41 | 0.27 | 0.22 | 0.1 | 0.03 | 0.02 | 0.44 | 0.14 | 0.08 | 0.04 | 0.001 | 0.0 | 0.0
128 | 0.9 | 0.77 | 0.63 | 0.57 | 0.45 | 0.38 | 0.25 | 0.14 | 0.09 | 0.41 | 0.1 | 0.06 | 0.04 | 0.002 | 0.0 | 0.0
256 | 0.92 | 0.81 | 0.7 | 0.63 | 0.54 | 0.5 | 0.44 | 0.3 | 0.23 | 0.47 | 0.14 | 0.07 | 0.04 | 0.002 | 0.0 | 0.0
512 | 0.94 | 0.88 | 0.79 | 0.74 | 0.68 | 0.63 | 0.57 | 0.46 | 0.39 | 0.48 | 0.18 | 0.08 | 0.05 | 0.003 | 0.0 | 0.0
1024 | 0.95 | 0.88 | 0.81 | 0.77 | 0.71 | 0.65 | 0.59 | 0.46 | 0.39 | 0.52 | 0.22 | 0.12 | 0.07 | 0.002 | 0.0 | 0.0
Table 3: Dancing the Ballet Grid Search Results. Each column shows the percentage of succeeding at 2, 4 or 8 dances. Note: 500 seeds were present during training. The training lasted for about 27 million steps. | Training Seeds | Novel Seeds
---|---|---
# Dances | 2 | 4 | 8 | 2 | 4 | 8
GRU | 0.96 | 0.91 | 0.81 | 0.6 | 0.23 | 0.17
LSTM | 0.96 | 0.81 | 0.66 | 0.59 | 0.26 | 0.15
1 | 0.93 | 0.8 | 0.64 | 0.6 | 0.24 | 0.16
8 | 0.97 | 0.82 | 0.72 | 0.59 | 0.24 | 0.17
16 | 0.95 | 0.88 | 0.72 | 0.57 | 0.24 | 0.14
32 | 0.96 | 0.87 | 0.77 | 0.6 | 0.24 | 0.17
64 | 0.97 | 0.9 | 0.77 | 0.62 | 0.28 | 0.17
Max | 0.97 | 0.89 | 0.8 | 0.59 | 0.27 | 0.17
64 | 0.95 | 0.82 | 0.68 | 0.6 | 0.2 | 0.16
128 | 0.96 | 0.85 | 0.76 | 0.62 | 0.24 | 0.16
256 | 0.97 | 0.9 | 0.74 | 0.58 | 0.25 | 0.18
512 | 0.96 | 0.85 | 0.72 | 0.59 | 0.27 | 0.16
1024 | 0.95 | 0.89 | 0.79 | 0.59 | 0.28 | 0.16

##  Appendix E Refreshing Stale Hidden States and Advantages

We showed that refreshing stale hidden states and advantages do not yield any
improvement concerning the IQM return (Section 5.3. When examining further
training metrics (Figure 16 and 17), no substantial divergence between not
refreshing and refreshing every or every other epoch is observed. Gradient
norms and the Kullback–Leibler divergence behave differently, while conducting
8 epochs of optimization. Note that gradient norms are clipped at $0.5$.

Figure 16: Monitoring of the training runs in Mortar Mayhem when refreshing
stale hidden states and advantages. a) shows the gradient norm of all model
parameters, whereas b) presents the gradient norm only for parameters of the
recurrent layer GRU. The Kullback–Leibler divergence between $\theta_{old}$
and $\theta$ is illustrated by c). All curves show the mean and the standard
deviation.

Figure 17: Monitoring of the training runs in Dancing the Ballet when
refreshing stale hidden states and advantages. a) shows the gradient norm of
all model parameters, whereas b) presents the gradient norm only for
parameters of the recurrent layer GRU. The Kullback–Leibler divergence between
$\theta_{old}$ and $\theta$ is illustrated by c). All curves show the mean and
the standard deviation.

##  Appendix F Adding the Last Action and Last Reward to the Observation

Figure 18 describes an ablation on the agent’s observation space to determine
the significance of adding the last action, last reward or both to the current
observation. Of utility is usually the last action, whereas the last reward
does not seem to affect performance. Because of no notable extra cost, the
last reward and last action can always be considered for the current
observation regardless of their individual impact. The figures also show an
experiment when hidden states are not reset to zero. Hence, the agent starts a
new episode using the final hidden state of the former episode.

(a) Mortar Mayhem Training Seeds

(b) Mortar Mayhem Novel Seeds

(c) Ballet 8 Dances Training Seeds

Figure 18: An ablation on the agent’s observation space. Also, the experiment
on not resetting the hidden state for new episodes is shown.


$\langle$reception date$\rangle$ $\langle$acception date$\rangle$
$\langle$publication date$\rangle$

11affiliationtext: Department of Physical Science, Graduate School of Science,
Osaka Prefecture University 1-1 Gakuen-cho, Naka-ku, Sakai, Osaka 599-8531,
Japan22affiliationtext: National Astronomical Observatory of Japan, Mitaka,
Tokyo, 181-8588, Japan33affiliationtext: Department of Physics, Institute of
Science and Technology, Keio University, 3-14-1 Hiyoshi, Kohoku-ku, Yokohama,
Kanagawa 223-8522, Japan44affiliationtext: Institute of Space and
Astronautical Science (ISAS) Japan Aerospace Exploration Agency (JAXA), 3-1-1
Yoshinodai, Chuo-ku, Sagamihara, Kanagawa 252-5210, Japan55affiliationtext:
Astronomy Section, Nagoya City Science Museum, 2-17-1 Sakae, Naka-ku, Nagoya,
Aichi 460-0008, Japan66affiliationtext: Dr. Karl Remeis Observatory, Erlangen
Centre for Astroparticle Physics, Friedrich-Alexander-Universität Erlangen-
Nürnberg, Sternwartstra$\beta$e 7, 96049 Bamberg, Germany77affiliationtext:
Department of Astrophysics, Nagoya University, Furo-cho, Chikusa-ku, Nagoya,
Aichi, 464-8602, Japan88affiliationtext: Institute of Astronomy, the
University of Tokyo, 2-21-1 Osawa, Mitaka, Tokyo 181-0015, Japan

ISM: clouds — ISM: individual objects (Carina) — radio lines: ISM

# Massive star formation in the Carina nebula complex and Gum 31 – II. a
cloud–cloud collision in Gum 31

Shinji FUJITA1∗ Hidetoshi SANO2 Rei ENOKIYA3 Katsuhiro HAYASHI4 Mikito KOHNO5
Kisetsu TSUGE6 Kengo TACHIHARA7 Atsushi Nishimura8 Akio OHAMA7 Yumiko YAMANE7
Takahiro OHNO7 Rin I. YAMADA7 Yasuo FUKUI7<EMAIL_ADDRESS>
###### Abstract

We present the results of analyses of the 12CO ($J$=1–0), 13CO ($J$=1–0), and
12CO ($J$=2–1) emission data toward Gum 31. Three molecular clouds separated
in velocity were detected at $-25$, $-20$, and $-10$ km s-1. The velocity
structure of the molecular clouds in Gum 31 cannot be interpreted as expanding
motion. Two of them, the $-25$ km s-1 cloud and the $-20$ km s-1 cloud, are
likely associated with Gum 31, because their 12CO ($J$=2–1)/12CO ($J$=1–0)
intensity ratios are high. We found that these two clouds show the
observational signatures of cloud–cloud collisions (CCCs): a complementary
spatial distribution and a V-shaped structure (bridge features) in the
position–velocity diagram. In addition, their morphology and velocity
structures are very similar to the numerical simulations conducted by the
previous studies. We propose a scenario that the $-25$ km s-1 cloud and the
$-20$ km s-1 cloud were collided and triggered the formation of the massive
star system HD 92206 in Gum 31. This scenario can explain the offset of the
stars from the center and the morphology of Gum 31 simultaneously. The
timescale of the collision was estimated to be $\sim$ 1 Myr by using the ratio
between the path length of the collision and the assumed velocity separation.
This is consistent with that of the CCCs in Carina Nebula Complex in our
previous study.

##  1 Introduction

Many recent observational studies of star forming region have revealed the
triggering of massive star formations via cloud–cloud collisions (CCCs). The
two observational signatures—the complementary spatial distributions and
bridging features between the colliding clouds—have been suggested as evidence
(e.g., [Fukui et al. (2021)]). Such evidence was detected in CCC regions both
in the Galactic (e.g., [Furukawa et al. (2009), Ohama et al. (2010), Fukui et
al. (2014), Torii et al. (2015), Fukui et al. (2016), Fujita et al. (2021a),
Fukui et al. (2017a), Torii et al. (2017a), Torii et al. (2017b), Enokiya et
al. (2018), Hayashi et al. (2018), Kohno et al. (2018), Nishimura et al.
(2018), Sano et al. (2018), Fujita et al. (2021b), Fujita et al. (2019), Fukui
et al. (2019), Torii et al. (2021), Torii et al. (2018b), Fukui et al. (2021),
Fujita et al. (2021c)]) and extragalactic (e.g., [Tokuda et al. (2019), Sano
et al. (2021)]) objects.

In Fujita et al. (2021c) (hereafter Paper I), we analysed a CO dataset towards
the Carina Nebula Complex (CNC). Four discrete molecular clouds at velocities
$-27$, $-20$, $-14$, and $-8$ km s-1 were found. Most of them might be
associated with the star clusters in the CNC, because their 12CO
($J$=2–1)/12CO ($J$=1–0) intensity ratios are high and their distributions
correspond to the Spitzer 8-$\mu$m distributions. In addition to this, we
found that these clouds show the observational signatures of a CCC.
Furthermore, we also found that the SiO emission (a tracer of a shocked
molecular gas) is significantly enhanced between the colliding clouds. For
these results, we propose a scenario that the formation of massive stars in
the clusters in the CNC was triggered by CCCs with a timescale of $\sim$ 1
Myr.

In this paper, we focus on Gum 31 (E127 in Hanaoka et al. (2019)). Gum 31
located $\sim 50$ pc from the CNC on the sky plane. The distance to Gum 31 has
been discussed and determined to be 2.5 $\pm$0.3 kpc (Barnes et al. (2010)).
We adopt the value of 2.5 kpc in this paper. Figure 1(a) shows a composite
color image of the WISE 22-$\mu$m (red) and Spitzer/GLIMPSE 8-$\mu$m (green)
emissions around Gum 31. It is ionized by the young stellar cluster NGC 3324
(e.g., Walborn (1982)) including the double or multiple O-type stars in HD
92206 [HD 92206A at $(l,\,b)=(286\fdg 223,\,-0\fdg 170)$, HD 92206B at
$(l,\,b)=(286\fdg 225,\,-0\fdg 169)$, and HD 92206C (CD-57$\fdg$3378) at
$(l,\,b)=(286\fdg 219,\,-0\fdg 178)$] (Mathys (1988)). Gum 31 is relatively a
small and resembles to the Spitzer bubbles (Churchwell et al. (2006, 2007)),
although the radius is somewhat larger ($\sim$10 pc) than is typical for those
bubbles. Gum31 is connected to the central parts of the CNC in the Herschel
far-infrared maps (Ohlendorf et al. (2013)) and CO maps (Yonekura et al.
(2005); Rebolledo et al. (2016)). In addition, the distance to the CNC is
estimated to be $\sim$2.3 kpc (e.g., Allen & Hillier (1993); Smith (2006)) and
thus Gum31 is probably associated with the CNC.

Cappa et al. (2008) investigated the ionized, neutral, and CO gas in the
environs of Gum 31 and found that the Hi structure is 10.0 $\pm$ 1.7 pc in
radius, has a neutral mass of 1500 $\pm$ 500 $M_{\odot}$, and is expanding at
11 km s-1. They concluded that the expansion of the Hii region triggered
stellar formation in the molecular shell, although the angular resolution of
their CO data is insufficient ($\sim\,2^{\prime}.7$). Detailed gas
observations and analyses in such a young [the age of the cluster NGC 3324 is
estimated to be $<\,3$ Myr (Carraro et al. (2001))] star forming region are
important in revealing the mechanism of massive star formation.

Figure 1: (a) Composite image of the WISE 22 $\mu$m (red) and Spitzer/GLIMPSE
8 $\mu$m (green) towards Gum 31. (b) The 12CO ($J$=1–0) integrated-intensity
map taken from the Mopra archive (Rebolledo et al. (2016, 2017)). Integrated
velocity range is from -35 to 10 km s-1. The black contours are plotted 25
($\sim 10\sigma$), 50, 75, …, 200 K km s-1. The black dotted rectangle
indicates the area observed with NANTEN2. The crosses represent the multiple
O-stars HD 92206 (Mathys (1988)). The triangles represent young stellar
objects (Ohlendorf et al. (2013)).

##  2 Dataset

###  2.1 Mopra

As in Paper I, we used the 12CO and 13CO ($J$=1–0) archived datasets obtained
with the Mopra 22-m telescope (Burton et al. (2013); Braiding et al. (2015);
Rebolledo et al. (2016, 2017)), the beam size and velocity resolution of which
are $\sim$35′′ and $\sim$0.09 km s-1, respectively. The rms noise levels for
the 12CO and 13CO ($J$=1–0) datasets are $\sim$4.0 K and $\sim$1.9 K per
channel on the $T_{\rm mb}$ scale, respectively, at a velocity grid of 0.09 km
s-1.

###  2.2 NANTEN2

The 12CO ($J$=2–1) data toward Gum 31 were obtained with the NANTEN2 4-m
millimeter/sub-millimeter telescope at Atacama, Chile in 2015 October. The
beam size of NANTEN2 at $\sim$230 GHz corresponds to $\sim$90′′, and we
adopted the on-the-fly (OTF) mapping mode with Nyquist sampling. The backend
and frontend system and calibration are the same as in Paper I. A typical
uncertainty in the intensity is $\sim$20%, and the typical rms noise level for
the 12CO ($J$=2–1) data is $\sim$0.3 K per channel on the $T_{\rm mb}$ scale
at the velocity grid of 0.5 km s-1.

##  3 Results

Figure 1(b) shows the 12CO ($J$=1–0) integrated-intensity distributions
(Mopra) with the velocity range from $-35$ to $10$ km s-1. It has a roughly
ring-shaped structure, and the double or multiple O-type stars in HD 92206 has
been identified (e.g., Mathys (1988); Maíz-Apellániz et al. (2004)) at a
location that is offset eastward from the center. We estimated that the
maximum column density of the molecular gas is $1\,\times\,10^{23}$ cm-2 and
the total molecular gas mass (within the dotted square in Figure 1) is about
$1.4\,\times\,10^{5}$ $M_{\odot}$ by using the same method as in Paper I.

Figure 2 shows the $l$–$v$ diagrams of the 12CO ($J$=1–0) toward Gum 31 with
the integrated ranges (a) from $-0$$4$ to $+0$$2$, (b) from $-0$$1$ to
$+0$$2$, and (c) from $-0$$4$ to $-0$$1$. In Figure 2(b), in the northern
part, we detected two clouds separated in velocity and centered at
$\sim\,$$-20$ and $\sim\,$$-10$ km s-1 (hereinafter called the $-20$ km s-1
cloud and the $-10$ km s-1 cloud, respectively). On the other hand, in Figure
2(c), there are two velocity components separated in velocity and centered at
$\sim\,$$-25$ (hereinafter called the $-25$ km s-1 cloud) and $\sim\,$$-20$ km
s-1 with a velocity gradient. In addition, there is also an intermediate
velocity component in between.

Figure 3 shows the integrated-intensity ratios 12CO ($J$=2–1)/12CO ($J$=1–0)
(hereinafter denoted by $R^{12}_{2-1/1-0}$) (color image) and the integrated-
intensity distributions of the 12CO ($J$=1–0) emission (contours) for the
three clouds. The $R^{12}_{2-1/1-0}$ of the $-25$ km s-1 cloud and the $-20$
km s-1 cloud are high ($>$0.8) at the edge of the clouds, indicating that they
are heated by the surrounding Hii region in the same way as the cloud in the
CNC. In contrast, the $R^{12}_{2-1/1-0}$ of the $-10$ km s-1 cloud is low
($\sim\,0.4$). For these reasons, it is probable that the former two are
associated with the Hii region, and the latter is not.

Figure 2: (a) Galactic longitude – Velocity ($l$–$v$) diagram of the 12CO
($J$=1–0) emissions integrated from -04 to +02. (b) The same as (a), but
integrated from -01 to +02 (A in Figure 1(b)). (c) The same as (a), but
integrated from -04 to -01. (B in Figure 1(b)) Figure 3: The integrated-
intensity ratio 12CO ($J$=2–1)/12CO ($J$=1–0) ($R^{12}_{2-1/1-0}$) map for the
velocity range (a) from $-29$ km s-1 to $-22$ km s-1, (b) from $-21$ km s-1 to
$-16$ km s-1, and (c) from $-12$ km s-1 to $-8$ km s-1, respectively. The
symbols are the same as in Figure 1.

##  4 Discussion

The observations with Mopra and NANTEN2 have shown that the two CO clouds, the
$-25$ km s-1 cloud and the $-20$ km s-1 cloud, are both associated with Gum
31. The molecular mass of the two are $4.9\,\times\,10^{4}$ $M_{\odot}$ (from
$-27$ km s-1 to $-21$ km s-1) and $2.3\,\times\,10^{4}$ $M_{\odot}$ (from
$-21$ km s-1 to $-15$ km s-1), respectively, within a radius of 15 pc centered
on Gum 31. By tentatively assuming the most plausible viewing angle of the
relative motion of the clouds to the line-of-sight is 60∘ (Fujita et al.
(2021b)), the relative velocity between the two is 10 km s-1, which requires
the total mass of $\frac{rv^{2}}{2G}\,=\,1.7\,\times\,10^{5}$ $M_{\odot}$ to
bind the two clouds gravitationally within 15 pc (where $r$ and $G$ are the
distance between the two clouds and the gravitational constant, respectively).
This is a factor of 2 larger than the total molecular mass of the two clouds
derived, indicating that the coexistence of the two clouds cannot be
understood as a gravitationally bound system. Therefore, the velocity
structure of the molecular gas in Gum 31 is thought to be in the middle of or
as a result of some dynamic event.

###  4.1 Expanding shell model

A possible explanation of the velocity components and the morphology of Gum 31
is the feedback effects from the exciting stars in HD 92206, such as the
stellar wind, UV radiation, and pressure of expanding Hii region. However, HD
92206 stars are offset from the center of the bubble-like structure of Gum 31.
In addition, in Figure 2(a) we can not see clear expanding motion, namely an
ellipse in the $p$–$v$ diagram, of the cloud either in the northern or the
southern part of Gum 31. Figures 4(a)(b) shows the first moment of the 12CO
($J$=1–0) emissions and the 13CO ($J$=1–0) emissions, respectively, with a
velocity range from $-27$ km s-1 to $-15$ km s-1. It is clear that the insider
cloud is blueshifted (the $-25$ km s-1 cloud) and the outsider is redshifted
(the $-20$ km s-1 cloud). If the cloud is isotropically expanding, such a
velocity gradient should not be seen. For these reasons, molecular gas in Gum
31 is likely to not have an expanding structure.

Figure 4: (a) Map of the 1st moment of the 12CO ($J$=1–0) emission between
$-27$ km s-1 and $-15$ km s-1. The contours show the moment 0 (integrated-
intensity) of the 12CO ($J$=1–0) emission. The contours are plotted every 25 K
km s-1 from 25 K km s-1 ($\sim 10\sigma$). The symbols are the same as in
Figure 1. (b) The same as (a), but for the 13CO ($J$=1–0). (c–e) The spectra
of the 12CO ($J$=1–0) and the 13CO ($J$=1–0) at the position P1, P2, and P3,
respectively, shown by square makers.

###  4.2 An Alternative: Cloud–cloud collision

Paper I suggests complex CCCs in the CNC. The existence of different velocity
components, relatively weak CO emissions at intermediate velocities between
two clouds (a bridge feature) in a $p$–$v$ diagram, and a complementary
spatial distribution were reported as evidence of CCCs. The combination of the
two characteristics of a bridge feature and a complementary spatial
distribution made the $p$–$v$ diagram V-shaped.

In Figure 2(c), we found that a V-shape structure in the $l$–$v$ diagrams can
be seen. In addition, it was found that the two velocity components show
complementary spatial distributions (Figure 5). These morphology and velocity
structures are very similar to the numerical simulations conducted by e.g.,
Habe & Ohta (1992), Takahira et al. (2014), and Haworth et al. (2015). That
is, a small cloud (the $-25$ km s-1 cloud) collided with the large cloud (the
$-20$ km s-1 cloud) at supersonic velocity from the eastern side, and formed a
cavity. The small cloud has a larger mass than the large cloud
($4.9\,\times\,10^{4}$ $M_{\odot}$ and $2.3\,\times\,10^{4}$ $M_{\odot}$,
respectively). It is considered that the initial density of the small cloud
was higher than that of the large cloud, if we assume that their thickness on
the line-of-sight direction are about the same. These condition are similar to
the numerical simulation conducted by Takahira et al. (2014). After the
collision, the two were merged in the interface layers, and turbulence was
predominant due to strong compression, forming a molecular cloud core that
formed massive stars in HD 92206. Furthermore, the collision scenario can
interpret the offset of HD 92206 from the center of Gum 31 seen in Figure 1(a)
and the first moment and spectra of the molecular gas in Figures 4. Figure 6
shows a sketch diagram of this collision scenario in Gum 31. Such a CCC
scenario has also been proposed for RCW 120 (Torii et al. (2015)) and N4
(Fujita et al. (2019)), which have a similar appearance to Gum 31. The time
scale of the collision is roughly estimated as $\sim$10 pc/(5 km
s${}^{-1}\,\times\,$2) $\sim$ 1 Myr from the size of the cavity and the
velocity separation of the molecular cloud, with the assumption that the angle
between the line of sight and the collision axis is $60^{\circ}$. There is no
large contradiction between this collision time scale and the young age of the
cluster NGC 3324 ($<\,3$ Myr, Carraro et al. (2001)).

We conclude that the massive star formation of the CNC and Gum 31 were
triggered by multiple CCCs and a CCC, respectively, because the CCC scenario
in Gum31 is consistent with Paper I results in terms of the collision
timescale and the collision direction. It is considered that the source that
caused the multiple CCCs in the CNC and Gum31 is possibly the same.  This
picture is similar to the star formation history in W51 (Fujita et al.
(2021b)), which has similar star formation activity (dozens of O-type stars)
to Carina. In addition, our results support the proposal that Gum 31 is part
of the CNC (Ohlendorf et al. (2013)).

For the YSOs (triangle markers in Figure 1(a)), it cannot be determined by
this study whether the formation was triggered by the collision or triggered
by the feedback of stars in Gum 31 as suggested by Cappa et al. (2008). In the
future, it will be possible to discuss this by e.g., detailed measurement of
their ages.

Figure 5: The red and blue show the integrated-intensity distributions of the
12CO ($J$=1–0) emissions toward Gum 31. The each integrated velocity ranges
are shown in the top left corner of the figure. The contour levels are 8, 16,
32, 64, 96, 128, and 160 K km s-1. The symbols are the same as in Figure 1.
Figure 6: A sketch diagram of the cloud–cloud collision scenario in Gum 31.

##  5 Summary

  1. 1.

We analysed the 12CO ($J$=1–0), 13CO ($J$=1–0) (Mopra), and 12CO ($J$=2–1)
(NANTEN2) emission lines toward Gum 31, and found the discrete three molecular
clouds at velocities $-25$, $-20$, and $-10$ km s-1.

  2. 2.

The $-25$ km s-1 cloud and the $-20$ km s-1 cloud are associated with Gum 31,
because their 12CO ($J$=2–1)/12CO ($J$=1–0) intensity ratios are high. In
addition, these clouds show the observational signatures of a CCC as same as
the clouds in the CNC. In addition, their morphology and velocity structures
are very similar to the numerical simulations conducted by the previous
studies.

  3. 3.

We propose a scenario that the $-25$ km s-1 cloud and the $-20$ km s-1 cloud
were collided and triggered the formation of the massive stars in Gum 31. This
CCC scenario can explain the offset of the stars from the center and the
morphology of Gum 31 simultaneously. We estimate the timescale of the
collision to be $\sim$ 1 Myr by using the path length of the collision and the
assumed velocity separation. This is roughly consistent with that of the CCCs
in the CNC estimated in our previous study, which support that Gum 31 is part
of the CNC.

We are deeply grateful to the financially support by Grants-in-Aid for
Scientific Research (KAKENHI) of the Japanese society for the Promotion of
Science (JSPS; grant numbers 15K17607, 17H06740, and 18K13580), and we would
like to thank the all members of the NASA, Mopra, and NANTEN2 for providing
the data. Also, we want to thanks Astropy (Astropy Collaboration et al.
(2013)) and APLpy (Robitaille & Bressert (2012)) development team for data
analysis.

## References

  * Allen & Hillier (1993) Allen, D. A., & Hillier, D. J. 1993, Proceedings of the Astronomical Society of Australia, 10, 338
  * Astropy Collaboration et al. (2013) Astropy Collaboration, Robitaille, T. P., Tollerud, E. J., et al. 2013, A&A, 558, A33
  * Barnes et al. (2010) Barnes, P. J., Yonekura, Y., Ryder, S. D., et al. 2010, MNRAS, 402, 73. doi:10.1111/j.1365-2966.2009.15890.x
  * Braiding et al. (2015) Braiding, C., Burton, M. G., Blackwell, R., et al. 2015, PASA, 32, e020
  * Burton et al. (2013) Burton, M. G., Braiding, C., Glueck, C., et al. 2013, PASA, 30, e044
  * Cappa et al. (2008) Cappa, C., Niemela, V. S., Amorín, R., et al. 2008, A&A, 477, 173. doi:10.1051/0004-6361:20067028
  * Carraro et al. (2001) Carraro, G., Patat, F., & Baumgardt, H. 2001, A&A, 371, 107. doi:10.1051/0004-6361:20010307
  * Churchwell et al. (2006) Churchwell, E., Povich, M. S., Allen, D., et al. 2006, ApJ, 649, 759
  * Churchwell et al. (2007) Churchwell, E., Watson, D. F., Povich, M. S., et al. 2007, ApJ, 670, 428
  * Enokiya et al. (2018) Enokiya, R., Sano, H., Hayashi, K., et al. 2018, PASJ, 70, S49
  * Fujita et al. (2019) Fujita, S., Torii, K., Tachihara, K., et al. 2019, ApJ, 872, 49
  * Fujita et al. (2021a) Fujita, S., Tsutsumi, D., Ohama, A., et al. 2021a, PASJ, 73, S273. doi:10.1093/pasj/psaa005
  * Fujita et al. (2021b) Fujita, S., Torii, K., Kuno, N., et al. 2021b, PASJ, 73, S172. doi:10.1093/pasj/psz028
  * Fujita et al. (2021c) Fujita, S., Sano, H., Enokiya, R., et al. 2021c, PASJ, 73, S201. doi:10.1093/pasj/psaa078
  * Fukui et al. (2014) Fukui, Y., Ohama, A., Hanaoka, N., et al. 2014, ApJ, 780, 36
  * Fukui et al. (2016) Fukui, Y., Torii, K., Ohama, A., et al. 2016, ApJ, 820, 26
  * Fukui et al. (2017a) Fukui, Y., Tsuge, K., Sano, H., et al. 2017, PASJ, 69, L5
  * Fukui et al. (2019) Fukui, Y., Tokuda, K., Saigo, K., et al. 2019, ApJ, 886, 14
  * Fukui et al. (2021) Fukui, Y., Habe, A., Inoue, T., et al. 2021, PASJ, 73, S1. doi:10.1093/pasj/psaa103
  * Furukawa et al. (2009) Furukawa, N., Dawson, J. R., Ohama, A., et al. 2009, ApJ, 696, L115
  * Habe & Ohta (1992) Habe, A., & Ohta, K. 1992, PASJ, 44, 203
  * Hanaoka et al. (2019) Hanaoka, M., Kaneda, H., Suzuki, T., et al. 2019, PASJ, 71, 6. doi:10.1093/pasj/psy126
  * Haworth et al. (2015) Haworth, T. J., Tasker, E. J., Fukui, Y., et al. 2015, MNRAS, 450, 10
  * Hayashi et al. (2018) Hayashi, K., Sano, H., Enokiya, R., et al. 2018, PASJ, 70, S48
  * Kohno et al. (2018) Kohno, M., Torii, K., Tachihara, K., et al. 2018, PASJ, 70, S50
  * Mathys (1988) Mathys, G. 1988, A&AS, 76, 427
  * Maíz-Apellániz et al. (2004) Maíz-Apellániz, J., Walborn, N. R., Galué, H. Á., et al. 2004, ApJS, 151, 103. doi:10.1086/381380
  * Nishimura et al. (2018) Nishimura, A., Minamidani, T., Umemoto, T., et al. 2018, PASJ, 70, S42. doi:10.1093/pasj/psx149
  * Ohama et al. (2010) Ohama, A., Dawson, J. R., Furukawa, N., et al. 2010, ApJ, 709, 975
  * Ohlendorf et al. (2013) Ohlendorf, H., Preibisch, T., Gaczkowski, B., et al. 2013, A&A, 552, A14. doi:10.1051/0004-6361/201220218
  * Rebolledo et al. (2016) Rebolledo, D., Burton, M., Green, A., et al. 2016, MNRAS, 456, 2406
  * Rebolledo et al. (2017) Rebolledo, D., Green, A. J., Burton, M., et al. 2017, MNRAS, 472, 1685
  * Robitaille & Bressert (2012) Robitaille, T., & Bressert, E. 2012, Astrophysics Source Code Library, ascl:1208.017
  * Sano et al. (2018) Sano, H., Enokiya, R., Hayashi, K., et al. 2018, PASJ, 70, S43
  * Sano et al. (2021) Sano, H., Tsuge, K., Tokuda, K., et al. 2021, PASJ, 73, S62. doi:10.1093/pasj/psaa045
  * Smith (2006) Smith, N. 2006, ApJ, 644, 1151
  * Takahira et al. (2014) Takahira, K., Tasker, E. J., & Habe, A. 2014, ApJ, 792, 63
  * Tokuda et al. (2019) Tokuda, K., Fukui, Y., Harada, R., et al. 2019, ApJ, 886, 15
  * Torii et al. (2015) Torii, K., Hasegawa, K., Hattori, Y., et al. 2015, ApJ, 806, 7
  * Torii et al. (2017a) Torii, K., Hattori, Y., Hasegawa, K., et al. 2017a, ApJ, 835, 142
  * Torii et al. (2017b) Torii, K., Hattori, Y., Hasegawa, K., et al. 2017b, ApJ, 840, 111
  * Torii et al. (2018b) Torii, K., Fujita, S., Matsuo, M., et al. 2018b, PASJ, 70, S51
  * Torii et al. (2021) Torii, K., Hattori, Y., Matsuo, M., et al. 2021, PASJ, 73, S368. doi:10.1093/pasj/psy098
  * Walborn (1982) Walborn, N. R. 1982, AJ, 87, 1300. doi:10.1086/113216
  * Yonekura et al. (2005) Yonekura, Y., Asayama, S., Kimura, K., et al. 2005, ApJ, 634, 476