As mentioned above, training is conducted without obstacles. The motivation for the environmental parameters is that the goal can be reached within a small number of steps without making it a trivial task.  This is done to increase the difficulty of restriction handling by offering several alternative paths. We assume that significant differences between algorithms would be more likely to be found in such a scenario.

In this context, the size of the map is closely related to the step size of the agent. The minimum number of steps on a large map with a simultaneously large step size is equivalent to a small environment with few steps. Based on a step size of 1.0, we set the width and height to $15$. This makes a minimum number of 15 steps necessary to reach the target location. Accordingly, we set the maximum length of an episode to $40$ which accounts for longer paths and exploration.
On one hand, if the maximum number of steps is too small, agents might not be able to reach the goal, and investigating the episode length loses effect. On the other hand, if it is too large, the probability rises that algorithms randomly drop into the goal.
A relatively larger step size has the advantage that restrictions increase since more obstacles fall into the radius.

The agent is placed in the bottom left of the environment, and the goal is located in the top right corner. The range of actions is set to $[-110,110]$ to not allow sharp turns in the opposite direction but still keep enough flexibility so that not the entire space can frequently get unavailable.
The reward function has been determined empirically. Each algorithm is trained for 50.000 steps. Note that static restrictions occur in this setup at the border of the environment to keep the agent on the map. Therefore, the training is not completely restriction-free and comparable to the usual approach in the literature [23]. It must also be taken into account that, because of an error during the training process, PPO with discrete was trained with parameters optimized for continuous masking. It was impossible to repeat the optimization due to the time constraints of this thesis.
We set the zero-padding for the baseline algorithms to support up to eight disjoint intervals. Due to the simplicity of our task, we do not expect the count to rise higher.

To test the fitness of our models, we evaluate each without exploration after the last training iteration. The same environment is used for the rollouts. We run the algorithms on six different seeds, including 40, 41, 42, 43, 44, and 45.

1cParameter                           Value
Agent                                                   1l

step size $s^{\text{agent}}$                                     $1.0$

radius $r^{\text{agent}}$                                        $0.4$

action space range $[a_\text{min},a_\text{max}]$                 $[-110,110]$

starting position $(x^{\text{agent}}_t, y^{\text{agent}}_t)$     $(1.0,1.0)$

starting perspective $p^{\text{agent}}_0$                        $90.0$

safety distance $d^{\text{safe}}$                                $0.05$
Environment                                             1l

height $h$                                                       $15.0$

width $w$                                                        $15.0$

maximum steps $T$                                                $40$

goal position $(x^{\text{goal}},y^{\text{goal}})$                $12.0$

goal distance threshold $r^{\text{goal}}$                        $0.5$
Reward                                                  1l

on goal $r_{\text{goal}}$                                        $50.0$

on collision $r_{\text{collision}}$                              $20.0$

improvement scale $c$                                            $5.0$

step penalty scale $p$                                           $0.05$

Environmental parameters for the experiments

§.§.§ Results

In line with the axes of variability, the following first depicts outcomes within and across training runs before we continue with the evaluation rollouts.

Within and across training runs
The results indicate a smooth learning curve for MPS-TD3 and most baseline algorithms that converges close to the maximum reward. Due to a large number of methods, figure <ref> shows exemplarily the results of MPS-TD3 and PPO with random replacement. They represent the optimal behavior by having a steady return increase without a drop in performance. The episodes between the last two training iterations achieve average cumulative rewards of 114.34 and 114.54 respectively.

Episode length

[Training progress without dynamic restrictions]Training progress without dynamic restrictions

However, PAM and PPO with discrete masking fail to maximize the expected return and converge to a sub-optimal policy. Both learning curves are also depicted in figure <ref>.
The progress of PAM's average return continuously has a high standard deviation with up- and downward trends. At the end of the training, the approach reaches a relatively low mean cumulative reward of $49.46$. Another fact that caught our attention is that the action values explode. They increase simultaneously to the reward and come close to the maximum achievable return after about 20.000 time steps. Then, the model becomes increasingly unstable and does not improve further.
In terms of PPO with discrete masking, the average return settles around 60. Additional time steps did not increase the performance. The final plateau is reached at approximately $40.000$ time steps although the learning curve in figure <ref> does not show the progress with more training iterations.

Moreover, our analysis reveals that MPS-TD3 and the off-policy baselines have a higher standard deviation than on-policy learning. Specifically, TD3 with projection demonstrates this trend in appendix <ref> most clearly but the dispersity can also be observed from MPS-TD3 in figure <ref>. Across separate runs, the metrics show more variation in the first 40,000 time steps, but the differences decrease as the agent nears the end of training. The higher standard deviation aligns with other results such that off-policy algorithms take longer to converge. For instance, random replacement requires a median time of five training iterations to solve 80% of the episodes when combined with PPO, while it takes 11.50 with MPS-TD3.

Furthermore, we discover that the average episode length follows a similar pattern to the reward. This means that with more training iterations, the agents find shorter ways to the target. All methods start with the maximum number of steps and gradually reduce them to approach the optimal value. The exceptions are penalty-based agents since they are not collision free from the beginning. Additionally, discrete PPO and PAM terminate with episode lengths above 30. The described behavior is depicted in Figure <ref>. The lines show for PPO with discrete masking and random replacement, MPS-TD3,  and PAM the step counts with their standard deviation over time. The episode lengths of the other algorithms are illustrated in appendix <ref> and <ref>. Also regarding the number of steps, off-policy has a higher standard deviation than on-policy learning.

Likewise, the average restrictions get lower as training continues. The trajectories show that the agents become more proficient at avoiding the borders of the environment and taking a direct path toward the goal. As a result, they have access to a wider range of actions. For example, the TD3 algorithm with continuous masking starts in appendix <ref> with an average allowed action space of less than 150 but the metric increases until almost the entire action space is available.

Similarly, the number of intervals decreases in appendixes <ref>, <ref>, and <ref> over time. Most agents deal with a single subset by the end of training. Only PAM, MPS-TD3, and discrete PPO maintain a higher average number of intervals. However, MPS-TD3 simultaneously increases the size of the available action space. A larger fraction is unrestricted but split over more intervals. Again our algorithms and the off-policy baselines have a higher standard deviation than the on-policy algorithms. Note that all results and metrics can be found in appendix <ref>.

After training
In the evaluation rollouts, MPS-TD3 and the baselines achieve a high average return which is similar to the end of training. Table <ref> describes the mean cumulative reward, episode length, and fraction of solved episodes without exploration.

1cApproach     1cReturn     1cSteps     1cSolved
TD3                                       $114.57 \pm 3.67$                         $17.33 \pm 2.88$                         $100.00\% \pm 0.00\%$

TD3-Projection                            $91.46 \pm 55.30$                         $17.60 \pm 4.22$                         $83.33\% \pm 40.82\%$

TD3-Masking                               $114.27 \pm6.21$                          $18.83 \pm 5.53$                         $100.00\% \pm 0.00\%$

TD3-Random                                $114.13 \pm 4.54$                         $18.17 \pm 3.87$                         $100.00\% \pm 0.00\%$

DQN-Masked                                $113.80 \pm 4.83$                         $18.83 \pm 4.22$                         $100.00\% \pm 0.00\%$

PPO                                       $115.16 \pm 0.13$                         $15.00 \pm 0.00$                         $100.00\% \pm 0.00\%$

PPO-Projection                            $115.12 \pm 1.03$                         $15.33 \pm 0.52$                         $100.00\% \pm 0.00\%$

PPO-Masking                               $114.81 \pm 0.38$                         $15.00 \pm 0.00$                         $100.00\% \pm 0.00\%$

PPO-Random                                $114.87 \pm 0.50$                         $15.00 \pm 0.00$                         $100.00\% \pm 0.00\%$

PPO-Masked                                $73.06 \pm 64.29$                         $18.00 \pm 1.83$                        $66.67\% \pm 51.64\%$

MPS-TD3                                   $115.53 \pm 1.70$                         $16.83 \pm 1.47$                         $100.00\% \pm 0.00\%$

PAM                                       $11.88 \pm 41.37$                         $38.00 \pm 0.00$                         $16.67\% \pm 40.8\%$

[Average return, steps, and solved episodes - learned without obstacles]Average return, steps, and solved episodes without obstacles

That the methods are comparable in the final return is supported by the Welch's t-test in which no mean, except for PAM, is significantly different from another ($p \geq 0.05$). However, we found that discrete PPO reaches only 73.06, and TD3 with projection also made a lower average return of $91.46$ contrary to the training convergence. We discovered that the reason for the latter is an episode in which the agent misses the goal and follows a sub-optimal policy. This can be seen in the corresponding videos.
In line with training results, PAM finishes with the lowest mean cumulative reward of $11.88$. It is the only difference that is significant at the 0.01 level to all algorithms with the exception of PPO with discrete masking (p = 0.08). The trajectories indicate that the actions have a random pattern. PAM has an average step count of $38.00$ and ends closer to the starting point than other algorithms.

Additionally, table <ref> shows that MPS-TD3 has a significantly different average episode length compared to the on-policy baselines (p $\leq$ 0.05). However, the effect size can be considered small since the mean is 16.4 and the smallest among all TD3 and DQN variants. Generally, off-policy algorithms take slightly more steps than PPO. However, the differences between all other algorithms are not statistically significant at the 0.05 level.

Except for PAM, TD3 with projection and the discrete PPO, MPS-TD3, and the baselines solve all of the evaluation episodes. The single run which misses the goal in projection with TD3 has been described earlier and leads to a decrease in the fraction of solved environments to $83.33\%$. The same holds for two runs of PPO with discrete action masking. The target location is slightly missed and the border of the environment is followed away from the goal. However, contrary to the random behavior of PAM, both still pass the midpoint of the map when moving toward the target. The specific paths can be again followed in the videos.

In addition to the above, the restrictions throughout the episodes are at a minimum. This is shown by the average fraction of the allowed actions and the number of intervals in the action space. The entire set is consistently available for most algorithms. Next to PAM, appendixes <ref>, <ref>, and <ref> show only PPO with discrete masking and TD3 with random replacement and projection having fluctuations after 20 steps. Figure <ref> illustrates the average fraction of the action space and <ref> the number of disjoint intervals at each time step. The lines depict MPS-TD3 and PAM to contrast optimal and sub-optimal behavior. The larger restrictions of PAM are due to when the episode is not solved and the agent runs into the border of the environment again. Contrary, MPS-TD3 represents the behavior when an optimal path is followed. Only a single rollout comes close to the border at the beginning of the episode and causes fluctuations in the graph. All metrics, visualizations, and significance values are contained for each algorithm in appendix <ref>.

Fraction of the allowed actions

Number of intervals in the action space

[MPS-TD3 and PAM action space behavior]MPS-TD3 and PAM action space behavior

§.§.§ Discussion

The results of the experiment have several implications for the research directions in this thesis. First, the study found that PPO, TD3, and DQN can navigate a mapless environment, which is consistent with previous research [92, 61].
Due to the task's simplicity, we do not need additional information such as artificial potential fields. MPS-TD3 and the baseline modifications do not seem to hinder learning. Specifically, the outcomes of MPS-TD3 show that utilizing the critic to decide between multiple intervals seems to work. We achieved comparable results to the continuous masking baseline that scales the action into concatenated intervals. However, the effect might be due to that the rollouts are almost constantly based on a single interval. In this case, MPS-TD3 and the continuous masking baseline work similarly. In terms of convergence speed, the unmodified algorithms reached the final performance as fast as the methods strictly respecting restrictions. A similar result was found by Krasowski et al. [49] in an inverted pendulum task with a safety controller and Dalal et al. [23] who train DDPG to reach a goal while staying on a similar map to this experiment with euclidean projection.

Nonetheless, on- and off-policy learning have small discrepancies. Changes in results with restrictions may be due to the algorithms' unconstrained performances. Nevertheless, we believe differences are caused by hyperparameters and less task suitability. TD3 has turned out very sensitive to small changes as has been already reported by Henderson et al. [36]. No model stands out as clearly superior in unrestricted training performance. Future studies can aim for more suitable hyperparameters while testing harder tasks. Differences might be visible then.

Third, the study found that more research is needed to stabilize the PAM algorithm. Contrary to previous applications in different domains [26, 12], the approach has poor performance when used to divide the action space into bins. It may be useful to control the explosion of action values to improve its performance. A starting point could be using more sophisticated parameterized models such as the architectures by Bester et al. [9] or Hu et al. [41], inverted gradients [35], and learning rates that satisfy the Robbins-Monro condition [75] as suggested by Xiong et al. [98]. The comparison in subsequent experiments is not fair, as PAM performs inadequately in the unrestricted domain. The algorithm hence already fails in view of the first research question.
The same holds for PPO with discrete masking which is probably due to the configuration.

Nevertheless, the similar performance of MPS-TD3 and most methods allows us to measure the effect of restrictions in the majority of cases.

§.§ Simple Restrictions

Following the aim of this thesis, we study next how the presented algorithms perform with simple and dynamic restrictions on their action space. This is done by incorporating random obstacles into the environment.

§.§.§ Setup

The properties of the environment are the same as in the previous section but with one obstacle located at the center of the map. This obstacle produces a single restriction interval at a time. When the midpoint is reached, the agent receives the same observation as during training but a different subset of the action space is available. The possible actions for the same state change between rollouts, i.e. $s_t = s_{t'} \nRightarrow \mathcal{C}_t = \mathcal{C}_{t'}$. Hence, the restrictions can be considered dynamic according to section <ref>. We select the center as the position for obstacles since the previous subsection found that most algorithms follow a straight path to the goal.

The prerequisite that agents must pass the midpoint is also examined visually and runs with incorrect behavior are discarded. This way ensures a fair comparison among the algorithms.
Accordingly, projection with TD3 on seed two, and seeds two and three of PAM are removed. TD3 with random projection moves to the top left corner before taking the straight way. This reduces the effect of midpoint restrictions. On one hand, this procedure reduces the power of the experiments. On the other hand, only in this way, validity is ensured.

Another question that arises is if models not reaching the goal without obstacles should be included. Although they narrowly miss the target, we believe they can still provide valuable insight into reactions to restrictions. Specifically, we are interested in whether these models would still make it close to the goal or get stuck when an obstacle is introduced. This information can help us better understand what adjustments may be needed to improve their performance. As a consequence, we decided to use them in the following experiment.

Additionally, the size of restrictions can have a significant impact. Larger obstacles increase the distance between the new optimal path and the learned behavior. We sample the obstacle dimensions from a uniform distribution of $U(0.2, 6.8)$ to avoid bias towards specific scales. Note that we still consider different forms as well. Our results must generalize over a variety of restrictions.

In total, we experiment with 40 evaluation environments created with seeds from 0 to 39. Except for projection with TD3 and PAM, every setup is tested with all six training runs, resulting in 240 evaluations for each method.

§.§.§ Results

The results show that DQN with discrete masking achieves the highest performance in terms of both, reward and fraction of reached goals. It is the only method that successfully solves all of the evaluation environments. This can be read from Figure <ref> comparing the percentages of all algorithms. The success of DQN with discrete masking is statistically confirmed. The difference to every other algorithm, except for PPO with projection and masking, is statistically significant at the 0.05 level. In particular the first comes close to DQN, achieving a 97.50% success rate.

In addition, trajectories exhibit DQN following a smooth path in most cases. The average episode length is 20.52, which is relatively low compared to other off-policy baselines. For instance, MPS-TD3 requires 24.72 steps on average. However, the variability is high and the differences are not statistical (p $\geq$ 0.05). The only exceptions are TD3 with random replacement and PAM (p = 0.02). The on-policy baselines also take a similar number of steps to DQN (p $\geq$ 0.05). For instance, PPO with projection has an average episode length of 20.54 (p = 0.99).

[Portions of solved episodes with simple restrictions]Portions of solved episodes with simple restrictions

Within the same training paradigm, MPS-TD3, projection, and continuous masking tend to perform similarly. By training paradigm, we mean that a learning type is applied with different modifications. We find no significant differences between the reward, solved fraction, and steps, of the same algorithm, such as MPS-TD3 compared to TD3 with continuous masking or projection (p $\geq$ 0.05). Only PPO with random replacement has less steps than masking (p $\leq$ 0.01) and fewer returns compared to projection (p = 0.03).

Overall, on-policy learning performs better than MPS-TD3 and the off-policy baselines in environments with continuous action spaces, while the opposite is true with discrete actions. For example, PPO with projection achieves an average reward of 109.43, compared to TD3's 78.09 (p = 0.02). Although the differences are not always significant, the mean is consistently lower, pointing to a trend. Similarly, MPS-TD3 has a lower average return than PPO with projection (p $\leq$ 0.01), masking (p = 0.16), and random replacement (p $\leq$ 0.01).

The PAM algorithm achieves the lowest level of performance in all fields. The mean return is 16.76 and the agent reaches the goal in 16.88% of the cases (p $\leq$ 0.01 except for penalty-based). On average 30.34 steps are required. The trajectories indicate that even when the agent passes an obstacle, it misses the goal.

Further investigating trajectories suggests for our algorithms and the baselines that the larger the restriction, the less the reward, and the path gets less smooth. Examples are shown in figures <ref> and <ref>. More trajectories are available from appendixes <ref> to <ref>. When PPO diverges minimally from its policy, the path continues smoothly and straight. When the alternative route gets further away, random oscillations occur and it is less likely that the agent makes it to the goal.

Smaller obstacle

Larger obstacle

Circular behavior

[PPO trajectory examples with a single obstacle]PPO trajectory examples with a single obstacle

Another finding is circular behavior. The pattern is best visualized by PPO in figure <ref> and occurs for MPS-TD3, PAM, and all baselines except for DQN. The situation arises when the agent selects the same invalid actions which bring back similar states. From these states, the same choices are made. This leads to repeated and inefficient behavior. However, the situation can usually be escaped after a few rounds. We found it happening less frequently with random replacement compared to methods that project an action always to the same value.
The whole results and visualizations for the algorithms can be found in appendix <ref>.

§.§.§ Action Space

The mean size of the available action space is between 180.38 and 194.47. For instance, MPS-TD3 has an average of 187.90, indicating that around 85.41% of the action space is unrestricted. In the beginning, more actions are allowed but the portion declines between 5 and 10 steps when an obstacle occurs. Then, the restrictions shrink until most algorithms reach the goal at around 20 steps. The fluctuations afterward are caused by rollouts that do not find a straight path to the target location but run into the border of the environment first. Figure <ref> illustrates exemplary the size of allowed actions with respect to the time steps for MPS-TD3, DQN with discrete, and PPO with continuous masking. The course of the other approaches in appendixes <ref>, <ref>, and <ref> is similar.

The same pattern holds for the single intervals which are the smallest at around 10 steps. The lowest average of simultaneously available subsets is 1.06 in the case of DQN. However, this number can increase up to 1.14 for TD3, PPO, and MPS-TD3. The mean standard deviation of the subset sizes across all algorithms is 47.01. Appendixes <ref>, <ref>, and <ref> indicate that the allowed space usually consists of a single interval. The average length of the subsets comes close to the mean of the total available action space. For example, the different parts have an average size of 177.55 in the case of PPO with continuous masking. The mean minimum length across all intervals in an action space is 174.59 and the maximum is 180.50. The control variables for all algorithms are contained in appendix <ref>.

Fraction of the allowed actions for MPS-TD3 and PPO with masking

§.§.§ Discussion

Aligning with our research aim, we proposed a first way to evaluate algorithms with dynamic interval action space restrictions.
From the perspective of this experiment, we see partial evidence for our hypothesis that constraint-learning methods require different action space subsets during training. On one hand, MPS-TD3 and the off-policy baselines performed worse with respect to the previous subsection. Larger fluctuations occurred and randomly replacing actions works similarly well. We hypothesize deviations from the policy can generally not be handled well. On the other hand, on-policy differences are relatively small. Note that most authors test algorithms with restrictions when the constraints occur during training. Therefore, we added to, for example, Krasowski et al. [49] and Huang et al. [43], how the approaches perform with unseen restrictions.

In our case, DQN was the best method for previously unknown restrictions, MPS-TD3 was similar to the continuous off-policy baselines, and restrictions learned with penalties failed in the majority of cases. The latter is similar to when agents are trained with restrictions since the performance drop was also found by Brosowski et al. [14] and Kochdumper et al. [48]. Likewise to the previous subsection, off-policy continuous masking achieved the same performance as MPS-TD3. Therefore, considering the intervals separately does not seem to have an advantage when the action values for subsets are not properly learned. Nevertheless, our method adds an alternative to zero-padding. It must be investigated in subsequent experiments if the methods are also comparable when more disjoint intervals occur. The success of DQN is contrary to the discretization error in the literature [27] and promotes the use of advanced discretization techniques [88, 83]. However, similarly to Krasowski et al. [49], the reason could be that the task is too simple to exaggerate differences between algorithms and it is not necessary to choose from a fine-grained action space to bypass an obstacle efficiently.

As previously, the on-policy tend to perform better than the continuous off-policy approaches and come close to DQN. We believe the difference is due to the performance in the unrestricted environment although the discrepancies are considered relatively large. Likewise to the previous subsection, the effect could come from hyperparameters and less task suitability since TD3 was built for high dimensional action spaces in complex environments [58]. The lower performance of PPO on the discretized action space is probably caused by the learned sub-optimal policy during training. It is also important to mention that not all algorithms follow a completely straight path and have an advantage with small obstacles. Nevertheless, we believe the bias is rather small and negligible.

As expected, PAM fails to find appropriate paths. The performance was anticipated since only about 16% of the environments were initially solved in the unrestricted domain. Following the suggestions from the previous section, it is unclear to what extent the performance changed due to the restrictions. However, future research should aim at stabilizing PAM. We still believe the procedure is promising because of the connection to DQN which reached remarkable results.

Overall, the experiment seems to have worked effectively even though not a single method turned out as superior. The control variables indicated similar restrictions on the action space on the straight path to the goal between all methods.

§.§ Complex Restrictions

Since we did not find clear answers to the research questions in a simple environment, a second experiment is conducted with more complex restrictions. This is intended to make differences more clearly recognizable.

§.§.§ Setup

The complex scenario is again the same environment as in previous experiments but with 14 obstacles randomly generated using algorithm <ref>. Examples can be found in appendixes <ref> to <ref>. We assume that differences in steps are more significant when agents have to bypass multiple obstacles with narrow paths in between. Since more barricades are spread on the map, we expect restrictions to occur constantly and to have more disjoint intervals on average. Locations are sampled from a normal distribution with mean $\mu = (7.5,7.5)$ and covariance matrix $\Sigma = \big(\begin{smallmatrix}
4.0 & 0.0\\
0.0 & 4.0
\end{smallmatrix}\big)$ according to algorithm <ref>. The size of the obstacles is yielded from $\mathcal{N}(1.0, 0.25)$ and clipped to fit into the range $[0.5,1.5]$. The covariance matrix is set to bias obstacles to occur on the line between the agent starting position and the goal more frequently. As in the previous experiment, we use the seeds from 0 to 39 to generate environments which make 240 evaluation rollouts per algorithm.

§.§.§ Results

DQN achieves the highest rewards and fraction of solved environments. The mean return is 108.31 with 97.50% solved environments and on average 21.51 steps are taken to the goal. The difference in the return and solved episodes is significant with respect to MPS-TD3, PAM, and the other algorithms at the 0.01 level. Figure <ref> illustrates the mean cumulative reward and standard deviation for each approach.

Average return with complex restrictions

As a counterexample, PPO with projection has an average return of 55.10 with 48.75% solved episodes and a mean of 28.60 steps. At the same time, PPO with projection reaches the highest rewards among all on-policy baselines.

This example shows that also in terms of path optimality DQN outperforms most algorithms (p $\leq$ 0.01). Only discrete (p = 0.21) and continuous masking (p = 0.07) come closer to DQN with average episode lengths of 24.22 and 25.81 respectively. Appendix <ref> illustrates that all methods, with DQN excluded, solve less than half of the episodes and need relatively long paths to reach the goal. Exemplary routes are illustrated for traceability in appendixes <ref> to <ref>. Most of the DQN trajectories are straight without temporary runaways.

Furthermore, the performance of our proposed algorithms, MPS-TD3 and PAM, falls. The average cumulative reward is 5.35 and 10.04 respectively. The trajectories reveal that the methods already struggle with leaving the first obstacles behind. A high number of steps is necessary even when the goal is reached. In the case of MPS-TD3, the average episode length is 35.16 while it is 28.27 with PAM. MPS-TD3 solves 5.83% and PAM 12.08% of the episodes. Additionally, we find circular behavior in some rollouts. This also holds for the baselines except for DQN.

In general, the results suggest that there are no significant differences within on- or off-policy learning with strict continuous restrictions (p $\geq$ 0.05). For instance, PPO has a mean return of 52.71, 55.10, and 49.73 for masking, projection, and random replacement. On the off-policy side, a similar picture emerges. Both of our proposed algorithms reach a likewise average return. The trend holds for the length of an episode and the fraction of solved environments either. The only difference is that masking requires slightly fewer steps than random replacement (p $\leq$ 0.01). However, note that the cumulative reward is off-policy constantly lower than on-policy although the significance is only found between the projection baselines.
All results, visualizations, and significance values can be found in appendix <ref>.

§.§.§ Action Space

The average allowed action space is similar between all algorithms and ranges from 64.67% with MPS-TD3 to 68.73% in the case of PAM. The portions make average sizes of 142.27 and 151.20 respectively. Only the unmodified TD3 and PPO can select with 75.04% and 74.70% from larger fractions. As an example, the average portion of available actions is visualized for MPS-TD3 and PAM in figure <ref> over time. The courses of the other algorithms behave likewise and can be found in appendixes <ref>, <ref>, and <ref>. The restrictions are slightly lower at the beginning and the trajectories show that the only constraints are caused by the border of the environment. Afterward, most algorithms fail to find a way to the goal. They maintain large restrictions by running into obstacles again.

Allowed action space

Variance in interval sizes

MPS-TD3 and PAM action space behavior with multiple obstacles

Between 1.32 and 1.44 disjoint intervals are on average available at the same time. Only MPS-TD3 has to choose with a mean of 1.50 from significantly more intervals. The average size for a single subset for MPS-TD3 is 114.46 and 130.00 in the case of PAM. Figure <ref> illustrates the variance in the interval sizes within an action space for both methods over time. The results show that the variability is approximately constant. More metrics are described in appendix <ref>.

§.§.§ Discussion

Overall, the outcomes are similar to the previous experiment but the differences are more clear.
The experiment indicates that DQN with masking is the only method to find alternative paths which maximize the reward.
Estimating a finite number of action values in parallel and then removing unavailable parts performs better than replacing actions heuristically or scaling the output. We hypothesize the advantage comes from the learning procedure. The implicit handling makes the task less complex since no scaling needs to be learned. However, the procedure is problematic in continuous action spaces since the number of actions is infinite. More research is necessary to develop techniques without discretization.

All other algorithms, including MPS-TD3, perform significantly worse than in simple environments and our architecture is not an alternative compared to on-policy baselines. However, the outcomes are partly expected. The previous subsection predicted the trend and all experience tuples on the optimal path are based on the full action space. The agent has no reference which could indicate an advantage for an interval or an action and may confuse the obstacle with the border of the map. Therefore, imposing artificial restrictions could improve performance. Nevertheless, we expect arbitrary action space subsets to impact the difficulty of the task. Future experiments must explore the influence of artificial restrictions with varying properties and can try MPS-TD3 with different models, such as PPO. Further, we encourage integrating the performance without trained restrictions in future works from other authors. For example, when using pre-trained models, it might not always be possible to incorporate restrictions from the beginning.

In line with previous results, no superior method for continuous action spaces could be found. The outcomes only suggest that penalty-based approaches are worse. The result is problematic since reward shaping is a common practice to handle restrictions in the literature [72, 91]. Future works should consider replacing their penalty with methods such as random replacement or using DQN with a discretization of the action space. Note that results are only transferable when the agents face dynamic restrictions in their application that they did not encounter before. As previously mentioned, research in this field is sparse and it is difficult to compare our results to previous works.

In obstacle avoidance, the results promote using local data to restrict the action space when unseen obstacles occur and suggest relying on discrete actions until better continuous methods are found. For example, the sensor data of Tai et al. [87] or Meyer et al. [63] could be used to exclude intervals in which an obstacle is detected too close instead of returning less reward. We hypothesize unseen restriction handling can be improved this way. It would also be interesting to compare the performance when global information becomes available.
Overall, the experiment indicated the anticipated effect. Compared to the previous subsection, the average fraction of the allowed action space decreased and the number of simultaneous disjoint intervals was increasing. Subsequent experiments can work on more restrictions and explore the limits of DQN.

§.§ Summary

In the context of this chapter, three experiments with dynamic restrictions were carried out. We included 10 baseline algorithms and our proposed improvements MPS-TD3 and PAM. However, we failed to stabilize PAM which makes implications difficult. We explored the case when agents are not trained with restrictions but face them when applied. Related to the research question of how the performance of RL algorithms can be measured with dynamic interval action space restrictions, the environments are used to provide the first benchmarks. For answers on how an optimal alternative policy can be found, our results suggest the following guidelines. Note that we do not consider PAM further since it failed to learn even in unrestricted episodes.

* The techniques do not seem to have a negative effect on learning. The baseline algorithms and MPS-TD3 converged close to the maximum reward. Discrete PPO was an exception. However, we believe this behavior comes from the sub-optimal configuration and may not due to the restriction-handling procedure.

* Penalty-based restrictions should be avoided. Returning a negative reward continuously performed the worst. While in simple environments, knowledge from the minimal restrictions could be used to surround at least some of the obstacles, the method collapsed completely in the complex domain.

* Any other continuous algorithm works for simple restrictions. When we assume that the baselines are compared in combination with PPO, since it was generally more suitable for the environment, then the differences were only marginal. Nevertheless, projection and DQN had a slight advantage.

* Discretization and DQN masking is necessary for complex subsets. DQN with discrete masking was the only method to achieve good results in an environment with multiple obstacles. Despite the discretization error, it is therefore inevitable for previously unseen and more complex dynamic restrictions.

§ LEARNING WITH DYNAMIC OBSTACLES

When dynamic restrictions emerge during training, architectures parameterizing some of their layers should have an advantage. Domain knowledge is directly incorporated into the structure of the neural network. In this case, experience tuples allow learning the scaling behavior and interval selection. Therefore, we assume that performances change compared to models trained without restrictions. For example, MPS-TD3 is supposed to surround obstacles more smoothly than postprocessing actions with random replacement in which an invalid selection is replaced with an arbitrary feasible value.

To explore each agent's behavior, we split the experiment again into three parts: First, we train the agent to reach the goal. This is similar to the previous section except that random restrictions can already appear at this point. Afterward, we repeat the evaluations in the simple and complex domains.

§.§ Training

As in earlier analyses, assessing the performance in an unrestricted episode is crucial when it comes to measuring the effect of obstacles on algorithms. In the context of this experiment, different learning outcomes must be considered since dynamic restrictions already occur on the optimal path during training. Hence, this subsection first presents the new training results. This is again done by describing the updated learning procedure before continuing with the final performance. The experiment is supposed to explore two folds: First, we are interested in whether agents can learn under dynamic restrictions and still find the optimal path when the environment is obstacle-free. Second, it measures the new baseline performance of each algorithm which can be compared with episodes containing restrictions.

§.§.§ Setup

The following experiments are based on the same environment as in previous subsections. The specific parameters can be read from table <ref> and are described in section <ref>. Before each episode, we place four random obstacles on the map. Each of them moves between two waypoints. An exemplary setup is illustrated in figure <ref>. On one hand, more obstacles might make learning more difficult and exhibit differences more clearly. On the other hand, the influence on the training behavior of even small restrictions is currently unknown. Four obstacles have the advantage, that, due to the spread, the probability rises that the agent actually encounters one of them without making the task too complex. This is important at the beginning of training when the path of the agent varies. The covariance matrix is set to be independent and identically distributed with $\Sigma = \big(\begin{smallmatrix}
7.5 & 0.0\\
0.0 & 7.5
\end{smallmatrix}\big)$ and describes the variance around mean $\mu = (7.5, 7.5)$. The size of the obstacles is sampled from $\mathcal{N}(1.0, 0.25)$ and clipped to fit into the range $[0.25,1.75]$. The waypoints have an average distance of $\mu = 4.0$ with variance $\sigma^2 = 0.25$. The step size is 0.1. This way, more than 40 episode steps are on average required to reach the second waypoint. The movement speed of the obstacles is relatively low compared to the agent. This is necessary so that the algorithms can always escape restrictions. Note that we train the discrete PPO with manually tuned hyperparameters. To ensure the reproducibility of our results, we sample the training environments from a fixed order of seeds. We start with seed 0 and gradually increase the value by one after an episode ends.

Example of a training episode with four dynamic obstacles

The algorithms are trained with seeds 40, 41, 42, 43, 44, and 45. We run each for 800.000 time steps because of the increased complexity.

§.§.§ Results

Similarly to the previous experiments, we start presenting the results within and across training runs and then continue with the outcomes of the evaluation rollouts according to the axes of variability.

Within and across training runs
Most methods show a smooth and sharp learning curve. This behavior is illustrated in appendix <ref>, <ref>, <ref>, and figure <ref> which illustrates MPS-TD3 and TD3 with projection exemplary. The lines depict the average return with the standard deviation. MPS-TD3 and TD3 achieve mean rewards of 114.63 and 113.95 after the last training iteration. Near the end of the training, almost all episodes are solved. The fraction of goals that are reached improves likewise to the reward and shows a similar steady rise. The solved portion while training progresses is depicted in figure <ref>. That the task is learned quickly is also indicated by the number of time steps necessary to solve more than 80% of the episodes on average. MPS-TD3 reaches the plateau of convergence after 8.000 and TD3 with projecting invalid actions requires 15.000 time steps. Afterward, the variability stays small and the algorithms keep a constant reward. The learning behavior holds for all methods with the exception of a few.

Portion of solved episodes

Training progress with dynamic restrictions

For example, PAM does not learn a proper policy. The reward constantly fluctuates around zero and the majority of episodes are not solved throughout all time steps. Both patterns are likewise depicted in figure <ref>. The shaded area additionally shows a larger standard deviation. It must be taken into account that, similarly to the previous section, we found that the action values are rising constantly. The average return increases at the beginning until the action value estimates get larger than the true values. Afterward, the model gets unstable and unlearns the previous. The inability to find a path is also indicated by the mean episode length in appendix <ref>, which sticks near the maximum of 40 steps.

Furthermore, PPO with discrete masking and penalty-based restrictions find a sub-optimal policy. The lower rewards and fewer solved episodes can be read from figure <ref> as well. The plot example shows the unmodified TD3 algorithm which behaves similarly to PPO. Compared to the other baselines, the progress has not the same sharpness and the across training runs variability is higher. The latter is indicated by the larger standard deviation.

However, the average episode length of penalty-based methods does not seem different from algorithms with a higher reward after convergence. Since collisions occur more frequently at the beginning, the average number of steps is increasing instead of declining. In appendixes <ref> and <ref>, the rise can be considered as strong as the progress of algorithms solving all episodes. Note that this does not hold for the discrete PPO which converges to approximately 26 steps on average. The decline in the average episode length is also less steady.
Generally, a smaller portion of actions is available near the beginning of training which increases simultaneously with the reward. For example, MPS-TD3 could only choose from approximately 70% of the unrestricted action space during the first iterations. After convergence, the average available fraction is constantly oscillating around 80%. Throughout the training, the actions are split over an average of approximately two intervals. Appendix <ref> indicates that this is similar between all algorithms and includes the other metrics from section <ref>.

After training
In the evaluation rollouts without obstacles, all runs, except for PAM, find proper paths. This is shown by the average return, episode length, and fraction of solved environments which are described in table <ref>.

1cApproach     1cReturn     1cSteps     1cSolved
TD3                                       $115.02 \pm 2.18$                       $16.50 \pm 1.76$                       $100.00\% \pm 0.00\%$

TD3-Projection                            $114.15 \pm 6.11$                       $17.83 \pm 5.49$                       $100.00\% \pm 0.00\%$

TD3-Masking                               $117.54 \pm 0.27$                       $16.00 \pm 0.00$                       $100.00\% \pm 0.00\%$

TD3-Random                                $115.56 \pm 5.11$                       $17.50 \pm 3.67$                       $100.00\% \pm 0.00\%$

DQN-Masked                                $117.61 \pm 0.09$                       $16.00 \pm 0.00$                       $100.00\% \pm 0.00\%$

PPO                                       $114.82 \pm 1.07$                       $15.17 \pm 0.41$                       $100.00\% \pm 0.00\%$

PPO-Projection                            $115.17 \pm 0.08$                       $15.00 \pm 0.00$                       $100.00\% \pm 0.00\%$

PPO-Masking                               $115.73 \pm 1.91$                       $16.00 \pm 0.00$                       $100.00\% \pm 0.00\%$

PPO-Random                                $115.03 \pm 0.23$                       $15.00 \pm 0.00$                       $100.00\% \pm 0.00\%$

PPO-Masked                                $115.04 \pm 1.54$                       $17.00 \pm 0.89$                       $100.00\% \pm 0.00\%$

MPS-TD3                                   $115.76 \pm 1.68$                       $15.67 \pm 0.52$                       $100.00\% \pm 0.00\%$

PAM                                       $0.20 \pm 13.78$                        $-$                                    $0.00\% \pm 0.00\%$

[Average return, steps, and solved episodes - learned with obstacles]Average return, steps, and solved episodes without obstacles

Although according to appendixes <ref> and <ref> some differences in the metrics are statistically significant, the effect size can be considered small. An example is TD3 with continuous masking and PPO projecting invalid actions. The rewards are 117.54 and 115.17 respectively so the difference is only 2.37 (p $\leq$ 0.01). The same pattern can be found in the average episode length where the mean number of steps is 16.00 and 15.00 (p $\leq$ 0.01). An equivalent fit is additionally supported by the fact that MPS-TD3 and the baselines reach the goal in all of the environments. It is only necessary to take into account that PAM fails to solve a single episode and reaches mean returns of 0.20 which is significantly lower (p $\leq$ 0.01). All results, visualizations, and significance values are illustrated in appendix <ref>.

The action space behaves likewise to the direct path found in most trajectories. Appendixes <ref>, <ref>, and <ref> indicate that typically only a single interval is available. The restrictions occur at the beginning of an episode for the same reason as in the previous section: The waypoints show that the agent starts next to a border of the environment. As an example, figure <ref> illustrates the fraction of the allowed actions for MPS-TD3, PAM, and TD3 with random projection over time. Near the end of an episode, single runs of the latter cause fluctuations.  By this means, the algorithms pass the center of the map but hit the border right next to the goal before entering it. The action space is summarized with all introduced metrics in appendixes <ref> and <ref>. Further, the variables are illustrated over time in appendixes <ref>, <ref>, and <ref>.

[Allowed action space over time]Allowed action space over time

§.§.§ Discussion

The results indicate that strictly respecting restrictions can improve learning progress and final performance. The conclusion is similar to Brosowski et al. [14] with static and convex, as well as Huang et al. [42] with discrete restrictions.
We hypothesize that the reason is the more accurate action space. Likewise, Kanervisto et al. [45] could improve PPO by experimenting with the number of actions.
When the actions contain invalid subsets, agents learn from more actions. Otherwise, they focus on the available parts from the beginning, making the task less complex.

The advantage seems to be connected to improved exploration. Specifically, in the presented environment this might have a significant impact. Since an episode ends when an invalid action is selected, it is less likely that the agent explores states further away from the starting location at the early stages of training. Restrictions have to be learned before approaching more distant locations. This is not the case for algorithms respecting restrictions by design and the agent might reach the goal earlier by chance.
This means that the penalty-based approaches find the target location less frequently at the beginning which could slow learning. The goal might have a greater impact than small improvements toward the target.
Our proposed algorithm MPS-TD3 comes close to the optimal path within the first training iterations. The outcome is similar to all baselines except for the penalty-based methods and discrete PPO. The unmodified algorithms confirm Karasowski et al. [49] who also report lower rewards near the beginning. We assume that the discrete PPO results are again due to the sub-optimal configuration. We add to the literature that MPS-TD3 can learn to choose between different intervals. However, the performance is the same as when the subsets are concatenated. It is unclear how the outcomes differ when more intervals occur.

As in the previous section, our results promote strictly respecting restrictions in obstacle avoidance as opposed to reward-shaping. Sensor or image data could cut dangerous angles out of the action space. By this means, the convergence time may improve and routes would be, contrary to Zhang et al. [107], completely collision-free. Future experiments can also try to combine the two methods: Likewise to Meyer et al. [63], the reward increases as agents approach barriers but actions are replaced if they lead to a collision. While Choi et al. [19] improved the performance by masking discrete actions that lead to collisions of every algorithm, we were investigating the actual effect of restrictions in a continuous domain.

Overall, the similar results of the algorithms ease the comparison in subsequent experiments. The desired impact of obstacles during training could be achieved. However, PAM could not solve any of the environments. In line with obstacle-free episodes, we anticipate the action values to be responsible for the instability.

§.§ Simple Restrictions

The following part of this thesis uses the agents from the previous subsection and investigates how the approaches perform with simple constraints when dynamic restrictions already occurred during the learning process.

§.§.§ Setup

Similarly to before, the evaluation environment contains a single obstacle in the center of the map. The implementation is the same as in section <ref>. Note that for some episodes, the obstacle has a comparable size and form to what the agents already encountered during training. For this reason, we expect generally higher performance. Nevertheless, algorithms might handle the restrictions in various ways revealing differences. The similarity to section <ref> allows comparing the results to agents trained without dynamic restrictions. As formerly, we check each algorithm to pass the midpoint of the map visually and remove the runs of seed four for the unmodified TD3 as well as seed five for TD3 with projection and seed two with random replacement. The setups are sampled using seeds from 0 to 39.

§.§.§ Main Results

The results show that DQN yields the highest average return and is the only approach reaching the goal in all episodes. Figure <ref> compares the mean cumulative rewards of MPS-TD3 and PAM to the baseline algorithms. Illustrations of the other metrics can be found in appendixes <ref> and <ref>. The difference of DQN is significant with respect to all on-policy methods (p $\leq$ 0.05) also because of the lower standard deviation as indicated by the black bars. The mean return is 115.34 suggesting a straight path. DQN shows relatively low episode lengths with an average number of 17.90 steps. In contrast, MPS-TD3 takes a mean of 19.99.

[Average return with simple restrictions]Average return with simple restrictions

However, no difference in the rewards could be found between off-policy algorithms (p $\geq$ 0.05). For example, MPS-TD3 is close to DQN with an average cumulative reward of 110.82. All methods reach mean returns of more than 100. A similar picture emerges for the fraction of solved episodes (p $\geq$ 0.05). MPS-TD3 reaches the goal in 97.92% of the cases which is only 2.09% more than masking.

Appendix <ref> shows that projection has the lowest average episode length on- and off-policy.
The approach needs off-policy a mean of 17.23 and therefore significantly fewer steps than all algorithms. Exceptions are the unmodified TD3 (p = 0.06), both masking approaches (p $\geq$ 0.05), and PPO with projection (p $\leq$ 0.01). The mean episode length of the on-policy projection is with 16.39 significantly the lowest (p $\geq$ 0.05). However, note that only 80% of the episodes are solved. DQN, as the approach with the highest reward, requires 17.90 actions on average.

Overall, on-policy learning shows marginally less performance compared to the off-policy counterparts. The average return is lower for penalty-based restrictions, projection, discrete masking, and random replacement (p $\leq$ 0.01). The figure above shows that the return drops on average below 100 for the unmodified PPO and the combination with projection, and discrete masking. For example, penalty-based restrictions reach 90.77 while TD3 makes an average return of 107.03.

Out of all on-policy baselines, continuous masking achieves 111.61 and therefore the highest reward (p $\leq$ 0.05). The same pattern holds for the fraction of solved episodes. The method solves 97.08% of the environments while the second highest percentage is a portion of only 91.67% in the case of random replacement.

Additionally, trajectory analysis reveals that circular behavior happens less frequently in this experiment than in agents trained without dynamic restrictions. This can be followed in the videos and paths for each approach in appendixes <ref> to <ref>.
The results for all algorithms and metrics which are not listed in this subsection are summarized in appendix <ref>.

§.§.§ Action Space

The action space follows a similar behavior to section <ref>.  We illustrate the allowed fraction and number of available intervals for MPS-TD3 and PPO with continuous masking in figure <ref>. Appendixes <ref>, <ref>, and <ref> show that the patterns are similar between all algorithms. The trajectories reveal that the agents start with constraints due to the border of the environment. Afterward, no restrictions occur until the obstacle blocks the path. On the straight way to the goal, the action space is the smallest on the 10th step. The average number of disjoint intervals increases since the obstacle could split the action space into multiple parts. As the previous subsection indicated that most algorithms reach the target location at approximately 20 steps, larger fractions of the actions are available until then. In most cases, the straight path to the goal without restrictions is taken after the obstacle has been passed. The variability after approximately 20 steps is due to runs that miss the target and follow a sub-optimal policy.

Fraction of the allowed actions

Number of intervals in the action space

[Action space behavior of MPS-TD3 and PPO with masking]Action space behavior of MPS-TD3 and PPO with masking

Over entire episodes, the average size of the allowed action space is between 165.93 for PPO with projection and 192.99 in the unmodified on-policy case. This makes 75.42% and 87.72% respectively. The mean of other algorithms is in between these two values. The single intervals are with lengths between 162.15 and 187.94 on average only slightly smaller. This is already indicating the low variance within an action space. Since the average number of disjoint intervals is relatively low, the dispersity does not increase unexpectedly high. The reason is that most of the time only a single interval is available causing the variance to be zero. For example, in the case of PPO, the average size of the smallest interval in the action space is 185.29 while the largest is 190.58. This pattern can also be observed in appendix <ref> over time. Until the 20th step, the mean standard deviation of the interval sizes is constantly below 22 and the lengths are close. From appendixes <ref>, <ref>, and <ref> follows that the standard deviation oscillates for most algorithms at around 15. Appendix <ref> further contains all metrics for every algorithm.

§.§.§ Discussion

Likewise to the previous section, the answer is unclear when looking for a preferred method to handle interval action space restrictions. Similarly to the agents trained without dynamic obstacles, DQN was the most stable and achieved the highest results. However, differences are only marginal, and it is uncertain if the results hold for more complex environments due to the discretization error.

MPS-TD3 is comparable to DQN but works in continuous action spaces. More tests can show if our approach is preferable to the other baselines due to the ability to handle a possibly infinite number of subsets. In contrast to section <ref>, the unmodified algorithms perform similarly to strictly following restrictions. These findings support that sensor [87], image [33], or global data [63] is sufficient to pass simple dynamic obstacles. Future works could explore the setups of Wang et al. [91] to see how all approaches deal with dead ends.

We hypothesize the general high performance of all approaches might be due to overfitting. The presented scenario could have appeared likewise during training in which all algorithms have learned proper paths. The exception is obstacles larger and smaller than the ones during training. Nevertheless, it is unlikely that the obstacle was found exactly in the middle of the map and the same restrictions occurred. Therefore, the constraints can still be considered new. Due to the equal results, it is questionable if the presented environment is useful in terms of the question of how the performance of algorithms with dynamic restrictions can be measured or if the performance is really almost equivalent. Future experiments are necessary.
During the analysis, it has been shown that continuous masking and projection achieved lower step counts than MPS-TD3. This could suggest better restriction handling and would mean that scaling into the concatenation of the multiple disjoint intervals performs better than considering each separately. However, the projection results could be related to the fewer solved episodes. These episodes might be the ones with larger obstacles requiring longer paths. Subsequent works should consider controlling for the size of obstacles more specifically. Note again that some algorithms do not follow a completely straight path and have an advantage with small obstacles.

However, in summary, the obstacles have shown the desired effect of restrictions similar to the agents trained without obstacles in section <ref>.

§.§ Complex Restrictions

After the previous subsection presented the outcomes with a simple environment, the following repeats the evaluation in the more complex domain.

§.§.§ Setup

Again, the setup is the same as in section <ref>. We use the previously presented implementation since we intend to compare the results with the earlier agents. The environments are sampled with seeds from 0 to 39 which makes a total of 240 evaluation rollouts per algorithm.

§.§.§ Results

The outcomes show that MPS-TD3, projection, masking, and DQN are similar in average return. All of them achieve high rewards with only a few minor differences being significant at the 0.05 level. For example, TD3 with continuous masking reaches 113.88 compared to MPS-TD3 with 110.73 (p $\leq$ 0.01). Contrarily, DQN, MPS-TD3, PPO with continuous masking, and both projection approaches are not significantly different from each other (p $\geq$ 0.05).

Likewise to the return, the average fraction of solved episodes can be considered generally high. MPS-TD3 and TD3 with continuous masking solve most episodes. Both reach the goal in 99.17% of the rollouts. This implies that the algorithms miss the target in only two out of the 240 environments. Exceptions are the unmodified TD3 and PPO that attain with 33.75% and 6.67% the target location less frequently. Additionally, PPO with discrete masking and random replacement find a way in only 82.01% and 83.33% of the cases. Our proposed PAM algorithm does not solve a single episode. Nevertheless, even randomly replacing invalid actions with TD3  arrives 95.42% of the time.

1cApproach     1cReturn     1cSteps     1cSolved

TD3                                       $43.94 \pm 19.87$                       $22.89 \pm 2.17$                       $33.75\% \pm 17.23\%$

TD3-Projection                            $107.36 \pm 6.57$                       $19.80 \pm 4.90$                       $95.00\% \pm 4.47\%$

TD3-Masking                               $113.88 \pm 1.28$                       $17.59 \pm 0.81$                       $99.17\% \pm 1.29\%$

TD3-Random                                $106.85 \pm 4.65$                       $21.36 \pm 2.34$                       $95.42\% \pm 4.31\%$

DQN-Masked                                $112.14 \pm 2.40$                       $17.88 \pm 0.28$                       $96.25\% \pm 2.62\%$

PPO                                       $12.70 \pm 7.96$                        $22.02 \pm 5.80$                       $6.67\% \pm 4.92\%$

PPO-Projection                            $110.61 \pm 3.06$                       $16.45 \pm 0.23$                       $95.00\% \pm2.74\%$

PPO-Masking                               $106.80 \pm 4.92$                       $18.15 \pm 0.75$                       $86.03\% \pm 2.86\%$

PPO-Random                                $94.56 \pm 4.39$                        $21.83 \pm 1.61$                       $83.33\% \pm 4.38\%$

PPO-Masked                                $67.56 \pm 1.45$                        $22.86 \pm 1.76$                       $82.01\% \pm 1.66\%$

MPS-TD3                                   $110.73 \pm 1.42$                       $21.10 \pm 0.10$                       $99.17\% \pm 2.04\%$

PAM                                       $-10.87 \pm 3.94$                       $-$                                    $0.00\% \pm 0.00\%$

[Average return, steps, and solved episodes - complex restrictions]Average return, steps, and solved episodes

In terms of path optimality, we found that MPS-TD3, random replacement, discrete PPO, and unmodified algorithms have higher average episode lengths. For example, while having relatively high rewards and many solved episodes, MPS-TD3 has an average episode length of 21.10 with a low standard deviation. Randomly replacing invalid actions takes mean steps of 21.36 and 21.83. Therefore the required actions are significantly higher than the ones of both continuous masking algorithms, PPO with projection and DQN (p $\leq$ 0.05).

Overall, PAM, discrete PPO, and unmodified algorithms perform worse across all metrics. For instance, TD3, and PAM achieve significantly lower mean returns of 43.94 and -10.87 compared to the 113.88 of TD3 with continuous masking (p $\leq$ 0.01). Also, the discrete PPO attains with an average of 67.56 lower cumulative rewards (p $\leq$ 0.01). Table <ref> shows that the pattern holds across metrics.

Just like in the previous sections, we observe that on-policy approaches tend to achieve lower values than off-policy methods for all the metrics being considered. For example, PPO with random replacement solves only 83.33% of the episodes and achieves a relatively low mean return of 94.56 compared to 95.42% and 106.85 (p $\leq$ 0.01). Although statistical significance is missing for the on- and off-policy difference between the continuous masking algorithms, the mean cumulative reward and fraction of solved episodes is higher. Only projection reaches in combination with PPO an average return of 110.61 as opposed to 107.36 with TD3 (p $\geq$ 0.05). The general trend is best illustrated in appendixes <ref>, <ref>, and <ref>. All visualizations of the metrics comparing our algorithms and the baselines can be found in appendix <ref>.

§.§.§ Action Space

The action space is similar to section <ref> but restrictions get fewer as the agent approaches the goal. The total average available length is between 140.90 for TD3 with projection and 160.93 for the unmodified PPO. Percentage-wise these are 64.04% and 73.15%. All other algorithms, including MPS-TD3 and PAM, have a mean in between the two values. On average, the actions are split over 1.28 and 1.39 disjoint intervals. Figure <ref> shows the allowed fraction and number of available intervals for MPS-TD3 and PPO over time. The algorithms are selected as exemplary. Appendix <ref>, <ref>, and <ref> demonstrate an equivalent course for the other methods. The restrictions are smallest near the beginning and then increase until the size of the allowed action space is smallest at approximately the 15th step. Afterward, more actions become available before most agents reach the goal. The standard deviation in the figure suggests that the variability is relatively small. Only the second half of the maximum episode length shows larger fluctuations caused by single runs missing and reaching the goal later.

The average size of a single interval is approximately 20 smaller than the sum of all subsets. The appendix shows that this is similar between all algorithms. For example, the available actions of MPS-TD3 make an average of 144.93 while the individual intervals have a mean length of 124.01. The smallest interval has an average size of 113.56 and the largest 134.53. The course behaves likewise to figure <ref>. The fewest actions are available before the 15th step. Correspondingly, the standard deviation of the interval sizes within the action space is highest between the 5th and 15th actions with means oscillating around approximately 22. The average and pattern over time of each metric are contained in appendix <ref>.

Fraction of the allowed actions

Number of intervals in the action space

[Action space behavior for MPS-TD3 and PPO]Action space behavior for MPS-TD3 and PPO

§.§.§ Discussion

The outcomes provide valuable insights that are relevant to the research question of this thesis.
In line with our expectations, MPS-TD3, continuous masking, and projection significantly improved their performance compared to section <ref> and are an alternative to DQN. Each of the algorithms comes with strengths and weaknesses. For example, while MPS-TD3 is worse in finding the shortest path, the approach is generally more likely to reach the goal. Training with obstacles seems to allow an understanding of the scaling behavior and action values of intervals. Likewise, to Krasowski et al. [49], the decision for an algorithm depends on the task at hand. However, the results suggest that the impact of the choice can be minor. Further tests may explore if differences emerge in more complex environments. We also believe that projection highly depends on the implementation. In our case, the projected action is close to the most efficient way to pass an obstacle. Hence, the performance may be overestimated. Subsequent studies can analyze the effect with specific environment layouts and obstacle forms.

Second, random replacement, PAM, and unmodified algorithms took more steps or solved significantly fewer episodes. The results are contrary to the experiments with a single obstacle indicating that the approaches are more suitable for simple domains.
Randomly replacing invalid actions was only slightly worse. We assume strategies must be learned or well-designed as in the projection case. It would be interesting if sophisticated techniques such as extra replay buffers [38] improve results. However, the approach is preferable over penalty-based restrictions and PAM. The unmodified algorithms should be avoided after they have also shown slower convergence. PAM can not be evaluated as the training failed.

In the context of obstacle avoidance, the outcomes are similar to section <ref> and suggest strictly avoiding obstacles in complex environments when the information is available. Contrary to the simple rollouts, the unmodified approaches have problems when many obstacles have to be passed. However, discretization for DQN is not required as opposed to training without restrictions.

Overall, the action space had similar sizes as in the previous subsection but restrictions emerged continuously and with more intervals. Nonetheless, experiments in different tasks are necessary to control the transferability of results.

§.§ Summary

In this part of the thesis, we repeated the three experiments from section <ref>. Contrary to before, we optimized MPS-TD3. PAM and each of the 10 baseline algorithms with dynamic interval restrictions during training. By this means, the procedure explored the research questions more similarly to recent literature. Nevertheless, we found that experiments are not complete and need to be expanded in subsequent works. As previously, the results suggest specific recommendations that can be used to address the research questions and guide future directions. Note that we again do not consider PAM since the method could not be stabilized in this section either.

* Strictly restricting restrictions can improve return and convergence. The original TD3 and PPO converged slower than the other approaches and reached fewer returns between the last training iterations. Note that it is questionable if this holds for the discrete PPO due to the sub-optimal hyperparameters in our experiments.

* All techniques can work properly with simple restrictions. When only off-policy algorithms are considered, the differences were only marginal and all achieved a high reward. However, this does not hold in combination with PPO. Overall, DQN had slight but not significant advantages and a lower variability on all metrics.

* Penalty-based restrictions should be avoided with complex subsets. Returning a negative reward achieved significantly fewer returns when restrictions were more complex compared to the simple domain. The method seems not suitable when restrictions increasingly differ from training.

* Complex replacement strategies must be learned or well defined. Randomly replacing actions achieved high rewards but took longer paths than more sophisticated techniques. However, we also believe that projecting to fixed valid points must be well-designed to reach similar or better results as in this experiment.

CHAPTER: CONCLUSION

After presenting the experimental results, the next sections conclude the paper with a summary, a description of limitations, and an outlook for future work.

§ SUMMARY

This thesis applied dynamic interval action space restrictions in deep RL.
The first goal was to find algorithms following an optimal policy with varying allowed actions. We developed two approaches that comply with an arbitrary number of subsets.
First, PAM builds on the success of parameterized RL and uses P-DQN to divide the action space into predefined bins. The bins allow applying the theoretically well-founded discrete action masking but lower the discretization error. The trick is to combine each bin with a fine-grained action. Second, MPS-TD3 modifies TD3 to dispense the discretization and considers each of the intervals separately through multiple forward passes. The actions are scaled into the specific subset by parameterizing the output layer of the neural network. Afterward, the critic decides on the action to execute in the environment. By this means, we added an approach to support disjoint subsets to ConstraintNet. The algorithms were compared to common approaches to handle restrictions in the literature. The baselines included projection, replacement, as well as discrete and continuous masking. We applied PPO and TD3 unmodified and with each extension.

The second goal was to find methods for evaluation algorithms with dynamic interval restrictions. We started by developing fundamental criteria. Afterward, we presented an obstacle avoidance environment with metrics to compare policies and monitor the effect of restrictions. We analyzed the learning behavior and tested agents in simple and complex environments. The experiments differentiated when algorithms were trained with and without obstacles.

The outcomes had several implications for future research directions.
In terms of training progress, our study demonstrated that also with dynamic constraints, strictly adhering to restrictions can improve convergence and final performance. MPS-TD3 performed better than the unmodified algorithms but worked similarly to projection, masking, and random replacement. However, the advantage disappeared without dynamic restrictions. Only PAM turned out to overestimate action values and requires further modifications. Therefore, the method can currently not be regarded as a solution for the first research question of this thesis. Nevertheless, we presented recommendations that might help in stabilizing the algorithm. Since suggestions between the other approaches could not be found, we believe that our tasks were too simple to exaggerate differences clearly and leave complicated environments for future work.

We also conducted experiments with already converged agents. The outcomes indicated that discretizing a continuous action space and masking action values with DQN is inevitable when similar restrictions did not emerge in the training episodes. All continuous algorithms suffered from major performance drops as the unseen constraints got more complex.
Afterward, we repeated the experiments with agents that already encountered obstacles during training. The results revealed that MPS-TD3, continuous masking, and projection can be used likewise to DQN. By this means, MPS-TD3 can be a continuous alternative without defining a maximum number of possible intervals prior to training. Nevertheless, the final decision between the algorithms seems to depend on the task at hand. We assume differences might become visible in other environments but no further experiments could be conducted due to the scope of this work.

Besides, our results provided a new perspective on obstacle avoidance. While former works focused on shaping the reward, action, and observation space prior to training, we assume that the available information can restrict the agent to subsets of the actions at each time step individually. Contrary to studies that projected invalid actions into static and convex intervals, we evaluated disjoint subsets that can vary for the same observation. Our approach was collision-free from the beginning and has shown evidence that the flexible but more precise action space may improve pathfinding around obstacles.

In summary, we could not conclusively clarify the question regarding RL algorithms for optimal policies with dynamic interval action space restrictions but gave insights for future work. We presented an obstacle avoidance task that provided first benchmarks and motivate extending the environment for further experiments.

§ LIMITATIONS

Although we monitored the effect of obstacles on the action space and varied the difficulty of our tasks, our study is due to several limitations.
A list of aspects that must be considered regarding the validity of our results is presented below.

One-Dimensional Action Space The insights only hold for actions of a single dimension. As soon as the action space becomes multi-dimensional, the complexity of the task itself and MPS-TD3 rises drastically as the number of convex subsets increases. The performances might change and algorithms may be better suited for low- or high-dimensional problems. Further, actions of different forms can be combined. For example, the gym framework allows the integration of continuous, discrete, binary, and textual actions into composites such as dictionaries, tuples, and graphs. The spaces can consist of arbitrary subsets in different forms. Generally, the proposed approaches are capable of dealing with more complex structures as the actions are translated to discrete or continuous outputs but combinations of approaches are necessary which was not explored in this thesis.

Simple Restriction Properties Obstacles in our experiments produce relatively simple restrictions so that often a single interval is available. In practice, the number of disjoint subsets can get large and vary drastically. It might be inefficient or impossible to apply extensive zero-padding to the input. In other cases, restrictions may not be appropriately represented by intervals at all. For example, when the continuous output of a neural network in a highly non-linear embedding space should be limited. Then storing all intervals for each dimension can be infeasible. Additionally, note that restrictions might be interdependent. For example, a range of a dimension might be only allowed when another action lies in a specific interval. Only the projection and replacement baselines could handle the dependencies as no input representation is required. The projection would need a distance function that projects to the closest valid combination. Contrary to the other methods, the allowed subset is not learned and masking is not performed separately for each dimension. However, the results are based on heuristics.

Generalization Our outcomes only hold for the presented domain of obstacle avoidance in mapless navigation. No claims can be made about the transferability of the results to other tasks or algorithms. For example, TD3 is generally more suitable for high-dimensional problems and performance may change in other environments. Specifically, real-world transfer has been previously shown to be difficult [111]. Learning physics for a robot to surround obstacles with extraneous impacts is more demanding and differences between the baselines and MPS-TD3 might become visible. Therefore, we expect the performance of each algorithm to be strongly task-dependent. Furthermore, properties, such as that the valid action of projection is close to the optimum, can not be assumed. By this means, the limitation includes different obstacle avoidance environments as well as unrelated fields.

Availability of Restrictions We assume that the restrictions can be calculated precisely and are provided with each step from the environment. However, it is not always possible to obtain fast and reliable constraints in practice. The computation time can differ and the interval boundaries might be not exact. For example, an autonomous driving situation might have already passed until the allowed motor parameters are approximated. Noisy restrictions simultaneously increase task complexity and would allow collisions even when the constraints are correctly followed. Besides, it might be possible that a finite set of restrictions can not be pre-computed at all. In this case, ways to limit an agent to unknown allowed subsets must be found.

Hyperparameter Sensitivity In this work, all algorithms were optimized with respect to the first 150.000 time steps. We believe that the differences in the performance without restrictions are due to the configurations. The initial differences between the approaches may change when hyperparameters are tuned with respect to more training iterations and seeds which was not possible due to the time scope in this thesis. The PPO with discrete masking was also learned with sub-optimal settings because of an error during the training process and the search could not be repeated. It is unclear if the approach would perform similarly to its off-policy counterpart.

§ RESEARCH OUTLOOK

The limitations from the previous section can provide starting points for future research. Besides adding more algorithms from the literature and performing more hyperparameter optimization, we consider the following points as the next steps.

PAM Stability We believe more effort should be spent on stabilizing the PAM algorithm. The approach masks action values similar to DQN. The discrete method was the only effective algorithm when restrictions were applied to the agents trained without obstacles. Therefore, PAM could be a promising continuous alternative with fine-grained actions. The stabilization may be achieved by inverting the gradients and using learning rates that satisfy the Robbins-Monro condition. However, more sophisticated parameterized on- and off-policy architectures, such as H-PPO proposed by Fan et al [28], can also be considered.

Action Space Properties The developed environment can be used to control for different properties of restrictions more specifically. For example, smaller and larger paths between obstacles can manipulate the fraction of the allowed action space during an episode. Our experiments indicated that the performance gets less when the alternative deviates more from the learned policy. However, future experiments should aim to implement a metric that controls for the deviation of the optimal restricted policy. It would also be interesting to compare the results to outcomes when the action space continuously consists of more disjoint intervals on average. The effect can be achieved by setting a large number of small obstacles on the map. Afterward, the impact can be measured and compared to the action space variables of this thesis. Moreover, a combination of static training and dynamic evaluation obstacles could make the selection of the interval actually matter. For example, an inefficient choice might make the agent take a longer path around a known static obstacle. Finally, future work can add more dimensions to the action space to approach more complex restrictions. The step size may be selected by the agent instead of being defined beforehand.

Transferability The generalization can be tested in different tasks. For example, interval restrictions arise in variations of multi-step bandit problems where taking an action blocks a subset temporarily in the next steps. Consider an action space $\mathcal{A} = [1,1]$ with an unknown underlying reward function and assume that a selection makes an interval of size 0.1 unavailable. Then action $a = 0.5$ produces the dynamic action space $\mathcal{A}_\varphi = \{[-1,0.4], [0.4,1.0]\}$ for the next time step.  An implementation of such a multi-step bandit problem based on maximizing oil extraction is implemented on the author's GitHub repository. Other applications are the cake-cutting problem, multi-agent systems in which agents depend on each other, and a governance that imposes external restrictions.

Restriction Correctness Different noise processes can be used to disturb the interval boundaries of restrictions. This way, approximations are simulated and the combination of collisions and dynamic restrictions is analyzed. The algorithms have to learn when to rely on the provided constraints and may handle perturbations differently well. This way, the approaches take a step toward real-life environments.

[1]
Joshua Achiam, David Held, Aviv Tamar, and Pieter Abbeel.
Constrained policy optimization.
In International Conference on Machine Learning, pages 22–31.
PMLR, 2017.

[2]
Laha Ale, Scott A King, Ning Zhang, Abdul Rahman Sattar, and Janahan
D3pg: Dirichlet ddpg for task partitioning and offloading with
constrained hybrid action space in mobile-edge computing.
IEEE Internet of Things Journal, 9(19):19260–19272, 2022.

[3]
Prithviraj Ammanabrolu and Matthew Hausknecht.
Graph constrained reinforcement learning for natural language action
arXiv preprint arXiv:2001.08837, 2020.

[4]
Leo Ardon, Alberto Pozanco, Daniel Borrajo, and Sumitra Ganesh.
Inapplicable actions learning for knowledge transfer in reinforcement
arXiv preprint arXiv:2211.15589, 2022.

[5]
Kaushik Balakrishnan, Punarjay Chakravarty, and Shubham Shrivastava.
An a* curriculum approach to reinforcement learning for rgbd indoor
robot navigation.
arXiv preprint arXiv:2101.01774, 2021.

[6]
Gianpietro Battocletti, Riccardo Urban, Simone Godio, and Giorgio Guglieri.
Rl-based path planning for autonomous aerial vehicles in unknown
In AIAA Aviation 2021 Forum, page 3016, 2021.

[7]
James Bergstra, Brent Komer, Chris Eliasmith, Dan Yamins, and David D Cox.
Hyperopt: a python library for model selection and hyperparameter
Computational Science & Discovery, 8(1):014008, 2015.

[8]
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysaw Debiak,
Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse,
et al.
Dota 2 with large scale deep reinforcement learning.
arXiv preprint arXiv:1912.06680, 2019.

[9]
Craig J Bester, Steven D James, and George D Konidaris.
Multi-pass q-networks for deep reinforcement learning with
parameterised action spaces.
arXiv preprint arXiv:1905.04388, 2019.

[10]
Homanga Bharadhwaj, Aviral Kumar, Nicholas Rhinehart, Sergey Levine, Florian
Shkurti, and Animesh Garg.
Conservative safety critics for exploration.
arXiv preprint arXiv:2010.14497, 2020.

[11]
Omar Bouhamed, Hakim Ghazzai, Hichem Besbes, and Yehia Massoud.
Autonomous uav navigation: A ddpg-based deep reinforcement learning
In 2020 IEEE International Symposium on Circuits and Systems,
pages 1–5. IEEE, 2020.

[12]
Salah Bouktif, Abderraouf Cheniki, and Ali Ouni.
Traffic signal control using hybrid action space deep reinforcement
Sensors, 21(7):2302, 2021.

[13]
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman,
Jie Tang, and Wojciech Zaremba.
Openai gym.
arXiv preprint arXiv:1606.01540, 2016.

[14]
Mathis Brosowsky, Florian Keck, Olaf Dünkel, and Marius Zöllner.
Sample-specific output constraints for neural networks.
In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 35, pages 6812–6821, 2021.

[15]
Stephanie CY Chan, Samuel Fishman, John Canny, Anoop Korattikara, and Sergio
Measuring the reliability of reinforcement learning algorithms.
arXiv preprint arXiv:1912.05663, 2019.

[16]
Weijia Chen, Yuedong Xu, and Xiaofeng Wu.
Deep reinforcement learning for multi-resource multi-machine job
arXiv preprint arXiv:1711.07440, 2017.

[17]
Wuhui Chen, Xiaoyu Qiu, Ting Cai, Hong-Ning Dai, Zibin Zheng, and Yan Zhang.
Deep reinforcement learning for internet of things: A comprehensive
IEEE Communications Surveys & Tutorials, 23(3):1659–1692,

[18]
Richard Cheng, Gábor Orosz, Richard M Murray, and Joel W Burdick.
End-to-end safe reinforcement learning through barrier functions for
safety-critical continuous control tasks.
In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 33, pages 3387–3395, 2019.

[19]
Ho-Bin Choi, Ju-Bong Kim, Youn-Hee Han, Se-Won Oh, and Kwihoon Kim.
Marl-based cooperative multi-agv control in warehouse systems.
IEEE Access, 10:100478–100488, 2022.

[20]
Jaewan Choi, Geonhee Lee, and Chibum Lee.
Reinforcement learning-based dynamic obstacle avoidance and
integration of path planning.
Intelligent Service Robotics, 14(5):663–677, 2021.

[21]
Reinis Cimurs, Jin Han Lee, and Il Hong Suh.
Goal-oriented obstacle avoidance with deep reinforcement learning in
continuous action space.
Electronics, 9(3):411, 2020.

[22]
Cédric Colas, Olivier Sigaud, and Pierre-Yves Oudeyer.
A hitchhiker's guide to statistical comparisons of reinforcement
learning algorithms.
arXiv preprint arXiv:1904.06979, 2019.

[23]
Gal Dalal, Krishnamurthy Dvijotham, Matej Vecerik, Todd Hester, Cosmin
Paduraru, and Yuval Tassa.
Safe exploration in continuous action spaces.
arXiv preprint arXiv:1801.08757, 2018.

[24]
Olivier Delalleau, Maxim Peter, Eloi Alonso, and Adrien Logut.
Discrete and continuous action representation for practical rl in
video games.
arXiv preprint arXiv:1912.11077, 2019.

[25]
Bruce The Deus.
Deus drone, Jan 2022.

[26]
Marina Dorokhova, Yann Martinson, Christophe Ballif, and Nicolas Wyrsch.
Deep reinforcement learning control of electric vehicle charging in
the presence of photovoltaic generation.
Applied Energy, 301:117504, 2021.

[27]
Gabriel Dulac-Arnold, Richard Evans, Hado van Hasselt, Peter Sunehag, Timothy
Lillicrap, Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, and
Ben Coppin.
Deep reinforcement learning in large discrete action spaces.
arXiv preprint arXiv:1512.07679, 2015.

[28]
Zhou Fan, Rui Su, Weinan Zhang, and Yong Yu.
Hybrid actor-critic reinforcement learning in parameterized action
arXiv preprint arXiv:1903.01344, 2019.

[29]
Scott Fujimoto, Herke Hoof, and David Meger.
Addressing function approximation error in actor-critic methods.
In International Conference on Machine Learning, pages
1587–1596. PMLR, 2018.

[30]
Sean Gillies et al.
Shapely: manipulation and analysis of geometric objects, 2007.

[31]
Ian Goodfellow, Yoshua Bengio, and Aaron Courville.
Deep learning.
MIT press, 2016.

[32]
Yang Guan, Yangang Ren, Qi Sun, Shengbo Eben Li, Haitong Ma, Jingliang Duan,
Yifan Dai, and Bo Cheng.
Integrated decision and control: toward interpretable and
computationally efficient driving intelligence.
IEEE Transactions on Cybernetics, 53(2):859–873, 2022.

[33]
Xinde Guo, Zhigang Ren, Zongze Wu, Jialun Lai, Deyu Zeng, and Shengli Xie.
A deep reinforcement learning based approach for agvs path planning.
In 2020 Chinese Automation Congress, pages 6833–6838. IEEE,

[34]
Peter E Hart, Nils J Nilsson, and Bertram Raphael.
A formal basis for the heuristic determination of minimum cost paths.
IEEE Transactions on Systems Science and Cybernetics,
4(2):100–107, 1968.

[35]
Matthew Hausknecht and Peter Stone.
Deep reinforcement learning in parameterized action space.
arXiv preprint arXiv:1511.04143, 2015.

[36]
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup,
and David Meger.
Deep reinforcement learning that matters.
In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 32, 2018.

[37]
Nick Hoernle, Rafael Michael Karampatsis, Vaishak Belle, and Kobi Gal.
Multiplexnet: Towards fully satisfied logical constraints in neural
In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 36, pages 5700–5709, 2022.

[38]
Hao-Lun Hsu, Qiuhua Huang, and Sehoon Ha.
Improving safety in deep reinforcement learning using unsupervised
action planning.
In 2022 International Conference on Robotics and Automation,
pages 5567–5573. IEEE, 2022.

[39]
Jueming Hu, Xuxi Yang, Weichang Wang, Peng Wei, Lei Ying, and Yongming Liu.
Obstacle avoidance for uas in continuous action space using deep
reinforcement learning.
arXiv preprint arXiv:2111.07037, 2021.

[40]
Yuh-Jong Hu and Shang-Jen Lin.
Deep reinforcement learning for optimizing finance portfolio
In 2019 Amity International Conference on Artificial
Intelligence, pages 14–20. IEEE, 2019.

[41]
Zhejie Hu and Tomoyuki Kaneko.
Hierarchical advantage for reinforcement learning in parameterized
action space.
In 2021 IEEE Conference on Games, pages 1–8. IEEE, 2021.

[42]
Shengyi Huang and Santiago Ontañón.
Comparing observation and action representations for deep
reinforcement learning in $\mu $rts.
arXiv preprint arXiv:1910.12134, 2019.

[43]
Shengyi Huang and Santiago Ontañón.
A closer look at invalid action masking in policy gradient
arXiv preprint arXiv:2006.14171, 2020.

[44]
Julian Ibarz, Jie Tan, Chelsea Finn, Mrinal Kalakrishnan, Peter Pastor, and
Sergey Levine.
How to train your robot with deep reinforcement learning: lessons we
have learned.
The International Journal of Robotics Research,
40(4-5):698–721, 2021.

[45]
Anssi Kanervisto, Christian Scheller, and Ville Hautamäki.
Action space shaping in deep reinforcement learning.
In 2020 IEEE Conference on Games, pages 479–486. IEEE, 2020.

[46]
Diederik P Kingma and Jimmy Ba.
Adam: A method for stochastic optimization.
arXiv preprint arXiv:1412.6980, 2014.

[47]
B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A
Al Sallab, Senthil Yogamani, and Patrick Pérez.
Deep reinforcement learning for autonomous driving: A survey.
IEEE Transactions on Intelligent Transportation Systems, 2021.

[48]
Niklas Kochdumper, Hanna Krasowski, Xiao Wang, Stanley Bak, and Matthias
Provably safe reinforcement learning via action projection using
reachability analysis and polynomial zonotopes.
arXiv preprint arXiv:2210.10691, 2022.

[49]
Hanna Krasowski, Jakob Thumm, Marlon Müller, Xiao Wang, and Matthias
Provably safe reinforcement learning: A theoretical and experimental
arXiv preprint arXiv:2205.06750, 2022.

[50]
Andreas Kuhnle, Jan-Philipp Kaiser, Felix Theiß, Nicole Stricker, and
Gisela Lanza.
Designing an adaptive production control system using reinforcement
Journal of Intelligent Manufacturing, 32(3):855–876, 2021.

[51]
Xiaoyun Lei, Zhian Zhang, and Peifang Dong.
Dynamic path planning of unknown environment based on deep
reinforcement learning.
Journal of Robotics, 2018, 2018.

[52]
Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Moritz Hardt,
Benjamin Recht, and Ameet Talwalkar.
Massively parallel hyperparameter tuning.
arXiv preprint arXiv:1810.05934, 5, 2018.

[53]
Yijing Li.
An rrt-based path planning strategy in a dynamic environment.
In 2021 7th International Conference on Automation, Robotics and
Applications, pages 1–5. IEEE, 2021.

[54]
Yuxi Li.
Deep reinforcement learning: An overview.
arXiv preprint arXiv:1701.07274, 2017.

[55]
Zhaojian Li, Uroš Kalabić, and Tianshu Chu.
Safe reinforcement learning: Learning with supervision using a
constraint-admissible set.
In 2018 Annual American Control Conference, pages 6390–6395.
IEEE, 2018.

[56]
Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken
Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica.
Rllib: Abstractions for distributed reinforcement learning.
In International Conference on Machine Learning, pages
3053–3062. PMLR, 2018.

[57]
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez,
and Ion Stoica.
Tune: A research platform for distributed model selection and
arXiv preprint arXiv:1807.05118, 2018.

[58]
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom
Erez, Yuval Tassa, David Silver, and Daan Wierstra.
Continuous control with deep reinforcement learning.
arXiv preprint arXiv:1509.02971, 2015.

[59]
Yongshuai Liu, Jiaxin Ding, and Xin Liu.
A constrained reinforcement learning based approach for network
In 2020 IEEE 28th International Conference on Network
Protocols, pages 1–6. IEEE, 2020.

[60]
Yongshuai Liu, Avishai Halev, and Xin Liu.
Policy learning with constraints in model-free reinforcement
learning: A survey.
In The 30th International Joint Conference on Artificial
Intelligence, 2021.

[61]
Enrico Marchesini and Alessandro Farinelli.
Discrete deep reinforcement learning for mapless navigation.
In 2020 IEEE International Conference on Robotics and
Automation, pages 10688–10694. IEEE, 2020.

[62]
Daniel Marta, Christian Pek, Gaspar I Melsión, Jana Tumova, and Iolanda
Human-feedback shield synthesis for perceived safety in deep
reinforcement learning.
IEEE Robotics and Automation Letters, 7(1):406–413, 2021.

[63]
Eivind Meyer, Haakon Robinson, Adil Rasheed, and Omer San.
Taming an autonomous surface vehicle for path following and collision
avoidance using deep reinforcement learning.
IEEE Access, 8:41466–41481, 2020.

[64]
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy
Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu.
Asynchronous methods for deep reinforcement learning.
In International Conference on Machine Learning, pages
1928–1937. PMLR, 2016.

[65]
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis
Antonoglou, Daan Wierstra, and Martin Riedmiller.
Playing atari with deep reinforcement learning.
arXiv preprint arXiv:1312.5602, 2013.

[66]
Fatih Ozhamaratli and Paolo Barucca.
Deep reinforcement learning for optimal investment and saving
strategy selection in heterogeneous profiles: Intelligent agents working
towards retirement.
arXiv preprint arXiv:2206.05835, 2022.

[67]
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary
DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer.
Automatic differentiation in pytorch.

[68]
Santiago Paternain, Luiz Chamon, Miguel Calvo-Fullana, and Alejandro Ribeiro.
Constrained reinforcement learning has zero duality gap.
Advances in Neural Information Processing Systems, 32, 2019.

[69]
Michael Pernpeintner, Christian Bartelt, and Heiner Stuckenschmidt.
Governing black-box agents in competitive multi-agent systems.
In European Conference on Multi-Agent Systems, pages 19–36.
Springer, 2021.

[70]
Tu-Hoa Pham, Giovanni De Magistris, and Ryuki Tachibana.
Optlayer-practical constrained optimization for deep reinforcement
learning in the real world.
In 2018 IEEE International Conference on Robotics and
Automation, pages 6236–6243. IEEE, 2018.

[71]
Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi
Kanervisto, and Noah Dormann.
Stable baselines3, 2019.

[72]
Yongyi Ran, Han Hu, Yonggang Wen, and Xin Zhou.
Optimizing energy efficiency for data center via parameterized deep
reinforcement learning.
IEEE Transactions on Services Computing, 2022.

[73]
Sathya N Ravi, Tuan Dinh, Vishnu Suresh Lokhande, and Vikas Singh.
Explicitly imposing constraints in deep networks via conditional
gradients gives improved generalization and faster convergence.
In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 33, pages 4772–4779, 2019.

[74]
Jing Ren and Xishi Huang.
Potential fields guided deep reinforcement learning for optimal path
planning in a warehouse.
In 2021 IEEE 7th International Conference on Control Science and
Systems Engineering, pages 257–261. IEEE, 2021.

[75]
Herbert Robbins and Sutton Monro.
A stochastic approximation method.
The Annals of Mathematical Statistics, pages 400–407, 1951.

[76]
Jeremy Roghair, Amir Niaraki, Kyungtae Ko, and Ali Jannesari.
A vision based deep reinforcement learning algorithm for uav obstacle
In Proceedings of SAI Intelligent Systems Conference, pages
115–128. Springer, 2022.

[77]
Shah Sanket, Arunesh Sinha, Pradeep Varakantham, Perrault Andrew, and Milind
Solving online threat screening games using constrained action space
reinforcement learning.
In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 34, pages 2226–2235, 2020.

[78]
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver.
Prioritized experience replay.
arXiv preprint arXiv:1511.05952, 2015.

[79]
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter
High-dimensional continuous control using generalized advantage
arXiv preprint arXiv:1506.02438, 2015.

[80]
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.
Proximal policy optimization algorithms.
arXiv preprint arXiv:1707.06347, 2017.

[81]
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George
Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda
Panneershelvam, Marc Lanctot, et al.
Mastering the game of go with deep neural networks and tree search.
Nature, 529(7587):484–489, 2016.

[82]
David Silver, Guy Lever, Nicolas Heess, Thomas Degris, Daan Wierstra, and
Martin Riedmiller.
Deterministic policy gradient algorithms.
In International Conference on Machine Learning, pages
387–395. PMLR, 2014.

[83]
Sean R Sinclair, Siddhartha Banerjee, and Christina Lee Yu.
Adaptive discretization for episodic reinforcement learning in metric
Proceedings of the ACM on Measurement and Analysis of Computing
Systems, 3(3):1–44, 2019.

[84]
Anthony Stentz.
Optimal and efficient path planning for partially known environments.
In Intelligent Unmanned Ground Vehicles, pages 203–220.
Springer, 1997.

[85]
Richard S Sutton and Andrew G Barto.
Reinforcement learning: An introduction.
MIT press, 2018.

[86]
Daniel Tabas and Baosen Zhang.
Computationally efficient safe reinforcement learning for power
In 2022 American Control Conference, pages 3303–3310. IEEE,

[87]
Lei Tai, Giuseppe Paolo, and Ming Liu.
Virtual-to-real deep reinforcement learning: Continuous control of
mobile robots for mapless navigation.
In 2017 IEEE/RSJ International Conference on Intelligent Robots
and Systems, pages 31–36. IEEE, 2017.

[88]
Yunhao Tang and Shipra Agrawal.
Discretizing continuous action space for on-policy optimization.
In Proceedings of the AAAI Conference on Artificial
Intelligence, volume 34, pages 5981–5988, 2020.

[89]
Andrew Taylor, Andrew Singletary, Yisong Yue, and Aaron Ames.
Learning for safety-critical control with control barrier functions.
In Learning for Dynamics and Control, pages 708–717. PMLR,

[90]
Akshay Thirugnanam, Jun Zeng, and Koushil Sreenath.
Safety-critical control and planning for obstacle avoidance between
polytopes with control barrier functions.
In IEEE International Conference on Robotics and Automation,

[91]
Chao Wang, Jian Wang, Yuan Shen, and Xudong Zhang.
Autonomous navigation of uavs in large-scale complex environments: A
deep reinforcement learning approach.
IEEE Transactions on Vehicular Technology, 68(3):2124–2136,

[92]
Yuanda Wang, Haibo He, and Changyin Sun.
Learning to navigate through complex dynamic environment with modular
deep reinforcement learning.
IEEE Transactions on Games, 10(4):400–412, 2018.

[93]
Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Marc Lanctot, and Nando
Dueling network architectures for deep reinforcement learning.
In International Conference on Machine Learning, pages
1995–2003. PMLR, 2016.

[94]
Ermo Wei, Drew Wicke, and Sean Luke.
Hierarchical approaches for reinforcement learning in parameterized
action space.
arXiv preprint arXiv:1810.09656, 2018.

[95]
Bernard L Welch.
The generalization of ‘student's’problem when several different
population varlances are involved.
Biometrika, 34(1-2):28–35, 1947.

[96]
Xing Wu, Haolei Chen, Changgu Chen, Mingyu Zhong, Shaorong Xie, Yike Guo, and
Hamido Fujita.
The autonomous navigation and obstacle avoidance for usvs with anoa
deep reinforcement learning method.
Knowledge-Based Systems, 196:105201, 2020.

[97]
Linhai Xie, Sen Wang, Andrew Markham, and Niki Trigoni.
Towards monocular vision based obstacle avoidance through deep
reinforcement learning.
arXiv preprint arXiv:1706.09829, 2017.

[98]
Jiechao Xiong, Qing Wang, Zhuoran Yang, Peng Sun, Lei Han, Yang Zheng, Haobo
Fu, Tong Zhang, Ji Liu, and Han Liu.
Parametrized deep q-networks learning: Reinforcement learning with
discrete-continuous hybrid action space.
arXiv preprint arXiv:1810.06394, 2018.

[99]
Jingyi Xu, Zilu Zhang, Tal Friedman, Yitao Liang, and Guy Broeck.
A semantic loss function for deep learning with symbolic knowledge.
In International Conference on Machine Learning, pages
5502–5511. PMLR, 2018.

[100]
Tengyu Xu, Yingbin Liang, and Guanghui Lan.
Crpo: A new approach for safe reinforcement learning with convergence
In International Conference on Machine Learning, pages
11480–11491. PMLR, 2021.

[101]
Long Yang, Jiaming Ji, Juntao Dai, Yu Zhang, Pengfei Li, and Gang Pan.
Cup: A conservative update policy algorithm for safe reinforcement
arXiv preprint arXiv:2202.07565, 2022.

[102]
Qingfeng Yao, Zeyu Zheng, Liang Qi, Haitao Yuan, Xiwang Guo, Ming Zhao, Zhi
Liu, and Tianji Yang.
Path planning method with improved artificial potential field—a
reinforcement learning perspective.
IEEE Access, 8:135513–135523, 2020.

[103]
Haeun Yoo, Victor M Zavala, and Jay H Lee.
A dynamic penalty function approach for constraint-handling in
reinforcement learning.
IFAC-PapersOnLine, 54(3):487–491, 2021.

[104]
Dongjie Yu, Haitong Ma, Shengbo Li, and Jianyu Chen.
Reachability constrained reinforcement learning.
In International Conference on Machine Learning, pages
25636–25655. PMLR, 2022.

[105]
Tom Zahavy, Matan Haroush, Nadav Merlis, Daniel J Mankowitz, and Shie Mannor.
Learn what not to learn: Action elimination with deep reinforcement
Advances in Neural Information Processing Systems, 31, 2018.

[106]
Hongming Zhang and Tianyang Yu.
Taxonomy of Reinforcement Learning Algorithms, pages 125–133.
Springer Singapore, Singapore, 2020.

[107]
Sitong Zhang, Yibing Li, and Qianhui Dong.
Autonomous navigation of uav in multi-obstacle environments based on
a deep reinforcement learning approach.
Applied Soft Computing, 115:108194, 2022.

[108]
Zihao Zhang, Stefan Zohren, and Stephen Roberts.
Deep reinforcement learning for trading.
The Journal of Financial Data Science, 2(2):25–40, 2020.

[109]
Huaidong Zhou, Pengbo Feng, and Wusheng Chou.
A hybrid obstacle avoidance method for mobile robot navigation in
unstructured environment.
Industrial Robot: The International Journal of Robotics Research
and Application, 2022.

[110]
Zhehua Zhou, Ozgur S Oguz, Marion Leibold, and Martin Buss.
Learning a low-dimensional representation of a safe region for safe
reinforcement learning on dynamical systems.
IEEE Transactions on Neural Networks and Learning Systems,

[111]
Kai Zhu and Tao Zhang.
Deep reinforcement learning based mobile robot navigation: A review.
Tsinghua Science and Technology, 26(5):674–691, 2021.

CHAPTER: HYPERPARAMETERS

The following appendices describe the setup for hyperparameter optimization and the final configurations.

§ SEARCH SPACES

We start by presenting the search spaces. Note that continuous parameters are denoted as floating points, discrete ranges as integers, and categorical choices in curly brackets. For example, $[5,50]$ means all integer values from $5$ to $50$.

* PPO:

1cHyperparameter     1cSearch Space
clip parameter                                   $[0.1,0.4]$

value function clip parameter                    $[5.0,15.0]$

value function loss coefficient                  $[0.3,2.0]$

entropy coefficient                              $[0.0,0.2]$

learning rate                                    $[5 \cdot 10^{-8},0.005]$

lambda                                           $[0.8,1.0]$

hidden neurons                                   $\{[128, 128], [256, 128], [256, 256],$

$[512, 256], [256, 512], [256]$

$[512], [128], [64]\}$

mini-batch size                                  $[4,200]$

stochastic gradient descent iterations           $[3,60]$

batch size                                       $[200,6000]$

PPO hyperparameter search spaces

* DQN and PAM: Hyperparamers below the dashed line are PAM-specific.

1cHyperparameter     1cSearch Space
learning rate                                    $[5 \cdot 10^{-8},0.1]$

sigma                                            $[0.0,2.0]$

hidden neurons                                   $\{[128, 128], [256, 128], [256, 256],$

$[512, 256], [256, 512], [256]$

$[512], [128], [64]\}$

initial epsilon                                  $[0.1,0.8]$

final epsilon                                    $[0.005,0.1]$

epsilon reduction time steps                     $[1000,50000]$

replay buffer capacity                           $[1500,70000]$

prioritized replay alpha                         $[0.3,0.7]$

prioritized replay beta                          $[0.2,0.5]$

batch size                                       $[28,400]$
categorical distribution temperature             $[0.0,2.0]$

parameter network hidden neurons                 $\{[128, 128], [256, 128], [256, 256],$

$[512, 256], [256, 512], [256]$

$[512], [128], [64]\}$

parameter network learning rate                  $[5 \cdot 10^{-8},0.1]$

dueling                                          $\{True, False\}$

double q learning                                $\{True, False\}$

DQN and PAM hyperparameter search spaces

* TD3 and MDS-TD3:

1cHyperparameter     1cSearch Space
policy delay                                     $[1, 4]$

target policy noise                              $[0.0,0.5]$

target noise clip factor                         $[0.0,0.5]$

hidden neurons actor                             $\{[128, 128], [256, 128], [256, 256],$

$[512, 256], [256, 512], [256]$

$[512], [128], [64]\}$

hidden neurons critic                            $\{[128, 128], [256, 128], [256, 256],$

$[512, 256], [256, 512], [256]$

$[512], [128], [64]\}$

learning rate actor                              $[5 \cdot 10^{-8}, 0.001]$

learning rate critic                             $[5 \cdot 10^{-8}, 0.001]$

tau                                              $[5 \cdot 10^{-4}, 0.005]$

l2 regularization                                $[10^{-8}, 0.001]$

initial epsilon                                  $[0.05,0.4]$

final epsilon                                    $[0.005,0.1]$

epsilon reduction time steps                     $[1000,30000]$

initial noise scale                              $[0.2,30.0]$

final noise scale                                $[0.01,3.0]$

noise reduction time steps                       $[1000,50000]$

replay buffer capacity                           $[1500,70000]$

prioritized replay alpha                         $[0.3,0.7]$

prioritized replay beta                          $[0.2,0.5]$

target update frequency                          $[1,4]$

batch size                                       $[50,512]$

TD3 and MDS-TD3 hyperparameter search spaces

§ FINAL CONFIGURATIONS

After having defined the search spaces, the following tables describe the final hyperparameter values for each experiment and algorithm. We round each value to two decimal points. The specific setting can be found in the corresponding GitHub repository.

1cHyperparameter     1cWithout Obstacles     1cWith Obstacles
clip parameter                                   $0.38$                    $0.34$

value function clip parameter                    $32.52$                       $7.15$

value function loss coefficient                  $1.36$                       $1.51$

entropy coefficient                              $1.23 \cdot 10^{-2}$          $1.55 \cdot 10^{-3}$

learning rate                                    $1.11 \cdot 10^{-4}$           $1.21 \cdot 10^{-4}$

lambda                                           $0.84$                       $0.92$

hidden neurons                                   $[256, 256]$                  $[256, 256]$

mini-batch size                                  $84$                           $87$

gradient descent iterations           $43$                           $4$

batch size                                       $1998$                        $2112$

PPO configuration

1cHyperparameter     1cWithout Obstacles     1cWith Obstacles
clip parameter                                   $0.30$                    $0.38$

value function clip parameter                    $15.75$                       $5.98$

value function loss coefficient                  $0.91$                       $1.33$

entropy coefficient                              $1.44 \cdot 10^{-3}$          $7.06 \cdot 10^{-3}$

learning rate                                    $5.29 \cdot 10^{-5}$           $5.22 \cdot 10^{-6}$

lambda                                           $0.85$                       $0.87$

hidden neurons                                   $[256, 256]$                  $[256, 256]$

mini-batch size                                  $31$                           $116$

gradient descent iterations           $31$                           $24$

batch size                                       $2022$                        $4868$

PPO with projection configuration

1cHyperparameter     1cWithout Obstacles     1cWith Obstacles
clip parameter                                   $0.40$                    $0.28$

value function clip parameter                    $31.07$                       $13.69$

value function loss coefficient                  $1.43$                       $1.64$

entropy coefficient                              $5.33 \cdot 10^{-3}$          $7.94 \cdot 10^{-4}$

learning rate                                    $8.60 \cdot 10^{-5}$           $1.01 \cdot 10^{-5}$

lambda                                           $0.84$                       $0.83$

hidden neurons                                   $[256, 256]$                  $[256, 256]$

mini-batch size                                  $123$                           $130$

gradient descent iterations           $25$                           $17$

batch size                                       $2653$                        $3643$

PPO with random replacement configuration

1cHyperparameter     1cWithout Obstacles     1cWith Obstacles
clip parameter                                   $0.35$                    $0.29$

value function clip parameter                    $43.88$                       $9.23$

value function loss coefficient                  $1.97$                       $1.33$

entropy coefficient                              $1.09 \cdot 10^{-2}$          $3.05 \cdot 10^{-3}$

learning rate                                    $1.28 \cdot 10^{-4}$           $1.53 \cdot 10^{-4}$

lambda                                           $0.86$                       $0.88$

hidden neurons                                   $[256, 256]$                  $[128]$

mini-batch size                                  $121$                           $83$

gradient descent iterations           $32$                           $18$

batch size                                       $875$                        $3771$

PPO with continuous masking configuration

1cHyperparameter     1cWithout Obstacles     1cWith Obstacles
clip parameter                                   $0.35$                    $0.32$

value function clip parameter                    $43.88$                       $9.5$

value function loss coefficient                  $1.97$                       $1.10$

entropy coefficient                              $1.09 \cdot 10^{-2}$          $3.41$

learning rate                                    $1.28 \cdot 10^{-4}$           $1.59 \cdot 10^{-4}$

lambda                                           $0.86$                       $0.89$

hidden neurons                                   $[256, 256]$                  $[128,128]$

mini-batch size                                  $121$                           $107$

gradient descent iterations           $32$                           $8$

batch size                                       $875$                        $3864$

PPO with discrete masking configuration

1cHyperparameter     1cWithout Obstacles     1cWith Obstacles
policy delay                                     $2$                                  $1$

target policy noise                              $0.38$                                 $0.20$

target noise clip factor                         $0.28$                                 $0.19$

hidden neurons actor                             $[256]$              $[256]$

hidden neurons critic                            $[512]$              $[512]$

learning rate actor                              $6.30 \cdot 10^{-5}$                 $6.70 \cdot 10^{-5}$

learning rate critic                             $8.49 \cdot 10^{-4}$                 $2.34 \cdot 10^{-4}$

tau                                              $2.13 \cdot 10^{-3}$                 $4.38 \cdot 10^{-3}$

l2 regularization                                $5.99 \cdot 10^{-5}$                          $5.59 \cdot 10^{-4}$

initial noise scale                              $16.25$                                $9.59$

final noise scale                                $0.56$                               $0.40$

noise reduction time steps                       $37184$                             $3073$

replay buffer capacity                           $21836$                             $66258$

prioritized replay alpha                         $0.67$                                $0.55$

prioritized replay beta                          $0.47$                                $0.37$

target update frequency                          $2$                                    $1$

batch size                                       $420$                                $438$

TD3 configuration

1cHyperparameter     1cWithout Obstacles     1cWith Obstacles
policy delay                                     $1$                                  $1$

target policy noise                              $0.09$                                 $0.40$

target noise clip factor                         $0.32$                                 $0.25$

hidden neurons actor                             $[256, 512]$              $[512, 256]$

hidden neurons critic                            $[128, 128]$              $[512]$

learning rate actor                              $3.54 \cdot 10^{-6}$                 $2.59 \cdot 10^{-6}$

learning rate critic                             $4.39 \cdot 10^{-4}$                 $9.85 \cdot 10^{-5}$

tau                                              $7.31 \cdot 10^{-4}$                 $1.52 \cdot 10^{-3}$

l2 regularization                                $4.38 \cdot 10^{-4}$                          $8.31 \cdot 10^{-4}$

initial noise scale                              $25.24$                                $15.89$

final noise scale                                $1.30$                               $1.67$

noise reduction time steps                       $36192$                             $7248$

replay buffer capacity                           $9337$                             $29410$

prioritized replay alpha                         $0.31$                                $0.60$

prioritized replay beta                          $0.45$                                $0.25$

target update frequency                          $3$                                    $1$

batch size                                       $473$                                $318$

TD3 with projection configuration

1cHyperparameter     1cWithout Obstacles     1cWith Obstacles
policy delay                                     $3$                                  $1$

target policy noise                              $0.27$                                 $0.39$

target noise clip factor                         $0.28$                                 $0.30$

hidden neurons actor                             $[256, 256]$              $[128, 128]$

hidden neurons critic                            $[256]$              $[128]$

learning rate actor                              $8.18 \cdot 10^{-6}$                 $6.69 \cdot 10^{-5}$

learning rate critic                             $6.46 \cdot 10^{-4}$                 $8.89 \cdot 10^{-4}$

tau                                              $9.10 \cdot 10^{-4}$                 $1.91 \cdot 10^{-3}$

l2 regularization                                $1.81 \cdot 10^{-5}$                          $5.83 \cdot 10^{-4}$

initial noise scale                              $13.29$                                $27.32$

final noise scale                                $0.82$                               $2.00$

noise reduction time steps                       $35129$                             $37168$

replay buffer capacity                           $56739$                             $43248$

prioritized replay alpha                         $0.49$                                $0.37$

prioritized replay beta                          $0.31$                                $0.42$

target update frequency                          $1$                                    $2$

batch size                                       $499$                                $478$

TD3 with continuous masking configuration

1cHyperparameter     1cWithout Obstacles     1cWith Obstacles
policy delay                                     $1$                                  $1$

target policy noise                              $0.08$                                 $0.21$

target noise clip factor                         $0.36$                                 $0.26$

hidden neurons actor                             $[256]$              $[256, 512]$

hidden neurons critic                            $[256, 512]$              $[512]$

learning rate actor                              $3.31 \cdot 10^{-5}$                 $1.03 \cdot 10^{-5}$

learning rate critic                             $2.15 \cdot 10^{-4}$                 $6.81 \cdot 10^{-4}$

tau                                              $3.19 \cdot 10^{-4}$                 $2.69 \cdot 10^{-3}$

l2 regularization                                $7.43 \cdot 10^{-4}$                          $2.64 \cdot 10^{-4}$

initial noise scale                              $5.47$                                $17.26$

final noise scale                                $2.40$                               $1.70$

noise reduction time steps                       $17759$                             $16524$

replay buffer capacity                           $50433$                             $32792$

prioritized replay alpha                         $0.46$                                $0.34$

prioritized replay beta                          $0.42$                                $0.37$

target update frequency                          $2$                                    $2$

batch size                                       $384$                                $496$

TD3 with continuous masking configuration

1cHyperparameter     1cWithout Obstacles     1cWith Obstacles
learning rate                                    $5.67 \cdot 10^{-4}$            $2.39 \cdot 10^{-4}$

sigma                                            $0.88$                          $1.76$

hidden neurons                                   $[256, 256]$                  $[256, 256]$

categorical distribution temperature             $1.0$                            $1.00$

initial epsilon                                  $0.8$                            $0.99$

final epsilon                                    $0.01$                         $0.05$

epsilon reduction time steps                     $50000$                       $14998$

replay buffer capacity                           $50000$                       $49766$

prioritized replay alpha                         $0.60$                          $0.62$

prioritized replay beta                          $0.40$                          $0.42$

batch size                                       $399$                           $211$

DQN with discrete masking configuration

1cHyperparameter     1cWithout Obstacles     1cWith Obstacles
learning rate                                    $9.46 \cdot 10^{-4}$            $3.45 \cdot 10^{-5}$

sigma                                            $1.81$                          $0.91$

hidden neurons                                   $[256]$                  $[64]$

categorical distribution temperature             $1.51$                            $0.57$

initial epsilon                                  $0.43$                            $0.65$

final epsilon                                    $6.04 \cdot 10^{-3}$                         $0.01$

epsilon reduction time steps                     $27070$                       $7157$

replay buffer capacity                           $2398$                       $19299$

prioritized replay alpha                         $0.53$                          $0.55$

prioritized replay beta                          $0.36$                          $0.30$

batch size                                       $237$                           $148$

parameter network hidden neurons           $[256]$          $[512, 56]$

parameter network learning rate                  $8.07 \cdot 10^{-4}$         $1.36 \cdot 10^{-5}$

dueling                                          $True$     $True$

double q learning                                $True$      $True$

PAM configuration

1cHyperparameter     1cWithout Obstacles     1cWith Obstacles
policy delay                                     $2$                                  $1$

target policy noise                              $0.05$                                 $0.32$

target noise clip factor                         $0.21$                                 $0.24$

hidden neurons actor                             $[256, 128]$              $[128, 128]$

hidden neurons critic                            $[256]$              $[512, 256]$

learning rate actor                              $2.70 \cdot 10^{-5}$                 $5.29 \cdot 10^{-5}$

learning rate critic                             $8.98 \cdot 10^{-4}$                 $6.47 \cdot 10^{-4}$

tau                                              $3.19 \cdot 10^{-4}$                 $2879 \cdot 10^{-3}$