# A Review of Reward Functions for Reinforcement Learning in the context of
Autonomous Driving

Ahmed Abouelazm1*, Jonas Michel1,2*, and J. Marius Zöllner1,2 1Authors are
with the FZI Research Center for Information Technology, Germany
<EMAIL_ADDRESS>are with the Karlsruhe Institute of Technology,
Germany* These authors contributed equally

###### Abstract

Reinforcement learning has emerged as an important approach for autonomous
driving. A reward function is used in reinforcement learning to establish the
learned skill objectives and guide the agent toward the optimal policy. Since
autonomous driving is a complex domain with partly conflicting objectives with
varying degrees of priority, developing a suitable reward function represents
a fundamental challenge. This paper aims to highlight the gap in such function
design by assessing different proposed formulations in the literature and
dividing individual objectives into Safety, Comfort, Progress, and Traffic
Rules compliance categories. Additionally, the limitations of the reviewed
reward functions are discussed, such as objectives aggregation and
indifference to driving context. Furthermore, the reward categories are
frequently inadequately formulated and lack standardization. This paper
concludes by proposing future research that potentially addresses the observed
shortcomings in rewards, including a reward validation framework and
structured rewards that are context-aware and able to resolve conflicts.

###### Index Terms:

autonomous driving, reinforcement learning, reward function, motion planning

##  I Introduction

Since human error accounts for 94% of traffic accidents[1, 2, 3], autonomous
vehicles are being pursued as an efficient and safe replacement for manual-
driven vehicles to reduce the high number of injuries and traffic
accidents[4]. In addition to increasing road safety, autonomous vehicles have
the potential to improve two major challenges in road transportation, namely,
the efficiency of road infrastructure as well as the efficiency of fuel
consumption and associated emissions[5].

Autonomous driving software architecture can be divided into modular and End-
to-End (E2E) learning approaches[6]. Modular approaches divide the complex
task of driving into multiple subtasks such as perception, localization,
planning, and control to compute vehicle actions from sensor data[6]. The
aforementioned approaches have high dependencies between different modules and
suffer from error propagation through individual modules [7]. These drawbacks
have led the research community to pay more attention to E2E approaches.
Compared to modular approaches, E2E approaches are more sustainable and have
fewer components, as they learn a driving policy directly from input sensor
data [7] without any intermediate explicit tasks.

Reinforcement learning (RL) has been one of the dominant approaches in E2E
driving. Since RL is a learning framework based on the process of repeated
interaction between an agent and its environment in contrast to data-focused
approaches such as supervised learning[8]. The main goal of an RL agent is to
maximize the cumulative reward, which is the discounted total of the rewards
received for each time step[9]. The reward is a singular value based on the
aggregation of a set of objectives, namely the reward function, based on the
action executed by the agent and the updated environment state[9]. To account
for the complexity and size of the state and action spaces, Deep Reinforcement
Learning (DRL) has been introduced in state-of-the-art approaches using a deep
neural network[10].

RL has recently been applied in autonomous driving at different motion
planning levels. In behavior planning, RL is utilized to learn a policy of
high-level decisions based on the environment state [11]. As demonstrated in
[12, 13, 14, 15], turning decisions and lane changes are the most common
behavior planning applications of reinforcement learning. Another planning
level where RL has been applied is trajectory planning in both Cartesian and
Frenet coordinate systems [16, 17, 18, 19, 20]. Finally, RL can also be
formulated to directly compute low-level control commands of the vehicle such
as steering, acceleration, and deceleration as illustrated in [21, 22, 23].

The objective of this research is to bring attention to the diverse challenges
associated with designing a suitable reward system for autonomous driving and
conduct a comprehensive analysis of the rewards employed in state-of-the-art
RL approaches. Individual objectives in the recently proposed formulations are
analyzed in depth and divided into different categories. Additionally, a
comprehensive discussion on the strengths and limitations of the reviewed
reward functions is presented, with the aim of uncovering potential areas for
refinement. Lastly, this research puts forward suggestions for enhancing the
structure of the reward function, with the overarching goal of improving the
safety and efficiency of RL approaches and addressing the need for validation
frameworks for such systems.

##  II Challenges of reward design in the autonomous driving domain

A pivotal aspect of autonomous driving is the conversion of the intricate
multi-objective nature of autonomous driving into a performance metric, as
discussed by [24]. State-of-the-art approaches employ objective functions or
reward functions to encode competing objectives like safety and efficiency.
Although this paper focuses on reward functions due to the prevalence of
reinforcement learning in recent driving approaches, it is crucial to
recognize that the limitations highlighted in this review extend to the design
of objective functions within modular architectures as well.

The reward function, a fundamental element in reinforcement learning, serves
the purpose of evaluating a specific action within the current environment
state. It is therefore necessary for the reward function to facilitate a
rational decision among multiple actions by representing the critical
performance indicators of the task [25]. Inadequately formulated reward
functions, particularly in complex environments, fail to accurately reflect
the agent’s objectives and may result in policies that are suboptimal in
relation to the agent’s actual task [26].

Insufficient emphasis is placed on the design phase of reward functions in
current autonomous driving approaches, resulting in notable limitations and
significant opportunities for enhancement in the overall design of reward
functions [24]. The process of designing reward functions for autonomous
driving is confronted with numerous challenges. Autonomous driving, being a
multi-objective problem, necessitates a reward function capable of not only
encompassing diverse objectives but also addressing the complex task of
combining these objectives effectively and resolving their conflicts [24].
Furthermore, autonomous driving is dependent on context such as the region,
weather conditions, and driving environment (urban environment vs. highway)
[24]. Additionally, the absence of robust performance metrics for evaluating
specific reward functions, coupled with delayed rewards, presents further
design difficulties [24]. Ill-defined reward attributes may lead to
undesirable or even safety-compromising driving behaviors, emphasizing the
need for careful consideration during the reward function design phase.

##  III Categories of Objectives in the reward function design

In order to establish comparability and facilitate additional analysis, the
reward functions are broken down into individual components. These components
are then assigned to a predefined set of categories. The selected categories,
safety, progress, comfort, traffic rules conformance, and related to model
performance, were chosen due to their prevalence in the approaches under
investigation and their intrinsic significance as fundamental metrics for
driving performance assessment. Additionally, the analysis delved into the
specific use case, considering constraints, strengths, and weaknesses of
objectives formulation. The suitability of existing industry standards for the
attribute categories of autonomous driving was examined and missing standards
were highlighted. Further, the different categories are discussed with their
various formulations and their strengths and limitations.

Certain publications use reward terms that don’t fit into the categories that
have been chosen. Typically, the purpose of these terms is to improve model
performance, either by implicitly encouraging exploration [27, 28, 29] or
through explicit methods [30]. Given that these terms do not capture aspects
related to autonomous driving objectives, they are not considered for further
analysis.

###  III-A Safety

The primary goal of driving is ensuring safety, which can be assessed from
different perspectives. Safety encompasses hardware elements like brake
performance, software design, and driving behavior that prioritizes avoiding
collisions. This study specifically emphasizes safe and collision-free
driving, as it aligns most closely with the reward function and RL objectives.
A number of standards, including ISO 26262 [31] and SOTIF [32], address other
aspects of safety; however, none of them explicitly address a definition of
safety that can be applied to the development of reward or objective functions
for autonomous driving. This standard, which is lacking in the literature to
date, is essential to defining reward functions that ensure RL agents’ safety.

To tackle the absence of a specific definition, this work delves into key
findings from the literature relevant to formulating such a definition. The
safety objective should promote safe driving and penalize risky behavior, such
as actions that result in collisions. Maintaining a safe distance from other
road users, and driving at the proper speed, are part of safe driving
behavior. Two general approaches can be distinguished in the current
implementations of safety in reward functions. The direct method penalizes
collisions to reduce the likelihood of accidents in the learned agent
behavior. The second, a more situational approach, seeks to enhance hazard
perception by estimating the risk of potential collisions and allocating
rewards based on the level of risk.

The simplest implementation of safety is a conditional function, wherein a
negative reward is applied if a collision occurs; otherwise, the reward is set
to zero, as exemplified in previous works [21, 33, 34, 28, 35, 36, 37, 38, 39,
40, 41, 42]. The mathematical formulation of a safety conditional reward is
illustrated in equation (1), where $x$ is a negative collision penalty that is
manually defined. One limitation of this approach is its failure to consider
the severity of the collision. For example, a minor collision at low speed is
penalized the same as a collision with pedestrians at high speed.

$r_{safety}=\left\\{\begin{matrix}x&\text{, if there is a collision}\\\
0&\text{, otherwise}\end{matrix}\right.$ (1)

Compared to the proposed definition of safety, only the aspect of punishing
unsafe driving behavior is covered. Even if this conditional function is
interpretable, a manually defined penalty $x$ is often difficult to interpret
and depends heavily on the magnitude assigned to other rewards and penalties.
Various levels of severity can be defined in the safety reward to impose
distinct penalties for different behaviors during collisions, enhancing
interpretability. Moreover, the safety reward can be tailored to distinguish
between accidents involving pedestrians, vehicles, and static obstacles A
realization of such a formulation is illustrated in [43] and [28], where
distinctions between different collision severity are made, and lower or
higher penalties are assigned based on the entity of the traffic participant
involved in a collision. Other approaches use collision damage to calculate
the penalty[30, 44]. While these formulations are applicable in simulation,
collision damage is extremely difficult to calculate in real scenarios.

While the aforementioned approaches enhance transparency and interpretability,
a significant challenge arises in accurately attributing collisions to
specific types of traffic participants, severity levels, or collision damage.
A feasible strategy for assessing collision severity involves incorporating
the speed of the autonomous vehicle at the time of the collision into the
reward, as suggested in [45]. The penalty for the collision increases with
higher collision speeds, providing a justifiable basis for certain penalty
values, as discussed in equation (2). Additionally, the speed can be
introduced exponentially in the formula to impose even more severe penalties
for higher speeds [45].

$r_{safety}=-x\times({velocity}^{2}+0.5)$ (2)

All the previously mentioned formulations involve penalizing penalties for
unsafe driving behavior, specifically collisions, but they lack any provision
for rewarding safe driving behavior relative to the assumed risk. In this
regard, trajectories exhibiting near-collision behavior are assessed
identically to trajectories exhibiting safer driving behavior. An extension of
the conditional reward function to incorporate collisions and near collisions
is utilized in a number of publications [27, 46, 47, 48].

Alternatively, the safety objective can be modeled using a continuous and
dense reward rather than a conditional sparse function. In order to integrate
collision risk into the reward function and represent the current situation’s
riskiness with a single continuous value, research efforts, as outlined in
[22, 49, 29], focus on factors such as the distance to the nearest vehicles or
obstacles. Recognizing the dynamic nature of other vehicles, a more fitting
approach than distance-based evaluation involves heuristics such as time-to-
collision [50, 51, 52]. The formulation of TTC-based reward is clarified in
equation (3) where the function $f$ describes the penalty according to the
current value of TTC. $f$ can either be a constant or, for a more realistic
representation, the inverse of TTC. In the latter case, the penalty increases
as TTC decreases [50]. The parameter $t$ is defined as the critical threshold
below which the TTC is considered risky.

$r_{safety}=\left\\{\begin{matrix}f(TTC)&\text{, if }TTC<t\\\ 0&\text{,
otherwise}\end{matrix}\right.$ (3)

Apart from TTC, other metrics like headway [50] and distance to other vehicles
[22] can also be taken into account when estimating the current risk level.
It’s important to highlight that, as indicated by [53], TTC is considered more
suitable than other utilized risk heuristics. This preference arises from the
fact that a small TTC value is distinctive to risky situations demanding an
immediate response, whereas a small headway or distance to other vehicles may
not necessarily signify a critical scenario.

Furthermore, this work suggests a safety reward formulation that has a
continuous dense term that penalizes risky driving behavior and encourages
safer driving on the road, such as TTC or headway, as well as a sparse penalty
for collisions. This penalty can be dynamically assigned based on different
collision severity levels and actor types.

This work recommends a formulation for a safety reward that includes both a
sparse penalty for collisions and a continuous dense term that penalizes risky
driving behavior and promotes safer driving on the road, such as TTC or
headway. Various collision severity levels and actor types can be taken into
account in the collision penalty.

Finally, recently introduced sophisticated mathematical decision models, such
as Nvidia Force Field (SFF) [54] and Responsibility-Sensitive Safety (RSS)
[55], could play a pivotal role in developing a transparent and interpretable
definition for safety. SFF aims to make decisions that minimize the
intersection between the autonomous vehicles and other road users claimed sets
[54], thereby averting critical scenarios. The claimed set represents the
union of trajectories computed via a safety procedure proposed in [54]. On the
other hand, RSS functions as an additional layer on top of the autonomous
vehicle planning module, ensuring the avoidance of critical situations in
longitudinal and lateral directions, as well as during yielding by calculating
the worst-case TTC for these maneuvers. To the best of our knowledge, none of
the state-of-the-art RL approaches incorporate the decision models mentioned
earlier. The sole occurrence of SFF usage is proposed in [56], where a neural
network is trained to predict the claimed sets of actors in Carla simulation
[57].

###  III-B Progress

The RL agent is motivated to advance toward a predetermined goal from its
current location by the progress objective. Efficiency is another name for
this objective. There is no universally standardized definition for progress
within the context of reward functions and autonomous driving tasks. A basic
formulation of this objective involves receiving a delayed reward upon
reaching the goal[35, 40, 58, 41]. The reward function discussed in Paleja et
al. [51] additionally rewards attaining 40% and 60% of total progress.
Additionally, other works simultaneously apply a penalty for each time step
elapsed since the beginning of the episode[27, 47] or penalize a speed of
zero[21, 34], incentivizing the agent to complete the task as fast as
possible.

In order to provide a dense reward, progress is often proxied by the distance
traveled[30, 35, 52, 49, 48] or the velocity [21, 43, 34, 28, 36, 37, 38, 51]
in the current time step. Another approach to modeling progress involves
defining a specific target velocity or distance to be covered within a single
time step. Deviations from these targets are then penalized, with larger
deviations incurring higher penalties. However, existing approaches often use
a fixed desired velocity, typically based on the road speed limit[46, 58, 47,
42, 29], without considering other factors such as traffic density and weather
conditions that may affect velocity. A more suitable formulation could involve
dynamically calculating the target velocity based on all relevant factors,
though this might pose challenges in real-world applications. Moreover, as an
alternative formulation of progress, rewarding acceleration is put forth in
[30, 44].

One major problem with dense formulations of progress mentioned earlier is
that the agent can move in the opposite direction of the goal and still
receive a progress reward.

Two papers [28, 44] determine the reward by calculating the distance to goal
[44] or the Euclidean distance to the goal[28], which seems to solve the
problem at first glance. However, a reward based on the road distance traveled
on the route to the destination is a more appropriate approach, as suggested
in [20], since the path to the agent’s destination is not necessarily a
straight line and is defined by the road topology and traffic rules.

Finally, overtaking is rewarded in [46, 35, 42] to encourage progress in
situations where overtaking a slow vehicle in front could lead to faster
progress at the expense of a minimal safety risk.

Although the reviewed formulations of progress can seem reasonable in
hindsight, they can lead to conflicts and accidents. As illustrated by Knox et
al.[24], such progress formulations are ineffective even in a simple situation
where a static obstacle blocks the agent’s path. Unlike a human driver in this
situation, the agent might choose to crash into the obstacle rather than
remain idle, since the cumulative progress penalty over an extended waiting
period can be greater than the collision penalty. This irrational decision is
a consequence of the flawed formulation and aggregation of the individual
objectives and is discussed further in IV-A.

In conclusion, a significant challenge with the progress objective lies in its
overlap with safety. The moment a vehicle departs from a stationary position,
the associated risk relatively increases. However, to reach the predefined
goal, movement is essential and, therefore, must be incentivized by the reward
function.

###  III-C Comfort

The degree to which passengers find their ride comfortable and agreeable is a
determining factor in the overall success and widespread adoption of
autonomous vehicles, in addition to their technical capabilities such as
safety and progress objectives. The comfort objective has no established
standard applicable to the reward function of autonomous driving. The
prevailing industry standard, as outlined in [59], defines comfort by
evaluating vibrations affecting the spine of passengers. This comfort standard
is predominantly passenger-focused, influenced by variables like exposure
duration, age, and height of the passenger [59].

Applying such a standard to autonomous driving proves impractical, given
scenarios where vehicles operate without passengers or experience frequent
changes in passengers. This dynamic change in passengers makes it difficult to
assess comfort levels during a journey, as conventional methods such as user
studies are not scalable. One of the most appropriate and complete definitions
of comfort in the context of autonomous driving, to the best of our knowledge,
is illustrated in[60]. The right-of-way, acceleration, and the derivative of
acceleration (jerk) in both longitudinal and lateral directions are taken into
account in the suggested formulation. However, it disregards the rate of
change of the steering angle and lateral acceleration in its formulation.

It should be emphasized that while some aspects of the previously mentioned
formulation are considered in the reviewed literature for this work, no
research offered a complete coverage of the formulation. Numerous methods
impose penalties on high acceleration and deceleration [45, 33, 22, 36, 52,
49], while others focus on penalizing the rate of change of acceleration
(jerk) [50, 22, 47]. When assessing comfort, only one paper takes headway into
account [50], and two of the examined publications introduce penalties for
hard braking [51, 41], potentially conflicting with the safety objective of
the reward function.

One aspect that is not among those previously defined in [60] is steering
smoothness, which is used as an evaluation measure for comfort [21, 23, 43,
36]. In detail, steering smoothness is achieved by penalizing high angles of
steering [21, 36] or counter steering [23, 43].

Across several publications, we observed a consistent pattern of leaving out
the comfort attribute entirely from the reward function design [27, 34, 30,
46, 28, 35, 44, 40, 58] which has a negative impact on the agent’s learned
policy.

###  III-D Traffic rules conformance

The goal of the traffic rules conformance objective is to motivate the agent
to comply with various traffic regulations. Detaching traffic regulations from
the concept of safety allows for a more contextualized approach, since the
laws may change depending on the situation. The majority of the reviewed
publications primarily address the basic applications of traffic regulations.
The specific rules currently implemented include rewarding adherence to
staying in the lane[21, 23, 34, 28, 38], imposing penalties for surpassing the
speed limit[21, 36, 37, 52], and for undercutting the required minimum
headway[50]. Additionally, in specific scenarios, regulations involve
rewarding compliance with the right-of-way[33] and remaining in the correct
lane while turning[33].

A prevalent limitation found in the current literature is the absence of a
mechanism for simultaneous compliance with multiple traffic laws and the use
of rule relaxations. For example, none of the publications strictly enforce
speed limits; rather, they penalize speeding based on the degree of deviation
from the speed limit.

##  IV General limitations

This section outlines the main limitations of current reward functions,
highlighting areas that could be addressed in future research to improve
autonomous driving models. Unlike the previous section, the focus here is on
the overall structure of the reward function rather than specific reward
terms. In particular, the focus is on comprehending how the different
objectives are combined to shape the overall reward function and how the
reward’s lack of adaptability to different driving contexts restricts its
generalizability.

###  IV-A Aggregation of attributes

####  IV-A1 Summation

A major limitation is the aggregation of attributes. Most of the reviewed
publications use summation to combine the different reward terms to obtain the
final reward [21, 22, 27, 43, 45, 35, 39, 51, 52, 40, 47, 41, 61].
Nevertheless, this simple formulation doesn’t encode any priority or
distinction between different objectives and fails to handle conflicts in
these objectives. As depicted in equation (4), safety is assigned the same
weight as progress.

$r=r_{safety}+r_{progress}+r_{comfort}+r_{rules}$ (4)

####  IV-A2 Weighted summation

The literature has explored the incorporation of a mechanism to assign
priorities in the final reward calculation, considering the different levels
of objective significance within autonomous driving.

Several works utilize a switching mechanism to condition the RL agent on
particular features of the current environment state and assign different
final reward values to it[34, 37, 38, 58, 48]. While the approach appears
effective in simplified scenarios, the scalability to realistic settings is
hindered by the growing complexity of conditions in a real-world environment.
Additionally, this method design relies on a substantial amount of expert
knowledge. These limitations motivate a need for more adaptive and scalable
approaches that can handle the conflict and trade-offs between objectives.

More sophisticated methodologies have been introduced, incorporating the use
of weights to assign varying degrees of importance to different reward terms.
This approach involves assigning a weight per attribute, as detailed in
equation (5). The introduced weights can be manually tuned, as demonstrated in
the reviewed papers [50, 30, 46, 28, 44, 36, 49, 29, 42]. Additionally,
Inverse Reinforcement Learning (IRL) approaches are proposed either to learn
the complete reward value [62, 63] at a given state or the individual
objectives’ reward contribution [64] or the weights per attribute [65, 66]
from recorded expert driving.

$r=w_{safety}\times r_{safety}+w_{progress}\times r_{progress}\\\
+w_{comfort}\times r_{comfort}+w_{traffic\;rules}\times r_{rules}$ (5)

Manually fine-tuning weights in the complex setting of autonomous driving
poses challenges, given the lack of intuitive guidelines on determining the
appropriate balance between its diverse objectives. Conversely, the
application of IRL demands substantial computational resources and a diverse
dataset to ensure effective generalization, as noted in previous research[33].
Additionally, a notable drawback of weight-based approaches lies in their lack
of adaptability, where optimal weights may vary based on context or the type
of maneuver.

####  IV-A3 Lexicographic ordering

Using a lexicographic order is one way to eliminate the need for explicit
weighting procedures and address some of the associated disadvantages. In a
lexicographic ordering approach, objectives are strictly ordered based on
their importance, and the decision is made by considering each objective
sequentially, without assigning specific weights, as demonstrated in[33]. The
paper introduces a lexicographic deep Q network that incorporates four
prioritized goals, correct lane changes, safety aspects, traffic rule
compliance, and comfort, each associated with assigned thresholds [33].

Although this approach mitigates the challenges associated with traditional
weighting processes, it introduces new ones. The method relies on a strict
ordering of objectives to calculate the reward, and is unable to manage
formulations where multiple objectives hold the same level of importance.
Additionally, the introduction of a threshold for each objective requires
manual tuning. Section V-A introduces Rulebooks as an alternative threshold-
free ordering approach with better trade-off handling.

###  IV-B Use Case Specific Reward design with lack of Context Awareness

The driving context plays a pivotal role in establishing criteria for desired
behavior in autonomous vehicles. This context significantly shapes the design
of the reward function, aiming to mathematically represent this behavior [24].
Typically, the reward function is customized to excel in the specific driving
context, incorporating terms specific to the use case in focus.

While urban driving is the most prevalent use case [21, 43, 45, 33, 28, 44,
36, 39], other scenarios such as merging procedures [22, 42, 41, 49], lane
change maneuvers [48, 47], or even racing track simulations [27, 34] are
occasionally studied. A drawback of use case specific reward functions is
their limited generalization to diverse or unexpected scenarios and limited
versatility in a different driving context or use case.

On the other hand, relying on a single, all-encompassing reward function might
prove insufficient in a dynamic domain such as autonomous driving[24]. A
universal and versatile reward function for autonomous driving can be
established using a hybrid strategy. This entails combining use cases specific
terms with more broadly applicable objectives while maintaining awareness of
the vehicle’s driving context. However, none of the reviewed papers made
modifications to the reward function to incorporate context awareness.

A limitation of the suggested reward formulation is that specific terms are
tailored for each use case, necessitating a transition mechanism between
different use cases during driving. As of now, no such mechanism has been
developed in the field of autonomous driving. An option for handling these
transitions is the use of Reward machines, which can be viewed as an extension
or specialization of finite state machines in the context of RL. Further
details about Reward machines are explored in section V-B.

###  IV-C Economic aspects disregarded

The examined reward functions have predominantly emphasized the objectives
outlined in section III, yet consistently overlook economic aspects. Economic
aspects in driving encompass factors such as fuel efficiency, and cost
optimization. While safety and regulations typically take precedence in social
and ethical priorities, economic aspects should receive additional attention
due to their significant financial and environmental impact.

##  V Proposals for future work

In this section, potential solutions for the identified limitations in current
reward functions are explored, with the intention of advancing the autonomous
driving reward design process. The suggested approaches center on exploring
alternative ways to aggregate individual objectives through Rulebooks and
incorporating driving context into reward functions. Additionally, the section
examines the absence of frameworks for validating and evaluating reward
functions.

###  V-A Rulebooks

When examining the primary issues with existing reward functions, attention is
drawn to challenges linked to the use of weighted reward functions for
different objectives. Proposing an alternative to the current reward function
formulation, the Rulebook emerges as a distinctive approach.

A Rulebook can be defined as a tuple 〈R, $\leq$〉 consisting of a certain
number of rules R and a pre-defined order $\leq$ amongst those rules to
account for different priorities[67]. This tuple can be represented as a
directed graph. The primary benefit lies in the elimination of the need for
manual weight assignment, replaced instead by the establishment of rule
priorities within a Rulebook. Furthermore, the rule priority can be acquired
either entirely or partially from data, as highlighted in the work by Censi et
al. [67].

A driving action or trajectory can be evaluated based on its rule violation
for the Rulebook rules, given their priority[67]. Subsequent studies apply an
iterative technique to search for an action with minimum rule violations [68].
Accordingly, a Rulebook can serve as a foundational framework for developing a
more robust reward function that can handle complex situations where certain
rules must be disregarded or are in conflict with other rules. Rulebooks have
demonstrated successful application in the planning modules of autonomous
driving systems, showcasing effectiveness even when defined based on only a
few critical rules[69, 67].

###  V-B Context and Reward Machines

One drawback of reward functions is their lack of context awareness, as
outlined in Section IV-B. Relying solely on a single reward function without
considering context may prove inadequate. To address this issue and improve
generalizability, an effective approach is to incorporate context as input and
enable learning across various contexts[24].

The utilization of reward machines presents a viable approach for encoding
driving context. A reward machine serves as an extension of finite state
machines within the context of RL. Distinguished by their expressive
capability, reward machines excel in hierarchically decomposing intricate
tasks into discrete subtasks [70]. Each subtask is associated with a unique
reward, and a transition mechanism governs the transitions between these
subtasks. This formulation enhances the adaptability of reward machines to
changes in environmental conditions over time [71].

In the context of autonomous driving, a subtask can be a different driving
context or maneuver type, with the requirement of transition mechanisms to
switch between these subtasks. Despite the drawback of relying on transition
mechanisms and the potential for over-engineering when decomposing driving
tasks into specific contexts or maneuvers, reward machines serve as a starting
point in the development of rewards that are context-aware.

###  V-C A Framework for reward function validation

Given the significant limitations and possible ill-definitions of reward
function design reviewed in this review, we contend that the establishment of
an automatic framework for the validation of reward functions is imperative.
This framework is crucial for guaranteeing the safety and reliability of
reinforcement learning agents developed for autonomous driving scenarios.

Following an extensive examination of the existing literature, no instances of
a comprehensive framework designed for the validation of reward functions were
identified, with the exception of a manual approach discussed in [24]. This
particular approach introduces eight sanity checks intended to reveal
potential issues within a given reward function. However, a notable challenge
associated with this method is the non-trivial evaluation of these sanity
checks. For example, one sanity check involves assessing whether the reward
function unintentionally encourages undesired behavior[24], a task that proves
impractical to manually verify within complex domains like autonomous driving.

In recent publications, several works have been introduced to address
automatic critical scenario generation. These studies focus on either
generating or modifying scenarios to produce adversarial examples, intending
to assess the agent’s behavior in critical situations that can lead to
accidents [72, 73, 74, 75]. From the perspective of this work, critical
scenario generation approaches can serve as an initial step for the design of
automatic validation frameworks of reward functions in autonomous driving.

##  VI Conclusion

In this study, an examination of reward functions applied to state-of-the-art
autonomous driving RL models was conducted. The review process included
breaking down the reward functions into individual terms and assigning them to
a predefined set of categories. This approach enables a comprehensive analysis
both in a broad sense and in terms of specific categories. At the category
analysis level, a prominent challenge observed was the absence of concrete
industry standards for these categories, resulting in diverse and typically
ill-defined formulations of these categories.

Furthermore, general limitations of the reward function design have been
discussed. The limitations included the simplistic approach for aggregation of
conflicting attributes, lack of awareness about the context of the driving
task, and limited generalization due to use case specific formulation.

Different aspects of enhancement of reward function are discussed for future
work. The combination of objectives can be addressed through Rulebooks and the
encoding of context can be enhanced through reward machines. Future research
recommendations also address the absence of a framework for the validation of
reward functions.

## ACKNOWLEDGMENT

The research leading to these results is funded by the German Federal Ministry
for Economic Affairs and Climate Action within the project “KI Wissen”
(project number: 19A20020L). The authors would like to thank the consortium
for the successful cooperation.

## References

  * [1] A. S. Mueller, J. B. Cicchino, and D. S. Zuby, “What humanlike errors do autonomous vehicles need to avoid to maximize safety?” _Journal of safety research_ , vol. 75, pp. 310–318, 2020.
  * [2] M. Cunningham and M. A. Regan, “Autonomous vehicles: human factors issues and future research,” in _Proceedings of the 2015 Australasian Road safety conference_ , vol. 14, 2015.
  * [3] S. Singh, “Critical reasons for crashes investigated in the national motor vehicle crash causation survey,” Tech. Rep., 2015.
  * [4] World Health Organization, _Global status report on road safety 2018_. Genève, Switzerland: World Health Organization, Jan. 2019.
  * [5] S. E. Shladover, “Cooperative (rather than autonomous) vehicle-highway automation systems,” _IEEE Intelligent Transportation Systems Magazine_ , vol. 1, no. 1, pp. 10–19, 2009.
  * [6] L. Liu, S. Lu, R. Zhong, B. Wu, Y. Yao, Q. Zhang, and W. Shi, “Computing systems for autonomous driving: State of the art and challenges,” _IEEE Internet of Things Journal_ , vol. 8, no. 8, pp. 6469–6486, 2020.
  * [7] D. Coelho and M. Oliveira, “A review of end-to-end autonomous driving in urban environments,” _IEEE Access_ , vol. 10, pp. 75 296–75 311, 2022.
  * [8] R. S. Sutton and A. G. Barto, “Reinforcement learning: An introduction,” _Robotica_ , vol. 17, no. 2, pp. 229–235, 1999.
  * [9] V. François-Lavet, P. Henderson, R. Islam, M. G. Bellemare, J. Pineau _et al._ , “An introduction to deep reinforcement learning,” _Foundations and Trends® in Machine Learning_ , vol. 11, no. 3-4, pp. 219–354, 2018.
  * [10] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” _arXiv preprint arXiv:1312.5602_ , 2013.
  * [11] A. Sadat, M. Ren, A. Pokrovsky, Y.-C. Lin, E. Yumer, and R. Urtasun, “Jointly learnable behavior and trajectory planning for self-driving vehicles,” in _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2019, pp. 3949–3956.
  * [12] C.-J. Hoel, K. Wolff, and L. Laine, “Automated speed and lane change decision making using deep reinforcement learning,” in _2018 21st International Conference on Intelligent Transportation Systems (ITSC)_. IEEE, 2018, pp. 2148–2155.
  * [13] A. R. Fayjie, S. Hossain, D. Oualid, and D.-J. Lee, “Driverless car: Autonomous driving using deep reinforcement learning in urban environment,” in _2018 15th international conference on ubiquitous robots (ur)_. IEEE, 2018, pp. 896–901.
  * [14] Z. Cao, D. Yang, S. Xu, H. Peng, B. Li, S. Feng, and D. Zhao, “Highway Exiting Planner for Automated Vehicles Using Reinforcement Learning,” _IEEE Transactions on Intelligent Transportation Systems_ , 2021.
  * [15] H. Wang, S. Yuan, M. Guo, C.-Y. Chan, X. Li, and W. Lan, “Tactical driving decisions of unmanned ground vehicles in complex highway environments: A deep reinforcement learning approach,” _Proceedings of the Institution of Mechanical Engineers, Part D: Journal of Automobile Engineering_ , vol. 235, no. 4, pp. 1113–1127, 2021.
  * [16] M. Zhu, X. Wang, and Y. Wang, “Human-like autonomous car-following model with deep reinforcement learning,” _Transportation research part C: emerging technologies_ , vol. 97, pp. 348–368, 2018.
  * [17] Á. Fehér, S. Aradi, F. Hegedüs, T. Bécsi, and P. Gáspár, “Hybrid DDPG approach for vehicle motion planning,” in _International Conference on Informatics in Control, Automation and Robotics (ICINCO)_ , 2019.
  * [18] C. Paxton, V. Raman, G. D. Hager, and M. Kobilarov, “Combining neural networks and tree search for task and motion planning in challenging environments,” in _2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2017, pp. 6059–6066.
  * [19] J. Coad, Z. Qiao, and J. M. Dolan, “Safe trajectory planning using reinforcement learning for self driving,” _arXiv:2011.04702_ , 2020.
  * [20] M. Moghadam, A. Alizadeh, E. Tekin, and G. H. Elkaim, “An end-to-end deep reinforcement learning approach for the long-term short-term planning on the frenet space,” _arXiv preprint arXiv:2011.13098_ , 2020.
  * [21] J. Chen, B. Yuan, and M. Tomizuka, “Model-free deep reinforcement learning for urban autonomous driving,” in _2019 IEEE intelligent transportation systems conference (ITSC)_. IEEE, 2019, pp. 2765–2771.
  * [22] P. Wang, H. Li, and C.-Y. Chan, “Quadratic q-network for learning continuous control for autonomous vehicles,” _arXiv preprint arXiv:1912.00074_ , 2019.
  * [23] P. Wolf, C. Hubschneider, M. Weber, A. Bauer, J. Härtl, F. Dürr, and J. M. Zöllner, “Learning how to drive in a real world simulation with deep q-networks,” in _2017 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2017, pp. 244–250.
  * [24] W. B. Knox, A. Allievi, H. Banzhaf, F. Schmitt, and P. Stone, “Reward (mis) design for autonomous driving,” _Artificial Intelligence_ , vol. 316, p. 103829, 2023.
  * [25] J. Eschmann, “Reward function design in reinforcement learning,” _Reinforcement Learning Algorithms: Analysis and Applications_ , pp. 25–33, 2021.
  * [26] R. T. Icarte, T. Q. Klassen, R. Valenzano, and S. A. McIlraith, “Reward machines: Exploiting reward function structure in reinforcement learning,” _Journal of Artificial Intelligence Research_ , vol. 73, pp. 173–208, 2022.
  * [27] L.-C. Wu, Z. Zhang, S. Haesaert, Z. Ma, and Z. Sun, “Risk-aware reward shaping of reinforcement learning agents for autonomous driving,” _arXiv preprint arXiv:2306.03220_ , 2023.
  * [28] A. Sharif and D. Marijan, “Evaluating the robustness of deep reinforcement learning for autonomous policies in a multi-agent urban driving environment,” in _2022 IEEE 22nd International Conference on Software Quality, Reliability and Security (QRS)_. IEEE, 2022, pp. 785–796.
  * [29] J. Wu, Z. Huang, and C. Lv, “Uncertainty-aware model-based reinforcement learning: Methodology and application in autonomous driving,” _IEEE Transactions on Intelligent Vehicles_ , vol. 8, no. 1, pp. 194–203, 2022.
  * [30] P. Palanisamy, “Multi-agent connected autonomous driving using deep reinforcement learning,” in _2020 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2020, pp. 1–7.
  * [31] ISO Central Secretary, “Road vehicles - functional safety - part 1: Vocabulary,” International Organization for Standardization, Geneva, CH, Standard ISO/PAS 21448:2019, 2018. [Online]. Available: https://www.iso.org/standard/62711.html
  * [32] ——, “Road vehicles - safety of the intended functionality (sotif),” International Organization for Standardization, Geneva, CH, Standard ISO/PAS 21448:2019, 2019. [Online]. Available: https://www.iso.org/standard/62711.html
  * [33] C. Li and K. Czarnecki, “Urban driving with multi-objective deep reinforcement learning,” _arXiv preprint arXiv:1811.08586_ , 2018.
  * [34] A. Yu, R. Palefsky-Smith, and R. Bedi, “Deep reinforcement learning for simulated autonomous vehicle control,” _Course Project Reports: Winter_ , vol. 2016, 2016.
  * [35] L. Wang, J. Liu, H. Shao, W. Wang, R. Chen, Y. Liu, and S. L. Waslander, “Efficient reinforcement learning for autonomous driving with parameterized skills and priors,” _arXiv preprint arXiv:2305.04412_ , 2023.
  * [36] J. Chen, S. E. Li, and M. Tomizuka, “Interpretable end-to-end urban autonomous driving with latent deep reinforcement learning,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 6, pp. 5068–5078, 2021.
  * [37] L. Anzalone, P. Barra, S. Barra, A. Castiglione, and M. Nappi, “An end-to-end curriculum learning approach for autonomous driving scenarios,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 23, no. 10, pp. 19 817–19 826, 2022.
  * [38] X. Pan, Y. You, Z. Wang, and C. Lu, “Virtual to real reinforcement learning for autonomous driving,” _arXiv preprint arXiv:1704.03952_ , 2017.
  * [39] A. R. Fayjie, S. Hossain, D. Oualid, and D.-J. Lee, “Driverless car: Autonomous driving using deep reinforcement learning in urban environment,” in _2018 15th international conference on ubiquitous robots (ur)_. IEEE, 2018, pp. 896–901.
  * [40] D. Isele, R. Rahimi, A. Cosgun, K. Subramanian, and K. Fujimura, “Navigating occluded intersections with autonomous vehicles using deep reinforcement learning,” in _2018 IEEE international conference on robotics and automation (ICRA)_. IEEE, 2018, pp. 2034–2039.
  * [41] Y. Hu, A. Nakhaei, M. Tomizuka, and K. Fujimura, “Interaction-aware decision making with adaptive strategies under merging scenarios,” in _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2019, pp. 151–158.
  * [42] M. Bouton, A. Nakhaei, D. Isele, K. Fujimura, and M. J. Kochenderfer, “Reinforcement learning with iterative reasoning for merging in dense traffic,” in _2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)_. IEEE, 2020, pp. 1–6.
  * [43] X. Liang, T. Wang, L. Yang, and E. Xing, “Cirl: Controllable imitative reinforcement learning for vision-based self-driving,” in _Proceedings of the European conference on computer vision (ECCV)_ , 2018, pp. 584–599.
  * [44] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “Carla: An open urban driving simulator,” in _Conference on robot learning_. PMLR, 2017, pp. 1–16.
  * [45] P. Cai, Y. Luo, A. Saxena, D. Hsu, and W. S. Lee, “Lets-drive: Driving in a crowd by learning from tree search,” _arXiv preprint arXiv:1905.12197_ , 2019.
  * [46] A. Karalakou, D. Troullinos, G. Chalkiadakis, and M. Papageorgiou, “Deep reinforcement learning reward function design for autonomous driving in lane-free traffic,” _Systems_ , vol. 11, no. 3, p. 134, 2023.
  * [47] F. Ye, X. Cheng, P. Wang, C.-Y. Chan, and J. Zhang, “Automated lane change strategy using proximal policy optimization-based deep reinforcement learning,” in _2020 IEEE Intelligent Vehicles Symposium (IV)_. IEEE, 2020, pp. 1746–1752.
  * [48] C.-J. Hoel, K. Wolff, and L. Laine, “Automated speed and lane change decision making using deep reinforcement learning,” in _2018 21st International Conference on Intelligent Transportation Systems (ITSC)_. IEEE, 2018, pp. 2148–2155.
  * [49] P. Wang and C.-Y. Chan, “Formulation of deep reinforcement learning architecture toward autonomous driving for on-ramp merge,” in _2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC)_. IEEE, 2017, pp. 1–6.
  * [50] M. Zhu, Y. Wang, Z. Pu, J. Hu, X. Wang, and R. Ke, “Safe, efficient, and comfortable velocity control based on reinforcement learning for autonomous driving,” _Transportation Research Part C: Emerging Technologies_ , vol. 117, p. 102662, 2020.
  * [51] R. Paleja, L. Chen, Y. Niu, A. Silva, Z. Li, S. Zhang, C. Ritchie, S. Choi, K. C. Chang, H. E. Tseng _et al._ , “Interpretable reinforcement learning for robotics and continuous control,” _arXiv preprint arXiv:2311.10041_ , 2023.
  * [52] Z. Zhang, “Autonomous car driving based on deep reinforcement learning,” in _2022 International Conference on Economics, Smart Finance and Contemporary Trade (ESFCT 2022)_. Atlantis Press, 2022, pp. 835–842.
  * [53] K. Vogel, “A comparison of headway and time to collision as safety indicators,” _Accident analysis & prevention_, vol. 35, no. 3, pp. 427–433, 2003.
  * [54] D. Nistér, H.-L. Lee, J. Ng, and Y. Wang, “The safety force field,” _NVIDIA White Paper_ , 2019.
  * [55] S. Shalev-Shwartz, S. Shammah, and A. Shashua, “On a formal model of safe and scalable self-driving cars,” _arXiv preprint arXiv:1708.06374_ , 2017.
  * [56] H. Suk, T. Kim, H. Park, P. Yadav, J. Lee, and S. Kim, “Rationale-aware autonomous driving policy utilizing safety force field implemented on carla simulator,” _arXiv preprint arXiv:2211.10237_ , 2022.
  * [57] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “Carla: An open urban driving simulator,” in _Conference on robot learning_. PMLR, 2017, pp. 1–16.
  * [58] C.-J. Hoel, K. Driggs-Campbell, K. Wolff, L. Laine, and M. J. Kochenderfer, “Combining planning and deep reinforcement learning in tactical decision making for autonomous driving,” _IEEE transactions on intelligent vehicles_ , vol. 5, no. 2, pp. 294–305, 2019.
  * [59] ISO Central Secretary, “Mechanical vibration and shock — evaluation of human exposure to whole-body vibration — part 1: General requirements,” International Organization for Standardization, Geneva, CH, Standard ISO 2631-1:1997, 1997. [Online]. Available: https://www.iso.org/standard/62711.html
  * [60] H. Bellem, B. Thiel, M. Schrauf, and J. F. Krems, “Comfort in automated driving: An analysis of preferences for different automated driving styles and their dependence on personality traits,” _Transportation research part F: traffic psychology and behaviour_ , vol. 55, pp. 90–100, 2018.
  * [61] Z. Cao, D. Yang, S. Xu, H. Peng, B. Li, S. Feng, and D. Zhao, “Highway exiting planner for automated vehicles using reinforcement learning,” _IEEE Transactions on Intelligent Transportation Systems_ , vol. 22, no. 2, pp. 990–1000, 2020.
  * [62] Z. Zhao, Z. Wang, K. Han, R. Gupta, P. Tiwari, G. Wu, and M. J. Barth, “Personalized car following for autonomous driving with inverse reinforcement learning,” in _2022 International Conference on Robotics and Automation (ICRA)_. IEEE, 2022, pp. 2891–2897.
  * [63] X. Wen, S. Jian, and D. He, “Modeling the effects of autonomous vehicles on human driver car-following behaviors using inverse reinforcement learning,” _IEEE Transactions on Intelligent Transportation Systems_ , 2023.
  * [64] P. Abbeel and A. Y. Ng, “Apprenticeship learning via inverse reinforcement learning,” in _Proceedings of the twenty-first international conference on Machine learning_ , 2004, p. 1.
  * [65] Z. Wu, L. Sun, W. Zhan, C. Yang, and M. Tomizuka, “Efficient sampling-based maximum entropy inverse reinforcement learning with application to autonomous driving,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 4, pp. 5355–5362, 2020.
  * [66] S. Rosbach, V. James, S. Großjohann, S. Homoceanu, and S. Roth, “Driving with style: Inverse reinforcement learning in general-purpose planning for automated driving,” in _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2019, pp. 2658–2665.
  * [67] A. Censi, K. Slutsky, T. Wongpiromsarn, D. Yershov, S. Pendleton, J. Fu, and E. Frazzoli, “Liability, ethics, and culture-aware behavior specification using rulebooks,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 8536–8542.
  * [68] W. Xiao, N. Mehdipour, A. Collin, A. Y. Bin-Nun, E. Frazzoli, R. D. Tebbens, and C. Belta, “Rule-based optimal control for autonomous driving,” in _Proceedings of the ACM/IEEE 12th International Conference on Cyber-Physical Systems_ , 2021, pp. 143–154.
  * [69] B. Helou, A. Dusi, A. Collin, N. Mehdipour, Z. Chen, C. Lizarazo, C. Belta, T. Wongpiromsarn, R. D. Tebbens, and O. Beijbom, “The reasonable crowd: Towards evidence-based and interpretable models of driving behavior,” in _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2021, pp. 6708–6715.
  * [70] R. T. Icarte, T. Q. Klassen, R. Valenzano, and S. A. McIlraith, “Reward machines: Exploiting reward function structure in reinforcement learning,” _Journal of Artificial Intelligence Research_ , vol. 73, pp. 173–208, 2022.
  * [71] A. Camacho, R. T. Icarte, T. Q. Klassen, R. A. Valenzano, and S. A. McIlraith, “Ltl and beyond: Formal languages for reward function specification in reinforcement learning.” in _IJCAI_ , vol. 19, 2019, pp. 6065–6073.
  * [72] A. Wachi, “Failure-scenario maker for rule-based agent using multi-agent adversarial reinforcement learning and its application to autonomous driving,” _arXiv preprint arXiv:1903.10654_ , 2019.
  * [73] D. Rempe, J. Philion, L. J. Guibas, S. Fidler, and O. Litany, “Generating useful accident-prone driving scenarios via a learned traffic prior,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2022, pp. 17 305–17 315.
  * [74] N. Hanselmann, K. Renz, K. Chitta, A. Bhattacharyya, and A. Geiger, “King: Generating safety-critical driving scenarios for robust imitation via kinematics gradients,” in _European Conference on Computer Vision_. Springer, 2022, pp. 335–352.
  * [75] W. Ding, B. Chen, M. Xu, and D. Zhao, “Learning to collide: An adaptive safety-critical scenarios generating method,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 2243–2250.