# Reinforcement Learning Your Way: Agent Characterization through Policy
Regularization

Charl Maree & Christian Omlin
Center for Artificial Intelligence Research
University of Agder
Grimstad, Norway
<EMAIL_ADDRESS>
Second affiliation: Chief Technology Office, Sparebank 1 SR-Bank, Stavanger,
Norway.

###### Abstract

The increased complexity of state-of-the-art reinforcement learning (RL)
algorithms have resulted in an opacity that inhibits explainability and
understanding. This has led to the development of several post-hoc
explainability methods that aim to extract information from learned policies
thus aiding explainability. These methods rely on empirical observations of
the policy and thus aim to generalize a characterization of agents’ behaviour.
In this study, we have instead developed a method to _imbue_ a characteristic
behaviour into agents’ policies through regularization of their objective
functions. Our method guides the agents’ behaviour during learning which
results in an intrinsic characterization; it connects the learning process
with model explanation. We provide a formal argument and empirical evidence
for the viability of our method. In future work, we intend to employ it to
develop agents that optimize individual financial customers’ investment
portfolios based on their spending personalities.

_K_ eywords Explainable AI; Multi-agent systems; Deterministic policy
gradients

##  1 Introduction

Recent advances in reinforcement learning (RL) have increased complexity
which, especially for deep RL, has brought forth challenges related to
explainability [1]. The opacity of state-of-the-art RL algorithms has led to
model developers having a limited understanding of their agents’ policies and
no influence over learned strategies [2]. While concerns surrounding
explainability have been noted for AI in general, it is only more recently
that attempts have been made to explain RL systems [3, 1, 4, 5]. These
attempts have resulted in a wide suite of methods requiring various degrees of
expert knowledge, either about the state-action domain or about the specific
RL algorithm. Further, they typically rely on post-hoc analysis of learned
policies which give only observational assurances of agents’ behaviour. We
instead propose an _intrinsic_ method of regularizing agents’ actions based on
a given prior. While current methods for RL regularization aim to improve
_training performance_ \- e.g., by maximizing the entropy of the action
distribution [6], or by minimising the distance to a prior sub-optimal state-
action distribution [7] \- our aim is the _characterization_ of our agents’
behaviours. We also extend the current regularization techniques to
accommodate multi-agent systems which allows intrinsic characterization of
individual agents. We provide a formal argument for the rationale of our
method and demonstrate its efficacy in a toy problem where agents learn to
navigate to a destination on a grid by performing, e.g., only right turns
(under the premise that right turns are considered safer than left turns [8]).
There are several useful applications beyond this toy problem, such as
financial advice based on personal goals.

##  2 Background and Related Work

###  2.1 Agent Characterization

There have been several approaches to characterizing RL agents, with most - if
not all - employing some form of post-hoc evaluation technique. Some notable
examples are:

Probabilistic argumentation [9]

in which a human expert creates an ‘argumentation graph’ with a set of
arguments and sub-arguments; sub-arguments attack or support main arguments
which attack or support discrete actions. Sub-arguments are labelled as ‘ON’
or ‘OFF’ depending on the state observation for each time-step. Main arguments
are labelled as ‘IN’, ‘OUT’, or ‘UNDECIDED’ in the following RL setting:
_states_ are the union of the argumentation graph and the learned policy,
_actions_ are the probabilistic ‘attitudes’ towards given arguments, and
_rewards_ are based on whether an argument attacks or supports an action. The
learned ‘attitudes’ towards certain arguments are used to characterize agents’
behaviour.

Structural causal modelling (SCM) [10]

learns causal relationships between states and actions through ‘action
influence graphs’ that trace all possible paths from a given initial state to
a set of terminal states, via all possible actions in each intermediate state.
The learned policy then identifies a causal chain as the single path in the
action influence graph that connects the initial state to the relevant
terminal state. The explanation is the vector of rewards along the causal
chain. Counter-explanations are a set of comparative reward vectors along
chains originating from counter-actions in the initial state.
Characterizations are made based on causal and counterfactual reasons for
agents’ choices of action.

Reward decomposition [11, 12]

decomposes the reward into a vector of intelligible reward classes using
expert knowledge. Agent characterization is done by evaluation of the reward
vector for each action post training.

Hierarchical reinforcement learning HRL [13, 14]

divides agents’ tasks into sub-tasks to be learned by different agents. This
simplifies the problem to be solved by each agent, making their behaviour
easier to interpret and thereby making them easier to characterize.

Introspection (interesting elements) [15]

is a statistical post-hoc analysis of the policy. It considers elements such
as the frequency of visits to states, the estimated values of states and
state-action pairs, state-transition probabilities, how much of the state
space is visited, etc. Interesting statistical properties from this analysis
are used to characterize the policy.

###  2.2 Multi-Agent Reinforcement Learning and Policy Regularization

We consider the multi-agent setting of partially observable Markov decision
processes (POMDPs) [16]: for $N$ agents, let $\mathcal{S}$ be a set of states,
$\mathcal{A}_{i}$ a set of actions, and $\mathcal{O}_{i}$ a set of incomplete
state observations where $i\in[1,..,N]$ and
$\mathcal{S}\mapsto\mathcal{O}_{i}$. Agents select actions according to
individual policies $\pi_{\theta_{i}}(\mathcal{O}_{i})\mapsto\mathcal{A}_{i}$
and receive rewards according to individual reward functions
$r_{i}(\mathcal{S},\mathcal{A}_{i})\mapsto\mathbb{R}$, where $\theta_{i}$ is
the set of parameters governing agent $i$’s policy. Finally, agents aim to
maximize their total discounted rewards:

$R_{i}(o,a)=\sum_{t=0}^{T}\gamma r_{i}(o_{t},a_{t})$

where $T$ is the time horizon and $\gamma\in[0,1]$ is a discount factor. For
single-agent systems, the deep deterministic policy gradient algorithm (DDPG)
defines the gradient of the objective $J(\theta)=\mathbb{E}_{s\sim
p^{\mu}}[R(s,a)]$ as [17]:

$\nabla_{\theta}J(\theta)=\mathbb{E}_{s\sim\mathcal{D}}\left[\nabla_{\theta}\mu_{\theta}(a|s)\nabla_{a}Q^{\mu_{\theta}}(s,a)|_{a=\mu_{\theta}(s)}\right]$
(1)

where $p^{\mu}$ is the state distribution, $\mathcal{D}$ is an experience
replay buffer storing observed state transition tuples $(s,a,r,s^{\prime})$,
and $Q^{\mu_{\theta}}(s,a)$ is a state-action value function where actions are
selected according to a policy $\mu_{\theta}(\mathcal{S})\mapsto\mathcal{A}$.
In DDPG, the policy $\mu$ \- also called the _actor_ \- and the value function
$Q$ \- also called the _critic_ \- are modelled by deep neural networks.
Equation (1) is extended to a multi-agent setting; the multi-agent deep
deterministic policy gradient algorithm (MADDPG) learns individual sets of
parameters for each agent $\theta_{i}$ [18]:

$\nabla_{\theta_{i}}J(\theta_{i})=\mathbb{E}_{o,a\sim\mathcal{D}}\left[\nabla_{\theta_{i}}\mu_{\theta_{i}}(a_{i}|o_{i})\nabla_{a_{i}}Q^{\mu_{\theta_{i}}}\left(o_{i},a_{1},...,a_{N}\right)|_{a_{i}=\mu_{\theta_{i}}(o_{i})}\right]$
(2)

where $o_{i}\in\mathcal{O}_{i}$ and the experience replay buffer $\mathcal{D}$
contains tuples $(o_{i},a_{i},r_{i},o^{\prime}_{i}),\ i\in[1,..,N]$.

In this work, we further extend MADDPG by adding a regularization term to the
actors’ objective functions, thus encouraging them to mimic the behaviours
specified by simple predefined prior policies. There have been several
approaches to regularizing RL algorithms, mostly for the purpose of improved
generalization or training performance. In [7], the authors defined an
objective function with a regularization term related to the statistical
difference between the current policy and a predefined prior:

$J(\theta)=\mathbb{E}_{s,a\sim\mathcal{D}}\left[R\left(s,a\right)-\alpha
D_{KL}\left(\pi_{\theta}\left(s,a\right)\|\pi_{0}\left(s,a\right)\right)\right]$
(3)

where $\alpha$ is a hyperparameter scaling the relative contribution of the
regularization term - the Kullback-Leibler (KL) divergence ($D_{KL}$) - and
$\pi_{0}$ is the prior policy which the agent attempts to mimic while
maximising the reward. The KL divergence is a statistical measure of the
difference between two probability distributions, formally:

$D_{KL}(P\|Q)=\sum_{x\in X}P(x)\log{\frac{P(x)}{Q(x)}}$

where $P$ and $Q$ are discrete probability distributions on the same
probability space $X$. The stated objective of KL regularization is increased
learning performance by penalising policies that stray too far from the prior.
The KL divergence is often also called the _relative entropy_ , with KL-
regularized RL being the generalization of entropy-regularized RL [19];
specifically if $\pi_{0}$ is the uniform distribution, Equation 3 reduces to,
up to a constant, the objective function for entropy-regulated RL as described
in [6]:

$J(\theta)=\mathbb{E}_{s,a\sim\mathcal{D}}\left[R(s,a)+\alpha
H\left[\pi_{\theta}(s,a)\right]\right]$ (4)

where $H(\pi)=P(\pi)\log(P(\pi))$ is the statistical entropy of the policy.
The goal of entropy-regularized RL is to encourage exploration by maximising
the policy’s entropy and is used as standard in certain state-of-the-art RL
algorithms, such as soft actor-critic (SAC) [6]. Other notable regularization
methods include _control regularization_ where, during learning, the action of
the actor is weighted with an action from a sub-optimal prior:
$\mu_{k}=\frac{1}{1+\lambda}\mu_{\theta}+\frac{\lambda}{1+\lambda}\mu_{prior}$
and _temporal difference regularization_ which adds a penalty for large
differences in the Q-values of successive states:
$J(\theta,\eta)=\mathbb{E}_{s,a,s^{\prime}\sim\mathcal{D}}\left[R(s,a)-\eta\delta_{Q}(s,a,s^{\prime})\right]$
where $\delta_{Q}(s,a,s^{\prime})=\left[R(s,a)+\gamma
Q(s^{\prime},a^{\prime})-Q(s,a))\right]^{2}$ [20, 21].

While our algorithm is based on regularization of the objective function, it
could be argued that it shares similar goals as those of algorithms based on
_constrained RL_ , namely the intrinsic manipulation of agents’ policies
towards given objectives. One example of constrained RL is [22] which finds a
policy whose long-term measurements lie within a set of constraints by
penalising the reward function with the Euclidean distance between the state
and a given set of restrictions, e.g., an agent’s location relative to
obstacles on a map. Another example is [23] which penalises the value function
with the accumulated cost of a series of actions, thus avoiding certain state-
action situations. However, where constrained RL attempts to _avoid_ certain
conditions - usually through a penalty based on expert knowledge of the state
- regularized RL aims to _promote_ desired behaviours, such as choosing
default actions during training or maximizing exploration by maximising action
entropy. The advantage of our system is that it does not require expert
knowledge of the state-action space to construct constraints; our
regularization term is independent of the state which allows agents to learn
simple behavioural patterns, thus improving the interpretability of their
characterization.

##  3 Methodology

We regulate our agents based on a state-independent prior to maximize rewards
while adhering to simple, predefined rules. In a toy problem, we demonstrate
that agents learn to find a destination on a map by taking only right turns.
Intuitively, we supply the probability distribution of three actions - _left_
, _straight_ , and _right_ \- as a regularization term in the objective
function, meaning the agents aim to mimic this given probability distribution
while maximising rewards. Such an agent can thus be characterized as an agent
that prefers, e.g., right turns over left turns. As opposed to post-hoc
characterization, ours is an intrinsic method that inserts a desirable
characteristic into an agent’s behaviour _during_ learning.

###  3.1 Action Regularization

We modify the objective function in Equation (4) and replace the
regularization term $H[\pi_{\theta}(s,a)]=P(\pi)\log(P(\pi))$ with the mean
squared error of the expected action and a specified prior $\pi_{0}$:

$\displaystyle J(\theta)$
$\displaystyle=\mathbb{E}_{s,a\sim\mathcal{D}}\left[R(s,a)\right]-\lambda L$
(5) $\displaystyle L$
$\displaystyle=\frac{1}{M}\sum_{j=0}^{M}\Big{[}\mathbb{E}_{a\sim\pi_{\theta}}\left[a_{j}\right]-(a_{j}|\pi_{0}(a))\Big{]}^{2}$
(6)

where $\lambda$ is a hyperparameter that scales the relative contribution of
the regularization term $L$, $a_{j}$ is the $j^{th}$ action in a vector of $M$
actions, $\pi_{\theta}$ is the current policy, and $\pi_{0}$ is the specified
prior distribution of actions which the agent aims to mimic while maximising
the reward. Note that $\pi_{0}(a)$ is _independent_ of the state and
$(a_{j}|\pi_{0}(a))$ is therefore constant across all observations and time-
steps. This is an important distinction from previous work, and results in a
prior that is simpler to construct and a characterization that can be
interpreted by non-experts. Since this is a special case of Equation (4), it
follows from the derivation given in [6].

We continue by extending our objective function to support a multi-agent
setting. From Equation (5) and following the derivation in [18], we derive a
multi-agent objective function with $i\in[1,N]$ where $N$ is the number of
agents:

$J(\theta_{i})=\mathbb{E}_{o_{i},a_{i}\sim\mathcal{D}}\left[R_{i}(o_{i},a_{i})\right]-\lambda\frac{1}{M_{i}}\sum_{j=0}^{M_{i}}\left[\mathbb{E}_{a\sim\pi_{\theta_{i}}(o_{i},a)}(a_{j})-(a_{j}|\pi_{0_{i}}(a))\right]^{2}$
(7)

Further, in accordance with the MADDPG algorithm, we model actions and rewards
with actors and critics, respectively [18]:

$\displaystyle\mathbb{E}_{a_{i}\sim\pi_{\theta_{i}}(o_{i},a_{i})}\left[a_{i}\right]$
$\displaystyle=\mu_{\theta_{i}}(o_{i})$ (8)
$\displaystyle\mathbb{E}_{o_{i},a_{i}\sim\mathcal{D}}\left[R_{i}(o_{i},a_{i})\right]$
$\displaystyle=Q_{\theta_{i}}(o_{i},\mu_{\theta_{1}}(o_{1}),\dots,\mu_{\theta_{N}}(o_{N}))$
(9)

Through simple substitution of Equations (8) and (9) into Equation (7) we
formulate our multi-agent regularized objective function:

$\displaystyle J(\theta_{i})$
$\displaystyle=Q_{\theta_{i}}(o_{i},\mu_{\theta_{1}}(o_{1}),\dots,\mu_{\theta_{N}}(o_{N}))-\lambda
L_{i}$ (10) $\displaystyle L_{i}$
$\displaystyle=\frac{1}{M_{i}}\sum_{j=0}^{M_{i}}\Big{[}\mu_{\theta_{i}}(o_{i})_{j}-(a_{j}|\pi_{0_{i}}(a))\Big{]}^{2}$
(11)

Algorithm 1 optimizes the policies of multiple agents given individual
regularization constraints $\pi_{0_{i}}$.

Algorithm 1 Action-regularized MADDPG algorithm

Set the number of agents $N\in\mathbb{N}$

for i in 1, N do $\triangleright$ For each agent

Initialize actor network $\mu_{\theta_{\mu,i}}$ with random parameters
$\theta_{\mu,i}$

Initialize critic network $Q_{\theta_{Q,i}}$ with random parameters
$\theta_{Q,i}$

Initialize target actor network $\mu^{\prime}_{\theta_{\mu^{\prime},i}}$ with
parameters $\theta_{\mu^{\prime},i}\leftarrow\theta_{\mu,i}$

Initialize target critic network $Q^{\prime}_{\theta_{Q^{\prime},i}}$ with
parameters $\theta_{Q^{\prime},i}\leftarrow\theta_{Q,i}$

Set the desired prior action distribution $\pi_{0_{i}}$

Set the number of actions $M_{i}\leftarrow|\pi_{0_{i}}|$

end for

Initialize replay buffer $\mathcal{D}$

Set regularization weight $\lambda$

for e = 1, Episodes do

Initialise random function $F(e)\sim N(0,\sigma_{e})$ for exploration

Reset environment and get the state observation $s_{1}\mapsto o_{[1..N]}$

$t\leftarrow 1$, $Done\leftarrow False$

while not Done do

for i in 1, N do $\triangleright$ For each agent

Select action with exploration
$a_{i,t}\leftarrow\mu_{\theta_{\mu,i}}(o_{i})+F(e)$

end for

Apply compounded action $a_{t}$

Retrieve rewards $r_{[1..N],t}$ and observations $s_{t}^{\prime}\mapsto
o^{\prime}_{[1..N],t}$

Store transition tuple
$\mathcal{T}=\left(o_{t},a_{t},r_{t},o^{\prime}_{t}\right)$ to replay buffer:
$\mathcal{D}\leftarrow\mathcal{D}\cup\mathcal{T}$

$t\leftarrow t+1$

if (end of episode) then

$Done\leftarrow True$

end if

end while

Sample a random batch from the replay buffer $\mathcal{B}\subset\mathcal{D}$

for i in 1, N do $\triangleright$ For each agent

$\widehat{Q_{i}}\leftarrow r_{\mathcal{B},i}+\gamma
Q^{\prime}_{i}\left(o^{\prime}_{\mathcal{B},i},\mu^{\prime}_{1}\left(o^{\prime}_{\mathcal{B},1}\right),\dots,\mu^{\prime}_{N}\left(o^{\prime}_{\mathcal{B},N}\right)\right)$

Update critic parameters $\theta_{Q,i}$ by minimising the loss:
$\mathcal{L}(\theta_{Q,i})=\frac{1}{|\mathcal{B}|}\sum_{\mathcal{B}}\left(Q_{\theta_{Q,i}}\left(o_{\mathcal{B}},a_{\mathcal{B},1},\dots,a_{\mathcal{B},N}\right)-\widehat{Q_{i}}\right)^{2}$

Update the actor parameters $\theta_{\mu,i}$ by minimising the loss:
$\triangleright$ From Equation 10 $\displaystyle\widehat{R_{i}}$
$\displaystyle=\overline{Q_{i}\left(o_{\mathcal{B},i},\mu_{1}(o_{\mathcal{B},1}),\dots,\mu_{N}(o_{\mathcal{B},N})\right)}$
$\displaystyle\mathcal{L}(\theta_{\mu,i})$
$\displaystyle=-\widehat{R_{i}}+\lambda\frac{1}{M_{i}}\sum_{j=1}^{M_{i}}\left[\overline{\mu_{i}(o_{\mathcal{B},i})_{j}}-\left(a_{j}|\pi_{0_{i}}\right)\right]^{2}$

Update target network parameters:  $\displaystyle\theta_{\mu^{\prime},i}$
$\displaystyle\leftarrow\tau\theta_{\mu,i}+\left(1-\tau\right)\theta_{\mu^{\prime},i}$
$\displaystyle\theta_{Q^{\prime},i}$
$\displaystyle\leftarrow\tau\theta_{Q,i}+\left(1-\tau\right)\theta_{Q^{\prime},i}$

end for

end for

##  4 Experiments

###  4.1 Empirical Setup

We created a toy problem in which one or more agents navigate a $6\times 6$
grid through a set of three actions: $\mathcal{A}_{1}=$ _turn left_ ,
$\mathcal{A}_{2}=$ _go straight_ , and $\mathcal{A}_{3}=$ _turn right_. Every
new episode randomly placed a set of destinations in the grid
$D_{i},\,i\in[1,N]$ \- one for each of $N\geq 1$ agents - with initial agent
locations $L_{i,0}=(3,0)$. Rewards were the agents’ Euclidean distances from
their destinations $R_{i,t}=\lVert D_{i}-L_{i,t}\rVert_{2}$ where $L_{i,t}$ is
the location of agent $i$ at time-step $t$. Finally, agents’ observations were
the two-dimensional distances to their destinations:
$\mathcal{O}_{i,t}=D_{i}-L_{i,t}$. An episode was completed when either both
agents had reached their destinations or a maximum of 50 time-steps had
passed.

We ran two sets of experiments, one for a single-agent setting, and one for a
dual-agent setting. We sized all networks in these two settings with two fully
connected feed-forward layers; the single agent networks had 200 nodes in each
layer, while the dual-agent networks had 700 nodes in each layer. Actor
networks had a softmax activation layer, while the critic networks remained
unactivated. Our training runs consisted of 3 000 iterations and we tuned the
hyperparameters using a simple one-at-a-time parameter sweep. We used training
batches of 256 time-steps and sized the reply buffers to hold 2 048 time-
steps. In each iteration we collected 256 time-steps and ran two training
epochs. We tuned the learning rates to 0.04 for the actors and 0.06 for the
critics, the target network update parameters $\tau$ to 0.06, and the discount
factors $\gamma$ to 0.95. We specified the regularization coefficient
$\lambda=2$, the regularization prior for the single-agent setting as
$\pi_{0}=[P(\mathcal{A}_{1}),P(\mathcal{A}_{2}),P(\mathcal{A}_{3})]=[0.0,0.6,0.4]$,
and the regularization priors for the dual-agent setting as
$\pi_{0,1}=[0.0,0.6,0.4]$ and $\pi_{0,2}=[0.4,0.6,0.0]$. This meant that the
single agent was regularized to not take any left turns, while slightly
favouring going straight above turning right. For the dual agents, agent 1 was
to avoid left turns while agent 2 was to instead avoid right turns; we did
this to demonstrate the characterization of the agents as preferring either
left or right turns while navigating to their destinations.

###  4.2 Results

In the single agent setting of our toy problem, we used our algorithm to
encourage an agent to prefer right turns over left turns; we used a
regularization prior $\pi_{0}=[0.0,0.6,0.4]$ to regulate the probability of
_left actions_ to $0.0$, _straight actions_ to $0.6$, and _right actions_ to
$0.4$. Figure 1 shows three different trajectories that demonstrate such an
agent’s behaviour for destinations which lie either to the left, straight
ahead, or to the right of the agent’s starting location. As expected, the
agent never turned left and always took the shortest route to its destination
given its constraints.

(a) The agent made a series of right turns to reach its destination on the
left. (b) The destination was straight ahead and the agent needed not turn.
(c) The agent followed the shortest path and made a right turn.

Figure 1: Three trajectories of a single agent in the navigation problem. The
starting locations are consistently (3,0) and the destinations are marked by
red circles. During learning, the agent received a regularization prior
$\pi_{0}=[0.0,0.6,0.4]$ where the values in $\pi_{0}$ correspond to the
probabilities of the actions _turn left_ , _go straight_ , and _turn right_
respectively.

Figure 2 shows the same grid navigation problem, but this time for a multi-
agent setting. Here, we used two agents with different regularization terms to
constrain their actions to 1) right turns only and 2) left turns only; the
first agent’s regularization prior $\pi_{0,1}=[0.0,0.6,0.4]$ specified a
probability of $0.0$ for the _left action_ , $0.6$ for the _straight action_ ,
and $0.4$ for the _right action_ , while the second agent’s regularization
prior $\pi_{0,2}=[0.4,0.6,0.0]$ specified a probability of $0.4$ for the _left
action_ , $0.6$ for the _straight action_ , and $0.0$ for the _right action_.
Clearly, the two agents have learned different strategies in the navigation
problem. In Figure 2 it is clear that the two agents consistently took the
shortest path to their respective destinations while adhering to their
individual constraints. We therefore characterize them as agents that
preferred to take right and left turns, respectively. Crucially, this is an
_intrinsic_ property of the agents imposed by the regularization of the
objective function. This separates our method of intrinsic characterization
from post-hoc characterization techniques.

(a) Both agents’ destinations were on the left, but only the agent regularized
to prefer left turns actually turned left. The other agent completed a series
of right turns to reach its destination. (b) Both agents’ destinations were
located such that they had to perform a series of turns in the direction
according to their regularization priors $\pi_{0,i}$

(c) Both agents’ destinations were located such that they could perform their
preferential turn - either to the left, or to the right - according to their
regularization priors $\pi_{0,i}$ (d) One agent’s destination was straight
ahead and it needed not turn. Clearly, the regularization priors $\pi_{0,i}$
allowed for such strategies.

Figure 2: Four sets of trajectories for a dual-agent environment in the
navigation problem. The first agent - labelled ’right turns’ - received a
regularization prior $\pi_{0,1}=[0.0,0.6,0.4]$ while the second agent -
labelled ’left-turns’ received a regularization prior
$\pi_{0,2}=[0.4,0.6,0.0]$ where the values in $\pi_{0,i}$ correspond to the
probabilities of the actions _turn left_ , _go straight_ , and _turn right_.

Finally, Figure 3 shows a typical curve of training and testing returns across
the 3 000 training iterations. The agents clearly demonstrate a good learning
response with steadily increasing returns both in training and testing.

Figure 3: Training and testing returns for a typical training run. The
learning process clearly followed a steady increase in returns.

##  5 Conclusions and Direction for Future Work

Our objective was the _intrinsic_ characterization of RL agents. To this end,
we investigated and briefly summarized the relevant state-of-the-art in
explainable RL and found that these methods have typically been relying on
post-hoc evaluations of a learned policy. Policy regularization is a method
that modifies a policy; however, it has typically been employed to enhance
training performance which does not necessarily aid in policy
characterization. We therefore adapted entropy regularization from maximizing
the entropy in the policy to minimizing the mean squared difference between
the expected action and a given prior. This encourages the agent to mimic a
predefined behaviour while maximizing its reward during learning. Finally, we
extended MADDPG with our regularization term. We provided a formal argument
for the validity of our algorithm and empirically demonstrated its functioning
in a toy problem. In this problem, we characterized two agents to follow
different approaches when navigating to a destination in a grid; while one
agent performed only right turns, the other performed only left turns. We
conclude that our fundamentally sound algorithm was able to imbue specific
characteristic behaviours into our agents’ policies. In future work, we intend
to use this algorithm to develop a set of financial advisors that will
optimize individual customers’ investment portfolios according to their
individual spending personalities [24]. While maximising portfolio values,
these agents may prefer, e.g., property investments over crypto currencies
which are analogous to right turns and left turns in our toy problem.

## References

  * [1] A. Heuillet, F. Couthouis, and N. Díaz-Rodríguez.  Explainability in deep reinforcement learning.  Knowledge-Based Systems, 214(106685):1–24, 2021.
  * [2] Javier García, Fern, and o Fernández.  A comprehensive survey on safe reinforcement learning.  Journal of Machine Learning Research, 16(42):1437–1480, 2015.
  * [3] Alejandro Barredo Arrieta, Natalia Díaz-Rodríguez, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera.  Explainable artificial intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI.  Information Fusion, 58:82–115, 2020.
  * [4] Lindsay Wells and Tomasz Bednarz.  Explainable AI and reinforcement learning: A systematic review of current approaches and trends.  Frontiers in Artificial Intelligence, 4:1–48, 2021.
  * [5] S. Gupta, G Singal, and D. Garg.  Deep reinforcement learning techniques in diversified domains: A survey.  Archives of Computational Methods in Engineering, 28:4715–4754, 2021.
  * [6] Tuomas Haarnoja, Haoran Tang, P. Abbeel, and Sergey Levine.  Reinforcement learning with deep energy-based policies.  In ICML, 2017.
  * [7] Alexandre Galashov, Siddhant Jayakumar, Leonard Hasenclever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins, Wojtek M. Czarnecki, Yee Whye Teh, Razvan Pascanu, and Nicolas Heess.  Information asymmetry in KL-regularized RL.  In International Conference on Learning Representations, 2019.
  * [8] John Lu, Sunanda Dissanayake, Nelson Castillo, and Kristine Williams.  Safety evaluation of right turns followed by u-turns as an alternative to direct left turns - conflict analysis.  Technical report, CUTR Research Reports, 2001.
  * [9] Régis Riveret, Yang Gao, Guido Governatori, Antonino Rotolo, Jeremy V. Pitt, and Giovanni Sartor.  A probabilistic argumentation framework for reinforcement learning agents.  Autonomous Agents and Multi-Agent Systems, 33:216–274, 2019.
  * [10] P. Madumal, T. Miller, L. Sonenberg, and F. Vetere.  Explainable reinforcement learning through a causal lens.  arXiv, 1905.10958v2, 2019.
  * [11] Harm van Seijen, Mehdi Fatemi, Joshua Romoff, Romain Laroche, Tavian Barnes, and Jeffrey Tsang.  Hybrid reward architecture for reinforcement learning.  arXiv, 1706.04208, 2017.
  * [12] Z. Juozapaitis, A. Koul, A. Fern, M. Erwig, and F. Doshi-Velez.  Explainable reinforcement learning via reward decomposition.  International Joint Conference on Artificial Intelligence. A Workshop on Explainable Artificial Intelligence., 2019.
  * [13] B. Beyret, A. Shafti, and A.A. Faisal.  Dot-to-dot: Explainable hierarchical reinforcement learning for robotic manipulation.  In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS, page 5014–5019, 2019.
  * [14] Luca Marzari, Ameya Pore, Diego Dall’Alba, Gerardo Aragon-Camarasa, Alessandro Farinelli, and Paolo Fiorini.  Towards hierarchical task decomposition using deep reinforcement learning for pick and place subtasks.  arXiv, 2102.04022, 2021.
  * [15] P. Sequeira, E. Yeh, and M.T. Gervasio.  Interestingness elements for explainable reinforcement learning through introspection.  IUI Workshops, page 1–7, 2019.
  * [16] Michael L. Littman.  Markov games as a framework for multi-agent reinforcement learning.  In In Proceedings of the Eleventh International Conference on Machine Learning, pages 157–163, 1994.
  * [17] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra.  Continuous control with deep reinforcement learning.  In ICLR (Poster), 2016.
  * [18] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, P. Abbeel, and Igor Mordatch.  Multi-agent actor-critic for mixed cooperative-competitive environments.  In NIPS, 2017.
  * [19] Brian D. Ziebart.  Modeling Purposeful Adaptive Behavior with the Principle of Maximum Causal Entropy.  PhD thesis, Machine Learning Department, Carnegie Mellon University, December 2010.
  * [20] Richard Cheng, Abhinav Verma, Gabor Orosz, Swarat Chaudhuri, Yisong Yue, and Joel W. Burdick.  Control regularization for reduced variance reinforcement learning.  arXiv, 1905.05380, 2019.
  * [21] Simone Parisi, Voot Tangkaratt, Jan Peters, and Mohammad Emtiyaz Khan.  Td-regularized actor-critic methods.  Machine Learning, 108(8-9):1467–1501, Feb 2019.
  * [22] Sobhan Miryoosefi, Kianté Brantley, Hal Daume III, Miro Dudik, and Robert E Schapire.  Reinforcement learning with convex constraints.  In Advances in Neural Information Processing Systems, volume 32, pages 1–10, 2019.
  * [23] Yinlam Chow, Mohammad Ghavamzadeh, Lucas Janson, and Marco Pavone.  Risk-constrained reinforcement learning with percentile risk criteria.  Journal of Machine Learning Research, 18:1–51, 12 2015.
  * [24] Charl Maree and Christian W. Omlin.  Clustering in recurrent neural networks for micro-segmentation using spending personality (in print).  In 2021 IEEE Symposium Series on Computational Intelligence (SSCI), pages 1–5, 2021.