###### Abstract

Development of distributed Multi-Agent Reinforcement Learning (MARL)
algorithms has attracted an increasing surge of interest lately mainly due to
the recent advancements of Deep Neural Networks (DNNs). Complex cooperative,
competitive or mixed behavior among the agents in the multi-agent
environments, make them more appealing to real world scenarios. Generally
speaking, conventional Model-Based (MB) or Model-Free (MF) RL algorithms are
not directly applicable to the MARL problems due to utilization of a fixed
reward model for learning the underlying value function. While DNN-based
solutions perform utterly well when a single agent is involved, such methods
fail to fully generalize to the complexities of MARL problems. In other words,
although recently developed approaches based on DNNs for multi-agent
environments have achieved superior performance, they are still prone to
overfiting, high sensitivity to parameter selection, and sample inefficiency.
In this paper, an adaptive Kalman filter-based framework is introduced as an
efficient alternative to address the aforementioned problems. More
specifically, the paper proposes the Multi-Agent Adaptive Kalman Temporal
Difference (MAK-TD) framework and its Successor Representation-based variant,
referred to as the MAK-SR. Intuitively speaking, the main objective is to
capitalize on unique characteristics of Kalman Filtering (KF) such as
uncertainty modeling and online second order learning. The proposed MAK-TD/SR
frameworks consider the continuous nature of the action-space that is
associated with high dimensional multi-agent environments and exploit Kalman
Temporal Difference (KTD) to address the parameter uncertainty. By leveraging
the KTD framework, SR learning procedure is modeled into a filtering problem,
where Radial Basis Function (RBF) estimators are used to encode the continuous
space into feature vectors. On the other hand, for learning localized reward
functions, we resort to Multiple Model Adaptive Estimation (MMAE), as a remedy
to deal with the lack of prior knowledge on the observation noise covariance
and observation mapping function. The proposed MAK-TD/SR frameworks are
evaluated via several experiments, which are implemented through the OpenAI
Gym MARL benchmarks. In these experiments, different number of agents in
cooperative, competitive and mixed (cooperative-competitive) scenarios are
utilized. The experimental results illustrate superior performance of the
proposed MAK-TD/SR frameworks compared to their state-of-the-art counterparts.

###### keywords:

Kalman Temporal Difference; Multiple Model Adaptive Estimation; Multi-Agent
Reinforcement Learning; Successor Representation.

x 10.3390/—— xx Multi-Agent Reinforcement Learning via Adaptive Kalman
Temporal Difference and Successor Representation Mohammad Salimibeni 1, Arash
Mohammadi ${}^{1}*$, Parvin Malekzadeh 2, and Konstantinos N. Plataniotis 2
Mohammad Salimibeni, Parvin Malekzadeh, Konstantinos N. Plataniotis, and Arash
Mohammadi Correspondence<EMAIL_ADDRESS>Tel.: +1-514-848-2712
ext. 2712 Current address: 1455 De Maisonneuve Blvd. W. EV-009.187, Montreal,
Quebec, Canada, HG-1M8

##  1 Introduction

Reinforcement Learning (RL), as a class of Machine Learning (ML) techniques,
targets providing human-level adaptive behavior by construction of an optimal
control policy Sutton . Generally speaking, the main underlying objective is
learning (via trial and error) from previous interactions of an autonomous
agent and its surrounding environment. The optimal control (action) policy can
be obtained via RL algorithms through the feedback that environment provides
to the agent after each of its actions salimiICASSP ; Parvin:access ; Spano ;
Seo ; DrMing1 ; DrMing2 . Policy optimality can be reached via such an
approach with the goal of increasing the reward over time. In most of the
successful RL applications, e.g., Go and Poker games, robotics, and autonomous
driving, typically, several autonomous agents are involved. This naturally
falls within the context of Multi-Agent RL (MARL), which is a relatively long-
established domain, however, it has recently been revitalized due to the
advancements made in the single-agent RL approaches. In the MARL domain, which
is the focus of this manuscript, multiple decision-making agents interact
(cooperate and/or compete) in a shared environment to gain a common or a
conflicting goal.

Literature Review: Traditionally, RL algorithms are classified as: (i) Model-
Free (MF) approaches Parvin:access ; Turchetta2020 ; Jing2021  where sample
trajectories are exploited for learning the value function, and; (ii) Model-
Based (MB) techniques Liu2020  where reward functions are estimated by
leveraging search trees or dynamic programming Bellman1954 . MF methods,
generally, do not adapt quickly to local changes in the reward function. On
the other hand, MB techniques can adapt quickly to changes in the environment,
but this comes with a high computational cost Song2021 ; Sam ; Vertes . To
address the above adaptation problems, Successor Representations (SR)
approaches Geerts2019 ; Machado2021  are proposed as an alternative RL
category. The SR method provides the flexibility of the MB algorithm and has
computational efficiency comparable to that of the MF algorithms. In SR-based
methods, both the immediate reward expected to be received after each action
and the discounted expected future state occupancy (which is called the SR)
are learned. Afterwards, in each of the successor states, the value function
is factorized into the SR and the immediate reward. This factorization only
needs learning of the reward function for new tasks, allowing rapid policy
evaluation when reward conditions are changed. In scenarios with a limited
number of states, the SR and the reward function (hence, the value function)
associated with each state can be readily computed. Computation of the value
function, however, is infeasible for MARL problems as in such scenarios we
deal with a large number of continuous states Moskovitz2021 . In other words,
conventional approaches developed for single agent scenarios such as single-
agent SR, Q-Learning, or policy gradient can not be directly adopted to MARL
to compute the value function. The main problem here is that, typically, from
a single agent’s perspective, the environment tends to become unstable as each
agent’s policies change during the training process. In the context of deep
Q-learning Hasselt2016 , this leads to stabilization issues as it is difficult
to properly use the previous localized experiences. From the perspective of
policy gradient, typically, observations demonstrate high variance in
coordinating multiple agents.

To leverage SR-based solutions for MARL, value function approximation is
unavoidable and one can use either linear or non-linear estimation approaches
Babu2012 ; Riedmiller2005 . In both categories, a set of adjustable parameters
define the value of the approximated function. Non-linear function
approximators, such as Deep Neural Networks (DNNs) Riedmiller2005 ; Tang ; Kim
; Xie  have enabled application of RL methods to complex multi-agent
scenarios. While DNN approaches like Deep Q-Networks (DQN) Mnih2013  and Deep
Deterministic Policy Gradient (DDPG) Lillicrap2015  achieved superior results,
they suffer from some major disadvantages including the overfitting problem;
high sensitivity in choosing parameters; sample inefficiency, and; high number
of episodes required for training the models. The linear function
approximators, on the other hand, transform the approximation problem into a
weight calculation problem in order to fuse several local estimators.
Convergence can be examined when linear function approximators are utilized as
they are better understood than their non-linear counterparts Tsitsiklis1997 ;
Bertsekas2004 . Cerebellar Model Articulation Controllers (CMACs) Miller1990
and Radial Basis Functions (RBFs) Haykin1994  are usually used as linear
estimators in this context. It has been shown, however, that the function
approximation process can be better represented via gradual-continuous
transitions Barreto2008 . Albeit the computation of the RBFs’ parameters is
usually based on prior knowledge of the problem at hand, these parameters can
also be adapted leveraging observed transitions in order to improve the
autonomy of the approach. In this context, cross entropy and gradient descent
methods Menache2005  can be utilized for the adaptation task. Stability of the
gradient descent-based approach was later improved by exploiting a restrictive
method in Barreto2008 , which is adopted in this manuscript.

After verifying the value function’s structure, to train the value function
approximator, the following methodologies can be used: (i) Bootstrapping
methods, e.g., Fixed-Point Kalman Filter (FPKF) Choi2006 ; (ii) Residual
techniques such as Kalman Temporal Difference (KTD), and Gaussian Process
Temporal Difference (GPTD) Engel2005 , which is a special form of the KTD,
and; (iii) Projected fixed-point methods such as Least Square Temporal
Difference (LSTD) Bradtke1996 . Among these methodologies, KTD Geist2010  is a
prominent technique as based on the selected structure, it provides both
uncertainty and Minimum Mean Square Error (MMSE) approximation of the value
function. In particular, uncertainty is beneficial for achieving higher sample
efficiency. The KTD approach, however, requires prior knowledge of the
filter’s parameters (e.g., noise covariance of the process and measurement
models), which are not readily available in realistic circumstances. Parameter
estimation is a well-studied problem within the context of Kalman Filtering
(KF), where several adaptive schemes are developed over the years including
but not limited to Multiple Model Adaptive Estimation (MMAE) methods
Arash:TNSRE:2015 ; Arash:Sensors2016 ; Arash:TSP:2015  and, innovation-based
adaptive schemes Mehra1970 . When the system’s mode is changing, the latter
has the superiority to adapt faster and its efficiency was shown in Assa2018 ,
where different suggested averaging and weighting patterns were compared. MMAE
methods were already utilized in the RL problems, for instance, Reference
Kitao2017  proposed a multiple model KTD coupled with a model selection
mechanism to address issues related to the parameter uncertainty. Existing
multiple model methodologies are, however, not easily generalizable to the
MARL problem.

In methods proposed in Sam ; Ma2018 ; Momennejad2017 ; Russek2017 , while the
classical TD learning is coupled with DNNs, uncertainty of the value function
and that of the SR are not studied. To deal with uncertainty, a good
combination of exploitation and exploration should be used to prevent the
agent’s overconfidence about its knowledge to fully rely on exploitation.
Alternatively, an agent can perform exploration over other possible actions,
which might lead to improved results and a reduction in the uncertainty.
Although, from computation points of view, it is intractable to find an
optimal trade-off between exploitation and exploration, it has been
represented that exploration can benefit from the uncertainty in two separate
ways, i.e., through added randomness to the value function, and via shifting
towards uncertain action selection Sutton . Consequently, the approximated
value function’s uncertainty, is a beneficial information for resolving the
available conflict between exploration and exploitation Sutton ; Geist2013 .
It was shown in Geist2013  that, the sensitivity of the framework to the
parameters of the model can be diminished via uncertainty incorporation within
the KTD method. Therefore, the required time and memory to find/learn the best
model will be reduced compared to DNN-based methods Sam ; Ma2018 ;
Momennejad2017 ; Russek2017 . The reduced sensitivity in setting the
parameters enhances the reproducibility feature of a reliable approach, which
leads to regeneration of more consistent outputs while running multiple
learning epochs. Consequently the risk of getting unacceptable results in real
scenarios will be decreased Chan2020 . Geerts et al. Geerts2019  leveraged KTD
framework to estimate the SR for problems with discrete state-spaces, however,
information related to uncertainty of the estimated SR is not considered in
the action selection procedure. Malekzadeh et al. Parvin2021  applied the KTD
approach within the SR learning context and used the uncertainty associated
with the estimated SR for action estimation process, but only considered
single agent environments.

Contributions: The paper proposes Multi-Agent Adaptive Kalman Temporal
Difference (MAK-TD) framework and its SR-based variant, the Multi-Agent
Adaptive Kalman Successor Representation (MAK-SR) framework. The MAK-TD/SR
frameworks consider the continuous nature of the action-space that is
associated with high dimensional multi-agent environments and exploit KTD to
address the parameter uncertainty. By leveraging the KTD framework, SR
learning procedure is modeled into a filtering problem in this work.
Intuitively speaking, the goal is to take advantage of the inherent benefits
of the KF, i.e., online second order learning, uncertainty estimation and non-
stationary handling. Afterwards, RBF-based estimation is utilized within the
MAK-TD/SR frameworks in order for continuous states to be encoded into feature
vectors and for the reward function to be projected as a linear function of
the extracted feature vectors. On the other hand, for learning localized
reward functions, we resort to MMAE, as a remedy to deal with the lack of
prior knowledge on observation noise covariance and observation mapping
function. In summary, the paper makes the following key contributions:

  * •

Within the MARL domain, the so-called MAK-TD framework is proposed as
compensation for the information inadequacy about a key unknown filter’s
parameter, which is the measurement noise covariance. For learning the optimal
policy and to simultaneously enhance sample efficiency of the proposed MAK-TD,
an off-policy Q-learning approach is implemented.

  * •

MAK-TD is extended to MAK-SR by incorporation of the SR learning process into
the filtering problem using KTD formulation for learned SR uncertainty
approximation. Moreover, adopting KTD is beneficial to reduce the required
memory/time to learn the SR while reducing model’s sensitivity to parameters
selection (i.e., more reliability) in comparison to DNN-based algorithms.

  * •

A coupled gradient descent and MMAE-based scheme is adopted for development of
the MAK-SR framework to form a KF-based approximation of the reward function.
The ultimate objective is reducing sensitivity to having prior knowledge on
the observation noise covariance and observation mapping function.

  * •

For establishing a trade off between exploration and exploitation, an
innovative active learning mechanism is implemented to incorporate uncertainty
of the value function obtained from the SR learning. Such a mechanism results
in efficiently enhancing performance in terms of cumulative reward.

A multi-agent extension of the OpenAI gym benchmark, a two-dimensional world
with continuous space Mordatch2018  is utilized to simulate cooperative,
competitive scenarios, and mix interaction settings. The proposed MAK-TD/SR
frameworks are evaluated through a comprehensive set of experiments and
simulations illustrating their superior performance compared to their
counterparts.

The remainder of the paper is organized as follows: In Section 2, the basics
of RL and MARL are briefly discussed. The proposed MAK-TD framework is
presented in Section 3, and its SR-based variant, the MAK-SR framework, is
introduced in Section 4. Experimental results based on multi-agent RL
benchmark are presented in Section 5. Section 6, finally, concludes the paper.

##  2 Problem Formulation

To provide the background required for development of the proposed MAK-TD/SR
frameworks, in this section, we present an overview of single agent and MARL
techniques.

###  2.1 Single-Agent Reinforcement Learning (RL)

In conventional RL scenarios, typically, a single agent is placed in an
unknown environment performing autonomous actions with the goal of maximizing
its accumulated reward. In such scenarios, the agent starts its interactions
with the environment in an initial state denoted by $\bm{s}_{0}$ and continues
to interact with the environment until reaching a pre-defined terminal state
$\bm{s}_{T}$. Action set $\mathcal{A}$ is defined from which the agent can
select potential actions following a constructed optimal policy. In other
words, given its current state $\bm{s}_{k}\in\mathcal{S}$, the single agent
follows a policy denoted by $\pi_{k}$ and performs action
$a_{k}\in\mathcal{A}$ at time $k$. Following the agent’s action, based on
transition probability of $P(\bm{s}_{k+1}|\bm{s}_{k},a_{k})\in\mathcal{P}_{a}$
it moves to a new state $\bm{s}_{k+1}\in\mathcal{S}$ receiving reward of
$r_{k}\in\mathcal{R}$. A discount factor $\gamma\in(0,1)$ is utilized to
incorporate future rewards as such balancing the immediate rewards and future
ones. In summary, a Markov Decision Process (MDP), denoted by $5$-tuple
$\\{\mathcal{S},\mathcal{A},\mathcal{P}_{a},\mathcal{R},\gamma\\}$, is,
typically, used as the underlying mathematical model that governs the RL
process. Therefore, the main objective is learning an optimal policy to map
states into actions by maximizing the expected sum of discounted rewards,
which is referred to as the optimal policy $\pi^{*}$ Sutton1998 . The optimal
policy $\pi^{*}$ is, typically, obtained based on the following state-action
value function

$\displaystyle
Q_{\pi}(\bm{s},a)=\mathbb{E}\left\\{\sum_{k=0}^{T}\gamma^{k}r_{k}|\bm{s}_{0}=\bm{s},a_{0}=a,a_{k}=\pi(\bm{s}_{k})\right\\}.$
(1)

Note that in Eq. (1), $\mathbb{E}\\{\cdot\\}$ denotes the expectation
operator. To perform an action at the learning stage, the current policy is
utilized. Once convergence is reached,
$a_{k}=\arg\max_{a\in\mathcal{A}}Q_{\pi^{*}}(\bm{s}_{k},a)$, which is the
optimal policy, can be used by the agent to perform the required tasks. This
completes a brief introduction to RL, next, the TD learning is reviewed as a
building block of the proposed MAK-TD/SR frameworks.

###  2.2 Off-Policy Temporal Difference (TD) Learning

By taking an action and moving from one state to another, based on the Bellman
equation and Bellman update scheme Hutter2008 , the value function is
gradually updated using sample transitions. This procedure is referred to as
Temporal Difference (TD) update Hutter2008 . There are two approaches to
update policy namely “on-policy learning” or “off-policy learning”. The former
techniques use the current policy for action selection. For example, SARSA
Sutton1996 ; Xia2019  is an on-policy approach that optimizes the network as

$\displaystyle
Q_{\pi}(\bm{s}_{k},a_{k})=Q_{\pi}(\bm{s}_{k},a_{k})+\alpha\Big{(}r_{k}+\gamma\,Q_{\pi}(\bm{s}_{k+1},a_{k+1})-Q_{\pi}(\bm{s}_{k},a_{k})\Big{)},$
(2)

where $\alpha$ denotes the learning rate and $Q_{\pi}(\bm{s}_{k},a_{k})$ is
the state-action value function. In on-policy methods, by following a defined
policy, selecting a new state becomes a non-optimal task. Additionally, this
approach seems to be inefficient in sample selection since the value function
is updated through the current policy instead of using the optimized one. In
“off-policy” solutions, such as Q-learning Watkins1992 ; Ge2019 ; Xia2019 ;
Li2019 , the information received from previous policies is exploited to
update the policy and reach a new one (exploitation). On the other hand, to
properly explore new states, a stochastic policy is usually chosen as the
behavior policy (exploration). In brief, Q-learning is formed based on the
Bellman optimal equation as follows

$\displaystyle
Q_{\pi^{*}}(\bm{s}_{k},a_{k})=Q_{\pi^{*}}(\bm{s}_{k},a_{k})+\alpha\Big{(}r_{k}+\gamma\max_{a\in\mathcal{A}}Q_{\pi^{*}}(\bm{s}_{k+1},a)-Q_{\pi^{*}}(\bm{s}_{k},a_{k})\Big{)},$
(3)

where the optimal policy $\pi^{*}$ is used to form the state-action value
function ${Q_{\pi^{*}}(\bm{s}_{k},a_{k})}$. The policy can be obtained via a
greedy approach as follows

$\displaystyle
V_{\pi^{*}}(\bm{s})=\max_{a\in\mathcal{A}}Q_{\pi^{*}}(\bm{s}_{k},a).$ (4)

Upon convergence, actions can be selected based on the optimal policy and not
the behaviour policy as follows

$\displaystyle a_{k}=\arg\max_{a\in\mathcal{A}}Q_{\pi^{*}}(\bm{s}_{k},a).$ (5)

This completes our discussion on TD learning. In what follows, we discuss the
MARL approaches as well as value function approximation using the proposed
algorithms in the multi-agent environments.

###  2.3 Multi-Agent Setting

Within the context of MARL, we consider a scenario with $N$ agents, each with
its localized observations, actions, and states. In other words, Agent $i$,
for ($1\leq i\leq N$), utilizes policy $\pi^{(i)}$, which is a function from
the cartesian product of its localized action set $\mathcal{A}^{(i)}$ and its
localized observation set $\mathcal{Z}^{(i)}$ to a real number within zero and
one. We use superset
$\mathbb{S}=\\{\mathcal{S}^{(1)},\ldots,\mathcal{S}^{(N)}\\}$ to collectively
represent all the localized states, $\mathcal{S}^{(i)}$, for ($1\leq i\leq
N$). Likewise, supersets
$\mathbb{A}=\\{\mathcal{A}^{(1)},\ldots,\mathcal{A}^{(N)}\\}$ and
$\mathbb{Z}=\\{\mathcal{Z}^{(1)},\ldots,\mathcal{Z}^{(N)}\\}$ are used to
jointly represent all the localized actions, and local observations,
respectively. Each agent makes localized decisions following the transition
function
$T:\mathbb{S}\times\mathcal{A}^{(1)}\times,\ldots,\times\mathcal{A}^{(N)}\rightarrow\mathbb{S}^{2}$.
Consequently, an action is performed locally resulting in a new localized
measurement and a localized reward
$r^{(i)}:\mathbb{S}\times\mathcal{A}^{(i)}\rightarrow{\mathbb{R}}$. The main
objective of each agent is to maximize its localized expected return
$R^{(i)}=\sum_{t=0}^{T}\gamma^{t}(r^{(i)})^{t}$ over a termination window of
$T$ using a predefined discount factor of $\gamma$.

Traditional models like policy gradient or Q-Learning are not suitable for
MARL scenarios Lowe2017 , since the policy of an agent changes during the
progress of the training, and the environment becomes non-stationary towards
that specific agent’s points of view. Consequently, most recently proposed
platforms for multi-agent scenarios employ other strategies, where the agents’
own observation (known as local information at the execution time) are
exploited to learn optimal localized policies. Typically, such methods do not
consider specific communication patterns between agents or any differentiable
model of the environment’s dynamics Lowe2017 . Moreover, these models support
different interactions between agents from cooperation to competition or their
combination Lowe2017 ; Singh2019 . In this context, an adaptation is made
between the decentralized execution and centralized training to be able to
feed the policy training steps with more available data to speed up the
process of finding the optimal policy.

###  2.4 Multi-Agent Successor Representation (SR)

Within the context of SR, given an initial action $a^{(i)}$, and an initial
state $\bm{s}^{(i)}$, the expected discounted future state occupancy of state
${\bm{s}^{\prime}}^{(i)}$ is estimated based on the current policy $\pi^{(i)}$
as follows

$\displaystyle\bm{M}_{\pi^{(i)}}(\bm{s}^{(i)},{\bm{s}^{\prime}}^{(i)},a^{(i)})=\mathbb{E}\left[\sum_{k=0}^{T}\gamma^{k}\mathbbm{1}[\bm{s}^{(i)}_{k}={\bm{s}^{\prime}}^{(i)}]|\bm{s}^{(i)}_{0}=\bm{s}^{(i)},a^{(i)}_{0}=a^{(i)}\right],$
(6)

where $\mathbbm{1}\\{\cdot\\}=1$ if
$\bm{s}^{(i)}_{k}={\bm{s}^{\prime}}^{(i)}$, otherwise it is zero. The SR can
be represented with a $N_{\bm{s}^{(i)}}\times N_{\bm{s}^{(i)}}$ matrix when
the state-space is discrete. The recursive approach used in Eq. (2), can be
leveraged to update SR as follows

$\displaystyle\bm{M}_{\pi^{(i)}}^{\text{new}}(\bm{s}^{(i)}_{k},{\bm{s}^{\prime}}^{(i)},a^{(i)}_{k})=\bm{M}_{\pi^{(i)}}^{\text{old}}(\bm{s}^{(i)}_{k},{\bm{s}^{\prime}}^{(i)},a^{(i)}_{k})+$
$\displaystyle\alpha\Big{(}\mathbbm{1}[\bm{s}^{(i)}_{k}={\bm{s}^{\prime}}^{(i)}]+\gamma\bm{M}_{\pi^{(i)}}(\bm{s}^{(i)}_{k+1},{\bm{s}^{\prime}}^{(i)},a^{(i)}_{k+1})-\bm{M}_{\pi^{(i)}}^{\text{old}}(\bm{s}^{(i)}_{k},{\bm{s}^{\prime}}^{(i)},a^{(i)}_{k})\Big{)}.$

After computation (approximation) of the SR, its inner product with the
estimated value of the immediate reward can be used to form the state-action
value function based on Eq. (1), i.e.,

$\displaystyle
Q_{\pi^{(i)}}(\bm{s}^{(i)}_{k},a^{(i)}_{k})=\sum_{{\bm{s}^{\prime}}^{(i)}\in\mathcal{S}^{(i)}}\bm{M}(\bm{s}^{(i)}_{k},{\bm{s}^{\prime}}^{(i)},a^{(i)}_{k})R^{(i)}({\bm{s}^{\prime}}^{(i)},a^{(i)}_{k}).$
(8)

As a final note, it is worth mentioning an important characteristic of the SR-
based approach, i.e., the state-action value function can be reconstructed
based on the reward function. The developed MARL/MASR formulation presented
here is used to develop the proposed MAK-TD/SR frameworks in the following
sections.

##  3 The MAK-TD Framework

As stated previously, the MAK-TD framework, is a Kalman-based off-policy
learning solution for multi-agent networks. More specifically, by exploiting
the TD approach represented in Eq. (3), the optimal value function associated
with the $i^{th}$ agent, for ($1\leq i\leq N$), can be approximated from its
one-step estimation as follows

$\displaystyle Q_{{\pi^{(i)}}^{*}}(\bm{s}_{k}^{(i)},a_{k}^{(i)})\approx
r_{k}^{(i)}+\gamma\max_{{a^{(i)}}\in\mathcal{A}}Q_{{\pi^{(i)}}^{*}}(\bm{s}_{k+1}^{(i)},a^{(i)}).$
(9)

By changing the variables’ order, the reward at each time can be represented
(modeled) as a noisy observation, i.e.,

$r_{k}^{(i)}=Q_{{\pi^{(i)}}^{*}}(\bm{s}_{k}^{(i)},a_{k}^{(i)})-\gamma\max_{{a^{(i)}}\in\mathcal{A}}Q_{\pi^{*}}(\bm{s}_{k+1}^{(i)},a^{(i)})+v_{k}^{(i)},$
(10)

where $v_{k}$ is modeled as a zero-mean normal distribution with variance of
$R^{(i)}$. By considering the local state-space of each agent, we use
localized basis functions to approximate each agent’s value function.
Therefore, the following value function can be formed for Agent $i$, for
($1\leq i\leq N$),

$\displaystyle
Q_{\pi^{(i)}}(\bm{s}_{k}^{(i)},a_{k}^{(i)})=\bm{\phi}(\bm{s}_{k}^{(i)},a_{k}^{(i)})^{T}\bm{\theta}_{k}^{(i)},$
(11)

where term $\bm{\phi^{(i)}}(\bm{s}^{(i)},a^{(i)})$ represents a vector of
basis functions; $\pi^{(i)}$ is the policy associated with Agent $i$, and;
finally, $\bm{\theta}^{(i)}_{k}$ denotes the vector of the weights.
Substituting Eq. (11) in Eq. (10) results in

$r_{k}^{(i)}=\Big{[}\bm{\phi}(\bm{s}_{k}^{(i)},a_{k}^{(i)})^{T}-\gamma\max_{{a^{(i)}}\in\mathcal{A}}\bm{\phi}(\bm{s}_{k+1}^{(i)},a^{(i)})^{T}\Big{]}\bm{\theta}_{k}^{(i)}+v_{k}^{(i)},$
(12)

which can be simplified into the following linear observation model

$r_{k}^{(i)}=[\bm{h}_{k}^{(i)}]^{T}\bm{\theta}_{k}^{(i)}+v_{k}^{(i)},$ (13)

with

$\displaystyle\bm{h}_{k}^{(i)}=\bm{\phi}(\bm{s}_{k}^{(i)},a_{k}^{(i)})-\gamma\max_{{a^{(i)}}\in\mathcal{A}}\bm{\phi}(\bm{s}_{k+1}^{(i)},a^{(i)}).$
(14)

In other words, Eq. (13) is the localized measurement (reward) of the
$i^{\text{th}}$ agent, which is a linear model of the weight vector
$\bm{\theta}_{k}^{(i)}$. For approximating localized weight
$\bm{\theta}_{k}^{(i)}$, first we leverage the observed reward, which is
obtained by transferring from state $s_{k}^{(i)}$ to $s_{k+1}^{(i)}$. Second,
given that the noise variance of the measurement is not known a-priori, we
exploit MMAE adaptation by representing it with $M$ different values
(${R^{j}}^{(i)}$, for ($1\leq j\leq M$). Consequently, a combination of $M$
KFs is used to estimate $\hat{\bm{\theta}}_{k}^{(i)}$ based on each of its
candidate values, i.e.,

$\displaystyle{\bm{K}_{k}^{j}}^{(i)}$ $\displaystyle=$
$\displaystyle\bm{P}_{(\bm{\theta},k|k-1)}^{(i)}\bm{h}_{k}^{(i)}\big{(}{\bm{h}_{k}^{T}}^{(i)}\bm{P}_{(\bm{\theta},k|k-1)}^{(i)}\bm{h}_{k}^{(i)}+{R^{j}}^{(i)}\big{)}^{-1}$
(15) $\displaystyle\hat{\bm{\theta}_{k}^{j}}^{(i)}$ $\displaystyle=$
$\displaystyle\hat{\bm{\theta}}_{(k|k-1)}^{(i)}+{\bm{K}_{k}^{j}}^{(i)}\big{(}r_{k}^{(i)}-{\bm{h}_{k}^{T}}^{(i)}\hat{\bm{\theta}}_{(k|k-1)}^{(i)}\big{)}$
(16) $\displaystyle{\bm{P}^{j}_{\bm{\theta},k}}^{(i)}$ $\displaystyle=$
$\displaystyle\big{(}\bm{I}-{\bm{K}_{k}^{j}}^{(i)}{\bm{h}_{k}^{T}}^{(i)}\big{)}{\bm{P}^{T}_{(\bm{\theta},k|k-1)}}^{(i)}\big{(}\bm{I}-{\bm{K}_{k}^{j}}^{(i)}{\bm{h}_{k}^{T}}^{(i)}\big{)}+{\bm{K}_{k}^{j}}^{(i)}{R^{j}}^{(i)}{{\bm{K}_{k}^{j}}^{T}}^{(i)},$
(17)

where superscript $j$ is used to refer to the $j^{\text{th}}$ matched KF, for
which a specific value (${R^{j}}^{(i)}$) is assigned to model covariance of
the observation model’s noise process. The posterior distribution associated
with each of the $M$ matched KFs is calculated based on its likelihood
function. All the matched aposteriori distributions are then added together
based on their corresponding weights to form the overall posterior
distribution given by

$\displaystyle
P^{(i)}(\bm{\theta}_{k}|\bm{Y}_{k})=\sum_{j=1}^{M}{\omega^{j}}^{(i)}P^{(i)}(\bm{\theta}_{k}^{(i)}|\bm{Y}_{k}^{(i)},{R^{j}}^{(i)}),$
(18)

where ${\omega^{j}}^{(i)}$ is the $j^{\text{th}}$ KF’s normalized observation
likelihood associated with the $i^{\text{th}}$ agent and is given by

${\omega^{j}}^{(i)}=P^{(i)}(r_{k}^{(i)}|\bm{\theta}_{(k|k-1)}^{(i)},R^{j})=c^{(i)}.e^{\big{[}\frac{-1}{2}\big{(}r_{k}^{(i)}-{\bm{h}_{k}^{T}}^{(i)}\hat{\bm{\theta}}_{(k|k-1)}^{(i)}\big{)}^{T}\big{(}{\bm{h}_{k}^{T}}^{(i)}\bm{P}_{(\bm{\theta},k|k-1)}^{(i)}\bm{h}_{k}^{(i)}+{R^{j}}^{(i)}\big{)}^{-1}}{\big{(}r_{k}^{(i)}-{\bm{h}_{k}^{T}}^{(i)}\hat{\bm{\theta}}_{(k|k-1)}^{(i)}\big{)}\big{]}},$
(19)

where $c^{(i)}=1/(\sum_{j=1}^{M}{w^{j}}^{(i)})$. Exploiting Eq. (18), the
weight and its error covariance are then updated as follows

$\displaystyle\hat{\bm{\theta}}_{k}^{(i)}$ $\displaystyle=$
$\displaystyle\sum_{j=1}^{M}{\omega^{j}}^{(i)}\hat{\bm{\theta}}{{}_{k}^{j}}^{(i)}$
(20) $\displaystyle\bm{P}_{\bm{\theta},k}^{(i)}$ $\displaystyle=$
$\displaystyle\sum_{j=1}^{M}{\omega^{j}}^{(i)}\left({\bm{P}_{\bm{\theta},k}^{j}}^{(i)}+(\hat{\bm{\theta}}{{}^{j}}^{(i)}-\hat{\bm{\theta}}^{(i)})(\hat{\bm{\theta}}{{}^{j}}^{(i)}-\hat{\bm{\theta}}^{(i)})^{T}\right).$
(21)

To finalize computation of $\hat{\bm{\theta}}_{k}^{(i)}$ based on Eqs.
(13)-(21), localized measurement mapping function $\bm{h}_{k}^{(i)}$ is
required. Since $\bm{h}_{k}^{(i)}$ is formed by the basis functions, its
adaptation necessitates the adaptation of the basis functions. The vector of
basis functions shown in Eq. (11) is formed as follows

$\bm{\phi}(\bm{s}^{(i)}_{k})=\big{[}\phi_{1}(\bm{s}^{(i)}_{k}),\phi_{2}(\bm{s}^{(i)}_{k}),\ldots,\phi_{N_{b}-1}(\bm{s}^{(i)}_{k}),\phi_{N_{b}}(\bm{s}^{(i)}_{k})\big{]}^{T},$
(22)

where $N_{b}$ is the number of basis functions. Each basis function is
represented by a RBF, which is defined by its mean and covariance parameters
as follows

$\phi_{n}(\bm{s}^{(i)}_{k})=\exp\\{\frac{-1}{2}(\bm{s}^{(i)}_{k}-\bm{\mu}^{(i)}_{n})^{T}{\bm{\Sigma}^{(i)}_{n}}^{-1}(\bm{s}^{(i)}_{k}-\bm{\mu}^{(i)}_{n})\\},$
(23)

where $\bm{\mu}^{(i)}_{n}$ and $\bm{\Sigma}^{(i)}_{n}$ are the mean and
covariance of $\phi_{n}(\bm{s}^{(i)}_{k})$, for ($1\leq n\leq N_{b}$).
Generally speaking, the state-action feature vector can be represented as
follows

$\displaystyle\bm{\phi}(\bm{s}_{k}^{(i)},a_{k}^{(i)})=[\phi_{1,a_{1}}(\bm{s}^{(i)}_{k}),\ldots\phi_{N_{b},a_{1}}(\bm{s}^{(i)}_{k}),\phi_{1,a_{2}}(\bm{s}^{(i)}_{k}),\ldots\phi_{N_{b},a_{D^{(i)}}}(\bm{s}^{(i)}_{k})]^{T},$
(24)

where
$\bm{\phi}(\cdot):\mathcal{A}^{(i)}\times\mathcal{S}\rightarrow\mathbb{R}^{N_{b}\times
D^{(i)}}$, and $D^{(i)}$ denotes the number of actions associated with the
$i^{\text{th}}$ agent. The state-action feature vector
$\bm{\phi}(\bm{s}^{(i)}_{k},a^{(i)}_{k}=a^{(i)}_{d})$, for ($1\leq d\leq
D^{(i)}$) in Eq. (24) is considered to be generated from
$\bm{\phi}(\bm{s}^{(i)}_{k})$ by placing this state feature vector in the
corresponding spot for action $a^{(i)}_{k}$ while the feature values for the
rest of the actions are set to zero, i.e.,

$\bm{\phi}(\bm{s}^{(i)}_{k},a^{(i)}_{k})=[0,\ldots
0,\phi_{1}(\bm{s}^{(i)}_{k}),\ldots,\phi_{N}(\bm{s}^{(i)}_{k}),0,\ldots
0,]^{T}.$ (25)

Due to the large number of parameters associated with the measurement mapping
function, the multiple model approach seems to be inapplicable. Alternatively,
Restricted Gradient Descent (RGD) Barreto2008  is employed, where the goal is
to minimize the following loss function

$L^{(i)}_{k}=(\bm{\phi}^{T}(\bm{s}^{(i)}_{k},a_{k})\,\bm{\theta}^{(i)}_{k}-r^{(i)}_{k})^{2}.$
(26)

The gradient of the objective function with respect to the parameters of each
basis function is then calculated using the chain rule as follows

$\displaystyle\Delta\bm{\mu}^{(i)}$ $\displaystyle=$
$\displaystyle-\frac{\partial
L_{k}^{(i)}}{\partial\bm{\mu}^{(i)}}=-\frac{\partial L_{k}^{(i)}}{\partial
Q_{{\pi^{*}}^{(i)}}}\frac{\partial
Q_{{\pi^{*}}^{(i)}}}{\partial\bm{\phi^{(i)}}}\frac{\partial\bm{\phi^{(i)}}}{\partial\bm{\mu}^{(i)}}$
(27) $\displaystyle\text{and }\Delta\bm{\Sigma}^{(i)}$ $\displaystyle=$
$\displaystyle-\frac{\partial\bm{\Sigma}_{k}^{(i)}}{\partial\bm{\mu}^{(i)}}=-\frac{\partial
L_{k}^{(i)}}{\partial Q_{{\pi^{*}}^{(i)}}}\frac{\partial
Q_{{\pi^{*}}^{(i)}}}{\partial\bm{\phi^{(i)}}}\frac{\partial\bm{\phi^{(i)}}}{\partial\bm{\Sigma}^{(i)}},$
(28)

where calculation of the partial derivations is done leveraging Eqs. (11),
(23), and (26). Therefore, the mean and covariance of the RBFs can be adapted
using the calculated partial derivative as follows

$\displaystyle\bm{\mu}^{(i)}_{n}$ $\displaystyle=$
$\displaystyle\bm{\mu}^{(i)}_{n}-2\lambda_{\bm{\mu}^{(i)}}{\left(L^{(i)}_{k}\right)}^{\frac{1}{2}}{\bm{\theta}^{(i)}_{k}}^{T}(\bm{\Sigma}^{(i)}_{n})^{-1}(\bm{s}^{(i)}_{k}-\bm{\mu}^{(i)}_{n})$
(29) $\displaystyle\bm{\Sigma}^{(i)}_{n}$ $\displaystyle=$
$\displaystyle\bm{\Sigma}^{(i)}_{n}-2\lambda_{\bm{\Sigma}^{(i)}}{\left(L^{(i)}_{k}\right)}^{\frac{1}{2}}{\bm{\theta^{(i)}}_{k}}^{T}(\bm{\Sigma}^{(i)}_{n})^{-1}\times(\bm{s}^{(i)}_{k}-\bm{\mu}^{(i)}_{n})(\bm{s}^{(i)}_{k}-\bm{\mu}^{(i)}_{n})^{T}{\bm{\Sigma}^{(i)}_{n}}^{-1},$
(30)

where both $\lambda_{\bm{\mu}^{(i)}}$ and $\lambda_{\bm{\Sigma}^{(i)}}$ denote
the adaptation rates. Based on Barreto2008 , for the sake of stability, only
one of the updates shown in Eqs. (29) and (30), will be applied. To be more
precise, when the size of the covariance is decreasing (i.e.,
${L^{(i)}_{k}}^{\frac{1}{2}}({\bm{\theta}^{(i)}_{k}}^{T}\bm{\phi}(\cdot))>0$),
the covariances of the RBFs are updated using Eq. (30), otherwise their means
are updated using Eqs. (29). Using this approach, unlimited expansion of the
RBF covariances is avoided.

One superiority that the proposed learning framework shows over other
optimization-based techniques (e.g., gradient descent-based methods) is the
calculation of the uncertainty for the weights $\bm{P}_{\bm{\theta},k}^{(i)}$,
which is directly related to the uncertainty of the value function. This
information can then be used at each step to select the actions, leading to
the most reduction in the weights’ uncertainty. Using the information form of
the KF (information filter AK2 ), the information of the weights denoted by
$\bm{P}_{\bm{\theta},k}^{(i)}$ is updated as follows

$\displaystyle{\bm{P}^{-1}_{\bm{\theta},k}}^{(i)}={\bm{P}^{-1}_{(\bm{\theta},_{k|k-1})}}^{(i)}+\bm{h}_{k}^{(i)}{R^{-1}}^{(i)}{\bm{h}_{k}^{T}}^{(i)}.$
(31)

In Eq. (31), the second element, i.e.,
$\bm{h}_{k}^{(i)}{R^{-1}}^{(i)}{\bm{h}_{k}^{T}}^{(i)}$, represents the
information received from the measurement. The action is obtained by
maximizing the information of the weights, i.e.,

$\displaystyle a_{k}^{(i)}$ $\displaystyle=$
$\displaystyle\arg\max_{a}\Big{(}\bm{h}_{k}^{(i)}(\bm{s}_{k}^{(i)},a^{(i)}){R^{-1}}^{(i)}{\bm{h}_{k}^{T}}^{(i)}(\bm{s}_{k}^{(i)},a^{(i)})\Big{)}$
(32) $\displaystyle=$
$\displaystyle\arg\max_{a}\Big{(}\bm{h}_{k}^{(i)}(\bm{s}_{k}^{(i)},a^{(i)}){\bm{h}^{T}_{k}}^{(i)}(\bm{s}_{k}^{(i)},a^{(i)})\Big{)}.$

The second equality in Eq. (32) is constructed as $R^{(i)}$ is a scalar. The
projected behaviour policy in Eq. (32) is different from that of Reference
Geist2010 , where a random policy was proposed, which favored actions with
less certainty of the value function. Although reducing the value function’s
uncertainty through action selection is an intelligent approach, it is less
efficient in sample selection due to the random nature of such policies.
Algorithm 1, briefly represents the MAK-TD framework proposed in this work.

Algorithm 1 The Proposed MAK-TD Framework

1:Learning Phase:

2:Set
$\bm{\theta}_{0},\bm{P}_{\bm{\theta},0},\bm{F},\bm{\mu}_{n,i_{d}},\bm{\Sigma}_{n,i_{d}}$
for $n={1,2,\ldots,N}$ and $i_{d}={1,2,\ldots,D}$

3:Repeat (for each episode):

4: Initialize $\bm{s}_{k}$

5: Repeat (for each agent $i$):

6: While $\bm{s}_{k}^{(i)}\neq\bm{s}_{T}$ do:

7:
$a_{k}^{(i)}=\arg\max\limits_{a}\Big{(}\bm{h}_{k}^{(i)}(\bm{s}_{k}^{(i)},a^{(i)}){\bm{h}_{k}^{T}}^{(i)}(\bm{s}_{k}^{(i)},a^{(i)})\Big{)}$

8: Take action $a_{k}^{(i)}$ , observe $\bm{s}_{k+1}^{(i)},r_{k}^{(i)}$

9: Calculate $\bm{\phi^{(i)}}(\bm{s}^{(i)},a^{(i)})$ via Eqs. (22) and (23)

10:
$\bm{h}_{k}^{(i)}(\bm{s}_{k}^{(i)},a_{k}^{(i)})=\bm{\phi^{(i)}}(\bm{s}_{k}^{(i)},a_{k}^{(i)})-\gamma\arg\max\limits_{a}\bm{\phi^{(i)}}(\bm{s}_{k+1}^{(i)},a^{(i)})$

11:
$\hat{\bm{\theta}}_{(k|k-1)}^{(i)}=\bm{F}^{(i)}{\hat{\bm{\theta}}_{k}}^{(i)}$

12:
$\bm{P}_{(\bm{\theta},k|k-1)}^{(i)}=\bm{F}^{(i)}\bm{P}_{\bm{\theta},k-1}^{(i)}{\bm{F}^{T}}^{(i)}+\bm{Q}^{(i)}$

13: for $j=1:M$ do:

14:
${\bm{k}_{k}^{j}}^{(i)}=\bm{P}_{(\bm{\theta},k|k-1)}^{(i)}\bm{h}_{k}^{(i)}({\bm{h}_{k}^{T}}^{(i)}\bm{P}_{(\bm{\theta},k|k-1)}^{(i)}\bm{h}_{k}^{(i)}+{R^{j}}^{(i)})^{-1}$

15:
$\hat{\bm{\theta}}{{}_{k}^{j}}^{(i)}=\hat{\bm{\theta}}_{(\bm{\theta},k|k-1)}^{(i)}+{\bm{k}_{k}^{j}}^{(i)}(r_{k}^{j}-\bm{h}{{}_{k}^{T}}^{(i)}\hat{\bm{\theta}}_{(k|k-1)}^{(i)})$

16:
$\bm{P}_{\bm{\theta},k}^{(i)}=(\bm{I}-{\bm{K}_{k}^{j}}^{(i)}{\bm{h}_{k}^{T}}^{(i)})\bm{P}_{(\bm{\theta},k|k-1)}^{(i)}(\bm{I}-{\bm{K}_{k}^{j}}^{(i)}{\bm{h}_{k}^{T}}^{(i)})^{T}+{\bm{K}_{k}^{j}}^{(i)}R^{j}{{\bm{K}_{k}^{j}}^{T}}^{(i)}$

17: end for

18: Compute the value of $c$ and ${w^{j}}^{(i)}$ by using
$\sum_{j=1}^{M}{w^{j}}^{(i)}=1$ and Eq. (19)

19:
$\hat{\bm{\theta}}_{k}^{(i)}=\sum_{j=1}^{M}{w^{j}}^{(i)}\hat{\bm{\theta}}{{}_{k}^{j}}^{(i)}$

20:
$\bm{P}_{\bm{\theta}_{k}}^{(i)}=\sum_{j=1}^{M}{\omega^{j}}^{(i)}\left({\bm{P}_{\bm{\theta},_{k}}^{j}}^{(i)}+(\hat{\bm{\theta}}{{}^{j}}^{(i)}-\hat{\bm{\theta}}^{(i)})(\hat{\bm{\theta}}{{}^{j}}^{(i)}-\hat{\bm{\theta}}^{(i)})^{T}\right)$

21: RBFs Parameters Update:

22:
$L^{(i)}_{k}=(\bm{\phi}^{T}(\bm{s}^{(i)}_{k},a_{k})\,\bm{\theta}^{(i)}_{k}-r^{(i)}_{k})^{2}$

23: if
${L^{(i)}_{k}}^{\frac{1}{2}}({\bm{\theta}^{(i)}_{k}}^{T}\bm{\phi}(\cdot))>0$
then:

24: Update $\bm{\Sigma}_{n,a_{d}}$ via Eq. (29)

25: else:

26: Update $\bm{\mu}_{n,a_{d}}$ via Eq. (30)

27: end if

28: end while

29:Testing Phase:

30:Repeat (for each trial episode):

31: While $\bm{s}_{k}\neq\bm{s}_{T}$ do:

32: Repeat (for each agent):

33: $a_{k}=\arg\max\limits_{a}\bm{\phi}(\bm{s}_{k},a)^{T}\bm{\theta}_{k}$

34: Take action $a_{k}$, and observe $\bm{s}_{k+1},r_{k}$

35: Calculate Loss $S_{k}$ for all agents

36: End While

##  4 The MAK-SR Framework

In the previous section, MAK-TD framework is proposed, which is a MM Kalman-
based off-policy learning solution for multi-agent networks. To learn the
value function, a fixed model for the reward function is considered, which
could restrict its application to more complex MARL problems. SR-based
algorithms are appealing solutions to tackle this issue where the focus is
instead on learning the immediate reward and the SR, which is the expected
discounted future state occupancy. In the existing SR-based approaches that
use standard temporal difference methods the uncertainty about the
approximated SR, is not captured. In order to address this issue, we extend
the MAK-TD framework and design its SR-based variant in this section. In other
words, MAK-TD is extended to MAK-SR by incorporation of the SR learning
procedure into the filtering problem using KTD formulation to estimate
uncertainty of the learned SR. Moreover, by applying KTD, we benefit from the
decrease in memory and time spent for the SR learning and also sensitivity of
the framework’s performance to its parameters (i.e., more reliable) when
compared to DNN-based algorithms.

Exact computation of the SR and the reward function is, typically, not
possible within the multi-agent settings as we are dealing with a large number
of continuous states. Therefore, we follow the approach developed in Section 3
and approximate the SR and the reward function via basis functions. For the
state-action feature vector $\bm{\phi}(\bm{s}^{(i)},a^{(i)})$, a feature-based
SR, which encodes the expected occupancy of the features, is defined as
follows

$\bm{M}_{\pi^{(i)}}(\bm{s}^{(i)},:,a^{(i)})=\mathbb{E}\left[\sum_{k=0}^{T}\gamma^{k}\bm{\phi}(\bm{s}^{(i)}_{k},a^{(i)}_{k})|\bm{s}^{(i)}_{0}=\bm{s}^{(i)},a^{(i)}_{0}=a^{(i)}\right].$
(33)

We consider that the immediate reward function for pair
$(\bm{s}^{(i)},a^{(i)})$ can be linearly factorized as

$\displaystyle
r^{(i)}(\bm{s}^{(i)}_{k},a^{(i)}_{k})\approx\bm{\phi}(\bm{s}^{(i)}_{k},a^{(i)}_{k})^{T}\bm{\theta}^{(i)}_{k},$
(34)

where $\bm{\theta}^{(i)}_{k}$ is the reward weight vector. The state-action
value function (Eq. (8)), therefore, can be computed as follows

$\displaystyle
Q(\bm{s}^{(i)}_{k},a^{(i)}_{k})={\bm{\theta}_{k}^{(i)}}^{T}\bm{M}(\bm{s}^{(i)}_{k},:,a^{(i)}_{k}).$
(35)

The SR matrix $\bm{M}(\bm{s}^{(i)}_{k},:,a^{(i)}_{k})$ can be approximated as
a linear function of the same feature vector as follows

$\displaystyle\bm{M}_{\pi^{(i)}}(\bm{s}^{(i)}_{k},:,a^{(i)}_{k})\approx\bm{M}_{k}\,\bm{\phi}(\bm{s}^{(i)}_{k},a^{(i)}_{k}).$
(36)

The TD learning of the SR then can be performed as follows

$\bm{M}_{\pi^{(i)}}^{\text{new}}(\bm{s}^{(i)}_{k},:,a^{(i)}_{k})=\bm{M}_{\pi^{(i)}}^{\text{old}}(\bm{s}^{(i)}_{k},:,a^{(i)}_{k})+\alpha\big{(}\bm{\phi^{(i)}}(\bm{s}^{(i)}_{k},a^{(i)}_{k})+\gamma\bm{M}_{\pi^{(i)}}(\bm{s}^{(i)}_{k+1},:,a^{(i)}_{k+1})-\bm{M}_{\pi^{(i)}}^{\text{old}}(\bm{s}^{(i)}_{k},:,a^{(i)}_{k})\big{)}.$
(37)

By defining the estimation structure of the SR and reward function, a suitable
method must be selected to learn (approximate) the weight vector of the reward
$\bm{\theta}^{(i)}$ and the weight matrix of the SR $\bm{M}$ for Agent $i$.
The proposed multi-agent MAK-SR algorithm contains two main components, i.e.,
KTD-based weight SR learning and radial basis function update. For the latter,
we apply the method developed in Section 3 to approximate the vector of basis
functions via representing each of them as a RBF. The gradient of the loss
function (26), with respect to the parameters of the RBFs is calculated using
the chain rule for the mean and covariance of RBFs using (29) and (30).

For KTD-based weight SR learning, the SR can be obtained from its one-step
approximation using the TD method of Eq. (37). In this regard, the state-
action feature vector at time step $k$ can be considered as a noisy
measurement from the system as follows

$\hat{\bm{\phi}}(\bm{s}^{(i)}_{k},a^{(i)}_{k})=\bm{M}^{\text{new}}(\bm{s}^{(i)}_{k},:,a^{(i)}_{k})-\gamma\bm{M}(\bm{s}^{(i)}_{k+1},:,a^{(i)}_{k+1})+\bm{n}^{(i)}_{k},$
(38)

where $\bm{n}^{(i)}_{k}$ follows a zero-mean normal distribution with
covariance of $\bm{R}^{(i)}_{\bm{M}}$. Considering Eqs. (36) and (38)
together, the feature vector $\bm{\phi}(\bm{s}^{(i)}_{k},a^{(i)}_{k})$ can be
approximated as

$\hat{\bm{\phi}}(\bm{s}^{(i)}_{k},a^{(i)}_{k})=\bm{M}_{k}\underbrace{\Big{[}\bm{\phi}(\bm{s}^{(i)}_{k},a^{(i)}_{k})-\gamma\bm{\phi}(\bm{s}^{(i)}_{k+1},a^{(i)}_{k+1})\Big{]}}_{\bm{g}^{(i)}_{k}}+\bm{n}^{(i)}_{k}.$
(39)

s Matrix $\bm{M}_{k}$ is then mapped to a column vector $\bm{m}^{(i)}_{k}$ by
concatenating its columns. Using the vec-trick characteristic of Kronecker
product denoted by $\otimes$, then we can rewrite Eq. (39) as follows

$\displaystyle\hat{\bm{\phi}}(\bm{s}^{(i)}_{k},a^{(i)}_{k})=({\bm{g}^{(i)}}^{T}_{k}\otimes\bm{I})\bm{m}^{(i)}_{k}+\bm{n}^{(i)}_{k},$
(40)

where $\bm{I}$ represents an identity matrix of appropriate dimension. More
specifically, Eq. (40) is used to represent the localized measurements
($\bm{\phi}(\bm{s}^{(i)}_{k},a^{(i)}_{k})$) linearly based on vector
$\bm{m}^{(i)}_{k}$, which requires estimation. Therefore, we use the following
linear state model

$\displaystyle\bm{m}^{(i)}_{k+1}=\bm{m}^{(i)}_{k}+\bm{\mu}^{(i)}_{k},$ (41)

to complete the required state-space representation for KF-based
implementation. The noise associated with the state model (Eq. (41)), i.e.,
$\bm{\mu}^{(i)}_{k}$, follows a zero-mean normal distribution with covariance
of $\bm{Q}_{\bm{M}}$. Via implementing the KF’s recursive equations, we use
the new localized observations to estimate $\bm{m}^{(i)}_{k}$ and its
corresponding covariance matrix $\bm{P}^{(i)}_{\bm{m}^{(i)},k}$. After this
step, vector $\bm{m}^{(i)}_{k}$ is reshaped to form a ($L\times L$) matrix in
order to reconstruct Matrix $\bm{M}_{k}$. Eq. (35) is finally used to form the
state-action value function for associated with
($\bm{s}^{(i)}_{k},a^{(i)}_{k}$). Algorithm 2 summarizes the proposed MAK-SR
framework.

Algorithm 2 The Proposed MAK-SR Framework

1:Learning Phase:

2:Initialize:
$\bm{\theta}_{0},\bm{P}_{\bm{\theta},0},\bm{m}_{0},\bm{P}_{\bm{M},0},{\bm{\mu}_{n},\text{and}\,\bm{\Sigma}_{n}}$
for $n={1,2,\ldots,N}$

3:Parameters:
$\bm{Q}_{\bm{\theta}},\bm{Q}_{\bm{M}},\lambda_{\bm{\mu}},\lambda_{\bm{\Sigma}},\text{and}\,\\{R^{j}_{\bm{\theta}},R^{j}_{\bm{M}}\\}$
for $j={1,2,\ldots,M}$

4:Repeat (for each episode):

5: Initialize $\bm{s}_{k}$

6: Repeat (for each agent $i$):

7: While $\bm{s}_{k}^{(i)}\neq\bm{s}_{T}$ do:

8: Reshape $\bm{m}_{k}$ into $L\times L$ to construct 2-D matrix $\bm{M}_{k}$.

9:
$a_{k}^{(i)}=\arg\max\limits_{a}\Big{(}\bm{g}_{k}^{(i)}(\bm{s}_{k}^{(i)},a){\bm{g}_{k}^{(i)}}^{T}(\bm{s}_{k}^{(i)},a^{(i)})\Big{)}$

10: Take action $a_{k}^{(i)}$ , observe $\bm{s}_{k+1}^{(i)}$ and
$r_{k}^{(i)}$.

11: Calculate $\bm{\phi}(\bm{s}_{k}^{(i)},a_{k}^{(i)})$ via Eqs. (23) and
(25).

12: Update reward weights vector: Perform MMAE to update
$\bm{\theta}_{k}^{(i)}$.

13: Update SR weights vector: Perform KF on Eqs. (40) and (41) to update
$\bm{m}_{k}^{(i)}$.

14: Update RBFs parameters: Perform RGD on the loss function $L_{k}$ to update
$\bm{\Sigma}_{n}$ and $\bm{\mu}_{n}$.

15: end while

It is worth mentioning that, unlike the DNN-based networks for multi-agent
scenarios, the proposed multiple-model frameworks require far less memory due
to their sequential data processing nature. In other words, storing the whole
episodes’ information for all the agents is not needed as the last measured
data (assuming one-step Markov decision process) can be leveraged given the
sequential nature of the incorporated filters. Finally, it should be noted
that the proposed MAK-SR and MAK-TD frameworks are designed for systems with a
finite number of actions. . One direction for future research is to consider
extending the proposed MAK-SR framework to applications where the to action-
space is infinite-dimensional. This might occur in continuous control problems
Li2019 ; Zhang:2021  where number of possible actions at each state is
infinite.

##  5 Experimental Results

Figure 1: Different multi-agent scenarios implemented within the OpenAI gym.

The performances of the proposed MAK-SR and MAK-TD frameworks are evaluated in
this section, where a multi-agent extension of the OpenAI gym benchmark is
utilized. Fig. 1 illustrates snapshots of the environment utilized for
evaluation of the proposed approaches. More specifically, a two-dimensional
world is implemented to simulate competitive, cooperative, and/or mix
interaction scenarios Mordatch2018 . The utilized benchmark is currently one
of the most standard environments to test different multi-agent algorithms,
where time, discrete action space, and continuous observations are the basics
of the environment. Such a multi-agent environment is a natural curriculum in
that the environment difficulty is determined based on the skills of the
agents cooperating or competing. The environment does not have a stable
equilibrium, therefore, allowing the participating agents to become smarter
irrespective of their intelligence level. In each step, the implemented
environment provides observations and rewards once the agents performed their
actions.

In what follows, we discuss different multi-agent environments exploited in
this work as well as the experimental assumptions considered during testing of
the proposed methods. Finally, the results of the experiments will be
represented and explained.

###  5.1 Environments

In the represented multi-agent environments, we do not impose any assumption
or requirement on having identical observations or action spaces for the
agents. Furthermore, agents are not restricted to follow the same policy $\pi$
while playing the game. In the environments, a different number of agents and
possible landmarks can be placed to establish different interactions such as
cooperative, competitive, or mixed strategies. The strategy in each
environment is to keep the agents in the game as long as possible. Each test
can be fully cooperative when agents communicate to maximize a shared return,
or can be fully competitive when the agents compete to achieve different
goals. The mixed scenario for the predator-prey environments (a variant of the
classical predator-prey) is defined in a way that a group of slower agents
must cooperate against another group of faster agents to maximize their
returned reward. Each agent takes a step by choosing one of five available
actions, i.e., no movement, left, right, up, and down, transiting to a new
state, and receiving a reward from the environment. Moreover, each agent will
receive a list of observations in each state, which contains the agent’s
position and velocity, relative positions of landmarks (if available), and its
relative position to other agents in the environment. That is how an agent
knows the position and general status of the agents (friends and adversaries),
enabling the decision making process of that agent. As shown in Fig. 1, each
environment has its own margins. An agent that leaves the area will be
punished by $-50$ points, the game will be reset, and a random configuration
will be initiated to start the next state, which begins immediately. The red
agents play the predator role and receive $+100$ points intercepting (hunting)
a prey (small green agents). The green agents that are faster than red agents
(predators) will receive $-100$ points by each interception with the red ones.
As their job is to follow the prey, the predators will be punished
proportionally to their distance to the prey (green agents). In contrast, the
opposite will happen to the green agents as they keep the maximum distance
from the predators. The proposed MAK-TD/SR frameworks are evaluated against
DQN Mnih2013 , DDPG Lillicrap2015 , and MADDPG Lowe2017 . We evaluate the
algorithms in terms of loss, returned discounted reward, and the number of
collisions between agents.

###  5.2 Experimental Assumptions

Table 1: Total loss averaged across all the episodes and for all the four implemented scenarios. Environment |  MAK-SR |  MAK-TD |  MADDPG |  DDPG |  DQN
---|---|---|---|---|---
Simple Cooperation |  8.93 |  2.4088 |  9649.84 |  10561.16 |  10.93
Simple Competition |  0.43 |  4.9301 |  10158.18 |  10710.37 |  107.39
Predator-Prey 1v2 |  0.005 |  1.9374 |  6816.34 |  6884.33 |  8.21
Predator-Prey 2v1 |  8.87 |  1.2421 |  7390.18 |  6882.2 |  10.24
Table 2: Total received reward by the agents averaged for all the four implemented scenarios. Environment |  MAK-SR |  MAK-TD |  MADDPG |  DDPG |  DQN
---|---|---|---|---|---
Simple Cooperation |  -16.0113 |  -23.0113 |  -69.28 |  -66.29 |  -39.96
Simple Competition |  -0.778 |  -13.358 |  -63.30 |  -61.34 |  -14.49
Predator-Prey 1v2 |  -0.0916 |  -13.432 |  -46.17 |  -20.53 |  -23.451
Predator-Prey 2v1 |  -0.081 |  -17.0058 |  -55.69 |  -49.41 |  -44.32
Table 3: Average steps taken by agents per episode for all the environments based on the implemented platforms. Environment |  MAK-SR |  MAK-TD |  MADDPG |  DDPG |  DQN
---|---|---|---|---|---
Simple Cooperation |  14.03 |  12.064 |  7.377 |  7.369 |  15.142
Simple Competition |  17.59 |  17.48 |  7.36 |  7.18 |  11.98
Predator-Prey 1v2 |  14.78 |  12.36 |  6.21 |  7.69 |  10.02
Predator-Prey 2v1 |  9.94 |  9.773 |  6.25 |  7.12 |  8.46

In the proposed frameworks, we exploit related RBFs based on the different
agents’ sizes of observations and a bias parameter. The size of the
observation vector at each local agent (localized observation vector), which
represents the number of global and local measurements available locally,
varies across different scenarios based on the type and the number of agents
present/active in the environment. Irrespective of size of the localized
observation vectors, the size of the localized feature vectors, which
represents the available five actions, is considered to be $50$. Mean and
covariance of the RBFs are initialized randomly for all the agents in all the
environments. For example, consider a Predator-Prey scenario with $2$ preys
optimizing their actions against one predator. In this toy-example (discussed
for clarification purposes), considering $9$ RBFs together with localized
observation vectors of size $12$ for the predator and $10$ for the preys, the
mean vector associated with the predator and the preys are of dimensions
$9\times 12$ and $9\times 10$, respectively. Consequently, for this Predator-
Prey scenario, $\bm{\mu}$, which is initialized randomly contains three agents
with random values with the mean size $((9,12),(9,10),(9,10))$ and the
covariance, $\bm{\Sigma}=(\bm{I}_{12},\bm{I}_{10},\bm{I}_{10})$ where
$\bm{I}_{12}$ and $\bm{I}_{10}$ are the identity matrices of size ($12\times
12$) and ($10\times 10$), respectively. Based on Eq. (25), the vector of basis
function is represented as follows

$\displaystyle\bm{\phi}(\bm{s}_{k},a_{k}=-2)$ $\displaystyle=$
$\displaystyle[0,\ldots,0,0,\ldots,0,1,\phi_{1,a_{d}},\ldots\phi_{9,a_{d}},0,\ldots,0,0,\ldots,0]^{T},$
(42) $\displaystyle\bm{\phi}(\bm{s}_{k},a_{k}=-1)$ $\displaystyle=$
$\displaystyle[0,\ldots,0,0,\ldots,0,0,\ldots,0,1,\phi_{1,a_{d}},\ldots\phi_{9,a_{d}},0,\ldots,0]^{T},$
(43) $\displaystyle\bm{\phi}(\bm{s}_{k},a_{k}=0)$ $\displaystyle=$
$\displaystyle[0,\ldots,0,0,\ldots,0,0,\ldots,0,0,\ldots,0,1,\phi_{1,a_{d}},\ldots\phi_{9,a_{d}}]^{T},$
(44) $\displaystyle\bm{\phi}(\bm{s}_{k},a_{k}=+1)$ $\displaystyle=$
$\displaystyle[0,\ldots,0,1,\phi_{1,a_{d}},\ldots,\phi_{9,a_{d}},0,\ldots
0,0,\ldots,0,0,\ldots,0]^{T}$ (45)
$\displaystyle\text{and}\quad\quad\bm{\phi}(\bm{s}_{k},a_{k}=+2)$
$\displaystyle=$
$\displaystyle[1,\phi_{1,a_{d}},\ldots,\phi_{9,a_{d}},0,\ldots,0,0,\ldots,0,0,\ldots
0,0,\ldots 0]^{T},$ (46)

where $\phi_{l,a_{d}}$ is calculated based on Eq. (24) for
($l\in\\{1,2,\dots,9\\}$. , $\gamma$, In all the scenarios, the time step
chosen to be $10$ milliseconds and the discount factor is $0.95$. The
transition matrix is initiated to $\bm{F}=\bm{I}_{50}$, and for the process
noise covariance, a small value of $\bm{Q}_{k}=10^{-7}\bm{I}_{50}$ is
considered. The covariance matrix associated with the noise of the measurement
model is selected from the following set

$\displaystyle R^{(i)}\in\\{0.01,0.1,0.5,1,5,10,50,100\\}.$ (47)

For initializing the weights, we sample from a zero mean Gaussian
initialization distribution
$\mathcal{N}(\bm{\theta}_{0},\bm{P}_{\bm{\theta},0})$, where
$\bm{\theta}_{0}=\bm{0}_{50}$ and $\bm{P}_{\bm{\theta},0}=10\bm{I}_{50}$. By
considering the aforementioned initial parameters, each experiment is
initiated randomly and consists of $1,000$ learning episodes together with
$1,000$ test episodes. Given small number of available learning episodes, the
proposed MAK-TD/SR frameworks outperformed their counterparts across different
metrics including sample efficiency, cumulative reward, cumulative steps, and
speed of the value function convergence.

Figure 2: The Predator-Prey environment: (a) Loss. (b) Received rewards.

(a)

(b)

(c)

(d)

Figure 3: Four different normalized loss functions results for all the agents
in the for the four algorithms in four different environments (a) Simple
Cooperation (b) Simple Competition (c) Predator-Prey 2v1, and (d) Predator-
Prey 1v2

(a)

(b)

(c)

(d)

Figure 4: Four different Reward functions results for all the agents in the
for the five algorithms in four different environments (a) Simple Cooperation
(b) Simple Competition (c) Predator-Prey 2v1, and (d) Predator-Prey 1v2

###  5.3 Results

Initially, the agents are trained over different number of episodes, after
which $10$ iteration each of $1,000$ episodes is implemented for testing to
compute different results evaluating performance and efficiency of the
proposed MAK-TD/SR frameworks. First, to evaluate stability of the
incorporated RBFs, a Monte-Carlo (MC) study is conducted where $10$ RBFs are
used across all the environments. The results are averaged over multiple
realizations leveraging MC sampling as shown in Tables 1-3. The results
illustrate the inherent stability of the utilized RBFs and the proposed MAK-TD
and MAK-SR frameworks. Capitalizing on the results of Tables 1-3, the MAK-SR
can be considered as the most sample efficient approach. It is worth noting
that although MAK-SR outperforms the MAK-TD approach, we included both since
the learned representation is not transferrable between optimal policies in
the SR learning. For such scenarios, MAK-TD is an alternative solution
providing, more or less, similar performance to that of the MAK-SR. To be more
precise, when solving a previously unseen MDP, a learned SR representation can
only be used for initialization. In other words, the agents still have to
adjust the SR representation to the policy, which is only optimal within the
existing MDP. This limitation urges us to represent the MAK-TD as another
trusted solution.

As it can be seen from Table 1, the average loss associated with the proposed
MAK-SR is better than that of the MAK-TD. Both frameworks, however, outperform
their counterparts, which can be attributed to their improved sample selection
efficiency. This excellence can also be seen for the Predator-Prey $1v2$
environment in Fig. 2(a). The calculated losses mostly have small values after
the beginning of the experiments, indicating stability of the implemented
frameworks. As can be seen, other approaches cannot provide that level of
performance that is achieved by MAK-SR and MAK-TD with such low number of
training episodes in this experiment. The other three DNN-based approaches can
reach such an efficiency with a much higher amount of experience (more than
$10,000$ experiments) and use much more memory space to save the batches of
the information. Fig. 2(b) shows the rewards gained by all the agents in a
Predator-Prey environment. As can be seen in Table 2 and Fig. 2(b), the
rewards gained in the MAK-SR are also better than those of the MAK-TD and are
much higher than the other approaches. This can be considered exceptional
considering the limited utilized experience. For all other environments this
better performance in the gained reward can be seen in Fig. 4 where four
different reward functions for five discussed algorithms in four experiment
environments are shown. It is worth mentioning that the average number of the
steps taken by all the agents in the defined environments are also represented
in Table 3, showing MAK-SR remarkable results in contrast with the other
algorithms. Results related to the number of the steps are also shown in Fig.
5 for different environments admitting superiority of the MAK-SR framework in
contrast with other solutions. The loss function associated with each of the
five implemented methods is shown in Fig. 3. As expected, the performance of
each model improves over time as being trained through different training
episodes. The proposed MAK-SR and MAK-TD provide exceptional performances
given the small number of training episodes utilized in these experiments.
$MADDPG$, $DDPG$, and $DQN$, however, fail to achieve the same performance
level.

(a)

(b)

(c)

(d)

Figure 5: Number of steps taken in four different environments based on the
five implemented algorithms (a) Simple Cooperation (b) Simple Competition (c)
Predator-Prey 2v1, and (d) Predator-Prey 1v2

##  6 Conclusion

In this paper, an adaptive Kalman filter-based framework, referred to as the
Multi-Agent Adaptive Kalman Temporal Difference (MAK-TD), and its Successor
Representation (SR)-based variant (the MAK-SR) are introduced as an efficient
alternative to Deep Neural Network (DNN)-based Multi-Agent Reinforcement
Learning (MARL) solutions. The main objective is to address sample
inefficiency, memory problems, and lack of prior information issues of DNN-
based MARL techniques. Intuitively speaking, we capitalize on unique
characteristics of Kalman Filtering (KF) such as uncertainty modeling and
online second order learning. The proposed MAK-TD/SR frameworks consider the
continuous nature of the action-space that is associated with high dimensional
multi-agent environments and exploit Kalman Temporal Difference (KTD) to
address the parameter uncertainty. By leveraging the KTD framework, SR
learning procedure is modeled into a filtering problem, where Radial Basis
Function (RBF) estimators are used to encode the continuous space into feature
vectors. On the other hand, for learning localized reward functions, we resort
to Multiple Model Adaptive Estimation (MMAE), as a remedy to deal with the
lack of prior knowledge on observation noise covariance and observation
mapping function. The proposed MAK-TD/SR frameworks are evaluated via several
experiments, which are implemented through the OpenAI Gym multi-agent RL
benchmarks. In these experiments, different number of agents in cooperative,
competitive and mixed (cooperative-competitive) scenarios are utilized. The
experimental results are discussed illustrating superior performance of the
proposed MAK-TD/SR frameworks compared to their counterparts. As a final note,
the MAK-SR framework outperforms the MAK-TD approach, however, as the learned
representation is not transferrable between optimal policies in the SR
learning, MAK-TD provides an alternative solution with, more or less, similar
performance to that of the MAK-SR.

## References

  * (1) P. Malekzadeh, M. Salimibeni, M. Hou, A. Mohammadi, K. N. Plataniotis,  “AKF-SR: Adaptive Kalman Filtering-Based Successor Representation,"  Neurocomputing, vol. 467, no. 7, pp. 476-490, January 2022.
  * (2) R. S. Sutton and A. G. Barto, F. Bach, et al.,  “Reinforcement Learning: An Introduction,"  Cambridge, MA, USA: MIT Press, 2018
  * (3) M. Salimibeni, P. Malekzadeh, A. Mohammadi, P. Spachos and K. N. Plataniotis,  “Makf-Sr: Multi-Agent Adaptive Kalman Filtering-Based Successor Representations,"  IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pp. 8037-8041.
  * (4) P. Malekzadeh, M. Salimibeni, A. Mohammadi, A. Assa and K. N. Plataniotis,  “MM-KTD: Multiple Model Kalman Temporal Differences for Reinforcement Learning,"  IEEE Access, vol. 8, pp. 128716-128729, 2020.
  * (5) S. Spanò, G.C. Cardarilli, L. Di Nunzio, R. Fazzolari, D. Giardino, M. Matta, A. Nannarelli, and M. Re,  “An Efficient Hardware Implementation of Reinforcement Learning: The Q-Learning," Algorithm,"  IEEE Access, vol. 7, pp. 186340-186351, 2019.
  * (6) M. Seo, L.F. Vecchietti, S. Lee, and D. Har,  “Rewards Prediction-Based Credit Assignment for Reinforcement Learning With Sparse Binary Rewards,"  IEEE Access, vol. 7, pp. 118776-118791, 2019.
  * (7) M. Salimibeni, P. Malekzadeh, A. Mohammadi and K. N. Plataniotis,  “Distributed Hybrid Kalman Temporal Differences for Reinforcement Learning,"  IEEE International Asilomar Conference on Signals, Systems, and Computers, 2020.
  * (8) A. Toubman et al.,  “Modeling behavior of Computer Generated Forces with Machine Learning Techniques, the NATO Task Group approach,"  IEEE Int. Con. Systems, Man, and Cyb. (SMC), Budapest, 2016, pp. 001906-001911.
  * (9) J. J. Roessingh et al.,  “Machine Learning Techniques for Autonomous Agents in Military Simulations - Multum in Parvo,"  IEEE Int. Con. Systems, Man, and Cyb. (SMC), Banff, AB, 2017, pp. 3445-3450.
  * (10) G. Jing, H. Bai, J. George and A. Chakrabortty,  “Model-free optimal control of linear multi-agent systems via decomposition and hierarchical approximation,"  IEEE Trans. Control Netw. Syst., 2021.
  * (11) M. Turchetta, A. Krause and S. Trimpe,  “Robust Model-free Reinforcement Learning with Multi-objective Bayesian Optimization,"  IEEE Int. Conf. on Robotics and Automation (ICRA), 2020.
  * (12) Q. Liu, T. Yu, Y. Bai, and C. Jin,  “A sharp analysis of model-based reinforcement learning with self-play,"  arXiv preprint arXiv:2010.01604, 2020.
  * (13) R. Bellman,  “The Theory of Dynamic Programming,"  RAND Corp Santa Monica CA, Tech. Rep., 1954.
  * (14) Y. Song, and W. Sun,  “Pc-mlp: Model-based reinforcement learning with policy cover guided exploration.,"  International Conference on Machine Learning, 2021.
  * (15) E. Vértes,and M. Sahani,  “A Neurally Plausible Model Learns Successor Representations in Partially Observable Environments,"  Advances in Neural Information Processing Systems 32, pp.13714-13724, 2019.
  * (16) S. Blakeman, and D. Mareschal,  “A Complementary Learning Systems Approach to Temporal Difference Learning,"  Neural Networks, vol. 122, 2020, pp. 218-230.
  * (17) M.C. Machado, A. Barreto, and D. Precup,  “Temporal Abstraction in Reinforcement Learning with the Successor Representation,"  ArXiv, 2021, abs/2110.05740.
  * (18) J. P. Geerts, K. L. Stachenfeld, and N. Burgess,  “Probabilistic Successor Representations with Kalman Temporal Differences,"  arXiv preprint arXiv:1910.02532, 2019.
  * (19) M. Riedmiller,  “Neural Fitted Q Iteration-first Experiences with a Data Efficient Neural Reinforcement Learning Method,"  European Conference on Machine Learning, Springer, 2005, pp. 317-328.
  * (20) V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, et al,  “Playing Atari with deep reinforcement learning,"  Technical report.DeepmindTechnologies,arXiv:1312.5602[cs.LG], 2013.
  * (21) R. Lowe, Y. Wu, A. Tamar, J. Harb, P. Abbeel, and I. Mordatch,  “Multi-agent actor-critic for mixed cooperative-competitive environments,"  Annual Conference on Neural Information Processing Systems (NIPS), 2017.
  * (22) T. H. Moskovitz, J. Parker-Holder, A. Pacchiano, M. Arbel, and M. I. Jordan,  “Tactical optimism and pessimism for deep reinforcement learning,"  NeurIPS, 2021.
  * (23) T. Lillicrap, J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra,  “Continuous control with deep reinforcement learning,"  arXiv, preprint arXiv:1509.02971, 2015.
  * (24) C. Ma, J. Wen, and Y. Bengio,  “Universal successor representations for transfer reinforcement learning,"  arXiv preprint, arXiv:1804.03758, 2018.
  * (25) I. Momennejad, E.M. Russek, J.H. Cheong, M.M. Botvinick, N.D. Daw, S.J.Gershman,  “The successor representation in human reinforcement learning,"  Nature Human Behaviour, 1 (9) (2017) 680–692.
  * (26) E.M. Russek, I. Momennejad, M.M. Botvinick, S.J. Gershman, N.D. Daw,  “Predictive representations can link model-based reinforcement learning to model-free mechanisms,"  PLoS Computational Biology, 13 (9) (2017) e1005768.
  * (27) S.C. Chan, S. Fishman, J. Canny, et al.,  “Measuring the reliability of reinforcement learning algorithms,"  International Conference on Learning Representations, 2020.
  * (28) Y. Tang, H. Guo,T. Yuan, X. Gao, X. Hong, Y. Li, J. Qiu, Y. Zuo, and J. Wu,  “Flow Splitter: A Deep Reinforcement Learning-Based Flow Scheduler for Hybrid Optical-Electrical Data Center Network,"  IEEE Access, vol. 7, pp.129955-129965, 2019.
  * (29) C. Hu, and M. Xu,  “Adaptive Exploration Strategy With Multi-Attribute Decision-Making for Reinforcement Learning,"  IEEE Access, vol. 8, pp. 32353-32364, 2020.
  * (30) M. Kim, S. Lee, J. Lim, J. Choi, and S.G. Kang,  “Unexpected Collision Avoidance Driving Strategy Using Deep Reinforcement Learning,"  IEEE Access, vol. 8, pp. 17243-17252, 2020.
  * (31) J. Xie, Z. Shao, Y. Li, Y. Guan, and J. Tan,  “Deep reinforcement learning with optimized reward functions for robotic trajectory planning,"  IEEE Access, vol. 7, pp. 105669-105679, 2019.
  * (32) J.N. Tsitsiklis, B. Van Roy,  “An analysis of temporal-difference learning with function approximation,"  IEEE Transactions on Automatic Control, 42 (5) (May 1997) 674–690.
  * (33) D.P. Bertsekas, V.S. Borkar, A. Nedic,  “Improved temporal difference methods with linear function approximation, Learning and Approximate Dynamic Programming,"  2004, pp. 231–255.
  * (34) H. Van Hasselt, A. Guez, and D. Silver, “Deep Reinforcement Learning with Double Q-learning."  AAAI, vol. 2. Phoenix, AZ, 2016, p. 5.
  * (35) S. Haykin,  “Neural Networks: A Comprehensive Foundation,”  Prentice Hall PTR, 1994.
  * (36) W. T. Miller, F. H. Glanz, and L. G. Kraft,  “Cmas: An Associative Neural Network Alternative to Backpropagation,"  Proceedings of the IEEE, vol. 78, no. 10, pp. 1561-1567, 1990.
  * (37) G.S. Babu, S. and Suresh,  “Meta-cognitive neural network for classification problems in a sequential learning framework,  Neurocomputing, 81, pp.86-96, 2012.
  * (38) A. d. M. S. Barreto and C. W. Anderson,  “Restricted Gradient-descent Algorithm for Value-function Approximation in Reinforcement Learning,"  Artificial Intelligence, vol. 172, no. 4-5, pp. 454-482, 2008.
  * (39) I. Menache, S. Mannor, and N. Shimkin,  “Basis Function Adaptation in Temporal Difference Reinforcement Learning,"  Annals of Operations Research, vol. 134, no. 1, pp. 215-238, 2005.
  * (40) D. Choi and B. Van Roy,  “A generalized Kalman filter for Fixed Point Approximation and Efficient Temporal-difference Learning,"  Discrete Event Dynamic Systems, vol. 16, no. 2, pp. 207-239, 2006.
  * (41) Y. Engel,  “Algorithms and Representations for Reinforcement Learning,”  Hebrew University of Jerusalem, 2005.
  * (42) S. J. Bradtke and A. G. Barto,  “Linear Least-squares Algorithms for Temporal Difference Learning,"  Machine Learning, vol. 22, no. 1-3, pp. 33-57, 1996.
  * (43) E.Z. Liu, A. Raghunathan, P. Liang, C. Finn,  “ Decoupling exploration and exploitation for meta-reinforcement learning without sacrifices,"  International Conference on Machine Learning, 2021, PMLR. pp. 6925–6935.
  * (44) M. Geist and O. Pietquin,  “Algorithmic Survey of Parametric Value Function Approximation,"  IEEE Transactions on Neural Networks and Learning Systems, vol. 24, no. 6, pp. 845-867, 2013.
  * (45) M. Geist and O. Pietquin,  “Kalman Temporal Differences,"  Journal of Artificial Intelligence Research, vol. 39, pp. 483-532, 2010.
  * (46) R. Mehra,  “On the Identification of Variances and Adaptive Kalman Filtering,"  IEEE Transactions on Automatic Control, vol. 15, no. 2, pp. 175-184, 1970.
  * (47) A. Mohammadi and K. N. Plataniotis,  “Distributed Widely Linear Multiple-Model Adaptive Estimation,"  IEEE Transactions on Signal and Information Processing over Networks, vol. 1, no. 3, pp. 164-179, 2015.
  * (48) C. Yang, A. Mohammadi, Q-W. Chen,  “ Multi-Sensor Fusion with Interaction Multiple Model and Chi-Square Test Tolerant Filter,"  Sensors, vol. 16, no. 11, 1835, 2016.
  * (49) A. Mohammadi and K. N. Plataniotis,  “Improper Complex-Valued Multiple-Model Adaptive Estimation,"  IEEE Transactions on Signal Processing, vol. 63, no. 6, pp. 1528-1542, 2015.
  * (50) A. Assa and K. N. Plataniotis,  “Similarity-based Multiple Model Adaptive Estimation,"  IEEE Access, vol. 6, pp. 36 632-36 644, 2018.
  * (51) T. Kitao, M. Shirai, and T. Miura, “Model Selection based on Kalman Temporal Differences Learning,"  IEEE International Conference on Collaboration and Internet Computing (CIC), 2017, pp. 41-47.
  * (52) R. S. Sutton, A. G. Barto, F. Bach et al.,  “Reinforcement Learning: An Introduction,”  MIT Press, 1998.
  * (53) M. Hutter and S. Legg,  “Temporal Difference Updating without a Learning Rate,"  Advances in Neural Information Processing Systems, 2008, pp. 705-712.
  * (54) R. S. Sutton,  “Generalization in Reinforcement Learning: Successful Examples using Sparse Coarse Coding,"  Advances in Neural Information Processing Systems, 1996, pp. 1038-1044.
  * (55) A. Singh, T. Jain, and S. Sukhbaatar,  “Learning when to communicate at scale in multiagent cooperative and competitive tasks,“  In ICLR, 2019.
  * (56) I. Mordatch, P. Abbeel,  “Emergence of grounded compositional language in multi-agent populations.“  Proc. AAAI Conference of Artificial Intelligence, 2018.
  * (57) W. Xia, C. Di, H. Guo, and S. Li,  “Reinforcement Learning Based Stochastic Shortest Path Finding in Wireless Sensor Networks,"  IEEE Access, vol. 7, pp.157807-157817, 2019.
  * (58) J. Li, T. Chai, F. L. Lewis, Z. Ding and Y. Jiang,  “Off-Policy Interleaved $Q$ -Learning: Optimal Control for Affine Nonlinear Discrete-Time Systems,"  IEEE Transactions on Neural Networks and Learning Systems,, vol. 30, no. 5, pp. 1308-1320, May 2019.
  * (59) K. Zhang, Z. Yang, and T. Başar,  “Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithms,"  , https://arxiv.org/abs/1911.10635, springer, 2021.
  * (60) C. J. Watkins and P. Dayan,  “Q-learning,"  Machine Learning, vol. 8, no. 3-4, pp. 279-292, 1992.
  * (61) Y. Ge, F. Zhu, X. Ling, and Q. Liu,  “Safe Q-Learning Method Based on Constrained Markov Decision Processes,"  IEEE Access, vol. 7, pp. 165007-165017, 2019.
  * (62) A. Mohammadi and K. N. Plataniotis,  “Event-Based Estimation With Information-Based Triggering and Adaptive Update,"  IEEE Transactions on Signal Processing, vol. 65, no. 18, pp. 4924-4939, 15 Sept. 2017.