#  cammarl: Conformal Action Modeling in
Multi Agent Reinforcement Learning

Nikunj Gupta  Somjit Nath  Samira Ebrahimi Kahou

###### Abstract

Before taking actions in an environment with more than one intelligent agent,
an autonomous agent may benefit from reasoning about the other agents and
utilizing a notion of a guarantee or confidence about the behavior of the
system. In this article, we propose a novel multi-agent reinforcement learning
(MARL) algorithm cammarl, which involves modeling the actions of other agents
in different situations in the form of confident sets, i.e., sets containing
their true actions with a high probability. We then use these estimates to
inform an agent’s decision-making. For estimating such sets, we use the
concept of conformal predictions, by means of which, we not only obtain an
estimate of the most probable outcome but get to quantify the operable
uncertainty as well. For instance, we can predict a set that provably covers
the true predictions with high probabilities (e.g., 95%). Through several
experiments in two fully cooperative multi-agent tasks, we show that cammarl
elevates the capabilities of an autonomous agent in MARL by modeling conformal
prediction sets over the behavior of other agents in the environment and
utilizing such estimates to enhance its policy learning. Implementation can be
found here: https://github.com/Nikunj-Gupta/conformal-agent-modelling.

Machine Learning, ICML

##  1 Introduction

Developing systems of autonomous agents capable of effective multi-agent
interactions can be very useful in modern cooperative artificial intelligence
(AI). For instance, service robots, surveillance agents, and many more similar
applications require profound collaboration among agents (and with humans),
without prior coordination. Now, to enable complex, constructive behaviors to
emerge from unsupervised interactions among agents, an essential skill for an
agent to have is the ability to reason about other agents in the environment.
There has been considerable research addressing this problem of agent or
opponent modeling (Albrecht & Stone, 2018). Generally, it involves
constructing models of other agents that learn useful attributes to inform its
own decision-making (such as the future actions of the other agents, or their
current goals and plans) from current or past interaction history (such as the
previous actions taken by other agents in different situations).

Figure 1: Our proposed methodology of informing an autonomous agent’s
decision-making by means of conformal predictions of action sets of other
agents in the environment illustrated with two agents for simplicity. Two
agents ($\mathcal{N}_{self}$, $\mathcal{N}_{other}$) receive their own partial
observations from the environment ($o_{self}$, $o_{other}$) and take their
actions ($a_{self}$, $a_{other}$). An independent conformal action prediction
model $\mathcal{C}$ learns to output a conformal action set,
$\\{a^{\prime}_{other}\\}$, corresponding to $\mathcal{N}_{other}$ which are
then used as additional inputs for training by $\mathcal{N}_{self}$ to inform
its policy and perform its action $a_{self}$.

We are interested in the particular aspect of an interactive, autonomous agent
that involves learning an additional, independent model to make predictions
about the actions of the other agents in the environment, supplemental to its
reinforcement learning-based policy to make decisions related to its
downstream task. An autonomous agent can then incorporate those estimates to
inform its decision-making and optimize its interaction with the other agents.
While there exist several methods for developing such models for other agents
(Albrecht & Stone, 2018), there is currently no method or theory to the best
of our knowledge that would allow an agent to consider the correctness or
confidence of the predictions of the learned model.

#### Conformal Predictions.

Conformal predictions or inference is a fitting method for generating
statistically accurate uncertainty sets for the predictions from machine
learning classifiers. It is steadily gaining popularity owing to its explicit
and non-asymptotic guarantees over the produced sets (Angelopoulos & Bates,
2021). In other words, we can obtain conformal sets that provably contain the
true predictions with high probabilities, such as 95%, chosen in advance. This
can be very useful and successful in high-risk learning settings, especially
in decision-making in medical applications from diagnostic information, for
instance, which demand quantifying uncertainties to avoid insufferable model
failures. What if we only prefer to use the predictions when the model is
confident? For example, doctors may only consider a predicted medical
diagnosis when the model is at least 95% accurate, or may want to use the
predicted set with high credence to consider ruling out relevant
possibilities. So, in this article, we aim to enhance the capabilities of an
agent in a multi-agent reinforcement learning (MARL) setting by modeling and
using conformal prediction sets (or the latent representations learned in the
process111More details in Appendix B.) over the behavior of an autonomous
system. In particular, we model other agents’ actions in the form of confident
sets, i.e., sets that contain other agents’ true actions with a high
probability. We hypothesize that these estimated conformal sets would inform
our learning agent’s decision-making and elevate its performance in MARL.
Figure 1 shows the high-level idea of our proposed model for learning agents
in any given environment.

In this work, we aim to introduce a novel framework to train an autonomous
agent that enhances its decision-making by modeling and predicting confident
conformal actions of other agents in the environment — the cammarl algorithm
(Section 3), and then empirically demonstrate that conformal action modeling
used in cammarl indeed can help make significant improvements in cooperative
policies learned by reinforcement learning agents in two multi-agent domains
(Section 4).

##  2 Related Works

Decision-making without reasoning about other agents in the environment can be
very challenging, for instance, due to weak or no theoretical guarantees, non-
stationarity (single agent’s perspective), and inefficient coordination for a
considerable coherent joint behavior (Matignon et al., 2012). Modeling other
agents in an environment is not new and has been studied in the past (Albrecht
& Stone, 2018; Albrecht et al., 2020). However, our proposal of predicting
conformal sets of actions of the other agents in the environment (with high
probability) is novel and has not been attempted to the best of our knowledge.

#### Learning world models.

Model-based reinforcement learning (MBRL) has certainly shown its advantages
in data efficiency, generalization, exploration, counterfactual reasoning, and
performance in many tasks and domains (Hafner et al., 2020, 2021; Jain et al.,
2022; Moerland et al., 2023; Pal & Leon, 2020; Polydoros & Nalpantidis, 2017)
in single-agent RL, and now, it has also started to attract attention in MARL
(Wang et al., 2022). However, most of the current works in model-based MARL do
not yet focus on teammate or opponent modeling. Some recent works (Park et
al., 2019b; Zhang et al., 2021) incorporated dynamics modeling and a
prediction module to estimate the actions of other agents within the
construction of the environment model. However, these prediction models were
trained without accessing the true trajectories from the other agents which
can be problematic in several use cases.

#### Learning agent models.

A widely popular technique to reason about other agents in the environment is
to learn representations of different properties of other agents. For
instance, learning to reconstruct the actions of other agents from their
partial observations (He et al., 2016; Mealing & Shapiro, 2015; Panella &
Gmytrasiewicz, 2017; Albrecht & Ramamoorthy, 2015), modeling an agent or its
policy using encoder-decoder-based architectures (Grover et al., 2018;
Zintgraf et al., 2021), learning latent representations from local information
with or without utilizing the modeled agent’s trajectories (Papoudakis et al.,
2021; Xie et al., 2021) or modeling the forward dynamics of the system through
relational reasoning using graph neural networks (Tacchetti et al., 2018).
Theory-of-Mind Network or TomNet learned embeddings corresponding to other
agents in the environment for meta-learning (Rabinowitz et al., 2018). Some
works also constructed I-POMDPs to utilize recursive reasoning (Albrecht &
Stone, 2018) assuming unrestricted knowledge of the observation models of
other agents. Nevertheless, cammarl involves no form of reconstruction of
other agent’s policy or rewards, or state models. Any of these techniques can
be used with cammarl which, however, is not the objective of this work. Also,
unlike cammarl, many of these aforementioned techniques evaluate in fully-
observable environments or rely upon direct access to other agents’ experience
trajectories even during execution. This can be infeasible in various
settings.

#### Multi-agent reinforcement learning (MARL).

Numerous deep MARL research works that focus on partial observability in fully
cooperative settings indirectly involve reasoning about the intentions of
teammates or opponents in an environment (Gronauer & Diepold, 2022). For
instance, many works allow agents to communicate, enabling them to indirectly
reason about the others’ intentions (Lazaridou et al., 2016; Foerster et al.,
2016; Sukhbaatar et al., 2016; Das et al., 2017; Mordatch & Abbeel, 2018;
Gupta et al., 2021; Zhu et al., 2022). On the other hand, some studied the
emergence of cooperative and competitive behaviors among agents in varying
environmental factors, for instance, task types or reward structures (Leibo et
al., 2017). Recent work in hierarchical reinforcement learning also attempts
to develop a hierarchical model to enable agents to strategically decide
whether to cooperate or compete with others in the environment and then
execute respective planning programs (Kleiman-Weiner et al., 2016). However,
none of these works study the improvement in an autonomous agent’s decision-
making via directly modeling the other agents in the environment or predicting
their actions or current or future intentions.

#### Inverse reinforcement learning (IRL).

Research in the field of IRL also relates to our work because we share the key
motive of inferring other agents’ intentions and then use it to learn a policy
that maximizes the utility of our learning agent (Arora & Doshi, 2021).
However, IRL addresses this by deducing the reward functions of other agents
based on their behavior, assuming it to be nearly optimal. On the other hand,
in cammarl we directly model the other agent’s actions based on their
observations and use these estimates to indirectly infer their goal in an
online manner.

#### Conformal prediction.

Estimating well-grounded uncertainty in predictions is a difficult and
unsolved problem and there have been numerous approaches for approximating it
in research in supervised learning (Gawlikowski et al., 2021). Recent works in
conformal predictions (Angelopoulos et al., 2020; Lei et al., 2018;
Hechtlinger et al., 2018; Park et al., 2019a; Cauchois et al., 2020; Messoudi
et al., 2020) have now significantly improved upon some of the early research
(Vovk et al., 2005; Platt et al., 1999; Papadopoulos et al., 2002), for
instance in terms of predicted set sizes, improved efficiency, and providing
formal guarantees. For this article, we adapt the core ideas from Regularized
Adaptive Prediction Sets (RAPS) (Angelopoulos et al., 2020) to our setting
owing to its demonstrated improved performance evaluation on classification
benchmarks in supervised learning (Angelopoulos et al., 2020).

##  3 The cammarl Algorithm

###  3.1 Mathematical Model

Formally, we consider two agents in the environment — learning agent denoted
by self and the other agent denoted by other. The partially observable Markov
game (Littman, 1994) for our setting can then be defined using the following
tuple222A tabular version can be found in Section C.1:

$\langle\mathcal{N}_{i},\mathcal{S},\mathcal{A}_{i},\mathcal{O}_{i},\mathcal{T},\mathcal{C},\pi_{\theta_{i}},r_{i}\rangle_{i\in\\{self,other_{1}\dots
other_{K-1}\\}}$

With the set $\mathcal{S}$ describing the possible true states (or full
observations) of the environment, $K$ agents, $\mathcal{N}_{self}$ and $(K-1)$
$\mathcal{N}_{other_{j}}$ ($j\in[1,K]$), observe the environment locally using
their sets of observations $\mathcal{O}_{self}$ and $(K-1)$
$\mathcal{O}_{other_{j}}$ respectively, and act using their set of actions,
$\mathcal{A}_{self}$ and $(K-1)$ $\mathcal{A}_{other_{j}}$. Each agent i can
select an action $a_{i}\in\mathcal{A}_{i}$ using their policy
$\pi_{\theta_{i}}$, and their joint action
$\textbf{a}\in\mathcal{A}_{self}\times\mathcal{A}_{other_{1}}\times\dots\mathcal{A}_{other_{K}}$
then imposes a transition to the next state in the environment according to
the state transition function $\mathcal{T}$, defined as a probability
distribution on the subsequent state based on current state and actions,
$\mathcal{T}$:
$\mathcal{S}\times\mathcal{A}_{self}\times\mathcal{A}_{other_{1}}\times\dots\mathcal{A}_{other_{K}}\times\mathcal{S}\rightarrow[0,1]$.
The agents use their individual reward function $r_{i}(s,a)$ :
$\mathcal{O}_{i}\times\mathcal{A}_{i}\rightarrow\mathbb{R}$. Both agents aim
to maximize their own total expected rewards
$R_{i}=\sum_{t=0}^{T}\gamma^{t}r_{i}^{t}$ where $\gamma\in[0,1)$ as the
discount factor and T is the time horizon.

In cammarl, at each time step t, we also use a conformal prediction model for
the $j$-th agent, defined as a set-valued function,
$\mathcal{C}:\mathbb{R}^{d}\rightarrow 2^{|\mathcal{A}_{other_{j}}|}$

$\mathcal{C}(o^{t}_{other_{j}})\rightarrow\\{A^{i}_{other_{j}}\\}$

which outputs a conformal action predictive set $\\{A^{t}_{other_{j}}\\}$ for
each input of $\mathcal{N}_{other_{j}}$’s local observation
$o^{t}_{other_{j}}\in\mathcal{O}_{other_{j}}$ at the time.

###  3.2 Conformal Action Modeling

Figure 2: A detailed illustration of conformal action modelling and inference
in cammarl to generate prediction sets of $\mathcal{N}_{other}$’s actions
using conformal predictors. Algorithm 1 Conformal action modeling in marl

[1] $N_{self}$, $N_{other_{j}}$ $\leftarrow$ Initialize Actor-Critic networks
for — $\mathcal{N}_{self}$ and $\mathcal{N}_{other_{j}}$, where $j\in[1,K]$
(K-1) conformalModels $\leftarrow$ Initialize the conformal model to predict
conformal action sets

$\texttt{episode}=1,2,\ldots$

Fetch observations $o_{self}$, $o_{other_{1}}\dots o_{other_{K}}$ from
environment

$\texttt{timesteps}=1,2,\ldots,T$

$\texttt{conformalActions}\leftarrow\texttt{conformalModels}(o_{other_{j}})$;
for $j\in[1,K]$ Predict conformal action set $o_{self}$ $\leftarrow$
$o_{self}$ \+ conformalActions Concatenate conformal actions to $o_{self}$ Run
agent policies in the environment

Collect trajectories of $\mathcal{N}_{self}$ and
$\mathcal{N}_{other_{1}}\dots\mathcal{N}_{other_{K}}$

update interval reached Train conformalModels using
$\mathcal{N}_{other_{j}}$’s state-action mappings; for $j\in[1,K]$ Train
$N_{self}$ using PPO Train $N_{other_{j}}$ using PPO; for $j\in[1,K]$

Now we formally describe our proposed algorithm — Conformal Action Modeling-
based Multi-Agent Reinforcement Learning or cammarl. Our objective is to
inform an $\mathcal{N}_{self}$’s decision-making by modeling the other agent’s
actions in the environment as conformal prediction sets that contain the true
actions with a high probability (for example, 95%). More specifically,
$\mathcal{N}_{self}$ uses a separate conformal action prediction model to
obtain sets of $\mathcal{N}_{other_{j}}$’s actions at each timestep that
contains latter’s true action in the environment at a given time step with
high prespecified probabilities.

Algorithm 1 describes the complete workflow of training of agents in cammarl.
We begin by initializing the actor-critic networks for both the agents in the
environment, the conformal model, and the memory buffers for each of these.
Now, at the beginning of each episode in the environment, both agents receive
their own partial observations (line 1). Next, the conformal model predicts
the actions of all the $\mathcal{N}_{other_{j}}$’s in the form of a set, which
is then provided as an additional input to $\mathcal{N}_{self}$ (lines 1-1),
whereas $\mathcal{N}_{other_{j}}$ has access to only its own partial
observation, $o_{other_{j}}$. Both agents now take actions in the environment
and continue collecting their experiences (lines 1-1). The agents and the
conformal model periodically train using their respective experience memory
(lines 1-1).

Figure 2 shows a detailed illustration of our conformal action modeling that
materializes internally at each time step. We use only one other agent
$\mathcal{N}_{other}$ for simplicity. The conformal predictor $\mathcal{C}$
collects $\mathcal{N}_{other}$’s state-action pairs and periodically learns
and updates a neural network classifier,
$f(\cdot):\mathbb{R}^{d}\rightarrow\mathbb{R}^{|\mathcal{A}_{other}|}$ (where
d is the number of dimensions in $\mathcal{N}_{other}$’s local observation and
$|\mathcal{A}_{other}|$ is the number of possible discrete actions available
for $\mathcal{N}_{other}$), to predict action from a given state. Then, we
adapt RAPS conformal calibration (Angelopoulos et al., 2020) to our setting.
Considering $\textbf{o}\in O_{other}$ as feature vectors, we use the updated f
to compute action probabilities
$\hat{\pi}_{o}\in\mathbbm{R}^{|\mathcal{A}_{other}|}$. The probabilities are
then ordered from most probable to least probable followed by the estimation
of the predictive action set for the given feature inputs. To promote small
predictive sets we also add a regularization term as also proposed in RAPS.
Formally, for a feature vector o and the corresponding possible prediction a,
to estimate a set which includes all the actions that will be included before
a, let us define the total probability mass of the set of actions that are
more probable than a:

$\rho_{o}(a)=\sum^{|\mathcal{A}_{other}|}_{a^{\prime}=1}\hat{\pi}_{o}(a^{\prime})\mathbbm{1}_{\\{\hat{\pi}_{o}(a^{\prime})\geq\hat{\pi}_{o}(a)\\}}$

Also, if we define a function to rank the possible action outcomes based on
their probabilities $\hat{\pi}$ as

$z_{o}(a)=|\\{a^{\prime}\in\mathcal{A}_{other}:\\{\hat{\pi}_{o}(a^{\prime})\geq\hat{\pi}_{o}(a)\\}\\}|$

we can then estimate a predictive action set as follows:

$\mathcal{C}^{*}(\textbf{o}):=\bigl{\\{}\textbf{a}:\rho_{\textbf{o}}(\textbf{a})+\hat{\pi}_{\textbf{o}}(\textbf{a})\cdot
u+\lambda\cdot(z_{\textbf{o}}(\textbf{a})-k_{reg})^{+}\leq\tau\bigr{\\}}$

where $(x)^{+}$ denotes the positive portion of x, and $\lambda,k_{reg}\geq 0$
are regularization hyperparameters to incentivize small set sizes. Here,
$u\sim uniform$ [0, 1] (used to allow for randomized procedures) and the
tuning parameter $\tau$ (the cumulative sum of the classifier scores after
sorting and penalization) to control the size of the sets are identical to as
used in RAPS for supervised tasks (Angelopoulos et al., 2020).

To summarize, in cammarl, $\mathcal{N}_{self}$ gets to use estimates of
$\mathcal{N}_{other}$’s actions at each time step to make informed decisions
in the environment. Instead of modeling exact actions with no uncertainty
estimation, we prefer to produce an action set carrying desirable guarantees
of containing $N_{other}$’s true action with high probability, integrate it
into an agent’s downstream task, and enable improved decision-making and
collaboration with $\mathcal{N}_{other}$.

##  4 Experiments

In this section, we discuss the cooperative tasks with multiple agents used in
this study. We note here that though we work in fully cooperative settings in
this article, cammarl as an idea can be generalized to competitive or mixed
settings too (more on this in Appendix).

###  4.1 Environments

We focus on four cooperative multi-agent environments: Cooperative Navigation
The agents learn to visit the two landmarks avoiding collisions. In Level-
based Foraging, agents collect foods based on their levels. In Pressure Plate,
the agents traverse a grid world and must cooperate by standing on a plate to
keep a gate open while the other agents collect the main reward. Finally, we
also run cammarlin Google Football, where 3 agents try to score a goal against
a defender and a keeper in a game of football. Further details on these
environments are in Section A.

###  4.2 Results

Figure 3: Comparison of agent performances (in terms of reward accumulation)
in CN (a) and LBF (b) in different settings with varying pieces of information
available to $\mathcal{N}_{self}$ during training. cammarl’s performance is
very close to the upper bound, GIAM, and is considerably better than the other
extreme, NOAM. It also outperforms the other defined benchmarks (TAAM, TOAM, &
EAP) in both tasks, along with the benefit of uncertainty quantification of
its estimates. Interestingly, in CN, cammarl can be seen to learn arguably
faster, but all methods converge to similar results, whereas in LBF, it
actually seems to converge to a better policy. The curves are averaged over
five independent trials and smoothed using a moving window average (100
points) for readability.

To show the benefits of conformal action set prediction, we compare cammarl
with the performances of agents in different settings with varying pieces of
information made available to $\mathcal{N}_{self}$ during training.

#### No-Other-Agent-Modeling (NOAM).

At first, we train $\mathcal{N}_{self}$ without allowing it to model
$\mathcal{N}_{other}$. This baseline, as expected, underperforms when compared
to any other settings (where any kind of agent modeling is allowed). It is
indicative of a lower bound to the learning performance of our model where no
kind of benefit from agent modeling is utilized by $\mathcal{N}_{self}$.
Results are shown in Figure 3. We call this baseline — No-Other-Agent-Modeling
or NOAM.

#### True-Action-Agent-Modeling (TAAM).

Advancing from the inputs available in NOAM, we implement TAAM by allowing
$\mathcal{N}_{self}$ to additionally utilize $\mathcal{N}_{other}$’s true
actions to train. This baseline helps us evaluate cammarl against works that
estimate other agents’ actions in the environment and use those predictions to
enhance the decision-making of their controlled autonomous agents. By giving
the true actions as inputs, this baseline can act as an upper bound to such
works(He et al., 2016; Grover et al., 2018; Zintgraf et al., 2021; Mealing &
Shapiro, 2015; Panella & Gmytrasiewicz, 2017; Albrecht & Ramamoorthy, 2015).
Figure 3 shows TAAM’s performance curve. Using additional information, it does
reasonably better than NOAM in both tasks.

#### True-Observation-Agent-Modeling (TOAM).

As discussed in Section 2, learning world models often involves reconstructing
observation as an additional task while learning task-related policies (Jain
et al., 2022; Hafner et al., 2020, 2021). Inspired by this research, we
implement the TOAM baseline where we allow access to $\mathcal{N}_{other}$’s
true observations to $\mathcal{N}_{self}$ during training and execution. In
other words, we augment $\mathcal{N}_{self}$’s partial observations with the
other agent’s local observations too. This baseline can act as an upper bound
to the performances of research works that learn to reconstruct states for
agents (Hafner et al., 2020, 2021; Wang et al., 2022; Park et al., 2019b;
Zhang et al., 2021). As expected, $\mathcal{N}_{self}$ performs considerably
better than NOAM in both environments (Figure 3). Here, the difference in
returns in TOAM and TAAM can be attributed to the fact that the local
observations of other agents include more information that can be useful to
infer their behavior. For instance, in CN, knowing the relative positions of
other agents with respect to the landmarks can be more useful to infer which
landmark that agent might be approaching when compared to knowing its current
(or history) actions.

#### Global-Information-Agent-Modeling (GIAM).

On the other extreme, we also implement GIAM, where $\mathcal{N}_{self}$
trains with complete access to both (1) $\mathcal{N}_{other}$’s true action
trajectories ($a_{other}$), and (2) $\mathcal{N}_{other}$’s true observations
($o_{other}$) as additional information. Figure 3 shows that GIAM achieves
higher returns compared to all other settings in both environments. This is
intuitive because it benefits from more information. GIAM is conditioned on
$\mathcal{N}_{other}$’s true experiences and consequently demands access to
them even during execution. This can be infeasible in real-world scenarios,
however, theoretically represents an upper bound on the performance of agents
in cammarl and other settings.

#### Exact-Action-Prediction (EAP).

Building over the inputs of TOAM, we construct a stronger baseline, EAP, in
which $\mathcal{N}_{self}$ uses an additional neural network classifier to
model a probability distribution over $\mathcal{N}_{other}$’s actions. In
other words, instead of predicting conformal sets of actions (like in
cammarl), in this baseline, $\mathcal{N}_{self}$ tries to model
$\mathcal{N}_{other}$’s actions from the latter’s observations without
accounting for any uncertainty quantification. This baseline is inspired by
works that explicitly model the other agent’s actions in the environments and
utilize them to inform their controlled agent’s decision-making (for instance,
(He et al., 2016; Grover et al., 2018; Zintgraf et al., 2021)). Hence, here, a
cross-entropy loss is used to train the added sub-module that predicts the
$\mathcal{N}_{other}$’s actions along with a PPO loss to train
$\mathcal{N}_{self}$’s policy network. Figure 3 shows that cammarl agents are
able to distinctly perform better than EAP in LBF, however, interestingly, the
performance curve for this baseline nearly overlaps with cammarl in CN. Also,
in LBF, the curves for TOAM and EAP seem to significantly overlap. We
speculate that in a complicated task like LBF, estimating the exact action of
$\mathcal{N}_{other}$ can be difficult, and with unaccounted uncertainty in
the predictions, $\mathcal{N}_{self}$ suffers from a lower return. In CN,
which is comparatively simpler, the closeness of returns in EAP and cammarl
seem reasonable as even the conformal model predictions eventually start
predicting the most probable action with higher probabilities and hence a set
of size one (more on this in Section 6).

####  cammarl.

Now, we implement cammarl, where the conformal action prediction model
periodically trains on collected observations of $\mathcal{N}_{other}$ and
predicts a corresponding conformal set of actions. $\mathcal{N}_{self}$ uses
these estimates of $\mathcal{N}_{other}$’s actions along with its own
observations and then decides upon its actions in the environment. Figure 3
shows that cammarl agents obtain returns that are much closer to the upper
bound, GIAM, than the lower bound, NOAM. Furthermore, cammarl’s better
performance compared to TOAM in both environments can be attributed to the
fact that it can be difficult to predict the $\mathcal{N}_{other}$’s
intentions by only using $o_{other}$ without any information pertaining to its
actions in those situations. And, in TAAM, $\mathcal{N}_{self}$ is expected to
implicitly encode information regarding $\mathcal{N}_{other}$’s observations
from its own local observations or in the latent space and map it to
$\mathcal{N}_{other}$’s true actions. We speculate that this could be a strong
assumption and consequently very difficult, hence, cammarl agents outperform
TAAM too. Note here that the sets output by the conformal action prediction
model are of varying sizes in each iteration. Now, to be able to use these
dynamically changing inputs for $\mathcal{N}_{self}$ in cammarl, we convert
the output sets to a corresponding binary encoding (by firing up the bits in a
zero vector at indices corresponding to the actions predicted by the model).
We discuss some more ways to be able to use conformal prediction sets with
dynamic sizes and compare cammarl’s performances in all variations later in
the supplementary.

In summary, through experiments in two complex cooperative tasks, we show that
(1) cammarl indeed works, (2) it outperforms common settings like NOAM, TOAM,
and TAAM which assume the availability of other agents’ true trajectories
during training and execution (generally infeasible in real-world scenarios),
(3) Its performance is closest to our upper bound of performance (GIAM), (4)
cammarl agents learn their policies faster than the other baselines, and (5)
cammarl can be preferred over strong benchmarks such as EAP owing to its
higher interpretability due to the theoretical guarantees of conformal
predictions in terms of coverage (Angelopoulos et al., 2020) (discussed more
in Section 6).

###  4.3 Results with more than 2 agents

Figure 4: Comparison of agent performances (in terms of reward accumulation)
in environments with more than 2 agents: (a) Pressure Plate and (b) Google
Football. Interestingly, Pressure Plate cammarl can be seen to learn arguably
faster, but all methods converge to similar results, whereas in Google
Football, cammarl reaches a higher reward than the baselines. In Google
Football, the observations are global, so we did not include TOAM and GIAM.
The curves are averaged over five independent runs.

cammarl can be extended to problems with more than 2 agents, where the main
agent uses a conformal model for each of the other agents. From Figure 4 (a)
we see that although all the algorithms ultimately converge to the optimal
reward, cammarl is more sample efficient and converges in roughly 10% fewer
episodes compared to GIAM and TOAM. In Google Football (Figure 4 (b)) we
notice a considerable improvement as well, with conformal predictions helping
in much better reward accumulation. These experiments also highlight the
benefit of using conformal predictions, as TAAM with accurate actions do not
converge as fast as cammarl.

Figure 5: Comparison of agent performances (in terms of reward accumulation)
in LBF with 2 agents (2p) and 6 foods (6f) respectively with cooperation
turned off. cammarl’s performance is very close to the upper bound, GIAM, and
is considerably better than the other extreme, NOAM. Interestingly, cammarl
also seems to converge faster than the other baselines. The curves are
averaged over five seeds. Figure 6: Comparison of agent performances in CN
with uncertainty information available to $\mathcal{N}_{self}$ during
training. This graph highlights the merit of using Conformal Action
Predictions over simple uncertainty estimation.

###  4.4 Results in Mixed Settings

For these experiments, we use the original LBF environment, but we turn off
the cooperation flag. This is a slightly easier task, however, each agent here
would fight for different food positions, so it is a mixed setting between
cooperation and opposition. Modeling actions of other agents can also help to
improve performance in this setting, although it is not as critical as in the
cooperative setting.

Figure 5 highlights the training performance on LBF with 2 agents
respectively. Although this is a simpler task, in which all algorithms roughly
converge to the same reward, cammarl seems to be on par with our upper bound
GIAM, especially in terms of faster convergence.

##  5 Motivating Conformal Predictions

Figure 7: Analysing conformal prediction in cammarl over time during the
training by looking at trends in conformal sets sizes, coverage of highly
probable predictions, model loss and accuracy during training of cammarl
agents.

In Figure 3 (a) and (b), we observe the improved performance of cammarl by
predicting conformal sets. One key ingredient to cammarl was the addition of
uncertainty predictions to the actions of the other agents. Thus, we can
attribute the increased performance of cammarl to this as well. To test this
theory, we added one more baseline where similar to the action prediction
baseline (EAP), we add the action prediction probabilities directly into the
state space. We call this Action Prediction with Uncertainty (APU), and both
APU and cammarl operate on the same information. From Figure 6, we can
conclude that just adding uncertain predictions is not enough to achieve an
uplift in performance that we see for cammarl and there is definite merit to
using conformal predictions. Another reason for the poor performance of APU
would be that the agent has to parse through the relevant information and
learn from it, whereas they are readily provided in a concise manner for
cammarl.

##  6 Discussion

In this section, we dig deeper and try to analyze the inner components of
cammarl. In particular, we plot some observable trends during the training of
cammarl’s agents in both the tasks (Figure 7) and discuss each of them here.

#### Set Sizes.

We collected the set sizes produced in cammarl throughout the training and
report them in Figure 7 (a) and 7 (f). Smaller sets are preferred, as they
carry specific information which can be more useful practically. The curves
show a decreasing trend in the set sizes in cammarl in both CN and LBF
respectively when tracked over the number of updates of the conformal
prediction model during training. This is a good sign for cammarl, as it shows
that the conformal predictions are becoming more precise with continued
training over time.

#### Coverage.

As also discussed earlier, it is desirable for the predicted sets to provide 1
- $\alpha$ coverage for a pre-defined user-specified $\alpha$ such as 10%.
Formally, to map a feature vector, $o_{other}\in\mathcal{O}_{other}$, to a
subset of discrete responses, $a^{\prime}_{other}\in\mathcal{A}_{other}$, it
is useful to define an uncertainty set function, $\mathcal{C}$($o_{other}$),
such that $P(a^{\prime}_{other}\in C(o_{other}))\geq 1-\alpha$. Figure 7 (b)
and 7 (f) shows the increasing trend of confidence coverage in cammarl.

#### Model accuracy and loss.

In Figure 7 (c) and Figure 7 (d) we show the conformal model’s accuracy and
loss respectively for CN and LBF in Figure 7 (g) and Figure 7 (h). The model
accuracy, as expected, increases with more data coming in to train over time
and the loss correspondingly decreases.

##  7 Conclusion

In this article, we propose a novel MARL algorithm, cammarl, which calls for
confident reasoning about other artificial agents in the environment and
benefiting from inferences about their behavior. Through experiments in two
cooperative multi-agent tasks, CN and LBF, we showed that guiding an agent’s
decision-making by inferring other agents’ actions in the form of conformal
sets, indeed helps in achieving better performances of the learning agents. By
using conformal prediction, we were also able to ensure the estimation of
predictive sets that covered the real predictions of the intentions of other
agents with a very high pre-specified probability of 95%.

Limitations and Future Works: In our paper, we analyzed cammarl with two
agents, however, cammarl is certainly generalizable to bigger networks or more
simple classifiers, and analyzing its changing performance on varying buffer
sizes can help in better comprehending its efficiency. Second, it would be
interesting to investigate the cammarl’s scalability to a system of many
agents (say 100 or 1000) or on more complicated multi-agent environments such
as tasks requiring a higher need for coordination. Thirdly, our mathematical
model in Section 3.1 makes an assumption that the state space is accessible
globally which may not be the case in some problems. Finally, in this work, we
restricted the agents to infer the behavior of other agents only via conformal
sets; it would be interesting to study the cases where more ways of sharing
information or modeling agents’ behavior are additionally allowed.

## References

  * Albrecht & Ramamoorthy (2015) Albrecht, S. V. and Ramamoorthy, S.  A game-theoretic model and best-response learning method for ad hoc coordination in multiagent systems.  _arXiv preprint arXiv:1506.01170_ , 2015.
  * Albrecht & Stone (2018) Albrecht, S. V. and Stone, P.  Autonomous agents modelling other agents: A comprehensive survey and open problems.  _Artificial Intelligence_ , 258:66–95, 2018.
  * Albrecht et al. (2020) Albrecht, S. V., Stone, P., and Wellman, M. P.  Special issue on autonomous agents modelling other agents: Guest editorial, 2020.
  * Angelopoulos et al. (2020) Angelopoulos, A., Bates, S., Malik, J., and Jordan, M. I.  Uncertainty sets for image classifiers using conformal prediction.  _arXiv preprint arXiv:2009.14193_ , 2020.
  * Angelopoulos & Bates (2021) Angelopoulos, A. N. and Bates, S.  A gentle introduction to conformal prediction and distribution-free uncertainty quantification.  _arXiv preprint arXiv:2107.07511_ , 2021.
  * Arora & Doshi (2021) Arora, S. and Doshi, P.  A survey of inverse reinforcement learning: Challenges, methods and progress.  _Artificial Intelligence_ , 297:103500, 2021.
  * Cauchois et al. (2020) Cauchois, M., Gupta, S., and Duchi, J.  Knowing what you know: valid and validated confidence sets in multiclass and multilabel prediction.  _arXiv preprint arXiv:2004.10181_ , 2020.
  * Das et al. (2017) Das, A., Kottur, S., Moura, J. M., Lee, S., and Batra, D.  Learning cooperative visual dialog agents with deep reinforcement learning.  In _Proceedings of the IEEE international conference on computer vision_ , pp. 2951–2960, 2017.
  * Foerster et al. (2016) Foerster, J. N., Assael, Y. M., de Freitas, N., and Whiteson, S.  Learning to communicate to solve riddles with deep distributed recurrent q-networks.  _arXiv preprint arXiv:1602.02672_ , 2016.
  * Gawlikowski et al. (2021) Gawlikowski, J., Tassi, C. R. N., Ali, M., Lee, J., Humt, M., Feng, J., Kruspe, A., Triebel, R., Jung, P., Roscher, R., et al.  A survey of uncertainty in deep neural networks.  _arXiv preprint arXiv:2107.03342_ , 2021.
  * Gronauer & Diepold (2022) Gronauer, S. and Diepold, K.  Multi-agent deep reinforcement learning: a survey.  _Artificial Intelligence Review_ , 55(2):895–943, 2022.
  * Grover et al. (2018) Grover, A., Al-Shedivat, M., Gupta, J., Burda, Y., and Edwards, H.  Learning policy representations in multiagent systems.  In _International conference on machine learning_ , pp. 1802–1811. PMLR, 2018.
  * Gupta et al. (2021) Gupta, N., Srinivasaraghavan, G., Mohalik, S. K., and Taylor, M. E.  Hammer: Multi-level coordination of reinforcement learning agents via learned messaging.  _arXiv preprint arXiv:2102.00824_ , 2021.
  * Hafner et al. (2020) Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M.  Dream to control: Learning behaviors by latent imagination.  In _International Conference on Learning Representations_ , 2020.  URL https://openreview.net/forum?id=S1lOTC4tDS.
  * Hafner et al. (2021) Hafner, D., Lillicrap, T. P., Norouzi, M., and Ba, J.  Mastering atari with discrete world models.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=0oabwyZbOu.
  * He et al. (2016) He, H., Boyd-Graber, J., Kwok, K., and Daumé III, H.  Opponent modeling in deep reinforcement learning.  In _International conference on machine learning_ , pp. 1804–1813. PMLR, 2016.
  * Hechtlinger et al. (2018) Hechtlinger, Y., Póczos, B., and Wasserman, L.  Cautious deep learning.  _arXiv preprint arXiv:1805.09460_ , 2018.
  * Jain et al. (2022) Jain, A. K., Sujit, S., Joshi, S., Michalski, V., Hafner, D., and Ebrahimi Kahou, S.  Learning robust dynamics through variational sparse gating.  _Advances in Neural Information Processing Systems_ , 35:1612–1626, 2022.
  * Kleiman-Weiner et al. (2016) Kleiman-Weiner, M., Ho, M. K., Austerweil, J. L., Littman, M. L., and Tenenbaum, J. B.  Coordinate to cooperate or compete: abstract goals and joint intentions in social interaction.  In _CogSci_ , 2016.
  * Kurach et al. (2020) Kurach, K., Raichuk, A., Stanczyk, P., Zajac, M., Bachem, O., Espeholt, L., Riquelme, C., Vincent, D., Michalski, M., Bousquet, O., et al.  Google research football: A novel reinforcement learning environment.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pp. 4501–4510, 2020.
  * Lazaridou et al. (2016) Lazaridou, A., Peysakhovich, A., and Baroni, M.  Multi-agent cooperation and the emergence of (natural) language.  _arXiv preprint arXiv:1612.07182_ , 2016.
  * Lei et al. (2018) Lei, J., G’Sell, M., Rinaldo, A., Tibshirani, R. J., and Wasserman, L.  Distribution-free predictive inference for regression.  _Journal of the American Statistical Association_ , 113(523):1094–1111, 2018.
  * Leibo et al. (2017) Leibo, J. Z., Zambaldi, V., Lanctot, M., Marecki, J., and Graepel, T.  Multi-agent reinforcement learning in sequential social dilemmas.  _arXiv preprint arXiv:1702.03037_ , 2017.
  * Littman (1994) Littman, M. L.  Markov games as a framework for multi-agent reinforcement learning.  In _Machine learning proceedings 1994_ , pp. 157–163. Elsevier, 1994.
  * Lowe et al. (2017) Lowe, R., Wu, Y., Tamar, A., Harb, J., Abbeel, P., and Mordatch, I.  Multi-agent actor-critic for mixed cooperative-competitive environments.  _Neural Information Processing Systems (NIPS)_ , 2017.
  * Matignon et al. (2012) Matignon, L., Laurent, G. J., and Le Fort-Piat, N.  Independent reinforcement learners in cooperative markov games: a survey regarding coordination problems.  _The Knowledge Engineering Review_ , 27(1):1–31, 2012.
  * Mealing & Shapiro (2015) Mealing, R. and Shapiro, J. L.  Opponent modeling by expectation–maximization and sequence prediction in simplified poker.  _IEEE Transactions on Computational Intelligence and AI in Games_ , 9(1):11–24, 2015.
  * Messoudi et al. (2020) Messoudi, S., Rousseau, S., and Destercke, S.  Deep conformal prediction for robust models.  In _Information Processing and Management of Uncertainty in Knowledge-Based Systems: 18th International Conference, IPMU 2020, Lisbon, Portugal, June 15–19, 2020, Proceedings, Part I 18_ , pp. 528–540. Springer, 2020.
  * Moerland et al. (2023) Moerland, T. M., Broekens, J., Plaat, A., Jonker, C. M., et al.  Model-based reinforcement learning: A survey.  _Foundations and Trends® in Machine Learning_ , 16(1):1–118, 2023.
  * Mordatch & Abbeel (2017) Mordatch, I. and Abbeel, P.  Emergence of grounded compositional language in multi-agent populations.  _arXiv preprint arXiv:1703.04908_ , 2017.
  * Mordatch & Abbeel (2018) Mordatch, I. and Abbeel, P.  Emergence of grounded compositional language in multi-agent populations.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 32, 2018.
  * Pal & Leon (2020) Pal, C.-V. and Leon, F.  Brief survey of model-based reinforcement learning techniques.  In _2020 24th International Conference on System Theory, Control and Computing (ICSTCC)_ , pp. 92–97. IEEE, 2020.
  * Panella & Gmytrasiewicz (2017) Panella, A. and Gmytrasiewicz, P.  Interactive pomdps with finite-state models of other agents.  _Autonomous Agents and Multi-Agent Systems_ , 31:861–904, 2017.
  * Papadopoulos et al. (2002) Papadopoulos, H., Proedrou, K., Vovk, V., and Gammerman, A.  Inductive confidence machines for regression.  In _Machine Learning: ECML 2002: 13th European Conference on Machine Learning Helsinki, Finland, August 19–23, 2002 Proceedings 13_ , pp. 345–356. Springer, 2002.
  * Papoudakis et al. (2021) Papoudakis, G., Christianos, F., and Albrecht, S.  Agent modelling under partial observability for deep reinforcement learning.  _Advances in Neural Information Processing Systems_ , 34:19210–19222, 2021.
  * Park et al. (2019a) Park, S., Bastani, O., Matni, N., and Lee, I.  Pac confidence sets for deep neural networks via calibrated prediction.  _arXiv preprint arXiv:2001.00106_ , 2019a.
  * Park et al. (2019b) Park, Y. J., Cho, Y. S., and Kim, S. B.  Multi-agent reinforcement learning with approximate model learning for competitive games.  _PloS one_ , 14(9):e0222215, 2019b.
  * Platt et al. (1999) Platt, J. et al.  Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods.  _Advances in large margin classifiers_ , 10(3):61–74, 1999.
  * Polydoros & Nalpantidis (2017) Polydoros, A. S. and Nalpantidis, L.  Survey of model-based reinforcement learning: Applications on robotics.  _Journal of Intelligent & Robotic Systems_, 86(2):153–173, 2017.
  * Rabinowitz et al. (2018) Rabinowitz, N., Perbet, F., Song, F., Zhang, C., Eslami, S. A., and Botvinick, M.  Machine theory of mind.  In _International conference on machine learning_ , pp. 4218–4227. PMLR, 2018.
  * Schulman et al. (2017) Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Sukhbaatar et al. (2016) Sukhbaatar, S., Fergus, R., et al.  Learning multiagent communication with backpropagation.  _Advances in neural information processing systems_ , 29, 2016.
  * Tacchetti et al. (2018) Tacchetti, A., Song, H. F., Mediano, P. A., Zambaldi, V., Rabinowitz, N. C., Graepel, T., Botvinick, M., and Battaglia, P. W.  Relational forward models for multi-agent learning.  _arXiv preprint arXiv:1809.11044_ , 2018.
  * Vovk et al. (2005) Vovk, V., Gammerman, A., and Shafer, G.  _Algorithmic learning in a random world_.  Springer Science & Business Media, 2005.
  * Wang et al. (2022) Wang, X., Zhang, Z., and Zhang, W.  Model-based multi-agent reinforcement learning: Recent progress and prospects.  _arXiv preprint arXiv:2203.10603_ , 2022.
  * Xie et al. (2021) Xie, A., Losey, D., Tolsma, R., Finn, C., and Sadigh, D.  Learning latent representations to influence multi-agent interaction.  In _Conference on robot learning_ , pp. 575–588. PMLR, 2021.
  * Zhang et al. (2021) Zhang, W., Wang, X., Shen, J., and Zhou, M.  Model-based multi-agent policy optimization with adaptive opponent-wise rollouts.  _arXiv preprint arXiv:2105.03363_ , 2021.
  * Zhu et al. (2022) Zhu, C., Dastani, M., and Wang, S.  A survey of multi-agent reinforcement learning with communication.  _arXiv preprint arXiv:2203.08975_ , 2022.
  * Zintgraf et al. (2021) Zintgraf, L., Devlin, S., Ciosek, K., Whiteson, S., and Hofmann, K.  Deep interactive bayesian reinforcement learning via meta-learning.  _arXiv preprint arXiv:2101.03864_ , 2021.

## Appendix

##  Appendix A Environments

Figure 8: Multi-agent cooperative environments used in this study: (a) OpenAI
MPE’s Cooperative Navigation: Agents (blue) learn to cover the two landmarks
(black) avoiding collisions. The figure shows cooperative navigation with 2
agents (N=2) and 2 landmarks (L=2). and (b) Level-based foraging: Agents must
collect food and learn to cooperate using sparse rewards. This is a 12
$\times$ 12 level-based foraging grid-world with 2 cooperative players and 4
food locations. (c) Pressure Plate: Agents must stand on the pressure plate to
keep the gates open for one of the agents to reach the goal. This is a
pressure plate environment with 4 agents.

#### Cooperative Navigation (CN) (Mordatch & Abbeel, 2017; Lowe et al., 2017).

In this task, agents are expected to learn to navigate and cover all the
landmarks cooperatively and without colliding. Each agent can perceive the
other agents and landmarks within its reference frame (in the form of relative
positions and velocities) and can take discrete actions to move around (left,
right, up, down, stay) in the environment. The agents receive a team reward
(so $r_{self}$ and $r_{other}$ are the same in this case) which is calculated
as the minimum of the distance of the agents’ and landmarks’ ($x_{i}$,
$y_{i}$) positions in the grid world. This reward, based on their proximity
(or distance) from each of the landmarks, forces the need for cooperation in
order to succeed in the task. Furthermore, agents are penalized upon
collisions. Formally, the reward function in this environment can be defined
as

$r=\left[-1*\sum_{l=1}^{L}min_{\\{n\in\mathcal{N}\\}}(distance(n,l))\right]-c$

where $|\mathcal{N}|$ is the number of agents and L are the landmarks in the
environment. Here, c is the number of collisions in an episode and the agents
are penalized with -1 for each time two agents collide.

#### Level-based foraging (LBF) (Albrecht & Ramamoorthy, 2015).

In this environment, $\mathcal{N}_{self}$ and $\mathcal{N}_{other}$ are part
of a 12$\times$12 grid world which contains four randomly scattered food
locations, each assigned a level. The agents also have a level of their own.
They attempt to collect food which is successful only if the sum of the levels
of the agents involved in loading is greater than or equal to the level of the
food. This is a challenging environment, requiring agents to learn to trade
off between collecting food for their own and cooperating with the other agent
to acquire higher team rewards. Moreover, this environment has sparse rewards
making it difficult to learn and operate independently in the environment. In
particular, each agent is rewarded equal to the level of the food it managed
to collect, divided by its level (its contribution).

#### Pressure Plate333https://github.com/uoe-agents/pressureplate

In this domain, the agents are expected to learn to co-operate in traversing
the gridworld to collect the yellow treasure chest. Each agent is assigned a
pressure plate that only they can activate. That agents must stand on the
pressure plate to keep the corresponding doorway to open while the other
agents can go through it leaving the agent behind. Each agent can only view a
limited portion of the environment. If an agent is in the room with their
assigned plate, their reward is the negative normalized Manhattan distance
between the agent and plate’s position, else the reward is the difference
between the current and desired room number.

Figure 9: Comparing the performance of agents when trained with different
variants of cammarl.

#### Google Football (Kurach et al., 2020)

For our task, we use the 3 vs 1 with keeper game which is one of the 5 domains
in the Football Academy of Google Research Football domain. The main agent
controls 3 agents whose objective is to score a goal. The opponent, consisting
of 1 defender and a keeper will try to prevent scoring the goal. This
environment requires cooperation among the 3 players to score, including
skills like passing the ball. The agents receive a reward when it successfully
scores a goal past the opponent.

##  Appendix B Variations of the cammarl algorithm

As noted earlier, the output of the conformal prediction model, i.e., the
conformal sets, are of varying sizes in different situations. So, we came up
with numerous ways to be able to use them to implement cammarl. In this
section, we discuss and compare them and speculate some pros and cons for each
of them.

####  cam-padding.

The conformal model outputs a set containing the action predictions in the
decreasing order of their probability of being the true actions. In other
words, the first element in the set is the highest probable action, the next
is second highest probable, and so on. The maximum set size being
$|\mathcal{A}_{other}|$, we try padding the set with zeros to fix its size
when used as input in cammarl. We hypothesise that in this way,
$\mathcal{N}_{self}$ must be able to learn to infer the information about the
actions along with the notion of their importance. Figure 9 also supports this
hypothesis by showing the reasonable performance of this version of
cammarl(green curve). Nevertheless, such padding can be undesirable,
particularly because zeros need not necessarily mean “no information”, and we
try to remove it in the versions discussed next.

####  cam-binary.

In this method (also discussed earlier in Section 4), we encode the conformal
sets into binary strings. We start with a vector of zeros of size
$|\mathcal{A}_{other}|$ and fire up the bits corresponding to the output
actions in the sets. The orange curve in Figure 9 shows that this version of
cammarl outperforms all the other versions.

####  cam-penultimate.

Here, we modify cammarl to now share the embedding (representations) learned
in the conformal prediction model as input to $\mathcal{N}_{self}$. The size
of embedding is fixed and this way, padding of conformal action sets in
cammarl can be avoided. As shown in Figure 9 (blue curve), cammarl with
representations also manages to obtain returns close to binary-cammarl and
padded-cammarl. We also note here that, binary-cammarl and padded-cammarl
still performs slightly better than cammarl with representations.

Interestingly, all versions perform almost equivalently for the use-cases
tested in this article. Any method can be used depending on the application at
hand. For instance in medical use-cases, the knowledge of ranks of conformal
actions is critical, and so cp-padding can be used. In this article, we choose
cp-binary. It is a simple way to ensure fixed-sized inputs for
$\mathcal{N}_{self}$ in cammarl and empirically works very well, and hence we
propose this as a generic solution. We note here that, the embeddings here are
not being passed in addition to the set of predictions.

##  Appendix C MARL agents: Implementation details

###  C.1 Partially observable MDP

The partially observable MDP defined and used in this study has numerous
parameters as defined previously. Here is a table for quick reference (Table
1).

Symbol | Description
---|---
i |  self or other
$N_{i}$ | Agent i
S | Set of possible states of the environment
$A_{i}$ | Set of available actions for agent i
$O_{i}$ | Set of local observations of agent i
T | State transition function
C | Conformal Prediction model
$\pi_{\theta_{i}}$ | Agent i’s decision policy
$r_{i}$ | Reward function for agent i
$\gamma$ | Discount factor
T | Time horizon (length of an episode)
Table 1: Parameters in our POMDP

We use proximal policy optimization (PPO) (Schulman et al., 2017) to update
the decision-making policy for both the RL agents, however, any other RL
algorithm could be used alternatively. For the individual actor and critic
networks, we used 2 fully-connected multi-layer perceptron (MLP) layers. For
the conformal prediction model $\mathcal{C}$, we use another fully-connected
MLP with 2 layers each with 64 hidden nodes. All the agents and the model
$\mathcal{C}$ collect their individual experiences and train their own
policies independently.

We add regularization to the conformal model to encourage small set sizes.
This regularization is controlled by two hyper-parameters $\lambda$ and
$k_{reg}$. Each time the conformal model is trained, we choose the $\lambda$
and $k_{reg}$ parameters from a set that optimizes for small sizes of the
action set. The $k_{reg}$ values are dependent on the logits, however, the
$\lambda$ values are in a set $[0.001,0.01,0.1,0.2,0.5]$.

##  Appendix D Visualization of Learned Policy

In this section, we visualize the learned policy in Pressure Plate with
cammarl demonstrating cooperation between the 4 agents.

Figure 10: Visualization of the policy learned by cammarl in Pressure Plate
with 4 agents