11institutetext: University of Maryland, College Park MD 20740, USA 11email:
<EMAIL_ADDRESS>
# Interpretable Deep Reinforcement Learning for Green Security Games with
Real-Time Information

Vishnu Dutt Sharma  John P. Dickerson  Pratap Tokekar

###### Abstract

Green Security Games with real-time information (GSG-I) add the real-time
information about the agents’ movement to the typical GSG formulation. Prior
works on GSG-I have used deep reinforcement learning (DRL) to learn the best
policy for the agent in such an environment without any need to store the huge
number of state representations for GSG-I. However, the decision-making
process of DRL methods is largely opaque, which results in a lack of trust in
their predictions. To tackle this issue, we present an interpretable DRL
method for GSG-I that generates visualization to explain the decisions taken
by the DRL algorithm. We also show that this approach performs better and
works well with a simpler training regimen compared to the existing method.

###### Keywords:

Green Security Games Deep Reinforcement Learning Interpretable Machine
Learning

##  1 Introduction

Green Security Games (GSGs) games are a sub-class of Stackelberg Security
Games which are employed to model the interaction between law enforcement
personnel and poachers in the green security domain like the security and
conservation of forest, wildlife, and fisheries against poachers and illegal
harvesters. Existing methods have used artificial intelligence for enhancing
the movement strategies for law enforcement personnel, to capture the illegal
and adversarial agents [14] [3] [13]. Wang et al [26] propose a variation of
GSG called GSG-I, which includes the real-time footprints of the players. The
addition of such information scales up the memory requirement to encode the
game representation and makes the traditional solutions unsuitable for solving
GSG-I. This issue can be resolved with deep reinforcement learning (DRL), a
deep learning technique, treating the GSG as a Markov Decision Process (MDP).
DRL can learn the policy or value function without supervision and does not
need to store the state representation, unlike the tabular reinforcement
learning approaches. The non-linear multi-layered structure of DRL networks
makes them capable to learn the complex mapping between inputs and output, and
therefore such networks can approximate the optimal policy well. Thus, DRL
methods have been quite successful for pursuit-evasion games like Ms. Pacman
[20][6]. Similarly, Wang et al. [26] present a DRL method combined with the
double oracle framework [19] called DeDOL to GSG-I. The agent trained with
this approach can use real-time information and performs well in terms of
average episodic rewards.

While the complex structure of the DRL methods makes them successful in a
variety of applications but also renders them opaque to humans i.e. decision-
making processes of such methods are difficult to interpret. Real-life
deployment of such systems can be dangerous in scenarios where the decision
can cause harm to the end-user. Many recent works propose explainable
frameworks and interpretable methods to allow the user to understand the key
factors behind a decision and thus engender trust in DRL techniques.

Motivated by these developments we present an interpretable DRL method to
solve GSG-I. We also show that despite using interpretable constructs, this
method performs better than the non-interpretable counterpart. Lastly, the
training procedure for this method is simpler and faster than that of DeDOL.

##  2 Related Work

The idea of using DRL for GSGs has previously been explored for the continuous
space GSGs [11] [12], uncertainty augmented GSGs [25], and Multi-agent games
[15]. The GSG definitions in these works lack real-time information that could
capture the real-world interaction of agents with the environment. Wang et al.
[26] fill this gap by proposing GSG-I with the adversary’s footprints as the
real-time information. They also propose a DRL method trained with the double
oracle framework [19]. All of these methods, however, treat the DRL models as
black-boxes, concealing their decision-making process. Making such systems
transparent can help us ensure that the DRL policies do not result in any harm
to the agents, and may provide further insights like robustness and
generalizability of these systems.

Efforts to bring transparency to the DRL models have gained momentum in the
past few years. The approaches to make DRL, and deep learning in general,
transparent can be classified into two directions: (a) interpretability, where
the mechanism of the underlying model is analyzed to understand how a decision
was reached, and (b) explainability, where we treat the DRL model as a black-
box and try to approximate a simple, but interpretable method like decision
tree over its prediction, to explain the reason behind an output.
Interpretability is usually preferred over explainability as it tries to
convey the actual decision-making process instead of approximating the
original solution. But it is also often assumed to result in a worse-
performing model, though no proof exists for this claim [23].

As DRL networks have been used for a variety of domains, like computer vision
and natural language processing, the existing work on transparent DRL doesn’t
have a general method to fit all the applications. While some focus on using
the intermediate layers of a trained DRL model to visualize the feature
influences [8] [16], others distill the knowledge into simpler prediction
models and infer the decision rules from them [18]. The states in GSG-I have
spatial relations and can be thought of as multi-channel images. For such
inputs, explainable approaches often employ saliency-based approaches [22] [1]
are often used as they can be visualized along with the inputs. Tree-based
methods like soft decision trees (SDTs) [4] and Linear Model U-Trees (LMUTs)
[17] approximate the DRL model into a piece-wise linear function with its tree
structure and visualize the importance of the input features.

For interpretability, the neural network architecture is required to have
intermediate units whose output can be interpreted. The existing saliency-
based methods [7][5][21] rely on perturbations and thus result in explanations
rather than interpretations. Object-based saliency maps [16] provide an
alternative by using the appearance of the objects in the environment. But
these maps also effectively generate explanations and rely heavily on domain
knowledge. Hilton et al [8] take a more rigorous approach of associating
activations and data points to first identify the type of object that the
neuron focuses on and then convert them to saliency maps. An simpler way to
generate saliency maps is to use attention mechanism [2]. Attention mechanism
is a neural network unit that generates the features by aggregating the input
feature, where the weight for aggregation is learned using the same input
features. The weights can be used to interpret the strength of influence of
the features in the prediction. Prior works have successfully used attention
mechanism for identifying the important part in images based on text for image
captioning [27], which supports their suitability for learning interpretable
features in spatially-correlated inputs, or encoded images. As the weights of
attention mechanism participate in the decision-making process, it is an
interpretable method. Attention-based saliency methods for DRL like Attention
Augmented Agents [17] and Mask-Attention-A3C [10] generate their saliency maps
by including spatial attention mechanism in the architecture. Mask-
Attention-A3C uses a simpler implementation of attention mechanism akin to
gates in the long short-term memory [9] units and leverages them as masks to
control the information flow form the feature maps to the classifier. This is
a simpler yet effective implementation that can produce interpretable maps for
both value and policy function, and thus we adapt it for solving GSG-I in this
work.

##  3 Experiment Setup

The state in GSG-I is represented as a multi-dimensional array, encoding the
information available to the patroller. However, Mask-Attention-A3C (MA-A3C)
was originally designed for 3-channel color images. Thus we run experiments
both on GSG-I states transformed to color images (referred to as _color
images_ with the original MA-A3C architecture, and on multi-array
representations (referred to as _encoded images_) with a modified MA-A3C
architecture.

In the following subsections, we first provide details of the environment and
then elaborate on the proposed DRL method.

Figure 1: The GSG-I environment used in the experiments

###  3.1 GSG-I Environment

Figure 1 shows a state from the GSG-I environment. This setting involves a
patroller (defender) and a poacher (attacker). Each cell has a fixed
probability of an animal being there, simulating areas in a forest, e.g. lake,
mountain, swamp, etc. that differ in the fauna density. At the start of the
game, the poacher, an adversary, enters the grid through one of the corners,
chosen at random. Her goal is to capture animals by placing snares in the
cells and exiting the environment through the starting location. A snare can
launch a successful attack in a cell $(i,j)$ with probability
$P^{attack}_{i,j}$ i.e. the probability of an animal’s presence in that cell.
She can place only 3 snares at max and tries to maximize the number of animals
captured. Any attack launched after her return to the start position does not
count. The patroller always starts patrolling from her post as the center of
the grid. She has a map of the forest and thus knows the fauna density in each
cell in the grid. The patroller can keep track of the real-time information
like her and the attacker’s footprints observed so far. The patroller’s
objective is to find and remove potential snares and catch the attacker as
soon as possible. The capture of the attacker results in the termination of
the game.

GSG-I is designed as a two-player, zero-sum game. The patroller, our agent,
gets a reward $\mathbb{R}$ of $+2$ when she removes a snare and a reward of
$+8$ if she captures the attacker. But she incurs a negative reward equal to
the $-P^{attack}_{i,j}$ if the attack is launched successfully in the cell
$(i,j)$. The patroller can choose an action from her action space
$\mathbb{A}^{d}$ : { up, down, right, left, stand still}. The attacker’s
actions space additionally includes placing snare as following,
$\mathbb{A}^{a}$ : {up, down, right, left, stand still } $\times$ { place
snare, do not place snare } . The game terminates if the patroller catches the
attacker and finds all the snare, or if the time horizon $T$ is reached. The
games state for encoded images $\mathbb{S}$ for the patroller, is represented
by a 20 channel square array with side 7, corresponding to the patroller’s
view of the environment and encodes the following information: the first 8
channels are the poachers’ footprints encoded as binary variables indicating
the direction of movement ({north, west, east, south}×{entering or leaving});
similarly, the next 8 channel encode the patroller’s footprints; the $17^{th}$
channel encodes the patroller’s location on the grid a 1 at the corresponding
location in the array, and 0 elsewhere; the $18^{th}$ channel stores the
animal density or the probability of successful attack by snare in each cell;
the $19^{th}$ channel stores the trail of the patroller by adding 0.1 to a
cell whenever the patroller visits it; the $20^{th}$ stores the time
normalized by the time horizon $T$ (75 for the $7\times 7$ grid GSG-I), which
is same across all the cells. The attacker can place a maximum of 6 snares in
this grid. In our experiments, we use a randomly generated animal density map
and attacker moves using heuristic random walk [26]. The colored image
representations are images of size $160\times 160$ pixels generated with the
information available to the patroller. Figure 2 shows two colored images from
an episode. As shown in Figure 2(b), the patroller can not see the attacker
and only knows about the attacker’s footprints observed so far.

(a) State after $7^{th}$ step

(b) State after $18^{th}$ step

Figure 2: GSG-I states as images in an example episode at $7^{th}$ and
$18^{th}$ state

###  3.2 DRL Network

MA-A3C network is composed of Convolution Neural Network (CNN) and Multi-Layer
Perceptron (MLP) layers (Figure 3(a)). The CNN layers act as feature
generators and for feature generation using the spatial relationship between
entities. The feature generator for encoded images is composed of one
convolutional and max-pooling, and for color images, we use 3 convolutional
and max-pooling layers. After each max-pooling operation, ReLU activation is
used. The output features maps have 64 channels and are shared between the
policy (actor) and value (critic) heads. At each head, the network learns an
attention map of the same size as the feature map. It is used as a mask over
the output of the feature layer as shown in Figure 3(b). The masks use sigmoid
activation and thus act as almost binary filters controlling the flow of
information; values close to 0 suppress the outputs at the corresponding
location in the feature map, an values close to 1 pass them with minute
change. The filters features are then fed to the MLP at each head. At policy
head, the MLP acts as a regressor, whereas the value head acts as a
classifier. The network additionally includes an optional ConvLSTM [24] unit
to share features across time and capture the temporal relations. We run
experiments without ConvLSTM first and then enabled it on the better
performing state representation to observe its effect. The experiments are
conducted using a 32-core, 2.10Ghz Xeon Silver-4208 CPU and an Nvidia GeForce
RTX 2080Ti GPU. We train the networks for 100000 episodes, which takes around
18 hours for MA-A3C without ConvLSTM and around 27 hours for MA-A3C with
ConvLSTM. The pre-trained DeDOL model, on the other hand, is reported to have
been trained in local and global mode with a double oracle framework for 5
days.

(a) Network

(b) Attention mechanism for masking

Figure 3: Mask-Attention-A3C Network Architecture

##  4 Results

Our experiments show that the visualizing the attention maps/masks in MA-A3C
provide some intuition about the next action. Figure 4 shows the examples for
MA-A3C without ConvLSTM on color images. Here, the patroller moves towards the
wall and follows along the edge of the environment until it comes across the
attacker’s footsteps. This behavior is similar to DeDOL’s strategy presented
by Wang et al [26]. In the attention map for the value function, initially,
the focus is on cells with high fauna density, which is reasonable as the
agent hasn’t collected any evidence about the attacker’s movement. Also, the
region of interest covers the patroller, a small part of the cell where she
moves next, and her past few footprints. This is likely due to the focus on
the new information that is available in the current cell. When multiple
footprints of the attacker are seen, this map is useful in identifying where
the patroller will move next. However, there are additional multiple regions
on the map which do not seem to influence the patroller’s actions. The
attention map for the policy function seems less noisy compared to the value
function, but it does not show the correct influential regions concerning the
patroller’s actions.

(a) Attention maps for value head

(b) Attention maps for policy head

Figure 4: Attention maps generated by Mask-Attention-A3C for GSG-I with
3-channel color image as the input

The interpretation maps of MA-A3C networks for the encoded images is
comparatively more clear, as shown in Figure 5. This was expected as the
encoded images ease the burden of processing images into features for the
neural network. In this case, the attention map for the value function is
sparser and the area indicating the patroller’s next step is not as salient
here. On the other hand, the attention map for the policy function provides a
better idea of the patroller’s next steps. In the second last frame, the
patroller could have moved left or down, but it is clear from the attention
map that she will move down. An interesting observation here is that the
initial path of the patroller is very similar to the path predicted by DeDOL
(Figure 2(b)). Enabling ConvLSTM used results more intuitive and focused
interpretable maps. As shown in Figure LABEL:fig:gsg1_enc_mask_lst, both the
value and policy maps give a clear sense of the agent’s next move. We believe
that this is because the agent is now able to compare the temporal changes in
the state when making a decision and thus can process new information well. We
also note that the attention maps in these experiments are not as clear and
sharp as generally observed on games like Ms. Pacman, possibly due to local
visibility of the agent in GSG-I as compared to Pacman, where the Pacman can
see the whole environment.

(a) Attention maps for value head

(b) Attention maps for policy head

Figure 5: Attention maps generated by Mask-Attention-A3C for GSG-I with
20-channel encoded image as the input

(a) Attention maps for value head

(b) Attention maps for policy head

Figure 6: Attention maps generated by Mask-Attention-A3C with ConvLSTM enabled
for GSG-I with 20-channel encoded image as the input

Figure 7 compares the episode length and the episodic reward i.e. the total
reward accumulated in an episode, aggregated over 1000 episodes of GSG-I for
DeDOL and the following variations of MA-A3C: MA-A3c over encoded image
(MA-A3C (Encoded Image)); MA-A3C over color images (MA-A3c (RGB Image));
MA-A3c with ConvLSTM over encoded image (MA-A3C-ConvLSTM (Encoded Image));
MA-A3C with ConvLSTM over encoded images trained for 1000,000 episodes
(MA-A3C-ConvLSTM_v2 (Encoded Image)). We present these results for the
randomly generated fauna density map and a Gaussian fauna density map (highest
probability at the center and lower towards the edges). A good model should
have low episode length (patroller catching the attacker as soon as possible)
and high episodic reward (patroller removing all the snares and catching the
attacker). We find the DeDOL to perform better than the MA-A3C models without
ConvLSTMs but fares badly in comparison to ConvLSTM augmented MA-A3C models.
While the MA-A3C-ConvLSTM (Encoded Image) is able to perform on par with
DeDOL, and further training (MA-A3C-ConvLSTM (Encoded Image)) improves the
margin of difference in metric from DeDOL. However, more training doesn’t help
the network generalize well and both the metric suffers severely compare to
all other models. We suspect that the networks begin to memorize the input at
this point. MA-A3C with the color image is similar to MA-A3C with the encoded
image in performance. Thus we can simplify the network architecture if the
preprocessed information is present, but raw information can also be used with
appropriate architectural modifications.

(a) Episode length

(b) Episodic reward

Figure 7: Performance comparison of the interpretable DRL networks with DeDOL

##  5 Conclusion and Future Work

The experiments with Mask-Attention-A3C present promising qualitative and
quantitative results. We found that similarly performing, but interpretable
DRL models can be obtained by appropriate augmentations of attention maps and
ConvLSTM units. Such models can be trained within lesser time and without
complex training, regime consisting of multiple modes and game theoretic
update mechanism. We believe that such methods can further improve with
inclusion on units like ConvLSTM that indirectly act priors for learning
methodology benefiting both the interpretation and performance. An interesting
direction for future work is the human evaluation of the map visualizations to
conclude whether they are interpretable in general. A comparative evaluation
of the two attention maps generated by Mask-Attention-A3C can add more
insights. These maps provide two different interpretations. While an expert
can relate them directly to the value function and the policy function, a non-
expert may need the answer to the question which maps is more useful in a
given condition?. Such studies would therefore be able to generalize the
meaning of interpretable visualizations.

## References

  * [1] Akanksha Atrey, Kaleigh Clary, and David Jensen.  Exploratory not explanatory: Counterfactual analysis of saliency maps for deep reinforcement learning.  arXiv preprint arXiv:1912.05743, 2019.
  * [2] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  arXiv preprint arXiv:1409.0473, 2014.
  * [3] Anjon Basak, Fei Fang, Thanh Hong Nguyen, and Christopher Kiekintveld.  Combining graph contraction and strategy generation for green security games.  In International Conference on Decision and Game Theory for Security, pages 251–271. Springer, 2016.
  * [4] Nathan Dahlin, Krishna Chaitanya Kalagarla, Nikhil Naik, Rahul Jain, and Pierluigi Nuzzo.  Designing interpretable approximations to deep reinforcement learning with soft decision trees.  arXiv preprint arXiv:2010.14785, 2020.
  * [5] Ruth C Fong and Andrea Vedaldi.  Interpretable explanations of black boxes by meaningful perturbation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3429–3437, 2017.
  * [6] Adrian Goldwaser and Michael Thielscher.  Deep reinforcement learning for general game playing.  In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 1701–1708, 2020.
  * [7] Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern.  Visualizing and understanding atari agents.  In International Conference on Machine Learning, pages 1792–1801. PMLR, 2018.
  * [8] Jacob Hilton, Nick Cammarata, Shan Carter, Gabriel Goh, and Chris Olah.  Understanding rl vision.  Distill, 5(11):e29, 2020.
  * [9] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural computation, 9(8):1735–1780, 1997.
  * [10] Hidenori Itaya, Tsubasa Hirakawa, Takayoshi Yamashita, Hironobu Fujiyoshi, and Komei Sugiura.  Visual explanation using attention mechanism in actor-critic-based deep reinforcement learning.  arXiv preprint arXiv:2103.04067, 2021.
  * [11] Nitin Kamra, Umang Gupta, Fei Fang, Yan Liu, and Milind Tambe.  Policy learning for continuous space security games using neural networks.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  * [12] Nitin Kamra, Umang Gupta, Kai Wang, Fei Fang, Yan Liu, and Milind Tambe.  Deepfp for finding nash equilibrium in continuous action spaces.  In International Conference on Decision and Game Theory for Security, pages 238–258. Springer, 2019.
  * [13] Debarun Kar.  When AI helps Wildlife Conservation: Learning Adversary Behaviors in Green Security Games.  PhD thesis, University of Southern California, 2017.
  * [14] Richard Klima, Karl Tuyls, and Frans Oliehoek.  Markov security games: Learning in spatial security problems.  In NIPS Workshop on Learning, Inference and Control of Multi-Agent Systems, pages 1–8, 2016.
  * [15] Marc Lanctot, Vinicius Zambaldi, Audrunas Gruslys, Angeliki Lazaridou, Karl Tuyls, Julien Pérolat, David Silver, and Thore Graepel.  A unified game-theoretic approach to multiagent reinforcement learning.  arXiv preprint arXiv:1711.00832, 2017.
  * [16] Michael Lewis, Huao Li, and Katia Sycara.  Deep learning, transparency, and trust in human robot teamwork.  In Trust in Human-Robot Interaction, pages 321–352. Elsevier, 2021\.
  * [17] Guiliang Liu, Oliver Schulte, Wang Zhu, and Qingcan Li.  Toward interpretable deep reinforcement learning with linear model u-trees.  In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 414–429. Springer, 2018.
  * [18] Prashan Madumal, Tim Miller, Liz Sonenberg, and Frank Vetere.  Explainable reinforcement learning through a causal lens.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 2493–2500, 2020.
  * [19] H Brendan McMahan, Geoffrey J Gordon, and Avrim Blum.  Planning in the presence of cost functions controlled by an adversary.  In Proceedings of the 20th International Conference on Machine Learning (ICML-03), pages 536–543, 2003.
  * [20] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller.  Playing atari with deep reinforcement learning.  arXiv preprint arXiv:1312.5602, 2013.
  * [21] Nikaash Puri, Sukriti Verma, Piyush Gupta, Dhruv Kayastha, Shripad Deshmukh, Balaji Krishnamurthy, and Sameer Singh.  Explain your move: Understanding agent actions using specific and relevant feature attribution.  arXiv preprint arXiv:1912.12191, 2019.
  * [22] Matthias Rosynski, Frank Kirchner, and Matias Valdenegro-Toro.  Are gradient-based saliency maps useful in deep reinforcement learning?  arXiv preprint arXiv:2012.01281, 2020.
  * [23] Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong.  Interpretable machine learning: Fundamental principles and 10 grand challenges.  arXiv preprint arXiv:2103.11251, 2021.
  * [24] Xingjian Shi, Zhourong Chen, Hao Wang, Dit-Yan Yeung, Wai-Kin Wong, and Wang-chun Woo.  Convolutional lstm network: A machine learning approach for precipitation nowcasting.  Advances in neural information processing systems, 28, 2015.
  * [25] Aravind Venugopal, Elizabeth Bondi, Harshavardhan Kamarthi, Keval Dholakia, Balaraman Ravindran, and Milind Tambe.  Reinforcement learning for unified allocation and patrolling in signaling games with uncertainty.  arXiv preprint arXiv:2012.10389, 2020.
  * [26] Yufei Wang, Zheyuan Ryan Shi, Lantao Yu, Yi Wu, Rohit Singh, Lucas Joppa, and Fei Fang.  Deep reinforcement learning for green security games with real-time information.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 1401–1408, 2019.
  * [27] Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio.  Show, attend and tell: Neural image caption generation with visual attention.  In International conference on machine learning, pages 2048–2057. PMLR, 2015.