# Deep Interpretable Models of Theory of Mind

Ini Oguntola, Dana Hughes, and Katia Sycara * All authors are with the School
of Computer Science, Carnegie Mellon University, Pittsburgh, PA, USA
(ioguntol, danahugh<EMAIL_ADDRESS>This work is supported by the Defense
Advanced Research Projects Agency (DARPA) under Contract No. HR001120C0036,
and by the AFRL/AFOSR award FA9550-18-1-0251.

###### Abstract

When developing AI systems that interact with humans, it is essential to
design both a system that can understand humans, and a system that humans can
understand. Most deep network based agent-modeling approaches are 1) not
interpretable and 2) only model external behavior, ignoring internal mental
states, which potentially limits their capability for assistance,
interventions, discovering false beliefs, etc. To this end, we develop an
interpretable modular neural framework for modeling the intentions of other
observed entities. We demonstrate the efficacy of our approach with
experiments on data from human participants on a search and rescue task in
Minecraft, and show that incorporating interpretability can significantly
increase predictive performance under the right conditions.

##  I INTRODUCTION

Human intelligence is remarkable not just for the way it allows us to navigate
environments individually, but also how it operates socially, engaging with
other intelligent actors. Humans naturally build high-level models of those
around them, and are able to make inferences about their beliefs, desires and
intentions (BDI) [1]. These inferences allow people to anticipate the
behaviors of others, use these predictions to condition their own behavior,
and then anticipate potential responses. In both psychology and machine
learning this is referred to as theory of mind (ToM) [2, 3, 4], which aims to
model not only the external behavior of other entities but their internal
mental states as well. The developmental psychology literature has found that
children as young as 4 years old have already developed a ToM, a crucial
ability in human social interaction [5]. ToM can enable discovery of false or
incomplete beliefs and knowledge and can thus facilitate interventions to
correct false beliefs. Therefore, work in enabling agents to develop ToM is a
crucial step not only in developing more effective multi-agent AI systems but
also for developing AI systems that interact with humans, both cooperatively
and competitively [4].

With a few exceptions [6], most agent-modeling approaches in the reinforcement
learning and imitation learning literature largely ignore these internal
mental states, usually only focusing on reproducing the external behavior [7,
8]. This limits their ability to reason in a deeper way about entities that
they interact with. While prior work has explored providing agents with models
of some aspect of a human’s mental state, such as reward [6] or rationality
[9], more complex models incorporating multiple properties of mental state
(e.g., beliefs over the environment, desires, personality characteristics,
etc.) in non-toy environments with remains largely unexplored.

While modeling ToM of a human is a very challenging task for an artificial AI
agent, understanding the reasoning of such an agent is even more challenging.
This paper focuses on developing human-interpretable ToM models. We believe
that the ability of a ToM model to both infer human mental states and allow
humans to understand what features contribute to the model’s reasoning would
1) enable trust with humans it interacts with 2) better enables the agent to
choose and justify interventions (e.g. advice).

In this paper we focus on modeling the intentions of observed human within a
ToM framework, and discuss the potential to leverage this reasoning for
interventions, i.e. give advice to the human in the context of human-agent
teaming. We perform experiments with human trajectories obtained from
simulated search and rescue tasks in a Minecraft environment, and find that
enforcing interpretability can also increase predictive accuracy under the
right conditions.

The primary contributions of this paper are the following:

  * •

We design a modular framework to enable AI agents to have a theory of mind
model of a human.111Our method is general and supports both humans and
artificial agents as observed entities. However, since our experimental
evaluation is on human data, we only refer to a human as the entity being
observed.

  * •

We present a method of combining neural and non-differentiable components
within our framework.

  * •

We extend this framework for interpretable intent inference with a novel
application of concept whitening [10].

  * •

We present experimental results with human participants and provide both
qualitative and quantitative interpretability analyses.

##  II RELATED WORK

###  II-A Theory of Mind

Theory of mind approaches to agent-modeling have shown to make inferences that
align with those of human observers [2, 11]. An early work from Breazeal et.
al took an approach based on embodied cognition with rule-based inference
using STRIPS-like approaches from the planning literature, and were able to
demonstrate false belief detection [12]. Other similar early approaches also
infer human goals and beliefs in the context of planning with pre-programmed
non-scalable approaches [13, 14]. Bayesian approaches to ToM [2] are also
difficult to scale to larger environments. Rabinowitz et al. used neural
networks to learn latent representations of “mental states”, but only for
artificial agents in small toy environments [3]. Other work concurrent to this
paper has explored theory of mind modeling of human state and behavior with
graph-theoretic approaches for scenarios such as autonomous driving [15].

This paper presents a neural ToM approach that supports reasoning about
humans, and provides experimental results within a complex environment
modeling actual human decision making. In addition, we explicitly incorporate
interpretability into our approach while still maintaining the performance and
scalability benefits of neural approaches.

###  II-B Imitation Learning

Theory of mind is closely related to imitation learning and inverse
reinforcement learning in that both attempt to model other agents, and both
can be applied to model either human or artificial agents [16]. Although most
imitation learning methods do not consider interpretability, there are
approaches such as InfoGAIL that take a heuristic approach towards
interpretability by using information theoretic losses to enforce structure on
the latent space and discover modes in observed behavior [17]. Others use
human-readable programmatic polices for agent-modeling rather than black-box
deep networks [18]. The approach presented in this paper falls under the
umbrella of behavioral cloning [19, 20], with additional constraints for
modeling internal states via theory of mind and for interpretable learning.

###  II-C Deep Interpretability

Saliency-based methods [21, 22] are the most popular means of providing post-
hoc explanations, and aim to highlight the most important input features by
assigning importance weights. However, many common saliency methods have been
found to be independent of both the model and the data, and often have similar
resulting explanations for all inputs, which is undesirable [23, 24]. While
saliency maps have also been applied in reinforcement learning contexts [25,
26], operating on low-level input features (e.g. raw pixels) may not coincide
with human ideas of explanation.

Concept-vector post-hoc methods focus on explanations based on higher-level
concepts rather than low-level input features [27, 28], but these also have
their own drawbacks. In general, most post-hoc explanations are based on
assumptions of the latent space that may not hold; for instance, the implicit
assumptions that 1) concepts are (linearly) separable in the latent space, and
that 2) each axis in the latent space only represents a single concept.

Alternatively, others have devised approaches such as concept whitening that
focus on learning models that are interpretable by design [24, 29, 10], i.e.
approaches that shape the latent space during training. These methods have
been used in classification tasks, e.g. in image classification for aligning
the latent space with predefined human-interpretable concepts. In contrast to
these approaches, this paper develops a variation of concept whitening [10]
for modeling decision-making processes (e.g. theory of mind, imitation
learning, reinforcement learning).

##  III NEURAL ToM FRAMEWORK

###  III-A Purpose

It is important to note that the ToM framework is used by an observer to infer
mental states of an observed entity, but not to define the entity’s observable
behavior. The action predictions produced by the ToM model can be treated as a
policy to forecast the entity’s future behavior, and imitation learning can
provide a training signal for the ToM model, but our purpose is not to train
an agent to perform a task.

###  III-B Overview

Figure 1: Modular theory of mind (ToM) framework.

We developed a modular theory of mind framework for an observing agent to use
to infer the mental state of an observed human. As shown in Figure 1, we model
the decision making process of an observed entity as follows:

##### Belief Model

The belief model $m:\mathcal{B}\times\mathcal{O}\rightarrow\mathcal{B}$ is
responsible for updating the belief state. Given current belief
$b\in\mathcal{B}$ and observation $o\in\mathcal{O}$, the belief model $m$
outputs an updated belief state $b_{new}=m(b,o)$.

##### Desire Model

The desire model $g:\mathcal{B}\rightarrow\mathcal{I}$ is responsible for
calculating the intent $z\in\mathcal{I}$ given updated belief $b$; that is,
$g(b)=z$.

##### Action Model

The action model $f:\mathcal{B}\times\mathcal{I}\rightarrow\mathcal{A}$ is
responsible for generating or predicting an action $a\in A$, given the belief
state $b$ and intent $z$; that is, $f(b,z)=a$.

The modules in our framework impose an inductive bias that reflects the BDI
model of agency in folk psychology [1]. Also, as we later describe,
modularizing the framework in this way allows for combining heuristic and
data-driven components (e.g., neural networks).

###  III-C Combining Differentiable and Heuristic Components

Practically speaking, we need to make sure that the belief space $\mathcal{B}$
corresponds to something that can be understood as belief, that the intent
space $\mathcal{I}$ corresponds to what can be understood as intent, etc. If
we were to implement all three components with neural networks and train end-
to-end, there is nothing that constrains the intermediate outputs to
correspond to what we expect (e.g. no way to ensure the desire model output
actually represents an “intent”).

We can mitigate this difficulty by imposing additional structure on the
pipeline. The simplest way to do this is to replace one or more of the models
with rule-based models, and/or to impose structural constraints on the
input/output space of these models. In this paper we are primarily interested
in inferring intent, so we choose to model the desire model with a neural
network, while modeling the belief and action models with rule-based
heuristics.

Given a planning task we structure the belief state as a grid/graph of
locations, and use a rule-based belief model to update this belief state given
an observation. Additionally, we replace the action model with A* search, and
structure the intents from the desire model as locations of subgoals.

In the setup described above, the belief and action models are rule-based, and
the desire model is the sole trainable component. The rule-based belief model
does not pose any issue with respect to the differentiability of the pipeline
(one can think of it as simply preprocessing the input observation). However,
we cannot optimize for the final output in any gradient-based way, as the
output of the desire model is the input to the non-differentiable action
model, which produces the final output. Unless we have ground-truth intents,
training the desire model becomes difficult.

###  III-D Inverse Action Model

Given belief state $b\in\mathcal{B}$, observed action $a\in\mathcal{A}$, a set
of intents $\mathcal{I}$, and non-differentiable action model
$f:\mathcal{B}\times\mathcal{I}\rightarrow\mathcal{A}$, we want to learn a
desire model $g$ to model a conditional distribution $p(z\mid b)$ such that

$\displaystyle\mathbb{E}[a\mid b]=\mathbb{E}_{z\sim g(b)}[f(b,z)\mid b]$

where $z\in\mathcal{I}$ is the intent. However, we may not have access to any
samples from such a distribution (i.e. no ground truth $z\in\mathcal{I}$ for
given $b,a$ pairs).

Alternatively, we can learn to model the distribution ${p(z\mid b,a)}$ with an
“inverse action model” $h$. This density of this distribution is proportional
to

$\displaystyle p(z\mid b,a)$ $\displaystyle\propto p(a,z\mid b)=p(a\mid
b,z)\cdot p(z\mid b)$ (1)

Because we have direct access to rule-based action model $f$ we can sample
from $p(a\mid b,z)$, and thus given some kind of prior $p(z\mid b)$ we can
sample from $p(z\mid b,a)$ to learn $h$.

Once we have learned an inverse action model $h$, then for each belief-action
pair $(b,a)$ we can then simply use $h$ to sample intents from $p(z\mid b,a)$,
and use these sampled intents to train the desire model $g$ in a supervised
manner.

###  III-E Training

Training is done in two stages: the first stage trains the inverse action
model, the second stage trains the desire model. In each stage, once we gather
the necessary data, we train using stochastic gradient descent (SGD).

####  III-E1 Training Inverse Action Model

To train the inverse action model, we first collect belief states by
sequentially running observations from human trajectories through the rule-
based belief model and storing the resulting belief states at each timestep.
These trajectories can be from human participants or potentially even from
artificial agents trained to perform the task.

Now that we have collected these belief states, for each belief state $b$, we
sample an intent $z$ given some prior $p(z\mid b)$, and create a set of $b,z$
pairs (for more implementation details on sampling and the prior, see the
Appendix). Next, for each belief-intent pair, we generate an action
$a=f(b,z)$, creating a dataset of belief-action-intent triples $(b,a,z)$.
Finally, we train the inverse action model on this dataset to predict intents
given beliefs and actions; that is, using our generated dataset of $(b,a,z)$
tuples, given $b,a$, predict $z$.

Pseudocode for this training process is provided in Algorithm 1.

####  III-E2 Training Desire Model

To train the desire model, we first collect belief states by running the
observations from human trajectories through the belief model. We also store
the corresponding observed actions for each belief state. We can then generate
belief-intent pairs for the desire model by sampling intents from our inverse
action model $z\sim h(b,a)$ for each belief-action pair. The target intents
are formed by combining a) the probability distribution from the inverse
action model over the previous belief state, and if available b) the next
realized intent (from the future, in a post-hoc manner). Finally we train the
desire model on this data to predict intent given belief.

Pseudocode for this process is provided in Algorithm 2.

Input : set of human trajectories $\mathcal{T}=\\{\tau_{1},\tau_{2},\dots\\}$,
belief model $s$, action model $f$

$\mathcal{D}_{baz}\rightarrow\emptyset$

for  _$\tau\in\mathcal{T}$_ do

$o_{1},\dots,o_{n}\rightarrow\tau$

$b_{0}\rightarrow\textrm{Uniform}$

for  _$t=1,\dots,n$_ do

$b_{t}\rightarrow s(b_{t-1},o_{t})$

for  _$i=1,\dots,m$_ do

$z_{t}\sim p(z\mid b_{t})$

$a=f(b_{t},z_{t})$

$\mathcal{D}_{baz}=\mathcal{D}_{baz}\cup\\{(b_{t},a,z_{t})\\}$

end for

end for

end for

Initialize neural network parameters $\theta_{h}$

Use SGD to train $h(b,a\mid\theta_{h})$ on dataset $\mathcal{D}_{baz}$

Algorithm 1 Training inverse action model

Input : set of human trajectories $\mathcal{T}=\\{\tau_{1},\tau_{2},\dots\\}$,
belief model $s$, inverse action model $h$

$\mathcal{D}_{bz}\rightarrow\emptyset$

for  _$\tau\in\mathcal{T}$_ do

$(o_{1},a_{1}),\dots,(o_{n},a_{n})\rightarrow\tau$

$b_{0}\rightarrow\textrm{Uniform}$

for  _$t=1,\dots,n$_ do

$b_{t}\rightarrow s(b_{t-1},o_{t})$

$z_{t}\sim h(b_{t},a_{t})$

$\mathcal{D}_{bz}=\mathcal{D}_{bz}\cup\\{(b_{t},z_{t})\\}$

end for

end for

Initialize neural network parameters $\theta_{g}$

Use SGD to train $g(b\mid\theta_{g})$ on dataset $\mathcal{D}_{bz}$

Algorithm 2 Training desire model Figure 2: Encoder-decoder architecture used
for the desire and inverse-action models in our experiments, inspired by
U-Nets for image-segmentation [30]. Blue indicates convolutional layers, green
indicates pooling, brown indicates up-sampling, and yellow is a final linear
layer. A more detailed description can be found in the appendix.

##  IV CONCEPT WHITENING

Concept whitening (CW) is a mechanism introduced by Chen et al. [10] for
modifying neural network layers to increase interpretability. Broadly, it aims
to enforce structure on the latent space by aligning the axes with predefined
human-interpretable concepts. While this technique was developed for the
purpose of image classification, here we adapt the idea in the context of
intent inference with the desire model. By explicitly defining a set of
concepts that can serve as “explanations” for intent inferences, we can use
concept whitening to allow for interpretability via identification of the most
important concepts for any inference.

We also note that although we consider concept whitening in the context that
can broadly be categorized as behavioral cloning, our approach to
interpretable agent-modeling is framework-agnostic and could potentially be
applied to other reinforcement learning and imitation learning contexts.

###  IV-A Technical Details

Given latent representation $\mathbf{Z}\in\mathbb{R}^{n\times d}$, let
$\mathbf{Z_{C}}\in\mathbb{R}^{n\times d}$ be the mean-centered latent
representation. We can calculate the ZCA-whitening matrix
$\mathbf{W}\in\mathbb{R}^{d\times d}$ as in [31], and thus decorrelate and
standardize the data via whitening operation $\psi$:

$\psi(\mathbf{Z})=\mathbf{WZ_{C}}=\mathbf{W}(\mathbf{Z}-\mu\mathbf{1}^{\top})$
(2)

where $\mu=\frac{1}{n}\sum_{i=1}^{n}\mathbf{z}_{i}$ is the latent sample mean.

Now say we are given concepts $c_{1}\dots c_{k}$ that can be characterized by
corresponding auxiliary datasets $\mathbf{X}_{c_{1}}\dots\mathbf{X}_{c_{k}}$,
and assume we have an orthogonal matrix $\mathbf{Q}\in\mathbb{R}^{d\times d}$
such that the data from $\mathbf{X}_{c_{j}}$ has high activation on the $j$-th
axis (i.e. column $\mathbf{q}_{j}$). Then the concept-whitened representation
is given by:

$\hat{\mathbf{Z}}=\mathbf{Q^{\top}WZ_{C}}$ (3)

Training alternates between optimizing for the main objective (i.e. the
network’s final output) and optimizing the orthogonal matrix $\mathbf{Q}$ for
concept-alignment. To optimize $\mathbf{Q}$, we maximize the following
objective:

$\max_{\mathbf{q_{1}}\dots\mathbf{q_{k}}}\sum_{j=1}^{k}\frac{1}{n_{j}}\sum_{\mathbf{x}_{c_{j}}\in\mathbf{X}_{c_{j}}}\mathbf{q}_{j}^{\top}\hat{\mathbf{z}}_{\mathbf{x}_{c_{j}}}$
(4)

where $\hat{\mathbf{z}}_{\mathbf{x}_{c_{j}}}$ denotes the concept-whitened
latent representation in the model on data sample from concept $c_{j}$.
Orthogonality can be maintained when optimization is performed via gradient
descent and curvilinear search on the Stiefel manifold [32]. A more detailed
description of concept whitening and the optimization algorithm can be found
in [10].

###  IV-B Concept Whitening for Intent Prediction

We can modify this idea to the context of explanatory concepts for inferring
intents. Specifically, we consider the desire model (Fig. 2) and insert a
concept whitening layer (for more detail see the appendix and Fig. 3).

First we define a set of concepts $C=\\{c_{1},\dots,c_{k}\\}$; these concepts
should correspond to appropriate human-interpretable reasons or “explanations”
for intent prediction given the problem domain. We also must be able to
identify a subset of timesteps from our trajectories where each concept
applies, either directly from the trajectory data, or from external labels.

Recall that the desire model’s inputs are belief states, which we can generate
sequentially by passing the observations from each trajectory timestep through
the belief model. Then for each concept $c_{j}$ we consider only the belief
states from the timesteps where $c_{j}$ is known to apply, and aggregate them
into auxiliary dataset $\mathbf{B}_{c_{j}}$.

Then training alternates between:

  1. 1.

Optimizing for intent prediction, given a belief state and a ground truth
intent

  2. 2.

Concept-aligning the CW orthogonal matrix $\mathbf{Q}$ by maximizing the
activation along axis $j$ for each auxiliary dataset $\mathbf{B}_{c_{j}}$

Pseudocode for this process is provided in Algorithm 3.

$\mathcal{D}\rightarrow\emptyset$

for  _$\tau\in\mathcal{T}$_ do

$(o_{1},a_{1}),\dots,(o_{n},a_{n})\rightarrow\tau$

$b_{0}\rightarrow\textrm{Uniform}$

for  _$t=1,\dots,n$_ do

$b_{t}\rightarrow\textrm{BM}(b_{t-1},o_{t})$

$z_{t}\sim\textrm{IAM}(b_{t},a_{t})$

$\mathcal{D}=\mathcal{D}\cup\\{(b_{t},z_{t})\\}$

end for

end for

for  _$e=1,...,\mathit{num\\_epochs}$_ do

Train DM on $\mathcal{D}$ with gradient descent

if  _$e\mod 5=0$_ then

for  _$j=1,...,k$_ do

Maximize activation of $\mathbf{B}_{c_{j}}$ on the $j$-th column of
$\mathbf{Q}$ (see [10])

end for

end if

end for

Algorithm 3 Training desire model with concept whitening Figure 3: Desire
model with concept whitening.

##  V EXPERIMENTS

###  V-A Task Domain

We consider a simulated search and rescue task in a Minecraft environment. The
scenario simulates a damaged building after a disaster, with areas of the
building layout perturbed with collapsed rubble, wall openings, and fires.
There are 34 injured victims within the building who will die if left
untreated. For convenience and simplicity, victims are represented as blocks.
Out of these victims, 10 of these are critically injured and will expire after
5 minutes. These critical victims take 15 seconds to triage and are worth 30
points each. Other victims are considered “non-critical”, but will expire
after 10 minutes. Non-critical victims take 7.5 seconds to triage and are
worth 10 points each. The goal of the task is to earn as many points as
possible within a 10 minute mission.

###  V-B Human Data For Training and Evaluation

All experiments are performed using a set of 75 trajectories previously
collected from human participants [33]. Prior to each mission, participants
were given information on the task and the original building layout. However,
the knowledge conditions of certain participants were manipulated by partially
withholding information. Some participants were not informed of the cost-
benefit tradeoffs (i.e. the knowledge that critical victims take 15 seconds to
rescue and are worth 30 points and non-critical victims take 7.5 seconds to
rescue and are worth 10 points). Knowledge was also manipulated via a beep
signal that activated whenever the participant was near a room with a victim
(1 beep for non-critical victim, 2 beeps for critical victim); certain
participants were not told the meaning of the signal.

Participants were run in 3 knowledge conditions:

  1. 1.

No knowledge of critical-victim tradeoff, no knowledge of signal meaning

  2. 2.

Knowledge of critical-victim tradeoff but not of signal meaning

  3. 3.

Knowledge of both critical-victim tradeoff and signal meaning

###  V-C Intent Prediction

We represent intents as $(x,y)$ positions the participant intends to navigate
towards. Specifically, we consider victims, doors, and room openings as
locations-of-interest, which frames the intent prediction task as predicting
either the next room to be visited or the next victim to be triaged. The
predictions are accumulated at each timestep ($\sim$ 0.5 seconds per timestep)
between visits of locations-of-interest, and then their mode is evaluated
against the ground truth. We evaluate on a held-out test set of 20% of
participant trajectories.

###  V-D Concepts

We defined a set of 10 concepts related to mission timer, knowledge condition,
and field of view (see Table I). We consider 3 subsets:

  * •

Concept Set I is the full concept set

  * •

Concept Set II omits the knowledge condition concepts

  * •

Concept Set III omits both knowledge condition and mission timer concepts

TABLE I: Concepts Concept | Concept Sets
---|---
Mission timer between 0-3 minutes | I, II
Mission timer between 3-5 minutes | I, II
Mission timer between 5-8 minutes | I, II
Mission timer $>$ 8 minutes | I, II
Knowledge condition 1 (no triage, no signal) | I
Knowledge condition 2 (triage, no signal) | I
Knowledge condition 3 (triage, signal) | I
Door / opening in field of view | I, II, III
Non-critical victim in field of view | I, II, III
Critical victim in field of view | I, II, III

The field of view and mission timer concepts were labeled directly from the
data; the knowledge condition concepts are labeled with external knowledge of
the condition for each participant trajectory.

###  V-E Results

We compare the accuracy of ToM model intent predictions under 3 methods:
training without Concept Whitening (CW), training from scratch with CW, and
transfer learning by initializing a CW model with the weights of a pretrained
non-CW model. Results are provided in Table II, where we see that introducing
concept whitening for interpretability actually results in increased accuracy
of the model. The inductive bias produced by shaping the latent space to
correspond to our selected concepts improves performance.

TABLE II: Intent Prediction Performance  Training Method | Intent Prediction Accuracy
---|---
Without CW | 73.0%
CW | 84.0%
CW + Transfer | 84.1%

###  V-F Concept Ablation

We also tested the effect of concept selection on performance (Table III). In
particular, we omitted the knowledge condition (KC) concepts and / or the
mission timer concepts, tested concept-whitened ToM models both with and
without transfer, and found noticeably diminished performance.

Compared to the non-CW model, CW with reduced concept sets resulted in worse
performance, and while transfer from the non-CW model somewhat mitigated this
effect, we still see a significant drop from the performance with full concept
set. This demonstrates the importance of good concept selection for the
resulting performance of concept-whitened ToM model.

TABLE III: Varying Concept Sets Training Method | Concept Set | Acc.
---|---|---
Without CW | N/A | 73.0%
CW | III | 41.2%
CW | II | 69.2%
CW | I | 84.0%
CW + Transfer | III | 54.9%
CW + Transfer | II | 77.9%
CW + Transfer | I | 84.1%

##  VI INTERPRETABILITY ANALYSIS

Generally, using a deep model for a task such as intent inference makes
interpreting model output difficult due to the “black-box” nature of neural
networks. Adding the concept whitening module provides us with a direct
mechanism that allows us to interpret the model output in terms of our chosen
concepts. For each observation and corresponding intent prediction made by our
ToM model, we can calculate the importance of each concept.

For example, if our ToM model observes that a player hears a beep and predicts
the player intends to bypass a particular door, we can examine the relative
importance of concepts such as “critical victim in field of view” or “no
knowledge of the beep signal’s meaning” or “over 8 minutes elapsed in mission
timer” for our model’s prediction. If for instance the most important concept
is “no knowledge of beep signal’s meaning”, an appropriate intervention could
be to inform the player of the signal’s meaning.

###  VI-A Qualitative Analysis

We can estimate the concept importance for each prediction via the activation
for each column of the CW orthogonal matrix $\mathbf{Q}$, given by:

$a_{j}=\mathbf{q}_{j}^{\top}\hat{\mathbf{z}}_{b}$ (5)

where $\hat{\mathbf{z}}_{\mathbf{x}}$ is the concept-whitened latent
representation for belief state $b$.

We can examine the activation vectors $\mathbf{a}=[a_{1}\dots a_{k}]$ for
different types of intent predictions by the learned model (CW + transfer,
full concept set). The mean normalized activations for non-critical victims,
critical victims, and doors / openings are visualized in Fig. 4.

These largely line up with intuition; unsurprisingly, the presence of an
intent-relevant entity in the field of view is an important concept for the
model’s prediction of said intent. We also see variability in the importance
of different mission time intervals and of knowledge condition for different
intent predictions.

(a) Mean concept activation for intent prediction of non-critical victim. We
can see that the presence of a non-critical victim in the field-of-view is the
most activated concept.

(b) Mean concept activation for intent prediction of critical victim. Here we
see zero activation for mission timer above 5 minutes (which corresponds with
critical victims expiring). We also see that the presence of a critical victim
or room opening in field-of-view is a common reason for predicting intent to
triage a critical victim.

(c) Mean concept activation for intent prediction of opening. The presence of
an opening in the field of view is the most highly activated concept. We also
see that compared to the other mission timer concepts, the last 2 minutes sees
the timer become a more important reason for predicting intent to go towards a
door or opening.

Figure 4: Mean concept activations for different intent prediction types.

###  VI-B Quantitative Analysis

We also attempt to quantitatively assess how well the concept activation
vectors characterize the ToM model’s intent inferences. Intuitively, we should
be able to deduce information about what the model’s prediction will be with
high accuracy, using only concept activation vectors as input.

This can be framed as a classification problem where, given an activation
vector, we predict the type of the corresponding predicted intent as one of:
non-critical victim, critical victim, or opening (i.e. door or hole in the
wall between rooms). Rather than use a complex model, we instead learn simpler
models – a decision tree and an SVM – and use the accuracy as a proxy for the
quality of our concept activations. As shown in Table IV, we achieve
relatively high accuracies with these simple models.

TABLE IV: Classifiying Activations as Intents Model | Accuracy
---|---
Decision Tree | 93.0%
SVM | 92.0%

##  VII CONCLUSION

We have presented a modular ToM approach for reasoning about humans that can
allow for both neural and heuristic components. Our approach explicitly
incorporates interpretability while still maintaining the performance and
scalability advantages of neural approaches. We move beyond simple toy
environments and apply our framework to a more complex setting. Our
experimental results demonstrate that enforcing interpretability can also
increase predictive accuracy.

The natural extension of this work is in exploring the benefits of ToM and
interpretability in assistance and interventions. A particularly interesting
direction would be to explore counterfactuals. This means examining how intent
inference changes given a change in concept activation, and then finding the
closest belief state that could result in the given change. Approaching this
through interpretable concept activations rather than in the belief space
could facilitate interventions to warn about or correct human errors when
working in Human + AI teams.

Another potential avenue of future work is in artificial agents explaining
their own decisions and interventions in terms of human-interpretable
concepts. We hope that this work can serve as a starting point for social
intelligence in AI systems that are designed to interact with humans.

## APPENDIX

###  VII-A Implementation Details

We consider Minecraft environments that can effectively be projected onto 2D
grid representations (e.g., building interiors). The specification for each of
the framework components is given below.

##### Observations

Observations are represented $X\times Y$ grids (where $X$ and $Y$ are
dimensions of the environment), and each $(x,y)$ coordinate contains one of
$K$ different block types.

##### Belief States

Each belief state $b$ is represented by a $X\times Y\times K$ grid, where the
value at $(x,y,k)$ represents the probability of the block at position $(x,y)$
having block type $k$.

##### Belief Model

We use a rule-based belief model that aggregates observations into our belief
state with probability 1, and decay probabilities over time to a uniform
distribution over block types by $b\rightarrow\frac{b+\epsilon}{1+K\epsilon}$
after each timestep, where $b$ is a belief state grid, $K$ is the number of
block types, and $\epsilon$ is a forgetfulness hyperparameter we set to
$0.01$.

##### Intents

We represent each intent as an $(x,y)$ position the player intends to navigate
towards.

##### Intent Prior

When generating data to train the inverse action model, for each belief state
$b$, we sample an intent $(x,y)$ given some prior $p(x,y|b)$, and create a set
of $b,(x,y)$ pairs. We specifically use the prior
$p(x,y|b)=\frac{1}{d_{b}(x,y)}$ if we belief a victim or door is at position
$(x,y)$, and 0 otherwise, where $d_{b}(x,y)$ is the L1 distance of point
$(x,y)$ from the player’s position.

##### Action Model

We use A* search as our action model, $\textrm{A}^{*}(b,(x,y))=a$, where $b$
is a belief state, $(x,y)$ represents the intent, and $a$ is an action from
discrete action set of: left_turn, right_turn, toggle_door, toggle_lever,
triage, or None.

###  VII-B Neural Architectures

##### Inverse Action Model

The inverse action model takes as input a belief state $b$ and an action $a$.
It outputs an $X\times Y$ grid of log-probabilities for the intent at each
grid cell. It is designed as an encoder-decoder model, inspired by image-
segmentation approaches, and uses the following architecture (Fig 2):

  * •

3 encoder blocks each consisting of a convolutional layer, followed by max-
pooling, ReLU activation and batch norm

  * •

A bottleneck layer, concatenating the downsampled input and action before
passing through a linear layer

  * •

3 decoder blocks each consisting of:

    * –

a deconvolutional upsampling layer

    * –

a residual connection with the output of the corresponding encoder block

    * –

a convolutional layer, followed by ReLU activation and batch norm

##### Desire Model

The desire model takes a belief state $b$ as input. It outputs an $X\times Y$
grid of log-probabilities for the intent at each grid cell. Its architecture
(Fig 2) is identical to that of the inverse action model, except without
concatenating the action in the bottleneck layer. When training concept
whitening, we replace batch normalization after the bottleneck layer with a
concept whitening layer.

## References

  * [1] M. Georgeff, B. Pell, M. Pollack, M. Tambe, and M. Wooldridge, “The belief-desire-intention model of agency,” in _International workshop on agent theories, architectures, and languages_. Springer, 1998, pp. 1–10.
  * [2] C. Baker, R. Saxe, and J. Tenenbaum, “Bayesian theory of mind: Modeling joint belief-desire attribution,” in _Proceedings of the annual meeting of the cognitive science society_ , vol. 33, 2011.
  * [3] N. Rabinowitz, F. Perbet, F. Song, C. Zhang, S. A. Eslami, and M. Botvinick, “Machine theory of mind,” in _International conference on machine learning_. PMLR, 2018, pp. 4218–4227.
  * [4] F. Cuzzolin, A. Morelli, B. Cirstea, and B. J. Sahakian, “Knowing me, knowing you: Theory of mind in ai,” _Psychological medicine_ , vol. 50, no. 7, pp. 1057–1061, 2020.
  * [5] J. W. Astington and M. J. Edward, “The development of theory of mind in early childhood,” _Encyclopedia on early childhood development_ , vol. 14, pp. 1–7, 2010.
  * [6] R. Choudhury, G. Swamy, D. Hadfield-Menell, and A. D. Dragan, “On the utility of model learning in hri,” in _2019 14th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_. IEEE, 2019, pp. 317–325.
  * [7] J. N. Foerster, R. Y. Chen, M. Al-Shedivat, S. Whiteson, P. Abbeel, and I. Mordatch, “Learning with opponent-learning awareness,” in _AAMAS 2018_ , 2018, pp. 122–130.
  * [8] Y. Wen, Y. Yang, R. Luo, J. Wang, and W. Pan, “Probabilistic recursive reasoning for multi-agent reinforcement learning,” in _7th International Conference on Learning Representations, ICLR 2019_ , 2019.
  * [9] R. Shah, N. Gundotra, P. Abbeel, and A. Dragan, “On the feasibility of learning, rather than assuming, human biases for reward inference,” in _International Conference on Machine Learning_. PMLR, 2019, pp. 5670–5679.
  * [10] Z. Chen, Y. Bei, and C. Rudin, “Concept whitening for interpretable image recognition,” _Nature Machine Intelligence_ , vol. 2, no. 12, pp. 772–782, 2020.
  * [11] B. Scassellati, “Theory of mind for a humanoid robot,” _Autonomous Robots_ , vol. 12, no. 1, pp. 13–24, 2002.
  * [12] C. Breazeal, J. Gray, and M. Berlin, “An embodied cognition approach to mindreading skills for socially intelligent robots,” _The International Journal of Robotics Research_ , vol. 28, no. 5, pp. 656–680, 2009.
  * [13] J. Oh, F. Meneguzzi, and K. Sycara, “Probabilistic plan recognition for proactive assistant agents,” _Plan, activity, and intent recognition._ , vol. 10, p. 23, 2014.
  * [14] J. Oh, F. Meneguzzi, K. Sycara, and T. J. Norman, “Prognostic normative reasoning,” _Engineering Applications of Artificial Intelligence_ , vol. 26, no. 2, pp. 863–872, 2013.
  * [15] R. Chandra, A. Bera, and D. Manocha, “Stylepredict: Machine theory of mind for human driver behavior from trajectories,” _arXiv preprint arXiv:2011.04816_ , 2020.
  * [16] J. Jara-Ettinger, “Theory of mind as inverse reinforcement learning,” _Current Opinion in Behavioral Sciences_ , vol. 29, pp. 105–110, 2019.
  * [17] Y. Li, J. Song, and S. Ermon, “Infogail: interpretable imitation learning from visual demonstrations,” in _NIPS 2017_ , 2017, pp. 3815–3825.
  * [18] A. Verma, V. Murali, R. Singh, P. Kohli, and S. Chaudhuri, “Programmatically interpretable reinforcement learning,” in _International Conference on Machine Learning_. PMLR, 2018, pp. 5045–5054.
  * [19] M. Bain and C. Sammut, “A framework for behavioural cloning.” in _Machine Intelligence 15_ , 1995, pp. 103–129.
  * [20] F. Torabi, G. Warnell, and P. Stone, “Behavioral cloning from observation,” in _IJCAI 2018_ , 2018, pp. 4950–4957.
  * [21] M. T. Ribeiro, S. Singh, and C. Guestrin, “” why should i trust you?” explaining the predictions of any classifier,” in _Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining_ , 2016, pp. 1135–1144.
  * [22] S. M. Lundberg and S. Lee, “A unified approach to interpreting model predictions,” in _NIPS 2017_ , 2017, pp. 4765–4774.
  * [23] J. Adebayo, J. Gilmer, M. Muelly, I. J. Goodfellow, M. Hardt, and B. Kim, “Sanity checks for saliency maps,” in _NeurIPS 2018_ , 2018, pp. 9525–9536.
  * [24] C. Rudin, “Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead,” _Nature Machine Intelligence_ , vol. 1, no. 5, pp. 206–215, 2019.
  * [25] R. Iyer, Y. Li, H. Li, M. Lewis, R. Sundar, and K. Sycara, “Transparency and explanation in deep reinforcement learning neural networks,” in _Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society_ , 2018, pp. 144–150.
  * [26] R. M. Annasamy and K. Sycara, “Towards better interpretability in deep q-networks,” in _Proceedings of the AAAI Conference on Artificial Intelligence_ , vol. 33, no. 01, 2019, pp. 4561–4569.
  * [27] B. Kim, M. Wattenberg, J. Gilmer, C. Cai, J. Wexler, F. Viegas, _et al._ , “Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (tcav),” in _International conference on machine learning_. PMLR, 2018, pp. 2668–2677.
  * [28] B. Zhou, Y. Sun, D. Bau, and A. Torralba, “Interpretable basis decomposition for visual explanation,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , 2018, pp. 119–134.
  * [29] P. W. Koh, T. Nguyen, Y. S. Tang, S. Mussmann, E. Pierson, B. Kim, and P. Liang, “Concept bottleneck models,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 5338–5348.
  * [30] O. Ronneberger, P. Fischer, and T. Brox, “U-net: Convolutional networks for biomedical image segmentation,” in _International Conference on Medical image computing and computer-assisted intervention_. Springer, 2015, pp. 234–241.
  * [31] L. Huang, Y. Zhou, F. Zhu, L. Liu, and L. Shao, “Iterative normalization: Beyond standardization towards efficient whitening,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 4874–4883.
  * [32] Z. Wen and W. Yin, “A feasible method for optimization with orthogonality constraints,” _Mathematical Programming_ , vol. 142, no. 1, pp. 397–434, 2013.
  * [33] L. Huang, J. Freeman, N. Cooke, M. Cohen, X. Yin, J. Clark, M. Wood, V. Buchanan, C. Carrol, F. Scholcover, A. Mudigonda, L. Thomas, A. Teo, M. Freiman, J. Colonna-Romano, L. Lapujade, and K. Tatapudi, “Using humans’ theory of mind to study artificial social intelligence in minecraft search and rescue,” in _(to be submitted to the) Journal of Cognitive Science_ , 2021.