# Smart Help: Strategic Opponent Modeling for
Proactive and Adaptive Robot Assistance in Households

Zhihao Cao1,2, , Zidong Wang1,2,∗, Siwen Xie3, Anji Liu4, Lifeng
Fan${}^{1,\textrm{\Letter}}$
1 State Key Laboratory of General Artificial Intelligence, Beijing Institute
for General Artificial Intelligence (BIGAI)
2 Department of Automation, Tsinghua University 3 Yuanpei College, Peking
University
4 Computer Science Department, University of California, Los Angeles
Indicates equal contribution. Work was conducted during Zhihao Cao and Zidong
Wang’s internships at BIGAI. 🖂 Corresponding author: Lifeng Fan
(lifengfan@bigai.ai). Our environment, dataset, and codes are available at
https://github.com/caozh20/SmartHelp.

###### Abstract

Despite the significant demand for assistive technology among vulnerable
groups (e.g., the elderly, children, and the disabled) in daily tasks,
research into advanced AI-driven assistive solutions that genuinely
accommodate their diverse needs remains sparse. Traditional human-machine
interaction tasks often require machines to simply help without nuanced
consideration of human abilities and feelings, such as their opportunity for
practice and learning, sense of self-improvement, and self-esteem. Addressing
this gap, we define a pivotal and novel challenge Smart Help, which aims to
provide proactive yet adaptive support to human agents with diverse
disabilities and dynamic goals in various tasks and environments. To establish
this challenge, we leverage AI2-THOR [32] to build a new interactive 3D
realistic household environment for the Smart Help task. We introduce an
innovative opponent modeling module that provides a nuanced understanding of
the main agent’s capabilities and goals, in order to optimize the assisting
agent’s helping policy. Rigorous experiments validate the efficacy of our
model components and show the superiority of our holistic approach against
established baselines. Our findings illustrate the potential of AI-imbued
assistive robots in improving the well-being of vulnerable groups.

##  1 Introduction

Figure 1:  An example of the Smart Help strategy for an assistive robot. The
top left shows the human user’s capability distribution across several
dimensions (e.g., weight for lifting heavy objects and height for grasping
objects from high positions), as well as the current task (e.g., making
coffee). The figure illustrates three types of helpers: (1) The  Basic Helper
merely observes the human user without inferring anything, and thus remains
idle and waits until the human user fails the task, leading to frustration.
(2) The Ordinary Helper infers the human user’s goals through observations and
always provides simple and direct assistance for each goal, taking over the
entire task. While the task is successfully completed, the human user is left
dissatisfied because the robot’s helping strategy is overly intrusive, causing
discomfort. (3) The  Smart Helper infers both the human user’s goals and her
capability distribution. It reasons about whether the human user is capable of
independently accomplishing each goal and thus identifies the bottleneck goal
that hinders task completion. The proactive helper then adapts its strategy to
assist solely with the bottleneck goal. As a result, the human user
successfully completes the task with tailored support, feeling comfortable and
satisfied.

All of us may find ourselves within vulnerable demographics at some point.
Throughout the human life span, we confront an array of challenges, whether
originating from physical discomfort [12], emotional turmoil [53], or the
inevitable march of aging [22], that can hinder our ability to perform even
the simplest tasks that we once accomplished effortlessly (e.g., lifting a
heavy object). This predicament often triggers a complex emotional response.
The fear of being labeled as “special” or “disabled” can evoke feelings of
diminished self-esteem, reduced self-efficacy and self-sufficiency, and a
sense of personal boundary violation [8, 40]. However, previous research in
human-robot interaction has primarily focused on pure cooperation [30, 57, 38,
42, 43, 52], and assistive technologies [4, 7] have been designed to simply
take over everything for human users, often disregarding their emotional well-
being. Considering the importance of sensitivity and consideration when
offering help, we need new assistance technologies with an emphasis not only
on the successful completion of the task but also on the recipient’s emotional
acceptance of the assistance [8, 40]. This introduces a new dimension to the
concept of learning to help in AI, extending the boundaries beyond technical
proficiency and into the realm of empathetic engagement.

To tackle this issue, we propose a novel and pivotal challenge Smart Help,
aiming to provide both proactive and adaptive support to human agents with
diverse disabilities and dynamic goals in different tasks and environments.
Fig. 1 exemplifies the concept of the Smart Help strategy as applied to an
assistive robot aiding a human in a household task. The Basic Helper fails to
provide proactive assistance due to its inability to infer the user’s goals
accurately. The Ordinary Helper successfully infers the user’s goals from
observations and offers proactive help, but it lacks adaptability to different
user needs. Its simple “take-over” helping strategy can cause a feeling of
discomfort in the human user. In contrast, the Smart Helper not only infers
both the user’s goals distribution but also reasons about the user’s
capability of completing each goal independently. As a result, the Smart
Helper can identify critical goals that hinder task completion and provide
proactive and adaptive help that selectively addresses those bottlenecks. This
approach allows the user to accomplish the task successfully and joyfully.

Leveraging AI2-THOR [32], a 3D interactive environment engineered for
realistic home simulations, we demonstrate a concrete realization of the
proposed novel challenge Smart Help. We create a multi-agent interactive
environment featuring a main agent with diverse capability distributions
(representing the vulnerable group) across dimensions, such as toggling,
opening, closing, weight, and height. Within this environment, the main agent
faces challenges in achieving dynamic goals across various tasks, and an
assistive robot is introduced to assist the main agent throughout their
endeavors. Unlike previous assistive tasks that were highly specialized and
constrained, such as the use of a robotic arm for the disabled [10, 51, 17],
our work represents, to the best of our knowledge, the first construction of a
3D home environment designed to assist the vulnerable group with various
general daily household tasks.

Furthermore, we build a new benchmark model for our proposed challenge, which
consists of 1) an opponent modeling module that jointly optimizes goal
inference and capability estimation, and 2) a helping policy module that
reasons about the bottleneck, and learns an optimal Smart Help policy in an
online interactive manner. To better evaluate performance in the Smart Help
task, we also introduce six assessment metrics. Contrary to traditional
evaluation metrics primarily focusing on cumulative reward or task completion,
our metrics emphasize the helper’s contribution to the task and the
essentiality of assistance. Rigorous experiments validate the efficacy of our
model components and show the superiority of our holistic approach against
baselines. We believe our proposed task, environment, model, and benchmark
will contribute to the development of next-generation advanced home assistive
robot technologies.

Our main contributions can be summarized as follows:

  * •

We propose a novel Smart Help challenge that aims at learning to provide both
proactive and adaptive help to diverse human users (especially vulnerable
groups) based on inferred goals and capabilities.

  * •

To the best of our knowledge, we contribute the first 3D realistic home
environment built upon AI2-THOR, that focuses on assisting the vulnerable
group with daily household tasks in an online and interactive manner.

  * •

We provide a benchmark model with a joint goal and capability inference,
bottleneck reasoning, and helping policy improvement. Strict experiments and
the proposed holistic evaluations validate the efficacy of our model.

##  2 Related Work

Assistive Robots. The domain of assistive robots encompasses a broad spectrum
of research, focusing on enhancing the life quality for individuals with
various needs and spanning diverse applications such as mobility aids [1, 20],
companion robots [46], and robotic arms [21]. Feil-Seifer and Mataric [18]
underscore that Socially Assistive Robotics entails robots assisting humans
through effective interaction in tasks such as food delivery [49], healthcare
[27], and other tasks necessitating social interaction. However, many of them
only rely on simple rules or programs to achieve social interaction [49, 27].
The deployment of assistive robots has also raised ethical and safety
considerations, particularly in terms of user autonomy, privacy, acceptance,
trust, etc. [5, 6, 18, 9]. The objective of our work is to devise an effective
algorithm for Socially Assistive Robotics to estimate people’s goals and
capabilities, thereby enabling comfortable and smart assistance.

Embodied Multi-agent Collaboration. A series of embodied collaboration tasks
have been developed recently thanks to the development of Embodied AI
simulators [32, 13, 14, 41, 33, 48, 47, 50]. However, these tasks always focus
on some limited task completion, such as collaborative navigation [56, 34] and
collaborative furniture rearrangement [28, 29]. Watch-and-Help [42] and NOPA
[43] study the cooperative tasks between two agents with the same
capabilities. Our work focuses on the collaboration between agents with
different capabilities and study the strategic assistance for the vulnerable
group on household tasks.

Opponent Modeling. Opponent modeling [24, 19, 16], a pivotal approach in
multi-agent interaction tasks, leverages a variety of methodologies such as
Inverse Reinforcement Learning [36], Bayesian methods [52, 45, 26, 25, 60],
Deep Q-Networks (DQN) [23, 35], Variational Auto-Encoders [39], Markov
Decision Process [36], etc. This technique is especially valuable in
competitive settings like games and strategic decision-making, where
understanding and anticipating an opponent’s behavior or unseen traits (e.g.,
Theory of Mind modeling [44]) can significantly influence the outcome. It
enables agents to predict future environmental state transitions and refine
their strategies accordingly [59, 58]. Projects like Watch-and-Help [42] and
NOPA [43] have explored estimating a main agent’s goals to improve
coordination. Yet, these initiatives often overlook the capabilities of the
main agent, leading to a gap in developing effective support strategies for
those in need. Our research addresses this gap by focusing on enhancing the
acceptance of AI assistance among users, as emphasized in [9], highlighting
the importance of aligning AI functionalities with user needs and preferences.
Additionally, we propose to model the capability of the opponent and learn a
smart, adaptive, and empathetic helping policy for vulnerable people in a
challenging 3D household interactive environment with partial observation and
high uncertainty.

##  3 The Smart Help Challenge

###  3.1 Problem Formulation

We model the interaction between the main agent and the helper with a multi-
agent Partially Observable Markov Decision Process (POMDP[11]), which is
formally defined as a tuple $G=\textless S,A,O,R,T,n,\gamma\textgreater$. $S$
represents the state space, including physical states and mental states. $A$
is the joint action space for $n$ agents, whose local observations compose
observation space $O$. $T(s^{\prime}|s,a)$ denotes the transition probability.
$R(s,a)$ denotes the shared reward function and $\gamma\in[0,1)$ is the
discount factor. Specifically, there are $n=2$ agents in our Smart Help
Challenge, i.e., a main agent and a helper agent. The objective is to train a
helper agent to assist the main agent in achieving goals, considering the
emotional state of the main agent simultaneously.

In previous work, the reward for the helper agent is:

$R_{h}(s,a_{h})=R^{g_{h}}(s,a_{h})+\beta R^{g_{m}}(s,a_{h}),\\\ $ (1)

where the $\beta\in(0,1)$ is a factor controlling the level of altruism of the
helper. $R^{g_{h}}(s,a_{h})$ is the reward attributed to the successful
completion of the helper agent’s goal at state $s$ through action $a_{h}$.
$R^{g_{m}}(s,a_{h})$ is the same for the main agent’s goal achievement. This
formulation assumes equal consideration for both agents when rewarding goal
completion. However, when the helper fulfills a goal of the main agent, it
triggers emotional responses in the main agent. Integrating this emotional
component with Eq. 1, we have:

$\displaystyle R_{h}(s,a_{h})$ $\displaystyle=R^{g_{h}}(s,a_{h})$ (2)
$\displaystyle+\beta(R^{g_{m}}(s,a_{h})+\lambda_{e}R^{e_{m}}(s,a_{h})),$

where $R^{e_{m}}(s,a_{h})$ means the reward correlated to the emotions of the
main agent when the helper does the action $a_{h}$, and $\lambda_{e}$ is a
hyper-parameter that controls the helper’s sensitivity to the emotional states
of the main agent. In our Smart Help challenge, the helper agent’s role is
solely to assist the main agent in accomplishing his/her goals, without its
own goals. So we omit the term $R^{g_{h}}(s,a_{h})$ and set $\beta=1$, leading
to the helper’s reward as:

$R_{h}(s,a_{h})=R^{g_{m}}(s,a_{h})+\lambda_{e}R^{e_{m}}(s,a_{h}).\\\ $ (3)

Indeed, this is what sets smart Help apart from other assistance-related
tasks. Through such design, we emphasize that AI agents must possess the
capacity to discern not just how to aid humans, but also when their assistance
is truly needed and will be valued.

###  3.2 Challenge Implementation

We implement a tangible version of Smart Help in the AI2THOR [32] simulator,
which emphasizes adaptive helping policy for individuals with various
disabilities, as shown in Fig. 1. Our new environment includes a main agent
that simulates human behavior in various assistive household tasks. The main
agent, assigned with a new task and a capability distribution at the beginning
of each round, attempts to complete the task with an expert planning policy,
which utilizes complete information to divide the task into distinct goals and
plan intentional actions for the goals. We also implement a low-level planner
that translates the intentional action to a sequence of executable actions in
the simulator, so as to enable the main agent to reach the target state
indicated by the input intentional action. The workflow of the low-level
planner is a loop involving the following steps:

  1. 1.

Navigate to the relevant object by following the shortest path determined by
the object’s position and the room layout; or be directly teleported to the
target position.

  2. 2.

Interact with the object in accordance with the goal.

However, actions taken by the main agent might fail due to certain
disabilities or the physical constraints of the scene. Thus, the helper agent,
as in Fig. 2, based on its symbolic observations of the world, should infer
both the goal and capability of the main agent and provide proactive
assistance with the bottleneck for the main agent. We will introduce the
detailed environment settings in the following part.

Object State. We define object state of object $i$ as
$e_{i}=(type_{i},pos_{i},w_{i},attr_{i},pr_{i})$, where:

  * •

$type_{i}\in\mathbb{N}$ denotes the type of object $i$ with an index;

  * •

$pos_{i}\in\mathbb{R}^{3}$ records the position of object $i$’s center;

  * •

$w_{i}\in\mathbb{R}$ represents the weight of object $i$;

  * •

$attr_{i}\in\\{0,1\\}^{4+2}$ denotes the attributes of object $i$, including
its 4 status (i.e., whether it is picked up, opened, cooked, and toggled on)
and its visibility to the 2 agents;

  * •

$pr_{i}\in\mathbb{R}$ signifies the type of the parent receptacle of the
object $i$ through an index; here, the parent receptacle means an object that
serves as the container or the support for another subordinate object.

Action Space. We employ intentional actions to construct the action space,
where intentional actions represent the target state that the agent wants to
achieve. It is defined as $a_{i}=(predicate_{i},e_{i})\in A$; for instance,
(PickUp, 3), where 3 is an object index, representing “Bread” in our
environment. There are seven kinds of different intentional actions: (Wait),
(PickUp, $e_{i}$), (Put, $pr_{i}$), (ToggleOn, $e_{i}$), (ToggleOff, $e_{i}$),
(Open, $e_{i}$), and (Close, $e_{i}$). In total, there are $|A|=373$ possible
intentional actions.

Agent Capability. The variance of agent capability is manifested in the
differences in the transition matrix, which means that different agents will
produce different outcomes for the same action at the same state. For example,
some agents may be able to pick an object up, while others cannot due to the
its weight. Learning the matrix itself is impractical due to the vast size of
the space. Instead, we select six parameters to represent the distinct
characteristics of different matrices. Therefore, the capability of agent $i$
is represented as
$\eta_{i}=(\alpha_{i},\beta_{i},\gamma_{i},\delta_{i},\epsilon_{i},\zeta_{i})$
where:

  * •

The maximum height an agent can reach $\alpha_{i}\in[0,1]$ and the maximum
weight an agent can lift $\beta_{i}\in[0,1]$ determine whether an agent can
execute the action PickUp;

  * •

$\gamma_{i},\delta_{i},\epsilon_{i},\zeta_{i}\in[0,1]$ respectively represent
the agent’s ability to complete the actions Open, Close, ToggleOn and
ToggleOff.

Agent State. We define the state of agent $i$ as
$h_{i}=(\eta_{i},pos_{i},rot_{i},e_{i},a_{i},succ_{i})$, wherein:

  * •

$\eta_{i}\in C=[0,1]^{6}$ denotes the capability of agent $i$,

  * •

$pos_{i},rot_{i}\in\mathbb{R}^{3}$ represent agent $i$’s position and
rotation,

  * •

$e_{i}\in E$, $a_{i}\in A$, and $succ_{i}\in\\{0,1\\}$ respectively represent
the object held by agent $i$, the action undertaken by agent $i$ in the last
step, and whether the last action was a success or a failure.

Figure 2:  This figure depicts the architecture of our smart help model and
its interaction mechanisms with the environment. The model is partitioned into
two primary components: an opponent modeling module and a helping policy
module. The opponent modeling module is designed to estimate the goal and
capability of the main agent. It adopts a sliding window paradigm and utilizes
several MLP layers to generate the state feature $\xi_{s}$. This feature is
then processed by a feature extractor to derive the capability feature
$\xi_{\eta}$ and the goal feature $\xi^{t}_{g}$. The helping policy module is
composed of two MLP layers to produce action decision and value estimation
respectively. During interaction with the environment, the helping model
outputs an intentional action, representing the target state it aims to reach.
The environment employs a low-level planner to decompose the intentional
action into a sequence of executable actions, which are then processed by the
AI2THOR simulator, guiding the assistant toward the intended state. Finally,
the simulator provides a new observation as feedback, triggering the next
cycle of interaction.

World. The world is denoted as $W=(S,A,T)$, wherein:

  * •

$S=\\{e_{i},h_{j}\\}_{i=1,...,N_{o},j=1,2}$ represents the world state, which
encompasses the state of $N_{o}$ objects and $2$ agents;

  * •

$A=\\{a_{i}\\}$ denotes the action space;

  * •

$T=P(S^{\prime}|S,A;\eta_{i})$ refers to the transition matrix controlled by
agent capability.

Task. Given our formulation of agent capabilities, we select three everyday
tasks for practical implementations of our challenge: Make Breakfast, Arrange
Room, and Make Coffee. These tasks respectively involves different
capabilities:
$(\alpha_{i},\beta_{i},\gamma_{i},\delta_{i},\epsilon_{i},\zeta_{i})$ for Cook
Potato, $(\alpha_{i},\beta_{i},\gamma_{i})$ for Arrange Room, and
$(\alpha_{i},\beta_{i},\epsilon_{i},\zeta_{i})$ for Make Coffee.

Goal. Upon setting a task, the agent need to identify several key points
(goals) that they must achieve. These goals are hidden from the helper, but
affect the helper’s reward for a particular action. Each goal is defined by
the target state of the involved objects. The goal at time $t$ is represented
as:

$\vspace{-0.1cm}g^{t}=(\text{target}\\_\text{state}^{t},e^{t},pr^{t})$ (4)

Target state includes Wait, Get, On, In, KeepOpen, KeepClose, KeepOn and
KeepOff, e.g., “(In, Cup, Cabinet)”.

Observation Space. The proposed environment supports both symbolic and visual
observations. This dual-mode observation system enables the helper to acquire
and refine helping behaviors in various contexts. Following previous work [42,
43], we use symbolic observations containing physical states of all
perceivable objects and agents. Hidden states, like goals and capabilities,
can not be observed.

##  4 Our Model

Our objective is to train a helper agent who can effectively adapt to main
agents with varying capabilities and goals. There are two main challenges: 1)
the challenge posed by the environment (e.g., partial observation and
occlusion), and 2) the challenge posed by the task (i.e., how to estimate the
current goal $g^{t}$ and capabilities $\eta$ of the main agent). Given the
observation $o^{0:t}$, the action policy for the helper is formulated as:

$P(a|o^{0:t})=\sum_{g^{t}\in G,\eta\in
C}P(a|g^{t},\eta,o^{0:t})P(g^{t},\eta|o^{0:t}).$ (5)

Therefore, as depicted in Fig. 2, our model consists of: 1) an opponent
modeling module to estimate the current goal and capabilities of the main
agent, i.e., $P(g^{t},\eta|o^{0:t})$; and 2) a helping policy module to learn
the distribution of the helper’s action conditioned on the predicted goal and
capabilities of the main agent, i.e., $P(a|g^{t},\eta,o^{0:t})$.

The training of our model has two stages. In the first stage, we train an
opponent modeling module using the collected main agent trajectories. This
stage is crucial as it enables the helper agent to understand and anticipate
the behavior of the main agent effectively. In the second stage, we utilize
the pre-trained opponent modeling module to train the helping policy module,
which handles dynamic and complex tasks by interacting in our environment.

###  4.1 The Opponent Modeling Module

To train this module, we collect simulated trajectories in our environment
with the main agent following an expert policy and the helper moving randomly,
where each frame contains the observations of the helper agent and the true
labels of the main agent’s goals and capabilities. Throughout data collection,
we utilize a main agent with various tasks and capabilities and a helper
executing random actions in the environment. See more details in the
supplementary.

We use a sliding window trick when inferring goals and capabilities from the
main agent’s trajectory since goals are transient and typically persist for
only a few steps. This method enables examining a segment of the trajectory at
any given time, and sliding the window as new actions are taken. It allows the
helper agent to make more immediate and relevant inferences about the main
agent’s current goals and capabilities, enhancing its adaptability and
responsiveness. Formally, we have:

$P(\eta_{i},g^{t}|o^{0:t})\approx P(\eta_{i},g^{t}|o^{t-w:t}),$ (6)

where $w$ is the window size.

We use an object encoder and an agent encoder to extract features from raw
observations (details in supplementary). Similar to the “class token” in [2],
we prepend a learnable embedding, which we call “object token” to the sequence
of object feature. We follow the usage of “class token” [2], and only retain
this object token to represent the object feature $\xi^{t}_{o}$ after the
transformer [54] layer (other tokens are removed). This object feature
$\xi^{t}_{o}$, together with the agent feature $\xi^{t}_{a}$, constitute the
state feature $\xi^{t}_{s}$. As in Eq. 6, we use a window size of $w$ and the
state features $\\{\xi^{t-w}_{s},\cdots,\xi^{t}_{s}\\}$ are fed into an MLP
layer to obtain the final state feature $\xi_{s}$. Furthermore, a feature
extractor is employed to derive the capability feature $\xi_{\eta}$ and the
target state feature $\xi^{t}_{g}$ of the main agent. A transformer layer is
utilized for the capability feature. As for the goal feature $\xi^{t}_{g}$, as
detailed in Eq. 4, one transformer layer is used for the target state feature,
and two MLP layers are employed for the features of the related object and its
target parent receptacle.

###  4.2 The Helping Policy Module

Our helping policy module has an actor-critic [55] architecture. At each time
step $t$, given the current observation $o^{t}$, together with all the
features $\xi_{\eta}$ and $\xi_{g}^{t}$ from the opponent modeling module, the
helper agent needs to learn the helping policy
$\pi_{\theta}(a|o^{t},\eta,g^{t})$. We employ an MLP layer for action
selection (the “Policy MLP” in Fig. 2), coupled with an LSTM network for
context memorization. Meanwhile, another MLP (the “Value MLP” in Fig. 2) is
utilized for the value network to estimate the value of the current state.

##  5 Experimental Setup

###  5.1 Two-stage Learning

We apply a two-stage training technique to train our model. In the first
stage, we use four auxiliary classifiers to respectively train the capability
feature $\xi_{\eta}$, target state feature, object feature and parent
receptacle feature (i.e., the goal feature $\xi^{t}_{g}$) in a supervised
learning manner. These classifiers are lightweight and efficient, utilizing
MLPs, and are trained using cross-entropy loss and the Adam optimizer, with a
learning rate of $1\times 10^{-6}$ and a batch size of 32.

In the second stage, we remove these four classifiers, keep the pre-trained
opponent modeling module frozen, directly use the learnt opponent features
$\xi_{\eta}$ and $\xi^{t}_{g}$, and focus on the training of helping policy
module. We use 20 kitchen rooms for helping policy training and 10 for
evaluation in AI2-THOR [32]. The RLlib [15] framework and PPO [31] algorithm
are applied to train our helping policy module. Unless otherwise stated, we
use the default settings of RLlib. During training, the scene is randomly
initialized with a new room, a new task, and a new capability distribution of
the main agent, at each episode.

We apply a progressive learning technique to train the help policy module.
Firstly, we train the helper policy module for 840 epochs, with a constant
learning rate of $5\times 10^{-5}$ without weight decay and a batch size of
128, using the reward defined in Eq. 3 with $\lambda_{e}=0.0$. We find such
“warm-up” quite important in helping the helper agent to familiarize itself
with basic skills to complete goals and tasks. Secondly, we progressively
train the model for 240 epochs with a learning rate of $5\times 10^{-7}$ and
$\lambda_{e}=1.0$ in Eq. 3 to enhance the smart help ability. In the
evaluation phase, we test the helper in 10 rooms (never used in training) and
with 14 different task-capability pairs, repeating three times with different
random seeds. We exclude 7 task-capability pairs as they will enable the main
agent to finish the task independently. See supplementary for more details.

###  5.2 Reward

The reward of the helper is influenced by the goal $g_{m}$ and the capability
$\eta_{m}$ of the main agent. During training, the helper agent will get a
reward of 20 after finishing a goal of the main agent, corresponding to the
first term in Eq. 3. The second term $R^{e_{m}}(s,a_{h})$ in Eq. 3 is
influenced by the capability of the main agent. Specifically, if the helper
finishes a goal that the main agent can handle independently, the helper will
get a punishment: $R^{e_{m}}(s,a_{h})=-30$. For every step, the helper agent
will get a cost of -0.12 normally and a cost of -0.5 if the action is illegal
(e.g., attempting to open something that can not be opened, such as an apple)
in the environment. If the task is not finished when the scenario is ended,
the helper will receive a punishment of -20.

###  5.3 Baselines

To make a fair comparison, the baselines are:

  * •

Random. This helper randomly selects an action.

  * •

End2end-$\lambda_{e}$=$0.0$. This model generates actions directly from the
observation with MLPs, trained with $\lambda_{e}=0.0$ in Eq. 3. We set
learning rate to $5\times 10^{-5}$, batch size to 128. The model is trained
for 350 epochs before convergence.

  * •

End2end-$\lambda_{e}$=$1.0$. Similar to End2end-$\lambda_{e}$=$0.0$, but with
$\lambda_{e}=1.0$ in Eq. 3.

  * •

MCTS. This model uses a Monte-Carlo Tree Search (MCTS [3]) algorithm to search
for an action with the highest value, given a predicted goal from the opponent
modeling module. The value of each action is estimated by the status of goal
completion after rollout, discounted by the number of steps taken to achieve
this result.

  * •

MCTS-heuristic. This model combines the MCTS model with expert rules for the
three selected tasks. Given a predicted goal from the opponent modeling
module, it decides on which action to simulate at a probability $p=0.5$ by
chance and $p=0.5$ by a rule-based selection of the next action to complete
the goal.

  * •

MCTS${}_{\text{TG}}$. This is an implementation to reproduce the high-level
planning policy of Watch-and-Help [42]. This model knows the true goal of the
main agent.

  * •

MCTS${}_{\text{RG}}$. This model is almost the same as MCTS${}_{\text{TG}}$,
except that it follows a random goal.

  * •

LLM. We use Large Language Model, i.e., gpt-3.5-turbo-instruct [37] as the
helper. We compose a prompt based on the state observations from the AI2-THOR
[32]. The prompt is fed into the LLM at every step. We extract the action
decision generated from the LLM and evaluate its actual effectiveness in the
AI2-THOR simulator. See supplementary for more details.

###  5.4 Ablation Study

To assess the contributions and efficacy of essential components in our
method, we derive the following variants:

  * •

BaseModel. This model is the aforementioned helping policy model with a pre-
trained parameter-frozen opponent modeling module. As in Sec. 5.1, we set
$\lambda_{e}=0.0$ in Eq. 3, and train it for 840 epochs.

  * •

BaseModel-w/o. Capability. Here we remove the capability embedding from the
training of the helping policy.

  * •

BaseModel-$\lambda_{e}$=$1.0$. This model is trained with $\lambda_{e}=1.0$ in
Eq. 3 for 840 epochs.

  * •

BaseModel-PL-$\lambda_{e}$=$0.0$. Here, “PL” means the Progressive Learning
technique (detailed in Sec. 5.1), but with $\lambda_{e}=0.0$ in the second
phase after the first warm-up phase. We simply continue to train the BaseModel
with a smaller learning rate for 240 epochs (with no change in
$\lambda_{e}=0.0$).

  * •

BaseModel-PL-$\lambda_{e}$=$1.0$ (our full model). We use the correct
Progressive Learning technique (detailed in Sec. 5.1), and change to
$\lambda_{e}=1.0$ in the second phase after warming up the basic skills of the
helper model.

Figure 3: Qualitative results of the learned smart help policy. In this
example, the main agent moves directly to finish the goal “(In, Potato,
Microwave)”, but fails at the goal “(Open, Microwave)”. (a) The BaseModel-
PL-$\lambda_{e}$=$0.0$ model not only helps the main agent with the bottleneck
“(Open, Microwave)”, but also continues to help with another goal “(ToggleOff,
Microwave)”. (b) Our full model BaseModel-PL-$\lambda_{e}$=$1.0$ only offer
necessary help with the bottleneck “(Open, Microwave)”, and let the main agent
to finish the rest goals independently, which helping policy is smarter since
it considers the needs and emotional feelings of the main agent.

###  5.5 Evaluation Metrics

To objectively evaluate an agent’s performance, we utilize six distinct
metrics, where Helping Necessity (HN) and Helping Rate (HR) are first proposed
in our work to better assess the Smart Help policy. Let $N$ denote the size of
the test scenarios, and let each scenario $i$ be characterized by an initial
room state $s^{0}_{i}$ and a goal of transitioning the room to the target
state $s^{*}_{i}$. Assuming that both the main agent and the helper require
$l_{i}$ steps (with a maximum of 30) to reach the final state $s^{l}_{i}$, we
elaborate on these metrics as follows.

  * •

Success Rate (SR and GSR). It includes Task-conditioned Success Rate (SR) and
Goal-conditioned Success Rate (GSR), respectively representing the completion
degree of tasks and goals. The SR is defined as
$SR=\frac{1}{N}\sum_{i=1}^{N}R_{i}$, where $R_{i}$ is 1 if the task is
finished. For task $i$, let $N_{g^{m}_{i}}$ denote the effective goals
completed by the main agent, $N_{g^{h}_{i}}$ represent the effective goals
completed by the helper, and $N_{g^{a}_{i}}$ be all the effective goals of the
task. The goal-conditioned success for task $i$ is:
$GS_{i}=\frac{N_{g^{m}_{i}}+N_{g^{h}_{i}}}{N_{g^{a}_{i}}}$. Hence, the overall
GSR is defined as $GSR=\frac{1}{N}\sum_{i=1}^{N}GS_{i}$. $R_{i}=1$ if and only
if $GS_{i}=1$.

  * •

Helping Necessity (HN). This metric reflects the necessity of the helper’s
involvement. Only when the capability required to finish the goal exceeds the
main agent’s capability, the helping is necessary. Let
$N_{g^{h}_{\text{nec},i}}$ denote the necessary goals completed by the helper,
the Helping Necessity is then calculated as
$HN=\frac{1}{N}\sum_{i=1}^{N}\frac{N_{g^{h}_{\text{nec},i}}}{N_{g^{h}_{i}}}R_{i}$,
where $R_{i}=1$ if the task is completed, otherwise $R_{i}=0$. This assesses
the helper’s ability to swiftly identify the main agent’s capability and
provide necessary assistance.

  * •

Helping Rate (HR). HR represents the helper’s initiative to help.
$HR=\frac{N_{\text{help}}}{N_{\text{need}\\_\text{help}}}$, reflecting the
probability of helping when the main agent needs help.

  * •

Episode Length (EL). We record the average episode length to reflect the
efficiency of helping policy. It is computed as:
$EL=\frac{1}{N}\sum_{i=1}^{N}l_{i}$.

  * •

Success-weighted by Path Length (SPL). We also include SPL to have a
comprehensive evaluation. It is computed as:
$SPL=\frac{1}{N}\sum_{i=1}^{N}R_{i}\frac{d_{i}}{\max(d_{i},l_{i})}$, where
$R_{i}\in\\{0,1\\}$ denotes whether the task is successfully completed,
$d_{i}$ represents the minimum number of steps to finish the task $i$, and
$l_{i}$ is the actual steps.

Beyond these key metrics, we also incorporate the average rewards as a metric
for evaluation. For the Smart Help challenge, we want to increase the HN and
HR, while simultaneously descending the EL, as well as keeping the SR, GSR,
and SPL as high as possible.

##  6 Results and Analyses

Comparison with baselines. As shown in Tab. 1, our full model exhibits
superior performance compared with all other baseline models without ground
truth knowledge of the main agent. The Random agent, selecting random actions,
and the MCTS${}_{\text{RG}}$ model, following random goals, both have minimal
ability to provide appropriate help. The superiority of our BaseModel compared
to the End2End model underscores the value of opponent modeling in the context
of assistance tasks. The LLM agent, although has the shortest EL, falls short
on other metrics compared with our model. The MCTS-heuristic model, which
incorporates rule-based heuristics, outperforms the pure MCTS model. However,
our model achieves higher scores in SR, GSR, HN, and HR, while maintaining
similar values for EL and SPL, indicating the effectiveness of its assistive
action planning. The MCTS${}_{\text{TG}}$ model, knowing the true goal of the
main agent, achieves the best performance among all the models, serving as an
upper bound for the task. Notably, while it surpasses other models in most
metrics, our model demonstrates competitive performance, particularly in $HN$.
This suggests that our model achieves a balance by providing only necessary
assistance and ensuring user comfort. The gap between our model and the upper
bound shows potential for future research and further improvement.

Ablation analysis. Comparing the BaseModel with BaseModel-$\lambda_{e}$=$1.0$,
we find that setting $\lambda_{e}=0.0$ can augment the assisting actions of
the helper. When compared with BaseModel-w/o. Capability, we find that the
capability module contributes to the learning of the smart helping policy.
Comparing the BaseModel-PL-$\lambda_{e}$=$0.0$ with BaseModel-
PL-$\lambda_{e}$=$1.0$, we find that after the initial “warm-up” of the basic
skills, setting $\lambda_{e}=1.0$ in the second phase could greatly improve
the HN while maintaining competitive performances across other metrics. Thus,
as in Sec. 5.1, for our full model, in the BaseModel “warm-up” training phase,
the model learns how to complete the goals and tasks, and the assistive
actions are greatly encouraged; while in the second progressive learning
phase, our full model focuses on improving the assistance strategy of the
helper based on the main agent’s needs and feelings.

Qualitative Results. Fig. 3 demonstrates two helping cases with a baseline
model and our full model. The two models exhibit exploration behaviors in the
scene and finally learn to solve the bottleneck problem of the main agent. In
comparison, the BaseModel-PL-$\lambda_{e}$=$0.0$ only learns to help all the
goals it successfully infers, while our full model BaseModel-
PL-$\lambda_{e}$=$1.0$ learns to decide whether to help based on its inferred
goals and capabilities, i.e., only to help with “(Open, Microwave)”.

Method | SR($\uparrow$) | GSR($\uparrow$) | HN($\uparrow$) | HR($\uparrow$) | Reward($\uparrow$) | EL($\downarrow$) | SPL($\uparrow$)
---|---|---|---|---|---|---|---
Random | 0.074 | 0.364 | 0.054 | 0.057 | -31.339 | 28.567 | 0.054
End2end-$\lambda_{e}$=$0.0$ | 0.267 | 0.441 | 0.149 | 0.315 | -23.218 | 24.343 | 0.212
End2end-$\lambda_{e}$=$1.0$ | 0.067 | 0.372 | 0.057 | 0.057 | -26.183 | 28.726 | 0.049
LLM | 0.345 | 0.506 | 0.382 | 0.381 | / | 22.914 | 0.210
MCTS | 0.178 | 0.482 | 0.266 | 0.328 | / | 26.743 | 0.126
MCTS-heuristic | 0.274 | 0.528 | 0.355 | 0.400 | / | 24.692 | 0.199
$\text{MCTS}_{\text{TG}}$ | 0.593 | 0.804 | 0.610 | 0.678 | / | 16.271 | 0.541
$\text{MCTS}_{\text{RG}}$ | 0.043 | 0.348 | 0.015 | 0.015 | / | 28.950 | 0.042
BaseModel | 0.419 | 0.515 | 0.408 | 0.412 | -15.179 | 24.945 | 0.176
BaseModel-w/o. Capability | 0.374 | 0.504 | 0.375 | 0.379 | -13.852 | 23.748 | 0.235
BaseModel-$\lambda_{e}$=$1.0$ | 0.181 | 0.466 | 0.193 | 0.200 | -17.185 | 27.448 | 0.122
BaseModel-PL-$\lambda_{e}$=$0.0$ | 0.488 | 0.556 | 0.455 | 0.496 | -10.959 | 24.383 | 0.198
BaseModel-PL-$\lambda_{e}$=$1.0$ (Ours) | 0.483 | 0.548 | 0.498 | 0.506 | -10.625 | 24.650 | 0.200

Table 1:  The quantitative results of our experiments. The last ten
environments in AI2-THOR are reserved for evaluation. The best results are
highlighted in bold. Note that we include some oracle baselines and the upper
bound performances are highlighted with underlines.

##  7 Conclusion

In this paper, we propose the novel challenge Smart Help and build an
environment and model that fosters smarter and more harmonious interaction
between humans and artificial agents. We hope our challenge, environment,
dataset, model and benchmark results will serve as valuable resources to
future studies of this important problem.

#### Acknowledgement

This work is supported by the National Science and Technology Major Project
(2022ZD0114900). We thank Zhenliang Zhang at BIGAI and Luyao Yuan at META for
their helpful discussions.

## References

  * Aigner and McCarragher [1999] Peter Aigner and Brenan McCarragher.  Shared control framework applied to a robotic aid for the blind.  _IEEE Control Systems Magazine_ , 19(2):40–46, 1999.
  * Alexey et al. [2020] Dosovitskiy Alexey, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, and Mostafa Dehghani et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  2020\.
  * B. et al. [2012] Browne Cameron B., Edward Powley, Daniel Whitehouse, Simon M. Lucas, Peter I. Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton.  A survey of monte carlo tree search methods.  _IEEE Transactions on Computational Intelligence and AI in games_ , 2012.
  * Bemelmans et al. [2012] Roger Bemelmans, Gert Jan Gelderblom, Pieter Jonker, and Luc De Witte.  Socially assistive robots in elderly care: a systematic review into effects and effectiveness.  _Journal of the American Medical Directors Association_ , 13(2):114–120, 2012.
  * Broadbent et al. [2009] Elizabeth Broadbent, Rebecca Stafford, and Bruce MacDonald.  Acceptance of healthcare robots for the older population: Review and future directions.  _International journal of social robotics_ , 1:319–330, 2009.
  * Broadbent et al. [2010] Elizabeth Broadbent, I Han Kuo, Yong In Lee, Joel Rabindran, Ngaire Kerse, Rebecca Stafford, and Bruce A MacDonald.  Attitudes and reactions to a healthcare robot.  _Telemedicine and e-Health_ , 16(5):608–613, 2010.
  * Brose et al. [2010] Steven W Brose, Douglas J Weber, Ben A Salatin, Garret G Grindle, Hongwu Wang, Juan J Vazquez, and Rory A Cooper.  The role of assistive robotics in the lives of persons with disability.  _American Journal of Physical Medicine & Rehabilitation_, 89(6):509–521, 2010.
  * Buoncompagni et al. [2017] Luca Buoncompagni, Barbara Bruno, Antonella Giuni, Fulvio Mastrogiovanni, and Renato Zaccaria.  Towards a new paradigm for assistive technology at home: research challenges, design issues and performance assessment.  _arXiv preprint arXiv:1710.10164_ , 2017.
  * Čaić et al. [2018] Martina Čaić, Gaby Odekerken-Schröder, and Dominik Mahr.  Service robots: value co-creation and co-destruction in elderly care networks.  _Journal of Service Management_ , 2018.
  * Casey et al. [2021] Anthony Casey, Hannan Azhar, Marek Grzes, and Mohamed Sakel.  Bci controlled robotic arm as assistance to the rehabilitation of neurologically disabled patients.  _Disability and Rehabilitation: Assistive Technology_ , 16(5):525–537, 2021.
  * Cassandra et al. [1994] Anthony R Cassandra, Leslie Pack Kaelbling, and Michael L Littman.  Acting optimally in partially observable stochastic domains.  In _Aaai_ , pages 1023–1028, 1994.
  * Davila et al. [2019] Victor J Davila, Andrew J Meltzer, M Susan Hallbeck, William M Stone, and Samuel R Money.  Physical discomfort, professional satisfaction, and burnout in vascular surgeons.  _Journal of vascular surgery_ , 70(3):913–920, 2019.
  * Deitke et al. [2020] Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozbeh Mottaghi, Jordi Salvador, and et al.  Robothor: An open simulation-to-real embodied ai platform.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 3164–3174, 2020.
  * Ehsani et al. [2021] Kiana Ehsani, Winson Han, Alvaro Herrasti, Eli VanderBilt, Luca Weihs, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi.  Manipulathor: A framework for visual object manipulation.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pages 4495–4504, 2021.
  * Eric et al. [2018] Liang Eric, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica.  Rllib: Abstractions for distributed reinforcement learning.  In _International conference on machine learning_ , pages 3053–3062. PMLR, 2018.
  * Everett and Roberts [2018] Richard Everett and Stephen Roberts.  Learning against non-stationary agents with opponent modelling and deep reinforcement learning.  In _2018 AAAI spring symposium series_ , 2018.
  * Fall et al. [2015] Cheikh Latyr Fall, Philippe Turgeon, Alexandre Campeau-Lecours, Véronique Maheu, Mounir Boukadoum, Sebastien Roy, Daniel Massicotte, Clément Gosselin, and Benoit Gosselin.  Intuitive wireless control of a robotic arm for people living with an upper body disability.  In _2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)_ , pages 4399–4402. IEEE, 2015.
  * Feil-Seifer and Mataric [2005] David Feil-Seifer and Maja J Mataric.  Defining socially assistive robotics.  In _9th International Conference on Rehabilitation Robotics, 2005. ICORR 2005._ , pages 465–468. IEEE, 2005.
  * Foerster et al. [2017] Jakob N Foerster, Richard Y Chen, Maruan Al-Shedivat, Shimon Whiteson, Pieter Abbeel, and Igor Mordatch.  Learning with opponent-learning awareness.  _arXiv preprint arXiv:1709.04326_ , 2017.
  * Glover [2003] Jared Glover.  A robotically-augmented walker for older adults.  2003\.
  * Graf et al. [2002] B Graf, A Hans, J Kubacki, and RD Schraft.  Robotic home assistant care-o-bot ii.  In _Proceedings of the Second Joint 24th Annual Conference and the Annual Fall Meeting of the Biomedical Engineering Society][Engineering in Medicine and Biology_ , pages 2343–2344. IEEE, 2002.
  * Harman [2001] Denham Harman.  Aging: overview.  _Annals of the New York Academy of Sciences_ , 928(1):1–21, 2001.
  * He et al. [2016a] He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé III.  Opponent modeling in deep reinforcement learning.  In _International conference on machine learning_ , pages 1804–1813. PMLR, 2016a.
  * He et al. [2016b] He He, Jordan Boyd-Graber, Kevin Kwok, and Hal Daumé III.  Opponent modeling in deep reinforcement learning.  In _International conference on machine learning_ , pages 1804–1813. PMLR, 2016b.
  * Hernandez-Leal and Kaisers [2017] Pablo Hernandez-Leal and Michael Kaisers.  Learning against sequential opponents in repeated stochastic games.  In _The 3rd Multi-disciplinary Conference on Reinforcement Learning and Decision Making, Ann Arbor_ , page 16, 2017.
  * Hernandez-Leal et al. [2016] Pablo Hernandez-Leal, Benjamin Rosman, Matthew E Taylor, L Enrique Sucar, and Enrique Munoz de Cote.  A bayesian approach for learning and tracking switching, non-stationary opponents.  In _Proceedings of the 2016 international conference on autonomous agents & multiagent systems_, pages 1315–1316, 2016.
  * Inoue et al. [2008] Kaoru Inoue, Kazuyoshi Wada, and Yuko Ito.  Effective application of paro: Seal type robots for disabled people in according to ideas of occupational therapists.  In _Computers Helping People with Special Needs: 11th International Conference, ICCHP 2008, Linz, Austria, July 9-11, 2008. Proceedings 11_ , pages 1321–1324. Springer, 2008.
  * Jain et al. [2019] Unnat Jain, Luca Weihs, Eric Kolve, Mohammad Rastegari, Svetlana Lazebnik, Ali Farhadi, Alexander G Schwing, and Aniruddha Kembhavi.  Two body problem: Collaborative visual task completion.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6689–6699, 2019.
  * Jain et al. [2020] Unnat Jain, Luca Weihs, Eric Kolve, Ali Farhadi, Svetlana Lazebnik, Aniruddha Kembhavi, and Alexander Schwing.  A cordial sync: Going beyond marginal policies for multi-agent embodied tasks.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16_ , pages 471–490. Springer, 2020.
  * Jiang and Lu [2018] Jiechuan Jiang and Zongqing Lu.  Learning attentional communication for multi-agent cooperation.  _Advances in neural information processing systems_ , 31, 2018.
  * John et al. [2017] Schulman John, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _arXiv preprint arXiv:1707.06347_ , 2017.
  * Kolve et al. [2017] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Matt Deitke, Kiana Ehsani, Daniel Gordon, Yuke Zhu, et al.  Ai2-thor: An interactive 3d environment for visual ai.  _arXiv preprint arXiv:1712.05474_ , 2017.
  * Li et al. [2021] Chengshu Li, Fei Xia, Roberto Martín-Martín, Michael Lingelbach, Sanjana Srivastava, Bokui Shen, Kent Vainio, Cem Gokmen, Gokul Dharan, Tanish Jain, et al.  igibson 2.0: Object-centric simulation for robot learning of everyday household tasks.  _arXiv preprint arXiv:2108.03272_ , 2021.
  * Liu et al. [2022] Xinzhu Liu, Di Guo, Huaping Liu, and Fuchun Sun.  Multi-agent embodied visual semantic navigation with scene prior knowledge.  _IEEE Robotics and Automation Letters_ , 7(2):3154–3161, 2022.
  * Mnih et al. [2015] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al.  Human-level control through deep reinforcement learning.  _nature_ , 518(7540):529–533, 2015.
  * OhnBar et al. [2018] Eshed OhnBar, Kris Kitani, and Chieko Asakawa.  Personalized dynamics models for adaptive assistive navigation systems.  In _Conference on Robot Learning_ , pages 16–39. PMLR, 2018.
  * [37] OpenAI.  Gpt-3.5 turbo models.  https://platform.openai.com/docs/models/gpt-3-5-turbo.
  * Panait and Luke [2005] Liviu Panait and Sean Luke.  Cooperative multi-agent learning: The state of the art.  _Autonomous agents and multi-agent systems_ , 11:387–434, 2005.
  * Papoudakis and Albrecht [2020] Georgios Papoudakis and Stefano V Albrecht.  Variational autoencoders for opponent modeling in multi-agent systems.  _arXiv preprint arXiv:2001.10829_ , 2020.
  * Parida and Sinha [2021] Monnie Parida and Manjira Sinha.  Pandemic and disability: Challenges faced and role of technology.  _Technology and Disability_ , 33(4):245–252, 2021.
  * Puig et al. [2018] Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba.  Virtua lhome: Simulating household activities via programs.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 8494–8502, 2018.
  * Puig et al. [2020] Xavier Puig, Tianmin Shu, Shuang Li, Zilin Wang, Yuan-Hong Liao, Joshua B Tenenbaum, Sanja Fidler, and Antonio Torralba.  Watch-and-help: A challenge for social perception and human-ai collaboration.  _arXiv preprint arXiv:2010.09890_ , 2020.
  * Puig et al. [2023] Xavier Puig, Tianmin Shu, Joshua B Tenenbaum, and Antonio Torralba.  Nopa: Neurally-guided online probabilistic assistance for building socially intelligent home assistants.  _arXiv preprint arXiv:2301.05223_ , 2023.
  * Rabinowitz et al. [2018] Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, SM Ali Eslami, and Matthew Botvinick.  Machine theory of mind.  In _International conference on machine learning_ , pages 4218–4227. PMLR, 2018.
  * Rosman et al. [2016] Benjamin Rosman, Majd Hawasly, and Subramanian Ramamoorthy.  Bayesian policy reuse.  _Machine Learning_ , 104:99–127, 2016.
  * Roy et al. [2000] Nicholas Roy, Gregory Baltus, Dieter Fox, Francine Gemperle, Jennifer Goetz, Tad Hirsch, Dimitris Margaritis, Michael Montemerlo, Joelle Pineau, Jamie Schulte, et al.  Towards personal service robots for the elderly.  In _Workshop on Interactive Robots and Entertainment (WIRE 2000)_ , page 184. Citeseer, 2000.
  * Savva et al. [2019] Manolis Savva, Abhishek Kadian, Oleksandr Maksymets, Yili Zhao, Erik Wijmans, Bhavana Jain, and Julian Straub et al.  Habitat: A platform for embodied ai research.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pages 9339–9347, 2019.
  * Shen et al. [2021] Bokui Shen, Fei Xia, Chengshu Li, Roberto Martín-Martín, Linxi Fan, Guanzhi Wang, Claudia Pérez-D’Arpino, Shyamal Buch, Sanjana Srivastava, Lyne Tchapmi, et al.  igibson 1.0: A simulation environment for interactive tasks in large realistic scenes.  In _2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , pages 7520–7527. IEEE, 2021.
  * Simmons et al. [1997] Reid Simmons, Richard Goodwin, Karen Zita Haigh, Sven Koenig, and Joseph O’Sullivan.  A layered architecture for office delivery robots.  In _Proceedings of the first international conference on Autonomous agents_ , pages 245–252, 1997.
  * Szot et al. [2021] Andrew Szot, Alexander Clegg, Eric Undersander, Erik Wijmans, Yili Zhao, John Turner, and Noah Maestre et al.  Habitat 2.0: Training home assistants to rearrange their habitat.  In _Advances in Neural Information Processing Systems_ , pages 251–266, 2021.
  * Uehara et al. [2010] Hideyuki Uehara, Hiroki Higa, and Takashi Soken.  A mobile robotic arm for people with severe disabilities.  In _2010 3rd IEEE RAS & EMBS International Conference on Biomedical Robotics and Biomechatronics_, pages 126–129. IEEE, 2010.
  * Ullman et al. [2009] Tomer Ullman, Chris Baker, Owen Macindoe, Owain Evans, Noah Goodman, and Joshua Tenenbaum.  Help or hinder: Bayesian models of social goal inference.  _Advances in neural information processing systems_ , 22, 2009.
  * Van Jaarsveld et al. [2021] Danielle D Van Jaarsveld, David D Walker, Simon Lloyd D Restubog, Daniel Skarlicki, Yueyang Chen, and Pascale H Frické.  Unpacking the relationship between customer (in) justice and employee turnover outcomes: can fair supervisor treatment reduce employees’ emotional turmoil?  _Journal of Service Research_ , 24(2):301–319, 2021.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  _Advances in neural information processing systems_ , 30, 2017.
  * Volodymyr et al. [2016] Mnih Volodymyr, Adrià Puigdomènech Badia, Mehdi Mirza, Alex Graves, Tim Harley, Timothy P. Lillicrap, David Silver, , and Koray Kavukcuoglu.  Asynchronous methods for deep reinforcement learning.  In _the 33rd International Conference on International Conference on Machine Learning_ , pages 1928–1937. PMLR, 2016.
  * Wang et al. [2021] Haiyang Wang, Wenguan Wang, Xizhou Zhu, Jifeng Dai, and Liwei Wang.  Collaborative visual navigation.  _arXiv preprint arXiv:2107.01151_ , 2021.
  * Xuan et al. [2001] Ping Xuan, Victor Lesser, and Shlomo Zilberstein.  Communication decisions in multi-agent cooperation: Model and experiments.  In _Proceedings of the fifth international conference on Autonomous agents_ , pages 616–623, 2001.
  * Yu et al. [2022] Xiaopeng Yu, Jiechuan Jiang, Wanpeng Zhang, Haobin Jiang, and Zongqing Lu.  Model-based opponent modeling.  _Advances in Neural Information Processing Systems_ , 35:28208–28221, 2022.
  * Zhang et al. [2021] Weinan Zhang, Xihuai Wang, Jian Shen, and Ming Zhou.  Model-based multi-agent policy optimization with adaptive opponent-wise rollouts.  _arXiv preprint arXiv:2105.03363_ , 2021.
  * Zheng et al. [2018] Yan Zheng, Zhaopeng Meng, Jianye Hao, Zongzhang Zhang, Tianpei Yang, and Changjie Fan.  A deep bayesian policy reuse approach against non-stationary agents.  _Advances in neural information processing systems_ , 31, 2018.

Supplementary Material

## Overview

This supplementary material includes:

  * •

Appendix A has the detailed setting of the environment.

  * •

Appendix B has the model implementation details.

##  Appendix A Environment Setting

Figure 1:  In our study, a total of 30 distinct scenes from AI2-THOR were
utilized to train our model. The planforms of them are provided here. These
scenes were divided into two datasets: 20 for training while the remaining 10
were reserved for testing. Different scenes have different size, but the
figures presented here are scaled to the same size. The object composition in
each scene is diverse. Thus the agent must learn a general policy for
different scenes. However, for our tasks, the focus was solely on those
objects that were common across all scenes. This approach was adopted to
ensure that the task was consistently applicable across all the scenes used in
the study.

#### Scenes

We use 30 scenes from AI2THOR kitchen scenes, which are shown in Fig. 1. In
these scenes, 20 are used for training while the remaining 10 are reserved for
testing. The object composition in each scene is diverse. We only pay
attention to objects that were common across all scenes when parsing the task
to goals.

#### Tasks

We have classified our tasks into three categories: MakeBreakfast,
ArrangeRoom, and MakeCoffee. Each task is further divided into a series of
goals. The Tab. 1 shows some examples about the parsing process. Different
room will leads to different parsing. For example, in some rooms the mug has
been already put in the coffee machine, thus the main agent does not need to
pick up the mug and put it in the coffee machine.

Task | Goals (example)
---|---
MakeBreakfast | (Get, Potato), (KeepOff, Mircowave),
| (KeepOpen, Mircowave), (In, Potato, Microwave),
| (KeepClose, Microwave), (KeepOn, Microwave),
| (KeepOff, Microwave)
ArrangeRoom | (Get, Potato), (KeepOpen, Fridge),
| (In, Potato, Fridge), (KeepClose Fridge),
MakeCoffee | (Get, Mug), (In, Mug, CoffeeMachine),
| (KeepOn, CoffeeMachine), (KeepOff, CoffeeMachine)

Table 1:  The table showcases examples of the parsing process, from
overarching tasks to specific goals. This process may vary across different
scenes and may also be probabilistic within the same scene. For instance, in
certain scenarios, the mug may already be situated in the coffee machine. In
such cases, the agent only needs to activate and then deactivate the machine
to complete the “MakeCoffee” task.

#### Goals

Goals denote the target states that an agent must reach before successfully
completing a task. We have identified seven potential goals for the agent to
choose from. These are: (Wait), (Get, $e_{i}$), (On, $e_{i}$, $pr_{i}$), (In,
$e_{i}$, $pr_{i}$), (KeepOpen, $e_{i}$), (KeepClose, $e_{i}$), (KeepOn,
$e_{i}$), and (KeepOff, $e_{i}$). In certain scenarios, the sequence of goals
is pre-determined. This means that the agent must accomplish the goals in the
given order, rather than arbitrarily. For instance, the microwave must be
closed before it can be toggled on. Failing to follow this sequence results in
an invalid action, as a microwave cannot be activated while open. Moreover,
once a specific state has been reached, certain properties of that state need
to be maintained until the next goal indicates a change. For example, if the
goal sequence comprises (Open, Fridge) followed by (Put, Bread, Fridge), the
agent must keep the fridge open after the first goal is achieved, until
successfully accomplishing the ensuing goal of placing the bread in the
fridge. These complexities present challenges both in the parsing process,
which converts tasks into goals, and in the execution of these goals, adding
layers of difficulty to the agent’s tasks.

#### Intentional action

We have defined seven intentional actions that the agent can perform: (Wait),
(PickUp, $e_{i}$), (Put, $pr_{i}$), (ToggleOn, $e_{i}$), (ToggleOff, $e_{i}$),
(Open, $e_{i}$), and (Close, $e_{i}$). Upon selecting an intentional action, a
low-level planner devises a sequence of executable actions to be sent to the
AI2THOR simulator. It’s important to note that actions may fail due to various
reasons. For instance, the action (Put, $pr_{i}$) will result in failure if
the agent is not currently holding any objects. Similarly, an action to open
an item that cannot be opened is destined to fail. Furthermore, the validity
of an action can be context-specific. For example, while a microwave can
typically be opened, it cannot be opened if it is currently active; it must
first be toggled off. This context-dependent validity of actions presents a
significant challenge for the model. It necessitates the model to learn the
intricate dependency relationships between the context and the action, based
on the reward and observation data provided by the environment.

#### Capability

Some examples about how the capability will affect agent’s interaction with
different objects are shown in Fig. 2. We represent the capability by
$\alpha$, $\beta$, $\gamma$, $\delta$, $\epsilon$, and $\zeta$.

  * •

$\alpha_{i}\in[0,1]$ denotes the maximum height that the agent can reach;

  * •

$\beta_{i}\in[0,1]$ represents the maximum weight that the agent can lift;

  * •

$\gamma_{i},\delta_{i},\epsilon_{i},\zeta_{i}\in[0,1]$ represent the agent’s
ability to complete open, close, toggle on, and toggle off tasks,
respectively.

In this challenge, we have seven types of agents to select from. These include
agents with full capabilities and agents with one randomly assigned
limitation. An agent with full capabilities will have
$(\alpha,\beta,\gamma,\delta,\epsilon,\zeta)=(1,1,1,1,1,1)$. If the agent has
a limitation in $\alpha$, the value will fall within the range of $(0.2,0.8)$.
The same concept applies to other capabilities, where $\beta$ will be within
$(0.1,0.7)$, and $\gamma,\delta,\epsilon,\zeta$ will range from $(0,0.49)$.

In the training process, the main agent will be randomly set to a kind of
capability. In the testing process, we will set the main agent capability to
the lowest level. For example, when the main agent’s type is $\beta$
limitation, his capability is (1, 0.1, 1, 1, 1, 1).

Type | Capability
---|---
Full capability | (1, 1, 1, 1, 1, 1)
$\alpha$ limitation |  ($\alpha$, 1, 1, 1, 1, 1), $\alpha\in(0.2,0.8)$
$\beta$ limitation |  (1, $\beta$, 1, 1, 1, 1), $\beta\in(0.1,0.7)$
$\gamma$ limitation |  (1, 1, $\gamma$, 1, 1, 1), $\gamma\in(0,0.49)$
$\delta$ limitation |  (1, 1, 1, $\delta$, 1, 1), $\delta\in(0,0.49)$
$\epsilon$ limitation |  (1, 1, 1, 1, $\epsilon$, 1), $\epsilon\in(0,0.49)$
$\zeta$ limitation |  (1, 1, 1, 1, 1, $\zeta$), $\zeta\in(0,0.49)$

Table 2:  We sample agents from seven different types. The types also include
the sampling of agents with full capabilities, serving as a benchmark to
examine the model’s ability to differentiate between agents with limitations
and those without. The range of capabilities for an agent with limitations is
established based on the statistical information derived from the scenes.
Figure 2:  This figure shows the impact of different capabilities, represented
by $\alpha$, $\beta$, $\gamma$, $\delta$, $\epsilon$, and $\zeta$, on the
agent’s interaction with various objects. $\alpha$ denotes the agent’s ability
to lift objects of different weights. A lower $\alpha$ value implies a limited
capacity to handle only lighter objects, such as a potato (weight 0.18) or a
tomato (weight 0.20). However, as $\alpha$ increases, the agent can manage
heavier items, such as bread (weight 0.7) or a cup (weight 1.0). These object
weights are determined by the AI2THOR simulator [32], reflecting the careful
considerations of its developers. $\beta$ symbolizes the agent’s ability to
reach varying heights. The floor, with a height of 0, is the most accessible,
while the other objects’ height, like the cabinet’s height, determined by its
presence in the scene, poses a greater challenge. $\gamma,\delta,\epsilon$ and
$\zeta$ respectively represent the agent’s ability to perform specific
operations: opening, closing, toggling on, and toggling off. We assume a
threshold of 0.5, which serves as the tipping point between success and
failure in these actions under normal circumstances.

##  Appendix B Implementation Details

###  B.1 Trajectory dataset

We assembled a main agent trajectory dataset for pretraining the opponent
modeling module. The pretraining phase equips the model with a more effective
starting point for the subsequent reinforcement learning (RL) training. During
data collection, we employ a human-derived expert policy to guide the main
agent’s actions. Simultaneously, the helper will randomly choose an action,
with its observations and the ground label constituting the dataset. This
dataset comprises 60,024 trajectories, each containing observations at five
discrete time points and the final goal of the main agent. When at all of the
five discrete time points the helper can not observe the main agent, the label
of the main agent’s goal will be set to None. The statistics of the dataset is
shown in Tab. 3.

Goal | KeepClose | Get | In | On | Open | ToggleOff | ToggleOn | Wait | None
---|---|---|---|---|---|---|---|---|---
# Trajectory | 4192 | 8295 | 3420 | 214 | 5066 | 2384 | 2337 | 12953 | 21154

Table 3:  Dataset Statistics. Our dataset is characterized by an imbalance in
the number of trajectories corresponding to each goal. This discrepancy is
addressed during the training process through a rebalancing strategy. The
frequency of “On” goals is significantly lower compared to the others. This is
primarily due to the interchangeable use of “In” goals to represent “On”
goals. This substitution is reasonable as the agent can achieve them via the
same singular “Put” action. This substitution is used particularly during the
parsing of the task “ArrangeRoom”, which aids in maintaining uniform
terminology across the parsing.

###  B.2 Opponent Modeling Module

We train an opponent modeling module utilizing the main agent trajectory
dataset.

First, we need to change the observations to embedding with the object
encoder. We use MLPs and embedding model to achieve this.

Property | Data type | Method | Output size
---|---|---|---
Object type | Int | Embedding | 32
Parent receptacle | Int | Embedding | 32
Height | Float | MLP | 32
Weight | Float | MLP | 32
Position | Float | MLP | 32
Distance | Float | MLP | 32
Other properties | Bool | MLP | 32
Table 4: The properties of object and how to change them to embedding. Here
other properties include ’isPickedUp’, ’isOpen’, ’isCooked’, ’isToggled’, and
’isVisible’.

After the change, we will get a feature will size 224 for every object. Then
we use a transformer to handle all the features. The transformer has 8 heads,
0.2 dropout, hidden size of 128, and 4 layers. With the transformer, we get
the object feature.

For dealing with the observation of agents, we use the agent encoder 5.

Property | Data type | Method | Output size
---|---|---|---
Held object type | Int | Embedding | 32
Action type | Int | Embedding | 32
Action success | Bool | MLP | 32
Position | Float | MLP | 32
Rotation | Float | MLP | 32
Table 5: The observations of objects and how to change them to embedding.

After the model processing, we concatenate the feature of action type and
action success and put them into a MLP to get the action feature. Then we use
other three features and the action feature to get the agent feature with MLP.

The object feature and the agent feature are concatenate to form the state
feature. Next, we concatenate state features of five time points and put them
into a MLP. The time feature output by the MLP has size of 128. Then, the time
feature is fed into two transformers, and then we will get two features, which
represent goal and capability respectively. The transformer has two heads, 0.1
dropout, hidden size of 256, and 4 layers. The parameters of these two
transformers are not shared. Then we concatenate the time feature and the goal
feature and put them to a MLP. The we get the feature of $o_{1}$, whose size
is 128. Then we concatenate the time feature, the goal feature and the $o_{1}$
feature and get $o_{1}$ feature with a MLP. The $o_{1}$ feature represents the
object that the goal involves. The $o_{2}$ feature represents the parent
receptacle that the goal involves.

In the training process, we use four MLP as classifiers to get the probability
of goals, the relevant objects, and the predicted capabiities. The we use
cross-entropy loss and the Adam optimizer, with a learning rate of $1\times
10^{-6}$ and a batch size of 32.

We achieved a commendable prediction accuracy of 83.6%. This high level of
precision indicates that our model possesses a remarkable capability to infer
the goals of human agents based on a constrained set of observational data.

###  B.3 Helping Model

Our model accepts symbolic observations as input, which encompass the helper’s
partial observation, as well as the symbolic agent states of both the helper
and the main agent. The observation includes the symbolic states of all
objects within the helper’s field of view. The object encoder and the agent
encoder have been introduced in the last part. With the assistance of our pre-
trained opponent modeling module, we can effectively model the main agent
using its trajectory. It is worth noting that the classifiers from the
opponent modeling module are not utilized within the helper model. Instead,
the latent representation from the opponent modeling module is concatenated
with the observation. This composite embedding is then processed through a
linear layer and a ReLU function before being input into a single-layer LSTM
with 512 hidden units. To generate an action policy, we apply a linear layer
to the output of the LSTM, with no additional nonlinearity required. This
streamlined approach ensures the effective processing of symbolic observations
and accurate policy generation.

###  B.4 LLM

In our setup, LLM, i.e., “gpt-3.5-turbo-instruct” is used as a planner to
generate a policy for the helper agent. For each scene, we provide the basic
information of this environment, as shown in Fig. 3. At each step, we give the
partial observations of scenes and agents to the LLM, as shown in Fig. 4.
Besides, as shown in Fig. 5, we stipulate how the LLM should output to select
a valid action, and give an example to help it understand. The LLM processes
the prompt and produces an action decision based on its understanding of the
given information. If the LLM outputs an invalid action decision, the “Wait”
action will be adopted as its next action. To assess the effectiveness of the
LLM-generated actions, we execute this action in the AI2-THOR simulator to
determine how well the LLM performs as a planner for the helper agent in the
given environment.

Figure 3:  The LLM prompt (the first part). We provide the basic setting
information for the LLM, including what needs to be completed, what can be
done, what types of the main agents are in the environment, etc.  Figure 4:
The LLM prompt (the second part). At each step, we provide the LLM with its
partial observations. For scene observations, we enumerate all observable
objects and their properties. For agent observations, we summarize the action
trajectory of the main agent, the completion of his previous action, and
whether this main agent holds an object. All the optional actions and the
previous action information of the LLM will also be mentioned.  Figure 5:  The
LLM prompt (the third part). We will specify the criteria for selecting a
valid action through the LLM’s output, along with providing illustrative
examples to facilitate its comprehension.

###  B.5 MCTS-heuristic

  * •

MCTS. We implement a MCTS algorithm which outputs an action based on a given
goal, whether predicted, ground truth, or randomly selected. In each
simulation, it samples an action under the guidance of the Upper Confidence
Bound (UCB), which balances exploration and exploitation according to node
values and visit counts. Then it performs a rollout to reach one possible end
state, with the maximum rollout depth limited to 5 steps. If the goal is
successfully achieved within this depth, the end state’s value is evaluated
by:

$v=50/d$

where $v$ represents the value and $d$ the number of steps taken to complete
the goal. The algorithm returns the action represented by the most visited
node after $n=500$ simulations.

  * •

MCTS-heuristic.To enhance search efficiency and task completion capabilities,
we use a heuristic MCTS method, which incorporates hand-written rules that are
specifically defined to break down a goal into a sequence of actions necessary
for task completion. This model employs a probabilistic strategy during the
sampling of the MCTS, with a probability parameter $p_{\text{sample}}$
determining whether to sample a random action for rollout or, at
$1-p_{\text{sample}}$, select the next action from the rule-based sequence,
excluding completed actions. When $p_{\text{sample}}$ is set to $1$, this
model is equivalent to the MCTS model, and when it is set to $0$, it strictly
follows the rule-based action sequence. Initially, we set $p_{\text{sample}}$
to $1$ and $0.9$ to encourage exploration for more efficient action sequences.
The action sequences obtained through this exploratory search align with the
hand-written rules, validating their optimality to a certain degree. During
testing, we adjust $p_{\text{sample}}=0.5$ to ensure a more efficient search
and a higher success rate in task completion.

  * •

$\text{MCTS}_{\text{TG}}$ and $\text{MCTS}_{\text{RG}}$. We develop
$\text{MCTS}_{\text{TG}}$ and $\text{MCTS}_{\text{RG}}$ to replicate the
baselines used in Watch-and-Help [42]. In the original work, a hierarchical
planner was implemented, using MCTS for high-level planning and regression
planning (RP) for low-level planning. Since our task environment does not
require low-level planning, we use the MCTS-heuristic model as a counterpart
of their planner in this specific environment. To provide comparison, we
implemented two additional baselines: $\text{MCTS}_{\text{TG}}$, which has
knowledge of the main agent’s true goal, and $\text{MCTS}_{\text{RG}}$, which
follows a random goal. As shown in the results table, the performance of
$\text{MCTS}_{\text{RG}}$ is similar to the Random agent, with minimal ability
to provide assistance and complete tasks. This highlights the importance of a
smart agent that can infer the main agent’s goal. $\text{MCTS}_{\text{TG}}$,
knowing the true goal of the main agent, achieves the best performance across
all metrics. However, while it significantly surpasses other models in other
metrics, completing the task better and faster, its HN is relatively low,
indicating the limitation of its simple take-over helping strategy.