# Creativity of AI: Hierarchical Planning Model Learning for Facilitating Deep
Reinforcement Learning

Hankz Hankui Zhuo<EMAIL_ADDRESS>Shuting Deng
<EMAIL_ADDRESS>Mu Jin<EMAIL_ADDRESS>Zhihao Ma
<EMAIL_ADDRESS>Kebing Jin<EMAIL_ADDRESS>Chen Chen
<EMAIL_ADDRESS>Chao Yu<EMAIL_ADDRESS>School of Computer
Science and Engineering, Sun Yat-sen University, Guangzhou, China  Huawei
Noah’s Ark Lab

###### Abstract

Despite of achieving great success in real-world applications, Deep
Reinforcement Learning (DRL) is still suffering from three critical issues,
i.e., data efficiency, lack of the interpretability and transferability.
Recent research shows that embedding symbolic knowledge into DRL is promising
in addressing those challenges. Inspired by this, we introduce a novel deep
reinforcement learning framework with symbolic options. Our framework features
a loop training procedure, which enables guiding the improvement of policy by
planning with planning models (including action models and hierarchical task
network models) and symbolic options learned from interactive trajectories
automatically. The learned symbolic options alleviate the dense requirement of
expert domain knowledge and provide inherent interpretability of policies.
Moreover, the transferability and data efficiency can be further improved by
planning with the symbolic planning models. To validate the effectiveness of
our framework, we conduct experiments on two domains, Montezuma’s Revenge and
Office World, respectively. The results demonstrate the comparable
performance, improved data efficiency, interpretability and transferability.

###### keywords:

Deep Reinforcement Learning, Option Model, Action Model, HTN Model.

##  1 Introduction

Deep Reinforcement Learning (DRL) has achieved tremendous success in complex
and high dimensional environments such as Go [20, 21] and Atari Games [14]. It
interacts with environments and improves its policy with the collected
experience, by maximizing the long term reward. Recent criticism on DRL mostly
focuses on the lack of transferability, interpretability, and data efficiency.
The policy learnt from an environment often fails in another unseen
environment. Due to the use of black-box neural networks for function
approximation, the intrinsic lack of interpretability issue naturally raises
in DRL, which disables the agent to explain its actions in a human-
understandable way and earn people’s trust in critical areas such as
autonomous driving [1] and chemical engineering [27]. Besides, DRL often
requires a large amount of data to learn a satisfying policy in complex
environments. The process of collecting experiences for learning the policy is
time-consuming and the sample efficiency is low.

To alleviate those issues, researchers have investigated the combination of
HRL and symbolic planning to improve transferability, interpretability, and
data efficiency [16, 11, 25, 12, 6, 17, 10]. In those approaches, the original
MDP is divided into two levels. The higher level utilizes a symbolic planner
with a given planning model to generate plans for selecting _options_ , while
the lower-level interacts with the environment to accomplish the selected
options. This two-level structure helps alleviating the sparse reward issue,
and improves sampling efficiency with the help of generated plans. In those
work, however, they require the planning models have been provided by domain
experts. In many real-world applications, however, it is often difficult to
create planning models by hand [26], especially when the environment is
complicated. A more realistic idea is to automatically learn planning models
from training data [31, DBLP:DBLP:journals/ai/YangWJ07, 15, 13, 7] and exploit
the learnt planning models to generate plans for guiding the exploration of
options. Although there is indeed an approach [17] proposed to learn planning
models automatically, they still need to manually define a major part of the
models in advance. Besides, the planning goal in this approach is kept
unchanged while it is dynamically adapted to maximize the external reward in
our framework.

Figure 1: State Mapping Function in Montezuma’s Revenge

In our previous work [8], we propose a novel framework, namely SORL, which
stands for Symbolic Options for Reinforcement Learning, to learn action models
to help the exploration of actions in reinforcement learning. We assume that
there exists a function $F$, mapping high dimensional states to symbolic
states and enabling us to learn symbolic action models and options. As shown
in Figure 1, we extract the position of the man in red and the key from the
high-dimensional state to obtain the corresponding symbolic state. When the
agent walks from the middle ladder to the right ladder, the key still exists
and the environment does not give any feedback (e.g., zero reward). This can
be seen as a symbolic transition and we can generate the corresponding action
model as shown in Figure 2. Then, we use a planner with the learned action
models and a planning goal as input to generate a plan and use it to instruct
the learning of the agent.

Figure 2: An Action model in Montezuma’s Revenge

Based on the assumption, SORL features a two-level structure, of which the
higher level is a symbolic planner and a meta-controller, and the lower level
is an RL agent interacting with the environment. The higher level utilizes the
collected trajectories from the lower level to learn action models and
symbolic options with minimal human knowledge. After that, the meta-controller
chooses an option according to the plan generated from the planner with the
learned action models and assigns it to the lower level. By interacting with
the environments, the lower level learns a policy to reach the assigned option
and sends the collected experience to the higher level. This cross-
fertilization structure not only helps alleviating the sparse and delayed
reward problem but also improves the data efficiency.

Inspired by our previous work on learning hierarchical task networks (HTNs)
[29, 32, 24], we conjecture that the hierarchical structures of actions can be
leveraged to help improve the exploration efficiency and the action model
learning accuracy. We thus, in this work, extend our previous work [8] into
learning hierarchical task network models (besides action models), namely
SORL-HTN, as well as presenting more details about our previous work. With the
hierarchical task network models learnt, we can use both task decomposition
relations and action models to better guide the exploration of actions in
reinforcement learning. We exhibit that our approach significantly outperforms
our previous work that only learns action models, when the scale of the
environment is large (with respect to the number of objects and their
relationships).

Different from previous hierarchical task network models learning approahces
[29, 32, 24] and action models learning approaches [26, 36, 37, 33, 34, 30,
35, 28, 31], which assume the number of task models (a task model is composed
of preconditions and effects of the task) and action models to be learnt is
known beforehand, both of our SORL and SORL-HTN do not know exactly “how many”
and “what” task models and action models to be learnt from the environment. We
expect the agent continuously creates new hierarchical task models and action
models (as well as task decomposition methods) via interactions with the
environment, and exploits the new models to guide the exploration of actions
to build policies creatively—we consider this as the _creativity_ property of
an AI agent. We claim that an agent with such creativity can build better
policies with respect to transferability, interpretability, and data-
efficiency.

We summarize our contribution as follows:

  * 1.

Our work is the first one to learn action models, hierarchical task network
models and option models automatically without being told any knowledge of
these models and simultaneously learn RL policies.

  * 2.

We propose a symbolic reinforcement learning framework capable of providing
transferability, interpretability, and improved data-efficiency.

  * 3.

The symbolic option learned by SORL and SORL-HTN is more general, which can
correspond to more than one action model.

##  2 Preliminaries

In this section we establish relevant notation and briefly introduce key
aspects of symbolic planning and reinforcement learning.

###  2.1 Symbolic Planning with PDDL

In PDDL language [2, 3], states are represented as set of propositions and we
call it symbolic states throughout the paper to distinguish  them from states
in RL. Propositions represent the properties of the world and in the  symbolic
state $s$, proposition $p\in s$ if $p$ is true otherwise $(not\ p)\in s$. An
action description called action model is a tuple
$(name,pre^{+},pre^{-},eff^{+},eff^{-})$, where $name$ is the name of the
action, $(pre^{+},pre^{-})$ are the preconditions and $(eff^{+},eff^{-})$ are
the effects. As shown in Fig.2, the action model describes that when the agent
walks from the middle ladder to the right ladder, the key keeps still and the
reward remains unchanged. If $pre^{+}\subset s$ and $s\cap pre^{-}=\emptyset$,
then we can execute action $a$ and obtain the next state

$s^{\prime}=((s-eff^{-})\cup eff^{+}).$

The planning domain $D=(P,A)$ includes the proposition set $P$ and the action
set $A$, which describe the state space and the action space, respectively. A
tuple $(s,a,s^{\prime})$ describes a symbolic transition from state $s$ to
state $s^{\prime}$ after executing action $a$. We define a planning problem
denoted as a triple $(I,P,A,G)$, of which $I$ is an initial state and $G$ is a
goal state. The solution to this problem is called a plan $\pi$, which is a
sequence of actions. After executing the plan, we can obtain a symbolic
transition trace from $I$ to $G$. To obtain such a plan with the maximum
reward, we use a planner called Metric-FF [4], which can handle planning
problems with continuous metrics.

###  2.2 Hierarchical Task Networks Planning

###  2.3 Reinforcement Learning

A Markov Decision Process (MDP) is defined as the tuple
$(\widetilde{S},\widetilde{A},P_{\widetilde{s}\widetilde{s^{\prime}}}^{\widetilde{a}},r^{\widetilde{a}}_{\widetilde{s}},\gamma)$
where $\widetilde{S}$ and $\widetilde{A}$ denote the state space and action
space, respectively, $P_{\widetilde{s}\widetilde{s^{\prime}}}^{\widetilde{a}}$
provides the transition probability of moving from state
$\widetilde{s}\in\widetilde{S}$ to state
$\widetilde{s^{\prime}}\in\widetilde{S}$ after taking action
$\widetilde{a}\in\widetilde{A}$, $r^{\widetilde{a}}_{\widetilde{s}}$ is the
immediate reward obtained after performing action $\widetilde{a}$ at state
$\widetilde{s}$ and $\gamma\in[0,1)$ is a discount factor. The task of RL is
to  obtain a policy

$\pi:\widetilde{S}\rightarrow\widetilde{A}$

that maximizes the expected return

$V_{\pi}(\widetilde{s})={\mathbb{E}_{\pi}}[\sum_{t=0}^{\infty}\gamma^{t}r_{t}\mid\widetilde{s}_{0}=\widetilde{s}]$

where $r_{t}$ is the reward at time step $t$ received by following $\pi$ from
state $\widetilde{s}_{0}=\widetilde{s}$. The state-action value function is
defined as follows:

$Q_{\pi}(\widetilde{s},\widetilde{a})={\mathbb{E}_{\pi}}[\sum_{t=0}^{\infty}\gamma^{t}r_{t}\mid\widetilde{s}_{0}=\widetilde{s},\widetilde{a}_{0}=\widetilde{a}].$

###  2.4 Option Framework

Hierarchical Reinforcement Learning (HRL) extends RL with temporally macro
actions that represent high-level behaviors. The option framework [22] models
macro actions as options. In particular, an option $o$ is defined as
$(I_{o}(s),\pi_{o}(s),\beta_{o}(s))$, where initiation condition $I_{o}(s)$
determines whether option $o$ can be executed at state $s$, termination
condition $\beta_{o}(s)$ determines whether option execution terminates at
state $s$ and $\pi_{o}(s)$ is a policy mapping state $s$ to a low-level
action. In this framework, an agent learns to choose an optimal option to be
executed in high level, i.e. the meta controller level, and low level, i.e.
the controller level that learns optimal policies to reach the option. An
explicit assumption is that the set of options is predefined by human experts.

##  3 Problem Formulation

We define the reinforcement learning environment by a tuple
$\langle\mathcal{P}^{sym},F,\mathcal{M}\rangle$, where
$\mathcal{P}^{sym}=\langle I,G,P,A\rangle$ is a symbolic planning problem,
$\mathcal{M}=\langle\widetilde{S},\widetilde{A},\widetilde{R},\widetilde{P},\gamma\rangle$
is a MDP problem, and $F$ is a function mapping a low-level state
$\widetilde{s}$ to high-level symbolic state $s$. In symbolic planning problem
$\mathcal{P}^{sym}$, $I$ is an initial state. $G$ is a goal state. $P$ is a
set of propositions represented by planning language PDDL with prior
knowledge, and it is used to describe symbolic states $S$, where $S\subseteq
2^{P}$. The initial state $I$ and goal $G$ satisfy $I\in S$ and $G\in S$. $A$
is a set of action models that is used to transit a symbolic state to another:
$S\times A\rightarrow S$. Each action model is learned by the meta-controller
module through symbolic state pairs. In MDP problem $\mathcal{M}$,
$\widetilde{A}$ and $\widetilde{S}$ are sets of low-level actions and low-
level states, respectively. $\widetilde{R}$ is the reward function.
$\widetilde{P}$ is a set of ???

Taking the Figure 1 as an example, in the game of the Montezuma’s Revenge, the
low-level state is the picture of the game scene and the high-level symbolic
state is composed of the propositions describing the location of the agent and
the existence of the key.

##  4 The SORL Framework with Learnt Action Models

Figure 3: The SORL framework

This framework aims to learn action models, which can be utilized by a
symbolic planner to generate a sequence of options and achieve the maximal
cumulative reward.

As shown in the Figure 3, the SORL framework includes three components: (1) a
planner for generating plans, (2) a meta-controller for generating action
models, goals and choosing the goal option, and (3) an option set for
interacting with the environment. The meta-controller first takes the symbolic
state pairs and their external rewards as input and outputs action models and
a goal. Noted that the state pairs set are empty in the beginning. Then the
planner takes the action models as input and computes a plan. Next, the meta-
controller receives the plan from the planner and chooses an option. Each
option in option set can be regarded as an agent. The chosen agent keeps
interacting with the environment until accomplishing the option or reaching
the maximal steps, and the low-level state traces will be transformed into
symbolic state pairs by the label function $F$ and sent back to meta
controller. The meta-controller continues learning action models and symbolic
options from gained symbolic state pairs and external rewards. We repeat these
procedures $num\\_episodes$ times. With the proceeding of learning, our
approach keeps updating action models and planning goals and the planner is
able to generate plans achieving better rewards.

###  4.1 Option Set

##### Symbolic Option

In this paper, we propose a novel option framework which is called _symbolic
option_. A symbolic option is computed by symbolic state pairs gained from
trajectories instead of manual setting in advance,  requiring less prior
knowledge in our approach. We define a symbolic option by $so=(pre,\pi,eff)$.
$\pi$ is a low-level policy. $pre$ is an union of preconditions, including
$pre^{+}$ and $pre^{-}$. It’s created and updated when the meta-controller
generates action models. Similarly, $eff$ is composed of $eff^{+}$ and
$eff^{-}$, describing the effects of the symbolic option. As for a symbolic
option $so$ and a high-dimension state $\widetilde{s}$, we compute initiation
condition $I_{so}(\widetilde{s})$ by Equation (1) and termination condition
$\beta_{so}(\widetilde{s})$ by Equation (2).  A symbolic option  can be
executed based on $\widetilde{s}$ only if $I_{so}(\widetilde{s})=True$.
Similarly, it terminates only if $\beta_{so}(\widetilde{s})=True$.

$I_{so}(\widetilde{s})=\left\\{\begin{array}[]{rcl}True&pre^{+}\subset
F(\widetilde{s}),F(\widetilde{s})\cap pre^{-}=\emptyset\\\
False&otherwise\end{array}\right.$ (1)

$\beta_{so}(\widetilde{s})=\left\\{\begin{array}[]{rcl}True&eff^{+}\subset
F(\widetilde{s}),eff^{-}\cap F(\widetilde{s})=\emptyset\\\
False&otherwise\end{array}\right.$ (2)

Note that the inherent symbolic propositions of our symbolic option provide
better interpretability compared to those approaches based on black-box neural
networks.

In terms of the low-level policy $\pi$, it can be learned by interacting with
the environments with the intrinsic rewards given by the meta-controller.

##### Global Option

At the beginning of our algorithm, the option set contains no symbolic options
but a global option

$o_{G}=(I_{G}(s),\pi,\beta_{G}(s)),$

where $I_{G}(\widetilde{s})\equiv True$, $\beta_{G}(\widetilde{s})=True$ if
symbolic state changes and $\pi=random(\widetilde{A})$. We use
$random(\widetilde{A})$ to indicate that the global option each time  chooses
a random action $\widetilde{a}\in\widetilde{A}$. Intuitively, the global
option is available for any state and it keeps random ly exploring until the
symbolic state changes.  Hence, in order to discover new action models,  the
meta-controller outputs the global option when the plan is empty or all action
models in the plan has been executed.

##### Symbolic State Pair and External Reward

Given an option $o_{j}$ under state $\widetilde{s}$, the lower level policy
interacts with the environment and output a pair of symbolic states
$(s_{1},s_{2})$ and external reward $r_{e}$, denoted by

$(s_{1},s_{2}),r_{e}=ExecuteOption(\widetilde{s},o_{j}).$

If the chosen option $o_{j}$ is not available for $\widetilde{s}$, i.e.,
$I_{j}(\widetilde{s})=False$, both of the output pair and the reward are
$None$. Otherwise, if the chosen option $o_{j}$ is able to be executed, we let
$s_{1}=F(\widetilde{s})$ and the policy $\pi_{j}$ first choose s an action
$\widetilde{a}$ and we can  obtain the next state $\widetilde{s^{\prime}}$ and
its reward $\widetilde{r}$ by interacting with the environments. Then the
controller adds experience
$(\widetilde{s},\widetilde{a},\widetilde{s^{\prime}},\widetilde{r})$ to the
$o_{j}$’s replay buffer.  We keep executing action by following the low-level
policy and update the states and rewards until until
$\beta_{j}(\widetilde{s^{\prime}})=True$ or reaching the maximum steps, which
means the option has been successfully executed  or not. Finally, if  the
option $o_{j}$ is successfully executed,  we set the output symbolic state
pair as $(s_{1},s_{2})$ of which $s_{2}=F(\widetilde{s^{\prime}})$ and the
external reward $r_{e}$ be the accumulated sum of the environment rewards
during interacting.

###  4.2 Meta-controller

In this section, we introduce our Meta-controller in detail. Meta-controller
takes symbolic state pairs and their external rewards as input, and first
generates action models and a planning goal and then chooses a an option
according to the plan from planner.

##### Action Model

Given symbolic state pairs and their rewards, meta-controller generates action
models by

$A,F_{A,O},O=GenerateActionModels(R,O,sr).$

The function indicates it takes a dictionary $R$,an option set $O$ and the
success ratio set $sr$ as inputs, and outputs a generated action set $A$, a
mapping function $F_{A,O}$ and the updated option set $O$. Dictionary $R$
includes mappings from a symbolic state pair to its external rewards.
$F_{A,O}$ transfers action models to options. The success ratio set $sr$
records the  percentage of action models successful executed each 100 times.

As for a symbolic state pair $(s_{1},s_{2})_{i}\in R$, we can get a
corresponding action model

$a_{i}=(name,pre^{+},pre^{-},eff^{+},eff^{-}).$

Noted that the action model and a symbolic state pair is a one-to-one match.
Given a state pair $(s_{1},s_{2})_{i}$, the $name$ of $a_{i}$ is the index of
action models, denoted by $act_{i}$, and $pre^{+}=\\{p|p\in s_{1}\\}$,
$pre^{-}=\\{p|p\notin s_{1}\\}$. Next we let $eff^{+}=s_{2}-s_{1}$ and
$eff^{-}=s_{1}-s_{2}$, where $a-b$ is a set subtraction indicating set $a$
subtracts the intersection of set $a$ and set $b$. In order to generate a plan
gaining a maximum reward, we use the metric constant $quality$ to denote the
cumulative reward of the plan and add the proposition
“$(increase\quad(quality)\quad\rho_{i})$” into $eff^{+}$. Finally, we get an
action which is called $act_{i}$, and we define the gain ed reward of
$act_{i}$ by $\rho_{i}$. To encourage the planner to  generate a plan
including the exploring action model, the reward $\rho_{i}$ is composed of
mean external reward and exploration reward, computed by Equation (3), where
$R[(s_{1},s_{2})_{i}]$ is the external rewards list and $r_{E}$ is the
exploration rewards. The exploration rewards is computed by Equation (4),
where c is a constant and $sr[i]$ is the success rate of $act_{i}$, which
means exploration reward decreases as success rate increases.

$\rho_{i}=mean(R[(s_{1},s_{2})_{i}])+r_{E}$ (3)

$r_{E}=\left\\{\begin{array}[]{ll}c(1-sr[i])&act_{i}\ is\ being\ explored\\\
0&otherwise\end{array}\right.$ (4)

If  there exists a symbolic option $o_{j}=(pre_{j},\pi_{j},eff_{j})$ where
$eff_{j}=eff$ after we attain an action model,  we update $pre_{j}^{+}$ to a
union of $pre^{+}$ and $pre_{j}^{+}$, and $pre_{j}^{-}$ to a union of
$pre^{-}$ and $pre_{j}^{-}$. Otherwise, we create a new symbolic option
$o_{j}=(pre,\pi_{j},eff)$ and add it to the option set $O$. At last, we set
the mapping function $F_{A,O}(act_{i})=o_{j}$. During the exploration,  we
explore each action model sequentially, in other words, we repeat exploring
$act_{i}$ until the success rate of $act_{0}$ to $act_{i-1}$ is higher than
the threshold.

##### Planning Goal

Next Meta-controller outputs a goal to guide planner, aiming at generating a
plan with a maximal reward. The goal is a label function $quality>q$, where
$q$ is the cumulative external rewards of the plan gained in the last episode.
Intuitively, the function constrains the planner to compute a plan with a
largest reward compared with the past plans.

##### Chosen Option and Intrinsic Reward

After the planner generates a plan $\Pi=(a_{1},a_{2},\dots,a_{n})$, as for
each action model $a_{i}$, the meta-controller selects a symbolic option from
option set by $o_{j}=F_{A,O}(act_{i})$, and we can get an series of options
$(o_{0},o_{1},\dots,o_{n})$. If all action models in $\Pi$ successfully
finish, which indicates the chosen symbolic options are executed sequentially
and termination conditions are satisfied , then the meta-controller would
choose the global option $o_{G}$ to explore the environment thoroughly.  For
each option $o_{i}=(pre_{i},\pi_{i},\beta_{i})$, we refer to [12] to design
intrinsic rewards:

$r_{i}(\widetilde{s})=\left\\{\begin{array}[]{rcl}\phi&\beta_{i}(\widetilde{s})=True\\\
r&otherwise\end{array}\right.$ (5)

where $\phi$ is a constant and $r$ is the reward gained from the environments
when reach state $\widetilde{s}$.

Algorithm 1 Planning and Learning algorithm for SORL

Input: proposition set $P$, state mapping function $F$, success ratio
threshold $\lambda$

1: Initialization: option set $O\leftarrow\\{o_{G}\\}$, action models set
$A\leftarrow\emptyset$, symbolic state pairs’ external rewards dictionary
$R\leftarrow\emptyset$, action models’ success ratio set
$sr\leftarrow\emptyset$, plan $\Pi_{0}\leftarrow\emptyset$, $q\leftarrow 0$

2: for t=1,2, …, $num\\_episodes$ do

3: Initialize game, get start state $\widetilde{s_{0}}$, $I\leftarrow
F(\widetilde{s_{0}})$, $\Pi^{*}\leftarrow\Pi_{t-1}$

4: $A,F_{A,O},O\leftarrow GenerateActionModels(R,O,sr)$

5: $G\leftarrow(quality>q)$

6: $\Pi_{t}\leftarrow metricFF.solve(I,P,A,G)$

7: if $\Pi_{t}=\emptyset$ then $\Pi_{t}\leftarrow\Pi^{*}$

8: $q\leftarrow 0$

9: for $a_{i}\in\Pi_{t}$ do

10: $o_{j}\leftarrow F_{A,O}[i]$, obtain current state $\widetilde{s}$

11: $(s_{1},s_{2}),r_{e}\leftarrow ExecuteOption(\widetilde{s},o_{j})$

12: append $r_{e}$ into $R[(s_{1},s_{2})]$, $q\leftarrow q+r_{e}$

13: end for

14: while env isn’t terminal do

15: obtain current state $\widetilde{s}$

16: $(s_{1},s_{2}),r_{e}\leftarrow ExecuteOption(\widetilde{s},o_{G})$

17: if $(s_{1},s_{2})$ not in $R$ then

18: $R[(s_{1},s_{2})]\leftarrow list(r_{e})$

19: else

20: append $r_{e}$ into $R[(s_{1},s_{2})]$

21: end if

22: end while

23: train options in $O$ and calculate $sr$

24: end for

###  4.3 Planning and Learning

As shown in Algorithm 1, we firstly initialize an option set $O$ only
including $o_{G}$, an empty action model set $A$, an empty dictionary mapping
symbolic state pairs to their external rewards $R$, an empty action models
success ratio set $sr$ and an empty plan $\Pi_{0}$. When an episode $t$
begins, we first get a start state $\widetilde{s_{0}}$ from environment. We
then compute the symbolic initial state $I$ by $F$ and record the best plan
$\Pi^{*}$, which is the plan generated in the last episode. Then meta-
controller updates action models $A$, symbolic options set $O$, their mapping
function $F_{A,O}$ and  the planning goal $G$. Given current action models $A$
and planning goal $G$, Metric-FF planner [4] generates a new plan $\Pi_{t}$
whose quality is higher than the last plan $\Pi_{t-1}$. If $\Pi_{t}$ is empty,
which indicates Metric-FF couldn’t find a solution to solve the problem, we
let $\Pi_{t}=\Pi^{*}$.

As for each action model $a_{i}$ in plan $\Pi_{t}$, meta-controller chooses a
corresponding symbolic option $o_{j}$ by $F_{A,O}$. Then the controller
interacts with environment by performing Deep Q-Learning, executes the action
chosen by $o_{j}$’s inner policy and stores experience into $o_{j}$’s replay
buffer until $o_{j}$ terminates. After that, we get $o_{j}$’s initial symbolic
state $s_{1}$ and a terminate symbolic state $s_{2}$ and an extrinsic reward
$r_{e}$. In this way, we compute symbolic state pairs and their extrinsic
rewards one by one and record these mappings by a dictionary $R$. Finally,
quality $q$ of plan $\Pi_{t}$ is defined as the accumulated sum of extrinsic
rewards.

If the environment is not finished after executing $\Pi_{t}$, the meta-
controller chooses the global option $o_{G}$ to explore new symbolic states
pairs in the environment. $o_{G}$ stops exploring when the computed symbolic
state changes and we calculate a symbolic state pair $(s_{1},s_{2})$ and its
external reward $r_{e}$. If $(s_{1},s_{2})$ is a new symbolic state pair, we
add it into $R$. This process repeats until the environment is  terminated.
Finally, when an episode ends, we train options in $O$ and calculate success
ratio for each action model.

##  5 The SORL-HTN Framework with Learnt HTN Models

Figure 4: The framework of learning HTN and action models

To further leverage the task structures for guiding the exploration of actions
in reinforcement learning, we build a novel algorithm framework to learn
hierarchical task networks based on the action models learnt by Algorithm 1.
An framework of learning HTN and action models is shown in Figure 4, where the
controller module uses the set of (learnt) option models (denoted by ”Option
Set”) to complete the set of subtasks (denoted by ”Subtask Set”) with
interaction to the environment, i.e., executing a low-level ”action” and
receiving ”reward” and new ”state” from the environment. After that, the Meta-
Controller module receives the extrinsic reward computed by the Controller
module and learn hierarchical task network models (i.e., decomposition
methods, which is denoted by ”Method model learner”) and action models
(denoted by ”Action model learner”) based on the extrinsic reward. Finally,
the Symbolic Planner module receives the ”Method models” and ”Action models”
from the Meta-Controller module and computes subtasks and intrinsic rewards
for the Controller module, with respect to reaching the input goal. In the
following, we will address the details of the three modules: Controller, Meta-
Controller, and Symbolic Planner.

Specifically, we provide an algorithm framework, as shown in Algorithm 2, to
detail the learning procedure of Figure 4. In Algorithm 2, Steps 4 and 5 aim
to learn action models and decomposition methods, i.e., the Meta-Controller
module. Step 6 aims to compute a list of subtasks and the intrinsic reward,
i.e., the Symbolic Planner module. Steps 9 to 32 aim to complete subtasks with
interaction to the environment, i.e., the Controller module.

Algorithm 2 An overview of our HTN-based reinforcement learning

Input: tasks $T$, propositions $P$, state mapping function $F$, success
threshold $\lambda$
Output: action models $A$, method models $M$, options $O$

1: Initialization: $O\leftarrow\\{o_{G}\\}$, $A\leftarrow\emptyset$,
$M\leftarrow\emptyset$, symbolic state pairs $SP\leftarrow\emptyset$, plan
$\Pi_{0}\leftarrow\emptyset$, exploration trace $\tau\leftarrow\emptyset$,
success ratios of action models $sr\leftarrow\emptyset$

2: for $t=1,2,\dots,num\\_episodes$ do

3: Initialize game, get initial state $\widetilde{s_{0}}$, $I\leftarrow
F(\widetilde{s_{0}})$

4: $A,F_{A,O},O\leftarrow GenerateActionModels(SP,O,sr)$

5: $M,F_{M,T}\leftarrow GenerateMethodModels(\tau,T,A)$

6: $\Pi_{t}\leftarrow HTNsolver(I,P,A,M,T)$

7: if $\Pi_{t}=\emptyset$, let $\Pi_{t}\leftarrow\Pi_{t-1}$

8: $\tau\leftarrow\emptyset$

9: for $m_{i}\in\Pi_{t}$ do

10: $t_{j}\leftarrow F_{M,T}[m_{i}]$

11: for $a_{k}\in\Pi_{t}(m_{i})$ do

12: $\tau\leftarrow\tau+a_{k}$

13: $o_{l}\leftarrow F_{A,O}[a_{k}]$, obtain current state $\widetilde{s}$

14: $(s_{1},s_{2}),r_{e}\leftarrow ExecuteOption(\widetilde{s},o_{l})$

15: If $\beta(t_{j})=\textbf{true}{}$, let $r_{e}\leftarrow r_{e}+R_{T}$

16: append $r_{e}$ into $SP[(s_{1},s_{2})]$

17: end for

18: end for

19: while env isn’t terminal do

20: obtain current state $\widetilde{s}$, $s\leftarrow F(\widetilde{s})$

21: $A_{s}\leftarrow\\{$ operator instances applicable to $s\\}$

22: if $A\neq\emptyset$ then

23: nondeterministically choose an $a_{s}\in A_{s}$

24: $\tau\leftarrow\tau+a_{s}$, $o_{m}\leftarrow F_{A,O}[a_{s}]$

25: $(s_{1},s_{2}),r_{e}\leftarrow ExecuteOption(\widetilde{s},o_{m})$

26: append $r_{e}$ into $SP[(s_{1},s_{2})]$

27: else

28: $(s_{1},s_{2}),r_{e}\leftarrow ExecuteOption(\widetilde{s},o_{G})$

29: If $(s_{1},s_{2})$ not in $SP$, let $SP[(s_{1},s_{2})]\leftarrow
list(r_{e})$, otherwise, append $r_{e}$ into $SP[(s_{1},s_{2})]$

30: end if

31: end while

32: train options in $O$ and calculate $sr$

33: end for

##  6 Experiment

In this section, we evaluate our approach on two domains, Office World and
Montezuma’s Revenge in terms of data-efficiency, interpretability and
transferability.

(a) Office World

(b) Learning Curve

(c) Propositions

(d) Learned Action Models and Options

(e) Learning Curve

Figure 5: Experimental Results in the Office World

###  6.1 Office World

We first evaluate our approach on  the Office World [5] which is a simple
multitask environment. In this environment,  being initialized at a random
location, the agent can move  towards one of the four cardinal directions.
Actions are valid only if the movement does not go through a wall. The agent
can pick up  cups of coffee or mail s when it reaches the cell marked with
blue cups or green envelops, respectively.  He can deliver coffee or mail to
the office  by reaching the cell marked with a purple hand. The symbol $*$
means the place where the agent can not stay or reach.

####  6.1.1 Setup

In this environment, the start location of the agent is randomly initialized
at every episode. The agent is required to finish three tasks. The first and
the second are to deliver a cup of coffee or a piece of mail to the office
while the third is to hand both objects to the office. We compared SORL to
h-DQN, a goal based HRL  approach [9]. Since the state and action space are
finite, we choose to implement these two approaches with q-table in both high
and low levels.

####  6.1.2 Results

We evaluate our approaches in terms of data-efficiency, interpretability and
transferability.

  * 1.

Data-efficiency In order to validate the data-efficiency, we train these two
approaches in the three tasks and compare the corresponding performance at the
same interaction steps. To demonstrate the transferability, we train the agent
in task 3 along with the options learned in tasks 1 and 2. To implement our
approach, we design the propositions as shown in Fig. 5(c). As shown in
Fig.5(b) , from Task1 to Task3, SORL can get rewards faster than HRL.

  * 2.

Interpretability Fig.5(d) shows action models and symbolic options learned in
each task. Those action models describe the reason of making decisions at each
step in a human understandable way. For example, we can explicitly know $act1$
in task1 can be executed when the agent gets coffee and does not deliver it to
the office, and the agent would deliver the coffee to the office and get a
reward of 100 when $act1$ is executed.

  * 3.

Transferability  By utilizing the options learned in tasks 1 and 2, we test
the tranferability of SORL and H-DQN in Task 3 and denote them as $SORL_{t}$
and $HRL_{t}$. As shown in Fig5(c), the performance of SORL and HRL is
improved when transferring the learned knowledge. It verifies that compared to
SORL, the converging speed of $SORL_{t}$ improves dramatically with only half
of samples. We conjecture that the SORL is able to transfer the learned
knowledge into other unseen environments.

###  6.2 Montezuma’s Revenge

Montezuma’s Revenge is an Atari game with sparse and delayed rewards. It
requires the player to navigate through several rooms while collecting
treasures. We conduct our experiments based on the first room shown in
Fig.6(a). In this room, the player only obtains positive rewards when it
fetches the key (+100) or opens a door (+300). Otherwise, the player would not
receive any reward signal. The optimal solution is to climb down the ladders
to obtain the key, then return back to the platform and open a door, resulting
in a maximum reward (+400).

####  6.2.1 Setup

We compare our approach with HRL [9] and SDRL [12] as baselines, where SDRL is
an approach that combines symbolic planning and RL with excellent results in
complex environments with sparse rewards. SORL can automatically learn the
action models while they are pre-defined by experts in SDRL. Besides, the
option model can correspond to multiple action models in SORL while one in
SDRL. We implement these approaches under an option-based HRL framework. In
terms of the low level, we follow the network architecture used in [9] and
train this network with double-Q learning [23] and prioritized experience
replay [18]. Besides, both SORL and SDRL use a planner to generate high level
policy while HRL utilizes a neural network. The intrinsic reward follows 5
with $\phi=100$. The maximum steps in an episode and the threshold of success
rate are set to be 500 and 0.95, respectively. To describe the environment, we
abstract four local propositions (e.g., MiddleLadder, RightDoor, LeftLadder
and RightLadder) and an object (Key).

(a) Learning Curve

(b) Learned Action Models

Figure 6: Experimental Results in Montezuma’s Revenge

####  6.2.2 Results

We present the experimental result s in Fig.6. It is evident that SORL can
achieve the maximum reward (+400) in 0.7M samples while both SDRL and HRL need
more than 1.5M samples, indicating the superior data-efficiency of SORL.
However, to pick up the key (reward +100), SORL needs to interact with the
environment with more than 0.3M steps, at which SDRL and HRL fall into the
local optimum. This is because SORL randomly explores symbolic options and it
is easier to find options closer to the starting point. After finding these
options, SORL would train them sequentially instead of directly learning
options on the path of getting the key. One option model corresponds to one
action model in SDRL while several action models in SORL. The ability of
reusing the learned symbolic options enables SORL to converge faster than
SDRL. Take Fig.8 as an example, the $opt1$ representing the move from middle
ladder to right door, is firstly trained at the beginning when the player does
not get the key. After the player picks up the key, SORL only needs a small
amount of data to fine-tune $opt1$ when the player moves from the middle
ladder to right door with a key. However, both SDRL and HRL start training the
options after the player moves to the middle ladder with a key, consuming more
interaction resources. Different from option-based HRL and SDRL, SORL can
learn the initial and termination condition of symbolic options automatically.
The action models used in SDRL need to be constructed by human in advance
while they are learned from the trajectories in SORL, saving labour resources.

Figure 7: Examples of Learned Action Models Figure 8: learned Symbolic Options

We present some of the learned action models in Fig.7 and the effects of
symbolic options in Fig.8. Fig. 7 describes the preconditions and effects of
each action model and we can see that if the player is at LeftLadder and the
key exists, then the player can obtain a key and reward (+100) by executing
action5. Fig. 8 shows the learned options in SORL and the order of options
actually does not match the optimal order because SORL randomly explore the
environment and options 0-3 are easier to learn. We describe the meaning of
all learned action models and their corresponding options in Fig. 6(b). It is
easy to see that act7 to act10 correspond to the options explored before, so
these options can be reused to improve the data-efficiency.

##  7 Conclusions

In this paper, we propose two novel frameworks SORL and SORL-HTN which can
automatically learn action models, hierarchical task models, and symbolic
options from the trajectories and the symbolic planner can instruct RL to
explore efficiently in environments with sparse and delayed rewards. Compared
with other approaches, the experimental results demonstrate the better
sampling efficiency of our approach. Moreover, SORL and SORL-HTN require less
prior knowledge and provides interpretability and transferability by the
learned action models and symbolic options. In the future, it would be
interesting to investigate possibility of learning more expressive planning
models in real-world applications, as well as different learning mechanisms,
such as transfer learning [35, 19].

## Acknowledgement

This research was funded by the National Natural Science Foundation of China
(Grant No. 62076263), Guangdong Natural Science Funds for Distinguished Young
Scholar (Grant No. 2017A030306028), Guangdong special branch plans young
talent with scientific and technological innovation (Grant No. 2017TQ04X866),
Pearl River Science and Technology New Star of Guangzhou and Guangdong
Province Key Laboratory of Big Data Analysis and Processing.

## References

  * Aradi [2020] Aradi, S., 2020. Survey of deep reinforcement learning for motion planning of autonomous vehicles. CoRR abs/2001.11231.
URL https://arxiv.org/abs/2001.11231

  * Gerevini [2020] Gerevini, A. E., 2020. An introduction to the planning domain definition language (PDDL): book review. Artif. Intell. 280, 103221.
URL https://doi.org/10.1016/j.artint.2019.103221

  * Ghallab et al. [1998] Ghallab, M., Knoblock, C., Wilkins, D., Barrett, A., Christianson, D., Friedman, M., Kwok, C., Golden, K., Penberthy, S., Smith, D., Sun, Y., Weld, D., 08 1998. Pddl - the planning domain definition language.
  * Hoffmann [2002] Hoffmann, J., 2002. Extending FF to numerical state variables. In: van Harmelen, F. (Ed.), Proceedings of the 15th Eureopean Conference on Artificial Intelligence, ECAI’2002, Lyon, France, Jusly 2002. IOS Press, pp. 571–575.
  * Icarte et al. [2018] Icarte, R. T., Klassen, T. Q., Valenzano, R. A., McIlraith, S. A., 2018. Advice-based exploration in model-based reinforcement learning. In: Bagheri, E., Cheung, J. C. K. (Eds.), Advances in Artificial Intelligence - 31st Canadian Conference on Artificial Intelligence, Canadian AI 2018, Toronto, ON, Canada, May 8-11, 2018, Proceedings. Vol. 10832 of Lecture Notes in Computer Science. Springer, pp. 72–83.
URL https://doi.org/10.1007/978-3-319-89656-4_6

  * Illanes et al. [2020] Illanes, L., Yan, X., Icarte, R. T., McIlraith, S. A., 2020. Symbolic plans as high-level instructions for reinforcement learning. In: Beck, J. C., Buffet, O., Hoffmann, J., Karpas, E., Sohrabi, S. (Eds.), Proceedings of ICAPS. AAAI Press, pp. 540–550.
  * James et al. [2020] James, S., Rosman, B., Konidaris, G., 2020. Learning portable representations for high-level planning. In: Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event. Vol. 119 of Proceedings of Machine Learning Research. PMLR, pp. 4682–4691.
URL http://proceedings.mlr.press/v119/james20a.html

  * Jin et al. [2022] Jin, M., Ma, Z., Jin, K., Zhuo, H. H., Chen, C., Yu, C., 2022. Creativity of AI: automatic symbolic option discovery for facilitating deep reinforcement learning. In: Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourth Conference on Innovative Applications of Artificial Intelligence, IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22 - March 1, 2022. AAAI Press, pp. 7042–7050.
URL https://ojs.aaai.org/index.php/AAAI/article/view/20663

  * Kulkarni et al. [2016] Kulkarni, T. D., Narasimhan, K., Saeedi, A., Tenenbaum, J., 2016. Hierarchical deep reinforcement learning: Integrating temporal abstraction and intrinsic motivation. In: Lee, D. D., Sugiyama, M., von Luxburg, U., Guyon, I., Garnett, R. (Eds.), Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain. pp. 3675–3683.
URL
https://proceedings.neurips.cc/paper/2016/hash/f442d33fa06832082290ad8544a8da27-Abstract.html

  * Lee et al. [2021] Lee, J., Katz, M., Agravante, D. J., Liu, M., Klinger, T., Campbell, M., Sohrabi, S., Tesauro, G., 2021. Ai planning annotation in reinforcement learning: Options and beyond. In: Planning and Reinforcement Learning PRL Workshop at ICAPS.
  * Leonetti et al. [2016] Leonetti, M., Iocchi, L., Stone, P., 2016. A synthesis of automated planning and reinforcement learning for efficient, robust decision-making. Artif. Intell. 241, 103–130.
URL https://doi.org/10.1016/j.artint.2016.07.004

  * Lyu et al. [2019] Lyu, D., Yang, F., Liu, B., Gustafson, S., 2019. SDRL: interpretable and data-efficient deep reinforcement learning leveraging symbolic planning. In: The Thirty-Third AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27 - February 1, 2019. AAAI Press, pp. 2970–2977.
URL https://doi.org/10.1609/aaai.v33i01.33012970

  * Martínez et al. [2016] Martínez, D., Alenyà, G., Torras, C., Ribeiro, T., Inoue, K., 2016\. Learning relational dynamics of stochastic domains for planning. In: Coles, A. J., Coles, A., Edelkamp, S., Magazzeni, D., Sanner, S. (Eds.), Proceedings of the Twenty-Sixth International Conference on Automated Planning and Scheduling, ICAPS 2016, London, UK, June 12-17, 2016. AAAI Press, pp. 235–243.
URL http://www.aaai.org/ocs/index.php/ICAPS/ICAPS16/paper/view/13079

  * Mnih et al. [2015] Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., et al., 2015\. Human-level control through deep reinforcement learning. nature 518 (7540), 529–533.
  * Ng and Petrick [2019] Ng, J. H. A., Petrick, R. P. A., 2019. Incremental learning of planning actions in model-based reinforcement learning. In: Kraus, S. (Ed.), Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019. ijcai.org, pp. 3195–3201.
URL https://doi.org/10.24963/ijcai.2019/443

  * Ryan [2002] Ryan, M. R. K., 2002. Using abstract models of behaviours to automatically generate reinforcement learning hierarchies. In: Sammut, C., Hoffmann, A. G. (Eds.), Machine Learning, Proceedings of the Nineteenth International Conference (ICML 2002), University of New South Wales, Sydney, Australia, July 8-12, 2002. Morgan Kaufmann, pp. 522–529.
  * Sarathy et al. [2020] Sarathy, V., Kasenberg, D., Goel, S., Sinapov, J., Scheutz, M., 2020. Spotter: Extending symbolic planning operators through targeted reinforcement learning. arXiv preprint arXiv:2012.13037.
  * Schaul et al. [2016] Schaul, T., Quan, J., Antonoglou, I., Silver, D., 2016. Prioritized experience replay. In: Bengio, Y., LeCun, Y. (Eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
URL http://arxiv.org/abs/1511.05952

  * Shen et al. [2020] Shen, J., Zhuo, H. H., Xu, J., Zhong, B., Pan, S. J., 2020. Transfer value iteration networks. In: The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020. AAAI Press, pp. 5676–5683.
URL https://aaai.org/ojs/index.php/AAAI/article/view/6022

  * Silver et al. [2016] Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., van den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M., Dieleman, S., Grewe, D., Nham, J., Kalchbrenner, N., Sutskever, I., Lillicrap, T. P., Leach, M., Kavukcuoglu, K., Graepel, T., Hassabis, D., 2016\. Mastering the game of go with deep neural networks and tree search. Nat. 529 (7587), 484–489.
URL https://doi.org/10.1038/nature16961

  * Silver et al. [2017] Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., et al., 2017. Mastering the game of go without human knowledge. Nature 550 (7676), 354–359.
  * Sutton et al. [1999] Sutton, R. S., Precup, D., Singh, S. P., 1999. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artif. Intell. 112 (1-2), 181–211.
URL https://doi.org/10.1016/S0004-3702(99)00052-1

  * van Hasselt et al. [2016] van Hasselt, H., Guez, A., Silver, D., 2016. Deep reinforcement learning with double q-learning. In: Schuurmans, D., Wellman, M. P. (Eds.), Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA. AAAI Press, pp. 2094–2100.
URL http://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12389

  * Xiao et al. [2020] Xiao, Z., Wan, H., Zhuo, H. H., Herzig, A., Perrussel, L., Chen, P., 2020. Refining HTN methods via task insertion with preferences. In: AAAI. AAAI Press, pp. 10009–10016.
  * Yang et al. [2018] Yang, F., Lyu, D., Liu, B., Gustafson, S., 2018. PEORL: integrating symbolic planning and hierarchical reinforcement learning for robust decision-making. In: Lang, J. (Ed.), Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI 2018, July 13-19, 2018, Stockholm, Sweden. ijcai.org, pp. 4860–4866.
URL https://doi.org/10.24963/ijcai.2018/675

  * Yang et al. [2007] Yang, Q., Wu, K., Jiang, Y., 2007. Learning action models from plan examples using weighted MAX-SAT. Artif. Intell. 171 (2-3), 107–143.
URL https://doi.org/10.1016/j.artint.2006.11.005

  * Zhou et al. [2017] Zhou, Z., Li, X., Zare, R. N., 2017. Optimizing chemical reactions with deep reinforcement learning. ACS central science 3 (12), 1337–1344.
  * Zhuo [2015] Zhuo, H. H., 2015. Crowdsourced action-model acquisition for planning. In: AAAI. pp. 3439–3446.
  * Zhuo et al. [2009] Zhuo, H. H., Hu, D. H., Hogg, C., Yang, Q., Muñoz-Avila, H., 2009. Learning HTN method preconditions and action models from partial observations. In: Boutilier, C. (Ed.), IJCAI 2009, Proceedings of the 21st International Joint Conference on Artificial Intelligence, Pasadena, California, USA, July 11-17, 2009. pp. 1804–1810.
URL http://ijcai.org/Proceedings/09/Papers/300.pdf

  * Zhuo and Kambhampati [2013] Zhuo, H. H., Kambhampati, S., 2013. Action-model acquisition from noisy plan traces. In: Rossi, F. (Ed.), IJCAI 2013, Proceedings of the 23rd International Joint Conference on Artificial Intelligence, Beijing, China, August 3-9, 2013. IJCAI/AAAI, pp. 2444–2450.
URL http://www.aaai.org/ocs/index.php/IJCAI/IJCAI13/paper/view/6873

  * Zhuo and Kambhampati [2017] Zhuo, H. H., Kambhampati, S., 2017. Model-lite planning: Case-based vs. model-based approaches. Artif. Intell. 246, 1–21.
URL https://doi.org/10.1016/j.artint.2017.01.004

  * Zhuo et al. [2014] Zhuo, H. H., Munoz-Avila, H., Yang, Q., 2014. Learning hierarchical task network domains from partially observed plan traces. Artif. Intell. 212, 134–157.
URL https://doi.org/10.1016/j.artint.2014.04.003

  * Zhuo et al. [2013a] Zhuo, H. H., Nguyen, T. A., Kambhampati, S., 2013a. Model-lite case-based planning. In: AAAI.
  * Zhuo et al. [2013b] Zhuo, H. H., Nguyen, T. A., Kambhampati, S., 2013b. Refining incomplete planning domain models through plan traces. In: IJCAI. pp. 2451–2458.
  * Zhuo and Yang [2014] Zhuo, H. H., Yang, Q., 2014. Action-model acquisition for planning via transfer learning. Artif. Intell. 212, 80–103.
URL https://doi.org/10.1016/j.artint.2014.03.004

  * Zhuo et al. [2010] Zhuo, H. H., Yang, Q., Hu, D. H., Li, L., 2010. Learning complex action models with quantifiers and logical implications. Artif. Intell. 174 (18), 1540–1569.
URL https://doi.org/10.1016/j.artint.2010.09.007

  * Zhuo et al. [2011] Zhuo, H. H., Yang, Q., Pan, R., Li, L., 2011. Cross-domain action-model acquisition for planning via web search. In: Bacchus, F., Domshlak, C., Edelkamp, S., Helmert, M. (Eds.), Proceedings of the 21st International Conference on Automated Planning and Scheduling, ICAPS 2011, Freiburg, Germany June 11-16, 2011. AAAI.
URL http://aaai.org/ocs/index.php/ICAPS/ICAPS11/paper/view/2678