# Imitation Learning: A Survey of Learning Methods, Environments and Metrics

Nathan Gavenski, Odinaldo Rodrigues, Michael Luck

###### Abstract

Imitation learning is an approach in which an agent learns how to execute a
task by trying to mimic how one or more teachers perform it. This learning
approach offers a compromise between the time it takes to learn a new task and
the effort needed to collect teacher samples for the agent. It achieves this
by balancing learning from the teacher, who has some information on how to
perform the task, and deviating from their examples when necessary, such as
states not present in the teacher samples. Consequently, the field of
imitation learning has received much attention from researchers in recent
years, resulting in many new methods and applications. However, with this
increase in published work and past surveys focusing mainly on methodology, a
lack of standardisation became more prominent in the field. This non-
standardisation is evident in the use of environments, which appear in no more
than two works, and evaluation processes, such as qualitative analysis, that
have become rare in current literature. In this survey, we systematically
review current imitation learning literature and present our findings by  (i)
classifying imitation learning techniques, environments and metrics by
introducing novel taxonomies;  (ii) reflecting on main problems from the
literature; and  (iii) presenting challenges and future directions for
researchers.

###### Index Terms:

Imitation Learning

##  I Introduction

Imitation learning is a socially-inspired machine learning approach that
consists of learning a task from examples provided by a teacher. In other
words, this approach refers to an agent’s acquisition of skills or behaviours
by observing a teacher performing a given task [1]. This approach benefits the
agent since it learns from samples that theoretically indicate a successful
behaviour, whether having to search which behaviours solve the task (a process
that random trial-and-error learning techniques have to go through). Moreover,
this idea from imitation learning is compelling since when learning something
new, the learner tries to learn from a proficient source and adopt more
expert-like behaviour. Hence, directing the learning process towards specific
observed behaviours also allows for more human-centric approaches.

In recent years, Imitation learning evolved significantly from Pomerleau’s
original work on behavioural cloning [2]. Some work expanded the application
of imitation learning with more complex learner structures, such as ensemble
methods [3]. Other work proposed novel learning schematics, such as
adversarial learning [4] and self-supervised learning [5]. Thus, these
approaches rely on machine learning techniques to optimise its agent’s
behaviour. In contrast, other researchers focused on agent techniques by
proposing the usage of exploration [6] and remapping of latent actions [7].
Given these improvements, imitation learning has seen many applications,
including robotics [8, 9, 10, 11, 12, 13, 14, 15, 16, 17], game-playing [18,
19, 20, 6, 21, 22, 7, 23, 24, 25, 9, 26, 27, 28], and natural language
processing [29, 30, 31, 32, 33].

With this increase in popularity, imitation learning has seen a rise in the
number of experimental environments and performance metrics, as we show in
this survey. These new metrics and environments, along with other surveys [1,
34, 35] focusing more on learning techniques, create a lack of standardisation
in the evaluation process, making meaningful comparisons between work
challenging. Hence, this survey examines the field from a different
perspective from the traditional model-free and model-based views employed in
the past. In it, we present the field based on the most predominant learning
techniques, metrics and environments. This work presents the first taxonomies
for environments and metrics and a new methodological one. We believe that our
taxonomy for methods does not nullify other existing methodological taxonomies
but rather complements them by classifying novel trends in imitation learning
methods in more detail. Nevertheless, this survey acknowledges the prior
contributions from previous surveys and incorporates them when necessary.

We start the survey by describing the selection criteria for including work in
the systematic review and highlighting the main differences between this
survey and previous ones in Section II. Section III lays out the formal
underpinning of imitation learning, defining the critical elements involved in
imitation learning tasks, allowing our discussion to remain mathematically
rigorous. We then present the different approaches to imitation learning in
Section IV, introducing a new taxonomy to highlight a new trend among learning
approaches. In Section V, we present the different environments used in
imitation learning and propose the first taxonomy for imitation learning
environments based on the role of the environment in the learning process.
Section VI explains the different metrics used to evaluate imitation learning
approaches, groups them based on each of their measures, and presents the
first taxonomy for imitation learning metrics. In Section VII, we discuss key
insights derived from the literature, pose key challenges for the field, and
possible future directions for imitation learning. Finally, Section VIII
summarises the main contributions of this survey by emphasising the need for
consistent use of evaluation processes and environment and how researchers can
use these new taxonomies to achieve a more solid and systematic approach.

##  II Review preliminaries

In this work, we use a snowballing approach and two different surveys:
‘Imitation Learning: A Survey of Learning Methods’ [1] from $2017$ and
‘Imitation Learning: Progress, Taxonomies and Challenges’ [35] from $2021$ as
a starting point. Hussein et al.’s work is the most cited imitation learning
survey, which makes a good starting point in a snowballing procedure. It cites
significant work on imitation learning and defines various approaches used to
learn from demonstrations. Moreover, with more than $800$ citations, the
snowballing process is bound to find the most relevant work to our desired
subject. Given that the first survey is older, we add Zheng et al. as a more
recent survey. By adding a more novel survey, we are more assured of finding
relevant work that we might miss from an older survey in a snowballing
process. To select all relevant work, we follow two general guidelines when
filtering all papers found during the snowballing process. The first criterion
is that all work must be from inverse reinforcement learning or imitation
learning literature because, historically, imitation learning approaches
appeared as part of reinforcement learning literature. Furthermore, imitation
learning approaches that use inverse reinforcement learning require teacher
demonstrations to be considered imitation learning, we do not exclude any
inverse reinforcement learning method without further inspection. The second
inclusion criterion refers to the publishing venues. In this survey, we only
use work published on a peer-review venue, with an exception for all pre-
prints published in $2023$, given the slow nature of machine learning venues’
double-blind review. However, this exception matches a single work (’Stable
Motion Primitives via Imitation and Contrastive Learning’ [15]). Although
these rules set a general framework for selecting relevant work for our
literature review, they are just guidelines that should not be seen as strict
rules.

We use abstract information to remove all work that uses imitation learning
outside agent-related research. An example of such work are meta-learning
publications, which use imitation learning techniques applied to other
domains, such as neural networks training [36]. Ultimately, we selected $65$
publications from a variety of venues, over half of which are from $2019$ or
more recent. From these, $30$ are not present in Zheng et al. [35] publication
due to the fast-growing nature of the imitation learning field. For
completeness, we look and include other surveys (which others work [34] lack)
since imitation learning research takes inspiration from other fields, such as
reinforcement learning [37], multi-agent [38, 39] and transfer learning [40,
41].

To help bring some uniformity to our review, we deviate slightly in approach
from past surveys and create a unifying taxonomy under which different
categories of environments and metrics can be understood and compared. Hence,
our survey looks at the field from a different perspective than the
traditional model-free and model-based view employed by Torabi et al.’s survey
[34]. We attribute the lack of standardisation in the imitation learning field
to the lack of a taxonomy for imitation learning environments and metrics,
which is the main reason for us to deviate from past surveys. In Hussein et
al. survey on imitation learning [1], the authors mainly focused on
methodologies, given the need for classifying new approaches from the
literature. In it, Hussein et al. set a background for future work, from which
we take inspiration, and introduce relevant work following their learning
approach, such as apprenticeship learning, behavioural cloning, and inverse
reinforcement learning. We follow a similar approach for our classification of
imitation learning methods, in which we present a taxonomy based on the most
predominant learning techniques during training, such as inverse dynamics
models and adversarial learning. Given its simplistic nature for classifying
imitation learning methods, we deviate from Zheng et al.’s survey. They
classify each approach as either a behavioural cloning, inverse reinforcement
learning or adversarial imitation learning approach or as high-level (which
requires higher-level cognitive functions) or low-level (concrete operations,
such as moving an object) tasks. Moreover, we maintain that our classification
system not only acknowledges their taxonomy but also enhances it by providing
a more detailed categorisation of emerging trends within imitation learning
methods.

Finally, we present a novel taxonomy for imitation learning environments based
on the role of the environment in the learning process and a taxonomy for
imitation learning metrics based on the meaning they convey. Although Hussein
et al. and Zheng et al. discuss different imitation learning environments,
they do not classify them and only group them by domain, which provides no
insight into the environment’s role in the learning process. Similarly to
environments, Hussein et al. present different metrics for evaluating
imitation learning methods. However, their work does not classify them or
present in-depth metrics analysis.

##  III Background

It is part of human nature to learn. Learning comes in many forms: observing,
experiencing, reading and sharing. One of these learning approaches comes
intuitively to humans: learning by observing, which is a skill humans develop
quite early (as children and newborns) to learn how to walk and talk [42].

The imitation learning paradigm refers to an agent’s acquisition of skills or
behaviours by observing a source performing a given task [1]. In the imitation
learning context, we follow Russell and Norvig’s definition [43, Chapter 3.]
of an agent where an agent is an entity that autonomously interacts within an
environment to achieve a goal. We refer to these sources of behaviour as
teachers. Unlike other agent-based learning methods, such as reinforcement
learning, where an agent learns through interactions with the environment,
imitation learning uses this teacher’s behaviours to guide the learning
process. Most commonly, imitation learning refers to the teacher as an
‘expert’, but we deviate from this nomenclature, which we further discuss the
reasoning behind in Section VII. In this setting, a teacher can be a human or
a computational agent. We generally define a teacher as an agent who provides
information on how to act in an environment. This information might carry
expert knowledge, but it is not a requirement.

More formally, imitation learning problems use Markov Decision Processes to
model the environment. This formulation represents a state-action network,
where the transition of states is mapped from its state-actions and,
therefore, is suitable for imitation learning. A more formal definition of the
Markov decision process is:

######  Definition 1.

Markov Decision Processes are represented by a five-tuple
${M}=\langle\mathbf{S},\mathbf{A},T,r,\gamma\rangle$, in which: $\mathbf{S}$
is the state space, $\mathbf{A}$ is the action space, $T$ is the transition
dynamics $T:\mathbf{S}\times\mathbf{A}\rightarrow\mathbf{S}$, $r$ is the
immediate reward function $r:\mathbf{S}\times\mathbf{A}\rightarrow\mathbb{R}$,
and $\gamma$ is the discount factor such that $\gamma\in[0,1)$.

The state space $\mathbf{S}$ is represented by vectors that contain
information regarding the environment’s current state. For the action space
$\mathbf{A}$, we can also encounter two forms to represent them: vector-based
and single values. Vector-based actions represent either a sequence of
decisions that the agent should enact in order, such as a sequence of buttons
to press, or a list of actions to act simultaneously, different joints from a
robot, for example. The transition dynamics of a Markov Decision Process
dictate how a transition in an environment will occur, while the discount
factor relates to the reward regarding the time domain. A discount factor of
$1$ indicates that future rewards are considered equally important as
immediate rewards, while a discount factor close to $0$ implies that only
immediate rewards are prioritised. Moreover, a critical property of these
networks is that the transition dynamics usually only depend on the previous
state and action, regardless of earlier states. Nevertheless, in the context
of imitation learning, although the network might carry information regarding
reward and discount factors, we consider that this information is inaccessible
to the agent, and the learning process does not depend on it. Figure 1
displays a common implementation of the Markovian problem where the
environment provides a state $s$ to the agent, which predicts the most likely
action $a$. The environment applies the state transition function $T$ on the
state-action pair to generate the new state $s^{\prime}$, and computes the
immediate reward using the discount factor $\gamma$ and immediate reward
function $r$.

[width=]figures/03-background/mdp-diagram

Figure 1: The Markovian Decision Process.

Following a Markov Decision Process definition, we can generally state that
the goal of imitation learning is for an agent to use $\mathbf{S}$ and
$\mathbf{A}$ from a teacher to learn a mapping function called policy. A
teacher also has a known or unknown policy that the agent is trying to
approximate, and we define this function as:

######  Definition 2.

A policy is a function ${\pi}$ that maps states to actions
${\pi}:=\mathbf{S}\rightarrow\mathbf{A}$. The policy has inner parameters
$\theta$, which represent the internal variables or weights within the policy
that are adjusted during the learning process. The set of all possible
internal variables is defined as
$\Pi=\left\\{\pi_{1},\pi_{2},\dots,\pi_{z}\right\\}$.

Strictly defined, a policy maps states to probabilities of selecting each
possible action $P(a\mid s)$ conditioned by the available teacher’s data.
Having the MDP definition allows us to define agents and teachers more
formally.

######  Definition 3.

An agent ${\pi_{\theta}}$ is one possible instantiation of ${\Pi}$. It selects
an action $a\in\mathbf{A}$ provided an environment state $s\in\mathbf{S}$
according to its fixed learned parameters $\theta$.

######  Definition 4.

A teacher ${\pi_{\psi}}$ is a special case of an agent, with respect to its
parameters $\psi$, whose behaviour imitation learning wants to approximate.

Applying an action $a$ to the environment results in a new state
representation $s^{\prime}$, which an agent can use to select a new action,
forming a sequence of states and actions. All possible sequences from a given
teacher refer to a smaller of all possible environment states $\mathbf{S}$ and
actions $\mathbf{A}$, which we denote as $S_{\pi_{\psi}}$ and
$A_{\pi_{\psi}}$, respectively. These states and actions are subsets because
solutions from ${\pi_{\psi}}$ may not require it to visit a specific state or
predict an action. Thus, $S_{\pi_{\psi}}\subseteq\mathbf{S}$ and
$A_{\pi_{\psi}}\subseteq\mathbf{A}$. For simplicity, in this work, we omit the
policy information when referring to ${\pi_{\psi}}$, and explicitly inform the
policy otherwise.

On the other hand, we still lack a definition for the sequential data
collected from teachers to learn the policy function. These sequences of state
and action interactions from the teacher in the environment are called
trajectories and are an essential part of the imitation learning processes.
The data within each trajectory is the only information an imitation learning
agent can access when learning how to perform a task in an environment.
Therefore, if too few samples are collected, the agent might encounter states
with no prior information on how to act, or if the data is corrupted, it may
act differently than its teacher. In this report, we formally define
trajectories as:

######  Definition 5.

A trajectory is an ordered list of state-action pairs
$\tau=\left[(s_{1},a_{1}),(s_{2},a_{2}),\dots,(s_{n},a_{n})\right]$, where
$n>1$ and $s_{i+1}=T(s_{i},a_{i})$, for all $i>1$.

A set of trajectories is composed of multiple trajectories, such that
$\mathcal{T}=\left\langle\tau_{1},\tau_{2},\dots,\tau_{k}\right\rangle$, where
$k\geqslant 1$. Given a set of trajectories $\mathcal{T}$, an agent
${\pi_{\theta}}$ and a teacher ${\pi_{\psi}}$, we can mathematically express
the learning process for imitation learning as minimising the error for a
given loss function ${\ell}$ between ${\pi_{\psi}}$ and ${\pi_{\theta}}$, as
follows:

$\operatorname*{arg\,min}_{\theta}\sum_{\tau\in\mathcal{T}}\sum_{s\in\tau}{\ell}({\pi_{\psi}}(s),{\pi_{\theta}}(s)),$
(1)

where $s$ is a state sampled from the teacher’s trajectory $\tau$ in the set
of trajectories $\mathcal{T}$. It is important to note that early iterations
of imitation learning used supervised learning losses to learn a policy.
However, novel work [6, 3] have shown that imitation learning can be
formulated as self-supervised and adversarial learning approaches (and further
discussed in Section IV).

Equation 1 assumes we have access to ${\pi_{\psi}}$. However, it is sensible
to assume that imitation learning could not depend on direct access to a
teacher’s policy. This level of access requires knowledge of the teacher’s
internal state, which cannot be done when the teacher is human. Therefore,
imitation learning may use demonstrations, also known as imitation learning
from demonstrations, to learn a policy, which we define below.

######  Definition 6.

A demonstration is a state-action pair taken from a teacher’s trajectory, such
that $(s,a)\sim\tau\in\mathcal{T}$.

A list of all sampled demonstrations from one or more teachers is denoted as
$D=\left[(s_{1},a_{1}),\dots,(s_{n},a_{n})\right]$. We denote the subset of
all states for a demonstration is $S=\left\\{s\mid(s,a)\in D\text{ for some
}a\right\\}$, while the subset of all actions is $A=\left\\{a\mid(s,a)\in
D\text{ for some }s\right\\}$. The probability of taking an action $a$ given
$s$ in a list of demonstration $D$ is:

$P_{D}(s,a)=\left\\{\begin{matrix}P(s,a)&\text{if }(s,a)\in D\\\
0&\text{otherwise}\end{matrix}\right.,\text{ where }$ (2)

$P(s,a)=\frac{\left|\left\\{(s_{i},a_{i})\in D\mid
s_{i}=s,a_{i}=a\right\\}\right|}{\left|\left\\{(s_{i},a)\in D\mid
s_{i}=s,\text{for some~{}}a\right\\}\right|}.$ (3)

Unlike trajectories, demonstration lists are not necessarily ordered lists;
hence, $s_{i+1}$ may not be the direct outcome from $T(s_{i},a_{i})$. As
trajectories, demonstrations are vital for imitation learning and highly
correlate with the agent’s ability to generalise in unseen scenarios. For
example, when solving different puzzles, the agent must adapt its strategy to
solve one puzzle where some pieces do not share the same shapes from previous
demonstrations. Consequently, when creating imitation learning from
demonstrations, researchers must consider whether the data is representative
enough of the environment, which we further discuss in Section VII.

Although imitation learning shies away from any direct signal from an
environment by using demonstration, experiences such as an agent’s
trajectories can be used as well. The difference between the two instances of
data is that demonstrations provide the teacher’s action to a given state,
allowing a supervised learning approach, and experiences show the performed
action, which may not be close to the source behaviour but also provides the
reward (or cost) of performing that action given the current state. However,
using the reward function to optimise its policy further goes against the
imitation learning paradigm. Hence, some work avoid using the returned reward
from the environment and use experiences as state-action pairs. More formally,
we define experiences as follows.

######  Definition 7.

An experience is a special case of a demonstration, in the sense that its
tuples are from the learned agent’s trajectories:
$(s,a)\sim\tau\in\mathcal{T}_{\pi_{\theta}}$ instead of the teacher’s.

All definitions so far help us understand early approaches to imitation
learning. Conversely, these definitions do not align with how humans acquire
knowledge through observation. Thus, imitation learning from observation
further restricts imitation learning by reducing demonstration information
only to the teacher’s states. By learning without knowing which actions were
performed, imitation learning from observation tries to employ the same
learning approach humans do by figuring out the action performed from the
observed changes between two successive states of the environment. In this
context, imitation learning from observation represents its examples with less
information.

######  Definition 8.

An observation is a state pair $(s_{i},s_{i+1})$ from a teacher.

If we have access to the teacher’s trajectory $\tau$ and the transition
function $T$, we can retrieve any observation with $(s_{i},T(s_{i},a_{i}))$,
such that $(s_{i},a_{i})\in\tau$ and $i\leqslant n$. On the other hand, if we
lack these pieces of information, we can only retrieve observations
$(s_{i},s_{i+1})$ for all sequences of states where $i<n$. A list of
observations is a not necessarily ordered list of all sampled observations
from ${\pi_{\psi}}$: $O=\left[(s_{1},s_{2}),\dots,(s_{n-1},s_{n})\right]$,
where $n>1$. Like demonstrations, the subset of states for observations is as
follows: $\left\\{s\mid(s_{i},s_{i+1})\in O\right\\}$. The probability for
observation pairs follows Equation 2 (and Equation 3, consequently) but uses
the observation’s information rather than the demonstration’s. Assuming access
only to the state information helps because datasets that explicitly give the
actions performed between state changes (also known as labelled datasets) are
uncommon in the real-world111This is commonly referred to as in the wild and
are costly to create. By lifting this restriction, imitation learning agents
do not need to create a dataset that involves recording humans playing or
training another agent to act as a teacher. However, by removing the action
information, two problems may arise when dealing with more complex tasks:  (i)
Markovian problems most often are non-injective, which means that the
transition function $T$ may map two different state-action pairs to the same
state, and smaller datasets will not cover the range of transitions necessary
for the policy to learn properly; or  (ii) computing whether the collected
data covers a significant part of the Markovian network might be impossible.
Therefore, focusing on imitation learning from observation is vital to create
more efficient imitation learning agents and removing the requirement for vast
datasets. Figure 2 shows a general approach to imitation learning, where the
agent learns from demonstrations or observations provided by a teacher and
uses the environment to collect new experiences.

[width=]figures/03-background/il-diagram

Figure 2: A general approach to imitation learning.

So far, we have described all possible inputs for imitation learning agents as
time-sensitive, but our agent formulation for now remains time-independent.
Thus, we have to consider time as a possible input for all types of data,
despite its ability to specify an instance of input and output. A policy that
maps a sequence of states to actions, such as $P(A\mid
s_{0},s_{1},\cdots,s_{t})$, is called non-stationary. These policies help map
lasting consequences between states and sequential actions, such as the agent
closing a path unintentionally and the different movements required to hit a
ball with a racquet, respectively. Therefore, non-stationary policies are more
naturally suited to learning motor trajectories [28]. Conversely, stationary
policies ignore time and predict actions solely on the present information
$P(A\mid s)$. The advantage of stationary policies is the ability to learn
tasks when their length might be too long or unknown [28], such as autonomous
driving, where the agent can drive for an undetermined time. Moreover, non-
stationary policies are difficult to adapt to unseen scenarios and changes in
the parameters of a task [44]. Given the temporal characteristics of these
policies, at one point, the trajectory can result in compounded errors as the
agent continues to perform the remainder of the actions, such as an agent
moving in a loop through a maze. The imitation learning literature focuses
heavily on stationary agents.

Lastly, Zheng et al. [35] proposes to further classify actions into two
groups:  (i) low-levelactions, which refer to atomic decisions that an agent
can perform in a given domain, typically involving simple commands such as
movement or interaction; or  (ii) high-levelactions that are decisions made by
an agent that determines the overall plan or strategy to be executed in a
given task or domain, which are often complex and involve multiple lower-level
actions or sequences of decisions.  However, most imitation learning work
focuses on low-level actions of single-value outputs, given the stationary
nature of most agents.

[width=]figures/04-methods/taxonomy

Figure 3: New proposed taxonomy for imitation learning methods.

##  IV Imitation Learning Methods

The most common form of classification for imitation learning methods is
obtained by partitioning them into model-based and model-free methods. Model-
based methods rely on learning to model the environment, such as predicting
the consequences of their actions and, afterwards, using this model in the
learning process of the policy. Conversely, model-free methods do not use a
model from the environment. Instead, they learn by trial and error, similar to
reinforcement learning approaches. Model-based methods offer a trade-off in
time and sample efficiency by leveraging the number of samples a teacher
provides, the less time it takes to train a policy. On the other hand, model-
free methods are usually more robust to unseen states since they learn by
trial and error, which gives them more access to diverse states (outside the
teacher’s dataset), resulting in more generalisation from their policies.
Lastly, imitation learning methods can also be classified as online. Online
imitation learning methods assume that we have access to the teacher’s policy.
They are less common learning methods due to the availability of teacher’s
policies not being common.

In this section, we divide all imitation learning works into dynamics model
methods (model-based – Section IV-B), adversarial methods (model-free –
Section IV-C), hybrid methods (Section IV-D), and online methods (Section
IV-E). We deviate from model-free and model-based classifications to clarify
the recent trends in imitation learning. Most surveys [1, 34, 35] consider
adversarial methods that use dynamics models adversarial by nature and model-
based, while we classify them as hybrid. We also briefly present the
behavioural cloning approach (Section IV-A), the simplest form of imitation
learning, its evolution with more novel methods and the base framework for
most dynamics models methods. In this work, we maintain the online
classification as Hussein et al. [1].

Fifty out of the $65$ publications were exclusively proposing a new method. We
expected a high percentage of work to be methodological papers since imitation
learning borrows environments and metrics from the reinforcement learning
literature. On the one hand, this trend is sensible since there were fewer
imitation learning methods in the past. On the other hand, this focus on
imitation learning methodology shows a lack of research on other subjects
pertinent to imitation learning. Nevertheless, these work that focus on
studying imitation learning methods more broadly, such as resiliency [45] and
evaluation [46, 47], have been published. The most common approach was
adversarial learning (following the taxonomy illustrated in Figure 3) with
$15$ published works, followed by $14$ dynamic, $11$ behavioural cloning, $8$
hybrid, and $2$ online methods. Although we expected to find many methods that
use dynamic models, an interesting trend is the use of hybrid approaches to
create imitation learning agents. We hypothesise that hybrid approaches and
dynamic models are also becoming more popular with more researchers from
backgrounds outside the reinforcement learning field. More so, hybrid methods
offer a better trade-off between efficiency and effectiveness, which is a
reason for their increase in popularity in recent years. Figure 4 shows the
difference between our taxonomy (colors) and other surveys taxonomies
(location) for each of the papers present in this work.

[width=]figures/04-methods/method

Figure 4:  Comparison between taxonomies. Each dot represents a method-focused
paper in this review. The colours represent the new taxonomy, while their
location represents how they would be traditionally classified. We remove the
online classification since both taxonomies are equal on that matter.

###  IV-A Behavioural Cloning

The earliest approach to imitation learning was Behavioural Cloning [2, 62],
which reduces the problem of learning to imitate a teacher into a supervised
problem. In it, the agent learns to predict the most likely action given a
state $argmaxP(a\mid s_{t})$ based on a teacher’s dataset. To be precise, the
agent uses a state-action pair of demonstrations $(s_{t},a_{t})$ at time $t$
from a proficient source to learn to act as the source based on previously
seen data. Behavioural cloning tries to approximate the agent’s trajectory
that of its teacher. A trajectory is a coherent sequence of demonstrations or
experiences from one cycle of the teacher or agent’s interaction with the
environment. Such an approach becomes costly for more complex scenarios,
requiring a large number of samples and information about the action’s effects
on the environment. For example, the number of samples used to solve tasks
involving a higher number of possible actions (e.g., continuous actions) or
intricate dynamics is approximately a hundred times higher than for classic
control tasks [5]. This problem occurs for behavioural cloning policies since
they fall short when approximating unseen states to known trajectories
(generalisation), a common problem from supervised learning approaches.
Additionally, requiring demonstrations becomes costly due to the need for
labelled pairs. Usually, recording these trajectories involves training an
agent via reinforcement learning, but some researchers record themselves
playing, and as a result, they may not be demonstrating an optimal solution to
the problem being solved. This scenario is not problematic as long as the
measure of success depends on how good the agent is at imitating the teacher,
rather than how good it is at actually solving the problem. We will come back
to this point in more detail in Section VI.

Newer approaches usually use behavioural cloning as a bootstrapping mechanism.
Bootstrapping a policy means using a machine learning approach, such as
supervised learning, to acquire knowledge from the environment or desired
behaviour before applying another learning technique to fine-tune the agent’s
performance. An example is the work from Lynch et al. [11], where the authors
use behavioural cloning to condition a policy to a set of goals and,
afterwards, use planning to create an optimal trajectory. A second possible
bootstrapping application is the work from Daftry et al. [63], in which the
researchers record themselves flying a drone, apply behavioural cloning to
learn their flying behaviour, and, as the final step, use reinforcement
learning so the agent learns to adapt to different seasons (when the images
look different). These approaches display some of the benefits of using
behavioural cloning. The agent learns more efficiently offline (without
requiring direct environment access) by applying this technique and,
afterwards, using the acquired knowledge to reduce the number of steps
required for less efficient learning approaches, such as reinforcement
learning. Additionally, behavioural cloning coupled with bootstrapping can
condition the policy to a more human-like behaviour.

Conversely, behaviour cloning remains the sole learning approach when the work
mainly focuses on aspects outside the learning process. One common approach in
these work [57, 56, 8, 64] is coupling behavioural cloning with other learning
mechanisms, such as attention [57] or domain-shift [56, 8]. We use these as
mere examples of mechanisms applied to imitation learning. Moreover, in these
cases, the behavioural cloning approach could be swapped for another imitation
learning paradigm with correct adaptations since the cost of acquiring
labelled demonstrations remains significant.

###  IV-B Dynamics Model methods

A possible approach for solving the task of mimicking a teacher without any
direct action information is by employing the use of dynamics models. These
approaches are classified as model-based imitation learning methods since they
learn a model from the environmental context/dynamics. Hence, these models
learn an approximation of $T$ from Definition 1. Developing dynamics models
involves some form of online play, which is when the agent interacts with the
environment and focuses on a specific task. Dynamic models can appear in two
forms: inverse and forward.

####  IV-B1 Inverse Dynamics Models

Inverse Dynamics Models avoid the need for labelled pairs in behavioural
cloning by encoding environmental physics and retrieving the likelihood of
each action given a state transition $P(a\mid s_{t},s_{t+1})$. Nair et al.
[10] use this form of learning to teach a robot how to manipulate a rope in
different ways. In their work, the dynamics model receives the current state
of the environment and a state from a sequence of human demonstrators as the
goal state from that sequence. With both states, the model predicts the action
responsible for the desired transition conditioned to a desired goal. This
process is applied sequentially, and the authors perform all experiments using
only matrices (image states).

Torabi et al. [5] later implement an inverse dynamics model, which they use in
vector states for control and robotic tasks and coined as Behavioural Cloning
from Observation. It uses its randomly initialised policy to learn a mapping
function without access to the teacher’s action by creating a dataset
containing labelled actions. Hence, Torabi et al.’s method does not rely on a
desired state (or goal) from its teacher. Their method uses demonstrations
$(s_{t},a_{t},s_{t+1})$ from $\pi_{\theta}$ to learn the dynamics model. This
dynamics model learns a uniform transition function (given the uniformly
distributed dataset due to the random initialisation from policy $\pi$ with
weights $\theta$) and creates self-supervised labels for the teacher’s
observations. Provided with these labelled samples, Torabi et al.’s work
trains its policy in a supervised manner using behavioural cloning. Following
Torabi et al.’s work, other methods augmented their approach to improve its
performance [55], stability [6], sample efficiency [67, 65], and to other
domains [13, 66].

Most notably, our work [6] improves Torabi et al.’s work by applying an
exploration mechanism and fine-tuning its sampling mechanism, the first
imitation learning method to use an exploration mechanism. The method assumes
that the dynamics and policy models are not always sure about the correct
action and samples from each model’s output using a softmax distribution.
Therefore, if a model has an equal distribution between two actions, it will
select each action $50\%$ of the time. Conversely, if a model’s output mainly
weighs towards an action, such as $[2.5,0.1]$, the model will pick the first
action approximately $90\%$ of the time. This mechanism has a twofold benefit:
(i) during its self-supervised part, it constantly changes its pseudo-labels,
which updates the policy weights, avoiding unwanted biases from the dynamics
model; and  (ii) during the creation of new samples, the policy creates more
diverse state transitions, which help the dynamics model learn different
transitions from the environment.

As behavioural cloning methods, inverse dynamics methods also combine their
approach with reinforced learning methods [67, 65]. Pavse et al.’s work
involves two phases. In the first phase, an inverse dynamics model is randomly
initialised and learns state transitions from a random policy, just as Torabi
et al.’s work would. In its second phase, the method alternates between
generating agent experiences (with environment rewards) and optimising the
policy with teacher demonstrations. This two-phase procedure aims to find the
optimal policy in terms of total task reward (which may outperform the
teacher) by using the expert demonstration as a guide. Our work [65] similarly
alternates reinforcement learning with imitation learning. The difference
between both works is that instead of using all experiences, the second
applies the goal-aware sampling mechanism from Gavenski et al.’s work and
limits the size of the replay buffer from its reinforced learning counterpart
to have less drastic updates when learning with experiences.

####  IV-B2 Forward Dynamics Models

Like inverse dynamics models, forward dynamics models model the environment’s
dynamics using state transitions. These forward models predict the next state,
given some conditioning. Most researchers condition these models’ predictions
with the environment’s actions to generate the next state [9, 7]. However,
others rely on temporal information, such as all using all states until the
current moment [14, 15].

Most notably, Pathak et al.’s work uses the same idea from Nair et al., where
it conditions its policy with a goal. However, instead of using current and
goal states to predict the actions, the authors use current and goal state
features coupled with the last action to predict the most likely next action
and use the current and the predicted action to generate the next state.
Pathak et al.’s policy uses a common technique to predict its information:
using all previous states until timestep $t$, which helps to create a
consistent action considering all previous states.

On the other hand, the action information might be unavailable or can be
insufficient. For example, suppose an agent walks towards an impassable wall
and hits it. Even though the agents performed an action, such as ‘move
forward’, the final transitions would show no movement. Thus, the agent should
consider that the action responsible for such a transition is ‘no action’,
sometimes absent in the environment dynamics. Edwards et al.’s work applies
this premise to forward dynamics models by using these latent actions to
condition its next state. By doing so, Edwards et al.’s method creates more
faithful transitions according to the environment dynamics. As a final step,
the method requires remapping the learned actions into environmental actions.

Forward dynamics models are less popular than inverse dynamics models, given
their nature of predicting the next state from the Markov Decision Process,
which is more challenging than predicting the action responsible for a state
transition. The agent can lower its generalisation capabilities by adding this
sequential information into a behavioural cloning technique when the
trajectory deviates from its teacher’s dataset.

###  IV-C Adversarial learning methods

Adversarial learning and inverse reinforcement learning methods share the same
task in imitation learning. Instead of trying to reproduce the teacher’s
behaviour by applying some form of supervised learning, these methods create
an artificial reward function, which conditions the agent by rewarding similar
behaviours. By creating this artificial reward function, imitation learning
approaches can use reinforcement learning optimisation techniques to learn
policies with similar behaviours from demonstrations. Most work use
adversarial learning methods to create these reward functions [72, 54, 53, 52,
51, 24, 16, 50, 49, 48]; however, some earlier work rely on other inverse
reinforcement learning techniques [12, 26, 18]. This method is model-free
since the policy freely acts in the environment and learns by trial and error
without modelling the environment dynamics. Nevertheless, not all inverse
reinforcement learning methods use online play to learn its policy.

Before applying generative models, inverse reinforcement learning in an
imitation learning setting used demonstrations to guide the agent’s learning
by applying a distance-based optimisation function to the teacher’s and
agent’s trajectories. Finn et al. [12]’s work uses inverse entropy to retrieve
an artificial reward function based on a random controller and the teacher’s
demonstrations. With the artificial reward, their method would optimise a
policy function, creating new demonstrations and allowing for further
refinement of the retrieved reward function.

Ho and Ermon proposed the usage of adversarial learning [73] and maximum
entropy inverse reinforcement learning [74]. Adversarial learning is a type of
training where a model trains to generate data according to a dataset, and a
discriminator model has to learn how to discriminate samples from the dataset
and those generated by the first model. The method uses a generative model,
the same one that forward dynamics models would use, to generate trajectories
and a discriminator model to discriminate between teachers’ and students’
trajectories. Therefore, the policy role in this setting is to ‘fool’ the
discriminator model by acting accordingly to the teacher demonstrations.
However, the method presents two problems:  (i) it assumes that the teacher’s
actions are available during training, which inherits the cost of behavioural
cloning approaches; and  (ii) it is susceptible to local minima during its
optimisation process, which requires prolonged environmental interactions.
Nevertheless, Ho and Ermon is more sample-efficient than behavioural cloning,
which decreases the severity of the first point. Ho and Ermon’s work [4] is
the most known work and perhaps one of the most influential, with several
methods following the same setting [52, 49, 24, 55]. Most notably, Li et al.’s
work [52] assumes that a dataset consists of various behaviours from one or
more teachers. So, instead of creating a policy that tries to predict the most
likely action solely on the state alone, it also conditions the agent with a
teacher’s parameter. Additionally, Peng et al.’s work changes how the
discriminator discriminates between teacher and student. In it, the authors
constrain the amount of information given to the discriminator by applying the
output of the generative model into an encoder. Therefore, the discriminator
does not use the state to discriminate but an encoded value. Peng et al. show
that applying this bottleneck helps the policy to generalise better and
achieve better results than other adversarial methods that rely on state
information.

Torabi et al.’s work [49] applies a similar strategy to Ho and Ermon’s work to
apply imitation learning from observation. It assumes that adversarial methods
follow a convex conjugate optimisation problem, where the generative model and
policy function learn to approximate state transitions alone. To approximate
teacher and student, a discriminator has to discriminate over the agent and
teacher trajectories observations. By learning to differentiate between
origins, the discriminator retrieves an artificial reward function, which the
method uses to optimise the policy. By applying these changes, Torabi et al.’s
work improves adversarial methods performance in more dynamic environments and
reduces the cost of learning new policies. Torabi et al.’s work has also seen
further improvement by Sun et al.’s work [16]. Sun et al. use an adversarial
learning approach to create a policy that minimises the policy with a no-
regret function bounded to one. Therefore, if the optimisation function is
zero, the policy acts as the teacher; however, if the artificial reward
function returns one, the policy is as far as possible from its teachers. As
with other works, acquiring new demonstrations to further policy improvement
is done via online play, which becomes costly if many interactions are needed.
Additionally, Torabi et al. further explore his methods capabilities by
applying it to vision tasks [54] by using convolutional neural networks as the
discriminator model. Since vectorised information is not typical for
applications, optimising a policy using only visual information from a
convolutional neural network while holding the vectorised representation as
the input for its policy allows for more optimal solutions.

Although adversarial learning methods became the standard for model-free
approaches, inverse reinforcement learning techniques have also demonstrated
remarkable advancements recently. Chang et al.’s work [48] discards the usage
of discriminator models to match the policy’s trajectories to the teacher’s
using a distance metric. However, using this approach, the method can enter a
seeking behaviour state, producing low probabilities for observations from the
teacher data. Therefore, Chang et al. implement a noise-conditioned
normalising flow, which adds noise to observations conditioned to the state
during training and nothing during evaluations. Doing so reduces the chance of
low probabilities and biased predictions from the policy. In multi-agent
systems, inverse reinforcement learning methods have also achieved some
success. In Song et al.’s work [53], the method is divided into centralised
(or cooperative), decentralised, and zero-sum. In centralised and
decentralised approaches, the policy operates with the environment to match
the teacher reward by using adversarial techniques. On the other hand, in the
last approach, the policy interacts directly with the discriminator to match
the reward from the teacher. Conditioning also appears in multi-agent
settings. For example, Shih et al. proposes to create a policy that adapts its
strategy depending on other agents [18]. Therefore, the method has to
condition itself by classifying which strategy from the teachers’ dataset is
being used.

Finally, adversarial learning approaches have also appeared in different
settings and tasks. For example, Shang and Ryoo assumes that the agent only
has access to third-person views from the teacher (a more human-like
experience of observing others) [19]. Thus, outside of learning the teacher’s
behaviour, the agent must also learn to adapt the data from its teacher to
suit its first-person view when enacting the same task. In Yin et al.’s work
[50], the method has to predict a sequence of actions from a behavioural
cloning and adversarial agent. After both policies predict all actions, a
planning model forms a plan of which actions should be used throughout the
trajectory. Finally, in the domain of self-driving vehicles, Hu et al.’s work
[51] experiments with sharing weights with the policy and generative model,
which the method uses to optimise the policy using a reinforcement learning
technique.

###  IV-D Hybrid approaches

Recent work in imitation learning has focused on taking inspiration from
adversarial and dynamics methods [21, 3, 22, 68, 55, 17, 71, 70, 69]. By
combining dynamics models and adversarial learning, these methods create more
robust policies since they have a temporal signal that helps minimise teacher
and student differences and some intrinsic knowledge about the environment
dynamics. Although they use discriminators to achieve their task, these
methods have significantly improved imitation learning efficiency, with some
requiring a single trajectory [21].

Towards sample efficiency and the issue of saddle points, Wang et al. [70]
propose the use of variational autoencoder to add some diversity with
relatively fewer demonstrations and achieve one-shot learning to new
trajectories. They apply a temporal neural network to create a non-stationary
policy. It is worth noting that even though the generative model relies on a
forward dynamics model, the agent’s optimisation does not rely on it,
increasing efficiency during optimisation.

In learning from demonstration hybrid methods, Baram et al. develop a method
that optimises the forward dynamics model [71]. Instead of concatenating state
and action into a neural network to predict the next state, the method uses a
recurrent neural network (temporal) to encode the state and another non-
temporal encoder and perform an element-wise multiplication to feed as the
input of a third neural network. By doing so, the authors achieve teacher
behaviours earlier during training and reduce the number of teacher samples.
In the same context, hybrid methods have also seen use in partial
demonstrations from teachers [22]. In their work, the demonstrator is a single
teacher’s life in Atari environments (less than a whole episode – three
lives), and Yu et al. use a variational autoencoder to encode a forward
dynamics model and a discriminator to learn an artificial reward function.
Although learning from demonstration work is not as common as its observation
counterpart, these methods tackle problems significant for imitation learning
agents. Removing the complexity of learning policies without action
information allows researchers to tackle other, more complex problems.

On the other hand, hybrid methods in the context of learning from observation
are more prevalent in recent research. These hybrid mehtods highly focus on
efficiency [21, 3, 68, 55] and eliminating the seeking behaviour [21, 17]. Zhu
et al.’s work [21] solves the issue of imitation learning agents adopting a
seeking mode by using a discriminator to swap between a covering and seeking
mode when the agent heavily diverges from the teacher. It does so by only
using ten teacher’s trajectories for each environment, but Zhu et al. assume
that no more than a single state-action pair can result in the same next
state, which is not valid in most environments where there is momentum for
example. Most notably, Kidambi et al.’s work uses a forward dynamics model
with adversarial learning and couples it with an exploration mechanism [3].
Kidambi et al. also achieves teacher’s results with ten trajectories. However,
their method assumes that all episodes initiate in the same state, which leads
the method to fail to generalise as state sizes grow. Moreover, Kidambi et
al.’s method exploration mechanism comes with the cost of using an ensemble
method (more than one model during prediction), which later work by Monteiro
et al.’s solves. In addition to solving efficiency problems, Liu et al.’s work
[17], the authors experiment with encoders across different contexts, creating
dynamics models to translate between teacher and student states and apply a
discriminator model to approximate both behaviours. By doing so, Liu et al.’s
work eliminates the need for fine-tuning a model every time the context of an
agent changes and only requires training a new encoder, allowing for
translating teacher behaviours into new contexts.

More recently, Gangwani et al.’s work assumes that its policy has to learn
from a teacher’s trajectory that diverges from the agent. Different from Liu
et al.’s work, the teachers and agents are in the same context, but the
transition function for both is different. In other words, the teacher’s
observations should guide the policy towards a goal, similar to other methods
that condition their policies as a goal state. In their work, the policy
learns a forward dynamics model to output the most likely next state from the
agent’s transition function perspective that aligns with the teacher’s
trajectories. Gangwani et al.’s work [69] shows that it performs better in
these settings than other methods. However, the results are far from the
teacher, which shows that such a setting needs additional research.

We note that hybrid methods have become more popular. Although adversarial
learning methods are less sample efficient, they help model-based methods
generalise and learn more human-like behaviour. Methods such as Kidambi et
al.’s and Zhu et al.’s work achieve teacher results with ten or fewer
trajectories, but some recent studies show that there are still other
mechanisms to research. For example, exploration mechanisms and cross-domain
contexts have been uncommon in imitation learning approaches. By exploring,
agents can acquire knowledge unattainable before due to the need for more
variety in teacher samples. When applying cross-domain contexts, imitation
learning agents can learn from a third-person view or different settings for
the agent, such as fewer joints in a robot.

###  IV-E Online Imitation Learning

A different approach to learning from a teacher is online imitation learning.
In this approach, the agent must also learn the teacher’s behaviours. However,
instead of using demonstrations from a pre-recorded dataset, online imitation
learning involves access to the teacher’s information in real-time. Online
learning can be detrimental when designing a reward function that is
challenging to define or a policy function that is difficult to model. Given
the requirement of direct access to a teacher, such as another agent, this
approach is less commonly researched than the other areas. Nevertheless,
online imitation learning is more suitable when real-time interaction with a
teacher is possible or when the agent needs to adapt to changing conditions.

Ross et al. implements an iterative online imitation learning algorithm that
uses demonstrations from online play and teachers to learn a behaviour [28].
It uses a Follow-The-Leader technique to optimise the policy behaviour based
on the demonstrations it retrieves during the first iterations and from the
teacher. Follow-The-Leader optimisation is one of the possible online learning
algorithms to optimise agents. It tries to minimise a cumulative loss function
or maximise the agent’s reward sequentially. Since reward values are not
available to the agent in imitation learning approaches, Follow-The-Leader
tries to minimise the difference between teacher and student behaviour.
Lavington et al. further improved the Follow-The-Leader strategy by applying a
regularisation term, which they coined as Follow-The-Regularised-Leader [75].
In it, Lavington et al. consider that the teacher’s demonstrations might have
noise, and the Follow-The-Leader approach can lead to oscillations resulting
in non-optimal behaviour. The researchers show that a policy achieves better
results after applying regularisation to the optimisation function.

###  IV-F Learning Methods Discussion

Upon analysing the literature from this section, we notice a change in the
development of new learning methods, which use hybrid mechanisms. Instead of
relying solely on dynamics models or adversarial approaches to learn the
teacher’s behaviour, these methods mix different aspects from both sides to
achieve better results. Furthermore, recent approaches focus more on imitation
learning from observation and efficiency problems. In Section VII, we further
expand on challenges researchers may focus on rather than efficiency and also
reflect on different aspects learning methods may not account for.

This literature review shows a lack of imitation learning methods applied to
online learning and multi-agent systems. We believe this shortage of work for
online learning comes from the premise of having access to the teacher, which
limits the number of applications and the complexity of learning from a policy
in an environment where more than one agent impacts the transition function.
Like online learning, multi-agent systems use highly dynamic environments to
learn their policies. In these environments, the overall combination of all
possible states can be virtually infinite, which leads to imitation learning
agents requiring a lot of data or drastic assumptions on how to model the
environment (that might not hold true in diverse cases). Therefore, we argue
that imitation learning methods tailored specifically to the challenges posed
by these applications need to be researched and developed.

##  V Imitation Learning Environments

Environments are a vital part of the imitation learning process. They simulate
various tasks and domains and allow the agent to interact with the simulation
as they would in a real-world application. Creating an environment most often
involves creating the environment’s rules, such as physics for each object,
and implementing rules to guide the agent, for example, how agents should
interact. Nevertheless, they allow different work to evaluate their results
using the same dynamics, creating a more fair comparison since all methods
receive the same demonstrations and solve the same problem. In the literature
review, we notice a lack of consensus on which environments imitation learning
literature should use.

In the literature on agents, there are different descriptions of what
environments consist of. We follow Russell and Norvig’s definition, which
describes environments in four different aspects [76, Chapter. 2]. The first
aspect entails the accessibility of the states from the environment. Most
commonly, environments use fully observable states, meaning the agent can
obtain complete, accurate, up-to-date information. However, most real-world
environments are inaccessible in this sense. The second aspect is certainty.
Deterministic environments are those in which any action has a single
guaranteed effect. In these environments, there is no uncertainty about the
state resulting from an action. We call environments that are not
deterministic non-deterministic. The third aspect describes the environment
dynamics. Static environments are those that are safe to assume will remain
unchanged except by the performance of an action by an agent. In contrast, a
dynamic environment has other processes operating, changing in ways beyond the
agent’s control, such as the physical world, which is a highly dynamic
environment. Lastly, the fourth aspect entails how things are represented.
Discrete environments are those with fixed, natural numbers and a finite set
of actions, while continuous are those represented by rational numbers, hence
an infinite set of actions with upper and lower limits.

Environments can also be domain-specific, such as autonomous driving, or more
general, such as robotic simulations, which use 2D and 3D simulations to
evaluate agents in various similar tasks. Most work use the application to
differentiate each environment, such as assistive and humanoid robotics [1].
However, such a classification is shallow since it only focuses on a specific
set of problems per environment, not what the environment tests the agent on.
Therefore, we propose a threefold classification for these environments:  (i)
validation, which helps researchers to validate ideas without added
complexity;  (ii) precision, which helps to test agents that require less
temporal modeling, but require precision; and  (iii) sequential, which tests
an agent’s actions and their temporal consequences.  Figure 5 presents the
proposed taxonomy for the environments.

[width=]figures/05-environments/taxonomy

Figure 5: Imitation learning environments taxonomy.

It is important to note that other forms of classifying environments exist,
such as how the states are represented (images or vectors), actions (discrete
or continuous), and whether a task is a maintenance task (a set of states are
unreachable to the agent) or an achievement task (the agent should reach a set
of states). However, our classification focuses more on the agent’s intent,
such as ‘to be as precise as possible’, than on the task or domain itself.
Additionally, this taxonomy also contains domain-specific and task-specific
classifications after the broader term. For each environment, we present
whether the environment presents domain-specific or task-specific problems,
which should help researchers decide if an environment is desirable for their
experiments.

(a) CartPole [77]

(b) Half-Cheetah [78]

(c) Humanoid [78]

(d) Breakout [79]

Figure 6: Examples of validation environments.

###  V-A Validation environments

Validation environments are more straightforward simulations in general. These
environments help to validate whether an agent can learn to perform a task
from a teacher with no extra complexity involved. Usually, the agent is only
required to learn the task, and the environment does not diverge much when
initialising the agent. These environments are more commonly split into
discrete and continuous action environments.

The most common discrete environment in the imitation learning literature is
the CartPole [77] (Figure 6(a)). This environment is a perfect example of a
validation environment. It consists of a pole attached by an un-actuated joint
to a cart, which moves along a frictionless track. The pole is placed upright
on the cart, and the goal is to balance the pole by applying forces to the
left and right of the cart. Thus, CartPole is a maintenance task-specific
environment. Each episode starts with uniformly random values between $-0.05$
and $0.05$ for the cart position, cart velocity, pole angle and pole angular
velocity. Therefore, the variety in each episode is small, and since there is
no differentiation between training and validation, such as different
mechanics, the agent is not required to generalise. Achieving an optimal
result within the CartPole environment does not demonstrate that the imitation
learning agent can learn and perform complex tasks. However, it is helpful as
an initial validation. Alongside the CartPole, the most common environments
for discrete actions are the MountainCar [80] environment and Atari games
[79]. In the MountainCar environment, a car is randomly placed at the bottom
of a valley shaped like a sine wave. The only available actions are
accelerating the car in either direction or not accelerating. The MountainCar
problem deals with momentum, making it harder for simple or random agents to
solve since it carries some sequential impact. If an agent builds momentum and
selects to use force opposite to its direction, the agent will lower its speed
and fail to reach the goal. On the other hand, Atari games are environments
that consist of visual state representations. They are commonly used in
imitation learning tasks because they allow for visual qualitative assessment
of the agent’s behaviour. Moreover, Atari games were a long benchmark for
comparing how agents perform in different tasks and allow for a direct
comparison with humans using the video game score [81].

The most common continuous environments are simulations from Multi-Joint
dynamics with Contact (MuJoCo) [82] (Figures 6(b) and 6(c)). In these
environments, the agent is a robotic simulation of different joints and body
parts configurations, such as a cheetah or a human. The agent aims to move
right by applying force to these joints and body parts. Similar to the
CartPole environment, these environments are simplistic in the sense of not
posing new challenges to the agent. The agent has no barrier to overcome or
other dynamics it has to adapt to. However, differently from the CartPole
environment, these environments still pose a challenge in the sense that the
agent has to learn to control its body joints together to achieve the goal.

Most work use a combination of validation and another environment type to
experiment with the limitations of imitation learning agents. However, some
work [20, 5, 49, 68, 50, 22, 16] use only validation environments, which can
be troublesome. By only using these environments, the agent can learn to
overimitate its teacher without comprehending the underlying task.
Additionally, considering the initialisation is not diverse, the agent can
force a teacher’s trajectory into the environment, causing the same problem as
reward-based metrics. In other words, during an agent’s first interactions
with an environment, the agent can predict actions that will take it to the
same states present in the teacher’s dataset, and afterwards, the agent will
only have to perform state matching to achieve optimal results. Other
frameworks for simulation, such as Google’s suite [83], promise a diverse set
of initialisations, but further testing shows that these simulations are
vulnerable to the same seeking behaviour. Table A shows all validation
environments with Russell and Norvig’s environment attributes information.

###  V-B Precision environments

(a) MuJoCo [82]

(b) MAV [63]

(c) Panda-gym [84]

(d) Robo-gym [85]

Figure 7: Examples of precision environments.

Precision environments are generally independent of long action consequences
and require precision, as the name suggests, from the agent. Russell et al.
classify this autonomy from time as episodic environments [86, Chapter. 2]. It
is important to note that an episode is a sequential set of states from the
agent in imitation learning literature, while Russell et al. classify an
episode as a single step from the agent (an experience). In episodic
environments, the task divides the agent’s experience into atomic experiences.
The agent receives a state in each experience and then performs a single
action. Additionally, the following experience is independent of the actions
taken in the previous one. However, this characteristic depends on how the
environment computes these experiences. For example, an experience usually
comprises four frames in Atari games, while it is a single frame in Mario
games. Therefore, being episodic has some space for interpretation.

Precision environments usually involve robotics tasks where the agent requires
precision, and time for each decision is not relevant. The most common type of
precision environment is the robotic arm. This simulation works for various
tasks, such as opening doors [68], pushing pins [61], and arranging blocks
[57]. The most common setting for these tasks is a robotic arm with an
unmovable base and some range of movement on other joints (Figure 7(c)). In
these tasks, researchers aim for agents that can overimitate a teacher in the
same setting and adapt slightly in newer settings, such as posing the blocks
in different positions [57] or moving with different body configurations [24].

Robot walking and navigation is also common tasks for precision environments.
For this task, the environment simulates a robot’s body, such as MuJoCo, but
to a higher degree of detail. The idea is to test whether the agent can
navigate different obstacles and spaces. Furthermore, the agent can have
different body configurations, from vacuums to bipedal designs. For example,
micro air vehicles, or MAVs, is a common domain for these precision
environments (Figure 7(b)), where the agent has to fly through obstacles
without getting hit. Some flying environments consider different seasons to
understand how agents adapt to variations in images [63].

Usually, work that use precision environments experiment with physical robots
as the final evaluation step [15, 14, 13, 9, 17, 8, 63, 12]. However, this
approach is expensive, given the requirement of a robotic arm or different
robot settings. Therefore, some authors [67] rely on simulations to test
different settings, such as blocks world [57], where the robotic arm has to
arrange blocks in order, and the authors split some initial settings and goals
only for evaluation, making sure the agent is not only overimitating and has
learned the underlying task correctly. Table II shows all precision
environments with some information regarding the environments’ attributes.

###  V-C Sequential environments

Sequential environments could have long-lasting consequences, given the
agent’s prior actions. They are the opposite of precision environments.
However, it is essential to observe that not all actions must impact future
experiences. These environments are harder to solve for imitation learning
agents because they require the agent to plan for future actions, which might
be learning an abstraction from the task, and usually rely on non-stationary
policies. For example, in Lunar Lander, an environment where a rocket has to
use its engines to land between two flags, the agent have to learn how to map
a trajectory depending on where the agent spawns. If we suppose the agent
overimitates teachers’ trajectories instead of adapting actions to consider
the difference between previous demonstrations and the current setting, the
agent will not learn how to land the rocket where it is suppose to, but
instead, it will learn how to land the rocket where the teacher landed it.

Sequential environments are most commonly used for autonomous driving domains,
where the agent must learn how to drive and navigate different scenarios. In
these environments, different aspects can be experimented with. For example,
while CARLA [87] has a more realistic city traffic simulation, TORCS [88]
proposes a scenario for open racing car simulation, where the agent competes
against other cars (Figures 8(a) and 8(b), respectively). Simulation of Urban
Mobility, or SUMO [89], allows for recreating various urban scenarios, but it
does not have CARLA’s realistic LIDAR and RADAR sensors. In contrast,
SuperTuxKart offers a driving simulation with a more cartoonish style where
the agent has different powers to use against other computer-controlled agents
(Figure 8(c)). Therefore, each environment can have drastic differences in
information and settings, and researchers must ponder which scenario is more
relevant to their research and correctly evaluates their agents.

On the other hand, if the domain of autonomous driving is not required, the
usual sequential environment is video games. Video games allow for easy
abstraction of tasks, such as navigation [90], or just for modelling temporal
consequences. In the latter, these environments allow for the manipulation of
maps to test specific behaviours from the agent. For example, VizDoom [91]
(Figure 8(f)) allows developing agents to play DOOM, a $1993$ video game where
you play a marine fighting through hordes of monsters in first person, using
visual information. The environment has visual editors that allow for custom
scenarios and to control variables for each level, allowing for a more
consistent benchmark between researchers. Another advantage of these
environments is that they provide diverse contexts for both learning and
assessment, such as changing the number of enemies, weapons and level design
between training and evaluation. CoinRun [92] (Figure 8(g)), where the agent
learns how to navigate through a level from a game where the goal is to pick
one or more coins, poses a similar setting. During training, the level has no
hindrances to the agent, but during the evaluation, the level will have
monsters and lava, which can kill the agent. Consequently, sequential
environments are great to evaluate not only long lasting consequences but also
the agent’s ability to generalise to different settings.

Nevertheless, this diversity of settings for sequential environments can cause
problems when comparing different work. For example, in Duan et al.’s work use
a maze to evaluate whether their agent learned the underlying task [57].
However, in Gavenski et al.’s the authors use a different maze to evaluate
their agent [6]. In the first, the authors use a more simplistic structure
(following a ‘C’ and ‘S’ shape). The latter, uses a more complex structure,
with more turns and dead ends with different sizes ($3\times 3$, $5\times 5$,
and $10\times 10$). Thus, making the comparison between these two work harder.

Combining sequential and validation environments is ideal when creating
imitation learning agents. These long-lasting consequences usually are the
downfall of these agents. More so, these environments allow for diverse
customisation of different levels of requirements. Table III shows all
sequential environments with some information regarding the environments’
attributes.

(a) CARLA [87]

(b) TORC [88]

(c) SuperTuxKart [93]

(d) Overcooked [64]

(e) Mario [90]

(f) DOOM [91]

(g) CoinRun [92]

(h) Minecraft [94]

Figure 8: Examples of sequential environments.

###  V-D Environments Discussion

In this section we analysed all environments used in the literature and
classified them in three different categories: validation, precision, and
sequential. In their usage, we observe a behaviour similar to Zipf’s law,
where from the $66$ environments used to evaluate the various imitation
learning work (presented in Tables A, II, and III), six environments account
for more than $80\%$ of the total number of times these environments were
used. Figure 9 illustrates this behaviour by displaying the distribution of
the environments used in the literature. Walker-2D, MuJoCo robotic arm
simulation, Hopper, CartPole, MountainCar and HalfCheetah are the most common
environments. We classify five of these six as validation environments, except
for the robotic arm. As pointed out in Section V-A, this trend worries us
since using only validation environments might incur the wrong evaluation of
the imitation learning agent. Nevertheless, $42$ out of the $66$ environments
appear only once in the literature. This behaviour is worrisome because it
shows a lack of experimental protocols. We recognise that various tasks and
domains may require particular settings. However, when analysing the work for
autonomous vehicles, for example, we observe that the simulations were used
only once on all domain-relevant work. Furthermore, Zheng et al. also point
out this behaviour in their survey [35], when presenting the high-level and
low-level tasks, where they argue that researchers have sometimes might bias
their evaluation by selecting different environments. We attribute this
behaviour to reviewers and researchers looking to evaluation metrics and not
to evaluation protocols, which shows a need for a more formal definition of
more thorough protocols.

Figure 9: Distribution of environments used in the literature.

##  VI Imitation Learning Metrics

[width=]figures/06-metrics/taxonomy

Figure 10: Imitation learning metrics taxonomy.

Imitation learning evaluation metrics can drastically differ from each work.
Some work rely on more domain-specific metrics, while others present metrics
that measure how the agent performs compared to its teacher. The most common
taxonomy for evaluation metrics is the quantitative and qualitative
classification. However, this differentiation fails to consider the context of
a metric. For example, a metric that evaluates the distance between an agent’s
trajectory and its teacher in a task requiring precision will convey that the
agent is performing well. Conversely, if the environment requires less
precision and a goal state holds higher importance, using a metric that
measures an entire trajectory might convey that an agent that does not reach
its goal state but follows the teacher’s trajectory until a point, such as
stopping right before the goal, is better than one that takes a significantly
different trajectory but reaches the goal. Therefore, we classify metrics as
quantitative and qualitative but under behaviour, domain and model metrics.
Behaviour metrics are those that measure the agent’s behaviour, they convey
how distant an agent is from its teacher. One might argue whether behaviour
metrics measure teachers behaviour [45], but we shy away from this discussion
for now. Domain metrics measure domain-specific properties; they bring a more
contextual measurement for tasks with more specific requirements, such as how
many traffic infractions an agent commits in a self-driving car environment.
Finally, model metrics measure the learning procedure of agents, such as how
accurate a model is when predicting an action.

###  VI-A Behaviour metrics

Behaviour metrics are the most common metrics among all imitation learning
works. Usually, they depend on the environment’s reward or the distance from
one property of the teacher. The most used metrics are reward-based ones since
comparing the reward from an agent to its teacher intuitively shows how the
agents perform comparatively. Three variations from the reward-based metrics
are present in the reviewed literature:  (i) accumulated reward;  (ii) average
episodic reward; and  (iii) performance.

Accumulated reward is how much reward the agent accumulates in an episode. We
reiterate here that an episode in this work differentiates from Russell and
Norvig’s definition [43] and refers to a set of experiences from an agent in
an environment. The final value is given by summing the reward retrieved from
the Markov decision process in an episode with $n$ steps (Equation 4). This
metric conveys how the agent performs the environment task and allows
researchers to compare student and teacher rewards. However, using a single
episode might be misleading. For example, suppose the seed (a random numerical
value used to initialise the environment) used in the evaluation is present in
the teacher dataset. In that case, the agent must only match the given states
to those in the training set to achieve the same reward. Moreover, even when
the seed is absent in the teacher dataset, some initial states might be closer
to the teacher’s initial state than others. Thus, some states require less
generalisation from the agent than others, making some initial states ‘easier’
than others. Therefore, the accumulated reward might not be the best metric to
measure how well the agent generalises in some unfavourable instances.

$AR({\pi})=\sum_{i=1}^{n}\gamma^{n}r(s_{i},{\pi}(s_{i}))$ (4)

A more statistical approach would be to use the average reward for a set
number of episodes and study the deviation between each experience. Therefore,
the chance of finding diverse seeds increases, and the standard deviation
among all episodes can show whether the agent is consistent among different
experiences. The average of all accumulated rewards is called the average
episodic reward. In it, the agent records its accumulated reward for $k$
number of episodes, and the average is calculated (Equation 5). The number of
episodes is usually $100$ or $10$, and the standard deviation is shown
alongside it. The average episodic reward has the same benefits as the
accumulated reward and solves the problem from untested generalisation, as
long as there are enough episodes. However, it can be the case that an agent’s
average episodic reward is closer to that of a random initialised agent than
to that of a teacher in a specific task. To understand how an agent compares
between all agents, all three reward values would have to be given.

$AER({\pi})=\frac{1}{k}\sum^{k}_{j=1}AR({\pi})$ (5)

Performance solves the comparison issue by applying normalisation in the
accumulated reward. It uses a random policy’s ${\pi_{\xi}}$ accumulated reward
as the minimum and the teacher’s as the maximum (Equation 6). In this metric,
if an agent accumulates as much reward as a random one, the performance will
be $0$. However, if the same agent accumulates the same reward as the teacher,
its performance will be $1$. An agent can achieve a performance higher than
$1$ when it accumulates more than its teacher and lower than $0$ when its
rewards are inferior to a random agent.

$\mathcal{P}_{\tau}({\pi})=\frac{AR({\pi})-AR({\pi_{\xi}})}{AR({\pi_{\psi}})-AR({\pi_{\xi}})}$
(6)

The performance metric is usually retrieved from $k$ trajectories to get the
same benefits the average episodic reward has. However, some work directly
uses the average episodic reward instead of the accumulated reward when
computing the metric (Equation 7).

$\mathcal{P}_{\mathcal{T}}({\pi})=\frac{AER({\pi})-AER({\pi_{\xi}})}{AER({\pi_{\psi}})-AER({\pi_{\xi}})}$
(7)

Even though the performance metric has all the benefits of the other reward-
based metrics, it inherits the same issues from linear normalisation
procedures when dealing with skewed data [45]. For example, suppose we assume
that the minimum reward (random) is $0$, the maximum (teacher) is $100$. The
mean score from agents is $50$, and the median score is $20$. In this case,
the majority of the performance for the agents will be clustered towards the
minimum, and there will be very little differentiation among them. Thus,
performance becomes a less useful metric when the agent’s performance is
close.

Although the metric does not optimise the agent, it can misrepresent the
agent’s behaviour in a given environment. For example, imagine two scenarios
for an environment where the threshold to be considered optimal is a $100$
reward:  (i) the teacher reward is $75$, and the random reward is $0$; and
(ii) the teacher reward is $100$, and the random reward is again $0$.  In the
first scenario, when an agent achieves teacher behaviour, it will score a
performance of $1$, even though the agent was $25$ points short from an
optimal result. By achieving the same reward from the first scenario ($75$) in
the second scenario, the agent will only achieve a performance of $0.75$. This
example shows the problem of assuming as a premise that the teacher is an
optimal policy. If the selected teacher is not optimal, the performance metric
will not accurately represent how good the agent is. Moreover, in the first
scenario, although the agent is $25$ points from being considered optimal for
the task, the metric conveys optimality. Hence, there is a need to know
beforehand how optimal the teacher is, which can be a complex problem in
several application domains. On the other hand, if an agent achieves a reward
of $150$ in the same example, the performance will be $2$ and $1.5$ for the
first and second scenarios, respectively. Although the agent achieves
performance higher than the teacher, this result shows that:  (i) the samples
might not be ideal; and  (ii) the policy might not have learned the exact
behaviour from the teacher.

Distance metrics are the most common metrics after reward-based ones. They
usually involve measuring how far the agent is from one or more properties of
the teacher. For example, some work measures the distance between the
teacher’s and agent’s trajectories or between the artificial reward function
and the MDP’s $r$. Although the distance metric can vary, the most common
distances are the Manhattan distance (Equation 8, where $p=1$) and Euclidean
distance (Equation 8, where $p=2$), where an input $I$ may be single states,
trajectories or state-action pairs, and the Kullback-Leibler divergence
(Equation 9), where $P$ and $Q$ are two probability functions. A common issue
with distance-based metrics is the requirement for teachers to be accessible
to measure the chosen property (when observations do not provide the desired
property). These metrics require some higher degree of control for measuring
the same property for the agent. An example would be measuring the precision
of unseen trajectories. One might reserve part of the trajectory data to
measure how close agents and teachers are. However, this setup would require a
more significant number of data from researchers. A second issue with
distance-based metrics is that they might misrepresent the behaviour of an
agent due to compounding errors from different actions during the early stages
of an episode. Suppose an agent acts differently from its teacher earlier on.
In that case, the action might cause the trajectory to be further apart even
though the agent’s behaviour might be close to the teacher after these earlier
steps. This approach to measuring would require researchers to have a tree of
all possible teacher’s actions to compare the agent’s actions given a
diverging path from its initial data.

$d(I_{\pi_{\theta}},I_{\pi_{\psi}},p)=\left(\sum_{i=1}^{n}\left|I^{i}_{\pi_{\theta}}-I^{i}_{\pi_{\psi}}\right|^{p}\right)^{\frac{1}{p}}$
(8)

$d(P\mid\mid Q)=\sum_{s\in S}P(s)\ln\left(\frac{P(s)}{Q(s)}\right)$ (9)

The success rate ($SR$) also appears as a viable quantitative behavioural
metric. In it, the authors usually select a goal $g$ for the environment (it
could be the environment’s goal or a user-defined one) and collect a set of
trajectories $\mathcal{T}$, one for each episode, and check whether each
trajectory $\tau$ has $g$ in it (Equation 10).

$SR(g,\mathcal{T})=\frac{1}{\mid\mathcal{T}\mid}\sum_{\tau\in\mathcal{T}}G(g,\tau),\text{
where}$ (10)
$G(g,\tau)=\left\\{\begin{matrix}1&\forall_{n}s_{n}\in\tau\mid\exists_{t}s_{n}=g\\\
0&otherwise\end{matrix}\right.$ (11)

Coupling a reward-based and distance-based metric with a success rate can help
researchers understand how the agent behaves. Both initial metrics convey
information about the agent’s trajectory while the latter conveys if the agent
achieves its intended goal.

Nevertheless, these metrics do not have information regarding how human-like
the agent behaviour is (considering the agent trained with human
demonstrations). Therefore, some works employ qualitative metrics like
questionnaires and Turing tests. For questionnaires, questions are usually
regarding how similar to humans the agents look [26] or how easy it is to play
with the agent [95]. In the Turing Test, participants classify if a video from
a agent interacting with the environment is human or agent. Thus, the agent
aims to deceive the human participants (similar to adversarial learning).
Moreover, other researchers [63, 12, 61] also observe the agent’s trajectory
and do some qualitative analysis. More related to robotics, this analysis
usually displays a graph with various trajectories for the robot’s movement,
and the authors explain why the divergence in trajectory is good or acceptable
in a precise task. Unfortunately, the qualitative metrics that require human
participants are costly in comparison to the quantitative ones. Thus, they are
less common in more recent work. Although these metrics are essential for
evaluating imitation learning agents, we understand that to use qualitative
metrics requires significant effort. Creating a questionnaire requires
question design, which the area of computer science usually does not rely on,
and the Turing test requires participants to watch numerous videos of humans
and agents playing to vote.

###  VI-B Domain metrics

It is not always the case that behaviour metrics convey the information
required to evaluate whether an agent performs adequately in an environment.
Some domains allow the usage of specific metrics to measure the optimality of
the agent. These metrics are specific to these domains and do not make sense
to use anywhere else. The most common domain-specific metrics are those used
in autonomous driving and robotics.

For driving tasks, different behaviours from the agent can be measured, such
as distance travelled [28, 63], traffic infractions [56], time to collision
and distance headway [51], allowing for more accurate measurement of how
safely the agent navigates through the environment. Even though the reward
function could contain this information, by separating them, the researcher
can better comprehend the agent’s behaviour and try to optimise different
things without altering the reward function to represent the desired behaviour
better.

In robotics, most domain-specific metrics measure time [59, 63, 28] or failure
rate [28, 10]. In the case of time, these metrics convey how efficient the
robot arm is in performing a task. For example, how long it takes to order a
set of blocks, or how precise the robot can perform a variety of movements,
such as the pixel difference between teacher and agent in the same settings.
For failure rate, most metrics display the percentage of times the robot
failed to reach its goal, such as running in a simulation. Conversely, we did
not find any work that gave intuition behind the failure, which is crucial to
understanding whether the final result is good or bad. We understand that
doing this analysis is more costly and adds a qualitative tone to it, which
the domain-specific metrics lack; however, only providing a number
representing failure, such as how many times the robot fell during a run,
makes it hard to compare. Compared to humans, competitors almost never fall
during a running competition, and to other robots, the results might be
significant.

Considering that imitation learning agents do not use reward functions to
learn, it would not be very sensible to require a reward function to measure
how optimal an agent is. Therefore, some work use domain-specific scoring to
show how an agent performs. For example, in some environments, researchers use
how many coins [7, 26] the agent collects during an episode or how many goals
a team scores [96, 59]. Nevertheless, these metrics lack the finesse of other
metrics and should be used with behavioural- or model-specific metrics.

###  VI-C Model metrics

Model metrics are the usual metrics we see in learning research. They convey
how accurate or inaccurate a model is. Most work use accuracy since imitation
learning approaches use some form of behavioural cloning, which is a
supervised-learning approach (Equation 12). Other work use the error rate from
their models, which usually rely on generative models, such as forward
dynamics models. However, these metrics do not convey useful information
regarding the agent’s performance in a task. An example is the accuracy of
inverse dynamics models. Although they can have high accuracy in their
demonstration datasets, they might have poor accuracy in the teacher’s
observation since usually it follows other distribution. Therefore, accuracy
might give the wrong impression that the model is classifying each transition
correctly when it is not. Model-specific metrics are used during training to
understand whether a model can learn the desired task. Additionally, some work
also uses these metrics to interpret when a model becomes biased toward a set
of actions [6].

$\text{Accuracy}=\frac{\text{Number of correct predictions}}{\text{Total
number of predictions}}$ (12)

On the other hand, some work [25, 52] use other model metrics to have
intuition behind the model’s learning. In these work, the authors use
dimensionality reduction techniques, such as t-SNE [97], to interpret how good
their models are in clustering different actions. Although these techniques
are promising to give some qualitative intuition behind the models’
performance in clustering information, these metrics are often used when the
work focuses outside imitation learning, for example, transfer learning.
Moreover, when using these dimensionality reduction techniques, one should
consider that:  (i) they are highly sensitive to hyperparameters, as pointed
out by van der Maaten and Hinton [97];  (ii) interpreting them might be
challenging [98], which makes them less suitable as an explainable method; and
(iii) and some of these techniques lack robustness and might not provide
consistent results for similar datasets [99].

Saliency maps are also used as qualitative metrics in imitation learning
literature [6, 24] as model-domain metrics. They display in the model’s input
where does are the most active gradients. It gives some intuition on what the
model focuses on when predicting an action. Figure 11 displays an example
where the authors use the input image (Figure 11(a)) to display the attention
map (Figure 11(b)) to understand what the agent may be using to classify the
action. On the other hand, saliency maps share three of the major drawbacks of
dimensionality reduction techniques, which we understand we need to focus on:
(i) subjectivity;  (ii) unreliability; and  (iii) interpretability.  The first
comes from the fact that saliency maps are influenced by the choice of the
saliency model and parameters used in their generation. Different models may
produce varying results for the same input, and there is no consensus on which
model is the best [100]. Given the subjectivity of saliency maps, it can be
sensible to assume that such behaviour might cause unreliable results [101,
102]. Although these models can produce impressive results when connecting
labels with raw input data, connecting more complex knowledge representations
might be difficult, especially when connecting internal knowledge
representations from neural networks to humans [103]. Finally, saliency maps
might be challenging to interpret [104]. When interpreting these results, one
should remember that there are different possible conclusions given a
combination of input and label.

(a) Image used as input.

(b) Saliency map.

Figure 11: Saliency map example from Gavenski et al.’s work [6].

Therefore, researchers should consider using these model metrics to provide an
intuition of what imitation learning agents learned or how they behave, but
not as concrete proof. Furthermore, a combination of these metrics with
domain-specific and behaviour-specific metrics, which show a more accurate
representation of how optimal the agent is, should be present for a more
complete conclusion.

###  VI-D Evaluation protocol

All work presented in Section IV follow in some form the structure in
Algorithm 1 when evaluating their method. They train a policy
${\pi_{\theta}}$, given a set of teacher demonstrations $D$ (Line 1). For
simplicity, here we consider that all samples are demonstrations (have action
notations) (Line 1), but one should assume that this is only true for learning
from demonstration scenarios since learning from observation does not require
action information during training. Algorithm 1 also only considers trained
policies since it refers to the final evaluation protocol, so we remove any
iterative training notation. Finally, researchers report using some metrics
presented in this section. The number of episodes from an agent in an
environment ($N$) varies between $5$ to $100$. Although imitation learning
work rarely inform how the seeds are selected. It is sensible to assume that
the seeds used to collect teacher demonstrations are absent from the
evaluation process. Unfortunately, this assumption is not always true since
the newest version from OpenAI Gym [105] removes the option for users to set a
specific seed for the environment. Therefore, some work might use the same
seed for both teacher and agent, which can lead to unfair comparisons.

Algorithm 1 Default evaluation protocol

$D\leftarrow\left\\{(s_{0},a_{0});\cdots;(s_{n},a_{n})\right\\}$

1:train $\pi_{\theta}$ with $\mathcal{T}$

2:$AER_{\pi_{\theta}}\leftarrow 0$

3:$\mathcal{P}_{\pi_{\theta}}\leftarrow 0$ $n\leftarrow 1$ to $N$

4:$AER_{\pi_{\theta}}\leftarrow
AER_{\pi_{\theta}}+\nicefrac{{AR({\pi_{\theta}})}}{{N}}$

5:$\mathcal{P}_{\pi_{\theta}}\leftarrow\mathcal{P}_{\pi_{\theta}}+\nicefrac{{\mathcal{P}({\pi_{\theta}})}}{{N}}$

Additionally, some methods employ other experiments to evaluate their
performance in scenarios with different amounts of samples (sample efficiency)
[55, 65, 3, 21]. In these scenarios, researchers usually reduce the number of
teacher episodes the agent has access to and perform the same evaluation
protocol in Algorithm 1. Other work experiment with different degrees of
optimality and measure each method’s robustness [45]. In these scenarios, they
create teacher samples with different degrees of optimality and perform the
evaluation protocol in Algorithm 1.

Finally, to compare their work, other approaches run the same steps from
Algorithm 1 for all baselines selected. Researchers frequently select the
traditional behavioural cloning method as a benchmark alongside newer
approaches in the same category as the proposed method. For example, learning
from observation methods usually compare their approach with behavioural
cloning since it has access to the teacher’s actions and works as a ground-
truth comparison. Moreover, adversarial learning methods compare their
approach with Ho and Ermon’s work [4], even though they are outdated.

###  VI-E Metrics Discussion

In this section, we listed different metrics with different contexts for
imitation learning work. We divided them into three different groups:
behaviour, domain and model. When studying these metrics, we found $50$
different ones (differentiating them by context and equation and not only by
name). As discussed in Section VI-A, most metrics focused on quantitative
analysis, with the most common one using reward to measure how optimal the
agent is. As with the environments, the metrics also present many that appear
once. However, unlike environments, comparing all methods with a new metric
presents fewer unfair comparison problems if researchers compare all methods
using the same metric. Additionally, we observe an increase in performance
usage, which considers policies from random and teachers agents to measure
optimality.

Figure 12: Count and usage of quantitative and qualitative metrics in the
imitation learning literature reviewed.

Qualitative metrics are rare in most novel works, which is understandable
since the cost of creating questionnaires or running Turing tests might make
some research unfeasible. However, the absence of qualitative metrics in the
literature is problematic since it does not allow researchers to understand
how human-like the agent is. Figure 12 shows the discrepancy between
quantitative and qualitative metrics and their usage.

Finally, we analyze different protocols for evaluating imitation learning
methods. Most work follows the same guidelines when evaluating novel methods
and baselines. In its essence, the presented protocol offers a straightforward
way to evaluate imitation learning methods. However, researchers fail to show
how their method compared outside of the optimal scenario by not applying the
same protocol to measure other characteristics outside the overall
performance, such as efficiency and effectiveness. We conclude that following
protocols presented in Section VI-D, such as our protocol where we reduce the
degree of optimality of experts [45], might alleviate some of the issues from
the original protocol but will not solve them.

##  VII Reflections

In this section, we reflect on the insights we had while writing this survey.
These reflections can help researchers better understand the field of
imitation learning and its challenges, which we further discuss at the end of
this section. Importantly, we aim to help new researchers in designing
imitation learning agents while avoiding common pitfalls.

For this section, we consider that under the loss function ${\ell}$, and agent
learn how to maximise the return of ${\ell}$ to perform a task under certain
conditions. ${\ell}$ might be the accumulated value of ${M}$’s reward function
($\sum_{i}\gamma^{i}r_{i+1}$) or capture some other specific desired
behaviour. For example, ${\ell}$ may want to minimise the number of vertices
in the solution of a path-finding problem, or instead compute the overall
length of the path according to the weight of the edges between vertices.
Training a policy ${\pi}$ with parameters $\theta$ and loss function ${\ell}$
yields the value function $v_{\ell}(\pi_{\theta},s)$, which measures the
quality of ${\pi}$ for each state $s$. A policy ${\pi}$ with parameters
$\theta_{o}$ is optimal for ${\ell}$ when for all states $s$ and all
parameters $\beta$,
$v_{\ell}({\pi_{\theta}}_{o},s)~{}\geqslant~{}v_{\ell}(\pi_{\beta},s)$.
Moreover, ${\pi}$ is theoretically optimal, when for all loss functions $h$
and all parameters $\beta$,
$v_{\ell}({\pi_{\theta}}_{o},s)~{}\geqslant~{}v_{h}(\pi_{\beta},s)$. In other
words, even though an agent may optimal under a given function ${\ell}$, it
may still under-perform under another loss function $h$.

###  VII-A Imitation Learning Evaluation

When we consider IL systems, we are likely to compare them to machine learning
(ML) and reinforcement learning (RL) approaches. After all, IL methods are a
relaxation of the RL approach into an ML supervised classification problem
(Section III). However, this comparison comes with some preconceptions that we
want to address:  (i) disparities over testing processes;  (ii) divergence in
benchmarking; and  (iii) the main goal of IL approaches.

Traditional ML applications divide their data into three different group sets:
training, validation and test sets. They assume that all sets share the same
distribution over data, and the test set should be only used at the end to
evaluate how the model will perform under a ‘real-world’ scenario. This allows
a well-defined and clear evaluation. Conversely, IL approaches shy away from
using test sets. Since the typical application for IL approaches is agent-
based, it is more practical to test the final learned policy via simulation of
the environment. This testing approach is similar to how RL methods are
evaluated. However, the training processes for RL and IL agents are different.
Training RL agents involves the use of exploration and exploitation phases
without a training set. During these phases, agents learn the different values
for states outside an optimal trajectory. Thus, even though a RL agent might
not visit a state often, the agent will have some information about how to act
on it. The training of IL agents is limited to the states in the training set,
which means they have no information about states outside this set. For
example, consider the grids in Figure 13, where the objective is for the agent
to reach square $g$. Suppose the training set of an IL agent consists only of
the two optimal trajectories in purple and blue given in Figure 13(a). After
training, the agent will have acquired some knowledge about how to get to $g$
from any square except for $s_{x}$. Furthermore, the probability of reaching
$s_{x}$ is $0\%$ from the learned policy’s perspective since the teacher
samples prevent the agent from reaching $s_{x}$ from any state. Now consider
the training set in Figure 13(b), containing $4$ green, $3$ purple, $3$ blue,
and $1$ orange trajectories. The learned policy will not be successful for
$50\%$ of the non-goal states, because the states $s_{0}$, $s_{x}$, $s_{y}$,
and $s_{z}$ lead to an infinite loop. If the agent starts in $s_{0}$, it will
go to $s_{y}$, since $P(\text{`{up}'}\mid s_{0})=6/11$. If it starts in
$s_{z}$, then it will go to $s_{x}$, since $P(\text{`{up}'}\mid s_{z})=4/5$.
Once in $s_{x}$ or $s_{y}$, it will loop because from $s_{x}$, it will go to
$s_{y}$, since $P(\text{`{left}'}\mid s_{x})=4/10$ (highest) and from $s_{y}$,
it will go to $s_{x}$, since $P(\text{`{right}'}\mid s_{y})=6/10$. Therefore,
it is important to understand how the simulations used for testing relate to
the samples used in the training since particular initialisations of the tests
might involve states and behaviours not present in the training data. In the
example given in Figure 13, it is possible to understand the looping behaviour
from a manual inspection alone. However, this may be impossible in
environments with a high number of states, such as continuous environments or
for higher dimensions. Moreover, other considerations are needed to understand
the evaluation results of IL agents, such as poor performance due to drastic
state variance from training and testing, or good performance due to lower
variance and data leakage, which occurs when the training data is present in
the test set.

[width=]figures/07-challenges/maze_all

(a) Optimal trajectories.

[width=]figures/07-challenges/maze_some

(b) Multiple trajectories.

Figure 13:  Examples for a teacher dataset with blue, green, orange and purple
trajectories. The numbers give the distribution of trajectories in the
dataset.

Secondly, with the popularity increase of ML approaches, benchmarking datasets
have helped to evaluate how different methodologies perform under the same
conditions and some applications hide test data from researchers to maintain
evaluation integrity [106, 107]. Moreover, using the same dataset (and the
same proportion of training/testing split) allows researchers to validate each
other’s results more efficiently, and simplifies the evaluation process
because there is no need to run every relevant baseline over the data again.
RL work uses the same benchmarking approach, with the performance of RL
methods evaluated on popular common environments [108]. Unfortunately, the
same uniformity in evaluation has not yet been employed in IL methods, which
usually create customised training data and compare performance with baselines
over the data for which the other methods were not initially created.
Therefore, when comparing a new approach, a teacher dataset must be created,
available reliable code found (with hyperparameters or requiring a vast search
if wanting to compare with the best results for each method), and every
baseline must be run over the generated data, leaving much room for issues,
such as lack of data or diversity. This divergence in benchmarking further
corroborates the main problems from the disparities in the test process [35].

Finally, we argue that IL’s ultimate goal is to learn how to accomplish a task
using the teacher’s data and not simply to copy the actions taken by the
teacher. In other words, the IL agent should act as the teacher would in
previously seen states, but it should also somehow learn how the teacher would
act in unseen states. For example, in Figure 13(a), given state $s_{x}$, ‘up’
and ‘right’ are optimal actions ideally to be taken by the IL agent. If that
is not possible, the agent should at least attempt to choose an action that
takes it back to an already-seen trajectory. Analogously, the problem
highlighted in Figure 13(b) stems from the fact that the IL alternates between
competing trajectories without ever completing either.

###  VII-B Expert optimality feasibility

Imitation learning always assumes that training samples come from the
execution of a task by an expert [1]. While it is fair to assume that the
source is proficient, the implicit presumption of the theoretical optimality
of the demonstration itself is problematic. Firstly, the more complex the
environment, the harder it is to ascertain whether the samples are optimal.
For example, while it might be easy to identify which episodes are near
optimality in the CartPole environment [77] (where performance is considered
optimal when the accumulated rewards match the number of steps), it is much
more challenging to evaluate how ‘good’ an agent is in autonomous driving
scenarios, such as SUMO [89]. In particular, some important aspects of the
driver’s behaviour (e.g., situational awareness) may be difficult to fully
measure. Moreover, in highly dynamic systems such as in real-world driving,
how can one be sure that the samples collected fully capture all aspects of
the optimality of the execution? In practice, even though the teacher might be
proficient, the samples collected may be corrupt or not cover all the
experiences necessary for the learner to achieve the same level of optimality.
Although it is not impossible to evaluate performance in both cases, it might
be impractical to do so. Since reproducibility is crucial, we expect an
increasing focus on presenting more data about the teacher’s behaviour, an
aspect that is lacking in most work, except for [45, 109].

Secondly, we argue that the demonstrations may not necessarily fully capture
the intended optimal behaviour. For example, they may not necessarily achieve
other secondary goals, such as fairness and safety, that may not play a part
in the scenarios available for the learning. We note that safe and fair agents
are not inherently suboptimal, but as in the first scenario, it might be
impossible to create a perfect fitness function to account for the desired
behaviour. In these cases, samples may unintentionally be used where the
teacher does not fully constrain actions, resulting in less than theoretical
optimal agents (with respect to e.g., safety requirements). For example,
suppose an agent is trained to drive a car and uses samples from a human
driver, who may drive incautiously. In this case, the agent will learn like
its teacher, which is optimal for some driving scenarios but not optimal in
situations such as emergencies where the driver must act with extreme caution.
We therefore believe that only striving for optimal agents is not always
sufficient.

Furthermore, consideration for aspects beyond the agent’s overall reward with
a focus on human feedback is central to ‘Reinforcement Learning from Human
Feedback’ [110, 111]. Here, theoretical optimality is sacrificed for other
four aspects. Firstly, humans can be more nuanced than machines when taking
action and safety and ethical considerations override overall performance.
Secondly, humans have intrinsic notions of solution quality that are difficult
to capture in the design of reward functions. Human feedback reduces the need
for intricate designs. Thirdly, human guidance in the agent’s exploration
phase helps to reduce the training time. Lastly, human feedback allows for
curriculum learning, where the agent starts with human-curated samples and
gradually transitions to more autonomous learning paradigms. Since imitation
learning, by definition, relies on human examples to guide the entire learning
process, we argue that IL agents should also consider these qualitative
aspects, which are possible avenues for future research.

Therefore, the trade-off between expert and teacher agents has to be
understood when developing a new agent. In some cases, such as robotics for
manufacturing, it is desirable to employ expert policies because they are
guaranteed to be most efficient. In others, such as driving, optimal behaviour
may be difficult to demonstrate, and teacher policies are acceptable. We argue
that by understanding the role of experts and teachers in a task, researchers
can make more informative decisions about the most appropriate type of
demonstrations and what the performance evaluation really measures.

###  VII-C Expert Necessary Requirements

[width=1]figures/07-challenges/teacher

Figure 14:  Diagram of teacher, optimal and expert policy sets.

As we saw in the previous section, it is desirable and perhaps inevitable that
IL agents sometimes learn from non-optimal sources. As we see it, if an
environment follows an MDP (as described in Section III), it has a universe of
all possible policies $\Pi$ (that either succeed or fail to solve the
Markovian problem). Some successful policies are also optimal (${\Pi_{op}}$).
The training data does not usually contain all optimal policies, restricting
itself to a subset of these (the ‘expert’ policies $\Pi^{*}_{\psi}$). In this
scenario, researchers should strive to acquire all optimal policies for
training the agent, such that the expert policy set is complete
($\Pi^{*}_{\psi}={\Pi_{op}}$). In general, it is impossible or impractical to
do so, resulting in incomplete coverage. Furthermore, in practice, the
successful policies used to teach a task or skill also contain policies that
are not optimal (the ‘teacher’ policies $\Pi_{\psi}$, which we also call
complete when $\Pi_{\psi}={\Pi_{op}}\cup\Pi_{np}$). The inclusion of non-
optimal policies in $\Pi_{\psi}$ can happen because researchers sometimes
provide their own demonstrations, and it may not be possible to check them for
optimality. Therefore, we argue that for clarity, we should call the policies
used for training simply as teachers, reserving the term ‘experts’ for
policies whose optimality has been ascertained. Figure 14 illustrates the
relationship between all policies.

TABLE I: Necessary requirements for expert validation. Aspect | Description
---|---
(i) | State space | Finite / Discrete
(ii) | Action space | Finite / Discrete
(iii) | State observability | Fully observable
(iv) | MDP sample coverage | All MDP states

We now look at the role played by the teachers’ policies. Table I presents a
set of necessary requirements all of which need to be fulfilled for the source
data to be reliably considered an expert. (i) and (ii) relate to the state and
action space from environments. Most often, work that use continuous
(infinite) states and actions spaces in their problem formulation impose
strict assumptions, such as injective MDPs [21] and trajectories with the same
initial states [3]. These assumptions are not applicable in most practical
scenarios or require extensive modifications. Therefore, we can assume that
work whose environments have infinite state and action spaces cannot guarantee
optimality of the training data. (iii) refers to the observability of the
environments. Optimality of policies for environments with partial or faulty
observations [22] cannot be guaranteed. (iv) considers how much of the state
space the samples collected cover. We argue that in cases where the samples do
not fully cover the Markovian problem, we cannot guarantee that the agent
fully learns the expert’s behaviour. Hence, the output from training might be
an optimal policy but does not guarantee optimal behaviour in unseen states.
Nevertheless, we observe that although researchers might be able to fulfil
these conditions for optimality, they should not be used as the sole criteria
for the evaluation of the overall performance of the learning approach, since
they are too restrictive. In real-world applications, it is often the case
that one of more conditions need to be relaxed for practical use.

###  VII-D Challenges and Future Directions

Throughout this survey, we have discussed several challenges that imitation
learning agents face. This section highlights some of them and discusses
possible future directions for imitation learning research.

Safety is a crucial aspect of agent-based systems. Without it, agents can
cause harm to themselves, other agents, or the environment. However, for
imitation learning agents, safety can be hard to achieve since they learn
through a teacher’s perspective and without a direct signal. When learning via
observation, the samples might lack information regarding the environment’s
dynamics, leading to unsafe behaviour due to the agent’s lack of knowledge.
Moreover, the teacher might not behave safely, leading to an unsafe agent.
Thus, we believe there is space to research how imitation learning agents can
learn safely from teachers. Possibly, imitation learning agents should learn
how to differentiate between safe and unsafe samples and teachers.

Learning efficiency became a trend for imitation learning algorithms [55, 50,
68, 16]. For imitation learning, researchers measure efficiency by the number
of samples needed to learn a task. However, we believe that efficiency by
itself might be a misleading metric. For example, an agent who learns a task
by only observing a single episode from a teacher might be considered
efficient. Conversely, suppose the agents fail to generalise to other
scenarios or act unsafely. In that case, their efficiency is not as good as it
seems. Thus, efficiency should be measured by the number of samples needed to
learn a task while measuring other factors, such as generalisation and safety.

Learning effectiveness is often overlooked by imitation learning researchers.
In this work, we define effectiveness as the ability of an agent to learn a
task or skill optimally (or close to it) with non-optimal teachers.
Effectiveness is as crucial as efficiency since it is not enough to learn a
task or skill with few samples if the agent does not learn it well. Moreover,
we believe effectiveness is more challenging than efficiency due to how
imitation learning agents learn. Considering our premise that is impossible to
know how optimal a teacher is in real-world applications, it is essential to
develop agents that can learn effectively from non-optimal teachers. Thus, we
believe that imitation learning researchers should focus more on effectiveness
than efficiency.

Adaptability comes from combining generalisation and transfer learning.
Generalisation is the ability of an agent to learn a task or skill from a
teacher and apply it to other unseen scenarios (similar to effective
learning). Transfer learning is the ability of an agent to learn a task or
skill from a teacher and apply it to other tasks or skills. More formally,
transfer learning in imitation learning refers to the agent’s ability to
translate knowledge it learned from one Markov Decision Process $M$ to another
$M^{\prime}$, either regarding the transition function $T$ (from $T$ to
$T^{\prime}$) or in the form of a new immediate reward function $r$.
Considering that imitation learning agents do not have access to $r$, this
form of transfer learning usually appears in the sense of $T$ remaining the
same but the goal of the environment changing. Given the nature of the
imitation learning agent’s training and lack of a direct signal, we believe
these two aspects are essential to strive for. By achieving generalisation or
transfer learning, imitation learning agents can succeed more in real-world
scenarios. However, researchers should focus on learning paradigms outside
imitation learning, such as symbolic learning, to achieve these goals.

Multi-agent systems are not a research topic as common as single agents for
imitation learning researchers. We believe such behaviour comes from imitation
learning being more adequate for learning low-level tasks, which usually
multi-agent systems do not have. However, imitation learning can be used to
learn high-level tasks, such as coordination and communication, which are
essential for multi-agent systems. Imitation learning is a highly adaptable
process that can be employed with other techniques, such as planning [50, 30],
explorations [3, 6]. Perhaps using imitation learning on more descriptive
processes, such as ‘Belief Desire and Intention’, can help imitation learning
agents achieve better results on multi-agent systems scenarios.

##  VIII Conclusions

In this survey, we present a comprehensive review of imitation learning
methods, environments and metrics. We provide background knowledge required to
understand the field and how different imitation learning methods use the data
to learn a policy. For each method we discuss the main ideas behind it, the
environments used to evaluate, and the metrics used to compare them.

Our contributions is an organisation of the field in terms of their methods,
environments and metrics. In methods, besides summarising the main approaches
for imitation learning highlighting their aspects, we provide a taxonomy that
focuses on new trends in the field, complementing existing ones. This taxonomy
displays newer trends and helps researchers identify new opportunities without
supplanting the traditional off-policy and on-policy taxonomy. For
environments, we introduce a novel taxonomy for environments that classifies
them based on their characteristics and role in an evaluation process
(validation, precision, and sequential). This classification will help
researchers understand how different environments play a role in their
evaluation process and should help reviewers judge how effective the
evaluation of a new method is. Being able to look at environments in a more
systematic way should help overcome the problem of assessing the effectiveness
of a method when evaluation occurs in a one-off environment. In metrics, we
systematically review the metrics used to evaluate imitation learning methods
and discussed their advantages and disadvantages. Like our taxonomies in
methods and environments, we classified the metrics into three categories:
behaviour, domain, and model. Each of these categories can be refined into
qualitative and quantitative options, with behaviour metrics further divided
into reward-based and distance-based metrics. We highlight the importance of
using these metrics groups to evaluate a method since they provide different
perspectives of the method’s performance. Given the variety of metrics, an
open question is, whether we can we aggregate them to rank methods. Finally,
we show that the imitation learning field lacks standardisation in the
evaluation process, making it difficult to compare different methods from the
perspectives of the environments or metrics. To the best of our knowledge, no
other survey includes such comprehensive taxonomies or provides in-depth
explanations for imitation learning environments and metrics.

While there has been much recent progress, we believe the imitation learning
field shows much potential for future research. This survey furnishes a solid
foundation from which to further develop the area, one that allows us to
systematically evaluate new approaches with regards to the state of the art.
Specifically, by contrasting the different ways in which past work has been
evaluated, we see that there is much to improve when one evaluates the
resulting behaviour, rather than relying on environmental metrics like
traditional reinforcement learning does. More specifically, imitation learning
can benefit from other research fields that pursue more human-like behaviour,
such as reinforcement learning from human feedback.

## References

  * Hussein et al. [2017] A. Hussein, M. M. Gaber, E. Elyan, and C. Jayne, “Imitation learning: A survey of learning methods,” _ACM Computing Surveys (CSUR)_ , vol. 50, no. 2, pp. 1–35, Jun 2017.
  * Pomerleau [1988] D. A. Pomerleau, “Alvinn: An autonomous land vehicle in a neural network,” in _International Conference on Neural Information Processing Systems_ , 1988, pp. 305–313.
  * Kidambi et al. [2021] R. Kidambi, J. Chang, and W. Sun, “Mobile: Model-based imitation learning from observation alone,” in _Advances in Neural Information Processing Systems_ , M. Ranzato, A. Beygelzimer, Y. Dauphin, P. Liang, and J. W. Vaughan, Eds., vol. 34. Curran Associates, Inc., 2021, pp. 28 598–28 611. [Online]. Available: https://proceedings.neurips.cc/paper/2021/file/f06048518ff8de2035363e00710c6a1d-Paper.pdf
  * Ho and Ermon [2016] J. Ho and S. Ermon, “Generative adversarial imitation learning,” in _Advances in neural information processing systems_. Advances in neural information processing systems, 2016, pp. 4565–4573.
  * Torabi et al. [2018a] F. Torabi, G. Warnell, and P. Stone, “Behavioral cloning from observation,” in _Proceedings of the 27th International Joint Conference on Artificial Intelligence_ , ser. IJCAI’18. AAAI Press, July 2018, pp. 4950–4957. [Online]. Available: https://www.ijcai.org/proceedings/2018/0687.pdf
  * Gavenski et al. [2020] N. Gavenski, J. Monteiro, R. Granada, F. Meneguzzi, and R. C. Barros, “Imitating unknown policies via exploration,” in _International British Machine Vision Virtual Conference_ , 2020, pp. 1–8. [Online]. Available: \url{https://www.bmvc2020-conference.com/conference/papers/paper_0774.html}
  * Edwards et al. [2019] A. D. Edwards, H. Sahni, Y. Schroecker, and C. L. Isbell, “Imitating latent policies from observation,” in _Proceedings of the 36th International Conference on Machine Learning_ , ser. ICML 2019. Proceedings of the 36th International Conference on Machine Learning, 2019, pp. 1755–1763.
  * Zhang et al. [2018] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel, “Deep imitation learning for complex manipulation tasks from virtual reality teleoperation,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, may 2018.
  * Pathak et al. [2018] D. Pathak, P. Mahmoudieh, G. Luo, P. Agrawal, D. Chen, Y. Shentu, E. Shelhamer, J. Malik, A. A. Efros, and T. Darrell, “Zero-shot visual imitation,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , June 2018.
  * Nair et al. [2017] A. Nair, D. Chen, P. Agrawal, P. Isola, P. Abbeel, J. Malik, and S. Levine, “Combining self-supervised learning and imitation for vision-based rope manipulation,” in _2017 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, may 2017.
  * Lynch et al. [2020] C. Lynch, M. Khansari, T. Xiao, V. Kumar, J. Tompson, S. Levine, and P. Sermanet, “Learning latent plans from play,” in _Proceedings of the Conference on Robot Learning_ , ser. Proceedings of Machine Learning Research, L. P. Kaelbling, D. Kragic, and K. Sugiura, Eds., vol. 100. PMLR, 30 Oct–01 Nov 2020, pp. 1113–1132. [Online]. Available: https://proceedings.mlr.press/v100/lynch20a.html
  * Finn et al. [2016] C. Finn, S. Levine, and P. Abbeel, “Guided cost learning: Deep inverse optimal control via policy optimization,” in _Proceedings of The 33rd International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, M. F. Balcan and K. Q. Weinberger, Eds., vol. 48. New York, New York, USA: PMLR, 20–22 Jun 2016, pp. 49–58. [Online]. Available: https://proceedings.mlr.press/v48/finn16.html
  * Yang et al. [2023] S. Yang, W. Zhang, R. Song, J. Cheng, H. Wang, and Y. Li, “Watch and act: Learning robotic manipulation from visual demonstration,” _IEEE Transactions on Systems, Man, and Cybernetics: Systems_ , pp. 1–13, 2023.
  * Paolillo et al. [2023] A. Paolillo, P. R. Giordano, and M. Saveriano, “Dynamical system-based imitation learning for visual servoing using the large projection formulation,” in _ICRA’23-IEEE Int. Conf. on Robotics and Automation_ , 2023.
  * Pérez-Dattari and Kober [2023] R. Pérez-Dattari and J. Kober, “Stable motion primitives via imitation and contrastive learning,” _arXiv_ , 2023.
  * Sun et al. [2019] W. Sun, A. Vemula, B. Boots, and D. Bagnell, “Provably efficient imitation learning from observation alone,” in _Proceedings of the 36th International Conference on Machine Learning_ , ser. Proceedings of Machine Learning Research, K. Chaudhuri and R. Salakhutdinov, Eds., vol. 97. PMLR, 09–15 Jun 2019, pp. 6036–6045. [Online]. Available: https://proceedings.mlr.press/v97/sun19b.html