$\cup_{i\in[m]}R_{i}=(\min_{i\in[m]}x_{i}+(1-\frac{|S_{C_{i}}|}{n}),1]$ if
there exists a nonempty $R_{i}$, and is empty otherwise. Therefore the length
of the feasible region is

$\min\left\\{\min_{i\in[m]}\left\\{x_{i}+(1-\frac{|S_{C_{i}}|}{n})\right\\},1\right\\}-\max\left\\{0,\max_{i\in[m]}\left\\{x_{i}-(1-\frac{|S_{C_{i}}|}{n})\right\\}\right\\}.$

By symmetry, we suppose without loss of generality that

$\min_{i\in[m]}\left\\{x_{i}+(1-\frac{|S_{C_{i}}|}{n})\right\\}\leq 1.$

We consider the following transformation. Let $j$ correspond to the agent
location with the largest right infeasible interval, i.e.

$j:=\arg\min_{i\in[m]}\left\\{x_{i}+(1-\frac{|S_{C_{i}}|}{n})\right\\},$

and we have $x_{j}<1$. If $x_{j}>0$, move all agents at $x_{j}$ to $0$. We
show that this transformation weakly decreases the feasible region length: in
$\min_{i\in[m]}\left\\{x_{i}+(1-\frac{|S_{C_{i}}|}{n})\right\\}$, $x_{i}$
decreases to $0$ and $|S_{C_{i}}|$ weakly increases (it strictly increases if
there are already agents at $0$). Furthermore, the
$\max_{i\in[m]}\left\\{x_{i}-(1-\frac{|S_{C_{i}}|}{n})\right\\}$ term is
unaffected unless the shifted group of agents originally corresponded to the
maximum value, in which case the $x_{i}$ term decreases by at most the length
of the shift, and the $(1-\frac{|S_{C_{i}}|}{n})$ term weakly decreases as
$|S_{C_{i}}|$ weakly increases. Therefore this transformation weakly decreases
the feasible region length. Now the agent location with the largest right
infeasible interval is $0$, so our feasible region length is

$(1-\frac{|S_{C_{j^{\prime}}}|}{n})-\max\left\\{0,\max_{i\in[m]}\left\\{x_{i}-(1-\frac{|S_{C_{i}}|}{n})\right\\}\right\\},$

where $j^{\prime}$ corresponds to the location $x_{j^{\prime}}=0$. If the
right boundary of the largest left infeasible interval
$\max_{i\in[m]}\left\\{x_{i}-(1-\frac{|S_{C_{i}}|}{n})\right\\}$ corresponds
to the agents at $x_{j^{\prime}}=0$, then it is at most $0$, and the feasible
region length is $(1-\frac{|S_{C_{j^{\prime}}}|}{n})$ which is at least
$\frac{n-n_{C}}{n}$. We now suppose that this is not the case.

We have

$\max_{i\in[m]}\left\\{x_{i}-(1-\frac{|S_{C_{i}}|}{n})\right\\}\leq\max_{i\in[m]}\left\\{1-(1-\frac{|S_{C_{i}}|}{n})\right\\},$

so the feasible region length is at least

$(1-\frac{|S_{C_{j^{\prime}}}|}{n})-\max\left\\{0,\max_{i\in[m]}\left\\{\frac{|S_{C_{i}}|}{n}\right\\}\right\\}.$

Since $\arg\max_{i\in[m]}\left\\{\frac{|S_{C_{i}}|}{n}\right\\}\neq
j^{\prime}$, the feasible region length is at least

$1-\frac{|S_{C_{j^{\prime}}}|+|S_{C_{k}}|}{n}\geq\frac{n-n_{C}}{n}$

where $k\neq j^{\prime}$.

We now know the length of the (continuous) feasible region corresponding to
the classic agents is at least $\frac{n-n_{C}}{n}=\frac{n_{O}}{n}$. We now
consider the obnoxious agents. Suppose we construct an open ball of radius
$\frac{|S_{O_{i}}|}{2n}$ around each group of obnoxious agents at the same
location. Any location within one of these open balls is infeasible. The sum
of ball lengths is $\frac{n_{O}}{n}$, so using a similar argument to that in
the proof of Proposition 2, we see that a feasible solution with respect to
both the classic agents’ and obnoxious agents’ distance inequalities always
exists. ∎


# Discretizing Dynamics for Maximum Likelihood Constraint Inference

Kaylene C. Stocking1, David L. McPherson1, Robert P. Matthew2, and Claire J.
Tomlin1  1 Department of Electrical and Computer Engineering
University of California, Berkeley  2 Department of Physical Therapy and
Rehabilitation Science
University of California, San Francisco

###### Abstract

Maximum likelihood constraint inference is a powerful technique for
identifying unmodeled constraints that affect the behavior of a demonstrator
acting under a known objective function. However, it was originally formulated
only for discrete state-action spaces. Continuous dynamics are more useful for
modeling many real-world systems of interest, including the movements of
humans and robots. We present a method to generate a tabular state-action
space that approximates continuous dynamics and can be used for constraint
inference on demonstrations that obey the true system dynamics. We then
demonstrate accurate constraint inference on nonlinear pendulum systems with
2- and 4-dimensional state spaces, and show that performance is robust to a
range of hyperparameters. The demonstrations are not required to be fully
optimal with respect to the objective, and the most likely constraints can be
identified even when demonstrations cover only a small portion of the state
space. For these reasons, the proposed approach may be especially useful for
inferring constraints on human demonstrators, which has important applications
in human-robot interaction and biomechanical medicine.

##  I Introduction

Inverse reinforcement learning (IRL) allows an agent to infer the goals
driving someone’s behavior and learn to complete the same task simply by
observing. This is a powerful paradigm for learning new behaviors from
scratch, but doesn’t encompass all of the useful information we may extract
from observations. For example, consider the case where the agent already has
a good policy for some task, but notices that an expert demonstrator is
deviating from the optimal behavior. A reasonable explanation would be that
the demonstrator is acting under new environmental constraints that the agent
is unaware of. As a concrete example, we can imagine a scenario where an
autonomous vehicle (the agent) is following a car that suddenly swerves (the
demonstrator). Since both vehicles have the same policy of avoiding collisions
and following the road, the agent can infer that an obstacle suddenly appeared
in the road and take evasive action even before it can detect the obstacle
directly. Taking cues from other agents is an important aspect of intelligent
behavior that can help compensate for problems such as sensor failure or
perceptual error.

Figure 1: An overview of maximum likelihood constraint inference on a
continuous system. (a) A human demonstrates task performance while obeying an
unknown constraint. (b) Their continuous trajectory is projected onto the
approximate state space $\mathcal{S}$ of a tabular Markov Decision Process
(MDP). (c) The human’s behavior deviates from what the MDP expects due to the
unmodeled constraint. (d) The most likely constraint $\mathcal{C}^{*}$ can be
inferred, causing the tabular MDP trajectory to more closely match that of the
human. The inferred constraint region can be mapped back into the continuous
state space for applications such as identifying potential biomechanical
problems.

The process of detecting constraints that help explain the behavior of a
demonstrator is called constraint inference. Scobee and Sastry [17] applied
the maximum entropy IRL framework to this problem, resulting in an algorithm
that can identify the most likely constraints from a hypothesis set. However,
this approach is limited to systems with tabular state-action spaces. This
precludes its use in many real systems of interest whose dynamics are
inherently continuous. In this paper, we describe a procedure for creating a
tabular approximation of an arbitrary continuous system, and show that maximum
likelihood constraint inference (MLCI) can be used to infer constraints on the
approximated system that transfer well to the original continuous system. We
analyze the effects of various approximation hyperparameters on the accuracy
of constraint inference on an example 2-dimensional pendulum system. Although
this analysis does not necessarily generalize to other sets of dynamics,
following a similar procedure on a system of interest can indicate whether the
approximation is sufficient for meaningful constraint inference. We also
present a technique for estimating confidence in the inferred constraint in
the form of a Bayesian probability update.

In addition to applications where the agent wishes to use learned constraints
to improve its own policy, this extension of MLCI allows us to perform
constraint inference directly from observed human movements. One exciting
potential application of this work is in individuals with _non-specific_ low-
back pain, pain that is not immediately attributable to a specific pathology.
This affects 84% of people in their lifetime, with around 12% of people being
disabled from this pain [1]. Our proposed constraint inference approach would
enable an explanatory biomechanical tool to infer joint level limitations from
a series of full-body movements, where traditional biomechanical methods have
seen limited success [14]. This motivates the telescoping inverted pendulum
model (section VI) which has been used to model different standing patterns in
clinical populations [13].

We first present related work in section II, before briefly introducing MLCI
in section III and our method for translating continuous dynamics into a
tabular Markov Decision Process that can be used with MLCI in section IV. We
then perform experimental analysis in section V, and finally show an example
with clinically motivated 4D telescoping inverted pendulum dynamics in section
VI.

##  II Related Work

Previous work on constraint inference can be split into two categories:
approaches that infer the most likely constraints but require a tabular state-
action space, and those that admit continuous dynamics but drop the maximum
likelihood feature. In the former category, in addition to Scobee and Sastry
[17], Vazquez-Chanlatte et al. [19] learn task specifications, which can be
thought of as a generalization of state-space constraints to include complex
multi-step behaviors. Unfortunately, neither of these approaches can be
applied directly to many real-world systems that are inherently continuous.

There are many proposed methods for identifying constraints in systems that
cannot be tabulated. Some use heuristics such as assuming that constrained
behaviors will have high intra-demonstration variance and low inter-
demonstration variance, or that a maintaining an end effector in the same
orientation throughout a demonstration suggests a constraint [12, 5]. [7]
presents a kinematics-based approach for learning constraints that affect how
a nominal policy is executed in different environments, but doesn’t assume an
objective function and therefore requires demonstrations to cover much of the
state space for the inference to be well-defined. [10] is specialized for
online constraint inference in the context of shared autonomy, where mis-
identified constraints can be corrected by the user. [2] provides a flexible
approach for learning state-space constraints by sampling from possible
trajectories with lower costs than the demonstrations.

Although the present work introduces error by estimating continuous dynamics
with a finite state-action space, it provides two key advantages over previous
methods that work with continuous dynamics. First, using the maximum entropy
framework allows us to model the demonstrators as soft-optimal with respect to
a reward function, which may be especially appropriate for human
demonstrators. Second, we are able to estimate and rank the most likely
constraints even in situations where demonstrations cover only a small portion
of the state space and do not provide enough information to fully resolve
ambiguity in possible constraints.

##  III Markov Decision Processes and Maximum Likelihood Constraint Inference

To perform maximum likelihood constraint inference (MLCI), we adapt the
approach developed by [17]. In this section, we present a brief overview of
the MLCI algorithm.

###  III-A Markov Decision Dynamics

MLCI is formulated as an operation on a tabular Markov Decision Process (MDP).
The MDP is a tuple of four elements:

  * •

A state space $\mathcal{S}$ to navigate. $\mathcal{S}$ is a finite set of
discrete state values:

$\mathcal{S}=\\{s^{1},s^{2},...,s^{M}\\}$

  * •

A set of actions $\mathcal{A}$ to decide between. $\mathcal{A}$ is a finite
set of discrete input values:

$\mathcal{A}=\\{a^{1},a^{2},\cdots,a^{W}\\}$

  * •

A transition kernel

$P_{a_{t-1}}(s_{t}|s_{t-1}):\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow[0,1]$

that determines the influence of $a_{t-1}\in\mathcal{A}$ on $s_{t}$. The
repeated action of this transition kernel generates a sequence of states over
a time horizon $t\in[0:T]$ given a sequence of action choices up to the
horizon. The couple of state sequence and action sequence is the trajectory
$\xi:[0:T]\rightarrow A\times S$ and the space of all possible trajectories is
$\Xi$.

  * •

An objective metric $R(\xi)$ that measures the quality of trajectories.

$R:\Xi\rightarrow\mathbb{R}$

This work focuses on deterministic dynamics, so the transition kernel will be
singleton distributions with zero probability of all next states except the
deterministic successor $s_{t}=f(a_{t-1},s_{t-1})$. That is, we focus on MDP’s
with transitions of the form:

$\displaystyle P_{a_{t-1}}(s_{t}|s_{t-1})=\begin{cases}1,\text{ if
}s_{t}=f(a_{t-1},s_{t-1})\\\ 0,\text{ otherwise}\end{cases}$ (1)

MLCI requires that $\mathcal{S}$ and $\mathcal{A}$ be finite sets, and we
refer to an MDP which satisfies this property as tabular.

###  III-B Maximum Entropy Likelihood on Trajectories

This work leverages the maximum entropy likelihood distribution advanced in
[21] and extended to constraint inference in [17]. This distribution’s
randomness reflects epistemological uncertainty in the estimated reward
function $R(\xi)$ of the demonstrator. Under this distribution, the likelihood
of a trajectory $\xi$ is defined on the deterministic MDP as:

$\displaystyle P(\xi)=\frac{e^{R(\xi)}}{Z}$ (2)

where $Z$ is the normalizing constant:

$Z=\sum_{\xi\in\Xi}e^{R(\xi)}$

This work investigates how dynamic agents avoid certain sets of states
$C\subset\mathcal{S}$. These constrained states further refine the choice
distribution by zeroing out illegal choices:

$\displaystyle P_{C}(\xi)=\begin{cases}\frac{e^{R(\xi)}}{Z_{C}},\text{ if
}s_{t}\notin C\quad\forall t\\\ 0,\text{ otherwise}\end{cases}$ (3)

Where the partition constant $Z$ decreases to $Z_{C}$ for this new
distribution that constrains out much of the previous support. Let
$\Xi_{C}\subset\Xi$ be the subset of trajectories that don’t violate the
constraint $C$:

$\Xi_{C}=\\{\xi|s_{t}\notin C\quad\forall t\\}$

So that $Z_{C}$ may be simply defined as:

$Z_{C}=\sum_{\xi\in\Xi_{C}}e^{R(\xi)}$

###  III-C Constraint Inference

The distribution in equation (3) describes the likelihood of observing any
demonstrated trajectory $\xi$ given a constraint set $C$. Given a set of
independent and identically distributed sample trajectories
$\tilde{\xi}=\\{\xi^{(j)}\text{ for }j\in[1,2,\cdots,N]\\}$, the likelihood of
observing this dataset is:

$\displaystyle P_{C}(\tilde{\xi})$ $\displaystyle=\Pi_{j=1}^{N}P(\xi^{(j)})$
(4)
$\displaystyle=\Pi_{j=1}^{N}\frac{e^{R(\xi^{(j)})}}{Z_{C}}\mathbb{I}[\xi^{(j)}\in\Xi_{C}]$
(5)
$\displaystyle=\frac{1}{Z_{C}}\Pi_{j=1}^{N}e^{R(\xi^{(j)})}\mathbb{I}[\xi^{(j)}\in\Xi_{C}]$
(6)

Adding a constraint to the model that helps explain the demonstrations will
increase this likelihood. Therefore, the most likely constraint $C^{*}$ is the
one that maximizes $P_{C^{*}}(\tilde{\xi})$. Note two properties that will aid
in finding $C^{*}$:

######  Remark 1.

The optimal constraint set $C^{*}$ must have all $\xi^{(j)}$ inside of its
corresponding $\Xi_{C^{*}}$. Otherwise its likelihood would be 0 – a lower
likelihood even than having no constraints at all. This would contradict its
being the optimum.

Therefore for any feasible candidate constraints, the indicator
$\mathbb{I}[\xi^{(j)}\in\Xi_{C}]$ will always evaluate to 1. With the zero-
case ruled out, the likelihood can be straightforwardly characterized by
factoring out the remaining $C$-dependent component:

######  Remark 2.

When comparing the likelihood amongst feasible constraint sets, they are only
re-scalings of the same dataset-determined constant
$\Pi_{j=1}^{N}e^{R(\xi^{(j)})}$ by $1/{Z_{C}}$. So the maximum likelihood
constraint set $C^{*}$ is simply whichever set $C$, amongst the feasible
constraint sets, has the smallest $Z_{C}$.

For every hypothesized constraint set $C_{i}$, $Z_{C_{i}}$ can be computed by
a Bellman backup or by forward simulation. The latter approach is favored by
[17] as it makes a direct parallel to the seminal Maximum Entropy IRL work
[21]. Let $C_{0}\subset\mathcal{S}$ be some baseline set of known constraints
(e.g. the empty set for the unconstrained case). The forward simulation relies
on the fact that $Z_{C_{i}}$ is proportional to
$P_{C^{0}}(\xi\in\Xi_{C_{i}})$.

$\displaystyle Z_{C_{i}}$ $\displaystyle=\sum_{\xi\in\Xi_{C_{i}}}e^{R(\xi)}$
$\displaystyle=Z_{C_{0}}\sum_{\xi\in\Xi_{C_{i}}}\frac{e^{R(\xi)}}{Z_{C_{0}}}$
$\displaystyle=Z_{C_{0}}P_{C^{0}}(\xi\in\Xi_{C_{i}})$

It can be calculated by forward simulating the state distribution under
$C_{0}$’s maximum entropy distribution and observing the probability that
trajectories violate the constraint $C_{i}$ up to time $t\in[0:T]$. Call that
quantity $\Phi_{i,t}$, then:

$\displaystyle Z_{C_{i}}$
$\displaystyle=Z_{C_{0}}P_{C^{0}}(\xi\in\Xi_{C_{i}})$
$\displaystyle=Z_{C_{0}}(1-P_{C^{0}}(\xi\notin\Xi_{C_{i}}))$
$\displaystyle=Z_{C_{0}}(1-\Phi_{i,T})$

Therefore, the most likely constraint minimizes $Z_{C_{i}}$, or equivalently,
maximizes $\Phi_{i,T}$. The quantity $\Phi_{i,T}$ will be useful in some of
our subsequent analysis.

##  IV Formulation of Approximate MDP

Given an arbitrary set of continuous dynamics of the form $\dot{x}=h(x,u)$, we
wish to generate an appropriate tabular state-action space that can be used
with the MLCI algorithm described in section III. We will illustrate this
process with a pendulum model that we return to for experimental analysis in
section V.

###  IV-A Running Example: Pendulum System

The pendulum model consists of a 2-dimensional state space (angle and angular
velocity). The 1-dimensional control input is the normalized torque applied at
the base of the pendulum:

$\ddot{\theta}=\frac{g}{l}\cdot sin(\theta)+u$ (7)

Where the gravitational constant $g$ and the length of the pendulum $l$ are
both assumed to be 1 for simplicity. The constraint hypothesis set is an
evenly spaced 10-by-10 grid of non-overlapping cells that cover the state
space, for a total of 100 possible constraints. (Note that any set of state
space regions is acceptable as the constraint hypothesis set, including
overlapping regions or ones that do not cover the whole state space, but it is
typically appropriate for them to be equally sized. This is because a larger
constraint region is able to ”explain away” more demonstrator sub-optimality
and is therefore likely to have a larger $\Phi_{T}$, making it difficult to
directly compare constraint regions of different sizes when choosing the most
likely one.) The demonstrator wants to arrive at a particular goal state at
the end of a $\tilde{T}$ = 5s period while minimizing the total squared torque
and avoiding the true constraint region, $K$:

$\begin{split}u^{*}=min\int_{0}^{\tilde{T}}u(\tilde{t})^{2}d\tilde{t}\\\
\mathrm{s.t.}(\theta(0),\dot{\theta}(0))=(\theta_{0},\dot{\theta}_{0})\\\
(\theta(\tilde{T}),\dot{\theta}(\tilde{T}))=(\theta_{T},\dot{\theta}_{T})\\\
(\theta(\tilde{t}),\dot{\theta}(\tilde{t}))\notin
K\quad\forall\tilde{t}\in[0,\tilde{T}]\end{split}$ (8)

Where we use $\tilde{t}$ to refer to continuous time and $t$ to refer to
discrete time steps.

###  IV-B Forming The Tabular State-Action Space

First, we choose appropriate bounds for each dimension of the state space and
control input, which can come from domain knowledge or observing the range of
values in the demonstrations. For the pendulum system, it is natural to bound
$\theta\in[0,2\pi]$, we select the velocity bound $\dot{\theta}\in[-6,6]$, and
the control input bound $u\in[-2,2]$ is chosen by observing that the controls
used by continuous trajectories optimizing the objective in equation (8)
rarely exceed this range. We then grid up the continuous state space by
dividing it into disjoint cells that completely cover the bounded area. A
reasonable default is to use equally sized boxes. For example, we can divide
the pendulum state space into 100 cells, 10 along each dimension, each
encompassing a $\frac{\pi}{5}$ rad angle width and a 1.2 rad/s angular
velocity range. The set of these cells is $\mathcal{S}$. Similarly, the range
of possible control inputs is divided into discrete points to give
$\mathcal{A}$. We use $s$ and $a$ to label the discrete states and actions,
respectively. $x_{s_{t}}$ is the value of the continuous state at the center
point of state cell $s_{t}$, while $u_{a_{t}}$ is the value of the control
input associated with discrete action choice $a_{t}$.

###  IV-C Tabular MDP Transition Kernel and Objective

To complete the tabular MDP representation, we need to determine the
transition and reward associated with each
$(s\in\mathcal{S},a\in\mathcal{A}$). For “gridworld” environments frequently
used in inverse reinforcement learning, the agent is allowed to transition to
any adjacent cell. However, for arbitrary continuous dynamics, this behavior
may result in trajectories that bear little resemblance to what is possible
under the true dynamics. For example, consider that in the pendulum system,
allowing a transition from $\theta\in[0,\frac{\pi}{5}]$ to
$\theta\in[\frac{\pi}{5},\frac{2\pi}{5}]$ is nonsensical if the current
velocity is a large negative value, regardless of the control input.

To resolve this problem, we select a constant time interval $\Delta t$ that
represents the amount of time that passes between state transitions in the
tabular model. For each discrete $(s\in\mathcal{S},a\in\mathcal{A})$, we use
an ODE solver to determine the trajectory that would result from starting at
the center of state cell $s$, $x_{s}$, and applying a constant control input
of $u_{a}$ for $\Delta t$ time. We can then determine which state cell the
agent would land in at the end of this trajectory segment, which becomes the
successor state $s^{\prime}$. While this is sufficient for determining
appropriate discrete transitions, the start and successor cells alone do not
tell us which state-based constraints may have been violated while taking a
particular transition. Therefore, we also keep track of which hypothesized
constraints would be violated while executing the continuous trajectory
underlying the discrete transition. This ensures that the agent isn’t allowed
to “warp through” constraints even when $s$ and $s^{\prime}$ are not adjacent
cells.

Finally, we assume that the ground-truth reward for an entire trajectory can
be expressed as $R=\int_{0}^{\tilde{T}}r(x(\tilde{t}),u(\tilde{t}))d\tilde{t}$
for some function $r(x,u)$ of the continuous state and control input. We
estimate the tabular reward function as
$\hat{R}=\sum_{t=0}^{T}r(x_{s_{t}},u_{a_{t}})\cdot\Delta t$, where the
sequence $(s_{[0:T]},a_{[0:T]})$ is the sequence of discrete state-action
pairs over the course of a trajectory on the tabular MDP.

It is worth noting that trajectories allowable under the tabular MDP described
above are not necessarily feasible or safe under the true continuous dynamics.
For example, starting from different points within the same cell might result
in slightly different constraint violations, while we only track violations
that result from starting in the center of each cell. This is acceptable for
our application because we are trying to obtain estimates of general behavior
that enable reasonable likelihood-based constraint inference. Similarly, there
is no well-defined mapping from a particular continuous trajectory to a
feasible discrete state-action sequence under the approximate tabular
dynamics. Since we only handle state-based constraints, it is sufficient to
determine which possible constraints a demonstration violates without trying
to construct a discrete version of the trajectory. This can be done by
sampling points along the trajectory to determine which constraint regions it
passes though.

The primary hyperparameters that determine the final tabular MDP are the
number of cells to use for each state dimension, the number of actions, and
the transition time step $\Delta t$. These parameters can be tuned using
domain knowledge or by running simulated experiments with known constraints to
determine which model obtains the best performance. An example of these
experiments and the resulting constraint inference performance for the
pendulum system is described in the following section. Once an appropriate
model has been selected, it can be used with any combination of reward
function and demonstration set. Additionally, if the objective of the
demonstrators is known in advance, MLCI can be performed on the appropriately
initialized discrete MDP as a pre-computation step, and constraints can be
inferred online with very little additional computation.

##  V Analysis On Pendulum System

After following the procedure outlined in section IV for the pendulum system,
we now have a tabular MDP representation that can be used with MLCI as
described in section III. We next turn to analyzing the behavior of this
approximate MDP. For our experiments, we tested two possible ground-truth
constraints: $C_{1}$ prohibits $\theta\in[\pi,\frac{6}{5}\pi]$ while
$\dot{\theta}\in[0,1.2]$, and $C_{2}$ prohibits
$\theta\in[\frac{8}{5}\pi,\frac{9}{5}\pi]$ while $\dot{\theta}\in[0,1.2]$ Both
ground-truth constraints are aligned with the constraint hypothesis set. The
constraint hypothesis space is illustrated in Fig. 2. For each ground truth
constraint, we randomly sampled 100 pairs of start and end states from
$\Omega$ (defined in equation (10) below) for agents to satisfy while
optimizing the objective in equation (8). Some of these start-end state pairs
were ill-posed since the pendulum could not reach across them in the fixed 5
second time horizon provided. After removing these configurations, the set of
demonstrations was reduced to $N=65$ trajectories.

Figure 2: We test our constraint inference on a fixed-base pendulum model. The
constraint hypothesis space evenly divides the state space into 100 cells, 10
along the angle axis and 10 along angular velocity. The two constraints used
in our experiments, $C_{1}$ and $C_{2}$, are shown here in different shades of
red. The constraints cover different angle regions but the same angular
velocity.

###  V-A Accuracy Of Tabular MDP Dynamics

We first examine how accurately the tabular MDP recovers the true continuous
dynamics under goal-directed behavior induced by the objective function. For
each ground-truth constraint and random start-goal pair, we initialized the
MDP while incorporating the true constraint into the MDP dynamics (i.e.,
actions that would result in violating the true constraint were not allowed).
We then performed a Bellman backup to determine the distribution of soft
optimal policies on the tabular MDP. Intuitively, if the MDP perfectly
describes the true continuous dynamics, we expect that running a simulation
with the ground-truth dynamics while taking the sequence of actions determined
by one of these policies will cause the agent to land exactly at the goal
state. Following this intuition, we sampled and executed a random policy from
each MDP and measured the normalized Euclidean distance between the final
state and goal state. As shown in Fig. 3, increasing the number of state cells
reduces the “round-off” error associated with each discrete state transition
and results in a final state that is closer to the intended goal. Since the
objective function specifies a fixed time horizon, increasing $\Delta t$
decreases the number of transitions over the course of a trajectory and
therefore reduces final state error as well.

Figure 3: Dividing the state space into larger numbers of smaller cells in the
MDP allows for a more accurate approximation of the continuous dynamics when
executing optimal policies over 5s. A smaller $\Delta t$ means that a larger
number of discrete transitions are taken over the same time horizon, leading
to decreased accuracy, especially for a coarser discretized state space. Pink
circles: 100 state cells, green crosses: 400, yellow squares: 900, blue
triangles: 1600. Note that there is decreasing benefit to further refining the
state space grid above 900 cells.

###  V-B Generating Simulated Expert Demonstrations

To understand the accuracy of constraint inference with the tabular MDP, we
first need expert demonstrations that follow the ground-truth dynamics. For
each ground-truth constraint, 100 random pairs of states were sampled to serve
as the start and goal points for independent demonstrations. These expert
continuous demonstrations were synthesized using a second-order descent method
with simulation time step ($\Delta t=0.01sec$), much finer than the $\Delta t$
used in the tabular MDP. The demonstrations are optimized using a Gauss-
Newton-style descent method known as Iterative Linear-Quadratic Regulators (or
iLQR) [6]. The optimization is halted after ten iterations. For each start-
goal pair, the best-of-three optimizations is picked (each with randomly
sampled controls initialization) to reject optimizations that get stuck in
local minima. The optimizations that could not succeed in reaching their goal
were filtered out from the dataset, reducing the dataset size to $N=65$.

The state constraints are blocked out as rectangular polytope constraints in
the continuous state-space. They are enforced using an interior-point method
that supersedes any controls (as in [3]) that would reach the constrained
states. This backwards-reachable set that forms the barrier-certificate [15]
is computed via a Hamilton-Jacobi Isaacs Partial Differential Equation [11].
For a continuous dynamic $\dot{x}=h(x,u)$, the robust backwards reachable set
$R$ of the constraint region $K\subset\mathcal{X}=\mathbb{R}^{m}$ can be
computed as the sub-zero level set of:

$\displaystyle\frac{\partial
V}{\partial\tilde{t}}(x,\tilde{t})=-\min\\{0,\max_{u}\nabla
V(x,\tilde{t})^{T}h(x,u)\\}$ (9)

where $V$ is initialized to the signed distance from $K$:

$V(x,0)=dist_{K}(x)$

Let $\Omega_{\tilde{t}}$ be the complement of this backwards reachable set:

$\displaystyle\Omega_{\tilde{t}}=\\{x|V(x,\tilde{t})\geq 0\\}$ (10)

As the complement of the reachable set, $\Omega_{\tilde{t}}$ is the set from
which there is a way to avoid the keepout set $K$. Since there exists an
avoidant strategy, this $\Omega_{\tilde{t}}$ is a control-invariant set. So
long as the system is initialized within $\Omega_{\tilde{t}}$ it is possible
to remain safe. Furthermore, any controls can be taken up to crossing the
border from $\Omega_{\tilde{t}}$ into $R$. At this point, the maximally safe
action $u(x,\tilde{t})^{*}=\arg\max_{u}\nabla V(x,\tilde{t})^{T}h(x,u)$ must
be taken. This is the safety strategy advanced in [3].

This safety strategy ensures the system will stay on the interior of the
feasible region. Due to intervening only when absolutely necessary (i.e. when
crossing into $R$), this intervention is also the _least restrictive_. It will
not eliminate any trajectories that weren’t already infeasible. Therefore the
set of feasible solutions remains unchanged after instituting these dynamics.
The optimal trajectory of the non-intervened dynamics will be the same as the
optimal trajectory on the intervened dynamics.

This constraint-enforcing switching control is non-differentiable, so
derivative-based optimizations on the controls cannot be used. Fortunately,
new relaxations of switched dynamics [20] can substitute a relaxed problem
whose solutions will converge to the true unrelaxed solution as the relaxation
is tightened.

###  V-C Constraint Inference Performance

Accurate constraint inference relies on a close match between expert
demonstrations and soft-optimal trajectories on the tabular MDP that
incorporates the ground-truth constraint. For the purposes of constraint
inference, two trajectories are equivalent if they violate the same
constraints in the constraint hypothesis set. Therefore, we next examine the
difference between the expected constraint violation under the tabular MDP and
the actual constraints violated by $N=65$ independent continuous
demonstrations. Over all of the possible constraints, this difference can be
expressed as

$\sum_{i=1}^{100}|\Phi_{i,T}-\frac{1}{N}\sum_{j=1}^{N}D^{(j)}_{i}|$ (11)

Where $D^{(j)}_{i}$ is an indicator for whether demonstration $j$ violates
constraint $C_{i}$. If the approximate MDP perfectly tracks the true
constraint violation distribution and demonstrations are distributed according
to soft optimality, we expect the quantity in equation (11) to go
asymptotically to 0 as the number of demonstrations increases. Results for
different model hyperparameters and a single demonstration (averaged over 65
trials and the two alternative ground-truth constraints) are shown in Fig. 4.
Increasing the number of state space grid cells from 100 to 400 lowers
constraint violation error, but increasing the number of states in the tabular
MDP beyond this point does not have much effect. This suggests that the
greater accuracy of the approximate MDP for larger numbers of discrete states
does not necessarily translate into improved constraint inference. Error is
stable across different values of the discrete time interval $\Delta t$.

Figure 4: Accuracy in estimating constraint violation for random combinations
of start and goal is relatively constant over various hyperparameter values,
although there is a small benefit for increasing the state space grid size
from 100 cells (pink circles) to 400 cells (green crosses). The metric used
here is quantified in equation (11). Pink circles: 100 state cells, green
crosses: 400, yellow squares: 900, blue triangles: 1600.

We see a very similar trend when examining the performance of constraint
inference across MDP’s generated with different hyperparameters, as can be
seen in Fig. 5. After choosing an appropriate $\Delta t$, tabular MDP’s with
100 to 1600 states are able to successfully identify the true constraint as
one of the top-5 likeliest constraints after 9 demonstrations. Increasing the
number of states to at least 400 stabilizes performance across different
choices of $\Delta t$. Even though the approximate MDP’s do not capture the
true continuous dynamics with high fidelity, especially for the coarsest
state-space grid, constraint inference still works well and is robust to a
range of hyperparameters.

Figure 5: The true constraint is ranked as the one of the most likely
possibilities after just a few demonstrations across many hyperparameter
choices. For the 10x10 state space grid (pink circles), $\Delta t=0.2$ gives
poor performance because it often isn’t possible for the agent to reach a new
cell within this time frame, so most transitions result in the tabular agent
erroneously staying in the same location. Overall, there is a small benefit
for increasing from 100 state cells to 400 cells (green crosses). A 1600 state
cell version performs equivalently to 900 cells (yellow squares) but is
omitted for clarity. Pink circles: 100 state cells, green crosses: 400, yellow
squares: 900.

In addition to these average trends, we can qualitatively examine the
approximation quality by sampling a trajectory from the discrete MDP and
comparing it to the original continuous demonstration. An example of this for
a single trial is shown in Fig. 6.

Figure 6: We can directly compare the approximated MDP to individual
demonstrated trajectories by sampling a discrete trajectory from the MDP when
the true constraint is known. From left to right, a trajectory sampled from an
MDP with 100, 900, and 1600 states, respectively. The blue line is the
demonstration, which is attempting to get from the start (cyan box) to the
goal (green box) while avoiding the true constraint (red box). The states in
the discrete trajectory are shown with shaded pink boxes. Possible constraints
violated by the demonstration are shown with blue diagonal lines going from
top left to bottom right, while possible constraints violated by the discrete
trajectory are shown with pink diagonal lines going from bottom left to top
right. For the finer grid state spaces in the center and right panels, the
states visited by the discrete trajectory are smaller than the constraint
regions. For this example, the trajectory from the 900-state MDP happens to
most closely match the demonstration.

For all of the analyses described above, we also varied the number of discrete
actions in the approximate MDP but found that this made little difference to
any of the measures we examined. A larger number of actions allows the
discrete agent more possible routes to the goal, but it may be that these
routes do not change constraint violation behavior in expectation across the
soft-optimal policy distribution. Fig. 3 through Fig. 6 show results using 9
actions evenly spaced from $u=-2$ to $u=2$.

###  V-D Confidence In Found Constraints

In addition to identifying the most likely constraints influencing agent
behavior, it is desirable to calculate the probability of there being a
constraint at all. First, consider the simple case where we assume that there
is at most one constraint, and if there is one, it is the most likely one
identified via MLCI. Let $C$ be the event that this is truly a constraint, and
$A_{N}$ be the event that N independent trajectories do not violate this
constraint. We would like to calculate $P(C|A_{N})$. We know that
$P(A_{N}|C)=0$ since no demonstrations may violate a constraint, and that
$P(\bar{A_{1}}|\bar{C})=\Phi_{T}$ (i.e. the probability of a demonstration not
violating this constraint by coincidence, even if the agent isn’t really
subject to it), which we obtain from the MLCI algorithm. We can therefore use
Bayes’ Rule to obtain the following formula:

$P(C|A_{N})=\frac{P(C)}{P(C)-(1-\Phi_{T})^{N}(1-P(C))}$ (12)

where $P(C)$ is a prior on the probability of the constraint being present.
This simple formula introduces no additional approximation error beyond what
is already present in the model under the assumptions described above, and
presents an important advantage of the MLCI approach to constraint inference
over previous approaches that cannot provide confidence estimates of found
constraints. Unfortunately, relaxing the assumptions on possible constraints
and calculating probabilities of all possible constraints quickly becomes
computationally intractable. Providing estimates of these probabilities is
left for future work.

##  VI Potential Application: Sit-to-Stand and Lower Back Pain

The robustness to hyperparameters selection, low number of required
demonstrations, and ability to provide a confidence interval on the identified
constraints supports the use of the MLCI approach to identify patient-specific
impairments from observed motion. One potential application is in the analysis
of individuals with Low back pain (LBP).

LBP affects 70-90% of adults during their lifetime and can be extremely
debilitating [18]. However, it is often difficult to determine the source of
the pain and therefore prescribe an appropriate treatment. Disorders of the
lower spine, hip, and pelvic region can all cause LBP [16]. Treating the wrong
problem may result in an unnecessary surgery that doesn’t resolve the
patient’s LBP. When a treatment plan that addresses the physical cause of the
pain can’t be identified, patients may be prescribed opioids for chronic pain
management, even though these are ineffective and can lead to abuse and
addiction [9]. Therefore, there is a pressing clinical need to develop better
methods for understanding the source of LBP.

There is a recent body of literature suggesting that LBP may be linked to
irregularities in movement patterns. For example, inappropriate amounts of
pelvic movement during various motions appears to contribute to LBP [8]. This
pelvic movement may be compensatory for a limited range of motion in other
joints - in other words, constraints on the achievable joint angles. We would
expect the resulting movement patterns to avoid regions of the biomechanical
state space associated with pain. Identifying both physical and pain-related
constraints on movement could therefore lead us to a better understanding of
the underlying cause of the LBP. For this reason, we would like to infer the
most likely constraints a person is acting under when observing their
movements. A particularly promising movement pattern for demonstrations is
completing a sit-to-stand trajectory, which exerts significant strain on
several joints implicated in LBP [4]. A telescoping inverted pendulum system
has been used to model this movement, which reduces the problem to 4
dimensions while allowing for clinically relevant discovery [13].

###  VI-A Constraint Inference on Telescoping Inverted Pendulum

Following the above motivation, we next demonstrate successful constraint
inference on a telescoping inverted pendulum (TIP) model. The dynamics for
this model are as follows:

$\displaystyle\ddot{\theta}$ $\displaystyle=\frac{g}{l}\cdot
sin(\theta)+u_{1}$ $\displaystyle\ddot{l}$ $\displaystyle=u_{2}$

These dynamics omit the cross-coupling term between angular acceleration and
linear velocity for simplicity. For this experiment, we chose the goal set as
the set of all states within a certain range of pendulum length and angle,
leaving velocity as a free parameter. The objective is to reach the goal set
at $\tilde{T}$ = 5s while minimizing $||u||_{2}^{2}$. The constraint
hypothesis space is a 10x10 evenly spaced grid along the angle and length
dimensions, so that if a particular (angle, length) combination is
constrained, the agent is not allowed to enter that combination at any
velocity. We generated 5 demonstrations with random start and goal states
following the same procedure as in section V-B. We then formulated a tabular
MDP with 2500 states (10 cells each for angle and length, and 5 cells each for
angular and linear velocity) and 15 actions (5 discrete torque options and 3
discrete linear force options) and performed constraint inference on the
demonstrations. The ground truth constraint and top 2 likeliest inferred
constraints are shown in Fig. 7. Despite the coarseness of the tabular state-
action space and a mismatch between the constraint hypothesis space and the
true constraint region, MLCI correctly identifies the ground truth constraint
and takes about 5 minutes with no optimization effort on a single CPU core. If
the start and goal states of demonstrations are known in advance, as is likely
to be the case in a clinical test, this computation can be done ahead of time
and inferring constraints after observing the actual demonstration
trajectories is virtually instantaneous.

Figure 7: Ground truth and inferred constraints for a 4-dimensional telescopic
inverted pendulum (TIP) model. The constraint hypothesis space is 100 evenly
spaced cells in length and angle. The ground truth constraint, which does not
evenly align with the constraint hypothesis set, is shown in red. After 5
demonstrations, the top 2 most likely constraints, shown as cells with dark
black outlines, coincide very well with the true constraint. 2 demonstrations
of those used in inference, along with the constraints they violated (shaded
cells), are shown in blue and purple. The tabular MDP must keep track of the
linear and angular velocity dimensions as well as the position dimensions
shown here.

##  VII Conclusion

We have presented methodology for forming a tabular MDP approximation of
continuous dynamics which can be used for maximum likelihood constraint
inference. Although the approximation introduces some error into the
estimation, constraint inference works well with pendulum dynamics over a
range of hyperparameters, including a small discrete state space. The present
approach allows for ranking possible constraints by their likelihood, which is
especially useful in applications with significant uncertainty, and uses the
maximum entropy framework, which may be an especially good fit for human
demonstrators, who tend to act sub-optimally. Future work should characterize
the kinds of dynamics for which this approach works well and whether
techniques such as variable grid size may allow for higher accuracy and
increased computational efficiency.

## References

  * Balagué et al. [2012] Federico Balagué, Anne F Mannion, Ferran Pellisé, and Christine Cedraschi.  Non-specific low back pain.  _The Lancet_ , 379(9814):482–491, February 2012\.  ISSN 0140-6736.  doi: 10.1016/S0140-6736(11)60610-7.  URL https://www.sciencedirect.com/science/article/pii/S0140673611606107.
  * Chou et al. [2020] G. Chou, N. Ozay, and D. Berenson.  Learning Constraints From Locally-Optimal Demonstrations Under Cost Function Uncertainty.  _IEEE Robotics and Automation Letters_ , 5(2):3682–3690, April 2020.  ISSN 2377-3766.  doi: 10.1109/LRA.2020.2974427.  Conference Name: IEEE Robotics and Automation Letters.
  * Hoffmann and Tomlin [2008] G. M. Hoffmann and C. J. Tomlin.  Decentralized cooperative collision avoidance for acceleration constrained vehicles.  In _2008 47th IEEE Conference on Decision and Control_ , pages 4357–4363, December 2008.  doi: 10.1109/CDC.2008.4739434.  ISSN: 0191-2216.
  * Hughes et al. [1994] M. A. Hughes, D. K. Weiner, M. L. Schenkman, R. M. Long, and S. A. Studenski.  Chair rise strategies in the elderly.  _Clinical Biomechanics_ , 9(3):187–192, May 1994\.  ISSN 0268-0033.  doi: 10.1016/0268-0033(94)90020-5.  URL http://www.sciencedirect.com/science/article/pii/0268003394900205.
  * Li and Berenson [2017] Changshuo Li and Dmitry Berenson.  Learning Object Orientation Constraints and Guiding Constraints for Narrow Passages from One Demonstration.  In Dana Kulić, Yoshihiko Nakamura, Oussama Khatib, and Gentiane Venture, editors, _2016 International Symposium on Experimental Robotics_ , Springer Proceedings in Advanced Robotics, pages 197–210, Cham, 2017. Springer International Publishing.  ISBN 978-3-319-50115-4.  doi: 10.1007/978-3-319-50115-4˙18.
  * Li and Emanuel [2004] Weiwei Li and Todorov Emanuel.  ITERATIVE LINEAR QUADRATIC REGULATOR DESIGN FOR NONLINEAR BIOLOGICAL MOVEMENT SYSTEMS:.  In _Proceedings of the First International Conference on Informatics in Control, Automation and Robotics_ , pages 222–229, Setúbal, Portugal, 2004. SciTePress - Science and and Technology Publications.  ISBN 978-972-8865-12-2.  doi: 10.5220/0001143902220229.  URL http://www.scitepress.org/DigitalLibrary/Link.aspx?doi=10.5220/0001143902220229.
  * Lin et al. [2015] H. Lin, M. Howard, and S. Vijayakumar.  Learning null space projections.  In _2015 IEEE International Conference on Robotics and Automation (ICRA)_ , pages 2613–2619, May 2015.  doi: 10.1109/ICRA.2015.7139551.  ISSN: 1050-4729.
  * M et al. [2015] Sadeghisani M, Manshadi Fd, Kalantari Kk, Rahimi A, Namnik N, Karimi Mt, and Oskouei Ae.  Correlation between Hip Rotation Range-of-Motion Impairment and Low Back Pain. A Literature Review.  _Ortopedia, Traumatologia, Rehabilitacja_ , 17(5):455–462, October 2015.  ISSN 1509-3492, 2084-4336.  doi: 10.5604/15093492.1186813.  URL https://europepmc.org/article/med/26751745.
  * Martell et al. [2007] Bridget A. Martell, Patrick G. O’Connor, Robert D. Kerns, William C. Becker, Knashawn H. Morales, Thomas R. Kosten, and David A. Fiellin.  Systematic Review: Opioid Treatment for Chronic Back Pain: Prevalence, Efficacy, and Association with Addiction.  _Annals of Internal Medicine_ , 146(2):116, January 2007.  ISSN 0003-4819.  doi: 10.7326/0003-4819-146-2-200701160-00006.  URL http://annals.org/article.aspx?doi=10.7326/0003-4819-146-2-200701160-00006.
  * Mehr et al. [2016] N. Mehr, R. Horowitz, and A. D. Dragan.  Inferring and assisting with constraints in shared autonomy.  In _2016 IEEE 55th Conference on Decision and Control (CDC)_ , pages 6689–6696, December 2016.  doi: 10.1109/CDC.2016.7799299.
  * Mitchell [2007] Ian Mitchell.  A Toolbox of Level Set Methods.  _UBC Department of Computer Science Technical Report TR-2007-11_ , page 31, 2007.
  * Pais et al. [2013] Lucia Pais, Keisuke Umezawa, Yoshihiko Nakamura, and Aude Billard.  Learning Robot Skills Through Motion Segmentation and Constraints Extraction.  _HRI Workshop on Collaborative Manipulation_ , page 5, 2013.
  * Papa and Cappozzo [1999] Elisabetta Papa and Aurelio Cappozzo.  A telescopic inverted-pendulum model of the musculo-skeletal system and its use for the analysis of the sit-to-stand motor task.  _Journal of Biomechanics_ , 32(11):1205–1212, November 1999.  ISSN 0021-9290.  doi: 10.1016/S0021-9290(99)00103-7.  URL http://www.sciencedirect.com/science/article/pii/S0021929099001037.
  * Papi et al. [2018] Enrica Papi, Anthony M. J. Bull, and Alison H. McGregor.  Is there evidence to use kinematic/kinetic measures clinically in low back pain patients? A systematic review.  _Clinical Biomechanics_ , 55:53–64, June 2018.  ISSN 0268-0033.  doi: 10.1016/j.clinbiomech.2018.04.006.  URL https://www.sciencedirect.com/science/article/pii/S0268003318303152.
  * Prajna and Jadbabaie [2004] Stephen Prajna and Ali Jadbabaie.  Safety Verification of Hybrid Systems Using Barrier Certificates.  In Rajeev Alur and George J. Pappas, editors, _Hybrid Systems: Computation and Control_ , Lecture Notes in Computer Science, pages 477–492, Berlin, Heidelberg, 2004. Springer.  ISBN 978-3-540-24743-2.  doi: 10.1007/978-3-540-24743-2˙32.
  * Prather and Dillen [2019] Heidi Prather and Linda van Dillen.  Links between the Hip and the Lumbar Spine (Hip Spine Syndrome) as they Relate to Clinical Decision Making for Patients with Lumbopelvic Pain.  _PM &R_, 11(S1):S64–S72, 2019.  ISSN 1934-1563.  doi: 10.1002/pmrj.12187.  URL https://onlinelibrary.wiley.com/doi/abs/10.1002/pmrj.12187.  _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1002/pmrj.12187.
  * Scobee and Sastry [2019] Dexter R. R. Scobee and S. Shankar Sastry.  Maximum Likelihood Constraint Inference for Inverse Reinforcement Learning.  _arXiv:1909.05477 [cs, eess, stat]_ , September 2019.  URL http://arxiv.org/abs/1909.05477.  arXiv: 1909.05477.
  * Thiruganasambandamoorthy et al. [2014] Venkatesh Thiruganasambandamoorthy, Ekaterina Turko, Dominique Ansell, Aparna Vaidyanathan, George A. Wells, and Ian G. Stiell.  Risk Factors for Serious Underlying Pathology in Adult Emergency Department Nontraumatic Low Back Pain Patients.  _The Journal of Emergency Medicine_ , 47(1):1–11, July 2014.  ISSN 0736-4679.  doi: 10.1016/j.jemermed.2013.08.140.  URL http://www.sciencedirect.com/science/article/pii/S0736467914001553.
  * Vazquez-Chanlatte et al. [2018] Marcell Vazquez-Chanlatte, Susmit Jha, Ashish Tiwari, Mark K. Ho, and Sanjit Seshia.  Learning Task Specifications from Demonstrations.  _Advances in Neural Information Processing Systems_ , 31:5367–5377, 2018.  URL https://proceedings.neurips.cc/paper/2018/hash/74934548253bcab8490ebd74afed7031-Abstract.html.
  * Westenbroek et al. [2018] T. Westenbroek, H. Gonzalez, and S. S. Sastry.  A New Solution Concept and Family of Relaxations for Hybrid Dynamical Systems.  In _2018 IEEE Conference on Decision and Control (CDC)_ , pages 743–750, December 2018.  doi: 10.1109/CDC.2018.8619050.  ISSN: 2576-2370.
  * Ziebart et al. [2010] Brian D Ziebart, J Andrew Bagnell, and Anind K Dey.  Modeling Interaction via the Principle of Maximum Causal Entropy.  In _International Conference on Machine Learning_ , page 8, 2010\.


# F-SIOL-310: A Robotic Dataset and Benchmark for Few-Shot Incremental Object
Learning

Ali Ayub1 and Alan R. Wagner2 1Department of Electrical Engineering, The
Pennsylvania State University, State College, PA 16802, USA
<EMAIL_ADDRESS>of Aerospace Engineering, The Pennsylvania State
University, State College, PA 16802, USA<EMAIL_ADDRESS>
###### Abstract

Deep learning has achieved remarkable success in object recognition tasks
through the availability of large scale datasets like ImageNet. However, deep
learning systems suffer from catastrophic forgetting when learning
incrementally without replaying old data. For real-world applications, robots
also need to incrementally learn new objects. Further, since robots have
limited human assistance available, they must learn from only a few examples.
However, very few object recognition datasets and benchmarks exist to test
incremental learning capability for robotic vision. Further, there is no
dataset or benchmark specifically designed for incremental object learning
from a few examples. To fill this gap, we present a new dataset termed
F-SIOL-310 (Few-Shot Incremental Object Learning) which is specifically
captured for testing few-shot incremental object learning capability for
robotic vision. We also provide benchmarks and evaluations of 8 incremental
learning algorithms on F-SIOL-310 for future comparisons. Our results
demonstrate that the few-shot incremental object learning problem for robotic
vision is far from being solved.

##  I INTRODUCTION

Humans have the ability to learn new concepts continually over their lifetime
from only a few examples. With robots increasingly becoming an integral part
of the society for a variety of different roles, such as household robots [1],
they should also learn about new concepts continually to adapt to their
dynamic environments. Further, since robots have limited human assistance in
real-world environments, they must learn from only a few examples [2]. The
ultimate goal of this paper is to develop an object recognition dataset for
continual learning with a few examples on robots.

One of the reasons for deep learning’s remarkable success in object
recognition is the availability of large-scale object datasets like ImageNet
[3] and CIFAR-100 [4]. These datasets, however, have been designed for batch
learning in which a model is trained on the data of all the classes in one
batch and then evaluated on a separate test set. Further, these datasets
generally contain objects in ideal conditions, such as no blurriness or
transparency etc. In contrast, robots generally do not have access to a large
set of idealistic, labeled images in the real world. Robots mostly have to
continually acquire data through their own camera autonomously in different
conditions which often leads to less than perfect images. Further, to get
labels for the captured object images, robots have to ask for human
assistance. Human teachers are usually unwilling to answer enormous numbers of
questions, hence robots may only have a few labeled object images to learn
from. Thus, large scale datasets like ImageNet and CIFAR-100 are not suitable
for evaluating incremental learning approaches for robot vision.

Most state-of-the-art (SOTA) incremental learning approaches [5, 6, 7, 8],
however, have been tested on less complex datasets like MNIST [9] or CIFAR-100
[4]. Further, most current incremental learning approaches [5, 6, 7, 8] are
not designed for few-shot incremental learning (FSIL), which is a challenging,
but realistic, learning problem for robotics applications that was first
considered in [10]. FSIL is necessary for real-world robotics applications
like household cleaning robots or industrial packaging robots or anywhere a
robot may have to quickly learn new objects from a few examples provided by
humans. For FSIL, an incremental learning model is required to learn object
classes incrementally but with only a few (usually 5 or 10) training examples
per class. CBCL [10] was specifically designed for FSIL, however, it was
primary tested on the CIFAR-100 dataset since no dataset had been specifically
created for FSIL.

Figure 1: Baxter robot used to capture the F-SIOL-310 object dataset. Datasets |  Classes |  Objects |  Acquisition |  Train/Test Difference |  FSIL Benchmarks |  Scale Variation |  Transparency
---|---|---|---|---|---|---|---
COIL-100 [11] |  - |  100 |  turntable |  No |  No |  No |  No
NORB [12] |  5 |  25 |  turntable |  No |  No |  No |  No
BigBIRD [13] |  - |  100 |  turntable |  No |  No |  No |  No
ROD [14] |  51 |  300 |  turntable |  No |  No |  No |  No
CORe-50 [15] |  10 |  50 |  hand held |  No |  No |  No |  No
ARID [16] |  51 |  153 |  robot |  No |  No |  Yes |  No
OpenLORIS [17] |  19 |  69 |  robot |  No |  No |  Yes |  No
F-SIOL-310 |  22 |  310 |  robot |  Yes |  Yes |  Yes |  Yes
TABLE I: Comparison of F-SIOL-310 with other robotics object recognition
datasets.

In order to close the gap between existing robot vision research and real-
world applications, we propose a new RGB dataset termed F-SIOL-310. A Baxter
robot (manufactured by Rethink robotics, see Figure 1) was used to actively
capture household objects on a table. The dataset is specifically designed for
FSIL with only a small set of training images and a larger set of test images
per object category captured by the robot using its own camera and it
considers various other robot vision challenges as well, such as different
object sizes, object transparency and a clear distinction between objects in
the train and test sets. We provide extensive evaluations of the state-of-the-
art incremental learning approaches using different backbone neural network
architectures pre-trained on different datastets, on F-SIOL-310 for FSIL. The
complete dataset is available at https://tinyurl.com/jer56jhr. This paper
contributes:

  1. 1.

A new RGB object dataset termed F-SIOL-310. The dataset is collected using a
diverse set of 310 household objects of various sizes. To the best of our
knowledge, no other dataset has been developed for the FSIL problem which
contains different objects belonging to a category in the train and test sets.

  2. 2.

Benchmarks for evaluating FSIL with 5-shot and 10-shot settings using
different test sets.

  3. 3.

An extensive set of evaluations of current SOTA incremental learning
approaches on F-SIOL-310 with 5-shot and 10-shot incremental learning which
show that the current approaches are far from being accurate enough for
application in real world robotics problems.

The remainder of the paper is organized as follows: Section II reviews the
related work including incremental learning approaches and other robotics
datasets for object recognition. Section III describes F-SIOL-310 in detail.
Section IV presents benchmarks and empirical evaluations of SOTA approaches on
F-SIOL-310 for FSIL. Finally, Section V offers conclusions and directions for
future research.

##  II RELATED WORK

###  II-A Incremental Learning Techniques

As discussed in [17], a true incremental learning system should have the
following characteristics: 1) it can learn new knowledge and patterns from new
data; 2) it can avoid catastrophic forgetting and remember past knowledge; 3)
it can generalize in response to new incoming data; 4) it should have few-shot
learning capability so that it can learn from limited data; 5) it can learn
from an infinite stream of data while keeping the memory footprint and
learning time from growing drastically. She et al. [17] examined the first
three capabilities in their work. In contrast, this paper evaluates current
incremental learning techniques on all five characteristics. These
characteristics require incremental learning algorithms to learn new object
classes from limited data and to perform competitively on new and old object
classes while keeping the system memory from growing substantially.

Figure 2: Examples of training and test images in CORe-50 (a) and OpenLORIS
(b) datasets. Note that same objects are used in train and test sets with
similar backgrounds.

Creating highly accurate incremental object learning classifiers, however, is
made more difficult by the catastrophic forgetting problem: the model (often a
neural network) forgets the previously learned classes when learning new
classes resulting in a significant decrease in classification accuracy. Most
existing class-incremental learning methods avoid this problem by storing a
portion of the training samples from previous classes and retraining the model
(typically a neural network) on a mixture of the stored data and new data [7,
18]. However, storing real samples quickly exhausts memory capacity and limits
performance for real-world applications. To avoid this problem, some
incremental learning approaches use regularization techniques [8, 5]. Although
these approaches solve the memory storage issues, their performance is
inferior to approaches that store old class data. Another genre of incremental
learning approaches use generative memory such as autoencoders or GANs
(Generative Adversarial Networks) [19, 6, 20]. These approaches are currently
only applicable on simpler datasets with a smaller number of classes. One of
the main issues with all of these approaches is that they require a large
number of labeled training images per class and cannot learn from few examples
(FSIL problem). To the best of our knowledge, CBCL [10, 21, 22, 23] is
currently the only approach that tackles the FSIL problem. Although CBCL
generates reasonable accuracy on FSIL problems, its accuracy depends on having
a good, task-specific feature extractor trained on a large dataset.

One of the main problems with the incremental learning approaches mentioned
above, is that they are evaluated on less complex datasets such as MNIST [9]
and CIFAR-100 [4]. These datasets contain idealistic images in constrained
environments. For real-world applications, however, robots do not have access
to images of objects in perfect conditions. Further, all of these datasets
contain a large number of training images per class which is unrealistic for
robots operating in real-world environments. Robots only have access to a
limited number of training images that are labeled by a human. Hence, to
develop more advanced incremental learning algorithms, there is a need for a
dataset that includes realistic images taken by an embodied, situated robot.
This dataset will act as a benchmark for future incremental object learning
approaches tested on a robot.

###  II-B Related Datasets

In recent years many robotics object recognition datasets have been publicly
released for research. Table I shows a comparison of robotics object
recognition datasets. Most of these datasets (COIL-100 [11], NORB [12],
BigBIRD [13], ROD [14] and ARID [16]) were designed for batch learning
scenario, however they could potentially be used for incremental learning. To
the best of our knowledge, only CORe-50 [15] and OpenLORIS [17] were
specifically designed for incremental learning and released incremental
learning benchmarks for robotics applications.

The datasets COIL-100, NORB, BigBIRD and ROD present images of objects
captured on a turntable in a systematically controlled environment with
perfect lighting conditions and object views. Although these datasets are
created for robotics applications, they miss many crucial challenges that
robots face in real-world scenarios such as scale variation of objects and the
presence of transparent objects. Further, NORB only has 5 object classes and
25 total objects, while COIL-100 and BigBIRD do not provide object classes and
contain only 100 objects in total, which is extremely small compared to other
datasets like CIFAR-100 or OpenLORIS. Classification results on such a small
number of classes also do not suffer from catastrophic forgetting as shown in
[7, 10]. ROD is the only dataset that has more than 51 object classes and 300
total number of objects. However, each class contains only $\sim$6 objects.
Another major issue with all of these datasets is that they contain the same
objects in the train and test sets. Hence, algorithms trained on these
datasets may overfit on the train set and nevertheless produce good results on
the test set instead of learning generalized representations of objects. For
real-world object recognition, algorithms need to learn general
representations of objects so that they can recognize unseen objects of the
same class [3, 24]. Finally, since theses datasets have a small number of
objects per class, there are too few different objects per class to create
training and test sets that do not contain the same objects for few-shot
incremental learning.

ARID is a recently captured robotics object recognition dataset which was
captured by manually driving a robot through different environments. The
dataset attends to many challenges of robot object recognition including that
the images were captured by cameras on a robot and that the images include
objects at different scales under different lighting conditions. However, this
dataset was not designed for incremental object learning and it did not
release any incremental learning benchmarks. ARID contains a large number of
object classes (51), however the total number of objects per class are
extremely small (3). Since there are only 3 objects per class, this dataset
has to use same objects in the training and test sets and it cannot be used
for learning from a few examples. Hence, this dataset cannot be used to
evaluate Few-shot incremental learning algorithms.

Figure 3: Examples images of 22 object categories in F-SIOL-310. One object
per category is shown from the training and test set splits for 5-shot
incremental learning. Note the difference between the objects between the
training and test sets.

Core-50 is the first robotics dataset specifically designed for incremental
object learning. It contains 11 video clips of objects in different
backgrounds. CORe-50 was captured with the operator holding the objects in his
hand, which makes it unsuitable for real-world autonomous robotics situations.
In real-world scenarios, robots have to capture data through their own camera
instead of asking for human assistance with every object. This dataset also
does not consider many other challenges faced by robots in real-world
scenarios such as scale variation of objects and the inclusion of transparent
objects. Further, this dataset only consists of 10 object classes with 5
objects per class (total 50 objects). OpenLORIS is the latest robotics dataset
that was designed for incremental object learning. Similar to ARID, this
dataset was manually captured using cameras on an autonomous robot. OpenLORIS
also captures the different challenges of robotics vision such as scale
variation, lighting conditions etc and it provides benchmarks for incremental
learning under different environmental conditions. Unfortunately, this dataset
contains only 19 object classes with only $\sim$3 objects per class. The
primary problems with CORe-50 and OpenLORIS are: 1) they contain same objects
in the train and test sets. Although the objects are captured in slightly
different environments, the objects themselves are the same in train and test
sets (see Figure 2 for examples). 2) These datasets test the incremental
learning capability of algorithms by providing the same objects in different
increments captured in different backgrounds. This is contrary to the spirit
of incremental learning where the algorithm does not have access to past
objects when learning new objects. 3) these datasets are not suitable for
testing Few-shot incremental learning and do not provide any FSIL benchmarks.

In contrast to these datasets, F-SIOL-310 contains different objects in the
training and test sets (see Figure 3) and it is specifically designed for
testing FSIL on robots. F-SIOL-310 contains 310 objects which is larger than
all of the above mentioned datasets. Similar to ARID and OpenLORIS, F-SIOL-310
was captured using a camera on a robot. However, unlike ARID and OpenLORIS,
F-SIOL-310 is captured using the arm camera of a manipulator robot (Baxter).
Further, note that unlike CORe-50 and OpenLORIS, F-SIOL-310 only provides
images of objects and not a set of videos with different views of objects. The
reason is that in real-world scenarios it is not always easy for robots to get
different views of objects and they usually have only one view of the object
available. Both CORe-50 and OpenLORIS contain manually taken videos of objects
from different views, which may not be possible for robots in real-world
situations.

##  III F-SIOL-310

F-SIOL-310 is a dataset specifically designed for few-shot incremental object
learning on an autonomous robot. F-SIOL-310 consists of common household
objects belonging to 22 classes: ballpoint, book, candy, cereal, comb, fork,
glass, hair_brush, hair_clip, keyboard, knife, lipstick, lotion, mouse, mug,
plate, shampoo, soap, spatula, spoon, toothbrush, toothpaste (see Figure 3).
For each object class, we purchased 11-17 different objects leading to a total
of 310 objects.

The dataset can be used to perform class level or object level classification.
Class level classification for FSIL is a much harder task than object-level
classification, since the model is only given a few training samples per class
but it is tested on a larger set which contains unseen objects. Hence, the
model is required to learn a general representation of each object class from
only a few training examples while learning classes incrementally, making the
task challenging. In this paper, we focus on the class-level classification
for FSIL experiments (Section IV).

The images that compose the dataset were collected on a table-top. The objects
were presented on the table in front of a Baxter robot and the robot’s hand
camera was used to capture the data. The robot’s hand hovered over the table
to caputre the objects. For each object two images from different views were
captured, which resulted in a total of 22-34 RGB images per class. The total
raw data consists of 620 RGB images, each of size 1020$\times$534.

Figure 4: Examples of images showing challenges in realistic robotic vision.
(a) Objects belonging to categories lipstick and hair_clip are relatively
small. (b) Objects belonging to categories hair_clip and glass are
transparent.

The images captured by the robot are realistic and not idealized. The
background of the objects is not perfect since the table has several
discolorations and some of the object sizes are rather small (Figure 4 (a)).
Moreover, the lighting conditions are not ideal and some objects are
transparent (Figure 4 (b)). Further, different objects are of different sizes
or at different distance from the camera (Figure 3). Hence, it was not
possible to crop the images to a fixed smaller size and we only provide the
raw data and used it in our FSIL experiments (Section IV). In case cropped
objects are needed, we provide the bounding boxes for objects. The bounding
boxes were generated by passing the images through the RetinaNet [25].

For evaluation on FSIL, we split the dataset into two different training and
test sets for 5-shot and 10-shot incremental learning experiments. For 5-shot
incremental learning, only 5 images per class are in the training set and the
rest of the images are in the test set. Similarly, 10 images per class are in
the training set for 10-shot incremental learning setting and the rest of the
images are in the test set. Objects in the training set do not overlap with
the test set i.e. training and test sets have different objects of the same
class (see Figure 3). This setup is in accordance with realistic robotics
scenarios, where the robot must learn the general concept of an object class
and recognize unseen objects rather than just memorizing and recognizing the
same objects.

##  IV EXPERIMENTS

We evaluate various state-of-the-art (SOTA) incremental learning approaches on
F-SIOL-310 for 5-shhot and 10-shot incremental learning scenarios. We further
evaluate the accuracy of a variety of deep convolutional networks pre-trained
on different datasets as backbone architectures in incremental learning
approaches for FSIL experiments.

Figure 5: Evaluation of 8 incremental learning approaches on F-SIOL-310 with
ImageNet features (a) and CIFAR-100 features (b), for 5-shot (left) and
10-shot (right) incremental learning with 2 classes per increment in terms of
classification accuracy (%). Mean and standard deviations of the
classification accuracy in each increment are reported across 10 runs with a
different seed in each run. Average incremental accuracy is reported in
parenthesis for each approach. FLB (in grey) is the batch learning upperbound
that trains the model on all the previous and new class data. (Best viewed in
color)

###  IV-A SOTA Incremental Learning Approaches

We evaluate 8 incremental learning approaches (FLB [26], FT, iCaRL [7], LWF
[8], EWC [5], CWR [15], CBCL [10], EEC [6]) on F-SIOL-310 for 5-shot and
10-shot incremental learning. Few-Shot learning baseline (FLB) uses the
features from a pre-trained neural network and then trains a linear layer
using the cross-entropy loss [26]. FLB is not designed for incremental
learning, hence we use the complete training set of the previous classes and
the new classes in each increment for training FLB. Thus, FLB is an upper
bound depicting the performance of batch learning strategy for FSIL
experiments. Fine-tuning (FT) is a naive approach in which the model is
trained only on the data of the new classes in each increment. Other
incremental learning approaches have been introduced in Section II.

###  IV-B Implementation Details

The Pytorch deep learning framework [27] and an Nvidia Titan RTX GPU were used
for implementation and training of all neural network models. All of the input
images were resized to $256\times 256$ and randomly cropped to $224\times 224$
as the input to the network during training. ResNet18 [28] pre-trained on
ImageNet dataset [3] was used as the backbone architecture for all the
approaches. For all the models except CBCL, we trained the network with cross-
entropy loss optimized with stochastic gradient descent (SGD) with momentum of
0.9 and a fixed learning rate of 0.1 for 40 epochs in each increment. For
CBCL, we used a fixed distance threshold $D$=17 for Agg-Var clustering and
number of closest centroids for classification $k$=1 in each increment.

For iCaRL, we allow it to store images of all the previous classes for both
5-shot and 10-shot incremental learning experiments which is similar to batch
learning. CBCL and EEC are allowed to store a maximum of 44 centroids, making
their memory consumption minimal as desired by incremental learning
approaches. EWC, LWF and CWR do not require storage of images or feature
vectors of previous classes. For both 5-shot and 10-shot incremental learning
experiments, we divided the training set into 11 batches with 2 classes per
batch learned in each increment. For evaluation, we tested each approach in
each increment on all the classes it had learned so far, resulting in
decreasing accuracy curves. We also report average incremental accuracy which
is the average of all the accuracies achieved by a model for all the
increments. For robustness, we ran all the experiments for all the models 10
times with different random seeds and report average and standard deviation of
the accuracies.

k-shot |  Networks |  FLB |  FT |  iCaRL |  LWF |  EWC |  CWR |  CBCL |  EEC
---|---|---|---|---|---|---|---|---|---
5-Shot |  ResNet18 |  82.1$\pm$8.4 |  18.7$\pm$15.3 |  19.4$\pm$17.6 |  14.9$\pm$12.8 |  13.8$\pm$13.0 |  39.1$\pm$25.1 |  80.6$\pm$7.9 |  22.5$\pm$15.8
|  VGG16 |  83.8$\pm$7.2 |  23.5$\pm$21.2 |  24.4$\pm$25.9 |  14.8$\pm$16.4 |  16.9$\pm$16.9 |  46.3$\pm$22.8 |  78.1$\pm$9.2 |  24.4$\pm$21.0
|  ResNet50 |  85.1$\pm$7.5 |  17.6$\pm$15.8 |  34.5$\pm$16.5 |  14.1$\pm$15.1 |  17.2$\pm$22.2 |  39.2$\pm$24.5 |  85.3$\pm$5.9 |  17.3$\pm$15.8
10-Shot |  ResNet18 |  89.2$\pm$5.2 |  22.1$\pm$19.6 |  19.0$\pm$18.0 |  18.7$\pm$19.6 |  15.4$\pm$18.6 |  39.5$\pm$24.6 |  86.6$\pm$5.9 |  22.8$\pm$17.7
|  VGG16 |  86.0$\pm$6.4 |  25.4$\pm$23.8 |  21.5$\pm$21.1 |  14.1$\pm$15.2 |  18.0$\pm$18.1 |  46.4$\pm$22.6 |  84.1$\pm$6.1 |  25.8$\pm$24.0
|  ResNet50 |  90.3$\pm$5.1 |  21.0$\pm$16.8 |  37.7$\pm$14.8 |  13.2$\pm$14.0 |  17.5$\pm$21.6 |  39.6$\pm$24.3 |  89.3$\pm$4.7 |  18.8$\pm$16.3
TABLE II: Comparison of 8 incremental learning approaches on F-SIOL-310 using
different backbone network architectures for 5-shot and 10-shot incremental
learning with 2 classes per increment in terms of average incremental accuracy
(%). Mean and standard deviations of the average incremental accuracy in 10
runs are reported.

###  IV-C Benchmarks for FSIL

Figure 5 (a) compares the 8 SOTA incremental learning approaches with ImageNet
features on F-SIOL-310 for 5-shot (Figure 5 (a) (left)) and 10-shot (Figure 5
(a) (right)) incremental learning experiments in terms of classification
accuracy. For both 5-shot and 10-shot incremental learning experiments, all
the models except CBCL and FLB suffer from catastrophic forgetting. Even
though iCaRL, EWC, LWF, EEC and CWR have been designed to reduce the effect of
catastrophic forgetting (as demonstrated on simple large scale datasets like
MNIST and CIFAR-100), they do not perform well on the FSIL scenario and
perform similarly to FT which is the baseline for catastrophic forgetting.
Note that iCaRL stores the complete data of all the previous classes, still it
suffers from catastrophic forgetting, which demonstrates that iCaRL struggles
with the FSIL vision challenge. FLB, the batch learning baseline (it does not
learn incrementally), uses the data of all the classes in each increment,
hence it produces the best results. CBCL is the only method that produces
favorable results on the two FSIL settings. Still, note that CBCL’s accuracy
drops on F-SIOL-310 compared to CIFAR-100 as reported in [10]. In the original
paper [10], CBCL outperforms FLB by a significant margin on FSIL experiments
on CIFAR-100. However, on F-SIOL-310 it produces $\sim$9% lower accuracy than
FLB after learning all the classes. These results clearly show that the
current incremental learning approaches are not suitable for few-shot
incremental object learning for robotics applications. CBCL is the most
promising method and it mitigates the effects of catastrophic forgetting, but
it still produces only $\sim$70% accuracy after learning only 22 object
classes. Many realistic domestic robotics applications will likely demand a
much higher object classification accuracy even after learning a large number
of object classes.

###  IV-D Effect of Using Different Base Features

For the previous experiment all the SOTA approaches used ResNet-18 pre-trained
on ImageNet as the feature extractor. To test the effect of the base features,
we trained ResNet-18 from scratch on the CIFAR-100 dataset and use it as the
backbone architecture for all the approaches. CIFAR-100 is a much smaller
dataset than ImageNet containing only 50,000 training images belonging to 100
classes. In comparison, ImageNet contains 1.2 million images belonging to 1000
classes. Hence, the CIFAR-100 base features are not as general as ImageNet
features.

Figure 5 (b) compares the 8 incremental learning approaches on 5-shot and
10-shot incremental learning using ResNet-18 pre-trained on CIFAR-100. Similar
to the previous experiment, all the models except CBCL and FLB suffer from
catastrophic forgetting for both 5-shot and 10-shot incremental experiments.
However, EEC and iCaRL show better performance ($\sim$10%) than when using
ImageNet features. Both of these approaches learn a representation and a
classifier together for incremental learning. Hence, it is difficult for the
representation to adjust after only a few training examples when starting from
ImageNet features. However, for CIFAR-100 the features learned by the model
are not as general (because of the smaller size of CIFAR-100) hence it is
easier for the representation to adjust even with a few training examples.

Both FLB and CBCL show significantly lower accuracy ($\sim$12%) when using
CIFAR-100 features than with ImageNet features. Both approaches use the fixed
features from the pre-trained network and do not learn their own
representation. Since CIFAR-100 features are not as general as ImageNet
features, both FLB and CBCL struggle in the FSIL scenario. These results
further show the limitations of the current incremental learning techniques.
Even methods that are specifically designed for FSIL, such as CBCL, depend on
the availability of good base features.

###  IV-E Effect of Using Different Backbone Architectures

Table II compares the average incremental accuracies of 8 incremantal learning
approaches on 5-shot and 10-shot incremental learning with three different
backbone network architectures (ResNet18 [28], VGG16 [29], ResNet50 [28]) pre-
trained on ImageNet dataset. Both FLB and CBCL produce the best results when
using a deeper backbone like ResNet50, since both FLB and CBCL use the network
as a fixed feature extractor and a deeper backbone has higher generalization
power. iCaRL also gets the best results with ResNet50 utilizing the higher
capacity of ResNet50 when learning using all the old and new class images. The
other 5 approaches achieve slightly better results using VGG16 over other
networks. However, all of the approaches (except FLB and CBCL) suffer from
catastrophic forgetting regardless of the backbone network used. Note that the
variance of average incremental accuracy for all the approaches (especially
FT, iCaRL, LWF, EWC, CWR and EEC) is significantly high. This shows that all
the approaches are highly susceptible to the order of the classes presented in
the training set depicting that they are unreliable in real-world applications
with unknown order of classes in different increments. Also, note that the
average incremental accuracy for 5-shot and 10-shot learning is similar for
all the approaches, except FLB and CBCL. These results show that a small
increase in number of images per class from 5 to 10 does not improve the
accuracy of these approaches since they all require a large number of training
examples per class. These findings further show that using a deeper backbone
network architecture can help some approaches achieve better performance.
However, current SOTA approaches are still far from being deployed in real-
world robotics applications. Hence, F-SIOL-310 can be used by future
incremental learning approaches to close the gap between research and real-
world robotics applications.

##  V CONCLUSION

In this paper we have presented a new dataset (F-SIOL-310) specifically
designed for the FSIL problem for robotics applications. The dataset
encompasses many real-world challenges faced by robots, like different objects
in training and test sets, scale variation and transparency in objects.
Through extensive experiments on F-SIOL-310 using 8 incremental learning
approaches, we show that the FSIL problem for robotics applications is far
from being solved.

Our results have shown that using deeper backbone architectures pre-trained on
a large scale image dataset helps improve the performance of incremental
learning techniques. Another finding from our results is that using a pre-
trained network as a feature extractor produces better accuracy than
finetuning the network. Among the incremental learning approaches, CBCL
produces the best results because it was specifically designed for FSIL.
Although CBCL produces comparable accuracy to the batch learning baseline when
the number of classes is small ($\sim$10), its accuracy starts to drop below
the batch learning baseline when the total number of classes learned increases
($\sim$22).

## ACKNOWLEDGMENT

This work was supported by Air Force Office of Scientific Research contract
FA9550-17-1-0017.

## References

  * [1] M. J. Matarić, “Socially assistive robotics: Human augmentation versus automation,” _Science Robotics_ , vol. 2, no. 4, 2017.
  * [2] T. Lesort, V. Lomonaco, A. Stoian, D. Maltoni, D. Filliat, and N. Díaz-Rodríguez, “Continual learning for robotics: Definition, framework, learning strategies, opportunities and challenges,” _Information Fusion_ , vol. 58, p. 52–68, 2020.
  * [3] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei, “Imagenet large scale visual recognition challenge,” _Int. J. Comput. Vision_ , vol. 115, no. 3, pp. 211–252, Dec. 2015.
  * [4] A. Krizhevsky, “Learning multiple layers of features from tiny images,” 2009, technical report, University of Toronto.
  * [5] J. Kirkpatrick, R. Pascanu, N. C. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, and R. Hadsell, “Overcoming catastrophic forgetting in neural networks,” _Proceedings of the National Academy of Sciences of the United States of America_ , vol. 114, no. 13, pp. 3521–3526, 2017.
  * [6] A. Ayub and A. R. Wagner, “Storing encoded episodes as concepts for continual learning,” _arXiv:2007.06637_ , 2020.
  * [7] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, “iCaRL: Incremental classifier and representation learning,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , July 2017.
  * [8] Z. Li and D. Hoiem, “Learning without forgetting,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 40, no. 12, pp. 2935–2947, Dec 2018.
  * [9] Y. LeChun, “The mnist database of handwritten digits,” 1998. [Online]. Available: http://yann.lecun.com/exdb/mnist/
  * [10] A. Ayub and A. R. Wagner, “Cognitively-inspired model for incremental learning using a few examples,” in _The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops_ , June 2020.
  * [11] S. A. Nene, S. K. Nayar, and H. Murase, “Columbia object image library (coil-20),” 1996, technical Report CUCS-005-96.
  * [12] Y. LeCun, F. J. Huang, and L. Bottou, “Learning methods for generic object recognition with invariance to pose and lighting,” in _Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004._ , vol. 2, 2004, pp. II–104 Vol.2.
  * [13] A. Singh, J. Sha, K. S. Narayan, T. Achim, and P. Abbeel, “Bigbird: A large-scale 3d database of object instances,” in _2014 IEEE International Conference on Robotics and Automation (ICRA)_ , 2014, pp. 509–516.
  * [14] K. Lai, L. Bo, X. Ren, and D. Fox, “A large-scale hierarchical multi-view rgb-d object dataset,” in _2011 IEEE International Conference on Robotics and Automation_ , 2011, pp. 1817–1824.
  * [15] V. Lomonaco and D. Maltoni, “Core50: a new dataset and benchmark for continuous object recognition,” in _Proceedings of the 1st Annual Conference on Robot Learning_ , vol. 78, 2017, pp. 17–26.
  * [16] M. R. Loghmani, B. Caputo, and M. Vincze, “Recognizing objects in-the-wild: Where do we stand?” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , 2018, pp. 2170–2177.
  * [17] Q. She, F. Feng, X. Hao, Q. Yang, C. Lan, V. Lomonaco, X. Shi, Z. Wang, Y. Guo, Y. Zhang, F. Qiao, and R. H. M. Chan, “Openloris-object: A robotic vision dataset and benchmark for lifelong deep learning,” _arXiv:1911.06487_ , 2019\.
  * [18] F. M. Castro, M. J. Marin-Jimenez, N. Guil, C. Schmid, and K. Alahari, “End-to-end incremental learning,” in _The European Conference on Computer Vision (ECCV)_ , September 2018.
  * [19] O. Ostapenko, M. Puscas, T. Klein, P. Jahnichen, and M. Nabi, “Learning to remember: A synaptic plasticity driven framework for continual learning,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019, pp. 11 321–11 329.
  * [20] A. Ayub and A. R. Wagner, “Eec: Learning to encode and regenerate images for continual learning,” in _International Conference on Learning Representations (ICLR)_ , 2021. [Online]. Available: https://openreview.net/forum?id=lWaz5a9lcFU
  * [21] ——, “Tell me what this is: Few-shot incremental object learning by a robot,” _arXiv:2008.00819_ , 2020.
  * [22] ——, “Centroid based concept learning for rgb-d indoor scene classification,” in _British Machine Vision Conference (BMVC)_ , 2020.
  * [23] K. Mokhtari, A. Ayub, V. Surendran, and A. R. Wagner, “Pedestrian density based path recognition and risk prediction for autonomous vehicles,” in _2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)_ , 2020, pp. 517–524.
  * [24] N. Pinto, D. D. Cox, and J. J. Dicarlo, “Why is real-world visual object recognition hard?” _PLoS Computational Biology_ , vol. 4, no. 1, 2008.
  * [25] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollar, “Focal loss for dense object detection,” in _The IEEE International Conference on Computer Vision (ICCV)_ , Oct 2017.
  * [26] W.-Y. Chen, Y.-C. Liu, Z. Kira, Y.-C. F. Wang, and J.-B. Huang, “A closer look at few-shot classification,” in _International Conference on Learning Representations_ , 2019.
  * [27] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in _Advances in Neural Information Processing Systems 32_. Curran Associates, Inc., 2019, pp. 8024–8035.
  * [28] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2016.
  * [29] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” _arXiv 1409.1556_ , 09 2014.