#

Linear independence of certain numbers in the base-$b$ number system

Shintaro Murakami , Yohei Tachiya  Hirosaki University, Graduate School of
Science and Technology, Hirosaki 036-8561, Japan
e-mail<EMAIL_ADDRESS>University, Graduate School of
Science and Technology, Hirosaki 036-8561, Japan
e-mail<EMAIL_ADDRESS>
###### Abstract

Let $(i,j)\in\mathbb{N}\times\mathbb{N}_{\geq 2}$ and $S_{i,j}$ be an infinite
subset of positive integers including all prime numbers in some arithmetic
progression. In this paper, we prove the linear independence over $\mathbb{Q}$
of the numbers

$1,\quad\sum_{n\in
S_{i,j}}\frac{a_{i,j}(n)}{b^{in^{j}}},\quad(i,j)\in\mathbb{N}\times\mathbb{N}_{\geq
2},$

where $b\geq 2$ is an integer and $a_{i,j}(n)$ are bounded nonzero integer-
valued functions on $S_{i,j}$. Moreover, we also establish a necessary and
sufficient condition on the subset $\mathcal{A}$ of
$\mathbb{N}\times\mathbb{N}_{\geq 2}$ for the numbers

$1,\quad\sum_{n\in
T_{i,j}}\frac{a_{i,j}(n)}{b^{in^{j}}},\quad(i,j)\in\mathcal{A}$

to be linearly independent over $\mathbb{Q}$ for any given infinite subsets
$T_{i,j}$ of positive integers. Our theorems generalize a result of V. Kumar.

Keywords: Linear independence, base-$b$ number system, primes in arithmetic
progression
AMS Subject Classification: 11J72.

##  1 Introduction

In 1996, Yu. V. Nesterenko [12] showed the lower bound for the transcendence
degree of the field generated over $\mathbb{Q}$ by the values of the
Eisenstein series. Nesterenko’s theorem derives a number of remarkable
transcendence and algebraic independence results for the values of various
modular functions. For example, applying Nesterenko’s theorem, D. Bertrand [2]
and independently D. Duverney, Ke. Nishioka, Ku. Nishioka and I. Shiokawa [4]
derived algebraic independence results for the values of the Jacobi theta
functions. In particular, they proved that the number
$\sum_{n=1}^{\infty}\alpha^{n^{2}}$ is transcendental for any algebraic number
$\alpha$ with $0<|\alpha|<1$.

On the other hand, we are not aware of the transcendence of the number
$\sum_{n=1}^{\infty}\alpha^{n^{k}}$ for an integer $k\geq 3$. Recently, V.
Kumar [9] obtained the following linear independence result by using the
properties of uniform distribution modulo $1$ of irrational numbers:

######  Theorem 1.1 ([9, Theorem 1]).

Let $k\geq 2$, $b\geq 2$ and $1\leq a_{1}<a_{2}<\cdots<a_{m}$ be integers such
that $\sqrt[k]{a_{i}/a_{j}}\notin\mathbb{Q}$ for any $i\neq j$. Then the
numbers

$1,\quad\sum_{n=1}^{\infty}\frac{1}{b^{a_{1}n^{k}}},\quad\sum_{n=1}^{\infty}\frac{1}{b^{a_{2}n^{k}}},\quad\dots,\quad\sum_{n=1}^{\infty}\frac{1}{b^{a_{m}n^{k}}}$
(1.1)

are linearly independent over $\mathbb{Q}$.

The aim of this paper is to extend Theorem 1.1 by showing linear independence
results for certain infinite series. In particular, as an application of our
Theorem 1.2, we will generalize Theorem 1.1 even without the condition
$\sqrt[k]{a_{i}/a_{j}}\notin\mathbb{Q}$ for any $i\neq j$. This gives a
positive answer to the question of V. Kumar [9, Section 4]. Moreover, we
establish in Theorem 1.3 a necessary and sufficient condition for a set of
certain infinite series to be linearly independent over $\mathbb{Q}$. We will
find by Theorem 1.3 that the same conclusion of Theorem 1.1 can be achieved in
case where $k\geq 3$ even if the infinite series in (1.1) are replaced by any
subseries (Corollary 1.3).

Before stating our Theorem 1.2, we prepare some notations. Let $\mathbb{N}$
denote the set of positive integers. Let
$(i,j)\in\mathbb{N}\times\mathbb{N}_{\geq 2}$ and $S_{i,j}$ be an infinite
subset of $\mathbb{N}$ including all prime numbers in the arithmetic
progression $n\equiv h_{i,j}$ (mod $d_{i,j}$), where $d_{i,j}$ and $h_{i,j}$
are positive integers and relatively prime. Note that Dirichlet’s theorem
states that such arithmetic progression contains infinitely many prime
numbers. Thus, we can choose various infinite sets $S_{i,j}$ such as the sets
of all prime numbers in arithmetic progressions, all prime numbers, all
squarefree integers and all positive integers.

Let $S_{i,j}$ be the above infinite set and $a_{i,j}(n)$ be a nonzero integer-
valued function on $S_{i,j}$. Define the function

$f_{i,j}(z):=\sum_{n\in S_{i,j}}a_{i,j}(n)z^{in^{j}},$ (1.2)

which converges in $|z|<1$. Since $j\geq 2$, the function (1.2) is not
rational. We establish linear independence results for the values of the
functions (1.2) at the rational argument $z=1/b$ with an integer $b\geq 2$.
Recall that an infinite set of numbers is called linearly independent if each
one of its finite subsets is linearly independent; otherwise it is called
linearly dependent. Our results are the following.

######  Theorem 1.2.

Let $b\geq 2$ be an integer and $f_{i,j}(z)$ be the functions defined in
(1.2). Then the set of the numbers

$1,\qquad f_{i,j}(1/b)=\sum_{n\in
S_{i,j}}\frac{a_{i,j}(n)}{b^{in^{j}}},\quad(i,j)\in\mathbb{N}\times\mathbb{N}_{\geq
2}$ (1.3)

is linearly independent over $\mathbb{Q}$.

Applying Theorem 1.2 with $S_{i,j}:=\mathbb{N}$ and $a_{i,j}(n):=(\pm 1)^{n}$
($n\geq 1$) for all $(i,j)\in\mathbb{N}\times\mathbb{N}_{\geq 2}$, we have

######  Corollary 1.1.

Let $b\geq 2$ be an integer. Then the set of the numbers

$1,\qquad\alpha_{i,j}:=\sum_{n=1}^{\infty}\frac{1}{b^{in^{j}}}\quad(i=1,2,\dots,\,\,\,j=2,3,\dots)$

is linearly independent over $\mathbb{Q}$. The same holds for the set of the
numbers

$1,\qquad\beta_{i,j}:=\sum_{n=1}^{\infty}\frac{(-1)^{n}}{b^{in^{j}}}\quad(i=1,2,\dots,\,\,\,j=2,3,\dots).$

Note that the first assertion in Corollary 1.1 gives an unconditional version
of Theorem 1.1.

######  Remark 1.1.

Let $\alpha_{i,2}$ and $\beta_{i,2}$ be as in Corollary 1.1. Then we have the
equalities $\alpha_{i,2}=(\vartheta_{3}(1/b^{i})-1)/2$ and
$\beta_{i,2}=(\vartheta_{4}(1/b^{i})-1)/2$, where

$\vartheta_{3}(q):=1+2\sum_{n=1}^{\infty}q^{n^{2}},\qquad\vartheta_{4}(q):=1+2\sum_{n=1}^{\infty}(-1)^{n}q^{n^{2}}$

are the Jacobi theta functions defined in $|q|<1$. Hence, Corollary 1.1
provides linear independence results for the values of the Jacobi theta
functions

$\vartheta_{\ell}(1/b^{i}),\quad i=1,2,\dots$ (1.4)

for each $\ell\in\\{3,4\\}$. As mentioned at the beginning of this section,
the numbers in (1.4) are all transcendental (cf. [2, 4]), and hence, so are
the numbers $\alpha_{i,2}$, $\beta_{i,2}$ $(i=1,2,\dots)$. More strongly, any
two numbers in the set $\\{\alpha_{i,2},\beta_{i,2}\mid i=1,2,\dots\\}$ are
algebraically independent over $\mathbb{Q}$, while any three are not (cf. [5,
6]). For example, the explicit algebraic relations among the first three
numbers $\alpha_{i,2}$ $(i=1,2,3)$ and $\beta_{i,2}$ $(i=1,2,3)$,
respectively, can be derived from the results in [6, § 4.1]. Recently, C.
Elsner and V. Kumar [7] obtained linear independence results over the field of
algebraic numbers for some three numbers in $\\{\alpha_{i,2}\mid i\geq 1\\}$.

Moreover, applying Theorem 1.2 with the sets $S_{i,j}$ of all prime numbers
and all squarefree integers, we have

######  Corollary 1.2.

Let $b\geq 2$ be an integer. Then the set of the numbers

$1,\qquad\gamma_{i,j}:=\sum_{p:{\rm
prime}}\frac{1}{b^{ip^{j}}}\quad(i=1,2,\dots,\,\,\,j=2,3,\dots)$

is linearly independent over $\mathbb{Q}$. The same holds for the set of the
numbers

$1,\qquad\sum_{n:{\rm
squarefree}}\frac{1}{b^{in^{j}}}\quad(i=1,2,\dots,\,\,\,j=2,3,\dots).$

It should be noted that D. H. Bailey, J. M. Borwein, R. E. Crandall and C.
Pomerance [1] proved that the numbers $\alpha_{i,j}$ ($j\geq 3$) and
$\gamma_{i,j}$ $(j\geq 2)$ are either transcendental or algebraic numbers with
degrees at least $j$ and $j+1$, respectively.

In Theorem 1.2 we can not replace the sets $S_{i,j}$ by any infinite subsets
of positive integers; indeed, the numbers (1.3) can be linearly dependent over
$\mathbb{Q}$ for suitable infinite sets $S_{i,j}$, e.g. clearly $\sum_{n\in
2\mathbb{N}}{b^{-n^{2}}}=\sum_{n\in\mathbb{N}}{b^{-4n^{2}}}$. The following
Theorem 1.3 gives a necessary and sufficient condition on the subset
$\mathcal{A}$ of $\mathbb{N}\times\mathbb{N}_{\geq 2}$ for the numbers in
(1.5) to be linearly independent over $\mathbb{Q}$ for arbitrary given
infinite sets $T_{i,j}$.

######  Theorem 1.3.

Let $b\geq 2$ be an integer and $\mathcal{A}$ be a subset of
$\mathbb{N}\times\mathbb{N}_{\geq 2}$. Then for any subsets $T_{i,j}$
$((i,j)\in\mathcal{A})$ of $\mathbb{N}$ and for any nonzero integer-valued
functions $a_{i,j}(n)$ on $T_{i,j}$ the set of the numbers

$1,\qquad\sum_{n\in
T_{i,j}}\frac{a_{i,j}(n)}{b^{in^{j}}},\quad(i,j)\in\mathcal{A}$ (1.5)

is linearly independent over $\mathbb{Q}$ if and only if the following two
conditions are fulfilled.

(i) Let $(i_{1},j_{1}),(i_{2},j_{2})\in\mathcal{A}$ be distinct. Then
$i_{1}u^{j_{1}}\neq i_{2}v^{j_{2}}$ for any positive integers $u$ and $v$.

(ii)There exists at most one $(i,j)\in\mathcal{A}$ with $j=2$.

Applying Theorem 1.3, we obtain the following Corollary 1.3 which gives an
another generalization of Theorem 1.1.

######  Corollary 1.3.

Let $k\geq 3$, $b\geq 2$ and $1\leq a_{1}<a_{2}<\cdots<a_{m}$ be integers such
that $\sqrt[k]{a_{i}/a_{j}}\notin\mathbb{Q}$ for any $i\neq j$. Let $T_{i}$
$(i=1,2,\dots,m)$ be any infinite subsets of $\mathbb{N}$. Then the numbers

$1,\quad\sum_{n\in T_{1}}\frac{1}{b^{a_{1}n^{k}}},\quad\sum_{n\in
T_{2}}\frac{1}{b^{a_{2}n^{k}}},\quad\dots,\quad\sum_{n\in
T_{m}}\frac{1}{b^{a_{m}n^{k}}}$

are linearly independent over $\mathbb{Q}$.

The present paper is organized as follows. Theorems 1.2 and 1.3 will be shown
in Sections 2 and 3, respectively. In the proofs, we will prove that the
base-$b$ representations of nontrivial linear forms over $\mathbb{Z}$ of the
numbers (1.3) and (1.5) contain arbitrarily long strings of zero without being
identically zero from some point on. To see this, we consider the system of
the simultaneous congruences in the proof of Theorem 1.2. This method is based
on elementary arguments used in the papers of S. Chowla [3] and P. Erdős [8].
On the other hand, in the proof of Theorem 1.3, we use Mahler’s result [11] on
the finiteness of integer solutions $x$ and $y$ for the Diophantine equation
$ax^{m}+by^{n}=c$. These approaches are completely different from those used
in the papers of V. Kumar [9, 10].

##  2 Proof of Theorem 1.2

We first show the following lemma.

######  Lemma 2.1.

Let $k\geq 2$, $u\geq 1$ and $v\neq 0$ be integers. Then there are infinitely
many prime numbers $p$ such that the congruence $uX^{k}+v\equiv p\pmod{p^{2}}$
has an integer solution.

###### Proof.

Let $g(X):=uX^{k}+v\in\mathbb{Z}[X]$. Then there are infinitely many prime
numbers $p$ such that $p$ divides $g(m)$ for some integer $m\geq 1$; in other
words, the congruence

$g(X)\equiv 0\pmod{p}$ (2.1)

has an integer solution for infinitely many prime numbers $p$. Let $p$ be such
a prime number with $p>\max\\{k,u,|v|\\}$ and let $x$ be an integer solution
of (2.1). Then $g(x)\equiv 0$ (mod $p$) and $g^{\prime}(x)=kux^{k-1}\not\equiv
0$ (mod $p$), since $v\neq 0$. Hence, there exists an integer $y$ satisfying

$g^{\prime}(x)y-1+g(x)/p\equiv 0\pmod{p}.$ (2.2)

By (2.2) we have

$g(py+x)=u(py+x)^{k}+v\equiv g^{\prime}(x)py+g(x)\equiv p\pmod{p^{2}}.$

This shows that $g(X)\equiv p$ (mod $p^{2}$) has an integer solution $X=py+x$
and the proof of Lemma 2.1 is completed. ∎

Proof of Theorem 1.2. Let $m\geq 2$ be an arbitrary integer. It suffices to
show that the set of the numbers

$1,\qquad f_{i,j}(1/b)=\sum_{n\in
S_{i,j}}\frac{a_{i,j}(n)}{b^{in^{j}}}\quad(i=1,2,\dots,m,\,\,j=2,3,\dots,m)$
(2.3)

is linearly independent over $\mathbb{Q}$. Define

$\mathcal{A}:=\\{(i,j)\mid i=1,2,\dots,m,\,\,j=2,3,\dots,m\\}.$

We fix $(i_{0},j_{0})\in\mathcal{A}$. Let $S_{i_{0},j_{0}}$ be an infinite
subset of positive integers including all prime numbers $p$ satisfying
$p\equiv h$ (mod $d$), where $d:=d_{i_{0},j_{0}}$ and $h:=h_{i_{0},j_{0}}$ are
positive integers and relatively prime. Let $N$ be an integer sufficiently
large. By Lemma 2.1 for each integer $\ell=1,2,\dots,N-1,N+1,\dots,2N-1$ there
exists a prime number $p_{\ell}>N$ such that the congruence

$i_{0}X^{j_{0}}+\ell-N\equiv p_{\ell}\pmod{p_{\ell}^{2}}$ (2.4)

has an integer solution. Since for each $\ell$ there are infinitely many such
prime numbers, we may assume that the prime numbers $p_{\ell}$ are distinct.
Let $x_{\ell}$ be a fixed integer solution of (2.4). Consider the system of
the $2N-1$ simultaneous congruences

$\left\\{\begin{array}[]{ll}X\equiv h&(\textrm{mod}\,\,d),\\\\[1.99997pt]
X\equiv
x_{\ell}&(\textrm{mod}\,\,p_{\ell}^{2}),\quad\ell=1,2,\dots,N-1,N+1,\dots,2N-1.\end{array}\right.$
(2.5)

We find by the Chinese remainder theorem that there exists the least positive
integer solution $x$ of (2.5) and all integer solutions of (2.5) are given by
$\alpha n+x$ $(n\in\mathbb{Z})$, where

$\alpha:=d\prod_{\begin{subarray}{c}\ell=1\\\ \ell\neq
N\end{subarray}}^{2N-1}p_{\ell}^{2}.$

Then the integers $\alpha$ and $x$ are relatively prime; indeed, if
$p_{\ell}\mid x$ for some prime number $p_{\ell}$, then by (2.4) and (2.5) we
have $p_{\ell}\mid\ell-N$, which is impossible by $1\leq|\ell-N|<N<p_{\ell}$.
Moreover, by (2.5) the integers $d$ and $x$ are relatively prime, since so are
the integers $d$ and $h$. Hence, by Dirichlet’s theorem on arithmetic
progressions, there exits a positive integer $n_{0}$ such that $q:=\alpha
n_{0}+x$ is a prime number sufficiently large. Note that $q\in
S_{i_{0},j_{0}}$, since $q\equiv x\equiv h\pmod{d}$. By (2.4) and (2.5) we
have

$i_{0}q^{j_{0}}+\ell-N\equiv p_{\ell}\pmod{p_{\ell}^{2}}$

for each integer $\ell=1,2,\dots,N-1,N+1,\dots,2N-1$. This implies that the
integer $i_{0}q^{j_{0}}+\ell-N$ is divisible by $p_{\ell}$ exactly once, and
hence, we obtain

$i_{0}q^{j_{0}}\pm u\notin\\{ik^{j}\mid
k\in\mathbb{N},\,\,(i,j)\in\mathcal{A}\\}$ (2.6)

for every integer $u=1,2,\dots,N-1$, since $p_{\ell}$ is a prime number
sufficiently large.

Let $c$ and $c_{i,j}$ $((i,j)\in\mathcal{A})$ be integers such that

$c+\sum_{(i,j)\in\mathcal{A}}c_{i,j}f_{i,j}(1/b)=0,$ (2.7)

where the sum is taken over all $i,j$ with $(i,j)\in\mathcal{A}$. Let

$f_{i,j}(1/b)=\sum_{n\in
S_{i,j}}\frac{a_{i,j}(n)}{b^{in^{j}}}=\sum_{n=1}^{\infty}\frac{e_{i,j}(n)}{b^{n}},$

where

$e_{i,j}(n):=\left\\{\begin{array}[]{ll}a_{i,j}(k)&{\rm
if}\,\,n=ik^{j}\,\,{\rm for\,\,some\,\,integer}\,\,k\in S_{i,j},\\\ 0&{\rm
otherwise}.\end{array}\right.$ (2.8)

Then by (2.6) and (2.8)

$e_{i,j}(i_{0}q^{j_{0}}\pm u)=0,\quad u=1,2,\dots,N-1$ (2.9)

for any $(i,j)\in\mathcal{A}$. Hence, by (2.7) and (2.9) we obtain

$\displaystyle P_{N}$
$\displaystyle:=-c-\sum_{n=1}^{i_{0}q^{j_{0}}-N}\frac{1}{b^{n}}\sum_{(i,j)\in\mathcal{A}}{c_{i,j}e_{i,j}(n)}$
$\displaystyle=\frac{1}{b^{i_{0}q^{j_{0}}}}\sum_{(i,j)\in\mathcal{A}}{c_{i,j}e_{i,j}(i_{0}q^{j_{0}})}+\sum_{n=i_{0}q^{j_{0}}+N}^{\infty}\frac{1}{b^{n}}\sum_{(i,j)\in\mathcal{A}}{c_{i,j}e_{i,j}(n)}.$
(2.10)

Clearly, $b^{i_{0}q^{j_{0}}-N}P_{N}$ is an integer. On the other hand, by
(2.10)

$\displaystyle|b^{i_{0}q^{j_{0}}-N}P_{N}|$ $\displaystyle\leq
b^{-N}C_{1}\sum_{(i,j)\in\mathcal{A}}|c_{i,j}|+\sum_{n=i_{0}q^{j_{0}}+N}^{\infty}\frac{C_{1}}{b^{n-i_{0}q^{j_{0}}+N}}\sum_{(i,j)\in\mathcal{A}}{|c_{i,j}|}$
$\displaystyle\leq C_{2}b^{-N}+2C_{2}b^{-2N}\to 0\quad(N\to\infty),$

where $C_{1}:=\max_{i,j}\\{|a_{i,j}(n)|\mid n\in S_{i,j}\\}$ and
$C_{2}:=C_{1}\sum_{(i,j)\in\mathcal{A}}|c_{i,j}|$. Thus, we obtain $P_{N}=0$
for every large integer $N$. Therefore, by (2.10) we find that

$Q_{N}:=\sum_{(i,j)\in\mathcal{A}}c_{i,j}e_{i,j}(i_{0}q^{j_{0}})=-\sum_{n=i_{0}q^{j_{0}}+N}^{\infty}\frac{1}{b^{n-i_{0}q^{j_{0}}}}\sum_{(i,j)\in\mathcal{A}}{c_{i,j}e_{i,j}(n)}$

is an integer for every large $N$ and

$|Q_{N}|\leq 2C_{2}b^{-N}\to 0\quad(N\to\infty).$

This implies that

$Q_{N}=\sum_{(i,j)\in\mathcal{A}}c_{i,j}e_{i,j}(i_{0}q^{j_{0}})=0$ (2.11)

for every large integer $N$. Let $(i,j)\in\mathcal{A}$ be satisfy
$i_{0}q^{j_{0}}=ik^{j}$ for some integer $k\in S_{i,j}$. Then we obtain $i\mid
i_{0}$ and $j\mid j_{0}$, since $q$ is a prime number sufficiently large.
Hence, the equality (2.11) is written as

$\sum_{\begin{subarray}{c}(i,j)\in\mathcal{A}\\\ i\leq i_{0},j\leq
j_{0}\end{subarray}}c_{i,j}e_{i,j}(i_{0}q^{j_{0}})=0.$ (2.12)

Now we show $c_{i,j}=0$ for any $(i,j)\in\mathcal{A}$ by induction on
$i+j(\geq 3)$. We first apply the above arguments to the case where
$(i_{0},j_{0})=(1,2)\in\mathcal{A}$ and the set $S_{1,2}$. Then there exists a
prime number $q:=q_{1,2}\in S_{1,2}$ satisfying (2.12), so that,
$c_{1,2}e_{1,2}(q^{2})=0$. Since $e_{1,2}(q^{2})=a_{1,2}(q)\neq 0$, we obtain
$c_{1,2}=0$. Let $M\geq 3$ be an integer and suppose that $c_{i,j}=0$ for any
$(i,j)\in\mathcal{A}$ with $i+j\leq M$. Then, applying the above arguments
again to the case where $(i_{0},j_{0})\in\mathcal{A}$ with $i_{0}+j_{0}=M+1$
and the set $S_{i_{0},j_{0}}$, we obtain (2.12) for some prime number
$q:=q_{i_{0},j_{0}}\in S_{i_{0},j_{0}}$. Hence, by the induction hypothesis

$0=\sum_{\begin{subarray}{c}(i,j)\in\mathcal{A}\\\ i\leq i_{0},j\leq
j_{0}\end{subarray}}c_{i,j}e_{i,j}(i_{0}q^{j_{0}})=c_{i_{0},j_{0}}e_{i_{0},j_{0}}(i_{0}q^{j_{0}}).$

Since $e_{i_{0},j_{0}}(i_{0}q^{j_{0}})=a_{i_{0},j_{0}}(q)\neq 0$, we have
$c_{i_{0},j_{0}}=0$. Thus, we obtain $c_{i,j}~{}=0$ for every
$(i,j)\in\mathcal{A}$, and so $c=0$ by (2.7). The proof of Theorem 1.2 is
completed. ∎

##  3 Proof of Theorem 1.3

We first show that if both conditions (i) and (ii) do not hold, then the
numbers in (1.5) can be linearly dependent over $\mathbb{Q}$ for suitable
infinite sets $T_{i,j}$ and the constant functions $a_{i,j}(n)\equiv 1$ $(n\in
T_{i,j})$. If condition (i) does not hold, then there exit distinct pairs
$(i_{1},j_{1}),(i_{2},j_{2})\in\mathcal{A}$ and positive integers $u,v$ such
that $i_{1}u^{j_{1}}=i_{2}v^{j_{2}}$. Hence, putting
$T_{1}:=\\{u2^{j_{2}m}\mid m\in\mathbb{N}\\}$ and $T_{2}:=\\{v2^{j_{1}m}\mid
m\in\mathbb{N}\\}$, we obtain the equality $\sum_{n\in
T_{1}}{b^{-i_{1}n^{j_{1}}}}=\sum_{n\in T_{2}}{b^{-i_{2}n^{j_{2}}}}.$ Next we
assume that condition (i) holds while condition (ii) does not. Then there
exist $(i_{1},2),(i_{2},2)\in\mathcal{A}$ such that $i_{1}i_{2}$ is not
square. Consider the set of positive integer solutions of the Pell equation
$x^{2}-i_{1}i_{2}y^{2}=1$:

$\mathcal{S}:=\left\\{(x,y)\in\mathbb{N}^{2}\mid
x^{2}-i_{1}i_{2}y^{2}=1\right\\}.$

As is well known, $\mathcal{S}$ is an infinite set. Hence, noting that
$i_{1}x^{2}-i_{2}(i_{1}y)^{2}=i_{1}$ for $(x,y)\in\mathcal{S}$, we have

$\sum_{n\in T_{1}}\frac{1}{b^{i_{1}n^{2}}}=\sum_{n\in
T_{2}}\frac{1}{b^{i_{2}n^{2}+i_{1}}}=\frac{1}{b^{i_{1}}}\sum_{n\in
T_{2}}\frac{1}{b^{i_{2}n^{2}}},$

where $T_{1}:=\\{x\mid(x,y)\in\mathcal{S}\\}$ and
$T_{2}:=\\{i_{1}y\mid(x,y)\in\mathcal{S}\\}$. Thus, our assertion is proved.

In what follows, we assume that both conditions (i) and (ii) hold. We fix
$(i_{0},j_{0})\in\mathcal{A}$ and let $N\geq 1$ be an integer sufficiently
large. Consider the Diophantine equations

$i_{0}x^{j_{0}}-iy^{j}=\pm u,$ (3.1)

where $(i,j)\in\mathcal{A}$ and $u=1,2,\dots,N-1$. If $j_{0}=j=~{}2$, we have
$i_{0}=i$ by condition (ii). Then clearly the equation (3.1) has only finitely
many integer solutions $x$ and $y$. Let $\max\\{j_{0},j\\}\geq 3$. Then K.
Mahler [11] proved that the greatest prime factor of $i_{0}x^{j_{0}}-iy^{j}$
tends to infinity as $\max\\{|x|,|y|\\}\to\infty$ with relatively prime
integers $x$ and $y$. Hence, also in this case, the equation (3.1) has only
finitely many integer solutions $x$ and $y$. Thus, there exists a positive
integer $M$ such that

$i_{0}t^{j_{0}}\pm u\notin\\{ik^{j}\mid k\in\mathbb{N},(i,j)\in\mathcal{A}\\}$
(3.2)

holds for every integer $t>M$ and for all integers $u=1,2,\dots,N-1$.

Let $c$ and $c_{i,j}$ $((i,j)\in\mathcal{A})$ be integers such that

$c+\sum_{(i,j)\in\mathcal{A}}c_{i,j}\delta_{i,j}=0,$ (3.3)

where

$\delta_{i,j}:=\sum_{n\in
T_{i,j}}\frac{a_{i,j}(n)}{b^{in^{j}}}=\sum_{n=1}^{\infty}\frac{e_{i,j}(n)}{b^{n}}$

and

$e_{i,j}(n):=\left\\{\begin{array}[]{ll}a_{i,j}(k)&{\rm
if}\,\,n=ik^{j}\,\,{\rm for\,\,some\,\,integer}\,\,k\in T_{i,j},\\\ 0&{\rm
otherwise}.\end{array}\right.$ (3.4)

Let $t\in T_{i_{0},j_{0}}$ $(t>M)$ be an integer satisfying (3.2). Then by
(3.2) and (3.4)

$e_{i,j}(i_{0}t^{j_{0}}\pm u)=0,\quad u=1,2,\dots,N-1$

for any $(i,j)\in\mathcal{A}$. Hence, similarly as in the proof of Theorem
1.2, we can derive

$\sum_{(i,j)\in\mathcal{A}}c_{i,j}e_{i,j}(i_{0}t^{j_{0}})=0$ (3.5)

for every large integer $N$. Clearly, we have
$e_{i_{0},j_{0}}(i_{0}t^{j_{0}})=a_{i_{0},j_{0}}(t)\neq 0$. Moreover, if
$(i,j)\neq(i_{0},j_{0})$, then by condition (i) we have $i_{0}t^{j_{0}}\neq
iv^{j}$ for every positive integer $v$, and so $e_{i,j}(i_{0}t^{j_{0}})=0$.
Therefore, by (3.5) we obtain $c_{i_{0},j_{0}}=0$. Since the integer pair
$(i_{0},j_{0})$ is chosen arbitrarily, we obtain $c_{i,j}=0$ for every
$(i,j)\in\mathcal{A}$, and hence, $c=0$ by (3.3). Theorem 1.3 is proved. ∎

### Acknowledgements

The authors would like to express their sincere gratitude to Professor Hajime
Kaneko for valuable comments and suggestions on this manuscript. The authors
are also grateful to Professor Takafumi Miyazaki for pointing out the paper of
K. Mahler [11]. This work was partly supported by JSPS KAKENHI Grant Number
JP18K03201.

## References

  * [1] D. H. Bailey, J. M. Borwein, R. E. Crandall and C. Pomerance, On the binary expansions of algebraic numbers, J. Théor. Nombres Bordeaux, 16 (2004), 487–518.
  * [2] D. Bertrand, Theta functions and transcendence, Ramanujan J. 1 (1997), 339–350.
  * [3] S. Chowla,  On series of the Lambert type which assume irrational values for rational values of the argument, Proc. Natl. Inst. Sci. India Part A 13 (1947), 171–173.
  * [4] D. Duverney, Ke. Nishioka, Ku. Nishioka and I. Shiokawa, Transcendence of Jacobi’s theta series, Proc. Japan Acad. Ser. A Math. Sci. 72 (1996), 202–203.
  * [5] C. Elsner, F. Luca and Y. Tachiya, Algebraic results for the values $\vartheta_{3}(m\tau)$ and $\vartheta_{3}(n\tau)$ of the Jacobi theta-constant, Mosc. J. Comb. Number Theory 8 (2019), 71–79.
  * [6] C. Elsner, M. Kaneko and Y. Tachiya, Algebraic independence results for the values of the theta-constants and some identities, J. Ramanujan Math. Soc. 35 (2020), 71–80.
  * [7] C. Elsner and V. Kumar, On linear forms in Jacobi theta-constants, arXiv: 1911.06513.
  * [8] P. Erdős, On arithmetical properties of Lambert series, J. Indian Math. Soc. (N.S.) 12 (1948), 63–66.
  * [9] V. Kumar, Linear independence of certain numbers, Arch. Math. (Basel) 112 (2019), 377–385.
  * [10] V. Kumar, Linear independence of certain numbers, J. Ramanujan Math. Soc. 35 (2020), 17–22.
  * [11] K. Mahler, On the greatest prime factor of $ax^{m}+by^{n}$, Nieuw Arch. Wisk. 1 (1953), 113–122.
  * [12] Yu. V. Nesterenko, Modular functions and transcendence questions, Mat. Sb. 187 (1996) 65–96; English transl. Sb. Math. 187 1319–1348.


# Offline Meta-Reinforcement Learning for Industrial Insertion ††thanks:
*Equal contribution ††thanks: 1X, The Moonshot Factory, Mountain View, CA,
USA††thanks: 2Intrinsic Innovation LLC, Mountain View, CA, USA††thanks:
3Deepmind, London, UK††thanks: 4Google Brain, Mountain View, CA, USA††thanks:
5Department of Electric Engineering and Computer Science, University of
California, Berkeley, Berkeley, CA, USA ††thanks: 6Stanford University (work
done as an intern at X)

Tony Z. Zhao∗1,6, Jianlan Luo∗2, Oleg Sushkov3, Rugile Pevceviciute3,
Nicolas Heess3, Jon Scholz3, Stefan Schaal2 and Sergey Levine4,5

###### Abstract

Reinforcement learning (RL) can in principle let robots automatically adapt to
new tasks, but current RL methods require a large number of trials to
accomplish this. In this paper, we tackle rapid adaptation to new tasks
through the framework of meta-learning, which utilizes past tasks to learn to
adapt with a specific focus on industrial insertion tasks. Fast adaptation is
crucial because prohibitively large number of on-robot trials will potentially
damage hardware pieces. Additionally, effective adaptation is also feasible in
that experience among different insertion applications can be largely
leveraged by each other. In this setting, we address two specific challenges
when applying meta-learning. First, conventional meta-RL algorithms require
lengthy online meta-training. We show that this can be replaced with
appropriately chosen offline data, resulting in an offline meta-RL method that
only requires demonstrations and trials from each of the prior tasks, without
the need to run costly meta-RL procedures online. Second, meta-RL methods can
fail to generalize to new tasks that are too different from those seen at
meta-training time, which poses a particular challenge in industrial
applications, where high success rates are critical. We address this by
combining contextual meta-learning with direct online finetuning: if the new
task is similar to those seen in the prior data, then the contextual meta-
learner adapts immediately, and if it is too different, it gradually adapts
through finetuning. We show that our approach is able to quickly adapt to a
variety of different insertion tasks, with a success rate of 100% using only a
fraction of the samples needed for learning the tasks from scratch. Experiment
videos and details are available at https://sites.google.com/view/offline-
metarl-insertion.

##  I INTRODUCTION

Industrial robotic manipulation tasks are gaining more traction than ever,
partially due to the sharply increasing interest in AI-enabled applications
and more suitable and affordable robot hardware [1, 2, 3]. Part mating and
part insertion are very common in the manufacturing industry, and the goal to
enable high-mix low-volume scenarios are among the most prominent research
topics [4]. Current manufacturing methods often rely on human-engineered
heuristics that must be finetuned or redesigned for each new instance. In this
context, reinforcement learning provides an appealing and automated
alternative. Previous works have demonstrated that reinforcement learning can
solve some of these manipulations tasks [5, 9, 10, 11, 12, 13, 14, 15, 16, 6,
7, 8]. However, lengthy exploration processes are often required. For example,
it would usually take a RL algorithm hours to solve one connector insertion
task [9]. This process needs to be repeated for each individual connector,
even though these connectors can bear similar insertion strategies; e.g., the
knowledge to insert a USB-A connector should provide guidance for inserting a
USB-C connector.

Figure 1: Our approach, Offline Meta-RL with Demonstration Adaptation (ODA),
uses offline data for pretraining, then adapts using both demonstrations and
online finetuning. Figure 2: During offline meta-training (left), ODA trains
the policy using offline data for each training task, with an encoder that
learns to summarize the demonstrations into a latent variable $z$. Next,
during adaptation to a new task (middle), we condition the encoder on demos
for the new task, obtaining a policy that either solves the new task
immediately, or else provides a initialization for the third stage (right),
where the entire policy is finetuned end-to-end with online interactions.

Meta-RL algorithms can make it possible to adapt to new tasks quickly by pre-
training on a distribution of tasks [17, 18, 19, 20, 21, 22, 23, 24]. However,
the lengthy online meta-training phase required can be prohibitive in
industrial robotics settings. On the other hand, robots that are already
deployed at manufacturing sites can generate large amounts of useful data from
their existing controllers. This makes it appealing to consider incorporating
ideas from offline RL [25, 26, 27, 28, 29, 30, 31], which utilizes previously
collected datasets in place of active data collection. Several works have
proposed offline algorithms for meta-RL [32, 33, 34, 35, 36]. Our offline
meta-RL approach builds on Pong et al. [33], but incorporates a number of
design choices that make it practical for industrial applications: (i) we
leverage demonstrations for both training tasks and the test task, which
reduces the need for lengthy exploration and accelerates adaptation; (ii) we
combine meta-learned adaptation via a PEARL-style encoder [18] with full
finetuning of the entire policy, such that even when the new task is very
different from those seen in meta-training, the policy can still adapt
gradually until it succeed.

Concretely, our method first meta-trains a policy on available datasets. For a
new task, we adapt the policy using user-provided demonstrations. This phase
may already solve most tasks, but if it does not, we follow up with a fine-
tuning phase until satisfactory performance is attained. The offline training
dataset can comprise demonstration data from humans, replay buffer data from
RL training, or even data from hand-designed (“scripted”) policies. During
meta-training, we first pass human demonstration data into an encoder to infer
the task variable $z$; we then condition both the actor and critic on $z$ in
addition to observations or actions. All networks are trained jointly with
offline RL. This encourages the encoder to pick up task information from
demonstrations that is useful for actor and critic. When adapting to a new
task, we infer a new task variable from a few human demonstrations and
evaluate the adapted policy. We only finetune the policy end-to-end with
additional online samples when adaptation alone does not solve the task.

Our contributions are summarized as follows: We present a novel offline meta-
RL framework that integrates a number of design choices to make it practical
in realistic industrial insertion settings. Our method can adapt to unseen
tasks reliably and rapidly by leveraging static offline datasets and optional
on-policy robot interaction. We evaluate it with a total of 13 industrial
insertion tasks, including ones like RAM insertion that has tight tolerance
and requires delicate handling. Our method is able to achieve 100/100 success
for all 13 tasks within 30 minutes of adaptation.

##  II BACKGROUND

In this section, we give a brief background of RL and offline meta-RL. We
consider a Markov Decision Process (MDP) associated with task $\mathcal{T}$
defined by
$\mathcal{M}_{\mathcal{T}}=\\{\mathcal{S},\mathcal{A},\mathcal{P},\mathcal{R},p_{0},\gamma\\}$.
$\mathcal{S}$ is the state space, $\mathcal{A}$ is the action space,
$\mathcal{P}$ governs the transition dynamics, $\mathcal{R}$ is the reward
function, $\gamma$ is the discount factor, $p_{0}(s)$ is the initial state
distribution. The replay buffer for the task $\mathcal{D}_{\mathcal{T}}$
consists of state, action, reward, next state tuples $(s,a,r,s^{\prime})$. We
also define the associated Q function of the policy as
$Q^{\pi}(s_{t},a_{t})=\sum_{t^{\prime}=t}^{T}\mathbb{E}_{\pi}[\gamma^{t^{\prime}-t}r(s_{t^{\prime}},a_{t^{\prime}})|s_{t},a_{t}]$,
with $a_{t^{\prime}}$ taken with the current policy $\pi$. We can learn this
Q-function via Bellman backups in actor-critic methods [37], and the objective
of reinforcement learning is to find the policy that maximize cumulative
discounted rewards $R=\sum_{t}\gamma^{t}r(s_{t},a_{t})$.

In the offline meta-RL setting, we assume a task distribution
$p_{\mathcal{T}}(\mathord{\cdot})$, and every task $\mathcal{T}$ is an
aforementioned MDP. We slightly abuse the notion of
$\mathcal{D}_{\mathcal{T}}$ to represent the static dataset linked to each
task. During offline meta-training, we have N training tasks
$\mathcal{T}_{i=1,...,N}$ sampled from $p_{\mathcal{T}}(\mathord{\cdot})$ and
we assume access to dataset $\mathcal{D}_{\mathcal{T}_{i}}$ for each sampled
task. We optimized the appropriate meta-training loss using the datasets to
learn a meta-policy $\pi_{\theta^{\star}}$ and a meta-adaptation procedure
$f_{\phi^{\star}}$. At meta-test time, we face a new task
$\mathcal{T^{\prime}}$ sampled from $p_{\mathcal{T}}(\mathord{\cdot})$, and we
assume access to the associated new dataset
$\mathcal{D}_{\mathcal{T^{\prime}}}$. The meta-adaptation procedure adapts the
meta-policy to the new task, i.e.
$\hat{\pi}=f_{\phi^{\star}}(\pi_{\theta^{\star}},\mathcal{D}_{\mathcal{T^{\prime}}})$,
and we aim to maximizes the return of $\hat{\pi}$:

$\displaystyle\theta^{\star},\phi^{\star}=\text{argmax}_{\theta,\phi}\,\,\sum_{t=0}^{T}[\gamma^{t}r(s_{t},a_{t})];$
(1)

where $s_{t},a_{t}$ denote the state and action taken at step $t$ when
executing the adapted policy $\hat{\pi}$ in $\mathcal{T^{\prime}}$.

##  III RELATED WORK

In addition the works cited in Sec. I, we discuss several previous works that
focus on insertion problems. Heuristic-based methods combined with compliance-
controlled robotic arms or even passive mechanisms [38, 39, 40, 41, 42, 43,
44, 45, 46] can solve some insertion tasks reliably, but such methods
typically require manual tuning for every type of task and connector. Gerrit
et al. [24] applied PEARL[18] for solving industrial insertion tasks in a
sim2real fashion, while it has yet to be shown that such domain randomization
approach could scale beyond the two insertion tasks considered in the paper.
We posit that it would require a significantly larger family of parameters to
randomize over in order to capture the complex physics like contact dynamics
and deformation, making it computationally expensive and involves careful
engineering. While our paper show that a broader range of connectors insertion
problems can be solved by collecting data in the real world and process them
with offline meta-RL. Another relevant work is by Spector et al. [47], which
learns a residual insertion policy on top of an existing PD controller through
“backward learning.” Unlike [48, 12], the residual policy is learned in a
supervised fashion for sample efficiency. They generate the training data by a
scripted policy producing random perturbations starting from the goal, and
show this type of template policy can in principle tackle a series of
industrial insertion tasks. However, as the authors pointed out, this method
has trouble inserting connectors with small clearance, thus resulting in lower
performance for connectors like USB-C, whereas our method is able to
consistently adapt to 13 new connectors with a 100% success rate, as shown in
our experiments. The paper also focused on visual adaptation, which can be a
different challenge than ours.

##  IV Offline RL with Demonstration Adaptation

In our method, which is summarized in Fig. 2, an adaptive policy is meta-
trained via offline RL on a user-provided dataset consisting of trials for a
variety of different tasks, together with corresponding policies for each
task. This adaptive policy is then used to solve new tasks, first by adapting
to them using a small number of demonstrations, and then by finetuning with
online RL. In the meta-training stage, we assume access to two buffers
containing demonstrations and offline data respectively, for each of the
pretrain tasks. Our goal is to meta-train the policy so that it can adapt to
new tasks using user-provided demonstrations, which are easy and safe to
provide for a new task in limited quantity, rather than adapting entirely via
online exploration trials as in prior work. Our meta-learned policy consists
of a policy network $\pi_{\theta}(a|s,z)$ that is conditioned on the current
state $s$ and a _latent code_ $z$ that denotes the task, as well as an encoder
network $q_{\phi}(z|c)$ that predicts $z$ for each task from a _context_ $c$,
which consists of the demonstrations. Intuitively, this encoder extracts the
information necessary to determine how to perform the task from the demos and
consolidates it into $z$, as we will discuss in this section. This design
resembles prior methods such as PEARL and MELD [18, 23], with the main
difference that we specifically encode demonstrations rather than online
experience. However, unlike meta-imitation methods [49], the policy and
encoder are still meta-trained with RL.

When our method needs to adapt to a new task (e.g., insert a new type of
connector), we assume that we are first provided with demonstrations, which
form the context $c$ for the new task and which we use to infer $z\sim
q(z|c)$. Then, we can run $\pi(a|s,z)$. In many cases, this policy will
already perform the task successfully. However, if the new task is
substantially different from the tasks seen during meta-training, the encoder
may be unable to infer a $z$ for which the policy is consistently successful.
In this case, we can finetune the entire policy and encoder with reinforcement
learning and additional online data collection. Typically, this finetuning
process is quite fast (5-10 minutes in our experiments), since $\pi(a|s,z)$
already provides a very good initialization. We describe the components of our
method in more detail below.

###  IV-A Contextual Meta-Learning with Demonstrations

The design of our meta-reinforcement learning model resembles prior contextual
meta-RL algorithms, such as PEARL and MELD [18, 23], with the principle
differences being that we utilize offline data and adapt using demonstrations.
In this design, adaptation is performed via an encoder network
$q_{\phi}(z|c)$, which reads in a context $c$ consisting of some trajectories
for the current task, and infers a vector of latent variables $z$ that is
passed to the policy $\pi_{\theta}(a|s,z)$. In this way, $z$ encapsulates the
information needed to perform the current task. We follow prior work and use
the same architecture for $q_{\phi}(z|c)$ as PEARL. Training is performed end-
to-end via RL, which trains both $\pi_{\theta}(a|s,z)$ and $q_{\phi}(z|c)$
together to maximize performance on the meta-training tasks. We discuss the
specific algorithm we use in Section IV-B. We deviate from prior work in our
choice of context $c$: while PEARL and MELD [18, 23] use trials from the
latest policy to generate trajectories for the context, our method uses user-
provided demonstrations. Such demonstrations are typically easy to provide in
small quantities, and spare the method from needing to explore via trial and
error. In our experiments, we collect demonstrations using a space mouse to
control the tool-center-point (TCP) pose. To meta-train $q_{\phi}(z|c)$ to
utilize demonstrations, we maintain two buffers during meta-training: a
demonstration buffer, which is used to form the context for each minibatch,
and another buffer that consists of all offline data and is used to sample the
transitions for actor-critic updates. Following Rakelly et al. [18], we add a
KL-divergence loss to $q_{\phi}(z|c)$ to minimize unnecessary information
content in $z$. This is summarized in Algorithm 1.

In the offline setting, demonstrations also allow us to sidestep the train-
test distribution shift problem in offline meta-RL pointed out by Pong et al.
[33]. When using the meta-trained policy to collect data to form the context
$c$, the data collected for a new task will differ systematically from the
offline data used for training $q_{\phi}(z|c)$ during the offline meta-
training phase, since the learned policy differs from the policy that
collected the offline data, and as pointed out by Pong et al. [33], this can
lead to very poor performance for standard offline meta-RL methods. Since we
instead use demonstrations, which do not experience this distributional shift,
our method performs well even with purely offline meta-training.

###  IV-B Offline and Online Reinforcement Learning

In order to meta-train on offline data, we require a suitable offline RL
algorithm. Since our aim is to meta-train a policy that can then be fine-tuned
to new tasks, we use Advantage-Weighted Actor-Critic (AWAC) [27], which has
been shown to perform well in offline training followed by fine-tuning. We
summarize AWAC in this section. AWAC aims to learn a policy that maximizes
reward while bounding the deviation from the data distribution $\pi_{\beta}$:

$\displaystyle\theta^{\star}=\operatorname*{arg\,max}_{\theta}\mathbb{E}_{{\mathbf{s}}\sim\mathcal{D}}\mathbb{E}_{\pi_{\theta}({\mathbf{a}}|{\mathbf{s}})}[Q_{\phi}({\mathbf{s}},{\mathbf{a}})]\,\mathbf{s}.\mathbf{t}.\,D_{KL}(\pi_{\theta}\|\pi_{\beta})\leq\epsilon$

This constrained optimization can be solved via Lagrangian duality, and the
solution can be approximated via weighted maximum likelihood:

$\displaystyle\theta^{\star}=\operatorname*{arg\,max}_{\theta}\mathbb{E}_{{\mathbf{s}},{\mathbf{a}}\sim\beta}[\log\pi_{\theta}({\mathbf{a}}|{\mathbf{s}})\exp(A^{\pi}({\mathbf{s}},{\mathbf{a}}))],$

where
$A^{\pi}({\mathbf{s}},{\mathbf{a}})=Q_{\phi}({\mathbf{s}},{\mathbf{a}})-E_{{\mathbf{a}}\sim\pi_{\theta}({\mathbf{a}}|{\mathbf{s}})}[Q_{\phi}({\mathbf{s}},{\mathbf{a}})]$
is the advantage under current policy $\pi$. The critic $Q_{\phi}$ is trained
to minimize the standard Bellman error. For further detail, we refer to prior
work [27].

This RL algorithm is used to meta-train the policy, critic and encoder, and
also to finetune it in the last stage of adaptation. The design of our
approach resembles the method of Pong et al. [33], with several critical
differences: (i) We condition the encoder on demonstrations, making it
possible to adapt to new tasks without risky exploration. This is important in
safety-critical industrial settings. (ii) Unlike Pong et al. [33], we do not
require any online meta-training phase, because the conditioning
demonstrations do not incur distributional shift. This makes our method
significantly easier to meta-train, since all meta-training uses offline data
with no online collection at all. (iii) We utilize a hybrid adaptation
procedure where we _first_ use the encoder to adapt to demonstrations, and
then finetune further with online AWAC to maximize performance. As we show in
our experiments, this finetuning stage can significantly improve performance
on difficult tasks. This combination of design decisions is novel to our
approach. Though it does combine a number of parts that are based on prior
work, as we demonstrate in our experiments, the particular combination of
parts we employ is important for good final performance. We describe both
stages in detail in Algorithm 1 and 2. To avoid clutter, we describe the
algorithm when batch size is 1 during data sampling (e.g. Algo. 1 line 4, 5),
when in practice we sample a mini-batch. For reproducibility, we include
network architectures, hyperparameters and other implementation details on our
website.

Algorithm 1 ODA Meta-training

0: $D^{i}_{demo}$ and $D^{i}_{offline}$ for each of $N$ training tasks,
learning rates $\eta_{1},\eta_{2},\eta_{3}$, temperature $\lambda$, KL weight
$\beta$

1: Init. encoder $q_{\phi}$, actor $\pi_{\theta}$, critic $Q_{\psi}$

2: while not converged do

3: for task $i=1,2,...,N$ do

4: Sample demo data as context $c_{i}\sim D^{i}_{demo}$

5: Sample offline data $(s,a,s^{\prime},a^{\prime},r)\sim D^{i}_{offline}$

6: Sample task variable $z_{i}\sim q_{\phi}(\cdot|c_{i})$

7: $y=r(s,a)+\gamma\mathop{\mathbb{E}}_{s^{\prime},a^{\prime}\sim
D^{i}_{offline}}Q_{\psi}(s^{\prime},a^{\prime},z_{i})$

8: $\mathcal{L}^{i}_{critic}=(Q_{\psi}(s,a,z_{i})-y)^{2}$

9:
$\mathcal{L}^{i}_{actor}=-\log\pi_{\theta}(a|s,z_{i})\exp(\frac{1}{\lambda}A^{\pi}(s,a,z_{i}))$

10: $\mathcal{L}^{i}_{KL}=\beta
D_{KL}(q_{\phi}(\cdot|c_{i})\|\mathcal{N}(0,I))$

11: end for

12:
$\phi\leftarrow\phi-\eta_{1}\nabla_{\phi}\sum_{i}(\mathcal{L}^{i}_{critic}+\mathcal{L}^{i}_{KL})$

13:
$\theta\leftarrow\theta-\eta_{2}\nabla_{\theta}\sum_{i}\mathcal{L}^{i}_{actor}$

14: $\psi\leftarrow\psi-\eta_{3}\nabla_{\psi}\sum_{i}\mathcal{L}^{i}_{critic}$

15: end while

Algorithm 2 ODA Adaptation and Finetuning

0: Test task demo $D_{test}$, learning rates $\eta_{1},\eta_{2},\eta_{3}$,
temperature $\lambda$, KL weight $\beta$, Pretrained $\pi_{\theta}$,
$Q_{\psi}$, $q_{\phi}$

1: Init. empty online buffer $\mathcal{B}$

2: Sample demo data as context $c\sim D_{test}$

3: Sample task variable $z\sim q_{\phi}(\cdot|c)$

4: Evaluate policy $\pi_{\theta}(\cdot|s,z)$, exit if policy solves the task

5: while not converged do

6: collect trajectory $\uptau$ by executing policy $\pi_{\theta}(\cdot|s,z)$

7: add $\uptau$ to $\mathcal{B}$

8: Sample demo data as context $c\sim D_{test}$

9: Sample offline data $(s,a,s^{\prime},r)\sim\mathcal{B}$

10: Sample task variable $z\sim q_{\phi}(\cdot|c)$

11: Calculate $\mathcal{L}_{actor}$, $\mathcal{L}_{critic}$,
$\mathcal{L}_{KL}$ same as Algo.1

12:
$\phi\leftarrow\phi-\eta_{1}\nabla_{\phi}(\mathcal{L}_{critic}+\mathcal{L}_{KL})$

13: $\theta\leftarrow\theta-\eta_{2}\nabla_{\theta}\mathcal{L}_{actor}$

14: $\psi\leftarrow\psi-\eta_{3}\nabla_{\psi}\mathcal{L}_{critic}$

15: end while

Figure 3: Offline training tasks. For each of the 11 training tasks, we run
DDPGfD [5] to collect data for offline training.

Figure 4: Test tasks and adaptation performance. We evaluate adaptation with 9
test tasks and compare it to AWAC [27]. Our method ODA achieves 100% success
rate for 6/9 tasks, and outperforms AWAC for 8/9 tasks. Adaptation only uses
the demos for each task, without additional online interaction.

Figure 5: Finetuning performance. For tasks where adapting with demos does not
reach 100/100, we finetune with online interaction. We plot the training
success rate and wall-clock time for ODA, finetuning AWAC[27], DDPGfD[5], as
well as behavioral cloning. Our method reduces wall-clock time by 11.75x
compared to DDPGfD, and outperforms AWAC. We note that the length of bar plot
reflects the number of episodes, and successful episodes are significantly
shorter wall-clock.

##  V EXPERIMENTS

We evaluate our algorithm by adapting to 12 new tasks (see Fig. 4 and 6),
studying the following questions:

  1. 1.

How well does our method perform compared to standard offline RL?

  2. 2.

Can finetuning provide fast adaptation even for very different novel tasks,
where just running the meta-trained encoder fails?

  3. 3.

Can our method handle the challenging tasks in Fig. 6, which are outside of
the training distribution?

  4. 4.

Does our meta-adaptation improve with more offline training data?

#### Experiment setup

In our insertion tasks, the robot starts holding an object (e.g., a
connector), and must position it into a goal pose where it is inserted into a
socket. The reward is a binary indicator for whether the insertion is
successful, and the episode terminates either when the robot succeeds, or the
episode length exceeds a maximum time limit. We run experiments with a KUKA
iiwa7 robot. The agent controls the TCP twist of the robot at 10Hz, which is
tracked and interpolated by a downstream impedance controller at 1000 Hz. The
observation to our agent consists of the robot’s TCP pose, velocity, and the
wrench measured at the tool tip. For both training and test tasks, we express
robot TCP information in a relative coordinate system, with its origin located
at the TCP, as in prior work [9]. We add perturbation noise from a uniform
distribution $\mathcal{U}$[-1mm, 1mm] at the beginning of each episode, at
both training and test time. The policy does not have access to this noise,
and therefore must be robust to the perturbations.

#### Offline dataset

We collected data for 11 different plug-socket pairs, as shown in Figure 3.
The data was collected by running DDPGfD [5] and saving the entire replay
buffer as offline data for our method, as well as demonstrations for each
task. On average, we obtain 500 episodes of RL interaction data and 20
episodes of human demonstration data for each of the training tasks. To
further improve the diversity of the dataset, we also reran the final DDPGfD
policies with added noise to provide good coverage of the workspace. We list
dataset details in our website.

###  V-A Adaptation via Learned Encoder

In our first set of experiments, we evaluate how well our learned encoder can
adapt to new tasks using only the demonstrations for those tasks. We use 9
test tasks as shown in Fig. 4. We collect 20 demonstrations for each task and
feed them into our encoder to infer the task information $z$. We compare to
standard AWAC pretrained on the same data, with demos added into the offline
data. As shown in Fig. 4, our method performs well across all of the tasks,
succeeding 100/100 times on 6 of the 9, while AWAC attains comparatively lower
success rates.

AWAC learns a single policy for all connectors, and relies on this policy to
generalize effectively, whereas our approach can adapt to each task using a
small amount of demonstration data. Let’s look at a motivating example to
further illustrate this difference. We tested pretrained policies both from
our method and AWAC on one of the training tasks, “E-Model-10p” (see Fig. 3).
Since this is a training task, we expect both methods to do well, but AWAC
only has a success rate of 49%, while our method gets 100%. This task requires
fine-grained adjustments to insert, which are quite distinct from other
connectors in the dataset, making it hard for the non-adaptive AWAC policy to
succeed on this task and all the others with the same strategy. On the other
hand, our method succeeds by determining the right strategy from the provided
demonstrations. Of course, we could finetune AWAC with additional data, which
would make for a more fair comparison. We study precisely this in the next set
of experiments.

###  V-B Handling Out-of-Distribution Tasks via Finetuning

If the task at test-time is out-of-distribution relative to tasks seen during
offline training, simply utilizing the demos with the encoder might not be
enough to succeed. In this section, we study how finetuning can alleviate this
issue, using the three tasks where our method fails to achieve 100%
performance from only the demos. In Fig. 5, we show finetuning results for (1)
our method, (2) finetuning AWAC with demonstrations added to initial buffer,
(3) training DDPG from demos, and (4) the performance of behavioral cloning
with the demos. (4) fails to reach 100% because imitating human demonstrations
will not produce a policy that is robust to starting pose randomization. (1),
(2), and (3) all attain a success rate of 100/100 after finetuning, while our
method finetunes significantly faster as shown in the bar plot below each
graph in Fig. 5. On average, we finish training 11.75 times faster than
DDPGfD. This is not completely surprising because the power of offline
training: it allows us to leverage the rich information in offline datasets.
We also find that in the two tasks where our method outperforms AWAC (left and
middle plot of Fig. 5), we finish training 1.73 times faster than AWAC. For
the third task (right plot of Fig. 5), our algorithm takes 5 mins to account
for the performance difference. Note the speedup comparison here is
conservative: we do not include the tasks where AWAC fails but our method
achieves 100% without any finetuning at all (i.e., the first 4 tasks in Fig.
4). It’s also worth noting that we have tried to pretrain a policy with DDPG
on the offline datasets, but it showed 0% success rate on test tasks. We then
finetune the policy pretrained with DDPG loss, while it takes similar amount
of time to succeed as randomly initialized networks. This validates the
necessity of offline RL training. We did not include it in the figure for
presentation clarity.

###  V-C Out-Of-Distribution Challenge Tasks

Figure 6: Challenge tasks We challenge our method with three additional out of
training distribution tasks. ODA is able to solve all three with 100/100
success within 30 minutes.

In realistic industrial assembly tasks, the robot may be tasked with problems
that differ significantly from the conditions seen in training. To study how
our method performs in these settings, we evaluate three additional challenge
tasks that differ significantly from the training connectors and present a
particular physical challenge (see Fig. 6): (1) RAM insertion, (2) network
card insertion, and (3) complex shape insertion. (1) and (2) are require
handling delicate electronics and significant application of force. Training
from scratch would be impractical here, since an untrained policy would easily
damage the circuit board. (3) is challenging because of the small clearance
and the very different contact shape compared to training tasks. We do not
inject uniform noise to the starting pose to make these three tasks more
tractable.

As shown in Figure 6, our method is able to solve all three tasks 100/100
within 30 minutes. For network card insertion, we are able to succeed without
any finetuning. We posit that inserting the RAM is significantly harder than
the network card because it is much longer (288 vs. 18 pins), requires high
precision along the yaw axis, and requires considerable application of force.
These characteristics makes it very different from all the training tasks, and
generally difficult for a robot to perform. During the 25 minutes required to
adapt the policy to perform this task 100% of the time, no damage is caused to
the RAM or the motherboard.

###  V-D Scaling with More Data

Figure 7: Scaling with more data. We plot the adaptation performance of three
tasks as the number of training tasks increases. The success rate increases
steadily as the algorithm has access to more data.

Lastly, we analyze how the number of training tasks affects adaptation
performance of our method, shown in Fig. 7. We evaluate the meta-adaptation
performance across three tasks, and it continues to improve as we increase the
number of tasks in the dataset from 1 to 11.

##  VI DISCUSSION

We introduced an offline meta-RL algorithm, ODA, that can meta-learn an
adaptive policy from offline data, quickly adapt based on a small number of
user-provided demonstrations for a new task, and then further adapt through
online finetuning. We show that this approach is highly effective at adapting
to new connector insertion tasks, learning to insert many connectors with a
100% success rate after just the initial demonstrations, and finetuning other
connectors to a 100% success rate with minutes of additional online training.
Although our algorithm is composed of parts proposed in prior work, the
particular combination and the use of demonstrations for adapting in
contextual meta-RL is novel to our method. Promising future directions include
incorporating visual perception, extending the approach to other tasks, and
extending it into a lifelong learning framework where each new task is
included into a lifelong meta-training process.

## References

  * [1] “Global Robotics Market - Growth, Trends, COVID-19 Impact, and Forecasts (2021 - 2026)”, 2021  URL: https://www.yahoo.com/now/global-robotics-market-growth-trends-125900343.html
  * [2] “Introducing Intrinsic”, 2021  URL: https://blog.x.company/introducing-intrinsic-1cf35b87651
  * [3] “Alphabet to launch robotics firm Intrinsic under its other bets unit”, 2021  URL: https://www.reuters.com/technology/alphabet-launch-robotics-firm-intrinsic-under-its-other-bets-unit-2021-07-23/
  * [4] Ben Wang  “The Future of Manufacturing: A New Perspective”  In _Engineering_ 4.5, 2018, pp. 722–728  DOI: https://doi.org/10.1016/j.eng.2018.07.020
  * [5] Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rothörl, Thomas Lampe and Martin Riedmiller  “Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards”  In _arXiv preprint arXiv:1707.08817_ , 2017
  * [6] A. Tamar, G. Thomas, T. Zhang, S. Levine and P. Abbeel  “Learning from the Hindsight Plan – Episodic MPC Improvement”  In _ArXiv e-prints_ , 2016  arXiv:1609.09001 [cs.RO]
  * [7] Matej Vecerík, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Manfred Otto Heess, Thomas Rothörl, Thomas Lampe and Martin A. Riedmiller  “Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewards”  In _ArXiv_ abs/1707.08817, 2017
  * [8] Garrett Thomas, Melissa Chien, Aviv Tamar, Juan Aparicio Ojea and P. Abbeel  “Learning Robotic Assembly from CAD”  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , 2018, pp. 1–9
  * [9] Jianlan Luo, Oleg Sushkov, Rugile Pevceviciute, Wenzhao Lian, Chang Su, Mel Vecerik, Ning Ye, Stefan Schaal and Jonathan Scholz  “Robust Multi-Modal Policies for Industrial Assembly via Reinforcement Learning and Demonstrations: A Large-Scale Study”  In _Proceedings of Robotics: Science and Systems_ , 2021  DOI: 10.15607/RSS.2021.XVII.088
  * [10] Mel Vecerik, Oleg Sushkov, David Barker, Thomas Rothörl, Todd Hester and Jon Scholz  “A practical approach to insertion with variable socket position using deep reinforcement learning”  In _2019 International Conference on Robotics and Automation (ICRA)_ , 2019, pp. 754–760  IEEE
  * [11] Jianlan Luo, Eugen Solowjow, Chengtao Wen, Juan Aparicio Ojea, Alice M Agogino, Aviv Tamar and Pieter Abbeel  “Reinforcement learning on variable impedance controller for high-precision robotic assembly”  In _2019 International Conference on Robotics and Automation (ICRA)_ , 2019, pp. 3080–3087  IEEE
  * [12] Gerrit Schoettler, Ashvin Nair, Jianlan Luo, Shikhar Bahl, Juan Aparicio Ojea, Eugen Solowjow and Sergey Levine  “Deep Reinforcement Learning for Industrial Insertion Tasks with Visual Inputs and Natural Rewards”  In _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2020, pp. 5548–5555  DOI: 10.1109/IROS45743.2020.9341714
  * [13] Sergey Levine, Chelsea Finn, Trevor Darrell and Pieter Abbeel  “End-to-end training of deep visuomotor policies”  In _The Journal of Machine Learning Research_ 17.1 JMLR. org, 2016, pp. 1334–1373
  * [14] Sergey Levine, Nolan Wagener and Pieter Abbeel  “Learning Contact-Rich Manipulation Skills with Guided Policy Search”  In _CoRR_ abs/1501.05611, 2015  arXiv: http://arxiv.org/abs/1501.05611
  * [15] Tuomas Haarnoja, Vitchyr Pong, Aurick Zhou, Murtaza Dalal, Pieter Abbeel and Sergey Levine  “Composable Deep Reinforcement Learning for Robotic Manipulation”  In _2018 IEEE International Conference on Robotics and Automation (ICRA)_ , 2018, pp. 6244–6251  DOI: 10.1109/ICRA.2018.8460756
  * [16] Tadanobu Inoue, Giovanni De Magistris, Asim Munawar, Tsuyoshi Yokoya and Ryuki Tachibana  “Deep Reinforcement Learning for High Precision Assembly Tasks”  In _CoRR_ abs/1708.04033, 2017  arXiv: http://arxiv.org/abs/1708.04033
  * [17] Chelsea Finn, Pieter Abbeel and Sergey Levine  “Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks”  In _Proceedings of the 34th International Conference on Machine Learning_ 70, Proceedings of Machine Learning Research  PMLR, 2017, pp. 1126–1135  URL: https://proceedings.mlr.press/v70/finn17a.html
  * [18] Kate Rakelly, Aurick Zhou, Chelsea Finn, Sergey Levine and Deirdre Quillen  “Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables”  In _Proceedings of the 36th International Conference on Machine Learning_ 97, Proceedings of Machine Learning Research  PMLR, 2019, pp. 5331–5340  URL: https://proceedings.mlr.press/v97/rakelly19a.html
  * [19] Abhishek Gupta, R. Mendonca, Yuxuan Liu, P. Abbeel and Sergey Levine  “Meta-Reinforcement Learning of Structured Exploration Strategies”  In _NeurIPS_ , 2018
  * [20] Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, Ilya Sutskever and Pieter Abbeel  “RL2: Fast Reinforcement Learning via Slow Reinforcement Learning”, 2016  arXiv:1611.02779 [cs.AI]
  * [21] Ignasi Clavera, Anusha Nagabandi, Ronald S. Fearing, Pieter Abbeel, Sergey Levine and Chelsea Finn  “Learning to Adapt: Meta-Learning for Model-Based Control”  In _CoRR_ abs/1803.11347, 2018  arXiv: http://arxiv.org/abs/1803.11347
  * [22] Mitchell Wortsman, Kiana Ehsani, Mohammad Rastegari, Ali Farhadi and Roozbeh Mottaghi  “Learning to Learn How to Learn: Self-Adaptive Visual Navigation Using Meta-Learning”  In _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2019, pp. 6743–6752
  * [23] Tony Z. Zhao, Anusha Nagabandi, Kate Rakelly, Chelsea Finn and Sergey Levine  “MELD: Meta-Reinforcement Learning from Images via Latent State Models”  In _CoRR_ abs/2010.13957, 2020  arXiv: https://arxiv.org/abs/2010.13957
  * [24] Gerrit Schoettler, Ashvin Nair, Juan Aparicio Ojea, Sergey Levine and Eugen Solowjow  “Meta-Reinforcement Learning for Robotic Industrial Insertion Tasks”  In _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_ , 2020, pp. 9728–9735  DOI: 10.1109/IROS45743.2020.9340848
  * [25] Sergey Levine, Aviral Kumar, George Tucker and Justin Fu  “Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems”, 2020  arXiv:2005.01643 [cs.LG]
  * [26] Aviral Kumar, Aurick Zhou, George Tucker and Sergey Levine  “Conservative Q-Learning for Offline Reinforcement Learning”, 2020  arXiv:2006.04779 [cs.LG]
  * [27] Ashvin Nair, Murtaza Dalal, Abhishek Gupta and Sergey Levine  “Accelerating Online Reinforcement Learning with Offline Datasets”, 2020  arXiv:2006.09359 [cs.LG]
  * [28] Avi Singh, Albert Yu, Jonathan Yang, Jesse Zhang, Aviral Kumar and Sergey Levine  “COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning”, 2020  arXiv:2010.14500 [cs.LG]
  * [29] Ron Dorfman and Aviv Tamar  “Offline Meta Learning of Exploration”  In _arXiv: Learning_ , 2020
  * [30] Ajay Mandlekar, Fabio Ramos, Byron Boots, Li Fei-Fei, Animesh Garg and Dieter Fox  “IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data”  In _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 4414–4420
  * [31] Scott Fujimoto and Shixiang Shane Gu  “A Minimalist Approach to Offline Reinforcement Learning”  In _ArXiv_ abs/2106.06860, 2021
  * [32] Ron Dorfman and Aviv Tamar  “Offline Meta Reinforcement Learning”  In _ArXiv_ abs/2008.02598, 2020
  * [33] Vitchyr H. Pong, Ashvin Nair, Laura Smith, Catherine Huang and Sergey Levine  “Offline Meta-Reinforcement Learning with Online Self-Supervision”, 2021  arXiv:2107.03974 [cs.LG]
  * [34] Eric Mitchell, Rafael Rafailov, Xue Bin Peng, Sergey Levine and Chelsea Finn  “Offline Meta-Reinforcement Learning with Advantage Weighting”  In _ICML_ , 2021
  * [35] Ron Dorfman and Aviv Tamar  “Offline Meta Reinforcement Learning”  In _CoRR_ abs/2008.02598, 2020  arXiv: https://arxiv.org/abs/2008.02598
  * [36] Lanqing Li, Rui Yang and Dijun Luo  “Efficient Fully-Offline Meta-Reinforcement Learning via Distance Metric Learning and Behavior Regularization”  In _ArXiv_ abs/2010.01112, 2021
  * [37] T. Lillicrap, Jonathan J. Hunt, A. Pritzel, N. Heess, T. Erez, Yuval Tassa, D. Silver and Daan Wierstra  “Continuous control with deep reinforcement learning”  In _CoRR_ abs/1509.02971, 2016
  * [38] W. Haskiya, K. Maycock and J. Knight  “Robotic assembly: chamferless peg-hole assembly”  In _Robotica_ 17, 1999, pp. 621–634
  * [39] Herman Bruyninckx, Stefan Dutré and Joris De Schutter  “Peg-on-hole: a model based solution to peg and hole alignment”  In _Proceedings of 1995 IEEE International Conference on Robotics and Automation_ 2, 1995, pp. 1919–1924 vol.2
  * [40] S. Chhatpar and M. Branicky  “Search strategies for peg-in-hole assemblies with position uncertainty”  In _Proceedings 2001 IEEE/RSJ International Conference on Intelligent Robots and Systems. Expanding the Societal Role of Robotics in the the Next Millennium (Cat. No.01CH37180)_ 3, 2001, pp. 1465–1470 vol.3
  * [41] D. Whitney  “Force Feedback Control of Manipulator Fine Motions”  In _Journal of Dynamic Systems Measurement and Control-transactions of The Asme_ 99, 1977, pp. 91–97
  * [42] D.. Whitney  “Quasi-Static Assembly of Compliantly Supported Rigid Parts”  In _Journal of Dynamic Systems, Measurement, and Control_ 104.1, 1982, pp. 65–77  DOI: 10.1115/1.3149634
  * [43] Hyeonjun Park, J. Bae, Jae-Han Park, M. Baeg and Jaeheung Park  “Intuitive peg-in-hole assembly strategy with a compliant manipulator”  In _IEEE ISR 2013_ , 2013, pp. 1–5
  * [44] Kamal Sharma, Varsha Shirwalkar and Prabir K. Pal  “Intelligent and environment-independent Peg-In-Hole search strategies”  In _2013 International Conference on Control, Automation, Robotics and Embedded Systems (CARE)_ , 2013, pp. 1–6  DOI: 10.1109/CARE.2013.6733716
  * [45] W.S. Newman, Y. Zhao and Y.-H. Pao  “Interpretation of force and moment signals for compliant peg-in-hole assembly”  In _Proceedings 2001 ICRA. IEEE International Conference on Robotics and Automation (Cat. No.01CH37164)_ 1, 2001, pp. 571–576 vol.1  DOI: 10.1109/ROBOT.2001.932611
  * [46] Te Tang, Hsien-Chung Lin, Yu Zhao, Wenjie Chen and Masayoshi Tomizuka  “Autonomous alignment of peg and hole by force/torque measurement for robotic assembly”  In _2016 IEEE International Conference on Automation Science and Engineering (CASE)_ , 2016, pp. 162–167  DOI: 10.1109/COASE.2016.7743375
  * [47] O. Spector and Dotan Di Castro  “InsertionNet - A Scalable Solution for Insertion”  In _IEEE Robotics and Automation Letters_ 6, 2021, pp. 5509–5516
  * [48] Tobias Johannink, Shikhar Bahl, Ashvin Nair, Jianlan Luo, Avinash Kumar, Matthias Loskyll, Juan Aparicio Ojea, Eugen Solowjow and Sergey Levine  “Residual Reinforcement Learning for Robot Control”  In _CoRR_ abs/1812.03201, 2018  arXiv: http://arxiv.org/abs/1812.03201
  * [49] Chelsea Finn, Tianhe Yu, Tianhao Zhang, Pieter Abbeel and Sergey Levine  “One-shot visual imitation learning via meta-learning”  In _Conference on Robot Learning_ , 2017, pp. 357–368  PMLR