# The rank of the $2$-class group of some fields with large degree

Mohamed Mahmoud CHEMS-EDDIN  Mohamed Mahmoud CHEMS-EDDIN: Mohammed First
University, Mathematics Department, Sciences Faculty, Oujda, Morocco
<EMAIL_ADDRESS>
###### Abstract.

Let $n\geq 3$ be an integer and $d$ an odd square-free integer. We shall
compute the rank of the $2$-class group of
$L_{n,d}:=\mathbb{Q}(\zeta_{2^{n}},\sqrt{d})$, when all the prime divisors of
$d$ are congruent to $\pm 3\pmod{8}$ or $9\pmod{16}$.

###### Key words and phrases:

Cyclotomic $Z_{2}$-extension, $2$-rank, $2$-class group.

###### 2010 Mathematics Subject Classification:

11R29; 11R23; 11R27; 11R04.

##  1\. Introduction

The explicit computation of the rank of the $2$-class group of a given number
field $K$ is one of the difficult problems of algebraic number theory,
especially for fields with large degree. For many years ago, several authors
studied this problem for number fields of degree $2$ or $4$ (cf. [5, 9, 8]).
The methods used therein are not enough to deal with the same problem for
number fields with large degree, although recently some papers studied this
question for some number fields of degree $2^{n}$ (cf. [6, 2]). Using the
cyclotomic units and some results of the theory of the cyclotomic
$\mathbb{Z}_{2}$-extension, we extend these methods to compute the rank of the
$2$-class group of some fields of degree $2^{n}$ of the form
$L_{n,d}:=\mathbb{Q}(\zeta_{2^{n}},\sqrt{d})$, where $d$ is an odd square-free
integer and $n\geq 3$ is a positive integer.

Let $\mathds{k}:=\mathbb{Q}(\sqrt{d},\sqrt{-1})$,
$\mathbb{Q}(\sqrt{-2},\sqrt{d})$ or $\mathbb{Q}(\sqrt{-2},\sqrt{-d})$. Then
the cyclotomic $\mathbb{Z}_{2}$-extension of $\mathds{k}$ is

$\mathds{k}(\sqrt{2})\subset\mathds{k}(\sqrt{2+\sqrt{2}})\subset\mathds{k}(\sqrt{2+\sqrt{2+\sqrt{2}}})\subset...$

which coincides with the tower $L_{3,d}\subset L_{4,d}\subset...\subset
L_{n,d}\subset...$

The present work is a continuation of our previous work [2], in which we
computed the rank of the $2$-class group of $L_{n,d}$ when the prime divisors
of $d$ are congruent to $3$ or $5\pmod{8}$. Thus, we compute the rank of the
$2$-class group of $L_{n,d}$ when the prime divisors of $d$ are congruent to
$\pm 3\pmod{8}$ or $9\pmod{16}$. Furthermore, we give the rank of the
$2$-class group of $L_{n,d}$ in terms of that of $L_{4,d}$, when the prime
divisors of $d$ are congruent to $\pm 3\pmod{8}$ or $\pm 7\pmod{16}$.

## Notations

The next notations will be used for the rest of this article:

  1. $\bullet$

$d$: An odd square-free integer,

  2. $\bullet$

$n$: A positive integer $\geq 3$,

  3. $\bullet$

$K_{n}$: $\mathbb{Q}(\zeta_{2^{n}})$,

  4. $\bullet$

$K_{n}^{+}$: The maximal real subfield of $K_{n}$,

  5. $\bullet$

$L_{n,d}$: $K_{n}(\sqrt{d})$,

  6. $\bullet$

$\mathcal{N}$: The norm map of the extension $L_{n,d}/K_{n}$,

  7. $\bullet$

$E_{k}$: The unit group of a number field $k$,

  8. $\bullet$

$\mathcal{O}_{k}$: The ring of integers of a number field $k$,

  9. $\bullet$

${Cl}(k)$: The class group of a number field $k$,

  10. $\bullet$

${Cl}_{2}(k)$: The $2$-class group of a number field $k$,

  11. $\bullet$

$rank_{2}(Cl(L_{n,d}))$: The rank of the $2$-class group of $L_{n,d}$,

  12. $\bullet$

$\mathfrak{p}$: A prime ideal of $K_{n}$,

  13. $\bullet$

$\left(\frac{\alpha,d}{\mathfrak{p}}\right)$: The quadratic norm residue
symbol for $L_{n,d}/K_{n}$,

  14. $\bullet$

$\left(\frac{\alpha}{\mathfrak{p}}\right)$: The quadratic power residue
symbol.

##  2\.  The preliminary results

Let us first collect some results that will be used in what follows.

######  Lemma 1 ([3]).

Let $K/k$ be a $\mathbb{Z}_{2}$-extension, $k_{n}$ its n-th layer and $n_{0}$
an integer such that any prime of $K$ which is ramified in $K/k$ is totally
ramified in $K/k_{n_{0}}$. If there exists an integer $n\geq n_{0}$ such that
$rank_{2}({Cl}(k_{n}))=rank_{2}({Cl}(k_{n+1}))$, then
$rank_{2}({Cl}(k_{m}))=rank_{2}({Cl}(k_{n}))$ for all $m\geq n$.

######  Lemma 2 ([10, Lamma 8.1, Corollary 4.13]).

  1. 1.

The cyclotomic units of $K_{n}^{+}$ are generated by $-1$ and

$\xi_{k,n}=\zeta_{2^{n}}^{(1-k)/2}\frac{1-\zeta_{2^{n}}^{k}}{1-\zeta_{2^{n}}},$

where $k$ is an odd integer such that $1<k<2^{n-1}$.

  2. 2.

The cyclotomic units of $K_{n}$ are generated by $\zeta_{2^{n}}$ and the
cyclotomic units of $K_{n}^{+}$ .

  3. 3.

The Hasse’s index $Q$ of $K_{n}$ equals 1.

######  Lemma 3 ([4]).

Let $K/k$ be a quadratic extension. If the class number of $k$ is odd, then
the rank of the $2$-class group of $K$ is given by

$rank_{2}({Cl}(K))=t-1-e,$

where $t$ is the number of ramified primes (finite or infinite) in the
extension $K/k$ and $e$ is defined by $2^{e}=[E_{k}:E_{k}\cap
N_{K/k}(K^{*})]$.

######  Remark 1.

Note that a unit $u$ of $K_{n}$ is a norm in $L_{n,d}/K_{n}$ if and only if
$\left(\frac{u,d}{\mathfrak{p}}\right)=1$, for all prime ideal $\mathfrak{p}$
of $K_{n}$ ramified in $L_{n,d}$.

Next, we need to characterize ideals of $K_{n}=\mathbb{Q}(\zeta_{2^{n}})$ that
ramify in $L_{n,d}=K_{n}(\sqrt{d})$. Let $n\geq 3$ and $d$ be an odd square-
free integer, then $d$ is congruent to $1$ or $3\pmod{4}$. So $2$ is
unramified in either $\mathbb{Q}(\sqrt{d})$ or $\mathbb{Q}(\sqrt{-d})$. Thus,
the ramification index of $2$ in $L_{n,d}$ is strictly inferior to $2^{n}$. As
$2$ is totally ramified in $K_{n}:=\mathbb{Q}(\zeta_{2^{n}})$, so the prime
ideal of $K_{n}$ lying over $2$ is unramified in $L_{n,d}$, as otherwise the
ramification index of $2$ in $L_{n,d}$ will be $2^{n}$, which is absurd. Hence
we prove the following result:

$\mathbb{Q}$$\mathbb{Q}(\sqrt{d})$$\mathbb{Q}(\sqrt{-d})$$\mathbb{Q}(\zeta_{2^{n}})$$\mathbb{Q}(\zeta_{2^{n}},\sqrt{d})$

######  Lemma 4.

Let $d$ be an odd square-free integer. Then a prime ideal $\mathfrak{p}$ of
$K_{n}$ is ramified in $L_{n,d}/K_{n}$ if and only if it divides $d.$

######  Proposition 1.

Let $n\geq 5$ and $p$ be a rational prime. Then $p$ decomposes into $4$ primes
of $K_{n}$ if and only if $p\equiv 7\text{ or }9\pmod{16}$.

###### Proof.

Let $K_{\infty}$ denote the cyclotomic $\mathbb{Z}_{2}$-extension of $K$. Note
that $\mathrm{Gal}(K_{\infty}:K)\simeq\mathbb{Z}_{2}$. Hence the decomposition
field of a prime above $p$ must be some $K_{n}$. Therefore if a prime
$\mathfrak{p}$ of $K_{n}$ is inert in $K_{n+1}$, then $\mathfrak{p}$ is inert
in $K_{\infty}/K_{n}$. Then Proposition 1 is from the easily verified fact
that a prime $p$ decomposes into $4$ primes in $K_{4}=\mathbb{Q}(\zeta_{16})$
and $K_{5}=\mathbb{Q}(\zeta_{32})$ if and only if $p\equiv 7,9\pmod{16}$. ∎

######  Remark 2.

Let $p\equiv 7\text{ or }9\pmod{16}$ be a prime. We have

  1. 1.

If $p\equiv 9\pmod{16}$, then $p$ decomposes into product of four prime ideals
in $K_{3}$ and $K_{4}$.

  2. 2.

If $p\equiv 7\pmod{16}$, then $p$ decomposes into product of two prime ideals
in $K_{3}$ and into product of four prime ideals in $K_{4}$.

######  Proposition 2 ([2]).

Let $m\geq 4$ and $p$ be a rational prime. Then $p$ decomposes into the
product of $2$ primes of $K_{m}$ if and only if $p\equiv 3$ or $5\pmod{8}$.

Now we shall do some computations:

######  Lemma 5 ([2]).

Let $n\geq 3$ be a positive integer and $p$ be a prime number. Let
$\mathfrak{p}_{K_{n}}$ denote a prime ideal of $K_{n}$ above $p$. We have

  1. 1.

If $p\equiv 5\pmod{8}$. Then

$\left(\frac{\zeta_{2^{n}},p}{\mathfrak{p}_{K_{n}}}\right)=-1\text{ and
}\left(\frac{\xi_{k,n},p}{\mathfrak{p}_{K_{n}}}\right)=1.$

  2. 2.

If $p\equiv 3\pmod{8}$. Then

$\left(\frac{\zeta_{2^{n}},p}{\mathfrak{p}_{K_{n}}}\right)=-1\text{ and
}\left(\frac{\xi_{k,n},p}{\mathfrak{p}_{K_{n}}}\right)=\left\\{\begin{array}[]{ccc}-1,&\text{
if }k\equiv\pm 3\pmod{8}&\\\ 1,&\text{ elsewhere. }&\end{array}\right.$

######  Lemma 6.

Let $n\geq 3$ be an integer and $p$ a prime congruent to $9\pmod{16}$,
$\mathfrak{p}_{K_{n}}$ a prime ideal of $K_{n}$ dividing $p$.

$\left(\frac{\zeta_{2^{n}},p}{\mathfrak{p}_{K_{n}}}\right)=-1\text{ and
}\left(\frac{\xi_{k,n},p}{\mathfrak{p}_{K_{n}}}\right)=\left\\{\begin{array}[]{cl}-\left(\frac{2}{p}\right)_{4},&\text{
if }k\equiv\pm 3\pmod{8}\\\ 1,&\text{elsewhere}.\\\ \end{array}\right.$

###### Proof.

For all $n\geq 3$, the prime $p$ decomposes into product of four prime ideals
of $K_{n}$ (see Proposition 1), denote by $\mathfrak{p}_{K_{n}}$ one of them
(such that $\mathfrak{p}_{K_{n-1}}\subset\mathfrak{p}_{K_{n}}$). We have
$\zeta_{2^{n}}^{2}=\zeta_{2^{{n-1}}}$, so the minimal polynomial of
$\zeta_{2^{n}}$ over $K_{{n-1}}$ is $X^{2}-\zeta_{n-1}$ and
$N_{K_{n}/K_{n-1}}(\zeta_{2^{n}})=-\zeta_{2^{n-1}}$. Thus

$\begin{array}[]{ll}\left(\frac{\zeta_{2^{n}},p}{\mathfrak{p}_{{K_{n}}}}\right)=\left(\frac{-\zeta_{2^{n-1}},p}{\mathfrak{p}_{{K_{n-1}}}}\right)=\left(\frac{\zeta_{2^{n-1}},p}{\mathfrak{p}_{{K_{n-1}}}}\right)=...=\left(\frac{\zeta_{8},p}{\mathfrak{p}_{{K_{3}}}}\right)=(-1)^{\frac{p-1}{8}}=-1(\text{see
\cite[cite]{[\@@bibref{}{chemsZkhnin1}{}{}]}}).\end{array}$
$\begin{array}[]{ll}\left(\frac{\xi_{k,n},p}{\mathfrak{p}_{K_{n}}}\right)&=\left(\frac{\zeta_{2^{n}}^{(1-k)/2},p}{\mathfrak{p}_{K_{n}}}\right)\left(\frac{\frac{1-\zeta_{2^{n}}^{k}}{1-\zeta_{2^{n}}},p}{\mathfrak{p}_{K_{n}}}\right)\\\
&=(-1)^{(1-k)/2}\left(\frac{(1-\zeta_{2^{n}}^{k})(1-\zeta_{2^{n}}),p}{\mathfrak{p}_{K_{n}}}\right)\\\
&=(-1)^{(1-k)/2}\left(\frac{1-\zeta_{8}^{k},p}{\mathfrak{p}_{{K_{3}}}}\right)\left(\frac{1-\zeta_{8},p}{\mathfrak{p}_{{K_{3}}}}\right)\\\
&=(-1)^{(3-k)/2}\left(\frac{\zeta_{8}^{-1},p}{\mathfrak{p}_{{K_{3}}}}\right)\left(\frac{1-\zeta_{8}^{k},p}{\mathfrak{p}_{{K_{3}}}}\right)\left(\frac{1-\zeta_{8},p}{\mathfrak{p}_{{K_{3}}}}\right)\\\
&\vspace{0.2cm}=\left\\{\begin{array}[]{cc}\left(\frac{\varepsilon_{2},p}{\mathfrak{p}_{K_{3}}}\right),&\text{
if }k\equiv 3\pmod{8}\\\
\left(\frac{1+\zeta_{8},p}{\mathfrak{p}_{{K_{3}}}}\right)\left(\frac{1-\zeta_{8},p}{\mathfrak{p}_{{K_{3}}}}\right),&\text{
if }k\equiv 5\pmod{8}\\\
-\left(\frac{1-\zeta_{8}^{-1},p}{\mathfrak{p}_{{K_{3}}}}\right)\left(\frac{1-\zeta_{8},p}{\mathfrak{p}_{{K_{3}}}}\right),&\text{
if }k\equiv 7\pmod{8}\\\
\left(\frac{1-\zeta_{8},p}{\mathfrak{p}_{{K_{3}}}}\right)\left(\frac{1-\zeta_{8},p}{\mathfrak{p}_{{K_{3}}}}\right),&\text{
if }k\equiv 1\pmod{8}\\\ \end{array}\right.\\\
&\vspace{0.2cm}=\left\\{\begin{array}[]{cc}-\left(\frac{2}{p}\right)_{4},&\text{
if }k\equiv 3\pmod{8}\;(\text{see
\cite[cite]{[\@@bibref{}{chemsZkhnin1}{}{}]}})\vspace{0.2cm}\\\
\left(\frac{1-i,p}{\mathfrak{p}_{{K_{3}}}}\right),&\text{ if }k\equiv
5\pmod{8}\vspace{0.2cm}\\\
-\left(\frac{2-\sqrt{2},p}{\mathfrak{p}_{{K_{3}}}}\right),&\text{ if }k\equiv
7\pmod{8}\vspace{0.2cm}\\\ 1,&\text{ if }k\equiv 1\pmod{8}.\\\
\end{array}\right.\\\ \end{array}$

Using the proof of [1, Lemma 3.4] we get
$\left(\frac{1-i,p}{\;\mathfrak{p}_{{K_{3}}}}\right)=\left(\frac{1-i}{\;\mathfrak{p}_{{K_{3}}}}\right)=\left(\frac{1+i}{\;\mathfrak{p}_{{K_{3}}}}\right)=\left(\frac{\zeta_{8},p}{\;\mathfrak{p}_{{K_{3}}}}\right)\left(\frac{\sqrt{2},p}{\;\mathfrak{p}_{{K_{3}}}}\right)=\left(\frac{\zeta_{8},p}{\;\mathfrak{p}_{{K_{3}}}}\right)(-1)^{\frac{p-1}{8}}\left(\frac{\varepsilon_{2},p}{\;\mathfrak{p}_{{K_{3}}}}\right)=\left(\frac{\varepsilon_{2},p}{\;\mathfrak{p}_{{K_{3}}}}\right)=-\left(\frac{2}{p}\right)_{4}.$
Similarly we have
$\left(\frac{2-\sqrt{2},p}{\mathfrak{p}_{{K_{3}}}}\right)=\left(\frac{2+\sqrt{2}}{\mathfrak{p}_{{K_{3}}}}\right)=\left(\frac{\sqrt{2},p}{\mathfrak{p}_{{K_{3}}}}\right)\left(\frac{\varepsilon_{2},p}{\mathfrak{p}_{{K_{3}}}}\right)=(-1)^{\frac{p-1}{8}}\left(\frac{\varepsilon_{2},p}{\;\mathfrak{p}_{{K_{3}}}}\right)\left(\frac{\varepsilon_{2},p}{\mathfrak{p}_{{K_{3}}}}\right)=-1$
. Which achieves the proof. ∎

##  3\. The main results

The authors of [2] computed the rank of the $2$-class group of $L_{n,d}$, when
the prime divisors of $d$ are congruent to $3$ or $5\pmod{8}$. In this section
we shall compute the rank of the $2$-class group of $L_{n,d}$, then the prime
divisors of $d$ are congruent to $\pm 3\pmod{8}$ or $9\pmod{16}$.

######  Theorem 1.

Let $n\geq 3$ and $d>2$ be an odd composite square-free integer of prime
divisors congruent to $\pm 3\pmod{8}$ or $9\pmod{16}$. Let $r$ denote the
number of prime divisors of $d$ which are congruent to $3$ or $5\pmod{8}$ and
$q$ the number of those which are congruent to $9\pmod{16}$. Set $t=4q+2r$. We
have

  1. 1.

If there are two primes $p_{1}$ and $p_{2}$ dividing $d$ such that
$p_{1}\equiv-p_{2}\equiv 5\pmod{8}$, then $rank_{2}(Cl(L_{n,d}))=t-3.$

  2. 2.

If $d$ is divisible by a prime congruent to $3\pmod{8}$ and none of the primes
$p|d$ is congruent to $5\pmod{8}$, then $rank_{2}(Cl(L_{n,d}))=t-2$ or $t-3$.
More precisely, $rank_{2}(Cl(L_{n,d}))=t-3$ if and only if there is a prime
$p\equiv 1\pmod{8}$ dividing $d$ such that $\left(\frac{2}{p}\right)_{4}=-1$.

  3. 3.

If $d$ is divisible by a prime congruent to $5\pmod{8}$ and none of the primes
$p|d$ is congruent to $3\pmod{8}$, then $rank_{2}(Cl(L_{n,d}))=t-2$ or $t-3$.
More precisely, $rank_{2}(Cl(L_{n,d}))=t-3$ if and only if there is a prime
$p\equiv 1\pmod{8}$ dividing $d$ such that $\left(\frac{2}{p}\right)_{4}=1$.

  4. 4.

If all the primes $p|d$ are congruent to $9\pmod{16}$, then
$rank_{2}(Cl(L_{n,d}))=4q-2$ or $4q-3$. More precisely,
$rank_{2}(Cl(L_{n,d}))=4q-3$ if and only if there are two prime divisors
$p_{1}$ and $p_{2}$ of $d$ such that $\left(\frac{2}{p_{1}}\right)_{4}=1$ and
$\left(\frac{2}{p_{2}}\right)_{4}=-1$.

###### Proof.

We shall firstly prove the items of the above theorem assuming that
$n\in\\{3,4,5\\}$. We have $h(K_{n}^{+})=1$ (see [7]). So by [10, Theorem
8.2], the unit group of $K_{n}^{+}$ is generated by $-1$ and the cyclotomic
units $\xi_{k,n}$, for odd integers $k$ such that $1<k<2^{n-1}$. Thus, by
Lemma 2, we have

$E_{K_{n}}=\langle\zeta_{2^{n}},\xi_{k,n},\text{ with $k$ is an odd integer
such that }1<k<2^{n-1}\rangle.$

By Lemma 3, Proposition 1 and Lemma 4 we have
$rank_{2}(Cl(L_{n,d}))=t-1-e_{n,d}$, where $e_{n,d}$ is defined by
$(E_{K_{n}}:E_{K_{n}}\cap\mathcal{N}(L_{n,d}))=2^{e_{n,d}}$. We shall
determine the classes representing
$E_{K_{n}}/(E_{K_{n}}\cap\mathcal{N}(L_{n,d}))$. Let $\alpha\in E_{K_{n}}$.
$\overline{\alpha}$ denotes the class of $\alpha$ in
$E_{K_{n}}/(E_{K_{n}}\cap\mathcal{N}(L_{n,d}))$. Let $p$ be a prime dividing
$d$ and $\mathfrak{p}_{K_{n}}$ be a prime ideal of $K_{n}$ lying over $p$. We
have

$\displaystyle\left(\frac{\alpha,d}{\mathfrak{p}_{K_{n}}}\right)=\prod_{q|d}\left(\frac{\alpha,d}{\mathfrak{p}_{K_{n}}}\right)$
$\displaystyle=$
$\displaystyle\left(\frac{\alpha,p}{\mathfrak{p}_{K_{n}}}\right)\prod_{q|d\;\text{
and }\;q\not=p}\left(\frac{\alpha,q}{\mathfrak{p}_{K_{n}}}\right)$
$\displaystyle=$
$\displaystyle\left(\frac{\alpha,p}{\mathfrak{p}_{K_{n}}}\right)\prod_{q|d\;\text{
and
}\;q\not=p}\left(\frac{q}{\;\mathfrak{p}_{K_{n}}}\right)^{0}=\left(\frac{\alpha,p}{\mathfrak{p}_{K_{n}}}\right).$

Note that the units $\xi_{k,n}$ for $k\equiv\pm 1\pmod{8}$ are norms in
$L_{n,d}/K_{n}$ so we will disregard them. Let $k$ and $k^{\prime}$ denote any
two positive integers such that $k,k^{\prime}\equiv\pm 3\pmod{8}.$

  1. 1.

By Lemmas 5 and 6, we have $\zeta_{2^{n}}$ and $\xi_{k,n}$ are not norms in
$L_{n,d}/K_{n}$. Furthermore

$\left\\{\begin{array}[]{cl}\left(\frac{\zeta_{2^{n}}\xi_{k,n},p}{\mathfrak{p}_{K_{n}}}\right)=-1,&\text{
if }p\equiv 5\pmod{8},\vspace{0.3cm}\\\
\left(\frac{\xi_{k,n}\xi_{k^{\prime},n},p}{\mathfrak{p}_{K_{n}}}\right)=1,&\text{
for all prime }p\text{ dividing }d.\\\ \end{array}\right.$

Thus $\overline{\zeta_{2^{n}}}\not=\overline{\xi_{k,n}}$ and
$\overline{\xi_{k,n}}=\overline{\xi_{k^{\prime},n}}$. Hence
$E_{K_{n}}/(E_{K_{n}}\cap\mathcal{N}(L_{n,d}))=\\{\overline{1},\overline{\zeta_{2^{n}}},\overline{\xi_{k,n}},\overline{\zeta_{2^{n}}\xi_{k,n}}\\}$.
It follows that $rank_{2}(Cl(L_{n,d}))=t-3$.

  2. 2.

By Lemmas 5 and 6, we have $\zeta_{2^{n}}$ and $\xi_{k,n}$ are not norms in
$L_{n,d}/K_{n}$. Furthermore

$\left\\{\begin{array}[]{cl}\left(\frac{\zeta_{2^{n}}\xi_{k,n},p}{\mathfrak{p}_{K_{n}}}\right)=\left(\frac{2}{p}\right)_{4},&\text{
if }p\equiv 3\pmod{8},\vspace{0.3cm}\\\
\left(\frac{\xi_{k,n}\xi_{k^{\prime},n},p}{\mathfrak{p}_{K_{n}}}\right)=1,&\text{
for all prime }p\text{ dividing }d.\\\ \end{array}\right.$

Then $\overline{\xi_{k,n}}=\overline{\xi_{k^{\prime},n}}$ and
$\overline{\zeta_{2^{n}}}\not=\overline{\xi_{k,n}}$ if and only if
$\left(\frac{2}{p}\right)_{4}=-1$. Thus,
$E_{K_{n}}/(E_{K_{n}}\cap\mathcal{N}(L_{n,d}))=\\{\overline{1},\overline{\zeta_{2^{n}}}\\}$,
if $\left(\frac{2}{p}\right)_{4}=1$ and
$E_{K_{n}}/(E_{K_{n}}\cap\mathcal{N}(L_{n,d}))=\\{\overline{1},\overline{\zeta_{2^{n}}},\overline{\xi_{k,n}},\overline{\zeta_{2^{n}}\xi_{k,n}}\\}$
if not. So the second item.

  3. 3.

By Lemmas 5 and 6, we have $\zeta_{2^{n}}$ are not norm in $L_{n,d}/K_{n}$ and
$\xi_{k,n}$ is not norm in $L_{n,d}/K_{n}$ is and only if
$\left(\frac{2}{p}\right)_{4}=1$. So with similar discussion as above, one
shows that
$E_{K_{n}}/(E_{K_{n}}\cap\mathcal{N}(L_{n,d}))=\\{\overline{1},\overline{\zeta_{2^{n}}}\\}$
if $\left(\frac{2}{p}\right)_{4}=-1$ and
$E_{K_{n}}/(E_{K_{n}}\cap\mathcal{N}(L_{n,d}))=\\{\overline{1},\overline{\zeta_{2^{n}}},\overline{\xi_{k,n}},\overline{\zeta_{2^{n}}\xi_{k,n}}\\}$
if not. So the results.

  4. 4.

    1. $\bullet$

Assume that for all prime $p$ dividing $d$ we have
$\left(\frac{2}{p}\right)_{4}=1$, then
$\left(\frac{\zeta_{2^{n}}\xi_{k,n},p}{\mathfrak{p}_{K_{n}}}\right)=\left(\frac{\zeta_{2^{n}},p}{\mathfrak{p}_{K_{n}}}\right)\left(\frac{\xi_{k,n},p}{\mathfrak{p}_{K_{n}}}\right)=1$.
It follows that $\overline{\zeta_{2^{n}}}=\overline{\xi_{k,n}}$. So
$E_{K_{n}}/(E_{K_{n}}\cap\mathcal{N}(L_{n,d}))=\\{\overline{1},\overline{\zeta_{2^{n}}}\\}$
and $e_{n,d}=1$.

    2. $\bullet$

If for all the primes $p$ dividing $d$ we have
$\left(\frac{2}{p}\right)_{4}=-1$, then $\xi_{k,n}$ is norm in
$L_{n,d}/K_{n}$. So
$E_{K_{n}}/(E_{K_{n}}\cap\mathcal{N}(L_{n,d}))=\\{\overline{1},\overline{\zeta_{2^{n}}}\\}$
and $e_{n,d}=1$.

    3. $\bullet$

Suppose now there are two primes $p_{1}$ and $p_{2}$ dividing $d$ such that
$\left(\frac{2}{p_{1}}\right)_{4}=1$ and
$\left(\frac{2}{p_{2}}\right)_{4}=-1$. We have
$\left(\frac{\xi_{k,n},p_{1}}{\mathfrak{p}_{K_{n}}}\right)=\left(\frac{\zeta_{2^{n}}\xi_{k,n},p_{2}}{\mathfrak{p}_{K_{n}}}\right)=-1$.
Thus $\overline{\zeta_{2^{n}}}\not=\overline{\xi_{k,n}}$ and
$\overline{\xi_{k,n}}\not=\overline{1}$. We infer that
$E_{K_{n}}/(E_{K_{n}}\cap\mathcal{N}(L_{n,d}))=\\{\overline{1},\overline{\zeta_{2^{n}}},\overline{\xi_{k,n}},\overline{\zeta_{2^{n}}\xi_{k,n}}\\}$
and $e_{n,d}=2$. So the third item.

Thus we proved the theorem for $n\in\\{3,4,5\\}$.
Since $rank_{2}(Cl(L_{3,d}))=rank_{2}(Cl(L_{4,d}))$, then Lemma 1 achieves the
proof.

∎

We similarly get the following result:

######  Theorem 2.

Let $n\geq 3$ be a positive integer and let $p$ denote a prime such that
$p\equiv 9\pmod{16}$. Then

$rank_{2}(Cl(L_{n,p}))=2.$

##  4\. Appendix

In this appendix, we give the rank of the $2$-class group of $L_{n,d}$
according to that of $L_{4,d}$, when the prime divisors of $d$ are congruent
to $\pm 3\pmod{8}$ or $\pm 7\pmod{16}$.

######  Lemma 7.

Let $n\geq 4$ be an integer and $p$ a prime integer congruent to $7\pmod{16}$.
Then for all prime ideal $\mathfrak{p}_{K_{n}}$ of $K_{n}$ dividing $p$, we
have

$\left(\frac{\zeta_{2^{n}},p}{\mathfrak{p}_{K_{n}}}\right)=\left(\frac{\zeta_{16},p}{\mathfrak{p}_{K_{4}}}\right)\text{
and
}\left(\frac{\xi_{k,n},p}{\mathfrak{p}_{K_{n}}}\right)=\left\\{\begin{array}[]{cc}\left(\frac{\xi_{3,4},p}{\mathfrak{p}_{K_{4}}}\right),&\text{
if }k\equiv 3\pmod{16}\\\
\left(\frac{\xi_{5,4},p}{\mathfrak{p}_{K_{4}}}\right),&\text{ if }k\equiv
5\pmod{16}\\\ \left(\frac{\xi_{7,4},p}{\mathfrak{p}_{K_{4}}}\right),&\text{ if
}k\equiv 7\pmod{16}\\\ 1,&\text{ if }k\equiv 1\pmod{16}.\\\
\end{array}\right.$

And there are $\varepsilon_{1}$, $\varepsilon_{2}$,
$\varepsilon_{3}\in\\{-1,1\\}$ such that for all prime ideal
$\mathfrak{p}_{K_{n}}$ of $K_{n}$ dividing $p$ we have

$\left(\frac{\xi_{k,n},p}{\mathfrak{p}_{K_{n}}}\right)=\left\\{\begin{array}[]{cl}\varepsilon_{1}\left(\frac{\xi_{3,4},p}{\mathfrak{p}_{K_{4}}}\right),&\text{
if }k\equiv 11\pmod{16}\\\
\varepsilon_{2}\left(\frac{\xi_{5,4},p}{\mathfrak{p}_{K_{4}}}\right),&\text{
if }k\equiv 13\pmod{16}\\\
\varepsilon_{3}\left(\frac{\xi_{7,4},p}{\mathfrak{p}_{K_{4}}}\right),&\text{
if }k\equiv 15\pmod{16}\\\ 1,&\text{ if }k\equiv 9\pmod{16}.\\\
\end{array}\right.$

###### Proof.

By Remark 2 and Proposition 1, there are four prime ideals of $K_{4}$ lying
over $p$, and these primes are inert in $K_{n}$ for all $n\geq 5$. Since the
minimal polynomial of $\zeta_{2^{n}}$ over $K_{{n-1}}$ is $X^{2}-\zeta_{n-1}$,
then $N_{K_{n}/K_{n-1}}(\zeta_{2^{n}})=-\zeta_{2^{n-1}}$. We have

$\left(\frac{\zeta_{2^{n}},p}{\mathfrak{p}_{K_{n}}}\right)=\left(\frac{\zeta_{2^{n}},p}{\mathfrak{p}_{K_{n}}}\right)=\left(\frac{\zeta_{{2^{n-1}}},p}{\mathfrak{p}_{{K_{n-1}}}}\right)=...=\left(\frac{\zeta_{16},p}{\mathfrak{p}_{{K_{4}}}}\right),$

and

$\left(\frac{1-\zeta_{{2^{n}}}^{k},p}{\mathfrak{p}_{K_{n}}}\right)=\left(\frac{N_{K_{n}/K_{n-1}}(1-\zeta_{2^{n}}^{k}),p}{\mathfrak{p}_{{K_{n-1}}}}\right)=...=\left(\frac{1-\zeta_{16}^{k},p}{\mathfrak{p}_{{K_{4}}}}\right).$

$\displaystyle\left(\frac{\xi_{k,n},p}{\mathfrak{p}_{K_{n}}}\right)$
$\displaystyle=$
$\displaystyle\left(\frac{\zeta_{2^{n}}^{(1-k)/2},p}{\mathfrak{p}_{{K_{n}}}}\right)\left(\frac{\frac{1-\zeta_{2^{n}}^{k}}{1-\zeta_{2^{n}}},p}{\mathfrak{p}_{{K_{n}}}}\right)$
$\displaystyle=$
$\displaystyle\left(\frac{\zeta_{2^{n}},p}{\mathfrak{p}_{{K_{n}}}}\right)^{(1-k)/2}\left(\frac{(1-\zeta_{2^{n}}^{k})(1-\zeta_{2^{n}}),p}{\mathfrak{p}_{{K_{n}}}}\right)$
$\displaystyle=$
$\displaystyle\left(\frac{\zeta_{2^{n}},p}{\mathfrak{p}_{{K_{n}}}}\right)^{(1-k)/2}\left(\frac{1-\zeta_{2^{n}}^{k},p}{\mathfrak{p}_{{K_{n}}}}\right)\left(\frac{1-\zeta_{2^{n}},p}{\mathfrak{p}_{{K_{n}}}}\right)$
$\displaystyle=$
$\displaystyle\left(\frac{\zeta_{16},p}{\mathfrak{p}_{{K_{4}}}}\right)^{(1-k)/2}\left(\frac{1-\zeta_{16}^{k},p}{\mathfrak{p}_{{K_{4}}}}\right)\left(\frac{1-\zeta_{16},p}{\mathfrak{p}_{{K_{4}}}}\right).$

We consider the following cases

  1. $\bullet$

If $k\equiv 1,3,5\text{ or }7\pmod{16}$, then we have directly the result.

  2. $\bullet$

If $k\equiv 11\pmod{16}$, then
$\zeta_{16}^{k}=e^{\frac{11\pi}{8}i}=-e^{\frac{3\pi}{8}i}=-\zeta_{16}^{3}$.
Thus

$\displaystyle\left(\frac{\xi_{k,n},p}{\mathfrak{p}_{K_{n}}}\right)$
$\displaystyle=$
$\displaystyle\left(\frac{\zeta_{16},p}{\mathfrak{p}_{{K_{n}}}}\right)^{(1-k)/2}\left(\frac{1+\zeta_{16}^{3},p}{\mathfrak{p}_{{K_{4}}}}\right)\left(\frac{1-\zeta_{16},p}{\mathfrak{p}_{{K_{4}}}}\right)$
$\displaystyle=$
$\displaystyle\left(\frac{\zeta_{16},p}{\mathfrak{p}_{{K_{n}}}}\right)^{(1-k)/2}\left(\frac{1+\zeta_{16}^{3},p}{\mathfrak{p}_{{K_{4}}}}\right)\left(\frac{1-\zeta_{16},p}{\mathfrak{p}_{{K_{4}}}}\right)\left(\frac{1-\zeta_{16}^{3},p}{\mathfrak{p}_{{K_{4}}}}\right)\left(\frac{1-\zeta_{16}^{3},p}{\mathfrak{p}_{{K_{4}}}}\right)$
$\displaystyle=$
$\displaystyle\left(\frac{(1+\zeta_{16}^{3})(1-\zeta_{16}^{3}),p}{\mathfrak{p}_{{K_{4}}}}\right)\left(\frac{\xi_{3,4},p}{\mathfrak{p}_{K_{4}}}\right)=\left(\frac{1-\zeta_{8}^{3},p}{\mathfrak{p}_{{K_{4}}}}\right)\left(\frac{\xi_{3,4},p}{\mathfrak{p}_{K_{4}}}\right).$

Since $\mathfrak{p}_{K_{3}}$ is totally decomposed in $K_{4}$, then
$\left(\frac{\xi_{k,n},p}{\mathfrak{p}_{K_{n}}}\right)=\left(\frac{1-\zeta_{8}^{3},p}{\mathfrak{p}_{{K_{3}}}}\right)\left(\frac{\xi_{3,4},p}{\mathfrak{p}_{K_{4}}}\right)$.
We have $p$ decomposes into product of two prime ideals,
$\mathfrak{p}_{K_{3}}$ and $\mathfrak{p}_{K_{3}}^{\prime}$ of $K_{3}$, then by
the product formula, we have :

$\left(\frac{1-\zeta_{8}^{3},p}{\mathfrak{p}_{{K_{3}}}}\right)\left(\frac{1-\zeta_{8}^{3},p}{\mathfrak{p}_{{K_{3}}}^{\prime}}\right)\left(\frac{1-\zeta_{8}^{3},p}{\mathfrak{q}_{{K_{3}}}}\right)=1,$

where $\mathfrak{q}_{{K_{3}}}$ is the prime ideal of $K_{3}$ above $2$. By
Remark 1 and Lemma 4 we easily deduce that
$\left(\frac{1-\zeta_{8}^{3},p}{\mathfrak{q}_{{K_{3}}}}\right)=1$. Thus
$\left(\frac{1-\zeta_{8}^{3},p}{\mathfrak{p}_{{K_{3}}}}\right)=\left(\frac{1-\zeta_{8}^{3},p}{\mathfrak{p}_{{K_{3}}}^{\prime}}\right)=\varepsilon_{1}$
and
$\left(\frac{\xi_{k,n},p}{\mathfrak{p}_{K_{n}}}\right)=\varepsilon_{1}\left(\frac{\xi_{3,4},p}{\mathfrak{p}_{K_{4}}}\right).$

  3. $\bullet$

Similarly, we show the relationships between the other norm residue symbols.

∎

######  Theorem 3.

Let $d$ be a square-free integer such that the prime divisors of $d$ are
congruent to $\pm 3\pmod{8}$ or $\pm 7\pmod{16}$. Then, for all positive
integer $n\geq 4$ , we have

$rank_{2}(Cl(L_{n,d}))=rank_{2}(Cl(L_{4,d})).$

###### Proof.

Suppose that $n\in\\{4,5\\}$. By Lemmas 5, 6 and 7, we have $e_{n,d}=e_{4,d}$.
By Propositions 1 and 2 the number of prime divisors of $d$ in $K_{n}$ is the
same. Then $rank_{2}(Cl(L_{5,d}))=rank_{2}(Cl(L_{4,d}))$ (see Lemma 3). Hence,
Lemma 1 completes the proof. ∎

## Acknowledgment

I would like to express my gratitude to my professor Abdelkader Zekhnini for
his support and remarks during the preparation of this paper. My thanks are
also due to the referee for his/her careful reading and constructive comments.

## References

  * [1] A. Azizi, M.M. Chems-Eddin, A. Zekhnini, On the rank of the $2$-class group of some imaginary triquadratic number fields, arXiv:1905.01225v2.
  * [2] M.M. Chems-Eddin, A. Azizi, A. Zekhnini, On the $2$-class group of some number fields with large degree, arXiv:1911.11198.
  * [3] T. Fukuda, Remarks on $Z_{p}$-extensions of number fields. Proc. Japan Acad. Ser. A Math. Sci., 70 $(1994)$, $264$–$266$.
  * [4] G. Gras, Sur les l-classes d’idéaux dans les extensions cycliques relatives de degré premier l, Ann. Inst. Fourier (Grenoble), 23 (1973), 1–48.
  * [5] P. Kaplan, Sur le $2$-groupe des classes d’idéaux des corps quadratiques, J. Reine. Angew. Math., 283/284 (1976), 313-363.
  * [6] J. Li, Y. Ouyang, Y. Xu, S. Zhang, $2$-class groups in dyadic kummer towers, arXiv:1905.04966.
  * [7] J.M. Masley, H. L. Montgomery, Cyclotomic fields with unique factorization, J. Reine Angew. Math., 286/287 (1976), 248–256.
  * [8] T.M. McCall, C.J. Parry, R.R. Ranalli, Imaginary bicyclic biquadratic fields with cyclic $2$-class group, J. Number Theory, 53 (1995), 88–99.
  * [9] M.T. McCall, C.J. Parry, R.R. Ranalli, The 2-rank of the class group of imaginary bicyclic biquadratic fields. Canad. J. Math., 49 (1997), 283–300.
  * [10] L.C. Washington, Introduction to cyclotomic fields. Graduate Texts in Mathematics, 83. Springer-Verlag, New York, 1982.


We consider three different SIMO systems where $(n,p)=(20,4)$, $(n,p)=(25,5)$ and $(n,p)=(30,6)$. We observe from Figs. <ref>, <ref> and <ref> that when the measurement noise is zero, in all three cases Algorithm <ref>  reaches very close to the machine epsilon. For the above three dimensions, the necessary truncation length and the learning rate do not change significantly when the hidden state dimension and output size increase. The spectral radius of $\mathbf{A}$ in Figs. <ref>, <ref> and <ref> is $0.93$, $0.95$ and $0.96$, respectively. The convergence of Algorithm <ref> for the three considered systems is depicted in Figs. <ref>-<ref>, when the batch size is $10,000$. For both algorithms, we have $(T,\eta)=\{(800,10^{-5}),(800, 10^{-5}),(800, 10^{-5})\}$. For the above systems, we consider measurement noise with zero mean and standard deviation $0.1$. For both approaches, we set $(T,\eta)=\{(300,4\times 10^{-8}),(500,3\times 10^{-8}),(700,3\times 10^{-8})\}$. The batch size is $10^7$ for Algorithm <ref>. The convergence of Algorithm <ref> for noisy systems is depicted in Figs. <ref>-<ref>, and the convergence of Algorithm <ref>  is depicted in Figs. <ref>-<ref>. The input to the system is Gaussian noise with zero mean and standard deviation $1$, and system noise has the standard deviation $0.1$.

Results for SISO systems.  In (a)-(f), the systems are noise-free. In (g)-(l), the systems are noisy. In (a), (d), (g) and (j), $n=20$, $m=1$, $p=1$. In (b), (e), (h) and (k), $n=25$, $m=1$, $p=1$. In (c), (f), (i) and (l), $n=30$, $m=1$, $p=1$.

§.§ MISO

We consider three different MISO systems for which the hidden state dimensions are 20, 25 and 30. For these three systems, the input sizes are 4, 5 and 6, respectively.  As depicted in Figs. <ref>, <ref> and <ref>, Algorithm <ref> learns the unknown parameters at a linear convergence rate. In Fig. <ref>, we have $\rho(\mathbf{A})=0.75$ and for Figs. <ref> and <ref>, we have $\rho(\mathbf{A})=0.70$. In Figs. <ref>-<ref>, we have $(T,\eta)=\{(800,10^{-5}),(800, 10^{-5}),(800, 10^{-5})\}$. Identical truncation length and the learning rate are considered for Algorithm <ref> in <ref>-<ref>. The batch size for Algorithm <ref>  is $10,000$. For noisy systems, the measurement noise is white, and its mean is zero and its variance is $0.01$. The standard deviation of the input signal is $0.1$. When the measurement noise is considered for the above systems, the convergence of Algorithm <ref> is depicted in Figs. <ref>-<ref>. Moreover, Figs. <ref>-<ref> show the convergence of Algorithm <ref> for the three noisy systems. The truncation length and step-size for both algorithms are $(T,\eta)=\{(60,5\times 10^{-8}),(60,5\times 10^{-8}),(60,5\times 10^{-8})\}$ for different systems. The batch size for Algorithm <ref> is $10^7$.

Results for MISO systems.  In (a)-(f), the systems are noise-free. In (g)-(l), the systems are noisy. In (a), (d), (g) and (j), $n=5$, $m=4$, $p=1$. In (b), (e), (h) and (k), $n=5$, $m=5$, $p=1$. In (c), (f), (i) and (l), $n=5$, $m=6$, $p=1$.

§.§ MIMO

Three different MIMO systems with hidden state dimensions 20, 25 and 30 are considered. In particular, for the three systems, we have $(n,m,p)=(5,4,4)$, $(n,m,p)=(5,5,4)$ and $(n,m,p)=(5,6,4)$. As the hidden state dimension increases, the number of required samples to reach a certain distance from the ground truth unknowns increases. The spectral radius of $\mathbf{A}$ in Figs. <ref>, <ref> and <ref> is $0.55$, $0.75$ and $0.64$, respectively. The convergence of Algorithm <ref> for three  noise-free systems is depicted in Figs. <ref>-<ref>, where $(T,\eta)=\{(800,10^{-5}),(800, 10^{-5}),(800, 10^{-5})\}$. Figs. <ref>-<ref> depict the convergence of Algorithm <ref> with the same truncation lengths and learning rates. For Algorithm <ref>, the batch size is $10^4$. The convergence of Algorithm <ref> for the three systems after the addition of white noise with standard deviation $0.1$ is given Figs. <ref>-<ref>. Furthermore, the convergence of Algorithm <ref> for identical systems is depicted Figs. <ref>-<ref>. For both approaches, we have $(T,\eta)=\{(60,5\times 10^{-8}),(60,5\times 10^{-8}),(60,5\times 10^{-8})\}$. The batch size is $10^7$. The control signal is white noise with zero mean and standard deviation $1$.

Results for MIMO systems. In (a)-(f), the systems are noise-free. In (g)-(l), the systems are noisy. In (a), (d), (g) and (j), $n=5$, $m=4$, $p=4$. In (b), (e), (h) and (k), $n=5$, $m=5$, $p=4$. In (c), (f), (i) and (l), $n=5$, $m=6$, $p=4$.


#  Distilling Motion Planner Augmented Policies into Visual Control Policies
for Robot Manipulation

I-Chun Arthur Liu2 Shagun Uppal111footnotemark: 1 Gaurav S. Sukhatme2 Joseph
J. Lim1
Peter Englert2 Youngwoon Lee1
1Cognitive Learning for Vision and Robotics Lab 2Robotic Embedded Systems
Laboratory
University of Southern California Equal contributionGaurav Sukhatme holds
concurrent appointments as a Professor at USC and as an Amazon Scholar. This
paper describes work performed at USC and is not associated with Amazon.AI
Advisor at NAVER AI LabWork done during an internship at NAVER AI Lab

###### Abstract

Learning complex manipulation tasks in realistic, obstructed environments is a
challenging problem due to hard exploration in the presence of obstacles and
high-dimensional visual observations. Prior work tackles the exploration
problem by integrating motion planning and reinforcement learning. However,
the motion planner augmented policy requires access to state information,
which is often not available in the real-world settings. To this end, we
propose to distill a state-based motion planner augmented policy to a visual
control policy via (1) visual behavioral cloning to remove the motion planner
dependency along with its jittery motion, and (2) vision-based reinforcement
learning with the guidance of the smoothed trajectories from the behavioral
cloning agent. We evaluate our method on three manipulation tasks in
obstructed environments and compare it against various reinforcement learning
and imitation learning baselines. The results demonstrate that our framework
is highly sample-efficient and outperforms the state-of-the-art algorithms.
Moreover, coupled with domain randomization, our policy is capable of zero-
shot transfer to unseen environment settings with distractors. Code and videos
are available at https://clvrai.com/mopa-pd.

> Keywords: Visual Policy Distillation, Motion Planning, Reinforcement
> Learning

##  1 Introduction

Solving complex manipulation tasks in obstructed environments is a challenging
problem in deep reinforcement learning (RL) since it requires precise object
interactions as well as collision-free movement across obstacles. To tackle
this problem, prior works [1, 2, 3] have proposed to combine the strengths of
motion planning (MP) and RL – safe collision-free maneuvers of MP and
sophisticated contact-rich interactions of RL, demonstrating promising
results. However, MP requires access to the geometric state of an environment
for collision checking, which is often not available in the real world, and is
also computationally expensive for a real-time control. To deploy such agents
in realistic settings, we need to resolve the dependency on the state
information and costly computation of MP, such that the agent can perform a
task in the visual domain.

To this end, we propose a two-step distillation framework, motion planner
augmented policy distillation (MoPA-PD), that transfers the state-based motion
planner augmented RL policy (MoPA-RL [1]) into a visual control policy,
thereby removing the motion planner and the dependency on the state
information. Concretely, our framework consists of two stages: (1) visual
behavioral cloning (BC [4]) with trajectories collected using the MoPA-RL
policy and (2) vision-based RL training with the guidance of smoothed
trajectories from the BC policy. The first step, visual BC, removes the
dependency on the motion planner and the resulting visual BC policy generates
smoother behaviors compared to the motion planner’s jittery behaviors. Then,
in the second step, we further improve the visual policy using asymmetric
actor-critic RL [5] directly from the image observations to overcome sub-
optimality in the MoPA-RL policy through self-exploration.

For efficient vision-based RL training, our method leverages the experience of
the BC policy and the state-based critic from MoPA-RL by initializing the
critic network with the MoPA-RL critic and the actor network through BC pre-
training; BC-trajectory guided RL training using a separate expert replay
buffer; tuning the entropy coefficient to encourage exploitation (i.e.,
maximizing reward) over exploration (i.e., maximizing entropy).

In summary, our contributions are as follows:

  * •

We propose a novel framework to learn a visual control policy in cluttered
scenarios by distilling a state-space policy that uses a motion planner into
an image-space policy by removing the motion planner dependency.

  * •

Our asymmetric visual policy leaning ensures high sample-efficiency using
weight initialization followed by BC trajectory-guided RL with entropy
coefficient tuning.

  * •

The distilled visual policy is capable of zero-shot domain transfer to unseen
environments with visual domain randomization during training.

  * •

Our method outperforms the state-of-the-art methods in terms of success rate,
sample efficiency, and path length on three manipulation tasks in obstructed
environments.

Figure 1:  Our two-step framework distills the state-based MoPA-RL agent into
a visual control policy. In stage 1, we learn a visual BC policy using MoPA-RL
trajectories, thereby ensuring smooth trajectories for the expert replay
buffer $\mathcal{R}_{e}$, while removing the motion planner. In stage 2, we
learn a BC trajectory-guided vision-based RL agent with an asymmetric actor-
critic algorithm using both the expert replay buffer $\mathcal{R}_{e}$ and
agent replay buffer $\mathcal{R}_{\pi}$. To improve sample efficiency, we
initialize the actor and the critic of the vision-based agent with the weights
of the BC policy and the MoPA-RL critic, respectively.

##  2 Related Work

The objective of our work is to learn complex manipulation tasks in realistic
and obstructed environments. Motion planner augmented RL methods [1, 2, 3]
have shown promising results in solving complex tasks in obstructed
environments by combining MP and RL. However, these methods cannot be deployed
in real-world settings due to their dependency on state information. To remove
this dependency, state-based policies must be transferred to the visual
domain.

A typical approach to distill state-based policies into vision-based policies
is behavioral cloning (BC [4]). However, BC often suffers when encountering
states unseen during training [6]. Therefore, prior works have proposed
various methods to improve BC agents using offline RL [7] and online RL [8].
However, these works do not involve the distillation of motion planners.

Several recent efforts have been made towards distilling MP algorithms into
neural motion planners using learning-based methods [9, 10, 11, 12, 13]. Most
of these efforts can be broadly categorized into imitation learning (IL) and
RL paradigms. For IL, supervised learning on MP trajectories has been used to
learn neural network-based planners [10, 11, 12, 13, 14]. However, the
performance of such supervised learning approaches are limited by the
collected dataset and demonstrator’s performance.

To discover better policies with additional interactions, various off-policy
RL methods have been studied for superseding motion planners with neural
network policies [13, 15, 16]. These approaches utilize expert trajectories
stored in the replay buffer for guided exploration, which leads to better
policies than experts [17, 18, 19, 20]. However, these works learn either in
unobstructed environments [17, 18, 19, 20] or on simpler tasks not involving
object manipulation [13]. They also assume fully observable environments and
learn policies in state space. In this work, we focus on complex manipulation
tasks in obstructed environments using visual observations.

Moreover, most offline RL approaches [21, 22] in sparse reward scenarios are
coupled with hindsight experience replay (HER [23]). They empirically show
that not using HER significantly hurts the agent’s performance [13, 5, 16,
17]. However, HER is designed to efficiently train goal-conditioned policies
for multi-goal RL, which makes it unsuitable for tasks with multiple
sequential steps without explicitly conditioning on goals. In contrast, our
method successfully learns composite manipulation tasks in cluttered
environments without using HER for tackling sparse rewards.

##  3 Method

Our goal is to solve complex manipulation tasks in obstructed environments
with visual inputs. While MP-based methods can efficiently solve such tasks,
they are restricted to learning a state-based policy which is difficult to
transfer to real environments. Moreover, sampling-based motion planning incurs
significant computational costs, making it hard to integrate them into real-
time controllers. Thus, we propose a method that distills an MP-augmented
state-based agent into a visual policy, removing the MP and state dependency.
We formally define our problem and introduce the MP-augmented RL in Section
3.1. Then, we describe our two-step visual distillation approach in Section
3.2.

###  3.1 Preliminaries

##### Problem formulation

We formulate our problem as a Markov Decision Process (MDP) $M$ defined by a
tuple $(\mathcal{S},\mathcal{O},\mathcal{A},P,R,\rho_{0},\gamma)$ that
consists of a state space $\mathcal{S}$, partial observation space
$\mathcal{O}$ (visual inputs corresponding to states), action space
$\mathcal{A}$, transition function
$P:\mathcal{S}\times\mathcal{A}\times\mathcal{S}\rightarrow\mathbb{R}$, reward
function $R$, initial state distribution $\rho_{0}$, and discount factor
$\gamma$. The agent is represented as a policy $\pi(a_{t}|o_{t})$, which takes
an action $a_{t}$ under a visual observation $o_{t}$ and receives a reward
$r_{t}=R(s_{t},a_{t})$, with the state transitioning to $s_{t+1}$. The goal of
the agent is to maximize the expected discounted sum of rewards
$\mathbb{E}_{(s,a)\sim\pi}\sum_{t=0}^{T-1}\gamma^{t}r_{t}$, where $T$ is the
episode horizon.

##### Motion Planner-Augmented RL (MoPA-RL)

To tackle manipulation tasks in cluttered environments, Yamada et al. [1]
proposed to incorporate MP and RL via an MP-augmented MDP
$\tilde{M}=(\mathcal{S},\tilde{\mathcal{A}},\tilde{P},\tilde{R},\rho_{0},\gamma)$
that consists of the state space $\mathcal{S}$, enlarged action space
$\tilde{\mathcal{A}}$ augmenting the direct action space $\mathcal{A}$ with
the MP action space $\hat{\mathcal{A}}$,111 MoPA-RL [1] defines the direct
action space as $\mathcal{A}=[-\Delta q_{\text{step}},\Delta
q_{\text{step}}]^{d}$, where $\Delta q_{\text{step}}$ is the maximum joint
displacement, and the enlarged MP-augmented action space as
$\tilde{\mathcal{A}}=[-\Delta q_{\text{MP}},\Delta q_{\text{MP}}]^{d}$, where
$\Delta q_{\text{MP}}$ is the motion planner action limit with $\Delta
q_{\text{MP}}>\Delta q_{\text{step}}$. In MoPA-RL, actions in the direct
action space $a\in\mathcal{A}$ are directly applied as a joint torque while
other large actions
$\hat{a}\in\hat{\mathcal{A}}=\tilde{\mathcal{A}}\setminus\mathcal{A}$ invoke
motion planning. We refer the readers to Yamada et al. [1] for more details.
augmented transition function
$\tilde{P}:\mathcal{S}\times\tilde{\mathcal{A}}\times\mathcal{S}\rightarrow\mathbb{R}$,
augmented reward function $\tilde{R}(s,\tilde{a})$, initial state distribution
$\rho_{0}$, and discount factor $\gamma$.

MoPA-RL [1] learns a policy $\pi_{\phi}(\tilde{a}_{t}|s_{t})$ on the augmented
MDP $\tilde{M}$ with an off-policy RL algorithm, SAC [24]. Precisely, given a
state $s_{t}$, the policy predicts an action $\tilde{a}_{t}$, which is defined
as a robot joint displacement $\Delta q_{t}$. If the action lies within the
direct action space $\mathcal{A}$, it is directly executed by the controller
as the agent performs sophisticated and contact-rich manipulations. However,
when the actions are larger in magnitude (i.e.
$\tilde{a}\in\hat{\mathcal{A}}$), the probability of collisions in the
presence of obstacles increases. Therefore, a sampling-based motion planner,
RRT-Connect [25], is called to realize such actions with large joint
displacements by computing collision-free paths. This allows the agent to
efficiently explore obstructed environments while avoiding collision [1].

###  3.2 Motion Planner Augmented Policy Distillation

Despite the advantages of MoPA-RL, its applicability to the real world is
limited due to the high computational cost of MP and dependency on fully
observable environment states. To this end, we propose a visual distillation
method, motion planner augmented policy distillation (MoPA-PD), that learns an
image-based control policy (without MP) from a state-based MP-augmented policy
by leveraging its rollouts and the learned state-based critic as a guidance.
Concretely, our proposed method consists of two stages, as illustrated in
Figure 1. Given a state-based MoPA-RL agent, we first use BC to train a
vision-based actor with trajectories collected from the MoPA-RL policy
(Section 3.2.1), and then we further improve the vision-based agent via BC
trajectory-guided asymmetric actor-critic RL [5] with the visual BC actor and
MoPA-RL critic (Section 3.2.2).

####  3.2.1 Stage 1: Visual Behavioral Cloning and Trajectory Smoothing

The MoPA-RL policy is learned with access to complete information about the
environment states and also utilizes MP for executing large action steps
without collisions. In this paper, we aim to utilize demonstrations from this
planner-based policy and distill it into a visual control policy, thereby
deducting expensive MP computations and training the actor in visual domain.
With the learned MoPA-RL policy, we first collect multiple transitions $d_{i}$
in low-level (direct) action space and store them into the MoPA-RL
demonstration dataset $\mathcal{D}_{\text{mp}}=\\{d_{1},d_{2},d_{3},...\\}$,
where $d_{i}=(s_{i},o_{i},a_{i},r_{i},s_{i+1})$. Then, we train a visual BC
actor $\pi_{\theta}(a_{t}|o_{t})$ using observation-action pairs
$(o_{i},a_{i})$ from the dataset $\mathcal{D}_{\text{mp}}$ by minimizing the
mean squared error.

Distilling the MoPA-RL trajectories using BC not only enables the BC actor to
work directly on visual inputs but also reduces jerky motion planning
behaviors of MoPA-RL, which occur due to motion planner’s priority on obstacle
avoidance over trajectory smoothness [26]. By removing unnecessary, jerky
movements through visual BC, the resulting trajectories become smoother and
even shorter, and thus help learning consistent and smooth motions.

However, the BC actor often fails when it encounters states not seen during
training due to covariate shift [6]. To improve the robustness of the policy,
we further train the visual BC actor using RL with additional environment
interactions. In other words, the BC actor can be a good starting policy for
visual RL training and the trajectories collected from the BC actor can
effectively guide exploration.

####  3.2.2 Stage 2: Vision-based RL with Asymmetric Actor-Critic

In the second stage, we further train the visual policy using RL directly on
image observations to enhance the robustness of the policy and overcome sub-
optimality in the planner-based policy. For efficient RL training, we adopt an
asymmetric actor-critic architecture [5], where the actor acts based on
environment images and robot joint states while the critic learns from
environment states. This architecture is motivated by learning transferable
visual policies, which do not require state information during inference but
benefit from them while learning in simulation. Our training procedure
comprises of the following components:

##### Weight initialization of actor and critic networks

Initializing actor and critic networks with suitable weights can accelerate RL
training by providing more optimal rollouts from the actor and informative
learning signals from the critic compared to the randomly initialized actor
and critic [27, 28]. Especially when learning from pixels, suitable
initialization can significantly improve sample efficiency [20, 29]. To this
end, we propose to leverage the state-based agent’s experience by utilizing
its weights for initializing our asymmetric visual agent’s networks, thereby
bringing their initial distributions closer. Thus, the visual actor network
$\pi_{\theta}(a_{t}|o_{t})$ is initialized with the weights of the visual BC
actor learned in Section 3.2.1. We also initialize the critic
$Q_{\psi}(s_{t},a_{t})$ with the critic of MoPA-RL. Note that the action
spaces of two critic networks are different, but initialization is still
feasible since the action space of MoPA-RL is a superset of the direct action
space.

##### BC trajectory-guided asymmetric RL

After initializing the actor and critic networks, we collect agent rollouts
into the agent replay buffer $\mathcal{R}_{\pi}$. As an RL algorithm, we adopt
Soft Actor-Critic (SAC [24]), an off-policy model-free RL algorithm for
continuous control. We optimize our asymmetric visual policy $\pi_{\theta}$ by
maximizing the following objective:

$J(\pi_{\theta})=\sum_{t=0}^{T}\mathbb{E}_{(\mathbf{s_{t}},\mathbf{a_{t}})\sim\rho_{\pi_{\theta}}}\left[Q_{\psi}\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)+\alpha\mathcal{H}\left(\pi_{\theta}\left(\cdot\mid\mathbf{o}_{t}\right)\right)\right],$
(1)

where the temperature parameter $\alpha$ balances between exploration and
exploitation using entropy $\mathcal{H}$.

RL training with a BC-initialized policy often quickly deviates from the
original policy. To prevent this problem and guide exploration during RL, we
update our RL agent not only with agent trajectories but also with BC policy
(i.e. expert) trajectories [18]. Thus, we collect smoothed trajectories from
our visual BC policy, $(s_{i},o_{i},a_{i},r_{i},s_{i+1})$, in an expert replay
buffer $\mathcal{R}_{e}$. Then, for each RL training iteration, we sample 1:3
transitions from $\mathcal{R}_{e}$ and $\mathcal{R}_{\pi}$, respectively. A
separate expert replay buffer ensures guided exploration from the expert and
also circumvents catastrophic forgetting after weight initialization. Note
that we use the BC trajectories $\mathcal{R}_{e}$ instead of the MoPA-RL data
$\mathcal{D}_{\text{mp}}$ because jittery motion planning paths in the MoPA-RL
data make RL training difficult and sub-optimal.

##### Entropy coefficient tuning

The performance of SAC is known to be sensitive to $\alpha$ [30]. Since our
asymmetric agent already starts with the well-trained BC actor and MoPA-RL
critic, we initialize $\alpha$ to values lower than the final $\alpha$
obtained in MoPA-RL. This is to ensure that with prior knowledge in the state-
based agent, the visual agent focuses more on maximizing rewards over
exploration. We describe the hyperparameter choice and implementation details
in Section A.1 and Section B.

In summary, we propose a two-step visual distillation method for a state-based
MoPA-RL agent using visual BC followed by BC trajectory-guided vision-based RL
to remove the dependency on MP and environment states. For efficient RL
training, our method leverages weight initialization of the actor and critic,
the expert replay buffer of BC smoothed trajectories, and entropy coefficient
tuning.

##  4 Experiments

In this paper, we propose to distill a motion planner augmented policy into a
visual control policy for complex manipulation tasks in obstructed
environments. In our experiments, we aim to answer the following questions:
(1) Does IL efficiently learn policies for obstructed environments in image
space? Moreover, is naively combining IL with RL sufficient for solving
complex tasks? (2) Is distillation better than directly learning a visual
policy using MP? (3) Does our approach, MoPA-PD, efficiently learn a visual
policy using prior state-based exploration knowledge? (4) Is our visual policy
capable of domain transfer and robust to unseen distractors?

###  4.1 Environments

(a) Sawyer Push

(b) Sawyer Lift

(c) Sawyer Assembly

Figure 2:  Manipulation tasks in obstructed environments from Yamada et al.
[1]. (a) Sawyer Push: Push the red cube in the box to the green goal. (b)
Sawyer Lift: Lift out the can inside the box. (c) Sawyer Assembly: Insert the
table leg in the hole in the table top.

We evaluate our approach on three obstructed environments (see Figure 2) from
Yamada et al. [1], simulated using the MuJoCo physics engine [31]. We use a
32x32 image as a visual observation.

  * •

Sawyer Push: A Rethink Sawyer robot arm with 7 DoF, initially located outside
the box, needs to reach an object placed inside the box and push it to the
goal location within the box.

  * •

Sawyer Lift: The Sawyer arm must reach an object placed inside a box, grasp,
and lift it outside the box, avoiding collisions. The Sawyer arm is always
initialized outside the box.

  * •

Sawyer Assembly: The Sawyer arm needs to assemble the fourth leg (already
attached to the gripper) of a table into the vacant hole while avoiding
collisions to other three legs. This environment is built upon the IKEA
furniture assembly environment [32].

For Sawyer Push and Sawyer Assembly, the agent receives a reward proportional
to the distance between the end-effector and object, or the object and goal
position when the distance is less than $\epsilon$. We use $\epsilon=0.1$ for
Sawyer Push and $\epsilon=0.3$ for Sawyer Assembly while the distance between
initial and goal state is around $1.2$. An episode is considered successful
when the distance between the cube and goal (Sawyer Push) or the peg-head and
goal (Sawyer Assembly) is less than 0.05. For Sawyer Lift, we use the reward
function similar to Fan et al. [33], where a reward is defined for all three
intermediate stages of the task: _reach_ , _grasp_ and _lift_. A bonus reward
signal of $150$ is received upon successful task completion in all the
environments. We refer readers to Section D for more details.

| Sawyer Push | Sawyer Lift | Sawyer Assembly
---|---|---|---
| ASR $\uparrow$ | AEL $\downarrow$ | ADR $\uparrow$ | ASR $\uparrow$ | AEL $\downarrow$ | ADR $\uparrow$ | ASR $\uparrow$ | AEL $\downarrow$ | ADR $\uparrow$
MoPA-RL | 98.2 | 111.0 | 52.7 | 95.0 | 109.2 | 52.7 | 99.8 | 63.3 | 82.4
BC-Visual | 99.4 | 118.0 | 46.9 | 62.0 | 108.8 | 34.6 | 97.0 | 115.1 | 50.4
Asym. SAC | 0.0 | 250.0 | 0.0 | 0.0 | 250.0 | 0.2 | 0.0 | 250.0 | 3.4
MoPA-Asym. SAC | 0.0 | 250.0 | 0.0 | 0.0 | 250.0 | 0.0 | 0.0 | 250.0 | 0.0
CoL | 0.0 | 250.0 | 0.0 | 29.8 | 173.3 | 9.7 | 0.0 | 250.0 | 0.0
CoL w/ BC smoothing | 0.0 | 250.0 | 0.0 | 0.0 | 250.0 | 5.1 | 0.0 | 250.0 | 0.0
Ours w/o BC smoothing | 100.0 | 34.0 | 108.7 | 99.4 | 43.6 | 100.7 | 0.0 | 250.0 | 0.0
Ours | 100.0 | 32.0 | 110.8 | 99.0 | 42.0 | 101.7 | 100.0 | 61.7 | 84.5
Table 1:  Average success rate (ASR), episode length (AEL), and discounted
return (ADR) of our method and baselines averaged over five seeds. Each method
is evaluated after 3M environment steps. MoPA-RL [1] is the expert agent
trained with state-based policy. All baselines below horizontal line are
trained with asymmetric actor-critic for fair comparison. Maximum horizon is
250 for all tasks.

###  4.2 Baselines

We compare our method with the following baselines to evaluate the merits of
our approach:

  * •

MoPA-RL [1]: A MP-augmented policy, for learning large displacement actions
with MP and smaller actions with RL. This policy also serves as our state-
based expert agent.

  * •

Visual Behavioral Cloning (BC-Visual): A behavioral cloning policy trained on
the image-action pairs collected from MoPA-RL trajectories.

  * •

Asym. SAC [5]: An asymmetric actor-critic method, trained using SAC where the
critic is learned in the state space and the actor is learned in the image
space.

  * •

MoPA Asym. SAC: MoPA-RL policy [1] learned using the asymmetric framework.
Note that this method still uses a motion planner with an augmented action
space. This method is a direct attempt at learning a visual policy using a MP-
augmented framework [1].

  * •

CoL [29]: A policy learned using the Cycle-of-Learning framework, which is the
state-of-the-art algorithm for learning from demonstrations (LfD).

  * •

CoL (w BC Smoothing) [29]: A policy similar to CoL [29], with BC for
trajectory smoothing.

  * •

Ours (w/o BC Smoothing): A policy learned using our approach described in
Section 4 without BC trajectory smoothing, i.e., we directly use MoPA-RL
trajectories in $\mathcal{R}_{e}$.

(a) Sawyer Push

(b) Sawyer Lift

(c) Sawyer Assembly

Figure 3: Learning curves of our method compared to baselines. All methods are
trained for 3M environment steps. For the methods that require MoPA-RL, we
train MoPA-RL for 1M steps and then train the methods for 2M steps (total 3M)
for a fair comparison. Our method solves all the tasks with the highest
average discounted return. Our method w/o BC smoothing can solve two tasks
with slower convergence compared to Ours, but fails on Sawyer Assembly.

###  4.3 Evaluation

We compare our approach against baselines on the following evaluation metrics
averaged over five random seeds and $100$ unseen episodes per seed:

  * •

Average Success Rate (ASR) is the average number of successful episodes.

  * •

Average Episode Length (AEL) is the average length of successful episodes.

  * •

Average Discounted Return (ADR) is the average discounted sum of rewards
$\sum_{t=0}^{T-1}\gamma^{t}R(s_{t},a_{t})$, with $T$ being the episode
horizon. An episode completed in more time steps has a lower discounted
reward, due to exponentially discounted reward with $\gamma=0.99$.

###  4.4 Results

##### Comparisons with baselines

Figure 3 illustrates the learning curves of our method and all other baselines
using discounted rewards against the number of training steps. As per the
trend, our method is far more sample-efficient and outperforms all the
baselines. Asym. SAC [5] fails to directly learn a visual policy for Sawyer
Lift and Sawyer Assembly in obstructed environments, where exploration is
hard. Moreover, MoPA Asym. SAC, which is a direct attempt at learning a visual
policy while using MP, does not successfully learn to solve the tasks. CoL
[29] which has a straightforward combination of RL and BC objective for actor
optimization receives partial rewards for some tasks at around 1.5-2M steps,
much slower than our method’s convergence.

As reported in Table 1, BC achieves decent performance for Sawyer Push and
Assembly, but not for the Sawyer Lift task. However, it achieves high AEL and
low ADR for all tasks, showing its inefficiency to solve the task fast. This
is because BC is bound to perform as good as the expert demonstrations (here
from MoPA-RL) at its best. In contrast, our method optimizes trajectories
beyond expert signals and achieves significantly lower AEL and higher ADR
compared to baselines.

##### Ablation on BC trajectory smoothing

Our method without BC smoothing has higher variance across seeds and converges
slower than our method for two tasks (see Figure 3). For Sawyer Assembly, it
does not learn to solve the task at all. In short, BC smoothing of the motion
planner trajectories is important for our method to work across all
environments. This is because the motion planner based trajectories are
usually jittery and non-smooth. Using behavioral cloning refines each
transition, thereby making the entire trajectories much smoother and help in
efficiently training the RL agent.

##### Ablation on weight initialization

We compare our method with and without actor-critic initialization in
appendix, Figure 7 and do not observe any episode success until 1.2M
environment steps for the latter. This shows the importance of our proposed
weight initialization for learning in the visual domain. We further elaborate
the ablation results in Section A.2.

##### Ablation on entropy coefficient tuning

We examine the effect of different values of $\alpha$ on the trade-off between
entropy maximization and reward maximization for the SAC objective as shown in
Section A.1. Compared to higher $\alpha$ values, smaller values of $\alpha$
improve the sample efficiency during visual policy learning. Since we utilize
prior knowledge from the state-based agent, we use a smaller alpha to exploit
the previously acquired knowledge instead of exploring the entire state space
again.

###  4.5 Policy Transfer to Different Domains

(a) Domain randomization

(b) Original environment

(c) Scenario 1

(d) Scenario 2

Figure 4:  Illustration of environments for domain transfer. (a) Domain randomized environments are used for training; For testing, we use (b) original environment without randomization, (c) Scenario 1 with distractors such as unseen cubes (blue and magenta), walls, floor, and furniture, and (d) Scenario 2 similar to Scenario 1 along with variation in size of the table and texture of the box and table.  | Sawyer Push | Sawyer Lift | Sawyer Assembly
---|---|---|---
| ASR $\uparrow$ | AEL $\downarrow$ | ADR $\uparrow$ | ASR $\uparrow$ | AEL $\downarrow$ | ADR $\uparrow$ | ASR $\uparrow$ | AEL $\downarrow$ | ADR $\uparrow$
Original env | 99.7 | 39.1 | 103.1 | 99.3 | 37.3 | 106.5 | 100.0 | 49.7 | 93.4
Scenario 1 | 100.0 | 40.4 | 102.2 | 96.7 | 38.3 | 103 | 100.0 | 47.9 | 94.8
Scenario 2 | 99.3 | 43.4 | 99.04 | 97.3 | 37.0 | 104.7 | 100.0 | 47.7 | 95.0
Table 2: Evaluation metrics for domain transfer to unseen environments,
illustrated in Figure 4.

In this experiment, we learn our policy with domain randomization (DR) to
verify its domain transfer capabilities. DR is a promising approach for
modelling transferable policies using a multitude of variations in simulation
during training [34, 35], which makes the policy invariant to changes that are
insignificant for the task. For this work, we randomize the simulation
environment using different textures, colors, and lighting conditions for
training as shown in Figure 4 and Figure 10. We then test our DR policy on
three unseen scenarios, as illustrated in Figure 4 comprising of (1) the
original environment without any randomization; (2) Scenario 1 with realistic
background distractions (e.g., furniture, walls, and floor) and unseen
distractor cubes (blue and magenta) near the target cube (red); and (3)
Scenario 2 with additional changes in the texture of the table and box, and
different size of the table. Our policy attains more than 96% ASR in all three
unseen scenarios for all the tasks (see Table 2). This illustrates the
robustness of our proposed framework in learning transferable visual policies,
robust to unseen distractors in a sample-efficient manner.

##  5 Conclusion

In this paper, we introduce a two-step distillation method for learning
manipulation tasks in obstructed environments in the visual domain. In step
one, we use a MP-augmented RL policy as the state-based expert and
subsequently learn a visual BC agent from it, removing the motion planner
dependency. In step two, we learn a vision-based agent using an asymmetric
actor-critic framework. This step is further expedited via proper weight
initialization, BC trajectory-guided RL training, and entropy coefficient
tuning, making our method highly sample-efficient. Our visual policy combined
with domain randomization demonstrates successful zero-shot transfer to unseen
environments with new visual domains and distractors. Beyond zero-shot
transfer, fine-tuning the policy in the real world by learning a vision-based
critic or applying simulation-to-real techniques [36] is our definite future
work to realize simulation-to-real transfer.

#### Acknowledgments

This research is supported by the Annenberg Fellowship from USC, NAVER AI Lab,
and NSF NRI-2024768. We thank our colleagues from the CLVR lab and RESL for
the valuable discussions that considerably assisted the research.

## References

  * Yamada et al. [2020] J. Yamada, Y. Lee, G. Salhotra, K. Pertsch, M. Pflueger, G. S. Sukhatme, J. J. Lim, and P. Englert.  Motion planner augmented reinforcement learning for robot manipulation in obstructed environments.  In _Conference on Robot Learning_ , 2020.
  * Xia et al. [2021] F. Xia, C. Li, R. Martín-Martín, O. Litany, A. Toshev, and S. Savarese.  Relmogen: Leveraging motion generation in reinforcement learning for mobile manipulation.  In _IEEE International Conference on Robotics and Automation_ , 2021\.
  * Matheron et al. [2020] G. Matheron, N. Perrin, and O. Sigaud.  Pbcs: Efficient exploration and exploitation using a synergy between reinforcement learning and motion planning.  In _International Conference on Artificial Neural Networks_ , pages 295–307. Springer, 2020.
  * Pomerleau [1989] D. A. Pomerleau.  Alvinn: An autonomous land vehicle in a neural network.  In _Advances in Neural Information Processing Systems_ , pages 305–313, 1989.
  * Pinto et al. [2018] L. Pinto, M. Andrychowicz, P. Welinder, W. Zaremba, and P. Abbeel.  Asymmetric actor critic for image-based robot learning.  In _Robotics: Science and Systems_ , 2018.
  * Ross et al. [2011] S. Ross, G. J. Gordon, and J. Bagnell.  A reduction of imitation learning and structured prediction to no-regret online learning.  In _International Conference on Artificial Intelligence and Statistics_ , 2011.
  * Lee et al. [2021] A. X. Lee, C. Devin, Y. Zhou, T. Lampe, K. Bousmalis, J. T. Springenberg, A. Byravan, A. Abdolmaleki, N. Gileadi, D. Khosid, C. Fantacci, J. E. Chen, A. Raju, R. Jeong, M. Neunert, A. Laurens, S. Saliceti, F. Casarini, M. Riedmiller, R. Hadsell, and F. Nori.  Beyond pick-and-place: Tackling robotic stacking of diverse shapes.  In _Conference on Robot Learning_ , 2021.
  * Jeong et al. [2020] R. Jeong, Y. Aytar, D. Khosid, Y. Zhou, J. Kay, T. Lampe, K. Bousmalis, and F. Nori.  Self-supervised sim-to-real adaptation for visual robotic manipulation.  In _IEEE International Conference on Robotics and Automation_ , 2020\.
  * Qureshi and Yip [2018] A. H. Qureshi and M. C. Yip.  Deeply informed neural sampling for robot motion planning.  _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pages 6582–6588, 2018.
  * Qureshi et al. [2019] A. H. Qureshi, M. J. Bency, and M. C. Yip.  Motion planning networks.  In _IEEE International Conference on Robotics and Automation_ , pages 2118–2124, 2019.
  * Bency [2018] M. J. Bency.  _Towards Neural Network Embeddings of Optimal Motion Planners_.  PhD thesis, UC San Diego, 2018.
  * Pfeiffer et al. [2017] M. Pfeiffer, M. Schaeuble, J. Nieto, R. Siegwart, and C. Cadena.  From perception to decision: A data-driven approach to end-to-end motion planning for autonomous ground robots.  In _IEEE International Conference on Robotics and Automation_ , pages 1527–1533, 2017.
  * Jurgenson and Tamar [2019] T. Jurgenson and A. Tamar.  Harnessing reinforcement learning for neural motion planning.  In _Robotics: Science and Systems_ , 2019.
  * Strudel et al. [2020] R. Strudel, R. Garcia, J. Carpentier, J. Laumond, I. Laptev, and C. Schmid.  Learning obstacle representations for neural motion planning.  In _Conference on Robot Learning_ , 2020.
  * Ha et al. [2020] H. Ha, J. Xu, and S. Song.  Learning a decentralized multi-arm motion planner.  In _Conference on Robot Learning_ , 2020.
  * Luo et al. [2021] S. Luo, H. Kasaei, and L. Schomaker.  Self-imitation learning by planning.  In _IEEE International Conference on Robotics and Automation_ , 2021\.
  * Nair et al. [2018] A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel.  Overcoming exploration in reinforcement learning with demonstrations.  _IEEE International Conference on Robotics and Automation_ , pages 6292–6299, 2018.
  * Hester et al. [2018] T. Hester, M. Vecerík, O. Pietquin, M. Lanctot, T. Schaul, B. Piot, D. Horgan, J. Quan, A. Sendonaris, I. Osband, G. Dulac-Arnold, J. Agapiou, J. Z. Leibo, and A. Gruslys.  Deep q-learning from demonstrations.  In _Association for the Advancement of Artificial Intelligence_ , 2018\.
  * Vecerík et al. [2017] M. Vecerík, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot, N. Heess, T. Rothörl, T. Lampe, and M. A. Riedmiller.  Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards.  _ArXiv_ , abs/1707.08817, 2017.
  * Rajeswaran et al. [2018] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine.  Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations.  In _Robotics: Science and Systems_ , 2018.
  * Nair et al. [2020] A. Nair, A. Gupta, M. Dalal, and S. Levine.  Awac: Accelerating online reinforcement learning with offline datasets.  _ArXiv_ , abs/2006.09359, 2020.
  * Endrawis et al. [2021] S. Endrawis, G. Leibovich, G. Jacob, G. Novik, and A. Tamar.  Efficient self-supervised data collection for offline robot learning.  In _IEEE International Conference on Robotics and Automation_ , 2021\.
  * Andrychowicz et al. [2017] M. Andrychowicz, D. Crow, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba.  Hindsight experience replay.  In _Advances in Neural Information Processing Systems_ , 2017.
  * Haarnoja et al. [2018] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  In _International Conference on Machine Learning_ , 2018.
  * Kuffner and LaValle [2000] J. J. Kuffner and S. M. LaValle.  Rrt-connect: An efficient approach to single-query path planning.  In _IEEE International Conference on Robotics and Automation_ , 2000\.
  * Zhu et al. [2015] Z. Zhu, E. Schmerling, and M. Pavone.  A convex optimization approach to smooth trajectories for motion planning with car-like robots.  _IEEE Conference on Decision and Control_ , 2015.
  * Cruz et al. [2017] G. Cruz, Y. Du, and M. E. Taylor.  Pre-training neural networks with human demonstrations for deep reinforcement learning.  _ArXiv_ , abs/1709.04083, 2017.
  * Anderson et al. [2015] C. Anderson, M. Lee, and D. L. Elliott.  Faster reinforcement learning after pretraining deep networks to predict state dynamics.  _International Joint Conference on Neural Networks_ , 2015.
  * Goecks et al. [2020] V. G. Goecks, G. Gremillion, V. Lawhern, J. Valasek, and N. R. Waytowich.  Integrating behavior cloning and reinforcement learning for improved performance in sparse reward environments.  _International Conference on Autonomous Agents and Multi-Agent Systems_ , page 465–473, 2020.
  * Wang and Ni [2020] Y. Wang and T. Ni.  Meta-sac: Auto-tune the entropy temperature of soft actor-critic via metagradient.  In _7th ICML Workshop on Automated Machine Learning_ , 2020.
  * Todorov et al. [2012] E. Todorov, T. Erez, and Y. Tassa.  Mujoco: A physics engine for model-based control.  In _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pages 5026–5033, 2012.
  * Lee et al. [2021] Y. Lee, E. S. Hu, and J. J. Lim.  IKEA furniture assembly environment for long-horizon complex manipulation tasks.  In _IEEE International Conference on Robotics and Automation_ , 2021\.
  * Fan et al. [2018] L. Fan, Y. Zhu, J. Zhu, Z. Liu, O. Zeng, A. Gupta, J. Creus-Costa, S. Savarese, and L. Fei-Fei.  Surreal: Open-source reinforcement learning framework and robot manipulation benchmark.  In _Conference on Robot Learning_ , pages 767–782, 2018.
  * Tobin et al. [2017] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel.  Domain randomization for transferring deep neural networks from simulation to the real world.  _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pages 23–30, 2017.
  * Sheckells et al. [2019] M. Sheckells, G. Garimella, S. Mishra, and M. Kobilarov.  Using data-driven domain randomization to transfer robust control policies to mobile robots.  _IEEE International Conference on Robotics and Automation_ , pages 3224–3230, 2019.
  * Zhang et al. [2021] G. Zhang, L. Zhong, Y. Lee, and J. J. Lim.  Policy transfer across visual and dynamics domain gaps via iterative grounding.  In _Robotics: Science and Systems_ , 2021.

##  Appendix A Additional Ablation Study

We discuss further ablation results regarding: (1) The effect of the entropy
coefficient $\alpha$ on the performance as well as convergence speed for
learning our visual policy. (2) The effect of using our proposed weight
initialization strategy for the asymmetric actor-critic framework. (3) The
effect of behavioral cloning for trajectory smoothing over the motion planner
augmented RL trajectories. (4) The dependency of our method on the success of
MoPA-RL.

###  A.1 Effect of varying the entropy coefficient $\alpha$

In Section Section 3.2, we discuss the role of tuning the entropy coefficient
$\alpha$ for maintaining the trade-off between entropy maximization as well as
reward maximization for the SAC objective. For our visual policy learning
using asymmetric actor-critic framework, we mainly focus on the exploration
achieved by our state-based agent using MoPA-RL [1]. This helps us leverage
our state agent’s experience for providing guided exploration rather than
exploring the entire state space again for visual policy learning. Therefore,
we focus on the reward maximization objective by choosing a very small value
of $\alpha$.

Table 3 reports the $\log(\alpha)$ values for our learned state-agent using
MoPA-RL. For accelerating our visual policy learning, we use smaller $\alpha$
values to further emphasize on reward maximization. In Figure 5, we show that
using smaller values of $\alpha$ assists in faster convergence and improving
sample efficiency as compared to using higher $\alpha$ values. Moreover,
smaller $\alpha$ values consistently converge for all the environments whereas
for larger $\alpha$ values, we can see unstable performance varying across
different tasks. We also observe that large values of $\alpha$ undergo large
perturbations during training making learning unstable whereas smaller values
of $\alpha$ remain stable throughout training, as shown in Figure 6.

| Sawyer Push | Sawyer Lift | Sawyer Assembly
---|---|---|---
$\log(\alpha)$ | -0.63 | -2.7 | -0.29
Table 3:  Values of the final $\log(\alpha)$ after MoPA-RL training for all
three environments.

(a) Sawyer Push

(b) Sawyer Lift

(c) Sawyer Assembly

Figure 5: Ablation curves for showing the effect of the entropy coefficient
$\alpha$ for visual policy learning via asymmetric actor-critic framework. As
the trends show, smaller $\alpha$ values lead to a stable and faster training
for all three environments.

(a) Sawyer Push

(b) Sawyer Lift

(c) Sawyer Assembly

Figure 6: Ablation curves for showing the change in the entropy coefficient
$\alpha$ during training for visual policy learning via asymmetric actor-
critic framework. As the trends show, larger $\alpha$ values fluctuate during
training leading to unstable learning, whereas smaller values of $\alpha$ do
not show large variations throughout training for all three environments.

###  A.2 Effect of weight initialization

As discussed in Section Section 4.3, we note that our weight initialization
technique for the actor-critic networks before visual policy learning
significantly aids in achieving optimal performance and also facilitates
improving learning speed. In Figure 7, we see that without appropriate
initialization for the asymmetric critic using our state agent and the visual
actor using behavioral cloning, the agent achieves minimal or no success rate
across all environments up till 1.2M environment steps. On the other hand, our
method with initialization converges to a success rate of $\approx$ 1.0 much
earlier than 1.0M environment steps for all three tasks.

(a) Sawyer Push

(b) Sawyer Lift

(c) Sawyer Assembly

Figure 7: Learning curves comparing our method with and without actor-critic
initialization for all three environments. Without initialization, the agent
achieves minimal or no success rate, whereas it quickly learns to achieve
optimal performance with weight initialization across all the tasks.

###  A.3 Behavioral cloning for trajectory smoothing

As mentioned in Section 3.2.1, we use BC to smooth out jittery motion planner
augmented trajectories and store them in an augmented expert replay buffer
$\mathcal{R}_{e}$. In Figure 8, we qualitatively compare BC and MoPA-RL’s
random rollouts with the same start and goal positions. As we can visualize,
the BC trajectory path is smoother compared to the trajectory obtained via the
MoPA-RL policy. This is indeed significant for a refined transition from a
state-based policy to a visual policy in terms of performance as well as
convergence speed and sample efficiency (see Figure 3).

(a) Sawyer Push

(b) Sawyer Lift

(c) Sawyer Assembly

Figure 8: Visualization of end-effector position for a randomly chosen rollout
comparing the smoothness of behavioral cloning trajectories with respect to
MoPA-RL trajectories with the same start state (marked in green) and final
end-effector position (marked in red) for all three tasks.

###  A.4 Sub-optimal training for MoPA-RL

To analyze the dependence of our approach on the success of the MoPA-RL
training in stage one, we experiment with sub-optimal models of MoPA-RL. We
train on lesser environment interactions and then use the critic for weight
initialization, followed by our asymmetric visual agent training. We train for
the same number of environment steps as Table 1 results, i.e., 2M environment
steps. We report the results in Table 4(a) and Table 4(b) for Sawyer Push and
Sawyer Lift tasks, respectively.

As reported in Table 4(a) and Table 4(b), we observe that although our method
benefits from initialization using the state-based agent compared to random
initialization, it does not heavily depend on the state-based agent’s success.
This shows that even when MoPA-RL cannot completely solve the task, our
framework for distilling the state-based policy into the visual policy
achieves optimal performance.

| Environment Steps
---|---
| 0.2M | 1.0M
| ASR $\uparrow$ | AEL $\downarrow$ | ADR $\uparrow$ | ASR $\uparrow$ | AEL $\downarrow$ | ADR $\uparrow$
MoPA-RL [1] | 69.2 | 102.7 | 41.0 | 98.0 | 105.4 | 54.5
BC-Visual | 93.2 | 137.4 | 37.1 | 98.8 | 57.4 | 97.2
Ours | 100.0 | 34.4 | 108.2 | 100 | 32.0 | 110.8
(a) Sawyer Push

| Environment Steps
---|---
| 0.5M | 0.52M | 1.0M
| ASR $\uparrow$ | AEL $\downarrow$ | ADR $\uparrow$ | ASR $\uparrow$ | AEL $\downarrow$ | ADR $\uparrow$ | ASR $\uparrow$ | AEL $\downarrow$ | ADR $\uparrow$
MoPA-RL [1] | 41.4 | 153.8 | 21.0 | 60.6 | 138.3 | 29.2 | 95 | 109.2 | 52.7
BC-Visual | 48 | 189.4 | 14.1 | 55.2 | 146.6 | 24.2 | 63 | 109.4 | 34.8
Ours | 99.0 | 42.0 | 101.6 | 99.4 | 42.9 | 101.0 | 99.0 | 42.0 | 101.7
(b) Sawyer Lift

Table 4: Average success rate (ASR), episode length (AEL), and discounted
return (ADR) of our method compared with MoPA-RL and BC using sub-optimal and
optimal MoPA-RL models. Even with sub-optimal training, when our state-based
agent does not learn to completely solve the task, our method for distillation
into a visual policy achieves high performance and successfully solves the
task in lesser steps (smaller AEL).

##  Appendix B Implementation Details

###  B.1 Network architecture and hyperparameter selection

For training our state-based agent in step one using MoPA-RL [1], we use three
layered fully connected neural networks with 256 hidden units and ReLU
activation for actor $\pi_{\phi}$ and critic $\mathcal{Q}_{\psi}$ networks and
MP implementation similar to [1] using the RRT-Connect algorithm. For our
asymmetric agent’s critic, we retain the critic network same as
$\mathcal{Q}_{\psi}$ and the visual actor $\pi_{\theta}$ is a 3-layered
convolution neural network followed by three fully connected layers with 256
hidden units and Leaky ReLU activation and another set of fully connected
layers outputting the mean and the standard deviation of the Gaussian
distribution over an action space. The behavioral cloning agent shares the
network architecture with the asymmetric visual actor $\pi_{\theta}$. We
specify other hyperparameter details for SAC and behavioral cloning training
in Table B.1, Table B.1 and Table 7.

Parameter | Value
---|---
Optimizer | Adam
Learning rate | 1e-5
Discount factor ($\gamma$) | 0.99
Replay buffer size | $10^{6}$
Image size | 32x32
Minibatch size | 256
Nonlinearity | ReLU
No. of expert trajectories | 100
Network update per env. step | 1
Table 5: SAC hyperparameters shared across all environments Parameter | Value
---|---
Optimizer | Adam
Learning rate | 5e-4
# observation-action pairs |  $\approx$ 1M
Train-test split | 9:1
Image size | 32x32
Minibatch size | 512
Nonlinearity | LeakyReLU
Scheduler step size | 5
Scheduler decay rate | 0.99
Table 6: Behavioral cloning hyperparameters shared across all environments

| Sawyer Push | Sawyer Lift | Sawyer Assembly
---|---|---|---
Action dimension | 7 | 8 | 7
Reward Scale | 0.8 | 0.5 | 1.0
Table 7: Environment-specific parameters for SAC training

###  B.2 Choosing the optimal model for behavioral cloning

For behavioral cloning training, we use validation success rate for choosing
the optimal model weights rather than using loss on the test dataset. As seen
in the Figure 9, we see that a lower error on the test set does not
necessarily provide the best validation accuracy. Therefore, we pick the model
on the first such epoch which receives 100% validation accuracy. For measuring
the validation accuracy after each training epoch, we use 5 episodes each on 6
randomly chosen seeds. We train our BC agent for a total of 140 epochs and
select the earliest model with the best validation accuracy for BC trajectory
smoothing as well as asymmetric visual actor’s weight initialization. We also
report the training time and number of epochs (for optimal convergence)
required to account for the additional steps required in learning the BC agent
in Table 8.

(a) Loss curves for behavioral cloning

(b) Validation Accuracy for behavioral cloning

Figure 9: Learning curves for behavioral cloning showing loss on the train set, loss on the test set as well as the validation accuracy per epoch for Sawyer Assembly environment. | Sawyer Push | Sawyer Lift | Sawyer Assembly
---|---|---|---
Number of epochs | 12 | 13 | 24
Wall-clock time | 30 | 39 | 120
Table 8: Number of epochs and wall-clock time (in minutes) for training a BC
agent used for weight initialization and BC Smoothing.

##  Appendix C Domain Randomization

For domain randomization, we randomly sample a variation for the texture,
color and lighting conditions for each episode during training. An
illustration of the domain randomization samples for each environment is shown
in Figure 10. Training our method with domain randomized environments helps in
learning robust policies that are capable of transfer to unseen domains and
distractor objects as shown in Section Figure 4.5.

(a) Sawyer Push

(b) Sawyer Lift

(c) Sawyer Assembly

Figure 10: Illustration for random samples of domain randomization for each
environment,

##  Appendix D Environment Details

We simulate all of our 3D environments using MuJoCo physics engine [31].
Subsequently, we describe the observation details for all three environments.
We maintain all other specifics regarding the reward functions, success
criteria as well as environment intial states same as explained in Yamada et
al. [1].

###  D.1 Sawyer Push

The task is to reach an object placed inside a box and push it towards the
goal region using a Sawyer Arm. We define the positions of the goal, object
and the end-effector as $p_{\text{goal}}$, $p_{\text{obj}}$ and
$p_{\text{eef}}$ respectively.

##### Observations.

Each state observation $s_{t}$ comprises of the joint state
$(\sin\theta,\cos\theta)$, angular joint velocity, quaternion end-effector
coordinates, $p_{\text{eef}}$ the position of the object, the position of the
goal $p_{\text{goal}}$, distance between the end-effector and the object, and
the distance between the object and goal position. This acts as an input to
the critic in the asymmetric framework.

The visual observation $o_{t}$ comprises of the simulated image corresponding
to $s_{t}$ and the robot joint space information. This is used as an input to
the actor in the asymmetric framework for learning the visual policy.

###  D.2 Sawyer Lift

For Sawyer Lift, the task is to reach an object placed inside a box, grasp it
using the gripper hand and then lift it up above the box height.

##### Observations.

The state observation $s_{t}$ comprises of the joint state
$(\sin\theta,\cos\theta)$, angular joint velocity, the position of the object
position and quaternion, end-effector coordinates, $p_{\text{eef}}$ , the
position of the goal $p_{\text{goal}}$, and distance between the end-effector
and the object. This acts as an input to the critic in the asymmetric
framework.

For an input to the asymmetric actor for visual policy learning, the visual
observation $o_{t}$ comprises of the simulated image corresponding to $s_{t}$
and the robot joint space information.

###  D.3 Sawyer Assembly

In Sawyer Assembly, the task is to assemble the fourth leg of a tabletop in
its gripper to its corresponding hole where the other three legs are already
in place. This needs to be done while avoiding collisions with the
obstructions posed by the other three table legs that are already assembled
since the table is free to move under collisions.

##### Observations.

The state observation $s_{t}$ comprises of the joint state
$(\sin\theta,\cos\theta)$, angular velocity, the position of the hole, the
head and tail positions of the leg in the gripper hand and its quaternion.

The input to the asymmetric actor for visual policy learning comprises of the
simulated image $o_{t}$ and the robot joint state information.