from the mean in (5.41) as usual, concluding the analysis of the transport
error.

###  7.4 Parameters and inequalities

We need to choose values for the parameters

$\displaystyle\beta\,,\quad
b\,,\quad\Gamma\,,\quad\gamma\,,\quad\ell\,,\quad{\mathbf{N}_{\textnormal{dec}}}\,,\quad{\mathsf{d}}\,,\quad\mathcal{C}_{\mathring{R}}\,,\quad
a\,$

such that all the inequalities throughout the paper are satisfied. We chosen
the parameters in the following order, and according to the following
methodology:

  1. (1)

Let $\beta^{\prime}=1$ be an auxiliary parameter which we use temporarily to
simplify the arithmetic in a number of inequalities. After demonstrating that
each inequality is strict, we shall later choose a value of $\beta>1$ which
preserves each inequality.

  2. (2)

Set $\Gamma=\nicefrac{{1}}{{2}}$.

  3. (3)

Choose $b>600$.

  4. (4)

Choose ${\mathbf{N}_{\textnormal{dec}}}$ such that
$64({\mathbf{N}_{\textnormal{dec}}}+4)<b{\mathbf{N}_{\textnormal{dec}}}(1-\Gamma-\nicefrac{{\Gamma}}{{b}})$.
This is possible since $\Gamma=\nicefrac{{1}}{{2}}$ and $b>600$.

  5. (5)

Choose ${\mathsf{d}}$ such that so that
${\mathsf{d}}(40-b+\Gamma(b-1))<-b+\Gamma(b-1)-32$. This is possible since
$b>600$ and $\Gamma=\nicefrac{{1}}{{2}}$.

  6. (6)

Choose $\gamma\in(0,1)$ such that

    1. (i)

$-2b+(1+\gamma)b+8<-\gamma b^{2}$.

    2. (ii)

$\gamma b-8<-\gamma b^{2}$.

    3. (iii)

$232-b+\Gamma(b-1)<-2\gamma b^{2}$.

    4. (iv)

$264+\Gamma(1-b)<-\gamma b^{2}$.

    5. (v)

$\Gamma(1-b)+\gamma b<-\gamma b^{2}$.

This is possible since we may choose $\gamma$ very close to $0$.

  7. (7)

Choose $\mathcal{C}_{\mathring{R}}$ to ensure that (6.32a) and (6.33a) are
satisfied; this is possible since $\mathcal{C}_{\mathring{R}}$ appears as a
prefactor in the inductive assumption in (2.3) for ${\mathring{R}}_{q}$, and
by extension in (6.24).

Given the choices of $\beta^{\prime}=1$, $\Gamma$, $b$,
${\mathbf{N}_{\textnormal{dec}}}$, ${\mathsf{d}}$, and $\gamma$ above, we may
now verify a number of strict inequalities. We leave room in each inequality
so that later we may choose $\beta>1$, and $a$ large enough to absorb implicit
constants throughout the argument. We introduce the auxiliary parameter

$\delta_{q}^{\prime}=\lambda_{q}^{-2\beta^{\prime}}=\lambda_{q}^{-2}\,$

in order to simplify the arithmetic, and we show at the end that one can in
fact substitute $\beta>1$ for $\beta^{\prime}=1$ while preserving all the
inequalities. For each inequality, we write out everything in terms of
$\lambda_{q}$. We remind the reader that, per definitions (2.2) and (3.1),

$\lambda_{q}=a^{b^{q}}\,,\qquad\delta_{q}=\lambda_{q}^{-2\beta}\,,\qquad\ell=\lambda_{q}^{-8}\,.$
(7.31)

  1. 1.

From the choice of $\ell=\lambda_{q}^{-8}$, and a sufficiently large choice of
$a$ so that $\lambda_{q}^{-1}\leq\alpha^{2}\mathcal{C}_{\mathring{R}}$, we
have that

$\ell^{2}\lambda_{q}^{9}<\alpha^{2}\mathcal{C}_{\mathring{R}}\delta^{\prime}_{q+2}\lambda_{q+2}^{2}\impliedby-16+10<-2\beta^{\prime}b^{2}+2b^{2}\,.$

  2. 2.

From the temporary choice of $\beta^{\prime}=1$, we have that

$\ell<\lambda_{q+1}^{2}\delta^{\prime}_{q+1}\iff-8<2b^{2}-2\beta^{\prime}b^{2}\,.$

  3. 3.

From the choices $\ell=\lambda_{q}^{-8}$ and $b>600$, we have that

$\ell^{-12}<\lambda_{q+1}\iff 96<b\,.$

  4. 4.

From the choice of ${\mathbf{N}_{\textnormal{dec}}}$, we have that

$\ell^{-8({\mathbf{N}_{\textnormal{dec}}}+4)}<\lambda_{q+1}^{{\mathbf{N}_{\textnormal{dec}}}(1-\Gamma-\nicefrac{{\Gamma}}{{b}})}\iff
64({\mathbf{N}_{\textnormal{dec}}}+4)<b{\mathbf{N}_{\textnormal{dec}}}(1-\Gamma-\nicefrac{{\Gamma}}{{b}})\,.$

  5. 5.

From $b>600$ and $\Gamma=\nicefrac{{1}}{{2}}$, we have that

$r_{q}^{-1}<\lambda_{q+1}\,\iff\Gamma(b-1)<b\,.$

  6. 6.

\- From the temporary choice of $\beta^{\prime}=1$ and the choice of $\gamma$,
we have that

$\delta^{\prime}_{q+1}\lambda_{q+1}^{2+\gamma}(\ell\lambda_{q+1})^{-1}<\lambda_{q+2}^{2-\gamma}\delta^{\prime}_{q+2}\iff-2\beta^{\prime}b+(1+\gamma)b+8<-2\beta^{\prime}b^{2}+(2-\gamma)b^{2}\,.$

  7. 7.

From the temporary choice of $\beta^{\prime}=1$ and the choice of $\gamma$, we
have that

$\delta^{\prime}_{q+1}\lambda_{q+1}^{2+\gamma}\ell<\lambda_{q+2}^{2-\gamma}\delta^{\prime}_{q+2}\iff-2\beta^{\prime}b+(2+\gamma)b-8<-2\beta^{\prime}b^{2}+(2-\gamma)b^{2}\,.$

  8. 8.

From the temporary choice of $\beta^{\prime}=1$, the choice of $b$, and the
choice of $\gamma$, we have that

$\delta^{\prime}_{q+1}r_{q}\lambda_{q+1}^{2+\gamma}<\delta^{\prime}_{q+2}\lambda_{q+2}^{2-\gamma}\iff-2\beta^{\prime}b+2b+\Gamma(1-b)+\gamma
b<-2\beta^{\prime}b^{2}+2b^{2}-\gamma b^{2}\,.$

  9. 9.

From the choice of ${\mathsf{d}}$, we have that

$\left(\frac{\ell^{-5}}{\lambda_{q+1}r_{q}}\right)^{\mathsf{d}}\ell^{-4}<\left(\lambda_{q+1}r_{q}\right)^{-1}\iff{\mathsf{d}}(40-b+\Gamma(b-1))<-b+\Gamma(b-1)-32\,.$

  10. 10.

As in the last inequality, the choice of ${\mathsf{d}}$ ensures that

$\left(\frac{\ell^{-5}}{\lambda_{q+1}}\right)^{\mathsf{d}}\ell^{-20}<\lambda_{q+1}^{-1}\iff{\mathsf{d}}(40-b)+160<-b\,.$

  11. 11.

From the temporary choice $\beta^{\prime}=1$, the inequality $b>600$, and the
choice of $\gamma$, we have that

$\ell^{-9}\ell^{-20}\left(\lambda_{q+1}r_{q}\right)^{-1}<\delta^{\prime}_{q+2}\lambda_{q+2}^{2-2\gamma}\iff
232-b+\Gamma(b-1)<-2\beta^{\prime}b^{2}+(2-2\gamma)b^{2}\,.$

  12. 12.

From the temporary choice of $\beta^{\prime}=1$, the choice
$\Gamma=\nicefrac{{1}}{{2}}$, and the choice of $b>600$, we have that

$\ell^{-33}r_{q}<\delta^{\prime}_{q+2}\lambda_{q+2}^{2-\gamma}\iff
264+\Gamma(1-b)<-2\beta^{\prime}b^{2}+(2-\gamma)b^{2}\,.$

Now observe that each of the inequalities above is strict, and all quantities
are continuous with respect to $\beta^{\prime}$. Therefore we may choose
$\beta>1$ so that all inequalities _remain_ strict after substitution of
$\beta>1$ for $\beta^{\prime}=1$. At this point, we may now choose $a$ large
enough so that all implicit constants throughout the paper do not exceed the
extra room still present in each inequality.

######  Remark 7.3.

Aside from estimating the intersection of Mikado flows which belong to the
same Lagrangian coordinate system by Proposition 5.3, the only significant
change required to apply our arguments to the 2D Euler-$\alpha$ equations is
the fact that the intermittency gain from $L^{2}$ to $L^{1}$ for intermittent
Mikado flows is only of order $r_{q}^{\nicefrac{{1}}{{2}}}$ instead of
$r_{q}$. When choosing parameters, this only affects the Nash and transport
error estimates, which held from inequality (12). Substituting
$\nicefrac{{1}}{{2}}\cdot\Gamma(1-b)$ for $\Gamma(1-b)$ necessitates a larger
choice of $b$, but the rest of the parameter choices may be made identically,
and so we achieve analogous results.

##  Appendix A Proof of Lemma 1.5

In this section we provide a proof of Lemma 1.5.

###### Proof.

We follow the proof from [CET94]. Mollifying (1.8) in space with a standard
radial, compactly supported mollifier $\varphi_{\varepsilon}$, assuming
differentiability in time, and integrating in space against
$u_{i}^{\varepsilon}=u\ast\varphi_{\varepsilon}$ gives

$\displaystyle\frac{1}{2}\frac{d}{dt}\left(\|u_{i}^{\varepsilon}\|_{L^{2}}^{2}+\alpha^{2}\|\nabla
u_{i}^{\varepsilon}\|_{L^{2}}^{2}\right)$
$\displaystyle=\int_{\mathbb{T}^{n}}(u_{j}u_{i})^{\varepsilon}\partial_{j}u_{i}^{\varepsilon}+\alpha^{2}(u_{j}\partial_{k}u_{i})^{\varepsilon}\partial_{j}\partial_{k}u_{i}^{\varepsilon}+\alpha^{2}(\partial_{k}u_{j}\partial_{k}u_{i})^{\varepsilon}\partial_{j}u_{i}^{\varepsilon}-\alpha^{2}(\partial_{k}u_{j}\partial_{i}u_{j})^{\varepsilon}\partial_{k}u_{i}^{\varepsilon}\,dx\,$
(A.1)

where we have used Remark 1.3.

We wish to show that the right-hand side vanishes when $\varepsilon$ is sent
to 0. We recall some standard estimates for mean-zero $f\in B_{3,\infty}^{s}$
where ${1<s<2}$; see for example [CET94, bcd].

$\displaystyle\|\nabla f(\cdot)-\nabla
f(\cdot-y)\|_{L^{3}}\lesssim|y|^{s-1}\|\nabla f\|_{B_{3,\infty}^{s-1}}$ (A.2a)
$\displaystyle\|f-f^{\varepsilon}\|_{L^{3}}\lesssim\varepsilon\|f\|_{B_{3,\infty}^{s}}$
(A.2b) $\displaystyle\|\nabla f-\nabla
f^{\varepsilon}\|_{L^{3}}\lesssim\varepsilon^{s-1}\|\nabla
f\|_{B_{3,\infty}^{s-1}}$ (A.2c)
$\displaystyle\|\nabla^{2}f^{\varepsilon}\|_{L^{3}}\lesssim\varepsilon^{s-2}\|\nabla
f\|_{B_{3,\infty}^{s-1}}$ (A.2d) $\displaystyle{\left\|\nabla
f\right\|_{L^{3}}\lesssim\left\|\nabla f\right\|_{B^{s-1}_{3,\infty}}}$ (A.2e)
$\displaystyle\left\|\nabla
f\right\|_{B^{s-1}_{3,\infty}}\lesssim\left\|f\right\|_{B^{s}_{3,\infty}}$
(A.2f)
$\displaystyle{\left\|f^{\varepsilon}\right\|_{B^{s}_{3,\infty}}\lesssim\left\|f\right\|_{B^{s}_{3,\infty}}}\,.$
(A.2g)

Finally, recall the double commutator identity from [CET94]:

$(fg)^{\varepsilon}=f^{\varepsilon}g^{\varepsilon}-(f-f^{\varepsilon})(g-g^{\varepsilon})+r_{\varepsilon}(f,g)\,,$
(A.3)

where $f^{\varepsilon}(x)=f*\varphi_{\varepsilon}(x)$ and

$r_{\varepsilon}(f,g)=\int_{\mathbb{T}^{n}}\varphi_{\varepsilon}(y)(f(x-y)-f(x))(g(x-y)-g(x))\,dy\,.$
(A.4)

Then the right-hand side of (A) can be written as

$\displaystyle\int_{\mathbb{T}^{n}}(u_{j}u_{i})^{\varepsilon}\partial_{j}u_{i}^{\varepsilon}+\alpha^{2}(u_{j}\partial_{k}u_{i})^{\varepsilon}\partial_{j}\partial_{k}u_{i}^{\varepsilon}+\alpha^{2}(\partial_{k}u_{j}\partial_{k}u_{i})^{\varepsilon}\partial_{j}u_{i}^{\varepsilon}-\alpha^{2}(\partial_{k}u_{j}\partial_{i}u_{j})^{\varepsilon}\partial_{k}u_{i}^{\varepsilon}dx$
$\displaystyle=\int_{\mathbb{T}^{n}}u_{j}^{\varepsilon}u_{i}^{\varepsilon}\partial_{j}u_{i}^{\varepsilon}+\alpha^{2}\left[u_{j}^{\varepsilon}\partial_{k}u_{i}^{\varepsilon}\partial_{j}\partial_{k}u_{i}^{\varepsilon}+\partial_{k}u_{j}^{\varepsilon}\partial_{k}u_{i}^{\varepsilon}\partial_{j}u_{i}^{\varepsilon}-\partial_{k}u_{j}^{\varepsilon}\partial_{i}u_{j}^{\varepsilon}\partial_{k}u_{i}^{\varepsilon}\right]dx$
$\displaystyle-\int_{\mathbb{T}^{n}}(u_{j}-u_{j}^{\varepsilon})(u_{i}-u_{i}^{\varepsilon})\partial_{j}u_{i}^{\varepsilon}dx$
$\displaystyle-\alpha^{2}\int_{\mathbb{T}^{n}}\left[(u_{j}-u_{j}^{\varepsilon})(\partial_{k}u_{i}-\partial_{k}u_{i}^{\varepsilon})\partial_{j}\partial_{k}u_{i}^{\varepsilon}+(\partial_{k}u_{j}-\partial_{k}u_{j}^{\varepsilon})(\partial_{k}u_{i}-\partial_{k}u_{i}^{\varepsilon})\partial_{j}u_{i}^{\varepsilon}-(\partial_{k}u_{j}-\partial_{k}u_{j}^{\varepsilon})(\partial_{i}u_{j}-\partial_{i}u_{j}^{\varepsilon})\partial_{k}u_{i}^{\varepsilon}\right]dx$
$\displaystyle+\int_{\mathbb{T}^{n}}r_{\varepsilon}(u_{j},u_{i})\partial_{j}u_{i}^{\varepsilon}+\alpha^{2}\left[r_{\varepsilon}(u_{j},\partial_{k}u_{i})\partial_{j}\partial_{k}u_{i}^{\varepsilon}+r_{\varepsilon}(\partial_{k}u_{j},\partial_{k}u_{i})\partial_{j}u_{i}^{\varepsilon}-r_{\varepsilon}(\partial_{k}u_{j},\partial_{i}u_{j})\partial_{k}u_{i}^{\varepsilon}\right]dx\,.$
(A.5)

We have

$\int_{\mathbb{T}^{n}}u_{j}^{\varepsilon}u_{i}^{\varepsilon}\partial_{j}u_{i}^{\varepsilon}dx=\int_{\mathbb{T}^{n}}u_{j}^{\varepsilon}\partial_{k}u_{i}^{\varepsilon}\partial_{j}\partial_{k}u_{i}^{\varepsilon}=0$

using integration by parts and that $\operatorname{\mathrm{div}}u=0$.
Furthermore, the equality

$\partial_{k}u_{j}^{\varepsilon}\partial_{k}u_{i}^{\varepsilon}\partial_{j}u_{i}^{\varepsilon}=\partial_{k}u_{j}^{\varepsilon}\partial_{i}u_{j}^{\varepsilon}\partial_{k}u_{i}^{\varepsilon}$

follows from switching the roles of $i$ and $j$. Therefore the first four
terms after the equal sign on the right-hand side of (A.5) vanish, and we can
bound the remaining terms by

$\displaystyle\lesssim\|u-u^{\varepsilon}\|_{L^{3}}^{2}\|\nabla
u^{\varepsilon}\|_{L^{3}}+\|u-u^{\varepsilon}\|_{L^{3}}\|\nabla u-\nabla
u^{\varepsilon}\|_{L^{3}}\|\nabla^{2}u^{\varepsilon}\|_{L^{3}}+\|\nabla
u-\nabla u^{\varepsilon}\|_{L^{3}}^{2}\|\nabla u^{\varepsilon}\|_{L^{3}}$
$\displaystyle+\|r_{\varepsilon}(u,u)\|_{L^{\frac{3}{2}}}\|\nabla
u^{\varepsilon}\|_{L^{3}}+\|r_{\varepsilon}(u,\nabla
u)\|_{L^{\frac{3}{2}}}\|\nabla^{2}u^{\varepsilon}\|_{L^{3}}+\|r_{\varepsilon}(\nabla
u,\nabla u)\|_{L^{\frac{3}{2}}}\|\nabla u^{\varepsilon}\|_{L^{3}}$
$\displaystyle\lesssim\varepsilon^{{2}}\|u\|_{B_{3,\infty}^{s}}^{3}+\varepsilon^{{2(s-1)}}\|u\|_{B_{3,\infty}^{s}}^{3}.$

Thus, taking $s>1$ shows that the right-hand side of (A) converges to zero
when $\varepsilon\rightarrow 0$. Concluding the proof in the case of
continuity in time may be done analogously as for the classical Euler
equations, and we omit further details. ∎

##  Appendix B Proof of Lemma 4.1

In this section we provide a proof of Lemma 4.1.

###### Proof.

We first construct $\mathcal{K}_{0}$ and $c_{i}^{0}$ by hand while allowing
$\left\|{\mathring{R}}\right\|_{0}\leq 1$, afterwards constructing
$\mathcal{K}_{n}$ for $1\leq n\leq N$ and choosing $\varepsilon$. Consider a
symmetric traceless matrix ${\mathring{R}}$, which without loss of generality
may be written as

${\mathring{R}}=\begin{pmatrix}a&c&d\\\ c&b&e\\\ d&e&-a-b\end{pmatrix}\,.$
(B.1)

Notice that the set of such matrices is 5-dimensional, and combined with the
condition on the sum of $(c_{i}^{0})^{2}$ in (4.1) will require a set of at
least six vectors. The extra three vectors will ensure that the coefficients
are all positive. Ignoring for now the upper subscript $0$ on the vectors
$k_{i}^{0}\in\mathcal{K}_{0}$, we set

$k_{1}=e_{1}\,,\qquad k_{2}=e_{2}\,,\qquad k_{3}=e_{3}\,.$

For ease of notation, let us define

$f(k_{i})=3k_{i}\otimes k_{i}-\textnormal{Id}\,.$

Then it is clear that $f(k_{1})$, $f(k_{2})$, and $f(k_{3})$ are symmetric
traceless matrices which only contain entries on the diagonal; specifically,
we have

$f(k_{1})=\begin{pmatrix}2&0&0\\\ 0&-1&0\\\ 0&0&-1\end{pmatrix}\,,\qquad
f(k_{2})=\begin{pmatrix}-1&0&0\\\ 0&2&0\\\ 0&0&-1\end{pmatrix}\,,\qquad
f(k_{3})=\begin{pmatrix}-1&0&0\\\ 0&-1&0\\\ 0&0&2\end{pmatrix}\,.\qquad$ (B.2)

We shall use $f(k_{1})$ and $f(k_{2})$ to engender the entries of the matrix
_on_ the diagonal in (B.1), while we shall use a “balanced” sum of the form
$c(f(k_{1})+f(k_{2})+f(k_{3}))$ to ensure the second condition in (4.1). We
further set

$\displaystyle k_{4}$ $\displaystyle=\frac{3e_{1}+4e_{2}}{5}\,,\qquad
k_{5}=\frac{3e_{1}+4e_{3}}{5}\,,\qquad k_{6}=\frac{3e_{2}+4e_{3}}{5}$
$\displaystyle k_{7}$ $\displaystyle=\frac{3e_{1}-4e_{2}}{5}\,,\qquad
k_{8}=\frac{3e_{1}-4e_{3}}{5}\,,\qquad k_{9}=\frac{3e_{2}-4e_{3}}{5}\,.$

Then

$\displaystyle f(k_{4})$ $\displaystyle=\frac{1}{25}\begin{pmatrix}2&36&0\\\
36&23&0\\\ 0&0&-25\end{pmatrix}\,,\qquad
f(k_{5})=\frac{1}{25}\begin{pmatrix}2&0&36\\\ 0&-25&0\\\
36&0&23\end{pmatrix}\,,\qquad f(k_{6})=\frac{1}{25}\begin{pmatrix}-25&0&0\\\
0&2&36\\\ 0&36&23\end{pmatrix}$ (B.3) $\displaystyle f(k_{7})$
$\displaystyle=\frac{1}{25}\begin{pmatrix}2&-36&0\\\ -36&23&0\\\
0&0&-25\end{pmatrix}\,,\quad f(k_{8})=\frac{1}{25}\begin{pmatrix}2&0&-36\\\
0&-25&0\\\ -36&0&23\end{pmatrix}\,,\quad
f(k_{9})=\frac{1}{25}\begin{pmatrix}-25&0&0\\\ 0&2&-36\\\
0&-36&23\end{pmatrix}\,.$

We shall use $f(k_{4})$, $f(k_{5})$, $f(k_{6})$, $f(k_{7})$, $f(k_{8})$, and
$f(k_{9})$ to engender the entries of (B.1) _off_ the diagonal.

It is simple to check that the set
$\\{f(k_{1}),f(k_{2}),f(k_{4}),f(k_{5}),f(k_{6})\\}$ is a linearly independent
set in the space of symmetric traceless matrices. Therefore, there exist
smooth functions
$\\{\tilde{c}_{1},\tilde{c}_{2},\tilde{c}_{4},\tilde{c}_{5},\tilde{c}_{6}\\}$
given by the solutions of a linear system of equations such that for all
symmetric traceless matrices ${\mathring{R}}$ satisfying
$\left\|{\mathring{R}}\right\|_{0}\leq 1$,

$\tilde{c}_{1}f(k_{1})+\tilde{c}_{2}f(k_{2})+\tilde{c}_{4}f(k_{4})+\tilde{c}_{5}f(k_{5})+\tilde{c}_{6}f(k_{6})={\mathring{R}}\,.$
(B.4)

Unfortunately the functions $\tilde{c}_{i}$ are not strictly positive. Let us
define the auxiliary parameter

$c_{0}=\max_{\begin{subarray}{c}\left\|{\mathring{R}}\right\|_{0}\leq
1,i=4,5,6\end{subarray}}|\tilde{c}_{i}({\mathring{R}})|\,.$

Then from (B.4) and (B.3), we have that

$\displaystyle\left(1-\delta_{ij}\right)$
$\displaystyle\bigg{(}\tilde{c}_{1}f(k_{1})+\tilde{c}_{2}f(k_{2})+\tilde{c}_{4}f(k_{4})+\tilde{c}_{5}f(k_{5})+\tilde{c}_{6}f(k_{6})$
$\displaystyle\qquad+2c_{0}\left(f(k_{4})+f(k_{5})+f(k_{6})+f(k_{7})+f(k_{8})+f(k_{9})\right)\bigg{)}^{ij}=\left(1-\delta_{ij}\right){\mathring{R}}^{ij}\,;$

that is, off the diagonal, the matrix on the left hand side of the equation is
equal to ${\mathring{R}}$. The advantage now is that the coefficients on
$f(k_{i})$ for $4\leq i\leq 9$ are all positive. In order to ensure equality
_on_ the diagonal, we may replace the coefficients $\tilde{c}_{1}$ and
$\tilde{c}_{2}$ on $f(k_{1})$ and $f(k_{2})$ with new coefficients
$\dot{c}_{1}$ and $\dot{c}_{2}$ such that

$\displaystyle\bigg{(}\dot{c}_{1}f(k_{1})+\dot{c}_{2}f(k_{2})+\tilde{c}_{4}f(k_{4})+\tilde{c}_{5}f(k_{5})+\tilde{c}_{6}f(k_{6})$
$\displaystyle\qquad\qquad+2c_{0}\left(f(k_{4})+f(k_{5})+f(k_{6})+f(k_{7})+f(k_{8})+f(k_{9})\right)\bigg{)}^{ij}={\mathring{R}}^{ij}\,$
(B.5)

for all $1\leq i,j\leq 3$. Since the coefficients $\dot{c}_{1}$ and
$\dot{c}_{2}$ are not necessarily positive, we set

$\dot{c}_{0}=\max_{\left\|{\mathring{R}}\right\|_{0}\leq
1,i=1,2}\left|\dot{c}_{i}({\mathring{R}})\right|\,.$

Then from (B.2) and (B.5), we have that

$\displaystyle\bigg{(}\dot{c}_{1}f(k_{1})+\dot{c}_{2}f(k_{2})+2\dot{c}_{0}\left(f(k_{1})+f(k_{2})+f(k_{3})\right)$
$\displaystyle\qquad+\tilde{c}_{4}f(k_{4})+\tilde{c}_{5}f(k_{5})+\tilde{c}_{6}f(k_{6})+2c_{0}\left(f(k_{4})+f(k_{5})+f(k_{6})+f(k_{7})+f(k_{8})+f(k_{9})\right)\bigg{)}^{ij}={\mathring{R}}^{ij}\,$

Aggregating coefficients for each of the nine tensors on the left-hand side,
we see that each is strictly positive. We still have to ensure the second
condition in (4.1); however, this is easily achieved by replacing the constant
$\dot{c}_{0}$ with a non-negative, bounded function
$c_{0}({\mathring{R}}):B_{1}(0)\rightarrow[0,\infty)$ which depends on
${\mathring{R}}$ and imposes that the sum of the coefficients is equal for all
${\mathring{R}}\in B_{1}(0)$. Thus we have achieved everything in (4.1) for
$n=0$. Note that the value of $\mathcal{C}_{\rm sum}$ could in principle be
computed explicitly and depends on the choice of the function
$c_{0}({\mathring{R}})$.

To construct $\mathcal{K}_{n}$ for $n=1$, choose a rational rotation matrix
$O_{1}$ with $0<\left\|O_{1}-\textnormal{Id}\right\|_{0}\ll 1$, and set
$\mathcal{K}_{1}=O_{1}\mathcal{K}_{0}$. Plugging the rotated vectors
$O_{1}k_{i}^{0}$ into (4.1), we find that the effect of $O_{1}$ is that

$\sum_{i=1}^{9}\left(c_{i}^{0}\right)^{2}\left({\mathring{R}}\right)\left(3k_{i}^{0}\otimes
k_{i}^{0}-\textnormal{Id}\right)={\mathring{R}}\quad\rightarrow\quad\sum_{i=1}^{9}\left(c_{i}^{0}\right)^{2}\left({\mathring{R}}\right)O_{1}\left(3k_{i}^{0}\otimes
k_{i}^{0}-\textnormal{Id}\right)O_{1}^{T}=O_{1}{\mathring{R}}O_{1}^{T}\,.$

Using the equalities $\textnormal{tr}(AB)=\textnormal{tr}(BA)$ and
$O^{T}=O^{-1}$, we have that $O_{1}{\mathring{R}}O_{1}^{T}$ is still traceless
and symmetric. Define

$c^{1}_{i}\left({\mathring{R}}\right)=c^{0}_{i}\left(O_{1}^{T}{\mathring{R}}O_{1}\right)\,$

for $\left\|{\mathring{R}}\right\|$ sufficiently small so that
$c^{0}_{i}\left(O_{1}{\mathring{R}}O_{1}^{T}\right)$ is well-defined and
strictly positive for each $i$. Then this specifies a value $\varepsilon_{1}$
such that for all $\left\|{\mathring{R}}\right\|_{0}\leq\varepsilon_{1}$,

$\displaystyle\sum_{i=1}^{9}\left(c_{i}^{1}\right)^{2}\left({\mathring{R}}\right)\left(3k_{i}^{1}\otimes
k_{i}^{1}-\textnormal{Id}\right)$
$\displaystyle=\sum_{i=1}^{9}\left(c^{0}_{i}\right)^{2}\left(O_{1}^{T}{\mathring{R}}O_{1}\right)O_{1}\left(3k_{i}^{0}\otimes
k_{i}^{0}-\textnormal{Id}\right)O_{1}^{T}$ $\displaystyle={\mathring{R}}\,,$

showing that the first equality in (4.1) is satisfied for $n=1$. The second
equality in (4.1) comes immediately from the construction of the $c_{i}^{0}$’s
and $c^{1}_{i}$’s. In addition, $\mathcal{K}_{0}$ and $\mathcal{K}_{1}$ will
be disjoint if $0<\left\|O_{1}-\textnormal{Id}\right\|_{0}\ll 1$. Iterating
this procedure and taking the minimum value of $\varepsilon_{n}$ concludes the
proof. ∎

### Acknowledgements

The authors thank Vlad Vicol and Steve Shkoller for stimulating conversations.
RB was partially supported by the NSF under Grant No. DMS-1911413 and No.
DMS-195435. MN was partially supported by the NSF under Grant No. DMS-1928930
while participating in a program hosted by the Mathematical Sciences Research
Institute during the spring 2021 semester, and by the NSF under Grant No.
DMS-1926686 while a member at the Institute for Advanced Study.

## References


# Self-Supervised Learning for Joint Pushing and Grasping Policies
in Highly Cluttered Environments

Kamal Mokhtar1, Cock Heemskerk2, Hamidreza Kasaei1 1Department of Artificial
Intelligence, Bernoulli Institute, University of Groningen, 9747 AG, The
Netherlands.2 Heemskerk Innovative Technology, Delft, the Netherlands
email<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Robots often face situations where grasping a goal object is desirable but not
feasible due to other present objects preventing the grasp action. We present
a deep Reinforcement Learning approach to learn grasping and pushing policies
for manipulating a goal object in highly cluttered environments to address
this problem. In particular, a dual reinforcement learning model approach is
proposed, which presents high resilience in handling complicated scenes,
reaching $98\%$ task completion using primitive objects in a simulation
environment. To evaluate the performance of the proposed approach, we
performed two extensive sets of experiments in packed objects and a pile of
objects scenarios. Experimental results showed that the proposed method worked
very well in both scenarios and outperformed the recent state-of-the-art
approach [1] with a large margin. Trained models and source code for the
results reproducibility purpose are publicly available. here.

###### Index Terms:

polyhedral objects, antipodal grasp, Graphics Processing Unit, Reinforcement
Learning

##  I Introduction

Grasping is a crucial skill for many robotics applications. It enables robots
to perform numerous tasks, which are otherwise impossible without a versed
object’s grasp. It further opens opportunities to robots to numerous skills
that are not attainable without a versed object’s grasp. In many scenarios,
objects do not appear in isolation, and therefore, grasping a target object
might not be feasible due to occlusion and cluttered presence in the
environment. Inspired by human dexterity, a robot should have a set of
primitive motions such as pushing to single out the target object from the
clutter and make future grasp feasible. Such a sequence of actions by a robot
is depicted in Fig. 1. In order to reach this level of reasoning, the robot
must consider the visual observation of the goal object(s) and emphasize the
spatial equivalence to reach a successful grasp. In other words, object
grasping is not a simple antipodal grasp action. It requires a cohesive system
that all operate in harmony resulting in a smooth action. Starting from the
sensory information of the robot acquired from its surrounding environment,
which could have flaws, the robot needs to handle such circumstances.
Additionally, the scene can change dynamically after each action and before a
task is completed, which requires the robot’s resilience. Furthermore, the
motion planning of a robotic arm to get into the desired pose is challenging
in itself. Therefore, the robot needs to observe the environment and adapt
accordingly constantly [2].

Figure 1: Infeasible grasp action as a result of densely packed blocks,
requiring pre-manipulation push action: (left) in this example, the target
object, shown in green, is not graspable due to other objects present; the
robot pushes the target object in a way to make it graspable; (right) finally,
the robot grasps the object successfully.

This paper mainly focuses on self-supervised learning for pushing and grasping
objects in highly cluttered environments via deep Reinforcement Learning (RL).
There has been number of study’s evolving around the concept of the synergy
between pushing and grasping, i.e., primitive motion push to grasp [3] [1] [4]
[5] [6] [7] [8]. However, most of the previous work is intended for bin
picking, emptying objects from one bin, and dropping them in another. In this
work, we exploit scenarios where the goal object is initially not graspable
due to surrounding clutter. The synergy between pushing and grasping
flourishes the coherent behavior between pushing and grasping. Unlike having
any models working independently from the other, which often result in
unpredictable changes in the scene and strain in solving the issue, we propose
a deep RL approach to exploit the potentials and limitations of the synergy
between pushing and grasping. The overall system architecture, which will be
discussed in more detail in section III is demonstrated in Fig. 2. In summary,
our contributions are threefold:

Figure 2: Overview of the entire system. RGB-D camera and UR5 robot arm in
CoppeliaSim simulation environment. The camera sensory data undergoes
orthographic transformation, and the goal and all objects mask are generated.
The color heightmap, depth heightmap, and goal mask are fed into the Grasp net
and the push net, all of which experience 16 rotation steps, covering 360
degrees rotation rotating clockwise and starting at 0 degrees. The models
produced pixel-wise Q values are considered whether to push or grasp and
eventually execute the action. In the testing phase, the all objects mask is
used.

  * •

We develop a self-supervised learning approach for a service robot to jointly
learn push and grasp policies to clear occlusion around goal objects, which
will facilitate grasping the goal object and completing the task.

  * •

We perform extensive sets of experiments to evaluate the proposed approach in
both packed (structured) and pile (unstructured and complicated) scenarios.
Experimental results show that the proposed method outperformed the current
state-of-the-art approach [1] by a large margin regarding task completion
rate, grasp success rate, and the number of motions (see Sec. IV).

  * •

To the benefit of research communities, we release the source code and the
trained models, making it possible to reproduce our
results111https://github.com/Kamalnl92/Self-Supervised-Learning-for-pushing-
and-grasping.

##  II Related Work

This section will mainly cover two fundamental vital aspects of the problem we
are tackling, the object grasping problem and the pre-grasp manipulation.

###  II-A Grasping

Autonomous grasping is a challenging problem that has undergone enormous
improvement in recent years, especially with Artificial Intelligence (AI)
practical approaches. However, robots’ dexterity in grasping is one of the
most primitive manipulations that allow robots to operate further complex
tasks [9]. Classical approaches rely on a model-based approach, where pre-
knowledge of the object model is used to estimate force closures and grasp
orientation. These methods impose limitations, as it is essential to have
accurate modeling of the objects, and that is not always possible [10].

A mathematical model describing objects is crucial to predict the physics of
the hand and the grasped object. Various conditions arise during grasping,
which is essential to be expressed by the model to achieve stable behavior
resisting the different loading conditions [11].

The substantial increase in computing power has allowed the deep-learning
methods to prevail in solving the issue. It is focused on data-driven
methodology; the data are perceived in the form of image and depth information
of the scene. Then in a simulation setting, the robot exploits the learned
visual features to perform a grasp action. The advantage of such an approach
is that it does not require explicit knowledge of the objects compared to the
classic approaches, i.e., model-based [9]. State-of-the-art and leading work
that uses observed geometry to evaluate antipodal grasp pose [12] [13], has
sparked enormous interest in the deep learning approach. Inspired by the
efficient framework, many studies have taken different approaches in
representing the data around the deep learning model, or different models have
been used, i.e., Generative Residual Convolutional Neural Network (GR-ConvNet)
as model [14]. Another example is research work that generates multi-view 3D
object grasping, where objects are represented in the orthographic
representation of the depth image from three virtual orthographic views
representing the object. After one of the views is selected, it is fed to an
auto-encoder that produces a heat-map for a grasp configuration selection
[15].

###  II-B Pre-grasp manipulation

RL is widely-used when there is a demand on a cognitive capability [16], i.e.,
approaching a goal object taking into consideration other objects that are
present in the environment. This field of occluded environments has not been
studied much, and most studies concerns bin picking (all objects are picked
from one bin and placed in another) [3]. Little work is done about having a
goal object in a cluttered scene, and a robot arm is attempting to grasp it,
which is a fundamental trait for a service robot. This requires efficient pre-
manipulation of other objects to be able to grasp the goal object.

Humans have a rich set of manipulation strategies desirable for service
robots. These traits are appropriate for a service robot to be able to
autonomously perform a task in our peoples’ environment [17]. Objects that are
in a configuration that is impossible to grasp, i.e., a flat object such as a
compact disc (CD), need to be slid to an edge of a table to permit a grasp
action [18]. In another case, objects cannot be grasped due to the presents of
other objects disabling a grasp action. The latter is the focus of this paper;
pre-grasping manipulation such as shifting is a popular approach that permits
consequent object grasping. A demonstration of this behavior, see Figure 1, in
the first state object is impossible to be grasped, the robot arm pushes the
object further away from the clutter such that it permits a grasp action.
There have been many recent studies concerning this problem, which we will
present in the remaining of this section.

Introducing Markov decision (MDP) with a state-space the scene, action space,
transition distribution, and reward functions are elements for the RL
approach. RL is bound to data consumption, and a time-dependent task
introduces the problem of spars-rewards. The solution to this problem in an
MDP setting is to map the current state to action and make the skill of
shifting explicitly dependent on the grasping [4].

Most of the former works are top-down grasps; however, that is not sufficient
at all times. Especially if the aim is to have safe, stable grapes, it is
crucial to include a grasp evaluation of objects in all different
orientations. A method that uses reasoning to find an effective grasp sequence
to grasp a goal object in a structured clutter uses a cascaded process. By
first learning the grasp distribution for a single isolated object and then a
discriminate model that captures the collision of the gripper and the present
objects in the scene. It is possible to eliminate specific unattainable
grasps. The Variation Autoencoder (VAE) is used to evaluate the grasp
distribution of the segmented point cloud of the target object as well as the
collision check [5].

The synergy between pushing and grasping of objects in clutter remains sample
inefficient for the model to receive a reward only after the goal object is
grasped. To mitigate the issue, we relabel the goal object, also known as
Hindsight Experience Replay (HER) [19]. It contributes to enriching the replay
buffer and, in turn, faster learning. We used Xu et al. [1] as a starting
point for our project. Thus, it will be explained in more detail. The modeling
synergy of the push-grasping goal-oriented problem as Markov Decision Process
(MDP) enriches the concept of model decision making in some parts of it are
random and others under control. For the process of pushing and grasping,
there are many reasons that action previously succeeded in execution will not
work in current or future attempts due to simulation physics inconsistency.
These factors are challenging to be all capture if the model requires total
dependence on it. The approach of using Visual Pushing for Grasping (VPG)
coined by [3] gives the system the flexibility of less relying on the physics
of the simulation environment. As it learns to cope with it from experience.
The push action is landing the robot arm on desired coordinates then
performing pushing action in one selected direction from the 16 directions.
Moreover, grasping in a similar strategy, with respect to the orientation of
the object grasping is performed.

##  III Method

This section discusses the overall system architecture, the grasp and the push
models hierarchical framework, and the training procedure in detail.

###  III-A Overall system

As shown in Fig. 2, the system produces two actions, push or grasp, depending
on what is feasible in the environment. Grasping the goal object is executed
when the likelihood of grasping the goal object exceeds the threshold.
Otherwise, the goal object or its surrounding objects are pushed approximately
$1.3$cm away from the current position to increase the probability of grasping
the goal object successfully. The grasp/push points and direction are
indicated on the image coordinate (output of the networks) and then
transformed to the robot’s reference frame to be executed.

We model the synergy between push-grasp and goal-oriented grasp as a goal-
oriented MDP problem. The policy is represented as $\pi(s_{t}|g)$, the reward
as $R(s_{t},a_{t},g)$, and lastly the $Q$-value function as
$Q(s_{t},a_{t},g)$. Pushing and grasping in an occluded environment is
formulated in a reinforcement learning setting. The camera data of the working
scene is transformed into orthographic images and rotated $16$ times to cover
$360$ degrees rotation. The rotations are approximately $22.5$ degrees
increase for each step. The rotation allows the models to express and learn
grasp orientation and push direction. The model maximum $Q$-value is selected
with respect to the rotation, and then action is executed. If the grasp
threshold exceeds a certain $Q$-value (we set it to $1.8$), a grasp action to
the goal object is executed. Else, push of the goal object or any other
objects is performed [1].

The pushing is performed in a manner that increases the probability of
successfully grasping the goal object in the future. Similar to what we would
have in Generative Adversarial Network (GAN), the grasp net $\phi_{g}$ serves
as a discriminator, and the push net $\phi_{p}$ is considered a generator. It
manipulates the scene to utilize an increase in the grasp $Q$-value,
particularly the $Q_{g}$ (see Fig. 2). The discrete mask of the goal object is
used both as input in the model and as output to remove irrelevant pixel
$Q$-values.

Unlike to [1], the mask of all objects is not clear when it is used. We
decided during training not to restrict the model to have the $Q$-values only
on the relevant objects. It enhances the exploration space of the model due to
considering all $Q$-values and executing at the max value. Nevertheless, we
used the objects mask at the output during inference.

We realized that the discrete masking proposed by [1] is not useful as it
consistently results in invalid actions (see Fig. 3 top-row). Moreover, their
approach requires different masking during training and testing phases. This
impacts the scalability of the system. More advanced instance segmentation
methods (e.g., [20]) can be used to cope with this problem. However, since the
perception is out of the scope of this paper, we simplify the problem by using
RGB-colors segmentation as we know the color of the goal object in advance
(i.e., green color). The working discrete mask after applying it to the pixel
$Q$-values is illustrated in Fig. 3 (lower-row).

Figure 3: Objects pixels $Q$-value after applying masking: (top-row) masking
problems in approach proposed by [1]; (lower-row) Accurate masking using our
method.

###  III-B Models hierarchical framework

The hierarchical framework consists of two networks (grasp $\phi_{g}$ and push
$\phi_{p}$) sharing the same structure similar to [1]. However, the training
is different. We use three parallel 121-layer DenseNets [8], pre-trained on
ImageNet [21], to extract visual features. The first DenseNet tower takes the
RGB data of the scene, the second one takes the normalized depth image, and
the last one takes the goal object’s goal mask (discrete masking). The output
of these networks is then concatenated to form a single feature vector for the
given observation. The obtained representation is fed into a fully
convolutions network [22], which contains two Kernels of $1\times 1$ and
Rectified Linear Unit (ReLU) activation function, and batch normalization
[23]. Finally, it is bilinearly upsampled to generate a pixel-wise prediction
(i.e., the output image will be the same size as the input cropped image).

###  III-C Models training

We train the model in three sequential stages: the first stage is training the
goal-condition grasping, the second stage is dedicated to goal-conditioned
pushing, and lastly, alternating the training between the goal-condition grasp
and push. The grasp reward $R_{g}$ and push reward $R_{p}$, are formulated as
follow:

$R_{g}=\begin{cases}1,&\text{if grasp success}\\\ 0,&\text{if not}\end{cases}$
(1)

$R_{p}=\begin{cases}0.5,&Q_{g}^{improved}>0.1\text{ \& no scene change}\\\
-0.5,&\text{ no scene change}\\\ 0,&\text{otherwise}\end{cases}$ (2)

where $Q_{g}^{improved}$ is calculated as

$Q_{g}^{improved}=Q_{g}^{post-push}-Q_{g}^{pre-push}$ (3)

Furthermore, the amount of changes in the scene is calculated from the depth
map surrounding the goal object. Inspired by the Bellman equation of
reinforcement learning, we can formulate the state action function with
respect to the goal object as $\pi(s|a,g)$. We use the epsilon-greedy action
selection strategy $\epsilon(\pi(s|a,g))$. It gives the agent the privilege to
both explore and exploit options [24]:

$\nu(S_{t},g)=E[R_{[t+1]}+\rho\nu(S_{[t+1]})|S_{t}=S,g]$ (4)

Where $\rho$ represents a discount function, an overview of the training
process and the grasp success rate for the different stages of training are
depicted in Fig. 4. In the following sub-sections, more details about the
training are discussed.

####  III-C1 Goal-conditioned grasping

To train the model, we randomly generate sparse scenes by putting five objects
in the workspace (e.g., see Fig. 3). In this stage, we consider the execution
of each grasp as an episode and train the model in two phases. The first
phase, i.e., grasp agnostic, is done by relabeling the goal object to increase
the sample efficiency. In the second phase, i.e., grasp explore as the model
learned to grasp objects, we assume it learned sufficient estimation of the
orientations and grasp pose; hence, we omit the relabeling of the goal-object.
In both phases, the $\epsilon$-greedy strategy is used for balancing the
trade-off between exploring and exploiting [25]. Unlike what has been used by
[1] which measures the $Q$-values to determine when to stop training, we
measure the grasp success rate directly. Figure 4 shows that the training
could be stopped in both phases after reaching $1400$ grasp epochs since the
model grasp success is mostly stable. Even though the grasp success does not
increase much in the grasp explore phase, the task is more challenging as only
goal objects grasp considered a success, and the environment might not have a
feasible grasp due to yet untrained push model.

####  III-C2 Goal-conditioned pushing

In this round of training, we fix the grasp model and only train the push
model based on adversarial training, as mentioned earlier. Each episode has
been defined as up to five pushes and a grasp in the end. In other words, if
the $Q$-value of the grasp for the goal object exceeds a given threshed, the
robot immediately executes the grasp action, and the episode is terminated.
After a short period of 120 training grasp epochs, we observed that the goal-
oriented grasp success rate increases dramatically as the push model
manipulates the scene. Consequently, it increases the probability of the goal
object’s grasp (see Fig. 4 third-row). A positive reward is given to the push
model only if it increases the future probability of grasping the goal object.

####  III-C3 Goal-conditioned alternating

In the previous grasping stage, we trained the grasp model in sparse
environments (i.e., around five objects per scene) to reduce the effect of
occlusion. In addition, the model was trained when the push was not trained
yet, which leads to the distribution mismatch problem. The grasp net is
further trained after the push model is trained to overcome this issue. Models
are alternately trained in an environment with ten objects present in the
scene to improve the synergy between the grasp and push models.

Illustration of the training process is depicted in Fig. 4. Since the grasping
success result is a binary result (i.e., $1$ success and $0$ for a failure),
we use exponential smoothing with a factor of $0.9$ to show trends in the
data, i.e., learning progress of the grasp success. Also, the averaging mean
is used to show the data variation. For the goal-condition grasping and goal-
conditioned pushing, we use the mean rolling window of size $50$, and the rest
we set to mean of $7$.

Figure 4: Grasp success rate versus number of grasp epochs. The top two are
related to the grasping network $\phi_{g}$ training, the middle one for the
push network $\phi_{p}$ training. The last two are when the training is
alternated between the grasp net and push net.

##  IV Experiment

To evaluate the proposed approach, we carried out extensive sets of
experiments in the CoppeliaSim simulation environment [26]. Our experimental
setup consists of a UR5 robot arm and RGB-D Intel RealSense SR300, as shown in
Fig. 2. We used inverse kinematics (IK) solver for motion planning purposes
[27]. The networks are trained with Adam optimizer [28], the learning rate is
fixed on $10^{-4}$, and weight decay of $2^{-4}$ has been considered. We
trained the models using NVIDIA V100 Graphics Processing Unit for faster
training time. We compare our approach to the Xu et al.[1] since, to the best
of our knowledge, it is the current state-of-the-art approach in this domain.
We train and test both approaches on the same hardware and same evaluation
scenes to achieve fair compression. We use the following evaluation metrics
that have been used previously by [1, 3]:

  * •

Completion (C): The mean percentage completion over $n$ test runs. Completions
are successful and equal to $1$ in a test run; if the system does not exceed
in failing to grasp the goal object $n=5$ times, else it is $0$. The metric
measures the system’s ability to complete the task.

  * •

Grasp success (GS): The mean percentage of successful grasp over all grasp
attempts. This metric represents the accuracy of the model and its ability to
estimate goal object successful grasp.

  * •

Motion number (MN): The mean number of push actions per completion. It
reflects action efficiency.

Figure 5: Two examples of the generated scene are packed and pile scenarios for evaluation purposes. In all experiments, the goal object is defined by green color: (top-row) an example of packed scene (structured) and its heat-maps (pixel $Q$-values); (bottom-row) an example of an unstructured scene (pile) and its heat-maps (pixel $Q$-values). TABLE I: Simulation consistency results on structured scene Approach | C% | GS% | MN
---|---|---|---
Xu et al.[1] | $83\pm 3.75$ | $26.48\pm 2.16$ | $3.43\pm 0.31$
Ours | $\mathbf{98.98.0\pm 1.01}$ | $\mathbf{86.08\pm 3.32}$ | $\mathbf{1.12\pm 0.03}$

To evaluate the proposed approach, we conducted two sets of experiments. One
is a packed (structured) scenario, and the other is a pile (unstructured and
complicated) scenario. In the first round of experiments, we evaluate the
system’s consistency by performing the same experiment $100$ times in a packed
scenario. An example scene of this experiment together with its its heat-maps
(pixel $Q$-values) are depicted in Fig. 5 (top-row). The obtained results are
summarized in Table I. The low standard values indicate that our approach
produced a consistent sequence of actions to accomplish the task successfully.
Furthermore, we observed that since the model strives to gain a reward with
respect to the push reward (see Eq. 2), the $Q$-values were high on and around
the goal object. Therefore, shifting the goal object facilitates free space
around the target object to be grasped (see Fig. 6). By looking at
representative scenarios of Fig. 5, we hypothesize that approximately one push
to grasp the goal object is highly efficient.

Figure 6: Sequence of snapshots showing the robot performing decluttering by
pushing the objects surrounding the target object to make grasping the goal
object feasible: (left) initial scene; (middle) pushing surrounding objects;
(right) grasping the goal object.

By comparing the results, it is clear that our proposed approach outperforms
the state-of-the-art [1] across all metrics by a large margin. In particular,
our approach achieves a $100\%$ completion rate (C), which is $10\%$ better
than Xu et al. [1]. Also, in terms of Grasp Success (GS), our proposed
approach shows ($\approx 60\%$) better performance. The underlying reason for
using their approach is that the robot tends to perform unsuccessful grasps of
the target object as it lacks the proper estimation of the space around the
goal object. It led to several consecutive failures and the termination of the
experiment. Our model has better accuracy in making feasible grasp
estimations. It has learned to pick up the target object with a remarkably
higher grasp success rate ($\approx 60\%$) difference margin compared to Xu et
al.[1].

In the second set of tests, we assess the performance of the proposed approach
by generating random scenes using $10$, $15$, and $20$ objects. It is worth
mentioning that having more objects in the scene makes the task more
complicated. For these experiments, we drop different primitive objects in the
scene sequentially from a pre-defined point in the middle and above the
working scene to generate test scenes randomly. It results in more dense
scenarios, and the goal object gets covered by other objects in some instances
(see Fig. 7 left). We selected a goal object for each experiment out of the
possible primitive set of objects. An example of such scenes is depicted on
the bottom-row of Fig. 5. We generated a set of random scenes using $10$
objects to train the model. We evaluate the proposed approach by generating
$100$ different scenes for each level of complexity (i.e., $10$, $15$, $20$
objects per scene). The obtained results are presented in Table II.

Across the completion metric and grasp success rate, we see that our system
performed significantly better. However, regarding the Motion Number metric,
our system scored lower.

We observed that in scenes containing $10$ objects, one push action, on
average, was highly efficient to complete the task.

By increasing the level of complexity (adding more objects to the scene), we
expected the model to score lower across all metrics since the model has never
encountered a scene containing more than $10$ objects during the training
phase. When we tested the agent with $15$ objects per scene, we observed that
the agent performed better in terms of task completion and grasp success rate.
On a closer inspection, we found that even though there are more objects in
the scene, in some instances, the goal object was pushed away and became
isolated after other objects were spawned into the scene (see Fig. 7 middle).
Such situations made grasping the goal object easier. By increasing the number
of objects to $20$, such cases occurred rarely, and complicated scenes were
often generated as shown in (see Fig. 7 right). Therefore, as we expected, the
performance of both approaches decreased. By comparing obtained results, it is
visible that the proposed approach outperformed Xu et al.[1] regarding task
completion and grasp success rate. With $20$ objects in the scene, our system
completed the task with at least ${97.22\pm 1.95}$ which is approximately
$20\%$ (on average) better than the result of Xu et al.[1].

Another interesting point is that the proposed system performed well regarding
task completion and grasp success rate metrics. It showed a higher MN than Xu
et al. [1]. As this metric reflects action efficiency, one wants to aim for
lower values. Since Xu et al. [1] model makes the wrong estimation of a
feasible goal object grasp, we observed that it keeps attempting to grasp the
goal object even when it is not feasible at all. Attempting to grasp the goal
object results in moving objects in the scene, and this action is not counted
in the MN metric (as MN is defined as the number of push actions per
completion). Therefore, Xu et al. [1] model showed a better MN performance.
Nevertheless, not in a desirable manner, it is not systematical to avoid
pushing though it is essential, it loses the purpose of synergy value between
the push and the grasp action. Our grasp success rate might be on the low
spectrum, but this can be improved by increasing the grasp probability
threshold. However, that will cause a decrease in the push efficiency as the
model will make more pushes and improve clearing to the degree that it has
higher grasp probability confidence. Overall, our system showed consistent
performance despite the increased number of objects.

Figure 7: Typical scenes with different number of present objects, starting
from left to right $10$, $15$, and $20$ respectively.

We observed some limitations of using pushing as a pre-manipulating step for
grasping during visual observation of the test scenes. The push is not always
smooth. There are cases where the objects flip up due to friction. In other
cases, objects keep rolling. It happens to cylinder-shaped objects. There are
cases where the objects cannot be pushed at all due to surface friction with
the surface or weak friction between the robot gripper and the object.

TABLE II: Simulation results on 100 different complicated scenes

Approach | #objects | C% | GS% | MN
---|---|---|---|---
Xu et al.[1] | 10 | $71.56\pm 4.48$ | $20.35\pm 1.57$ | $\mathbf{0.64\pm 0.25}$
Xu et al.[1] | 15 | $71.0\pm 4.56$ | $22.72\pm 1.75$ | $\mathbf{1.1\pm 0.54}$
Xu et al.[1] | 20 | $77.89\pm 4.27$ | $21.59\pm 1.78$ | $\mathbf{1.13\pm 0.45}$
Ours | 10 | $\mathbf{98.97\pm 1.02}$ | $\mathbf{66.21\pm 3.91}$ | $1.02\pm 0.14$
Ours | 15 | $\mathbf{100\pm 0}$ | $\mathbf{74.60\pm 3.89}$ | $3.67\pm 0.98$
Ours | 20 | $\mathbf{97.22\pm 1.95}$ | $\mathbf{69.23\pm 4.62}$ | $6.09\pm 1.82$

##  V Conclusion and Future Work

This paper presents a self-supervised deep RL approach to learn grasping and
pushing policies for manipulating a goal object in highly cluttered
environments. In particular, our approach enables a robot to grasp a goal
object by pushing other objects away when needed. We present a self-supervised
deep RL approach, which learns grasping and pushing policies for manipulating
a goal object in highly cluttered environments. We trained and tested the
model in two challenging scenarios: packed objects and a pile of objects.
Objects in a packed scenario are tightly packed together and form a structured
scene, while in the pile scenario, we randomly dropped $20$ objects inside the
robot’s workspace to make a complicated and unstructured scene. Experimental
results showed that the proposed approach enabled the robot to interact
robustly with the environment and successfully manipulate the goal object in
both scenarios. Our system outperformed the selected state-of-the-art in terms
of task completion rate and grasp success in both scenarios. In the case of MN
metric, our approach outperformed Xu et al. in the packed scenario, while Xu
et al. worked better than ours in the pile scenario.

While evaluating the system, we realized that the pushing strategy has some
limitations for objects that are hard to push due to friction with the surface
under the object or causing objects to fall from their standing positions. In
real-life settings, this limitation can cause problems as the robot is often
confronted with various levels of friction. Undoubtedly, objects that have
three times the heights of the object that we used in the testing, shifting
them will cause the object to be laid on the surface. There is a certain
tolerance for causing chaos in the scene. Imagine having a case where a robot
is instructed to pick a goal object from a shelf. In that case, it is more
relevant to use some sequence grasping to reach the goal object in place of
pushing. In the continuation of this work, we would like to investigate such
scenarios by grasping and clearing other objects before grasping the goal
object.

## Acknowledgment

We thank the Center for Information Technology of the University of Groningen
for their support and for providing access to the Peregrine high performance
computing cluster.

## References

  * [1] K. Xu, H. Yu, Q. Lai, Y. Wang, and R. Xiong, “Efficient learning of goal-oriented push-grasping synergy in clutter,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 4, pp. 6337–6344, 2021.
  * [2] T. Kunz, U. Reiser, M. Stilman, and A. Verl, “Real-time path planning for a robot arm in changing environments,” in _2010 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, 2010, pp. 5906–5911.
  * [3] M. Fujita, Y. Domae, A. Noda, G. Garcia Ricardez, T. Nagatani, A. Zeng, S. Song, A. Rodriguez, A. Causo, I.-M. Chen _et al._ , “What are the important technologies for bin picking? technology analysis of robots in competitions based on a set of performance metrics,” _Advanced Robotics_ , vol. 34, no. 7-8, pp. 560–574, 2020.
  * [4] L. Berscheid, P. Meißner, and T. Kröger, “Robot learning of shifting objects for grasping in cluttered environments,” in _2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2019, pp. 612–618.
  * [5] A. Murali, A. Mousavian, C. Eppner, C. Paxton, and D. Fox, “6-dof grasping for target-driven object manipulation in clutter,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2020, pp. 6232–6238.
  * [6] B. Tang, M. Corsaro, G. Konidaris, S. Nikolaidis, and S. Tellex, “Learning collaborative pushing and grasping policies in dense clutter,” in _2021 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2021, pp. 6177–6184.
  * [7] Y. Yang, H. Liang, and C. Choi, “A deep learning approach to grasping the invisible,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 2232–2239, 2020.
  * [8] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 4700–4708.
  * [9] H. Duan, P. Wang, Y. Huang, G. Xu, W. Wei, and X. Shen, “Robotics dexterous grasping: The methods based on point cloud and deep learning,” _Frontiers in Neurorobotics_ , vol. 15, p. 73, 2021.
  * [10] A. Sahbani, S. El-Khoury, and P. Bidaud, “An overview of 3d object grasp synthesis algorithms,” _Robotics and Autonomous Systems_ , vol. 60, no. 3, pp. 326–336, 2012.
  * [11] D. Prattichizzo and J. C. Trinkle, “Grasping,” in _Springer handbook of robotics_. Springer, 2016, pp. 955–988.
  * [12] A. t. Pas and R. Platt, “Using geometry to detect grasps in 3d point clouds,” _arXiv preprint arXiv:1501.03100_ , 2015.
  * [13] A. ten Pas, M. Gualtieri, K. Saenko, and R. Platt, “Grasp pose detection in point clouds,” _The International Journal of Robotics Research_ , vol. 36, no. 13-14, pp. 1455–1473, 2017.
  * [14] S. Kumra, S. Joshi, and F. Sahin, “Antipodal robotic grasping using generative residual convolutional neural network,” in _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2020, pp. 9626–9633.
  * [15] H. Kasaei and M. Kasaei, “Mvgrasp: Real-time multi-view 3d object grasping in highly cluttered environments,” _arXiv preprint arXiv:2103.10997_ , 2021\.
  * [16] M. Q. Mohammed, K. L. Chung, and C. S. Chyi, “Review of deep reinforcement learning-based object grasping: Techniques, open challenges, and recommendations,” _IEEE Access_ , vol. 8, pp. 178 450–178 481, 2020.
  * [17] L. Y. Chang, G. J. Zeglin, and N. S. Pollard, “Preparatory object rotation as a human-inspired grasping strategy,” in _Humanoids 2008-8th IEEE-RAS International Conference on Humanoid Robots_. IEEE, 2008, pp. 527–534.
  * [18] D. Kappler, L. Chang, M. Przybylski, N. Pollard, T. Asfour, and R. Dillmann, “Representation of pre-grasp strategies for object manipulation,” in _2010 10th IEEE-RAS International Conference on Humanoid Robots_. IEEE, 2010, pp. 617–624.
  * [19] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, O. Pieter Abbeel, and W. Zaremba, “Hindsight experience replay,” _Advances in neural information processing systems_ , vol. 30, 2017\.
  * [20] Y. Xiang, C. Xie, A. Mousavian, and D. Fox, “Learning RGB-D feature embeddings for unseen object instance segmentation,” in _Conference on Robot Learning (CoRL)_ , 2020.
  * [21] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in _2009 IEEE conference on computer vision and pattern recognition_. Ieee, 2009, pp. 248–255.
  * [22] J. Long, E. Shelhamer, and T. Darrell, “Fully convolutional networks for semantic segmentation,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 3431–3440.
  * [23] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in _International conference on machine learning_. PMLR, 2015, pp. 448–456.
  * [24] M. Gimelfarb, S. Sanner, and C.-G. Lee, “$\\{$$\backslash$epsilon$\\}$-bmc: A bayesian ensemble approach to epsilon-greedy exploration in model-free reinforcement learning,” _arXiv preprint arXiv:2007.00869_ , 2020.
  * [25] C. D’Eramo, A. Cini, and M. Restelli, “Exploiting action-value uncertainty to drive exploration in reinforcement learning,” in _2019 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2019, pp. 1–8.
  * [26] E. Rohmer, S. P. N. Singh, and M. Freese, “V-rep: A versatile and scalable robot simulation framework,” in _2013 IEEE/RSJ International Conference on Intelligent Robots and Systems_ , 2013, pp. 1321–1326.
  * [27] R. Diankov, “Automated construction of robotic manipulation programs,” 2010.
  * [28] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.