# Computing Chaotic Time-Averages from a Small Number of Periodic Orbits

Joshua L. Pughe-Sanford<EMAIL_ADDRESS>Sam Quinn  Teodor Balabanski
Roman O. Grigoriev  School of Physics, Georgia Institute of Technology, 837
State St NW, Atlanta, GA 30332, USA

###### Abstract

Temporal averages in chaotic systems can often be approximated using
collections of periodic orbits—unstable time-periodic solutions of the
governing equations—embedded within the chaotic set. This connection is
formalized by periodic orbit theory, with a large number of such solutions
required to obtain a good approximation. Here, we describe an alternative,
data-driven approach that allows for an accurate approximation of temporal
averages even when the number of time-periodic solutions used is quite small.
Moreover, this approach yields convergence to temporal averages, in terms of
the number of solutions used, that far outperforms periodic orbit theory.

Deterministic chaos is pervasive in natural and engineered systems, with
turbulence in fluids [1], plasma [2] and nonlinear optics [3] representing
some well-known and practically important examples of high-dimensional chaotic
systems. Despite being governed by deterministic equations, chaotic dynamics
can only be forecast on short temporal intervals due to their sensitivity to
initial conditions. Hence, on long time scales, one is forced to resort to a
statistical description with a key objective to compute temporal averages of
various physical observables. In ergodic systems, temporal averages of $a({\bf
x})$,

$\displaystyle\left\langle
a\right\rangle_{t}\equiv\lim_{T\to\infty}\frac{1}{T}\int_{0}^{T}a(\mathbf{x}(t))\,dt,$
(1)

may be computed as state-space averages over an SRB measure [4, 5, 6],

$\left\langle a\right\rangle_{\mu}\equiv\int a(\mathbf{x})\,d\mu(\mathbf{x}),$
(2)

where $\rho({\bf x})=d\mu({\bf x})/d{\bf x}$ is the corresponding natural
density. However, constructing such a measure can be extremely challenging
even in low dimensions and is likely intractable in high dimensions.

A popular alternative to constructing a state-space measure is to evaluate a
temporal average using an infinite set of unstable orbits ${\bf x}_{p}(t)$,
e.g., unstable periodic orbits (UPOs) or their generalizations such as
relative periodic orbits (RPOs) [7] for systems with continuous symmetries,

$\displaystyle\left\langle
a\right\rangle_{t}=\sum_{p=1}^{\infty}w_{p}\left\langle a\right\rangle_{p},$
(3)

provided these orbits are dense in the chaotic set. The orbit averages

$\displaystyle\left\langle
a\right\rangle_{p}\equiv\frac{1}{T_{p}}\int_{0}^{T_{p}}a(\mathbf{x}_{p}(t))\
dt,$ (4)

are computed over one period of motion, $T_{p}$, and the weights $w_{p}$ can
be computed from properties of the orbits according to periodic orbit theory
(POT) [8, 9, 10] or its alternatives [11].

Practical applications of POT in higher dimensions face two fundamental
challenges: (i) it is hard to compute and store suitably comprehensive sets of
UPOs [12, 13] and (ii) a symbolic dynamics [14, Ch. 14] is required to
guarantee that the POT expansion converges quickly. Applications of POT in the
absence of a symbolic dynamics have seen little success [12, 13], as have
those using heuristic choices of weights [15, 16]. The most effective existing
methods for selecting weights are data-driven [17, 18], since they adapt most
freely to the choice of the orbits used. However, none of these studies has
examined how the accuracy is related to this choice.

In this letter, we derive an interpretable data-driven approach to determining
optimal weights, in the least-squares sense, for a finite number of UPOs. We
illustrate the method using the 1963-Lorenz system

$\displaystyle\dot{x}$ $\displaystyle=10(y-x)$ $\displaystyle\dot{y}$
$\displaystyle=x(28-z)-y$ (5) $\displaystyle\dot{z}$
$\displaystyle=xy-(8/3)z,$

which is ergodic and admits symbolic dynamics [19]—properties which make the
system amenable to POT analysis.

Let us consider a coarse-grained density

$\displaystyle\tilde{\rho}(\mathbf{y})$
$\displaystyle=\int\rho(\mathbf{x})G(\mathbf{y},\mathbf{x})d\mathbf{x},$ (6)

with a Gaussian kernel

$\displaystyle G(\mathbf{y},\mathbf{x})$
$\displaystyle=\left(2\pi\sigma^{2}\right)^{-d/2}\exp\left[{-|\mathbf{y}-\mathbf{x}|^{2}/2\sigma^{2}}\right],$
(7)

where $d$ is the dimension of the full state space; $d=3$ for Lorenz. Note
that $\tilde{\rho}({\bf y})$ represents not just a blurred version of
$\rho({\bf y})$ but also the temporal average of a Gaussian observable
$G(\mathbf{y},\mathbf{x})$ centered at a fixed point $\mathbf{y}$ in state
space. Similarly $\tilde{\rho}_{p}({\bf y})$, defined as the temporal average
(4) of $G(\mathbf{y},\mathbf{x})$ over the orbit ${\bf x}_{p}(t)$, also
represents a blurred version of the invariant density associated with that
orbit.

Let us now consider Equation 3 with the sum truncated to include only finitely
many periodic orbits. While it no longer holds for any choice of the weights
$w_{p}$, this truncation can provide very useful estimates of temporal
averages for a particular choice of $w_{p}$. Let us start by evaluating this
truncation for the Gaussian observable $G(\mathbf{y},\mathbf{x})$ which yields

$\displaystyle\tilde{\rho}(\mathbf{y})\approx\sum_{p=1}^{P}w_{p}\tilde{\rho}_{p}(\mathbf{y})$
(8)

for an arbitrary ${\bf y}$. Since both the coarse-grained natural density
$\tilde{\rho}(\mathbf{y})$ and orbital density $\tilde{\rho}_{p}(\mathbf{y})$
can be computed independently for any ${\bf y}$, Equation 8 provides one
constraint on the weights $w_{p}$ for each ${\bf y}$ it is evaluated at.
Evaluating this relation at a sufficiently large number of different locations
yields a system of equations which can be solved, in the least squares sense,
to find the weights.

To incorporate information from the all ${\bf y}$ in the chaotic set, we take
the inner product of Equation 8 with $\tilde{\rho}_{q}$ for $q=1,\dots,P$.
This results in the normal equations,

$\displaystyle\int\tilde{\rho}_{q}(\mathbf{y})\tilde{\rho}(\mathbf{y})\,d\mathbf{y}\approx\sum_{p=1}^{P}w_{p}\int\tilde{\rho}_{q}(\mathbf{y})\tilde{\rho}_{p}(\mathbf{y})\,d{\mathbf{y}}$
(9)

which are a linear system

$\displaystyle{\bf b}\approx A{\bf w}$ (10)

of $P$ equations for $P$ unknown weights. Note that we have chosen a uniform
inner product over the state space, which is indeed a choice, but one that
allows for $A$ and ${\bf b}$ to be cast in analytic form,

$\displaystyle b_{q}$
$\displaystyle=\left\langle\int_{0}^{T_{q}}\frac{d\tau}{T_{q}}\exp\left[-\frac{\|{\bf
x}(t)-{\bf
x}_{q}(\tau)\|^{2}}{2(\sqrt{2}\sigma)^{2}}\right]\right\rangle_{t},$ (11)
$\displaystyle A_{qp}$
$\displaystyle=\int_{0}^{T_{p}}\frac{dt}{T_{p}}\int_{0}^{T_{q}}\frac{d\tau}{T_{q}}\exp\left[-\frac{\|{\bf
x}_{p}(t)-{\bf x}_{q}(\tau)\|^{2}}{2(\sqrt{2}\sigma)^{2}}\right].$

The matrix $A$ describes the overlap between coarse-grained orbit densities in
the full state space, encoding which orbits are grouped and which are well-
separated. In turn, the components of ${\bf b}$ correspond to time averages
over coarse-grained orbit densities. Note that matrix $A$ involves integrals
over the periods of different orbits which are cheap to compute. The elements
of the vector ${\bf b}$ require a long chaotic trajectory that samples the
attractor well and explicit evaluation of the temporal averages.

(a)

(b)

(c)

(d)

(e)

(f)

Figure 1:  (a-c) Blurred natural density for $\sigma=1,\sqrt{10},10$ (from
left to right). (d-f) Orbital expansions using the shortest $P=21$ orbits at
the same choices of $\sigma$. The boundary of the Lorenz attractor is drawn in
solid red. White corresponds to zero density and darker colors corresponds to
larger densities.

The solution ${\bf w}$ of the system (10) defines a least-squares weighted
(LSW) approximation of temporal averages

$\displaystyle\langle\ \cdot\ \rangle_{P}=\sum_{p=1}^{P}w_{p}\langle\ \cdot\
\rangle_{p},$ (12)

for any observable from only finitely many orbits. In practice, $A$ should be
replaced with a finite-rank approximation to limit the effect of marginal
modes, $A{\bf w}_{0}\approx 0$, which correspond to linear combinations of
orbits whose densities look nearly identical under coarse-graining. We use a
singular value decomposition (SVD) to perform the truncation and only retain
singular values larger than $10^{-8}$ times the leading singular value.

Before discussing the results, let us consider choice of the free parameter
$\sigma$ which controls the coarse-graining. In the limit $\sigma\to\infty$,
all orbital densities $\tilde{\rho}_{p}(\mathbf{x})$ become identical constant
functions, consequently the weights become equal, $w_{p}=1/P$. Conversely, in
the limit $\sigma\to 0$, the orbital densities become non-overlapping,
consequently $A$ becomes diagonal. In both limits, the accuracy of LSW
deteriorates. Figure 1 compares the coarse-grained natural density alongside
its approximation (8) based on the weighted sum of orbital densities for
various choices of sigma. For $\sigma=10$, the approximation is accurate, but
coarse-graining smears out $\tilde{\rho}$ far beyond the boundaries of the
chaotic attractor. For $\sigma=1$, coarse-graining has little adverse effect
on the natural density, but the approximation in terms of the orbital
densities becomes inaccurate, with gaps between orbits opening up. This
suggests that there is an optimal – for a given collection of orbits – choice
of $\sigma$.

To quantify the accuracy of the approximation $\left\langle
a\right\rangle_{P}$ – with any choice of the weights $w_{p}$ – we define the
error as the difference between $\left\langle a\right\rangle_{P}$ and
$\left\langle a\right\rangle_{t}$

$\displaystyle E(a)=\frac{|\left\langle a\right\rangle_{P}-\left\langle
a\right\rangle_{t}|}{s_{a}}$ (13)

normalized by the standard deviation

$\displaystyle s_{a}=\sqrt{\langle a^{2}\rangle_{t}-\langle
a\rangle_{t}^{2}}.$ (14)

Standard deviation represents the magnitude of fluctuations of $a$ around its
mean value, so it is a natural scale against which to measure the precision of
estimates of $\langle a\rangle_{t}$.

Table 1 lists reference values of the temporal mean $\left\langle
a\right\rangle_{t}$ and standard deviation $s_{a}$ for three representative
observables: the state space speed $v=|\dot{\bf x}|$, the variable $z$, and
the distance $d=|{\bf x}-{\bf p}|$ to the fixed point
$\mathbf{p}=(6\sqrt{2},6\sqrt{2},27)$ represented graphically in Figure 2(a).
All reference values are computed with enough precision to ensure that $E(a)$
is accurate up to five significant digits.

$a$ | $v$ | $z$ | $d$
---|---|---|---
$\langle a\rangle_{t}$ | 95.69514(41) | 23.549969(58) | 17.629464(68)
$s_{a}$ | 57.070(1) | 8.6224(2) | 7.9661(2)
Table 1:  Temporal averages and standard deviations for three representative
observables. The method of estimating uncertainties is described in Appendix.

(a)

(b)

(c)

Figure 2:  Comparison of different weighting schemes for $v$ (left column),
$z$ (center column), and $d$ (right column). (a) The values of the
observables, minimized over $y$. The boundary of the Lorenz attractor is drawn
in solid red. (b) The approximation error for libraries of orbits that are
complete to length $n$. POT (black diamonds) and uniform weighting (green
diamonds) are plotted alongside LSW at $\sigma=1$ (blue), $\sigma=\sqrt{10}$
(red), and $\sigma=10$ (yellow). The dashed line corresponds to
$\Delta_{a}/s_{a}$. (c) A histogram of the error for POT vs. LSW at
$\sigma={10}$ over $5000$ randomly chosen sub-libraries of $P=21$ orbits.
Darker colors indicate larger bin counts. Black contours are drawn at even
values of $\log_{10}(E_{\text{LSW}}/E_{\text{POT}})$. The area shaded gray
indicates the region in which POT outperforms LSW.

To test the accuracy of the orbit expansion $\left\langle a\right\rangle_{P}$
for different choices of the weights $w_{p}$ we use the shortest (in period)
$1375$ orbits from those reported by Viswanath [20, 21]. This library includes
all prime orbits of symbolic length up to 12. Trajectories that describe an
orbit being repeated multiple times are not prime. Figure 2 compares the
accuracy of several different weighting schemes. Figure 2(b) considers the
error $E(a)$ of each method computed using the $P$ shortest (in time) orbits.
This is a favorable setting for POT, which predicts $E(a)$ to decrease as a
power law in $P$ [14, Sec. 23.1]. A description of how to compute averages
using POT is provided in the Appendix. We find that, uniform weighting
$w_{p}=1/P$ performs the worst, overall, although for $P\lesssim 10$, it gives
essentially the same accuracy as POT. As the number of orbits increases to
$P=1375$, the accuracy of the POT-based expansion improves by more than an
order of magnitude while, for uniform weighting, the accuracy shows
essentially no improvement when more than $10$ or so orbits are included.

Among the weighting schemes considered here, LSW yields an error which
decreases the fastest in $P$, with a rate that sensitively depends on the
coarse-graining parameter $\sigma$. For $P$-large, the error asymptotes to a
low, but finite, value which appears to be fairly insensitive to the choice of
$\sigma$. For $\sigma=\sqrt{10}$ and $\sigma=10$, the error bottoms out very
quickly, when only about 20 or so orbits are included. For $\sigma=1$, the
error decreases the slowest, requiring a few hundred orbits to bottom out.
Remarkably, for all three observables, with the proper choice of $\sigma$, LSW
outperforms POT by a wide margin. The accuracy of LSW is better than that
based on POT by an order of magnitude while requiring two orders of magnitude
fewer orbits.

In practice, symbolic dynamics are rarely available, and orbits are found by
converging initial guesses generated by a particular approach such as
recurrence analysis [22] or topological analysis [23, Sec 4.5]. The resulting
libraries of orbits will therefore be both incomplete and biased. To emulate
these biases, we constructed an ensemble of 5000 random choices of $P=21$
orbits and computed the error $E(a)$ for each member of the ensemble. This
represents a situation typical for high-dimensional chaotic systems such as
turbulent fluid flows, where only a few tens of orbits are available. Figure
2(c) compares the accuracy of LSW against POT, with the results binned into a
uniform mesh on a logarithmic scale, with darker colors representing higher
probabilities. For both LSW and POT, the error is found to vary by roughly an
order of magnitude, with the accuracy of LSW being two orders of magnitude
better than POT, for both $\left\langle v\right\rangle$ and $\left\langle
d\right\rangle$. For $\left\langle z\right\rangle$, POT has the highest
accuracy, but LSW still outperforms POT by more than an order of magnitude.
This observable is special: very few orbits come close to the origin, leading
to a natural bias in the orbit library.

Note that the accuracy of LSW is limited by the precision with which the
vector ${\bf b}$ is evaluated. Equation 10 involves a conventional time
average. In practice, computing a temporal average as an integral in time via
Equation 1 is computationally wasteful, since repeated evaluations of an
observable will be highly correlated over short time scales. Instead, we
leverage ergodicity and evaluate the observable using a sparsely sampled
chaotic trajectrory:

$\displaystyle\langle a\rangle_{M}\equiv\lim_{\Delta
t\to\infty}\frac{1}{M}\sum_{m=1}^{M}a({\bf x}(m\Delta t)).$ (15)

Thus defined, $\langle a\rangle_{M}$ may be considered a mean over $M$
independent samples from the natural density. We take $\Delta t=2$ to be
slightly larger than the auto-correlation time of the system and $M=10^{6}$.
The error in computing a time average with $\langle\ \cdot\ \rangle_{M}$ is

$\displaystyle\Delta_{a}\equiv|\left\langle a\right\rangle_{M}-\left\langle
a\right\rangle_{t}|$ $\displaystyle\sim M^{-\frac{1}{2}}.$ (16)

and a characteristic lower bound for the order of magnitude of $E(a)$ is given
by $\Delta_{a}/s_{a}$, regardless of the library size, as illustrated in
Figure 2(b). For all observables considered here, LSW appears to converge to
an accuracy slightly better than $\Delta_{a}/s_{a}$. This bound can be made
arbitrarily low by computing ${\bf b}$ with sufficient precision.

An unusual property of LSW is that it often returns negative weights.
Approximation methods based on finding the invariant distribution for a Markov
graph of orbits [18, 24] always yield weights that represent probabilities and
hence are convex, i.e., both non-negative, $w_{p}\geq 0$, and normalized,
$\sum_{p}w_{p}=1$. POT appears to always yield convex weights. In other
methods, convexity is simply assumed [17]. While the normalization follows
naturally from the desired property $\langle 1\rangle_{P}=\sum_{p}w_{p}=1$,
there is no fundamental reason why the weights have to be non-negative. In
fact, the assumption of non-negativity is neither grounded nor sensible for
small orbit libraries. For example, the shortest 12 orbits in the Lorenz
systems all overestimate the average state space speed, $\langle
v\rangle_{p}>\langle v\rangle$. Hence, the true average lies outside the
convex hull of the orbit averages. As a result, no choice of convex weights
can predict the true average to within better than about 3% for this library.
Meanwhile, LSW generally yields non-convex weights yielding a prediction
accurate to within 0.005%. LSW appears to return normalized weights, e.g., it
was found that $|1-\sum_{p}w_{p}|=O(10^{-3})$ in the previous example.

Non-convex sets of weights can be interpreted as a quasiprobability
distribution over the orbits, meaning that the neighborhoods of orbits are not
disjoint [25, §16][26]. This interpretation is empirically supported by the
fact that chaotic trajectories frequently approach multiple periodic orbits
simultaneously [27, 28]. It has been noted [29] that quasiprobability
distributions can appear “due to the mathematical segregation of systems or
states which physically only exist in combination.” Here, orbit measures
$\left\langle a\right\rangle_{p}$ have been mathematically segregated even
though only $\left\langle a\right\rangle_{t}$ is physically relevant due to
noise [30].

Figure 3:  The size of the largest gap, $\sigma_{h}$, between orbits in the
Lorenz system for the $P$ shortest orbits.

As Figure 2 shows, the worst accuracy of LSW is achieved at small values of
$\sigma$, for which the orbit neighborhoods represented by $\tilde{\rho}_{p}$
tend to become disjoint. Overlap between neighborhoods allows LSW to explore
non-convex combinations of weights. The size of the largest gap between
neighboring orbits can be computed as

$\displaystyle\sigma_{\text{h}}=\max_{p}\min_{q\neq
p}\max_{\tau}\min_{t}\|\mathbf{x}_{p}(\tau)-\mathbf{x}_{q}(t)\|.$ (17)

According to Figure 3, this gap decreases from $\sigma_{\text{h}}\approx 4$ to
$\sigma_{\text{h}}\approx 1$ as more orbits are included. The overlap between
orbital densities allows for non-zero off-diagonal entries in the matrix $A$.
These entries facilitate “hybridization” in the right eigenvectors of $A$,
directly analogous to hybridization in Molecular Orbital Theory [31, Ch. 9].
For $\sigma$ much smaller than the shortest inter-orbit distance, $A$ becomes
diagonal and the weights all become positive. Hence, it is the relatively high
value of $\sigma_{h}$ at low $P$ which explains why the accuracy of LSW is the
poorest at the smallest $\sigma$. For $\sigma\lesssim\sigma_{h}$ the orbit
neighborhoods become disjoint and the weights become convex and hence sub-
optimal.

To summarize, we have shown that it is possible to construct an accurate
approximation of temporal averages of generic smooth observables as a weighted
sum over unstable periodic orbits of a chaotic system. The accuracy of the
approximation far surpasses that of periodic orbit theory which has long been
considered the golden standard. It is important to emphasize that, using only
$O(10)$ orbits which need to be neither particularly long nor chosen in any
particular way, a very accurate and robust approximation can be constructed.

Interestingly, the approach presented here is not limited to unstable periodic
orbits or even invariant sets. It should work just as well for an arbitrary
collection of open trajectories or other subsets of the chaotic set.
Furthermore, unlike POT, this approach does not require the system to be
uniformly hyperbolic. This is particularly important for high-dimensional
chaotic systems, such as turbulent fluid flows, which are not uniformly
hyperbolic [27] and for which finding unstable periodic orbits tends to be
both practically challenging and expensive.

We gratefully acknowledge financial support from the National Science
Foundation under Grant No. 2032657.

Appendix: Periodic Orbit Theory (POT).—The theoretical foundations of POT are
summarized in Ref. [10], and the monograph [14] provides a detailed reference
text on the subject. The POT calculation of orbit averages for the Lorenz
system has been carried out in Ref. [8]. Here, we provide a short summary of
the essential steps. The following algorithm works for systems with symbolic
dynamics, however, averages can be computed from POT without one [14, §23.7].

Given a library of periodic orbits which contains all orbits up to symbolic
length $n$, the temporal average is approximated as

$\displaystyle\langle
a\rangle_{t}\approx-\frac{\partial_{\beta}F_{n}(s,\beta)}{\partial_{s}F_{n}(s,\beta)}\Bigg{|}_{\begin{subarray}{c}\
s=s_{0},\\\ \beta=0\ \end{subarray}},$ (18)

where $F_{n}$ approximates a spectral determinant[14, Ch. 22] and $s_{0}$ is
the real zero of $F_{n}$, i.e., $F_{n}(s_{0},0)=0$. Equality is reached in
Equation 18 in the limit $n\to\infty$. While POT does not define the weights
$w_{p}$ explicitly, they may be computed from Equation 18:

$\displaystyle w_{p}=-\frac{\partial_{\langle
a\rangle_{p}}\partial_{\beta}F_{n}(s,\beta)}{\partial_{s}F_{n}(s,\beta)}\Bigg{|}_{\begin{subarray}{c}\
s=s_{0},\\\ \beta=0\ \end{subarray}}.$ (19)

The spectral determinant can be computed using the trace coefficients [14,
§23.2.2]

$\displaystyle
C_{j}=-\sum_{n_{p}r=j}\frac{1}{r}\frac{e^{-rT_{p}\left(s-\beta\langle
a\rangle_{p}\right)}}{\lvert\det(1-(M_{p})^{r})\rvert},$ (20)

where the sum is over all prime orbits $p$ such that $j=rn_{p}$, where
$r=1,2,\dots$ and $n_{p}$ is the symbol length of orbit $p$. Notice that
$C_{j}$ sums over all periodic orbits (not just prime) of symbol length $j$.
$M_{p}$ is the transverse monodromy matrix [14, §21.2.2] of orbit $p$ over one
period. For the Lorenz system, $\det(1-(M_{p})^{r})\approx
1-\lvert\Lambda_{p}\rvert^{r}$, where $\Lambda_{p}$ is the largest Floquet
multiplier of the orbit over one period.

The spectral determinant is then

$\displaystyle F_{n}(s,\beta)$ $\displaystyle=1+\sum_{j=1}^{n}Q_{j}$ (21)

with the help of a recurrence relation

$\displaystyle Q_{j}$
$\displaystyle=C_{j}+\sum_{i=1}^{j-1}\frac{j-i}{j}C_{j-i}Q_{i},$ (22)

where $Q_{1}=C_{1}$. The partial derivatives of $F_{n}$ are computed via the
chain rule with

$\displaystyle\partial_{s}C_{j}\Big{|}_{\beta=0}$
$\displaystyle={\sum_{n_{p}r=j}}\frac{e^{-rT_{p}s}}{\lvert
1-|\Lambda_{p}|^{r}\rvert}T_{p},$ (23)
$\displaystyle\partial_{\beta}C_{j}\Big{|}_{\beta=0}$
$\displaystyle=-{\sum_{n_{p}r=j}}\frac{e^{-rT_{p}s}}{\lvert
1-|\Lambda_{p}|^{r}\rvert}T_{p}\langle a\rangle_{p},$ (24)
$\displaystyle\partial_{\langle
a\rangle_{p}}\partial_{\beta}C_{j}\Big{|}_{\beta=0}$
$\displaystyle=-\sum_{n_{p}r=j}\frac{e^{-rT_{p}s}}{\lvert
1-|\Lambda_{p}|^{r}\rvert}T_{p},$ (25) $\displaystyle\partial_{\langle
a\rangle_{p}}C_{j}\Big{|}_{\beta=0}$ $\displaystyle=0.$ (26)

In Equation 25, note that $p$ is fixed and the sum runs only over $r$. The
root $s_{0}$ is found using Newton’s method by iterating the relation

$\displaystyle s_{0}=s_{0}-\frac{F_{n}(s_{0},0)}{\partial_{s}F_{n}(s_{0},0)}$
(27)

until $|F_{n}(s_{0},0)|<\epsilon$ using $s_{0}=0$ as the initial guess. We use
$\epsilon=10^{-14}$.

A code that computes POT weights using the algorithm described here is
provided in a code repository [32].

Appendix: Computing reference averages.—The reference values of $\left\langle
a\right\rangle_{t}$ and $s_{a}$ listed in Table 1 were computed using sparse
samplings of chaotic trajectories, similarly to Equation 15. To estimate the
uncertainty, partial averages were computed over $L=12,762$ trajectories:

$\displaystyle\bar{a}_{l}=\frac{1}{M}\sum_{m=1}^{M}a\left(\mathbf{x}^{(l)}(m\Delta
t)\right)$ (28)

for $M=10^{6}$ and $\Delta t=5$. The reference values and their uncertainties
were then estimated as:

$\displaystyle\langle
a\rangle_{t}=\left(\frac{1}{L}\sum_{l=1}^{L}\bar{a}_{l}\right)\pm\frac{\Sigma_{l}[a]}{\sqrt{L}}$
(29) $\displaystyle s_{a}=\sqrt{\langle a^{2}\rangle_{t}-\langle
a\rangle_{t}^{2}}\pm\frac{\sqrt{(\Sigma_{l}[a^{2}])^{2}+(2\langle
a\rangle_{t}\Sigma_{l}[a])^{2}}}{2s_{a}\sqrt{L}}$ (30)
$\displaystyle(\Sigma_{l}[a])^{2}={\frac{1}{L-1}\sum_{l=1}^{L}(\bar{a}_{l}-\langle
a\rangle_{t})^{2}},$ (31)

We intentionally computed these reference values to a much greater precision
than that of the $\mathbf{b}$ vector (i.e. $\Delta_{a}$ from Equation 16),
ensuring that uncertainty in the reference value of $\langle a\rangle_{t}$
does not dominate the calculation of $E(a)$.

## References

  * Christensen  _et al._ [2021] A. Christensen, K. Tsiaras, J. Murawski, Y. Hatzonikolakis, J. She, M. St. John, U. Lips, and R. Brouwer, Front. Mar. Sci. 8, 744208 (2021).
  * Yamada  _et al._ [2008] T. Yamada, S.-I. Itoh, T. Maruta, N. Kasuya, Y. Nagashima, S. Shinohara, K. Terasaka, M. Yagi, S. Inagaki, Y. Kawai, A. Fujisawa, and K. Itoh, Nature Physics 4, 721 (2008).
  * Picozzi  _et al._ [2014] A. Picozzi, J. Garnier, T. Hansson, P. Suret, S. Randoux, G. Millot, and D. Christodoulides, Phys. Reports 542, 1 (2014).
  * Sinai [1972] Y. G. Sinai, Russian Math. Surveys 27, 21 (1972).
  * Ruelle [1976] D. Ruelle, Am. J. Math. 98, 619 (1976).
  * Bowen  _et al._ [2008] R. E. Bowen, J.-R. Chazottes, and D. Ruelle, Ergodic Theory of Axiom a Diffeomorphisms, in  _Equilibrium States and the Ergodic Theory of Anosov Diffeomorphisms_, Vol. 470 (Springer Berlin Heidelberg, 2008) pp. 61–73.
  * Budanur  _et al._ [2015] N. B. Budanur, D. Borrero-Echeverry, and P. Cvitanović, Chaos 25, 073112 (2015).
  * Eckhardt and Ott [1994] B. Eckhardt and G. Ott, Z. Physik B 93, 259 (1994).
  * Cvitanović [1995] P. Cvitanović, Physica D: Nonlinear Phenomena 83, 109 (1995).
  * Cvitanović [1999] P. Cvitanović, in  _Supersymmetry and Trace Formulae_, Vol. 370, edited by I. V. Lerner, J. P. Keating, and D. E. Khmelnitskii (Springer US, 1999) pp. 85–102.
  * Lai  _et al._ [1997] Y.-C. Lai, Y. Nagai, and C. Grebogi, Phys. Rev. Lett. 79, 649 (1997).
  * Chandler and Kerswell [2013] G. J. Chandler and R. R. Kerswell, J. Fluid Mech. 722, 554 (2013).
  * Lucas and Kerswell [2015] D. Lucas and R. R. Kerswell, Phys. Fluids 27, 045106 (2015).
  * Cvitanović  _et al._ [2016] P. Cvitanović, R. Artuso, G. Tanner, G. Vattay, and R. Manieri,  _Chaos: Classical and Quantum_ (Niels Bohr Inst., 2016).
  * Kazantsev [1998] E. Kazantsev, Nonlin. Processes Geophys. 5, 193 (1998).
  * Zoldi and Greenside [1998] S. Zoldi and H. Greenside, Phys. Rev. E 57, R2511 (1998).
  * Page  _et al._ [2022] J. Page, P. Norgaard, M. P. Brenner, and R. R. Kerswell (2022), arXiv:2212.11886.
  * Yalnız  _et al._ [2021] G. Yalnız, B. Hof, and N. B. Budanur, Phys. Rev. Lett. 126, 244502 (2021).
  * Eckmann and Ruelle [1985] J. P. Eckmann and D. Ruelle, Rev. Mod. Phys. 57, 617 (1985).
  * [20] http://www.math.lsa.umich.edu/~divakar/lorenz/index.html.
  * Viswanath [2003] D. Viswanath, Nonlinearity 16, 1035 (2003).
  * Cvitanović and Gibson [2010] P. Cvitanović and J. F. Gibson, Phys. Scr. T142, 014007 (2010).
  * Gudorf [2020] M. N. Gudorf,  _Spatiotemporal Tiling of the Kuramoto-Sivashinsky Equation_ , Ph.D. thesis (2020).
  * Schneider  _et al._ [2007] T. M. Schneider, B. Eckhardt, and J. Vollmer, Phys. Rev. E 75, 066313 (2007).
  * Moyal [1949] J. E. Moyal, Math. Proc. Camb. Phil. Soc. 45, 99 (1949).
  * Wigner [1932] E. Wigner, Phys. Rev. 40, 749 (1932).
  * Krygier  _et al._ [2021] M. C. Krygier, J. L. Pughe-Sanford, and R. O. Grigoriev, J. Fluid Mech. 923, A7 (2021).
  * Crowley  _et al._ [2023] C. J. Crowley, J. L. Pughe-Sanford, W. Toler, R. O. Grigoriev, and M. F. Schatz, Phil. Trans. R. Soc. A. 381, 20220137 (2023).
  * Bartlett [1945] M. S. Bartlett, Math. Proc. Camb. Phil. Soc. 41, 71 (1945).
  * Cowieson and Young [2005] W. Cowieson and L.-S. Young, Ergod. Th. Dynam. Sys. 25, 1115 (2005).
  * McQuarrie and Simon [1997] D. A. McQuarrie and J. D. Simon,  _Physical Chemistry: A Molecular Approach_ (University Science Books, 1997).
  * [32] Certain codes have been made public at https://github.com/cdggt/lsw.


# OVGNet: A Unified Visual-Linguistic Framework for Open-Vocabulary Robotic
Grasping

Meng Li1,2, Qi Zhao1, Shuchang Lyu1, Chunlei Wang1, Yujing Ma3, Guangliang
Cheng4 and Chenguang Yang4 Corresponding author: Guangliang Cheng*This work
was supported by the National Natural Science Foundation of China (grant
number 62072021)1Meng Li, Qi Zhao, Shuchang Lyu and Chunlei Wang are with
Department of Electronic and Information Engineering, Beihang University.
{limenglm, zhaoqi, lyushuchang<EMAIL_ADDRESS>Li is also with
Peng Cheng Laboratory, ShenZhen, China3Yujing Ma is with SenseTime, Beijing,
China<EMAIL_ADDRESS>Cheng and Chenguang Yang are with the
Department of Computer Science, at the University of Liverpool.
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Recognizing and grasping novel-category objects remains a crucial yet
challenging problem in real-world robotic applications. Despite its
significance, limited research has been conducted in this specific domain. To
address this, we seamlessly propose a novel framework that integrates open-
vocabulary learning into the domain of robotic grasping, empowering robots
with the capability to adeptly handle novel objects. Our contributions are
threefold. Firstly, we present a large-scale benchmark dataset specifically
tailored for evaluating the performance of open-vocabulary grasping tasks.
Secondly, we propose a unified visual-linguistic framework that serves as a
guide for robots in successfully grasping both base and novel objects.
Thirdly, we introduce two alignment modules designed to enhance visual-
linguistic perception in the robotic grasping process. Extensive experiments
validate the efficacy and utility of our approach. Notably, our framework
achieves an average accuracy of 71.2% and 64.4% on base and novel categories
in our new dataset, respectively. Our code and dataset are available at
https://github.com/cv516Buaa/OVGNet.

##  I Introduction

The increasing demand for intelligent robots in diverse real-world
applications, including warehousing logistics [1], domestic service [2], and
smart agriculture [3], exposes limitations in current robotic perception and
grasping methods. Existing methods are limited to recognizing predefined
(seen) categories, hindering their ability to handle novel-category (unseen)
objects in real-world scenarios where listing all possible categories is
impractical. Hence, it is highly necessary to introduce an algorithm for
detecting and grasping novel-category objects in the field of robotics.

Recent advancements in the field of visual-linguistic robotic grasping have
facilitated human-guided interaction with robots through language. Various
methods [4, 5, 6] have been proposed to develop deep learning-based frameworks
that address the challenge of visual-linguistic feature representation. These
frameworks empower robots to execute object grasping tasks based on language
guidance. However, there still exists two main weaknesses in visual-linguistic
robotic grasping task. First, previous methods overlook the representation
potential on localizing and grasping novel (unseen) objects. The benchmark
datasets on this task is also lacking. Second, with the development of open-
vocabulary learning (OVL), some notable methods [7, 8] provide an insight on
detecting novel objects. However, a unified framework, which integrates OVL
mechanism into robotic grasping still remains unexplored.

Figure 1: The diagram of open-vocabulary grasping. Objects and fonts in red
and blue respectively indicate the base and novel categories.

To address the aforementioned weaknesses, we introduce an innovative benchmark
dataset named OVGrasping specifically tailored for the open-vocabulary
grasping (OVG) task. Comprising 63,385 instances across 117 categories, the
dataset is strategically partitioned into base and novel categories. The
primary objective is to guide robotic systems in recognizing and effectively
grasping novel objects. Additionally, we also introduce the Open-Vocabulary
Learning (OVL) concept into the realm of robotic grasping, presenting a
comprehensive framework termed as OVGNet for the OVG task. The framework focus
on localizing and grasping novel objects by leveraging the learned knowledge
from open-vocabulary foundation models and fine-tuning knowledge with only
base categories. As shown in Fig. 1, our framework comprises a visual-
linguistic perception system for locating target objects based on language
references and a grasping system for acquiring them using specific grasping
poses. To improve the perceptual capability for novel objects, we introduce an
image guided language attention module (IGLA) and a language guided image
attention module (LGIA) within the visual-linguistic system. These modules
ensure alignment between visual and linguistic features, thereby promoting
generalization from base-category to novel-category objects. Extensive
experiments conducted on the OVGrasping dataset validate the efficacy of
OVGNet. Visualization and analysis demonstrate the interpretability of our
method.

Comparing to previous methods (Fig. 1). The VLG task focuses on locating and
grasping base-category objects, which may fail when encountering with novel-
category objects. In contrast, the OVG task aims to generalize the locating
and grasping ability, consequently facilitating open-vocabulary grasping for
novel-category objects. In summary, the main contributions are listed as
follows:

  * •

We are the first to introduce a large-scale dataset for benchmarking the OVG
task, providing a valuable resource for guiding robots in grasping both base
and novel categories.

  * •

We propose a novel OVG framework that integrates the OVL into robotic
grasping, which leverages both prior knowledge and fine-tuning techniques,
significantly enhancing the robot’s ability to grasp novel-category objects.

  * •

We introduce two alignment modules to enhance feature consistency, thereby
improving the generalization capabilities of our visual-linguistic perception
system.

##  II Related Works

###  II-A Grasp Pose Detection

Grasping pose detection mainly falls into two categories. 2D grasping [9]
detects the grasping points and orientations within a two-dimensional plane.
In contrast, 6-DOF grasping [10, 11, 12, 13] is the most extensively method.
GraspNet [14] has constructed a dataset containing one billion grasp poses and
provides a baseline for detecting grasp poses. Furthermore, Lu et al. [15]
propose the FGC-Grasp which integrates force-closure with measurements of
flatness, gravity, and collision. Xu et al. [16] propose a single-stage
grasping network to execute instance-level grasping. Wang et al. [17] propose
a visual grasping framework based transformer, which captures both local and
global features.

###  II-B Open-Vocabulary Visual Grounding

Open-Vocabulary learning [18, 19] aims to expand the range of vocabulary and
comprehension. Previous works mainly involved two aspects: Open-Vocabulary
Detection (OV-D)[20, 21, 22, 23] and Open-Vocabulary Segmentation (OV-S) [24,
25, 26, 27]. OV-D methods aim to detect the objects without the predefined
category. These approaches can accurately detect various objects instructed by
language. Liu et al. [8] propose an OV-D framework based on DINO [28], which
outputs the score between phrases and objects. Unlike the OV-D and OV-S
methods, Open-Vocabulary Visual Grounding (OV-VG) aims to detect and locate
the novel-category objects referred by natural language, establishing a more
extended correspondence between language and vision. Wang et al. [29] are the
first to introduce the concept of OV-VG and benchmark various models on the
OV-VG task.

###  II-C Vision-Language Guided Robotic Grasping

With the development of the large language models [30, 31] and multi-modal
models [8, 7], vision-language guided robotic grasping has become a cutting-
edge research topic. Xu et al. [5] access the probability of each grasping
pose by jointly model action, language, and vision. Tang et al. [32] design a
task-oriented grasp method to address the challenge of task grounding in
addition to object grounding. Sun et al [33] employ CLIP [7] for object
localization and 6-DoF pose estimation, enhancing performance with a novel
dynamic mask strategy for feature fusion. Lu et al. [4] propose a VG dataset
called RoboRefIt to train visual-linguistic model. Despite the successful
integration of visual-linguistic models into the robotic grasping task,
allowing for effective manipulation of target objects within the base
category, the challenge of grasping novel objects persists.

##  III Proposed Method

###  III-A OVGrasping Dataset

To empower robots with proficiency in locating and grasping both base and
novel objects, we construct a new large-scale language-guided grasping dataset
termed as OVGrasping, by reorganizing the RoboRefIt [4], GraspNet [14]
dataset, and incorporating newly collected data.

####  III-A1 Dataset Descriptions

The OVGrapsing dataset comprises 117 categories and 63,385 instances.
Instances are sourced from three distinct origins: RoboRefIt, contributing a
total of 66 classes; GraspNet, enriching the dataset with 34 classes; and a
simulated environment, encompassing 17 classes. Notably, we employ pybullet to
craft the simulation environment, incorporating 3D models made available by
OmniObject3D [34]. Subsequently, we employ a camera positioned orthogonally to
the grasping plane to capture images with dimensions of 480x640 pixels.

Specifically, The dataset is divided into two categories: the base category
consists of 68 classes with 51,857 instances, and the novel category comprises
49 classes totaling 11,528 instances. The base category is used for training
our framework, whereas the novel category is only utilized to evaluate the
performance of open-vocabulary. Additionally, to evaluate the performance
within the base category, we randomly extract 10$\%$ from the base category as
the test set with 4,830 instances. Totally, the OVGrasping dataset encompasses
various categories of objects and distinctly divide them into base and novel
categories.

####  III-A2 Data Annotation and Samples

During the annotation process, to ensure data accuracy and reliability, we
hired a team of six annotation experts and three quality inspectors to check
the consistency and accuracy of the annotation data. In the dataset, we
provide a comprehensive description of each object, guided by the contextual
semantics of the image, including color, shape, and location. Considering that
the images provided by OVGrasping potentially includes similar or identical
objects, we design a new annotation format for guiding the network to better
detect and locate the target object, which is inspired by RoboRefIt.
Specifically, as shown in equation 1, we introduce subject-object relationship
to determine the relative position, thereby distinguishing the target object
from two identical objects.

$\begin{split}{Describtion}&={<Template>}+{<Target\;Object>}\\\
&+{<Position>}+{<Relative\;Object>}.\end{split}$ (1)

During the annotation process, we randomly generate template for each target
object. Annotators are tasked with providing input solely for the relative
location and the corresponding object. Examples are shown in Fig. 2. We
provide a detailed description to locate the target object from two identical
objects. In essence, the OVGrasping dataset not only includes a plenty of base
and novel classes, but also introduces the subject-object relationship to
describe the relative position, providing guidance for robots to develop
proficiency in grasping the target from identical objects.

Figure 2: Samples of OVGrasping dataset. Red boxes indicate the target
objects, and green boxes denote the relative objects.

####  III-A3 Dataset Comparison

The OVGrasping dataset aims to facilitate robotic perception and grasping of
base and novel objects. To fully analyze the main features of the OVGrsaping,
we conduct a thorough comparison with four extensive datasets, among which the
RoboRefIt is the most relevant to robotic grasping. As indicated in Table I,
OVGrasping exhibits a higher average word count in comparison to RoboRefIt,
accompanied by more detailed descriptions. Furthermore, compared to Sun-Spot
and RoboRefIt, the OVGrasping demonstrates a notable increase in both the
number of object classes and instances. Lastly, and most importantly, the
OVGrasping is an open-vocabulary (OV) dataset oriented toward robotic
grasping.

To summarize, comparing to previous datasets, the OVGrasping has the following
advantages: 1) It introduces open-vocabulary into robot grasping, clearly
divides base and novel categories. 2) It uses detailed language to describe
the target object, including subject-object relationship, shape, color. 3) It
reorganizes data from multiple datasets, including both real and virtual. 4)
It contains diverse and abundant instances, which can guide robots to perceive
novel objects.

TABLE I: Comparison of primary characteristics between previous datasets and
OVGrasping dataset.

Datasets | Data | Number | Avg. | OV | Object | Data
---|---|---|---|---|---|---
Format | Instances | Word | Classes | Source
OVGrasping | RGB | 63,385 | 10.0 | ✓ | 117 | Real, Virtual
RoboRefIt | RGBD | 50,758 | 9.5 | $\times$ | 66 | Real
ScanRefer | 3D | 51,583 | 20.0 | $\times$ | 250 | Real
SUNRefer | 3D | 38,495 | 16.3 | $\times$ | - | Real
Sun-Spot | RGBD | 7,990 | 14.0 | $\times$ | 38 | Real

###  III-B The Architecture: OVGNet

Figure 3: The overview of OVGNet. The visual-linguistic perception system
locates the target object referred by natural language. The grasping system
generates grasping pose for the target object. MHA stands for multi head
attention. Feature constrain represents the scaling of image feature using the
constraint score. LGQS represents the language guided query selection module.

To achieve open-vocabulary grasping, as shown in Fig. 3, we design a unified
visual-linguistic grasping framework inspired by GroundingDino [8] and
GraspNet, which comprises two integral components: a visual-linguistic
perception system and a grasping system.

####  III-B1 Visual-Linguistic Perception System

The system is designed to locate the base and novel target object based on
language input. Given an image ${i}$ and a text description ${l}$, following
the GroundingDINO, we extract image features and text features using the image
backbone ${f_{i}(\cdot)}$, text backbone ${f_{l}(\cdot)}$, and cross-modality
encoder ${f_{ce}(\cdot)}$, which parameters remain fixed throughout the
training process.

${v_{i}},{v_{l}}={f_{ce}({f_{i}}(i),{f_{l}(l)})}$ (2)

Considering that the OVGrasping dataset includes complex linguistic structures
such as subject-object relationship. To align the consistency between vision-
language features, we design an Image Guided Language Attention (IGLA) to
enhance the text feature, which is based on multi-head attention (MHA). Then,
we use image feature ${v_{i}}$ as query, and text feature ${v_{l}}$ as key and
value. Through multi-head attention, image features are aligned with text
features, generating visual-linguistic feature ${v_{il}}$. Subsequently, we
scale the vision-language features by $\alpha$, and employ a skip connection
to add them to the original text feature ${v_{l}}$, yielding the image-guided
text feature ${v^{{}^{\prime}}_{l}}$. The formulation for IGLA is as follows:

${v^{{}^{\prime}}_{l}}=\alpha{v_{il}}+{v_{l}},\quad{v_{il}}=softmax(\frac{{v_{i}}{v^{T}_{l}}}{\sqrt{{d_{{v_{l}}}}}}){v_{l}}$
(3)

Empirically, we set $\alpha$ to 0.5, where ${d_{{v_{l}}}}$ represents the
dimension of ${v_{l}}$. Furthermore, to enhance visual-linguistic
comprehension and heighten attention to regions described in the language
within the image, we introduce a Language Guided Image Attention (LGIA) to
refine the image feature ${v_{i}}$. Specifically, we employ a fully connected
layer and L2 regularization to map ${v_{i}}$ and ${v^{{}^{\prime}}_{l}}$ to
the same dimensional space, obtaining ${v_{id}}$ and ${v^{{}^{\prime}}_{ld}}$.
These features are utilized to compute the constraint score ${S_{c}}$, as
depicted below:

${S_{c}}={\beta}*e^{\bigg{(}-\frac{{\big{(}1-{v_{id}(x)^{T}}{v^{{}^{\prime}}_{ld}(x)}\big{)}^{2}}}{2\theta^{2}}\bigg{)}}$
(4)

where $\beta$ and $\theta$ are learnable parameters.
${v_{id}(x)^{T}}{v^{{}^{\prime}}_{ld}(x)}$ represents the constraint score for
each point ${x}$ in the vector. After obtaining the constraint score, we
employ it to dampen the significance of areas in the image that are irrelevant
to the language, obtaining language guided image feature
${v_{i}^{{}^{\prime}}}$.

${v_{i}^{{}^{\prime}}}={\lambda}*{S_{c}}*{v_{i}}+(1-\lambda)*{v_{i}}$ (5)

where $\lambda$ denotes the balance parameter, which is empirically set to
0.6. Finally, following the GroundingDINO, we use language-guided query
selection ${f_{ls}(\cdot)}$ to select queries ${v_{q}}$ from
${v_{i}^{{}^{\prime}}}$. The number of queries is set to 900. Subsequently, we
input ${v_{i}^{{}^{\prime}}}$, ${v_{l}^{{}^{\prime}}}$, and ${v_{q}}$ into
cross-modality decoder ${f_{cd}(\cdot)}$, outputting the bounding box ${y}$
with the maximum score.

${y}=argmax({f_{cd}({f_{ls}({v_{i}^{{}^{\prime}}},{v_{l}^{{}^{\prime}}})},{v_{i}^{{}^{\prime}},{v_{l}^{{}^{\prime}}}})})$
(6)

####  III-B2 Grasping System

The system is designed to guide the robots to grasp target object. During the
experimental process, we observed that generating an accurate grasping pose
for smaller objects is challenging. Therefore, we use the target bounding box
obtained from visual-linguistic perception system to segment the original
point cloud, obtaining the target point cloud. Subsequently, we input the
target point cloud into pre-trained GraspNet to generate 6-DOF grasping poses.
To enhance the success rate of grasping and filter out unsuitable postures, we
use the grasp score obtained by GraspNet and grasp angle as thresholds to
filter grasp poses. Following this, in the process of selecting the target
grasp pose, we employ Euclidean distance to compare the distances between the
center point of the bounding box and the center point of each grasp pose. The
grasp pose with the shortest distance is chosen as the target grasp pose.

##  IV Experiments

In this section, we evaluate the effectiveness and rationality of our proposed
framework on OVGrasping dataset and virtual environment, which includes
visual-linguistic perception and grasping success rate.

###  IV-A Visual-Linguistic Perception

####  IV-A1 Training Settings

We train the visual-linguistic system on OVGrasping dataset by using two
NVIDIA Geforce RTX 3090 GPUs. The learning rate and batch size are set to 1e-5
and 16, respectively. We initialize the OVGNet parameters with GroundingDINO,
which has been pre-trained on diverse datasets including COCO[35], O365[36],
LIVS[37], V3Det[38], Flickr30k[39], and GRIT-200K. Empirically, we set the
fine-tuning epoch to 10. In the training process, we applied a random cropping
for data augmentation. The longest edge of image and the max text length are
set to 1333 and 256, respectively.

####  IV-A2 Result on OVGrasping Dataset

The accuracy of visual-linguistic perception is an important factor in
ensuring the efficiency of open-vocabulary grasping. To fully demonstrate the
effectiveness of OVGNet, we conducted a comparative experiment on the
OVGrasping dataset. Following previous works [40, 4], we use<EMAIL_ADDRESS>as
the evaluation metric, which requires the intersection over union (IoU)
between the predicted bounding box and the ground-truth box to be greater than
0.5. To ensure consistency in data usage between VL-Grasp and OVGNet, we
retrained VL-Grasp on the OVGrasping dataset, in which the batch size and
training epoch were set to 8 and 90, respectively.

As shown in Table II, our approach exhibits a marginally lower performance
compared to VL-Grasp in the base category, which may be attributed to the
fewer training epochs compared with VL-Grasp. However, in contrast, our method
demonstrates outstanding perceptual and generalization ability in novel
category. Specifically, OVGNet achieves 47.38$\%$ improvement over VL-Grasp.
Such significant improvement can be attributed to prior knowledge from the
foundation model, and fine-tuning knowledge from base objects. These
experimental results validate the effectiveness and rationality of our visual-
linguistic perception system.

TABLE II: COMPARISON of DETECTION RESULTS with other METHODS ON OVGRASPING.

Methods | Pre-trained Data | OVGrasping
---|---|---
Base | Novel
VL-Grasp | None | 86.08 | 18.88
OVGNet(ours) | COCO, O365, LIVS, V3Det, | 84.22 | 66.26
GRIT-200K, Flickr30k.

TABLE III:  Ablation study of our proposed modules on OVGrasping dataset.

IGLA | LGIA | OVGrasping
---|---|---
Base | Novel
$\times$ | $\times$ | 81.90 | 64.09
$\checkmark$ | $\times$ | 82.98 | 65.08
$\times$ | $\checkmark$ | 83.03 | 65.21
$\checkmark$ | $\checkmark$ | 84.22 | 66.26

####  IV-A3 Visualization Experiments

In this section, to further analyze the characteristics of OVGNet and VL-
Grasp, we conducted the visualization analysis on OVGrasping dataset. As shown
in Fig. 4 (a), in the base category, both VL-Grasp and OVGNet can accurately
detect the target object referred by natural language. As indicated in Fig. 4
(b), VL-Grasp can not efficiently detect novel objects, while OVGnet can
accurately locate the novel objects. As shown in Fig. 4 (c), for the target
object described with subject-object relationships, VL-Grasp tends to detect
the relative object that belong to the base class. In contrast, the OVGNet can
effectively recognize and locate the target object.

####  IV-A4 Ablation Study

In this section, to fully demonstrate the effectiveness of the proposed
modules, we conducted the ablation studies on the OVGrasping dataset. As shown
in Table III, compared to the baseline, the performance in the base category
exhibits enhancements of 1.08% and 1.13% with the inclusion of IGLA and LGIA,
respectively. In novel category, the two modules achieve an improvement with
0.99$\%$ and 1.12$\%$, respectively. With the addition of the two modules, our
method outperforms the baseline by 2.32$\%$ and 2.17$\%$ in base and novel
categories, respectively. The above experiments effectively validate that the
integration of two modules effectively enhances the alignment between vision
and language, thus improving the performance of detection accuracy in both
base and novel.

Figure 4: Visualization on OVGrasping dataset. Green boxes indicate the
ground-truth, and red boxes denote the detection results.

###  IV-B Grasping in Virtual Environment

####  IV-B1 Grasping Setting

We constructed a grasping simulation environment based on pybullet, employing
the UR5 arm and ROBOTIQ-85 to grasp target object, and using Intel RealSense
L515 to capture image for the visual-language perception system. We randomly
generated 135 grasping scenes to test grasping accuracy, of which 65 are used
to test base and novel categories, and 70 are used to evaluate different
tasks. All grasping scenes are stored in URDF format, which can be loaded by
pybullet. Each scene includes 8 objects and may contain multiple identical or
similar objects. To demonstrate our framework can locate the target object
from identical objects, we further classify grasping scenes into single-
grasping and multiple-grasping scenarios.

####  IV-B2 Grasping Test

In practical scenarios, the high trial-and-error cost for the robot hinders
repeated attempts to grasp the target object. To comprehensively assess the
grasping efficiency of our proposed framework, we constrain the attempts for
each scene, setting the maximum attempts to 3. Specifically, once the robot
successfully grasps an object in a scene, the scene automatically terminates,
irrespective of whether the grasped object is the intended target or not.

Following the above criteria, we conducted grasping tests on 65 scenes
including base and novel. As shown in Table IV, our grasping framework
demonstrates a remarkable success rate, requiring fewer grasping attempts.
Specifically, with just one grasping attempt, our framework attains an average
success rate of 44.9% for the base objects and 34.4% for novel objects. The
success rate exhibits a consistent upward trend with additional grasping
attempts. Upon reaching three attempts, our framework achieves its peak
success rate of 71.2% for base objects and 64.4% for novel objects. In the
base category, we find that our framework tends to the single grasping scenes,
achieving a maximum success rate of 73.1$\%$. While in the novel category, it
prefers to multiple-grasping scenes, and achieves a success rate of 65.1$\%$.
The above results effectively validate the grasping performance of our
framework on base and novel categories.

Additionally, we select 4 simple objects and 3 hard objects to conduct the
grasping test of different tasks. Each object includes 10 grasping scenes,
which may contain identical objects, with the maximum of attempts as 3. As
shown in Table V, in the simple task, our framework achieves an average
success rate of 87.5$\%$, surpassing the average level in base category.
Meanwhile, in the hard task, our framework attains an average success rate of
56.6$\%$, approaching the average level in novel class. The above results
validate the generalization of our framework in various tasks.

TABLE IV:  grasping success rate test on base and novel scenes.

Attempts | Base | Novel
---|---|---
Single | Multi | Total | Single | Multi | Total
1 | 50.3 | 39.5 | 44.9 | 36.3 | 32.4 | 34.4
2 | 62.1 | 53.2 | 57.7 | 54.6 | 44.5 | 49.6
3 | 73.1 | 69.2 | 71.2 | 63.6 | 65.1 | 64.4

TABLE V:  grasping success rate test on different tasks.

Task | Object | Base | Novel | Success Rate
---|---|---|---|---
Simple | Apple | $\checkmark$ |  | 10/10
Orange | $\checkmark$ |  | 8/10
Toothpaste Box | $\checkmark$ |  | 8/10
Pear | $\checkmark$ |  | 9/10
Total | - | - | 87.5
Hard | Dragon |  | $\checkmark$ | 5/10
Power Drill |  | $\checkmark$ | 6/10
battery |  | $\checkmark$ | 6/10
Total | - | - | 56.6

####  IV-B3 Comparison with other methods

In this section, to further verify the effectiveness of our framework, we
employ 8 grasping scenes provided by VLAGrasp [5] to test the grasping success
rate. The grasping and linguistic settings provided by VLAGrasp are different
from our framework. First, in our setting, once an object is successfully
grasped, regardless of whether it is the target object, the scene
automatically terminates. While in VLAGrasp, the grasping process persists
even if the initially grasped object is not the intended target, extending
until the maximum allowed attempts are exhausted. Obviously, our grasping
setting is better suited for real-world applications. Second, the language
provided by VLAGrasp can refer to multiple target objects. Nevertheless, in
our setting, the language must refer to single target object. Therefore, we
adjust the language provided by VLAGrasp to fit our linguistic setting. The
examples are shown as follows, where Ori. and Adj. represents original and
adjustment language, respectively.

  * •

Ori. : grasp a round object.

  * •

Adj. : grasp a blue round object.

As shown in Table VI, our framework achieves excellent success rate with fewer
attempts. It is worth noting that the grasping scenes provided by VLAGrasp
include 15 objects that belong to crowded scene, but our framework can still
accurately locate and grasp the target objects. Such performance adequately
illustrates the robust environmental adaptability of our framework.

TABLE VI:  GRASPING SUCCESS RATE test on eight scenes provided by VLAGrasp
[5].

Scenes | Target Object | Base | Novel | Success | attempts
---|---|---|---|---|---
1 | Racquetball |  | $\checkmark$ | $\checkmark$ | 1
2 | Apple | $\checkmark$ |  | $\checkmark$ | 1
3 | Blue cup | $\checkmark$ |  | $\checkmark$ | 1
4 | Orange | $\checkmark$ |  | $\checkmark$ | 2
5 | Banana | $\checkmark$ |  | $\times$ | -
6 | Weiquan |  |  | $\checkmark$ | 1
7 | Theramed |  | $\checkmark$ | $\times$ | -
8 | Pear | $\checkmark$ |  | $\checkmark$ | 1
Total | - | - | - | 75.0 | 1.16

####  IV-B4 Case Analysis

In this section, to better analyze the characteristics and challenges of our
framework, we visualize the grasping poses of the target object and cases of
grasping failures. Obviously, as shown in Fig. 5 (a), when the visual-
linguistic perception system outputs incorrect result, our framework is unable
to locate the target object, resulting in failed grasping attempt.
Furthermore, as shown in Fig. 5 (b), although the target object is accurately
located, there exists some occlusion in the target-level point cloud. This
creates interference when generating grasping poses. Lastly, as shown in Fig.
5 (c), in the attempts of grasping irregular objects with indentations or
protrusions, the grasping poses are prone to these regions that are not
optimal for grasping, which leads to failures in grasping.

Figure 5: Case analysis. Green boxes indicate the ground-truth, red boxes
denote the predict results, and yellow area represents the defect.

##  V Conclusion

In this paper, we present a novel approach to enhance robotic grasping by
integrating open-vocabulary learning. Consequently, we contribute a
challenging dataset named OVGrasping to benchmark the open-vocabulary task,
and propose a unified visual-linguistic framework for open-vocabulary
grasping. In the future, we will continue to explore the open-vocabulary
grasping task, aiming to design an end-to-end open-vocabulary grasping system.

## References

  * [1] Adheesh Shenoy, Tianze Chen, and Yu Sun,  “Multi-object grasping-efficient robotic picking and transferring policy for batch picking,”  in IEEE International Conference on Intelligent Robots and Systems (IROS), 2022, pp. 2741–2747.
  * [2] Julia Borras, Guillem Alenya, and Carme Torras,  “A grasping-centered analysis for cloth manipulation,”  IEEE Transactions on Robotics (TRO), vol. 36, no. 3, pp. 924–936, 2020.
  * [3] Xing Wang, Hanwen Kang, Hongyu Zhou, et al.,  “Geometry-aware fruit grasping estimation for robotic harvesting in apple orchards,”  Computers and Electronics in Agriculture (CEA), vol. 193, pp. 106716, 2022.
  * [4] Yuhao Lu, Yixuan Fan, Beixing Deng, et al.,  “Vl-grasp: a 6-dof interactive grasp policy for language-oriented objects in cluttered indoor scenes,”  in IEEE International Conference on Intelligent Robots and Systems (IROS), 2023, pp. 976–983.
  * [5] Kechun Xu, Shuqi Zhao, Zhongxiang Zhou, et al.,  “A joint modeling of vision-language-action for target-oriented grasping in clutter,”  in IEEE International Conference on Robotics and Automation (ICRA), 2023, pp. 11597–11604.
  * [6] Chao Tang, Dehao Huang, Wenqi Ge, et al.,  “Graspgpt: Leveraging semantic knowledge from a large language model for task-oriented grasping,”  IEEE Robotics and Automation Letters (RAL), vol. 8, no. 11, pp. 7551–7558, 2023.
  * [7] Alec Radford, Jong Wook Kim, Chris Hallacy, et al.,  “Learning transferable visual models from natural language supervision,”  in International Conference on Machine Learning (ICML), 2021, vol. 139, pp. 8748–8763.
  * [8] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, et al.,  “Grounding dino: Marrying dino with grounded pre-training for open-set object detection,”  arXiv preprint arXiv:2303.05499, 2023.
  * [9] Valerija Holomjova, Andrew J Starkey, and Pascal Meißner,  “Gsmr-cnn: An end-to-end trainable architecture for grasping target objects from multi-object scenes,”  in IEEE International Conference on Robotics and Automation (ICRA), 2023, pp. 3808–3814.
  * [10] Ran Qin, Haoxiang Ma, Boyang Gao, et al.,  “Rgb-d grasp detection via depth guided learning with cross-modal attention,”  in IEEE International Conference on Robotics and Automation (ICRA), 2023.
  * [11] Guangyao Zhai, Dianye Huang, Shun-Cheng Wu, et al.,  “Monograspnet: 6-dof grasping with a single rgb image,”  in IEEE International Conference on Robotics and Automation (ICRA), 2023, pp. 1708–1714.
  * [12] Qiyu Dai, Yan Zhu, Yiran Geng, et al.,  “Graspnerf: Multiview-based 6-dof grasp detection for transparent and specular objects using generalizable nerf,”  in IEEE International Conference on Robotics and Automation (ICRA), 2023, pp. 1757–1763.
  * [13] Hu Cheng, Yingying Wang, and Max Q.-H. Meng,  “A robot grasping system with single-stage anchor-free deep grasp detector,”  IEEE Transactions On Instrumentation And Measurement (TIM), vol. 71, pp. 1–12, 2022.
  * [14] Haoshu Fang, Chenxi Wang, Minghao Gou, et al.,  “Graspnet-1billion: A large-scale benchmark for general object grasping,”  in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020, pp. 11441–11450.
  * [15] Yuhao Lu, Beixing Deng, Zhenyu Wang, et al.,  “Hybrid physical metric for 6-dof grasp pose detection,”  in International Conference on Robotics and Automation (ICRA), 2022, pp. 8238–8244.
  * [16] Yucheng Xu, Mohammadreza Kasaei, Hamidreza Kasaei, et al.,  “Instance-wise grasp synthesis for robotic grasping,”  in IEEE International Conference on Robotics and Automation (ICRA), 2023, pp. 1744–1750.
  * [17] Shaochen Wang, Zhangli Zhou, Zhen Kan, et al.,  “When transformer meets robotic grasping: Exploits context for efficient grasp detection,”  IEEE Robotics and Automation Letters (RAL), vol. 7, no. 3, pp. 8170–8177, 2022.
  * [18] Xiangtai Li, Xia Li, Li Zhang, et al.,  “Improving semantic segmentation via decoupled body and edge supervision,”  in European Conference Computer Vision (ECCV), 2020, vol. 12362, pp. 435–452.
  * [19] Xiangtai Li, Xia Li, Ansheng You, et al.,  “Towards efficient scene understanding via squeeze reasoning,”  IEEE Transactions on Image Processing (TIP), vol. 30, pp. 7050–7063, 2021.
  * [20] Hanoona Abdul Rasheed, Muhammad Maaz, Muhammad Uzair, et al.,  “Bridging the gap between object and image-level representations for open-vocabulary detection,”  in Neural Information Processing Systems (NeuralIPS), 2022.
  * [21] Yu Du, Fangyun Wei, Zihe Zhang, et al.,  “Learning to prompt for open-vocabulary object detection with vision-language model,”  in IEEEConference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 14064–14073.
  * [22] Hao Zhang, Feng Li, Xueyan Zou, and other,  “A simple framework for open-vocabulary segmentation and detection,”  in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 1020–1031.
  * [23] Jie Wu, Chunlei Wu, Jing Lu, et al.,  “Region reinforcement network with topic constraint for image-text matching,”  IEEE Transactions on Circuits and Systems for Video Technology (TCSVT), vol. 32, no. 1, pp. 388–397, 2022.
  * [24] Chaofan Ma, Yuhuan Yang, Yan-Feng Wang, et al.,  “Open-vocabulary semantic segmentation with frozen vision-language models,”  in British Machine Vision Conference (BMVC), 2022, p. 45.
  * [25] Feng Liang, Bichen Wu, Xiaoliang Dai, et al.,  “Open-vocabulary semantic segmentation with mask-adapted CLIP,”  in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 7061–7070.
  * [26] Jianzong Wu, Xiangtai Li, Henghui Ding, et al.,  “Betrayed by captions: Joint caption grounding and generation for open vocabulary instance segmentation,”  in IEEE International Conference on Computer Vision (ICCV)), 2023, pp. 21881–21891.
  * [27] Jiarui Xu, Sifei Liu, Arash Vahdat, et al.,  “Open-vocabulary panoptic segmentation with text-to-image diffusion models,”  in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 2955–2966.
  * [28] Mathilde Caron, Hugo Touvron, Ishan Misra, et al.,  “Emerging properties in self-supervised vision transformers,”  in IEEE International Conference on Computer Vision (ICCV), 2021, pp. 9630–9640.
  * [29] Chunlei Wang, Wenquan Feng, Xiangtai Li, et al.,  “Ov-vg: A benchmark for open-vocabulary visual grounding,”  arXiv preprint arXiv:2310.14374, 2023.
  * [30] Jacob Devlin, Ming-Wei Chang, Kenton Lee, et al.,  “BERT: pre-training of deep bidirectional transformers for language understanding,”  in North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), 2019, pp. 4171–4186.
  * [31] Ajay Patel, Bryan Li, Mohammad Sadegh Rasooli, et al.,  “Bidirectional language models are also few-shot learners,”  in International Conference on Learning Representations (ICLR), 2023.
  * [32] Chao Tang, Dehao Huang, Lingxiao Meng, et al.,  “Task-oriented grasp prediction with visual-language inputs,”  in IEEE International Conference on Intelligent Robots and Systems (IROS), 2023, pp. 4881–4888.
  * [33] Qiang Sun, Haitao Lin, Ying Fu, et al.,  “Language guided robotic grasping with fine-grained instructions,”  in IEEE International Conference on Intelligent Robots and Systems (IROS), 2023, pp. 1319–1326.
  * [34] Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang, et al.,  “Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation,”  in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 803–814.
  * [35] Tsung-Yi Lin, Michael Maire, Serge Belongie, et al.,  “Microsoft coco: Common objects in context,”  in European Conference Computer Vision (ECCV), 2014, pp. 740–755.
  * [36] Shuai Shao, Zeming Li, Tianyuan Zhang, et al.,  “Objects365: A large-scale, high-quality dataset for object detection,”  in IEEE International Conference on Computer Vision (ICCV), 2019, pp. 8430–8439.
  * [37] Agrim Gupta, Piotr Dollar, and Ross Girshick,  “Lvis: A dataset for large vocabulary instance segmentation,”  in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 5356–5364.
  * [38] Jiaqi Wang, Pan Zhang, Tao Chu, et al.,  “V3det: Vast vocabulary visual detection dataset,”  arXiv preprint arXiv:2304.03752, 2023.
  * [39] Bryan A Plummer, Liwei Wang, Chris M Cervantes, et al.,  “Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models,”  in IEEE International Conference on Computer Vision (ICCV), 2015, pp. 2641–2649.
  * [40] Muchen Li and Leonid Sigal,  “Referring transformer: A one-step approach to multi-task visual grounding,”  Advances in neural information processing systems (NeuralIPS), vol. 34, pp. 19652–19664, 2021.