In the present paper we study the geometry of the closed Białynicki-Birula cells of the quiver Grassmannians associated to a nilpotent representation of a cyclic quiver defined by a single matrix. For the special case, where we choose subrepresentations of dimension $\mathbf{1}=(1,\dots,1)$, the main result of this paper is that the closed Białynicki-Birula cells are smooth. We also discuss the multiplicative structure of the cohomology ring of such spaces. Namely, we describe the so-called Knutson-Tao basis in context of the basis of equivariant cohomology that is dual to fundamental classes in equivariant homology.

§ INTRODUCTION

The theory of quiver Grassmannians, first introduced by Schofield in <cit.> has its roots in representation theory, but it found its way to other fields, such as algebraic geometry. This theory, from the perspective of algebraic geometry, is a rich one – we know that these spaces describe all projective varieties as was shown in <cit.>, so they are an attractive object to study. For acyclic quivers many properties such as smoothness, irreducibility or dimension have known criteria, for more information one can consult <cit.>.

More recently, M. Lanini and A. Pütz in <cit.> define torus actions on the quiver Grassmannians associated to the so-called nilpotent representations of cyclic quivers, which endows the quiver Grassmannians with a GKM-variety structure and so lets us compute equivariant cohomology of these spaces. These spaces were further analyzed in  <cit.>, <cit.> in context of group actions and resolution of singularities respectively.

This paper delves into the GKM-variety structure and the Białynicki-Birula decomposition of these quiver Grassmannians in a special case. Let $\Delta_n$ be the equioriented cycle with $n$ vertices and $\mathcal{M}$ be a $\Delta_n$-representation that has the same matrix $J:\mathbb{C}^N\to\mathbb{C}^N$ over each arrow in $\Delta_n$. In this paper we study the quiver Grassmannian $X=\qgr{1}{\mathcal{M}}\subseteq\prod_{i=1}^n\proj{}{N-1}$ of subrepresentations of dimension vector $\mathbf{1}\in\mathbb{Z}^{n}$. Without loss of generality $J$ can be assumed to be a Jordan matrix with $M$ Jordan blocks. Following <cit.> we introduce a torus action on $X$ and study its $T$-equivariant cohomology. Authors of <cit.> tackled a similar problem – authors considered the quiver Grassmannian $\qgr{k}{\mathcal{M}}$, where $\mathbf{k}=(k,\dots,k)$, but $\mathcal{M}$ was defined by $J$ which is a single Jordan block of size $n$. In that paper one can find, for example, the Poincaré polynomials of such quiver Grassmannians, resolution of singularities and the connection between these quiver Grassmannians and totally nonnegative Grassmannians.

The Białynicki-Birula cells form a stratification of the quiver Grassmannian into affine cells, as was shown in <cit.>. In our case  we prove the following two results concerning the structure of the closed Białynicki-Birula cells
Let $p\in X^{T}$ be a fixed point. Then the closure of the BB-cell $\mathcal{C}_p$ is a product of smooth quiver Grassmannians.

The nilpotent Jordan matrix $J$ corresponds to a partition of $N$ and so we can visualize coordinates of $\mathbb{C}^N$ using the Young tableau $\mathcal{J}$ of shape associated to $J$. Appropriately identifying the basis vectors of $\mathbb{C}^N$ with boxes of $\mathcal{J}$ leads to the definition of twisted lexicographic order on this basis. More precisely, suppose that $J$ has $M$ blocks and the $l$-th block is of size $j_l$. Let $v_i^l$ be the $i$-th Jordan basis vector of the $l$-th block of $J$. We then set $v_i^l<v_k^s$ if either $j_s-k<j_l-i$ or $j_s-k=j_l-i$ and $l<k$. The closed BB-cells form a poset with respect to inclusion and we prove that these two relations are inverse to each other
For two fixed points $p,q\in X^T$ we have $\clBB{p}\subseteq\clBB{q}$ if and only if $q_i\le p_i$ for every $i\in\mathbb{Z}/n\mathbb{Z}$.

We can use the visual nature of coordinates to visualize possible edges in the GKM-graph of $X$. Namely, each fixed point $p\in X^T$ consists of choosing the basis vectors $v_i^l$ over each vertex. Thus, we can divide $p$ into movable parts, i.e. segments of these lines in $\mathbb{C}^N$ that do not go to $0$ under $J$. Each movable part can be seen as a coloring of first couple of boxes in a fixed row in $\mathcal{J}$ and a mutation of movable part is defined to be either a coloring of the same amount of boxes, but in a lower row or two colorings – we split the first coloring into two. We then show that these mutations define edges in the GKM-graph of $X$ in the following Proposition
Let $p,q\in X^T$ be two fixed points. Then an arrow $p\to q$ in the GKM graph of $X$ exists if and only if $q$ is obtained from $p$ by a mutation of exactly one movable part in $p$.

We also give a way to get the label of such an edge in Proposition $\ref{GKMedges}$.

The closed Białynicki-Birula cells from a basis of equivariant homology and since they are smooth, integration along these cells can be done using the Atiyah-Bott, Berline-Vergne, which heavily simplifies actual computations. We can construct the dual basis $\{p^x\in H_T^*(X):x\in X^T\}$ and this basis is immediately seen to be a Knutson-Tao basis of $H_T^*(X)$, which were defined in <cit.>. Again, as the cells are smooth, then computing things such as structure constants $c_{x,y}^z$, defined by

\begin{equation*}
p^x\cdot p^y = \sum_{z\in X^T}c_{x,y}^zp^z
\end{equation*}

is immediate, since this comes down to linear algebra (albeit linear algebra over large vector spaces).

This paper is constructed as follows – Chapter $\ref{BasicDefinitions}$ is meant to fix notation and recall some very basic notions regarding the quiver Grasssmannians and nilpotent representations. Chapter $\ref{TorusActions}$ recalls the construction of actions of two tori on quiver Grassmannians for nilpotent $\Delta_n$-representations from <cit.>, but only for our case. Chapter $\ref{BBDecomposition}$ is about the Białynicki-Birula decomposition of this particular family of quiver Grassmannians. Namely, we recall a result from <cit.>, that the induced Białynicki-Birula decomposition is a cellular decomposition into affine cells. We further investigate how these cells look like after taking the closure in the ambient space of the quiver Grassmannian, which is a product of projective spaces. This chapter proves that the closed Białynicki-Birula cells are smooth. In Chapter $\ref{GKMStructure}$ we describe the GKM graph of these quiver Grassmannians, which was also computed in <cit.>, but we provide a description using colorings of Young tableaux. In Chapter $\ref{MultiplicativeStructure}$ we define the basis dual to the equivariant fundamental classes in equivariant homology and discuss computing structure constants of this basis.

§ BASIC DEFINITIONS AND NOTATION

Let $T=(\mathbb{C}^\times)^r$ be an algebraic torus. For a $T$-space $X$ we define its equivariant cohomology via the Borel construction. We denote by $H_T^*(X)$ the equivariant cohomology ring with coefficients in $\mathbb{Q}$ (for a proper introduction into equivariant cohomology see <cit.>).

Now suppose that $X$ is a $\mathbb{C}^\times$-space and we decompose the fixed point set into connected components as follows

\begin{equation*}
X^{\mathbb{C}^\times} = \bigcup_{k\ge0} X_k,
\end{equation*}

then we define the Białynicki-Birula cells or the BB-cells as

\begin{equation*}
\mathcal{C}_k = \{x\in X:\lim_{\lambda\to0}\lambda\cdot x\in X_k\}.
\end{equation*}

Together these cells form the BB-decomposition of $X$. If these cells are affine spaces, then $X$ is an equivariantly formal $\mathbb{C}^\times$-variety.

Let $Q=(Q_0,Q_1)$ be a finite and connected quiver and $M=(M_i,M_{\alpha})_{i\in Q_0,\alpha\in Q_1}$ be a representation of $Q$ over $\mathbb{C}$ (equivalently, a $\mathbb{C}Q$-module). We denote by ${\dim M\in\mathbb{Z}_{\ge0}^{|Q_0|}}$ the dimension vector of $M$, i.e. $(\dim M)_{i}=\dim_{\mathbb{C}} M_i$ (for a proper introduction to representations of quivers one can consult <cit.>). For a fixed vector $\mathbf{e}\le \dim M$ we denote by $\qgr{e}{M}$ the quiver Grassmannian of subrepresentations of $M$ of dimension $\mathbf{e}$. These spaces are known to have a (projective) scheme structure given by their embedding into a product of usual Grassmannians, since a subrepresentation of $M$ is determined by a choice of subspaces for all $M_i$'s. Note that these spaces in general are not varieties as they need not be irreducible or even reduced.

Let $M\in\text{rep}(Q)$ be a representation and $B$ be the set of basis elements of each vector space in $M$. We divide $B$ into subsets corresponding to each vertex – let $B^i\subset B$ be the subset of $B$ that is the basis of $M_i$. The coefficient quiver of $M$ with respect to the basis $B$ is defined as the following quiver:

* The set of vertices is $B$.

* For $v_k^i\in B^i, v_l^j\in B^j$ an arrow $v_k^i\to v_l^j$ exists if and only if there exists an arrow $\alpha:i\to j$ such that the $l$-th coordinate of $M_\alpha(v_k^i)$ in $B^j$ is non-zero.

We denote this coefficient quiver by $Q(M,B)$. This tool was used to find a well-suited tori actions on quiver Grassmannians, for example in <cit.> and this paper is following this construction.

In this paper we will be interested in a special family of representations of a certain family of quivers. Namely, let $\Delta_n$ be the oriented cycle quiver, that is an equioriented $n$-gon. We label both the vertices and edges using the set $\mathbb{Z}/n\mathbb{Z}$.

A $\Delta_n$-representation $M$ is called nilpotent if there exists a number $N>0$ such that $M$, considered as a $\mathbb{C}\Delta_n$-module, is annihilated by the ideal of paths of length at least $N$. Explicitly, this means that for any $i\in Q_0$ the composition of $N$ consecutive maps from $M$ vanishes.

A basic example of a nilpotent representation is a representation that has the same vector space, say of dimension $N$, over each vertex and the same nilpotent map over each arrow. Since we can simultaneously change basis, we can without loss of generality assume that over each vertex we have $\mathbb{C}^N$ and our map is a Jordan matrix $J:\mathbb{C}^N\to\mathbb{C}^N$, whose Jordan-block decomposition is given by

\begin{equation*}
J = J_1\oplus \dots\oplus J_{M}, \text{ where } J_i\in M_{j_i\times j_i}(\{0,1\}), j_1\ge j_2\ge\dots\ge j_M.
\end{equation*}

As $J$ must be nilpotent, each Jordan block has zeroes on the diagonal. We will denote this representation by $\mathcal{M}=(\mathcal{M}_i=\mathbb{C}^N,\mathcal{M}_\alpha=J)_{i,\alpha\in\mathbb{Z}/n\mathbb{Z}}$.

Following <cit.> we can construct a GKM-variety structure on any quiver Grassmannian associated to any nilpotent $\Delta_n$-representation and we will use this result to study the geometry of $\qgr{1}{\mathcal{M}}$, where $\mathbf{1}=(1,\dots,1)$.

Throughout this paper we will talk about subspaces defined as a linear span of some basis vector and we will shorthand $\text{span}(v_1,\dots,v_n)$ by $\langle v_1,\dots,v_n\rangle$. We denote the set $\{1,\dots,M\}$ by $[M]$.

§ TORUS ACTIONS ON THE MAIN OBJECT

In this section we recall known results from <cit.> applied to our example of $X:=\qgr{1}{\mathcal{M}}$, where $\mathcal{M}$ is the representaion defined in Example ${MainObject}$ and $\mathbf{1}=(1,\dots,1)$. This space clearly embeds into $\mathbb{P}:=(\mathbb{P}^{N-1})^n$ and authors of <cit.> in fact define actions of two tori on $\mathbb{P}$ that fixes $X$.

Let us start with the $\mathbb{C}^\times$-action. Fix $l\in\{1,\dots,M\}$ and let $V_l\subseteq\mathbb{C}^N$ be the subspace fixed by the $l$-th Jordan block in $J$. We denote the Jordan basis of $V_l$ by $v_1^l,\dots,v_{j_l}^l$. Consider a linear action of $\mathbb{C}^\times$ on $\mathbb{C}^N$, defined on basis vectors $v_i^l\in\mathbb{C}^N$ as follows

\begin{equation}\label{C-action}
\lambda\cdot v_i^l := \lambda^{l-M(j_l-i)}v_i^l, \text{ for }\lambda\in\mathbb{C}^\times.
\end{equation}

In other words, the weight of $v_{j_l}^l$ is $l$ and as we decrease the index of the basis vector we decrease the weight by $M$ (so long as the vectors lie in the same $V_l$). Note that we act on each coordinate by different weights. After we pass to the projective space $\proj{}{N-1}$ the fixed point set under the induced torus action is the set of (linear spans of) standard basis vectors. We extend this action to the diagonal action on $\mathbb{P}$. It requires a check, that $X$ is $\mathbb{C}^\times$-invariant under this action and we refer the reader to <cit.> for the proof.

Next, we describe an action of $T=(\mathbb{C}^\times)^{nM+1}$ on $\mathbb{P}$ that fixes $X$, but omit describing the whole construction here and state the resulting torus action; readers interested in where this action comes from are referred to <cit.>. We start with the $0$-th vertex and define the following torus action on basis vectors

\begin{equation*}\label{T-action}
(t_0,\{t_{r,s}\}_{r\in\mathbb{Z}/n\mathbb{Z},s\in[M]})\cdot v_{i}^l := t_0^{i-1}t_{j_l-i,l}v_i^l, \text{ for }(t_0,\{t_{r,s}\}_{r\in\mathbb{Z}/n\mathbb{Z},s\in[M]})\in T.
\end{equation*}

This naturally extends to $\proj{}{N-1}$ and to extend this to $\mathbb{P}$ we use an automorphism $\rho:T\to T$, which we define as follows

\begin{align}\label{Rotation}
\rho(t_0):=t_0, && \rho(t_{j,k}) := t_{j+1,k}.
\end{align}

Now, if $v_i^l$ lies over the $j$-th vertex, then we set

\begin{equation*}
(t_0,\{t_{r,s}\}_{r\in\mathbb{Z}/n\mathbb{Z},s\in[M]})\cdot v_i^l := \rho^j(t_0,\{t_{r,s}\}_{r\in\mathbb{Z}/n\mathbb{Z},s\in[M]})\cdot v_i^l=t_0^{i-1}t_{j_l-i+j,l}v_i^l.
\end{equation*}

This torus action makes $X$ a $T$-equivariant subvariety of its ambient space, by <cit.>.

Consider the case of a Jordan matrix with two blocks $J=J_3\oplus J_2$ and $3$ vertices. Then the $\mathbb{C}^\times$-action over each vertex is given by

\begin{equation*}
\lambda\cdot (a_1^1,a_2^1,a_3^1,a_1^2,a_2^2) = (\lambda^{-2} a_1^1,\lambda^{-1} a_2^1,\lambda^1 a_3^1, a_1^2,\lambda^2 a_2^2)
\end{equation*}

and the $T=(\mathbb{C}^\times)^{7}$-action over the zeroth, first and second vertex is given by

\begin{align*}
& t\cdot (a_1^1,a_2^1,a_3^1,a_1^2,a_2^2) = (t_{2,1}a_1^1,t_0t_{1,1}a_2^1,t_0^2t_{0,1}a_3^1,t_{1,2}a_1^2,t_{0,2}a_2^2),\\
& t\cdot(a_1^1,a_2^1,a_3^1,a_1^2,a_2^2) = (t_{0,1}a_1^1,t_0t_{2,1}a_2^1,t_0^2t_{1,1}a_3^1,t_{2,2}a_1^2,t_{1,2}a_2^2),\\
& t\cdot(a_1^1,a_2^1,a_3^1,a_1^2,a_2^2) = (t_{1,1}a_1^1,t_0t_{0,1}a_2^1,t_0^2t_{2,1}a_3^1,t_{0,2}a_1^2,t_{2,2}a_2^2).
\end{align*}

There is a beautiful interplay between these two actions – the $\mathbb{C}^\times$-action induces a BB-decomposition of $X$ that turns out to be a cellular decomposition by <cit.>, which implies that $X$ is equivariantly formal. The fixed point sets $X^{\mathbb{C}^\times}$ and $X^T$ are equal and the $\mathbb{C}^\times$-action can be recovered from the $T$-action via a cocharacter $\chi:\mathbb{C}^\times\to T$ defined by

\begin{equation*}
\chi(t) := (t^M,\{t^{k-Mj_k+M}\}_{j\in\mathbb{Z}/n\mathbb{Z},k\in[M]}).
\end{equation*}

The $T$-variety $X$ is a GKM-variety by <cit.> and its GKM graph can be described using the coefficient quiver of $\mathcal{M}$, but in our specific case we can give descriptions of the BB-cells and the GKM of $X$ in a simpler language.

After fixing the basis of the vector space $\mathcal{M}$ to be the sets $\{v_i^l\}$ over every vertex, the coefficient quiver of $\mathcal{M}$ with respect to this basis can be drawn as $M$ spirals starting at each vertex of $\Delta_n$ and the $l$-th spiral spans $j_l$ vertices. We adopt the notation from <cit.> and denote this quiver by $Q(\mathcal{M})$.

Take for example $n=3$ and $J=J_3$ to be a single Jordan block of size $3\times 3$. Then the coefficient quiver $Q(\mathcal{M})$ looks as follows

\[\resizebox{0.7\textwidth}{!}{
\begin{tikzcd}[ampersand replacement=\&]
\&\&\& v_1^1 \\
\&\&\& v_2^1 \\
\&\&\& v_3^1 \\
\\
\&\& v_3^1 \&\&\& v_3^1 \\
\& v_2^1 \&\&\&\&\& v_2^1 \\
v_1^1 \&\&\&\&\&\&\& v_1^1
\arrow[from=1-4, to=6-7]
\arrow[from=6-7, to=5-3]
\arrow[from=7-1, to=2-4]
\arrow[from=2-4, to=5-6]
\arrow[from=7-8, to=6-2]
\arrow[from=6-2, to=3-4]
\end{tikzcd}

Authors of <cit.> describe the GKM graph of $X$ in <cit.>. In later chapters we will remark how the description of the GKM graph of $X$ comes from the one given in <cit.>.

§ BIAŁYNICKI-BIRULA DECOMPOSITION

Here we will focus on the $\mathbb{C}^\times$-action on $X$, given by Formula $\eqref{C-action}$ and the induced BB-decomposition of $X$. We start with the known description of the fixed point set.
Fixed points of $X$ under both tori actions consists of representations that over all vertices have vector spaces spanned by the fixed basis $\{v_i^l:i=1,\dots,j_l,l=1,\dots,M\}$. These subrepresentations are in bijective correspondence with successor closed subquivers $S\subseteq Q(\mathcal{M})$, where $Q(\mathcal{M})$ is the coefficient quiver of $\mathcal{M}$ defined in Notation $\ref{CoefficientQuiver}$, such that $S$, restricted to vertices corresponding to a fixed vertex, has exactly one vertex.

In particular any fixed point $p\in X^T$, considered as a subquiver of $Q(\mathcal{M})$ decomposes into connected components.

Let $p\in X^T$ be a fixed point and $S_p\subseteq Q(\mathcal{M})$ be the subquiver defined by $p$. Then $S_p$ decomposes into connected succesor-closed subquivers

\begin{equation*}
\end{equation*}

where $\sigma_i\subseteq S_p$ are the maximal connected subquivers of $S_p$. Each $\sigma_i$ is called a movable part in $p$ and any such $\sigma_i$ defines a representation of the $A$-type equioriented quiver representation

\begin{equation*}
\sigma_i \rightsquigarrow (\langle v_{j_l-m}\rangle,\langle v_{j_l-m+1}\rangle,\dots,\langle v_{j_l}\rangle)
\end{equation*}

for some $m\ge 0$ and $l=1,\dots,M$. We shall say that $\sigma_i$ is thus of length $m$ and comes from the $l$-th block. To fully define $\sigma_i$ we also need either a starting vertex or the terminal vertex, since $\sigma_i$ corresponds to a part of $p$.

Here we turn to Young tableaux. Our fixed Jordan matrix $J$ corresponds to a Young tableau of shape $J$ and we shall denote this Young tableau by $\mathcal{J}$. Each box thus corresponds to a basis vector $v_i^l$ in $\mathbb{C}^N$. We can fill $\mathcal{J}$ with $\mathbb{C}^\times$-weights that act on these vectors, getting a nice table of weights.

Consider the case $J=J_4\oplus J_3\oplus J_3\oplus J_2$. Then we get the following Young tableaux
centertableaux, boxsize=2em

\begin{align*}
J \rightsquigarrow \begin{ytableau}
v_4^1 & v_3^1 & v_2^1 & v_1^1  \\
v_3^2 & v_2^2 & v_1^2  \\
v_3^3 & v_2^3 & v_1^3 \\
v_2^4 & v_1^4
\end{ytableau}  && \mathbb{C}^\times\text{-weights} = \begin{ytableau}
1 & -3 & -7 & -11  \\
2 & -2 & -6  \\
3 & -1 & -5 \\
4 & 0
\end{ytableau}
\end{align*}

We can endow the set of basis elements with a "twisted" lexicographic order. We state the exact definition below.

On the set of basis elements $\{v_i^l:i=1,\dots,j_l,l=1,\dots,M\}$ consider the twisted lexicographic order

\begin{equation*}
v_i^l < v_j^s \iff \text{either } j_s-j<j_l-i \text{ or } j_s-j=j_l-i \text{ and }l<s.
\end{equation*}

This order is of course equivalent to the order of $\mathbb{C}^\times$-weights, i.e.

\begin{equation*}
\text{weight}(v_i^l)<\text{weight}(v_j^s) \iff v_i^l<v_j^s.
\end{equation*}

We also extend this order to a partial order on fixed points: For two fixed points $p,q\in X^T$ we say that $p<q$ if $p_i\le q_i$ for all $i\in \mathbb{Z}/n\mathbb{Z}$ and $p_j< q_j$ for at least one $j\in\mathbb{Z}/n\mathbb{Z}$.

Note that we can check the inequality $p<q$ for two fixed points $p,q\in X^T$ only at vertices either ending or starting movable parts in $p$. This follows simply from the fact these vertices uniquely determine fixed points.

With this we can parametrize the BB-cells.

Let $p\in X^{T}$ be a fixed point. Then the BB-cell $\mathcal{C}_p\subset X$ is an affine space.

This was already proven in <cit.> in a much more general setting,  but for our special case we will present a proof which will result in an explicit description of these cells.

Fix $p\in X^T$. First let us consider $i\in\mathbb{Z}/n\mathbb{Z}$ and we shall study

\begin{equation*}
(\mathcal{C}_p)_i =\{\nu_i\in\proj{}{N-1}:\nu\in\mathcal{C}_p\}.
\end{equation*}

Suppose that $p_i=\langle v_j^l\rangle$. Then $(\mathcal{C}_p)_i$ lies in the BB-cell of the ambient variety $\mathcal{C}_i\subseteq\proj{}{N-1}$ associated to $p_i\in\proj{}{N-1}$. This space is easy to describe

\begin{equation*}
\mathcal{C}_i = \{u\in\mathbb{C}^N:u_j^l\neq0, u_k^s=0\text{ for all }(k,s)\text{ such that }v_k^s< v_j^l\}.
\end{equation*}

Visually, the points of $\mathcal{C}_i$ look as follows

\begin{equation*}
\begin{ytableau}
u_{j_1}^1 & u_{j_1-1}^1 & \none[\dots] & u_{j+1}^1 & 0 & 0 & \none[\dots] \\
u_{j_2}^2 & u_{j_1-1}^1  & \none[\dots] & u_{j+1}^2 & 0 & 0 & \none[\dots] \\
\none[\vdots] & \none[\vdots] & \none & \none[\vdots] & \none[\vdots] & \none[\vdots] \\
u_{j_l}^l & u_{j_l-1}^l &  \none[\dots] & u_{j+1}^l & 1 & 0 & \none[\dots]\\
\none[\vdots] & \none[\vdots] & \none & \none[\vdots] & \none[\vdots] & \none[\vdots] \\
u_{j_M}^M & u_{j_M-1}^M & \none[\dots] & u_{j+1}^M & u_{j}^M & 0 & \none[\dots] \\
\end{ytableau}
\end{equation*}

We can choose $u_j^l=1$ to get a parametrization of $\mathcal{C}_i$ as an affine space. The BB-cell of the quiver Grassmannian is the intersection

\begin{equation*}
\mathcal{C}_p = \left( \prod_{i=0}^{n-1}\mathcal{C}_{i} \right)\cap X.
\end{equation*}

To pass to another vertex we distinguish two cases

* $Jv_i^l=0$. Then for any $u\in\mathcal{C}_i$ we have $Ju=0$ as $v_k^s\ge v_j^l$ implies $Jv_k^s=0$. Thus vectors from $(\mathcal{C}_p)_i$ do not impose any conditions on $(\mathcal{C}_p)_{i+1}$.

* $Jv_{i}^l=v_{i+1}^l$ is non-zero. Then for any $\nu\in \mathcal{C}_p$ we have

\begin{equation*}
\end{equation*}

This simply follows from the fact that $\nu_i$ must be spanned by a vector, which has a non-zero coordinate next to $v_i^l$, which does not go to $0$ under $J$. So vectors in $(\mathcal{C}_p)_{i+1}$ are completely determined by vectors in $(\mathcal{C}_p)_i$.

Therefore if we have a movable part in $p$ running through vertices $i$ through $m$, then $(\mathcal{C}_p)_i=\mathcal{C}_i$ is an affine space and $(\mathcal{C}_p)_{i+k}=J^k(\mathcal{C}_i)$ for all $k=0,\dots,m$. Dimension of $\mathcal{C}_i$ is given by

\begin{equation*}
\dim(\mathcal{C}_i) = |\{ v_j^s:v_j^s>p_i \}|.
\end{equation*}

The whole cell $\mathcal{C}_p$ is the product of either $\mathcal{C}_i$'s, in case of $i$ being a starting vertex of a movable part in $p$, or images of elements of $\mathcal{C}_i$ for the other vertices. Thus the dimension of $\mathcal{C}_p$ is the sum of dimensions of $\mathcal{C}_i$'s for all $i$ that start a movable part in $p$.

Next, we shall focus on the closures of the cells. One of the more crucial results regarding the closed cells is that they are smooth, which is exactly what we shall prove now. This is a miracle of subrepresentation vector being $\mathbf{1}=(1,\dots,1)$. Even for $J$ being a single Jordan blocks if we choose other dimension vectors we get singular closed BB cells as was observed in <cit.>.

Let $p\in X^{T}$ be a fixed point. Then the closure of the BB-cell $\mathcal{C}_p$ is a product of smooth quiver Grassmannians.

We will provide an explicit description of these subvarieties. First consider the case where $p\in X^T$ is a single movable part. Again, denote by $\clBB{i}\subseteq\proj{}{N-1}$ the closed BB-cell associated to the fixed point $p_i$ for $i\in\mathbb{Z}/n\mathbb{Z}$. Each $\mathcal{C}_i$ is isomorphic to a projective space $\proj{}{d_i-1}$ by forgetting all zero's imposed on vectors spanning lines in $\mathcal{C}_i$, so

\begin{equation*}
d_i = |\{v_k^s:v_k^s\ge p_i\}|.
\end{equation*}

Let $R_p\in\text{rep}A_n$ be the representation of the equioriented $A_n$-quiver defined by

\begin{equation*}
R_p: \mathbb{C}^{d_0}\xrightarrow{\pi_0}\mathbb{C}^{d_1}\xrightarrow{\pi_1}\mathbb{C}^{d_2}\to\dots\to\mathbb{C}^{d_n},
\end{equation*}

where $\pi_i:\mathbb{C}^{d_i}\to\mathbb{C}^{d_{i+1}}$ is the map induced from $J$ after forgetting these basis vectors $v_j^s$ such that $v_j^s<p_i$. In particular the $\pi_i$'s are epimorphisms! By forgetting all zeros in the description of $\clBB{p}$ we get an isomorphism

\begin{equation*}
\clBB{p}\cong \qgr{1}{R_p}.
\end{equation*}

Since all arrows in $R_p$ are epimorphisms, then the decomposition of $R_p$ into irreducible components is a decomposition into components that are all supported at the first vertex. As these are all injective, we get that $\text{Ext}^1(R_p,R_p)=0$. From the known theory of acyclic quiver Grassmannians (<cit.>) we get that $\qgr{1}{R_p}$ is smooth, irreducible and of dimension

\begin{equation*}
\dim \qgr{1}{R_p} = \langle \mathbf{1},\mathbf{d}-\mathbf{1}\rangle_{A_{m+1}} = \sum_{i=0}^m(d_i-1) - \sum_{i=0}^{m-1}(d_i-1) = d_m-1.
\end{equation*}

If $p$ is comprised of many movable parts, then since each movable part ends at a vector space $p_i$ such that $Jp_i=0$, we have that $\clBB{p}$ is the product of quiver Grassmannians associated to each movable part alone.

Another crucial property of this BB-decomposition is that the closed cells form a stratification. Since the quiver Grassmannian $X$ has a stratification by affine cells, then it is equivariantly formal, which is one of the ingredients baked into the definition of GKM spaces. It has first been proven in <cit.>.

The BB-decomposition of $X$ forms a stratification, i.e. if for $p,q\in X^T$ we have $\mathcal{C}_p\cap \clBB{q}\neq\emptyset$, then $\mathcal{C}_p\subseteq\clBB{q}$.

This follows immediately from the fact that if we fix $i\in\mathbb{Z}/n\mathbb{Z}$ and pick two fixed points $p,q\in X^T$, then both $p_i$ and $q_i$ are fixed points in $\proj{}{N-1}$ under the $\mathbb{C}^\times$-action defined over the $i$-th vertex of $\Delta_n$. In particular if $\mathcal{C},\mathcal{D}\subseteq\proj{}{N-1}$ are the BB-cells corresponding to $p_i$ and $q_i$ respectively in the ambient space, then

\begin{align*}
& \mathcal{C}\cap X = (\mathcal{C}_p)_i,\\
& \mathcal{D}\cap X = (\mathcal{C}_q)_i.
\end{align*}

These larger BB-cell do form a stratification, so if $\mathcal{C}_p\cap\clBB{q}\neq\emptyset$, then $\mathcal{C}_p\subseteq\clBB{q}$, since we are actually comparing the larger BB-cells.

The partial order on fixed points defined in Definition $\ref{PartialOrder}$ lets us neatly describe the partial order of the closed BB-cells.

For two fixed points $p,q\in X^T$ we have $\clBB{p}\subseteq\clBB{q}$ if and only if $q\le p$.

This is true for the BB-cells of all $\proj{}{N-1}$ as the lexicographic order is the order of weights acting on this space.

When it comes to calculations in cohomology this becomes combinatorially intense, even if we restrict ourselves to just a single Jordan block. This particular case has been studied in <cit.>. We shall give a few observations about this case below.

Suppose that $J$ is a single Jordan block of size $N$. In this case the BB-decomposition simplifies heavily as we do not have to worry about other Jordan blocks. Our one-dimensional torus acts on each $\mathbb{C}^N$ in $\mathcal{M}$ in the standard way.

A fixed point $p\in X^T$ in this case corresponds uniquely to a non-empty subset $I_p\subseteq\mathbb{Z}/n\mathbb{Z}$ – the subset of these vertices, where a movable part in $p$ ends. In other words, we have

\begin{equation*}
I_p = \{i\in\mathbb{Z}/n\mathbb{Z}:Jp_i=0\}.
\end{equation*}

Subsets that define a fixed point must be non-empty and the difference of two consecutive points in such a subset must be at most $N$. The partial order from Definition $\ref{PartialOrder}$ simplifies to the usual order on the power set of $\mathbb{Z}/n\mathbb{Z}$.

The open BB-cells are, of course, affine spaces; their dimension obtained from Proposition $\ref{BBCellsAffine}$ simplifies to

\begin{equation*}
\dim \mathcal{C}_p = |I_p^c|.
\end{equation*}

Movable parts in fixed points are parametrized by fixing a vertex (for example fixing the ending vertex of a segment) and length.

Observe that in this case we can restrict ourselves only to the case where $N\le n$.

Let $\mathcal{M}_N$ be the representation of $\Delta_n$, which has the same matrix over each arrow $J$ that is a single Jordan block of size $N$. If $N\ge n$, then we have

\begin{equation*}
\qgr{1}{\mathcal{M}_N}\cong\qgr{1}{\mathcal{M}_n}.
\end{equation*}

If $N\le N'$, then there exists an embedding

\begin{equation*}
\qgr{1}{\mathcal{M}_N}\hookrightarrow\qgr{1}{\mathcal{M}_{N'}}.
\end{equation*}

To see the first part let us assume that $M\in\qgr{1}{\mathcal{M}_N}$ with $N\ge n$ and fix a vertex $i\in\mathbb{Z}/n\mathbb{Z}$. Let $M_i=\langle u\rangle$ and define

\begin{equation*}
j = \min(k:u_k\neq 0).
\end{equation*}

After applying $J$ to $u$ $n$ times we arrive at the condition $J^nu\in\langle u \rangle$, note that either $j+n>N$, in which case $J^nu=0$ or $j+n\le N$ and then

\begin{equation*}
\min(k:(J^nu)_k\neq0) = j+n.
\end{equation*}

For the condition $J^nu\in\langle u\rangle$ to be met we must thus have $J^nu=0$ and this is equivalent to $u$ having $0$ as its first $N-n$ coordinates. Thus any element of $\qgr{1}{\mathcal{M}_N}$ is in fact a subrepresentation of $\mathcal{M}_n$.

For $N\le N'$ the embedding is simply given by the map $\mathcal{M}_N\to\mathcal{M}_{N'}$, which is induced by the map $\mathbb{C}^N\to\mathbb{C}^{N'}$ that embeds $\mathbb{C}^N$ into the higher dimensional space by putting an appropriate amount of zeroes at the start.

We can endow $X^{T}$ with a (commutative) semigroup structure by letting $p\cap q$ be the fixed point defined by the union of subsets

\begin{equation*}
I_{p\cap q} := I_p \cup I_q\subseteq\mathbb{Z}/n\mathbb{Z},
\end{equation*}

where $I_p,I_q\subseteq\mathbb{Z}/n\mathbb{Z}$ are the subsets corresponding to $p$ and $q$ respectively. Observe that for all $i\in\mathbb{Z}/n\mathbb{Z}$ we have

\begin{equation}\label{CapOfPoints}
(p\cap q)_i=\min(p_i,q_i),
\end{equation}

where $\min(p_i,q_i)$ is the point in $\proj{}{N-1}$ spanned by the vector that is lesser of vectors spanning $p_i$ and $q_i$.

This construction lets us describe the intersections of the closed BB-cells

Let $p,q\in X^{T}$ be two fixed points. Then the following formula holds

\begin{equation*}
\clBB{p}\cap\clBB{q} = \clBB{p\cap q}.
\end{equation*}

This is immediate when we consider BB-cells on $\proj{}{N-1}$ for fixed $i\in\mathbb{Z}$, since we have $\eqref{CapOfPoints}$.

§ GKM STRUCTURE

As we have already recalled in Section 3, the $T$-variety $X$ is a GKM variety. In the traditionally defined GKM graph the set of fixed points $X^T$ is the set of vertices and the set of one-dimensional orbits is the set of edges. This graph is not oriented, but we can orient it in our case using the $\mathbb{C}^\times$-action on $X$, as in <cit.>. We give a precise definiton below.

Let $X$ be a GKM variety with respect to a fixed $T$-action and ${\chi:\mathbb{C}^\times\to T}$ be a cocharacter such that the induced $\mathbb{C}^\times$-action on $X$ via $\chi$ has the same fixed point set

\begin{equation*}
X^T = X^{\mathbb{C}^\times}.
\end{equation*}

Let $x,y\in X^T$ be two fixed points such that they are connected by an edge, i.e. we have a one-dimensional orbit $O$ such that $x,y\in\overline{O}$. We orient this edge as $x\to y$ if for any (and thus all) $\nu\in O$ we have

\begin{equation*}
\lim_{\lambda\to0}\lambda\cdot \nu = x,
\end{equation*}

where $\lambda\in\mathbb{C}^\times$ acts on $\nu$ via $\chi$.

In particular if an arrow $p\to q$ exists, then for the orbit $O$ that induces this arrow we have $O\subseteq\mathcal{C}_p$. In <cit.> there is a description of one-dimensional orbits of any quiver Grassmannian endowed with their torus action and in our case this description becomes simpler.

We can encode movable parts using Young tableaux. If $\mathcal{J}$ is the Young tableau of shape corresponding to the partition corresponding to $J$, then a movable part $S$ of length $m$ and from the $l$-th block is a coloring of $\mathcal{J}$, where we color in first $m+1$ boxes in the $l$-th row in $\mathcal{J}$ and leave the rest blank.

Consider $J=J_4\oplus J_3\oplus J_3\oplus J_2\oplus J_1$ and a movable part of length $1$ in the second block. Its coloring is

\begin{equation*}
\begin{ytableau}
\, & \, &  & \, \\
*(gray) & *(gray) & \, \\
\, & \, & \, \\
\, & \, \\
\,
\end{ytableau}
\end{equation*}

These colorings let us visualize fundamental mutations (defined in <cit.>), i.e. edges in the GKM graph of $X$.

Let $S$ be a movable part of length $m$ in some fixed point and $\lambda$ be the coloring of $\mathcal{J}$ associated with $S$. A mutation of $S$ is either

* A single coloring $\lambda'$, whose colored segment is the same length as in $\lambda$, but in a lower row, or

* Two colorings $\lambda_1,\lambda_2$, where $\lambda_1$ is a coloring of the same row and length $m'$ and $\lambda_2$ is a coloring of any row and first $m''$ elements so that

\begin{equation*}
\end{equation*}

In other words, we split the colored boxes in $\lambda$ into two colorings that correspond to movable parts, with the restriction that at least one movable part must be from the same block as $\lambda$.

Consider the movable part from Example $\ref{YoungTableauExample}$. Then there are two possible mutations of Type 1. Namely, we have
baseline, boxsize=1em

\begin{align*}
\begin{ytableau}
\, & \, &  & \, \\
\, & \,& \, \\
*(gray) & *(gray) & \, \\
\, & \, \\
\,
\end{ytableau} && \begin{ytableau}
\, & \, &  & \, \\
\, & \, & \, \\
\, & \, & \, \\
*(gray) & *(gray) \\
\,
\end{ytableau}
\end{align*}

There are also five possible mutations of Type 2.
baseline, boxsize=1em

\begin{gather*}
\left( \, \begin{ytableau}
\, & \, &  & \, \\
*(gray) & \, & \, \\
\, & \, & \, \\
\, & \, \\
\,
\end{ytableau},\begin{ytableau}
*(gray) & \, &  & \, \\
\, & \,& \, \\
\, & \, & \, \\
\, & \, \\
\,
\end{ytableau} \, \right) , \left( \, \begin{ytableau}
\, & \, &  & \, \\
*(gray) & \, & \, \\
\, & \, & \, \\
\, & \, \\
\,
\end{ytableau},\begin{ytableau}
\, & \, &  & \, \\
*(gray) & \,& \, \\
\, & \, & \, \\
\, & \, \\
\,
\end{ytableau} \, \right) , \left( \, \begin{ytableau}
\, & \, &  & \, \\
*(gray) & \, & \, \\
\, & \, & \, \\
\, & \, \\
\,
\end{ytableau},\begin{ytableau}
\, & \, &  & \, \\
\, & \,& \, \\
*(gray) & \, & \, \\
\, & \, \\
\,
\end{ytableau} \, \right) \\
\, \\
\left( \, \begin{ytableau}
\, & \, &  & \, \\
*(gray) & \, & \, \\
\, & \, & \, \\
\, & \, \\
\,
\end{ytableau},\begin{ytableau}
\, & \, &  & \, \\
\, & \,& \, \\
\, & \, & \, \\
*(gray) & \, \\
\,
\end{ytableau} \, \right) , \left( \, \begin{ytableau}
\, & \, &  & \, \\
*(gray) & \, & \, \\
\, & \, & \, \\
\, & \, \\
\,
\end{ytableau},\begin{ytableau}
\, & \, &  & \, \\
\, & \,& \, \\
\, & \, & \, \\
\, & \, \\
\end{ytableau} \, \right)
\end{gather*}

One can imagine the colored boxes as, for example, a stack of cards in a game of solitaire. Mutations correspond to either moving the whole stack "downwards" or making two stacks, but we can put the new stack wherever we want (we can put the stacks on top of each other in this scenario). These actions characterize exactly the edges of the GKM graph of $X$ as it is illustrated by the following proposition.

Let $p,q\in X^T$ be two fixed points. Then an arrow $p\to q$ in the GKM graph of $X$ exists if and only if $q$ is obtained from $p$ by a mutation of exactly one movable part in $p$.

Before the proof we shall give an example of a GKM graph.

Let $J$ be a single Jordan block of size $3$ and suppose we have $n=3$ vertices. We will denote the fixed points by the subsets of $\{0,1,2\}$ that they define. Mutations in this case can only correspond to the action of dividing movable parts into two, which on the level of subsets correspond to adding a single element to the subset defining a fixed point. The GKM graph of $X$ in this case turns out to be

\begin{equation*}
\resizebox{0.9\textwidth}{!}{
\begin{tikzcd}[ampersand replacement=\&]
\&\&\&\&\& {\{2\}} \\
\\
\&\&\& {\{0,2\}} \&\&\&\& {\{1,2\}} \\
\&\&\&\&\& {\{0,1,2\}} \\
\\
{\{0\}} \&\&\&\&\& {\{0,1\}} \&\&\&\&\& {\{1\}}
\arrow["{t_2+2t_0-t_1}", from=1-6, to=3-4,sloped]
\arrow["{t_1+t_0-t_2}", from=6-1, to=3-4,sloped]
\arrow["{t_3+2t_0-t_2}"', from=6-1, to=6-6,sloped]
\arrow["{t_3+t_0-t_1}", from=1-6, to=3-8,sloped]
\arrow["{t_1+2t_0-t_3}" , from=6-11, to=3-8,sloped]
\arrow["{t_2+t_0-t_3}"', from=6-11, to=6-6,sloped]
\arrow["{t_1+t_0-t_2}"', from=6-6, to=4-6,sloped]
\arrow["{t_3+t_0-t_1}"', from=3-4, to=4-6,sloped]
\arrow["{t_2+t_0-t_3}", from=3-8, to=4-6,sloped]
\end{tikzcd}
\end{equation*}

For two fixed points $p,q\in X^T$ let $p+q$ be the subrepresentation given by

\begin{equation*}
(p+q)_i = \langle v_{i}^l+v_j^s \rangle,
\end{equation*}

where $i\in\mathbb{Z}/n\mathbb{Z}, p_i=\langle v_i^l\rangle$ and $q_i=\langle v_j^s\rangle$.

It is easy to see that if we assume the conditions on $I_p$ and $I_q$, then $p_i=q_i$ for all vertices besides those that correspond to the movable part in $p$ that gets mutated to create $q$. On this movable part we have $p_i<q_i$. To see the arrow $p\to q$, we just take the orbit $T\cdot(p+q)$. The representation $p+q$ has one non-zero coordinate everywhere besides one movable part in $p$, so the orbit of this element is one-dimensional and

\begin{align*}
&\lambda\cdot (p+q) \xrightarrow[]{\lambda\to 0}p,
&\lambda\cdot(p+q) \xrightarrow[]{\lambda\to\infty}q.
\end{align*}

Now suppose that there exists an orbit $O\subseteq X$ such that $p,q\in\overline{O}$ and $O\subseteq\mathcal{C}_p$. As for any $\nu\in O$ we have $\lambda\cdot \nu\xrightarrow[]{\lambda\to\infty}q$, then we must have $\clBB{q}\subseteq\clBB{p}$ and by Theorem $\ref{OrderOfBBCells}$ we have $p_i\le q_i$ for all $i$. Let us pick an element $\nu\in O\subseteq \mathcal{C}_p$. By the explicit description of the BB-cells in Proposition $\ref{BBCellsAffine}$ we know that $\nu$ is determined by vector spaces put over the vertices which are starting movable parts in $p$. As the orbit $O=T\cdot \nu$ has to be one-dimensional, then we can only have one movable part in $p$ over which $q$ differs from $p$, since we would have an orbit of larger dimension otherwise. We can therefore assume that $p$ is a single movable part, say starting at the $i$-th vertex. The whole $T$-orbit is purely determined by what happens at the $i$-th vertex, since lines in other vertices are just images by $J$. This reduces the case to $\proj{}{N-1}$. We know the $1$-dimensional orbits there – in the GKM graph of $\proj{}{N-1}$ under the induced torus action an edge $p_i\to q_i$ exists if and only if $p_i< q_i$. Let $p_i=\langle v_j^l\rangle, q_i=\langle v_k^s\rangle$ and $\lambda_p,\lambda_q$ be colorings of $\mathcal{J}$ associated to $p$ and $q$ respectively. We can distinguish two cases

* If $j=k$, then we must have $l<s$. In this case $\lambda_q$ came from $\lambda_p$ by moving the whole colored row in $\lambda_p$ down.

* If $j<k$, then $\lambda_q$ came about from dividing the colored row in $\lambda_p$ into two parts. We split this row at the $(i+j_s-j)$-th vertex.

We also need to describe the labels of the edges of the GKM graph, which is done below.

Suppose that in the GKM graph of $X$ there exists an edge $p\to q$ between two fixed points $p,q\in X^T$. Then the label of this edge can be chosen to be the following polynomial

\begin{equation*}
w(p,q) = t_{m+j,l} + (j_l-j_k+i-m)t_0 - t_{i+j,l},
\end{equation*}

where $q$ comes by mutating a movable part in $p$ that starts at the $j$-th vertex, is of length $m$ and comes from the $l$-th block. The augmentation occurs at the $(i+j)$-th vertex.

We need the weight of the $T$-action on which we act on the $\proj{}{1}$ that is equiavariantly isomorphic to the closed orbit $\overline{T\cdot(p+q)}$. The isomorphism $\overline{T\cdot(p+q)}\to\proj{}{1}$ is given by looking at the vector space lying over the starting point of the movable part in $p$ which we augment to obtain $q$.

Suppose first that this movable part in $p$ starts at the $0$-th vertex, is of length $m$ and comes from the $l$-th block; in particular $p_i=\langle v_{j_l-m}^l\rangle$. Suppose further that $q_i=\langle v_{j_k-i}^k$, i.e. the augmentation occurs at the $i$-th vertex and at one of the movable parts is from the $k$-th block. The torus acts on these vectors with weights

\begin{align*}
& w_1 = t_0^{j_l-m-1}t_{m,l},\\
& w_2 = t_0^{j_k-i-1}t_{i,l}.
\end{align*}

So the torus acts on the closed orbit identified with $\proj{}{1}$ with weight

\begin{equation*}
w = t_{m,l} + (j_l-j_k+i-m)t_0 - t_{i,l}.
\end{equation*}

The above polynomial is also a weight in the torus representation $T_q\clBB{p}$. Now, if the movable part in $p$ that we are augmenting starts at the $j$-th vertex, then we have to translate everything by $j$:

\begin{equation*}
w(p,q) = t_{m+j,l} + (j_l-j_k+i-m)t_0 - t_{i+j,l}.
\end{equation*}

This is the label of the edge $p\to q$ in the GKM graph of $X$ and a weight in the torus representation $T_q\clBB{p}$.

There is a natural $(\mathbb{Z}/n\mathbb{Z})$-action on $X$, which comes from rotating the quiver along the direction given by the arrows in $\Delta_n$. Since the $\mathbb{C}^\times$-action defined by Formula $\eqref{C-action}$ is independent of the vertex over which we define this action, then the $\mathbb{Z}/n\mathbb{Z}$ is immediately seen to lift to the set of fixed point $X^T$ as well as the set of (closed) BB-cells. The $T$-action on $X$ commutes with the $(\mathbb{Z}/n\mathbb{Z})$-action up to a twist by $\rho:T\to T$ and so, similarly as is the case with Schubert classes in the usual Grassmannians, this gives a particular symmetry on the GKM-graph of $X$. Namely, if $\alpha\in H_T^*(X)$ is a class, then for the generator $\tau\in\mathbb{Z}/n\mathbb{Z}$ we can set

\begin{equation}\label{Z/nZ-ActionOnGraph}
(\tau\cdot\alpha)|_p = \alpha|_{\tau(p)}\circ \rho,
\end{equation}

where $\rho:H_T^*(*)\to H_T^*(*)$ is the map induced by $\rho:T\to T$ defined by $\eqref{Rotation}$.

With this section concluded we are ready to delve into the analysis of the multiplcative structure of $H_T^*(X)$.

§ MULTIPLICATIVE STRUCTURE OF EQUIVARIANT COHOMOLOGY

This last section is devoted to analyzing the multiplicative structure of the equivariant cohomology ring $H_T^*(X)$. M. Lanini and A. Pütz provide a construction of the so called Knutson-Tao basis for equivaraint cohomology <cit.> for certain quiver Grassmannians. This basis is dependent on a chosen orientation of the GKM-graph of $X$ using cocharacters as we recalled in Definition $\ref{Orientation}$. We give a precise definition below
Let $X$ be any GKM variety under a fixed $T$-action and $\chi:\mathbb{C}^\times\to T$ be a cocharacter such that the induced $\mathbb{C}^\times$-action by $\chi$ preserves fixed points $X^T=X^{\mathbb{C}^\times}$. This induced $\mathbb{C}^\times$-action determines an orientation on the GKM graph of $X$ as in Definition $\ref{Orientation}$. A set $\{q^x:x\in X^T\}$ is called a Knutson-Tao basis if its elements satisfy the following

* For each $y\in X^T$ the polynomial $q^x|_y$ is either $0$ in the case that there is no oriented path from $y$ to $x$ or is homogeneous of degree $\deg(q^x|_x)$ in the case where there is such a path.

* The restriction $q^x|_x$ is the product of weights associated to edges going from $x$ in the GKM-graph of $X$.

If a Knutson-Tao basis exists, then it is a basis of $H_T^*(X)$ as an $H_T^*(*)$-module <cit.>. If our GKM variety $X$ is Palais-Smale, meaning that $X$ satisfies hypothesis of Definition $\ref{Knutson-TaoBasis}$ and for each edge $x\to y$ in the GKM graph of $X$ the amount of edges going out of $x$ is strictly higher than the amount of edges going out of $y$, then this basis is unique <cit.>.

This basis behaves well with respect to the order on $X^T$ given by paths in the graph. In our case this order coincides with the inverted lexicographic order. These basis' arose from the Schubert basis for equivariant cohomology of the usual Grassmannians in <cit.>. Similarly as for flag varieties and usual Grassmannians, a central question regarding these basis is how they multiply. Since $\{q^x:x\in X^T\}$ is a basis of $H_T^*(X)$ as a $H_T^*(*)$-module, then computation of the structure constants

\begin{equation*}
q^x \cdot q^y = \sum_{z\in X^T}c_{x,y}^zq^z
\end{equation*}

determines the multiplicative structure of $H_T^*(X)$, considered as a $H_T^*(*)$-algebra.

In our case each cell $\clBB{x}$ gives an equivariant fundamental class $[\clBB{x}]\in H_{*}^T(X)$ and the set of fundamental classes is a basis of equivariant homology. By Theorem $\ref{ClBBCellsSmooth}$ we know the closed BB cells are smooth and so integration along such a cell is much simpler to grasp. Using this pairing we can get the dual basis $\{p^x\in H_T^*(X):x\in X^T\}$, i.e. characterized by

\begin{equation*}
\int_{\clBB{y}}p^x|_{\clBB{y}} = \delta_{x,y}.
\end{equation*}

Using Atiyah-Bott, Berline-Vergne we can further rewrite the above formula as

\begin{equation*}
\int_{\clBB{y}}p^x|_{\clBB{y}} = \sum_{z\in X^T:z\in\clBB{y}}\frac{p^x|_z}{e(T_z\clBB{y})} = \delta_{x,y},
\end{equation*}

where $e(T_z\clBB{y})$ is the product of weights of the torus representation $T_z\clBB{y}$. These products of weights can be read directly from the GKM graph of $X$. Namely, for a fixed point $x\in X^T$ we first consider the subgraph of $X$ spanned by all oriented paths starting from $x$. For any $y\in X^T$ such that $y\in\clBB{x}$ we have that the vertex corresponding to $y$ lies in this subgraph. To get the Euler class of the torus representation $T_y\clBB{x}$ we take the product of labels of edges either from or into $y$ in this subgraph. However, one has to be careful with the sign of these labels – normally, labels of edges in the GKM graph are defined up to a scalar multiple and since we need weights of the torus representation $T_y\clBB{x}$, then one has to take the appropriately scaled generator of the ideal that generates labels of edges in the GKM graph of $X$; in Proposition $\ref{GKMedges}$ we give the appropriately scaled labels.
Take $n=3$ and $J=J_3$. Then there are $7$ fixed points associated to non-empty subsets of $\{0,1,2\}$. The GKM graph of $X$ in this case is given in Example $\ref{GKMGraphn=3}$. The Knutson-Tau basis of $H_T^*(X)$ consists of three types of elements:

* The identity $1\in H_T^*(X)$, dual to the smallest cell $\clBB{\{0,1,2\}}$.

* Three elements dual to the medium-sized cells, i.e. those corresponding to two-element subsets

\begin{align*}
& p^{\{0,1\}}=(t_1+t_0-t_2,t_1+2t_0-t_3,0,t_1+t_0-t_2,0,0,0),\\
& p^{\{1,2\}}=(0,t_2+t_0-t_3,t_2+2t_0-t_1,0,t_2+t_2-t_3,0,0),\\
& p^{\{0,2\}}=(t_3+2t_0-t_2,0,t_3+t_0-t_1,0,0,t_3+t_0-t_1,0),
\end{align*}

where the order follows the following order on fixed points

\begin{equation*}
\end{equation*}

* Three elements dual to the largest cells:

\begin{align*}
& p^{\{0\}}=((t_2-2t_0-t_3)(t_2-t_0-t_1),0,\dots,0),\\
& p^{\{1\}}=(0,(t_3-2t_0-t_1)(t_3-t_0-t_2),0,\dots,0),\\
& p^{\{2\}}=(0,0,(t_1-2t_0-t_2)(t_1-t_0-t_3),0,\dots,0).
\end{align*}

From simple linear algebra one can immediately observe that this dual basis always is a Knutson-Tao basis. Authors in <cit.> define a family of quiver Grassmannians for which a Knutson-Tao basis exists (and is unique); our GKM space $X$ is a member of this family.

Elements of this dual basis are homogeneous (we grade $H_T^*(X)$ in the usual fashion, i.e. $\deg t_0=\deg t_{i,j}=2$ for all $i,j$); degree of $p^x$ is given by the dimension of the cell it corresponds to

\begin{equation*}
\deg(p^x) = 2\dim(\clBB{x}).
\end{equation*}

By definition of the dual basis $p^x$, the structure constants are given by integrals

\begin{equation*}
c_{x,y}^z = \int_{\clBB{z}}p^x|_{\clBB{z}}\cdot p^y|_{\clBB{z}} = \sum_{s\in X^T: \clBB{x}\cup\clBB{y}\subseteq\clBB{s}\subseteq \clBB{z}}\frac{p^x|_s\cdot p^y|_s}{e(T_s\clBB{z})},
\end{equation*}

We can outline that the basis $\{p^x:x\in X^T\}$ satisfies the same symmetry as the closed BB-cells that these classes are dual to, i.e. if $\tau\in\mathbb{Z}/n\mathbb{Z}$ is the generator, then

\begin{equation*}
p^{\tau(x)} = \tau(p^x),
\end{equation*}

where $\tau$ acts on fixed point by rotation of the quiver and on $H_T^*(X)$ by Formula $\eqref{Z/nZ-ActionOnGraph}$.

Missing 'biblatex' package
The bibliography requires the 'biblatex' package.

Cambridge University Press

titleEquivariant cohomology in algebraic geometry

Cambridge University Press

titleElements of the representation theory of associative algebras

journaltitleJ. Algebraic Combin.
titleQuiver Grassmannians associated with string modules

Amer. Math. Soc.

booktitleRepresentation theory and beyond
seriesContemp. Math.
titleThree lectures on quiver Grassmannians

journaltitleJ. Pure Appl. Algebra
titleOn the quiver Grassmannian in the acyclic case

journaltitleCanad. J. Math.
titleTotally nonnegative Grassmannians, Grassmann necklaces, and quiver Grassmannians

journaltitleDuke Math. J.
titlePuzzles and equivariant cohomology of Grassmannians

journaltitleAlgebra Number Theory
titleGKM-theory for torus actions on cyclic quiver Grassmannians

journaltitleJ. Algebraic Combin.
titlePermutation actions on quiver Grassmannians for the equioriented cycle via GKM-theory

journaltitlePacific J. Math.
titleDesingularizations of quiver Grassmannians for the equioriented cycle quiver

journaltitleAlgebras and Representation Theory
titleEvery projective variety is a quiver Grassmannian

Proceedings of the London Mathematical Society

titleGeneral representations of quivers

journaltitleAmer. J. Math.
titlePermutation representations on Schubert varieties


# Multi-View Object Pose Refinement with Differentiable Renderer

Ivan Shugurov∗,1,2, Ivan Pavlov∗,1,2, Sergey Zakharov1,2,3 and Slobodan
Ilic1,2 Manuscript received: October, 15, 2020; Revised: January, 14, 2021;
Accepted: February, 8, 2021.This paper was recommended for publication by
Editor Markus Vincze upon evaluation of the Associate Editor and Reviewers’
comments.)1Department of Informatics, Technical University of Munich, Germany
<EMAIL_ADDRESS>AG, Munich, Germany3Currently at Toyota Research
Institute, Los Altos, USA∗These authors contributed equally to the
work.Digital Object Identifier (DOI): 10.1109/LRA.2021.3062350

###### Abstract

This paper introduces a novel multi-view 6 DoF object pose refinement approach
focusing on improving methods trained on synthetic data. It is based on the
DPOD detector, which produces dense 2D-3D correspondences between the model
vertices and the image pixels in each frame. We have opted for the use of
multiple frames with known relative camera transformations, as it allows
introduction of geometrical constraints via an interpretable ICP-like loss
function. The loss function is implemented with a differentiable renderer and
is optimized iteratively. We also demonstrate that a full detection and
refinement pipeline, which is trained solely on synthetic data, can be used
for auto-labeling real data. We perform quantitative evaluation on LineMOD,
Occlusion, Homebrewed and YCB-V datasets and report excellent performance in
comparison to the state-of-the-art methods trained on the synthetic and real
data. We demonstrate empirically that our approach requires only a few frames
and is robust to close camera locations and noise in extrinsic camera
calibration, making its practical usage easier and more ubiquitous.

###### Index Terms:

Object Detection, Perception for Grasping and Manipulation; Deep Learning for
Visual Perception

##  I Introduction

Object detection and 6D pose estimation in RGB images are among the most
fundamental problems in computer vision, with applications encompassing
autonomous driving, augmented reality and robotics. A large body of work has
already been presented, but recent advances in deep learning have opened up
new horizons for RGB-based algorithms, which have now started to dominate the
field. However, precise 6 DoF pose estimation, required, in autonomous robotic
grasping systems, for example, still remains a challenging problem. This is
mainly due to the perspective ambiguity, lightning changes, clutter and
occlusions. Current industrial implementations [1] are not based on deep
learning and rely on depth data for increased pose accuracy. Moreover, these
approaches use 3D models directly, while deep learning methods still struggle
with the domain gap when trained on synthetic data rendered from 3D CAD
models. Recently, CosyPose [2], a deep learning method that operates on RGB
images, assumed the lead over traditional methods in the BOP challenge[3],
especially when multiple images are used for pose refinement.

The task of pose refinement has also been addressed by deep learning
approaches. Methods like [4, 5, 6] trained CNN networks on pairs of images,
the aim being to learn to predict the pose offset between the predicted object
pose and the observed detected object. There, 3D models were rendered in the
predicted pose and real images were given as patches with the detected object.
The main disadvantage of these refiners is that the are dependent on the
correct pose error priors used during training coupled with the necessity to
re-train them for each new object in order to obtain high-quality results.

We address the aforementioned problems in this paper by introducing a multi-
view refinement procedure. We first train the DPOD detector [4] on synthetic
data to overcome the dataset bias and the lack of real training data. We then
exploit geometric multi-view constraints during refinement to cope with
perspective ambiguities and occlusions in monocular RGB images that cause
their sub-par performance. Compared to existing deep learning refiners [4, 5,
6], the proposed approach has the following advantages: 1) it does not require
training of the refiner itself; 2) it is not object-specific; 3) it can be
applied to arbitrary many images without the need for any modification; 4) it
has explicit geometric constraints and, thus, an explicit objective function
which is optimized during the refinement.

Figure 1: Multi-view inference and pose optimization. 1) Inputs to the
algorithm are an unordered set of images and corresponding relative camera
transformations. 2) YOLO is applied to each image separately to detect the
object of interest in each of them. 3) Dense correspondences are predicted
with the DPOD network. 4) Rough object pose in the reference frame is
estimated using PnP and RANSAC using the predicted 2D-3D dense
correspondences. 5) The final pose is iteratively refined using the multi-view
optimization based on differentiable rendering.

The proposed pose refinement procedure is based on differentiable renderering.
It uses multiple views to add relative camera poses as constraints to the pose
optimization procedure. It is done by comparing predicted and rendered dense
correspondences in each frame and then transmitting the error back though the
differentiable renderer to update the pose. The proposed loss function allows
varying numbers of frames to be used without any changes to the optimization
procedure. The loss formulation also does not impose any restrictions on where
in the 3D world the cameras are placed and whether or not views from different
cameras overlap as long as the object is visible in the images. We assume the
availability of relative camera poses. In practice, they can be easily
obtained by a number of various methods, such as placing the object on the
markerboard and either using an actual multi-camera system or using a single
camera but moving the markerboard. In the scenario of robotic grasping, a
camera can be mounted on the robotic arm to enable observation of the object
from several viewpoints.

We evaluate this approach on LineMOD [7], Occlusion [8], Homebrewed [9] and
YCB-V [10] datasets and report performance that is superior to any related
method trained on synthetic data and similar to or better than methods which
use real training data and post-refinement. Our experiments show that our
approach can robustly perform multi-view pose refinement even when relative
poses are imprecise. Our results also demonstrate that the proposed refinement
remains effective even in degenerate cases, when cameras are in close
proximity to each other.

Further, we show that our framework can be used for the task of auto-labeling
real data as in [11], thus removing the need for manual pose labeling. The
networks are first trained on synthetic data and then used to label the real
images. This procedure enables a considerable reduction in the time and effort
needed to annotate the data. Our multi-view refinement pipeline enables us to
automatically produce high quality pose annotations for these real images and
re-train the detector on them.

##  II Related Work

This section first discusses the state of the art of monocular pose
estimation, before going on to review popular pose refinement techniques and
discuss methods that use multiple images for object detection and pose
estimation.

(a) Refinement with two views.

(b) Refinement with four views.

Figure 2: Example refinement results on the Homewbrewed dataset [9]. The top
row shows initial per-frame poses produced by PnP before refinement, while the
bottom row shows them after refinement. The outline of the object is
visualized in green for the ground truth pose, and in blue for the estimated
pose. This illustrates that the proposed refiner is capable of selecting a
reference frame with a good initial pose and refining it even in the presence
of occlusions and imprecise correspondences or when some of the initial pose
hypotheses are completely incorrect, as in (b).

6 DoF pose estimation SSD-6D [12] was one of the first deep learning methods
for 6 DoF pose estimation from RGB images. SSD-6D extends the off-the-shelf
SSD object detector and was fully trained on synthetic data. Since the method
is based on learning discrete poses, it requires refinement to obtain
reasonable results. Subsequent approaches predicted a fixed number of 2D
keypoints, whose locations on the 3D model are known. Using those 2D-3D
correspondences, the full pose is estimated using a PnP algorithm. BB8[13]
proposed predicting 2D projections of the 3D bounding box corners. In
contrast, PVNet [14] used per-pixel Hough voting to allow all object’s pixels
to vote for the few keypoints lying on the object. HybridPose [15] extended
correspondence prediction by predicting edge vectors and symmetry
correspondences, which were then used jointly in an augmented PnP algorithm.
The concept of 2D-3D correspondence estimation was further extended in the
methods that rely on dense correspondences: DPOD [4], Pix2Pose [16], CDPN [17]
and EPOS [18]. They predicted a 3D correspondence for each foreground object
pixel, thus increasing the number of correspondences. AAE [19] used an off-
the-shelf 2D object detector trained on real data and an autoencoder, trained
on sythetic data, followed by feature matching to predict discrete object
poses. PoseCNN [10] relied on Hough voting to locate the 2D projection of
object’s center and distance from the object to estimate the translation.
Rotation was directly regressed as a quaternion.

Of the aforementioned methods, only SSD6D, DPOD, AAE and EPOS reported results
on synthetic train data in the original papers, though for some of them the
results on synthetic data were later submitted to the BOP challenge [3].

6D pose refinement. Object pose refinement has been studied thoroughly in the
past. Iterative Closest Point [20] (ICP) is one of the most classical and used
approaches. ICP refines a given pose iteratively by establishing one-to-one
correspondences between the point cloud and object’s vertices based on their
spatial proximity and then minimizing distances between them. There are
various variants of ICP and ICP-like algorithms, but their common fundamental
limitation is that they require reliable registered depth information, which
is not always readily available. It restricts their applicability in many
scenarios, for example, on images from mobile phones or from edge devices.
Another line of research attempts to replace depth-based ICP with deep
learning on point clouds. The most relevant example is DenseFusion [21], which
first applies a convolutional neural network to segment out the objects of
interests. It then fuses RGB and depth features to predict the pose. A
trainable iterative deep refiner is applied subsequently to directly regress
the pose offset.

In RGB images, the object pose has been traditionally refined using edge
alignment. The idea is to align the edges of the object rendered in the pose
hypothesis with the edges observed in the image. This approach is very
sensitive to the image quality, clutter, occlusions and to the way in which
the edge correspondences are computed. Later, deep learning pose refinement
approaches [4, 5, 6] operating on RGB images were introduced. The idea here is
to use an external pose estimation algorithm to obtain the initial pose
approximation. Then, the object is rendered in the predicted pose. A rendered
image and a given input RGB image are then put into another convolutional
neural network, which directly regresses a pose offset. In BB8 [13], on the
other hand, the refiner was trained to update predicted locations of
keypoints.

Disadvantages of the aforementioned deep learning-based refiners are their
dependence on the correct pose error priors used during training and the need
to retrain them for each new object in order to obtain high-quality results.

Object pose estimation and refinement from multiple views. This topic has been
studied in the past in several works, most notably in [22, 23, 24]. The papers
display two main trends. First, they all rely on independent pose hypothesis
prediction for each monocular image. Secondly, relative camera transformations
between frames are assumed to be known beforehand. Known camera
transformations are used to fuse pose predictions from several views in the
global coordinate system. Then, either the pose hypothesis that aligns the
best with the other hypotheses is chosen or poses are refined to align better
in the 3D space. Orthogonal to those methods, are the approaches proposed in
[25, 2]. In [25], a set of uncalibrated RGB images is used to create a scale-
ambiguous 3D reconstruction. Then, objects are detected in the 3D
reconstruction that correspond to joint detection in all the frames. Despite
its good performance, the method had the downside that it comprised a number
of complicated and time-consuming steps and the need to use a large number
(72) of frames to obtain reliable reconstructions. CosyPose [2] also relies on
independent pose hypotheses from each frame. However, in contrast to [22, 23,
24], they are then used in a RANSAC scheme to match pose hypotheses from
several frames and produce a unified object-level scene reconstruction and
approximate relative camera poses. They are jointly optimized by minimizing
the multi-view reprojection error. Unfortunately, each of these methods is
evaluated on different datasets, and different pose metrics are reported,
which makes it impossible to compare them either with each other or with the
recent pose estimation papers. We compare our refiner to CosyPose on
Homebrewed [9] and YCB [10] datasets.

##  III Proposed Method

The complete inference pipeline is shown in Figure 1. The proposed method is
divided into the following steps. Step 1: takes a set of images together with
their known intrinsic parameters and relative transformations between cameras;
Step 2: The objects of interest in each image are detected separately; Step 3:
The per-object per-pixel 2D-3D correspondences are predicted independently for
each detected object; Step 4: The object 6 DoF pose is estimated with EPnP
[26] and RANSAC using the predicted 2D-3D correspondences; Step 5: The initial
rough pose is iteratively refined to find a pose which better aligns with
predicted dense correspondences in all the frames. This is done by defining a
loss function over the predicted correspondences and the ideal 2D-3D
correspondences which correspond to the object in the given pose. These ideal
correspondences are produced with a differentiable renderer so that the entire
multi-view alignment procedure is differentiable. We will now describe each
step in more detail. However, the main contribution of the paper comprises the
multi-view refinement in Step 5. In all experiments, we used the Soft
Rasterizer renderer [27].

###  III-A Object Detection and Pose Estimation

As the focus of the paper is on refinement and not on the whole pipeline, we
have opted for one of the already available dense correspondence-based
detectors. We slightly extended the DPOD [4] and trained it on synthetic
training data (later referred to as PBR data) provided by the BOP challenge
[3]. We separated the original DPOD architecture into two parts: 1) the YOLOv3
[28] detector trained to output tight object bounding boxes with corresponding
semantic labels, and 2) the DPOD-like architecture, which predicts object
masks and dense correspondences from these detections. The proposed two-stage
approach simplifies and accelerates the training procedure of each component,
slightly improves the quality of correspondences and leads to better
performance on more challenging Homebrewed [9] and YCB-V[10] datasets. In
contrast to DPOD [4], which relied on two-dimensional UV maps, we use the
three-dimensional Normalized Object Coordinates Space (NOCS) [29]. This
parameterization permits trivial conversion between the object coordinate
system and the NOCS coordinate system. Additionally, we switched from the
ResNet18 backbone to the ResNet34 backbone. The first block of layers is
frozen to avoid overfitting when training on synthetic data. Data augmentation
and transfer learning allowed the reliable training of the networks. YOLO was
trained for 100 epochs, the augmented DPOD for 240. The last checkpoint was
used in all experiments.

Let us formally define a model as a set of its vertices:
$\mathcal{M}\coloneqq\\{v\in\mathbb{R}^{3}\\}$. Operators that compute minimum
and maximum coordinates along the vertex dimension $i$ of all
$v\in\mathcal{M}$ are defined as:

$\begin{aligned}
min_{i}(\mathcal{M})\coloneqq\mathop{\mathrm{min}}\limits_{v\in\mathcal{M}}v_{i},\end{aligned}\qquad\qquad\begin{aligned}
max_{i}(\mathcal{M})\coloneqq\mathop{\mathrm{max}}\limits_{v\in\mathcal{M}}v_{i}\end{aligned}$
(1)

Then, for any point $p$, the NOCS projection operator is defined w.r.t. the
model as

$\pi_{\mathcal{M}}(p)\coloneqq\left\\{\frac{p_{d}-min_{d}(\mathcal{M})}{max_{d}(\mathcal{M})-min_{d}(\mathcal{M})}\right\\}_{d\in\left\\{x,y,z\right\\}}$
(2)

and its inverse as $\pi_{\mathcal{M}}^{-1}$. The 6 DoF pose is defined as the
standard rigid body transformation, where $\mathbf{T}\in\mathcal{SE}(3)$.

###  III-B Pose refinement with differentiable renderer

Given a ground truth pose $\mathbf{T}_{gt}$ and a predicted pose
$\mathbf{T}_{pr}$, the aim of pose refinement is to find a pose update
$\mathbf{T}_{\Delta}$ that satisfies
$\mathbf{T}_{\Delta}\cdot\mathbf{T}_{pr}=\mathbf{T}_{gt}$. It is, however,
impossible to estimate a perfect $\mathbf{T}_{\Delta}$, because the ground
truth pose $\mathbf{T}_{gt}$ is not available. For this reason, proxy loss
functions have to be used, which results in a sub-optimal pose update
$\mathbf{T}_{\Delta}$. We propose refining $\mathbf{T}_{pr}$ by optimizing the
discrepancy between the predicted noisy NOCS maps from several frames and the
perfect NOCS map rendered in the estimated pose with a differentiable
renderer.

TABLE I: Percentages of correctly estimated poses w.r.t. the ADD on the
LineMOD [7] dataset. DR1, DR2 and DR4 stand for the proposed refinement with
1, 2 and 4 views respectively. Single-asterisked (*) methods use real training
data. Double-asterisked (**) methods report only refinement time instead of
the time of the whole pipeline.

| RGB Single-View | RGB Multi-View | RGBD
---|---|---|---
| Closest views | Random views | Farthest views
Method | SSD6D[12] | OURS | BB8*[13] | DPOD[4] | OURS | PoseCNN*[10] | OURS | OURS | OURS | AAE[19] | SSD6D[12] | DenFus*[21] | DenFus.*[21]
Refinement | DL[5] | - | DL[13] | DL[4] | DR1 | DeepIM[6] | DR2 | DR4 | DR2 | DR4 | DR2 | DR4 | ICP | ICP | - | DL[21]
Ape | - | 28.04 | 40.40 | 55.23 | 38.351 | 76.95 | 55.28 | 85.38 | 97.89 | 100 | 97.7 | 100 | 24.35 | - | 80 | 92
Bvs. | - | 71.65 | 91.80 | 72.69 | 85.354 | 97.48 | 94.95 | 99.61 | 99.81 | 100 | 100 | 99.61 | 89.13 | - | 84 | 93
Cam | - | 58.62 | 55.70 | 34.76 | 76.399 | 93.53 | 87.57 | 98.18 | 99.7 | 100 | 99.7 | 100 | 82.1 | - | 77 | 94
Can | - | 62.8 | 64.10 | 83.59 | 87.316 | 96.46 | 89.57 | 96.06 | 95.47 | 96.06 | 95.87 | 96.85 | 70.82 | - | 87 | 93
Cat | - | 40.12 | 62.60 | 65.1 | 55.988 | 82.14 | 78.4 | 97.2 | 99.4 | 100 | 99.4 | 100 | 72.18 | - | 89 | 97
Driller | - | 61.46 | 74.40 | 73.32 | 79.801 | 94.95 | 86.5 | 96.02 | 97.13 | 97.79 | 98.23 | 97.79 | 44.87 | - | 78 | 87
Duck | - | 26.17 | 44.30 | 50.04 | 38.18 | 77.65 | 53.57 | 81.2 | 94.56 | 98.87 | 97.56 | 98.87 | 54.63 | - | 76 | 92
Eggbox | - | 87.65 | 57.80 | 89.05 | 93.003 | 97.09 | 95.53 | 98.83 | 97.27 | 99.22 | 98.83 | 99.61 | 96.62 | - | 100 | 100
Glue | - | 67.48 | 41.20 | 84.37 | 67.988 | 99.42 | 89.15 | 97.67 | 95.94 | 98.06 | 93.22 | 98.06 | 94.18 | - | 99 | 100
Holep. | - | 19.09 | 67.20 | 35.35 | 24.568 | 52.81 | 35.38 | 52.69 | 79.27 | 87.69 | 84.81 | 87.31 | 51.25 | - | 79 | 92
Iron | - | 83.85 | 84.70 | 98.78 | 94.28 | 98.26 | 98.36 | 100 | 100 | 100 | 100 | 100 | 77.86 | - | 92 | 97
Lamp | - | 43.52 | 76.50 | 74.27 | 65.664 | 97.5 | 80.38 | 91.05 | 94.37 | 97.28 | 95.92 | 96.5 | 86.31 | - | 92 | 95
Phone | - | 45.77 | 54.00 | 46.98 | 74.472 | 87.72 | 87.32 | 97.18 | 99.6 | 100 | 99.4 | 99.6 | 86.24 | - | 88 | 93
Mean | 34.1 | 53.55 | 62.70 | 66.42 | 67.79 | 88.61 | 79.38 | 91.62 | 96.19 | 97.19 | 96.97 | 98.02 | 71.58 | 90.9 | 86 | 94
Time (ms) | - | 56 | 330 | 36 | 398 | \+ 83** | 233 | 159 | 232 | 162 | 227 | 163 | 224 | 100 | - | -

Assuming that N frames are used for multi-view refinement, we define the
corresponding set of predicted segmentations
$\tilde{\mathcal{S}}\coloneqq\\{\tilde{\mathbf{S}}_{1},\dots,\tilde{\mathbf{S}}_{N}\\}$,
each of which is a binary segmentation mask
$\tilde{\mathbf{S}}\in\\{0,1\\}^{W\times H}$. A set of predicted NOCS
correspondences is defined as
$\tilde{\mathcal{C}}\coloneqq\\{\tilde{\mathbf{C}}_{1},\dots,\tilde{\mathbf{C}}_{N}\\}$,
where each $\tilde{\mathbf{C}}\in[0,1]^{W\times H\times 3}$. These are noisy
estimates obtained with the modified DPOD. They are computed once and remain
unchanged during refinement. The differentiable renderer is used for a
differentiable definition of the following functions: binary
foreground/background rendering
$S:\mathcal{M}\times\mathcal{SE}(3)\rightarrow\\{0,1\\}^{W\times H}$ and NOCS
rendering $C:\mathcal{M}\times\mathcal{SE}(3)\rightarrow[0,1]^{W\times H\times
3}$. We will omit their dependence on the CAD model $\mathcal{M}$ to keep the
notation concise. For each set of images used for multi-view refinement, one
of the images is taken as the reference frame, and then for each $f$-th image,
its relative pose $\mathbf{\Xi}_{ref\rightarrow f}\in\mathcal{SE}(3)$ is
recomputed w.r.t. to the reference frame. Initial pose hypotheses
$\mathcal{T}\coloneqq\left\\{\mathbf{T}_{1},...,\mathbf{T}_{N}\right\\}$ are
estimated separately for each frame with PnP+RANSAC. The pose of the object in
the coordinate system of the reference frame is later denoted by
$\mathbf{T}_{pr}$.

We use pre-computed NOCS maps in each frame and NOCS maps of the model in the
estimated pose in the reference frame transformed to the coordinate system of
the $f$-th frame by transformation $\mathbf{\Xi}_{ref\rightarrow
f}\cdot\mathbf{T}_{pr}$ to define a per-pixel loss function over predicted and
rendered correspondences. The per-pixel NOCS discrepancy relates directly to
the 3D structure of the object and the error in 3D. The per-pixel loss for
pixel $p$ in the $f$-th frame is defined as follows:

$\mathcal{L}_{f,p}\coloneqq\rho\left(\pi^{-1}\left(\tilde{\mathbf{C}}_{f,p}\right),\pi^{-1}\left(C\left(\mathbf{\Xi}_{ref\rightarrow
f}\cdot\mathbf{T}_{\Delta}\cdot\mathbf{T}_{pr}\right)_{p}\right)\right)$ (3)

In the above equation, $\rho$ stands for the arbitrary distance function in
3D. Essentially, for each frame, the object pose
$\mathbf{T}_{\Delta}\cdot\mathbf{T}_{pr}$ is first transformed to the frame’s
coordinate system using $\mathbf{\Xi}_{ref\rightarrow f}$ and then rendered.
For each pixel, the predicted and rendered NOCS correspondences are projected
into the 3D coordinate system of the model. Then, the discrepancy between them
is penalized. The overall loss function is fully differentiable and dependent
only on $\mathbf{T}_{\Delta}$, since the rendering of NOCS maps is performed
using the differentiable renderer.

On the level of the full set of images used for refinement, the objective of
the refinement is defined as:

$\mathbf{T}_{\Delta}^{*}=\mathop{\mathrm{argmin}}\limits_{\mathbf{T}_{\Delta}\in\mathcal{SE}(3)}\sum_{f=1}^{N}\sum_{p\in\mathcal{I}}\left[\tilde{\mathbf{S}}_{f,p}\cdot
S\left(\mathbf{\Xi}_{ref\rightarrow
f}\cdot\mathbf{T}_{\Delta}\cdot\mathbf{T}_{pr}\right)_{p}\right]\cdot\mathcal{L}_{f,p}$

(4)

Here, $\left[\tilde{\mathbf{S}}_{f,p}\cdot S\left(\mathbf{\Xi}_{ref\rightarrow
f}\cdot\mathbf{T}_{\Delta}\cdot\mathbf{T}_{pr}\right)_{p}\right]$ serves as an
indicator function regarding whether or not the pixel is foreground in both
the rendered and the predicted NOCS maps.

The minimization problem, however, cannot be solved optimally due to its non-
convexity. Therefore, the loss function is minimized iteratively by gradient
descent over the pose update $\mathbf{T}_{\Delta}$. This can be done using any
gradient-based method. We normally observe convergence within 50 optimization
steps. A trivial degenerate solution to the optimization problem exists, which
sets the loss to zero, namely non-overlapping rendered and predicted
$\tilde{\mathcal{S}}$ segmentation maps. But in reality, this is not a
problem, because PnP+RANSAC provides reliable initial pose estimates.
Moreover, degenerate pose hypotheses are explicitly handled in the choice of
the reference frame.

There are numerous ways os implementing the distance function
$\rho\colon\mathbb{R}^{3}\times\mathbb{R}^{3}\to\mathbb{R}^{+}$ and
parameterizing $\mathcal{SO}(3)$ rotations. We use the continuous rotation
parameterization from [30], which enables faster and more stable convergence
during the optimization procedure than quaternions and Euler angles. As the
predicted correspondences and matched correspondences might contain a
potentially large number of outliers, a robust $\rho$ function must be used to
mitigate this. We experimented with several options and ended up with a
particular case of the general robust function introduced in [31]. This
function is defined as follows:

$\rho(e,c)\coloneqq
1-\exp{\left(-\frac{1}{2}\left(\frac{e}{c}\right)^{2}\right)}$ (5)

$c$ is a hyper-parameter that stands for the scale of the loss function. In
our experiments, we adjusted it dynamically according to the median absolute
residuals: $c\coloneqq 2\cdot MEDIAN\left(\left|\mathbf{e}\right|\right)$.

The choice of the reference frame can affect the effectiveness of pose
refinement. The goal is to automatically choose a pose which has a high
overlap with predicted segmentations when transformed and rendered in other
views. Additionally, it should have the smallest possible loss
$\mathcal{L}_{f}$. For each image batch, the reference frame is chosen as
follows:

$\mathop{\mathrm{argmin}}\limits_{ref\in\left[1,..,N\right]}\frac{1}{K}\sum_{f=1}^{N}\frac{\sum_{p\in\mathcal{I}}\left[\tilde{\mathbf{S}}_{f,p}\cdot
S\left(\mathbf{\Xi}_{ref\rightarrow
f}\cdot\mathbf{T}_{ref}\right)_{p}\right]\cdot\mathcal{L}_{f,p}}{IOU\left(\tilde{\mathbf{S}}_{f},S\left(\mathbf{\Xi}_{ref\rightarrow
f}\cdot\mathbf{T}_{ref}\right)\right)}$ (6)

where $K$ stands for the number of frames with non-zero loss. Additionally,
$\mathop{\mathrm{argmin}}\limits$ ignores zero values, as they correspond to
degenerate poses that are not re-projected correctly onto other frames.

TABLE II: Percentages of correctly estimated poses w.r.t. the ADD on the
Occlusion [8] dataset. DR1, DR2 and DR4 stand for the proposed refinement with
1, 2 and 4 views respectively.

Train data | Synthetic | Real - GT labels
---|---|---
Method | OURS | SSD6D[12] | OURS | OURS | Pix2Pose[16] | PVNet[14] | DPOD[4] | HybPose[15]
closest views | random views | farthest views
Refinement | - |  DL [5] | DR1 | DR2 | DR4 | DR2 | DR4 | DR2 | DR4 | - | - | - | -
Ape | 26.1 | - | 15.9 | 21.59 | 31.45 | 57.85 | 79.81 | 61.74 | 79.81 | 22.0 | 15.8 | 18.8 | 53.3
Can | 37.8 | - | 45.4 | 63.61 | 81.67 | 71.57 | 88.05 | 72.37 | 87.65 | 44.7 | 63.3 | 60.8 | 86.5
Cat | 6.3 | - | 8.3 | 20.45 | 34.54 | 26.82 | 41.82 | 23.64 | 48.18 | 22.7 | 16.7 | 15.4 | 73.4
Driller | 43.7 | - | 52.4 | 80.3 | 97.79 | 91.61 | 98.67 | 93.80 | 99.11 | 44.7 | 65.7 | 62.0 | 92.8
Duck | 23.4 | - | 22.4 | 54.3 | 76.95 | 67.44 | 87.01 | 72.27 | 85.94 | 15.0 | 25.2 | 37.8 | 62.8
Eggbox | 22.7 | - | 21.0 | 34.19 | 44.32 | 41.54 | 43.08 | 39.07 | 48.45 | 25.2 | 50.2 | 70.9 | 95.3
Glue | 43.0 | - | 50.2 | 67.3 | 80.92 | 66.15 | 75.57 | 66.92 | 74.81 | 32.4 | 49.6 | 65.4 | 92.5
Holep. | 12.9 | - | 15.4 | 45.62 | 74.08 | 60.27 | 84.51 | 63.30 | 83.50 | 49.5 | 39.7 | 46.9 | 76.7
Mean | 27.0 | 27.5 | 28.9 | 48.42 | 65.22 | 60.41 | 74.82 | 61.64 | 75.93 | 32.0 | 40.8 | 47.3 | 79.2

The proposed refinement procedure has a certain similarity to the point-to-
point ICP as well as to PnP. In terms of the ICP, point-to-point
correspondences are established by rendering NOCS maps and using the predicted
NOCS maps with the same spatial location in 2D, as opposed to the nearest
neighbor search in the standard 3D ICP. The backpropagation through the
renderer corresponds directly to the distance minimization step of the ICP.
However, projective transformation is essential. This is because if only RGB
information is used, and therefore predicted NOCS maps in all frames are
already in the model’s coordinate system and do not impose any additional
spatial 3D constraints. On the other hand, rendering in order to establish
correspondences and minimize the discrepancy directly relates to PnP. The
proposed refiner can be seen as a multi-view PnP, where direct pixel-wise
error, rather than the reprojection error, is minimized.

###  III-C Autolabeling

We explore the usefulness of the proposed refiner for the self-annotation of
weakly labeled real data, similar to the work of [11]. We assume access to
weakly labeled real images with no pose labels. Again, relative
transformations between cameras are needed, which is a weaker assumption than
having precises per-object pose annotations. For the sake of simplicity, we
use the weak 2D labels to filter out correct detections, although this is not
necessary, as the detections can be filtered out with the epipolar
constraints.

The autolabeling pipeline operates as follows. First, the YOLO detector and
the DPOD network are trained in the usual way on synthetically generated data.
They are then applied to a partition of the real data and detections are
filtered out. The poses are computed using PnP+RANSAC and fed into our multi-
view refinement pipeline. Finally, we retrieve the newly estimated labels and
use them to build a dataset with real images. The DPOD part is trained as
usual, but the predicted NOCS maps are used instead of the ground truth NOCS.
The procedure allows us to achieve the performance at the level of DPOD
trained on fully annotated real data. No filtering of pose predictions is
performed, as ground truth poses are not available in the given scenario. This
results in a few images with bad pose annotations being used for training, but
they have no significant negative effect on the final performance. The
augmented DPOD trained in the standard way for 240 epochs, and the last
checkpoint is then used for evaluation.

##  IV Experiments

In this section, we evaluate our multi-view refinement pipeline on LineMOD
[7], Occlusion [8], Homebrewed [9] and YCB-V [10] datasets to assess its
properties. We then evaluate the proposed autolabeling pipeline. Lastly, we
test the robustness of the proposed refiner to the imprecision of relative
camera transformations and the choice of frames for refinement. We follow the
standard evaluation procedure of [12, 5, 4] and report the pose accuracy only
for objects correctly detected in 2D on Linemod and Occlusion. Pose quality is
computed in accordance with the ADD metric [7]. On Homebrewed and YCB-V, on
the other hand, we submit the predicted poses to the BOP challenge [3] for
evaluation and report the Average Recall (AR) metric returned by it.

Single object pose estimation. Here, we compare the quality of poses predicted
with our method to different approaches on the LineMOD [7] dataset. The
results are summarized in Table I. The table compares our refinement method to
various top-performing deep learning methods which reported an ADD score with
different pose refinement approaches. It is not our aim to make direct
comparisons with monocular methods, but rather to compare the quality of poses
after various refinement methods.

Our augmented DPOD, labeled as OURS, achieves 53.55% of the average ADD pose
accuracy without any refinement, slightly outperforming the original DPOD,
which showed 50% pose accuracy. Next, we evaluate the performance of our
differentiable rendering refiner in the monocular scenario (OURS DR1) and
compare it to previous state-of-the-art methods based on deep learning (DL):
[13, 4, 6, 5]. In this case, the pose accuracy increases from 53.55% to
67.69%, even though only one frame is used and no additional multi-view
constraints. The refiner outperforms all the competitors in that category
apart from DeepIM, even though BB8 and DeepIM have the advantage of using real
train data.

The addition of multiple frames introduces spatial geometric constraints that
result in a significant performance boost w.r.t. the ADD score when two (OURS
DR2) or four (OURS DR4) frames are used. Even though, the ADD score depends on
the choice of the frames, i.e. the closest frames corresponding to weaker
constraints, even the two-view refiner can outperform or perform similarly to
other approaches using depth-based ICP ([19] and [12]) and DenseFusion, which
uses real train data, RGBD inference and iterative DL-based refinement. The
table clearly shows that even a minimum multi-view setup can bring a
significant performance boost without any need for precise calibrated depth
information.

Robustness to occlusions. The robustness of our method to occlusions was
evaluated on Occlusion [8], Homebrewed [9] and YCB-V [10] datasets. The
results are presented in Table II, Table III and Table IV respectively.

Unfortunately, detectors trained on synthetic data are seldom evaluated on
Occlusion (Table II). To the best of our knowledge, there is only an ADD score
from SSD6D with the DL refinement [5]. Therefore, we instead compare to the
approaches trained with real data. The advantage of these approaches is that
they use real data from the target domain and also overfit to the particular
occlusions present in the test images. This enables comparison with the best
detectors and assessment of how closely the refiner is able to approach the
performance of the algorithms trained on real data. As can be seen in Table
II, our refiner is able to significantly improve the performance of the non-
refined baseline. If 4 views are used for refinement, the performance of our
approach is on-par with the state-of-the-art results, even though we do not
use any real data annotations.

With the Homebrewed dataset [9] (Table III), all top-performing methods were
trained on the synthetic PBR images.Our main aim here is to compare our
refiner to CosyPose [2], which also utilizes multi-view refinement. It is
clear from the table that correspondence-based pose estimation methods (ours
and CDPN) confidently outperform the direct pose prediction of CosyPose if no
refinement is used. With the multi-view refinement, the proposed methods
achieves top results even if only 2 closest views are used for refinement.
Additionally, the proposed approach outperforms CDPN, CosyPose and Pix2Pose
even if their predictions are refined with ICP.

Table IV shows a comparison of various top-performing methods on the YCB-V
[10] dataset. The dataset comes with a real train set and a set of pre-
rendered synthetic images (marked as ’real’ and ’synt’ in the table). On the
other hand, synthetic PBR images are also available. When trained on PBR
images, raw non-refined CosyPose poses outperform ours. With refinement, our
methods outperforms CosyPose, CDPN and EPOS trained on the same PBR data.
However, training on real data still has a huge advantage on this dataset.

TABLE III: Results on the Homebrewed dataset [9] reported according to the Average Recall (AR) metric of the BOP challenge [3] on the BOP challenge subset of test images.  Method | Train data | Refinement | AR | Time (s)
---|---|---|---|---
OURS | PBR | 4 furthest views | 0.841 | 0.578
OURS | PBR | 4 random views | 0.835 | 0.584
OURS | PBR | 4 closest views | 0.83 | 0.600
OURS | PBR | 2 random views | 0.818 | 0.926
OURS | PBR | 2 furthest views | 0.817 | 0.926
OURS | PBR | 2 closest views | 0.787 | 0.91
CosyPose[2] | PBR | 8 views | 0.746 | 0.427
OURS | PBR | no | 0.725 | 0.163
CDPNv2[17] | PBR | no | 0.722 | 0.273
CosyPose[2] | PBR | ICP | 0.712 | 5.326
CDPNv2 | synt, PBR | ICP | 0.712 | 0.713
CosyPose[2] | PBR | 4 views | 0.696 | 0.445
Pix2Pose[16] | PBR | ICP | 0.695 | 3.248
CosyPose[2] | PBR | no | 0.656 | 0.417

Runtime. To allow faster refinement time, the camera intrinsics were re-
computed such that the rendered image always has the dimensions
128$\times$128\. Moreover, each model was sub-sampled to contain only 1000
faces. After performing an ablation study on the Linemod dataset, we set the
number of refinement iterations to 50. As a result, if 2 views are used for
refinement, the average per-image refinement time for one object is 170
milliseconds; with 4 views - 100 milliseconds. Runtime scales linearly with
the number of objects in the scene. Tables I,III, IV indicate that the
proposed refinement, while not being real-time-capable, displays a performance
that is similar in terms of time to the multi-view matching and refinement of
CosyPose and similar to or faster than ICP-based methods, depending on the
dataset. All experiments were conducted on an Intel Core i9-9900K CPU 3.60GHz
with NVIDIA Geforce RTX 2080 TI GPU.

Robustness to camera choices. For all datasets used for evaluation, we split
each sequence into sets of 2 or 4 images. Images in the different sets do not
overlap. For each set, we compute relative camera poses from individual ground
truth camera poses. The pose is optimized jointly for images in the set. We
experiment with three different view sampling strategies: closest views,
random views and furthest views. Tables I, II, III, IVshow a quantitative
comparison of these strategies. It is clear from the tables that even though
close camera locations essentially constitute a bad setup for multi-view pose
refinement, as the pose error in one frame is not necessarily visible in other
frames, the refiner still improves the poses. Random and furthest view
selections tend to perform similarly on all the datasets.

Relative Pose Noise. The aim of this experiment is to demonstrate how noise in
relative poses affects the overall performance of the multi-view optimization
pipeline. To do this, we add noise separately to translation and rotation. For
rotation transformation, we sample the perturbation angle for each axis from a
normal distribution with zero mean and a standard deviation of 5, 7.5 or 10
degrees. For translation transformation, perturbations are sampled from the
normal distribution with zero mean and a standard deviation computed on the
basis of an object diameter: $\sigma=\frac{1}{3}\cdot(0.1\cdot diam_{obj})$.
Larger deviations render the problem innately ill-posed due to the definition
of the ADD measure. If the camera is at a distance of more than 10% of the
model’s diameter, the pose will always be classified as incorrect according to
the ADD metric. The results can be seen in Table V. Even with the noisy poses,
the refiner still ensures a reasonable pose quality. This table also shows
that having more views is beneficial in the noisy scenario.

TABLE IV: Results on the YCB-V dataset reported according to the Average Recall (AR) metric of the BOP challenge [3] on the BOP challenge subset of test images. CosyPose [2] results labeled with * were obtained by re-running the official implementation of the paper.  Method | Train data | Refinement | AR | Time (s)
---|---|---|---|---
CosyPose[2] | synt + real | ICP | 0.861 | 2.736
CosyPose[2] | synt + real | 8 views | 0.853 | 0.285
CosyPose[2] | synt + real | 4 views | 0.84 | 0.318
CosyPose[2] | synt + real | no | 0.821 | 0.241
Pix2Pose[16] | synt + real | ICP | 0.78 | 2.59
EPOS[18] | synt | no | 0.696 | 0.572
OURS | PBR | 4 furthest views | 0.674 | 0.555
OURS | PBR | 4 random views | 0.645 | 0.557
OURS | PBR | 2 furthest views | 0.621 | 848
OURS | PBR | 2 random views | 0.61 | 0.855
CosyPose*[2] | PBR | 8 views | 0.61 | 0.412
CosyPose*[2] | PBR | 4 views | 0.592 | 0.415
CosyPose[2] | PBR | no | 0.574 | 0.342
OURS | PBR | 4 closest views | 0.564 | 0.564
OURS | PBR | 2 closest views | 0.541 | 0.853
CDPNv2[17] | synt + real | no | 0.532 | 0.143
CDPNv2[17] | PBR | ICP | 0.532 | 1.034
OURS | PBR | no | 0.525 | 0.187
EPOS[18] | PBR | no | 0.499 | 0.764
CDPNv2[17] | PBR | no | 0.39 | 0.448

Autolabeling. In this experiment, we retrieve the estimated OURS+DR2 and
OURS+DR4 labels and use them to replace the synthetic training set by the
automatically generated real labels as discussed above. The extended training
set is then used to fine-tune the synthetically trained baseline. The results
can be seen in Table VI. It can be clearly seen that the fine-tuned network
trained on autolabels significantly outperforms the synthetic DPOD and is very
competitive when compared to the state-of-the-art methods trained on full
ground truth labels.

TABLE V: Percentages of correctly estimated poses w.r.t. the ADD score on the
Linemod [7] dataset with noisy relative camera transformations.

| 2 views | 4 views
---|---|---
| Trans. (% diam.) | Rot. (deg.) | Trans. (% diam.) | Rot. (deg.)
| 5 | 10 | 15 | 5 | 7.5 | 10 | 5 | 10 | 15 | 5 | 7.5 | 10
Ape | 95 | 79 | 58 | 93 | 82 | 68 | 99 | 89 | 66 | 98 | 89 | 73
Bvise | 97 | 85 | 63 | 97 | 91 | 78 | 100 | 91 | 70 | 99 | 93 | 80
Cam | 97 | 86 | 65 | 95 | 78 | 65 | 100 | 93 | 56 | 99 | 86 | 66
Can | 87 | 75 | 53 | 86 | 73 | 63 | 89 | 81 | 56 | 90 | 80 | 62
Cat | 95 | 81 | 62 | 95 | 85 | 72 | 99 | 89 | 63 | 99 | 91 | 76
Driller | 95 | 83 | 61 | 94 | 85 | 67 | 99 | 91 | 68 | 99 | 91 | 75
Duck | 89 | 74 | 57 | 91 | 78 | 60 | 97 | 86 | 62 | 96 | 81 | 63
Eggbox | 97 | 96 | 94 | 97 | 98 | 96 | 99 | 98 | 98 | 98 | 98 | 98
Glue | 94 | 93 | 86 | 94 | 94 | 91 | 98 | 96 | 96 | 97 | 99 | 97
Holep. | 79 | 65 | 44 | 73 | 63 | 51 | 89 | 76 | 52 | 85 | 73 | 62
Iron | 96 | 79 | 60 | 97 | 87 | 75 | 99 | 87 | 62 | 99 | 93 | 78
Lamp | 92 | 80 | 61 | 91 | 79 | 68 | 96 | 88 | 65 | 96 | 91 | 77
Phone | 95 | 79 | 60 | 95 | 86 | 71 | 100 | 87 | 60 | 99 | 87 | 72
Mean | 93 | 81 | 63 | 92 | 83 | 71 | 97 | 89 | 67 | 96 | 89 | 75

TABLE VI: Percentages of correctly estimated poses w.r.t. the ADD score when
automatically annotated data is used to train the monocular detector.

Train data | Synt | Real - Autolabels | Real - GT labels
---|---|---|---
DR2 | DR4
Method | OURS | OURS | OURS | DPOD[4] | PVNet[14] | OURS | CDPN[17] | HybP.[15]
Ape | 28.04 | 65.29 | 68.84 | 53.28 | 43.62 | 74.78 | 64.38 | 77.6
Bvise | 71.65 | 98.84 | 99.81 | 95.34 | 99.90 | 99.71 | 97.77 | 99.6
Cam | 58.62 | 93.19 | 94.25 | 90.36 | 86.86 | 96.52 | 91.67 | 95.9
Can | 62.80 | 97.44 | 98.62 | 94.1 | 95.47 | 99.02 | 95.87 | 93.6
Cat | 40.12 | 78.54 | 82.04 | 60.38 | 79.34 | 90.52 | 83.83 | 93.5
Driller | 61.46 | 94.81 | 96.47 | 97.72 | 96.43 | 97.9 | 96.23 | 97.2
Duck | 26.17 | 31.05 | 32.46 | 66.01 | 52.58 | 48.41 | 66.76 | 87
Eggbox | 87.65 | 93.48 | 95.24 | 99.72 | 99.15 | 99.9 | 99.72 | 99.6
Glue | 67.48 | 95.26 | 96.91 | 93.83 | 95.66 | 97.68 | 99.61 | 98.7
Holep. | 19.09 | 25.43 | 43.47 | 65.83 | 81.92 | 49.62 | 85.82 | 92.5
Iron | 83.85 | 99.26 | 99.59 | 99.8 | 98.88 | 99.39 | 97.85 | 98.1
Lamp | 43.52 | 90.79 | 93.31 | 88.11 | 99.33 | 92.92 | 97.89 | 96.9
Phone | 45.77 | 88.14 | 90.65 | 74.24 | 92.41 | 93.07 | 90.75 | 98.3
Mean | 53.55 | 80.89 | 83.97 | 82.98 | 86.27 | 87.65 | 89.86 | 94.5

##  V Conclusions

In this paper, we propose a novel object pose refinement pipeline. We adopt
the idea of using multiple frames to produce a single joint object pose
estimate. The use of multiple frames enables effective use of geometric
constraints. The proposed refiner is based on the external object detector,
which outputs deep 2D-3D correspondences in the form of Normalized Object
Coordinate space. Predicted dense correspondences from calibrated multiple
frames can be brought together to obtain a better pose estimate using a
differentiable renderer. The proposed approach imposes no constraints on the
number of frames used and the exact positions of the cameras in the 3D world,
and it remains robust even when the cameras are in close proximity to one
another. We experimentally show that the refiner works excellently on Linemod,
Occlusion, Homebrewed and YCB-V datasets. As an alternative use case, we
demonstrate how the proposed approach can be applied to automatically annotate
real train data, which has no object pose annotations. The approach even
remains effective if the relative transformations in-between frames are
imprecise. Moreover, even a multi-view setup with only two frames already
produces excellent results. This demonstrates that the approach can be used
successfully in practical applications.

## References

  * [1] B. Drost, M. Ulrich, N. Navab, and S. Ilic, “Model globally, match locally: Efficient and robust 3d object recognition,” in _2010 IEEE computer society conference on computer vision and pattern recognition_. Ieee, 2010, pp. 998–1005.
  * [2] Y. Labbe, J. Carpentier, M. Aubry, and J. Sivic, “Cosypose: Consistent multi-view multi-object 6d pose estimation,” in _ECCV_ , 2020.
  * [3] T. Hodan and A. Melenovsky. (2019) Bop: Benchmark for 6d object pose estimation: https://bop.felk.cvut.cz/home/.
  * [4] S. Zakharov, I. Shugurov, and S. Ilic, “Dpod: 6d pose object detector and refiner,” in _ICCV_ , 2019.
  * [5] F. Manhardt, W. Kehl, N. Navab, and F. Tombari, “Deep model-based 6d pose refinement in rgb,” in _ECCV_ , 2018.
  * [6] Y. Li, G. Wang, X. Ji, Y. Xiang, and D. Fox, “Deepim: Deep iterative matching for 6d pose estimation,” in _ECCV_ , 2018.
  * [7] S. Hinterstoisser, V. Lepetit, S. Ilic, S. Holzer, G. Bradski, K. Konolige, and N. Navab, “Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes,” in _ACCV_ , 2012.
  * [8] E. Brachmann, A. Krull, F. Michel, S. Gumhold, J. Shotton, and C. Rother, “Learning 6d object pose estimation using 3d object coordinates,” in _ECCV_ , 2014.
  * [9] R. Kaskman, S. Zakharov, I. Shugurov, and S. Ilic, “Homebreweddb: Rgb-d dataset for 6d pose estimation of 3d objects,” in _ICCVW_ , 2019.
  * [10] Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox, “Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes,” 2018.
  * [11] S. Zakharov, W. Kehl, A. Bhargava, and A. Gaidon, “Autolabeling 3d objects with differentiable rendering of sdf shape priors,” in _CVPR_ , 2020.
  * [12] W. Kehl, F. Manhardt, F. Tombari, S. Ilic, and N. Navab, “Ssd-6d: Making rgb-based 3d detection and 6d pose estimation great again,” in _ICCV_ , 2017\.
  * [13] M. Rad and V. Lepetit, “Bb8: A scalable, accurate, robust to partial occlusion method for predicting the 3d poses of challenging objects without using depth,” in _ICCV_ , 2017.
  * [14] S. Peng, Y. Liu, Q. Huang, X. Zhou, and H. Bao, “Pvnet: Pixel-wise voting network for 6dof pose estimation,” in _CVPR_ , 2019.
  * [15] C. Song, J. Song, and Q. Huang, “Hybridpose: 6d object pose estimation under hybrid representations,” in _ICCV_ , 2020.
  * [16] K. Park, T. Patten, and M. Vincze, “Pix2pose: Pixel-wise coordinate regression of objects for 6d pose estimation,” in _ICCV_ , 2019.
  * [17] Z. Li, G. Wang, and X. Ji, “Cdpn: Coordinates-based disentangled pose network for real-time rgb-based 6-dof object pose estimation,” in _ICCV_ , 2019.
  * [18] T. Hodan, D. Barath, and J. Matas, “Epos: Estimating 6d pose of objects with symmetries,” in _CVPR_ , 2020.
  * [19] M. Sundermeyer, Z.-C. Marton, M. Durner, M. Brucker, and R. Triebel, “Implicit 3d orientation learning for 6d object detection from rgb images,” in _ECCV_ , 2018.
  * [20] P. J. Besl and N. D. McKay, “Method for registration of 3-d shapes,” in _Sensor fusion IV: control paradigms and data structures_ , 1992.
  * [21] C. Wang, D. Xu, Y. Zhu, R. Martín-Martín, C. Lu, L. Fei-Fei, and S. Savarese, “Densefusion: 6d object pose estimation by iterative dense fusion,” in _CVPR_ , 2019.
  * [22] J. Sock, S. Hamidreza Kasaei, L. Seabra Lopes, and T.-K. Kim, “Multi-view 6d object pose estimation and camera motion planning using rgbd images,” in _ICCV_ , 2017.
  * [23] Ö. Erkent, D. Shukla, and J. Piater, “Integration of probabilistic pose estimates from multiple views,” in _ECCV_ , 2016.
  * [24] C. Li, J. Bai, and G. D. Hager, “A unified framework for multi-view multi-class object pose estimation,” in _ECCV_ , 2018.
  * [25] R. Kaskman, I. Shugurov, S. Zakharov, and S. Ilic, “6 dof pose estimation of textureless objects from multiple rgb frames,” in _ECCV Workshops_ , 2020\.
  * [26] V. Lepetit, F. Moreno-Noguer, and P. Fua, “Epnp: An accurate o (n) solution to the pnp problem,” _IJCV_ , 2009.
  * [27] S. Liu, T. Li, W. Chen, and H. Li, “Soft rasterizer: A differentiable renderer for image-based 3d reasoning,” in _ICCV_ , 2019.
  * [28] J. Redmon and A. Farhadi, “Yolov3: An incremental improvement,” _arXiv preprint arXiv:1804.02767_ , 2018.
  * [29] H. Wang, S. Sridhar, J. Huang, J. Valentin, S. Song, and L. J. Guibas, “Normalized object coordinate space for category-level 6d object pose and size estimation,” in _CVPR_ , 2019.
  * [30] Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li, “On the continuity of rotation representations in neural networks,” in _CVPR_ , 2019.
  * [31] J. T. Barron, “A general and adaptive robust loss function,” in _CVPR_ , 2019\.