ITEP-TH-10/15

Zamolodchikov tetrahedral equation
and higher Hamiltonians
of 2d quantum integrable systems

D.V<EMAIL_ADDRESS>Geometry and Topology Dep. Faculty of
Mechanics and Mathematics, Moscow State University, 119991 Moscow RUSSIA, Tel
/ Fax: (+7 495) 939 3798  (MSU, ITEP)

Abstract. The main aim of this work is to develop a method of constructing
higher Hamiltonians of quantum integrable systems associated with the solution
of the Zamolodchikov tetrahedral equation. As opposed to the series of papers
[1] the approach presented here is effective for generic solutions of the
tetrahedral equation without spectral parameter. In a sense, this result is a
two-dimensional generalization of the method of [2]. The work is a part of the
project relating the tetrahedral equation with the quasi-invariants of
2-knots.

###### Contents

  1. 1 Introduction
    1. 1.1 Yang-Baxter equation and its generalizations
    2. 1.2 Universal integrability in $d=1$
    3. 1.3 Tetrahedral equation
  2. 2 Commutative family
    1. 2.1 The commutativity demonstration in $d=1$
    2. 2.2 Regular $3$-d lattices and statistical models
    3. 2.3 Some consequences of the tetrahedral equation
    4. 2.4 Two families
    5. 2.5 The commutativity proof in $d=2$
  3. 3 Conclusion

##  1 Introduction

###  1.1 Yang-Baxter equation and its generalizations

This work is mainly focusing on the matrix Yang-Baxter equation

$\displaystyle R_{12}R_{13}R_{23}=R_{23}R_{13}R_{12}\in End(V^{\otimes
3}),\qquad R\in End(V^{\otimes 2}),$ (1)

and the matrix tetrahedral Zamolodchikov equation [3]

$\displaystyle\Phi_{123}\Phi_{145}\Phi_{246}\Phi_{356}=\Phi_{356}\Phi_{246}\Phi_{145}\Phi_{123}\in
End(V^{\otimes 6}),\qquad\Phi\in End(V^{\otimes 3}).$ (2)

In both cases $V$ is a finite dimensional vector space, the indices denote the
numbers of the space copies in which linear operators act non-trivially. We
should also mention the universal description of the $n$-simplex equation (eg.
[4]), generalizing the Yang-Baxter and the Zamolodchikov equations.

The work is aimed to generalize one of the existing applications of the theory
of the Yang-Baxter equation to the theory of quantum integrable systems. It
should be noted that this equation and subsequently the theory of quantum
groups marked a new era in the field of exactly-solvable models of
mathematical physics. This concerned both models of statistical physics in
dimension 2, and one-dimensional quantum mechanical models of the theory of
magnets. The established connection has been effective in the purely physical
issues like the effect of the spontaneous magnetization as like as in many
mathematical problems. The language of Hopf algebras not only expanded the
machine of modern algebra but also allowed to explore new patterns, for
example, in topology.

The mentioning of such diverse areas of modern mathematical physics here is
not accidental. The transition from the Yang-Baxter equations to the
tetrahedral equation appears natural from many points of view: low-dimensional
topology, combinatorics, statistical models, topological field theories,
homotopy algebraic structures. Here we present a brief table showing some
heredity in subjects related to the Yang-Baxter equation and the tetrahedral
one.

Table 1: Relations |  Yang-Baxter equation |  Zamolodchikov equation
---|---|---
Statistical models |  $d=2$ |  $d=3$
Spin chains |  $d=1$ |  $d=2$
Homotopy Lie algebras |  Lie algebras |  $2$-Lie algebras (eg. thesis of A. Crans)
Topological invariants |  Turaev-Reshetikhin-type knot invariants |  $2$-knot quasi-invariants in [7]
Hopf algebras |  Quasi-triangular Hopf algebras |  ?

###  1.2 Universal integrability in $d=1$

Besides the equation (1)

$\displaystyle R_{12}R_{13}R_{23}=R_{23}R_{13}R_{12}$

we consider the structural equations in the combinatorial form:

$\displaystyle R^{\prime}_{12}R^{\prime}_{23}R^{\prime}_{12}$ $\displaystyle=$
$\displaystyle R^{\prime}_{23}R^{\prime}_{12}R^{\prime}_{23}$ (3)
$\displaystyle R^{\prime}L\otimes L$ $\displaystyle=$ $\displaystyle L\otimes
LR^{\prime}$ (4)

where $R^{\prime}\in End(V)^{\otimes 2}$ and $L\in End(V)\otimes End(V_{q}),$
here $V_{q}$ is the quantum vector space (i.e. one another vector space,
distinguished from $V$.) The transition from (1) to (3) can be realized as
follows: if $R$ satisfies (1) then $R^{\prime}_{12}=R_{12}P_{12}$ as like as
$R^{\prime\prime}_{12}=P_{12}R_{12}$ satisfy (3), here $P_{12}$ is the
transposition linear operator in $V\otimes V.$ The equation (4) is
traditionally called the RLL-relation, it plays a substantial role in
Reshetikhin-Takhtadzhyan algebras. A solution example of (4) is provided by

$\displaystyle L=R_{1i_{1}}R_{1i_{2}}\ldots R_{1i_{k}},$

where $L$ is an operator in the quantum
space$V_{q}=V_{i_{1}}\otimes\ldots\otimes V_{i_{k}},$ here $V_{i_{j}}$ is just
a copy of the space $V.$

The work [2] presents a construction of a commutative family in the algebra
$End(V_{q})$ containing the trace $I_{1}=Tr_{V}L.$

######  Lemma 1 ([2])

Let us introduce a notation $L_{i}$ for the corresponding element in
$End(V_{i})\otimes End(V_{q}).$ Then the operators

$\displaystyle I_{k}=Tr_{1\ldots k}L_{1}\ldots
L_{k}R^{\prime}_{12}R^{\prime}_{23}\ldots R^{\prime}_{k-1,k}$ (5)

commute in $End(V_{q}).$ The trace is meant with respect to the auxiliary
spaces $V_{i}.$

This statement has an important role in the technique of constructing quantum-
mechanical integrable systems. This is applicable in quantum Gaudin systems,
Ruijenaars-Schneider system and others. Moreover, this is directly associated
with the theory of exactly-solvable models of statistical physics on
$2$-dimensional lattices. The study of the spectrum of the transfer-matrix is
a key ingredient in the problem of finding the partition function asymptotics
of some statistical models (eg. [5]).

###  1.3 Tetrahedral equation

As like as in the Yang-Baxter case for many purposes it is more convenient to
consider the set-theoretic tetrahedral equation (STTE) which may be defined as
follows: let $X$ be a finite set, we say that there is a solution for STTE on
$X$ if there is a map

$X\times X\times X\stackrel{{\scriptstyle\Phi}}{{\longrightarrow}}X\times
X\times X,$

satisfying the relation (graphically coinciding with (2))

$\Phi_{123}\circ\Phi_{145}\circ\Phi_{246}\circ\Phi_{356}=\Phi_{356}\circ\Phi_{246}\circ\Phi_{145}\circ\Phi_{123}:X^{\times
6}\to X^{\times 6}.$ (6)

Here, however, unlike (2), $X^{\times 6}$ denotes the Cartesian 6-th power of
$X$ and the subscripts denote the number of factors to which $\Phi$ is
applied, in other factors the map acts identically. For example

$\displaystyle\Phi_{356}(a_{1},a_{2},a_{3},a_{4},a_{5},a_{6})$
$\displaystyle=(a_{1},a_{2},\Phi_{1}(a_{3},a_{5},a_{6}),a_{4},\Phi_{2}(a_{3},a_{5},a_{6}),\Phi_{3}(a_{3},a_{5},a_{6}))$
(7)
$\displaystyle=(a_{1},a_{2},a_{3}^{\prime},a_{4},a_{5}^{\prime},a_{6}^{\prime}),$

where

$\Phi(x,y,z)=(\Phi_{1}(x,y,z),\Phi_{2}(x,y,z),\Phi_{3}(x,y,z))=(x^{\prime},y^{\prime},z^{\prime}).$

One distinguishes the functional tetrahedral equation (FTE), for example the
so-called electric solution (eg. [6]), represented as a transformation acting
on the space of functions of three variables:

$\displaystyle\Phi(x,y,z)=(x_{1},y_{1},z_{1});$ (8) $\displaystyle
x_{1}=\frac{xy}{x+z+xyz},$ $\displaystyle y_{1}=x+z+xyz,$ $\displaystyle
z_{1}=\frac{yz}{x+z+xyz}.$

This solution is relevant to the well-known star-triangle relation in the
theory of electric circuits.

There is another one interpretation of the tetrahedral equation in the task of
coloring of $2$-faces of the $4$-dimensional cube with elements of the set
$X$. Let $\Phi:X\times X\times X\rightarrow X\times X\times X$ be a map. A
coloring is called admissible if the colors of the incoming faces of all
$3$-cubes $x,y,z$ are related with the colors of outgoing faces
$x^{\prime},y^{\prime},z^{\prime}$ by the action of the map $\Phi:$

$\displaystyle(x^{\prime},y^{\prime},z^{\prime})=\Phi(x,y,z)$

At figure 1 we design a projection of the $4$-cube to a $3$-space

Figure 1: Tesseract

and at figure 2 \- two alternative sequences of coloring steps.

Figure 2: Coloring

It turns out that the condition of equivalence of the coloring obtained by
these two ways is equivalent to the tetrahedral equation on $\Phi$. The
problem is described in more details in [7].

The $N$-cube $2$-faces coloring problem allows us to construct a complex
analogous to those calculating the Yang-Baxter cohomology for the case of set-
theoretic tetrahedral equations in [7]. The $3$-cocycles of the complex play a
special role in this subject, they are determined by the condition:

$\displaystyle\varphi(a_{1},a_{2},a_{3})\varphi(a_{1}^{\prime},a_{4},a_{5})\varphi(a_{2}^{\prime},a_{4}^{\prime},a_{6})\varphi(a_{3}^{\prime},a_{5}^{\prime},a_{6}^{\prime})=$
$\displaystyle=\varphi(a_{3},a_{5},a_{6})\varphi(a_{2},a_{4},a_{6}^{\prime})\varphi(a_{1},a_{4}^{\prime},a_{5}^{\prime})\varphi(a_{1}^{\prime},a_{2}^{\prime},a_{3}^{\prime})$
(9)

in the notation of the picture 2. In particular the following lemma fulfills:

######  Lemma 2 (Th. 4 of [7])

Let $\Phi$ be a solution for the STTE, $\varphi$ be a $3$-cocycle of the
tetrahedral complex. Let $V$ be the vector space generated by the elements of
the set $X$. Then let us define a linear operator $A$ on $V^{\otimes 3}$
specifying its values on tensor products of basis vectors. We say that

$A(s)(e_{x}\otimes e_{y}\otimes
e_{z})=\varphi(x,y,z)^{s}(e_{x^{\prime}}\otimes e_{y^{\prime}}\otimes
e_{z^{\prime}})$

if and only if $\Phi(x,y,z)=(x^{\prime},y^{\prime},z^{\prime})$. In this case
$A(s)$ provides a solution for the matrix tetrahedral equation.

###### Remark 1

To construct such matrix solution from the electric solution for the FTE one
should realize the finite-dimentional reduction of the latter. In lemma 6 of
[7] it is demonstrated that the electric solution reduces correctly to the set
$X$ obtained as follows: consider the residue ring
${\mathbb{Z}}/p^{k}{\mathbb{Z}}$, where $p$ is either a prime number of the
form $p=4l+1$, or $p=2$, and $k$ is an integer $\geq 2$ (the Legendre symbol
$\left(\frac{-1}{p}\right)$ equals $1$). We fix one the square roots of $-1$
and call it $\varepsilon\in{\mathbb{Z}}/p{\mathbb{Z}}$. Our set $X$ will be
the following subset of ${\mathbb{Z}}/p^{k}{\mathbb{Z}}$:

$X=\\{x\in{\mathbb{Z}}/p^{k}{\mathbb{Z}}\,\colon\;\;x=\varepsilon\mod p\\}.$

##  2 Commutative family

###  2.1 The commutativity demonstration in $d=1$

Let us present here our own proof of the Maillet result in lemma 1. It
demonstrates the main technique and suggests the path for generalization. In
this section we denote by $R$ and $L$ solutions for the equations (3) and (4)
(do not confuse with equation (1)). We assume that $R$ is invertible. We
present here the Maillet generators (5) in a slightly more algebraic way:

$\displaystyle I_{k}=Tr_{V_{1}\otimes\ldots\otimes V_{k}}L^{\otimes
k}R_{12}\ldots R_{k-1,k},$ (10)

where the tensor product is taken with respect to the auxiliary spaces
$V_{i},$ the product in quantum space $V_{q}$ is implied. To prove the
commutativity of $I_{k}$ we demonstrate that there exists a linear operator
$A\in End(V)^{\otimes k+l}$ such that

$\displaystyle AL^{\otimes k+l}R_{12}\ldots R_{k-1,k}R_{k+1,k+2}\ldots
R_{k+l-1,k+l}A^{-1}=L^{\otimes k+l}R_{12}\ldots R_{l-1,l}R_{l+1,l+2}\ldots
R_{k+l-1,k+l}.$

This yields that the traces of these expressions with respect to the auxiliary
spaces coincide. This in turn produce the identity:

$\displaystyle[I_{k},I_{l}]=0.$

In what follows $Ad_{g}X$ means $gXg^{-1}.$ Let us introduce some accessory
notations: $R_{\overline{1k}}=R_{12}R_{23}\ldots R_{k-1,k}.$ This expression
is subject to some relations

###### Lemma 3

$\displaystyle Ad_{R_{\overline{1k}}}R_{m,m+1}=R_{m+1,m+2},\qquad 1\leq m\leq
k-2.$ (11)

Proof

$\displaystyle R_{12}\ldots R_{m-1,m}R_{m,m+1}R_{m+1,m+2}\ldots
R_{k-1,k}R_{m,m+1}R_{k-1,k}^{-1}\ldots
R_{m+1,m+2}^{-1}R_{m,m+1}^{-1}R_{m-1,m}^{-1}\ldots R_{12}^{-1}$
$\displaystyle=$ $\displaystyle R_{12}\ldots
R_{m-1,m}R_{m,m+1}\underline{R_{m+1,m+2}R_{m,m+1}R_{m+1,m+2}^{-1}}R_{m,m+1}^{-1}R_{m-1,m}^{-1}\ldots
R_{12}^{-1}$ $\displaystyle=$ $\displaystyle R_{12}\ldots
R_{m-1,m}\underline{R_{m,m+1}R_{m,m+1}^{-1}}R_{m+1,m+2}\underline{R_{m,m+1}R_{m,m+1}^{-1}}R_{m-1,m}^{-1}\ldots
R_{12}^{-1}$ $\displaystyle=$ $\displaystyle R_{12}\ldots
R_{m-1,m}R_{m+1,m+2}R_{m-1,m}^{-1}\ldots R_{12}^{-1}=R_{m+1,m+2}.$

$\square$
One another relation is expressed by the

###### Lemma 4

$\displaystyle
Ad_{R_{\overline{2k}}}Ad_{R_{\overline{1,k-1}}}R_{k-1,k}=R_{12}.$ (12)

Proof
We demonstrate the statement by induction. For $k=3$ we have

$\displaystyle
R_{23}\underline{R_{12}R_{23}R_{12}^{-1}}R_{23}^{-1}=R_{23}\underline{R_{23}^{-1}R_{12}R_{23}}R_{23}^{-1}=R_{12}.$

Let the statement be true for $k-1$, then

$\displaystyle Ad_{R_{\overline{2k}}}Ad_{R_{\overline{1,k-1}}}R_{k-1,k}$
$\displaystyle=$ $\displaystyle R_{23}\ldots R_{k-1,k}R_{12}\ldots
R_{k-3,k-2}\underline{R_{k-2,k-1}R_{k-1,k}R_{k-2,k-1}^{-1}}R_{k-3,k-2}^{-1}\ldots
R_{12}^{-1}R_{k-1,k}^{-1}\ldots R_{23}^{-1}$ $\displaystyle=$ $\displaystyle
R_{23}\ldots\underline{R_{k-1,k}}R_{12}\ldots
R_{k-3,k-2}\underline{R_{k-1,k}^{-1}}R_{k-2,k-1}\underline{R_{k-1,k}}R_{k-3,k-2}^{-1}\ldots
R_{12}^{-1}\underline{R_{k-1,k}^{-1}}\ldots R_{23}^{-1}$ $\displaystyle=$
$\displaystyle R_{23}\ldots R_{k-2,k-1}R_{12}\ldots
R_{k-3,k-2}R_{k-2,k-1}R_{k-3,k-2}^{-1}\ldots R_{12}^{-1}R_{k-2,k-1}^{-1}\ldots
R_{23}^{-1}$ $\displaystyle=$ $\displaystyle
Ad_{R_{\overline{2,k-1}}}Ad_{R_{\overline{1,k-2}}}R_{k-2,k-1}=R_{12}.$

$\square$
We may now fabricate an operator $A$ by the formula:

$\displaystyle A=R_{\overline{l,k+l}}R_{\overline{l-1,k+l-1}}\ldots
R_{\overline{1,k+1}}.$ (13)

###### Lemma 5

$\displaystyle Ad_{A}(L^{\otimes k+l}R_{12}\ldots R_{k-1,k}R_{k+1,k+2}\ldots
R_{k+l-1,k+l})=L^{\otimes k+l}R_{12}\ldots R_{l-1,l}R_{l+1,l+2}\ldots
R_{k+l-1,k+l}.$

Proof
Let us note that in virtue of (4) we need to demonstrate only

$\displaystyle Ad_{A}(R_{12}\ldots R_{k-1,k}R_{k+1,k+2}\ldots
R_{k+l-1,k+l})=R_{12}\ldots R_{l-1,l}R_{l+1,l+2}\ldots R_{k+l-1,k+l}$

By the other hand lemma 3 yields

$\displaystyle Ad_{A}(R_{12}\ldots R_{k-1,k})=R_{l+1,l+2}\ldots
R_{k+l-1,k+l}.$ (14)

Lemma 4 provides

$\displaystyle Ad_{A}(R_{k+1,k+2}\ldots R_{k+l-1,k+l})=R_{12}R_{23}\ldots
R_{l-1,l}.$ (15)

The commutativity argument of the right hand sides of (14) and (15) finishes
the proof.
$\square$

This reasoning can be generalized for generating functions of integrals. Let
us introduce a notation:

$\displaystyle S(t)=L^{\otimes
N}(1+tR_{12}+t^{2}R_{12}R_{23}+\ldots+t^{N-1}R_{12}\ldots R_{N-1,N})$

Then

$\displaystyle Q(t)=TrS(t)=\sum_{k=0}^{N-1}t^{k}I_{k}I_{1}^{N-k}.$

If one now considers the conjugation operator

$\displaystyle A=R_{\overline{N,2N}}R_{\overline{N-1,2N-1}}\ldots
R_{\overline{1,N+1}}.$ (16)

then

$\displaystyle Ad_{A}(S(t)\otimes S(u))=S(u)\otimes S(t).$

This immediately implies

$\displaystyle[Q(t),Q(u)]=0.$

###  2.2 Regular $3$-d lattices and statistical models

Consider a periodic three-dimensional lattice of size $K\times L\times M$, we
mark the edges incoming to the node $(i,j,k)$ as
$x_{i,j,k},y_{i,j,k},z_{i,j,k}$. The periodicity conditions imply
$*_{N+1,j,k}=*_{1,j,k},$ and similar identities for other indexes. Consider a
statistical model with the Boltzmann weights in the nodes of the lattice sites
determined by the value of the 3-cocycle $\varphi$ of the tetrahedral complex.
The states are defined as admissible coloring of the edges, i.e. such that in
each node the condition fulfills:

$\displaystyle\Phi(x_{i,j,k},y_{i,j,k},z_{i,j,k})=(x_{i+1,j,k},y_{i,j+1,k},z_{i,j,k+1}).$
(17)

A partition function is defined as follows:

$\displaystyle
Z(s)=\sum_{Col}\prod_{i,j,k}\varphi(x_{i,j,k},y_{i,j,k},z_{i,j,k})^{s}.$ (18)

To explore the ”integrability” of the subsidiary quantum problem one needs to
recognize the layer-to-layer transfer-matrix. In order to determine what it
is, we need another interpretation of the partition function.
We associate a copy of the space $V$ to each line of the lattice. For
convenience, we denote the vertical spaces by characters $V_{ik}$ and the
horizontal ones - by $E_{i}$ and $N_{k}.$
We construct an operator $A_{ik}(s)$ with the chosen $3$-cocycle according to
lemma 2 construction.

Figure 3: 1-layer configuration

Let us define the transfer-matrix by a $1$-layer product:

$\displaystyle T(s)=Tr\prod_{i}\prod_{k}A_{ik}(s).$

The formula implies the product and trace of matrices with respect to
horizontal spaces. This operator acts on the tensor product of vertical
spaces. Here $A_{ik}(s)$ is an operator on the space $E_{i}\otimes
V_{ik}\otimes N_{k}$. It turns out that the partition function takes the form

$\displaystyle Z(s)=Tr_{V_{jk}}T(s)^{L}.$

Such issues as the asymptotic behavior of partition functions with respect to
increasing the size of the lattice may be solved by the study of the spectrum
of the transfer-matrix. The integrability condition, i.e. the possibility of
including the transfer-matrix in a large commutative family, simplifies the
problem of finding the spectrum.

###  2.3 Some consequences of the tetrahedral equation

In this section we give a generalization of the Maillet construction in the
case of the three-dimensional lattice and the transfer-matrix associated with
the solution of the matrix tetrahedral equation. Consider a lattice of size
$K\times L\times M$ and several forms of $1$-layer product:

$\displaystyle\Phi_{(i)*(j)}=\Phi_{(i_{1}\ldots i_{k})*(j_{1}\ldots
j_{m})}=\prod_{s\in\overrightarrow{(1,\ldots,k)}}^{t\in\overrightarrow{(1,\ldots,m)}}\Phi_{i_{s}l_{st}j_{t}}=\prod_{s\in\overrightarrow{(1,\ldots,k)}}\Phi_{i_{s}(l_{s1}\ldots
l_{sm})(j_{1}\ldots
j_{m})}=\prod_{s\in\overrightarrow{(1,\ldots,k)}}\Phi_{i_{s}(l_{s})(j)}.$ (19)

The arrows over indexing sets mean the direction in products: for example

$\displaystyle\prod_{s\in\overrightarrow{(1,\ldots,k)}}A_{s}=A_{1}\ldots
A_{k}.$

The transfer-matrix can be represented as the trace of the expression (19)

$\displaystyle T=I_{1}=Tr_{(i)(j)}\Phi_{(i)*(j)}.$ (20)

Let us write down several identities which are straightforward consequences of
the tetrahedral equations. They can be considered as generalizations of the
$RLL$\- relations.

###### Lemma 6

$\displaystyle\Phi_{123}\Phi_{1(i)(j)}\Phi_{2(i)(l)}\Phi_{3(j)(l)}$
$\displaystyle=$
$\displaystyle\Phi_{3(j)(l)}\Phi_{2(i)(l)}\Phi_{1(i)(j)}\Phi_{123},$ (21)
$\displaystyle\Phi_{(i)(j)1}\Phi_{(i)(l)2}\Phi_{(j)(l)3}\Phi_{123}$
$\displaystyle=$
$\displaystyle\Phi_{123}\Phi_{(j)(l)3}\Phi_{(i)(l)2}\Phi_{(i)(j)1},$ (22)
$\displaystyle\Phi_{(i)(i^{\prime})0}\Phi_{(i)*(j)}\Phi_{(i^{\prime})*(j^{\prime})}\Phi_{0(j)(j^{\prime})}$
$\displaystyle=$
$\displaystyle\Phi_{0(j)(j^{\prime})}\Phi_{(i^{\prime})*(j^{\prime})}\Phi_{(i)*(j)}\Phi_{(i)(i^{\prime})0}.$
(23)

where:

$\displaystyle\Phi_{1(i)(j)}$ $\displaystyle=$
$\displaystyle\Phi_{1(i_{1}\ldots i_{k})(j_{1}\ldots
j_{k})}=\prod_{s\in\overrightarrow{(1,\ldots,k)}}\Phi_{1i_{s}j_{s}},$
$\displaystyle\Phi_{(i)(j)1}$ $\displaystyle=$
$\displaystyle\Phi_{(i_{1}\ldots i_{k})(j_{1}\ldots
j_{k})1}=\prod_{s\in\overrightarrow{(1,\ldots,k)}}\Phi_{i_{s}j_{s}1}.$

We introduce also some twisted versions of solutions for the tetrahedral
equation:

$\displaystyle\Phi_{123}^{L}$ $\displaystyle=$ $\displaystyle
P_{12}\Phi_{123},\qquad\tilde{\Phi}_{123}^{L}=P_{23}\Phi_{123},$
$\displaystyle\Phi_{123}^{R}$ $\displaystyle=$
$\displaystyle\Phi_{123}P_{23},\qquad\tilde{\Phi}_{123}^{R}=\Phi_{123}P_{12}.$

The identities fulfill:

$\displaystyle\Phi_{123}^{L}\Phi_{145}\Phi_{246}\Phi_{356}^{R}$
$\displaystyle=$
$\displaystyle\Phi_{356}^{R}\Phi_{145}\Phi_{246}\Phi_{123}^{L},$ (24)
$\displaystyle\tilde{\Phi}^{L}_{\alpha\beta\gamma}\Phi_{\alpha
12}^{R}\Phi_{\beta 23}\Phi_{\gamma 13}$ $\displaystyle=$
$\displaystyle\Phi_{\beta 23}\Phi_{\gamma 13}\Phi_{\alpha
12}^{R}\tilde{\Phi}^{L}_{\alpha\beta\gamma},$ (25)
$\displaystyle\Phi_{145}^{R}\Phi_{123}^{R}\Phi_{356}^{L}\Phi_{246}^{L}$
$\displaystyle=$
$\displaystyle\Phi_{356}^{L}\Phi_{246}^{L}\Phi_{145}^{R}\Phi_{123}^{R},$ (26)
$\displaystyle\tilde{\Phi}_{123}^{L}\tilde{\Phi}_{145}^{L}\tilde{\Phi}_{246}^{R}\tilde{\Phi}_{356}^{R}$
$\displaystyle=$
$\displaystyle\tilde{\Phi}_{246}^{R}\tilde{\Phi}_{356}^{R}\tilde{\Phi}_{123}^{L}\tilde{\Phi}_{145}^{L}.$
(27)

They perform a role similar to the Yang-Baxter equation in combinatorial
notation. These equalities are also simple consequences of the tetrahedral
equation, we do not provide here the proofs with the only purpose to simplify
the exposition.

###  2.4 Two families

First we pay attention to the connection of the tetrahedral equation and the
Yang-Baxter equations of the following type: for an invertible $\Phi$ the
formula (21) can be transformed to the kind

$\displaystyle\Phi_{123}\Phi_{1(i)(j)}\Phi_{2(i)(l)}\Phi_{3(j)(l)}\Phi_{123}^{-1}$
$\displaystyle=$ $\displaystyle\Phi_{3(j)(l)}\Phi_{2(i)(l)}\Phi_{1(i)(j)}.$

Taking the trace of both parts with respect to indices $1,2,3$, one deduces
that the expression $R_{(i)(j)}=Tr_{1}\Phi_{1(i)(j)}$ satisfies the Yang-
Baxter equation (1). Moreover as a consequence of (22) we have

$\displaystyle\Phi_{(i)(i^{\prime})0}\Phi_{(i)*(j)}\Phi_{(i^{\prime})*(j^{\prime})}\Phi_{0(j)(j^{\prime})}\Phi^{-1}_{(i)(i^{\prime})0}=\Phi_{0(j)(j^{\prime})}\Phi_{(i^{\prime})*(j^{\prime})}\Phi_{(i)*(j)}.$

If one takes the trace of both parts with respect to the spaces
$i,i^{\prime},0$ it turns out that the expression
$L_{*(j)}=Tr_{(i)}\Phi_{(i)*(j)}$ satisfies the $RLL$ relation:

$\displaystyle L_{*(i)}L_{*(j)}R_{(i)(j)}=R_{(i)(j)}L_{*(j)}L_{*(i)}.$

Thus the data

$\displaystyle L_{*(j)}=Tr_{(i)}\Phi_{(i)*(j)},\qquad
R^{\prime}_{(i)(j)}=Tr_{1}\Phi^{R}_{1(i)(j)}$

meets lemma 1 conditions. The immediate consequence of this is that the
following set of operators:

$\displaystyle
I_{0,k}=Tr_{j_{l},l=1,\ldots,k}\prod_{l=\overrightarrow{1,\ldots,k}}L_{*(j_{l})}\prod_{m=\overrightarrow{1,\ldots,k-1}}R^{\prime}_{(j_{m})(j_{m+1})}$

is commutative and contains $I_{1}=Tr_{j}L_{*(j)}=Tr_{ij}\Phi_{(i)*(j)}.$
There is a slightly more general form for $I_{0,k}$

$\displaystyle
I_{0,k}=Tr_{i_{l},j_{l},s_{m}}\prod_{l=\overrightarrow{1,\ldots,k}}\Phi_{(i_{m})*(j_{m})}\prod_{m=\overrightarrow{1,\ldots,k-1}}\Phi^{R}_{s_{m}(j_{m})(j_{m+1})}.$

A similar argument can show that, due to lemma 1 the family

$\displaystyle
I_{n,0}=Tr_{i_{l},j_{l},t_{m}}\prod_{l=\overrightarrow{1,\ldots,n}}\Phi_{(i_{m})*(j_{m})}\prod_{m=\overrightarrow{1,\ldots,n-1}}\Phi^{L}_{(i_{m})(i_{m+1})t_{m}}$

is also commutative and include $I_{1}.$ The main accomplishment of the paper
is that these two families commute among themselves. In order to give a
precise formulation let us introduce some notation.

$\displaystyle B_{k}$ $\displaystyle=$
$\displaystyle\Phi_{(i_{1})\ast(j_{1})}\ldots\Phi_{(i_{k})\ast(j_{k})}=\prod_{\alpha\in\overrightarrow{(1,\ldots,k)}}\Phi_{(i_{\alpha})\ast(j_{\alpha})},$
$\displaystyle R^{s}_{k}$ $\displaystyle=$
$\displaystyle\Phi_{s_{1}(j_{1})(j_{2})}^{R}\ldots\Phi_{s_{k-1}(j_{k-1})(j_{k})}^{R}=\prod_{\alpha\in\overrightarrow{(1,\ldots,k-1)}}\Phi^{R}_{s_{\alpha}(j_{\alpha})(j_{\alpha+1})},$
$\displaystyle L^{s}_{l}$ $\displaystyle=$
$\displaystyle\Phi_{(i_{1})(i_{2})s_{1}}^{L}\ldots\Phi_{(i_{l-1})(i_{l})s_{l-1}}^{L}=\prod_{\alpha\in\overrightarrow{(1,\ldots,l-1)}}\Phi^{L}_{(i_{\alpha})(i_{\alpha+1})s_{\alpha}},$
$\displaystyle A^{p}_{L}$ $\displaystyle=$
$\displaystyle\prod_{\overrightarrow{\alpha}\overleftarrow{\beta}}\Phi_{(i_{\alpha})(i_{\beta})p_{\alpha\beta}},$
$\displaystyle A^{p}_{R}$ $\displaystyle=$
$\displaystyle\prod_{\overrightarrow{\beta}\overleftarrow{\alpha}}\Phi_{p_{\alpha\beta}(j_{\alpha})(j_{\beta})}.$

$p_{\alpha\beta}$ is the set of accessory indices in last two formulas. In
addition, we need two auxiliary elements:

$\displaystyle\Phi_{p}$ $\displaystyle=$
$\displaystyle\prod_{\overrightarrow{\alpha}\overleftarrow{\beta}}\tilde{\Phi}^{L}_{s_{\alpha}p_{\alpha+1,\beta}p_{\alpha,\beta}},$
$\displaystyle\Phi^{*}_{p}$ $\displaystyle=$
$\displaystyle\prod_{\overrightarrow{\beta}\overleftarrow{\alpha}}\tilde{\Phi}^{R}_{p_{\alpha,\beta+1}p_{\alpha,\beta}t_{\beta}}.$

###  2.5 The commutativity proof in $d=2$

###### Definition 1

Let us call a solution of the tetrahedral equation generic if it is invertible
and the operator $Tr_{p}\left(A_{R}^{p}\Phi_{p}^{*}\right)$ is invertible too.

###### Theorem 1

For any generic solution $\Phi$ of the tetrahedral equation the following is
true

$\displaystyle[I_{0,k},I_{n,0}]=0.$

Proof
Let us consider an expression:

$\displaystyle\underbrace{A_{L}^{p}}_{i_{\alpha},i_{\beta},p}\underbrace{\Phi_{p}}_{p,s_{\alpha}}\underbrace{B_{k}}_{i_{\alpha},j_{\alpha}}\otimes\underbrace{B_{l}}_{i_{\beta},j_{\beta}}\underbrace{R_{k}^{s}}_{j_{\alpha},s_{\alpha}}\underbrace{L_{l}^{t}}_{i_{\beta},t_{\beta}}\underbrace{A_{R}^{p}}_{j_{\alpha},j_{\beta},p}\underbrace{\Phi_{p}^{*}}_{p,t_{\beta}}.$
(28)

The indices under each multiplier indicate the spaces in which it acts. Next,
we shall omit the tensor product sign, considering the indices of the
corresponding spaces. This expression can be transmuted to the form:

$\displaystyle
A_{L}^{p}B_{k}B_{l}\boxed{\Phi_{p}R_{k}^{s}A_{R}^{p}}L_{l}^{t}\Phi_{p}^{*}\overset{Lemma\ref{l1}}{=}A_{L}^{p}B_{k}B_{l}\boxed{A_{R}^{p}R_{k}^{s}\Phi_{p}}L_{l}^{t}\Phi_{p}^{*}$
$\displaystyle=\boxed{A_{L}^{p}B_{k}B_{l}A_{R}^{p}}R_{k}^{s}\Phi_{p}L_{l}^{t}\Phi_{p}^{*}\overset{Lemma\ref{l2}}{=}\boxed{A_{R}^{p}B_{l}B_{k}A_{L}^{p}}R_{k}^{s}\Phi_{p}L_{l}^{t}\Phi_{p}^{*}$
$\displaystyle=A_{R}^{p}B_{l}B_{k}A_{L}^{p}R_{k}^{s}L_{l}^{t}\boxed{\Phi_{p}\Phi_{p}^{*}}\overset{Lemma\ref{l3}}{=}A_{R}^{p}B_{l}B_{k}A_{L}^{p}R_{k}^{s}L_{l}^{t}\boxed{\Phi_{p}^{*}\Phi_{p}}$
$\displaystyle=A_{R}^{p}B_{l}B_{k}R_{k}^{s}\boxed{A_{L}^{p}L_{l}^{t}\Phi_{p}^{*}}\Phi_{p}\overset{Lemma\ref{l4}}{=}A_{R}^{p}B_{l}B_{k}R_{k}^{s}\boxed{\Phi_{p}^{*}L_{l}^{t}A_{L}^{p}}\Phi_{p}$
$\displaystyle=A_{R}^{p}\Phi_{p}^{*}B_{l}B_{k}R_{k}^{s}L_{l}^{t}A_{L}^{p}\Phi_{p}.$
(29)

Then we deduce:

$\displaystyle{A_{L}^{p}}{\Phi_{p}}{B_{k}B_{l}}{R_{k}^{s}}{L_{l}^{t}}{A_{R}^{p}}{\Phi_{p}^{*}}({A_{L}^{p}}{\Phi_{p}})^{-1}=A_{R}^{p}\Phi_{p}^{*}B_{l}B_{k}R_{k}^{s}L_{l}^{t}.$

In particular this implies

$\displaystyle
Tr_{i_{\alpha},i_{\beta},p,s_{\alpha}}{B_{k+l}}{R_{k}^{s}}{L_{l}^{t}}{A_{R}^{p}}{\Phi_{p}^{*}}=Tr_{i_{\alpha},i_{\beta},p,s_{\alpha}}A_{R}^{p}\Phi_{p}^{*}B_{l}B_{k}R_{k}^{s}L_{l}^{t}.$

Let us note that the trace procedure factorizes

$\displaystyle
Tr_{i_{\alpha},i_{\beta},s_{\alpha}}\left({B_{k}B_{l}}{R_{k}^{s}}{L_{l}^{t}}\right)Tr_{p}\left({A_{R}^{p}}{\Phi_{p}^{*}}\right)=Tr_{p}\left(A_{R}^{p}\Phi_{p}^{*}\right)Tr_{i_{\alpha},i_{\beta},s_{\alpha}}\left(B_{l}B_{k}R_{k}^{s}L_{l}^{t}\right).$

One may notice that

$\displaystyle\left(Tr_{p}\left(A_{R}^{p}\Phi_{p}^{*}\right)\right)^{-1}Tr_{i_{\alpha},i_{\beta},s_{\alpha}}\left({B_{k}B_{l}}{R_{k}^{s}}{L_{l}^{t}}\right)Tr_{p}\left({A_{R}^{p}}{\Phi_{p}^{*}}\right)=Tr_{i_{\alpha},i_{\beta},s_{\alpha}}\left(B_{l}B_{k}R_{k}^{s}L_{l}^{t}\right).$

Taking a trace over the remaining auxiliary spaces completes the proof:

$\displaystyle
I_{0k}I_{l0}=Tr_{j_{\alpha},j_{\beta},t}\left(Tr_{i_{\alpha},i_{\beta},s_{\alpha}}\left({B_{k+l}}{R_{k}^{s}}{L_{l}^{t}}\right)\right)=Tr_{j_{\alpha},j_{\beta},t}\left(Tr_{i_{\alpha},i_{\beta},s_{\alpha}}\left(B_{l}B_{k}R_{k}^{s}L_{l}^{t}\right)\right)=I_{l0}I_{0k}.$

Proceed now to the proof of lemmas.

###### Lemma 7

$\displaystyle\Phi_{p}R_{k}^{s}A_{R}^{p}=A_{R}^{p}R_{k}^{s}\Phi_{p}.$ (30)

Proof
Let us deduce a useful formula:

$\displaystyle\tilde{\Phi}^{L}_{\alpha\beta\gamma}\Phi^{R}_{\alpha(i)(j)}\Phi_{\beta(j)(k)}\Phi_{\gamma(i)(k)}=\Phi_{\beta(j)(k)}\Phi_{\gamma(i)(k)}\Phi^{R}_{\alpha(i)(j)}\tilde{\Phi}^{L}_{\alpha\beta\gamma}.$
(31)

It is a direct consequence of equation (25). Let us write more thoroughly (30)

$\displaystyle\prod_{\overrightarrow{\alpha}\overleftarrow{\beta}}\tilde{\Phi}^{L}_{s_{\alpha}p_{\alpha+1,\beta}p_{\alpha,\beta}}\prod_{\overrightarrow{\alpha}}\Phi^{R}_{s_{\alpha}(j_{\alpha})(j_{\alpha+1})}\prod_{\overrightarrow{\beta}\overleftarrow{\alpha}}\Phi_{p_{\alpha,\beta}(j_{\alpha})(j_{\beta})}.$

Note that the left and right multipliers commute at different values of
$\beta$ so it is enough to check the relation at fixed $\beta=\beta_{0}$
performing further dressing starting with the multipliers closest to the
center.

$\displaystyle\prod_{\overrightarrow{\alpha}}\tilde{\Phi}^{L}_{s_{\alpha}p_{\alpha+1,\beta_{0}}p_{\alpha,\beta_{0}}}\prod_{\overrightarrow{\alpha}}\Phi^{R}_{s_{\alpha}(j_{\alpha})(j_{\alpha+1})}\prod_{\overleftarrow{\alpha}}\Phi_{p_{\alpha,\beta_{0}}(j_{\alpha})(j_{\beta_{0}})}$
$\displaystyle=$
$\displaystyle\prod_{\overrightarrow{\alpha}}\left(\tilde{\Phi}^{L}_{s_{\alpha}p_{\alpha+1,\beta_{0}}p_{\alpha,\beta_{0}}}\Phi^{R}_{s_{\alpha}(j_{\alpha})(j_{\alpha+1})}\right)\prod_{\overleftarrow{\alpha}}\Phi_{p_{\alpha,\beta_{0}}(j_{\alpha})(j_{\beta_{0}})}.$

Now we will focus attention on how the elements of the left product are
transferred through the right one. This can be done consequently: fix the
index $\alpha=\alpha_{0}$ in the left product

$\displaystyle\tilde{\Phi}^{L}_{s_{\alpha_{0}}p_{\alpha_{0}+1,\beta_{0}}p_{\alpha_{0},\beta_{0}}}\Phi^{R}_{s_{\alpha}(j_{\alpha_{0}})(j_{\alpha_{0}+1})}\prod_{\overleftarrow{\alpha}}\Phi_{p_{\alpha,\beta_{0}}(j_{\alpha})(j_{\beta_{0}})}.$

The only multipliers of the right product which do not commute with the
elements on the left have indices $\alpha=\alpha_{0}$ and
$\alpha=\alpha_{0}+1$. When moving through them we use equality (31)

$\displaystyle\tilde{\Phi}^{L}_{s_{\alpha_{0}}p_{\alpha_{0}+1,\beta_{0}}p_{\alpha_{0},\beta_{0}}}\Phi^{R}_{s_{\alpha}(j_{\alpha_{0}})(j_{\alpha_{0}+1})}\Phi_{p_{\alpha_{0}+1,\beta_{0}}(j_{\alpha_{0}+1})(j_{\beta_{0}})}\Phi_{p_{\alpha_{0},\beta_{0}}(j_{\alpha_{0}})(j_{\beta_{0}})}$
$\displaystyle=$
$\displaystyle\Phi_{p_{\alpha_{0}+1,\beta_{0}}(j_{\alpha_{0}+1})(j_{\beta_{0}})}\Phi_{p_{\alpha_{0},\beta_{0}}(j_{\alpha_{0}})(j_{\beta_{0}})}\Phi^{R}_{s_{\alpha}(j_{\alpha_{0}})(j_{\alpha_{0}+1})}\tilde{\Phi}^{L}_{s_{\alpha_{0}}p_{\alpha_{0}+1,\beta_{0}}p_{\alpha_{0},\beta_{0}}}.$

$\square$

###### Lemma 8

$\displaystyle A_{L}^{p}B_{k}B_{l}A_{R}^{p}=A_{R}^{p}B_{l}B_{k}A_{L}^{p}.$

Proof
The statement is an immediate consequence of equation (22). We will illustrate
the basic techniques of generalization. We write the left part of the
expression:

$\displaystyle\prod_{\overrightarrow{\alpha}\overleftarrow{\beta}}\Phi_{(i_{\alpha})(i_{\beta})p_{\alpha\beta}}\prod_{\overrightarrow{\alpha}}\Phi_{(i_{\alpha})\ast(j_{\alpha})}\prod_{\overrightarrow{\beta}}\Phi_{(i_{\beta})\ast(j_{\beta})}\prod_{\overleftarrow{\alpha}\overrightarrow{\beta}}\Phi_{p_{\alpha\beta}(j_{\alpha})(j_{\beta})}.$

We are going to move the multipliers of the third product through the
multipliers of the second one. For doing this we use a twisting multipliers of
the first and fourth factors. Note that the multipliers of the first and
fourth factors commute except those with both coinciding indices. It remains
to verify that the multiplier order is suitable. We will move the elements of
the third product consequently from the left through the second product. To do
this, the order relation on $\beta$ in the first product should be such that
the junior members stand on the right, and in the fourth on the left.
Similarly, one can check that the order relation on $\alpha$ in the first
product should be such that the senior members are on the right, and in the
fourth on the left.
$\square$

###### Lemma 9

$\displaystyle\Phi_{p}\Phi_{p}^{*}=\Phi_{p}^{*}\Phi_{p}.$

Proof
Let us write the left side of the expression

$\displaystyle\prod_{\overrightarrow{\alpha}\overleftarrow{\beta}}\tilde{\Phi}^{L}_{s_{\alpha}p_{\alpha+1,\beta}p_{\alpha,\beta}}\prod_{\overrightarrow{\beta}\overleftarrow{\alpha}}\tilde{\Phi}^{R}_{p_{\alpha,\beta+1}p_{\alpha,\beta}t_{\beta}}.$

We move the right product through the left one component-wise, for fixed
$\alpha$ on the left and fixed $\beta$ on the right. Let us consider these
multipliers:

$\displaystyle\prod_{\overleftarrow{\beta}}\tilde{\Phi}^{L}_{s_{\alpha_{0}}p_{\alpha_{0}+1,\beta}p_{\alpha_{0},\beta}}\prod_{\overleftarrow{\alpha}}\tilde{\Phi}^{R}_{p_{\alpha,\beta_{0}+1}p_{\alpha,\beta_{0}}t_{\beta_{0}}}.$

The only non-commutative elements of these products have neighboring indices
$\beta=\beta_{0},\beta_{0}+1$ and $\alpha=\alpha_{0},\alpha_{0}+1.$ We write
only them

$\displaystyle\tilde{\Phi}^{L}_{s_{\alpha_{0}}p_{\alpha_{0}+1,\beta_{0}+1}p_{\alpha_{0},\beta_{0}+1}}\tilde{\Phi}^{L}_{s_{\alpha_{0}}p_{\alpha_{0}+1,\beta_{0}}p_{\alpha_{0},\beta_{0}}}\tilde{\Phi}^{R}_{p_{\alpha_{0}+1,\beta_{0}+1}p_{\alpha_{0}+1,\beta_{0}}t_{\beta_{0}}}\tilde{\Phi}^{R}_{p_{\alpha_{0},\beta_{0}+1}p_{\alpha_{0},\beta_{0}}t_{\beta_{0}}}.$

Note that the right pair can be moved through the left pair of multipliers
according to (26).
$\square$

###### Lemma 10

$\displaystyle A_{L}^{p}L_{l}^{t}\Phi_{p}^{*}=\Phi_{p}^{*}L_{l}^{t}A_{L}^{p}.$

Proof
Analogously to lemma 8.
$\square$

##  3 Conclusion

The choice of the notations $I_{l,0}$ and $I_{0,k}$ used in the main part of
the work pursued a definitive goal. We hope that the following hypothesis is
correct

###### Hypothesis 1

For any generic solution of the tetrahedral equation it can be constructed a
two-parameter commutative family $I_{l,k}$, which includes two families
presented above.

###### Remark 2

Note that in the work [1] it is constructed a two-parameter family of
operators commuting with a layer-by-layer transfer-matrix related to the
special solution of the tetrahedral equation. It is curious to compare the
result in the $q$-oscillator realization case and the result presented here.

However, the main result of this work reveals some important perspectives:

  * •

Primarily, it provides the opportunity to study the spectra of the
corresponding quantum models and models of statistical physics in dimension
$d=3.$ We hope that in this case the Bethe ansatz technique also can be
applied.

  * •

Another interesting direction is the generalization of the notion of
integrability in the case of $2$-dimensional surfaces comprising the classical
case. It is interesting to juxtapose the approach of this work with the
language of paper [8] on the theory of the Hitchin systems on surfaces. We
would also be enthusiastic in the study of moduli spaces of 2-bundles on
surfaces and an appropriate analogue of the Hitchin theory. In this case one
has a significant difficulty in constructing nonabelian gerbs.

  * •

In [7] we present a construction of quasi-invariant of $2$-knots in the form
of a partition function on the graph of double points of the $2$-knot diagram.
This structure is resembling to the approach of [9]. The similarity of the
partition function expressions is intriguing and gives hope for further
development activities in low-dimensional topology and topological quantum
field theory. Presumably a certain connection of this method with four-
dimensional quantum field theories, like the BF-theory, can be established.

## Acknowledgments

The work was partially supported by the RFBR grant of Russian Foundation of
Basic Research 14-01-00012, grant of the support of scientific schools
4833.2014.1, and the grant of the Dynasty Foundation.

## References

  * [1] V.V. Bazhanov, S.M. Sergeev, Zamolodchikov’s Tetrahedron Equation and Hidden Structure of Quantum Groups. J.Phys.A 39:3295-3310,2006,
  * [2] J.M Maillet, Lax equations and quantum groups, Phys. Lett. B 245, 480 (1990).
  * [3] A. B. Zamolodchikov, Tetrahedra equations and integrable systems in three-dimensional space, Zh. Eksp. Teor. Fiz. 79 (1980) 641 664. [English translation: Soviet Phys. JETP 52 (1980) 325-326].
  * [4] J. Hietarinta, Permutation-type solutions to the Yang-Baxter and other n- simplex equations. J. Phys. A: Math. Gen., 1997, Vol. 30, 4757.
  * [5] R. J. Baxter, Exactly Solved Models in Statistical Mechanics. Academic, London, 1982.
  * [6] R.M. Kashaev, I.G. Korepanov, S.M. Sergeev, Functional tetrahedron equation. Theor Math Phys (1998) 117: 1402.
  * [7] I.G. Korepanov, G.I. Sharygin, D.V. Talalaev, Cohomologies of n-simplex relations. Mathematical Proceedings of the Cambridge Philosophical Society. 2016. Vol. 161, no. 2. P. 203 222.
  * [8] D.V.Osipov, A.B.Zheglov, On some questions related to the Krichever correspondence, Mathematical Notes, 2007, Vol. 81, No. 4, pp. 467-476
  * [9] Carter, J.S., Jelsovsky, D., Langford, L., Kamada, S., Saito, M., Quandle cohomology and state-sum invariants of knotted curves and surfaces, Trans. Amer. Math. Soc. 355 (2003), no. 10, 3947-3989.


# Self-Supervised Learning of Object Segmentation
from Unlabeled RGB-D Videos

Shiyang Lu1, Yunfu Deng2, Abdeslam Boularias1, Kostas Bekris1 1The authors are
affiliated with the Department of Computer Science at Rutgers University, New
Brunswick, NJ, 08901, USA. Email: {shiyang.lu, abdeslam.boularias,
<EMAIL_ADDRESS>2This author is affiliated with the Department of
Electrical and Computer Engineering at Rutgers University, New Brunswick, NJ,
08901. This work is supported by NSF awards 1734492, 1846043 and 2132972.

###### Abstract

This work proposes a self-supervised learning system for segmenting rigid
objects in RGB images. The proposed pipeline is trained on unlabeled RGB-D
videos of static objects, which can be captured with a camera carried by a
mobile robot. A key feature of the self-supervised training process is a
graph-matching algorithm that operates on the over-segmentation output of the
point cloud that is reconstructed from each video. The graph matching, along
with point cloud registration, is able to find reoccurring object patterns
across videos and combine them into 3D object pseudo labels, even under
occlusions or different viewing angles. Projected 2D object masks from 3D
pseudo labels are used to train a pixel-wise feature extractor through
contrastive learning. During online inference, a clustering method uses the
learned features to cluster foreground pixels into object segments.
Experiments highlight the method’s effectiveness on both real and synthetic
video datasets, which include cluttered scenes of tabletop objects. The
proposed method outperforms existing unsupervised methods for object
segmentation by a large margin.

##  I Introduction

Autonomous robots should be able to reason about objects even when human
supervision is not available [1]. This work considers a setup where a robot
navigates in a static environment and passively collects RGB-D videos. The
environment contains unknown rigid objects, which rest stably, potentially in
cluttered configurations, on supporting surfaces like tabletops. The same
object instances can reappear in various scenes with arbitrary 6D poses while
being partially occluded by different surrounding objects. This raises the
following question: Can a robot learn to correctly segment objects in RGB
images given its prior experience of collecting unlabeled videos that contain
the involved objects without any human supervision?

Figure 1: Self-Supervised Learning Pipeline. Top: Step 1. Object over-
segmentation on the 3D reconstruction of each video. Step 2. Generating a
distinctive feature for each 3D segment by training an initial feature
extractor $\phi_{1}$ using contrastive learning. Step 3. A graph-matching
algorithm operates over the 3D segments and their $\phi_{1}$ features to
identify sets of segments to be grouped as objects. Step 4. Using contrastive
learning to train another feature extractor $\phi_{2}$, which uses as positive
examples pixels that belong to the same hypothesized objects. Bottom: Upon
inference, given an RGB image and a foreground mask, the network $\phi_{2}$
generates pixel-wise features as input to a clustering algorithm for object
segmentation.

While some objects, such as a coke can or a sugar box, can be relatively
easily singled out from a scene due to their characteristic texture or simple
geometries, other more complex and articulated objects pose a more serious
challenge in the absence of prior knowledge. For example, a lamp can have a
concave lampshade, a gooseneck, and a base. It’s hard for a robot to tell from
a single snapshot if they are individual objects or parts of the same object.
This paper argues that robots can address this problem without prior knowledge
or human supervision by simply collecting videos of scenes, which can be
extended to a lifelong learning process. The idea is to identify object parts
that appear simultaneously in the same spatial arrangement in different
scenes. The key contribution of this work towards this idea is a graph-
matching algorithm that identifies reoccurring sets of neighboring 3D object
parts. The overall approach utilizes the results of the matching algorithm to
train a pixel-wise feature extractor through contrastive learning, which can
be directly consumed by a clustering method for object segmentation.

The proposed pipeline is shown in Fig. 1. The learning system receives as
input a collection of unlabeled RGB-D videos from multiple scenes and returns
a pixel-wise feature extractor $\phi_{2}$ that can generate distinctive object
features for segmentation. The system first performs (over-)segmentation of
the 3D point cloud for each input video resulting in 3D segments corresponding
to simple geometries. This is done by reconstructing each 3D scene, removing
the background using plane detection, and segmenting the scene using existing
non-learning, geometry-based methods [2, 3]. The 3D segments are projected to
the 2D frames, and a pixel-wise feature extractor $\phi_{1}$ is learned to
generate distinctive features for each 3D segment through contrastive
learning, similar to MaskContrast [4]. The 3D segments are then abstracted as
nodes of a graph, where edges connect adjacent nodes, and nodes are
represented by their features according to $\phi_{1}$. Reoccurring graph
patterns are searched across different scenes by a feature-based, sub-graph
matching algorithm [5]. Graph matching is used here as a mechanism to identify
candidate pairs of object instances with multiple parts. The corresponding
point clouds of the candidates are registered and matched nodes with low
registration scores are pruned out. The remaining matched nodes and the
segments represented by those nodes are considered parts of the same object.
Then, a second network $\phi_{2}$ is learned using contrastive learning to
generate pixel-wise features of object instances. During inference, given an
RGB image and a foreground mask, $\phi_{2}$ returns pixel-wise object
features, which are then clustered using a variant of the mean-shift algorithm
[6, 7] to output the object segmentation of the input RGB image.

The experimental results of Section VI show that the proposed technique
significantly outperforms multiple baselines both on real-world and photo-
realistic synthetic datasets.

##  II Related Work

Self-Supervised Learning. Many recent studies on self-supervised learning [8,
9, 10, 11, 12] focus on generating a pre-trained model from a large dataset
using pseudo labels without human supervision. Given the pre-trained model,
downstream tasks (e.g., object segmentation) can be trained with less
annotated data. Many of the existing methods require object region proposals
that contain one or very few salient objects. This object localization process
is usually performed by 2D region proposals [13] and saliency detection [4]
for static images, or object tracking [14, 15] and optical flow [16, 17] for
videos. Nevertheless, they are not as suitable for cluttered scenes [18] as
the errors introduced during region proposal are often too large. Meanwhile,
generating a pre-trained model is not ideal as it still requires manually
labeled ground truth to finetune for downstream tasks, such as segmentation.
This work, on the other hand, focuses on finding object pseudo labels that can
be directly used for training.

Segmentation without Manual Annotation. Many unsupervised methods have been
proposed for object/scene segmentation. Notable non-learning methods are Mean
Shift [6] for image segmentation, and LCCP [2], SymSeg [3], CPC [19] for point
cloud segmentation. Some efforts have studied RGB-D segmentation for unknown
objects, including UOC [20], UOIS [21], RICE [22], SD-MaskRCNN [23]. These
methods typically train a neural network on synthetic data, where ground-truth
labels are automatically generated. Fine-tuning these models on real data is
nontrivial without manual annotation. Recently, some work has been done on
unsupervised semantic segmentation. PiCIE [24] uses invariance to photometric
effects and equivariance to geometric transformations as an inductive bias.
STEGO [25] proposes a novel contrastive loss function that encourages features
to form compact clusters while preserving their relationships across the
corpora.

Unsupervised Object Discovery This work is also related to unsupervised object
discovery in general. Some representative works include [26], which is an
object detection method that uses part-based matching with bottom-up region
proposals, and [27] that formulate the problem as ranking amenable to
distributed methods available for eigenvalue problems and link analysis.

##  III Problem Setup and Notation

This work addresses the problem of object segmentation given an RGB-D image
$I$ of a novel scene, which contains various, potentially cluttered objects.
Each object is assumed to have appeared before in a collection of $m$
unlabeled raw RGB-D videos $V=\\{V_{1},V_{2},...,V_{m}\\}$ that were passively
recorded by a mobile robot while observing static scenes. There are $K$ unique
object instances $O=\\{O_{1},O_{2},...,O_{K}\\}$ that the robot can observe.
Image frame at time $t$ in the training $i$-th video $V_{i}$ is denoted as
$I^{t}_{i}$, i.e., $V_{i}=(I^{1}_{i},I^{2}_{i},...,I^{N_{i}}_{i})$, where
$N_{i}$ is the total number of frames in $V_{i}$. Each frame $I^{t}_{i}$ is
composed of a color and a depth image denoted as $I^{t}_{i}.color$ and
$I^{t}_{i}.depth$ respectively. Each frame $I$ of a video, as well as the test
image, may contain only a subset of the object set $O$, denoted as
$O^{I}\subseteq O$. Background objects, such as furniture, are assumed to be
known a priori and are automatically removed from the RGB-D images. No
additional information, such as the number of objects in each scene, or labels
that help with segmentation is available.

##  IV Object Segmentation during Testing

During online inference, a learned function
$\phi_{2}(x):\mathcal{X}\rightarrow\mathcal{Z}$ is used to extract a
normalized feature vector $z$ for each pixel $x$ in the given test RGB image.
A clustering algorithm is applied over the foreground pixels based on their
extracted features. This foreground mask can be generated with a saliency
detection method, such as BasNet [28] and DeepUSPS [29]. See the example of
Fig. 2.

Figure 2: Example of saliency detection [28] for generating the foreground
mask (right) given RGB input (left).

Since the number of objects in a given image is not known, clustering methods,
such as K-Means, which require this input, are not applicable. Thus, this work
adapts a variant of the von Mises-Fisher mean-shift [7] for feature
clustering. This variant not only considers the feature similarity of pixels
but also the 2D proximity of pixels. A simple post-processing step is applied
so that pixels in a feature cluster are separated if the corresponding pixels
do not form a connected component. The online inference is computationally
efficient ($\sim$ 6fps). The detailed training process of $\phi_{2}$ is
described in the following section.

##  V Self-Supervised Learning Pipeline

###  V-A Video Pre-processing

Given a set of unlabeled RGB-D videos of static objects, a dense
reconstruction method, where KinectFusion [30] is adopted in this work, is
used to estimate the camera pose for each frame and reconstruct a 3D scene for
each video. The foreground 3D region of interest is extracted by removing
known background objects, e.g. table. And the 2D foreground masks during
training are generated by projecting the filtered point cloud to individual 2D
frames. 3D reconstruction is employed here instead of individual depth images
for multiple reasons: a) It allows to generate pseudo labels on the
reconstructed point cloud once and then acquiring consistent labels for
segments on all of the frames of each video via 3D to 2D projection, b) It
increases the visibility of each object given multiple views, which reduces
issues due to occlusions that can arise for individual viewpoints, c) It
avoids having to find correspondences between segments across frames, which
can be error-prone.

###  V-B Object Over-Segmentation

Each reconstructed 3D scene is over-segmented into a set of small segments
corresponding to simple geometries. Ideally, this over-segmentation should be
consistent across different scenes, where consistency means that the same
object surface is segmented in the same way across different scenes. This work
first adopts LCCP [2] for this purpose. LCCP is a non-learning segmentation
method based on the local convexity of point clouds. It is able to segment
convex objects, such as a box, into a whole, and segment complex objects into
small, approximately convex parts. During the development of this work, it is
found that LCCP can fail on certain objects with a large concave surface, such
as a bowl.

To alleviate this issue, SymSeg [3] is introduced to assist with improving
(over-)segmentation quality for symmetric concave objects. SymSeg first
detects symmetries in a point cloud and then uses them for segmentation. If no
symmetry is detected, then SymSeg is not applied. In this work, only
rotational symmetries are used, since reflection symmetries often lead to
under-segmentation in clutter. A simple strategy to fuse the results of LCCP
and SymSeg is illustrated in Fig. 3. A 3D segment generated by LCCP is first
split into multiple parts if an overlapping symmetric segment is detected by
SymSeg. LCCP segments are then merged if they appear in the same symmetric
segment of SymSeg, with an exception when both parts share the same symmetric
axis (e.g. a bottle standing in a bowl). Small segments with less than 200
points are pruned. Note that this work does not argue that this combination is
the best for over-segmentation, but they work sufficiently well as a submodule
in the pipeline. Any recent progress in class-agnostic part segmentation may
benefit this work.

Figure 3: Splitting and Merging Segmentation Results of LCCP and SymSeg. (a)
Reconstructed point cloud of a “Jell-O” box in a bowl. (b) Irregular segments
generated by LCCP due to notable concavity. (c) Detected rotational axis and
segments generated by SymSeg. (d) Splitting segments. (e) Merging segments.

###  V-C Contrastive Learning

To generate a distinctive feature for each 3D segment, contrastive learning is
adopted to train a pixel-wise feature extractor $\phi$. The features will be
first used during graph matching and later in object segmentation. 3D segments
are projected to 2D pixels using estimated camera poses of video frames. Given
the assumption that points in the same segment belong to the same object and
their features should be similar, the embeddings of a pair of pixels $(i,j)$
are pulled closer if they belong to the same 3D segment, or pushed away
otherwise. In practice, $P\gg N$, where $P$ is the number of foreground pixels
per image, and $N$ is the number of segments per scene. The number of pixel
pairs is $O(P^{2})$, which results in a computationally expensive sampling
process. To alleviate this issue, the strategy from MaskContrast [4] is
adopted to reduce the computational complexity to $O(PN)$. Let $M_{i}$ be the
set of pixels from segment $i$. Define the mean pixel embedding of $M_{i}$ as
$\bar{z}_{i}=\frac{1}{|M_{i}|}\sum_{k=1}^{|M_{i}|}z^{k}_{i}$. The mean feature
$\bar{z}_{i}$ is used to represent the features of all pixels in $M_{i}$.
Given positive pairs $(z_{k},\bar{z}_{i})$, for $k\in M_{i}$, and negative
pairs ($z_{k},\bar{z}_{j})$, for $k\not\in M_{j}$, the contrastive loss for
each foreground pixel $k$ is defined as:
$\mathcal{L}_{k}=-log\frac{exp(z_{k}\cdot\bar{z}_{i})/\tau}{\sum_{j=1}^{N}exp(z_{k}\cdot\bar{z}_{j}/\tau)}$,
where $\tau$ is a temperature hyperparameter that has a constant value of
$0.07$ in all our experiments (as in MoCo [9]). The mean feature $\bar{z}_{i}$
is used to represent the 3D segment $i$ after training.

Figure 4: Graph Matching. (a) Colored 3D reconstruction of two scenes. (b) 3D
segments, and corresponding graphs, resulting from over-segmentation of the
point clouds. (c) Matched nodes from graph matching are highlighted. Parts of
the “dumbbell” and the “lamp” are correctly matched, while the “bleach
cleanser” and the “master chef can” are incorrectly matched. (d) Matched
connected components are shown in the same color. (e) 3D point cloud
registration of matched components. (f) Segments/nodes that do not register
well (such as the “bleach cleanser” and the “master chef can”) are rejected
before merging the remaining segments into objects.

###  V-D Graph Matching

For each scene, a graph is generated by linking each 3D segment to a node and
defining an edge between every two segments that share a boundary in the 3D
space. Reoccurring subgraphs across multiple graphs (i.e., scenes) are
hypothesized objects since it is unlikely for two separate objects to always
appear in the same poses relative to each other in all the scenes encountered
by the robot. To find these reoccurring subgraphs, this work adapts an
algorithm based on error-tolerant, minimum-cost subgraph matching [5]. Given a
source graph $G_{1}=(\mathcal{V}_{1},\mathcal{E}_{1})$ and a target graph
$G_{2}=(\mathcal{V}_{2},\mathcal{E}_{2})$, the algorithm identifies a subgraph
$S\subseteq G_{2}$, that minimizes the cost of matching $S$ to a subgraph in
$G_{1}$ given both structural and feature distortions. This optimization
problem is formulated into a binary linear program (BLP) with the following
objective function:

$\vspace{-.1in}\footnotesize\begin{split}J=\min_{x,y,\alpha,\beta}\Big{(}&\sum_{i\in\mathcal{V}_{1}}\sum_{k\in\mathcal{V}_{2}}x_{i,k}\cdot
c(i\rightarrow k))+\sum_{i\in\mathcal{V}_{1}}\alpha_{i}\cdot
c(i\rightarrow\epsilon)+\\\
&\sum_{ij\in\mathcal{E}_{1}}\sum_{kl\in\mathcal{E}_{2}}y_{ij,kl}\cdot
c(ij\rightarrow kl)+\sum_{ij\in\mathcal{E}_{1}}\beta_{ij}\cdot
c(ij\rightarrow\epsilon)\Big{)}\end{split}\vspace{-.1in}$ (1)

where $x_{i,k}$, $y_{ij,kl}$, $\alpha_{i}$ and $\beta_{ij}$ are binary
variables to optimize: $x_{i,k}$ expresses the mapping of node $i$ from source
graph $G_{1}$ to node $k$ from target graph $G_{2}$; $y_{ij,kl}$ expresses the
mapping of edge $(i,j)$ from source graph $G_{1}$ to edge $(k,l)$ from target
graph $G_{2}$; deletion variables $\alpha_{i}$ and $\beta_{ij}$ are set to be
1, if node $i$ and edge $(i,j)$ are deleted, respectively, according to the
match. This can happen when the 3D segment corresponding to node $i$ in the
source scene does not appear anywhere in the target scene. The cost of node
mapping $c(i\rightarrow k)$ is defined as the cosine distance between the
corresponding segment features $\bar{z}_{i}$ and $\bar{z}_{k}$. The cost of
edge mapping $c(ij\rightarrow kl)$ is set to 0 here as no edge feature is
provided. The cost $c(i\rightarrow\epsilon)$ (or $c(ij\rightarrow\epsilon)$)
expresses the cost of not assigning a vertex (or edge) of the source graph to
a vertex (or edge) in the target graph. Both costs are set to 0.1 in the
implementation, which can be viewed as thresholds for rejection.

$\sum_{k\in\mathcal{V}_{2}}x_{i,k}\leq 1,\forall i\in\mathcal{V}_{1};\hskip
7.22743pt\sum_{i\in\mathcal{V}_{1}}x_{i,k}\leq 1,\forall
k\in\mathcal{V}_{2}\\\ $ (2a)
$\alpha_{i}=1-\sum_{k\in\mathcal{V}_{2}}x_{i,k},\forall
i\in\mathcal{V}_{1};\hskip
7.22743pt\beta_{ij}=1-\sum_{kl\in\mathcal{V}_{2}}y_{ij,kl},\forall
ij\in\mathcal{E}_{1}\\\ $ (2b) $\begin{split}\sum_{l\in\mathcal{V}_{2}\textrm{
s.t. }kl\in\mathcal{E}_{2}}y_{ij,kl}\leq x_{i,k},\forall
ij\in\mathcal{E}_{1},\forall k\in\mathcal{V}_{2};\hskip 7.22743pt\\\
\sum_{k\in\mathcal{V}_{2}\textrm{ s.t. }kl\in\mathcal{E}_{2}}y_{ij,kl}\leq
x_{j,l},\forall ij\in\mathcal{E}_{1},\forall l\in\mathcal{V}_{2}\end{split}$
(2c) $\begin{split}x_{i,k}\in\\{0,1\\},\forall i\in\mathcal{V}_{1},\forall
k\in\mathcal{V}_{2};\hskip 7.22743pt\\\ y_{ij,kl}\in\\{0,1\\},\forall
ij\in\mathcal{E}_{1},\forall kl\in\mathcal{E}_{2}\end{split}$ (2d)

The objective function is subject to constraints 2a-2d. Constraints 2a
indicate that each node from either graph can be mapped to at most one node
from the other graph. Constraint 2b indicates that if a node in $G_{1}$ is not
matched to any vertex in $G_{2}$, then it must be deleted. Constraints 2c
indicate that an edge should be mapped only if the corresponding vertices are
also mapped. The constraints 2d ensure that the variables to be optimized are
binary variables. Note that constraints 2b are implicitly respected given
constraints 2a and 2c. Thus, the deletion variables $\alpha_{i}$ and
$\beta_{ij}$ in equation 1 can be removed to reduce the search space resulting
in the following objective function:

$\footnotesize\begin{split}J=\min_{x,y}\Big{(}&\sum_{i\in\mathcal{V}_{1}}\sum_{k\in\mathcal{V}_{2}}x_{i,k}\big{(}c(i\rightarrow
k))-c(i\rightarrow\epsilon)\big{)}\\\
+&\sum_{ij\in\mathcal{E}_{1}}\sum_{kl\in\mathcal{E}_{2}}y_{ij,kl}\big{(}c(ij\rightarrow
kl)-c(ij\rightarrow\epsilon)\big{)}\\\
+&\sum_{i\in\mathcal{V}_{1}}c(i\rightarrow\epsilon)+\sum_{ij\in\mathcal{E}_{1}}c(ij\rightarrow\epsilon)\Big{)}\\\
\end{split}$ (3)

Furthermore, since the 3D segment graphs are undirected, constraints 2c can be
further combined into a single one, i.e.

$\vspace{-.05in}\footnotesize\sum_{l\in\mathcal{V}_{2}\textrm{ s.t.
}kl\in\mathcal{E}_{2}}y_{ij,kl}\leq x_{i,k}+x_{j,k},\hskip 7.22743pt\forall
ij\in\mathcal{E}_{1},\forall k\in\mathcal{V}_{2}$ (4)

The final formulation is to minimize the objective function according to Eq. 3
given the constraints of Eqs. 2a, 4 and 2d. Each weakly connected component in
every 3D scene graph is matched against all the weakly connected components in
all other scenes. Note that source and target graphs are not commutative,
i.e., $J(G_{1},G_{2})\neq J(G_{2},G_{1})$.

###  V-E False Matches Pruning

This optimization returns matched node pairs with minimum cost, which are used
to find matched subgraphs. Three criteria are used to prune falsely matched
subgraphs.
1. A threshold $T_{1}$ is set so that only pairs of subgraphs with a total matching cost $J$ below this threshold are considered valid matches. The implementation sets $T_{1}$ to be proportional to $N=|\mathcal{V}_{1}|+|\mathcal{E}_{1}|$, i.e., $T_{1}=\alpha N$, because the total cost is a sum over both node and edge costs. A large graph is likely to generate a higher cost than a small one. The empirical ratio $\alpha$ is set to be $0.1$.
2. The matching pairs of subgraphs that satisfy criteria 1 are filtered by RANSAC-based point cloud registration [31]. An empirical threshold $T_{2}=0.9$ is set for both precision and recall of the registered point cloud of each pair of matched nodes. If a pair of nodes does not meet this threshold, the nodes will not be merged with other nodes in their respective subgraphs, as shown in Fig. 4.
3. If an object is found to be isolated in a scene, which is performed automatically, then other segments should not be merged with this object. To achieve this, if a source graph $G_{1}=(\mathcal{V}_{1},\mathcal{E}_{1})$ is matched with a subgraph $S$ of the target graph $G_{2}$, such that $S=(\mathcal{V}^{\prime},\mathcal{E}^{\prime}),|\mathcal{V}_{1}|=|\mathcal{V}^{\prime}|$, then any matched subgraph $\hat{S}=(\hat{\mathcal{V}},\hat{\mathcal{E}})\subseteq G_{2}$ that contains all the nodes of $S$ as a proper subset, i.e., $\mathcal{V}^{\prime}\subset\hat{\mathcal{V}}$, will be filtered.

Once the pruning is completed, the remaining nodes in the matched graphs are
merged. The merged segments are considered objects and projected back to each
frame of the video as object masks for training. The training process for the
network $\phi_{2}$ is the same as in Sec. V-C.

##  VI Experimental Results

###  VI-A Dataset

Two video datasets are used for experiments and evaluation, which mimic the
behavior of a robot navigating around a table with cluttered objects on top of
it [32]. Some sample frames are shown in Fig. 5. The first one is a physics-
aware photo-realistic video dataset generated by BlenderProc [33]. All 17 toys
from the HomeBrewedDB dataset [34] are selected to create random scenes in
simulation. For each scene, 12 unique toys are randomly selected and dropped
on a surface. A virtual camera captures a video of each scene by rotating
around it at a fluctuating height. In total, 36 videos are collected with 800
frames each. The second dataset is collected in real environments with 20
unique objects. Among them, 13 objects are from the YCB dataset [35], and 7
objects with more complicated geometries: “lamp”, “dumbbell”, “toy dinosaur”,
“toy duck”, “Clorox bottle”, “detergent”, and “toy Charmander” are included.
Each of these non-convex objects contains several sub-parts of varying shapes.
The real dataset contains 30 videos with 10 unique objects and $\sim$1000
frames each. The ground-truth segmentation for the real dataset is manually
annotated in 3D space for each scene and then projected in 2D frames.
Background removal in these scenes is performed using RANSAC [31] for
detection and removal of the support plane.

Figure 5: Datasets. (a) A photo-realistic synthetic dataset of 36 videos using
all toys from the HomebrewedDB dataset [34] rendered by BlenderProc [33]. (b)
A manually collected real dataset with 30 videos and 20 unique objects.

###  VI-B Evaluation Metrics

Standard metrics for object segmentation are used, i.e., average precision,
recall, and Intersection over Union (IoU). The association of predicted and
ground-truth segments is performed by using the Hungarian algorithm [36] where
the cost of association is based on the IoU of two segments. To remove the
factor of imperfect foreground detection, ground-truth foreground masks are
provided for all methods, and only the segmentation of foreground pixels is
evaluated. The segmentation results are tested on every 10 frames selected
from the testing videos since neighboring frames in a 30 FPS video are very
similar. The testing frames are selected from videos that are different from
the training ones.

###  VI-C Baseline Methods

A set of baseline methods are considered for comparison. A. DeepLabV3+ [37] is
a widely-used segmentation model for RGB images. This baseline is trained with
supervision, which is considered an upper bound. The proposed method also uses
DeepLabV3+ as a backbone network $\phi$ for pixel-wise feature extraction by
removing its classification layer and adding a 16-dimensional embedding layer.
B. UOIS [21] is an RGB-D object segmentation method designed for unseen
tabletop objects. RICE [22] is a follow-up work over UOIS, which refines its
segmentation results. These methods are trained over a large amount of
synthetic data with ground-truth annotations. Since they can barely be
finetuned on datasets without ground-truth annotations, weights provided by
the authors are directly used for testing. C. LCCP [2] does object
segmentation based on local convexity. Unlike its usage in the proposed
method, which does segmentation on the whole reconstructed point cloud, here
it performs segmentation on the point cloud that is converted from each
testing frame using the camera intrinsic matrix. D. SD-MaskRCNN [23] is a
method for unknown object segmentation similar to UOIS. It is also trained
with simulated data with ground truth but only takes depth images as input. E.
PiCIE [24] is an unsupervised semantic segmentation method using Invariance
and Equivariance in Clustering. F. STEGO [25] is another unsupervised
segmentation method, which distills correspondences between images into a set
of class labels using a contrastive loss. The proposed method, PiCIE (E) and
STEGO (F) are trained from scratch and tested using 3-fold cross-validation on
the unlabeled videos, where background pixels are masked out.

| Real Dataset | Synthetic Dataset
---|---|---
Method | Prec. | Recall | IoU | Prec. | Recall | IoU
A. [37] | 0.948 | 0.947 | 0.910 | 0.977 | 0.932 | 0.915
B. [21]+[22] | 0.859 | 0.753 | 0.711 | 0.963 | 0.854 | 0.838
C. [2] | 0.882 | 0.749 | 0.706 | 0.993 | 0.815 | 0.820
D. [23] | 0.782 | 0.773 | 0.662 | 0.812 | 0.751 | 0.663
E. [24] | 0.427 | 0.342 | 0.230 | 0.361 | 0.326 | 0.158
F. [25] | 0.318 | 0.343 | 0.162 | 0.325 | 0.337 | 0.142
Ours | 0.925 | 0.911 | 0.870 | 0.938 | 0.925 | 0.880
TABLE I: Segmentation results for different methods. Method A is a supervised
solution provided as an upper bound of efficiency. The best results among
unsupervised methods are in bold.

###  VI-D Quantitative Results

Quantitative results of the proposed and baseline methods are shown in Table
I. The proposed method works well on both real and synthetic datasets. It
outperforms all baseline methods in terms of mean IoU except A, which was
trained with supervision. B and D work well given that they are designed for
unknown object segmentation and are only trained in simulation, but they can
barely be improved without manual annotation on the test dataset. It is
surprising that the unsupervised semantic segmentation methods E and F
performed poorly on this task. This may be due to the lack of careful tuning
of hyperparameters (default is used) and the lack of inductive bias provided
for segmenting object instances. The work argues that learning based on
geometric inductive bias and object reappearance across different scenes is
effective for segmenting previously seen objects when deploying a robot in
real environments.

###  VI-E Intermediate 3D Segmentation Results

To assess the advantage of using graph matching for object pseudo-label
generation, intermediate point cloud segmentation results are provided. The
proposed method is compared against LCCP [2] and SymSeg [3] on the
reconstructed object point cloud of each video. Intermediate quantitative and
qualitative point cloud segmentation results are provided in Table II and Fig.
6 respectively. The benefits of combining LCCP with SymSeg are not entirely
shown in numbers, as SymSeg sometimes can introduce object-level segmentation
errors, such as over-segmenting a round cup and its handle. Nevertheless, it
makes the 3D over-segmentation more consistent for cases like in Fig. 3. The
over-segmented objects, however, can be recovered by the graph-matching
algorithm if an object pattern is found across scenes. The improvement brought
by graph matching is even more pronounced over the more challenging geometries
present in the synthetic dataset.

Figure 6: Intermediate Qualitative 3D Segmentation Results (a) Reconstructed object point clouds in their original color. (b) Segmentation results from LCCP + SymSeg. (c) Object pseudo labels after graph matching and pruning given over-segmentation results as inputs. A failure case of over-segmentation is shown in the last row, where part of the ”pitcher” and ”cup” are not separated and the ”cup” is segmented into inconsistent pieces. This cannot be addressed during graph matching. | Real Dataset | Synthetic Dataset
---|---|---
| Prec. | Recall | IoU | Prec. | Recall | IoU
[2] | 0.969 | 0.858 | 0.831 | 0.984 | 0.771 | 0.761
[2] & [3] | 0.967 | 0.861 | 0.833 | 0.987 | 0.768 | 0.758
Ours | 0.957 | 0.964 | 0.923 | 0.978 | 0.966 | 0.950
TABLE II: Point Cloud Segmentation Results. ”Ours” here corresponds to the
object segmentation results achieved after graph matching given the input of
over-segmentation from [2] $\&$ [3]. Best results in bold.

###  VI-F Ablation Study

The results of an ablation study are shown in Table III. All the alternatives
use the same feature extractor (DeepLabV3+ [37]) and clustering method
(MeanShift [7]). The ablations use different object pseudo-labels for training
and are tested given ground truth foreground masks similar to Table I. V1. The
first one is trained on the COCO dataset [38]. Its final prediction layer is
removed and 256-dim pixel features are used directly for clustering. It has
the lowest performance. V2. The second one is trained using the output of LCCP
[2] without graph matching. Its performance is similar to that of LCCP-based
3D segmentation as expected. V3. The third one is trained using pseudo-labels
with graph matching but without pruning. The performance is better than not
using graph matching, but it’s not as good as the proposed approach since
nodes can be incorrectly merged, which results in erroneous object pseudo-
labels. V4. The last alternative tries to exhaustively register point clouds
of connected components without using the candidates from graph matching. If
the precision, recall, and feature cosine similarity of two registered nodes
are greater than 0.9, then these two nodes are considered a match. This
underperforms the proposed method since point cloud registration is more
likely to fail due to clutter, and takes significantly more time to find
object patterns.

| Real Dataset | Synthetic Dataset
---|---|---
Ablation | Prec. | Recall | IoU | Prec. | Recall | IoU
V1. | 0.716 | 0.665 | 0.547 | 0.806 | 0.775 | 0.665
V2. | 0.915 | 0.815 | 0.776 | 0.935 | 0.746 | 0.711
V3. | 0.904 | 0.832 | 0.790 | 0.934 | 0.800 | 0.762
V4. | 0.915 | 0.883 | 0.842 | 0.931 | 0.823 | 0.782
Ours | 0.925 | 0.911 | 0.870 | 0.938 | 0.925 | 0.880
TABLE III: Ablation results. Best results in bold.

###  VI-G Running Time

Binary linear programming (BLP) is an NP-hard problem. In practice, however,
solutions can be acquired very fast given advanced solvers, such as Gurobi
[39], which is used in companion implementation. The processes of graph
matching (Sec.V-D) and pruning (Sec.V-E) between two scenes can be done in
about 2s and pairwise matching across 30 scenes can be done within $10$
minutes on an AMD Ryzen 5900 CPU.

##  VII Conclusion and Limitations

This work proposes a self-supervised learning pipeline given unlabeled RGB-D
videos captured with a moving camera observing static scenes, which is a
common scenario for household mobile robots. A graph-matching algorithm is
adapted to find object patterns across videos and generate object pseudo-
labels for learning. The proposed method achieves better results in object
segmentation than existing unsupervised segmentation methods and methods that
used simulation data for supervised training.

One limitation of the proposed approach is that it relies on the initial
object over-segmentation to be consistent across different scenes. Despite
using the synergy of LCCP and SymSeg, consistent over-segmentation may not
always be achieved when reconstruction noise is large and when objects are not
able to be separated using geometric features. This can potentially be
addressed with more data that cover multiple over-segmentation patterns and
future improvements on the matching algorithm that considers the operation of
splitting a node when under-segmentation happens. Another limitation is that
the proposed method cannot handle deformable objects in general, since it
requires matched objects to be identical in both texture and geometry.

## References

  * [1] S. Lu, R. Wang, Y. Miao, C. Mitash, and K. Bekris, “Online object model reconstruction and reuse for lifelong improvement of robot manipulation,” in _2022 International Conference on Robotics and Automation (ICRA)_. IEEE, 2022, pp. 1540–1546.

  * [2] S. Christoph Stein, M. Schoeler, J. Papon, and F. Worgotter, “Object partitioning using local convexity,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2014, pp. 304–311.
  * [3] A. Ecins, C. Fermüller, and Y. Aloimonos, “Seeing behind the scene: Using symmetry to reason about objects in cluttered environments,” in _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2018, pp. 7193–7200.
  * [4] W. Van Gansbeke, S. Vandenhende, S. Georgoulis, and L. Van Gool, “Unsupervised semantic segmentation by contrasting object mask proposals,” in _International Conference on Computer Vision_ , 2021.
  * [5] J. Lerouge, M. Hammami, P. Héroux, and S. Adam, “Minimum cost subgraph matching using a binary linear program,” _Pattern Recognition Letters_ , vol. 71, pp. 45–51, 2016.
  * [6] D. Comaniciu and P. Meer, “Mean shift: A robust approach toward feature space analysis,” _IEEE Transactions on pattern analysis and machine intelligence_ , vol. 24, no. 5, pp. 603–619, 2002.
  * [7] A. Banerjee, I. S. Dhillon, J. Ghosh, S. Sra, and G. Ridgeway, “Clustering on the Unit Hypersphere using von Mises-Fisher Distributions,” _Journal of Machine Learning Research_ , vol. 6, no. 9, 2005.
  * [8] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in _International conference on machine learning_. PMLR, 2020, pp. 1597–1607.
  * [9] X. Chen, H. Fan, R. Girshick, and K. He, “Improved baselines with momentum contrastive learning,” _arXiv preprint arXiv:2003.04297_ , 2020.
  * [10] X. Chen and K. He, “Exploring simple siamese representation learning,” _arXiv preprint arXiv:2011.10566_ , 2020.
  * [11] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. H. Richemond, E. Buchatskaya, C. Doersch, B. A. Pires, Z. D. Guo, M. G. Azar, _et al._ , “Bootstrap your own latent: A new approach to self-supervised learning,” _arXiv preprint arXiv:2006.07733_ , 2020.
  * [12] C. S. Juntao Tan and A. Boularias, “A self-supervised learning system for object detection in videos using random walks on graphs,” in _Proceedings of the 2021 International Conference on Robotics and Automation (ICRA), Xi’an, China_ , 2021.
  * [13] S. Pirk, M. Khansari, Y. Bai, C. Lynch, and P. Sermanet, “Online learning of object representations by appearance space feature alignment,” in _2020 IEEE International Conference on Robotics and Automation (ICRA)_ , 2020, pp. 10 473–10 479.
  * [14] X. Wang and A. Gupta, “Unsupervised learning of visual representations using videos,” in _Proceedings of the IEEE international conference on computer vision_ , 2015, pp. 2794–2802.
  * [15] X. Wang, K. He, and A. Gupta, “Transitive invariance for self-supervised visual representation learning,” in _Proceedings of the IEEE international conference on computer vision_ , 2017, pp. 1329–1338.
  * [16] D. Pathak, R. Girshick, P. Dollár, T. Darrell, and B. Hariharan, “Learning features by watching objects move,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 2701–2710.
  * [17] R. Liu, Z. Wu, S. Yu, and S. Lin, “The emergence of objectness: Learning zero-shot segmentation from videos,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [18] S. Purushwalkam and A. Gupta, “Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases,” _arXiv preprint arXiv:2007.13916_ , 2020.
  * [19] M. Schoeler, J. Papon, and F. Worgotter, “Constrained planar cuts-object partitioning for point clouds,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 5207–5215.
  * [20] Y. Xiang, C. Xie, A. Mousavian, and D. Fox, “Learning RGB-D feature embeddings for unseen object instance segmentation,” _arXiv preprint arXiv:2007.15157_ , 2020.
  * [21] C. Xie, Y. Xiang, A. Mousavian, and D. Fox, “Unseen object instance segmentation for robotic environments,” _IEEE Transactions on Robotics_ , 2021.
  * [22] C. Xie, A. Mousavian, Y. Xiang, and D. Fox, “Rice: Refining instance masks in cluttered environments with graph neural networks,” in _Conference on Robot Learning (CoRL)_ , 2021.
  * [23] M. Danielczuk, M. Matl, S. Gupta, A. Li, A. Lee, J. Mahler, and K. Goldberg, “Segmenting Unknown 3D Objects from Real Depth Images using Mask R-CNN Trained on Synthetic Data,” in _Proc. IEEE Int. Conf. Robotics and Automation (ICRA)_ , 2019.
  * [24] J. H. Cho, U. Mall, K. Bala, and B. Hariharan, “PiCIE: Unsupervised Semantic Segmentation Using Invariance and Equivariance in Clustering,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2021, pp. 16 794–16 804.
  * [25] M. Hamilton, Z. Zhang, B. Hariharan, N. Snavely, and W. T. Freeman, “Unsupervised semantic segmentation by distilling feature correspondences,” in _International Conference on Learning Representations_ , 2022. [Online]. Available: https://openreview.net/forum?id=SaKO6z6Hl0c
  * [26] M. Cho, S. Kwak, C. Schmid, and J. Ponce, “Unsupervised object discovery and localization in the wild: Part-based matching with bottom-up region proposals,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 1201–1210.
  * [27] V. H. Vo, E. Sizikova, C. Schmid, P. Pérez, and J. Ponce, “Large-scale unsupervised object discovery,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 16 764–16 778, 2021.
  * [28] X. Qin, Z. Zhang, C. Huang, C. Gao, M. Dehghan, and M. Jagersand, “Basnet: Boundary-aware salient object detection,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2019, pp. 7479–7489.
  * [29] T. Nguyen, M. Dax, C. K. Mummadi, N. Ngo, T. H. P. Nguyen, Z. Lou, and T. Brox, “Deepusps: Deep robust unsupervised saliency prediction via self-supervision,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [30] R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohi, J. Shotton, S. Hodges, and A. Fitzgibbon, “KinectFusion: Real-time dense surface mapping and tracking,” in _2011 10th IEEE international symposium on mixed and augmented reality_. IEEE, 2011, pp. 127–136.
  * [31] M. A. Fischler and R. C. Bolles, “Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography,” in _Readings in Computer Vision_ , M. A. Fischler and O. Firschein, Eds. San Francisco (CA): Morgan Kaufmann, 1987, pp. 726–740.
  * [32] Z. Sui, H. Chang, N. Xu, and O. C. Jenkins, “Geofusion: Geometric consistency informed scene estimation in dense clutter,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 4, pp. 5913–5920, 2020.
  * [33] M. Denninger, M. Sundermeyer, D. Winkelbauer, Y. Zidan, D. Olefir, M. Elbadrawy, A. Lodhi, and H. Katam, “BlenderProc,” _arXiv preprint arXiv:1911.01911_ , 2019.
  * [34] R. Kaskman, S. Zakharov, I. Shugurov, and S. Ilic, “HomebrewedDB: RGB-D Dataset for 6D Pose Estimation of 3D Objects,” _International Conference on Computer Vision (ICCV) Workshops_ , 2019.
  * [35] B. Calli, A. Singh, J. Bruce, A. Walsman, K. Konolige, S. Srinivasa, P. Abbeel, and A. M. Dollar, “Yale-CMU-Berkeley dataset for robotic manipulation research,” _The International Journal of Robotics Research_ , vol. 36, no. 3, pp. 261–268, 2017.
  * [36] H. W. Kuhn, “The Hungarian method for the assignment problem,” _Naval research logistics quarterly_ , vol. 2, no. 1-2, pp. 83–97, 1955.
  * [37] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _Proceedings of the European Conference on Computer Vision (ECCV)_ , September 2018.
  * [38] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” in _European conference on computer vision_. Springer, 2014, pp. 740–755.
  * [39] Gurobi Optimization, LLC, “Gurobi Optimizer Reference Manual,” 2021. [Online]. Available: https://www.gurobi.com