11institutetext: Marco Buratti 22institutetext: Dipartimento di Scienze di
Base e Applicate per l’Ingegneria (S.B.A.I.), Università Sapienza di Roma,
Italy, 22email<EMAIL_ADDRESS>
# Tight Heffter arrays from finite fields

Marco Buratti

###### Abstract

After extending the classic notion of a tight Heffter array H$(m,n)$ to any
group of order $2mn+1$, we give direct constructions for elementary abelian
tight Heffter arrays, hence in particular for prime tight Heffter arrays. If
$q=2mn+1$ is a prime power, we say that an elementary abelian H$(m,n)$ is
“over $\mathbb{F}_{q}$” since, for its construction, we exploit both the
additive and multiplicative structure of the field of order $q$. We show that
in many cases a direct construction of an H$(m,n)$ over $\mathbb{F}_{q}$, say
$A$, can be obtained very easily by imposing that $A$ has rank 1 and,
possibly, a rich group of multipliers, that are elements $u$ of
$\mathbb{F}_{q}$ such that $uA=A$ up to a permutation of rows and columns. An
H$(m,n)$ over $\mathbb{F}_{q}$ will be said optimal if the order of its group
of multipliers is the least common multiple of the odd parts of $m$ and $n$,
since this is the maximum possible order for it. The main result is an
explicit construction of a rank-one H$(m,n)$ – reaching almost always the
optimality – for all admissible pairs $(m,n)$ for which there exist two
distinct odd primes $p$, $p^{\prime}$ dividing $m$ and $n$, respectively.

Dedicated to Doug Stinson on the occasion of his 66th birthday.

##  1 Introduction

Heffter arrays are interesting combinatorial designs introduced by Archdeacon
A  in 2015. For a rich survey on the many variations and the related results
we refer to A&J . In this note we consider Heffter arrays in the original
meaning of “tight” Heffter arrays extending the definition to any group of odd
order as proposed in CPP .

A half-set of an additive group $G$ of odd order $2\ell+1$ is a $\ell$-subset
$L$ of $G$ such that $L\ \cup\ -L=G\setminus\\{0\\}$, i.e., a complete set of
representatives for the so-called patterned starter of $G$. A subset or
multisubset $S$ of an additive abelian group $G$ will be said zero-sum if the
sum of all elements of $S$ is zero. This can be extended to non-abelian groups
saying that an ordered subset or multisubset $S=\\{s_{1},\dots,s_{k}\\}$ of
$G$ is zero-sum if $s_{1}+...+s_{k}=0$.

###### Definition 1

A Heffter array H$(m,n)$ over an additive group $G$ of order $2mn+1$ is a
$m\times n$ matrix whose entries form a half-set of $G$ and whose rows and
columns are all zero-sum.

Additive designs have been introduced in CFP  as block designs whose point set
$V$ is a subset of an abelian group $G$ and whose blocks are all zero-sum.
They are strictly $G$-additive when $V$ is the whole $G$. The concrete
construction of a strictly $G$-additive Steiner 2-design in which $G$ is not
elementary abelian is one of the many hard problems concerning this
interesting theory BN . The theory can be extended to other combinatorial
designs. For instance, a Heffter array H$(n,n)$ can be seen as an additive
resolvable $(n^{2}_{2},(2n)_{n})$-configuration (see BP ).

A Heffter array is cyclic or elementary abelian if it is over a cyclic or
elementary abelian group, respectively. If an H$(m,n)$ is elementary abelian,
then $q=2mn+1$ is obviously a prime power and we say that the array is over
$\mathbb{F}_{q}$, that is the field of order $q$.

The existence problem for a cyclic H$(m,n)$ has been completely solved ABD .
On the other hand the solution has been obtained via recursive methods and, as
far as we are aware, just a few direct constructions are known. In this note
we will give direct constructions for infinitely many elementary abelian
H$(m,n)$. Thus, in particular, for infinitely many prime H$(m,n)$.

We get our constructions by imposing, first of all, that the arrays are rank-
one, i.e., of rank equal to 1 in the usual sense of linear algebra. Thus, a
Heffter array H$(m,n)$ over $\mathbb{F}_{q}$ is rank-one if its rows are all
multiples of a non-zero vector of $\mathbb{F}_{q}^{n}$ (and consequently its
columns are all multiples of a non-zero vector of $\mathbb{F}_{q}^{m}$).

###### Example 1

It is readily seen that the following is a rank-one H$(3,3)$ over
$\mathbb{Z}_{19}$.

$A=\begin{pmatrix}1&3&-4\cr 7&2&-9\cr-8&-5&-6\cr\end{pmatrix}$

###### Example 2

Let $g$ be a root of the primitive polynomial $x^{2}+x+2$ over
$\mathbb{F}_{5}$. Then the following is a rank-one H$(3,4)$ over
$\mathbb{F}_{25}$.

$A=\begin{pmatrix}1&&g&&g+4&&3g\cr 3g+1&&3g+4&&3&&4g+2\cr
2g+3&&g+1&&4g+3&&3g+3\end{pmatrix}$

In CDY  Cavenagh et al. propose a definition of equivalent Heffter arrays. On
the other hand, as far as we are aware, there is no “official” definition of
isomorphic Heffter arrays. From the perspective of the classic design theory,
we think it is natural to give the following.

###### Definition 2

An isomorphism between a Heffter array $A$ over a group $G$ and a Heffter
array $A^{\prime}$ over a group $G^{\prime}$ is a group isomorphism $\phi$
between $G$ and $G^{\prime}$ such that $\phi(A)$ or its transposal can be
obtained from $A^{\prime}$ by suitable permutations of its rows and columns.

Thus, in particular, an automorphism of a Heffter array $A$ over a group $G$
is an automorphism of the group $G$ mapping $A$ or $A^{T}$ into a matrix
obtainable from $A$ itself by permuting its rows and columns.

Two Heffter arrays are isomorphic if there exists an isomorphism between them.

By saying that a Heffter array $A$ is over a ring $R$ we will mean that it is
over the additive group of $R$. If $R$ is with identity and $u$ is a unit of
$R$, it is clear that $A$ and $uA$ are isomorphic since the map
$\widehat{u}:x\in R\longrightarrow ux\in R$ is an isomorphism between them. On
the other hand it is not said that $\widehat{u}$ is an automorphism of $A$. In
the event this happens we will say that $u$ is a multiplier of $A$.

###### Definition 3

Let $A$ be a Heffter array over a ring $R$ with identity. A multiplier of $A$
is any unit $u$ of $R$ such that $uA$ or $uA^{T}$ can be obtained from $A$ by
suitable permutations of its rows and columns.

It is evident that if $u$ is a multiplier of a Heffter array $A$, then the set
of entries of $A$ and $uA$ coincide. Thus $-1$ is never a multiplier since, by
the definition of a half-set, the set of entries of $-A$ is exactly the
complement of the set of entries of $A$. Note, however, that $A$ and $-A$ are
equivalent in the sense of Cavenagh et al. CDY .

The set $M$ of all multipliers of a Heffter array over a ring $R$ with
identity clearly form a subgroup of the group $U(R)$ of units of $R$. For
instance, it is easy to see that the group of multipliers of the H$(3,3)$ of
Example 1 is $\\{1,7,11\\}$ and that the group of multipliers of the H$(3,4)$
of Example 2 is $\\{1,3g+1,2g+3\\}$.

Starting from the fundamental paper by R.M. Wilson W  – or even from the
earlier paper by R.C. Bose Bose  – arriving to very recent work by the present
author et al. BBGRT  the search for difference families and all their variants
(a well-know topic of design theory) has been made easier by imposing “many”
multipliers. The same can be applied to rank-one Heffter arrays. Indeed all
the rank-one Heffter arrays constructed in this paper have a non-trivial group
of multipliers.

Throughout the paper, saying that a pair $(m,n)$ is admissible we mean that
$2mn+1$ is a prime power and that both $m$ and $n$ are greater than 2. Also,
speaking of “a rank-one Heffter array H$(m,n)$” it will be understood that
$(m,n)$ is admissible and that the array is over $\mathbb{F}_{2mn+1}$.

The paper will be organized as follows.

In the next section we will give some very elementary prerequisites that are
necessary to understand the paper.

In Section 3 it is proved that the existence of a rank-one H$(m,n)$ is
completely equivalent to a factorization of a half-set of $\mathbb{F}_{2mn+1}$
into the product $X\cdot Y$ of a zero-sum $m$-set $X$ by a zero-sum $n$-set
$Y$. As a corollary we get the easiest and nicest construction of a rank-one
H$(m,n)$: if $(m,n)$ is admissible with $m$, $n$ odd and coprime, then the
$m\times n$ array $[r^{2ni+2mj}]$ where $r$ is a primitive element of
$\mathbb{F}_{2mn+1}$, is a rank-one H$(m,n)$ whose multipliers are the non-
zero squares of $\mathbb{F}_{2mn+1}$.

In Section 4 we prove that the order $\mu$ of the group of multipliers of a
rank-one H$(m,n)$ is at most equal to the least common multiple of the odd
parts of $m$ and $n$. So we say that a rank-one H$(m,n)$ is optimal when $\mu$
reaches this value. In particular, we say that it is perfect when $\mu=mn$.
The perfect rank-one Heffter arrays are precisely the “easiest and nicest”
obtained in Section 2 and we prove that they have the useful property of being
globally simple in Section 5.

In Section 6, using cyclotomy, we give our main result that is an explicit
construction of a rank-one H$(m,n)$ for every admissible pair $(m,n)$ such
that there exist two distinct odd primes $p$, $p^{\prime}$ dividing $m$ and
$n$, respectively. We also prove that this construction reaches optimality
unless $m$ and $n$ have the same radical $r$ and either $mr\over n$ or
$nr\over m$ is an integer.

##  2 Preliminaries

We will need some elementary facts about cyclotomy and the related standard
notation/terminology. Given a prime power $q=de+1$, the multiplicative group
of $\mathbb{F}_{q}$ will be denoted by $\mathbb{F}_{q}^{*}$ and the subgroup
of $\mathbb{F}_{q}^{*}$ of index $e$ – or, equivalently, the subgroup of
$\mathbb{F}_{q}^{*}$ of order $d$ – will be denoted by $C^{e}$. If $r$ is a
primitive element of $\mathbb{F}_{q}^{*}$, then the set of cosets of $C^{e}$
in $\mathbb{F}_{q}^{*}$ (the so-called cyclotomic classes of index $e$) is
$\\{r^{i}C^{e}\ |\ 0\leq i\leq e-1\\}$. It will be always understood that $r$
is fixed and, as it is standard, the coset $r^{i}C^{e}$ will be denoted by
$C^{e}_{i}$ whichever is $i$. Note that we have $C^{e}_{i}=C^{e}_{j}$ if and
only if $i\equiv j$ (mod $e$) and that $C^{e}_{i}\cdot C^{e}_{j}=C^{e}_{i+j}$.
Anyone who has a little bit of familiarity with finite fields should know the
elementary facts below that we recall for convenience.

###### Lemma 1

Let $q$ be a prime power. Then we have:

  * (i)

Every union of cosets of a non-trivial subgroup of $\mathbb{F}_{q}^{*}$ is
zero-sum.

  * (ii)

If $q=2ed+1$ with $d$ odd, then $-1\in C^{2e}_{e}$ so that $C^{2e}_{0}\ \cup\
C^{2e}_{1}\ \cup\ \dots\ \cup\ C^{2e}_{e-1}$ is a half-set of
$\mathbb{F}_{q}$.

  * $(ii)^{\prime}$

In particular, if $q=2d+1$ with $d$ odd, then $-1\in C^{2}_{1}$ and $C^{2}$ is
a half-set of $\mathbb{F}_{q}$.

  * (iii)

The product of two subgroups of $\mathbb{F}_{q}^{*}$ of orders $s$ and $t$, is
the subgroup of $\mathbb{F}_{q}^{*}$ of order $lcm(s,t)$.

We also need the following.

###### Proposition 1

Let $q=de+1$ be a prime power and let $X$ be a subset of $\mathbb{F}_{q}^{*}$.
Then the stabilizer of $X$ under the natural action of $\mathbb{F}_{q}^{*}$ is
divisible by $d$ if and only if $X$ is a union of cosets of $C^{e}$.

###### Proof

The “if part” is obvious. Let us prove the “only if” part. Let $S$ be the
$\mathbb{F}_{q}^{*}$-stabilizer of $X$ and assume that its order is divisible
by $d$. Then $S$ contains the subgroup of $\mathbb{F}_{q}^{*}$ of order $d$,
that is $C^{e}$. Thus it makes sense to consider the action of $C^{e}$ on $X$.
The orbits of this action partition $X$ and each of them is of the form
$xC^{e}$ for some $x\in X$, that is a coset of $C^{e}$ in
$\mathbb{F}_{q}^{*}$. The assertion follows.

##  3 Characterization

One says that a subset $S$ of a multiplicative (resp. additive) group $G$, can
be factorized into the product $X\cdot Y$ (resp. sum $X+Y$) of two subsets
$X$, $Y$ of $S$ if every element of $S$ can be written in exactly one way as
$x\cdot y$ (resp. $x+y$) with $x\in X$ and $y\in Y$. One also says that
$X\cdot Y$ (resp. $X+Y$) is a factorization of $S$. By saying that $X$ is a
factor of $S$ one means that $X$ is a subset of $S$ for which there exists
another subset $Y$ of $S$ such that $S=X\cdot Y$ (resp. $S=X+Y$). In other
words, $X$ is a factor of $S$ if it is possible to “tile” $S$ with suitable
translates of $X$.

We are going to show that the rank-one Heffter arrays can be characterized in
terms of factorizations of a half-set of a finite field.

###### Proposition 2

Let $q=2mn+1$ be a prime power. There exists a rank-one H$(m,n)$ if and only
if there is a suitable half-set of $\mathbb{F}_{q}$ admitting a factorization
into the product of two zero-sum subsets of $\mathbb{F}_{q}^{*}$ of sizes $m$
and $n$.

###### Proof

$(\Longrightarrow).$ If $A$ is a rank-one H$(m,n)$ over $\mathbb{F}_{q}$, then
all columns of $A$ are multiples of a vector
$(x_{1},\dots,x_{m})\in\mathbb{F}_{q}^{m}$. Thus, if $A^{j}$ is the $j$-th
column of $A$, there is a suitable $y_{j}\in\mathbb{F}_{q}$ such that
$A^{j}=(x_{1},\dots,x_{m})\cdot y_{j}$ for $1\leq j\leq n$. It follows that
$a_{i,j}=x_{i}y_{j}$. Hence the set of all entries of $A$, which is a half-set
of $(\mathbb{F}_{q},+)$ by definition of a Heffter array, coincides with the
product of the two sets $X=\\{x_{1},\dots,x_{m}\\}$ and
$Y=\\{y_{1},\dots,y_{n}\\}$.

$(\Longleftarrow).$ Let $V$ be a half-set of $\mathbb{F}_{q}$ and assume that
$V=X\cdot Y$ where $X=\\{x_{1},\dots,x_{m}\\}$ and $Y=\\{y_{1},\dots,y_{n}\\}$
are zero-sum subsets of $\mathbb{F}_{q}^{*}$ of sizes $m$ and $n$,
respectively. Consider the $m\times n$ array $A$ over $\mathbb{F}_{q}$ defined
by $a_{i,j}=x_{i}y_{j}$ for $1\leq i\leq m$ and for $1\leq j\leq n$. It is
obvious that the entries of $A$ are precisely the elements of $V$, hence they
form a half-set of $(\mathbb{F}_{q},+)$. Note that the $i$-th row of $A$ is
$x_{i}\cdot(y_{1},\dots,y_{n})$ and that the $j$-th column of $A$ is
$(x_{1},\dots,x_{m})\cdot y_{j}$. Thus, given that both $X$ and $Y$ are zero-
sum, we infer that all rows and columns of $A$ are also zero-sum. We conclude
that $A$ is a rank-one H$(m,n)$.

We will briefly refer to the sets $X$ and $Y$ in the above proof as the
factors of the Heffter array $A$.

The next result is just a special case of the more general Theorem 6.1 that we
will see later. On the other hand we think it is appropriate to give its
statement now and separately, since it is an almost immediate consequence of
Proposition 2. Actually, this is the easiest and nicest construction of a
rank-one Heffter array.

###### Corollary 1

If $(m,n)$ is an admissible pair with $m$, $n$ odd and coprime, then $C^{2n}$
and $C^{2m}$ are the factors of a rank-one H$(m,n)$.

###### Proof

By assumption, $q=2mn+1$ is a prime power. The set $X=C^{2n}$ and $Y=C^{2m}$
are the subgroups of $\mathbb{F}_{q}^{*}$ of orders $m$ and $n$, respectively.
They are zero-sum by Lemma 1$(i)$. Also, their product is the subgroup of
$\mathbb{F}_{q}^{*}$ of order $mn$ by Lemma 1$(iii)$, i.e., the group $C^{2}$
of non-zero squares of $\mathbb{F}_{q}$ which is a half-set by Lemma
1$(ii)^{\prime}$. The assertion then follows from Proposition 2.

The above gives, in particular, infinitely many Heffter arrays which are
elementary abelian but not prime. For instance, we have $7^{3}=2\cdot 9\cdot
19+1$ and hence, applying Corollary 1 with $m=9$ and $n=19$, we get a
H$(9,19)$ over $\mathbb{F}_{7^{3}}$.

We will see that the group of multipliers of an H$(m,n)$ obtainable via
Corollary 1 is $C^{2}$, i.e., the group of non-zero squares of
$\mathbb{F}_{2mn+1}$.

###### Example 3

Let us construct a rank-one H$(3,5)$. The request makes sense since $q=2\cdot
3\cdot 5+1=31$ is a prime. Note that $x=5$ is a cubic root of unity (mod 31)
and that $y=2$ is a 5th root of unity (mod 31). Thus $X=\langle
x\rangle=\\{1,5,25\\}$ is the subgroup of $\mathbb{F}_{31}^{*}$ of order 3 and
$Y=\langle y\rangle=\\{1,2,4,8,16\\}$ is the subgroup of $\mathbb{F}_{31}^{*}$
of order 5. The desired rank-one H$(3,5)$ is therefore

$A=\begin{pmatrix}1&2&4&8&16\cr 5&10&20&9&18\cr 25&19&7&14&28\end{pmatrix}$

Note that multiplying $A$ by 9 (which is a generator of the squares of
$\mathbb{F}_{31}$) corresponds to permute its rows cyclically
$A_{i}\rightarrow A_{i+1\ (mod\ 3)}$ and then to permute the columns
cyclically $A^{j}\rightarrow A^{j+3\ (mod\ 5)}$. This implies that the non-
zero squares of $\mathbb{F}_{31}$ form a group of multipliers of the above
array.

##  4 Multipliers of a rank-one Heffter array

From now on, given any integer $k$ we denote by $k_{o}$ the odd part of $k$,
that is the greatest odd divisor of $k$.

###### Proposition 3

Let $A$ be a rank-one H$(m,n)$ over $\mathbb{F}_{q}$ with factors $X$ and $Y$.
Then the group of multipliers of $A$ is the product of the
$\mathbb{F}_{q}^{*}$-stabilizers of $X$ and $Y$. Also, its order is at most
equal to $lcm(m_{o},n_{o})$.

###### Proof

Set $X=\\{x_{1},\dots,x_{m}\\}$ and $Y=\\{y_{1},\dots,y_{n}\\}$ so that we
have $a_{i,j}=x_{i}y_{j}$ for $1\leq i\leq m,1\leq j\leq n$. We have to prove
that the group $M$ of multipliers of $A$ is the product $S\cdot T$ where $S$
and $T$ are the $\mathbb{F}_{q}^{*}$-stabilizers of $X$ and $Y$, respectively.
In the following, $A_{i}$ and $\overline{A_{i}}$ will denote the $i$-th row of
$A$ and the set of its elements, respectively. Analogously, $A^{j}$ and
$\overline{A^{j}}$ will denote the $j$-th column of $A$ and the set of its
elements, respectively.

If $s\in S$, then $s(x_{1},\dots,x_{m})=(x_{\pi(1)},\dots,x_{\pi(m)})$ for a
suitable permutation $\pi$ on the set $\\{1,\dots,m\\}$. It easily follows
that $sA$ is the matrix whose $j$-th column is $A^{\pi(j)}$ for $1\leq j\leq
m$, hence a matrix obtainable from $A$ by permuting its columns. Thus, by
definition, $s$ is a multiplier of $A$.

Analogously, if $t\in T$, then
$t(y_{1},\dots,y_{n})=(y_{\psi(1)},\dots,y_{\psi(n)})$ for a suitable
permutation $\psi$ on the set $\\{1,\dots,n\\}$. It follows that $tA$ is the
matrix whose $i$-th row is $A_{\psi(i)}$ for $1\leq i\leq n$, hence a matrix
obtainable from $A$ by permuting its rows and then $t$ is a multiplier of $A$.

Thus $M$ contains both $S$ and $T$, hence it contains their product $S\cdot
T$.

Now we prove the inverse inclusion. Let $u\in M$. By definition of a
multiplier, there is a suitable pair $(i,j)$ such that either

(1) $uA_{1}$ is a permutation of $A_{i}$ and $uA^{1}$ is a permutation of
$A^{j}$;

or

(2) $uA_{1}$ is a permutation of $A^{j}$ and $uA^{1}$ is a permutation of
$A_{i}$.

Of course (2) may happen only in the case $m=n$.

Assume that (1) holds. It follows, in particular, that $ua_{1,1}=ux_{1}y_{1}$
must be the common element of $A_{i}$ and $A^{j}$, that is
$a_{i,j}=x_{i}y_{j}$. Thus we have

$u=x_{i}y_{j}x_{1}^{-1}y_{1}^{-1}$ (1)

For $1\leq i\leq m$, we have $A_{i}=(x_{i}y_{1},x_{i}y_{2},\dots,x_{i}y_{n})$,
hence $\overline{A_{i}}=x_{i}Y$. Analogously, for $1\leq j\leq n$, we have
$A^{j}=(x_{1}y_{j},x_{2}y_{j},\dots,x_{m}y_{j})$, hence
$\overline{A^{j}}=Xy_{j}$. In particular, we have $\overline{A_{1}}=x_{1}Y$
and $\overline{A^{1}}=Xy_{1}$.

The fact that $uA_{1}$ is a permutation of $A_{i}$ implies that
$u\overline{A_{1}}=\overline{A_{i}}$, i.e., $ux_{1}Y=x_{i}Y$. Analogously, the
fact that $uA^{1}$ is a permutation of $A^{j}$ implies that
$u\overline{A^{1}}=\overline{A^{j}}$, i.e., $uXy_{1}=Xy_{j}$. Thus, using (1),
we get $y_{j}y_{1}^{-1}Y=Y$ and $x_{i}x_{1}^{-1}X=X$. This means that
$y_{j}y_{1}^{-1}$ stabilizes $Y$ and $x_{i}x_{1}^{-1}$ stabilizes $X$, i.e.,
$y_{j}y_{1}^{-1}\in T$ and $x_{i}x_{1}^{-1}\in S$. The conclusion is that
$u=(x_{i}x_{1}^{-1})(y_{j}y_{1}^{-1})\in S\cdot T$.

If $m=n$ and (2) holds, reasoning in a very similar way one finds that $u\in
S\cap T$, hence $u$ is the unity. Indeed, if $u$ fixes both $X$ and $Y$ then
we have $ux_{1}=x_{i}$ and $uy_{1}=y_{j}$ for a suitable pair $(i,j)$. It
follows that $ux_{1}y_{1}=x_{i}y_{1}=x_{1}y_{j}$ which implies $i=j=1$, hence
$u=1$.

By Proposition 1, the order of $S$ is a divisor of $|X|=m$. Remember, however,
that $-1$ cannot be a multiplier so that $-1\notin S$ which means that $S$ has
odd order. Hence $|S|$ is a divisor of $m_{o}$ and then $S$ is a subgroup of
the group $M_{o}$ of the $m_{o}$-th roots of unity. Reasoning exactly in the
same way, one can see that $T$ is a subgroup of the group $N_{o}$ of the
$n_{o}$-th roots of unity. Thus $M=S\cdot T$ is a subgroup of $M_{o}\cdot
N_{o}$. We get the assertion observing that the order of $M_{o}\cdot N_{o}$ is
$lcm(m_{o},n_{o})$ (see Lemma 1$(iii)$).

The above suggests the following definition.

###### Definition 4

A rank-one H$(m,n)$ is optimal if the order of its group of multipliers is the
least common multiple of the odd parts of $m$ and $n$. It is perfect if it has
order $mn$, i.e., it is the group $C^{2}$ of non-zero squares of
$\mathbb{F}_{2mn+1}$.

The perfect rank-one H$(m,n)$ can be completely characterized in view of
Corollary 1.

###### Proposition 4

There exists a perfect rank-one H$(m,n)$ if and only if $(m,n)$ is an
admissible pair with $m$, $n$ odd and coprime.

###### Proof

Let $A$ be a perfect rank-one H$(m,n)$ and let $M$ be its group of
multipliers. By definition, we have $|M|=mn$ but we also have $|M|\leq
lcm(m_{o},n_{o})$ by Proposition 3. Thus, considering that $lcm(m_{o},n_{o})$
is a divisor of $mn$, we necessarily have $lcm(m_{o},n_{o})=mn$. Clearly, this
is possible only if $m$ and $n$ are odd and coprime.

Now let $(m,n)$ be an admissible pair with $m$, $n$ odd and coprime. By
Corollary 1 there exists a rank-one H$(m,n)$ whose factors are the group $X$
of $m$-th roots of unity and the group $Y$ of $n$-th roots of unity. Their
$\mathbb{F}_{q}^{*}$-stabilizers are $X$ and $Y$, respectively (see
Proposition 1). So the group of multipliers is $X\cdot Y$ whose order is $mn$
by Lemma 1$(iii)$.

##  5 Globally simple rank-one Heffter arrays

The useful notion of a globally simple Heffter array over a cyclic group has
been introduced in CMPP  and can be extended to Heffter arrays over any group
in the obvious way. In particular, a tight H$(m,n)$, say $A=(a_{i,j})$, is
globally simple if each of its rows and each of its columns has pairwise
distinct partial sums, i.e., none of the lists
$\displaystyle\sigma_{i}=(\sum_{k=1}^{j}a_{i,k}\ |\ 1\leq j\leq n)$,
$i=1,\dots,m$, and $\displaystyle\tau_{j}=(\sum_{k=1}^{i}a_{k,j}\ |\ 1\leq
i\leq m)$, $j=1,\dots,n$, has repeated elements.

In general, to check whether an H$(m,n)$ is globally simple could require long
and tedious calculations. This task is much easier in the case of rank-one
Heffter arrays.

###### Proposition 5

Let $A=(a_{i,j})$ be a rank-one H$(m,n)$ such that the first of its rows and
the first of its columns have pairwise distinct partial sums. Then $A$ is
globally simple.

###### Proof

Let $\sigma_{i}=(\sigma_{i,1},\dots,\sigma_{i,n})$ be the list of the partial
sums of $A_{i}$. We have $A_{i}=\rho_{i}A_{1}$ for a suitable
$\rho_{i}\in\mathbb{F}_{q}^{*}$ since $A$ is rank-one. Hence we have
$\sigma_{i}=\rho_{i}\sigma_{1}$ which implies that $\sigma_{i}$ does not have
repeated elements since this is true for $\sigma_{1}$ by assumption. Thus the
partial sums of any row of $A$ are pairwise distinct. Reasoning in the same
way, we can see that the partial sums of any column of $A$ are also pairwise
distinct.

The perfect Heffter arrays of Corollary 1 are globally simple.

###### Theorem 5.1

If $q=2mn+1$ is a prime power with $m$, $n$ odd and coprime, then there exists
a globally simple H$(m,n)$ over $\mathbb{F}_{q}$.

###### Proof

Let $x$ be a primitive $n$-th root of unity in $\mathbb{F}_{q}$ and let $y$ be
a primitive $m$-th root of unity in $\mathbb{F}_{q}$. By Corollary 1, we can
take $X=(1,x,x^{2},...,x^{n-1})$ and $Y=(1,y,y^{2},...,y^{m-1})$ as first row
and first column of a perfect rank-one H$(m,n)$. The $j$-th partial sum of $X$
is $\sigma_{j}=\sum_{k=0}^{j-1}x^{k}={x^{j}-1\over x-1}$ for $1\leq j\leq n$.
If we have $\sigma_{j_{1}}=\sigma_{j_{2}}$ with
$j_{1},j_{2}\in\\{1,\dots,n\\}$, then we have $x^{j_{1}-j_{2}}=1$, hence $n$
divides $j_{1}-j_{2}$ since $x$ has order $n$ in $\mathbb{F}_{q}^{*}$. This is
possible only for $j_{1}=j_{2}$, hence the partial sums of $X$ are pairwise
distinct.

Analogously, the $i$-th partial sum of $Y$ is
$\tau_{i}=\sum_{k=0}^{i-1}y^{k}={y^{i}-1\over y-1}$ for $1\leq i\leq m$. If we
have $\tau_{i_{1}}=\tau_{i_{2}}$ with $i_{1},i_{2}\in\\{1,\dots,m\\}$, then we
have $y^{i_{1}-i_{2}}=1$, hence $m$ divides $i_{1}-i_{2}$ since $y$ has order
$m$ in $\mathbb{F}_{q}^{*}$. This is possible only for $i_{1}=i_{2}$, hence
the partial sums of $Y$ are pairwise distinct.

The assertion then follows from Proposition 5.

##  6 Main results

Let us say that an admissible pair $(m,n)$ is agreeable if there exist two
distinct odd primes $p$ and $p^{\prime}$ dividing $m$ and $n$, respectively.
We are going to give an explicit construction of a rank-one H$(m,n)$ for any
admissible agreeable pair $(m,n)$. Then we will discuss its possible
optimality.

###### Theorem 6.1

If $q=2mn+1$ is a prime power with $(m,n)$ agreeable, then there exists a
rank-one H$(m,n)$ over $\mathbb{F}_{q}$.

###### Proof

Take any pair $(m_{1},n_{1})$ of odd coprime integers greater than 1 with
$m_{1}$ dividing $m_{o}$ and $n_{1}$ dividing $n_{o}$. Such a pair certainly
exists by definition of an agreeable pair; in the worst of the cases $m_{1}$
and $n_{1}$ may be suitable odd primes. Set $m=m_{1}m_{2}$, $n=n_{1}n_{2}$ and
consider the following subsets of $\mathbb{F}_{q}^{*}$:

$X=\bigcup_{i=0}^{m_{2}-1}C^{2m_{2}n}_{i};\quad\quad
Y=\bigcup_{j=0}^{n_{2}-1}C^{2mn_{2}}_{jn_{2}}.$

We have $2m_{2}n={q-1\over m_{1}}$, hence $C^{2m_{2}n}$ is the subgroup of
$\mathbb{F}_{q}^{*}$ of order $m_{1}$. So we see that $X$ is a union of
$m_{2}$ cosets of this subgroup, hence a $m$-subset of $\mathbb{F}_{q}^{*}$
which is zero-sum by Lemma 1$(i)$. Analogously, we have $2mn_{2}={q-1\over
n_{1}}$, hence $C^{2mn_{2}}$ is the subgroup of $\mathbb{F}_{q}^{*}$ of order
$n_{1}$. Thus $Y$ is a union of $n_{2}$ cosets of this subgroup, hence a
$n$-subset of $\mathbb{F}_{q}^{*}$ which is zero-sum by Lemma 1$(i)$.

Given that $\gcd(m_{1},n_{1})=1$, by Lemma 1$(iii)$ the product of the
subgroups of $\mathbb{F}_{q}^{*}$ of orders $m_{1}$ and $n_{1}$ is the
subgroup of $\mathbb{F}_{q}^{*}$ of order $m_{1}n_{1}$:

$C^{2m_{2}n}\cdot C^{2mn_{2}}=C^{2m_{2}n_{2}}$

It follows that

$C^{2m_{2}n}_{i}\cdot C^{2mn_{2}}_{j}=C^{2m_{2}n_{2}}_{i+j}$

and then we can write:

$X\cdot Y=\displaystyle\bigcup_{(i,j)\in I\times J}C^{2m_{2}n_{2}}_{i+j}$

with $I=\\{0,1,\dots,m_{2}-1\\}$ and
$J=\\{0,m_{2},2m_{2},\dots,(n_{2}-1)m_{2}\\}$. Now note that $I+J$ is a
factorization of the whole interval $[0,m_{2}n_{2}-1]$. Thus, setting
$e=m_{2}n_{2}$, we can write

$X\cdot Y=C^{2e}_{0}\ \cup\ C^{2e}_{1}\ \cup\ \dots\ \cup\ C^{2e}_{e-1}$

It follows, by Lemma 1$(ii)$, that $X\cdot Y$ is a half-set of
$\mathbb{F}_{q}$. Then, if we take an arbitrary ordering $(x_{1},\dots,x_{m})$
of $X$ and an arbitrary ordering $(y_{1},\dots,y_{n})$ of $Y$, using the “if
part” of Proposition 2 we can finally say that the $m\times n$ array
$[x_{i}y_{j}]$ is a rank-one H$(m,n)$.

###### Example 4

Let us construct a rank-one H$(6,15)$. The pair $(6,15)$ is agreeable. Indeed
$q=2\cdot 6\cdot 15+1=181$ is a prime and $m_{1}=3$, $n_{1}=5$ are distinct
primes dividing the odd parts of 6 and 15, respectively. Using $r=2$ as
primitive element of $\mathbb{F}_{181}$ and following the instructions of
Theorem 6.1 we find that the factors of a rank-one H$(6,15)$ are

$X=C^{60}_{0}\ \cup\ C^{60}_{1}=\\{1,48,132,2,96,83\\}\quad{\rm and}$
$Y=C^{36}_{0}\ \cup\ C^{36}_{2}\ \cup\
C^{36}_{4}=\\{1,59,42,125,135,4,55,168,138,178,16,39,129,9,169\\}.$

Therefore the desired H$(6,15)$ is the following

$\left(\begin{array}[]{*{24}c}1&59&42&125&135&4&55&168&138&178&16&39&129&9&169\cr
48&117&25&27&145&11&106&100&108&37&44&62&38&70&148\cr
132&5&114&29&82&166&20&94&116&147&121&80&14&102&45\cr
2&118&84&69&89&8&110&155&95&175&32&78&77&18&157\cr
96&53&50&54&109&22&31&19&35&74&88&124&76&140&115\cr
83&10&47&58&164&151&40&7&51&113&61&160&28&23&90\end{array}\right)$

We recall that the radical of an integer $k$, denoted by $rad(k)$, is the
product of the individual prime factors of $k$ if $k>1$, and it is equal to 1
if $k=1$. We give the following definition.

###### Definition 5

An admissible pair $(m,n)$ is optimal if it is agreeable and neither
$m_{o}\cdot rad(m_{o})$ divides $n_{o}$ nor $n_{o}\cdot rad(n_{o})$ divides
$m_{o}$.

We want to exploit the proof of Theorem 3 in the best possible way in order to
establish whether it may lead to an optimal rank-one Heffter array.

###### Theorem 6.2

If an admissible pair $(m,n)$ is optimal, then there exists an optimal rank-
one H$(m,n)$.

###### Proof

Let $A$ be the rank-one H$(m,n)$ constructed in Theorem 6.1, let $X$, $Y$ be
its factors and let $S$, $T$ be their respective
$\mathbb{F}_{q}^{*}$-stabilizers. Recall that $X$ is a union of cosets of the
$m_{1}$-th roots of unity and that $Y$ is a union of cosets of the $n_{1}$-th
roots of unity. Then, by Proposition 1, the orders of $S$ and $T$ are
divisible by $m_{1}$ and $n_{1}$, respectively. Recalling that the group of
multipliers of $A$ is $S\cdot T$ by Proposition 3, we deduce that its order is
at least equal to $m_{1}n_{1}$.

Thus, to prove the assertion it is enough to show that $lcm(m_{o},n_{o})$ can
be written as a product $m_{1}n_{1}$ with $m_{1}>1$ a divisor of $m_{0}$,
$n_{1}>1$ a divisor of $n_{0}$, and $\gcd(m_{1},n_{1})=1$.

Consider first the case $m_{o}=n_{o}$. Here, to say that $(m,n)$ is optimal
simply means that $m_{o}$ has at least two distinct prime factors. Let $p$ be
one of them and let $p^{\alpha}$ be the largest power of $p$ dividing $m_{o}$.
Then $m_{1}=p^{\alpha}$ and $n_{1}={m_{o}\over p^{\alpha}}$ satisfy the
requirement.

Now assume that $m_{o}<n_{o}$. Let $\\{p_{1},\dots,p_{t}\\}$ be the set of
prime divisors of $m_{o}n_{o}$, let $p_{i}^{\alpha_{i}}$ be the largest power
of $p_{i}$ dividing $m_{o}$, and let $p_{i}^{\beta_{i}}$ be the largest power
of $p_{i}$ dividing $n_{o}$. Now let $I$ be the set of $i$’s such that
$\alpha_{i}\geq\beta_{i}$ and let $J$ be the complement of $I$ in
$\\{1,\dots,t\\}$, i.e., the set of $j$’s such that $\alpha_{j}<\beta_{j}$. It
is obvious that
$lcm(m_{o},n_{o})=\prod_{i=1}^{t}p_{i}^{\max(\alpha_{i},\beta_{i})}$, hence we
can write $lcm(m_{o},n_{o})$ as a product of a divisor $m_{1}$ of $m_{o}$ and
a divisor $n_{1}$ of $n_{o}$ as follows

$lcm(m,n)=m_{1}n_{1}\quad{\rm with}\quad m_{1}=\prod_{i\in
I}p_{i}^{\alpha_{i}}\quad{\rm and}\quad n_{1}=\prod_{j\in
J}p_{j}^{\beta_{j}}.$

It is also obvious that $m_{1}$ and $n_{1}$ are coprime. It remains to show
that $m_{1}$ and $n_{1}$ are both greater than 1. Indeed, if $m_{1}=1$, then
$I$ would be empty which means that, for every $h$, the exponent of $p_{h}$ in
the prime factorization of $m_{o}$ is strictly less than the exponent of
$p_{h}$ in the prime factorization of $n_{o}$. This clearly implies that
$m_{o}\cdot rad(m_{o})$ is a divisor of $n_{o}$ contradicting that $(m,n)$ is
optimal. Also, if $n_{1}=1$ then $J$ would be empty which means that, for
every $h$, the exponent of $p_{h}$ in the prime factorization of $m_{o}$ is
greater than or equal to the exponent of $p_{h}$ in the prime factorization of
$n_{o}$. This would imply that $m_{o}\geq n_{o}$ against the assumption. We
conclude that $m_{1}$, $n_{1}$ satisfy the requirement.

The remaining case $m_{o}>n_{o}$ can be proved exactly in the same way by
inverting the roles of $m_{o}$ and $n_{o}$.

Note that the agreeable pairs which are not optimal are, up to the order, of
the form $(2^{\alpha}\nu\rho,2^{\beta}\nu)$ where $\nu$, $\rho$ are odd
integers such that $\nu$ has at least two distinct prime divisors and
$rad(\rho)=rad(\nu)$. The least admissible pair which is agreeable but not
optimal is $(21^{2},21)$. Indeed the pair $(15^{2},15)$ is not admissible
since we have $2\cdot 15^{3}+1=43\cdot 157$.

##  7 Conclusion

We have constructed, explicitly, a rank-one H$(m,n)$ for any admissible
agreeable pair $(m,n)$.

The existence question for rank-one Heffter arrays remains open for admissible
disagreeable pairs, which are precisely the pairs $(m,n)$ where either one of
the two parameters is equal to a power of 2, or their product has exactly one
odd prime factor. Apart from some examples where $m$ and $n$ are “small” (see,
e.g,. Example 2), we doubt that rank-one “disagreeable” Heffter arrays are
obtainable by means of a very explicit construction. Yet, we strongly believe
that their existence can be proved in any admissible case by means of some
tools which are a little bit more sophisticated than those used in this paper.
We hope to find the final proof in a future work.

## Acknowledgement

The author would like to thank Anita Pasotti for helpful discussions on the
topic and the anonymous referee who pointed out that the perfect rank-one
Heffter arrays of Corollary 1 are globally simple.

This work has been performed under the auspices of the G.N.S.A.G.A. of the
C.N.R. (National Research Council) of Italy.

## References

  * (1) Archdeacon, D.S.: Heffter arrays and biembedding graphs on surfaces, Electron. J. Combin. 22, $\sharp$P1.74 (2015).
  * (2) Archdeacon, D.S., Boothby, T., Dinitz, J.H.: Tight Heffter arrays exist for all possible values, J. Combin. Des. 25, 5–35 (2017).
  * (3) Bose, R.C.: On the construction of balanced incomplete block designs, Ann. Eugenics 9, 353–399 (1939).
  * (4) Bonvicini, S., Buratti, M., Garonzi, M., Rinaldi, G., Traetta, T.: The first families of highly symmetric Kirkman Triple Systems whose orders fill a congruence class, Des. Codes Cryptogr. 89, 2725–2757 (2021).
  * (5) Buratti, M., Nakic, A.: Super-regular Steiner $2$-designs, Finite Fields Appl. 85 (2023), Article number 102116.
  * (6) Buratti, M., Pasotti A.: Heffter and magic configurations, in preparation.
  * (7) Caggegi, A., Falcone, G., Pavone, M.: On the additivity of block designs, J. Algebr. Comb. 45, 271–294 (2017).
  * (8) Cavenagh, N.J., Donovan, D.M., Yazici, E.S.: Biembeddings of cycle systems using integer Heffter arrays, J. Combin. Designs 29, 750–764 (2021).
  * (9) Costa, S., Pasotti, A., Pellegrini, M.A.: Relative Heffter arrays and biembeddings, Ars Math. Contemp. 18, 241–271 (2020).
  * (10) Costa, S., Morini, F., Pasotti, A., Pellegrini M.A.: Globally simple Heffter arrays and orthogonal cyclic cycle decompositions, Australas. J. Combin. 72, 549–593 (2018).
  * (11) Pasotti, A., Dinitz, J.H.: A survey of Heffter arrays, arXiv:2209.13879
  * (12) Wilson, R.M.: Cyclotomy and difference families in elementary abelian groups, J. Number Theory 4, 17–47 (1972).


# Self-supervised Visual-LiDAR Odometry with Flip Consistency

Bin Li  College of Information Science and Electronic Engineering, Zhejiang
University, China
Mu Hu  College of Information Science and Electronic Engineering, Zhejiang
University, China
Shuling Wang  College of Information Science and Electronic Engineering,
Zhejiang University, China
Lianghao Wang  Zhejiang Lab, Hangzhou, China  Xiaojin Gong The corresponding
authorThis work was supported in part by the Natural Science Foundation of
Zhejiang Province, China under Grant No. LY17F010007 and LY18F010004. College
of Information Science and Electronic Engineering, Zhejiang University, China

###### Abstract

Most learning-based methods estimate ego-motion by utilizing visual sensors,
which suffer from dramatic lighting variations and textureless scenarios. In
this paper, we incorporate sparse but accurate depth measurements obtained
from lidars to overcome the limitation of visual methods. To this end, we
design a self-supervised visual-lidar odometry (Self-VLO) framework. It takes
both monocular images and sparse depth maps projected from 3D lidar points as
input, and produces pose and depth estimations in an end-to-end learning
manner, without using any ground truth labels. To effectively fuse two
modalities, we design a two-pathway encoder to extract features from visual
and depth images and fuse the encoded features with those in decoders at
multiple scales by our fusion module. We also adopt a siamese architecture and
design an adaptively weighted flip consistency loss to facilitate the self-
supervised learning of our VLO. Experiments on the KITTI odometry benchmark
show that the proposed approach outperforms all self-supervised visual or
lidar odometries. It also performs better than fully supervised VOs,
demonstrating the power of fusion.

##  1 Introduction

Ego-motion or pose estimation is a key problem in simultaneous localization
and mapping (SLAM), which plays an important role in various applications such
as autonomous driving[35], 3D reconstruction [6], and augmented reality [16].
Most traditional methods [29, 35, 50, 37] address this problem by following a
feature extraction and matching pipeline, relying on geometry models for
optimization. These methods have shown excellent performance, but they depend
on hand-crafted modules and lack the ability of adapting to all challenging
scenarios. Recently, learning-based techniques [52, 45, 19, 22, 2] have
attracted more and more research interest. Although these methods perform
inferior to traditional counterparts, they demonstrate advantages in dealing
with tough cases such as rotation-only motion or tracking lost scenarios.

To date, most learning-based methods use visual sensors. These methods, also
referred to as deep visual odometries (VOs) [52, 20, 19], take advantage of
rich information in monocular or stereo images, but are sensitive to lighting
conditions. Several works [18, 22, 2] adopt ranging sensors such as lidars for
ego-motion estimation. These sensors can provide accurate depth measurements
and are robust to the change of lighting conditions, but their data are
sparse. Therefore, visual and ranging sensors are highly complementary so that
it is desirable to integrate them to achieve better performance. In addition,
considering that most autonomous vehicles are equipped with both visual and
ranging sensors nowadays, the study of visual-lidar odometries (VLOs) will be
of great value for academic research and industrial applications.

In this work, we propose a learning-based method to implement a visual-lidar
odometry. In order to effectively fuse visual and lidar information, we first
project 3D lidar points to obtain image-aligned sparse depth maps. Then, we
design a network composed of a two-pathway encoder to extract features from a
monocular image sequence and a sparse depth map sequence, respectively. The
extracted features are fused at multiple scales and employed to jointly
predict pose and dense depth values. Considering the difficulty of obtaining
ground-truth labels, we opt to learn our VLO under self-supervision. To this
end, we adopt a siamese network architecture and design a flip consistency
loss to facilitate the self-supervised learning.

The main contributions of our work are summarized as follows:

  * •

To the best of our knowledge, this work is the first end-to-end learning-based
visual-lidar odometry. It inputs monocular images and sparse depth maps, and
outputs both depth and pose estimations.

  * •

We adopt a siamese network architecture and design a flip consistency loss to
learn our VLO under self-supervision, without using any ground-truth pose and
depth labels.

  * •

Evaluations on the KITTI odometry benchmark [5] show that our proposed method
outperforms all self-supervised visual or lidar odometries. It also performs
better than all fully supervised VOs, demonstrating the power of fusion.

##  2 Related Work

###  2.1 Visual & Lidar Odometry

Visual odometry aims to estimate ego-motion based on images. Traditional VOs
can be classified into feature-based or direct methods. The former [29, 16,
26] follows a standard pipeline including feature extraction and matching,
motion estimation, and local/global optimization. The direct methods [28, 13]
minimize a photometric error that takes all pixels into account, assuming that
scenes are stationary and of a Lambertian reflectance. However, these
assumptions are hard to be fully satisfied so that most direct methods perform
inferior to their feature-based counterparts.

Recently, deep learning techniques have been firstly applied to supervised VO
[38, 39, 42, 43] that requires ground-truth pose labels. Considering the
hardness of obtaining ground-truth, more and more works focus on unsupervised
VO. The pioneer work, SfMLearner [52], takes a monocular video sequence as
input to jointly learn depth and pose networks by minimizing a photometric
loss. Following it, various end-to-end learning methods have been developed.
For instance, UnDeepVO[33], Zhu et al.[53], and Zhan et al.[49] take stereo
sequences as training inputs to address the scale ambiguity in monocular VO.
SAVO [20] and Li et al. [19] adopt recurrent neural networks (RNNs) to encode
temporal information. Additional predictions such as optical flow [45],
disparity map [53], and motion mask [33, 8] are also included to address
motion or occlusion problem. In addition, some recent works such as DVSO [27]
and D3VO [44] integrate the end-to-end pose estimation with traditional global
optimization modules to boost VO performance further.

Lidar odometry estimates the relative pose between two 3D point cloud frames.
Although a lidar provides accurate depth information, the irregular sparsity
of point clouds makes LO a challenging problem. Most previous lidar odometry
works, such as LOAM [50], LeGO-LOAM [37], and LOL [34], are based on
traditional pipelines and achieve state-of-the-art performance. Recently, deep
learning techniques have also been applied to this task. For instance, L3-Net
[22] replaces each module in the traditional pipeline with a deep neural
network in order to learn from data. DeepVCP [21], DeepLO [2] and LO-Net [18]
design various end-to-end learning frameworks. The former [21] adopts a 3D
convolutional neural network (CNN) for feature extraction and matching, while
the latter two [2, 18] project point clouds into 2D maps via a cylindrical or
spherical projection and use 2D CNNs for processing. Most learning-based LOs
are performed under full supervision, except DeepLO [2].

Visual-lidar odometry takes advantage of visual and lidar information for ego-
motion estimation. Up to now, most of the existing VLOs are based on
traditional methods. For instance, V-LOAM [51] integrates the results of VO
and LO, both of which adopt the feature-based pipelines, to boost the
performance. LIMO [9] provides a feature-based monocular VO with the scale
estimated from lidar data to achieve ego-motion estimation. Recently, there
are several visual and lidar based methods making use of learning techniques.
Yue et al. [46] exploit a deep network to enrich 3D point clouds with the
guidance of high-resolution images. But they employ a traditional point cloud
registration method NDT [25] to estimate pose. Vitor et al. [10] incorporates
depth supervision into a self-supervised VO to produce scale-aware pose
estimation. But the input of their network only contains visual images. In
this work, we propose a self-supervised visual-lidar odometry. It takes a
sequence of monocular images and sparse depth maps projected from
corresponding 3D lidar points as inputs to learn a depth and pose estimation
network in an end-to-end manner, without using any ground-truth pose and depth
labels.

###  2.2 Depth Estimation

Depth estimation works can be classified into depth prediction and depth
completion categories, according to the KITTI benchmark [5]. The former [4]
estimates depth from a single color image, while the latter takes a color
image and a sparse depth map as the inputs to produce a dense depth map. In
contrast to learning-based VOs [38, 52, 49] that input an image sequence, most
depth estimation works take a single frame as input. They focus more on the
design of network architectures to encode multi-scale contexts [17], semantics
[3] or surface normals [32] for performance improvement. Moreover, most depth
estimation methods are based on supervised learning, requiring ground-truth
depth maps that are extremely hard to obtain. Recently, unsupervised methods
have also been developed. For instance, Ma et al. [24] implements self-
supervised depth completion via also taking a sequence of images and sparse
depth maps as inputs. Their work shares the same inputs with ours, but it uses
a traditional method to estimate ego-motion between two frames.

Figure 1: The network architecture of our visual-lidar odometry, referred to
as VLONet. It follows an encoder-decoder structure and consists of three major
components: a two-pathway encoder, a pose estimator and a depth prediction
decoder.

###  2.3 Consistency Regularization

Consistency regularization aims to improve a model’s performance via enforcing
the consistency between two output predictions. For instance, some stereo
matching [48] and depth estimation [30, 7, 23] works take a stereo pair as
inputs and place a left-right consistency on their corresponding outputs.
Optical flow [54] takes consecutive images as inputs and impose a forward-
backward consistency. Recently, the consistency between the outputs of an
original image and its transformed (e.g. rescaled, rotated, or flipped)
version has been exploited to construct supervision for various self-
supervised learning tasks in image classification [1, 11], object detection
[14], semantic segmentation [40], etc. In this work, we adopt the flip
consistency and design an adaptively weighted consistency loss specifically
for self-supervised visual-lidar odometry.

##  3 The Proposed Method

In this section, we first present the network architecture designed for our
visual-lidar odometry. Then we introduce the proposed losses to achieve the
self-supervised learning.

###  3.1 Network Architecture

Visual-lidar fusion: When designing the network architecture, the first
problem we need to address is how to fuse visual and lidar modalities.
Inspired by recent depth completion works [32, 36], we adopt an encoder-
decoder fusion scheme and design the network as shown in Figure 1. First, in
contrast to deep LOs [2, 18] that take a cylindrical or spherical projection,
we project each 3D point cloud into its aligned image frame and obtain a 2D
sparse depth map. Although this projection manner leaves out the 3D points
that are not in the camera’s field of view, it facilitates the fusion of color
and depth information.

We refer to our visual-lidar odometry network as VLONet. It has a two-pathway
encoder. Each takes either an image sequence or a depth map sequence as input.
The features extracted in each pathway are fused by a point-wise addition for
the prediction of relative pose between two consecutive frames. In addition,
the depth and visual features are also fused together with decoder features
via a feature fusion module at each scale, as shown in Figure 1, to predict a
dense depth map. In contrast to most deep VOs [52, 33, 49] that construct two
separate networks for depth and pose prediction, our depth and pose estimation
share the same encoder, which can not only reduce the computational cost but
also extract more powerful features.

A siamese architecture for the self-supervised learning: Inspired by recent
unsupervised learning works [1, 11, 14, 40], we adopt a siamese architecture
for our self-supervised VLO. More specifically, the architecture consists of
two siamese VLONets, as shown in Figure 2. One network takes original image
and depth sequences as input, and the other takes the horizontally flipped
images and depth maps as input. These two networks share the same structure
and parameters. Based on this architecture, we can impose a consistency
regularization to facilitate our self-supervised learning. This siamese
architecture is used for training, while only one VLONet is needed for test.

Figure 2: An overview of our Self-VLO. It consists of two VLONets that share
the same structures and parameters with each other. One VLONet takes original
sequences as input and the other inputs horizontally flipped sequences. Each
network is supervised with corresponding depth and pose prediction losses.
There is also a flip consistency loss imposed to constrain the outputs of two
VLONets.

###  3.2 Depth and Pose Prediction Losses

We first introduce the losses for the prediction of depth and pose in a single
VLONet. As no ground-truth labels are available, we adopt a view synthesis
loss $\mathcal{L}_{vs}$, together with a depth fidelity loss
$\mathcal{L}_{df}$ and a depth smoothness loss $\mathcal{L}_{ds}$ for
supervision. Therefore, the entire loss of a single VLONet is defined by

$\mathcal{L}_{vlo}=\lambda_{vs}\mathcal{L}_{vs}+\lambda_{df}\mathcal{L}_{df}+\lambda_{ds}\mathcal{L}_{ds},$
(1)

in which $\lambda_{vs}$, $\lambda_{df}$, and $\lambda_{ds}$ are scaling
factors to balance the terms. For the purpose of self-containedness, we
briefly introduce each term as follows.

View synthesis loss, which is extensively used in unsupervised VOs [52, 33],
measures the difference between a target image and an image synthesized from a
source image. Assuming that each sequence contains $2N+1$ frames, we denote
the input image sequence by
$\\{I_{t-N},\cdots,I_{t-1},I_{t},I_{t+1},\cdots,I_{t+N}\\}$ and the depth map
sequence by $\\{D_{t-N},\cdots,D_{t-1},D_{t},D_{t+1},\cdots,D_{t+N}\\}$. The
frame $I_{t}$ and $D_{t}$ are the target views and the rest are the source
views denoted by $I_{s}$ and $D_{s}$, $s\in\\{t-N,\cdots,t+N\\}$, $s\neq t$.
The VLONet predicts a dense depth map $\widehat{D}_{t}$ and a relative pose
$\widehat{P}_{t\rightarrow
s}=[\hat{t}_{x},\hat{t}_{y},\hat{t}_{z},\hat{r}_{x},\hat{r}_{y},\hat{r}_{z}]$
for each target and source pair. Here, $[\hat{t}_{x},\hat{t}_{y},\hat{t}_{z}]$
is a 3D translational vector and $[\hat{r}_{x},\hat{r}_{y},\hat{r}_{z}]$ is a
3D Euler angle, from which we can construct a $4\times 4$ transformation
matrix $\widehat{T}_{t\rightarrow s}$.

Given a pixel $p_{t}$ in the target image $I_{t}$, we can get its
corresponding pixel $p_{s}$ in a source image $I_{s}$ by the following
transform:

$p_{s}=K\widehat{T}_{t\rightarrow s}\widehat{D}_{t}(p_{t})K^{-1}p_{t},$ (2)

where

$K=\left[\begin{array}[]{ccc}f_{x}&0&c_{x}\\\ 0&f_{y}&c_{y}\\\
0&0&1\end{array}\right]$ (3)

is a camera intrinsic matrix provided in training. Further, by taking the
differentiable bilinear sampling mechanism [52], we warp the source image into
the target frame to get the synthesized image $\tilde{I}_{s\rightarrow t}$, in
which $\tilde{I}_{s\rightarrow t}(p_{t})=I_{s}(\left\langle
p_{s}\right\rangle)$.

Then, a view synthesis loss is designed to measure the difference between
$I_{t}$ and $\tilde{I}_{s\rightarrow t}$. To this end, we combine a L1 error
with a SSIM [47] and define the loss as follows:

$\mathcal{L}_{vs}=\sum_{s}(1-\alpha_{s})\|I_{t}-\tilde{I}_{s\rightarrow
t}\|_{1}+\alpha_{s}(1-SSIM(I_{t},\tilde{I}_{s\rightarrow t})),$ (4)

where $\alpha_{s}$ is a scalar to balance the two terms. According to Equation
(2), we know that this loss provides a supervision for both depth and pose
estimations.

Depth fidelity loss is to ensure the predicted dense depth map have the same
values with the input sparse depth map at these valid pixels. Therefore, this
loss is defined by

$\mathcal{L}_{ds}=\|M(D_{t})\odot(D_{t}-\widehat{D}_{t})\|_{1},$ (5)

in which $M(D_{t})=\mathbbm{1}(D_{t}>0)$ indicates the valid pixels in the
sparse map. Note that the supervision of depth comes from the inputs and no
extra ground-truth depth labels are required. Therefore, our method is self-
supervised.

Depth smoothness loss is imposed to improve the quality of the predicted dense
depth map. Considering the correlation between color and depth images, we
introduce an image-guided loss to smooth depth while keep sharp on object
boundaries. This loss is defined by

$\displaystyle\mathcal{L}_{sm}$
$\displaystyle=\sum_{p}\exp\left(-|\nabla_{x}I_{t}(p)|\right)|\nabla_{x}\widehat{D}_{t}(p)|$
(6)
$\displaystyle+\exp\left(-|\nabla_{y}I_{t}(p)|\right)|\nabla_{y}\widehat{D}_{t}(p)|,$
(7)

where $\nabla_{x}$, $\nabla_{y}$ denote the gradients.

###  3.3 The Flip Consistency Loss

By taking advantage of the siamese architecture, we also impose a consistency
regularization to facilitate our self-supervised training. It is based on the
following assumption [40]: if there is an affine transformation $A(\cdot)$ on
the input, the corresponding output is inclined to be equivariant. That is,
$G(A(I,D))=A(G(I,D))$, where $G(\cdot)$ denotes our VLONet function mapping
the input to the output. In this work, we only take the horizontal flip into
account for simplicity. Therefore, we first derive the relation between two
outputs and then design a flip consistency loss to regularize the two outputs.

Recall that the original input sequences are $\left\langle
I_{t},I_{s}\right\rangle$ and $\left\langle D_{t},D_{s}\right\rangle$, and
their outputs are $\widehat{D}_{t}$ and $\widehat{P}_{t\rightarrow
s}=[\hat{t}_{x},\hat{t}_{y},\hat{t}_{z},\hat{r}_{x},\hat{r}_{y},\hat{r}_{z}]$.
We horizontally flip each image and depth frame while keep their temporal
order unchanged. The flipped sequences are denoted by $\left\langle
I^{f}_{t},I^{f}_{s}\right\rangle$ and $\left\langle
D^{f}_{t},D^{f}_{s}\right\rangle$, and the corresponding outputs are denoted
by $\widehat{D}_{t}^{f}$ and $\widehat{P}^{f}_{t\rightarrow
s}=[\hat{t}^{f}_{x},~{}\hat{t}^{f}_{y},~{}\hat{t}^{f}_{z},~{}\hat{r}^{f}_{x},~{}\hat{r}^{f}_{y},~{}\hat{r}^{f}_{z}]$.
These two outputs have the following relations:

$\left\\{\begin{array}[]{lr}\widehat{D}_{t}^{f}=flip(\widehat{D}_{t})&\\\ &\\\
\begin{aligned} \widehat{P}^{f}_{t\rightarrow
s}&=[\hat{t}^{f}_{x},~{}\hat{t}^{f}_{y},~{}\hat{t}^{f}_{z},~{}\hat{r}^{f}_{x},~{}\hat{r}^{f}_{y},~{}\hat{r}^{f}_{z}]\\\
&=[-\hat{t}_{x},\hat{t}_{y},\hat{t}_{z},\hat{r}_{x},-\hat{r}_{y},-\hat{r}_{z}]&\end{aligned}\end{array}\right.$
(8)

where $flip(\cdot)$ denotes the horizontal flip operation. Note that, when an
image is flipped, the intrinsic matrix of the camera is also changed into the
following one

$K^{f}=\left[\begin{array}[]{ccc}f_{x}&0&W-c_{x}\\\ 0&f_{y}&c_{y}\\\
0&0&1\end{array}\right]$ (9)

where $W$ is the image width.

Based on the above-mentioned relations, we place consistency constraints on
both depth and pose estimation results. Therefore, the flip consistency loss
is defined as follows.

$\mathcal{L}_{fc}=\mathcal{L}_{dfc}+\mathcal{L}_{pfc},$ (10)

in which

$\mathcal{L}_{dfc}=\|flip(\widehat{D}_{t})-\widehat{D}_{t}^{f}\|_{1},$ (11)

and

$\displaystyle\mathcal{L}_{pfc}$
$\displaystyle=\|\hat{t}_{x}+\hat{t}_{x}^{f}+\hat{t}_{y}-\hat{t}_{y}^{f}+\hat{t}_{z}-\hat{t}_{z}^{f}\|_{1}$
(12)
$\displaystyle+\alpha_{r}\|\hat{r}_{x}-\hat{r}^{f}_{x}+\hat{r}_{y}+\hat{r}^{f}_{y}+\hat{r}_{z}+\hat{r}^{f}_{z}\|_{1}$

are two losses to constrain the consistency of the depth and pose estimations,
respectively. $\alpha_{r}$ is a scalar to balance the translational error and
the rotational error.

###  3.4 The Entire Loss

In summary, when training with the siamese network architecture, we have two
depth and pose estimation losses $\mathcal{L}_{vlo}$ and
$\mathcal{L}^{f}_{vlo}$ , respectively, for the original and flipped
sequences, together with a flip consistency loss $\mathcal{L}_{fc}$ to
constrain the two outputs. Therefore, the entire loss is

$\mathcal{L}=\mathcal{L}_{vlo}+\mathcal{L}^{f}_{vlo}+\phi\mathcal{L}_{fc}.$
(13)

Here,

$\phi=\lambda_{fc}\cdot\exp\left(-\frac{\mathcal{L}_{vs}}{\sigma}\right)$ (14)

is an adaptive weight controlled by the view synthesis loss $\mathcal{L}_{vs}$
to balance the prediction losses and the consistency loss. $\lambda_{fc}$ and
$\sigma$ are two scalars to ensure the adaptive weight $\phi$ range from 0 to
1.

The reason to employ this adaptive weighting scheme [41] is because the
consistency loss $\mathcal{L}_{fc}$ is prone to result in a trivial solution
with all zeros in depth and pose values. Involving the consistency loss
permaturely may make the model parameters oscillate with time when the depth
and pose estimates are incorrect, leading to a slow convergence at the early
epochs. Therefore, we use this adaptive weighting scheme to vary the weight
during training time by letting $\phi$ be inversely proportional to the
$\mathcal{L}_{vs}$. $\phi$ is small when $\mathcal{L}_{vs}$ is large and
$\phi$ will be close to 1 when $\mathcal{L}_{vs}$ goes to 0 as the training
converges.

Models | Inputs | DA | SA | FC | Seq.09 | Seq.10 | Mean
---|---|---|---|---|---|---|---
$t_{rel}$ | $r_{rel}$ | $t_{rel}$ | $r_{rel}$ | $t_{rel}$ | $r_{rel}$
VLO1 | M+L |  |  |  | 4.33 | 1.72 | 3.30 | 1.40 | 3.82 | 1.56
VLO2 | M+L | $\surd$ |  |  | 3.42 | 1.59 | 4.04 | 1.40 | 3.73 | 1.50
VLO3 | M+L | $\surd$ | $\surd$ |  | 3.21 | 1.33 | 3.56 | 1.14 | 3.39 | 1.24
VLO4 | M+L | $\surd$ | $\surd$ | $\surd$ | 2.58 | 1.13 | 2.67 | 1.28 | 2.62 | 1.21
VO1 | M |  |  |  | 11.34 | 3.15 | 16.70 | 5.04 | 14.02 | 4.10
VO2 | M | $\surd$ |  |  | 9.83 | 3.53 | 14.92 | 4.43 | 12.38 | 3.98
VO3 | M | $\surd$ | $\surd$ |  | 8.30 | 2.56 | 15.90 | 4.92 | 12.10 | 3.74
VO4 | M | $\surd$ | $\surd$ | $\surd$ | 7.00 | 2.41 | 11.74 | 3.33 | 9.37 | 2.87

Table 1: Comparison of the proposed method and its variants. Here, M denotes
the input taking only monocular image sequences, M+L is the input taking both
monocular images and depth maps projected from lidar points. DA denotes the
data augmentation strategy, SA is the siamese architecture, and FC is the flip
consistency loss. $t_{rel}$ is the translational error (%) and $r_{rel}$ is
the rotational error (deg/100m). The best results of VLOs or VOs are marked in
bold.

##  4 Experiments

###  4.1 Experiment Setting

Dataset and evaluation metrics. We evaluate our self-supervised visual-lidar
odometry and its variants on the KITTI odometry benchmark [5]. As the common
practice [52, 33, 53], we take sequence 00-08 for training and test on
sequence 09 and 10. To evaluate the pose estimation results, we compute the
average translational error (%) and rotational error (deg/100m) on all
possible sub-sequences of length $(100,200,\cdots,800)$ meters, following the
official criteria provided in the KITTI benchmark .

Implementation details. We implement the proposed model based on the PyTorch
[31] framework. In all experiments, the length of each input sequence is 3.
Each image or depth map is resized into $192\times 624$, which is half of the
original resolution, to save computational cost. We train our full model using
a single NVIDIA GTX 2080Ti. The full model takes 50 hours or so for training.
It infers both depth and pose at a rate of 40Hz during test time.

The details of our network, including the operations and channel size, are
marked in Figure 1. In addition, we use batch normalization [12] and ReLU
activation for all convolutional layers except those in the prediction layers.
During training, we use the Adam optimizer [15] with $\beta_{1}=0.9$,
$\beta_{2}=0.999$ and mini-batch size of 4 to train the network for 180K
iterations. The initial learning rate starts from 0.0002 and decreases by half
for every 70K iterations. We empirically set the hyper-parameters as follows:
$\lambda_{vs}=2.0$, $\lambda_{df}=0.2$, $\lambda_{ds}=40.0$,
$\alpha_{s}=0.85$, and $\alpha_{r}=2.0$. In the adaptive weight $\phi$, we set
the scaling factors $\lambda_{fc}=2.5$ and $\sigma=0.2$.

Data augmentation. In our network, each input sequence is a 3-frame snippet.
If we only select consecutive frames as inputs, then the low-speed snippets
may take the majority while the high-speed snippets take a very small portion,
as shown in Figure 3(a). This imbalanced distribution makes a model perform
poorly at high-speed scenarios. To address this issue, we opt to enlarge the
sampling interval by sampling 3 frames, respectively, at time $t-2$, $t$, and
$t+2$ with a probability $p=0.6$. By this means, the speed distribution of two
consecutive frames in 3-frame snippets is more balanced, as shown in Figure
3(b). In addition, we also observe that extremely low-speed snippets (e.g. the
speed between two frames is slower than 10km/h) may hurt the performance. The
reason is that the geometry model of pose estimation degenerates when two
frames are too close to each other. Therefore, we leave out these snippets
during training. Except these, no other data augmentation techniques such as
random scaling, cropping or flipping are adopted.

(a) Without DA

(b) With DA

Figure 3: The distributions of the speed between two consecutive frames in
3-frame snippets sampled from the KITTI odometry benchmark (Seq. 00-08). DA
refers to the data augmentation strategy designed in our work.

###  4.2 Ablation Studies

We first conduct a series of experiments to investigate the effectiveness of
each component proposed in our model. To this end, we test the following four
model variants: 1) VLO1: the model using only a single VLONet and without our
data augmentation (DA) strategy; 2) VLO2: the model using one VLONet; 3) VLO3:
the model using the siamese architecture composed of two VLONets but without
the flip consistency loss; 4) VLO4: the full model. All of the latter three
models are trained with DA. In addition, to validate the effectiveness of
visual-lidar fusion, we also investigate four visual models, denoted by VO1-4,
which are respectively corresponding to VLO1-4 but take only monocular images
as input.

Table 1 presents the pose estimation results of all variants. Note that the VO
models take only monocular images as input that lead to a scale ambiguity
problem, we recover the scales by a post-processing step as done in [52, 33].
When comparing each VLO model with the corresponding VO counterpart, we
observe that the fusion of visual and lidar information improves the pose
estimation performance dramatically. In addition, all proposed components
including the data augmentation strategy, the siamese architecture, and the
flip consistency loss, gradually boost the performance in both VLOs and VOs,
demonstrating their effectiveness. Figure 4 presents the trajectories of
sequence 09 and 10, produced by four model variants. As shown in the figure,
both VLO1 and VLO4 generate the trajectories much closer to the ground truth
than the VO models, and the full model VLO4 is the closest.

(a) Seq. 09

(b) Seq. 10

Figure 4: Trajectories of sequence 09 and 10. Figure 5: Translational and
rotational errors along each axis, averaged on sub-sequences of sequence 09
with different length. Top row: the averaged translational errors along $X$,
$Y$, $Z$ axis, respectively. Bottom row: the averaged rotational errors along
pitch, yaw, and roll.

| Method | Inputs | Supervision | Seq.09 | Seq.10 | Mean(09-10)
---|---|---|---|---|---|---
$t_{rel}$ | $r_{rel}$ | $t_{rel}$ | $r_{rel}$ | $t_{rel}$ | $r_{rel}$
Supervised |  DeepVO[38] | Mono | Pose | - | - | 8.11 | 8.83 | 8.11 | 8.83
ESP-VO[39] | Mono | Pose | - | - | 9.77 | 10.2 | 9.77 | 10.2
GFS-VO[42] | Mono | Pose | - | - | 6.32 | 2.33 | 6.32 | 2.33
Xue et al.[43] | Mono | Pose | - | - | 3.94 | 1.72 | 3.94 | 1.72
LO-Net[18] | LiDAR | Pose | 1.37 | 0.58 | 1.80 | 0.93 | 1.59 | 0.76
Self-supervised |  SfMLearner[52] | Mono | - | 18.77 | 3.21 | 14.33 | 3.30 | 16.55 | 3.26
UnDeepVO[33] | Stereo | - | 7.01 | 3.61 | 10.63 | 4.65 | 8.82 | 4.13
Zhu et al.[53] | Stereo | - | 4.66 | 1.69 | 6.30 | 1.59 | 5.48 | 1.64
Zhan et al.[49] | Stereo | - | 11.92 | 3.60 | 12.62 | 3.43 | 12.27 | 3.52
Gordon et al.[8] | Mono | - | 3.10 | - | 5.40 | - | 4.25 | -
SAVO [20] | Mono | - | 9.52 | 3.64 | 6.45 | 2.41 | 7.99 | 3.03
Li et al. [19] | Mono | - | 5.89 | 3.34 | 4.79 | 0.83 | 5.34 | 2.09
DeepLO[2] | LiDAR | - | 4.87 | 1.95 | 5.02 | 1.83 | 4.95 | 1.89
Self-VLO | Mono+LiDAR | - | 2.58 | 1.13 | 2.67 | 1.28 | 2.62 | 1.21

Table 2: Comparison of the proposed method with state-of-the-art learning
based methods. Note that the self-supervised methods are trained on sequence
00-08. For the supervised methods, [38, 39, 42, 43] are trained on sequence
00, 02, 08 and 09, and [18] is trained on sequence 00-06.

In order to take a close look at how the proposed models make improvements,
Figure 5 also plots the translational and rotational errors along each axis,
averaged on sub-sequences of sequence 09 with a length of
$(100,200,\cdots,800)$ meters. We see that the fusion of visual and lidar
information, as shown by VLO1 and VLO4 in Figure 5, dramatically improves the
performance on $t_{x}$ (along the lateral direction), $t_{z}$ (along the
driving direction), and $r_{y}$ (yaw), which are the directions dominate a
vehicle’s motion. Besides, the errors of VLO4 are more stable than the other
variants, indicating that the full model is more robust to the change of
driving speed and road scenarios.

###  4.3 Comparison with State-of-the-arts

In this section, we compare our proposed full model (named as Self-VLO) with
both fully supervised and unsupervised/self-supervised methods. All previous
learning-based methods either take visual images or lidar data as input.
Therefore, the state-of-the-art fully supervised methods include four visual
odometries [38, 39, 42, 43] and one lidar odometry [18]. The self-supervised
methods include seven VOs [52, 33, 53, 49, 8, 20, 19] and one LO [2]. These
visual methods may take monocular or stereo image sequences as input for
training, but all of them infer pose from monocular images during test.

Table 2 summarizes the comparison results. From it we see that our Self-VLO
outperforms all visual based methods, including both fully supervised and
self-supervised ones, on test sequences 09 and 10. Moreover, Self-VLO also
performs better than the unsupervised LO [2] although it takes $360^{o}$ field
of view into account. These comparisons demonstrate the effectiveness of our
method.

##  5 Conclusions

In this paper, we have presented a self-supervised visual-lidar odometry. It
takes advantage of rich visual information and accurate depth information to
improve the performance of ego-motion estimation. We adopt a siamese
architecture and design an adaptively weighted flip consistency loss to
facilitate the learning of our VLO under self-supervision, requiring no manual
annotations. Experiments on KITTI validate the effectiveness of our proposed
method.

## References

  * [1] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton.  A simple framework for contrastive learning of visual representations.  arXiv preprint arXiv:2002.05709, 2020.
  * [2] Y. Cho, G. Kim, and A. Kim.  Unsupervised geometry-aware deep lidar odometry.  In ICRA, 2020.
  * [3] D. Eigen and R. Fergus.  Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture.  In ICCV, 2015.
  * [4] D. Eigen, C. Puhrsch, and R. Fergus.  Depth map prediction from a single image using a multi-scale deep network.  In NIPS, 2014.
  * [5] A. Geiger, P. Lenz, and R. Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In CVPR, 2012.
  * [6] A. Geiger, J. Ziegler, and C. Stiller.  Stereoscan: Dense 3d reconstruction in real-time.  In IVS, 2011.
  * [7] C. Godard, O. Mac, and G. Brostow.  Unsupervised monocular depth estimation with left-right consistency.  In CVPR, 2017.
  * [8] A. Gordon, H. Li, R. Jonschkowski, and A. Angelova.  Depth from videos in the wild: Unsupervised monocular depth learning from unknown cameras.  In ICCV, 2019.
  * [9] J. Graeter, A. Wilczynski, and M. Lauer.  Limo: Lidar-monocular visual odometry.  In IROS, 2018.
  * [10] V. Guizilini, J. Li, R. Ambrus, S. Pillai, and A. Gaidon.  Robust semi-supervised monocular depth estimation with reprojected distances.  In CoRL, 2019.
  * [11] H. Guo, K. Zheng, X. Fan, H. Yu, and S. Wang.  Visual attention consistency under image transforms for multi-label image classification.  In CVPR, 2019.
  * [12] S. Ioffe and C. Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  arXiv preprint arXiv:1502.03167, 2015.
  * [13] T. Schöps J. Engel and D. Cremers.  LSD-SLAM: Large-Scale Direct monocular SLAM.  In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 8690 LNCS, pages 834--849, 2014.
  * [14] J. Jeong, S. Lee, J. Kim, and N. Kwak.  Consistency-based semi-supervised learning for object detection.  In NIPS, 2019.
  * [15] D. Kingma and J. Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [16] G. Klein and D. Murray.  Parallel tracking and mapping for small ar workspaces.  In ISMAR, 2007.
  * [17] A. Li, Z. Yuan, Y. Ling, W. Chi, S. Zhang, and C. Zhang.  A multi-scale guided cascade hourglass network for depth completion.  In WACV, 2020.
  * [18] Q. Li, S. Chen, C. Wang, X. Li, C. Wen, M. Cheng, and J. Li.  Lo-net: Deep real-time lidar odometry.  In CVPR, 2019.
  * [19] S. Li, X. Wang, Y. Cao, F. Xue, Z. Yan, and H. Zha.  Self-supervised deep visual odometry with online adaptation.  In CVPR, 2020.
  * [20] S. Li, F. Xue, X. Wang, Z. Yan, and H. Zha.  Sequential adversarial learning for self-supervised deep visual odometry.  In ICCV, 2019.
  * [21] W. Lu, G. Wan, Y. Zhou, X. Fu, P. Yuan, and S. Song.  Deepvcp: An end-to-end deep neural network for point cloud registration.  In ICCV, 2019.
  * [22] Y. Lu, W.and Zhou, G. Wan, S. Hou, and S. Song.  L3-net: Towards learning based lidar localization for autonomous driving.  In CVPR, 2019.
  * [23] T. Hassner M. Goldman and S. Avidan.  Learn stereo, infer mono: Siamese networks for self-supervised, monocular, depth estimation.  In CVPRW, 2019.
  * [24] F. Ma, G. V. Cavalheiro, and S. Karaman.  Self-supervised sparse-to-dense: Self-supervised depth completion from lidar and monocular camera.  In ICRA, 2019.
  * [25] M. Magnusson, A. Lilienthal, and T. Duckett.  Scan registration for autonomous mining vehicles using 3d‐ndt.  Journal of Field Robotics, 24(10):803--827, 2007.
  * [26] R. Mur-Artal and J. Tardos.  ORB-SLAM2: An Open-Source SLAM System for Monocular, Stereo, and RGB-D Cameras.  IEEE Transactions on Robotics, 33(5):1255--1262, 2017.
  * [27] J. Stuckler N. Yang, R. Wang and D. Cremers.  Deep virtual stereo odometry: Leveraging deep depth prediction for monocular direct sparse odometry.  In ECCV, pages 817--833, 2018.
  * [28] R. Newcombe, S. Lovegrove, and A. Davison.  DTAM: Dense tracking and mapping in real-time.  In ICCV, pages 2320--2327, 2011.
  * [29] D. Nister, O. Naroditsky, and J. Bergen.  Visual odometry.  In CVPR, 2004.
  * [30] Y. Liu P. Chen, A. Liu and Y. Wang.  Towards scene understanding: Unsupervised monocular depth estimation with semantic-aware representation.  In CVPR, 2019.
  * [31] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, Z. Killeen, T.and Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In NeuroIPS, 2019.
  * [32] J. Qiu, Z. Cui, Y. Zhang, X. Zhang, S. Liu, B. Zeng, and M. Pollefeys.  Deeplidar: Deep surface normal guided depth prediction for outdoor scene from sparse lidar data and single color image.  In CVPR, 2019.
  * [33] Z. Long R. Li, S. Wang and D. Gu.  Undeepvo: Monocular visual odometry through unsupervised deep learning.  In ICRA, 2018.
  * [34] D. Rozenberszki and A. L. Majdik.  Lol: Lidar-only odometry and localization in 3d point cloud maps.  In ICRA, 2020.
  * [35] J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers.  A benchmark for the evaluation of rgb-d slam systems.  In IROS, 2012.
  * [36] J. Tang, F. Tian, W. Feng, J. Li, and P. Tan.  Learning guided convolutional network for depth completion.  2019\.
  * [37] T.Shan and B.Englot.  Lego-loam: Lightweight and ground-optimized lidar odometry and mapping on variable terrain.  In IROS, 2018.
  * [38] S. Wang, R. Clark, H. Wen, and N. Trigoni.  Deepvo: Towards end-to-end visual odometry with deep recurrent convolutional neural networks.  In ICRA, 2017.
  * [39] S. Wang, R. Clark, H. Wen, and N. Trigoni.  End-to-end, sequence-to-sequence probabilistic visual odometry through deep neural networks.  International Journal of Robotics Research, 37(4-5):513--542, 2018\.
  * [40] Y. Wang, J. Zhang, M. Kan, S. Shan, and X. Chen.  Self-supervised equivariant attention mechanism for weakly supervised semantic segmentation.  In CVPR, 2020.
  * [41] A. Wong and S. Soatto.  Bilateral cyclic constraint and adaptive regularization for unsupervised monocular depth prediction.  In CVPR, 2019.
  * [42] F. Xue, Q. Wang, X. Wang, W. Dong, J. Wang, and H. Zha.  Guided feature selection for deep visual odometry.  In Asian Conference on Computer Vision, pages 293--308. Springer, 2018.
  * [43] F. Xue, X. Wang, S. Li, Q. Wang, J. Wang, and H. Zha.  Beyond tracking: Selecting memory and refining poses for deep visual odometry.  In CVPR, 2019.
  * [44] N. Yang, L. Stumberg, R. Wang, and D. Cremers.  D3vo: Deep depth, deep pose and deep uncertainty for monocular visual odometry.  In CVPR, 2020.
  * [45] Z. Yin and J. Shi.  Geonet: Unsupervised learning of dense depth, optical flow and camera pose.  In CVPR, 2018.
  * [46] J. Yue, W. Wen, J. Han, and L. Hsu.  Lidar data enrichment using deep learning based on high-resolution image: An approach to achieve high-performance lidar slam using low-cost lidar.  arXiv preprint arXiv:2008.03694, 2020.
  * [47] H. Sheikh Z. Wang, A. Bovik and E. Simoncelli.  Image quality assessment: From error visibility to structural similarity.  IEEE Transactions on Image Processing, 13(4):600--612, 2004.
  * [48] J. Zbontar and Y. LeCun.  Stereo matching by training a convolutional neural network to compare image patches.  Journal of Machine Learning Research, 17:1--32, 2016.
  * [49] H. Zhan, R. Garg, C. Weerasekera, K. Li, H. Agarwal, and I. Reid.  Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction.  In CVPR, 2018.
  * [50] J. Zhang and S. Singh.  Loam: Lidar odometry and mapping in real-time.  In RSS, 2014.
  * [51] J. Zhang and S. Singh.  Visual-lidar odometry and mapping: Low drift, robust, and fast.  In ICRA, 2015.
  * [52] T. Zhou, M. Brown, N. Snavely, and D. G. Lowe.  Unsupervised learning of depth and ego-motion from video.  In CVPR, 2017.
  * [53] A. Zhu, W. Liu, Z. Wang, V. Kumar, and K. Daniilidis.  Robustness meets deep learning: An end-to-end hybrid pipeline for unsupervised learning of egomotion.  arXiv preprint arXiv:1812.08351, 2018.
  * [54] Y. Zou, Z. Luo, and J. Huang.  Df-net: Unsupervised joint learning of depth and flow using cross-task consistency.  In ECCV, 2018.