# A note on a deterministic property to obtain the long run behavior of the
range of a stochastic process

Maher Boudabra111Mathematics department, King Fahd University for Petroleum
and Minerals, KSA Binghao Wu222School of Mathematics, Monash University,
Australia

###### Abstract

A Brownian motion with drift is simply a process $V^{\eta}_{t}$ of the form
$V^{\eta}_{t}=B_{t}+\eta t$ where $B_{t}$ is a standard Brownian motion and
$\eta>0$ 333The case $\eta<0$ is deducible by remarking
$V^{-\eta}(t)=-V^{\eta}(t)$. In [13], the authors considered the drifted
Brownian motion and studied the statistics of some related sequences defined
by certain stopping times. In particular, they provided the law of the range
$R_{t}(V^{\eta})$ of $V^{\eta}$ as well as its first range process
$\theta_{V^{\eta}}(a)$. In particular, they investigated the asymptotic
comportment of $R_{t}(V^{\eta})$ and $\theta_{V^{\eta}}(a)$. They proved that
if $V_{t}^{\eta}$ is a Brownian motion with a positive drift $\eta$ then its
range $R_{t}(V^{\eta})=\sup_{0\leq s\leq t}V_{t}^{\eta}-\inf_{0\leq s\leq
t}V_{t}^{\eta}$ is asymptotically equivalent to $\eta t$. In other words

$\frac{R_{t}(V^{\eta})}{t}\overset{a.e}{\underset{t\rightarrow\infty}{\longrightarrow}}\eta.$
(1)

In this paper, we show that (1) follows from a striking deterministic
property. More precisely, we show that the long run behavior of the range of a
deterministic continuous function is obtainable straightaway from that of the
function itself. Our result can be deemed as the continuous version of a
similar one appeared in [1].

##  1 Introduction

The range of a real valued process $X=(X_{t})_{t\in T}$ is defined to be the
"measure" (in a certain sense) of the set of values generated by this process
up to time $t$. We shall denote the range of the process $X$ by $R_{t}(X)$.
When $X$ is discrete, i.e $X=(X_{n})_{n\in\mathbb{N}}$, $R_{n}(X)$ simplifies
to the number of explored sites up to time $n$ by the process $X$, i.e
$R_{n}(X)=|\\{X_{0},...,X_{n}\\}|$. The range of discrete processes (random
walks, Markov chains etc…) has been investigated in several seminal works [4,
5, 2, 15, 16]. In particular, we cite Kesten-Spitzer-Whitman theorem [12]
which says that the average of the range of a nearest neighbor simple random
walk (referred to as its speed in [1]) converges to the probability of never
returning to the origin (the starting point more generally). That is, let
$(\xi_{n})_{n}$ be a sequence of i.i.d Redmacher random variables of parameter
$p$ and set $X_{n}=\xi_{1}+...+\xi_{n}$. Then

$\frac{R_{n}(X)}{n}\overset{a.e}{\underset{n\rightarrow\infty}{\longrightarrow}}\mathbb{P}(X\,\,\text{never
returns to the origin}).$ (2)

Note that $\mathbb{P}(X\,\,\text{never returns to the origin})$ is simply
$2p-1=|\mathbb{E}(X_{1})|$. In [1], the authors showed that (2) is a
deterministic property, and no requirement to involve randomness to get it. In
fact they showed the following result.

######  Theorem 1.

[1] if $x=(x_{n})_{n}$ is any deterministic sequence mimicking a nearest
neighbor random walk, i.e $|x_{n+1}-x_{n}|\leq 1$ for all $n$, then if

$\frac{x_{n}}{n}\underset{{\scriptscriptstyle
n\rightarrow+\infty}}{\longrightarrow}\ell$

then

$\frac{R_{n}(x)}{n}\underset{{\scriptscriptstyle
n\rightarrow+\infty}}{\longrightarrow}|\ell|.$

Theorem (1) shows that the asymptotic behavior of the range of any nearest
neighbor random walk follows immediately from the asymptotic behavior of the
random walk itself. This deterministic result covers also the case of random
walks in random environments [11]. An immediate consequence of theorem (1) is
that if $X$ is a random walk for which the Birkoff ergodic theorem [10]
applies then

$\frac{R_{n}(X)}{n}\underset{{\scriptscriptstyle
n\rightarrow+\infty}}{\longrightarrow}|\mathbb{E}(X_{1})|.$ (3)

In continuous time, the range of a process $X=(X_{t})_{t\geq 0}$ is defined
similarly. It is the Lebesgue measure of the interval $[\sup_{0\leq s\leq
t}X_{s}-\inf_{0\leq s\leq t}X_{s}]$. In other words

$R_{t}(X)=\sup_{0\leq s\leq t}X_{s}-\inf_{0\leq s\leq t}X_{s}.$ (4)

One should notice that the path of $R_{t}(X)$ is non-decreasing. Thus, we can
define its right continuous inverse

$\theta_{X}(a):=\inf\\{t\geq 0\mid R_{t}(X)>a\\}.$

Note that $R_{t}(X)$ and $\theta_{X}(a)$ share the same monotonicity. In
addition, the following two relations encodes the duality in between them:

  * •

$R_{t}(X)<a\Longleftrightarrow\theta_{X}(a)<t.$

  * •

$\theta_{X}(R_{t}(X))\geq t.$

A typical process to study its range in continuous time is obviously the
Brownian motion as well as its derivatives [13, 6, 14]. The density of the
range of a standard Brownian motion $B=B_{t}$ is computed in [4] for a fixed
time $t$ while the Laplace transform of $\theta_{B}(a)$ is given in [6, 14].
In [13], the authors considered the case of a positive drifted Brownian motion
$V_{t}^{\eta}:=B_{t}+\eta t$ where $\eta>0$. In section $4,$ they provided the
distribution functions of $R_{t}(V^{\eta})$ and $\theta_{V^{\eta}}(a)$. In
addition, they proved two limit theorems for $\theta_{V^{\eta}}(a)$ that can
be seen as a law of large numbers and a central limit theorem. What matters
for us in this work is their result about the long run behavior of the range.
They showed that the range of $V_{t}^{\eta}$ is asymptotically equivalent to
$\eta t$ (a.e), or equivalently

$\frac{R_{t}(V^{\eta})}{t}\overset{a.e}{\underset{t\rightarrow\infty}{\longrightarrow}}\eta.$

However, their proof is quite technical and it is based on the study of the
right continuous inverse process $\theta_{V^{\eta}}(a):=\inf\\{t\geq 0\mid
R_{t}(V^{\eta})>a\\}$, combined with the use of the subadditive ergodic
theorem. Their approach will be discussed in the sequel. What we have
discovered is that the comportment of the range of $V_{t}^{\eta}$ at infinity
does not require that whole machinery to obtain. In fact, it is a
deterministic property that follows directly from the behavior of the
underlying process. More generally, we show that the long run behavior of the
range of a function $f:[0,\infty)\rightarrow\mathbb{R}$ is deducible from the
long run behavior of the function itself. To this end, recall that the range
of $f$ is defined exactly as in (4), i.e

$R_{t}(f)=\sup_{0\leq s\leq t}f_{s}-\inf_{0\leq s\leq t}f_{s}.$

When $f$ is continuous then its range path is also continuous. Moreover, when
$f$ s monotonic then its range increases over the time. Finally note that

$R_{t}(f)=R_{t}(-f).$ (5)

##  2 Results and applications

Now, we are ready to state the main result of the paper which is inspired from
the work in [1] but has more consequences.

######  Theorem 2.

Let $f,\psi:[0,\infty)\rightarrow\mathbb{R}$ be two functions such that
$\psi(t)$ is positively increasing to infinity. If

$\frac{f(t)}{\psi(t)}\underset{t\rightarrow\infty}{\longrightarrow}\ell\not=0$
(6)

then

$\frac{R_{t}(f)}{\psi(t)}\underset{t\rightarrow\infty}{\longrightarrow}|\ell|.$

###### Proof.

Without loss of generality we may assume $\ell$ is positive thanks to (5). Let
$\varepsilon\in(0,\ell)$. The assumption (6) yields

$\psi(t)(\ell-\varepsilon)\leq f(t)\leq\psi(t)(\ell+\varepsilon)$ (7)

for $t$ greater than some positive $T$. As $\psi(t)$ is positive then
$\psi(t)(\ell-\varepsilon)\leq R_{t}(f)$. On the other hand, we have

$R_{t}(f)\leq R_{T}(f)+\psi(t)(\ell+\varepsilon)$ (8)

since $f(t)>0$ for $t>T$. By combining (7) and (8) we get

$-2\varepsilon\leq\frac{R_{t}(f)}{\psi(t)}-\ell\leq\frac{R_{T}(f)}{\psi(t)}+\varepsilon.$

Since $\frac{R_{T}(f)}{\psi(t)}$ will vanish at infinity, we can find
$T^{\sharp}\geq T$ such that

$-2\varepsilon\leq\frac{R_{t}(f)}{\psi(t)}-\ell\leq 2\varepsilon$

for all $t$ greater than $T^{\sharp}$ which completes the proof by taking into
account the case $\ell<0$. ∎

Note that theorem (2) covers also the case when $\ell$ is infinite. This is
done by a minor change in the previous proof. Using the same assumptions and
notations of theorem (2), the case when $\ell$ is zero is similar and it is
the subject of the next theorem.

######  Theorem 3.

If

$\frac{f(t)}{\psi(t)}\underset{t\rightarrow\infty}{\longrightarrow}0$ (9)

then

$\frac{R_{t}(f)}{\psi(t)}\underset{t\rightarrow\infty}{\longrightarrow}0.$

###### Proof.

The assumption (9) yields

$-\psi(t)\varepsilon\leq f(t)\leq\psi(t)\varepsilon$ (10)

for $t$ greater than some positive $T$. In particular we get

$R_{t}(f)\leq R_{T}(f)+2\psi(t)\varepsilon$ (11)

The inequality (11) yields

$\frac{R_{t}(f)}{\psi(t)}\leq\frac{R_{T}(f)}{\psi(t)}+2\varepsilon.$

Since $\frac{R_{T}(f)}{\psi(t)}$ will vanish at infinity, we can find
$T^{\dagger}\geq T$ such that

$0\leq\frac{R_{t}(f)}{\psi(t)}\leq 3\varepsilon$

for all $t$ greater than $T^{\dagger}$ which completes the proof. ∎

The long run behavior of the the range of the drifted Brownian motion
$V^{\eta}_{t}$ follows easily by looking at the behavior of $V^{\eta}_{t}$
itself followed by applying theorems (2) and (3).

######  Corollary 4.

If $V_{t}^{\eta}$ is a Brownian motion with a drift $\eta$ (possibly $\eta=0$)
then

$\frac{R_{t}(V^{\eta})}{t}\overset{a.e}{\underset{t\rightarrow\infty}{\longrightarrow}}|\eta|.$

###### Proof.

Recall that $V_{t}^{\eta}=B_{t}+\eta t$ where $B_{t}$ is a standard Brownian
motion. Then by the law of large numbers for Brownian motion (See [9] for
example), $B_{t}$ is almost surely negligible against $t$, that is
$\frac{B_{t}}{t}\overset{a.e}{\underset{t\rightarrow\infty}{\longrightarrow}}0$.
Thus
$\frac{V_{t}^{\eta}}{t}\overset{a.e}{\underset{t\rightarrow\infty}{\longrightarrow}}|\eta|$
. Therefore by applying theorems (2) and (3) we get

$\frac{R_{t}(V^{\eta})}{t}\overset{a.e}{\underset{t\rightarrow\infty}{\longrightarrow}}|\eta|\,\,\,\,\,\,$
(12)

∎

In particular we obtain the behavior of the range of $B_{t}$

$\frac{R_{t}(B)}{t}\overset{a.e}{\underset{t\rightarrow\infty}{\longrightarrow}}0$

which is not covered in [13]. The reason we think is because the authors
derived the limit (12) by proving fist the following limit of
$\theta_{V^{\eta}}(a)$

$\frac{\theta_{V^{\eta}}(a)}{a}\overset{a.e}{\underset{a\rightarrow\infty}{\longrightarrow}}\frac{1}{|\eta|}$

which itself requires computations in terms of $\frac{1}{\eta}$ (valid only
when $\eta\neq 0$). Then they deduced the asymptotic behavior of the range
$R_{t}(V^{\eta})$ leaning on the fact that $\theta_{V^{\eta}}(a)$ is the right
continuous process of $R_{t}(V^{\eta})$. Now by theorems (2) and (3), the
asymptotic comportment of the range of $V^{\eta}$ are obtained independently
of $\theta_{V^{\eta}}(x)$ and follows exclusively from the behavior of the
underlying process $V^{\eta}_{t}$.

Now we show that the behavior of $\theta_{V^{\eta}}(a)$ is obtainable from
that of $R_{t}(V^{\eta})$ and vice versa. The main tool to do so is the
following result relating the asymptotic behaviors of a non decreasing
function and its generalized inverse. For a survey about generalized inverses,
we refer the reader to [3].

######  Proposition 5.

Let $\xi:\mathbb{R}_{+}\rightarrow\mathbb{R}$ be a non decreasing function and
let $\xi^{\dagger}$ be its generalized inverse, i.e

$\xi^{\dagger}(y)=\inf\\{x\mid\xi(x)>y\\}.$ (13)

If

$\frac{\xi(x)}{x}\underset{x\rightarrow\infty}{\longrightarrow}\ell$

then

$\frac{\xi^{\dagger}(x)}{x}\underset{x\rightarrow\infty}{\longrightarrow}\frac{1}{\ell}.$

###### Proof.

The statement is straightforward when $\xi$ is continuously increasing since
in such a case $\xi^{\dagger}(y)=\xi^{-1}(y)$. Otherwise fix a positive
$\varepsilon$ (less than $\ell$ if $\ell>0$) and let $\tau$ be a large number
so that

$x(\ell-\varepsilon)\leq\xi(x)\leq x(\ell+\varepsilon)$

for $x\geq\tau$. It is not hard to see that for $x>\tau$, the curve of $\zeta$
is squeezed between $\frac{x}{\ell+\varepsilon}$ and
$\frac{x}{\ell-\varepsilon}$ 444Jumps (resp. flat parts) of $\xi$ becomes flat
parts (resp. jumps) for $\zeta$. In other words

$\displaystyle\frac{x}{(\ell+\varepsilon)}\leq\xi^{\dagger}(x)\leq\frac{x}{(\ell-\varepsilon)}$
if $\ell>0$ (14)
$\displaystyle-\frac{x}{\varepsilon}\leq\xi^{\dagger}(x)\leq\frac{x}{\varepsilon}$
if $\ell=0$

for $x$ larger than $\tau$. The result follows by considering the reciprocals
in (14). ∎

######  Remark 6.

Another common definition of the generalized inverse is
$\xi^{*}(y):=\inf\\{x\mid\xi(x)\geq y\\}$. proposition (5) is still valid if
we use $\xi^{*}$ since $\xi^{*}=\xi^{\dagger}$ a.e 555It is because the set of
discontinuity of a non decreasing function is countable. The only difference
is that the formula (13) generates a right continuous function while $\xi^{*}$
generates a left continuous one. A very known usage of the left continuous
inverse is when $\xi$ represents the cumulative distribution function of some
distribution $\mu.$ In such a case, $\xi^{*}$ is used to simulate a random
variable of the same law $\mu$. More precisely

$\xi^{*}(\text{Uni}(0,1))\sim\mu.$

That is, the uniform distribution over $(0,1)$ is mapped to a random variable
of distribution $\mu$. Note also that proposition (5) works in both directions
since $\left(\xi^{\dagger}\right)^{\dagger}=\xi$ a.e. In particular, $\ell$
could be infinite.

A straightforward consequence of proposition (5) is the equivalence between
the comportment of $\theta_{V^{\eta}}(x)$ and $R_{t}(V^{\eta})$. This appeared
only as implication in [13].

######  Proposition 7.

We have

$\frac{\theta_{V^{\eta}}(x)}{x}\overset{a.e}{\underset{x\rightarrow\infty}{\longrightarrow}}\frac{1}{\eta}.$

In particular

$\frac{\theta_{B}(x)}{x}\overset{a.e}{\underset{x\rightarrow\infty}{\longrightarrow}}+\infty.$

The following diagram illustrates the difference between our approach and that
in [13].

|
---|---
Behavior of $V_{t}^{\eta}$Behavior of $R_{t}(V^{\eta})$Behavior of
$\theta_{V^{\eta}}(x)$ Figure 1: The continuous bent arrows illustrate our
approach while the dashed ones illustrate that in [13].

Another application of proposition (5) is the recovery of the limit theorem
for renewal processes (We refer the reader to [8] for more details). That is,
let $N_{t}$ be a renewal counting process associated to some renewal sequence
$T_{n}$ defined by

$N_{t}=\sup\\{n\mid T_{n}\leq t\\}.$ (15)

Note that the RHS of (15) is simply
$\sum_{n=1}^{\infty}\mathbf{1}_{\\{T_{n}\leq t\\}}$. The process $N_{t}$ is
the generalized inverse of the renewal sequence $T_{n}$ (the right continuous
one). More precisely, $T_{n}$ is obtainable from $N_{t}$ by

$T_{n}=\inf\\{t\mid N_{t}\geq n\\}.$

That is, we recover the following equivalence

$\frac{T_{n}}{n}\overset{a.e}{\underset{n\rightarrow\infty}{\longrightarrow}}\gamma\Longleftrightarrow\frac{N_{t}}{t}\overset{a.e}{\underset{t\rightarrow\infty}{\longrightarrow}}\frac{1}{\gamma}$

valid even in the case when the sequence of inter-arrivals $T_{n}-T_{n-1}$ is
ergodic. Both theorems (2) and (3) can be seen as the continuous version of
the main result stated in [1]. However, the discrete version can follow easily
from the continuous case. That is, let $\left(x_{n}\right)$ be a sequence of
integers mimicking a simple walk on $\mathbb{Z}$ and let $r_{n}$ be the range
of $\left(x_{n}\right)$, i.e the number of explored sites up to time $n$ by
$\left(x_{n}\right)$. Let $x_{t}^{\star}$ be the function obtained from
$\left(x_{n}\right)$ by connecting the dots. It is clear that

$|R_{t}(x^{\star})-r_{\lfloor t\rfloor}|\leq 2.$

Hence, in case of existence of limits, we obtain

$\lim_{n\rightarrow\infty}\frac{r_{n}}{n}=\lim_{\lfloor
t\rfloor\rightarrow\infty}\frac{r_{\lfloor t\rfloor}}{\lfloor
t\rfloor}=\lim_{t\rightarrow\infty}\frac{R_{t}(x^{\star})}{t}.$

A typical process derived from Brownian motion is the so called Bessel
process. Bessel processes are important as they interfere in financial
modeling of stock market prices etc. A $n$-dimensional Bessel process, denoted
by $BES(n)$, is defined by

$BES(n)=\|\boldsymbol{B}_{t}\|$

where $\|\cdot\|$ is the Euclidean norm and $\mathbf{B}_{t}$ is a
$n$-dimensional Brownian motion 666People often add the starting point as a
subscript.. Another alternative definition of Bessel processes uses SDE
theory. More precisely, a Bessel process is the solution of the following SDE

$dX_{t}=dB_{t}+\frac{n-1}{2}\frac{dt}{X_{t}}.$

We refer the reader to [7] for more details about the subject. The
determination of the statistic of the range of a Bessel process is not an easy
task, not to mention its long run behavior. Now, theorems (2) and (3) allows
us to obtain the asymptotic behavior of such a range by knowing only the
behavior of the process itself. However, we provide a result that covers more
than the case of Bessel processes.

######  Proposition 8.

Let
$\boldsymbol{B}_{t}=(\boldsymbol{B}_{t}^{(1)},\boldsymbol{B}_{t}^{(2)},...,\boldsymbol{B}_{t}^{(n)})$
be an $n$-dimensional Brownian motion and let $0<p\leq+\infty$. Then

$\frac{R_{t}(\|\boldsymbol{B}\|_{p})}{t}\overset{a.e}{\underset{t\rightarrow\infty}{\longrightarrow}}0\,\,\,\,\,\,(a.e)$

where $\|\boldsymbol{B}\|_{p}$ is the $p^{th}$ norm 777 It is not a true norm
when $p<1$. We use the word by abuse of of terminology. of
$\boldsymbol{B}_{t}$, i.e

$\|\boldsymbol{B}\|_{p}:=\begin{cases}\left(|B_{t}^{(1)}|^{p}+|B_{t}^{(2)}|^{p}+...+|B_{t}^{(n)}|^{p}\right)^{\frac{1}{p}}&p<+\infty\\\
\max_{1\leq i\leq n}|\boldsymbol{B}_{t}^{(i)}|&p=+\infty.\end{cases}$

###### Proof.

Let $p<+\infty$. The law of large number implies that

$\frac{|B_{t}^{(j)}|^{p}}{t^{p}}\overset{a.e}{\underset{t\rightarrow\infty}{\longrightarrow}}0$

for $j=1,...,n$. It follows that

$\begin{aligned}
\frac{\|\boldsymbol{B}\|_{p}}{t}&=\left(\frac{|B_{t}^{(1)}|^{p}}{t^{p}}+\frac{|B_{t}^{(2)}|^{p}}{t^{p}}+...+\frac{|B_{t}^{(n)}|^{p}}{t^{p}}\right)^{\frac{1}{p}}\\\
&\overset{a.e}{\underset{t\rightarrow\infty}{\longrightarrow}}0\end{aligned}.$

For $p=+\infty$ we have

$\begin{aligned}
\frac{\|\boldsymbol{B}\|_{p}}{t}&\leq\left(\frac{|B_{t}^{(1)}|}{t}+\frac{|B_{t}^{(2)}|}{t}+...+\frac{|B_{t}^{(n)}|}{t}\right)\\\
&\overset{a.e}{\underset{t\rightarrow\infty}{\longrightarrow}}0\end{aligned}.$

Therefore, in both cases

$\frac{R_{t}(\|\boldsymbol{B}\|_{p})}{t}\overset{a.e}{\underset{t\rightarrow\infty}{\longrightarrow}}0.$

∎

######  Corollary 9.

The range of a Bessel process is negligible against the time.

The following result illustrates the relation between a deterministic function
$f$ and its underlying supremum.

######  Proposition 10.

Let $f,\psi:[0,\infty)\rightarrow\mathbb{R}$ be two functions such that
$\psi(t)$ is positively increasing to infinity and let $\ell>0$. If

$\frac{f(t)}{\psi(t)}\underset{t\rightarrow\infty}{\longrightarrow}\ell$ (16)

then

$\frac{\sup_{0\leq s\leq
t}f_{s}}{f(t)}\underset{t\rightarrow\infty}{\longrightarrow}1.$

###### Proof.

From the assumption, we know for every $\varepsilon>0$ there exists a large
enough $T$ such that, when $t>T$

$\psi(t)(\ell-\varepsilon)\leq f(t)\leq\psi(t)(\ell+\varepsilon).$ (17)

Also,

$1=\frac{f(t)}{f(t)}\leq\frac{\sup_{0\leq s\leq t}f_{s}}{f(t)}$ (18)

for $t>T$. On the other hand, we have

$\frac{\sup_{0\leq s\leq t}f_{s}}{f(t)}\leq\frac{\sup_{0\leq s\leq
T}f_{s}+\sup_{T\leq s\leq t}f_{s}}{f(t)}\leq\frac{\sup_{0\leq s\leq
T}f_{s}+\psi(t)(\ell+\varepsilon)}{f(t)}$ (19)

Letting $t$ go to infinity, $\frac{\sup_{0\leq s\leq T}f_{s}}{f(t)}$ will
converge to $0$ as $f(t)$ diverges to $+\infty$.
$\frac{\psi(t)(\ell+\varepsilon)}{f(t)}$ will converge to
$1+\frac{\varepsilon}{\ell}$. Therefore,

$1\leq\liminf_{t\to\infty}\frac{\sup_{0\leq s\leq
t}f_{s}}{f(t)}\leq\limsup_{t\to\infty}\frac{\sup_{0\leq s\leq
t}f_{s}}{f(t)}\leq 1+\frac{\varepsilon}{\ell}$ (20)

The inequality (19) holds for any $\varepsilon>0$. Thus,

$\frac{\sup_{0\leq s\leq
t}f_{s}}{f(t)}\underset{t\rightarrow\infty}{\longrightarrow}1.$ (21)

∎

######  Corollary 11.

Under the same assumptions of Proposition (10), we have

$\frac{\sup_{0\leq s\leq
t}f_{s}}{\psi(t)}\underset{t\rightarrow\infty}{\longrightarrow}\ell$

and

$\lim_{t\rightarrow+\infty}\frac{R_{t}(f)}{\psi(t)}=\lim_{t\rightarrow+\infty}\frac{R_{t}(\sup
f)}{\psi(t)}=\ell.$

In other words, the four functions $f,\sup_{0\leq s\leq
t}f_{s},R_{t}(f),R_{t}(\sup f)$ have the same long run behavior under the
assumptions of proposition (10). Note that proposition (10) is still valid
when $\ell=0$ provided that $f$ becomes eventually positive after certain
time. The following result is a consequence of proposition (10). It concerns
the underlying supremum process of a drifted Brownian motion.

######  Proposition 12.

Let $\widetilde{V}_{t}^{\eta}=\sup_{0\leq s\leq t}V_{s}^{\eta}$. Then

$\frac{\widetilde{V}_{t}^{\eta}}{t}\overset{a.e}{\underset{t\rightarrow\infty}{\longrightarrow}}\eta$

and

$\frac{R_{t}(\widetilde{V}^{\eta})}{t}\overset{a.e}{\underset{t\rightarrow\infty}{\longrightarrow}}\eta.$

A corollary of proposition (12) is that the supremum of a Brownian motion, as
well as its range are negligible against the time evolution. However, it would
be interesting to derive estimates for the range like the law of iterated
logarithm for Brownian motion, and on top of that investigate whether such
potential properties are deterministic or not.

### Acknowledgments

The authors would like to thank Greg Markowsky for helpful communications.

## References

  * [1] M. Boudabra and G. Markowsky.  Remarks on the speeds of a class of random walks on the integers.  Discrete Mathematics, 344(8):112436, 2021.
  * [2] K. S. Chong, R. Cowan, and L. Holst.  The ruin problem and cover times of asymmetric random walks and brownian motions.  Advances in Applied Probability, 32(1):177–192, 2000.
  * [3] A. de La Fortelle.  A study on generalized inverses and increasing functions part i: generalized inverses.  2015\.
  * [4] W. Feller.  The asymptotic distribution of the range of sums of independent random variables.  The annals of mathematical statistics, pages 427–432, 1951.
  * [5] P. W. Glynn.  On the range of a regenerative sequence.  Stochastic Processes and Their Applications, 20(1):105–113, 1985\.
  * [6] J. P. Imhof.  On the range of brownian motion and its inverse process.  The Annals of Probability, 13(3):1011–1017, 1985.
  * [7] G. F. Lawler.  Notes on the bessel process.  Lecture notes. Available on the webpage of the author, pages 523–531, 2018.
  * [8] K. V. Mitov and E. Omey.  Renewal processes.  In Renewal Processes, pages 1–51. Springer, 2014.
  * [9] P. Mörters and Y. Peres.  Brownian Motion, volume 30.  Cambridge University Press, 2010.
  * [10] A. N Shiryaev.  Probability-1, volume 95.  Springer, 2016.
  * [11] F. Solomon.  Random walks in a random environment.  The Annals of Probability, 3(1):1–31, 1975.
  * [12] F. Spitzer.  Principles of random walk, volume 34.  Springer Science & Business Media, 2001.
  * [13] E. Tanré and P. Vallois.  Range of brownian motion with drift.  Journal of Theoretical Probability, 19(1):45–69, 2006.
  * [14] P. Vallois.  Decomposing the brownian path via the range process.  Stochastic Processes and Their Applications, 55(2):211–226, 1995\.
  * [15] P. Vallois.  The range of a simple random walk on $\mathbb{Z}$.  Advances in Applied Probability, 28(4):1014–1033, 1996.
  * [16] P. Vallois and Charles S Tapiero.  Range reliability in random walks.  Mathematical Methods of Operations Research, 45:325–345, 1997.


# Representation Disparity-aware Distillation for 3D Object Detection

Yanjing Li1†, Sheng Xu1†, Mingbao Lin3, Jihao Yin1, Baochang Zhang1,2,4,
Xianbin Cao1∗
1 Beihang University 2 Zhongguancun Laboratory 3 Tencent
4 Nanchang Institute of Technology

###### Abstract

In this paper, we focus on developing knowledge distillation (KD) for compact
3D detectors. We observe that off-the-shelf KD methods manifest their efficacy
only when the teacher model and student counterpart share similar intermediate
feature representations. This might explain why they are less effective in
building extreme-compact 3D detectors where significant representation
disparity arises due primarily to the intrinsic sparsity and irregularity in
3D point clouds. This paper presents a novel representation disparity-aware
distillation (RDD) method to address the representation disparity issue and
reduce performance gap between compact students and over-parameterized
teachers. This is accomplished by building our RDD from an innovative
perspective of information bottleneck (IB), which can effectively minimize the
disparity of proposal region pairs from student and teacher in features and
logits. Extensive experiments are performed to demonstrate the superiority of
our RDD over existing KD methods. For example, our RDD increases mAP of CP-
Voxel-S to 57.1% on nuScenes dataset, which even surpasses teacher performance
while taking up only 42% FLOPs.

††footnotetext: † Equal contribution.††footnotetext: ∗ Corresponding author:
<EMAIL_ADDRESS>1 Code: https://github.com/YanjingLi0202/RDD

##  1 Introduction

3D object detection in point clouds [25, 39, 29] is a fundamental perception
task with broad applications on autonomous driving, robotics and smart city,
_etc_. Beneficial from the large-scale 3D perception datasets [8, 2, 32] as
well as advanced point [25], pillar [18, 37] and voxel based [9, 21]
representations of sparse and irregular LiDAR point cloud scenes, 3D detection
has achieved remarkable progress [28, 43]. Unfortunately, stronger performance
is often accompanied with heavier computation burden, therefore the adoption
in real-world applications still remains a challenging problem.

Recent attempts to improve efficiency focus on developing specified
architectures for point-based 3D object detectors [5, 45], not generalizable
to a wide spectrum of pillar/voxel-based methods [46, 18, 28, 43, 7, 39].
Here, we aim at a model-agnostic framework for obtaining efficient and
accurate 3D object detectors with knowledge distillation (KD). Due to its
effectiveness, generality and simplicity, KD has become a popular strategy to
develop efficient models in a variety of 2D tasks [12, 21, 6, 13], which
improves the performance of a lightweight student model by harvesting
knowledge from an accurate yet computationally heavy teacher model.

Figure 1: Visualization of intermediate neck features from teacher CP-Voxel
[43] and student CP-Voxel-XXS [40]. Student† denotes CP-Voxel-XXS distilled by
[40]. “F-P” denotes false positive predictions from the detector. The second
and fourth column images show false positives on the original inputs where the
red boxes denote false positives from the detector. Off-the-shelf
implementation fails to tackle false positives if significant disparity exists
between teacher (c) and student (g) feature maps.

The recent art [40] employs pivotal position logit KD to enhance the
performance of compact 3D detectors. However, as we analyze in this paper, the
intrinsic representation disparity, stemming from region distances between
compact students and pre-trained teachers, is crucial to 3D student detectors
and solely neglected in existing methods [40].

Figure 2: Histogram of mean square distance between feature maps of CP-Voxel
$\&$ CP-Voxel-XXS ( orange), and CP-Voxel $\&$ distilled CP-Voxel-XXS-PP (
blue) on KITTI [8] and nuScenes [2].

For an in-depth analysis, in Fig. 1, we visualize the feature maps and
predictions of pre-trained heavy CP-Voxel [43], a state-of-the-art 3D LiDAR-
based detector, and these of compact CP-Voxel-XXS and its distilled version by
[40]. The visual results indicate that off-the-shelf KD methods manifest their
efficacy only when the teacher model and student counterpart share similar
feature maps as in Fig. 1 $\&$ Fig. 1e. Otherwise, false positives are
overmuch if significant representation disparity arises as like Fig. 1c $\&$
Fig. 1g, which greatly deteriorate the performance of compact 3D detectors.
For a comprehensive verification, in Fig. 2, we calculate the mean square
distance between feature maps of teacher CP-Voxel and student CP-Voxel-XXS as
a metric to reflect if teacher knowledge can be well transferred to student.
It is intuitive that a large distance indicates a higher representation
disparity. We perform upon two datasets including KITTI [8] and nuScenes [2].
Fig. 2 manifests the statistics where the Fig. 2 $\&$ Fig. 2b represents
distance histogram between teacher and vanilla student while the Fig. 2c $\&$
Fig. 2d represents distance histogram between teacher and distilled student.
Each histogram can be separated into 1) the first one in compliance with small
distance ( blue area) and 2) the second one in line with large distance (
orange area). We observe that the small-distance one is further reduced after
distilling, which indicates an efficient distillation. While the large-
distance one almost remains unchangeable, which indicates an inefficient
distillation. Therefore, it remains an open issue to tackle the representation
disparity in existing methods.

Therefore, in this paper we propose a novel 3D detector oriented
representation disparity-aware distillation (RDD) method to address the above
issue and reduce performance gap between compact students and over-
parameterized teachers. Framework of our RDD is illustrated in Fig. 3, where
the distillation objective is actually formulated under the principle of
information bottleneck (IB) to maximize the mutual information between
intermediate features of teacher and students. To this end, for each region
proposal in teacher (student) model, our RDD first pair it by cropping a
counterpart region in the same location of student (teacher) model. We measure
representation disparity in each pair with mutual information under the IB
framework and then learn to weight the region pairs to better bilaterally
transfer information between teacher and student. In contrast to off-the-shelf
pivotal position logit KD or simply involving ground truths [36, 40], the
weighted information is transferred by a feature-level representation
disparity-aware distillation as well as logit-level representation disparity-
aware distillation loss.

We compare our RDD against state-of-the-art 2D and 3D KD methods [6, 34, 24,
40, 44] on datasets of KITTI [8] and large-scale nuScenes [2]. Extensive
results reveal that our method outperforms the others by a considerable
margin. For instance, on nuScenes, the CP-Voxel-S [40] distilled by our RDD
obtains 57.1% mAP with only 42% FLOPs of CP-Voxel [43], achieving a new state-
of-the-art.

Figure 3: Overview of the proposed representation disparity-aware distillation
(RDD) framework. We first select representative region pairs based on the
representation disparity and formulate a balanced mask for each region pair.
Then we distill student in both feature-level and logit-level to effectively
eliminate the representation disparity.

##  2 Related work

3D LiDAR-based Object Detection targets to localize and classify 3D objects
from point clouds. Point-based methods [29, 5, 42, 45] take raw point clouds
and leverage PointNet++ [25] to extract sparse point features and generate
point-wise 3D proposals. Pillar-based works [18, 37] finish voxelization in
bird eye’s view and extract pillar-wise features with PointNet++. Voxel-based
methods [46, 39, 28] voxelize point clouds and obtain voxel-wise features with
3D sparse convolutional networks, which has become one of the most popular
data treatment. Besides, range-based works [1, 33] were proposed for long-
range and fast detection. Recently, designing efficient 3D detectors has drawn
some attentions [5, 45] with raw point data treatment. In this work, we focus
on exploring model-agnostic knowledge distillation methods to boost the
performance of lightweight 3D detectors.

Knowledge Distillation aims to transfer knowledge from a large teacher model
to a lightweight student network, which has become a thriving area in
efficient deep learning. The simple-yet-common used KD method [12] distills
knowledge between teacher and student on the output prediction logits. Another
line of research proposed to help student’s optimization with hints stored in
informative intermediate features from teacher [27, 14, 17, 11, 16, 3]. In 2D
detection task, some works attempt distillation techniques [19, 36, 6, 26, 41,
24] by emphasizing instance-wise distillation and feature knowledge. Mimic
[19], FG [36] and GID [6] sample local region features with box proposals or
custom indicators for foreground-aware feature imitation. Label KD [24]
utilizes teacher’s information for label assignment of student. Moreover,
recent methods have also been proposed to distill the 3D LiDAR-based
detectors. PointDistiller [44] captures and makes usage of the semantic
information in the local geometric structure of point clouds for compressing
the student. [40] leverages cues in teacher prediction to determine the
important areas for distillation. Nevertheless, existing 3D LiDAR-based KD
only leverage the information from well-trained teachers but neglect the
distillation-needed areas in the students. In the contrary, in this work, we
propose an enhanced 3D detection KD method which takes into consideration
representation disparity and effectively transfer the comprehensive
information from well-trained teachers to compact students.

##  3 Representation Disparity-aware Distillation

Fig. 3 illustrates the framework of our RDD for 3D object detection. To
complete our landscape, in Sec. 3.1, we first model our distillation objective
under the principle of information bottleneck (IB) [30, 38, 35]. Then, Sec.
3.2 depicts the generation of region proposal pairs between teacher and
student, and representation disparity in each pair. Lastly, we detail our
distillation losses including feature-level representation disparity-aware
distillation and logit-level representation disparity-aware distillation.

###  3.1 Knowledge Distillation Objective

Usually, knowledge distillation involves a to-be-trained student detector and
a pre-trained teacher detector, and we distinguish them with scripts
$\mathcal{S}$ and $\mathcal{T}$, respectively. We start with a novel
perspective of information bottleneck (IB) principle [30] to explore KD in 3D
detectors. As discussed in [38], IB is commensurate with the best compression
hypothesis that advocates minimizing data misfitting and model complexity in
concert such that the task-irrelevant information can be well diminished in
the compressed model for a better performance. Considering the facts in 3D
object detection that the point clouds are overwhelmingly sparse and the
extreme imbalance keeps going between informative instances and redundant
background, an efficient information extraction is therefore particularly
important.

Given a set of point clouds $X$, KD objective in the IB principle is written
as:

$\displaystyle\mathop{\min}_{\theta_{\bm{B}}^{\mathcal{S}},\theta_{\bm{D}}^{\mathcal{S}}}[I(X;f^{\mathcal{S}})-\delta
I(f^{\mathcal{S}};y^{GT})]-\beta I(f^{\mathcal{S}};f^{\mathcal{T}}),$ (1)

where $f^{\mathcal{T}}$ and $f^{\mathcal{S}}$ are the high-level feature maps
from the neck/backbone of the teacher and the student, respectively. $y^{GT}$
denotes ground-truth. $\theta_{\bm{B}}^{\mathcal{S}}$ and
$\theta_{\bm{D}}^{\mathcal{S}}$ are the parameters of backbone and detection
part in student respectively. Meanwhile, $\delta,\beta$ are Lagrange
multipliers [30]. $I()$ returns the mutual information between its two input
variables. With $I(f^{\mathcal{S}};y^{GT})$ maximizing the mutual information
between the features and ground-truth, the first item $I(X;f^{\mathcal{S}})$
minimizes the mutual information between the point cloud data and the high-
level feature maps of student to control the noise introduction. This part can
be treated as the original detection loss of the detector [43, 18]. With
teacher model’s features as guidance, the second item $\beta
I(f^{\mathcal{S}};f^{\mathcal{T}})$ maximizes the mutual information to
preserve more teacher information in student. The collective cooperation
between the two items guides student to focus more on beneficial information
and less on noise information [35, 38, 30].

###  3.2 Region Pairs with Representation Disparity

Region Pairs. We denote
$\\{R_{i}^{\mathcal{T}}|(p_{reg,i}^{\mathcal{T}},p_{cls,i}^{\mathcal{T}})\\}_{i=1}^{M}$
and
$\\{R_{i}^{\mathcal{S}}|(p_{reg,i}^{\mathcal{S}},p_{cls,i}^{\mathcal{S}})\\}_{i=M+1}^{M+N}$
as the outputs of teacher and student where the $i$-th region proposal
$R^{\mathcal{T}}_{i}/R^{\mathcal{S}}_{i}\in\mathbb{R}^{C\times H\times W}$
contains two information including a regression coordinate $p_{reg,i}$ to
model the proposal position and a classification probability $p_{cls,i}$ to
tell the proposal category. Note that in the center-based 3D detectors, e.g.,
CenterPoints [43], each proposal corresponds to a Gaussian area in the
heatmaps, while in the anchor-based 3D detectors, e.g., PointPillars [18],
each proposal corresponds to a region in the intermediate features.

As shown in the left of Fig. 4, the recent study [40] unilaterally passes on
teacher proposals of higher classification probability to the corresponding
regions of student. On the one hand, it ignores the efficacy of student
information; on the other hand, the representation disparity is neglected as
discussed in Sec. 1. In this paper, we propose to bilaterally transfer
information between teacher and student. Specifically, as depicted in the
right of Fig. 4, for each single region in teacher (student) model, we crops a
counterpart feature map patch in the same location of student (teacher) model
to form a total of $M+N$ region pairs
$\\{(R^{\mathcal{T}}_{i},R^{\mathcal{S}}_{i})\\}_{i=1}^{M+N}$. Here,
$\\{R^{\mathcal{S}}_{i}\\}_{i=1}^{M}$ accords with cropped region patches in
student and $\\{R^{\mathcal{T}}_{i}\\}_{i=M+1}^{M+N}$ is in tune with these
patches in teacher. For ease of representation, the superscripts $\mathcal{T}$
and $\mathcal{S}$ will be dropped from time to time in the following contents.

Then, our distillation considers representation disparity issue by weighting
the patch-level distance under the IB principle. Before diving into details,
we first channel-wise normalize proposal $R_{i}$ considering the large-scale
magnitude gap between the pre-trained teacher and the to-be-trained student,
as:

$\displaystyle\hat{R}_{i;c,:,:}=\frac{\operatorname{exp}(\frac{R_{i;c,:,:}}{\tau})}{\sum_{c^{\prime}\in\\{1,2,...,C\\}}\operatorname{exp}(\frac{R_{i;c^{\prime},:,:}}{\tau})},$
(2)

where $R_{i;c,:,:}$ denotes the $c$-th channel of $R_{i}$ and $\tau=4$ in this
paper denotes a hyper-parameter controlling the statistical attributions of
the channel-wise alignment operation

Representation Disparity. Under the framework of IB principle, we define and
evaluate representation disparity as mutual information between student patch
$\hat{R}^{\mathcal{S}}_{i}$ and teacher patch $\hat{R}^{\mathcal{T}}_{i}$.
This is formulated as:

$\begin{split}I(\hat{R}^{\mathcal{S}}_{i};\hat{R}^{\mathcal{T}}_{i})=H(\hat{R}^{\mathcal{S}}_{i})-H(\hat{R}^{\mathcal{S}}_{i}|\hat{R}^{\mathcal{T}}_{i}),\end{split}$
(3)

where $H()$ returns the information entropy. A smaller
$I(\hat{R}^{\mathcal{S}}_{i};\hat{R}^{\mathcal{T}}_{i})$ indicates higher
disparity between $\hat{R}^{\mathcal{S}}_{i}$ and $\hat{R}^{\mathcal{T}}_{i}$.

Figure 4: Illustration for the generation of the region pairs. Each single
region in one model generates a counterpart feature map patch in the same
location of the other model.

Then, a naive way to measure mutual information between teacher and student
$I(f^{\mathcal{S}};f^{\mathcal{T}})$ in the KD objective of Eq. (1) is to sum
up mutual information of all patch pairs as:

$\displaystyle
I(\hat{R}^{\mathcal{S}}_{i};\hat{R}^{\mathcal{T}}_{i})=\sum_{i=1}^{M+N}m_{i}I(\hat{R}^{\mathcal{S}}_{i};\hat{R}^{\mathcal{T}}_{i}).$
(4)

Nevertheless, it does not take into account the representation disparity among
different region pairs. Given this, prior to a formal distillation, we propose
to optimize a weighting vector
${\bm{m}}=[m_{1},m_{2},...,m_{M+N}]\in\mathbb{R}^{M+N}$ to identify disparate
region pairs, leading to our learning objective as:

$\displaystyle\mathop{\min}_{{\bf
m}}\underbrace{\sum_{i=1}^{M+N}m_{i}I(\hat{R}^{\mathcal{S}}_{i};\hat{R}^{\mathcal{T}}_{i})}_{I(f^{\mathcal{S}};f^{\mathcal{T}}|{\bm{m}})}+\lambda\|{\bf
m}\|_{1},$ (5)

where the minimization leads $m_{i}\in{\bm{m}}$ to be large to penalize high
disparity stemming from pair $\hat{R}^{\mathcal{S}}_{i}$ and
$\hat{R}^{\mathcal{T}}_{i}$. The term $\mathop{\min}_{{\bf m}}\|{\bf m}\|_{1}$
is involved to prevent the model to equally distill all alternative region
pairs. Also, to indicate the disparity degree. we clip $m_{i}$ to $1$ or $0$
if its value is beyond $[0,1]$. $\lambda>0$ is a hyper-parameter to determine
the sparsity of ${\bm{m}}$.

Moreover, the introduction of ${\bm{m}}$ leads our format of
$I(f^{\mathcal{S}};f^{\mathcal{T}})$ to
$I(f^{\mathcal{S}};f^{\mathcal{T}}|{\bm{m}})$ as manifested in Eq. (5).
Together with our representation disparity, the KD objective under IB
framework finally changes from Eq. (1) to:

$\displaystyle\mathop{\min}_{\theta_{\bm{B}}^{\mathcal{S}},\theta_{\bm{D}}^{\mathcal{S}}}$
$\displaystyle[I(X;f^{\mathcal{S}})-\delta I(f^{\mathcal{S}};y^{GT})]-\beta
I(f^{\mathcal{S}};f^{\mathcal{T}}|{\bm{m}}^{*}),$ (6)
$\displaystyle\operatorname{s.t.}\;{\bf m}^{*}=\mathop{\arg\min}_{\bf
m}\;I(f^{\mathcal{S}};f^{\mathcal{T}}|{\bm{m}})+\lambda\|{\bm{m}}\|_{1}.$

Our objective involve two sub-problems. In each training iteration, we first
perform inner-level optimization to derive a current optimal ${\bm{m}}^{*}$;
and then solves the upper-level optimization to conduct distillation based on
explicit distillation losses in Sec. 3.3. Notice the inner-level optimization
causes negligible costs compared to the upper-level one since the size of
region pairs is not large. On the contrary, it derives distillation to focus
more on disparate region pairs for a better performance.

###  3.3 Knowledge Transferring

We present the upper-level optimization in Eq. (6). Recall in Sec. 3.1 we
analyze that the first item $[I(X;f^{\mathcal{S}})-\delta
I(f^{\mathcal{S}};y^{GT})]$ is in compliance with the original detection loss
such as proposal classification and coordinate regression. Our central is to
hand over the specific format of the second term
$I(f^{\mathcal{S}};f^{\mathcal{T}}|{\bm{m}}^{*})$, which according to Eq. (3)
and Eq. (5) can be explicitly derived as:

$I(f^{\mathcal{S}};f^{\mathcal{T}}|{\bm{m}}^{*})=\sum_{i=1}^{M+N}m^{*}_{i}(H(\hat{R}^{\mathcal{S}}_{i})-H(\hat{R}^{\mathcal{S}}_{i}|\hat{R}^{\mathcal{T}}_{i})).$
(7)

Considering the intrinsic entropy of $\hat{R}^{\mathcal{S}}_{i}$ remains
unchanged within each iteration, therefore $H(\hat{R}^{\mathcal{S}}_{i})$ is
regarded as a constant. Maximizing
$I(f^{\mathcal{S}};f^{\mathcal{T}}|{\bm{m}}^{*})$ turns to minimizing
$H(\hat{R}^{\mathcal{S}}_{i}|\hat{R}^{\mathcal{T}}_{i})$. Unfortunately, it is
hard to directly minimize
$H(\hat{R}^{\mathcal{S}}_{i}|\hat{R}^{\mathcal{T}}_{i})$. Instead, we choose
to minimize norm distance between $\hat{R}^{\mathcal{S}}_{i}$ and
$\hat{R}^{\mathcal{T}}_{i}$ as a substitute since both of them reach optimal
when $\hat{R}^{\mathcal{S}}_{i}=\hat{R}^{\mathcal{T}}_{i}$.

In view of that feature pyramid network (FPN) [20] has been adopted in most 3D
detectors for robustness of multi-scale detection [43, 18, 39], it is natural
to choose the neck feature maps after FPN for distillation. After forming the
region pairs, the feature-level representation disparity-aware distillation
loss is computed as:

$\displaystyle\mathcal{L}_{feat}$
$\displaystyle=\frac{1}{M+N}\sum_{i=1}^{M+N}m_{i}^{*}\|\varphi(\psi(\hat{R}_{i}^{\mathcal{S}}))-\hat{R}_{i}^{\mathcal{T}}\|_{2},$
(8)

where $\varphi$ indicates the RoI Align [10]. $\psi$ is $1\times 1$
convolution followed by batch normalization [15] and ReLU [23] to align
channel-wise discrepancy between teacher region $\hat{R}_{i}^{\mathcal{T}}$
and and student region $\hat{R}_{i}^{\mathcal{S}}$ [6, 36].

In addition to FPN outputs, another neglected special kind of feature maps
come to the logits in the classification and regression branches. Previous 2D
and 3D methods [40, 36, 12] conduct distillation on the whole or pivotal part
of the detection head outputs or on the conventional feature maps, degrading
the student performance [6]. The probable cause can be attributed to the
extreme imbalance between instances and backgrounds in 3D LiDAR-based
detection task. Therefore, based on the weighting vector ${\bm{m}}$, we also
take into consideration the regression coordinate and classification
probability of each region proposal $\hat{R}_{i}$, and form our logit-level
representation disparity-aware distillation loss as:

$\displaystyle\mathcal{L}_{logit}=\frac{1}{M+N}\sum_{i=1}^{M+N}$
$\displaystyle
m_{i}^{*}(\|p_{cls,i}^{\mathcal{S}}-p_{cls,i}^{\mathcal{T}}\|_{1}$ (9)
$\displaystyle+\|(p_{reg,i}^{\mathcal{S}}-p_{reg,i}^{\mathcal{T}})\|_{1}),$

in which $p_{cls,i}$ and $p_{reg,i}$ denote the classification probability and
proposal coordinates for the $i$-th region proposal $R_{i}$ as introduced in
Sec. 3.2. It should be noted that, for the center-based 3D detector [43], the
regression loss item do not exist.

Finally, the exact definition of our KD objective in Eq. (1) is given as:

$\mathcal{L}=\underbrace{\mathcal{L}_{cls}+\gamma\mathcal{L}_{reg}}_{I(X;f^{\mathcal{S}})-\delta
I(f^{\mathcal{S}};y^{GT})}+\underbrace{\alpha_{1}\mathcal{L}_{feat}+\alpha_{2}\mathcal{L}_{logit}}_{-\beta
I(f^{\mathcal{S}};f^{\mathcal{T}})},$ (10)

where $\mathcal{L}_{cls}$ and $\mathcal{L}_{reg}$ are the original detection
losses supervised by the ground-truth labels. The $\gamma,\alpha_{1}$ and
$\alpha_{2}$ are trade-off parameters between different objectives.

##  4 Experiments

Comprehensive experiments are conducted to evaluate our proposed method on two
datasets for object detection: nuScenes [2] and KITTI [8]. First, we introduce
the datasets, metrics, implementation details and compact model architecture
in Sec. 4.1. Then we select the hyper-parameters, validate the effectiveness
of the components, and analyze the information of our method through ablation
studies in Sec. 4.2. Finally, in Sec. 4.3 and Sec. 4.4, we compare our method
with other image-based 2D distillation methods implemented on 3D detectors,
and other 3D distillation methods to demonstrate the superiority of RDD.

###  4.1 Experimental Settings

Datasets and Evaluation Metrics. For nuScenes dataset [2], metrics are mean
average precision (mAP) and the nuScenes detection score (NDS). These metrics
are computed in the physical unit. For KITTI dataset [8], we report the
average precision calculated by 40 sampling recall positions for 3D object
detection on the validation split. Following the typical protocol, the IoU
threshold is set as 0.7 for class Car and 0.5 for class Pedestrians and
Cyclists.

Training $\&$ Validation. For experiments conducted on nuScenes dataset, we
follow the same setups as the original CenterPoint [43]. We use AdamW [22] to
train the model. The weight decay for AdamW is $1e-2$. Following a cyclic
schedule [31], the learning rate is initially $1e-4$ and gradually increased
to $1e-3$, and finally decreased to $1e-8$. We train for 20 epochs on 8 NVIDIA
Tesla V100 GPUs. During inference, we take the top 100 highest-scored objects
as the final predictions. We do not use any post-processing such as non
maximum suppression (NMS). We use the toolkit provided with the nuScenes
dataset for evaluation.

For experiments conducted on KITTI dataset, we use AdamW [22] to train the
model. The weight decay for AdamW is $1e-2$. Following a cyclic schedule, the
learning rate is initially $1e-4$ and gradually increased to $1e-3$, which is
finally decreased to $1e-8$. We train for 80 epochs on 8 NVIDIA Tesla V100
GPUs. We apply axis aligned non maximum suppression (NMS) with an overlap
threshold of 0.5 IoU, following [18] for inference. We evaluate the model
performance using the toolkit provided with the KITTI dataset.

Compact Model Architecture. We apply CenterPoint-Voxel and CenterPoint-Pillar
[43] on nuScenes, and SECOND [39] and PointPillars [18] on KITTI.

Figure 5: Influence of $\lambda$, $\alpha_{1}$ and $\alpha_{1}$ using CP-Voxel-S [40] on nuScenes [2]. Table 1: Effects of different components in RDD with CP-Voxel-S and CP-Pillar-v0.4 [40] on nuScenes [2]. All experiments use $\ell_{2}$ loss for distillation. Teacher models are marked in grey. Model |  | Region
---
Selection
mAP | NDS
[HTML]EFEFEF CP-Voxel | - | 56.6 | 64.7
CP-Voxel-S | - | 54.0 | 62.1
Hint | 52.9 | 61.1
FG | 54.3 | 62.6
RDD | 57.1 | 65.0
[HTML]EFEFEF CP-Pillar | - | 49.1 | 59.7
CP-Pillar-v0.4 | - | 46.5 | 55.5
Hint | 45.6 | 55.1
FG | 47.6 | 57.2
RDD | 50.0 | 58.9

For CP-Voxel and CP-Pillar [43], we follow [40] to adopt width and input
resolution compression. Specifically, we compress the channels of
$\\{$encoder, backbone$\&$neck, head$\\}$ (i.e., $\\{$Pillar Feature Encoding
(PFE), Bird eye’s view Feature Encoding (BFE), Head$\\}$ in [40]) into
$\\{$1$\times$, 0.5$\times$, 0.5$\times\\}$, $\\{$0.75$\times$, 0.5$\times$,
0.5$\times\\}$ and $\\{$0.5$\times$, 0.25$\times$, 0.25$\times\\}$, forming
the CP-Voxel-S, CP-Voxel-XS and CP-Voxel-XXS. We follow [40] to change the
voxel size of CP-Pillar from 0.32 to 0.4, 0.48 and 0.64, forming the CP-
Pillar-v0.4, CP-Pillar-v0.48 and CP-Pillar-v0.64. For SECOND [39], we also
follow [40] to compress the width of SECOND [39], where the channels of
$\\{$encoder, backbone$\&$neck$\\}$ (i.e., $\\{$PFE, BFE$\\}$ in [40]) are
reduced to $\\{$0.75$\times$, 0.5$\times\\}$ and $\\{$0.5$\times$,
0.5$\times\\}$, forming the SECOND-S and SECOND-XS. For PointPillars [18], we
compress the width of PointPillars [18] forming the PointPillars-S and
PointPillars-XS, in which the channels of $\\{$encoder, backbone$\&$neck$\\}$
are reduced to $\\{$0.75$\times$, 0.5$\times\\}$ and $\\{$0.5$\times$,
0.5$\times\\}$.

Table 2: Evaluating the components of RDD based on CP-Voxel-S and CP-Pillar-v0.4. RDD-F and RDD-L denote w/ or w/o distillation loss in Eq. (8) and Eq. (9), respectively. Student | RDD-F | RDD-L | mAP | NDS
---|---|---|---|---
CP-Voxel-S | - | - | 54.0 | 62.1
$\checkmark$ |  | 56.8 | 64.0
| $\checkmark$ | 57.0 | 64.3
$\checkmark$ | $\checkmark$ | 57.1 | 65.0
CP-Pillar-v0.4 | - | - | 46.5 | 55.5
$\checkmark$ |  | 49.7 | 58.0
| $\checkmark$ | 49.2 | 57.9
$\checkmark$ | $\checkmark$ | 50.0 | 58.9

###  4.2 Ablation Study

Hyper-parameter Selection. We first select hyper-parameters $\lambda$ of Eq.
(5) and $\alpha_{1}$, $\alpha_{2}$ of Eq.(10) in this part, with experiments
conducted on nuScenes dataset. Note that $\gamma$ in Eq.(10) is sett following
CenterPoint [43], SECOND [39], and Pointpillar [18] when conducted our
expeirments on these detectors. We show the model performance (mAP) with
different setups of hyper-parameters $\lambda$ in Fig. 5, in which the
performances increase first and then decrease with the uplift of $\lambda$
from left to right. Since $\lambda$ controls the decline of $\|{\bf m}\|_{1}$
related to the proportion of selected distillation-desired region pairs,
$\lambda=0$ denotes all the alternative region pairs are equally distilled
leading to a unsatisfactory performance. While with $\lambda$ increasing to
more than 0.1, the proportion of selected distillation-desired region pairs is
not large enough for effectively eliminating the representation disparity,
which also affects the performance. In conclusion, the CP-Voxel-S [40] obtains
better performances with $\lambda$ set as 0.1. The figure also shows that the
CP-Voxel-S [40] obtains best performances with $\alpha_{1}$ and $\alpha_{2}$
set as 0.2 and 0.2. However, enlarging $\alpha_{1}$ and $\alpha_{2}$
degenerates the performance of the detectors. Based on the ablative study
above, we set hyper-parameters $\\{\lambda,\alpha_{1},\alpha_{2}\\}$ as
$\\{$0.1, 0.2, 0.2$\\}$ for the experiments in this paper.

Component Ablation. We first compare our representation disparity-aware (RDD)
region selecting method with other methods to select regions: Hint [4] (using
the neck feature without region mask) and FG [36]. We show the effectiveness
of RDD with center-based 3D detectores [43] on nuScenes dataset [2] in Tab. 1.
On the CP-Voxel-S [40], the introducing of RDD achieves improvements of the
mAP and NDS by $\\{$ 3.1%, 4.2%, 2.8%$\\}$ and $\\{$2.9%, 3.9%, 2.4%$\\}$
compared to non-distillation, Hint [4], and FGFI [36], under the same student-
teacher framework. We also evaluate the RDD on the Pillar-based 3D detector,
i.e., CP-Pillar-v0.4 [40]. Our RDD selection method improves the mAP and NDS
by $\\{$3.5%, 4.4%, 2.4%$\\}$ and $\\{$3.4%, 3.8%, 1.7%$\\}$ compared to non-
distillation, Hint [4], and FGFI [36], supervised by the same teacher (CP-
Pillar [43]).

Then we evaluate the proposed distillation losses, i.e., $\mathcal{L}_{feat}$
in Eq. (8) (RDD-F) and $\mathcal{L}_{logit}$ in Eq. (9) (RDD-L), in Tab. 2. As
listed, RDD-F and RDD-L improve the performance when used alone, and the two
losses further boost the performance considerably when combined together. For
example, the RDD-F improve the mAP of CP-Voxel-S [40] by 2.8% and the RDD-L
achieves 3.0% mAP improvement. While combining the RDD-F and RDD-L together,
the performance improvement achieves 3.1%. These ablative experiments further
validates the effectiveness of our method.

Table 3: Experimental results on nuScenes [2]. $\\#$ F and $\\#$ P indicate float operations (FLOPs) and parameters of the detector. Teacher models are marked in gray shadow. Detector |  |  $\\#$ F/$\\#$ P
---
(G/M)
Method |  mAP ($\uparrow$) |  NDS ($\uparrow$)
[HTML]EFEFEF CP-Voxel | 114.8 / 7.8 | - | 56.6 | 64.7
|  | No Distill | 54.0 | 62.1
3-5 |  | GID-L | 54.1 | 62.1
|  | GID-F | 53.4 | 61.8
|  | FG | 54.3 | 62.6
|  | LAD | 53.0 | 61.7
3-5 |  | PP Logit KD | 55.7 | 64.3
CP-Voxel -S | 47.8 / 4.0 | RDD | 57.1 | 65.0
|  | No Distill | 53.0 | 61.8
3-5 |  | GID-L | 53.2 | 61.6
|  | GID-F | 52.9 | 61.3
|  | FG | 53.3 | 61.8
|  | LAD | 53.0 | 61.5
3-5 |  | PP Logit KD | 53.5 | 61.7
CP-Voxel -XS | 36.0 / 2.8 | RDD | 54.0 | 62.1
|  | No Distill | 46.7 | 55.5
3-5 |  | GID-L | 46.8 | 56.5
|  | GID-F | 47.0 | 56.4
|  | FG | 47.1 | 56.2
|  | LAD | 46.6 | 55.3
3-5 |  | PP Logit KD | 47.9 | 57.8
CP-Voxel -XXS | 12.0 / 1.0 | RDD | 49.4 | 57.9
[HTML]EFEFEF CP-Pillar | 333.9 / 5.2 | - | 49.1 | 59.7
|  | No Distill | 46.5 | 55.5
3-5 |  | GID-L | 47.3 | 56.4
|  | GID-F | 47.6 | 56.8
|  | FG | 47.7 | 57.2
|  | LAD | 46.9 | 55.7
3-5 |  | PP Logit KD | 48.6 | 57.5
CP-Pillar -v0.4 | 212.9 / 5.2 | RDD | 50.0 | 58.9
|  | No Distill | 45.3 | 54.2
3-5 |  | GID-L | 45.4 | 54.5
|  | GID-F | 46.1 | 55.3
|  | FG | 47.0 | 56.2
|  | LAD | 45.2 | 54.4
3-5 |  | PP Logit KD | 47.3 | 57.5
CP-Pillar -v0.48 | 149.4 / 5.2 | RDD | 48.8 | 58.5
|  | No Distill | 44.0 | 52.3
3-5 |  | GID-L | 44.2 | 52.6
|  | GID-F | 44.4 | 53.9
|  | FG | 44.7 | 53.7
|  | LAD | 43.9 | 53.2
3-5 |  | PP Logit KD | 45.0 | 55.9
CP-Pillar -v0.64 | 85.1 / 5.2 | RDD | 45.8 | 56.1

###  4.3 Results on nuScenes

We first compare our method with image-based 2D and 3D distillation methods
for the task of 3D object detection on nuScenes [2]. Note that for width
compressed students, we use the same input voxel size as [43], i.e. $0.1$. We
mainly discuss the mAP and NDS (default nuScenes metric) in the following. We
evaluate the proposed RDD on CenterPoint detectors [43] in Tab. 3. For CP-
Voxel-S, compared to non-distillation of GID-L [6], GID-F [6], FG [36] and LAD
[24], our RDD boosts the performance of mAP and NDS by $\\{$3.1%, 3.0%, 3.7%,
2.8%, 4.1%$\\}$ and $\\{$2.9%, 2.9%, 3.2%, 2.4%, 3.3%$\\}$, respectively.
Moreover, our RDD improves the mAP and NDS of CP-Voxel-S by $1.4\%$ and
$0.7\%$ respectively, compared with previous state-of-the-art 3D distillation
method (PP-logit-KD). It is worth noting that our RDD trained CP-Voxel-S even
surpasses the teacher detectors by 0.5% mAP and 0.3% NDS but taking up only
41.6% FLOPs and 51.3% parameters of the teacher, which is a significant
achievement. For CP-Voxel-XS, our RDD improves the performance of mAP and NDS
by $\\{$1.0%, 0.8%, 1.1%, 0.7%, 1.0%$\\}$ and $\\{$0.3%, 0.5%, 0.8%, 0.3%,
0.6%$\\}$, compared to non-distillation of GID-L [6], GID-F [6], FG [36] and
LAD [24]. In addition, our RDD improves the mAP and NDS of CP-Voxel-XS by
$0.5\%$ and $0.4\%$, compared with previous state-of-the-art 3D distillation
method (PP Logit KD). For CP-Voxel-XXS, our RDD surpasses non-distillation of
GID-L [6], GID-F [6], FG [36] and LAD [24] by $\\{$2.7%, 2.6%, 2.4%, 2.3%,
2.8%$\\}$ mAP and $\\{$2.4%, 1.4%, 1.5%, 1.7%, 1.6%$\\}$ NDS. Moreover, our
RDD boosts the mAP and NDS of CP-Voxel-XXS by $1.5\%$ and $0.1\%$
respectively, compared with previous state-of-the-art 3D distillation method
(PP Logit KD). Above experiments well validates the effectiveness of our
method.

Besides, our method generates convincing results on CP-Pillar based detectors
[43]. As shown in the 24-th to 45-th rows of Tab. 3, the performance of the
proposed RDD with CP-Pillar-v0.4, CP-Pillar-v0.48 and CP-Pillar-v0.64
outperforms the non-distillation baseline by $\\{$3.5%, 3.5%, 1.8%$\\}$ and
$\\{$3.4%, 4.3%, 3.8%$\\}$ on mAP and NDS, a large margin. Compared with
previous state-of-the-art 3D distillation methods, our RDD achieves $\\{$1.4%,
1.5%, 0.8%$\\}$ and $\\{$1.4%, 1.0%, 0.2%$\\}$ improvement on mAP and NAS
respectively with CP-Pillar-v0.4, CP-Pillar-v0.48 and CP-Pillar-v0.64, which
well validates the efficacy of our method. The above experimental results
prove the superiority of our RDD method on both Voxel-based and Pillar-based
CenterPoint [43].

Table 4: Experimental results for 3D detection on KITTI [8]. $\\#$ F and $\\#$ P indicate float operations (FLOPs) and parameters of the detector. Teacher models are marked in gray shadow. |  |  |  | 3D
---|---|---|---|---
Detector |  $\\#$ F (G) |  $\\#$ P (M) | Method |  | Moderate
---
mAP@R40($\uparrow$)
[HTML]EFEFEF SECOND | 80.5 | 5.3 | - | 67.2
|  |  | No Distill | 65.6
4-5 |  |  | GID-L | 66.3
|  |  | GID-F | 66.8
|  |  | FG | 66.6
|  |  | LAD | 67.0
4-5 |  |  | PointDistiller | 67.8
|  |  | PP Logit KD | 67.7
SECOND -S | 23.0 | 1.6 | RDD | 68.2
|  |  | No Distill | 64.2
4-5 |  |  | GID-L | 65.0
|  |  | GID-F | 65.2
|  |  | FG | 65.3
|  |  | LAD | 65.7
4-5 |  |  | PointDistiller | 66.3
|  |  | PP Logit KD | 66.4
SECOND -XS | 20.5 | 1.4 | RDD | 67.0
[HTML]EFEFEF PointPillars | 34.3 | 4.8 | - | 60.3
|  |  | No Distill | 58.6
4-5 |  |  | GID-L | 58.9
|  |  | GID-F | 59.1
|  |  | FG | 59.4
|  |  | LAD | 59.2
4-5 |  |  | PointDistiller | 62.3
|  |  | PP Logit KD | 62.2
PointPillars -S | 9.8 | 1.5 | RDD | 63.0
|  |  | No Distill | 58.9
4-5 |  |  | GID-L | 59.2
|  |  | GID-F | 59.6
|  |  | FG | 59.4
|  |  | LAD | 59.7
4-5 |  |  | PointDistiller | 60.0
|  |  | PP Logit KD | 60.2
PointPillars -XS | 8.7 | 1.3 | RDD | 60.9

###  4.4 Results on KITTI

We further show that our RDD can generalize well to KITTI [8] dataset with
anchor-based detectors SECOND [39] and PointPillar [18]. As shown in Tab. 4,
for SECOND-S, compared to non-distillation, GID-L [6], GID-F [6], FG [36] and
LAD [24], our RDD boosts the performance of moderate mAP@R40 by $\\{$2.6%,
1.9%, 1.4%, 1.6%, 1.2%$\\}$. Moreover, our RDD improves the moderate mAP@R40
of SECOND-S by 0.4% and 0.5%, compared with PointDistiller [44] and PP Logit
KD [40]. And SECOND-S trained with our RDD even surpasses its teacher model by
1.0%. And for SECOND-XS, our RDD surpasses previous image-based 2D
distillation methods GID-L [6], GID-F [6], FG [36] and LAD [24] by by
$\\{$2.8%, 2.0%, 1.8%, 1.7%, 1.3%$\\}$ in moderate mAP@R40. SECOND-XS trained
with our RDD also surpasses previous 3D distillation method, PoinrDistiller
and PP Logit KD, by 0.7% and 0.6% on the moderate mAP@R40, which well
validates the effectiveness of our method. Moreover, our method can also be
generalized to PointPillars [18]. As shown in the 20-th to 36-th rows of Tab.
4, the performance of the proposed RDD with PointPillars-S and PointPillars-XS
outperforms the non-distillation baseline by 4.5% and 2.0% on the moderate
mAP@R40, a large margin. Compared with PoinrDistiller, our RDD achieves 1.7%
and 0.9% improvement on the moderate mAP@R40 with PointPillars-S and
PointPillars-XS. Our RDD also surpasses PP Logit KD by 1.8% and 0.7%
improvement on the moderate mAP@R40 with PointPillars-S and PointPillars-XS,
which well validates the efficacy of our method. The above results are of
great significance in the 3D LiDAR-based object detection.

##  5 Conclusion

This paper presents a novel method for training compact 3D LiDAR-based
detectors with knowledge distillation to eliminate the representation
disparity (RDD). RDD employs a information bottleneck (IB) principle to select
the regions with maximum representation disparity and proposes effective
distillation losses to supervise the representation disparity. As a result,
our RDD significantly boosts the performance of compact 3D detectors.
Extensive experiments show that RDD surpasses state-of-the-art compact 3D
detectors and other knowledge distillation methods in 3D LiDAR-based object
detection.

##  6 Acknowledgement

This work was supported in part by the National Natural Science Foundation of
China under Grant 62076016, under Grant 61827901, “One Thousand Plan” projects
in Jiangxi Province Jxsg2023102268, Foundation of China Energy Project
GJNY-19-90.

## References

  * [1] Alex Bewley, Pei Sun, Thomas Mensink, Dragomir Anguelov, and Cristian Sminchisescu.  Range conditioned dilated convolutions for scale invariant 3d object detection.  arXiv preprint arXiv:2005.09927, 2020.
  * [2] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom.  nuscenes: A multimodal dataset for autonomous driving.  In Proc. of CVPR, pages 11621–11631, 2020.
  * [3] Defang Chen, Jian-Ping Mei, Yuan Zhang, Can Wang, Zhe Wang, Yan Feng, and Chun Chen.  Cross-layer distillation with semantic calibration.  In Proc. of AAAI, pages 7028–7036, 2021.
  * [4] Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker.  Learning efficient object detection models with knowledge distillation.  In Proc. of NeurIPS, 2017.
  * [5] Yilun Chen, Shu Liu, Xiaoyong Shen, and Jiaya Jia.  Fast point r-cnn.  In Proc. of ICCV, pages 9775–9784, 2019.
  * [6] Xing Dai, Zeren Jiang, Zhao Wu, Yiping Bao, Zhicheng Wang, Si Liu, and Erjin Zhou.  General instance distillation for object detection.  In Proc. of CVPR, pages 7842–7851, 2021.
  * [7] Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li.  Voxel r-cnn: Towards high performance voxel-based 3d object detection.  In Proc. of AAAI, pages 1201–1209, 2021.
  * [8] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the kitti vision benchmark suite.  In Proc. of CVPR, pages 3354–3361, 2012.
  * [9] Benjamin Graham and Laurens van der Maaten.  Submanifold sparse convolutional networks.  arXiv preprint arXiv:1706.01307, pages 1–10, 2017.
  * [10] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proc. of ICCV, pages 2961–2969, 2017.
  * [11] Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi.  Knowledge transfer via distillation of activation boundaries formed by hidden neurons.  In Proc. of AAAI, pages 3779–3787, 2019.
  * [12] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [13] Yuenan Hou, Zheng Ma, Chunxiao Liu, and Chen Change Loy.  Learning lightweight lane detection cnns by self attention distillation.  In Proc. of ICCV, pages 1013–1021, 2019.
  * [14] Zehao Huang and Naiyan Wang.  Like what you like: Knowledge distill via neuron selectivity transfer.  arXiv preprint arXiv:1707.01219, 2017.
  * [15] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In Proc. of ICML, pages 448–456, 2015.
  * [16] Xiao Jin, Baoyun Peng, Yichao Wu, Yu Liu, Jiaheng Liu, Ding Liang, Junjie Yan, and Xiaolin Hu.  Knowledge distillation via route constrained optimization.  In Proc. of ICCV, pages 1345–1354, 2019.
  * [17] Nikos Komodakis and Sergey Zagoruyko.  Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer.  In Proc. of ICLR, pages 1–13, 2017.
  * [18] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom.  Pointpillars: Fast encoders for object detection from point clouds.  In Proc. of CVPR, pages 12697–12705, 2019.
  * [19] Quanquan Li, Shengying Jin, and Junjie Yan.  Mimicking very efficient network for object detection.  In Proc. of CVPR, pages 6356–6364, 2017.
  * [20] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In Proc. of CVPR, pages 2117–2125, 2017.
  * [21] Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang.  Structured knowledge distillation for semantic segmentation.  In Proc. of CVPR, pages 2604–2613, 2019.
  * [22] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  In Proc. of ICLR, pages 1–18, 2017.
  * [23] Vinod Nair and Geoffrey E Hinton.  Rectified linear units improve restricted boltzmann machines.  In Proc. of ICML, pages 1–8, 2010.
  * [24] Chuong H Nguyen, Thuy C Nguyen, Tuan N Tang, and Nam LH Phan.  Improving object detection by label assignment distillation.  In Proc. of WACV, pages 1005–1014, 2022.
  * [25] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  In Proc. of NeurIPS, page 5099–5108, 2017.
  * [26] Lu Qi, Jason Kuen, Jiuxiang Gu, Zhe Lin, Yi Wang, Yukang Chen, Yanwei Li, and Jiaya Jia.  Multi-scale aligned distillation for low-resolution detection.  In Proc. of CVPR, pages 14443–14453, 2021.
  * [27] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio.  Fitnets: Hints for thin deep nets.  arXiv preprint arXiv:1412.6550, 2014.
  * [28] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li.  Pv-rcnn: Point-voxel feature set abstraction for 3d object detection.  In Proc. of CVPR, pages 10529–10538, 2020.
  * [29] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li.  Pointrcnn: 3d object proposal generation and detection from point cloud.  In Proc. of CVPR, pages 770–779, 2019.
  * [30] Ravid Shwartz-Ziv and Naftali Tishby.  Opening the black box of deep neural networks via information.  arXiv preprint arXiv:1703.00810, 2017.
  * [31] Leslie N Smith.  Cyclical learning rates for training neural networks.  In Proc. of WACV, pages 464–472. IEEE, 2017.
  * [32] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al.  Scalability in perception for autonomous driving: Waymo open dataset.  In Proc. of CVPR, pages 2446–2454, 2020.
  * [33] Pei Sun, Weiyue Wang, Yuning Chai, Gamaleldin Elsayed, Alex Bewley, Xiao Zhang, Cristian Sminchisescu, and Dragomir Anguelov.  Rsn: Range sparse net for efficient, accurate lidar 3d object detection.  In Proc. of CVPR, pages 5725–5734, 2021.
  * [34] Ruoyu Sun, Fuhui Tang, Xiaopeng Zhang, Hongkai Xiong, and Qi Tian.  Distilling object detectors with task adaptive regularization.  arXiv preprint arXiv:2006.13108, 2020.
  * [35] Chaofei Wang, Qisen Yang, Rui Huang, Shiji Song, and Gao Huang.  Efficient knowledge distillation from model checkpoints.  arXiv preprint arXiv:2210.06458, 2022.
  * [36] Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng.  Distilling object detectors with fine-grained feature imitation.  In Proc. of CVPR, pages 4933–4942, 2019.
  * [37] Yue Wang, Alireza Fathi, Abhijit Kundu, David A Ross, Caroline Pantofaru, Tom Funkhouser, and Justin Solomon.  Pillar-based object detection for autonomous driving.  In Proc. of ECCV, pages 18–34, 2020.
  * [38] Ziwei Wang, Ziyi Wu, Jiwen Lu, and Jie Zhou.  Bidet: An efficient binarized object detector.  In Proc. of CVPR, pages 2049–2058, 2020.
  * [39] Yan Yan, Yuxing Mao, and Bo Li.  Second: Sparsely embedded convolutional detection.  Sensors, 18(10):3337, 2018.
  * [40] Jihan Yang, Shaoshuai Shi, Runyu Ding, Zhe Wang, and Xiaojuan Qi.  Towards efficient 3d object detection with knowledge distillation.  In Proc. of NeurIPS, 2022.
  * [41] Zhendong Yang, Zhe Li, Xiaohu Jiang, Yuan Gong, Zehuan Yuan, Danpei Zhao, and Chun Yuan.  Focal and global knowledge distillation for detectors.  In Proc. of CVPR, pages 4643–4652, 2022.
  * [42] Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia.  Std: Sparse-to-dense 3d object detector for point cloud.  In Proc. of ICCV, pages 1951–1960, 2019.
  * [43] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl.  Center-based 3d object detection and tracking.  In Proc. of CVPR, pages 11784–11793, 2021.
  * [44] Linfeng Zhang, Runpei Dong, Hung-Shuo Tai, and Kaisheng Ma.  Pointdistiller: Structured knowledge distillation towards efficient and compact 3d detection.  arXiv preprint arXiv:2205.11098, 2022.
  * [45] Yifan Zhang, Qingyong Hu, Guoquan Xu, Yanxin Ma, Jianwei Wan, and Yulan Guo.  Not all points are equal: Learning highly efficient point-based detectors for 3d lidar point clouds.  In Proc. of CVPR, pages 18953–18962, 2022.
  * [46] Yin Zhou and Oncel Tuzel.  Voxelnet: End-to-end learning for point cloud based 3d object detection.  In Proc. of CVPR, pages 4490–4499, 2018.