Rational approximations to two irrational numbers222This research is supported
by the Russian Science Foundation under grant [19-11-00065] and performed in
Khabarovsk Division of the Institute for Applied Mathematics, Far Eastern
Branch, Russian Academy of Sciences..
Nikita Shulga

###### Abstract

For real $\xi$ we consider the irrationality measure function
$\psi_{\xi}(t)=\min_{1\leqslant q\leqslant t,q\in\mathbb{Z}}||q\xi||$. We
prove that in the case $\alpha\pm\beta\notin\mathbb{Z}$ there exist arbitrary
large values of $t$ with

$\Bigl{|}\frac{1}{\psi_{\alpha}(t)}-\frac{1}{\psi_{\beta}(t)}\Bigl{|}\geqslant\sqrt{5}\left(1-\sqrt{\frac{\sqrt{5}-1}{2}}\right)t.$

The constant on the right-hand side is optimal.

##  1 Introduction

For irrational number $\xi\in\mathbb{R}$ we consider irrationality measure
function

$\psi_{\xi}(t)=\min_{1\leq q\leq t,\,q\in\mathbb{Z}}\|q\xi\|,$

where $||.||$ denotes distance to the nearest integer.
Let

$Q_{0}\leq Q_{1}<Q_{2}<\ldots<Q_{n}<Q_{n+1}<\ldots$

be the sequence of denominators of convergents to $\xi$. It it a well-known
fact (see [1] or [2]) that

$\psi_{\xi}(t)=\|Q_{n}\xi\|\text{ for }Q_{n}\leq t<Q_{n+1}.$ (1)

It is clear for any $\xi\in\mathbb{R}$ and for every $t\geq 1$ we have

$\psi_{\xi}(t)\leq\frac{1}{t}.$ (2)

In [3], Kan and Moshchevitin proved that for any two irrational numbers
$\alpha,\beta$ satisfying $\alpha\pm\beta\notin\mathbb{Z}$ the difference

$\psi_{\alpha}(t)-\phi_{\beta}(t)$

changes its sign infinitely many times as $t\to\infty$.

Obviously, it follows that the difference of their reciprocals

$d(t)=d_{\alpha,\beta}(t)=\frac{1}{\psi_{\beta}(t)}-\frac{1}{\psi_{\alpha}(t)}$
(3)

also changes sign infinitely many times as $t\to\infty$.

In 2017 Dubickas [4] proved the following result.

######  Theorem A.

For any irrational numbers $\alpha,\beta$ satisfying
$\alpha\pm\beta\notin\mathbb{Z}$ the sequence $d(n),n\in\mathbb{Z_{+}}$ is
unbounded.

To formulate further results, we need constants

$\tau=\frac{\sqrt{5}+1}{2},$ (4)

$\phi=\frac{\sqrt{5}-1}{2},$ (5)

$K=\sqrt{\tau}-1=0.2720^{+},$ (6)

$C=K(\sqrt{\tau}+\tau^{-3/2})=\sqrt{5}(1-\sqrt{\phi})=0.47818^{+}.$ (7)

In 2019 Moshchevitin [5] proved the following

######  Theorem B.

Let $\alpha$ and $\beta$ be the two irrational numbers. If
$\alpha\pm\beta\notin\mathbb{Z}$, then for every $T\geq 1$ there exists $t\geq
T$ such that

$|\psi_{\alpha}(t)-\psi_{\beta}(t)|\geqslant
K\cdot\min(\psi_{\alpha}(t),\psi_{\beta}(t)).$ (8)

It was also proved that constant $K$ in (8) is optimal.
Using Theorem B and (2) Moshchevitin deduced stronger version of Theorem A.

######  Corollary A.

Let $\alpha$ and $\beta$ be the two irrational numbers. If
$\alpha\pm\beta\notin\mathbb{Z}$, then for every $T\geq 1$ there exists $t\geq
T$ such

$\Bigl{|}\frac{1}{\psi_{\alpha}(t)}-\frac{1}{\psi_{\beta}(t)}\Bigl{|}\geqslant
Kt,\text{\,\,\, with $K$ from \eqref{constK}}.$ (9)

In this paper we give an improvement of this result with the best possible
constant.

######  Theorem 1.

1) Let $\alpha$ and $\beta$ be the two irrational numbers. If
$\alpha\pm\beta\notin\mathbb{Z}$, then for every $T\geq 1$ there exists $t\geq
T$ such

$\Bigl{|}\frac{1}{\psi_{\alpha}(t)}-\frac{1}{\psi_{\beta}(t)}\Bigl{|}\geqslant
Ct,\text{\,\,\, where $C$ is defined in \eqref{constC}}.$ (10)

2) Constant $C$ in (10) is optimal, that is for any $\varepsilon>0$, there
exist $\alpha$ and $\beta$ with $\alpha\pm\beta\notin\mathbb{Z}$ such that

$\Bigl{|}\frac{1}{\psi_{\alpha}(t)}-\frac{1}{\psi_{\beta}(t)}\Bigl{|}\leqslant(C+\varepsilon)t$

for all $t$ large enough.

##  2 Auxiliary results

Throughout, we consider $\alpha\notin\mathbb{Q}$ and its continued fraction
expansion

$\alpha=a_{0}+\cfrac{1}{a_{1}+\cfrac{1}{a_{2}+\cdots}}=[a_{0};a_{1},a_{2},\ldots],\,\,\,a_{j}\in\mathbb{Z}_{+}$
(11)

By $\alpha_{r}$ we denote a tail of continued fraction

$\alpha_{r}=[a_{r};a_{r+1},a_{r+2},\ldots]$ (12)

Similarly, let $\beta=[b_{0};b_{1},b_{2},\ldots]$ and
$\beta_{m}=[b_{m};b_{m+1},b_{m+2},\ldots]$.
For irrational numbers $\alpha$ and $\beta$ we denote the denominators of
their convergents by $q_{n}$ and $t_{m}$ respectively.
Define

$\xi_{n}=\psi_{\alpha}(q_{n})\text{\,\,\,\,\, and
\,\,\,\,\,\,}\eta_{s}=\psi_{\beta}(t_{s}).$

It is well known that

$q_{n}\alpha-\frac{(-1)^{n}}{q_{n}\alpha_{n+1}+q_{n-1}}\in\mathbb{Z}.$ (13)

Using this fact as well as (1) we see that

$\frac{1}{\psi_{\alpha}(n)}=q_{r}\alpha_{r+1}+q_{r-1}=q_{r+1}+\frac{q_{r}}{\alpha_{r+2}}$
(14)

where $r\geqslant 0$ is the largest integer satisfying $q_{r}\leqslant n$.

Similarly,

$\frac{1}{\psi_{\beta}(n)}=t_{l}\beta_{l+1}+t_{l-1}=t_{l+1}+\frac{t_{l}}{\beta_{l+2}}$
(15)

where $l\geqslant 0$ is the largest integer satisfying $t_{l}\leqslant n$.

From now on we consider only irrational numbers $\alpha$ and $\beta$ with
$\alpha\pm\beta\notin\mathbb{Z}$.
The following two simple lemmas in fact were used in [5]. Here we formulate
them without a proof.

######  Lemma 2.1.

There exist only finitely many pairs $(q_{n},q_{n+1})=(t_{m},t_{m+1})$.

######  Lemma 2.2.

There exist only finitely many pairs $(q_{n},q_{n+2})=(t_{m+1},t_{m+2})$ with
$a_{n+2}=1$.

Our next auxiliary result is the following

######  Lemma 2.3.

If $\eta_{s}\in(\xi_{n},\xi_{n-1})$, then either

$\frac{1}{\eta_{s}}-\frac{1}{\xi_{n-1}}\geq
t_{s}\left(\beta_{s+1}+\frac{t_{s-1}}{t_{s}}\right)\left(1-\frac{1}{\sqrt{\alpha_{n+1}}}\right)$

or

$\frac{1}{\xi_{n}}-\frac{1}{\eta_{s}}\geq
q_{n}\left(\alpha_{n+1}+\frac{q_{n-1}}{q_{n}}\right)\left(1-\frac{1}{\sqrt{\alpha_{n+1}}}\right).$

###### Proof.

It is a well-known fact, that for any $n\in\mathbb{N}$ one has

$\frac{\xi_{n-1}}{\xi_{n}}=\alpha_{n+1}.$

As

$\frac{\xi_{n-1}}{\eta_{s}}\cdot\frac{\eta_{s}}{\xi_{n}}=\frac{\xi_{n-1}}{\xi_{n}}=\alpha_{n+1},$

we see that either $\frac{\xi_{n-1}}{\eta_{s}}\geq\sqrt{\alpha_{n+1}}$ or
$\frac{\eta_{s}}{\xi_{n}}\geq\sqrt{\alpha_{n+1}}$.

In the first case, one has

$\frac{1}{\eta_{s}}-\frac{1}{\xi_{n-1}}\geq\frac{1}{\eta_{s}}-\frac{1}{\sqrt{\alpha_{n+1}}\eta_{s}}=\frac{1}{\eta_{s}}\left(1-\frac{1}{\sqrt{\alpha_{n+1}}}\right)=t_{s}\left(\beta_{s+1}+\frac{t_{s-1}}{t_{s}}\right)\left(1-\frac{1}{\sqrt{\alpha_{n+1}}}\right).$

Last equality is due to (15).
Similarly, in the case $\frac{\eta_{s}}{\xi_{n}}\geq\sqrt{\alpha_{n+1}}$,
using (14) we get

$\frac{1}{\xi_{n}}-\frac{1}{\eta_{s}}\geq\frac{1}{\xi_{n}}-\frac{1}{\sqrt{\alpha_{n+1}}\xi_{n}}=\frac{1}{\xi_{n}}\left(1-\frac{1}{\sqrt{\alpha_{n+1}}}\right)=q_{n}\left(\alpha_{n+1}+\frac{q_{n-1}}{q_{n}}\right)\left(1-\frac{1}{\sqrt{\alpha_{n+1}}}\right).$

∎

By $F_{n}$ we denote $n$th Fibonacci number, that is

$F_{1}=F_{2}=1,F_{n+1}=F_{n}+F_{n-1}.$

By $\langle a_{1},\ldots,a_{n}\rangle$ we denote a denominator of the
continued fraction $[0;a_{1},\ldots,a_{n}]$. Using this notation, we recall
that

$\langle a_{1},\ldots,a_{n},b_{1},\ldots,b_{k}\rangle=\langle a_{1},\ldots
a_{n}\rangle\langle b_{1}\ldots b_{k}\rangle+\langle a_{1},\ldots
a_{n-1}\rangle\langle b_{2}\ldots b_{k}\rangle.$ (16)

##  3 Proof of the main inequality

In this section we prove statement 1) of Theorem 1.

For a contradiction, assume that for some $\alpha$ and $\beta$ with
$\alpha\pm\beta\notin\mathbb{Z}$ there exists $T\in\mathbb{Z}_{+}$ such that
for $d(t)$ defined in (3) we have

$|d(t)|<Ct$ (17)

for each $t>T$.

######  Lemma 3.1.

Suppose that

$q_{n-1}\leqslant t_{m-1}<q_{n}<t_{m}$

and (17) holds for all $t\geq t_{m-1}$, then

$a_{n+1}=1.$ (18)

Similarly, if

$t_{m-1}\leqslant q_{n-1}<t_{m}<q_{n}$

and (17) holds for all $t\geq q_{n-1}$, then

$b_{m+1}=1.$ (19)

###### Proof.

Let us prove (18). Using (3), (14) and (15) we obtain

$d(t_{m-1})=t_{m}+\frac{t_{m-1}}{\beta_{m+1}}-q_{n}-\frac{q_{n-1}}{\alpha_{n+1}}$

and

$d(q_{n})=t_{m}+\frac{t_{m-1}}{\beta_{m+1}}-q_{n+1}-\frac{q_{n}}{\alpha_{n+2}}.$

Subtracting the second formula from the first one we get

$2Cq_{n}>d(t_{m-1})-d(q_{n})=q_{n+1}+\frac{q_{n}}{\alpha_{n+2}}-q_{n}-\frac{q_{n-1}}{\alpha_{n+1}}$
$>q_{n}\Bigl{(}a_{n+1}-1+\frac{1}{\alpha_{n+2}}\Bigl{)}+q_{n-1}\Bigl{(}1-\frac{1}{\alpha_{n+1}}\Bigl{)}>q_{n}(a_{n+1}-1).$

Hence, $a_{n+1}<2C+1=1.95636^{+}$. As $a_{n+1}\in\mathbb{N}$, we see that
$a_{n+1}=1$. The proof of (19) is the same. ∎

To prove statement 1) of Theorem 1 we consider two cases.
Case 1. Either there exist infinitely many $n$, such that $a_{n+1}\geq 2$ or
infinitely many $m$, such that $b_{m+1}\geq 2$.
Case 2. Both $\alpha$ and $\beta$ are equivalent to $\tau$.

In Case 1 we consider several subcases.
1.1) There exist infinitely many $a_{n+1}\geq 2$, such that $q_{n}$ is not a
denominator of convergent to $\beta$ or there exist infinitely many
$b_{m+1}\geq 2$, such that $t_{m}$ is not a denominator of convergent to
$\alpha$.
Without loss of generality suppose that this condition holds for partial
quotients and denominators of $\alpha$.
Consider $a_{n+1}\geq 2$, such that $q_{n}$ is not a denominator of convergent
to $\beta$ and (17) being true for $t\geq q_{n-1}$.
Consider indices $n$ and $m$ such that $t_{m-1}<q_{n}<t_{m}$. Suppose that one
of the

$q_{n-1}=t_{m-1}<q_{n}<t_{m}\text{\,\,\,\,or\,\,\,\,\,\,\,}q_{n-1}<t_{m-1}<q_{n}<t_{m}$

is true. Then by Lemma 3.1 $a_{n+1}$ has to be equal to 1. There is a
contradiction with $a_{n+1}\geq 2$. Hence the only possible case is

$t_{m-1}<q_{n-1}<q_{n}<t_{m}.$ (20)

Now using (3) and (20) we obtain

$d(q_{n-1})=t_{s+1}+\frac{t_{s}}{\beta_{s+2}}-q_{n}-\frac{q_{n-1}}{\alpha_{n+1}}$

and

$d(q_{n})=t_{s+1}+\frac{t_{s}}{\beta_{s+2}}-q_{n+1}-\frac{q_{n}}{\alpha_{n+2}}.$

By subtracting the second formula from the first we get

$d(q_{n-1})-d(q_{n})=q_{n+1}+\frac{q_{n}}{\alpha_{n+2}}-q_{n}-\frac{q_{n-1}}{\alpha_{n+1}}$
$=q_{n}\Bigl{(}a_{n+1}-1+\frac{1}{\alpha_{n+2}}\Bigl{)}+q_{n-1}\Bigl{(}1-\frac{1}{\alpha_{n+1}}\Bigl{)}>q_{n}.$

The last inequality is due to $a_{n+1}\geq 2$. As (17) is true for $t\geq
q_{n-1}$, we can estimate

$d(q_{n-1})-d(q_{n})<Cq_{n-1}+Cq_{n}<2Cq_{n}.$

We come to a contradiction $2Cq_{n}>q_{n}$, as $2C=0.95636^{+}$.
1.2) There exists $N$, such that for every $a_{n+1}\geq 2$ with $n>N$, $q_{n}$
is a denominator of convergent to $\beta$, for every $b_{n+1}\geq 2$ with
$n>N$, $t_{n}$ is a denominator of convergent to $\alpha$ and (17) is true for
$t\geq\min(q_{N},t_{N})$.
Without loss of generality suppose that there exist infinitely many
$a_{n+1}\geq 2$ for $\alpha$.
Consider $a_{n+1}\geq 2$ with $n>N$. Then $q_{n}=t_{s}$ for some index $s$.

Next, for the local convenience we introduce the union $U=D_{\alpha}\cup
D_{\beta}$ of two sequences $D_{\alpha}=\\{q_{0}=1\leq q_{1}<q_{2}<\ldots\\}$
and $D_{\beta}=\\{t_{0}=1\leq t_{1}<t_{2}<\ldots\\}$. We construct an infinite
word $W$ on an alphabet $\\{B^{*}_{*},Q^{*},T_{*}\\}$, where $*$ will match
some indices from $D_{\alpha}$ and $D_{\beta}$, by following procedure. The
first element of $U$ is 1. It belongs to both sequences, so we start our
infinite word with $B^{0}_{0}$. Then, we write the letter $Q^{j}$ if the next
element of $U$ is $q_{j}$ from the set $D_{\alpha}$, but not from the set
$D_{\beta}$. We write letter $T_{i}$ if it is $t_{i}$ from the set
$D_{\beta}$, but not from the set $D_{\alpha}$. Also, we write $B^{j}_{i}$ if
the next element of $U$ is present in both $D_{\alpha}$ and $D_{\beta}$ as
$q_{j}$ and $t_{i}$ respectively.

As $q_{n}=t_{s}$, the word $W$ contains letter $B^{n}_{s}$. We consider
possible cases of the location of the next letter $B^{*}_{*}$.
1.2.1) The word $W$ can not contain $B^{n}_{s}B^{n+1}_{s+1}$ as a subword
infinitely many times, since it is forbidden by Lemma 2.1.
1.2.2) The word $W$ can not contain subwords $B^{n}_{s}Q^{n+1}B^{n+2}_{s+1}$
or $B^{n}_{s}T_{s+1}B^{n+1}_{s+2}$ infinitely many times as it is forbidden by
Lemma 3.1 and Lemma 2.2. Indeed, by Lemma 3.1, if $W$ has a subword
$B^{n}_{s}Q^{n+1}$ or $B^{n}_{s}T_{s+1}$, then $a_{n+2}=1$ or $b_{s+2}=1$
respectively. Both cases are covered by Lemma 2.2, which states that there
exist only finitely many subwords $B^{n}_{s}Q^{n+1}B^{n+2}_{s+1}$ or
$B^{n}_{s}T_{s+1}B^{n+1}_{s+2}$ with $a_{n+2}=1$ or $b_{s+2}=1$ respectively.
So there are not such subwords for $n,s$ large enough.
1.2.3a) Let the word $W$ contain a subword $B^{n}_{s}Q^{n+1}\ldots
Q^{n+k-1}B^{n+k}_{s+1}$ with $k\geq 3$. Then

$t_{s+1}\geq q_{n+3}=q_{n+2}+q_{n+1}=2q_{n+1}+q_{n}$

and by (3) we have

$d(q_{n})=t_{s+1}-q_{n+1}+\frac{t_{s}}{\beta_{s+2}}-\frac{q_{n}}{\alpha_{n+2}}$

$\geq
q_{n+3}-q_{n+1}+q_{n}\Bigl{(}\frac{1}{\beta_{s+2}}-\frac{1}{\alpha_{n+2}}\Bigl{)}$

$=q_{n+1}+q_{n}\Bigl{(}1+\frac{1}{\beta_{s+2}}-\frac{1}{\alpha_{n+2}}\Bigl{)}>q_{n+1}>a_{n+1}q_{n}\geq
2q_{n}.$

As $|d(q_{n})|<Cq_{n}$, where $C=\sqrt{5}(1-\sqrt{\phi})=0.47818^{+}$, we come
to a contradiction with $d(q_{n})>2q_{n}$.
1.2.3b) Let the word $W$ contain a subword $B^{n}_{s}T_{s+1}\ldots
T_{s+m-1}B^{n+1}_{s+m}$ with $m\geq 3$. Then

$q_{n+1}\geq t_{s+3}=t_{s+2}+t_{s+1}=2t_{s+1}+t_{s}$

by (3) we have

$-d(t_{s})=q_{n+1}-t_{s+1}+\frac{q_{n}}{\alpha_{n+2}}-\frac{t_{s}}{\beta_{s+2}}$

$\geq
t_{s+3}-t_{s+1}+t_{s}\Bigl{(}\frac{1}{\beta_{s+2}}-\frac{1}{\alpha_{n+2}}\Bigl{)}$

$=t_{s+1}+t_{s}\Bigl{(}1+\frac{1}{\beta_{s+2}}-\frac{1}{\alpha_{n+2}}\Bigl{)}>t_{s+1}>t_{s}.$

As $|d(t_{s})|<Ct_{s}$, where $C=\sqrt{5}(1-\sqrt{\phi})=0.47818^{+}$, we come
to a contradiction with $-d(t_{s})>t_{s}$.
1.2.4) The word $W$ contains subword $B^{n}_{s}\ldots B^{n+k+1}_{s+m+1}$,
where between $B^{n}_{s}$ and $B^{n+k+1}_{s+m+1}$ there are $k\geq 1$ letters
$Q^{*}$ and $m\geq 1$ letters $T_{*}$ in any order.

We know that

$\langle a_{1},\ldots,a_{n}\rangle=q_{n}=t_{s}=\langle
b_{1},\ldots,b_{s}\rangle$ and $\langle
a_{1},\ldots,a_{n+k+1}\rangle=q_{n+k+1}=t_{s+m+1}=\langle
b_{1},\ldots,b_{s+m+1}\rangle.$

By the conditions of the case 1.2), if there is a partial quotient
$a_{n+1}\geq 2$ or $b_{n+1}\geq 2$ for $n>N$, then the word $W$ contains a
letter $B^{n}_{*}$ or $B^{*}_{n}$ respectively. As we consider letter
$B^{n}_{s}$ along with the closest to it letter $B^{*}_{*}$, we deduce that
there are only partial quotients equal to $1$ in sequences
$(a_{n+2},\ldots,a_{n+k+1})$ and $(b_{s+2},\ldots,b_{s+m+1})$. So we have

$q_{n+k+1}=\langle a_{1},\ldots,a_{n},a_{n+1},\underbrace{1,\ldots,1}_{k\geq
1\text{ times}}\rangle\text{, \,\,\, where \,\,\,}a_{n+1}\geq 2$

and

$t_{s+m+1}=\langle b_{1},\ldots,b_{s},b_{s+1},\underbrace{1,\ldots,1}_{m\geq
1\text{ times}}\rangle.$

Using formula (16), we have

$\langle a_{1},\ldots a_{n+1}\rangle\langle\underbrace{1,\ldots,1}_{k\text{
times}}\rangle+\langle a_{1},\ldots
a_{n}\rangle\langle\underbrace{1,\ldots,1}_{k-1\text{ times}}\rangle$
$=\langle b_{1},\ldots b_{s+1}\rangle\langle\underbrace{1,\ldots,1}_{m\text{
times}}\rangle+\langle b_{1},\ldots
b_{s}\rangle\langle\underbrace{1,\ldots,1}_{m-1\text{ times}}\rangle.$

Or, for short,

$q_{n+1}F_{k+1}+q_{n}F_{k}=t_{s+1}F_{m+1}+t_{s}F_{m}.$

From this equality we can express $t_{s+1}$ as

$t_{s+1}=q_{n+1}\frac{F_{k+1}}{F_{m+1}}+q_{n}\frac{F_{k}-F_{m}}{F_{m+1}}.$
(21)

Now, if $t_{s+1}>q_{n+1}$, then $m\leq k-1$. Indeed, assume that $m\geq k$.
Then from (21) we have

$t_{s+1}=q_{n+1}\frac{F_{k+1}}{F_{m+1}}+q_{n}\frac{F_{k}-F_{m}}{F_{m+1}}\leq
q_{n+1}\frac{F_{m+1}}{F_{m+1}}+q_{n}\frac{F_{m}-F_{m}}{F_{m+1}}=q_{n+1}.$

Next we estimate tails of continued fractions of $\alpha$ and $\beta$ as

$\frac{1}{\beta_{s+2}}=[b_{s+2},b_{s+3},\ldots]>\frac{1}{b_{s+2}+1}=\frac{1}{2},$
(22)
$\frac{1}{\alpha_{n+2}}=[a_{n+2},a_{n+3},\ldots]<\frac{1}{a_{n+2}+\frac{1}{a_{n+3}+1}}=\frac{1}{1+\frac{1}{1+1}}=\frac{2}{3}.$
(23)

We apply (21),(22), (23), the fact that $m\leq k-1$ and $q_{n+1}>a_{n+1}q_{n}$
to get

$d(q_{n})=t_{s+1}-q_{n+1}+\frac{t_{s}}{\beta_{s+2}}-\frac{q_{n}}{\alpha_{n+2}}$
$=q_{n+1}\Bigl{(}\frac{F_{k+1}-F_{m+1}}{F_{m+1}}\Bigl{)}+q_{n}\Bigl{(}\frac{F_{k}-F_{m}}{F_{m+1}}+\frac{1}{\beta_{s+2}}-\frac{1}{\alpha_{n+2}}\Bigl{)}$
$\geq(a_{n+1}-1)q_{n}\frac{F_{m}}{F_{m+1}}+q_{n}\Bigl{(}1+\frac{1}{\beta_{s+2}}-\frac{1}{\alpha_{n+2}}\Bigl{)}>\frac{3a_{n+1}+2}{6}q_{n}\geq\frac{4}{3}q_{n}.$

As before, we come to a contradiction with $d(q_{n})<Cq_{n}=0.47818^{+}q_{n}$.
If $t_{s+1}<q_{n+1}$, then we use

$t_{s+1}=q_{n+1}\frac{F_{k+1}}{F_{m+1}}+q_{n}\frac{F_{k}-F_{m}}{F_{m+1}},$
$m\geq k+1,$
$\frac{1}{\alpha_{n+2}}=[a_{n+2},a_{n+3},\ldots]>\frac{1}{a_{n+2}+1}=\frac{1}{2},$
$\frac{1}{\beta_{s+2}}=[b_{s+2},b_{s+3},\ldots]<\frac{1}{b_{s+2}+\frac{1}{b_{s+3}+1}}=\frac{1}{1+\frac{1}{1+1}}=\frac{2}{3}$

to get

$Cq_{n}>-d(q_{n})=q_{n+1}-t_{s+1}+\frac{q_{n}}{\alpha_{n+2}}-\frac{t_{s}}{\beta_{s+2}}$
$=q_{n+1}\frac{F_{m+1}-F_{k+1}}{F_{m+1}}+q_{n}\left(\frac{F_{m}-F_{k}}{F_{m+1}}+\frac{1}{\alpha_{n+2}}-\frac{1}{\beta_{s+2}}\right)$
$>a_{n+1}q_{n}\frac{F_{m-1}}{F_{m+1}}+q_{n}\left(\frac{F_{m-2}}{F_{m+1}}-\frac{1}{6}\right)>q_{n}\left(\frac{2F_{m-1}+F_{m-2}}{F_{m+1}}-\frac{1}{6}\right)=\frac{5}{6}q_{n}.$

2) There exists $N_{0}$, such that for every $n,m>N_{0}$ one has $a_{n+1}=1$
and $b_{m+1}=1$. This means that $\beta_{n+2}=\alpha_{n+2}=\tau$ for any
$n>N_{0}$. Let us construct a word $W$ as in the case 1.2).

######  Lemma 3.2.

There are only finitely many subwords $Q^{n}Q^{n+1}$ or $T_{s}T_{s+1}$ in word
$W$ under the conditions of the case 2).

###### Proof.

Assume that $W$ has infinitely many subwords $Q^{n}Q^{n+1}$. Then, it has
infinitely many subwords $T_{s}Q^{n}Q^{n+1}$ or $B_{s}^{n-1}Q^{n}Q^{n+1}$. In
both cases we have

$q_{n-1}\leq t_{s}<q_{n}<q_{n+1}<t_{s+1},$

so we can deduce

$-d(q_{n+1})=q_{n+1}\alpha_{n+2}+q_{n}-t_{s}\beta_{s+1}-t_{s-1}$

and

$-d(t_{s})=q_{n}+\frac{q_{n-1}}{\alpha_{n+1}}-t_{s}\beta_{s+1}-t_{s-1}.$

By subtracting the second equality from the first we obtain

$C(q_{n+1}+q_{n})>C(q_{n+1}+t_{s})>d(t_{s})-d(q_{n-1})=q_{n+1}\alpha_{n+2}-\frac{q_{n-1}}{\alpha_{n+1}}=\tau
q_{n+1}-\phi q_{n-1},$

or

$q_{n+1}<q_{n+1}(\tau-C)<Cq_{n}+\phi q_{n-1}<\phi(q_{n}+q_{n-1})=\phi
q_{n+1},$

which is a contradiction. In case $W$ has infinitely many subwords
$T_{s}T_{s+1}$ the proof is the same. ∎

Now let us show that there are only finitely many letters $B^{*}_{*}$. Assume
that the word $W$ contains letter $B^{n}_{s}$ for $n,s$ large enough. Then by
Lemma 2.1 the next letter is either $Q^{n+1}$ or $T_{s+1}$. Without loss of
generality suppose that it is $Q^{n+1}$. Then by Lemma 2.2 and Lemma 3.2 the
next letter is $T_{s+1}$. If the next letter is $B_{s+2}^{n+2}$, then, as
$b_{s+2}=a_{n+2}=1$, we have

$q_{n+1}+q_{n}=q_{n+2}=t_{s+2}=t_{s+1}+t_{s}$

and as $q_{n}=t_{s}$, we get $q_{n+1}=t_{s+1}$, which is a contradiction.
$B^{n}_{s}Q^{n+1}T_{s+1}T_{s+2}$ is forbidden by Lemma 3.2, hence we always
have a subword $B^{n}_{s}Q^{n+1}T_{s+1}Q^{n+2}$. Now we can show that a
subword $Q^{i}T_{j}Q^{i+1}B_{j+1}^{i+2}$ is also forbidden for any $i>n,j>s$,
since in the equality

$t_{j}+t_{j-1}=t_{j+1}=q_{i+2}=q_{i+1}+q_{i}$

the terms on the right side are greater than the corresponding terms on the
left. The same argument works with a subword $T_{j-1}Q^{i}T_{j}B_{j+1}^{i+1}$.

We have shown that for $n$ large enough, $W$ has an infinite subword
$Q^{n-1}T_{s}Q^{n}T_{s+1}\ldots$.
In our case this also implies that

$\xi_{n-1}>\eta_{s}>\xi_{n}>\eta_{s+1}>\xi_{n+1}>\ldots.$

At this point we use Lemma 2.3 twice: for $\eta_{s}\in(\xi_{n},\xi_{n-1})$ and
for $\xi_{n}\in(\eta_{s+1},\eta_{s})$. Then, if
$\frac{\eta_{s}}{\xi_{n}}\leq\sqrt{\tau}$, we have

$\frac{\xi_{n-1}}{\eta_{s}}\geq\sqrt{\tau}\text{ \,\,\,\,\, and \,\,\,\,\,
}\frac{\xi_{n}}{\eta_{s+1}}\geq\sqrt{\tau}.$

Hence both

$d(t_{s})=\frac{1}{\eta_{s}}-\frac{1}{\xi_{n-1}}\geq
t_{s}\left(\tau+\frac{t_{s-1}}{t_{s}}\right)\left(1-\sqrt{\phi}\right)$ (24)

and

$d(t_{s+1})=\frac{1}{\eta_{s+1}}-\frac{1}{\xi_{n}}\geq
t_{s+1}\left(\tau+\frac{t_{s}}{t_{s+1}}\right)\left(1-\sqrt{\phi}\right)$ (25)

are true. As $\beta\sim\tau$, we know that $\frac{t_{s}}{t_{s+1}}\to\phi$ as
$s\to\infty$. Moreover, one of $\frac{t_{s}}{t_{s+1}}$ and
$\frac{t_{s-1}}{t_{s}}$ is less than $\phi$, and another one is greater than
$\phi$ for every $s$ big enough. Now from (24) and (25) we have that

$d(t_{s})>t_{s}\sqrt{5}(1-\sqrt{\phi})=t_{s}C\text{\,\,\,\,\,\,or\,\,\,\,\,\,}d(t_{s+1})>t_{s+1}\sqrt{5}(1-\sqrt{\phi})=t_{s+1}C.$

In case $\frac{\eta_{s}}{\xi_{n}}\geq\sqrt{\tau}$ we get that
$\frac{\eta_{s+1}}{\xi_{n+1}}\geq\sqrt{\tau}$ and similarly we will come to
one of

$-d(q_{n})>q_{n}\sqrt{5}(1-\sqrt{\phi})=q_{n}C\text{\,\,\,\,\,\,or\,\,\,\,\,\,}-d(q_{n+1})>q_{n+1}\sqrt{5}(1-\sqrt{\phi})=q_{n+1}C$

being true by the same argument.

In every case we came to a contradiction, hence the assumption (17) was wrong.

##  4 Optimality of Theorem 1

Now let us prove the second statement of Theorem 1, that is the constant $C$
is optimal. We consider two irrational numbers, $\tau$ from (4) and $\theta$,
which we will define below.
First, for every $\varepsilon>0$ there exist $U,V\in\mathbb{Z}$, such that

$\left|V+\frac{U}{\tau}-\sqrt{\tau}\right|<\varepsilon.$

Define the sequence $X_{n}$ as

$X_{0}=U,\,\,\,X_{1}=V,\,\,\,X_{n+1}=X_{n}+X_{n-1}\text{\, for
every\,\,\,}n\geq 1.$

Then we have

$X_{n}=A\tau^{n}+B(-\tau)^{-n}\text{,\,\,\,\,\,where\,\,\,\,\,}A=\frac{\tau
V+U}{\tau+2}.$

As $A>0$, there exists $k\in\mathbb{Z_{+}}$, such that $X_{k-1},X_{k}\geqslant
1$. Let

$\frac{X_{k-1}}{X_{k}}=[0;b_{w},\ldots,b_{1}],\,\,\,\,\,\,\,\,b_{j}\in\mathbb{Z_{+}}$

and define $\theta$ as

$\theta=[0;b_{1},\ldots,b_{w},\overline{1}].$

We can show that

$\Bigl{|}\frac{1}{\psi_{\tau}(t)}-\frac{1}{\psi_{\theta}(t)}\Bigl{|}\leqslant(C+\varepsilon)t$
(26)

for all $t$ large enough.
Denominator $q_{n}$ of convergent $p_{n}/q_{n}$ to $\tau$ is equal to

$q_{n}=F_{n}=\frac{1}{\sqrt{5}}(\tau^{n}-(-\tau)^{-n}).$

Denominator $s_{n}$ of convergent $r_{n}/s_{n}$ to $\theta$ is equal to

$s_{n}=X_{n-n_{0}},\,\,\,\,\,n_{0}=k-w$

for $n$ large enough. Note, that

$\frac{X_{n}}{F_{n}}\sim
A\sqrt{5}=V+U\tau^{-1},\,\,\,\,\,|A\sqrt{5}-\sqrt{\tau}|<\varepsilon.$

We use

$q_{n}=F_{n}<X_{n}=s_{n+n_{0}}<F_{n+1}=q_{n+1}$

and

$\frac{1}{\psi_{\tau}(t)}=\frac{1}{\xi_{n}}=\frac{1}{||q_{n}\tau||}\sim\sqrt{5}F_{n},\,\,\,\,F_{n}\leqslant
t<F_{n+1},$
$\frac{1}{\psi_{\theta}(t)}=\frac{1}{\eta_{n}}=\frac{1}{||s_{n+n_{0}}\eta||}\sim\sqrt{5}X_{n},\,\,\,\,X_{n}\leqslant
t<X_{n+1}$

so that

$\frac{\xi_{n-1}}{\eta_{n}}\sim\frac{F_{n}}{X_{n-1}}=\sqrt{\tau}+O(\varepsilon),\,\,\,\,\,\frac{\eta_{n}}{\xi_{n}}\sim\frac{X_{n}}{F_{n}}=\sqrt{\tau}+O(\varepsilon),$

and we have

$\Bigl{|}\frac{1}{\psi_{\tau}(t)}-\frac{1}{\psi_{\theta}(t)}\Bigl{|}\sim
X_{n}\sqrt{5}\left|1-\frac{1}{\sqrt{\tau}}\right|,\,\,\,\,\,\,\,\,X_{n}\leqslant
t<F_{n+1},$
$\Bigl{|}\frac{1}{\psi_{\tau}(t)}-\frac{1}{\psi_{\theta}(t)}\Bigl{|}\sim
F_{n+1}\sqrt{5}\left|1-\frac{1}{\sqrt{\tau}}\right|,\,\,\,\,\,\,\,\,F_{n+1}\leqslant
t<X_{n+1}.\,\,\,\,\,\,\,\square$

## Acknowledgements

The author thanks Nikolay Moshchevitin for careful reading and helpful
comments.

N. Shulga is a scholarship holder of "BASIS" Foundation for Development of
Theoretical Physics and Mathematics and is supported in part by the Moebius
Contest Foundation for Young Scientists.

## References

  * [1] Perron, O.: Die Lehre von den Kettenbrüchen, B.G. Teubner, Leipzig und Berlin (1929)
  * [2] Schmidt, W.M.: Diophantine approximations, Lecture Notes Mathematics, vol. 785. Springer, Berlin (1980)
  * [3] Kan, I.D., Moshchevitin, N.G.: Approximations to two real numbers, Unif. Distrib. Theory 5(2), 79-86 (2010)
  * [4] Dubickas, A.: On rational approximations to two irrational numbers, J. Number Theory 177, 43-59 (2017)
  * [5] Moshchevitin, N.G.: Über die Funktionen des Irrationalitätsmaßes für zwei irrationale Zahlen, Arch. Math. 112 (2019), 161-168


[I-]Supplementary

# Scalable End-to-End RF Classification:
A Case Study on Undersized Dataset Regularization by Convolutional-MST

Khalid Youssef, Greg Schuette, Yubin Cai, Daisong Zhang, Yikun Huang, Yahya
Rahmat-Samii, and Louis-S. Bouchard K. Youssef, G. Schuette and L.-S. Bouchard
are with the Department of Chemistry and Biochemistry, University of
California, Los Angeles, 607 Charles E Young Drive East, Los Angeles,
California 90095-1569.
E-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>Y. Cai, D. Zhang, Y. Huang
and Y. Rahmat-Samii are with the Department of Electrical and Computer
Engineering, University of California, Los Angeles, 420 Westwood Plaza, Los
Angeles, California 90095-1594.
Email<EMAIL_ADDRESS>
###### Abstract

Unlike areas such as computer vision and speech recognition where
convolutional and recurrent neural networks-based approaches have proven
effective to the nature of the respective areas of application, deep learning
(DL) still lacks a general approach suitable for the unique nature and
challenges of RF systems such as radar, signals intelligence, electronic
warfare, and communications. Existing approaches face problems in robustness,
consistency, efficiency, repeatability and scalability. One of the main
challenges in RF sensing such as radar target identification is the difficulty
and cost of obtaining data. Hundreds to thousands of samples per class are
typically used when training for classifying signals into 2 to 12 classes with
reported accuracy ranging from 87% to 99%, where accuracy generally decreases
with more classes added. In this paper, we present a new DL approach based on
multistage training and demonstrate it on RF sensing signal classification. We
consistently achieve over 99% accuracy for up to 17 diverse classes using only
11 samples per class for training, yielding up to 35% improvement in accuracy
over standard DL approaches.

##  I Introduction

With applications in defense, retail, healthcare and tomography, [1, 2, 3, 4,
5] RF-based sensor systems that can detect, locate and identify targets at
long distances and under different weather conditions are being developed. One
example is ground-based and aircraft-mounted radars for the correct
identification of targets in the battlefield [6]. Another example is ground
penetrating radar, which has important applications to mineral resources
evaluation [7]. Other examples include advanced personnel screening imagers
[8], concealed weapon detection [9], through-the-wall imaging [10] and
vehicular access control using RFID [11]. The signal received, which is a
result of electromagnetic scattering [12], is difficult to interpret and model
using hand-engineered methods, and this is where blackbox-type approaches such
as deep learning (DL) can be useful.

In our previous work [13], we have presented a new multistage training (MST)
approach to DL for RF transmitter identification. MST is a highly
distributable structure-parallel DL approach that comprises multiple stages of
neural network ensembles, each consisting of several small networks, which
allows efficient utilization of Newton based second-order optimization, and
provides a highly effective data-driven regularization. It was originally
designed for high-fidelity denoising of magnetic resonance images (MRI) with
non-additive noise [14, 15]. The multistage approach allows “very early
stopping” at each individual stage, where a target error is assigned as a
stopping criterion in the first stage and is gradually decreased at successive
stages. By systematically assigning specific stopping criteria to each stage
we can control the speed of convergence in the system as a whole to optimize
overall performance and generalization, where a minimal error is reached in
the final stage without over-fitting. Our second-order MST has proven superior
to standard gradient descent based first-order convolutional and deep neural
networks (DNN), including MST trained using first-order optimization [13] .

Given that MST has proven superior to other DL methods in RF signal
classification [13], we investigated an extension of MST employing a
convolutional front-end as a feature extraction stage, for a fully automated
end-to-end implementation. We refer to the new implementation as convolutional
multistage training (C-MST), and we refer to the network architecture as
convolutional multistage network (C-MSN). We demonstrate our method on the
classification of radar-like signals, where a linearly polarized
electromagnetic wave illuminates an object which in turn creates a scattered
field detected by another antenna. In this experiment we consistently obtain
over 99% accuracy for 17 classes using only 11 samples per class for training,
as well as consistency, robustness, wall clock time, and scalability.

The dataset, which contains many acquisitions of scattered electromagnetic
waves acquired in the frequency domain using a network analyzer and measuring
the S-parameter for all 17 object classes, will be made publicly available.
The environment mimics radar-like detection of objects under ideal conditions,
i.e., without any clutter or motion. Our contribution in this article is
three-fold: 1) providing an overview of existing techniques, 2) extending our
method to include a convolutional front-end thereby enabling us to increase
the input size while maintaining the excellent generalization properties of
MST, and 3) providing a new benchmark dataset to help standardize the
comparison between different algorithms for RF classification. Our algorithm
is computationally efficient and allows incremental learning where only part
of the network needs to be trained when new targets are added. It can be run
on modest computers and may be a good candidate for deployment in the field,
for real-time, low-shot-number learning.

##  II Non-DL Methods for Target Identification

Target recognition algorithms operate on a measured target signature data for
comparison with the previously-derived computer representations of the targets
to provide an estimate of the target’s identity. The recognition process is
limited by noise in the radar measurements, errors in the generation of
signature reference data and the use of classifiers, which usually involves
design compromises. A conventional algorithm is template matching using cross-
correlation analysis [16, 17, 18, 19, 20, 21]. The accuracy of template
matching can be improved using statistical pattern recognition techniques
designed to determine the class or identity of a measured object by means of
the features extracted from the measured pattern or signature [22, 23, 24, 25,
26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36]. Features can be hand-engineered,
including polarization enhancement, resonant-frequency poles, multi-path
reflection signatures, target structure-induced modulations, microphone
effect, jet-engine modulations or features derived from a transform domain
representation of the signal [37]. Features can also be learned from the data
using techniques such as $k_{n}$-nearest-neighbor estimation or Fisher linear
discriminant analysis [38]. The use of features for recognition provides
advantages in reducing the requirements on the size of reference databases.
The extracted features are then compared to a database content to maximize the
target recognition performance using rule-based Euclidean distance or Bayesian
techniques [39, 40, 41, 42, 43, 44, 45, 46]. Distance-based methods invoke the
sum of all the distance measures for all features and the minimum distance
measure is the best assessment of the target’s identity. Unlike end-to-end
approaches that would ideally extract optimal features directly from the raw
data, manual feature selection can be limited by rough approximations and
subjectivity.

##  III Target Identification by DL

Thanks to the great success of DL methods in recent years in areas such as
computer vision and speech recognition, there has been a growing interest to
use DL for RF applications. However, unlike computer vision and speech
recognition where standard convolutional and recurrent neural network methods
have proven to be very effective to the nature of these applications, DL still
lacks a standardized approach suitable to the unique nature and challenges of
RF. While an ideal target identification method should be able to operate
directly on raw data in an end-to-end fashion, fully connected neural network
(FCN) implementations trained using standard algorithms are limited by the
number of inputs and the number of training samples they need in order to
achieve proper generalization and avoid over-fitting. In general, as the input
size increases, the number of required training samples increases [47].
Typically, the number of samples becomes impractical after the input size
exceeds only a few hundred inputs. However, digitized and demodulated RF
signals are complex-valued and may be thousands of samples long. Standard
regularization techniques such as L2-regularization help remedy the problem,
but only to a certain extent [48]. For this reason, feature extraction remains
a crucial step for a FCN to work, and the shortcomings of feature engineering
which can be limited by subjectivity and crude approximations inherently
remain.

The introduction of convolutional neural networks (CNN) delivered a tremendous
advantage to overcoming this DL problem by limiting the number of connections
from the inputs to the network [49, 47]. Convolutional neurons scan through
the entire input but are connected only to a few inputs at a time. This, in
addition to the introduction of DL-specific regularization techniques such as
dropout layers [50], revolutionized the field of computer vision. CNNs have
the ability to automatically extract features from raw inputs, which is a
crucial step towards end-to-end implementations. However, due to the immense
computational requirements, CNN training algorithms available today are
largely based on first-order gradient descent optimization, which poses
limitations on the performance and capabilities of CNNs and DL in general. For
example, it is not uncommon for an effective computer vision application to
require tens or hundreds of thousands of training samples, though it is
possible for humans to distinguish between different objects after only few
encounters [51, 52]. A recent study on using CNNs for RF identification shows
that several thousand samples per class are needed to achieve high accuracy
[53]. While customized approaches have been proposed for specific
applications, performance is very sensitive to hyperparameters and require
considerable expert effort to tune for proper parameter selection and tuning.
This makes DL implementations narrowly applicable, mainly limited to the exact
application and dataset at hand, which poses significant limitations on
repeatability. Furthermore, even with suitable hyperparameters, training with
different initial conditions, e.g. initial neuron weights and biases, can
cause large variations in performance. A typical approach to address this
problem is to train one or more network models several times with different
initial conditions and use the average output or vote of a committee of
networks as the final result [54, 55, 56], which can help improve performance
but is ultimately limited by the success rate of the individual networks.
Second-order Newton based optimization methods offer several advantages in
robustness to hyperparameters, efficiency, accuracy, convergence speed, and
require lower network complexity where they can achieve results that are
superior to first-order methods with fewer neurons [57, 58, 59, 60, 61, 62,
63]. However, second-order optimization is generally deemed unfeasible for
training large networks due to the intractable computational requirements of
traditional second-order methods.

Provided a large number of training samples is available, most shortcomings of
first-order based DL approaches are usually manageable. Unfortunately,
obtaining data in large quantities is not trivial in RF problems and can be a
difficult, time-consuming, and expensive process. Synthetically-generated data
is often used to compensate for the lack of real data, but it can only
approximate the fine details in the signals which poses a limitation on
accuracy and scalability. Generative adversarial networks (GAN) have also been
used to generate new training data [64], but GANs can face similar issues
depending on application. Recent work [65] studying the use of GANs for
synthetic aperture radar showed promise at improving the quality of
synthetically-generated data. The study also showed that GANs were difficult
to train and were not a perfect substitute for real data [65]. Artificial
neural networks (ANN) [66], support vector machines (SVM) [67] and Markov [68,
69] models have been studied to perform feature extraction and classifier
functions [70, 71]. Several attempts have been made to adapt standard DL
approaches to radar classification with varying degrees of success [72, 73,
74, 75, 76, 77, 78, 79]. Hundreds to thousands of samples per class were used
for training in order to classify 2 to 12 classes with reported accuracy
ranging from 87% to 99% where accuracy generally decreases with more classes.
Existing DL approaches face problems in robustness, consistency, efficiency,
repeatability and scalability.

##  IV Implementation of DL by Convolutional MST

This paper presents a new variation of MST with a convolutional front end, and
its application to the classification of radar-like signals. Given the
inherent regularization properties of CNNs for large inputs [47, 49] and their
robustness to translational variance, which has been demonstrated in radar ATR
[80], we have developed C-MST to further improve the performance and
applicability of our original MST method. The addition of CNNs not only helps
with translational invariance, but also helps in situations where training
data availability is scarce. Unlike our previous work where only part of the
RF signal is used (the onset) by the MST, the convolutional front-end allows
using the entire RF signal for a fully automated end-to-end implementation.

We test the new method regularization performance when trained with an
undersized dataset (11 samples per class) for a varying number of classes (8
to 17). We term the new method convolutional multi stage training (C-MST).
Here we describe the implementation of our method as well as conventional DL
methods used for benchmarking purposes. The C-MST method and its
implementation are described in Section IV-A below. The C-MST method is
validated against other DL methods: CNN, CNN Committee, FCN and FCN Committee,
as described below. In Section V we describe experimental methods for
acquiring the radar data.

All networks were implemented in MATLAB® R2019b using the Deep Learning
toolbox (The Math Works, Natick, MA) on a CentOS 7-based server featuring two
Xeon processors, each with 10 cores, and 128 GB RAM. Hyperparameters for all
methods were selected to maximize generalization. Architecture details and
hyperparameters values are provided in the appendix. The dataset will be made
available for download at https://to_be_determined

###  IV-A Convolutional MST (C-MST)

Figure 1:  C-MSN architecture for 17 object classes. A representive vector of
the output of each stage in response to one sample is shown for further
illustration. Actual stage outputs in response to all samples are shown in
Fig. 2

Twelve CNNs constitute the first stage of the C-MSN illustrated in Fig. 1.
Each CNN consists of 4 inner blocks and one outer block. Each inner block
contains a convolutional layer followed by a batch normalization layer, a ReLU
layer, and a pooling layer. The outer block consists of a dropout layer,
followed by two fully connected layers, a softmax layer, and a classification
layer. The architecture details are illustrated in Fig. 11 and the
hyperparameters for each layer are listed in Fig. 12. Each CNN is trained for
3 epochs, and the softmax layer outputs from all CNNs are concatenated to form
the input of stage 2. It is important to note that it is the outputs from the
softmax layer — not the output layer — that are passed on to stage 2. The
remaining C-MSN stages are similar to [13], where the original MST approach is
described in detail. Multiple identical FCNs are stacked together at each
stage, where each FCN is randomly assigned different initial conditions.
Herein, each FCN consists of two fully connected layers with 10 neurons in
each layer. The FCNs in each stage are separately trained for 3 epochs using
the second-order Levenberg-Marquardt algorithm; hyperparameters are listed in
Fig. 12. Each FCN has one output, and is trained to fire in response to one of
the object classes only, where 4 FCNs in each stage are assigned to each
object class. In the case of classifying 17 objects, the number of FCNs in
each stage is equal to 4 $\times$ 17, which yields 68 FCNs per stage. The
concatenated outputs of all FCNs in each stage are passed on as the input to
the next stage. In the final stage, the outputs from each group of 4 FCNs
corresponding to an object class are averaged together to obtain the final
response to that object class. Figure 2 shows an example that demonstrates the
gradual improvement in the outputs of subsequent C-MSN stages.

Figure 2:  Outputs from different C-MSN stages. The y-axis is the stacked
stage output number and the x-axis is the sample ID. Stage 1 has 12 networks
with 17 outputs per network, so the y-axis is from 1 to 204 (17x12). Stages 2
to 4 each have 68 networks with 1 output each. The final output represents a
confusion matrix. This figure illustrates the successive evolution of the
output, where accuracy gradually improves and noise decreases with each stage.

###  IV-B CNN

A comparison of C-MST against standard CNN was performed. A CNN identical to
the one described in section IV-A was trained for a maximum of 50 epochs using
the Adam optimizer [81]; Training/validation ratio was set to 70% and 30%
respectively. Early stoppage based on validation error was used. The weights
from the epoch with minimal validation error were selected as the final model
weights. Hyperparameters are listed in Fig. 12.

###  IV-C CNN Committee

Twelve CNNs identical to the one described in section IV-B are trained. Each
CNN is assigned different randomly-generated initial conditions. The final
output is the majority vote of all CNNs.

###  IV-D FCN

The FCN consists of 3 fully connected layers with 200, 200, and 17 neurons,
respectively. This is followed by a softmax layer and a classification layer.
The FCN was trained for 100 epochs using the Adam optimizer [81]. Early
stoppage based on validation error was used. The weights from the epoch with
minimal validation error were selected as the final model weights.
Hyperparameters are listed in Fig. 12. L2-regularization was used. The
training-to-validation ratio was set to 70% and 30%, respectively.

###  IV-E FCN Committee

The FCN committee used twelve FCNs, each identical to the one described in
section IV-D. Each FCN is assigned different randomly-generated initial
conditions. The final output is the majority vote of the outputs from each
FCN.

##  V Radar Experiments

###  V-A Data Collection

Figure 3:  (a) Vivaldi antenna model as used in the CST Studio Suite
simulation of the empty chamber. (b) S21 parameter traces for the empty
chamber. This plot compares simulated vs experimental data. The experimental
S21 parameter trace shown was collected on the first day of data collection.

Radar data was collected inside an anechoic chamber (band rejection 1 MHz - 10
GHz) using a pair of TSA900 900 MHz - 12 GHz PCB Vivaldi Antennas (RFSPACE
Inc., Atlanta, GA) connected to a vector network analyzer (VNA) model E5071C 9
kHz-8.5 GHz ENA Series (Agilent, Santa Clara, CA). The VNA was operated in S21
parameter mode with ports 1 (Tx) and 2 (Rx) connected to antennas 1 and 2.
Frequency sweeping was performed in the range 675 MHz - 8.5 GHz (corresponding
to wavelengths in free space: 0.44 m - 3.5 cm) with IF bandwidth 70 kHz, 1,600
points/trace and 512 averages/trace. Both Vivaldi antennas were mounted
vertically to make the readout from the Rx antenna sensitive to waves of the
same polarization as the Tx antenna. The experimental setup is shown in Fig.
4. Each trace (S21 parameter) was saved as a file with 1,600 real/imaginary
(complex) data points versus frequency on a linear amplitude scale from the
Smith Chart mode of the VNA.

Next, seventeen (17) objects were placed approximately 10 feet from the pair
of antennas, which were themselves 5 feet apart. The triangular configuration
was kept fixed and the objects were rotated through 3 angles (0, 45 and 90∘),
and the experiments were repeated 12 times (for each object and angle) on
different days over a total period of 3 months. All targets in subsequent
trials and angle rotations were placed at the same locations with intentional
positioning errors of up to 10 centimeters in-plane and 5∘ for the angles.
Other uncertainties in the measurements are due to the limited SNR and
possible drifts in the VNA S21 parameter calibration over the 3-month span, as
the VNA was calibrated only once on the first day of experiments. Photos of
the 17 objects and their positioning from the antennas’ perspective are shown
in Fig. 10.

Figure 4: Schematic of experimental setup. (a) Top view of the anechoic
chamber. (b) Side view of the chamber. The “target” indicated in the figure
represents any of the 17 different objects studied in this project; the 17
objects are shown in Fig. 10. Relative proportions are drawn to scale. Scale
bar, 1 ft.

###  V-B Data Processing

Since the purpose of this data is to assess end-to-end performance, minimal
pre-processing was conducted. Fingerprints used for training the neural
networks were created directly from the raw data. Each fingerprint consisted
of traces from 3 angles (0, 45 and 90∘), with each trace consisting of 1,600
real and imaginary values, for a total of 9,600 data points/fingerprint. Real
and imaginary components of each trace were normalized separately by
subtracting the mean and dividing by the standard deviation. The dataset
consisted of 204 fingerprints; 17 object classes with 12 fingerprints per
class. Different augmentation and pre-processing strategies can improve the
results for all aproaches. We omit such strategies herein in order to more
accurately assess the role of the DL approaches in the performance comparison.

###  V-C Analysis

Manual identification of targets from raw data proved difficult, as shown in
Fig. 5(b) and 6(b). Figure 5 shows log-magnitude plots of the S21 parameter
for all 17 objects in the 0∘ orientation. (The objects and our definition of
orientation are shown in Fig. 10.) In the low-frequency limit of the S21
parameter, most objects were indistinguishable. Several targets whose S21
parameters are shown in Fig. 5(a) displayed traces that can be manually
distinguished over the medium-to-high frequency range. Each of these objects
were metallic with large cross-sectional areas. This is expected from objects
with high conductivity and a large enough area to create conditions for high
reflectivity and scattering. On the other hand, several objects, shown in Fig.
5(b), have no readily-identifiable characteristics in the S21 parameter that
allow us to manually distinguish and identify them. We note that the high-
frequency limit (6-8 GHz) of our experiment was unreliable for manual target
identification due to high variance between experiments in cases of low radar
cross-sections (RCS).

(a) (b)

Figure 5:  S21 parameters (raw data, log-magnitude traces) for all 17 objects
at their respective 0∘ orientation. Traces show the raw data as acquired
without any post processing or averaging. (a) Several of the targets (objects
#1, 2, 4, 6, 13 and 15) have S21 parameters with obvious differences in the
log-magnitude plot. (b) Other targets (objects #3, 5, 7, 8, 9, 10, 11, 12, 14,
16 and 17) exhibit S21 parameters with no obvious visual differences.

Figure 6 examines the angle dependence of the S21 parameter for each object.
Certain objects, such as object #1 in Fig. 6(a), reflect the radio waves in
different ways depending on orientation. Therefore, once an object is
identified, its orientation can be determined from the log-magnitude plot of
the S21 parameter. This was the case for relatively large, metallic objects
that possess low symmetry and present a unique radar cross-section (RCS) at
each orientation. An exception was object #1, whose 0∘ and 90∘ orientations
have similar cross sections while exhibiting unique traces (S21 parameters in
log-magnitude mode). A possible explanation for this exception is that a
crimped copper seam was present on the right and left side of the object for
0∘ and 90∘ orientations, respectively. This seam likely reflects and scatters
radio waves differently depending on its orientation, allowing each
orientation’s trace to be manually distinguished. However, other objects
displayed little or no clear differences between orientations in the low-
frequency, low-noise region, as shown in Fig. 6(b). In our experiment, objects
with the least-distinguishable orientations tended to be those presenting
small RCS areas, regardless of material composition. Small RCSs imply weak
scattering intensity and overshadowing of the object’s characteristics in the
log-magnitude plot. Such objects with traces that cannot be manually
identified present a unique challenge to radar target identification. This is
generally the case for targets that present small RCS due to their sizes,
reflective properties or range of the measurement. This apparent difficulty
drives the need to develop DL algorithms for radar target identification.

(a) (b)

Figure 6:  S21 parameters for two targets (objects #1 and #17) at three
different orientations (the objects and a definition of orientation are
indicated in Fig. 10). No noise removal or averaging was performed on the
traces shown. (a) Object #1 is shown at 0, 45, and 90∘ orientations, each of
which is easily identifiable. Objects #2, 4, 6, and 13 (not shown here) also
demonstrate easily distinguishable orientations. (b) Object #17 is shown at 0,
45, and 90∘ orientations, each of which are indistinguishable. Orientations
for Objects #9, 10, 11, 14, and 16 (not shown here) were also visually
indistinguishable from their log-magnitude plots.

##  VI Results

(a)
(b)

Figure 7:  Accuracy trends (a) and standard deviation trends (b) for different
numbers of object classes are shown. Each point in the graph is calculated
from 60 trials.

Experiments were designed to test the aspects of accuracy, consistency,
robustness and wall clock time of the different DL approaches. As expected,
the FCN-based approaches were not suitable for an end-to-end implementation,
yet they were included for completeness. The CNN committee was the closest
competitor to C-MSN. Therefore, our discussion will focus on comparing CNN to
C-MSN results.

###  VI-A Accuracy

Leave-one-out cross validation (LOOCV) was used for testing the accuracy of
each approach. LOCCV is a special case of K-fold cross validation where one
sample is left out at a time, and K is equal to the number of samples [82].
All combinations of 11 samples per object for training / 1 sample per object
for validation were tested. The process was repeated 5 times for a total of 60
trials per accuracy measurement. The average accuracy of the 60 trials was
used to calculate each accuracy value shown in Fig. 7(a). The confusion
matrices for C-MSN, CNN, and CNN Committee are shown in Fig. 8.

###  VI-B Consistency

The standard deviation of the individual accuracies from the 60 trials was
calculated to assess the consistency of each approach. The results are shown
in Fig. 7(b). Zero standard deviation indicates that performance remains
consistent when training with different initial conditions. With the exception
of transfer learning which requires a previous model that has already been
trained, initial conditions are typically selected randomly even when
following specific initialization techniques. Consistency is a highly-
desirable feature that reduces the number of times a model must be trained in
order to achieve desirable performance, which increases implementation
efficiency. C-MST clearly excels in this aspect, outperforming the other
approaches by a large margin.

###  VI-C Robustness to Hyperparameters

While second-order optimizers are robust to hyperparameter settings, their
computational complexity is prohibitive for DL. Common DL algorithms instead
rely on first-order (gradient descent) based optimization, which is sensitive
to the settings of hyperparameters such as learning rate and the number of
layers and neurons in the network [83]. As a distributive training algorithm,
C-MST allows the efficient utilization of second-order optimization, making it
inherently robust to hyperparameter settings.

Robustness was tested by conducting several classification experiments with
varying number of objects. We note, as explained in section V-C, that certain
objects are more difficult to classify than others. For the purpose of this
experiment, the hyperparameters of each approach were first optimized for
classifying 17 objects, after which they were fixed and each approach was
tested with a decreasing number of objects, decreasing the level of
classification complexity. As seen in Fig. 7, C-MSN remains very consistent
whereas the performance of both CNN and CNN Committee degrades for less than
12 objects despite the decrease in difficulty.

Figure 8:  Confusion Matrices for C-MSN, CNN and CNN committee.

###  VI-D Wall Clock Time

Despite the seeming complexity of C-MST, which involves hundreds of individual
FCNs and CNNs, and although second-order training – which is typically
associated with high computational complexity – is used, the approach is
highly computationally efficient. The computational time in C-MST accelerates
virtually linearly with the number of computational cores in a processor, as
shown in Fig. 13. Given enough cores, C-MST can run in less wall clock time
than the CNN and CNN committee approaches. This is due to the highly-
distributable nature of C-MST, where individual networks in each stage can be
trained independently and in parallel. This is also due to the gradual
multistage convergence in C-MST as demonstrated in Fig. 2, where individual
networks require a very small number of training epochs relative to standard
approaches. Furthermore, the partial connectivity in C-MST typically results
in a relatively small number of parameters for each individual network in
inner stages. The average training time for C-MST was 20.2 seconds, while the
average training time for CNN was 23.9 seconds, and the average training time
for CNN Committee was 31.5 seconds.

##  VII Conclusion

Herein we presented a new DL approach for RF classification, and collected a
new benchmarking dataset for proof-of-concept radar applications. The
experiments conducted in this study confirm that, while standard CNN
approaches can work sufficiently well in some scenarios, their performance
drops dramatically as the classification complexity increases. Since
classification complexity increases with the number of classes, it is clear
that standard DL approaches do not scale effectively in such RF classification
applications. In contrast, C-MST is more stable and maintains significantly
higher performance across all experiments. Most notably, C-MST maintains a 99%
accuracy in the most difficult classification experiment conducted,
representing a 35% advantage over standard CNN approaches. Additionally, C-MST
is robust, computationally efficient and highly distributable. Therefore,
C-MST effectively scales with computational complexity and training time as
the number of classes increases. We propose C-MST as a scalable end-to-end
approach suitable to the nature and challenges of RF data.

We note that while the main purpose of this study is to compare accuracy and
generalization ability of DL approaches when trained with an undersized
training dataset, C-MST is equally applicable to pre-processed data and to
other radar modalities including 2-D modalities such as synthetic aperture
radar, where the input dimensions of the front end CNN stage can be adjusted
accordingly. Hyperparameters such as number of stages and number of networks
per stage can also be further adjusted according to the problem complexity.

Furthermore, our newly collected benchmarking dataset will be made publicly
available to enable other groups to validate their work independently when
applying our method to other challenging radar classification problems. Data
collected from pulsed radar in the field generally include additional factors
such as polarization, background clutter, time-domain acquisition, variable
target-to-radar distance (range), moving targets, additional sources of noise
and interference and radar jamming. The effects of radar clutter were not
included here, as the main point of the study was to compare classification
accuracy of the new algorithm (C-MSN) to existing, state-of-the-art algorithms
under identical conditions. The numerous advantages offered by our approach
will help improve RF-based signal classification performance under these
challenging scenarios.

## References

  * [1] S. M. Patole, M. Torlak, D. Wang, and M. Ali, “Automotive radars: A review of signal processing techniques,” _IEEE Signal Processing Magazine_ , vol. 34, no. 2, pp. 22–35, 2017.
  * [2] K. W. O’Haver, C. K. Barker, G. D. Dockery, and J. D. Huffaker, “Radar development for air and missile defense,” _Johns Hopkins APL Technical Digest_ , vol. 34, no. 2, pp. 140–153, 2018.
  * [3] J. Muñoz-Ferreras, Z. Peng, R. Gómez-García, and C. Li, “Review on advanced short-range multimode continuous-wave radar architectures for healthcare applications,” _IEEE Journal of Electromagnetics, RF and Microwaves in Medicine and Biology_ , vol. 1, no. 1, pp. 14–25, 2017.
  * [4] N. Joshi, M. Baumann, A. Ehammer, R. Fensholt, K. Grogan, P. Hostert, M. R. Jepsen, T. Kuemmerle, P. Meyfroidt, E. T. Mitchard _et al._ , “A review of the application of optical and radar remote sensing data fusion to land use mapping and monitoring,” _Remote Sensing_ , vol. 8, no. 1, p. 70, 2016\.
  * [5] W. W.-L. Lai, X. Derobert, and P. Annan, “A review of ground penetrating radar application in civil engineering: A 30-year journey from locating and testing to imaging and diagnosis,” _NDT & E International_, vol. 96, pp. 58–78, 2018.
  * [6] P. Tait, _Introduction to Radar Target Recognition_. IET, 2005, vol. 18.
  * [7] J. Francke, “A review of selected ground penetrating radar applications to mineral resource evaluations,” _Journal of Applied Geophysics_ , vol. 81, pp. 29–37, 2012.
  * [8] S. Ahmed, A. Genghammer, A. Schiessl, and L.-P. Schmidt, “Fully electronic e-band personnel imager of 2 m2 aperture based on a multistatic architecture,” _IEEE Transactions on Microwave Theory and Techniques_ , vol. 61, no. 1, pp. 651–657, 2012.
  * [9] A. Agurto, Y. Li, G. Tian, N. Bowring, and S. Lockwood, “A review of concealed weapon detection and research in perspective,” in _2007 IEEE International Conference on Networking, Sensing and Control_. IEEE, 2007, pp. 443–448.
  * [10] S. Sadeghi, K. Mohammadpour-Aghdam, R. Faraji-Dana, and R. Burkholder, “A DORT-uniform diffraction tomography algorithm for through-the-wall imaging,” _IEEE Transactions on Antennas and Propagation_ , vol. 68, no. 4, pp. 3176–3183, 2019.
  * [11] D. Almanza-Ojeda, A. Hernandez-Gutierrez, and M. Ibarra-Manzano, “Design and implementation of a vehicular access control using RFID,” in _2006 Multiconference on Electronics and Photonics_. IEEE, 2006, pp. 223–225.
  * [12] M. Mishchenko, “Measurement and modeling of electromagnetic scattering by particles and particle groups,” _Polarimetry of Stars and Planetary Systems, edited by: Kolokolova, L., Hough, J., and Levasseur-Regourd, A.-C., Cambridge University Press, Cambridge_ , pp. 13–34, 2015.
  * [13] K. Youssef, L.-S. Bouchard, K. Haigh, J. Silovsky, B. Thapa, and C. Valk, “Machine learning approach to RF transmitter identification,” _IEEE Journal of Radio Frequency Identification_ , vol. 2, pp. 197–205, 2018.
  * [14] K. Youssef, N. N. Jarenwattananon, and L. Bouchard, “Feature-preserving noise removal,” _IEEE Transactions on Medical Imaging_ , vol. 34, no. 9, pp. 1822–1829, 2015.
  * [15] K. Youssef and L. Bouchard, “Feature-preserving noise removal,” 2014, US Patent US9953246B2. [Online]. Available: https://patents.google.com/patent/US9953246B2
  * [16] D. Pearson, “Introduction,” in _Image Processing_ , D. Pearson, Ed. London: McGraw-Hill, 1991, pp. 1–14.
  * [17] L. Novak, G. Owirka, and W. Brower, “Performance of 10- and 20-target MSE classifiers,” _IEEE Transactions on Aerospace and Electronic Systems_ , vol. 36, no. 4, pp. 1279–1289, Oct 2000.
  * [18] R. Heiden, “Aircraft recognition with radar range profiles,” Ph.D. dissertation, Doctoral Thesis, University of Amsterdam, Chapter 5, 1998.
  * [19] R. Heiden and F. Groen, “Distance based range profile classification techniques for aircraft recognition by radar-a comparison on real radar data,” in _AGARD Conference Proceedings on Tactical Aerospace C3I in Coming Years-Papers presented at the Mission Systems Panel 3rd Symposium held in Lisbon, Portugal from 15th May to 18th May 1995, 17/1-17/7_. Advisory Group for Aerospace Research and Development (AGARD; NATO), 1995.
  * [20] S. Hudson and D. Psaltis, “Correlation filters for aircraft identification from radar range profiles,” _IEEE Transactions on Aerospace and Electronic Systems_ , vol. 29, no. 3, pp. 741–748, July 1993.
  * [21] J. Tang and Z. Zhu, “Comparison study on high resolution radar target recognition,” in _Proceedings of the IEEE 1996 National Aerospace and Electronics Conference NAECON 1996_ , vol. 1, May 1996, pp. 250–253 vol.1.
  * [22] R. Boyle and R. Thomas, _Computer Vision: A First Course_. Blackwell Scientific Publications, Ltd., 1988.
  * [23] E. Davies, “Machine vision: Theory, algorithms, and practicalities,” 2004.
  * [24] L. Roberts, “Machine perception of three-dimensional solids,” 1963.
  * [25] J. Tippett, _Optical and Electro-optical Information Processing_. Massachusetts Institute of Technology Press, 1965, 1965.
  * [26] D. Vernon, “Machine vision: Automated visual inspection and robot vision,” 1991\.
  * [27] A. Jain, _Fundamentals of Digital Image Processing_. Prentice Hall (Englewood Cliffs, NJ), 1989.
  * [28] K. Rosenbach and J. Schiller, “Identification of aircraft on the basis of 2-D radar images,” in _Proceedings International Radar Conference_ , May 1995, pp. 405–409.
  * [29] F. Zernike, “Beugungstheorie des schneidenver-fahrens und seiner verbesserten form, der phasenkontrastmethode,” _Physica_ , vol. 1, no. 7-12, pp. 689–704, 1934.
  * [30] T. Kohonen, Ed., _Self-Organizing Maps_. Berlin, Heidelberg: Springer-Verlag, 1997.
  * [31] R. Hu and Z. Zhu, “Researches on radar target classification based on high resolution range profiles,” in _Proceedings of the IEEE 1997 National Aerospace and Electronics Conference. NAECON 1997_ , vol. 2, July 1997, pp. 951–955 vol.2.
  * [32] B. Pei and Z. Bao, “Radar target recognition based on peak location of HRR profile and HMMs classifiers,” _IET Conference Proceedings_ , pp. 414–418(4), January 2002.
  * [33] A. Maki, K. Fukui, Y. Kawawada, and M. Kiya, “Automatic ship identification in ISAR imagery: an on-line system using CMSM,” in _Proceedings of the 2002 IEEE Radar Conference (IEEE Cat. No.02CH37322)_ , April 2002, pp. 206–211.
  * [34] M. Cassabaum, J. Rodriguez, J. Riddle, and D. Waagen, “Feature analysis using millimeter-wave real beam and doppler beam sharpening techniques,” in _Proceedings Fifth IEEE Southwest Symposium on Image Analysis and Interpretation_ , April 2002, pp. 101–105.
  * [35] S. He, W. Zhang, and G. Guo, “High range resolution MMW radar target recognition approaches with application,” in _Proceedings of the IEEE 1996 National Aerospace and Electronics Conference NAECON 1996_ , vol. 1, May 1996, pp. 192–195 vol.1.
  * [36] C. Nieuwoudt and E. Botha, “Relative performance of correlation-based and feature-based classifiers of aircraft using radar range profiles,” in _Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No.98EX170)_ , vol. 2, Aug 1998, pp. 1828–1832 vol.2.
  * [37] B. Borden, _Radar Imaging of Airborne Targets: A Primer for Applied Mathematicians and Physicists_. CRC Press, 1999.
  * [38] R. Duda, P. Hart, and D. Stork, _Pattern classification_. John Wiley & Sons, 2012.
  * [39] T. Bayes, “An essay towards solving a problem in the doctrine of chances,” _Philosophical Transactions of the Royal Society_ , vol. 53, pp. 370–415, 1794.
  * [40] M. DeGroot, _Optimal Statistical Decisions_. John Wiley & Sons, 2005, vol. 82.
  * [41] D. Sivia and J. Skilling, _Data Analysis: A Bayesian Tutorial_. OUP Oxford, 2006.
  * [42] A. Gelman, J. Carlin, H. Stern, D. Dunson, A. Vehtari, and D. Rubin, _Bayesian Data Analysis_. Chapman and Hall (CRC Press), 2013.
  * [43] P. Lee, _Bayesian Statistics: An Introduction_. John Wiley, 2012.
  * [44] G. D’Agostini, _Bayesian Reasoning in Data Analysis: A Critical Introduction_. World Scientific, 2003.
  * [45] J. Yan, X. Feng, and P. Huang, “Bayes-optimality based feature transform for high resolution range profile identification,” in _6th International Conference on Signal Processing, 2002._ , vol. 2, Aug 2002, pp. 1396–1398 vol.2.
  * [46] D. Zhou, L. Wu, and G. Liu, “Bayesian classifier based on discretized continuous feature space,” in _ICSP ’98. 1998 Fourth International Conference on Signal Processing (Cat. No.98TH8344)_ , vol. 2, Oct 1998, pp. 1225–1228 vol.2.
  * [47] Y. LeCun and Y. Bengio, _Convolutional networks for images, speech, and time-series_. MIT Press, 1995.
  * [48] A. Y. Ng, “Feature selection, ${L}_{1}$ vs. ${L}_{2}$ regularization, and rotational invariance,” in _Proceedings of the Twenty-First International Conference on Machine Learning_ , ser. ICML ’04. New York, NY, USA: Association for Computing Machinery, 2004, p. 78. [Online]. Available: https://doi.org/10.1145/1015330.1015435
  * [49] J. Denker, R. Howard, L. Jackel, and Y. LeCun, “Hierarchical constrained automatic learning network for character recognition,” January 1990, US Patent US5058179A. [Online]. Available: https://patents.google.com/patent/US5058179A
  * [50] G. Hinton, A. Krizhevsky, I. Sutskever, and N. Srivastva, “System and method for addressing overfitting in a neural network,” 2012, US Patent US9406017B2. [Online]. Available: https://patents.google.com/patent/US9406017B2
  * [51] F. Fleuret, T. Li, C. Dubout, E. K. Wampler, S. Yantis, and D. Geman, “Comparing machines and humans on a visual categorization test,” _Proceedings of the National Academy of Sciences_ , vol. 108, no. 43, pp. 17 621–17 625, 2011.
  * [52] S. Dodge and L. Karam, “A study and comparison of human and deep learning recognition performance under visual distortions,” in _2017 26th International Conference on Computer Communication and Networks (ICCCN)_. IEEE, 2017, pp. 1–7.
  * [53] T. Oyedare and J.-M. Park, “Estimating the required training dataset size for transmitter classification using deep learning,” in _International Symposium on Dynamic Spectrum Access Networks_. IEEE, 2019.
  * [54] D. C. Ciresan, U. Meier, L. M. Gambardella, and J. Schmidhuber, “Convolutional neural network committees for handwritten character classification,” in _2011 International Conference on Document Analysis and Recognition_ , 2011, pp. 1135–1139.
  * [55] D. Cireşan, U. Meier, J. Masci, and J. Schmidhuber, “A committee of neural networks for traffic sign classification,” in _The 2011 International Joint Conference on Neural Networks_ , 2011, pp. 1918–1921.
  * [56] Z.-Q. Zhao, D.-S. Huang, and B.-Y. Sun, “Human face recognition based on multi-features using neural networks committee,” _Pattern Recognition Letters_ , vol. 25, no. 12, pp. 1351–1358, 2004.
  * [57] B. M. Ozyildirim and M. Kiran, “Do optimization methods in deep learning applications matter?” _arXiv e-prints_ , February 2020.
  * [58] Q. V. Le, J. Ngiam, A. Coates, A. Lahiri, B. Prochnow, and A. Y. Ng, “On optimization methods for deep learning,” in _Proceedings of the 28th International Conference on Machine Learning_ , ser. ICML’11. Madison, WI, USA: Omnipress, 2011, p. 265–272.
  * [59] R. Battiti, “First- and second-order methods for learning: Between steepest descent and newton’s method,” _Neural Computation_ , vol. 4, no. 2, pp. 141–166, March 1992.
  * [60] H. H. Tan and K. H. Lim, “Review of second-order optimization techniques in artificial neural networks backpropagation,” _IOP Conference Series: Materials Science and Engineering_ , vol. 495, June 2019. [Online]. Available: https://doi.org/10.1088%2F1757-899x%2F495%2F1%2F012003
  * [61] Y. A. LeCun, L. Bottou, G. B. Orr, and K.-R. Müller, _Neural Networks: Tricks of the Trade_ , 2nd ed. Springer, Berlin, Heidelberg, 2012, ch. 1, pp. 9–48.
  * [62] B. M. Wilamowski and H. Yu, “Improved computation for levenberg–marquardt training,” _IEEE Transactions on Neural Networks_ , vol. 21, no. 6, pp. 930–937, June 2010.
  * [63] H. Yu and B. Wilamowski, “Levenberg-marquardt training,” in _The Industrial Electronics Handbook_ , 2nd ed. CRC Press, January 2011, vol. 5, ch. 12, pp. 12–1 – 12–16.
  * [64] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in _Advances in Neural Information Processing Systems_ , 2014, pp. 2672–2680.
  * [65] J. Guo, B. Lei, C. Ding, and Y. Zhang, “Synthetic aperture radar image synthesis by using generative adversarial nets,” _IEEE Geoscience and Remote Sensing Letters_ , vol. 14, no. 7, pp. 1111–1115, 2017.
  * [66] D. Nandagopal, N. Martin, R. Johnson, P. Lozo, and M. Palaniswami, “Performance of radar target recognition schemes using neural networks - a comparative study,” in _Proceedings of ICASSP ’94. IEEE International Conference on Acoustics, Speech and Signal Processing_ , vol. ii, April 1994, pp. II/641–II/644 vol.2.
  * [67] Z. Li, Z. Weida, and J. Licheng, “Radar target recognition based on support vector machine,” in _WCC 2000 - ICSP 2000. 2000 5th International Conference on Signal Processing Proceedings. 16th World Computer Congress 2000_ , vol. 3, Aug 2000, pp. 1453–1456 vol.3.
  * [68] P. Bharadwaj, P. Runkle, L. Carin, J. A. Berrie, and J. A. Hughes, “Multiaspect classification of airborne targets via physics-based HMMs and matching pursuits,” _IEEE Transactions on Aerospace and Electronic Systems_ , vol. 37, no. 2, pp. 595–606, April 2001.
  * [69] M. Jahangir, K. Ponting, and J. O’Loghlen, “Correction to ’robust doppler classification technique based on hidden Markov models’,” _IEE Proceedings - Radar, Sonar and Navigation_ , vol. 150, no. 5, p. 387, Oct 2003.
  * [70] T. Cooke, N. Redding, J. Schroeder, and J. Zhang, “Target discrimination in complex synthetic aperture radar imagery,” in _Conference Record of the Thirty-Fourth Asilomar Conference on Signals, Systems and Computers (Cat. No.00CH37154)_ , vol. 2, Oct 2000, pp. 1540–1544 vol.2.
  * [71] P. Han, R. Wu, Y. Wang, and Z. Wang, “An efficient SAR ATR approach,” in _2003 IEEE International Conference on Acoustics, Speech, and Signal Processing, 2003. Proceedings. (ICASSP ’03)._ , vol. 2, 6—10 April 2003, pp. II–429—32.
  * [72] M. Seyfioğlu, A. Özbayoğlu, and S. Gürbüz, “Deep convolutional autoencoder for radar-based classification of similar aided and unaided human activities,” _IEEE Transactions on Aerospace and Electronic Systems_ , vol. 54, no. 4, pp. 1709–1723, 2018.
  * [73] D. Avrahami, M. Patel, Y. Yamaura, and S. Kratz, “Below the surface: Unobtrusive activity recognition for work surfaces using RF-radar sensing,” in _23rd International Conference on Intelligent User Interfaces_ , ser. IUI ’18. ACM, 2018, pp. 439–451.
  * [74] B. Jokanović and M. Amin, “Fall detection using deep learning in range-doppler radars,” _IEEE Transactions on Aerospace and Electronic Systems_ , vol. 54, no. 1, pp. 180–189, 2018.
  * [75] S. Chen, H. Wang, F. Xu, and Y. Jin, “Target classification using the deep convolutional networks for SAR images,” _IEEE Transactions on Geoscience and Remote Sensing_ , vol. 54, no. 8, pp. 4806–4817, 2016.
  * [76] D. Morgan, “Deep convolutional neural networks for ATR from SAR imagery,” in _Algorithms for Synthetic Aperture Radar Imagery XXII_ , vol. 9475. International Society for Optics and Photonics, 2015, p. 94750F.
  * [77] J. Lien, N. Gillian, M. E. Karagozler, P. Amihood, C. Schwesig, E. Olson, H. Raja, and I. Poupyrev, “Soli: Ubiquitous gesture sensing with millimeter wave radar,” _ACM Transactions on Graphics (TOG)_ , vol. 35, no. 4, p. 142, 2016.
  * [78] S. Wang, J. Song, J. Lien, I. Poupyrev, and O. Hilliges, “Interacting with soli: Exploring fine-grained dynamic gesture recognition in the radio-frequency spectrum,” in _Proceedings of the 29th Annual Symposium on User Interface Software and Technology_. ACM, 2016, pp. 851–860.
  * [79] X. Cao, X. Zhang, R. Togneri, and Y. Yu, “Underwater target classification at greater depths using deep neural network with joint multiple-domain feature,” _IET Radar, Sonar & Navigation_, vol. 13, no. 3, pp. 484–491, 2018.
  * [80] D. Malmgren-Hansen, R. Engholm, and M. O. Pedersen, “Training convolutional neural networks for translational invariance on sar atr,” in _Proceedings of EUSAR 2016: 11th European Conference on Synthetic Aperture Radar_. VDE, 2016, pp. 1–4.
  * [81] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” 2017.
  * [82] A. Elisseeff, M. Pontil _et al._ , “Leave-one-out error and stability of learning algorithms with applications,” _NATO Science Series, III: Computer and Systems Sciences_ , vol. 190, pp. 111–130, 2003.
  * [83] P. Xu, F. Roosta, and M. W. Mahoney, “Second-order optimization for non-convex machine learning: An empirical study,” in _Proceedings of the 2020 SIAM International Conference on Data Mining_. SIAM, 2020, pp. 199–207.

## Photos of Objects and Chamber

Seventeen objects were collected from engineering and chemistry laboratories
at UCLA for use as radar targets. The objects were selected to present a
diversity of sizes, shapes and material composition. The 47.0 by 30.5 cm
cardboard platform seen in the pictures provide a scale for each object’s
size. All 17 targets were placed in the radar’s path and rotated through 3
different angles (0, 45 and 90∘). Photos of all 17 rotated objects are shown
in Fig. 10, as seen from the perspective of the radar. Photos of the anechoic
chamber setup are shown in Fig. 9.

Figure 9:  Anechoic chamber and experimental setup. The photo outlined in
green shows the two Vivaldi antennas in the lower left and right corners, both
oriented vertically, as used during experimental data collection. The photo
outlined in red shows the VNA used for all S21 readouts (Agilent model E5071C
9 kHz - 8.5 GHz ENA Series). The photo outlined in blue shows the radar’s
position relative to the cardboard platform. Figure 10:  The seventeen targets
are shown on the platform. Object size can be gauged by the 47 cm wide and
30.5 cm deep cardboard platform. 0, 45, and 90∘ orientations are shown in
(ii), (iii), and (iv), respectively. The red arrows (i) depict the
orientations on the cardboard. Unless otherwise noted, photos are taken from
the radar’s perspective at a 45∘ orientation. The objects vary in size and
composition, affecting rRCS. (0) Platform: Roughly 60∘ from the radar’s
perspective. (1) Object 1: Empty metal (copper) box with hole on top face. (2)
Object 2: Metal box (closed cover) with home-built circuit. (3a) Object 3:
Plastic toolbox (closed). (3b) Object 3: Open; data was collected with the
toolbox closed. (4) Object 4: Metal box (open cover) with custom circuit. (5a)
Object 5: Plastic box (closed). (5b) Object 5: Open; data was collected with
the box closed. (6) Object 6: DC power supply (metal cover). (7) Object 7:
Front cover of a power amplifier (metal) at 90∘ orientation. (8) Object 8:
Rogers duroid laminate (copper) at 90∘ orientation. (9) Object 9: Data
transfer switch box (with plastic cover). (10) Object 10: Variable capacitor
box (metal cover). (11) Object 11: Data transfer switch box (with metal
cover). (12) Object 12: Port converter (metallic). (13) Object 13: Data
transfer switch box (with plastic and metal cover). (14) Object 14: Vise
(metal) at 0∘ orientation. (15) Object 15: Metal box. (16) Object 16:
Chemistry hotplate stirrer. (17) Object 17: Black & Decker drill at 0∘
orientation.

## Signal Strength

Twelve (12) traces were recorded for each of the seventeen (17) objects, and
each of the three (3) orientations per object. Additionally, 112 traces of the
empty anechoic chamber were recorded under otherwise equal conditions. Each
trace (S21 parameter) was recorded as a string of complex numbers (real,
imaginary) representing the complex-valued signal amplitude as a function of
sweep frequency. Signal-to-noise ratios (SNR) and relative radar cross
sections (rRCS) were calculated for each of the 17 objects and their three
orientations using this data and tabulated below (Table I). Radar cross
sections (RCS) are provided as dimensionless numbers between 0 and 1 (rRCS).
SNR values reflect the maximum amplitude in the frequency domain of the S21
parameter, whereas rRCS values reflect the total signal over the same domain.
This is analogous to the pulse radar echo amplitude. Uncertainties in the
values represent sample standard deviations, which were calculated using the
12 traces for each object (and orientation).

From each complex-valued trace for each object (and orientation), the
magnitude of the complex data was computed and stored in vectors. Each trace
vector was 1,600 points long, corresponding to linearly spaced frequency
values in the range 675 MHz - 8.5 GHz. For the empty anechoic chamber data,
all 112 traces were averaged to provide a clean (low noise) trace. This low-
noise trace was used to subtract the background signal for each object (and
orientation) in order to produce traces whose features only reflect the
characteristics of an object.

From the background-subtracted traces, SNR was calculated as follows. The
location of maximum signal was identified and 100 nearby points were averaged
to reduce noise. For each trace, a flat region was identified in order to
estimate the noise as the standard deviation of 100 points taken in the flat
region. SNR was computed as the above-mentioned signal strength divided by the
standard deviation of the flat region. The resulting value was inserted into
Eq. (1),

$\left.\text{SNR}\right|_{\text{dB}}=20\log_{10}\left(\frac{\text{S21}_{\text{target
removed}}}{\text{S21}_{\text{with target}}}\right).$ (1)

to yield a value in dB. rRCS values were estimated from the background-
subtracted traces by taking the area under each curve, which represents the
total radar signal (integrated across all frequencies). The integral was
computed in MATLAB using the trapezoidal rule. This area represents the “pulse
amplitude” in a pulsed radar echo experiment. These pulse amplitudes were then
averaged for each object-orientation combination and normalized to the largest
object-orientation average to yield a number between 0 and 1 (what we call the
“relative” RCS).

Object |  SNR, 0∘ |  SNR, 45∘ (dB) |  SNR, 90∘ (dB) |  rRCS, 0∘ |  rRCS, 45∘ |  rRCS, 90∘
---|---|---|---|---|---|---
1 | 20(10) | 13(8) | 19(9) | 0.9(3) | 0.10(2) | 1.0(4)
2 | 14(7) | 14(7) | 8(7) | 0.34(5) | 0.10(2) | 0.4(1)
3 | 13(7) | 14(8) | 12(7) | 0.10(2) | 0.10(2) | 0.14(2)
4 | 12(7) | 14(7) | 11(6) | 0.22(6) | 0.09(3) | 0.17(3)
5 | 14(8) | 13(7) | 14(8) | 0.10(2) | 0.10(2) | 0.17(4)
6 | 12(8) | 13(7) | 10(6) | 0.23(3) | 0.10(3) | 0.17(2)
7 | 13(8) | 13(8) | 14(9) | 0.10(3) | 0.10(2) | 0.5(2)
8 | 12(7) | 14(8) | 13(8) | 0.10(3) | 0.10(3) | 0.20(3)
9 | 12(6) | 14(7) | 10(6) | 0.10(3) | 0.10(3) | 0.11(3)
10 | 10(6) | 13(7) | 12(7) | 0.12(3) | 0.09(3) | 0.09(3)
11 | 10(6) | 14(8) | 11(6) | 0.12(2) | 0.09(3) | 0.10(3)
12 | 10(6) | 13(7) | 10(6) | 0.14(3) | 0.09(3) | 0.11(3)
13 | 11(7) | 12(7) | 10(6) | 0.23(2) | 0.10(3) | 0.11(3)
14 | 11(6) | 13(7) | 12(7) | 0.11(3) | 0.09(3) | 0.10(3)
15 | 14(8) | 11(6) | 11(7) | 0.44(5) | 0.10(3) | 0.14(2)
16 | 11(7) | 13(7) | 11(6) | 0.10(3) | 0.10(3) | 0.10(3)
17 | 12(7) | 13(7) | 13(7) | 0.10(3) | 0.10(3) | 0.10(3)
TABLE I:  SNR and rRCS values for each object (1–17) and their three
orientations. The standard deviation of each signal is shown in parentheses.

## CNN Architecture and Hyperparameters

The detailed CNN architecture and hyperparameters are shown in Fig. 11. The
hyperparameters used for Adam and Levenberg-Marquardt are provided in Fig. 12.

Figure 11:  CNN architecture and hyperparameters.  Figure 12:  Levenberg-
Marquardt and Adam parameters.

## Wall clock time

Fig. 13 shows wall clock time and speedup relative to 1 CPU core. Measurements
were obtained by activating different number of CPU cores and measuring the
training time.

Figure 13:  Wall clock time and acceleration.

## Detailed Results

Full results from each individual iteration are shown in Fig. 14. These are
the results used to calculate the values of the average accuracy and standard
deviation in Fig. 7.

Figure 14:  Detailed classification results obtained by different machine
learning algorithms.