# A note on the GFF with one free boundary condition

Yong Han111College of Mathematics and Statistics, Shenzhen University,
Shenzhen 518060, Guandong, China<EMAIL_ADDRESS>Yuefei Wang222College of
Mathematics and Statistics, Shenzhen University, Shenzhen 518060, Guandong,
China and Institute of Mathematics, Academy of Mathematics and Systems
Sciences, Chinese Academy of Sciences, Beijing 100190, China.
<EMAIL_ADDRESS>Zipeng Wang 333College of Mathematics and Statistics,
Chongqing University, Chongqing, 401331, China<EMAIL_ADDRESS>
###### Abstract

In this note, we shall prove an explicit formula on the probability of the
level line of the Gaussian Free Field (GFF) with mixed boundary condition
terminating at the free boundary, which generalizes the results of GFF with
Dirichlet boundary condition.

##  1 Background

Let us first recall some basic knowledge on Loewner chains ([Law05, LSW01,
RS05]). Given a continuous function $W$ from $\mathbb{R}_{+}$ to $\mathbb{R}$:
$t\rightarrow W_{t}$, the Loewner’s ordinary differential equation (ODE) is
defined by

$\displaystyle\partial_{t}g_{t}(z)=\frac{2}{g_{t}(z)-W_{t}},\quad
g_{0}(z)=z\in\mathbb{H}.$ (1.1)

For $z\in\overline{\mathbb{H}}\setminus\\{0\\}$, let $\tau(z)$ be the blowup
time of the Loewner’s ODE above. For $t\geq 0$, let $K_{t}$ be the set of
$z\in\mathbb{H}$ with $\tau(z)\leq t$. Then the collection
$\\{K_{t}:t\in\mathbb{R}_{+}\\}$ forms an increasing family of compact subsets
of $\overline{\mathbb{H}}$ such that for each $t$, $g_{t}$ is the unique
conformal map from $\mathbb{H}\setminus K_{t}$ onto $\mathbb{H}$ satisfying
$g_{t}(z)=z+2t/z+O(1/|z|^{2})$ as $z\rightarrow\infty$. The conformal maps
$(g_{t})_{t\geq 0}$ (or the compact hulls $(K_{t})_{t\geq 0}$ ) are called the
Loewner chain driven by $W$. For $\kappa>0$ and $B_{t}$ being the one
dimensional standard Brownian motion, the random Loewner chain driven by
$W_{t}=\sqrt{\kappa}B_{t}$ is called the Schramm Loewner Evolution
($\mathrm{SLE}_{\kappa}$) process. The $\mathrm{SLE}_{\kappa}$ process is
generated by a curve $\gamma$ from $0$ to $\infty$, i.e., for each $t\geq 0$,
$\mathbb{H}\setminus K_{t}$ is the unbounded component of
$\mathbb{H}\setminus\gamma[0,t]$. There is an variant of
$\mathrm{SLE}_{\kappa}$ which is known as the $\mathrm{SLE}_{\kappa}(\rho)$
process. For $\rho\geq 0$, and $y\leq x\in\mathbb{R}$, consider the following
Stochastic differential equation system:

$\displaystyle dW_{t}=\sqrt{\kappa}dB_{t}+\frac{\rho}{W_{t}-V_{t}}dt,\quad
dV_{t}=\frac{2}{V_{t}-W_{t}},\quad W_{0}=x,\quad V_{0}=y.$ (1.2)

The random Loewner chain driven by $W$ defined in (1.2) is called the
$\mathrm{SLE}_{\kappa}(\rho)$ process from $x$ to $\infty$ with force point
$y$. More properties of $\mathrm{SLE}_{\kappa}(\rho)$ process can be found in
[MS16].

Next, let us give a brief introduction to the GFF ([DS11, MS16]). For
$a\in\mathbb{R}$, the Green function with Dirichlet boundary condition on
$(a,\infty)$ and Neumann boundary condition (also called free boundary
condition ) on $(-\infty,a)$ is given by

$\displaystyle
G_{\mathrm{mix}}(z,w):=\frac{1}{2\pi}\Re\left(\frac{\left(\sqrt{z-a}+\sqrt{w-a}\right)\left(\sqrt{z-a}-\overline{\sqrt{w-a}}\right)}{\left(\sqrt{z-a}-\sqrt{w-a}\right)\left(\sqrt{z-a}+\overline{\sqrt{w-a}}\right)}\right),\quad
z,w\in\mathbb{H},$ (1.3)

where we take the branch of the square root such that it takes value in the
upper half plane.

The GFF with Dirichlet boundary condition on $(a,\infty)$ and Neumann boundary
condition (also called free boundary condition ) on $(-\infty,a)$ is a
centered Gaussian process $\Gamma$ indexed by the set of continuous functions
with compact support in $\mathbb{H}$ such that

$\mathbb{E}\left[\Gamma(f)\Gamma(g)\right]=\int_{\mathbb{H}\times\mathbb{H}}f(z)G_{\mathrm{mix}}(z,w)g(w)dzdw.$

Notice that when $a\rightarrow-\infty$, $\Gamma$ is reduced to the GFF with
Dirichlet boundary condition. In [PW19] the authors gave a full description of
the crossing probabilities of all possible patterns of the level lines of the
GFF with alternating boundary conditions. In particular, they gave a formula
of the probability that the level line of the GFF originated from one given
point terminates at another given point ((1.8) of [PW19, Theorem 1.4] ). In
this note we will use martingale methods to deal with the GFF with the mixture
of one free boundary data and Dirichlet boundary data and give a formula
corresponding to (1.8) of [PW19].

##  2 Results

Given $n+1$ points $a<b_{1}<b_{2}<...<b_{n}$. Let $\phi$ be the harmonic
function on $\mathbb{H}$ satisfying the Neumann boundary condition on
$(-\infty,a)$ and taking values $\pm\lambda$ alternatively in the intervals
$(b_{i},b_{i+1})$ for $i=0,1,2,...,n$, where we take $b_{0}=a,b_{n+1}=+\infty$
and $\lambda=\sqrt{\pi/8}$ (see the following figure).

The GFF (denoted by $h$) with free boundary condition on $(\infty,a)$ and
alternating boundary conditions $\pm\lambda$ on $(b_{i},b_{i+1})$ for
$i=1,2,...,n$ is defined as the Gaussian process $\Gamma+\phi$, where $\Gamma$
is defined as the previous section.

Fix $k\in\\{1,2,3,...,n\\}$, let $I_{k}$ be the subset of $\\{1,2,3,...,n\\}$
that consists of numbers having the same parity as $k$ and
$J_{k}=\\{1,2,3,...,n\\}\setminus I_{k}$. In [IK13, Proposition 6], the
authors gave a coupling between the GFF in the strip with arbitrary mixed
boundary condition which can be stated as following:

######  Theorem 2.1.

There exists a coupling between $h$ and the random Loewner chain started from
$b_{k}$ driven by the following SDE systems:

$\displaystyle\begin{cases}db_{k}(t)=2dB_{t}+\frac{-1}{b_{k}(t)-a(t)}dt+\sum_{i\in
I_{k}}F(b_{k}(t),b_{i}(t),a(t))dt-\sum_{i\in
J_{k}}F(b_{k}(t),b_{j}(t),a(t))dt\\\\[11.38109pt]
da(t)=\frac{2}{a(t)-b_{k}(t)}dt,\quad
db_{i}(t)=\frac{2}{b_{i}(t)-b_{k}(t)}dt,\quad\forall i\neq k,\end{cases}$
(2.1)

where $F$ is defined as

$\displaystyle F(x,y,z):=\frac{2}{x-y}\sqrt{\frac{y-z}{x-z}},\quad z<x,z<y.$
(2.2)

Now we state our main result.

######  Theorem 2.2.

The Loewner chain driven by $b_{k}(t)$ in (2.1) is almost surely generated by
a simple curve $\gamma[0,T]\rightarrow\overline{\mathbb{H}}$ such that

$\displaystyle\gamma(0)=b_{k},\quad\gamma(0,T)\subset\mathbb{H},\quad\gamma(T)=b_{i}\text{
for some }i\in J_{k}\text{ or }\gamma(T)\in(-\infty,a).$ (2.3)

Moreover, we have

$\displaystyle\mathbb{P}(\gamma(T)\in(-\infty,a))=\prod_{i\in
I_{k},i<k}\frac{1+\sqrt{\frac{b_{i}-a}{b_{k}-a}}}{1-\sqrt{\frac{b_{i}-a}{b_{k}-a}}}\prod_{i\in
I_{k},i>k}\frac{1+\sqrt{\frac{b_{k}-a}{b_{i}-a}}}{1-\sqrt{\frac{b_{k}-a}{b_{i}-a}}}\prod_{i\in
J_{k},i<k}\frac{1-\sqrt{\frac{b_{i}-a}{b_{k}-a}}}{1+\sqrt{\frac{b_{i}-a}{b_{k}-a}}}\prod_{i\in
J_{k},i>k}\frac{1-\sqrt{\frac{b_{k}-a}{b_{i}-a}}}{1+\sqrt{\frac{b_{k}-a}{b_{i}-a}}}.$
(2.4)

We begin the proof with the following lemma.

######  Lemma 2.3.

Suppose that $(g_{t})_{t\geq 0}$ is $\mathrm{SLE}_{4}(-1)$ process from
$b_{k}$ to $\infty$ with force point $a$. Define $b_{j}(t):=g_{t}(b_{j})$ for
$j\neq k$ and $a(t)=g_{t}(a)$. Let $F$ be the function defined as (2.2),

$\displaystyle J_{t}:=\sum_{i\in I_{k}}F(b_{k}(t),b_{i}(t),a(t))dt-\sum_{i\in
J_{k}}F(b_{k}(t),b_{j}(t),a(t))dt,$ (2.5)

and

$Z_{t}:=\left(\prod_{i\in
I_{k}}\frac{\sqrt{b_{i}(t)-a(t)}-\sqrt{b_{k}(t)-a(t)}}{\sqrt{b_{i}(t)-a(t)}+\sqrt{b_{k}(t)-a(t)}}\prod_{j\in
J_{k}}\frac{\sqrt{b_{j}(t)-a(t)}+\sqrt{b_{k}(t)-a(t)}}{\sqrt{b_{j}(t)-a(t)}-\sqrt{b_{k}(t)-a(t)}}\right)^{2}.$

Then $\log(Z_{t})$ is a local martingale and $d\log(Z_{t})=2J_{t}dB_{t}$.

###### Proof.

By [MS16, Theorem 1.3] $\mathrm{SLE}_{4}(-1)$ is generated by a simple curve.
So ${b_{j}(t):j\neq k}$ and $a(t)$ are well-defined. By definition we have

$\displaystyle\frac{1}{2}\log(Z_{t})=$ $\displaystyle\sum_{i\in
I_{k}}\log\left(\sqrt{b_{i}(t)-a(t)}-\sqrt{b_{k}(t)-a(t)}\right)-\log\left(\sqrt{b_{i}(t)-a(t)}+\sqrt{b_{k}(t)-a(t)}\right)$
$\displaystyle+$ $\displaystyle\sum_{j\in
J_{k}}\log\left(\sqrt{b_{j}(t)-a(t)}+\sqrt{b_{k}(t)-a(t)}\right)-\log\left(\sqrt{b_{j}(t)-a(t)}-\sqrt{b_{k}(t)-a(t)}\right).$

So from Itô’s formula, we have

$\displaystyle d\log(Z_{t})=$ $\displaystyle\sum_{i\in
I_{k}}\frac{-2}{(b_{i}(t)-b_{k}(t))(b_{k}(t)-a(t))}\sqrt{\frac{b_{i}(t)-a(t)}{b_{k}(t)-a(t)}}dt+\frac{2}{b_{k}(t)-b_{i}(t)}\sqrt{\frac{b_{i}(t)-a(t)}{b_{k}(t)-a(t)}}\left(2dB_{t}+\frac{-1}{b_{k}(t)-a(t)}\right)$
$\displaystyle+$ $\displaystyle\sum_{i\in
J_{k}}\frac{2}{(b_{i}(t)-b_{k}(t))(b_{k}(t)-a(t))}\sqrt{\frac{b_{i}(t)-a(t)}{b_{k}(t)-a(t)}}dt+\frac{-2}{b_{k}(t)-b_{i}(t)}\sqrt{\frac{b_{i}(t)-a(t)}{b_{k}(t)-a(t)}}\left(2dB_{t}+\frac{-1}{b_{k}(t)-a(t)}\right)$
$\displaystyle=$ $\displaystyle\sum_{i\in
I_{k}}\frac{2}{b_{k}(t)-b_{i}(t)}\sqrt{\frac{b_{i}(t)-a(t)}{b_{k}(t)-a(t)}}2dB_{t}+\sum_{i\in
J_{k}}\frac{-2}{b_{k}(t)-b_{i}(t)}\sqrt{\frac{b_{i}(t)-a(t)}{b_{k}(t)-a(t)}}2dB_{t}$
$\displaystyle=$ $\displaystyle 2J_{t}dB_{t}.$

∎

Now define $N_{t}:=\exp\\{-\frac{1}{8}\int_{0}^{t}J_{s}^{2}ds\\}$, where
$J_{t}$ is defined as (2.5). And it can be checked that
$M_{t}:=Z_{t}^{\frac{1}{4}}N_{t}$ is a local martingale up to the first time
$T$ that one of $\\{a,b_{1},...,b_{k-1},b_{k+1},...,b_{n}\\}$ is swallowed and
$dM_{t}=\frac{1}{2}M_{t}J_{t}dB_{t}$. In fact,

$\displaystyle T:=\inf\left\\{t:b_{k}(t)-a(t)=0\text{ or
}b_{k}(t)-g_{t}(b_{i})=0\text{ for some }i\neq k\right\\}.$ (2.6)

Now define

$\displaystyle
T_{n}^{1}:=\inf\left\\{t:\left|\frac{b_{k}(t)-a(t)}{b_{i}(t)-a(t)}\right|\leq\frac{1}{n}\text{
or }\left|\frac{b_{k}(t)-a(t)}{b_{i}(t)-a(t)}\right|\geq 1-\frac{1}{n}\text{
for some }i>k\right\\}.$ (2.7) $\displaystyle
T_{n}^{2}:=\inf\left\\{t:\left|\frac{b_{i}(t)-a(t)}{b_{k}(t)-a(t)}\right|\leq\frac{1}{n}\text{
or }\left|\frac{b_{i}(t)-a(t)}{b_{k}(t)-a(t)}\right|\geq 1-\frac{1}{n}\text{
for some }i<k\right\\}.$ (2.8)

And $T_{n}:=T_{n}^{1}\wedge T_{n}^{2}$. Then we have $(M_{t}:0\leq t\leq
T_{n})$ is a bounded martingale for any $n$. Since the Loewner chain driven by
(2.1) can be obtained by weighting $\operatorname{SLE}_{4}(-1)$ with the local
martingale $M_{t}$. We can see that the Loewner chain driven by (2.1) is
absolutely continuous with respect to $\operatorname{SLE}_{4}(-1)$ up to
$T_{n}$ and therefore the Loewner chain driven by (2.1) is generated by a
continuous curve up to $T_{n}$. Since $n$ is arbitrary, we have the Loewner
chain driven by (2.1) is generated by a continuous curve up to $T$, which we
will denote by $\gamma$. We need to show that as $t\rightarrow T$, $\gamma(t)$
converges almost surely. Notice that when $\gamma(t)$ accumulates at
$\\{b_{i}:i\in I_{k}\\}$, the local martingale is uniformly bounded and
therefore it is absolutely continuous with respect to
$\operatorname{SLE}_{4}(-1)$ up to and include $T$. But
$\operatorname{SLE}_{4}(-1)$ is a continuous curve from $b_{k}$ to
$(-\infty,a)$. So $\gamma$ can not accumulate at $\\{b_{i}:i\in I_{k}\\}$. The
same reason excludes the event that $\gamma$ accumulates at $b_{k}$. So almost
surely as $t\rightarrow T$, $\gamma(t)$ accumulates at
$(\infty,a)\cup\\{b_{i}:i\in J_{k}\\}$.

Let $\mathcal{E}$ be the event that $\gamma$ has accumulation point in
$(-\infty,a)$. Then on the event $\mathcal{E}$, $M_{t}$ is bounded and
$\gamma$ is absolutely continuous with respect to $\operatorname{SLE}_{4}(-1)$
and therefore is a continuous curve from $b_{k}$ to $(-\infty,a)$. We have
finished the first part of Theorem 2.2.

Now we prove (2.4). Define

$\displaystyle g(a,b_{1},b_{2},...,b_{n})=\prod_{i\in
I_{k},i<k}\frac{1+\sqrt{\frac{b_{i}-a}{b_{k}-a}}}{1-\sqrt{\frac{b_{i}-a}{b_{k}-a}}}\prod_{i\in
I_{k},i>k}\frac{1+\sqrt{\frac{b_{k}-a}{b_{i}-a}}}{1-\sqrt{\frac{b_{k}-a}{b_{i}-a}}}\prod_{i\in
J_{k},i<k}\frac{1-\sqrt{\frac{b_{i}-a}{b_{k}-a}}}{1+\sqrt{\frac{b_{i}-a}{b_{k}-a}}}\prod_{i\in
J_{k},i>k}\frac{1-\sqrt{\frac{b_{k}-a}{b_{i}-a}}}{1+\sqrt{\frac{b_{k}-a}{b_{i}-a}}}.$
(2.9)

One can check that $\tilde{M}_{t}:=g(a(t),b_{1}(t),b_{2}(t),...,b_{n}(t))$ is
a local martingale up to $T$.

######  Lemma 2.4.

Take the notations as above, $(\tilde{M}_{t}:0\leq t\leq T)$ is a martingale
and we have $\lim_{t\rightarrow T}\tilde{M}_{t}=1$ if
$\gamma(T)\in(-\infty,a)$; $\lim_{t\rightarrow T}\tilde{M}_{t}=0$ otherwise.

###### Proof.

Suppose that $\gamma(T)=b_{i_{0}}$ for some $i_{0}\in J_{k}$. Without lost of
generality, we may assume $b_{i_{0}}<b_{k}$. Then as $t\rightarrow T$ we have
$b_{j}(t)-b_{k}(t)\rightarrow 0$ for any $i_{0}\leq j\leq k$. Hence we have
$\prod_{i\in
J_{k},i>k}\frac{1-\sqrt{\frac{b_{k}(t)-a(t)}{b_{i}(t)-a(t)}}}{1+\sqrt{\frac{b_{k}(t)-a(t)}{b_{i}(t)-a(t)}}}$
tends to some finite value, so are

$\prod_{i\in
I_{k},i>k}\frac{1+\sqrt{\frac{b_{k}(t)-a(t)}{b_{i}(t)-a(t)}}}{1-\sqrt{\frac{b_{k}(t)-a(t)}{b_{i}(t)-a(t)}}},\quad\prod_{i\in
I_{k},i<i_{0}}\frac{1+\sqrt{\frac{b_{i}(t)-a(t)}{b_{k}(t)-a(t)}}}{1-\sqrt{\frac{b_{i}(t)-a(t)}{b_{k}(t)-a(t)}}},\quad\text{
and }\prod_{i\in
J_{k},i<i_{0}}\frac{1-\sqrt{\frac{b_{i}(t)-a(t)}{b_{k}(t)-a(t)}}}{1+\sqrt{\frac{b_{i}(t)-a(t)}{b_{k}(t)-a(t)}}}.$

We are left to investigate

$I:=\prod_{i\in I_{k},i_{0}\leq
i<k}\frac{1+\sqrt{\frac{b_{i}(t)-a(t)}{b_{k}(t)-a(t)}}}{1-\sqrt{\frac{b_{i}(t)-a(t)}{b_{k}(t)-a(t)}}}\prod_{i\in
J_{k},i_{0}\leq
i<k}\frac{1-\sqrt{\frac{b_{i}(t)-a(t)}{b_{k}(t)-a(t)}}}{1+\sqrt{\frac{b_{i}(t)-a(t)}{b_{k}(t)-a(t)}}}.$

We can put the products in pairs
$(i_{0},i_{0}+1),(i_{0}+2,i_{0}+3),...,(k-3,k-2),(k-1)$. For each pair
$(j,j+1)$, we have

$\frac{1-\sqrt{\frac{b_{j}(t)-a(t)}{b_{k}(t)-a(t)}}}{1+\sqrt{\frac{b_{j}(t)-a(t)}{b_{k}(t)-a(t)}}}\cdot\frac{1+\sqrt{\frac{b_{j+1}(t)-a(t)}{b_{k}(t)-a(t)}}}{1-\sqrt{\frac{b_{j+1}(t)-a(t)}{b_{k}(t)-a(t)}}}=\left(\frac{1+\sqrt{\frac{b_{j+1}(t)-a(t)}{b_{k}(t)-a(t)}}}{1+\sqrt{\frac{b_{j}(t)-a(t)}{b_{k}(t)-a(t)}}}\right)^{2}\frac{b_{k}(t)-b_{j}(t)}{b_{k}(t)-b_{j+1}(t)}$

Notice that as $t\rightarrow T$

$\frac{b_{k}(t)-b_{j}(t)}{b_{k}(t)-b_{j+1}(t)}\rightarrow 1\quad\forall
i_{0}\leq j<k-1.$

So

$\lim_{t\rightarrow T}\prod_{i\in I_{k},i_{0}\leq
i<k}\frac{1+\sqrt{\frac{b_{i}(t)-a(t)}{b_{k}(t)-a(t)}}}{1-\sqrt{\frac{b_{i}(t)-a(t)}{b_{k}(t)-a(t)}}}\prod_{i\in
J_{k},i_{0}\leq
i<k}\frac{1-\sqrt{\frac{b_{i}(t)-a(t)}{b_{k}(t)-a(t)}}}{1+\sqrt{\frac{b_{i}(t)-a(t)}{b_{k}(t)-a(t)}}}=\lim_{t\rightarrow
T}\frac{1-\sqrt{\frac{b_{k-1}(t)-a(t)}{b_{k}(t)-a(t)}}}{1+\sqrt{\frac{b_{k-1}(t)-a(t)}{b_{k}(t)-a(t)}}}=0.$

We get $\tilde{M_{t}}\rightarrow 0$ as $t\rightarrow T$ when
$\gamma(T)=b_{i_{0}}$ for some $i_{0}\in J_{k}$.

Now suppose that we have $\gamma(T)\in(-\infty,a)$. Then as $t\rightarrow T$,
$a(t)-b_{k}(t)\rightarrow 0$ and $b_{i}(t)-b_{k}(t)\rightarrow 0$ for $i<k$.
So

$\prod_{i\in
J_{k},i>k}\frac{1-\sqrt{\frac{b_{k}(t)-a(t)}{b_{i}(t)-a(t)}}}{1+\sqrt{\frac{b_{k}(t)-a(t)}{b_{i}(t)-a(t)}}}\cdot\prod_{i\in
I_{k},i>k}\frac{1+\sqrt{\frac{b_{k}(t)-a(t)}{b_{i}(t)-a(t)}}}{1-\sqrt{\frac{b_{k}(t)-a(t)}{b_{i}(t)-a(t)}}}\rightarrow
1.$

By Lemma B.2 of [PW19], we have as $t\rightarrow T$,

$\frac{b_{i}(t)-a(t)}{b_{k}(t)-a(t)}\rightarrow 0,\quad\forall i<k.$

Therefore if $\gamma(T)\in(-\infty,a)$, $\tilde{M}_{t}\rightarrow 1$ as
$t\rightarrow T$. So $(\tilde{M}_{t}:0\leq t\leq T)$ is a uniformly bounded
martingale and by the stopping theorem we can get (2.4). ∎

######  Remark 2.5.

Since $k$ is arbitrary, we can get all the probabilities that the level line
started from $b_{k}$ terminates at the free boundary arc $(-\infty,a)$. Let
$a$ goes to $\infty$, we can get the crossing probabilities of the GFF with
Dirichlet boundary condition which is the Theorem 1.4 of [PW19]. For example,
take $k=1$ and $n=2N-1$, we have

$g(a,b_{1},b_{2},...,b_{n})=\prod_{i\in
I_{k},i>k}\frac{1+\sqrt{\frac{b_{k}-a}{b_{i}-a}}}{1-\sqrt{\frac{b_{k}-a}{b_{i}-a}}}\prod_{i\in
J_{k},i>k}\frac{1-\sqrt{\frac{b_{k}-a}{b_{i}-a}}}{1+\sqrt{\frac{b_{k}-a}{b_{i}-a}}}.$

By taking $a\rightarrow\infty$, the limit above is equal to

$\frac{b_{2}-b_{1}}{b_{3}-b_{1}}\cdot\frac{b_{4}-b_{1}}{b_{5}-b_{1}}\cdot...\cdot\frac{b_{2N-2}-b_{1}}{b_{2N-1}-b_{1}},$

which is equation $(1.8)$ of [PW19].

## Acknowledgement

Y.Han and Y.Wang are supported in part by NSF of China (Grants No. 11688101)
and Z.Wang is supported by NSF of China (Grants No.11601296)

## References

  * [DS11] Bertrand Duplantier and Scott Sheffield.  Liouville quantum gravity and KPZ.  Invent. Math., 185(2):333–393, 2011.
  * [IK13] Konstantin Izyurov and Kalle Kytölä.  Hadamard’s formula and couplings of SLEs with free field.  Probab. Theory Related Fields, 155(1-2):35–69, 2013.
  * [Law05] Gregory F. Lawler.  Conformally invariant processes in the plane, volume 114 of Mathematical Surveys and Monographs.  American Mathematical Society, Providence, RI, 2005.
  * [LSW01] Gregory F. Lawler, Oded Schramm, and Wendelin Werner.  Values of Brownian intersection exponents. I. Half-plane exponents.  Acta Math., 187(2):237–273, 2001.
  * [MS16] Jason Miller and Scott Sheffield.  Imaginary geometry I: Interacting SLEs.  Probab. Theory Related Fields, 164(3-4):553–705, 2016.
  * [PW19] Eveliina Peltola and Hao Wu.  Global and local multiple SLEs for ${\kappa\leq 4}$ and connection probabilities for level lines of GFF.  Communications in Mathematical Physics, 366(2):469–536, 2019.
  * [RS05] Steffen Rohde and Oded Schramm.  Basic properties of SLE.  Ann. of Math. (2), 161(2):883–924, 2005.


# Deep Learning on Mobile Devices Through Neural Processing Units and Edge
Computing

Tianxiang Tan and Guohong Cao  Department of Computer Science and Engineering
The Pennsylvania State University
Email: {txt51<EMAIL_ADDRESS>
###### Abstract

Deep Neural Network (DNN) is becoming adopted for video analytics on mobile
devices. To reduce the delay of running DNNs, many mobile devices are equipped
with Neural Processing Units (NPU). However, due to the resource limitations
of NPU, these DNNs have to be compressed to increase the processing speed at
the cost of accuracy. To address the low accuracy problem, we propose a
Confidence Based Offloading (CBO) framework for deep learning video analytics.
The major challenge is to determine when to return the NPU classification
result based on the confidence level of running the DNN, and when to offload
the video frames to the server for further processing to increase the
accuracy. We first identify the problem of using existing confidence scores to
make offloading decisions, and propose confidence score calibration techniques
to improve the performance. Then, we formulate the CBO problem where the goal
is to maximize accuracy under some time constraint, and propose an adaptive
solution that determines which frames to offload at what resolution based on
the confidence score and the network condition. Through real implementations
and extensive evaluations, we demonstrate that the proposed solution can
significantly outperform other approaches.

##  I Introduction

Deep Neural Networks (DNN) have been successfully applied to various computer
vision and natural language processing problems. Recently, many applications
based on DNNs have been developed to provide more intelligent video analytics.
For example, some drones such as DJI Mavic Pro can recognize and follow a
target based on video analytics; law enforcement officers can use smart
glasses to identify suspects [1]. In these applications, only lightweight DNNs
can be run locally and their accuracy is much lower than advanced DNNs.
Although advanced DNNs can provide us with better results, they also suffer
from high computational overhead which means long delay and more energy
consumption when running on mobile devices.

In recent years, many companies such as Huawei, Qualcomm, and Samsung have
developed dedicated Neural Processing Units (NPUs) for mobile devices, which
can process AI features. With NPU, the running time of these DNNs can be
significantly reduced. For example, on HUAWEI mate 10 pro, running AlexNet (a
DNN) [2] on NPU is 30 times faster than running it on CPU. Although GPUs on
mobile devices can also be used to run DNNs, they are not as powerful as those
running on desktops and servers. Compared to GPUs on mobile devices, NPUs are
much faster and more energy efficient [3]. However, there are some fundamental
limitations with NPU. The most significant limitation of NPU is the precision
of the floating-point numbers. NPU uses 16 bits or 8 bits to represent the
floating-point numbers instead of 32 bits in CPU. As a result, it runs DNNs
much faster but less accurate compared to CPU. Moreover, NPU has its own
memory which is usually too small for advanced DNNs. The advanced DNNs have to
be compressed in order to be loaded and executed by NPU at the cost of
accuracy.

To address the low accuracy problem on NPU, mobile devices can offload the
data to the edge server and let the edge server run the DNNs. Since the server
has more computational power, high accuracy can be achieved by processing the
data with the most advanced DNNs in a short time. However, when the network
condition is poor or when the data size is large which is usually true in
video analytics, offloading may take longer time to transmit the data. There
is a tradeoff between the offloading based approach and the NPU based
approach. The offloading based approach has good accuracy, but has longer
delay under poor network condition. On the other hand, the NPU based approach
is faster, but with less accuracy. Recent research (e.g., FastVA [4]) tries to
combine these two approaches for video analytics on mobile devices. To
maximize accuracy under some time constraint, FastVA determines which frames
should be processed locally on NPU and which frames should be offloaded to the
edge server. However, existing research treats DNN as a black box and never
considers exploring insights about running DNNs on NPU to improve performance.

We address this problem by considering the confidence level of running DNNs on
NPU. The confidence level can be represented by the confidence score, which is
computed based on the output of the DNN. If the confidence score is higher
than a threshold, the classification result on NPU is most likely accurate and
can be directly used; otherwise, the data is offloaded for further processing
to improve the accuracy. Similar ideas have been studied in existing work
although for different purpose. For example, in [5, 6], confidence score is
leveraged to reduce the processing delay by early exit; i.e., returning the
classification results without running all DNN layers. However, there are two
unique challenges for applying this idea to video analytics on mobile devices
with NPU. First, the confidence score of many advanced DNNs cannot accurately
estimate the classification results, and then may not be effective for making
offloading decisions. Second, existing research relies on fixed confidence
score threshold to determine when to offload. However, in video analytics, the
decision can be affected by many factors such as the confidence score, the
network condition, the video frame resolution, and thus fixed threshold will
not work well.

In this paper, we address these challenges by proposing a Confidence Based
Offloading (CBO) framework for deep learning video analytics on mobile devices
with NPU. To address the first challenge, we propose techniques to calibrate
the confidence score so that it can accurately reflect the correctness of the
classification results on NPU. To address the second challenge, we formulate
the CBO problem, where the goal is to maximize accuracy under some time
constraint. To achieve this goal, the confidence score threshold is adaptively
adjusted based on the network condition, confidence score and the selected
frame resolution, before being used for determining if the video frame should
be offloaded for further processing.

Our contributions can be summarized as follows.

  * •

To address the accuracy loss problem in deep learning video analytics on NPU
based mobile devices, we study confidence based offloading and propose
confidence score calibration techniques to improve the performance.

  * •

We formulate the CBO problem, and propose an optimal solution. Since the
optimal solution relies on the perfect knowledge of all future video frames,
we further propose an online algorithm to relax this assumption.

  * •

Through real implementations and extensive evaluations, we demonstrate that
the proposed solution can significantly outperform other approaches.

The rest of the paper is organized as follows. Section II presents the
background and motivation. In Section III, we evaluate the effectiveness of
CBO and propose calibration techniques to improve performance. In Section IV,
we present our CBO framework for video analytics. Section V presents the
evaluation results and Section VI reviews related work. Section VII concludes
the paper.

##  II Preliminary

###  II-A Characteristics of NPU

(a) Processing time comparison (b) Accuracy comparison

Figure 1: Performance comparisons of running DNNs on NPU and CPU.

To have a better understanding of NPU, we conducted some experiments on HUAWEI
mate 10 pro, using the following four DNN models: 1) VGG [7] with the LFW
dataset [8], 2) VocNet [9] with 4000 images from the VOC dataset [10], 3)
AlexNet [2] with the VOC dataset, 4) YOLO Small [11] with 4000 images randomly
chosen from the MS COCO dataset [12]. As shown in Figure 1(a), compared to
CPU, running VGG, VocNet, AlexNet, Yolo Small on NPU can significantly reduce
the processing time by 95%; however, this may be at the cost of accuracy loss
as illustrated in Figure 1(b). Specifically, compared to CPU, using NPU has
similar accuracy when running VGG, 30% accuracy loss when running VocNet, 11%
accuracy loss for AlexNet, and the F1-score drops to 0.3 for YOLO Small.

The accuracy loss is mainly because NPU can only support FP16 operations and
store the intermediate result of each layer using FP16. Running DNNs with FP16
can save memory and reduce the processing time, but it may reduce the accuracy
due to the numerical instability caused by floating point overflow or
underflow.

The amount of accuracy loss is related to the DNN model. In VGG, the extracted
feature vectors are compared, and they represent the same person if the
similarity is above a predefined threshold. Although NPU introduced error may
change some values, the relationship between the similarity and the threshold
will remain and thus keep the same level of accuracy. However, AlexNet, VocNet
and Yolo Small use more information in the feature vectors to identify or
locate multiple objects in the images. Each value in the feature vector
represents the category, the location or the size of an object and a small
error introduced by NPU can change the prediction completely. As a result,
they have lower accuracy when running on NPU.

As shown in Figure 1, NPU runs much faster than CPU. It is ideal for running
some DNNs such as VGG, where using NPU can significantly reduce the processing
time while maintaining high accuracy. However, it may not be the best choice
for running DNNs such as VocNet and Yolo small, due to the high accuracy loss.
In this paper, we address this problem by proposing techniques to improve its
accuracy.

###  II-B Motivation

To address the accuracy loss problem, one solution is to offload the video
frames to the edge server or cloud for processing. However, transmitting all
video frames to the server may take much longer time and may violate the delay
constraint, compared to processing them locally on NPU, especially when the
wireless bandwidth is limited. Instead of processing video frames only on NPU
or only through offloading, we take advantage of both approaches by exploring
insights about running DNNs on NPU.

Although running DNNs on NPU has lower accuracy in general, the accuracy of
running DNNs for some specific objects can be much higher, due to the skewed
accuracy of the DNNs. This is because DNNs are becoming more and more complex,
with many parameters and layers designed to achieve high accuracy for general
inference scenarios. These DNNs have lots of redundancy especially when the
inference scenario is simple; i.e., even when running on NPU with lower
precision parameter and simple model structure, the compressed DNN can still
perform well for some simple scenarios.

Figure 2: Accuracy for different objects with VocNet

Figure 2 shows the accuracy of running VocNet on CPU and NPU for different
objects. When considering all objects, the accuracy of running VocNet on NPU
is 0.54, which is lower than that on CPU (0.89). When considering specific
objects, VocNet on NPU performs much better for recognizing airplanes with
accuracy of 0.96, which is similar to the accuracy on CPU. This is because
there is much higher visual difference between the background (i.e., blue sky)
and the airplane. Even though the VocNet has been compressed to be run on NPU,
with lower precision parameters and simpler model structure, it can still
recognize airplanes accurately. On the other hand, VocNet performs poorly on
NPU for recognizing tables, with accuracy of 0.1. This is because most tables
in the data set have much complex background, and the visual difference
between the background and table is much smaller. Then, a more complex and
advanced DNN (i.e., the original DNN) is needed to classify these objects
correctly.

To consider both cases, two different DNNs should be applied, a compressed DNN
to process simple cases and the original DNN to process more complex cases.
Due to the resource limitation of NPU, the compressed DNN should be run on NPU
and the original DNN should be run on the edge server or cloud. Then, images
such as airplane should be processed with local NPU and other images such as
tables should be offloaded to the server for further processing. However, the
system does not know whether an image is an airplane or table before hand.
Thus, we propose the following solution. First, the image is processed locally
on NPU, which has negligible delay. If the image can be recognized with a high
level of confidence, the classification result is returned; otherwise, the
image is offloaded for further processing. Then, most images can be processed
locally with high accuracy and low delay, and only images with low accuracy
are offloaded. The key component of this solution is how to quantify the
confidence level of running DNNs on NPU, which will be discussed in the next
section.

Figure 3: Making offloading decisions based on confidence score.

Figure 4: The effects of confidence score threshold $\theta$.

Figure 5: Accuracy vs. confidence score.

Figure 6: Calibration model training.

##  III Quantifying the confidence level

The result of the DNN is represented by a feature vector which can be
extracted from the last layer of the DNN. To quantify the confidence level of
the DNN, the feature vector is used to calculate the confidence score. In this
section, we first present an offloading framework based on the calculated
confidence score, and and then present confidence score calibration techniques
to improve its accuracy.

###  III-A Confidence Based Offloading (CBO)

Figure 6 illustrates the basic idea of CBO. To determine which images should
be offloaded, confidence score is used to predict the correctness of the
result. The confidence score can be computed based on the feature vector
produced by the DNN. Let $\mathbf{x}=[x_{1},x_{2},x_{3},\ldots,x_{N}]$ denote
the extracted feature vector, where each element $x_{i}$ represents the
probability that object $i$ appears in the image. Normally, the values of
these elements are not normalized and the confidence score is computed as
$\max\sigma(x_{i})$, where $\sigma(x_{i})$ is the softmax function and it is
defined as $\sigma(x_{i})=\frac{e^{x_{i}}}{\sum_{k=1}^{N}e^{x_{k}}}$. As shown
in Figure 6, when the confidence score is larger than a threshold $\theta$,
the result is returned; otherwise, the image is offloaded for further
processing.

We conducted experiments to evaluate the effectiveness of CBO. The AlexNet and
ResNet-152 which are trained based on the ImageNet dataset, are used to
process videos randomly selected from the FCVID dataset, which includes
thousands of real-world videos. In the experiment, AlexNet is run on NPU and
ResNet-152 is run on the server. The offload percentage is defined as
$\frac{n^{o}}{n}$, where $n^{o}$ is the number of offloaded frames and $n$ is
the total number of video frames.

The results are shown in Figure 6. When $\theta$ is 0, all frames are only
processed by NPU. When $\theta$ is 1, all frames are offloaded to the server.
As shown in the figure, as $\theta$ increases from 0 to 1, more frames are
offloaded to the server and the accuracy increases from 0.42 to 0.81. However,
this is not as expected due to the following reason. When the accuracy
requirement is 0.8, the threshold has to be above 0.8 at which the offloaded
traffic is above 90%. Then, CBO does not save any bandwidth.

The reason of such poor performance is because the confidence score generated
by the DNN on NPU cannot accurately estimate the correctness of the result. We
conducted the following experiment. With the same DNN and the dataset to
identify the reason. In this experiment, the video frames are divided into 10
bins with 0.1 confidence interval. For example, the first bin includes the
frames with confidence score from 0 to 0.1. For each bin, we process the
frames on NPU and calculate the accuracy. The result is shown in Figure 6. As
the confidence score increases from 0 to 1, the accuracy increases from 0.29
to 0.5. For example, the accuracy is 0.42 for the frames with confidence score
between 0.2 to 0.3.

CBO is based on the assumption that frames classified incorrectly on NPU will
be offloaded for further processing. This can be easily achieved if the
confidence score of running the DNN to process the frame is the same as its
accuracy. Then, the classification result of processing frames with confidence
score 0.9 is more likely to be correct than that with confidence score 0.5,
and hence the accuracy can be improved by offloading frames with lower
confidence score. However, as shown in Figure 6, the accuracy remains to be
0.5 for frames with confidence score much higher than 0.5 (e.g., 0.9). Then,
it is hard to use the confidence score to determine which frame should be
offloaded. This explains why CBO performs poorly as shown in Figure 6.
Fortunately, there is nothing wrong with CBO. The problem is due to the fact
that the confidence score cannot accurately estimate the correctness of the
classification result, and it should be calibrated.

###  III-B Confidence Score Calibration

Although modern DNNs can achieve better accuracy in classification problems,
the confidence scores produced by these DNNs are poorly calibrated [13]. To
address this issue, researchers have proposed many confidence score
calibration techniques, following two different methods.

Platt Calibration: Platt Calibration is a parametric method [14] and its key
idea is to train logistic regression models which can transform the confidence
score into a calibrated one to provide better correctness estimation. The
logistic model can be defined as
$P(y_{i}=1|\mathbf{x})=\frac{1}{1+e^{\mathbf{A}f(\mathbf{x})+\mathbf{B}}}$,
where $\mathbf{A,B}$ are the parameters that needs to be trained.

Isotonic Regression: Different from Platt Calibration, the Isotonic Regression
is a common non-parametric calibration method [15]. It learns a piecewise
constant function $f$ to transform the confidence score into a calibrated one,
which means $x^{\prime}_{i}=f(x_{i})$. To train the Isotonic Regression model,
$f$ is trained by minimizing the square loss function
$\sum_{i=1}^{n}(f(x_{i})-y_{i})^{2}$.

Although these two methods use different machine learning models to calibrate
the confidence score, they have the same training procedure for confidence
score calibration, and this procedure is shown in Figure 6. As shown in the
figure, $N$ models are trained for calibrating the confidence score
$x_{1},x_{2},\ldots,x_{N}$. For each model $i$, it takes the feature vector
$\mathbf{x}=[x_{1},x_{2},x_{3},\ldots,x_{N}]$ as input and outputs a new
confidence score $x^{\prime}_{i}$. When the $i^{th}$ model is being trained,
the output variable $y_{j}$ is set to be 1 if object $i$ appears in the frame
$I_{j}$. $y_{j}$ is set to be 0 if $I_{j}$ does not include object $i$.

In our CBO, our goal is to choose a method which can calibrate the confidence
score and make it match the accuracy of the DNN. To achieve this goal, we use
the following metrics to evaluate them and select the best.

  * •

Expected Calibration Error (ECE): ECE measures the expectation difference
between the confidence score and accuracy of the DNN. ECE is defined as
$\sum_{i=1}^{10}\frac{B_{i}}{n}|A(B_{i})-C(B_{i})|$, where $B_{i}$ ($1\leq
i\leq 10$) is the bins with 0.1 confidence interval. For example $B_{1}$
includes the frames with confidence score between 0 to 0.1. For each bin
$B_{i}$, we process the frames on NPU and calculate the accuracy $A(B_{i})$
and the average confidence score $C(B_{i})$.

  * •

Maximum Calibration Error (MCE): MCE measures the worst-case difference
between the confidence score and accuracy of the DNN. MCE is defined as
$\max_{1\leq i\leq 10}|A(B_{i})-C(B_{i})|$.

Method | ECE | MCE
---|---|---
Uncalibrated | 0.27 | 0.48
Platt Calibration | 0.07 | 0.29
Isotonic Regression | 0.16 | 0.41
TABLE I: Comparison of different calibration techniques.

With AlexNet and the FCVID dataset, we perform an experiment to evaluate the
performance of different calibration techniques. In the experiment, the video
frames are divided into 10 bins with 0.1 confidence interval. For example, the
first bin includes the frames with confidence score from 0 to 0.1. For each
bin, we process the frames on NPU and calculate the accuracy. We use the
images from ImageNet validation dataset and a subset of video frames from
FCVID dataset as the training data for Platt Calibration and Isotonic
Regression. The result is shown in Table I. The uncalibrated confidence score
has the worst performance among the three methods. Compared to the Isotonic
Regression, Platt Calibration has lower ECE and MCE, which means its
performance is better. This is because the Isotonic Regression suffers from
overfitting, where the model can learn the training data well but is not
generalized to new data. Therefore, Platt Calibration is used for calibrating
the confidence score.

(a) Accuracy and offload traffic under different calibrated confidence score
thresholds. (b) Accuracy vs. calibrated confidence score.

Figure 7: The performance of Platt Calibration. Figure 8: The CBO framework.

The effectiveness of confidence calibration We perform experiments to evaluate
the performance of the calibration technique using the same setting as that in
the last subsection except that the confidence score is replaced with the
calibrated confidence score based on Platt Calibration.

As shown in Figure 7 (a), the accuracy is increased as the number of offloaded
frames increases. Compared to the results shown in Figure 6, calibrated
confidence score is more effective. For example, when the accuracy requirement
is 0.7, the threshold has to be above 0.35 at which the offloaded traffic is
30%. However, as shown in Figure 6, at least 70% of the frames are offloaded
in order to satisfy the same accuracy requirement.

With the same setting, Figure 7 (b) explains why calibrated confidence score
is more effective. As the confidence score increases from 0 to 1, the accuracy
increases from 0.11 to 1. The accuracy variation is much larger than that
shown in Figure 6. Thus, it is much easier to determine which frames should be
offloaded.

##  IV The CBO Framework

###  IV-A Overview

Figure 8 shows our CBO framework for video analytics. The video frames are
first processed locally on NPU. Based on the calibrated confidence score, our
framework determines which frames should be offloaded. To provide real time
video analytics, the processing of each video frame should be completed within
a time constraint. Then, for some offloaded frames, the resolution may be
reduced to save bandwidth and delay, at the cost of accuracy.

In the figure, CBO is based on the idea presented in Figure 6. However, the
confidence threshold $\theta$ is not fixed; it is adaptively adjusted based on
the network condition, the confidence score and the selected frame resolution.
That is, based on the accuracy and processing time requirement, we study the
CBO problem which adaptively selects $\theta$ and the frame resolution, to
maximize accuracy under some time constraint. In the following, we first
formulate the problem and then propose an adaptive solution which determines
which frames to offload at what resolution based on the confidence score and
the network condition.

###  IV-B Problem Formulation

For each frame $I_{i}$ ($1\leq i\leq n$), it is first processed by the DNN on
NPU. Let $p_{i}$ denote the calibrated confidence score and let
$A^{npu}_{p_{i}}$ denote the accuracy of running the DNN to process the frame
on NPU. Assume the video frame rate is $f$, the time interval between two
consecutive video frames is $\gamma=\frac{1}{f}$. Since NPU is very fast, the
local processing time for each frame is shorter than $\gamma$ and it is not
the bottleneck. For the $i^{th}$ video frame, assume its arrival time is
$i\gamma$, our system ensures that it is processed before time $T+i\gamma$,
where $T$ is the time constraint.

Notation | Description
---|---
$I_{i}$ | The $i$th frame
$A^{o}_{r}$ | The accuracy of the DNN with input images
| in resolution $r$
$S(I_{i},r)$ | The data size of the frame $I_{i}$ in resolution $r$
$T^{o}$ | The processing time on the server
$B$ | Upload bandwidth (data rate)
$f$ | Frame rate (fps)
$T$ | The time constraint for each frame
$n$ | The number of video frames that needs to be processed
$\theta$ | The confidence score threshold for offloading.
TABLE II: Notation

Based on the calibrated confidence score, a frame may be offloaded to the
server for further processing to improve accuracy. If $p_{i}$ is higher than
the threshold $\theta$, the classification result is returned. Otherwise, the
classification result is considered to be incorrect and $I_{i}$ is offloaded
to the server in the original resolution or reduced to resolution $r$ before
being offloaded. Let $B$ denote the upload bandwidth and let $L$ denote the
network latency between the server and the mobile device. Then, it takes
$\frac{S(I_{i},r)}{B}+T^{o}+L$ to transmit the $i^{th}$ frame in resolution
$r$ and receive the result from the server. In this way, the transmission time
can be reduced by resizing the frame to a lower resolution, at a cost of lower
accuracy.

The notations used in the problem formulation are listed in Table II. The CBO
problem can be formulated as an integer programming in the following way.

$\displaystyle\max\quad$
$\displaystyle\frac{1}{n}\displaystyle\sum^{n}_{i=1}(A^{npu}_{p_{i}}(1-X_{i})+\sum_{r}A^{o}_{r}Y^{r}_{i}X_{i})$
(1) s.t. $\displaystyle D(k)\leq(i-k)*\gamma+T,\,\forall i,k$ (2)
$\displaystyle\theta-p_{i}<X_{i},\,\forall i$ (3) $\displaystyle
p_{i}-\theta\leq 1-X_{i},\,\forall i$ (4)
$\displaystyle\sum_{r}Y^{r}_{i}=1,\forall i$ (5) $\displaystyle
X_{i},Y^{r}_{i}\in\\{0,1\\},\,\forall i$ (6)

Where $D(k)=\sum_{r}\sum_{k\leq i}\frac{S(k,r)Y^{r}_{k}X_{k}}{B}+T^{o}X_{i}+L$
is the offloading time for the frames that arrive between $I_{k}$ and $I_{i}$.
$X_{i}$ is a variable to show whether the frame is needed to be offloaded and
$Y^{r}_{i}$ is a variable to show which resolution the frame is resized to
before offloading. If $X_{i}=0$, the frame $I_{i}$ is only processed locally.
If $X_{i}=1$, the frame $I_{i}$ is offloaded to the server for further
processing. If $Y^{r}_{i}=1$, the frame $I_{i}$ is resized to resolution $r$
before offloading.

Objective (1) is to maximize the accuracy of the processed frames. Constraint
(2) specifies that the classification results should be returned within the
time constraint, and constraint (3) specifies that the frames with confidence
score lower than or equal to threshold $\theta$ should be offloaded.
Constraint (4) specifies that frames with confidence score higher than
$\theta$ should not be offloaded.

###  IV-C Finding the Optimal Solution

In this subsection, we build a solution graph based on which we can find the
optimal solution for the CBO problem. As shown in Figure 9, nodes at different
levels represent the schedule options for different frames. More specifically,
nodes at level $i(1\leq i\leq n)$ represent the schedule options of frame
$I_{i}$. There are $(m+1)$ nodes at level $i$, representing $m+1$ scheduling
options, where $m$ is the number of frame resolutions. For example, at level 1
(i.e., for frame $I_{1}$), node $V^{npu}_{1}$ represents that the frame is
processed on local NPU, and $V^{r_{m}}_{1}$ represents that frame $I_{1}$ is
offloaded to the server in resolution $r_{m}$. Each node is associated with a
time window $[i\gamma,i\gamma+T]$ which represents the time constraint for the
frame to be processed. We also create two dummy nodes: $V_{start}$ and
$V_{end}$. They are the source node and the destination node at level 0 and
$n+1$.

Figure 9: The solution graph.

For edges, we add a link from each node at level $i$ to all nodes at level
$i+1$. An edge $(V^{r}_{i},V^{r}_{i+1})$ has two attributes, cost and time
duration. The cost is defined as the negative of the accuracy
$c(V^{r}_{i},V^{r}_{i+1})=-A^{o}_{r}$ and
$c(V^{npu}_{i},V^{r}_{i+1})=A^{npu}_{p_{i}}$. The time duration is the
offloading time since local processing time on NPU is very short and it is not
the bottleneck. The time duration between $V^{r}_{i}$ and $V^{r}_{i+1}$ (for
frame $I_{i}$ with resolution $r$) can be computed as
$t(V^{r}_{i},V^{r}_{i+1})=\frac{S(i,r)}{B}$. In this way, our problem is
converted to the problem of finding the least cost path from $V_{start}$ to
$V_{end}$ while visiting each chosen node within its specific time window.

######  Theorem 1.

The CBO problem is NP-hard.

###### Proof.

We reduce a well known NP-hard problem, the subset sum problem to our problem.
In the subset sum problem, there is a set $U$ which includes $n$ numbers
$(a_{1},a_{2},a_{3},\ldots,a_{n})$ and the goal is to find a subset of numbers
so that its sum is equal to a value $K$.

For an arbitrary instance of the subset sum problem, we can construct an
instance of our solution graph as follows. For each number $a_{i}$, two nodes
$V^{0}_{i}$ and $V^{1}_{i}$ are added to the graph at level $i$. The time
window of these two nodes are set to be $[-\sum_{i}|a_{i}|,\sum_{i}|a_{i}|]$.
Specially, the time window for the destination node $V_{end}$ is set to be
$[K,K]$.

For edges, we add links from nodes at level $i$ to nodes at level $i+1$. More
specifically, the cost is set to be 0 and the time duration
$t(V^{p}_{i},V^{q}_{i+1})$ is set to $a_{i}$ if $p=1$; otherwise,
$t(V^{p}_{i},V^{q}_{i+1})=0$.

A solution to our problem must satisfy the requirement that the destination
node $V_{end}$ must be visited at the exact time $K$. As a result, the sum of
the time duration of the selected nodes is equal to $K$. Since the time
duration of a node is also equivalent to the value of the corresponding
number, the sum of the selected numbers is also equal to $K$. This completes
the reduction and hence the proof. ∎

Due to the time window constraint, the shortest path cannot be found using the
Dijkstra algorithm. Instead, a dynamic programming algorithm is used to search
for the optimal path.

Let $P_{j}(V^{r}_{i})$ denote the $j^{th}$ feasible path from $V_{start}$ to
$V^{r}_{i}$ and each path has two attributes $T_{j}(V^{r}_{i})$ and
$C_{j}(V^{r}_{i})$, which are used to record the time duration and the cost of
the path. Initially, $T(V_{start})$ and $C_{(}V_{end})$ are set to be 0. For
each path $P_{j}(V^{r}_{i})=(V_{start},V^{r}_{1},\ldots,V^{r}_{i})$, its
attributes are iteratively computed as follows

$\displaystyle
T_{j}(V^{r}_{k})=\max(T_{j}(V^{r}_{k-1})+t(V^{r}_{k-1},V^{r}_{k}),k\gamma)$
$\displaystyle C_{j}(V^{r}_{k})=C_{j}(V^{r}_{k-1})+c(V^{r}_{k-1},V^{r}_{k})$

Since the node $V_{i}$ must be visited during $[i\gamma,i\gamma+T]$, a
feasible path should satisfy $i\gamma\leq T_{j}(V^{r}_{k})\leq i\gamma+T$.
Although a lot of feasible paths can be found in the iterations, the algorithm
only considers the most efficient ones. More specifically, for two paths
$P_{1}(V^{r}_{i})$ and $P_{2}(V^{r}_{i})$, if
$T_{1}(V^{r}_{i})<T_{2}(V^{r}_{i})$ and $C_{1}(V^{r}_{i})<C_{2}(V^{r}_{i})$,
$P_{1}(V^{r}_{i})$ is more efficient than $P_{2}(V^{r}_{i})$ and
$P_{2}(V^{r}_{i})$ will not be considered in future iterations. The optimal
path is $P(V_{end})$ which has the minimum cost $\min C(V_{end})$.

Since there are at most $T$ different efficient paths from $V_{start}$ to
$V^{r}_{i}$, there are at most $mT$ paths at level $i(1\leq i\leq n)$.
Therefore, the time complexity of the optimal algorithm is $O(nm^{2}T)$.

###  IV-D The CBO Algorithm

The optimal solution can maximize the accuracy within the time constraint.
However, it is not practical since it requires the complete knowledge of all
frames, such as the frame sizes and the confidence score of running DNN to
process the frame. In this subsection, we remove this assumption and propose
an adaptive solution, called CBO algorithm.

Since the frames with lower confidence scores are classified with lower
accuracy on NPU, they should be offloaded to increase the accuracy as long as
there is available bandwidth. However, due to bandwidth limitation, some
frames cannot be offloaded and have to rely on local NPU for classification.

We use the following dynamic programming algorithm to determine which frames
should be offloaded with what resolution. Suppose $k$ frames have been
processed locally. For each frame $I_{i}$ ($1\leq i\leq k$), its arrival time
is $t^{arr}_{i}$ and the confidence score is $p_{i}$. The frames are sorted in
the descending order of the confidence scores, which means $p_{i}>p_{j}$ if
$i<j$. In our algorithm, a list $l_{j}$ ($j\in[0,k]$) is used to find the
schedule decision for maximizing the accuracy. Each element in the list
$l_{j}$ is a pair $(t,A)$, where $A$ is the accuracy improvement which can be
achieved by offloading the first $j$ frames within time $t$. If $I_{i}$ is
offloaded in resolution $r$, the accuracy improvement can be computed as
$A=A^{o}_{r}-A^{npu}_{p_{i}}$. Initially, $l_{0}={(0,0)}$. To add pairs to the
list $l_{j}$, we consider the following two cases.

No offloading:  In this case, the $j^{th}$ frame will not be offloaded to the
server for further processing. All pairs in $l_{j-1}$ will be added to
$l_{j}$.

Offloading:  In this case, the $j^{th}$ frame will be offloaded in resolution
$r$. It takes $\frac{S(I_{j},r)}{B}$ to transmit the frame and the accuracy
improvement is $A^{o}_{r}-A^{npu}_{p_{j}}$. For each pair $(t,A)\in l_{j-1}$,
a new pair
$(max(t,t^{arr}_{j})+\frac{S(I_{j},r)}{B},A+A^{o}_{r}-A^{npu}_{p_{j}})$ is
added to $l_{j}$. Notice that the frames should be processed within the time
constraint. Therefore, $max(t,t^{arr}_{j})+\frac{S(I_{j},r)}{B}+T^{o}+L\leq
T+t^{arr}_{j}$ must be satisfied for all new pairs.

To improve the efficiency of our algorithm, only the most efficient pairs in
$l_{j}$ are kept. More specifically, a pair $(t^{\prime},A^{\prime})$ is said
to dominate another pair $(t,A)$ if and only if $t^{\prime}\leq t$ and
$A^{\prime}\geq A$. The pair $(t^{\prime},A^{\prime})$ is more efficient than
the pair $(t,A)$ and all the dominated pair will be removed from $l_{j}$.

With the list of $l_{k}$, we can find the confidence threshold $\theta$ and
offloaded frame resolution $r^{o}$. After the first frame is offloaded, the
algorithm will be run again for the frames that have been processed locally.
The CBO algorithm is summarized in Algorithm 1. In Lines 1-10, dynamic
algorithm is applied to maximize the accuracy, and the schedule decision is
determines for frame $I_{1}$ in Lines 11-20. The running time of the algorithm
is $O(k^{2}*m)$.

Result: Confidence threshold $\theta$, frame resolution $r^{o}$

1

2$l_{0}\leftarrow\\{(0,0)\\}$

3

4for  _$j\leftarrow 1$ to $k$_ do

5 for  _each $(t,A)\in l_{j-1}$_ do

6 Add $(t,A)$ to $l_{j}$

7 for  _each possible resolution r_ do

8 $t^{\prime}\leftarrow max(t,t^{arr}_{j})+\frac{S(j,r)}{B}$

9 if  _$t^{\prime}+T^{o}+L\leq T+t^{arr}_{j}$_ then

10 $A^{\prime}\leftarrow A^{o}_{r}-A^{npu}_{p_{j}}$

11 Add $(t^{\prime},A+A^{\prime})$ to $l_{j}$

12

13 Remove the dominated pairs from $l_{j}$

14

15$(t^{\prime},A^{\prime})\leftarrow\arg\max_{(t,A)\in l_{k}}A$

16

17for  _$j$ from $k-1$ to $0$_ do

18 for  _each pair $(t,A)$ in $l_{j}$_ do

19 for  _each possible resolution $r$_ do

20 if  _$t+\frac{S(I_{j},r)}{B}=t^{\prime}$ and
$A+A^{o}_{r}-A^{npu}_{p_{j}}=A^{\prime}$_ then

21 $A^{\prime}\leftarrow A$, $t^{\prime}\leftarrow t$

22 $\theta\leftarrow p_{j}$, $r^{o}\leftarrow r$

23

24return $r^{o}$, $\theta$

Algorithm 1 CBO algorithm

##  V Performance Evaluations

In this section, we evaluate the performance of the CBO algorithm and compare
it with other approaches.

###  V-A Evaluation Setup

The evaluations are performed on HUAWEI mate 10 pro, which is equipped with 6
GB memory, octa-core CPU ($4\times 2.4$ GHz and $4\times 1.8$ GHz) and a NPU.
HUAWEI has published the HUAWEI DDK [16] toolset for developers to run the DNN
on NPU. The pre-trained DNNs must be optimized before they can be run on NPU
since NPU has a different architecture from CPU. The HUAWEI DDK includes
toolsets to perform such optimizations for DNNs. It also includes the APIs to
run the DNNs, and a few Java Native Interface (JNI) functions are provided to
use the APIs on Android. Since these JNI functions cannot extract the
confidence scores, we add JNI functions to use the confidence score
calibration model which is trained on a powerful desktop.

In the experiment, AlexNet is deployed on the mobile device and ResNet-152 is
deployed on the server. These DNNs are used for object recognition, and they
are very popular in computer vision community and have been fine-tuned for
many problem. Moreover, AlexNet has a simple model structure and it can be
executed efficiently on NPU to provide real-time video analytic. In contrast,
ResNet-152 is more complex and it can achieve higher accuracy than AlexNet at
the cost of more computational power.

To measure the performance of our algorithm, we use a subset of videos from
the FCVID dataset, which includes many real-world videos. These videos have
been used for training models related to object classification and activity
recognition. In our experiment, we focus on object classification, and thus
activity recognition clips are not used. Since the dataset is very large,
about 1.9 TB, we randomly select 40 videos from the dataset and filter out the
noisy data.

In the experiment, the frames are offloaded in the lossless PNG format. The
server is a desktop with AMD Ryzen 7 1700 CPU, GeForce GTX1070 Ti graphics
card and 16 GB RAM. We have installed the Caffe [17] framework to run the DNNs
on GPU.

Figure 10: Accuracy vs. Resolution.

Figure 11: The performance of different approaches under different network
conditions.

Figure 12: The performance of different approaches under different frame
rates.

Figure 13: The performance of different approaches under different network
latency.

We evaluate the proposed algorithms with different frame rates. Most videos in
the dataset use 30 fps, and thus we have to change their frame rate by
decoding/encoding. For ResNet-152, the maximum resolution of the input image
is 224x224 pixels. This resolution can be downsized for some offloading
images, and we consider 5 different resolutions: 45x45, 90x90, 134x134,
179x179 and 224x224 pixels. The tradeoff between accuracy and frame resolution
is shown in Figure 10.

| Running Time (ms)
---|---
AlexNet on NPU | 20
ResNet-152 on server | 37
Confidence Score Calibration | 8
TABLE III: The running time of different models for each frame.

The time constraint for each frame is set to be 200 ms in all experiments. The
running time of our CBO algorithm is less than 1 ms on smartphone and it is
negligible compared to the time constraint (100 ms level). The running time of
different DNNs are shown in Table III. The data shown in Figure 10 and Table
III will be used for making scheduling decisions in Algorithm 1. We compare
CBO with the following approaches.

  * •

Local:  All frames are processed locally on NPU. After a frame has been
processed on NPU, the classification result is returned.

  * •

Server:  All frames are offloaded to the server for processing. Each frame is
resized to a resolution due to bandwidth limitation so that it can be
offloaded before the next frame arrives.

  * •

FastVA:  This is an implementation of the FastVA framework [4] which maximizes
accuracy under some time constraint. Based on the optimization, it determines
which frames should be processed locally on NPU and which frames should be
offloaded to the edge server.

  * •

Compress:  This method is similar to FastVA, except that it runs a compressed
DNN to process the frames locally on CPU. We leverage Tensorflow which is a
well-known deep learning framework to perform pruning and dynamic range
quantization on the original DNN. Specifically, the sparsity of the model is
set to 0.9 in the pruning step. After compression, the model size is reduced
by 95% and 50% of the processing time is saved on CPU, but still much slower
than NPU.

  * •

CBO without Calibration (CBO-w/o):  This algorithm is the same as CBO, except
that the confidence score is not calibrated.

  * •

Optimal:  This shows the performance upper bound for all algorithms. It tries
all possible combinations and chooses the schedule that maximizes the
accuracy. Note that this method cannot be used for processing videos in real
time since it takes too much time to search all possible schedules. We can
only find the optimal solution offline by replaying the data trace.

###  V-B Evaluation Results

The performance of the algorithm depends on several factors, the bandwidth,
the network latency, and the video frame rate.

In Figure 13, we compare CBO with Local, Server, FastVA, Compress and CBO-w/o
under different network conditions. In the evaluation, the network latency is
set to be 100 ms and the frame rate is set to be 30 fps. Since no frame is
offloaded to the server in the Local approach, its performance does not change
under different network conditions. Local, FastVA, CBO and CBO-w/o have the
same accuracy when the bandwidth is 0, since no frame can be offloaded to the
server. When the bandwidth is lower than 3 Mbps, the Local approach achieves
higher accuracy than the Server approach. This is because the frames have to
be offloaded in an extremely low resolution. Even with advanced DNN running at
the server, the accuracy is still pretty low with low resolution frames.
Compared to the Compress approach, FastVA achieves higher accuracy when the
network bandwidth is lower than 15 Mbps. This is because the running time of
compressed DNN on CPU is long and Compress offloads most frames in low
resolution. Note that the processing time of compressed DNN (50%) is not
reduced as much as the model size (95%). This is because the reduction of the
model size is mainly due to the removal of the redundant parameters in the
fully connected layers. After compression, there are still many convolutional
layers which are computationally intensive, and it takes a large amount of
time to run these layers. In contrast, FastVA avoids offloading frames in low
resolution by processing video frames on NPU. As can be seen from the figure,
CBO outperforms FastVA, since CBO knows which frames are classified
incorrectly based on the calibrated confidence scores and it can improve the
accuracy effectively by offloading the frames with low confidence scores.

From Figure 13, we can also see that CBO-w/o underperforms CBO and FastVA
because the uncalibrated confidence scores cannot accurately estimate the
correctness of the classification result. With uncalibrated confidence score,
CBO-w/o may offload frames which have been classified correctly on NPU,
wasting bandwidth resources, and it may return misclassified results and
reduce the accuracy. Compared to the Server approach, CBO-w/o achieves higher
accuracy when the network bandwidth is low. This is because CBO-w/o avoids
offloading frames in very low resolution by returning the classification
result with high confidence score. When the bandwidth is high, the Server
approach outperforms CBO-w/o since it can offload frames with higher
resolution and then increase the accuracy. As the bandwidth increases above
36Mbps, the difference among CBO, CBO-w/o, FastVA, Compress and Server becomes
smaller since most frames can be offloaded to the server in higher resolution
to achieve higher accuracy.

In Figure 13, we evaluate the impact of frame rate on accuracy for different
approaches. We set the uplink network bandwidth to be 5 Mbps and set the
network latency to be 100 ms. As shown in the figure, CBO significantly
outperforms CBO-w/o, Server, FastVA, Compress and Local. In general, the
accuracy of all approaches drops when the frame rate increases. The Server
approach suffers a 15% accuracy drop when the frame rate increases from 5 fps
to 30 fps. This is because most frames have to be resized to low resolution
when the frame rate increases to 30 fps. In contrast, only 6% accuracy drop is
observed in CBO, since it only offloads the frames with low confidence scores.
With the same amount of bandwidth, CBO can offload more frames with high
resolution compared to other approaches.

In Figure 13, we evaluate the impact of network latency on accuracy. We set
the uplink network bandwidth to be 5 Mbps and set the frame rate to be 30 fps.
As shown in the figure, CBO significantly outperforms CBO-w/o, Server, FastVA,
Compress and Local. Since the Local approach does not offload any frames, its
performance remains the same. As the latency increases, less frames can be
offloaded to the server for processing due to the delay constraint
requirement. Therefore, the performance of Server, Compress, FastVA, CBO and
CBO-w/o degrades as the latency increases. Compared to the Server approach,
the accuracy drop in CBO is much smaller. This is because CBO can reduce the
chance of offloading frames in low resolution by using NPU based
classification results that have high level of confidence.

(a) The performance of Optimal (b) Difference between Optimal and CBO

Figure 14: Comparison between CBO and Optimal.

In Figure 14, we compare CBO with Optimal under various frame rate and network
conditions. As shown in Figure 14(a), the accuracy of Optimal increases when
the network bandwidth increases, because the mobile device can upload more
frames with higher resolution. As frame rate requirement increases, less
frames can be offloaded and Optimal has to offload frames with lower
resolution. Therefore, the accuracy becomes lower.

In Figure 14(b), we show the accuracy difference between Optimal and CBO. The
accuracy difference is computed using the accuracy of the Optimal approach
minus that of CBO. As can be seen from the figure, the difference is almost
zero in most cases, which indicates that CBO is close to Optimal.

##  VI Related Work

In recent years, researchers have made tremendous progress in applying DNNs
for various classification problems [7, 9, 18, 19]. However, these DNNs are
designed for machines with powerful CPU and GPU, and it is difficult to run
them on mobile devices due to the resource limitations. To address this issue,
researchers leveraged model compression technique to reduce the resource
demand of DNNs. For example, in [20, 21], the authors proposed techniques to
remove the redundant parameters and operations from the neural network to
reduce the model size and processing time. Although the efficiency can be
improved through these model compression techniques, the accuracy also drops.

Computation offloading represents another kind of solution for enabling video
analytics on mobile devices. Some general offloading frameworks [22, 23, 24]
have been proposed to optimize energy and reduce the computation time for
mobile applications. However, these frameworks have limitations when applied
to deep learning based video analytics since a large amount of data has to be
offloaded to the server. To address this issue, researchers propose offloading
framework for deep learning applications [25, 26, 27, 28, 29, 30]. There have
been some studies on confidence based offloading. For example, in [5] and [6],
confidence score is leveraged to reduce the processing delay by early exit;
i.e., returning the classification results without running all DNN layers.
Different from them, our framework is designed for video analytics on mobile
device with NPU, where the goal is to maximize accuracy under some time
constraint. Moreover, we propose confidence score calibration technique to
improve the performance.

Considerable amount of work has been done on improving the execution
efficiency of DNNs on mobile devices through hardware support. For example,
Tan et al. [31] developed model partitioning techniques to schedule some
neural network layers on CPU while executing other layers on NPU to achieve
better tradeoffs between processing time and accuracy. Cappuccino [32]
optimized computation by exploiting imprecise computation on the mobile
system-on-chip (SoC). Oskouei et al. [33] developed an Android library called
CNNdroid for running DNNs on mobile GPU. DeepMon [34] leveraged GPU for
continuous vision analysis on mobile devices. FastVA [4] leveraged NPU and
offloading technique for video analytics on mobile devices. Different from
FastVA, we explore insights about running DNNs on NPU to improve performance.

##  VII Conclusions

In this paper, we proposed a CBO framework for video analytics to address the
low accuracy problem of running DNNs on NPU. The major challenge is to
determine when to return the NPU classification result based on the confidence
level of running the DNN, and when to offload the video frames to the server
for further processing to increase the accuracy. We found that existing
confidence scores were not effective for making offloading decisions, and thus
proposed techniques to calibrate the confidence score so that it could
accurately reflect the correctness of the classification results on NPU. We
formulated the CBO problem, where the goal is to maximize accuracy under some
time constraint. To achieve this goal, the confidence score threshold is
adaptively adjusted based on the network condition, the confidence score and
the selected frame resolution, before being used for determining if the video
frame should be offloaded for further processing. Extensive evaluation results
show that the proposed solution can significantly outperform other approaches.

## Acknowledgement

This work was supported in part by the National Science Foundation under grant
number 2125208.

## References

  * [1] “Chinese police spot suspects with surveillance sunglasses,” https://www.bbc.com/news/world-asia-china-42973456.
  * [2] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” _Advances in neural information processing systems (NIPS)_ , 2012.
  * [3] “HiSilicon Kirin 970 Performance Overview,” https://www.anandtech.com/show/12195/hisilicon-kirin-970-power-performance-overview/6f.
  * [4] T. Tan and G. Cao, “FastVA: Deep Learning Video Analytics Through Edge Processing and NPU in Mobile,” _IEEE INFOCOM_ , 2020.
  * [5] S. Teerapittayanon, B. McDanel, and H. Kung, “Distributed deep neural networks over the cloud, the edge and end devices,” _IEEE ICDCS_ , 2017\.
  * [6] S. Wang, S. Yang, and C. Zhao, “SurveilEdge: Real-time Video Query based on Collaborative Cloud-Edge Deep Learning,” _IEEE INFOCOM_ , 2020.
  * [7] A. Z. O. Parkhi, A. Vedaldi, “Deep Face Recognition,” _British Machine Vision Conference_ , 2015.
  * [8] G. B. Huang, M. Ramesh, T. Berg, and E. Learned-Miller, “Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments,” University of Massachusetts, Amherst, Tech. Rep., 2007.
  * [9] S. Lapuschkin, A. Binder, G. Montavon, K.-R. Müller, and W. Samek, “Analyzing classifiers: Fisher vectors and deep neural networks,” _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016\.
  * [10] M. Everingham, S. Eslami, L. Van Gool, C. Williams, J. Winn, and A. Zisserman, “The Pascal Visual Object Classes Challenge: A Retrospective,” _Springer International Journal of Computer Vision (IJCV)_ , 2015.
  * [11] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” _IEEE CVPR_ , 2016.
  * [12] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft coco: Common objects in context,” _European Conference on Computer Vision_ , 2014.
  * [13] C. Guo, G. Pleiss, Y. Sun, and K. Weinberger, “On Calibration of Modern Neural Networks,” _ACM International Conference on Machine Learning (ICML)_ , 2017.
  * [14] A. Niculescu-Mizil and R. Caruana, “Predicting Good Probabilities with Supervised Learning,” _ACM International Conference on Machine Learning (ICML)_ , 2005.
  * [15] B. Zadrozny and C. Elkan, “Transforming Classifier Scores into Accurate Multiclass Probability Estimates,” _ACM International Conference on Knowledge Discovery and Data Mining (SIGKDD)_ , 2002.
  * [16] “HiAI,” https://developer.huawei.com/consumer/en/devservice/doc/
2020315\.

  * [17] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell, “Caffe: Convolutional Architecture for Fast Feature Embedding,” _ACM International Conference on Multimedia_ , 2014\.
  * [18] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” _IEEE CVPR_ , 2016.
  * [19] A. .Barbu, D. Mayo, J. Alverio, W. Luo, C. Wang, D. Gutfreund, J. Tenenbaum, and B. Katz, “Objectnet: A Large-Scale Bias-Controlled Dataset for Pushing the Limits of Object Recognition Models,” _Advances in neural information processing systems (NIPS)_ , 2019.
  * [20] J. Luo, J. Wu, and W. Lin, “Thinet: A Filter Level Pruning Method for Deep Neural Network Compression,” _IEEE ICCV_ , 2017.
  * [21] R. Yu, A. Li, C. Chen, J. Lai, V. Morariu, X. Han, M. Gao, C. Lin, and L. Davis, “NISP: Pruning Networks Using Neuron Importance Score Propagation,” _IEEE CVPR_ , 2018.
  * [22] E. Cuervo, A. Balasubramanian, D. Cho, A. Wolman, S. Saroiu, R. Chandra, and P. Bahl, “MAUI: making smartphones last longer with code offload,” _ACM Mobisys_ , 2010.
  * [23] Y. Geng, W. Hu, Y. Yang, W. Gao, and G. Cao, “Energy-efficient computation offloading in cellular networks,” _IEEE ICNP_ , 2015.
  * [24] Y. Geng, Y. Yang, and G. Cao, “Energy-efficient computation offloading for multicore-based mobile devices,” _IEEE INFOCOM_ , 2018.
  * [25] S. Han, H. Shen, M. Philipose, S. Agarwal, A. Wolman, and A. Krishnamurthy, “MCDNN: An Approximation-Based Execution Framework for Deep Stream Processing Under Resource Constraints,” _ACM Mobisys_ , 2016.
  * [26] T. Tan and G. Cao, “Deep Learning Video Analytics Through Edge Computing and Neural Processing Units on Mobile Devices,” _IEEE Transactions on Mobile Computing_ , To Appear.
  * [27] K. Chen, T. Li, H.-S. Kim, D. E. Culler, and R. H. Katz, “MARVEL: Enabling Mobile Augmented Reality with Low Energy and Low Latency,” _ACM Sensys_ , 2018.
  * [28] X. Ran, H. Chen, X. Zhu, Z. Liu, and J. Chen, “DeepDecision: A Mobile Deep Learning Framework for Edge Video Analytics,” _IEEE INFOCOM_ , 2018.
  * [29] T. Tan and G. Cao, “Deep Learning Video Analytics on Edge Computing Devices,” _IEEE SECON_ , 2021.
  * [30] L. Liu, H. Li, and M. Gruteser, “Edge Assisted Real-Time Object Detection for Mobile Augmented Reality,” _ACM International Conference on Mobile Computing and Networking_ , 2019.
  * [31] T. Tan and G. Cao, “Efficient Execution of Deep Neural Networks on Mobile Devices with NPU ,” _ACM/IEEE International Conference on Information Processing in Sensor Networks (IPSN)_ , 2021.
  * [32] M. Motamedi, D. Fong, and S. Ghiasi, “Cappuccino: efficient CNN inference software synthesis for mobile system-on-chips,” _IEEE Embedded Systems Letters_ , 2019.
  * [33] Latifi Oskouei, Seyyed Salar and Golestani, Hossein and Hashemi, Matin and Ghiasi, Soheil, “Cnndroid: Gpu-accelerated execution of trained deep convolutional neural networks on android,” _ACM International Conference on Multimedia_ , 2016.
  * [34] L. N. Huynh, Y. Lee, and R. K. Balan, “Deepmon: Mobile gpu-based deep learning framework for continuous vision applications,” _ACM Mobisys_ , 2017\.