###### Abstract

The purpose of this paper is to extend the embedding theorem of Sobolev spaces
involving general kernels and we provide a sharp critical exponent in these
embeddings. As an application, solutions for equations driven by a general
integro-differential operator, with homogeneous Dirichlet boundary conditions,
is established by using the Mountain Pass Theorem.

Sharp embedding of Sobolev spaces involving general kernels and its
application

Huyuan<EMAIL_ADDRESS>Hichem<EMAIL_ADDRESS>
1Department of Mathematics, Jiangxi Normal University, Nanchang, Jiangxi
330022, PR China

and

2Department of Mathematics, College of Science, King Saud University P.O. Box
2455, Riyadh 11451, Saudi Arabia.

Key words: Sobolev space involving general kernel, Sobolev embedding, Integro-
differential operator, Mountain Pass Theorem

MSC2010: 35R09, 35J61, 46E35

##  1  Introduction

In the study of weak solutions for semilinear elliptic equations, the
embedding from corresponding Sobolev space to $L^{q}$ space plays a
fundamental role, especially the compact embedding. In a recent work, Di
Nazza, Palatucci and Valdinoci in [11] made a clear description for the
fractional Sobolev space $W^{s,p}(\Omega)$ and gave an elegant proof for the
embedding theorem from $W^{s,p}(\Omega)$ to $L^{q}(\Omega)$, which is
continuous when $q\in[1,\frac{Np}{N-sp}]$ and compact when
$q\in[1,\frac{Np}{N-sp})$, where $s\in(0,1)$, $sp<N$ and
$\Omega\subset\mathbb{R}^{N}$ is a bounded extension domain with $N\geq 2$.

Motivated by the above work, our purpose of this paper is to build a sharp
embedding theorem of Sobolev space involving general kernel $K$ and by using
this embedding theorem to search for weak solutions to problem

$\begin{array}[]{lll}\mathcal{L}_{K}u+f(x,u)=0&\rm{in}\quad\Omega,\\\\[5.69054pt]
\phantom{\mathcal{L}_{K}+u^{p}--}u=0&\rm{in}\quad\Omega^{c},\end{array}$ (1.1)

where $\Omega\subset\mathbb{R}^{N}$ is an open bounded $C^{2}$ domain with
$N\geq 2$ and the nonlocal operator $\mathcal{L}_{K}$ is defined by

$\mathcal{L}_{K}u(x)=\frac{1}{2}\int_{\mathbb{R}^{N}}[u(x+y)+u(x-y)-2u(x)]K(y)dy$

with the kernel $K:\mathbb{R}^{N}\setminus\\{0\\}\to(0,+\infty)$ satisfying

$\int_{\mathbb{R}^{N}}\min\\{|x|^{2},1\\}K(x)dx<+\infty$ (1.2)

and

$K(x)=K(-x),\qquad x\in\mathbb{R}^{N}\setminus\\{0\\}.$ (1.3)

Moreover, we assume that $K$ is decreasing monotone in the following sense

$K(x)\geq K(y)\qquad{\rm{if}}\ \ |x|\leq|y|.$ (1.4)

A typical example for $K$ is given by $K(x)=|x|^{-(N+2s)}$ with $s\in(0,1)$
and then $\mathcal{L}_{K}$ is the fractional Laplacian operator
$-(-\Delta)^{s}$.

During the last years, non-linear equations involving general integro-
differential operators, especially, fractional Laplacian, have been studied by
many authors. Caffarelli and Silvestre [4] studied the fractional Laplacian
through extension theory. Caffarelli and Silvestre [5, 6], Ros-Oton and Serra
[18] investigated regularity results for fractional elliptic equations. Sire
and Valdinoci in [21], Felmer and Wang in [13], Hajaiej [15, 16] and Felmer,
Quaas and Tan [12] obtained symmetry property of solutions for semilinear
equation involving the fractional Laplacin. More interests on fractional
elliptic equations see [7, 8, 9, 10, 14] and the references therein.

Recently, Servadei and Valdinoci in [20] obtained a solution of (1.1) via
Mountain Pass Theorem under the hypothesis that there exist $\lambda>0$ and
$s\in(0,1)$ such that

$K(x)\geq\lambda|x|^{-(N+2s)},\quad x\in\mathbb{R}^{N}\setminus\\{0\\}$

and nonlinear term $f$ is subcritical, that is,

$|f(x,t)|\leq a_{1}+a_{2}|t|^{q-1}\quad{\rm a.e.}\ x\in\Omega,\
t\in\mathbb{R}$

with $q\in(2,\frac{2N}{N-2s})$ and constants $a_{1},a_{2}>0$. We say that
$\frac{2N}{N-2s}$ is the critical exponent, denoted by $2^{*}(s)$.

In this paper, we are also interested in studying problem (1.1) with more
general kernels and our purpose is to find new criterion for critical
exponent, where we could deal with the following case

$\liminf_{|x|\to 0^{+}}K(x)|x|^{N}\in(0,\infty).$ (1.5)

To this end, we define

$s_{0}=\sup\\{s\geq 0\ |\ \lim_{r\to
0^{+}}r^{2s}\int_{B_{r}^{c}(0)}K(y)dy=+\infty\\}.$ (1.6)

We remark that if $K$ satisfies (1.2) and is nonnegative, then the definition
in (1.6) is equivalent to

$s_{0}=\sup\\{s\geq 0\ |\ \lim_{r\to 0^{+}}r^{2s}\int_{B_{1}(0)\setminus
B_{r}(0)}K(x)dx=+\infty\\}$

By the fact that $\int_{B_{1}^{c}(0)}K(x)dx$ is bounded from (1.2).

Our basic setting is that $s_{0}>0.$ In section 2, we will prove that
$s_{0}\leq 1$ and exhibit an example in which the kernel $K$ satisfying (1.5)
makes $s_{0}\in(0,1)$. We note that the limit of
$r^{2s_{0}}\int_{B_{r}^{c}(0)}K(y)dy$, as $r\to 0$, could be in $[0,\infty]$
or even no exists. Denote

$l_{\infty}:=\liminf_{r\to 0^{+}}r^{2s_{0}}\int_{B_{r}^{c}(0)}K(y)dy,$ (1.7)

then it occurs one of the cases: Case 1: $l_{\infty}=0$ and Case 2:
$l_{\infty}\in(0,\infty]$,

Our first aim is to study the Sobolev space involving general kernel $K$.
Denote by $X$ the linear space of Lebesgue measurable functions from
$\mathbb{R}^{N}$ to $\mathbb{R}$ such that the restriction to $\Omega$ of any
function $g$ in $X$ belongs to $L^{2}(\Omega)$ and

$\int_{\mathbb{R}^{2N}\setminus\mathcal{O}}(g(x)-g(y))^{2}K(x-y)dxdy<+\infty,$

where $\mathcal{O}:=\Omega^{c}\times\Omega^{c}$. The space $X$ is endowed with
the norm as

$\|g\|_{X}=(\|g\|^{2}_{L^{2}(\Omega)}+\int_{\mathbb{R}^{2N}\setminus\mathcal{O}}(g(x)-g(y))^{2}K(x-y)dxdy)^{1/2}.$
(1.8)

Now we define the following Sobolev space

$X_{0}=\\{g\in X\ |\ g=0\ \ {\rm a.e.\ in}\ \Omega^{c}\\}$

equipped the norm (1.8). From (1.2), we stress that
$C^{2}_{0}(\Omega)\subseteq X_{0},$ see [20], and so $X$ and $X_{0}$ are
nonempty.

Now we are ready for an embedding theorem.

###### Theorem 1.1

Assume that $K$ satisfies (1.2), (1.4), (1.6) with $s_{0}\in(0,1]$,
$2^{*}(s_{0})=\frac{2N}{N-2s_{0}}$ and $l_{\infty}$ is defined by (1.7). Then
$(X_{0},\|\cdot\|_{X})$ is a Hilbert space and
$(i)$ if $l_{\infty}=0$, the embedding

$X_{0}\hookrightarrow L^{q}(\Omega)$ (1.9)

is continuous and compact for $q\in[1,2^{*}(s_{0}))$. Moreover, for
$q\in[1,2^{*}(s_{0}))$ there exists $C>0$ such that

$\|g\|_{L^{q}}\leq C\|g\|_{X},\quad\forall g\in X_{0};$ (1.10)

$(ii)$ if $l_{\infty}\in(0,\infty]$, the embedding (1.9) is continuous for
$q\in[1,2^{*}(s_{0})]$ and compact for $q\in[1,2^{*}(s_{0}))$, and the
embedding inequality (1.10) holds for for $q\in[1,2^{*}(s_{0})]$.

###### Example 1.1

Let

$K(x)=\frac{1}{|x|^{N+2s_{0}}}\left[(-\log|x|)_{+}+1\right]^{\sigma},\quad
x\in\mathbb{R}^{N}\setminus\\{0\\},$ (1.11)

where $\sigma\in\mathbb{R}$ and $(-\log|x|)_{+}=\max\\{-\log|x|,0\\}$. When
$s_{0}\in(0,1)$ $\sigma\in\mathbb{R}$ or $s_{0}=1$ $\sigma<-1$, the kernel $K$
defined by (1.11) satisfies (1.2) and (1.4).

We note that $l_{\infty}=0$ if $\sigma<0$, $l_{\infty}\in(0,\infty)$ if
$\sigma=0$ and $l_{\infty}=\infty$ if $\sigma>0$. In particular,
$s_{0}\in(0,1)$ and $\sigma=0$, the embedding (1.9) coincides the results in
[11]. Especially, when $s_{0}=1$ and $\sigma<-1$, $2^{*}(s_{0})=2^{*}$ the
critical exponent for $H^{1}_{0}(\Omega)\Subset L^{2^{*}}(\Omega)$.

Now we are able to make use of Theorem 1.1 to study the existence of weak
solutions of (1.1). Before stating the existence result we make precise the
definition of weak solution that we use in the article. We say that a function
$u\in X_{0}$ is a weak solution of (1.1) if

$\int_{\mathbb{R}^{N}\times\mathbb{R}^{N}}[u(x)-u(y)][\varphi(x)-\varphi(y)]K(x-y)dxdy=\int_{\Omega}f(x,u(x))\varphi(x)dx,$
(1.12)

for any $\varphi\in X_{0}$.

The existence result can be stated as follows.

###### Theorem 1.2

Assume that $f(x,u)=|u|^{p-2}u$, $K$ satisfies (1.2-1.4),
$2^{*}(s_{0})=\frac{2N}{N-2s_{0}}$, where $s_{0}\in(0,1]$ defined in (1.6).

Then problem (1.1) admits a nontrivial weak solution for
$p\in(2,2^{*}(s_{0}))$.

###### Remark 1.1

Take $K$ as example 1.1 with $s_{0}\in(0,1)$ and $\sigma\in\mathbb{R}$ or
$s_{0}=1$ and $\sigma<-1$, then problem (1.1) admits a weak solution for
$f(x,u)=|u|^{p-2}u$ with $p\in(2,2^{*}(s_{0}))$.

Take $K$ as example 2.1, problem (1.1) admits a weak solution for
$f(x,u)=|u|^{p-2}u$ with $p\in(2,2^{*}(s_{0}))$.

The paper is organized as follows. In Section 2, we analyze some basic
properties of the kernel $K$ and give an example showing that $s_{0}$ makes
sense. In Section 3, we study the Sobolev embedding theorem in our setting.
Finally, we prove the existence of weak solution to (1.1) in Section 4.

##  2 Discussion to the kernel $K$

This section is devoted to the properties of the kernel $K$.

###### Proposition 2.1

Assume that $s_{0}$ is defined by (1.6) and $K$ satisfies (1.2). Then $(i)$
for any $s<s_{0}$, we have

$\lim_{r\to 0^{+}}r^{2s}\int_{B_{r}^{c}(0)}K(y)dy=+\infty;$

$(ii)$

$s_{0}\leq\inf\\{s\geq 0\ |\ \lim_{r\to
0^{+}}r^{2s}\int_{B_{r}^{c}(0)}K(y)dy=0\\}\leq 1;$ (2.1)

$(iii)$ if there exists $s_{1}\leq s_{2}$ such that

$\liminf_{|x|\to 0^{+}}K(x)|x|^{N+2s_{1}}>0\quad{\rm and}\quad\limsup_{|x|\to
0^{+}}K(x)|x|^{N+2s_{2}}<\infty,$ (2.2)

then $s_{0}\in[s_{1},s_{2}]$.

Proof. _$(i)$_ By the definition of $s_{0}$, there at least are a sequence of
positive numbers $\\{s_{n}\\}$ such that

$s_{n}<s_{0},\quad\lim_{n\to\infty}s_{n}=s_{0},\quad\lim_{r\to
0^{+}}r^{2s_{n}}\int_{B_{r}(0)}K(y)dy=+\infty.$

Then for any $s<s_{0}$, there exists $s_{n}$ such that $s<s_{n}$ and then

$\lim_{r\to 0^{+}}r^{2s}\int_{B_{r}(0)}K(y)dy\geq\lim_{r\to
0^{+}}r^{2s_{n}}\int_{B_{r}(0)}K(y)dy=+\infty.$

_$(ii)$_ By (1.2) and $K$ being nonnegative, we have that for any $r\in(0,1)$,

$\displaystyle\infty$ $\displaystyle>$
$\displaystyle\int_{\mathbb{R}^{N}}\min\\{|x|^{2},1\\}K(x)dx$ $\displaystyle>$
$\displaystyle\int_{B_{1}(0)\setminus
B_{r}(0)}|x|^{2}K(x)dx+\int_{\mathbb{R}^{N}\setminus B_{1}(0)}K(x)dx$
$\displaystyle\geq$ $\displaystyle r^{2}\int_{\mathbb{R}^{N}\setminus
B_{r}(0)}K(x)dx.$

Then for any $s>1$, we have that

$\displaystyle
r^{2s}\int_{B_{r}^{c}(0)}K(x)dx=r^{2(s-1)}[r^{2}\int_{B_{r}^{c}(0)}K(x)dx]\to
0\quad{\rm as}\ r\to 0.$

Thus, $\inf\\{s\geq 0\ |\ \lim_{r\to
0^{+}}r^{2s}\int_{B_{r}^{c}(0)}K(y)dy=0\\}\leq 1$.

We now prove the first inequality (2.1). We denote

$s_{00}=\inf\\{s\geq 0\ |\ \lim_{r\to
0^{+}}r^{2s}\int_{B_{r}^{c}(0)}K(y)dy=0\\}.$

Since for any $s>s_{00}$, we have that

$\lim_{r\to 0^{+}}r^{2s}\int_{B_{r}^{c}(0)}K(x)dx=0.$

By the definition of $s_{0}$, we have $s_{0}\leq s$ and then by arbitrary of
$s>s_{00}$, we obtain that $s_{0}\leq s_{00}$.

_$(iii)$_ For any $s<s_{1}$ by (2.2), we have

$\displaystyle r^{2s}\int_{B_{r}^{c}(0)}K(x)dx$ $\displaystyle=$
$\displaystyle r^{2(s-s_{1})}[r^{2s_{1}}\int_{B_{r}^{c}(0)}K(x)dx]$
$\displaystyle\geq$ $\displaystyle
r^{2(s-s_{1})}\inf_{|x|\in(0,1)}(K(x)|x|^{N+2s_{1}})[r^{2s_{1}}\int_{r}^{1}\tau^{-2s_{1}-1}d\tau]$
$\displaystyle\geq$ $\displaystyle
r^{2(s-s_{1})}\int_{r}^{1}\tau^{-1}d\tau\inf_{|x|\in(0,1)}(K(x)|x|^{N+2s_{1}})$
$\displaystyle\to$ $\displaystyle\infty\quad{\rm as}\ r\to 0.$

By the definition of $s_{0}$, we have $s_{0}\geq s$ and then by arbitrary of
$s<s_{1}$, we obtain that $s_{0}\geq s_{1}$. Similarly to prove $s_{0}\leq
s_{2}$. $\Box$

###### Lemma 2.1

$(i)$ Assume that the kernel $K$ satisfies (1.4) and is continuous in
$\mathbb{R}^{N}\setminus\\{0\\}$, then $K$ is radially symmetric about the
origin.

$(ii)$ Assume that the kernel $K$ satisfies (1.2), (1.4) and (1.6) with
$s_{0}>0$. Then for any $s\in(0,s_{0})$, there exists a sequence $\\{r_{n}\\}$
of positive numbers which converges to 0 and

$\lim_{r_{n}\to 0^{+}}r_{n}^{N+2s}\inf_{|x|=r_{n}}K(x)=+\infty.$ (2.3)

Proof. $(i)$ By contradiction, we may assume that there exist
$x_{1},y_{1}\in\mathbb{R}^{N}\setminus\\{0\\}$ such that $|x_{1}|=|y_{1}|$ and
$K(x_{1})>K(y_{1})$. Since $K$ is continuous in
$\mathbb{R}^{N}\setminus\\{0\\}$, then there exists
$x_{2}\in\mathbb{R}^{N}\setminus\\{0\\}$ such that $|x_{2}|>|x_{1}|$ and

$K(x_{2})\geq K(x_{1})-\frac{K(x_{1})-K(y_{1})}{2}>K(y_{1}),$

which is impossible with the assumption (1.4).

$(ii)$ By Proposition 2.1 $(i)$, we have that for $s\in(0,s_{0})$ and
$\epsilon\in(0,s_{0}-s)$,

$\lim_{r\to 0^{+}}r^{2(s+\epsilon)}\int_{B_{r}^{c}(0)}K(x)dx=+\infty.$ (2.4)

Let $\tilde{K}(r)=\inf_{|x|=r}K(x)$, then by (1.4), we have
$\tilde{K}(r_{1})\leq\tilde{K}(r_{2})$ for $r_{1}\geq r_{2}$ and
$K(x)\leq\tilde{K}(r)$ for any $|x|>r$.

If (2.3) doesn’t hold, then there no exist any sequence $\\{r_{n}\\}$
converging to zero such that (2.3) holds, that is

$\limsup_{r\to 0^{+}}r^{N+2s}\tilde{K}(r)<+\infty.$

Together with $\tilde{K}$ is decreasing, then there exists $C>0$ such that

$\tilde{K}(r)\leq Cr^{-N-2s},\quad r\in(0,1).$

For any $x\in B_{1}(0)\setminus\\{0\\}$, we have
$K(x)\leq\tilde{K}(\frac{|x|}{2})$,

$\displaystyle r^{2(s+\epsilon)}\int_{B_{1}(0)\setminus B_{r}(0)}K(x)dx$
$\displaystyle\leq$ $\displaystyle r^{2(s+\epsilon)}\int_{B_{1}(0)\setminus
B_{r}(0)}\tilde{K}(\frac{|x|}{2})dx$ $\displaystyle\leq$ $\displaystyle
C2^{N+2s}r^{2(s+\epsilon)}\int_{r}^{1}\tau^{-1-2s}d\tau$ $\displaystyle\leq$
$\displaystyle Cr^{2\epsilon}.$

Together with (1.2), we have

$\lim_{r\to 0^{+}}r^{2(s+\epsilon)}\int_{B_{r}^{c}(0)}K(x)dx=0.$

which contradicts with (2.4). The proof is complete. $\Box$

To end this section, we construct an example of $K$ satisfying (1.5) for which
$s_{0}\in(0,1)$.

###### Example 2.1

Let

$K(x)=\left\\{\begin{array}[]{lll}a_{n}^{-N-2s},\ \
|x|\in[a_{n+1},a_{n}),\\\\[5.69054pt] |x|^{-N},\ \
|x|\in[a_{1},1),\\\\[5.69054pt] |x|^{-N-2s},\ \
|x|\in[1,+\infty).\end{array}\right.$ (2.5)

where $s\in(0,1)$, $a_{0}\in(0,1)$, $a_{n}=a_{0}^{b^{n}}$ with
$n\in\mathbb{N}$ and $b=\frac{N+2s}{N}$.
Then

$\liminf_{r\to 0^{+}}K(r)r^{N}=1\quad{\rm and}\quad s_{0}\in(0,s).$

Proof. We observe that $\lim_{n\to+\infty}a_{n}=0$ and

$\displaystyle
K(a_{n})a_{n}^{N}=a_{n-1}^{-N-2s}a_{n}^{N}=a_{0}^{-b^{n-1}(N+2s)}a_{n}^{N}=(a_{0}^{b^{n}})^{-N}a_{n}^{N}=1,$

then we have

$\liminf_{r\to 0^{+}}K(r)r^{N}=1.$

Combining Proposition 2.1 $(iii)$ and the fact of $\limsup_{r\to
0^{+}}K(r)r^{N+2s}\leq 1$, we have that

$s_{0}\in[0,s).$

Now we prove that $s_{0}>0$. For $r\in(0,a_{1})$, there exists
$n\in\mathbb{N}$ such that $a_{n+1}\leq r<a_{n}.$ If $n$ big enough, we have
$a_{n}\leq\frac{1}{2}a_{n-1}$. Combining with $b>1$, then

$\displaystyle\int_{B_{a_{1}(0)}\setminus B_{r}(0)}K(y)dy$ $\displaystyle=$
$\displaystyle|\omega_{N}|[(a_{n}-r)^{N}a_{n}^{-N-2s}+\sum_{k=2}^{n}(a_{k-1}-a_{k})^{N}a_{k-1}^{-N-2s}]$
$\displaystyle\geq$
$\displaystyle|\omega_{N}|\sum_{k=2}^{n}(a_{k-1}-a_{k})^{N}a_{k-1}^{-N-2s}$
$\displaystyle\geq$ $\displaystyle|\omega_{N}|2^{-N}a_{n-1}^{-2s},$

where $w_{N}$ is the unit sphere of $\mathbb{R}^{N}$. Choose
$\beta=b^{-2}s>0$, then we obtain that

$a_{n-1}^{-2s}\geq a_{n+1}^{-2\beta}.$

Therefore,

$\liminf_{r\to 0^{+}}r^{2\beta}\int_{B_{a_{1}(0)}\setminus B_{r}(0)}K(y)dy\geq
2^{-N}|\omega_{N}|.$

By Proposition 2.1 $(iii)$, we obtain that $s_{0}\geq\beta>0.$ $\Box$

##  3 Sobolev spaces

In this section, we will consider some embedding results inspired from [11].
First we introduce some basic spaces and some useful tools to prove embedding
theorems.

###### Lemma 3.1

Assume that $K$ satisfies (1.2), (1.4), (1.6) with $s_{0}>0$ and $l_{\infty}$
is defined by (1.7). Let $x\in\mathbb{R}^{N}$ and $E\subset\mathbb{R}^{N}$ be
a measurable set with $|E|\in(0,+\infty)$, then
$(i)$ if $l_{\infty}=0$, for any $s\in(0,s_{0})$, there exists $C>0$ such that

$\int_{E^{c}}K(x-y)dy\geq C|E|^{-\frac{2s}{N}};$ (3.1)

$(ii)$ if $l_{\infty}\in(0,\infty]$, there exists $C>0$ such that (3.1) holds
with $s\in(0,s_{0}]$.

Proof. We just need to prove that the conclusion of Lemma 3.1 holds for a
sequence $E_{n}$ with $|E_{n}|>0$ and $\lim_{n\to\infty}|E_{n}|=0$. Let
$\rho_{n}=(\frac{|E_{n}|}{\omega_{N}})^{1/N}$, then it follows that
$|E_{n}^{c}\cap B_{\rho_{n}}(x)|=|E_{n}\cap B_{\rho_{n}}^{c}(x)|$. Therefore,
by (1.4), we have that

$K(x-y)\geq\inf_{|z|=\rho_{n}}K(z),\quad y\in E_{n}^{c}\cap B_{\rho_{n}}(x),$
$K(x-y)\leq\inf_{|z|=\rho_{n}}K(z),\quad y\in
E_{n}\cap\bar{B}_{\rho_{n}}^{c}(x).$

Thus

$\displaystyle\int_{E_{n}^{c}}K(x-y)dy$ $\displaystyle=$
$\displaystyle\int_{E_{n}^{c}\cap
B^{c}_{\rho_{n}}(x)}K(x-y)dy+\int_{E_{n}^{c}\cap B_{\rho_{n}}(x)}K(x-y)dy$
(3.2) $\displaystyle\geq$ $\displaystyle\int_{E_{n}^{c}\cap
B^{c}_{\rho_{n}}(x)}K(x-y)dy+\inf_{|z|=\rho_{n}}K(z)|E_{n}^{c}\cap
B_{\rho_{n}}(x)|$ $\displaystyle\geq$ $\displaystyle\int_{E_{n}^{c}\cap
B^{c}_{\rho_{n}}(x)}K(x-y)dy+\inf_{|z|=\rho_{n}}K(z)|E_{n}\cap\bar{B}_{\rho_{n}}^{c}(x)|$
$\displaystyle=$ $\displaystyle\int_{B^{c}_{\rho_{n}}}K(x-y)dy.$

_$(i)$_ By Proposition 2.1 $(i)$ and $s_{0}>0$, we observe that for any
$s\in(0,s_{0})$

$\lim_{r\to 0^{+}}r^{2s}\int_{B_{r}^{c}(0)}K(y)dy=\infty.$ (3.3)

Then by (3.2), there exists $C>0$ such that

$\int_{E_{n}^{c}}K(x-y)dy\geq C|E_{n}|^{-\frac{2s}{N}}.$

_$(ii)$_ Since $l_{\infty}>0$, then there exists $\sigma\in(0,1)$ such that
for $r\in(0,1)$

$r^{2s_{0}}\int_{B_{r}^{c}(0)}K(y)dy\geq\sigma l_{\infty},$

which, together with (3.2), implies that

$\int_{E_{n}^{c}}K(x-y)dy\geq\sigma l_{\infty}|E_{n}|^{-\frac{2s_{0}}{N}}.$

For $s\in(0,s_{0})$, it is the same as the proof of $(i)$. $\Box$

###### Lemma 3.2

[11, Lemma 6.2] Assume that $s\in(0,1)$, $2s<N$ and $T>1$. Let
$n\in\mathbb{Z}$ and $\\{a_{k}\\}$ be a bounded, nonnegative, decreasing
sequence with $a_{k}=0$ for any $k\geq n$. Then,

$\displaystyle\sum_{k\in\mathbb{Z}}a_{k}^{1-\frac{2s}{N}}T^{k}\leq
C\sum_{k\in\mathbb{Z},a_{k}\not=0}a_{k+1}a_{k}^{-\frac{2s}{N}}T^{k},$

for a suitable constant $C=C(s,T,N)>0$, independent of $n$.

###### Lemma 3.3

Assume that $K$ satisfies (1.2), (1.4), (1.6) with $s_{0}\in(0,1)$ and
$l_{\infty}$ is defined by (1.7). Let $f\in L^{\infty}(\mathbb{R}^{N})$ be
compactly supported, then

$\int_{\mathbb{R}^{2N}}|f(x)-f(y)|^{2}K(x-y)dxdy\geq
C\sum_{k\in\mathbb{Z},a_{k}\not=0}a_{k+1}a_{k}^{-\frac{2s}{N}}2^{2k},$

where $a_{k}=|\\{|f|>2^{k}\\}|$, $k\in\mathbb{Z}$, $C=C(N,K)>0$ and the choice
of $s$ is the same as in Lemma 3.1.

Proof. The proof is similar to Lemma 6.3 in [11] just replaced the kernel by
$K$. For reader’s convenience, we give the detail below. Firstly, we assume
that $f$ is nonnegative. If not, we replace $f$ by $|f|$. Let
$A_{k}:=\\{f>2^{k}\\}$, $D_{k}:=A_{k}\setminus A_{k+1}$, $d_{k}:=|D_{k}|$ and
$S:=\sum_{j\in\mathbb{Z},a_{j-1}\not=0}2^{2j}a_{j-1}^{-\frac{2s}{N}}d_{j}.$
Then

$\\{(i,j)\in\mathbb{Z}^{2}\ s.t.\ a_{i-1}\not=0\ {\rm and}\
a_{j-1}^{-\frac{2s}{N}}d_{j}\not=0\\}\subset\\{(i,j)\in\mathbb{Z}^{2}\ s.t.\
a_{j-1}\not=0\\}.$ (3.4)

Then we have that

$\displaystyle\sum_{i\in\mathbb{Z},a_{i-1}\not=0}\sum_{j\in\mathbb{Z},j\geq
i+1}2^{2i}a_{i-1}^{-\frac{2s}{N}}d_{j}$ $\displaystyle=$
$\displaystyle\sum_{i\in\mathbb{Z},a_{i-1}\not=0}\sum_{j\in\mathbb{Z},j\geq
i+1,a_{i-1}^{s}d_{j}\not=0}2^{2i}a_{i-1}^{-\frac{2s}{N}}d_{j}$
$\displaystyle\leq$
$\displaystyle\sum_{i\in\mathbb{Z}}\sum_{j\in\mathbb{Z},j\geq
i+1,a_{i-1}\not=0}2^{2i}a_{i-1}^{-\frac{2s}{N}}d_{j}$ $\displaystyle=$
$\displaystyle\sum_{j\in\mathbb{Z},a_{j-1}\not=0}\sum_{i\in\mathbb{Z},i\leq
j-1}2^{2i}a_{i-1}^{-\frac{2s}{N}}d_{j}$ $\displaystyle\leq$
$\displaystyle\sum_{j\in\mathbb{Z},a_{j-1}\not=0}\sum_{i\in\mathbb{Z},i\leq
j-1}2^{2i}a_{j-1}^{-\frac{2s}{N}}d_{j}$ $\displaystyle=$
$\displaystyle\sum_{j\in\mathbb{Z},a_{j-1}\not=0}\sum_{k=0}^{+\infty}2^{2j-2}2^{-2k}a_{j-1}^{-\frac{2s_{0}}{N}}d_{j}$
$\displaystyle\leq$ $\displaystyle S.$

Fixed $i\in\mathbb{Z}$ and $x\in D_{i}$, for any $l\in\mathbb{Z}$ with $l\leq
i-2$ and any $y\in D_{l}$, we have that

$|f(x)-f(y)|\geq 2^{i-1}$

and therefore,

$\displaystyle\sum_{l\in\mathbb{Z},l\leq
i-2}\int_{D_{j}}|f(x)-f(y)|^{2}K(x-y)dy$ $\displaystyle\geq$ $\displaystyle
2^{2i-2}\sum_{l\in\mathbb{Z},l\leq i-2}\int_{D_{j}}K(x-y)dy$ $\displaystyle=$
$\displaystyle 2^{2i-2}\int_{A^{c}_{i-1}}K(x-y)dy.$

By Lemma 3.1, we have

$\sum_{l\in\mathbb{Z},l\leq i-2}\int_{D_{l}}|f(x)-f(y)|^{2}K(x-y)dy\geq
c_{0}2^{2i}a_{i-1}^{-\frac{2s}{N}},$

for some suitable $c_{0}>0$. As a consequence, for any $i\in\mathbb{Z}$,

$\sum_{l\in\mathbb{Z},l\leq i-2}\int_{D_{i}\times
D_{l}}|f(x)-f(y)|^{2}K(x-y)dxdy\geq c_{0}2^{2i}a_{i-1}^{-\frac{2s}{N}}d_{i}$

and then,

$\displaystyle\sum_{i\in\mathbb{Z},a_{i-1}\not=0}\sum_{l\in\mathbb{Z},l\leq
i-2}\int_{D_{i}\times D_{l}}|f(x)-f(y)|^{2}K(x-y)dxdy\geq c_{0}S.$

Thus, we obtain

$\displaystyle\sum_{i\in\mathbb{Z},a_{i-1}\not=0}\sum_{l\in\mathbb{Z},l\leq
i-2}\int_{D_{i}\times D_{l}}|f(x)-f(y)|^{2}K(x-y)dxdy$ $\displaystyle\geq
c_{0}[\sum_{i\in\mathbb{Z},a_{i-1}\not=0}2^{2i}a_{i-1}^{-\frac{2s}{N}}a_{i}-\sum_{i\mathbb{Z},a_{i-1}\not=0}\sum_{j\in\mathbb{Z},j\geq
i+1}2^{2i}a_{i-1}^{-\frac{2s}{N}}d_{j}]$ $\displaystyle\geq
c_{0}(2^{2i}a_{i-1}^{-\frac{2s}{N}}a_{i}-S).$

So, it follows that

$\displaystyle\int_{\mathbb{R}^{N}\times\mathbb{R}^{N}}|f(x)-f(y)|^{2}K(x-y)dxdy$
$\displaystyle\geq$ $\displaystyle
2\sum_{i\in\mathbb{Z},a_{i-1}\not=0}\sum_{l\in\mathbb{Z},l\leq
i-2}\int_{D_{i}\times D_{l}}|f(x)-f(y)|^{2}K(x-y)dxdy$ $\displaystyle\geq$
$\displaystyle
c_{0}(\sum_{i\in\mathbb{Z},a_{i-1}\not=0}2^{2i}a_{i-1}^{-\frac{2s}{N}}a_{i}).$

$\Box$

###### Lemma 3.4

Assume that $q\in[1,+\infty)$, $f:\mathbb{R}^{N}\to\mathbb{R}$ is a measurable
function. For any $n\in\mathbb{N}$,

$f_{n}(x):=\max\\{\min\\{f(x),n\\},-n\\},\quad\forall x\in\mathbb{R}^{N}.$

Then

$\lim_{n\to+\infty}\|f_{n}\|_{L^{q}(\mathbb{R}^{N})}=\|f\|_{L^{q}(\mathbb{R}^{N})}.$

Proof.  The details of the proof refers to [11, Lemma 6.4] or [2]. $\Box$

Now we can give the statement of embedding theorem as follows:

###### Theorem 3.1

Assume that $K$ satisfies (1.2), (1.4), (1.6) with $s_{0}>0$ and $l_{\infty}$
is defined by (1.7). Then
$(i)$ if $l_{\infty}=0$, then for $s\in(0,s_{0})$ there exists $C>0$ such that
for any $f\in X_{0}$, we have

$\|f\|_{L^{2^{*}(s)}(\Omega)}\leq
C(\int_{\mathbb{R}^{N}}\int_{\mathbb{R}^{N}}|f(x)-f(y)|^{2}K(x-y)dxdy)^{\frac{1}{2}};$
(3.5)

$(ii)$ if $l_{\infty}\in(0,\infty]$, then (3.5) holds with $s=s_{0}$.

Proof. First we note that

$\int_{\mathbb{R}^{N}}\int_{\mathbb{R}^{N}}|f(x)-f(y)|^{2}K(x-y)dxdy<+\infty.$
(3.6)

Without loss of generality, we can assume that $f\in
L^{\infty}(\mathbb{R}^{N})$. Indeed, let $f_{n}$ be defined as in Lemma 3.4,
then combining with Lemma 3.4 and (3.6), we make use of the Dominated
Convergence Theorem to imply

$\lim_{n\to\infty}\int_{\mathbb{R}^{2N}}|f_{n}(x)-f_{n}(y)|^{2}K(x-y)dxdy=\int_{\mathbb{R}^{2N}}|f(x)-f(y)|^{2}K(x-y)dxdy,$

which allows us to obtain estimate for function $f\in X_{0}$.

Take $s$, $a_{k}$ and $A_{k}$ defined as in Lemma 3.3, then we have that

$\displaystyle\|f\|^{2^{*}(s)}_{L^{2^{*}(s)}(\mathbb{R}^{N})}=\sum_{k\in\mathbb{Z}}\int_{A_{k}\setminus
A_{k+1}}|f(x)|^{2^{*}(s)}dx\leq\sum_{k\in\mathbb{Z}}2^{2^{*}(s)(k+1)}a_{k},$

that is,

$\|f\|^{2}_{L^{2^{*}(s)}(\mathbb{R}^{N})}\leq
4(\sum_{k\in\mathbb{Z}}2^{2^{*}(s)k}a_{k})^{2/2^{*}(s)}.$

Since $2<2^{*}(s)$, then

$\|f\|^{2}_{L^{2^{*}(s)}(\mathbb{R}^{N})}\leq
4\sum_{k\in\mathbb{Z}}2^{2k}a_{k}^{2/2^{*}(s)}.$

By Lemma 3.2 with $T=4$, it follows that

$\|f\|^{2}_{L^{2^{*}(s)}(\mathbb{R}^{N})}\leq
C\sum_{k\in\mathbb{Z}}2^{2k}a_{k+1}a_{k}^{-\frac{2s}{N}}.$

for a suitable constant $C$ depending on $N,K$.

Finally, it suffices to apply Lemma 3.4 to obtain that

$\|f\|_{L^{2^{*}(s)}(\mathbb{R}^{N})}\leq
C(\int_{\mathbb{R}^{N}}\int_{\mathbb{R}^{N}}|f(x)-f(y)|^{2}K(x-y)dxdy)^{\frac{1}{2}},$

up to relabeling the constant $C$. Since $f\in X_{0}$, $f=0$ in $\Omega^{c}$,
then (3.5) holds. $\Box$

###### Corollary 3.1

The norm (1.8) in $X_{0}$ is equivalent to

$\|f\|_{X_{0}}:=(\int_{\mathbb{R}^{N}}\int_{\mathbb{R}^{N}}|f(x)-f(y)|^{2}K(x-y)dxdy)^{\frac{1}{2}}.$
(3.7)

Proof. We only need to prove that there exists $C>0$ such that for any $f\in
X_{0}$,

$\|f\|_{X}\leq C\|f\|_{X_{0}}.$

It follows by Theorem 3.1 that

$\displaystyle\|f\|_{X}^{2}=\int_{\Omega}f^{2}(x)dx+\int_{\mathbb{R}^{N}}\int_{\mathbb{R}^{N}}|f(x)-f(y)|^{2}K(x-y)dxdy$
$\displaystyle\leq|\Omega|^{1-\frac{2}{2^{*}(s)}}(\int_{\Omega}|f|^{2^{*}(s)}(x)dx)^{\frac{2}{2^{*}(s)}}+\int_{\mathbb{R}^{N}}\int_{\mathbb{R}^{N}}|f(x)-f(y)|^{2}K(x-y)dxdy$
$\displaystyle\leq
C\int_{\mathbb{R}^{N}}\int_{\mathbb{R}^{N}}|f(x)-f(y)|^{2}K(x-y)dxdy.$

The proof is complete. $\Box$

###### Theorem 3.2

Assume that $K$ satisfies (1.2), (1.4), (1.6) with $s_{0}>0$ and $\mathcal{T}$
is a bounded subset of $X_{0}$.

Then $\mathcal{T}$ is pre-compact in $L^{q}(\Omega)$, $q\in[1,2^{*}(s_{0}))$.

Proof. We first prove that $\mathcal{T}$ is pre-compact in $L^{2}(\Omega)$. To
this end, we only show that $\mathcal{T}$ is totally bounded in
$L^{2}(\Omega)$. By Lemma 2.1$(ii)$, there exists $\\{r_{n}\\}$ positive and
convergent to 0 such that

$\lim_{n\to\infty}r_{n}^{N}K(r_{n})=+\infty.$

Let $\rho:\mathbb{R}_{+}\to\\{\frac{r_{n}}{2},n\in\mathbb{N}\\}$ such that,
denoting $\rho_{\epsilon}=\rho(\epsilon)$, for any $\epsilon>0$,
$\rho_{\epsilon}=r_{n}$ for some $n$ and

$\lim_{\epsilon\to 0^{+}}\rho_{\epsilon}=0.$

It is obvious that

$\lim_{\epsilon\to 0^{+}}(2\rho_{\epsilon})^{N}K(2\rho_{\epsilon})=+\infty.$
(3.8)

Let $\eta_{\epsilon}=\epsilon\rho_{\epsilon}^{\frac{N}{2}}$ and take a
collection of disjoints cubes $Q_{1},....,Q_{M}$ of side $\rho_{\epsilon}$
such that

$\Omega\subset\bigcup_{j=1}^{N}Q_{j}.$

For any $x\in\Omega$, there exists a unique integer $j(x)$ in $\\{1,...,M\\}$
such that $x\in Q_{j(x)}$. Let

$P(f)(x):=\frac{1}{|Q_{j(x)}|}\int_{Q_{j(x)}}f(y)dy,$

then $P$ is linear and $P(f)$ is constant in $Q_{j}$, which we denote by
$q_{j}(f)$. We define the linear operator $R$ by

$R(f)=\rho_{\epsilon}^{\frac{N}{2}}(q_{1}(f),...,q_{M}(f))\in\mathbb{R}^{M}$

and

$\|v\|_{2}:=(\sum^{M}_{j=1}|v_{j}|^{2})^{\frac{1}{2}},\quad
v\in\mathbb{R}^{M}.$

We observe that for any $f\in\mathcal{T}$,

$\displaystyle\|P(f)\|^{2}_{L^{2}(\Omega)}$ $\displaystyle=$
$\displaystyle\sum_{j=1}^{M}\int_{Q_{j}}|P(f)(x)|^{2}dx=\rho_{\epsilon}^{N}\sum_{j=1}^{M}|q_{j}(f)|^{2}$
$\displaystyle=$ $\displaystyle\|R(f)\|_{2}^{2}=\int_{\Omega}|f(y)|^{2}dy$
$\displaystyle=$ $\displaystyle\|f\|_{L^{2}(\Omega)}^{2}\ \leq C_{0}^{2}.$

Therefore, there exist $b_{1},.....b_{I}\in\mathbb{R}^{M}$ such that

$R(\mathcal{T})\subset\bigcup_{i=1}^{I}B_{\eta_{\epsilon}}(b_{i}),$

where the balls $\\{B_{\eta_{\epsilon}}\\}$ are taken in $\mathbb{R}^{M}$. For
any $x\in\Omega$, we set

$\beta_{j}(x)=\rho_{\epsilon}^{-\frac{N}{2}}b_{i,j(x)},$

where $b_{i,j(x)}$ is the $j(x)$th coordinates of $b_{i}$. Noticing that
$\beta_{j}$ is constant on $Q_{j}$, i.e. for $x\in Q_{j}$, it follows that

$P(\beta_{i})(x)=\rho_{\epsilon}^{-\frac{N}{2}}b_{i,j}=\beta_{i}(x)$

and so $q_{j}(\beta_{i})=\rho_{\epsilon}^{-\frac{N}{2}}b_{i,j}$. Thus
$R(\beta_{i})=b_{i}$. Furthermore, for any $f\in\mathcal{T}$

$\displaystyle\|f-P(f)\|^{2}_{L^{2}(\Omega)}$ $\displaystyle=$
$\displaystyle\sum_{j=1}^{M}\int_{Q_{j}}|f(x)-P(f)(x)|^{2}dx$ $\displaystyle=$
$\displaystyle\sum_{j=1}^{M}\int_{Q_{j}}\frac{1}{|Q_{j}|^{2}}|\int_{Q_{j}}f(x)-f(y)dy|^{2}dx$
$\displaystyle\leq$
$\displaystyle\frac{1}{\rho_{\epsilon}^{2N}}\sum_{j=1}^{M}\int_{Q_{j}}[\int_{Q_{j}}|f(x)-f(y)|dy]^{2}dx$

and for any fixed $j\in\\{1,...,M\\}$, by Hölder inequality, we get

$\displaystyle\frac{1}{\rho_{\epsilon}^{2N}}[\int_{Q_{j}}|f(x)-f(y)|dy]^{2}\leq\frac{1}{\rho_{\epsilon}^{2N}}|Q_{j}|\int_{Q_{j}}|f(x)-f(y)|^{2}dy$
$\displaystyle\qquad\qquad\leq\frac{1}{\rho_{\epsilon}^{N}}\frac{1}{K(2\rho_{\epsilon})}\int_{Q_{j}}|f(x)-f(y)|^{2}K(x-y)dy$
$\displaystyle\qquad\qquad\qquad\qquad\leq\frac{1}{\rho_{\epsilon}^{N}K(2\rho_{\epsilon})}\|f\|_{X}^{2},$

where $K(2\rho_{\epsilon})=\inf_{|x|=2\rho_{\epsilon}}K(x)$. Therefore,

$\displaystyle\|f-P(f)\|^{2}_{L^{2}(\Omega)}\leq\frac{1}{\rho_{\epsilon}^{N}K(2\rho_{\epsilon})}\|f\|_{X}^{2}\sum_{j=1}^{M}|Q_{j}|\leq\frac{C}{\rho_{\epsilon}^{N}K(2\rho_{\epsilon})}.$
(3.9)

Consequently, for any $f$, there exists $j\in\\{1,....M\\}$ such that $P(f)\in
B_{\eta_{\epsilon}}(b_{j})$ and then we derive that

$\displaystyle\|f-\beta_{j}\|_{L^{2}(\Omega)}$
$\displaystyle\quad\leq\|f-P(f)\|_{L^{2}(\Omega)}+\|P(f)-P(\beta_{j})\|_{L^{2}(\Omega)}+\|P(\beta_{j})-\beta_{j}\|_{L^{2}(\Omega)}$
$\displaystyle\quad\leq\frac{C}{\rho_{\epsilon}^{N}K(2\rho_{\epsilon})}+\frac{\|R(f)-R(\beta_{j})\|_{L^{2}(\Omega)}}{\rho_{\epsilon}^{\frac{N}{2}}}$
$\displaystyle\quad\leq\frac{C}{\rho_{\epsilon}^{N}K(2\rho_{\epsilon})}+\frac{\eta_{\epsilon}}{\rho_{\epsilon}^{\frac{N}{2}}},$

where by (3.8), $\frac{1}{(2\rho_{\epsilon})^{N}K(2\rho_{\epsilon})}\to 0$ as
$\epsilon\to 0$ and $\frac{\eta_{\epsilon}}{\rho_{\epsilon}^{N/2}}=\epsilon$.
As a consequence, $\mathcal{T}$ is pre-compact in $L^{2}(\Omega)$.

Now we are in the position to prove that $\mathcal{T}$ is pre-compact in
$L^{q}(\Omega)$ with $q\in[1,2^{*}(s_{0}))$. Since $L^{2}(\Omega)\subset
L^{q}(\Omega)$ with $q\in[1,2)$, then $\mathcal{T}$ is pre-compact in
$L^{q}(\Omega)$. For $q\in(2,2^{*}(s_{0}))$, there exists $s\in(0,s_{0})$ such
that $q<2^{*}(s)$, then using Hölder inequality with
$\theta=\frac{2(2^{*}(s)-q)}{q(2^{*}(s)-2)}$, we get that

$\displaystyle\|f-\beta_{j}\|_{L^{q}(\Omega)}$ $\displaystyle=$
$\displaystyle\left(\int_{\Omega}|f-\beta_{j}|^{\theta
q}|f-\beta_{j}|^{q(1-\theta)}dx\right)^{\frac{1}{q}}$ $\displaystyle\leq$
$\displaystyle\||f-\beta_{j}|\|^{\frac{\theta}{2}}_{L^{2}(\Omega)}\||f-\beta_{j}|\|^{\frac{1}{q}-\frac{\theta}{2}}_{L^{2^{*}(s)}(\Omega)}$
$\displaystyle\leq$
$\displaystyle\left(\frac{C}{\rho_{\epsilon}^{N}K(2\rho_{\epsilon})}+\frac{\eta_{\epsilon}}{\rho_{\epsilon}^{\frac{N}{2}}}\right)^{\frac{\theta}{2}}\|f\|_{X}^{\frac{1}{q}-\frac{\theta}{2}},$

thus, $\mathcal{T}$ is pre-compact in $L^{q}(\Omega)$ with
$q\in(2,2^{*}(s_{0}))$. The proof ends. $\Box$

Proof of Theorem 1.1. For Theorem 1.1 part $(i)$, let $(f_{n})$ be a sequence
functions in $X_{0}$ such that

$\|f_{n}\|_{X}\leq C,\quad\forall n\in\mathbb{N}$

where $C>0$. By Theorem 3.1, Inequality (1.10) follows by (3.5). We obtain
that the sequence $(f_{n})$ is pre-compact in $L^{q}$ with
$q\in[1,2^{*}(s_{0}))$, then the compactness in Theorem 1.1 follows. $\Box$

##  4 Existence of weak solution to (1.1)

For the proof of Theorem 1.1, we observe that problem (1.1) has a variational
structure, indeed it is the Euler-Lagrange equation of the functional
$\mathcal{J}:X_{0}\to\mathbb{R}$ defined as follows

$\mathcal{J}(u)=\frac{1}{2}\|u\|_{X_{0}}^{2}-\frac{1}{p}\int_{\Omega}|u|^{p}dx.$

Note the functional $\mathcal{J}$ is Fréchet differentiable in $u\in X_{0}$
and for any $\varphi\in X_{0}$,

$\langle\mathcal{J}^{\prime}(u),\varphi\rangle=\int_{Q}\big{(}u(x)-u(y)\big{)}\big{(}\varphi(x)-\varphi(y)\big{)}K(x-y)dxdy-\int_{\Omega}|u|^{p-2}u(x)\varphi(x)dx.$

We will make use of Mountain Pass theorem to obtain the weak solution. In what
follows, we check the structure condition of Mountain Pass theorem. It is
obvious that $\mathcal{J}(0)=0$.

###### Proposition 4.1

Under the hypotheses of Theorem 1.2, there exist $\rho>0$ and $\beta>0$ such
that $\mathcal{J}(u)\geq\beta$, for any $u\in X_{0}$ with
$\|u\|_{X_{0}}=\rho$.

Proof. Let $u\in X_{0}$, then

$\displaystyle\mathcal{J}(u)$ $\displaystyle=$
$\displaystyle\frac{1}{2}\|u\|_{X_{0}}^{2}-\frac{1}{p}\int_{\Omega}|u(x)|^{p}\,dx$
$\displaystyle\geq$
$\displaystyle\frac{1}{2}\|u\|_{X_{0}}^{2}-C\|u\|_{X_{0}}^{p}$
$\displaystyle=$
$\displaystyle\frac{1}{2}\|u\|_{X_{0}}^{2}(1-C\|u\|_{X_{0}}^{p-2}),$

where we used Theorem 1.1 and Corollary 3.1 for the inequality. We choose
$\sigma>0$ such that $1-C\sigma^{\frac{p-2}{2}}=\frac{1}{2}$, since $p>2$.
Then for $\|u\|_{X_{0}}^{2}=\sigma$, $1-C\|u\|_{X_{0}}^{p-2}=\frac{1}{2}$,
then we have

$\mathcal{J}(u)\geq\frac{1}{4}\sigma.$

The proof is complete. $\Box$

###### Proposition 4.2

Under the hypotheses of Theorem 1.2, there exists $e\in X_{0}$ such that
$\|e\|_{X_{0}}>\rho$ and $\mathcal{J}(e)\leq 0$, where $\rho$ is given in
Proposition 4.1.

Proof. We fix a function $u_{0}\in X_{0}$ with $\|u_{0}\|=1$ in $\Omega$.
Since the space of $\\{tu_{0}:t\in\mathbb{R}\\}$ is a subspace of $X_{0}$ with
dimension 1 and all the norms are equivalent, then
$\int_{\Omega}|u_{0}(x)|^{p}dx>0$. Then there exists $t_{0}>0$ such that for
$t\geq t_{0}$,

$\displaystyle\mathcal{J}(tu_{0})$ $\displaystyle=$
$\displaystyle\frac{t^{2}}{2}\|u_{0}\|_{X_{0}}^{2}-\frac{t^{p}}{p}\int_{\Omega}|u_{0}(x)|^{p}dx$
$\displaystyle\leq$ $\displaystyle C(t^{2}-t^{p})\leq 0.$

We choose $e=t_{0}u_{0}$. The proof is complete. $\Box$

We say that $\mathcal{J}$ has $P.S.$ condition, if for any sequence
$\\{u_{n}\\}$ in $X_{0}$ satisfying $\mathcal{J}(u_{n})\to c$ and
$\mathcal{J}^{\prime}(u_{n})\to 0$ as $n\to\infty$, there is a convergent
subsequence, where $c\in\mathbb{R}$.

###### Proposition 4.3

Under the hypotheses of Theorem 1.2, $\mathcal{J}$ has $P.S.$ condition in
$X_{0}$.

Proof. Let $\\{u_{n}\\}$ be a $P.S.$ sequence, then we need to show that there
are a subsequence $\\{u_{n_{k}}\\}$ and $u$ such that

$u_{n_{k}}\to u\quad{\rm in}\ \ L^{p}(\Omega)\quad{\rm as}\ k\to\infty.$

For some $C>0$, we have that

$\displaystyle
C\|u_{n}\|_{X_{0}}\geq\mathcal{J}^{\prime}(u_{n})u_{n}=\|u_{n}\|^{2}_{X_{0}}-\int_{\Omega}|u_{n}|^{p}dx$
(4.1)

and

$\displaystyle
c-1\leq\mathcal{J}(u_{n})=\frac{1}{2}\|u_{n}\|^{2}_{X_{0}}-\frac{1}{p}\int_{\Omega}|u_{n}|^{p}dx.$
(4.2)

Then $p\times$(4.2)-(4.1) implies that

$(\frac{p}{2}-1)\|u_{n}\|^{2}_{X_{0}}\leq c+C\|u_{n}\|_{X_{0}},$

then $u_{n}$ is uniformly bounded in $X_{0}$.

Thus, by Theorem 1.1 and Corollary 3.1, there exists a subsequence
$(u_{n_{k}})$ and $u$ such that

$u_{n_{k}}\rightharpoonup u,\quad{\rm in}\quad X_{0},$ $u_{n_{k}}\to
u,\quad{\rm a.e.\ in}\ \Omega\quad{\rm and\ \ in}\quad L^{p}(\Omega),$

when $k\to\infty$. Together with $\lim_{k\to\infty}\mathcal{J}(u_{n_{k}})=c$,
we have $\|u_{n_{k}}\|_{X_{0}}\to\|u\|_{X_{0}}$ as $k\to\infty$. Then we have
$u_{n_{k}}\to u$ in $X_{0}$ as $k\to\infty$. $\Box$

Proof of Theorem 1.2. By Proposition 4.1, Proposition 4.2 and Proposition 4.3,
we may use Mountain Pass Theorem (for instance, [22, Theorem 6.1]; see also
[1, 17]) to obtain that there exists a critical point $u\in X_{0}$ of
$\mathcal{J}$ at some value $c\geq\beta>0$. By $\beta>0$, we have $u$ is
nontrivial. Therefore, (1.1) admits a nonnegative weak solution. The proof is
complete. $\Box$

###### Remark 4.1

Suppose that $s_{0}\in(0,1)$ and $f:\Omega\times\mathbb{R}\to\mathbb{R}$ is a
Carathéodory function verifying the following hypothesis:

  * $(f_{1})\ $

    1. there exist $a_{1},a_{2}>0$ and $q\in(2,2^{*}(s_{0}))$ such that

$|f(x,t)|\leq a_{1}+a_{2}|t|^{q-1}\quad a.e.\ x\in\Omega,\ t\in\mathbb{R};$

  * $(f_{2})\ $

    1. $\lim_{t\to 0}\frac{f(x,t)}{|t|}=0\quad{\rm uniformly\ in}\ x\in\Omega;$

  * $(f_{3})\ $

    1. there exist $\mu>2$ and $r>0$ such that a.e. $x\in\Omega,t\in\mathbb{R},|t|\geq r$

$0<\mu F(x,t)\leq tf(x,t),$

where the function $F$ is the primitive of $f$ with respect to the variable
$t$, that is

$F(x,t)=\int_{0}^{t}f(x,\tau)d\tau.$

Then fractional elliptic problem (1.1) admits a nontrivial weak solution.

Proof. Using the technique in the proof of Theorem 1 in [20] and Theorem 1.1
part $(ii)$, we derive a nontrivial weak solution of (1.1) by Mountain Pass
Theorem. $\Box$

## References

  * [1] A. Ambrosetti and P. Rabinowitz, Dual variational methods in critical point theory and applications, J. Funct. Anal. 14, 349–381 (1973).
  * [2] A. Burchard and H. Hajaiej, Rearrangement inequalities for functionals with monotone integrands, J. Funct. Anal. 2, 561-582 (2006).
  * [3] H. Brezis, Functional analysis, Sobolev spaces and partial differential equations, Springer (2010).
  * [4] L. Caffarelli and L. Silvestre, An extension problem related to the fractional laplacian, Comm. Partial Differential Equations 32, 1245-1260 (2007).
  * [5] L. Caffarelli and L. Silvestre, Regularity theory for fully non-linear integrodifferential equations, Comm. Pure Appl. Math. 62, 597-638 (2009).
  * [6] L. Caffarelli and L. Silvestre, Regularity results for nonlocal equations by approximation, Arch. Ration. Mech. Anal. 200(1), 59-88 (2011).
  * [7] H. Chen, P. Felmer and A. Quaas, Large solution to elliptic equations involving fractional Laplacian, Accepted by Ann. Ins.Henri Poincaré, arXiv:1311.6044 (2013)
  * [8] H. Chen and L. Véron, Semilinear fractional elliptic equations involving measures, Accepted by J. Diff. Eq., arXiv:1305.0945 (2013).
  * [9] H. Chen and L. Véron, Semilinear fractional elliptic equations with gradient nonlinearity involving measures, J. Funct. Anal., 266(8), 5467-5492 (2014).
  * [10] H. Chen and L. Véron, Weak and strong singular solutions of semilinear fractional elliptic equations, Accepted by Asymp. Anal., arXiv:1307.7023 (2013).
  * [11] E. Di Nazza, G.Palatucci and E. Valdinoci, Hitchhiker’s guide to the fractional Sobolev spaces, Bull. Sci. Math., 136 (5), 521-573 (2012).
  * [12] P. Felmer, A. Quaas and J. Tan, Positive solutions of non-linear Schrödinger equation with the fractional laplacian, Proc. Roy. Soc. Edinburgh., 142, 1237-1262 (2012).
  * [13] P. Felmer and Y. Wang, Radial symmetry of positive solutions to equations involving the fractional laplacian, Accepted by Comm. Contem. Math.
  * [14] H. Hajaiej, Variational problems related to some fractional kinetic equations, arXiv:1205.1202 (2012).
  * [15] H. Hajaiej, On the optimality of the conditions used to prove the symmetry of the minimizers of some fractional constrained variational problems, Ann. Inst. H. Poincaré 14(5), 1425-1433 (2013).
  * [16] H. Hajaiej, Existence of minimizers of functionals involving the fractional gradient in the abscence of compactness, symmetry and monotonicity, J. Math. Anal. Appl. 399(1), 17-26 (2013).
  * [17] P. H. Rabinowitz, Minimax methods in critical point theory with applications to differential equations, CBMS Reg. Conf. Ser. Math., 65, American Mathematical Society, Providence, RI (1986).
  * [18] X. Ros-Oton and J. Serra, The Dirichlet problem for the fractional laplacian: regularity up to the boundary, J. Math. Pures Appl., 101(3), 275-302 (2014)
  * [19] O. Savin and E. Valdinoci, Density estimates for a nonlocal variational model via the Sobolev inequality, SIAM J. Math. Anal., 43(6), 2675-2687 (2011).
  * [20] R. Servadei and E. Valdinoci, Moutain Pass solutions for non-local elliptic operators, J. Math. Anal. Appl. 389(2), 887-898 (2012).
  * [21] Y. Sire and E. Valdinoci, Fractional laplacian phase transitions and boundary reactions: a geometric inequality and a symmetry result, J. Funct. Anal. 256, 1842-1864 (2009).
  * [22] M. Struwe, Variational methods, applications to nonlinear partial differential equations and Hamiltonian systems, Ergebnisse der Mathematik und ihrer Grenzgebiete, 3, Springer Verlag, Berlin–Heidelberg (1990).


# HHF: Hashing-guided Hinge Function for Deep Hashing Retrieval

Chengyin Xu∗, Zenghao Chai∗, Zhengzhuo Xu∗,
Hongjia Li, Qiruyi Zuo, Lingyu Yang, Chun Yuan†, Senior Member, IEEE
†Corresponding author: C. Yuan is with the Tsinghua Shenzhen International
Graduate School, Tsinghua University, Shenzhen 518055, China. (email:
yuanc@sz.tsinghua.edu.cn). ∗Equal contribution authors, listing order is
random: C. Xu, Z. Chai and Z. Xu are with the Tsinghua Shenzhen International
Graduate School, Tsinghua University, Shenzhen 518055, China. (e-mail:
{xucy20<EMAIL_ADDRESS>zenghaochai@gmail.com). H. Li, Q. Zuo,
and L. Yang are with the Tsinghua Shenzhen International Graduate School,
Tsinghua University, Shenzhen 518055, China. (e-mail: {lhj20, zqry20,
yly20}@mails.tsinghua.edu.cn).

###### Abstract

Deep hashing has shown promising performance in large-scale image retrieval.
However, latent codes extracted by Deep Neural Networks (DNNs) will inevitably
lose semantic information during the binarization process, which damages the
retrieval accuracy and makes it challenging. Although many existing approaches
perform regularization to alleviate quantization errors, we figure out an
incompatible conflict between metric learning and quantization learning. The
metric loss penalizes the inter-class distances to push different classes
unconstrained far away. Worse still, it tends to map the latent code deviate
from ideal binarization point and generate severe ambiguity in the
binarization process. Based on the minimum distance of the binary linear code,
we creatively propose Hashing-guided Hinge Function (HHF) to avoid such
conflict. In detail, the carefully-designed inflection point, which relies on
the hash bit length and category numbers, is explicitly adopted to balance the
metric term and quantization term. Such a modification prevents the network
from falling into local metric optimal minima in deep hashing. Extensive
experiments in CIFAR-10, CIFAR-100, ImageNet, and MS-COCO show that HHF
consistently outperforms existing techniques, and is robust and flexible to
transplant into other methods. Code is available at
https://github.com/JerryXu0129/HHF.

###### Index Terms:

Image retrieval, deep hash, metric learning, quantization learning

##  I Introduction

Large-scale image retrieval has reached glary attention in the field of
computer vision and multimedia computing over the past years [1, 2, 3, 4, 5,
6]. With the arrival of the era of big data, billions of images are uploaded
to social platforms and search engines day and night. Therefore, how to
effectively retrieve approximate nearest neighbors of given queries from
massive image databases is a very meaningful research topic with broad
applications.

Figure 1: (a): Standard pipeline for image retrieval and corresponding loss
functions with the proposed HHF. (b): the red cluster is balanced under the
opposite gradient direction from metric loss and the quantization loss. The
possibility of samples being error hashed in binarization increases. (c) the
red cluster is only affected by the quantization loss until it reaches the
hashing position with the help of HHF.

Image retrieval, especially content-based retrieval, adopts hashing [7, 8, 9]
to handle high-dimensional and large-scale data for efficient retrieval and
low storage overhead. In light of the promising performance of Deep Neural
Networks (DNNs) in feature extraction, Deep Hashing [10, 11, 12, 13] has
become an important research branch in recent years, which aims at mapping
high-dimensional feature vectors extracted by DNNs into low-dimensional
vectors through specific hash functions. The encoded binary hash codes enable
to minimize the storage overhead and maintain the distinguishable feature of
given 2D images [10, 11, 12, 13].

One of the challenges in Deep Hashing lies in semantic information
disappearance in the binarization process (i.e., from continuous latent codes
to discrete hash codes). Numerous researchers have carried out in-depth
efforts to tackle inevitable information loss in binarization. HashNet [11] is
proposed to tackle information loss by a novel activation strategy. CSQ [14],
OrthoHash [15] and DCSH [16] constrain the samples to be aligned with fixed or
learnable hash centers to alleviate the information loss. DHN [12] firstly
proposed quantization loss during training and demonstrated its flexibility in
other works [17, 18, 19, 20, 21, 22, 23] by achieving promising performance
gains.

However, we figure out an incompatible conflict between metric loss and
quantization loss of previous works [12, 19, 24]. As Fig. 1(b) illustrates,
the metric loss enforces samples of different classes to be alienated and
cluster the same categories. The gradient direction from the metric loss for
the red cluster is opposite to that from the quantization loss. Such universal
conflict causes the clusters to deviate from the ideal hashing position when
converged, which results in error hashed issues and harms retrieval results.

To tackle the above challenges without sacrificing metric learning, the
elaborate-designed Hashing-guided Hinge Function (HHF) is proposed to overcome
such a dilemma between metric learning and quantization learning. In specific,
we propose a novel threshold $\zeta$, determined by hash bit length and
category numbers, to prevent inter-distance among different classes (or
proxies) being unlimitedly alienated (see Fig. 1(c)). Inspired by the minimum
distance of the binary linear code [25], we deduce the proper $\zeta$ in
different cases and generate a table look-at. The proposed hinge threshold
controls negative samples to be punished iff their inter-distance is less than
$\zeta$, and finally, keep stabilizing at the hashing position.

We improve the linear correlated cosine similarity in naïve metric loss with
the proposed HHF that contains: a). an inflection point $\zeta$ to prevent
negative samples unlimited alienated, and b). a relaxation margin $\delta$ to
prevent simply mapping similar samples to the same latent codes. We
comprehensively validate that the proposed HHF can be easily plugged-and-play
and consistently improve the performance among various methods, which shows
great superiority and robustness in CIFAR-10, CIFAR-100, ImageNet, and MS-COCO
datasets. Our contributions are summarized as follows:

  * •

We pinpoint the conflict between metric loss and quantization loss, which is
proved harmful for models to converge at the optimal hashing solution. Such a
conflict will severely increase error hashed risks that lead to wrong
retrieval results.

  * •

We overcome such a universal conflict by proposing HHF to modify the metric
loss term. HHF is robust in various hash bit lengths and category numbers. We
verify the flexibility of HHF with plenty of methods. The proposed HHF
achieves better retrieval results and reaches the ideal hash position without
deteriorating metric learning.

  * •

We conduct extensive experiments to demonstrate that HHF outperforms existing
techniques and achieves remarkable state-of-the-art performance gains on four
public benchmarks. We will release our code for research purposes in the image
retrieval field.

The rest of this paper is organized as follows: Section II briefly reviews the
related work. Section III describes the proposed Hashing-guided Hinge
Function, which solves the contradiction between metric learning and
quantization learning. Section IV demonstrates the effectiveness and
flexibility of the proposed method by extensive experiments on four
benchmarks, and finally, Section V concludes our work.

##  II Related Work

Deep Neural Network (DNN), especially Convolution Neural Network (CNN), has
shown great superiority in feature extraction of 2D images [26]. Deep Hashing
[10, 27] has surpassed traditional hashing methods [1, 2, 3, 5, 4, 6] and
becomes the mainstream for image retrieval. In large-scale image retrieval,
Deep Hashing related methods are mainly composed of two parts: 1). metric
learning and 2). binary hashing. Therefore, according to different concerns,
previous work can be divided into two categories: methods focusing on 1).
metric learning and 2). binary hashing. Below we review the methods that are
most closely related to our work, and a full in-depth review can be found in
[7, 8, 9].

Methods focusing on metric learning can be classified into the pair-based and
proxy-based methods. Contrastive Loss [28, 29] and Pair-wise Loss [17, 11, 18,
12] are primary pair-based methods that mine data-to-data relations from the
sample pairs by metric learning. However, Pair-wise Loss only correlates given
pairs but ignores the relationship between other categories. Hence it will
easily drop into locally optimal solutions and cannot deal with hard pair
challenges.

To tackle the challenge, Triplet Loss [30, 27, 19] associates the anchor input
with a positive and a negative sample. It tackles the hard triplet by
comparing the distance between the anchor and the two samples. Based on
Triplet Loss, Sohn et al. [31] propose $N$-pair Loss that contains an
efficient batch construction strategy to optimize one positive sample and
$N-1$ negative samples simultaneously, which speeds up convergence and reduces
computational complexity. Lifted Structure Loss [32] also considers the
distance among one positive anchor and multiple negative anchors, and
penalizes them according to their hardness, alleviating the challenging hard
pair problem. Although pair-based losses [17, 30] can sufficiently exploit the
relationships of given samples and accurately model the distance of each
category, the large computational overhead limits these methods to consider
the whole datasets but only constrains on a single mini-batch. As a result,
metric learning based on pair-based methods only contains limited local
information.

Proxy-based methods are proposed to alleviate the computation complexity of
pair-based losses. Proxy-NCA [13] replaces the pair-wise error with a set of
learnable class centers (i.e., proxies), and calculates the distance between
each proxy and positive/negative samples. Because the category number is far
less than samples, Proxy-based losses significantly improve the convergence
speed by only considering the distance of each sample and proxy. Combining the
advantages of proxy and $N$-pair Loss, Manifold Proxy Loss [33] utilizes the
manifold-aware distance as the metric in the feature space to improve
performance further. Proxy-Anchor Loss [24] serves as the state-of-the-art
proxy-based method for metric learning. It integrates both advantages of the
two methods that adopts LogSumExp function to excavate semantic relations of
different samples and solves hard sample challenges effectively.

Methods focusing on binary hashing aim at dealing with information loss. The
latent codes obtained by DNNs will be hashed into discrete $\pm 1$ for easy
storage and fast retrieval. However, plenty of continuous semantic information
will get lost during the binarization process. To tackle the problem, HashNet
[11] optimizes the pair-wise loss with non-smooth binary activations, and
addresses the gradient difficulty in optimization via sign activation function
$\text{sgn}(\cdot)$. The gradient of Tanh is close to $0$ in the domain, while
the considerable distance between the latent codes of similar samples would
lead to a minor penalty, hence it cannot guarantee all similar samples well
clustered in the feature space.

In addition, some methods directly learn the semantic relationship between
samples and the predefined hash centers. CSQ [14] constructs fixed hash center
sets randomly or based on the Hadamard matrix, which alleviates information
loss during the binarization process. Based on the orthogonal hashing centers
[14], OrthoHash [15] optimizes the distance between samples and hash centers
by proposing Scaled Cosine Similarity loss. However, the artificially defined
hash centers miss the semantic relationship of intra-class and inter-class.
The Hadamard matrix-based or randomly generated codes are either inflexible to
deal with the changed length and category number or cannot achieve
satisfactory performance.

Compared to those hash center fixed methods [14, 15], DCWH [21] and IDCH [22]
set learnable hash centers and optimize them by calculating the sample offset
and voting per bit in a mini-batch. DCSH [16] obtains the latent codes in each
epoch and calculates the average of each bit to obtain the binarized hash
centers. Pair-wise loss based DHN [12] integrates quantization loss with
metric loss to constrain latent codes, and controls the quantization error to
improve hashing quality. Most recent works [17, 18, 19, 14, 21, 22, 23] add
the quantization loss to reduce information loss during binarization and show
promising performance gains.

However, we figure out an incompatible conflict when combining metric loss
with quantization loss, which causes much information loss during the
binarization process. Hence, the carefully designed HHF overcomes such
contradiction, which can be applied in most existing techniques and achieve
significant performance gains.

##  III Methodology

###  III-A Problem Definition

Given $M$ images $\mathcal{X}=\\{x_{i}\\}_{i=1}^{M}$ and corresponding label
matrix $\mathcal{Y}\in\mathds{R}^{C\times M}$, where $C$ is the category
numbers of the whole dataset, Deep Hashing learns a feature extractor
$\mathcal{E}$ parameterized by $\Theta$ and a hashing function $\mathcal{H}$.
For each $x_{i}\in\mathcal{X}$, we first obtain the latent code
$\bm{h_{i}}=\\{h_{1}h_{2}\cdots h_{K}|h_{i}\in\mathbb{R}\\}\in\mathbb{R}^{K}$
by $\mathcal{E}$ with the supervision of a series of losses:
$\bm{h_{i}}\leftarrow\mathcal{E}(x_{i}|\Theta)$. Since we adopt the
binarization as hashing function $\mathcal{H}$, the latent code $\bm{h_{i}}$
is quantizated into a compact $K$-bit binary code
$\bm{b_{i}}=\\{b_{1}b_{2}\cdots b_{K}|b_{i}\in\\{\pm 1\\}\\}$ by the hashing
function $\mathcal{H}$: $\bm{b_{i}}\leftarrow\mathcal{H}(\bm{h_{i}})$. We
reasonably assume that the hashing code $\bm{b}$ in the Hamming space shares
the similar semantic information with the representation $\bm{h_{i}}$ in the
feature space. For given query image $x_{q}$, we sort the hash codes for all
the samples in the database according to their Hamming distance, and return
the top-$N$ images as the query results. The core challenge of this task is to
learn a reliable feature extractor $\mathcal{E}$ to cluster images of
different categories with proper and distinguishable hash positions.

###  III-B Loss Functions in Image Retrieval

Metric Loss. Without loss of generality, for any two $K$-dimension binary code
$\bm{b_{1}},\bm{b_{2}}\in\\{\pm 1\\}^{K}$, the Hamming distance estimates the
similarity of the two binary codes, which is defined as the summary of XOR
operation per bits as Eq. 1.

$\displaystyle\mathbb{D}_{\text{Ham}}(\bm{b_{1}},\bm{b_{2}})$
$\displaystyle=\sum_{i=1}^{K}\mathds{1}({b_{1}}_{i}\neq{b_{2}}_{i})$ (1)

However, in practice, the Hamming distance in Eq. 1 is non-differentiable,
which is not qualified to measure the distance among latent codes for
training. Considering the Hamming distance
$\mathds{D}_{\text{Ham}}(\bm{b_{1}},\bm{b_{2}})$ is correlated to cosine
similarity $\cos(\bm{b_{1}},\bm{b_{2}})$, cosine similarity is adopted instead
to approximate the distance between latent code pairs $\bm{h_{1}},\bm{h_{2}}$
for backpropagation. i.e., the distance between them only depends on their
angle in hypersphere space as Eq. 2.

$\displaystyle\mathbb{D}_{\text{Ham}}(\text{sgn}(\bm{h_{1}}),$
$\displaystyle\text{sgn}(\bm{h_{2}}))=\frac{K}{2}\left(1-\frac{\bm{b_{1}}\cdot\bm{b_{2}}}{||\bm{b_{1}}||\cdot||\bm{b_{2}}||}\right)$
(2)
$\displaystyle\propto-\cos(\bm{b_{1}},\bm{b_{2}})\approx-\cos(\bm{h_{1}},\bm{h_{2}})$

Metric loss aims at clustering, i.e., to push different category samples far
as possible and pull samples of the same categories close as possible. In
specific, existing techniques [17, 18, 19, 14, 21, 22, 23] generally adopt the
estimated cosine similarity in a mini-batch $\mathcal{B}$ to design such loss
term, which can be abstractly recorded as Eq. 3:

$\displaystyle\mathcal{L}_{\text{Metric}}$
$\displaystyle=\sum_{\bm{h_{1}}\in\mathds{H}}\sum_{\begin{subarray}{c}\bm{h_{2}}\in\mathds{H}^{\prime}\end{subarray}}\mathds{1}(y_{1}=y_{2})\mathcal{F}_{+}\left(-\cos(\bm{h_{1}},\bm{h_{2}})\right)$
(3)
$\displaystyle\quad+\sum_{\bm{h_{1}}\in\mathds{H}}\sum_{\begin{subarray}{c}\bm{h_{2}}\in\mathds{H}^{\prime}\end{subarray}}\mathds{1}(y_{1}\neq
y_{2})\mathcal{F}_{-}\left(\cos(\bm{h_{1}},\bm{h_{2}})\right),$

where $\mathds{1}(\cdot)$ is the indicator function that equals $1$ iff
$(\cdot)$ is true, otherwise is $0$. $\mathcal{F}_{+}$ and $\mathcal{F}_{-}$
indicate the designed loss functions for positive or negative samples,
respectively, which either estimate the similarity of pair-based methods or
proxy-based methods. $\mathds{H}$ represent the latent codes of given mini-
batch images, $\mathds{H}^{\prime}$ indicates the set of other samples’ latent
codes or class proxies. $y_{i}$ indicates the class label.

Quantization Loss. However, in the image retrieval field, metric loss
considers neither efficient query nor easy storage requirements, but it only
focuses on clustering images (or latent codes) of the same categories and
alienating those of different labels. Therefore, metric loss cannot guarantee
the binarized latent codes well-located in the ideal hash position because of
the missing quantization constraints. If the latent code $\bm{h}$ is directly
binarized, a large amount of feature information will get lost. e.g., some
bits close to $0$ of similar samples will be ambiguously binarized to $\pm 1$,
which is highly undesirable for clustering images with similar hash codes.

To handle ambiguity issues that cause information loss, quantization loss [12]
is proposed to constrain latent codes. By adding quantization loss [18, 19,
14, 21, 22, 23], retrieval performance is demonstrated to achieve promising
improvement. It’s often calculated by $l_{1}$ or $l_{2}$ regulation loss
between latent codes $\bm{h}$ and their corresponding binary vectors $\bm{b}$
obtained by sign function $\text{sgn}(\cdot)$:

$\mathcal{L}_{\text{Quan}}=\sum_{\bm{h}\in\mathds{H}}||\bm{h}-\text{sgn}(\bm{h})||_{F}^{n},$
(4)

where existing techniques [17, 18, 19, 14, 21, 22, 23] generally set $n=1,2$
to compute $l_{1}$ or $l_{2}$ loss of the quantization term.

Figure 2: Comparison between metric loss with/without HHF. (a) Metric loss
pushes different category samples far as possible in the feature space. (b)
After adding the quantization loss, the class centers will move towards the
binary points. But due to the repulsive effect of other classes in the metric
loss, the class centers cannot fit the hashing position. (c) Our HHF method
only encourages sample (or proxy) to push away samples in different categories
whose angle with the sample (or proxy) is $\leq 90^{\circ}$. (d) After adding
quantization loss to metric loss with HHF, the result achieves the global
optimal solution of this case.

###  III-C Conflict on Metric Loss and Quantization Loss

However, the learning purposes of metric loss and quantization loss are
inherently contradictory. We intuitively illustrate such a conflict in Fig. 2
for clarity, which contains three different categories in a 2D feature space
(i.e., a hypersphere).

Therefore, all samples in Fig. 2 are scaled to a feature circle with radius
$r=\sqrt{2}$ in 2D space (correspondingly, in $K$-dimension space, all samples
are distributed on a hypersphere with radius $r=\sqrt{K}$). The binarization
operation directly pulls all samples to four binary points (i.e., $(\pm 1,\pm
1)$), corresponding to four quadrants. The more samples away from the binary
points, the more information will be lost after hashing $\mathcal{H}$.

In Fig. 2(a), the motivation of metric learning is to push the clustering
centers of different categories as far as possible. Then, finally, it forms an
equilateral triangle on the 2D space to obtain the optimal solution (i.e.,
each cluster is pushed far as possible). At the same time, the purpose of
quantization learning in Fig. 2(b) is to pull the class centers to one of the
four binary points as close as possible to get the optimal solution for
quantization (i.e., towards least information loss in the binarization
process).

It is intuitive to find the incompatible conflict where the optimal metric and
quantization solutions cannot be satisfied simultaneously. In fact, the ideal
situation is that the three class centers are strictly at three of the four
binary points, which is the optimal global solution (Fig. 2(d)) that limited
information lost during the binarization operation.

Methods integrate metric loss and quantization loss will finally get an
intermediate situation (Fig. 2(b)) between the optimal metric solution
(equilateral triangle) and the optimal global solution (isosceles right
triangle), resulting in more information disappeared in the binarization
process with poor retrieval accuracy. Specifically, samples farther from the
clustering centers are more likely to be misclassified to other binary points
(as we will prove in Fig. 5).

###  III-D HHF: Novel Conflict Solution

Figure 3: Yellow line in (a) shows that we modify the linear correlation of
cosine similarity in metric loss with a hinge function by a specific
inflection point $\zeta$ for negative pairs. Red lines in (a)(b) illustrate
the hinge function after optimization for both negative and positive pairs.

Overview of HHF. We propose Hashing Guided Hinge Function (HHF) to solve the
conflict between metric loss and quantization loss. The effectiveness of HHF
is intuitively illustrated in Fig. 2(c)(d). We modify cosine similarity in
metric loss with hinge function controlled by a specific inflection point
$\zeta$, shown as the yellow line in Fig. 3(a). The inflection point $\zeta$,
which depends on code length and category number, illustrates the minimum
distance of the optimal global solution $d_{\min}$. We do not expect the
metric distances between different classes to be infinitely alienated.
Instead, we enforce to push them apart until reaching a certain distance
$\zeta$, deduced by $d_{\min}$. When the distance between different classes
satisfies the $\zeta$, we consider the optimal metric solution is achieved.

In Fig. 2, when the cosine similarity between two categories is exactly $0$,
the optimal global solution can be satisfied. For any samples (or class
proxies), HHF only pushes away those different category samples until their
angle reaches $90^{\circ}$, and does not constrain angles greater than
$90^{\circ}$. In this way, the optimal metric solution does not conflict with
the optimal quantization solution, and finally, the optimal global solution
can be obtained.

Noteworthy, inspired by the Triplet Loss [30], our empirical study shows that
models will be overfitting if all positive samples are uncontrolled close to
each other. Therefore, we make an additional margin on intra-class and inter-
class distance. To be specific, we add a hyperparameter $\delta$ to the
original HHF as the relaxation factor. The optimized cosine similarity
function is shown as the red line in Fig. 3(a)(b).

Details of Inflection Constant $\bm{\zeta}$. We then discuss the value and
calculation approach of obtaining the inflection point constant $\zeta$. The
target is equivalent to minimizing the distance of given binary linear codes.
However, it proves to be an NP-hard problem [34], i.e., calculating the ideal
minimum distance requires exponential time complexity. Fortunately,
researchers [25] propose to estimate the upper and lower bound of the minimum
distance, which constrains the minimum distance through serious inequalities
and presents minimum distance bound tables $\mathcal{T}$ for binary linear
code.

Inspired by [25], the minimum distance $d_{\min}$ is set according to the
following rules. a). If the value obtained from $\mathcal{T}$ is a constant,
then we take this value as $d_{\min}$. b). Otherwise, if the value obtained is
a range, then the median of this range is chosen as $d_{\min}$. Finally, to
calculate the ideal inflection point for metric learning, we obtain the value
of $\zeta$ by normalizing $d_{\min}$ according to the code length, i.e.,
numerically equivalent to the hash code $\bm{b}$’s length $K$:

$\displaystyle\zeta=1-2\times\frac{d_{\min}}{K}$ (5)

For convenience, we generate a $256\times 256$ table that can directly obtain
the value of the inflection point $\zeta$ uniquely determined by the hash code
length and category numbers, which is entirely available in our public code
repository.

Proxy-Anchor Loss with HHF. To better quantify each input $x\in\mathcal{X}$
into an appropriate hash code, a proper loss function is required to supervise
the metric distance. We choose the previous state-of-the-art approach Proxy-
Anchor [24] as the backbone metric on HHF for illustration. The metric loss of
Proxy-Anchor is calculated as Eq. 6:

$\displaystyle\mathcal{L}$
$\displaystyle=\frac{1}{|\mathds{P}|}\sum_{\bm{p}\in\mathds{P}}\log\left(1+\sum_{\bm{h}\in\bm{h_{p}^{-}}}e^{\alpha(\cos(\bm{h},\bm{p})+\gamma)}\right)$
(6)
$\displaystyle+\frac{1}{|\mathds{P}^{+}|}\sum_{\bm{p}\in\mathds{P}^{+}}\log\left(1+\sum_{\bm{h}\in\bm{h_{p}^{+}}}e^{-\alpha(\cos(\bm{h},\bm{p})-\gamma)}\right),$

where $\alpha$ is a scaling factor, and $\gamma$ is a margin. $\mathds{P}$
denotes the set of the proxies and $\mathds{P}^{+}$ represents the positive
proxies of samples in a mini-batch $\mathcal{B}$. For a proxy, the set of
latent codes $\mathds{H}=\\{\bm{h_{i}}\\}_{i=1}^{\mathcal{B}}$ is categorized
into two sets: $\bm{h^{+}_{p}}$ indicates the set of all samples of same
categories, and the rest $\bm{h^{-}_{p}}$ = $\mathds{H}-\bm{h^{+}_{p}}$
indicates different classes to proxy.

In Eq. 6, the error component $-\alpha(\cos(\bm{h},\bm{p})-\gamma)$ and
$\alpha(\cos(\bm{h},\bm{p})+\gamma$) is linear correlated to cosine similarity
for all samples, which contradicts quantization learning as we analyzed
before. According to the proposed HHF, we modify the linear calculation part
of cosine similarity in naïve Proxy-Anchor Loss to the Hinge function with a
specific inflection point $\zeta$, as Eq. 7 illustrates:

$\displaystyle\mathcal{L}$
$\displaystyle{}_{\text{Metric}}=\frac{1}{|\mathds{P}|}\sum_{\bm{p}\in\mathds{P}}\log\left(1+\sum_{\bm{h}\in\bm{h_{p}^{-}}}(e^{\alpha
H(\cos(\bm{h},\bm{p}),-\zeta-\delta)}-1)\right)$ (7)
$\displaystyle+\frac{1}{|\mathds{P}^{+}|}\sum_{\bm{p}\in\mathds{P}^{+}}\log\left(1+\sum_{\bm{h}\in\bm{h_{p}^{+}}}(e^{\alpha
H(-\cos(\bm{h},\bm{p}),1-\delta)}-1)\right),$

where $H(x,y)=\max(0,x+y)$ is a hinge loss function, $\delta$ is a relaxation
factor to prevent overfitting. Since the cosine similarity with HHF is always
$\geq 0$, its exponential part is $\geq 1$. As a result, each sample has
limited influence on loss when calculating gradient, which damages
backpropagation, especially when the large batch size is adopted. Therefore,
we set the exponential part minus to $1$ to enhance the influence of each
sample and enlarge the gradient.

In addition, Proxy-Anchor only considers metric loss but ignores the
quantization term, hence cannot perform as expected for retrieval. Therefore,
we follow [19, 22] to adopt $l_{2}$ regularization as quantization loss
$\mathcal{L}_{\text{Quan}}$, so as to extend Proxy-Anchor Loss to obtain the
final modified loss function that integrates the metric term
$\mathcal{L}_{\text{Metric}}$ and the quantization term
$\mathcal{L}_{\text{Quan}}$ by weight $\beta$ as Eq. 8.

$\mathcal{L}_{\text{Total}}=\mathcal{L}_{\text{Metric}}+\beta\mathcal{L}_{\text{Quan}}$
(8)

Learning Algorithm with HHF. HHF is flexible and robust to plugged-and-play
into other methods that require metric terms and quantization terms. Algo. 1
shows the training algorithm of given proxy-based loss as an illustration. The
algorithm of pair-based losses with HHF is in a similar fashion.

Algorithm 1 Training process with the proposed HHF.

0: Training images $\mathcal{X}=\\{x_{i}\\}^{M}_{i=1}$, label matrix
$\mathcal{Y}\in\mathds{R}^{C\times M}$, hash code length $K$, mini-batch
$\mathcal{B}$.

0: Optimized model param $\Theta^{*}$, encoded hash position
$\mathds{B}^{*}\in\\{\bm{b}\\}^{M}_{i=1}$, proxy matrix
$\mathds{P}^{*}\in\mathds{R}^{C\times N}$.

1: Calculate the dimension $k$ of the corresponding binary linear code by the
number of $C$ categories: $k=\lceil\log C\rceil$;

2: Look-up the Minimum Distance Bound $\Omega=[\Omega_{\min},\Omega_{\max}]$
of $k$-dim $K$-bits binary linear code via $\mathcal{T}$ [25] and obtain the
$d_{\min}$ according to our rule;

3: Normalize $d_{\min}$ according to Eq. 5 to obtain the ideal $\zeta$;

4: Random initialize $\Theta\leftarrow\Theta^{(0)}$,
$\bm{\mathds{P}}\leftarrow\bm{\mathds{P}}^{(0)}$;

5: Initialize Training epoch $T\leftarrow 0$;

6: repeat

7: Randomly sample a mini-batch data
$\mathcal{X}^{\prime}=\\{x_{i}\\}^{\mathcal{B}}_{i=1}$;

8: Compute 1). latent codes $\bm{h_{i}}\leftarrow\mathcal{E}(x_{i}|\Theta)$
and 2). hash codes $\bm{b_{i}}\leftarrow\text{sgn}(\bm{h_{i}})$ by forward
propagation;

9: for $\bm{h}=\bm{h_{1}}$ to $\bm{h_{\mathcal{B}}}$, $\bm{p}=\bm{p_{1}}$ to
$\bm{p_{C}}$ do

10: if $\bm{h}$ and $\bm{p}$ is a Positive Pair then

11: the similarity is $\max(0,1-\cos(\bm{h},\bm{p})-\delta)$

12: else if $\bm{h}$ and $\bm{p}$ is a Negative Pair then

13: the similarity is $\max(0,\cos(\bm{h},\bm{p})-\zeta-\delta)$

14: end if

15: Compute Metric loss $\mathcal{L}_{\text{Metric}}$ via Eq. 3;

16: end for

17: Compute Quantization loss $\mathcal{L}_{\text{Quan}}$ via Eq. 4 and total
loss $\mathcal{L}_{\text{Total}}$ via Eq. 8;

18: Update $\Theta^{(T+1)}$ and $\mathds{P}^{(T+1)}$ by back propagation;

19: $T\leftarrow T+1$;

20: until Convergence

21: Return
$\Theta^{*}\leftarrow\Theta^{(T)},\mathds{B}^{*}\leftarrow\mathcal{E}(\mathcal{X}|\Theta^{*}),\mathds{P}^{*}\leftarrow\mathds{P}^{(T)}$.

##  IV Experiments

In this section, we verify the effectiveness of the proposed HHF by conducting
extensive experiments on four representative benchmarks (w.r.t. CIFAR-10 [35],
CIFAR-100 [35], MS-COCO [36], and ImageNet [37]) for comprehensive
comparisons. We integrate HHF with various off-the-shelf metric losses [24,
13, 12] and compare them to the current state-of-the-art methods
quantitatively and qualitatively. In addition, we conduct hyperparameter
sensitivity experiments to fully demonstrate the robustness of the proposed
method.

###  IV-A Datasets

CIFAR-10 [35] is a widely used dataset for image recognition and is
appropriate for image retrieval tasks. It contains $60,000$ images at
$32\times 32$ resolution with $10$ different categories. We follow previous
work [17, 19, 18] to construct CIFAR-10-MINI and CIFAR-10-FULL with different
protocols for experiments. In CIFAR-10-MINI, we randomly select $500$ and
$100$ images from each class as the train set and query set, respectively, the
remaining images as the database. In CIFAR-10-FULL, $50,000$ images both serve
as the train set and database, and the rest $10,000$ images as the query set.

CIFAR-100 [35] contains the same image numbers ($60,000$ in total) as
CIFAR-10, while it has $100$ classes with $600$ images per category. Follow
[21, 22], we use the official $50,000$ images as train set and database, and
$10,000$ images for query in our experiment.

MS-COCO [36] is a popular multi-label dataset for image recognition. It
contains more than $120,000$ images, each of which is labeled as several of
$80$ categories. Follow previous work [14, 21, 22], we first preprocess to
remove the images without any labels from the dataset and then randomly select
$5,000$ images from the remaining $122,218$ ones, containing at least one
label, as the query set, $10,000$ images as the training set, and the rest as
the database.

ImageNet [37] is one of the most popular benchmark datasets for the Large
Scale Visual Recognition Challenge (ILSVRC 2012). It contains more than
$120,000$ images in $1,000$ categories. We use the same data and settings as
[11, 21]. The main challenge lies in training sample paucity and the large
number of categories.

###  IV-B Implementation Details

The proposed HHF is implemented in the PyTorch framework [38]. We
comprehensively adopt ResNet-50 [39] and GoogLeNet [40] pretrained on ImageNet
to illustrate the robustness of our HHF. We fine-tune the pre-trained
backbones for all layers up to the FC layer and map the output layer dimension
to the hash bit $K$. We adopt stochastic gradient descent (SGD) [41] as the
optimizer with momentum $0.9$ and weight decay $5e-4$. The initial learning
rate is $0.001$ for feature extraction and $0.01$ for proxy adjustment if the
proxy-based loss is adopted as the baseline metric. The learning rate
decreases by $0.5$ every $10$ epochs for $100$ epochs in total. For the
hyperparameters, if without special instructions, $\delta$ in the HHF method
is empirically set to $0.2$, and for the rest of the hyperparameters related
to other methods, we follow the settings in the previous work [12, 13, 24].

###  IV-C Evaluation Metrics

Given any query image, we retrieve top-$N$ images in the database based on the
closest Hamming distance and calculate their similarity. We adopt the standard
similarity protocol from the previous work [11, 17, 21]. For single-label
datasets (i.e., CIFAR, ImageNet), two images are considered similar iff they
have the same label. For multi-label datasets (i.e., MS-COCO), two images are
considered similar iff they share at least one label. We evaluate the proposed
method with several strong baselines [10, 27, 12, 17, 19, 18, 11, 13, 20, 42]
and current state-of-the-art methods [21, 14, 22, 24, 16, 15] for
comprehensive comparisons.

mean Average Precision (mAP) serves as the primary metric in our experiment,
which is the most widely used metric in image retrieval. Average Precision
(AP) refers to the area under the Precision-recall curve. Larger AP indicates
better retrieval performance. mAP is the mean of average precision for each
query image that can be calculated as Eq. 9.

$\text{mAP}@N=\frac{1}{|\mathcal{Q}|}\sum_{x_{q}\in\mathcal{Q}}AP(x_{q},y_{q})@N\vspace{-5pt}$
(9)

where $N$ is the number of relevant images w.r.t. the query image
$x_{q}\in\mathcal{Q}$ with corresponding label(s) $y_{q}$.

Hash Position Error (HPE) is proposed and adopted to measure the quality of
quantization learning in image retrieval quantitatively. Specifically, HPE
measures the average $l_{2}$ distance between the latent codes and
corresponding hash codes, as Eq. 10 illustrates.

$\text{HPE}=\frac{1}{N}\sum_{i=1}^{N}||\bm{h_{i}}-\text{sgn}(\bm{h_{i}})||_{2}^{2},\vspace{-5pt}$
(10)

where $\text{sgn}(\cdot)$ is a sign function that maps continuous latent codes
into binary values. Smaller HPE indicates that the latent codes are closer to
the ideal hash position, and shows less information loss during the
binarization process.

Global Inter-intra Distance Ratio. Apart from mAP and HPE, we follow [22] to
measure the Global Inter-intra Distance Ratio $\eta_{\text{global}}$ for
quantitative analyses on the quality of metric learning.
$\eta_{\text{global}}$ refers to the average distance ratio of intra-class and
inter-class, which is defined as Eq. 11:

$\eta_{\text{global}}=\frac{\frac{1}{N}\sum\limits_{i=1}^{N}||\bm{h_{i}}-\bm{\bar{h}^{c}_{i}}||_{2}^{2}}{\frac{1}{C}\sum\limits_{i=1}^{C}\frac{1}{C-1}\sum\limits_{\begin{subarray}{c}i\neq
j,j=1\end{subarray}}^{C}||\bm{\bar{h}^{c}_{i}}-\bm{\bar{h}^{c}_{j}}||_{2}^{2}},\vspace{-6pt}$
(11)

where $\bm{\bar{h}^{c}_{i}}$ represents the class center of the $i\\_$th
class. The Global Inter-intra Distance Ratio considers the average distance
among class centers, i.e., $\eta_{\text{global}}$ estimates the inter-intra
distance of each sample among all other categories.

Local Inter-intra Distance Ratio. In practice, we notice those semantic-
distinguishable classes (or remote centers), which away from the given class,
have seldom retrieval side-effect. Hence we believe the evaluation of the most
confusing classes is necessary. To achieve this, we propose Local Inter-intra
Distance Ratio $\eta_{\text{local}}$ to estimate the worst entangled class
centers, which is defined as Eq. 12.

$\eta_{\text{local}}=\frac{1}{N}\sum_{i=1}^{N}\frac{||\bm{h_{i}}-\bm{\bar{h}^{c}_{i}}||_{2}^{2}}{||\bm{h_{i}}-\bm{\bar{h}^{c}_{\phi(i)}}||_{2}^{2}},$
(12)

where $\phi(i)=\mathop{\arg\min}\nolimits_{\begin{subarray}{c}i\neq
j\end{subarray}}||\bm{h_{i}}-\bm{\bar{h}^{c}_{j}}||_{2}^{2}$ represents the
closest class center index of the $i\\_$th center. Both $\eta_{\text{global}}$
and $\eta_{\text{local}}$ measure the clustering quality of metric learning,
smaller $\eta_{\text{global}}$ and $\eta_{\text{local}}$ indicates better
performance in clustering.

###  IV-D Results on CIFAR-10 and CIFAR-100

TABLE I: mAP performance by Hamming Ranking for different hash bits on
CIFAR-10-MINI, CIFAR-10-FULL, and CIFAR-100-FULL. ⋆: reported results with the
same GoogLeNet backbone from [21, 22]. †: our reproduced results with
GoogLeNet.

Dataset | CIFAR-10-MINI | CIFAR-10-FULL | CIFAR-100-FULL
---|---|---|---
Hash bits | 12 bits | 24 bits | 32 bits | 48 bits | 12 bits | 24 bits | 32 bits | 48 bits | 12 bits | 24 bits | 32 bits | 48 bits
DPSH⋆ [17] | 0.797 | 0.806 | 0.820 | 0.802 | 0.908 | 0.922 | 0.925 | 0.935 | 0.060 | 0.101 | 0.120 | 0.159
DTSH⋆ [19] | 0.790 | 0.797 | 0.794 | 0.775 | 0.928 | 0.935 | 0.940 | 0.942 | 0.607 | 0.706 | 0.712 | 0.725
DSDH⋆ [18] | 0.800 | 0.802 | 0.804 | 0.808 | 0.913 | 0.925 | 0.943 | 0.930 | 0.078 | 0.150 | 0.187 | 0.227
DCWH⋆ [21] | 0.818 | 0.840 | 0.848 | 0.854 | 0.940 | 0.950 | 0.954 | 0.952 | 0.723 | 0.744 | 0.757 | 0.766
IDCWH⋆ [22] | 0.828 | 0.865 | 0.868 | 0.849 | 0.964 | 0.969 | 0.967 | 0.968 | 0.764 | 0.813 | 0.824 | 0.835
Proxy-Anchor† [24] | 0.878 | 0.886 | 0.889 | 0.904 | 0.975 | 0.976 | 0.977 | 0.978 | 0.773 | 0.860 | 0.870 | 0.870
Proxy-Anchor† \+ HHF | 0.889 | 0.903 | 0.909 | 0.908 | 0.975 | 0.976 | 0.978 | 0.979 | 0.851 | 0.868 | 0.873 | 0.875

TABLE II: mAP performance with different hash bits on ImageNet and MS-COCO. ⋆:
reported results from [11, 21, 14, 22, 42, 15, 16]. †: our reproduced results
with GoogLeNet and ResNet-50 through the open-source code and the loss
equation by existing literature. $\Delta$: average improvement with HHF for
different hash bits compared with the original method.

| Dataset | ImageNet (mAP@1000) | MS-COCO (mAP@5000)
---|---|---|---
Backbone | Hash bits | 16 bits | 32 bits | 48 bits | 64 bits | $\Delta$ | 16 bits | 32 bits | 48 bits | 64 bits | $\Delta$
AlexNet [43] | CNNH⋆ [10] | 0.281 | 0.450 | 0.525 | 0.554 |  | 0.564 | 0.574 | 0.571 | 0.567 |
DNNH⋆ [27] | 0.290 | 0.461 | 0.530 | 0.565 |  | 0.593 | 0.603 | 0.605 | 0.610 |
HashNet⋆ [11] | 0.506 | 0.631 | 0.663 | 0.684 |  | 0.687 | 0.718 | 0.730 | 0.736 |
DCCH⋆ [42] | - | - | - | - |  | 0.659 | 0.729 | 0.731 | 0.739 |
OrthoCos⋆ [15] | 0.606 | 0.679 | - | 0.711 |  | 0.709 | 0.762 | - | 0.787 |
GoogLeNet [40] | DPSH⋆ [17] | - | - | - | - |  | 0.349 | 0.355 | 0.360 | 0.367 |
DSDH⋆ [18] | - | - | - | - |  | 0.347 | 0.359 | 0.366 | 0.370 |
DCWH⋆ [21] | 0.782 | 0.799 | 0.835 | 0.849 |  | 0.742 | 0.776 | 0.786 | 0.779 |
IDCWH⋆ [22] | - | - | - | - |  | 0.732 | 0.760 | 0.764 | 0.770 |
DHN† [12] | 0.542 | 0.582 | 0.541 | 0.627 | - | 0.677 | 0.701 | 0.695 | 0.694 | -
DHN† \+ HHF | 0.721 | 0.810 | 0.812 | 0.802 | +21.33% | 0.722 | 0.752 | 0.763 | 0.772 | +6.05%
Proxy-NCA† [13] | 0.613 | 0.690 | 0.749 | 0.755 | - | 0.562 | 0.608 | 0.674 | 0.649 | -
Proxy-NCA† \+ HHF | 0.803 | 0.854 | 0.873 | 0.872 | +14.88% | 0.708 | 0.839 | 0.860 | 0.870 | +19.6%
Proxy-Anchor† [24] | 0.856 | 0.883 | 0.880 | 0.883 | - | 0.736 | 0.822 | 0.845 | 0.850 | -
Proxy-Anchor† \+ HHF | 0.882 | 0.898 | 0.901 | 0.897 | +1.90% | 0.805 | 0.850 | 0.858 | 0.851 | +2.78%
ResNet-50 [39] | DCH⋆ [20] | 0.652 | 0.737 | - | 0.758 |  | 0.759 | 0.801 | - | 0.825 |
CSQ⋆ [14] | 0.851 | 0.865 | 0.866 | 0.873 |  | 0.796 | 0.838 | - | 0.861 |
DCSH⋆ [16] | - | - | - | - |  | 0.805 | 0.847 | 0.859 | 0.861 |
DHN† [12] | 0.429 | 0.501 | 0.516 | 0.552 | - | 0.650 | 0.663 | 0.667 | 0.667 | -
DHN† \+ HHF | 0.653 | 0.769 | 0.799 | 0.807 | +25.75% | 0.746 | 0.798 | 0.814 | 0.826 | +13.43%
Proxy-NCA† [13] | 0.620 | 0.715 | 0.737 | 0.769 | - | 0.545 | 0.609 | 0.659 | 0.679 | -
Proxy-NCA† \+ HHF | 0.835 | 0.861 | 0.859 | 0.858 | +14.3% | 0.700 | 0.796 | 0.850 | 0.857 | +17.78%
Proxy-Anchor† [24] | 0.824 | 0.871 | 0.874 | 0.873 | - | 0.713 | 0.805 | 0.840 | 0.846 | -
Proxy-Anchor† \+ HHF | 0.871 | 0.891 | 0.895 | 0.896 | +2.78% | 0.798 | 0.850 | 0.872 | 0.870 | +4.65%

Settings. We integrate Proxy-Anchor Loss [24] with HHF and quantization terms
for experiments. The overall loss function has illustrated in Eq. 8. To
eliminate the influence of backbones on the performance, we fairly compare
various methods on CIFAR [35] datasets with the same GoogLeNet for all
methods. In Tab. I, “⋆” represents that we directly report the results from
[21, 22]. DPSH [17], DTSH [19], and DSDH [18] are reproduced by IDCH [22] on
GoogLeNet, and we directly adopt the results from [22]. We also reproduce and
report the results of Proxy-Anchor Loss with quantization term on GoogLeNet
for reference (denoted as “†”).

Results. In simple CIFAR-10, the conflict gets relieved by expanding train
data under little category numbers. The bottleneck originates from the
expressiveness of backbones, which accounts for the similar performance on
CIFAR-10-FULL between Proxy-Anchor with/without HHF. Tab. I demonstrates that
HHF outperforms existing techniques over different hash bits, especially in
CIFAR-100-FULL. Since CIFAR-100 has more categories than CIFAR-10, the
conflict between metric learning and quantization learning is more prominent
and challenging. In CIFAR-100-FULL, HHF achieves over $\bm{5.78\%}$ and
$\bm{2.35\%}$ average gains compared to IDCWH [22] and Proxy-Anchor [24],
respectively, which verifies the effectiveness of the proposed method.

TABLE III: Ablation study with proxy-based methods on ImageNet under
ResNet-50. Underlined results: Smaller HPE, $\eta_{\text{global}}$, and
$\eta_{\text{local}}$, which indicate better performance. HHF enables to
achieve better HPE without sacrificing metric learning.

Hash Bits | 16 bits | 32 bits | 48 bits | 64 bits
---|---|---|---|---
Metric | HPE $\downarrow$ | $\eta_{\text{global}}\downarrow$ | $\eta_{\text{local}}\downarrow$ | HPE $\downarrow$ | $\eta_{\text{global}}\downarrow$ | $\eta_{\text{local}}\downarrow$ | HPE$\downarrow$ | $\eta_{\text{global}}\downarrow$ | $\eta_{\text{local}}\downarrow$ | HPE$\downarrow$ | $\eta_{\text{global}}\downarrow$ | $\eta_{\text{local}}\downarrow$
Proxy-NCA | 1.858 | 0.298 | 0.947 | 2.519 | 0.286 | 0.806 | 2.875 | 0.281 | 0.748 | 3.039 | 0.263 | 0.679
Proxy-NCA + HHF | 1.334 | 0.262 | 0.466 | 1.690 | 0.250 | 0.444 | 2.052 | 0.243 | 0.438 | 2.250 | 0.235 | 0.450
Proxy-Anchor | 1.862 | 0.295 | 0.434 | 1.834 | 0.291 | 0.413 | 2.072 | 0.285 | 0.406 | 2.300 | 0.282 | 0.406
Proxy-Anchor + HHF | 1.296 | 0.265 | 0.433 | 1.807 | 0.267 | 0.407 | 1.846 | 0.264 | 0.399 | 2.103 | 0.262 | 0.393

Figure 4: Precision-recall curves (solid lines) and Precision@top-$N$ curves
(dot lines) on ImageNet (@first row) and MS-COCO (@second row) under ResNet-50
with binary codes @16 / 32 / 48 / 64 bits (left $\rightarrow$ right). “*”
indicates method with HHF. PA: Proxy-Anchor. PN: Proxy-NCA. Three baselines
can be further improved with our HHF.

###  IV-E Results on Large-Scale Datasets

Settings. To illustrate the effectiveness and generalization of HHF
comprehensively, we integrate HHF with three different methods using different
backbones on challenging large-scale datasets (i.e., ImageNet and MS-COCO).
Specifically, we adopt HHF in 1). DHN [12], a pair-based loss, combines
quantization loss with metric loss to constrain latent codes. 2). Proxy-NCA
[13], a proxy-based loss, proposes a set of learnable class centers. 3).
Proxy-Anchor [24], a proxy-based loss, combines the advantages of proxy-based
methods and pair-based methods. We comprehensively compare them to various
state-of-the-art methods in Tab. II.

Results. Tab. II demonstrates that HHF enables different methods to achieve
significant performance gains in the two large-scale datasets over different
hash bits. As mentioned before, the conflict issue becomes increasingly severe
as the category increases, HHF keeps generalizations on challenging datasets
and shows portability in many approaches of dealing with conflicts, especially
in the naïve methods DHN and Proxy-NCA. Experiments on different backbones
also verify that HHF is robust and effective in tackling the conflict between
metric learning and quantization learning.

###  IV-F Ablation study.

Qualitative analysis. Fig. 4 shows the Precision w.r.t. different numbers of
Top Returned Samples and Recall curves on ImageNet and MS-COCO. Compared to
methods with/without HHF, HHF achieves considerable performance improvements
compared with original ones. Specifically, Proxy-Anchor with HHF achieves the
best performance among others.

Quantitative analysis. As one of the main methodologies of HHF, we propose the
hinge inflection point to explicitly constrain the metric loss to avoid class
centers becoming unlimited alienated from each other, hence HHF seems to
damage the results of feature learning. However, thanks to the theory of
minimum distance of the binary linear code [25], we quantitatively demonstrate
that HHF generally will not damage metric learning, instead it slightly
promotes the metric learning process and ensures clusters more
distinguishable, because it guarantees metric loss when distance $\leq\zeta$
and fully focuses on quantization learning if distance $\geq\zeta$.

Tab. III shows that the state-of-the-art proxy-based methods keep stable
$\eta$ with/without HHF, i.e., HHF has seldom side-effect on metric learning
(as we will verify in Fig. 6). Noteworthy, methods with HHF get HPE
significantly decreased, i.e., the latent codes are closer to the ideal binary
position, which is crucial for efficient image retrieval. In most cases, HHF
enables distinguishing confusion categories, i.e., it gets
$\eta_{\text{local}}$ significantly decreased. The result proves that the
proposed HHF generally reduces semantic information loss in the binarization
process without loss of cluster performance.

(a) Query: doormat

(b) Query: butterfly

Figure 5: Retrieval results of methods with/without HHF on ImageNet. Red box
indicates misclassified results, while HHF enables to significantly improve
the retrieval quality. “*” indicates method with HHF. PA: Proxy-Anchor. PN:
Proxy-NCA.

###  IV-G Visualized Comparison

Retrieval results. We represent visualized retrieval results for empirical
comparisons. We randomly give some queries to retrieve the top-$10$ results
from ImageNet database and compare the retrieved images of different methods
(i.e., DHN, Proxy-NCA, Proxy-Anchor with/without HHF). The results are
illustrated in Fig. 5.

Results in Fig. 5 show that the proposed HHF enables to improve the retrieval
performance significantly. Specifically, due to the conflict between metric
learning and quantization learning, previous methods cannot balance the
optimal clustering results and ideal hashing position. Intuitively, although
some methods with/without HHF perform comparable on clustering (i.e., similar
$\eta$) in Tab. III, there are many misclassified or error hashed results in
retrieval when without HHF (e.g., for a butterfly query, Proxy-Anchor
misclassify the results and make all error retrieval results; for a doormat
query, DHN retrieve the all error human results).

In contrast, with the combination of the proposed HHF, the retrieval accuracy
gets remarkably improved. HHF enables images of the same category to
accurately binarize into similar hash codes (i.e., smaller HPE and larger
mAP), which alleviates the errors with satisfactory retrieval results and is
robust in most cases. Intuitively, the retrieval results in Fig. 5 are more
reliable than those without HHF.

(a) DHN [12] (b) Proxy-NCA [13] (c) Proxy-Anchor [24] (d) DHN+HHF

(e) Proxy-NCA+HHF (f) Proxy-Anchor+HHF

Figure 6: Visualized t-SNE results of methods with/without HHF on CIFAR-100.
The scatters of the same color indicate the same categories, the combination
of HHF enables to disentangle different categories.

Clustering results. To fully illustrate how the proposed HHF contributes to
metric learning, besides the detailed quantitative comparisons in Tab. III, we
further present our empirical study to demonstrate that HHF will not harm
metric learning. Specifically, we utilize t-SNE [44] to map $K$-dimension
latent codes into 2D distribution for visualization. We compared DHN [12],
Proxy-NCA [13], and Proxy-Anchor [24] with/without HHF on CIFAR-100 with $12$
hash bits, as illustrated in Fig. 6.

Fig. 6(a) shows that the latent codes of different categories are coupled
together, indicating that DHN is incapable of achieving satisfactory
clustering results. The poor clustering performance also deteriorates the
subsequent quantization operation, making the retrieval results less promising
(as Fig. 5 illustrates). Noteworthy, DHN uses a pair-wise loss for metric
learning. However, such pair-wise loss methods only consider a negative sample
once in back-propagate, regardless of other negative ones. Hence, the randomly
sampled pairs cannot guarantee that each category is alienated and harmful to
other negative samples, making it easy to trap in a locally optimal solution.
In contrast, HHF makes specific constraints to avoid samples being over-
alienated, which tackles the conflict of intra-class and inter-class.

Compared to DHN, Proxy-NCA slightly disentangles the hash centers of different
categories. However, most categories are still ambiguous of each other. As a
comparison, Proxy-NCA with HHF significantly improves both metric learning
(Fig. 6(e)) and quantization learning (Tab. 5). Proxy-Anchor is proved to be
helpful in metric learning. Intuitively, the latent codes of each category can
get perfectly disentangled compared to DHN. Fig. 5(c)(f) demonstrate that both
Proxy-Anchor with and without HHF achieve appealing results. HHF successfully
solves the challenging conflict and improves retrieval performance without
losing of metric learning.

TABLE IV: mAP performance of HHF@48bits w.r.t. the margin hyperparameter
$\delta$ on three datasets under GoogLeNet, Above which the performance is
optimal when $\delta=0.2$.

Margin Params $\delta$ | 0.0 | 0.1 | 0.2 | 0.3 | 0.4
---|---|---|---|---|---
CIFAR-100-FULL | 0.818 | 0.858 | 0.875 | 0.874 | 0.367
ImageNet | 0.879 | 0.889 | 0.901 | 0.879 | 0.420
MS-COCO | 0.841 | 0.847 | 0.858 | 0.837 | 0.541

###  IV-H Hyperparameter Sensitivity Analysis

Settings. We further present in-depth investigations on the influence of
$\delta$. The main effect of $\delta$ in the image retrieval field is to
prevent overfitting [30]. $\delta$ shows more interpretability in HHF that it
prevents different images of the same category encoded into the same latent
codes. It ensures images to distribute in an ideal hypersphere. We
comprehensively compare Proxy-Anchor with HHF setting the margin $\delta$
range from $0$ to $0.4$. We conduct experiments in three standard datasets:
CIFAR-100, ImageNet, and MS-COCO with hash bits of $48$. The mAP results are
illustrated in Tab. IV.

Results. Tab. IV illustrates that $\delta$ has similar performance influence
trends on these benchmarks of a specific approach, where the best performance
achieves nearby $\delta=0.2$. We also report HHF without adding $\delta$ for
reference, to intuitively demonstrate that the missing margin will generally
damage the performance. Specifically, due to the existence of overfitting
issues, mAP of each dataset gets decreased compared to the best results.
Noteworthy, the improving $\delta$ does not guarantee the consistent
performance gains of different methods. When $\delta$ is away from $0.2$, we
notice that performance will oppositely decrease. Hence the empirical study
also proves the importance of an appropriate $\delta$ for better hash
performance.

##  V Conclusion

In this paper, a novel hinge loss function is proposed towards the ideal hash
position of given images for large-scale retrieval, which is robust to
plugged-and-play into existing methods to consistently improve their
performance. To achieve this, we explore in-depth investigations to excavate
the relationship between metric learning and quantization learning, and
systematically analyze the challenging conflict issues between the two
learning procedures for deep hashing. By revealing the unsolved fact that the
ideal metric solution cannot satisfy the optimal quantization solution for
retrieval, we focus on addressing semantic information loss during the hashing
process in large-scale image datasets and carefully design Hashing guided
hinge loss to effectively tackle such conflict. Extensive experiments on four
standard datasets verify the superiority and flexibility of HHF to integrate
with various off-the-shelf methods to achieve significant performance gains.
We conduct both quantitative and qualitative comparisons to demonstrate the
effectiveness of our method. Various evaluation metrics and visualizations
fully demonstrate that HHF enables to achieve state-of-the-art retrieval
performance with better visually consistent results.

## References

  * [1] R. Datta, D. Joshi, J. Li, and J. Z. Wang, “Image retrieval: Ideas, influences, and trends of the new age,” _ACM Csur_ , 2008.
  * [2] L. Zhang and Y. Rui, “Image search—from thousands to billions in 20 years,” _ACM TOMM_ , 2013.
  * [3] Y. Gong, S. Lazebnik, A. Gordo, and F. Perronnin, “Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval,” _IEEE TPAMI_ , 2012.
  * [4] W. Liu, J. Wang, R. Ji, Y.-G. Jiang, and S.-F. Chang, “Supervised hashing with kernels,” in _CVPR_ , 2012.
  * [5] B. Kulis and T. Darrell, “Learning to hash with binary reconstructive embeddings.” in _NIPS_ , 2009.
  * [6] F. Shen, C. Shen, W. Liu, and H. Tao Shen, “Supervised discrete hashing,” in _CVPR_ , 2015.
  * [7] I. M. Hameed, S. H. Abdulhussain, and B. M. Mahmmod, “Content-based image retrieval: A review of recent trends,” _Cogent Engineering_ , 2021.
  * [8] W. Chen, Y. Liu, W. Wang, E. M. Bakker, T. Georgiou, P. W. Fieguth, L. Liu, and M. S. Lew, “Deep image retrieval: A survey,” _arXiv preprint:abs/2101.11282_ , 2021.
  * [9] J. Wang, W. Liu, S. Kumar, and S. Chang, “Learning to hash for indexing big data - A survey,” _Proc. IEEE_ , 2016.
  * [10] R. Xia, Y. Pan, H. Lai, C. Liu, and S. Yan, “Supervised hashing for image retrieval via image representation learning,” in _AAAI_ , 2014.
  * [11] Z. Cao, M. Long, J. Wang, and P. S. Yu, “Hashnet: Deep learning to hash by continuation,” in _ICCV_ , 2017.
  * [12] H. Zhu, M. Long, J. Wang, and Y. Cao, “Deep hashing network for efficient similarity retrieval,” in _AAAI_ , 2016.
  * [13] Y. Movshovitz, A. Toshev, T. K. Leung, S. Ioffe, and S. Singh, “No fuss distance metric learning using proxies,” in _ICCV_ , 2017.
  * [14] L. Yuan, T. Wang, X. Zhang, F. E. H. Tay, Z. Jie, W. Liu, and J. Feng, “Central similarity quantization for efficient image and video retrieval,” in _CVPR_ , 2020.
  * [15] J. T. Hoe, K. Ng, T. Zhang, C. S. Chan, Y. Song, and T. Xiang, “One loss for all: Deep hashing with a single cosine similarity based learning objective,” _arXiv preprint:abs/2109.14449_ , 2021.
  * [16] A. Jose, D. Filbert, C. Rohlfing, and J.-R. Ohm, “Deep hashing with hash center update for efficient image retrieval,” _arXiv preprint:abs/2106.06269_ , 2021.
  * [17] W. Li, S. Wang, and W. Kang, “Feature learning based deep supervised hashing with pairwise labels,” in _IJCAI_ , 2016.
  * [18] Q. Li, Z. Sun, R. He, and T. Tan, “Deep supervised discrete hashing,” in _NIPS_ , 2017.
  * [19] X. Wang, Y. Shi, and K. M. Kitani, “Deep supervised hashing with triplet labels,” in _ACCV_ , 2016.
  * [20] Y. Cao, M. Long, B. Liu, and J. Wang, “Deep cauchy hashing for hamming space retrieval,” in _CVPR_ , 2018.
  * [21] X. Zhe, S. Chen, and H. Yan, “Deep class-wise hashing: Semantics-preserving hashing via class-wise loss,” _IEEE Trans. Neural Networks Learn. Syst._ , 2020.
  * [22] M. Zhang and H. Yan, “Improved deep classwise hashing with centers similarity learning for image retrieval,” in _ICPR_ , 2020.
  * [23] J. Wang, B. Chen, Q. Zhang, Z. Meng, S. Liang, and S. Xia, “Weakly supervised deep hyperspherical quantization for image retrieval,” in _AAAI_ , 2021.
  * [24] S. Kim, D. Kim, M. Cho, and S. Kwak, “Proxy anchor loss for deep metric learning,” in _CVPR_ , 2020.
  * [25] A. E. Brouwer and T. Verhoeff, “An updated table of minimum-distance bounds for binary linear codes,” _IEEE Trans. Inf. Theory_ , 1993.
  * [26] W. Rawat and Z. Wang, “Deep convolutional neural networks for image classification: A comprehensive review,” _Neural Comput._ , 2017.
  * [27] H. Lai, Y. Pan, Y. Liu, and S. Yan, “Simultaneous feature learning and hash coding with deep neural networks,” in _CVPR_ , 2015.
  * [28] S. Chopra, R. Hadsell, and Y. LeCun, “Learning a similarity metric discriminatively, with application to face verification,” in _CVPR_ , 2005\.
  * [29] R. Hadsell, S. Chopra, and Y. LeCun, “Dimensionality reduction by learning an invariant mapping,” in _CVPR_ , 2006.
  * [30] F. Schroff, D. Kalenichenko, and J. Philbin, “Facenet: A unified embedding for face recognition and clustering,” in _CVPR_ , 2015.
  * [31] K. Sohn, “Improved deep metric learning with multi-class n-pair loss objective,” in _NIPS_ , 2016.
  * [32] H. O. Song, Y. Xiang, S. Jegelka, and S. Savarese, “Deep metric learning via lifted structured feature embedding,” in _CVPR_ , 2016.
  * [33] N. Aziere and S. Todorovic, “Ensemble deep manifold similarity learning using hard proxies,” in _CVPR_ , 2019.
  * [34] A. Vardy, “The intractability of computing the minimum distance of a code,” _IEEE Trans. Inf. Theory_ , 1997.
  * [35] A. Krizhevsky, G. Hinton _et al._ , “Learning multiple layers of features from tiny images,” 2009.
  * [36] T. Lin, M. Maire, S. J. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: common objects in context,” in _ECCV_ , 2014.
  * [37] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. S. Bernstein, A. C. Berg, and F. Li, “Imagenet large scale visual recognition challenge,” _IJCV_ , 2015.
  * [38] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “Pytorch: An imperative style, high-performance deep learning library,” in _NeurIPS_ , 2019.
  * [39] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in _CVPR_ , 2016.
  * [40] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. E. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in _CVPR_ , 2015.
  * [41] L. Bottou, “Large-scale machine learning with stochastic gradient descent,” in _Proceedings of COMPSTAT_ , 2010.
  * [42] A. Jose, E. S. Ottlik, C. Rohlfing, and J.-R. Ohm, “Optimized feature space learning for generating efficient binary codes for image retrieval,” _Signal Process. Image Commun._ , 2021.
  * [43] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in _NIPS_ , 2012.
  * [44] L. Van der Maaten and G. Hinton, “Visualizing data using t-sne.” _JMLR_ , 2008\.