A Semilinear Elliptic Problem with Critical Exponent and Potential Terms

Haoyu Li

Departamento de Matemática, Universidade Federal de S$\tilde{a}$o Carlos,
S$\tilde{a}$o Carlos-SP, 13565-905, Brazil
E-mail: e-mail<EMAIL_ADDRESS>
Li Ma

School of Mathematics and Physics, University of Science and Technology
Beijing,
30 Xueyuan Road, Haidian District, Beijing 100083, China
and
Department of Mathematics, Henan Normal University,
Xinxiang 453007, China
E-mail<EMAIL_ADDRESS>
Abstract. This paper addresses the following problem.

$\left\\{\begin{array}[]{lr}-{\Delta}u=\lambda
I_{\alpha}*_{\Omega}u+|u|^{2^{*}-2}u\mbox{ in }\Omega,\\\ u\in
H_{0}^{1}(\Omega).\end{array}\right.$

Here, $\Omega$ is a bounded domain in $\mathbb{R}^{N}$ with $N\geq 3$,
$2^{*}=\frac{2N}{N-2}$, $\lambda\in\mathbb{R}$, $\lambda\in(0,N)$,
$I_{\alpha}$ is the Riesz potential and

$\displaystyle
I_{\alpha}*_{\Omega}u(x):=\int_{\Omega}\frac{\Gamma(\frac{N-\alpha}{2})}{\Gamma(\frac{\alpha}{2})\pi^{\frac{N}{2}}2^{\alpha}|x-y|^{N-\alpha}}u(y)dy.$

We study the non-existence, existence and multiplicity results. Our argument
combines Brezis-Nirenberg’s method with the regularity results involving
potential terms. Especially, we study the following nonlocal eigenvalue
problem.

$\left\\{\begin{array}[]{lr}-{\Delta}u=\lambda I_{\alpha}*_{\Omega}u\mbox{ in
}\Omega,\\\ \lambda\in\mathbb{R},\,u\in H_{0}^{1}(\Omega).\end{array}\right.$

Keywords: Sobolev critical exponent; Nonlocal semilinear elliptic equations;
Riesz potential; Pohozaev identiy; Nonlocal eigenvalue problem; Ground state
solution; Maximum principle for nonlocal operator. 2020 MSC: 35J20, 35B33,
35J61, 35J61, 45K05, 47A75, 35B50

###### Contents

  1. 1 Introduction
    1. 1.1 Introduction and historical remarks
    2. 1.2 Main results
    3. 1.3 Organization of this paper
  2. 2 Proof of Theorem 1.4: The eigenvalue problems
    1. 2.1 The existence of eigenfunctions
    2. 2.2 The regularity of eigenfunctions
    3. 2.3 Orthogonality and completeness
    4. 2.4 Principal eigenfunction
    5. 2.5 Proof of Theorem 1.4
  3. 3 Proof of Theorems 1.1 and 1.2: Non-existence results
    1. 3.1 The regularity of weak solutions
    2. 3.2 Pohozaev identity
    3. 3.3 Proof of Theorem 1.1
    4. 3.4 Proof of Theorem 1.2
  4. 4 Proof of Theorems 1.3 and 1.5: Existence and multiplicity results
    1. 4.1 Estimate of the mountain-pass level
    2. 4.2 A local compactness
    3. 4.3 A maximum principle for a nonlocal operator
    4. 4.4 Proof of Theorem 1.3
    5. 4.5 Proof of Theorem 1.5

##  1\. Introduction

###  1.1. Introduction and historical remarks

In this paper, we consider the following problem.

(1.1) $\left\\{\begin{array}[]{lr}-{\Delta}u=\lambda
I_{\alpha}*_{\Omega}u+|u|^{2^{*}-2}u\mbox{ in }\Omega,\\\ u\in
H_{0}^{1}(\Omega).\end{array}\right.$

Here, $\Omega$ is a bounded domain in $\mathbb{R}^{N}$ with smooth boundary
$\partial\Omega$ and $N\geq 3$, $2^{*}=\frac{2N}{N-2}$, $I_{\alpha}$ is the
Riesz potential defined as

$\displaystyle
I_{\alpha}=\frac{\Gamma(\frac{N-\alpha}{2})}{\Gamma(\frac{\alpha}{2})\pi^{\frac{N}{2}}2^{\alpha}|x|^{N-\alpha}}$

and the corresponding potential term is defined as

$\displaystyle
I_{\alpha}*_{\Omega}u(x):=\int_{\Omega}\frac{\Gamma(\frac{N-\alpha}{2})}{\Gamma(\frac{\alpha}{2})\pi^{\frac{N}{2}}2^{\alpha}|x-y|^{N-\alpha}}u(y)dy.$

In follows, we denote
$C_{N,\alpha}=\frac{\Gamma(\frac{N-\alpha}{2})}{\Gamma(\frac{\alpha}{2})\pi^{\frac{N}{2}}2^{\alpha}}$
for the sake of brevity. Our topic is stream out of the following problem
involving a convolution term.

(1.2) $\left\\{\begin{array}[]{lr}-\Delta u+\omega
u+\lambda(I_{\alpha}*F(u))F^{\prime}(u)=g(x,u)\mbox{ in }\Omega,\\\ u\in
H_{0}^{1}(\Omega).\end{array}\right.$

The equation in the form of (1.2) arise in the context of many physical
models, such as the quantum transport and the non-relativistic Newtonian
gravity, see [6, 3, 35] and the references therein. In particular, if
$\lambda=-1$ and $g(x,u)\equiv 0$, Problem (1.2) turns out to be a Choquard
equation or a nonlinear Schrödinger-Newton equation. And if $F(u)=u^{2}$,
Problem (1.2) becomes into a Schrödinger-Poisson equation. There has been a
large number of recent mathematical research on these problems.

For Choquard equation, in the research of Moroz and van Schaftingen [32, 33],
they proved the existence of the ground state solution and verified the
related properties. Like the scalar field equation $-\Delta u+u=u^{p}$, the
uniqueness of the positive solution was an open problem for a long time until
Ma and Zhao [30] gave a partial affirmative answer. We refer [34] for a
survey. There are also references that focus on more specific solutions. For
instance, the existence of multi-bump solutions to a non-autonomous Choquard
equation is proved by Wei and Winter [41]. For the autonomous Choquard
equation, [43, 39] shows the existence of saddle solutions. Recently, Liu, Ma
and Xia [29] studied the singularly perturbed Choquard equation and obtained
multiple bounded state solutions.

For Schrödinger-Poisson equation, Benci and Fortunato [5] studied the
corresponding eigenvalue problem. Their attention is focused on Problem (1.2)
with $g(x,u)=\lambda u$. For problem with nonlinear $g(x,u)$, [38] and [24]
obtained the ground state solution. Schrödinger-Poisson equations with
critical nonlinear term are studied in [1] for planar case and in [11] for the
three-dimensional case. There are also numerous references on more specific
cases. For example, in [22], Hebey and Wei considered the analogue to
Schrödinger-Poisson equation defined on sphere.

In this paper, we analyze Problem (1.2) with $\omega=0$, $F(u)=u$ and
$g(x,u)=|u|^{2^{*}-2}u$, i.e. Problem (1.1). The solvability of elliptic
problem involving critical exponent was open question for a long time, until
the publication of the ground-breaking work by Brézis and Nirenberg [7].
Subsequent works, such as [10, 26], continued to study the multiplicity result
for critical elliptic problems and the non-autonomous variants of them. Among
all of the problems, we emphasize that the two-dimensional critical problem is
with exponentially growing nonlinear terms, cf. [20, 27].

In this paper, we study the critical problem involving potential terms. The
potential term $I_{\alpha}*_{\Omega}u$ brings additional difficulties in
regularity and the nonlocal eigenvalue problem. To address this problem, we
introduce regularity estimates for potential terms. As to the nolocal
eigenvalue problem involving term $I_{\alpha}*_{\Omega}u$, we apply a minimax
approach [37, 36, 31] instead of the classical linear functional analytic
method [17, Chapter 6].

From the point of view of nonlinear analysis, the solution to Problem (1.1) is
a critical point of the energy functional

$\displaystyle E_{\alpha,\lambda}(u)=\frac{1}{2}\int_{\Omega}|\nabla
u|^{2}dx-\frac{\lambda}{2}\mathcal{D}_{\alpha}(u)-\frac{1}{2^{*}}\int_{\Omega}|u|^{2^{*}}dx$

with

$\displaystyle\mathcal{D}_{\alpha}(u)=C_{N,\alpha}\int_{\Omega}dy\int_{\Omega}\frac{u(x)u(y)}{|x-y|^{N-\alpha}}dy.$

To deal with $\mathcal{D}_{\alpha}(u)$, Hardy-Littlewood-Sobolev inequality is
an important tool. To be precise, for any $p,q\in(1,\infty)$ and any
$\lambda\in(0,n)$, there exists a constant $C_{p,q,\lambda,N}>0$ such that for
any $f\in L^{p}(\mathbb{R}^{N})$ and any $g\in L^{q}(\mathbb{R}^{N})$ it holds
that

(1.3)
$\displaystyle\int_{\mathbb{R}^{N}}dy\int_{\mathbb{R}^{N}}\frac{f(x)g(x)}{|x-y|^{\lambda}}dx\leq
C_{p,q,\lambda,N}|f|_{p}|g|_{q}.$

This inequality can be traced back to the work by Hardy and Littlewood [21] of
1928. In [28], Lieb studied its sharp constant. We refer [12, 14, 13, 25, 4,
15, 16] and the references therein for more detailed researches on Inequality
(1.3) and its variants. In this paper, we will frequently use Inequality with
$p=q=\frac{2N}{N+\alpha}$ and $\lambda=N-\alpha$. The best constant in this
case is denoted by $C_{HLS}$.

Now we consider term $\mathcal{D}_{\alpha}(u)$. First, notice that
$\mathcal{D}_{\alpha}(u)$ is well-defined on $H_{0}^{1}(\Omega)$ since

$\displaystyle\mathcal{D}_{\alpha}(u)$
$\displaystyle=C_{N,\alpha}\int_{\Omega}dy\int_{\Omega}\frac{u(x)u(y)}{|x-y|^{N-\alpha}}dx=C_{N,\alpha}\int_{\mathbb{R}^{N}}dy\int_{\mathbb{R}^{N}}\frac{\widetilde{u}(x)\widetilde{u}(y)}{|x-y|^{N-\alpha}}dx$
$\displaystyle\leq C|\widetilde{u}|_{\frac{2N}{N+\alpha}}^{2}\leq C|\nabla
u|_{2}^{2}.$

Here, $\widetilde{u}$ is the zero extension of $u$, i.e.

$\widetilde{u}(x)=\left\\{\begin{aligned} u(x)&\qquad&x\in\Omega,\\\
0&\qquad&x\notin\Omega.\end{aligned}\right.$

By a similar approach, we can prove that $\mathcal{D}_{\alpha}(u)$ is a
$C^{2}$ functional.

###  1.2. Main results

The first part of our result concerns the non-existence of solutions to
Problem (1.1).

######  Theorem 1.1.

Suppose that $\Omega$ is star shaped and that $\lambda\leq 0$, then Problem
(1.1) admits no non-trivial solutions.

Under certain assumptions, there is also a nonexistence result for Problem
(1.1) for some positive $\lambda>0$.

######  Theorem 1.2.

Suppose that $N-\alpha-4<0$ and that there exists a constant $c_{0}>0$ such
that $x\cdot n_{x}\geq c_{0}$ for any $x\in\partial\Omega$, then there exists
a constant $\Lambda_{0}=\Lambda_{0}(\alpha,\Omega)>0$ such that for any
$\lambda\leq\Lambda_{0}$, Problem (1.1) admits no positive solution.

On the other hand, we establish the existence result.

######  Theorem 1.3.

Problem (1.1) admits a solution if one of the following cases occurs.

  * $(1).$

$N-\alpha-4\geq 0$ and $\lambda>0$;

  * $(2).$

$N-\alpha-4<0$ and $\lambda>0$ large.

If, additionally, we assume that $0<\lambda<\lambda_{1}$ with

$\displaystyle\lambda_{1}=\inf_{u\neq 0}\frac{\int_{\Omega}|\nabla
u|^{2}dx}{\mathcal{D}_{\alpha}(u)},$

Problem (1.1) admits a positive solution.

To discuss the multiplicity results, we need the following nonlocal eigenvalue
problem.

(1.4) $\left\\{\begin{array}[]{lr}-{\Delta}u=\lambda
I_{\alpha}*_{\Omega}u\mbox{ in }\Omega,\\\ \lambda\in\mathbb{R},\,u\in
H_{0}^{1}(\Omega).\end{array}\right.$

For Problem (1.4), the following theorem holds.

######  Theorem 1.4.

There exists a sequence
$\\{(\lambda_{k},\phi_{k})\\}_{k}\subset\mathbb{R}\times H_{0}^{1}(\Omega)$
such that

  * $(1).$

$(\lambda_{k},\phi_{k})$ solves Problem (1.4);

  * $(2).$

$\phi_{k}\in C^{2,\theta}(\Omega)\cap C_{0}(\overline{\Omega})$ for some
$\theta\in(0,1)$ and $\\{\phi_{k}\\}_{k}$ forms an normalized orthogonal base
of $H_{0}^{1}(\Omega)$;

  * $(3).$

$0<\lambda_{1}<\lambda_{2}\leq\lambda_{3}\leq\cdots\to\infty$ and
$\phi_{1}>0$.

Based on Theorem 1.4, we get the following multiplicity results.

######  Theorem 1.5.

Under the assumption of Theorem 1.3, denote

  * •

$\upsilon=S\cdot|\Omega|_{N}^{-\frac{2+\alpha}{N}}\cdot
C_{HLS}^{-1}C_{N,\alpha}^{-1}$;

  * •

$m=\\#\\{j\in\mathbb{Z}_{+}|\lambda<\lambda_{j}<\lambda+\upsilon\\}$.

Then, Problem (1.1) admits at least $m$ pairs of solutions. Here,
$S=\inf_{u\neq 0}\frac{\int_{\mathbb{R}^{N}}|\nabla
u|^{2}dx}{(\int_{\mathbb{R}^{N}}|u|^{2^{*}}dx)^{\frac{2}{2^{*}}}}$ and
$\lambda_{k}$ are the eigenvalues in Theorem 1.4.

###  1.3. Organization of this paper

In Section 2, we prove Theorem 1.4. Theorems 1.1 and Theorem 1.2 are proved in
Section 3. To this end, we first show the regularity of the weak solutions to
Problem (1.1) in Subsection 3.1 and in Subsection 3.2 we prove the
corresponding Pohozaev identity. In Section 4, Theorem 1.3 is proved by
testing the mountain pass level and verifying a local compactness result and a
maximum principle for a nonlocal operator. In order to prove 1.5, we need to
estimate the genus of certain energy level sets. This is shown in Subsection
4.5.

Notations.

  * •

Throughout this paper, $C$ denotes a generic positive constant that may vary
from line to line;

  * •

$o_{n}(1)$ denotes a generic infinitesimal as $n\to\infty$;

  * •

$dS_{x}$ denotes the area element of $\partial\Omega$ and
$d\vec{S}_{x}=n_{x}dS_{x}$ where $n_{x}$ is the outer normal vector of
$\partial\Omega$ at $x$;

  * •

Without causing confusion, the norm of the spaces $L^{p}(\Omega)$ and of the
space $H_{0}^{1}(\Omega)$ is denoted by $|\cdot|_{p}$ and $\|\cdot\|$,
respectively;

  * •

The constant
$C_{N,\alpha}=\frac{\Gamma(\frac{N-\alpha}{2})}{\Gamma(\frac{\alpha}{2})\pi^{\frac{N}{2}}2^{\alpha}}$
and $C_{HLS}$ denotes the best constant of Inequality (1.3) with
$p=q=\frac{2N}{N+\alpha}$ and $\lambda=N-\alpha$;

  * •

$S=\inf\\{\int_{\mathbb{R}^{N}}|\nabla
u|^{2}dx|\int_{\mathbb{R}^{N}}|u|^{2^{*}}dx=1\\}>0$.

##  2\. Proof of Theorem 1.4: The eigenvalue problems

In this part, we prove Theorem 1.4 in steps. Different from the classical
approach [17, Chapter 6], we apply a minimax approach which can also be
applied to nonlinear eigenvalue problems and other nonlocal eigenvalue
problems, cf. [37, 36, 31].

###  2.1. The existence of eigenfunctions

######  Lemma 2.1.

The set $\mathcal{M}=\\{u\in H_{0}^{1}(\Omega)|\mathcal{D}_{\alpha}(u)=1\\}$
is a $C^{1}$ manifold in $H_{0}^{1}(\Omega)$ that is homomorphic to $S$. Here
$S=\\{u\in H_{0}^{1}(\Omega)|\int_{\Omega}|\nabla u|^{2}dx=1\\}$.

Proof. To verify this, it is sufficient to notice that

$\displaystyle\mathcal{D}_{\alpha}(u)=C_{N,\alpha}\int_{\Omega}dy\int_{\Omega}\frac{u(x)u(y)dx}{|x-y|^{N-\alpha}}=C_{N,\alpha}\int_{\mathbb{R}^{N}}dy\int_{\mathbb{R}^{N}}\frac{\widetilde{u}(x)\widetilde{u}(y)dx}{|x-y|^{N-\alpha}}.$

Here, $\widetilde{u}$ is the zero-extension of $u$ outside of $\Omega$. Notice
that

$\displaystyle|\mathcal{D}_{\alpha}(u)|\leq
C|\widetilde{u}|^{2}_{\frac{2N}{N+\alpha}}\leq C|\nabla u|_{2}^{2}.$

This is due to Hardy-Littlewood-Sobolev inequality and Hölder’s inequality.
This implies that $\mathcal{D}_{\alpha}(u)<\infty$ for any
$H_{0}^{1}(\Omega)$. On the other hand, by Parseval’s identity,

$\displaystyle\mathcal{D}_{\alpha}(u)=C_{N,\alpha}\int_{\mathbb{R}^{N}}|I_{\frac{\alpha}{2}}*\widetilde{u}|^{2}dx.$

Therefore, $D_{\alpha}(u)=0$ if and only if $\widetilde{u}=0$, i.e. $u=0$ in
$\Omega$.

$\Box$

Another necessary result is that

######  Lemma 2.2.

$\int_{\Omega}|\nabla u|^{2}dx$ satisfies $(PS)$ condition on $\mathcal{M}$.
To be precise, if the sequence $\\{u_{n}\\}_{n}\subset H_{0}^{1}(\Omega)$ and
$\\{\lambda_{n}\\}_{n}\subset\mathbb{R}$ and a number
$\lambda_{*}\in\mathbb{R}$ satisfy

  * $(a).$

$\int_{\Omega}|\nabla u_{n}|^{2}dx\to\lambda_{*}$;

  * $(b).$

$u_{n}-\lambda_{n}(-\Delta)_{\Omega}^{-1}I_{\alpha}*_{\Omega}u_{n}=o_{n}(1)$,

then $\lambda_{n}\to\lambda_{*}$ and there exists $u_{\infty}\in\mathcal{M}$
such that $u_{n}\to u_{\infty}$ in $H_{0}^{1}(\Omega)$.

Proof. Notice that $\\{u_{n}\\}_{n}$ is bounded in $H_{0}^{1}(\Omega)$, then
define $u_{\infty}$ as the weak limit. By a direct computation,

(2.1) $\displaystyle\mathcal{D}_{\alpha}(u_{n}-u_{\infty})\leq
C|u_{n}-u_{\infty}|_{\frac{2N}{N+\alpha}}\to 0$

due to the Sobolev embedding and

$\displaystyle\mathcal{D}_{\alpha}(u_{n}-u_{\infty})=\mathcal{D}_{\alpha}(u_{n})+\mathcal{D}_{\alpha}(u_{\infty})-2C_{N,\alpha}\int_{\Omega}dy\int_{\Omega}\frac{u_{n}(x)u_{\infty}(y)}{|x-y|^{N-\alpha}}dx.$

Since

  * •

$u_{n}\to u_{\infty}$ in $L^{2}(\Omega)$;

  * •

$I_{\alpha}*_{\Omega}u_{\infty}=I_{\alpha}*_{\mathbb{R}^{N}}\widetilde{u}_{\infty}|_{\Omega}\in
L^{2}(\Omega)$. Here, $\widetilde{u}_{\infty}$ is the zero extension of
$u_{\infty}$,

we get

(2.2)
$\displaystyle\mathcal{D}_{\alpha}(u_{n}-u_{\infty})=\mathcal{D}_{\alpha}(u_{n})-\mathcal{D}_{\alpha}(u_{\infty}).$

Moreover,

(2.3) $\displaystyle\mathcal{D}_{\alpha}(u_{n}-u_{\infty})\geq 0.$

Then, (2.1), (2.2) and (2.3) imply that $\mathcal{D}_{\alpha}(u_{\infty})=1$.
Multiplying $(b)$ by $u_{n}$, we get

$\displaystyle\int_{\Omega}|\nabla
u_{n}|^{2}dx=\lambda_{n}\mathcal{D}_{\alpha}(u_{n})+o_{n}(1).$

Combining $(a)$, we get $\lambda_{n}\to\lambda_{*}$. Multiplying $(b)$ by
$u_{\infty}$,

$\displaystyle\int_{\Omega}\nabla u_{n}\cdot\nabla
u_{\infty}dx-\lambda_{n}C_{N,\alpha}\int_{\Omega}dy\int_{\Omega}\frac{u_{n}(x)u_{\infty}(y)}{|x-y|^{N-\alpha}}dx+o_{n}(1)=0.$

Then,

$\displaystyle\|u_{n}-u_{\infty}\|^{2}=\int_{\Omega}|\nabla
u_{n}|^{2}dx-\int_{\Omega}|\nabla
u_{\infty}|^{2}dx=\lambda_{n}-\lambda_{*}+o_{n}(1)\to 0.$

This completes the proof.

$\Box$

Consider the following minimax problem

(2.4) $\displaystyle\lambda_{k}=\inf_{A\subset\mathcal{M},\gamma(A)\geq
k}\sup_{u\in A}\frac{\int|\nabla u|^{2}dx}{\mathcal{D}_{\alpha}(u)}.$

Here, $\gamma$ is the $\mathbb{Z}_{2}$-genus generated by the anti-podal
action $u\mapsto-u$. By a standard method (see, for instance, [2] and [37]),
we obtain a sequence
$\\{(\lambda_{k},\phi_{k})\\}_{k}\subset\mathbb{R}\times\mathcal{M}$ solving
Problem (1.4) and $\lambda_{k}\to+\infty$.

Especially,

$\displaystyle\lambda_{1}=\inf_{u\in\mathcal{M}}\frac{\int|\nabla
u|^{2}dx}{\mathcal{D}_{\alpha}(u)}.$

By Hardy-Littlewood-Sobolev inequality,

$\displaystyle\mathcal{D}_{\alpha}(u)$
$\displaystyle=C_{N,\alpha}\int_{\Omega}dy\int_{\Omega}\frac{u(x)u(y)}{|x-y|^{N-\alpha}}dx=C_{N,\alpha}\int_{\mathbb{R}^{N}}dy\int_{\mathbb{R}^{N}}\frac{\widetilde{u}(x)\widetilde{u}(y)}{|x-y|^{N-\alpha}}dx$
$\displaystyle\leq
C|\widetilde{u}|^{2}_{L^{\frac{2N}{N+\alpha}}(\mathbb{R}^{N})}\leq C|\nabla
u|_{2}^{2}.$

Here, $\widetilde{u}$ is the zero-extension of $u$. [37, Theorem 8.10] and [2]
imply the following result.

######  Proposition 2.3.

There exists a sequence of solutions $\\{(\lambda_{k},\phi_{k})\\}_{k}$ to
Problem (1.4) such that
$0<\lambda_{1}\leq\lambda_{2}\leq\lambda_{3}\leq\cdots$.

###  2.2. The regularity of eigenfunctions

In this part, we prove a regularity result for the solutions to Problem (1.4).
We start with a function $u\in H_{0}^{1}(\Omega)$ such that

(2.5) $\displaystyle-\Delta u=\lambda
C_{N,\alpha}\int_{\Omega}\frac{u(y)}{|x-y|^{N-\alpha}}dy$

for some $\lambda>0$. Recall [18, Lemma 7.12].

######  Lemma 2.4.

The operator $I_{\alpha}*_{\Omega}\cdot:L^{p}(\Omega)\to L^{q}(\Omega)$
continuously for any $q\in[1,\infty]$ and $p$ satisfying
$0\leq\frac{1}{p}-\frac{1}{q}<\frac{\alpha}{N}$. To be precise, there exists a
constant $C>0$ such that $|I_{\alpha}*_{\Omega}u|_{q}\leq C|u|_{p}$ for any
$u\in L^{p}(\Omega)$.

In follows, it is sufficient to use the case $p=q$.

######  Lemma 2.5.

Let $(\lambda,u)$ solving Problem (2.5). For any $r\geq 1$, we get $u\in
W^{2,r}(\Omega)$.

Proof. To begin with, let us recall that $u\in H_{0}^{1}(\Omega)$. Then, $u\in
L^{\frac{2N}{N-2}}(\Omega)$ and Sobolev inequality and Lemma 2.4 give

  * $\mbox{Step 1}.$

$u\in W^{2,\frac{2N}{N-2}}(\Omega)$, then $u\in L^{\frac{2N}{N-6}}(\Omega)$
and $I_{\alpha}*_{\Omega}u\in L^{\frac{2N}{N-6}}(\Omega)$;

  * $\mbox{Step 2}.$

$u\in W^{2,\frac{2N}{N-6}}(\Omega)$, then $u\in L^{\frac{2N}{N-10}}$ and hence
$I_{\alpha}*_{\Omega}u\in L^{\frac{2N}{N-10}}(\Omega)$;

  * ……

  * $\mbox{Step k}.$

$u\in W^{2,\frac{2N}{N-4k+2}}(\Omega)$, then $u\in
L^{\frac{2N}{N-4k-2}}(\Omega)$ and hence $I_{\alpha}*_{\Omega}u\in
L^{\frac{2N}{N-4k-2}}(\Omega)$;

  * ……

Let $k_{m}:=\max\\{k\in\mathbb{N}|N-4k>2\\}$, then

  * $\mbox{Case 1}.$

$2=N-4(k_{m}+1)$, then for any $r\geq 1$, $u\in L^{r}(\Omega)$;

  * $\mbox{Case 2}.$

$2>N-4(k_{m}+1)$, then $u\in L^{\infty}(\Omega)$ and for any $r\geq 1$, $u\in
L^{r}(\Omega)$.

This implies that $u\in W^{2,r}(\Omega)$ for any $r\geq 1$.

$\Box$

To proceed, we need to introduce the $M^{p}(\Omega)$ space and the associated
potential estimate.

######  Definition 2.6.

$M^{p}(\Omega)=\\{u\in L^{1}_{loc}(\Omega)|\exists K>0\mbox{ s.t. }\forall
x\in\Omega\mbox{ and }R>0,\,\int_{\Omega\cap B_{R}(x)}|u|dx\leq
KR^{N(1-\frac{1}{p})}\\}$. The norm of $M^{p}(\Omega)$ is defined as

$\displaystyle\|u\|_{M^{p}(\Omega)}=\inf\Big{\\{}K>0\Big{|}\forall
x\in\Omega\mbox{ and }R>0,\,\int_{\Omega\cap B_{R}(x)}|u|dx\leq
KR^{N(1-\frac{1}{p})}\Big{\\}}.$

######  Remark 2.7.

It holds that $L^{1}(\Omega)=M^{1}(\Omega)$, $L^{p}(\Omega)\subset
M^{p}(\Omega)$ and $L^{\infty}(\Omega)=M^{\infty}(\Omega)$. We refer [18,
Chapter 7.9].

The following result is [18, Lemma 7.18].

######  Lemma 2.8.

If $\frac{N}{\alpha}<p$, there exists a constant $C>0$,
$|I_{\alpha}*_{\Omega}u|_{\infty}\leq C\|u\|_{M^{p}(\Omega)}$ for any $u\in
M^{p}(\Omega)$.

######  Lemma 2.9.

Let $(\lambda,u)$ solving Problem (2.5). Then, we get $\nabla
I_{\alpha}*_{\Omega}u\in L^{\infty}(\Omega)$.

Proof. Using Lemma 2.5, we get a $\theta\in(0,1)$ such that $u\in
C_{0}^{1,\theta}(\Omega)$. Then, $|\nabla u|\in C^{0,\theta}(\Omega)\subset
L^{\infty}(\Omega)$. On the other hand, for any $x\in\Omega$,

$\displaystyle\nabla_{x}\int_{\Omega}\frac{u(y)}{|x-y|^{N-\alpha}}dy$
$\displaystyle=\nabla_{x}\int_{\mathbb{R}^{N}}\frac{\widetilde{u}(y)}{|x-y|^{N-\alpha}}dy=\int_{\mathbb{R}^{N}}\frac{\nabla\widetilde{u}(y)}{|x-y|^{N-\alpha}}dy$
$\displaystyle=\int_{\Omega}\frac{\nabla u(y)}{|x-y|^{N-\alpha}}dy.$

Here, $\widetilde{u}$ is the zero extension of $u$. Then,
$|\nabla_{x}\int_{\Omega}\frac{u(y)}{|x-y|^{N-\alpha}}dy|\leq\int_{\Omega}\frac{|\nabla
u(y)|}{|x-y|^{N-\alpha}}$ for any $x\in\Omega$. Then, Lemma 2.8 gives that
$\nabla I_{\alpha}*_{\Omega}u\in L^{\infty}(\Omega)$.

$\Box$

######  Proposition 2.10.

For any solution $(\lambda,u)$ to Problem (2.5), $u\in
C^{2,\theta}(\Omega)\cap C_{0}(\overline{\Omega})$ for some $\theta\in(0,1)$.

Proof. Lemma 2.9 implies that $I_{\alpha}*_{\Omega}u\in C^{0,\theta}(\Omega)$
for some $\theta\in(0,1)$. The lemma follows the classical Hölder estimate
[18, Chapter 6].

$\Box$

###  2.3. Orthogonality and completeness

The following result is evident.

######  Claim 2.11.

For the sequence we obtain in Proposition 2.3, it holds that

  * $(1).$

For any $k$ and any $\mu\in\mathbb{R}$, $(\lambda_{k},\mu\phi_{k})$ solves
Problem (1.4);

  * $(2).$

If there exists $k_{0},p_{0}$ such that
$\lambda_{k_{0}+1}=\cdots=\lambda_{k_{0}+p_{0}}$, then for any
$\mu_{1},\cdots,\mu_{p_{0}}$,
$(\lambda_{k_{0}+1},\sum_{i=1}^{p_{0}}\mu_{i}\phi_{i})$ solves Problem (1.4).

Furthermore, we get

######  Lemma 2.12.

If $i,j=1,2,\cdots$ such that $\lambda_{i}\neq\lambda_{j}$, then
$\int_{\Omega}\nabla\phi_{i}\cdot\nabla\phi_{j}dx=0$.

Proof. Notice that

(2.6) $\displaystyle-{\Delta}\phi_{i}=\lambda_{i}I_{\alpha}*_{\Omega}\phi_{i}$

and

(2.7) $\displaystyle-\Delta\phi_{j}=\lambda_{j}I_{\alpha}*_{\Omega}\phi_{j}.$

Multiplying (2.6) by $\frac{1}{\lambda_{i}}\phi_{j}$, (2.7) by
$\frac{1}{\lambda_{j}}\phi_{i}$, integrating over $\Omega$ and taking the
difference, we get

$\displaystyle\int_{\Omega}\nabla\phi_{i}\cdot\nabla\phi_{j}dx=0.$

$\Box$

######  Remark 2.13.

Moreover, under the assumptions of Lemma 2.12, we also get
$\int_{\Omega}dy\int_{\Omega}\frac{\phi_{i}(x)\phi_{j}(y)}{|x-y|^{N-\alpha}}dx=0$.

Therefore, without loss of generality, we can assume that $\\{\phi_{k}\\}_{k}$
are pairwise orthogonal. By the $(PS)$ condition, it is also clear that

######  Claim 2.14.

For any $k=1,2,\cdots$,
$\sup\\{l>0|\lambda_{k+1}=\cdots=\lambda_{k+l}\\}<\infty$.

Now we verify the completeness of $\\{\phi_{k}\\}_{k}$.

######  Proposition 2.15.

It holds that
$H_{0}^{1}(\Omega)=\overline{\mbox{span}\\{\phi_{1},\cdots,\phi_{k},\cdots\\}}$.

Proof. Otherwise, if there exists a function $\phi_{*}\in H_{0}^{1}(\Omega)$
such that
$\phi_{*}\perp\overline{\mbox{span}\\{\phi_{1},\cdots,\phi_{k},\cdots\\}}$.
Denote $H_{0}:=\overline{\mbox{span}\\{\phi_{1},\cdots,\phi_{k},\cdots\\}}$
and
$H_{0}^{\perp}:=(\overline{\mbox{span}\\{\phi_{1},\cdots,\phi_{k},\cdots\\}})^{\perp}$.
Then $\phi_{*}\in H_{0}^{\perp}$. Now we claim that

######  Claim 2.16.

$\mathcal{N}=\mathcal{M}\cap H_{0}^{\perp}$ is a natural constrain for
$\int|\nabla u|^{2}dx$ on $\mathcal{M}$.

If $\nabla_{\mathcal{N}}\int|\nabla u|^{2}dx=0$ for $u\in\mathcal{N}$, we want
to prove $\nabla_{\mathcal{M}}\int|\nabla u|^{2}dx=0$. To do this, let us
select a $\phi_{j}\in H_{0}$. Then,

$\displaystyle 0=\int_{\Omega}\nabla
u\cdot\nabla\phi_{j}dx=\int_{\Omega}u(-\Delta\phi_{j})dx=\lambda_{j}C_{N,\alpha}\int_{\Omega}dy\int_{\Omega}\frac{u(x)\phi_{j}(y)}{|x-y|^{N-\alpha}}dx,$

since $u\perp\phi_{j}$ in $H_{0}^{1}(\Omega)$ and $\phi_{j}$ is the $j$-th
eigenfunction. In this computation, we also proved that $v\in
T_{u}\mathcal{M}$. The same computation implies that
$\nabla_{\mathcal{M}}\int|\nabla u|^{2}dx=0$ and the claim is proved.

Then, we obtain an eigenvalue and an eigenfunction $(\lambda_{0},u_{0})$ with
$0<\lambda_{0}<\infty$. This contradicts with the order of the eigenvalues
$\lambda_{1}\leq\lambda_{2}\leq\cdots$. Therefore, $H_{0}^{\perp}=0$, i.e.
$H_{0}^{1}(\Omega)=\overline{\mbox{span}\\{\phi_{1},\cdots,\phi_{k},\cdots\\}}$.

$\Box$

###  2.4. Principal eigenfunction

In this part, we analyze the principal eigenfunction and the principle
eigenvalue. Recall that

(2.8) $\displaystyle\lambda_{1}=\inf_{u\in\mathcal{M}}\int_{\Omega}|\nabla
u|^{2}dx$

and $\phi_{1}$ attends the above minimum.

######  Proposition 2.17.

It holds that $\phi_{1}>0$.

Proof. Otherwise, we assume that $\phi_{1}$ changes its sign. Let
$\phi_{1,+}=\max\\{\phi_{1},0\\}$ and $\phi_{1,-}=\min\\{\phi_{1},0\\}$. Then,

$\displaystyle\int_{\Omega}|\nabla\phi_{1}|^{2}dx$
$\displaystyle=\lambda_{1}C_{N,\alpha}\int_{\Omega}dy\int_{\Omega}\frac{\phi_{1}(x)\phi_{1}(y)}{|x-y|^{N-\alpha}}dx$
$\displaystyle=\lambda_{1}C_{N,\alpha}\int_{\Omega}dy\int_{\Omega}\frac{\phi_{1,+}(x)\phi_{1,+}(y)}{|x-y|^{N-\alpha}}dx+\lambda_{1}C_{N,\alpha}\int_{\Omega}dy\int_{\Omega}\frac{\phi_{1,-}(x)\phi_{1,-}(y)}{|x-y|^{N-\alpha}}dx$
$\displaystyle\quad+2\lambda_{1}C_{N,\alpha}\int_{\Omega}dy\int_{\Omega}\frac{\phi_{1,+}(x)\phi_{1,-}(y)}{|x-y|^{N-\alpha}}dx.$

Here,

$\displaystyle\lambda_{1}C_{N,\alpha}\int_{\Omega}dy\int_{\Omega}\frac{\phi_{1,+}(x)\phi_{1,-}(y)}{|x-y|^{N-\alpha}}dx\leq
0.$

If
$\lambda_{1}C_{N,\alpha}\int_{\Omega}dy\int_{\Omega}\frac{\phi_{1,+}(x)\phi_{1,-}(y)}{|x-y|^{N-\alpha}}dx<0$,
then

$\displaystyle\int_{\Omega}|\nabla\phi_{1}|^{2}dx<\lambda_{1}C_{N,\alpha}\int_{\Omega}dy\int_{\Omega}\frac{\phi_{1,+}(x)\phi_{1,-}(y)}{|x-y|^{N-\alpha}}dx+\lambda_{1}C_{N,\alpha}\int_{\Omega}dy\int_{\Omega}\frac{\phi_{1,-}(x)\phi_{1,-}(y)}{|x-y|^{N-\alpha}}dx.$

Then, either

$\displaystyle\int_{\Omega}|\nabla\phi_{1,+}|^{2}dx<\lambda_{1}C_{N,\alpha}\int_{\Omega}dy\int_{\Omega}\frac{\phi_{1,+}(x)\phi_{1,+}(y)}{|x-y|^{N-\alpha}}dx$

or

$\displaystyle\int_{\Omega}|\nabla\phi_{1,-}|^{2}dx<\lambda_{1}C_{N,\alpha}\int_{\Omega}dy\int_{\Omega}\frac{\phi_{1,-}(x)\phi_{1,-}(y)}{|x-y|^{N-\alpha}}dx.$

This contradict with (2.8). Then,

$\displaystyle\lambda_{1}C_{N,\alpha}\int_{\Omega}dy\int_{\Omega}\frac{\phi_{1,+}(x)\phi_{1,-}(y)}{|x-y|^{N-\alpha}}dx=0.$

Hence,

$\displaystyle\int_{\Omega}|\nabla\phi_{1}|^{2}dx=\lambda_{1}C_{N,\alpha}\int_{\Omega}dy\int_{\Omega}\frac{\phi_{1,+}(x)\phi_{1,-}(y)}{|x-y|^{N-\alpha}}dx+\lambda_{1}C_{N,\alpha}\int_{\Omega}dy\int_{\Omega}\frac{\phi_{1,-}(x)\phi_{1,-}(y)}{|x-y|^{N-\alpha}}dx.$

Then, $\phi_{1,+}$ and $\phi_{1,-}$ also minimize (2.8) and

$\displaystyle-\Delta\phi_{1,\pm}=\lambda_{1}I_{\alpha}*_{\Omega}\phi_{1,\pm}\mbox{
in }\Omega.$

This contradicts with the strong maximum principle of $-\Delta$, cf. [17,
Chapter 6.4].

$\Box$

###  2.5. Proof of Theorem 1.4

The proof of Theorem 1.4 follows the above considerations.

Proof of Theorem 1.4. Assertion $(1)$ of Theorem 1.4 is proved by Proposition
2.3, i.e. there exists a sequence of pairs $\\{(\lambda_{k},\phi_{k})\\}_{k}$
whose elements solve Problem (1.4). By Proposition 2.10, for any
$k=1,2,\cdots$, $\phi_{k}\in C^{2,\theta}(\Omega)\cap
C_{0}(\overline{\Omega})$ for some $\theta\in(0,1)$. For the set
$\\{\phi_{k}\\}_{k}$, the orthogonality and the completeness are guaranteed by
Lemma 2.12 and Proposition 2.15. This proves Assertion $(2)$ of Theorem 1.4.

As for Assertion $(3)$, using Proposition 2.17, the principal eigenvalue
$\lambda_{1}$ is simple. The proof is complete.

$\Box$

##  3\. Proof of Theorems 1.1 and 1.2: Non-existence results

###  3.1. The regularity of weak solutions

The proofs of Theorems 1.1 and 1.2 are based on [7]. We start with $u\in
H^{1}_{0}(\Omega)$ solving Problem (1.1), i.e.,

$\left\\{\begin{array}[]{lr}-{\Delta}u=\lambda
I_{\alpha}*_{\Omega}u+|u|^{2^{*}-2}u\mbox{ in }\Omega,\\\ u\in
H_{0}^{1}(\Omega).\end{array}\right.$

We want to prove that $u\in C^{1,\theta}(\Omega)$ for some $\theta\in(0,1)$.
Recall the following Hardy-Littlewood-Sobolev inequality:

$\displaystyle\Big{|}\int_{\mathbb{R}^{N}}dy\int_{\mathbb{R}^{N}}\frac{f(x)g(y)}{|x-y|^{N-\alpha}}dx\Big{|}\leq
C|f|_{p}|g|_{t}$

with $\frac{1}{p}+\frac{1}{t}=1+\frac{\alpha}{N}$. First, we have that

######  Lemma 3.1.

Let $u$ be a solution to Problem (1.1) in $H_{0}^{1}(\Omega)$. For any $r\geq
1$, $u\in L^{r}(\Omega)$.

Proof. Multiplying Problem (1.1) by $|u|^{s-1}u$ and integrating over
$\Omega$, we get

$\displaystyle\frac{4s}{(s+1)^{2}}\int_{\Omega}|\nabla(u^{\frac{s+1}{2}})|^{2}dx=\int_{\Omega}(-\Delta
u)|u|^{s-1}udx\leq\lambda
C\int_{\mathbb{R}^{N}}dy\int_{\mathbb{R}^{N}}\frac{|\widetilde{u}(y)\widetilde{u}(x)^{s}|}{|x-y|^{N-\alpha}}dx+|u|_{\frac{N+2}{N-2}+s}^{\frac{N+2}{N-2}+s}.$

Here, $\widetilde{u}$ is the zero extension of $u$. Using Sobolev inequality
and Hardy-Littlewood-Sobolev inequality,

$\displaystyle|u|_{\frac{(s+1)N}{N-2}}^{s}\leq
C\Big{(}|u|^{s+1}_{\frac{N(s+1)}{N+\alpha}}+|u|_{\frac{N+2}{N-2}+s}^{\frac{N+2}{N-2}+s}\Big{)}.$

Noticing $\frac{N(s+1)}{N+\alpha}\leq\frac{N+2}{N-2}+s$, then

(3.1) $\displaystyle|u|_{\frac{(s+1)N}{N-2}}^{s}\leq
C\Big{(}|u|^{s+1}_{\frac{N+2}{N-2}+s}+|u|_{\frac{N+2}{N-2}+s}^{\frac{N+2}{N-2}+s}\Big{)}.$

Letting $s_{i+1}=\frac{N}{N-2}\cdot s_{i}-\frac{2}{N-2}$. We can prove that
for any $r\geq 1$, $u\in L^{r}(\Omega)$.

$\Box$

######  Lemma 3.2.

Let $u$ be a solution to Problem (1.1) in $H_{0}^{1}(\Omega)$. There exists a
$\theta\in(0,1)$, $u\in C^{2,\theta}(\Omega)$.

Proof. By Lemma 3.1 and 2.4, we get for any $r\geq 1$, $u\in W^{2,r}(\Omega)$.
Then, for any $r\geq 1$, $|\nabla u|\in L^{r}(\Omega)$. Notice that for any
$x\in\Omega$,

$\displaystyle\nabla_{x}\int_{\Omega}\frac{u(y)}{|x-y|^{N-\alpha}}dy$
$\displaystyle=\nabla_{x}\int_{\mathbb{R}^{N}}\frac{\widetilde{u}(y)}{|x-y|^{N-\alpha}}dy=\int_{\mathbb{R}^{N}}\frac{\nabla\widetilde{u}(y)}{|x-y|^{N-\alpha}}dy$
$\displaystyle=\int_{\Omega}\frac{\nabla u(y)}{|x-y|^{N-\alpha}}dy.$

Lemma 2.9 implies that
$|\nabla_{x}\int_{\Omega}\frac{u(y)}{|x-y|^{N-\alpha}}dy|\in
L^{\infty}(\Omega)$. Then, $I_{\alpha}*_{\Omega}u\in C^{0,\theta}(\Omega)$ for
some $\theta\in(0,1)$. Combing the classical Hölder estimate (cf. [18, Chapter
6]), we complete the proof.

$\Box$

###  3.2. Pohozaev identity

In this part, we derive a Pohozaev identity for Problem (1.1). First, let us
consider the potential term $I_{\alpha}*_{\Omega}u$.

######  Lemma 3.3.

It holds that

$\displaystyle\int_{\Omega}(I_{\alpha}*_{\Omega}u)(x\cdot\nabla
u)dx=-\frac{N+\alpha}{2}C_{N,\alpha}\int_{\Omega}dx\int_{\Omega}\frac{u(x)u(y)}{|x-y|^{N-\alpha}}dy.$

Proof. By a direct computation,

$\displaystyle\mbox{div}_{x}\Big{(}(I_{\alpha}*_{\Omega}u)x\cdot
u\Big{)}=\mbox{div}_{x}\Big{(}(I_{\alpha}*_{\Omega})x\Big{)}u+(I_{\alpha}*_{\Omega})x\cdot\nabla
u.$

This gives that

$\displaystyle\int_{\Omega}(I_{\alpha}*_{\Omega}u)(x\cdot\nabla u)dx$
$\displaystyle=-\int_{\Omega}dx\Big{[}\mbox{div}_{x}\Big{(}(I_{\alpha}*_{\Omega}u)x\Big{)}u\Big{]}+\int_{\Omega}dx\Big{[}\mbox{div}_{x}\Big{(}(I_{\alpha}*_{\Omega}u)xu(x)\Big{)}\Big{]}$
$\displaystyle=-\int_{\Omega}dx\Big{[}\mbox{div}_{x}\Big{(}(I_{\alpha}*_{\Omega}u)x\Big{)}u\Big{]}$

since $u\in H_{0}^{1}(\Omega)$. Moreover, notice that

$\displaystyle\mbox{div}_{x}\Big{(}\frac{x}{|x-y|^{N-\alpha}}\Big{)}=\frac{N}{|x-y|^{N-\alpha}}-(N-\alpha)\frac{(x-y)\cdot
x}{|x-y|^{N-\alpha+2}}.$

We get

$\displaystyle\int_{\Omega}(I_{\alpha}*_{\Omega}u)(x\cdot\nabla u)dx$
$\displaystyle=-NC_{N,\alpha}\int_{\Omega}dx\int_{\Omega}\frac{u(x)u(y)}{|x-y|^{N-\alpha}}dy$
$\displaystyle\qquad+(N-\alpha)C_{N,\alpha}\int_{\Omega}dx\int_{\Omega}\frac{(x-y)\cdot
xu(x)u(y)}{|x-y|^{N-\alpha+2}}dy.$

By the symmetry of the double integral,

$\displaystyle 2\int_{\Omega}dx\int_{\Omega}\frac{(x-y)\cdot
xu(x)u(y)}{|x-y|^{N-\alpha+2}}dy$
$\displaystyle=\int_{\Omega}dx\int_{\Omega}\frac{(x-y)\cdot
xu(x)u(y)}{|x-y|^{N-\alpha+2}}dy$
$\displaystyle\qquad-\int_{\Omega}dx\int_{\Omega}\frac{(x-y)\cdot
yu(x)u(y)}{|x-y|^{N-\alpha+2}}dy$
$\displaystyle=\int_{\Omega}dx\int_{\Omega}\frac{u(x)u(y)}{|x-y|^{N-\alpha}}dy.$

Hence,

$\displaystyle\int_{\Omega}(I_{\alpha}*_{\Omega}u)(x\cdot\nabla
u)dx=-\frac{N+\alpha}{2}C_{N,\alpha}\int_{\Omega}dx\int_{\Omega}\frac{u(x)u(y)}{|x-y|^{N-\alpha}}dy.$

$\Box$

Now we prove Pohozaev identity.

######  Lemma 3.4.

For any solution to Problem (1.1), it holds that

$\displaystyle 0$ $\displaystyle=\int_{\partial\Omega}\Big{|}\frac{\partial
u}{\partial n}\Big{|}^{2}x\cdot d\vec{S}_{x}+(N-2)\int_{\Omega}|\nabla
u|^{2}dx-\lambda
C_{N,\alpha}(N+\alpha)\int_{\Omega}dy\int_{\Omega}\frac{u(x)u(y)}{|x-y|^{N-\alpha}}dx$
$\displaystyle\quad-(N-2)\int_{\Omega}|u|^{2^{*}}dx.$

Proof. To do this, only need to notice that

$\displaystyle 0=\int_{\Omega}(\Delta u+\lambda
I_{\alpha}*_{\Omega}u+|u|^{2^{*}-2}u)(x\cdot\nabla u)=I_{1}+I_{2}+I_{3}.$

Here,

$\displaystyle I_{1}$ $\displaystyle=\int_{\Omega}\Delta u(x\cdot\nabla u)dx$
$\displaystyle=\int_{\Omega}\Big{[}\mbox{div}\big{(}\nabla u(x\cdot\nabla
u)\big{)}-|\nabla u|^{2}-x\cdot\nabla\big{(}\frac{|\nabla
u|^{2}}{2}\big{)}\Big{]}dx$
$\displaystyle=\int_{\Omega}\Big{[}\mbox{div}\big{(}\nabla(x\cdot\nabla
u)-x\frac{|\nabla u|^{2}}{2}\big{)}+\frac{N-2}{2}|\nabla u|^{2}\Big{]}dx.$

Since for any $x\in\Omega$, $x\cdot\nabla u=x\cdot n\frac{\partial u}{\partial
n}$, then

$\displaystyle I_{1}=\frac{1}{2}\int_{\partial\Omega}\Big{|}\frac{\partial
u}{\partial n}\Big{|}^{2}x\cdot d\vec{S}_{x}+\frac{N-2}{2}\int_{\Omega}|\nabla
u|^{2}dx.$

Moreover,

$\displaystyle I_{2}=\lambda\int_{\Omega}(I_{\alpha}*_{\Omega}u)(x\cdot\nabla
u)dx=-\frac{\lambda
C_{N,\alpha}(N+\alpha)}{2}\int_{\Omega}dx\int_{\Omega}\frac{u(x)u(y)}{|x-y|^{N-\alpha}}dy.$

due to Lemma 3.3. And,

$\displaystyle I_{3}$
$\displaystyle=\int_{\Omega}|u|^{2^{*}-2}u\cdot(x\cdot\nabla
u)dx=\int_{\Omega}\Big{[}\mbox{div}\big{(}\frac{x}{2^{*}}|u|^{2^{*}}\big{)}-\frac{N-2}{2}|u|^{2^{*}}\Big{]}dx$
$\displaystyle=-\frac{N-2}{2}\int_{\Omega}|u|^{2^{*}}dx.$

The lemmas follows immediately.

$\Box$

###  3.3. Proof of Theorem 1.1

In this part, we prove Theorem 1.1.

Proof of Theorem 1.1. Let $u$ be a solution to Problem (1.1), then

$\displaystyle\int_{\Omega}|\nabla u|^{2}dx-\lambda
C_{N,\alpha}\int_{\Omega}dx\int_{\Omega}\frac{u(x)u(y)}{|x-y|^{N-\alpha}}dy=\int_{\Omega}|u|^{2^{*}}dx.$

By Lemma 3.4, we get

(3.2) $\displaystyle 0=\int_{\partial\Omega}\Big{|}\frac{\partial u}{\partial
n}\Big{|}^{2}x\cdot d\vec{S}_{x}-(\alpha+2)\lambda
C_{N,\alpha}\int_{\Omega}dx\int_{\Omega}\frac{u(x)u(y)}{|x-y|^{N-\alpha}}dy.$

Suppose that $\Omega$ is star-shaped, if $\lambda<0$, we get $u\equiv 0$
immediately. If $\lambda=0$, $u\equiv 0$ due to the unique continuation, cf.
[23].

$\Box$

###  3.4. Proof of Theorem 1.2

In this part, we prove Theorem 1.2.

Proof of Theorem 1.2. By Pohozaev identity, if Problem (1.1) admits a positive
solution,

$\displaystyle\lambda
C_{N,\alpha}\int_{\Omega}dx\int_{\Omega}\frac{u(x)u(y)}{|x-y|^{N-\alpha}}dy$
$\displaystyle=\frac{1}{\alpha+2}\int_{\partial\Omega}\Big{|}\frac{\partial
u}{\partial n}\Big{|}^{2}x\cdot d\vec{S}_{x}$
$\displaystyle\geq\frac{c_{0}}{\alpha+2}\int_{\partial\Omega}\Big{|}\frac{\partial
u}{\partial n}\Big{|}^{2}dS_{x}$
$\displaystyle\geq\frac{c_{0}|\partial\Omega|_{N-1}^{-1}}{\alpha+2}\Big{(}\int_{\partial\Omega}\frac{\partial
u}{\partial n}dS_{x}\Big{)}^{2}$
$\displaystyle=\frac{c_{0}|\partial\Omega|_{N-1}^{-1}}{\alpha+2}\Big{(}\int_{\Omega}(-\Delta
u)dx\Big{)}^{2}.$

Noticing that $u|_{\partial\Omega}=0$, by the Sobolev inequality involving
$|\Delta u|_{1}$ (see, for instance, [8, Theorem 8]), if $N-\alpha-4<0$, there
exists a constant $C_{*}>0$ independence in $u$ and

$\displaystyle|u|_{\frac{2N}{N+\alpha}}\leq C\|u\|_{W^{1,q}(\Omega)}\leq
C_{*}|\Delta u|_{1}$

for any $1\leq q<\frac{N}{N-1}$. Then, by Hardy-Littlewood-Sobolev inequality,
there exists a constant $C=C(N,\alpha)>0$ such that

$\displaystyle\lambda
C_{N,\alpha}\int_{\Omega}dx\int_{\Omega}\frac{u(x)u(y)}{|x-y|^{N-\alpha}}dy\geq\frac{C|\Omega|_{N}^{\frac{N-\alpha-4}{N}}}{|\partial\Omega|_{N-1}}C_{N,\alpha}\int_{\Omega}dx\int_{\Omega}\frac{u(x)u(y)}{|x-y|^{N-\alpha}}dy.$

This proves the result.

$\Box$

##  4\. Proof of Theorems 1.3 and 1.5: Existence and multiplicity results

###  4.1. Estimate of the mountain-pass level

The proofs of Theorems 1.3 and 1.5 are based on [7, 10]. To begin with, we use
the Talenti instanton to test the mountain pass level of functional
$I_{\alpha}$. Talenti instanton is defined as

$\displaystyle
U_{x_{0},\mu}(x)=C_{0}\frac{\mu^{\frac{N-2}{2}}}{(1+\mu^{2}|x-x_{0}|^{2})^{\frac{N-2}{2}}}$

with $C_{0}$ is a positive constant depends on $N$ only, cf. [9, pp. 18]. A
minimizing formulation of the mountain pass level is that

$\displaystyle
m_{\alpha,\lambda}=\inf\Big{\\{}J_{\alpha,\lambda}(u)\Big{|}\int_{\Omega}|u|^{2^{*}}dx=1\Big{\\}}$

with

$\displaystyle J_{\alpha,\lambda}(u)=\int_{\Omega}|\nabla
u|^{2}dx-\lambda\mathcal{D}_{\alpha}(u).$

Consider the minimization

$\displaystyle S=\inf\Big{\\{}\int_{\mathbb{R}^{N}}|\nabla
u|^{2}dx\Big{|}\int_{\mathbb{R}^{N}}|u|^{2^{*}}dx=1\Big{\\}}.$

Without loss of generality, let us assume that
$\int_{\mathbb{R}^{N}}|U_{x_{0},\mu}|^{2^{*}}dx=1$ and that $U_{x_{0},\mu}$
attains $S$. Moreover, let us assume that $0\in\Omega$. For a small positive
number $\delta>0$, define a smooth cutoff function

$\xi(x)=\left\\{\begin{aligned} 1,&|x|<\delta,\\\
0,&|x|>2\delta.\end{aligned}\right.$

By a routine computation as in [7] and in [9, pp. 19],

######  Proposition 4.1.

It holds that

  * $(1).$

$\int_{\Omega}|\xi
U_{0,\mu}|^{2^{*}}=\int_{\mathbb{R}^{N}}U_{0,1}^{2^{*}}+O(\frac{1}{\mu^{N}})$;

  * $(2).$

$\int_{\Omega}|\nabla(\xi U_{0,\mu})|^{2}=\int_{\mathbb{R}^{N}}|\nabla
U_{0,1}|^{2}+O(\frac{1}{\mu^{N-2}})$.

Now we will give an estimate on
$C_{N,\alpha}\int_{\Omega}dx\int_{\Omega}\frac{\xi(x)\xi(y)U_{0,\mu}(x)U_{0,\mu}(y)}{|x-y|^{N-\alpha}}dy$.
To this end, we first study its limit, i.e. the integral

(4.1) $\displaystyle
C_{N,\alpha}\int_{\mathbb{R}^{N}}dx\int_{\mathbb{R}^{N}}\frac{U_{0,1}(x)U_{0,1}(y)}{|x-y|^{N-\alpha}}dy.$

######  Lemma 4.2.

Integral (4.1) converges if and only if $N-\alpha-4>0$.

Proof. Notice that

$\displaystyle\mbox{(\ref{inte:limit})}=|I_{\frac{\alpha}{2}}*U_{0,1}|_{2}^{2}=C||\xi|^{-\frac{\alpha}{2}}\widehat{U_{0,1}}(\xi)|_{2}^{2}.$

Now we consider $\widehat{U_{0,1}}(\xi)$, the Fourier transform of $U_{0,1}$.
Do as in [28, pp. 360],

$\displaystyle\widehat{\Big{[}\Big{(}\frac{1}{1+|x|^{2}}\Big{)}^{\frac{N-2}{2}}\Big{]}}(\xi)=\frac{\pi^{\frac{N}{2}}2^{\frac{2}{N}-\frac{N}{2}+2}}{\Gamma\Big{(}\frac{N}{2}-1\Big{)}}|\xi|^{-1}K_{1}(|\xi|).$

Here, $K_{1}$ is the modified Bessel function of the second kind
$K_{\upsilon}$ with $\upsilon=1$. Using [40, (14)/pp. 88], we get

(4.2) $\displaystyle|\xi|^{-\alpha}|\widehat{U_{0,1}}(\xi)|^{2}|\xi|^{N-1}\sim
C|\xi|^{(N-\alpha-4)-1}$

as $|\xi|\to 0$. By [40, (1)/pp. 202], we get

(4.3) $\displaystyle|\xi|^{-\alpha}|\widehat{U_{0,1}}(\xi)|^{2}|\xi|^{N-1}\sim
Ce^{-|\xi|}|\xi|^{N-\alpha-4}$

as $|\xi|\to\infty$. Combining (4.2) and (4.3), we prove the lemma.

$\Box$

We only need to check $\mathcal{D}_{\alpha}(\xi U_{0,\mu})$. By an easy
computation, we get

$\displaystyle\mathcal{D}_{\alpha}(\xi U_{0,\mu})$
$\displaystyle=C_{N,\alpha}\int_{\Omega}dy\int_{\Omega}\frac{\xi(x)\xi(y)U_{0,\mu}(x)U_{0,\mu}(y)}{|x-y|^{N-\alpha}}$
$\displaystyle=C_{N,\alpha}\mu^{-2-\alpha}\int_{\mathbb{R}^{N}\times\mathbb{R}^{N}}\frac{\xi\big{(}\frac{x}{\mu}\big{)}\xi\big{(}\frac{x}{\mu}\big{)}U_{0,1}(x)U_{0,1}(y)}{|x-y|^{N-\alpha}}dxdy.$

Denote

$\displaystyle D_{R}:=C_{N,\alpha}\int_{R_{R}(0)\times
B_{R}(0)}\frac{U_{0,1}(x)U_{0,1}(y)}{|x-y|^{N-\alpha}}dxdy.$

######  Lemma 4.3.

It holds that

  * $(1).$

If $N-\alpha-4>0$, $D_{R}\to C$ for some $C>0$ as $R\to\infty$;

  * $(2).$

If $N-\alpha-4\leq 0$, $D_{R}\to+\infty$ as $R\to\infty$;

  * $(3).$

Given $R>0$, for large $\mu>0$, $\mathcal{D}_{\alpha}(\xi U_{0,\mu})\geq
D_{R}\mu^{-2-\alpha}$.

The next lemma follows immediately.

######  Lemma 4.4.

$m_{\alpha,\lambda}<S$ if one of the following holds:

  * $(1).$

$N-\alpha-4\geq 0$ and any $\lambda>0$;

  * $(2).$

$N-\alpha-4<0$ and $\lambda>0$ large.

Proof. Denote $v_{\mu}=\frac{\xi U_{0,\mu}}{|\xi U_{0,\mu}|_{2^{*}}}$. Then,

$\displaystyle m_{\alpha,\lambda}$ $\displaystyle\leq
I_{\lambda}(v_{\mu})=\frac{\int_{\Omega}|\nabla(\xi
U_{0,\mu})|^{2}-\lambda\mathcal{D}_{\alpha}(\xi U_{0,\mu})}{(\int_{\Omega}|\xi
U_{0,\mu}|^{2^{*}})^{\frac{N-2}{N}}}$
$\displaystyle\leq\frac{\int_{\mathbb{R}^{N}}|\nabla
U_{0,1}|^{2}+O(\mu^{2-N})-\lambda
D_{R}\mu^{-2-\alpha}}{(1+O(\mu^{-N}))^{\frac{N-2}{N}}}=\int_{\mathbb{R}^{N}}|\nabla
U_{0,1}|^{2}+\frac{O(\mu^{2-N})-\lambda D_{R}\mu^{-2-\alpha}}{1+O(\mu^{2-N})}$
$\displaystyle=S+T_{\lambda,\mu,\alpha,N}.$

Here,

$\displaystyle T_{\lambda,\mu,\alpha,N}=\frac{O(\mu^{2-N})-\lambda
D_{R}\mu^{-2-\alpha}}{1+O(\mu^{2-N})}.$

In order to prove $m_{\alpha,\lambda}\leq I_{\lambda}(v_{\mu})<S$, it is
sufficient to check that $T_{\lambda,\mu,\alpha,N}<0$.

Case. 1. $N-\alpha-4>0$.

If $N-\alpha-4>0$, for any $\lambda>0$, letting $\mu>0$ be sufficiently large,
we get $T_{\lambda,\mu,\alpha,N}<0$.

Case. 2. $N-\alpha-4=0$.

If $N-\alpha-4=0$, for any $\lambda>0$, Lemma 4.3 implies that
$D_{R}\to+\infty$ as $R\to\infty$. Then, for large $R>0$ and $\mu>0$, we get
$T_{\lambda,\mu,\alpha,N}<0$ again.

Case. 3. $N-\alpha-4<0$.

If $N-\alpha-4<0$, for arbitrarily fixed $\mu$ and $R$, let $\lambda>0$ be
sufficiently large, we get $T_{\lambda,\mu,\alpha,N}<0$.

$\Box$

###  4.2. A local compactness

In this subsection, we prove a convergence theorem for the $(PS)$ sequence of
the energy functional

$\displaystyle E_{\alpha,\lambda}(u)=\frac{1}{2}\int_{\Omega}|\nabla
u|^{2}-\frac{\lambda}{2}\mathcal{D}_{\alpha}(u)-\frac{1}{2^{*}}\int_{\Omega}|u|^{2^{*}}$

satisfying certain assumptions. To be precise, we get

######  Lemma 4.5.

For any sequence $\\{u_{n}\\}_{n}\in H_{0}^{1}(\Omega)$ such that

  * $(1).$

$E_{\alpha,\lambda}(u_{n})\to c<\frac{1}{N}S^{\frac{N}{2}}$;

  * $(2).$

$\nabla E_{\alpha,\lambda}(u_{n})\to 0$ in $H^{-1}(\Omega)$,

then $\\{u_{k}\\}_{k}$ admits a convergent subsequence in $H_{0}^{1}(\Omega)$.

Proof. We prove it in steps.

Step 1. $\\{u_{n}\\}_{n}$ is bounded in $H^{1}_{0}(\Omega)$.

It is evident that

$\displaystyle o_{n}(1)\|u_{n}\|+c+1\geq\frac{1}{N}|u_{n}|_{2^{*}}^{2^{*}}.$

Then,

$\displaystyle|\nabla u_{n}|_{2}^{2}$ $\displaystyle\leq
C+o_{n}(1)\|u_{n}\|+\frac{|\lambda|}{N}\mathcal{D}_{\alpha}(u_{n})$
$\displaystyle\leq C+o_{n}(1)\|u_{n}\|+(C+o_{n}(1)\|u_{n}\|)^{\frac{N-2}{N}}.$

This implies the boundedness of $\\{u_{n}\\}_{n}$ in $H^{1}_{0}(\Omega)$.

Step 2. $\\{u_{n}\\}_{n}$ converges in $H^{1}_{0}(\Omega)$.

First notice that there exists a $u_{\infty}\in H_{0}^{1}(\Omega)$ such that
$u_{n}\rightharpoonup u_{\infty}$ in $H_{0}^{1}(\Omega)$. We will prove that
$u_{\infty}\to u_{\infty}$ in $H_{0}^{1}(\Omega)$. Then, it is evident that

  * •

$\nabla E_{\alpha,\lambda}(u_{\infty})=0$;

  * •

$\int_{\Omega}|\nabla
u_{n}|^{2}dx=\int_{\Omega}|\nabla(u_{n}-u_{\infty})|^{2}dx+\int_{\Omega}|\nabla
u_{\infty}|^{2}dx+o_{n}(1)$;

  * •

$\mathcal{D}_{\alpha}(u_{n})\to\mathcal{D}_{\alpha}(u_{\infty})$;

  * •

$\int_{\Omega}|u_{n}|^{2^{*}}dx=\int_{\Omega}|u_{n}-u_{\infty}|^{2^{*}}dx+\int_{\Omega}|u_{\infty}|^{2^{*}}dx+o_{n}(1)$.

Here, the last assertion applied Brezis-Lieb theorem. On the other hand,
notice that

$\displaystyle o_{n}(1)$ $\displaystyle=\nabla
E_{\alpha,\lambda}(u_{n})(u_{n}-u_{\infty})$ $\displaystyle=\Big{(}\nabla
E_{\alpha,\lambda}(u_{n})-\nabla
E_{\alpha,\lambda}(u_{\infty})\Big{)}(u_{n}-u_{\infty})$
$\displaystyle=\int_{\Omega}|\nabla(u_{n}-u_{\infty})|^{2}dx-\int_{\Omega}|u_{n}|^{2^{*}-2}u_{n}(u_{n}-u_{\infty})dx+o_{n}(1).$

Here, using Brezis-Lieb theorem,

$\displaystyle\int_{\Omega}|u_{n}|^{2^{*}-2}u_{n}(u_{n}-u_{\infty})dx=\int_{\Omega}|u_{n}|^{2^{*}}dx-\int_{\Omega}|u_{\infty}|^{2^{*}}dx=\int_{\Omega}|u_{n}-u_{\infty}|^{2^{*}}dx+o_{n}(1).$

Then, we get

$\displaystyle\int_{\Omega}|\nabla(u_{n}-u_{\infty})|^{2}dx=\int_{\Omega}|u_{n}-u_{\infty}|^{2^{*}}dx+o_{n}(1).$

It follows that

$\displaystyle
E_{\alpha,0}(u_{n}-u_{\infty})=\frac{1}{N}\int_{\Omega}|\nabla(u_{n}-u_{\infty})|^{2}dx+o_{n}(1).$

Then,

(4.4)
$\displaystyle\int_{\Omega}|\nabla(u_{n}-u_{\infty})|^{2}dx=N(E_{\alpha,\lambda}(u_{n})-E_{\alpha,\lambda}(u_{\infty}))+o_{n}(1)<S^{\frac{N}{2}}.$

On the other hand, notice that

(4.5)
$\displaystyle\|u_{n}-u_{\infty}\|^{2}(1-S^{-\frac{2^{*}}{2}}\|u_{n}-u_{\infty}\|^{2^{*}-2})\leq\int_{\Omega}|\nabla(u_{n}-u_{\infty})|^{2}dx-\int_{\Omega}|u_{n}-u_{\infty}|^{2^{*}}dx=o_{n}(1).$

Combining (4.4) and (4.5), we get $\|u_{n}-u_{\infty}\|\to 0$.

$\Box$

###  4.3. A maximum principle for a nonlocal operator

In this part, we prove the following maximum principle for the nonlocal
operator

$\displaystyle S_{\lambda}(u):=-\Delta u-\lambda I_{\alpha}*_{\Omega}u.$

######  Lemma 4.6.

Suppose $0<\lambda<\lambda_{1}$ and $u\in H_{0}^{1}(\Omega)\backslash\\{0\\}$
satisfies that $S_{\lambda}(u)\geq 0$, then $u\geq 0$ in $\Omega$. Here,

$\displaystyle\lambda_{1}=\inf_{u\neq 0}\frac{\int_{\Omega}|\nabla
u|^{2}dx}{\mathcal{D}_{\alpha}(u)}.$

Proof. According to the assumption, it holds that

(4.6) $\displaystyle-\Delta u-\lambda I_{\alpha}*_{\Omega}u\geq 0$

Testing (4.6) with $u_{-}(x)=\min\\{0,u(x)\\}$, we get

$\displaystyle 0$ $\displaystyle\geq\int_{\Omega}|\nabla u|^{2}dx-\lambda
C_{N,\alpha}\int_{\Omega}\int_{\Omega}\frac{u_{+}(x)u_{-}(y)}{|x-y|^{N-\alpha}}dxdy-\lambda\int_{\Omega}\int_{\Omega}\frac{u_{-}(x)u_{-}(y)}{|x-y|^{N-\alpha}}dxdy$
(4.7) $\displaystyle\geq\int_{\Omega}|\nabla u_{-}|^{2}dx-\lambda
C_{N,\alpha}\int_{\Omega}\int_{\Omega}\frac{u_{-}(x)u_{-}(y)}{|x-y|^{N-\alpha}}dxdy.$

with $u_{+}(x)=\max\\{0,u(x)\\}$. Now we study $\lambda
C_{N,\alpha}\int_{\Omega}\int_{\Omega}\frac{u_{-}(x)u_{-}(y)}{|x-y|^{N-\alpha}}dxdy$.
By Hardy-Littlewood-Sobolev inequality and Sobolev inequality, we get

$\displaystyle\lambda
C_{N,\alpha}\int_{\Omega}\int_{\Omega}\frac{u_{-}(x)u_{-}(y)}{|x-y|^{N-\alpha}}dxdy$
$\displaystyle=\lambda\mathcal{D}_{\alpha}(u)\leq\lambda\cdot\lambda_{1}^{-1}\int_{\Omega}|\nabla
u_{-}|^{2}dx.$

Here,

$\displaystyle\lambda_{1}=\inf_{u\in\mathcal{M}}\int_{\Omega}|\nabla
u|^{2}=\inf_{u\neq 0}\frac{\int|\nabla u|^{2}dx}{\mathcal{D}_{\alpha}(u)}.$

Plugging this into (4.3), we get

$\displaystyle 0\geq|\nabla u_{-}|_{2}^{2}-\lambda\cdot\lambda_{1}^{-1}|\nabla
u_{-}|_{2}^{2}=\Big{(}1-\frac{\lambda}{\lambda_{1}}\Big{)}|\nabla
u_{-}|_{2}^{2}.$

Notice that $0<\lambda<\lambda_{1}$. We get $u_{-}=0$ and complete the proof.

$\Box$

###  4.4. Proof of Theorem 1.3

In this part, we prove Theorem 1.3.

Proof of Theorem 1.3. By Lemma 4.4, for the cases

  * $(1).$

$N-\alpha-4\geq 0$ and any $\lambda>0$;

  * $(2).$

$N-\alpha-4<0$ and $\lambda>0$ large,

there exists a a sequence $\\{u_{n}\\}_{n}\subset H_{0}^{1}(\Omega)$ such that
as $n\to\infty$,

(4.8) $\displaystyle J_{\alpha,\lambda}(u_{n})\to
m_{\alpha,\lambda}=\inf\Big{\\{}\int_{\Omega}|\nabla
u|^{2}dx-\lambda\mathcal{D}_{\alpha}(u)\Big{|}\int_{\Omega}|u|^{2^{*}}dx=1\Big{\\}}<S.$

First, we show that $\\{u_{n}\\}_{n}$ is bounded in $H_{0}^{1}(\Omega)$.
Indeed, by a direct computation,

$\displaystyle m_{\alpha,\lambda}\geq\int_{\Omega}|\nabla
u_{n}|^{2}dx-\lambda\mathcal{D}_{\alpha}(u_{n})\geq\int_{\Omega}|\nabla
u_{n}|^{2}dx-C|u_{n}|_{2^{*}}^{2}\geq\int_{\Omega}|\nabla u_{n}|^{2}dx-C.$

Then, $\\{u_{n}\\}_{n}$ is bounded in $H_{0}^{1}(\Omega)$ and therefore there
exists a $u\in H_{0}^{1}(\Omega)$ such that $u_{n}\rightharpoonup u$ in
$H_{0}^{1}(\Omega)$.

On the other hand, by Brezis-Lieb theorem and the weak convergence of
$\\{u_{n}\\}_{n}$ to $u$, we get

$\displaystyle\int_{\Omega}|u_{n}|^{2^{*}}dx-\int_{\Omega}|u_{n}-u|^{2^{*}}dx=\int_{\Omega}|u|^{2^{*}}dx+o_{n}(1),$
$\displaystyle\int_{\Omega}|\nabla
u_{n}|^{2}dx-\int_{\Omega}|\nabla(u_{n}-u)|^{2}dx=\int_{\Omega}|\nabla
u|^{2}dx+o_{n}(1)$

and

$\displaystyle\mathcal{D}_{\alpha}(u_{m})=\mathcal{D}_{\alpha}(u)+o_{n}(1).$

Then,

$\displaystyle m_{\alpha,\lambda}+o_{n}(1)$
$\displaystyle=\int_{\Omega}|\nabla(u_{n}-u)|^{2}dx+\int_{\Omega}|\nabla
u|^{2}dx-\lambda\mathcal{D}_{\alpha}(u)+o_{n}(1)$ $\displaystyle\geq
S|u_{n}-u|_{2^{*}}^{2}+m_{\alpha,\lambda}|u|_{2^{*}}^{2}+o_{n}(1)$
$\displaystyle\geq
S|u_{n}-u|_{2^{*}}^{2^{*}}+m_{\alpha,\lambda}|u|_{2^{*}}^{2^{*}}+o_{n}(1)$
$\displaystyle\geq(S-m_{\alpha,\lambda})|u_{n}-u|_{2^{*}}^{2^{*}}+m_{\alpha,\lambda}+o_{n}(1).$

Since $m_{\alpha,\lambda}<S$, $u_{n}\to u$ in $L^{2^{*}}(\Omega)$ and
$|u|_{2^{*}}=1$. Therefore,

$\displaystyle m_{\alpha,\lambda}\leq
J_{\alpha,\lambda}(u)\leq\lim_{n\to\infty}J_{\alpha,\lambda}(u_{n})=m_{\alpha,\lambda},$

which proves that $u$ attains the minimum $m_{\alpha,\lambda}$. This implies
that $u$ is a solution to Problem (1.1).

To prove the second part, it is necessary to modify the settings as follows.
Denote

$\displaystyle
m^{+}_{\alpha,\lambda}=\inf\Big{\\{}J_{\alpha,\lambda}(u)\Big{|}\int_{\Omega}|u_{+}|^{2^{*}}dx=1\Big{\\}}.$

Notice that the set $\Big{\\{}u\in
H_{0}^{1}(\Omega)\Big{|}\int_{\Omega}|u_{+}|^{2^{*}}dx=1\Big{\\}}$ is a
$C^{1}$ manifold. Then the above computation can be repeated. To be precise,
we get

######  Claim 4.7.

$m_{\alpha,\lambda}^{+}<S$ if one of the following holds:

  * $(1).$

$N-\alpha-4\geq 0$ and any $\lambda>0$;

  * $(2).$

$N-\alpha-4<0$ and $\lambda>0$ large.

Then, there exists a minimizing sequence $\\{u^{\prime}_{n}\\}_{n}$ of
$m_{\alpha,\lambda}^{+}$. Repeating the argument as above, we find a non-
negative minimizer $u^{\prime}$ of $m_{\alpha,\lambda}^{+}$, which solves the
problem

$\left\\{\begin{array}[]{lr}-{\Delta}u=\lambda
I_{\alpha}*_{\Omega}u+|u_{+}|^{2^{*}-2}u_{+}\mbox{ in }\Omega,\\\ u\in
H_{0}^{1}(\Omega).\end{array}\right.$

Since $0<\lambda<\lambda_{1}$, by Lemma 4.6 and Lemma 3.2, $u^{\prime}$ is a
positive solution to Problem (1.1).

$\Box$

###  4.5. Proof of Theorem 1.5

We follow the notation in the last subsection.

Proof of Theorem 1.5. We apply the idea in [10]. Let
$A_{j}=\mbox{span}\\{\phi_{1},\cdots,\phi_{j}\\}\cap\\{u\in
H_{0}^{1}(\Omega)||u|_{2^{*}}=1\\}$. By a direct computation,

$\displaystyle\sup_{u\in A_{j}}\frac{\int_{\Omega}|\nabla
u|^{2}dx-\lambda\mathcal{D}_{\alpha}(u)}{|u|_{2^{*}}^{2}}\leq(\lambda_{j}-\lambda)\sup_{A_{j}}\frac{\mathcal{D}_{\alpha}(u)}{|u|_{2^{*}}^{2}}<\upsilon\cdot
C_{HLS}C_{N,\alpha}|\Omega|_{N}^{\frac{2+\alpha}{N}}.$

Here, $C_{HLS}$ is the best constant in the Hardy-Littlewood-Sobolev
inequality

$\displaystyle\int_{\mathbb{R}^{N}}dy\int_{\mathbb{R}^{N}}\frac{u(x)v(y)}{|x-y|^{N-\alpha}}dx\leq
C_{HLS}|u|_{\frac{2N}{N+\alpha}}|v|_{\frac{2N}{N+\alpha}}.$

By the assumption in Theorem 1.5,

$\displaystyle\sup_{u\in A_{j}}\frac{\int_{\Omega}|\nabla
u|^{2}dx-\lambda\mathcal{D}_{\alpha}(u)}{|u|_{2^{*}}^{2}}\leq(\lambda_{j}-\lambda)\sup_{A_{j}}\frac{\mathcal{D}_{\alpha}(u)}{|u|_{2^{*}}^{2}}<S.$

Denote

$\displaystyle F_{\alpha,\lambda}(u)=\frac{\int_{\Omega}|\nabla
u|^{2}dx-\lambda\mathcal{D}_{\alpha}(u)}{|u|_{2^{*}}^{2}}.$

Then, $\gamma(\mbox{int}(F_{\alpha,\lambda}^{S})\cap\\{u\in
H_{0}^{1}(\Omega)||u|_{2^{*}}=1\\})\geq j+m$. Denote $\mathcal{L}=\\{u\in
H_{0}^{1}(\Omega)|\int_{\Omega}|u|^{2^{*}}dx=1\\}$,
$L:=\overline{\mbox{span}\\{\phi_{j},\phi_{j+1},\cdots\\}}\cap\mathcal{L}$ and
$\Gamma_{k}=\\{A\subset\mathcal{L}|\gamma(A)\geq j+k\\}$ for $k=1,\cdots,m$.
Here, $\gamma$ is the $\mathbb{Z}_{2}$ genus generated by the anti-podal
action $u\mapsto-u$. Define the minimax values

(4.9) $\displaystyle c_{k}=\inf_{A\in\Gamma_{k}}\sup_{x\in
A}J_{\alpha,\lambda}(u).$

Notice that for any $k=1,\cdots,m$ and any $A\in\Gamma_{k}$, $A\cap
L\neq\emptyset$. By the intersection lemma [37, Proposition 7.8], we get

$\displaystyle c_{k}=\inf_{A\in\Gamma_{k}}\sup_{u\in
A}J_{\alpha,\lambda}(u)\geq\inf_{u\in
L}J_{\alpha,\lambda}(u)\geq(\lambda_{j}-\lambda)\int_{\Omega}|\nabla
u|^{2}dx\Big{|}_{\mathcal{L}}\geq c_{*}>0.$

By [2] and [37, Theorem 8.10], $c_{k}$ are critical values of
$J_{\alpha,\lambda}$ for $k=1,\cdots,m$. For any $k=1,\cdots,m$, there exists
a $(PS)$ sequence $\\{u_{n}^{k}\\}_{n}$ of
$J_{\alpha,\lambda}(u)|_{\mathcal{L}}$ such that
$J_{\alpha,\lambda}(u_{n}^{k})\to c_{k}$ as $n\to\infty$. Then, the sequence
$\\{\widetilde{u}_{n}^{k}\\}_{n}$ is a $(PS)$ sequence of $E_{\alpha,\lambda}$
such that
$\lim_{n\to\infty}E_{\alpha,\lambda}(\widetilde{u}_{n}^{k})<\frac{1}{N}S^{\frac{N}{2}}$.
By Lemma 4.5, $\\{\widetilde{u}_{n}^{k}\\}_{n}$ converges to a critical point
of $E_{\alpha,\lambda}$ and Theorem 1.5 follows immediately.

$\Box$

Acknowledgment.  Both of the authors would like to express their sincere
appreciation to the anonymous reviewers for their helpful suggestions on
improving this paper.

Fundings.  In this research, HL was supported by FAPESP PROC 2022/15812-0.

Competing Interests.  The authors have no relevant financial or non-financial
competing interests.

Data availability statement.  No datasets were generated or analysed during
the current study.

## References

  * [1] Alves, C. O, Cassani, D, Tarsi, C, Yang, M, Existence and concentration of ground state solutions for a critical nonlocal Schrödinger equation in $\mathbb{R}^{2}$. J. Differ. Equations 261, No. 3, 1933-1972 (2016).
  * [2] Ambrosetti, A, Rabinowitz, P. H, Dual variational methods in critical point theory and applications. J. Funct. Anal. 14, 349-381 (1973).
  * [3] Bahrami, M, Gro$\beta$ardt, A, Donadi, S, Bassi, A, The Schrödinger-Newton equation and its foundations. New J. Phys. 16, No. 11, Article ID 115007, 17 p. (2014).
  * [4] Bellazzini, J, Frank, R. L, Visciglia, N, Maximizers for Gagliardo-Nirenberg inequalities and related non-local problems. Math. Ann. 360, No. 3-4, 653-673 (2014).
  * [5] Benci, V, Fortunato, D, An eigenvalue problem for the Schrödinger-Maxwell equations. Topol. Methods Nonlinear Anal. 11, No. 2, 283-293 (1998).
  * [6] Bokanowski, O, López, José L, Soler, J, On an exchange interaction model for quantum transport: the Schrödinger-Poisson-Slater system. Math. Models Methods Appl. Sci. 13, No. 10, 1397-1412 (2003).
  * [7] Brézis, H, Nirenberg, L, Positive solutions of nonlinear elliptic equations involving critical Sobolev exponents. Commun. Pure Appl. Math. 36, 437-477 (1983).
  * [8] Brézis, H, Strauss, W. A, Semi-linear second-order elliptic equations in $L^{1}$. J. Math. Soc. Japan 25, 565-590 (1973).
  * [9] Cao, D, Peng, S, Yan, S, Singularly perturbed methods for nonlinear elliptic problems. Cambridge Studies in Advanced Mathematics 191. Cambridge: Cambridge University Press. ix, 252 p. (2021).
  * [10] Cerami, G, Fortunato, D, Struwe, M, Bifurcation and multiplicity results for nonlinear elliptic problems involving critical Sobolev exponents. Ann. Inst. Henri Poincaré, Anal. Non Linéaire 1, 341-350 (1984).
  * [11] Cerami, G, Molle, R, Multiple positive bound states for critical Schrödinger-Poisson systems. ESAIM, Control Optim. Calc. Var. 25, Paper No. 73, 29 p. (2019).
  * [12] Chen, W, Jin, C, Li, C, Lim, J, Weighted Hardy-Littlewood-Sobolev inequalities and systems of integral equations. (English) Zbl 1147.45301 Discrete Contin. Dyn. Syst. 2005, Suppl., 164-172 (2005).
  * [13] Chen, W, Li, C, The best constant in a weighted Hardy-Littlewood-Sobolev inequality. Proc. Am. Math. Soc. 136, No. 3, 955-962 (2008).
  * [14] Chen, W, Li, C, Ou, B, Classification of solutions for a system of integral equations. Commun. Partial Differ. Equations 30, No. 1, 59-65 (2005).
  * [15] Dou, J, Zhu, M, Reversed Hardy-Littewood-Sobolev inequality. Int. Math. Res. Not. 2015, No. 19, 9696-9726 (2015).
  * [16] Dou, J, Zhu, M, Sharp Hardy-Littlewood-Sobolev inequality on the upper half space. Int. Math. Res. Not. 2015, No. 3, 651-687 (2015).
  * [17] Evans, L. C, Partial differential equations. 2nd ed. Graduate Studies in Mathematics 19. Providence, RI: American Mathematical Society. xxi, 749 p. (2010).
  * [18] Gilbarg, D, Trudinger, N. S, Elliptic partial differential equations of second order. Reprint of the 1998 ed. Classics in Mathematics. Berlin: Springer. xiii, 517 p. (2001).
  * [19] Guo, Y, Peng, S, Asymptotic behavior and classification of solutions to Hartree type equations with exponential nonlinearity. J. Geom. Anal. 34, No. 1, Paper No. 23, 21 p. (2024).
  * [20] Han, Z.-C, Li, Y, On the local solvability of the Nirenberg problem on $\mathbb{S}^{2}$. Discrete Contin. Dyn. Syst. 28, No. 2, 607-615 (2010).
  * [21] Hardy, G. H, Littlewood, J. E, Some properties of fractional integrals. I. M. Z. 27, 565-606 (1928).
  * [22] Hebey, E, Wei, J, Schrödinger-Poisson systems in the 3-sphere. Calc. Var. Partial Differ. Equ. 47, No. 1-2, 25-54 (2013).
  * [23] Heinz, E, Über die Eindeutigkeit beim Cauchyschen Anfangswertproblem einer elliptischen Differentialgleichung zweiter Ordnung. Nachr. Akad. Wiss. G?ttingen, Math.-Phys. Kl., Math.-Phys.-Chem. Abt. 1955, 1-12 (1955).
  * [24] Ianni, I, Ruiz, D, Ground and bound states for a static Schrödinger-Poisson-Slater problem. Commun. Contemp. Math. 14, No. 1, 1250003, 22 p. (2012).
  * [25] Lei, Y, Li, C, Ma, C, Asymptotic radial symmetry and growth estimates of positive solutions to weighted Hardy-Littlewood-Sobolev system of integral equations. Calc. Var. Partial Differ. Equ. 45, No. 1-2, 43-61 (2012).
  * [26] Li, Y. Y, On $-\Delta u=K(x)u^{5}$ in $\mathbb{R}^{3}$. Commun. Pure Appl. Math. 46, No. 3, 303-340 (1993).
  * [27] Li, Y, Shafrir, I, Blow-up analysis for solutions of $-\Delta u=Ve^{u}$ in dimension two. Indiana Univ. Math. J. 43, No. 4, 1255-1270 (1994).
  * [28] Lieb, E. H, Sharp constants in the Hardy-Littlewood-Sobolev and related inequalities. Ann. Math. (2) 118, 349-374 (1983).
  * [29] Liu, X, Ma, S, Xia, J, Multiple bound states of higher topological type for semi-classical Choquard equations. Proc. R. Soc. Edinb., Sect. A, Math. 151, No. 1, 329-355 (2021).
  * [30] Ma, L, Zhao, L, Classification of positive solitary solutions of the nonlinear Choquard equation. Arch. Ration. Mech. Anal. 195, No. 2, 455-467 (2010).
  * [31] Molica, B. G, Radulescu, V. D, Servadei, R, Variational methods for nonlocal fractional problems. Encyclopedia of Mathematics and its Applications 162. Cambridge: Cambridge University Press. xvi, 383 p. (2016).
  * [32] Moroz, V, Van Schaftingen, J, Groundstates of nonlinear Choquard equations: existence, qualitative properties and decay asymptotics. J. Funct. Anal. 265, No. 2, 153-184 (2013).
  * [33] Moroz, V, Van Schaftingen, J, Existence of groundstates for a class of nonlinear Choquard equations. Trans. Am. Math. Soc. 367, No. 9, 6557-6579 (2015).
  * [34] Moroz, V, Van Schaftingen, J, A guide to the Choquard equation. J. Fixed Point Theory Appl. 19, No. 1, 773-813 (2017).
  * [35] Penrose, R, On gravity’s role in quantum state reduction. Gen. Relativ. Gravitation 28, No. 5, 581-600 (1996).
  * [36] Perera, K, Agarwal, R. P, O’Regan, D, Morse theoretic aspects of $p$-Laplacian type operators. Mathematical Surveys and Monographs 161. Providence, RI: American Mathematical Society. xx, 141 p. (2010).
  * [37] Rabinowitz, P. H, Minimax methods in critical point theory with applications to differential equations. Regional Conference Series in Mathematics 65. Providence, RI: American Mathematical Society. viii, 100 p. (1986).
  * [38] Vaira, G, Ground states for Schrödinger-Poisson type systems. Ric. Mat. 60, No. 2, 263-297 (2011).
  * [39] Wang, Z.-Q, Xia, J, Saddle solutions for the Choquard equation. II. Nonlinear Anal., Theory Methods Appl., Ser. A, Theory Methods 201, Article ID 112053, 25 p. (2020).
  * [40] Watson, G. N, A treatise on the theory of Bessel functions. 2nd ed. Cambridge Mathematical Library. Cambridge: Cambridge Univ. Press. vi, 804 p. (1995).
  * [41] Wei, J, Winter, M, Strongly interacting bumps for the Schrödinger-Newton equations. J. Math. Phys. 50, No. 1, 012905, 22 p. (2009).
  * [42] Willem, M, Minimax theorems. Progress in Nonlinear Differential Equations and their Applications. 24. Boston: Birkhäuser. viii, 159 p. (1996).
  * [43] Xia, J, Wang, Z.-Q, Saddle solutions for the Choquard equation. Calc. Var. Partial Differ. Equ. 58, No. 3, Paper No. 85, 30 p. (2019).