# Solitary waves for the nonlinear Schrödinger-Poisson system with positron-
electron interaction

Sangdon Jin  Department of mathematics, Chung-Ang University, Seoul 06974,
Korea<EMAIL_ADDRESS>and  Jinmyoung Seok  Department of Mathematics, Kyonggi
University, Suwon 16227, Republic of Korea<EMAIL_ADDRESS>
###### Abstract.

In this paper, we study the existence of positive solutions to the nonlinear
elliptic system

$\begin{cases}-\Delta
u_{1}+u_{1}+(\mu_{11}\phi_{u_{1}}-\mu_{12}\phi_{u_{2}})u_{1}=\frac{1}{2\pi}\int_{0}^{2\pi}|u_{1}+e^{i\theta}u_{2}|^{p-1}(u_{1}+e^{i\theta}u_{2})d\theta\
\ \mbox{ in }\ \ \mathbb{R}^{3},\\\ -\Delta
u_{2}+u_{2}+(\mu_{22}\phi_{u_{2}}-\mu_{12}\phi_{u_{1}})u_{2}=\frac{1}{2\pi}\int_{0}^{2\pi}|u_{2}+e^{i\theta}u_{1}|^{p-1}(u_{2}+e^{i\theta}u_{1})d\theta\
\ \mbox{ in }\ \ \mathbb{R}^{3},\end{cases}$

which is derived from taking the nonrelativistic limit of the nonlinear
Maxwell-Klein-Gordon equations under the decomposition of waves functions into
positron and electron parts. We characterize the existence and nonexistence of
positive vector solutions, depending on parameters $p$ and $\mu_{ij}$.

###### Key words and phrases:

semilinear elliptic system; Maxwell-Klein-Gordon; Schrödinger-Poisson;
variational method

###### 2010 Mathematics Subject Classification:

35J50, 35J61, 35Q40

##  1\.  Introduction

This paper is devoted to classify the structure of standing waves for the
following system of Schrödinger-Poisson type,

$\left\\{\begin{aligned} &2i\dot{v}_{+}-\Delta
v_{+}+(\mu_{11}\phi_{v_{+}}-\mu_{12}\phi_{v_{-}})v_{+}-\frac{1}{2\pi}\int_{0}^{2\pi}g(v_{+}+e^{i\theta}\overline{v}_{-})d\theta=0,\\\
&2i\dot{v}_{-}-\Delta
v_{-}+(\mu_{22}\phi_{v_{-}}-\mu_{12}\phi_{v_{+}})v_{-}-\frac{1}{2\pi}\int_{0}^{2\pi}g(v_{-}+e^{i\theta}\overline{v}_{+})d\theta=0,\end{aligned}\right.$
(1)

where $v_{\pm}(t,x):\mathbb{R}^{1+3}\rightarrow\mathbb{C}$,
$\phi_{u}=(4\pi|\cdot|)^{-1}\ast|u|^{2}$, $\mu_{ij}>0$ for $i,j=1,2$ and
$g(v):\mathbb{C}\rightarrow\mathbb{C}$ satisfying $g(e^{is}v)=e^{is}g(v)$ for
all $s\in\mathbb{R}$. This system arises as a limit system of the nonlinear
Maxwell-Klein-Gordon equations when we consider the nonrelativistic limit,
taking the speed of light to infinity. The nonlinear Maxwell-Klein-Gordon
equations are written as follows:

$\left\\{\begin{aligned} &D_{\alpha}D^{\alpha}\psi=(mc)^{2}\psi-g(\psi),\\\
&\partial^{\beta}F_{\alpha\beta}=\frac{1}{c}\text{Im}(\psi\overline{D_{\alpha}\psi}),\end{aligned}\right.$
(NMKG)

where $\psi(t,x):\mathbb{R}^{1+3}\rightarrow\mathbb{C}$ is a wave function,
$g:\mathbb{C}\rightarrow\mathbb{C}$ is a nonlinear potential satisfying
$g(e^{is}v)=e^{is}g(v)$ for all $s\in\mathbb{R}$,
$D_{\alpha}=\partial_{\alpha}+\frac{i}{c}A_{\alpha},\alpha=0,1,2,3$ is the
covariant derivative and
$F_{\alpha\beta}=\partial_{\alpha}A_{\beta}-\partial_{\beta}A_{\alpha}$ is the
curvature tensor. Here, we write $\partial_{0}=\frac{\partial}{c\partial t}$,
$\partial_{j}=\frac{\partial}{\partial x_{j}},j=1,2,3$. Indices are raised
under the Minkowski metric $g_{\alpha\beta}=\text{diag}(-1,1,1,1)$, i.e.,
$X^{\alpha}=g_{\alpha\beta}X_{\beta}$. The positron part $v_{+}$ and electron
part $v_{-}$ of a solution $\psi$ to (NMKG) are given by

$v_{+}\coloneqq
e^{-ic^{2}t}\frac{1}{2}\left(\psi-i\langle\nabla\rangle_{c}^{-1}D^{+}_{0}\psi\right),\quad
v_{-}\coloneqq
e^{-ic^{2}t}\frac{1}{2}\left(\bar{\psi}-i\langle\nabla\rangle_{c}^{-1}D^{-}_{0}\bar{\psi}\right),$

where $\langle\nabla\rangle_{c}$ denotes the Fourier multiplier with the
symbol $\sqrt{|\cdot|^{2}+c^{2}}$ and
$D_{\alpha}^{\pm}=\partial_{\alpha}\pm\frac{i}{c}A_{\alpha}$. Some formal
computation (for example, see [21]) shows that by taking the nonrelativistic
limit $c\to\infty$, $(v_{+},v_{-})$ approaches to a solution of the system (1)
with $\mu_{ij}=2$. In other words, a solution of (NMKG) can be approximated as
the two different modes of oscillations, namely
$\psi=e^{ic^{2}t}v_{+}+e^{-ic^{2}t}\overline{v_{-}}+o_{c}(1)$ as
$c\rightarrow\infty$, where $(v_{+},v_{-})$ is a solution to the system (1).
The integration
$\frac{1}{2\pi}\int_{0}^{2\pi}g(v_{\pm}+e^{i\theta}\overline{v}_{\mp})d\theta$
in (1) describes the resonances and if we take the cubic nonlinearity
$g(v)=|v|^{2}v$, then it simply becomes
$(|v_{\pm}|^{2}+2|v_{\mp}|^{2})v_{\pm}$.

In [21, 22], Masmoudi-Nakanishi rigorously justified the convergence of
nonrelativistic limit in the space $C[0,T],H^{1})$ either the Maxwell gauge
terms are not involved, that is, $A_{\alpha}=0$ for $\alpha=0,1,2,3$ or the
nonlinear potential term $g$ is absent. In these cases, the system (1) reduces
to respectively the systems

$2i\dot{v}_{\pm}-\Delta
v_{\pm}-\frac{1}{2\pi}\int_{0}^{2\pi}g(v_{\pm}+e^{i\theta}\overline{v_{\mp}})d\theta=0$
(2)

or

$\left\\{\begin{aligned} &2i\dot{v}_{+}-\Delta
v_{+}+(\mu_{11}\phi_{v_{+}}-\mu_{12}\phi_{v_{-}})v_{+}=0,\\\
&2i\dot{v}_{-}-\Delta
v_{-}+(\mu_{22}\phi_{v_{-}}-\mu_{12}\phi_{v_{+}})v_{-}=0.\end{aligned}\right.$
(3)

We note that the system (1) serves as a physically meaningful generalization
of several well-studied semilinear PDEs. For example, when either $v_{+}$ or
$v_{-}$ vanishes, the system (1) is reduced to the so-called Schrödinger-
Poisson equation

$2i\dot{v}-\Delta v+\mu\phi_{v}v-g(v)=0,$

which is extensively studied during past two decades. We refer to the
literatures [7, 8, 9, 10, 23, 29] for the existence and qualitative properties
of positive standing waves. The system (2) with $g(u)=|u|^{2}u$ is called the
coupled system of cubic nonlinear Schrödinger equations and appeared in theory
of Bose–Einstein condensates and nonlinear optics (see [1, 2, 5, 20, 25, 26,
27, 28, 30, 32, 35] and references therein). Moreover, if we take
$v_{+}=v_{-}$ and $\mu_{11}=\mu_{22}$ in the system (3), we obtain the
nonlinear Hartree equation

$2i\dot{v}-\Delta v+\mu\phi_{v}v=0,$

which arises in the self gravitational collapse of quantum mechanical system
and physics of laser beams. We refer to [16, 18, 19, 24] and references
therein for the study of ground states and standing waves. In relating with
the system (3), the existence of positive standing waves in [34] is studied
when the parameters $\mu_{11}$ and $\mu_{22}$ are negative.

The main novelty of this paper therefore is to investigate the existence of
standing waves having nontrivial components $v_{+},\,v_{-}$ to the full system
(1) with positive coupling parameters $\mu_{ij}>0$. We will see that the
nontrivial standing waves of the aforementioned reduced systems play a
significant role in analyzing structures of standing waves to (1).

Now, we insert the standing wave ansatz

$v_{+}(x,t)=u_{1}(x)e^{i\frac{\lambda}{2}t}\mbox{ and
}v_{-}(x,t)=u_{2}(x)e^{i\frac{\lambda}{2}t}$

into the system (1) to obtain

$\begin{cases}-\Delta u_{1}+\lambda
u_{1}+(\mu_{11}\phi_{u_{1}}-\mu_{12}\phi_{u_{2}})u_{1}=\frac{1}{2\pi}\int_{0}^{2\pi}g(u_{1}+e^{i\theta}u_{2})d\theta\
\ \mbox{ in }\ \ \mathbb{R}^{3},\\\ -\Delta u_{2}+\lambda
u_{2}+(\mu_{22}\phi_{u_{2}}-\mu_{12}\phi_{u_{1}})u_{2}=\frac{1}{2\pi}\int_{0}^{2\pi}g(u_{2}+e^{i\theta}u_{1})d\theta\
\ \mbox{ in }\ \ \mathbb{R}^{3},\end{cases}$ (4)

where $u_{1},u_{2}:\mathbb{R}^{3}\rightarrow\mathbb{R}$, $\lambda,\mu_{ij}>0$
for $i,j=1,2$. We denote by $\vec{u}$ a pair of functions $(u_{1},\,u_{2})$.
Here we define some concepts of triviality and positiveness of a vector
function $\vec{u}$.

######  Definition.

  1. A vector function $\vec{u}=(u_{1},u_{2})$ is said to be

  2. $*$

nontrivial if either $u_{1}\neq 0$ or $u_{2}\neq 0$;

  3. $*$

semi-trivial if it is nontrivial but either $u_{1}=0$ or $u_{2}=0$;

  4. $*$

vectorial if both of $u_{1}$ and $u_{2}$ are not zero;

  5. $*$

nonnegative if $u_{1}\geq 0$ and $u_{2}\geq 0$;

  6. $*$

positive if $u_{1}>0$ and $u_{2}>0$.

We denote by $H^{1}=H^{1}(\mathbb{R}^{3})$ the completion of
$C^{\infty}_{c}(\mathbb{R}^{3})$, the space of real valued smooth function
with compact support, with respect to the standard Sobolev norm

$\|u\|_{\lambda}=\Big{(}\int_{\mathbb{R}^{3}}|\nabla u|^{2}+\lambda
u^{2}dx\Big{)}^{\frac{1}{2}}.$

The notation $H_{r}^{1}$ denotes the space of radially symmetric functions in
$H^{1}$. We also define ${\bf H}\coloneqq H_{r}^{1}\times H_{r}^{1}$ equipped
with the norm

$\|\vec{u}\|_{\bf{H}}^{2}=\|u_{1}\|_{\lambda}^{2}+\|u_{2}\|_{\lambda}^{2}.$

We now are ready to state the main results. Our first result deals with the
case $g=0$, in which the system (4) is reduced to

$\begin{cases}-\Delta u_{1}+\lambda
u_{1}+(\mu_{11}\phi_{u_{1}}-\mu_{12}\phi_{u_{2}})u_{1}=0\ \ \mbox{ in }\ \
\mathbb{R}^{3},\\\ -\Delta u_{2}+\lambda
u_{2}+(\mu_{22}\phi_{u_{2}}-\mu_{12}\phi_{u_{1}})u_{2}=0\ \ \mbox{ in }\ \
\mathbb{R}^{3}.\end{cases}$ (5)

In this case we can provide a satisfactory picture for the solution structure
of (5). Interestingly, it turns out that the sign of determinant of the matrix
$(\mu_{ij})$ completely determines the existence of nontrivial solutions to
(5).

######  Theorem 1.1.

Let $\lambda,\mu_{ij}>0$ for $i,j=1,2$. If $\mu_{11}\mu_{22}\geq\mu_{12}^{2}$,
then (5) has only the trivial solution in $H^{1}(\mathbb{R}^{3})\times
H^{1}(\mathbb{R}^{3})$. If $\mu_{11}\mu_{22}<\mu_{12}^{2}$, The system (5)
admits a positive vector solution $\vec{u}\in\mathbf{H}$, which is unique in
the class of nontrivial nonnegative functions in H. Actually, it is determined
as $\vec{u}=(V,a_{0}V)$, where
$a_{0}=\sqrt{\frac{\mu_{11}+\mu_{12}}{\mu_{22}+\mu_{12}}}$ and $V$ is a unique
positive radial solution of the equation

$-\Delta u+\lambda
u+\frac{\mu_{11}\mu_{22}-\mu_{12}^{2}}{\mu_{22}+\mu_{12}}\phi_{u}u=0.$ (6)

######  Remark 1.

It seems interesting to compare the system (5) with the corresponding single
equation, so-called nonlinear Hartree equation

$-\Delta u+\lambda u+\mu\phi_{u}u=0.$

It is proved in [16, 19] that it admit a unique positive radial solution when
$\mu<0$. (Note that this makes the statement of Theorem 1.1 valid.) By
multiplying by $u$ and integrating, it is also easy to see it has only trivial
solution if $\mu\geq 0$. Here, we point out that the sign of
$\text{det}(\mu_{ij})$ of the system (5) plays the same role with the sign of
$\mu$.

######  Remark 2.

It is immediate to show that the system (5) does not have any semi-trivial
solution. Suppose, for example, $u_{2}=0$ for a solution $\vec{u}$ so that (5)
reduces to $-\Delta u_{1}+\lambda u_{1}+\mu_{11}\phi_{u_{1}}u_{1}=0$. Then as
mentioned in the previous remark, we see $u_{1}=0$.

We next turn our attention to the case $g\neq 0$. In this paper, we only focus
on the standard power function $g(v)=|v|^{p-1}v$. Then we can write the system
(4) as

$\begin{cases}-\Delta u_{1}+\lambda
u_{1}+(\mu_{11}\phi_{u_{1}}-\mu_{12}\phi_{u_{2}})u_{1}=\frac{1}{2\pi}\int_{0}^{2\pi}|u_{1}+e^{i\theta}u_{2}|^{p-1}(u_{1}+e^{i\theta}u_{2})d\theta\
\ \mbox{ in }\ \ \mathbb{R}^{3},\\\ -\Delta u_{2}+\lambda
u_{2}+(\mu_{22}\phi_{u_{2}}-\mu_{12}\phi_{u_{1}})u_{2}=\frac{1}{2\pi}\int_{0}^{2\pi}|u_{2}+e^{i\theta}u_{1}|^{p-1}(u_{2}+e^{i\theta}u_{1})d\theta\
\ \mbox{ in }\ \ \mathbb{R}^{3}.\end{cases}$ (7)

In this case, one can compare the system (7) with

$-\Delta u+\lambda u+\mu\phi_{u}u=|u|^{p-1}u\mbox{ in }\mathbb{R}^{3},$ (8)

which is called the nonlinear Schrödinger-Poisson equation. It is proved in
[10] that when $p\in(0,1]\cup[5,\infty)$, there exists no nontrivial solution
to (8). In the interesting paper [29], Ruiz investigated the existence of
positive radial solutions to (8) with the subcritical range of $p\in(1,5)$. He
shows that while a positive radial solution exists when $p\in(2,5)$ for each
$\lambda,\mu>0$, the situation is changed when we consider the case
$p\in(1,2]$, where there exists no nontrivial solution for large $\mu>0$ and
there exist two positive radial solutions for small $\mu>0$.

It is therefore naturally expected that the system (7) admits only trivial
solution if the range of the exponent $p>0$ is either sub-linear or $H^{1}$
super-critical. The second result of this paper confirms this whenever
$\det{(\mu_{ij})}\geq 0$.

######  Theorem 1.2.

Let $\lambda,\mu_{ij}>0$ for $i,j=1,2$ satisfying
$\mu_{11}\mu_{22}\geq\mu_{12}^{2}$. If $p\in(0,1]\cup[5,\infty)$, then (7) has
only the trivial solution in $H^{1}(\mathbb{R}^{3})\times
H^{1}(\mathbb{R}^{3})$.

######  Remark 3.

The condition $\mu_{11}\mu_{22}\geq\mu_{12}^{2}$ is not a technical matter.
Indeed, we note that, when $p=1$, (7) becomes the following systems

$\begin{cases}-\Delta
u_{1}+(\lambda-1)u_{1}+(\mu_{11}\phi_{u_{1}}-\mu_{12}\phi_{u_{2}})u_{1}=0\ \
\mbox{ in }\ \ \mathbb{R}^{3},\\\ -\Delta
u_{2}+(\lambda-1)u_{2}+(\mu_{22}\phi_{u_{2}}-\mu_{12}\phi_{u_{1}})u_{2}=0\ \
\mbox{ in }\ \ \mathbb{R}^{3}.\end{cases}$ (9)

Then, by Theorem 1.1, we deduce that if $\lambda>1$ and $\mu_{ij}>0$ with
$\mu_{11}\mu_{22}<\mu_{12}^{2}$, then (9) has a positive solution.

In the subsequent theorems, we investigate the structure of ground states to
(7) in the super-linear and sub-critical range of $p\in(1,5)$ to search for
positive vector solutions. Observe that the system (7) is the Euler-Lagrange
system of the action functional

$\displaystyle I(\vec{u})$
$\displaystyle=\frac{1}{2}\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}+|\nabla
u_{2}|^{2}+\lambda u_{1}^{2}+\lambda
u_{2}^{2}dx+\frac{1}{4}\int_{\mathbb{R}^{3}}\mu_{11}u_{1}^{2}\phi_{u_{1}}+\mu_{22}u_{2}^{2}\phi_{u_{2}}-2\mu_{12}u_{1}^{2}\phi_{u_{2}}dx$
$\displaystyle\qquad-\frac{1}{p+1}\frac{1}{2\pi}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}|u_{1}+e^{i\theta}u_{2}|^{p+1}d\theta
dx.$

We say that a solution $\vec{u}\in\mathbf{H}$ to (7) is a (radial) ground
state if $\vec{u}$ is nontrivial and minimizes the value of $I$ among all
nontrivial solutions to (7) in $\mathbf{H}$.

Similarly with the results on the nonlinear Schrödinger-Poisson equations, it
turns out that the solution structure of (7) changes at the borderline $p=2$.
The next theorem is concerned with the range of $p\in(2,5)$.

######  Theorem 1.3.

Let $2<p<5$ and $\lambda,\mu_{ij}>0$ for $i,j=1,2$. Then there exists a non-
negative ground state solution $\vec{w}$ of (7). If either $2<p<5$ and
$\mu_{11}=\mu_{22}$; or $\frac{1}{3}(-2+\sqrt{73})\leq p<5$, then $\vec{w}$ is
positive. Moreover, if $2<p<5$ and $\mu_{11}=\mu_{22}$, then $\vec{w}$ must be
of the form $\vec{w}=(U,U)$, where $U$ is a positive ground state solution of

$-\Delta u+\lambda
u+({\mu_{11}-\mu_{12}})\phi_{u}u=\frac{c_{p}}{2}|u|^{p-1}u\mbox{ in
}\mathbb{R}^{3}$ (10)

with
$c_{p}=\frac{2^{\frac{p-1}{2}}}{\pi}\int_{0}^{2\pi}(1+\cos\theta)^{\frac{p+1}{2}}d\theta$.

######  Remark 4.

We note that the number $\frac{1}{3}(-2+\sqrt{73})\cong 2.18133$ is slightly
larger than 2. We believe that this number appears in just a technical matter
and any ground state is actually positive in the whole range of $2<p<5$
without the assumption $\mu_{11}=\mu_{22}$.

Now, we turn to the remaining case $p\in(1,2]$. It turns out that the solution
structure varies when we change the sign of $\det{(\mu_{ij})}$. We first
construct a positive solution of (7) when $\mu_{12}>0$ is taken to be large,
i.e., $\det(\mu_{ij})<0$ and $|\det(\mu_{ij})|$ is large. The solution is
obtained as a perturbation of $(V,V)$, where $V$ is the solution of a
nonlinear Hartree equation

$-\Delta V+\lambda V-\phi_{V}V=0\mbox{ in }\mathbb{R}^{3}.$

######  Theorem 1.4.

Let $1<p\leq 2$ and $\lambda,\mu_{11},\,\mu_{22}>0$ be fixed. Then there
exists a constant $\mu_{0}>0$ such that if $\mu_{12}>\mu_{0}$, the system (7)
admits a positive solution.

The next two theorems cover the case $\det{(\mu_{ij})}>0$. We will see that,
unlike the case $\det{(\mu_{ij})<0}$, the system (7) does not admit any
nontrivial solution when $\det(\mu_{ij})$ is taken to be large.

######  Theorem 1.5.

Let $1<p\leq 2$ and $\mu_{ij}>0$ for $i,j=1,2$. Assume that $\lambda\geq 2$,
$\mu_{11}>4$ and $(\mu_{11}-4)(\mu_{22}-4)>\mu_{12}^{2}.$ Then (7) has only
the trivial solution in $H^{1}(\mathbb{R}^{3})\times H^{1}(\mathbb{R}^{3})$.

Our final result deals with the case that $\det{(\mu_{j})}>0$ and $\mu_{ij}$
is small. We construct two positive solutions to (7) such that one has
positive energy and the other one is an energy minimizer which has negative
energy. This corresponds to the work by Ruiz [29] in which the same kind of
positive solutions for (10) are found.

######  Theorem 1.6.

Let $1<p<2$ and $\lambda,\mu_{ij}>0$ for $i,j=1,2$. If
$\mu_{11}\mu_{22}-\mu_{12}^{2}>0$ and $\mu_{ij}$ is small enough, where
$i,j=1,2$, then there are at least two different positive radial solutions
$\vec{u}_{1}$, $\vec{u}_{2}$ of (7), where $\vec{u}_{1}$ is a positive
minimizer of (7) with negative energy, and $\vec{u}_{2}$ is a positive
solution of (7) with positive energy. Moreover, if we assume
$\mu_{11}\mu_{22}-\mu_{12}^{2}>0$, $\mu_{11}=\mu_{22}$ and $\mu_{ij}>0$ is
small enough, where $i,j=1,2$, then $\vec{u}_{1}$ and $\vec{u}_{2}$ have the
form $\vec{u}_{1}=(U_{1},U_{1})$ and $\vec{u}_{2}=(U_{2},U_{2})$, where
$U_{1}$ is a positive minimizer of (10) with negative energy, and $U_{2}$ is a
positive solution of (10) with positive energy.

######  Remark 5.

Theorem 1.4 and 1.6 show that the value of $\det(\mu_{ij})$ affects the
solution structure of (7) in the case $p\in(1,2]$. If $\det(\mu_{ij})$ is
largely negative, then there exists a positive solution bifurcating from the
ground state of the nonlinear Hartree equation. If $\det(\mu_{ij})$ is
positive and the coefficient are small, then there exists two positive
solution bifurcating from two positive solution of the nonlinear Schrödinger-
Poisson equations.

The rest of paper is organized as follows. In Section 2, we study the system
(4) with $g=0$ and prove Theorem 1.1. Section 3 is devoted to provide
classification results of positive solutions to (7), and some properties of
the semi-trivial solutions to (7) such as the energy levels or Morse indices
of them. In Section 4, we study the existence of positive solutions to (7) for
$2<p<5$, and prove Theorem 1.3. In Section 5, we construct positive solutions
to (7) for $1<p\leq 2$ and $\det(\mu_{ij})<0$, and prove Theorem 1.4. In
Section 6, we present the multiple existence and non-existence results to (7)
for $1<p<2$ and $\det(\mu_{ij})>0$, and give the proofs of Theorem 1.5 and
Theorem 1.6. In Appendix A, we prove Pohozaev identity for (7), and give non-
existence results to (7) when $p\in(0,1]\cup[5,\infty)$, which provide the
proof of Theorem 1.2. Finally, in Appendix B, we discuss some regularities of
the energy functional for (7).

##  2\. Classification of solutions in zero potential case

In this section, we prove Theorem 1.1, which gives a complete classification
of nonnegative solutions for (5). Throughout this section, we assume
$\lambda,\mu_{ij}>0$ for $i,j=1,2$ and $\vec{u}=(u_{1},u_{2})\in
H^{1}(\mathbb{R}^{3})\times H^{1}(\mathbb{R}^{3})$ always denotes an arbitrary
solution of (5).

###  2.1. Triviality of $\vec{u}$ when $\det(\mu_{ij})\geq 0$

We first deal with the case $\det(\mu_{ij})\geq 0$, i.e.,
$\mu_{11}\mu_{22}\geq\mu_{12}^{2}$. Our claim is that $\vec{u}$ is trivial.
Multiplying the first equation of (5) by $u_{1}$ and the second equation by
$u_{2}$ respectively, and integrating, we see that

$\displaystyle 0$ $\displaystyle=\int_{\mathbb{R}^{3}}\Big{(}-\Delta
u_{1}+\lambda
u_{1}+(\mu_{11}\phi_{u_{1}}-\mu_{12}\phi_{u_{2}})u_{1}\Big{)}u_{1}dx=\int_{\mathbb{R}^{3}}|\nabla
u_{1}|^{2}+\lambda
u_{1}^{2}+\mu_{11}\phi_{u_{1}}u_{1}^{2}-\mu_{12}\phi_{u_{2}}u_{1}^{2}dx$ (11)
$\displaystyle=\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}+\lambda
u_{1}^{2}+\mu_{11}\phi_{u_{1}}u_{1}^{2}-\mu_{12}\phi_{u_{1}}u_{2}^{2}dx$

and

$0=\int_{\mathbb{R}^{3}}\Big{(}-\Delta u_{2}+\lambda
u_{2}+(\mu_{22}\phi_{u_{2}}-\mu_{12}\phi_{u_{1}})u_{2}\Big{)}u_{2}dx=\int_{\mathbb{R}^{3}}|\nabla
u_{2}|^{2}+\lambda
u_{2}^{2}+\mu_{22}\phi_{u_{2}}u_{2}^{2}-\mu_{12}\phi_{u_{1}}u_{2}^{2}dx.$ (12)

Note that, since $\mu_{ij}>0,\mu_{11}\mu_{22}\geq\mu_{12}^{2}$ and
$\nabla\phi_{u_{1}}\cdot\nabla\phi_{u_{2}}\leq\frac{\mu_{11}}{2\mu_{12}}|\nabla\phi_{u_{1}}|^{2}+\frac{\mu_{12}}{2\mu_{11}}|\nabla\phi_{u_{2}}|^{2},$
where $i,j=1,2$, we have

$\mu_{11}|\nabla\phi_{u_{1}}|^{2}+\mu_{22}|\nabla\phi_{u_{2}}|^{2}-2\mu_{12}\nabla\phi_{u_{1}}\cdot\nabla\phi_{u_{2}}\geq\frac{\mu_{11}\mu_{22}-\mu_{12}^{2}}{\mu_{11}}|\nabla\phi_{u_{2}}|^{2}\geq
0.$ (13)

Then, by (13), adding (11) and (12), we get

$\displaystyle 0$ $\displaystyle=\int_{\mathbb{R}^{3}}|\nabla
u_{1}|^{2}+|\nabla u_{2}|^{2}+\lambda u_{1}^{2}+\lambda
u_{2}^{2}+\mu_{11}\phi_{u_{1}}u_{1}^{2}+\mu_{22}\phi_{u_{2}}u_{2}^{2}-2\mu_{12}\phi_{u_{1}}u_{2}^{2}dx$
$\displaystyle=\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}+|\nabla
u_{2}|^{2}+\lambda u_{1}^{2}+\lambda
u_{2}^{2}+\mu_{11}|\nabla\phi_{u_{1}}|^{2}+\mu_{22}|\nabla\phi_{u_{2}}|^{2}-2\mu_{12}\nabla\phi_{u_{1}}\cdot\nabla\phi_{u_{2}}dx$
$\displaystyle\geq\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}+|\nabla
u_{2}|^{2}+\lambda u_{1}^{2}+\lambda u_{2}^{2}dx,$

which implies that $\vec{u}\in H^{1}(\mathbb{R}^{3})\times
H^{1}(\mathbb{R}^{3})$ is trivial.

###  2.2. Explicit form of $\vec{u}$ when $\det(\mu_{ij})<0$

We next assume $\mu_{11}\mu_{22}<\mu_{12}^{2}$. Suppose that $\vec{u}$ is
nontrivial, nonnegative and radial. We have already seen from Remark 2 that
$\vec{u}$ is vectorial. Then the strong maximum principle applies to see that
$\vec{u}$ is positive. We now state a lemma which clarifies a structure of
$\vec{u}$.

######  Lemma 2.1.

Assume $\lambda,\mu_{12}>0$, $\mu_{ii}\geq 0$ for $i=1,2$. Then for any
positive radial solutions $\vec{u}=(u_{1},u_{2})$ of (5), we have $u_{2}\equiv
a_{0}u_{1}$, where $a_{0}=\sqrt{\frac{\mu_{11}+\mu_{12}}{\mu_{22}+\mu_{12}}}$.

###### Proof.

Let $\vec{u}=(u_{1},u_{2})$ be a positive radial solution of (5). By Newton’s
Theorem and Holder’s inequality, we see that

$\phi_{u_{i}}(x)=\frac{1}{|x|}\int_{0}^{\infty}u_{i}^{2}(s)s^{2}\min\\{|x|s^{-1},1\\}ds\leq
C\|u_{i}\|_{H^{1}}^{2}\min\\{1,|x|^{-1}\\}$ (14)

for $i=1,2$. Thus, by the comparison principle, for any
$\lambda^{\prime}<\lambda<\lambda^{\prime\prime}$, there exist constants
$C_{1},C_{2}>0$ such that

$C_{1}\exp(-\lambda^{\prime\prime}|x|)\leq u_{i}(x)\leq
C_{2}\exp(-\lambda^{\prime}|x|)\mbox{ for }x\in\mathbb{R}^{3}$

and thus $u_{i}^{2}/u_{j}\in H^{1}(\mathbb{R}^{3})$, where $i,j=1,2$. Denote
$v_{2}\equiv a_{0}^{-1}u_{2}$, where
$a_{0}=\sqrt{\frac{\mu_{11}+\mu_{12}}{\mu_{22}+\mu_{12}}}$. Then, since
$\phi_{u_{2}}=a_{0}^{2}\phi_{v_{2}}$, we have

$\begin{cases}-\Delta u_{1}+\lambda
u_{1}+(\mu_{11}\phi_{u_{1}}-a_{0}^{2}\mu_{12}\phi_{v_{2}})u_{1}=0,\\\ -\Delta
v_{2}+\lambda
v_{2}+(a_{0}^{2}\mu_{22}\phi_{v_{2}}-\mu_{12}\phi_{u_{1}})v_{2}=0.\end{cases}$
(15)

By dividing the first and second equations of (15) by $u_{1}$ and $v_{2}$,
respectively, and subtracting the second equation from the first equation, we
have

$\displaystyle 0$ $\displaystyle=-(\Delta u_{1})u_{1}^{-1}+(\Delta
v_{2})v_{2}^{-1}+\Big{(}(\mu_{11}+\mu_{12})\phi_{u_{1}}-a_{0}^{2}(\mu_{22}+\mu_{12})\phi_{v_{2}}\Big{)}$
$\displaystyle=-(\Delta u_{1})u_{1}^{-1}+(\Delta
v_{2})v_{2}^{-1}+(\mu_{11}+\mu_{12})(\phi_{u_{1}}-\phi_{v_{2}}).$

Multiplying the above equation by $u_{1}^{2}-v_{2}^{2}$, then integration by
parts yields

$\displaystyle 0$ $\displaystyle=\int_{\mathbb{R}^{3}}\nabla
u_{1}\cdot\nabla\Big{(}\frac{u_{1}^{2}-v_{2}^{2}}{u_{1}}\Big{)}-\nabla
v_{2}\cdot\Big{(}\frac{u_{1}^{2}-v_{2}^{2}}{v_{2}}\Big{)}+(\mu_{11}+\mu_{12})(\phi_{u_{1}}-\phi_{v_{2}})(u_{1}^{2}-v_{2}^{2})$
$\displaystyle=\int_{\mathbb{R}^{3}}(u_{1}^{2}+v_{2}^{2})\Big{|}\frac{\nabla
u_{1}}{u_{1}}-\frac{\nabla
v_{2}}{v_{2}}\Big{|}^{2}+(\mu_{11}+\mu_{12})|\nabla(\phi_{u_{1}}-\phi_{v_{2}})|^{2}dx,$

which implies that $u_{1}\equiv v_{2}$. ∎

By applying Lemma 2.1, we can immediately see that $\vec{u}$ has the form
$\vec{u}=(V,a_{0}V)$, where
$a_{0}=\sqrt{\frac{\mu_{11}+\mu_{12}}{\mu_{22}+\mu_{12}}}$ and $V$ is a
positive solution of the Hartree equation

$-\Delta u+\lambda
u+\frac{\mu_{11}\mu_{22}-\mu_{12}^{2}}{\mu_{22}+\mu_{12}}\phi_{u}u=0.$ (16)

We finally refer to [16, Theorem 10] or [19, Lemma 9] to conclude that if any
positive radial solution of (16) is unique when
$\mu_{11}\mu_{22}<\mu_{12}^{2}$. This completes the proof of Theorem 1.1.

##  3\. Structure theorems for nonnegative solutions in power potential case

The rest of the paper is devoted to the study of the system (7), that is, the
power potential case. We begin with defining the energy functional for (7):

$\displaystyle I(\vec{u})$
$\displaystyle=\frac{1}{2}\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}+|\nabla
u_{2}|^{2}+\lambda u_{1}^{2}+\lambda
u_{2}^{2}dx+\frac{1}{4}\int_{\mathbb{R}^{3}}\mu_{11}u_{1}^{2}\phi_{u_{1}}+\mu_{22}u_{2}^{2}\phi_{u_{2}}-2\mu_{12}u_{1}^{2}\phi_{u_{2}}dx$
$\displaystyle\qquad-\frac{1}{p+1}\frac{1}{2\pi}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}|u_{1}+e^{i\theta}u_{2}|^{p+1}d\theta
dx$ $\displaystyle=\frac{1}{2}\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}+|\nabla
u_{2}|^{2}+\lambda u_{1}^{2}+\lambda
u_{2}^{2}dx+\frac{1}{4}\int_{\mathbb{R}^{3}}\mu_{11}u_{1}^{2}\phi_{u_{1}}+\mu_{22}u_{2}^{2}\phi_{u_{2}}-2\mu_{12}u_{1}^{2}\phi_{u_{2}}dx$
$\displaystyle\qquad-\frac{1}{p+1}\frac{1}{2\pi}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p+1}{2}}d\theta
dx.$

We prove in Section B that $I$ is twice continuously differentiable on
$\mathbf{H}$ and its critical points are solutions to (7). By finding nonzero
critical points of $I$, one therefore can provide several nontrivial solutions
to (7). This shall be done throughout subsequent sections.

One fundamental task of this paper is to distinguish these solutions from the
semi-trivial ones. We will see that this can be accomplished by obtaining some
information on mathematical structures of the semi-trivial solutions such as
the energy levels or Morse indices of them. The main purpose of this section
is to investigate such structures of semi-trivial solutions. We also prove a
rigidity theorem for positive solutions, which are required to give a
classification on positive solutions.

###  3.1. Morse indices of semi-trivial solutions

In this subsection, we compute a lower bound of Morse indices of semi-trivial
critical points of $I$. We say $n$ is the Morse index of a critical point
$\vec{u}$ of $I$ if $n$ is the maximal dimension of subspaces
$V\subset\mathbf{H}$ satisfying

$I^{\prime\prime}(\vec{u})[h,h]<0\quad\text{for all }h\in V.$

The following identity is straightforward from (79) in Appendix B but
frequently invoked throughout this section.

######  Lemma 3.1.

Assume $1<p<5$. For $i=1,2$, let $\lambda>0,\mu_{ij}\geq
0,(t_{1},t_{2})\in\mathbb{R}^{2}$ and $(u_{1},u_{2})\in{\bf H}$. Then we have

$\displaystyle
I^{\prime\prime}(u_{1},0)\Big{[}(t_{1}u_{1},t_{2}u_{1}),(t_{1}u_{1},t_{2}u_{1})\Big{]}$
$\displaystyle=t_{1}^{2}\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}+\lambda
u_{1}^{2}+3\mu_{11}\phi_{u_{1}}u_{1}^{2}-p|u_{1}|^{p+1}dx$ (17)
$\displaystyle\quad+t_{2}^{2}\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}+\lambda
u_{1}^{2}-\mu_{12}\phi_{u_{1}}u_{1}^{2}-\frac{p+1}{2}|u_{1}|^{p+1}dx$

and

$\displaystyle
I^{\prime\prime}(0,u_{2})\Big{[}(t_{1}u_{2},t_{2}u_{2}),(t_{1}u_{2},t_{2}u_{2})\Big{]}$
$\displaystyle=t_{1}^{2}\int_{\mathbb{R}^{3}}|\nabla u_{2}|^{2}+\lambda
u_{2}^{2}-\mu_{12}\phi_{u_{2}}u_{2}^{2}-\frac{p+1}{2}|u_{2}|^{p+1}dx$
$\displaystyle\quad+t_{2}^{2}\int_{\mathbb{R}^{3}}|\nabla u_{2}|^{2}+\lambda
u_{2}^{2}+3\mu_{22}\phi_{u_{2}}u_{2}^{2}-p|u_{2}|^{p+1}dx.$

######  Proposition 3.2.

Assume that $1<p<5$. Let $\lambda$ and $\mu_{ij}$ be positive constants for
$i=1,2$. Then every semi-trivial critical point of $I$ has Morse index larger
than or equal to $1$.

###### Proof.

We may only deal with the semi-trivial critical point $(u_{1},0)$ due to the
symmetry of the system (7). Since $u_{2}=0$, $u_{1}$ solves

$-\Delta u_{1}+\lambda u_{1}+\mu_{11}\phi_{u_{1}}u_{1}=|u_{1}|^{p-1}u_{1}$

so that one has

$\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}+\lambda
u_{1}^{2}+\mu_{11}\phi_{u_{1}}u_{1}^{2}-|u_{1}|^{p+1}dx=0.$

Then by (17), we see that for $t\in\mathbb{R}\setminus\\{0\\}$,

$\displaystyle I^{\prime\prime}(u_{1},0)\Big{[}(0,tu_{1}),(0,tu_{1})\Big{]}$
$\displaystyle=t^{2}\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}+\lambda
u_{1}^{2}-\mu_{12}\phi_{u_{1}}u_{1}^{2}-\frac{p+1}{2}|u_{1}|^{p+1}dx$
$\displaystyle=-t^{2}\int_{\mathbb{R}^{3}}(\mu_{11}+\mu_{12})\phi_{u_{1}}u_{1}^{2}+\frac{p-1}{2}|u_{1}|^{p+1}dx<0.$

This shows that $1$ is a lower bound of Morse index of $(u_{1},0)$. ∎

######  Proposition 3.3.

Let $\lambda$ and $\mu_{ij}$ be positive constants for $i=1,2$. Assume that
$\frac{1}{3}(-2+\sqrt{73})\leq p<5$. Then every semi-trivial critical point of
$I$ has Morse index larger than or equal to $2$.

###### Proof.

Let $(u_{1},0)$ be a semi-trivial critical point of $I$ so that $u_{1}$ solves

$-\Delta u_{1}+\lambda u_{1}+\mu_{11}\phi_{u_{1}}u_{1}=|u_{1}|^{p-1}u_{1}.$

Define $a=\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}dx,\ \
b=\int_{\mathbb{R}^{3}}\lambda u_{1}^{2},\ \
c=\int_{\mathbb{R}^{3}}\mu_{11}\phi_{u_{1}}u_{1}^{2},\ \
d=\int_{\mathbb{R}^{3}}|u_{1}|^{p+1}dx.$ Then we have

$\begin{cases}a+b+c-d=0,\\\
\frac{1}{2}a+\frac{3}{2}b+\frac{5}{4}c-\frac{3}{p+1}d=0.\end{cases}$ (18)

Here, the second one is so-called the Pohozaev identity. We refer to [29] or
Proposition A.1 applied to the solution $(u_{1},0)$ in Appendix.

By (17), we have

$\displaystyle
I^{\prime\prime}(u_{1},0)\Big{[}(t_{1}u_{1},t_{2}u_{1}),(t_{1}u_{1},t_{2}u_{1})\Big{]}$
$\displaystyle=t_{1}^{2}\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}+\lambda
u_{1}^{2}+3\mu_{11}\phi_{u_{1}}u_{1}^{2}-p|u_{1}|^{p+1}dx$ (19)
$\displaystyle\quad+t_{2}^{2}\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}+\lambda
u_{1}^{2}-\mu_{12}\phi_{u_{1}}u_{1}^{2}-\frac{p+1}{2}|u_{1}|^{p+1}dx$
$\displaystyle=t_{1}^{2}(\@slowromancap i@)+t_{2}^{2}(\@slowromancap ii@).$

Since $a+b+c-d=0$, we have

$\displaystyle(\@slowromancap
ii@)=-\int_{\mathbb{R}^{3}}(\mu_{11}+\mu_{12})\phi_{u_{1}}u_{1}^{2}+\frac{p-1}{2}|u_{1}|^{p+1}dx<0.$
(20)

On the other hand, by (18), we have

$a=\frac{1}{3}b+\frac{5p-7}{3(p+1)}d,\ \
c=-\frac{4}{3}b+\frac{2(5-p)}{3(p+1)}d$

and

$(\@slowromancap
i@)=a+b+3c-pd=\frac{1}{3}\Big{(}-8b-\frac{3p^{2}+4p-23}{p+1}d\Big{)}.$

Thus, $\frac{3p^{2}+4p-23}{p+1}>0$ if $p>\frac{1}{3}(-2+\sqrt{73})\cong
2.18133$, which implies that $(\@slowromancap i@)<0$ if
$p\geq\frac{1}{3}(-2+\sqrt{73})$ and consequently the Morse index of
$(u_{1},0)$ is larger than or equal to 2. By the symmetry, Morse index of the
semi-trivial solution $(0,u_{2})$ for (7) is also larger than or equal to two.
∎

###  3.2. Energy comparison of semi-trivial solutions

In this subsection, we prove that a semi-trivial solution to (7) is not a
ground state when $\mu_{11}=\mu_{22}$ and $2<p<5$. We accomplish this by
comparing the energy of semi-trivial solutions with the energy of the solution
of the form $(W,W)$, whose existence is guaranteed for the case
$\mu_{11}=\mu_{22}$. In fact, if $\mu_{11}=\mu_{22}$ then $(W,W)$ gives a
solution of (7) for any solution $W$ of the equation

$-\Delta u+\lambda
u+({\mu_{11}-\mu_{12}})\phi_{u}u=\frac{c_{p}}{2}|u|^{p-1}u,$

where we define

$\displaystyle
c_{p}\coloneqq\frac{1}{2\pi}\int_{0}^{2\pi}|1+e^{i\theta}|^{p+1}d\theta=\frac{2^{\frac{p-1}{2}}}{\pi}\int_{0}^{2\pi}(1+\cos\theta)^{\frac{p+1}{2}}d\theta.$
(21)

We shall show that the energy level of $(W,W)$ is lower than that of semi-
trivial solutions.

We first prepare some auxiliary lemmas.

######  Lemma 3.4.

[29, Lemma 3.3] Let $2<p<5$, $a,c>0$ and $b\in\mathbb{R}$. Define
$h:[0,\infty)\rightarrow\mathbb{R}$ by

$h(t)=at+bt^{3}-ct^{2p-1}.$

Then $h$ has a unique critical point, which corresponds to its maximum.

Let us denote

$I_{\gamma}(u)\coloneqq\int_{\mathbb{R}^{3}}\frac{1}{2}\Big{(}|\nabla
u|^{2}+u^{2}\Big{)}+\frac{\gamma}{4}u^{2}\phi_{u}-\frac{1}{p+1}|u|^{p+1}dx.$

######  Lemma 3.5.

Let $v$ and $w$ be positive ground state solutions of

$-\Delta v+v+\gamma\phi_{v}v=|v|^{p-1}v\ \mbox{ and }\ -\Delta
w+w+\mu\phi_{w}w=|w|^{p-1}w,$

respectively, where $2<p<5$ and $\gamma,\mu\in\mathbb{R}$ with $\gamma>\mu.$
Then we have

$I_{\gamma}(v)>I_{\mu}(w).$

######  Remark 6.

The existence of a positive radial ground states for $I_{\gamma}$ with $2<p<5$
was proved in [29] when $\gamma>0$ and in [33] when $\gamma<0$.

###### Proof.

In fact, we recall

$I_{\gamma}(v)=\inf_{u\not\neq 0}\max_{t\geq 0}I_{\gamma}(u_{t})\mbox{ and
}I_{\mu}(w)=\inf_{u\not\neq 0}\max_{t\geq 0}I_{\mu}(u_{t}),$

where $u_{t}(x)\equiv t^{2}u(tx)$ (see [4, Lemma 2.4]). Combining the Nehari’s
identity

$\int_{\mathbb{R}^{3}}|\nabla v|^{2}+v^{2}+\gamma v^{2}\phi_{v}-|v|^{p+1}dx=0$

and the Pohozaev’s identity (see [29] or Proposition A.1)

$\int_{\mathbb{R}^{3}}\frac{1}{2}|\nabla
v|^{2}+\frac{3}{2}v^{2}+\frac{5\gamma}{4}v^{2}\phi_{v}-\frac{3}{p+1}v^{p+1}dx=0,$

we get

$\int_{\mathbb{R}^{3}}\frac{3}{2}|\nabla
v|^{2}+\frac{1}{2}v^{2}+\frac{3\gamma}{4}v^{2}\phi_{v}-\frac{2p-1}{p+1}v^{p+1}dx=0.$

From this and Lemma 3.4, we see that $I_{\gamma}(v)=\max_{t\geq
0}I_{\gamma}(v_{t})$, and thus

$\displaystyle I_{\gamma}(v)$ $\displaystyle=\max_{t\geq
0}I_{\gamma}(v_{t})=\max_{t\geq
0}\bigg{[}\int_{\mathbb{R}^{3}}\frac{1}{2}\Big{(}t^{3}|\nabla
v|^{2}+tv^{2}\Big{)}+\frac{\gamma}{4}t^{3}v^{2}\phi_{v}-\frac{t^{2p-1}}{p+1}|v|^{p+1}dx\bigg{]}$
$\displaystyle>\max_{t\geq
0}\bigg{[}\int_{\mathbb{R}^{3}}\frac{1}{2}\Big{(}t^{3}|\nabla
v|^{2}+tv^{2}\Big{)}+\frac{\mu}{4}t^{3}v^{2}\phi_{v}-\frac{t^{2p-1}}{p+1}|v|^{p+1}dx\bigg{]}\geq
I_{\mu}(w).$

∎

A direct computation shows that there holds the following scaling properties.

######  Lemma 3.6.

Let $v$ be a solution of

$-\Delta v+av+b\phi_{v}v=c|v|^{p-1}v,$

where $1<p<5$ and $a,b,c>0$, which is a Euler-Lagrange equation of

$I_{a,b,c}(v)=\int_{\mathbb{R}^{3}}\frac{1}{2}|\nabla
v|^{2}+\frac{a}{2}v^{2}+\frac{b}{4}\phi_{v}v^{2}-\frac{c}{p+1}|v|^{p+1}dx.$

Then a rescaled function
$\tilde{v}(x)=\Big{(}\frac{c}{a}\Big{)}^{\frac{1}{p-1}}v\Big{(}\frac{x}{\sqrt{a}}\Big{)}$
solves

$-\Delta\tilde{v}+\tilde{v}+\gamma(a,b,c)\phi_{\tilde{v}}\tilde{v}=|\tilde{v}|^{p-1}\tilde{v},$

where $\gamma(a,b,c)=\frac{b}{a^{2}}\left(\frac{a}{c}\right)^{\frac{2}{p-1}}$
and satisfies

$I_{\gamma(a,b,c)}(\tilde{v})=I_{1,\gamma(a,b,c),1}(\tilde{v})=\Big{(}\frac{c}{a}\Big{)}^{\frac{2}{p-1}}a^{\frac{1}{2}}I_{a,b,c}(v).$

Now, we are ready to prove the main result of this subsection.

######  Proposition 3.7.

Assume $2<p<5$, $\lambda>0,\mu_{ij}>0$, $\mu_{11}=\mu_{22}$, where $i,j=1,2$.
Then we have

$I(V,0)=I(0,V)>I(W,W),$ (22)

where $W$ and $V$ are positive ground state solutions of

$-\Delta u+\lambda u+({\mu_{11}-\mu_{12}})\phi_{u}u=\frac{c_{p}}{2}|u|^{p-1}u\
\mbox{ and }\ -\Delta u+\lambda u+\mu_{11}\phi_{u}u=|u|^{p-1}u,$

respectively.

###### Proof.

Denote
$\tilde{W}(x)=\Big{(}\frac{c_{p}}{2\lambda}\Big{)}^{\frac{1}{p-1}}W\Big{(}\frac{x}{\sqrt{\lambda}}\Big{)}$
and
$\tilde{V}(x)=\Big{(}\frac{1}{\lambda}\Big{)}^{\frac{1}{p-1}}V\Big{(}\frac{x}{\sqrt{\lambda}}\Big{)}$.
Then, by Lemma 3.6, we see that $\tilde{W}$ and $\tilde{V}$ are positive
ground state solutions of

$-\Delta
u+u+\frac{2^{\frac{2}{p-1}}(\mu_{11}-\mu_{12})}{\lambda^{\frac{2p-4}{p-1}}c_{p}^{\frac{2}{p-1}}}\phi_{u}u=|u|^{p-1}u\
\mbox{ and }\ -\Delta
u+u+\frac{\mu_{11}}{\lambda^{\frac{2p-4}{p-1}}}\phi_{u}u=|u|^{p-1}u,$

respectively. By Jensen’s inequality, we see that for $1<p<5$,

$1=\left(\frac{1}{2\pi}\int_{0}^{2\pi}(1+\cos\theta)d\theta\right)^{\frac{p+1}{2}}<\frac{1}{2\pi}\int_{0}^{2\pi}(1+\cos\theta)^{\frac{p+1}{2}}d\theta.$

which implies that $2^{\frac{p+1}{2}}<c_{p}$ for $1<p<5$. Note that this means
$\Big{(}\frac{c_{p}}{2}\Big{)}^{\frac{2}{p-1}}>1$ for $2<p<5$ and thus we have
$\mu_{11}>\frac{2^{\frac{2}{p-1}}(\mu_{11}-\mu_{12})}{c_{p}^{\frac{2}{p-1}}}$.
Then, by Lemma 3.5 and Lemma 3.6,

$\displaystyle\int_{\mathbb{R}^{3}}\frac{1}{2}\Big{(}|\nabla\tilde{V}|^{2}+\tilde{V}^{2}\Big{)}+\frac{\mu_{11}}{4\lambda^{\frac{2p-4}{p-1}}}\tilde{V}^{2}\phi_{\tilde{V}}-\frac{1}{p+1}|\tilde{V}|^{p+1}dx$
$\displaystyle=\lambda^{-\frac{5-p}{2(p-1)}}\int_{\mathbb{R}^{3}}\frac{1}{2}\Big{(}|\nabla
V|^{2}+\lambda
V^{2}\Big{)}+\frac{\mu_{11}}{4}V^{2}\phi_{V}-\frac{1}{p+1}|V|^{p+1}dx$
$\displaystyle=\lambda^{-\frac{5-p}{2(p-1)}}I(V,0)$
$\displaystyle>\int_{\mathbb{R}^{3}}\frac{1}{2}\Big{(}|\nabla\tilde{W}|^{2}+\tilde{W}^{2}\Big{)}+\frac{2^{\frac{2}{p-1}}(\mu_{11}-\mu_{12})}{4\lambda^{\frac{2p-4}{p-1}}c_{p}^{\frac{2}{p-1}}}\phi_{\tilde{W}}\tilde{W}^{2}-\frac{1}{p+1}|\tilde{W}|^{p+1}dx$
$\displaystyle=\frac{c_{p}^{\frac{2}{p-1}}}{2^{\frac{2}{p-1}}\lambda^{\frac{5-p}{2(p-1)}}}\int_{\mathbb{R}^{3}}\frac{1}{2}\Big{(}|\nabla
W|^{2}+\lambda
W^{2}\Big{)}+\frac{\mu_{11}-\mu_{12}}{4}\phi_{W}W^{2}-\frac{c_{p}}{2(p+1)}|W|^{p+1}dx$
$\displaystyle=\frac{c_{p}^{\frac{2}{p-1}}}{2^{\frac{2}{p-1}}\lambda^{\frac{5-p}{2(p-1)}}}\frac{1}{2}I(W,W).$

Note that, since $2^{\frac{p+1}{2}}<c_{p}$ for $2<p<5$, we get
$\lambda^{-\frac{5-p}{2(p-1)}}<\frac{c_{p}^{\frac{2}{p-1}}}{2^{\frac{p+1}{p-1}}\lambda^{\frac{5-p}{2(p-1)}}}.$
From this, we see that for $2<p<5$,

$I(V,0)=I(0,V)>I(W,W).$

∎

###  3.3. Rigidity results for positive solutions

Analogously with Lemma 2.1, we can show the coincidence of components for
positive solutions to (7) with $\mu_{11}=\mu_{22}$.

######  Proposition 3.8.

Assume $1<p<5$, $\lambda>0$, $\mu_{ij}\geq 0$ and $\mu_{11}=\mu_{22}$, where
$i,j=1,2$. For any positive radial solutions $\vec{u}=(u_{1},u_{2})$ of (7),
we have $u_{1}\equiv u_{2}$.

###### Proof.

Let $\vec{u}=(u_{1},u_{2})\in{\bf H}$ be a positive solution of

$\begin{cases}-\Delta u_{1}+\lambda
u_{1}+(\mu_{11}\phi_{u_{1}}-\mu_{12}\phi_{u_{2}})u_{1}=\frac{1}{2\pi}\int_{0}^{2\pi}(u_{1}+u_{2}\cos\theta)\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p-1}{2}}d\theta,\\\
-\Delta u_{2}+\lambda
u_{2}+(\mu_{22}\phi_{u_{2}}-\mu_{12}\phi_{u_{1}})u_{2}=\frac{1}{2\pi}\int_{0}^{2\pi}(u_{2}+u_{1}\cos\theta)\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p-1}{2}}d\theta.\end{cases}$

By (75), we see that

$\displaystyle\int_{0}^{2\pi}(u_{1}+u_{2}\cos\theta)\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p-1}{2}}d\theta\leq
Cu_{1}(u_{1}^{p-1}+u_{2}^{p-1})\leq\tilde{C}|x|^{-(p-1)}u_{1},$

where we used the Strauss inequality $|u_{i}(x)|\leq
C\|u_{i}\|_{H^{1}}|x|^{-1}$ for $i=1,2$. Moreover, by Newton’s Theorem,

$\phi_{u_{i}}(x)=\frac{1}{|x|}\int_{0}^{\infty}u_{i}^{2}(s)s\min\\{|x|,s\\}ds\leq
C\min\\{1,\|u_{i}\|_{H^{1}}^{2}|x|^{-1}\\}$

for $i=1,2$. Thus, by the comparison principle, for any
$\lambda^{\prime}<\lambda<\lambda^{\prime\prime}$, there exist constants
$C_{1},C_{2}>0$ such that

$C_{1}\exp(-\lambda^{\prime\prime}|x|)\leq u_{i}(x)\leq
C_{2}\exp(-\lambda^{\prime}|x|)\mbox{ for }x\in\mathbb{R}^{3},$ (23)

where $i=1,2$. From (23), we deduce that $u_{i}^{2}/u_{j}\in
H^{1}(\mathbb{R}^{3})$, where $i,j=1,2$. Moreover, by (72), we see that for
$a,b>0$,

$\displaystyle\int_{0}^{2\pi}\frac{a+b\cos\theta}{a}\Big{(}a^{2}+2ab\cos\theta+b^{2}\Big{)}^{\frac{p-1}{2}}-\frac{b+a\cos\theta}{b}\Big{(}a^{2}+2ab\cos\theta+b^{2}\Big{)}^{\frac{p-1}{2}}d\theta$
(24)
$\displaystyle=\frac{b^{2}-a^{2}}{ab}\int_{0}^{2\pi}\cos\theta\Big{(}a^{2}+2ab\cos\theta+b^{2}\Big{)}^{\frac{p-1}{2}}d\theta$
$\displaystyle=(p-1)(b^{2}-a^{2})\int_{0}^{2\pi}\sin^{2}\theta\Big{(}a^{2}+2ab\cos\theta+b^{2}\Big{)}^{\frac{p-3}{2}}d\theta.$

Then, by dividing the first and second equations of (7) by $u_{1}$ and
$u_{2}$, respectively, and subtracting the second one from the first one, we
see from (24) that

$\displaystyle 0$ $\displaystyle=-(\Delta u_{1})u_{1}^{-1}+(\Delta
u_{2})u_{2}^{-1}+\Big{(}(\mu_{11}+\mu_{12})\phi_{u_{1}}-(\mu_{22}+\mu_{12})\phi_{u_{2}}\Big{)}$
$\displaystyle\qquad+\frac{1}{2\pi}\int_{0}^{2\pi}\bigg{[}\frac{u_{2}+u_{1}\cos\theta}{u_{2}}-\frac{(u_{1}+u_{2}\cos\theta)}{u_{1}}\bigg{]}(u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2})^{\frac{p-1}{2}}d\theta$
$\displaystyle=-(\Delta u_{1})u_{1}^{-1}+(\Delta
u_{2})u_{2}^{-1}+(\mu_{11}+\mu_{12})(\phi_{u_{1}}-\phi_{u_{2}})$
$\displaystyle\quad+\frac{(p-1)(u_{1}^{2}-u_{2}^{2})}{2\pi}\int_{0}^{2\pi}\sin^{2}\theta\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p-3}{2}}d\theta.$

By (23) and the fact that

$(u_{1}^{2}-u_{2}^{2})^{2}(u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2})^{\frac{p-3}{2}}\leq
C\begin{cases}(u_{1}+u_{2})^{2}|u_{1}-u_{2}|^{p-1}&\mbox{ if }1<p\leq 3,\\\
(u_{1}^{4}+u_{2}^{4})(u_{1}^{p-3}+u_{2}^{p-3})&\mbox{ if }3<p<5,\end{cases}$

multiplying the above equation by $u_{1}^{2}-u_{2}^{2}$, then integration by
parts yields

$\displaystyle 0$ $\displaystyle=\int_{\mathbb{R}^{3}}\nabla
u_{1}\cdot\nabla\Big{(}\frac{u_{1}^{2}-u_{2}^{2}}{u_{1}}\Big{)}-\nabla
u_{2}\cdot\Big{(}\frac{u_{1}^{2}-u_{2}^{2}}{u_{2}}\Big{)}+(\mu_{11}+\mu_{12})(\phi_{u_{1}}-\phi_{u_{2}})(u_{1}^{2}-u_{2}^{2})$
(25)
$\displaystyle\qquad+\frac{(p-1)(u_{1}^{2}-u_{2}^{2})^{2}}{2\pi}\int_{0}^{2\pi}\sin^{2}\theta\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p-3}{2}}d\theta
dx$
$\displaystyle=\int_{\mathbb{R}^{3}}(u_{1}^{2}+u_{2}^{2})\Big{|}\frac{\nabla
u_{1}}{u_{1}}-\frac{\nabla
u_{2}}{u_{2}}\Big{|}^{2}+(\mu_{11}+\mu_{12})|\nabla(\phi_{u_{1}}-\phi_{u_{2}})|^{2}$
$\displaystyle\qquad+\frac{(p-1)(u_{1}^{2}-u_{2}^{2})^{2}}{2\pi}\int_{0}^{2\pi}\sin^{2}\theta\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p-3}{2}}d\theta
dx.$

If $u_{1}\not\equiv u_{2}$ in $\mathbb{R}^{3}$, then we may assume that
$\Omega=\\{x\in\mathbb{R}^{3}:u_{1}(x)<u_{2}(x)\\}\neq\emptyset.$ Then by (25)
and the fact that

$u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}=(u_{2}-u_{1})^{2}+2u_{1}u_{2}(1+\cos\theta)>0\mbox{
for all }\theta\in[0,2\pi]\mbox{ and }x\in\Omega,$

we see that $u_{1}\equiv u_{2}$ in $\Omega$, which is a contradiction. Thus,
we have $u_{1}\equiv u_{2}$ in $\mathbb{R}^{3}$. ∎

##  4\. Power potential case with $2<p<5$

In this section, we prove the existence results of (7) for $2<p<5$, and give
the proof of Theoreom 1.3.

###  4.1. Construction of a nonnegative ground state

In this subsection, we construct a nontrivial nonnegative ground state
solution of (7) for $2<p<5$. We mainly follow the approach adopted in [29] but
we need to modify arguments because the term

$\int_{\mathbb{R}^{3}}\mu_{11}u_{1}^{2}\phi_{u_{1}}+\mu_{22}u_{2}^{2}\phi_{u_{2}}-2\mu_{12}u_{1}^{2}\phi_{u_{2}}\,dx$

appearing in the energy functional $I$ is not sign definite.

We begin with by setting the manifold

$\mathcal{M}\equiv\\{\vec{u}=(u_{1},u_{2})\in{\bf H}\setminus\\{\vec{0}\\}\ |\
G(\vec{u})=0\\},$ (26)

where

$\displaystyle G(\vec{u})$ $\displaystyle\equiv 2J(\vec{u})+P(\vec{u})$
$\displaystyle=\int_{\mathbb{R}^{3}}\frac{3}{2}\Big{(}|\nabla
u_{1}|^{2}+|\nabla
u_{2}|^{2}\Big{)}+\frac{1}{2}\lambda(u_{1}^{2}+u_{2}^{2})+\frac{3}{4}\Big{(}\mu_{11}u_{1}^{2}\phi_{u_{1}}+\mu_{22}u_{2}^{2}\phi_{u_{2}}-2\mu_{12}u_{1}^{2}\phi_{u_{2}}\Big{)}$
$\displaystyle\qquad-\frac{2p-1}{p+1}\bigg{[}\frac{1}{2\pi}\int_{0}^{2\pi}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p+1}{2}}d\theta\bigg{]}dx$

and $J$ and $P$ are given by

$\displaystyle J(\vec{u})$ $\displaystyle\equiv I^{\prime}(\vec{u})\vec{u}$
$\displaystyle=\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}+|\nabla
u_{2}|^{2}+\lambda u_{1}^{2}+\lambda
u_{2}^{2}dx+\int_{\mathbb{R}^{3}}\mu_{11}u_{1}^{2}\phi_{u_{1}}+\mu_{22}u_{2}^{2}\phi_{u_{2}}-2\mu_{12}u_{1}^{2}\phi_{u_{2}}dx$
$\displaystyle\qquad-\frac{1}{2\pi}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p+1}{2}}d\theta
dx,$ $\displaystyle P(\vec{u})$
$\displaystyle\equiv\int_{\mathbb{R}^{3}}-\frac{1}{2}\Big{(}|\nabla
u_{1}|^{2}+|\nabla
u_{2}|^{2}\Big{)}-\frac{3}{2}\lambda(u_{1}^{2}+u_{2}^{2})dx-\frac{5}{4}\int_{\mathbb{R}^{3}}\mu_{11}u_{1}^{2}\phi_{u_{1}}+\mu_{22}u_{2}^{2}\phi_{u_{2}}-2\mu_{12}u_{1}^{2}\phi_{u_{2}}dx$
$\displaystyle\qquad+\frac{3}{p+1}\int_{\mathbb{R}^{3}}\frac{1}{2\pi}\int_{0}^{2\pi}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p+1}{2}}d\theta
dx.$

Note that, since

$\displaystyle G^{\prime}(\vec{u})[\psi_{1},\psi_{2}]$ (27)
$\displaystyle=\int_{\mathbb{R}^{3}}3(\nabla u_{1}\cdot\nabla\psi_{1}+\nabla
u_{2}\cdot\nabla\psi_{2})+\lambda u_{1}\psi_{1}+\lambda u_{2}\psi_{2}dx$
$\displaystyle\quad+3\int_{\mathbb{R}^{3}}\mu_{11}u_{1}\phi_{u_{1}}\psi_{1}+\mu_{22}u_{2}\phi_{u_{2}}\psi_{2}-\mu_{12}u_{1}\phi_{u_{2}}\psi_{1}-\mu_{12}u_{2}\phi_{u_{1}}\psi_{2}dx$
$\displaystyle\quad-(2p-1)\frac{1}{2\pi}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\Big{(}(u_{1}+u_{2}\cos\theta)\psi_{1}+(u_{1}\cos\theta+u_{2})\psi_{2}\Big{)}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p-1}{2}}d\theta
dx$ $\displaystyle=\int_{\mathbb{R}^{3}}3(\nabla
u_{1}\cdot\nabla\psi_{1}+\nabla u_{2}\cdot\nabla\psi_{2})+\lambda
u_{1}\psi_{1}+\lambda u_{2}\psi_{2}dx$
$\displaystyle\quad+3\int_{\mathbb{R}^{3}}\mu_{11}u_{1}\phi_{u_{1}}\psi_{1}+\mu_{22}u_{2}\phi_{u_{2}}\psi_{2}-\mu_{12}u_{1}\phi_{u_{2}}\psi_{1}-\mu_{12}u_{2}\phi_{u_{1}}\psi_{2}dx$
$\displaystyle\quad-(2p-1)\frac{1}{2\pi}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}|u_{1}+e^{i\theta}u_{2}|^{p-1}(u_{1}+e^{i\theta}u_{2})\psi_{1}+|u_{2}+e^{i\theta}u_{1}|^{p-1}(u_{2}+e^{i\theta}u_{1})\psi_{2}d\theta
dx,$

if $G^{\prime}(\vec{u})=0$, $\vec{u}$ satisfies

$\begin{cases}-3\Delta u_{1}+\lambda
u_{1}+3(\mu_{11}\phi_{u_{1}}-\mu_{12}\phi_{u_{2}})u_{1}=(2p-1)\frac{1}{2\pi}\int_{0}^{2\pi}|u_{1}+e^{i\theta}u_{2}|^{p-1}(u_{1}+e^{i\theta}u_{2})d\theta,\\\
-3\Delta u_{2}+\lambda
u_{2}+3(\mu_{22}\phi_{u_{2}}-\mu_{12}\phi_{u_{1}})u_{2}=(2p-1)\frac{1}{2\pi}\int_{0}^{2\pi}|u_{2}+e^{i\theta}u_{1}|^{p-1}(u_{2}+e^{i\theta}u_{1})d\theta.\end{cases}$
(28)

######  Lemma 4.1.

Let $2<p<5$ and $\lambda,\mu_{ij}>0$ for $i,j=1,2$. Then $\mathcal{M}$ is non-
empty and $\vec{0}\notin\partial\mathcal{M}$. Moreover,
$\inf_{\mathcal{M}}I>0$, and if $\inf_{\mathcal{M}}I=I(\vec{u})$,
$G^{\prime}(\vec{u})\neq 0$.

###### Proof.

Note that, by Jensen’s inequality, for $\vec{u}=(u_{1},u_{2})\in{\bf
H}\setminus\\{\vec{0}\\}$

$\displaystyle 0<\int_{\mathbb{R}^{3}}(u_{1}^{2}+u_{2}^{2})^{\frac{p+1}{2}}dx$
$\displaystyle=\int_{\mathbb{R}^{3}}\Big{(}\frac{1}{2\pi}\int_{0}^{2\pi}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}d\theta\Big{)}^{\frac{p+1}{2}}dx$
(29)
$\displaystyle\leq\frac{1}{2\pi}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p+1}{2}}d\theta
dx.$

For any $\vec{u}=(u_{1},u_{2})\in{\bf H}\setminus\\{\vec{0}\\}$, one has that

$\displaystyle t^{2}(u_{1}(t\cdot),u_{2}(t\cdot))\in\mathcal{M}$
$\displaystyle\iff\int_{\mathbb{R}^{3}}\frac{3}{2}t^{3}\Big{(}|\nabla
u_{1}|^{2}+|\nabla
u_{2}|^{2}\Big{)}+\frac{1}{2}t\lambda(u_{1}^{2}+u_{2}^{2})+\frac{3}{4}t^{3}\Big{(}\mu_{11}u_{1}^{2}\phi_{u_{1}}+\mu_{22}u_{2}^{2}\phi_{u_{2}}-2\mu_{12}u_{1}^{2}\phi_{u_{2}}\Big{)}dx$
$\displaystyle\qquad\qquad=t^{2p-1}\frac{2p-1}{p+1}\int_{\mathbb{R}^{3}}\bigg{[}\frac{1}{2\pi}\int_{0}^{2\pi}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p+1}{2}}d\theta\bigg{]}dx.$

From this and (29), for all $\vec{u}=(u_{1},u_{2})\in{\bf
H}\setminus\\{\vec{0}\\}$, there exists a unique $t>0$ such that
$t\vec{u}\in\mathcal{M}$, which implies that $\mathcal{M}$ is non-empty.

By the facts that $\int_{\mathbb{R}^{3}}|u|^{p+1}dx\leq
C_{1}\|u\|_{H^{1}}^{p+1}$ and

$\int_{\mathbb{R}^{3}}u^{2}\phi_{v}dx\leq\|u\|_{L^{\frac{12}{5}}}^{2}\|\phi_{v}\|_{L^{6}}\leq\frac{1}{2}\Big{(}\|u\|_{L^{\frac{12}{5}}}^{4}+\|\phi_{v}\|_{L^{6}}^{2}\Big{)}\leq
C_{2}\Big{(}\|u\|_{H^{1}}^{4}+\|v\|_{H^{1}}^{4}\Big{)},$ (30)

where $C_{1},C_{2}>0$ are constants, we have

$G(\vec{u})\geq\frac{1}{2}(\|u_{1}\|_{\lambda}^{2}+\|u_{2}\|_{\lambda}^{2})\left(1-C\frac{\|u_{1}\|_{\lambda}^{4}+\|u_{2}\|_{\lambda}^{4}}{\|u_{1}\|_{\lambda}^{2}+\|u_{2}\|_{\lambda}^{2}}-C\frac{\|u_{1}\|_{\lambda}^{p+1}+\|u_{2}\|_{\lambda}^{p+1}}{\|u_{1}\|_{\lambda}^{2}+\|u_{2}\|_{\lambda}^{2}}\right),$

and hence we deduce that there is $\rho>0$ such that

$\|u_{1}\|_{\lambda}^{2}+\|u_{2}\|_{\lambda}^{2}\geq\rho\mbox{ for all
}\vec{u}\in\mathcal{M},$ (31)

which implies that $\vec{0}\notin\partial\mathcal{M}$.

We claim that $\inf_{\mathcal{M}}I>0$. Let $k=I(\vec{u})$, where
$\vec{u}=(u_{1},u_{2})\in\mathcal{M}$. Define

$\displaystyle a$ $\displaystyle=\int_{\mathbb{R}^{3}}|\nabla
u_{1}|^{2}+|\nabla u_{2}|^{2}dx,\ \ b=\int_{\mathbb{R}^{3}}\lambda
u_{1}^{2}+\lambda u_{2}^{2}dx,$ (32) $\displaystyle c$
$\displaystyle=\int_{\mathbb{R}^{3}}\mu_{11}u_{1}^{2}\phi_{u_{1}}+\mu_{22}u_{2}^{2}\phi_{u_{2}}-2\mu_{12}u_{1}^{2}\phi_{u_{2}}dx,\
\
d=\int_{\mathbb{R}^{3}}\bigg{[}\frac{1}{2\pi}\int_{0}^{2\pi}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p+1}{2}}d\theta\bigg{]}dx.$

Note that $a,b,c$ and $d$ satisfy

$\begin{cases}\frac{1}{2}a+\frac{1}{2}b+\frac{1}{4}c-\frac{1}{p+1}d=k,\\\
\frac{3}{2}a+\frac{1}{2}b+\frac{3}{4}c-\frac{2p-1}{p+1}d=0.\end{cases}$ (33)

We solve the above system of equations for arbitrarily given $a,b$ and $k$ to
get

$c=-2\frac{a(p-2)+b(p-1)+k(1-2p)}{p-2},\ \ d=-\frac{b(p+1)-3k(p+1)}{2(p-2)}.$

From the latter equality, we have

$(p+1)b+2(p-2)d=3k(p+1)$ (34)

so that $k>0$ and consequently $\inf_{\mathcal{M}}I\geq 0$. Moreover, since
$2<\frac{12}{5}<p+1$,

$\displaystyle\int_{\mathbb{R}^{3}}|\nabla\phi_{u}|^{2}dx$
$\displaystyle=\int_{\mathbb{R}^{3}}\phi_{u}u^{2}dx\leq\|\phi_{u}\|_{L^{6}}\|u\|_{L^{\frac{12}{5}}}^{2}\leq
C\|\phi_{u}\|_{D^{1,2}}\|u\|_{L^{2}}^{2\Lambda}\|u\|_{L^{p+1}}^{2(1-\Lambda)},$

where $\Lambda=\frac{5p-7}{6(p-1)}$, and hence by (29) we deduce that

$|c|\leq C(\|\phi_{u_{1}}\|_{D^{1,2}}^{2}+\|\phi_{u_{2}}\|_{D^{1,2}}^{2})\leq
C(b^{4\Lambda}+d^{\frac{8(1-\Lambda)}{p+1}}).$ (35)

Now, suppose that $\inf_{\mathcal{M}}I=0$. Then there exists a sequence
$\\{\vec{u}_{n}\\}\in\mathcal{M}$ satisfying $k_{n}=I(\vec{u_{n}})\to 0$ as
$n\to\infty$. We define the constants $a_{n},b_{n},c_{n},d_{n}$ as in (32) for
$\vec{u}_{n}$. Since $p>2,\,b_{n},d_{n}>0$, (34) tells us that $b_{n},d_{n}\to
0$ as $n\to\infty$. This shows by (35) $c_{n}\to 0$ as $n\to\infty$ but this
contradicts with (31) and (33). Thus we conclude that $\inf_{\mathcal{M}}I>0$.

Finally, we claim that if $\inf_{\mathcal{M}}I=I(\vec{u})=k$, then
$G^{\prime}(\vec{u})\neq 0$. Contrary to the claim, we assume that
$\inf_{\mathcal{M}}I=I(\vec{u})=k$ and $G^{\prime}(\vec{u})=0$. By (27), one
has

$\displaystyle G^{\prime}(\vec{u})\vec{u}$
$\displaystyle=\int_{\mathbb{R}^{3}}3(|\nabla u_{1}|^{2}+|\nabla
u_{2}|^{2})+\lambda(u_{1}^{2}+u_{2}^{2})+3\Big{(}\mu_{11}u_{1}^{2}\phi_{u_{1}}+\mu_{22}u_{2}^{2}\phi_{u_{2}}-2\mu_{12}u_{1}^{2}\phi_{u_{2}}\Big{)}$
$\displaystyle\qquad-(2p-1)\bigg{[}\frac{1}{2\pi}\int_{0}^{2\pi}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p+1}{2}}d\theta\bigg{]}dx=0.$

Then if $G^{\prime}(\vec{u})=0$, we have

$\begin{cases}\frac{1}{2}a+\frac{1}{2}b+\frac{1}{4}c-\frac{1}{p+1}d=k,\\\
\frac{3}{2}a+\frac{1}{2}b+\frac{3}{4}c-\frac{2p-1}{p+1}d=0\\\
3a+b+3c-(2p-1)d=0\\\
\frac{3}{2}a+\frac{3}{2}b+3\frac{5}{4}c-(2p-1)\frac{3}{p+1}d=0.\end{cases}$

The fourth equation is the Pohozaev identity(see Proposition A.1) applies to
(28). Then we see that

$a=-k\frac{2p-1}{4(p-2)},\ b=3k\frac{2p-1}{2(p-1)},\ c=-k\frac{2p-1}{2(p-2)},\
d=-3k\frac{p+1}{4(2-3p+p^{2})},$

and thus by the facts that $k>0,a\geq 0$ and $2<p<5$, we can deduce a
contradiction. ∎

######  Lemma 4.2.

Let $2<p<5$ and $\lambda,\mu_{ij}>0$ for $i,j=1,2$. If $\vec{u}\in{\bf H}$ is
a minimizer of $I$ on $\mathcal{M}$, then $\vec{u}$ is a non-trivial critical
point of $I$.

###### Proof.

Suppose that $\vec{u}=(u_{1},u_{2})\in{\bf H}$ is a minimizer of $I$ on
$\mathcal{M}$. Then, by Lemma 4.1, there exists $\omega\in\mathbb{R}$ such
that

$I^{\prime}(\vec{u})=\omega G^{\prime}(\vec{u}),$

which can be written as, in a weak sense,

$\begin{cases}&-(1-3\omega)\Delta u_{1}+(1-\omega)\lambda
u_{1}+(1-3\omega)(\mu_{11}\phi_{u_{1}}-\mu_{12}\phi_{u_{2}})u_{1}\\\
&\qquad+\Big{(}-1+(2p-1)\omega\Big{)}\frac{1}{2\pi}\int_{0}^{2\pi}|u_{1}+e^{i\theta}u_{2}|^{p-1}(u_{1}+e^{i\theta}u_{2})d\theta=0,\\\
&-(1-3\omega)\Delta u_{2}+(1-\omega)\lambda
u_{2}+(1-3\omega)(\mu_{22}\phi_{u_{2}}-\mu_{12}\phi_{u_{1}})u_{2}\\\
&\qquad+\Big{(}-1+(2p-1)\omega\Big{)}\frac{1}{2\pi}\int_{0}^{2\pi}|u_{2}+e^{i\theta}u_{1}|^{p-1}(u_{2}+e^{i\theta}u_{1})d\theta=0.\end{cases}$
(36)

Then, since $\vec{u}\in\mathcal{M}$, $k=I(\vec{u})$,
$I^{\prime}(\vec{u})\vec{u}=\omega G^{\prime}(\vec{u})\vec{u}$ and the
Pohozaev identity(see Proposition A.1) applied to (36), we have

$\displaystyle\begin{cases}\frac{1}{2}a+\frac{1}{2}b+\frac{1}{4}c-\frac{1}{p+1}d=k,\\\
\frac{3}{2}a+\frac{1}{2}b+\frac{3}{4}c-\frac{2p-1}{p+1}d=0,\\\
(1-3\omega)a+(1-\omega)b+(1-3\omega)c+\Big{(}-1+(2p-1)\omega\Big{)}d=0,\\\
(1-3\omega)\frac{1}{2}a+(1-\omega)\frac{3}{2}b+(1-3\omega)\frac{5}{4}c-\Big{(}1-(2p-1)\omega\Big{)}\frac{3}{p+1}d=0,\end{cases}$

where $a,b,c,d$ are given in (32). Then since $b,d>0$ and $2<p<5$, we can
prove $\omega=0$ (see [29, Theorem 3.2]). ∎

######  Lemma 4.3.

It holds that

$\|\phi_{u}-\phi_{v}\|_{D^{1,2}}\leq C\|u-v\|_{L^{3}}\|u+v\|_{L^{2}},$

where $C>0$ is a constant.

###### Proof.

Since $-\Delta(\phi_{u}-\phi_{v})=u^{2}-v^{2}$, we have

$\displaystyle\int_{\mathbb{R}^{3}}|\nabla(\phi_{u}-\phi_{v})|^{2}dx$
$\displaystyle\leq\int_{\mathbb{R}^{3}}(u-v)(u+v)(\phi_{u}-\phi_{v})dx\leq\|u-v\|_{L^{3}}\|u+v\|_{L^{2}}\|\phi_{u}-\phi_{v}\|_{L^{6}}$
$\displaystyle\leq
C\|u-v\|_{L^{3}}\|u+v\|_{L^{2}}\|\phi_{u}-\phi_{v}\|_{D^{1,2}}.$

∎

######  Proposition 4.4.

Let $2<p<5$ and $\lambda,\mu_{ij}>0$ for $i,j=1,2$. Then $I$ has a global
minimum $\vec{u}$ on $\mathcal{M}$. Moreover, $\vec{u}$ is a non-trivial, non-
negative critical point of $I$.

###### Proof.

Let $\vec{u}_{n}=(u_{1,n},u_{2,n})\in\mathcal{M}$ such that
$I(\vec{u}_{n})\rightarrow\inf_{\mathcal{M}}I$. We claim that
$\\{\vec{u}_{n}\\}$ is bounded in $H^{1}(\mathbb{R}^{3})\times
H^{1}(\mathbb{R}^{3})$. Denote

$\displaystyle a_{n}=\int_{\mathbb{R}^{3}}|\nabla u_{1,n}|^{2}+|\nabla
u_{2,n}|^{2}dx,\ \ b_{n}=\int_{\mathbb{R}^{3}}\lambda u_{1,n}^{2}+\lambda
u_{2,n}^{2}dx,$ (37) $\displaystyle
c_{n}=\int_{\mathbb{R}^{3}}\mu_{11}u_{1,n}^{2}\phi_{u_{1,n}}+\mu_{22}u_{2,n}^{2}\phi_{u_{2,n}}^{2}-2\mu_{12}u_{1,n}^{2}\phi_{u_{2,n}}dx,$
$\displaystyle
d_{n}=\int_{\mathbb{R}^{3}}\bigg{[}\frac{1}{2\pi}\int_{0}^{2\pi}\Big{(}u_{1,n}^{2}+2u_{1,n}u_{2,n}\cos\theta+u_{2,n}^{2}\Big{)}^{\frac{p+1}{2}}d\theta\bigg{]}dx.$

From (34), we get that $b_{n}$ and $d_{n}$ are bounded, and hence by (35),
$c_{n}$ is bounded. Thus, since $\vec{u}_{n}=(u_{1,n},u_{2,n})\in\mathcal{M}$
and $b_{n},c_{n},d_{n}$ are bounded, we have that $a_{n}$ is bounded.

We may assume that $\vec{u}_{n}\rightharpoonup\vec{u}=(u_{1},u_{2})$ in
$H^{1}(\mathbb{R}^{3})\times H^{1}(\mathbb{R}^{3})$ and
$\vec{u}_{n}\rightarrow\vec{u}$ in $L^{q}(\mathbb{R}^{3})\times
L^{q}(\mathbb{R}^{3})$, where $2<q<6$. Define

$\displaystyle a=\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}+|\nabla
u_{2}|^{2}dx,\ \ b=\int_{\mathbb{R}^{3}}\lambda u_{1}^{2}+\lambda
u_{2}^{2}dx,$ $\displaystyle
c=\int_{\mathbb{R}^{3}}\mu_{11}u_{1}^{2}\phi_{u_{1}}+\mu_{22}u_{2}^{2}\phi_{u_{2}}-2\mu_{12}u_{1}^{2}\phi_{u_{2}}dx,\
\
d=\int_{\mathbb{R}^{3}}\bigg{[}\frac{1}{2\pi}\int_{0}^{2\pi}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p+1}{2}}d\theta\bigg{]}dx$
$\displaystyle\bar{a}=\lim_{n\rightarrow\infty}a_{n},\ \
\bar{b}=\lim_{n\rightarrow\infty}b_{n},\ \
\bar{c}=\lim_{n\rightarrow\infty}c_{n},\ \
\bar{d}=\lim_{n\rightarrow\infty}d_{n},$

where $a_{n},b_{n},c_{n}$ and $d_{n}$ are given in (37). Then, by the
compactness of the embedding $H_{r}^{1}\rightarrow L^{q}$ and Lemma 4.3, where
$2<q<6$, we have $c=\bar{c}$ and $d=\bar{d}$. We claim that
$\vec{u}_{n}\rightarrow\vec{u}$ in $H^{1}(\mathbb{R}^{3})\times
H^{1}(\mathbb{R}^{3})$ and $\vec{u}\in\mathcal{M}$. To prove the claim, it
suffices to show that $a+b=\bar{a}+\bar{b}$. From the weak convergence, we
have $a\leq\bar{a}$ and $b\leq\bar{b}$. Suppose that
$\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}+\lambda
u_{1}^{2}dx<\lim_{n\rightarrow\infty}\int_{\mathbb{R}^{3}}|\nabla
u_{1,n}|^{2}+\lambda u_{1,n}^{2}dx$. This implies that $a+b<\bar{a}+\bar{b}$.
Since $I(\vec{u}_{n})\rightarrow\inf_{\mathcal{M}}I$ and $G(\vec{u}_{n})=0$,
we have

$\begin{cases}\frac{1}{2}\bar{a}+\frac{1}{2}\bar{b}+\frac{1}{4}\bar{c}-\frac{1}{p+1}\bar{d}=\inf_{\mathcal{M}}I,\\\
\frac{3}{2}\bar{a}+\frac{1}{2}\bar{b}+\frac{3}{4}\bar{c}-\frac{2p-1}{p+1}\bar{d}=0.\end{cases}$
(38)

By (31) and the second equation of (38),
$0<\frac{p+1}{2p-1}\Big{(}\frac{3}{2}\bar{a}+\frac{1}{2}\bar{b}\Big{)}=-\frac{3(p+1)}{4(2p-1)}\bar{c}+\bar{d}=-\frac{3(p+1)}{4(2p-1)}c+d$,
which implies that $\vec{u}\neq\vec{0}$. Then, by (29), we have $\bar{a}\geq
a>0,\bar{b}\geq b>0$ and $\bar{d}=d>0$. Define

$\displaystyle\bar{h}(t)=\frac{1}{2}\bar{a}t^{3}+\frac{1}{2}\bar{b}t+\frac{1}{4}\bar{c}t^{3}-\frac{1}{p+1}\bar{d}t^{2p-1}\
\ \mbox{ and }\ \
h(t)=\frac{1}{2}at^{3}+\frac{1}{2}bt+\frac{1}{4}ct^{3}-\frac{1}{p+1}dt^{2p-1}.$

By Lemma 3.4, both functions have a unique critical point, corresponding to
their maxima. From (38), the maximum of $\bar{h}$ is equal to
$\inf_{\mathcal{M}}I$ and is achieved at $t=1$. We observe that since
$c=\bar{c}$, $d=\bar{d}$, $a\leq\bar{a}$, $b\leq\bar{b}$ and
$a+b<\bar{a}+\bar{b}$, $h(t)<\bar{h}(t)$ for all $t>0$. Let $t_{0}>0$ be the
point such that the maximum of $h$ is achieved. Then $h^{\prime}(t_{0})=0$ and
$h(t_{0})<\max\bar{h}=\inf_{\mathcal{M}}I.$ Define
$\vec{v}_{0}(x)=(t_{0}^{2}u_{1}(t_{0}x),t_{0}^{2}u_{2}(t_{0}x)).$ Then we have

$\displaystyle I(\vec{v}_{0})$
$\displaystyle=\frac{1}{2}at_{0}^{3}+\frac{1}{2}bt_{0}+\frac{1}{4}ct_{0}^{3}-\frac{1}{p+1}dt_{0}^{2p-1}=h(t_{0})<\inf_{\mathcal{M}}I,$
$\displaystyle G(\vec{v}_{0})$
$\displaystyle=\frac{3}{2}at_{0}^{3}+\frac{1}{2}bt_{0}+\frac{3}{4}ct_{0}^{3}-\frac{2p-1}{p+1}dt_{0}^{2p-1}=t_{0}h^{\prime}(t_{0})=0.$

Thus, $\vec{v}_{0}\in\mathcal{M}$ and $I(\vec{v}_{0})<\inf_{\mathcal{M}}I$,
which is a contradiction. Thus, we have $\vec{u}_{n}\rightarrow\vec{u}$ in
$H^{1}(\mathbb{R}^{3})\times H^{1}(\mathbb{R}^{3})$ and
$\vec{u}\in\mathcal{M}$.

To prove the existence of a non-negative non-trivial critical point of $I$, we
claim that

$\displaystyle\int_{0}^{2\pi}\Big{(}a^{2}+2ab\cos\theta+b^{2}\Big{)}^{\frac{p+1}{2}}d\theta=\int_{0}^{2\pi}\Big{(}a^{2}+2|a||b|\cos\theta+b^{2}\Big{)}^{\frac{p+1}{2}}d\theta,$
(39)

where $a,b\in\mathbb{R}$. Indeed, since
$t^{2}-2t\cos\theta+1>0,t^{2}+2t\cos\theta+1>0$ for $|t|<1$, $0\leq\theta\leq
2\pi$ and

$\int_{0}^{2\pi}(2t\cos\theta+t^{2})^{n}d\theta=\int_{0}^{2\pi}(-2t\cos\theta+t^{2})^{n}d\theta,$

where $n\in\mathbb{N}$ and $|t|<1$, we have

$\displaystyle\int_{0}^{2\pi}(1+2t\cos\theta+t^{2})^{\frac{p+1}{2}}d\theta$
$\displaystyle=\sum_{n=0}^{\infty}\frac{h^{(n)}(0)}{n!}\int_{0}^{2\pi}(2t\cos\theta+t^{2})^{n}d\theta$
$\displaystyle=\sum_{n=0}^{\infty}\frac{h^{(n)}(0)}{n!}\int_{0}^{2\pi}(-2t\cos\theta+t^{2})^{n}d\theta$
$\displaystyle=\int_{0}^{2\pi}(1-2t\cos\theta+t^{2})^{\frac{p+1}{2}}d\theta,$

where $|t|<1$ and $h(r)=(1+r)^{\frac{p+1}{2}}$, and thus we deduce the claim
(39). By (39), if $\vec{u}=(u_{1},u_{2})$ is a minimizer of $I$ restricted to
$\mathcal{M}$, then so is $(|u_{1}|,|u_{2}|)$. Then, by Lemma 4.2, we see that
$(|u_{1}|,|u_{2}|)$ is a non-negative non-trivial critical point of $I$. ∎

###  4.2. Proof of Theorem 1.3

We prove the existence part in subsection 4.1. By Proposition 4.4, we see that
for $2<p<5$, there exists a non-trivial, non-negative ground state solution
$\vec{w}$ of (7). Moreover, note that by Lemma 4.1, $\vec{w}$ is a minimizer
for $I$ in $\mathcal{M}$, $\mathcal{M}$ is smooth in a neighborhood of
$\vec{w}$ and $\mathcal{M}$ is of codimension $1$ in a neighborhood of
$\vec{w}$, where $\mathcal{M}$ is given in (26). Hence, a non-negative ground
state solution $\vec{w}$ for (7) has Morse index less than equal to $1$.

For $\mu_{ij}>0$, let $W$ be a ground state solution of

$-\Delta u+\lambda
u+({\mu_{11}-\mu_{12}})\phi_{u}u=\frac{c_{p}}{2}|u|^{p-1}u,$ (40)

where $c_{p}$ is given in (21). The existence of a ground state solution to
(40) was proved in [33] when $\mu_{11}-\mu_{12}<0$ and [29, Theorem 3.2] when
$\mu_{11}-\mu_{12}>0$. Clearly, there exists a ground state to (40) when
$\mu_{11}=\mu_{12}$. We note that if $\mu_{11}=\mu_{22}$, $(W,W)$ is a
solution of (7), and thus $(W,W)\in\mathcal{M}$.

First, we assume that $\frac{1}{3}(-2+\sqrt{73})\leq p<5$ and $\mu_{ij}>0$,
where $i,j=1,2$. Then, by Proposition 3.3, a ground state solution of (7)
$\vec{w}$ is positive, where we used the fact that a non-negative ground state
solution $\vec{w}$ for (7) has Morse index less than equal to $1$.

Next, we assume that $2<p<5$ and $\mu_{11}=\mu_{22}$. Then, by Proposition 3.7
and the fact that $I(W,W)\geq I(\vec{w})$, we see that a ground state solution
$\vec{w}$ is positive. Moreover, by Proposition 3.8, we deduce that
$\vec{w}=(\tilde{W},\tilde{W})$, where $\tilde{W}$ is a positive ground state
solution of (40). $\Box$

##  5\. Power potential case with $1<p\leq 2$ and $\det(\mu_{ij})<0$

In this subsection, we prove Theorem 1.4, the existence of a positive solution
of (7) for $1<p\leq 2$ and large $\mu_{12}>0$. This can be considered as an
extreme case of $\det(\mu_{ij})<0$.

###  5.1. Proof of Theorem 1.4

Let $v_{i}(x)=\sqrt{\mu_{12}}u_{i}(x)$ for $i=1,2$. Then

$\displaystyle I_{+}(\vec{u})$
$\displaystyle=\frac{1}{2}\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}+|\nabla
u_{2}|^{2}+\lambda u_{1}^{2}+\lambda u_{2}^{2}dx$
$\displaystyle\quad+\frac{1}{4}\int_{\mathbb{R}^{3}}\mu_{11}(u_{1})_{+}^{2}\phi_{(u_{1})_{+}}+\mu_{22}(u_{2})_{+}^{2}\phi_{(u_{2})_{+}}-2\mu_{12}(u_{1})_{+}^{2}\phi_{(u_{2})_{+}}dx$
$\displaystyle\qquad-\frac{1}{p+1}\frac{1}{2\pi}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\Big{(}(u_{1})_{+}^{2}+2(u_{1})_{+}(u_{2})_{+}\cos\theta+(u_{2})_{+}^{2}\Big{)}^{\frac{p+1}{2}}d\theta
dx$
$\displaystyle=\mu_{12}^{-1}\bigg{[}\frac{1}{2}\int_{\mathbb{R}^{3}}|\nabla
v_{1}|^{2}+|\nabla v_{2}|^{2}+\lambda v_{1}^{2}+\lambda v_{2}^{2}dx$
$\displaystyle\qquad+\frac{1}{4\mu_{12}}\int_{\mathbb{R}^{3}}\mu_{11}(v_{1})_{+}^{2}\phi_{(v_{1})_{+}}+\mu_{22}(v_{2})_{+}^{2}\phi_{(v_{2})_{+}}-2\mu_{12}(v_{1})_{+}^{2}\phi_{(v_{2})_{+}}dx$
$\displaystyle\qquad-\frac{1}{(p+1)\mu_{12}^{\frac{p-1}{2}}}\frac{1}{2\pi}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\Big{(}(v_{1})_{+}^{2}+2(v_{1})_{+}(v_{2})_{+}\cos\theta+(v_{2})_{+}^{2}\Big{)}^{\frac{p+1}{2}}d\theta
dx\bigg{]}$
$\displaystyle=\mu_{12}^{-1}\Big{(}I_{0}(\vec{v})+H_{\mu_{12}}(\vec{v})\Big{)},$

where $\vec{u}=(u_{1},u_{2})$, $\vec{v}=(v_{1},v_{2})$,

$I_{0}(\vec{v})=\frac{1}{2}\int_{\mathbb{R}^{3}}|\nabla v_{1}|^{2}+|\nabla
v_{2}|^{2}+\lambda v_{1}^{2}+\lambda
v_{2}^{2}dx-\frac{1}{2}\int_{\mathbb{R}^{3}}(v_{1})_{+}^{2}\phi_{(v_{2})_{+}}dx$

and

$\displaystyle H_{\mu_{12}}(\vec{v})$
$\displaystyle=\frac{1}{4\mu_{12}}\int_{\mathbb{R}^{3}}\mu_{11}(v_{1})_{+}^{2}\phi_{(v_{1})_{+}}+\mu_{22}(v_{2})_{+}^{2}\phi_{(v_{2})_{+}}dx$
$\displaystyle\qquad-\frac{1}{2\pi(p+1)\mu_{12}^{\frac{p-1}{2}}}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\Big{(}(v_{1})_{+}^{2}+2(v_{1})_{+}(v_{2})_{+}\cos\theta+(v_{2})_{+}^{2}\Big{)}^{\frac{p+1}{2}}d\theta
dx.$

By applying the result in [15, Theorem 1], we try to find a critical point
$\vec{v}$ of $I_{0}+H_{\mu_{12}}$. Indeed, we observe that by (75), Lemma 4.3,
the compact embeddings $H^{1}_{r}(\mathbb{R}^{3})\subset
L^{q}(\mathbb{R}^{3})$, where $2<q<6$ and the fact that

$\displaystyle H_{\mu_{12}}^{\prime}(\vec{v})\vec{\psi}$
$\displaystyle=\frac{1}{\mu_{12}}\int_{\mathbb{R}^{3}}\mu_{11}(v_{1})_{+}\psi_{1}\phi_{(v_{1})_{+}}+\mu_{22}(v_{2})_{+}\psi_{2}\phi_{(v_{2})_{+}}dx$
$\displaystyle-\frac{1}{2\pi\mu_{12}^{\frac{p-1}{2}}}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\Big{(}(v_{1})_{+}^{2}+2(v_{1})_{+}(v_{2})_{+}\cos\theta+(v_{2})_{+}^{2}\Big{)}^{\frac{p-1}{2}}$
$\displaystyle\qquad\times\left(\Big{(}(v_{1})_{+}+1_{\\{v_{1}>0\\}}(v_{2})_{+}\cos\theta\Big{)}\psi_{1}+\Big{(}(v_{2})_{+}+1_{\\{v_{2}>0\\}}(v_{1})_{+}\cos\theta\Big{)}\psi_{2}\right)d\theta
dx,$

where $\vec{\psi}=(\psi_{1},\psi_{2})$, we see that $H_{\mu_{12}}:{\bf
H}\rightarrow\mathbb{R}$ is a $C^{1}$ functional such that $H_{\mu_{12}}:{\bf
H}\rightarrow\mathbb{R}$, $H_{\mu_{12}}^{\prime}:{\bf H}\rightarrow{\bf
H}^{-1}$ are compact, and for any $M>0$,

$\lim_{\mu_{12}\rightarrow\infty}\sup_{\|\vec{v}\|_{\bf{H}}\leq
M}|H_{\mu_{12}}(\vec{v})|=\lim_{\mu_{12}\rightarrow\infty}\sup_{\|\vec{v}\|_{\bf{H}}\leq
M}\|H_{\mu_{12}}^{\prime}(\vec{v})\|_{{\bf H}^{-1}}=0.$

Also, note that $I_{0}$ satisfies the Palais-Smale condition and the following
properties:

  1. (1)

there exist $c,r>0$ such that if $\|\vec{u}\|_{\bf{H}}=r$, then
$I_{0}(\vec{u})\geq c$, and there exists $\vec{w}\in{\bf H}$ such that
$\|\vec{w}\|_{\bf{H}}>r$ and $I_{0}(\vec{w})<0$;

  2. (2)

$\min_{\gamma\in\Gamma}\max_{s\in[0,1]}I_{0}(\gamma(s))=\inf\\{I_{0}(\vec{v})\
|\ \vec{v}\in{\bf H}\setminus\\{\vec{0}\\},I_{0}^{\prime}(\vec{v})=0\\},$

where $\Gamma=\\{\gamma\in C([0,1],{\bf H})\ |\
\gamma(0)=0,\gamma(1)=\vec{w}\\}$;

  3. (3)

there exists a curve $\gamma_{0}(s)\in\Gamma$ passing throungh $\vec{v}_{0}$
at $s=s_{0}$ and satisfying

$I_{0}(\vec{v}_{0})>I_{0}(\gamma_{0}(s))\mbox{ for all }s\neq s_{0}.$

Indeed, using the compact embeddings $H^{1}_{r}(\mathbb{R}^{3})\subset
L^{q}(\mathbb{R}^{3})$, where $2<q<6$, we can prove that the functional
$I_{0}$ satisfies the Palais-Smale condition, that is, for any sequence
$\\{\vec{u}_{n}\\}\subset{\bf H}$ satisfying

$|I_{0}(\vec{u}_{n})|\leq C\mbox{ uniformly in }n,\mbox{ and
}\|I_{0}^{\prime}(\vec{u}_{n})\|_{{\bf H}^{-1}}\rightarrow 0\mbox{ as
}n\rightarrow\infty,$

$\\{\vec{u}_{n}\\}$ has a strong convergence subsequence in
$H^{1}(\mathbb{R}^{3})\times H^{1}(\mathbb{R}^{3})$.

For $w\in C_{c}^{\infty}(\mathbb{R}^{3})\setminus\\{0\\}$, $w>0$ and for large
$t>0$,

$I_{0}(tw,tw)=t^{2}\int_{\mathbb{R}^{3}}|\nabla w|^{2}+\lambda
w^{2}dx-\frac{t^{4}}{2}\int_{\mathbb{R}^{3}}w^{2}\phi_{w}dx<0$

and for some $C_{1}>0$,

$\displaystyle I_{0}(\vec{u})$
$\displaystyle\geq\frac{1}{2}(\|u_{1}\|_{\lambda}^{2}+\|u_{2}\|_{\lambda}^{2})-\frac{1}{2}\|\phi_{u_{2}}\|_{L^{6}}\|u_{1}\|_{L^{\frac{12}{5}}}^{2}\geq\frac{1}{2}(\|u_{1}\|_{\lambda}^{2}+\|u_{2}\|_{\lambda}^{2})-C_{1}\|u_{2}\|_{\lambda}^{2}\|u_{1}\|_{\lambda}^{2}$
$\displaystyle\geq\frac{1}{2}\Big{[}\|u_{1}\|_{\lambda}^{2}(1-C_{1}\|u_{1}\|_{\lambda}^{2})+\|u_{2}\|_{\lambda}^{2}(1-C_{1}\|u_{2}\|_{\lambda}^{2})\Big{]},$

which imply that there exist $c,r>0$ such that if $\|\vec{u}\|_{{\bf H}}=r$,
then $I_{0}(\vec{u})\geq c>0$ and there exists $(w,w)\in{\bf H}$ such that
$\|(w,w)\|_{\bf{H}}>r$ and $I_{0}(w,w)<0$. This proves $(1)$. By the mountain
pass theorem [3], we see that there exists a critical point
$\vec{v}_{0}\in{\bf H}\setminus\\{0\\}$ of $I_{0}$ such that

$I_{0}(\vec{v}_{0})=\inf_{\gamma\in\Gamma}\max_{s\in[0,1]}I_{0}(\gamma(s))>0,$

where $\Gamma=\\{\gamma\in C([0,1],{\bf H})\ |\
\gamma(0)=0,\gamma(1)=(w,w)\\}$.

On the other hand, let $\vec{v}\in{\bf H}\setminus\\{0\\}$ and
$I_{0}^{\prime}(\vec{v})=0.$ By a strong maximum principle and the fact that
$\vec{v}=(v_{1},v_{2})$ satisfies

$\begin{cases}-\Delta v_{1}+\lambda v_{1}-\phi_{(v_{2})_{+}}(v_{1})_{+}=0\ \
\mbox{ in }\ \ \mathbb{R}^{3},\\\ -\Delta v_{2}+\lambda
v_{2}-\phi_{(v_{1})_{+}}(v_{2})_{+}=0\ \ \mbox{ in }\ \
\mathbb{R}^{3},\end{cases}$

we see that $\vec{v}$ is positive. Then, by Lemma 2.1, $\vec{v}=(V,V)$, where
$V\in H_{r}^{1}$ satisfies $V>0$ and $-\Delta V+\lambda V-\phi_{V}V=0$.
Moreover, from [19, Lemma 9], we see that $V$ is a unique radial, positive
solution of $-\Delta w+\lambda w-\phi_{w}w=0$. Thus, we see that
$\vec{v}_{0}=(V,V)$,

$I_{0}(\vec{v}_{0})=\inf\\{I_{0}(\vec{u})\ |\ \vec{u}\in{\bf
H}\setminus\\{0\\},I_{0}^{\prime}(\vec{u})=0\\}$

and for all $s\neq 1$,

$I_{0}(\vec{v}_{0})>I_{0}(\gamma_{0}(s))=s^{2}\int_{\mathbb{R}^{3}}|\nabla
V|^{2}+\lambda
V^{2}dx-\frac{s^{4}}{2}\int_{\mathbb{R}^{3}}V^{2}\phi_{V}dx=\left(s^{2}-\frac{s^{4}}{2}\right)\int_{\mathbb{R}^{3}}|\nabla
V|^{2}+\lambda V^{2}dx,$

where $\gamma_{0}(s)=(sV,sV)$. These prove $(2)$ and $(3)$. Thus, by [15,
Theorem 1], we can find a critical point $\vec{v}_{\mu_{12}}$ of
$I_{0}+H_{\mu_{12}}$ such that
$\vec{v}_{\mu_{12}}\rightarrow\vec{v}_{0}=(V,V)$ in
$H^{1}(\mathbb{R}^{3})\times H^{1}(\mathbb{R}^{3})$ as
$\mu_{12}\rightarrow\infty$. Then from this, (75), a strong maximum principle
and the fact that $\vec{v}_{\mu_{12}}=(v_{1,\mu_{12}},v_{2,\mu_{12}})$
satisfies

$\begin{cases}-\Delta v_{1}+\lambda
v_{1}+\frac{1}{\mu_{12}}(\mu_{11}\phi_{(v_{1})_{+}}-\mu_{12}\phi_{(v_{2})_{+}})(v_{1})_{+}\\\
\ \
=\frac{1}{2\pi\mu_{12}^{\frac{p-1}{2}}}\int_{0}^{2\pi}\Big{(}(v_{1})_{+}^{2}+(v_{2})_{+}^{2}+2(v_{1})_{+}(v_{2})_{+}\cos\theta\Big{)}^{\frac{p-1}{2}}\Big{(}(v_{1})_{+}+1_{\\{v_{1}>0\\}}(v_{2})_{+}\cos\theta\Big{)}d\theta\
\mbox{ in }\ \mathbb{R}^{3},\\\ -\Delta v_{2}+\lambda
v_{2}+\frac{1}{\mu_{12}}(\mu_{22}\phi_{(v_{2})_{+}}-\mu_{12}\phi_{(v_{1})_{+}})(v_{2})_{+}\\\
\ \
=\frac{1}{2\pi\mu_{12}^{\frac{p-1}{2}}}\int_{0}^{2\pi}\Big{(}(v_{1})_{+}^{2}+(v_{2})_{+}^{2}+2(v_{1})_{+}(v_{2})_{+}\cos\theta\Big{)}^{\frac{p-1}{2}}\Big{(}(v_{2})_{+}+1_{\\{v_{2}>0\\}}(v_{1})_{+}\cos\theta\Big{)}d\theta\
\mbox{ in }\ \mathbb{R}^{3}.\end{cases}$

we see that for large $\mu_{12}$, $\vec{v}_{\mu_{12}}$ is positive. Thus, we
deduce that for large $\mu_{12}$,
$(\mu_{12})^{-\frac{1}{2}}\vec{v}_{\mu_{12}}$ is a positive critical point of
$I$.

##  6\. Power potential case with $1<p<2$ and $\det(\mu_{ij})>0$

In this section, we prove Theorem 1.5 and Theorem 1.6, which give the non-
existence result to (7) when $\mu_{11}$ and $\mu_{22}$ are large, and the
existence result of two positive solutions to (7) when $\det(\mu_{ij})>0$ and
$\mu_{ij}$ is small, respectively.

###  6.1.  Triviality of $\vec{u}$ when $\mu_{11}$ and $\mu_{22}$ are large

We first prove that any solution $\vec{u}$ to (7) is trivial when $1<p\leq 2$,
$\lambda\geq 2$, $\mu_{ij}>0$, $\mu_{11}>4$ and
$(\mu_{11}-4)(\mu_{22}-4)>\mu_{12}^{2}.$

Proof of Theorem 1.5. Suppose that $\vec{u}=(u_{1},u_{2})\in
H^{1}(\mathbb{R}^{3})\times H^{1}(\mathbb{R}^{3})$ is a solution of (7). Then
we have

$\displaystyle I^{\prime}(\vec{u})\vec{u}$
$\displaystyle=\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}+|\nabla
u_{2}|^{2}+\lambda u_{1}^{2}+\lambda
u_{2}^{2}dx+\int_{\mathbb{R}^{3}}\mu_{11}u_{1}^{2}\phi_{u_{1}}+\mu_{22}u_{2}^{2}\phi_{u_{2}}-2\mu_{12}u_{1}^{2}\phi_{u_{2}}dx$
$\displaystyle\qquad-\frac{1}{2\pi}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p+1}{2}}d\theta
dx=0.$

We note that for $a,b\in\mathbb{R}$,

$\frac{1}{2\pi}\int_{0}^{2\pi}\Big{(}a^{2}+2ab\cos\theta+b^{2}\Big{)}^{\frac{p+1}{2}}d\theta\leq(|a|+|b|)^{p+1}\leq
2^{p}(|a|^{p+1}+|b|^{p+1})$

and

$4\int_{\mathbb{R}^{3}}|u|^{3}dx=4\int_{\mathbb{R}^{3}}(-\Delta\phi_{u})|u|dx=4\int_{\mathbb{R}^{3}}\nabla\phi_{u}\cdot\nabla|u|dx\leq\int_{\mathbb{R}^{3}}|\nabla
u|^{2}+4|\nabla\phi_{u}|^{2}dx.$

Then we see that if $\mu_{11}>4$ and
$\Big{(}\mu_{11}-4\Big{)}\Big{(}\mu_{22}-4\Big{)}>\mu_{12}^{2},$

$\displaystyle 0=I^{\prime}(\vec{u})\vec{u}$
$\displaystyle\geq\int_{\mathbb{R}^{3}}\lambda u_{1}^{2}+\lambda
u_{2}^{2}dx+\int_{\mathbb{R}^{3}}\Big{(}\mu_{11}-4\Big{)}u_{1}^{2}\phi_{u_{1}}+\Big{(}\mu_{22}-4\Big{)}u_{2}^{2}\phi_{u_{2}}-2\mu_{12}u_{1}^{2}\phi_{u_{2}}dx$
$\displaystyle\quad+4\int_{\mathbb{R}^{3}}|u_{1}|^{3}+|u_{2}|^{3}dx-2^{p}\int_{\mathbb{R}^{3}}|u_{1}|^{p+1}+|u_{2}|^{p+1}dx$
$\displaystyle\geq\int_{\mathbb{R}^{3}}\lambda u_{1}^{2}+\lambda
u_{2}^{2}dx+4\int_{\mathbb{R}^{3}}|u_{1}|^{3}+|u_{2}|^{3}dx-2^{p}\int_{\mathbb{R}^{3}}|u_{1}|^{p+1}+|u_{2}|^{p+1}dx.$

To complete the proof, we claim that if $\lambda\geq 2$ and $1<p\leq 2$, then
$h(t)\geq 0$ for $t\geq 0$, where $h:\mathbb{R}^{+}\rightarrow\mathbb{R}$ is
given by $h(t)=\lambda+4t-2^{p}t^{p-1}$. We observe that

$h^{\prime}(t)\begin{cases}<0&\mbox{ if
}t<\frac{1}{2}(p-1)^{\frac{1}{2-p}},\\\ >0&\mbox{ if
}t>\frac{1}{2}(p-1)^{\frac{1}{2-p}},\\\ =0&\mbox{ if
}t=\frac{1}{2}(p-1)^{\frac{1}{2-p}},\end{cases}$ $h(t)\geq 0\ \ \mbox{ for
}t\geq 0\ \ \mbox{ if }\ \ \lambda\geq
2(p-1)^{\frac{p-1}{2-p}}-2(p-1)^{\frac{1}{2-p}}=2(p-1)^{\frac{p-1}{2-p}}(2-p),$

and for $1<p\leq 2$, $(p-1)^{\frac{p-1}{2-p}}(2-p)<(p-1)^{\frac{p-1}{2-p}}\leq
1$. From these, we can prove the claim, and thus, $(u_{1},u_{2})=(0,0)$ if
$\lambda\geq 2$ and $1<p\leq 2$. $\Box$

###  6.2. Construction of two positive solutions

Following the idea in [29], we study the existence of two positive solutions
of (7) for $1<p<2$ and $\mu_{11}\mu_{22}-\mu_{12}^{2}>0$.

######  Proposition 6.1.

Let $\lambda,\mu_{ij}>0$ for $i,j=1,2$. Assume $1<p<2$ and
$\mu_{11}\mu_{22}-\mu_{12}^{2}>0$. Then we have

  1. (i)

$\inf_{\vec{u}\in{\bf H}}I(\vec{u})>-\infty$;

  2. (ii)

$I$ satisfies (PS) condition.

###### Proof.

We first claim that $\inf_{\vec{u}\in{\bf H}}I(\vec{u})>-\infty.$ We note that
for $a,b\in\mathbb{R}$,

$\frac{1}{2\pi}\int_{0}^{2\pi}\Big{(}a^{2}+2ab\cos\theta+b^{2}\Big{)}^{\frac{p+1}{2}}d\theta\leq(|a|+|b|)^{p+1}\leq
2^{p+1}(|a|^{p+1}+|b|^{p+1}),$

and for some $0<\tilde{\mu}_{11}<\mu_{11}$, $0<\tilde{\mu}_{22}<\mu_{22}$,
$\mu_{12}<\tilde{\mu}_{12}$, we have
$\tilde{\mu}_{11}\tilde{\mu}_{22}-\tilde{\mu}_{12}^{2}>0$. Then we have

$\displaystyle I(\vec{u})$
$\displaystyle\geq\frac{1}{2}\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}+|\nabla
u_{2}|^{2}+\lambda u_{1}^{2}+\lambda u_{2}^{2}dx$
$\displaystyle\quad+\frac{1}{4}\int_{\mathbb{R}^{3}}(\mu_{11}-\tilde{\mu}_{11})u_{1}^{2}\phi_{u_{1}}+(\mu_{22}-\tilde{\mu}_{22})u_{2}^{2}\phi_{u_{2}}+2(\tilde{\mu}_{12}-\mu_{12})u_{1}^{2}\phi_{u_{2}}dx$
$\displaystyle\quad+\frac{1}{4}\int_{\mathbb{R}^{3}}\tilde{\mu}_{11}u_{1}^{2}\phi_{u_{1}}+\tilde{\mu}_{22}u_{2}^{2}\phi_{u_{2}}-2\tilde{\mu}_{12}u_{1}^{2}\phi_{u_{2}}dx-\frac{2^{p+1}}{p+1}\int_{\mathbb{R}^{3}}|u_{1}|^{p+1}+|u_{2}|^{p+1}dx$
$\displaystyle\geq\frac{1}{2}\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}+|\nabla
u_{2}|^{2}+\lambda u_{1}^{2}+\lambda
u_{2}^{2}dx+\frac{\hat{\mu}}{4}\int_{\mathbb{R}^{3}}u_{1}^{2}\phi_{u_{1}}+u_{2}^{2}\phi_{u_{2}}+u_{1}^{2}\phi_{u_{2}}dx$
$\displaystyle\quad-\frac{2^{p+1}}{p+1}\int_{\mathbb{R}^{3}}|u_{1}|^{p+1}+|u_{2}|^{p+1}dx,$

where
$\hat{\mu}\in\Big{(}0,\min\\{\mu_{11}-\tilde{\mu}_{11},\mu_{22}-\tilde{\mu}_{22},\tilde{\mu}_{12}-\mu_{12}\\}\Big{)}.$
From this and the fact that

$\sqrt{\frac{\hat{\mu}}{8}}\int_{\mathbb{R}^{3}}|u|^{3}dx=\sqrt{\frac{\hat{\mu}}{8}}\int_{\mathbb{R}^{3}}(-\Delta\phi_{u})|u|dx=\sqrt{\frac{\hat{\mu}}{8}}\int_{\mathbb{R}^{3}}\nabla\phi_{u}\cdot\nabla|u|dx\leq\int_{\mathbb{R}^{3}}\frac{1}{4}|\nabla
u|^{2}+\frac{\hat{\mu}}{8}|\nabla\phi_{u}|^{2}dx,$

we obtain

$\displaystyle I(\vec{u})$
$\displaystyle\geq\frac{1}{4}\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}+|\nabla
u_{2}|^{2}dx+\frac{\lambda}{2}\int_{\mathbb{R}^{3}}u_{1}^{2}+u_{2}^{2}dx+\frac{\hat{\mu}}{8}\int_{\mathbb{R}^{3}}u_{1}^{2}\phi_{u_{1}}+u_{2}^{2}\phi_{u_{2}}+2u_{1}^{2}\phi_{u_{2}}dx$
(41)
$\displaystyle\quad+\sqrt{\frac{\hat{\mu}}{8}}\int_{\mathbb{R}^{3}}|u_{1}|^{3}+|u_{2}|^{3}dx-\frac{2^{p+1}}{p+1}\int_{\mathbb{R}^{3}}|u_{1}|^{p+1}+|u_{2}|^{p+1}dx.$

Define

$h:\mathbb{R}^{+}\times\mathbb{R}^{+}\rightarrow\mathbb{R},\ \
h(s,t)=\frac{\lambda}{4}(s^{2}+t^{2})+\sqrt{\frac{\hat{\mu}}{8}}(s^{3}+t^{3})-\frac{2^{p+1}}{p+1}(s^{p+1}+t^{p+1}),$

where $p\in(1,2)$. We claim that

$h\mbox{ is positive for }\\{(s,t)\in\mathbb{R}^{+}\times\mathbb{R}^{+}\ |\
s^{2}+t^{2}<r_{0}\mbox{ or }s^{2}+t^{2}>R_{0}\\},$ (42)

where $r_{0},R_{0}>0$ are constants. Indeed, note that if
$(s,t)\in\\{(s,t)\in\mathbb{R}^{+}\times\mathbb{R}^{+}\ |\
s^{2}+t^{2}<r_{0}\\}$,
$\frac{s^{p+1}+t^{p+1}}{s^{2}+t^{2}}<r_{0}^{\frac{p-1}{2}}$ and if
$(s,t)\in\\{(s,t)\in\mathbb{R}^{+}\times\mathbb{R}^{+}\ |\ s<t,\
s^{2}+t^{2}>R_{0}\\},$
$\frac{s^{p+1}+t^{p+1}}{s^{3}+t^{3}}\leq\frac{2t^{p+1}}{t^{3}}<2\Big{(}\frac{2}{R_{0}}\Big{)}^{\frac{2-p}{2}}.$
These imply (42).

Define $m=\inf_{s>0,t>0}h(s,t)$. If $m\geq 0$, we are done, beacuse
$I(\vec{u})\geq 0$, and thus we assume $m<0$. Then, by (41) and (42), we have

$\\{(s,t)\in\mathbb{R}^{+}\times\mathbb{R}^{+}\ |\
h(s,t)<0\\}\subset\\{(s,t)\in\mathbb{R}^{+}\times\mathbb{R}^{+}\ |\
r_{0}<s^{2}+t^{2}<R_{0}\\}$

and

$\displaystyle I(\vec{u})$
$\displaystyle\geq\frac{1}{4}\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}+|\nabla
u_{2}|^{2}dx+\frac{\lambda}{4}\int_{\mathbb{R}^{3}}u_{1}^{2}+u_{2}^{2}dx+\frac{\hat{\mu}}{8}\int_{\mathbb{R}^{3}}u_{1}^{2}\phi_{u_{1}}+u_{2}^{2}\phi_{u_{2}}+2u_{1}^{2}\phi_{u_{2}}dx$
(43) $\displaystyle\quad+\int_{\mathbb{R}^{3}}h(|u_{1}|,|u_{2}|)dx$
$\displaystyle\geq\frac{1}{4}\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}+|\nabla
u_{2}|^{2}dx+\frac{\lambda}{4}\int_{\mathbb{R}^{3}}u_{1}^{2}+u_{2}^{2}dx+\frac{\hat{\mu}}{8}\int_{\mathbb{R}^{3}}u_{1}^{2}\phi_{u_{1}}+u_{2}^{2}\phi_{u_{2}}+2u_{1}^{2}\phi_{u_{2}}dx$
$\displaystyle\quad+m\Big{|}\\{x\in\mathbb{R}^{3}\ |\
h(|u_{1}(x)|,|u_{2}(x)|)<0\\}\Big{|}$
$\displaystyle\geq\frac{1}{4}\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}+|\nabla
u_{2}|^{2}dx+\frac{\lambda}{4}\int_{\mathbb{R}^{3}}u_{1}^{2}+u_{2}^{2}dx+\frac{\hat{\mu}}{8}\int_{\mathbb{R}^{3}}u_{1}^{2}\phi_{u_{1}}+u_{2}^{2}\phi_{u_{2}}+2u_{1}^{2}\phi_{u_{2}}dx+m|A|,$

where $A=\\{x\in\mathbb{R}^{3}\ |\ r_{0}<u_{1}^{2}(x)+u_{2}^{2}(x)<R_{0}\\}.$
Note that $A$ is spherically symmetric.

Suppose to the contrary that there exists $\\{\vec{u}_{n}\\}\subset{\bf H}$
such that $I(\vec{u}_{n})\rightarrow-\infty$. Clearly,
$\|u_{1,n}\|_{\lambda}+\|u_{2,n}\|_{\lambda}\rightarrow\infty$ as
$n\rightarrow\infty$. For each $\vec{u}_{n}=(u_{1,n},u_{2,n})$, define
$A_{n}=\\{x\in\mathbb{R}^{3}\ |\ r_{0}<u_{1,n}^{2}(x)+u_{2,n}^{2}(x)<R_{0}\\}$
and $\rho_{n}=\sup\\{|x|\ |\ x\in A_{n}\\}$. By (43) and the fact that
$I(\vec{u}_{n})<0$, we have

$|m||A_{n}|\geq\frac{1}{4}(\|u_{1,n}\|_{\lambda}^{2}+\|u_{2,n}\|_{\lambda}^{2}),$

which implies that $|A_{n}|\rightarrow\infty$ as $n\rightarrow\infty$. Since

$|u(x)|\leq C|x|^{-1}\|u\|_{H^{1}}\mbox{ for }u\in H_{r}^{1},$

where $C>0$ (see [31]), we see that for $x\in\mathbb{R}^{3}$ with
$|x|=\rho_{n}$, $u_{1,n}^{2}(x)+u_{2,n}^{2}(x)=r_{0}$ and

$r_{0}=u_{1,n}^{2}(x)+u_{2,n}^{2}(x)\leq
C^{2}\rho_{n}^{-2}(\|u_{1,n}\|_{H^{1}}^{2}+\|u_{2,n}\|_{H^{1}}^{2})\leq
C_{1}\rho_{n}^{-2}|m||A_{n}|,$

which implies that $\rho_{n}\leq C_{2}|A_{n}|^{\frac{1}{2}},$ where
$C_{1},C_{2}>0$.

On the other hand, by (43) and the fact that $I(\vec{u}_{n})<0$, we have

$\frac{\hat{\mu}}{8}\int_{\mathbb{R}^{3}}u_{1,n}^{2}\phi_{u_{1,n}}+u_{2,n}^{2}\phi_{u_{2,n}}+2u_{1,n}^{2}\phi_{u_{2,n}}dx\leq|m||A|.$

Then from this and the facts that $|x-y|\leq 2\rho_{n}\mbox{ for }x,y\in
A_{n}$ and

$\displaystyle\int_{\mathbb{R}^{3}}u_{1,n}^{2}\phi_{u_{1,n}}+u_{2,n}^{2}\phi_{u_{2,n}}+2u_{1,n}^{2}\phi_{u_{2,n}}dx$
$\displaystyle=\int_{\mathbb{R}^{3}}\int_{\mathbb{R}^{3}}\frac{u_{1,n}^{2}(x)u_{1,n}^{2}(y)+u_{2,n}^{2}(x)u_{2,n}^{2}(y)+2u_{1,n}^{2}(x)u_{2,n}^{2}(y)}{|x-y|}dxdy$
$\displaystyle=\int_{\mathbb{R}^{3}}\int_{\mathbb{R}^{3}}\frac{\Big{(}u_{1,n}^{2}(x)+u_{2,n}^{2}(x)\Big{)}\Big{(}u_{1,n}^{2}(y)+u_{2,n}^{2}(y)\Big{)}}{|x-y|}dxdy,$

we have

$\displaystyle\frac{8}{\hat{\mu}}|m||A_{n}|$
$\displaystyle\geq\int_{\mathbb{R}^{3}}\int_{\mathbb{R}^{3}}\frac{\Big{(}u_{1,n}^{2}(x)+u_{2,n}^{2}(x)\Big{)}\Big{(}u_{1,n}^{2}(y)+u_{2,n}^{2}(y)\Big{)}}{|x-y|}dxdy$
$\displaystyle\geq\int_{A_{n}}\int_{A_{n}}\frac{\Big{(}u_{1,n}^{2}(x)+u_{2,n}^{2}(x)\Big{)}\Big{(}u_{1,n}^{2}(y)+u_{2,n}^{2}(y)\Big{)}}{|x-y|}dxdy\geq
r_{0}^{2}\frac{|A_{n}|^{2}}{2\rho_{n}}.$

This implies that $|A_{n}|\leq C_{3}\rho_{n}$, where $C_{3}>0$, which is a
contradiction to the facts that $|A_{n}|\rightarrow\infty$ as
$n\rightarrow\infty$ and $\rho_{n}\leq C_{2}|A_{n}|^{\frac{1}{2}}.$ Thus, we
have $\inf_{\vec{u}\in{\bf H}}I(\vec{u})>-\infty.$

Next, we prove that $I$ satisfies (PS) condition. Let
$\vec{u}_{n}=(u_{1,n},u_{2,n})$ be a sequence in ${\bf H}$ such that
$I^{\prime}(\vec{u}_{n})\rightarrow 0$ as $n\rightarrow\infty$. Then we have

$\displaystyle\|u_{1,n}\|_{\lambda}+\|u_{2,n}\|_{\lambda}$ $\displaystyle\geq
I^{\prime}(\vec{u}_{n})\vec{u}_{n}$
$\displaystyle=\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}+|\nabla
u_{2}|^{2}+\lambda u_{1}^{2}+\lambda
u_{2}^{2}dx+\int_{\mathbb{R}^{3}}\mu_{11}u_{1}^{2}\phi_{u_{1}}+\mu_{22}u_{2}^{2}\phi_{u_{2}}-2\mu_{12}u_{1}^{2}\phi_{u_{2}}dx$
$\displaystyle\qquad-\frac{1}{2\pi}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p+1}{2}}d\theta
dx.$

Following the first part of the proof (41), we deduce that for some
$\bar{\mu}>0$,

$\displaystyle\|u_{1,n}\|_{\lambda}+\|u_{2,n}\|_{\lambda}$
$\displaystyle\geq\frac{1}{2}\int_{\mathbb{R}^{3}}|\nabla u_{1,n}|^{2}+|\nabla
u_{2,n}|^{2}+\lambda u_{1,n}^{2}+\lambda u_{2,n}^{2}dx$ (44)
$\displaystyle\quad+\frac{\bar{\mu}}{2}\int_{\mathbb{R}^{3}}u_{1,n}^{2}\phi_{u_{1,n}}+u_{2,n}^{2}\phi_{u_{2,n}}+2u_{1,n}^{2}\phi_{u_{2,n}}dx+\int_{\mathbb{R}^{3}}g(|u_{1,n}|,|u_{2,n}|)dx,$

where
$g(s,t)=\frac{1}{2}\lambda(s^{2}+t^{2})+\sqrt{\bar{\mu}}(s^{3}+t^{3})-2^{p+1}(s^{p+1}+t^{p+1})$.
If $\|u_{1,n}\|_{\lambda}+\|u_{2,n}\|_{\lambda}$ is unbounded in $n$, then by
(44), for large $n$, we have

$\displaystyle\frac{1}{3}\int_{\mathbb{R}^{3}}|\nabla u_{1,n}|^{2}+|\nabla
u_{2,n}|^{2}+\lambda u_{1,n}^{2}+\lambda
u_{2,n}^{2}dx+\frac{\bar{\mu}}{2}\int_{\mathbb{R}^{3}}u_{1,n}^{2}\phi_{u_{1,n}}+u_{2,n}^{2}\phi_{u_{2,n}}+2u_{1,n}^{2}\phi_{u_{2,n}}dx$
$\displaystyle\quad+\int_{\mathbb{R}^{3}}g(|u_{1,n}|,|u_{2,n}|)dx<0.$

Then, by the same arguments in (43) below, we deduce a contradiction. Since
$\|u_{1,n}\|_{\lambda}+\|u_{2,n}\|_{\lambda}$ is bounded, we assume that
$\vec{u}_{n}\rightharpoonup\vec{u}=(u_{1},u_{2})$ in
$H^{1}(\mathbb{R}^{3})\times H^{1}(\mathbb{R}^{3})$ and
$\vec{u}_{n}\rightharpoonup\vec{u}$ in $L^{q}(\mathbb{R}^{3})\times
L^{q}(\mathbb{R}^{3})$, where $2<q<6$. From this and Lemma 4.3, we see that

$\displaystyle\lim_{n\rightarrow\infty}I^{\prime}(\vec{u}_{n})\vec{u}$
$\displaystyle=\lim_{n\rightarrow\infty}\bigg{[}\int_{\mathbb{R}^{3}}\nabla
u_{1,n}\cdot\nabla u_{1}+\nabla u_{2,n}\cdot\nabla u_{2}+\lambda
u_{1,n}u_{1}+\lambda u_{2,n}u_{2}dx$
$\displaystyle\quad+\int_{\mathbb{R}^{3}}\mu_{11}u_{1,n}\phi_{u_{1,n}}u_{1}+\mu_{22}u_{2,n}\phi_{u_{2,n}}u_{2}-\mu_{12}u_{1,n}\phi_{u_{2,n}}u_{1}-\mu_{12}u_{2,n}\phi_{u_{1,n}}u_{2}dx$
$\displaystyle-\frac{1}{2\pi}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\Big{(}(u_{1,n}+u_{2,n}\cos\theta)u_{1}+(u_{2,n}+u_{1,n}\cos\theta)u_{2}\Big{)}$
$\displaystyle\quad\times\Big{(}u_{1,n}^{2}+2u_{1,n}u_{2,n}\cos\theta+u_{2,n}^{2}\Big{)}^{\frac{p-1}{2}}d\theta
dx\bigg{]}$ $\displaystyle=\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}+|\nabla
u_{2}|^{2}+\lambda u_{1}^{2}+\lambda
u_{2}^{2}dx+\int_{\mathbb{R}^{3}}\mu_{11}u_{1}^{2}\phi_{u_{1}}+\mu_{22}u_{2}^{2}\phi_{u_{2}}-2\mu_{12}u_{1}^{2}\phi_{u_{2}}dx$
$\displaystyle\quad-\frac{1}{2\pi}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p+1}{2}}d\theta
dx=0$

and

$\displaystyle\lim_{n\rightarrow\infty}I^{\prime}(\vec{u}_{n})\vec{u}_{n}$
$\displaystyle=\lim_{n\rightarrow\infty}\bigg{[}\int_{\mathbb{R}^{3}}|\nabla
u_{1,n}|^{2}+|\nabla u_{2,n}|^{2}+\lambda u_{1,n}^{2}+\lambda u_{2,n}^{2}dx$
$\displaystyle\quad+\int_{\mathbb{R}^{3}}\mu_{11}u_{1,n}^{2}\phi_{u_{1,n}}+\mu_{22}u_{2,n}^{2}\phi_{u_{2,n}}-2\mu_{12}u_{1,n}^{2}\phi_{u_{2,n}}dx$
$\displaystyle\quad-\frac{1}{2\pi}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\Big{(}u_{1,n}^{2}+2u_{1,n}u_{2,n}\cos\theta+u_{2,n}^{2}\Big{)}^{\frac{p+1}{2}}d\theta
dx\bigg{]}$
$\displaystyle=\lim_{n\rightarrow\infty}\bigg{[}\int_{\mathbb{R}^{3}}|\nabla
u_{1,n}|^{2}+|\nabla u_{2,n}|^{2}+\lambda u_{1,n}^{2}+\lambda
u_{2,n}^{2}dx\bigg{]}+\int_{\mathbb{R}^{3}}\mu_{11}u_{1}^{2}\phi_{u_{1}}+\mu_{22}u_{2}^{2}\phi_{u_{2}}-2\mu_{12}u_{1}^{2}\phi_{u_{2}}dx$
$\displaystyle\quad-\frac{1}{2\pi}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p+1}{2}}d\theta
dx=0.$

Thus, we deduce that $\vec{u}_{n}\rightarrow\vec{u}=(u_{1},u_{2})$ in
$H^{1}(\mathbb{R}^{3})\times H^{1}(\mathbb{R}^{3})$. ∎

######  Proposition 6.2.

Let $1<p<2$ and $\lambda,\mu_{ij}>0$ for $i,j=1,2$. If
$\mu_{11}\mu_{22}-\mu_{12}^{2}>0$ and $\mu_{ij}$ is small enough, where
$i,j=1,2$, then there are at least two different non-negative radial solutions
$\vec{u}_{1,\mu_{ij}}$, $\vec{u}_{2,\mu_{ij}}$ of (7), where
$\vec{u}_{1,\mu_{ij}}$ is a non-negative minimizer of (7) with negative
energy, and $\vec{u}_{2,\mu_{ij}}$ is a positive solution of (7) with positive
energy.

###### Proof.

We observe that if $\mu_{ij}=0$ for $i,j=1,2$, $I$ is not bounded below, and
thus there exists $\mu_{0}>0$ such that if $0<\mu_{ij}<\mu_{0}$, then
$\inf_{\vec{u}\in{\bf H}}I(\vec{u})<0$. By Proposition 6.1, $I$ is bounded
below and satisfies (PS) condition. Then, by the Ekeland variational principle
[11], the infimum is achieved, that is, there exists
$\vec{u}_{1,\mu_{ij}}\in{\bf H}\setminus\\{(0,0)\\}$ such that
$\inf_{\vec{u}\in{\bf H}}I(\vec{u})=I(\vec{u}_{1,\mu_{ij}})<0$ for
$0<\mu_{ij}<\mu_{0}$. Moreover, by (39), we may assume that the minimizer
$\vec{u}_{1,\mu_{ij}}$ is non-negative.

Next, in order to prove the existence of second positive solution to (7), we
consider a limit problem

$\begin{cases}-\Delta u_{1}+\lambda
u_{1}=\frac{1}{2\pi}\int_{0}^{2\pi}\Big{(}(u_{1})_{+}^{2}+(u_{2})_{+}^{2}+2(u_{1})_{+}(u_{2})_{+}\cos\theta\Big{)}^{\frac{p-1}{2}}((u_{1})_{+}+1_{\\{u_{1}>0\\}}(u_{2})_{+}\cos\theta)d\theta,\\\
-\Delta u_{2}+\lambda
u_{2}=\frac{1}{2\pi}\int_{0}^{2\pi}\Big{(}(u_{1})_{+}^{2}+(u_{2})_{+}^{2}+2(u_{1})_{+}(u_{2})_{+}\cos\theta\Big{)}^{\frac{p-1}{2}}((u_{2})_{+}+1_{\\{u_{2}>0\\}}(u_{1})_{+}\cos\theta)d\theta\end{cases}$
(45)

and a corresponding energy functional

$\displaystyle I_{+,0}(\vec{u})=$
$\displaystyle\frac{1}{2}\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}+|\nabla
u_{2}|^{2}+\lambda u_{1}^{2}+\lambda u_{2}^{2}dx$
$\displaystyle\quad-\frac{1}{p+1}\frac{1}{2\pi}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\Big{(}(u_{1})_{+}^{2}+2(u_{1})_{+}(u_{2})_{+}\cos\theta+(u_{2})_{+}^{2}\Big{)}^{\frac{p+1}{2}}d\theta
dx.$

Then by (74) and a maximum principle, $I_{+,0}$ is $C^{1}$ and any critical
point of $I_{+,0}$ is a non-negative solution of (45). It is standard to show
that

$I_{+,0}\mbox{ satisfies (PS) condition }$ (46)

because of the arguments in Proposition 6.1 (ii) and the fact that
$I_{+,0}(\vec{u})-\frac{1}{p+1}I_{+,0}^{\prime}(\vec{u})\vec{u}=(\frac{1}{2}-\frac{1}{p+1})\|\vec{u}\|_{\bf{H}}^{2}$.
Moreover, we see that there exist $c,r>0$ such that

$\mbox{if }\|\vec{u}\|_{\bf{H}}=r,\mbox{ then }I_{+,0}(\vec{u})\geq c,\mbox{
and there exists }\vec{w}\in{\bf H}\mbox{ such that
}\|\vec{w}\|_{\bf{H}}>r\mbox{ and }I_{+,0}(\vec{w})<0.$ (47)

Then by the mountain pass theorem of Ambrosetti and Rabinowitz [3], we can
prove that there exists a non-negative mountain pass solution $\vec{v}$ with
Morse index for (45) less than equal to $1$. Note that by Lemma 3.1, the Morse
index of semi-trivial solutions $(w,0)$ and $(0,w)$ for (45) are larger than
equal to $2$, where $w$ is a unique positive solution of

$-\Delta w+\lambda w=w^{p}.$ (48)

Thus, a non-negative mountain pass solution $\vec{v}$ is a positive solution.
Then by Proposition 3.8, we see that $\vec{v}=(U,U),$ where $U$ is a unique
positive solution of

$-\Delta U+\lambda U=\frac{c_{p}}{2}U^{p}$ (49)

and $c_{p}$ is given in (21) Moreover,

$I_{+,0}(U,U)=(\frac{1}{2}-\frac{1}{p+1})\|(U,U)\|_{\bf{H}}^{2}>0.$ (50)

We employ the result in [15, Theorem 1] to find a critical point
$\vec{u}_{2,\mu_{ij}}$ of $I_{+}$ such that $\vec{u}_{2,\mu_{ij}}$ converges
to $(U,U)$ as $\mu_{ij}\rightarrow 0$. We write

$\displaystyle I_{+}(\vec{u})$
$\displaystyle=I_{+,0}(\vec{u})+H_{\mu_{11},\mu_{22},\mu_{12}}(\vec{u}),$

where $I_{+}$ is given in (73) and

$H_{\mu_{11},\mu_{22},\mu_{12}}(\vec{u})=\frac{1}{4}\int_{\mathbb{R}^{3}}\mu_{11}(u_{1})_{+}^{2}\phi_{(u_{1})_{+}}+\mu_{22}(u_{2})_{+}^{2}\phi_{(u_{2})_{+}}-2\mu_{12}(u_{1})_{+}^{2}\phi_{(u_{2})_{+}}dx.$

Observe that since $J_{+,0}^{\prime}(\vec{u})\vec{u}<-\rho$ if
$\vec{u}\in{\bf{H}}\setminus\\{(0,0)\\}$ and
$I_{+,0}^{\prime}(\vec{u})\vec{u}=0$, where
$J_{+,0}(\vec{u})=I_{+,0}^{\prime}(\vec{u})\vec{u}$ and $\rho$ is a positive
constant, we get

$\displaystyle\inf\\{I_{+,0}(\vec{u})\ |\
\vec{u}\in{\bf{H}}\setminus\\{(0,0)\\},I_{+,0}^{\prime}(\vec{u})\vec{u}=0\\}$
$\displaystyle=\inf\\{I_{+,0}(\vec{u})\ |\
\vec{u}\in{\bf{H}}\setminus\\{(0,0)\\},I_{+,0}^{\prime}(\vec{u})=0\\}$
$\displaystyle=\min\\{I_{+,0}(U,U),I_{+,0}(w,0),I_{+,0}(0,w)\\},$

where $U$ is a positive solution of (49) and $w$ is a positive solution of
(48). Then by the fact that the Morse index of semi-trivial solutions $(w,0)$
and $(0,w)$ for (45) are larger than equal to $2$, we see that

$I_{+,0}(U,U)=\inf\\{I_{+,0}(\vec{u})\ |\
\vec{u}\in{\bf{H}}\setminus\\{(0,0)\\},I_{+,0}^{\prime}(\vec{u})=0\\}.$ (51)

Also, we see that for $s\neq 1$,

$I_{+,0}(U,U)>I_{+,0}(\gamma(s))=\left(s^{2}-\frac{2}{p+1}s^{p+1}\right)\int_{\mathbb{R}^{3}}|\nabla
U|^{2}+\lambda U^{2}dx$ (52)

where $\gamma(s)=(sU,sU)$, and by the compact embeddings
$H^{1}_{r}(\mathbb{R}^{3})\subset L^{q}(\mathbb{R}^{3})$, where $2<q<6$,
$H_{\mu_{11},\mu_{22},\mu_{12}}:\bf{H}\rightarrow\mathbb{R}$ is a $C^{1}$
functional such that

$\displaystyle\lim_{\mu_{11},\mu_{22},\mu_{12}\rightarrow
0}\sup_{\|\vec{u}\|_{{\bf H}}\leq
M}|H_{\mu_{11},\mu_{22},\mu_{12}}(\vec{u})|=\lim_{\mu_{11},\mu_{22},\mu_{12}\rightarrow
0}\sup_{\|\vec{u}\|_{{\bf H}}\leq
M}\|H_{\mu_{11},\mu_{22},\mu_{12}}^{\prime}(\vec{u})\|_{{\bf{H}}^{-1}}=0,$
(53) $\displaystyle
H_{\mu_{11},\mu_{22},\mu_{12}}:{\bf{H}}\rightarrow\mathbb{R}\mbox{ and
}H_{\mu_{11},\mu_{22},\mu_{12}}^{\prime}:{\bf{H}}\rightarrow{\bf{H}}^{-1}\mbox{
are compact}.$

Thus, by (46), (47) and (51)-(53), applying the result in [15, Theorem 1],
there exists a critical point $\vec{u}_{2,\mu_{ij}}$ of $I_{+}$ such that
$\vec{u}_{2,\mu_{ij}}$ converges to $(U,U)$ in $H^{1}(\mathbb{R}^{3})\times
H^{1}(\mathbb{R}^{3})$ as $\mu_{ij}\rightarrow 0$. Then from this, (75) and a
strong maximum principle, we obtain that $\vec{u}_{2,\mu_{ij}}$ is positive.
Moreover, the convergence $\vec{u}_{2,\mu_{ij}}\rightarrow(U,U)$ and (50)
imply that for small $\mu_{ij}>0$, $I_{+}(\vec{u}_{2,\mu_{ij}})>0.$ ∎

###  6.3. Proof of Theorem 1.6

In Proposition 6.2, we have shown the existence of a non-negative minimizer
$\vec{u}_{1,\mu_{ij}}$ to (7) with negative energy, and a positive solution
$\vec{u}_{2,\mu_{ij}}$ to (7) with positive energy if
$\mu_{11}\mu_{22}-\mu_{12}^{2}>0$ and $\mu_{ij}>0$ is small, where $i,j=1,2$.

We claim that a non-negative minimizer $\vec{u}_{1,\mu_{ij}}$ of (7) is
positive. Suppose to the contrary that a minimizer $\vec{u}_{1,\mu_{ij}}$ of
(7) is semi-trivial, that is, $\vec{u}_{1,\mu_{ij}}=(v_{1,\mu_{ij}},0).$ Then,
by Proposition 3.2, we get a contradiction to the fact that
$\vec{u}_{1,\mu_{ij}}$ is a minimizer of (7). Thus, we deduce that for small
$\mu_{ij}>0$ with $\det(\mu_{ij})>0$, a minimizer $\vec{u}_{1}$ of (7) is
positive.

Finally, we further assume $\mu_{11}=\mu_{22}$. For small $\mu_{ij}>0$ with
$\mu_{11}>\mu_{12}$, let $\vec{u}_{1,\mu_{ij}}$ and $\vec{u}_{2,\mu_{ij}}$ be
a positive minimizer of (7) with negative energy and a positive solution of
(7) with positive energy, respectively. Then, by Proposition 3.8, we deduce
that $\vec{u}_{1,\mu_{ij}}$ and $\vec{u}_{2,\mu_{ij}}$ have the form
$\vec{u}_{1,\mu_{ij}}=(U_{1,\mu_{ij}},U_{1,\mu_{ij}})$ and
$\vec{u}_{2,\mu_{ij}}=(U_{2,\mu_{ij}},U_{2,\mu_{ij}})$, where $U_{1,\mu_{ij}}$
and $U_{2,\mu_{ij}}$ are solutions of

$-\Delta u+\lambda
u+({\mu_{11}-\mu_{12}})\phi_{u}u=\frac{c_{p}}{2}|u|^{p-1}u.$ (54)

We claim that for small $\mu_{ij}>0$ with $\mu_{11}>\mu_{12}$,
$U_{1,\mu_{ij}}$ is a positive minimizer of (54) with negative energy and
$U_{2,\mu_{ij}}$ is a positive solution of (54) with positive energy. Since
$I(u,u)=2\tilde{I}(u)$, where

$\tilde{I}(u)=\int_{\mathbb{R}^{3}}\frac{1}{2}\Big{(}|\nabla u|^{2}+\lambda
u^{2}\Big{)}+\frac{\mu_{11}-\mu_{12}}{4}\phi_{u}u^{2}-\frac{c_{p}}{2(p+1)}|u|^{p+1}dx,$

we deduce that $U_{1,\mu_{ij}}$ is a minimizer of (54) with negative energy.

On the other hand, by the construction in subsection 6.2, we see that
$U_{2,\mu_{ij}}\rightarrow U\not\equiv 0$ in $H^{1}(\mathbb{R}^{3})$ as
$\mu_{ij}\rightarrow 0$, where $i,j=1,2$, $U>0$ and $U$ satisfies (49). Then
we see that as $\mu_{ij}\rightarrow 0$,

$\tilde{I}(U_{2,\mu_{ij}})\rightarrow\int_{\mathbb{R}^{3}}\frac{1}{2}\Big{(}|\nabla
U|^{2}+\lambda
U^{2}\Big{)}-\frac{c_{p}}{2(p+1)}|U|^{p+1}dx=\left(\frac{1}{2}-\frac{1}{p+1}\right)\int_{\mathbb{R}^{3}}|\nabla
U|^{2}+\lambda U^{2}dx>0.$

##  Appendix A Pohozaev’s identity and Nonexistence result

In this section, we prove Pohozaev’s identity for (7) and give the proof of
Theorem 1.2.

######  Proposition A.1.

Let $A,B,C$ and $D$ are real constants and $(u_{1},u_{2})\in{\bf H}$ be a weak
solution of

$\begin{cases}-A\Delta u_{1}+B\lambda
u_{1}+C(\mu_{11}\phi_{u_{1}}-\mu_{12}\phi_{u_{2}})u_{1}=\frac{D}{2\pi}\int_{0}^{2\pi}|u_{1}+e^{i\theta}u_{2}|^{p-1}(u_{1}+e^{i\theta}u_{2})d\theta\\\
-A\Delta u_{2}+B\lambda
u_{2}+C(\mu_{22}\phi_{u_{2}}-\mu_{12}\phi_{u_{1}})u_{2}=\frac{D}{2\pi}\int_{0}^{2\pi}|u_{2}+e^{i\theta}u_{1}|^{p-1}(u_{2}+e^{i\theta}u_{1})d\theta,\end{cases}$
(55)

where $\lambda,\mu_{ij}>0$ for $i,j=1,2$, $-\Delta\phi_{u}=u^{2}$ and $1<p<5$.
Then it holds

$\displaystyle 0$
$\displaystyle=\int_{\mathbb{R}^{3}}\frac{A}{2}\Big{(}|\nabla
u_{1}|^{2}+|\nabla
u_{2}|^{2}\Big{)}+\frac{3B}{2}\lambda(u_{1}^{2}+u_{2}^{2})+\frac{5C}{4}\Big{(}\mu_{11}|\nabla\phi_{u_{1}}|^{2}+\mu_{22}|\nabla\phi_{u_{2}}|^{2}-2\mu_{12}\nabla\phi_{u_{1}}\cdot\nabla\phi_{u_{2}}\Big{)}$
(56)
$\displaystyle\qquad-\frac{3D}{p+1}\bigg{[}\frac{1}{2\pi}\int_{0}^{2\pi}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p+1}{2}}d\theta\bigg{]}dx.$

###### Proof.

Let

$(u,v)\in{\bf H}\cap\Big{(}H^{2}_{loc}(\mathbb{R}^{3})\times
H^{2}_{loc}(\mathbb{R}^{3})\Big{)}\mbox{ and
}(\phi_{u},\phi_{v})\in\Big{(}D^{1,2}(\mathbb{R}^{3})\times
D^{1,2}(\mathbb{R}^{3})\Big{)}\cap\Big{(}H^{2}_{loc}(\mathbb{R}^{3})\times
H^{2}_{loc}(\mathbb{R}^{3})\Big{)},$

where $-\Delta\phi_{u}=u^{2}$ and $-\Delta\phi_{v}=v^{2}$. By (72) and
integration by parts, we have

$\displaystyle\int_{B(0,R)}\int_{0}^{2\pi}|u+e^{i\theta}v|^{p-1}(u+e^{i\theta}v)(x\cdot\nabla
u)+|v+e^{i\theta}u|^{p-1}(v+e^{i\theta}u)(x\cdot\nabla v)d\theta dx$ (57)
$\displaystyle=\int_{0}^{2\pi}\int_{B(0,R)}\Big{(}(u+v\cos\theta)(x\cdot\nabla
u)+(v+u\cos\theta)(x\cdot\nabla
v)\Big{)}\Big{(}u^{2}+2uv\cos\theta+v^{2}\Big{)}^{\frac{p-1}{2}}dxd\theta$
$\displaystyle=\frac{1}{p+1}\int_{0}^{2\pi}\int_{B(0,R)}x\cdot\nabla\Big{(}u^{2}+2uv\cos\theta+v^{2}\Big{)}^{\frac{p+1}{2}}dxd\theta$
$\displaystyle=\int_{0}^{2\pi}\bigg{[}-\frac{3}{p+1}\int_{B(0,R)}\Big{(}u^{2}+2uv\cos\theta+v^{2}\Big{)}^{\frac{p+1}{2}}dx$
$\displaystyle\quad+\frac{R}{p+1}\int_{\partial
B(0,R)}(u^{2}+2uv\cos\theta+v^{2})^{\frac{p+1}{2}}d\sigma\bigg{]}d\theta.$

On the other hand, by integration by parts,

$\displaystyle\int_{B(0,R)}(-\Delta\phi_{u})(x\cdot\nabla\phi_{v})dx=-\int_{B(0,R)}(\partial_{jj}\phi_{u})(x_{i}\partial_{i}\phi_{v})dx$
$\displaystyle=\int_{B(0,R)}(\partial_{i}\phi_{u})(\partial_{i}\phi_{v})+x_{i}(\partial_{j}\phi_{u})(\partial_{ij}\phi_{v})dx-\int_{\partial
B(0,R)}(\partial_{j}\phi_{u})(\partial_{i}\phi_{v})\frac{x_{i}x_{j}}{|x|}d\sigma$
$\displaystyle=\int_{B(0,R)}(\partial_{i}\phi_{u})(\partial_{i}\phi_{v})-3(\partial_{j}\phi_{u})(\partial_{j}\phi_{v})-x_{i}(\partial_{ij}\phi_{u})(\partial_{j}\phi_{v})dx$
$\displaystyle\quad-\int_{\partial
B(0,R)}(\partial_{j}\phi_{u})(\partial_{i}\phi_{v})\frac{x_{i}x_{j}}{|x|}d\sigma+\int_{\partial
B(0,R)}|x|(\partial_{j}\phi_{u})(\partial_{j}\phi_{v})d\sigma.$

Here, we adopt the summation convention on repeated indices. From this, we see
that the term $\int_{B(0,R)}u^{2}(x\cdot\nabla\phi_{v})dx$ can be expressed in
the following two ways:

$\displaystyle\int_{B(0,R)}u^{2}(x\cdot\nabla\phi_{v})dx$
$\displaystyle=\int_{B(0,R)}\nabla\phi_{u}\cdot\nabla\phi_{v}+x_{i}(\partial_{j}\phi_{u})(\partial_{ij}\phi_{v})dx$
(58) $\displaystyle\quad-\frac{1}{R}\int_{\partial
B(0,R)}(x\cdot\nabla\phi_{u})(x\cdot\nabla\phi_{v})d\sigma$

and

$\displaystyle\int_{B(0,R)}u^{2}(x\cdot\nabla\phi_{v})dx$
$\displaystyle=\int_{B(0,R)}-2\nabla\phi_{u}\cdot\nabla\phi_{v}-x_{i}(\partial_{ij}\phi_{u})(\partial_{j}\phi_{v})dx$
(59) $\displaystyle\quad+\int_{\partial
B(0,R)}R\nabla\phi_{u}\cdot\nabla\phi_{v}-\frac{1}{R}(x\cdot\nabla\phi_{u})(x\cdot\nabla\phi_{v})d\sigma.$

Then, by (58) and [10, Lemma 3.1 (3.13)],

$\displaystyle\int_{B(0,R)}\phi_{v}u(x\cdot\nabla u)dx$ (60)
$\displaystyle=-\int_{B(0,R)}\frac{1}{2}u^{2}(x\cdot\nabla\phi_{v})+\frac{3}{2}\phi_{v}u^{2}dx+\frac{R}{2}\int_{\partial
B(0,R)}\phi_{v}u^{2}d\sigma$
$\displaystyle=-\int_{B(0,R)}\frac{1}{2}u^{2}(x\cdot\nabla\phi_{v})+\frac{3}{2}\phi_{v}(-\Delta\phi_{u})dx+\frac{R}{2}\int_{\partial
B(0,R)}\phi_{v}u^{2}d\sigma$
$\displaystyle=-\int_{B(0,R)}\frac{1}{2}u^{2}(x\cdot\nabla\phi_{v})+\frac{3}{2}\nabla\phi_{u}\cdot\nabla\phi_{v}dx+\int_{\partial
B(0,R)}\frac{R}{2}\phi_{v}u^{2}+\frac{3}{2R}\phi_{v}(\nabla\phi_{u}\cdot
x)d\sigma$
$\displaystyle=\int_{B(0,R)}-2\nabla\phi_{u}\cdot\nabla\phi_{v}-\frac{1}{2}x_{i}(\partial_{j}\phi_{u})(\partial_{ij}\phi_{v})dx$
$\displaystyle\quad+\int_{\partial
B(0,R)}\frac{R}{2}\phi_{v}u^{2}+\frac{3}{2R}\phi_{v}(\nabla\phi_{u}\cdot
x)+\frac{1}{2R}(x\cdot\nabla\phi_{u})(x\cdot\nabla\phi_{v})d\sigma$

and similarily, by (59) and [10, Lemma 3.1 (3.13)],

$\displaystyle\int_{B(0,R)}\phi_{u}v(x\cdot\nabla v)dx$ (61)
$\displaystyle=-\int_{B(0,R)}\frac{1}{2}v^{2}(x\cdot\nabla\phi_{u})+\frac{3}{2}\phi_{u}v^{2}dx+\frac{R}{2}\int_{\partial
B(0,R)}\phi_{u}v^{2}d\sigma$
$\displaystyle=-\int_{B(0,R)}\frac{1}{2}v^{2}(x\cdot\nabla\phi_{u})+\frac{3}{2}\nabla\phi_{u}\cdot\nabla\phi_{v}dx+\int_{\partial
B(0,R)}\frac{R}{2}\phi_{u}v^{2}+\frac{3}{2R}\phi_{u}(\nabla\phi_{v}\cdot
x)d\sigma$
$\displaystyle=\int_{B(0,R)}-\frac{1}{2}\nabla\phi_{u}\cdot\nabla\phi_{v}+\frac{1}{2}x_{i}(\partial_{j}\phi_{u})(\partial_{ij}\phi_{v})dx$
$\displaystyle\quad+\int_{\partial
B(0,R)}\frac{R}{2}\phi_{u}v^{2}+\frac{3}{2R}\phi_{u}(\nabla\phi_{v}\cdot
x)-\frac{R}{2}\nabla\phi_{u}\cdot\nabla\phi_{v}+\frac{1}{2R}(x\cdot\nabla\phi_{u})(x\cdot\nabla\phi_{v})d\sigma.$

Thus, by (60) and (61), we see that

$\displaystyle\int_{B(0,R)}\phi_{v}u(x\cdot\nabla u)+\phi_{u}v(x\cdot\nabla
v)dx$ (62)
$\displaystyle=\int_{B(0,R)}-\frac{5}{2}\nabla\phi_{u}\cdot\nabla\phi_{v}dx+\int_{\partial
B(0,R)}\frac{R}{2}(\phi_{v}u^{2}+\phi_{u}v^{2})+\frac{3}{2R}\Big{(}\phi_{v}(\nabla\phi_{u}\cdot
x)+\phi_{u}(\nabla\phi_{v}\cdot x)\Big{)}$
$\displaystyle\quad-\frac{R}{2}\nabla\phi_{u}\cdot\nabla\phi_{v}+\frac{1}{R}(x\cdot\nabla\phi_{u})(x\cdot\nabla\phi_{v})d\sigma.$

Moreover, by [10, Lemma 3.1],

$\displaystyle\int_{B(0,R)}(-A\Delta u+B\lambda u)(x\cdot\nabla u)dx$ (63)
$\displaystyle=-\int_{B(0,R)}\frac{A}{2}|\nabla
u|^{2}+\frac{3B\lambda}{2}u^{2}dx+\int_{\partial
B(0,R)}-\frac{A}{R}|x\cdot\nabla u|^{2}+\frac{RA}{2}|\nabla
u|^{2}+\frac{RB\lambda}{2}u^{2}d\sigma$

and

$\displaystyle\int_{B(0,R)}\phi_{u}u(x\cdot\nabla u)dx$ (64)
$\displaystyle=-\int_{B(0,R)}\frac{1}{2}u^{2}(x\cdot\nabla\phi_{u})+\frac{3}{2}\phi_{u}u^{2}dx+\frac{R}{2}\int_{\partial
B(0,R)}\phi_{u}u^{2}d\sigma$
$\displaystyle=-\int_{B(0,R)}\frac{1}{2}(-\Delta\phi_{u})(x\cdot\nabla\phi_{u})+\frac{3}{2}\phi_{u}u^{2}dx+\frac{R}{2}\int_{\partial
B(0,R)}\phi_{u}u^{2}d\sigma$
$\displaystyle=\int_{B(0,R)}\frac{1}{4}|\nabla\phi_{u}|^{2}-\frac{3}{2}\phi_{u}u^{2}dx+\int_{\partial
B(0,R)}\frac{1}{2R}|x\cdot\nabla\phi_{u}|^{2}-\frac{R}{4}|\nabla\phi_{u}|^{2}+\frac{R}{2}\phi_{u}u^{2}d\sigma.$

Let $(u_{1},u_{2})\in{\bf H}$ be a weak solution of (55). Then, by Newton’s
Theorem, the Strauss inequality and [12, Theorem 8.17],
$(\phi_{u_{1}},\phi_{u_{2}})\in L^{\infty}_{loc}(\mathbb{R}^{3})\times
L^{\infty}_{loc}(\mathbb{R}^{3})$, $(u_{1},u_{2})\in
L^{\infty}_{loc}(\mathbb{R}^{3})\times L^{\infty}_{loc}(\mathbb{R}^{3})$, and
thus standard regularity theory [12, Theorem 9.19] implies
$u_{1},u_{2},\phi_{u_{1}},\phi_{u_{2}}\in W^{2,q}_{loc}(\mathbb{R}^{3})$ for
any $q\in[1,\infty)$. Moreover, by a comparision principle, we have

$u_{i}(x)\leq\tilde{C}_{1}\exp(-\tilde{C}_{2}|x|)\mbox{ for
}x\in\mathbb{R}^{3},$

where $i=1,2$ and $\tilde{C}_{1},\tilde{C}_{2}>$ are constants. Since
$(u_{1},u_{2})\in{\bf H}$ is a weak solution of (55), we have

$\displaystyle 0$ $\displaystyle=\int_{B(0,R)}(-A\Delta u_{1}+B\lambda
u_{1})(x\cdot\nabla u_{1})+(-A\Delta u_{2}+B\lambda u_{2})(x\cdot\nabla
u_{2})$ (65)
$\displaystyle\quad+C\bigg{(}\mu_{11}\phi_{u_{1}}u_{1}(x\cdot\nabla
u_{1})+\mu_{22}\phi_{u_{2}}u_{2}(x\cdot\nabla
u_{2})-\mu_{12}\Big{(}\phi_{u_{2}}u_{1}(x\cdot\nabla
u_{1})+\phi_{u_{1}}u_{2}(x\cdot\nabla u_{2})\Big{)}\bigg{)}$
$\displaystyle\quad-\frac{D}{2\pi}\int_{0}^{2\pi}|u_{1}+e^{i\theta}u_{2}|^{p-1}(u_{1}+e^{i\theta}u_{2})(x\cdot\nabla
u_{1})+|u_{2}+e^{i\theta}u_{1}|^{p-1}(u_{2}+e^{i\theta}u_{1})(x\cdot\nabla
u_{2})d\theta\ dx.$

Then, by (57) and (62)-(65), we deduce that

$\displaystyle\int_{B(0,R)}\frac{A}{2}\Big{(}|\nabla u_{1}|^{2}+|\nabla
u_{2}|^{2}\Big{)}+\frac{3B}{2}\lambda(u_{1}^{2}+u_{2}^{2})$ (66)
$\displaystyle\quad+C\Big{(}-\frac{\mu_{11}}{4}|\nabla\phi_{u_{1}}|^{2}+\frac{3\mu_{11}}{2}\phi_{u_{1}}u_{1}^{2}-\frac{\mu_{22}}{4}|\nabla\phi_{u_{2}}|^{2}+\frac{3\mu_{22}}{2}\phi_{u_{2}}u_{2}^{2}-\frac{5}{2}\mu_{12}\nabla\phi_{u_{1}}\cdot\nabla\phi_{u_{2}}\Big{)}$
$\displaystyle\quad-\frac{3D}{p+1}\bigg{[}\frac{1}{2\pi}\int_{0}^{2\pi}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p+1}{2}}d\theta\bigg{]}dx$
$\displaystyle=\int_{\partial B(0,R)}-\frac{A}{R}|x\cdot\nabla
u_{1}|^{2}+\frac{RA}{2}|\nabla
u_{1}|^{2}+\frac{RB\lambda}{2}u_{1}^{2}-\frac{A}{R}|x\cdot\nabla
u_{2}|^{2}+\frac{RA}{2}|\nabla u_{2}|^{2}+\frac{RB\lambda}{2}u_{2}^{2}$
$\displaystyle+C\bigg{[}\frac{\mu_{11}}{2R}|x\cdot\nabla\phi_{u_{1}}|^{2}-\frac{R\mu_{11}}{4}|\nabla\phi_{u_{1}}|^{2}+\frac{R\mu_{11}}{2}\phi_{u_{1}}u_{1}^{2}+\frac{\mu_{22}}{2R}|x\cdot\nabla\phi_{u_{2}}|^{2}-\frac{R\mu_{22}}{4}|\nabla\phi_{u_{2}}|^{2}+\frac{R\mu_{22}}{2}\phi_{u_{2}}u_{2}^{2}$
$\displaystyle\quad-\frac{R\mu_{12}}{2}(\phi_{u_{2}}u_{1}^{2}+\phi_{u_{1}}u_{2}^{2})-\frac{3\mu_{12}}{2R}\Big{(}\phi_{u_{2}}(\nabla\phi_{u_{1}}\cdot
x)+\phi_{u_{1}}(\nabla\phi_{u_{2}}\cdot
x)\Big{)}+\frac{R\mu_{12}}{2}\nabla\phi_{u_{1}}\cdot\nabla\phi_{u_{2}}$
$\displaystyle\quad-\frac{\mu_{12}}{R}(x\cdot\nabla\phi_{u_{1}})(x\cdot\nabla\phi_{u_{2}})\bigg{]}-\frac{RD}{2\pi(p+1)}\int_{0}^{2\pi}(u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2})^{\frac{p+1}{2}}d\theta
d\sigma.$

We will prove that the right hand side in (66) converges to zero for a
suitable sequence $R_{n}\rightarrow\infty$. Assuming this, we have

$\displaystyle 0$
$\displaystyle=\int_{\mathbb{R}^{3}}\frac{A}{2}\Big{(}|\nabla
u_{1}|^{2}+|\nabla u_{2}|^{2}\Big{)}+\frac{3B}{2}\lambda(u_{1}^{2}+u_{2}^{2})$
$\displaystyle\quad+C\Big{(}-\frac{\mu_{11}}{4}|\nabla\phi_{u_{1}}|^{2}+\frac{3\mu_{11}}{2}\phi_{u_{1}}u_{1}^{2}-\frac{\mu_{22}}{4}|\nabla\phi_{u_{2}}|^{2}+\frac{3\mu_{22}}{2}\phi_{u_{2}}u_{2}^{2}-\frac{5}{2}\mu_{12}\nabla\phi_{u_{1}}\cdot\nabla\phi_{u_{2}}\Big{)}$
$\displaystyle\quad-\frac{3D}{p+1}\bigg{[}\frac{1}{2\pi}\int_{0}^{2\pi}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p+1}{2}}d\theta\bigg{]}dx$
$\displaystyle=\int_{\mathbb{R}^{3}}\frac{A}{2}\Big{(}|\nabla
u_{1}|^{2}+|\nabla
u_{2}|^{2}\Big{)}+\frac{3B}{2}\lambda(u_{1}^{2}+u_{2}^{2})+\frac{5C}{4}\Big{(}\mu_{11}|\nabla\phi_{u_{1}}|^{2}+\mu_{22}|\nabla\phi_{u_{2}}|^{2}-2\mu_{12}\nabla\phi_{u_{1}}\cdot\nabla\phi_{u_{2}}\Big{)}$
$\displaystyle\qquad-\frac{3D}{p+1}\bigg{[}\frac{1}{2\pi}\int_{0}^{2\pi}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p+1}{2}}d\theta\bigg{]}dx.$

Following the arguments in [6, Proposition 1], we can prove that the right
hand side in (66) converges to zero for a suitable sequence
$R_{n}\rightarrow\infty$. Indeed, note that by (14),

$\phi_{u_{i}}(x)\leq\frac{\|u_{i}\|_{H^{1}}^{2}}{|x|}\ \ \mbox{ and }\ \
|\nabla\phi_{u_{i}}(x)|\leq\frac{\|u_{i}\|_{H^{1}}^{2}}{|x|^{2}}.$

From this, we see that

$\Big{|}\int_{\partial B(0,R)}\frac{1}{R}\phi_{u_{i}}(\nabla\phi_{u_{j}}\cdot
x)d\sigma\Big{|}\leq CR^{-1},$ (67)

where $C$ is a constant independent of $R$. Moreover, by [6, Proposition 1],
if $G\in L^{1}(\mathbb{R}^{3})$, then there exists $R_{n}\rightarrow\infty$
such that

$R_{n}\int_{\partial B(0,R_{n})}|G|d\sigma\rightarrow 0\mbox{ as
}n\rightarrow\infty.$ (68)

Then, by (67), (68), Sobolev embedding theorem and the fact

$\int_{\mathbb{R}^{3}}\phi_{v}u^{2}dx=\int_{\mathbb{R}^{3}}\nabla\phi_{u}\cdot\nabla\phi_{v}dx\leq\int_{\mathbb{R}^{3}}|\nabla\phi_{u}||\nabla\phi_{v}|dx\leq\|\phi_{u}\|_{D^{1,2}}\|\phi_{v}\|_{D^{1,2}},$

we can prove the claim. ∎

Proof of Theorem 1.2 As a direction application of Pohozaev’s identity (56),
we prove Theorem 1.2, the nonexistence of nontrivial solutions to (7) when
$\mu_{11}\mu_{22}-\mu_{12}^{2}\geq 0$ and $p\in(0,1]\cup[5,\infty)$. Assume
that $\vec{u}=(u_{1},u_{2})\in H^{1}(\mathbb{R}^{3})\times
H^{1}(\mathbb{R}^{3})$ satisfies (7). Then, by Proposition A.1 and the fact
that $I^{\prime}(\vec{u})\vec{u}=0$, we see that

$\displaystyle 0=$ $\displaystyle\int_{\mathbb{R}^{3}}|\nabla
u_{1}|^{2}+|\nabla u_{2}|^{2}+\lambda u_{1}^{2}+\lambda
u_{2}^{2}dx+\int_{\mathbb{R}^{3}}\mu_{11}u_{1}^{2}\phi_{u_{1}}+\mu_{22}u_{2}^{2}\phi_{u_{2}}-2\mu_{12}u_{1}^{2}\phi_{u_{2}}dx$
(69)
$\displaystyle\qquad-\frac{1}{2\pi}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p+1}{2}}d\theta
dx$

and

$\displaystyle 0$ $\displaystyle=\int_{\mathbb{R}^{3}}|\nabla
u_{1}|^{2}+|\nabla
u_{2}|^{2}+3\lambda(u_{1}^{2}+u_{2}^{2})dx+\frac{5}{2}\int_{\mathbb{R}^{3}}\mu_{11}u_{1}^{2}\phi_{u_{1}}+\mu_{22}u_{2}^{2}\phi_{u_{2}}-2\mu_{12}u_{1}^{2}\phi_{u_{2}}dx$
(70)
$\displaystyle\qquad-\frac{6}{p+1}\bigg{[}\frac{1}{2\pi}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p+1}{2}}d\theta
dx\bigg{]}.$

Subtracting the first equation (69) from the second one (70), we have

$\displaystyle 0=$
$\displaystyle\int_{\mathbb{R}^{3}}2\lambda(u_{1}^{2}+u_{2}^{2})dx+\frac{3}{2}\int_{\mathbb{R}^{3}}\mu_{11}u_{1}^{2}\phi_{u_{1}}+\mu_{22}u_{2}^{2}\phi_{u_{2}}-2\mu_{12}u_{1}^{2}\phi_{u_{2}}dx$
$\displaystyle\qquad+\Big{(}1-\frac{6}{p+1}\Big{)}\bigg{[}\frac{1}{2\pi}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p+1}{2}}d\theta
dx\bigg{]}.$

From this, (13) and the facts that
$\mu_{11}>0,\mu_{11}\mu_{22}-\mu_{12}^{2}\geq 0$, we see that
$(u_{1},u_{2})=(0,0)$ if $p\geq 5$.

On the other hand, we multiply the second equation (70) by $\frac{1}{3}$ and
subtract the calculated one from the first equation (69), we have

$\displaystyle 0=$ $\displaystyle\frac{2}{3}\int_{\mathbb{R}^{3}}|\nabla
u_{1}|^{2}+|\nabla
u_{2}|^{2}dx+\frac{1}{6}\int_{\mathbb{R}^{3}}\mu_{11}u_{1}^{2}\phi_{u_{1}}+\mu_{22}u_{2}^{2}\phi_{u_{2}}-2\mu_{12}u_{1}^{2}\phi_{u_{2}}dx$
$\displaystyle\qquad+\Big{(}\frac{2}{p+1}-1\Big{)}\frac{1}{2\pi}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p+1}{2}}d\theta
dx.$

From this, (13) and the facts that
$\mu_{11}>0,\mu_{11}\mu_{22}-\mu_{12}^{2}\geq 0$, we see that
$(u_{1},u_{2})=(0,0)$ if $p\leq 1$. $\Box$

##  Appendix B Regularity of the action functional $I(\vec{u})$

In this section, we study regularities for the action function $I$ for (7). We
first show (7) is the Euler-Lagrange equation of $I$.

######  Proposition B.1.

The functional $I$ is well-defined and continuously differentiable on ${\bf
H}$. In addition, any critical point of $I$ is a weak solution of (7).

###### Proof.

Standard computation shows that $I$ is $C^{1}$ on $\mathbf{H}$ and satisfies

$\displaystyle I^{\prime}(\vec{u})\vec{\psi}$
$\displaystyle=\int_{\mathbb{R}^{3}}\nabla u_{1}\cdot\nabla\psi_{1}+\nabla
u_{2}\cdot\nabla\psi_{2}+\lambda u_{1}\psi_{1}+\lambda u_{2}\psi_{2}dx$ (71)
$\displaystyle\quad+\int_{\mathbb{R}^{3}}\mu_{11}u_{1}\phi_{u_{1}}\psi_{1}+\mu_{22}u_{2}\phi_{u_{2}}\psi_{2}-\mu_{12}u_{1}\phi_{u_{2}}\psi_{1}-\mu_{12}u_{2}\phi_{u_{1}}\psi_{2}dx$
$\displaystyle\quad-\frac{1}{2\pi}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\Big{(}(u_{1}+u_{2}\cos\theta)\psi_{1}+(u_{2}+u_{1}\cos\theta)\psi_{2}\Big{)}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p-1}{2}}d\theta
dx$ $\displaystyle=\int_{\mathbb{R}^{3}}\nabla u_{1}\cdot\nabla\psi_{1}+\nabla
u_{2}\cdot\nabla\psi_{2}+\lambda u_{1}\psi_{1}+\lambda u_{2}\psi_{2}dx$
$\displaystyle\quad+\int_{\mathbb{R}^{3}}\mu_{11}u_{1}\phi_{u_{1}}\psi_{1}+\mu_{22}u_{2}\phi_{u_{2}}\psi_{2}-\mu_{12}u_{1}\phi_{u_{2}}\psi_{1}-\mu_{12}u_{2}\phi_{u_{1}}\psi_{2}dx$
$\displaystyle\quad-\frac{1}{2\pi}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}|u_{1}+e^{i\theta}u_{2}|^{p-1}(u_{1}+e^{i\theta}u_{2})\psi_{1}+|u_{2}+e^{i\theta}u_{1}|^{p-1}(u_{2}+e^{i\theta}u_{1})\psi_{2}d\theta
dx$

which implies that any critical point of $I$ is a weak solution of (7). Here
we used the fact that for $a,b\in\mathbb{R}$,

$\displaystyle\int_{0}^{2\pi}(a+be^{i\theta})|a+be^{i\theta}|^{p-1}d\theta$
$\displaystyle=\int_{0}^{2\pi}(a+b\cos\theta+ib\sin\theta)\Big{(}(a+b\cos\theta)^{2}+(b\sin\theta)^{2}\Big{)}^{\frac{p-1}{2}}d\theta$
(72)
$\displaystyle=\int_{0}^{2\pi}(a+b\cos\theta)\Big{(}a^{2}+2ab\cos\theta+b^{2}\Big{)}^{\frac{p-1}{2}}d\theta.$

∎

We note that a nontrivial critical point of $I$ needs not to be nonnegative.
Thus, in order to find a nonnegative solution of (7), we consider a truncated
functional $I_{+}:{\bf H}\rightarrow\mathbb{R}$ given by

$\displaystyle I_{+}(\vec{u})=$
$\displaystyle\frac{1}{2}\int_{\mathbb{R}^{3}}|\nabla u_{1}|^{2}+|\nabla
u_{2}|^{2}+\lambda u_{1}^{2}+\lambda u_{2}^{2}dx$ (73)
$\displaystyle\quad+\frac{1}{4}\int_{\mathbb{R}^{3}}\mu_{11}(u_{1})_{+}^{2}\phi_{(u_{1})_{+}}+\mu_{22}(u_{2})_{+}^{2}\phi_{(u_{2})_{+}}-2\mu_{12}(u_{1})_{+}^{2}\phi_{(u_{2})_{+}}dx$
$\displaystyle\quad-\frac{1}{p+1}\frac{1}{2\pi}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\Big{(}(u_{1})_{+}^{2}+2(u_{1})_{+}(u_{2})_{+}\cos\theta+(u_{2})_{+}^{2}\Big{)}^{\frac{p+1}{2}}d\theta
dx,$

where $u_{+}=\max\\{u,0\\}$.

######  Proposition B.2.

The functional $I_{+}$ is continuously differentiable on $\mathbf{H}$. In
addition, any critical point of $I_{+}$ is a nonnegative weak solution of (7).

###### Proof.

Let us denote

$A_{+}(\vec{u})=\frac{1}{p+1}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\Big{(}(u_{1})_{+}^{2}+2(u_{1})_{+}(u_{2})_{+}\cos\theta+(u_{2})_{+}^{2}\Big{)}^{\frac{p+1}{2}}d\theta
dx.$

To prove that $A_{+}$ is $C^{1}$, with derivative given by

$\displaystyle A^{\prime}_{+}(\vec{u})[\psi_{1},\psi_{2}]$
$\displaystyle=\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\Big{(}(u_{1})_{+}^{2}+2(u_{1})_{+}(u_{2})_{+}\cos\theta+(u_{2})_{+}^{2}\Big{)}^{\frac{p-1}{2}}$
(74)
$\displaystyle\qquad\times\left(\Big{(}(u_{1})_{+}+1_{\\{u_{1}>0\\}}(u_{2})_{+}\cos\theta\Big{)}\psi_{1}+\Big{(}(u_{2})_{+}+1_{\\{u_{2}>0\\}}(u_{1})_{+}\cos\theta\Big{)}\psi_{2}\right)d\theta
dx,$

we will show that the map $\vec{u}\mapsto A^{\prime}_{+}(\vec{u})$ is
continuous. We define

$\alpha(t)=\int_{0}^{2\pi}(1+t\cos\theta)(1+2t\cos\theta+t^{2})^{\frac{p-1}{2}}d\theta.$

Then since for large $t>0$,

$\displaystyle\alpha(t)/t^{p-1}$
$\displaystyle=\int_{0}^{2\pi}(1+t\cos\theta)\Big{(}1+2t^{-1}\cos\theta+t^{-2}\Big{)}^{\frac{p-1}{2}}d\theta$
$\displaystyle=\int_{0}^{2\pi}(1+t\cos\theta)\Big{(}1+\frac{p-1}{2}(2t^{-1}\cos\theta+t^{-2})+O(t^{-2})\Big{)}d\theta$
$\displaystyle=\int_{0}^{2\pi}1+(p-1)(\cos\theta)^{2}d\theta+O(t^{-1})=\pi(p+1)+O(t^{-1}),$

we see that $\lim_{t\rightarrow 0}\alpha(t)=2\pi$ and
$\lim_{t\rightarrow\infty}\alpha(t)/t^{p-1}=\pi(p+1)$. Since $\alpha$ is
continuous, we see that

$\alpha(t)\leq C(1+t^{p-1})\mbox{ for all }t\geq 0.$

From this, we get

$\displaystyle\int_{0}^{2\pi}\Big{(}(u_{1})_{+}^{2}+2(u_{1})_{+}(u_{2})_{+}\cos\theta+(u_{2})_{+}^{2}\Big{)}^{\frac{p-1}{2}}\left((u_{1})_{+}+1_{\\{u_{1}>0\\}}(u_{2})_{+}\cos\theta\right)d\theta$
(75)
$\displaystyle=(u_{1})_{+}^{p}\int_{0}^{2\pi}\Big{(}(1+2(u_{1})_{+}^{-1}(u_{2})_{+}\cos\theta+(u_{1})_{+}^{-2}(u_{2})_{+}^{2}\Big{)}^{\frac{p-1}{2}}\left(1+(u_{1})_{+}^{-1}(u_{2})_{+}\cos\theta\right)d\theta$
$\displaystyle\leq
C(u_{1})_{+}\left((u_{1})_{+}^{p-1}+(u_{2})_{+}^{p-1}\right).$

From this, we see that

$\displaystyle\bigg{\|}\int_{0}^{2\pi}\Big{(}(u_{1})_{+}^{2}+2(u_{1})_{+}(u_{2})_{+}\cos\theta+(u_{2})_{+}^{2}\Big{)}^{\frac{p-1}{2}}\left((u_{1})_{+}+1_{\\{u_{1}>0\\}}(u_{2})_{+}\cos\theta\right)\bigg{\|}_{L^{\frac{p+1}{p}}}$
$\displaystyle\leq
C\left\|(u_{1})_{+}\left((u_{1})_{+}^{p-1}+(u_{2})_{+}^{p-1}\right)\right\|_{L^{\frac{p+1}{p}}}\leq
C(\|u_{1}\|_{L^{p+1}}^{p}+\|u_{2}\|_{L^{p+1}}^{p}),$

and hence if $(u_{1},u_{2})\rightarrow(v_{1},v_{2})$ in ${\bf H}$, then for
$\psi_{1}\in H^{1}(\mathbb{R}^{3})$,

$\displaystyle\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\Big{(}(u_{1})_{+}^{2}+2(u_{1})_{+}(u_{2})_{+}\cos\theta+(u_{2})_{+}^{2}\Big{)}^{\frac{p-1}{2}}\left((u_{1})_{+}+1_{\\{u_{1}>0\\}}(u_{2})_{+}\cos\theta\right)$
$\displaystyle\quad-\Big{(}(v_{1})_{+}^{2}+2(v_{1})_{+}(v_{2})_{+}\cos\theta+(v_{2})_{+}^{2}\Big{)}^{\frac{p-1}{2}}\left((v_{1})_{+}+1_{\\{v_{1}>0\\}}(v_{2})_{+}\cos\theta\right)d\theta\psi_{1}dx$
$\displaystyle\leq\|\psi_{1}\|_{L^{p+1}}\bigg{\|}\int_{0}^{2\pi}\Big{(}(u_{1})_{+}^{2}+2(u_{1})_{+}(u_{2})_{+}\cos\theta+(u_{2})_{+}^{2}\Big{)}^{\frac{p-1}{2}}\left((u_{1})_{+}+1_{\\{u_{1}>0\\}}(u_{2})_{+}\cos\theta\right)$
$\displaystyle\quad-\Big{(}(v_{1})_{+}^{2}+2(v_{1})_{+}(v_{2})_{+}\cos\theta+(v_{2})_{+}^{2}\Big{)}^{\frac{p-1}{2}}\left((v_{1})_{+}+1_{\\{v_{1}>0\\}}(v_{2})_{+}\cos\theta\right)d\theta\bigg{\|}_{L^{\frac{p+1}{p}}}\rightarrow
0.$

This implies that the map $\vec{u}\mapsto A^{\prime}_{+}(\vec{u})$ is
continuous.

Then we see that $I_{+}$ is $C^{1}$ and

$\displaystyle I^{\prime}_{+}(\vec{u})[\psi_{1},\psi_{2}]$
$\displaystyle=\int_{\mathbb{R}^{3}}\nabla u_{1}\cdot\nabla\psi_{1}+\nabla
u_{2}\cdot\nabla\psi_{2}+\lambda u_{1}\psi_{1}+\lambda u_{2}\psi_{2}dx$
$\displaystyle\quad+\int_{\mathbb{R}^{3}}\mu_{11}(u_{1})_{+}\phi_{(u_{1})_{+}}\psi_{1}+\mu_{22}(u_{2})_{+}\phi_{(u_{2})_{+}}\psi_{2}-\mu_{12}(u_{1})_{+}\phi_{(u_{2})_{+}}\psi_{1}-\mu_{12}(u_{2})_{+}\phi_{(u_{1})_{+}}\psi_{2}dx$
$\displaystyle\quad-\frac{1}{2\pi}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\Big{(}(u_{1})_{+}^{2}+2(u_{1})_{+}(u_{2})_{+}\cos\theta+(u_{2})_{+}^{2}\Big{)}^{\frac{p-1}{2}}$
$\displaystyle\qquad\times\left(\Big{(}(u_{1})_{+}+1_{\\{u_{1}>0\\}}(u_{2})_{+}\cos\theta\Big{)}\psi_{1}+\Big{(}(u_{2})_{+}+1_{\\{u_{2}>0\\}}(u_{1})_{+}\cos\theta\Big{)}\psi_{2}\right)d\theta
dx.$

Moreover, from a maximum principle, we see that any critical point of $I_{+}$
is a non-negative solution of (7). ∎

We next show that $I$ is actually $C^{2}$. This regularity is required to
compute the Morse indices of critical points of $I$.

######  Proposition B.3.

The functional $I$ is twice continuously differentiable on $\mathbf{H}$.

###### Proof.

Let $\vec{\psi}=(\psi_{1},\psi_{2})$,

$T(\vec{u})=\frac{1}{4}\int_{\mathbb{R}^{3}}\mu_{11}u_{1}^{2}\phi_{u_{1}}+\mu_{22}u_{2}^{2}\phi_{u_{2}}-2\mu_{12}u_{1}^{2}\phi_{u_{2}}dx.$

and

$A(\vec{u})=\frac{1}{p+1}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p+1}{2}}d\theta
dx.$

Note that $T$ is $C^{2}$ functional, and hence we have

$T^{\prime}(\vec{u})\vec{\psi}=\int_{\mathbb{R}^{3}}\mu_{11}u_{1}\phi_{u_{1}}\psi_{1}+\mu_{22}u_{2}\phi_{u_{2}}\psi_{2}-\mu_{12}u_{1}\phi_{u_{2}}\psi_{1}-\mu_{12}u_{2}\phi_{u_{1}}\psi_{2}dx$

and

$\displaystyle T^{\prime\prime}(\vec{u})[\vec{\psi},\vec{\psi}]$
$\displaystyle=\int_{\mathbb{R}^{3}}\mu_{11}\phi_{u_{1}}\psi_{1}^{2}+\mu_{11}u_{1}(\phi_{u_{1}}^{\prime}\psi_{1})\psi_{1}-\mu_{12}\phi_{u_{2}}\psi_{1}^{2}-\mu_{12}u_{2}(\phi_{u_{1}}^{\prime}\psi_{1})\psi_{2}$
(76)
$\displaystyle\quad+\mu_{22}\phi_{u_{2}}\psi_{2}^{2}+\mu_{22}u_{2}(\phi_{u_{2}}^{\prime}\psi_{2})\psi_{2}-\mu_{12}u_{1}(\phi_{u_{2}}^{\prime}\psi_{2})\psi_{1}-\mu_{12}\phi_{u_{1}}\psi_{2}^{2}dx$
$\displaystyle=\int_{\mathbb{R}^{3}}\mu_{11}\phi_{u_{1}}\psi_{1}^{2}+\mu_{11}u_{1}(\phi_{u_{1}}^{\prime}\psi_{1})\psi_{1}-\mu_{12}\phi_{u_{2}}\psi_{1}^{2}-2\mu_{12}u_{2}(\phi_{u_{1}}^{\prime}\psi_{1})\psi_{2}$
$\displaystyle\quad+\mu_{22}\phi_{u_{2}}\psi_{2}^{2}+\mu_{22}u_{2}(\phi_{u_{2}}^{\prime}\psi_{2})\psi_{2}-\mu_{12}\phi_{u_{1}}\psi_{2}^{2}dx.$

Moreover, note that since

$u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}=(u_{1}+u_{2}\cos\theta)^{2}+u_{2}^{2}\sin^{2}\theta=(u_{2}+u_{1}\cos\theta)^{2}+u_{1}^{2}\sin^{2}\theta,$

we have

$(u_{1}+u_{2}\cos\theta)^{2}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{-1}\leq
1$ (77)

and

$(u_{2}+u_{1}\cos\theta)(u_{1}+u_{2}\cos\theta)\Big{(}u_{2}^{2}+2u_{2}u_{1}\cos\theta+u_{1}^{2}\Big{)}^{-1}\leq
1,$

and hence we get

$\left\|\int_{0}^{2\pi}(u_{2}+u_{1}\cos\theta)(u_{1}+u_{2}\cos\theta)\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p-3}{2}}d\theta\right\|_{L^{\frac{p+1}{p-1}}}\leq
C(\|u_{1}\|_{L^{p+1}}^{p-1}+\|u_{2}\|_{L^{p+1}}^{p-1})$

From this, we see that if $(u_{1},u_{2})\rightarrow(v_{1},v_{2})$ in ${\bf
H}$, then for $\psi_{1},\varphi_{2}\in H^{1}(\mathbb{R}^{3})$,

$\displaystyle\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}(u_{2}+u_{1}\cos\theta)(u_{1}+u_{2}\cos\theta)\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p-3}{2}}$
$\displaystyle\quad-(v_{2}+v_{1}\cos\theta)(v_{1}+v_{2}\cos\theta)\Big{(}v_{1}^{2}+2v_{1}v_{2}\cos\theta+v_{2}^{2}\Big{)}^{\frac{p-3}{2}}d\theta\psi_{1}\varphi_{2}dx$
$\displaystyle\leq\|\psi_{1}\|_{L^{p+1}}\|\varphi_{2}\|_{L^{p+1}}\bigg{\|}\int_{0}^{2\pi}(u_{2}+u_{1}\cos\theta)(u_{1}+u_{2}\cos\theta)\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p-3}{2}}$
$\displaystyle\quad-(v_{2}+v_{1}\cos\theta)(v_{1}+v_{2}\cos\theta)\Big{(}v_{1}^{2}+2v_{1}v_{2}\cos\theta+v_{2}^{2}\Big{)}^{\frac{p-3}{2}}d\theta\bigg{\|}_{L^{\frac{p+1}{p-1}}}\rightarrow
0.$

Thus we deduce that $A$ is $C^{2}$, with derivatives given by

$\displaystyle A^{\prime}(\vec{u})\vec{\psi}$
$\displaystyle=\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p-1}{2}}\Big{(}(u_{1}+u_{2}\cos\theta)\psi_{1}+(u_{2}+u_{1}\cos\theta)\psi_{2}\Big{)}d\theta
dx$

and

$\displaystyle A^{\prime\prime}(\vec{u})[\vec{\psi},\vec{\psi}]$ (78)
$\displaystyle=\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\bigg{[}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p-1}{2}}+(p-1)(u_{1}+u_{2}\cos\theta)^{2}\Big{(}u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2}\Big{)}^{\frac{p-3}{2}}\bigg{]}\psi_{1}^{2}$
$\displaystyle\quad+2\bigg{[}\cos\theta\Big{(}u_{2}^{2}+2u_{2}u_{1}\cos\theta+u_{1}^{2}\Big{)}^{\frac{p-1}{2}}$
$\displaystyle\quad+(p-1)(u_{2}+u_{1}\cos\theta)(u_{1}+u_{2}\cos\theta)\Big{(}u_{2}^{2}+2u_{2}u_{1}\cos\theta+u_{1}^{2}\Big{)}^{\frac{p-3}{2}}\bigg{]}\psi_{1}\psi_{2}$
$\displaystyle\quad+\bigg{[}\Big{(}u_{2}^{2}+2u_{2}u_{1}\cos\theta+u_{1}^{2}\Big{)}^{\frac{p-1}{2}}+(p-1)(u_{2}+u_{1}\cos\theta)^{2}\Big{(}u_{2}^{2}+2u_{2}u_{1}\cos\theta+u_{1}^{2}\Big{)}^{\frac{p-3}{2}}\bigg{]}\psi_{2}^{2}d\theta
dx$
$\displaystyle=\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\bigg{\\{}\Big{[}1+(p-1)(u_{1}+u_{2}\cos\theta)^{2}(u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2})^{-1}\Big{]}\psi_{1}^{2}$
$\displaystyle\quad+2\Big{[}\cos\theta+(p-1)(u_{2}+u_{1}\cos\theta)(u_{1}+u_{2}\cos\theta)(u_{2}^{2}+2u_{2}u_{1}\cos\theta+u_{1}^{2})^{-1}\Big{]}\psi_{1}\psi_{2}$
$\displaystyle\quad+\Big{[}1+(p-1)(u_{2}+u_{1}\cos\theta)^{2}(u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2})^{-1}\Big{]}\psi_{2}^{2}\bigg{\\}}\Big{(}u_{2}^{2}+2u_{2}u_{1}\cos\theta+u_{1}^{2}\Big{)}^{\frac{p-1}{2}}d\theta
dx.$

Thus, from (76) and (78), we have

$\displaystyle I^{\prime\prime}(\vec{u})[\vec{\psi},\vec{\psi}]$ (79)
$\displaystyle=\int_{\mathbb{R}^{3}}|\nabla\psi_{1}|^{2}+|\nabla\psi_{2}|^{2}+\lambda\psi_{1}^{2}+\lambda\psi_{2}^{2}dx$
$\displaystyle\quad+\int_{\mathbb{R}^{3}}\mu_{11}\phi_{u_{1}}\psi_{1}^{2}+\mu_{11}u_{1}(\phi_{u_{1}}^{\prime}\psi_{1})\psi_{1}-\mu_{12}\phi_{u_{2}}\psi_{1}^{2}-2\mu_{12}u_{2}(\phi_{u_{1}}^{\prime}\psi_{1})\psi_{2}$
$\displaystyle\quad+\mu_{22}\phi_{u_{2}}\psi_{2}^{2}+\mu_{22}u_{2}(\phi_{u_{2}}^{\prime}\psi_{2})\psi_{2}-\mu_{12}\phi_{u_{1}}\psi_{2}^{2}dx$
$\displaystyle\quad-\frac{1}{2\pi}\int_{\mathbb{R}^{3}}\int_{0}^{2\pi}\bigg{\\{}\Big{[}1+(p-1)(u_{1}+u_{2}\cos\theta)^{2}(u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2})^{-1}\Big{]}\psi_{1}^{2}$
$\displaystyle\quad+2\Big{[}\cos\theta+(p-1)(u_{2}+u_{1}\cos\theta)(u_{1}+u_{2}\cos\theta)(u_{2}^{2}+2u_{2}u_{1}\cos\theta+u_{1}^{2})^{-1}\Big{]}\psi_{1}\psi_{2}$
$\displaystyle\quad+\Big{[}1+(p-1)(u_{2}+u_{1}\cos\theta)^{2}(u_{1}^{2}+2u_{1}u_{2}\cos\theta+u_{2}^{2})^{-1}\Big{]}\psi_{2}^{2}\bigg{\\}}\Big{(}u_{2}^{2}+2u_{2}u_{1}\cos\theta+u_{1}^{2}\Big{)}^{\frac{p-1}{2}}d\theta
dx.$

∎

Acknowledgement. This research of the first author was supported by the Basic
Science Research Program through the National Research Foundation of Korea
(NRF) funded by the Ministry of Science and ICT (NRF-2020R1A2C4002615). This
research of the second author was supported by Basic Science Research Program
through the National Research Foundation of Korea (NRF) funded by the Ministry
of Science and ICT (NRF-2020R1C1C1A01006415).

## References

  * [1] N. Akhmediev, A. Ankiewicz, Partially Coherent Solitons on a Finite Background, Physical Review Letters, 82 (1999), 2661–2664.
  * [2] A. Ambrosetti and E. Colorado, Standing waves of some coupled nonlinear Schrödinger equations, J. Lond. Math. Soc. 75 (2007), 67–82.
  * [3] A. Ambrosetti, P. H. Rabinowitz, Dual variational methods in critical point theory and applications. J. Functional Analysis 14 (1973), 349–381.
  * [4] A. Azzollini, A. Pomponio, Ground state solutions for the nonlinear Schrödinger-Maxwell equations. J. Math. Anal. Appl. 345 (2008), no. 1, 90–108.
  * [5] T. Bartsch, E. N. Dancer and Z.-Q. Wang, A Liouville theorem, a-priori bounds, and bifurcating branches of positive solutions for a nonlinear elliptic system, Calc. Var. Partial Differential Equations 37 (2010) 345–361.
  * [6] H. Berestycki, P. L. Lions, Nonlinear scalar field equations I. Existence of a ground state. Arch. Rational Mech. Anal. 82 (1983), no. 4, 313–345.
  * [7] G. M. Coclite, A multiplicity result for the nonlinear Schrödinger-Maxwell equations, Commun. Appl. Anal. 7 (2003), no. 2-3, 417–423.
  * [8] M. Colin, T. Watanabe, Standing waves for the nonlinear Schrödinger equation coupled with the Maxwell equation, Nonlinearity 30 (2017), no. 5, 1920–1947.
  * [9] T. D’Aprile, D. Mugnai, Solitary waves for nonlinear Klein-Gordon-Maxwell and Schrödinger-Maxwell equations, Proc. Roy. Soc. Edinburgh Sect. A 134 (2004), no. 5, 893–906.
  * [10] T. D’Aprile, D. Mugnai, Non-existence results for the coupled Klein-Gordon-Maxwell equations. Adv. Nonlinear Stud.4 (2004), no. 3, 307–322.
  * [11] I. Ekeland, On the variational principle. J. Math. Anal. Appl. 47 (1974), 324–353.
  * [12] D. Gilbarg and N. Trudinger, Elliptic Partial Differential Equations of Second Order, Springer-Verlag, 1983.
  * [13] H. Hofer, A note on the topological degree at a critical point of mountainpass-type, Proc. Amer. Math. Soc. 90 (1984), no. 2, 309–315.
  * [14] L. Jeanjean, K. Tanaka, A remark on least energy solutions in $\mathbb{R}^{N}$. Proc. Amer. Math. Soc. 131 (2003), no. 8, 2399–2408.
  * [15] W. Jeong, J. Seok, On perturbation of a functional with the mountain pass geometry: applications to the nonlinear Schrödinger-Poisson equations and the nonlinear Klein-Gordon-Maxwell equations. Calc. Var. Partial Differential Equations 49 (2014), no. 1-2, 649–668.
  * [16] E. H. Lieb, Existence and uniqueness of the minimizing solution of Choquard’s nonlinear equation. Studies in Appl. Math. 57 (1976/77), no. 2, 93–105.
  * [17] E.H. Lieb, M. Loss, Analysis. Second edition. Graduate Studies in Mathematics, 14. American Mathematical Society, Providence, RI, 2001.
  * [18] P. L. Lions, The Choquard equation and related questions, Nonlinear Anal. 4 (1980), no. 6, 1063–1072.
  * [19] E. Lenzmann, Uniqueness of ground states for pseudorelativistic Hartree equations. Anal. PDE 2 (2009), no. 1, 1–27.
  * [20] Z. Liu and Z.-Q. Wang, Ground states and bound states of a nonlinear Schrdinger system, Adv. Nonlinear Stud. 10 (2010), 175–193.
  * [21] N. Masmoudi, K. Nakanishi, From nonlinear Klein-Gordon equation to a system of coupled nonlinear Schrödinger equations, Math. Ann. 324 (2002), no. 2, 359–389.
  * [22] N. Masmoudi, K. Nakanishi, Nonrelativistic limit from Maxwell-Klein-Gordon and Maxwell-Dirac to Poisson-Schrödinger, Int. Math. Res. Not. 2003, no. 13, 697–734.
  * [23] P.A. Markowich, C. A. Ringhofer, C. Schmeiser, Semiconductor equations, Springer-Verlag, Vienna, 1990.
  * [24] L. Ma, L. Zhao, Classification of positive solitary solutions of the nonlinear Choquard equation. Arch. Ration. Mech. Anal. 195 (2010), no. 2, 455–467.
  * [25] M. Mitchell, Z. Chen, M. Shih and M. Segev, Self-Trapping of partially spatially incoherent light, Phys. Rev. Lett. 77  (1996), 490–493.
  * [26] M. Mitchell and M. Segev, Self-trapping of inconherent white light, Nature, 387 (1997), 880–882.
  * [27] S. Peng, Z.Q. Wang, Segregated and synchronized vector solutions for nonlinear Schrödinger systems. Arch. Ration. Mech. Anal. 208 (2013), no. 1, 305–339.
  * [28] CH. Regg, N. Cavadini, A. Furrer, et al, Bose-Einstein condensation of the triple states in the magnetic insulator $TlCuCl_{3}$, Nature, 423 (2003), 62–65.
  * [29] D. Ruiz, The Schrödinger-Poisson equation under the effect of a nonlinear local term. J. Funct. Anal. 237  (2006), no. 2, 655–674.
  * [30] B. Sirakov, Least energy solitary waves for a system of nonlinear Schrödinger equations in $R^{n}$, Comm. Math. Phys. 271 (2007), 199–221.
  * [31] W. A. Strauss, Existence of solitary waves in higher dimensions. Comm. Math. Phys. 55 (1977), no. 2, 149–162.
  * [32] G. Thalhammer, G. Barontini, L. De Sarlo L, J. Catani, F. Minardi and M. Inguscio, Double species Bose-Einstein condensate with tunable interspecies interactions, Phys Rev Lett. 100 (2008), 210402.
  * [33] G. Vaira, Ground states for Schrödinger-Poisson type systems, Ric. Mat. 60 (2011), no. 2, 263–297.
  * [34] J. Wang, J. Shi, Standing waves for a coupled nonlinear Hartree equations with nonlocal interaction. Calc. Var. Partial Differential Equations 56 (2017), no. 6, Paper No. 168.
  * [35] J. Wei and W. Yao, Uniqueness of positive solutions to some coupled nonlinear Schrödinger equations, Commun. Pure Appl. Anal. 11 (2012), 1003–1011.