# Normalized solutions for Schrödinger system with quadratic and cubic
interactions

###### Abstract

In this paper, we give a complete study on the existence and non-existence of
solutions to the following mixed coupled nonlinear Schrödinger system

$\begin{cases}-\Delta u+\lambda_{1}u=\beta uv+\mu_{1}u^{3}+\rho
v^{2}u&\text{in}\ \ \mathbb{R}^{N},\\\ -\Delta
v+\lambda_{2}v=\frac{\beta}{2}u^{2}+\mu_{2}v^{3}+\rho u^{2}v&\text{in}\ \
\mathbb{R}^{N},\end{cases}$

under the normalized mass conditions $\int_{\mathbb{R}^{N}}u^{2}dx=b^{2}_{1}$
and $\int_{\mathbb{R}^{N}}v^{2}dx=b^{2}_{2}$. Here $b_{1},b_{2}>0$ are
prescribed constants, $N\geq 1$, $\mu_{1},\mu_{2},\rho>0$,
$\beta\in\mathbb{R}$ and the frequencies $\lambda_{1},\lambda_{2}$ are unknown
and will appear as Lagrange multipliers. In the one dimension case, the energy
functional is bounded from below on the product of $L^{2}$-spheres, normalized
ground states exist and are obtained as global minimizers. When $N=2$, the
energy functional is not always bounded on the product of $L^{2}$-spheres. We
give a classification of the existence and nonexistence of global minimizers.
Then under suitable conditions on $b_{1}$ and $b_{2}$, we prove the existence
of normalized solutions. When $N=3$, the energy functional is always unbounded
on the product of $L^{2}$-spheres. We show that under suitable conditions on
$b_{1}$ and $b_{2}$, at least two normalized solutions exist, one is a ground
state and the other is an excited state. Furthermore, by refining the upper
bound of the ground state energy, we provide a precise mass collapse behavior
of the ground state and a precise limit behavior of the excited state as
$\beta\rightarrow 0$. Finally, we deal with the high dimensional cases $N\geq
4$. Several non-existence results are obtained if $\beta<0$. When $N=4$,
$\beta>0$, the system is a mass-energy double critical problem, we obtain the
existence of a normalized ground state and its synchronized mass collapse
behavior. Comparing with the well studied homogeneous case $\beta=0$, our main
results indicate that the quadratic interaction term not only enriches the set
of solutions to the above Schrödinger system but also leads to a stabilization
of the related evolution system.

_Keywords:_ Schrödinger system; mixed couplings; normalized solution; mass
collapse behavior.

_2010 Mathematics Subject Classification:_ 35J50, 35B33, 35R11, 58E05.

Xiao Luo 111Email addresses<EMAIL_ADDRESS>(Luo). Juncheng Wei 222Email
addresses<EMAIL_ADDRESS>(Wei). Xiaolong Yang 333Email addresses:
<EMAIL_ADDRESS>(Yang). Maoding Zhen 444Email addresses:
<EMAIL_ADDRESS>(Zhen).

1 School of Mathematics, Hefei University of Technology, Hefei, 230009, P. R.
China
2 Department of Mathematics, University of British Columbia, Vancouver, B.C.,
V6T 1Z2, Canada
3 School of Mathematics and Statistics, Central China Normal University,
Wuhan, 430079, P. R. China
4 School of Mathematics, Hefei University of Technology, Hefei, 230009, P. R.
China

##  1 Introduction and main results

In this paper, we look for solutions to the following coupled Schrödinger
system

$\begin{cases}-\Delta u+\lambda_{1}u=\beta uv+\mu_{1}u^{3}+\rho
v^{2}u&\text{in}\ \ \mathbb{R}^{N},\\\ -\Delta
v+\lambda_{2}v=\frac{\beta}{2}u^{2}+\mu_{2}v^{3}+\rho u^{2}v&\text{in}\ \
\mathbb{R}^{N},\end{cases}$ (1.1)

satisfying the additional constraints

$\int_{\mathbb{R}^{N}}u^{2}dx=b^{2}_{1}\ \text{and}\
\int_{\mathbb{R}^{N}}v^{2}dx=b^{2}_{2}.$ (1.2)

Here, $b_{1},b_{2}>0$ are prescribed constants, $N\geq 1$,
$\mu_{1},\mu_{2},\rho>0$, $\beta\in\mathbb{R}$ and the frequencies
$\lambda_{1},\lambda_{2}$ are unknown and will appear as Lagrange multipliers.

Problem (1.1)-(1.2) arises from the research of stationary exponentially
localized bright solitary waves for the following two-wave mixing system

$\begin{cases}i\frac{\partial\Phi_{1}}{\partial
z}+\Delta\Phi_{1}+\beta\Phi^{\ast}_{1}\Phi_{2}+s(|\Phi_{1}|^{2}+\rho|\Phi_{2}|^{2})=0,\\\
i\frac{\partial\Phi_{2}}{\partial
z}+\Delta\Phi_{2}-\gamma\Phi_{2}+\frac{\beta}{2}\Phi^{2}_{1}+s(\eta|\Phi_{2}|^{2}+\rho|\Phi_{1}|^{2})\Phi_{2}=0,\end{cases}$
(1.3)

which describes the dynamics of beam propagation in lossless bulk $\chi^{(2)}$
media inhering cubic nonlinearity, under conditions for second-harmonic
generation type-I. $z$ is the propagation distance coordinate, the superscript
$\ast$ denotes the complex conjugate function and $\beta$ is a constant. The
slowly varying complex envelope functions of the fundamental wave
$\Phi_{1}=\Phi_{1}(x,z)$ and of the second harmonic $\Phi_{2}=\Phi_{2}(x,z)$
are assumed to propagate with a constant polarization,
$\overrightarrow{e}_{1},\overrightarrow{e}_{2}$ along the $z$ axis. The
electric field $\overrightarrow{E}=\overrightarrow{E}(\overrightarrow{R},Z,M)$
is given by

$\displaystyle\overrightarrow{E}=\overrightarrow{E}(\overrightarrow{R},Z,M)=E_{0}(\Phi_{1}e^{i\theta_{1}}\overrightarrow{e}_{1}+2\Phi_{2}e^{i2\theta_{1}}\overrightarrow{e}_{2}),$
(1.4)

where $\overrightarrow{R}=r_{0}x,\ Z=z_{0}z,\ \theta_{1}=k_{1}Z-\omega_{1}M$,
$\omega_{1}$ is fundamental frequency and $(\Phi_{1},\ \Phi_{2})$ satisfies
(1.3). The real normalization parameters $E_{0},\ z_{0}$ and $R_{0}$ are given
by

$\displaystyle
E_{0}=\frac{4\widetilde{\chi}^{(2)}_{1}}{3|\widetilde{\chi}^{(3)}_{1k}|},\ \
z_{0}=2k_{1}r^{2}_{0}\ \text{and}\
r^{2}_{0}=\frac{3|\widetilde{\chi}^{(3)}_{1k}|}{16\mu_{0}\omega^{2}_{1}(\widetilde{\chi}^{(2)}_{1})^{2}},$
(1.5)

where $\mu_{0}$ is the vacuum permeability and $k_{p}$ is the wave number at
the frequency $\omega_{p}$. The real parameters $\gamma,s,\eta$ and $\rho$ are
given by

$\displaystyle\gamma=2z_{0}(2k_{1}-k_{2}),\
s=sign(\widetilde{\chi}^{(3)}_{1k}),\
\eta=16\frac{\widetilde{\chi}^{(3)}_{2k}}{\widetilde{\chi}^{(3)}_{1k}},\ \
\rho=8\frac{\widetilde{\chi}^{(3)}_{1c}}{\widetilde{\chi}^{(3)}_{1k}},$ (1.6)

where $2k_{1}-k_{2}\ll k_{1}$ is the phase-mismatch parameter,
$\widetilde{\chi}^{j}_{p}=\widetilde{\chi}^{j}(\omega_{p})$ denote the Fourier
components at frequency $\omega_{p}$ of the $j$th order susceptibility tensor
and the scalar $\chi^{3}_{p,q}$ is the vectorial Fourier transform of
$\chi^{3}$ $(p,q=1,2)$. Thus,
$\widetilde{\chi}^{(2)}_{1}=\widetilde{\chi}^{(2)}_{2}$ represents the
quadratic nonlinearity, $\widetilde{\chi}^{(3)}_{pk}$ and
$\widetilde{\chi}^{(3)}_{1c}=\widetilde{\chi}^{(3)}_{2c}$ are the parts of the
cubic nonlinearity responsible for self-phase and cross-phase modulation,
respectively. For more details about physical meaning of system (1.3), one can
refer to the papers [1, 2, 3, 9].

Indeed, if we substitute

$\displaystyle\Phi_{1}(x,z)=u(x)e^{i\lambda_{0}z}\ \text{ and }\
\Phi_{2}(x,z)=v(x)e^{i2\lambda_{0}z}$ (1.7)

into (1.3), then $(u,v)$ solves the following stationary system

$\begin{cases}-\Delta u+\lambda_{1}u=\beta uv+s(u^{2}+\rho v^{2})u&\text{in}\
\ \mathbb{R}^{N},\\\ -\Delta v+\lambda_{2}v=\frac{\beta}{2}u^{2}+s(\eta
v^{2}+\rho u^{2})v&\text{in}\ \ \mathbb{R}^{N},\end{cases}$ (1.8)

with $\lambda_{1}=\lambda_{0}$, $\lambda_{2}=4\lambda_{0}+\gamma$ and $s=\pm
1$. In this paper we consider the focusing case, i.e., $s=1$ in (1.8). After
rescaling and renaming the parameters, we obtain system (1.1).

Motivated by the fact that the $L^{2}$-norm is a preserved quantity of the
evolution, (see [3]), we are interested in searching solutions to (1.1) with
prescribed $L^{2}$-norm–the so-called normalized solutions to (1.1). It is
standard that the solutions of (1.1)-(1.2) can be obtained as critical points
of the energy functional

$J_{\beta}(u,v)=\frac{1}{2}\int_{\mathbb{R}^{N}}(|\nabla u|^{2}+|\nabla
v|^{2})dx-\frac{1}{4}\int_{\mathbb{R}^{N}}(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2})dx-\frac{\beta}{2}\int_{\mathbb{R}^{N}}u^{2}vdx,$

on the constraint space $\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$, where
for $b\in\mathbb{R}$ we define

$\mathrm{T}_{b}:=\big{\\{}u\in
H^{1}(\mathbb{R}^{N}):\int_{\mathbb{R}^{N}}u^{2}=b^{2}\big{\\}}.$

In the last ten years, the study of normalized solutions for Schrödinger
equations or systems has received lots of attention. However almost all the
results deal with the cubic interactions. When $\beta=0,\ N=1,\ \rho>0$, N.
Nguyen and Z. Wang in [34] proved the existence of normalized solutions to
problem (1.1) by minimizing the corresponding energy functional constrained on
the product of the $L^{2}$-sphere and using concentration-compactness
arguments, they also studied the stability properties of these solutions.
Since the corresponding constrained functional is unbounded both from above
and from below on the $L^{2}$-sphere, the approach used in [34] does not work
for problem (1.1) with $N=3$. When $\beta=0,\ N=3$, T. Bartsch, L. Jeanjean
and N. Soave [5] proved that for arbitrary masses $b_{i}$ and positive
parameter $\mu_{i}$, there exists $\rho_{2}>\rho_{1}>0$ such that for both
$0<\rho<\rho_{1}$ and $\rho>\rho_{2}$, system (1.1)-(1.2) has a positive
radial solution. T. Bartsch and N. Soave in [6] proved the existence of at
least one normalized solution to problem (1.1) in the case of $\rho<0$ by a
new approach based on the introduction of a natural constraint associated to
problem (1.1), and in [8], they proved the existence of infinitely many
solutions of problem (1.1) with $\mu_{1}=\mu_{2}>0$ and $\rho\leq-\mu_{1}$ by
using a suitable minimax argument. Later, by using bifurcation theory and the
continuation method, T. Bartsch, X. Zhong and W. Zou [7] obtained the
existence of normalized solutions for any given $b_{1},b_{2}>0$ for $\rho$ in
a large range. They also got a result about the nonexistence of positive
solutions which shows that their existence theorem is almost optimal. By using
standard Ljusternik-Schnirelmann theory, when $\beta=0$, the authors in [35,
43] considered problem (1.1) in bounded domains under the case $\mu_{1}$,
$\mu_{2}$, $\rho<0$, they proved the existence of infinitely many normalized
solutions and occurrence of phase-separation as $\rho\rightarrow-\infty$. In
[36], B. Noris et al. studied problem (1.1) in bounded domains of
$\mathbb{R}^{N}$, or the problem with trapping potentials in the whole space
$\mathbb{R}^{N}$ (the presence of a trapping potential makes the two problems
essentially equivalent) with $N\leq 3$. In both cases, they proved the
existence of positive solutions with small masses $b_{1}$ and $b_{2}$, and the
orbital stability of the associated solitary waves. When $\beta=0,\ N=2$, Guo
et al. in [18, 19, 20] considered the existence, non-existence, uniqueness and
asymptotic behavior of solutions to problem (1.1)-(1.2) with certain type of
trapping potentials.

Since competing quadratic and cubic nonlinearities is a general physical
phenomenon, it is important to know the effect of such a competition on
normalized solutions. In [45] the authors gave a first study to problem (1.1)
with both quadratic and cubic interactions (without the mass constraints). In
the present paper, we give a complete study on the existence of normalized
solutions in the less studied case $\beta\neq 0.$ First, we consider the one
dimensional case, in which the energy functional is bounded from blow on the
product of $L^{2}$-spheres $\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$(see
Lemma 2.2), so we define

$m_{\beta}(b_{1},b_{2})=\inf_{(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}}J_{\beta}(u,v)$
(1.9)

and then we prove the following result

######  Theorem 1.1.

If $N=1$, $\mu_{1},\mu_{2},\rho>0$, then for every $\beta>0$,
$-\infty<m_{\beta}(b_{1},b_{2})<0$ is achieved. In addition any minimizing
sequence for (1.9) is, up to translation, strongly convergent in
$H^{1}(\mathbb{R})\times H^{1}(\mathbb{R})$ to a solution of (1.1)-(1.2).

Next, we turn to $N=2.$ In this case, the energy functional is not always
bounded on $\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$, relating to the
values $b_{1}$ and $b_{2}$. Let $Q$ be the unique positive radial solution to

$-\Delta Q+Q=Q^{3},\ Q\in H^{1}(\mathbb{R}^{2}).$ (1.10)

The main results in dimension $N=2$ can be stated as follows:

######  Theorem 1.2.

Let $N=2$ and $\mu_{1},\mu_{2},\rho>0$:

(i) If
$\max\\{(\mu_{1}+\rho)b^{2}_{1},(\mu_{2}+\rho)b^{2}_{2}\\}<\|Q\|_{L^{2}(\mathbb{R}^{2})}^{2}$,
then

a) for every $\beta<0$, problem (1.1)-(1.2) has no positive solutions,

b) for every $\beta>0$, when $0<b_{2}\leq\|Q\|_{L^{2}(\mathbb{R}^{2})}$,
$-\infty<m_{\beta}(b_{1},b_{2})<0$ is achieved by $(u,v)$ which is a positive
solution of (1.1)-(1.2).

(ii) If $\frac{\mu_{1}b^{4}_{1}+\mu_{2}b^{4}_{2}+2\rho
b^{2}_{1}b^{2}_{2}}{(b^{2}_{1}+b^{2}_{2})}>\|Q\|_{L^{2}(\mathbb{R}^{2})}^{2}$,
then $m_{\beta}(b_{1},b_{2})=-\infty$ for any $\beta\in\mathbb{R}$.

In order to prove Theorem 1.2, in section 2, we introduce the following
minimization problem

$A=A(\mu_{1},\mu_{1},\rho,b_{1},b_{2}):=\inf_{(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}}\frac{\int_{\mathbb{R}^{2}}(|\nabla
u|^{2}+|\nabla
v|^{2})dx}{\int_{\mathbb{R}^{2}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx}$

and prove that

$\displaystyle\frac{\|Q\|_{L^{2}(\mathbb{R}^{2})}^{2}}{2\max\\{(\mu_{1}+\rho)b^{2}_{1},(\mu_{2}+\rho)b^{2}_{2}\\}}\leq
A\leq\frac{\frac{1}{2}(b^{2}_{1}+b^{2}_{2})\|Q\|_{L^{2}(\mathbb{R}^{2})}^{2}}{\mu_{1}b^{4}_{1}+\mu_{2}b^{4}_{2}+2\rho
b^{2}_{1}b^{2}_{2}}.$

We shall prove that if $A>\frac{1}{2}$, then $J_{\beta}(u,v)$ is coercive on
$\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$ and
$m_{\beta}(b_{1},b_{2})>-\infty$ is achieved, so there exists at least one
normalized ground state for (1.1).

######  Remark 1.1.

When $N=2$, if $A<\frac{1}{2}$, then $m_{\beta}(b_{1},b_{2})=-\infty.$ Thus
the minimization problem constrained on
$\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$ does not work. Since
$\int_{\mathbb{R}^{2}}|\nabla u|^{2}+|\nabla v|^{2}$ and
$\int_{\mathbb{R}^{2}}\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho u^{2}v^{2}$ behave at
the same way under $L^{2}$ preserving scaling of $(u,v)$, $J_{\beta}(\iota
u(\iota x),\iota v(\iota x))$ may strictly increasing with respect to
$\iota\in\mathbb{R}$, so the usual methods developed on the Pohozaev-Nehari
constraint can not be applied directly here, see [5, 6, 41, 42]. Inspired by
the recent works [48, 29], we try to construct a submanifold of
$\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$ as following

$\displaystyle
N_{b_{1},b_{2}}:=\Big{\\{}(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}\mid
P_{b_{1},b_{2}}(u,v)=0\text{ and }\int_{\mathbb{R}^{2}}(|\nabla u|^{2}+|\nabla
v|^{2})dx<\frac{1}{2}\int_{\mathbb{R}^{2}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx\Big{\\}},$

on which $J(u,v)$ may admits a minimizer, where $P_{b_{1},b_{2}}(u,v)=0$ is
the related Pohozaev-Nehari identity. Indeed, for any $(u,v)$ on the set
$N_{b_{1},b_{2}}$, $J_{\beta}(\iota u(\iota x),\iota v(\iota x))$ has a unique
maximum point ${\iota}_{u,v}$ and is strictly increasing in
$(-\infty,{\iota}_{u,v})$ and decreasing in $({\iota}_{u,v},+\infty)$.
Therefore, we expect a constrained variation can be used to obtain a
normalized solution to (1.1). However, due to the uncertainty of sign of the
quadratic interaction term and inhomogeneity of the coupling term, it seems
difficult to prove the compactness of the minimizing sequences (or Palais-
Smale sequences) developed in $N_{b_{1},b_{2}}$. We believe that when
$A<\frac{1}{2}$, the existence of normalized solutions to (1.1) in dimension
two is an expected interesting result.

Now, we deal with the three dimensional case. In this case,
$m_{\beta}(b_{1},b_{2})=-\infty$ for any $b_{1},b_{2}>0.$ Indeed, the leading
term is $L^{2}$ supercritical and Sobolev subcritical, the energy function
$J_{\beta}(u,v)$ is unbounded both from above and from below on
$\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$. In order to search for two
normalized solutions, we use the ideas introduced by N. Soave [42, 41] to
study the corresponding fiber maps $\Psi_{u,v}(t)$ (see (4.2)), which has the
same Mountain pass structure as the original functional. The benefits of the
fiber map are that the critical point of $\Psi_{u,v}(t)$ allow to project a
function on $\mathcal{P}_{b_{1},b_{2}}$ (see (4.1)). The monotonicity and
convexity of $\Psi_{u,v}(t)$ has a strongly affect the structure of
$\mathcal{P}_{b_{1},b_{2}}$ and then intimately related to the minimax
structure of $J_{\beta}(u,v)|_{\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}}$.
To show our main results, we first introduce the Gagliardo-Nirenberg-Sobolev
inequality

$\|u\|_{L^{p}(\mathbb{R}^{N})}\leq C_{N,p}\|\nabla
u\|_{L^{2}(\mathbb{R}^{N})}^{\gamma_{p}}\|u\|_{L^{2}(\mathbb{R}^{N})}^{1-\gamma_{p}}\
\ \text{for all}\ \ u\in H^{1}(\mathbb{R}^{N}),$ (1.11)

where

$\displaystyle\gamma_{p}=\frac{N(p-2)}{2p},$

and we denote by $C_{N,p}$ the best constant in the the Gagliardo-Nirenberg-
Sobolev inequality $H^{1}(\mathbb{R}^{N})\hookrightarrow
L^{p}(\mathbb{R}^{N}),\ 2<p<2^{\ast}=\frac{2N}{N-2}(N\geq 3).$ Our main
results are the following:

######  Theorem 1.3.

When $N=3$, $\mu_{1},\mu_{2},\rho,\beta>0$ and

$\beta\left(2b^{\frac{3}{2}}_{1}+b^{\frac{3}{2}}_{2}\right)C^{3}_{3,3}C^{2}_{3,4}\sqrt{\mu_{1}b_{1}+\mu_{2}b_{2}+\rho
b^{\frac{1}{2}}_{1}b^{\frac{1}{2}}_{2}}<\frac{2\sqrt{6}}{3},$

then (1.1)-(1.2) has at least two positive normalized solutions, one is a
ground state $(\widehat{u}_{\beta},\widehat{v}_{\beta})$, the other is an
excited state $(\widetilde{u}_{\beta},\widetilde{v}_{\beta})$. Moreover,
$J_{\beta}(\widehat{u}_{\beta},\widehat{v}_{\beta})\rightarrow 0^{+}$,
$\int_{\mathbb{R}^{3}}(|\nabla\widehat{u}_{\beta}|^{2}+|\nabla\widehat{v}_{\beta}|^{2})dx\rightarrow
0$ and there exists $\rho_{2}>0$ such that when $\rho>\rho_{2}$, then
$(\widetilde{u}_{\beta},\widetilde{v}_{\beta})\rightarrow(\widetilde{u}_{0},\widetilde{v}_{0})$
strongly in $H^{1}(\mathbb{R}^{3})\times H^{1}(\mathbb{R}^{3})$ as
$\beta\rightarrow 0$, where $(\widetilde{u}_{0},\widetilde{v}_{0})$ is a
normalized ground state of (1.1) with $\beta=0$.

######  Remark 1.2.

Theorem 1.3 gives the existence of two normalized solutions for (1.1). The
first one is a local minimizer for which we establish the compactness of
minimizing sequence. The second solution is obtained through a constrained
linking. Theorem 1.3 also shows the limit behavior of the solutions as
$\beta\rightarrow 0$. The first solution will disappear and the second
solution will converge to the normalized solution of system (1.1) with
$\beta=0$, which has been studied by T. Bartsch, L. Jeanjean and N. Soave in
[5]. When $\beta=0,\ N=3$, in [5] the authors proved that for arbitrary masses
$b_{i}$ and parameter $\mu_{i}$, there exists $\rho_{2}>\rho_{1}>0$ depending
on the masses such that for both $0<\rho<\rho_{1}$ and $\rho>\rho_{2}$ system
(1.1)–(1.2) has a positive radial solution. Recently, T. Bartsch, X. Zhong and
W. Zou in [7] obtained the existence of normalized solutions for any given
$b_{1},b_{2}>0$ and $\rho>0$ in a large range, which is independent of the
masses. They also have a result about the nonexistence of positive solutions
which shows that their existence theorem is almost optimal. Therefore our
results indicate that the quadratic interaction term not only enriches the set
of solutions to the above Schrödinger system but also expands the permissible
range of $\rho$.

Furthermore, we give a mass collapse behavior of the ground states obtained in
Theorem 1.3. From Theorem 1.2 of [49], we know that
$(u_{0},v_{0})=(\sqrt{2}\beta^{-1}w,\beta^{-1}w)$ is the unique positive
solution of

$\begin{cases}-\Delta u+u=\beta uv&\text{in}\ \ \mathbb{R}^{3},\\\ -\Delta
v+v=\frac{\beta}{2}u^{2}&\text{in}\ \ \mathbb{R}^{3},\end{cases}$ (1.12)

where $w$ is the unique positive solution of

$-\Delta u+u=u^{2},\ u\in H^{1}(\mathbb{R}^{3}).$ (1.13)

######  Theorem 1.4.

Assume that the assumptions in Theorem 1.3 hold, and
$(u_{b_{1},b_{2}},v_{b_{1},b_{2}})$ is a ground state for (1.1)-(1.2). Up to a
subsequence, we have

$\big{(}L^{-1}_{1}u_{b_{1},b_{2}}(\theta^{-1}_{1}(x)),L^{-1}_{2}v_{b_{1},b_{2}}(\theta^{-1}_{2}(x))\big{)}\rightarrow(\bar{u},\bar{v})$

in $H^{1}(\mathbb{R}^{3})\times H^{1}(\mathbb{R}^{3})$ as $b_{1},b_{2}\to 0$
and $b_{1}\backsim b_{2}$, where $(\bar{u},\bar{v})$ satisfies

$\begin{cases}-\Delta u+\lambda^{\ast}_{1}u=\beta uv&\text{in}\ \
\mathbb{R}^{3},\\\ -\Delta
v+\lambda^{\ast}_{2}v=\frac{\beta}{2}u^{2}&\text{in}\ \
\mathbb{R}^{3},\end{cases}$

for some $\lambda^{\ast}_{1},\ \lambda^{\ast}_{2}>0$, where
$\theta_{1}=\frac{2\beta^{2}b_{1}^{\frac{6}{5}}b_{2}^{\frac{4}{5}}}{16^{\frac{2}{5}}\|w\|^{2}_{L^{2}(\mathbb{R}^{3})}},\
\theta_{2}=\frac{\beta^{2}b_{1}^{\frac{8}{5}}b_{2}^{\frac{2}{5}}}{16^{\frac{1}{5}}\|w\|^{2}_{L^{2}(\mathbb{R}^{3})}},\
L_{1}=\frac{2\beta^{4}b_{1}^{\frac{14}{5}}b_{2}^{\frac{6}{5}}}{16^{\frac{3}{5}}\|w\|^{4}_{L^{2}(\mathbb{R}^{3})}}$,
$L_{2}=\frac{4\beta^{4}b_{1}^{\frac{12}{5}}b_{2}^{\frac{8}{5}}}{16^{\frac{4}{5}}\|w\|^{4}_{L^{2}(\mathbb{R}^{3})}}$.
Moreover, if $\lambda^{\ast}_{1}=\lambda^{\ast}_{2}$, then
$(\bar{u},\bar{v})=(\sqrt{2}\beta^{-1}w,\beta^{-1}w)$.

Denote the set of ground states to (1.1)-(1.2) by $\mathcal{M}_{b_{1},b_{2}}$.
If $(u,v)$ solves (1.1) with some $\lambda_{1},\lambda_{2}\in\mathbb{R}$, then
$\Phi(z,x)=e^{-i\lambda_{1}z}u(x),\ \Psi(z,x)=e^{-i\lambda_{2}z}v(x)$
satisfies the time-dependent system

$\begin{cases}i\partial_{z}\Phi-\Delta\Phi=\beta\Phi\Psi+\mu_{1}\Phi^{3}+\rho\Psi^{2}\Phi&\text{in}\
\ \mathbb{R}^{3},\\\
i\partial_{z}\Psi-\Delta\Psi=\frac{\beta}{2}\Phi^{2}+\mu_{2}\Psi^{3}+\rho\Phi^{2}\Psi&\text{in}\
\ \mathbb{R}^{3},\end{cases}$ (1.14)

where $(z,x)\in\mathbb{R}\times\mathbb{R}^{3},\ i=\sqrt{-1}$. From [10], we
know the local well-posedness of solutions to (1.14) holds for $\beta>0$, then
we can consider the stability of $\mathcal{M}_{b_{1},b_{2}}$. The set
$\mathcal{M}_{b_{1},b_{2}}$ is said to be stable under the Cauchy flow of
(1.14) if for any $\epsilon>0$, there exists $\delta>0$ such that for any
$(\Phi_{0},\Psi_{0})\in H^{1}(\mathbb{R}^{3})\times H^{1}(\mathbb{R}^{3})$
satisfying

$dist_{H^{1}(\mathbb{R}^{3})\times
H^{1}(\mathbb{R}^{3})}((\Phi_{0},\Psi_{0}),\mathcal{M}_{b_{1},b_{2}})<\delta,$

then the solution $((\Phi(z,\cdot),\Psi(z,\cdot))$ of (1.14) with
$((\Phi(0,\cdot),\Psi(0,\cdot))=(\Phi_{0},\Psi_{0})$ satisfies

$\sup dist_{H^{1}(\mathbb{R}^{3})\times
H^{1}(\mathbb{R}^{3})}((\Phi(z,\cdot),\Psi(z,\cdot)),\mathcal{M}_{b_{1},b_{2}})<\epsilon,$

where $Z$ is the maximal existence time for $((\Phi(z,\cdot),\Psi(z,\cdot))$.
We have

######  Theorem 1.5.

When $\mu_{1},\mu_{2},\rho,\beta>0$ and

$\beta\left(2b^{\frac{3}{2}}_{1}+b^{\frac{3}{2}}_{2}\right)C^{3}_{3,3}C^{2}_{3,4}\sqrt{\mu_{1}b_{1}+\mu_{2}b_{2}+\rho
b^{\frac{1}{2}}_{1}b^{\frac{1}{2}}_{2}}<\frac{2\sqrt{6}}{3},$

then the set $\mathcal{M}_{b_{1},b_{2}}$ is compact, up to translation, and it
is stable.

######  Remark 1.3.

Theorem 1.5 indicates that the small quadratic interaction term leads to a
stabilizations of standing waves corresponding to (1.14). Indeed, when
$\beta=0$, T. Bartsch, L. Jeanjean and N. Soave in [5] showed that the
associated solitary wave is orbitally unstable by blowing up in finite time.
Therefore, by creating a gap in the ground state energy level of the system
(from positive to negative), the quadratic coupling term not only makes the
ground state solutions stable, but also changes the structure of the energy
functional and enriches the solution set.

Now, we deal with the four dimensional case. In this case,
$m_{\beta}(b_{1},b_{2})=-\infty$ for any $b_{1},b_{2}>0.$ Indeed, the leading
term is $L^{2}$-critical and Sobolev critical, the energy function
$J_{\beta}(u,v)$ is unbounded both from above and from below on
$\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$. Denote
$\mathcal{S}=\inf_{D^{1,2}(\mathbb{R}^{4})\backslash\\{0\\}}\frac{\|\nabla
u\|^{2}_{L^{2}(\mathbb{R}^{4})}}{\|u\|^{2}_{L^{4}(\mathbb{R}^{4})}}.$ From
[44], we know that $\mathcal{S}$ is attained by the Aubin-Talanti bubbles

$\displaystyle
U_{\epsilon}(x):=\frac{2\sqrt{2}\epsilon}{\epsilon^{2}+|x|^{2}},\ \epsilon>0,\
x\in\mathbb{R}^{4}.$ (1.15)

Then $U_{\epsilon}$ satisfies $-\Delta u=u^{3}$ and
$\int_{\mathbb{R}^{4}}|\nabla
U_{\epsilon}|^{2}dx=\int_{\mathbb{R}^{4}}|U_{\epsilon}|^{4}dx=\mathcal{S}^{2}$.
On the other hand, if $0<\rho<\min\\{\mu_{1},\mu_{2}\\}$ or
$\rho>\max\\{\mu_{1},\mu_{2}\\}$(see Lemma 6.3),
$\left(\sqrt{\frac{\rho-\mu_{2}}{\rho^{2}-\mu_{1}\mu_{2}}}U_{\epsilon},\sqrt{\frac{\rho-\mu_{1}}{\rho^{2}-\mu_{1}\mu_{2}}}U_{\epsilon}\right)$
is the least energy solutions to the following elliptic system:

$\begin{cases}-\Delta u=\mu_{1}u^{3}+\rho v^{2}u&\text{in}\ \
\mathbb{R}^{4},\\\ -\Delta v=\mu_{2}v^{3}+\rho u^{2}v&\text{in}\ \
\mathbb{R}^{4},\\\ u,v\in D^{1,2}(\mathbb{R}^{4}).\end{cases}$ (1.16)

Define

$\mathcal{S}_{\mu_{1},\mu_{2},\rho}=\inf_{(u,v)\in[D^{1,2}(\mathbb{R}^{4})]^{2}\backslash\\{(0,0)\\}}\frac{\int_{\mathbb{R}^{4}}(|\nabla
u|^{2}+|\nabla
v|^{2})dx}{\left(\int_{\mathbb{R}^{4}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx\right)^{\frac{1}{2}}}.$ (1.17)

The main results on this aspect can be stated as follows:

If $\beta<0$, we have the following non-existence results.

######  Theorem 1.6.

Let $\mu_{i},b_{i},\rho>0(i=1,2)$ and $\beta<0$.

  1. 1.

If $N=4$, then problem (1.1)-(1.2) has no positive solution $(u,v)\in
H^{1}(\mathbb{R}^{4})\times H^{1}(\mathbb{R}^{4})$.

  2. 2.

If $N=5$, problem (1.1)-(1.2) has no positive solution $(u,v)\in
H^{1}(\mathbb{R}^{5})\times H^{1}(\mathbb{R}^{5})$ satisfying the additional
assumption that $u\in L^{p}(\mathbb{R}^{5})$ for some $p\in(0,\frac{5}{3}]$.

  3. 3.

Moreover, if $N\geq 4$, problem (1.1)-(1.2) has no non-trivial radial
solution.

Next, we consider the case $\beta>0$.

######  Theorem 1.7.

When $N=4$, let $\mu_{i},b_{i}>0(i=1,2)$, $\beta>0$, and
$\rho\in\big{(}0,\min\\{\mu_{1},\mu_{2}\\}\big{)}\cup\big{(}\max\\{\mu_{1},\mu_{2}\\},\infty\big{)}$,
then the following conclusions hold.

  1. 1.

If $0<\beta b_{1}<\\!\frac{3}{2|C_{4,3}|^{3}}$ and $0<\beta
b_{2}<\\!\frac{3}{|C_{4,3}|^{3}}$, then (1.1)-(1.2) has a positive ground
state solution
$(u_{b_{1},b_{2}},v_{b_{1},b_{2}})\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$.

  2. 2.

Moreover, there exists $\sigma_{1}>0$ such that

$\big{(}\sigma_{1}u_{b_{1},b_{2}}(\sigma_{1}x),\sigma_{1}v_{b_{1},b_{2}}(\sigma_{1}x)\big{)}\to\Big{(}\sqrt{\frac{\rho-\mu_{2}}{\rho^{2}-\mu_{1}\mu_{2}}}U_{\varepsilon_{0}},\sqrt{\frac{\rho-\mu_{1}}{\rho^{2}-\mu_{1}\mu_{2}}}U_{\varepsilon_{0}}\Big{)}$

in $D^{1,2}(\mathbb{R}^{4})\times D^{1,2}(\mathbb{R}^{4})$, for some
$\varepsilon_{0}>0$ as $(b_{1},b_{2})\to(0,0)$, up to a subsequence.

######  Remark 1.4.

If $N=4$, in problem (1.1), $\beta uv$ and $\frac{\beta}{2}u^{2}$ can be
regarded as mass-critical terms. Moreover, the terms $u^{3},v^{3}$ and the
coupled terms $u^{2}v,v^{2}u$ are mass super-critical and energy critical.
Theorem 1.7 indicate that problem (1.1) with mass critical lower order
perturbation term possesses at least one normalized ground state solution,
whose two components both converge to the Aubin-Talanti bubble in related
Sobolev space by making appropriate scaling, as the masses of two components
vanish. More recently, in [31] jointly with W. Zou, the first and third
authors in this present paper considered the equation

$\begin{cases}-\Delta u+\lambda_{1}u=\alpha_{1}|u|^{p-2}u+\mu_{1}u^{3}+\beta
v^{2}u\quad&\hbox{in}~{}\mathbb{R}^{4},\\\ -\Delta
v+\lambda_{2}v=\alpha_{2}|v|^{p-2}v+\mu_{2}v^{3}+\beta
u^{2}v\quad&\hbox{in}~{}\mathbb{R}^{4},\\\ \end{cases}$ (1.18)

under the mass constraint
$(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$, where
$\mu_{1},\mu_{2},\beta>0$, $\alpha_{1},\alpha_{2}\in\mathbb{R}$,
$p\\!\in\\!(2,4)$ and $\lambda_{1},\lambda_{2}\\!\in\\!\mathbb{R}$ appear as
Lagrange multipliers. We must point out that compared with [31], the problem
(1.1) in $\mathbb{R}^{4}$ is more delicate due to the uncertainty of sign of
the quadratic interaction term and inhomogeneity of the coupling term.
Moreover, since system (1.1) with mixed couplings is asymmetric, the
permissible range of $b_{1},b_{2}$ obtained in Theorem 1.7 may not be optimal,
so it is difficult to prove the non-existence of normalized solutions to (1.1)
for larger $b_{1}$ and $b_{2}$.

Finally, we give outline of the proofs. In the one dimension case, the energy
functional is bounded from below on the product of $L^{2}$-spheres, the
constrained minimization method developed by L. Jeanjean [17] can be used to
obtained a normalized ground state, which is obtained by establishing the
compactness of the minimizing sequences. When $N=3$, the energy functional is
always unbounded on the product of $L^{2}$-spheres. We use the ideas
introduced by N. Soave [42, 41] to study the related fiber maps
$\Psi_{u,v}(t)$(see (4.2)). It is easy to see that the critical point of
$\Psi_{u,v}(t)$ allow to project a function on $\mathcal{P}_{b_{1},b_{2}}$(see
(4.1)) and the monotonicity and convexity properties of $\Psi_{u,v}(t)$ has a
strongly affect the structure of $\mathcal{P}_{b_{1},b_{2}}$ and then
intimately related to the minimax structure of
$J_{\beta}(u,v)|_{\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}}$.

However, for the system (1.1) we study, due to the uncertainty of the sign of
the term $\beta\int_{\mathbb{R}^{N}}u^{2}vdx$ in the corresponding energy
functional, the above method cannot be used directly, and we need to introduce
additional constrains on the previous Pohozaev manifold. On the new manifold
with additional constrains $\mathcal{P}_{b_{1},b_{2}}$(see (4.3)-(4.5)), under
suitable condition on $b_{1}$ and $b_{2}$, we can prove that
$J_{\beta}(u,v)|_{\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}}$ admits a
convex-concave geometry and the new manifold $\mathcal{P}_{b_{1},b_{2}}$ is
non-empty. To show that $\mathcal{P}_{b_{1},b_{2}}$ is a natural constraint,
we use some ideas introduced by F. Clarke in [13] and already used by J.
Mederski and J. Schino in [33] to deal with minimization problems whose
constraints are given by inequalities. Therefore,
$J_{\beta}(u,v)|_{\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}}$ has a local
minimizer and a mountain pass critical point. By establish the compactness of
minimizing sequence, we obtain a solution which is a local minimizer. The
second solution is obtained through a constrained mountain pass. It is worth
pointing out that after proving the strongly convergence of minimizing
sequences, we need to verify that the limit function $(u,v)$ is still in the
allowed set $\mathcal{P}_{b_{1},b_{2}}$. Indeed, if
$\beta\int_{\mathbb{R}^{N}}u^{2}vdx\leq 0,$ then the local minimum point will
disappear, thus we get a contradiction.

When $N=3$, obviously, we obtain that
$J_{\beta}(u,v)|_{\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}}$ has no local
minimizer as $\beta\rightarrow 0^{+}$, i.e. the ground state energy converges
to $0$ as $\beta\to 0^{+}$. The mountain pass solutions
$(\widetilde{u}_{\beta},\widetilde{v}_{\beta})$ obtained in Theorem 1.3
depends on $\beta$, we shall analyze the convergence properties of
$(\widetilde{u}_{\beta},\widetilde{v}_{\beta})$ as $\beta\to 0^{+}$. We first
deduce that for the function $\beta\in[0,\infty)\mapsto
m_{\beta}(b_{1},b_{2})\in\mathbb{R}$ is monotone non-increasing, and
$\\{(\widetilde{u}_{\beta},\widetilde{v}_{\beta})\\}$ is bounded in
$H^{1}(\mathbb{R}^{3})\times H^{1}(\mathbb{R}^{3})$. Then there exists a
subsequence, such that $(\widetilde{u}_{\beta},\widetilde{v}_{\beta})$
converge to $(\widetilde{u}_{0},\widetilde{v}_{0})$ strongly in
$H^{1}(\mathbb{R}^{3})\times H^{1}(\mathbb{R}^{3})$ as $\beta\to 0^{+}$, where
$(\widetilde{u}_{0},\widetilde{v}_{0})$ is a ground state solution of
(1.1)-(1.2) with $\beta=0$.

In the proof of Theorem 1.4, the main ingredient is the refined upper bound of
$m^{+}(b_{1},b_{2})$ (see Lemma 5.2) i.e.

$m^{+}_{\beta}(b_{1},b_{2})<-\frac{1}{6}\Big{[}\frac{4\beta^{4}b_{1}^{\frac{22}{5}}b_{2}^{\frac{8}{5}}}{16^{\frac{4}{5}}\|w\|^{6}_{L^{2}(\mathbb{R}^{3})}}+\frac{\beta^{4}b_{1}^{\frac{16}{5}}b_{2}^{\frac{14}{5}}}{16^{\frac{2}{5}}\|w\|^{6}_{L^{2}(\mathbb{R}^{3})}}\Big{]}\|\nabla
w\|^{2}_{L^{2}(\mathbb{R}^{3})}.$

This refinement needs to keep the testing functions staying in the admissible
set
$A_{R_{0}}=\big{\\{}(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}:\|\nabla
u\|^{2}_{L^{2}(\mathbb{R}^{3})}+\|\nabla
v\|^{2}_{L^{2}(\mathbb{R}^{3})}<R^{2}_{0}\big{\\}}$. We overcome this
difficulty by utilising the properties of the unique positive radial ground
state solution of (1.12). By accurate estimation and careful analysis, we give
a precise description of the asymptotic behavior of solutions as the mass
$b_{1},b_{2}$ goes to zero.

To prove item 1 of Theorem 1.7, we follow the approach of [46]. We can obtain
a bounded minimizing sequence by using the Pohozaev constraint
approach(see[41, 42]). However, it is very difficult to prove the compactness
of a minimizing sequence at positive energy levels. Motivated by [46], we
drive a better energy estimate on the associated mountain pass energy level
(see Lemma 6.7 ), i.e

$0<m_{\beta}(b_{1},b_{2})<\frac{k_{1}+k_{2}}{4}\mathcal{S}^{2}.$

This is enough to guarantee the compactness of minimizing sequences at the
energy level $m_{\beta}(b_{1},b_{2}).$ To prove item 2 of Theorem 1.7, we
first prove the following energy splitting asymptotic property of the solution

$||\nabla u_{b_{1},b_{2}}||^{2}_{2}+\|\nabla
v_{b_{1},b_{2}}\|^{2}_{2}\to(k_{1}+k_{2})\mathcal{S}^{2},\ \
\mu_{1}\|u_{b_{1},b_{2}}\|^{4}_{4}+\mu_{2}\|v_{b_{1},b_{2}}\|^{4}_{4}+2\rho\|u_{b_{1},b_{2}}v_{b_{1},b_{2}}\|^{2}_{2}\to(k_{1}+k_{2})\mathcal{S}^{2},$

as $(b_{1},b_{2})\to(0,0)$. Then we obtain that
$(u_{b_{1},b_{2}},v_{b_{1},b_{2}})$ is a special minimizing sequence of the
minimizing problem (1.17). We follow some ideas from Lemma 3.5 of [26], and by
the uniqueness of ground state solution of (1.17), we obtain the precisely
asymptotic behavior of $(u_{b_{1},b_{2}},v_{b_{1},b_{2}})$ as
$(b_{1},b_{2})\rightarrow(0,0)$.

Throughout the paper, we shall write $a\backsim b$ if $C_{1}a\leq b\leq
C_{2}a$ where $C_{i},i=1,2$ are constants. $H^{1}_{r}$ denotes the subspace of
functions in $H^{1}$ which are radially symmetric with respect to 0, and
$\mathrm{T}_{b_{i,r}}=\mathrm{T}_{b_{i}}\cap H^{1}_{r},i=1,2$. The rest of
this paper is organized as follows. In section 2, we prove Theorem 1.1. In
section 3, we prove Theorem1.2. In section 4, we prove Theorem 1.3. In section
6, we prove Theorems 1.6 and 1.7.

##  2 Proof of Theorem 1.1

To prove Theorem 1.1, we use the ideas introduced in [16, 17]. First we recall
the rearrangement results of M. Shibata [40] as presented in [22, 16]. Let $u$
be a Borel measurable function on $\mathbb{R}^{N}$. It is said to vanish at
infinity if the level set $|\\{x\in\mathbb{R}^{N}:u(x)>t\\}|<+\infty$ for
every $t>0$. Here $|A|$ stands for the N-dimensional Lebesgue measure of a
Lebesgue measurable set $A\subset\mathbb{R}^{N}$. Considering two Borel
mesurable functions $u,v$ which vanish at infinity in $\mathbb{R}^{N}$, for
$t>0,$ we define $A^{\star}(u,v:t):=\\{x\in\mathbb{R}^{N}:|x|<r\\}$, where
$r>0$ is chosen so that

$B(0,r)=|\\{x\in\mathbb{R}^{N}:|u(x)|>t\\}|+|\\{x\in\mathbb{R}^{N}:|v(x)|>t\\}|,$

and $\\{u,v\\}^{\star}$ by

$\\{u,v\\}^{\star}(x):=\int^{\infty}_{0}\chi_{A^{\star}(u,v:t)}(x)dt,$

where $\chi_{A}(x)$ is a characteristic function of the set
$A\subset\mathbb{R}^{N}.$

######  Lemma 2.1 ([22] Lemma A.1).

(i) The function $\\{u,v\\}^{\star}$ is radially symmetric, non-increasing and
lower semi-continuous. More, for each $t>0$ there holds
$\\{x\in\mathbb{R}^{N}:\\{u,v\\}^{\star}>t\\}=A^{\star}(u,v:t)$.

(ii) Let $\Phi:[0,\infty)\rightarrow[0,\infty)$ be non-decreasing lower semi-
continuous, continuous at 0 and $\Phi(0)=0.$ Then
$\\{\Phi(u),\Phi(v)\\}^{\star}=\Phi(\\{u,v\\}^{\star})$.

(iii)
$\|\\{u,v\\}^{\star}\|^{p}_{L^{p}(\mathbb{R}^{N})}=\|u\|^{p}_{L^{p}(\mathbb{R}^{N})}+\|v\|^{p}_{L^{p}(\mathbb{R}^{N})}$
for $1\leq p<\infty.$

(iv) If $u,v\in H^{1}(\mathbb{R}^{N})$, then $\\{u,v\\}^{\star}\in
H^{1}(\mathbb{R}^{N})$ and

$\|\nabla\\{u,v\\}^{\star}\|^{2}_{L^{2}(\mathbb{R}^{N})}\leq\|\nabla
u\|^{2}_{L^{2}(\mathbb{R}^{N})}+\|\nabla v\|^{2}_{L^{2}(\mathbb{R}^{N})}.$

In addition, if $u,v\in(H^{1}(\mathbb{R}^{N})\cap
C^{1}(\mathbb{R}^{N}))\setminus\\{0\\}$ are radially symmetric, positive and
non-increasing, then

$\int_{\mathbb{R}^{N}}|\nabla\\{u,v\\}^{\star}|^{2}dx<\int_{\mathbb{R}^{N}}|\nabla
u|^{2}dx+\int_{\mathbb{R}^{N}}|\nabla v|^{2}dx.$

(v) Let $u_{1},u_{2},v_{1},v_{2}\geq 0$ be Borel measurable functions which
vanish at infinity, then

$\int_{\mathbb{R}^{N}}(u_{1}u_{2}+v_{1}v_{2})dx\leq\int_{\mathbb{R}^{N}}\\{u_{1},v_{1}\\}^{\star}\\{u_{2},v_{2}\\}^{\star}dx.$

The solution of (1.1)–(1.2) can be found as a critical point of the following
energy functional

$J_{\beta}(u,v)=\frac{1}{2}\int_{\mathbb{R}^{N}}(|\nabla u|^{2}+|\nabla
v|^{2})dx-\frac{1}{4}\int_{\mathbb{R}^{N}}(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2})dx-\frac{\beta}{2}\int_{\mathbb{R}^{N}}u^{2}vdx.$

From Gagliardo-Nirenberg-Sobolev inequality (1.11), we have

$\displaystyle\frac{\mu_{1}}{4}\int_{\mathbb{R}^{N}}u^{4}dx\leq\frac{\mu_{1}}{4}C^{4}_{N,4}b^{4-N}_{1}\|\nabla
u\|^{N}_{L^{2}(\mathbb{R}^{N})},\ \
\frac{\mu_{2}}{4}\int_{\mathbb{R}^{N}}v^{4}dx\leq\frac{\mu_{2}}{4}C^{4}_{N,4}b^{4-N}_{2}\|\nabla
v\|^{N}_{L^{2}(\mathbb{R}^{N})},$ (2.1)
$\displaystyle\int_{\mathbb{R}^{N}}u^{2}v^{2}dx$
$\displaystyle\leq\left(\int_{\mathbb{R}^{N}}u^{4}dx\right)^{\frac{1}{2}}\left(\int_{\mathbb{R}^{N}}v^{4}dx\right)^{\frac{1}{2}}$
(2.2) $\displaystyle\leq
C^{4}_{N,4}b^{\frac{4-N}{2}}_{1}b^{\frac{4-N}{2}}_{2}\|\nabla
u\|^{\frac{N}{2}}_{L^{2}(\mathbb{R}^{N})}\|\nabla
v\|^{\frac{N}{2}}_{L^{2}(\mathbb{R}^{N})}$
$\displaystyle\leq\frac{1}{2}C^{4}_{N,4}b^{\frac{4-N}{2}}_{1}b^{\frac{4-N}{2}}_{2}\left[\|\nabla
u\|^{2}_{L^{2}(\mathbb{R}^{N})}+\|\nabla
u\|^{2}_{L^{2}(\mathbb{R}^{N})}\right]^{\frac{N}{2}},$
$\displaystyle\beta\int_{\mathbb{R}^{N}}u^{2}vdx$
$\displaystyle\leq|\beta|\left(\int_{\mathbb{R}^{N}}|u|^{3}dx\right)^{\frac{2}{3}}\left(\int_{\mathbb{R}^{N}}|v|^{3}dx\right)^{\frac{1}{3}}$
(2.3)
$\displaystyle\leq|\beta|\left[\frac{2}{3}\int_{\mathbb{R}^{N}}|u|^{3}dx+\frac{1}{3}\int_{\mathbb{R}^{N}}|v|^{3}dx\right]$
$\displaystyle\leq|\beta|\left[\frac{2}{3}C^{3}_{N,3}b^{\frac{6-N}{2}}_{1}\|\nabla
u\|^{\frac{N}{2}}_{L^{2}(\mathbb{R}^{N})}+\frac{1}{3}C^{3}_{N,3}b^{\frac{6-N}{2}}_{2}\|\nabla
v\|^{\frac{N}{2}}_{L^{2}(\mathbb{R}^{N})}\right]$
$\displaystyle\leq|\beta|\left[\left(\frac{2}{3}b^{\frac{6-N}{2}}_{1}+\frac{1}{3}b^{\frac{6-N}{2}}_{2}\right)C^{3}_{N,3}\left[\|\nabla
u\|^{2}_{L^{2}(\mathbb{R}^{N})}+\|\nabla
v\|^{2}_{L^{2}(\mathbb{R}^{N})}\right]^{\frac{N}{4}}\right].$

Next, we show that $m(b_{1},b_{2})<0.$ We now focus on Sobolev subcritical
nonlinear Schrödinger equation with prescribed $L^{2}$ norm. For fixed
$\mu>0,2<p<2+\frac{4}{N}$, we search for $H^{1}$ and $\lambda\in\mathbb{R}$
solving

$\begin{cases}-\Delta u+u=\mu|u|^{p-2}u&\text{in}\ \ \mathbb{R}^{N},\\\
\int_{\mathbb{R}^{N}}u^{2}dx=b^{2}.\end{cases}$ (2.4)

Solutions of (2.4) can be found as critical points of
$J_{\mu,p}:H^{1}\rightarrow\mathbb{R}$,

$\displaystyle J_{\mu,p}=\frac{1}{2}\int_{\mathbb{R}}|\nabla
u_{n}|^{2}dx-\frac{\mu}{p}\int_{\mathbb{R}}|u|^{p}dx$ (2.5)

constrained on $S_{b}=\\{u\in H^{1}|\int_{\mathbb{R}^{N}}u^{2}dx=b^{2}\\}$.

######  Lemma 2.2.

When $N=1$, for any $\rho>0,\beta>0$, we have

$m_{\beta}(b_{1},b_{2})=\inf_{(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}}J_{\beta}(u,v)<0.$

###### Proof.

From (2.1), (2.2) and (2.3), we can deduce that

$\displaystyle J_{\beta}(u,v)$
$\displaystyle=\frac{1}{2}\int_{\mathbb{R}}(|\nabla u|^{2}+|\nabla
v|^{2})dx-\frac{1}{4}\int_{\mathbb{R}}(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2})dx-\frac{\beta}{2}\int_{\mathbb{R}}u^{2}vdx$
$\displaystyle\geq\frac{1}{2}\int_{\mathbb{R}}(|\nabla u|^{2}+|\nabla
v|^{2})dx-\left[\frac{\mu_{1}}{4}C^{4}_{1,4}b^{3}_{1}+\frac{\mu_{2}}{4}C^{4}_{1,4}b^{3}_{2}+\frac{\rho}{4}C^{4}_{1,4}b^{\frac{3}{2}}_{1}b^{\frac{3}{2}}_{2}\right]\left[\|\nabla
u\|^{2}_{L^{2}(\mathbb{R})}+\|\nabla
u\|^{2}_{L^{2}(\mathbb{R})}\right]^{\frac{1}{2}}$
$\displaystyle\quad-\frac{|\beta|}{2}\left[\left(\frac{2}{3}b^{\frac{5}{2}}_{1}+\frac{1}{3}b^{\frac{5}{2}}_{2}\right)C^{3}_{1,3}\left[\|\nabla
u\|^{2}_{L^{2}(\mathbb{R})}+\|\nabla
v\|^{2}_{L^{2}(\mathbb{R})}\right]^{\frac{1}{4}}\right],$

so $J$ is coercive and in particular $m_{\beta}(b_{1},b_{2})>-\infty.$ Since

$\displaystyle J_{\beta}(u,v)$
$\displaystyle=\frac{1}{2}\int_{\mathbb{R}}(|\nabla u|^{2}+|\nabla
v|^{2})dx-\frac{\beta}{2}\int_{\mathbb{R}}u^{2}vdx-\frac{1}{4}\int_{\mathbb{R}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx$
$\displaystyle=J_{\mu_{1},4}(u)+J_{\mu_{2},4}(v)-\frac{1}{2}\int_{\mathbb{R}}\rho
u^{2}v^{2}dx-\frac{\beta}{2}\int_{\mathbb{R}}u^{2}vdx,$

where $J_{\mu_{i},4},i=1,2$ is defined in (2.5). Then we choose
$u=u_{\mu_{1},p,b_{1}}$ and $v=u_{\mu_{2},4,b_{2}}$, where
$u_{\mu_{i},4,b_{i}}(i=1,2)$ is the unique positive solution of (2.4) with $b$
and $\mu$ are replaced by $b_{i}$ and $\mu_{i}$. We have
$J_{\mu_{1},4}(u_{\mu_{i},4,b_{i}})<0,$ (see details in Lemma 2.1 of [11]).
Therefore

$m_{\beta}(b_{1},b_{2})\leq
J_{\beta}(u_{\mu_{1},4,b_{1}},u_{\mu_{2},4,b_{2}})<0,$

so for any $\beta>0$, we obtain

$m_{\beta}(b_{1},b_{2})=\inf_{(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}}J_{\beta}(u,v)<0.$

∎

######  Lemma 2.3.

Let $\\{(u_{n},v_{n})\\}\subset\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$ be
a minimizing sequence for $m(b_{1},b_{2})$. Then for $\beta>0$,
$\\{(|u_{n}|,|v_{n}|)\\}$ is also a minimizing sequence.

###### Proof.

Since

$\int_{\mathbb{R}^{N}}|\nabla|u_{n}||^{2}dx\leq\int_{\mathbb{R}}|\nabla
u_{n}|^{2}dx,\
\int_{\mathbb{R}^{N}}|\nabla|v_{n}||^{2}dx\leq\int_{\mathbb{R}}|\nabla
v_{n}|^{2}dx,$
$\int_{\mathbb{R}}u_{n}^{2}v_{n}dx\leq\int_{\mathbb{R}}|u_{n}|^{2}|v_{n}|dx,$

we have

$\displaystyle J_{\beta}(|u_{n}|,|v_{n}|)$
$\displaystyle=\frac{1}{2}\int_{\mathbb{R}}(|\nabla|u_{n}||^{2}+|\nabla|v_{n}||^{2})dx-\frac{\beta}{2}\int_{\mathbb{R}}|u_{n}|^{2}|v_{n}|dx$
$\displaystyle\quad-\frac{1}{4}\int_{\mathbb{R}}\left(\mu_{1}|u_{n}|^{4}+\mu_{2}|v_{n}|^{4}+2\rho|u_{n}|^{2}|v_{n}|^{2}\right)dx$
$\displaystyle\leq\frac{1}{2}\int_{\mathbb{R}}(|\nabla u_{n}|^{2}+|\nabla
v_{n}|^{2})dx-\frac{\beta}{2}\int_{\mathbb{R}}u_{n}^{2}v_{n}dx$
$\displaystyle\quad-\frac{1}{4}\int_{\mathbb{R}}\left(\mu_{1}u_{n}^{4}+\mu_{2}v_{n}^{4}+2\rho
u_{n}^{2}v_{n}^{2}\right)dx=J_{\beta}(u_{n},v_{n}).$

Thus, $J_{\beta}(|u_{n}|,|v_{n}|)\leq J_{\beta}(u_{n},v_{n}).$ Therefore, when
$\beta>0$, $\\{(|u_{n}|,|v_{n}|)\\}$ is also a minimizing sequence. ∎

######  Lemma 2.4.

(i) If $(d^{n}_{1},d^{n}_{2})$ is such that
$(d^{n}_{1},d^{n}_{2})\rightarrow(d_{1},d_{2})$ as $n\rightarrow+\infty$ with
$0\leq d^{n}_{i}\leq b_{i}$ for $i=1,2$, we have
$m_{\beta}(d^{n}_{1},d^{n}_{2})\rightarrow m_{\beta}(d_{1},d_{2})$ as
$n\rightarrow+\infty$.

(ii) Let $d_{i}\geq 0,\ b_{i}\geq 0,i=1,2$ such that
$b^{2}_{1}+d^{2}_{1}=c_{1}^{2},\ b^{2}_{2}+d^{2}_{2}=c_{2}^{2}$, then
$m_{\beta}(b_{1},b_{2})+m_{\beta}(d_{1},d_{2})\geq m_{\beta}(c_{1},c_{2}).$

###### Proof.

The proof is similar to that of Lemma 3.1 in [16], we omit the details here. ∎

######  Lemma 2.5.

For any $\rho>0,\beta>0$, let $(u_{n},v_{n})\subset H^{1}(\mathbb{R})\times
H^{1}(\mathbb{R})$ be a sequence such that

$J_{\beta}(u_{n},v_{n})\rightarrow m_{\beta}(b_{1},b_{2})\ \text{and}\
\int_{\mathbb{R}}u_{n}^{2}dx=b^{2}_{1},\
\int_{\mathbb{R}^{N}}v_{n}^{2}dx=b^{2}_{2},$

then $\\{(u_{n},v_{n})\\}$ is relatively compact in $H^{1}(\mathbb{R})\times
H^{1}(\mathbb{R})$ up to translations, that is there exists a subsequence
$(u_{n_{k}},v_{n_{k}})$, a sequence of points
$\\{y_{k}\\}\subset\mathbb{R}^{N}$ and a function
$(\widetilde{u},\widetilde{v})\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}\times
H^{1}(\mathbb{R})\times H^{1}(\mathbb{R})$ such that
$(u_{n_{k}}(\cdot+y_{k}),v_{n_{k}}(\cdot+y_{k}))\rightarrow(\widetilde{u},\widetilde{v})$
strongly in $H^{1}(\mathbb{R})\times H^{1}(\mathbb{R}).$

###### Proof.

We use the ideas introduced in [16, 17]. Assume that $(u^{n}_{1},u^{n}_{2})$
is a minimizing sequence associated to the functional $J$ on
$\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$, from (3.7) and the coerciveness
of functional $J$ on $\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$, the
sequence $(u^{n}_{1},u^{n}_{2})$ is bounded in $H^{1}(\mathbb{R}^{N})\times
H^{1}(\mathbb{R}).$ If

$\sup_{y\in\mathbb{R}}\int_{B(y,R)}(v_{n}^{2}+v_{n}^{2})dx=o(1),\ \text{for
some}\ R>0,$

then by Vanishing lemma (see Lemma I.1 in [27]), we have that
$(u_{n},v_{n})\rightarrow(0,0)$ in $2<p<2^{\ast}$, contrary with the fact
$m(b_{1},b_{2})<0$, therefore there exists a $\beta_{0}>0$ and a sequence
$\\{y_{n}\\}\subset\mathbb{R}$ such that

$\int_{B(y_{n},R)}(v_{n}^{2}+v_{n}^{2})dx\geq\beta_{0}>0.$

Since $(u^{n}_{1},u^{n}_{2})$ is bounded in $H^{1}(\mathbb{R})\times
H^{1}(\mathbb{R})$, $(u^{n}_{1},u^{n}_{2})$ is weakly convergence in
$H^{1}(\mathbb{R})\times H^{1}(\mathbb{R})$ and local compactness in
$L^{2}(\mathbb{R})\times L^{2}(\mathbb{R})$, so

$(u_{n}(x-y_{n}),v_{n}(x-y_{n}))\rightharpoonup(u,v)\neq(0,0)\ \text{in}\
H^{1}(\mathbb{R})\times H^{1}(\mathbb{R}).$

Let

$w_{n}(x)=u_{n}(x)-u(x+y_{n}),\ \sigma_{n}(x)=v_{n}(x)-v(x+y_{n}),$

we show that $w_{n}(x)\rightarrow 0,\sigma_{n}(x)\rightarrow 0$ in
$L^{p}(\mathbb{R})$ for $2<p<2^{\star}$, suppose by contradiction that there
exists a sequence $\\{z_{n}\\}\subset\mathbb{R}^{N}$ such that

$(w_{n}(x-z_{n}),\sigma_{n}(x-z_{n}))\rightharpoonup(w,\sigma)\neq(0,0)\
\text{in}\ H^{1}(\mathbb{R})\times H^{1}(\mathbb{R}).$

By the Brezis-Lieb lemma and Lemma 2.4 in [17], we have

$\displaystyle J_{\beta}(u_{n},v_{n})$
$\displaystyle=J_{\beta}(u_{n}(x-y_{n}),v_{n}(x-y_{n}))$ (2.6)
$\displaystyle=J_{\beta}(w_{n}(x-y_{n}),\sigma_{n}(x-y_{n}))+J_{\beta}(u,v)+o(1)$
$\displaystyle=J_{\beta}(w_{n}(x-z_{n})-w,\sigma_{n}(x-z_{n})-\sigma)+J_{\beta}(w,\sigma)+J_{\beta}(u,v)+o(1),$
$\displaystyle\|u_{n}(x-y_{n})\|^{2}_{L^{2}(\mathbb{R})}$
$\displaystyle=\|w_{n}(x-z_{n})\|^{2}_{L^{2}(\mathbb{R})}+\|u\|^{2}_{L^{2}(\mathbb{R})}+o(1)$
$\displaystyle=\|w_{n}(x-z_{n})-w\|^{2}_{L^{2}(\mathbb{R})}+\|w\|^{2}_{L^{2}(\mathbb{R})}+\|u\|^{2}_{L^{2}(\mathbb{R})}+o(1),$

and

$\displaystyle\|v_{n}(x-y_{n})\|^{2}_{L^{2}(\mathbb{R})}$
$\displaystyle=\|\sigma_{n}(x-z_{n})\|^{2}_{L^{2}(\mathbb{R})}+\|v\|^{2}_{L^{2}(\mathbb{R})}+o(1)$
$\displaystyle=\|\sigma_{n}(x-z_{n})-\sigma\|^{2}_{L^{2}(\mathbb{R})}+\|\sigma\|^{2}_{L^{2}(\mathbb{R})}+\|v\|^{2}_{L^{2}(\mathbb{R})}+o(1).$

So, we obtain

$\displaystyle\|w_{n}(x-z_{n})-w\|^{2}_{L^{2}(\mathbb{R})}$
$\displaystyle=b^{2}_{1}-\|w\|^{2}_{L^{2}(\mathbb{R})}-\|u\|^{2}_{L^{2}(\mathbb{R})}+o(1)$
(2.7) $\displaystyle:=d^{2}_{1}+o(1),$
$\displaystyle\|\sigma_{n}(x-z_{n})-\sigma\|^{2}_{L^{2}(\mathbb{R})}$
$\displaystyle=b^{2}_{2}-\|\sigma\|^{2}_{L^{2}(\mathbb{R})}-\|v\|^{2}_{L^{2}(\mathbb{R})}+o(1)$
(2.8) $\displaystyle:=d^{2}_{2}+o(1).$

By (2.6)-(2.8), (i) of Lemma 2.4 and $J_{\beta}(u_{n},v_{n})\rightarrow
m_{\beta}(b_{1},b_{2})$, we obtain

$\displaystyle m_{\beta}(b_{1},b_{2})\geq
m_{\beta}(d_{1},d_{2})+J_{\beta}(w,\sigma)+J_{\beta}(u,v).$ (2.9)

If
$J_{\beta}(w,\sigma)>m_{\beta}(\|w\|_{L^{2}(\mathbb{R})},\|\sigma\|_{L^{2}(\mathbb{R})})$
or
$J_{\beta}(u,v)>m_{\beta}(\|u\|_{L^{2}(\mathbb{R})},\|v\|_{L^{2}(\mathbb{R})})$,
then by (2.7)-(2.9) and (ii) of Lemma 2.4, we have

$\displaystyle m_{\beta}(b_{1},b_{2})$
$\displaystyle>m_{\beta}(d_{1},d_{2})+m_{\beta}(\|w\|_{L^{2}(\mathbb{R})},\|\sigma\|_{L^{2}(\mathbb{R})})+m_{\beta}(\|u\|_{L^{2}(\mathbb{R})},\|v\|_{L^{2}(\mathbb{R})})$
$\displaystyle>m_{\beta}(d_{1},d_{2})+m_{\beta}(\sqrt{b^{2}_{1}-d^{2}_{1}},\sqrt{b^{2}_{2}-d^{2}_{2}})\geq
m_{\beta}(b_{1},b_{2})$

which is impossible. So,

$J_{\beta}(w,\sigma)=m_{\beta}(\|w\|_{L^{2}(\mathbb{R})},\|\sigma\|_{L^{2}(\mathbb{R})}),\
J_{\beta}(u,v)=m_{\beta}(\|u\|_{L^{2}(\mathbb{R})},\|v\|_{L^{2}(\mathbb{R})}).$

Let $\widetilde{w},\widetilde{\sigma},\widetilde{u},\widetilde{v}$ be the
Schwarz symmetric-decreasing rearrangement of $w,\sigma,u,v$. From Section 3.3
in [25], we know that
$\widetilde{w},\widetilde{\sigma},\widetilde{u},\widetilde{v}$ are
nonnegative. Since

$\|\widetilde{w}\|^{2}_{L^{2}(\mathbb{R})}=\|w\|^{2}_{L^{2}(\mathbb{R})},\
\|\widetilde{\sigma}\|^{2}_{L^{2}(\mathbb{R})}=\|\sigma\|^{2}_{L^{2}(\mathbb{R})},$
$\|\widetilde{u}\|^{2}_{L^{2}(\mathbb{R})}=\|u\|^{2}_{L^{2}(\mathbb{R})},\
\|\widetilde{v}\|^{2}_{L^{2}(\mathbb{R})}=\|v\|^{2}_{L^{2}(\mathbb{R})},$
$J_{\beta}(\widetilde{w},\widetilde{\sigma})\leq J_{\beta}(w,\sigma),\
J_{\beta}(\widetilde{u},\widetilde{v})\leq J_{\beta}(u,v).$

By the standard argument as [25], we can deduce that

$J_{\beta}(\widetilde{w},\widetilde{\sigma})=m_{\beta}(\|w\|_{L^{2}(\mathbb{R})},\|\sigma\|_{L^{2}(\mathbb{R})}),\
J_{\beta}(\widetilde{u},\widetilde{v})=m_{\beta}(\|u\|_{L^{2}(\mathbb{R})},\|v\|_{L^{2}(\mathbb{R})}).$

Therefore, $(\widetilde{w},\widetilde{\sigma}),\
(\widetilde{u},\widetilde{v})$ are solutions of system (1.1), by the standard
regularity results, we can get
$\widetilde{w},\widetilde{\sigma},\widetilde{u},\widetilde{v}\in
C^{2}(\mathbb{R}^{2})$. Without restriction, we may assume $w\neq 0$. We
divide into two cases.

Case 1: $w\neq 0$ and $u\neq 0.$

From (ii), (iv), (v) of Lemma 2.1, we have

$\int_{\mathbb{R}}|\nabla\\{\widetilde{w},\widetilde{u}\\}^{\star}|^{2}dx<\int_{\mathbb{R}}|\nabla\widetilde{w}|^{2}dx+\int_{\mathbb{R}}|\nabla\widetilde{u}|^{2}dx\leq\int_{\mathbb{R}}|\nabla
w|^{2}dx+\int_{\mathbb{R}}|\nabla u|^{2}dx,$
$\displaystyle\int_{\mathbb{R}}|\\{\widetilde{w},\widetilde{u}\\}^{\star}|^{2}|\\{\widetilde{\sigma},\widetilde{v}\\}^{\star}|^{2}dx$
$\displaystyle=\int_{\mathbb{R}}\\{|\widetilde{w}|^{2},|\widetilde{u}|^{2}\\}^{\star}\\{|\widetilde{\sigma}|^{2},|\widetilde{v}|^{2}\\}^{\star}dx$
$\displaystyle\geq\int_{\mathbb{R}}|\widetilde{w}|^{2}|\widetilde{\sigma}|^{2}+|\widetilde{u}|^{2}|\widetilde{v}|^{2}dx$
$\displaystyle=\int_{\mathbb{R}}\widetilde{(|w|^{2})}\widetilde{(|\sigma|^{2})}+\widetilde{(|u|^{2})}\widetilde{(|v|^{2})}dx$
$\displaystyle\geq\int_{\mathbb{R}}|w|^{2}|\sigma|^{2}+|u|^{2}|v|^{2}dx,$
$\displaystyle\int_{\mathbb{R}}|\\{\widetilde{w},\widetilde{u}\\}^{\star}|^{2}\\{\widetilde{\sigma},\widetilde{v}\\}^{\star}dx$
$\displaystyle=\int_{\mathbb{R}}\\{|\widetilde{w}|^{2},|\widetilde{u}|^{2}\\}^{\star}\\{\widetilde{\sigma},\widetilde{v}\\}^{\star}dx$
$\displaystyle\geq\int_{\mathbb{R}}|\widetilde{w}|^{2}\widetilde{\sigma}+|\widetilde{u}|^{2}\widetilde{v}dx$
$\displaystyle=\int_{\mathbb{R}}\widetilde{(|w|^{2})}\widetilde{(|\sigma|)}+\widetilde{(|u|^{2})}\widetilde{(|v|)}dx$
$\displaystyle\geq\int_{\mathbb{R}}|w|^{2}|\sigma|+|u|^{2}|v|dx,$

so

$\displaystyle
J_{\beta}(w,\sigma)+J_{\beta}(u,v)>J_{\beta}(\\{\widetilde{w},\widetilde{u}\\}^{\star},\\{\widetilde{\sigma},\widetilde{v}\\}^{\star}).$
(2.10)

By (iii) of Lemma 2.1, we get

$\displaystyle\int_{\mathbb{R}}|\\{\widetilde{w},\widetilde{u}\\}^{\star}|^{2}dx=\int_{\mathbb{R}}(|\widetilde{w}|^{2}+|\widetilde{u}|^{2})dx=\int_{\mathbb{R}}(|w|^{2}+|u|^{2})dx,$
(2.11)
$\displaystyle\int_{\mathbb{R}}|\\{\widetilde{\sigma},\widetilde{v}\\}^{\star}|^{2}dx=\int_{\mathbb{R}}(|\widetilde{\sigma}|^{2}+|\widetilde{v}|^{2})dx=\int_{\mathbb{R}}(|\sigma|^{2}+|v|^{2})dx.$
(2.12)

By (2.7)-(2.12), (iii) of Lemma 2.1 and (ii) of Lemma 2.4, we obtain

$\displaystyle
m_{\beta}(b_{1},b_{2})>m_{\beta}(d_{1},d_{2})+m_{\beta}(\sqrt{b^{2}_{1}-d^{2}_{1}},\sqrt{b^{2}_{2}-d^{2}_{2}})\geq
m_{\beta}(b_{1},b_{2}),$

a contradiction.

Case 2: $w\neq 0,\ u=0$ and $v\neq 0.$

If $\sigma\neq 0$, we only need to replaced $w,u$ by $\sigma,v$ in Case 1, by
the same argument as Case 1, we can get a contradiction. Thus, we suppose that
$\sigma=0$. By (ii)-(v) of Lemma 2.1, we have

$\displaystyle
J_{\beta}(\\{\widetilde{w},0\\}^{\star},\\{\widetilde{v},0\\}^{\star})$
$\displaystyle\leq\frac{1}{2}\int_{\mathbb{R}}(|\nabla\widetilde{w}|^{2}+|\nabla\widetilde{v}|^{2})dx-\frac{\mu_{2}}{4}\int_{\mathbb{R}^{N}}|\widetilde{w}|^{4}dx-\frac{\mu_{1}}{4}\int_{\mathbb{R}}|\widetilde{v}|^{4}dx$
(2.13)
$\displaystyle\quad-\frac{\rho}{2}\int_{\mathbb{R}}|\widetilde{w}|^{2}|\widetilde{v}|^{2}dx-\frac{\beta}{2}\int_{\mathbb{R}}|\widetilde{w}|^{2}\widetilde{v}dx$
$\displaystyle\leq\frac{1}{2}\int_{\mathbb{R}}(|\nabla\widetilde{w}|^{2}+|\nabla\widetilde{v}|^{2})dx-\frac{\mu_{2}}{4}\int_{\mathbb{R}}|\widetilde{w}|^{4}dx-\frac{\mu_{1}}{4}\int_{\mathbb{R}}|\widetilde{v}|^{4}dx$
$\displaystyle=J_{\beta}(\widetilde{w},0)+J_{\beta}(0,\widetilde{v})\leq
J_{\beta}(w,0)+J_{\beta}(0,v).$

By (iii) of Lemma 2.1, we get

$\displaystyle\int_{\mathbb{R}}|\\{\widetilde{w},0\\}^{\star}|^{2}dx=\int_{\mathbb{R}}|\widetilde{w}|^{2}dx=\int_{\mathbb{R}}|w|^{2}dx,$
(2.14)
$\displaystyle\int_{\mathbb{R}}|\\{\widetilde{v},0\\}^{\star}|^{2}dx=\int_{\mathbb{R}}|\widetilde{v}|^{2}dx=\int_{\mathbb{R}}|v|^{2}dx.$
(2.15)

By (2.7)-(2.8), (2.9),(2.13),(2.14)(2.15), (iii) of Lemma 2.1 and (ii) of
Lemma 2.4, we obtain

$\displaystyle
m_{\beta}(b_{1},b_{2})>m_{\beta}(d_{1},d_{2})+m_{\beta}(\sqrt{b^{2}_{1}-d^{2}_{1}},\sqrt{b^{2}_{2}-d^{2}_{2}})\geq
m_{\beta}(b_{1},b_{2}),$

a contradiction. The contradictions obtained in Case 1 and Case 2 indicate
that $w_{n}(x)=u_{n}(x)-u(x+y_{n})\rightarrow 0,\
\sigma_{n}(x)=v_{n}(x)-v(x+y_{n})\rightarrow 0$ in $L^{p}(\mathbb{R})$ for
$2<p<2^{\ast}$. ∎

###### Proof of Theorem 1.1.

Let $\\{w_{n},\sigma_{n}\\}$ be any minimizing sequence for the functional $J$
on $\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$. By Lemma 2.5, we know that
there exists $\\{y_{n}\\}\subset\mathbb{R}$ such that
$(w_{n},\sigma_{n})\rightharpoonup(w,\sigma)$ in $H^{1}(\mathbb{R})\times
H^{1}(\mathbb{R})$ and $(w_{n},\sigma_{n})\rightarrow(w,\sigma)$ in
$L^{p}(\mathbb{R})\times L^{p}(\mathbb{R})$ for $2<p<2^{\ast}$. Hence, by the
weakly lower semi-continuity of the norm, we have

$\displaystyle J_{\beta}(w,\sigma)\leq m_{\beta}(b_{1},b_{2})<0.$ (2.16)

To show the compactness of $\\{w_{n},\sigma_{n}\\}$ in
$H^{1}(\mathbb{R})\times H^{1}(\mathbb{R})$, it suffices to prove that
$(w,\sigma)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$. If
$\int_{\mathbb{R}}|w|^{2}dx=b^{2}_{1}$ and $\
\int_{\mathbb{R}^{N}}|\sigma|^{2}dx=b^{2}_{2}$, we are done. Assume by
contradiction that there exists $\overline{b}_{1}<b_{1}$ and
$\overline{b}_{2}<b_{2}$ such that
$\int_{\mathbb{R}}|w|^{2}dx=\overline{b}^{2}_{1}<b^{2}_{1},$ or $\
\int_{\mathbb{R}}|\sigma|^{2}dx=\overline{b}^{2}_{2}<b^{2}_{2}$. Then, by the
definition of $m_{\beta}(b_{1},b_{2})$, we have

$m_{\beta}(\overline{b}_{1},\overline{b}_{2})\leq J_{\beta}(w,\sigma).$

Case 1 If $\int_{\mathbb{R}}|w|^{2}dx=\overline{b}^{2}_{1}<b^{2}_{1}$ and $\
\int_{\mathbb{R}}|\sigma|^{2}dx=\overline{b}^{2}_{2}<b^{2}_{2}$. By Lemma 2.2
and (ii) of Lemma 2.4, we have

$J_{\beta}(w,\sigma)\leq m_{\beta}(b_{1},b_{2})\leq
m_{\beta}(\overline{b}_{1},\overline{b}_{2})+m_{\beta}\big{(}\sqrt{b^{2}_{1}-\overline{b}^{2}_{1}},\sqrt{b^{2}_{2}-\overline{b}^{2}_{2}}\big{)}<m_{\beta}(\overline{b}_{1},\overline{b}_{2})\leq
J_{\beta}(w,\sigma),$

a contradiction.

Case 2 If $\int_{\mathbb{R}}|w|^{2}dx=b^{2}_{1}$ and $\
\int_{\mathbb{R}}|\sigma|^{2}dx=\overline{b}^{2}_{2}<b^{2}_{2}$. By Lemma 2.2
and (ii) of Lemma 2.4 and $m(0,\sqrt{b^{2}_{2}-\overline{b}^{2}_{2}})<0,$ we
have

$J_{\beta}(w,\sigma)\leq m_{\beta}(b_{1},b_{2})\leq
m_{\beta}(b_{1},\overline{b}_{2})+m_{\beta}\big{(}0,\sqrt{b^{2}_{2}-\overline{b}^{2}_{2}}\big{)}<m_{\beta}(b_{1},\overline{b}_{2})\leq
J_{\beta}(w,\sigma),$

a contradiction.

Case 3 If $\int_{\mathbb{R}}|w|^{2}dx=\overline{b}^{2}_{1}<b^{2}_{1}$ and $\
\int_{\mathbb{R}}|\sigma|^{2}dx=b^{2}_{2}$. By Lemma 2.2 and (ii) of Lemma 2.4
and $m(\sqrt{b^{2}_{1}-\overline{b}^{2}_{1}},0)<0,$ we have

$J_{\beta}(w,\sigma)\leq m_{\beta}(b_{1},b_{2})\leq
m_{\beta}(\overline{b}_{1},b_{2})+m_{\beta}\big{(}\sqrt{b^{2}_{1}-\overline{b}^{2}_{1}},0\big{)}<m_{\beta}(\overline{b}_{1},b_{2})\leq
J_{\beta}(w,\sigma),$

a contradiction.

Therefore,

$(w,\sigma)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}.$

∎

##  3 Proof of Theorem 1.2

To show our results, we first define following constant $A$,

$\displaystyle
A=\inf_{(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}}\frac{\int_{\mathbb{R}^{2}}(|\nabla
u|^{2}+|\nabla
v|^{2})dx}{\int_{\mathbb{R}^{2}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx}.$ (3.1)

From Gagliardo-Nirenberg-Sobolev inequality in [39],

$\|u\|^{4}_{L^{4}(\mathbb{R}^{2})}\leq\frac{2}{\|Q(x)\|^{2}_{L^{2}}}\|\nabla
u\|_{L^{2}(\mathbb{R}^{2})}^{2}\|u\|_{L^{2}(\mathbb{R}^{2})}^{2}\ \ \text{for
all}\ \ u\in H^{1}(\mathbb{R}^{2}),$

where $Q(x)$ is the unique positive solution of

$-\Delta u+u=u^{3},\ u\in H^{1}(\mathbb{R}^{2}),$

and the identity is achieved at $u(x)=Q(|x|)$. It is easy to see that $Q(|x|)$
satisfies that

$\|\nabla
Q\|_{L^{2}(\mathbb{R}^{2})}^{2}=\|Q\|_{L^{2}(\mathbb{R}^{2})}^{2}=\frac{1}{2}\|Q\|^{4}_{L^{4}(\mathbb{R}^{2})}.$

From above Gagliardo-Nirenberg-Sobolev inequality, we have

$\int_{\mathbb{R}^{2}}(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2})dx\leq\frac{2}{\|Q(x)\|^{2}_{L^{2}}}\left[\mu_{1}b^{2}_{1}+\mu_{2}b^{2}_{2}+2\rho
b_{1}b_{2}\right]\int_{\mathbb{R}^{2}}(|\nabla u|^{2}+|\nabla v|^{2})dx,$

so

$\displaystyle A\geq\frac{1}{C}>0.$ (3.2)

Next, we prove that

$\displaystyle\frac{\|Q\|_{L^{2}(\mathbb{R}^{2})}^{2}}{2\max\\{(\mu_{1}+\rho)b^{2}_{1},(\mu_{2}+\rho)b^{2}_{2}\\}}\leq
A\leq\frac{\frac{1}{2}(b^{2}_{1}+b^{2}_{2})\|Q\|_{L^{2}(\mathbb{R}^{2})}^{2}}{\mu_{1}b^{4}_{1}+\mu_{2}b^{4}_{2}+2\rho
b^{2}_{1}b^{2}_{2}}.$ (3.3)

It is easy to see that
$(\frac{b_{1}Q(x)}{\|Q(x)\|_{L^{2}}},\frac{b_{2}Q(x)}{\|Q(x)\|_{L^{2}}})\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}.$
Thus,

$\displaystyle
A\leq\frac{b^{2}_{1}+b^{2}_{2}}{\mu_{1}b^{4}_{1}+\mu_{2}b^{4}_{2}+2\rho
b^{2}_{1}b^{2}_{2}}\frac{\|Q(x)\|^{2}_{L^{2}}\int_{\mathbb{R}^{2}}|\nabla
Q|^{2}dx}{\int_{\mathbb{R}^{2}}Q^{4}dx}=\frac{\frac{1}{2}(b^{2}_{1}+b^{2}_{2})\|Q\|_{L^{2}(\mathbb{R}^{2})}^{2}}{\mu_{1}b^{4}_{1}+\mu_{2}b^{4}_{2}+2\rho
b^{2}_{1}b^{2}_{2}}.$

On the other hand, for any $(u,v)\in H^{1}(\mathbb{R}^{2})\times
H^{1}(\mathbb{R}^{2})$ and $\int_{\mathbb{R}^{2}}u^{2}dx=b^{2}_{1},\ \
\int_{\mathbb{R}^{2}}v^{2}dx=b^{2}_{2}.$ Then from Gagliardo-Nirenberg-Sobolev
inequality, we have

$\displaystyle\frac{\|Q\|_{L^{2}(\mathbb{R}^{2})}^{2}}{2\max\\{(\mu_{1}+\rho)b^{2}_{1},(\mu_{2}+\rho)b^{2}_{2}\\}}$
$\displaystyle\leq\frac{\int_{\mathbb{R}^{2}}(|\nabla u|^{2}+|\nabla
v|^{2})dx}{\int_{\mathbb{R}^{2}}(\mu_{1}+\rho)u^{4}dx+\int_{\mathbb{R}^{2}}(\mu_{2}+\rho)v^{4}dx}$
$\displaystyle\leq\frac{\int_{\mathbb{R}^{2}}(|\nabla u|^{2}+|\nabla
v|^{2})dx}{\int_{\mathbb{R}^{2}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx}.$

From (2.1), (2.2) and the definition of $A$, we have

$\displaystyle J_{0}(u,v)$
$\displaystyle=\frac{1}{2}\int_{\mathbb{R}^{2}}(|\nabla u|^{2}+|\nabla
v|^{2})dx-\frac{1}{4}\int_{\mathbb{R}^{2}}(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2})dx$ (3.4)
$\displaystyle\geq\left[\frac{1}{2}-\frac{1}{4A}\right]\int_{\mathbb{R}^{2}}(|\nabla
u|^{2}+|\nabla v|^{2})dx,$

so $J_{0}(u,v)$ is bounded from below on
$\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$ provided that

$\displaystyle A>\frac{1}{2}.$ (3.5)

Let $u\in\mathrm{T}_{b}$ and $t\star u=e^{\frac{Nt}{2}}u(e^{t}x)$, then
$t\star u\in\mathrm{T}_{b}$, we define

$t\star(u,v)=(t\star u,t\star v).$ (3.6)

Under the condition of
$\max\\{(\mu_{1}+\rho)b^{2}_{1},(\mu_{2}+\rho)b^{2}_{2}\\}<\|Q\|_{L^{2}(\mathbb{R}^{2})}^{2}$,
we have

$A>\frac{1}{2},$

so

$\frac{1}{2}\int_{\mathbb{R}^{2}}(|\nabla u|^{2}+|\nabla
v|^{2})dx-\frac{1}{4}\int_{\mathbb{R}^{2}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx>0.$

Therefore

$\displaystyle\frac{1}{2}\beta\int_{\mathbb{R}^{2}}u^{2}vdx=\frac{1}{2}\int_{\mathbb{R}^{2}}(|\nabla
u|^{2}+|\nabla
v|^{2})dx-\frac{1}{4}\int_{\mathbb{R}^{2}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx>0.$

Since

$\displaystyle\Psi_{u,v}(t)=J_{\beta}(t\star(u,v))$
$\displaystyle=\frac{e^{2t}}{2}\bigg{[}\int_{\mathbb{R}^{2}}(|\nabla
u|^{2}+|\nabla
v|^{2})dx-\frac{1}{2}\int_{\mathbb{R}^{2}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx\bigg{]}$
$\displaystyle-\frac{\beta}{2}e^{t}\int_{\mathbb{R}^{2}}u^{2}vdx.$

$J_{\beta}(u,v)=\frac{1}{2}\int_{\mathbb{R}^{N}}(|\nabla u|^{2}+|\nabla
v|^{2})dx-\frac{1}{4}\int_{\mathbb{R}^{N}}(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2})dx-\frac{\beta}{2}\int_{\mathbb{R}^{N}}u^{2}vdx.$

######  Lemma 3.1.

Under the condition of
$\max\\{(\mu_{1}+\rho)b^{2}_{1},(\mu_{2}+\rho)b^{2}_{2}\\}<\|Q\|_{L^{2}(\mathbb{R}^{2})}^{2}$,
we have

$-\infty<m_{\beta}(b_{1},b_{2})=\inf_{(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}}J_{\beta}(u,v)<0.$

###### Proof.

If
$\frac{\|Q\|_{L^{2}(\mathbb{R}^{2})}^{2}}{\max\\{(\mu_{1}+\rho)b^{2}_{1},(\mu_{2}+\rho)b^{2}_{2}\\}}>1$
and $\beta>0$, we have $A>\frac{1}{2}$, thus

$\displaystyle J_{\beta}(u,v)$
$\displaystyle=\frac{1}{2}\int_{\mathbb{R}^{2}}(|\nabla u|^{2}+|\nabla
v|^{2})dx-\frac{1}{4}\int_{\mathbb{R}^{2}}(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2})dx-\frac{\beta}{2}\int_{\mathbb{R}^{2}}u^{2}vdx$ (3.7)
$\displaystyle\geq\left[\frac{1}{2}-\frac{1}{4A}\right]\int_{\mathbb{R}^{2}}(|\nabla
u|^{2}+|\nabla v|^{2})dx$
$\displaystyle\quad-\frac{\beta}{2}\left[\left(\frac{2}{3}b^{2}_{1}+\frac{1}{3}b^{2}_{2}\right)C^{3}_{2,3}\left[\|\nabla
u\|^{2}_{L^{2}(\mathbb{R}^{2})}+\|\nabla
v\|^{2}_{L^{2}(\mathbb{R}^{2})}\right]^{\frac{1}{2}}\right],$

so $J_{\beta}(u,v)$ is coercive on
$\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$ and

$m_{\beta}(b_{1},b_{2})=\inf_{(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}}J_{\beta}(u,v)>-\infty.$

Next, we prove

$m_{\beta}(b_{1},b_{2})=\inf_{(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}}J_{\beta}(u,v)<0.$

It is easy to see that

$\displaystyle J_{\beta}(t\star(u,v))$
$\displaystyle=\frac{e^{2t}}{2}\int_{\mathbb{R}^{2}}(|\nabla u|^{2}+|\nabla
v|^{2})dx-\frac{\beta}{2}e^{t}\int_{\mathbb{R}^{2}}u^{2}vdx$
$\displaystyle\quad-\frac{e^{2t}}{4}\int_{\mathbb{R}^{2}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx,$

when $\beta>0$ we choose $u>0,v>0$ and
$(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$, it is easy to see
$(t\star u,t\star v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$ and

$\lim_{t\rightarrow-\infty}J_{\beta}(t\star(u,v))<0,$

when $\beta<0$ we choose $u>0,v<0$ and
$(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$, it is easy to see
$(t\star u,t\star v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$ and

$\lim_{t\rightarrow-\infty}J_{\beta}(t\star(u,v))<0.$

Thus for any $\beta\in\mathbb{R}\setminus\\{0\\}$, we obtain

$m_{\beta}(b_{1},b_{2})=\inf_{(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}}J_{\beta}(u,v)<0.$

∎

###### Proof of Theorem 1.2.

When
$\max\\{(\mu_{1}+\rho)b^{2}_{1},(\mu_{2}+\rho)b^{2}_{2}\\}<\|Q\|_{L^{2}(\mathbb{R}^{2})}^{2}$,
by the definition of $A$, we have $A>\frac{1}{2}$ . Thus, if $\beta<0$ and
there exists a positive solution $(u,v)$ to (1.1)-(1.2), we have

$\displaystyle\frac{1}{2}\beta\int_{\mathbb{R}^{2}}u^{2}vdx$
$\displaystyle=\int_{\mathbb{R}^{2}}(|\nabla u|^{2}+|\nabla
v|^{2})dx-\frac{1}{2}\int_{\mathbb{R}^{2}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx\geq 0,$

a contradiction.

Next, we consider $\beta>0$. When
$\max\\{(\mu_{1}+\rho)b^{2}_{1},(\mu_{2}+\rho)b^{2}_{2}\\}<\|Q\|_{L^{2}(\mathbb{R}^{2})}^{2}$,
from Lemma 3.1, we get $J_{\beta}(u,v)$ is coercive on
$\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$ and

$m_{\beta}(b_{1},b_{2})=\inf_{(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}}J_{\beta}(u,v)>-\infty.$

Let $\\{(u_{n},v_{n})\\}\subset\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$ be
any minimizing sequence for $m_{\beta}(b_{1},b_{2})$. By Lemma 2.3, we know
that $\\{|u_{n}|,|v_{n}|\\}$ is also a minimizing sequence. Then by taking
$\\{(|u_{n}|,|v_{n}|)\\}$ and adapting the Schwarz symmetrization
to$\\{(|u_{n}|,|v_{n}|)\\}$ if necessary, we can obtain a new minimizing
sequence (up to a subsequence), such that$\\{(u_{n},v_{n})\\}$ are all real
valued, nonnegative, radially symmetric. Since $\\{(u_{n},v_{n})\\}$ is
bounded in $H_{r}^{1}(\mathbb{R}^{2})\times H_{r}^{1}(\mathbb{R}^{2})$. By the
Sobolev embedding theorem, we have
$H_{r}^{1}(\mathbb{R}^{2})\hookrightarrow\hookrightarrow
L^{p}_{r}(\mathbb{R}^{2})$ for $2<p<+\infty$, thus there exists a $(u,v)\in
H_{r}^{1}(\mathbb{R}^{2})\times H_{r}^{1}(\mathbb{R}^{2})$ such that
$(u_{n},v_{n})\rightharpoonup(u,v)$ in $H_{r}^{1}(\mathbb{R}^{2})\times
H_{r}^{1}(\mathbb{R}^{2})$, $(u_{n},v_{n})\rightarrow(u,v)$ in
$L^{p}_{r}(\mathbb{R}^{2})\times L^{p}_{r}(\mathbb{R}^{2})\ \ \text{for }\
2<p<+\infty$ and $(u_{n},v_{n})\rightarrow(u,v)$ a.e in $\mathbb{R}^{2}$.
Hence $u,v\geq 0$ are radial functions.

Step 1: Prove $\lambda_{1}>0$ and $\lambda_{2}>0$. By Ekeland’s variational
principle yields in a standard way the existence of a new minimizing sequence,
which is also a Palais-Smale sequence for $J_{\beta}$ on
$\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}.$ So
$J_{\beta}^{\prime}\mid_{\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}}(u_{n},v_{n})\rightarrow
0$, by the Lagrange multipliers rule, we know that there exists a sequence
$(\lambda_{1,n},\lambda_{2,n})\in\mathbb{R}^{2}$ such that

$\displaystyle\int_{\mathbb{R}^{2}}(\nabla u_{n}\nabla\varphi+\nabla
v_{n}\nabla\psi)dx+\int_{\mathbb{R}^{2}}(\lambda_{1,n}u_{n}\varphi+\lambda_{2,n}v_{n}\psi)dx$
(3.8)
$\displaystyle-\int_{\mathbb{R}^{2}}(\mu_{1}u_{n}^{3}\varphi+\mu_{2}v_{n}^{3}\psi+\rho
v_{n}^{2}u_{n}\varphi+\rho u_{n}^{2}v_{n}\psi)dx$
$\displaystyle-\beta\int_{\mathbb{R}^{2}}u_{n}\varphi
v_{n}-\frac{\beta}{2}\int_{\mathbb{R}^{2}}u_{n}^{2}\psi=o(1)\|(\psi,\varphi)\|_{H^{1}(\mathbb{R}^{2})\times
H^{1}(\mathbb{R}^{2})}\ \ \text{in}\ \ \mathbb{R}^{2},$

for every $(\varphi,\psi)\in H^{1}(\mathbb{R}^{2})\times
H^{1}(\mathbb{R}^{2})$. We claim both $\lambda_{1,n}$ and $\lambda_{2,n}$ are
bounded sequence, and at least one of them is converging, up to a subsequence,
to a strictly negative value. Indeed, we can using $(u_{n},0)$ and $(0,v_{n})$
as text function in (3.8), we have

$\int_{\mathbb{R}^{2}}\lambda_{1,n}u^{2}_{n}dx=-\int_{\mathbb{R}^{2}}|\nabla
u_{n}|^{2}dx+\int_{\mathbb{R}^{2}}(\mu_{1}u_{n}^{4}+\rho
v_{n}^{2}u^{2}_{n})dx+\beta\int_{\mathbb{R}^{2}}u^{2}_{n}v_{n}+o(1)\|\varphi\|_{H^{1}(\mathbb{R}^{2})},$
$\int_{\mathbb{R}^{2}}\lambda_{2,n}v^{2}_{n}dx=-\int_{\mathbb{R}^{2}}|\nabla
v_{n}|^{2}dx+\int_{\mathbb{R}^{2}}(\mu_{2}v_{n}^{4}+\rho
v_{n}^{2}u^{2}_{n})dx+\frac{\beta}{2}\int_{\mathbb{R}^{2}}u^{2}_{n}v_{n}+o(1)\|\psi\|_{H^{1}(\mathbb{R}^{2})},$

so

$\displaystyle\int_{\mathbb{R}^{2}}(\lambda_{1,n}u^{2}_{n}+\lambda_{2,n}v^{2}_{n})dx$
$\displaystyle=-\int_{\mathbb{R}^{2}}(|\nabla u_{n}|^{2}+|\nabla
v_{n}|^{2})dx+\int_{\mathbb{R}^{2}}(\mu_{1}u_{n}^{4}+\mu_{2}v_{n}^{4}+2\rho
v_{n}^{2}u^{2}_{n})dx$ (3.9)
$\displaystyle\quad+\frac{3}{2}\beta\int_{\mathbb{R}^{2}}u_{n}^{2}v_{n}dx.$

By (2.1)-(2.3) and the boundedness of $\\{(u_{n},v_{n})\\}$, we can deduce
that $(\lambda_{1,n},\lambda_{2,n})$ is bounded, hence up to a subsequence
$(\lambda_{1,n},\lambda_{2,n})\rightarrow(\lambda_{1},\lambda_{2})\in\mathbb{R}^{2}$,
passing to limits in (3.8), we can deduce that $(u,v)$ is a nonnegative
solutions of (1.1) . Therefore

$\displaystyle\int_{\mathbb{R}^{2}}(|\nabla u|^{2}+|\nabla
v|^{2})dx+\int_{\mathbb{R}^{2}}(\lambda_{1}u^{2}+\lambda_{2}v^{2})dx=\int_{\mathbb{R}^{2}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx+\frac{3}{2}\beta\int_{\mathbb{R}^{2}}u^{2}vdx.$

On the one hand, we multiply the first equation of (1.1) by $x\cdot\nabla u$
and the second equation of (1.1) by $x\cdot\nabla v$, integrate by parts, we
can get

$\displaystyle\int_{\mathbb{R}^{2}}(\lambda_{1}u^{2}+\lambda_{2}v^{2})dx=\frac{1}{2}\int_{\mathbb{R}^{2}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx+\beta\int_{\mathbb{R}^{2}}u^{2}vdx.$ (3.10)

Since $P_{b_{1},b_{2}}(u_{n},v_{n})\rightarrow 0$, which implies that

$\displaystyle\int_{\mathbb{R}^{2}}(|\nabla u_{n}|^{2}+|\nabla
v_{n}|^{2})dx-\frac{1}{2}\int_{\mathbb{R}^{2}}\left(\mu_{1}u_{n}^{4}+\mu_{2}v_{n}^{4}+2\rho
u_{n}^{2}v_{n}^{2}\right)dx-\frac{1}{2}\beta\int_{\mathbb{R}^{2}}u_{n}^{2}v_{n}dx=o_{n}(1).$
(3.11)

Together (3.9) with (3.11), we can get

$\displaystyle\lambda_{1,n}b^{2}_{1}+\lambda_{2,n}b^{2}_{2}=\frac{1}{2}\int_{\mathbb{R}^{2}}\left(\mu_{1}u_{n}^{4}+\mu_{2}v_{n}^{4}+2\rho
u_{n}^{2}v_{n}^{2}\right)dx+\beta\int_{\mathbb{R}^{2}}u_{n}^{2}v_{n}dx,$
(3.12)

when $\beta>0$, it is easy to see that at least one sequence of
$(\lambda_{i,n})$ is positive and bounded away from 0. Let
$n\rightarrow+\infty$ in (3.12), we have

$\displaystyle\lambda_{1}b^{2}_{1}+\lambda_{2}b^{2}_{2}=\frac{1}{2}\int_{\mathbb{R}^{2}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx+\beta\int_{\mathbb{R}^{2}}u^{2}vdx.$ (3.13)

We claim that if $\lambda_{1}>0$(resp.$\lambda_{2}>0)$, then
$\lambda_{2}>0$(resp.$\lambda_{1}>0$). Indeed, we know that at least one
sequence of $(\lambda_{i})$ is positive and bounded away from 0. If
$\lambda_{2}>0$, now we argue by contradiction and assume that
$\lambda_{1}\leq 0$, then

$-\Delta u=-\lambda_{1}u+\beta uv+\mu_{1}u^{3}+\rho v^{2}u\geq 0.$

Using a Liouville type theorem[[22], Lemma A.2], we can deduce that $u=0$. So,
$v$ satisfies that

$\begin{cases}-\Delta v+\lambda_{2}v=\mu_{2}v^{3},&\text{in}\ \
\mathbb{R}^{2},\\\ v>0,&\text{in}\ \ \mathbb{R}^{2},\\\
\int_{\mathbb{R}^{3}}v^{2}dx=b^{2}_{2},&\text{in}\ \ \mathbb{R}^{2}.\\\
\end{cases}$

Therefore, when $0<b_{2}\leq\|Q\|_{L^{2}(\mathbb{R}^{2})}$ we have

$\displaystyle m_{\beta}(b_{1},b_{2})$
$\displaystyle=\lim_{n\rightarrow+\infty}J_{\beta}(u_{n},v_{n})=\lim_{n\rightarrow+\infty}J_{\beta}(u_{n},v_{n}-v)+J(0,v)$
$\displaystyle=\lim_{n\rightarrow+\infty}\bigg{[}\frac{1}{2}\int_{\mathbb{R}^{2}}(|\nabla
u_{n}|^{2}+|\nabla(v_{n}-v)|^{2})dx-\frac{1}{4}\int_{\mathbb{R}^{2}}\Big{(}\mu_{1}u_{n}^{4}+\mu_{2}(v_{n}-v)^{4}+2\rho
u_{n}^{2}(v_{n}-v)^{2}\Big{)}dx$
$\displaystyle\quad-\frac{1}{2}\beta\int_{\mathbb{R}^{2}}u_{n}^{2}(v_{n}-v)dx\bigg{]}+J(0,v)\geq\lim_{n\rightarrow+\infty}\frac{1}{2}\int_{\mathbb{R}^{2}}(|\nabla(u_{n}-u)|^{2}+|\nabla
v_{n}|^{2})dx+m_{\beta}(0,b_{2})\geq 0,$

in contradiction with $m_{\beta}(b_{1},b_{2})<0$, where we use the fact that
$m_{\beta}(0,b_{2})=0$ for $0<b_{2}\leq\|Q\|_{L^{2}(\mathbb{R}^{2})}$, (see
Theorem 1.2 in [30]). Thus, $\lambda_{1}>0,\ \lambda_{2}>0.$

If $\lambda_{1}>0$, now we argue by contradiction and assume that
$\lambda_{2}\leq 0$, then

$-\Delta v=-\lambda_{2}v\frac{\beta}{2}u^{2}+\mu_{2}v^{3}+\rho u^{2}v\geq 0.$

Using a Liouville type theorem[[22], Lemma A.2], we can deduce that $v=0$. So,
by the structure of system (1.1), we get $u=0$, which is impossible.

Step 2: Prove the $L^{2}$ convergence. Indeed, it is easy to see that
$\int_{\mathbb{R}^{2}}u^{2}\leq b^{2}_{1},\ \int_{\mathbb{R}^{2}}v^{2}\leq
b^{2}_{2}.$ From (3.10) and (3.12), we have

$\lambda_{1}\left(b^{2}_{1}-\int_{\mathbb{R}^{2}}u^{2}dx\right)+\lambda_{2}\left(b^{2}_{2}-\int_{\mathbb{R}^{2}}v^{2}dx\right)=0,$

so

$\displaystyle\int_{\mathbb{R}^{2}}u^{2}=b^{2}_{1},\
\int_{\mathbb{R}^{2}}v^{2}=b^{2}_{2}.$

Step 3: Prove the $H^{1}$ convergence. It is easy to see that

$\displaystyle m_{\beta}(b_{1},b_{2})$
$\displaystyle=\lim_{n\rightarrow+\infty}J_{\beta}(u_{n},v_{n})$
$\displaystyle=\lim_{n\rightarrow+\infty}\bigg{[}\frac{1}{2}\int_{\mathbb{R}^{2}}(|\nabla
u_{n}|^{2}+|\nabla
v_{n}|^{2})dx-\frac{1}{4}\int_{\mathbb{R}^{2}}\left(\mu_{1}u_{n}^{4}+\mu_{2}v_{n}^{4}+2\rho
u_{n}^{2}v_{n}^{2}\right)dx-\frac{1}{2}\beta\int_{\mathbb{R}^{2}}u_{n}^{2}v_{n}dx\bigg{]}.$

Case 1 If $u=0,v=0$, by compact Sobolev embedding, we have

$\displaystyle
m_{\beta}(b_{1},b_{2})=\lim_{n\rightarrow+\infty}J_{\beta}(u_{n},v_{n})\geq\lim_{n\rightarrow+\infty}\frac{1}{2}\int_{\mathbb{R}^{2}}(|\nabla
u_{n}|^{2}+|\nabla v_{n}|^{2})dx\geq 0.$

contradict with Lemma 3.1.

Case 2 If $u\neq 0,v=0$, indeed, if $v=0$, by the structure of system (1.1),
we get $u=0$, so Case 2 doesn’t happen.

Case 3 If $u=0,\ v\neq 0$, let

$\overline{u}_{n}=u_{n},\ \overline{v}_{n}=v_{n}-v,$

from [17, Lemma 2.4], we get

$\int_{\mathbb{R}^{2}}|\overline{u}_{n}|^{2}|\overline{v}_{n}|^{2}dx=\int_{\mathbb{R}^{2}}|u_{n}|^{2}|v_{n}|^{2}dx-\int_{\mathbb{R}^{2}}|u|^{2}|v|^{2}dx+o(1).$
(3.14)

By Brezis Lieb Lemma in [47], we can obtain that

$\int_{\mathbb{R}^{2}}|\overline{u}_{n}|^{2}\overline{v}_{n}dx=\int_{\mathbb{R}^{2}}|u_{n}|^{2}v_{n}dx-\int_{\mathbb{R}^{2}}|u|^{2}vdx+o(1).$
(3.15)

So, from (3.14) and (3.15), we have

$\displaystyle
J_{\beta}(u_{n},v_{n})=J_{\beta}(\overline{u}_{n},\overline{v}_{n})+J_{\beta}(u,v)+o(1).$

Therefore

$\displaystyle m_{\beta}(b_{1},b_{2})$
$\displaystyle=\lim_{n\rightarrow+\infty}J_{\beta}(u_{n},v_{n})=\lim_{n\rightarrow+\infty}J_{\beta}(\overline{u}_{n},\overline{v}_{n})+J_{\beta}(0,v)$
$\displaystyle=\lim_{n\rightarrow+\infty}\bigg{[}\frac{1}{2}\int_{\mathbb{R}^{2}}(|\nabla\overline{u}_{n}|^{2}+|\nabla\overline{v}_{n}|^{2})dx-\frac{1}{4}\int_{\mathbb{R}^{2}}\left(\mu_{1}\overline{u}_{n}^{4}+\mu_{2}\overline{v}_{n}^{4}+2\rho\overline{u}_{n}^{2}\overline{v}_{n}^{2}\right)dx$
$\displaystyle\quad-\frac{1}{2}\beta\int_{\mathbb{R}^{2}}\overline{u}_{n}^{2}\overline{v}_{n}dx\bigg{]}+J_{\beta}(0,v)$
$\displaystyle\geq\lim_{n\rightarrow+\infty}\frac{1}{2}\int_{\mathbb{R}^{2}}(|\nabla\overline{u}_{n}|^{2}+|\nabla\overline{v}_{n}|^{2})dx+J_{\beta}(0,v)\geq
m_{\beta}(0,b_{2})=0.$

contradict with Lemma 3.1, where we use the fact that $m_{\beta}(0,b_{2})=0$
for $0<b_{2}\leq\|Q\|_{L^{2}(\mathbb{R}^{2})}$.

Case 4 If $u\neq 0,v\neq 0$, let

$\widetilde{u}_{n}=u_{n}-u,\ \widetilde{v}_{n}=v_{n}-v,$

from [21, Lemma 2.1], we get

$\int_{\mathbb{R}^{2}}|\widetilde{u}_{n}|^{2}|\widetilde{v}_{n}|^{2}dx=\int_{\mathbb{R}^{2}}|u_{n}|^{2}|v_{n}|^{2}dx-\int_{\mathbb{R}^{2}}|u|^{2}|v|^{2}dx+o(1).$
(3.16)

By Brezis Lieb Lemma in [47], we can obtain

$\int_{\mathbb{R}^{2}}|\widetilde{u}_{n}|^{2}\widetilde{v}_{n}dx=\int_{\mathbb{R}^{2}}|u_{n}|^{2}v_{n}dx-\int_{\mathbb{R}^{2}}|u|^{2}vdx+o(1).$
(3.17)

So, from (3.16) and (3.17), we have

$\displaystyle
J_{\beta}(u_{n},v_{n})=J_{\beta}(\widetilde{u}_{n},\widetilde{v}_{n})+J_{\beta}(u,v)+o(1).$

Therefore

$\displaystyle m_{\beta}(b_{1},b_{2})$
$\displaystyle=\lim_{n\rightarrow+\infty}J_{\beta}(u_{n},v_{n})=\lim_{n\rightarrow+\infty}J_{\beta}(\widetilde{u}_{n},\widetilde{v}_{n})+J_{\beta}(u,v)$
$\displaystyle=\lim_{n\rightarrow+\infty}\bigg{[}\frac{1}{2}\int_{\mathbb{R}^{2}}(|\nabla\widetilde{u}_{n}|^{2}+|\nabla\widetilde{v}_{n}|^{2})dx$
$\displaystyle\quad-\frac{1}{4}\int_{\mathbb{R}^{2}}\left(\mu_{1}\widetilde{u}_{n}^{4}+\mu_{2}\widetilde{v}_{n}^{4}+2\rho\widetilde{u}_{n}^{2}\widetilde{v}_{n}^{2}\right)dx-\frac{1}{2}\beta\int_{\mathbb{R}^{2}}\widetilde{u}_{n}^{2}\widetilde{v}_{n}dx\bigg{]}+J_{\beta}(u,v)$
$\displaystyle\geq\lim_{n\rightarrow+\infty}\frac{1}{2}\int_{\mathbb{R}^{2}}(|\nabla\widetilde{u}_{n}|^{2}+|\nabla\widetilde{v}_{n}|^{2})dx+J_{\beta}(u,v)\geq
m_{\beta}(b_{1},b_{2}).$

Thus, $J_{\beta}(u,v)=m_{\beta}(b_{1},b_{2})\ \text{and}\
(u_{n},v_{n})\rightarrow(u,v)$ in $H^{1}(\mathbb{R}^{2})\times
H^{1}(\mathbb{R}^{2}).$

Next, we prove the second part of Theorem 1.2. It is easy to see that

$\displaystyle J_{0}(t\star(u,v))$
$\displaystyle=\frac{e^{2t}}{2}\int_{\mathbb{R}^{2}}(|\nabla u|^{2}+|\nabla
v|^{2})dx-\frac{e^{2t}}{4}\int_{\mathbb{R}^{2}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx.$

If

$\displaystyle\frac{1}{2}\int_{\mathbb{R}^{2}}(|\nabla u|^{2}+|\nabla
v|^{2})dx<\frac{1}{4}\int_{\mathbb{R}^{2}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx,$ (3.18)

then

$J_{0}(t\star(u,v))\rightarrow-\infty\ \text{as}\ t\rightarrow+\infty.$

It is easy to see

$\displaystyle J_{\beta}(t\star(u,v))$
$\displaystyle=\frac{e^{2t}}{2}\int_{\mathbb{R}^{2}}(|\nabla u|^{2}+|\nabla
v|^{2})dx-\frac{\beta}{2}e^{t}\int_{\mathbb{R}^{2}}u^{2}vdx-\frac{e^{2t}}{4}\int_{\mathbb{R}^{2}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx$ (3.19)
$\displaystyle=e^{2t}J_{0}(u,v)-\frac{\beta}{2}e^{t}\int_{\mathbb{R}^{2}}u^{2}vdx.$

If (3.18) holds, there exists
$(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$ such that $J_{0}(u,v)<0,$
by (3.19) we can deduce

$\inf_{(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}}J_{\beta}(u,v)=-\infty.$

∎

##  4 Proof of Theorem 1.3

Define the set

$\displaystyle\mathcal{P}_{b_{1},b_{2}}(u,v):=\Big{\\{}$
$\displaystyle(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}:P_{b_{1},b_{2}}(u,v)=0,\
\beta\int_{\mathbb{R}^{3}}u^{2}vdx>0\Big{\\}},$ (4.1)

where

$\displaystyle P_{b_{1},b_{2}}(u,v)$
$\displaystyle=\int_{\mathbb{R}^{3}}(|\nabla u|^{2}+|\nabla
v|^{2})dx-\frac{3}{4}\int_{\mathbb{R}^{3}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx-\frac{3}{4}\beta\int_{\mathbb{R}^{3}}u^{2}vdx,$

and

$\displaystyle\Psi_{u,v}(t)=J_{\beta}(t\star(u,v))=$
$\displaystyle\frac{e^{2t}}{2}\int_{\mathbb{R}^{3}}(|\nabla u|^{2}+|\nabla
v|^{2})dx-\frac{\beta}{2}\int_{\mathbb{R}^{3}}e^{\frac{3t}{2}}u^{2}vdx$ (4.2)
$\displaystyle-\frac{e^{3t}}{4}\int_{\mathbb{R}^{3}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx.$

It is easy to check

$\displaystyle\Psi^{\prime}_{u,v}(t)=e^{2t}\int_{\mathbb{R}^{3}}(|\nabla
u|^{2}+|\nabla
v|^{2})dx-\frac{3\beta}{4}\int_{\mathbb{R}^{3}}e^{\frac{3t}{2}}u^{2}vdx-\frac{3e^{3t}}{4}\int_{\mathbb{R}^{3}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx,$

and

$\Psi^{\prime}_{u,v}(t)=\mathcal{P}_{b_{1},b_{2}}(t\star u,t\star v),\ \
\mathcal{P}_{b_{1},b_{2}}(u,v)=\Big{\\{}(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}:\Psi^{\prime}_{u,v}(0)=0,\
\beta\int_{\mathbb{R}^{3}}u^{2}vdx>0\Big{\\}}.$

We decompose $\mathcal{P}_{b_{1},b_{2}}(u,v)$ into three disjoint unions

$\mathcal{P}_{b_{1},b_{2}}(u,v)=\mathcal{P}^{+}_{b_{1},b_{2}}(u,v)\cup\mathcal{P}^{0}_{b_{1},b_{2}}(u,v)\cup\mathcal{P}^{-}_{b_{1},b_{2}}(u,v),$

where

$\displaystyle\mathcal{P}^{+}_{b_{1},b_{2}}(u,v):=\Big{\\{}(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}},\
(u,v)\in\mathcal{P}_{b_{1},b_{2}},\
\beta\int_{\mathbb{R}^{3}}u^{2}vdx>0:\Psi^{\prime\prime}_{u,v}(0)>0\Big{\\}},$
(4.3)
$\displaystyle\mathcal{P}^{0}_{b_{1},b_{2}}(u,v):=\Big{\\{}(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}},\
(u,v)\in\mathcal{P}_{b_{1},b_{2}},\
\beta\int_{\mathbb{R}^{3}}u^{2}vdx>0:\Psi^{\prime\prime}_{u,v}(0)=0\Big{\\}},$
(4.4)
$\displaystyle\mathcal{P}^{-}_{b_{1},b_{2}}(u,v):=\Big{\\{}(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}},\
(u,v)\in\mathcal{P}_{b_{1},b_{2}},\
\beta\int_{\mathbb{R}^{3}}u^{2}vdx>0:\Psi^{\prime\prime}_{u,v}(0)<0\Big{\\}}.$
(4.5)

To prove Theorem 1.3, we first give some lemmas. The following auxiliary
result shows the role of $\mathcal{P}_{b_{1},b_{2}}(u,v)$.

######  Lemma 4.1.

If $(u,v)$ is a solution of problem (1.1)–(1.2) for some
$\lambda_{1},\lambda_{2}\in\mathbb{R}$, then
$(u,v)\in\mathcal{P}_{b_{1},b_{2}}(u,v)$.

###### Proof.

On the one hand, we multiply the first equation of (1.1) by $x\cdot\nabla u$
and the second equation of (1.1) by $x\cdot\nabla v$, integrate by parts, we
can get

$\displaystyle\frac{1}{2}\int_{\mathbb{R}^{3}}(|\nabla u|^{2}+|\nabla
v|^{2})dx+\frac{3}{2}\int_{\mathbb{R}^{3}}(\lambda_{1}u^{2}+\lambda_{2}v^{2})dx$
(4.6)
$\displaystyle=\frac{3}{4}\int_{\mathbb{R}^{3}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx+\frac{3}{2}\beta\int_{\mathbb{R}^{3}}u^{2}vdx.$

On the other hand, we multiply the first equation of (1.1) by $u$ and the
second equation of (1.1) by $v$, integrate by parts, we have

$\displaystyle\int_{\mathbb{R}^{3}}(|\nabla u|^{2}+|\nabla
v|^{2})dx+\int_{\mathbb{R}^{3}}(\lambda_{1}u^{2}+\lambda_{2}v^{2})dx=\int_{\mathbb{R}^{3}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx+\frac{3}{2}\beta\int_{\mathbb{R}^{3}}u^{2}vdx.$ (4.7)

Together (4.6) with (4.7), we obtain

$\displaystyle\int_{\mathbb{R}^{3}}(|\nabla u|^{2}+|\nabla
v|^{2})dx-\frac{3}{4}\int_{\mathbb{R}^{3}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx-\frac{3}{4}\beta\int_{\mathbb{R}^{3}}u^{2}vdx=0.$ (4.8)

∎

To show that the energy functional has a concave-convex structure, by
(2.1)-(2.3), we introduce

$\mathcal{D}_{1}=\mu_{1}C^{4}_{3,4}b_{1},\
\mathcal{D}_{2}=\mu_{2}C^{4}_{4,4}b_{2},\
\mathcal{D}_{3}=C^{4}_{3,4}b^{\frac{1}{2}}_{1}b^{\frac{1}{2}}_{2},\ \
\mathcal{D}_{4}=\big{(}\frac{2}{3}b^{\frac{3}{2}}_{1}+\frac{1}{3}b^{\frac{3}{2}}_{2}\big{)}C^{3}_{3,3}.$

If $\rho>0$

$\displaystyle J_{\beta}(u,v)$
$\displaystyle\geq\frac{1}{2}\int_{\mathbb{R}^{3}}(|\nabla u|^{2}+|\nabla
v|^{2})dx-\frac{1}{4}\mathcal{D}_{1}\big{(}\int_{\mathbb{R}^{3}}|\nabla
u|^{2}dx\big{)}^{\frac{3}{2}}-\frac{1}{4}\mathcal{D}_{2}\big{(}\int_{\mathbb{R}^{3}}|\nabla
v|^{2}dx\big{)}^{\frac{3}{2}}$ (4.9)
$\displaystyle\quad-\frac{1}{4}\rho\mathcal{D}_{3}\big{(}\int_{\mathbb{R}^{3}}(|\nabla
u|^{2}+|\nabla
v|^{2})dx\big{)}^{\frac{3}{2}}-\frac{1}{2}|\beta|\mathcal{D}_{4}\big{(}\int_{\mathbb{R}^{3}}(|\nabla
u|^{2}+|\nabla v|^{2})dx\big{)}^{\frac{3}{4}}$ $\displaystyle\geq
h\Big{(}\big{(}\int_{\mathbb{R}^{3}}(|\nabla u|^{2}+|\nabla
v|^{2})dx\big{)}^{\frac{1}{2}}\Big{)},$

where $h(t):(0,+\infty)\rightarrow\mathbb{R}$ defined by

$h(t)=\frac{1}{2}t^{2}-\frac{1}{4}(\mathcal{D}_{1}+\mathcal{D}_{2}+\rho\mathcal{D}_{3})t^{3}-\frac{1}{2}|\beta|\mathcal{D}_{4}t^{\frac{3}{2}}.$

######  Lemma 4.2.

When
$|\beta|\left(2b^{\frac{3}{2}}_{1}+b^{\frac{3}{2}}_{2}\right)C^{3}_{3,3}C^{2}_{3,4}\sqrt{\mu_{1}b_{1}+\mu_{2}b_{2}+\rho
b^{\frac{1}{2}}_{1}b^{\frac{1}{2}}_{2}}<\frac{2\sqrt{6}}{3}$ holds, then
$h(t)$ has exactly two critical points, one is a local minimum at negative
level, the other one is a global maximum at positive level. Further, there
exists $0<R_{0}<R_{1}$ such that $h(R_{0})=h(R_{1})=0$, $h(t)>0$ if and only
if $t\in(R_{0},R_{1})$.

###### Proof.

Since

$h(t)=t^{\frac{3}{2}}\Big{[}\frac{1}{2}t^{\frac{1}{2}}-\frac{1}{4}C^{4}_{3,4}(\mu_{1}b_{1}+\mu_{2}b_{2}+\rho
b^{\frac{1}{2}}_{1}b^{\frac{1}{2}}_{2})t^{\frac{3}{2}}-\frac{1}{2}|\beta|\big{(}\frac{2}{3}b^{\frac{3}{2}}_{1}+\frac{1}{3}b^{\frac{3}{2}}_{2}\big{)}C^{3}_{3,3}\Big{]}$

it is easy to see that $h(t)>0$ if and only if
$\varphi(t)>\frac{1}{2}|\beta|\big{(}\frac{2}{3}b^{\frac{3}{2}}_{1}+\frac{1}{3}b^{\frac{3}{2}}_{2}\big{)}C^{3}_{3,3}$,
where
$\varphi(t)=\frac{1}{2}t^{\frac{1}{2}}-\frac{1}{4}C^{4}_{3,4}(\mu_{1}b_{1}+\mu_{2}b_{2}+\rho
b^{\frac{1}{2}}_{1}b^{\frac{1}{2}}_{2})t^{\frac{3}{2}}$, since

$\varphi^{\prime}(t)=\frac{1}{4}t^{-\frac{1}{2}}-\frac{3}{8}C^{4}_{3,4}(\mu_{1}b_{1}+\mu_{2}b_{2}+\rho
b^{\frac{1}{2}}_{1}b^{\frac{1}{2}}_{2})t^{\frac{1}{2}},$

it is easy to see that $\varphi(t)$ has a unique global maximum point at
positive level in
$\widetilde{t}=\frac{2}{3C^{4}_{3,4}(\mu_{1}b_{1}+\mu_{2}b_{1}+\rho
b^{\frac{1}{2}}_{1}b^{\frac{1}{2}}_{2})}$ and

$\varphi(\widetilde{t})=\frac{\sqrt{6}}{9}\frac{1}{\sqrt{C^{4}_{3,4}(\mu_{1}b_{1}+\mu_{2}b_{2}+\rho
b^{\frac{1}{2}}_{1}b^{\frac{1}{2}}_{2})}},$

therefore $h(t)$ is positive on an open interval $(R_{0},R_{1})$ if and only
if
$\varphi(\widetilde{t})>\frac{1}{2}|\beta|\left(\frac{2}{3}b^{\frac{3}{2}}_{1}+\frac{1}{3}b^{\frac{3}{2}}_{2}\right)C^{3}_{3,3}$,
so

$\displaystyle|\beta|\big{(}2b^{\frac{3}{2}}_{1}+b^{\frac{3}{2}}_{2}\big{)}C^{3}_{3,3}C^{2}_{3,4}\sqrt{\mu_{1}b_{1}+\mu_{2}b_{2}+\rho
b^{\frac{1}{2}}_{1}b^{\frac{1}{2}}_{2}}<\frac{2\sqrt{6}}{3}.$ (4.10)

Since

$\displaystyle h^{\prime}(t)$
$\displaystyle=t^{\frac{1}{2}}\Big{[}t^{\frac{1}{2}}-\frac{3}{4}C^{4}_{3,4}(\mu_{1}b_{1}+\mu_{2}b_{2}+\rho
b^{\frac{1}{2}}_{1}b^{\frac{1}{2}}_{2})t^{\frac{3}{2}}-\frac{3}{4}|\beta|\big{(}\frac{2}{3}b^{\frac{3}{2}}_{1}+\frac{1}{3}b^{\frac{3}{2}}_{2}\big{)}C^{3}_{3,3}\Big{]}=t^{\frac{1}{2}}g(t),$

we can deduce that when

$\displaystyle g(\frac{4}{9C^{4}_{3,4}(\mu_{1}b_{1}+\mu_{2}b_{1}+\rho
b^{\frac{1}{2}}_{1}b^{\frac{1}{2}}_{2})})=\frac{4}{9\sqrt{C^{4}_{3,4}(\mu_{1}b_{1}+\mu_{2}b_{1}+\rho
b^{\frac{1}{2}}_{1}b^{\frac{1}{2}}_{2})}}-\frac{3}{4}|\beta|\big{(}\frac{2}{3}b^{\frac{3}{2}}_{1}+\frac{1}{3}b^{\frac{3}{2}}_{2}\big{)}C^{3}_{3,3}>0,$
(4.11)

then $h(t)$ has exactly two critical points, one is a local minimum at
negative level, the other one is a global maximum at positive level. It is
easy to see that when (4.10) holds, then (4.11) also holds. ∎

Let

$N:=\Big{\\{}(u,v)\in H^{1}(\mathbb{R}^{4})\times
H^{1}(\mathbb{R}^{4})~{}|~{}\beta\int_{\mathbb{R}^{3}}u^{2}vdx>0\Big{\\}},$

and for simplicity, we still denote

$\mathcal{P}_{b_{1},b_{2}}=\Big{\\{}(u,v)\in(\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}})\cap
N~{}|~{}P_{b_{1},b_{2}}(u,v)=0\Big{\\}}.$

######  Lemma 4.3.

Under the assumption (4.10), then $\mathcal{P}^{0}_{b_{1},b_{2}}=\emptyset$
and $\mathcal{P}_{b_{1},b_{2}}$ is a $C^{1}$ submanifold in
$H^{1}(\mathbb{R}^{3})\times H^{1}(\mathbb{R}^{3})$ with codimension 3.

###### Proof.

Assume by contradiction that there exists a
$(u,v)\in\mathcal{P}^{0}_{b_{1},b_{2}}$ such that

$\displaystyle\int_{\mathbb{R}^{3}}(|\nabla u|^{2}+|\nabla
v|^{2})dx=\frac{3}{4}\int_{\mathbb{R}^{3}}(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2})dx+\frac{3}{4}\beta\int_{\mathbb{R}^{3}}u^{2}vdx$ (4.12)

and

$\displaystyle 2\int_{\mathbb{R}^{3}}(|\nabla u|^{2}+|\nabla
v|^{2})dx-\frac{9\beta}{8}\int_{\mathbb{R}^{3}}u^{2}vdx-\frac{9}{4}\int_{\mathbb{R}^{3}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx=0.$ (4.13)

From (4.12) and (4.13), we obtain

$\displaystyle\beta\int_{\mathbb{R}^{3}}u^{2}vdx=2\int_{\mathbb{R}^{3}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx.$ (4.14)

Thus, from (2.1)-(2.3), we have

$\displaystyle\int_{\mathbb{R}^{3}}(|\nabla u|^{2}+|\nabla v|^{2})dx$
$\displaystyle=\frac{9}{4}\int_{\mathbb{R}^{3}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx$ (4.15)
$\displaystyle\leq\frac{9\mu_{1}}{4}C^{4}_{3,4}b_{1}\|\nabla
u\|^{3}_{L^{2}(\mathbb{R}^{3})}+\frac{9\mu_{2}}{4}C^{4}_{3,4}b_{2}\|\nabla
v\|^{3}_{L^{2}(\mathbb{R}^{3})}$ $\displaystyle\quad+\frac{9}{4}\rho
C^{4}_{3,4}b^{\frac{1}{2}}_{1}b^{\frac{1}{2}}_{2}\left[\|\nabla
u\|^{2}_{L^{2}(\mathbb{R}^{3})}+\|\nabla
v\|^{2}_{L^{2}(\mathbb{R}^{3})}\right]^{\frac{3}{2}}$
$\displaystyle\leq\frac{9}{4}C^{4}_{3,4}(\mu_{1}b_{1}+\mu_{2}b_{2}+\rho
b^{\frac{1}{2}}_{1}b^{\frac{1}{2}}_{2})\left[\|\nabla
u\|^{2}_{L^{2}(\mathbb{R}^{3})}+\|\nabla
v\|^{2}_{L^{2}(\mathbb{R}^{3})}\right]^{\frac{3}{2}},$
$\displaystyle\int_{\mathbb{R}^{3}}(|\nabla u|^{2}+|\nabla v|^{2})dx$
$\displaystyle=\frac{9}{8}\beta\int_{\mathbb{R}^{3}}u^{2}vdx$ (4.16)
$\displaystyle\leq\frac{9}{8}|\beta|\left[\left(\frac{2}{3}b^{\frac{3}{2}}_{1}+\frac{1}{3}b^{\frac{3}{2}}_{2}\right)C^{3}_{3,3}\left[\|\nabla
u\|^{2}_{L^{2}(\mathbb{R}^{3})}+\|\nabla
v\|^{2}_{L^{2}(\mathbb{R}^{3})}\right]^{\frac{3}{4}}\right].$

So, by (4.15) and (4.16), we get

$\frac{1}{3\sqrt{C^{4}_{3,4}(\mu_{1}b_{1}+\mu_{2}b_{1}+\rho
b^{\frac{1}{2}}_{1}b^{\frac{1}{2}}_{2})}}\leq\frac{9}{16}|\beta|\mathcal{D}_{4},$

therefore

$\displaystyle|\beta|\left(2b^{\frac{3}{2}}_{1}+b^{\frac{3}{2}}_{2}\right)C^{3}_{3,3}C^{2}_{3,4}\sqrt{\mu_{1}b_{1}+\mu_{2}b_{2}+\rho
b^{\frac{1}{2}}_{1}b^{\frac{1}{2}}_{2}}\geq\frac{16}{9},$ (4.17)

contract the assumption (4.10), which implies that
$\mathcal{P}^{0}_{b_{1},b_{2}}=\emptyset$. Next, we show that
$\mathcal{P}_{b_{1},b_{2}}$ is a smooth manifold of codimension 3 on
$H^{1}(\mathbb{R}^{3})\times H^{1}(\mathbb{R}^{3})$. It is easy to see that
$\mathcal{P}_{b_{1},b_{2}}$ is defined by $P_{b_{1},b_{2}}(u,v)=0$, $G(u)=0$,
$F(u)=0$ and $E(u,v)>0$, where

$G(u)=\int_{\mathbb{R}^{3}}u^{2}dx-b^{2}_{1},\ \
F(v)=\int_{\mathbb{R}^{3}}v^{2}dx-b^{2}_{2}\ \ \text{and}\ \
E(u,v)=\beta\int_{\mathbb{R}^{3}}u^{2}v.$

Since $P_{b_{1},b_{2}}(u,v)$, $G(u)$, $F(u)$ and $E(u,v)$ are class of
$C^{1}$, we only need to check that

$d(P_{b_{1},b_{2}}(u,v),G(u),F(v),E(u,v)):H\rightarrow\mathbb{R}^{3}\ \text{
is surjective},$

for every $(u,v)\in(G^{-1}(0)\times F^{-1}(0))\cap E^{-1}(0)\cap
P^{-1}_{b_{1},b_{2}}(0)$. If this is not true, $dP_{b_{1},b_{2}}(u,v)$ has to
be linearly dependent from $dG(u)$, $dF(v)$ and $dE(u,v)$ i.e. there exist
$\nu_{1},\nu_{2},\nu_{3}\in\mathbb{R}$ such that

$\begin{cases}2\int_{\mathbb{R}^{3}}\nabla
u\nabla\varphi+\nu_{1}u\varphi+\nu_{3}\beta
uv\varphi=3\int_{\mathbb{R}^{3}}(\mu_{1}u^{3}\varphi+\rho
v^{2}u\varphi)+\frac{3\beta}{2}\int_{\mathbb{R}^{3}}u\varphi v&\text{in}\ \
\mathbb{R}^{3},\\\ 2\int_{\mathbb{R}^{3}}\nabla
v\nabla\psi+\nu_{2}v\psi+\frac{\nu_{3}}{2}\beta
u^{2}\psi=3\int_{\mathbb{R}^{3}}(\mu_{2}v^{3}\psi+\rho
u^{2}v\psi)+\frac{3\beta}{4}\int_{\mathbb{R}^{3}}u^{2}\psi&\text{in}\ \
\mathbb{R}^{3},\end{cases}$

for every $(\varphi,\psi)\in H^{1}(\mathbb{R}^{3})\times
H^{1}(\mathbb{R}^{3})$, so

$\begin{cases}-2[\Delta u+\nu_{1}u+\nu_{3}\beta uv]=3(\mu_{1}u^{3}+\rho
v^{2}u)+\frac{3\beta}{2}uv&\text{in}\ \ \mathbb{R}^{3},\\\ -2[\Delta
v+\nu_{2}v+\frac{\nu_{3}}{2}\beta u^{2}]=3(\mu_{2}v^{3}+\rho
u^{2}v)+\frac{3\beta}{4}u^{2}&\text{in}\ \ \mathbb{R}^{3}.\end{cases}$

The Pohozaev identity for above system is

$\displaystyle\int_{\mathbb{R}^{3}}(|\nabla u|^{2}+|\nabla
v|^{2})dx-\frac{9}{8}\int_{\mathbb{R}^{3}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx-\frac{3}{2}\big{(}\frac{3}{8}-\frac{\nu_{3}}{2}\big{)}\beta\int_{\mathbb{R}^{3}}u^{2}vdx=0.$

Then $\int_{\mathbb{R}^{3}}(|\nabla u|^{2}+|\nabla
v|^{2})dx=\frac{9}{8}\int_{\mathbb{R}^{3}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx$. Since $(u,v)\in(G^{-1}(0)\times F^{-1}(0))\cap
E^{-1}(0)\cap P^{-1}_{b_{1},b_{2}}(0)$, we have $\int_{\mathbb{R}^{3}}(|\nabla
u|^{2}+|\nabla
v|^{2})dx=\frac{3}{4}\int_{\mathbb{R}^{3}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx$, this is a contradiction. By Proposition A.1 of [33], we
get that if
$m_{\beta}:=\inf\limits_{(u,v)\in\mathcal{P}_{b_{1},b_{2}}}J_{\beta}=J_{\beta}(\bar{u},\bar{v})$,
then there exist $\lambda_{i}\in\mathbb{R}(i=4)$, such that

$\displaystyle-(1+\lambda_{4})\Delta\bar{u}+\lambda_{1}\nu_{1}\bar{u}+\lambda_{3}\beta\bar{u}\bar{v}=(1+\frac{3\lambda_{4}}{2})(\mu_{1}\bar{u}^{3}+\rho\bar{v}^{2}u)+(1+\frac{3\lambda_{4}}{4})\beta\bar{u}\bar{v},$
$\displaystyle-(1+\lambda_{4})\Delta\bar{v}+\lambda_{2}\bar{v}+\frac{\lambda_{3}}{2}\beta\bar{u}^{2}=(1+\frac{3\lambda_{4}}{2}\lambda_{4})(\mu_{2}\bar{v}^{3}+\rho\bar{u}^{2}\bar{v})+(1+\frac{3\lambda_{4}}{4})\frac{\beta}{2}\bar{u}^{2}.$

Since $E(\bar{u},\bar{v})>0$, from Theorem 1 of [13] or the proof of Lemma
2.11 in [33], we have $\lambda_{3}=0$. Therefore, we obtain that the
restricted set $N$ does not change the structure of the manifold
$\mathcal{P}_{b_{1},b_{2}}$. Then, we have that $\mathcal{P}_{b_{1},b_{2}}$ is
a smooth manifold of codimension 3 on
$\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$. ∎

######  Lemma 4.4.

When $\beta\int_{\mathbb{R}^{3}}u^{2}vdx>0$. For every
$(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$, the function
$\Psi_{u,v}(t)$ has exactly two critical points $s_{u,v}<t_{u,v}\in\mathbb{R}$
and two zeros $c_{u,v}<d_{u,v}\in\mathbb{R}$ with
$s_{u,v}<c_{u,v}<t_{u,v}<d_{u,v}$. Moreover,

  1. $(1)$

$s_{u,v}\star(u,v)\in\mathcal{P}^{+}_{b_{1},b_{2}}(u,v)$ and
$t_{u,v}\star(u,v)\in\mathcal{P}^{-}_{b_{1},b_{2}}(u,v)$, and if
$t\star(u,v)\in\mathcal{P}_{b_{1},b_{2}}(u,v)$, then either $t=s_{u,v}$ or
$t=t_{u,v}.$

  2. $(2)$

$(\int_{\mathbb{R}^{3}}(|\nabla(t\star u)|^{2}+|\nabla(t\star
v)|^{2})dx)^{\frac{1}{2}}\leq R_{0}$ for every $t\leq c_{u,v},$ and

$\displaystyle J_{\beta}(u,v)(s_{u,v}\star(u,v))$
$\displaystyle=\min\bigg{\\{}J_{\beta}(t\star(u,v):t\in\mathbb{R}\ \text{and}\
(\int_{\mathbb{R}^{3}}(|\nabla(t\star u)|^{2}+|\nabla(t\star
v)|^{2})dx)^{\frac{1}{2}}<R_{0}\bigg{\\}}<0,$

where $R_{0}$ is defined in Lemma 4.2.

  3. $(3)$

We have

$J_{\beta}(t_{u,v}\star(u,v))=\max\\{J_{\beta}(t\star(u,v)):t\in\mathbb{R}\\}>0$

and $\Psi_{u,v}(t)$ is strictly decreasing and concave on $(t_{u,v},+\infty)$.
In particular, if $t_{u,v}<0$, then $P_{b_{1},b_{2}}(u,v)<0.$

  4. $(4)$

The maps
$(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}:s_{u,v}\in\mathbb{R}$ and
$(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}:t_{u,v}\in\mathbb{R}$ are
of class $C^{1}$.

###### Proof.

Let $(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$, since
$t\star(u,v)\in\mathcal{P}_{b_{1},b_{2}}(u,v)$ if and only if
$(\Psi_{u,v})^{\prime}(t)=0$. Thus, we first show that $\Psi_{u,v}(t)$ has at
least two critical points. From (4.9), we have

$\displaystyle\Psi_{u,v}(t)$ $\displaystyle=J_{\beta}(t\star(u,v))\geq
h\Big{(}\big{[}\int_{\mathbb{R}^{3}}(|\nabla(t\star u)|^{2}+|\nabla(t\star
v)|^{2})dx\big{]}^{\frac{1}{2}}\Big{)}$
$\displaystyle=h\Big{(}e^{t}\big{[}\int_{\mathbb{R}^{3}}(|\nabla
u|^{2}+|\nabla v|^{2})dx\big{]})^{\frac{1}{2}}\Big{)}.$

Thus, the $C^{2}$ function $\Psi_{u,v}(t)$ is positive on

$\Big{(}\ln\frac{R_{0}}{\big{[}\int_{\mathbb{R}^{3}}(|\nabla u|^{2}+|\nabla
v|^{2})dx\big{]}^{\frac{1}{2}}},\ln\frac{R_{1}}{\big{[}\int_{\mathbb{R}^{3}}(|\nabla
u|^{2}+|\nabla v|^{2})dx\big{]}^{\frac{1}{2}}}\Big{)},$

$\Psi_{u,v}(+\infty)=-\infty$ and $\Psi_{u,v}(-\infty)=0^{-}$.

Indeed, $(\Psi_{u,v})^{\prime}(t)=0$ implies that
$g(t)=\frac{3\beta}{4}\int_{\mathbb{R}^{3}}u^{2}vdx$ where

$g(t)=e^{\frac{t}{2}}\int_{\mathbb{R}^{3}}(|\nabla u|^{2}+|\nabla
v|^{2})dx-\frac{3e^{\frac{3t}{2}}}{4}\int_{\mathbb{R}^{3}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx.$

It is easy to see that $g(t)$ has a unique maximum point, thus the above
equation has at most two solutions. If
$\frac{3\beta}{4}\int_{\mathbb{R}^{3}}u^{2}vdx>0$. So,
$\Psi_{u,v}(-\infty)=0^{-}$, $\Psi_{u,v}(+\infty)=-\infty$. It is easy to see
that $\Psi_{u,v}(t)$ has a local minimum point $s_{u,v}$ at negative level in
$(0,\ln\frac{R_{0}}{\left(\int_{\mathbb{R}^{3}}(|\nabla u|^{2}+|\nabla
v|^{2})dx\right)^{\frac{1}{2}}})$ and has a global maximum point $t_{u,v}$ at
positive level in $\Big{(}\ln\frac{R_{0}}{\big{[}\int_{\mathbb{R}^{3}}(|\nabla
u|^{2}+|\nabla
v|^{2})dx\big{]}^{\frac{1}{2}}},\ln\frac{R_{1}}{\big{[}\int_{\mathbb{R}^{3}}(|\nabla
u|^{2}+|\nabla v|^{2})dx\big{]}^{\frac{1}{2}}}\Big{)}.$ Next, we show that
$\Psi_{u,v}(t)$ has no other critical points. From
$(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}},$ $t\in\mathbb{R}$ is a
critical point of $\Psi_{u,v}(t)$ if and only if
$t\star(u,v)\in\mathcal{P}_{b_{1},b_{2}},$ we have $s_{u,v}\star(u,v),\ \
t_{u,v}\star(u,v)\in\mathcal{P}_{b_{1}\times b_{2}}$ and
$t\star(u,v)\in\mathcal{P}_{b_{1}\times b_{2}}$ if and only if $t=s_{u,v}$ or
$t=t_{u,v}$. Since $s_{u,v}$ is a local minimum point of $\Psi_{u,v}(t)$, we
know that $(\Psi_{u,v})^{\prime\prime}(s_{u,v})\geq 0$, from
$\mathcal{P}^{0}_{b_{1},b_{2}}=\emptyset$, we know that
$(\Psi_{u,v})^{\prime\prime}(s_{u,v})\neq 0$, thus
$(\Psi_{u,v})^{\prime\prime}(s_{u,v})>0$, therefore
$s_{u,v}\star(u,v)\in\mathcal{P}^{+}_{b_{1},b_{2}}.$ Similarly, we have
$t_{u,v}\star(u,v)\in\mathcal{P}^{-}_{b_{1},b_{2}}.$ By the monotonicity and
the behavior at infinity of $\Psi_{u,v}(t)$, we know that $\Psi_{u,v}(t)$ has
exactly two zeros $c_{u,v}<d_{u,v}$ with $s_{u,v}<c_{u,v}<t_{u,v}<d_{u,v}$ and
$\Psi_{u,v}(t)$ has exactly two inflection points, in particular,
$\Psi_{u,v}(t)$ is concave on $[t_{u,v},+\infty)$ and hence if $t_{u,v}<0$,
then $P_{b_{1},b_{2}}(u,v)=(\Psi_{u,v})^{\prime}(0)<0.$ Finally, we prove that
$(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}\mapsto
s_{u,v}\in\mathbb{R}$ and
$(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}\mapsto
t_{u,v}\in\mathbb{R}$ are of class $C^{1}$. Indeed, we can apply the implicit
function theorem on the $C^{1}$ function
$\Phi(t,u,v)=(\Psi_{u,v})^{\prime}(t)$, then
$\Phi(s_{u,v},u,v)=(\Psi_{u,v})^{\prime}(s_{u,v})=0,\partial_{s}\Phi(s_{u,v},u,v)=(\Psi_{u,v})^{\prime\prime}(s_{u,v})<0$,
thus $(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}\mapsto
s_{u,v}\in\mathbb{R}$ is class of $C^{1}$. Similarly, we can prove that
$(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}\mapsto
t_{u,v}\in\mathbb{R}$ is class of $C^{1}$. ∎

###  4.1 Existence of a local minimizer

For $k>0$, set

$A_{k}=\Big{\\{}(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}:\big{(}\int_{\mathbb{R}^{3}}(|\nabla
u|^{2}+|\nabla v|^{2})dx\big{)}^{\frac{1}{2}}<k\Big{\\}},$

and

$m^{+}_{\beta}(b_{1},b_{2})=\inf_{(u,v)\in A_{R_{0}}}J_{\beta}(u,v).$

From Lemma 4.4, we have following corollary

######  Corollary 4.1.

The set $\mathcal{P}^{+}_{b_{1},b_{2}}$ is contained in

$A_{R_{0}}=\Big{\\{}(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}:\big{(}\int_{\mathbb{R}^{3}}(|\nabla
u|^{2}+|\nabla v|^{2})dx\big{)}^{\frac{1}{2}}<R_{0}\Big{\\}}$

and

$\sup_{\mathcal{P}^{+}_{b_{1},b_{2}}}J_{\beta}(u,v)\leq
0\leq\inf_{\mathcal{P}^{-}_{b_{1},b_{2}}}J_{\beta}(u,v).$

######  Lemma 4.5.

We have $m^{+}_{\beta}(b_{1},b_{2})\in(-\infty,0)$ that

$m^{+}_{\beta}(b_{1},b_{2})=\inf_{\mathcal{P}_{b_{1},b_{2}}}J_{\beta}(u,v)=\inf_{\mathcal{P}^{+}_{b_{1},b_{2}}}J_{\beta}(u,v)\
\text{and}\ m^{+}_{\beta}(b_{1},b_{2})<\inf_{\overline{A_{R_{0}}}\setminus
A_{R_{0}-\rho}}J_{\beta}(u,v).$

###### Proof.

For $(u,v)\in A_{R_{0}}$, we have

$J_{\beta}(u,v)\geq h\Big{(}\big{(}\int_{\mathbb{R}^{3}}(|\nabla
u|^{2}+|\nabla
v|^{2})dx\big{)}^{\frac{1}{2}}\Big{)}\geq\min_{t\in[0,R_{0}]}h(t)>-\infty.$

Therefore, $m^{+}_{\beta}(b_{1},b_{2})>-\infty.$ Moreover, there exists
$(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$, such that

$\Big{(}\int_{\mathbb{R}^{3}}(|\nabla(t\star u)|^{2}+|\nabla(t\star
v)|^{2})dx\Big{)}^{\frac{1}{2}}<R_{0}$

and $J_{\beta}(t\star(u,v))<0$ for $t\ll-1$. Hence
$m^{+}_{\beta}(b_{1},b_{2})<0.$ Since $\mathcal{P}^{+}_{b_{1},b_{2}}\subset
A_{R_{0}}$, we know that
$m^{+}_{\beta}(b_{1},b_{2})\leq\inf_{\mathcal{P}^{+}_{b_{1},b_{2}}}J_{\beta}(u,v).$
On the other hand, if $(u,v)\in A_{R_{0}}$, then
$s_{u,v}\star(u,v)\in\mathcal{P}^{+}_{b_{1},b_{2}}\subset A_{R_{0}}$ and

$\displaystyle J_{\beta}(u,v)(s_{u,v}\star(u,v))=\min\Big{\\{}J_{\beta}$
$\displaystyle(t\star(u,v)):t\in\mathbb{R}\ \text{and}\
\big{(}\int_{\mathbb{R}^{3}}(|\nabla(t\star u)|^{2}+|\nabla(t\star
v)|^{2})dx\big{)}^{\frac{1}{2}}<R_{0}\Big{\\}}$ $\displaystyle\leq
J_{\beta}(u,v),$

where $R_{0}$ is defined in Lemma 4.2, so
$\inf_{\mathcal{P}^{+}_{b_{1},b_{2}}}J_{\beta}(u,v)\leq
m^{+}_{\beta}(b_{1},b_{2})$. Since $J_{\beta}(u,v)>0$ on
$\mathcal{P}^{-}_{b_{1},b_{2}}$, we know that
$\inf_{\mathcal{P}^{+}_{b_{1},b_{2}}}J_{\beta}(u,v)=\inf_{\mathcal{P}_{b_{1},b_{2}}}J_{\beta}(u,v).$
Finally, by the continuity of $h$, there exists $\rho>0$ such that
$h(t)\geq\frac{m^{+}_{\beta}(b_{1},b_{2})}{2}$ if $t\in[R_{0}-\rho,R_{0}]$.
Therefore,

$J_{\beta}(u,v)\geq h\Big{(}\big{(}\int_{\mathbb{R}^{3}}(|\nabla
u|^{2}+|\nabla
v|^{2})dx\big{)}^{\frac{1}{2}}\Big{)}\geq\frac{m^{+}_{\beta}(b_{1},b_{2})}{2}>m^{+}_{\beta}(b_{1},b_{2})$

for every $(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$ with
$R_{0}-\rho\leq\left(\int_{\mathbb{R}^{3}}(|\nabla u|^{2}+|\nabla
v|^{2})dx\right)^{\frac{1}{2}}\leq R_{0}$. This completes the proof. ∎

######  Lemma 4.6.

Under the assumption (4.10) holds, we have

$m^{+}(b_{1},b_{2})<\min\\{m^{+}(b_{1},0),m^{+}(0,b_{2})\\}.$

###### Proof.

From [5], we know that $m^{+}(0,b_{2})$ can be achieved by
$v^{\ast}\in\mathrm{T}_{b_{2}}$ and $v^{\ast}$ is radially symmetric and
decreasing. We choose aproper test function $u\in\mathrm{T}_{b_{1}}$ such that
$(t\star u,v^{\ast})\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$. From Lemma
4.4, we obtain

$\displaystyle
h(t)<h_{1}(t)=\frac{1}{2}t^{2}-\frac{1}{4}\mu_{2}C^{4}_{3,4}b_{1}t^{3}-\frac{1}{2}|\beta|\frac{2}{3}b^{\frac{3}{2}}_{2}C^{3}_{3,3}t^{\frac{3}{2}}.$
(4.18)

By direct calculations, there exists $0<t^{\ast}<R_{0}$ such that
$<h_{1}(t^{\ast})=0$. From [41], we have

$m^{+}(0,b_{2})=\inf_{v\in\mathcal{P}^{+}_{0,b_{2}}}J_{\beta}(0,v)=\inf_{v\in\mathrm{T}_{b_{2}}\cap
B(t^{\ast})}J_{\beta}(0,v).$

Therefore, from the analysis in Lemma 4.2, we have

$\|\nabla v^{\ast}\|_{L^{2}}\leq
t^{\ast}<R_{0}<\widetilde{t}=\frac{2}{3C^{4}_{3,4}(\mu_{1}b_{1}+\mu_{2}b_{1}+\rho
b^{\frac{1}{2}}_{1}b^{\frac{1}{2}}_{2})}.$

Since $h(R_{0})=h(R_{1})=0$ and the monotonicity of $h(t)$, we deduce that
$(t\star u,v^{\ast})\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}\cap
A_{R_{0}}$ for $t\ll-1$, therefore,

$\displaystyle m^{+}(b_{1},b_{2})$
$\displaystyle=\inf_{(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}\cap
A_{R_{0}}}J(u,v)\leq J(t\star u,v^{\ast})$
$\displaystyle=\frac{1}{2}\int_{\mathbb{R}^{3}}|\nabla
v^{\ast}|^{2}dx-\frac{1}{4}\mu_{2}\int_{\mathbb{R}^{N}}|v^{\ast}|^{4}dx-\frac{\beta}{2}\int_{\mathbb{R}^{3}}|t\star
u|^{2}(v^{\ast})dx$
$\displaystyle\quad+\frac{e^{2t}}{2}\int_{\mathbb{R}^{3}}|\nabla
u|^{2}dx-\frac{e^{3t}}{4}\int_{\mathbb{R}^{N}}\mu_{2}u^{4}dx-\frac{1}{2}\int_{\mathbb{R}^{N}}\rho|t\star
u|^{2}(v^{\ast})^{2}dx$ $\displaystyle<J(0,v^{\ast})=m^{+}(0,b_{2}).$

Similarly, we have

$m^{+}(b_{1},b_{2})<m^{+}(b_{1},0).$

Hence, the proof is completed. ∎

######  Lemma 4.7.

Let $\\{(u_{n},v_{n})\\}\subset H^{1}(\mathbb{R}^{3})\times
H^{1}(\mathbb{R}^{3})$ be a minimizing sequence for
$J_{\beta}(u,v)|_{\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}}$ at level
$m^{+}_{\beta}(b_{1},b_{2})$. Then $\\{(u_{n},v_{n})\\}$ is bounded in
$H^{1}(\mathbb{R}^{3})\times H^{1}(\mathbb{R}^{3})$.

###### Proof.

Since $P_{b_{1},b_{2}}(u_{n},v_{n})\rightarrow 0$, we have

$\displaystyle P_{b_{1},b_{2}}(u_{n},v_{n})=\int_{\mathbb{R}^{3}}(|\nabla
u_{n}|^{2}+|\nabla
v_{n}|^{2})dx-\frac{3}{4}\int_{\mathbb{R}^{3}}\left(\mu_{1}u_{n}^{4}+\mu_{2}v_{n}^{4}+2\rho
u_{n}^{2}v_{n}^{2}\right)dx-\frac{3}{4}\beta\int_{\mathbb{R}^{3}}u_{n}^{2}v_{n}dx=o_{n}(1).$

Thus, from (2.3), we have

$\displaystyle J_{\beta}(u_{n},v_{n})$
$\displaystyle=\frac{1}{6}\int_{\mathbb{R}^{3}}(|\nabla u_{n}|^{2}+|\nabla
v_{n}|^{2})dx-\frac{\beta}{4}\int_{\mathbb{R}^{3}}u_{n}^{2}v_{n}dx+o_{n}(1)$
$\displaystyle\geq\frac{1}{6}\int_{\mathbb{R}^{3}}(|\nabla u_{n}|^{2}+|\nabla
v_{n}|^{2})dx$
$\displaystyle\quad-\frac{|\beta|}{4}\Big{[}\big{(}\frac{2}{3}b^{\frac{3}{2}}_{1}+\frac{1}{3}b^{\frac{3}{2}}_{2}\big{)}C^{3}_{3,3}\big{[}\|\nabla
u_{n}\|^{2}_{L^{2}(\mathbb{R}^{3})}+\|\nabla
v_{n}\|^{2}_{L^{2}(\mathbb{R}^{3})}\big{]}^{\frac{3}{4}}\Big{]}.$

Since $\\{(u_{n},v_{n})\\}$ is a minimizer sequence for
$J_{\beta}(u,v)|_{\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}}$ at level
$m^{+}_{\beta}(b_{1},b_{2})$, we have $J_{\beta}(u_{n},v_{n})\leq
m^{+}_{\beta}(b_{1},b_{2})+1$ for $n$ large. Hence

$\displaystyle\frac{1}{6}\int_{\mathbb{R}^{3}}(|\nabla u_{n}|^{2}+|\nabla
v_{n}|^{2})dx$
$\displaystyle\leq\frac{|\beta|}{4}\Big{[}\big{(}\frac{2}{3}b^{\frac{3}{2}}_{1}+\frac{1}{3}b^{\frac{3}{2}}_{2}\big{)}C^{3}_{3,3}\big{[}\|\nabla
u_{n}\|^{2}_{L^{2}(\mathbb{R}^{3})}+\|\nabla
v_{n}\|^{2}_{L^{2}(\mathbb{R}^{3})}\big{]}^{\frac{3}{4}}\Big{]}+m^{+}_{\beta}(b_{1},b_{2})+2,$

so $\\{(u_{n},v_{n})\\}$ is bounded in $H^{1}(\mathbb{R}^{3})\times
H^{1}(\mathbb{R}^{3})$. This completes the proof. ∎

######  Lemma 4.8.

Let $\\{(u_{n},v_{n})\\}\subset\mathrm{T}_{b_{1,r}}\times\mathrm{T}_{b_{2,r}}$
be a nonnegative minimizing sequence for
$J_{\beta}(u,v)|_{\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}}$ at level
$m^{+}_{\beta}(b_{1},b_{2})$ with additional properties
$P_{b_{1},b_{2}}(u_{n},v_{n})\rightarrow 0$ and
$u^{-}_{n},v^{-}_{n}\rightarrow 0$ a.e. in $\mathbb{R}^{3}$, then up to a
subsequence $(u_{n},v_{n})\rightarrow(u,v)$ in $H^{1}(\mathbb{R}^{3})\times
H^{1}(\mathbb{R}^{3})$, where $(u,v)$ is a positive solution of (1.1) for some
$\lambda_{1},\lambda_{2}>0.$

###### Proof.

From Lemma 4.7, we known that $\\{(u_{n},v_{n})\\}$ is bounded in
$H_{r}^{1}(\mathbb{R}^{3})\times H_{r}^{1}(\mathbb{R}^{3})$, thus, by the
Sobolev embedding theorem, we have
$H_{r}^{1}(\mathbb{R}^{3})\hookrightarrow\hookrightarrow
L^{p}_{r}(\mathbb{R}^{3})$ for $2<p<6$, thus there exists a $(u,v)\in
H_{r}^{1}(\mathbb{R}^{3})\times H_{r}^{1}(\mathbb{R}^{3})$ such that
$(u_{n},v_{n})\rightharpoonup(u,v)$ in $H_{r}^{1}(\mathbb{R}^{3})\times
H_{r}^{1}(\mathbb{R}^{3})$, $(u_{n},v_{n})\rightarrow(u,v)$ in
$L^{p}_{r}(\mathbb{R}^{3})\times L^{p}_{r}(\mathbb{R}^{3})\ \ \text{for }\
2<p<6$ and $(u_{n},v_{n})\rightarrow(u,v)$ a.e in $\mathbb{R}^{3}$. Hence
$u,v\geq 0$ are radial functions. Since
$J^{\prime}\mid_{\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}}(u_{n},v_{n})\rightarrow
0$, by the Lagrange multipliers rule, we know that there exists a sequence
$(\lambda_{1,n},\lambda_{2,n})\in\mathbb{R}^{2}$ such that

$\displaystyle\int_{\mathbb{R}^{3}}(\nabla u_{n}\nabla\varphi+\nabla
v_{n}\nabla\psi)dx+\int_{\mathbb{R}^{3}}(\lambda_{1,n}u_{n}\varphi+\lambda_{2,n}v_{n}\psi)dx$
(4.19)
$\displaystyle-\int_{\mathbb{R}^{3}}(\mu_{1}u_{n}^{3}\varphi+\mu_{2}v_{n}^{3}\psi+\rho
v_{n}^{2}u_{n}\varphi+\rho u_{n}^{2}v_{n}\psi)dx$
$\displaystyle-\beta\int_{\mathbb{R}^{3}}u_{n}\varphi
v_{n}-\frac{\beta}{2}\int_{\mathbb{R}^{3}}u_{n}^{2}\psi=o(1)\|(\psi,\varphi)\|_{H^{1}(\mathbb{R}^{3})\times
H^{1}(\mathbb{R}^{3})}\ \ \text{in}\ \ \mathbb{R}^{3},$

for every $(\varphi,\psi)\in H^{1}(\mathbb{R}^{3})\times
H^{1}(\mathbb{R}^{3})$. We claim both $\lambda_{1,n}$ and $\lambda_{2,n}$ are
bounded sequence, and at least one of them is converging, up to a subsequence,
to a strictly negative value. Indeed, we can using $(u_{n},0)$ and $(0,v_{n})$
as text function in (4.19), we have

$\int_{\mathbb{R}^{3}}\lambda_{1,n}u^{2}_{n}dx=-\int_{\mathbb{R}^{3}}|\nabla
u_{n}|^{2}dx+\int_{\mathbb{R}^{3}}(\mu_{1}u_{n}^{4}+\rho
v_{n}^{2}u^{2}_{n})dx+\beta\int_{\mathbb{R}^{3}}u^{2}_{n}v_{n}+o(1)\|\varphi\|_{H^{1}(\mathbb{R}^{3})},$
$\int_{\mathbb{R}^{3}}\lambda_{2,n}v^{2}_{n}dx=-\int_{\mathbb{R}^{3}}|\nabla
v_{n}|^{2}dx+\int_{\mathbb{R}^{3}}(\mu_{2}v_{n}^{4}+\rho
v_{n}^{2}u^{2}_{n})dx+\frac{\beta}{2}\int_{\mathbb{R}^{3}}u^{2}_{n}v_{n}+o(1)\|\psi\|_{H^{1}(\mathbb{R}^{3})},$

so

$\displaystyle\int_{\mathbb{R}^{3}}(\lambda_{1,n}u^{2}_{n}+\lambda_{2,n}v^{2}_{n})dx$
$\displaystyle=-\int_{\mathbb{R}^{3}}(|\nabla u_{n}|^{2}+|\nabla
v_{n}|^{2})dx+\int_{\mathbb{R}^{3}}(\mu_{1}u_{n}^{4}+\mu_{2}v_{n}^{4}+2\rho
v_{n}^{2}u^{2}_{n})dx$ (4.20)
$\displaystyle\quad+\frac{3}{2}\beta\int_{\mathbb{R}^{3}}u_{n}^{2}v_{n}dx.$

By (2.1)-(2.3) and the boundedness of $\\{(u_{n},v_{n})\\}$, we can deduce
that $(\lambda_{1,n},\lambda_{2,n})$ is bounded, hence up to a subsequence
$(\lambda_{1,n},\lambda_{2,n})\rightarrow(\lambda_{1},\lambda_{2})\in\mathbb{R}^{2}$,
passing to limits in (4.19), we can deduce that $(u,v)$ is a nonnegative
solutions of (1.1) . Therefore

$\displaystyle\int_{\mathbb{R}^{3}}(|\nabla u|^{2}+|\nabla
v|^{2})dx+\int_{\mathbb{R}^{3}}(\lambda_{1}u^{2}+\lambda_{2}v^{2})dx=\int_{\mathbb{R}^{3}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx+\frac{3}{2}\beta\int_{\mathbb{R}^{3}}u^{2}vdx.$

From (2.1), we can get

$\displaystyle\frac{1}{2}\int_{\mathbb{R}^{3}}(|\nabla u|^{2}+|\nabla
v|^{2})dx+\frac{3}{2}\int_{\mathbb{R}^{3}}(\lambda_{1}u^{2}+\lambda_{2}v^{2})dx=\frac{3}{4}\int_{\mathbb{R}^{3}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx+\frac{3}{2}\beta\int_{\mathbb{R}^{3}}u^{2}vdx.$

Thus we obtain

$\displaystyle\int_{\mathbb{R}^{3}}(\lambda_{1}u^{2}+\lambda_{2}v^{2})dx=\frac{1}{4}\int_{\mathbb{R}^{3}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx+\frac{3}{4}\beta\int_{\mathbb{R}^{3}}u^{2}vdx.$ (4.21)

From $P_{b_{1},b_{2}}(u_{n},v_{n})\rightarrow 0$, we have

$\displaystyle\int_{\mathbb{R}^{3}}(|\nabla u_{n}|^{2}+|\nabla
v_{n}|^{2})dx-\frac{3}{4}\int_{\mathbb{R}^{3}}\left(\mu_{1}u_{n}^{4}+\mu_{2}v_{n}^{4}+2\rho
u_{n}^{2}v_{n}^{2}\right)dx-\frac{3}{4}\beta\int_{\mathbb{R}^{3}}u_{n}^{2}v_{n}dx=o_{n}(1).$
(4.22)

Together (4.20) with (4.22), we can get

$\displaystyle\lambda_{1,n}b^{2}_{1}+\lambda_{2,n}b^{2}_{2}=\frac{1}{4}\int_{\mathbb{R}^{3}}\left(\mu_{1}u_{n}^{4}+\mu_{2}v_{n}^{4}+2\rho
u_{n}^{2}v_{n}^{2}\right)dx+\frac{3}{4}\beta\int_{\mathbb{R}^{3}}u_{n}^{2}v_{n}dx.$
(4.23)

When $\beta>0$, it is easy to see that at least one sequence of
$(\lambda_{i,n})$ is positive and bounded away from 0. Let
$n\rightarrow+\infty$ in (4.23), we have

$\displaystyle\lambda_{1}b^{2}_{1}+\lambda_{2}b^{2}_{2}=\frac{1}{4}\int_{\mathbb{R}^{3}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx+\frac{3}{4}\beta\int_{\mathbb{R}^{3}}u^{2}vdx.$ (4.24)

We claim that if $\lambda_{1}>0$(resp.$\lambda_{2}>0)$, then
$\lambda_{2}>0$(resp.$\lambda_{1}>0$). Indeed, we know that at least one
sequence of $(\lambda_{i})$ is positive and bounded away from 0. If
$\lambda_{2}>0$, now we argue by contradiction and assume that
$\lambda_{1}\leq 0$, then

$-\Delta u=-\lambda_{1}u+\beta uv+\mu_{1}u^{3}+\rho v^{2}u\geq 0.$

Using a Liouville type theorem[[22], Lemma A.2], we can deduce that $u=0$. So,
$v$ satisfies that

$\begin{cases}-\Delta v+\lambda_{2}v=\mu_{2}v^{3},&\text{in}\ \
\mathbb{R}^{3},\\\ v>0,&\text{in}\ \ \mathbb{R}^{3},\\\
\int_{\mathbb{R}^{3}}v^{2}dx=b^{2}_{2},&\text{in}\ \ \mathbb{R}^{3}.\\\
\end{cases}$

Therefore

$\displaystyle
m^{+}_{\beta}(b_{1},b_{2})=\lim_{n\rightarrow+\infty}J_{\beta}(u_{n},v_{n})$
$\displaystyle=\lim_{n\rightarrow+\infty}\left[\frac{1}{8}\int_{\mathbb{R}^{3}}(\mu_{1}u_{n}^{4}+\mu_{2}v_{n}^{4}+2\rho
u_{n}^{2}v_{n}^{2})dx-\frac{\beta}{8}\int_{\mathbb{R}^{3}}u_{n}^{2}v_{n}dx\right]$
$\displaystyle=\frac{\mu_{1}}{8}\int_{\mathbb{R}^{3}}v^{4}dx=m^{+}_{\beta}(0,b_{2}),$

in contradiction with Lemma 4.6. Thus, $\lambda_{1}>0,\ \lambda_{2}>0.$

If $\lambda_{1}>0$, now we argue by contradiction and assume that
$\lambda_{2}\leq 0$, then

$-\Delta v=-\lambda_{2}v\frac{\beta}{2}u^{2}+\mu_{2}v^{3}+\rho u^{2}v\geq 0.$

Using a Liouville type theorem[[22], Lemma A.2], we can deduce that $v=0$. So,
by the structure of system (1.1), we get $u=0$, which is impossible.

It is easy to see that

$\displaystyle
m^{+}_{\beta}(b_{1},b_{2})=\lim_{n\rightarrow+\infty}J_{\beta}(u_{n},v_{n})=\lim_{n\rightarrow+\infty}\left[\frac{1}{6}\int_{\mathbb{R}^{3}}(|\nabla
u_{n}|^{2}+|\nabla
v_{n}|^{2})dx-\frac{\beta}{4}\int_{\mathbb{R}^{3}}u_{n}^{2}v_{n}dx\right].$

Case 1 If $u=0,v=0$, by compact Sobolev embedding,

$\displaystyle
m^{+}_{\beta}(b_{1},b_{2})=\lim_{n\rightarrow+\infty}J_{\beta}(u_{n},v_{n})\geq\lim_{n\rightarrow+\infty}\frac{1}{6}\int_{\mathbb{R}^{3}}(|\nabla
u_{n}|^{2}+|\nabla v_{n}|^{2})dx\geq 0.$

However, from Lemma 4.4, we know that $m^{+}_{\beta}(b_{1},b_{2})<0$, which is
a contraction.

Case 2 If $u\neq 0,v=0$, indeed, if $v=0$, by the structure of system (1.1),
we get $u=0$, so Case 2 doesn’t happen.

Case 3 If $u=0,\ v\neq 0$, we have

$\displaystyle
m^{+}_{\beta}(b_{1},b_{2})=\lim_{n\rightarrow+\infty}J_{\beta}(u_{n},v_{n})\geq\lim_{n\rightarrow+\infty}\frac{1}{6}\int_{\mathbb{R}^{3}}(|\nabla
u_{n}|^{2}+|\nabla v_{n}|^{2})dx\geq 0,$

contradicts with $m^{+}_{\beta}(b_{1},b_{2})<0$.

Case 4 if $u\neq 0,v\neq 0$, let

$\widetilde{u}_{n}=u_{n}-u,\ \widetilde{v}_{n}=v_{n}-v,$

from [17, Lemma 2.4], we get

$\int_{\mathbb{R}^{3}}|\widetilde{u}_{n}|^{2}|\widetilde{v}_{n}|^{2}dx=\int_{\mathbb{R}^{3}}|u_{n}|^{2}|v_{n}|^{2}dx-\int_{\mathbb{R}^{3}}|u|^{2}|v|^{2}dx+o(1).$
(4.25)

By Brezis Lieb Lemma in [47], we can obtain that

$\int_{\mathbb{R}^{3}}|\widetilde{u}_{n}|^{2}\widetilde{v}_{n}dx=\int_{\mathbb{R}^{3}}|u_{n}|^{2}v_{n}dx-\int_{\mathbb{R}^{3}}|u|^{2}vdx+o(1).$
(4.26)

So, from (4.25) and (4.26), we have

$\displaystyle 0$
$\displaystyle=P_{b_{1},b_{2}}(u_{n},v_{n})+o(1)=P_{b_{1},b_{2}}(\widetilde{u}_{n},\widetilde{v}_{n})+P_{b_{1},b_{2}}(u,v)+o(1)$
$\displaystyle=\int_{\mathbb{R}^{3}}(|\nabla\widetilde{u}_{n}|^{2}+|\nabla\widetilde{v}_{n}|^{2})dx-\frac{3}{4}\int_{\mathbb{R}^{3}}\left(\mu_{1}\widetilde{u}_{n}^{4}+\mu_{2}\widetilde{v}_{n}^{4}+2\rho\widetilde{u}_{n}^{2}\widetilde{v}_{n}^{2}\right)dx-\frac{3}{4}\beta\int_{\mathbb{R}^{3}}\widetilde{u}_{n}^{2}\widetilde{v}_{n}dx+o(1).$

It is easy to see that $\int_{\mathbb{R}^{3}}u^{2}\leq b^{2}_{1},\
\int_{\mathbb{R}^{3}}v^{2}\leq b^{2}_{2}.$ From (4.21) and (4.24), we have

$\lambda_{1}\left(b^{2}_{1}-\int_{\mathbb{R}^{3}}u^{2}dx\right)+\lambda_{2}\left(b^{2}_{2}-\int_{\mathbb{R}^{3}}v^{2}dx\right)=0,$

so

$\displaystyle\int_{\mathbb{R}^{3}}u^{2}=b^{2}_{1},\
\int_{\mathbb{R}^{3}}v^{2}=b^{2}_{2}.$ (4.27)

Therefore, we know that

$\displaystyle m^{+}_{\beta}(b_{1},b_{2})$
$\displaystyle=\lim_{n\rightarrow+\infty}J_{\beta}(u_{n},v_{n})=\lim_{n\rightarrow+\infty}J_{\beta}(\widetilde{u}_{n},\widetilde{v}_{n})+J_{\beta}(u,v)$
$\displaystyle=\lim_{n\rightarrow+\infty}\left[\frac{1}{6}\int_{\mathbb{R}^{3}}(|\nabla\widetilde{u}_{n}|^{2}+|\nabla\widetilde{v}_{n}|^{2})dx-\frac{\beta}{4}\int_{\mathbb{R}^{3}}\widetilde{u}_{n}^{2}\widetilde{v}_{n}dx\right]+J_{\beta}(u,v)$
$\displaystyle\geq\lim_{n\rightarrow+\infty}\bigg{[}\frac{1}{6}\int_{\mathbb{R}^{3}}(|\nabla\widetilde{u}_{n}|^{2}+|\nabla\widetilde{v}_{n}|^{2})dx$
$\displaystyle\quad-\frac{|\beta|}{4}\left(\int_{\mathbb{R}^{3}}|\widetilde{u}_{n}|^{3}dx\right)^{\frac{2}{3}}\left(\int_{\mathbb{R}^{3}}|\widetilde{v}_{n}|^{3}dx\right)^{\frac{1}{3}}\bigg{]}+J_{\beta}(u,v)$
$\displaystyle\geq\lim_{n\rightarrow+\infty}\frac{1}{6}\int_{\mathbb{R}^{3}}(|\nabla\widetilde{u}_{n}|^{2}+|\nabla\widetilde{v}_{n}|^{2})dx+J_{\beta}(u,v)\geq
m^{+}_{\beta}(b_{1},b_{2}).$

Thus, $J_{\beta}(u,v)=m^{+}_{\beta}(b_{1},b_{2})\ \text{and}\
(u_{n},v_{n})\rightarrow(u,v)$ in $H^{1}(\mathbb{R}^{3})\times
H^{1}(\mathbb{R}^{3}).$ ∎

###  4.2 Existence of a second critical point of mountain pass type

Next, we prove the existence of second critical point of mountain pass type
for $J_{\beta}(u,v)\mid_{\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}}$.

######  Lemma 4.9.

Suppose that $J_{\beta}(u,v)<m^{+}_{\beta}(b_{1},b_{2})$. Then the value
$t_{u,v}$ defined by Lemma 4.4 is negative.

###### Proof.

From Lemma 4.4, we know that the function $\Psi_{u,v}(t)$ has exactly two
critical points $s_{u,v}<t_{u,v}\in\mathbb{R}$ and two zeros
$c_{u,v}<d_{u,v}\in\mathbb{R}$ with $s_{u,v}<c_{u,v}<t_{u,v}<d_{u,v}$. If
$d_{u,v}\leq 0$, then $t_{u,v}<0.$ Assume by contradiction that $d_{u,v}>0$,
if $0\in(c_{u,v},t_{u,v})$, then $J_{\beta}(u,v)=\Psi_{u,v}(0)>0$ contract
with $J_{\beta}(u,v)<m^{+}_{\beta}(b_{1},b_{2})<0$. Therefore, $c_{u,v}>0.$ By
Lemma 4.4-(2), we know that

$\displaystyle
m^{+}_{\beta}(b_{1},b_{2})>J_{\beta}(u,v)=\Psi_{u,v}(0)\geq\inf_{t\in(-\infty,c_{u,v}]}\Psi_{u,v}(t)\geq
J_{\beta}(u,v)(s_{u,v}\star(u,v))$
$\displaystyle=\min\left\\{J_{\beta}(t\star(u,v):t\in\mathbb{R}\ \text{and}\
(\int_{\mathbb{R}^{3}}(|\nabla(t\star u)|^{2}+|\nabla(t\star
v)|^{2})dx)^{\frac{1}{2}}<R_{0}\right\\}$
$\displaystyle=J_{\beta}(s_{u,v}\star(u,v)\geq m^{+}_{\beta}(b_{1},b_{2}),$

which is a contradiction. ∎

######  Lemma 4.10.

We have

$m^{-}_{\beta}(b_{1},b_{2}):=\inf_{(u,v)\in\mathcal{P}^{-}_{b_{1},b_{2}}}J_{\beta}(u,v)>0.$

###### Proof.

From Lemma 4.2, there exists $0<R_{0}<R_{1}$ such that $h(R_{0})=h(R_{1})=0$,
$h(t)>0$ if and only if $t\in(R_{0},R_{1})$. Let $t_{max}$ be the maximum
point of the function $h(t)$ at the positive level $(R_{0},R_{1})$. For every
$(u,v)\in\mathcal{P}^{-}_{b_{1},b_{2}}$, there exists
$\tau_{u,v}\in\mathbb{R}$ such that

$\big{(}\int_{\mathbb{R}^{3}}(|\nabla(\tau_{u,v}\star
u)|^{2}+|\nabla(\tau_{u,v}\star v)|^{2})dx\big{)}^{\frac{1}{2}}=t_{max},$

and from Lemma 4.4 we know that
$t_{u,v}\star(u,v)\in\mathcal{P}^{-}_{b_{1},b_{2}}$ then $t=t_{u,v},$ which
implies that 0 is the unique strict maximum of $\Psi_{u,v}(t)$. Therefore,

$\displaystyle J_{\beta}(u,v)$
$\displaystyle=\Psi_{u,v}(0)\geq\Psi_{u,v}(\tau_{u,v})=J_{\beta}(\tau_{u,v}\star(u,v))$
$\displaystyle\geq
h\Big{(}\big{(}\int_{\mathbb{R}^{3}}(|\nabla(\tau_{u,v}\star
u)|^{2}+|\nabla(\tau_{u,v}\star
v)|^{2})dx\big{)}^{\frac{1}{2}}\Big{)}=h(t_{max})>0,$

so for any $(u,v)\in\mathcal{P}^{-}_{b_{1},b_{2}}$, we have

$m^{-}_{\beta}(b_{1},b_{2}):=\inf_{(u,v)\in\mathcal{P}^{-}_{b_{1},b_{2}}}J_{\beta}(u,v)>0.$

∎

Next, we use the idea introduced by Li and Zou [28] to give the energy
estimate. We observe that
$w_{\lambda,\mu}=\Big{(}\frac{\|w\|^{2}_{L^{2}(\mathbb{R}^{3})}}{\mu
b^{2}}\Big{)}^{2}\frac{\|w\|_{L^{2}(\mathbb{R}^{3})}^{2}}{\mu^{\frac{3}{2}}b^{2}}w\Big{(}\big{(}\frac{\|w\|^{2}_{L^{2}(\mathbb{R}^{3})}}{\mu
b^{2}}\big{)}x\Big{)}$ is a solution of the problem

$\begin{cases}-\Delta u+\lambda u=\mu u^{3}&\text{in}\ \ \mathbb{R}^{3},\\\
u>0\ \ \ &\text{in}\ \ \mathbb{R}^{3},\\\
\int_{\mathbb{R}^{3}}u^{2}dx=b^{2},\end{cases}$ (4.28)

where $w$ is the unique positive solution of $-\Delta u+u=u^{3}\ \text{in}\ \
\mathbb{R}^{3}$.

######  Lemma 4.11.

For any $b_{1},b_{2}>0,$ $\rho>0,\beta>0$, we have

$m^{-}_{\beta}(b_{1},b_{2})<\min\\{m^{-}_{\beta}(0,b_{2}),m^{-}_{\beta}(b_{1},0)\\}.$

###### Proof.

Let $(v,\lambda_{0})\in\mathrm{T}_{b_{2}}\times\mathbb{R}^{+}$ be the unique
solution of equation (4.28) with parameters $\mu_{2}$ and $b_{2}$. Let
$u(x)=c\frac{\varphi(x)}{|x|^{m}},\ m<\frac{1}{2}$

$\varphi(x)\in C^{\infty}_{0}(B_{2}(0)),\ 0\leq\varphi(x)\leq 1,\
\varphi(x)=1\ \text{in}\ B_{1}(0).$

Then $u\in H^{1}(\mathbb{R}^{3})$ and $u\in\mathrm{T}_{b_{1}}$ for some
suitable c. Therefore, $(t\star
u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}$ for any $t\in\mathbb{R}$.
So,

$\int_{\mathbb{R}^{3}}|t\star
u|^{2}v^{2}dx=C_{0}e^{(3-2m)t}\int_{\mathbb{R}^{3}}\frac{\varphi^{2}(e^{t}(x))}{|x|^{2m}}v^{2}(x)dx,$
$\int_{\mathbb{R}^{3}}|t\star
u|^{2}vdx=C_{0}e^{(3-2m)t}\int_{\mathbb{R}^{3}}\frac{\varphi^{2}(e^{t}(x))}{|x|^{2m}}v(x)dx.$

From the decay properties of [32] that $v$ decays exponentially

$v(x)=O(|x|^{-\frac{1}{2}}e^{-\lambda^{\frac{1}{2}}_{0}|x|})\ \text{and}\
|u(x)|\leq M\ \text{in}\ \mathbb{R}^{3}.$

Thus,

$0<\int_{\mathbb{R}^{3}}\frac{v^{2}(x)}{|x|^{2m}}dx\leq
C\Big{(}\int_{B_{R}(0)}\frac{1}{|x|^{2m}}+\int_{\mathbb{R}^{3}\setminus
B_{R}(0)}|x|^{-1-2m}e^{-\lambda^{\frac{1}{2}}_{0}2|x|}\Big{)}<+\infty,$
$0<\int_{\mathbb{R}^{3}}\frac{v(x)}{|x|^{2m}}dx\leq
C\Big{(}\int_{B_{R}(0)}\frac{1}{|x|^{2m}}+\int_{\mathbb{R}^{3}\setminus
B_{R}(0)}|x|^{-\frac{1}{2}-2m}e^{-\lambda^{\frac{1}{2}}_{0}|x|}\Big{)}<+\infty.$

By the Dominated Convergence Theorem, we obtain that

$\lim_{t\rightarrow-\infty}\int_{\mathbb{R}^{3}}\frac{\varphi^{2}(e^{t}(x))}{|x|^{2m}}v^{2}(x)dx=\int_{\mathbb{R}^{3}}\frac{v^{2}(x)}{|x|^{2m}}dx=C_{1}\in(0,+\infty),$
$\lim_{t\rightarrow-\infty}\int_{\mathbb{R}^{3}}\frac{\varphi^{2}(e^{t}(x))}{|x|^{2m}}v(x)dx=\int_{\mathbb{R}^{3}}\frac{v(x)}{|x|^{2m}}dx=C_{2}\in(0,+\infty),$

so

$\displaystyle\int_{\mathbb{R}^{3}}|t\star
u|^{2}v^{2}dx=C_{0}e^{(3-2m)t}\int_{\mathbb{R}^{3}}\frac{\varphi^{2}(e^{t}(x))}{|x|^{2m}}v^{2}(x)dx$
$\displaystyle=C_{0}e^{(3-2m)t}(C_{1}+o(1))\geq\frac{C_{0}C_{1}}{2}e^{(3-2m)t},$
(4.29)

as $t\rightarrow-\infty$.

$\displaystyle\int_{\mathbb{R}^{3}}|t\star u|^{2}vdx$
$\displaystyle=C_{0}e^{(3-2m)t}\int_{\mathbb{R}^{3}}\frac{\varphi^{2}(e^{t}(x))}{|x|^{2m}}v(x)dx$
(4.30)
$\displaystyle=C_{0}e^{(3-2m)t}(C_{2}+o(1))\geq\frac{C_{0}C_{2}}{2}e^{(3-2m)t}\
\text{as}\ t\rightarrow-\infty.$

From Lemma 4.4, we know that $t_{\ast}=t_{(t\star u,v)}$ is a local maximum of
$\Psi_{u,v}(t)$ and $t_{(t\star u,v)}\star(t\star
u,v)\in\mathcal{P}^{-}_{b_{1},b_{2}}$, so

$\displaystyle 0=P(t_{\star}\star(t\star u,v))$
$\displaystyle=e^{2(t+t_{\star})}\int_{\mathbb{R}^{3}}|\nabla
u|^{2}dx+e^{2t_{\star}}\int_{\mathbb{R}^{3}}|\nabla
v|^{2}dx-\frac{3e^{3(t+t_{\star})}}{4}\mu_{1}\int_{\mathbb{R}^{3}}u^{4}dx$
$\displaystyle\quad-\frac{3e^{3t_{\star}}}{4}\mu_{2}\int_{\mathbb{R}^{3}}v^{4}dx-\frac{3e^{3t_{\star}}}{2}\rho\int_{\mathbb{R}^{3}}|t\star
u|^{2}v^{2}dx-\frac{3\beta}{4}e^{\frac{3t_{\star}}{2}}\int_{\mathbb{R}^{3}}|t\star
u|^{2}vdx.$

From (4.29), (4.30), (4.2) and let $t\rightarrow-\infty$, we have

$e^{2t_{\star}}\int_{\mathbb{R}^{3}}|\nabla
v|^{2}dx=\frac{3e^{3t_{\star}}}{4}\mu_{2}\int_{\mathbb{R}^{3}}v^{4}dx.$

Since $(v,\lambda_{0})\in\mathrm{T}_{b_{2}}\times\mathbb{R}^{+}$ be the unique
solution of equation (4.28) with parameters $\mu_{2}$ and $b_{2}$, by the
Pohozaev identity for equation (4.28) with parameters $\mu_{2}$ and $b_{2}$,
we can get $e^{3t_{\star}}\rightarrow 1$ as $t\rightarrow-\infty.$ Therefore,
from Lemma 2.1 in [28], we have

$e^{2t_{\star}}\int_{\mathbb{R}^{3}}|\nabla
v|^{2}dx-\frac{3e^{3t_{\star}}}{4}\mu_{2}\int_{\mathbb{R}^{3}}v^{4}dx<m_{\beta}(0,b_{2}),$

therefore

$\displaystyle m^{-}_{\beta}(b_{1},b_{2})$ $\displaystyle\leq
J_{\beta}(t_{\star}\star(t\star u,v))$
$\displaystyle=\frac{e^{2(t+t_{\star})}}{2}\int_{\mathbb{R}^{3}}|\nabla
u|^{2}dx+\frac{e^{2t_{\star}}}{2}\int_{\mathbb{R}^{3}}|\nabla
v|^{2}dx-\frac{e^{3(t+t_{\star})}}{4}\mu_{1}\int_{\mathbb{R}^{3}}u^{4}dx$
$\displaystyle\quad-\frac{e^{3t_{\star}}}{4}\mu_{2}\int_{\mathbb{R}^{3}}v^{4}dx-\frac{e^{3t_{\star}}}{2}\rho\int_{\mathbb{R}^{3}}|t\star
u|^{2}v^{2}dx-\frac{\beta}{2}e^{\frac{3t_{\star}}{2}}\int_{\mathbb{R}^{3}}|t\star
u|^{2}vdx$ $\displaystyle=\frac{e^{2t_{\star}}}{2}\int_{\mathbb{R}^{3}}|\nabla
v|^{2}dx-\frac{e^{3t_{\star}}}{4}\mu_{2}\int_{\mathbb{R}^{3}}v^{4}dx-\frac{C_{0}C_{1}}{2}e^{(3-2m)t}-\beta\frac{C_{0}C_{2}}{2}e^{(3-2m)t}$
$\displaystyle<m^{-}_{\beta}(0,b_{2})-\frac{C_{0}C_{1}}{2}e^{(3-2m)t}-\beta\frac{C_{0}C_{2}}{2}e^{(3-2m)t},$

from which, we see that for sufficiently small $t\ll-1$, there holds
$m^{-}_{\beta}(b_{1},b_{2})<m^{-}_{\beta}(0,b_{2}).$ Similarly, we can deduce
that

$m^{-}_{\beta}(b_{1},b_{2})<m^{-}_{\beta}(b_{1},0).$

∎

######  Lemma 4.12.

There is a radial symmetric Palais-Smale sequence of
$J|_{\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}}$ at level
$m^{-}_{\beta}(b_{1},b_{2})$ with the additional properties
$P(u_{n},v_{n})\rightarrow 0$ and $u^{-}_{n},v^{-}_{n}\rightarrow 0$ a.e. in
$\mathbb{R}^{3}.$ Then up to a subsequence $(u_{n},v_{n})\rightarrow(u,v)$ in
$H^{1}(\mathbb{R}^{3})\times H^{1}(\mathbb{R}^{3})$, where $(u,v)$ is a
positive solution of (1.1) for some $\lambda_{1},\lambda_{2}>0.$

###### Proof.

From (4.2), we define the functional $\widetilde{J}:\mathbb{R}\times
H^{1}(\mathbb{R}^{3})\times H^{1}(\mathbb{R}^{3})\rightarrow\mathbb{R}$ as
following

$\displaystyle\widetilde{J}_{\beta}(t,u,v)=J_{\beta}(t\star(u,v))$
$\displaystyle=\frac{e^{2t}}{2}\int_{\mathbb{R}^{3}}(|\nabla u|^{2}+|\nabla
v|^{2})dx-\frac{\beta}{2}\int_{\mathbb{R}^{3}}e^{\frac{3t}{2}}u^{2}vdx$
$\displaystyle\quad-\frac{e^{3t}}{4}\int_{\mathbb{R}^{3}}\left(\mu_{1}u^{4}+\mu_{2}v^{4}+2\rho
u^{2}v^{2}\right)dx.$

It is easy to see that $\widetilde{J}$ is of class $C^{1}$. Denoting
$J_{\beta}^{c}=\\{(u,v)\in\mathrm{T}_{b_{1}}\times\mathrm{T}_{b_{2}}:J_{\beta}(u,v)\leq
c\\}$ and introduce the minimax class

$\displaystyle\Gamma=\big{\\{}\gamma=(\alpha,\varphi_{1},\varphi_{2})\in
C([0,1],\mathbb{R}\times\mathrm{T}_{b_{1},r}\times\mathrm{T}_{b_{2},r}):\gamma(0)\in\\{0\\}\times\mathcal{P}^{+}_{b_{1},b_{2}},\gamma(1)\in\\{0\\}\times
J_{\beta}^{2m(b_{1},b_{2})}\big{\\}},$ (4.31)

with associated minimax level

$\sigma_{\beta}(b_{1},b_{2})=\inf_{\gamma\in\Gamma}\max_{t\in[0,1]}\widetilde{J}_{\beta}(\gamma(t)).$

Let $(u,v)\in\mathrm{T}_{b_{1},r}\times\mathrm{T}_{b_{2},r}$, it is easy to
see

$\int_{\mathbb{R}^{3}}\big{(}|\nabla(t\star u)|^{2}+|\nabla(t\star
v)|^{2}\big{)}dx\rightarrow 0\ \text{as}\ t\rightarrow-\infty,$
$J_{\beta}(t\star u,t\star v)\rightarrow-\infty\ \text{as}\
t\rightarrow+\infty.$

From Lemma 4.4, there exists $t_{0}\ll-1$ and $t_{1}\gg 1$ such that

$t_{0}\star(u,v)\in\mathcal{P}^{+}_{b_{1},b_{2}},\ \ t_{1}\star(u,v)\in
J_{\beta}^{2m_{\beta}(b_{1},b_{2})}$

and

$\gamma_{u,v}(t):t\in[0,1]\mapsto(0,((1-t)t_{0})+tt_{1})\star(u,v))\in\mathbb{R}\times\mathrm{T}_{b_{1},r}\times\mathrm{T}_{b_{2},r}$
(4.32)

is a path in $\Gamma$. Thus, $\sigma(b_{1},b_{2})$ is a real number. Next, we
prove that for every

$\gamma=(\alpha,\varphi_{1},\varphi_{2})\in\Gamma\ \text{there exists}\
t_{\gamma}\in(0,1)\ \text{such that}\
\alpha(t_{\gamma})\star(\varphi_{1}(t_{\gamma}),\varphi_{2}(t_{\gamma}))\in\mathcal{P}^{-}_{b_{1},b_{2}}.$
(4.33)

Indeed,
$\gamma(0)=(0,\varphi_{1}(0),\varphi_{2}(0))\in\\{0\\}\times\mathcal{P}^{+}_{b_{1},b_{2}}$,
by Lemma 4.4 and the fact that $t\star(u,v)\in\mathcal{P}_{b_{1},b_{2}}$ if
and only if $(\Psi_{u,v})^{\prime}(t)=0$, we have

$t_{\alpha(0)\star(\varphi_{1}(0),\varphi_{2}(0))}=t_{(\varphi_{1}(0),\varphi_{2}(0))}>s_{(\varphi_{1}(0),\varphi_{2}(0))}=0.$

Since
$J_{\beta}(\alpha(1)\star(\varphi_{1}(1),\varphi_{2}(1)))=\widetilde{J}_{\beta}(\gamma(1))\leq
2m_{\beta}(b_{1},b_{2})$, from Lemma 4.9, we have that

$t_{(\varphi_{1}(1),\varphi_{2}(1))}<0.$

From Lemma 4.4, the map $t_{\alpha(t)\star(\varphi_{1}(t),\varphi_{2}(t))}$ is
continuous in $t,$ so there exists
$t_{\gamma}\in(0,1)$ such that
$t_{\alpha(t_{\gamma})\star(\varphi_{1}(t_{\gamma}),\varphi_{2}(t_{\gamma}))}=0$,
so $\text{for every }\ \gamma=(\alpha,\varphi_{1},\varphi_{2})\in\Gamma\
\text{there exists}\ t_{\gamma}\in(0,1)\ \text{such that}\
\alpha(t_{\gamma})\star(\varphi_{1}(t_{\gamma}),\varphi_{2}(t_{\gamma}))\in\mathcal{P}^{-}_{b_{1},b_{2}}.$
This implies that

$\max_{\gamma([0,1])}\widetilde{J}_{\beta}\geq\widetilde{J}_{\beta}(\gamma(t_{\gamma}))=J_{\beta}(\alpha(t_{\gamma})\star(\varphi_{1}(t_{\gamma}),\varphi_{2}(t_{\gamma})\geq\inf_{\mathcal{P}^{-}_{b_{1},b_{2}}\cap\mathrm{T}_{b_{1},r}\times\mathrm{T}_{b_{2},r}}J_{\beta}.$

So

$\sigma_{\beta}(b_{1},b_{2})\geq\inf_{\mathcal{P}^{-}_{b_{1},b_{2}}\cap\mathrm{T}_{b_{1},r}\times\mathrm{T}_{b_{2},r}}J_{\beta}.$

On the other hand, if
$(u,v)\in\mathcal{P}^{-}_{b_{1},b_{2}}\cap\mathrm{T}_{b_{1},r}\times\mathrm{T}_{b_{2},r}$,
then $\gamma_{u,v}$ defined in (4.32) is a path in $\Gamma$ with

$J_{\beta}(u,v)=\widetilde{J}_{\beta}(t,u,v)=\max_{\gamma_{u,v}([0,1])}\widetilde{J}_{\beta}\geq\sigma_{\beta}(b_{1},b_{2}),$

so

$\inf_{\mathcal{P}^{-}_{b_{1},b_{2}}\cap\mathrm{T}_{b_{1},r}\times\mathrm{T}_{b_{2},r}}J_{\beta}\geq\sigma_{\beta}(b_{1},b_{2}).$

Thus,

$\inf_{\mathcal{P}^{-}_{b_{1},b_{2}}\cap\mathrm{T}_{b_{1},r}\times\mathrm{T}_{b_{2},r}}J_{\beta}=\sigma_{\beta}(b_{1},b_{2}).$

From Lemma 4.10, we know that

$\displaystyle\sigma_{\beta}(b_{1},b_{2})$
$\displaystyle=\inf_{\mathcal{P}^{-}_{b_{1},b_{2}}\cap\mathrm{T}_{b_{1},r}\times\mathrm{T}_{b_{2},r}}J_{\beta}>0\geq\sup_{\mathcal{P}^{+}_{b_{1},b_{2}}\cup
J_{\beta}^{2m(b_{1},b_{2})}\cap\mathrm{T}_{b_{1},r}\times\mathrm{T}_{b_{2},r}}J_{\beta}$
(4.34)
$\displaystyle=\sup_{\\{0\\}\times\mathcal{P}^{+}_{b_{1},b_{2}}\cup\\{0\\}\times
J^{2m(b_{1},b_{2})}\cap\mathrm{T}_{b_{1},r}\times\mathrm{T}_{b_{2},r}}\widetilde{J}_{\beta}.$

Let $\mathcal{F}=\\{\gamma([0,1]):\gamma\in\Gamma\\}$. By the terminology in
[[15], section 5], which implies that $\\{\gamma([0,1]):\gamma\in\Gamma\\}$ is
a homotopy stable family of compact subset of
$\mathbb{R}\times\mathrm{T}_{b_{1},r}\times\mathrm{T}_{b_{2},r}$ with extended
closed boundary
$\\{0\\}\cup\times\mathcal{P}^{+}_{b_{1},b_{2}}\times\\{0\\}\times
J^{2m(b_{1},b_{2})}_{\beta}$, and the superlevel
set$\\{\widetilde{J}_{\beta}\geq\sigma\\}$ is a dual set for $\mathcal{F}$,
which implies that the assumption in [[15], Theorem 5.2] are satisfied.
Therefore, taking any minimizing sequence
$\\{\gamma_{n}([0,1]),\gamma_{n}=(\alpha_{n},\varphi_{1,n},\varphi_{2,n})\\}$
for $\sigma$ with the property that $\alpha(t)=0,\ \varphi_{1,n}\geq 0,\
\varphi_{2,n}\geq 0$ for every $t\in[0,1]$, there exists a sequence
$(t_{n},u_{n},v_{n})\subset\mathbb{R}\times\mathrm{T}_{b_{1},r}\times\mathrm{T}_{b_{2},r}$
such that
$\widetilde{J}_{\beta}(t_{n},u_{n},v_{n})\rightarrow\sigma(b_{1},b_{2})$ and

$\partial_{t}\widetilde{J}_{\beta}(t_{n},u_{n},v_{n})\rightarrow 0,\
\|\partial_{(u,v)}\widetilde{J}_{\beta}(t_{n},u_{n},v_{n})\|_{T_{u_{n}\mathrm{T}_{b_{1},r}}\times
T_{v_{n}\mathrm{T}_{b_{2},r}}}\rightarrow 0$ (4.35)
$|t_{n}|+dist((u_{n},v_{n}),(\varphi_{1,n}([0,1]),\varphi_{2,n}([0,1])))\rightarrow
0.$ (4.36)

Let
$(\widetilde{u}_{n},\widetilde{v}_{n})=t_{n}\star(u_{n},v_{n})\in\mathrm{T}_{b_{1},r}\times\mathrm{T}_{b_{2},r}$.
By (4.36), we know that $t_{n}$ is bounded and
$\widetilde{u}^{-}_{n},\widetilde{v}^{-}_{n}\rightarrow 0$ a.e. in
$\mathbb{R}^{3}$. From (4.35), we can get

$P(\widetilde{u}_{n},\widetilde{v}_{n})=\partial_{t}\widetilde{J}_{\beta}(t_{n},u_{n},v_{n})\rightarrow
0,$

and

$\displaystyle
J_{\beta}^{\prime}(\widetilde{u}_{n},\widetilde{v}_{n})[\phi,\psi]=\partial_{(u,v)}\widetilde{J}_{\beta}(t_{n},u_{n},v_{n})[(-t_{n})\star(\phi,\psi)]$