# Coexistence of heterogenous predator-prey systems with density-dependent
dispersal

De Tang  De Tang
School of Mathematics (Zhuhai)
Sun Yat-sen University
Zhuhai, 519082, Guangdong,P.R. China<EMAIL_ADDRESS>and  Zhi-an
Wang$\ast$ Zhi-an Wang
Department of Applied Mathematics
Hong Kong Polytechnic University
Hung Hom, Kowloon, Hong Kong, P. R. China<EMAIL_ADDRESS>
###### Abstract.

This paper is concerned with existence, non-existence and uniqueness of
positive (coexistence) steady states to a predator-prey system with density-
dependent dispersal. To overcome the analytical obstacle caused by the cross-
diffusion structure embedded in the density-dependent dispersal, we use a
variable transformation to convert the problem into an elliptic system without
cross-diffusion structure. The transformed system and pre-transformed system
are equivalent in terms of the existence or non-existence of positive
solutions. Then we employ the index theory alongside the method of the
principle eigenvalue to give a nearly complete classification for the
existence and non-existence of positive solutions. Furthermore we show the
uniqueness of positive solutions and characterize the asymptotic profile of
solutions for small or large diffusion rates of species. Our results pinpoint
the positive role of density-dependent dispersal on the population dynamics
for the first time by showing that the density-dependent dispersal is a
beneficial strategy promoting the coexistence of species in the predator-prey
system by increasing the chance of predator’s survival.

###### Key words and phrases:

Predator-prey system, density-dependent dispersal, coexistence, principle
eigenvalue, index theory

###### 2020 Mathematics Subject Classification:

35B09, 25J25, 35J60, 35K57, 35Q92

∗Corresponding author.

##  1\. Introduction

Dispersal, an ecological process involving the movement of individual/multiple
species, is one of the main determinants shaping the structure of ecological
communities and maintaining the biodiversity [22, 23, 13]. The causes and
consequences as well as the selection and evolution of dispersal strategies
have been central questions in ecology extensively investigated in the
literature (cf. [47, 44, 53]). A variety of mathematical models have been
constructed to explore the effects of dispersal strategies on the population
dynamics and to predict their biological consequences (cf. [9, 4, 5, 36, 3,
26, 43]) where most of existing theoretical studies are focused on the random
dispersal. However, biological species will more likely employ the non-random
dispersal strategy to optimize their ecological fitness in changing
environments such as local population size, resource competition, habitat
quality/size, inbreeding avoidance, crowding effect and so on. Among various
possible non-random dispersal strategies, the density-dependent dispersal
(meaning that the dispersal rate of one species depends on the densities of
others) has been a major topic for discussion in the biological literature
(cf. [50, 39, 48, 38]). A prototype of two interacting species models with
density-dependent dispersal generally reads as

$\begin{cases}u_{t}=\mu_{1}\Delta
u+f(x,u,v),&\mbox{in}\;\Omega\times(0,\infty),\\\
v_{t}=\mu_{2}\Delta(d(u)v)+g(x,u,v),&\mbox{in}\;\Omega\times(0,\infty),\\\
\end{cases}$ (1.1)

where $u(x,t)$ and $v(x,t)$ denote the population densities of two interacting
species at location $x\in\Omega$ and at time $t>0$ in a bounded habitat
$\Omega\subset\mathbb{R}^{\color[rgb]{0,0,0}N}(N\geq 2)$.
$\Delta=\sum_{i=1}^{{\color[rgb]{0,0,0}n}}\frac{\partial^{2}}{\partial
x_{i}^{2}}$ is the usual Laplace operator. $\mu_{1}$ and $\mu_{2}$ are
positive constants accounting for the diffusion rates of the two interacting
species. The functions $f$ and $g$ describe the intra-specific and inter-
specific interactions between species in a possibly heterogenous environment.
The term $\Delta(d(u)v)$ entails that the dispersal of species $v$ depends on
the density of species $u$ via the dispersal rate function $d(u)$. The
dispersal strategy of species $v$ is said to be random if $d(u)$ is constant,
while to be density-dependent if $d(u)$ is non-constant. Endowing $f(x,u,v)$
and $g(x,u,v)$ with different forms, model (1.1) may include many well-known
mathematical biology models with density-dependent dispersal, such as the
Keller-Segel model [29, 28] describing chemotaxis, density-suppressed motility
model [19] describing the bacterial strip pattern formation driven by self-
trapping mechanism, and prey-taxis system [27, 51], starvation-driven
diffusion [8, 6]. Moreover model (1.1) with competitive dynamics can be
regarded as a special case of Shigesada-Kawasaki-Teramoto (STK) competition
model originally proposed in [46] (see also [37]). Most of existing
theoretical studies of (1.1) have been focused on the random dispersal for
which a large number of results have been developed on the steady state
problem of (1.1), for example see [11, 10, 17, 18, 34, 33, 41, 52]) for
predator-prey systems and [15, 2, 31, 35, 21]) for competition systems; see
also [3, 9] and references therein. In recent years mathematical models with
density-dependent dispersal have increasingly received attentions. Among other
things, this paper is concerned with the following predator-prey system with
density-dependent dispersal proposed in [27]

$\begin{cases}u_{t}=\epsilon\Delta
u+u(m(x)-u)-vF(u),&\mbox{in}\;\Omega\times\mathbb{R}^{+},\\\
v_{t}=\mu\Delta(d(u)v)+\alpha vF(u)-\theta
v,&\mbox{in}\;\Omega\times\mathbb{R}^{+},\\\ \nabla u\cdot n=\nabla v\cdot
n=0,&\mbox{on}\;\partial\Omega\times\mathbb{R}^{+},\\\ \end{cases}$ (1.2)

where $u$ denotes the prey density and $v$ the predator density; $\epsilon>0$
and $\mu>0$ account for the diffusion rates of the prey and predator,
respectively; $F(u)$ denotes the functional response function and $\alpha>0$
is the conversion rate, $\theta>0$ is the predator’s motility rate. $n$
denotes the outward unit normal vector of $\partial\Omega$ and Neumann
boundary conditions are prescribed to warrant that no individual crosses the
habitat boundary. The system (1.2) is a special form of prey-taxis models
proposed in [27] to describe the non-random foraging behavior of predators,
where the dispersal rate function $d(u)$ satisfies the property
$d^{\prime}(u)<0$ complying with the field observation that the predator will
reduce its random motility in the area of higher density of prey.

When $d(u)$ is constant, model (1.2) becomes the classical diffusive predator-
prey systems extensively studied in the literature, such as the steady-state
problem (cf. [11, 10, 17, 18, 34, 33, 41, 52]) and traveling wave problem (cf.
[24, 14]), just to mention some. In contrast the available results for non-
constant $d(u)$ are much less. The global boundedness of classical solutions
and stability of constant steady states of (1.2) with constant $m(x)$ was
first established in [25]. When $d(u)$ is a special form of piecewise
decreasing function, the existence of non-constant steady state solutions of
(1.2) with non-constant $m(x)$ for large $\epsilon>0$ was obtained in [7]
under certain conditions and effect of predator satisfaction on predator’s
survival was examined. When $F(u)$ is replaced by a Leslie-Gower type
functional response function, the global boundedness of solutions and global
stability of constant positive solutions were recently obtained in [40] for
constant $m(x)$. From application point of view, an important question is how
the density-dependent dispersal rate function $d(u)$ plays roles in the
population dynamics, which, however, has not been explored in the above
mentioned works.

The goal of this paper is to explore the steady state problem of (1.2) with
non-constant $d(u)$ and $m(x)$ to find conditions under which positive
solutions exists, by which we pinpoint the role of density-dependent dispersal
on the species coexistence. This is not only a biologically relevant question
(since coexistence of species is the center question concerned in ecology),
but also an interesting mathematical question due to the inherent cross-
diffusion structure in the model which makes many conventional methods fail to
use. The steady state problem of (1.2) reads as

$\begin{cases}\epsilon\Delta u+u(m(x)-u)-vF(u)=0,&\mbox{in}\;\Omega,\\\
\mu\Delta(d(u)v)+\alpha vF(u)-\theta v=0,&\mbox{in}\;\Omega,\\\ \nabla u\cdot
n=\nabla v\cdot n=0,&\mbox{on}\;\partial\Omega.\end{cases}$ (1.3)

Throughout the paper, we make the following basic assumptions on $m(x),F(u)$
and $d(u)$:

  1. ($H_{1}$)

$m\in C(\bar{\Omega})$, $\int_{\Omega}m(x)\mathrm{d}x>0$ and $m$ is not
constant;

  2. ($H_{2}$)

$F(0)=0$ and $F^{\prime}(u)>0$ for all $u\in[0,\infty)$;

  3. ($H_{3}$)

$d(u)\in C^{2}([0,\infty))$, $d(u)>0$ and $d^{\prime}(u)\leq,\not\equiv 0$ on
$[0,\infty)$.

The assumption ($H_{1}$) indicates that the resource $m(x)$ could be
beneficial or harmful but the total mass of the resource is advantageous. The
assumption ($H_{2}$) gives some basic property of the functional response
function which can be fulfilled by a large class of function like Holling type
I, II and III. The assumption ($H_{3}$) indicates that the random diffusion of
the predator decreases with respect to the prey density, which is a biological
postulation as in [27]. Though system (1.3) has a cross-diffusion structure,
fortunately we can circumvent this obstacle by invoking a change of variable

$w:=d(u)v$ (1.4)

which reformulates (1.3) to the following elliptic problem without cross-
diffusion

$\begin{cases}\epsilon\Delta
u+u(m(x)-u)-\frac{F(u)}{d(u)}w=0,&\mbox{in}\;\Omega,\\\ \mu\Delta
w+\frac{\alpha F(u)-\theta}{d(u)}w=0,&\mbox{in}\;\Omega,\\\ \nabla u\cdot
n=\nabla w\cdot n=0,&\mbox{on}\;\partial\Omega.\end{cases}$ (1.5)

The reformulated problem (1.5) has conventional random diffusions only and
many well-developed methods are potentially applicable. However the reaction
terms in (1.5) become more complicated under the transformation (1.4) and
existing methods and results for the predator-prey systems can not be applied
directly. For example, the function $F(u)$ in (1.3) is monotonic but the
function $\frac{F(u)}{d(u)}$ in the transformed system (1.5) is no longer
monotonic, which makes the analysis more difficult. The main goal of this
paper is to find the existence conditions for the positive solutions of (1.3)
and hence pinpoint the effects of density-dependent dispersal on the
coexistence of species. Noting that the existence/non-existence of positive
solutions of (1.3) is equivalent to that of (1.5) via the transformation
(1.4), in what follows we shall focus on the transformed system (1.5) and
fully exploit its structure alongside the delicate analysis to show that the
parameter regimes of species coexistence is broadened by the density-dependent
dispersal. This implies that the density-dependent dispersal is an
advantageous strategy of increasing the biodiversity in a heterogenous
landscape.

The main results of this paper consist of two parts. The first part is to find
conditions for the existence and non-existence of positive (coexistence)
solutions of (1.5), which are given in Theorem 3.1 by which we are able to
pinpoint the positive role of density-dependent dispersal in promoting the
species coexistence. The second part is to further explore the uniqueness and
asymptotic profiles of positive solutions in some limiting cases of
large/small diffusion rates $\epsilon$ and $\mu$, see Theorem 4.1 (large
$\epsilon$), Theorem 4.2 (large $\mu$) and Theorem 4.3 (small $\mu$). The
uniqueness of solutions is not only a difficult mathematical question for
elliptic problems in general, but also predict the possible long time dynamics
of the system, while the solution profiles describe the spatial distribution
patterns of species. These results can be carried over to the original problem
(1.3) directly via the transform (1.4).

The rest of this paper is organized. In section 2, we shall study the
eigenvalue problem of (1.5) and find the conditions for the
stability/instability of the unique semi-trivial solution. Furthermore we
establish some preliminary results for later use. In section 3, we employ the
topological degree method (index theory) to show that the instability of the
semi-trivial solutions ensures the existence of positive solutions and hence
to establish our main result on the existence/non-existence of positive
solutions of (1.5). In section 4, we prove the uniqueness and characterize the
asymptotical profile of positive solutions of (1.5) for large/small diffusion
rates $\epsilon$ and $\mu$.

##  2\. Stability of semi-trivial solutions

In this section, we study the eigenvalue problem associated with the problem
(1.5) and give some conditions for the stability/instability of the semi-
trivial solution of (1.5). We begin with the following linear eigenvalue
problem

$\begin{cases}\ell\Delta\phi+r(x)\phi=\lambda\phi,&\mbox{in}\;\Omega,\\\
\nabla\phi\cdot n=0,&\mbox{on}\;\partial\Omega,\end{cases}$ (2.1)

where $r\in C({\Omega})$. We denote the principal eigenvalue and eigenfunction
by $\lambda_{1}(\ell,r)$ and $\phi_{1}(\ell,r)$, respectively, where one can
choose $\phi_{1}(\ell,r)>0$ and $\|\phi_{1}(\ell,r)\|_{L^{\infty}}=1$ and
there is no other eigenvalue with a positive eigenfunction [30]. Moreover, by
the variational approach, $\lambda_{1}(\ell,r)$ can be characterized as

$\lambda_{1}(\ell,r)=\sup\limits_{0\neq\phi\in
H^{1}(\Omega)}\frac{\int_{\Omega}(-\ell|\nabla\phi|^{2}+r\phi^{2})\mathrm{d}x}{\int_{\Omega}\phi^{2}\mathrm{d}x}.$
(2.2)

Under the assumption $(H_{1})$, it is straightforward to see (cf. [3]) that
system (1.5) admits a unique semi-trivial solution $(\tilde{u},0)$ for any
$\epsilon>0$ , where $\tilde{u}>0$ satisfies

$\begin{cases}\epsilon\Delta\tilde{u}+\tilde{u}(m(x)-\tilde{u})=0,&\mbox{in}\;\Omega,\\\
\nabla\tilde{u}\cdot n=0,&\mbox{on}\;\partial\Omega.\end{cases}$ (2.3)

The problem (2.3) has been well studied in the literature and there are
wealthy results available (cf. [3, 42]). Below we cite a result that shall be
used later.

######  Proposition 2.1.

([31, Propositon 2.5]) The problem (2.3) has a unique positive solution
$\tilde{u}$ satisfying

  * (i)

$\tilde{u}\to m^{+}=\max\\{m,0\\}$ in $L^{\infty}(\Omega)$ as $\epsilon\to 0$;

  * (ii)

$\tilde{u}\to\frac{1}{|\Omega|}\int_{\Omega}m\mathrm{d}x$ in
$L^{\infty}(\Omega)$ as $\epsilon\to+\infty$.

We also collect some results on the principal eigenvalue and eigenfunction of
(2.1).

######  Lemma 2.1.

If $r\in C(\Omega)$, then the following statements on the principal eigenvalue
$\lambda_{1}(\ell,r)$ and eigenfunction $\phi_{1}(\ell,r)$ of problem (2.1)
are true.

  * (i)

$\lambda_{1}(\ell,r)$ and $\phi_{1}(\ell,r)$ depend smoothly on parameters
$\ell\in(0,+\infty)$ and continuously on $r\in C(\Omega)$.

  * (ii)

If $r$ is constant on $(0,L)$, then $\lambda_{1}(\ell,r)=r$; otherwise, the
principal eigenvalue $\lambda_{1}(\ell,r)$ is strictly decreasing with respect
to $\ell\in(0,+\infty)$ and

$\lim\limits_{\ell\to
0}\lambda_{1}(\ell,r)=\max\limits_{x\in\bar{\Omega}}r(x)\quad\mbox{and\quad$\lim\limits_{\ell\to+\infty}\lambda_{1}(\ell,r)=\frac{1}{|\Omega|}\int_{\Omega}r(x)\mathrm{d}x$.}$
(2.4)

  * (iii)

If $r_{i}\in C(\Omega)$ ($i=1$, $2$) and $r_{1}\geq,\not\equiv r_{2}$ in
$\Omega$, then $\lambda_{1}(\ell,r_{1})>\lambda_{1}(\ell,r_{2})$.

###### Proof.

The proofs of statements (i)-(iii) are quite standard. See, for example, [3,
Page 95 and Page 162] and [31]. ∎

Then, we define the notion of linear stability of a given steady state
$(u,w)$. The eigenvalue problem of the linearized system (1.5) at $(u,w)$,
reads

$\begin{cases}\epsilon\Delta\phi+(m-2u)\phi-\left(\frac{F(u)}{d(u)}\right)^{\prime}w\phi-\frac{F(u)}{d(u)}\psi=\tau\phi,&\mbox{in}\;\Omega,\\\
\mu\Delta\psi+\alpha\left(\frac{F(u)}{d(u)}\right)^{\prime}w\phi+\frac{\theta
d^{\prime}(u)}{d^{2}(u)}w\phi+\frac{\alpha
F(u)-\theta}{d(u)}\psi=\tau\psi,&\mbox{in}\;\Omega,\\\\[5.69054pt]
\nabla\phi\cdot n=\nabla\psi\cdot n=0,&\mbox{on}\;\partial\Omega,\end{cases}$
(2.5)

where ′ denotes the differentiation with respect to $u$, and $(\phi,\psi)$ is
the eigenfunction associated with the eigenvalue $\tau$.

Throughout the paper, the following convention will be adopted.

######  Definition 2.1.

An eigenvalue $\tau_{1}$ of problem (2.5) is called a principal eigenvalue if
$\tau_{1}\in\mathbb{C}$ and for any eigenvalue $\tau$ with $\tau\neq\tau_{1}$,
we have Re $\tau\leq$ Re $\tau_{1}$. If Re $\tau_{1}<0$, then $(u,w)$ is
linearly stable; while if Re $\tau_{1}>0$, then $(u,w)$ is linearly unstable;
we call $(u,w)$ is neutrally stable if Re $\tau_{1}=0$.

We remark here that the principal eigenvalue of problem (2.5) may not be
unique but the real part of $\tau_{1}$ are equal. Following the approach as
that in [31, Lemma 2.9 and Corollary 2.10], we can readily derive the
following result and omit the details for brevity.

######  Lemma 2.2.

For system (1.5), the following results hold.

  * (1)

$(0,0)$ is linearly stable if and only if
$\max\big{\\{}\lambda_{1}(\epsilon,m),\lambda_{1}\big{(}\mu,-\frac{\theta}{d(0)}\big{)}\big{\\}}<0.$

  * (2)

$(\tilde{u},0)$ is linearly stable if and only if
$\max\big{\\{}\lambda_{1}\big{(}\mu,\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u})}\big{)},\lambda_{1}(\epsilon,m-2\tilde{u})\big{\\}}<0.$

Based on Lemma 2.1 and Lemma 2.2, we have the following result.

######  Lemma 2.3.

The trivial solution $(0,0)$ of (1.5) is linearly unstable.

###### Proof.

From Lemma 2.1 and assumption $(H_{1})$, it follows that

$\lambda_{1}(\epsilon,m)>\lim\limits_{\mu\to+\infty}\lambda_{1}(\mu,m)=\frac{\int_{\Omega}m\mathrm{d}x}{|\Omega|}>0,$

which alongside Lemma 2.2 and the fact
$\lambda_{1}\big{(}\mu,-\frac{\theta}{d(0)}\big{)}=-\frac{\theta}{d(0)}<0$
shows that $(0,0)$ is linearly unstable. ∎

Next, we study the stability of semi-trivial solution $(\tilde{u},0)$ to
system (1.5).

######  Lemma 2.4.

$(\tilde{u},0)$ is linearly stable if and only if
$\lambda_{1}\left(\mu,\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u})}\right)<0.$

###### Proof.

From $\lambda_{1}(\epsilon,m-\tilde{u})=0$ and Lemma 2.1 (iii), it follows
that $\lambda_{1}(\epsilon,m-2\tilde{u})<0$. This alongside Lemma 2.2 implies
that $(\tilde{u},0)$ is linearly stable if and only if
$\lambda_{1}\left(\mu,\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u})}\right)<0.$ ∎

In the sequel, in some cases, instead of general density-dependent dispersal
rate function $d(u)$, we shall consider the following specialized forms for
the definiteness

$d(u):=d(u;k)=e^{-ku}\ \text{or}\ (1+u)^{-k},\ \ k\geq 0.$ (2.6)

Subsequent to this, we shall denote

$\theta_{0}=\frac{\alpha}{|\Omega|}\int_{\Omega}F(\tilde{u})dx,\
\theta_{k}=\frac{\alpha\int_{\Omega}\frac{F(\tilde{u})}{d(\tilde{u};k)}\mathrm{d}x}{\int_{\Omega}\frac{1}{d(\tilde{u};k)}\mathrm{d}x}\
\text{for}\ k>0.$ (2.7)

We also denote

$\tilde{u}_{\min}=\min\limits_{x\in\bar{\Omega}}\tilde{u}\ \ \mathrm{and}\ \
\tilde{u}_{\max}=\max\limits_{x\in\bar{\Omega}}\tilde{u}.$

Then it follows from Lemma 2.1 and assumption $(H_{2})$ that $\alpha
F(\tilde{u}_{\min})<\lambda_{1}(\mu,\alpha F(\tilde{u}))<\alpha
F(\tilde{u}_{\max})$.

Then we have the following key results.

######  Lemma 2.5.

There exists some $\tilde{\theta}\in(\alpha F(\tilde{u}_{\min}),\alpha
F(\tilde{u}_{\max}))$ satisfying $\lambda_{1}\left(\mu,\frac{\alpha
F(\tilde{u})-\tilde{\theta}}{d(\tilde{u})}\right)=0$ such that

$(\tilde{u},0)\ \text{is}\ \begin{cases}\mbox{linearly stable}&\ \text{if}\ \
\theta>\tilde{\theta},\\\ \mbox{linearly unstable}&\ \text{if}\ \
0\leq\theta<\tilde{\theta},\end{cases}$ (2.8)

and hence

$(\tilde{u},0)\ \text{is}\ \begin{cases}\mbox{linearly stable}&\ \text{if}\ \
\theta\geq\alpha F(\tilde{u}_{\max}),\\\ \mbox{linearly unstable}&\ \text{if}\
\ 0\leq\theta\leq\alpha F(\tilde{u}_{\min}).\end{cases}$ (2.9)

Moreover, for any $\theta\in(\alpha F(\tilde{u}_{\min}),\alpha
F(\tilde{u}_{\max}))$, if $d(u)=d(u;k)$, where $d(u;k)=e^{-ku}$ or
$(1+u)^{-k}$ with $k\geq 0$, the following results on the linear
stability/instability of $(\tilde{u},0)$ hold true.

  * (i)

Fixing all the parameters except $\mu$, if $\theta\in\left(\alpha
F(\tilde{u}_{\min}),\theta_{k}\right]$, then $(\tilde{u},0)$ is linearly
unstable for any $\mu>0$; while if $\theta\in\left(\theta_{k},\alpha
F(\tilde{u}_{\max})\right)$, then there exits some $\mu^{*}>0$ (depending on
$k$ and $\theta$) satisfying $\lambda_{1}\left(\mu^{*},\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u};k)}\right)=0$ such that

$(\tilde{u},0)\ \text{is}\ \begin{cases}\mbox{linearly stable}&\ \text{if}\ \
{\mu}>\mu^{*},\\\ \mbox{linearly unstable}&\ \text{if}\ \
0<{\mu}<\mu^{*}.\end{cases}$

  * (ii)

Fix all the parameters except ${\mu}$ and $k$. We have the following
statements.

    * (ii.1)

If $\theta\in\left(\alpha F(\tilde{u}_{\min}),\theta_{0}\right]$, then
$(\tilde{u},0)$ is linearly unstable for any ${\mu}>0$ and $k\geq 0$.

    * (ii.2)

If $\theta\in\left[\theta_{0},\alpha F(\tilde{u}_{\max})\right)$, there exists
$k^{*}(\theta)>0$ satisfying

$\mathrm{sgn}\bigg{(}\int_{\Omega}\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u};k)}\mathrm{d}x\bigg{)}=\mathrm{sgn}(k-k^{*})$
(2.10)

such that $(\tilde{u},0)$ is linearly unstable for any ${\mu}>0$ provided that
$k\geq k^{*}$. Moreover, there exists some $\tilde{\mu}$ such that
$(\tilde{u},0)$ is linearly unstable for any $k\in[0,k^{*})$ and
${\mu}\in(0,\tilde{\mu})$.

###### Proof.

From Lemma 2.1 (iii) and $d(\tilde{u})>0$ in $\Omega$, it follows that
$\lambda_{1}\left(\mu,\frac{\alpha F(\tilde{u})-\theta}{d(\tilde{u})}\right)$
is strictly decreasing with respect to $\theta$. Therefore, it suffices to
consider the values of $\lambda_{1}\Big{(}\mu,\frac{\alpha
F(\tilde{u})}{d(\tilde{u})}\Big{)}$ and
$\lim\limits_{\theta\to+\infty}\lambda_{1}\left(\mu,\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u})}\right)$. Based on the variational formula
(2.2), one has

$\lim\limits_{\theta\to+\infty}\lambda_{1}\left(\mu,\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u})}\right)=-\infty,$

and

$\displaystyle\lambda_{1}\left(\mu,\frac{\alpha
F(\tilde{u})}{d(\tilde{u})}\right)$ $\displaystyle=\sup\limits_{0\neq\phi\in
H^{1}(\Omega)}\frac{\int_{\Omega}\left(-\mu|\nabla\phi|^{2}+\frac{\alpha
F(\tilde{u})}{d(\tilde{u})}\phi^{2}\right)\mathrm{d}x}{\int_{\Omega}\phi^{2}\mathrm{d}x}\geq\frac{\int_{\Omega}\frac{\alpha
F(\tilde{u})}{d(\tilde{u})}\mathrm{d}x}{|\Omega|}>0,$

which suggests that there exists some $\tilde{\theta}\in(0,+\infty)$ such that
$\lambda_{1}\left(\mu,\frac{\alpha
F(\tilde{u})-\tilde{\theta}}{d(\tilde{u})}\right)=0$. To prove that
$\tilde{\theta}\in(\alpha F(\tilde{u}_{\min}),\alpha F(\tilde{u}_{\max}))$, it
suffices to show that

$\lambda_{1}\left(\mu,\frac{\alpha F(\tilde{u})-\alpha
F(\tilde{u}_{\min})}{d(\tilde{u})}\right)>0\quad\mbox{and}\quad\lambda_{1}\left(\mu,\frac{\alpha
F(\tilde{u})-\alpha F(\tilde{u}_{\max})}{d(\tilde{u})}\right)<0.$ (2.11)

Recalling the assumption $(H_{2})$ and the fact $\tilde{u}$ is not a constant
function in $\Omega$, one obtains

$\alpha F(\tilde{u})-\alpha F(\tilde{u}_{\min})\geq,\not\equiv
0\quad\mbox{and}\quad\alpha F(\tilde{u})-\alpha
F(\tilde{u}_{\max})\leq,\not\equiv 0,$

which combined with Lemma 2.1 (iii) implies that (2.11) holds. Therefore,
(2.8) and (2.9) hold.

Next, we consider the case $\theta\in(\alpha F(\tilde{u}_{\min}),\alpha
F(\tilde{u}_{\max}))$. Since the proofs are similar, we only consider the case
$d(u)=e^{-ku}$. By Lemma 2.1 (ii), one obtains

$\displaystyle\lim\limits_{{\mu}\to 0}\lambda_{1}({\mu},(\alpha
F(\tilde{u})-\theta)e^{k\tilde{u}})=\max\limits_{x\in\bar{\Omega}}(\alpha
F(\tilde{u})-\theta)e^{k\tilde{u}})>0.$ (2.12)

To proceed, we recall the notation $\theta_{k}=\frac{\int_{\Omega}\alpha
F(\tilde{u})e^{k\tilde{u}}\mathrm{d}x}{\int_{\Omega}e^{k\tilde{u}}\mathrm{d}x}$.
Then clearly $\alpha F(\tilde{u}_{\min})<\theta_{k}<\alpha
F(\tilde{u}_{\max})$ for any $k\geq 0$. Using (2.4), one has

$\displaystyle\lim\limits_{{\mu}\to+\infty}\lambda_{1}({\mu},(\alpha
F(\tilde{u})-\theta)e^{k\tilde{u}})=\frac{\int_{\Omega}(\alpha
F(\tilde{u})-\theta)e^{k\tilde{u}}\mathrm{d}x}{|\Omega|}\begin{cases}\geq
0,&\mbox{if}\;\theta\leq\theta_{k},\\\
<0,&\mbox{if}\;\theta>\theta_{k}.\end{cases}$ (2.13)

From Lemma 2.1 (i)-(ii), it follows that the principal eigenvalue
$\lambda_{1}\left({\mu},(\alpha F(\tilde{u})-\theta)e^{k\tilde{u}}\right)$
smoothly depends on ${\mu}$ and is strictly decreasing with respect to
${\mu}\in(0,+\infty)$. Therefore from (2.12)-(2.13), when
$\theta\leq\theta_{k}$, we have $\lambda_{1}\left(\mu,(\alpha
F(\tilde{u})-\theta)e^{k\tilde{u}}\right)>0$ for any ${\mu}>0$, which
indicates that $(\tilde{u},0)$ is linearly unstable from Lemma 2.4. As
$\theta>\theta_{k}$, from (2.12)-(2.13), we find a constant $\mu^{*}>0$ such
that $\lambda_{1}\left(\mu^{*},\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u};k)}\right)=0$ and
$\lambda_{1}\left(\mu,(\alpha F(\tilde{u})-\theta)e^{k\tilde{u}}\right)>0$ if
$0<\mu<\mu^{*}$ while $\lambda_{1}\left(\mu,(\alpha
F(\tilde{u})-\theta)e^{k\tilde{u}}\right)<0$ if $\mu>\mu^{*}$. This completes
the proof of statement (i) by using Lemma 2.4.

Next we prove the results in statement (ii). To this end, we first prove a
claim as below.

Claim 1: $\int_{\Omega}(\alpha F(\tilde{u})-\rho)e^{k\tilde{u}}\mathrm{d}x$ is
strictly increasing with respect to $k\in[0,+\infty)$ provided
$\rho\leq\theta_{0}=:\frac{\int_{\Omega}\alpha F(\tilde{u})dx}{|\Omega|}$. To
prove this, for any $\rho\leq\rho_{0}$, we define
$\mathfrak{f}(k)=\int_{\Omega}(\alpha
F(\tilde{u})-\rho)e^{k\tilde{u}}\mathrm{d}x$. Direct computations show that

$\displaystyle\mathfrak{f}^{\prime}(k)$
$\displaystyle=\int_{\Omega}\tilde{u}(\alpha
F(\tilde{u})-\rho)e^{k\tilde{u}}\mathrm{d}x$ (2.14)
$\displaystyle=\int_{\\{x\in\Omega|\tilde{u}(x)\geq
F^{-1}\left(\frac{\rho}{\alpha}\right)\\}}\tilde{u}(\alpha
F(\tilde{u})-\rho)e^{k\tilde{u}}\mathrm{d}x+\int_{\\{x\in\Omega|\tilde{u}(x)<F^{-1}\left(\frac{\rho}{\alpha}\right)\\}}\tilde{u}(\alpha
F(\tilde{u})-\rho)e^{k\tilde{u}}\mathrm{d}x$
$\displaystyle>\int_{\\{x\in\Omega|\tilde{u}(x)\geq
F^{-1}\left(\frac{\rho}{\alpha}\right)\\}}F^{-1}\left(\frac{\rho}{\alpha}\right)(\alpha
F(\tilde{u})-\rho)e^{kF^{-1}\left(\frac{\rho}{\alpha}\right)}\mathrm{d}x$
$\displaystyle\quad\quad+\int_{\\{x\in\Omega|\tilde{u}(x)<F^{-1}\left(\frac{\rho}{\alpha}\right)\\}}F^{-1}\left(\frac{\rho}{\alpha}\right)(\alpha
F(\tilde{u})-\rho)e^{kF^{-1}\left(\frac{\rho}{\alpha}\right)}\mathrm{d}x$
$\displaystyle=\int_{\Omega}F^{-1}\left(\frac{\rho}{\alpha}\right)(\alpha
F(\tilde{u})-\rho)e^{kF^{-1}\left(\frac{\rho}{\alpha}\right)}\mathrm{d}x\geq
0,$

where we have used the assumption $(H_{2})$. Therefore, Claim 1 holds.

If $\theta\in\left(\alpha F(\tilde{u}_{\min}),\theta_{0}\right]$, using claim
1, then we have

$\displaystyle\int_{\Omega}(\alpha
F(\tilde{u})-\theta)e^{k\tilde{u}}\mathrm{d}x$
$\displaystyle\geq\int_{\Omega}\left(\alpha
F(\tilde{u})-\theta_{0}\right)e^{k\tilde{u}}\mathrm{d}x\geq\int_{\Omega}\left(\alpha
F(\tilde{u})-\theta_{0}\right)\mathrm{d}x=0,$

which together with Lemma 2.1 (ii) and Lemma 2.4 implies that $(\tilde{u},0)$
is linearly unstable for any ${\mu}>0$ and $k\geq 0$. This proves the results
in (ii.1).

If $\theta\in\left(\theta_{0},\alpha F(\tilde{u}_{\max})\right)$, we define
$\mathcal{F}(k):=\int_{\Omega}(\alpha
F(\tilde{u})-\theta)e^{k\tilde{u}}\mathrm{d}x$ as a function of the parameter
$k$. Similar to the arguments as those in [49, Lemma 2.6], one can derive that

$\lim\limits_{k\to+\infty}\frac{\int_{\Omega}e^{k\tilde{u}}F(\tilde{u})\mathrm{d}x}{\int_{\Omega}e^{k\tilde{u}}\mathrm{d}x}=\lim\limits_{k\to+\infty}\frac{\int_{\Omega}e^{k(\tilde{u}-\tilde{u}_{\max})}F(\tilde{u})\mathrm{d}x}{\int_{\Omega}e^{k(\tilde{u}-\tilde{u}_{\max})}\mathrm{d}x}=F(\tilde{u}_{\max}),$
(2.15)

which along with the assumption $\theta\in\left(\theta_{0},\alpha
F(\tilde{u}_{\max})\right)$ gives that

$\mathcal{F}(0)<0\quad\mbox{and}\quad\mathcal{F}(\infty)>0.$

This together with the continuity of $\mathcal{F}(\cdot)$ yields a constant
$k^{*}>0$ such that $\mathcal{F}(k^{*})=\int_{\Omega}(\alpha
F(\tilde{u})-\theta)e^{k^{*}\tilde{u}}\mathrm{d}x=0$. Next we prove that the
positive root of $\mathcal{F}(k)=0$ is unique, for which it suffices to show
the following claim.

Claim 2: For any $k_{0}>0$ satisfying $\mathcal{F}(k_{0})=0$, then
$\mathcal{F}^{\prime}(k_{0})>0$. Indeed similar to (2.14), one can deduce that

$\displaystyle\mathcal{F}^{\prime}({k_{0}})$
$\displaystyle=\int_{\Omega}\tilde{u}(\alpha
F(\tilde{u})-\theta)e^{{k_{0}}\tilde{u}}\mathrm{d}x$
$\displaystyle=\int_{\\{x\in\Omega|\tilde{u}(x)\geq
F^{-1}(\theta/\alpha)\\}}\tilde{u}(\alpha
F(\tilde{u})-\theta)e^{{k_{0}}\tilde{u}}\mathrm{d}x+\int_{\\{x\in\Omega|\tilde{u}(x)<F^{-1}(\theta/\alpha)\\}}\tilde{u}(\alpha
F(\tilde{u})-\theta)e^{{k_{0}}\tilde{u}}\mathrm{d}x$
$\displaystyle>\int_{\\{x\in\Omega|\tilde{u}(x)\geq
F^{-1}(\theta/\alpha)\\}}F^{-1}(\theta/\alpha)(\alpha
F(\tilde{u})-\theta)e^{{k_{0}}\tilde{u}}\mathrm{d}x$
$\displaystyle\quad\quad+\int_{\\{x\in\Omega|\tilde{u}(x)<F^{-1}(\theta/\alpha)\\}}F^{-1}(\theta/\alpha)(\alpha
F(\tilde{u})-\theta)e^{{k_{0}}\tilde{u}}\mathrm{d}x$
$\displaystyle=\int_{\Omega}F^{-1}(\theta/\alpha)(\alpha
F(\tilde{u})-\theta)e^{{k_{0}}\tilde{u}}\mathrm{d}x$
$\displaystyle=F^{-1}(\theta/\alpha)\mathcal{F}(k_{0})=0.$

Therefore, $k^{*}$ is the unique positive root of $\mathcal{F}(k)=0$ and hence
(2.10) holds. Combining the facts in (2.10), Lemma 2.1 and Lemma 2.4, one
concludes that $(\tilde{u},0)$ is linearly unstable for any ${\mu}>0$ and
$k\geq k^{*}$. This shows the first part (ii.1) of assertion(ii). From the
results in statement (i), one has that

$\tilde{\mu}=\inf\limits_{k\in[0,k^{*})}\mu^{*}(k)>0,$

which directly implies the second part (ii.2) of statement (ii) by the same
argument as in the proof of statement (i). This complete the proof. ∎

######  Remark 2.1.

Fixing all the parameters except $\mu$ and $\theta$, for any
$\theta\in\left(\theta_{k},\alpha F(\tilde{u}_{\max})\right)$, Lemma 2.5 (i)
ensures a number $\mu^{*}(\theta)>0$ such that
$\lambda_{1}\left(\mu^{*}(\theta),\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u};k)}\right)=0$. We will show that
$\mu^{*}(\theta)$ is a convex function with respect to
$\theta\in\left(\theta_{k},\alpha F(\tilde{u}_{\max})\right)$ by proving that

$\rho\mu^{*}(\theta_{1})+(1-\rho)\mu^{*}(\theta_{2})>\mu^{*}(\rho\theta_{1}+(1-\rho)\theta_{2}),$
(2.16)

for any $\rho\in(0,1)$ and $\theta_{i}\in\left(\theta_{k},\alpha
F(\tilde{u}_{\max})\right)$ $(i=1,2)$. From (2.2),
$\lambda_{1}\left(\mu^{*}(\theta_{1}),\frac{\alpha
F(\tilde{u})-\theta_{1}}{d(\tilde{u};k)}\right)=\lambda_{1}\left(\mu^{*}(\theta_{2}),\frac{\alpha
F(\tilde{u})-\theta_{2}}{d(\tilde{u};k)}\right)=0$, and
$\phi_{1}\left(\mu^{*}(\theta_{1}),\frac{\alpha
F(\tilde{u})-\theta_{1}}{d(\tilde{u};k)}\right)\neq\phi_{1}\left(\mu^{*}(\theta_{2}),\frac{\alpha
F(\tilde{u})-\theta_{2}}{d(\tilde{u};k)}\right)$, it follows that

$\displaystyle 0$
$\displaystyle=\lambda_{1}\left(\mu^{*}(\rho\theta_{1}+(1-\rho)\theta_{2}),\frac{\alpha
F(\tilde{u})-(\rho\theta_{1}+(1-\rho)\theta_{2})}{d(\tilde{u};k)}\right)$
$\displaystyle=\sup\limits_{0\neq\phi\in
H^{1}(\Omega)}\frac{\int_{\Omega}\left(-\mu^{*}(\rho\theta_{1}+(1-\rho)\theta_{2})|\nabla\phi|^{2}+\frac{\alpha
F(\tilde{u})-(\rho\theta_{1}+(1-\rho)\theta_{2})}{d(\tilde{u};k)}\phi^{2}\right)\mathrm{d}x}{\int_{\Omega}\phi^{2}\mathrm{d}x}$
$\displaystyle<\sup\limits_{0\neq\phi\in
H^{1}(\Omega)}\frac{\int_{\Omega}\rho\left(-\mu^{*}(\theta_{1})|\nabla\phi|^{2}+\frac{\alpha
F(\tilde{u})-\theta_{1}}{d(\tilde{u};k)}\right)\mathrm{d}x}{\int_{\Omega}\phi^{2}\mathrm{d}x}$
$\displaystyle\ \ \ +\sup\limits_{0\neq\phi\in
H^{1}(\Omega)}\frac{\int_{\Omega}(1-\rho)\left(-\mu^{*}(\theta_{2})|\nabla\phi|^{2}+\frac{\alpha
F(\tilde{u})-\theta_{2}}{d(\tilde{u};k)}\right)\mathrm{d}x}{\int_{\Omega}\phi^{2}\mathrm{d}x}$
$\displaystyle\ \ \ +\sup\limits_{0\neq\phi\in
H^{1}(\Omega)}\frac{\int_{\Omega}(\rho\mu^{*}(\theta_{1})+(1-\rho)\mu^{*}(\theta_{2})-\mu^{*}(\rho\theta_{1}+(1-\rho)\theta_{2}))|\nabla\phi|^{2}\mathrm{d}x}{\int_{\Omega}\phi^{2}\mathrm{d}x}$
$\displaystyle=(\rho\mu^{*}(\theta_{1})+(1-\rho)\mu^{*}(\theta_{2})-\mu^{*}(\rho\theta_{1}+(1-\rho)\theta_{2}))\sup\limits_{0\neq\phi\in
H^{1}(\Omega)}\frac{\int_{\Omega}|\nabla\phi|^{2}\mathrm{d}x}{\int_{\Omega}\phi^{2}\mathrm{d}x},$

which implies that (2.16) holds.

To proceed, we present a generalized result of [12, Lemma 26] below, which can
be proved readily by the mathematical induction.

######  Proposition 2.2.

Suppose there are three sequences of nonnegative real numbers such that $0\leq
a_{1}\leq a_{2}\leq\cdots\leq a_{n}$, $0\leq b_{1}\leq b_{2}\leq\cdots\leq
b_{n}$ and $0\leq c_{1}\leq c_{2}\leq\cdots\leq c_{n}$. Then

$\left(\sum_{j=1}^{n}a_{j}b_{j}c_{j}\right)\left(\sum_{j=1}^{n}c_{j}\right)\geq\left(\sum_{j=1}^{n}a_{j}c_{j}\right)\left(\sum_{j=1}^{n}b_{j}c_{j}\right)$
(2.17)

where “=” holds if and only if $a_{1}=a_{n}$ or $b_{1}=b_{n}$ or
$c_{1}=c_{n-1}=0$.

###### Proof.

We use induction. If $n=1$, then (2.17) holds. Now we assume that (2.17) holds
when $n=i$, we need to show that it holds for $n=i+1$. Direct computations
give

$\displaystyle(a_{1}b_{1}c_{1}+a_{2}b_{2}c_{2}+\cdots+a_{i}b_{i}c_{i}+a_{i+1}b_{i+1}c_{i+1})(c_{1}+c_{2}+\cdots+c_{i}+c_{i+1})$
$\displaystyle\textstyle\quad=\left(\sum_{j=1}^{i}a_{j}b_{j}c_{j}\right)\left(\sum_{j=1}^{i}c_{j}\right)+a_{i+1}b_{i+1}c_{i+1}^{2}+c_{i+1}\left(\sum_{j=1}^{i}a_{j}b_{j}c_{j}\right)+a_{i+1}b_{i+1}c_{i+1}\left(\sum_{j=1}^{i}c_{j}\right)$
$\displaystyle\textstyle\quad\geq\left(\sum_{j=1}^{i}a_{j}c_{j}\right)\left(\sum_{j=1}^{i}b_{j}c_{j}\right)+a_{i+1}b_{i+1}c_{i+1}^{2}+\left(\sum_{j=1}^{i}c_{j}c_{i+1}(a_{j}b_{j}+a_{i+1}b_{i+1})\right)$
$\displaystyle\textstyle\quad=\left(\sum_{j=1}^{i}a_{j}c_{j}\right)\left(\sum_{j=1}^{i}b_{j}c_{j}\right)+a_{i+1}b_{i+1}c_{i+1}^{2}+\left(\sum_{j=1}^{i}c_{j}c_{i+1}(a_{j}b_{i+1}+b_{j}a_{i+1})\right)$
$\displaystyle\textstyle\quad\quad+\left(\sum_{j=1}^{i}c_{j}c_{i+1}(a_{j}b_{j}+a_{i+1}b_{i+1}-a_{j}b_{i+1}-b_{j}a_{i+1})\right)$
$\displaystyle\textstyle\quad=\left(\sum_{j=1}^{i}a_{j}c_{j}\right)\left(\sum_{j=1}^{i}b_{j}c_{j}\right)+a_{i+1}b_{i+1}c_{i+1}^{2}+b_{i+1}c_{i+1}\left(\sum_{j=1}^{i}a_{j}c_{j}\right)+a_{i+1}c_{i+1}\left(\sum_{j=1}^{i}b_{j}c_{j}\right)$
$\displaystyle\textstyle\quad\quad+\left(\sum_{j=1}^{i}c_{j}c_{i+1}(b_{j}-b_{i+1})(a_{j}-a_{i+1})\right)$
$\displaystyle\textstyle\quad\geq\left(\sum_{j=1}^{i+1}a_{j}c_{j}\right)\left(\sum_{j=1}^{i+1}b_{j}c_{j}\right).$

where the “=” in the last inequality holds if and only if $a_{j}=a_{i+1}$ or
$b_{j}=b_{i+1}$ or $c_{j}=0$ for all $j=1,2,\cdots,i$. This along with the
fact that $a_{j},b_{i},c_{j}$ are non-decreasing with respect to $j$ completes
the proof. ∎

We remark that the results in Proposition 2.2 can be considered as a
generalization of [12, Lemma 26] where $c_{i}=1$ $(i=1,2,\cdots,n)$.

######  Lemma 2.6.

Let $d(u)=:d(u;k)$ be given in (2.6). Fix all the parameters except $k$ and
define
$\tilde{F}(k):=\frac{\int_{\Omega}\frac{F(\tilde{u})}{d(\tilde{u};k)}\mathrm{d}x}{\int_{\Omega}\frac{1}{d(\tilde{u};k)}\mathrm{d}x}$.
Then $\tilde{F}(k)$ is strictly increasing with respect to $k\in[0,+\infty)$.

###### Proof.

We first consider the case $d(u;k)=e^{-ku}$ for which one has

$\tilde{F}^{\prime}(k)=\frac{\int_{\Omega}e^{k\tilde{u}}\mathrm{d}x\int_{\Omega}F(\tilde{u})\tilde{u}e^{k\tilde{u}}\mathrm{d}x-\int_{\Omega}\tilde{u}e^{k\tilde{u}}\mathrm{d}x\int_{\Omega}F(\tilde{u})e^{k\tilde{u}}\mathrm{d}x}{(\int_{\Omega}e^{k\tilde{u}}\mathrm{d}x)^{2}}.$

Next we shall approximate the integrals by their Riemann sums with

$a_{i}=F(\tilde{u}(x_{i})),\;\;b_{i}=\tilde{u}(x_{i}),\;\;\mbox{and}\;\;c_{i}=e^{k\tilde{u}(x_{i})},\quad
i=1,2,\cdots,n.$

Since we can rearrange the terms in the Riemann sums in the order that $b_{i}$
is ascending (then $a_{i}$ and $c_{i}$ are automatically ascending by the
assumption $(H_{2})$), by (2.17), one obtains

$\left(\frac{1}{n}\sum_{i=1}^{n}F(\tilde{u}(x_{i}))\tilde{u}(x_{i})e^{k\tilde{u}(x_{i})}\right)\left(\frac{1}{n}\sum_{i=1}^{n}e^{k\tilde{u}(x_{i})}\right)>\left(\frac{1}{n}\sum_{i=1}^{n}F(\tilde{u}(x_{i}))e^{k\tilde{u}(x_{i})}\right)\left(\frac{1}{n}\sum_{i=1}^{n}\tilde{u}(x_{i})e^{k\tilde{u}(x_{i})}\right)$

where the strict inequality results from the fact that $\tilde{u}$ is not a
constant function in $\Omega$ (cf. Proposition 2.1). Thus, one has

$\tilde{F}^{\prime}(k)>0,\quad\mbox{for any}\;k\geq 0.$

On the other hand, if $d(u;k)=(1+u)^{-k}$, then we have

$\tilde{F}^{\prime}(k)=\frac{\int_{\Omega}(1+\tilde{u})^{k}\mathrm{d}x\int_{\Omega}F(\tilde{u})(1+\tilde{u})^{k}\ln(1+\tilde{u})\mathrm{d}x-\int_{\Omega}(1+\tilde{u})^{k}\ln(1+\tilde{u})\mathrm{d}x\int_{\Omega}F(\tilde{u})(1+\tilde{u})^{k}\mathrm{d}x}{(\int_{\Omega}(1+\tilde{u})^{k}\mathrm{d}x)^{2}}.$

Let

$a_{i}=F(\tilde{u}(x_{i})),\;\;b_{i}=\ln(1+\tilde{u}(x_{i})),\;\;\mbox{and}\;\;c_{i}=(1+\tilde{u}(x_{i}))^{k},\quad
i=1,2,\cdots,n.$

Similarly, one can show that $\tilde{F}^{\prime}(k)>0$ for any $k\geq 0$,
which completes the proof. ∎

By Lemma 2.4, the linear stability of the semi-trivial steady state
$(\tilde{u},0)$ is determined by the sign of
$\textstyle\lambda_{1}\left({\mu},\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u};k)}\right)$. Then, it is natural to study the
level set

$\mathcal{S}_{0}:=\left\\{({\mu},k)|\lambda_{1}\left({\mu},\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u};k)}\right)=0\right\\}.$

Fixing all the parameters except ${\mu}$ and $k$, if
$\theta\in\left[\theta_{0},\alpha F(\tilde{u}_{\max})\right)$, for any
$k\in[0,k^{*})$, from Lemma 2.5 (ii), it follows that there exists unique
$\mu^{*}(k)>0$ such that $\textstyle\lambda_{1}\left(\mu^{*}(k),\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u};k)}\right)=0$. Next, we investigate the
property of $\mu^{*}(k)$ by varying $k$ from $0$ to $k^{*}$, that is, to
characterize the level set $\mathcal{S}_{0}$.

######  Lemma 2.7.

Let $d(u)=d(u;k)$ be given by (2.6) and all the parameters except for ${\mu}$
and $k$ fixed. Let $k^{*}$ and $\tilde{\mu}$ be as defined in Lemma
2.5-(ii.2). Assume $\theta\in\left(\theta_{0},\alpha
F(\tilde{u}_{\max})\right)$. For any $k_{0}\in[0,k^{*})$, we have

$\frac{\partial\lambda_{1}\left({\mu},\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u};k)}\right)}{\partial
k}=\frac{-\int_{\Omega}\frac{\alpha
F(\tilde{u})-\theta}{d^{2}(\tilde{u};k)}\cdot\frac{\partial
d(\tilde{u};k)}{\partial
k}\phi_{1}^{2}\mathrm{d}x}{\int_{\Omega}\phi_{1}^{2}\mathrm{d}x},$ (2.18)

where $\phi_{1}=\phi_{1}\left({\mu},\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u};k)}\right)$. In particular,

$\frac{\partial\lambda_{1}\left({\mu},(\alpha
F(\tilde{u})-\theta)e^{k\tilde{u}}\right)}{\partial
k}\Big{|}_{({\mu},k)=(\mu^{*}(k_{0}),k_{0})}=\mu^{*}(k_{0})\frac{\int_{\Omega}\left[\tilde{u}|\nabla\phi|^{2}+\phi\nabla\phi\cdot\nabla\tilde{u}\right]\mathrm{d}x}{\int_{\Omega}\phi^{2}\mathrm{d}x},$
(2.19)

where $\phi=\phi_{1}(\mu^{*}(k_{0}),(\alpha
F(\tilde{u})-\theta)e^{k_{0}\tilde{u}})$, and

$\frac{\partial\lambda_{1}\left({\mu},(\alpha
F(\tilde{u})-\theta)(1+\tilde{u})^{k}\right)}{\partial
k}\Big{|}_{({\mu},k)=(\mu^{*}(k_{0}),k_{0})}=\mu^{*}(k_{0})\frac{\int_{\Omega}\left[\ln(1+\tilde{u})|\nabla\phi|^{2}+\frac{\phi\nabla\phi\cdot\nabla\tilde{u}}{1+\tilde{u}}\right]\mathrm{d}x}{\int_{\Omega}\phi^{2}\mathrm{d}x},$
(2.20)

where $\phi=\phi_{1}(\mu^{*}(k_{0}),(\alpha
F(\tilde{u})-\theta)(1+\tilde{u})^{k_{0}})$. If

$\frac{\partial\lambda_{1}\left({\mu},\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u};k)}\right)}{\partial
k}\Big{|}_{({\mu},k)=(\mu^{*}(k_{0}),k_{0})}>0\ (\mathrm{resp}.<0),$ (2.21)

then $\frac{\partial\mu^{*}(k)}{\partial k}|_{k=k_{0}}>0\ (\mathrm{resp}.<0)$.
Moreover, $\lim\limits_{k\to 0}\mu^{*}(k)=\mu^{*}(0)$, $\lim\limits_{k\nearrow
k^{*}}\mu^{*}(k)=+\infty$ and $\mu^{*}(k)\in(\tilde{\mu},+\infty)$ for any
$k\in[0,k^{*})$.

###### Proof.

For simplicity, we denote $\lambda_{1}\left({\mu},\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u};k)}\right)$ and
$\phi_{1}\left({\mu},\frac{\alpha F(\tilde{u})-\theta}{d(\tilde{u};k)}\right)$
by $\lambda_{1}$ and $\phi_{1}$, respectively. Recall that $\lambda_{1}$ and
$\phi_{1}$ satisfy

$\begin{cases}{\mu}\Delta\phi_{1}+\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u};k)}\phi_{1}=\lambda_{1}\phi_{1},&\mbox{in}\;\Omega,\\\
\nabla\phi_{1}\cdot n=0,&\mbox{on}\;\partial\Omega.\end{cases}$ (2.22)

Differentiating (2.24) with respect to $k$, we get

$\begin{cases}{\mu}\Delta\phi^{\prime}_{1}+\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u};k)}\phi_{1}^{\prime}-d^{\prime}(\tilde{u};k)\frac{\alpha
F(\tilde{u})-\theta}{d^{2}(\tilde{u};k)}\phi_{1}=\lambda_{1}\phi^{\prime}_{1}+\lambda_{1}^{\prime}\phi_{1},&x\in\Omega,\\\
\nabla\phi^{\prime}_{1}\cdot n=0,&x\in\partial\Omega\end{cases}$ (2.23)

where we have used ′ to denote $\frac{\partial}{\partial k}$. Multiplying the
first equation of (2.24) by $\phi_{1}^{\prime}$, and then integrating the
resulting equation on $\Omega$, one obtains

$\int_{\Omega}\left({\mu}\phi^{\prime}_{1}\Delta\phi_{1}+\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u};k)}\phi_{1}\phi_{1}^{\prime}\right)\textrm{d}x=\lambda_{1}\int_{\Omega}\phi_{1}\phi_{1}^{\prime}\mathrm{d}x.$

Similarly, multiplying the first equation of (2.23) by $\phi_{1}$, and
integrating the resulting equation on $\Omega$, we obtain

$\int_{\Omega}\left({\mu}\phi_{1}\Delta\phi^{\prime}_{1}+\frac{(\alpha
F(\tilde{u})-\theta)\phi_{1}\phi_{1}^{\prime}}{d(\tilde{u};k)}-\frac{(\alpha
F(\tilde{u})-\theta)\phi^{2}_{1}d^{\prime}(\tilde{u};k)}{d^{2}(\tilde{u};k)}\right)\mathrm{d}x=\lambda_{1}\int_{\Omega}\phi_{1}\phi^{\prime}_{1}\mathrm{d}x+\lambda^{\prime}_{1}\int_{\Omega}\phi_{1}^{2}\mathrm{d}x.$

Subtracting the above two equations and applying the integration by parts
immediately give (2.18). Since the proofs of (2.19) and (2.20) are similar, we
only prove (2.19). Recall from Lemma 2.5 that
$(\lambda_{1}(\mu^{*}(k_{0}),(\alpha
F(\tilde{u})-\theta)e^{k_{0}\tilde{u}}),\phi_{1}(\mu^{*}(k_{0}),(\alpha
F(\tilde{u})-\theta)e^{k_{0}\tilde{u}}))=:(\lambda_{1},\phi_{1})=(0,\phi_{1})$
satisfies

$\begin{cases}\mu^{*}(k_{0})\Delta\phi_{1}+(\alpha
F(\tilde{u})-\theta)e^{k_{0}\tilde{u}}\phi_{1}=0,&\mbox{in}\;\Omega,\\\
\nabla\phi_{1}\cdot n=0,&\mbox{on}\;\partial\Omega.\end{cases}$ (2.24)

Then it follows from (2.18) that

$\displaystyle\frac{\partial\lambda_{1}\left({\mu},(\alpha
F(\tilde{u})-\theta)e^{k\tilde{u}}\right)}{\partial
k}\Big{|}_{({\mu},k)=(\mu^{*}(k_{0}),k_{0})}$
$\displaystyle=\frac{\int_{\Omega}(\alpha
F(\tilde{u})-\theta)\tilde{u}e^{k_{0}\tilde{u}}\phi_{1}^{2}\mathrm{d}x}{\int_{\Omega}\phi_{1}^{2}\mathrm{d}x}$
$\displaystyle=\frac{-\int_{\Omega}\mu^{*}(k_{0})\phi_{1}\tilde{u}\Delta\phi_{1}\mathrm{d}x}{\int_{\Omega}\phi_{1}^{2}\mathrm{d}x}$
$\displaystyle=\mu^{*}(k_{0})\frac{\int_{\Omega}\left[\tilde{u}|\nabla\phi_{1}|^{2}+\phi_{1}\nabla\phi_{1}\cdot\nabla\tilde{u}\right]\mathrm{d}x}{\int_{\Omega}\phi_{1}^{2}\mathrm{d}x}.$

This proves (2.19).

As to (2.21), we only consider the case
$\frac{\partial\lambda_{1}\left({\mu},\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u};k)}\right)}{\partial
k}\Big{|}_{({\mu},k)=(\mu^{*}(k_{0}),k_{0})}>0$ and the other case can been
treated similarly. For this case, we recall that
$\lambda_{1}(\mu^{*}(k_{0}),\frac{\alpha
F(\tilde{u})-\theta}{d(k_{0};\tilde{u})})=0$, which yields that (differentiate
it with respect to $k$)

$\frac{\partial\lambda_{1}\left({\mu},\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u};k)}\right)}{\partial{\mu}}\bigg{|}_{({\mu},k)=(\mu^{*}(k_{0}),k_{0})}\cdot\frac{\partial\mu^{*}(k)}{\partial
k}\bigg{|}_{k=k_{0}}+\frac{\partial\lambda_{1}\left({\mu},\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u};k)}\right)}{\partial
k}\bigg{|}_{({\mu},k)=(\mu^{*}(k_{0}),k_{0})}=0.$

This combined with Lemma 2.1 (ii) gives $\frac{\partial\mu^{*}(k)}{\partial
k}|_{k=k_{0}}>0$. Finally, the last part of this lemma is derived directly
from Lemma 2.5. ∎

Lemma 2.7 tells us that the sign of quantities defined in (2.19) or (2.20)
determines the monotonicity of $\mu^{*}(k)$ with respect to $k$. In general
these quantities may change signs by as $k$ varies from $0$ to $k^{*}$. In the
following Lemma, we shall show that the sign of quantities defined in (2.19)
or (2.20) can be determined if $m(x)$ is monotonic.

######  Lemma 2.8.

Assume $\Omega=[0,L]$, $m_{x}\geq 0$ in $(0,L)$ or $m_{x}\leq 0$ in $(0,L)$
and $d(u)=d(u;k)$, where $d(u;k)=e^{-ku}$ or $(1+u)^{-k}$. If
$\theta\in[\theta_{0},\alpha F(\tilde{u}_{\max}))$, then
$\frac{\partial\mu^{*}(k)}{\partial k}>0$ for $k\in[0,k^{*})$.

###### Proof.

We only consider the case $m_{x}\geq 0$ in $(0,L)$ and $d(u;k)=e^{-ku}$ while
other cases can be proven similarly. Recall that $\tilde{u}$ satisfies

$\begin{cases}\epsilon\tilde{u}_{xx}+\tilde{u}(m(x)-\tilde{u})=0,&\mbox{in}\;(0,L),\\\
\tilde{u}_{x}(0)=\tilde{u}_{x}(L)=0.\end{cases}$

Define $\eta:=\frac{\tilde{u}_{x}}{\tilde{u}}$ on $[0,L]$. Then $\eta$
satisfies

$\begin{cases}-\epsilon\eta_{xx}+(\tilde{u}-2\epsilon\eta_{x})\eta=m_{x}\geq
0,&\mbox{in}\;(0,L),\\\ \eta(0)=\eta(L)=0.\end{cases}$

By the strong maximum principle, one finds that

$\eta>0\quad\mbox{in}\;(0,L),$

which yields that $\tilde{u}_{x}>0$ in $(0,L)$. Recall that
$\phi_{1}(\mu^{*}(k),(\alpha F(\tilde{u})-\theta)e^{k\tilde{u}})$ satisfies

$\begin{cases}\mu^{*}(k)\phi_{xx}+\phi(\alpha
F(\tilde{u})-\theta)e^{k\tilde{u}}=0,&\mbox{in}\;(0,L),\\\
\phi_{x}(0)=\phi_{x}(L)=0.\end{cases}$ (2.25)

Integrating the first equation of (2.25) over $(0,L)$, one obtains

$\int_{0}^{L}\phi_{1}(\alpha F(\tilde{u})-\theta)e^{k\tilde{u}}\mathrm{d}x=0,$

where $\phi_{1}$ denotes $\phi_{1}(\mu^{*}(k),(\alpha
F(\tilde{u})-\theta)e^{k\tilde{u}})$ for simplicity. This fact combined with
$\tilde{u}_{x}>0$ in $(0,L)$, implies that there exist some $x^{*}\in(0,L)$
such that

$\mathrm{sgn}(\alpha F(\tilde{u}(x))-\theta)=\mathrm{sgn}(x-x^{*}).$

This fact together with $\phi_{1}>0$ on $[0,L]$ and the first equation of
(2.25) yields that

$\mathrm{sgn}((\phi_{1})_{xx})=-\mathrm{sgn}(x-x^{*}).$

which alongside the boundary conditions
$(\phi_{1})_{x}(0)=(\phi_{1})_{x}(L)=0$ indicates that

$(\phi_{1})_{x}>0\quad\mbox{in}\;(0,L).$ (2.26)

Combining (2.26), $\tilde{u}_{x}>0$ in $(0,L)$, (2.19) and Lemma 2.7, one
concludes that $\frac{\partial\mu^{*}(k)}{\partial k}>0$ for any
$k\in[0,k^{*})$. ∎

##  3\. Existence and non-existence of positive solutions to system (1.5)

In this section, we shall prove the existence and non-existence of positive
solutions to system (1.5) with help of index theory based on the results
established in section 2. We start by reviewing some well-known results of the
index theory.

###  3.1. Index theory

Let $E$ be a real Banach space and $W$ be a closed convex set of $E$ such that
$W-W$ is dense in $E$. The closed convex set $W$ is said to be a cone if
$eW\subset W$ for all $e\geq 0$ and $W\cap\\{-W\\}=0$. Define

$W_{y}\triangleq\overline{\\{x\in E|y+ex\in W\mbox{ for some $e>0$}\\}}.$

Denote the maximal linear subspace of $E$ contained in $W_{y}$ by $S_{y}$.
Assume that $T:\ E\to E$ is a compact linear and Fréchet differentiable
operator on $E$ such that $y\in W$ is a fixed point of $T$ and $T(W)\subseteq
W$. If there exists a closed linear subspace $E_{y}$ of $E$ such that
$E=S_{y}\bigoplus E_{y}$ and $W_{y}$ is generating, then the following result
holds.

######  Lemma 3.1 ([10, 45]).

Let $P:\ E\to E_{y}$ be the projection operator. Then
$\mathrm{index}_{W}(T,y)$ exists if the Fréchet derivative $T^{\prime}(y)$ of
$T$ at $y$ has no non-zero fixed point in $W_{y}$. Moreover,

  * (i)

$\mathrm{index}_{W}(T,y)=0$ if $P\circ T^{\prime}(y)$ has an eigenvalue bigger
than $1$; Otherwise,

  * (ii)

$\mathrm{index}_{W}(T,y)=\mathrm{index}_{S_{y}}(T^{\prime}(y),0)=(-1)^{\imath}$,
where ${\mathrm{index}}_{S_{y}}(T^{\prime}(y),0)$ is the index of the linear
operator $T^{\prime}(y)$ at $0$ in the space $S_{y}$ and $\imath$ is the sum
of algebraic multiplicities of the eigenvalues of $T^{\prime}(y)$ restricted
in $S_{y}$ which are greater than 1.

###  3.2. Preliminary results

We first quote an important result on the eigenvalue problem [11, 32].

######  Lemma 3.2.

Assume $r(x)\in C([0,L])$, $\mu>0$, and $M>0$ such that $M+r>0$ on $\Omega$.
If $\lambda_{1}(\mu,r)>0$, then the weighted eigenvalue problem,

$\begin{cases}-\mu\Delta\phi+M\phi=\kappa(M+r)\phi,\qquad&x\in\Omega,\\\
\nabla\phi\cdot n=0,&x\in\partial\Omega,\end{cases}$ (3.1)

has an eigenvalue $\kappa$ smaller than $1$. If $\lambda_{1}(\mu,r)<0$, then
it has no eigenvalue smaller than or equal to $1$.

Next we give an upper bound on possible positive solutions of system (1.5).

######  Lemma 3.3.

Let $(u,w)$ be a positive solution of system (1.5). Then

$u<\tilde{u}\leq{m}_{\max}\mbox{ and $w\leq c_{0}$ on $\Omega$, }$ (3.2)

where $m_{\max}=\max\limits_{x\in\bar{\Omega}}m$ and $c_{0}>0$ is a constant
depending on $m$, $\alpha$, $\theta$, $F(\cdot)$ and $d(\cdot)$.

###### Proof.

Combining the standard method of upper-lower solutions and the maximum
principle, one can deduce that

$u<\tilde{u}\leq{m}_{\max}\;\;\mbox{on}\;\;\Omega.$

Multiplying the first equation of system (1.5) by $\alpha$, adding the
resulting equation to the second equation of system (1.5) and integrating it
on $\Omega$, one obtains

$\int_{\Omega}\frac{\theta}{d(u)}w\mathrm{d}x=\alpha\int_{\Omega}u(m-u)\mathrm{d}x.$

This combined with $u<{m}_{\max}$ on $\Omega$ and $(H_{3})$ yields that

$\int_{\Omega}w\mathrm{d}x<\frac{\alpha
d(0)}{\theta}\int_{\Omega}u(m-u)\mathrm{d}x<\frac{\alpha
d(0)m_{\max}^{2}}{4\theta},$

which together with [1, Theorem 3.1] implies that there exists $c_{0}$
depending on $m$, $\alpha$, $\theta$, $F(\cdot)$ and $d(\cdot)$ (independents
on $\mu$ and $\epsilon$) such that $w\leq c_{0}\;\;\mbox{on}\;\;\Omega.$ ∎

Before moving forward, we introduce some notations.

$\displaystyle X$ $\displaystyle=$ $\displaystyle\\{u\in
C^{1}(\bar{\Omega})\cap C^{2}(\Omega)\ |\ \nabla u\cdot
n=0\;\mbox{on}\;\partial\Omega\\},$ $\displaystyle E$ $\displaystyle=$
$\displaystyle C(\bar{\Omega})\times C(\bar{\Omega}),$ $\displaystyle W$
$\displaystyle=$ $\displaystyle C^{+}(\bar{\Omega})\times
C^{+}(\bar{\Omega}),\mbox{ where $C^{+}(\bar{\Omega})=\\{u\in C(\bar{\Omega})\
|\ u\geq 0\\}$,}$ $\displaystyle\mathcal{D}$ $\displaystyle=$
$\displaystyle\left\\{(u,w)\in W\ |\ u<1+m_{\max},w<1+c_{0}\right\\}.$

Let $\mathcal{T}_{1}^{-1}$ be the inverse operator of $\mathcal{T}_{1}$ with
$\mathcal{T}_{1}(u)=-\epsilon\Delta u+\tilde{M}u$ for $u\in X$ and
$\mathcal{T}_{2}^{-1}$ be the inverse operator of $\mathcal{T}_{2}$ with
$\mathcal{T}_{2}(w)=-\mu\Delta w+\tilde{M}w$ for $u\in X$. For any
$\delta\in[0,1]$, we define $T_{\delta}:\mathcal{D}\rightarrow W$ by

$T_{\delta}(u,w)=\begin{pmatrix}\mathcal{T}_{1}^{-1}\left[u\left(\tilde{M}+\delta
m(x)-u-\frac{F(u)w}{d(u)u}\right)\right]\\\\[6.0pt]
\mathcal{T}_{2}^{-1}\left[w\left(\tilde{M}+\frac{\alpha
F(u)-\theta}{d(u)}\right)\right]\end{pmatrix},\qquad(u,w)\in\mathcal{D},$

where $\tilde{M}$ is large such that

$\tilde{M}-|m(x)|-u-\frac{F(u)w}{d(u)u}>0\quad\mbox{and}\quad\tilde{M}+\frac{\alpha
F(u)-\theta}{d(u)}>0,\quad\mbox{for}\;\;(u,w)\in\mathcal{D}.$

For example, one can choose
$\tilde{M}=\frac{\theta}{d(m_{\max})}+2\|m\|_{L^{\infty}}+\frac{c_{0}}{d(m_{\max})}\max\limits_{u\in[0,m_{\max}]}\frac{F(u)}{u}$,
where $\max\limits_{u\in[0,m_{\max}]}\frac{F(u)}{u}$ is bounded due to the
assumption $(H_{2})$. It is well-known that $T_{1}$ is a compact operator and
$T_{1}(\mathcal{D})\subseteq W$. Clearly system (1.5) has a positive solution
if and only if $T_{1}$ admits a positive fixed point on $\mathcal{D}$ by Lemma
3.3.

Direct computations yield

$\displaystyle W_{(0,0)}=C^{+}(\bar{\Omega})\times
C^{+}(\bar{\Omega}),\;S_{(0,0)}=\\{(0,0)\\},\;E_{(0,0)}=E,$ $\displaystyle
W_{(\tilde{u},0)}=C(\bar{\Omega})\times
C^{+}(\bar{\Omega}),\;S_{(\tilde{u},0)}=C(\bar{\Omega})\times\\{0\\},\;E_{(\tilde{u},0)}=\\{0\\}\times
C(\bar{\Omega}).$

With the above preparations, we start to calculate the index of $(0,0)$ and
$(\tilde{u},0)$.

######  Lemma 3.4.

The following results on the index holds.

  * (i)

$\mathrm{index}_{W}(T_{1},(0,0))=0$.

  * (ii)

$\mathrm{index}_{W}(T_{1},(\tilde{u},0))=\begin{cases}1,&\mbox{if
$(\tilde{u},0)$ is linearly stable},\\\ 0,&\mbox{if $(\tilde{u},0)$ is
linearly unstable}.\end{cases}$

  * (iii)

${\rm{\rm{deg}}}_{W}(I-T_{1},\mathcal{D})=1$.

###### Proof.

For statement (i), we linearize $T_{1}$ at $(0,0)$ to obtain

$DT_{1}(0,0)(\phi,\psi)=\begin{pmatrix}\mathcal{T}_{1}^{-1}[(\tilde{M}+m)\phi]\\\
\mathcal{T}_{2}^{-1}\left[\left(\tilde{M}-\frac{\theta}{d(0)}\right)\psi\right]\end{pmatrix}.$

It is straightforward to see that the operator $DT_{1}(0,0)$ has no non-zero
fixed point in $W_{(0,0)}$ due to the fact that $\lambda_{1}(\epsilon,m)>0$
and $\lambda_{1}\left(\mu,-\frac{\theta}{d(0)}\right)=-\frac{\theta}{d(0)}<0$.
From Lemma 2.3 and Lemma 3.2, it follows that $DT_{1}(0,0)$ admits an
eigenvalue bigger than 1 with corresponding eigenfunction
$(\phi_{1}(\epsilon,m),0)$. Therefore, by Lemma 3.1, we get
$\mathrm{index}_{W}(T_{1},(0,0))=0$.

As to assertion (ii), linearizing $T_{1}$ at $(\tilde{u},0)$, one has

$DT_{1}(\tilde{u},0)(\phi,\psi)=\begin{pmatrix}\mathcal{T}_{1}^{-1}\left[\phi\left(\tilde{M}+m(x)-2\tilde{u}\right)-\frac{F(\tilde{u})}{d(\tilde{u})}\psi\right]\\\\[6.0pt]
\mathcal{T}_{2}^{-1}\left[\psi\left(\tilde{M}+\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u})}\right)\right]\end{pmatrix}.$

We will show that the operator $DT_{1}(\tilde{u},0)$ has no non-zero fixed
point in $W_{(\tilde{u},0)}$. Otherwise, assume that $DT_{1}(\tilde{u},0)$ has
a non-zero fixed point $(\phi,\psi)$ in $W_{(\tilde{u},0)}$. Then
$(\phi,\psi)$ satisfies

$\begin{cases}\epsilon\Delta\phi+(m-2\tilde{u})\phi-\frac{F(\tilde{u})}{d(\tilde{u})}\psi=0,\quad&\mbox{in}\;\Omega,\\\
\mu\Delta\psi+\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u})}\psi=0,\quad&\mbox{in}\;\Omega,\\\
\nabla\phi\cdot n=\nabla\psi\cdot
n=0,\quad&\mbox{on}\;\partial\Omega.\end{cases}$

If $\psi=0$, then $\phi=0$ due to
$\lambda_{1}(\epsilon,m-2\tilde{u})<\lambda_{1}(\epsilon,m-\tilde{u})=0$ by
Lemma 2.1(iii). Thus, one obtains $\psi\in
C^{+}(\bar{\Omega})\setminus\\{0\\}$, which further implies that
$\lambda_{1}\big{(}\mu,\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u})}\big{)}=0$. This contradicts our assumption
that $(\tilde{u},0)$ is linearly stable ($\lambda_{1}\big{(}\mu,\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u})}\big{)}<0$) or unstable
($\lambda_{1}\big{(}\mu,\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u})}\big{)}>0$). Hence, the operator
$DT(\tilde{u},0)$ does not have non-zero fixed points in $W_{(\tilde{u},0)}$.
If $(\tilde{u},0)$ is linearly unstable, that is,
$\lambda_{1}\big{(}\mu,\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u})}\big{)}>0$, one attains that
$\mathcal{T}_{2}^{-1}\big{[}\cdot\big{(}\tilde{M}+\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u})}\big{)}\big{]}$ has an eigenvalue bigger
than 1 by Lemma 3.2. This combined with Lemma 3.1 gives that
$\mathrm{index}_{W}(T_{1},(\tilde{u},0))=0$. On the other hand, if
$(\tilde{u},0)$ is linearly stable, by Lemma 3.2, one knows that all the
eigenvalues of the operator
$\mathcal{T}_{2}^{-1}\big{[}\cdot\big{(}\tilde{M}+\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u})}\big{)}\big{]}$ are smaller than 1. This
together with Lemma 3.1 yields that

$\mathrm{index}_{W}(T_{1},(\tilde{u},0))=(-1)^{\imath},$

where $\imath$ is the sum of algebraic multiplicities of the eigenvalues of
the operator $DT_{1}(\tilde{u},0)$ restricted in $S_{y}$ which are greater
than 1.

Next we show that all the eigenvalues of the operator $DT_{1}(\tilde{u},0)$
restricted in $S_{y}$ are smaller than 1. If not, we assume the operator
$DT_{1}(\tilde{u},0)$ admits an eigenvalue $\kappa_{0}\geq 1$ with
eigenfunction $(\phi,0)\in S_{y}$ satisfying $\|\phi\|_{L^{2}}=1$. Then
$\kappa_{0}$ and $(\phi,0)$ satisfy

$\begin{cases}-\epsilon\Delta\phi+\tilde{M}\phi=\frac{\phi}{\kappa_{0}}(\tilde{M}+m-2\tilde{u}),\quad&\mbox{in}\;\Omega,\\\
\nabla\phi\cdot n=0,\quad&\mbox{on}\;\partial\Omega.\end{cases}$

This contradicts the fact that $\lambda_{1}(\epsilon,m-2\tilde{u})<0$ and
Lemma 3.2. Therefore, one concludes

$\mathrm{index}_{W}(T_{1},(\tilde{u},0)=(-1)^{0}=1,\quad\mbox{when}\;(\tilde{u},0)\;\mbox{is
linearly stable}.$

Finally, we prove that ${\rm{\rm{deg}}}_{W}(I-T_{1},\mathcal{D})=1$. If
$T_{\delta}$ has a fixed point $(u,w)$, then it satisfies

$\begin{cases}\epsilon\Delta u+u(\delta
m(x)-u)-\frac{F(u)w}{d(u)}=0,&\mbox{in}\;\Omega,\\\ \mu\Delta
w+\alpha\frac{F(u)w}{d(u)}-\theta\frac{w}{d(u)}=0,&\mbox{in}\;\Omega,\\\
\nabla u\cdot n=\nabla w\cdot n=0,&\mbox{on}\;\partial\Omega.\end{cases}$
(3.3)

Similar to Lemma 3.3, for all $\delta\in[0,1]$, one can show that if system
(3.3) has a positive solution $(u,w)$ then it satisfies (3.2) (if necessary,
one can choose large $c_{0}$). Then, $T_{\delta}$ doesn’t have any fixed point
on $\partial\mathcal{D}$. Thus, by the homotopy invariance, one obtains

$\deg_{W}(I-T_{1},\mathcal{D})=\deg_{W}(I-T_{\delta},\mathcal{D}).$ (3.4)

Obviously, system (3.3) only admits non-negative solution $(0,0)$ and
$(\tilde{u}_{\delta m},0)$ ($\tilde{u}_{\delta m}$ denotes the unique positive
solution of (2.3) by replacing $m$ with $\delta m$) when $\delta$ is small
enough. Therefore, one has

$\deg_{W}(I-T_{\delta},\mathcal{D})=\mathrm{index}_{W}(T_{\delta},(0,0))+\mathrm{index}_{W}(T_{\delta},(\tilde{u}_{\delta
m},0)),$ (3.5)

where $\delta$ is small enough. Linearizing $T_{\delta}$ at $(0,0)$, one gets

$DT_{\delta}(0,0)(\phi,\psi)=\begin{pmatrix}\mathcal{T}_{1}^{-1}[(\tilde{M}+\delta
m)\phi]\\\
\mathcal{T}_{2}^{-1}\left[\left(\tilde{M}-\frac{\theta}{d(0)}\right)\psi\right]\end{pmatrix}.$

Since $\lambda_{1}(\epsilon,\delta m)>0$ and
$\lambda_{1}\left(\mu,-\frac{\theta}{d(0)}\right)=-\frac{\theta}{d(0)}<0$,
similar to the results in (i), by Lemma 3.2 and Lemma 3.1, we have

$\mathrm{index}_{W}(T_{\delta},(0,0))=0,$ (3.6)

where $\delta$ is small enough. It is easy to derive that $(\tilde{u}_{\delta
m},0)$ is linearly stable when $\delta$ is small enough. Therefore, from
statement (ii), it follows that

$\mathrm{index}_{W}(T_{\delta},(\tilde{u}_{\delta m},0))=1,\quad\mbox{when
$\delta$ is small enough},$

which along with (3.4), (3.5) and (3.6) completes the proof. ∎

With the help of Lemma 3.4, we give the sufficient conditions for the
existence of positive solution to system (1.5).

######  Lemma 3.5.

If $(\tilde{u},0)$ is linearly unstable, then system (1.5) admits at least one
positive solution.

###### Proof.

If system (1.5) doesn’t have any positive solution, by the additivity of
indices and Lemma 3.4, we have

$1={\rm{\rm{deg}}}_{W}(I-T_{1},\mathcal{D})=\mathrm{index}_{W}(T_{1},(0,0))+\mathrm{index}_{W}(T_{1},(\tilde{u},0))=0+0=0,$

which is impossible. Hence, system (1.5) admits at least one positive solution
when $(\tilde{u},0)$ is linearly unstable. ∎

###  3.3. Main results

Now it is in a position to state our main results on the existence and non-
existence of positive solutions to (1.5).

######  Theorem 3.1.

Given $\epsilon,\alpha>0$, assume $(H_{1})$, $(H_{2})$ and $(H_{3})$ hold. Let
$\tilde{u}$ be the unique solution of (2.3) and
$\theta_{0}=\frac{\alpha}{|\Omega|}\int_{\Omega}F(\tilde{u})\mathrm{d}x.$ Then
the following results hold.

  * (i)

If $\theta\in[0,\alpha F(\tilde{u}_{\min})]$, then system (1.5) admits at
least a positive solution.

  * (ii)

If $\theta\in[\alpha F(\tilde{u}_{\max}),\infty)$, then system (1.5) doesn’t
admit any positive solution.

  * (iii)

If $\theta\in(\alpha F(\tilde{u}_{\min}),\alpha F(\tilde{u}_{\max}))$ and
$d(u)=d(u;k)$ where $d(u;k)$ is given in (2.6), then the following results
follow.

    * (a)

Fixing all the parameters except $\mu$, if $\theta\in\left(\alpha
F(\tilde{u}_{\min}),\theta_{k}\right]$ with $\theta_{k}$ defined in (2.7),
then system (1.5) admits a positive solution for any $\mu>0$; while if
$\theta\in\left(\theta_{k},\alpha F(\tilde{u}_{\max})\right)$, then there
exits some $\mu^{*}(\theta)>0$ satisfying
$\textstyle\lambda_{1}\big{(}\mu^{*},\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u};k)}\big{)}=0$ such that system (1.5) admits a
positive solution for all ${\mu}\in(0,\mu^{*}(\theta))$ and the semi-trivial
solution $(\tilde{u},0)$ is linearly stable for all ${\mu}>\mu^{*}(\theta)$.

    * (b)

Fixing all the parameters except $\mu$ and $k$, we have

      * (b.1)

If $\theta\in\left(\alpha F(\tilde{u}_{\min}),\theta_{0}\right]$, then system
(1.5) admits at least a positive solution for any ${\mu}>0$ and $k\geq 0$.

      * (b.2)

If $\theta\in\left(\theta_{0},\alpha F(\tilde{u}_{\max})\right)$, then there
exist $k^{*}(\theta)>0$ satisfying (2.10) such that:

        * (b.2A)

If $k\geq k^{*}(\theta)$, then system (1.5) admits a positive solution for any
${\mu}>0$;

        * (b.2B)

If $k\in[0,k^{*}(\theta))$, then there exists $\mu^{*}(k)$ satisfying
$\lambda_{1}\big{(}\mu^{*}(k),\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u})}\big{)}=0$ such that system (1.5) admits a
positive solution for all ${\mu}\in(0,\mu^{*}(k))$ and $(\tilde{u},0)$ is
linearly stable for all ${\mu}>\mu^{*}(k)$. Furthermore there exist a constant
$\tilde{k}(\theta)>0$ so that system (1.5) has no positive solutions for all
$k\in[0,\tilde{k}(\theta)]$ and ${\mu}>\mu^{*}(k)$.

###### Proof.

The results stated in assertions (i), (iii)-(a), and (iii)-(b.1) follow
directly from Lemma 2.5 and Lemma 3.5. For the assertion (ii), we use a
contradictive argument by assuming that system (1.5) admits a positive
solution $(u,w)$. From the second equation of system (1.5) and the Krein-
Rutman Theorem [30], it follows that

$\lambda_{1}\left(\mu,\frac{\alpha F(u)-\theta}{d(u)}\right)=0.$ (3.7)

On the other hand, due to assumptions $\theta\geq\alpha F(\tilde{u}_{\max})$,
$(H_{1})$ and $(H_{2})$, one concludes that

$\alpha F(u)-\theta\leq,\not\equiv 0\;\;\mbox{in}\;\Omega,$

which together with $\lambda_{1}(\mu,0)=0$ and Lemma 2.1-(iii) implies that
$\lambda_{1}\big{(}\mu,\frac{\alpha F(u)-\theta}{d(u)}\big{)}<0.$ This
contradicts (3.7). Therefore, the results in statement (ii) holds.

Finally we prove the results in the assertion (iii)-(b.2). First the result
(b.2A) in the statement (b.2) come from Lemma 2.5 and Lemma 3.5 directly. It
remains only to show result (B) in the statement (b.2). Given
$k\in[0,k^{*}(\theta))$, using Lemma 2.1 and $\theta\in(\alpha
F(\tilde{u}_{\min}),\alpha F(\tilde{u}_{\max}))$, we have

$\lim\limits_{{\mu}\to 0}\lambda_{1}\left({\mu},\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u};k)}\right)=\max\limits_{x\in\bar{\Omega}}\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u};k)}>0$

and

$\lim\limits_{{\mu}\to\infty}\lambda_{1}\left({\mu},\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u};k)}\right)=\frac{\int_{\Omega}\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u};k)}\mathrm{d}x}{|\Omega|}<0,\ \mbox{due to
$k<k^{*}(\theta)$}\ \mbox{and}\ \eqref{2022062802}.$

These facts combined with Lemma 2.1 (ii) and Lemma 3.5 imply the first part of
(B) in (b.2). Next proceed to prove the existence of $\tilde{k}(\theta)$. To
this end, we consider the case $d(u;k)=e^{-ku}$ only and the other case
$d(u;k)=(1+u)^{-k}$ can be treated similarly. We define
$\mathfrak{h}(x):=(\alpha F(x)-\theta)e^{kx}$, $x\in[0,\tilde{u}_{\max}]$.
Then, one has

$\mathfrak{h}_{x}(x)=[\alpha F_{x}(x)+k(\alpha F(x)-\theta)]e^{kx},$

which combined with assumption $(H_{2})$ implies that there exists
$\tilde{k}>0$, such that

$\mathfrak{h}_{x}(x)>0,\quad x\in[0,\tilde{u}_{\max}].$ (3.8)

Assume $k\leq\tilde{k}$ and ${\mu}>\mu^{*}(k)$, we will show that system (1.5)
doesn’t admit any positive solution. By contradiction, assume that system
(1.5) admits a positive solution $(u,w)$. From the second equation of (1.5),
it follows that

$\lambda_{1}({\mu},(\alpha F(u)-\theta)e^{ku})=0.$

This together with (3.8), Lemma 2.1, and Lemma 3.3 yields that
$\lambda_{1}({\mu},(\alpha F(\tilde{u})-\theta)e^{k\tilde{u}})>0,$ which
alongside Lemma 2.1 indicates that

$\lambda_{1}(\mu^{*}(k),(\alpha F(\tilde{u})-\theta)e^{k\tilde{u}})>0.$

This contradicts the definition of $\mu^{*}(k)$, that is,
$\lambda_{1}(\mu^{*}(k),(\alpha F(\tilde{u})-\theta)e^{k\tilde{u}})=0$. So,
system (1.5) doesn’t admit any positive solution. ∎

As a direct consequence of Theorem 3.1, we have the following results for the
predator-prey system with random dispersal.

######  Corollary 3.1.

Given $\epsilon,\alpha>0$, assume $d(u)=1$, $(H_{1})$ and $(H_{2})$ hold. Then
the following results hold.

  * (i)

If $\theta\in[0,\alpha F(\tilde{u}_{\min})]$, then system (1.5) admits at
least a positive solution.

  * (ii)

If $\theta\in[\alpha F(\tilde{u}_{\max}),\infty)$, then system (1.5) doesn’t
admit any positive solution.

  * (iii)

If $\theta\in(\alpha F(\tilde{u}_{\min}),\alpha F(\tilde{u}_{\max}))$, the
following results hold true.

    * (a)

If $\theta\in\left(\alpha F(\tilde{u}_{\min}),\theta_{0}\right]$, then system
(1.5) admits at least a positive solution for any ${\mu}>0$.

    * (b)

If $\theta\in\left(\theta_{0},\alpha F(\tilde{u}_{\max})\right)$, then there
exists a constant $\mu^{*}$ satisfying $\lambda_{1}(\mu^{*},\alpha
F(\tilde{u})-\theta))=0$ such that system (1.5) doesn’t admit any positive
solution for $\mu>\mu^{*}$ while admits a positive solution for all
$\mu\in(0,\mu^{*})$.

######  Remark 3.1.

We have several remarks in connection with the results of Theorem 3.1.

  * •

Comparing the results of Theorem 3.1 with those of Corollary 3.1, we see that
the density-dependent dispersal will have no impact on the species coexistence
when the predator’s death rate $\theta>0$ is small (i.e. $\theta\leq\alpha
F(\tilde{u}_{\min})$) or large (i.e. $\theta\geq\alpha F(\tilde{u}_{\max})$).
However if the value of $\theta$ is moderate (i.e. $\theta\in(\alpha
F(\tilde{u}_{\min}),\alpha F(\tilde{u}_{\max}))$), the density-dependent
dispersal will have evident impact on the species coexistence. Considering the
case $d(u)=e^{-ku}$ or $(1+u)^{-k}$ with $k\geq 0$, the results stated in
(iii)-(a) of Theorem 3.1 can be illustrated in Figure 1(a) and Figure 1(b)
where we see that the parameter regions of $(\theta,\mu)$ for the existence of
positive solutions (i.e. $\lambda_{1}>0$) increases as $k$ increases. This
implies that the density-dependent dispersal will increase the chance of
species coexistence. The result in (iii)-(b) of Theorem 3.1 gives another way
of understanding the impact of density-dependent dispersal, where for given
$\theta\in(\theta_{0},\alpha F(\tilde{u}_{\max}))$ coexistence (positive)
solutions exist only if $0<\mu<\mu^{*}(0)$ when $k=0$ (see (iii)-(b) of
Corollary 3.1) while exit for any $\mu>0$ when $k>k^{*}(\theta)$ (see
(iii)-(b.2) in Theorem 3.1), as illustrated in Figure 1(c). For
$k\in[0,k^{*}(\theta))$ and in dimension one, we have shown that $\mu^{*}(k)$
increases with respect to $k$ (see Lemma 2.8) and hence the range of $\mu$ for
the coexistence (i.e. $\lambda_{1}>0$) increases as $k$ increases (see Figure
1(c)). This is also verified by our numerical simulations shown in Figure 2.

Figure 1. Illustration of parameter regimes (shaded regions) for existence of
positive solutions to (1.5) (i.e. $\lambda_{1}>0$), where $\theta^{*}=\alpha
F(\tilde{u}_{\max})$ and
$\theta_{k}=\frac{\alpha\int_{\Omega}\frac{F(\tilde{u})}{d(\tilde{u};k)}\mathrm{d}x}{\int_{\Omega}\frac{1}{d(\tilde{u};k)}\mathrm{d}x}$
with $d(u;k)=e^{-ku}$ or $d(u;k)=(1+u)^{-k},k\geq 0$.

(a) $d(u)=1$ (b) $d(u)=e^{-8u}$

Figure 2. Numerical simulations of steady state profiles of system (1.2) with
random dispersal shown in (a) and density-dependent dispersal shown in (b),
where $\epsilon=0.1,\mu=10,\alpha=1,\theta=0.8,F(u)=u,m(x)=0.5+0.5\sin(x)$.
The dispersal rate function $d(u)$ is chosen as indicated in the Figure.

  * •

The constant $\tilde{k}$ in (b.2B) of Theorem 3.1 can be explicitly determined
for specific1 $F(u)$. For instance, we can choose

$\tilde{k}=\begin{cases}\frac{\alpha}{\theta},&\mbox{if}\;F(u)=u,\\\
\frac{\alpha}{\theta(1+\tilde{u}_{max})^{2}},&\mbox{if}\;F(u)=\frac{u}{1+u}.\end{cases}$

##  4\. Uniqueness and asymptotic profiles

In this section, we are devoted to investigating the uniqueness and asymptotic
profiles of solutions of (1.5) as $\varepsilon\to\infty$ (fast prey diffuse)
as well as $\mu\to\infty/0$ (fast/slow predator diffusion).

###  4.1. Fast prey diffusion

######  Theorem 4.1.

Suppose that $(H_{1})$, $(H_{2})$ and $(H_{3})$ hold. Let
$\bar{m}=\frac{1}{|\Omega|}\int_{\Omega}m(x)\mathrm{d}x$. If $\epsilon>0$ is
sufficiently large, then the following results hold.

  * (i)

If $\theta>\alpha F(\bar{m})$, then system (1.5) doesn’t have any positive
solution;

  * (ii)

If $0<\theta<\alpha F(\bar{m})$, then system (1.5) admits a unique positive
solution.

###### Proof.

For the assertion (i), arguing by contradiction, we suppose that system (1.5)
admits a positive solution $(u^{i},w^{i})$ with $\epsilon=\epsilon^{i}$, where
$\epsilon^{i}\to+\infty$ as $i\to\infty$. Then $\epsilon^{i}$ and
$(u^{i},w^{i})$ satisfy

$\begin{cases}\epsilon^{i}\Delta
u^{i}+u^{i}(m(x)-u^{i})-\frac{F(u^{i})}{d(u^{i})}w^{i}=0,&\mbox{in}\;\Omega,\\\
\mu\Delta w^{i}+\frac{\alpha
F(u^{i})-\theta}{d(u^{i})}w^{i}=0,&\mbox{in}\;\Omega,\\\ \nabla u^{i}\cdot
n=\nabla w^{i}\cdot n=0,&\mbox{on}\;\partial\Omega.\end{cases}$ (4.1)

Using Lemma 3.3, for any $i\geq 1$, one has

$\|u^{i}\|_{L^{\infty}(\Omega)}\leq
m_{\max}\;\;\mbox{and}\;\;\|w^{i}\|_{L^{\infty}(\Omega)}\leq c_{0}.$

Applying the elliptic regularity (cf. [20]), we have
$\|u^{i}\|_{W^{2,p}(\Omega)}$ and $\|w^{i}\|_{W^{2,p}(\Omega)}$ are uniformly
bounded for any $1<p<\infty$ and $i\geq 1$. By the Sobolev imbedding theorem,
one can deduce from (4.1) that $(u^{i},w^{i})$, passing to a subsequence if
necessary, converges to some nonnegative function $(u^{\infty},w^{\infty})$ in
$C^{1}(\Omega)$ as $i\to\infty$, where $(u^{\infty},w^{\infty})$ satisfies (in
the weak sense)

$\begin{cases}\Delta u_{\infty}=0,&\mbox{in}\;\Omega,\\\ \mu\Delta
w_{\infty}+w_{\infty}\frac{\alpha
F(u_{\infty})-\theta}{d(u_{\infty})}=0,&\mbox{in}\;\Omega,\\\ \nabla
u^{\infty}\cdot n=\nabla w^{\infty}\cdot
n=0,&\mbox{on}\;\partial\Omega.\end{cases}$ (4.2)

From Proposition 2.1 (ii), Lemma 3.3, $(H_{2})$, and the assumption
$\theta>\alpha F(\bar{m})$, it follows that

$\alpha F(u^{\infty})-\theta\leq\alpha F(\bar{m})-\theta<0,$

which indicates that

$\alpha F(u^{i})-\theta<0,\quad\mbox{when $i$ is large}.$ (4.3)

Integrating the second equation of (4.1) on $\Omega$, one obtains

$\int_{\Omega}\frac{\alpha F(u^{i})-\theta}{d(u^{i})}w^{i}\mathrm{d}x=0,$

which contradicts (4.3) when $i$ is large. Hence the results in assertion (i)
are obtained.

For assertion (ii), by Proposition 2.1 (ii) and Lemma 2.1 (i), one has

$\lim\limits_{\epsilon\to\infty}\lambda_{1}\left(\mu,\frac{\alpha
F(\tilde{u}_{\epsilon})-\theta}{d(\tilde{u}_{\epsilon})}\right)=\lambda_{1}\left(\mu,\frac{\alpha
F(\bar{m})-\theta}{d(\bar{m})}\right)=\frac{\alpha
F(\bar{m})-\theta}{d(\bar{m})}>0,$

where $\tilde{u}_{\epsilon}$ denotes the unique positive solution of (2.3).
This combined with Lemma 3.5 implies that, there exits some large
$\epsilon^{*}$ such that system (1.5) admits at least one positive solution
for $\epsilon\geq\epsilon^{*}$. For $\epsilon^{i}\geq\epsilon^{*}$ satisfying
$\epsilon^{i}\to+\infty$ as $i\to\infty$, we will prove that any positive
solution $(u^{i},w^{i})$ of system (1.5) with $\epsilon=\epsilon^{i}$
satisfies that

$\mbox{$(u^{i},w^{i})$ converge to $(c,w_{c})$ in $C^{1}(\Omega)$ as
$i\to\infty$},$ (4.4)

where $c=F^{-1}(\theta/\alpha)$ and $w_{c}=\frac{c\alpha
d(c)}{\theta|\Omega|}\int_{\Omega}(m-c)\mathrm{d}x$. Here $F^{-1}(\cdot)$
denotes the inverse of $F(\cdot)$. Following the approach as that in the proof
of assertion (i), it suffices to show that
$(u^{\infty},w^{\infty})=(c,w_{c})$. From the first equation of (4.2), it
follows that $u^{\infty}=c_{1}$ for some constant $c_{1}>0$. Let
$\hat{w}^{i}:=\frac{w^{i}}{\|w^{i}\|_{L^{\infty}}}$. Then, $\hat{w}^{i}$
satisfies

$\begin{cases}\mu\Delta\hat{w}^{i}+\frac{\alpha
F(u^{i})-\theta}{d(u^{i})}\hat{w}^{i}=0,&\mbox{in}\;\Omega,\\\
\nabla\hat{w}^{i}\cdot n=0,&\mbox{on}\;\partial\Omega.\end{cases}$ (4.5)

We may assume that $\hat{w}^{i}\to\hat{w}^{\infty}$ in $C^{1}(\Omega)$ as
$i\to\infty$ (passing to a subsequence if necessary). Integrating the second
equation of (4.5) on $\Omega$ and letting $i\to\infty$, we have

$\int_{\Omega}\frac{\alpha
F(c_{1})-\theta}{d(c_{1})}\hat{w}^{\infty}\mathrm{d}x=0,$

which along with the facts $\hat{w}^{\infty}\geq 0$ and
$\|\hat{w}^{\infty}\|_{L^{\infty}}=1$ implies that
$c_{1}=F^{-1}(\theta/\alpha).$ This together with (4.2) yields that

$w^{\infty}=c_{2}\geq 0.$ (4.6)

Integrating the first equation of (4.5) on $\Omega$ and letting $i\to\infty$,
one has

$\int_{\Omega}\bigg{(}c(m-c)-\frac{F(c)}{d(c)}w^{\infty}\bigg{)}\mathrm{d}x=0,$

which combined with (4.6) indicates that $w^{\infty}=\frac{c\alpha
d(c)}{\theta|\Omega|}\int_{\Omega}(m-c)\mathrm{d}x$. Hence, (4.4) holds.

Define $\mathcal{L}_{1}:\mathbb{R}\times\bar{H}_{0}^{2}(\Omega)\times
H_{n}^{2}(\Omega)\times[0,+\infty)\to\mathbb{R}\times\bar{L}^{2}(\Omega)\times
L^{2}(\Omega)$ by

$\displaystyle\mathcal{L}_{1}(\xi,\zeta,w,\gamma)=$
$\displaystyle\begin{pmatrix}\frac{1}{|\Omega|}\int_{\Omega}\left((\xi+\zeta)(m(x)-\xi-\zeta)-\frac{F(\xi+\zeta)}{d(\xi+\zeta)}w\right)\mathrm{d}x\\\\[6.0pt]
\Delta\zeta+\gamma\left[(\xi+\zeta)(m(x)-\xi-\zeta)-\frac{F(\xi+\zeta)}{d(\xi+\zeta)}w-\frac{1}{|\Omega|}\int_{\Omega}\left((\xi+\zeta)(m(x)-\xi-\zeta)-\frac{F(\xi+\zeta)}{d(\xi+\zeta)}w\right)\mathrm{d}x\right]\\\
\mu\Delta w+\frac{\alpha F(\xi+\zeta)-\theta}{d(\xi+\zeta)}w\end{pmatrix},$

where $H_{n}^{2}(\Omega)=\\{u\in H^{2}(\Omega)|\nabla u\cdot
n=0\;\mbox{on}\;\partial\Omega\\}$, $\bar{H}_{0}^{2}(\Omega)=\\{u\in
H_{n}^{2}(\Omega)|\int_{\Omega}u\mathrm{d}x=0\\}$, and
$\bar{L}^{2}(\Omega)=\\{u\in L^{2}(\Omega)|\int_{\Omega}u\mathrm{d}x=0\\}$.
Then, one has

$\displaystyle
D_{(\xi,\zeta,w)}{\mathcal{L}_{1}}|_{(\xi,\zeta,w,\gamma)=(c,0,w_{c},0)}(\phi,\psi,\eta)$
$\displaystyle\quad\quad=\begin{pmatrix}\frac{1}{|\Omega|}\int_{\Omega}\left((m-2c)(\phi+\psi)-\frac{F(c)}{d(c)}\eta-\frac{F^{\prime}(c)d(c)-d^{\prime}(c)F(c)}{d^{2}(c)}w_{c}(\phi+\psi)\right)\mathrm{d}x\\\\[6.0pt]
\Delta\psi\\\ \mu\Delta\eta+\frac{\alpha
F^{\prime}(c)}{d(c)}w_{c}(\phi+\psi)\end{pmatrix},$

where $c=F^{-1}(\theta/\alpha)$ and $w_{c}=\frac{c\alpha
d(c)}{\theta|\Omega|}\int_{\Omega}(m-c)\mathrm{d}x$.

Claim:
$D_{(\xi,\zeta,w)}{\mathcal{L}_{1}}|_{(\xi,\zeta,w,\gamma)=(c,0,w_{c},0)}$ is
non-degenerate. To show this, it amounts to show that problem

$\begin{cases}\frac{1}{|\Omega|}\int_{\Omega}\left((m-2c)(\phi+\psi)-\frac{F(c)}{d(c)}\eta-\frac{F^{\prime}(c)d(c)-d^{\prime}(c)F^{\prime}(c)}{d^{2}(c)}w_{c}(\phi+\psi)\right)\mathrm{d}x=0,\\\
\Delta\psi=0,&\mbox{in}\;\Omega,\\\ \mu\Delta\eta+\frac{\alpha
F(c)-\theta}{d(c)}\eta+\frac{\alpha F^{\prime}(c)d(c)-(\alpha
F(c)-\theta)d^{\prime}(c)}{d^{2}(c)}w_{c}(\phi+\psi)=0,&\mbox{in}\;\Omega,\end{cases}$
(4.7)

only has the trivial solution $(0,0)$ in
$\mathbb{R}\times\bar{H}_{0}^{2}(\Omega)\times H_{n}^{2}(\Omega)$. From the
second equation of (4.7) and the definition of ${H}_{0}^{2}(\Omega)$, it
follows that $\psi\equiv 0$. Integrating the third equation of (4.7), one
obtains

$\int_{\Omega}\frac{\alpha F^{\prime}(c)}{d(c)}w_{c}\phi\mathrm{d}x=0,$

which together with $(H_{2})$, $(H_{3})$, $\phi\in\mathbb{R}$ and
$w_{c}=\frac{c\alpha d(c)}{\theta|\Omega|}\int_{\Omega}(m-c)\mathrm{d}x>0$,
implies $\phi=0.$ This combined with the first and third equations of (4.7)
gives that $\eta\equiv 0.$ So, the claim holds.

From the above claim and the implicit function theorem, it follows that there
exists a neighborhood
$\mathcal{U}_{1}\in\mathbb{R}\times\bar{H}_{0}^{2}(\Omega)\times
H_{n}^{2}(\Omega)$ containing $(c,0,w_{c})$ and a function
$(\xi_{\gamma},\zeta_{\gamma},w_{\gamma})$ defined for all $\gamma$ close to
zero such that if $(\xi,\zeta,w)\in\mathcal{U}_{1}$ is a solution of
$\mathcal{L}_{1}(\xi,\zeta,w,\gamma)=0$ for some $\gamma$ close to zero, then
we must have that $(u,w)=(\xi_{\gamma}+\zeta_{\gamma},w_{\gamma})$ is a
positive solution of (1.5). This along with (4.4) shows that (1.5) admits a
unique positive solution when $\epsilon$ is large, and hence completes the
proof. ∎

###  4.2. Large/small predator diffusion

In this section, we shall investigate the uniqueness and asymptotic profile of
solutions to (1.5) as $\mu\to\infty$ and $\mu\to 0$. First we define

$g(u)=\displaystyle\frac{\int_{\Omega}{F(u)}{d^{-1}(u)}\mathrm{d}x}{\int_{\Omega}{d^{-1}(u)}\mathrm{d}x},$

where $u\in C(\Omega;[0,+\infty))$. On top of assumptions $(H_{2})$ and
$(H_{3})$, we impose two additional assumptions:

  1. ($H_{4}$)

$g^{\prime}(u)>0$ for any $u\in C(\Omega;[0,+\infty))$, where $g^{\prime}(u)$
denotes the Frechet derivative.

  2. ($H_{5}$)

$\left(\frac{F(u)}{ud(u)}\right)^{\prime}\geq 0$ for any $u\geq 0$, where ′
denotes the differentiation with respect to $u$.

We give some examples where $(H_{4})$ or $(H_{5})$ holds. If $F(u)$ satisfies
$(H_{2})$ and $d(u)=e^{-ku}$ (or $(1+u)^{-k}$) for any $k\geq 0$, then
$(H_{4})$ holds, see Lemma 2.6 and Proposition 2.2 for the proof. If $F(u)=u$
and $d(u)$ satisfies $(H_{3})$, or $F(u)=\frac{u}{1+u}$ (Holling-II) and
$d(u)=(1+u)^{-k}$ (or $e^{-ku}$) with $k\geq 1$, then $(H_{5})$ holds.

######  Theorem 4.2.

Assume $(H_{1})$, $(H_{2})$, $(H_{3})$ and $(H_{4})$ hold. Then the following
results hold true.

  * (i)

If $\theta>\alpha g(\tilde{u})$, then system (1.5) doesn’t have any positive
solution when $\mu$ is large;

  * (ii)

If $0<\theta<\alpha g(\tilde{u})$ and $(H_{5})$ holds, then any positive
solution of system (1.5) will converge to $(u^{*},c^{*})$ in $C^{1}(\Omega)$
as $\mu\to\infty$, where $c^{*}$ is a positive constant and $(u^{*},c^{*})$ is
the unique positive solution of

$\begin{cases}\epsilon\Delta
u^{*}+u^{*}(m(x)-u^{*})-\frac{F(u^{*})}{d(u^{*})}c^{*}=0,&\mbox{in}\;\Omega,\\\
\nabla u^{*}\cdot n=0,&\mbox{on}\;\partial\Omega,\\\ \int_{\Omega}\frac{\alpha
F(u^{*})-\theta}{d(u^{*})}\mathrm{d}x=0.\end{cases}$ (4.8)

Moreover, if $F(u)=u$, $d(u)=e^{-ku}$ (or $(1+u)^{-k}$) with
$k\in[0,\frac{\alpha}{\theta}]$, then system (1.5) admits a unique positive
solution when $\mu$ is large.

###### Proof.

For assertion (i), suppose by contradiction that system (1.5) admits a
positive solution $(u_{i},w_{i})$ with $\mu=\mu_{i}$, where
$\mu_{i}\to+\infty$ as $i\to\infty$. Then $\mu_{i}$ and $(u_{i},w_{i})$
satisfy

$\begin{cases}\epsilon\Delta
u_{i}+u_{i}(m(x)-u_{i})-\frac{F(u_{i})}{d(u_{i})}w_{i}=0,&\mbox{in}\;\Omega,\\\
\mu_{i}\Delta w_{i}+\frac{\alpha
F(u_{i})-\theta}{d(u_{i})}w_{i}=0,&\mbox{in}\;\Omega,\\\ \nabla u_{i}\cdot
n=\nabla w_{i}\cdot n=0,&\mbox{on}\;\partial\Omega.\end{cases}$ (4.9)

Using Lemma 3.3, for any $i\geq 1$, one has

$\|u_{i}\|_{L^{\infty}(\Omega)}\leq
m_{\max}\;\;\mbox{and}\;\;\|w_{i}\|_{L^{\infty}(\Omega)}\leq c_{0}.$

Similar to the analysis as that in the proof of Theorem 4.1, one can deduce
from (4.9) that $(u_{i},w_{i})$, passing to a subsequence if necessary,
converges to some nonnegative function $(u_{\infty},w_{\infty})$ in
$C^{1}(\Omega)$ as $i\to\infty$, where $(u_{\infty},w_{\infty})$ satisfies (in
the weak sense)

$\begin{cases}\epsilon\Delta
u_{\infty}+u_{\infty}(m(x)-u_{\infty})-\frac{F(u_{\infty})}{d(u_{\infty})}w_{\infty}=0,&\mbox{in}\;\Omega,\\\
\Delta w_{\infty}=0,&\mbox{in}\;\Omega,\\\ \nabla u_{\infty}\cdot n=\nabla
w_{\infty}\cdot n=0,&\mbox{on}\;\partial\Omega.\end{cases}$ (4.10)

Therefore, there exists some constant $c_{0}\geq 0$ such that
$w_{\infty}=c_{0}$. We will show that $w_{\infty}=c_{0}\geq 0$ can’t occur.

If $w_{\infty}=0$, from the first equation of system (4.10), it follows that
$u_{\infty}=0$ or $u_{\infty}=\tilde{u}$. We first show that $u_{\infty}=0$
can’t occur. If not, assume $u_{\infty}=0$. Let
$\hat{u}_{i}=\frac{u_{i}}{\|u_{i}\|_{L^{\infty}}}$. Then $\hat{u}_{i}$
satisfies

$\begin{cases}\epsilon\Delta\tilde{u}_{i}+\tilde{u}_{i}\left(m(x)-u_{i}-\frac{F(u_{i})w_{i}}{u_{i}d(u_{i})}\right)=0,&\mbox{in}\;\Omega,\\\
\nabla\tilde{u}_{i}\cdot n=0,&\mbox{on}\;\partial\Omega.\end{cases}$ (4.11)

By assumption $(H_{2})$, one has $\lim\limits_{u\to
0}\frac{F(u)}{u}=F^{\prime}(0)$. Similar to the above analysis, one can deduce
that $\hat{u}_{i}\to\hat{u}_{\infty}\geq 0$ in $C^{1}(\Omega)$ as $i\to\infty$
(passing to a subsequence if necessary) and $\hat{u}_{\infty}$ satisfies
$\|\hat{u}_{\infty}\|_{L^{\infty}}=1$. Multiplying the first equation of
(4.11) by $\frac{1}{\hat{u}_{i}}$, integrating the resulting equation on
$\Omega$ and letting $i\to\infty$, one gets

$\int_{\Omega}m(x)\mathrm{d}x=-\epsilon\int_{\Omega}\frac{|\nabla\hat{u}_{\infty}|^{2}}{\hat{u}_{\infty}}\mathrm{d}x\leq
0,$

which contradicts the assumption $(H_{1})$. On the other hand, if
$u_{\infty}=\tilde{u}$, let
$\tilde{w}_{i}=\frac{w_{i}}{\|w_{i}\|_{L^{\infty}}}$. Then $\tilde{w}_{i}$
satisfies

$\begin{cases}\mu_{i}\Delta\tilde{w}_{i}+\frac{\alpha
F(u_{i})-\theta}{d(u_{i})}\tilde{w}_{i}=0,&\mbox{in}\;\Omega,\\\
\nabla\tilde{w}_{i}\cdot n=0,&\mbox{on}\;\partial\Omega.\end{cases}$ (4.12)

Similarly, one can derive that $\tilde{w}_{i}\to 1$ in $C^{1}(\Omega)$ as
$i\to\infty$ which is equivalent to $\theta>\alpha g(\tilde{u})$. Integrating
the first equation of (4.12) on $\Omega$ and letting $i\to\infty$, one obtains

$\int_{\Omega}\frac{\alpha F(\tilde{u})-\theta}{d(\tilde{u})}\mathrm{d}x=0,$

which contradicts $\int_{\Omega}\frac{\alpha
F(\tilde{u})-\theta}{d(\tilde{u})}\mathrm{d}x<0$. Therefore, $w_{\infty}=0$
can’t occur.

If $w_{\infty}=C>0$, integrating the second equation of system (4.9) on
$\Omega$ and letting $i\to\infty$, one has $\int_{\Omega}\frac{\alpha
F(u_{\infty})-\theta}{d(u_{\infty})}\mathrm{d}x=0$. This indicates that

$\theta=\alpha g(u_{\infty}).$ (4.13)

Combining systems (2.3) and (4.9) alongside the method of upper-lower
solutions, we have

$u_{\infty}<\tilde{u}\quad\mbox{on}\;\;\Omega,$

which combined with $(H_{4})$ gives that $g(u_{\infty})<g(\tilde{u}).$ This
together with (4.13) implies that

$\theta<\alpha g(\tilde{u}),$

which contradicts our assumption $\theta>\alpha g(\tilde{u})$ and the
assertion in statement (i) is proved.

Next we show the results stated in statement (ii). From lemma 2.1 (ii) and
Lemma 2.2 (ii), it follows that $(\tilde{u},0)$ is linearly unstable for any
$\mu>0$, which combined with Lemma 3.5 suggests that system (1.5) admits at
least one positive solution for any $\mu>0$. We next establish the following
claim.

Claim 1: any positive solution of system (1.5), denoted by
$(u_{\mu},w_{\mu})$, converges to $(u^{*},c^{*})$ in $C^{1}(\Omega)$ as
$\mu\to\infty$, where $c^{*}$ is a positive constant and $(u^{*},c^{*})$ is
the unique positive solution of (4.8). Similar to the argument as that in
proving statement (i), it suffices to show that system (4.8) admits a unique
positive solution $(u^{*},c^{*})$ with $c^{*}$ being a positive constant. To
this end, we introduce an auxiliary question

$\begin{cases}\epsilon\Delta
z+z\Big{(}m(x)-z-\frac{F(z)}{zd(z)}c\Big{)}=0,&\mbox{in}\;\Omega,\\\ \nabla
z\cdot n=0,&\mbox{on}\;\partial\Omega.\end{cases}$ (4.14)

Since $\lim\limits_{z\to 0}\frac{F(z)}{z}=F^{\prime}(0)$, using the assumption
$(H_{5})$, it is standard to show (cf. [3]) that for any $c\geq 0$, (4.14)
admits a unique positive solution denoted by $z_{c}$. By the method of upper-
lower solutions, one has that

$\mbox{if $c_{1}>c_{2}\geq 0$, then $z_{c_{1}}<z_{c_{2}}$ on $\bar{\Omega}$}.$
(4.15)

Clearly, we have

$z_{c}=\tilde{u}$ when $c=0$. (4.16)

Integrating the first equation of (4.14) on $\Omega$ for any $c\geq 0$, one
obtains

$\int_{\Omega}z_{c}\left(m(x)-z_{c}-\frac{F(z_{c})}{z_{c}d(z_{c})}c\right)\mathrm{d}x=0,$

which together with (4.15) yields that

$z_{c}\to 0\;\;\mbox{in}\;\;C(\bar{\Omega})\;\;\mbox{as}\;\;c\to+\infty.$
(4.17)

By the assumption $\theta<\alpha g(\tilde{u})$, we have from (4.16) and (4.17)
that

$\lim\limits_{c\to+\infty}\alpha g(z_{c})<\theta<\alpha g(\tilde{u}).$ (4.18)

This combined with the fact that $g(z_{c})$ depends continuously and
monotonically on $c$ shows that system (4.14) admits a unique positive
solution $u^{*}$ with $c=c^{*}$ such that $\theta=\alpha g(u^{*}).$ Hence,
Claim 1 holds.

Finally, we prove the second part of statement (ii). From now on, we assume
$F(u)=u$, and $d(u)=e^{-ku}$ with $k\in[0,\frac{\alpha}{\theta}]$. The other
case $d(u)=(1+u)^{-k}$ can be treated similarly. Define
$\mathcal{L}:H_{n}^{2}(\Omega)\times\mathbb{R}\times\bar{H}_{0}^{2}(\Omega)\times[0,+\infty)\to
L^{2}(\Omega)\times\mathbb{R}\times\bar{L}^{2}(\Omega)$ by

$\mathcal{L}(u,\xi,\zeta,\beta)=\begin{pmatrix}\epsilon\Delta
u+u(m(x)-u)-(\xi+\zeta)ue^{ku}\\\\[6.0pt]
\frac{1}{|\Omega|}\int_{\Omega}\left[(\alpha
u-\theta)(\xi+\zeta)e^{ku}\right]\mathrm{d}x\\\ \Delta\zeta+\beta[(\alpha
u-\theta)(\xi+\zeta)e^{ku}-\frac{1}{|\Omega|}\int_{\Omega}(\alpha
u-\theta)(\xi+\zeta)e^{ku}\mathrm{d}x]\end{pmatrix}.$

Then, we have

$\displaystyle
D_{(u,\xi,\zeta)}\mathcal{L}|_{(u,\xi,\zeta,\beta)=(u^{*},c^{*},0,0)}(\phi,\psi,\eta)$
$\displaystyle\quad\quad=\begin{pmatrix}\epsilon\Delta\phi+\phi(m(x)-2u^{*})-c^{*}(1+ku^{*})\phi
e^{ku^{*}}-u^{*}\psi e^{ku^{*}}-u^{*}e^{ku^{*}}\eta\\\\[6.0pt]
\frac{1}{|\Omega|}\int_{\Omega}\left[(\alpha+k\alpha
u^{*}-k\theta)c^{*}\phi+(\alpha u^{*}-\theta)\psi+(\alpha
u^{*}-\theta)\eta\right]e^{ku^{*}}\mathrm{d}x\\\ \Delta\eta\end{pmatrix},$

where $(u^{*},c^{*})$ is the unique positive solution of (4.8).

Claim 2:
$D_{(u,\xi,\zeta)}\mathcal{L}|_{(u,\xi,\zeta,\beta)=(u^{*},c^{*},0,0)}$ is
non-degenerate. It suffices to show that problem

$\begin{cases}\epsilon\Delta\phi+\phi(m(x)-2u^{*})-c^{*}(1+ku^{*})\phi
e^{ku^{*}}-u^{*}\psi e^{ku^{*}}-u^{*}e^{ku^{*}}\eta=0,&\mbox{in}\;\Omega,\\\
\frac{1}{|\Omega|}\int_{\Omega}\left[(\alpha+k\alpha
u^{*}-k\theta)c^{*}\phi+(\alpha u^{*}-\theta)\psi+(\alpha
u^{*}-\theta)\eta\right]e^{ku^{*}}\mathrm{d}x=0\\\
\Delta\eta=0,&\mbox{in}\;\Omega,\end{cases}$ (4.19)

only admits trivial solution in
$H_{n}^{2}(\Omega)\times\mathbb{R}\times\bar{H}_{0}^{2}(\Omega)$. The third
equation of (4.19) and the definition of ${H}_{0}^{2}(\Omega)$ suggest that
$\eta\equiv 0$. This along with the fact $\int_{\Omega}(\alpha
u^{*}-\theta)e^{ku^{*}}\mathrm{d}x$, and the second equation of (4.19) gives
that

$\int_{\Omega}(\alpha+k\alpha u^{*}-k\theta)\phi e^{ku^{*}}\mathrm{d}x=0.$
(4.20)

From the first equation of (4.8) and the Krein-Rutman Theorem (cf. [16, 30]),
one finds that

$\lambda_{1}(\epsilon,m-u^{*}-c^{*}e^{ku^{*}})=0$

which together with Lemma 2.1 yields that

$\lambda_{1}(\epsilon,m-2u^{*}-c^{*}(1+ku^{*})e^{ku^{*}})<0.$

This combined with the first equation of (4.19) further implies that $\phi<0$
(resp. $>0$) on $\overline{\Omega}$ if $\psi>0$ (resp. $<0$) on
$\overline{\Omega}$ with $\phi\equiv 0$ if $\psi\equiv 0$ on
$\overline{\Omega}$. This together with (4.20) shows that
$\phi\equiv\psi\equiv 0$. So, Claim 2 holds.

Based on the Claim 2, $\mathcal{L}(u^{*},c^{*},0,0)=0$, and the implicit
function theorem implies that there exists a neighborhood $\mathcal{U}\in
H_{n}^{2}(\Omega)\times\mathbb{R}\times\bar{H}_{0}^{2}(\Omega)$ containing
$(u^{*},c^{*},0)$ and a function $(u_{\beta},\xi_{\beta},\zeta_{\beta})$
defined for all $\beta$ close to zero such that if
$(u,\xi,\zeta)\in\mathcal{U}$ is a solution of
$\mathcal{L}(u,\xi,\zeta,\beta)=0$ for some $\beta$ close to zero, then we
must have that $(u,w)=(u_{\beta},\xi_{\beta}+\zeta_{\beta})$ is a positive
solution of (1.5). This together with Claim 1 shows that (1.5) admits a unique
positive solution when $\mu$ is large, which completes the proof. ∎

######  Theorem 4.3.

Suppose that $(H_{1})$, $(H_{2})$ and $(H_{3})$ hold. Fixing all the
parameters except $\mu$, assume that $0<\theta<\alpha F(\tilde{u}_{\max})$.
Then every positive solution $(u_{\mu},w_{\mu})$ of system (1.5) satisfies
that $u_{\mu}\to u_{0}>0$ uniformly on $\bar{\Omega}$ and $w_{\mu}\to
w_{0}\geq 0$ in $L^{p}(\Omega)$ for any $p\geq 1$ as $\mu\to 0$, where
$u_{0}\leq F^{-1}(\theta/\alpha)$ on $\bar{\Omega}$ and satisfies (in the weak
sense)

$\begin{cases}\epsilon\Delta
u_{0}+u_{0}(m(x)-u_{0})-\frac{F(u_{0})}{d(u_{0})}w_{0}=0,&\mbox{in}\;\Omega,\\\
\nabla u_{0}\cdot n=0,&\mbox{on}\;\partial\Omega,\end{cases}$ (4.21)

and

$w_{0}(x)=0$ a.e. in $\\{x\in\Omega|u_{0}(x)<F^{-1}(\theta/\alpha)\\}$ and
$|\\{x\in\Omega|w_{0}>0\\}|>0$. (4.22)

Moreover, the following uniqueness results hold.

  * (a)

If $m_{\min}\geq F^{-1}(\theta/\alpha)$, then the solution of (4.21) is unique
and given by

$u_{0}\equiv
F^{-1}(\theta/\alpha)\;\mbox{and}\;w_{0}(x)=\frac{\alpha}{\theta}\cdot
d\left(F^{-1}(\theta/\alpha)\right)F^{-1}(\theta/\alpha)\left[m(x)-F^{-1}(\theta/\alpha)\right]\;\mbox{a.e.
in}\;\Omega.$ (4.23)

  * (b)

If $m_{\min}<F^{-1}(\theta/\alpha)$ and $\Omega=(0,L)$, we have the following
result: if $m_{x}\geq 0$ (resp. $m_{x}\leq 0$) in $(0,L)$, then there exists
unique $y^{*}\in(0,L)$, where $y^{*}$ may be different for the cases
$m_{x}\geq 0$ and $m_{x}\leq 0$, such that

$u_{0}(x)=\begin{cases}F^{-1}(\theta/\alpha),&\mbox{if}\;x\in[y^{*},L]\
(\mathrm{resp}.\ x\in[0,y^{*}]),\\\
\tilde{u}_{y^{*}},&\mbox{if}\;x\in[0,y^{*})\ (\mathrm{resp}.\
x\in[y^{*},L]),\end{cases}$ (4.24)

and

$w_{0}(x)=\begin{cases}\frac{\alpha}{\theta}\cdot
d\left(F^{-1}(\theta/\alpha)\right)F^{-1}(\theta/\alpha)\left[m(x)-F^{-1}(\theta/\alpha)\right],&\mbox{a.e.
in}\;(y^{*},L)\ (\mathrm{resp}.\ in\ [0,y^{*}]),\\\ 0,&\mbox{a.e.
in}\;(0,y^{*})\ (\mathrm{resp}.\ in\ [y^{*},L]),\end{cases}$ (4.25)

where $\tilde{u}_{y^{*}}$ is the unique positive solution of

$\begin{cases}\epsilon u_{xx}+u(m(x)-u)=0,&\mbox{in}\;(0,y^{*}),\\\
u_{x}(0)=u_{x}(y^{*})=0\ (\mathrm{resp}.\ u_{x}(y^{*})=u_{x}(L)=0),\quad
u(y^{*})=F^{-1}(\theta/\alpha).\end{cases}$ (4.26)

###### Proof.

From Lemma 2.5 and Lemma 3.5, it follows that system (1.5) admits at least one
positive solution denoted by $(u_{\mu},w_{\mu})$ when $\mu$ is small. By Lemma
3.3, one obtains that

$(u_{\mu},w_{\mu})\to(u_{0},w_{0})$ in $L^{p}(\Omega)$ for any $p\geq 1$, as
$\mu\to 0$. (4.27)

Clearly, $u_{0}\geq 0$ and $w_{0}\geq 0$ on $\bar{\Omega}$. Moreover, applying
the elliptic regularity (cf. [20]) and the Sobolev imbedding theorem, we may
assume that $u_{\mu}\to u_{0}$ in $C^{1}(\bar{\Omega})$ as $\mu\to 0$ and
$(u_{0},w_{0})$ satisfies (4.21). Using the strong maximum principle to
(4.21), we obtain that $u_{0}>0$ on $\bar{\Omega}$ or $u_{0}\equiv 0$. If
$u_{0}\equiv 0$, similar to the proofs as those for Theorem 4.2, one can
deduce that $\lambda_{1}(\epsilon,m)=0$, which is impossible due to assumption
$(H_{1})$. So, $u_{0}>0$ on $\bar{\Omega}$. The equation for $w_{\mu}$
suggests that

$\lambda_{1}\left(\mu,\frac{\alpha
F(u_{\mu})-\theta}{d(u_{\mu})}\right)=0,\quad\mbox{for any small}\;\;\mu>0,$

which combined with Lemma 2.1 (ii) implies that

$0=\lim\limits_{\mu\to 0}\lambda_{1}\left(\mu,\frac{\alpha
F(u_{\mu})-\theta}{d(u_{\mu})}\right)=\max\limits_{x\in\bar{\Omega}}\frac{\alpha
F(u_{0})-\theta}{d(u_{0})}.$

This together with assumptions $(H_{2})$ and $(H_{3})$ yields that

$u_{0}\leq
F^{-1}(\theta/\alpha)\;\mbox{on}\;\bar{\Omega}\;\mbox{and}\;u_{0}(x)=F^{-1}(\theta/\alpha)\;\mbox{for
some $x\in\bar{\Omega}$}.$ (4.28)

Integrating the equation that $w_{\mu}$ satisfies on $\Omega$, one has

$\int_{\Omega}\frac{\alpha
F(u_{\mu})-\theta}{d(u_{\mu})}w_{\mu}\mathrm{d}x=0,\quad\mbox{for any
small}\;\;\mu>0.$

Sending $\mu\to 0$, we have $\int_{\Omega}\frac{\alpha
F(u_{0})-\theta}{d(u_{0})}w_{0}\mathrm{d}x=0,$ which shows that

$w_{0}=0$ a.e. in $\\{x\in\Omega|u_{0}(x)<F^{-1}(\theta/\alpha)\\}$. (4.29)

We proceed to prove that $|\\{x\in\Omega|w_{0}>0\\}|>0$. If not, assume
$w_{0}=0$ a.e. in $\Omega$. Combining (4.21) and $u_{0}>0$, one has
$u_{0}=\tilde{u}$ on $\bar{\Omega}$. Then, by the assumption $0<\theta<\alpha
F(\tilde{u}_{\max})$ and $(H_{2})$, we have

$\max\limits_{x\in\bar{\Omega}}u_{0}=\tilde{u}_{\max}>F^{-1}(\theta/\alpha),$

which contradicts (4.28). Therefore, $|\\{x\in\Omega|w_{0}>0\\}|>0$.

Next we consider the case $m_{\min}\geq F^{-1}(\theta/\alpha)$. It is easy to
verify that $(u_{0},w_{0})$ given in (4.23) is well-defined and it satisfies
(4.21), (4.22) and $0<u_{0}\leq F^{-1}(\theta/\alpha)$ on $\bar{\Omega}$. So,
it suffices to show that (4.21) admits a unique non-negative solution
$(u_{0},w_{0})$ which satisfies (4.22) and $0<u_{0}\leq F^{-1}(\theta/\alpha)$
on $\bar{\Omega}$. If not, assume that (4.21) admits a non-negative solution
$(u_{1},w_{1})\neq(u_{0},w_{0})$ (see (4.23)) satisfying (4.22) and
$0<u_{1}\leq F^{-1}(\theta/\alpha)$ on $\bar{\Omega}$. Then,
$u_{1}\leq,\not\equiv F^{-1}(\theta/\alpha)$ in $\Omega$ and there exists some
$x_{0}\in\Omega$ such that $u_{1}(x_{0})<F^{-1}(\theta/\alpha)$. By
continuity, one can find some neighborhood $x_{0}\in\Omega_{1}\subset\Omega$
such that

$u_{1}<F^{-1}(\theta/\alpha)\;\mbox{in}\;\Omega_{1}\quad\mbox{and}\quad
u_{1}(x)=F^{-1}(\theta/\alpha),\;\mbox{for}\;x\in\partial\Omega_{1}\cap\Omega,$

which together with $u_{1}\leq,\not\equiv F^{-1}(\theta/\alpha)$ in $\Omega$
further implies that $\nabla
u_{1}(x)=0,\;\mbox{for}\;x\in\partial\Omega_{1}\cap\Omega.$ By (4.22), one has
$w_{1}=0$ a.e. in $\Omega_{1}$. Therefore, $u_{1}$ satisfies

$\begin{cases}\epsilon\Delta u+u(m(x)-u)=0,&\mbox{in}\;\Omega_{1},\\\ \nabla
u\cdot n=0,&\mbox{on}\;\partial\Omega_{1}.\end{cases}$

However the maximum principle applied to the above equations yields that

$m_{\min}\leq\min\limits_{x\in\Omega_{1}}m(x)\leq\min\limits_{x\in\Omega_{1}}u_{1}(x)<F^{-1}(\theta/\alpha),$

which contradicts our assumption $m_{\min}\geq F^{-1}(\theta/\alpha)$. These
facts complete the proof of the first part.

Finally, we consider the scenario $m_{\min}<F^{-1}(\theta/\alpha)$ and
$\Omega=(0,L)$. We shall only prove the case $m_{x}\geq 0$ and the case
$m_{x}\leq 0$ can be shown similarly. For any $y\in(0,L]$, we consider an
auxiliary problem

$\begin{cases}\epsilon u_{xx}+u(m(x)-u)=0,&\mbox{in}\;(0,y),\\\
u_{x}(0)=u_{x}(y)=0.\end{cases}$ (4.30)

Without loss of generality, we assume $m(0)>0$ (if $m(0)<0$, the method is
still valid). It is well-known that (4.30) admits a unique positive solution
denoted by $\tilde{u}_{y}$ (see Proposition 2.1). Moreover, if
$m_{x}\geq,\not\equiv 0$ in $(0,y)$, then $\frac{d\tilde{u}_{y}}{dx}>0$ in
$(0,y)$ (see the proof of Lemma 2.8); while if $m_{x}\equiv 0$ in $(0,y)$,
then $u\equiv m(0)$ in $(0,y)$.

Claim A: if $0<y_{1}<y_{2}\leq L$, then
$\tilde{u}_{y_{2}}(y_{2})\geq\tilde{u}_{y_{1}}(y_{1})$, where “=” holds if and
only if $m_{x}\equiv 0$ in $(0,y_{2})$. If $m_{x}\equiv 0$ in $(0,y_{2})$,
then $\tilde{u}_{y_{2}}(y_{2})=\tilde{u}_{y_{1}}(y_{1})=m(0)$. If
$m_{x}\geq,\not\equiv 0$ in $(0,y_{2})$, then
$\frac{d\tilde{u}_{y_{2}}}{dx}>0$ in $(0,y_{2})$ by Lemma 2.8. Thus,
$\tilde{u}_{y_{2}}$ restricted in $(0,y_{1})$ is a strictly upper-solution of
(4.30) with $y=y_{1}$ due to
$\frac{d\tilde{u}_{y_{2}}(y_{1})}{dx}>0=\frac{d\tilde{u}_{y_{1}}(y_{1})}{dx}$.
Moreover let $\phi_{1}^{*}(\epsilon,m)>0$ be the principal eigenfunction of
the following eigenvalue problem

$\begin{cases}\epsilon\phi_{xx}+m\phi=\lambda\phi,&\mbox{in}\;(0,y_{1}),\\\
\phi_{x}(0)=\phi_{x}(y_{1})=0.\end{cases}$

Then one can choose sufficiently small enough $\sigma>0$ such that
$\sigma\phi_{1}^{*}(\epsilon,m)<\tilde{u}_{y_{2}}$ in $(0,y_{1})$ and
$\sigma\phi_{1}^{*}(\epsilon,m)$ is a strictly lower-solution of (4.30) with
$y=y_{1}$. Therefore, from the methods of upper-lower solution, it follows
that

$\tilde{u}_{y_{2}}(y_{1})>\tilde{u}_{y_{1}}(y_{1})$

which together with $\frac{d\tilde{u}_{y_{2}}}{dx}>0$ in $(0,y_{2})$ implies
that

$\tilde{u}_{y_{2}}(y_{2})>\tilde{u}_{y_{1}}(y_{1}).$

Hence Claim A is proved. On the other hand, one observes that

$\lim\limits_{y\to
0}\tilde{u}_{y}(y)=m(0)=m_{\min}<F^{-1}\left(\frac{\theta}{\alpha}\right)\quad\mbox{and}\quad\lim\limits_{y\to
L}\tilde{u}_{y}(y)=\tilde{u}(L)=\tilde{u}_{\max}>F^{-1}\left(\frac{\theta}{\alpha}\right),$

which together with Claim A implies that there exists unique $y^{*}$ such that
$\tilde{u}_{y^{*}}$ satisfies (4.26). By
$\frac{d\tilde{u}_{y^{*}}}{dx}(y^{*})=0$ and
$\frac{d\tilde{u}_{y^{*}}}{dx}(x)>0$ in $(0,y^{*})$, one has
$\frac{d^{2}\tilde{u}_{y^{*}}}{dx^{2}}(y^{*})\leq 0$, which substituted into
the first equation of (4.26) further gives that

$m(y^{*})\geq\tilde{u}_{y^{*}}(y^{*})=F^{-1}\left(\frac{\theta}{\alpha}\right).$

Therefore, $(u_{0},w_{0})$ defined in (4.24) and (4.25) satisfies (4.21),
(4.22), and $0<u_{0}\leq F^{-1}\left(\frac{\theta}{\alpha}\right)$. To
complete the proof, it suffices to show that (4.21) admits a unique non-
negative solution $(u_{0},w_{0})$ which satisfies (4.22) and $0<u_{0}\leq
F^{-1}\left(\frac{\theta}{\alpha}\right)$ on $[0,L]$. Assume that (4.21)
admits another non-negative solution $(u_{2},w_{2})$ which satisfies (4.22)
and $0<u_{2}\leq F^{-1}\left(\frac{\theta}{\alpha}\right)$ on $[0,L]$.

Claim B: $u_{2}=u_{0}$ on $[0,L]$, where $u_{0}$ is given in (4.24). We first
prove that $u_{2}=u_{0}$ on $[0,y^{*}]$. We note here that
$u_{0}=\tilde{u}_{y^{*}}$ on $[0,y^{*}]$. It suffices to consider two cases

$(1)\;\;u_{2}(0)<F^{-1}\left(\frac{\theta}{\alpha}\right)\;\;\mbox{and}\;\;(2)\;\;u_{2}(0)=F^{-1}\left(\frac{\theta}{\alpha}\right).$

For case (1), it suffices to consider two cases

$\text{(1a)}\;\mbox{$\exists$ $x_{1}\in(0,L)$ such that
$u_{2}(x_{1})=F^{-1}\left(\frac{\theta}{\alpha}\right)$}\;,\
\mbox{and}\;\mathrm{(1b)}\;u_{2}<F^{-1}\left(\frac{\theta}{\alpha}\right)\;\mbox{in}\;(0,L).$

For case (1a), we define
$x_{2}=\inf\limits_{x\in[0,L]}\\{u_{2}(x)=F^{-1}\left(\frac{\theta}{\alpha}\right)\\}$.
Then, we have

$u_{2}<F^{-1}\left(\frac{\theta}{\alpha}\right)\;\;\mbox{on}\;[0,x_{2})\;\;\mbox{and}\;\;u_{2}(x_{2})=F^{-1}\left(\frac{\theta}{\alpha}\right).$

By (4.22), we have $w_{2}=0$ a.e. in $(0,x_{2})$ and $u_{2}$ satisfies

$\begin{cases}\epsilon u_{xx}+u(m(x)-u)=0,&\mbox{in}\;(0,x_{2}),\\\
u_{x}(0)=u_{x}(x_{2})=0,\quad
u(x_{2})=F^{-1}\left(\frac{\theta}{\alpha}\right).\end{cases}$

Claim A shows that $x_{2}=y^{*}$ and $u_{2}=\tilde{u}_{y^{*}}$ on $[0,y^{*}]$.
For case (1b), by (4.21) and (4.22), one has that $w_{2}=0$ a.e. in $(0,L)$
and $u_{2}=\tilde{u}$, which is impossible due to the fact that
$\tilde{u}(L)>F^{-1}\left(\frac{\theta}{\alpha}\right)$.

For case (2), it suffices to consider two cases

$\text{(2a)}\;u_{2}\equiv
F^{-1}\left(\frac{\theta}{\alpha}\right)\;\mbox{on}\;[0,L]\;,\
\mbox{and}\;\text{(2b)}\;\mbox{$\exists$ $x_{3}\in(0,L)$ such that
$u_{2}(x_{3})<F^{-1}\left(\frac{\theta}{\alpha}\right)$}.$

For case (2a), (4.21) tells us that $w_{0}(0)<0$ due to the assumption
$m(0)=m_{\min}<F^{-1}\left(\frac{\theta}{\alpha}\right)$. This is impossible.
For case (2b), we define

$x_{4}=\sup\limits_{x\in[0,x_{3}]}\left\\{u_{2}(x)=F^{-1}\left(\frac{\theta}{\alpha}\right)\right\\}$

and

$x_{5}=\inf\limits_{x\in[x_{3},L]}\left\\{u_{2}(x)=F^{-1}\left(\frac{\theta}{\alpha}\right)\right\\}.$

We note here that $x_{4}\in(0,x_{3})$ and $x_{5}\in(x_{3},L]$. Similarly, one
obtains that $w_{2}=0$ a.e. in $(x_{4},x_{5})$ and $u_{2}$ satisfies

$\begin{cases}\epsilon u_{xx}+u(m(x)-u)=0,&\mbox{in}\;(x_{4},x_{5}),\\\
u_{x}(x_{4})=u_{x}(x_{5})=0,\quad
u(x_{4})=F^{-1}\left(\frac{\theta}{\alpha}\right)\end{cases}$ (4.31)

which admits a unique solution which is non-decreasing with respect to $x$.
This contradicts the fact
$u_{2}<F^{-1}\left(\frac{\theta}{\alpha}\right)\mbox{in}\;(x_{4},x_{5})$ by
the definition of $x_{4}$ and $x_{5}$. Hence, $u_{2}=\tilde{u}_{y^{*}}=u_{0}$
on $[0,y^{*}]$.

Finally, it remains to show that $u_{2}\equiv
F^{-1}\left(\frac{\theta}{\alpha}\right)$ on $[y^{*},L]$. Recall that
$u_{2}(y^{*})=F^{-1}\left(\frac{\theta}{\alpha}\right)$. Arguing by
contradiction, we assume that $\exists$ $x_{6}\in(0,L)$ such that
$u_{2}(x_{6})<F^{-1}\left(\frac{\theta}{\alpha}\right)$. Define

$x_{7}=\sup\limits_{x\in[y^{*},x_{6}]}\left\\{u_{2}(x)=F^{-1}\left(\frac{\theta}{\alpha}\right)\right\\}$

and

$x_{8}=\inf\limits_{x\in[x_{6},L]}\left\\{u_{2}(x)=F^{-1}\left(\frac{\theta}{\alpha}\right)\right\\}.$

Then, similar to the analysis in case (2b), one can deduce a contradiction.
Therefore, $u_{2}\equiv F^{-1}\left(\frac{\theta}{\alpha}\right)$ on
$[y^{*},L]$. This completes the proof of Claim B and hence the proof of
Theorem 4.3. ∎

Acknowledgement. The research of D. Tang was supported by the National Natural
Science Foundation of China (No. 11901596), Science and Technology Program of
Guangzhou (No. 202102020772), and the Fundamental Research Funds for the
Central Universities (No. 2021qntd20). Z.-A. Wang is partially supported by a
grant from the NSFC/RGC Joint Research Scheme sponsored by the Research Grants
Council of Hong Kong and the National Natural Science Foundation of China
(Project No. $\mathrm{N}_{-}$PolyU509/22).

## References

  * [1] N.D. Alikakos.  An application of the invariance principle to reaction-diffusion equations.  J. Diff. Eqns., 33(2):201–225, 1979.
  * [2] I. Averill, K.-Y. Lam, and Y. Lou.  The role of advection in a two-species competition model: a bifurcation approach.  Mem. Amer. Math. Soc., 245(1161):v+117, 2017.
  * [3] R.S. Cantrell and C. Cosner.  Spatial ecology via reaction-diffusion equations.  John Wiley & Sons, 2004.
  * [4] R.S. Cantrell, C. Cosner, and Y. Lou.  Movement toward better environments and the evolution of rapid diffusion.  Math. Biosci., 204(2):199–214, 2006.
  * [5] R.S. Cantrell, C. Cosner, and Y. Lou.  Evolution of dispersal and the ideal free distribution.  Math. Biosci. Eng., 7(1):17–36, 2010.
  * [6] E. Cho and Y.-J. Kim.  Starvation driven diffusion as a survival strategy of biological organisms.  Bull. Math. Biol., 75(5):845–870, 2013.
  * [7] W. Choi, K. Kim, and I. Ahn.  Predator-prey models with prey-dependent diffusion on predators in spatially heterogeneous habitat.  J. Math. Anal. Appl., 99:127130, 2023.
  * [8] J. Chung, Y.-J. Kim, O. Kwon, and C.-W. Yoon.  Biological advection and cross-diffusion with parameter regimes.  AIMS Math., 4(6):1721–1744, 2019.
  * [9] C. Cosner.  Reaction-diffusion-advection models for the effects and evolution of dispersal.  Discrete Contin. Dyn. Syst., 34:1701–1745, 2014.
  * [10] E.N. Dancer.  On the indices of fixed points of mappings in cones and applications.  J. Math. Anal. Appl., 91(1):131–151, 1983.
  * [11] E.N. Dancer.  On positive solutions of some pairs of differential equations.  Trans. Amer. Math. Soc., 284(2):729–743, 1984.
  * [12] D.L. DeAngelis, W.-M. Ni, and B. Zhang.  Dispersal and spatial heterogeneity: single species.  J. Math. Biol., 72(1):239–254, 2016.
  * [13] U. Dieckmann, B. O’Hara, and W. Weisser.  The evolutionary ecology of dispersal.  Trends Ecology & Evolution, 14(3):88–90, 1999.
  * [14] W. Ding, W.and Huang.  Traveling wave solutions for some classes of diffusive predator-prey models.  J. Dynam. Differential Equations, 28(3-4):1293–1308, 2016.
  * [15] J. Dockery, V. Hutson, K. Mischaikow, and M. Pernarowski.  The evolution of slow dispersal rates: a reaction diffusion model.  J. Math. Biol., 37(1):61–83, 1998.
  * [16] Y. Du.  Order structure and topological methods in nonlinear partial differential equations: Vol. 1: Maximum principles and applications, volume 2.  World Scientific, 2006.
  * [17] Y. Du and Y. Lou.  Some uniqueness and exact multiplicity results for a predator-prey model.  Trans. Amer. Math. Soc., 349(6):2443–2475, 1997.
  * [18] Y. Du and J. Shi.  Allee effect and bistability in a spatially heterogeneous predator-prey model.  Trans. Amer. Math. Soc., 359(9):4557–4593, 2007.
  * [19] X. Fu, L.-H. Tang, C. Liu, J.-D. Huang, T. Hwa, and P. Lenz.  Stripe formation in bacterial systems with density-suppressed motility.  Phys. Rev. Lett., 108:198102, 2012.
  * [20] D. Gilbarg and N.S. Trudinger.  Elliptic Partial Differential Equations of Second Order. Springer-Verlag.  Berlin, 2001.
  * [21] X. He and W.-M. Ni.  Global dynamics of the Lotka-Volterra competition-diffusion system: Diffusion and spatial heterogeneity I.  Comm. Pure Appl. Math., 69(5):981–1014, 2016.
  * [22] T. Hiltunen and J. Laakso.  The relative importance of competition and predation in environment characterized by resource pulses-an experimental test with a microbial community.  BMC ecology, 13(1):1–8, 2013.
  * [23] R.D. Holt.  Predation, apparent competition, and the structure of prey communities.  Theor. Popu. Biol., 12(2):197–229, 1977.
  * [24] W. Huang.  Traveling wave solutions for a class of predator-prey systems.  J. Dynam. Differential Equations, 24(3):633–644, 2012.
  * [25] H.-Y. Jin and Z.A. Wang.  Global dynamics and spatio-temporal patterns of predator-prey systems with density-dependent motion.  European J. Appl. Math., 32(4):652–682, 2021.
  * [26] A. Jüngel.  Diffusive and nondiffusive population models.  In Mathematical Modeling of Collective Behavior in Socio-Economic and Life Sciences, pages 397–425. Springer, 2010.
  * [27] P. Kareiva and G. Odell.  Swarms of predators exhibit ”preytaxis” if individual predators use area-restricted search.  Amer. Nat., 130(2):233–270, 1987.
  * [28] E.F. Keller and L.A. Segel.  Model for chemotaxis.  J. Theor. Biol., 30(2):225–234, 1971.
  * [29] E.F. Keller and L.A. Segel.  Traveling bands of chemotactic bacteria: A theoretical analysis.  J. Theor. Biol., 30:377–380, 1971.
  * [30] M.G. Krein and M.A. Rutman.  Linear operators leaving invariant a cone in a banach space.  Uspekhi Matem. Nauk, 3(1):3–95, 1948.
  * [31] K.-Y. Lam and W.-M. Ni.  Uniqueness and complete dynamics in heterogeneous competition-diffusion systems.  SIAM J. Appl. Math., 72(6):1695–1712, 2012.
  * [32] L. Li.  Coexistence theorems of steady states for predator-prey interacting systems.  Trans. Amer. Math. Soc., 305(1):143–166, 1988.
  * [33] S. Li, J. Wu, and Y. Dong.  Effects of degeneracy and response function in a diffusion predator-prey model.  Nonlinearity, 31(4):1461–1483, 2018.
  * [34] S. Li, J. Wu, and Y. Dong.  Uniqueness and stability of positive solutions for a diffusive predator-prey model in heterogeneous environment.  Calc. Var. Partial Differential Equations, 58(3):Paper No. 110, 42, 2019.
  * [35] Y. Lou.  On the effects of migration and spatial heterogeneity on single and multiple species.  J. Diff. Eqns., 223(2):400–426, 2006.
  * [36] Y. Lou.  Some challenging mathematical problems in evolution of dispersal and population dynamics.  In Tutorials in Mathematical Biosciences IV, pages 171–205. Springer, 2008.
  * [37] Y. Lou and W.-M. Ni.  Diffusion, self-diffusion and cross-diffusion.  J. Diff. Eqns., 131(1):79–131, 1996.
  * [38] N. Maag, G. Cozzi, T. Clutton-Brock, and A. Ozgul.  Density-dependent dispersal strategies in a cooperative breeder.  Ecology, 99(9):1932–1941, 2018.
  * [39] E. Matthysen.  Density-dependent dispersal in birds and mammals.  Ecography, 28(3):403–416, 2005.
  * [40] Y.-Y. Mi, C. Song, and Z.-C. Wang.  Global boundedness and dynamics of a diffusive predator-prey model with modified Leslie-Gower functional response and density-dependent motion.  Commun. Nonlinear Sci. Numer. Simul., 119:Paper No. 107115, 21 pp, 2023.
  * [41] K. Nakashima and Y. Yamada.  Positive steady states for prey-predator models with cross-diffusion.  Adv. Differ. Eqns., 1(6):1099–1122, 1996.
  * [42] W.-M. Ni.  The mathematics of diffusion, volume 82 of CBMS-NSF Regional Conference Series in Applied Mathematics.  Society for Industrial and Applied Mathematics (SIAM), Philadelphia, PA, 2011.
  * [43] A. Okubo and S.A. Levin.  Diffusion and ecological problems: modern perspectives, volume 14.  Springer, 2001.
  * [44] R. Ron, O. Fragman-Sapir, and R. Kadmon.  Dispersal increases ecological selection by increasing effective community size.  Proc. Nat. Acad. Sci., 115(44):11280–11285, 2018.
  * [45] W. Ruan and W. Feng.  On the fixed point index and multiple steady-state solutions of reaction-diffusion systems.  Differential Integral Equations, 8(2):371–392, 1995.
  * [46] N. Shigesada, K. Kawasaki, and E. Teramoto.  Spatial segregation of interacting species.  J. Theor. Biol., 79(1):83–99, 1979.
  * [47] J.B. Shurin and E.G. Allen.  Effects of competition, predation, and dispersal on species richness at local and regional scales.  Amer Nat., 158(6):624–637, 2001.
  * [48] M.J. Smith, J.A. Sherratt, and X. Lambin.  The effects of density-dependent dispersal on the spatiotemporal dynamics of cyclic populations.  J. Theor. Biol., 254(2):264–274, 2008.
  * [49] D. Tang and Z.A. Wang.  Population dynamics with resource-dependent dispersal: single- and two-species models.  J. Math. Biol., 86(2):Paper No. 23, 42 pp, 2023.
  * [50] J.M.J Travis, D.J Murrell, and C. Dytham.  The evolution of density–dependent dispersal.  Proc. R Soc. Lond. Series B., 266(1431):1837–1842, 1999.
  * [51] Y.V. Tyutyunov, L.I. Titova, and I.N. Senina.  Prey-taxis destabilizes homogeneous stationary state in spatial gause-kolmogorov-type model for predator-prey system.  Ecological complexity, 31:170–180, 2017.
  * [52] F. Yi, J. Wei, and J. Shi.  Bifurcation and spatiotemporal patterns in a homogeneous diffusive predator-prey system.  J. Diff. Eqns, 246(5):1944–1977, 2009.
  * [53] B. Zhang, A. Kula, K. Mack, L. Zhai, A.L. Ryce, W.-M. Ni, D.L. DeAngelis, and J.D. Van Dyken.  Carrying capacity in a heterogeneous environment with habitat connectivity.  Ecology Letters, 20(9):1118–1128, 2017.