# On the solutions to weakly coupled system of $\boldsymbol{k_{i}}$-Hessian
equations

JingWen Ji  College of Mathematics and Statistics, Nanjing University of
Information Science and Technology, Nanjing 210094, P.R. China
<EMAIL_ADDRESS>,  Feida Jiang  School of Mathematics and Shing-Tung
Yau Center of Southeast University, Southeast University, Nanjing 211189, P.R.
China<EMAIL_ADDRESS>and  BaoHua Dong  College of Mathematics and
Statistics, Nanjing University of Information Science and Technology, Nanjing
210094, P.R. China<EMAIL_ADDRESS>
###### Abstract.

In this paper, the existence and multiplicity of nontrivial radial convex
solutions to general coupled system of $k_{i}$-Hessian equations in a unit
ball are studied via a fixed-point theorem. In particular, we obtain the
uniqueness of nontrivial radial convex solution and nonexistence of nontrivial
radial $\boldsymbol{k}$-admissible solution to a power-type system coupled by
$k_{i}$-Hessian equations in a unit ball. Moreover, using a generalized Krein-
Rutman theorem, the existence of $\boldsymbol{k}$-admissible solutions to an
eigenvalue problem in a general strictly $(k-1)$-convex domain is also
obtained.

###### Key words and phrases:

$k_{i}$-Hessian equations; existence; multiplicity; uniqueness; nonexistence;
eigenvalue problem.

###### 2010 Mathematics Subject Classification:

35J70, 35J60, 35J96, 35J57, 34A34

This work was supported by the National Natural Science Foundation of China
(No. 11771214, No. 11901303).

*corresponding author

##  1\. Introduction

In this paper, we consider the existence and multiplicity of nontrivial radial
$\boldsymbol{k}$-admissible solutions to the coupled system of the following
$k_{i}$-Hessian equations:

(1.1) $\left\\{\begin{aligned}
&S_{k_{1}}\left(D^{2}u_{1}\right)=f_{1}\left(|x|,-u_{2}\right),&{\rm in}\
B,\\\ &S_{k_{2}}\left(D^{2}u_{2}\right)=f_{2}\left(|x|,-u_{3}\right),&{\rm
in}\ B,\\\ &\qquad\qquad\quad\vdots\\\
&S_{k_{n-1}}\left(D^{2}u_{n-1}\right)=f_{n-1}\left(|x|,-u_{n}\right),&{\rm
in}\ B,\\\
&S_{k_{n}}\left(D^{2}u_{n}\right)=f_{n}\left(|x|,-u_{1}\right),&{\rm in}\
B,\\\ &u_{i}=0,i=1,\ldots,n,&{\rm on}\ \partial B,\end{aligned}\right.$

where $\boldsymbol{k}=(k_{1},\ldots,k_{n})$, $k_{i}=1,\ldots,N\
(i=1,\ldots,n)$, $B=\left\\{x\in\mathbb{R}^{N}:|x|<1\right\\}$ is a unit ball,
$n\geq 2$ and $N\geq 2$ are integers. The nonlinearities $f_{i}\
(i=1,\ldots,n)$ satisfy

$({\rm F}):f_{i}\in C\left([0,1]\times[0,+\infty),[0,+\infty)\right),\quad
i=1,\ldots,n$

and each $f_{i}$ is not identical to zero.

The $k$-Hessian operator $S_{k}$ is defined by the $k$-th elementary symmetric
function of eigenvalues of $D^{2}u$, i.e.

$S_{k}\left(D^{2}u\right):=S_{k}\left(\lambda(D^{2}u)\right)=\sum_{1\leq
i_{1}<\cdots<i_{k}\leq N}\lambda_{i_{1}}\cdots\lambda_{i_{k}},\quad
k=1,\ldots,N,$

where $\lambda(D^{2}u)=(\lambda_{1},\ldots,\lambda_{N})$ is the vector of
eigenvalues of $D^{2}u=\left[\frac{\partial^{2}u}{\partial x_{i}\partial
x_{j}}\right]_{n\times n}$, (see [21, 25] for instance). Notice that when
$k=1$, the Hessian operator reduces to the classical Laplace operator
$S_{1}(D^{2}u)=\sum_{i=1}^{N}\lambda_{i}=\Delta u$. When $k=N$, the Hessian
operator is the Monge-Ampère operator
$S_{N}(D^{2}u)=\prod_{i=1}^{N}\lambda_{i}=\det(D^{2}u)$. In fact, the
$k$-Hessian operator can be regarded as an extension of the Laplace operator
and the Monge-Ampère operator. When $k\geq 2$, the $k$-Hessian operator is a
fully nonlinear operator.

Let $u\in C^{2}(\Omega)$ and
$\sigma_{k}=\left\\{\lambda\in\mathbb{R}^{N}:S_{l}(\lambda)>0,\forall
l=1,\ldots,k\right\\}$ be a convex cone and its vertex be the origin. If
$\lambda(D^{2}u)\in\overline{\sigma}_{k}\ (\sigma_{k})$, $u$ is said to be
$k$-convex (uniformly $k$-convex) in $\Omega$. Equivalently, if
$\lambda(-D^{2}u)\in\overline{\sigma}_{k}\ (\sigma_{k})$, $u$ is $k$-concave
(uniformly $k$-concave) in $\Omega$. We say $u\in C^{2}(\Omega)\cup
C^{0}(\overline{\Omega})$ is $k$-admissible if
$\lambda(D^{2}u)\in\overline{\sigma}_{k}$. In particular, an $N$-admissible
function $u$ satisfying $\lambda(D^{2}u)\in\overline{\sigma}_{N}$ is said to
be convex. It is clear that
$\sigma_{N}\subset\cdots\subset\sigma_{k}\subset\cdots\subset\sigma_{1}$,
which implies that convex functions are contained in $k$-admissible functions.
Actually, we know from [2] that for a $k$-Hessian equation, it is elliptic
when restricted to $k$-admissible functions. For
$\boldsymbol{k}=(k_{1},\ldots,k_{n})$, $\boldsymbol{u}=(u_{1},\ldots,u_{n})$,
if $u_{i}$ is $k_{i}$-admissible and satisfies (1.1) for all $i=1,\ldots,n$,
we say $\boldsymbol{u}$ is a $\boldsymbol{k}$-admissible solution of (1.1).

Recalling that $f_{i}\in[0,+\infty)\ (i=1,\ldots,n)$ and $\boldsymbol{u}\in
C^{2}(B)$ is $\boldsymbol{k}$-admissible solution of (1.1) vanishing on the
boundary, we can achieve that $\boldsymbol{u}$ is sub-harmonic in $B$ from
[25]. Hence, we apply the maximum principle to conclude that $\boldsymbol{u}$
is negative in $B$.

The study of $k$-Hessian equations plays an important role in differential
geometry, fluid mechanics and other applied disciplines. In the past years,
many authors show great interest in solutions of $k$-Hessian equations and
many excellent results on $k$-Hessian equations have been obtained, for
instance, see [1, 9, 15, 17, 16, 18, 19, 25, 3, 2, 21, 22, 20]. However, there
are few studies that consider the fully nonlinear coupled systems except [7,
24, 29, 23, 4, 5, 6] based on our cognition. For example, by using fixed point
theorem, Wang [24] established the existence, multiplicity and nonexistence of
convex radial solutions to a coupled system of Monge-Ampère equations in
superlinear and sublinear cases. In [7], the authors studied the existence and
multiplicity of nontrivial radial solutions for system coupled by
multiparameter $k$-Hessian equations and obtained sufficient conditions for
the existence of nontrivial radial solutions to power-type coupled $k$-Hessian
system based on a eigenvalue theory in cones. In particular, Cui considered a
Hessian type system coupled by different $k$-Hessian equations and obtained
the existence of entire $k$-convex radial solutions, see [4].

Inspired by the above works, we are interested in a system coupled by
different $k$-Hessian equations with general nonlinearities which satisfy
$\alpha_{i}$ or $\beta_{i}$-asymptotic growth conditions. In this paper, we
shall establish the existence and multiplicity of nontrivial radial
$\boldsymbol{k}$-admissible solutions of the weakly coupled degenerated system
(1.1). It is worth to notice that the system (1.1) contains a variety of
different $k$-Hessian equations which is significantly different from that in
[23, 7, 29, 5] such that the problem we considered can contain Laplace
equations and Monge-Ampère equations at the same time. This kind of system can
represent the coupling of different types of elliptic equations, which makes
our problem more comprehensive and more applicable.

If $\alpha_{i},\beta_{i}>0$, we let

$\underline{f}_{i}^{0}=\liminf_{c\to 0^{+}}\min_{0\leq t\leq
1}\frac{f_{i}(t,c)}{c^{\alpha_{i}}},\qquad\qquad\underline{f}_{i}^{\infty}=\liminf_{c\to\infty}\min_{0\leq
t\leq 1}\frac{f_{i}(t,c)}{c^{\beta_{i}}},$

$\overline{f}_{i}^{0}=\limsup_{c\to 0^{+}}\max_{0\leq t\leq
1}\frac{f_{i}(t,c)}{c^{\alpha_{i}}},\qquad\qquad\overline{f}_{i}^{\infty}=\limsup_{c\to\infty}\max_{0\leq
t\leq 1}\frac{f_{i}(t,c)}{c^{\beta_{i}}}.$

Here, we call them $\alpha_{i}$ or $\beta_{i}$-asymptotic growth condition,
super-$\alpha_{i}$ or $\beta_{i}$-asymptotic growth condition and
sub-$\alpha_{i}$ or $\beta_{i}$-asymptotic growth condition. Compared with
some $N$-asymptotic growth (see, for instance [8, 24, 6] where the constants
$\alpha_{i}=\beta_{i}=N$) in studying Monge-Ampère equations and some
$k$-asymptotic growth (see, for instance [7, 26, 27] where the constants
$\alpha_{i}=\beta_{i}=k$) in studying $k$-Hessian equations, our conditions
are more flexible. By imposing suitable conditions on $\underline{f}_{i}^{0}$,
$\underline{f}_{i}^{\infty}$, $\overline{f}_{i}^{0}$,
$\overline{f}_{i}^{\infty}$ and coordinating inequality relations between
$\alpha_{i},\beta_{i}$ and $k_{i}$, we obtain existence and multiplicity
results in general cases as follows.

We will assume $\boldsymbol{f}=\\{f_{1},\ldots,f_{n}\\}$ satisfies one of the
following conditions:
(C1) $\underline{f}_{i}^{0},\overline{f}_{i}^{\infty}\in(0,+\infty)$,
$i=1,\ldots,n$ and $f_{i}(t,0)=0$, $i=2,\ldots,n$;
(C2) $\overline{f}_{i}^{0},\underline{f}_{i}^{\infty}\in(0,+\infty)$,
$i=1,\ldots,n$;
(C3) $\underline{f}_{i}^{0},\underline{f}_{i}^{\infty}\in(0,+\infty)$,
$i=1,\ldots,n$ and $f_{i}(t,0)=0$, $i=2,\ldots,n$;
(C4) $\overline{f}_{i}^{0},\overline{f}_{i}^{\infty}\in(0,+\infty)$,
$i=1,\ldots,n$.

######  Theorem 1.1.

(Existence theorem) Suppose that (F) and one of the following conditions hold:
(a). (C1) holds and positive constants $\alpha_{i}$, $\beta_{i}\
(i=1,\ldots,n)$ satisfy

$\prod_{i=1}^{n}\alpha_{i}<\prod_{i=1}^{n}k_{i},\quad\prod_{i=1}^{n}\beta_{i}<\prod_{i=1}^{n}k_{i};$

(b). (C2) holds and positive constants $\alpha_{i}$, $\beta_{i}\
(i=1,\ldots,n)$ satisfy

$\prod_{i=1}^{n}\alpha_{i}>\prod_{i=1}^{n}k_{i},\quad\prod_{i=1}^{n}\beta_{i}>\prod_{i=1}^{n}k_{i}.$

Then system (1.1) has at least one nontrivial radial convex solution.

Theorem 1.1 is concerning the existence of nontrivial radial convex solutions
to the weakly coupled degenerate system (1.1) with general nonlinear terms.
Furthermore, we can consider the result of multiplicity as well.

Let

$\displaystyle
G_{i}=\max\left\\{f_{i}(t,v_{i+1}(t)):(t,v_{i+1}(t))\in[0,1]\times[0,G_{i+1}^{\frac{1}{k_{i+1}}}]\right\\},\quad
i=1,\ldots,n-1,$ $\displaystyle
G_{n}=\max\left\\{f_{n}(t,v_{1}(t)):(t,v_{1}(t))\in[0,1]\times[0,\frac{r_{0}}{4}]\right\\},$
$\displaystyle\tilde{G}_{i}=\max\left\\{f_{i}(t,v_{i+1}(t)):(t,v_{i+1}(t))\in[0,1]\times[0,\tilde{G}_{i+1}^{\frac{1}{k_{i+1}}}]\right\\},\quad
i=2,\ldots,n-1,$
$\displaystyle\tilde{G}_{n}=\max\left\\{f_{n}(t,v_{1}(t)):(t,v_{1}(t))\in[0,1]\times[0,R_{0}]\right\\},$
$\displaystyle
E_{i}=\min\left\\{f_{i}(t,v_{i+1}(t)):(t,v_{i+1}(t))\in[\frac{1}{4},\frac{3}{4}]\times[\frac{1}{4}\Gamma_{i+1}E_{i+1}^{\frac{1}{k_{i+1}}},\tilde{G}_{i+1}^{\frac{1}{k_{i+1}}}]\right\\},\quad
i=1,\ldots,n-1,$ $\displaystyle
E_{n}=\min\left\\{f_{n}(t,v_{1}(t)):(t,v_{1}(t))\in[\frac{1}{4},\frac{3}{4}]\times[\frac{1}{4}R_{0},R_{0}]\right\\}.$

######  Theorem 1.2.

(Multiplicity theorem) Suppose that (F) and one of the following conditions
hold:
(c). (C3) holds, positive constants $\alpha_{i}$, $\beta_{i}\ (i=1,\ldots,n)$
satisfy

$\prod_{i=1}^{n}\alpha_{i}<\prod_{i=1}^{n}k_{i},\quad\prod_{i=1}^{n}\beta_{i}>\prod_{i=1}^{n}k_{i},$

and there exists a positive constant $r_{0}$ such that
$r_{0}>G_{1}^{\frac{1}{k_{1}}}$;
(d). (C4) holds, positive constants $\alpha_{i}$, $\beta_{i}(i=1,\ldots,n)$
satisfy

$\prod_{i=1}^{n}\alpha_{i}>\prod_{i=1}^{n}k_{i},\quad\prod_{i=1}^{n}\beta_{i}<\prod_{i=1}^{n}k_{i},$

and there exists a positive constant $R_{0}$ such that
$R_{0}<\Gamma_{1}E_{1}^{\frac{1}{k_{1}}}.$ Then system (1.1) has at least two
nontrivial radial convex solutions.

######  Remark 1.1.

Theorem 1.1 and Theorem 1.2 show that the existence and multiplicity of
nontrivial radial convex solutions to system (1.1) respectively, see the
penultimate paragraph in this section for more detailed explanations of convex
solutions. Since the convex solutions are contained in the
$\boldsymbol{k}$-admissible solutions, Theorems 1.1 and 1.2 show the existence
and multiplicity of $\boldsymbol{k}$-admissible solutions as well.

######  Remark 1.2.

It is worth to mention that the condition (C1) and (C3) can be replaced by
$f_{m_{k}}(t,0)=0,\ k=1,\ldots,n-1$, where
$\\{m_{1},\ldots,m_{n-1}\\}\subset\\{1,\ldots,n\\}$ and we describe as
$f_{i}(t,0)=0$, $i=2,\ldots,n$ for the sake of proof.

Specifically, we also study the uniqueness and nonexistence of nontrivial
radial solutions to a power-type coupled system of $\boldsymbol{k}$-Hessian
equations:

(1.2) $\left\\{\begin{aligned}
&S_{k_{1}}\left(D^{2}u_{1}\right)=\left(-u_{2}\right)^{\gamma_{1}},&{\rm in}\
B,\\\ &S_{k_{2}}\left(D^{2}u_{2}\right)=\left(-u_{3}\right)^{\gamma_{2}},&{\rm
in}\ B,\\\ &\qquad\qquad\quad\vdots\\\
&S_{k_{n-1}}\left(D^{2}u_{n-1}\right)=\left(-u_{n}\right)^{\gamma_{n-1}},&{\rm
in}\ B,\\\
&S_{k_{n}}\left(D^{2}u_{n}\right)=\left(-u_{1}\right)^{\gamma_{n}},&{\rm in}\
B,\\\ &u_{i}=0,i=1,\ldots,n,&{\rm on}\ \partial B,\end{aligned}\right.$

where $\gamma_{i}\ (i=1,\ldots,n)$ are positive constants.

It is obvious that system (1.2) is a special case of system (1.1). By the
definitions of $\alpha_{i}$ or $\beta_{i}$-asymptotic growth condition, the
growth of nonlinearities of the power-type system (1.2) satisfies
$\alpha_{i}=\beta_{i}=\gamma_{i}$, which asserts the existence of nontrivial
radial convex solutions to system (1.2) by Theorem 1.1 when
$\prod_{i=1}^{n}\gamma_{i}\neq\prod_{i=1}^{n}k_{i}$. Next, we go further to
study the uniqueness of nontrivial radial convex solution to system (1.2) in
Theorem 1.3.

######  Theorem 1.3.

(Uniqueness theorem) Suppose that positive constant
$\prod_{i=1}^{n}\gamma_{i}$ satisfies

$\prod_{i=1}^{n}\gamma_{i}<\prod_{i=1}^{n}k_{i},$

then system (1.2) has a unique nontrivial radial convex solution.

Here, we get the uniqueness result of nontrivial radial convex solution to
system (1.2) in the assumption of
$\prod_{i=1}^{n}\gamma_{i}<\prod_{i=1}^{n}k_{i}$. Besides, we obtain the
nonexistence of nontrivial radial $\boldsymbol{k}$-admissible solution in $B$
when $\prod_{i=1}^{n}\gamma_{i}=\prod_{i=1}^{n}k_{i}$.

######  Theorem 1.4.

(Nonexistence theorem) Suppose that positive constant
$\prod_{i=1}^{n}\gamma_{i}$ satisfies

$\prod_{i=1}^{n}\gamma_{i}=\prod_{i=1}^{n}k_{i},$

then system (1.2) admits no nontrivial radial $\boldsymbol{k}$-admissible
solution.

When $\prod_{i=1}^{n}\gamma_{i}=\prod_{i=1}^{n}k_{i}$, we are interested in
the existence of nonzero $\boldsymbol{k}$-admissible solutions for the
eigenvalue problem:

(1.3) $\left\\{\begin{aligned}
&S_{k_{1}}\left(D^{2}u_{1}\right)=\lambda_{1}\left(-u_{2}\right)^{\gamma_{1}},&{\rm
in}\ \Omega,\\\
&S_{k_{2}}\left(D^{2}u_{2}\right)=\lambda_{2}\left(-u_{3}\right)^{\gamma_{2}},&{\rm
in}\ \Omega,\\\ &\qquad\qquad\quad\vdots\\\
&S_{k_{n-1}}\left(D^{2}u_{n-1}\right)=\lambda_{n-1}\left(-u_{n}\right)^{\gamma_{n-1}},&{\rm
in}\ \Omega,\\\
&S_{k_{n}}\left(D^{2}u_{n}\right)=\lambda_{n}\left(-u_{1}\right)^{\gamma_{n}},&{\rm
in}\ \Omega,\\\ &u_{i}=0,i=1,\ldots,n,&{\rm on}\
\partial\Omega,\end{aligned}\right.$

with positive parameters $\lambda_{i}\ (i=1,\ldots,n)$, where
$\Omega\in\mathbb{R}^{N}$ is a bounded, smooth and strictly $(k-1)$-convex
domain, $N\geq 2$.

In fact, Wang has proved the existence of a positive eigenvalue $\lambda^{*}$
for a single $k$-Hessian equation with $f(u)=\lambda|u|^{k}(k<N)$ in [25].
When $\lambda=\lambda^{*}$, the corresponding eigenfunction $\varphi^{*}$ is
nonzero $k$-admissible and that any other eigenfunction would be a positive
constant multiple of $\varphi^{*}$. Since $\lambda^{*}$ acts like a
bifurcation point for system (1.3), we can be reminiscent of the generalized
Krein-Rutman theorem in [13] to obtain the existence of
$\boldsymbol{k}$-admissible solutions to eigenvalue problem (1.3).

######  Theorem 1.5.

(Eigenvalue problem) Suppose that $\Omega\in\mathbb{R}^{N}$ is a bounded,
smooth and strictly $(k-1)$-convex domain, positive constant
$\prod_{i=1}^{n}\gamma_{i}$ satisfies

$\prod_{i=1}^{n}\gamma_{i}=\prod_{i=1}^{n}k_{i},$

then system (1.3) admits a nonzero $\boldsymbol{k}$-admissible solution if and
only if
$\lambda_{1}\lambda_{2}^{\frac{\gamma_{1}}{k_{2}}}\cdots\lambda_{n}^{\frac{\prod_{i=1}^{n-1}\gamma_{i}}{\prod_{i=2}^{n}k_{i}}}=\lambda_{0}^{k_{1}}$,
where $\lambda_{0}\neq 1$ is a positive constant, such that the system

(1.4) $\left\\{\begin{aligned}
&S_{k_{1}}\left(D^{2}(\frac{u_{1}}{\lambda_{0}})\right)=\left(-u_{2}\right)^{\gamma_{1}},&{\rm
in}\ \Omega,\\\
&S_{k_{2}}\left(D^{2}u_{2}\right)=\left(-u_{3}\right)^{\gamma_{2}},&{\rm in}\
\Omega,\\\ &\qquad\qquad\quad\vdots\\\
&S_{k_{n-1}}\left(D^{2}u_{n-1}\right)=\left(-u_{n}\right)^{\gamma_{n-1}},&{\rm
in}\ \Omega,\\\
&S_{k_{n}}\left(D^{2}u_{n}\right)=\left(-u_{1}\right)^{\gamma_{n}},&{\rm in}\
\Omega,\\\ &u_{i}=0,\quad i=1,\ldots,n,&{\rm on}\
\partial\Omega,\end{aligned}\right.$

has a nonzero $\boldsymbol{k}$-admissible solution.

Note that the existence of nonzero $\boldsymbol{k}$-admissible solution of
(1.4) is guaranteed by a generalized Krein-Rutman theorem, see Section 5 for
details.

In this article, we study the existence and multiplicity of radial convex
solutions to system (1.1), the uniqueness of radial convex solution and
nonexistence of radial $\boldsymbol{k}$-admissible solution to system (1.2),
and the existence of radial $\boldsymbol{k}$-admissible solutions to the
related eigenvalue problem (1.3). The reasons why Theorems 1.1, 1.2 and 1.3
are only restricted to the convex solutions will be further explained in
Remarks 2.1 and 4.1. The improvement from convex solutions to
$\boldsymbol{k}$-admissible solutions in Theorem 1.1, Theorem 1.2 and Theorem
1.3 is still an interesting problem, which attracts us to find another way or
technique to solve this problem in a sequel.

The rest of the paper is organized as follows. In Section 2, we make some
preliminary calculations of $C^{2}$ radial solutions and present a fixed point
theorem in Theorem 2.1. In Section 3, we give the proof of existence and
multiplicity results for system (1.1) with general nonlinearities by using the
fixed point theorem. In Section 4, the uniqueness and nonexistence results for
power-type coupled system (1.2) which is a special case of (1.1) are
considered. In Section 5, by overcoming the difficulties caused by verifying
the condition of generalized Krein-Rutman theorem which to prove the operator
is strong, we obtain the existence of nonzero $\boldsymbol{k}$-admissible
solutions to the eigenvalue problem (1.3) in a general strictly $(k-1)$-convex
domain.

##  2\. Preliminaries

To study radial classical solutions of system (1.1), we assume $u(|x|)=u(t)$
be the radial function with $t=\sqrt{\sum_{i=1}^{N}x_{i}^{2}}$, then it
follows from Lemma 2.1 in [14] that the $k$-Hessian operator becomes

$S_{k}(D^{2}u)=C_{N-1}^{k-1}u^{\prime\prime}(t)(\frac{u^{\prime}(t)}{t})^{k-1}+C_{N-1}^{k}(\frac{u^{\prime}(t)}{t})^{k},\quad
t\in(0,1).$

Then we can convert (1.1) to the following system of ordinary differential
equations:

(2.1)
$\left\\{\begin{array}[]{ll}C_{N-1}^{k_{1}-1}u_{1}^{\prime\prime}(t)(\frac{u_{1}^{\prime}(t)}{t})^{k_{1}-1}+C_{N-1}^{k_{1}}(\frac{u_{1}^{\prime}(t)}{t})^{k_{1}}=f_{1}(t,-u_{2}),&0<t<1,\\\\[4.30554pt]
C_{N-1}^{k_{2}-1}u_{2}^{\prime\prime}(t)(\frac{u_{2}^{\prime}(t)}{t})^{k_{2}-1}+C_{N-1}^{k_{2}}(\frac{u_{2}^{\prime}(t)}{t})^{k_{2}}=f_{2}(t,-u_{3}),&0<t<1,\\\
\qquad\qquad\vdots\\\
C_{N-1}^{k_{n-1}-1}u_{n-1}^{\prime\prime}(t)(\frac{u_{n-1}^{\prime}(t)}{t})^{k_{n-1}-1}+C_{N-1}^{k_{n-1}}(\frac{u_{n-1}^{\prime}(t)}{t})^{k_{n-1}}=f_{n-1}(t,-u_{n}),&0<t<1,\\\\[4.30554pt]
C_{N-1}^{k_{n}-1}u_{n}^{\prime\prime}(t)(\frac{u_{n}^{\prime}(t)}{t})^{k_{n}-1}+C_{N-1}^{k_{n}}(\frac{u_{n}^{\prime}(t)}{t})^{k_{n}}=f_{n}(t,-u_{1}),&0<t<1,\\\\[4.30554pt]
u_{i}(1)=u_{i}^{\prime}(0)=0,\quad i=1,\ldots,n.\end{array}\right.$

Equivalently, we seek nonnegative $k$-concave solutions for convenience by
making a simple transformation $v_{i}=-u_{i}\ (i=1,\ldots,n)$ in (2.1), which
leads to the following system:

(2.2)
$\left\\{\begin{array}[]{ll}C_{N-1}^{k_{1}-1}(-v_{1})^{\prime\prime}(t)(\frac{(-v_{1})^{\prime}(t)}{t})^{k_{1}-1}+C_{N-1}^{k_{1}}(\frac{(-v_{1})^{\prime}(t)}{t})^{k_{1}}=f_{1}(t,v_{2}),&0<t<1,\\\\[4.30554pt]
C_{N-1}^{k_{2}-1}(-v_{2})^{\prime\prime}(t)(\frac{(-v_{2})^{\prime}(t)}{t})^{k_{2}-1}+C_{N-1}^{k_{2}}(\frac{(-v_{2})^{\prime}(t)}{t})^{k_{2}}=f_{2}(t,v_{3}),&0<t<1,\\\
\qquad\qquad\vdots\\\
C_{N-1}^{k_{n-1}-1}(-v_{n-1})^{\prime\prime}(t)(\frac{(-v_{n-1})^{\prime}(t)}{t})^{k_{n-1}-1}+C_{N-1}^{k_{n-1}}(\frac{(-v_{n-1})^{\prime}(t)}{t})^{k_{n-1}}=f_{n-1}(t,v_{n}),&0<t<1,\\\\[4.30554pt]
C_{N-1}^{k_{n}-1}(-v_{n})^{\prime\prime}(t)(\frac{(-v_{n})^{\prime}(t)}{t})^{k_{n}-1}+C_{N-1}^{k_{n}}(\frac{(-v_{n})^{\prime}(t)}{t})^{k_{n}}=f_{n}(t,v_{1}),&0<t<1,\\\\[4.30554pt]
v_{i}(1)=v_{i}^{\prime}(0)=0,\quad i=1,\ldots,n.\end{array}\right.$

By integration, we get from (2.2) that

$\left\\{\begin{array}[]{ll}v_{1}(t)=\int_{t}^{1}\left(\frac{k_{1}}{\tau^{N-k_{1}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{1}-1}}f_{1}\left(s,v_{2}(s)\right)\,{\rm
d}s\right)^{\frac{1}{k_{1}}}\,{\rm d}\tau,&0\leq t\leq 1,\\\
v_{2}(t)=\int_{t}^{1}\left(\frac{k_{2}}{\tau^{N-k_{2}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{2}-1}}f_{2}\left(s,v_{3}(s)\right)\,{\rm
d}s\right)^{\frac{1}{k_{2}}}\,{\rm d}\tau,&0\leq t\leq 1,\\\
\qquad\qquad\vdots\\\
v_{n-1}(t)=\int_{t}^{1}\left(\frac{k_{n-1}}{\tau^{N-k_{n-1}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{n-1}-1}}f_{n-1}\left(s,v_{n}(s)\right)\,{\rm
d}s\right)^{\frac{1}{k_{n-1}}}\,{\rm d}\tau,&0\leq t\leq 1,\\\
v_{n}(t)=\int_{t}^{1}\left(\frac{k_{n}}{\tau^{N-k_{n}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{n}-1}}f_{n}\left(s,v_{1}(s)\right)\,{\rm
d}s\right)^{\frac{1}{k_{n}}}\,{\rm d}\tau,&0\leq t\leq 1.\end{array}\right.$

Considering the Banach space $X:=C[0,1]$, for
$\boldsymbol{v}=(v_{1},\ldots,v_{n})\in\underbrace{X\times\cdots\times
X}_{n}$, we define
$||\boldsymbol{v}||=\sum_{i=1}^{n}||\boldsymbol{v_{i}}(t)||=\sum_{i=1}^{n}\sup\limits_{t\in[0,1]}|v_{i}(t)|$.
Let $K$ be a cone in $X$ defined as

(2.3) $K:=\left\\{v\in X:v(t)\geq 0,t\in[0,1],\min_{\frac{1}{4}\leq
t\leq\frac{3}{4}}v(t)\geq\frac{1}{4}||v||\right\\}.$

We define the operators $T_{i}:K\to X\ (i=1,\ldots,n)$ to be

$\displaystyle
T_{1}(v_{2})(t)=\int_{t}^{1}\left(\frac{k_{1}}{\tau^{N-k_{1}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{1}-1}}f_{1}\left(s,v_{2}(s)\right)\,{\rm
d}s\right)^{\frac{1}{k_{1}}}\,{\rm d}\tau,$ $\displaystyle
T_{2}(v_{3})(t)=\int_{t}^{1}\left(\frac{k_{2}}{\tau^{N-k_{2}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{2}-1}}f_{2}\left(s,v_{3}(s)\right)\,{\rm
d}s\right)^{\frac{1}{k_{2}}}\,{\rm d}\tau,$ $\displaystyle\qquad\qquad\vdots$
$\displaystyle
T_{n-1}(v_{n})(t)=\int_{t}^{1}\left(\frac{k_{n-1}}{\tau^{N-k_{n-1}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{n-1}-1}}f_{n-1}\left(s,v_{n}(s)\right)\,{\rm
d}s\right)^{\frac{1}{k_{n-1}}}\,{\rm d}\tau,$ $\displaystyle
T_{n}(v_{1})(t)=\int_{t}^{1}\left(\frac{k_{n}}{\tau^{N-k_{n}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{n}-1}}f_{n}\left(s,v_{1}(s)\right)\,{\rm
d}s\right)^{\frac{1}{k_{n}}}\,{\rm d}\tau.$

Note that each image of operator is a nonnegative $k$-concave function on
$[0,1]$ and we define
$T_{1}(v_{2})=v_{1},T_{2}(v_{3})=v_{2},\cdots,T_{n}(v_{1})=v_{n}$ in $K$.
Thus, by the concavity of $v_{i}\ (i=1,\ldots,n)$, it is easy to see that
$T_{i}\ (i=1,\ldots,n)$ maps $K$ into itself. Besides, by standard arguments,
we know that every operator is completely continuous.

Next, we define a composite operator $Tv_{1}=T_{1}T_{2}\cdots T_{n}(v_{1})$,
which is also completely continuous from $K$ to $K$. We can see that positive
solutions of (2.2) are equivalent to nonzero fixed points of operator $T$ in
cone $K$. If
$\boldsymbol{v}=(v_{1},\ldots,v_{n})\in\underbrace{C[0,1]\times\cdots\times
C[0,1]}_{n}$ is a positive solution of (2.2), then $v_{1}$ must be a nonzero
fixed point of $T$ in $K$; conversely if $v_{1}\in
K\setminus\left\\{0\right\\}$ is a fixed point of $T$, we can define
$v_{n}=T_{n}(v_{1}),v_{n-1}=T_{n-1}(v_{n}),\cdots,v_{2}=T_{2}(v_{3})$ such
that $(v_{1},\ldots,v_{n})\in\underbrace{C[0,1]\times\cdots\times C[0,1]}_{n}$
solves (2.2).

######  Remark 2.1.

As we shall see in the last two paragraphs, we let each $T_{i}\ (i=1\ldots,n)$
maps $K$ to itself which implies that $v^{\prime}(t)=(-u)^{\prime}(t)$ is
nonincreasing from Lemma 2.2 in [24]. On the other hand, the eigenvalues of
the second derivative of radial classical function in a unit ball can be
represented by
$\lambda(D^{2}u)=(u^{\prime\prime}(t),\frac{u^{\prime}(t)}{t},\ldots,\frac{u^{\prime}(t)}{t}),t\in[0,1]$,
we combine this with the definition of $\boldsymbol{k}$-admissible function,
an immediate consequence is that we essentially achieve the $(N-1)$-admissible
function in $\mathbb{R}^{N}$, that is, all $\frac{u^{\prime}(t)}{t}\geq 0$. To
sum up, all eigenvalues of the Hessian matrix of nontrivial radial
$\boldsymbol{k}$-admissible solutions of system (1.1) and system (1.2) are
nonnegative and exist in its closure of convex cone, which can draw our
conclusion.

The proofs of our existence and multiplicity results are based on the
following well-known fixed point theorem of cone, (see Theorem 2.3.4 in Guo
[10]).

######  Theorem 2.1.

Let $X$ be a Banach space and $K$ is a cone in $X$. Assume that $\Omega_{1}$,
$\Omega_{2}$ are bounded open subsets of $X$ with $0\in\Omega_{1}$,
$\overline{\Omega}_{1}\subset\Omega_{2}$ and let

$T:K\cap(\overline{\Omega}_{2}\setminus\Omega_{1})\to K$

be completely continuous such that either

  1. (i)

$||Tu||\geq||u||$, $u\in K\cap\partial\Omega_{1}$ and $||Tu||\leq||u||$, $u\in
K\cap\partial\Omega_{2}$; or

  2. (ii)

$||Tu||\leq||u||$, $u\in K\cap\partial\Omega_{1}$ and $||Tu||\geq||u||$, $u\in
K\cap\partial\Omega_{2}$

holds, where $||\cdot||$ is a norm in X, $\Omega_{R}=\\{u\in K:||u||<R\\}$ and
$\partial\Omega_{R}=\\{u\in K:||u||=R\\}$. Then T has a fixed point in
$K\cap(\overline{\Omega}_{2}\setminus\Omega_{1})$.

##  3\. Existence and multiplicity

In this section, we apply the fixed point theorem of cone in Theorem 2.1 to
prove the existence and multiplicity results in Theorem 1.1 and Theorem 1.2.
To simplify notation, we denote $v_{1}$ by $v_{n+1}$ .

###  3.1. Existence

In order to prove the Theorem 1.1, we first introduce two useful lemmas.

######  Lemma 3.1.

Assume (F) holds. Let $\eta,m>0$ and $v_{i}\in K,\ i=1,\ldots,n$. If for any
$t\in[\frac{1}{4},\frac{3}{4}]$ and $i=1,\ldots,n$, we have

$f_{i}\left(t,v_{i+1}(t)\right)\geq\eta v_{i+1}^{m}(t),$

then

$T_{i}(v_{i+1})(\frac{1}{4})\geq\Gamma_{i}\eta^{\frac{1}{k_{i}}}(\frac{1}{4})^{\frac{m}{k_{i}}}||v_{i+1}||^{\frac{m}{k_{i}}},\quad
i=1,\ldots,n,$

where $\Gamma_{i}$ are positive constants given by
$\Gamma_{i}=\int_{\frac{1}{4}}^{\frac{3}{4}}\left(\frac{k_{i}}{\tau^{N-k_{i}}}\int_{\frac{1}{4}}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{i}-1}}\,{\rm
d}s\right)^{\frac{1}{k_{i}}}\,{\rm d}\tau,i=1,\ldots,n$.

###### Proof.

For $v_{1}\in K$, we have

$\begin{split}T_{n}(v_{1})(\frac{1}{4})=&\int_{\frac{1}{4}}^{1}\left(\frac{k_{n}}{\tau^{N-k_{n}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{n}-1}}f_{n}\left(s,v_{1}(s)\right)\,{\rm
d}s\right)^{\frac{1}{k_{n}}}\,{\rm d}\tau\\\
\geq&\int_{\frac{1}{4}}^{\frac{3}{4}}\left(\frac{k_{n}}{\tau^{N-k_{n}}}\int_{\frac{1}{4}}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{n}-1}}\eta
v_{1}^{m}(s)\,{\rm d}s\right)^{\frac{1}{k_{n}}}\,{\rm d}\tau\\\
\geq&\int_{\frac{1}{4}}^{\frac{3}{4}}\left(\frac{k_{n}}{\tau^{N-k_{n}}}\int_{\frac{1}{4}}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{n}-1}}\eta\left(\frac{1}{4}||v_{1}||\right)^{m}\,{\rm
d}s\right)^{\frac{1}{k_{n}}}\,{\rm d}\tau\\\
=&\Gamma_{n}\eta^{\frac{1}{k_{n}}}\left(\frac{1}{4}||v_{1}||\right)^{\frac{m}{k_{n}}}.\end{split}$

For $v_{i}\in K\ (i=2,\ldots,n)$, we have similar calculations. Here we omit
them for simplicity. ∎

######  Lemma 3.2.

Assume (F) holds. Let $\varepsilon,d>0$ and $v_{i}\in K,i=1,\ldots,n$. If for
any $t\in[0,1]$ and $i=1,\ldots,n$, we have

$f_{i}\left(t,v_{i+1}(t)\right)\leq\varepsilon v_{i+1}^{d}(t),$

then

$T_{i}(v_{i+1})(t)<\left(\varepsilon||v_{i+1}||^{d}\right)^{\frac{1}{k_{i}}},\quad
i=1,\ldots,n.$

###### Proof.

Since $v_{1}(t)\in K$, ${\forall t}$ $\in[0,1]$, we have

(3.1)
$\begin{split}T_{n}(v_{1})(t)\leq&\int_{0}^{1}\left(\frac{k_{n}}{\tau^{N-k_{n}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{n}-1}}f_{n}\left(s,v_{1}(s)\right)\,{\rm
d}s\right)^{\frac{1}{k_{n}}}\,{\rm d}\tau\\\
\leq&\int_{0}^{1}\left(\frac{k_{n}}{\tau^{N-k_{n}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{n}-1}}\varepsilon
v_{1}^{d}(s)\,{\rm d}s\right)^{\frac{1}{k_{n}}}\,{\rm d}\tau\\\
\leq&\frac{1}{2}\left(\frac{k_{n}}{NC_{N-1}^{k_{n}-1}}\right)^{\frac{1}{k_{n}}}\left(\varepsilon||v_{1}||^{d}\right)^{\frac{1}{k_{n}}}\\\
<&\left(\varepsilon||v_{1}||^{d}\right)^{\frac{1}{k_{n}}},\end{split}$

where the fact
$\frac{1}{2}\left(\frac{k_{n}}{NC_{N-1}^{k_{n}-1}}\right)^{\frac{1}{k_{n}}}<1$
is used in the last inequality, which is easily checked. For $v_{i}\in K\
(i=2,\ldots,n)$, we also have similar conclusions. ∎

On the basis of the above preparations, we give the proof for the existence
result in Theorem 1.1 with the aid of the fixed point theorem of cone.

###### Proof of Theorem 1.1.

(a). It follows from $\underline{f}_{i}^{0}\in(0,+\infty)\ (i=1,\ldots,n)$
that for any given
$\varepsilon_{1}\in(0,\min\\{\underline{f}_{i}^{0},i=1,\ldots,n\\})$, there
exists a constant $r_{1}\in(0,1)$ such that

(3.2)
$f_{i}\left(t,v_{i+1}(t)\right)\geq(\underline{f}_{i}^{0}-\varepsilon_{1})v_{i+1}^{\alpha_{i}},\quad
0\leq v_{i+1}\leq r_{1},$

for any $t\in[0,1]$ and $i=1,\ldots,n$. Let

$L_{1}:=\Gamma_{1}\cdots\Gamma_{n}^{\frac{\prod_{i=1}^{n-1}\alpha_{i}}{\prod_{i=1}^{n-1}k_{i}}}(\underline{f}_{1}^{0}-\varepsilon_{1})^{\frac{1}{k_{1}}}\cdots(\underline{f}_{n}^{0}-\varepsilon_{1})^{\frac{\prod_{i=1}^{n-1}\alpha_{i}}{\prod_{i=1}^{n}k_{i}}}(\frac{1}{4})^{\frac{\alpha_{1}}{k_{1}}+\cdots+\frac{\prod_{i=1}^{n}\alpha_{i}}{\prod_{i=1}^{n}k_{i}}}$

be a positive constant. Since $f_{i}(t,0)=0$ for $i=2,\ldots,n$, there exists
another constant $r_{2}$:

$0<r_{2}<{\rm
min}\left\\{r_{1},L_{1}^{\frac{\prod_{i=1}^{n}k_{i}}{\prod_{i=1}^{n}k_{i}-\prod_{i=1}^{n}\alpha_{i}}}\right\\}$

such that

(3.3) $f_{i}\left(t,v_{i+1}(t)\right)\leq r_{1}^{k_{i}},\quad 0\leq
v_{i+1}\leq r_{2},$

for any $t\in[0,1]$ and $i=2,\ldots,n$. For $v_{1}\in
K\cap\partial\Omega_{r_{2}}$, it follows from Lemma 3.2 and (3.3) that

$v_{i}(t)=T_{i}(v_{i+1})(t)<r_{1},\quad i=2,\ldots,n$

which shows that for any $v_{1}\in K\cap\partial\Omega_{r_{2}}$, we have
$v_{i}\in(0,r_{1})$, for all $i=1,\ldots,n$. Then by Lemma 3.1 and (3.2), we
get

$T_{i}(v_{i+1})(\frac{1}{4})\geq\Gamma_{i}(\underline{f}_{i}^{0}-\varepsilon_{1})^{\frac{1}{k_{i}}}(\frac{1}{4})^{\frac{\alpha_{i}}{k_{i}}}||v_{i+1}||^{\frac{\alpha_{i}}{k_{i}}},\quad
i=1,\ldots,n.$

This suggests that for any $v_{1}\in K\cap\partial\Omega_{r_{2}}$, we have

$\begin{split}||Tv_{1}||=&\sup_{t\in[0,1]}\left|T_{1}T_{2}\cdots
T_{n}(v_{1})(t)\right|\\\ \geq&T_{1}T_{2}\cdots T_{n}(v_{1})(\frac{1}{4})\\\
\geq&\Gamma_{1}(\underline{f}_{1}^{0}-\varepsilon_{1})^{\frac{1}{k_{1}}}(\frac{1}{4})^{\frac{\alpha_{1}}{k_{1}}}||T_{2}(v_{3})||^{\frac{\alpha_{1}}{k_{1}}}\\\
\geq&\Gamma_{1}(\underline{f}_{1}^{0}-\varepsilon_{1})^{\frac{1}{k_{1}}}(\frac{1}{4})^{\frac{\alpha_{1}}{k_{1}}}|T_{2}(v_{3})(\frac{1}{4})|^{\frac{\alpha_{1}}{k_{1}}}\\\
\geq&\Gamma_{1}\Gamma_{2}^{\frac{\alpha_{1}}{k_{1}}}(\underline{f}_{1}^{0}-\varepsilon_{1})^{\frac{1}{k_{1}}}(\underline{f}_{2}^{0}-\varepsilon_{1})^{\frac{\alpha_{1}}{k_{1}k_{2}}}(\frac{1}{4})^{\frac{\alpha_{1}}{k_{1}}+\frac{\alpha_{2}\alpha_{2}}{k_{1}k_{2}}}||T_{3}(v_{4})||^{\frac{\alpha_{1}\alpha_{2}}{k_{1}k_{2}}}\\\
&\qquad\vdots\\\
\geq&L_{1}||v_{1}||^{\frac{\prod_{i=1}^{n}\alpha_{i}}{\prod_{i=1}^{n}k_{i}}}.\end{split}$

Notice that
$||v_{1}||=r_{2}<L_{1}^{\frac{\prod_{i=1}^{n}k_{i}}{\prod_{i=1}^{n}k_{i}-\prod_{i=1}^{n}\alpha_{i}}}$
and $\prod_{i=1}^{n}\alpha_{i}<\prod_{i=1}^{n}k_{i}$, then

$\frac{L_{1}||v_{1}||^{\frac{\prod_{i=1}^{n}\alpha_{i}}{\prod_{i=1}^{n}k_{i}}}}{||v_{1}||}=\frac{L_{1}}{||v_{1}||^{\frac{\prod_{i=1}^{n}k_{i}-\prod_{i=1}^{n}\alpha_{i}}{\prod_{i=1}^{n}k_{i}}}}>1,$

which implies that

(3.4) $||Tv_{1}||>||v_{1}||,\quad v_{1}\in K\cap\partial\Omega_{r_{2}}.$

On the other hand, it can be obtained from
$\overline{f}_{i}^{\infty}\in(0,+\infty)$ that for any given
$\varepsilon_{2}>0$, there exists a constant $R_{1}>1$ such that for any
$t\in[0,1]$ and $i=1,\ldots,n$,

(3.5)
$f_{i}\left(t,v_{i+1}(t)\right)\leq(\overline{f}_{i}^{\infty}+\varepsilon_{2})v_{i+1}^{\beta_{i}},\quad
v_{i+1}\geq R_{1}.$

Furthermore, by the continuity of $f_{i}\ (i=1,\ldots,n)$, there exist
constants $M_{i}(R_{1})>0\ (i=1,\ldots,n)$ such that for any
$(t,v_{i+1}(t))\in[0,1]\times[0,R_{1}]$,

(3.6) $f_{i}\left(t,v_{i+1}(t)\right)\leq M_{i}(R_{1}),\quad i=1,\ldots,n.$

Combining (3.5) with (3.6), we have for any
$(t,v_{i+1}(t))\in[0,1]\times\left[0,+\infty\right)$,

(3.7) $f_{i}\left(t,v_{i+1}(t)\right)\leq
M_{i}(R_{1})+(\overline{f}_{i}^{\infty}+\varepsilon_{2})v_{i+1}^{\beta_{i}},\quad
i=1,\ldots,n.$

Then from the Lemma 3.2 and (3.7), we have for any $t\in[0,1]$,

$T_{i}(v_{i+1})(t)\leq\left[M_{i}(R_{1})+(\overline{f}_{i}^{\infty}+\varepsilon_{2})||v_{i+1}||^{\beta_{i}}\right]^{\frac{1}{k_{i}}},\quad
i=1,\ldots,n.$

Let

$\displaystyle
H:=M_{1}(R_{1})^{\frac{1}{k_{1}}}+\cdots+(\overline{f}_{1}^{\infty}+\varepsilon_{2})^{\frac{1}{k_{1}}}\cdots(\overline{f}_{n-1}^{\infty}+\varepsilon_{2})^{\frac{\prod_{i=1}^{n-2}\beta_{i}}{\prod_{i=1}^{n-1}k_{i}}}M_{n}(R_{1})^{\frac{\prod_{i=1}^{n-1}\beta_{i}}{\prod_{i=1}^{n}k_{i}}},$
$\displaystyle
L_{2}:=(\overline{f}_{1}^{\infty}+\varepsilon_{2})^{\frac{1}{k_{1}}}\cdots(\overline{f}_{n}^{\infty}+\varepsilon_{2})^{\frac{\prod_{i=1}^{n-1}\beta_{i}}{\prod_{i=1}^{n}k_{i}}}.$

Thus, there exists a large constant $R_{2}$:

$R_{2}>{\rm
max}\left\\{R_{1},2H,(2L_{2})^{\frac{\prod_{i=1}^{n}k_{i}}{\prod_{i=1}^{n}k_{i}-\prod_{i=1}^{n}\beta_{i}}}\right\\}$

such that for any $v_{1}\in K\cap\partial\Omega_{R_{2}}$ and $t\in[0,1]$,

$\begin{split}Tv_{1}(t)=&T_{1}T_{2}\cdots T_{n}(v_{1})(t)\\\
\leq&\left[M_{1}(R_{1})+(\overline{f}_{1}^{\infty}+\varepsilon_{2})||T_{2}(v_{3})||^{\beta_{1}}\right]^{\frac{1}{k_{1}}}\\\
\leq&M_{1}(R_{1})^{\frac{1}{k_{1}}}+\left[(\overline{f}_{1}^{\infty}+\varepsilon_{2})||T_{2}(v_{3})||^{\beta_{1}}\right]^{\frac{1}{k_{1}}}\\\
\leq&M_{1}(R_{1})^{\frac{1}{k_{1}}}+(\overline{f}_{1}^{\infty}+\varepsilon_{2})^{\frac{1}{k_{1}}}M_{2}(R_{1})^{\frac{\beta_{1}}{k_{1}k_{2}}}+(\overline{f}_{1}^{\infty}+\varepsilon_{2})^{\frac{1}{k_{1}}}(\overline{f}_{2}^{\infty}+\varepsilon_{2})^{\frac{\beta_{1}}{k_{1}k_{2}}}||v_{3}||^{\frac{\beta_{1}\beta_{2}}{k_{1}k_{2}}}\\\
&\qquad\vdots\\\
\leq&H+L_{2}||v_{1}||^{\frac{\prod_{i=1}^{n}\beta_{i}}{\prod_{i=1}^{n}k_{i}}}.\end{split}$

Since $\prod_{i=1}^{n}\beta_{i}<\prod_{i=1}^{n}k_{i}$, we get that for any
$v_{1}\in K\cap\partial\Omega_{R_{2}}$,

$\frac{H+L_{2}||v_{1}||^{\frac{\prod_{i=1}^{n}\beta_{i}}{\prod_{i=1}^{n}k_{i}}}}{||v_{1}||}=\frac{H}{||v_{1}||}+\frac{L_{2}}{||v_{1}||^{\frac{\prod_{i=1}^{n}k_{i}-\prod_{i=1}^{n}\beta_{i}}{\prod_{i=1}^{n}k_{i}}}}<1,$

which implies that

(3.8) $||Tv_{1}||<||v_{1}||,\quad v_{1}\in K\cap\partial\Omega_{R_{2}}.$

Therefore, combining with (3.4) and (3.8), it follows from Theorem 2.1 that
$T$ has at least one fixed point in
$K\cap\left(\overline{\Omega}_{R_{2}}\setminus\Omega_{r_{2}}\right)$.

(b). By the assumption of $\overline{f}_{i}^{0}\in(0,+\infty)$, for any given
$\eta_{1}>0$, there exists a positive constant $r_{3}<1$ such that for any
$t\in[0,1]$ and $i=1,\ldots,n$,

(3.9)
$f_{i}\left(t,v_{i+1}(t)\right)\leq(\overline{f}_{i}^{0}+\eta_{1})v_{i+1}^{\alpha_{i}},\quad
v_{i+1}\in[0,r_{3}].$

Let

$L_{3}:=(\overline{f}_{1}^{0}+\eta_{1})^{\frac{1}{k_{1}}}(\overline{f}_{2}^{0}+\eta_{1})^{\frac{\alpha_{1}}{k_{1}k_{2}}}\cdots(\overline{f}_{n}^{0}+\eta_{1})^{\frac{\prod_{i=1}^{n-1}\alpha_{i}}{\prod_{i=1}^{n}k_{i}}}.$

Since $\overline{f}_{i}^{0}\in(0,+\infty)$, there exists another constant
$r_{4}$:

$0<r_{4}<\min\left\\{r_{3},L_{3}^{\frac{\prod_{i=1}^{n}k_{i}}{\prod_{i=1}^{n}k_{i}-\prod_{i=1}^{n}\alpha_{i}}}\right\\}$

such that for $i=2,\ldots,n$,

(3.10) $f_{i}(t,v_{i+1}(t))\leq
r_{3}^{k_{i}},\quad(t,v_{i+1}(t))\in[0,1]\times[0,r_{4}].$

Then for any $v_{1}\in K\cap\partial\Omega_{r_{4}}$, it follows from Lemma 3.2
and (3.10) that

$v_{i}(t)=T_{i}(v_{i+1})(t)\leq
r_{3},\quad(t,v_{i+1}(t))\in[0,1]\times[0,r_{4}],\quad i=2,\ldots,n.$

Thus, By Lemma 3.2 and (3.9), we get

$T_{i}(v_{i+1})(t)\leq\left[(\overline{f}_{i}^{0}+\eta_{1})||v_{i+1}||^{\alpha_{i}}\right]^{\frac{1}{k_{i}}},\quad
i=1,\ldots,n,$

for any $t\in[0,1]$. For $v_{1}\in K\cap\partial\Omega_{r_{4}}$, we have

$\begin{split}||Tv_{1}||=&\sup_{t\in[0,1]}\left|T_{1}T_{2}\cdots
T_{n}(v_{1})(t)\right|\\\
\leq&\left[(\overline{f}_{1}^{0}+\eta_{1})||T_{2}(v_{3})||^{\alpha_{1}}\right]^{\frac{1}{k_{1}}}\\\
\leq&(\overline{f}_{1}^{0}+\eta_{1})^{\frac{1}{k_{1}}}\left[(\overline{f}_{2}^{0}+\eta_{1})||T_{3}(v_{4})||^{\alpha_{2}}\right]^{\frac{\alpha_{1}}{k_{1}k_{2}}}\\\
\leq&(\overline{f}_{1}^{0}+\eta_{1})^{\frac{1}{k_{1}}}(\overline{f}_{2}^{0}+\eta_{1})^{\frac{\alpha_{1}}{k_{1}k_{2}}}\left[(\overline{f}_{3}^{0}+\eta_{1})||T_{4}(v_{5})||^{\alpha_{3}}\right]^{\frac{\alpha_{1}\alpha_{2}}{k_{1}k_{2}k_{3}}}\\\
&\qquad\vdots\\\
\leq&L_{3}||v_{1}||^{\frac{\prod_{i=1}^{n}\alpha_{i}}{\prod_{i=1}^{n}k_{i}}}.\end{split}$

Recalling that $\prod_{i=1}^{n}\alpha_{i}>\prod_{i=1}^{n}k_{i}$, then

$\frac{L_{3}||v_{1}||^{\frac{\prod_{i=1}^{n}\alpha_{i}}{\prod_{i=1}^{n}k_{i}}}}{||v_{1}||}=\frac{L_{3}}{||v_{1}||^{\frac{\prod_{i=1}^{n}k_{i}-\prod_{i=1}^{n}\alpha_{i}}{\prod_{i=1}^{n}k_{i}}}}<1,$

which implies that

(3.11) $||Tv_{1}||<||v_{1}||,\quad v_{1}\in K\cap\partial\Omega_{r_{4}}.$

On the other hand, it follows from $\underline{f}_{i}^{\infty}\in(0,+\infty)$
that for any given
$\eta_{2}\in(0,\min\\{\underline{f}_{i}^{\infty},i=1,\ldots,n\\})$, there
exists a constant $R_{3}>1$ such that

(3.12)
$f_{i}\left(t,v_{i+1}(t)\right)\geq(\underline{f}_{i}^{\infty}-\eta_{2})v_{i+1}^{\beta_{i}},\quad
v_{i+1}\geq R_{3},$

for any $t\in[0,1]$ and $i=1,\ldots,n$. Let

$L_{4}:=\Gamma_{1}\cdots\Gamma_{n}^{\frac{\prod_{i=1}^{n-1}\beta_{i}}{\prod_{i=1}^{n-1}k_{i}}}(\underline{f}_{1}^{\infty}-\eta_{2})^{\frac{1}{k_{1}}}\cdots(\underline{f}_{n}^{\infty}-\eta_{2})^{\frac{\prod_{i=1}^{n-1}\beta_{i}}{\prod_{i=1}^{n}k_{i}}}(\frac{1}{4})^{\frac{\beta_{1}}{k_{1}}+\cdots+\frac{\prod_{i=1}^{n}\beta_{i}}{\prod_{i=1}^{n}k_{i}}}.$

There exists another constant $R_{4}$:

(3.13)
$R_{4}>\left\\{4R_{3},L_{4}^{\frac{\prod_{i=1}^{n}k_{i}}{\prod_{i=1}^{n}k_{i}-\prod_{i=1}^{n}\beta_{i}}},L_{5}\right\\}$

such that for any $v_{1}\in K\cap\partial\Omega_{R_{4}}$, we have

(3.14) $\min_{\frac{1}{4}\leq
t\leq\frac{3}{4}}v_{1}(t)\geq\frac{1}{4}||v_{1}||=\frac{1}{4}R_{4}>R_{3},$

where

$L_{5}:=\max_{l\in\\{2,\ldots,n\\}}\left(\frac{4R_{3}}{\Gamma_{l}\cdots\Gamma_{n}^{\frac{\prod_{i=l}^{n-1}\beta_{i}}{\prod_{i=l}^{n-1}k_{i}}}(\underline{f}_{l}^{\infty}-\eta_{2})^{\frac{1}{k_{l}}}\cdots(\underline{f}_{n}^{\infty}-\eta_{2})^{\frac{\prod_{i=l}^{n-1}\beta_{i}}{\prod_{i=l}^{n}k_{i}}}(\frac{1}{4})^{\frac{\beta_{l}}{k_{l}}+\cdots+\frac{\prod_{i=l}^{n}\beta_{l}}{\prod_{i=l}^{n}k_{l}}}}\right)^{\frac{\prod_{i=l}^{n}k_{i}}{\prod_{i=l}^{n}\beta_{i}}}.$

Here in $L_{5}$, when $l=n$ we set
$\frac{\prod_{i=l}^{n-1}\beta_{i}}{\prod_{i=l}^{n-1}k_{i}}=1$, so that the
terms $\frac{\prod_{i=l}^{n-1}\beta_{i}}{\prod_{i=l}^{n-1}k_{i}}$ make sense
for all $i=2,\ldots,n$. Combining (3.12) and (3.13), it follows from Lemma 3.1
that for any $v_{1}\in K\cap\partial\Omega_{R_{4}}$,

(3.15) $\displaystyle||v_{n}||\geq
v_{n}(\frac{1}{4})=T_{n}(v_{1})(\frac{1}{4})\geq\Gamma_{n}(\underline{f}_{n}^{\infty}-\eta_{2})^{\frac{1}{k_{n}}}(\frac{1}{4})^{\frac{\beta_{n}}{k_{n}}}||v_{1}||^{\frac{\beta_{n}}{k_{n}}}>4R_{3},$
$\displaystyle||v_{n-1}||\geq
v_{n-1}(\frac{1}{4})=T_{n-1}(v_{n})(\frac{1}{4})\geq\Gamma_{n-1}(\underline{f}_{n-1}^{\infty}-\eta_{2})^{\frac{1}{k_{n-1}}}(\frac{1}{4})^{\frac{\beta_{n-1}}{k_{n-1}}}||v_{n}||^{\frac{\beta_{n-1}}{k_{n-1}}}>4R_{3},$
$\displaystyle\qquad\vdots$ $\displaystyle||v_{2}||\geq
v_{2}(\frac{1}{4})=T_{2}(v_{3})(\frac{1}{4})\geq\Gamma_{2}(\underline{f}_{2}^{\infty}-\eta_{2})^{\frac{1}{k_{2}}}(\frac{1}{4})^{\frac{\beta_{2}}{k_{2}}}||v_{3}||^{\frac{\beta_{2}}{k_{2}}}>4R_{3}.$

From (3.14) and (3.15), we get that for any $v_{1}\in
K\cap\partial\Omega_{R_{4}}$,

$\min_{\frac{1}{4}\leq t\leq\frac{3}{4}}v_{i}(t)\geq\frac{1}{4}||v_{i}||\geq
R_{3},\quad i=1,\ldots,n.$

Then by Lemma 3.1, we deduce that

$\begin{split}Tv_{1}(\frac{1}{4})=&T_{1}T_{2}\cdots
T_{n}(v_{1})(\frac{1}{4})\\\
\geq&\Gamma_{1}(\underline{f}_{1}^{\infty}-\eta_{2})^{\frac{1}{k_{1}}}(\frac{1}{4})^{\frac{\beta_{1}}{k_{1}}}||T_{2}(v_{3})||^{\frac{\beta_{1}}{k_{1}}}\\\
\geq&\Gamma_{1}(\underline{f}_{1}^{\infty}-\eta_{2})^{\frac{1}{k_{1}}}(\frac{1}{4})^{\frac{\beta_{1}}{k_{1}}}\left(\Gamma_{2}(\underline{f}_{2}^{\infty}-\eta_{2})^{\frac{1}{k_{2}}}(\frac{1}{4})^{\frac{\beta_{2}}{k_{2}}}||T_{3}(v_{4})||^{\frac{\beta_{2}}{k_{2}}}\right)^{\frac{\beta_{1}}{k_{1}}}\\\
&\qquad\vdots\\\
\geq&L_{4}||v_{1}||^{\frac{\prod_{i=1}^{n}\beta_{i}}{\prod_{i=1}^{n}k_{i}}}.\end{split}$

Since $\prod_{i=1}^{n}\beta_{i}>\prod_{i=1}^{n}k_{i}$, it follows from

$\frac{L_{4}||v_{1}||^{\frac{\prod_{i=1}^{n}\beta_{i}}{\prod_{i=1}^{n}k_{i}}}}{||v_{1}||}=\frac{L_{4}}{||v_{1}||^{\frac{\prod_{i=1}^{n}k_{i}-\prod_{i=1}^{n}\beta_{i}}{\prod_{i=1}^{n}k_{i}}}}>1$

that

(3.16) $||Tv_{1}||>||v_{1}||,\quad v_{1}\in K\cap\partial\Omega_{R_{4}}.$

Therefore, from Theorem 2.1 combining (3.11) and (3.16), we obtain that $T$
has at least one fixed point in
$K\cap\left(\overline{\Omega}_{R_{4}}\setminus\Omega_{r_{4}}\right)$. ∎

###  3.2. Multiplicity

In Section 3.1, applying the fixed-point theorem in Theorem 2.1, we achieve
the existence result in a cone with different combinations of asymptotic
growth condition and relations of $\alpha_{i},\beta_{i}$ and $k_{i}$. In order
to obtain the multiplicity of nontrivial radial convex solutions of (1.1), we
recombine the conditions in Theorem 1.1 and find two kinds of “intermediate
state” as in (3.17) and (3.19).

###### Proof of Theorem 1.2.

(c). As we assumed, for any $(t,v_{1}(t))\in[0,1]\times[0,\frac{r_{0}}{4}]$,
we have

$||v_{1}||\leq 4\min_{\frac{1}{4}\leq t\leq\frac{3}{4}}v_{1}(t)\leq r_{0}.$

Then for $v_{1}\in K\cap\partial\Omega_{r_{0}}$, by the definition of $G_{n}$,
we have

$\begin{split}v_{n}(t)=T_{n}(v_{1})(t)\leq&\int_{0}^{1}\left(\frac{k_{n}}{\tau^{N-k_{n}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{n}-1}}f_{n}\left(s,v_{1}(s)\right)\,{\rm
d}s\right)^{\frac{1}{k_{n}}}\,{\rm d}\tau\\\
\leq&\int_{0}^{1}\left(\frac{k_{n}}{\tau^{N-k_{n}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{n}-1}}G_{n}\right)^{\frac{1}{k_{n}}}\,{\rm
d}\tau\\\
=&\frac{1}{2}\left(\frac{k_{n}G_{n}}{NC_{N-1}^{k_{n}-1}}\right)^{\frac{1}{k_{n}}}\\\
<&G_{n}^{\frac{1}{k_{n}}},\end{split}$

for any $t\in[0,1]$. Similarly, for $v_{1}\in K\cap\partial\Omega_{r_{0}}$, we
have $v_{i}(t)\leq G_{i}^{\frac{1}{k_{i}}}\ (i=2,\ldots,n-1)$, $\forall
t\in[0,1]$. Therefore, by the definition of $G_{1}$, we have

$\begin{split}Tv_{1}(t)=&T_{1}T_{2}\cdots T_{n}(v_{1})(t)\\\
=&\int_{t}^{1}\left(\frac{k_{1}}{\tau^{N-k_{1}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{1}-1}}f_{1}\left(s,v_{2}(s)\right)\,{\rm
d}s\right)^{\frac{1}{k_{1}}}\,{\rm d}\tau\\\
\leq&\int_{0}^{1}\left(\frac{k_{1}}{\tau^{N-k_{1}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{1}-1}}G_{1}\,{\rm
d}s\right)^{\frac{1}{k_{1}}}\,{\rm d}\tau\\\
=&\frac{1}{2}\left(\frac{k_{1}G_{1}}{NC_{N-1}^{k_{1}-1}}\right)^{\frac{1}{k_{1}}}\\\
<&G_{1}^{\frac{1}{k_{1}}},\quad\forall t\in[0,1],\end{split}$

which implies that

(3.17) $||Tv_{1}||<||v_{1}||,\quad v_{1}\in K\cap\partial\Omega_{r_{0}}.$

Since $\prod_{i=1}^{n}\alpha_{i}<\prod_{i=1}^{n}k_{i}$,
$\prod_{i=1}^{n}\beta_{i}>\prod_{i=1}^{n}k_{i}$, it follows from Theorem 1.1
that there exist sufficient small constant $r_{2}\in(0,r_{0})$ and sufficient
large constant $R_{4}>r_{0}$ such that

(3.18) $||Tv_{1}||\geq||v_{1}||,\quad v_{1}\in
K\cap\partial\Omega_{r_{2}}\quad{\rm and}\quad||Tv_{1}||\geq||v_{1}||,\quad
v_{1}\in K\cap\partial\Omega_{R_{4}}.$

Note that for any $v_{1}\in K\cap\partial\Omega_{r_{0}}$,
$||Tv_{1}||\neq||v_{1}||$, which shows that $T$ has no fixed point in
$K\cap\partial\Omega_{r_{0}}$. Combining (3.17) and (3.18), it follows from
Theorem 2.1 that there exist at least two fixed points of $T$ in
$K\cap\left(\overline{\Omega}_{r_{0}}\setminus\Omega_{r_{2}}\right)$ and
$K\cap\left(\overline{\Omega}_{R_{4}}\setminus\Omega_{r_{0}}\right)$
respectively.

(d). For $v_{1}\in K\cap\partial\Omega_{R_{0}}$, by the definition of
$\tilde{G}_{n}$, we have

$\begin{split}v_{n}(t)=T_{n}(v_{1})(t)\leq&\int_{0}^{1}\left(\frac{k_{n}}{\tau^{N-k_{n}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{n}-1}}f_{n}\left(s,v_{1}(s)\right)\,{\rm
d}s\right)^{\frac{1}{k_{n}}}\,{\rm d}\tau\\\
\leq&\int_{0}^{1}\left(\frac{k_{n}}{\tau^{N-k_{n}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{n}-1}}\tilde{G}_{n}\,{\rm
d}s\right)^{\frac{1}{k_{n}}}\,{\rm d}\tau\\\
=&\frac{1}{2}\left(\frac{k_{n}\tilde{G}_{n}}{NC_{N-1}^{k_{n}-1}}\right)^{\frac{1}{k_{n}}}\\\
<&\tilde{G}_{n}^{\frac{1}{k_{n}}},\quad\forall t\in[0,1].\end{split}$

Besides, by the definition of $E_{n}$, we get that for any $v_{1}\in
K\cap\partial\Omega_{R_{0}}$,

$\begin{split}v_{n}(\frac{1}{4})=T_{n}(v_{1})(\frac{1}{4})=&\int_{\frac{1}{4}}^{1}\left(\frac{k_{n}}{\tau^{N-k_{n}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{n}-1}}f_{n}\left(s,v_{1}(s)\right)\,{\rm
d}s\right)^{\frac{1}{k_{n}}}\,{\rm d}\tau\\\
\geq&\int_{\frac{1}{4}}^{\frac{3}{4}}\left(\frac{k_{n}}{\tau^{N-k_{n}}}\int_{\frac{1}{4}}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{n}-1}}E_{n}\,{\rm
d}s\right)^{\frac{1}{k_{n}}}\,{\rm d}\tau\\\
=&\Gamma_{n}E_{n}^{\frac{1}{k_{n}}},\end{split}$

then

$\min_{\frac{1}{4}\leq
t\leq\frac{3}{4}}v_{n}(t)\geq\frac{1}{4}||v_{n}||\geq\frac{1}{4}v_{n}(\frac{1}{4})\geq\frac{1}{4}\Gamma_{n}E_{n}^{\frac{1}{k_{n}}}.$

Thus for any $t\in[\frac{1}{4},\frac{3}{4}]$, we have
$\frac{1}{4}\Gamma_{n}E_{n}^{\frac{1}{k_{n}}}\leq
v_{n}(t)\leq\tilde{G}_{n}^{\frac{1}{k_{n}}}$. Repeating the above steps, we
have $\frac{1}{4}\Gamma_{i}E_{i}^{\frac{1}{k_{i}}}\leq
v_{i}(t)\leq\tilde{G}_{i}^{\frac{1}{k_{i}}}$, for any
$t\in[\frac{1}{4},\frac{3}{4}],\ (i=2,\ldots,n)$. It follows from the
assumption of $E_{1}$ that for any $v_{1}\in K\cap\partial\Omega_{R_{0}}$,

$\begin{split}Tv_{1}(\frac{1}{4})=&T_{1}T_{2}\cdots
T_{n}(v_{1})(\frac{1}{4})\\\
=&\int_{\frac{1}{4}}^{1}\left(\frac{k_{1}}{\tau^{N-k_{1}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{1}-1}}f_{1}\left(s,T_{2}(v_{3})(s)\right)\,{\rm
d}s\right)^{\frac{1}{k_{1}}}\,{\rm d}\tau\\\
\geq&\int_{\frac{1}{4}}^{\frac{3}{4}}\left(\frac{k_{1}}{\tau^{N-k_{1}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{1}-1}}E_{1}\,{\rm
d}s\right)^{\frac{1}{k_{1}}}\,{\rm d}\tau\\\
\geq&\Gamma_{1}E_{1}^{\frac{1}{k_{1}}}>R_{0},\end{split}$

which deduce that

(3.19) $||Tv_{1}||>||v_{1}||,\quad v_{1}\in K\cap\partial\Omega_{R_{0}}.$

Moreover, since
$\prod_{i=1}^{n}\alpha_{i}>\prod_{i=1}^{n}k_{i},\prod_{i=1}^{n}\beta_{i}<\prod_{i=1}^{n}k_{i}$,
we know from Theorem 1.1 that there exist sufficient small constant
$r_{4}\in(0,R_{0})$ and sufficient large constant $R_{2}>R_{0}$ such that

(3.20) $||Tv_{1}||\leq||v_{1}||,\quad v_{1}\in
K\cap\partial\Omega_{R_{2}}\quad{\rm and}\quad||Tv_{1}||\leq||v_{1}||,\quad
v_{1}\in K\cap\partial\Omega_{r_{4}}.$

Note that for $v_{1}\in K\cap\partial\Omega_{R_{0}}$, the norm of $Tv_{1}$ is
strictly greater than $||v_{1}||$, which shows that fixed point of $T$ can not
exists on $K\cap\partial\Omega_{R_{0}}$. Thus, basing on (3.19) and (3.20), it
follows from Theorem 2.1 that there exist at least two fixed points of $T$ in
$K\cap\left(\overline{\Omega}_{R_{0}}\setminus\Omega_{r_{4}}\right)$ and
$K\cap\left(\overline{\Omega}_{R_{2}}\setminus\Omega_{R_{0}}\right)$
respectively. ∎

##  4\. Uniqueness and nonexistence

In this section, we study the uniqueness and nonexistence results for a
special case of the system (1.1) where the nonlinearities are power functions
with respect to $u$.

###  4.1. Uniqueness

In [12], the authors gave a proof of uniqueness and approximation by
iterations of the solution to a general Dirichlet problem of Monge-Ampère
equation. Here, we will use their method to prove Theorem 1.3. We first
introduce the definition of $u_{0}$-sublinear operator and a corresponding
existence result.

######  Definition 4.1.

Let $P$ be a cone from a Banach space $Y$. With some $u_{0}\in P$ positive,
$A:P\to P$ is called $u_{0}$-sublinear if

  1. (i)

for any $x>0$, there exist positive constants $\theta_{1}$ and $\theta_{2}$
which depend on $x$, such that

$\theta_{1}u_{0}\leq Ax\leq\theta_{2}u_{0};$

  2. (ii)

for any $\theta_{1}u_{0}\leq x\leq\theta_{2}u_{0}$ and $0<\xi<1$, there always
exists some $\eta>0$ such that

$A(\xi x)\geq(1+\eta)\xi Ax.$

######  Lemma 4.1.

An increasing and $u_{0}$-sublinear operator $A$ can have at most one positive
fixed-point.

The proof can be found in [12], we omit it here.

###### Proof of Theorem 1.3.

Let $X:=C[0,1]$ and cone $P:=\\{v\in X:v(t)\geq 0,t\in[0,1]\\}$. It is easy to
see that $K\subset P$, where $K$ is defined in (2.3). We define $T_{i}\
(i=1,\ldots,n)$ and composite operator $T=T_{1}T_{2}\cdots T_{n}$ as in
Section 2. The existence of nontrivial radial convex solutions to system (1.2)
is obtained in Theorem 1.1 and therefore investigate $T$ has at most one
fixed-point in $K$ is enough. By Lemma 4.1, it suffices to verify that $T:K\to
K$ is an increasing and $u_{0}$-sublinear for some $u_{0}$ positive in
$C[0,1]$. By the definitions of $T_{i}$, it is clear that each $T_{i}\
(i=1,\ldots,n)$ is a increasing operator, so is the composite operator $T$,
then we just need to prove that $T$ satisfies the Definition 4.1.

Firstly, we show that $T$ satisfies the Definition 4.1 (i).

$\begin{split}Tv_{1}(t)=&\int_{t}^{1}\left(\frac{k_{1}}{\tau^{N-k_{1}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{1}-1}}v_{2}^{\gamma_{1}}(s)\,{\rm
d}s\right)^{\frac{1}{k_{1}}}\,{\rm d}\tau\\\
\leq&||v_{2}||^{\frac{\gamma_{1}}{k_{1}}}\int_{t}^{1}\left(\frac{k_{1}}{\tau^{N-k_{1}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{1}-1}}\,{\rm
d}s\right)^{\frac{1}{k_{1}}}\,{\rm d}\tau\\\
\leq&||T_{2}(v_{3})||^{\frac{\gamma_{1}}{k_{1}}}\left(\frac{k_{1}}{NC_{N-1}^{k_{1}-1}}\right)^{\frac{1}{k_{1}}}\int_{t}^{1}\tau\,{\rm
d}\tau\\\
\leq&\left[\int_{0}^{1}\left(\frac{k_{2}}{\tau^{N-k_{2}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{2}-1}}v_{3}^{\gamma_{2}}(s)\,{\rm
d}s\right)^{\frac{1}{k_{2}}}\,{\rm
d}\tau\right]^{\frac{\gamma_{1}}{k_{1}}}(1-t)\\\
\leq&||v_{3}||^{\frac{\gamma_{1}\gamma_{2}}{k_{1}k_{2}}}(1-t)\\\
&\qquad\vdots\\\
\leq&||v_{1}||^{\frac{\prod_{i=1}^{n}\gamma_{i}}{\prod_{i=1}^{n}k_{i}}}(1-t).\end{split}$

Here, let $u_{0}=1-t$, $t\in[0,1)$ and
$\theta_{2}=||v_{1}||^{\frac{\prod_{i=1}^{n}\gamma_{i}}{\prod_{i=1}^{n}k_{i}}}$,
then $Tv_{1}(t)\leq\theta_{2}u_{0}$. Set

$\Gamma:=\left(\frac{9}{32}\right)^{\frac{\gamma_{1}}{k_{1}}+\cdots+\frac{\prod_{i=1}^{n-1}\gamma_{i}}{\prod_{i=1}^{n-1}k_{i}}}\left(\frac{k_{1}}{4^{\gamma_{1}}NC_{N-1}^{k_{1}-1}}\right)^{\frac{1}{k_{1}}}\cdots\left(\frac{k_{n}}{4^{\gamma_{n}}NC_{N-1}^{k_{n}-1}}\right)^{\frac{\prod_{i=1}^{n-1}\gamma_{i}}{\prod_{i=1}^{n}k_{i}}}||v_{1}||^{\frac{\prod_{i=1}^{n}\gamma_{i}}{\prod_{i=1}^{n}k_{i}}}$

be a positive constant which depends only on $||v_{1}||$. Next, let
$c\in(\frac{1}{4},\frac{3}{4})$ be a fixed number. Notice that $Tv_{1}(t)$ is
decreasing with $t$, we have for $t\in[0,c)$,

$\begin{split}Tv_{1}(t)\geq&Tv_{1}(c)=\int_{c}^{1}\left(\frac{k_{1}}{\tau^{N-k_{1}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{1}-1}}v_{2}^{\gamma_{1}}(s)\,{\rm
d}s\right)^{\frac{1}{k_{1}}}\,{\rm d}\tau\\\
\geq&\int_{c}^{\frac{3}{4}}\left(\frac{k_{1}}{\tau^{N-k_{1}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{1}-1}}(\frac{1}{4}||v_{2}||)^{\gamma_{1}}\,{\rm
d}s\right)^{\frac{1}{k_{1}}}\,{\rm d}\tau\\\
\geq&\left(\frac{k_{1}}{4^{\gamma_{1}}NC_{N-1}^{k_{1}-1}}\right)^{\frac{1}{k_{1}}}\left(\frac{9}{32}-\frac{1}{2}c^{2}\right)||T_{2}(v_{3})||^{\frac{\gamma_{1}}{k_{1}}}\\\
&\qquad\qquad\vdots\\\
\geq&\Gamma\left(\frac{9}{32}-\frac{1}{2}c^{2}\right)\\\
\geq&\Gamma\left(\frac{9}{32}-\frac{1}{2}c^{2}\right)(1-t),\end{split}$

where we use the fact $\min_{0\leq t\leq\frac{3}{4}}v(t)\geq\frac{1}{4}||v||$
in the above inequality which follows from (2.3) combining with the concavity
of $v_{i}\ (i=1,\ldots,n)$ and $v_{i}^{\prime}(0)=0$. For $t\in[c,1)$, we let

$\zeta(\tau):=\left(\frac{1}{\tau^{N-k_{1}}}\int_{0}^{\tau}s^{N-1}(1-s)^{\gamma_{1}}\,{\rm
d}s\right)^{\frac{1}{k_{1}}},\ \tau\in[c,1].$

Notice that $\zeta(\tau)\in C[c,1]$ and $\zeta(\tau)>0,\ \tau\in[c,1]$ is
well-defined, then $\zeta([c,1])$ is the image of a compact set and so is
compact which shows that it is both closed and bounded. So $\zeta([c,1])$ has
a positive absolute minimum. Besides, by the concavity of $v_{i}(t)\
(i=1,\ldots,n)$ and $v_{i}^{\prime}(0)=v_{i}(1)=0$, we have

(4.1) $v_{i}(t)\geq v_{i}(0)(1-t),\quad\forall t\in[0,1].$

Then we have for $t\in[c,1)$,

$\begin{split}Tv_{1}(t)=&\int_{t}^{1}\left(\frac{k_{1}}{\tau^{N-k_{1}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{1}-1}}v_{2}^{\gamma_{1}}(s)\,{\rm
d}s\right)^{\frac{1}{k_{1}}}\,{\rm d}\tau\\\
\geq&\int_{t}^{1}\left(\frac{k_{1}}{\tau^{N-k_{1}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{1}-1}}[v_{2}(0)(1-s)]^{\gamma_{1}}\,{\rm
d}s\right)^{\frac{1}{k_{1}}}\,{\rm d}\tau\\\
=&||v_{2}||^{\frac{\gamma_{1}}{k_{1}}}\left(\frac{k_{1}}{C_{N-1}^{k_{1}-1}}\right)^{\frac{1}{k_{1}}}\int_{t}^{1}\left(\frac{1}{\tau^{N-k_{1}}}\int_{0}^{\tau}s^{N-1}(1-s)^{\gamma_{1}}\,{\rm
d}s\right)^{\frac{1}{k_{1}}}\,{\rm d}\tau\\\
\geq&||T_{2}(v_{3})||^{\frac{\gamma_{1}}{k_{1}}}\left(\frac{k_{1}}{C_{N-1}^{k_{1}-1}}\right)^{\frac{1}{k_{1}}}\min_{\tau\in[c,1]}\zeta(\tau)\int_{t}^{1}\,{\rm
d}\tau\\\
=&\left[\int_{0}^{1}\left(\frac{k_{2}}{\tau^{N-k_{2}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{2}-1}}v_{3}^{\gamma_{2}}(s)\,{\rm
d}s\right)^{\frac{1}{k_{2}}}\,{\rm
d}\tau\right]^{\frac{\gamma_{1}}{k_{1}}}\left(\frac{k_{1}}{C_{N-1}^{k_{1}-1}}\right)^{\frac{1}{k_{1}}}\min_{\tau\in[c,1]}\zeta(\tau)(1-t)\\\
\geq&\left[\int_{0}^{\frac{3}{4}}\left(\frac{k_{2}}{\tau^{N-k_{2}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{2}-1}}(\frac{1}{4}||v_{3}||)^{\gamma_{2}}\,{\rm
d}s\right)^{\frac{1}{k_{2}}}\,{\rm
d}\tau\right]^{\frac{\gamma_{1}}{k_{1}}}\left(\frac{k_{1}}{C_{N-1}^{k_{1}-1}}\right)^{\frac{1}{k_{1}}}\min_{\tau\in[c,1]}\zeta(\tau)(1-t)\\\
=&||T_{3}(v_{4})||^{\frac{\gamma_{1}\gamma_{2}}{k_{1}k_{2}}}\left(\frac{k_{1}}{C_{N-1}^{k_{1}-1}}\right)^{\frac{1}{k_{1}}}\left(\frac{k_{2}}{4^{\gamma_{2}}C_{N-1}^{k_{2}-1}}\right)^{\frac{\gamma_{1}}{k_{1}k_{2}}}\left(\frac{9}{32}\right)^{\frac{\gamma_{1}}{k_{1}}}\min_{\tau\in[c,1]}\zeta(\tau)(1-t)\\\
&\qquad\qquad\vdots\\\ \geq&\Gamma
4^{\frac{\gamma_{1}}{k_{1}}}\min_{\tau\in[c,1]}\zeta(\tau)(1-t).\end{split}$

Let $\theta_{1}={\rm
min}\left\\{\Gamma\left(\frac{9}{32}-\frac{1}{2}c^{2}\right),\Gamma
4^{\frac{\gamma_{1}}{k_{1}}}\min_{\tau\in[c,1]}\zeta(\tau)\right\\}$, then we
have $Tv_{1}(t)\geq\theta_{1}u_{0}$.

To verify the Definition 4.1 (ii), we have for any $\theta_{1}u_{0}\leq
v_{1}\leq\theta_{2}u_{0}$ and $\xi\in(0,1)$, $T_{1}(\xi
v_{2})=\xi^{\frac{\gamma_{1}}{k_{1}}}T_{1}(v_{2}),\ T_{2}(\xi
v_{3})=\xi^{\frac{\gamma_{2}}{k_{2}}}T_{2}(v_{3}),\ \ldots,\ T_{n}(\xi
v_{1})=\xi^{\frac{\gamma_{n}}{k_{n}}}T_{n}(v_{1})$. Notice that
$\prod_{i=1}^{n}\gamma_{i}<\prod_{i=1}^{n}k_{i}$, then there exists $\eta>0$
such that

$T(\xi v_{1})=T_{1}T_{2}\cdots T_{n}(\xi v_{1})=T_{1}T_{2}\cdots
T_{n-1}(\xi^{\frac{\gamma_{n}}{k_{n}}}T_{n}(v_{1}))=\cdots=\xi^{\frac{\prod_{i=1}^{n}\gamma_{i}}{\prod_{i=1}^{n}k_{i}}}Tv_{1}\geq(1+\eta)\xi
Tv_{1}.$

Thus $T$ is a $u_{0}$-sublinear operator and $T$ has at most one fixed-point
in $K$ by Lemma 4.1 which shows that the system (1.2) has a unique nontrivial
radial convex solution. ∎

######  Remark 4.1.

Note that we also use the convexity of $u_{i}=-v_{i}\ (i=1,\ldots,n)$ in this
subsection, namely, the inequality (4.1).

###  4.2. Nonexistence

In the case of $\prod_{i=1}^{n}\gamma_{i}=\prod_{i=1}^{n}k_{i}$, we can get
nonexistence result by contradiction.

###### Proof of Theorem 1.4.

Suppose, to the contrary, that $v_{0}$ is a fixed-point of $T$ in $K$, then
$Tv_{0}=v_{0}$. It follows immediately from the definition of $T$ that $v_{0}$
is a concave function satisfying $v_{0}(1)=0$ and $v_{0}(t)>0,t\in[0,1)$.

On the other hand, for any $v_{1}\in K$, we have

$\begin{split}||T(v_{1})||=&\int_{0}^{1}\left(\frac{k_{1}}{\tau^{N-k_{1}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{1}-1}}v_{2}^{\gamma_{1}}(s)\,{\rm
d}s\right)^{\frac{1}{k_{1}}}\,{\rm d}\tau\\\
\leq&||v_{2}||^{\frac{\gamma_{1}}{k_{1}}}\int_{0}^{1}\left(\frac{k_{1}}{\tau^{N-k_{1}}}\int_{0}^{\tau}\frac{s^{N-1}}{C_{N-1}^{k_{1}-1}}\,{\rm
d}s\right)^{\frac{1}{k_{1}}}\,{\rm d}\tau\\\
=&\frac{1}{2}\left(\frac{k_{1}}{NC_{N-1}^{k_{1}-1}}\right)^{\frac{1}{k_{1}}}||v_{2}||^{\frac{\gamma_{1}}{k_{1}}}\\\
<&||v_{2}||^{\frac{\gamma_{1}}{k_{1}}}\\\ &\qquad\vdots\\\
<&||v_{1}||^{\frac{\prod_{i=1}^{n}\gamma_{i}}{\prod_{i=1}^{n}k_{i}}}=||v_{1}||.\end{split}$

Here, we also use the fact
$\frac{1}{2}\left(\frac{k_{1}}{NC_{N-1}^{k_{1}-1}}\right)^{\frac{1}{k_{1}}}<1$
for the same reason in (3.1). Thus if we take $v_{1}=v_{0}$ in the above
estimate, we have $||v_{0}||$ is strictly larger than $||Tv_{0}||$. This
contradicts $Tv_{0}=v_{0}$ and concludes the proof. ∎

######  Remark 4.2.

Due to the fact
$\frac{1}{2}\left(\frac{k_{1}}{NC_{N-1}^{k_{1}-1}}\right)^{\frac{1}{k_{1}}}<1$,
we have a direct proof of the nonexistence theorem by reduction to absurdity
without using the fixed-point theorem in Theorem 2.1. Therefore, we can obtain
the nonexistence for $\boldsymbol{k}$-admissible solutions of system (1.2) in
the assumption of $\prod_{i=1}^{n}\gamma_{i}=\prod_{i=1}^{n}k_{i}$, (not just
for the convex solutions of system (1.2)).

##  5\. Eigenvalue problem

In the previous section, we proved the nonexistence of nontrivial radial
convex solution to the power-type system (1.2) in a unit ball. Then by
imposing a suitable condition on positive parameters of eigenvalue problem
(1.3), we can also get the existence of $\boldsymbol{k}$-admissible solution
in a general strictly $(k-1)$-convex domain. In this section, our main tool is
the generalized Krein-Rutman theorem in [13].

We first recall some basic concepts:

Let $E$ be a Banach space, $M\subset E$ be a cone.

######  Definition 5.1.

The cone $M$ introduces a partial order in $E$ by the relation

$u\prec v$ if and only if $u-v\in M$.

######  Definition 5.2.

Define an operator $A:E\to E$. Then

  1. (i)

$A$ is called positive if $A(M)\subset M$;

  2. (ii)

$A$ is said to be homogeneous if it is positively homogeneous with degree 1;

  3. (iii)

$A$ is monotone if it satisfies $x\prec y\Rightarrow A(x)\prec A(y)$;

  4. (iv)

$A$ is called strong (relative to $M$), if for all $u,v\in Im(A)\cap
M\setminus\\{\theta\\}$, there exist positive constants $\delta$ and $\gamma$
that depend on $u$ and $v$ such that $u-\delta v\in M$, $v-\gamma u\in M$.

The following is the generalized Krein-Rutman theorem developed by Jacobsen in
[13].

######  Lemma 5.1.

Let $E$ contain a cone $M$, $A:E\to E$ be a completely continuous operator
with $A|_{M}:M\to M$ homogeneous, monotone, and strong. Furthermore, assume
that there exists a nonzero element $\omega$, $A(\omega)\in Im(A)\cap M$. Then
there exists a constant $\lambda_{0}>0$ with the following properties:

  1. (i)

There exists $u\in M\setminus\\{\theta\\}$, with $u=\lambda_{0}A(u)$;

  2. (ii)

If $v\in M\setminus\\{\theta\\}$ and $\lambda>0$ such that $v=\lambda A(v)$,
then $\lambda=\lambda_{0}$.

For the convenience of the reader, we also present the existence theorems in
[20, 25].

######  Lemma 5.2.

(see [20]) Let $\Omega$ be a uniformly $(k-1)$-convex domain in
$\mathbb{R}^{N}$, $k=2,\ldots,N$, $\varphi\in C^{0}(\overline{\Omega})$ and
$\psi\geq 0,\in L^{p}(\Omega)$, for $p>\frac{N}{2k}$. Then there exists a
unique admissible weak solution $u\in C^{0}(\overline{\Omega})$ to the problem

$\left\\{\begin{aligned} &S_{k}\left(D^{2}u\right)=\psi,&{\rm in}\ \Omega,\\\
&u=\varphi,&{\rm on}\ \partial\Omega.\end{aligned}\right.$

######  Lemma 5.3.

(see [25]) Assume that $\Omega$ is $(k-1)$-convex, $\varphi$, $\Omega\in
C^{3,1}$, $f\in C^{1,1}(\overline{\Omega})$, and $f\geq f_{0}>0$. Then there
is a unique $k$-admissible solution $u\in C^{3,\alpha}(\overline{\Omega})$ to
the Dirichlet problem

$\left\\{\begin{aligned} &S_{k}\left(D^{2}u\right)=f(x),&{\rm in}\ \Omega,\\\
&u=\varphi,&{\rm on}\ \partial\Omega.\end{aligned}\right.$

###### Proof of Theorem 1.5.

Let $X$ be a Banach space $C(\overline{\Omega})$ equipped with the supremum
norm. Define a cone $P:=\left\\{u\in X:u(x)\leq 0,\forall
x\in\Omega\right\\}$. Then by the Definition 5.1, we notice that the partial
order induced by $P$ implies that $u\prec v\Longleftrightarrow u(x)\leq
v(x),\forall x\in\Omega$.

For $i=1,\ldots,n$, we define $\overline{T}_{i}:X\to X$,
$\overline{T}_{i}(u_{i+1})=u_{i}$, where $u_{i}$ is the unique admissible weak
solution of the problem

(5.1) $\left\\{\begin{aligned}
&S_{k_{i}}\left(D^{2}u_{i}\right)=|u_{i+1}|^{\gamma_{i}},&{\rm in}\ \Omega,\\\
&u_{i}=0,&{\rm on}\ \partial\Omega.\end{aligned}\right.$

Notice that we denote $u_{1}:=u_{n+1}$ here. It follows from Lemma 5.2 that
the admissible weak solution $\overline{T}_{i}(u_{i+1})\in
C^{0}(\overline{\Omega})(i=1,\ldots,n)$. Define a composite operator
$\overline{T}:=\overline{T}_{1}\overline{T}_{2}\cdots\overline{T}_{n}$, which
is a completely continuous operator. Next, we verify that $\overline{T}$
satisfies the assumptions of Lemma 5.1.

Due to the $k$-convexity property of the admissible weak solution and the
boundary data, we have $\overline{T}(X)\subseteq P$, which implies that
$\overline{T}$ is positive and the operator $\overline{T}$ maps $P$ into
itself. For $t>0$, we have

$\overline{T}_{1}(tu_{2})=t^{\frac{\gamma_{1}}{k_{1}}}\overline{T}_{1}(u_{2}),\quad\overline{T}_{2}(tu_{3})=t^{\frac{\gamma_{2}}{k_{2}}}\overline{T}_{2}(u_{3}),\quad\cdots,\quad\overline{T}_{n}(tu_{1})=t^{\frac{\gamma_{n}}{k_{n}}}\overline{T}_{n}(u_{1}).$

Since the assumption $\prod_{i=1}^{n}\gamma_{i}=\prod_{i=1}^{n}k_{i}$, we
deduce that

$\overline{T}(tu_{1})=t^{\frac{\prod_{i=1}^{n}\gamma_{i}}{\prod_{i=1}^{n}k_{i}}}\overline{T}(u_{1})=t\overline{T}(u_{1}),$

which implies that $\overline{T}$ is homogeneous. By comparison principle in
Lemma 2.1 in [20] and the definition of $\overline{T}_{i}$, we get that
$\overline{T}_{i}\ (i=1,\ldots,n)$ are all monotone, so is $\overline{T}$.
Finally, we just have to verify that $\overline{T}$ is strong, that is, for
all $u,v\in Im(\overline{T})\cap P\setminus\\{\theta\\}$, there exist
$\delta>0$ and $\gamma>0$ such that $u-\delta v\leq 0$ in $\Omega$ and
$v-\gamma u\leq 0$ in $\Omega$. If $u\in Im(\overline{T})\cap
P\setminus\\{\theta\\}$, then there exists a function $v\in
X\setminus\\{\theta\\}$ such that
$u=\overline{T}v=\overline{T}_{1}\overline{T}_{2}\cdots\overline{T}_{n}(v)$,
where $u$ is nonzero $k$-admissible and strictly negative in $\Omega$
satisfying

$\left\\{\begin{aligned}
&S_{k_{1}}\left(D^{2}u\right)=|v|^{\gamma{{}_{1}}},&{\rm in}\ \Omega,\\\
&u=0,&{\rm on}\ \partial\Omega.\end{aligned}\right.$

It follows from Lemma 5.2 that $u=\overline{T}v\in C^{0}(\overline{\Omega})$.
Notice that $v$ is also the solution of (5.1) satisfying $v\in
C^{0}(\overline{\Omega})$, we let $v$ attains its minimum at
$x_{0}\in\overline{\Omega}$ and
$G:=\max|v|^{\gamma_{1}}=(-v(x_{0}))^{\gamma_{1}}$, then we have
$0<|v|^{\gamma_{1}}\leq G$ in $\Omega$. Consider a function $\omega$ which
satisfies

$\left\\{\begin{aligned} &S_{k_{1}}\left(D^{2}\omega\right)=G,&{\rm in}\
\Omega,\\\ &\omega=0,&{\rm on}\ \partial\Omega.\end{aligned}\right.$

Then, it follows from Lemma 5.3 that $\omega\in
C^{2,\alpha}(\overline{\Omega})$. By comparison principle in Lemma 2.1 in
[20], we have $\omega\leq u\leq 0$, in $\overline{\Omega}$ and $\omega=u=0$,
on $\partial\Omega$. Thus, for some small $t>0$, we have

$0\leq\frac{u(x-t\nu)-u(x)}{-t}\leq\frac{\omega(x-t\nu)-\omega(x)}{-t},\quad{\rm
for}\ x\in\partial\Omega,$

where $\nu$ is the unit outer normal vector field on $\partial\Omega$. Take a
limit in the last inequality, we have

$\displaystyle 0\leq\limsup_{t\to 0^{+}}\frac{u(x-t\nu)-u(x)}{-t}\leq$
$\displaystyle\limsup_{t\to 0^{+}}\frac{\omega(x-t\nu)-\omega(x)}{-t}$
$\displaystyle=$ $\displaystyle\frac{\partial\omega(x)}{\partial\nu},\quad{\rm
for}\ x\in\partial\Omega.$

With the same argument, there also exists $\tilde{\omega}\in
C^{2,\alpha}(\overline{\Omega})$ such that

(5.2) $\displaystyle 0\leq\limsup_{t\to 0^{+}}\frac{v(x-t\nu)-v(x)}{-t}\leq$
$\displaystyle\limsup_{t\to
0^{+}}\frac{\tilde{\omega}(x-t\nu)-\tilde{\omega}(x)}{-t}$ $\displaystyle=$
$\displaystyle\frac{\partial\tilde{\omega}(x)}{\partial\nu},\quad{\rm for}\
x\in\partial\Omega,$

then by Hopf Lemma in [11], we have

(5.3) $\liminf_{t\to 0^{+}}\frac{u(x-t\nu)-u(x)}{-t}>0,\quad{\rm on}\
\partial\Omega.$

By choosing a sufficiently small constant $\delta_{1}>0$, we have

$\displaystyle\liminf_{t\to
0^{+}}\frac{(u-\delta_{1}v)(x-t\nu)-(u-\delta_{1}v)(x)}{-t}$ $\displaystyle=$
$\displaystyle\liminf_{t\to
0^{+}}\frac{u(x-t\nu)-u(x)}{-t}-\delta_{1}\limsup_{t\to
0^{+}}\frac{v(x-t\nu)-v(x)}{-t}$ $\displaystyle>$ $\displaystyle 0,\quad{\rm
on}\ \partial\Omega,$

where (5.2) and (5.3) are used in the last inequality. Since $u,v$ are the
solutions of (5.1) which satisfy $u=v=0$ on $\partial\Omega$. Then for
$x\in\partial\Omega$, there exists a constant $t_{0}>0$ such that

(5.4) $(u-\delta_{1}v)(x-t\nu)<0,\quad{\rm for}\ t<t_{0}.$

Now, (5.4) implies that

$u-\delta_{1}v<0,\quad{\rm in}\ \Omega_{t_{0}}:=\\{x\in\Omega|{\rm
dist}(x,\partial\Omega)<t_{0}\\},$

where $dist(x,\partial\Omega)$ denotes the distance from $x$ to
$\partial\Omega$. For $x\in\Omega\setminus\Omega_{t_{0}}$, we set

$\delta_{2}:=\inf_{x\in\Omega\setminus\Omega_{t_{0}}}\frac{u(x)}{v(x)}>0.$

Fixing a constant $\delta\leq\min\\{\delta_{1},\delta_{2}\\}$, then we have

$u-\delta v\leq u-\delta_{1}v<0,\quad{\rm in}\ \Omega_{t_{0}}\quad{\rm
and}\quad u-\delta v\leq u-\delta_{2}v\leq 0,\quad{\rm in}\
\Omega\setminus\Omega_{t_{0}},$

which implies

$u-\delta v\leq 0,\quad{\rm in}\ \Omega.$

The same argument shows that there exists a constant $\gamma$ such that
$v-\gamma u\leq 0$ in $\Omega$. Now we have shown that $\overline{T}$ is
strong. Moreover, $\mathcal{N}(\overline{T}):=\\{u\in
P|\overline{T}(u)=0\\}=\\{0\\}$.

Combining this with Lemma 5.1 (i), we obtain that there exists $u_{1}^{*}\in
P\setminus\\{\theta\\}$ and constant $\lambda_{0}>0$ such that
$u_{1}^{*}=\lambda_{0}\overline{T}(u_{1}^{*})=\lambda_{0}\overline{T}_{1}\overline{T}_{2}\cdots\overline{T}_{n}(u_{1}^{*})$.
Let
$u_{n}^{*}=\overline{T}_{n}(u_{1}^{*}),\ldots,u_{2}^{*}=\overline{T}_{2}(u_{3}^{*})$.
Then $(u_{1}^{*},u_{2}^{*},\ldots,u_{n}^{*})$ is a solution of the following
system

$\left\\{\begin{aligned}
&S_{k_{1}}\left(D^{2}(\frac{u_{1}}{\lambda_{0}})\right)=\left(-u_{2}\right)^{\gamma_{1}},&{\rm
in}\ \Omega,\\\
&S_{k_{2}}\left(D^{2}u_{2}\right)=\left(-u_{3}\right)^{\gamma_{2}},&{\rm in}\
\Omega,\\\ &\qquad\qquad\quad\vdots\\\
&S_{k_{n-1}}\left(D^{2}u_{n-1}\right)=\left(-u_{n}\right)^{\gamma_{n-1}},&{\rm
in}\ \Omega,\\\
&S_{k_{n}}\left(D^{2}u_{n}\right)=\left(-u_{1}\right)^{\gamma_{n}},&{\rm in}\
\Omega,\\\ &u_{i}=0,i=1,\ldots,n,&{\rm on}\
\partial\Omega.\end{aligned}\right.$

By Lemma 5.1 (ii), if there exist $u_{0}\in P\setminus\\{\theta\\}$ and
$\lambda_{1}>0$ such that $u_{0}=\lambda_{1}\overline{T}(u_{0})$, then
$\lambda_{1}=\lambda_{0}$.

For this reason, the eigenvalue problem

(5.5) $\left\\{\begin{aligned}
&S_{k_{1}}\left(D^{2}u_{1}\right)=\tilde{\lambda}\left(-u_{2}\right)^{\gamma_{1}},&{\rm
in}\ \Omega,\\\
&S_{k_{2}}\left(D^{2}u_{2}\right)=\left(-u_{3}\right)^{\gamma_{2}},&{\rm in}\
\Omega,\\\ &\qquad\qquad\quad\vdots\\\
&S_{k_{n-1}}\left(D^{2}u_{n-1}\right)=\left(-u_{n}\right)^{\gamma_{n-1}},&{\rm
in}\ \Omega,\\\
&S_{k_{n}}\left(D^{2}u_{n}\right)=\left(-u_{1}\right)^{\gamma_{n}},&{\rm in}\
\Omega,\\\ &u_{i}=0,i=1,\ldots,n,&{\rm on}\
\partial\Omega,\end{aligned}\right.$

admits a solution ($\boldsymbol{k}$-admissible solution) if and only if
$\tilde{\lambda}=\lambda_{0}^{k_{1}}$.

Next, we prove that the system (1.3) has a nonzero $\boldsymbol{k}$-admissible
solution if and only if

$\lambda_{1}\lambda_{2}^{\frac{\gamma_{1}}{k_{2}}}\cdots\lambda_{n}^{\frac{\prod_{i=1}^{n-1}\gamma_{i}}{\prod_{i=2}^{n}k_{i}}}=\lambda_{0}^{k_{1}}.$

In fact, if $(u_{1},\ldots,u_{n})$ is a $\boldsymbol{k}$-admissible solution
of the system (1.3), then

$S_{k_{n}}(D^{2}u_{n})=\lambda_{n}(-u_{1})^{\gamma_{n}},$

which implies that

$S_{k_{n}}\left(D^{2}(\lambda_{n}^{-\frac{1}{k_{n}}}u_{n})\right)=(-u_{1})^{\gamma_{n}},$

Let $\tilde{u}_{n}=\lambda_{n}^{-\frac{1}{k_{n}}}u_{n}$, we have
$S_{k_{n}}(D^{2}\tilde{u}_{n})=(-u_{1})^{\gamma_{n}}$ and

$S_{k_{n-1}}(D^{2}u_{n-1})=\lambda_{n-1}(-u_{n})^{\gamma_{n-1}}=\lambda_{n-1}\lambda_{n}^{\frac{\gamma_{n-1}}{k_{n}}}(-\tilde{u}_{n})^{\gamma_{n-1}}.$

By the same argument, we let

$\displaystyle\tilde{u}_{n-1}=\lambda_{n-1}^{-\frac{1}{k_{n-1}}}\lambda_{n}^{-\frac{\gamma_{n-1}}{k_{n-1}k_{n}}}u_{n-1},$
$\displaystyle\qquad\vdots$
$\displaystyle\tilde{u}_{2}=\lambda_{2}^{-\frac{1}{k_{2}}}\lambda_{3}^{-\frac{\gamma_{2}}{k_{2}k_{3}}}\cdots\lambda_{n}^{-\frac{\prod_{i=2}^{n-1}\gamma_{i}}{\prod_{i=2}^{n}k_{i}}}u_{2},$

therefore we have
$S_{k_{n-1}}(D^{2}\tilde{u}_{n-1})=(-\tilde{u}_{n})^{\gamma_{n-1}}$ ,$\cdots$,
$S_{k_{1}}(D^{2}u_{1})=\lambda_{1}\lambda_{2}^{\frac{\gamma_{1}}{k_{2}}}\cdots\lambda_{n}^{\frac{\prod_{i=1}^{n-1}\gamma_{i}}{\prod_{i=2}^{n}k_{i}}}(-\tilde{u}_{2})^{\gamma_{1}}$.
From the previous discussion, we know that (5.5) admits a
$\boldsymbol{k}$-admissible solution if and only if
$\tilde{\lambda}=\lambda_{0}^{k_{1}}$. So,
$\lambda_{1}\lambda_{2}^{\frac{\gamma_{1}}{k_{2}}}\cdots\lambda_{n}^{\frac{\prod_{i=1}^{n-1}\gamma_{i}}{\prod_{i=2}^{n}k_{i}}}=\lambda_{0}^{k_{1}}$.

On the other hand, if
$\lambda_{1}\lambda_{2}^{\frac{\gamma_{1}}{k_{2}}}\cdots\lambda_{n}^{\frac{\prod_{i=1}^{n-1}\gamma_{i}}{\prod_{i=2}^{n}k_{i}}}=\lambda_{0}^{k_{1}}$,
we let
$\tilde{\lambda}=\lambda_{1}\lambda_{2}^{\frac{\gamma_{1}}{k_{2}}}\cdots\lambda_{n}^{\frac{\prod_{i=1}^{n-1}\gamma_{i}}{\prod_{i=2}^{n}k_{i}}}$.
Then $\tilde{\lambda}=\lambda_{0}^{k_{1}}$, which implies that (5.5) has a
$\boldsymbol{k}$-admissible solution $(u_{1},\ldots,u_{n})$. Define

$\displaystyle
u_{2}^{*}=\lambda_{2}^{\frac{1}{k_{2}}}\lambda_{3}^{\frac{\gamma_{2}}{k_{2}k_{3}}}\cdots\lambda_{n}^{\frac{\prod_{i=2}^{n-1}\gamma_{i}}{\prod_{i=2}^{n}k_{i}}}u_{2},$
$\displaystyle\qquad\vdots$ $\displaystyle
u_{n-1}^{*}=\lambda_{n-1}^{\frac{1}{k_{n-1}}}\lambda_{n}^{\frac{\gamma_{n-1}}{k_{n-1}k_{n}}}u_{n-1},$
$\displaystyle u_{n}^{*}=\lambda_{n}^{\frac{1}{k_{n}}}u_{n}.$

Then $(u_{1},u_{2}^{*},\ldots,u_{n}^{*})$ is a $\boldsymbol{k}$-admissible
solution of system (1.3). ∎

######  Remark 5.1.

It is necessary to emphasis that if we define different composite operator,
for example
$\overline{T}:=\overline{T}_{2}\cdots\overline{T}_{n}\overline{T}_{1}$, etc,
then we can let $\tilde{\lambda}$ be related to each $k_{i}\ (i=1,\ldots,n)$.
Here, we only take $\tilde{\lambda}=\lambda_{0}^{k_{1}}$ for a detailed
explanation.

######  Remark 5.2.

Here, we point out that our proof of the strong property of $\overline{T}$ is
different from that in [28]. We overcome the difficult caused by the non-
differentiability of solutions to degenerate $k$-Hessian equations and find
the corresponding sub-solutions equipped with the higher regularity. Thanks to
the Hopf lemma in [11] which applied to the non-differentiable function, we
derive the proof.

## References

  * [1] J. Bao, H. Li and Y.Y. Li, On the exterior Dirichlet problem for Hessian equations, Trans. Amer. Math. Soc., 366 (2014), no. 12, 6183–6200.
  * [2] L. Caffarelli, L. Nirenberg and J. Spruck, Dirichlet problem for nonlinear second order elliptic equations III, Functions of the eigenvalues of the Hessian, Acta Math., 155 (1985), 261-301.
  * [3] K. Chou and X.-J. Wang, A variational theory of the Hessian equation, Comm. Pure Appl. Math., 54 (2001), 1029-1064.
  * [4] J. Cui, Existence and nonexistence of entire $k$-convex radial solutions to Hessian type system, Adv. Difference Equ., (2021), Paper No. 462, 9pp.
  * [5] M. Feng, New results of coupled system of $k$-Hessian equations, Appl. Math. Lett., 94 (2019), 196-203.
  * [6] M. Feng, Convex solutions of Monge-Ampère equations and systems: existence, uniqueness and asymptotic behavior, Adv. Nonlinear Anal., 10 (2021), 371-399.
  * [7] M. Feng and X. Zhang, A coupled system of $k$-Hessian equations, Math. Methods Appl. Sci., 2 (2019), 1-18.
  * [8] J. Goncalves and C. Santos, Classical solutions of singular Monge-Ampère equations in a ball, J. Math. Anal. Appl., 305 (2005), 240-252.
  * [9] B. Guan, The Dirichlet problem for Hessian equations on Riemannian manifolds, Calc. Var. PDE, 8 (1999), 45-69.
  * [10] D. Guo and V. Lakshmikantham, Nonlinear Problems in Abstract Cones, Academic Press, Boston, 1988.
  * [11] E. Hopf, A remark on linear elliptic differential equations of second order, Proc. Amer. Math. Soc., 3 (1952), 791-793.
  * [12] S. Hu and H. Wang, Convex solutions of boundary value problems arising from Monge-Ampère equation, Discrete Contin. Dyn. Syst., 16 (2006), no. 3, 705-720.
  * [13] J. Jacobsen, Global bifurcation problems associated with $k$-Hessian operators, Topol. Methods Nonlinear Anal., 14 (1999), 81-130.
  * [14] X. Ji and J. Bao, Necessary and sufficient conditions on solvability for Hessian inequalities, Proc. Amer. Math. Soc., 138 (2010), no. 1, 175–188.
  * [15] H. Jian, Hessian equations with infinite Dirichlet boundary value, Indiana Univ. Math. J., 55 (2006), 1045-1062.
  * [16] F. Jiang and N. Trudinger, On the Dirichlet problem for general augmented Hessian equations, J. Differential Equations, 269 (2020), 5204-5227.
  * [17] F. Jiang, N. Trudinger and X.-P. Yang, On the Dirichlet problem for a class of augmented Hessian equations, J. Differ. Equ., 258 (2015), 1548-1576.
  * [18] X.N. Ma and G. Qiu, The Neumann problem for Hessian equations. Comm. Math. Phys., 366 (2019), no. 1, 1–28.
  * [19] N. Trudinger, On the Dirichlet problem for Hessian equations, Acta Math., 175 (1995), 151-164.
  * [20] N. Trudinger, Weak solutions of Hessian equation, Comm. Partial Differential Equations, 22 (1997), 1251-1261.
  * [21] N. Trudinger and X.-J. Wang, Hessian measures I, Topol. Methods Nonlin. Anal., 10 (1997), 225-239.
  * [22] N. Trudinger and X.-J. Wang, Hessian measures II, Ann. Math., 150 (1999), 579-604.
  * [23] G. Wang, Z. Yang and L. Zhang, D. Baleanu, Radial solutions of a nonlinear $k$-Hessian system involving a nonlinear operator, Commun. Nonlinear Sci. Numer. Simul., 91 (2020), 105396, 13pp.
  * [24] H. Wang, Convex solutions of systems arising from Monge-Ampère equations, Electron. J. Qual. Theory Differ. Equ. Special Edition I., 26 (2009), 1-8.
  * [25] X.-J. Wang, The $k$-Hessian Equation, Lecture Notes in Math., 1977 (2009), 177-252.
  * [26] X. Zhang, Analysis of nontrivial radial solutions for singular superlinear $k$-Hessian equations, Appl. Math. Lett., 106 (2020), 106409, 6pp.
  * [27] X. Zhang, Existence and uniqueness of nontrivial radial solutions for $k$-Hessian equations, J. Math. Anal. Appl., 492 (2020), 124439, 19pp.
  * [28] Z. Zhang and Z. Qi, On a power-type coupled system of Monge-Ampère equations, Topol. Methods Nonlinear Anal., 46 (2015), 717-729.
  * [29] Z. Zhang and S. Zhou, Existence of entire positive $k$-convex radial solutions to Hessian equations and systems with weights, Appl. Math. Lett., 50 (2015), 48-55.