# Mod 2 Representation of the Symmetric Group of Order 2 over Cohomology
Groups of 2-Configuration Space of Torus

Tomoki TOKUDA<EMAIL_ADDRESS>Joint Graduate School of
Mathematics for Innovation, Kyushu University, Fukuoka, 819-0395, Japan

###### Abstract.

In this article, we compute the mod 2 representarion of the symmetric group of
order 2 over the singular cohomology groups of orderd 2-configuration space
$C_{2}(T^{d})$ of the $d$-torus $T^{d}$ for $d\geqslant 1$. As applications of
the computation, we determine the Stiefel-Whitney height of $C_{2}(T^{d})$ for
any $d$, and determine $H^{*}(\mathbb{R}P^{\infty};\mathbb{F}_{2})$-module
structure of the cohomology groups of the unordered 2-configuration space of
the $d$-torus for $d=2,3$ using the Serre spectral sequence.

##  1\. Introduction and Main Theorems

The $n$-configuration space $C_{n}({X})$ of a space $X$ is the subspace of the
$n$-fold product space $X^{n}$ defined by

(1.0.1) $\displaystyle C_{n}({X})=\\{{(x_{1},\dots,x_{n})\in X^{n}}|{x_{i}\neq
x_{j}\text{ if }i\neq j}\\}.$

The symmetric group $\Sigma_{n}$ acts on $C_{n}({X})$ freely by permuting
coordinates. The unordered $n$-configuration space is the orbit space
$B_{n}({X})=C_{n}({X})/\Sigma_{n}$. In general, given an action of a group $G$
on a space $X$, the singular chain complex of $X$ inherits the $G$-action.
Then $H_{i}({X};{R})$ and $H^{i}({X};{R})$ become a module over the group ring
$R[G]$ where $R$ is a ring with unity.

The homology groups of the unordered configuration spaces $B_{n}({M})$ of a
manifold $M$ are studied by Bödigheimer, Cohen and Taylor in [1]. And this
result is generalised by Chen and Zhang in [2]. By their works, the Betti
numbers of $H_{i}({B_{n}({M})};{k})$ are almost determined where $k$ is a
prime field.

The cohomology algebra of unordered configuration spaces over closed manifolds
was computed by Félix and Tanré in [3] with coefficients $\mathbb{Q}$ and
$\mathbb{F}_{p}$. For the $B_{n}({T^{2}})$, Pagaria computed its cohomology
ring structure in [6]. However, in all these papers, the characteristics of
the coefficient fields are required to be zero or greater than the number of
coordinates of configurations.

The main result of this paper is the description of the
$\mathbb{F}_{2}[\Sigma_{2}]$-module structure of the cohomology groups
$H^{i}({C_{2}({T^{d}})};{\mathbb{F}_{2}})$ for $d\geqslant 2$. The point of
this study is that the characteristic of the field $\mathbb{F}_{2}$ divides
the order of $\Sigma_{2}$. In general, any representation of a finite group
$G$ over a finite dimensional $k$-vector space $V$ is completely reducible
unless the characteristic of $k$ divides the order of $G$. However, without
the assumption, the representation become more complicated in general.

Here is the main result of this paper.

###### Theorem 1.1

The representation of $\Sigma_{2}$ over
$H^{i}({C_{2}({T^{d}})};{\mathbb{F}_{2}})$ is decomposed as following:

(1.0.2) $\displaystyle
H^{i}({C_{2}({T^{d}})};{\mathbb{F}_{2}})\cong\begin{cases}\mathbb{F}_{2}^{\oplus\binom{d}{k}}\oplus\mathbb{F}_{2}[\Sigma_{2}]^{\oplus\frac{1}{2}\left(\sum_{j=0}^{i}\binom{d}{j}\binom{d}{i-j}-\binom{d}{k}\right)}&\text{
if $i=2k<d$,}\\\
\mathbb{F}_{2}[\Sigma_{2}]^{\oplus\frac{1}{2}\sum_{j=0}^{i}\binom{d}{j}\binom{d}{i-j}}&\text{
if $i=2k+1<d$,}\\\
\mathbb{F}_{2}^{\oplus\binom{d}{k}+\binom{d}{i-d}}\oplus\mathbb{F}_{2}[\Sigma_{2}]^{\oplus\frac{1}{2}\left(\sum_{j=0}^{i}\binom{d}{j}\binom{d}{i-j}-\binom{d}{k}\right)-\binom{d}{i-d}}&\text{
if $d\leqslant i=2k<2d$ ,}\\\
\mathbb{F}_{2}^{\oplus\binom{d}{i-d}}\oplus\mathbb{F}_{2}[\Sigma_{2}]^{\oplus\frac{1}{2}\left(\sum_{j=0}^{i}\binom{d}{j}\binom{d}{i-j}-\binom{d}{k}\right)-\binom{d}{i-d}}&\text{
if $d\leqslant i=2k+1<2d$ ,}\\\ 0&\text{ if $i\geqslant 2d$.}\\\ \end{cases}$

As a one of the additional results, we can determine the
$\mathbb{F}_{2}[\Sigma_{2}]$-module structure of
$H^{i}({B_{2}({T^{d}})};{\mathbb{F}_{2}})$ for $d=2,3$ by computing the Serre
spectral sequence of the Borel construction

(1.0.3) $\displaystyle C_{2}(T^{d})\longrightarrow
C_{2}(T^{d})\times_{\Sigma_{2}}E\Sigma_{2}\longrightarrow B\Sigma_{2}.$

Since the action of $\Sigma_{2}$ on $C_{2}(T^{d})$ is free, there is canonical
homotopy equivalence $C_{2}(T^{d})\times_{\Sigma_{2}}E\Sigma_{2}\rightarrow
B_{2}(T^{d})$. Hence, we obtain the homotopy fibration

(1.0.4) $\displaystyle C_{2}(T^{d})\longrightarrow B_{2}(T^{d})\longrightarrow
B\Sigma_{2}.$

Notice that the first map is the quotient map by the $\Sigma_{2}$ action, and
the classifying space $B\Sigma_{2}$ is the projective space
$\mathbb{R}P^{\infty}$. By computing the Serre spectral sequence of the
homotopy fibration 1.0.4, we obtain the following result.

###### Theorem 1.2

We identify
$H^{*}({\mathbb{R}P^{\infty}};{\mathbb{F}_{2}})\cong\mathbb{F}_{2}[\alpha]$.
When $d=2,3$, there are isomorphisms of $\mathbb{F}_{2}[\alpha]$-modules

(1.0.5) $\displaystyle
H^{*}({B_{2}({T^{2}})};{\mathbb{F}_{2}})\cong\mathbb{F}_{2}$
$\displaystyle[\alpha]/(\alpha^{3})\oplus(\oplus_{j=1}^{2}\mathbb{F}_{2}u_{1,j})\oplus\mathbb{F}_{2}u_{2,1}\oplus(\oplus_{j=1}^{2}\mathbb{F}_{2}[\alpha]/(\alpha^{3})x_{2,j}),$
(1.0.6) $\displaystyle
H^{*}({B_{2}({T^{3}})};{\mathbb{F}_{2}})\cong\mathbb{F}_{2}$
$\displaystyle[\alpha]/(\alpha^{4})\oplus(\oplus_{j=1}^{3}\mathbb{F}_{2}u_{1,j})\oplus(\oplus_{j=1}^{6}\mathbb{F}_{2}u_{2,j})\oplus(\oplus_{j=1}^{3}\mathbb{F}_{2}[\alpha]/(\alpha^{3})x_{2,j})$
(1.0.7)
$\displaystyle\oplus(\oplus_{j=1}^{9}\mathbb{F}_{2}u_{3,j})\oplus(\oplus_{j=1}^{3}\mathbb{F}_{2}u_{4,j})\oplus(\oplus_{j=1}^{3}\mathbb{F}_{2}[\alpha]/(\alpha^{2})x_{4,j})$

where $u_{i,j}$ and $x_{i,j}$ are elements with degree $i$.

To determine the module structure for the case of $d=3$, we use the following
result:

###### Theorem 1.3

For any $d\geqslant 2$, the Stiefel-Whitney height of the
$\mathbb{Z}_{2}$-bundle ${C_{2}({T^{d}})}\rightarrow{B_{2}({T^{d}})}$ is
exactly $d$. Here the Stiefel-Whitney height of a $\mathbb{Z}_{2}$-bundle is
the maximal integer $i$ such that $\alpha^{i}\neq 0$ where $\alpha$ is the
first Stiefel-Whitney class of the bundle.

The Stiefel-Whitney height of a manifold $M$ is an interesting invariant since
it is related to the minimal dimension $n$ such that a triangulated skeleton
of $M$ can be embedded in $\mathbb{R}^{n}$ (see [5]).

##  2\. Computation

In this section, we compute the $\mathbb{F}_{2}[\Sigma_{2}]$-module structure
of $H^{i}({C_{2}({T^{d}})};{\mathbb{F}_{2}})$ and prove Theorem 1.1 for
$d\geqslant 1$.

$T^{d}=\mathbb{R}^{d}/\mathbb{Z}^{d}$ inherits the Lie group structure from
$\mathbb{R}^{d}$. So we can define a map ${\varphi}\colon{T^{d}\times
T^{d}}\rightarrow{T^{d}\times T^{d}}$ by

(2.0.1) $\displaystyle\varphi(x,y)=(x,y-x).$

Note that the restriction
${\varphi}\colon{C_{2}({T^{d}})}\rightarrow{T^{d}\times(T^{d}\setminus 0)}$ is
a homeomorphism. Via this map, the desired representation is closely related
to $H^{i}({T^{d}\times T^{d}};{\mathbb{F}_{2}})$. So we first compute the the
$\mathbb{F}_{2}[\Sigma_{2}]$-module structure of $H^{i}({T^{d}\times
T^{d}};{\mathbb{F}_{2}})$.

###### Lemma 2.1

There is an isomorphism of $\mathbb{F}_{2}[\Sigma_{2}]$-module

(2.0.2) $\displaystyle H^{i}({T^{d}\times
T^{d}};{\mathbb{F}_{2}})\cong\begin{cases}{\mathbb{F}_{2}}^{\oplus\binom{d}{k}}\oplus\mathbb{F}_{2}[\Sigma_{2}]^{\oplus\frac{1}{2}\left(\sum_{j=0}^{i}\binom{d}{j}\binom{d}{i-j}-\binom{d}{k}\right)}&\text{
if $i=2k$ for some $0\leqslant k\leqslant d$,}\\\
\mathbb{F}_{2}[\Sigma_{2}]^{\oplus\frac{1}{2}\sum_{j=0}^{i}\binom{d}{j}\binom{d}{i-j}}&\text{
if $i=2k+1$ for some $0\leqslant k\leqslant d-1$,}\\\ 0&\text{
otherwise.}\end{cases}$

Here we assume that $\binom{m}{n}=0$ when $m<n$.

Before the proof, we decompose $T^{d}$ into cells, to use cellular homology
and cohomology. We regard $T^{d}$ to the quotient space obtained from the
$d$-dimensional cube $[-\frac{1}{2},\frac{1}{2}]^{d}$ by identifying each pair
of opposite faces. Let $e_{i}$ denote the 1-cell of $T^{d}$ corresponding to
the edge from $(-\frac{1}{2},\dots,-\frac{1}{2})$ to
$\underset{\widehat{i}}{(-\frac{1}{2},\dots,-\frac{1}{2},\frac{1}{2},-\frac{1}{2},\dots,-\frac{1}{2})}$(Fig.
1). Then each higher dimensional cell is a direct product of these 1-cells,
and the only cell containing the origin is the $d$-cell
$e_{1}\times\cdots\times e_{d}$. To simplify our notation, we omit the symbol
$\times$ and the 0-cell for direct product. Note that
$H_{i}({T^{d}};{\mathbb{Z}})$ is a free abelian group generated by the
$i$-dimensional cells $e_{k_{1}}e_{k_{2}}\cdots e_{k_{i}}$ for $0\leqslant
k_{1}<k_{2}<\cdots<k_{i}\leqslant d$.

$e_{1}$$e_{2}$$e_{1}$$e_{2}$$e_{3}$$e_{2}$$e_{3}$$e_{1}$$e_{3}$ Figure. 1.
Cell decomposition of $T^{3}$

  * Proof of Lemma 2.1.By the universal coefficient theorem, there is a natural isomorphism

(2.0.3) $\displaystyle
H^{i}({T^{d}};{\mathbb{F}_{2}})\cong\text{Hom}_{\mathbb{Z}}({H_{i}({T^{d}};{\mathbb{Z}})},{\mathbb{F}_{2}}).$

Let $e^{*}_{j}$ be the dual basis of $e_{j}$. By the Künneth theorem, we
obtain a natural isomorphism

(2.0.4) $\displaystyle H^{i}({T^{d}\times T^{d}};{\mathbb{F}_{2}})$
$\displaystyle\cong\bigoplus_{j=0}^{i}H^{j}({T^{d}};{\mathbb{F}_{2}})\otimes
H^{i-j}({T^{d}};{\mathbb{F}_{2}})$ (2.0.5)
$\displaystyle\cong\bigoplus_{j=0}^{i}\text{Hom}_{\mathbb{Z}}({H_{j}({T^{d}};{\mathbb{Z}})},{\mathbb{F}_{2}})\otimes\text{Hom}_{\mathbb{Z}}({H_{i-j}({T^{d}};{\mathbb{Z}})},{\mathbb{F}_{2}}).$

Via this isomorphism, the $\Sigma_{2}$-action on $H^{i}({T^{d}\times
T^{d}};{\mathbb{F}_{2}})$ is equivalent to swapping the left and right of the
tensor product $H^{j}({T^{d}};{\mathbb{F}_{2}})\otimes
H^{i-j}({T^{d}};{\mathbb{F}_{2}})$. Thus the subgroup generated by
$e^{*}_{m_{1}}\cdots e^{*}_{m_{s}}\otimes e^{*}_{n_{1}}\cdots e^{*}_{n_{t}}$
and $e^{*}_{n_{1}}\cdots e^{*}_{n_{t}}\otimes e^{*}_{m_{1}}\cdots
e^{*}_{m_{s}}$ is a subrepresentation of $\Sigma_{2}$. Unless
$e^{*}_{m_{1}}\cdots e^{*}_{m_{s}}=e^{*}_{n_{1}}\cdots e^{*}_{n_{t}}$, such
representation is isomorphic to
$\mathbb{F}_{2}[\Sigma_{2}]=\\{0,1,\sigma,1+\sigma\\}$ by the homomorphism
defined by

(2.0.6) $\displaystyle e^{*}_{m_{1}}\cdots e^{*}_{m_{s}}\otimes
e^{*}_{n_{1}}\cdots e^{*}_{n_{t}}\mapsto 1,\quad e^{*}_{n_{1}}\cdots
e^{*}_{n_{t}}\otimes e^{*}_{m_{1}}\cdots e^{*}_{m_{s}}\mapsto\sigma$

When $e^{*}_{m_{1}}\cdots e^{*}_{m_{s}}=e^{*}_{n_{1}}\cdots e^{*}_{n_{t}}$,
such a subrepresentation is trivial.

If $i=2k+1$ for some $0\leqslant k\leqslant d$, there is no trivial
subrepresentation. So we have an isomorphism of
$\mathbb{F}_{2}[\Sigma_{2}]$-module

(2.0.7) $\displaystyle H^{i}({T^{d}\times
T^{d}};{\mathbb{F}_{2}})\cong\mathbb{F}_{2}[\Sigma_{2}]^{\oplus\frac{1}{2}\dim{H^{i}({T^{d}\times
T^{d}};{\mathbb{F}_{2}})}}=\mathbb{F}_{2}[\Sigma_{2}]^{\oplus\frac{1}{2}\sum_{j=0}^{i}\binom{d}{j}\binom{d}{i-j}}.$

If $i=2k$ for some $0\leqslant k\leqslant d$, there are $\binom{d}{k}$ cells
of the form $e_{m_{1}}\cdots e_{m_{k}}\otimes e_{m_{1}}\cdots e_{m_{k}}$, and
for every other $i$-cell $e^{*}_{m_{1}}\cdots e^{*}_{m_{s}}\otimes
e^{*}_{n_{1}}\cdots e^{*}_{n_{i-s}}$, there is unique swapped $i$-cell
$e^{*}_{n_{1}}\cdots e^{*}_{n_{i-s}}\otimes e^{*}_{m_{1}}\cdots
e^{*}_{m_{s}}$. So we have the direct sum decomposition of representation

(2.0.8) $\displaystyle H^{i}({T^{d}\times
T^{d}};{\mathbb{F}_{2}})\cong\mathbb{F}_{2}^{\oplus\binom{d}{k}}\oplus\mathbb{F}_{2}[\Sigma_{2}]^{\oplus\frac{1}{2}\left(\sum_{j=0}^{i}\binom{d}{j}\binom{d}{i-j}-\binom{d}{k}\right)}.$

∎

We next show Theorem 1.1. The proof is supported by the commutative diagram

(2.0.13)

where the vertical maps are the inclusions.

  * Proof of Theorem 1.1.The diagram 2.0.13 induces the commutative diagram of cohomology groups

(2.0.18)

This diagram implies that the image of $e^{*}_{m_{1}}\cdots
e^{*}_{m_{s}}\otimes e^{*}_{n_{1}}\cdots e^{*}_{n_{t}}\in H^{i}({T^{d}\times
T^{d}};{\mathbb{F}_{2}})$ by $\iota^{*}$ generates
$H^{i}({C_{2}({T^{d}})};{\mathbb{F}_{2}})$.

If $i<d$, $\iota^{*}$ is an isomorphism since $\iota^{\prime*}$ and
$\varphi^{*}$ are isomorphisms. So $H^{i}({C_{2}({T^{d}})};{\mathbb{F}_{2}})$
is isomorphic to $H^{i}({T^{d}\times T^{d}};{\mathbb{F}_{2}})$ as
$\mathbb{F}_{2}$-module. Since the inclusion $\iota:C_{2}({T^{d}})\rightarrow
T^{d}\times T^{d}$ is $\Sigma_{2}$-equivariant, the diagram below is
commutative, hence, $H^{i}({C_{2}({T^{d}})};{\mathbb{F}_{2}})$ is isomorphic
to $H^{i}({T^{d}\times T^{d}};{\mathbb{F}_{2}})$ as
$\mathbb{F}_{2}[\Sigma_{2}]$-module ($\sigma$ is the generator of
$\Sigma_{2}$).

(2.0.23)

If $i=d$, $\iota^{\prime*}:H^{i}({T^{d}\times
T^{d}};{\mathbb{F}_{2}})\rightarrow H^{i}({T^{d}\times(T^{d}\setminus
0)};{\mathbb{F}_{2}})$ ,in 2.0.18, is not an isomorphism, but it is a
surjection whose kernel is the $1$-dimensional $\mathbb{F}_{2}$-vector space
with basis $\varphi^{*}(1\otimes e^{*}_{1}\cdots e^{*}_{d})$. The
commutativity of 2.0.18 implies that ${\iota^{*}}\colon{H^{i}({T^{d}\times
T^{d}};{\mathbb{F}_{2}})}\rightarrow{H^{i}({C_{2}({T^{d}})};{\mathbb{F}_{2}})}$
is surjective. Hence we obtain an isomorphism of
$\mathbb{F}_{2}[\Sigma_{2}]$-module

(2.0.24) $\displaystyle H^{i}({C_{2}({T^{d}})};{\mathbb{F}_{2}})\cong
H^{i}({T^{d}\times T^{d}};{\mathbb{F}_{2}})/\ker{\iota^{*}}=H^{i}({T^{d}\times
T^{d}};{\mathbb{F}_{2}})/\varphi^{*}(\langle 1\otimes e^{*}_{1}\cdots
e^{*}_{d}\rangle).$

Here we note that $\ker{\iota^{*}}$ is a
$\mathbb{F}_{2}[\Sigma_{2}]$-submodule of $H^{i}({T^{d}\times
T^{d}};{\mathbb{F}_{2}})$ since $\iota^{*}$ is $\Sigma_{2}$-equivariant. To
determine the $\mathbb{F}_{2}[\Sigma_{2}]$-module structure, we next compute
the induced map $\varphi^{*}:H^{i}({T^{d}\times
T^{d}};{\mathbb{F}_{2}})\rightarrow H^{i}({T^{d}\times
T^{d}};{\mathbb{F}_{2}})$.

Since all the cells of dimension greater than 1 of $T^{d}\times T^{d}$ are
expressed as a product cell of 1-cells, so all the cohomology classes of such
dimension of $T^{d}\times T^{d}$ are generated by cup products of some classes
of 1-cocycles. Since $\varphi^{*}:H^{*}({T^{d}\times
T^{d}};{\mathbb{F}_{2}})\rightarrow H^{*}({T^{d}\times
T^{d}};{\mathbb{F}_{2}})$ is a ring homomorphism, $\varphi^{*}$ is completely
determined by the restriction on the first cohomology group
$H^{1}({T^{d}\times T^{d}};{\mathbb{F}_{2}})$. By the universal coefficient
theorem, it is the dual map $(\varphi_{*})_{\\#}$ of
$\varphi_{*}:H_{1}({T^{d}\times T^{d}};{\mathbb{Z}})\rightarrow
H_{1}({T^{d}\times T^{d}};{\mathbb{Z}})$.

(2.0.29)

By the definition of $\varphi:T^{d}\times T^{d}\rightarrow T^{d}\times T^{d}$,
we have

(2.0.30) $\displaystyle\varphi_{*}(e_{i}\otimes*)$
$\displaystyle=(e_{i}\otimes*)-(*\otimes e_{i}),$ (2.0.31)
$\displaystyle\varphi_{*}(*\otimes e_{i})$ $\displaystyle=*\otimes e_{i},$

where $*$ is the 0-cell of $T^{d}$. So the dual map is determined by

(2.0.32) $\displaystyle\varphi^{*}(e_{i}^{*}\otimes 1)$
$\displaystyle=e_{i}^{*}\otimes 1,$ (2.0.33)
$\displaystyle\varphi^{*}(1\otimes e_{i}^{*})$
$\displaystyle=(e_{i}^{*}\otimes 1)-(1\otimes e_{i}^{*})=(e_{i}^{*}\otimes
1)+(1\otimes e_{i}^{*}).$

We return to the consideration for the case $i=d$. By the preceding
calculation, we have

(2.0.34) $\displaystyle\varphi^{*}(1\otimes e^{*}_{1}\cdots e^{*}_{d})$
$\displaystyle=\prod_{j=1}^{d}\varphi^{*}(1\otimes e_{j}^{*})$ (2.0.35)
$\displaystyle=\prod_{j=1}^{d}(e_{j}^{*}\otimes 1+1\otimes e_{j}^{*})$
(2.0.36) $\displaystyle=\sum_{l=0}^{d}\sum_{1\leqslant
j_{k_{1}}<\cdots<j_{k_{l}}\leqslant
d}e_{1}^{*}\cdots\widehat{e_{j_{k_{1}}}^{*}}\cdots\widehat{e_{j_{k_{l}}}^{*}}\cdots
e_{d}^{*}\otimes e_{j_{k_{1}}}^{*}\cdots e_{j_{k_{l}}}^{*}.$

Thus $H^{i}({T^{d}\times T^{d}};{\mathbb{F}_{2}})/\varphi^{*}(\langle 1\otimes
e^{*}_{1}\cdots e^{*}_{d}\rangle)$ is the $\mathbb{F}_{2}[\Sigma_{2}]$-module
generated by $e^{*}_{m_{1}}\cdots e^{*}_{m_{s}}\otimes e^{*}_{n_{1}}\cdots
e^{*}_{n_{d-s}}$ with one relation

(2.0.37) $\displaystyle\sum_{l=0}^{d}\sum_{1\leqslant
j_{k_{1}}<\cdots<j_{k_{l}}\leqslant
d}e_{1}^{*}\cdots\widehat{e_{j_{k_{1}}}^{*}}\cdots\widehat{e_{j_{k_{l}}}^{*}}\cdots
e_{d}^{*}\otimes e_{j_{k_{1}}}^{*}\cdots e_{j_{k_{l}}}^{*}=0.$

Notice that every summand
$e_{1}^{*}\cdots\widehat{e_{j_{k_{1}}}^{*}}\cdots\widehat{e_{j_{k_{l}}}^{*}}\cdots
e_{d}^{*}\otimes e_{j_{k_{1}}}^{*}\cdots e_{j_{k_{l}}}^{*}$ corresponds to the
unique choice of the indices in the left side of the tensor product, which is
equivalent to the choice of the indices in the right side, and this is one-to-
one. Thus choosing a half of the summands, we obtain a trivial
suburepresentation of $\Sigma_{2}$. More explicitly, for example, define $x$
by the formula

(2.0.38) $\displaystyle x=\begin{dcases}\sum_{l=0}^{(d/2)-1}\sum_{1\leqslant
j_{k_{1}}<\cdots<j_{k_{l}}\leqslant
d}e_{1}^{*}\cdots\widehat{e_{j_{k_{1}}}^{*}}\cdots\widehat{e_{j_{k_{l}}}^{*}}\cdots
e_{d}^{*}\otimes e_{j_{k_{1}}}^{*}\cdots e_{j_{k_{l}}}^{*}\\\
\quad\quad\quad+\sum_{1\leqslant
j_{k_{1}}<\cdots<j_{k_{d/2}}<d}e_{1}^{*}\cdots\widehat{e_{j_{k_{1}}}^{*}}\cdots\widehat{e_{j_{k_{d/2}}}^{*}}\cdots
e_{d}^{*}\otimes e_{j_{k_{1}}}^{*}\cdots e_{j_{k_{d/2}}}^{*}&\text{ if $d$ is
even,}\\\ \sum_{l=0}^{(d-1)/2}\sum_{1\leqslant
j_{k_{1}}<\cdots<j_{k_{l}}\leqslant
d}e_{1}^{*}\cdots\widehat{e_{j_{k_{1}}}^{*}}\cdots\widehat{e_{j_{k_{l}}}^{*}}\cdots
e_{d}^{*}\otimes e_{j_{k_{1}}}^{*}\cdots e_{j_{k_{l}}}^{*}&\text{ if $d$ is
odd.}\end{dcases}$

Then $x$ is a nontrivial element satisfying $\sigma(x)+x=0$, which is
equivalent to that $\sigma(x)=x$. So we have that, in $H^{i}({T^{d}\times
T^{d}};{\mathbb{F}_{2}})/\varphi^{*}(\langle 1\otimes e^{*}_{1}\cdots
e^{*}_{d}\rangle)$, the regular representation $\langle e^{*}_{1}\cdots
e^{*}_{d}\otimes 1,1\otimes e^{*}_{1}\cdots e^{*}_{d}\rangle$ in
$H^{i}({T^{d}\times T^{d}};{\mathbb{F}_{2}})$ has changed into the trivial
representation $\langle x\rangle$, and all the other subrepresentations of
$H^{i}({T^{d}\times T^{d}};{\mathbb{F}_{2}})$ are the same. Therefore, by
Lemma 2.1, we have

(2.0.39) $\displaystyle H^{d}({C_{2}({T^{d}})};{\mathbb{F}_{2}})$
$\displaystyle\cong H^{d}({T^{d}\times
T^{d}};{\mathbb{F}_{2}})/\varphi^{*}(\langle 1\otimes e^{*}_{1}\cdots
e^{*}_{d}\rangle)$ (2.0.40)
$\displaystyle\cong\begin{cases}{\mathbb{F}_{2}}^{\oplus\binom{d}{k}+1}\oplus\mathbb{F}_{2}[\Sigma_{2}]^{\oplus\frac{1}{2}\left(\sum_{j=0}^{i}\binom{d}{j}\binom{d}{d-j}-\binom{d}{k}\right)-1}&\text{
if $d=2k$ for some $k\geqslant 0$,}\\\
\mathbb{F}_{2}\oplus\mathbb{F}_{2}[\Sigma_{2}]^{\oplus\frac{1}{2}\sum_{j=0}^{i}\binom{d}{j}\binom{d}{d-j}-1}&\text{
if $d=2k+1$ for some $k\geqslant 0$}.\end{cases}$

It remains the case $i>d$. By the same diagram 2.0.18,
$H^{i}({C_{2}({T^{d}})};{\mathbb{F}_{2}})$ is isomorphic to
$H^{i}({T^{d}\times T^{d}};{\mathbb{F}_{2}})/\ker{\iota^{*}}$. Now the kernel
of $\iota^{*}$ is generated by the images

(2.0.41) $\displaystyle\varphi^{*}(e^{*}_{m_{1}}\cdots e^{*}_{m_{i-d}}\otimes
e^{*}_{1}\cdots e^{*}_{d})$ $\displaystyle=\varphi^{*}(e^{*}_{m_{1}}\cdots
e^{*}_{m_{i-d}}\otimes 1)\varphi^{*}(1\otimes e^{*}_{1}\cdots e^{*}_{d})$
(2.0.42)
$\displaystyle=\left(\prod_{j=1}^{i-d}\varphi^{*}(e^{*}_{m_{j}}\otimes
1)\right)\varphi^{*}(1\otimes e^{*}_{1}\cdots e^{*}_{d})$ (2.0.43)
$\displaystyle=\left(\prod_{j=1}^{i-d}(e^{*}_{m_{j}}\otimes
1)\right)\varphi^{*}(1\otimes e^{*}_{1}\cdots e^{*}_{d})$ (2.0.44)
$\displaystyle=(e^{*}_{m_{1}}\cdots e^{*}_{m_{i-d}}\otimes
1)\varphi^{*}(1\otimes e^{*}_{1}\cdots e^{*}_{d}).$

By eq. 2.0.34, in $H^{i}({T^{d}\times
T^{d}};{\mathbb{F}_{2}})/\ker{\iota^{*}}$, there are $\binom{d}{i-d}$
relations, each of which is given by

(2.0.45) $\displaystyle(e^{*}_{m_{1}}\cdots e^{*}_{m_{i-d}}\otimes
1)\left(\sum_{l=0}^{d}\sum_{1\leqslant j_{k_{1}}<\cdots<j_{k_{l}}\leqslant
d}e_{1}^{*}\cdots\widehat{e_{j_{k_{1}}}^{*}}\cdots\widehat{e_{j_{k_{l}}}^{*}}\cdots
e_{d}^{*}\otimes e_{j_{k_{1}}}^{*}\cdots e_{j_{k_{l}}}^{*}\right)=0.$

Since $(e_{j}^{*})^{2}=0$ foe each $j$, this relation is equivalent to

(2.0.46) $\displaystyle\left(\sum
e_{1}^{*}\cdots\widehat{e_{j_{k_{1}}}^{*}}\cdots\widehat{e_{j_{k_{l}}}^{*}}\cdots
e_{d}^{*}\otimes
e_{j_{k_{1}}}^{*}\cdots\widehat{e^{*}_{m_{1}}}\cdots\widehat{e^{*}_{m_{i-d}}}\cdots
e_{j_{k_{l}}}^{*}\right)(e^{*}_{m_{1}}\cdots e^{*}_{m_{i-d}}\otimes
e^{*}_{m_{1}}\cdots e^{*}_{m_{i-d}})=0$

where $l$ ranges $0$ to $d$, and $j_{k_{1}},\dots,j_{k_{l}}$ range $1\leqslant
j_{k_{1}}<\cdots<j_{k_{l}}\leqslant d$ under $j_{k_{s}}\neq
m_{1},\dots,m_{i-d}$ for each $1\leqslant s\leqslant l$. Notice that, in the
summation, each summand does not contain
$e^{*}_{m_{1}},\dots,e^{*}_{m_{i-d}}$, and the indices in each summand do not
duplicate. Since the summation is summing up all of such summands, we can pair
the summands of the form $e^{*}_{m_{1}}\cdots e^{*}_{m_{s}}\otimes
e^{*}_{n_{1}}\cdots e^{*}_{n_{t}}$ and $e^{*}_{n_{1}}\cdots
e^{*}_{n_{t}}\otimes e^{*}_{m_{1}}\cdots e^{*}_{m_{s}}$, then we can construct
a nontrivial element $x^{\prime}$ so that $\sigma(x^{\prime})=x^{\prime}$ as
well as the previous $x$. In the end, in $H^{i}({T^{d}\times
T^{d}};{\mathbb{F}_{2}})/\ker{\iota^{*}}$, $\binom{d}{i-d}$ regular
representations $\langle e^{*}_{1}\cdots e^{*}_{d}\otimes e^{*}_{m_{1}}\cdots
e^{*}_{m_{i-d}},e^{*}_{m_{1}}\cdots e^{*}_{m_{i-d}}\otimes e^{*}_{1}\cdots
e^{*}_{d}\rangle$ in $H^{i}({T^{d}\times T^{d}};{\mathbb{F}_{2}})$ have
changed into trivial representations. However, when $i=2d$, the relation means
that $\ker{\iota^{*}}=H^{2d}({T^{d}\times T^{d}};{\mathbb{F}_{2}})$.
Therefore, by Lemma 2.1, we finally obtain

(2.0.47) $\displaystyle H^{i}({C_{2}({T^{d}})};{\mathbb{F}_{2}})\cong
H^{i}({T^{d}\times T^{d}};{\mathbb{F}_{2}})/\ker{\iota^{*}}$ (2.0.48)
$\displaystyle\cong\begin{cases}{\mathbb{F}_{2}}^{\oplus\binom{d}{k}+\binom{d}{i-d}}\oplus\mathbb{F}_{2}[\Sigma_{2}]^{\oplus\frac{1}{2}\left(\sum_{j=0}^{i}\binom{d}{j}\binom{d}{i-j}-\binom{d}{k}\right)-\binom{d}{i-d}}&\text{
if $d\leqslant i=2k<d$,}\\\
\mathbb{F}_{2}^{\oplus\binom{d}{i-d}}\mathbb{F}_{2}[\Sigma_{2}]^{\oplus\frac{1}{2}\left(\sum_{j=0}^{i}\binom{d}{j}\binom{d}{i-j}\right)-\binom{d}{i-d}}&\text{
if $d\leqslant i=2k+1<d$,}\\\ 0&\text{ if $i\geqslant 2d$.}\end{cases}$

∎

In particular, when $d\leqslant 3$, we have

(2.0.49)
$\displaystyle\widetilde{H}^{i}({C_{2}({T^{1}})};{\mathbb{F}_{2}})=\widetilde{H}^{i}({C_{2}({S^{1}})};{\mathbb{F}_{2}})$
$\displaystyle\cong\begin{cases}0&\text{$i=0$,}\\\
\mathbb{F}_{2}[\Sigma_{2}]&\text{$i=1$,}\\\ 0&\text{otherwise.}\\\
\end{cases}$ (2.0.50)
$\displaystyle\widetilde{H}^{i}({C_{2}({T^{2}})};{\mathbb{F}_{2}})$
$\displaystyle\cong\begin{cases}0&\text{$i=0$,}\\\
\mathbb{F}_{2}[\Sigma_{2}]^{\oplus 2}&\text{$i=1$,}\\\ \mathbb{F}_{2}^{\oplus
3}\oplus\mathbb{F}_{2}[\Sigma_{2}]&\text{$i=2$,}\\\ \mathbb{F}_{2}^{\oplus
2}&\text{$i=3$,}\\\ 0&\text{otherwise.}\\\ \end{cases}$ (2.0.51)
$\displaystyle\widetilde{H}^{i}({C_{2}({T^{3}})};{\mathbb{F}_{2}})$
$\displaystyle\cong\begin{cases}0&\text{$i=0$,}\\\
\mathbb{F}_{2}[\Sigma_{2}]^{\oplus 3}&\text{$i=1$,}\\\ \mathbb{F}_{2}^{\oplus
3}\oplus\mathbb{F}_{2}[\Sigma_{2}]^{\oplus 6}&\text{$i=2$,}\\\
\mathbb{F}_{2}^{\oplus 1}\oplus\mathbb{F}_{2}[\Sigma_{2}]^{\oplus
9}&\text{$i=3$,}\\\ \mathbb{F}_{2}^{\oplus
6}\oplus\mathbb{F}_{2}[\Sigma_{2}]^{\oplus 3}&\text{$i=4$,}\\\
\mathbb{F}_{2}^{\oplus 3}&\text{$i=5$,}\\\ 0&\text{otherwise.}\end{cases}$

for the reduced cohomology groups. In these cases, the rank of the trivial
representation in $H^{i}$ and that of the regular one in $H^{d-i}$ are
symmetrically swapped. Although such ‘duality’-like situation does not occur
for $d\geqslant 4$, this is a little interesting.

##  3\. Applications

In this section, we prove Theorems 1.2 and 1.3. We first see the
$H^{*}({\mathbb{R}P^{\infty}};{\mathbb{F}_{2}})$-module structure of
$H^{i}({B_{2}({T^{2}})};{\mathbb{F}_{2}})$ since it can be determined
directly. The $E_{2}$-term of the Serre spectral sequence for the homotopy
fibration 1.0.4 is given by

(3.0.1) $\displaystyle
E_{2}^{p,q}=H^{p}({B\Sigma_{2}};{\mathcal{H}^{q}({C_{2}({T^{2}})};{\mathbb{F}_{2}})})$

where $\mathcal{H}^{q}({C_{2}({T^{d}})};{\mathbb{F}_{2}})$ denotes the local
system over $\mathbb{R}P^{\infty}$ associated to the homotopy fibration 1.0.4.
By the preceding result, we have

(3.0.2) $\displaystyle
E_{2}^{p,q}=\begin{cases}H^{p}({\mathbb{R}P^{\infty}};{\mathbb{F}_{2}})&\text{
$q=0$,}\\\ H^{p}({\mathbb{R}P^{\infty}};{\mathbb{F}_{2}[\Sigma_{2}]})^{\oplus
2}&\text{ $q=1$,}\\\ H^{p}({\mathbb{R}P^{\infty}};{\mathbb{F}_{2}})^{\oplus
3}\oplus H^{p}({\mathbb{R}P^{\infty}};{\mathbb{F}_{2}[\Sigma_{2}]})^{\oplus
2}&\text{ $q=2$,}\\\ H^{p}({\mathbb{R}P^{\infty}};{\mathbb{F}_{2}})^{\oplus
2}&\text{ $q=3$,}\\\ 0&\text{ otherwise.}\end{cases}$

In general, for a connected CW complex $X$ with finite fundamental group $\pi$
and the universal cover $\widetilde{X}$, there is an isomorphism
$H^{i}({\widetilde{X}};{R})\cong H^{i}({X};{R[\pi]})$ (see the proof of
Proposition 3H.5 in [4]). Since
$\pi_{1}({\mathbb{R}P^{\infty}})\cong\Sigma_{2}$, we have
$H^{i}({\mathbb{R}P^{\infty}};{\mathbb{F}_{2}[\Sigma_{2}]})\cong
H^{i}({S^{\infty}};{\mathbb{F}_{2}})$ by considering the universal cover
${S^{\infty}}\rightarrow{\mathbb{R}P^{\infty}}$. This isomorphism is derived
from the isomorphism of cochain complexes
${C^{*}({S^{\infty}};{\mathbb{F}_{2}})}\rightarrow{\text{Hom}_{\mathbb{F}_{2}[\Sigma_{2}]}({C_{*}({S^{\infty}})},{\mathbb{F}_{2}[\Sigma_{2}]})}$
where $C_{*}({S^{\infty}})$ and $C^{*}({S^{\infty}};{\mathbb{F}_{2}})$ denote
the cellular chain and cochain complexes, with coefficients in integer and
$\mathbb{F}_{2}$, respectively. More explicitly, this map is given by

(3.0.3) $\displaystyle\varphi\mapsto\left(\widehat{\varphi}\colon
x\mapsto\sum_{\gamma\in\pi_{1}}\varphi(\gamma^{-1}x)\gamma\right)$

where $\pi_{1}$ is the fundamental group of $\mathbb{R}P^{\infty}$. Hence,

(3.0.4) $\displaystyle
H^{p}({\mathbb{R}P^{\infty}};{\mathbb{F}_{2}[\Sigma_{2}]})\cong
H^{p}({S^{\infty}};{\mathbb{F}_{2}})\cong\begin{cases}\mathbb{F}_{2}&\text{if
$p=0$,}\\\ 0&\text{otherwise.}\end{cases}$

Since the generator of $H^{0}({S^{\infty}};{\mathbb{F}_{2}})$ is the
$0$-cochain $1$ of $S^{\infty}$ corresponding to $\widehat{1}\colon
e^{0}\mapsto 1+\sigma$, thus
$H^{0}({\mathbb{R}P^{\infty}};{\mathbb{F}_{2}[\Sigma_{2}]})$ is generated by
$e_{i}^{*}\otimes 1+1\otimes e_{i}^{*}$. Thus the $E_{2}$-term is described as
follows:

$p$$q$$1$$1$$1$$1$$1$$1$$\alpha$$\alpha^{2}$$\alpha^{3}$$\alpha^{4}$$\alpha^{5}$$2$$0$$0$$0$$0$$0$$4$$3$$3$$3$$3$$3$$2$$2$$2$$2$$2$$2$$d_{2}$$d_{2}$
Figure. 2. $E_{2}$-term

Here $\alpha$ denotes the generator of the polynomial ring
$H^{*}({\mathbb{R}P^{\infty}};{\mathbb{F}_{2}})$. Since $B_{2}({T^{2}})$ is a
$4$-dimensional open manifold, the region of total degree $4$ or greater must
vanish in $E_{\infty}$-term. So the upper $d_{2}$ in the figure is injective.
On the other hand, the lower $d_{2}$ is zero because if $d_{2}(u)=\alpha^{2}$,
then $\alpha^{3}=d_{2}(u)\alpha=d_{2}(u\alpha)=d_{2}(0)=0$. This contradicts
to $\alpha^{3}\neq 0$. Thus the $E_{3}$-term is given by

$p$$q$$1$$1$$1$$1$$1$$1$$\alpha$$\alpha^{2}$$\alpha^{3}$$\alpha^{4}$$\alpha^{5}$$2$$0$$0$$0$$0$$0$$4$$3$$1$$1$$1$$1$$0$$0$$0$$0$$0$$0$$d_{3}$
Figure. 3. $E_{3}$-term

If the $d_{3}$ in the figure is zero, then $\alpha^{4}$ remains in
$E_{\infty}$. Thus $d_{3}$ is nonzero, and we have the $E_{\infty}$ term
indicated in the following figure:

$p$$q$$1$$1$$1$$0$$0$$0$$\alpha$$\alpha^{2}$$2$$0$$0$$0$$0$$0$$3$$2$$0$$0$$0$$0$$0$$0$$0$$0$$0$$0$
Figure. 4. $E_{\infty}$-term

So we can see that the Stiefel-Whitney height of the bundle
${C_{2}({T^{2}})}\rightarrow{B_{2}({T^{2}})}$ is exactly $2$.

Before the computation for the case of $d=3$, we prove Theorem 1.3. This
follows from the observation of the Serre spectral sequence and the fact that
$T^{d}$ can be embedded into $\mathbb{R}^{d+1}$.

  * Proof of Theorem 1.3.An embedding ${T^{d}}\rightarrow{\mathbb{R}^{d+1}}$ induces a $\Sigma_{2}$-equivariant embedding ${C_{2}({T^{d}})}\rightarrow{C_{2}({\mathbb{R}^{d+1}})}$. Since there is a $\Sigma_{2}$-equivariant homotopy equivalence $C_{2}({\mathbb{R}^{d+1}})\simeq S^{d}$, we have $B_{2}({\mathbb{R}^{d+1}})\simeq\mathbb{R}P^{d}$. Thus the Stiefel-Whitney height of $B_{2}({T^{d}})$ is $d$ or less. On the other hand, the inclusion ${C_{2}({T^{d}})}\rightarrow{T^{d}\times T^{d}}$ induces a bundle map between the Borel constructions. We denote the Serre spectral sequence of the Borel construction

(3.0.7)

by $\\{E^{\prime},d^{\prime}\\}$. So its second page is indicated as

(3.0.8) $\displaystyle E^{\prime
p,q}_{2}=H^{p}({B\Sigma_{2}};{\mathcal{H}^{q}({T^{d}\times
T^{d}};{\mathbb{F}_{2}})}).$

Then, by Theorem 1.1, the induced homomorphism
${E^{\prime}_{r}}\rightarrow{E_{r}}$ gives surjections restricted on the part
of $q\leqslant d$. In fact, every differential of $E^{\prime}$ whose source is
in such part is zero. Thus that of $E$ so is. Therefore the Stiefel-Whitney
classes $\alpha^{i}$ remain for $i=0,1,\dots,d$, and the Stiefel-Whitney
height of $B_{2}({T^{d}})$ is $d$ or greater.

To complete the proof, we show that the Serre spectral sequence
$\\{E^{\prime},d^{\prime}\\}$ associated to 3.0.7 collapses at
$E^{\prime}_{2}$. This is checked by cup product of $E^{\prime}$. First, we
notice that $d^{\prime}_{2}(u)=0$ if $u$ is in a regular representation part
$H^{p}({\mathbb{R}P^{\infty}};{\mathbb{F}_{2}[\Sigma_{2}]})$ of $E_{2}^{\prime
p,q}$. Indeed, if $d^{\prime}(u)\neq 0$, we have
$d^{\prime}(u)\alpha=d^{\prime}(u\alpha)\neq 0$. However, $u\alpha$ is zero
because $u\alpha$ is an element in the regular representation part
$H^{p+1}({\mathbb{R}P^{\infty}};{\mathbb{F}_{2}[\Sigma_{2}]\otimes_{\mathbb{F}_{2}}\mathbb{F}_{2}})$
of $E_{2}^{\prime p+1,q}$. Thus we have $d^{\prime}_{2}=0$.

On the other hand, the basis of the trivial representation part of
$H^{*}({B\Sigma_{2}};{\mathcal{H}^{q}({T^{d}\times T^{d}};{\mathbb{F}_{2}})})$
consists of $e^{*}_{i}\otimes e^{*}_{i},\alpha$ and all of their products.
However, we have $d^{\prime}_{3}(e^{*}_{i}\otimes e^{*}_{i})=0$ as follows.
Since the $\Sigma_{2}$-action on $T^{d}\times T^{d}$ has a fixed point, a
section of the Borel construction 3.0.7 is obtained as the middle vertical map
in the diagram

(3.0.13)

So any power $\alpha^{i}$ of the first Stiefel-Whitney class $\alpha$ cannot
be zero in $H^{*}({(T^{d}\times
T^{d})\times_{\Sigma_{2}}E\Sigma_{2}};{\mathbb{F}_{2}})$. Hence,
$d^{\prime}_{3}(e^{*}_{i}\otimes e^{*}_{i})=0$, and the Leibniz rule implies
that $d^{\prime}_{3}=0$. Inductively, we have $d^{\prime}_{r}=0$ for arbitrary
$r\geqslant 2$. ∎

Now we can determine the $E_{\infty}$-term of the Serre spectral sequence for
$d=3$. The $E_{2}$-term is indicated as the figure below:

$p$$q$$1$$1$$1$$1$$1$$1$$1$$1$$\alpha$$\alpha^{2}$$\alpha^{3}$$\alpha^{4}$$\alpha^{5}$$\alpha^{6}$$\alpha^{7}$$3$$0$$0$$0$$0$$0$$0$$0$$9$$e^{*}_{i}\otimes
e^{*}_{i}$$3$$3$$3$$3$$3$$3$$3$$10$$x$$1$$1$$1$$1$$1$$1$$1$$9$$e^{*}_{i}e^{*}_{j}\otimes
e^{*}_{i}e^{*}_{j}$$6$$6$$6$$6$$6$$6$$6$$3$$3$$3$$3$$3$$3$$3$$3$$d_{2}$$d_{2}$$d_{2}$$d_{2}$
Figure. 5. $E_{2}$-term

The bottom $d_{2}$ is zero similar to the case $d=2$. Note that
$d_{r}(e^{*}_{i}\otimes e^{*}_{i})=0$ for arbitrary $r\geqslant 2$. This
implies that each $e^{*}_{i}e^{*}_{j}\otimes e^{*}_{i}e^{*}_{j}$ is a
permanent cycle, hence so is $(e^{*}_{i}e^{*}_{j}\otimes
e^{*}_{i}e^{*}_{j})\alpha^{2}$. So $x\alpha^{2}$ cannot be hit. To vanish
these three permanent cycles $(e^{*}_{i}e^{*}_{j}\otimes
e^{*}_{i}e^{*}_{j})\alpha^{2}$, the rank of the $d_{2}$ at the top in the
figure has to be $3$. By Theorem 1.3, the element $x$, given by the formula in
the proof of Theorem 1.1, has to hit to $\alpha^{4}$ at $E_{4}$-term. So
$d_{2}(x)$ and $d_{3}(x)$ are zero, and $d_{4}(x)=\alpha^{4}$. In summary, the
rank of the top $d_{2}$ in the figure is $3$, and the others are zero.
Therefore, the $E_{3}$-term is indicated as the following figure.

$p$$q$$1$$1$$1$$1$$1$$1$$1$$1$$\alpha$$\alpha^{2}$$\alpha^{3}$$\alpha^{4}$$\alpha^{5}$$\alpha^{6}$$\alpha^{7}$$3$$0$$0$$0$$0$$0$$0$$0$$9$$e^{*}_{i}\otimes
e^{*}_{i}$$3$$3$$3$$3$$3$$3$$3$$10$$x$$1$$1$$1$$1$$1$$1$$1$$9$$6$$3$$3$$3$$3$$3$$3$$0$$0$$0$$0$$0$$0$$0$$0$$d_{3}$$d_{3}$
Figure. 6. $E_{3}$-term

At the $E_{3}$-term, the remarkable differential is the right $d_{3}$ in the
figure. Since its target is generated by the permanent cycles
$(e^{*}_{i}\otimes e^{*}_{i})\alpha^{4}$, the rank of the right $d_{3}$ has to
be $3$ to vanish such cycles at $E_{\infty}$. Hence, the left $d_{3}$ is rank
$3$. Finally, $d_{4}(x)=\alpha^{4}$ implies that the $E_{\infty}$-term is
indicated as the following figure.

$p$$q$$1$$1$$1$$1$$0$$0$$0$$0$$\alpha$$\alpha^{2}$$\alpha^{3}$$3$$0$$0$$0$$0$$0$$0$$0$$9$$3$$3$$0$$0$$0$$0$$0$$9$$0$$0$$0$$0$$0$$0$$0$$6$$3$$0$$0$$0$$0$$0$$0$$0$$0$$0$$0$$0$$0$$0$$0$
Figure. 7. $E_{\infty}$-term

Finally, we choose generators $u_{i,j}$ and $x_{i,j}$ as generators of
$E_{\infty}^{0,i}$. In particular, let $u_{i,j}$ be in the regular
representation parts, and $x_{i,j}$ in the trivial representation parts. Here
we pay attention to that each $u_{i,j}\alpha^{k}(k\geqslant 1)$ might be
nonzero in $H^{*}({B_{2}({T^{d}})};{\mathbb{F}_{2}})$ in spite of it is zero
in $E_{\infty}$ (extension problem). However, we can choose $u_{i,j}$ so that
$u_{i,j}\alpha$ is zero in $H^{*}({B_{2}({T^{d}})};{\mathbb{F}_{2}})$. Indeed,
if $u_{i,j}\alpha$ is nonzero in $H^{*}({B_{2}({T^{d}})};{\mathbb{F}_{2}})$,
there is a positive integer $m$ and an element $y\in
H^{i}({B_{2}({T^{d}})};{\mathbb{F}_{2}})$ such that
$u_{i,j}\alpha=y\alpha^{m}$. Then replacing $u_{i,j}$ to
$u^{\prime}_{i,j}=u_{i,j}+y\alpha^{m-1}$, we obtain $u^{\prime}_{i,j}\alpha=0$
in $H^{*}({B_{2}({T^{d}})};{\mathbb{F}_{2}})$. This completes the proof of
Theorem 1.2.

## Acknowledgement

The author appreciates Mitsunobu Tsutaya and Takahiro Matsushita for
commenting on this study. The author was also supported by WISE program (MEXT)
at Kyushu University.

## References

  * [1] C.-F.Bödigheimer, F.Cohen, and L.Taylor. On the homology of configuration spaces. Topology, 28(1):111–123, 1989.
  * [2] Matthew Chen and Adela YiYu Zhang. Mod p homology of unordered configuration spaces of surfaces, preprint, arXiv:2208.10293, 2022.
  * [3] Yves Félix and Daniel Tanré. The Cohomology Algebra of Unordered Configuration Spaces. Journal of the London Mathematical Society, 72(2):525–544, 10 2005.
  * [4] Allen Hatcher. Algebraic topology. Cambridge University Press, Cambridge, 2002.
  * [5] Daisuke Kishimoto and Takahiro Matsushita. Van Kampen-Flores theorem and Stiefel-Whitney classes, preprint, arXiv:2304.08043, 2023.
  * [6] Roberto Pagaria. The cohomology rings of the unordered configuration spaces of elliptic curves. Algebraic Geo- metric Topology, 20(6):2995 – 3012, 2020.


# Non-cooperative Systems Modeling Repulsive Interaction of Bose-Einstein
Condensates in $\mathbb{R}^{N}$

Liliane A. Maia  Departamento de Matemática, UNB, 70910-900 Brasília, Brazil.
<EMAIL_ADDRESS>,  Mayra Soares  Departamento de Matemática, UNB, 70910-900
Brasília, Brazil<EMAIL_ADDRESS>and  Ricardo Ruviaro  Departamento de
Matemática, UNB, 70910-900 Brasília, Brazil<EMAIL_ADDRESS>
###### Abstract.

Inspired by so many possible applications of this class of problems, we seek
solution for non-cooperative elliptic systems of two Schrödinger equations.
General conditions are assumed under the potentials, which produces convenient
spectral properties on the elliptic operator concerned, hence the non-
cooperation characterizes it as a strongly indefinite problem. An Abstract
Linking Theorem developed previously by the first two authors is the main
tool, since variational approach is applied. Furthermore, super and
asymptotically quadratic nonlinearities are considered.

Research supported by FAPDF 0193.001300/2016, CNPq/PQ 308378/2017-2 (Brazil),
and PROEX/CAPES, Brazil.

##  1\. Introduction

This work is devoted to solve non-cooperative elliptic systems under
hypotheses on the potentials introduced by B. Sirakov in [13]. For $N\geq 2$
we consider the elliptic system given by

(1.1) $\left\\{\begin{array}[]{ll}-\Delta
u+V_{1}(x)u=F_{u}(x,u,v)\quad\;\;\text{in}\quad\mathbb{R}^{N}\\\ -\Delta
v+V_{2}(x)v=-F_{v}(x,u,v)\quad\text{in}\quad\mathbb{R}^{N},\end{array}\right.$

where, for $i=1,2$, the potentials $V_{i}\in
L^{\infty}_{loc}(\mathbb{R}^{N}),$ satisfy

$(V_{1})_{i}$ There exist constants $a_{i}\geq 0$ such that $V_{i}(x)\geq-
a_{i}\quad\text{for \ all}\quad x\in\mathbb{R}^{N};$

$(V_{2})_{i}$ Defining for all $\Omega\subset\mathbb{R}^{N}$ open,

$\nu_{i}(\Omega):=\inf\left\\{\int_{\Omega}\left(|\nabla
u(x)|^{2}+V_{i}(x)u^{2}(x)\right)\,dx:u\in H^{1}_{0}(\Omega),\
||u||_{L^{2}(\Omega)}=1\right\\},$

assume that $\displaystyle\lim_{r\to+\infty}\nu_{i}(\mathbb{R}^{N}\setminus
B_{r})=+\infty$, where $B_{r}=\\{x\in\mathbb{R}^{N}:|x|\leq r\\}$;

$(V_{3})_{i}$ Setting the first eigenvalue of operator
$A_{i}:=-\Delta+V_{i}(x)$ in $L^{2}(\mathbb{R}^{N})$ by

$\lambda^{i}_{1}:=\inf\left\\{\int_{\mathbb{R}^{N}}\left(|\nabla
u(x)|^{2}+V_{i}(x)u^{2}(x)\right)\,dx:u\in H^{1}(\mathbb{R}^{N}),\
||u||_{L^{2}(\mathbb{R}^{N})}=1\right\\},$

assume that $\lambda^{i}_{1}>0.$

Note that classes of potentials satisfying conditions $(V_{1})_{i}$ and
$(V_{2})_{i}$ are

$V_{i}(x)\to+\infty\quad\text{as}\quad|x|\to+\infty,$

or

$V_{i}(x)>V_{0}>0\quad\text{and}\quad\dfrac{1}{V_{i}(x)}\in
L^{1}(\mathbb{R}^{N}),$

or even such that

$|\Omega^{V_{i}}_{M}|<+\infty\quad\text{for \ all}\quad M>0,$

where $\Omega^{V_{i}}_{M}=\\{x\in\mathbb{R}^{N}:V_{i}(x)<M\\}$. However,
potentials such as $V_{i}(x)=x_{1}^{2}x_{2}^{2}\ ...\ x_{n}^{2}-a_{i}$, with
constants $a_{i}>0$ such that $\lambda^{i}_{1}>0$, satisfy conditions
$(V_{1})_{i}-(V_{3})_{i}$ although they do not satisfy any of the conditions
mentioned above. Furthermore, the following hypothesis

$\lim_{r\to+\infty}\left|\bigcap_{i=1,2}\Big{(}\Omega^{V_{i}}_{M}\Big{)}\setminus(B_{r})\right|=0,\quad\text{for
\ all}\quad M>0,$

is a sufficient condition for hypothesis $(V_{2})_{i}$ (cf. [13], Theorem 1.4
and [12], Remark 1.3).

$(V)$ There exists $h(x)\in L^{\infty}_{loc}(\mathbb{R}^{N})$ and constants
$\alpha>1$, $c_{0}>0$, $r_{0}>0$ such that

$h(x)\leq
c_{0}\left[1+\big{(}\max_{i=1,2}\\{0,V_{i}(x)\\}\big{)}^{\frac{1}{\alpha}}\right],\quad\text{if}\quad|x|\geq
r_{0}.$

Moreover, setting $a_{0}:=\displaystyle\inf_{x\in\mathbb{R}^{N}}h(x)$, assume
that $a_{0}>{\lambda^{1}_{1}}>0$.

An example which fulfils this hypothesis is given just before the main result
of Section 2.

Once again inspired by [13], suitable hypotheses involving function $h(x)$ are
required on $F$ for the both asymptotically and super quadratic cases.

A relevant motivation to study problem (1.1) stems from the system

$\left\\{\begin{array}[]{lll}i\dfrac{\partial\Phi_{1}}{\partial
t}=\Delta\Phi_{1}-V_{1}(x)\Phi_{1}-\mu_{1}|\Phi_{1}|^{2}\Phi_{1}-\beta_{12}|\Phi_{2}|^{2}\Phi_{1}\\\
\\\ i\dfrac{\partial\Phi_{2}}{\partial
t}=\Delta\Phi_{2}-V_{2}(x)\Phi_{2}-\mu_{2}|\Phi_{2}|^{2}\Phi_{2}-\beta_{21}|\Phi_{1}|^{2}\Phi_{2},\end{array}\right.$

which has been used to model Bose-Einstein condensates in two different
hyperfine spin states with the corresponding condensate wave functions
$\Phi_{j},\ j=1,2$. In this case, $V_{j}$ is the magnetic trapping potential
for the respective hyperfine spin state, the constants $\mu_{j}$ and
$\beta_{jk}$ are the intra-species and interspecies scattering lengths, which
represent the interactions between particles. If $\beta_{12}>0$ it means there
is repulsive interaction between particles $1$ and $2$, on the other hand, if
$\beta_{12}<0$, there exists attractive interaction between them. In purpose
of finding its solitary wave solutions of the form
$\Phi_{j}=e^{-i\lambda_{j}t}u_{j}(x),\ \lambda_{j}>0,$ for $j=1,2$, the system
above is transformed into the elliptic system

$\left\\{\begin{array}[]{ll}-\Delta
u_{1}+\big{(}V_{1}(x)+\lambda_{1}\big{)}u_{1}+\mu_{1}u^{3}_{1}+\beta_{12}u^{2}_{2}u_{1}=0\\\
-\Delta
u_{2}+\big{(}V_{2}(x)+\lambda_{2}\big{)}u_{2}+\mu_{2}u^{3}_{2}+\beta_{21}u^{2}_{1}u_{2}=0.\end{array}\right.$

Assuming $\beta_{12}\beta_{21}<0$ this elliptic system is non-cooperative and
therefore it is an practical example of our problem. For details of physical
applications see [3, 9] and references therein.

Originally, our motivation to study (1.1) came from [4, 5]. Indeed, in [4] D.
G. Costa treated the cooperative elliptic system

$\left\\{\begin{array}[]{ll}-\Delta
u+a(x)u=F_{u}(x,u,v)\quad\text{in}\quad\mathbb{R}^{N}\\\ -\Delta
v+b(x)v=F_{v}(x,u,v)\quad\;\text{in}\quad\mathbb{R}^{N},\end{array}\right.$

under the hypotheses that $a(x),\ b(x)\to+\infty$ as $n\to+\infty$ and the
nonlinear term $F$ is non-quadratic at infinity. Moreover, he mentioned that
his results could be generalized to non-cooperative elliptic systems applying
techniques found in [5]. Following this spirit and trying to generalize the
results in [4] for non-cooperative elliptic systems, we have profited from
ideas in the remarkable paper [13], where B. Sirakov considered a scalar
problem under more general hypotheses on the potentials and worked with a
super-quadratic nonlinearity.

To the best of our knowledge, non-cooperative elliptic systems under
assumptions similar to those introduced by B. Sirakov have not been treated
yet. In fact, most of the well known papers in non-cooperative elliptic
systems deals with constant potentials or even considers only bounded domains,
as for instance, [5, 6, 11, 2] and references therein. Although we are also
able to find in the literature some relevant works on non-cooperative elliptic
systems with non-constant potentials and treated on the whole space, normally
their authors consider $V=V_{i},\ i=1,2$, and assume hypotheses away from
those required by us, check for example [15, 7].

Due to hypotheses $(V_{1})_{i}$ and $(V_{3})_{i}$, problem (1.1) must be
tackled in the subspace of $H^{1}(\mathbb{R}^{N},\mathbb{R}^{2})$ defined by

(1.2) $E:=\left\\{(u_{1},u_{2})\in
H^{1}(\mathbb{R}^{N},\mathbb{R}^{2}):\int_{\mathbb{R}^{N}}\big{(}V_{1}(x)u_{1}^{2}(x)+V_{2}(x)u_{2}^{2}(x)\big{)}\,dx<+\infty\right\\},$

which is a Hilbert space continuously embedded in
$H^{1}(\mathbb{R}^{N},\mathbb{R}^{2})$ when endowed with the inner product

(1.3)
$\Big{(}(u_{1},u_{2}),(v_{1},v_{2})\Big{)}_{E}:=\sum_{i=1,2}\int_{\mathbb{R}^{N}}\big{(}\nabla
u_{i}(x)\cdot\nabla v_{i}(x)+V_{i}(x)u_{i}(x)v_{i}(x)\big{)}\,dx,$

and the correspondent norm
$\big{|}\big{|}(u_{1},u_{2})\big{|}\big{|}^{2}_{E}:=\Big{(}(u_{1},u_{2}),(u_{1},u_{2})\Big{)}_{E}$
(cf. [12] and [13], both Lemma 2.1). Furthermore, $E$ is compactly embedded in
the weighted spaces $L^{s}_{h(x)}(\mathbb{R}^{N},\mathbb{R}^{2})$, for $2\leq
s<+\infty$ if $N=2$, and for $2\leq s<2^{\\#}:=2^{*}-\dfrac{4}{\alpha(N-2)}$
if $N>2$ (cf. [12], Proposition 2.4 and also [13], Proposition 3.1). Hence,
the spectra of operators $A_{i}$ are discrete and consist of a sequence
${\lambda_{n}^{i}}$ of positive eigenvalues such that
$\lambda^{i}_{n}\to+\infty$ as $n\to+\infty.$ (cf. [13], Lemma 3.2). Note that
$2^{\\#}$ is determined such that $2<2^{\\#}\leq 2^{*}$ taking into account
that $\alpha>1$.

In our point of view, the most interesting feature of hypotheses on $V_{i}$ is
that the elliptic operator associated to problem (1.1) has pure point
spectrum. Although such a spectrum has negative and positive parts composed by
unbounded sequences, which makes the problem strongly indefinite, its
discreteness brings the necessary compactness for the problem. Furthermore,
the narrow relation between potentials $V_{i}$ and nonlinearity $F$, enables
us to get the desired geometry for the problem. In view of these facts, we are
able to find a non trivial solution for (1.1).

All hypotheses and observations made up to now, will be assumed throughout
this paper. From now, asymptotically and super quadratic cases are going to be
treated separately. Section 2 is dedicated to the asymptotically quadratic
problem. In order to do so, we recall the Abstract Linking Theorem in [10],
which is used to prove the existence of solution in this case, as well as in
the super-quadratic case. First, exploiting the profile of spectrum presented
by the associated operator we decompose space $E$ in appropriate subspaces for
the linking structure. In this environment, the requirements of the abstract
result are verified: compactness, linking geometry and boundedness of Cerami
sequences of the functional associated to the problem. Throughout Section 3,
we follow the same script for the super quadratic case, under the necessary
changes. In this case, it is worth to focus in the argument for the
boundedness of the sequences. Here, the well known Ambrosetti-Rabinowitz
condition [1] would not be sufficient as in the scalar problem. So, to
circumvent this difficulty we betake the resourceful hypothesis $(f5)$
introduced by L. Jeanjean and K. Tanaka in [8]. Finally, we point out that
this scheme can be applied to systems with more than two equations.

##  2\. Asymptotically Quadratic Elliptic Systems

With the purpose of studying the asymptotically quadratic case the following
assumptions on the Hamiltonian $F$ are required.

$(F_{1})$ $F(x,s,t)\in
C^{1}(\mathbb{R}^{N}\times\mathbb{R}^{2},\mathbb{R}^{+})$ and

$\dfrac{F(x,s,t)}{h(x)}=o(|(s,t)|^{2}),\quad\text{as}\quad|(s,t)|\to 0,$
$\dfrac{F(x,s,t)}{|(s,t)|^{2}}=h(x),\quad\text{as}\quad|(s,t)|\to+\infty,$

uniformly in $x\in\mathbb{R}^{N}$.

$(F_{2})$ Setting $Q(x,s,t):=F_{s}(x,s,t)s+F_{t}(x,s,t)t-2F(x,s,t)$, for
$(x,s,t)\in\mathbb{R}^{N}\times\mathbb{R}^{2}$, one has

$\lim_{|(s,t)|\to+\infty}Q(x,s,t)=+\infty,$

uniformly in $x\in\mathbb{R}^{N}$.

######  Remark 1.

It is worth to highlight the close relation between hypotheses $(V)$ and
$(F_{1})$. Since a linking geometry is sought, these assumptions provide the
required features for the functional associated to problem (1.1). Thereby,
note that, if instead of $F\geq 0$ as in $(F_{1})$, it was considered $F\leq
0$, then in $(V)$ it would be asked $a_{0}>\lambda_{1}^{2}$. Indeed, in this
case, problem (1.1) could be treated in the same way, provided that the
positions of the equations were exchanged.

In view of $(F_{1})$, given $\varepsilon>0$, for $2\leq p<+\infty$ if $N=2$
and for $2\leq p<2^{\\#}$ if $N>2$, there exist constants $C>0$ and
$\tilde{C}_{\varepsilon}>0$ such that

(2.1) $\big{|}F(x,u(x),v(x))\big{|}\leq
Ch(x)\Big{(}\varepsilon|(u(x),v(x))|^{2}+\tilde{C}_{\varepsilon}|(u(x),v(x))|^{p}\Big{)},$

almost everywhere in $\mathbb{R}^{N}$. In addition, assuming also hypotheses
$(V_{1})_{i}-(V_{3})$ for $i=1,2$ and $(V)$, the functional

$\int_{\mathbb{R}^{N}}\big{|}F(x,u(x),v(x))\big{|}\ dx$

belongs to $C^{1}(E,\mathbb{R})$, since for all $\varepsilon>0$ given it
satisfies

(2.2) $\int_{\mathbb{R}^{N}}\big{|}F(x,u(x),v(x))\big{|}\
dx\leq\varepsilon||(u,v)||^{2}_{E}+C_{\varepsilon}||(u,v)||^{p}_{E},$

for some constant $C_{\varepsilon}>0$ (cf. [12], Lemma 2.2 and also [13],
Lemma 3.1).

In virtue of $(V)$ and $(F_{1})$ it is possible to assert that the growth of
$F$ is controlled by the growth of $V_{i}$ by means of $h$, hence $F$ can be
unbounded but under some restrictions. For instance, a function $F$ such that

$\Big{(}F_{s}(x,s,t),F_{t}(x,s,t)\Big{)}=c_{0}h(x)\left(\dfrac{s^{3}}{1+s^{2}},\dfrac{t^{3}}{1+t^{2}}\right),$

or

$\Big{(}F_{s}(x,s,t),F_{t}(x,s,t)\Big{)}=c_{0}h(x)\left(\dfrac{s(s^{2}+t^{2})}{1+s^{2}+t^{2}},\dfrac{t(s^{2}+t^{2})}{1+s^{2}+t^{2}}\right),$

where $h(x)\geq a_{0}>\lambda^{1}_{1}$,
$V_{i}(x)\geq\big{[}h(x)\big{]}^{\alpha}$ for $|x|>r_{0}$, with $c_{0},\
a_{0},\ r_{0}$ positive constants and $\alpha>1$, satisfy assumptions $(V)$
and $(F_{1})$.

The main result of this section is stated below.

######  Theorem 2.1.

Under the assumptions $(V_{1})_{i}-(V_{3})_{i},(V)$ and $(F_{1})-(F_{2})$,
system (1.1) has a nontrivial solution in
$H^{1}(\mathbb{R}^{N},\mathbb{R}^{2}).$

###  2.1. Variational Framework

In order to tackle problem (1.1) by means of variational methods, the strongly
indefinite functional $I:E\to\mathbb{R}$ given by

(2.3) $\displaystyle I(u,v)$ $\displaystyle=$
$\displaystyle\dfrac{1}{2}\int_{\mathbb{R}^{N}}\Big{(}|\nabla
u(x)|^{2}+V_{1}(x)u^{2}(x)\Big{)}\,dx$
$\displaystyle-\dfrac{1}{2}\int_{\mathbb{R}^{N}}\Big{(}|\nabla
v(x)|^{2}+V_{2}(x)v^{2}(x)\ \Big{)}\,dx-\int_{\mathbb{R}^{N}}F(x,u(x),v(x))\
dx$ $\displaystyle=$
$\displaystyle\dfrac{1}{2}\Big{(}||(u,0)||^{2}_{E}-||(0,v)||_{E}^{2}\Big{)}-\int_{\mathbb{R}^{N}}F(x,u(x),v(x))\
dx,$

is associated to the elliptic system (1.1). Thereby, $I\in
C^{1}(E,\mathbb{R})$ and

(2.4) $\displaystyle I^{\prime}(u,v)(\varphi,\psi)$ $\displaystyle=$
$\displaystyle\int_{\mathbb{R}^{N}}\big{(}\nabla
u(x)\cdot\nabla\varphi(x)+V_{1}(x)u(x)\varphi(x)\big{)}\,dx$
$\displaystyle-\int_{\mathbb{R}^{N}}\big{(}\nabla
v(x)\cdot\nabla\psi(x)+V_{2}(x)v(x)\psi(x)\big{)}\;dx$
$\displaystyle-\int_{\mathbb{R}^{N}}F_{u}(x,u(x),v(x))\varphi(x)\
dx-\int_{\mathbb{R}^{N}}F_{v}(x,u(x),v(x))\psi(x)\ dx$ $\displaystyle=$
$\displaystyle\Big{(}(u,0),(\varphi,0)\Big{)}_{E}-\Big{(}(0,v),(0,\psi)\Big{)}_{E}$
$\displaystyle-\int_{\mathbb{R}^{N}}\Big{(}F_{u}(x,u(x),v(x))\varphi(x)+F_{v}(x,u(x),v(x))\psi(x)\Big{)}\;dx,$

hence critical points of $I$ are weak solutions for (1.1).

To obtain a nontrivial critical point of the functional $I$ we make use of an
abstract linking theorem proved by the authors in [10], which we now recall.

######  Theorem 2.2.

Linking Theorem for Cerami Sequences: Let $E$ be a real Hilbert space, with
inner product $\big{(}\cdot,\cdot\big{)}$, $E_{1}$ a closed subspace of $E$
and $E_{2}=E_{1}^{\perp}$. Let $I\in C^{1}(E,\mathbb{R})$ satisfying:

$(I_{1})\ \ I(u)=\dfrac{1}{2}\big{(}Lu,u\big{)}+B(u),$ for all $u\in E$, where
$u=u_{1}+u_{2}\in E_{1}\oplus E_{2}$, $Lu=L_{1}u_{1}+L_{2}u_{2}$ and
$L_{i}:E_{i}\rightarrow E_{i},\ i=1,2$ is a bounded linear self adjoint
mapping.

$(I_{2})\ \ B$ is weakly continuous and uniformly differentiable on bounded
subsets of $E$.

$(I_{3})\ $ There exist Hilbert manifolds $S,Q\subset E$, such that $Q$ is
bounded and has boundary $\partial Q$, constants $\alpha>\omega$ and $v\in
E_{2}$ such that
$(i)\ S\subset v+E_{1}$ and $I\geq\alpha$ on $S$;
$(ii)\ I\leq\omega$ on $\partial Q$;
$(iii)\ S$ and $\partial Q$ link.

$(I_{4})\ \ $ If for a sequence $(u_{n})$, $I(u_{n})$ is bounded and
$\left(1+||u_{n}||\right)||I^{\prime}(u_{n})||\to 0$, as $n\to+\infty$, then
$(u_{n})$ is bounded.
Then I possesses a critical value $c\geq\alpha$.

Thus, from now on, the plan is to find a critical point of $I$ by applying
Theorem 2.2, since $I$ is an indefinite functional. Henceforth, it is
necessary to check that $I$ satisfies $(I_{1})-(I_{4})$.

First, it is convenient to analyse operator $A:=(A_{1},-A_{2})$ for $A_{i}$
defined in $(V_{3})_{i}$ as an operator of
$L^{2}(\mathbb{R}^{N},\mathbb{R}^{2})$ such that $A(u,v):=(A_{1}u,-A_{2}v)\in
L^{2}(\mathbb{R}^{N},\mathbb{R}^{2}),$ for all $(u,v)$ in the domain of $A$.
Moreover, for all $(u,v)\in E$, one has

(2.5)
$\Big{(}A(u,v),(u,v)\Big{)}_{L^{2}(\mathbb{R}^{N},\mathbb{R}^{2})}=\big{|}\big{|}(u,0)\big{|}\big{|}_{E}^{2}-\big{|}\big{|}(0,v)\big{|}\big{|}_{E}^{2}.$

Hence, defining

$E_{1}:=\Big{\\{}(u,0)\in
H^{1}(\mathbb{R}^{N},\mathbb{R}^{2}):\int_{\mathbb{R}^{N}}V_{1}(x)u^{2}(x)\
dx<+\infty\Big{\\}}$

and

$E_{2}:=\Big{\\{}(0,v)\in
H^{1}(\mathbb{R}^{N},\mathbb{R}^{2}):\int_{\mathbb{R}^{N}}V_{2}(x)v^{2}(x)\
dx<+\infty\Big{\\}},$

it yields $E:=E_{1}\oplus E_{2}$. Thus, setting $L_{1}:=Id:E_{1}\to E_{1}$,
$L_{2}:=-Id:E_{2}\to E_{2}$ and $L:E\to E$ with $L:=L_{1}+L_{2}$, where $Id$
is identity operator, it follows that

$\Big{(}L(u,v),(\varphi,\psi)\Big{)}_{E}=\Big{(}(u,0),(\varphi,0)\Big{)}_{E}-\Big{(}(0,v),(0,\psi)\Big{)}_{E}.$

Therefore, $I$ satisfies $(I_{1})$ in Theorem 2.2 since

(2.6) $\displaystyle\Big{(}L(u,v),(u,v)\Big{)}_{E}$ $\displaystyle=$
$\displaystyle\Big{(}(u,0),(u,0)\Big{)}_{E}-\Big{(}(0,v),(0,v)\Big{)}_{E}$
$\displaystyle=$ $\displaystyle||(u,0)||_{E}^{2}-||(0,v)||_{E}^{2}$
$\displaystyle=$
$\displaystyle\Big{(}A(u,v),(u,v)\Big{)}_{L^{2}(\mathbb{R}^{N},\mathbb{R}^{2})},$

and then

$I(u,v)=\dfrac{1}{2}\Big{(}L(u,v),(u,v)\Big{)}_{E}+B(u,v),$

where

(2.7) $B(u,v):=-\displaystyle\int_{\mathbb{R}^{N}}F(x,u(x),v(x))\ dx.$

######  Remark 2.

Since each $A_{i}$ has discrete and positive spectrum
$\sigma(A_{i})=\\{\lambda^{i}_{n}\\}$ for $i=1,2$, $A$ has discrete spectrum
$\sigma(A)=\displaystyle\\{\lambda^{1}_{n}\\}\displaystyle\cup\\{-\lambda^{2}_{n}\\}$.
In fact, since $E$ is compactly embedded in
$L^{s}_{h(x)}(\mathbb{R}^{N},\mathbb{R}^{2})$, it implies that $\sigma(A)$ is
discrete (cf. [12], Proposition 2.4). In addition, $\lambda$ is an eigenvalue
of $A$ iff there exists $(u,v)\in E\setminus\\{0\\}$ such that
$A(u,v)=\lambda(u,v)$, namely, $A_{1}u=\lambda u$ and $A_{2}v=(-\lambda)v$.
Provided that, $u\not=0$ or $v\not=0$, then $\lambda$ is an eigenvalue of $A$
iff it is an eigenvalue of $A_{1}$ or $-\lambda$ is an eigenvalue of $A_{2}$.
Therefore, $A$ has positive and negative spectra composed respectively by
$\lambda^{1}_{n}\to+\infty$ and $-\lambda^{2}_{n}\to-\infty$ as $n\to+\infty.$
Furthermore, $\lambda^{1}_{1}$ is the smallest positive eigenvalue of $A$ and
$-\lambda^{2}_{1}$ is the largest negative eigenvalue of $A$, and in virtue of
$(V_{3})_{i}$, $0\notin\sigma(A)$. In addition, $E_{1}$ and $E_{2}$ correspond
to the subspaces of $E$ where $A$ is positive definite, and negative definite,
respectively, and both are infinite dimensional.

###  2.2. Establishing Compactness

With the aim of showing that $I$ satisfies $(I_{2})$ in Theorem 2.2, the
following lemmas are stated and proved.

######  Lemma 2.3.

Under the hypotheses $(V_{1})_{i}-(V_{3})_{i},(V)$ and $(F_{1})$, functional
$B:E\to\mathbb{R}$ defined as in (2.7) is weakly continuous.

###### Proof.

Let $(u_{n},v_{n})\rightharpoonup(u,v)$ weakly in $E$, since $E$ is compactly
embedded in $L^{s}_{h(x)}(\mathbb{R}^{N},\mathbb{R}^{2})$ for $2\leq
s<+\infty$ if $N=2$ and for $2\leq s<2^{\\#}$ if $N>2$, then
$(u_{n},v_{n})\to(u,v)$, up to subsequences, strongly in
$L^{s}_{h(x)}(\mathbb{R}^{N},\mathbb{R}^{2})$. In view of (2.1), using
standard arguments and applying Lebesgue Dominated Convergence Theorem, it
yields

$\int_{\mathbb{R}^{N}}F(x,u_{n}(x),v_{n}(x))\
dx\to\int_{\mathbb{R}^{N}}F(x,u(x),v(x))\ dx,$

as $n\to+\infty,$ namely, $B(u_{n},v_{n})\to B(u,v)$ and $B$ is weakly
continuous. ∎

######  Lemma 2.4.

Assuming that $(V_{1})_{i}-(V_{3})_{i},(V)$ and $(F_{1})$ hold, functional $I$
is uniformly differentiable on bounded sets of $E$.

###### Proof.

Fixed $R>0$ and given $(u_{1},u_{2})+(v_{1},v_{2}),\ (v_{1},v_{2})\in
B_{R}\subset E$, the closed ball centered on the origin, one has

$\displaystyle\Big{|}B(u_{1}+v_{1},u_{2}+v_{2})-B(u_{1},u_{2})-B^{\prime}(u_{1},u_{2})(v_{1},v_{2})\Big{|}$
$\displaystyle=$
$\displaystyle\int_{\mathbb{R}^{N}}\Big{|}F\big{(}x,u_{1}+v_{1},u_{2}+v_{2}\big{)}-F\big{(}x,u_{1},u_{2}\big{)}-F_{u}\big{(}x,u_{1},u_{2}\big{)}v_{1}-F_{v}\big{(}x,u_{1},u_{2}\big{)}v_{2}\Big{|}dx$
$\displaystyle\leq$
$\displaystyle\int_{\mathbb{R}^{N}}\dfrac{1}{h(x)^{\frac{1}{2}}}\Big{|}F_{u}\big{(}x,z_{1},z_{2}\big{)}+F_{v}\big{(}x,z_{1},z_{2}\big{)}-F_{u}\big{(}x,u_{1},u_{2}\big{)}-F_{v}\big{(}x,u_{1},u_{2}\big{)}\Big{|}\
\big{|}h(x)^{\frac{1}{2}}(v_{1},v_{2})\big{|}dx$ $\displaystyle\leq$
$\displaystyle C_{2}||\xi||_{L^{2}(\mathbb{R}^{N})}||(v_{1},v_{2})||_{E},$

where

$\xi(x):=\dfrac{1}{h(x)^{\frac{1}{2}}}\Big{|}F_{u}\big{(}x,z_{1}(x),z_{2}(x)\big{)}+F_{v}\big{(}x,z_{1}(x),z_{2}(x)\big{)}-F_{u}\big{(}x,u_{1}(x),u_{2}(x)\big{)}-F_{v}\big{(}x,u_{1}(x),u_{2}(x)\big{)}\Big{|}$

and
$(z_{1}(x),z_{2}(x))=(u_{1}(x),u_{2}(x))+(\theta_{1}(x)v_{1}(x),\theta_{2}(x)v_{2}(x))$,
where $0\leq\theta_{i}(x)\leq 1$ is given by Mean Value Theorem and $C_{2}>0$
is the constant given by the continuous embedding $E\hookrightarrow
L^{2}_{h(x)}(\mathbb{R}^{N},\mathbb{R}^{2})$. So as to prove that $B$ is
uniformly differentiable on bounded sets of $E$, it is sufficient to show that
given $\varepsilon>0$ there exist $\delta>0$ such that
$C_{2}||\xi||_{L^{2}(\mathbb{R}^{N})}\leq\varepsilon$ for all
$(u_{1},u_{2})+(v_{1},v_{2}),\ (v_{1},v_{2})\in B_{R}\subset E$ with
$||(v_{1},v_{2})||_{E}\leq\delta$. However, by definition of limit, this is
equivalent to show that $\xi\to 0$ in $L^{2}(\mathbb{R}^{N})$ as
$(v_{1},v_{2})\to 0$ in $E$.

Observe that $(v_{1},v_{2})\to 0$ in $E$ implies $(v_{1},v_{2})\to 0$ in
$L^{s}_{h(x)}(\mathbb{R}^{N},\mathbb{R}^{2})$ for $2\leq s<+\infty$ if $N=2$
and for $2\leq s<2^{\\#}$ if $N>2$, then $(v_{1}(x),v_{2}(x))\to 0$ almost
everywhere in $\mathbb{R}^{N}$ and there exist $(\varphi_{s},\psi_{s})\in
L^{s}_{h(x)}(\mathbb{R}^{N},\mathbb{R}^{2})$ such that
$|v_{1}(x)|\leq\varphi_{s}(x)$, $|v_{2}(x)|\leq\psi_{s}(x)$ almost everywhere
in $\mathbb{R}^{N}$. Moreover, $(z_{1}(x),z_{2}(x))\to(u_{1}(x),u_{2}(x))$
almost everywhere in $\mathbb{R}^{N}$. Thus, $\xi(x)\to 0$ almost everywhere
in $\mathbb{R}^{N}$. Furthermore, from $(F_{1})$ and L’Hospital rule, one has

$\big{|}F_{u}(x,z_{1}(x),z_{2}(x))\big{|}\leq
Ch(x)\big{|}(z_{1}(x),z_{2}(x))\big{|}\quad\text{and}\quad\big{|}F_{v}(x,z_{1}(x),z_{2}(x))\big{|}\leq
Ch(x)\big{|}(z_{1}(x),z_{2}(x))\big{|},$

almost everywhere in $\mathbb{R}^{N}$, for some constant $C>0$ and the
analogous holds for $u_{i}$ instead of $z_{i}$. Then, for $s=2$ it follows
that

(2.9) $\displaystyle|\xi(x)|^{2}$ $\displaystyle\leq$
$\displaystyle\dfrac{2}{h(x)}\left[\Big{|}F_{u}\big{(}x,z_{1}(x),z_{2}(x)\big{)}\Big{|}+\Big{|}F_{v}\big{(}x,z_{1}(x),z_{2}(x)\big{)}\Big{|}\right]^{2}$
$\displaystyle+\,\dfrac{2}{h(x)}\left[\Big{|}F_{u}\big{(}x,u_{1}(x),u_{2}(x)\big{)}\Big{|}+\Big{|}F_{v}\big{(}x,u_{1}(x),u_{2}(x)\big{)}\Big{|}\right]^{2}$
$\displaystyle\leq$
$\displaystyle\dfrac{2}{h(x)}\left[\Big{(}2Ch(x)|(z_{1}(x),z_{2}(x))|\Big{)}^{2}+\Big{(}2Ch(x)|(u_{1}(x),u_{2}(x))|\Big{)}^{2}\right]$
$\displaystyle\leq$ $\displaystyle
24C^{2}\Big{[}h(x)|(\varphi_{2}(x),\psi_{2}(x))|^{2}+h(x)|(u_{1}(x),u_{2}(x))|^{2}\Big{]}.$

Therefore, by applying Lebesgue Dominated Convergence Theorem it yields

$||\xi||^{2}_{L^{2}(\mathbb{R}^{N})}=\int_{\mathbb{R}^{N}}|\xi(x)|^{2}\ dx\to
0,$

as $(v_{1},v_{2})\to 0$ in $E$ and the result hold. ∎

###  2.3. Linking Geometry

Under the purpose of proving that $I$ satisfies $(I_{3})$ in Theorem 2.2, as
usual, set

$S:=(\partial B_{\rho}\cap E_{1})\quad\text{and}\quad{Q:=\\{(re,v):r\geq
0,(0,v)\in E_{2},||(re,v)||_{E}\leq r_{1}\\}},$

where $0<\rho<r_{1}$ are constants, $(e,0)\in E_{1}$ and $e$ is an
eigenfunction associated to $\lambda_{1}^{1}$ the first eigenvalue of $A_{1}$,
with $||(e,0)||_{E}=1$, therefore, such $S$ and $Q$ “link”. Due to the strict
inequality $a_{0}>\lambda_{1}^{1}$ in hypothesis $(V)$ and from (2.5), such an
$e$ satisfies

(2.10) $\displaystyle 1$ $\displaystyle=$
$\displaystyle||(e,0)||_{E}^{2}=\Big{(}A(e,0),(e,0)\Big{)}_{L^{2}(\mathbb{R}^{N},\mathbb{R}^{2})}$
$\displaystyle=$ $\displaystyle\int_{\mathbb{R}^{N}}\big{(}|\nabla
e(x)|^{2}+V_{1}(x)e^{2}(x)\big{)}\,dx=(A_{1}e,e)_{L^{2}(\mathbb{R}^{N})}$
$\displaystyle=$
$\displaystyle\lambda_{1}^{1}||e||^{2}_{L^{2}(\mathbb{R}^{N})}$
$\displaystyle<$ $\displaystyle
a_{0}||(e,0)||^{2}_{L^{2}(\mathbb{R}^{N},\mathbb{R}^{2})}$ $\displaystyle\leq$
$\displaystyle\int_{\mathbb{R}^{N}}h(x)|(e(x),0)|^{2}\ dx.$

Thus, by means of (2.10) it is possible to show that if $r_{1}>0$ is large
enough, then $I|_{S}\geq\alpha>0$ and $I|_{\partial Q}\leq 0$ for some
$\alpha>0$. Thus, $I$ satisfies $(I_{3})$ for some $\alpha>0,\ \omega=0$ and
arbitrary $(0,v)\in E_{2}$. Indeed, next lemma gives the result.

######  Lemma 2.5.

Suppose that $(V_{1})_{i}-(V_{3})_{i}$, $(V)$ and $(F_{1})$ hold, then $I$
satisfies $(I_{3})$.

###### Proof.

Note that $S\subset E_{1}$, then from (2.3) and (2.2), for all $(u_{1},0)\in
S$, it yields

(2.11) $\displaystyle I(u_{1},0)$ $\displaystyle\geq$
$\displaystyle\dfrac{1}{2}\rho^{2}-\Big{(}\varepsilon||(u_{1},0)||^{2}_{E}+C_{\varepsilon}||(u_{1},0)||^{p}_{E}\Big{)}$
$\displaystyle=$
$\displaystyle\rho^{2}\left[\left(\dfrac{1}{2}-\varepsilon\right)-C_{\varepsilon}\rho^{p-2}\right]$
$\displaystyle\geq$ $\displaystyle\alpha>0,$

where $\varepsilon,\rho$ are chosen small enough. Therefore, from (2.11),
$(I_{3})\ (i)$ holds for $I$.

With the purpose of proving that $I$ satisfies $(I_{3})\ (ii)$ in Theorem 2.2,
with $\omega=0$, observe that $I(0,v)\leq 0$, for all $(0,v)\in E_{2}$, then
it suffices to show that $I(re,v)\leq 0$ for $r>0,\ (0,v)\in E_{2}$ and
$||(re,v)||_{E}\geq r_{1}$, for some $r_{1}>0$ large enough. Arguing
indirectly assume that some sequence
$(r_{n}e,v_{n})\subset\mathbb{R}^{+}e\oplus E_{2}$ satisfies
$||(r_{n}e,v_{n})||_{E}\to+\infty$ and $I(r_{n}e,v_{n})>0$ for all
$n\in\mathbb{N}$. Seeking a contradiction, set

$(s_{n}e,w_{n}):=\dfrac{(r_{n}e,v_{n})}{||(r_{n}e,v_{n})||_{E}},$

where $s_{n}\in\mathbb{R}^{+},(0,w_{n})\in E_{2}$ and
$||(s_{n}e,w_{n})||_{E}=1$. Provided that $(s_{n}e,w_{n})$ is bounded, up to
subsequences, it follows that $(s_{n}e,w_{n})\rightharpoonup(se,w)$ in $E$,
hence $(s_{n}e,w_{n})\to(se,w)$ in
$L^{2}_{h(x)}(\mathbb{R}^{N},\mathbb{R}^{2})$, due to the compact embedding
$E\hookrightarrow L^{2}_{h(x)}(\mathbb{R}^{N},\mathbb{R}^{2})$. Then, up to
subsequences, $(s_{n}e(x),w_{n}(x))\to(se(x),w(x))$ almost everywhere in
$\mathbb{R}^{N}$, $s_{n}\to s$ in $\mathbb{R}^{+}$ and
$(0,w_{n})\rightharpoonup(0,w)$ in $E_{2}$, since $s_{n}$ and $(0,w_{n})$ are
also bounded. Noting that

$1=||(s_{n}e,w_{n})||_{E}^{2}=s_{n}^{2}+||(0,w_{n})||_{E}^{2},$

it follows that $0\leq s^{2}_{n}\leq 1$, and it yields

(2.12) $\displaystyle\dfrac{I(r_{n}e,v_{n})}{||(r_{n}e,v_{n})||_{E}^{2}}$
$\displaystyle=$
$\displaystyle\dfrac{1}{2}\Big{(}{s^{2}_{n}}||(e,0)||_{E}^{2}-||(0,w_{n})||_{E}^{2}\Big{)}-\int_{\mathbb{R}^{N}}\dfrac{F(x,r_{n}e(x),v_{n}(x))}{||(r_{n}e,v_{n})||_{E}^{2}}\
dx$ $\displaystyle=$ $\displaystyle
s^{2}_{n}-\dfrac{1}{2}-\int_{\mathbb{R}^{N}}\dfrac{F(x,r_{n}e(x),v_{n}(x))}{||(r_{n}e,v_{n})||_{E}^{2}}\
dx>0,$

hence $0<s\leq 1$. Moreover, from (2.10) one has
$1<\displaystyle\int_{\mathbb{R}^{N}}h(x)|(e(x),0)|^{2}\ dx.$ Then,

(2.13) $\displaystyle 0$ $\displaystyle>$ $\displaystyle
s^{2}-s^{2}\displaystyle\int_{\mathbb{R}^{N}}h(x)|(e(x),0)|^{2}\ dx$
$\displaystyle>$ $\displaystyle
s^{2}\left(1-\displaystyle\int_{\mathbb{R}^{N}}h(x)|(e(x),0)|^{2}\
dx\right)-\dfrac{1}{2}-\displaystyle\int_{\mathbb{R}^{N}}h(x)|(0,w(x))|^{2}\
dx.$

On the other hand, since $(s_{n}e,v_{n})$ is convergent in
$L^{2}_{h(x)}(\mathbb{R}^{N},\mathbb{R}^{2})$, there exists some
$(\varphi_{2},\psi_{2})\in L^{2}_{h(x)}(\mathbb{R}^{N},\mathbb{R}^{2})$ such
that $|s_{n}e(x)|\leq\varphi_{2}(x)$ and $|w_{n}(x)|\leq\psi_{2}(x)$ almost
everywhere in $\mathbb{R}^{N}$ and from (2.1) with $p=2$, it follows that

$\left|\dfrac{F(x,r_{n}e(x),v_{n}(x))}{||(r_{n}e,v_{n})||_{E}^{2}}\right|\leq\tilde{C}h(x)|(s_{n}e(x),w_{n}(x))|^{2}\leq\tilde{C}h(x)|(\varphi_{2}(x),\psi_{2}(x))|^{2},$

almost everywhere in $\mathbb{R}^{N}$, for some constant $\tilde{C}>0$.
Provided that $||(r_{n}e,v_{n})||_{E}\to+\infty$, and
$(s_{n}e(x),w_{n}(x))\to(se(x),w(x))\not=0$, almost everywhere in $supp(e)$,
it follows that
$(r_{n}e(x),v_{n}(x))=(s_{n}e(x),w_{n}(x))||(r_{n}e,v_{n})||_{E}\to+\infty$
almost everywhere in $supp(e)$, as $n\to+\infty$, hence in view of $(F_{1})$
one has

$\displaystyle\dfrac{F(x,r_{n}e(x),v_{n}(x))}{||(r_{n}e,v_{n})||_{E}^{2}}$
$\displaystyle=$
$\displaystyle\dfrac{F\big{(}x,||r_{n}e+u_{n}||_{E}(s_{n}e(x),w_{n}(x)\big{)}|(s_{n}e(x),w_{n}(x))|^{2}}{||r_{n}e+u_{n}||_{E}^{2}|(s_{n}e(x),w_{n}(x))|^{2}}$
$\displaystyle=$ $\displaystyle h(x)|(se(x),w(x))|^{2}+o_{n}(1),$

almost everywhere in $supp(se,w)$ as $n\to+\infty$. Note that,
$supp(se,w)\not=\emptyset$, because $supp(e)\not=\emptyset$. Thus, by Lebesgue
Dominated Convergence Theorem,

$\int_{\mathbb{R}^{N}}\dfrac{F(x,r_{n}e(x),v_{n}(x))}{||(r_{n}e,v_{n})||_{E}^{2}}dx\to\int_{\mathbb{R}^{N}}h(x)\big{|}(se(x),w(x))\big{|}^{2}\
dx,$

as $n\to+\infty.$ From (2.12) one has

$s^{2}_{n}-\dfrac{1}{2}-\int_{\mathbb{R}^{N}}\dfrac{F(x,r_{n}e(x),v_{n}(x))}{||(r_{n}e,v_{n})||_{E}^{2}}\
dx>0.$

Passing to the limit as $n\to+\infty$, it yields

(2.14) $\displaystyle 0$ $\displaystyle\leq$ $\displaystyle
s^{2}-\dfrac{1}{2}-\int_{\mathbb{R}^{N}}h(x)\big{|}(se(x),w(x))\big{|}^{2}\
dx$ $\displaystyle=$ $\displaystyle
s^{2}\left(1-\displaystyle\int_{\Omega}h(x)|(e(x),0)|^{2}\
dx\right)-\dfrac{1}{2}-\displaystyle\int_{\Omega}h(x)|(0,w(x))|^{2}\ dx,$

which is contrary to (2.13). Therefore the result holds. ∎

###  2.4. Boundedness of Cerami Sequences

Next lemma ensures $I$ satisfies last hypothesis in Theorem 2.2. Finally, with
this result it will be possible to prove Theorem 2.1.

######  Lemma 2.6.

Assuming that $(V_{1})_{i}-(V_{3})_{i}$, $(V)$ and $(F_{1})-(F_{2})$ hold, $I$
satisfies $(I_{4})$.

###### Proof.

Let $b>0$ be an arbitrary constant, and take $(u_{n},v_{n})\subset
I^{-1}([c-b,c+b])$ such that
$\big{(}1+||(u_{n},v_{n})||_{E}\big{)}||I^{\prime}(u_{n},v_{n})||_{E^{\prime}}\to
0$, it is necessary to show that $(u_{n},v_{n})$ is bounded. Suppose by
contradiction that $||(u_{n},v_{n})||_{E}\to+\infty$, up to subsequences.
Setting
$(\tilde{u}_{n},\tilde{v}_{n}):=\dfrac{(u_{n},v_{n})}{||(u_{n},v_{n})||_{E}}$,
it is bounded, hence
$(\tilde{u}_{n},\tilde{v}_{n})\rightharpoonup(\tilde{u},\tilde{v})$ in $E$ and
due to the compact embeddings $E\hookrightarrow
L^{s}_{h(x)}(\mathbb{R}^{N},\mathbb{R}^{2})$ for $2\leq s<+\infty$ if $N=2$
and $2\leq s<2^{\\#}$ if $N>2$, it implies that
$(\tilde{u}_{n},\tilde{v}_{n})\to(\tilde{u},\tilde{v})$ in
$L^{s}_{h(x)}(\mathbb{R}^{N},\mathbb{R}^{2})$. Writing
$(u_{n},v_{n})=(u_{n},0)+(0,v_{n})\in E_{1}\oplus E_{2}$, it satisfies

(2.15) $\displaystyle o_{n}(1)$ $\displaystyle=$
$\displaystyle\dfrac{1}{||(u_{n},v_{n})||_{E}}I^{\prime}(u_{n},v_{n})(\tilde{u}_{n},0)$
$\displaystyle=$
$\displaystyle||(\tilde{u}_{n},0)||_{E}^{2}-\int_{\mathbb{R}^{N}}\dfrac{F_{u}(x,u_{n}(x),v_{n}(x))}{||(u_{n},v_{n})||_{E}}\tilde{u}_{n}(x)\
dx$ $\displaystyle=$
$\displaystyle||(\tilde{u}_{n},0)||_{E}^{2}-\int_{\mathbb{R}^{N}}\dfrac{F_{u}(x,u_{n}(x),v_{n}(x))}{u_{n}(x)}\tilde{u}^{2}_{n}(x)\
dx$

and

(2.16) $\displaystyle o_{n}(1)$ $\displaystyle=$
$\displaystyle\dfrac{1}{||(u_{n},v_{n})||_{E}}I^{\prime}(u_{n},v_{n})(0,\tilde{v}_{n})$
$\displaystyle=$
$\displaystyle-||(0,\tilde{v}_{n})||_{E}^{2}-\int_{\mathbb{R}^{N}}\dfrac{F_{v}(x,u_{n}(x),v_{n}(x))}{v_{n}(x)}\tilde{v}^{2}_{n}(x)\
dx.$

Subtracting (2.16) from (2.15), and using that
$1=||(\tilde{u}_{n},0)||_{E}^{2}+||(0,\tilde{v}_{n})||_{E}^{2}$, it yields

(2.17)
$o_{n}(1)=1-\int_{\mathbb{R}^{N}}\left[\dfrac{F_{u}(x,u_{n}(x),v_{n}(x))}{u_{n}(x)}\tilde{u}^{2}_{n}(x)-\dfrac{F_{v}(x,u_{n}(x),v_{n}(x))}{v_{n}(x)}\tilde{v}^{2}_{n}(x)\right]dx.\\\
$

Provided that $(\tilde{u}_{n},\tilde{v}_{n})\to(\tilde{u},\tilde{v})$ in
$L_{h(x)}^{2}(\mathbb{R}^{N},\mathbb{R}^{2})$, there exist
$(\varphi_{2},\psi_{2})\in L^{2}_{h(x)}(\mathbb{R}^{N},\mathbb{R}^{2})$ such
that $|\tilde{u}_{n}(x)|\leq\varphi_{2}(x)$ and
$|\tilde{v}_{n}(x)|\leq\psi_{2}(x)$, almost everywhere in $\mathbb{R}^{N}$.
Furthermore, from $(F_{1})$ it follows that

$\left|\dfrac{F_{u}(x,u_{n}(x),v_{n}(x))}{u_{n}(x)}\right|\leq
Ch(x)\quad\text{and}\quad\left|\dfrac{F_{v}(x,u_{n}(x),v_{n}(x))}{v_{n}(x)}\right|\leq
Ch(x),$

for some constant $C>0$, almost everywhere in $\mathbb{R}^{N}$. Hence,

$\displaystyle\left|\dfrac{F_{u}(x,u_{n}(x),v_{n}(x))}{u_{n}(x)}\tilde{u}^{2}_{n}(x)-\dfrac{F_{v}(x,u_{n}(x),v_{n}(x))}{v_{n}(x)}\tilde{v}^{2}_{n}(x)\right|$
$\displaystyle\leq$ $\displaystyle
Ch(x)\Big{(}\tilde{u}^{2}_{n}(x)+\tilde{v}^{2}_{n}(x)\Big{)}$
$\displaystyle\leq$ $\displaystyle
Ch(x)\Big{(}\varphi^{2}_{2}(x)+\psi^{2}_{2}(x)\Big{)}.$

Since $(\tilde{u}_{n},\tilde{v}_{n})\to(\tilde{u},\tilde{v})$ in
$L^{2}_{h(x)}(\mathbb{R}^{N},\mathbb{R}^{2})$,
$|(u_{n}(x),v_{n}(x))|\to+\infty$, for all $x\in\mathbb{R}^{N}$ such that
$(\tilde{u}(x),\tilde{v}(x))\not=0$, then from $(F_{1})$ and L’Hospital rule,
it follows that

$\dfrac{F_{u}(x,u_{n}(x),v_{n}(x))}{u_{n}(x)}\tilde{u}^{2}_{n}(x)-\dfrac{F_{v}(x,u_{n}(x),v_{n}(x))}{v_{n}(x)}\tilde{v}^{2}_{n}(x)\to
2h(x)\left[\tilde{u}^{2}(x)-\tilde{v}^{2}(x)\right],$

as $n\to+\infty$, for all $x\in\mathbb{R}^{N}$
$(\tilde{u}(x),\tilde{v}(x))\not=0$. Therefore, by Lebesgue Dominated
Convergence Theorem one has

(2.18)
$\int_{\mathbb{R}^{N}}\left[\dfrac{F_{u}(x,u_{n}(x),v_{n}(x))}{u_{n}(x)}\tilde{u}^{2}_{n}(x)-\dfrac{F_{v}(x,u_{n}(x),v_{n}(x))}{v_{n}(x)}\tilde{v}^{2}_{n}(x)\right]dx\to\int_{\mathbb{R}^{N}}2h(x)\left[\tilde{u}^{2}(x)-\tilde{v}^{2}(x)\right]dx.$

Hence, passing to the limit in (2.17) as $n\to+\infty$, it yields

(2.19)
$\int_{\mathbb{R}^{N}}2h(x)\left[\tilde{u}^{2}(x)-\tilde{v}^{2}(x)\right]dx=1,$

which implies there exists $\Omega\subset\mathbb{R}^{N}$ such that
$|\Omega|>0$ and $(\tilde{u}(x),\tilde{v}(x))\not=0$ for all $x\in\Omega$,
otherwise (2.19) yields a contradiction.

On the other hand, since $(u_{n},v_{n})$ is a Cerami sequence, there exists a
constant $M_{0}>0$ such that

(2.20) $\displaystyle M_{0}$ $\displaystyle\geq$ $\displaystyle
I(u_{n},v_{n})-\dfrac{1}{2}I^{\prime}(u_{n},v_{n})(u_{n},v_{n})$
$\displaystyle=$
$\displaystyle\dfrac{1}{2}\int_{\mathbb{R}^{N}}\Big{(}F_{u}(x,u_{n},v_{n})u_{n}-F_{v}(x,u_{n},v_{n})v_{n}-2F(x,u_{n},v_{n})\Big{)}\,dx$
$\displaystyle\geq$
$\displaystyle\dfrac{1}{2}\int_{\Omega}Q(x,u_{n}(x),v_{n}(x))\ dx.$

Provided that $|(u_{n}(x),v_{n}(x))|\to+\infty$, for all $x\in\Omega$, in view
of $(F_{2})$, it yields

$\dfrac{1}{2}\int_{\Omega}Q(x,u_{n}(x),v_{n}(x))\ dx\to+\infty,$

as $n\to+\infty.$ Passing (2.20) to the limit, a contradiction holds. ∎

At long last, the main result is proved.

###### Proof of Theorem 2.1.

Provided that $I$ satisfies all assumptions $(I_{1})-(I_{4})$ in Theorem 2.2,
it ensures a critical point $(u,v)\in E$ of $I$, with $I(u,v)=c\geq\alpha>0$.
It implies that $I^{\prime}(u,v)(\varphi,\psi)=0$, for all $(\varphi,\psi)\in
E$ and $I(u,v)>=I(0,0)$, thus $(u,v)\not=0$. Since $I\in C^{1}(E,\mathbb{R})$,
it yields that $(u,v)$ is a nontrivial solution for system (1.1). ∎

##  3\. Super Quadratic Elliptic Systems

The following assumptions on the Hamiltonian $F$ are required for the super
quadratic case.

$(F^{\prime}_{1})$ $F(x,s,t)\in
C^{1}(\mathbb{R}^{N}\times\mathbb{R}^{2},\mathbb{R}^{+})$,

$\dfrac{\left|\big{(}F_{s}(x,s,t),F_{t}(x,s,t)\big{)}\right|}{h(x)}=o(|(s,t)|)\quad\text{as}\quad|(s,t)|\to
0,\quad\text{uniformly \ in}\quad x,$

and for some $2<p<+\infty$ if $N=2$ or $2<p<2^{\\#}:=\dfrac{4}{\alpha(N-2)}$
if $N>2$,

$\left|\big{(}F_{s}(x,s,t),F_{t}(x,s,t)\big{)}\right|\leq
c_{0}h(x)\big{(}1+|(s,t)|^{p-1}\big{)},\quad\text{for \
all}\quad(x,s,t)\in\mathbb{R}^{N}\times\mathbb{R}^{2};$

$(F^{\prime}_{2})$
$\displaystyle\lim_{|(s,t)|\to+\infty}\dfrac{F(x,s,t)}{|(s,t)|^{2}}=+\infty,\quad\text{uniformly
\ in}\quad x.$

Note that for such a $p$ given in $(F^{\prime}_{1})$ and arbitrary
$\varepsilon>0$, there exist constants $C>0,\ \tilde{C}_{\varepsilon}>0$ such
that (2.1) and (2.2) hold. Hence, the functional

$\int_{\mathbb{R}^{N}}\big{|}F(x,u(x),v(x))\big{|}\ dx$

belongs to $C^{1}(E,\mathbb{R})$ (cf. [12], Lemma 2.2 and also [13], Lemma
3.1).

In addition, for $Q(x,u,v)$ defined as in $(F_{2})$, so as to ensure the
boundedness of Cerami sequences, the following condition is required.

$(F^{\prime}_{3})$ There exists a constant $D\geq 1$ such that

$Q(x,w,z)\leq DQ(x,u,v),$

for all $(u,v),(w,z)\in\mathbb{R}^{2}$ with $|(w,z)|\leq|(u,v)|$.

It is worth to point out that hypothesis $(F^{\prime}_{3})$ was introduced by
Jeanjean and Tanaka [8], in order to bound Cerami sequences in cases where the
so called Ambrosetti-Rabinowitz condition [1] is not satisfied by the
nonlinearity, for example, when
$F(x,s,t)=c_{0}h(x)(\log|(s,t)|)^{\frac{1}{2}}(s^{2},t^{2})$, for $|(s,t)|\geq
1$. An alternative condition for the boundedness is mentioned at the end of
this work.

Similarly to the asymptotically quadratic case, the relation between $(V)$ and
$(F^{\prime}_{1})$ shows that the growth of $F$ is controlled by the growth of
$V_{i}$ by means of $h$, hence $F$ can be unbounded but under some
restrictions. For instance, a function $F$ such that

$\Big{(}F_{s}(x,s,t),F_{t}(x,s,t)\Big{)}=c_{0}h(x)\left(|(s,t)|^{p-2}s,|(s,t)|^{p-2}t\right),$

where $h(x)\geq a_{0}>\lambda^{1}_{1}$,
$V_{i}(x)\geq\big{[}h(x)\big{]}^{\alpha}$ for $|x|>r_{0}$, with $c_{0},\
a_{0},\ r_{0}$ positive constants and $\alpha>1$, satisfy assumptions $(V)$
and $(F_{1})$.

Now, the main result of this section is stated.

######  Theorem 3.1.

Suppose that $(V_{1})_{i}-(V_{3})_{i},(V)$ and
$(F^{\prime}_{1})-(F^{\prime}_{3})$ hold, then system (1.1) has a nontrivial
solution in $H^{1}(\mathbb{R}^{N},\mathbb{R}^{2}).$

Under the purpose of proving Theorem 3.1, the variational structure
established in Subsection 2.1. is assumed here. Thereby, $I$ satisfies
$(I_{1})$ in Theorem 2.2, which must be applied so as to get a non trivial
critical point of $I$. Furthermore, the conclusions of Lemmas 2.3 and 2.4, in
Subsection 2.2, are exactly the same replacing $(F_{1})$ by
$(F^{\prime}_{1})$. Thus, $(I_{2})$ is satisfied by $I$ as well.

###  3.1. Linking Structure

In order to prove that $I$ satisfies $(I_{3})$ in Theorem 2.2, all arguments
in Subsection 2.3 are adapted to the super quadratic case. As before,

$S:=(\partial B_{\rho}\cap E_{1})\quad\text{and}\quad{Q:=\\{(re,v):r\geq
0,(0,v)\in E_{2},||(re,v)||_{E}\leq r_{1}\\}},$

where $0<\rho<r_{1}$ are constants and $(e,0)\in E_{1}$ is an arbitrary vector
with $||(e,0)||_{E}=1$, therefore, such $S$ and $Q$ “link”. Next lemma shows
that if $r_{1}>0$ is large enough, then $I|_{S}\geq\alpha>0$ and $I|_{\partial
Q}\leq 0$ for some $\alpha>0$. Thus, $I$ satisfies $(I_{3})$ for some
$\alpha>0,\ \omega=0$ and arbitrary $(0,v)\in E_{2}$.

######  Lemma 3.2.

Assume that $(V_{1})_{i}-(V_{3})_{i}$, $(V)$ and
$(F^{\prime}_{1})-(F^{\prime}_{2})$ hold, then $I$ satisfies $(I_{3})$.

Although $(I_{3})\ (i)$ holds for $I$ by the same argument found in Lemma 2.5,
for the sake of completeness, this proof is repeated here.

###### Proof.

From (2.3) and (2.2), for all $(u_{1},0)\in S$, it yields (2.11) and likewise
$(I_{3})\ (i)$ holds for $I$.

For showing $(I_{3})\ (ii)$ with $\omega=0$, since $I(0,v)\leq 0$, for all
$(0,v)\in E_{2}$, it is enough to prove that $I(re,v)\leq 0$ for $r>0,\
(0,v)\in E_{2}$ and $||(re,v)||_{E}\geq r_{1}$, for some $r_{1}>0$
sufficiently large. Seeking a contradiction, assume that some sequence
$(r_{n}e,v_{n})\subset\mathbb{R}^{+}e\oplus E_{2}$ satisfies
$||(r_{n}e,v_{n})||_{E}\to+\infty$ and $I(r_{n}e,v_{n})>0$ for all
$n\in\mathbb{N}$. Set

$(s_{n}e,w_{n}):=\dfrac{(r_{n}e,v_{n})}{||(r_{n}e,v_{n})||_{E}},$

where $s_{n}\in\mathbb{R}^{+},(0,w_{n})\in E_{2}$ and
$||(s_{n}e,w_{n})||_{E}=1$. Provided that $(s_{n}e,w_{n})$ is bounded, up to
subsequences, it follows that $(s_{n}e,w_{n})\rightharpoonup(se,w)$ in $E$,
hence $(s_{n}e,w_{n})\to(se,w)$ in
$L^{2}_{h(x)}(\mathbb{R}^{N},\mathbb{R}^{2})$, due to the compact embedding
$E\hookrightarrow L^{2}_{h(x)}(\mathbb{R}^{N},\mathbb{R}^{2})$. Then, up to
subsequences, $(s_{n}e(x),w_{n}(x))\to(se(x),w(x))$ almost everywhere in
$\mathbb{R}^{N}$, $s_{n}\to s$ in $\mathbb{R}^{+}$ and
$(0,w_{n})\rightharpoonup(0,w)$ in $E_{2}$. Since
$1=||(s_{n}e,w_{n})||_{E}^{2}=s_{n}^{2}+||(0,w_{n})||_{E}^{2},$ it follows
that $0\leq s^{2}_{n}\leq 1$, and it yields

(3.1) $\displaystyle 0<\dfrac{I(r_{n}e,v_{n})}{||(r_{n}e,v_{n})||_{E}^{2}}$
$\displaystyle=$
$\displaystyle\dfrac{1}{2}\Big{(}{s^{2}_{n}}||(e,0)||_{E}^{2}-||(0,w_{n})||_{E}^{2}\Big{)}-\int_{\mathbb{R}^{N}}\dfrac{F(x,r_{n}e(x),v_{n}(x))}{||(r_{n}e,v_{n})||_{E}^{2}}\
dx$ $\displaystyle=$ $\displaystyle
s^{2}_{n}-\dfrac{1}{2}-\int_{\mathbb{R}^{N}}\dfrac{F(x,r_{n}e(x),v_{n}(x))}{||(r_{n}e,v_{n})||_{E}^{2}}\
dx$ $\displaystyle=$ $\displaystyle
s^{2}_{n}-\dfrac{1}{2}-\int_{\mathbb{R}^{N}}\dfrac{F(x,r_{n}e(x),v_{n}(x))}{|(r_{n}e(x),v_{n}(x))|^{2}}{|(s_{n}e(x),w_{n}(x))|^{2}}\
dx,$

hence $0<s\leq 1$. Provided that $||(r_{n}e,v_{n})||_{E}\to+\infty$ and

$(s_{n}e(x),w_{n}(x))\to(se(x),w(x))\not=0$

almost everywhere in supp$(e)$, it follows that

$(r_{n}e(x),v_{n}(x))=(s_{n}e(x),w_{n}(x))||(r_{n}e,v_{n})||_{E}\to+\infty,$

almost everywhere in supp$(e)$, as $n\to+\infty$, hence in view of
$(F^{\prime}_{2})$ one has

$\dfrac{F(x,r_{n}e(x),v_{n}(x))}{|(r_{n}e(x),v_{n}(x))|^{2}}{|(s_{n}e(x),w_{n}(x))|^{2}}\to+\infty,$

almost everywhere in supp$(se,w)$ as $n\to+\infty$. Note that,
$supp(se,w)\not=\emptyset$, because supp$(e)\not=\emptyset$. Thus, by Fatou’s
Lemma,

(3.2)
$\int_{\mathbb{R}^{N}}\dfrac{F(x,r_{n}e(x),v_{n}(x))}{|(r_{n}e(x),v_{n}(x))|^{2}}{|(s_{n}e(x),w_{n}(x))|^{2}}\
dx\to+\infty,$

as $n\to+\infty.$ In virtue of (3.1) one has

$\dfrac{1}{2}\geq
s^{2}_{n}-\dfrac{1}{2}>\int_{\mathbb{R}^{N}}\dfrac{F(x,r_{n}e(x),v_{n}(x))}{|(r_{n}e(x),v_{n}(x))|^{2}}{|(s_{n}e(x),w_{n}(x))|^{2}}\
dx.$

Due to (3.2), passing to the limit as $n\to+\infty$ it yields a contradiction.
∎

######  Remark 3.

Note that the assumption $a_{0}>\lambda_{1}^{1}$ in $(V)$, is irrelevant in
the super quadratic case, since inequality (2.10) is not used to establish the
linking geometry as was done in Lemma 2.5. Therefore, for the super quadratic
case, it is just necessary to require $a_{0}>0$ in $(V)$. Thereby, if $F\leq
0$ were considered instead of $F\geq 0$, to treat the problem it would be
necessary only to exchange one equation for the another, requiring hypothesis
$(F^{\prime}_{2})$ for $-F\geq 0$ instead of $F$.

###  3.2. Boundedness of Cerami Sequences

Last hypothesis in Theorem 2.2 is ensured in this subsection, and subsequently
Theorem 3.1 is going to be proved.

######  Lemma 3.3.

Supposing that $(V_{1})_{i}-(V_{3})_{i},(V),(F^{\prime}_{1})-(F^{\prime}_{2})$
and $(F^{\prime\prime}_{3})$ hold true for $I$, it satisfies $(I_{4})$.

###### Proof.

Let $b>0$ be a constant such that $c-b>0$ and take ${(u_{n},v_{n})\subset
I^{-1}([c-b,c+b])}$ such that
$\big{(}1+||(u_{n},v_{n})||_{E}\big{)}||I^{\prime}(u_{n},v_{n})||_{E^{\prime}}\to
0$, it is necessary to show that $(u_{n},v_{n})$ is bounded. Seeking a
contradiction, suppose that $||(u_{n},v_{n})||_{E}\to+\infty$, up to
subsequences and define
$(\tilde{u}_{n},\tilde{v}_{n}):=\dfrac{(u_{n},v_{n})}{||(u_{n},v_{n})||_{E}}$,
the normalized sequence. Since it is bounded,
$(\tilde{u}_{n},\tilde{v}_{n})\rightharpoonup(\tilde{u},\tilde{v})$ in $E$, up
to subsequences, and due to the compact embeddings $E\hookrightarrow
L^{s}_{h(x)}(\mathbb{R}^{N},\mathbb{R}^{2})$ for $2\leq s<+\infty$ if $N=2$
and $2\leq s<2^{\\#}$ if $N>2$, it implies that
$(\tilde{u}_{n},\tilde{v}_{n})\to(\tilde{u},\tilde{v})$ in
$L^{s}_{h(x)}(\mathbb{R}^{N},\mathbb{R}^{2})$.

Suppose that $(\tilde{u}(x),\tilde{v}(x))=(0,0)$ almost everywhere in
$\mathbb{R}^{N}$ and for ${\varepsilon\in(0,1)}$ define
$U^{\varepsilon}_{n}=(t^{\varepsilon}_{n}u_{n},\varepsilon{t^{\varepsilon}_{n}}v_{n})\in
E$ such that

(3.3)
$I(U^{\varepsilon}_{n})=\max_{t\in[0,1]}I\big{(}tu_{n},\varepsilon{tv_{n}}\big{)}.$

Provided that $(\tilde{u},\tilde{v})\to 0$ in
$L^{s}_{h(x)}(\mathbb{R}^{N},\mathbb{R}^{2})$, from $(F^{\prime}_{1})$ for
each $s>0$ fixed, there exists $C_{s}>0$, which does not depend on
$\varepsilon$, such that

$\displaystyle\int_{\mathbb{R}^{N}}F\big{(}x,s\tilde{u}_{n}(x),\varepsilon{s\tilde{v}_{n}(x)}\big{)}\
dx$ (3.4) $\displaystyle\leq
C_{s}\left\\{||(\tilde{u}_{n},\tilde{v}_{n})||^{2}_{L^{2}_{h(x)}(\mathbb{R}^{N},\mathbb{R}^{2})}+||(\tilde{u}_{n},\tilde{v}_{n})||^{p}_{L^{p}_{h(x)}(\mathbb{R}^{N},\mathbb{R}^{2})}\right\\}\to
0,$

as $n\to+\infty$. Moreover, since $||(u_{n},v_{n})||_{E}\to+\infty$, for $n$
large enough one has

(3.5) $\displaystyle I(U^{\varepsilon}_{n})$ $\displaystyle\geq$
$\displaystyle I\left(\frac{su_{n}}{||(u_{n},v_{n})||_{E}},\frac{\varepsilon
sv_{n}}{||(u_{n},v_{n})||_{E}}\right)$ $\displaystyle=$
$\displaystyle\dfrac{s^{2}}{2}\Big{(}||(\tilde{u}_{n},0)||_{E}^{2}-\varepsilon^{2}||(0,\tilde{v}_{n})||_{E}^{2}\Big{)}-\int_{\mathbb{R}^{N}}F\big{(}x,s\tilde{u}_{n}(x),{\varepsilon
s\tilde{v}_{n}(x)}\big{)}\ dx.$

Since $I(u_{n},v_{n})\geq c-b>0$, it follows that
$||(\tilde{u}_{n},0)||_{E}^{2}>||(0,\tilde{v}_{n})||_{E}^{2}$. Combining this
inequality with
$1=||(\tilde{u}_{n},0)||_{E}^{2}+||(0,\tilde{v}_{n})||_{E}^{2}$, it implies
that $||(\tilde{u}_{n},0)||_{E}^{2}>\dfrac{1}{2}$. Hence, from (3.2) and (3.5)
it implies that

$\displaystyle\liminf_{n\to+\infty}I(U^{\varepsilon}_{n})$ $\displaystyle\geq$
$\displaystyle\dfrac{s^{2}}{2}\liminf_{n\to+\infty}\Big{(}||(\tilde{u}_{n},0)||_{E}^{2}-\varepsilon^{2}||(\tilde{u}_{n},0)||_{E}^{2}\Big{)}$
$\displaystyle=$
$\displaystyle\dfrac{s^{2}}{2}(1-\varepsilon^{2})\liminf_{n\to+\infty}||(\tilde{u}_{n},0)||_{E}^{2}$
$\displaystyle\geq$ $\displaystyle\dfrac{s^{2}}{4}(1-\varepsilon^{2}),$

for all $s>0$ and $\varepsilon\in(0,1)$. Thus,

(3.6) $\liminf_{n\to+\infty}I(U^{\varepsilon}_{n})=+\infty\quad\text{for \
all}\quad\varepsilon\in(0,1).$

On the other hand, if there exists a subsequence $(n_{k})\subset\mathbb{N}$,
such that for some $\varepsilon_{n_{k}}\in(0,1)$ one has
$t^{\varepsilon_{n_{k}}}_{n_{k}}\in(0,1)$, in view of (3.3) it implies that
$I^{\prime}(U^{\varepsilon_{n_{k}}}_{n_{k}})U^{\varepsilon_{n_{k}}}_{n_{k}}=0$.
In addition, since $t^{\varepsilon_{n_{k}}}_{n_{k}}\in(0,1)$, in virtue of
$(F^{\prime}_{3})$ it follows that

(3.7) $\displaystyle I(U^{\varepsilon_{n_{k}}}_{n_{k}})$ $\displaystyle=$
$\displaystyle
I(U^{\varepsilon_{n_{k}}}_{n_{k}})-\dfrac{1}{2}I^{\prime}(U^{\varepsilon_{n_{k}}}_{n_{k}})U^{\varepsilon_{n_{k}}}_{n_{k}}$
$\displaystyle=$
$\displaystyle\dfrac{1}{2}\int_{\mathbb{R}^{N}}Q\big{(}x,t^{\varepsilon_{n_{k}}}_{n_{k}}u_{n_{k}}(x),{\varepsilon_{n_{k}}}t^{\varepsilon_{n_{k}}}_{n_{k}}v_{n_{k}}(x)\big{)}\,dx$
$\displaystyle\leq$
$\displaystyle\dfrac{D}{2}\int_{\mathbb{R}^{N}}Q\big{(}x,u_{n_{k}}(x),v_{n_{k}}(x)\big{)}\,dx$
$\displaystyle=$
$\displaystyle\dfrac{D}{2}\Big{\\{}I(u_{n_{k}},v_{n_{k}})-\dfrac{1}{2}I^{\prime}(u_{n_{k}},v_{n_{k}})(u_{n_{k}},v_{n_{k}})\Big{\\}}$
$\displaystyle\leq$ $\displaystyle M,$

for some $M>0$, provided that $(u_{n_{k}},v_{n_{k}})$ is also a Cerami
sequence. However, from (3.7) one arrives at

$\limsup_{k\to+\infty}I(U^{\varepsilon_{n_{k}}}_{n_{k}})\leq M,$

contradicting (3.6). Thus, such a subsequence $(n_{k})\subset\mathbb{N}$
cannot exist and hence, there exists $n_{0}\in\mathbb{N}$ such that for all
$n\geq n_{0}$ and for all $\varepsilon\in(0,1)$ one has
$t^{\varepsilon}_{n}=1$. Furthermore, by the continuity of $I$ it yields

$I(U^{\varepsilon}_{n})=I(u_{n},\varepsilon v_{n})\to
I(u_{n},v_{n})\quad\text{as}\quad\varepsilon\to 1,$

for each $n\geq n_{0}$. Then, for each $n\geq n_{0}$ there exists
$\varepsilon_{n}\in(0,1)$ such that

(3.8) $I(U^{\varepsilon}_{n})<I(u_{n},v_{n})+\dfrac{1}{n},$

for all $\varepsilon\in[\varepsilon_{n},1]$. Hence, if
$\tilde{\varepsilon}_{n}\in[\varepsilon_{n},1]$ is such that
$I(U^{\tilde{\varepsilon}_{n}}_{n})\geq I(U^{\varepsilon}_{n})$ for all
$\varepsilon\in[\varepsilon_{n},1]$, in view of (3.8) it follows that

(3.9)
$\limsup_{n\to+\infty}I(U^{\tilde{\varepsilon}_{n}}_{n})\leq\limsup_{n\to+\infty}I(u_{n},v_{n})=c.$

However, from (3.6) for each $\varepsilon\in[\varepsilon_{n},1)$ there exists
$n_{\varepsilon}\in\mathbb{N}$ such that $n\geq n_{\varepsilon}$ implies
$I(U^{\varepsilon}_{n})\geq\dfrac{1}{1-\varepsilon}\geq\dfrac{1}{1-\varepsilon_{n}}$,
then, it yields

(3.10)
$\liminf_{n\to+\infty}I(U^{\tilde{\varepsilon}_{n}}_{n})\geq\liminf_{n\to+\infty}I(U^{\varepsilon}_{n})\geq\lim_{n\to+\infty}\dfrac{1}{1-\varepsilon_{n}}=+\infty,$

since $\varepsilon_{n}\to 1,$ in virtue of (3.8). Thus, (3.10) contradicts
(3.9).

Therefore, $(\tilde{u},\tilde{v})$ cannot be the null function, namely there
exists $\Omega\subset\mathbb{R}^{N}$ with $|\Omega|>0$, such that
$(\tilde{u}(x),\tilde{v}(x))\not=(0,0)$ for all $x\in\Omega$. Furthermore,
since

$\displaystyle\int_{\mathbb{R}^{N}}\dfrac{F(x,u_{n}(x),v_{n}(x))}{||(u_{n},v_{n})||_{E}^{2}}dx|$
$\displaystyle=$
$\displaystyle\dfrac{1}{2}\Big{(}||(\tilde{u}_{n},0)||_{E}^{2}-||(0,\tilde{v}_{n})||_{E}^{2}\Big{)}-\dfrac{I(u_{n},v_{n})}{||(u_{n},v_{n})||_{E}^{2}}$
$\displaystyle\leq$
$\displaystyle\dfrac{1}{2}||(\tilde{u}_{n},0)||_{E}^{2}\leq\dfrac{1}{2},$

one has

(3.11)
$\dfrac{1}{2}\geq\int_{\Omega}\dfrac{F(x,u_{n}(x),v_{n}(x))}{|(u_{n}(x),v_{n}(x))|^{2}}|(\tilde{u}_{n},\tilde{v}_{n})|^{2}dx.$

Nevertheless, in view of $(F^{\prime}_{2})$, for all $x\in\Omega$ it implies
that

$\dfrac{F(x,u_{n}(x),v_{n}(x))}{|(u_{n}(x),v_{n}(x))|^{2}}|(\tilde{u}_{n},\tilde{v}_{n})|^{2}\to+\infty,$

as $n\to+\infty$, since
$|(u_{n}(x),v_{n}(x))|=|(\tilde{u}_{n}(x),\tilde{v}_{n}(x))|\
||(u_{n},v_{n})||_{E}\to+\infty$ for all $x\in\Omega$, as $n\to+\infty$.
Passing (3.11) to the limit and applying Fatou’s Lemma, it yields

$\dfrac{1}{2}\geq\liminf_{n\to+\infty}\int_{\Omega}\dfrac{F(x,u_{n}(x),v_{n}(x))}{|(u_{n}(x),v_{n}(x))|^{2}}|(\tilde{u}_{n},\tilde{v}_{n})|^{2}dx=+\infty,$

which gives a contradiction. Therefore, $(u_{n},v_{n})$ is bounded, and the
result holds. ∎

At long last, the main result is proved.

###### Proof of Theorem 3.1.

The same proof of Theorem 2.1 is applicable. ∎

It is worth to mention that depending on the nonlinearity, the solutions
$(u,v)$ found in Theorem 2.1 or in Theorem 3.1 could be either semi-trivial
$(u,0)$ or vectorial $(u,v)$, but never semi-trivial $(0,v)$, since such
solutions have positive energy $c>0$.

We also would like to point out that, instead of asking for hypothesis
$(F^{\prime}_{3})$ one, alternatively, could ask for the following assumption
so as to get the boundedness of Cerami sequences.

$(F^{\prime\prime}_{3})$ There exist constants $c_{1}>0$ and
$\theta>\min\left\\{1,\dfrac{N}{2}(p-2)\right\\}$ such that for all
${(x,u,v)\in\mathbb{R}^{N}\times\mathbb{R}^{2}}$,

$Q(x,u,v)\geq c_{1}h(x)|(u,v)|^{\theta}.$

Since the functional associated to system (1.1) is indefinite, hypothesis
$(F^{\prime\prime}_{3})$ is relevant to establish the boundedness of Cerami
sequences. Indeed, $(F^{\prime\prime}_{3})$ is a consequence of both following
conditions $(AR)$ and $(F^{\prime}_{4})$, which also give the boundedness of
Cerami sequences.

$(AR)$ There exists $\mu>2$ such that for all
$(x,u,v)\in\mathbb{R}^{N}\times\mathbb{R}^{2}$,

$0\leq\mu F(x,u,v)\leq F_{u}(x,u,v)u+F_{v}(x,u,v)v;$

$(F^{\prime}_{4})$ There exist constants $c_{1}>0$ and
$\theta>\min\left\\{1,\dfrac{N}{2}(p-2)\right\\}$ such that for all
${(x,u,v)\in\mathbb{R}^{N}\times\mathbb{R}^{2}}$,

$F(x,u,v)\geq c_{1}h(x)|u,v|^{\theta}.$

Moreover, only hypothesis $(AR)$ is not sufficient for the boundedness,
provided that standard arguments involving this assumption only bounds the
difference

$||(u_{n},0)||_{E}^{2}-||(0,v_{n})||_{E}^{2},$

which appears in the functional. Furthermore, considering
$\theta>\dfrac{N}{2}\dfrac{\alpha}{\alpha-1}(p-2)$ instead of
$\theta>\dfrac{N}{2}(p-2)$ and exploiting the hypotheses assumed on $V_{i}$,
it is possible to weak hypothesis $(F^{\prime\prime}_{3})$ only assuming

$F_{u}(x,u,v)u+F_{v}(x,u,v)v-2F(x,u,v)\geq c_{1}|(u,v)|^{\theta}.$

In this way, it is possible to prove the following lemma.

######  Lemma 3.4.

Under the hypotheses $(V_{1})_{i}-(V_{3})_{i}$, $(V)$,
$(F^{\prime}_{1})-(F^{\prime}_{2})$ and $(F^{\prime\prime}_{3})$, $I$
satisfies $(I_{4})$.

This result is obtained by bounding the Cerami sequences first in
$L^{\theta}_{h(x)}(\mathbb{R}^{N},\mathbb{R}^{2})$ and after in $E$, making
use of the growth of $F$. For the details on this argument, see [14] and [12].

## References

  * [1] Ambrosetti, A. and Rabinowitz, P. H.: Dual Variational Methods in Critical Point Theory and Applications. J. Func. Anal., 14, 349-381, 1973.
  * [2] Bartsch, T. and Clapp, M.: Critical Point Theory for Indefinite Functionals with Symmetries. J. Func. Anal. 138, 107-136, 1996.
  * [3] Chang, S. M., Lin, C. S. Lin, T. C. and Lin, W. W.: Segregated Nodal Domains of Two-dimensional Multi-species Bose–Einstein Condensates. Physica D 196, 341-361, 2004.
  * [4] Costa, D. G.: On a Class of Elliptic Systems in $\mathbb{R}^{N}$. _Electronic Journal of Differential Equations_ , 7, 1-14, 1994.
  * [5] Costa, D. G. and Magalhães, C. A.: A Variational Approach to Non-cooperative Elliptic Systems. _Nonlinear Analysis, Theory, Methods & Applications_, 25 No 7, 699-715, 1995.
  * [6] De Figueiredo, D. G. and Ding, Y.: Strongly Indefinite Functionals and Multiple Solutions of Elliptic Systems. _Transactions of the American Mathematical Society_ , 365 No. 7, 2973-2989, 2003.
  * [7] Fan, M., Wang, J., Xiao, L. and Zhang, F.: Existence and multiplicity of semiclassical solutions for asymptotically Hamiltonian elliptic systems. _J. Math. Anal. Appl._ 339, 340-351, 2013.
  * [8] Jeanjean, L. and Tanaka, K.: Singularly Perturbed Elliptic Problems with Superlinear or Asymptotically Linear Nonlinearities. Calculus of Variations 21, 287-318, 2004.
  * [9] Lin, T. C. and Wei, J.: Symbiotic Bright Solitary Wave Solutions of Coupled Nonlinear Schrödinger equations. Nonlinearity 19, 2755-2773, 2006.
  * [10] Maia, L. A. and Soares, M.: An Abstract Linking Theorem Applied to Indefinite Problems via Spectral Properties. ArXiv:1807.11155v1, Preprint, 2018.
  * [11] Pomponio, A.:An asymptotically linear non-cooperative elliptic system with lack of compactness. The Royal Society, Mathematical, Physical and Engineering Sciences 459 No. 2037, 2265-2279, 2007.
  * [12] Rabelo, P.: Existence and Multiplicity of Solutions for a Class of Elliptic Systems in $\mathbb{R}^{N}$. Nonlinear Analysis 71, 2585-2599, 2009.
  * [13] Sirakov, B.: Existence and Multiplicity of Solutions of Semi-linear Elliptic Equations in $\mathbb{R}^{N}$. Calculus of Variations 11, 119-142, 2000.
  * [14] Soares, M.: _An Abstract Linking Theorem Applied to Indefinite Problems via Spectral Properties._ PhD Thesis, Department of Mathematics - UnB, 2018.
  * [15] Wang, J., Xu, J. and Zhang, F.: Existence and Multiplicity of Solutions for Asymptotically Hamiltonian Elliptic Systems in $\mathbb{R}^{N}$. _J. Math. Anal. Appl._ 367, 193-203, 2010.