# Linear stability of the elliptic relative equilibria for the restricted
$4$-body problem: the Euler case

Bowen Liu1 and Qinglong Zhou2
1 School of Mathematical Science,
Shanghai Jiao Tong University, Shanghai 200240, China
2 Department of Mathematics,
Zhejiang University, Hangzhou 310027, Zhejiang, China
Partially supported by NSFC (Nos. 12101394, 12171426), Science and Technology
Innovation Action Program of STCSM (No. 20JC1413200), Natural Science
Foundation of Shanghai No. 22ZR1433100 and Innovation Program of Shanghai
Municipal Education Commission. E-mail<EMAIL_ADDRESS>author. Partially supported by NSFC (No.12171426), the Natural Science
Foundation of Zhejiang Province (No. Y19A010072) and the Fundamental Research
Funds for the Central Universities (No. 2021FZZX001-01). E-mail:
<EMAIL_ADDRESS>
###### Abstract

In this paper, we consider the elliptic relative equilibria of the restricted
$4$-body problems, where the three primaries form an Euler collinear
configuration and the four bodies span ${\bf R}^{2}$. We obtain the symplectic
reduction to the general restricted $N$-body problem. By analyzing the
relationship between this restricted $4$-body problems and the elliptic
Lagrangian solutions, we obtain the linear stability of the restricted
$4$-body problem by the ${\omega}$-Maslov index. Via numerical computations,
we also obtain conditions of the stability on the mass parameters for the
symmetric cases.

Keywords: restricted $N$-body problem, elliptic Euler collinear solution,
reduction, linear stability.

AMS Subject Classification: 70F10, 70H14, 34C25.

##  1 Introduction and main results

In the classical planar $N$-body problems of celestial mechanics, the position
vectors of the $N$-particles are denoted by $q_{1},\dots,{q_{N}}\in{\bf
R}^{2}$, and the masses are represented by $m_{1},\dots,{m_{N}}>0$. By
Newton’s second law and the law of universal gravitation, the system of
equations is

$\displaystyle m_{i}\ddot{q_{i}}=\frac{\partial U}{\partial q_{i}},\quad
i=1,{\dots,N},$ (1.1)

where $U(q)=U(q_{1},{\dots,q_{N}})=\sum_{1\leq
i<j\leq{N}}\frac{m_{i}m_{j}}{|q_{i}-q_{j}|}$ is the potential function and
$|\cdot|$ is the standard norm of vector in ${\bf R}^{2}$. Suppose the
configuration space is

$\hat{\chi}:=\left\\{q=(q_{1},{\dots,q_{N}})\in({\bf
R}^{2})^{{N}}\l|\sum_{i=1}^{{N}}m_{i}q_{i}=0,q_{i}\neq q_{j},\forall i\neq
j\right\\}.$

For the period $T$, the corresponding action functional is

$\displaystyle\mathbf{A}(q)=\int_{0}^{T}\left[\sum_{i=1}^{{N}}\frac{m_{i}|\dot{q}_{i}(t)|^{2}}{2}+U(q(t))\right]{\mathrm{d}}t,$
(1.2)

which is defined on the loop space $W^{1,2}({\bf R}/T{\bf Z},\hat{\chi})$. The
periodic solutions of (1.1) correspond to critical points of the action
functional (1.2). Let $p_{1},{\dots,p_{N}}\in{\bf R}^{2}$ be the momentum
vectors of the particles respectively. It is well-known that (1.1) can be
reformulated as a Hamiltonian system by

$\displaystyle\dot{p}_{i}=-\frac{\partial H}{\partial
q_{i}},\;\dot{q}_{i}=\frac{\partial H}{\partial
p_{i}},\quad\mbox{for}\;i=1,{\dots,N},$ (1.3)

with the Hamiltonian function

$\displaystyle
H(p,q)=\sum_{i=1}^{{N}}\frac{|p_{i}|^{2}}{2m_{i}}-U(q_{1},{\dots,q_{N}}).$
(1.4)

One special class of periodic solutions to the planar $N$-body problem is the
elliptic relative equilibrium (ERE for short) [20]. It is generated by a
central configuration and the Keplerian motion. A central configuration (C.C.
for short) is formed by $N$ position vectors
$\left(q_{1},\ldots,q_{N}\right)=\left(a_{1},\ldots,a_{N}\right)$ which
satisfy

$\displaystyle-\lambda m_{i}q_{i}=\frac{\partial U}{\partial
q_{i}},\forall\;1\leq i\leq N,$ (1.5)

where $\lambda=U(a)/{I(a)}>0$ and $I(a)={\sum_{i=1}^{N}}m_{i}|a_{i}|^{2}$ is
the moment of inertia. A planar central configuration of the $N$-body problem
gives rise to a solution of (1.1) where each particle moves on a specific
Keplerian orbit while the totality of the particles move according to a
homothety motion.

The restricted 4-body problem is one special case of the general $4$-body
problem which are three primaries and one massless body. We assume that
$m_{1}+m_{2}+m_{3}=1$ and $m_{4}=0$. By $m_{4}=0$, (1.5) can be reduced to

$\displaystyle-\lambda q_{i}=$ $\displaystyle\sum_{1\leq j\leq 3,i\neq
j}\frac{m_{i}(q_{j}-q_{i})}{|q_{j}-q_{i}|^{3}},\quad\mbox{for}\quad 1\leq
i\leq 3$ (1.6) $\displaystyle-\lambda q_{4}=$ $\displaystyle\sum_{1\leq j\leq
3}\frac{m_{i}(q_{j}-q_{4})}{|q_{j}-q_{4}|^{3}}.$ (1.7)

The C.C. is decomposed to a 3-body problem and the actions on the massless
body from three primaries. For the 3-body problem, there exists two types of
the C.C.s: the Lagrangian equilateral configurations [8] and the Euler
collinear configurations [1]. If the three primaries form a Lagrangian
equilateral, the number of the C.C. of the corresponding restricted 4-body
problem has been well-studied [9]. For the Euler collinear case, we still have
two cases: the $4$ bodies are collinear and the four bodies span ${\bf
R}^{2}$. The case of the four bodies form a collinear configuration has been
discussed in [13]. In this paper, we focus on the case of $4$ bodies span
${\bf R}^{2}$.

The linear stability of the ERE is determined by the eigenvalues of the
linearized Poincaré map. Let ${\bf U}$ denote the unit circle in the complex
plane. The ERE is spectrally stable if all eigenvalues of linearized Poincaré
map are on ${\bf U}$; it is linearly stable if Poincaré map is semi-simple and
spectrally stable; it is linearly unstable if at least one pair of eigenvalues
are not on ${\bf U}$. Since the nineteenth century [23], the researches on the
stability have always been active in celestial mechanics because it reveals
the dynamics near the period orbits. However, it has always been one difficult
task to obtain the linear stability of ERE, because the linearized Hamiltonian
systems are non-autonomous, especially when for the elliptic orbits. Many
results on linear stability of the three-body problems have been obtained over
the past decades by numerical methods [17, 18, 19], bifurcation theory [22]
and the index theory [5, 3, 6, 25]. To the best of our knowledge, the
${\omega}$-Maslov index theory is the only analytical method to obtain the
full picture of the stability and instability to the ERE, such as the elliptic
Lagrangian solution [5, 3, 6], and the elliptic Euler solution [25, 26].

When it comes to four-body problems, the research on stability of ERE is quite
active in the past decades. One well-studied case is the elliptic rhombus
solution [16, 10, 12] which are linearly instable. For the restricted $4$-body
problem with three primaries forming Lagrangian equilateral configuration, the
full bifurcation diagram of the stability and instability has been obtained
for all possible masses and all eccentricity [13, 9]. Furthermore, one stable
region of the linear stability has been found. Regarding the linear stability
of other ERE to the four-body problem, readers may refer to [24, 4]

In this paper, we focus on the restricted 4-body problem with three primaries
forming Euller collinear configurations and the four bodies span ${\bf
R}^{2}$. Since the ERE of the three primaries is always linearly unstable
[25], it is reasonable to study the stability problem of the massless
particle. For example, the “massless body” can be imaged as a space station
and the three massive bodies form a planetary system.

We first apply the symplectic reduction method [20] to the general restricted
$N$-body problem. Let $P$ denote the inertial position of the massless body
zero, which moves in the gravitational field of the $N-1$ primaries, without
disturbing their motion. The corresponding Hamiltonian function for this $N$th
body is given by

$H(P,q,t)={1\over
2}|P|^{2}-\sum_{i=1}^{N-1}{m_{i}\over|r(t)R({\theta}(t))a_{i}-q|}.$ (1.8)

Here $z=z(t)$ is the Kepler elliptic orbit given through the true anomaly
${\theta}={\theta}(t)$by

$\displaystyle r({\theta}(t))=|z(t)|=\frac{p}{1+e\cos{\theta}(t)},$ (1.9)

where $p=a(1-e^{2})$ and $a>0$ is the latus rectum of the ellipse (1.9). By
Proposition 2.1, the ERE $(P(t),q(t))^{T}$ of the system (1.3) is in time $t$
where $q(t)=r(t)R(\theta(t))a_{N}$ and $P(t)=\dot{q}(t)$. It can be
transformed to the new solution
$(\bar{Z}(\theta),\bar{z}(\theta))^{T}=(0,\sigma,\sigma,0)^{T}$ in the true
anomaly $\theta$ as the new variable for the original Hamiltonian function $H$
given by (2.2). We then have the reduced Hamiltonian system is given in the
theorem.

######  Theorem 1.1.

The linearized Hamiltonian system of (2.2) at the ERE
$\zeta_{0}\equiv(\bar{Z}(\theta),\bar{z}(\theta))^{T}=(0,\sigma,\sigma,0)^{T}\in{\bf
R}^{4}$ depending on the true anomaly $\theta$ is given by

$\displaystyle\dot{\xi}(\theta)=JB(\theta)\xi(\theta),$ (1.10)

with

$\displaystyle
B(\theta)=H^{\prime\prime}(\theta,\bar{Z},\bar{z})|_{\bar{\xi}=\xi_{0}}=\left(\begin{array}[]{cccc|cccc}I_{2}&-J_{2}\\\
J_{2}&I_{2}-{1\over 1+e\cos{\theta}}D\end{array}\right),$ (1.13)

where

$\displaystyle
D=I_{2}-{1\over\mu}\left(\sum_{i=1}^{N-1}{m_{i}\over|a_{i}-a_{N}|^{3}}\right)I+{3\over\mu}\sum_{i=1}^{N-1}m_{i}\frac{(a_{i}-a_{N})(a_{i}-a_{N})^{T}}{|a_{i}-a_{N}|^{5}}.$
(1.14)

The corresponding quadratic Hamiltonian function is given by

$\displaystyle
H_{2}(\theta,\bar{Z},\bar{z})=\frac{1}{2}|\bar{Z}|^{2}+\bar{Z}\cdot
J\bar{z}+\frac{1}{2}\left(I_{2}-{D\over 1+e\cos{\theta}}\right)|\bar{z}|^{2}.$
(1.15)

We apply Theorem 3.8 to the restricted $4$-body problem. Denote the
eigenvalues of $D$ by ${\lambda}_{3}$ and ${\lambda}_{4}$. By Proposition 3.1,
we have that both ${\lambda}_{3}$ and ${\lambda}_{4}$ are positive and
${\lambda}_{3}+{\lambda}_{4}=3$. By this property, we show the relationship
between the Maslov index of $\xi_{{\alpha},e}$ given by (1.10) and the Maslov
index of ${\gamma}_{\beta,e}$ which is the elliptic Lagrangian solutions in
[3]. Since the elliptic Lagrangian solutions has already been well-studied in
[3], we have the Maslov index of $\xi_{{\alpha},e}$ in Proposition 3.3. By the
properties of the Maslov index, we define three curves ${\alpha}_{k}(e)$,
${\alpha}_{s}(e)$ and ${\alpha}_{m}(e)$ (cf. (3.20), (3.22) below) from left
to right in $({\alpha},e)\in[0,3]\times[0,1)$ according to the $\omega$-Maslov
indices of $\xi_{{\alpha},e}$. By the ${\omega}$-Maslov index theory [14], we
obtain the normal forms of $\xi_{a,e}(2\pi)$ and then the linear stability as
follows.

######  Theorem 1.2.

  1. (i)

We have $\xi_{\alpha,e}(2\pi)\approx R\left(\theta_{1}\right)\diamond
R\left(\theta_{2}\right)$ for some $\theta_{1}$ and $\theta_{2}\in(\pi,2\pi)$,
and thus it is strongly linearly stable on the segment
$\alpha_{m}(e)<\alpha<3$;

  2. (ii)

We have $\left.\xi_{\alpha,e}(2\pi)\approx D(\lambda)\diamond
R(\theta)\right)$ for some $0>\lambda\neq-1$ and $\theta\in(\pi,2\pi)$ and it
is elliptic-hyperbolic, and thus linearly unstable on the segment
$\alpha_{s}(e)<\alpha<\alpha_{m}(e)$.

  3. (iii)

We have $\xi_{\alpha,e}(2\pi)\approx R\left(\theta_{1}\right)\diamond
R\left(\theta_{2}\right)$ for some $\theta_{1}\in(0,\pi)$ and
$\theta_{2}\in(\pi,2\pi)$ with $2\pi-\theta_{2}<\theta_{1}$, and thus it is
strongly linearly stable on the segment
$\alpha_{k}(e)<\alpha<{\alpha}_{s}(e)$.

Note that it is also possible that ${\alpha}_{k}(e)={\alpha}_{s}(e)$, or
${\alpha}_{s}(e)={\alpha}_{m}(e)$ for some $e$ in $[0,1)$. For these cases,
the corresponding normal forms of $\xi_{{\alpha},e}$ are given in Theorem
3.10.

For the circular case, we first numerically relationships between ${\alpha}$
and $m_{1}$, $m_{2}$ and $m_{3}$. Since $m_{1}+m_{2}+m_{3}=1$, we can plot the
stable regions with respect to in $m_{1}$ and $m_{2}$ by definition of
${\alpha}$ in (3), (3.2) and Corollary 3.11. We show this results in (a) of
Figure 1.

|
---|---
(a) The linear stability of circular solution. | (b) The symmetric case.
Figure 1: In Figure (a), the shade regions in the $(m_{1},m_{3})$-plane show
the all the possible choices of
$(m_{1},m_{2},m_{3})=(m_{1},1-m_{1}-m_{3},m_{3})$ of the circular orbit such
that the system is linear stable. The Figure (b) shows the changes of $\alpha$
with respect to $m_{2}$ in the symmetric case

If we further assume that $m_{1}=m_{3}$. We have the following numerical
results holds shown in (b) of Figure 1.

######  Theorem 1.3.

If $e=0$ and $m_{1}=m_{3}$, $\xi_{{\alpha},e}$ is linear stable for
$m_{2}\in(0.854,1)$.

This paper is organized as follows. We first introduce the generalized
symplectic reduction method to the restricted $N$-body problem in Section 2.
We then use the ${\omega}$-Maslo index theory to obtain the linear stability
of the restricted $4$-body problem in Section 3. We consider the symmetric
case by assuming $m_{1}=m_{3}$ in Section 3. We apply the linear stability
results in Section 3 and obtain the condition on $m_{1}$, $m_{2}$ and $m_{3}$
such that the system is linear stable.

##  2 Reduction for the restricted $N$-body problem

The central configuration coordinates for a class of periodic solutions of the
$n$-body problem was introduced [20]. In this section, we generalize this
reducution to the restricted $N$-body problem. For the given masses
$m=(m_{1},m_{2},\ldots,m_{N-1})\in({\bf R}^{+})^{N-1}$ of $N-1$ primaries let
$a=(a_{1},a_{2},\ldots,a_{N-1})$ be an $(N-1)$-body central configuration of
$m$ satisfying (1.6). Using normalization
$2I(a)=\sum_{i=1}^{N-1}m_{i}|a_{i}|^{2}=1$ and assuming $\mu=U(a)=\sum_{1\leq
i<j\leq N-1}\frac{m_{i}m_{j}}{|a_{i}-a_{j}|}$, we have

$\displaystyle\sum_{j=1,j\neq
i}^{N-1}\frac{m_{j}(a_{j}-a_{i})}{|a_{j}-a_{i}|^{3}}=-\frac{U(a)}{2I(a)}a_{i}=-\mu
a_{i}.$ (2.1)

######  Proposition 2.1.

There exists a symplectic coordinate change
$\xi=(P,q)^{T}\;\mapsto\;\bar{\xi}=(\bar{Z},\bar{z})^{T},$ such that using the
true anomaly ${\theta}$ as the variable the resulting Hamiltonian function of
the $n$-body problem is given by

$\displaystyle H(\theta,\bar{Z},\bar{z})=\frac{1}{2}|\bar{Z}|^{2}+\bar{z}\cdot
J\bar{Z}+\frac{p-r}{2p}|\bar{z}|^{2}-\frac{r}{{\sigma}}\sum_{i=1}^{N-1}{m_{i}\over|{\sigma}a_{i}-\bar{z}|},$
(2.2)

where $J=(\begin{smallmatrix}0&-1\\\ 1&0\end{smallmatrix})$,
$r({\theta})=\frac{p}{1+e\cos{\theta}}$, $\mu$ defined by (2.1),
${\sigma}=(\mu p)^{1/4}$ and $p$ is given in (1.9).

###### Proof.

Inspired by Lemma 3.1 of [20], we carry the coordinate changes in four steps.

Step 1. Rotating coordinates via the matrix $R({\theta}(t))$ in time $t$.

We change first the coordinates $\xi$ to
$\hat{\xi}=(\hat{Z},\hat{z})^{T}\in({\bf R}^{2})^{2}$ which rotates with the
speed of the true anomaly. The transformation matrix is given by the rotation
matrix $R({\theta})=(\begin{smallmatrix}\cos{\theta}&-\sin{\theta}\\\
\sin{\theta}&\cos{\theta}\end{smallmatrix})$. The generating function of this
transformation is given by

$\hat{F}(t,P,\hat{z})=-P\cdot R({\theta})\hat{z},$ (2.3)

and for $1\leq i\leq n-1$ the transformation is given by

$q=-\frac{\partial\hat{F}}{\partial
P}=R({\theta})\hat{z},\quad\hat{Z}=-\frac{\partial\hat{F}}{\partial\hat{z}}=R({\theta})^{T}P,.$
(2.4)

Writing $\dot{R}({\theta}(t))=\frac{d}{dt}R({\theta}(t))$, and noting that
$R({\theta})^{T}=R({\theta})^{-1}$ and
$\dot{R}({\theta})=\dot{{\theta}}JR({\theta})$ we obtain the function

$\displaystyle\hat{F}_{t}\equiv\frac{\partial\hat{F}}{\partial
t}=-P\cdot\dot{R}({\theta})\hat{z}=-\hat{Z}\cdot
R({\theta})^{T}\dot{R}({\theta})\hat{z}=-\dot{{\theta}}\left(\hat{Z}\cdot
R({\theta})^{T}JR({\theta})\hat{z}\right)=\dot{{\theta}}(\hat{z}\cdot
J\hat{Z}).$ (2.5)

Because ${\theta}={\theta}(t)$ depends on $t$, by adding the function
$\frac{\partial\hat{F}}{\partial t}$ to the Hamiltonian function $H$ in (1.8),
as in Line 5 in p.272 of [20], we obtain the Hamiltonian function $\hat{H}$ in
the new coordinates:

$\hat{H}(t,\hat{Z},\hat{z})=H_{0}(P,q)+\hat{F}_{t}=\frac{1}{2}|\hat{Z}|^{2}+(\hat{z}\cdot
J\hat{Z})\dot{{\theta}}-\sum_{i=1}^{N-1}{m_{i}\over|r(t)a_{i}-\hat{z}|},$
(2.6)

where the variables of $H_{0}$ are functions of ${\theta}$, $\hat{Z}$,
$\hat{z}$, given by (2.4).

Step 2. Dilating coordinates via the polar radius $r=|z(t)|$.

We change the coordinates $\hat{\xi}$ to $\tilde{\xi}=(\tilde{Z},\tilde{z})$
which dilate with $r=|z(t)|$ given by (1.9). The position coordinates are
transformed by $\hat{z}=r\tilde{z}$. It is natural to scale the momenta by
$1/r$ to get $\hat{Z}=\tilde{Z}/r$. But it turns out that the new
transformation

$\hat{Z}=\frac{1}{r}\tilde{Z}+\dot{r}\tilde{z}$ (2.7)

makes the resulting Hamiltonian function simpler. This transformation is
generated by the function

$\tilde{F}(t,\tilde{Z},\hat{z})=\frac{1}{r}\tilde{Z}\cdot\hat{z}+\frac{\dot{r}}{2r}|\hat{z}|^{2},$
(2.8)

and is given by

$\tilde{z}=\frac{\partial\tilde{F}}{\partial\tilde{Z}}=\frac{1}{r}\hat{z},\hat{Z}=\frac{\partial\tilde{F}}{\partial\hat{z}}=\frac{1}{r}\tilde{Z}+\frac{\dot{r}}{r}\hat{z}=\frac{1}{r}\tilde{Z}+\dot{r}\tilde{z},$

with

$\frac{\partial\tilde{F}}{\partial
t}=-\frac{\dot{r}}{r^{2}}\tilde{Z}\cdot\hat{z}+\frac{\ddot{r}r-\dot{r}^{2}}{2r^{2}}|\hat{z}|^{2}=-\frac{\dot{r}}{r}\tilde{Z}\cdot\tilde{z}+\frac{\ddot{r}r-\dot{r}^{2}}{2}|\tilde{z}|^{2},$
(2.9)

by (2.7).

In this case, as in the last two lines on p.272 of [20], the Hamiltonian
function $\hat{H}$ in (2.6) becomes the new Hamiltonian function $\tilde{H}$
in the new coordinates:

$\displaystyle\tilde{H}(t,\tilde{Z},\tilde{z})$ $\displaystyle\equiv$
$\displaystyle\hat{H}(t,\hat{Z},\hat{z})+\tilde{F}_{t}$ (2.10)
$\displaystyle=$
$\displaystyle\frac{1}{2r^{2}}|\tilde{Z}|^{2}+\frac{\dot{r}}{r}\tilde{Z}\cdot\tilde{z}+\frac{\dot{r}^{2}}{2}(|\tilde{z}|^{2})+(\tilde{z}\cdot
J\tilde{Z})\dot{{\theta}}-\sum_{i=1}^{N-1}{m_{i}\over|r(t)a_{i}-r(t)\tilde{z}|}+\tilde{F}_{t}$
$\displaystyle=$
$\displaystyle\frac{1}{2r^{2}}|\tilde{Z}|^{2}+\frac{r\ddot{r}}{2}|\tilde{z}|^{2}+(\tilde{z}\cdot
J\tilde{Z})\dot{{\theta}}-\frac{1}{r}\sum_{i=1}^{N-1}{m_{i}\over|a_{i}-\tilde{z}|}.$

Step 3. Coordinates via the true anomaly ${\theta}$ as the independent
variable.

Here we use the true anomaly ${\theta}\in[0,2\pi]$ as an independent variable
instead of $t\in[0,T]$ to simplify the study. This is achieved by dividing the
Hamiltonian function $\tilde{H}$ in (2.10) by $\dot{{\theta}}$. Assuming
$\dot{{\theta}}(t)>0$ for all $t\in[0,T]$ and $\tilde{\xi}\in W^{1,2}({\bf
R}/(T{\bf Z}),{\bf R}^{8})$ we consider the action functional corresponding to
the Hamiltonian system:

$\displaystyle f(\tilde{\xi})$ $\displaystyle=$
$\displaystyle\int_{0}^{T}(\frac{1}{2}\dot{\tilde{\xi}}(t)\cdot
J\tilde{\xi}(t)-\tilde{H}(t,\tilde{\xi}(t)))dt$ $\displaystyle=$
$\displaystyle\int_{0}^{2\pi}\left(\frac{1}{2}\frac{\dot{\tilde{\xi}}(t({\theta}))}{\dot{{\theta}}(t)}\cdot
J\tilde{\xi}(t)-\frac{\tilde{H}(t,\tilde{\xi}(t({\theta})))}{\dot{{\theta}}(t)}\right)d{\theta}$
$\displaystyle=$
$\displaystyle\int_{0}^{2\pi}\left(\frac{1}{2}\tilde{\xi}^{\prime}({\theta})\cdot
J\tilde{\xi}({\theta})-\tilde{H}({\theta},\tilde{\xi}({\theta}))\right)d{\theta}.$

Here we used $\tilde{\xi}^{\prime}({\theta})$ to denote the derivative of
$\tilde{\xi}({\theta})$ with respect to ${\theta}$. But in the following we
shall still write $\dot{\tilde{\xi}}({\theta})$ for the derivative with
respect to ${\theta}$ instead of $\tilde{\xi}^{\prime}({\theta})$ for
notational simplicity.

Note that the elliptic Kepler orbit (1.9) satisfies
$r(t)^{2}\dot{{\theta}}(t)=\sqrt{\mu p}=\sqrt{\mu a(1-e^{2})}={\sigma}^{2}$
with ${\sigma}=(\mu p)^{1/4}.$ Note that $a=\mu^{1/3}(T/2\pi)^{2/3}$ with $T$
being the minimal period of the orbit (1.9), we have

${\sigma}=(\mu
a(1-e^{2}))^{1/4}=\mu^{1/3}(\frac{T}{2\pi})^{1/6}(1-e^{2})^{1/4}\;\in\;(0,\mu^{1/3}(\frac{T}{2\pi})^{1/6}]$

depending on $e$, when the mass $\mu$ and the period $T$ are fixed. Note that
similarly we have $p={\sigma}^{4}/\mu$ depends on $e$ too. Note that the
function $r$ satisfies

$\ddot{r}=\frac{\mu
p}{r^{3}}-\frac{\mu}{r^{2}}=\mu\left(\frac{p}{r^{3}}-\frac{1}{r^{2}}\right).$

Therefore we get the Hamiltonian function $\tilde{H}$ in the new coordinates:

$\displaystyle\tilde{H}({\theta},\tilde{Z},\tilde{z})$ $\displaystyle\equiv$
$\displaystyle\frac{1}{\dot{{\theta}}}\tilde{H}(t,\tilde{Z},\tilde{z})$ (2.11)
$\displaystyle=$
$\displaystyle\frac{1}{2r^{2}(t)\dot{{\theta}}(t)}|\tilde{Z}|^{2}+\frac{r(t)\ddot{r}(t)}{2\dot{{\theta}}(t)}|\tilde{z}|^{2}+\tilde{z}\cdot
J\tilde{Z}-\frac{1}{r(t)\dot{{\theta}}(t)}\sum_{i=1}^{N-1}{m_{i}\over|a_{i}-\tilde{z}|}$
$\displaystyle=$
$\displaystyle\frac{1}{2{\sigma}^{2}}|\tilde{Z}|^{2}+\tilde{z}\cdot
J\tilde{Z}+\frac{\mu(p-r({\theta}))}{2{\sigma}^{2}}|\tilde{z}|^{2}-\frac{r({\theta})}{{\sigma}^{2}}\sum_{i=1}^{N-1}{m_{i}\over|a_{i}-\tilde{z}|},$

where $r({\theta})=p/(1+e\cos{\theta})$. Note that now the minimal period $T$
of the elliptic solution $\tilde{z}=\tilde{z}({\theta})$ becomes $2\pi$ in the
new coordinates in terms of true anomaly ${\theta}$ as an independent
variable.

Step 4. Coordinates via the dilation of ${\sigma}=(p\mu)^{1/4}$.

The last transformation is the dilation
$(\tilde{Z},\tilde{z})\;=\;({\sigma}\bar{Z},{\sigma}^{-1}\bar{z})$. This
transformation is symplectic and independent of the true anomaly ${\theta}$.
Thus the Hamiltonian function $\tilde{H}$ in (2.11) becomes a new Hamiltonian
function:

$H({\theta},\bar{Z},\bar{z})\equiv\tilde{H}({\theta},{\sigma}\bar{Z},{\sigma}^{-1}\bar{z})=\frac{1}{2}|\bar{Z}|^{2}+\bar{z}\cdot
J\bar{Z}+\frac{p-r}{2p}|\bar{z}|^{2}-\frac{r}{{\sigma}}\sum_{i=1}^{N-1}{m_{i}\over|{\sigma}a_{i}-\bar{z}|}.$
(2.12)

The proof is complete. ∎

Suppose that $(P(t),q(t))^{T}$ is the solution of system (1.3) with
$q(t)=r(t)R(\theta(t))a_{N}$, and $P(t)=\dot{q}(t).$ By Proposition 2.1, it is
transformed to the new solution $(\bar{Z}(\theta),\bar{z}(\theta))^{T}$ in the
true anomaly $\theta$ as the new variable for the original Hamiltonian
function $H$ of (1.8), which is given by

$\bar{Z}(\theta)=(0,\sigma)^{T}\qquad\bar{z}(\theta)=(\sigma,0)^{T}$ (2.13)

Therefore, we can prove the Theorem 1.1 directly.

###### Proof of Theorem 1.1.

In this proof we omit all the upper bars on the variables of $H$ in (2.2). By
(2.2), we have

$\displaystyle H_{z}=JZ+\frac{p-r}{p}z-\frac{r}{\sigma}{\partial\over\partial
z}\left(\sum_{i=1}^{N-1}{m_{i}\over|{\sigma}a_{i}-\bar{z}|}\right)|_{z={\sigma}a_{N}},\quad
H_{Z}=Z-Jz,$ (2.14)

and

$\displaystyle H_{ZZ}=I,\quad H_{Zz}=-J,\quad H_{zz}=J,\quad
H_{zz}=\frac{p-r}{p}I-\frac{r}{\sigma}{\partial^{2}\over\partial
z^{2}}\left(\sum_{i=1}^{N-1}{m_{i}\over|{\sigma}a_{i}-\bar{z}|}\right)|_{z={\sigma}a_{N}},$
(2.15)

where all the items above are $2\times 2$ matrices, and we denote by $H_{x}$
and $H_{xy}$ the derivative of $H$ with respect to $x$, and the second
derivative of $H$ with respect to $x$ and then $y$ respectively for $x$ and
$y\in{\bf R}$.

Now evaluating the corresponding functions at the special solution
$({0,{\sigma}},{{\sigma},0})^{T}\in{\bf R}^{4}$ of (2.13) with
$z=({\sigma},0)^{T}$, and summing them up, we obtain

$\displaystyle\frac{\partial^{2}}{\partial
z^{2}}\left(\sum_{i=1}^{N-1}{m_{i}\over|{\sigma}a_{i}-\bar{z}|}\right)\Bigg{|}_{\xi_{0}}$
$\displaystyle=$ $\displaystyle-\left(-\sum_{i=1}^{N-1}{m_{i}\over|\sigma
a_{i}-z|^{3}}\right)I+3\sum_{i=1}^{N-1}m_{i}\frac{({\sigma}a_{i}-z)({\sigma}a_{i}-z)^{T}}{|\sigma
a_{i}-z|^{5}}$ (2.16) $\displaystyle=$
$\displaystyle-{1\over{\sigma}^{3}}\left(\sum_{i=1}^{N-1}{m_{i}\over|a_{i}-a_{N}|^{3}}\right)I+{3\over{\sigma}^{3}}\sum_{i=1}^{N-1}m_{i}\frac{(a_{i}-a_{N})(a_{i}-a_{N})^{T}}{|a_{i}-a_{N}|^{5}}.$

Hence, we have

$\displaystyle H_{zz}$ $\displaystyle=$ $\displaystyle I_{2}-{r\over
p}I_{2}-{r\over
p}{{\sigma}^{3}\over\mu}\left[-{1\over{\sigma}^{3}}\left(\sum_{i=1}^{N-1}{m_{i}\over|a_{i}-a_{N}|^{3}}\right)I+{3\over{\sigma}^{3}}\sum_{i=1}^{N-1}m_{i}\frac{(a_{i}-a_{N+1})(a_{i}-a_{N})^{T}}{|a_{i}-a_{N}|^{5}}\right]$
(2.17) $\displaystyle=$ $\displaystyle I_{2}-{r\over
p}\left[I_{2}-{1\over\mu}\left(\sum_{i=1}^{N-1}{m_{i}\over|a_{i}-a_{N}|^{3}}\right)I+{3\over\mu}\sum_{i=1}^{N-1}m_{i}\frac{(a_{i}-a_{N})(a_{i}-a_{N})^{T}}{|a_{i}-a_{N}|^{5}}\right],$

where the last equality of the first formula follows from the definition of
$\mu$ in (2.1). Thus the proof is complete. ∎

##  3 The linear stability restricted $4$-body problem

We now consider the linear stability of special relative equilibrium in the
four body problem with one small mass away form the line of the three
primaries which form an Euler central configuration. A typical example is the
ERE orbit of the restricted $4$-bodies, the Sun, the Earth, the Moon and one
space station.

######  Proposition 3.1.

Suppose that $a_{i}$ for $1\leq i\leq 3$ forms an Euler collinear
configuration and $a_{i}$ for $1\leq i\leq 4$ span ${\bf R}^{2}$. The
eigenvalues ${\lambda}_{3}$ and ${\lambda}_{4}$ of $D$ defined by (1.14) are
both positive and satisfy

$\displaystyle{\lambda}_{3}+{\lambda}_{4}=3.$ (3.1)

###### Proof.

By (1.14) and the direct computations, we have that

$\displaystyle\lambda_{3}+\lambda_{4}=tr(D)=2+{1\over\mu}\sum_{i=1}^{3}{m_{i}\over|a_{i}-a_{4}|^{3}}=3+\beta_{2,0},$
(3.2)

where $\beta_{2,0}={1\over\mu}\sum_{i=1}^{3}{m_{i}\over|a_{i}-a_{4}|^{3}}-1.$
Moreover, we have

$\displaystyle D-{3+\beta_{2,0}\over 2}I_{2}$ $\displaystyle=-{3\over
2\mu}\sum_{i=1}^{3}{m_{i}\over|a_{i}-a_{4}|^{3}}I_{2}+{3\over\mu}\sum_{i=1}^{3}{m_{i}(a_{i}-a_{4})(a_{i}-a_{4})^{T}\over|a_{i}-a_{4}|^{5}}$
(3.3) $\displaystyle={3\over
2\mu}\left[\sum_{i=1}^{3}m_{i}{2(a_{i}-a_{4})(a_{i}-a_{4})^{T}-|a_{i}-a_{4}|^{2}\over|a_{i}-a_{4}|^{5}}\right]$
(3.4) $\displaystyle=\Psi(\beta_{22,0}),$ (3.5)

where
$\beta_{22,0}=\sum_{i=1}^{3}m_{i}{(z_{a_{i}}-z_{a_{4}})^{2}\over|a_{i}-a_{4}|^{5}}$
and $\Psi(z)=(\begin{smallmatrix}x&y\\\ -y&x\end{smallmatrix})$ with
$z=x+\sqrt{-1}y$. Note that $\beta_{2,0}$ and $\beta_{22,0}$ are coincide with
those of (2.10) and (A.14) in [13]. Therefore, following the discussion in
[13],we have

$\displaystyle\lambda_{3},\lambda_{4}={3+\beta_{22,0}\over
2}\pm|\beta_{22,0}|.$ (3.6)

Now suppose $a_{1}$, $a_{2}$and $a_{3}$ form an Euler collinear configuration
where they are locate on the $x$-axis in ${\bf R}^{2}$. We set
$a_{i}=(a_{ix},a_{iy})$ for $i=1,2,3,4$, and hence $a_{iy}=0$ when $i=1,2,3$.
Then the central configuration equation (2.1) of $m_{4}$ gives
$\sum_{j=1}^{3}{m_{j}(a_{j}-a_{4})\over|a_{j}-a_{4}|^{3}}=-\mu a_{4}.$
Especially, for $a_{4y}$, we have that

$\sum_{j=1}^{3}{m_{j}a_{4y}\over|a_{j}-a_{4}|^{3}}=\mu a_{4y}.$ (3.7)

Since $a_{4y}\neq 0$, it follows that

$\mu=\sum_{j=1}^{3}{m_{j}\over|a_{j}-a_{4}|^{3}},$ (3.8)

and hence $\beta_{2,0}=0$. By (3.6), we have $\lambda_{3}+\lambda_{4}=3$ and

$\displaystyle
D={3\over\mu}\sum_{j=1}^{3}{m_{j}(a_{j}-a_{4})(a_{j}-a_{4})^{T}\over|a_{j}-a_{4}|^{5}}={3\over\mu}\begin{pmatrix}\sum_{i=1}^{3}{m_{i}x_{4i}^{2}\over
r_{4i}^{5}}&\sum_{i=1}^{3}{m_{i}x_{4i}y_{4i}\over r_{4i}^{5}}\\\
\sum_{i=1}^{3}{m_{i}x_{4i}y_{4i}\over
r_{4i}^{5}}&\sum_{i=1}^{3}{m_{i}y_{4i}^{2}\over r_{4i}^{5}}\end{pmatrix},$
(3.9)

where $(x_{4i},y_{4i})=a_{4}-a_{i}$ and $r_{4i}=|a_{4}-a_{i}|$. Using Cauchy’s
inequality, we have

$\left(\sum_{i=1}^{3}{m_{i}x_{4i}y_{4i}\over
r_{4i}^{5}}\right)^{2}=\left(\sum_{i=1}^{3}\sqrt{m_{i}x_{4i}^{2}\over
r_{4i}^{5}}\sqrt{m_{i}y_{4i}^{2}\over
r_{4i}^{5}}\right)^{2}\leq\left(\sum_{i=1}^{3}{m_{i}x_{4i}^{2}\over
r_{4i}^{5}}\right)\left(\sum_{i=1}^{3}{m_{i}y_{4i}^{2}\over
r_{4i}^{5}}\right).$ (3.10)

Then we have $\det(D)\geq 0$ and hence $\lambda_{3},\lambda_{4}\geq 0$. ∎

Without loss of the generality, we assume that $\lambda_{3}\geq\lambda_{4}$ in
the following. We define the operator ${\cal A}({\alpha},e)$ by

$\displaystyle{\cal A}({\alpha},e)$
$\displaystyle=-\frac{{\mathrm{d}}^{2}}{{\mathrm{d}}t^{2}}I_{2}-I_{2}+\frac{1}{1+e\cos
t}R(t)D_{\alpha,e}R(t)^{T}$
$\displaystyle=-\frac{{\mathrm{d}}^{2}}{{\mathrm{d}}t^{2}}I_{2}-I_{2}+\frac{1}{2(1+e\cos
t)}((\lambda_{3}+\lambda_{4})I_{2}+(\lambda_{3}-\lambda_{4})S(t))$ (3.11)
$\displaystyle=-\frac{{\mathrm{d}}^{2}}{{\mathrm{d}}t^{2}}I_{2}-I_{2}+\frac{1}{2(1+e\cos
t)}(3I_{2}+\alpha S(t)),$ (3.12)

where $R(t)=(\begin{smallmatrix}\cos t&-\sin t\\\ \sin t&\cos
t\end{smallmatrix})$, $D_{{\beta},e}=(\begin{smallmatrix}\lambda_{3}&0\\\
0&\lambda_{4}\end{smallmatrix})$, $S(t)=(\begin{smallmatrix}\cos 2t&\sin 2t\\\
\sin 2t&-\cos 2t\end{smallmatrix})$ and
$\alpha:=\lambda_{3}-\lambda_{4}\in[0,3]$. We denote the Morse indices and
nullity of ${\cal A}({\alpha},e)$ on the domain $\overline{D({\omega},2\pi)}$
by $\phi_{{\omega}}({\cal A}({\alpha},e))$ and $\nu_{{\omega}}({\cal
A}({\alpha},e))$ respectively. We also denote the ${\omega}$-Maslov index of
$\xi_{{\alpha},e}$ by $i_{{\omega}}(\xi_{{\alpha},e})$ and the nullity by
$\nu_{{\omega}}(\xi_{{\alpha},e})$. By Lemma A.1 and the transformation
introduced by Section 2.4 of [3], we have for any ${\alpha}\in[0,3]$,
$e\in[0,1)$ and ${\omega}\in{\bf U}$,

$\displaystyle\phi_{{\omega}}({\cal
A}({\alpha},e))=i_{{\omega}}(\xi_{{\alpha},e}),\quad\nu_{{\omega}}({\cal
A}({\alpha},e))=\nu_{{\omega}}(\xi_{{\alpha},e}).$ (3.13)

According to (2.17) and (2.18) of [3], the essential part
$\gamma=\gamma_{\beta,e}(t)$ of the fundamental solution of the Lagrangian
orbit satisfies

$\displaystyle\begin{cases}&\dot{\gamma}(t)=JB(t)\gamma(t),\\\
&\gamma(0)=I_{4},\end{cases}$ (3.14)

with $B(t)=\left(\begin{smallmatrix}I&-J\\\ J&I_{2}-\frac{K_{\beta}}{1+e\cos
t}\end{smallmatrix}\right),$ where $e$ is the eccentricity and $K_{\beta}={\rm
diag}\\{\frac{3+\sqrt{9-\beta}}{2},\frac{3-\sqrt{9-\beta}}{2}\\}$. For
$(\beta,e)\in[0,9)\times[0,1)$, the second order differential operator
corresponding to (3.14) is given by

$\displaystyle
A(\beta,e)=-\frac{\mathrm{d}^{2}}{\mathrm{~{}d}t^{2}}I_{2}-I_{2}+\frac{1}{2(1+e\cos
t)}\left(3I_{2}+\sqrt{9-\beta}S(t)\right),$ (3.15)

where $S(t)=\left(\begin{smallmatrix}\cos 2t&\sin 2t\\\ \sin 2t&-\cos
2t\end{smallmatrix}\right)$, defined on the domain
$\overline{D({\omega},2\pi)}$ in (A.6). Then it is self-adjoint and depends on
the parameters $\beta$ and $e$. Let $\beta:=9-\alpha^{2}$ in (3.15), we have
that

$\displaystyle A(9-\alpha^{2},e)={\cal A}(\alpha,e).$ (3.16)

It follows that the Maslov index of
$i_{\omega}(\xi_{\alpha,e})=i_{\omega}(\gamma_{9-\alpha^{2},e})$ by letting
$\beta=9-\alpha^{2}$ in (3.14). By [3], the Maslov index of the
$\xi_{\alpha,e}$ is given as follows.

######  Proposition 3.2.

  1. (i)

For all $(\alpha,e)\in[0,3]\times[0,1)$, the Maslov index satisfies
$i_{1}(\xi_{\alpha,e})=0;$

  2. (ii)

for $e\in[0,1)$,

$\displaystyle i_{\omega}\left(\xi_{3,e}\right)=\begin{cases}2,\mbox{ if
}{\omega}\in{\bf U}\setminus\\{0\\},\\\ 0,\mbox{ if
}{\omega}=1;\end{cases}\quad
v_{\omega}\left(\xi_{3,e}\right)=\begin{cases}3,&\text{ if }\omega=1,\\\
0,&\text{ if }\omega\in\mathbf{U}\backslash\\{1\\};\end{cases}$ (3.17)

  3. (iii)

for $\omega\in\mathbf{U}$ and $e\in[0,1)$, we have
$i_{\omega}\left(\xi_{0,e}\right)=0,$ and
$v_{\omega}\left(\xi_{0,e}\right)=0.$

###### Proof.

By Theorem 1 of [3] and (3.16), we have (i) of this proposition holds. By (C)
of Section 3 of [3], we have (ii) of this proposition holds. By Section 3.2 of
[3], we have (iii) of this proposition holds. ∎

To study the monotonic of the operator ${\cal A}({\alpha},e)$, we define the
operator $\bar{{\cal A}}$ as the following.

$\displaystyle\bar{{\cal A}}({\alpha},e)=\frac{{\cal
A}(0,e)}{{\alpha}}+\frac{S(t)}{2(1+e\cos t)}.$ (3.18)

It follows that ${\cal A}({\alpha},e)={\alpha}\bar{{\cal A}}({\alpha},e).$
Then we have $\phi_{\omega}({\cal A}({\alpha},e))=\phi_{\omega}(\bar{{\cal
A}}({\alpha},e))$ and $\nu_{\omega}({\cal
A}({\alpha},e))=\nu_{\omega}(\bar{{\cal A}}({\alpha},e))$. Following from
(iii) of Proposition 3.3 and Lemma A.1, ${\cal A}(0,e)$ is positive definite
for any $\omega$ boundary condition. Using similar arguments of Lemma 4.4 and
Corollary 4.5 of [3],we get the following lemma holds. We omit the proof here.

######  Proposition 3.3 (cf. Lemma 4.4 and Corollary 4.5 of [3]).

  1. (i)

For each fixed $e\in[0,1)$, the operator $\bar{{\cal A}}({\alpha},e)$ is non-
increasing with respect to ${\alpha}\in(0,3]$ for any fixed
$\omega\in\mathbf{U}$. Especially
$\left.\frac{\partial}{\partial{\alpha}}\bar{{\cal
A}}({\alpha},e)\right|_{{\alpha}={\alpha}_{0}}=-\frac{1}{{\alpha}^{2}}{\cal
A}(0,e),$ for ${\alpha}\in(0,3]$ is a negative definite operator.

  2. (ii)

For every eigenvalue $\lambda_{{\alpha}_{0}}=0$ of
$\bar{A}\left({\alpha}_{0},e_{0}\right)$ with $\omega\in\mathbf{U}$ for some
$\left({\alpha}_{0},e_{0}\right)\in$ $(0,3]\times[0,1)$, there holds

$\displaystyle\left.\frac{\mathrm{d}}{\mathrm{d}{\alpha}}\lambda_{{\alpha}}\right|_{{\alpha}={\alpha}_{0}}>0.$
(3.19)

  3. (iii)

For every fixed $e\in[0,1)$ and $\omega\in\mathbf{U}$, the index function
$i_{\omega}\left(\xi_{{\alpha},e}\right)$, is non-decreasing in
$\alpha\in[0,3]$. Especially, if $\omega\in\mathbf{U}\backslash\\{1\\}$,
$i_{\omega}\left(\xi_{{\alpha},e}\right)$ increases from $0$ to $2$.

By Proposition 3.3, the $-1$-index $i_{-1}\left(\xi_{\alpha,e}\right)$ changes
monotonically increases from $0$ to $2$ as ${\alpha}$ increases from $0$ to
$3$. Then there exists three distinct curves $\Gamma_{k}$, $\Gamma_{s}$, and
$\Gamma_{m}$ locating from left to right in the parameter $(\alpha,e)$
rectangle $[0,3]\times[0,1)$. The curves $\Gamma_{s}$ and $\Gamma_{m}$ are the
$-1$-degenerate curves. Since the $i_{{\omega}}(\xi_{{\alpha},e})$ is non-
decreasing, the hyperbolic region is connected. We then can use the curve
$\Gamma_{k}$ to denote the boundary of the hyperbolic region.

More specifically, for every $e\in[0,1)$, the $-1$ index
$i_{-1}\left(\xi_{\alpha,e}\right)$ is non-increasing, and strictly decreasing
only on two values of $\alpha=\alpha_{1}(e)$ and
$\alpha=\alpha_{2}(e)\in(0,3)$. Define

$\displaystyle\alpha_{s}(e)=\min\left\\{\alpha_{1}(e),\alpha_{2}(e)\right\\}\text{
and
}\alpha_{m}(e)=\max\left\\{\alpha_{1}(e),\alpha_{2}(e)\right\\},\quad\text{
for }e\in[0,1),$ (3.20)

and

$\displaystyle\Gamma_{s}=\left\\{\left(\alpha_{s}(e),e\right)\mid
e\in[0,1)\right\\},\text{ and
}\Gamma_{m}=\left\\{\left(\alpha_{m}(e),e\right),\mid e\in[0,1)\right\\}.$
(3.21)

For every $e\in[0,1)$, we define

$\displaystyle\alpha_{k}(e)=\inf\left\\{\alpha^{\prime}\in[0,3]\mid\sigma\left(\xi_{\alpha,e}(2\pi)\right)\cap\mathbf{U}\neq\emptyset,\quad\forall\alpha\in\left[0,\alpha^{\prime}\right]\right\\},$
(3.22)

and

$\displaystyle\Gamma_{k}=\left\\{\left(\alpha_{k}(e),e\right)\in[0,3]\times[0,1)\mid
e\in[0,1)\right\\}.$ (3.23)

Therefore, $\Gamma_{s},\Gamma_{m}$ and $\Gamma_{k}$ form three curves.

######  Lemma 3.4.

(i) For given $e_{0}$, if $({\alpha}_{1},e_{0})$ and $({\alpha}_{1},e_{0})$
satisfy that $0<{\alpha}_{2}\leq{\alpha}_{1}<3$ and
$\xi_{{\alpha}_{1},e_{0}}(2\pi)$ is hyperbolic, then
$\xi_{{\alpha}_{2},e_{0}}(2\pi)$ is also hyperbolic. Consequently, the
hyperbolic region of $\xi_{{\alpha},e}$ is connected.

(ii) For $({\alpha},e)\in\Gamma_{h}$, every matrix $\xi_{{\alpha},e}(2\pi)$ is
hyperbolic. Thus $\Gamma_{k}$ is the boundary surface of this hyperbolic
region.

(iii) For any $e\in[0,1)$, the total multiplicity of ${\omega}$ degeneracy of
$\xi_{{\alpha},e}(2\pi)$ for ${\alpha}\in[0,3]$ satisfies that
$\sum_{0\leq{\alpha}\leq 3}\nu_{{\omega}}(\xi_{{\alpha},e}(2\pi))=2,$ for all
$\omega\in\mathbf{U}\backslash\\{1\\}.$

###### Proof.

(i) By Lemma A.1, for any fixed ${\omega}$ and $e$, if
$\xi_{{\alpha}_{1},e}(2\pi)$ is hyperbolic, then ${\cal A}({\alpha}_{1},e)$ is
positive definite on $\overline{D}({\omega},2\pi)$ for any given
${\omega}\in{\bf U}$. By (3.18), $\bar{{\cal A}}({\alpha}_{1},e)$ is positive
definite. By (3.19), we have $\bar{{\cal A}}({\alpha}_{1},e)<\bar{{\cal
A}}({\alpha}_{2},e)$. It follows that $\bar{{\cal A}}({\alpha}_{2},e)$ is
positive definite and non-degenerate for any ${\omega}\in{\bf U}$. Therefore
$\xi_{{\alpha}_{2},e}(2\pi)$ must be hyperbolic for all
${\alpha}\in[0,{\alpha}_{2})$.

Note that when ${\alpha}=0$ the matrix $\xi_{{\alpha},e}$ is hyperbolic by
(iii) of Proposition 3.3. Therefore, the hyperbolic region of
$\xi_{{\alpha},e}$ is connected.

(ii) By the definition of ${\alpha}_{k}(e)$, there exists a
sequence$\\{{\alpha}_{i}\\}_{i\in{\bf N}}$ satisfying
${\alpha}_{i}<{\alpha}_{k}(e)$, ${\alpha}_{i}\to{\alpha}_{k}(e)$, and
$\xi_{{\alpha}_{i},e}(2\pi)$ is hyperbolic. Therefore $\xi_{{\alpha},e}(2\pi)$
is hyperbolic for every ${\alpha}\in[0,{\alpha}_{k}(e))$ by (i). Then (3.22)
holds and ${\alpha}_{k}(e)$ is the envelope surface of this hyperbolic region.

(iii) Note that the operator $\bar{{\cal A}}({\alpha},e)$ on
$\overline{D}({\omega},2\pi)$ for $0$ and $3$ are both non-degenerate if
${\omega}\in{\bf U}\setminus\\{1\\}$. By (ii) and (iii) of Proposition 3.3 and
(3.19), there exist at most two ${\alpha}_{1}$ and ${\alpha}_{2}$ such that at
each of which the ${\omega}$-index decreases by 1 if
${\alpha}_{1}\neq{\alpha}_{2}$, or the ${\omega}$-index decreases by 2 if
${\alpha}_{1}={\alpha}_{2}$. Suppose that the two values are given by
${\alpha}_{1}={\alpha}_{1}(e)$ and ${\alpha}_{2}={\alpha}_{2}(e)$ such that
for ${\epsilon}>0$ small enough, we have that
$\phi_{{\omega}}(\bar{A}(0,e))=\phi_{{\omega}}(\bar{A}({\alpha}_{1}-{\epsilon},e))$,
$\phi_{{\omega}}(\bar{{\cal A}}({\alpha}_{1},e))=\phi_{{\omega}}(\bar{{\cal
A}}({\alpha}_{2}-{\epsilon},e))$ and $\phi_{{\omega}}(\bar{{\cal
A}}({\alpha}_{2},e))=\phi_{{\omega}}(\bar{{\cal A}}(3,e))$. Then we have that

$\displaystyle 2$ $\displaystyle=\phi_{{\omega}}(\bar{{\cal
A}}(3,e))-\phi_{{\omega}}(\bar{{\cal A}}(0,e))$
$\displaystyle=\phi_{{\omega}}(\bar{{\cal
A}}({\alpha}_{2},e))-\phi_{{\omega}}(\bar{{\cal
A}}({\alpha}_{2}-{\epsilon},e))+\phi_{{\omega}}(\bar{{\cal
A}}({\alpha}_{1},e))-\phi_{{\omega}}(\bar{{\cal
A}}({\alpha}_{1}-{\epsilon},e))$ $\displaystyle=\dim\ker(\bar{{\cal
A}}({\alpha}_{2},e))+\dim\ker(\bar{{\cal A}}({\alpha}_{1},e))$
$\displaystyle=\nu_{{\omega}}(\xi_{{\alpha}_{2},e}(2\pi))+\nu_{{\omega}}(\xi_{{\alpha}_{1},e}(2\pi))$
$\displaystyle=\sum_{{\alpha}\in[0,3]}\nu_{{\omega}}(\xi_{{\alpha},e}(2\pi)).$
(3.24)

Then we have that (iii) holds. ∎

Following similar arguments as Corollary 9.2 of [3], we have the following
corollary holds.

######  Corollary 3.5 (cf. Corollary 9.2 of [3]).

For every $e\in[0,1)$, we have
$\sum_{{\alpha}\in[0,{\alpha}_{m}(e))}\nu_{-1}(\xi_{{\alpha},e}(2\pi))=0,$ and
$\sum_{{\alpha}\in[{\alpha}_{m}(e),3)}\nu_{-1}(\xi_{{\alpha},e}(2\pi))=2.$

######  Proposition 3.6.

The function ${\alpha}_{k}(e)$ is continuous in $e$.

###### Proof.

We prove this proposition by contradiction. Suppose that ${\alpha}_{k}(e)$ is
not continuous in $e$. There must exist some $\hat{e}$ and a sequence
$\\{e_{i}\\}_{i=1}^{\infty}\subset[0,1)\setminus\\{\hat{e}\\}$ and
${\alpha}_{0}\in[0,3]$ such that
${\alpha}_{k}(e_{i})\to{\alpha}_{0}\neq{\alpha}_{k}(\hat{e})$, if
$e_{i}\to\hat{e}$ as $i\to\infty$. We discuss the two cases of the continuity
according to the sign of ${\alpha}_{0}-{\alpha}_{k}(\hat{e})$. By the
continuity of the eigenvalues of the matrix $\xi_{{\alpha},e}(2\pi)$ and by
(3.22), ${\sigma}(\xi_{{{\alpha}}_{0},\hat{e}}(2\pi))\cap{\bf U}\neq\emptyset$
holds.

By the definition of ${\alpha}_{k}(\hat{e})$ and (i) of Lemma 3.4, we must
have ${\alpha}_{k}(\hat{e})<{\alpha}_{0}$.

Now we suppose ${\alpha}_{k}(\hat{e})<{\alpha}_{0}$. By the continuity of
${\alpha}_{s}(e)$ and the definition of ${\alpha}_{0}$,

$\displaystyle{\alpha}_{k}(\hat{e})<{\alpha}_{0}\leq{\alpha}_{s}(\hat{e}).$
(3.25)

By the definition of ${\alpha}_{k}(\hat{e})$, let
${\omega}_{0}\in{\sigma}(\xi_{{\alpha}_{k}(\hat{e}),\hat{e}}(2\pi))\cap{\bf
U}$. Let
$L=\\{(\hat{{\alpha}},\hat{e})|{\alpha}\in[0,{\alpha}_{k}(\hat{e}))\\}$,
$V=\\{(0,e)|e\in[0,1)\\}$, and
$L_{i}=\\{({\alpha},e_{i})|{\alpha}\in[0,{\alpha}_{k}(e_{i})]\\}$.

$\displaystyle
i_{\omega_{0}}\left(\xi_{{\alpha},e}\right)=\nu_{\omega_{0}}\left(\xi_{{\alpha},e}\right)=0,\quad\forall({\alpha},e)\in
L\cup V\cup\bigcup_{i\geq 1}L_{i}.$ (3.26)

In particular, we have
$i_{\omega_{0}}\left(\xi_{{\alpha}_{k}(\hat{e}),\hat{e}}\right)=0$ and
$\nu_{\omega_{0}}\left(\xi_{{\alpha}_{k}(\hat{e}),\hat{e}}\right)\geq 1$.
Therefore, by the definition of ${\omega}_{0}$, there exists
$\hat{\alpha}\in({\alpha}_{k}(\hat{e}),{\alpha}_{0})$ sufficiently close to
${\alpha}_{k}(\hat{e})$ such that

$\displaystyle
i_{\omega_{0}}\left(\xi_{\hat{{\alpha}},\hat{e}}\right)=i_{\omega_{0}}\left(\xi_{{\alpha}_{k}(\hat{e}),\hat{e}}\right)+\nu_{\omega_{0}}\left(\xi_{{\alpha}_{k}(\hat{e}),\hat{e}}(2\pi)\right)\geq
1.$ (3.27)

Note that (3.27) holds for all ${\alpha}\in({\alpha}_{k}(e),{\alpha}_{0}]$.
Also $({\alpha},\hat{e})$ is an accumulation point of $\cup_{i\geq 1}L_{i}$.
This yields there exists $({\alpha}_{i},e_{i})\in L_{i}$ such that
$\xi_{{\alpha}_{i},e_{i}}$ is ${\omega}_{0}$-degenerate. Moreover
${\alpha}_{i}\to{\alpha}_{0}$ and
$\xi_{{\alpha}_{i},e_{i}}(2\pi)\to\xi_{{\alpha}_{0},\hat{e}}(2\pi)$ as
$i\to\infty$. Then we have following contradiction for $i\geq 1$ large enough

$\displaystyle 1\leq i_{\omega_{0}}(\xi_{{\alpha}_{0},\hat{e}})\leq
i_{\omega_{0}}(\xi_{{\alpha}_{i}(e),e_{i}})=0.$ (3.28)

Then we have the continuity of ${\alpha}_{k}(e)$ in ${\alpha}$ and $e$. ∎

######  Proposition 3.7.

The region $\Gamma_{s},\Gamma_{m}$ and $\Gamma_{k}$ form three curves which
possess the following properties.

  1. (i)

We have

$\displaystyle i_{-1}\left(\xi_{\alpha,e}\right)=\begin{dcases}0,\text{ if
}0\leq\alpha\leq\alpha_{s}(e),\\\ 1,\text{ if
}\alpha_{s}(e)<\alpha\leq\alpha_{m}(e),\\\ 2,\text{ if
}\alpha_{m}(e)<\alpha\leq 3,\end{dcases}$ (3.29)

and both $\Gamma_{s}$ and $\Gamma_{m}$ are precisely the $-1$ degeneracy
curves of the matrix $\xi_{\alpha,e}(2\pi)$ in the $(\alpha,e)$ rectangle
$[0,3]\times[0,1)$.

  2. (ii)

There holds $\alpha_{k}(e)\leq\alpha_{s}(e)\leq\alpha_{m}(e)<3$ for all
$e\in[0,1)$.

  3. (iii)

Every matrix $\xi_{\alpha,e}(2\pi)$ is hyperbolic when
$\alpha\in\left(0,\alpha_{k}(e)\right]$ and $e\in[0,1)$, and there holds

$\displaystyle\alpha_{k}(e)=\sup\left\\{\alpha\in[0,3]\mid\sigma\left(\xi_{\alpha,e}(2\pi)\right)\cap\mathbf{U}=\emptyset\right\\},\quad\forall
e\in[0,1).$

Consequently $\Gamma_{k}$ is the boundary curve of the hyperbolic region of
$\xi_{\alpha,e}(2\pi)$ in the $(\alpha,e)$ rectangle $[0,3]\times[0,1)$.

###### Proof.

For $0\leq e<1$, by the definitions of ${\alpha}_{s}({\alpha},e)$ and
${\alpha}_{m}(e)$ satisfying for ${\alpha}>0$ and $e\in[0,1)$ in (3.21), we
have that $-1$-index stays the same and only changes when
$({\alpha},e)\in\Gamma_{m}\cup\Gamma_{s}$.

Note that if ${\alpha}=0$,
$\sigma\left(\xi_{{\alpha},e}(2\pi)\right)\cap\mathbf{U}=\emptyset$. Therefore
for any $e\in[0,1)$, ${\alpha}_{k}(e)$ is well defined. By the definition of
(3.21) and (3.22), we have that for given $e_{0}\in[0,1)$, if
${\alpha}_{s}(e_{0})$ exists, we must have
${\alpha}_{k}(e_{0})\leq{\alpha}_{s}(e_{0})\leq{\alpha}_{m}(e_{0})$. By Lemma
3.4, (ii) and (iii) of this propositon can be obtianed directly. Then we have
this proposition holds. ∎

######  Theorem 3.8.

  1. (i)

We have $\xi_{\alpha,e}(2\pi)\approx R\left(\theta_{1}\right)\diamond
R\left(\theta_{2}\right)$ for some $\theta_{1}$ and $\theta_{2}\in(\pi,2\pi)$,
and thus it is strongly linearly stable on the segment
$\alpha_{m}(e)<\alpha<3$;

  2. (ii)

We have $\left.\xi_{\alpha,e}(2\pi)\approx D(\lambda)\diamond
R(\theta)\right)$ for some $0>\lambda\neq-1$ and $\theta\in(\pi,2\pi)$ and it
is elliptic-hyperbolic, and thus linearly unstable on the segment
$\alpha_{s}(e)<\alpha<\alpha_{m}(e)$.

  3. (iii)

We have $\xi_{\alpha,e}(2\pi)\approx R\left(\theta_{1}\right)\diamond
R\left(\theta_{2}\right)$ for some $\theta_{1}\in(0,\pi)$ and
$\theta_{2}\in(\pi,2\pi)$ with $2\pi-\theta_{2}<\theta_{1}$, and thus it is
strongly linearly stable on the segment
$\alpha_{k}(e)<\alpha<{\alpha}_{s}(e)$.

###### Proof.

(i) By Lemma A.4, the index and nullity of $2$nd-iteration of the symplectic
path $\xi_{{\alpha},e}(t)$ satisfies

$\displaystyle i_{1}(\xi_{{\alpha},e}^{2})$
$\displaystyle=i_{-1}\left(\xi_{{\alpha},e}\right)+i_{1}(\xi_{{\alpha},e}),$
$\displaystyle\nu_{1}\left(\xi_{{\alpha},e}^{2}\right)$
$\displaystyle=\nu_{1}\left(\xi_{{\alpha},e}\right)+\nu_{-1}\left(\xi_{{\alpha},e}\right)=\nu_{-1}\left(\xi_{{\alpha},e}\right),$
(3.30)

where $\nu_{1}(\xi_{{\alpha},e})=0$ if $({\alpha},e)\in[0,3]\times[0,1)$.
Therefore, the 2-iteration of the index is given by

$\displaystyle
i_{1}(\xi^{2}_{{\alpha},e})=\left\\{\begin{array}[]{ll}{2,}&{\text{ if
}({\alpha},e)\in\Gamma_{m}};\\\ {1,}&{\text{ if
}({\alpha},e)\in\Gamma_{s}};\\\ {0,}&{\text{ if
}({\alpha},e)\in\Gamma_{k}}.\end{array}\right.$ (3.34)

Follow the discussion in the proof of Theorem 1.2 of [5], if $({\alpha},e)$
satisfies ${\alpha}\neq{\alpha}_{m}(e)$ and ${\alpha}\neq{\alpha}_{s}(e)$, the
matrix $\xi_{{\alpha},e}(4\pi)=\xi_{{\alpha},e}^{2}(2\pi)$ is non-degenerate
with respect with eigenvalue $1$.

For (i), suppose
$\omega_{i}=e^{\sqrt{-1\theta_{i}}}\in\sigma\left(\xi_{{\alpha},e}(2\pi)\cap\mathbf{U}\right)$
with ${\theta}_{i}\in(0,\pi)$. Note that $i_{-1}(\xi_{{\alpha},e})=2$, and by
(A.9),

$\displaystyle i_{-1}(\xi_{{\alpha},e})$
$\displaystyle=i_{1}(\xi_{{\alpha},e})+\sum_{\omega_{i}}(S_{\xi_{{\alpha},e}(2\pi)}^{+}(\omega_{i})-S_{\xi_{{\alpha},e}(2\pi)}^{-}(\omega_{i}))+S_{\xi_{{\alpha},e}(2\pi)}^{+}(-1)$
$\displaystyle=\sum_{\omega_{i}}(S_{\xi_{{\alpha},e}(2\pi)}^{+}(\omega_{i})-S_{\xi_{{\alpha},e}(2\pi)}^{-}(\omega_{i})).$
(3.35)

It yields that

$\displaystyle
2=\sum_{\omega_{i}}\left(S_{\xi_{{\alpha},e}(2\pi)}^{+}(\omega_{i})-S_{\xi_{{\alpha},e}(2\pi)}^{-}(\omega_{i})\right)\leq\sum_{\omega_{i}}\left(S_{\xi_{{\alpha},e}(2\pi)}^{+}(\omega_{i})+S_{\xi_{{\alpha},e}(2\pi)}^{-}(\omega_{i})\right)\leq
2.$ (3.36)

Then we have that $S_{\xi_{{\alpha},e}}^{-}(\omega_{i})=0$, by the list of
splitting number in Section A.1. Therefore, there exist the two ${\omega}_{1}$
and ${\omega}_{2}$ such that $S_{\xi_{{\alpha},e}}^{+}(\omega_{i})=1$. Then we
have (i) of Theorem 3.8 holds.

(ii) Note that $i_{1}(\xi^{2}_{{\alpha},e})=1$ implies that
$i_{-1}(\xi_{{\alpha},e})=1$. Therefore, still by (3.35), there exists exact
one eigenvalue
$\omega=e^{\sqrt{-1\theta_{i}}}\in\sigma\left(\xi_{{\alpha},e}(2\pi)\cap\mathbf{U}\right)$
for ${\theta}\in(0,\pi)$ with the splitting number $(1,0)$. By the splitting
number the list of splitting number in Section A.1, we must have
$\xi_{{\alpha},e}(2\pi)\approx D({\lambda})\diamond R({\theta})$. Also note
that $i_{1}(\xi_{{\alpha},e})=0$ implies ${\lambda}<0$. Then we have (ii) of
Theorem 3.8 holds.

(iii) Suppose that $({\alpha}_{0},e_{0})\in\Gamma_{k}$. By (i) and (ii) of
Lemma 3.4, the matrix $M\equiv\xi_{{\alpha}_{0},e_{0}}(2\pi)$ is not
hyperbolic with at least one pair of on the unit circle ${\bf U}$.
Furthermore, by the definition of ${\alpha}_{k}(e_{0})$ and
${\alpha}_{s}(e_{0})$, $\pm 1\notin{\sigma}(\xi_{{\alpha}_{0},e_{0}}(2\pi))$.
Suppose that
$\sigma(\xi_{{\alpha}_{0},e_{0}}(2\pi))=\\{\lambda_{1},\lambda_{1}^{-1},\lambda_{2},\lambda_{2}^{-1}\\}$,
with $\lambda_{1}\in\mathbf{U}\backslash\mathbf{R}$ and
$\lambda_{2}\in(\mathbf{U}\cup\mathbf{R})\backslash\\{\pm 1,0\\}$. We must
have $\lambda_{2}\in\mathbf{U}\setminus\mathbf{R}$ and
$\xi_{{\alpha}_{0},e_{0}}(2\pi)\approx R({\theta}_{1})\diamond
R({\theta}_{2})$ for ${\theta}_{1}$, ${\theta}_{2}\in(0,\pi)\cup(\pi,2\pi)$.

If not, $\lambda_{2}\in{\bf R}\setminus\\{\pm 1,0\\}$. The normal form is
given by $\xi_{{\alpha}_{0},e_{0}}(2\pi)\approx D({\lambda})\diamond
R({\theta})$ for some ${\theta}\in(0,\pi)\cup(\pi,2\pi)$. Then by (A.9), we
obtain following contradiction.

$\displaystyle 0$ $\displaystyle=i_{-1}\left(\xi_{{\alpha}_{0},e_{0}}\right)$
$\displaystyle=i_{1}\left(\xi_{{\alpha}_{0},e_{0}}\right)+S_{M}^{+}(1)-S_{M}^{-}\left(e^{\pm\sqrt{-1}\theta}\right)+S_{M}^{+}\left(e^{\pm\sqrt{-1}\theta}\right)-S_{M}^{-}(-1)$
$\displaystyle=0+0-S_{R(\theta)}^{-}\left(e^{\pm\sqrt{-1}\theta}\right)+S_{R(\theta)}^{+}\left(e^{\pm\sqrt{-1}\theta}\right)-0$
$\displaystyle=\pm 1.$ (3.37)

Again as (3.37), we have that

$\displaystyle 0$ $\displaystyle=i_{-1}\left(\xi_{{\alpha}_{0},e_{0}}\right)$
$\displaystyle=i_{1}\left(\xi_{{\alpha}_{0},e_{0}}\right)+S_{M}^{+}(1)-S_{R\left(\theta_{1}\right)}^{-}\left(e^{\pm\sqrt{-1}\theta_{1}}\right)+S_{R\left(\theta_{1}\right)}^{+}\left(e^{\pm\sqrt{-1}\theta_{1}}\right)$
(3.38) $\displaystyle\quad-
S_{R\left(\theta_{2}\right)}^{-}\left(e^{\pm\sqrt{-1}\theta_{2}}\right)+S_{R\left(\theta_{1}\right)}^{+}\left(e^{\pm\sqrt{-1}\theta_{2}}\right)-S_{M}^{-}(-1)$
$\displaystyle=-S_{R\left(\theta_{1}\right)}^{-}\left(e^{\pm\sqrt{-1}\theta_{1}}\right)+S_{R\left(\theta_{1}\right)}^{+}\left(e^{\pm\sqrt{-1}\theta_{1}}\right)-S_{R\left(\theta_{2}\right)}^{-}\left(e^{\pm\sqrt{-1}\theta_{2}}\right)+S_{R\left(\theta_{2}\right)}^{+}\left(e^{\pm\sqrt{-1}\theta_{2}}\right).$
(3.39)

Note that if ${\theta}_{1}$ and ${\theta}_{2}$ locate at the same interval
$(0,\pi)$ or $(\pi,2\pi)$, the right hand side will be $\pm 2$. Thus, we must
have that ${\theta}_{1}\in(0,\pi)$ and ${\theta}_{2}\in(\pi,2\pi)$.

If ${\theta}_{1}=2\pi-{\theta}_{2}$, following equation holds.

$\displaystyle\sum_{{\alpha}_{k}<{\alpha}\leq
3}\nu_{\omega}(\xi_{{\alpha}_{0},e_{0}})\geq\sum_{{\alpha}_{s}\leq{\beta}\leq
3}\nu_{\omega}(\xi_{{\alpha}_{0},e_{0}})+\nu_{\omega}(\xi_{{\alpha}_{0},e_{0}})\geq
1+2,$ (3.40)

where ${\omega}=\exp(\sqrt{-1}{\theta}_{1})$. This contradicts to (iii) of
Lemma 3.4.

If $2\pi-{\theta}_{2}>{\theta}_{1}$, for
${\omega}=\exp(\sqrt{-1}{\theta}_{1})$, we have that

$\displaystyle 0\leq
i_{\omega}(\xi_{{\alpha}_{0},e_{0}})=i_{1}(\xi_{{\alpha}_{0},e_{0}})+S_{M}^{+}(1)-S_{R(\theta_{1})}^{-}(e^{\sqrt{-1}\theta_{1}})=-S_{R(\theta_{1})}^{-}(e^{\sqrt{-1}\theta_{1}})=-1,$
(3.41)

where $M=\xi_{{\alpha}_{0},e_{0}}(2\pi)$. This contradiction yields that
$2\pi-{\theta}_{2}<{\theta}_{1}$. Then we have (ii) of Theorem 3.8 holds. ∎

######  Lemma 3.9.

For some $({\alpha}_{0},e)\in[0,3]\times[0,1)$, if
$\xi_{{\alpha}_{0},e}(2\pi)\approx M_{2}(-1,c)$ which is given by (A.2) for
$c_{1}$, $c_{2}\in{\bf R}$, or it possesses the basic normal form
$N_{1}(-1,a)\diamond N_{1}(-1,b)$ for $a$, $b\in{\bf R}$, then
$\xi_{{\alpha},e}(2\pi)$ is hyperbolic for all ${\alpha}\in[0,{\alpha}_{0})$.

###### Proof.

Note that the basic normal form of the matrix $M_{2}(-1,c)$ is either
$N_{1}(-1,a)\diamond N_{1}(-1,b)$ or $N_{1}(-1,a)\diamond D({\lambda})$ for
some $a$, $b\in{\bf R}$ and $0>{\lambda}\neq-1$. Thus for any ${\omega}\in{\bf
U}\setminus\\{1\\}$, by (A.9), we obtain

$\displaystyle 0\leq
i_{\omega}(\xi_{{\alpha}_{0},e})=i_{1}(\xi_{{\alpha}_{0},e})+S_{M}^{+}(1)-S_{M}^{-}(\omega)=-S_{M}^{-}(\omega)\leq
0,$ (3.42)

where $M=\xi_{{\alpha}_{0},e}(2\pi)$. Then we have that
$i_{\omega}(\xi_{{\alpha}_{0},e})=0$ for all ${\omega}\in{\bf U}$. Note that
$\phi_{{\omega}}(\bar{\cal A}({\alpha},e))=i_{\omega}(\xi_{{\alpha}_{0},e})$
and $\nu_{{\omega}}(\bar{\cal
A}({\alpha},e))=\nu_{\omega}(\xi_{{\alpha}_{0},e})$. Now from
$\phi_{{\omega}}({\cal A}({\alpha},e))=0$ and the monotonic of eigenvalues of
$\bar{\cal A}({\alpha},e)$ in Lemma 3.4, we have that $\bar{\cal
A}({\alpha},e)>0$ for all ${\alpha}\in[0,{\alpha}_{0})$ on $D({\omega},2\pi)$.
Therefore, $\nu_{\omega}(\xi_{{\alpha},e})=\nu_{{\omega}}(\bar{\cal
A}({\alpha},e))=0$ holds for all ${\alpha}\in[0,{\alpha}_{0})$. Then this
lemma holds. ∎

######  Theorem 3.10.

When $({\alpha},e)\in\Gamma_{s}$, $\Gamma_{m}$ or $\Gamma_{k}$, the normal
form and the linear stability of $\xi_{{\alpha},e}(2\pi)$ satisfy followings.

  1. (i)

If ${\alpha}_{s}(e)<{\alpha}_{m}(e)$, we have
$\xi_{{\alpha}_{m}(e),e}(2\pi)\approx N_{1}(-1,1)\diamond R({\theta})$ for
some ${\theta}\in(\pi,2\pi)$. Thus it is spectrally stable and linearly
unstable.

  2. (ii)

If ${\alpha}_{k}(e)<{\alpha}_{s}(e)={\alpha}_{m}(e)$, we have
$\xi_{{\alpha}_{s}(e),e}(2\pi)\approx-I_{2}\diamond R({\theta})$ for some
${\theta}\in(\pi,2\pi)$. Thus it is linearly stable, but not strongly linearly
stable.

  3. (iii)

If ${\alpha}_{k}(e)<{\alpha}_{s}(e)<{\alpha}_{m}(e)$, we have
$\xi_{{\alpha}_{s}(e),e}(2\pi)\approx N_{1}(-1,-1)\diamond R({\theta})$ for
some ${\theta}\in(\pi,2\pi)$. Thus it is spectrally stable and linearly
unstable.

  4. (iv)

If ${\alpha}_{k}(e)<{\alpha}_{s}(e)\leq{\alpha}_{m}(e)$, we have
$\xi_{{\alpha}_{k}(e),e}(2\pi)\approx N_{2}(e^{\sqrt{-1}{\theta}},b)$ for some
${\theta}\in(0,\pi)$ and $b=\begin{pmatrix}b_{1}&b_{2}\\\
b_{3}&b_{4}\end{pmatrix}$ satisfying $(b_{2}-b_{3})\sin{\theta}>0$, that is,
$N_{2}(e^{\sqrt{-1}{\theta}},b)$ is trivial in the sense of Definition 1.8.11
in p.41 of [14]. Consequently the matrix $\xi_{{\alpha}_{k}(e),e}(2\pi)$ is
spectrally stable and linearly unstable.

  5. (v)

If ${\alpha}_{k}(e)={\alpha}_{s}(e)<{\alpha}_{m}(e)$, we have either
$\xi_{{\alpha}_{k}(e),e}(2\pi)\approx N_{1}(-1,1)\diamond D({\lambda})$ for
some $-1\neq{\lambda}<0$ and is linearly unstable; or
$\xi_{{\alpha}_{k}(e),e}(2\pi)\approx M_{2}(-1,c)$ with $c_{1}$, $c_{2}\in{\bf
R}$ and $c_{2}\neq 0$. Thus it is spectrally stable and linearly unstable.

  6. (vi)

If ${\alpha}_{k}(e)={\alpha}_{s}(e)={\alpha}_{m}(e)$, either
$\xi_{{\alpha}_{k}(e),e}(2\pi)\approx M_{2}(-1,c)$ with $c_{1}$, $c_{2}\in{\bf
R}$ with $c_{2}\neq 0$, or $\xi_{{\alpha}_{k}(e),e}\approx N_{1}(-1,1)\diamond
N_{1}(-1,1)$. Thus $\xi_{{\alpha}_{k}(e),e}(2\pi)$ is spectrally stable and
linearly unstable.

###### Proof.

(i) Let $e\in[0,1)$ satisfy ${\alpha}_{s}(e)<{\alpha}_{m}(e)$. Then Corollary
3.5 implies $\nu_{-1}(\xi_{{\alpha}_{m}(e),e}(2\pi))=1$. As the limit case of
(i) and (ii) of Theorem 3.8, we have the eigenvalues of matrix
$\xi_{{\alpha}_{m}(e),e}(2\pi)$ are all on ${\bf U}$ and the normal form is
either $M\approx M_{2}(-1,c)$ for some $c_{2}\neq 0$ or $M\approx
N_{1}(-1,1)\diamond R({\theta})$ for some ${\theta}\in(\pi,2\pi)$.

By Lemma 3.9 and ${\alpha}_{s}(e)<{\alpha}_{m}(e)$, we have that $M\approx
M_{2}(-1,c)$ for some $c_{2}\neq 0$ cannot holds. The normal form is given by
$M\approx N_{1}(-1,1)\diamond R({\theta})$. So $M$ is spectrally stable and
linearly unstable.

(ii) Let $e\in[0,1)$ satisfy
${\alpha}_{k}(e)<{\alpha}_{s}(e)={\alpha}_{m}(e)$. As the limit case of (i)
and (ii) of the Theorem 3.8 and Lemma 3.9, the normal form of the matrix
$M\equiv\xi_{{\alpha}_{s}(e),e}(2\pi)$ is either $M\approx N_{1}(-1,a)\diamond
N_{1}(-1,b)$ for some $a$, $b\in\\{-1,1\\}$, or $M\approx-I_{2}\diamond
R({\theta})$ for some ${\theta}\in(\pi,2\pi)$. However, the case of $M\approx
N_{1}(-1,a)\diamond N_{1}(-1,b)$ is impossible by Lemma 3.9. Then we have that
$M\approx-I_{2}\diamond R({\theta})$ for some ${\theta}\in(\pi,2\pi)$ and it
is linear stable but not strongly linear stable.

(iii) Let $e\in[0,1)$ satisfy
${\alpha}_{k}(e)<{\alpha}_{s}(e)<{\alpha}_{m}(e)$. As the limiting case of
Cases (ii) and (iii) of Theorem 3.8, the normal form of the matrix
$M\equiv\xi_{{\alpha}_{s}(e),e}(2\pi)$ must satisfy either $M\approx
N_{1}(-1,-1)\diamond R({\theta})$ for some ${\theta}\in(\pi,2\pi)$ or
$M\approx M_{2}(-1,c)$ for some $c_{2}\neq 0$. Here the second case is also
impossible by Lemma 3.9, and the conclusion of (iii) follows.

(iv) Let $e\in[0,1)$ satisfy
${\alpha}_{k}(e)<{\alpha}_{s}(e)\leq{\alpha}_{m}(e)$. As the limiting case of
the cases (iii) of Theorem 3.8, the matrix $\xi_{{\alpha}_{k}(e),e}(2\pi)$
must have Krein collision eigenvalues
${\sigma}(M)=\\{{\lambda}_{1},\bar{{\lambda}}_{1},{\lambda}_{2},\bar{{\lambda}}_{2}\\}$
with ${\lambda}_{1}=\bar{{\lambda}}_{2}=e^{\sqrt{-1}{\theta}}$ for some
${\theta}\in(0,\pi)\cup(\pi,2\pi)$. By Theorem 3.3 and the definition of
${\alpha}_{k}(e)$, $\pm 1$ cannot be the eigenvalue of
$\xi_{{\alpha}_{k}(e),e}(2\pi)$. Therefore, we must have that $M\approx
N_{2}({\omega},b)$ for ${\omega}=e^{\sqrt{-1}{\theta}}$ and some matrix
$b=(\begin{smallmatrix}b_{1}&b_{2}\\\ b_{3}&b_{4}\end{smallmatrix})$ which has
the form of (25-27) by Theorem 1.6.11 in p. 34 of [14]. Because
$(I_{2}\diamond-I_{2})^{-1}N_{2}(e^{\sqrt{-1}{\theta}},b)(I_{2}\diamond-
I_{2})=N_{2}(e^{\sqrt{-1}(2\pi-{\theta})},\hat{b})$ where
$\hat{b}=(\begin{smallmatrix}b_{1}&-b_{2}\\\ -b_{3}&b_{4}\end{smallmatrix})$.
We can always suppose ${\theta}\in(0,\pi)$ without changing the fact $M\approx
N_{2}({\omega},b)$.

Note that by (3.22) and Lemma 3.4, we have
$i_{{\omega}}(\xi_{{\alpha}_{k},e})=0$. Suppose $b_{2}-b_{3}=0$, by Lemma
1.9.2 in p. 43 of [15], $\nu_{{\omega}}(N_{2}({\omega},b))=2$ and then
$N_{2}({\omega},b)$ has basic normal form $R({\theta})\diamond
R(2\pi-{\theta})$ by the study in case 4 in p. 40 of [14]. Thus we have
following contradiction.

$\displaystyle
0=i_{\omega}\left(\xi_{{\alpha}_{k}(e),e}\right)=i_{1}\left(\xi_{{\alpha}_{k}(e),e}\right)+S_{M}^{+}(1)-S_{R(\theta)}^{-}(\omega)-S_{R(\theta)}^{-}(\bar{\omega})\leq-1.$
(3.43)

Therefore $b_{2}-b_{3}\neq 0$ must hold. Then we obtain

$\displaystyle
0=i_{\omega}(\xi_{{\alpha},{\beta}_{k}(e),e})=i_{1}(\xi_{{\alpha},{\beta}_{k}(e),e})+S_{M}^{+}(1)-S_{N_{2}(\omega,b)}^{-}(\omega)=-S_{N_{2}(\omega,b)}^{-}(\omega).$
(3.44)

By $\langle 6\rangle$ and $\langle 7\rangle$ in the list of splitting number
in Section A.1, we obtain that $N_{2}({\omega},b)$ must be trivial. Then by
Theorem 1 of [27], the matrix $M$ is spectrally stable and is linearly
unstable as claimed.

(v) Let $e\in[0,1)$ satisfy ${\alpha}_{k}(e)={\alpha}_{s}(e)<{\alpha}_{m}(e)$.
Note that $-1$ must be an eigenvalue of
$M\approx\xi_{{\alpha}_{k}(e),e}(2\pi)$ with the geometric multiplicity 1 by
Proposition 3.3. As the limit case of (ii) and (iii) of Theorem 3.8, the
matrix $M$ must satisfy either $M\approx M_{2}(-1,b)$ with $b_{1}$,
$b_{2}\in{\bf R}$ and $b_{2}\neq 0$, and thus is spectrally stable and
linearly unstable; or $M\approx N_{-1}(-1,a)\diamond D({\lambda})$ for some
$a\in\\{-1,1\\}$ and $-1\neq{\lambda}<0$. Then in the later case we obtain

$\displaystyle
0=i_{-1}(\xi_{{\alpha}_{k}(e),e})=i_{1}(\xi_{{\alpha}_{k}(e),e})+S_{M}^{+}(1)-S_{N_{1}(-1,a)}^{-}(-1)=-S_{N_{1}(-1,a)}^{-}(-1).$
(3.45)

Then by $\langle 3\rangle$ and $\langle 4\rangle$ in the list of splitting
number in Section A.1, we must have $a=1$. Thus
$M=\xi_{{\alpha}_{k}(e),e}(2\pi)$ is hyperbolic (elliptic-hyperbolic) and
linearly unstable. Note that by the above argument, the matrix $M_{2}(-1,b)$
also has the basic normal form $N_{1}(-1,1)\diamond D({\lambda})$ for some
$-1\neq{\lambda}<0$.

(vi) Let $e\in[0,1)$ satisfy
${\alpha}_{k}(e)={\alpha}_{s}(e)={\alpha}_{m}(e)$. As the limiting case of
cases (i), (ii) and (iii) of Theorem 3.8, $-1$ must be the only eigenvalue of
$M=\xi_{{\alpha}_{k}(e),e}$ with $\nu_{-1}(M)=2$ by Proposition 3.3. Thus the
matrix $M$ must satisfy $M\approx M_{1}(-1,c)$ with $c=0$ and
$v_{-1}(M_{2}(-1,c))=2$; or $M\approx N_{1}(-1,\hat{a})\diamond
N_{1}(-1,\hat{b})$ for some $\hat{a}$ and $\hat{b}\in\\{-1,1\\}$. In both case
$M$ possesses the basic normal form $N_{1}(-1,\hat{a})\diamond
N_{1}(-1,\hat{b})$ for some $\hat{a}$ and $\hat{b}\in\\{-1,1\\}$. Thus we
obtain

$\displaystyle 0$ $\displaystyle=i_{-1}\left(\xi_{{\alpha}_{k}(e),e}\right)$
$\displaystyle=i_{1}\left(\xi_{{\alpha}_{k}({\alpha},e),e}\right)+S_{M}^{+}(1)-S_{N_{1}(-1,a)}^{-}(-1)-S_{N_{1}(-1,b)}^{-}(-1)$
$\displaystyle=-S_{N_{1}(-1,a)}^{-}(-1)-S_{N_{1}(-1,b)}^{-}(-1).$ (3.46)

Then by $\langle 3\rangle$ and $\langle 4\rangle$ in the list of splitting
number in Section A.1, we must have $a=b=1$ similar to our above study for
(v). Therefore it is spectrally stable and linearly unstable as claimed. ∎

For the circular case, by $e=0$, we obtain the value of ${\alpha}_{s}$ and
${\alpha}_{m}$ and directly by Section 3.3 of [3] and
${\beta}=9-{\alpha}^{2}$.

######  Corollary 3.11.

For the circular orbit, namely $e=0$, we have that
${\alpha}_{s}={\alpha}_{m}=\frac{\sqrt{33}}{2}$ and ${\alpha}=2\sqrt{2}$.

  1. (i)

If $0\leq{\alpha}<2\sqrt{2}$, we have that
$\sigma(\xi_{{\alpha},e})\subset{\bf C}\setminus(U\cup{\bf R})$, and thus it
is linearly unstable.

  2. (ii)

If ${\alpha}=2\sqrt{2}$, we have that $\xi_{{\alpha},e}\approx
N_{2}(\omega_{0},b)$ with ${\omega}=e^{\sqrt{-1}\sqrt{2}\pi}$ and
$b=(\begin{smallmatrix}b_{1}&b_{2}\\\ b_{3}&b_{4}\end{smallmatrix})$ with
$b_{2}\neq b_{3}$ and thus it is linearly unstable.

  3. (iii)

If $2\sqrt{2}<{\alpha}<\frac{\sqrt{33}}{2}$, we have that
$\xi_{{\alpha},e}\approx R\left(\theta_{1}\right)\diamond
R\left(\theta_{2}\right)$ for some $\theta_{1}\in(0,\pi)$ and
$\theta_{2}\in(\pi,2\pi)$, and thus it is linearly stable.

  4. (iv)

If ${\alpha}=\frac{\sqrt{33}}{2}$, we have that $\xi_{{\alpha},e}\approx-
I_{2}\diamond R\left(\sqrt{3}\pi\right)$ and thus it is linearly stable.

  5. (v)

If $\frac{\sqrt{33}}{2}<{\alpha}\leq 3$, we have that $\xi_{{\alpha},e}\approx
R\left(\theta_{1}\right)\diamond R\left(\theta_{2}\right)$ for some
$\theta_{1}$ and $\theta_{2}\in(\pi,2\pi]$, and thus it is linearly stable.

##  4 The linear stability of the symmetric case $m_{1}=m_{3}$

In this section, we consider the symmetric case. Because of the relationship
between the linear stability of the restrict 4-body problem and the linear
stability of the Lagrangian solutions. It is well-known that the linear
stability of an elliptic Euler solution of the $3$-body problem with masses
$m=(m_{1},m_{2},m_{3})\in({\bf R}^{+})^{3}$ is determined by the eccentricity
$e\in[0,1)$ and the mass parameter

$\displaystyle{\beta}=\frac{m_{1}(3x^{2}+3x+1)+m_{3}x^{2}(x^{2}+3x+3)}{x^{2}+m_{2}[(x+1)^{2}(x^{2}+1)-x^{2}]},$
(4.1)

where $x$ is the unique positive solution of the Euler quintic polynomial
equation

$(m_{3}+m_{2})x^{5}+(3m_{3}+2m_{2})x^{4}+(3m_{3}+m_{2})x^{3}-(3m_{1}+m_{2})x^{2}-(3m_{1}+2m_{2})x-(m_{1}+m_{2})=0,$
(4.2)

and the three bodies form a central configuration of $m$, which are denoted by
$q_{1}=0$, $q_{2}=(x\alpha,0)^{T}$ and $q_{3}=((1+x)\alpha,0)^{T}$ with
$\alpha=|q_{2}-q_{3}|>0$, $x\alpha=|q_{1}-q_{2}|$ [25].

If further assuming that $m_{1}=m_{3}$, we have that $x=1$ is the unique
positive root of (4.2). Since the center of mass of the three primaries is the
origin, we have the position $a_{i}$ can be written in $m_{2}$ as follows.

$\displaystyle a_{1}=(-(1-m_{2})^{-{1\over 2}},0)^{T},\quad
a_{2}=(0,0)^{T},\quad a_{3}=((1-m_{2})^{-{1\over 2}},0)^{T}.$ (4.3)

Since the center of mass is the origin, we suppose
$a_{4}=(0,y(1-m_{2})^{-{1\over 2}})^{T}$ for some $y\in{\bf R}^{+}$. By (3.8),
we have

${1-m_{2}\over(y^{2}+1)^{3\over 2}}+{m_{2}\over y^{3}}={1+7m_{2}\over 8}.$
(4.4)

Note that when $m_{2}=0$, (4.4) gives $y=\sqrt{3}$; when $m_{2}=1$, (4.4)
gives $y=1$. If $0<y<1$, we have

${1-m_{2}\over(y^{2}+1)^{3\over 2}}+{m_{2}\over
y^{3}}\geq{1-m_{2}\over\sqrt{8}}+m_{2}\geq{1-m_{2}\over
8}+m_{2}={1+7m_{2}\over 8},$ (4.5)

where the equality holds if and only if $m_{2}=1$. Hence we must have $y\geq
1$. Moreover, we take the derivative of $y$ and obtain that

${{\mathrm{d}}y\over{\mathrm{d}}m_{2}}=-\frac{{7\over
8}+{1\over(y^{2}+1)^{3\over 2}}-{1\over
y^{3}}}{3y\Big{[}{1-m_{2}\over(y^{2}+1)^{5\over 2}}+{m_{2}\over
y^{5}}\Big{]}}<0,$ (4.6)

where $m_{2}\in[0,1)$. Here we used

${1\over y^{3}}-{1\over(y^{2}+1)^{3\over 2}}<{1\over
y^{3}}-{1\over(y+1)^{3}}={3y^{2}+3y+1\over y^{3}(y+1)^{3}}={3\over
y^{2}(y+1)^{2}}+{1\over y^{3}(y+1)^{3}}<{3\over 4}+{1\over 8}={7\over 8}.$

Therefore, the range of $y$ is $[1,\sqrt{3}]$, and $y$ is strictly decreasing
with respect to $m_{2}\in[0,1]$. By direct computations, we can reduce $D$,
defined in (3.9), to

$D=3\left(\begin{matrix}{8(1-m_{2})\over(1+7m_{2})(y^{2}+1)^{5\over 2}}&0\\\
0&1-{8(1-m_{2})\over(1+7m_{2})(y^{2}+1)^{5\over 2}}\end{matrix}\right).$ (4.7)

Let

$z={8(1-m_{2})\over(1+7m_{2})(y^{2}+1)^{5\over 2}}.$ (4.8)

We then have $\lambda_{3}=3(1-z)$, $\lambda_{4}=3z$ and $\det D=9z(1-z)$. It
follows that

$\alpha=\lambda_{3}-\lambda_{4}=6({1\over 2}-z).$ (4.9)

If $m_{2}=0$, then $y=\sqrt{3}$, hence (4.8) gives $z={1\over 4}$, and (4.9)
gives $\alpha={3\over 2}$; if $m_{2}=1$, (4.8) gives $z=0$, and (4.9) gives
$\alpha=3$. Figure 1 and Corollary 3.11 we have the relation of $\alpha$ and
$m_{2}$ and we obtain the stability of the system in term of $m_{2}$ and
Theorem 1.3 holds.

##  Appendix A Appendix

###  A.1 Preliminaries of ${\omega}$-Maslov-type indices and ${\omega}$-Morse
indices

Denote by ${\rm Sp}(2n)$ the symplectic group of real $2n\times 2n$ matrices.
Following [14], for any $\omega\in{\bf U}=\\{z\in{\bf C}\;|\;|z|=1\\}$, define
a real function
$D_{\omega}(M)=(-1)^{n-1}\overline{{\omega}}^{n}det(M-{\omega}I_{2n})$ for any
$M$ in the symplectic group ${\rm Sp}(2n)$. Then we can define ${\rm
Sp}(2n)_{{\omega}}^{0}=\\{M\in{\rm Sp}(2n)\,|\,D_{{\omega}}(M)=0\\}$ and ${\rm
Sp}(2n)_{{\omega}}^{\ast}={\rm Sp}(2n)\setminus{\rm Sp}(2n)_{{\omega}}^{0}$.
The orientation of ${\rm Sp}(2n)_{{\omega}}^{0}$ at any of its point $M$ is
defined to be the positive direction
$\frac{{\mathrm{d}}}{{\mathrm{d}}t}Me^{tJ}|_{t=0}$ of the path $Me^{tJ}$ with
$t>0$ small enough. Let $\nu_{{\omega}}(M)=\dim_{{\bf C}}\ker_{{\bf
C}}(M-{\omega}I_{2n})$. Let $\mathcal{P}_{2\pi}(2n)=\\{{\gamma}\in
C([0,2\pi],{\rm Sp}(2n))\;|\;{\gamma}(0)=I\\}$ and ${\gamma}_{0}(t)={\rm
diag}(2-\frac{t}{2\pi},(2-\frac{t}{2\pi})^{-1})$ for $0\leq t\leq 2\pi$.

Given any two $2m_{k}\times 2m_{k}$ matrices of square block form
$M_{k}=(\begin{smallmatrix}A_{k}&B_{k}\\\ C_{k}&D_{k}\end{smallmatrix})$ with
$k=1,2$, the symplectic sum of $M_{1}$ and $M_{2}$ is defined (cf. [14]) by
the following $2(m_{1}+m_{2})\times 2(m_{1}+m_{2})$ matrix
$M_{1}{\rm\diamond}M_{2}$:

$M_{1}{\rm\diamond}M_{2}=\begin{pmatrix}A_{1}&0&B_{1}&0\\\ 0&A_{2}&0&B_{2}\\\
C_{1}&0&D_{1}&0\\\ 0&C_{2}&0&D_{2}\end{pmatrix}.$

For any two paths $\xi_{j}\in{\cal P}_{\tau}(2n_{j})$ with $j=0$ and $1$, let
$\xi_{0}{\rm\diamond}\xi_{1}(t)=\xi_{0}(t){\rm\diamond}\xi_{1}(t)$ for all
$t\in[0,\tau]$. As in [14], for ${\lambda}\in{\bf R}\setminus\\{0\\}$,
$a\in{\bf R}$, ${\theta}\in(0,\pi)\cup(\pi,2\pi)$,
$b=(\begin{smallmatrix}b_{1}&b_{2}\\\ b_{3}&b_{4}\end{smallmatrix})$ with
$b_{i}\in{\bf R}$ for $i=1,\ldots,4$, and $c_{j}\in{\bf R}$ for $j=1,2$, some
normal forms are given by

$\displaystyle D({\lambda})=\begin{pmatrix}{\lambda}&0\\\
0&{\lambda}^{-1}\end{pmatrix},\qquad
R({\theta})=\begin{pmatrix}\cos{\theta}&-\sin{\theta}\\\
\sin{\theta}&\cos{\theta}\end{pmatrix},\qquad
N_{1}({\lambda},a)=\begin{pmatrix}{\lambda}&a\\\ 0&{\lambda}\end{pmatrix},$
(A.1) $\displaystyle
N_{2}(e^{\sqrt{-1}{\theta}},b)=\begin{pmatrix}R({\theta})&b\\\
0&R({\theta})\end{pmatrix},\quad
M_{2}({\lambda},c)=\begin{pmatrix}{\lambda}&1&c_{1}&0\\\
0&{\lambda}&c_{2}&(-{\lambda})c_{2}\\\ 0&0&{\lambda}^{-1}&0\\\
0&0&-{\lambda}^{-2}&{\lambda}^{-1}\end{pmatrix}.$ (A.2)

Here $N_{2}(e^{\sqrt{-1}{\theta}},b)$ is trivial if
$(b_{2}-b_{3})\sin{\theta}>0$, or non-trivial if
$(b_{2}-b_{3})\sin{\theta}<0$, in the sense of Definition 1.8.11 on p.41 of
[14]. Note that by Theorem 1.5.1 on pp.24-25 and (1.4.7)-(1.4.8) on p.18 of
[14], when ${\lambda}=-1$ there hold $c_{2}\not=0$ if and only if
$\dim\ker(M_{2}(-1,c)+I)=1$ and $c_{2}=0$ if and only if
$\dim\ker(M_{2}(-1,c)+I)=2$. For more details, readers may refer Section
1.4-1.8 of [14].

For any symplectic path $\xi\in\mathcal{P}_{2\pi}(2n)$ we define
$\nu_{\omega}({\gamma})=\nu_{\omega}(\xi(2\pi))$ and

$i_{\omega}(\xi)=[{\rm Sp}(2n)_{\omega}^{0}:\xi\ast\xi_{n}],\qquad{\rm
if}\;\;\xi(2\pi)\not\in{\rm Sp}(2n)_{{\omega}}^{0},$

i.e., the usual homotopy intersection number, and the orientation of the joint
path $\xi\ast\xi_{n}$ is its positive time direction under homotopy with fixed
end points, where the path
$\xi_{n}(t)=(\begin{smallmatrix}2-\frac{t}{\tau}&0\\\
0&(2-\frac{t}{\tau})^{-1}\end{smallmatrix})$. When $\xi(2\pi)\in{\rm
Sp}(2n)_{{\omega}}^{0}$, the index $i_{{\omega}}(\xi)$ follows [14]. The pair
$(i_{{\omega}}(\xi),\nu_{{\omega}}(\xi))\in{\bf Z}\times\\{0,1,\ldots,2n\\}$
is called the index function of $\xi$ at ${\omega}$. When
$\nu_{{\omega}}(\xi)=0$ or $\nu_{{\omega}}(\xi)>0$, the path $\xi$ is called
${\omega}$-non-degenerate or ${\omega}$-degenerate respectively. For more
details readers may refer to [14].

For $T>0$, suppose $x$ is a critical point of the functional

$F(x)=\int_{0}^{T}L(t,x,\dot{x})dt,\qquad\forall\,\,x\in W^{1,2}({\bf R}/T{\bf
Z},{\bf R}^{n}),$

where $L\in C^{2}(({\bf R}/T{\bf Z})\times{\bf R}^{2n},{\bf R})$ and satisfies
the Legendrian convexity condition $L_{p,p}(t,x,p)>0$. It is well known that
$x$ satisfies the corresponding Euler-Lagrangian equation:

$\displaystyle\frac{{\mathrm{d}}}{{\mathrm{d}}t}L_{p}(t,x,\dot{x})-L_{x}(t,x,\dot{x})=0,$
(A.3) $\displaystyle x(0)=x(T),\qquad\dot{x}(0)=\dot{x}(T).$ (A.4)

For such an extremal loop, define

$\displaystyle P(t)=L_{p,p}(t,x(t),\dot{x}(t)),\quad
Q(t)=L_{x,p}(t,x(t),\dot{x}(t)),\quad R(t)=L_{x,x}(t,x(t),\dot{x}(t)).$ (A.5)

Note that
$F\,^{\prime\prime}(x)=-\frac{{\mathrm{d}}}{{\mathrm{d}}t}(P\frac{{\mathrm{d}}}{{\mathrm{d}}t}+Q)+Q^{T}\frac{{\mathrm{d}}}{{\mathrm{d}}t}+R$.

For $\omega\in{\bf U}$, set $D(\omega,T)=\\{y\in W^{1,2}([0,T],{\bf
C}^{n})\,|\,y(T)=\omega y(0)\\}.$ We define the $\omega$-Morse index
$\phi_{\omega}(x)$ of $x$ to be the dimension of the largest negative definite
subspace of $\langle F\,^{\prime\prime}(x)y_{1},y_{2}\rangle$, for all
$y_{1},y_{2}\in D(\omega,T)$, where $\langle\cdot,\cdot\rangle$ is the inner
product in $L^{2}$. For $\omega\in{\bf U}$, we also set

$\displaystyle\overline{D}(\omega,T)=\\{y\in W^{2,2}([0,T],{\bf
C}^{n})\,|\,y(T)=\omega y(0),\dot{y}(T)={\omega}\dot{y}(0)\\}.$ (A.6)

Then $F^{\prime\prime}(x)$ is a self-adjoint operator on $L^{2}([0,T],{\bf
R}^{n})$ with domain $\overline{D}(\omega,T)$. We also define the
${\omega}$-nullity $\nu_{{\omega}}(x)$ of $x$ by

$\nu_{{\omega}}(x)=\dim\ker(F^{\prime\prime}(x)).$

Note that we only use $n=2$ in (A.6) from in this paper.

In general, for a self-adjoint linear operator $A$ on the Hilbert space
$\mathscr{H}$, we set $\nu(A)=\dim\ker(A)$ and denote by $\phi(A)$ its Morse
index which is the maximum dimension of the negative definite subspace of the
symmetric form $\langle A\cdot,\cdot\rangle$. Note that the Morse index of $A$
is equal to the total multiplicity of the negative eigenvalues of $A$.

On the other hand, $\tilde{x}(t)=(\partial L/\partial\dot{x}(t),x(t))^{T}$ is
the solution of the corresponding Hamiltonian system of (A.3)-(A.4), and its
fundamental solution $\gamma(t)$ is given by

$\displaystyle\begin{cases}\dot{\gamma}(t)&=JB(t)\gamma(t),\\\
\gamma(0)&=I_{2n},\end{cases}$ (A.7)

with $B(t)=\left(\begin{smallmatrix}P^{-1}(t)&-P^{-1}(t)Q(t)\\\
-Q(t)^{T}P^{-1}(t)&Q(t)^{T}P^{-1}(t)Q(t)-R(t)\end{smallmatrix}\right)$.

######  Lemma A.1.

([14], p.172) For the $\omega$-Morse index $\phi_{\omega}(x)$ and nullity
$\nu_{\omega}(x)$ of the solution $x=x(t)$ and the $\omega$-Maslov-type index
$i_{\omega}(\gamma)$ and nullity $\nu_{\omega}(\gamma)$ of the symplectic path
${\gamma}$ corresponding to $x$, for any $\omega\in{\bf U}$ we have
$\phi_{\omega}(x)=i_{\omega}(\gamma)$, and
$\nu_{\omega}(x)=\nu_{\omega}(\gamma).$

A generalization of the above lemma to arbitrary boundary conditions is given
in [HS1]. For more information on these topics, we refer to [14]. To measure
the jumps between $i_{\omega}({\gamma})$ and $i_{\lambda}({\gamma})$ with
$\lambda\in{\bf U}$ near ${\omega}$ from two sides of ${\omega}$ in ${\bf U}$,
the splitting numbers $S_{M}^{\pm}({\omega})$ is defined by followings.

######  Definition A.2.

([14]) For any $M\in{\rm Sp}(2n)$ and ${\omega}\in{\bf U}$, choosing $\tau>0$
and ${\gamma}\in{\cal P}_{\tau}(2n)$ with ${\gamma}(\tau)=M$, we define

$\displaystyle S_{M}^{\pm}({\omega})=\lim_{\epsilon\rightarrow
0^{+}}\;i_{\exp(\pm\epsilon\sqrt{-1}){\omega}}({\gamma})-i_{\omega}({\gamma}).$
(A.8)

They are called the splitting numbers of $M$ at ${\omega}$.

For any ${\omega}_{0}=e^{\sqrt{-1}{\theta}_{0}}\in{\bf U}$ with
$0\leq{\theta}_{0}<2\pi$, the eigenvalues of $M$ on ${\bf U}$ are denote by
${\omega}_{j}$ with $1\leq j\leq p_{0}$ which are distributed counterclockwise
from $1$ to ${\omega}_{0}$ and located strictly between $1$ and
${\omega}_{0}$. Then we have

$\displaystyle
i_{{\omega}_{0}}({\gamma})=i_{1}({\gamma})+S_{M}^{+}(1)+\sum_{j=1}^{p_{0}}(-S_{M}^{-}({\omega}_{j})+S_{M}^{+}({\omega}_{j}))-S_{M}^{-}({\omega}_{0}).$
(A.9)

The splitting numbers have following properties.

######  Lemma A.3.

([14], p.198) The integer valued splitting number pair
$(S_{M}^{+}({\omega}),S_{M}^{-}({\omega}))$ defined for all
$({\omega},M)\in{\bf U}\times\cup_{n\geq 1}{\rm Sp}(2n)$ are uniquely
determined by the following axioms:

$1^{\circ}$ (Homotopy invariant) $S_{M}^{\pm}({\omega})=S_{N}^{\pm}({\omega})$
for all $N\in\Omega^{0}(M)$.

$2^{\circ}$ (Symplectic additivity) $S_{M_{1}\diamond
M_{2}}^{\pm}({\omega})=S_{M_{1}}^{\pm}({\omega})+S_{M_{2}}^{\pm}({\omega})$
for all $M_{i}\in{\rm Sp}(2n_{i})$ with $i=1,2$.

$3^{\circ}$ (Vanishing) $S_{M}^{\pm}({\omega})=0$ if
${\omega}\not\in\sigma(M)$.

$4^{\circ}$ (Normality) $(S_{M}^{+}({\omega}),S_{M}^{-}({\omega}))$ coincides
with the ultimate type of ${\omega}$ for $M$ when $M$ is any basic normal
form.

Moreover, by Lemma 9.1.6 on p.192 of [14] for ${\omega}\in{\bf C}$ and
$M\in{\rm Sp}(2n)$, we have

$\displaystyle S_{M}^{+}({\omega})=S_{M}^{-}(\overline{\omega}).$ (A.10)

The ultimate type of ${\omega}\in{\bf U}$ for a symplectic matrix $M$
mentioned in the above lemma is given in Definition 1.8.12 on pp.41-42 of [14]
algebraically with its more properties studied there.

For the reader’s conveniences, following the List 9.1.12 on pp.198-199 of
[14], the splitting numbers (i.e., the ultimate types) for all basic normal
forms are given by:

$\langle$1$\rangle$ $(S_{M}^{+}(1),S_{M}^{-}(1))=(1,1)$ for $M=N_{1}(1,b)$
with $b=1$ or $0$.

$\langle$2$\rangle$ $(S_{M}^{+}(1),S_{M}^{-}(1))=(0,0)$ for $M=N_{1}(1,-1)$.

$\langle$3$\rangle$ $(S_{M}^{+}(-1),S_{M}^{-}(-1))=(1,1)$ for $M=N_{1}(-1,b)$
with $b=-1$ or $0$.

$\langle$4$\rangle$ $(S_{M}^{+}(-1),S_{M}^{-}(-1))=(0,0)$ for $M=N_{1}(-1,1)$.

$\langle$5$\rangle$
$(S_{M}^{+}(e^{\sqrt{-1}{\theta}}),S_{M}^{-}(e^{\sqrt{-1}{\theta}}))=(0,1)$
for $M=R(\theta)$ with ${\theta}\in(0,\pi)\cup(\pi,2\pi)$.

$\langle$6$\rangle$ $(S_{M}^{+}({\omega}),S_{M}^{-}({\omega}))=(1,1)$ for
$M=N_{2}({\omega},b)$ being non-trivial (cf. Definition 1.8.11 on p.41 of
[14]) with ${\omega}=e^{\sqrt{-1}{\theta}}\in{\bf U}\backslash{\bf R}$.

$\langle$7$\rangle$ $(S_{M}^{+}({\omega}),S_{M}^{-}({\omega}))=(0,0)$ for
$M=N_{2}({\omega},b)$ being trivial (cf. Definition 1.8.11 on p.41 of [14])
with ${\omega}=e^{\sqrt{-1}{\theta}}\in{\bf U}\backslash{\bf R}$.

$\langle$8$\rangle$ $(S_{M}^{+}({\omega}),S_{M}^{-}({\omega}))=(0,0)$ for
${\omega}\in{\bf U}$ and $M\in{\rm Sp}(2n)$ satisfying $\sigma(M)\cap{\bf
U}=\emptyset$.

For any symplectic path ${\gamma}\in\mathcal{P}_{2\pi}(2n)$ and $m\in{\bf N}$,
the $m$-th iteration ${\gamma}_{m}:[0,\tau]\longrightarrow{\rm Sp}(2n)$ is
given by ${\gamma}^{m}:[0,m\tau]\longrightarrow{\rm Sp}(2n),$ with
${\gamma}^{m}(t)={\gamma}(t-j\tau){\gamma}(\tau)^{j}$ for $g\tau\leq
t\leq(j+1)\tau$ and $j\in\\{0,1,\dots,m-1\\}$. The next Bott-type iteration
formula is will be used in this paper.

######  Lemma A.4 (cf. Theorem 9.2.1 of [14]).

For any $z\in{\bf U}$,
$i_{z}({\gamma}^{m})=\sum_{{\omega}^{m}=z}i_{{\omega}}({\gamma}).$

## References

  * [1] L. Euler, De motu rectilineo trium corporum se mutuo attrahentium. Novi Comm. Acad. Sci. Imp. Petrop. 11\. (1767) 144-151.
  * [2] G. Gascheau, Examen d’une classe d’équations différentielles et application à un cas particulier du problème des trois corps. Comptes Rend. Acad. Sciences. 16\. (1843) 393-394.
  * [3] X. Hu, Y. Long, S. Sun, Linear stability of elliptic Lagrangian solutions of the classical planar three-body problem via index theory. Arch. Ration. Mech. Anal. 213\. (2014) 993-1045.
  * [4] X. Hu, Y. Long, and Y. Ou.  Linear stability of the elliptic relative equilibrium with (1+n)-gon central configurations in planar n-body problem.  Nonlinearity, 33(3):1016–1045, 2020.
  * [5] X. Hu, S. Sun, Morse index and stability of elliptic Lagrangian solutions in the planar three-body problem. Adv. Math. 223\. (2010) 98-119.
  * [6] X. Hu, Y. Ou, and P. Wang.  Trace formula for linear Hamiltonian systems with its applications to elliptic Lagrangian solutions.  Arch. Ration. Mech. Anal., 216(1):313–357, 2015.
  * [7] R. Iturriaga, E. Maderna, Generic uniqueness of the minimal Moulton central configuration. Celest. Mech. Dyn. Astr. 123 (2015), 351-361.
  * [8] J. L. Lagrange, Essai sur le problème des trois corps. Chapitre II. Œuvres Tome 6, Gauthier-Villars, Paris. (1772) 272-292.
  * [9] E. S. G. Leandro.  Finiteness and bifurcations of some symmetrical classes of central configurations.  Arch. Ration. Mech. Anal., 167(2):147–177, 2003\.
  * [10] E. S. G. Leandro.  Structure and stability of the rhombus family of relative equilibria under general homogeneous forces.  J. Dynam. Differential Equations, 31(2):933–958, 2018.
  * [11] J. Liouville, Sur un cas particulier du problème des trois corps. J. Math. Pures Appl. 7\. (1842) 110-113.
  * [12] B. Liu.  Linear instability of elliptic rhombus solutions to the planar four-body problem.  Nonlinearity, Nonlinearity 34 (11), 7728–7749, 2021.
  * [13] B. Liu, Q. Zhou, Linear stability of elliptic relative equilibria of restricted four-body problem. J. Diff. Equa. 269\. (2020) 4751-4798.
  * [14] Y. Long, Index Theory for Symplectic Paths with Applications. Progress in Math. 207, Birkhäuser. Basel. 2002.
  * [15] Y. Long, Lectures on Celestial Mechanics and Variational Methods. Preprint. 2012\.
  * [16] A. Mansur, D. Offin, and M. Lewis.  Instability for a family of homographic periodic solutions in the parallelogram four body problem.  Qual. Theory Dyn. Syst., 16(3):671–688, 2017.
  * [17] R. Martínez, A. Samà, C. Simó, Stability of homograpgic solutions of the planar three-body problem with homogeneous potentials. in International conference on Differential equations. Hasselt, 2003, eds, Dumortier, Broer, Mawhin, Vanderbauwhede and Lunel, World Scientific, (2004) 1005-1010.
  * [18] R. Martínez, A. Samà, C. Simó, Stability diagram for 4D linear periodic systems with applications to homographic solutions. J. Diff. Equa. 226\. (2006) 619-651.
  * [19] R. Martínez, A. Samà, C. Simó, Analysis of the stability of a family of singular-limit linear periodic systems in ${\bf R}^{4}$. Applications. J. Diff. Equa. 226\. (2006) 652-686.
  * [20] K. Meyer, D. Schmidt, Elliptic relative equilibria in the N-body problem. J. Diff. Equa. 214\. (2005) 256-298.
  * [21] F. Moulton, The straight line solutions of the $n$-body problem. Ann. of Math. II Ser. 12 (1910) 1-17.
  * [22] G. Roberts, Linear stability of the elliptic Lagrangian triangle solutions in the three-body problem. J. Diff. Equa. 182\. (2002) 191-218.
  * [23] E. Routh, On Laplace’s three particles with a supplement on the stability or their motion. Proc. London Math. Soc. 6\. (1875) 86-97.
  * [24] Q. Zhou.  Linear stability of elliptic relative equilibria of four-body problem with two infinitesimal masses.  arXiv preprint arXiv:1908.01345, 2019.
  * [25] Q. Zhou and Y. Long.  Maslov-type indices and linear stability of elliptic Euler solutions of the three-body problem.  Arch. Ration. Mech. Anal., 226(3):1249–1301, 2017\.
  * [26] Q. Zhou and Y. Long.  The reduction of the linear stability of elliptic Euler-Moulton solutions of the $n$-body problem to those of 3-body problems.  Celestial Mech. Dynam. Astronom., 127(4):397–428, 2017.
  * [27] G. Zhu, Y. Long Linear stability of some symplectic matrices. Front. Math. China. 5(2010), no. 2, 361–368.