#  Robust Auto-landing Control of an agile Regional Jet Using Fuzzy Q-learning

Mohsen Zahmatkesh  Seyyed Ali Emami  Afshin Banazadeh<EMAIL_ADDRESS>Paolo Castaldi  Department of Aerospace Engineering, Sharif University of
Technology, Tehran, Iran  Department of Electrical, Electronic and Information
Engineering ”Guglielmo Marconi”, University of Bologna, Via Dell’Universit‘a
50, Cesena, Italy

###### Abstract

A robust auto-landing problem of a Truss-braced Wing (TBW) regional jet
aircraft with poor stability characteristics is presented in this study
employing a Fuzzy Reinforcement Learning scheme. Reinforcement Learning (RL)
has seen a recent surge in practical uses in control systems. In contrast to
many studies implementing Deep Learning in RL algorithms to generate
continuous actions, the methodology of this study is straightforward and
avoids complex neural network architectures by applying Fuzzy rules. An
innovative, agile civil aircraft is selected not only to meet future aviation
community expectations but also to demonstrate the robustness of the suggested
method. In order to create a multi-objective RL environment, a Six-degree-of-
freedom (6-DoF) simulation is first developed. By transforming the auto-
landing problem of the aircraft into a Markov Decision Process (MDP)
formulation, the problem is solved by designing a low-level Fuzzy Q-learning
(FQL) controller. More specifically, the well-known Q-learning method, which
is a discrete RL algorithm, is supplemented by Fuzzy rules to provide
continuous actions with no need to complex learning structures. The
performance of the proposed system is then evaluated by extensive flight
simulations in different flight conditions considering severe wind gusts,
measurement noises, actuator faults, and model uncertainties. Besides, the
controller effectiveness would be compared with existing competing techniques
such as Dynamic Inversion (DI) and Q-learning. The simulation results indicate
the superior performance of the proposed control system as a reliable and
robust control method to be employed in real applications.

###### keywords:

Fuzzy Q-learning, Q-learning, Reinforcement Learning, Auto-landing

††journal: Engineering Applications of Artificial Intelligence

##  1 Introduction

Recently, the aviation industry has faced a number of challenges, including
increased emissions and congested airspace. As a result, the future world
requirements will be characterized as safety and efficiency. In the efficiency
segment, reducing emissions by using less fuel and making economical flights
alongside faster flights is the main goal. On another hand, high-speed flight
management in moreover congested airspace falls within the subject of safety.
In conclusion, the discussed parameters strongly encourage the development of
novel aircraft configurations to gain advantageous elements. The Scope Clause,
on the other hand, is an agreement that places a cap on the number of aircraft
seats in order to prevent outsourcing and guarantee the jobs of union pilots.
Therefore, it is inevitable that the Modern Regional Jet (MRJ) fleet would
grow. That demonstrates the importance of reliable flight control systems. In
high-performance aerodynamic configurations, such as TBW aircraft, there are
some re-raised interests [1, 2, 3]. Apart from that, the aviation industry has
expressed interest in TBWs because of their fuel burn efficiency [4]. Despite
considerable dynamic modeling research [5], it appears that no reliable auto-
landing controller has been developed for these configurations.

Studies show that the landing procedure is the riskiest part of flying and
calls for expert pilotage abilities. The International Civil Aviation
Organization (ICAO) 6th annex document introduces three alternate
aerodromes—Take-off, En-route, and Destination—where an MRJ shall land in the
event of particular failure scenarios. In this case, [6] examined the most
frequent causes of flying incidents and accidents during the approach phase.
The most hazardous elements included are pilot erroneous decisions (74%),
skipping or completing activities incorrectly (72%), and ineffective crew
communication, mutual cooperation, and mutual control (63%). The cited
elements strengthen the function of trendy autonomous controllers with great
dependability to bolster safety. There are several studies with advanced
control centralization in the landing procedure. For instance, [7] developed a
double-loop nonlinear dynamic inversion controller based on a deep failure
estimator made up of layers of Long-short Term Memory (LSTM) and Convolution
Neural Networks (CNN). These neural networks were trained for severe stuck
failures, with time-series data on landing trajectory patterns in actuator
faulty and healthy landing condition simulations. In order to control the
trajectory of a tailless and blended wing UAV confronting air turbulences and
sensor measurement errors, a unique auto-landing framework was presented in
[8]. In this research, a Backstepping-based controller is used to control the
attitude angles. A Dynamic Inversion-based controller creates the throttle
signal to keep a constant velocity, and an adaptive disturbance observer
estimates the atmospheric turbulences to track the proposed landing
trajectory. In [9] a generalized Anti-windup based on traditional PID
controllers as well as a phase compensation system used to train a neural-
assisted Sliding Mode controller. In order to tackle stuck actuator failures
during an auto-landing scenario. It is demonstrated that the capacity of the
neural-aided Sliding Mode controller to tolerate faults is greatly improved by
adding Anti-windup and phase compensation. An aircraft heading angle is guided
by a Deep Q-learning (DQL) in [10], allowing it to land in the desired
2-dimensional field. In this study, the dynamic modeling of aircraft is not
included. In [11] a Deep Deterministic Policy Gradient (DDPG) was used to
control an Unmanned Aerial Vehicle (UAV’s) desired path for the landing flare
phase in the longitudinal channel in existing wind disturbances. The structure
of the proposed method includes two Deep ANNs as an Actor-critic architecture.
Similar to this, the DDPG approach is employed in the outer loop of a landing
procedure in [12] whereas the inner loop is controlled by a Proportional-
integral-derivative (PID) controller. It is important to note that several
classic techniques for aircraft attitude control have been created to improve
the quality of landings. The main shortcoming of these traditional theories is
their lack of adaptation in extended working points. To overcome this
weakness, some existing approaches have increased their robustness by
utilizing ANNs.

In this regard, publications like [13] and [14] are considered. In the first
paper, a $H_{2}$ controller is addressed to actuator faults and wind
disturbances. The focus of the second paper is on designing an online neural-
aided controller to increase the robustness of existing controllers for fault
tolerance. Although not specified in the landing phase, a different group of
publications also deals with controller design. For instance, [15] presented a
layered Model Predictive Control system based on simple sparse rapidly
exploring random trees for path planning, path control, velocity control, and
angular velocity control. This method improved the estimation of a real-time,
nonlinear, and onboard convex Flight-envelop calculation. For hydraulic
actuator failures, [16] created an Integral Sliding Mode controller featuring
Control Allocation. This controller eliminates the need to redesign another
controller owing to distributing control signals among redundant actuators. An
ANN-based adaptive controller using Feedback Linearization was developed in
[17] to address the dramatic roll and unsteady longitudinal behavior in the
condition of partial wing damage. In another study, to achieve hydraulic fault
tolerance in the longitudinal axis, [18] evaluated the performance of three
controllers, including Adaptive Back-stepping, Robust Sliding Mode, and PID.
In this instance, the initial controller overcame the other methods. A
Q-learning horizontal trajectory tracking controller with an ANN foundation
was developed in [19] using the MDP model of an airship with fine stability
characteristics. In this study, the method of action selection was optimized
using a Cerebellar Model Articulation Controller (CMAC) neural network. A Soft
Actor-critic (SAC) technique was used in [20] to solve a path planning problem
for a Long-endurance, Solar-powered UAV that took energy consumption into
account. Another study cited as [21] focused on a Skywalker X8 inner loop
control employing SAC and comparing it with a PID controller.

Proximal Policy Optimization (PPO), was used in [22] for orientation control
of a typical extremely dynamic coupled Fixed-wing aircraft in the stall
circumstance. After 100,000 episodes, there was a successful convergence of
the PPO. The efforts that have been mentioned thus far use ANNs to improve
convergence and robustness. To the best of our knowledge, there are discrete
RL-based attitude control studies without using ANNs. A Q-learning method was
used in [23] to control longitudinal and lateral angles in a general aviation
airplane (Cessna 172). This research controls the desired angles of zero and
the airplane profits good stability characteristics.

There are some Fuzzy adaptations on [24] work like [25] where the Q-functions
and action selection strategy are inferred from Fuzzy rules. Also, in order to
reduce the number of states needed to shape an MDP model for mobile robots
that avoid obstacles, [26] suggests a Fuzzy technique. Because the mobile
robot may encounter an infinite number of different conditions. The Fuzzy
method is used to generalize the condition and minimize processor requirements
alongside reducing states. Also, [27] proposed a dynamic Fuzzy Q-learning for
online and continuous tasks in mobile robots. In [28], the Fuzzy Q-learning
(FQL) method and Strictly Negative Imaginary (SNI) property are used to
provide a novel robust adaptive control for quadrotor attitude and altitude
stabilization. The objective is to develop a control strategy that dynamically
adapts the SNI controller using FQL. Another study, [29] used Q-learning as an
attitude controller for a unique, highly maneuverable regional jet aircraft in
MDP and POMDP scenarios. The simulation results in a variable pitch angle
tracking were satisfactory.

Motivated by the preceding discussions, the following are the main
contributions of the current study:

  1. 1.

A novel continuous action generator is developed as a general connector
between every (discrete/continuous) optimal policy and the RL environment.

  2. 2.

In response to worldwide aviation community expectations, a TBW aircraft
(figure 1) with specific stability characteristics is selected for the auto-
landing problem, where the high maneuverability of the aircraft brings
significant challenges into the design process.

  3. 3.

In contrast to many studies, the complexity of ANN architectures and the low
adaptation of classic methods are well resolved using Fuzzy Q-learning.

  4. 4.

The robustness and reliability of the proposed FQL are examined under
different flight conditions consisting of sensor measurement noises,
atmospheric disturbances, actuator faults, and model uncertainties.

Figure 1: Chaka 50 and 76 Modern Regional Jet (MRJ) Family [4]

##  2 Six-DoF Aircraft Dynamic Modeling

In order to develop a nonlinear RL environment, the 6-DoF nonlinear equations
of motions cited in [30, 31] are utilized in this section. Many environments
based on Gym and Flight Gear are open-source, such as GymFG ([32]). But this
plant must model and simulate from scratch due to the unique characteristics
of the innovative configuration. In this approach, it is presumed that the
earth is flat. Consequently, the body frame translational and rotational
equations are as follows:

$mD^{B}\bm{v}^{B}+m\bm{\Omega}^{B}\bm{v}^{B}=\bm{f}_{a}^{B}+\bm{f}_{p}^{B}+m\bm{g}^{B},$
(1)
$D^{B}(\bm{I}^{B}\bm{\omega}^{B})+\bm{\Omega}^{B}\bm{I}^{B}\bm{\omega}^{B}=\bm{m}_{a}^{B}+\bm{m}_{p}^{B}.$
(2)

Where $u,v,w$ are the velocity components, and $D^{B}$ is defined as the
Rotational Time Derivative in body frame. So the $D^{B}\bm{v}^{B}$ equals
$[\frac{dv}{dt}]^{B}=[\dot{u}\ \ \dot{v}\ \ \dot{w}]^{T}$, and
$[\omega]^{B}=[p\ \ q\ \ r]^{T}$ are the roll, pitch, and yaw angular rates
vector in body frame. Also $\bm{\Omega}^{B}$ is the skew-symmetric form of
angular rates vector.

$[\Omega]^{B}=\begin{bmatrix}0&-r&q\\\ r&0&-p\\\ -q&p&0\end{bmatrix}.$ (3)

Furthermore, $m$ is the mass of aircraft, and $\bm{I}^{B}$ is the moment of
inertia matrix in body frame;

$[I]^{B}=\begin{bmatrix}I_{x}&0&I_{xz}\\\ 0&I_{y}&0\\\
I_{xz}&0&I_{z}\end{bmatrix}.$ (4)

In equations (1) and (2), some variables on the right-hand side are considered
to be zero including $\bm{m}_{p}^{B}$ vector which is engine power moments.
Also, $\bm{f}_{a}^{B}$, $\bm{f}_{p}^{B}$, and $\bm{m}_{a}^{B}$ are considered
as aerodynamic force, engine power force, and aerodynamic moment vectors.
Where except engine power, the other non-zero forces and moments are computed
in aerodynamic frame as follows;

$\begin{aligned} \begin{bmatrix}L\\\ D\\\
m_{a}\end{bmatrix}^{S}=\bar{q}S\bar{c}\begin{bmatrix}c_{L_{0}}&c_{L_{\alpha}}&c_{L_{\dot{\alpha}}}&c_{L_{u}}&c_{L_{q}}&c_{L_{\delta_{E}}}\\\
c_{D_{0}}&c_{D_{\alpha}}&c_{D_{\dot{\alpha}}}&c_{D_{u}}&c_{D_{q}}&c_{D_{\delta_{E}}}\\\
c_{m_{0}}&c_{m_{\alpha}}&c_{m_{\dot{\alpha}}}&c_{m_{u}}&c_{m_{q}}&c_{m_{\delta_{E}}}\end{bmatrix}\begin{bmatrix}1\\\
\alpha\\\ \frac{\dot{\alpha}\bar{c}}{2V_{P_{1}}}\\\ \frac{u}{V_{P_{1}}}\\\
\frac{q\bar{c}}{2V_{P_{1}}}\\\ \delta_{E}\end{bmatrix}\end{aligned}.$ (5)

Furthermore, aerodynamic forces need a transfer from stability to the body
frames of the angle of attack $\alpha$ around $y_{S}$ axis.

$\displaystyle\begin{bmatrix}f_{a_{x}}\\\ f_{a_{y}}\\\
f_{a_{z}}\end{bmatrix}^{B}=\begin{bmatrix}cos\alpha&0&-sin\alpha\\\ 0&1&0\\\
sin\alpha&0&cos\alpha\end{bmatrix}^{BS}\begin{bmatrix}-D\\\ 0\\\
-L\end{bmatrix}^{S}.$ (6)

Also, the gravitational acceleration vector $\bm{g}^{B}$ in the body frame is
as follows:

$\begin{bmatrix}g_{x}\\\ g_{y}\\\
g_{z}\end{bmatrix}^{B}=\begin{Bmatrix}-g\sin(\theta)\\\
g\cos(\theta)\sin(\phi)\\\ g\cos(\theta)\cos(\phi)\end{Bmatrix}.$ (7)

Additionally, rotational kinematic equations are required for transfer from
body to inertial frames.

$\begin{bmatrix}\dot{\phi}\\\ \dot{\theta}\\\
\dot{\psi}\end{bmatrix}=\begin{bmatrix}1&\sin\varphi\tan\theta&\cos\varphi\tan\theta\\\
0&\cos\varphi&-\sin\varphi\\\
0&\sin\varphi/\cos\theta&\cos\varphi/\cos\theta\end{bmatrix}\begin{bmatrix}p\\\
q\\\ r\end{bmatrix}^{B}.$ (8)

So, the translational kinematic equations using (1), and (8), in the inertial
frame is achievable.

$\begin{bmatrix}\dot{x}\\\ \dot{y}\\\
\dot{z}\end{bmatrix}^{E}=\begin{bmatrix}\cos\psi\cos\theta&\cos\psi\sin\theta\sin\varphi-\sin\psi\cos\varphi&\cos\psi\sin\theta\cos\varphi+\sin\psi\sin\varphi\\\
\sin\psi\cos\theta&\sin\psi\sin\theta\sin\varphi+\cos\psi\cos\varphi&\sin\psi\sin\theta\cos\varphi-\cos\psi\sin\varphi\\\
-\sin\theta&\cos\theta\sin\varphi&\cos\theta\cos\varphi\end{bmatrix}\begin{bmatrix}u\\\
v\\\ w\end{bmatrix}^{B}.$ (9)

Based on Computational Fluid Dynamics (CFD), stability and control derivatives
for the Chaka-50 are presented in [4]. Table (1) summarises these derivatives
for two flying scenarios. Trim conditions in a wings-level flight are
calculated for simulation verification utilizing trim equations in [33]. The
drag equation takes into account the absolute values of $\delta_{E}$,
$i_{H_{1}}$, and $\alpha_{1}$. In addition, the flight path angle
$\gamma_{1}$, motor installation angle $\phi_{T}$, and horizontal tail
incidence angle $i_{H}$ are all equal to zero. By solving the trim equation,
the elevator deflection $\delta_{E}$ and required thrust $f_{a_{x}}$ for a
trim flight are obtained and shown in table (2). The numbers in the
aforementioned table (2) are crucial for the validation of the 6-DoF
simulation.

Table 1: Stability and control derivatives of Chaka 50 MRJ (1/rad) | Longitudinal
---
Derivatives
Ideal |  | Random
---
Uncertainty
([-10 10])%
| Longitudinal
---
Derivatives
Ideal |  | Random
---
Uncertainty
([-10 10])%
$c_{D_{0}}$ | 0.0338 | 0.0358 | $c_{L_{u}}$ | 0.081 | 0.076
$c_{L_{0}}$ | 0.3180 | 0.3363 | $c_{m_{u}}$ | -0.039 | -0.041
$c_{m_{0}}$ | -0.06 | -0.061 | $c_{L_{q}}$ | 12.53 | 12.56
$c_{D_{\alpha}}$ | 0.8930 | 0.893 | $c_{m_{q}}$ | -40.69 | -37.27
$c_{L_{\alpha}}$ | 14.88 | 14.52 | $c_{D_{\delta_{E}}}$ | 0.1570 | 01483
$c_{m_{\alpha}}$ | -11.84 | -11.84 | $c_{L_{\delta_{E}}}$ | 0.78 | 0.74
$c_{D_{u}}$ | 0.041 | 0.373 | $c_{m_{\delta_{E}}}$ | -5.98 | -5.93
Table 2: Trim parameters of Chaka 50 MRJ Parameter | Value
---|---
Required Thrust ($f_{a_{x}}$) | 21433.02 (lbs)
Required Elevator ($\delta_{E}$) | 0.39 (deg)
Angle of Attack ($\alpha$) | -2.28 (deg)
Table 3: Chaka 50 MRJ specifics for simulation Parameter | Value | Parameter | Value
---|---|---|---
Wing Area($m^{2}$) | 43.42 |  $I_{xx}$($kg.m^{2}$) | 378056.535
Mean Aerodynamic Chord($m$) | 1.216 |  $I_{yy}$($kg.m^{2}$) | 4914073.496
Span($m$) | 28 |  $I_{zz}$($kg.m^{2}$) | 5670084.803
Mass($kg$) | 18418.27 |  $I_{xz}$($kg.m^{2}$) | 0
Initial Speed $V_{P_{1}}(\frac{m}{s})$ | 160 | Initial height $h_{1}(m)$ | 100

###  2.1 Six-DoF Aircraft Dynamic Model considering Atmospheric Disturbances

Aircraft landing quality is affected by atmospheric disturbance which is air
turbulence in the small areas of the atmosphere that often happens close to
the ground. The coordinates of disturbances that result in a loss of lift and
altitude are the most hazardous since the aircraft is getting close to the
final approach. The atmospheric disturbance is described in the literature as
a stochastic process that is characterized by velocity spectra. There are two
widely used models that are typically used in flight dynamic simulations:
(1) Dryden Continuous Turbulence Model; and
(2) Von Karman Continuous Turbulence Model
The Dryden atmospheric turbulence is used in this study for two reasons.
First, it allows for easier mathematical modeling and covers both linear and
rotational components of disturbance velocity.

$\displaystyle G_{u}(s)=\sigma_{u}\sqrt{\frac{2L_{u}}{\pi
u_{1}}}\Bigg{[}\frac{1}{1+(\frac{L_{u}}{u_{1}}s)}\Bigg{]},$ (10)
$\displaystyle G_{v}(s)=\sigma_{v}\sqrt{\frac{L_{v}}{\pi
u_{1}}}\Bigg{[}\frac{1+2\sqrt{3}\frac{L_{v}}{u_{1}}s}{(1+\frac{2L_{v}}{u_{1}}s)^{2}}\Bigg{]},$
$\displaystyle G_{w}(s)=\sigma_{w}\sqrt{\frac{2L_{w}}{\pi
u_{1}}}\Bigg{[}\frac{1+2\sqrt{3}\frac{L_{w}}{u_{1}}s}{(1+\frac{2L_{w}}{u_{1}}s)^{2}}\Bigg{]},$

where according to [34], $L_{w}$, $L_{v}$, and $L_{u}$ are the scaling length;

$\displaystyle L_{u}=L_{v}=\frac{z}{(0.177+0.000823z)^{1.2}},$ (11)
$\displaystyle L_{w}=z,$

and $\sigma_{u}$, $\sigma_{v}$, and $\sigma_{w}$ are the intensity of
turbulence.

$\displaystyle\sigma_{u}=\sigma_{v}=\frac{\sigma_{w}}{(0.177+0.000823z)^{0.4}},$
(12) $\displaystyle\sigma_{w}=0.1u_{20},$

where wind speed at 20 feet is specified by $u_{20}$. The motion equations
have now been modified to include the effects of the wind based on [35]. In
general, the inertial frame is used to compute the wind and its derivatives.
But complex computations are required for its transition into the body. As an
alternative, one may use the derivatives in the body reference to reach; where
the $\bm{W}^{B}$ is the wind velocity vector in body frame; $[W]^{B}=[W_{x}\ \
W_{y}\ \ W_{z}]^{T}$.

$\displaystyle\dot{W}_{x}$ $\displaystyle=\bigg{[}\frac{\partial
W_{x}}{\partial x}\bigg{]}^{B}(u+W_{x})+\bigg{[}\frac{\partial W_{x}}{\partial
y}\bigg{]}^{B}(v+W_{y})+\bigg{[}\frac{\partial W_{x}}{\partial
z}\bigg{]}^{B}(w+W_{z})+\bigg{[}\frac{\partial W_{x}}{\partial
t}\bigg{]}^{B},$ (13) $\displaystyle\dot{W}_{y}$
$\displaystyle=\bigg{[}\frac{\partial W_{y}}{\partial
x}\bigg{]}^{B}(u+W_{x})+\bigg{[}\frac{\partial W_{y}}{\partial
y}\bigg{]}^{B}(v+W_{y})+\bigg{[}\frac{\partial W_{y}}{\partial
z}\bigg{]}^{B}(w+W_{z})+\bigg{[}\frac{\partial W_{y}}{\partial
t}\bigg{]}^{B},$ $\displaystyle\dot{W}_{z}$
$\displaystyle=\bigg{[}\frac{\partial W_{z}}{\partial
x}\bigg{]}^{B}(u+W_{x})+\bigg{[}\frac{\partial W_{z}}{\partial
y}\bigg{]}^{B}(v+W_{y})+\bigg{[}\frac{\partial W_{z}}{\partial
z}\bigg{]}^{B}(w+W_{z})+\bigg{[}\frac{\partial W_{z}}{\partial
t}\bigg{]}^{B}.$

The spatial derivatives of the wind speed, which are often stated in the
inertial frame, must be transferred to the body frames of reference in (13);

$[\nabla W]^{B}=[T]^{BE}[\nabla W]^{E}[\bar{T}]^{BE}.$ (14)

The effect of wind on angular rates $\bm{\omega}_{w}^{E}$ can be defined as a
rigid solid air caused by fluid stresses, and is expressed in the inertial
frame as;

E $\displaystyle=\frac{1}{2}\bigg{[}(\frac{\partial W_{z}}{\partial
y}-\frac{\partial W_{y}}{\partial
z})\bigg{]}^{E}{i}+\frac{1}{2}\bigg{[}(\frac{\partial W_{x}}{\partial
z}-\frac{\partial W_{z}}{\partial
x})\bigg{]}^{E}{j}+\frac{1}{2}\bigg{[}(\frac{\partial W_{y}}{\partial
x}-\frac{\partial W_{x}}{\partial y})\bigg{]}^{E}{k}.$ (15)

The above equation must be transferred to the body axis so as to use in the
6-DoF equation;

$\begin{bmatrix}p\\\ q\\\ r\end{bmatrix}^{B}=\begin{bmatrix}p\\\ q\\\
r\end{bmatrix}^{B}-[T]^{BE}\begin{bmatrix}(\frac{\partial W_{z}}{\partial
y}-\frac{\partial W_{y}}{\partial z})\\\ (\frac{\partial W_{x}}{\partial
z}-\frac{\partial W_{z}}{\partial x})\\\ (\frac{\partial W_{y}}{\partial
x}-\frac{\partial W_{x}}{\partial y})\end{bmatrix}^{E}.$ (16)

###  2.2 Actuator Fault

A type of failure that affects the plant inputs is an actuator fault. Actuator
faults in the aircraft might result from improper maintenance procedures, the
age of the material, or improper operation. In this study, the actuator fault
is expressed in two terms: the first, is a multiplicative term, which is the
elevator’s inability to achieve the required amount, and the second, is an
additive term, which is the output quantity bias.

$\displaystyle\delta_{E_{t}}=$ $\displaystyle
0.3\delta_{E_{t}}-0.7^{\circ},\qquad\textrm{If}\ t>12s,$ (17) $\displaystyle
0.4\delta_{E_{t}}+0.6^{\circ},\qquad\textrm{If}\ 12s>t>8s,$ $\displaystyle
0.5\delta_{E_{t}}-0.5^{\circ},\qquad\textrm{If}\ 8s>t>4s.$

According to equation (17), the elevator operates with $30\%$ power and
concurrently biases its output by $-0.7^{\circ}$ after 12 seconds of flight
because it is anticipated that the issue would worsen over time. After 4 and 8
seconds of flight, identical faults with different parameter values will have
occurred. In this study, $50\%$ deficiency and $60^{\circ}$ additive bias
after 4 seconds and $30\%$ deficiency and $0.6^{\circ}$ additive bias after 8
seconds were taken into consideration.

Figure 2: Landing Path and Landing Distance Diagram

##  3 Landing Path Planning

Examining Figure 2, the desired approach angle is $\theta_{a}\leqslant 3$
according to [36]. In this case assuming aircraft speed in touch down point
$V_{TD}=1.15V_{stall}$, and also speed in flare zone $V_{f}=1.23V_{stall}$.
So, the formula of flare circular arc is:

$R=\frac{V_{f}^{2}}{0.2g}.$ (18)

Now, in order to calculate $\theta_{f}$ in each time-step, flare altitude
$h_{f}$ is required.

$h_{f}=R-R\cos\theta_{a}.$ (19)

By considering aforementioned formulas, the approach distance $s_{a}$, and the
flare distance $s_{f}$ are as follows:

$s_{a}=\frac{-(50-h_{f})}{\tan\theta_{a}},$ (20) $s_{f}=-R\sin\theta_{a}.$
(21)

Also, by using the formula $s_{td}=s_{a}+s_{f}$ to represent touch-down
distance, the desired $\theta_{f}$ at each time step after covering approach
distance is produced by:

$\theta_{f}=\arcsin(\frac{x-x_{td}}{R}).$ (22)

##  4 Dynamic Inversion Auto Landing Structure

Consider the following formulation to represent the nonlinear dynamic system:

$\displaystyle\bm{\dot{x}}=\bm{f}(\bm{x})+\bm{g}(\bm{x})\bm{u},\
\bm{x}(0)=\bm{x}_{0},$ (23) $\displaystyle\bm{y}=\bm{h}(\bm{x}),$

where $\bm{x}\in\bm{R}$, $\bm{y}\in\bm{R}$, $\bm{u}\in\bm{R}$ are the vector
of state, measurement output, and control input. The goal of the problem is
often to design a suitable $\bm{u}$ such that $\bm{y}$ tracks desired
$\bm{y}_{des}$. By differentiating the output vector $\bm{y}$ with respect to
the state vector $\bm{x}$, this would be accomplished.

$\displaystyle\bm{\dot{y}}=\frac{\partial\bm{h}}{\partial\bm{x}}\frac{d\bm{x}}{dt}=\frac{\partial\bm{h}}{\partial\bm{x}}\bm{f}(\bm{x})+\frac{\partial\bm{h}}{\partial\bm{x}}\bm{g}(\bm{x})\bm{u}.$
(24)

The error between $\bm{y}$ and $\bm{y}_{des}$ must be eliminated by
establishing a first-order dynamic error, which is defined as
$\bm{e}_{y}=\bm{y}-\bm{y}_{des}$:

$\displaystyle\bm{\dot{e}}_{y}+\bm{K}\bm{e}_{y}=0,$ (25)
$\displaystyle\bm{\dot{y}}-\bm{\dot{y}}_{des}+\bm{K}(\bm{y}-\bm{y}_{des})=0.$

The error can be exponentially reduced by calculating the preceding equation
while assuming that $\bm{K}$ is a positive definite matrix. Let assume
$\bm{F_{y}}(\bm{x})=\frac{\partial\bm{h}}{\partial\bm{x}}\bm{f}(\bm{x})$,
$\bm{G_{y}}(\bm{x})=\frac{\partial\bm{h}}{\partial\bm{x}}\bm{g}(\bm{x})$ and
$\bm{g}(\bm{x})$ as an invertible matrix to derive the control signal as
follows:

$\displaystyle\bm{u}=[\bm{G}(\bm{x})]^{-1}[\bm{\dot{y}}-\bm{F_{y}}(\bm{x})].$
(26)

By substituting $\bm{\dot{y}}$ as a function of $\bm{e}$:

$\displaystyle\bm{u}=[\bm{G_{y}}(\bm{x})]^{-1}[\bm{\dot{y}}_{des}-\bm{K}(\bm{y}-\bm{y}_{des})-\bm{F_{y}}(\bm{x})].$
(27)

The main challenge is assigning $\bm{K}$ for error reduction. Extreme non-
linearity, couplings, modeling uncertainty, and aerodynamic uncertainty
combined with atmospheric disturbances lead to adaptive $\bm{K}$ designation
in this problem, which increases complexity. In this section, the Dynamic
Inversion controller was developed based on [37]. According to aforesaid
theories, first-order error in outer loop is considered
$\dot{e}_{h}+k_{h}e_{h}=0$, where $e_{h}=h-h_{des}$. So the desired path angle
is expressed as:

$\displaystyle\theta_{des}=\arcsin\Biggl{(}\frac{\dot{h}_{des}-k_{h}(h-h_{des})}{\sqrt{a_{h}^{2}+b_{h}^{2}}}\Biggl{)}-\arctan\biggl{(}\frac{b_{h}}{a_{h}}\biggl{)},$
(28)

where $a_{h}=u$, and $b_{h}=v\sin\phi+w\cos\phi$. Then for inner loop control,
the first-order error is used respectively. So;

$\displaystyle\dot{\theta}-\dot{\theta}_{des}+k_{\theta}(\theta-\theta_{des})=0.$
(29)

Obviously, forces and moments are applied to aircraft in body axis.

$\displaystyle
q_{des}=\sec\phi(\dot{\theta}_{des}-k_{\theta}(\theta-\theta_{des})+r\sin\phi).$
(30)

Now the control of $q$ in order to track $q_{des}$ is applicable. So the
first-order error is applied:

$\displaystyle\dot{q}-\dot{q}_{des}+k_{q}(q-q_{des})=0.$ (31)

By driving $\dot{q}$ in equation (1) and substituting equation (31), the
desired longitudinal control input $\delta_{E}$ is accessible.

$\displaystyle\delta_{E}=\frac{I_{y}(\dot{q}_{des}-k_{q}(q-q_{des}))+(I_{x}-I_{z})rp-
M_{A}-M_{\delta_{E}}\delta_{E}}{M_{\delta_{E}}},$ (32)

where $M_{\delta_{E}}=\bar{q}S\bar{c}c_{m_{\delta_{E}}}$.
Remark: It is apparent that the elevator deflection computation needs to
desired pitch angle and desired pitch rate states. But in the FQL method, it
will be seen later that the control signals are computed by just the desired
pitch angle state during the trajectory tracking phase.

##  5 Fuzzy Q-learning Auto-landing Structure

Because of their narrow Mean Aerodynamic Chord (MAC), TBW aircraft typically
exhibit insufficient longitudinal stability. More specifically, this finding
may be confirmed by evaluating the Phugoid and Short-period modes of the
Boeing N+3 TTBW [5] and Chaka 50 against the Cessna 172 [38]. Table (4)
contains an overview of the numerical data for the longitudinal modes of the
aforementioned transport aircraft. This table can support the claim made
above. To be clear, due to their superior manoeuvrability, the Chaka and
Boeing, which have similar designs, experience poor longitudinal stability
qualities. The Cessna 172, on the other hand, benefits from stable dynamics
behaviour. For clarification, the damping ratio of the Chaka 50’s Phugoid mode
causes low-stability behavior resulting in long-term oscillations as depicted
in [29].

Table 4: Longitudinal Dynamics Characteristic Aircraft Roots | Short Period Roots | Phugoid Roots
---|---|---
Chaka 50 | $-0.8\pm 0.61i$ | $-0.0064\pm 0.05i$
Cessna 172 | $-3.23\pm 5.71i$ | $-0.025\pm 0.19i$
Boeing N+3 | $-0.35\pm 0.35i$ | $-0.0082\pm 0.07i$

###  5.1 MDP Definition in Auto-landing

A sequential decision-making process, such as an auto-landing problem, must be
formalized as MDPs since one action affects not only the next state and its
immediate reward but also forthcoming states and their future rewards [39]. To
be clear, at each time-step $t$, the controller receives an state observation
from the 6-DoF simulation, including $\theta_{t}\in\bm{S_{1}}$ and
$\dot{\theta_{t}}\in\bm{S_{2}}$. Based on it, the controller specifies an
action, $\delta_{E_{t}}\in\bm{A}(\bm{s})$, which is the elevator deflection.
The simulation runs, and in the following time step $t+1$, the controller
receives a reward $R_{t+1}\in\bm{R}$ to assess its performance and find itself
in the next state $\theta_{t+1},\dot{\theta}_{t+1}$ until reaching to terminal
state $\theta_{T},\dot{\theta}_{T}$.

$\theta_{0},\ \dot{\theta}_{0},\ \delta_{E_{0}},\ R_{1},\ \theta_{1},\
\dot{\theta}_{1},\ \delta_{E_{1}},\ R_{2},\ ...\ \theta_{T},\
\dot{\theta}_{T}.$ (33)

Here, $\theta_{0}$, and $\dot{\theta}_{0}$ are the initial states, and
$\delta_{E_{0}}$ is the initial elevator deflection, moreover $R_{t}$ defines
the instant reward at time-step $t$. A random pick of $R_{t}$, $\theta_{t}$,
and $\dot{\theta}_{t}$ have a clear discrete probability distribution that is
solely reliant on the past state-action in this issue. Thus, by treating
$\theta_{t}$ and $\dot{\theta}_{t}$ as states, the Markov property is
satisfied because they contain complete information about all aspects of the
controller-aircraft (agent-environment) interaction history that matter in the
future. So, for all $\theta_{c}$, $\theta_{p}\in\bm{S}_{1}$,
$\dot{\theta}_{c}$, $\dot{\theta}_{p}\in\bm{S}_{2}$, and
$\delta_{E}\in\bm{A}(\bm{s})$, the equation of the MDP problem is defined as
$P$;

$\displaystyle P(\theta_{c},{\dot{\theta}}_{c}\ |\
\theta_{p},\dot{\theta}_{p},\delta_{E})=Pr\\{\theta_{t}=\theta_{c},\dot{\theta_{t}}={\dot{\theta}}_{c}\
|\
\theta_{t-1}=\theta_{p},\dot{\theta}_{t-1}={\dot{\theta}}_{p},\delta_{E_{t}}=\delta_{E}\\},$
(34)

where $c$ and $p$ are used for the current state and the previous state,
respectively. In this problem, according to table (5), equation (34) always
has a deterministic numerical amount in $[0\ 1]$ for all states. All states
are well-defined and each receives a distinct reward.

$\displaystyle{\sum}_{\theta_{c}\in\bm{S}_{1}}{\sum}_{\dot{\theta}_{c}\in\bm{S}_{2}}P(\theta_{c},{\dot{\theta}}_{c}\
|\ \theta_{p},\dot{\theta}_{p},\delta_{E})=1.$ (35)

The goal of finite MDP is to design a policy that maximizes reward over time.
To find an optimal policy for taking $\delta_{E}$ in state $\theta$,
$\dot{\theta}$, the state-action value function
$\bm{Q}\pi(\theta,\dot{\theta},\delta_{E})$ must be maximized, which is
defined as the expected return as the sum of discounted instant rewards by
starting from one specific state and pursuing policy $\pi$ to terminal state
$\theta_{T}$, $\dot{\theta}_{T}$:

$\displaystyle\bm{Q}\pi(\theta,\,\dot{\theta},\,\delta_{E})=\mathbb{E}_{\pi}\bigg{[}{\sum}_{k=0}^{\infty}\gamma^{k}R_{t+k+1}\
\bigg{|}\
\theta_{t}=\theta,\,\dot{\theta}_{t}=\dot{\theta},\,\delta_{E_{t}}=\delta_{E}\bigg{]},$
(36)

where $0<\gamma<1$ is the discount factor, whereas $\gamma\simeq 0$ denotes
the agent nearsightedness, and $\gamma\simeq 1$ means the agent is long-
sighted.

###  5.2 Structure of Fuzzy Q-learning Controller

In the current study, Q-learning, an early breakthrough in Reinforcement
Learning, is used to directly approximate the optimal elevator selection
policy in each condition [24]. Q-Learning is an off-policy, model-free control
algorithm based on the Temporal Difference (TD) method. In general, the system
state ($\theta_{t},\dot{\theta}_{t}$) is obtained each time step, and by using
the current elevator command ($\delta_{E_{t}}$) the action selection policy
will be updated. Because of the highly nonlinear and poor stability
characteristics of TBW aircraft, the Q-learning implementation results can be
unsatisfactory without employing continuous state-action [29]. Accordingly,
two Q-tables are trained by the Fuzzy and basic Q-learning methodologies.
Then, they are Incorporated in different auto-landing scenarios through a
novel technique namely the Fuzzy Action Assignment (FAA), which can be
introduced as a general connector between various trained Q-tables and
continuous environments. Instead of computing a discrete greedy action in a
particular state $\theta$, $\dot{\theta}$, FAA technique assigns a relative
weight (also defined as the validity function or membership function) to each
cell of the grid of system states (see Figure 3). This assignment is performed
based on the current value of the state-action value function. The membership
function of each grid cell with centers $\theta_{i}$ and $\dot{\theta}_{j}$ is
described as follows:

$\displaystyle
MF_{i,j}=\exp\left(-\frac{1}{2}\left(\frac{\theta_{t}-\theta_{i_{t}}}{\sigma_{\theta}}\right)^{2}\right)\exp\left(-\frac{1}{2}\left(\frac{q_{t}-\dot{\theta}_{j_{t}}}{\sigma_{\dot{\theta}}}\right)^{2}\right),$
(37)

where $\sigma_{\theta}$, and $\sigma_{\dot{\theta}}$ define the validity
widths of the membership functions. In this study, the elevator commands of
TBW aircraft are specified into $-0.25$ to $+0.25$ radians with $0.025$
intervals, corresponding to 21 elevator deflections. Also, the
$\epsilon$-greedy action selection strategy with epsilon decay is utilized in
this research so as to select greedy elevator commands in the last episodes
(when the trained policy is near-optimum).

$\delta_{E_{t}}=\begin{cases}{\arg\max}_{\delta_{E}}\
\bm{Q}(\theta_{t},\dot{\theta}_{t},\delta_{E})&\text{with probability
$1-\epsilon$}\\\ \text{random action}&\text{with probability $\epsilon$}\\\
\end{cases}$ (38)

Following that, $\delta_{E}$ is determined using a weighted average of
neighbor membership functions at each time step as regards:

$\displaystyle\delta_{E_{t}}=\frac{\sum_{i}\sum_{j}MF_{{i,j}_{t}}\
{\arg\max}_{\delta_{E}}\,\bm{Q}(\theta_{i_{t}},\dot{\theta}_{j_{t}},\delta_{E})}{\sum_{i}\sum_{j}MF_{{i,j}_{t}}}.$
(39)

The computed elevator deflection is applied to the aircraft 6-DoF simulation
environment and receives a scalar reward signal as performance feedback which
is defined in the Reward Function Definition section.

Figure 3: Grid of state variables used for tabular Q-learning (The center of
each cell, which is used to compute the membership function of the cell is
shown by a circle point.)

####  5.2.1 State-action Value Function Updating Rule

In this study, the state-action value function directly estimates the optimal
Q-table controller which is performed over episodes utilizing Fuzzy rules. The
base updating rule equation is unchanged but the calculation of its different
parts is improved dramatically. So in this part, the updating formalization of
the terms of updating function is discussed. In this case, $Q_{f}$ defines as
the value of state-action in the previous time step that is computed with its
neighbor grids as:

$\displaystyle{Q}_{f}(\theta_{t},\dot{\theta_{t}},\delta_{E_{t}})=\frac{\sum_{i}\sum_{j}MF_{{i,j}_{t}}\,{Q}(\theta_{i_{t}},\dot{\theta}_{j_{t}},\delta_{E})}{\sum_{i}\sum_{j}MF_{{i,j}_{t}}}.$
(40)

Then, estimating the optimal state-action value of the next time step using a
Fuzzy scheme is important in order to, find an estimation of a sequence of
best elevator deflection values that will be selected from that state until
the terminal state. But note that the quota of neighbor elevators is seen
using the membership function. The reason for this computation is obvious. The
effect of one specific $\delta_{E}$ is not only on one pair of $\theta$,
$\dot{\theta}$ but also has a nearly similar effect on their near neighbors.
So the Fuzzy optimal future value is as follows;

$\displaystyle\max_{\delta_{E}}{Q_{f}}(\theta_{t+1},\dot{\theta}_{t+1},\delta_{E})=\frac{\sum_{i}\sum_{j}MF_{{i,j}_{t+1}}\
{\max}_{\delta_{E}}\,{Q}(\theta_{i_{t+1}},\dot{\theta}_{j_{t+1}},\delta_{E})}{\sum_{i}\sum_{j}MF_{{i,j}_{t+1}}},$
(41)

where the membership function is related to the next time-step. Another step
is to calculate the Temporal Difference (TD) in Fuzzy format. Therefore, the
TD formulation is as follows;

$TD_{{i,j}_{t}}=\frac{\sum_{i}\sum_{j}MF_{{i,j}_{t}}\
\bigg{[}R_{t+1}+\gamma\,\underset{\delta_{E}}{max}\,{Q}_{f}(\theta_{t+1},\dot{\theta}_{t+1},\delta_{E})-{Q}_{f}(\theta_{t},\dot{\theta_{t}},\delta_{E_{t}})\bigg{]}}{\sum_{i}\sum_{j}MF_{t}},$
(42)

where the membership function in this step is related to the previous time-
step. The pseudocode of the proposed control strategy is summarized in
Algorithm 1.

####  5.2.2 Fuzzy Action Assignment Structure

As discussed earlier, the FAA receives optimal policies produced by various
continuous or discrete RL algorithms. Then generates continuous actions for
different control problems based on Fuzzy methodologies. Its implication are
easy-going using just two equations 37, and 39. The pseudocode of FAA is
explained in algorithm 2.

####  5.2.3 Reward Function Definition

The definition of an appropriate reward function is critical to the learning
processes’ convergence. Therefore, the reward function design and hyper-
parameter adjustment have received significant attention in this study
likewise. In this manner, the reward function is generated in three phases and
comprises plant states such as $\theta,q,\delta_{E}$.

To begin with, in order to limit the elevator’s high-frequency deflecting,
severe penalties in the case of aggressive elevator selection are required.
This penalty is performed whenever an elevator changes more than 0.1 radians.

$R_{t}=-10000,\quad\textrm{If}\
\left(|\delta_{E_{t}}|-|\delta_{E_{t-1}}|\right)>5.73^{\circ}.$ (43)

The reward function will then be calculated as follows if the aircraft is
close to the desired angle and the elevator operation frequency is reasonable.

$\displaystyle R_{t}=$ $\displaystyle(300,\qquad\textrm{If}\
|e_{\theta_{t}}|<0.05^{\circ}),$ (44) $\displaystyle+$
$\displaystyle(300,\qquad\textrm{If}\ |e_{\theta_{t}}|<0.02^{\circ}),\ $
$\displaystyle+$ $\displaystyle(400,\qquad\textrm{If}\ |q_{t}|<0.04^{\circ}),\
$ $\displaystyle+$ $\displaystyle(600,\qquad\textrm{If}\
|q_{t}|<0.02^{\circ}),\ $ $\displaystyle+$
$\displaystyle(800,\qquad\textrm{If}\ |q_{t}|<0.005^{\circ}),\ $

where $e_{\theta_{t}}=\theta_{t}-\theta_{des}$ is the proportional error. This
definition first examines the state of pitch tracking. The controller then
detects and prioritizes fewer pitch rates using higher reward allocations in
the last episodes. The concepts stated above were defined for learning
convergence. In other words, they are activated when the pitch angles are near
the desired values. However, it is essential to guide the learning process in
early episodes with another phrase. As a result, if none of the above two
requirements are satisfied, we should urge the air vehicle to proceed at the
desired angle. Using the following reward function, this demand is achievable:

$R_{t}=-(100\times|e_{\theta_{t}}|)^{2}-(40\times|q_{t}|)^{2}.$ (45)

As a result, the further the system deviates from the desired state, the lower
the reward. In addition, to avoid excessive pitch rates, a derivative term
(the second term) has been added into reward function. More specifically, the
presence of the pitch rate ($q_{t}$) in equation (44), as well as its weight
in equation (45), influences the convergence rate significantly.

Data: Learning Rate $\alpha$, Discount Factor $\gamma$, Desired Angle
$\theta_{des}=1$deg, Validity Widths $\sigma_{\theta},\sigma_{\dot{\theta}}$,
Elevator Deflections $\delta_{E}$, Pitch (Rate) Angle Intervals $\theta$,
$\dot{\theta}$.

Result: $\bm{Q}_{\pi^{*}}(\theta,\dot{\theta},\delta_{E})$

1 $\bm{Q}(\theta_{0},\dot{\theta}_{0},\delta_{E_{0}})\leftarrow 0$; for all$\
\theta\in{\bm{S}}_{1},\dot{\theta}\in{\bm{S}}_{2},\delta_{E}\in\bm{A}(\bm{s})$;

2 for  _Episode Number = 1 to 20000_ do

3 Initialize 6-DoF simulation with a random $\theta_{0}\in[0\ 2]$ deg.

4 for  _time-step (0.01) = 0 to 5 sec_ do

5 if  _$\epsilon <$ random number $\in[0\ 1]$_ then

6 $\delta_{E_{t}}\leftarrow random\ \delta_{E}\in\bm{A}(\bm{s})$;

7

8 else

9 for  _i,j = 1 to length I,J_ do

10 $MF_{i,j_{t}}\leftarrow$ Compute membership function eq 37;

11 $\delta_{E_{t}}\leftarrow$ Compute $\delta_{E_{t}}$ eq 39;

12

13 end for

14

15 end if

16 Execute 6-DoF simulation using computed $\delta_{E_{t}}$, observe
$R_{t+1}$, $\theta_{t+1}$, $\dot{\theta}_{t+1}$;

17 for  _i,j = 1 to length I,J_ do

18 $MF_{{i,j}_{t+1}}\leftarrow$ Compute membership function eq 37;

19
${Q_{f}}(\theta_{t},\dot{\theta}_{t},\delta_{E_{t}})\leftarrow{Q}(\theta_{t},\dot{\theta}_{t},\delta_{E_{t}})$
Computeed by eq 40;

20
$\max_{\delta_{E}}{Q_{f}}(\theta_{t+1},\dot{\theta}_{t+1},\delta_{E})\leftarrow\max_{\delta_{E}}{Q}(\theta_{t+1},\dot{\theta}_{t+1},\delta_{E})$
Computeed by eq 41;

21
${Q}(\theta_{i_{t}},\dot{\theta}_{j_{t}},\delta_{E_{t}})\leftarrow{Q}(\theta_{i_{t}},\dot{\theta}_{j_{t}},\delta_{E_{t}})+\alpha\bigg{[}TD_{i,j_{t}}\bigg{]}$
Based on eq 42;

22

23 end for

24 Substitute simulation parameters in time-step $t$ with $t+1$.

25 end for

26

27 end for

Return $\bm{Q}{\pi^{*}}(\theta,\dot{\theta},\delta_{E})$

Algorithm 1 Fuzzy Q-learning Aircraft Attitude Controller

Data: $\bm{Q}{\pi^{*}}(\theta,\dot{\theta},\delta_{E})$

Result: Contineuous Elevator Deflection $\delta_{E}$

1 Initialize 6-DoF simulation using predefined initial conditions;

2 for  _time-step (0.01) = 0 to 5 sec_ do

3 for  _i,j = 1 to length I,J_ do

4 $MF_{i,j_{t}}\leftarrow$ Compute membership function eq 37;

5 $\delta_{E_{t}}\leftarrow$ Compute $\delta_{E_{t}}$ eq 39;

6

7 end for

8 Execute 6-DoF simulation using computed $\delta_{E_{t}}$, compute the next
system states;

9 Substitute simulation parameters in time-step $t$ with $t+1$.

10 end for

Algorithm 2 Fuzzy Action Assignment Scheme

##  6 Simulation Results and Discussion

The auto-landing control of an innovative TBW regional jet aircraft is
developed utilizing Fuzzy Q-learning (FQL). The development process and the
justification of the findings will therefore be touched upon in this section.
The procedure is divided into two segments, as will be detailed later. The
first is the learning phase, and the second is the trajectory tracking phase,
which is the execution of the optimal policy. The proposed approach is then
compared with Dynamic Inversion, a well-known robust control method according
to studies. A novel continuous action selection method is developed in this
study to function as a useful link between every trained Q-table and
environment, as shown in figure 4.

In the learning phase, the desired pitch angle is set to 1 degree, and the
initial pitch angle is selected as a random number between 0 and 2 degrees
during episodes. The observation vector contains the pitch angle $\theta$, and
pitch rate $(q)$ in each time step. Furthermore, the pitch and pitch angle
rate state intervals are defined as a 3D table alongside with action vector
(Elevator deflection intervals) which are located in the policy block. This
block selects an action based on the $\epsilon$-greedy strategy. The FQL core
receives the reward, observation vector, and the relevant states of the pitch
angle and pitch rate from the policy block (Figure 4) and updates the policy
based on the aforementioned algorithm 1. Also, Q-learning receives all
mentioned signals except the observation vector for policy updating.

The second phase is specified for the auto-landing control. In this case, the
path planning block generates desired pitch angle in each time step. Then, one
of the controllers out of three generates elevator signals. The output of the
Fuzzy Q-table and Q-table is sent to the FAA block to produce continuous
action (elevator deflections) based on algorithm 2. Several scenarios are
defined in this stage including actuator faults, model uncertainties, and
atmospheric disturbance plus sensor measurement noise. The parameters required
for simulation are gathered in table 5. The positive part of pitch angle and
pitch rate intervals are not mentioned in this table because of considering
similar intervals. Useful to mention that all steps are developed in MATLAB
R2022a and the PC is characterized by an 8 cores processor with 2.30 GHz, and
8 GiB RAM.

Figure 4: Block diagram of learning and trajectory tracking phases of all 3
methods

The learning results of the two methods are shown in figure 5. The main
difference between MDP and POMDP models is observing the pitch rate in the MDP
problem definition which in the POMDP is omitted. Obviously, the FQL conquered
in comparison with Q-learning, and this insignificant difference will prove
the robustness of the FQL later. Furthermore, its fluctuations are less than
Q-learning (MDP). The POMDP policy is not included in the trajectory tracking
phase owing to its unsuccessful findings [29]. The results of attitude
tracking are gathered in figure 6. The first row is specified for ideal flight
conditions. All three methods were prosperous to track the desired angle. But
the main difference is related to attitude tracking errors ($TE_{\theta}$),
altitude tracking errors ($TE_{h}$), and control effort ($CE$) which are
defined as follows;

$TE_{\theta}=\frac{\int_{0}^{t}|\theta_{t}-\theta_{des}|dt}{t},$ (46)
$TE_{h}=\frac{\int_{0}^{t}|h_{t}-h_{des}|dt}{t},$ (47)
$CE=\frac{\int_{0}^{t}|\delta_{E_{t}}|dt}{t}.$ (48)

The second row contains simulation results in atmospheric disturbance and
sensor measurement noise. The DI and FQL are able to track tough high-
frequency responses. Although the path-planning block tried to generate less
desired pith angles to guide the aircraft to the desired trajectory, the QL
diverged. The third row includes the results of actuator faults and finally,
the last row gathers the results of model parameter uncertainties which the
performance of all controllers are reasonable.

Figure 5: Learning result of Q-learning and Fuzzy Q-learning during $20000$ Episodes Table 5: Controller parameters in learning and tracking phases Parameter | Definition | Value
---|---|---
Epsilon($\epsilon$) | Exploration Probability | $[0.1:{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}3e{-6}}:0.04]$
Alpha($\alpha$) | Learning Rate | $[0.02:{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}9e{-7}}:0.002]$
Gamma($\gamma$) | Discount Factor | $0.99$
Episode number | - | $20000$
$\theta$(rad) | Pitch Angle Intervals | $[-10,-0.024:{\color[rgb]{0,0,1}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,1}0.002}:-0.002,-0.001,0]$
$\dot{\theta}$(rad) | Pitch Rate Intervals | $[-10,-0.04,-0.02,-0.005]$
I | Adjacent pitch grids | $[i-2:i+2]$
J | Adjacent pitch rate grids | $[j-2:j+2]$
$k_{h}$ | Altitude Control Coefficient | 1.3
$k_{\theta}$ | Pitch Control Coefficient | 5
$k_{q}$ | Pitch Rate Control Coefficient | 10
Figure 6: Attitude tracking of all 3 methods in (1) Ideal flight conditions
(2) Atmospheric disturbance and sensor measurement noise (3) Actuator faults
(4) Model parameters uncertainties.

According to figure 7, the elevator deflection of all three methods explains
significant data. To clarify, the first subplot indicates a larger initial
deflection for DI in ideal flying conditions than the others. Furthermore, the
second subplot demonstrates the poor performance of DI in the face of sensor
noise and atmospheric disturbances. This result may be successful in
trajectory tracking but is unable to be applicable. The third subplot is drawn
for faulty elevator deflections with conspicuous jumps in all schemes.
Finally, in contrast to the first subplot, the last subplot yields roughly
analogous findings. It is useful to note that the DI overshoot before 10
seconds demonstrates its sensitivity to the commencing flare phase.

Figure 7: Computed elevator deflections of all 3 methods in different flight
conditions

The next figure 8 demonstrates the altitude tracking of all three controllers.
In an overview, the first, third, and last subplots prove the robustness of
all three methodologies theoretically. But the second subplot is an exception.
The QL controller is unable to accomplish its task properly but the FQL and DI
have reasonable findings although the performance of DI is not useful for real
applications. However, it is noticeable that the altitude tracking error of
FQL in the middle of the approach distance is larger than DI. But during the
flare phase that is more critical, the story changes, where the FQL surpasses
DI.

Figure 8: Altitude tracking of all 3 methods in various scenarios

The angle of attack (AoA) results of 15 seconds flight simulations are shown
in figure 9. The less magnitude of AoA changes is clear in the first subplot
compared to others. After the first subplot, the actuator faults have less
influence on AoA although, there are some little jumps. The second subplot
demonstrates an AoA change between -1 to 1 degree which is more than others
despite its noisy specificity. This can amplify the drag where its
consequences are clear in the next figure. Generally, the effect of actuator
faults on AoA is less than model uncertainties.

Figure 9: Aircraft Angle of Attack results in different flight simulations

The last figure depicts the speed of the airplane along its longitudinal body
axis. According to the path planning section, the speed of aircraft at the
touch-down point should be $161\frac{m}{sec}$. The simulation of all flight
conditions yields satisfactory results, however, the second subplot reveals a
problem with DI. The performance of this controller leads the aircraft speed
to stall margin, and the authors attribute this occurrence to high drag
production caused by the elevator.

Figure 10: Simulated longitudinal speed results during various scenarios

A summary of results is gathered numerically in table 6 for better
comprehension. At a first glance, the superiority of FQL is noticeable in
attitude tracking of all flight conditions except in presence of noise and
disturbance where DI depicts better findings apparently. But the performance
of DI in this circumstance is called into question owing to its control
effort. More precisely about ideal conditions, the altitude tracking errors of
QL and FQL are slightly better than DI, unlike their control effort. However,
the differences are insignificant. As previously stated in the second
scenario, despite DI superiority in pitch and altitude tracking errors,
elevator deflection is saturated drastically. In the simulated flight
influenced by actuator faults, the FQL controller is the forerunner. In this
scenario, QL performance is inferior to that of other controllers, yet all
controllers are successful. The final simulated flight includes model
parameter uncertainties, and the FQL attitude tracking error is better than
the other two. Although altitude tracking error and control effort of DI are
superior, the differences are insignificant.

Table 6: Attitude tracking error, altitude tracking error, and control effort
of three methods in different flight conditions.

Flight Condition | Control Method |  | Attitude Tracking
---
Error (deg)
| Altitude Tracking
---
Error (m)
| Control Effort
---
(deg)
| Dynamic Inv | 0.057 | 0.662 | 0.593
Ideal | Q-learning | 0.047 | 0.643 | 0.649
| Fuzzy QL | 0.040 | 0.644 | 0.669
| Dynamic Inv | 2.541 | 1.477 | 13.772
Noise + Disturbance | Q-learning | 11.178 | 21.655 | 1.998
| Fuzzy QL | 2.629 | 2.17 | 3.954
| Dynamic Inv | 0.075 | 0.717 | 0.592
Actuator Fault | Q-learning | 0.091 | 0.755 | 0.624
| Fuzzy QL | 0.064 | 0.708 | 0.654
| Dynamic Inv | 0.060 | 0.903 | 0.734
Model Uncertainty | Q-learning | 0.056 | 0.936 | 0.793
| Fuzzy QL | 0.042 | 0.928 | 0.812

Another discussion is around the working area of the proposed FQL controller
to prove its robustness. Although the previous simulations were in different
flight conditions, they were designed just around one working point. More
precisely, they were simulated in initial aircraft longitudinal speed
$160\frac{m}{s}$, and constant aircraft longitudinal aerodynamic and control
coefficients. In this part, the performance of both FQL and DI controllers are
examined in a wider alteration of aforesaid coefficients and aircraft speed.
In this case, the coefficients are varied between $\pm 30\%$, and also the
aircraft speed initializes between $150$ to $220\frac{m}{s}$. The FQL findings
were satisfactory in comparison with one of the well-known robust controllers
namely Dynamic Inversion. According to figure 10, the FQL pitch angle tracking
error varies between $0.04$ to $0.07$ degrees during $81$ defined initial
conditions. Correspondingly, the control effort related to this controller
varies between 0.64 to 2.6 degrees. On the other hand, the tracking error of
DI computed between 0.05 to 0.07 degrees where the elevator control effort is
varied from 0.7 to 2,6 degrees.

Figure 11: Fuzzy Q-learning and Dynamic Inversion tracking error and control
effort examined by alteration of model parameters, and aircraft speeds

##  7 Conclusion

In this research, auto-landing control of a regional jet aircraft with a novel
configuration and specific longitudinal dynamic stability characteristics was
addressed using Fuzzy Q-learning. The robustness property of this method was
evaluated in several probable scenarios including atmospheric disturbances,
sensor measurement noise, actuator faults, and model parameter uncertainties.
The simulation results illustrated comparable improvements in contrast to
Dynamic Inversion and classic Q-learning controllers. An innovative continuous
action generator was proposed in this research to be a connector between
optimal Q-tables and RL environments. In order to depict the robustness and
working area of the proposed method, the aircraft’s longitudinal speed, and
coefficients have been altered widely, and the pitch angle tracking error as
well as control effort are reported numerically. Summing up, the competency of
the Fuzzy Q-learning method was proved in this problem approaching different
flight conditions without being stuck in complicated Artificial Neural Network
architectures.

## References

  * [1] Li Li, Junqiang Bai, and Feng Qu.  Multipoint aerodynamic shape optimization of a truss-braced-wing aircraft.  Journal of Aircraft, pages 1–16, 2022.
  * [2] Martin Sohst, José Lobo do Vale, Frederico Afonso, and Afzal Suleman.  Optimization and comparison of strut-braced and high aspect ratio wing aircraft configurations including flutter analysis with geometric non-linearities.  Aerospace Science and Technology, 124:107531, 2022.
  * [3] Timothy Chau and David W Zingg.  Aerodynamic design optimization of a transonic strut-braced-wing regional aircraft.  Journal of Aircraft, 59(1):253–271, 2022.
  * [4] S. Zavaree, M. Zahmatkesh, K. Eghbali, K. Zahiremami, E. Vaezi, s.A. Madani, A. Kariman, Z. Heidari, A. Mahmoudi, F. Rassouli, et al.  Modern regional jet family (chaka: A high- performance, cost-efficient, semi-conventional regional jet family), 2021.
  * [5] Nhan T Nguyen and Juntao Xiong.  Dynamic aeroelastic flight dynamic modeling of mach 0.745 transonic truss-braced wing.  In AIAA SCITECH 2022 Forum, page 1325, 2022.
  * [6] Arthur Suharev, Vladimir Shestakov, and Konrad Stefański.  Analysis of the affecting factors on aircraft takeoff and landing ground path length.  2019\.
  * [7] Batuhan Eroglu, M. Cagatay Sahin, and Nazim Kemal Ure.  Autolanding control system design with deep learning based fault estimation.  Aerospace Science and Technology, 102:105855, 2020.
  * [8] Mihai Lungu.  Backstepping and dynamic inversion control techniques for automatic landing of fixed wing unmanned aerial vehicles.  Aerospace Science and Technology, 120:107261, 2022.
  * [9] Shaik Ismail, Abhay A. Pashilkar, Ramakalyan Ayyagari, and Narasimhan Sundararajan.  Improved neural-aided sliding mode controller for autolanding under actuator failures and severe winds.  Aerospace Science and Technology, 33(1):55–64, 2014.
  * [10] Zhuang Wang, Hui Li, Haolin Wu, Feng Shen, and Ruixuan Lu.  Design of agent training environment for aircraft landing guidance based on deep reinforcement learning.  In 2018 11th International Symposium on Computational Intelligence and Design (ISCID), volume 02, pages 76–79, 2018.
  * [11] Xin Yuan, Yuewen Sun, Yuanda Wang, and Changyin Sun.  Deterministic policy gradient with advantage function for fixed wing uav automatic landing.  In 2019 Chinese Control Conference (CCC), pages 8305–8310, 2019\.
  * [12] Chi Tang and Ying-Chih Lai.  Deep reinforcement learning automatic landing control of fixed-wing aircraft using deep deterministic policy gradient.  In 2020 International Conference on Unmanned Aircraft Systems (ICUAS), pages 1–9, 2020.
  * [13] Fang Liao, Jian Liang Wang, Eng Kee Poh, and Dong Li.  Fault-tolerant robust automatic landing control design.  Journal of guidance, control, and dynamics, 28(5):854–871, 2005\.
  * [14] A.A. Pashilkar, N. Sundararajan, and P. Saratchandran.  A fault-tolerant neural aided controller for aircraft auto-landing.  Aerospace Science and Technology, 10(1):49–61, 2006.
  * [15] George Zogopoulos-Papaliakos, George C Karras, and Kostas J Kyriakopoulos.  A fault-tolerant control scheme for fixed-wing uavs with flight envelope awareness.  Journal of Intelligent & Robotic Systems, 102(2):1–33, 2021.
  * [16] Salman Ijaz, Lin Yan, Mirza Tariq Hamayun, and Cun Shi.  Active fault tolerant control scheme for aircraft with dissimilar redundant actuation system subject to hydraulic failure.  Journal of the Franklin Institute, 356(3):1302–1332, 2019.
  * [17] Kijoon Kim, Seungkeun Kim, Jinyoung Suk, Jongmin Ahn, Nakwan Kim, and Byoung-Soo Kim.  Flight test of flying-wing type unmanned aerial vehicle with partial wing-loss.  Proceedings of the Institution of Mechanical Engineers, Part G: Journal of Aerospace Engineering, 233(5):1611–1628, 2019.
  * [18] Mikkel Eske Nørgaard Sørensen, Søren Hansen, Morten Breivik, and Mogens Blanke.  Performance comparison of controllers with fault-dependent control allocation for uavs.  Journal of Intelligent & Robotic Systems, 87(1):187–207, 2017\.
  * [19] Xiaowei Yang, Xixiang Yang, and Xiaolong Deng.  Horizontal trajectory control of stratospheric airships in wind field using q-learning algorithm.  Aerospace Science and Technology, 106:106100, 2020.
  * [20] Zeyi Xi, Di Wu, Wenjun Ni, and Xiaoping Ma.  Energy-optimized trajectory planning for solar-powered aircraft in a wind field using reinforcement learning.  IEEE Access, 10:87715–87732, 2022.
  * [21] Eivind Bøhn, Erlend M Coates, Dirk Reinhardt, and Tor Arne Johansen.  Data-efficient deep reinforcement learning for attitude control of fixed-wing uavs: Field experiments.  arXiv preprint arXiv:2111.04153, 2021.
  * [22] Weijun Hu, Zhiqiang Gao, Jiale Quan, Xianlong Ma, Jingyi Xiong, and Weijie Zhang.  Fixed-wing stalled maneuver control technology based on deep reinforcement learning.  In 2022 IEEE 5th International Conference on Big Data and Artificial Intelligence (BDAI), pages 19–25. IEEE, 2022.
  * [23] David J. Richter, Lance Natonski, Xiaxin Shen, and Ricardo A. Calix.  Attitude control for fixed-wing aircraft using q-learning.  In Jong-Hoon Kim, Madhusudan Singh, Javed Khan, Uma Shanker Tiwary, Marigankar Sur, and Dhananjay Singh, editors, Intelligent Human Computer Interaction, pages 647–658, Cham, 2022. Springer International Publishing.
  * [24] Christopher JCH Watkins and Peter Dayan.  Q-learning.  Machine learning, 8(3):279–292, 1992.
  * [25] P.Y. Glorennec and L. Jouffe.  Fuzzy q-learning.  In Proceedings of 6th International Fuzzy Systems Conference, volume 2, pages 659–662 vol.2, 1997.
  * [26] Arga Dwi Pambudi, Trihastuti Agustinah, and Rusdhianto Effendi.  Reinforcement point and fuzzy input design of fuzzy q-learning for mobile robot navigation system.  In 2019 International Conference of Artificial Intelligence and Information Technology (ICAIIT), pages 186–191, 2019.
  * [27] Meng Joo Er and Chang Deng.  Online tuning of fuzzy inference systems using dynamic fuzzy q-learning.  IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 34(3):1478–1489, 2004.
  * [28] Vu Phi Tran, Mohamed A. Mabrok, Sreenatha G. Anavatti, Matthew A. Garratt, and Ian R. Petersen.  Robust fuzzy q-learning-based strictly negative imaginary tracking controllers for the uncertain quadrotor systems.  IEEE Transactions on Cybernetics, pages 1–13, 2022.
  * [29] Mohsen Zahmatkesh, Seyyed Ali Emami, Afshin Banazadeh, and Paolo Castaldi.  Robust attitude control of an agile aircraft using improved q-learning.  Actuators, 11(12), 2022.
  * [30] Marcello R Napolitano.  Aircraft Dynamics.  Wiley, 2012.
  * [31] PH Zipfel.  Modeling and simulation of aerospace vehicle dynamics–third edition, 2014\.
  * [32] Andrew Wood, Ali Sydney, Peter Chin, Bishal Thapa, and Ryan Ross.  Gymfg: A framework with a gym interface for flightgear.  arXiv preprint arXiv:2004.12481, 2020.
  * [33] Jan Roskam.  Airplane flight dynamics and automatic flight controls.  DARcorporation, 1998.
  * [34] VA Mil-f.  8785c: Flying qualities of piloted airplanes.  US Air Force, 5, 1980.
  * [35] Walter Frost and Roland L Bowles.  Wind shear terms in the equations of aircraft motion.  Journal of Aircraft, 21(11):866–872, 1984.
  * [36] John Anderson.  Aircraft Performance and Design.  WCB/McGraw-Hill, 1 edition, 1999.
  * [37] Pradeep R. Ambati and Radhakant Padhi.  Robust auto-landing of fixed-wing uavs using neuro-adaptive design.  Control Engineering Practice, 60:218–232, 2017.
  * [38] Ender Çetin.  System identification and control of a fixed wing aircraft by using flight data obtained from x-plane flight simulator.  Master’s thesis, Middle East Technical University, 2018.
  * [39] Richard S Sutton and Andrew G Barto.  Reinforcement learning: An introduction.  MIT press, 2018.