# The Langevin approach for fission of heavy and super-heavy nuclei††thanks:
Presented at the Zakopane Conference on Nuclear Physics, Zakopane, Poland, 28
August - 4 September 2022

F.A.Ivanyuk  S.V.Radionov
C.Ishizuka, S.Chiba  Institute for Nuclear Research, Kyiv, Ukraine  Tokyo
Institute of Technology, Tokyo, Japan

###### Abstract

In this contribution, we present the main relations of the Langevin approach
to the description of fission or fusion-fission reactions. The results of
Langevin calculations are shown for the mass distributions of fission
fragments of super-heavy elements and used for the investigation of memory
effects in nuclear fission.

##  1 Introduction

We describe the nuclear fission process by the four-dimensional set of the
Langevin equations for the shape degrees of freedom with the shape given by
the two-center shell model (TCSM) shape parametrization. The potential energy
is calculated within the macroscopic-microscopic method. The collective mass,
$M$, and friction, $\gamma$, tensors are defined in macroscopic (Werner-
Wheller and wall-and-window formula) or microscopic (linear response theory)
approaches.

We start calculations from the ground state shape with zero collective
velocities and solve equations until the neck radius of the nucleus turns zero
(scission point). At the scission point, the solutions of Langevin equations
supply complete information about the system, its shape, excitation energy,
and collective velocities. This information makes it possible to calculate the
mass distributions, the total kinetic energy, and the excitation energies of
fission fragments. The results of numerous previous calculations are in
reasonable agreement with the available experimental data.

Below in this contribution, we present the calculated results for the mass
distributions of super-heavy nuclei and clarify the impact of memory effects
on the fission width of heavy nuclei.

The physics of super-heavy elements (SHE) has a long history. The existence of
the “island of stability” was predicted at the end of the 1960s [1].
Nevertheless, it took almost 30 years until the alpha-decay of the element
with Z=114 was observed experimentally at Flerov Nuclear Reactions Laboratory
in Dubna [2].

With the development of experimental facility, it became possible not only to
fix the fact of formation of SHE, but examine their properties. One of the
first property of interest – the process of fission of SHEs. For the
successful planning and carrying out of experiments, it is crucial to
understand what kind of fission fragments mass distribution (FFMD) one should
expect in the result of the fission of SHEs. The two double magic nuclei 132Sn
and 208Pb may contribute. Both have the shell correction in the ground state
of the same magnitude.

In order to clarify what kind of FFMD one could expect in the fission of SHEs,
we have carried out the calculations of FFMD for a number of SHEs. The results
are given in Section 3.

Another problem we address in this contribution is the influence of memory
effects on the probability of the fission process. Commonly one uses the
Markovian approximation to Langevin approach in which all quantities are
defined at the same moment. This approximation provides reasonable results,
but its accuracy is not well established. In publications, one can find
statements that the memory effects have a significant influence on the fusion
or fission processes and the statements that memory effects are very small.

To clarify this uncertainty, we have calculated the fission width using the
Langevin approach with memory effects included in a wide range of important
parameters: the excitation energy $E^{*}$ of the system, the damping parameter
$\eta$, the relaxation time $\tau$. The details and results of the
calculations are given in Section 4.

##  2 The Langevin approach for the fission process

Within the Langevin approach, the fission process is described by solving the
equations for the time evolution of the shape of nuclear surface of the
fissioning system. For the shape parametrization, we use that of the two-
center shell model (TCSM) [3] with 4 deformation parameters
$q_{\mu}=z_{0}/R_{0},\delta_{1},\delta_{2},\alpha$. Here z${}_{0}/R_{0}$
refers to the distance between the centers of left and right oscillator
potentials, $R_{0}$ being the radius of spherical nucleus with the mass number
A. The parameters $\delta_{i}$ describe the deformation of the right and left
fragment tips. The fourth parameter $\alpha$ is the mass asymmetry and the
fifth parameter of the TCSM shape parametrization $\epsilon$ was kept
constant, $\epsilon$=0.35, in all our calculations.

The first-order differential equations (Langevin equations) for the time
dependence of collective variables $q_{\mu}$ and the conjugated momenta
$p_{\mu}$ are:

$\displaystyle\frac{dq_{\mu}}{dt}$ $\displaystyle=$
$\displaystyle\left(m^{-1}\right)_{\mu\nu}p_{\nu},$ (1)
$\displaystyle\frac{dp_{\mu}}{dt}$ $\displaystyle=$
$\displaystyle-\frac{\partial F(q,T)}{\partial
q_{\mu}}-\frac{1}{2}\frac{\partial m^{-1}_{\nu\sigma}}{\partial
q_{\mu}}p_{\nu}p_{\sigma}-\gamma_{\mu\nu}m^{-1}_{\nu\sigma}p_{\sigma}+R_{\mu}(t).$

In Eqs. (1) the $F(q,T)$ is the temperature-dependent free energy of the
system, and $\gamma_{\mu\nu}$ and (m${}^{-1})_{\mu\nu}$ are the friction and
inverse of mass tensors.

The free energy $F(q,T)$ is calculated within the shell correction method. The
single particle energies are calculated with the deformed Woods-Saxon
potential fitted to the mentioned above TCSM shapes.

The collective inertia tensor $m_{\mu\nu}$ is calculated by the Werner-Wheeler
approximation and for the friction tensor $\gamma_{\mu\nu}$ we used the wall-
and-window formula. The random force $R_{\mu}(t)$ is the product of the
temperature-dependent strength factors gμν and the white noise $\xi_{\nu}$(t),
$R_{\mu}(t)=g_{\mu\nu}\xi_{\nu}$(t). The factors gμν are related to the
temperature and friction tensor via the Einstein relation,

$g_{\mu\sigma}g_{\sigma\nu}=T\gamma_{\mu\nu}$ (2)

The temperature T is kept constant, $aT^{2}=E^{*}$, or adjusted to the local
excitation energy on each step of integration by the relation,

$aT^{2}=E^{*}-p^{2}(t)/2M-[E_{pot}(q)-E_{pot}(q_{gs})].$ (3)

Here $q_{gs}$ is the ground state deformation. More details are given in our
earlier publications [4, 5, 6, 7].

Initially, the momenta $p_{\mu}$ are set to zero, and calculations are started
from the ground state deformation. Such calculations are continued until the
trajectories reach the ”scission point”, defined as the point in deformation
space where the neck radius turns zero.

##  3 Fission fragments mass distributions of super-heavy nuclei

In order to understand what kind of mass distributions one can expect from the
solution of Langevin equations for super-heavy nuclei, we looked first at the
potential energy of fissioning nuclei. Fig. 1 shows the potential energy Edef
of nuclei 296Lv and 302120 at zero temperature as a function of elongation
(the distance R12 between the centers of mass of left and right parts of a
nucleus) and the mass asymmetry (fragment mass number). In the top part of
Fig. 1 the energy was minimized with respect to the deformation parameters
$\delta_{1}$ and $\delta_{2}$. One sees the bottom of potential energy leading
to almost symmetric mass splitting. There is also a hint on the mass
asymmetric valley at $A_{F}$ close to $A_{F}$=208.

Figure 1: (top) The potential energy of ${}^{296}Lv$ and ${}^{302}120$ at
$T=0$ minimized with respect to deformation parameters $\delta_{1}$ and
$\delta_{2}$ (bottom), and at fixed values $\delta_{1}=-0.2$ and
$\delta_{2}=0.2$.

If the trajectories followed the bottom of potential energy, the mass
distributions would be symmetric. However, it is well known that the
trajectories may deviate substantially from the bottom of the potential valley
due to dynamic effects. We calculate the trajectories in four-dimensional
deformation space. In this space, the local minima could lead away from the
bottom of the potential valley. An example is shown in the bottom part of Fig.
1. Here we show the potential energy for fixed $\delta_{1}$= \- 0.2 and
$\delta_{2}$=0.2. One clearly sees another valley, leading to strongly mass
asymmetric splitting.

In Fig. 2, we show the fission fragment mass distributions of super-heavy
nuclei from 276Hs to 308122 as a function of fragment mass number $A_{F}$. The
FFMDs of nuclei from 276Cn to 308122 have three or four peak structures. The
main component is the symmetric peak, split into two components in some
isotopes. The peaks of lighter fragments are located around $A_{F}$=140.

Figure 2: The fission fragment mass distributions of super-heavy nuclei from
276Hs to 308122 calculated for the excitation energies $E^{*}$=10, 20 and 30
MeV as a function of the fragment mass number

One can also see the strongly asymmetric peak at the mass number close to
$A_{F}$=208. The strength of the (almost) symmetric and asymmetric components
in FFMD of SHEs depends on the proton and neutron numbers of the compound
nucleus. For 276Cn, the contribution of a strongly asymmetric peak is tiny.
This contribution becomes larger for more heavy SHE. In some elements of SHEs
with $Z=$116-122, the symmetric and mass-asymmetric peaks are of the same
magnitude. More details can be found in [8].

The similar strongly mass-asymmetric peaks in FFMD of SHEs were also found
recently in [9] within the Langevin approach with the so call Fourier shape
parametrization.

##  4 The memory effects in nuclear fission

In order to investigate the role of memory effects in nuclear fission, we
exploit a simple one-dimensional model with the potential energy given by the
two-parabolic potential (Kramers potential), see Fig. 3.

${\rm
E}_{pot}(q)=2V_{b}q(q-q_{0})/q_{0}^{2},\,0<q<q_{0};2V_{b}(q-q_{0})(2q_{0}-q)q_{0}^{2},\,q_{0}<q<2q_{0}.$
(4)

The potential (4) depends on two parameters, the barrier height $V_{b}$ and
the barrier width $q_{0}$. We have fixed the barrier height $V_{b}$ = 6 MeV,
which is close to the value of the fission barrier of actinide nuclei. The
width of the barrier is somewhat uncertain. It depends on the definition of
the collective coordinate $q$ and the model for the potential energy. For
simplicity, we have put here $q_{0}=1.0$.

For the potential (4) one can define the stiffness $C=d^{2}E_{pot}/dq^{2}$ and
the frequency of harmonic vibrations $\omega_{0}=\sqrt{C/M}$. In the present
work, we fix $\hbar\omega_{0}$ =1.0 MeV, which is close to the frequency of
collective vibrations calculated for 224Th in [10] within the microscopic
linear response theory. Then, for the mass parameter we will have the
deformation and temperature-independent value,

$M=C/\omega_{0}^{2}=4V_{b}/(\omega_{0}^{2}q_{0}^{2}).$ (5)

For the friction coefficient $\bar{\gamma}$ we use a slightly modified
approximation of [10],

$\bar{\gamma}/M=0.6(T^{2}+\hbar^{2}\omega_{0}^{2}/\pi^{2}))/(1+T^{2}/40).$ (6)

For the temperature, we consider here two options: constant temperature regime
and constant energy regime. In a constant temperature regime, the temperature
is time-independent, related to the initial excitation energy $E^{*}$ by the
Fermi-gas relation, $aT^{2}=E^{*}$, where $a$ is the level density parameter
of Töke and Swiatecki [11]. The fission width calculated in a constant
temperature regime will be denoted as $\Gamma_{f}(T)$.

At small excitations, the temperature varies with deformation and time, and
there is no reason to consider it constant. So, it should be adjusted to the
local excitation energy on each integration step by the relation (3).
Correspondingly, fission width calculated in a constant energy regime is
denoted as $\Gamma_{f}(E)$.

The fission width, $\Gamma_{f}$, is defined assuming the exponential decay of
the number of ”particles” in the potential well,

$P(t)=e^{-\Gamma_{f}t/\hbar}\to\Gamma_{f}=-\hbar\ln[P(t)]/t.$ (7)

By solving the Langevin equations one will get the set of time moments
$t_{b}$, at which some trajectories would cross the barrier. From this
information, one can find the probability $P(t)$ and the fission width
$\Gamma_{f}$, see [12].

Figure 3: (left) The two-parabolic potential (4) and few examples of the
dynamical trajectories. (right) The fission width as the solution of Eqs. (1,
4, 7) calculated at constant temperature (open dots), and the Kramers
approximations (8) for high and low damping limits.

The Markovian fission width $\Gamma_{f}(T)$ calculated by Eqs. (1, 4, 7) is
plotted as function of the damping parameter $\eta$ in the right part of Fig.
3. To present the results in a broader range of parameters, the damping
parameter $\eta\equiv\bar{\gamma}/2M\omega_{0}$ in these calculations was
considered as a free parameter.

For the comparison, in Fig.3 we also show the Kramers decay width
$\Gamma_{HV},\Gamma_{LV}$ in limits of high and low viscosity (friction) [13],

$\Gamma_{HV}=\frac{\hbar\omega_{0}}{2\pi}e^{-V_{b}/T}(\sqrt{1+\eta^{2}}-\eta)\,,\quad\Gamma_{LV}=\frac{\hbar\bar{\gamma}}{M}\frac{V_{b}}{T}e^{-V_{b}/T}.$
(8)

As one can see, the dependence of $\Gamma_{f}(T)$ on $\eta$ is rather
complicated. The fission width $\Gamma_{f}(T)$ grows as function of $\eta$ in
low damping region ($\eta<0.1$). For $\eta>0.2$, the fission width
$\Gamma_{f}(T)$ decreases as function of $\eta$.

In nuclear systems, the Markovian assumption is often too restrictive. We thus
have to generalize the above Langevin equations to allow for finite memory
effects. They read as [14],

$\displaystyle dq/dt$ $\displaystyle=$ $\displaystyle p(t)/M,$ (9)
$\displaystyle\frac{dp}{dt}$ $\displaystyle=$ $\displaystyle-\frac{\partial
E_{pot}}{\partial
q}-\int_{0}^{t}dt^{\prime}\gamma(t-t^{\prime})p(t^{\prime})/M+\zeta\,,\,\quad\gamma(t-t^{\prime})\equiv\bar{\gamma}e^{-\frac{t-t^{\prime}}{\tau}}/\tau\,,$

where $\tau$ is the memory (or relaxation) time. The extension consists in
allowing the friction to have a memory time, i.e., the friction reacts on past
stages of the system, what is called a retarded friction.

The random numbers $\zeta$ in (9) are the normally distributed random numbers
with the properties $<\zeta(t)>=0$,
$<\zeta(t)\zeta(t^{\prime})>=T\gamma(t-t^{\prime})$. In the limit
$\omega_{0}\tau<<1$, one recovers the Markovian limit of nuclear fission
dynamics, i.e., when the friction force is simply given by $\gamma\dot{q}(t)$.
The random numbers $\zeta(t)$ in (9) satisfy the equation

$d\zeta(t)/dt=-\zeta(t)/\tau+R(t)/\tau\,,$ (10)

and are used in the description of the so-called Ornstein-Uhlenbeck processes.

In the top part of Fig. 4 the calculated fission width $\Gamma_{f}(E)$ is
shown as a function of the damping parameter $\eta$ both for small and large
excitation energies, $E^{*}$=10, 25 and 60 MeV, for few values of the
relaxation time. Besides $\tau=0$, we choose in calculations below the two
values of $\tau$, $\tau=5\cdot 10^{-22}$ sec and $\tau=10^{-21}$ sec.

Figure 4: (top) The dependence of the fission width $\Gamma_{f}(E)$ (solid)
and the approximation (11) (dashed) on the damping parameter $\eta$ for few
values of the relaxation time $\tau$, $\tau$=0, $\tau=5\cdot 10^{-22}$ sec,
$\tau=10^{-21}$ sec and the initial excitation energies $E_{in}^{*}$=10, 25
and 60 MeV. (bottom) The dependence of the fission width $\Gamma_{f}(E)$
(solid) and the approximation (11) (dashed) on the relaxation time $\tau$ for
a few values of the damping parameter $\eta$, $\eta$=0.1, 0.5 and 1.0.

The results of Langevin calculations satisfying the energy conservation
condition are shown in Fig. 4 by solid lines. The fission width
$\Gamma_{f}(E)$ grows as a function of $\eta$ and decreases as a function of
$\tau$ in low damping region. The tendency is the opposite in the high damping
region; the fission width $\Gamma_{f}$ falls as a function of $\eta$ and
increases as a function of $\tau$. Such dependence is common both for small
and large excitation energies.

In the bottom part of Fig. 4, the fission width $\Gamma_{f}(E)$ (solid lines)
is shown as a function of the relaxation time $\tau$ for a few fixed values of
the damping parameter $\eta$. The bottom part of Fig. 4 confirms the above
conclusion: the dependence of fission width $\Gamma_{f}$ on $\eta$ and $\tau$
is opposite in low and high damping regions.

For the comparison, we show by dashed lines in Fig. 4 the available analytical
approximation for $\Gamma_{f}(T,\tau)$ [14, 15, 16],

$\frac{1}{\Gamma_{eff}}=\frac{1}{\Gamma_{LV}}+\frac{1}{\Gamma_{HV}}\,,\quad\Gamma_{LV}(\tau)=\frac{\Gamma_{LV}(0)}{1+\omega_{0}^{2}\tau^{2}},\quad\Gamma_{HV}(\tau)=\frac{\hbar\lambda}{2\pi}e^{-V_{b}/T}\,,$
(11)

where $\lambda$ is the largest positive solution of the secular equation,

$\lambda^{3}+\lambda^{2}/\tau+(\bar{\gamma}/M\tau-\omega_{0}^{2})\lambda-\omega_{0}^{2}/\tau=0\,.$
(12)

As can be seen, the results of Langevin calculations for $\Gamma_{f}(E)$ are
smaller than the analytical estimate (11) both in low and high damping limits.
The ratio $\Gamma_{f}(E)/\Gamma_{eff}$ is close to 1 at $E^{*}$=60 MeV and
close to 0.1 at $E^{*}$=10 MeV.

##  5 Summary

The calculated mass distributions of fission fragments of super-heavy nuclei
from 268Hs to 308122 demonstrate a three-four peaks structure of mass
distributions. In light super-heavies, we see the dominant mass symmetric peak
at $A_{F}\approx$ 140\. With increasing mass and charge numbers of fissioning
nuclei, the highly asymmetric peaks at $A_{H}\approx$ 208 appears. In
290-296Lv and 290-296Og, the three peaks in FFMD are approximately of the same
magnitude at E*=10 MeV.

The investigation of memory effects in nuclear fission is carried out. The
calculations presented here offer complete information on the dependence of
fission probability on all essential parameters, the relaxation time $\tau$,
the damping parameter $\eta$, and the excitation energy E*.

It turned out that the fission width $\Gamma_{f}(E)$ calculated under the
constant energy requirement is generally smaller than that calculated in the
constant temperature regime, $\Gamma_{f}(T)$, or the Bohr-Wheeler
approximation.

The dependence of the fission width $\Gamma_{f}(E)$ on the relaxation time
$\tau$ is very sensitive to the damping parameter $\eta$. In the low viscosity
region, the fission width $\Gamma_{f}(E)$ grows as a function of $\eta$ and
decreases as a function of $\tau$. In the high-viscosity region, the tendency
is the opposite. Such dependence is common both for small and large excitation
energies.

Acknowledgements. The authors are grateful to Prof. K.Pomorski for the
valuable discussions and presentation of our results at the Zakopane
Conference

## References

  * [1] S.G.Nilsson, C.F. Tsang, A. Sobiczewski et al, Nucl. Phys. A 131, 1 (1969).
  * [2] Yu.Ts. Oganessian, A.V. Yeremin, A.G. Popeko et al, Nature 400, 242 (1999).
  * [3] J. Maruhn and W. Greiner, Zeit. f. Phys. 251, 431 (1972).
  * [4] M.D. Usang, F.A. Ivanyuk, C. Ishisuka, and S. Chiba, Phys. Rev. C 94, 044602 (2016).
  * [5] C. Ishizuka, M.D. Usang, F.A. Ivanyuk  et al, Phys. Rev. C 96, 064616 (2017).
  * [6] M.D. Usang, F.A. Ivanyuk, C. Ishizuka, and S. Chiba, Phys. Rev. C 96, 064617 (2017).
  * [7] M.D. Usang, F.A. Ivanyuk, C. Ishisuka, and S. Chiba, Scientific Reports 9, 1525 (2019).
  * [8] C. Ishisuka, X. Zhang, M. D. Usang, F. A. Ivanyuk, and S. Chiba, Phys. Rev. C 101, 011601(R) (2020).
  * [9] P.V. Kostryukov, A. Dobrowolski, B. Nerlo-Pomorska et al, Chin. Phys. C 45, 124108 (2021).
  * [10] H. Hofmann, F. A. Ivanyuk, C. Rummel, and S. Yamaji, Phys. Rev. C 64, 054316 (2001).
  * [11] J. Töke, W. J. Swiatecki, Nucl. Phys. A 372, 141 (1981).
  * [12] F.A. Ivanyuk, S.V. Radionov, C. ishizuka and S. Chiba, Nucl. Phys. A 1028, 122526 (2022).
  * [13] H. A. Kramers, Physica VII, 284 (1940).
  * [14] Y. Abe, S. Ayik, P.-G. Reinhard, and E. Suraud, Phys. Rep.  275, 49 (1996).
  * [15] R.F. Grote and J.T. Hynes, Jour. Chem. Phys. 73, 2715 (1980).
  * [16] D.Boilley, Y.Lallouet, Jour. Stat. Phys. 125, 477 (2006).


# MorphoLander: Reinforcement Learning Based Landing of a Group of Drones on
the Adaptive Morphogenetic UAV

Sausar Karaf, Aleksey Fedoseev, Mikhail Martynov,
Zhanibek Darush, Aleksei Shcherbak, and Dzmitry Tsetserukou The authors are
with the Intelligent Space Robotics Laboratory, Skoltech, Bolshoy Boulevard
30, bld. 1, 121205, Moscow, Russia email: {sausar.karaf, aleksey.fedoseev,
mikhail.martynov, zhanibek.darush, aleksei.shcherbak,
<EMAIL_ADDRESS>
###### Abstract

This paper focuses on a novel robotic system MorphoLander representing
heterogeneous swarm of drones for exploring rough terrain environments. The
morphogenetic leader drone is capable of landing on uneven terrain, traversing
it, and maintaining horizontal position to deploy smaller drones for extensive
area exploration. After completing their tasks, these drones return and land
back on the landing pads of MorphoGear. The reinforcement learning algorithm
was developed for a precise landing of drones on the leader robot that either
remains static during their mission or relocates to the new position. Several
experiments were conducted to evaluate the performance of the developed
landing algorithm under both even and uneven terrain conditions. The
experiments revealed that the proposed system results in high landing accuracy
of 0.5 cm when landing on the leader drone under even terrain conditions and
2.35 cm under uneven terrain conditions. MorphoLander has the potential to
significantly enhance the efficiency of the industrial inspections, seismic
surveys, and rescue missions in highly cluttered and unstructured
environments.

_Keywords — Swarm of Drones, Precise UAV Landing, Morphogenetic UAV,
Reinforcement Learning_

##  I Introduction

The use of drones in monitoring and inspection tasks has increased
significantly in recent years due to their high mobility and ability to access
areas isolated from unmanned ground vehicles (UGVs). Teams of heterogeneous
robots can collaborate to achieve tasks with high efficiency. For example,
construction site monitoring, high-altitude operations, and exploration of
rough terrain can be done more efficiently with the collaborative capabilities
of multi-agent systems, where drones working alongside a leader robot can
execute operations. However, continuous exploration by drones remains a
challenging problem caused by the lack of a power supply and the inability of
drones to land on uneven surfaces.

Figure 1: Landing of the group of drones on the MorphoLander robot standing on
an uneven surface.

Intelligent unmanned aerial vehicles (UAVs) shown superior capabilities for
autonomous inspections, path planning, and data collection, through adaptation
to dynamic uncertain environments and complex tasks [1]. While multi-agent
aerial systems were widely studied [2], the complementary skills of agents in
heterogeneous formations performed more efficiently in missions [3]. Several
concepts of heterogeneous robotic teams were proposed in previous research,
such as object classification, inspection or digital art with UAVs carrying
different tools [4, 5, 6, 7].

However, applications for multi-agent teams are still limited due to their
inability to take off and land in unstructured terrains. While there were
efforts to address this issue with different robots, e.g., the application of
UAVs to deliver mini-UGVs for exploration in environments with sparse
obstacles[10], there are remaining challenges in exploration of cluttered
terrains, which require novel approaches to expand the scope of UAV
applications.

In this paper, we propose a novel system MorphoLander (Fig. 1) that utilizes
the capabilities of the hybrid drone as a landing platform for smaller drones.
Two major contributions of this work are the development of a compensation
system for surface inclination with a multi-terrain drone and the development
of an algorithm for landing in unstructured environments.

##  II Related Works

In UAV missions performed in unstructured environments, taking off and landing
pose the highest challenge for the system due to the surface inclination and
low controllability of the drone at these stages of a mission.

Several researchers proposed utilizing human capabilities as a means to
achieve a safe landing with mini- and nano-UAVs. For example, Tsykunov et al.
[11] and Auda et al. [12] explored drone landing on human limbs with the help
of visual and haptic cues. While showing sufficient precision and stability in
landing with several drones, these scenarios require the substantial presence
of a human at the remote site and are not feasible for autonomous systems.

To achieve a precise docking without a human-in-the-loop, more complex
algorithms are required for both the drone and the platform carrying the
landing pads. Several systems were proposed to achieve precise landing on a
platform with limited sizes, e.g., Nguyen et al. [13] developed a precise
drone landing system for charging, while Kooi et al. [14] proposed an RL-based
algorithm for landing on an inclined platform. While showing high stability in
static environments, the performance of both algorithms in environments with a
changing surface layout requires further investigation. A landing approach on
a moving wheeled platform was developed by Gupta et al. [15], however, the
proposed wheeled platform is not able to compensate for the inclination of the
surface. Fedoseev et al. [16] proposed a system for drone docking in midair by
utilizing robotic arm with soft gripper. This approach is indifferent to
dynamic unstructured terrain, however, it requires additional manipulators to
be placed on the landing site for each drone. Finally, Jain et al. [17]
suggested using a mothership drone for docking in midair. This approach
achieved a precise landing through the mobility of both platform and single
landing drone. However, its stability in dynamic conditions and the presence
of several landing agents require further exploration due to the changing
dynamics of the hovering platform.

##  III System Overview

###  III-A Design of the Landing Platform

To achieve efficient and safe autonomous landing we designed a platform that
would allow for drone docking both on ground and in midair. Thus, landing on a
morphogenetic UAV [18] provides a realistic and versatile test environment.
MorphoLander’s landing platform is the morphogenetic robot with four robotic
legs. Each leg has three degrees of freedom (DoFs) with the shoulder oriented
perpendicularly to the central axis of the robot. The legs are actuated by
Dynamixel MX-106 and MX-28 servomotors in the hip joints and MX-64 servomotors
in the knee joints. The landing gear is constructed using lightweight PLA
material except for the base, which comprises two 3 mm thick carbon disks
supporting the legs and hexacopter axes. An overview of the multi-terrain
drone, including the locomotion algorithm, was described in our previous work
[20]. In [19], we explored various leader-follower drone formations. In this
work, we have enhanced the design of the morphogenetic leader drone by
developing landing sites for the follower drones to land on. To accomplish
this, we embedded two landing pads with diameter of 20 cm each into the body
of the landing platform (Fig. 3).

Figure 2: 3D CAD model of MorphoLander.

###  III-B Adaptive Landing Gear

To land the mothership robot on an uneven surface, a stabilization algorithm
was developed based on the currently applied load on the servomotors, since no
external force-torque sensors were applied to preserve the weight of the
drone. Dynamixel servomotors are capable of measuring load as the internal
output, proportional to the percentage of the maximum motor torque.
Consequently, the load on the drives depends on the force exerted on the
robot’s end-effector perpendicular to the surface. By selecting the rate of
the current load proportionally to the weight of the robot and its limbs, we
were able to filter out the noise present in statics from the values obtained
in dynamics. When moving clockwise, the torque on the servo is within 20-50%
of stall torque and when moving counterclockwise it is from -50% to -20%. If
the limb does not touch the ground, the load on the servo is lower then 4% of
stall torque and reaches 4% when standing. When a single limb is at a higher
stage than the others, the torque on it increases up to 15-20% of stall
torque. When receiving such values, drone raises this limb higher to reduce
the load and return the robot into a stable position.

The algorithm below shows the given data filtering method. The idea of this
stabilization algorithm is that the end-effector of each limb moves along a
strictly vertical line through inverse kinematics (Fig. 3).

Figure 3: The currently applied load (generated torque in % of stall torque)
acting on the landing gear during the MorphoLander platform adaptation to the
uneven surface.

For the developed algorithm the maximum height of the raised limb equals 10
cm. This height is subdivided into 100 segments (percent). When subjected to a
load, the limb ascends by 10%, while in the absence of any load, it descends
by 5%. The algorithm for filtering the values and moving the limb depending on
the load in the shoulder joint is shown in Alg. 1.

Algorithm 1 Landing Control Algorithm

1:Update $\tilde{F}\leftarrow Dynamixel~{}Load$ each 0.05 seconds

2:Average $F\leftarrow\tilde{F}$ from the last 0.15 seconds

3:if $5\%\leq F\leq 15\%$ then

4: $x\leftarrow x+10$

5: if $x\geq 100$ then

6: $x\leftarrow 100$

7: end if

8:end if

9:if $\ \textbf{then}-4\%\geq F\geq-9\%$

10: $x\leftarrow x-5$

11: if $x\leq 0$ then

12: $x\leftarrow 0$

13: end if

14:end if

##  IV Reinforcement Learning Methodology

In order to effectively train the Reinforcement Learning (RL) agent, it is
necessary to have a thorough understanding of the system that is being
controlled. The focus of this study is on the landing maneuvers of Bitcraze
Crazyflie 2.1 micro-UAVs, which are equipped with both low- and high-level
controllers. The controller scheme is depicted in Fig. 4.

To utilize the capability of the pre-existing controllers, the reinforcement
learning agent that we train outputs an optimal velocity vector, represented
as follows:

$\mathbf{u}=\begin{bmatrix}v_{x}\ v_{y}\ v_{z}\end{bmatrix}^{T},$ (1)

where $v_{i}$ is the velocity along the $i$-th axis. This velocity vector will
then be fed into a cascade of proportional-integral-derivative (PID)
controllers, which are implemented on the Crazyflie drone.

Figure 4: Crazyflie controller scheme. The state of the drone is the
observation of the RL agent. The output of the agent, which is the target
velocity, is sent to the onboard controller. The onboard controller regulates
the RPM of the propellers to create thrust.

To enable the RL agent to learn an optimal control policy, it should be able
to interact with the environment and receive feedback in the form of rewards.
It is widely accepted that the usage of simulated environments for training of
the agent can ensure a safe and efficient learning process. However, the
accuracy of the simulation is crucial for the success of the learned policy in
real-world scenarios. Hence, we designed a simulation environment that closely
resembles the real-world environment to ensure that the learned policy is
transferable to the physical system.

###  IV-A Drone Landing Simulator

A simulated environment was developed with the Unity real-time development
platform to train the RL agent (Fig. 5). Unity game engine was applied based
on its ability to simulate physics and to use machine learning tools of the
MLAgents package [21] for RL agent training. Additionally, communication
between ROS and Unity scripts was established via RosBridge library, allowing
a low-delay control of the drone. In the simulated environment, the drone is
controlled by thrust forces generated by four motors, as shown in Fig. 4. To
match a real-world scenario, three PID regulators were implemented to control
the attitude rate, attitude, and velocity of the drone.

Figure 5: The simulated environment in Unity 3D. Blue drones are being
controlled by the RL agent. Orange drones show the goal landing positions.

The accuracy of the simulated environment plays a critical role in the
effectiveness of the RL agent. Therefore, the simulation was designed to
closely resemble real-world conditions to ensure that the agent could operate
effectively not only in the simulated environment but also in real-world
scenarios.

###  IV-B Training the Reinforcement Learning Controller

The primary objective of the RL algorithm is to discover a stochastic policy
$\pi_{k}(a|s)$ that maximizes the discounted return, denoted as follows:

$\eta(\pi_{\phi})=E_{\tau}[\sum_{t=0}^{T}\gamma^{t}r(s_{k},a_{k})],$ (2)

where $\tau$ represents the trajectory followed by policy $\pi_{\phi}$,
$r(s_{k},a_{k})$ is the reward function, and $\gamma$ is the discount factor.

In this study, we define the state of the drone as the linear kinematic values
and denote it as:

$s=[x,y,z,v_{x},v_{y},v_{z},a_{x},a_{y},a_{z}]^{T}.$ (3)

We define the state of the landing platform $s_{p}$ up to third order as:

$s_{p}=[x_{p},y_{p},z_{p},v_{x}^{p},v_{y}^{p},v_{z}^{p},a_{x}^{p},a_{y}^{p},a_{z}^{p}]^{T}.$
(4)

Finally, the observation of drone is derived as the difference between the two
states:

$\Delta s=s_{p}-s.$ (5)

The reward function is defined as:

$r=-e_{d}-\alpha\cdot e_{v}-\beta\cdot e_{a}-\gamma\cdot e_{u}+\xi,$ (6)

where $e_{d}$, $e_{v}$, and $e_{a}$ denote the Euclidean distance error of the
position, velocity, and acceleration, respectively. The value $e_{u}$ is the
magnitude of the control velocity generated by the RL agent, and $\xi$ is the
additional reward if the distance between the drone and the platform is less
than the threshold value.

To train the RL agent, we employed the Proximal Policy Optimization (PPO)
algorithm [22] due to its high performance and sample efficiency.
Additionally, we employed curriculum learning to speed up the learning
process. The agent training was conducted in two stages: position hold, to
teach the drone to maintain its position after reaching the desired location,
and position set, to teach the drone to reach a specific position and land on
the platform.

##  V Experimental Evaluation

To assess the efficacy of the developed MorphoLander system, we performed a
series of three experiments. In each experiment, a swarm consisting of two
Bitcraze Crazyflie drones was tasked with landing on the MorphoLander
platform. The positions of the drones were estimated using the Vicon V5 indoor
localization system, and the control commands were generated by the RL agent
in the Unity environment and transmitted to the drones through the Robot
Operating System (ROS). Each experiment was conducted in a series of 16
trials. In the subsequent sections, we will provide a detailed account of each
experiment and evaluate the controller’s performance.

###  V-A Landing on a Static Platform and Even Terrain

####  V-A1 Experiment Procedure

During the experiment, two Crazyflie drones took off from the floor and landed
on the MorphoLander robot placed on an even terrain. The distance between the
drones was 75 cm, and the starting distance between the drones and the landing
platform was 1.5 m. The drones were required to land on opposite platforms,
with the right drone landing on the left platform and vice versa.

####  V-A2 Experimental Results

Fig. 6 shows the results of the experiments with robot standing on even
terrain.

(a)

(b)

Figure 6: Landing positions (a) and trajectories (b) of drones landing on the
MorphoLander leader robot that stands on an even terrain. Orange and blue
lines represent the trajectories of the drones. The green and red cylinders
are the landing platforms.

The landing shift of a drone was measured and analyzed over six experiments
with a mean value of 0.55 cm, indicating that the RL agent was successful in
landing the swarm of two drones on the platform with high precision.

###  V-B Landing on a Static Platform and Uneven Terrain

####  V-B1 Experiment Procedure

This experiment evaluated the performance of the RL landing controller on the
MorphoLander leader robot in adaptive stabilization mode on unstructured
terrain. Two Crazyflie drones took off from the ground and landed on the
platform.

####  V-B2 Experiment Results

The results of the experiment are presented in Fig. 7.

(a)

(b)

Figure 7: Landing positions (a) and trajectories (b) of drones landing on the
MorphoLander robot that stands on uneven terrain. Orange and blue lines
represent the trajectories of the drones. The green and red cylinders are the
landing platforms.

The mean drone shift was 2.35 cm. This value was considered sufficient as it
falls within the precise landing error of 6.8 cm achieved in previous
experiments in this field [23]. However, drone landing shift was higher than
in the first experiment due to the added complexity of the terrain. The
performance of the RL agent was also affected by low tolerance of the landing
gear.

###  V-C Landing on a Relocated Platform and Uneven Terrain

####  V-C1 Experiment Procedure

This experiment aimed to replicate a real-world scenario, where the landing
controller is expected to operate in long-term missions. The experiment
involves transporting the Crazyflie drones using landing gear. The drones then
took off and hovered in place while the platform relocated to a different
position. Subsequently, the platform will come to a halt and await the landing
of the drones.

####  V-C2 Experiment Results

The detailed results of the experiment are presented in Fig. 8.

(a)

(b)

Figure 8: Landing positions (a) and trajectories (b) of drones landing on the
MorphoLander robot after its relocation. Orange and blue lines represent the
trajectories of the drones. The green and red cylinders are the landing
platforms.

In this experiment, the displacement of the landing gear to certain positions
obstructed the landing trajectory of the drones, resulting in a maximum mean
landing shift of 3.5 cm.

##  VI Conclusion and Future Work

This paper presented a novel robotic system MorphoLander consisting of a
morphogenetic leader drone and follower micro-drones capable of docking on the
leader by RL-based algorithm. The system leverages the ability of the leader
drone to land on uneven terrain and deploy smaller drones for extensive area
exploration. The RL-based controller was developed to achieve precise drone
landing on a static and relocated leader robot under even and uneven terrain
conditions. Experimental evaluations demonstrated a high level of landing
accuracy with mean drone shift of 2.35 cm from the landing pad center under
even terrain condition. The lowest mean landing shift of 0.55 cm was achieved
while landing on even terrain.

The proposed technology can potentially enhance industrial inspections,
delivery, rescue missions with supply preservation, and even provide on-board
recharging of the micro-drones. In the future, we will increase the robustness
of the system by teaching the agent to directly change the PWM signals that
control the individual thrusts. This scenario will allow the RL agent to learn
how to control drones in the presence of ground effect, downwash from other
drones, wind, and etc. We will additionally increase the stability of the
drone by teaching the RL agent to directly control the thrust of each motor,
allowing for more reliable and agile control in various external conditions.

## References

  * [1] Y. Zhou, B. Rao, and W. Wang, “Uav swarm intelligence: Recent advances and future trends,” IEEE Access, vol. 8, pp. 183 856–183 878, 2020.
  * [2] M. Abdelkader, S. Guler, H. Jaleel, and J. Shamma, “Aerial swarms: Recent applications and challenges,” Current Robotics Reports, vol. 2, pp. 1–12, 09 2021
  * [3] R. Arnold, J. Jablonski, B. Abruzzo, and E. Mezzacappa, “Heterogeneous uav multi-role swarming behaviors for search and rescue,” in 2020 IEEE Conference on Cognitive and Computational Aspects of Situation Management (CogSIMA), 2020, pp. 122–128.
  * [4] R. Arnold, B. Abruzzo, and C. Korpela, “Towards a heterogeneous swarm for object classification,” in 2019 IEEE National Aerospace and Electronics Conference (NAECON), 2019, pp. 139–147.
  * [5] S. Kojima, T. Takahashi, R. Bezerra, T. Nara, M. Takahashi, N. Saiki, K. Gunji, P. Songsuroj, R. Suzuki, K. Sato, Z. Han, K. Takahashi, Y. Okada, M. Watanabe, K. Tadakuma, K. Ohno, and S. Tadokoro, “Heterogeneous robots coordination for industrial plant inspection and evaluation at world robot summit 2020,”Advanced Robotics, vol. 0, no. 0, pp. 1–18, 2022. [Online]. Available: https://doi.org/10.1080/01691864.2022.2111230
  * [6] L. Sultan, M. Anjum, M. Rehman, S. Murawwat and H. Kosar, “Communication Among Heterogeneous Unmanned Aerial Vehicles (UAVs): Classification, Trends, and Analysis,” in IEEE Access, vol. 9, pp. 118815-118836, 2021, doi: 10.1109/ACCESS.2021.3107479.
  * [7] A. Uryasheva, M. Kulbeda, N. Rodichenko, and D. Tsetserukou, “DroneGraffiti: Autonomous Multi-UAV Spray Painting,” in ACM SIGGRAPH 2019 Studio (SIGGRAPH ’19), Article 4, pp. 1-2, New York, NY, USA, 2019, doi: 10.1145/3306306.3328000.
  * [8] W. L. Salas, L. M. Valent´ın-Coronado, I. Becerra, and A. Ram´ırezPedraza, “Collaborative object search using heterogeneous mobile robots,” in 2021 IEEE International Autumn Meeting on Power, Electronics and Computing (ROPEC), vol. 5, 2021, pp. 1–6.
  * [9] D. Pushp, S. Kalhapure, K. Das, and L. Liu, “Uav-miniugv hybrid system for hidden area exploration and manipulation,” in The 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2022), 09 2022.
  * [10] D. Pushp, S. Kalhapure, K. Das, and L. Liu, “Uav-miniugv hybrid system for hidden area exploration and manipulation,” in The 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2022), 09 2022.
  * [11] E. Tsykunov, R. Agishev, R. Ibrahimov, L. Labazanova, T. Moriyama, H. Kajimoto, and D. Tsetserukou. 2019. “SwarmCloak: Landing of a Swarm of Nano-Quadrotors on Human Arms,” In SIGGRAPH Asia 2019 Emerging Technologies (SA ’19). Association for Computing Machinery, New York, NY, USA, 46–47. https://doi.org/10.1145/3355049.3360542
  * [12] J. Auda, M. Weigel, J. R. Cauchard, and S. Schneegass. 2021. “Understanding Drone Landing on the Human Body,” In Proceedings of the 23rd International Conference on Mobile Human-Computer Interaction (MobileHCI ’21). Association for Computing Machinery, New York, NY, USA, Article 23, 1–13. https://doi.org/10.1145/3447526.3472031
  * [13] N.P. Nguyen, B.H. Lee, N. Xuan-Mung, L.N.N.T. Ha, H.S. Jeong, S.T. Lee, S.K. Hong. “Persistent Charging System for Crazyflie Platform,” Drones 2022, 6, 212. https://doi.org/10.3390/drones6080212
  * [14] J. E. Kooi and R. Babuška, “Inclined Quadrotor Landing using Deep Reinforcement Learning,” 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Prague, Czech Republic, 2021, pp. 2361-2368, doi: 10.1109/IROS51168.2021.9636096.
  * [15] A. Gupta, E. Dorzhieva, A. Baza, M. Alper, A. Fedoseev and D. Tsetserukou, “SwarmHawk: Self-Sustaining Multi-Agent System for Landing on a Moving Platform through an Agent Supervision,” 2022 International Conference on Unmanned Aircraft Systems (ICUAS), Dubrovnik, Croatia, 2022, pp. 990-997, doi: 10.1109/ICUAS54217.2022.9836080.
  * [16] A. Fedoseev, V. Serpiva, E. Karmanova, M. A. Cabrera, V. Shirokun, I. Vasilev, S. Savushkin, and D. Tsetserukou. “DroneTrap: Drone Catching in Midair by Soft Robotic Hand with Color-Based Force Detection and Hand Gesture Recognition,” 2021 IEEE 4th International Conference on Soft Robotics (RoboSoft), New Haven, CT, USA, 2021, pp. 261-266, doi: 10.1109/RoboSoft51838.2021.9479353.
  * [17] K. P. Jain and M. W. Mueller, “Flying batteries: In-flight battery switching to increase multirotor flight time,” 2020 IEEE International Conference on Robotics and Automation (ICRA), Paris, France, 2020, pp. 3510-3516, doi: 10.1109/ICRA40945.2020.9197580.
  * [18] R. Baines, S.K. Patiballa, J. Booth, et al. “Multi-environment robotic transitions through adaptive morphogenesis,” Nature 610, 283–289 (2022). https://doi.org/10.1038/s41586-022-05188-w
  * [19] Z. Darush, M. Martynov, A. Fedoseev, A. Shcherbak and D. Tsetserukou, “SwarmGear: Heterogeneous Swarm of Drones with Reconfigurable Leader Drone and Virtual Impedance Links for Multi-Robot Inspection,” IEEE International Conference on Unmanned Aircraft System (ICUAS 2023), Warsaw, Poland, June 6-9, 2023, in print.
  * [20] M. Martynov, Z. Darush, A. Fedoseev, D. Tsetserukou, “MorphoGear: An UAV with Multi-Limb Morphogenetic Gear for Rough-Terrain Locomotion,” IEEE/ASME International Conference on Advanced Intelligent Mechatronics, Seattle, Washington, USA, June 27-July 1, 2023, in print.
  * [21] A. Juliani, V. Berges, E. Teng, A. Cohen, J. Harper, C. Elion, C. Goy, Y. Gao, H. Henry, M. Mattar, D. Lange, “Unity: A general platform for intelligent agents,” arXiv preprint arXiv:1809.02627, 2020.
  * [22] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, “Proximal Policy Optimization Algorithms,” ArXiv, 2017, DOI:arXiv.1707.06347.
  * [23] Y. Guo, J. Guo, C. Liu, H. Xiong, L. Chai, and D. He, “Precision Landing Test and Simulation of the Agricultural UAV on Apron,” Sensors (Basel), vol. 20, no. 12, p. 3369, Jun. 2020. doi: 10.3390/s20123369. PMID: 32545886; PMCID: PMC7349159.