# Asymptotic integrability of nonlinear wave equations

A. M. Kamchatnov  Institute of Spectroscopy, Russian Academy of Sciences,
Troitsk, Moscow, 108840, Russia  Skolkovo Institute of Science and Technology,
Skolkovo, Moscow, 143026, Russia

###### Abstract

We introduce the notion of asymptotic integrability into the theory of
nonlinear wave equations. It means that the Hamiltonian structure of equations
describing propagation of high-frequency wave packets is preserved by
hydrodynamic evolution of the large-scale background wave, so that these
equations have an additional integral of motion. This condition is expressed
mathematically as a system of equations for the carrier wave number as a
function of the background variables. We show that a solution of this system
for a given dispersion relation of linear waves is related with the
quasiclassical limit of the Lax pair for the completely integrable equation
having the corresponding dispersionless and linear dispersive behavior. We
illustrate the theory by several examples.

###### pacs:

02.30.Ik, 05.45.Yv, 47.35.Fg

> At first sight it seems that the physical explanation of solitonic
> propagation as a manifestation of subtle balance of dispersive and nonlinear
> effects, on one side, and the mathematical idea of complete integrability of
> nonlinear wave equations, on the other side, are not related with each
> other. We show in this paper that a combined consideration of two limiting
> situations—the dispersionless evolution of large-scale waves and the
> propagation of high-frequency wave packets according to Hamilton’s optical-
> mechanical analogy—sheds new light on the complete integrability condition.
> As a result, we formulate a much weaker condition of asymptotic
> integrability which means that the Hamiltonian structure of equations for
> the packet’s propagation is preserved by the dispersionless evolution of the
> background wave providing thus an additional integral of motion. If this
> criterion is fulfilled, then we arrive at the quasiclassical limit of the
> Lax pair for some completely integrable equation. This theory is illustrated
> by several examples.

##  I Introduction

As was explained by classics of physics in the 19th century bouss-1872 ;
rayleigh-1876 ; stokes-91 ; kdv-95 , the phenomenon of solitary wave (or
solitonic) propagation is a result of subtle balance of nonlinear and
dispersive effects of wave motion. Independently, the mathematicians of the
19th century developed the idea of complete integrability of equations of
Newton dynamics (see, e.g., whitt-27  and references therein), and this idea
was based on some specific properties of Hamilton’s formulation of classical
mechanics. Apparently, these two fundamental ideas had not been related with
each other until the discovery of the inverse scattering transform method
(IST) of integration of nonlinear wave equations ggkm-67 ; lax-68 ; zs-71 .
Soon after this discovery, it was realized zf-71 ; gardner-71  that the
nonlinear wave equations solvable by the IST method possess the property of
complete integrability generalized to systems with infinite number of degrees
of freedom. Since then, the Hamiltonian approach to nonlinear wave equations
has become an important tool for their investigation and, as a result, the
soliton physics has become a well-developed part of the modern mathematical
physics (see, e.g., dickey ; ft-07  and references therein). At the same time,
it seems that the original physical idea of interplay of nonlinear and
dispersive effects has not played any essential role in this formal
development. The aim of this article is to demonstrate that at least in some
situations elaboration of this classical idea leads to interesting results
useful for the nonlinear wave physics.

##  II Asymptotic integrability

We suppose that our physical system is described by two variables which we
will call for definiteness the “density” $\rho$ and the “flow velocity” $u$.
In one-dimensional geometry, they depend on the space $x$ and time $t$
variables: $\rho=\rho(x,t),u=u(x,t)$. In the dispersionless limit of long
wavelengths, the dynamics of our system is described by some hydrodynamic
equations which can always be transformed (in case of two dependent variables)
to a diagonal Riemann form (see, e.g., ry-83 )

$\frac{\partial r_{+}}{\partial t}+v_{+}\frac{\partial r_{+}}{\partial
x}=0,\quad\frac{\partial r_{-}}{\partial t}+v_{-}\frac{\partial
r_{-}}{\partial x}=0$ (1)

for the so-called Riemann invariants $r_{\pm}=r_{\pm}(\rho,u)$. They are the
most convenient variables in wave physics, since in many typical situations
the ‘velocities’ $v_{\pm}$ are given by the expressions

$v_{\pm}=u\pm c,$ (2)

where $c$ is velocity of long sound waves propagating along a uniform
background with constant density $\rho$ and zero flow velocity $u$. This means
that the velocities $v_{\pm}$ correspond to the upstream or downstream
propagation of sound waves. Naturally, both $u$ and $c$ as well as $v_{\pm}$
can be expressed as functions of the Riemann invariants $r_{\pm}$. Equations
(1) are nonlinear and they express the nonlinear properties of our system.

The dispersive properties of the system are described by the dispersion
relations $\omega=\omega_{\pm}(k,r_{+},r_{-})$ (or
$\omega=\omega_{\pm}(k,\rho,u)$) for harmonic linear waves
$\propto\exp[i(kx-\omega t)]$ propagating along a uniform background with
constant values of $r_{+},r_{-}$ (or $\rho,u$). It is important that in our
formulation the dispersive and nonlinear properties are not independent of
each other: the velocities (2) can be expressed as a long wavelength limit of
the phase velocities of linear waves:

$v_{\pm}=\lim_{k\to 0}\frac{\omega_{\pm}(k)}{k},$ (3)

so the two dispersion relations correspond to the two directions of
propagation of linear waves.

Let now the background variables change little at distances of the order of
the size of the short-wavelength wave packet built from the linear waves with
wave numbers around the carrier wave number $k$. Then we can introduce the
coordinate $x(t)$ of such a wave packet and consider its motion along a
nonuniform time-dependent background wave whose evolution obeys Eqs. (1). In
this case, the frequencies $\omega_{\pm}$ become slow functions of $x$ and $t$
via the background variables $r_{\pm}=r_{\pm}(x,t)$. Consequently, propagation
of our wave packet obeys the Hamilton equations (see, e.g., LL-2 ; ko-90 )

$\frac{dx}{dt}=\frac{\partial\omega}{\partial
k},\quad\frac{dk}{dt}=-\frac{\partial\omega}{\partial x},$ (4)

where $\omega$ is given by one of the branches $\omega_{\pm}$ of the
dispersion relation. Thus, to find the path of the wave packet, we have to
solve at first the hydrodynamics equations (1), then substitute the solution
into the dispersion relation $\omega(k,r_{+},r_{-})$, and, at last, to solve
the Hamilton equations (4).

The above general formulation of the packet’s propagation problem refers to
arbitrary dispersion relations and related with them according to Eq. (3)
large-scale hydrodynamic equations. Now we want to specify this formulation in
such a way, that it becomes ‘integrable’. This means that the Hamilton
equations (4) must have an integral of motion which allows one to solve these
equations in quadratures. Thus, we demand that there exists such a function
$k=k(r_{+},r_{-})$ that the Hamilton equations (4) are compatible with the
background evolution according to the hydrodynamic equations (1). To express
this condition in analytical form, let us consider two moments of time
separated by a small interval $dt$ and, consequently, by a small distance
$dx=v_{g}dt$ between the corresponding locations of the wave packet, where
$v_{g}=\partial\omega/\partial k$ is the group velocity of the wave packet
(see the first Eq. (4)). Then the differences of the values of the background
Riemann invariants at the locations of the wave packet are equal to

$dr_{\pm}=\frac{\partial r_{\pm}}{\partial x}dx+\frac{\partial
r_{\pm}}{\partial t}dt=\frac{\partial r_{\pm}}{\partial x}(v_{g}-v_{\pm})dt,$

where we have taken into account the hydrodynamic equations (1). This relation
gives

$\frac{\partial r_{\pm}}{\partial x}dt=-\frac{dr_{\pm}}{v_{\pm}-v_{g}}$

and, consequently, the second Hamilton equation (4) yields the following
expression for the small change of the wave number:

$\begin{split}dk&=-\frac{\partial\omega}{\partial
x}dt=-\left(\frac{\partial\omega}{\partial r_{+}}\frac{\partial
r_{+}}{\partial x}+\frac{\partial\omega}{\partial r_{-}}\frac{\partial
r_{-}}{\partial x}\right)dt\\\ &=\frac{\partial\omega/\partial
r_{+}}{v_{+}-v_{g}}dr_{+}+\frac{\partial\omega/\partial
r_{-}}{v_{-}-v_{g}}dr_{-}.\end{split}$

On the other hand, our assumption that the wave number $k$ is only a function
$k=k(r_{+},r_{-})$ of the background variables gives

$dk=\frac{\partial k}{\partial r_{+}}dr_{+}+\frac{\partial k}{\partial
r_{-}}dr_{-}.$

Since we consider an arbitrary general solution of Eqs. (1), the differentials
$dr_{+}$ and $dr_{-}$ can be considered as independent of each other, so
comparison of the above expressions yields the system of equations

$\frac{\partial k}{\partial r_{+}}=\frac{\partial\omega/\partial
r_{+}}{v_{+}-v_{g}},\quad\frac{\partial k}{\partial
r_{-}}=\frac{\partial\omega/\partial r_{-}}{v_{-}-v_{g}},$ (5)

which define the function $k=k(r_{+},r_{-})$ of two independent variables
$r_{+},r_{-}$, which is the desirable integral of motion. Naturally, for
existence of such a function these derivatives must commute,

$\frac{\partial}{\partial r_{-}}\left(\frac{\partial k}{\partial
r_{+}}\right)=\frac{\partial}{\partial r_{+}}\left(\frac{\partial k}{\partial
r_{-}}\right),$ (6)

and this asymptotic integrability condition is only fulfilled for some special
choices of the dispersion relation $\omega=\omega(k,r_{+},r_{-})$ and,
consequently, of the hydrodynamic equations (1).

It is worth noticing that if we consider particular solutions of Eqs. (1) in
the form of simple waves, then one of the Riemann invariants is constant and
the above calculation leads to a single equation for the wave number $k=k(r)$
which becomes a function of the only varying Riemann invariant $r$,

$\frac{dk}{dr}=\frac{\partial\omega/\partial r}{V-v_{g}},$ (7)

where $\omega=\omega(k,r)$ and $V=\lim_{k\to 0}(\omega(k)/k)$,
$v_{g}=\partial\omega(k)/\partial k$. This equation was obtained in Ref. el-05
in the theory of dispersive shock waves as a small-amplitude limit of the
Whitham modulation equations and in Refs. kamch-20  from the Hamilton theory
of wave packets propagation. The general equations (5) were considered in
Refs. sk-23 ; kamch-23  with applications to the problems of propagation of
linear wave packets and asymptotic theory of solitons produced from an
intensive initial pulse. Here we shall consider the relation of the asymptotic
integrability condition to the complete integrability of nonlinear wave
equations.

##  III Transition to the complete integrability

As was noticed in Ref. ks-24 , the solution of Eqs. (5) is related with the
quasiclassical limit of Lax pairs of nonlinear wave equations completely
integrable in framework of the Ablowitz-Kaup-Newell-Segur (AKNS) scheme
akns-74  written in a scalar form kk-02 . Let the completely integrable
equation can be written as a compatibility condition of two linear equations

$\phi_{xx}=\mathcal{A}\phi,\qquad\phi_{t}=-\frac{1}{2}\mathcal{B}_{x}\phi+\mathcal{B}\phi_{x},$
(8)

where the functions $\mathcal{A}$ and $\mathcal{B}$ depend on the wave
variables $\rho,u$, their space derivatives of different orders, and on the
spectral parameter $\lambda$. If we turn to the quasiclassical limit of these
equations with fast oscillations of $\phi$, so that we can neglect all the
derivatives of the smooth functions $\rho$ and $u$ in the expressions for
${\mathcal{A}}$ and ${\mathcal{B}}$, then we arrive at their quasiclassical
limits $\overline{\mathcal{A}}$ and $\overline{\mathcal{B}}$ which only depend
on the wave variables $\rho$ and $u$ and the spectral parameter.

The nonlinear equations under consideration reduce in the dispersionless limit
to some hydrodynamic-like equations which can be transformed to the form (1)
for the Riemann invariants $r_{+},r_{-}$. On the other hand, linearization of
our completely integrable equations yields the dispersion relation
$\omega=\omega(k,r_{+},r_{-})$. Thus, we arrive at the system (5) which in
case of completely integrable equations has a solution

$k=k(r_{+},r_{-},q),$ (9)

where $q$ is an integration constant. As was shown in Ref. ks-24 , the
functions $\overline{\mathcal{A}}$ and $\overline{\mathcal{B}}$ are related
with $k$ and $\omega$ by the formulas

$\begin{split}\overline{\mathcal{A}}&=-\frac{1}{4}k^{2}(r_{+},r_{-},q),\\\
\overline{\mathcal{B}}&=-\frac{\omega(k(r_{+},r_{-},q),r_{+},r_{-})}{k(r_{+},r_{-},q)},\end{split}$
(10)

where $q$ is related with the spectral parameter $\lambda$ of the IST method.
It is natural to ask, whether it is possible to go in the opposite direction
and starting from the hydrodynamic equations and dispersion relation which
satisfy the asymptotic integrability condition (6), so that the solution of
Eqs. (5) is known, to arrive at the Lax pair (8) of some completely integrable
nonlinear wave equation.

Actually, this procedure of generalization of quasiclassical formulas to their
fully dispersive counterparts is analogous to the well-known quantization
procedure of classical mechanical systems resulting in equations for their
quantum-mechanical behavior. As is well known, such a quantization procedure
is not trivial and unique, so we shall confine ourselves to the simplest
methods widely used in physics.

In the first method, we can consider the functions
$\overline{\mathcal{A}},\overline{\mathcal{B}}$ given by Eqs. (10) as exact
functions defining the Lax pair (8), so their compatibility condition
$(\phi_{xx})_{t}=(\phi_{t})_{xx}$, i.e.,

$\mathcal{A}_{t}-2\mathcal{B}_{x}\mathcal{A}-\mathcal{B}\mathcal{A}_{x}+\frac{1}{2}\mathcal{B}_{xxx}=0$
(11)

yields the corresponding nonlinear wave system.

In the second method, we can transform Eqs. (10) to the variables $\rho,u$ and
combine these variables to a complex field

$\psi=\sqrt{\rho}\exp\left(i\int^{x}u(x^{\prime},t)dx^{\prime}\right)$ (12)

according to the well-known Madelung transformation madelung  of the quantum-
mechanical wave function. Then in the quasiclassical limit we have
$u\approx-i\psi_{x}/\psi$ or $u\approx i\psi^{*}_{x}/\psi^{*}$, so we can try
to return to the nonlinear dispersive equation for the field $\psi$ by the
replacements

$\rho\to\psi\psi^{*},\quad u\to-\frac{i\psi_{x}}{\psi}\quad\text{or}\quad
u\to\frac{i\psi^{*}_{x}}{\psi^{*}}$ (13)

in the quasiclassical expressions $\overline{\mathcal{A}}$ and
$\overline{\mathcal{B}}$. Besides that, we might have to add to these
functions some terms vanishing in the quasiclassical limit. The form of these
term can be guessed by means of analogy with known results and correctness of
their choice can be proved by checking the compatibility condition (11).

Let us illustrate this approach by several examples.

##  IV Shallow water and Bose-Einstein condensate

Let the system be described in the dispersionless limit by the shallow water
equations

$\rho_{t}+(\rho u)_{x}=0,\quad u_{t}+uu_{x}+\rho_{x}=0.$ (14)

As a dispersion relation we choose the formula

$\omega=k\left(u\pm\sqrt{\rho+\frac{k^{2}}{4}}\right)$ (15)

which in the long wavelength limit gives Eqs. (2) with $c=\sqrt{\rho}$, that
is

$v_{\pm}=u\pm\sqrt{\rho}.$ (16)

The Riemann invariants are defined by the formulas

$r_{\pm}=u/2\pm\sqrt{\rho}.$ (17)

It is convenient to rewrite Eqs. (5) in terms of $\rho$ and $u$, so we get the
equations

$\frac{\partial k^{2}}{\partial\rho}=-4,\quad\frac{\partial k^{2}}{\partial
u}=-2\sqrt{k^{2}+4\rho},$ (18)

which define the function $k=k(\rho,u)$. The asymptotic integrability
condition (6) is fulfilled and Eqs. (18) have the solution

$\begin{split}k^{2}&=(q-u)^{2}-4\rho\\\ &=(q-2r_{+})(q-2r_{-}).\end{split}$
(19)

If we assume that $q>u$ and choose the upper sign in Eq. (15), then we obtain

$\frac{\omega}{k}=\frac{1}{2}(q+u).$ (20)

The first method of quantization yields the Lax pair (8) with

$\mathcal{A}=\rho-\frac{1}{4}(q-u)^{2},\quad\mathcal{B}=-\frac{1}{2}(q+u),$
(21)

where $q$ plays the role of the spectral parameter. This is the Lax pair for
the Kaup-Boussinesq system kaup-75

$\rho_{t}+(\rho u)_{x}-\frac{1}{4}\rho_{xxx}=0,\quad u_{t}+uu_{x}+\rho_{x}=0.$
(22)

It finds applications to dynamics of shallow water dispersive waves.

If we treat the system (14) as Landau equations landau-41  for the superfluid
component and assume that Eq. (15) is the Bogoliubov dispersion relation
bogol-47  for linear waves propagating along a uniform condensate, then the
second method of quantization seems appropriate for the condensate’s wave
function
$\psi=\sqrt{\rho}\exp\left(i\int^{x}u(x^{\prime},t)dx^{\prime}\right)$. We
make replacements (13) in Eqs. (21) and after adding a higher order term for
fulfillment of Eq. (11) we arrive at the Lax pair with

$\begin{split}\mathcal{A}&=-\frac{1}{4}\left(q+\frac{i\psi_{x}}{\psi}\right)^{2}+\psi\psi^{*}-\left(\frac{\psi_{x}}{2\psi}\right)_{x},\\\
\mathcal{B}&=-\frac{1}{2}\left(q-\frac{i\psi_{x}}{\psi}\right).\end{split}$
(23)

This is the well-known scalar form of the Lax pair for the Gross-Pitaevskii
gross-61 ; pit-61  (or nonlinear Schrödinger (NLS) bn-67 ) equation

$i\psi_{t}+\frac{1}{2}\psi_{xx}-|\psi|^{2}\psi=0.$ (24)

##  V Derivative NLS equations

As we saw above, the shallow water equations (14) and the dispersion relation
(15) lead to different completely integrable systems depending on the method
of ‘quantization’ of the solution (19) of the asymptotic integrability
equations. Consequently, if we have some completely integrable equation for a
complex field
$\psi=\sqrt{\rho}\exp\left(i\int^{x}u(x^{\prime},t)dx^{\prime}\right)$, then
the corresponding dispersionless system and dispersion relation for linear
waves are also known and they satisfy the asymptotic integrability condition,
so the solution (9) can be found together with the quasiclassical limit (10)
of the functions ${\mathcal{A}}$ and ${\mathcal{B}}$. Then they define another
completely integrable system for the variables $\rho$ and $u$ which can be
considered as a ‘shallow water’ counterpart of the equation for the complex
field $\psi$. Here we shall consider such systems obtained from three forms of
the derivative NLS (DNLS) equation.

###  V.1 Kaup-Newell DNLS equation

The Kaup-Newell DNLS equationkn-78  has the form

$i\psi_{t}+\frac{1}{2}\psi_{xx}-i(|\psi|^{2}\psi)_{x}=0.$ (25)

The substitution
$\psi=\sqrt{\rho}\exp\left(i\int^{x}u(x^{\prime},t)dx^{\prime}\right)$ yields
in the dispersionless limit the hydrodynamic system

$\begin{split}&\rho_{t}+\left[\rho\left(u-\frac{3}{2}\rho\right)\right]_{x}=0,\\\
&u_{t}+uu_{x}-(\rho u)_{x}=0.\end{split}$ (26)

The characteristic velocities are given by the formulas

$v_{\pm}=u-2\rho\pm\sqrt{\rho(\rho-u)}$ (27)

and Eqs. (26) can be cast to the diagonal form (1) for the Riemann invariants

$r_{\pm}=\frac{u}{2}-\rho\pm\sqrt{\rho(\rho-u)}.$ (28)

Linearization of Eq. (25) for small-amplitude waves propagating along a
uniform background yields the dispersion relation

$\omega=k\left(u-2\rho\pm\sqrt{\rho(\rho-u)+\frac{k^{2}}{4}}\right).$ (29)

Eqs. (5) can be transformed to the variables $\rho,u$, so they read

$\begin{split}&\frac{\partial
k^{2}}{\partial\rho}=4\left[\sqrt{k^{2}+4\rho(\rho-u)}-(2\rho-u)\right],\\\
&\frac{\partial k^{2}}{\partial
u}=-2\left[\sqrt{k^{2}+4\rho(\rho-u)}-2\rho\right]\end{split}$ (30)

and have the solution

$\begin{split}k^{2}&=(q-u)^{2}+4q\rho\\\ &=(q-2r_{+})(q-2r_{-}),\end{split}$
(31)

where $q$ is an integration constant, so that

$\frac{\omega}{k}=\frac{q}{2}+\frac{u}{2}-\rho.$ (32)

We define the Lax pair according to Eqs. (10),

$\mathcal{A}=-\frac{1}{4}(q-u)^{2}-q\rho,\qquad\mathcal{B}=-\frac{q}{2}-\frac{u}{2}+\rho,$
(33)

where $q$ plays the role of the spectral parameter. The compatibility
condition yields the completely integrable system

$\begin{split}&\rho_{t}+\left[\rho\left(u-\frac{3}{2}\rho\right)\right]_{x}+\frac{1}{4u}(u-2\rho)_{xxx}=0,\\\
&u_{t}+uu_{x}-(\rho u)_{x}+\frac{1}{2u}(u-2\rho)_{xxx}=0.\end{split}$ (34)

Obviously, this is a dispersive generalization of the hydrodynamic system (26)
different from the Kaup-Newell equation transformed to the
$(\rho,u)$-variables:

$\begin{split}&\rho_{t}+\left[\rho\left(u-\frac{3}{2}\rho\right)\right]_{x}=0,\\\
&u_{t}+uu_{x}-(\rho
u)_{x}+\left(\frac{\rho_{x}^{2}}{8\rho^{2}}-\frac{\rho_{xx}}{4\rho}\right)_{x}=0.\end{split}$
(35)

Although the substitutions

$\overline{\rho}=\rho(\rho-u),\quad\overline{u}=u-2\rho$ (36)

transform Eqs. (34) to the Kaup-Boussinesq system (22) for the variables
$(\overline{\rho},\overline{u})$, the inverse transform is not single-valued,
the system (34) is not genuinely nonlinear and describes nonlinear waves
different from those for the Kaup-Boussinesq system.

###  V.2 Chen-Lee-Liu DNLS equation

The Chen-Lee-Liu equation cll-79

$i\psi_{t}+\frac{1}{2}\psi_{xx}-i|\psi|^{2}\psi_{x}=0$ (37)

reduces after Madelung transformation (12) to the dispersionless equations

$\rho_{t}+\left[\rho\left(u-\frac{\rho}{2}\right)\right]_{x}=0,\quad
u_{t}+uu_{x}-(\rho u)_{x}=0$ (38)

and leads to the dispersion relation

$\omega=k\left(u-\rho\pm\sqrt{-\rho u+\frac{k^{2}}{4}}\right),$ (39)

so we get the characteristic velocities

$v_{\pm}=u-\rho\pm\sqrt{-\rho u}.$ (40)

The Riemann invariants can be defined by the formulas

$r_{\pm}=\frac{1}{2}(u-\rho)\pm\sqrt{-u\rho}.$ (41)

The asymptotic integrability equations

$\begin{split}&\frac{\partial k^{2}}{\partial\rho}=4u+2\sqrt{k^{2}-4\rho
u},\\\ &\frac{\partial k^{2}}{\partial u}=4\rho-2\sqrt{k^{2}-4\rho
u}\end{split}$ (42)

have the solution

$\begin{split}k^{2}&=(q-u+\rho)^{2}+4\rho u\\\
&=(q-2r_{+})(q-2r_{-}),\end{split}$ (43)

so

$\frac{\omega}{k}=\frac{1}{2}(q+u-\rho).$ (44)

As a result we arrive at the Lax pair with

$\mathcal{A}=-\frac{1}{4}(q-u+\rho)^{2}-\rho
u,\qquad\mathcal{B}=-\frac{1}{2}(q+u-\rho),$ (45)

and, consequently, obtain the completely integrable system

$\begin{split}&\rho_{t}+\left[\rho\left(u-\frac{\rho}{2}\right)\right]_{x}+\frac{1}{4(\rho+u)}(u-\rho)_{xxx}=0,\\\
&u_{t}+uu_{x}-(\rho u)_{x}+\frac{1}{4(\rho+u)}(u-\rho)_{xxx}=0.\end{split}$
(46)

Again this system reduces to the Kaup-Boussinesq system for the variables

$\overline{\rho}=-\rho u,\quad\overline{u}=u-\rho.$ (47)

###  V.3 Gerdjikov-Ivanov DNLS equation

The Gerdjikov-Ivanov equation gi-83

$i\psi_{t}+\psi_{xx}+i\psi^{2}\psi^{*}_{x}+\frac{1}{2}|\psi|^{4}\psi=0$ (48)

yields the dispersionless equations

$\begin{split}&\rho_{t}+\left[\rho\left(2u+\frac{\rho}{2}\right)\right]_{x}=0,\\\
&u_{t}+2uu_{x}-(\rho u)_{x}-\rho\rho_{x}=0,\end{split}$ (49)

the dispersion relation

$\omega=k\left(2u\pm\sqrt{-\rho(\rho+2u)+{k^{2}}}\right),$ (50)

and the characteristic velocities

$v_{\pm}=2u\pm\sqrt{-\rho(\rho+2u)}.$ (51)

The Riemann invariants are defined by the formulas

$r_{\pm}=u\pm\sqrt{-\rho(\rho+2u)}.$ (52)

The asymptotic integrability equations

$\begin{split}&\frac{\partial k^{2}}{\partial\rho}=2(u+\rho),\\\
&\frac{\partial k^{2}}{\partial
u}=2\rho-2\sqrt{k^{2}-4\rho(\rho+2u)}\end{split}$ (53)

give the solution

$\begin{split}k^{2}&=(q-u)^{2}+\rho(\rho+2u)\\\
&=(q-r_{+})(q-r_{-}),\end{split}$ (54)

so

$\frac{\omega}{k}=q+u.$ (55)

Consequently, we obtain the Lax pair with

$\mathcal{A}=-\frac{1}{4}(q-u)^{2}-\frac{1}{4}\rho(\rho+2u),\qquad\mathcal{B}=-q-u,$
(56)

and the corresponding completely integrable system

$\begin{split}&\rho_{t}+\left[\rho\left(2u+\frac{\rho}{2}\right)\right]_{x}+\frac{1}{\rho+u}u_{xxx}=0,\\\
&u_{t}+2uu_{x}-(\rho u)_{x}-\rho\rho_{x}=0.\end{split}$ (57)

which can be transformed to the Kaup-Boussinesq system for the variables

$\overline{\rho}=-\rho(\rho+2u),\quad\overline{u}=2u.$ (58)

As is known, the above three DNLS equations are related with each other by
gauge transformations ws-83 ; kundu-84 ; kundu-87 . As we see, their ‘shallow
water’ counterparts are related with each other by simple algebraic formulas.
At the same time, the physical meaning of these equations can be different, as
it will be especially clearly seen in one more example considered in the next
section.

##  VI Landau-Lifshitz equation and two-layer ‘shallow water’ model

The Landau-Lifshitz equation

$\mathbf{M}_{t}=(\mathbf{M}_{xx}-M_{3}\mathbf{e}_{3})\wedge\mathbf{M}$ (59)

for the magnetization vector $\mathbf{M}=(M_{1},M_{2},M_{3})$,
$|\mathbf{M}|=1$, with an easy-plane anisotropy, can be transformed to more
convenient variables $w,v$ by a Madelung-like substitution ikcp-17 ; ckp-16 ;
ish-17

$\begin{split}M_{3}&=-w,\\\ M_{-}&\equiv M_{1}-iM_{2}\\\
&=\sqrt{1-w^{2}}\exp\left(-i\int^{x}vdx^{\prime}\right).\end{split}$ (60)

The resulting equations

$\begin{split}&w_{-}\left[v(1-w^{2})\right]_{x}=0,\\\
&v_{t}-\left[w(1-v^{2})\right]_{x}+\left[\frac{1}{\sqrt{1-w^{2}}}\left(\frac{w_{x}}{\sqrt{1-w^{2}}}\right)_{x}\right]_{x}=0\end{split}$
(61)

reduce to the dispersionless equations

$\begin{split}w_{t}&-\left[v(1-w^{2})\right]_{x}=0,\\\
v_{t}&-\left[w(1-v^{2})\right]_{x}=0\end{split}$ (62)

which can be transformed to the form (1) by introduction of the Riemann
invariants

$r_{\pm}=vw\pm\sqrt{(1-v^{2})(1-w^{2})}.$ (63)

The characteristic velocities of Eqs. (62) are equal to

$v_{\pm}=2vw\pm\sqrt{(1-v^{2})(1-w^{2})}.$ (64)

The whole system (61) after linearization with respect to small-amplitude
waves propagating along a uniform background yields the dispersion relation

$\omega=k\left(2wv\pm\sqrt{k^{2}+(1-v^{2})(1-w^{2})}\right)$ (65)

for harmonic waves. The asymptotic integrability equations (5) written in
terms of the $(v,w)$-variables

$\begin{split}&\frac{\partial k^{2}}{\partial
w}=2\left[w(1-v^{2})-v\sqrt{k^{2}+(1-v^{2})(1-w^{2})}\right],\\\
&\frac{\partial k^{2}}{\partial
v}=2\left[v(1-w^{2})-w\sqrt{k^{2}+(1-v^{2})(1-w^{2})}\right],\end{split}$ (66)

have the solution

$\begin{split}k^{2}&=(q-vw)^{2}-(1-v^{2})(1-w^{2})\\\
&=(q-r_{+})(q-r_{-})\end{split}$ (67)

so (assuming $q-vw>0$) we get

$\frac{\omega}{k}=q+vw.$ (68)

We define the Lax pair (8) with

$\begin{split}&\mathcal{A}=\frac{1}{4}[(1-v^{2})(1-w^{2})-(q-vw)^{2}],\\\
&\mathcal{B}=-q-vw\end{split}$ (69)

and obtain the corresponding completely integrable equations

$\begin{split}w_{t}&-\left[v(1-w^{2})\right]_{x}+\frac{v(vw)_{xxx}}{v^{2}-w^{2}}=0,\\\
v_{t}&-\left[w(1-v^{2})\right]_{x}-\frac{w(vw)_{xxx}}{v^{2}-w^{2}}=0.\end{split}$
(70)

This is a ‘shallow water’ counterpart of the Landau-Lifshitz equation (59).

It is remarkable that the dispersionless equations (62) were derived in Ref.
ovs-79  as a long wave shallow water approximation for dynamics of inner waves
in the fluid composed of two thin layers with very small difference of
densities of liquids in them (the lighter layer above the heavier one). The
dispersion relation for linear inner waves in a two-layer system is described
by a more complicated formula than Eq. (65) (see, e.g., Ref. lamb ).
Nevertheless, it has similar general structure, so Eqs. (70) can be considered
as a ‘toy model’ for dispersive dynamics of shallow inner waves. Although Eqs.
(70) can be reduced to the Kaup-Boussinesq system for the variables

$\overline{\rho}=(1-v^{2})(1-w^{2}),\quad\overline{u}=2vw,$ (71)

the qualitative properties of these two systems are very different. For
example, the Kaup-Boussinesq system is genuinely nonlinear and an initial
discontinuity evolves according to it into one of six possible wave structures
eggk-95 ; cikp-17 . The system (70) is not genuinely nonlinear and
classification of possible wave structures includes various combined
dispersive shock waves similar to the structures appearing in the theory of
the Landau-Lifshitz equation ikcp-17 . One might expect that such a
classification of possible wave structures evolving from initial
discontinuities in the theory of Eqs. (70) is qualitatively similar to such a
classification for the actual two-layer dispersion relation.

As one can see, although the considered above equations belong to different
physical situations, they all correspond to the same solution of the
asymptotic integrability equations with the dispersion relation

$\omega=k\left(r_{+}+r_{-}\pm\sqrt{\frac{1}{4}(r_{+}-r_{-})^{2}+k^{2}}\right),$
(72)

so that the long wavelength characteristic velocities are given by

$v_{\pm}=r_{+}+r_{-}\pm\frac{1}{2}(r_{+}-r_{-}),$ (73)

and the solution can be written in the form

$k^{2}=\left[q-\frac{1}{2}(r_{+}+r_{-})\right]^{2}-\frac{1}{4}(r_{+}-r_{-})^{2}.$
(74)

Only the formulas connecting the Riemann invariants $r_{\pm}$ with the
physical variables depend on the choice of the physical system and the above
list of equations is not exhaustive. In particular, one can easily check that
the Jaulent-Miodek system jm-76 ; zhou-97  is described by the same solution
of the asymptotic integrability equations.

##  VII Zakharov-Ito system

To demonstrate another type of solution of the asymptotic integrability
equations, let us consider the hydrodynamic-like equations written in the
Riemann form (1), where

$v_{+}=6r_{+}+2r_{-},\quad v_{-}=2r_{+}+6r_{-}.$ (75)

One can check that the asymptotic integrability condition is fulfilled if we
choose the dispersion relation

$\begin{split}\omega&=k\Big{\\{}4(r_{+}+r_{-})+{k^{2}}/{2}\\\
&\pm\sqrt{\left[2(r_{+}+r_{-})+{k^{2}}/2\right]^{2}-16r_{+}r_{-}}\Big{\\}},\end{split}$
(76)

so that Eqs. (5) have the solution

$k^{2}=\frac{4}{q}(q-r_{+})(q-r_{-}),$ (77)

and then

$\frac{\omega}{k}=4q+2(r_{+}+r_{-}).$ (78)

Let the Riemann invariants be related with the ‘physical’ variables $\rho,u$
by the formulas

$r_{+}=\frac{1}{2}(u+\sqrt{u^{2}-4\rho}),\quad
r_{-}=\frac{1}{2}(u-\sqrt{u^{2}-4\rho}),$ (79)

so that

$k^{2}=4\left(q-u+\frac{\rho}{q}\right),\quad\frac{\omega}{k}=4q+2u.$ (80)

Transition to the Lax pairs according to the rules (10) yields

$\mathcal{A}=-q+u-\frac{\rho}{q},\quad\mathcal{B}=-4q-2u,$ (81)

where $q$ plays the role of the spectral parameter. The resulting completely
integrable system is the well-known Zakharov-Ito system zakh-80 ; ito-82

$\begin{split}&\rho_{t}+2u\rho_{x}+4\rho u_{x}=0,\\\
&u_{t}+6uu_{x}-4\rho_{x}-u_{xxx}=0.\end{split}$ (82)

It is worth noticing that although the dispersionless system with the
velocities (75) written in $(\rho,u)$-variables

$\begin{split}&\rho_{t}+2u\rho_{x}+4\rho u_{x}=0,\\\
&u_{t}+6uu_{x}-4\rho_{x}=0\end{split}$ (83)

can be transformed to the shallow water system (14) for the variables

$\overline{\rho}=4u^{2}-16\rho,\quad\overline{u}=4u,$ (84)

these substitutions are not compatible with transformation of the dispersion
relation (15). In fact, they cast the dispersion relation (76) to the form

$\omega=k\left(4u+\frac{k^{2}}{2}\pm\sqrt{\left(2u+\frac{k^{2}}{2}\right)^{2}-16\rho}\right)$
(85)

different from the relation (15) for the Kaup-Boussinesq system.

##  VIII Conclusion

In this paper, we introduced the physically natural condition of asymptotic
integrability of nonlinear wave equations which means that two asymptotic
limits—dispersionless (hydrodynamic) evolution of a smooth background wave and
propagation of high-frequency wave packets—are compatible with each other in
the sense that the Hamiltonian structure of the packet’s propagation is
preserved by evolution of the background wave and, consequently, the Hamilton
equations have an integral of motion. In fact, this weaker integrability
condition leads to the quasiclassical limit of complete integrability in
framework of the AKNS scheme ks-24  and if the Lax pair of the equations under
consideration does not depend on the space derivatives of the wave variables,
then the asymptotic integrability condition reproduces the exact Lax pair.
This observation sheds new light on the origin of the completely integrable
nonlinear wave equations.

Besides that, it should be noticed that the relationship for the dependence of
the packet’s wave number on the background variables can be converted to a
similar relationship for the inverse half-width of narrow solitons propagating
along smooth background, and this remark allows one to develop a Hamiltonian
theory for propagation of narrow solitons along a non-uniform and time
dependent background ks-23 ; kamch-24b ; ks-24b .

At last, Eqs. (5) of asymptotic integrability can also have approximate
solutions correct in the limit of large wave numbers $k$ even for not
completely integrable equations. In these quite general situations, one can
develop the theory of propagation of small-amplitude wave packets along a non-
uniform and time-dependent background sk-23  and to formulate a generalized
Bohr-Sommerfeld quantization rule which determines the parameters of solitons
produced from an intensive initial pulse kamch-23 . Thus, the condition of
asymptotic integrability turns out to be a useful tool for investigation of
various problems in nonlinear physics.

###### Acknowledgements.

I thank E. A. Kuznetsov and M. V. Pavlov for useful discussions. This research
is funded by the research project FFUU-2021-0003 of the Institute of
Spectroscopy of the Russian Academy of Sciences (Sections 1-3) and by the RSF
grant number 19-72-30028 (Sections 4-7).

## Data Availability Statement

The data that support the findings of this study are available from the author
upon reasonable request.

## References

  * (1) J. Boussinesq, Théorie des ondes et des remous qui se propagent le long d’un canal rectangulaire horizontal, en communiquant au liquide contenu dans ce canal des vitesses sensiblement pareilles de la surface au fond, J. Math. Pures Appl., 17, 55–108 (1872).
  * (2) Lord Rayleigh, On waves, Phil. Mag., 1, 257–279 (1876).
  * (3) J. J. Stokes, Note on the theory of the solitary waves, Phil. Mag. 31, 314–316 (1891).
  * (4) D. J. Korteweg and G. de Vries, On the Change of Form of Long Waves advancing in a Rectangular Canal, and on a New Type of Long Stationary Waves, Phil. Mag., 39, 422–443 (1895).
  * (5) E. T. Whittaker, Analytical Dynamics, (Cambridge, CUP, 1927).
  * (6) S. C. Gardner, J. M. Greene, M. D. Kruskal, R. M. Miura, Method for solving the Korteweg-de Vries equation, Phys. Rev. Lett., 19, 1095–1097 (1967).
  * (7) P. D. Lax, Integrals of Nonlinear Equations of Evolution and Solitary Waves, Comm. Pure Appl. Math. 21, 467–490 (1968).
  * (8) V. E. Zakharov, A. B. Shabat, Exact theory of two-dimensional self-focusing and one-dimensional self-modulation of waves in nonlinear media, Zh. Eksp. Teor. Fiz. 61, 118–134 (1971) [Sov. Phys. JETP, 34, 62–69 (1972)].
  * (9) V. E. Zakharov and L. D. Faddeev, Korteweg-de Vries equation: A completely integrable Hamiltonian system, Funl. Analiz Prilozh., 5, 18–27 (1971) [Func. Anal. Appl. 5, 280–287 (1971)].
  * (10) S. C. Gardner, Korteweg–de Vries equation and generalizations. IV. Korteweg–de Vries as a Hamiltonian system, J. Math. Phys.. 12, 1548–1551 (1971).
  * (11) L. A. Dickey, Soliton Equations and Hamiltonian Systems, (World Scientific, Singapore, 2003).
  * (12) L. D. Faddeev, L. A. Takhtajan, Hamiltonian Methods in the Theory of Solitons, (Springer, Berlin, 2007).
  * (13) B. L. Roždestvenskii and N. N. Yanenko, Systems of Quasilinear Equations and Their Applications to Gas Dynamics, (Providence, AMS, 1983).
  * (14) L. D. Landau and E. M. Lifshitz, The Classical Theory of Fields, (Pergamon, Oxford, 1971).
  * (15) Yu. A. Kravtsov, Yu. I. Orlov, Geometrical Optics of Inhomogeneous Media, (Springer, Berlin, 1990).
  * (16) G. A. El, Chaos, Resolution of a shock in hyperbolic systems modified by weak dispersion, 15, 037103 (2005).
  * (17) A. M. Kamchatnov, Theory of quasi-simple dispersive shock waves and number of solitons evolved from a nonlinear pulse, Chaos 30, 123148 (2020).
  * (18) D. V. Shaykin, and A. M. Kamchatnov, Propagation of wave packets along large-scale background waves, Phys. Fluids, 35, 062108 (2023).
  * (19) A. M. Kamchatnov, Asymptotic theory of not completely integrable soliton equations, Chaos 33, 093105 (2023).
  * (20) A. M. Kamchatnov, D. V. Shaykin, Quasiclassical integrability condition in AKNS scheme, Physica D, 460, 134085 (2024).
  * (21) M. J. Ablowitz, D. J. Kaup, A. C. Newell, H. Segur, The inverse scattering transform—Fourier analysis for nonlinear problems, Stud. Appl. Math., 53, 249–315 (1974).
  * (22) A. M. Kamchatnov, R. A. Kraenkel, On the relationship between $2\times 2$ matrix and second-order scalar spectral problems for integrable equations, J. Phys. A: Math. Gen. 35, L13–L18 (2002).
  * (23) E. Madelung, Quantentheorie in hydrodynamischer Form, Z. Phys. 40, 322–326 (1927).
  * (24) D. J. Kaup, A higher-order water-wave equation and the method for solving it, Prog. Theor. Phys. 54, 396–408 (1975).
  * (25) L. D. Landau, The theory of superfuidity of helium II, J. Phys. (USSR) 5, 71–100 (1941).
  * (26) N. N. Bogoliubov. On the theory of superfluidity, J. Phys. (USSR), 11, 23–32 (1947).
  * (27) E. P. Gross, Structure of a Quantized Vertex in Boson Systems, Nuovo Cimento, 20, 454–477 (1961).
  * (28) L. P. Pitaevskii, Vortex Lines in an Imperfect Bose Gas, Zh. Eksp. Teor. Fiz. 40, 646–651 (1961) [Sov. Phys. JETP 13, 451–454 (1961)].
  * (29) D. J. Benney and A. C. Newell, The propagation of nonlinear wave envelopes, J. Math. & Phys., 46, 133–139 (1967).
  * (30) D. J. Kaup and A. C. Newell, An Exact Solution for a Derivative Nonlinear Schrödinger Equation, J. Math. Phys. 19, 798–801 (1978).
  * (31) H. H. Chen, Y. C. Lee, C. S. Liu, Integrability of Nonlinear Hamiltonian Systems by Inverse Scattering Method, Phys. Scr. 20, 490–492 (1979).
  * (32) V. S. Gerdjikov, M. I. Ivanov, The quadratic bundle of general form and the nonlinear evolution equations, Bulg. J. Phys. 10, 130–143 (1983).
  * (33) M. Wadati and K. Sogo, Gauge Transformations in Soliton Theory, J. Phys. Soc. Japan 52, 394–338 (1983).
  * (34) A. Kundu, Landau-Lifshitz and higher-order nonlinear systems gauge generated from nonlinear Schrödinger type equations, J. Math. Phys. 25, 3433–3438 (1984).
  * (35) A. Kundu, Exact solutions to higher-order nonlinear equations through gauge transformations, Physica D, 25, 399–406 (1987).
  * (36) T. Congy, A. M. Kamchatnov, and N. Pavloff, Dispersive hydrodynamics of nonlinear polarization waves in two-component Bose-Einstein condensates, SciPost Phys., 1, 006 (2016).
  * (37) S. K. Ivanov, A. M. Kamchatnov, T. Congy, and N. Pavloff, Solution of the Riemann problem for polarization waves in a two-component Bose-Einstein condensate, Phys. Rev. E 96, 062202 (2017).
  * (38) E. Iacocca, T. J. Silva, and M. A. Hoefer, Breaking of Galilean Invariance in the Hydrodynamic Formulation of Ferromagnetic Thin Films, Phys. Rev. Lett. 118, 017203 (2017).
  * (39) L. V. Ovsyannikov, Two-layer “shallow water” model, J. Appl. Mech. Tech. Phys. 20, 127–135 (1979).
  * (40) H. Lamb, Hydrodynamics, (Cambridge University Press, Cambridge, 1994).
  * (41) G. A. El, V. V. Geogjaev, A. V. Gurevich, and A. L. Krylov, Decay of an initial discontinuity in the defocusing NLS hydrodynamics, Physica D 87, 186–192 (1995).
  * (42) T. Congy, S. K. Ivanov, A. M. Kamchatnov, and N. Pavloff, Evolution of initial discontinuities in the Riemann problem for the Kaup-Boussinesq equation with positive dispersion, Chaos, 27, 083107 (2017).
  * (43) V. E. Zakharov, The inverse scattering method, Solitons, edited by R. K. Bullough and P. J. Caudrey, pp. 243–285 (Springer, Berlin, 1980).
  * (44) M. Ito, Symmetries and conservation laws of a coupled nonlinear wave equation, Phys. Lett. A 91, 335–338 (1982).
  * (45) M. Jaulent, I. Miodek, Nonlinear evolution equations associated with ‘energy-dependent Schrödinger potentials’, Lett. Math. Phys., 1, 243–259 (1976).
  * (46) R. Zhou, The finite-band solution of the Jaulent-Miodek equation, J. Math. Phys., 38, 2535–2546 (1997).
  * (47) A. M. Kamchatnov, D. V. Shaykin, Propagation of generalized Korteweg–de Vries solitons along large-scale waves, Phys. Rev. E 108, 054205 (2023).
  * (48) A. M. Kamchatnov, Hamilton theory of dark soliton motion in the nonlinear Schrödinger equation theory, Theor. Math. Phys. 219, 44–55 (2024).
  * (49) A. M. Kamchatnov, D. V. Shaykin, Propagation of dark solitons of DNLS equations along a large-scale background, Wave Motion, 129, 103349 (2024),


# TaskFlex Solver for Multi-Agent Pursuit via Automatic
Curriculum Learning

Jiayu Chen1, Guosheng Li1, Chao Yu1🖂, Xinyi Yang1, Botian Xu∗, Huazhong Yang1,
Yu Wang1🖂 🖂 Corresponding Authors<EMAIL_ADDRESS>of Electronic Engineering, Tsinghua University, Beijing, 100084, China. jiayu-
<EMAIL_ADDRESS>Work done as an intern in Tsinghua UniversityThis
research was supported by National Natural Science Foundation of China
(No.62325405, U19B2019, M-0248), Tsinghua University Initiative Scientific
Research Program, Tsinghua-Meituan Joint Institute for Digital Life, Beijing
National Research Center for Information Science, Technology (BNRist) and
Beijing Innovation Center for Future Chips.

###### Abstract

This paper addresses the problem of multi-agent pursuit, where slow pursuers
cooperate to capture fast evaders in a confined environment with obstacles.
Existing heuristic algorithms often lack expressive coordination strategies
and are highly sensitive to task conditions, requiring extensive
hyperparameter tuning. In contrast, reinforcement learning (RL) has been
applied to this problem and is capable of obtaining cooperative pursuit
strategies. However, RL-based methods face challenges in training for complex
scenarios due to the vast amount of training data and limited adaptability to
varying task conditions, such as different scene sizes, varying numbers and
speeds of obstacles, and flexible speed ratios of the evader to the pursuer.
In this work, we combine RL and curriculum learning to introduce a flexible
solver for multi-agent pursuit problems, named TaskFlex Solver (TFS), which is
capable of solving multi-agent pursuit problems with diverse and dynamically
changing task conditions in both 2-dimensional and 3-dimensional scenarios.
TFS utilizes a curriculum learning method that constructs task distributions
based on training progress, enhancing training efficiency and final
performance. Our algorithm consists of two main components: the _Task
Evaluator_ , which evaluates task success rates and selects tasks of moderate
difficulty to maintain a curriculum archive, and the _Task Sampler_ , which
constructs training distributions by sampling tasks from the curriculum
archive to maximize policy improvement. Experiments show that TFS produces
much stronger performance than baselines and achieves close to ${100\%}$
capture rates in both 2-dimensional and 3-dimensional multi-agent pursuit
problems with diverse and dynamically changing scenes. The project website is
at https://sites.google.com/view/tfs-2023.

##  I INTRODUCTION

Pursuit-evasion is the problem of multiple slow pursuers cooperating to
capture one or more fast evaders in a confined environment with obstacles. In
mobile robotics, pursuit-evasion tasks are used to study motion planning
problems that arise in adversarial settings, such as missile interception [1],
aircraft control [2], criminal pursuit as well as search and rescue [3]. The
main challenge of this problem is how multiple pursuers can cooperate to
capture evaders efficiently.

Heuristic algorithms have been widely used for multi-agent pursuit problems
[4, 5, 6]. Heuristic approaches require no training and can be directly
applied to different scenarios. However, these methods often cannot express
complex strategies that involve close coordination. The performance is
severely affected by task conditions, such as the scene size, the number and
the speed of obstacles, as well as the speed ratio of the evader to the
pursuer. Heuristic methods also require much hyperparameter tuning for each
test scenario. By contrast, reinforcement learning (RL) has been a promising
solution for a wide range of decision-making problems [7, 8], including multi-
agent cooperative pursuit problems [9, 10, 11, 12]. Deep reinforcement
learning is possible to obtain cooperative pursuit strategies, some of which
are difficult to hard-code via explicit rules.

However, training effective RL policies in complex scenarios can be
particularly challenging. This includes twofold: (1) learning a cooperative
policy over multiple agents in a large scene with multiple obstacles,
especially in a 3-dimensional scenario, consumes millions or even billions of
training samples due to an exponentially large action space and observation
space. (2) RL policies often suffer from narrow applicability to different
scenarios compared with classical heuristic approaches. A significant
challenge is to design an RL policy that can successfully address diverse task
conditions. An approximate optimal RL policy for scenarios with different task
conditions often requires an enormous amount of training time. Hence, most RL-
based pursuit methods focus on the 2-dimensional cases and only consider fixed
task conditions [9, 13, 14].

One way to accelerate RL training is curriculum learning – training agents in
tasks from easy to hard. In our work, we combine deep reinforcement learning
and the idea of curriculum learning to propose a TaskFlex Solver (TFS) for
multi-agent pursuit that can solve the pursuit-evasion problem with different
task conditions in both 2-dimensional and 3-dimensional environments. In TFS,
we design a curriculum learning algorithm that automatically constructs task
distributions based on the training progress, accelerating training and
obtaining better final performance. Our approach consists of two components, a
_Task Evaluator_ and a _Task Sampler_. The _Task Evaluator_ is a difficulty
partitioning mechanism that evaluates all tasks in a given task pool and
selects tasks of moderate difficulty to construct a curriculum learning
archive. The _Task Sampler_ constructs the training distribution that
maximizes the improvement of the current policy by both sampling from the
curriculum learning archive and the task pool.

We compare TFS with existing works in two testbeds, i.e., _2-dimensional
pursuit-evasion_ in the multi-agent particle-world environment (MPE) [15] and
_3-dimensional pursuit-evasion_ in the NVIDIA Isaac Sim-based Omnidrones
environment (Omni) [16]. We designed three different scenarios, the
_Challenging_ with extremely hard and fixed task conditions, the _Dynamic_
with dynamically changing task conditions, and the _Diverse_ with different
task conditions in the two testbeds. Our method outperforms all baselines with
a clear margin and achieves close to $100\%$ _capture rates_ in three
scenarios in both _2-dimensional pursuit-evasion_ and _3-dimensional pursuit-
evasion_ problems. Moreover, in the _3-dimensional pursuit-evasion_ , all the
baselines fail while our method obtains more than $90\%$ _capture rates_ in
all three scenarios and obtains at least a $2.36\times$ improvement in
_capture frequency_.

Figure 1: Overview of the TaskFlex Solver. We first initialize a task pool
$\mathcal{Q}_{pool}$ where the tasks in this task pool have $n$ variables. We
then feed the task pool to the _Task Evaluator_. The _Task Evaluator_
evaluates all the tasks in the task pool $\mathcal{Q}_{pool}$ using the
current policy $\pi_{\theta}$ and maintains a curriculum archive
$\mathcal{Q}_{cur}$ using the tasks with moderate capture rates. The _Task
Sampler_ samples from both the curriculum archive $\mathcal{Q}_{cur}$ and the
task pool $\mathcal{Q}_{pool}$ to construct the training distribution
$\mathcal{Q}_{train}$.

##  II RELATED WORK

###  II-A Multi-Agent Pursuit

For multi-agent pursuit problems, there are two kinds of methods: heuristic
pursuit algorithms and reinforcement learning-based algorithms. Heuristic
pursuit algorithms design the mathematical forms of force to guide the
pursuers’ move [4, 5, 6, 17]. For example, [4] assumes that each pursuer is
attracted by the evader and moves toward the evader while being repelled by
its teammates and obstacles to prevent collisions. [6] introduces a
surrounding force that enables pursuers to form an encirclement and approach
the faster evader. However, pursuers in these methods greedily chase the
evader and avoid obstacles, leading to a lack of cooperation among multiple
pursuers. Moreover, task conditions strongly influence the capture
performance.

Another promising method for multi-agent cooperative pursuit is via deep RL,
which learns cooperative pursuit strategies [13, 9, 18, 19]. For example, [9]
learns a pursuit strategy using a parameter-sharing Trust Region Policy
Optimization (TRPO) algorithm that coordinates multiple pursuers to capture
the evader. [13] applies Multi-agent Deep Deterministic Policy Gradient
(MADDPG) [15] to learn cooperative strategies to chase an evader. [18] focuses
on pursuit-evader games with non-holonomic agents, where new agents can join
the game. [19] proposes a new state representation for deep multi-agent RL
training by considering the agents as interchangeable and the exact number
irrelevant. RL-based methods allow for greater cooperation among pursuers,
some of which cannot be hard-coded by explicit rules. But the training method
of reinforcement learning has low sampling efficiency, and it usually takes
several days or even weeks of training time to solve the pursuit problem under
a given condition. Moreover, the pursuer strategies based on fixed task
conditions are challenging to apply to different scenarios. Compared to RL-
based methods, our algorithm focuses on accelerating the RL training procedure
and improve the policy performance under various task conditions with limited
training samples.

###  II-B Curriculum Learning

Curriculum learning [20] is a learning paradigm that accelerates training by
gradually increasing the difficulty of the training tasks. Automatic
curriculum learning (ACL) [21, 22, 23, 24] is a method that evaluates the
difficulty of a task by some metrics, such as task success rate, and
automatically generates the training distribution based on the current policy.
The core idea of ACL is to train the policy using tasks with moderate
difficulty, i.e., neither too hard nor too easy. In single-agent tasks, [22,
23] assume a fixed goal and generate tasks with starting states increasingly
far away from the goal to obtain a curriculum. [25, 26] consider a large task
space over simulator configurations and train from the easiest settings to the
hardest for sim-to-real adaptation. Recently, several works have applied ACL
in multi-agent systems to efficiently train a cooperative policy [27, 28, 10].
[27] proposes a curriculum learning algorithm to tackle goal-conditioned
cooperative MARL problems with sparse rewards through a novel variational
inference perspective. [28, 10] propose _population curriculum_ , which trains
entity-invariant policies over training tasks with a growing number of agents.
Our work, borrowing ideas from ACL, focuses on using curriculum learning to
improve the sample efficiency of deep reinforcement learning to solve multi-
agent pursuit problems with diverse task conditions and achieve high capture
performance.

##  III PRELIMINARY

###  III-A Problem Definition

In this paper, the multi-agent pursuit problem is defined as N pursuers
pursuing a faster evader in a scenario with obstacles. The goal of pursuers is
to catch the evader as quickly as possible while avoiding obstacles.
Conversely, the goal of the evader is to get as far away from the pursuers and
obstacles as possible. The evader will be captured by a pursuer if the
distance between them is less than $d_{c}$. Each episode of the capture
contains $T$ timesteps. If the evader is captured, the pursuers release the
evader, and the capture continues until the time $T$ runs out. If the time $T$
runs out and the evader is still not captured, the capture fails. The
following assumptions are further made:

The pursuers, the evader, and the obstacles are spawned randomly within a
square area of side $w_{s}$. Neither the pursuers nor the evader can get out
of the area: if they take an action that would end up outside the area, their
position is updated to be on the nearest area border. The number of obstacles
is $N_{o}$. The collision radius of pursuers is $d_{col}$. The distance
between pursuers, as well as between pursuers and obstacles, needs to be
larger than $d_{col}$ to avoid collision. The maximum speed of the pursuer is
$v_{p}$, and the evader maintains a constant speed $v_{e}$, where
$v_{p}<v_{e}$.

We consider the evader as a part of the environment and use a potential field
method [29] to control the evader. Pursuers and obstacles exert repulsion on
the evader along the vector direction. The boundary also exerts a force
perpendicular to the boundary on the evader so that the evader can avoid
boundarys. These forces decrease with distance. The speed vector of the evader
$\boldsymbol{v_{e}}$ is calculated by:

$\boldsymbol{v_{e}}=v_{e}\times\left(\sum_{i}\left(\frac{\boldsymbol{x_{p,i}}-\boldsymbol{x_{e}}}{{||\boldsymbol{x_{p,i}}-\boldsymbol{x_{e}}||}^{2}}\right)+\sum_{j}\left(\frac{\boldsymbol{x_{o,j}}-\boldsymbol{x_{e}}}{{||\boldsymbol{x_{o,j}}-\boldsymbol{x_{e}}||}^{2}}\right)+\sum_{k}\left(\frac{\boldsymbol{u_{k}}}{d_{k}}\right)\right)$

(1)

where $\boldsymbol{x_{e}}$ is the current position of the evader,
$\boldsymbol{x_{p,i}}$ is the current position of the pursuer $i$,
$\boldsymbol{x_{o,j}}$ is the position of the obstacle $j$ and
$\boldsymbol{u_{k}}$ is the unit vector pointing inward the boundary $k$.
$d_{k}$ is the vertical distance from the evader to the boundary $k$.

###  III-B Task Pool

We use a continuous vector $\tau$ to represent each task. $\tau$ is defined as
$({\tau}^{1},{\tau}^{2},\ldots,{\tau}^{n})$, where ${\tau}^{g}$ denotes a task
configuration parameter, such as the speed ratio of the evader to the pursuer
$v_{e}/v_{p}$. We define the task pool $\mathcal{Q}_{pool}$ as

$\small\mathcal{Q}_{pool}\coloneqq\\{({\tau}^{1},{\tau}^{2},\ldots,{\tau}^{n})|{\tau}^{g}_{min}\leq{\tau}^{g}\leq{\tau}^{g}_{max},1\leq
g\leq n\\},$ (2)

where the $max$ and $min$ subscripts represent the upper and lower bounds of
the corresponding dimension. $n$ is the number of task variables. We use the
tasks sampled from the task pool to formulate the multi-agent pursuit problem
for each episode. Fig. 2 illustrates a showcase of a task pool containing
different task conditions, such as the side length, the number of obstacles
and the speed ratio.

Figure 2: A showcase of the task pool. A task $\tau$ has three variables,
including the scene size, the number of obstacles, and the speed ratio of the
evader to the pursuer.

###  III-C Multi-agent Reinforcement Learning

Markov Game We formulate the multi-agent pursuit problem as a multi-agent
Markov decision process (MDP),
$M=<\mathcal{N},\mathcal{Q}_{pool},\mathcal{S},\mathcal{A},\mathcal{O},O,P,R,\gamma>$,
where the task condition is variable according to the task pool
$\mathcal{Q}_{pool}$. $\mathcal{N}\equiv\\{1,...,N\\}$ is a set of
$N=|\mathcal{N}|$ pursuers. $\mathcal{S}$ is the state space. $\mathcal{A}$ is
the shared action space for each pursuer. $\mathcal{O}$ is the observation
space. $o_{i}=O(s;i)$ denotes the observation for pursuer $i$ under state
$s\in\mathcal{S}$. $P(s^{\prime}|s,A)$ and $R(s,A)$ are the transition
probability and the reward function given state $s$ and joint actions
$A=(a_{1},\ldots,a_{N})$. $\gamma$ is the discount factor.

We consider homogeneous pursuers and learn a shared policy
$\pi_{\theta}(a_{i}|o_{i})$ parameterized by $\theta$ for each pursuer $i$.
The final objective $J(\theta)$ is to maximize the expected accumulative
reward over the entire task pool $\mathcal{Q}_{pool}$, i.e.,

$\displaystyle
J(\theta)=\mathbb{E}_{\tau\sim\mathcal{Q}_{pool},a_{i}^{t},s^{t}}\left[\sum_{t}\gamma^{t}R(s^{t},A^{t};\tau)\right].$
(3)

##  IV METHODOLOGY

To solve the lack of cooperativeness in heuristic algorithms and low training
efficiency of RL-based methods, we combine RL with curriculum learning to
train pursuers in the order of easy-to-hard, resulting in accelerating the
training of RL and improving the final performance. As shown in Fig. 1, we
introduce the overall algorithm, the TaskFlex Solver for multi-agent pursuit,
which consists of two components, a _Task Evaluator_ for evaluating and
selecting tasks with moderate difficulties and a _Task Sampler_ for
constructing the training distribution that maximizes the policy improvement.

First, we initiate with the setup of the task pool, represented as
$\mathcal{Q}_{pool}$. The task pool is then passed to the _Task Evaluator_
module. The _Task Evaluator_ performs an evaluation of all tasks in the task
pool using the TaskFlex policy $\pi_{\theta}$ and then selects a series of
tasks with a moderate level of complexity, which are then integrated into the
curriculum learning archive, denoted as $\mathcal{Q}_{cur}$. Subsequently, the
_Task Sampler_ samples tasks from the curriculum archive $\mathcal{Q}_{cur}$
and the task pool $\mathcal{Q}_{pool}$ to construct a training distribution of
tasks known as $\mathcal{Q}_{train}$. We generate parallel simulation
environments for MARL training according to the training distribution
$\mathcal{Q}_{train}$. Note that this framework is compatible with any MARL
algorithm. In this paper, we use a state-of-the-art MARL algorithm MAPPO [30]
as our training backbone. A detailed pseudo code of this entire process can be
found in Algo. 1.

Input: $\theta$, $p$, $\mathcal{Q}_{pool}$, $\alpha$, $\mathcal{Q}_{cur}$,
$T_{eval}$;

// $\mathcal{Q}_{cur}:$ the curriculum archive, $p:$ probability of sampling
from $\mathcal{Q}_{cur}$, $\alpha$: decay coefficient of $p$, $T_{eval}:$
update interval of $\mathcal{Q}_{cur}$.

Output: final policy $\pi_{\theta}$;

// use easy tasks to initialize $\mathcal{Q}_{cur}$.

$epoch\leftarrow 0$,
$\mathcal{Q}_{cur}\leftarrow\textbf{Init}(\mathcal{Q}_{pool})$;

repeat

// Task sampler.

Sample $\tau\sim\mathrm{Unif}(\mathcal{Q}_{cur})$ with probability $p$, else
$\tau\sim\mathrm{Unif}(\mathcal{Q}_{pool})$ to $\mathcal{Q}_{train}$;

// Train taskflex policy $\pi_{\theta}$.

Rollout with $\mathcal{Q}_{train}$ and train $\pi_{\theta}$ via MARL;

if  _$epoch~{}\%~{}T_{eval}==0$_ then

// Task evaluator.

$\mathcal{Q}_{cur}\leftarrow\\{{\tau}_{i}|{\sigma_{min}}~{}\leq~{}\textbf{Eval}(\pi_{\theta},{\tau}_{i})\newline
~{}\leq~{}{\sigma_{max}},{\tau}_{i}\in\mathcal{Q}_{pool}\\};$

// Sampling probability $p$ decays.

$p\leftarrow p\times(1-\alpha\times epoch)$;

$epoch\leftarrow{epoch+1}$;

until  _$\pi_{\theta}$ converges_;

Algorithm 1 TaskFlex Solver

###  IV-A Task Evaluator

As shown in Fig. 1, the _Task Evaluator_ serves as a crucial quality
evaluation mechanism responsible for assessing the performance of the learned
policy within the task pool. To facilitate this evaluation, we take the task
variables ${\tau}^{g}$ within each task
$\tau=({\tau}^{0},{\tau}^{1},\ldots,{\tau}^{g},\ldots,{\tau}^{n})$ and
uniformly divide them into $D_{g}$ intervals between their upper and lower
bounds, i.e. ${\tau}^{g}_{max}$ and ${\tau}^{g}_{min}$. For each task variable
${\tau}^{g}$, we randomly sample a value from each interval and combine all
the variables to generate $D_{1}\times D_{2}\times\ldots\times D_{n}$ specific
tasks. This sampling method ensures a good coverage of the entire task pool by
sampling only a finite number of tasks. During the evaluation phase, we
iterate through the sampled tasks to generate the corresponding simulation
environments. For each task ${\tau}$, we set the environment parameters
according to ${\tau}$ to generate a series of simulation environments. By
executing the policy $\pi_{\theta}$ in these simulation environments, we
calculate the average capture rate as the success rate of the task ${\tau}$.
We denote this process as $\textbf{Eval}(\pi_{\theta},{\tau})$.

After evaluation, we store tasks that meet the success rate criteria in the
curriculum archive, known as $\mathcal{Q}_{cur}$. Specifically, a task $\tau$
is included in $\mathcal{Q}_{cur}$ if it falls within the range defined by
hyperparameters $\sigma_{min}$ and $\sigma_{max}$. These hyperparameters set
the lower and upper thresholds for the success rate, ensuring that only tasks
exhibiting an acceptable level of performance are preserved in
$\mathcal{Q}_{cur}$. The curriculum archive $\mathcal{Q}_{cur}$ effectively
represents a distribution of tasks that are selected to maximize the learning
progress of the current policy $\pi_{\theta}$. In our paper, we follow a
general setting in curriculum learning [27] using $\sigma_{min}=0.5$ and
$\sigma_{max}=0.95$. For efficiency in the evaluation phase, we update the
curriculum archive $\mathcal{Q}_{cur}$ once every $T_{eval}$ policy
$\pi_{\theta}$ updates.

###  IV-B Task Sampler

In this section, we introduce the _Task Sampler_ , a pivotal component that
efficiently obtains the training distribution from a given pool of tasks,
allowing the algorithm to adapt and learn across diverse task scenarios. Our
method involves the construction of the training distribution, denoted as
$\mathcal{Q}_{train}$, by employing a probabilistic selection process.
Specifically, we select tasks from the curriculum archive $\mathcal{Q}_{cur}$
with probability $p$, while those from the task pool $\mathcal{Q}_{pool}$ are
selected with probability $1-p$.

Our choice to sample from $\mathcal{Q}_{cur}$ is driven by the objective of
accelerating convergence through the utilization of tasks of moderate
complexity. Conversely, sampling from the task pool, $\mathcal{Q}_{pool}$, is
essential for promoting global exploration and preventing the agents from
forgetting previously acquired knowledge and skills. Notably, the sampling
probability $p$ undergoes a decay process as the number of training episodes
progresses, and we use the hyperparameter $\alpha$ to control the decay rate.
In addition, before training, we select tasks whose task variables have all
the values corresponding to _min_ to form an easy task set that is added to
$\mathcal{Q}_{cur}$ to prevent $\mathcal{Q}_{cur}$ from being an empty set.

###  IV-C Reward Design

To encourage pursuers to move towards the evader efficiently, we introduce a
distance reward, $r_{dis}$, at each step:

$\displaystyle r_{dis}=-min_{i}(d_{i}),$ (4)

where $d_{i}$ represents the distance of the pursuer $i$ to the evader.

We further consider a capture reward, $r_{cap}$, for successfully capturing
the evader. If the distance between any pursuer and the evader falls below a
threshold, $d_{c}$, all pursuers are rewarded with a bonus of $10$. This
reward encourages the pursuers to work collectively in capturing the evader.

The collision penalty, $r_{col}$, acts as a critical safety measure to
discourage pursuers from colliding with obstacles. When the distance between a
pursuer $i$ and an obstacle is less than a threshold, $d_{col}$, the pursuer
$i$ will receive a penalty of $-100$.

The overall reward, $r_{total}$, is the linear combination of the above three
reward components to motivate pursuers to capture the evader while avoiding
potential obstacles effectively. $r_{total}$ is expressed as:

$\displaystyle r_{total}=r_{dis}+r_{cap}+r_{col}.$ (5)

##  V EXPERIMENT

(a) _2-D_. (b) _3-D_.

Figure 3: Illustration of _2-D_ and _3-D_.

###  V-A Testbeds

We consider the multi-agent pursuit problem over two testbeds, _2-dimensional
pursuit-evasion (2-D)_ in the multi-agent particle-world environment (MPE)
[15], and _3-dimensional pursuit-evasion (3-D)_ in the OmniDrones-Sim
environment (Omni) [16].

_2-dimensional pursuit-evasion_ As shown in Fig. 3, we define $\tau$ as
$(w_{s},v_{e}/v_{p},N_{o})$ in _2-D_ , where $w_{s}$ denotes the side length
of the scene area, $v_{e}/v_{p}$ denotes the speed ratio of the evader to the
pursuer and $N_{o}$ denotes the number of static obstacles. In this testbed,
one evader, three pursuers, and $N_{o}$ static obstacles are randomly spawned
in a square plane with side length $w_{s}$. The 2-dimensional environment is
fully-observable. Each pursuer’s observation includes the speed and position
of itself, other pursuers, and the evader. Each pursuer can also observe the
position of all obstacles and the current time step. The action space of
pursuers is discrete which contains stop and acceleration in four directions:
up, down, left, and right.

_3-dimensional pursuit-evasion_ As shown in Fig. 3, we define $\tau$ as
$(w_{s},v_{e}/v_{p},v_{o}/v_{p})$ in _3-D_ , where the definition of $w_{s}$
and $v_{e}/v_{p}$ are the same as _2-D_ and $v_{o}/v_{p}$ denotes the speed
ratio of the dynamic obstacles to the pursuer. In this testbed, one evader,
three pursuers, and five dynamic obstacles are randomly spawned in a square
area with side length $w_{s}$. Pursuers and the evader can fly in the air up
to a maximum height of $w_{s}$. Each pursuer is a drone that contains four
motors. The evader is a free-moving sphere with no constraints. This
3-dimensional environment is fully-observable. Each pursuer’s observations
include their self-state, the positions of other pursuers, the speed and
position of the evader, as well as the speed and position of obstacles. The
self-state comprises velocity, position, personal identity, and the current
time step. We define the action space of each pursuer in the form of multi-
discrete as $\mathcal{A}=\\{m_{0},m_{1},m_{2},m_{3}\\},m_{c}\in\\{0,1,2,3\\}$,
where $m_{c}$ represents the thrust of the $c$th motor.

###  V-B Task Setting

We design three different scenarios, _Challenging_ , _Dynamic_ and _Diverse_ ,
in the two testbeds.

_Challenging_ We evaluate the ability to solve extremely difficult pursuit
tasks with _Challenging_ where the task scene is large, contains multiple
static or dynamic obstacles, and the speed difference between the evader and
the pursuers is large. In this task, the speed ratio $v_{e}/v_{p}$ is $5.0$.
In _2-D_ , we create a large map with a side length of $4.0$ and add five
static obstacles. In contrast, in _3-D_ , we create a large region with a side
length of $1.0$ and introduce five fast dynamic obstacles with a speed of
$5.0$, increasing the complexity of the task.

_Dynamic_ In _Dynamic_ , we consider the scenarios in which the task
conditions change over time in an episode. In _2-D_ , the pursuers and evader
can move within a scene with $w_{s}=2.0$ with five static obstacles, and the
speed ratio of the evader to the pursuer $v_{e}/v_{p}$ is $1.3$ in the first
100 timesteps in an episode. In the last 100 timesteps, $w_{s}=4.0$ and
$v_{e}/v_{p}=5.0$. In _3-D_ , the task condition is defined as
$(w_{s},v_{e}/v_{p},v_{o}/v_{p})=(0.5,1.3,1.0)$ in the first 250 timesteps in
an episode and $(w_{s},v_{e}/v_{p},v_{o}/v_{p})=(1.0,5.0,5.0)$ in the last 250
timesteps.

_Diverse_ In _Diverse_ , all task parameters are variable, which means that
the algorithm must be able to solve the pursuit problem under different task
conditions. In _2-D_ , the task pool $\mathcal{Q}_{pool}$ is defined as
$\mathcal{Q}_{pool}\coloneqq\\{(w_{s},v_{e}/v_{p},N_{o})|2.0\leq w_{s}\leq
4.0,1.3\leq v_{e}/v_{p}\leq 5.0,0\leq N_{o}\leq 5\\}$. In _3-D_ , the task
pool $\mathcal{Q}_{pool}$ is defined as
$\mathcal{Q}_{pool}\coloneqq\\{(w_{s},v_{e}/v_{p},v_{o}/v_{p})|0.5\leq
w_{s}\leq 1.0,1.3\leq v_{e}/v_{p}\leq 5.0,1.0\leq v_{o}/v_{p}\leq 5.0\\}$.

###  V-C Evaluation Metrics

We consider three statistical metrics to capture different characteristics of
a particular pursuit strategy. Every experiment is repeated over three seeds.
Each evaluation score is expressed in the format of "mean (standard
deviation)", which is averaged over a total of 1500 testing episodes, i.e.,
500 episodes per random seed.

  * •

Capture Rate: This metric shows the average success rate of capturing the
evader in an episode. We define a successful capture as the evader being
captured at least once in an episode. Higher _capture rate_ indicates stronger
capture abilities.

  * •

Capture Frequency: This metric represents the average number of captures per
timestep in an episode. Higher _capture frequency_ indicates faster capture.

  * •

Coverage: This metric denotes the proportion of tasks in the task pool with
_capture rate_ greater than $95\%$. A higher _coverage_ ratio reflects greater
adaptability across various tasks.

Tasks | Scenarios | Metrics | Angelani | Janosov | APF | DACOOP | MAPPO | TFS
---|---|---|---|---|---|---|---|---
_2-D_ | _Challenging_ | Cap. Rate$\uparrow$ |  0.717(0.021) |  0.780(0.010) |  0.743(0.015) |  0.803(0.015) |  0.769(0.119) | 0.957(0.009)
Cap. Fre.$\uparrow$ |  0.007(0.000) |  0.008(0.000) |  0.007(0.000) |  0.012(0.000) |  0.039(0.028) | 0.085(0.013)
_Dynamic_ | Cap. Rate$\uparrow$ |  0.733(0.021) |  0.737(0.021) |  0.687(0.015) |  0.757(0.021) |  0.880(0.073) | 0.989(0.002)
Cap. Fre.$\uparrow$ |  0.011(0.000) |  0.010(0.001) |  0.009(0.000) |  0.013(0.000) |  0.137(0.038) | 0.219(0.001)
_Diverse_ | Cap. Rate$\uparrow$ |  0.807(0.006) |  0.827(0.015) |  0.810(0.010) |  0.833(0.006) |  0.916(0.047) | 0.985(0.002)
Cap. Fre.$\uparrow$ |  0.013(0.000) |  0.016(0.000) |  0.014(0.000) |  0.018(0.000) |  0.105(0.048) | 0.191(0.005)
Coverage$\uparrow$ |  0.136(0.036) |  0.273(0.016) |  0.150(0.017) |  0.250(0.014) |  0.417(0.416) | 1.000(0.000)
_3-D_ | _Challenging_ | Cap. Rate$\uparrow$ |  0.231(0.016) |  0.186(0.028) |  0.213(0.019) |  0.260(0.018) |  0.540(0.379) | 0.909(0.013)
Cap. Fre.$\uparrow$ |  0.002(0.000) |  0.002(0.000) |  0.001(0.001) |  0.003(0.001) |  0.151(0.116) | 0.308(0.031)
_Dynamic_ | Cap. Rate$\uparrow$ |  0.333(0.035) |  0.275(0.020) |  0.365(0.037) |  0.389(0.028) |  0.683(0.433) | 0.992(0.001)
Cap. Fre.$\uparrow$ |  0.006(0.001) |  0.004(0.001) |  0.005(0.000) |  0.005(0.000) |  0.210(0.157) | 0.492(0.016)
_Diverse_ | Cap. Rate$\uparrow$ |  0.387(0.030) |  0.375(0.032) |  0.398(0.038) |  0.460(0.034) |  0.659(0.440) | 0.990(0.001)
Cap. Fre. $\uparrow$ |  0.006(0.001) |  0.006(0.001) |  0.003(0.000) |  0.009(0.001) |  0.387(0.288) | 0.895(0.107)
Coverage$\uparrow$ |  0.00(0.000) |  0.00(0.000) |  0.00(0.000) |  0.00(0.000) |  0.552(0.391) | 0.979(0.007)
TABLE I: Results of TFS and baselines in _2-D_ and _3-D_.

(a) Capture rate. (b) Capture frequency.

(c) Capture rate. (d) Capture frequency.

(e) Coverage.

Figure 4: We compare MAPPO and MAPPO-manual with TFS in _2-D_. (a-b) Capture
rate and capture frequency in _Challenging_. (c-e) Capture rate, capture
frequency and coverage in _Diverse_.

###  V-D Baselines

We challenge TFS with three heuristic algorithms (Angelani, Janosov, and APF)
and two RL-based algorithms (DACOOP and MAPPO).

  * •

Angelani [5]: Angelani describes a pursuit-evasion phenomenon arising from
intergroup interactions, where pursuers are attracted to the nearest particles
of the opposing group, i.e., the evader.

  * •

Janosov [4]: Following the concepts of the Vicsek particle model [5], Janosov
designs greedy chasing of evaders and collision avoidance in a realistic
scenario with inertia, time delay and noise.

  * •

APF [31]: APF guides pursuers to a target position via the combination of
attractive, repulsive, and inter-individual forces. By setting the target of
pursuers to the evader’s position and adjusting the hyperparameters for
different forces, pursuers can move towards the evader with obstacle avoidance
in APF.

  * •

DACOOP [14]: DACOOP is an RL-based adaptive cooperative pursuit algorithm that
combines RL and APF. DACOOP uses RL to modify the primary hyperparameters of
APF to adapt to different task conditions.

  * •

MAPPO [30]: MAPPO is one of the most advanced multi-agent reinforcement
learning algorithms for general cooperative problems. We consider MAPPO as a
naive baseline in which we use random sampling from the task pool to construct
the training distribution for policy training instead of CL modules.

(a) Capture rate. (b) Capture frequency.

(c) Coverage.

Figure 5: Ablations on the sample probability in _Diverse_.

(a) Capture rate. (b) Capture frequency.

(c) Coverage.

Figure 6: Ablations on the decay coefficient in _Diverse_.

(a) Side length. (b) Speed ratio.

(c) Number of obstacles.

Figure 7: Visualization of task variables in the training distribution of TFS,
MAPPO and MAPPO-manual in _2-D_.

(a) Learned strategy of TFS. (b) Learned strategy of DACOOP.

Figure 8: The comparison in learned strategies of TFS and DACOOP in _3-D_.

###  V-E Main Results

We compare TFS with all baselines in _2-D_ and _3-D_ with three scenarios,
_Challenging_ , _Dynamic_ and _Diverse_ in Tab. I. For a fair comparison, we
train all RL-based methods for a total number of 300 million timesteps in
_2-D_ and 10 billion timesteps in _3-D_.

####  V-E1 Challenging

The results in Tab. I show that TFS outperforms all baselines with a clear
margin in _Challenging_ , which indicates that TFS is able to solve multi-
agent pursuit problems with extremely hard conditions. Specifically, TFS
obtains over $90\%$ capture rate in _Challenging_ and gains a $2.18\times$
improvement in _2-D_ as well as $2.05\times$ boost in _3-D_ in capture
frequency compared to the best baseline. All heuristic methods achieve
_capture rate_ below $90\%$ and obtain much lower _capture frequency_ than
TFS. This is because the hard-coded capture policy is conservative and poorly
cooperative, failing to capture evaders in tasks with large scenes with
multiple obstacles and significant speed differences between the evader and
the pursuers. DACOOP, which is slightly better than APF, dynamically adjusts
the hyperparameters of APF using RL, but its policy still lacks cooperation.
MAPPO is the best competitor among all the baselines, but it is data
inefficient and challenging to converge in a reasonable amount of time. We
also compare MAPPO with TFS with the training curves to demonstrate the
effectiveness of our curriculum learning module in Sec. V-F.

####  V-E2 Dynamic

As shown in Tab. I, TFS is robust to the dynamically changing scenario and
achieves the highest _capture rate_ and _capture frequency_ in _Dynamic_.
Specifically, TFS achieves close to $100\%$ capture rate and obtains
$1.60\times$ capture frequency in _2-D_ as well as $2.33\times$ capture
frequency in _3-D_ over the best baseline. Since all heuristic methods require
different hyperparameters for different task conditions, such methods cannot
be directly applied to scenarios with dynamically changing conditions. The
policy of DACOOP lacks cooperation and performs poorly due to complex task
conditions in the second half of an episode. MAPPO also performs worse than
TFS on the challenging tasks in the last half of an episode.

####  V-E3 Diverse

In _Diverse_ , TFS is the only algorithm that can solve the entire task pool.
In summary, TFS achieves close to $100\%$ capture rate and coverage in
_Diverse_. Moreover, our method obtains $1.82\times$ capture frequency in
_2-D_ and $2.37\times$ capture frequency in _3-D_ over the best baseline. All
heuristic methods and DACOOP are unable to solve tasks with diverse conditions
and have less than $30\%$ coverage of the task pool. The policy in the hard-
coded methods is poorly cooperative and can not solve complex tasks in the
task pool. MAPPO can solve easy tasks well but performs poorly on the
challenging tasks in the task pool, resulting in a high _capture rate_ and
_capture frequency_ but low _coverage_. This is due to the fact that MAPPO
samples uniformly from the task pool for training, so it cannot target to
improve the performance on hard tasks. TFS, on the other hand, can dynamically
adjust the training distribution to target moderately difficult tasks, thus
obtaining good coverage of the entire task pool.

###  V-F Abaltion Studies

To illustrate the effectiveness of the curriculum learning module, we compare
the training curves and visualize the tasks generated by TFS, MAPPO and MAPPO-
manual, which is a manually designed curriculum method, in the _Challenging_
and _Diverse_ scenarios in _2-D_. We also analyze the sensitivity of TFS to
the sample probability $p$ and the decay coefficient $\alpha$ by comparing the
training curves in _2-D_.

Effectiveness of the curriculum learning module. To illustrate the
effectiveness of TFS for training efficiency, we compare the training curves
of TFS with MAPPO and MAPPO-manual in Fig. 4. In MAPPO-manual, We manually
design a series of tasks to train the pursuers in order from easy to hard.
When the _capture rate_ on the current task is higher than $0.95$, we transfer
the policy to the next task and continue training until the policy converges
on the final task. The results in Fig. 4 show that MAPPO cannot converge
within a limited number of training samples and has poor performance. MAPPO-
manual is comparable to TFS in terms of _capture frequency_ but worse than TFS
in terms of _capture rate_. MAPPO-manual also has a significant variance in
_capture rate_ and _capture frequency_ because manually designed curricula are
based on human experience in dividing the difficulty of tasks and may not be
the most appropriate training tasks for the current policy. In contrast, TFS
can automatically select training tasks and speed up convergence. In _Diverse_
, the results in Fig. 4 show that MAPPO converges very slowly. MAPPO-manual
first gets better and then worse with training because MAPPO-manual overfits
to the current task and finally converges to the last task, leading to
cataclysmic forgetting in already learned tasks. In contrast, TFS samples
tasks from both $\mathcal{Q}_{cur}$ and $\mathcal{Q}_{pool}$, keeping
efficient training while not forgetting tasks that have already been learned.

As shown in Fig. 8, we visualize the generated tasks of TFS, MAPPO, and MAPPO-
manual by the mean-standard deviation curves of task variables in the training
distribution $\mathcal{Q}_{train}$ during training. We find that the training
tasks generated by TFS followed the pattern from easy to hard, i.e., from
tasks with small scenes with few obstacles and a slow evader to tasks with
large scenes with many obstacles and a fast evader. In the later stage of
training, TFS still keeps easy tasks in $\mathcal{Q}_{train}$ to prevent
forgetting. MAPPO, on the other hand, samples uniformly from the task pool
from the beginning to the end, which leads to slower convergence and poor
performance. MAPPO-manual also generates tasks from easy to hard, but the
training tasks in MAPPO-manual are fixed in each phase, which leads to
forgetting the previously learned tasks.

Choice of sample probability $p$. We investigate the choice of the sample
probability $p$ in _2-D_ in Fig. 8. As shown in Fig. 8, we need more samples
from the curriculum archive $\mathcal{Q}_{cur}$ than the task pool
$\mathcal{Q}_{pool}$ in the training batch, and sampling from the task pool
$\mathcal{Q}_{pool}$ ensures global exploration. When $p$ is too small, TFS
degenerates into MAPPO, resulting in slow convergence. When $p=1.0$, the lack
of global exploration also leads to poor performance at the beginning of
training. Finally, we choose $p=0.9$ in both _2-D_ and _3-D_.

Choice of decay coefficient $\alpha$. We perform experiments in _2-D_ to
analyze the sensitivity of the hyperparameter $\alpha$. As shown in Fig. 8, we
need to choose the decay coefficient carefully for the specific tasks. The
main principle is that the decay coefficient should not be too large. If
$\alpha$ is too large, the probability of sampling from the curriculum archive
$\mathcal{Q}_{cur}$ rapidly decreases, and TFS degrades to MAPPO, leading to a
large variance and slow convergence. Finally we suggest to choose $\alpha=2.0$
in _2-D_ and $\alpha=0.0$ in _3-D_.

###  V-G Behavior Analysis

To illustrate the cooperativeness of pursuit strategies in TFS, we visualize
the policy of TFS and a strong baseline, DACOOP in _3-D_. As shown in Fig. 8,
we find that the capture strategy in TFS comprises three key phases: chasing,
formation, and staying in formation on approach. Initially, the drones move to
intercept the target and establish an encirclement. Once the perimeter is set,
the drones progressively close in on the evader, adjusting the formation to
eliminate potential escape routes. In DACOOP, pursuers form a circle through
hard-coded strategies. However, there is no cooperation among pursuers, and it
is difficult to adjust the perimeter in real time to enclose all escape routes
of the evader so that the fast evader can easily escape from the pursuers’
circle.

##  VI CONCLUSION

We introduce the TaskFlex Solver, a novel approach that combines reinforcement
learning and curriculum learning to effectively address the multi-agent
pursuit problem under diverse and dynamic task conditions. Our method
outperforms existing approaches, consistently achieving close to $100\%$
capture rates in both 2-dimensional and 3-dimensional scenarios. Future work
can explore the extension of the TaskFlex Solver to other robotic
applications.

## References

  * [1] V. Turetsky and J. Shinar, “Missile guidance laws based on pursuit–evasion game formulations,” _Automatica_ , vol. 39, no. 4, pp. 607–618, 2003.
  * [2] J. M. Eklund, J. Sprinkle, and S. S. Sastry, “Switched and symmetric pursuit/evasion games using online model predictive control with application to autonomous aircraft,” _IEEE Transactions on Control Systems Technology_ , vol. 20, no. 3, pp. 604–620, 2011.
  * [3] D. W. Oyler, P. T. Kabamba, and A. R. Girard, “Pursuit–evasion games in the presence of obstacles,” _Automatica_ , vol. 65, pp. 1–11, 2016.
  * [4] M. Janosov, C. Virágh, G. Vásárhelyi, and T. Vicsek, “Group chasing tactics: how to catch a faster prey,” _New Journal of Physics_ , vol. 19, no. 5, p. 053003, 2017.
  * [5] L. Angelani, “Collective predation and escape strategies,” _Physical review letters_ , vol. 109, no. 11, p. 118104, 2012.
  * [6] X. Fang, C. Wang, L. Xie, and J. Chen, “Cooperative pursuit with multi-pursuer and one faster free-moving evader,” _IEEE transactions on cybernetics_ , vol. 52, no. 3, pp. 1405–1414, 2020.
  * [7] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, “Continuous control with deep reinforcement learning,” _arXiv preprint arXiv:1509.02971_ , 2015.
  * [8] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller, “Playing atari with deep reinforcement learning,” _arXiv preprint arXiv:1312.5602_ , 2013.
  * [9] J. K. Gupta, M. Egorov, and M. Kochenderfer, “Cooperative multi-agent control using deep reinforcement learning,” in _Autonomous Agents and Multiagent Systems: AAMAS 2017 Workshops, Best Papers, São Paulo, Brazil, May 8-12, 2017, Revised Selected Papers 16_. Springer, 2017, pp. 66–83.
  * [10] W. Wang, T. Yang, Y. Liu, J. Hao, X. Hao, Y. Hu, Y. Chen, C. Fan, and Y. Gao, “From few to more: Large-scale dynamic multiagent curriculum learning,” 2019\.
  * [11] S. F. Desouky and H. M. Schwartz, “Q ($\lambda$)-learning adaptive fuzzy logic controllers for pursuit–evasion differential games,” _International Journal of Adaptive Control and Signal Processing_ , vol. 25, no. 10, pp. 910–927, 2011.
  * [12] M. D. Awheda and H. M. Schwartz, “The residual gradient facl algorithm for differential games,” in _2015 IEEE 28th Canadian Conference on Electrical and Computer Engineering (CCECE)_. IEEE, 2015, pp. 1006–1011.
  * [13] Y. Wang, L. Dong, and C. Sun, “Cooperative control for multi-player pursuit-evasion games with reinforcement learning,” _Neurocomputing_ , vol. 412, pp. 101–114, 2020.
  * [14] Z. Zhang, X. Wang, Q. Zhang, and T. Hu, “Multi-robot cooperative pursuit via potential field-enhanced reinforcement learning,” in _2022 International Conference on Robotics and Automation (ICRA)_. IEEE, 2022, pp. 8808–8814.
  * [15] R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch, “Multi-agent actor-critic for mixed cooperative-competitive environments,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [16] B. Xu, F. Gao, C. Yu, R. Zhang, Y. Wu, and Y. Wang, “Omnidrones: An efficient and flexible platform for reinforcement learning in drone control,” _arXiv preprint arXiv:2309.12825_ , 2023.
  * [17] C. Muro, R. Escobedo, L. Spector, and R. Coppinger, “Wolf-pack (canis lupus) hunting strategies emerge from simple rules in computational simulations,” _Behavioural processes_ , vol. 88, no. 3, pp. 192–197, 2011.
  * [18] L. Xu, B. Hu, Z. Guan, X. Cheng, T. Li, and J. Xiao, “Multi-agent deep reinforcement learning for pursuit-evasion game scalability,” in _Proceedings of 2019 Chinese Intelligent Systems Conference: Volume I 15th_. Springer, 2020, pp. 658–669.
  * [19] M. Hüttenrauch, S. Adrian, G. Neumann, _et al._ , “Deep reinforcement learning for swarm systems,” _Journal of Machine Learning Research_ , vol. 20, no. 54, pp. 1–31, 2019.
  * [20] Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculum learning,” in _Proceedings of the 26th annual international conference on machine learning_ , 2009, pp. 41–48.
  * [21] R. Portelas, C. Colas, L. Weng, K. Hofmann, and P.-Y. Oudeyer, “Automatic curriculum learning for deep rl: A short survey,” _arXiv preprint arXiv:2003.04664_ , 2020.
  * [22] C. Florensa, D. Held, M. Wulfmeier, M. Zhang, and P. Abbeel, “Reverse curriculum generation for reinforcement learning,” in _Conference on robot learning_. PMLR, 2017, pp. 482–495.
  * [23] C. Florensa, D. Held, X. Geng, and P. Abbeel, “Automatic goal generation for reinforcement learning agents,” in _International conference on machine learning_. PMLR, 2018, pp. 1515–1528.
  * [24] B. Ivanovic, J. Harrison, A. Sharma, M. Chen, and M. Pavone, “Barc: Backward reachability curriculum for robotic reinforcement learning,” in _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, 2019, pp. 15–21.
  * [25] I. Akkaya, M. Andrychowicz, M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, _et al._ , “Solving rubik’s cube with a robot hand,” _arXiv preprint arXiv:1910.07113_ , 2019.
  * [26] B. Mehta, M. Diaz, F. Golemo, C. J. Pal, and L. Paull, “Active domain randomization,” in _Conference on Robot Learning_. PMLR, 2020, pp. 1162–1176.
  * [27] J. Chen, Y. Zhang, Y. Xu, H. Ma, H. Yang, J. Song, Y. Wang, and Y. Wu, “Variational automatic curriculum learning for sparse-reward cooperative multi-agent problems,” _Advances in Neural Information Processing Systems_ , vol. 34, pp. 9681–9693, 2021.
  * [28] Q. Long, Z. Zhou, A. Gupta, F. Fang, Y. Wu, and X. Wang, “Evolutionary population curriculum for scaling multi-agent reinforcement learning,” in _International Conference on Learning Representations_ , 2020.
  * [29] C. De Souza, R. Newbury, A. Cosgun, P. Castillo, B. Vidolov, and D. Kulić, “Decentralized multi-agent pursuit using deep reinforcement learning,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 3, pp. 4552–4559, 2021\.
  * [30] C. Yu, A. Velu, E. Vinitsky, Y. Wang, A. Bayen, and Y. Wu, “The surprising effectiveness of ppo in cooperative, multi-agent games,” _arXiv preprint arXiv:2103.01955_ , 2021.
  * [31] Y. Koren, J. Borenstein, _et al._ , “Potential field methods and their inherent limitations for mobile robot navigation.” in _Icra_ , vol. 2, no. 1991, 1991, pp. 1398–1404.