In the non-relativistic limit, the electric part and the magnetic part of
electromagnetism are decoupled. Starting with a non-relativistic many–electron
system, one may ask what is the quantum mechanical analog of the Poisson
equation that forms the basis of electrostatics. It turns out that this is
easily obtained via computing $\left.\delta Z[J]/\delta J\right|_{J=0}$ in two
different ways. First, upon taking the derivative of (15) and using Eqs. (16)
and (17), one obtains

$\left.\frac{\delta Z[J]}{\delta J({\bf x})}\right|_{J=0}=-\beta
Z_{J=0}\langle\psi^{{\dagger}}({\bf x})\psi({\bf x})\rangle=-\beta
Z_{J=0}\frac{\langle e\psi^{{\dagger}}({\bf x})\psi({\bf x})\rangle}{e}\;.$

Second, while taking the derivative of (15), if using Eqs. (19) and (20) one
arrives at

$\left.\frac{\delta Z[J]}{\delta J({\bf x})}\right|_{J=0}=-\beta
Z_{J=0}\langle i\phi\rangle=-\beta
Z_{J=0}\,e\;U^{-1}{\scriptstyle\circ}\langle A_{0}\rangle=\beta
Z_{J=0}\frac{1}{4\pi e}\nabla^{2}_{{\bf x}}\langle A_{0}({\bf x})\rangle\;.$

We therefore obtain the thermal quantum-mechanical analog of the Poisson
equation

$\nabla^{2}_{{\bf x}}\langle A_{0}({\bf x})\rangle=-4\pi\langle
e\psi^{{\dagger}}({\bf x})\psi({\bf x})\rangle\;,$

a result also obtained in reference Valiev and Fernando, 1996. This connection
to classical electrostatics is essential since it provides the quantum-
mechanical correspondence of an important ingredient in (bio)molecular
interactions that have been extensively studied in the presence of
dielectrics. Yu (2003); Doerr and Yu (2004, 2006); Obolensky et al. (2009)

The UDF described in this paper is systematically constructed, uniquely
determined, and in principle exact. However, in terms of real computations,
one can only keep $\Gamma_{i}$ terms up to some order in $\lambda$. A natural
question thus arises. How well will the truncated version work? In general,
this question can only be answered with numerical results. However, by
providing theoretical arguments and comparisons to other approaches, we wish
to convey that this method is likely to produce good results and thus to
attract computational efforts towards using the proposed approach.

It is worth pointing out the relation between the expression (46) and the
scheme Polonyi and Sailer (2002) motivated by the renormalization-group. The
vertex functions $I^{(j\leq l)}$ in (46) will contribute to the so-called
$l$-local approximation of reference Polonyi and Sailer, 2002. The absence of
$I^{(2)}$ manifests the absence of the correction term due to the bi-local
contribution, as shown in reference Polonyi and Sailer, 2002. Furthermore,
with the bilocal approximation Polonyi and Sailer (2002) included, Polonyi and
Sailer obtained an approximate energy functional which corresponds exactly to
our $\Gamma_{0}+\Gamma_{1}$. To reach an equivalent form of the proposed
$l$-local approximation of reference Polonyi and Sailer, 2002, we simply keep
terms up to $\Gamma_{l/2}$. Therefore, our formulation provides an explicit
means for achieving an $l$-local approximation without resorting to the
Hellmann-Feynman theorem.

As shown in (87), the propagator $\tilde{\mathcal{D}}_{0}$ can be expanded as

$\tilde{\mathcal{D}}_{0}=U+U{\scriptstyle\circ}D_{0}{\scriptstyle\circ}\,U+U{\scriptstyle\circ}D_{0}{\scriptstyle\circ}\,U{\scriptstyle\circ}D_{0}{\scriptstyle\circ}\,U+\ldots\;,$

when $U$ can be viewed as a small quantity and treated perturbatively. When
one is not allowed to treat $U$ as a small parameter (say in the strong
coupling regime), or when one needs to treat $U^{-1}$ as a small parameter
instead, the conventional perturbative expansion in $e^{2}$ breaks down
completely while our approach is still applicable. In the case when $U^{-1}$
must be treated as small, we expand $\tilde{\mathcal{D}}_{0}$ as

$\tilde{\mathcal{D}}_{0}=-D_{0}^{-1}-D_{0}^{-1}{\scriptstyle\circ}\,U^{-1}{\scriptstyle\circ}D_{0}^{-1}-D_{0}^{-1}{\scriptstyle\circ}\,U^{-1}{\scriptstyle\circ}D_{0}^{-1}{\scriptstyle\circ}\,U^{-1}{\scriptstyle\circ}D_{0}^{-1}-\ldots\;.$
(213)

And in this case, $U\gg 1$, our effective action expansion does have the
Hartree term $\frac{1}{2}n{\scriptstyle\circ}\,U{\scriptstyle\circ}n$ as the
leading order, followed by terms of order $U^{0}$ and then the expansion of
$\tilde{\mathcal{D}}_{0}$ provides series in powers of $U^{-1}$. Note that in
this case, the exchange-correlation functional is not led by order $U$ at all,
but is led by order $U^{0}$. This feature is not present in the conventional
perturbative approach using $U$ (or $e^{2}$) as the expansion parameter.

As mentioned earlier, there also exist different functional methods for many
electron systems. For example, the exchange correlation functional outlined by
Sham Sham (1985) is founded on the perturbative functional approach developed
by Luttinger and Ward Luttinger and Ward (1960) or equivalently by Klein.Klein
(1961) A succinct review of the Luttinger-Ward/Klein functional and its
applications can be found in reference Kotliar et al., 2006. The Luttinger-
Ward/Klein functional yields the grand-potential/ground-state-energy only when
the functional argument is equal to the fully-interacting, physical, one-
particle Green’s function. Instead of allowing the physical, full, one-
particle Green’s function, Dahlen et al. Dahlen et al. (2006) proposed to find
that stationary point of the Luttinger-Ward/Klein functional while restricting
the argument to the Hartree-Fock Green’s functions or the Green’s functions of
non-interacting systems. However, even if the Luttinger-Ward/Klein functional
is computed to all orders, the error in the value of the grand potential (or
the ground state energy) due to restriction on the Green’s functions remains
unknown.

Furthermore, it should be noted that when the functional argument is equal to
the physical one-particle Green’s function, the Luttinger-Ward/Klein
functional reaches a stationary point, not the minimum.Klein (1961) This means
that it is possible for the Klein/Luttinger-Ward functional to assume an even
lower value than the ground state energy (or the grand potential) when the
functional argument deviates from the physical Green’s function. In other
words, the Klein/Luttinger-Ward functional only retains its meaning as the
ground state energy (or grand potential) when the Green’s function takes the
value of the true (physical) Green’s function. Our effective action expression
of the energy functional, on the other hand, truly represents energy of the
system. Our effective action energy functional, when no truncation on the
series is made, reaches its minimum when the electron density assumes the true
(physical) density, and for any other $\upsilon$-representable density profile
prescribed, it represents the lowest energy possible associated with that
prescribed density profile.

The method of Hedin Hedin (1965) is largely identical to that of Luttinger and
Ward.Luttinger and Ward (1960) This includes the fact that the energy
functional reaches a stationary point, rather than the minimum, when the
functional argument is the fully-interacting one-particle Green’s function.
However, Hedin aims to replace the $e^{2}$-based (bare Coulomb) perturbative
expansion of the electron self energy by another expansion using a screened
interaction ${\mathcal{W}}$. Hedin expresses the electron self energy and the
screened interaction as a functional of the electron Green’s function of the
interacting system. Interestingly, the first order result, also termed the GW
approximation, of Hedin Hedin (1965) has been shown Dahlen et al. (2006) to
produce good results when compared to other density functionals. This suggests
that not treating $e^{2}$ as small might have some advantage. It is worth
pointing out that the first order term,
$\Gamma_{1}=-\frac{1}{2}\text{Tr}\ln(\tilde{\mathcal{D}}_{0}^{-1}{\scriptstyle\circ}\,U)$,
of the UDF described here is equivalent to the celebrated GW approximation.

Like the ${\mathcal{W}}$ propagator of Hedin,Hedin (1965) the
$\tilde{\mathcal{D}}_{0}$ propagator introduced here also corresponds to that
of a screened interaction (see section IV.4), thereby avoiding any possible
infrared divergence associated with perturbative expansion based on bare
Coulomb interactions. However, the screening associated with
$\tilde{\mathcal{D}}_{0}$ is from the KS particles and thus keeps the same
form no matter how many orders one wishes to include. This is different from
that of Hedin’s where the expression of ${\mathcal{W}}$ in terms of the
electron Green’s function changes with the order included. The other
difference between the proposed approach and reference Hedin, 1965 is that the
UDF proposed here depends on $J_{0}$, a function of three spatial variables
(and possibly with one additional time variable), while the method of
reference Hedin, 1965 expresses via ${\mathcal{W}}$ the electron self energy
as a functional of the Green’s function, a function of six spatial variables
(and possibly with two additional time variables).

It is well known that a loopwise expansion may also be viewed as an $\hbar$
expansion Itzykson and Zuber (1980), that is, an expansion of quantum-
mechanical effects. By first integrating out the fermionic degrees of freedom
completely, the proposed method is an expansion of bosonic loops formed by
$\tilde{\mathcal{D}}_{0}$ propagators associated with the auxiliary field $b$.
The $b$ field describes the potential produced by electron density
fluctuations around $n_{g}$. Since the ground state charge density $n_{g}$
captures the full quantum information of the ground state thanks to the HK
theorem, one anticipates a weaker quantum effect associated with the auxiliary
$b$ field than with the fermionic field. This makes the auxiliary $b$ field a
suitable candidate for loop (or quantum effect) expansion, the approach
pursued in this paper.

Finally, let us remark on the issue of convexity. The full $\Gamma[n]$ is
supposed to be convex, Valiev and Fernando (1997) thus guaranteeing a unique
solution without any local minima when searching for the minimum of
$\Gamma[n]$. However, in real computations only a finite number of terms of
the effective action can be kept. This approximate/truncated expression may
not warrant convexity and thus it is not guaranteed to be free of local minima
while numerically searching for the ground state density $n_{g}$ (or thermal
averaged density $n_{T}$ at finite temperature). In the near future, we plan
to implement numerically the methods presented in this paper, and will
describe in a separate publication the results obtained as well as the
investigation on the issue of convexity.

## Acknowledgement

This research was supported by the Intramural Research Program of the National
Library of Medicine of the National Institutes of Health. The author thanks
Dr. Oleg Obolensky and Professor John Neumeier for useful comments. He is
particularly indebted to Professor Richard Friedberg, who has provided
numerous useful suggestions and correspondence during the writing of the
paper. The Feynman diagrams are made using the style file contained in the
package developed in reference Binosi and Theußl, 2004

## References

  * Kohn (1999) W. Kohn, Rev. Mod. Phys. 71, 1253 (1999).
  * Hohenberg and Kohn (1964) P. Hohenberg and W. Kohn, Phys. Rev. 136, B864 (1964).
  * Mermin (1965) N. D. Mermin, Phys. Rev. 137, A1441 (1965).
  * Kohn and Sham (1965) W. Kohn and L. J. Sham, Phys. Rev. 140, A1133 (1965).
  * Hohenberg et al. (1990) P. C. Hohenberg, W. Kohn, and L. J. Sham, in _Density Functional Theory of Many-Fermion Systems_ , edited by P.-O. Lowdin (Academic Press, 1990), vol. 21 of _Advances in Quantum Chemistry_ , pp. 7 – 26.
  * Andersson and Gru1ning (2004) S. Andersson and M. Gru1ning, J. Phys. Chem. A 108, 7621 (2004).
  * Sousa et al. (2007) S. F. Sousa, P. A. Fernandes, and M. J. Ramos, J. Phys. Chem. A 111, 10439 (2007).
  * Tekarli et al. (2009) S. M. Tekarli, M. L. Drummond, T. G. Williams, T. R. Cundari, and A. K. Wilson, J. Phys. Chem. A 113, 86078614 (2009).
  * Kosztyu and Lendvay (2009) R. Kosztyu and G. Lendvay, Reaction Kinetics and Catalysis Letters 96, 233 (2009).
  * Kümmel and Kronik (2008) S. Kümmel and L. Kronik, Rev. Mod. Phys. 80, 3 (2008).
  * Cohen et al. (2008) A. J. Cohen, P. Mori-Sanchez, and W. Yang, Science 321, 792 (2008).
  * Kohn (1983) W. Kohn, Phys. Rev. Lett. 51, 1596 (1983).
  * Lieb (1982) E. H. Lieb, in _Physics as Natural Philosophy: Essays in Honor of Laszlo Tisza on His 75th Birthday_ , edited by A. Shimony and H. Feshbach (MIT Press, 1982), pp. 111 – 149.
  * Levy (1982) M. Levy, Phys. Rev. A 26, 1200 (1982).
  * Valiev and Fernando (1995) M. M. Valiev and G. W. Fernando, Phys. Rev. B 52, 10697 (1995).
  * Janak (1978) J. F. Janak, Phys. Rev. B 18, 7165 (1978).
  * Jona-Lasinio (1964) G. Jona-Lasinio, Il Nuovo Cimento 34, 1790 (1964).
  * Fukuda et al. (1994) R. Fukuda, T. Kotani, Y. Suzuki, and S. Yokojima, Prog. Theor. Phys. 92, 833 (1994).
  * Fukuda et al. (1995) R. Fukuda, M. Komachiya, S. Yokojima, Y. Suzuki, K. Okumura, and T. Inagaki, Prog. Theo. Phys. Suppl. 121, 1 (1995).
  * Valiev and Fernando (1997) M. Valiev and G. W. Fernando (1997), eprint cond-mat/9702247.
  * Polonyi and Sailer (2002) J. Polonyi and K. Sailer, Phys. Rev. B 66, 155113 (2002).
  * Jackiw (1974) R. Jackiw, Phys. Rev. D 9, 1686 (1974).
  * Valiev and Fernando (1996) M. Valiev and G. W. Fernando, Phys. Rev. B 54, 7765 (1996).
  * Görling and Levy (1994) A. Görling and M. Levy, Phys. Rev. A 50, 196 (1994).
  * Luttinger and Ward (1960) J. M. Luttinger and J. C. Ward, Phys. Rev. 118, 1417 (1960).
  * Sham and Schlüter (1983) L. J. Sham and M. Schlüter, Phys. Rev. Lett. 51, 1888 (1983).
  * Sham (1985) L. J. Sham, Phys. Rev. B 32, 3876 (1985).
  * Tokatly and Pankratov (2001) I. V. Tokatly and O. Pankratov, Phys. Rev. Lett. 86, 2078 (2001).
  * Negele and Orland (1988) J. W. Negele and H. Orland, _Quantum Many-Particle Systems_ (Addison-Wesley, Redwood city, CA, 1988).
  * Dahlen et al. (2006) N. E. Dahlen, R. van Leeuwen, and U. von Barth, Phys. Rev. A 73, 012511 (2006).
  * Hedin (1965) L. Hedin, Phys. Rev. 139, A796 (1965).
  * Hellmann (1937) H. Hellmann, _Einfḧrung in die Quantenchemie_ (Leipzig: Franz Deuticke, 1937).
  * Feynman (1939) R. P. Feynman, Phys. Rev. 56, 340 (1939).
  * Sharp and Horton (1953) R. T. Sharp and G. K. Horton, Phys. Rev. 90, 317 (1953).
  * Talman and Shadwick (1976) J. D. Talman and W. F. Shadwick, Phys. Rev. A 14, 36 (1976).
  * Petersilka et al. (1996) M. Petersilka, U. J. Gossmann, and E. K. U. Gross, Phys. Rev. Lett. 76, 1212 (1996).
  * Kümmel and Perdew (2003) S. Kümmel and J. P. Perdew, Phys. Rev. B 68, 035103 (2003).
  * Casida (1995a) M. E. Casida, Phys. Rev. A 51, 2005 (1995a).
  * von Barth et al. (2005) U. von Barth, N. E. Dahlen, R. van Leeuwen, and G. Stefanucci, Phys. Rev. B 72, 235109 (2005).
  * Baroni et al. (2001) S. Baroni, S. de Gironcoli, A. Dal Corso, and P. Giannozzi, Rev. Mod. Phys. 73, 515 (2001).
  * (41) R. Furnstahl, _EFT for DFT_ , eprint nucl-th/0702040.
  * Jones and Gunnarsson (1989) R. O. Jones and O. Gunnarsson, Rev. Mod. Phys. 61, 689 (1989).
  * Bartlett and Musiał (2007) R. J. Bartlett and M. Musiał, Reviews of Modern Physics 79, 291 (pages 62) (2007).
  * Hellgren and von Barth (2007) M. Hellgren and U. von Barth, Phys. Rev. B 76, 075107 (2007).
  * Perdew and Zunger (1981) J. P. Perdew and A. Zunger, Phys. Rev. B 23, 5048 (1981).
  * DuBois (1959) D. F. DuBois, Annals of Physics 7, 174 (1959).
  * Wigner (1934) E. Wigner, Phys. Rev. 46, 1002 (1934).
  * Fetter and Walecka (1971) A. L. Fetter and J. D. Walecka, _Quantum Theory of Many-Particle Systems_ (McGraw-Hill: San Francisco, 1971).
  * Jones and March (1973) W. Jones and N. H. March, _Theoretical Solid State Physics_ (Wiley: New York, 1973).
  * (50) J. Goldstone, Proceedings of the Royal Society of London. Series A, Mathematical and Physical Sciences 239, 267 (????).
  * Brueckner and Levinson (1955) K. A. Brueckner and C. A. Levinson, Phys. Rev. 97, 1344 (1955).
  * Klein (1961) A. Klein, Phys. Rev. 121, 950 (1961).
  * Brueckner and Gammel (1958) K. A. Brueckner and J. L. Gammel, Phys. Rev. 109, 1023 (1958).
  * Gell-Mann and Brueckner (1957) M. Gell-Mann and K. A. Brueckner, Phys. Rev. 106, 364 (1957).
  * Carr and Maradudin (1964) W. J. Carr and A. A. Maradudin, Phys. Rev. 133, A371 (1964).
  * Ferrell (1958) R. A. Ferrell, Phys. Rev. Lett. 1, 443 (1958).
  * Kohn and Luttinger (1960) W. Kohn and J. M. Luttinger, Phys. Rev. 118, 41 (1960).
  * Gell-Mann and Low (1951) M. Gell-Mann and F. Low, Phys. Rev. 84, 350 (1951).
  * Runge and Gross (1984) E. Runge and E. K. U. Gross, Phys. Rev. Lett. 52, 997 (1984).
  * Casida (1995b) M. E. Casida, in _Recent advances in density functional methods, part I_ , edited by D. P. Chong (World Scientific, 1995b), pp. 155 – 192.
  * Kobe (1962) D. H. Kobe, Annals of Physics 19, 448 (1962).
  * Baym and Mermin (1961) G. Baym and N. D. Mermin, Journal of Mathematical Physics 2, 232 (1961).
  * Bauernschmitt and Ahlrichs (1996) R. Bauernschmitt and R. Ahlrichs, Chemical Physics Letters 256, 454 (1996).
  * Slater (1930) J. C. Slater, Phys. Rev. 35, 210 (1930).
  * Yu (2003) Y.-K. Yu, Physica A: Statistical Mechanics and its Applications 326, 522 (2003).
  * Doerr and Yu (2004) T. P. Doerr and Y.-K. Yu, American Journal of Physics 72, 190 (2004).
  * Doerr and Yu (2006) T. P. Doerr and Y.-K. Yu, Phys. Rev. E 73, 061902 (pages 14) (2006).
  * Obolensky et al. (2009) O. I. Obolensky, T. P. Doerr, R. Ray, and Y.-K. Yu, Phys. Rev. E 79, 041907 (pages 15) (2009).
  * Kotliar et al. (2006) G. Kotliar, S. Y. Savrasov, K. Haule, V. S. Oudovenko, O. Parcollet, and C. A. Marianetti, Rev. Mod. Phys. 78, 865 (pages 87) (2006).
  * Itzykson and Zuber (1980) C. Itzykson and J.-B. Zuber, _Quantum Field Theory_ (New York: McGraw-Hill, 1980).
  * Binosi and Theußl (2004) D. Binosi and L. Theußl, Computer Physics Communications 161, 76 (2004).


# Trajectory Planning of Robotic Manipulator in Dynamic Environment Exploiting
DRL

Osama Ahmad  Department of Electrical Engineering
Lahore University of Management and Sciences
Lahore, Pakistan
<EMAIL_ADDRESS>Zawar Hussain  Department of Electrical Engineering
Lahore University of Management and Sciences
Lahore, Pakistan
<EMAIL_ADDRESS>Hammad Naeem  Speridian Technologies

Albuquerque, USA
<EMAIL_ADDRESS>
###### Abstract

This study is about the implementation of a reinforcement learning algorithm
in the trajectory planning of manipulators. We have a 7-DOF robotic arm to
pick & place the randomly placed block at a random target point in an unknown
environment. The obstacle is randomly moving which creates a hurdle in picking
the object. The objective of the robot is to avoid the obstacle and pick the
block with constraints to a fixed timestamp. In this literature, we have
applied a deep deterministic policy gradient (DDPG) algorithm and compared the
models’ efficiency with dense and sparse rewards.

###### Index Terms:

Trajectory-planning, Robotics, Control, Deep Reinforcement Learning.

††publicationid: pubid: 979-8-3503-4863-7/24/$31.00 ©2024 IEEE

##  I Introduction

In this modern era, robots are commonly used in many applications such as
picking and placing objects, welding, surgical, agricultural sectors, and many
more. Industrial robots operate in complex environments where uncertainties
and causalities may happen. In some applications, humans have to physically
interact with the robot which is commonly known as Human-Robot Interaction
(HRI). However, this interaction can raise the risk of safety concerns for
humans and the environment.
To ensure safety in industrial zones, many control strategies such as
impedance control, and admittance control have been introduced. These
techniques incorporate many torque or force sensors, proximity sensors, and
environment sensing sensors. But these mechanisms also introduced more
complexities in the robotic manipulator. Artificial potential fields are also
used to avoid obstacles in the operative space of the manipulator [12], [13].
Whereas the sampling method and path planner are used to find out the
trajectory of the manipulator. In these types of trajectory planning methods,
the dynamics of the robot should known, or if the obstacle is moving we have
to re-compute the trajectory which further adds real-time computational
complexity.
With the advancement in deep neural networks (DNN) and optimization in
hardware, AI techniques have been implemented in robots to perform industrial
tasks in complex environments. In the last decade, many control strategies
using Reinforcement Learning (RL) and neural networks have been implemented
[9], [11].
In [2] continuous control for robots using reinforcement learning has been
introduced. In this literature, two major control techniques have been
discussed: low-dimensionality and raw pixels using deep deterministic policy
gradient (DDPG). Normalized advantage function (NAF) Q-learning algorithm
shows promising results for complex systems [3].
However, for convergence, RL techniques take long training time and resources.
To simulate complex problems that resemble a real-time environment is still a
hard problem in the reinforcement domain. In [1] the transfer learning
approach to train the DRL model. The convergence will occur in fewer
iterations compared to the other approaches. The learning of the manipulator
can be improved by assigning efficient sparse rewards [4]. A multi-goal
reinforcement environment for continuous control tasks such as pushing,
sliding, and pick & place for the 7-DOF robotic arm has been demonstrated in
[5], [10], [11], [14]. Hindsight experience replay is a crucial building block
that makes the training process smooth in these complex and challenging
environments [4], [5].
A hybrid model using transfer learning is applied to the Comau industrial
robot, which switches between two modes [1]. For distant obstacles, it employs
a Single-Query Bi-Directional Probabilistic Roadmap planner with Lazy
Collision Checking (SBL). When encountering unforeseen obstacles, it uses deep
reinforcement learning (DRL) to compute joint positions and velocities for
evasion.
This paper concentrates on trajectory planning for manipulators, aiming to
navigate through an operative space while avoiding obstacles. The primary
challenges lie in dealing with unknown dynamics and the movement of obstacles
in any direction with unknown distributions within the operative field. The
goal is to execute tasks collision-free while minimizing the magnitude of
joint velocities or positions. This research explores various scenarios: S1)
where the target object and position randomly change with no obstacle in the
operative space; and S2) a significant contribution in a multi-goal
environment where the target object and position is randomly positioned, and
the obstacle also moves randomly.
The subsequent sections of this paper are structured as follows: Section II
introduces methodology, Section III explains the implementation of deep
reinforcement learning-based algorithms, Section IV provides simulation and
results, Section V presents future study and work, and Section VI wraps up the
discussions with conclusion.

Figure 1: Basic Diagram of Reinforcement Learning Scheme Figure 2: Deep
Deterministic Framework for Trajectory Planning for Robotic Arm

##  II Methodology

This paper presents a simulation of the robotic manipulator designed for pick-
and-place tasks in an industrial setting. The simulation considers the impact
of unforeseen obstacles, such as humans or other machinery, that might disrupt
the robot’s planned trajectory. A camera is mounted on the manipulator that
provides essential information about the surrounding environment. Moreover,
the robot is equipped with sensors that track the position and velocity of the
joints.

###  II-A Modeling of Robotic Arm

Consider a robotic manipulator having n degree of freedom with an open
kinematic chain. Let joint position and joint velocity be represented by
$q,\dot{q}\in\mathbb{R}^{n}$. Through the position $p_{e}$ and orientation
$\Phi_{e}$ of the end-effector, the joint position of the robotic arm is
computed by the inverse kinematic chain. The workspace $\mathcal{W}$ is a set
of all points spanned by the end-effector in a cartesian coordinates system.

###  II-B Formulation of Control Strategies

The formulation of sparse and dense reward is provided in eq. 1 and eq. 2
where $R_{d}$ is the Euclidean distance of the end-effector from the target
position, $R_{o}$ is the Euclidean distance of the end-effector from the
obstacle. $\delta_{d}$ is the threshold for minimum error threshold with
target position and $\delta_{o}$ is the maximum distance it should avoid
before hitting to obstacle. $c_{1}$ and $c_{2}$ are gains for distance to
target body and distance to obstacle body respectively.

$\boldsymbol{r_{t}}=-(\boldsymbol{R_{d}>\delta_{d}}+\boldsymbol{R_{o}<\delta_{o}}),$
(1) $\boldsymbol{r_{t}}=-(c_{1}\boldsymbol{R_{d}}+c_{2}\boldsymbol{R_{o}}),$
(2)

###  II-C Optimization Problem

The trajectory planning is a minimization problem of joint velocity over the
finite time horizon T as depicted in eq. 3. The dynamic of a robotic arm is
represented by function $f$ based on the current position and velocity. The
L2-norm of joint velocity $\lVert\dot{q}\rVert$ in the objective function
evaluates the minimum velocity to ensure smoothness in motion. The $c_{3}$ is
the penalty term to control the magnitude of the joint velocity $\dot{q}$.
$h(q,\dot{q})$ represents the holonomic constraints for the robotics arm. It
means that the position, orientation, and as well as joint velocities will be
applied within certain bounds. Through reward assignment $r_{t}$, it should
learn how to avoid the collision with the obstacle $\mathcal{O}$ such that
$\mathcal{W}(q)\cap\mathcal{O}\equiv\emptyset$. The formulated optimization
problem is given by

$\begin{split}\min_{\dot{q}}\int_{0}^{T}r_{t}+\,c_{3}\,\lVert\dot{q(t)}\rVert\
^{2}dt,\\\ \quad s.t.\quad\ddot{q}=f(q,\dot{q}),\quad
h(q,\dot{q})<=0\end{split}$ (3)

##  III Implementation of DRL based algorithm

Reinforcement Learning (RL) enables an agent to optimize its actions through a
process of trial and error within an interactive environment, guided by the
feedback received from its actions and accumulated past experiences. Distinct
from the principles of unsupervised learning, the primary objective in RL is
to develop an action strategy that maximizes the agent’s total cumulative
reward. Fig. 1 illustrates the basic framework and the action-reward feedback
cycle of a generic RL model.
The application of trajectory planning for a robotic arm involves states S and
action A which are continuous. Deep Deterministic Policy Gradient (DDPG) [2]
and Normalized Advantage Function (NAF) [3] are commonly used techniques to
compute policy $\pi$ with continuous space and continuous action applications.
In this literature, the DDPG technique has been applied for trajectory
planning and obstacle avoidance. The complete definition of the state space S
and action A are defined in the Appendix.

$S=\\{\mathbf{p_{e}^{0}},\mathbf{p_{o}^{0}},\mathbf{p_{r}^{0}},\mathbf{p_{obs}^{0}},\mathbf{v_{o}^{0}},\mathbf{\omega^{0}},\mathbf{v_{e}},d_{r},d_{l},v_{g_{r}},v_{g_{l}},\alpha,\beta,\gamma\\},$

$A=\\{d_{x_{e}},d_{y_{e}},d_{z_{e}},d_{g_{r}},d_{g_{l}}\\},$

DDPG is a model-free algorithm and it is based on the actor-critic approach.
It is an off-policy RL algorithm based on the assumption that the system is
deterministic such as for every state $s\in S$, there exists a unique action
$a=\pi(s)$. The actor-critic is a two-time scale method in which the actor
provides the action and the critic evaluates the action of the actor. The
basic algorithm and equation have been picked up from [2].
In deep reinforcement learning, the robotic environment autonomously interacts
with the environment and learns how to accomplish a given task. While
interacting with the environment, it stores its experience in the form of
tuples $(s_{t},a_{t},r_{t},s_{t+1})$. Majorly two neural networks are used to
compute action (actor) and value (critic). The target actor and target critic
are used for soft updates of the parameters. Fig. 2 demonstrates DDPG
framework for trajectory planning for a robotic manipulator.
Since DDPG is an off-policy method, the states-action samples are generated
from the actor-network $\mu(s|\theta^{\mu})$ for $T$ timestamp. Where $\mu$ is
the action provided by actor-network, $s\in S$, and $\theta^{\mu}$ is the
current weights of the network. Now, include the Ornstein–Uhlenbeck process
$\mathcal{N}_{t}$ noise in these action values
$a_{t}=\mu(s|\theta^{\mu})+\mathcal{N}_{t}$ for exploration purposes. Store
these experiences in the form of tuples and observe new states $s_{t+1}$.
After collecting these samples from the environment, generate random samples
from these tuples to train the networks.
The weights of the target actor and target critic networks are represented by
$\theta^{\mu^{{}^{\prime}}}$ and $\theta^{Q^{{}^{\prime}}}$ respectively. Eq.
4 shows the cumulative value based on the reward and value of the next state
action provided by the target critic network. The loss of the network is
computed using mean-squared error in Eq. 5.

$y_{i}=r_{i}+\gamma
Q^{\prime}(s_{i+1},\mu^{\prime}(s_{i+1}|\theta^{\mu^{\prime}})|\theta^{Q^{\prime}}),$
(4) $L=\frac{1}{N}\sum_{i}(y_{i}-Q(s_{i},a_{i}|\theta^{Q}))^{2}$ (5)

Eq. 6 shows how to update the actor network’s parameter during back-
propagation to maximize the rewards. It defines the gradient of the objective
function $J$ with respect to an actor weights. The right-hand side term shows
the derivative of the value of critic network
$Q(s,a|\theta^{Q})|_{s=s_{i},a=\mu(s_{i})}$ with respect to action $a$ times
the gradient of the action $\nabla_{\theta^{\mu}}\mu(s|\theta^{\mu})$ of an
actor at state $s_{i}$.

$\nabla_{\theta^{\mu}}J\approx\frac{1}{N}\sum_{i}\nabla_{a}Q(s,a|\theta^{Q})|_{s=s_{i},a=\mu(s_{i})}\nabla_{\theta^{\mu}}\mu(s|\theta^{\mu})|_{s_{i}}$
(6)

After computing the back-propagation at all replay batches, update the weights
of the target critic and target actor network. The polyak $\tau$ is used to
update the weights of target networks. It is used to soft update the weights
of the network. Because of the sudden change in weights disturbs the model
performance.

$\begin{split}\theta^{Q^{\prime}}\leftarrow\tau\theta^{Q}+(1-\tau)\theta^{Q^{\prime}}\\\
\theta^{\mu^{\prime}}\leftarrow\tau\theta^{\mu}+(1-\tau)\theta^{\mu^{\prime}}\end{split}$
(7)

Algorithm 1 DDPG algorithm [2]

1:Initialize critic network $\theta^{Q}$ and actor $\theta^{\mu}$

2:Initialize target network
$\theta^{Q^{\prime}}=\theta^{Q},\theta^{\mu^{\prime}}=\theta^{\mu}$

3:Initialize replay buffer $R$

4:Initialize a random process $N$ for action exploration

5:for episode $=1,M$ do

6: Receive initial observation state $s_{1}$

7: for t = 1, T do

8: Apply action $a_{t}=\mu(s_{t}|\theta^{\mu})+N_{t}$

9: Append $(s_{t},a_{t},r_{t},s_{t+1})$ in $R$ based on $a_{t}$ and $s_{t}$

10: end for

11: Random sample the minibatch of $K$ from $R$

12: for samples = 1, K do

13: compute $y_{i}$ using Eq. 4

14: Update critic by minimizing the loss: using Eq.5

15: Update the actor policy using Eq. 6

16: end for

17: Update the target networks using Eq. 7

18:end for

##  IV Simulation

###  IV-A Experimental Setup

The discussed algorithm has been implemented on a 7-DOF fetch robotic arm in
the gymnasium robotics environment proposed by [5]. Mujoco is a physics engine
used to simulate the robot’s interaction in a given environment under some
dynamics. Fig. 3 shows a snapshot of the environment, the challenge was to
enable a robotic arm to pick up a block placed at random locations and track a
randomly moving red target, all while avoiding unpredictable obstacles.
In every action, the robot is moved by a small displacement of the end-
effector in the Cartesian coordinate system. The joint positions of the
robotic arm are computed by the Mujoco framework internally using inverse
kinematics. The block is uniformly distributed within a range of [-0.15, 0.15]
m with reference to the initial end-effector (x,y) coordinates. Similarly, the
target is defined as uniformly distributed with a range of [-0.15, 0.15] m in
(x,y) coordinates, it can either be in mid-air or over the table The height of
the target is sampled from a uniform distribution within a range of [0, 0.45]
m. In the case of an obstacle, the x-axis is randomly distributed while the
y-axis changes constantly with time. The timestamp to accomplish the task per
episode is set to $T=100$.
For these experiments, four layers of neural network for actor-network have
been used with $256$ hidden units. In the output layer, the $tanh$ function is
used to provide normalized output. There are four layers for a critic network
having a linear output layer. There are $256$ units used in each hidden layer
with Relu as an activation function.

Figure 3: Robotic environment in OpenAI gym a) no obstacle b) obstacle

Experiments are conducted on a Linux machine equipped with an Intel(R) i9
$12^{th}$ Gen @ 2.40 GHz, 32 GB RAM, and NVIDIA 3080Ti GPU. The proposed
method was implemented using PyTorch, specifically Mujoco and gymnasium
robotics for robot operations. Each model was trained using the Adam optimizer
and the hyper-parameters used for experimentation mentioned in the Appendix.

###  IV-B Results and Discussion

We perform simulation for two case scenarios in this environment: 1) when
there is no obstacle robot has to pick the block and reach the target 2)
obstacle is randomly moving, block and target are randomly placed. In case 1,
the success rate is measured that the robot has accomplished the task. In case
2, the success rate is measured by the robot having achieved its task and not
colliding with an obstacle. Fig. 4 a) shows the success rate after every
training episode for case 1 when a reward is sparse and Fig. 4 b) shows the
success rate when a reward is dense. It is depicted that the model with sparse
reward converges in a few episodes compared with dense reward.

Figure 4: Success rate with no obstacle a) sparse reward b) dense reward

Fig. 5 a) and b) represent the training response of the actor-network for
sparse and dense rewards respectively. In the case of sparse reward, agents
receive only binary values which results in more exploration hence actor loss
increases sharply at the beginning. In contrast, the dense rewards agent
exploits its knowledge of the environment more consistently because of the
difference in distance of the agent’s actions, allowing it to refine its
policy with less exploration. Dense reward results in more quicker and stable
learning with lower loss values, suggesting a more efficient learning process
than the sparse reward case.

Figure 5: Actor loss when an obstacle is moving with a) sparse reward b) dense
reward

The training curve of critic loss with sparse and dense rewards is shown in
Fig. 6. The higher loss in the case of sparse reward leads to exhibits the
difficulty for the critic in predicting value as opposed to the critic in
dense reward where the agents appear to learn more efficiently indicated by
the steeper initial drop and overall lower loss values. In general, the
critic’s task is more complex as compared to the actor’s because it evaluates
the actions based on rewards which can vary widely as the actor explores the
environment.

Figure 6: Critic loss with an obstacle when a) sparse reward b) dense reward

The performance of the model has been evaluated on the test set, it consists
of 40 random cases. The success rate with sparse and dense rewards is shown in
Fig. 7, the results indicate the average test results of random cases after
each training epoch. The response of success rate shows that it is increasing
after every episode. The model with sparse reward shows overall good
performance in comparison to the dense reward. The model’s efficiency is
determined by computing the success rate on $10^{5}$ random test cases. Table
I shows the results of the training on different cases. Case 1 with dense
reward shows poor performance when a model is trained on $200$ iterations. If
this model is trained further, it can show significant performance
improvement. But in this analysis, we are comparing reward functions for fixed
episodes.

Figure 7: Success rate with an obstacle when a) sparse reward b) dense reward

It has been observed that the failure cases happen because of two main
reasons: the number of steps is limited and if the block falls below the table
during pick, the block falls beyond the workspace of the robot. When the
reward is sparse for case 2, DDPG shows promising results. With the dense
reward, the model didn’t converge even for $500$ training epoch. Because the
task’s complexity is relatively increased. The sparse reward introduces
flexibility in accomplishing the task. The collision with obstacles can be
improved further.

TABLE I: Evaluation on test cases Cases |  Success Rate(%) |  Fail to reach the target position (%) |  Collision with obstacle (%)
---|---|---|---
Case 1 (sparse reward) |  $94.265$ |  $5.735$ |  $NA$
Case 1 (dense reward) |  $36.03$ |  $63.97$ |  $NA$
Case 2 (sparse reward) |  $71.844$ |  $9.618$ |  $18.538$
Case 2 (dense reward) |  $16.477$ |  $83.271$ |  $0.252$

##  V Future Study and Work

Since DDPG uses linear layers in a deep neural network, it works with a fixed
number of observations. But in the current scenario, due to the random nature
of the obstacle, it can appear at any instance in the operative field of the
robot. This problem can be visualized as a dynamic graph where the obstacle
(node) appears at any time, so the robot’s behavior will change. Similar
approaches like [6], [7], and [15] can be applied to these problem statements.
These methods used Graph Neural Networks (GNN) and Reinforcement Learning (RL)
to explore the environment or accomplish goals. Another approach is also
commonly used to combine Model Predictive Control (MPC) and Reinforcement
Learning (RL) in path planning applications [8]. It solves the mixed static
and dynamic obstacle avoidance problem in an unknown environment.

##  VI Conclusion

This work successfully implements a reinforcement learning-based trajectory-
finding algorithm in a complex and unknown environment. It has been found that
sparse rewards make learning smooth and the performance efficiency is better
than with a model having dense rewards. In the moving obstacles environment, a
deep deterministic policy gradient shows a good response. The robot
prioritized obstacle avoidance over picking the block in finite steps.

## References

  * [1] B. Sangiovanni, G. P. Incremona, M. Piastra, and A. Ferrara, “Self-configuring Robot Path Planning with Obstacle Avoidance via Deep Reinforcement Learning,” IEEE control systems letters, vol. 5, 2021.

  * [2] Lillicrap, Timothy P., et al. “Continuous control with deep reinforcement learning,” arXiv preprint arXiv:1509.02971, 2015.

  * [3] Gu, Shixiang, et al, “Continuous deep q-learning with model-based acceleration,” International conference on machine learning, PMLR, 2016.

  * [4] Andrychowicz, Marcin, et al. “Hindsight experience replay,” Advances in neural information processing systems, 2017.

  * [5] Plappert, Matthias, et alm, “Multi-goal reinforcement learning: Challenging robotics environments and request for research,” arXiv preprint arXiv:1802.09464, 2018.

  * [6] Chen, Fanfei, et al., “Autonomous exploration under uncertainty via deep reinforcement learning on graphs,” 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020.

  * [7] Deng, Yuhong, et al., “Deep reinforcement learning based on local GNN for goal-conditioned deformable object rearranging,” IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2022.

  * [8] Xue, Junxiao, et al., “Multi-agent path planning based on MPC and DDPG,” arXiv preprint arXiv:2102.13283, 2021.

  * [9] Sangiovanni, Bianca, et al., “Deep reinforcement learning for collision avoidance of robotic manipulators,” 2018 European Control Conference (ECC)., 2018.

  * [10] Chen, Lienhung, et al., “Deep reinforcement learning based trajectory planning under uncertain constraints,”  Frontiers in Neurorobotics 16, 2022.

  * [11] Hu, Junyan, et al., “Voronoi-based multi-robot autonomous exploration in unknown environments via deep reinforcement learning,” IEEE Transactions on Vehicular Technology, 2020.

  * [12] Wang, Wenrui, et al., “An improved artificial potential field method of trajectory planning and obstacle avoidance for redundant manipulators,” International Journal of Advanced Robotic Systems, 2018.

  * [13] Liu, S., Q. Zhang, and D. Zhou, “Obstacle avoidance path planning of space manipulator based on improved artificial potential field method,” Journal of The Institution of Engineers (India): Series C 95, 2014.

  * [14] Bing, Zhenshan, et al., “Complex robotic manipulation via graph-based hindsight goal generation,”  IEEE transactions on neural networks and learning systems , 2021.

  * [15] Bing, Zhenshan, et al., “Solving robotic manipulation with sparse reward reinforcement learning via graph-based diversity and proximity,” IEEE Transactions on Industrial Electronics, 2022.

###  -A Notation Definition

The notations in state space S are defined as:
$\mathbf{p_{e}^{0}}:$ position of end-effector w.r.t global frame
$\mathbf{p_{o}^{0}}:$ position of object w.r.t global frame
$\mathbf{p_{o}^{g}}:$ Position of block w.r.t gripper
$\mathbf{p_{obs}^{0}}:$ position of obstacle w.r.t global frame
$\mathbf{v_{o}^{0}}:$ velocity of obstacle w.r.t global frame
$\mathbf{\omega^{0}}:$ angular velocity of block w.r.t global frame
$\mathbf{v_{e}}:$ velocity of end-effector
$d_{r}:$ displacement of right side gripper
$d_{l}:$ displacement of left side gripper
$v_{g_{r}}:$ velocity of right side gripper
$v_{g_{l}}:$ velocity of left side gripper
$\alpha:$ global x rotation of a block in XYZ Euler frame rotation
$\beta:$ global y rotation of a block in XYZ Euler frame rotation
$\gamma:$ global z rotation of a block in XYZ Euler frame rotation

The notations in action space A are defined as:
$d_{x_{e}}:$ displacement of end-effector along x axis
$d_{y_{e}}:$ displacement of end-effector along y axis
$d_{z_{e}}:$ displacement of end-effector along z axis
$d_{g_{r}}:$ displacement of right side gripper
$d_{g_{l}}:$ displacement of left side gripper

###  -B HyperParameter

The list of parameters when an obstacle is moving are:
critic learning rate: $0.0001$
Actor learning rate: $0.0001$
buffer size: $int(1E6)$
polyak: $0.96$
L2 action penalty: $1.0$
No. of observation clip: $200.$
gamma: $0.98$
epochs: $500$
cycles per epoch: $50$
rollout batch size per mpi thread: $2$
training batches per cycle: $40$
batch size: $256$
number of test rollouts per epoch: $10$
percentage of time a random action: $0.3$
noise: $0.2$
replay strategy: ‘future’, supported modes: future, none
replay k: $4$, number of additional goals used for replay, only used if off
policy data=future