# Impact of higher-order dispersion on frequency-modulated combs

Nikola Opa$\mathrm{\check{c}}$ak<EMAIL_ADDRESS>Institute of
Solid State Electronics, TU Wien, Vienna, Austria  Barbara Schneider
Institute for Quantum Electronics, ETH Zurich, Zurich, Switzerland  Jérôme
Faist  Institute for Quantum Electronics, ETH Zurich, Zurich, Switzerland
Benedikt Schwarz<EMAIL_ADDRESS>Institute of Solid State
Electronics, TU Wien, Vienna, Austria

###### Abstract

Frequency-modulated (FM) combs form spontaneously in free-running
semiconductor lasers and possess a vast potential for spectroscopic
applications. Despite recent progress in obtaining a conclusive theoretical
description, experimental FM combs often exhibit non-ideal traits, which
prevents their widespread use. Here we explain this by providing a clear
theoretical and experimental study of the impact of the higher-order
dispersion on FM combs. We reveal that spectrally-dependent dispersion is
detrimental for comb performance and leads to a decreased comb bandwidth and
the appearance of spectral holes. These undesirable traits can be mended by
applying a radio-frequency modulation of the laser bias. We show that
electrical injection-locking of the laser leads to a significant increase of
the comb bandwidth, a uniform-like spectral amplitudes, and the rectification
of the instantaneous frequency to recover a nearly linear frequency chirp of
FM combs.

Perfectly periodic waveforms of light, known as optical frequency combs, stand
as one of the pillars of modern optics, with applications ranging from
fundamental science to precise frequency metrology [1]. Recent years have
witnessed to the enormous efforts behind decreasing the footprint of comb
solutions – inciting rapid progress of chip-scale integrated comb generators.
Among these, semiconductor Fabry-Pérot (FP) lasers are of large importance as
they are compact and possess substantial broadband gain provided by electrical
pumping. The research interest in semiconductor lasers has recently
particularly peaked, largely owing to their capacity to emit the so-called
frequency-modulated (FM) combs. These combs constitute an exciting novel
alternative to generate equidistant comb spectra, reported to date in various
semiconductor laser types such as the quantum cascade laser (QCL) [2, 3, 4],
interband cascade laser [5], quantum dot laser [6], VCSEL [7], and quantum
well laser diode [8, 9, 10]. The traditional and well-established amplitude-
modulated (AM) combs, comprising a train of short periodic light pulses
emitted from mode-locked lasers [11], often rely on a series of external
optical elements to form, potentially requiring tabletop-sized setups. In
sharp contrast, FM combs form spontaneously in semiconductor FP lasers without
the need of any additional optical elements, which makes them especially
appealing for integrated applications. FM combs stand out all the more as they
are not characterized by pulses, but rather a constant intensity where the
instantaneous frequency is modulated instead with a periodic linear chirp.

Figure 1:  Simulated FM frequency combs induced by constant GVD. (a) The
intensity spectrum (top) and the intermodal phases (bottom) of a comb obtained
for GVD=1500$\,\mathrm{fs}^{2}/\mathrm{mm}$. (b) Calculated autocorrelation as
both the GVD and the laser pump are swept. The sign of the GVD only affects
the direction of the frequency chirp. The pump is defined as
$(J-J_{tr})/(J_{th}-J_{tr})$, where $J$ is the laser current density, $J_{th}$
is the lasing threshold, and $J_{tr}$ is the transparency current. Lasing
occurs for pump values larger than 1. FM combs are obtained in the indicated
parameter space, separated with a dashed-dotted line from the unlocked states.
Each pixel represents a state obtained after simulated 20 000 cavity
roundtrips. (c) Obtained spectral width for the same parameter sweep. Optimal
GVD which yields the largest comb bandwidth coincides with the minimum
dispersion necessary for stable comb formation given with the dashed-dotted
black line. (d) Comb bandwidth at a fixed GVD value as the pump is swept for
different FP cavity lengths.

The physical origin of FM combs was the topic of exhaustive theoretical
studies in recent years. Initial studies revealed the crucial roles of spatial
hole burning (SHB) and four-wave mixing (FWM) in the formation of equidistant
multimode spectra [12, 13, 14]. However, the particular modal phase
arrangement of FM combs which leads to their strikingly conspicuous linear
frequency chirp originates purely from a Kerr third-order optical nonlinearity
or a group velocity dispersion (GVD) present in the laser system with finite
facet reflectivities [15, 16, 17, 18, 19]. The latter was particularly
shocking, as even a small GVD was believed to be detrimental for coherent
laser operation, as in the case of AM combs. Despite this notion, a finite GVD
was shown to not only be potentially necessary for coherent comb emission, but
also to increase the optical bandwidth of the comb [20]. Regardless of the
evident breakthroughs in theoretical understanding, experimental FM combs
often continue to be plagued by poor performance exhibiting nonuniform spectra
and low optical bandwidth, thus limiting their use.

In this work, we present a combined experimental and theoretical study of the
influence of higher-order dispersion on FM combs. Although the role of higher-
order dispersion is well understood in the formation of other comb types e.g.
Kerr combs [21], where it is even utilized to create octave-spanning spectra
[22], the impact it has on FM combs remained unclear. We demonstrate that a
spectrally-dependent dispersion has a severe impact on FM comb performance.
Increasing the higher-order (third) contribution leads to poor experimentally-
observed characteristics such as nonuniform comb spectra, nonlinear frequency
chirp, and spectral narrowing – which are undesirable for applications.
Lastly, we show that even in the case of high third-order dispersion, the
negative impact on FM combs can be diminished and even reversed by applying a
radio-frequency (RF) modulation of the laser bias. The resulting injection-
locking of the laser yields the highly-desired larger comb bandwidth, a
uniform optical spectrum, and recovers the linear-like frequency chirp.

Figure 2:  Experimentally-measured FM comb with a large higher-order
dispersion. (a) Intensity spectrum exhibiting nonuniform amplitude
distribution and a spectral hole in the middle (top); SWIFTS spectrum (middle)
matches the geometric average of neighboring intensity spectrum amplitudes,
indicating comb operation; intermodal phases (bottom). (b) Piecewise-linear
instantaneous frequency, indicated with the dashed lines. The discontinuity
occurs due to the hole in the middle of the comb spectrum. (c) Measured
subthreshold GVD contains linear and higher-order contributions. The
measurement was taken at the bias of 0.59$\,A$, just below the lasing
threshold.

To theoretically study the behavior of FM combs, we employ spatio-temporal
numerical simulations of the master equation, derived from the Maxwell-Bloch
system [15, 16]. A typical theoretical FM comb, obtained with a second-order
(constant) dispersion of 1500$\,\mathrm{fs}^{2}/\mathrm{mm}$ in a
4$\,\mathrm{mm}$ long FP cavity is displayed in Fig. 1a). It exhibits an ideal
uniform-like spectrum with the distinct linear intermodal phases that cover
the full spectral range of $2\pi$, corresponding to a linear frequency chirp
and a quasi-constant intensity output. In the absence of Kerr nonlinearity,
GVD provides the only mechanism that shapes the linear chirp of FM combs in FP
lasers with partially reflective facets. Fig. 1b) displays the autocorrelation
value of simulated laser states, as the second-order dispersion and the laser
pump are swept, where lasing occurs for pump values larger than 1. The
autocorrelation is calculated between the emitted field over one roundtrip and
the emitted field delayed by 500 roundtrips. It is evident that an FM comb,
characterized by the autocorrelation of 1 (smaller values indicate an unlocked
state), is obtained only for a nonzero value of the GVD. Furthermore, the
minimum value of GVD required for stable comb formation increases with the
laser pump (dashed-dotted line). Potential presence of a nonzero Kerr
nonlinearity would shift the FM comb parameter space horizontally along the
GVD-axis [15, 20, 16, 18]. Not all obtained comb states possess equally
appealing characteristics, which is apparent from Fig. 1c), where we display
the comb bandwidth. Aiming for the largest spectral width, the laser should be
operated at a high bias point, and have a dispersion that is just large enough
to obtain stable comb operation, but not larger. The influence of the FP
cavity length is visible in Fig. 1d), where we have swept the laser pump for 4
different lengths while keeping the GVD constant. While strongly dependent on
the laser bias, the comb bandwidth increases for shorter cavities, at a fixed
pump. Identical qualitative behavior was obtained by following analytic theory
[18, 19], although the predicted dependence on the cavity length was stronger,
probably due to neglecting of the finite laser gainwidth in the analytic
approach.

Figure 3:  Influence of the third-order dispersion on FM combs in numerical
simulations. (a) Total spectrally-dependent GVD$(\omega)$ with an increasing
third-order (linear) dispersion $k^{(3)}$. Fourth- and higher-order
contributions are omitted for the sake of numerical stability. The evolution
of the (b) intensity spectra, (c) intermodal phases, and the (d) instantaneous
frequency as the third-order dispersion is increased.

Unlike the ideal theoretical FM combs, experimentally-obtained results are
often riddled with undesired traits such as nonuniform spectra with holes, as
shown in the top of Fig. 2a), measured for a 4$\,\mathrm{mm}$ long FP QCL.
More details on the laser active region and cavity design can be found in
[23]. Employing Shifted-Wave Interference Fourier Transform Spectroscopy
(SWIFTS) [24] we can prove the coherence of the comb state and extract the
intermodal phases by recording interferograms at the comb roundtrip frequency.
Due to the hole in the middle of the intensity spectrum, the intermodal phases
arrange themselves in two distinct linear patterns, together covering the
whole range of $2\pi$. As a consequence, the instantaneous frequency is a
piecewise linear function during one cavity roundtrip (Fig. 2b)). The
experimental FM comb state does not resemble much the simulated ideal state in
Fig. 1, urging to discover the cause behind the observed degradation. Fig. 2c)
displays the measured subthreshold dispersion [25], showing non-constant
values with a significant linear contribution around the lasing range,
corresponding to a large third-order dispersion. This is in sharp contrast
with the simulations in Fig. 1 where constant second-order dispersion values
were used, already pointing that the probable culprit behind nonuniform
experimental spectra is the higher-order dispersion.

To corroborate this hypothesis, we performed numerical simulations of the
master equation incorporating the third-order dispersion $k^{(3)}$, to account
just for the linear dependence of the dispersion observed in Fig. 2d). This
contribution is sufficient to explain the experimental behavior. The total
dispersion is then GVD$(\omega)=k^{(2)}+k^{(3)}(\omega-\omega_{0})$, where
$k^{(2)}$ is the second-order dispersion, and $\omega$ is the optical angular
frequency. Mathematically, the additional term in the master equation is
introduced as a third-order temporal derivative of the light field [26]. Terms
$\mathcal{O}(k^{(4)})$ describe nonlinear dispersion dependence
$\mathcal{O}(\omega^{(2)})$ and are omitted as they would introduce higher-
order derivates and issues with numerical stability. The origin of $k^{(3)}$
is purely considered to be from the cavity and material dispersion. An
additional contribution is due to the gain spectral profile itself and is
expected to increase in importance for narrower gainwidths or inhomogenous
gain broadening e.g. in heterogenous QCLs [27]. Column a) in Fig. 3 shows the
gradual increase of the the third-order dispersion which was inserted in
numerical simulations to study its impact on the comb state shown in Fig. 1a).
Columns b), c), and d) in Fig. 3 display the resulting comb spectra,
intermodal phases, and the instantaneous frequency, respectively. Although
maintaining comb operation, the intensity spectra progressively develop an
irregular amplitude distribution, resembling experimental comb spectra that
are often found in literature [2, 3, 4, 6, 5, 8, 10, 7, 9]. At the same time,
the intermodal phases and the instantaneous frequency increasingly deviate
from a linear pattern. For the highest value of third-order dispersion
($k^{(3)}=400\,\mathrm{fs}^{2}/\upmu m$), the spectrum splits in two separated
lobes, alike in Fig. 2a). Further resemblance to the experimental state is
observed from the intermodal phases and the instantaneous frequency, which in
approximation becomes a piecewise linear function during one roundtrip.

Figure 4:  The effect of RF modulation of the laser bias. Experimental data
(left) and numerical simulations (right) indicate that modulating the laser
bias leads to the broadening of the spectrum and flattening of the spectral
amplitudes. The intermodal phases and the instantaneous frequency restore
nearly linear behavior characteristic to ideal FM combs.

In order to reach the full potential of FM combs for widespread applications,
it is of crucial importance to achieve reproducible high-quality comb
behavior, as in the numerical simulations. First of all, having a broadband
uniform comb spectra is of high interest for spectroscopy. On the other hand,
obtaining a linear frequency chirp allows the use of a pulse compressor via
group delay compensation [28] to achieve femtosecond pulse emission [29]. In
Fig. 4 we demonstrate, both experimentally and theoretically, how to recover
these highly desired traits even in FM combs that possess a large higher-order
dispersion by employing an RF modulation of the laser bias. Modulating the
bias of a semiconductor laser at the roundtrip frequency enables coherent
control of many of its characteristics – which was so far successfully used to
injection-lock the comb beatnote [30, 5, 6], eliminate higher-order transverse
cavity modes [31], emit actively mode-locked pulses [32], and increase the
spectral bandwidth of FM combs [20, 23]. The modulation should be done on a
short end-section of the FP cavity to maximize the effect [33]. In the top of
Fig. 4 we show free-running experimental and simulated FM comb states with
large higher-order dispersion, taken from Fig.2a) and bottom of Fig. 3b),
respectively. The impact of the RF injection is visible from the panels below,
showing agreement between the experimental and simulation results. The comb
spectra become more uniform and the spectral hole disappears. Apart from this,
another striking effect is reflected in a significant broadening of the comb
bandwidth of around 100 % [20, 23]. In the theoretical model, the modulation
is directly implemented to the laser bias of a short section covering 10 % of
the cavity as $J(t)=J_{\mathrm{DC}}+J_{\mathrm{m}}\cos(\omega_{\mathrm{m}}t)$,
where $\omega_{\mathrm{m}}$ is the modulation frequency and we set
$J_{\mathrm{m}}=0.05J_{\mathrm{DC}}$ [32]. In the experiment, a 4 mm Fabry-
Pérot QCL with a microstrip-like line geometry [23] was used. Both, a DC-bias
of 950 mA and a 25 dBm AC bias modulation of 11.090 GHz, which matches the
free-running intermode beating, were supplied using a high-power coplanar RF-
probe. Whereas the device has no dedicated modulation section, the point of
contact with the probe is very close to the front-facet, which has a similar
effect as modulating a short end-section. The impact of bias modulation is
visible from the intermodal phases as well, which splay over $2\pi$ in a
single nearly linear pattern. Slightly worse behavior in experiments could be
explained by the existence of finite $\mathcal{O}(k^{(4)})$ terms. The
piecewise linear instantaneous frequency of free-running combs becomes
significantly rectified, thus enabling the use of group delay compensation
schemes to achieve pulse compression.

In this work, we provided a study of the impact of higher-order dispersion on
FM combs by utilizing both numerical simulations of a spatio-temporal
theoretical model and experimental measurements. We reveal that the presence
of finite higher-order dispersion $\mathcal{O}(k^{(3)})$ negatively affects FM
comb characteristics by lowering the spectral bandwidth, producing spectral
holes in the spectrum, and resulting in a nonlinear frequency chirp. In order
to boost FM comb uses for broadband spectroscopic applications, it is
therefore crucial to mitigate higher-order dispersion if possible. One way of
achieving this would be to tailor the spectral gain profile or change the
laser cavity in an effort to make the GVD constant within the lasing range.
Here we demonstrated a simple technique which achieves this and does not rely
on any modifications of the existing laser system. By modulating the laser
current around the roundtrip frequency we overcome the severe limitations
imposed by the higher-order dispersion by doubling the comb bandwidth,
flattening the intensity spectrum and recovering a nearly linear frequency
chirp. The further use of RF modulation could enhance the use of FM combs for
dual-comb spectroscopy.

## Acknowledgements

N. O. and B. Schwarz have received funding from the European Research Council
(ERC) under the European Union’s Horizon 2020 research and innovation
programme (Grant agreement No. 853014). B. Schneider and J. F. have received
funding from the European Research Council (ERC) under the European Union’s
Horizon 2020 research and innovation programme (Grant agreement No. 820419)
and from the Swiss Innovation Agency Innosuisse (Grant agreement No.
2155008433).

FM

    frequency-modulated
AM

    amplitude-modulated
QCL

    quantum cascade laser
FP

    Fabry-Pérot
SHB

    spatial hole burning
GVD

    group velocity dispersion
LEF

    linewidth enhancement factor
FWM

    four-wave mixing
RF

    radio-frequency
SWIFTS

    Shifted-Wave Interference Fourier Transform Spectroscopy

## References

  * Diddams  _et al._ [2020] S. A. Diddams, K. Vahala, and T. Udem, Optical frequency combs: Coherently uniting the electromagnetic spectrum, Science 369, 10.1126/science.aay3676 (2020).
  * Hugi  _et al._ [2012] A. Hugi, G. Villares, S. Blaser, H. C. Liu, and J. Faist, Mid-infrared frequency comb based on a quantum cascade laser, Nature 492, 229 (2012).
  * Burghoff  _et al._ [2014] D. Burghoff, T.-Y. Kao, N. Han, C. W. I. Chan, X. Cai, Y. Yang, D. J. Hayton, J.-R. Gao, J. L. Reno, and Q. Hu, Terahertz laser frequency combs, Nature Photonics 8, 462 (2014).
  * Singleton  _et al._ [2018] M. Singleton, P. Jouy, M. Beck, and J. Faist, Evidence of linear chirp in mid-infrared quantum cascade lasers, Optica 5, 948 (2018).
  * Schwarz  _et al._ [2019] B. Schwarz, J. Hillbrand, M. Beiser, A. M. Andrews, G. Strasser, H. Detz, A. Schade, R. Weih, and S. Höfling, Monolithic frequency comb platform based on interband cascade lasers and detectors, Optica 6, 890 (2019).
  * Hillbrand  _et al._ [2020a] J. Hillbrand, D. Auth, M. Piccardo, N. Opačak, E. Gornik, G. Strasser, F. Capasso, S. Breuer, and B. Schwarz, In-Phase and Anti-Phase Synchronization in a Laser Frequency Comb, Physical Review Letters 124, 10.1103/physrevlett.124.023901 (2020a).
  * Kriso  _et al._ [2021] C. Kriso, A. Barua, O. Mohiuddin, C. Möller, A. Ruiz-Perez, W. Stolz, M. Koch, and A. Rahimi-Iman, Signatures of a frequency-modulated comb in a VECSEL, Optica 8, 458 (2021).
  * Day  _et al._ [2020] M. W. Day, M. Dong, B. C. Smith, R. C. Owen, G. C. Kerber, T. Ma, H. G. Winful, and S. T. Cundiff, Simple single-section diode frequency combs, APL Photonics 5, 121303 (2020).
  * Sterczewski  _et al._ [2020] L. A. Sterczewski, C. Frez, S. Forouhar, D. Burghoff, and M. Bagheri, Frequency-modulated diode laser frequency combs at 2 $\upmu$m wavelength, APL Photonics 5, 076111 (2020).
  * Sterczewski  _et al._ [2022] L. A. Sterczewski, M. Fradet, C. Frez, S. Forouhar, and M. Bagheri, Battery-Operated Mid-Infrared Diode Laser Frequency Combs, Laser & Photonics Reviews 17, 2200224 (2022).
  * Haus [2000] H. Haus, Mode-locking of lasers, IEEE Journal of Selected Topics in Quantum Electronics 6, 1173 (2000).
  * Khurgin  _et al._ [2014] J. B. Khurgin, Y. Dikmelik, A. Hugi, and J. Faist, Coherent frequency combs produced by self frequency modulation in quantum cascade lasers, Applied Physics Letters 104, 081118 (2014).
  * Mansuripur  _et al._ [2016] T. S. Mansuripur, C. Vernet, P. Chevalier, G. Aoust, B. Schwarz, F. Xie, C. Caneau, K. Lascola, C. en Zah, D. P. Caffey, T. Day, L. J. Missaggia, M. K. Connors, C. A. Wang, A. Belyanin, and F. Capasso, Single-mode instability in standing-wave lasers: The quantum cascade laser as a self-pumped parametric oscillator, Physical Review A 94, 10.1103/physreva.94.063807 (2016).
  * Dong  _et al._ [2018] M. Dong, S. T. Cundiff, and H. G. Winful, Physics of frequency-modulated comb generation in quantum-well diode lasers, Physical Review A 97, 10.1103/physreva.97.053822 (2018).
  * Opačak and Schwarz [2019] N. Opačak and B. Schwarz, Theory of Frequency-Modulated Combs in Lasers with Spatial Hole Burning, Dispersion, and Kerr Nonlinearity, Physical Review Letters 123, 10.1103/physrevlett.123.243902 (2019).
  * Opačak  _et al._ [2021] N. Opačak, S. D. Cin, J. Hillbrand, and B. Schwarz, Frequency Comb Generation by Bloch Gain Induced Giant Kerr Nonlinearity, Physical Review Letters 127, 10.1103/physrevlett.127.093902 (2021).
  * Silvestri  _et al._ [2020] C. Silvestri, L. L. Columbo, M. Brambilla, and M. Gioannini, Coherent multi-mode dynamics in a quantum cascade laser: amplitude- and frequency-modulated optical frequency combs, Optics Express 28, 23846 (2020).
  * Burghoff [2020] D. Burghoff, Unraveling the origin of frequency modulated combs using active cavity mean-field theory, Optica 7, 1781 (2020).
  * Humbard and Burghoff [2022] L. Humbard and D. Burghoff, Analytical theory of frequency-modulated combs: generalized mean-field theory, complex cavities, and harmonic states, Optics Express 30, 5376 (2022).
  * Beiser  _et al._ [2021] M. Beiser, N. Opačak, J. Hillbrand, G. Strasser, and B. Schwarz, Engineering the spectral bandwidth of quantum cascade laser frequency combs, Optics Letters 46, 3416 (2021).
  * Bao  _et al._ [2017] C. Bao, H. Taheri, L. Zhang, A. Matsko, Y. Yan, P. Liao, L. Maleki, and A. E. Willner, High-order dispersion in Kerr comb oscillators, Journal of the Optical Society of America B 34, 715 (2017).
  * Pfeiffer  _et al._ [2017] M. H. P. Pfeiffer, C. Herkommer, J. Liu, H. Guo, M. Karpov, E. Lucas, M. Zervas, and T. J. Kippenberg, Octave-spanning dissipative Kerr soliton frequency combs in Si_3N_4 microresonators, Optica 4, 684 (2017).
  * Schneider  _et al._ [2021] B. Schneider, F. Kapsalidis, M. Bertrand, M. Singleton, J. Hillbrand, M. Beck, and J. Faist, Controlling Quantum Cascade Laser Optical Frequency Combs through Microwave Injection, Laser & Photonics Reviews 15, 2100242 (2021).
  * Burghoff  _et al._ [2015] D. Burghoff, Y. Yang, D. J. Hayton, J.-R. Gao, J. L. Reno, and Q. Hu, Evaluating the coherence and time-domain profile of quantum cascade laser frequency combs, Optics Express 23, 1190 (2015).
  * Hofstetter and Faist [1999] D. Hofstetter and J. Faist, Measurement of semiconductor laser gain and dispersion curves utilizing fourier transforms of the emission spectra, IEEE Photonics Technology Letters 11, 1372 (1999).
  * Opačak [2022] N. Opačak,  _The origin of frequency combs in free-running quantum cascade lasers_ , Ph.D. thesis, TU Wien (2022).
  * Bachmann  _et al._ [2016] D. Bachmann, M. Rösch, G. Scalari, M. Beck, J. Faist, K. Unterrainer, and J. Darmo, Dispersion in a broadband terahertz quantum cascade laser, Applied Physics Letters 109, 10.1063/1.4969065 (2016).
  * Singleton  _et al._ [2019] M. Singleton, M. Beck, and J. Faist, Pulses from a mid-infrared quantum cascade laser frequency comb using an external compressor, Journal of the Optical Society of America B 36, 1676 (2019).
  * Täschler  _et al._ [2021] P. Täschler, M. Bertrand, B. Schneider, M. Singleton, P. Jouy, F. Kapsalidis, M. Beck, and J. Faist, Femtosecond pulses from a mid-infrared quantum cascade laser, Nature Photonics 15, 919 (2021).
  * Hillbrand  _et al._ [2018] J. Hillbrand, A. M. Andrews, H. Detz, G. Strasser, and B. Schwarz, Coherent injection locking of quantum cascade laser frequency combs, Nature Photonics 13, 101 (2018).
  * Cin  _et al._ [2022] S. D. Cin, F. Pilat, A. Konečný, N. Opačak, G. Strasser, and B. Schwarz, Coherent control of transverse modes in semiconductor laser frequency combs via radio-frequency injection, Applied Physics Letters 121, 071106 (2022).
  * Hillbrand  _et al._ [2020b] J. Hillbrand, N. Opačak, M. Piccardo, H. Schneider, G. Strasser, F. Capasso, and B. Schwarz, Mode-locked short pulses from an 8 $\upmu$m wavelength semiconductor laser, Nature Communications 11, 10.1038/s41467-020-19592-1 (2020b).
  * Piccardo  _et al._ [2018] M. Piccardo, D. Kazakov, N. A. Rubin, P. Chevalier, Y. Wang, F. Xie, K. Lascola, A. Belyanin, and F. Capasso, Time-dependent population inversion gratings in laser frequency combs, Optica 5, 475 (2018).

  *[FP]: Fabry-Pérot
  *[FM]: frequency-modulated
  *[QCL]: quantum cascade laser
  *[AM]: amplitude-modulated
  *[SHB]: spatial hole burning
  *[FWM]: four-wave mixing
  *[GVD]: group velocity dispersion
  *[RF]: radio-frequency
  *[SWIFTS]: Shifted-Wave Interference Fourier Transform
Spectroscopy

  *[QCLs]: quantum cascade laser


# A Human-centered Explainable AI Framework for Burn Depth Characterization

Maxwell J. Jacobson
Purdue University
<EMAIL_ADDRESS>
&Daniela Chanci Arrubla
Emory University
<EMAIL_ADDRESS>
&Maria Romeo Tricas
Purdue University
<EMAIL_ADDRESS>
&Gayle Gordillo
Indiana University
<EMAIL_ADDRESS>
&Yexiang Xue
Purdue University
<EMAIL_ADDRESS>
&Chandan Sen
Indiana University
<EMAIL_ADDRESS>
&Juan Wachs
Purdue University
<EMAIL_ADDRESS>

###### Abstract

Approximately 1.25 million people in the United States are treated each year
for burn injuries. Precise burn injury classification is an important aspect
of the medical AI field. In this work, we propose an explainable human-in-the-
loop framework for improving burn ultrasound classification models. Our
framework leverages an explanation system based on the LIME classification
explainer to corroborate and integrate a burn expert’s knowledge — suggesting
new features and ensuring the validity of the model. Using this framework, we
discover that B-mode ultrasound classifiers can be enhanced by supplying
textural features. More specifically, we confirm that texture features based
on the Gray Level Co-occurance Matrix (GLCM) of ultrasound frames can increase
the accuracy of transfer learned burn depth classifiers. We test our
hypothesis on real data from porcine subjects. We show improvements in the
accuracy of burn depth classification — from 88% to 94% — once modified
according to our framework.

_Keywords_ Burn Analysis $\cdot$ Ultrasound $\cdot$ Deep Learning $\cdot$
Computer Vision $\cdot$ Explainability $\cdot$ Human-in-the-loop

##  1 Introduction

Figure 1: Our human-in-the-loop through explainability framework. A human
possessing expert knowledge trains a classifier. Our LIME-based explainer
provides insights to the human based on the current model. The human applies
expert knowledge to interpret the explanation. The human expert modifies the
model appropriately, and the process can be repeated. In our case, we modify
the network by adding new features based on expert medical knowledge regarding
burns.

Our work focuses on the task of burn depth estimation. Performing this task
accurately can be critical for the welfare of burn victims, but it is also a
challenge due to the high variability in the visual appearance of burns.
Different imaging modalities have been explored to solve this problem and
improve the accuracy of the diagnosis Sen et al. (2016) — e.g. color
photographs, ultrasound, infrared thermography, laser speckle imaging, and
laser doppler imaging Thatcher et al. (2016). Recent studies combine imaging
modalities with machine learning and deep learning models. Cirillo et al
Cirillo et al. (2019) compared the performance of VGG16 Simonyan and Zisserman
(2015), GoogleNet Szegedy et al. (2015), ResNet50 He et al. (2016), and
ResNet101, pretrained on ImageNet Deng et al. (2009), to classify their
labeled dataset of RGB burn images. Similarly, Chauhan & Goyal Chauhan and
Goyal (2020) carried out the burn depth classification based on the specific
characteristics of the body region in which the injury was located. However,
RGB images can negatively influence the accuracy of predictions due to
lighting conditions, skin color, or general variability in wound presentation.
Besides, RGB images, optical coherence tomography Singla et al. (2018),
spatial frequency-domain imaging Rowland et al. (2019), and ultrasound Lee et
al. (2020) have been utilized for feature extraction. Harmonic B-mode
ultrasound (HUSD) — a non-invasive sound-based imaging technique — is used in
this work. B-mode ultrasound is based on the transmission of small pulses of
ultrasound. Echoes reflected back to the transducer from body tissues that
have different acoustic impedances, which can be measured to build a 3D map of
tissues. Second harmonic frequency echoes are used in order to reduce the
artifacts in the image produced by the reflection of echoes at different
frequencies Narouze (2018).

Computer vision based techniques lack the human expertise in current medical
ML models. Therefore, a human-in-the-loop system built on Explainable
Artificial Intelligence is proposed here. Human-in-the-loop (HITL) is an
Artificial Intelligence (AI) paradigm that assumes the presence of human
experts that can guide the learning or operation of the otherwise-autonomous
system. Lundberg et al. Lundberg et al. (2018) developed and tested a system
to prevent hypoxaemia during surgery by providing anaesthesiologists with
interpretable hypoxaemia risks and contributing factors. Later, Sayres et al.
Sayres et al. (2019) proposed and evaluated a system to assist diabetic
retinopathy grading by ophtalmologists using a deep learning model and
integrated gradients explanation.

An Explainable Artificial Intelligence (XAI) is an intelligent system which
can be explained and understood by a human Gohel et al. (2021). For this, we
utilize LIME (Local Interpretable Model-agnostic Explanations) Ribeiro et al.
(2016), a recent method that is able to explain the predictions of any
classifier model in an interpretable manner. LIME operates by roughly
segmenting an image into feature regions, then assigning saliency scores to
each region. Higher scoring zones are more important in arriving at the
classification result of the studied model. The algorithm first creates random
permutations of the image to be explained. Then, the classification model is
run on those samples. Distances between those samples and the original image
can be calculated, which are then converted to weights by being mapped between
zero and one using a kernel function. Finally, a simple linear model is fitted
around the initial prediction to generate explanations. This explanation
provided by LIME is the result of the following minimization:

$\xi(x)=\operatorname*{argmin}_{g\in G}\mathcal{L}(f,g,\pi_{x})+\Omega(g)$ (1)

Let $x$ and $f$ be the image and classifier to be examined and $G$ as the
class of interpretable models like decision trees and linear models. The
complexity of the model (e.g. depth of a decision tree, number of non-zero
weights in a linear model) should be as small as possible to maintain
explainability. This complexity can be defined as $\Omega(g)$. Explanations by
LIME are found by fitting explainable models — minimizing the sum of the local
faithfulness loss $L$ and the complexity score. Permutated sampling is used to
approximate this local faithfulness loss. For this reason, a proximity measure
$\pi_{x}(z)$ calculates the distance between $x$ and another image $z$. The
objective is minimizing the fidelity function while maintaining a measure of
complexity low enough to be interpretable. This minimization makes no
assumptions about $f$ in order to generate a model-agnostic explanation.

Explainable intelligence is useful when combined with HITL systems because
they provide understandable and qualitative information about the relationship
between the instance’s components and the model’s prediction. Therefore, an
expert can make an informed decision about whether the model is reliable, and
can make the necessary changes if it is not — eventually reaching a confident
result. This is extremely important, above all in the medical field, because
of the severe ethical implications that suppose a wrong medical diagnosis.

By deploying our explainable human-in-the-loop method, we were able to confirm
the importance of one family of features which can enhance a convolutional
burn prediction classifier — statistical texture. From the Gray Level Co-
ocurrence Matrix (GLCM) — a method that represents the second-order
statistical information of gray levels between neighboring pixels in an image
— many important sub-features can be extracted Seal et al. (2018). In this
work, the features used are: contrast, homogeneity, angular second moment
(ASM), energy, and dissimilarity. More information on GLCM can be found in
appendix A.

We find that by discovering weaknesses in our model through explainable AI and
utilizing expert knowledge from a human in the loop, burn depth can be
classified more accurately from B-mode ultrasound data.

Figure 2: When applying our framework to the question of texture in ultrasound
burn analysis, we can show support for the hypothesis in three ways: from
explanation, from expert knowledge, and from scientific experimentation. This
allows for greater understanding and trust in the classifier.

##  2 Method

Figure 3: Burn classification model. The primary feature vector ($v_{1}$) is
calculated using the pre-trained model. This model is not updated during
training. The secondary vector ($v_{2}$) is calculated from GLCM texture
features. Both are concatenated as inputs to a simple 2-layer Softmax
classifier. This layer contains 1024 unit ReLU layer before the output layer.

Explainable Human-in-the-loop System. The primary contribution of this paper
is our XAI-based human-in-the-loop system (See figure 1). The human expert
starts the process — in this case, the expert was a clinician on the team.
They train and test an initial classifier model on a burn ultrasound dataset.
If this does not yield satisfactory results, the expert activates the
explainer system. Using hand-selected or random records from the dataset, the
LIME-based explainer gives a series of explanations about the model to the
expert. These explanations are in the form of saliency maps — visually
communicating to the expert what parts of the input are being utilized for any
given prediction.

For our purposes, LIME conducts its sampling using the quickshift Vedaldi and
Soatto (2008) and Felzenszwalb Felzenszwalb and Huttenlocher (2004)
segmentation methods. 10,000 permutations are sampled, and the LIME algorithm
is run. From the LIME outputs, we create a heatmap, a visual overlay of the
top K features, and a quantitative list of the LIME scores for each feature.
Besides LIME, we also produce a pixel-wise saliency map using back-propagation
on the input image Simonyan et al. (2014). This approach is not model
agnostic, but it provides a second explanation modality to include when the
classifier is a neural net.

The expert combines this new information with their prior medical knowledge.
In this case, the burn expert knows that severe burns change the echogenicity
of healthy skin. Further, the expert knows that physical textural features in
ultrasound translate to statistical textural changes in the image Lee et al.
(2020). This can be interpreted and implies that, in addition to key areas
like skin layer transitions, a solid classifier should have a nebulous
saliency map, extracting features from across the image. In our case, the
expert was provided with these explanations and observed dense saliency maps
were more apparent on misclassified examples. This view is demonstrated in
Figure 4.

After the interaction between the explainer and the expert, the necessary
modifications in the feature set of the convolutional neural network can be
made. In this work, this is done by adding GLCM texture features — selected in
the hope that they can assuage the "feature bottleneck" and improve accuracy.
Finally, the model is re-trained and re-tested to prove that the updated
classifier leads to a good prediction. Consequently, this human-in-the-loop
system allows the design of a trustworthy, ethical, and robust medical
diagnosis system (See figure 2.

Classification Model. In this work, ResNet18 is used as a pretrained —on
ImageNet dataset— convolutional neural network with all his layers frozen
except from the last fully connected layer, which is replaced by a 1-layer
softmax classifier. The hyperparameters set for this model are: 15 epochs, a
batch size equal to 8 and a $10^{-5}$ learning rate with no decay. The cross-
entropy loss function was utilized alongside the Adam optimizer Kingma and Ba
(2015) for training. The B-mode ultrasound imaging modality consists of
grayscale images with different textures, which are very important in the
diagnostic process. Therefore, in addition to the presented architecture, a
feature approach is followed in this work (See figure 3) . The last layer of
the ResNet18 neural network outputs a 30-dimension feature vector from the 512
features that the architecture initially extracts. This vector ($v_{1}$)
constitutes the primary feature vector, which is combined with the hand-
crafted Haralick texture features vector ($v_{2}$). Hence, the Haralick
features extracted from the GLCM matrix are concatenated with the CNN-based
features, and afterwards, the fused vector constitutes the input to the fully
connected softmax classifier.

Figure 4: Demonstration of explainer output. A) Shows a correct
classification. While certain zones of the image are more salient than others,
information is utilized from across the image. It is likely that texture
information is being naturally extracted. Note for scale that LIME scores
range between -0.027 and 0.016. B) Shows a misclassified example. The output
is largely determined by one feature of the input. The correct texture
features are either not being calculated or are being ignored. Note for scale
that LIME scores range between -0.061 and 0.012 — A nearly 70% increase.

##  3 Data

An unreleased B-mode HUSD dataset was acquired for each of the three classes
included in the burn depth classification task: full-thickness burn, partial-
thickness burn, and unburnt skin. Pig models were selected as the in vivo
wound-healing animal models based on previous studies, which indicate that
there is a high similarity between pig skin and human skin. Similar
characteristics include the dermal-epidermal ratio, the dermal collagen, the
distribution of blood vessels, and an abundant subdermal adipose tissue. In
addition, the healing process is also comparable, more specifically, it was
demostrated that there is a similarity of 78% between pigs and humans. This
number decreases to 53% for small-mammal models, and 57% for in vitro studies
Sullivan et al. (2001). B-mode HUSD videos of approximately 15 seconds were
obtained. The data corresponding to the skin class was collected first, in the
same region where the burn injuries were going to be located. Then, in order
to collect the data for the two remaining classes, eight full-thickness burn
injuries were generated. The data for the full-thickness class was collected
immediately after wounding (day 0). The wounds were then treated, and the
healing process started. By day 14 post-burn, the injuries had not healed
completely. More specifically, by that time the subcutaneous tissue had been
regenerated, while the dermis and the epidermis were still affected. These are
the same characteristics that can be seen in second-degree burn injuries,
reason why day 14 post-burn data was used for the second-degree class.

##  4 Experiments & Results

Features |  Mean
Acc |  Acc
Std |  Mean
Prec |  Prec
Std |  Mean
Rec |  Rec
Std |  Mean F1 |  F1 Std
---|---|---|---|---|---|---|---|---
SVM | 91.90% | 0.016 | 0.852 | 0.045 | 0.935 | 0.014 | 0.874 | 0.036
ResNet18 | 87.53% | 0.014 | 0.872 | 0.014 | 0.848 | 0.015 | 0.853 | 0.015
ResNet18 + |  |  |  |  |  |  |  |
Contrast | 93.62% | 0.014 | 0.934 | 0.011 | 0.913 | 0.013 | 0.918 | 0.013
ResNet18 + |  |  |  |  |  |  |  |
Homogeneity | 88.75% | 0.006 | 0.886 | 0.006 | 0.863 | 0.006 | 0.868 | 0.007
ResNet18 + |  |  |  |  |  |  |  |
ASM | 88.54% | 0.007 | 0.883 | 0.011 | 0.860 | 0.012 | 0.865 | 0.011
ResNet18 + |  |  |  |  |  |  |  |
Energy | 88.54% | 0.005 | 0.881 | 0.009 | 0.856 | 0.009 | 0.862 | 0.009
ResNet18 + |  |  |  |  |  |  |  |
Dissimilarity | 89.56% | 0.010 | 0.890 | 0.009 | 0.868 | 0.010 | 0.874 | 0.010
ResNet18 + |  |  |  |  |  |  |  |
All Features | 92.56% | 0.004 | 0.927 | 0.005 | 0.903 | 0.004 | 0.909 | 0.004
Table 1: Results on burn depth classification. Each method was trained and
tested 6 times. 30 Epochs of training were conducted using a random subset of
the training data containing 1,400 frames per class. Pure ResNet18 is compared
with the feature-augmented versions, as well as an SVM due to its common use
in this domain. All GLCM features show significant improvement over pure
ResNet18. Furthermore, only the Contrast feature shows very high improvement.
Although, combining all 5 features shows a similar improvement but with lower
variability.

Tukey Test Results
---
I | J | MD | P-adj | Reject
All | ASM | -0.0402 | 0 | ✓
Contr | 0.0106 | 0.461 |
Dissim | -0.03 | 0 | ✓
Energy | -0.0402 | 0 | ✓
Homog | -0.0381 | 0 | ✓
None | -0.0503 | 0 | ✓
ASM | Contr | 0.0508 | 0 | ✓
Dissim | 0.0102 | 0.501 |
Energy | 0 | 1 |
Homog | 0.0021 | 1 |
None | -0.0101 | 0.514 |

Tukey Test Results Continued
---
I | J | MD | P-adj | Reject
Contr | Dissim | -0.0406 | 0 | ✓
Energy | -0.0508 | 0 | ✓
Homog | -0.0487 | 0 | ✓
None | -0.0609 | 0 | ✓
Dissim | Energy | -0.0102 | 0.501 |
Homog | -0.0081 | 0.741 |
None | -0.0203 | 0.01 | ✓
Energy | Homog | 0.0021 | 1 |
None | -0.0101 | 0.514 |
Homog | None | -0.0122 | 0.295 |

Table 2: Tukey test results for accuracy across each experimental feature
method. The rejections are the same for all the metrics. Mean difference,
adjusted P values, and null hypothesis rejection status are given for each
pair. There does not exist a statistically significant difference between all
features and contrast, which are the two methods with better results in table
1. All features and Contrast show statistically significant improvement over
other feature sets.

To test our classifier before and after the HITL inspired improvement, we
trained it on our ultrasound dataset for 30 epochs. The basic data record was
chosen to be frames instead of full videos. The training set was composed of
1,974 full thickness, 1,802 partial thickness, and 1,493 unburnt skin
examples. The testing set was composed of 1,309 full thickness, 986 partial
thickness, and 141 unburnt skin examples. Another 1072, 805, and 141
respective samples were set aside for validation and hyper-parameter tuning.
Each dataset was composed of frames from unique videos.

Preprocessing consisted of a normalization step to translate image tensors to
the [0, 1] domain. A center crop was also performed to isolate the true
ultrasound data from scan metadata, which could otherwise contaminate the
results. This crop brought all images to the size 800x1,000. Resizing was also
done, compressing the image to the size 224x224 — this being the standard
input size to ResNet18.

As this dataset was unbalanced, average F1 score and score standard deviation
was collected alongside accuracy. All statistics were averaged over 6 training
runs with different random seeds for initialization. To show that no single
feature is sufficient to gain the best results, we train models with no extra
features (baseline), one extra feature (for each GLCM method), and all
features combined. We also train a Support Vector Machine SVM for comparison,
as this model is common in the burn analysis domain Huang et al. (2021); Lee
et al. (2020); Yadav et al. (2019). Results can be seen in Table 1 and
statistical analysis can be seen in Table 2. In summary, our results show
statistically significant improvements when adding some or all of the GLCM
texture features. In particular, contrast and dissimilarity are important
additions to our ResNet features.

##  5 Conclusions

This work has introduced a method for improving the accuracy of ultrasound
burn classifiers using explainable AI and a human-in-the-loop framework. Our
real-data experiments show that human experts can leverage their knowledge and
the insights of explainable AI to uncover new "feature bottlenecks". Further,
the results of these experiments show that by modifying the network according
to these insights, significant improvements can be gained. In our future work,
we will be adapting this process to different ultrasound modalities and
increasing automation and explainability within the framework.

##  6 Acknowledgments

This work was supported by the Office of the Assistant Secretary of Defense
for Health Affairs under Award No. 6W81XWH-21-2-0030, by the National
Institutes of Health under award 5R21LM013711-02, and by the National Science
Foundation under Grant NSF #2140612. Opinions, interpretations, conclusions
and recommendations are those of the authors and are not necessarily endorsed
by the Department of Defense or by the National Science Foundation.

## References

  * Chauhan and Goyal [2020] Joohi Chauhan and Puneet Goyal.  Convolution neural network for effective burn region segmentation of color images.  _Burns_ , 2020.  Publisher: Elsevier.
  * Cirillo et al. [2019] Marco D Cirillo, Robin Mirdell, Folke Sjöberg, and Tuan D Pham.  Tensor decomposition for colour image segmentation of burn wounds.  _Scientific reports_ , 9(1):1–13, 2019.  Publisher: Nature Publishing Group.
  * Deng et al. [2009] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Felzenszwalb and Huttenlocher [2004] Pedro F Felzenszwalb and Daniel P Huttenlocher.  Efficient graph-based image segmentation.  _International journal of computer vision_ , 59(2):167–181, 2004.
  * Gohel et al. [2021] Prashant Gohel, Priyanka Singh, and Manoranjan Mohanty.  Explainable ai: current status and future directions.  _arXiv preprint arXiv:2107.07045_ , 2021.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 770–778, 2016.  doi:10.1109/CVPR.2016.90.
  * Huang et al. [2021] Samantha Huang, Justin Dang, Clifford C. Sheckter, Haig A. Yenikomshian, and Justin Gillenwater.  A systematic review of machine learning and automation in burn wound evaluation: A promising but developing frontier.  _Burns_ , 47(8):1691–1704, 2021.  ISSN 0305-4179.  doi:https://doi.org/10.1016/j.burns.2021.07.007.  URL https://www.sciencedirect.com/science/article/pii/S0305417921001777.
  * Kingma and Ba [2015] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Yoshua Bengio and Yann LeCun, editors, _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.  URL http://arxiv.org/abs/1412.6980.
  * Lee et al. [2020] Sangrock Lee, Hanglin Ye, Deepak Chittajallu, Uwe Kruger, Tatiana Boyko, James K Lukan, Andinet Enquobahrie, Jack Norfleet, Suvranu De, and others.  Real-time burn classification using ultrasound imaging.  _Scientific reports_ , 10(1):1–13, 2020.  Publisher: Nature Publishing Group.
  * Lundberg et al. [2018] S Lundberg, B Nair, M Vavilala, and et al.  Explainable machine-learning predictions for the prevention of hypoxaemia during surgery.  _Nature Biomedical Engineering_ , 2:749–760, 2018.
  * Narouze [2018] Samer N Narouze.  _Atlas of ultrasound-guided procedures in interventional pain management_.  Springer, 2018.
  * Ribeiro et al. [2016] Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin.  "why should I trust you?": Explaining the predictions of any classifier.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016_ , pages 1135–1144, 2016.
  * Rowland et al. [2019] Rebecca A Rowland, Adrien Ponticorvo, Melissa L Baldado, Gordon T Kennedy, David M Burmeister, Robert JJ Christy, Nicole P Bernal, and Anthony J Durkin.  Burn wound classification model using spatial frequency-domain imaging and machine learning.  _Journal of biomedical optics_ , 24(5):056007, 2019.  Publisher: International Society for Optics and Photonics.
  * Sayres et al. [2019] Rory Sayres, Ankur Taly, Ehsan Rahimy, Katy Blumer, David Coz, Naama Hammel, Jonathan Krause, Arunachalam Narayanaswamy, Zahra Rastegar, Derek Wu, Shawn Xu, Scott Barb, Anthony Joseph, Michael Shumski, Jesse Smith, Arjun B. Sood, Greg S. Corrado, Lily Peng, and Dale R. Webster.  Using a deep learning algorithm and integrated gradients explanation to assist grading for diabetic retinopathy.  _Ophthalmology_ , 126(4):552–564, 2019.  ISSN 0161-6420.  doi:https://doi.org/10.1016/j.ophtha.2018.11.016.  URL https://www.sciencedirect.com/science/article/pii/S0161642018315756.
  * Seal et al. [2018] Ayan Seal, Debotosh Bhattacharjee, and Mita Nasipuri.  Predictive and probabilistic model for cancer detection using computer tomography images.  _Multimedia Tools and Applications_ , 77(3):3991–4010, 2018.
  * Sen et al. [2016] Chandan K Sen, Subhadip Ghatak, Surya C Gnyawali, Sashwati Roy, and Gayle M Gordillo.  Cutaneous imaging technologies in acute burn and chronic wound care.  _Plastic and reconstructive surgery_ , 138(3 Suppl):119S, 2016.  Publisher: NIH Public Access.
  * Simonyan and Zisserman [2015] Karen Simonyan and Andrew Zisserman.  Spatial reasoning network for zero-shot constrained scene generation.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.  URL http://arxiv.org/abs/1409.1556.
  * Simonyan et al. [2014] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps.  In _In Workshop at International Conference on Learning Representations_. Citeseer, 2014.
  * Singla et al. [2018] Neeru Singla, Vishal Srivastava, and Dalip Singh Mehta.  In vivo classification of human skin burns using machine learning and quantitative features captured by optical coherence tomography.  _Laser Physics Letters_ , 15(2):025601, 2018.  Publisher: IOP Publishing.
  * Sullivan et al. [2001] Tory P Sullivan, William H Eaglstein, Stephen C Davis, and Patricia Mertz.  The pig as a model for human wound healing.  _Wound repair and regeneration_ , 9(2):66–76, 2001.  Publisher: Wiley Online Library.
  * Szegedy et al. [2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1–9, 2015.  doi:10.1109/CVPR.2015.7298594.
  * Thatcher et al. [2016] Jeffrey E Thatcher, John J Squiers, Stephen C Kanick, Darlene R King, Yang Lu, Yulin Wang, Rachit Mohan, Eric W Sellke, and J Michael DiMaio.  Imaging techniques for clinical burn assessment with a focus on multispectral imaging.  _Advances in wound care_ , 5(8):360–378, 2016\.  Publisher: Mary Ann Liebert, Inc. 140 Huguenot Street, 3rd Floor New Rochelle, NY 10801 USA.
  * Vedaldi and Soatto [2008] Andrea Vedaldi and Stefano Soatto.  Quick shift and kernel methods for mode seeking.  In _European conference on computer vision_ , pages 705–718. Springer, 2008.
  * Yadav et al. [2019] DP Yadav, Ashish Sharma, Madhusudan Singh, and Ayush Goyal.  Feature Extraction Based Machine Learning for Human Burn Diagnosis From Burn Images.  _IEEE Journal of Translational Engineering in Health and Medicine_ , 7:1–7, 2019.  Publisher: IEEE.


# CBAGAN-RRT: Convolutional Block Attention Generative Adversarial Network for
Sampling-Based Path Planning

Abhinav Sagar
University of Maryland
<EMAIL_ADDRESS>Sai Teja Gilukara
University of Maryland
<EMAIL_ADDRESS>
###### Abstract

Sampling-based path planning algorithms play an important role in autonomous
robotics. However, a common problem among the RRT-based algorithms is that the
initial path generated is not optimal and the convergence is too slow to be
used in real-world applications. In this paper, we propose a novel image-based
learning algorithm (CBAGAN-RRT) using a Convolutional Block Attention
Generative Adversarial Network with a combination of spatial and channel
attention and a novel loss function to design the heuristics, find a better
optimal path, and improve the convergence of the algorithm both concerning
time and speed. The probability distribution of the paths generated from our
GAN model is used to guide the sampling process for the RRT algorithm. We
train and test our network on the dataset generated by [55] and demonstrate
that our algorithm outperforms the previous state-of-the-art algorithms using
both the image quality generation metrics like IOU Score, Dice Score, FID
score, and path planning metrics like time cost and the number of nodes. We
conduct detailed experiments and ablation studies to illustrate the
feasibility of our study and show that our model performs well not only on the
training dataset but also on the unseen test dataset. The advantage of our
approach is that we can avoid the complicated preprocessing in the state
space, our model can be generalized to complicated environments like those
containing turns and narrow passages without loss of accuracy, and our model
can be easily integrated with other sampling-based path planning algorithms.

##  1 Introduction

Path planning is one of the fundamental topics in robotics which aims to find
a collision-free path around obstacles in the environment. Various approaches
have been proposed over the years like grid-based ones using Dijkstra’s and A*
which suffer from the high time and memory cost as the dimensionality of the
search space increases. Sampling-based path planning algorithm solves the
aforementioned problem by drawing samples from the state space, constructing a
random graph, and searching for a feasible path using the Probabilistic
Roadmap (PRM) or the Rapidly Random Exploring Tree (RRT) thus offering better
scalability to higher dimensional problems. A random graph is constructed in
PRM after the sampling process and hence works better for multi-query problems
while the random tree is constructed randomly and incrementally from the start
to the goal node thus making it better for single-query problems. The best
part about the RRT or the PRM-based algorithms is that they guarantee that a
solution is found if it exists because of the probabilistic completeness
property. However, sampling-based path planning algorithms’ convergence speed
is slow and the initial paths generated are not optimal because they sample
uniformly in the state space and have many time-consuming and repeated
functions like collision checking and the tree rewiring operation in the case
of the RRT* algorithm. The path quality generated is proven to be not optimal
and these algorithms are too slow in challenging environments like those
having many turns or narrow passages. In this work, we propose a novel GAN-
based neural network that can quickly predict the probability distribution of
feasible paths on a map before being fed to the sampling-based path planning
algorithm thus speeding up the convergence both in terms of the time and the
speed.

##  2 Related Work

Sampling-based path planning algorithms [18]have been highly successful in
various robotics problems from autonomous driving to drones to warehouse
robots. The first sampling-based path planning algorithm was the original RRT
algorithm by [20] which solved several challenges faced by the grid-based
algorithms like faster convergence and better initial solution. It was
considered the perfect sweet spot between probabilistic road maps and grid-
based path planning algorithms getting the best of both worlds as shown in
[19] and [22].

[14] showed that the RRT* algorithm possesses asymptotically optimal property
and the quality of the path generated was much better than the RRT algorithm
using a novel rewiring operation and a novel nearest neighbor search
operation. However, the algorithm was much slower than the RRT algorithm and
it took a long time to converge and the paths obtained were not always
correct. The RRT* algorithm works under holonomic constraints, offline
planning mode, point kinematic model, uniform sampling strategy, and Euclidean
term as the distance metric.

[15] proposed Anytime RRT* which was able to generate better trajectories
using a branch and bound adaptation technique and committed trajectory
technique and was able to generate better optimal paths while also speeding up
the convergence thus demanding lesser computational resources. However, the
paths generated were still not optimal in a lot of cases and the algorithm
forced node removal while building the tree. Anytime RRT* algorithm works
under non-holonomic constraints, online planning mode, Dublin car kinematic
model, uniform sampling strategy, and a Euclidean plus a velocity term as the
distance metric.

[33] proposed RRT* Smart which improved the efficiency and accelerated the
rate of the convergence of the algorithm using a novel path optimization
technique and intelligent sampling. However, this method had its drawbacks
like manually defining the heuristics upon which the algorithm depended and
the fact that there was a trade-off between exploration space and the
convergence rate. The RRT* Smart algorithm works under holonomic constraints,
offline planning mode, point kinematic model, intelligent sampling strategy,
and Euclidean term as the distance metric.

Machine learning-based approaches have been applied and have had their share
of success by either improving the heuristics to speed up the convergence of
the algorithm or avoiding getting stuck in the local minimum. Deep learning
powered by deep neural networks has ushered a new era in this domain and has
been quite successful in solving various problems in sampling-based path
planning [43], [44], [30], [1], [2], [52], [25], [3].

Recent years have also seen an advent of generative models like the Generative
Adversarial Network like Pix2pix [11] and Variational Autoencoder to generate
the promising region using paired labeled training data and have shown to
outperform all previous numerical and algorithmic-based approaches while
reducing the time taken, number of nodes, and the length of the initial path
to convergence [55], [27], and [29]. Attention mechanism [42] has resulted in
a breakthrough and has been a driving force behind various state-of-the-art
architectures across modalities like computer vision and natural language
processing [53], [45], [48], [5], and [5] and offers great potential in the
image based path planning landscape.

##  3 Method

###  3.1 Path Planning Problem

Assuming the state space of the planning problem is represented by X and
$X_{f}ree$ is the collision-free subspace of the state X. Then
$X_{o}bs=X-X_{f}ree$ would denote the obstacle space in the map. Let’s assume
the start and the goal state be represented by $x_{s}$ and $x_{g}$ and any 2
random points by $x_{1}$ and $x_{2}$ in the map. Let’s also denote the ball
center of radius r at x by $B(x,r)$. The Euclidean distance between the 2 can
be denoted by $x_{1}-x_{2}$. Assuming $\sigma$ denotes a feasible path and for
the sequence of states we have $\sigma(0)$ = $x_{s}$, $\sigma(t)$ =
$B(x_{g},r_{g}$ where $\sigma(t)\subset X_{f}ree$. Hence the cost of the
optimal path can be represented by
$c(\sigma)=\Sigma_{t=1}^{t}\sigma(t)-\sigma(t-1)$. The path length can be
obtained by minimizing the cost function. The following definition is used in
the RRT algorithm:

\- V: Set of vertices in the sampling tree. \- E: Set of edges between the
vertices in the sampling tree. \- Sample free: Procedure of sampling a random
node from the state space. \- Nearest(V, x): Finding the nearest node from the
vertex V to point X using Euclidean distance as the metric. \- Steer($x_{1}$,
$x_{2}$): Steer from $x_{1}$ to $x_{2}$ along the path $\delta(t)$. \-
$Obs_{free}\delta(t)$: Find if a collision-free and feasible path exists in
the map or not.

###  3.2 Dataset

We used the dataset generated by [55] to validate our results. The dataset was
generated by randomly placing different obstacles on the map and randomly
sampling the start and goal nodes which are denoted by red and blue dots on
the map respectively. The RRT algorithm was run to generate the feasible path
which is shown in green color or the ground truth. The dimensions of all the
images are (3x256x256) where the height and the width of the images are 256
and the number of channels is 3. We use 8000 images for training and 2000
images for testing respectively using the dataset by [55].

###  3.3 Data Augmentation

The parameters used in the data augmentation like height shift of the map,
width shift of the map, shift step of the map, rotation probability of the
map, and the number of maps generated of the map are shown below:

Table 1: Data Augmentation Parameters Parameter | Value
---|---
Height shift of map | 2
Width shift of map | 2
Shift step of map | 1
Rotation probability of map | 0.5
Number of maps to be generated | 10

The following data augmentation methods were used to increase the quality and
quantity of the dataset:

1\. Rescaling: We rescale the pixel values by rescaling factor 1/255.

2\. Rotation: Random rotations with the setup degree range between [0, 360]
was used.

3\. Height and Width Shift: Shifting the input to the left or right and up or
down was performed.

4\. Shearing Intensity: It refers to the shear angle (unit in degrees) in a
counter-clockwise direction.

5\. Brightness: It uses a brightness shift value from the setup range.

A few sample ground truth images from the dataset generated by [55] showing
the promising region in the environment map are illustrated below which goes
into the training:

Figure 1: The first row shows images from the dataset with the white region as
the free space, the black region as the obstacle space, the start node in
color red, the goal node in color blue, and the green area as the promising
region superimposed on the map. The second row shows only the promising region
on the map. The promising region or the region of interest here is the ground
truth labels which go into the training along with the images depicted in the
first row.

###  3.4 Network Architecture

The fundamental goal in this image-based heuristic generation for the
environment is to map an RGB image $X\in R^{H\times W\times 3}$ to a semantic
map $Y\in R^{H\times W\times C}$ with the same spatial resolution $H\times W$,
where $C$ is the number of classes of objects present in image which is 2, in
this case, being the free space and the obstacle space respectively. The input
image $X$ is converted to a set of feature maps ${F_{l}}$ where l=1,…,3 from
each network stage, where $F_{l}\in R^{H_{l}×W_{l}×C_{l}}$ is a
$C_{l}$-dimensional feature map. The input image is first passed through a
block comprising convolutional, batch normalization, and ReLU activation
functions. We express a convolution layer $W^{n}(x)$ as follows:

$\mathrm{W}^{n}(x)=\mathbf{W}^{n\times n}\odot x+\mathbf{b}$ (1)

where $\odot$ represents the convolution operator, $W^{n\times n}$ represents
the $n\times n$ convolutional kernel, $x$ represents the input data and $b$
represents the bias vector.

####  3.4.1 Spatial Attention Module

The spatial attention module is used for capturing the spatial dependencies of
the feature maps. The spatial attention (SA) module in our network is defined
below:

$f_{{SA}}(x)=f_{sigmoid}\left({W}_{2}\left(f_{{ReLU}}\left({W}_{1}(x)\right)\right)\right)$
(2)

where $W_{1}$ and $W_{2}$ denotes the first and second $1\times 1$ convolution
layer respectively, $x$ denotes the input data, $f_{Sigmoid}$ denotes the
sigmoid function, $f_{ReLU}$ denotes the ReLu activation function. The spatial
attention module used in this work is shown in the figure below:

Figure 2: Details of our spatial attention module

.

####  3.4.2 Channel Attention Module

The channel attention module is used for extracting high-level multi-scale
semantic information. The channel attention (CA) module in our network is
defined below:

$f_{{CA}}(x)=f_{sigmoid}({W}_{2}(f_{{ReLU}}({W}_{1}f_{{AvgPool}}^{1}(x))))$
(3)

where $W_{1}$ and $W_{2}$ denotes the first and second $1\times 1$ convolution
layer, $x$ denotes the input data. $f^{1}_{AvgPool}$ denotes the global
average pooling function, $f_{Sigmoid}$ denotes the Sigmoid function,
$f_{ReLU}$ denotes ReLU activation function. The channel attention module used
in this work is shown in the figure below:

Figure 3: Details of our channel attention module

.

####  3.4.3 Convolutional Block Attention Generative Adversarial Network

GANs are a family of unsupervised generative models which learns to generate
samples from a given distribution [9]. Given a noise distribution, Generator
$G$ tries to generate samples while the Discriminator $D$ tries to tell
whether the generated samples are from the correct distribution or not. Both
the generator and discriminator are trying to fool each other, thus playing a
zero-sum game. In other words, both are in a state of Nash Equilibrium. Let
$G$ represent the generator, $D$ the discriminator, loss function used for
training GAN can be written as shown in Equation 1:

$\mathcal{F}(\mathcal{D},\mathcal{G})=\mathbb{E}_{\mathbf{x}\sim
p_{\mathbf{x}}}[-\log\mathcal{D}(\mathbf{x})]+\\\ \mathbb{E}_{\mathbf{z}\sim
p_{\mathbf{z}}}[-\log(1-\mathcal{D}(\mathcal{G}(\mathbf{z})))]$ (4)

where $z$ is latent vector, $x$ is data sample, $p_{z}$ is probability
distribution over latent space and $p_{x}$ is the probability distribution
over data samples. The zero-sum condition is defined in Equation 2:

$\min_{G}\max_{D}\mathcal{F}(\mathcal{D},\mathcal{G})$ (5)

The Convolutional layer has a kernel size of 4, a stride of 2, a padding of 1,
and uses batch normalization and LeakyReLU as the activation function. The
Transposed Convolutional layer uses a kernel size of 4, a stride of 2, and
padding of 1, and uses instance normalization and LeakyReLU as the activation
function. The number of input and output channels in the Residual Block is
128. The kernel size is 4, the stride is 2, the padding is 1, and uses
instance normalization and ReLU as the activation function. The Convolutional
Block Attention Module shown has a reduction in channel attention value of 16,
number of channels in the input layer of channel attention as 512, kernel size
in spatial attention of 7, uses ReLU as the activation function, and Kaiming
type as the weight initialization. The parameters for the Generator in our GAN
are: the number of map input channels is 3, the number of point input channels
is 3, the number of noise channels is 1, the number of hidden channels is 64,
and the number of output (ROI) channels is 3. The parameters used in the Map
Discriminator of our GAN are the number of Map channels as 3, the number of
ROI channels as 3, the number of hidden channels as 64, and the number of
output (ROI) channels as 1. The parameters used in the Point Discriminator of
our GAN are the number of Point channels as 3, the number of ROI channels as
3, the number of hidden channels as 64, and the number of output (ROI)
channels as 1. The overall structure of our GAN network is shown in the figure
below:

Figure 4: Details of the GAN network architecture

.

###  3.5 Loss Functions

A combination of binary cross entropy and dice losses has been used to train
the network. The first part binary cross-entropy is a commonly used loss
function for classification problems as shown by [8]. Every pixel in the image
needs to be classified and hence loss function can be written as shown in the
equation below:

$\mathcal{L}_{CE}=-\sum_{i,j}y_{i,j}\log\widehat{y}_{i,j}+\left(1-y_{i,j}\right)\log\left(1-\widehat{y}_{i,j}\right)$
(6)

The first of them is a binary cross-entropy loss in the x-direction as shown
below:

$L_{bce}^{x}\left(\mathcal{P}^{x},\hat{\mathcal{P}}^{x}\right)=\sum_{i=1}^{H}\sum_{j=1}^{W}\mathcal{P}_{i,j}^{x}\log\left(\hat{\mathcal{P}}_{i,j}^{x}\right)+\\\
\left(1-\mathcal{P}_{i,j}^{x}\right)\log\left(1-\hat{\mathcal{P}}_{i,j}^{x}\right)$
(7)

A similar binary cross-entropy loss term is also used in the y-direction. The
total binary cross entropy loss term can be obtained by summing up the two
individual terms as shown below:

$L_{bce}^{xy}(\mathcal{P},\hat{\mathcal{P}})=L_{bce}^{x}\left(\mathcal{P}^{x},\hat{\mathcal{P}}^{x}\right)+L_{bce}^{y}\left(\mathcal{P}^{y},\hat{\mathcal{P}}^{y}\right)$
(8)

The problem with binary cross entropy loss is that it doesn’t take into
account the class imbalance as the background is the dominant class. This is
one of the fundamental challenges in image segmentation problems. Dice Loss is
robust to the aforementioned problem which is based on Dice Similarity
Coefficient as defined using the equation below:

$L_{\text{dice
}}^{xy}(\mathcal{P},\hat{\mathcal{P}})=1-\frac{2\left|\mathcal{P}^{x}\cap\hat{\mathcal{P}}^{x}\right|+2\left|\mathcal{P}^{y}\cap\hat{\mathcal{P}}^{y}\right|}{\left|\mathcal{P}^{x2}\right|+\left|\hat{\mathcal{P}}^{x2}\right|+\left|\mathcal{P}^{y2}\right|+\left|\hat{\mathcal{P}}^{y2}\right|}$
(9)

Hence, the total loss can be obtained which is a summation of the binary
cross-entropy loss and dice loss which is used for training the model is shown
below:

$L(\mathcal{P},\hat{\mathcal{P}})=L_{\text{bce
}}^{xy}(\mathcal{P},\hat{\mathcal{P}})+L_{\text{dice
}}^{xy}(\mathcal{P},\hat{\mathcal{P}})$ (10)

The loss function used for training the map discriminator can be represented
as shown below:

$\displaystyle\mathcal{L}_{D_{\text{map }}}=L(\mathcal{P},\hat{\mathcal{P}})+$
$\displaystyle\mathbb{E}\left[\log D_{\text{map }}(s,m)\right]+$ (11)
$\displaystyle\left.\mathbb{E}_{[}\log\left(1-D_{\text{map
}}(G(z,m,p),m)\right)\right]$

The loss function used for training the point discriminator can be represented
as shown below:

$\displaystyle\mathcal{L}_{D_{\text{point
}}}=L(\mathcal{P},\hat{\mathcal{P}})+$ $\displaystyle\mathbb{E}\left[\log
D_{\text{point }}(s,p)\right]+$ (12)
$\displaystyle\left.\mathbb{E}_{[}\log\left(1-D_{\text{point
}}(G(z,m,p),p)\right)\right]$

The mean square pixel-wise loss function is defined using the below equation:

$\mathcal{L}_{mse}(X,Y)=\ell_{mse}(G(X),Y)=\|G(X)-Y\|^{2}$ (13)

Hence, the loss function used for training the generator can be represented as
shown below:

$\displaystyle\mathcal{L}_{G}=$ $\displaystyle\alpha_{1}\mathbb{E}\left[\log
D_{\text{map }}(G(z,m,p),m)\right]+$ (14)
$\displaystyle\alpha_{2}\mathbb{E}\left[\log D_{\text{point
}}(G(z,m,p),p)\right]+\mathcal{L}_{mse}(X,Y)$

###  3.6 Path Planning Algorithm

The RRT algorithm is used for single-query applications and is suitable for
dealing with high-dimensional problems. The algorithm incrementally builds a
tree of feasible paths rooted at a start node by randomly expanding nodes in a
collision-free space. Initially, there is an empty vertex set V, edge set E,
and an initial state. At every iteration, a node is randomly sampled and the
nearest node of the existing vertex sex V is connected to the new sampling
node. The vertex set and the edge set get added if the connection works. The
process keeps on repeating until a new sampling node in the goal region is
contained in the expanding tree. Space-filling trees are constructed to search
a path connecting start and goal nodes while incrementally drawing random
samples from the free space. We generate the promising region using GAN to
generate the heuristics which denotes that a feasible path exists with a high
probability. We use RRT as the sampling-based path-planning algorithm. We use
our pre-trained GAN model to guide the sampling process of the RRT algorithm.
The algorithm attempts to find the nearest vertex to the new sample and
connect them and the new vertex is added to the vertex set V and the edge is
added to the edge set E if no obstacles are present in the connection.
Finally, the algorithm returns a new vertex set V and edge set E. The
Heuristic RRT algorithm used in this work is shown below:

Algorithm 1: RRT Algorithm

⬇

RRT(Map, start, goal)

qs = [start]

qs_parent = [0]

while True{

q_rand = random_vertex_generated()

n_idx,q_near = nearest_vertex_check(q_rand)

q_new = new_point_generate(q_near,

q_rand,

n_idx)

if connection_check(q_new){

break

}

}

###  3.7 Training Details

We train the model on Nvidia T4 GPU in Pytorch using mini-batch Stochastic
Gradient Descent (SGD) as the optimizer. The momentum and weight decay of the
optimizer is set to 0.9 and 0.0001 respectively. We use exponential decay as
the learning rate scheduler. We train the model for 50 epochs and 8 as the
batch size. The input maps and the ground truth feasible region map are
combined and sent to training. There are 200 different types of environment
maps present in the dataset.

###  3.8 Evaluation Metrics

We use the time cost which represents the time taken for the algorithm to find
a solution and the number of nodes which denotes the number of nodes explored
on the way to solve as the path planning metrics to evaluate the quality of
our algorithm from the path planning perspective. We use Dice Score (Dice)
also known as F1-score and Intersection over union (IoU) as image quality
metrics to evaluate the performance of our network. Here True positive (TR),
false negative (FN), and false positive (FP) number of pixels calculated can
be calculated separately for each image and averaged over the test set. The
corresponding equations for both these metrics are shown using the equation
below respectively.

$\mathrm{Dice}=\frac{2TP}{2TP+FN+FP}$ (15)

$\mathrm{IoU}=\frac{TP}{TP+FN+FP}$ (16)

We also use Fréchet Inception Distance (FID) to evaluate the quality of
generated promising regions. A lower FID is preferred for better-performing
generative models. Let $D$ represent the CNN used to extract features,
($m_{r}$, $\sigma_{r}$) be the mean and covariance of features extracted from
real samples, and ($m_{f}$, $\sigma_{f}$) be mean and covariance of features
extracted from fake samples with $D$, then the Frechet Inception distance is
defined using the below equation:

$d^{2}\left(\left(m_{r},\Sigma_{r}\right),\left(m_{f},\Sigma_{f}\right)\right)=\left\|m_{r}-m_{f}\right\|_{2}^{2}+\\\
\operatorname{Tr}\left(\Sigma_{r}+\Sigma_{f}-2\left(\Sigma_{r}\Sigma_{f}\right)^{1/2}\right)$
(17)

###  3.9 Experimental Details

The hyperparameters used in training our model like the batch size, number of
epochs, generator learning rate, map discriminator learning rate, point
discriminator learning rate, and the optimizer type with its parameters are
shown below:

Table 2: Hyperparameters Parameter | Value
---|---
Batch Size | 8
Epochs | 20
Generator Learning Rate | 0.0001
Map Discriminator Learning Rate | 0.00005
Point Discriminator Learning Rate | 0.00005
Optimizer | Adam

In addition to the above hyperparameters, the exponential learning rate
scheduler and ReduceLROnPlateau were used. In gradient descent, the value of
momentum was taken as 0.9, $\gamma$ value of 0.1, and weight decay of 0.0005.
The hyperparameter values for the Adam optimizer are ($\beta_{1}$ = 0.5,
$\beta_{2}$ = 0.999. The parameters used in the various loss functions like
Dice loss, IOU loss, Pixel Wise MSE loss, Pixel Wise L1 loss, generator loss,
self-attention generator loss, and adaptive self-attention generator loss
which goes into training are shown below:

Table 3: Loss Function Parameters Parameter | Value
---|---
$\alpha$ in Dice loss | 10
The smooth parameter in Dice loss | 1
$\alpha$ in IOU loss | 10
The smooth parameter in IOU loss | 1
$\alpha$ in Pixel Wise MSE loss | 20
$\alpha$ in Generator loss | 100
$\alpha$ in CBAM Generator loss | 1
$\beta$ in CBAM Generator loss | 1
$\alpha$ in Adaptive CBAM Generator loss | 3

The parameters used in the RRT path planning algorithm are shown below:

Table 4: RRT Algorithm Parameters Parameter | Value
---|---
Step Length | 0.2
Maximum Iterations | 5000
Resolution of path | 1

##  4 Results

Our GAN model outperforms the GAN model by [55] using IOU, Dice, and FID as
the image quality evaluation metrics. Moreover, our model has a lesser number
of parameters and thus takes lesser time for both training and inference which
makes it a better choice for use as a heuristic for a sampling-based path
planning algorithm as shown in Table 5:

Table 5: Comparison of our results vs state of the art networks using IOU, Dice, FID, and the number of parameters as the evaluation metrics. Metrics | IOU | Dice | FID | No. of Parameters
---|---|---|---|---
SAGAN [55] | 88.45 | 93.50 | 66.47 | 1.24 Million
Our Network | 89.22 | 94.16 | 62.04 | 0.88 Million

The qualitative comparison of our results using CBAGAN-RRT versus the state-
of-the-art network architecture SAGAN-RRT in terms of the promising region
generated is shown in Figure 5:

Figure 5: Illustration of the comparison of results obtained. The first column
shows the obstacle map with the white region as the free space and the black
region as the obstacle space. The second column shows the promising region of
the ground truth maps which goes into the training. The third column shows the
promising region generated using the Self-Attention Generative Adversarial
Network used in [55]. The fourth column shows the promising region generated
using our Convolutional Block Attention Generative Adversarial Network. The
red and the black nodes in all the maps denote the start and goal nodes
respectively.

Our model outperforms the previous state-of-the-art network architecture using
both the time cost and the number of nodes as the path planning evaluation
metric. The comparison of using various algorithms like the original RRT
algorithm, SAGAN-RRT, and our CBAGAN-RRT network as the heuristic for the path
planning algorithm using time cost and the number of nodes is shown in Table
6:

Table 6: Comparison of our results versus RRT and SAGAN-RRT on 3 different maps using time cost and the number of nodes as the path planning evaluation metrics. Map | Algorithm | Time Cost(s) | Number of Nodes
---|---|---|---
Map1 | RRT | 2.12 | 202
Map1 |  SAGAN [55] | 1.37 | 157
Map1 | Our Network | 1.26 | 143
Map2 | RRT | 3.65 | 351
Map2 |  SAGAN [55] | 2.40 | 246
Map2 | Our Network | 2.08 | 218
Map3 | RRT | 1.85 | 184
Map3 |  SAGAN [55] | 1.28 | 129
Map3 | Our Network | 1.06 | 110

The qualitative comparison of our results using CBAGAN-RRT versus the original
RRT in terms of the path generated is shown in Figure 6:

Figure 6: Illustration of our results using CBAGAN-RRT versus the original RRT
in terms of the path generated. The left image depicts the path generated on
the original obstacle map where the obstacles are shown in the dark color and
the light color denotes the free space. The right image depicts the path
generated using only the region of interest generated from our GAN model where
the lighter color denotes the area inside the region of interest while the
darker color denotes the region outside the region of interest. Only the
region of interest is used for finding the feasible path and because the area
is much smaller than that in the other image, we can improve the speed of
convergence of the algorithm. The start and the goal nodes are shown in color
blue and red respectively while the red line denotes the path taken.

###  4.1 Ablation Study

We conduct experiments using all the combinations of using spatial and channel
attention modules and noticed that the best results using IOU, Dice, and FID
as the image quality metrics and the time cost and number of nodes as the path
planning metrics were achieved by using a combination of both spatial and
channel attention modules in our network architecture as shown in Table 7:

Table 7: Comparison of our results with a combination of spatial and channel attention modules using IOU, Dice, and FID as the image quality evaluation metrics and time cost and the number of nodes as the path planning metrics. Spatial Attention | Channel Attention | IOU | Dice | FID | Time Cost | No. of Nodes
---|---|---|---|---|---|---
$\checkmark$ | $\checkmark$ | 85.22 | 95.86 | 65.47 | 1.08 | 127
$\checkmark$ | $\times$ | 83.56 | 92.69 | 75.74 | 2.02 | 193
$\times$ | $\checkmark$ | 84.73 | 94.06 | 66.49 | 1.75 | 166
$\times$ | $\times$ | 83.08 | 92.10 | 82.53 | 2.56 | 235

We also conduct experiments using different attention modules and noticed that
CBAM using a combination of spatial and channel attention modules led to the
best results using IOU, Dice, and FID as the image quality metrics and the
time cost and number of nodes as the path planning metrics as shown in Table
8:

Table 8: Comparison of our results with various attention modules like Convolutional Block Attention Module, Dual Attention Module, Efficient Channel Attention Module, Efficient Multi-Scale Attention Module, and Shuffle Attention Module using IOU, Dice, and FID as the image quality metrics and the time cost and number of nodes as the path planning metrics. Attention Module | IOU | Dice | FID | Time Cost | No. of Nodes
---|---|---|---|---|---
CBAM [48] | 85.22 | 95.86 | 65.47 | 1.08 | 127
DAModule [5] | 84.54 | 94.29 | 68.45 | 1.23 | 146
ECAAttention [45] | 84.81 | 94.05 | 72.41 | 1.34 | 150
ShuffleAttention [54] | 85.06 | 95.42 | 70.56 | 1.16 | 133
TripletAttention [31] | 83.60 | 92.87 | 77.15 | 1.40 | 164
SKAttention [24] | 84.58 | 93.70 | 69.95 | 1.28 | 150
SpatialGroupEnhance [23] | 83.82 | 94.99 | 72.63 | 1.42 | 142
SEAttention [10] | 83.19 | 93.15 | 71.75 | 1.27 | 139
CoTAttention [26] | 84.14 | 94.06 | 69.25 | 1.36 | 145

##  5 Conclusions and Future Work

In this paper, we proposed a novel image-based approach using convolutional
block attention-based Generative Adversarial Network network CBAGAN-RRT to
design the heuristics and find better optimal solutions and improve the
convergence thus reducing the computational requirements for sampling-based
path planning algorithms. We use the predicted probability distribution of
paths generated from our model to guide the sampling process of the RRT
algorithm. We demonstrate that our approach performs better than the state-of-
the-art approach both in terms of the quality of feasible paths generated from
an image perspective using IOU, Dice, and FID as the evaluation metrics as
well as metrics like time cost and several nodes, from the path planning
perspective. Our algorithm CBAGAN-RRT differs only in the sampling strategy
from the RRT algorithm hence it can be easily integrated with other sampling-
based path planning algorithms. A future idea could be to solve the problem by
mapping it only as image segmentation and not image generation that is
segmenting the feasible region using a state-of-the-art image segmentation
algorithm and feeding that as heuristics for the path planning to improve the
results. Another idea could be to integrate our GAN model with other sampling-
based path-planning algorithms to study, test and validate the effectiveness
of our approach.

## References

  * [1] Oktay Arslan and Panagiotis Tsiotras.  Machine learning guided exploration for sampling-based motion planning algorithms.  In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2646–2652. IEEE, 2015.
  * [2] Brendan Burns and Oliver Brock.  Sampling-based motion planning using predictive models.  In Proceedings of the 2005 IEEE international conference on robotics and automation, pages 3120–3125. IEEE, 2005.
  * [3] Xuzhe Dang, Lukáš Chrpa, and Stefan Edelkamp.  Deep rrt.  In Proceedings of the International Symposium on Combinatorial Search, volume 15, pages 333–335, 2022.
  * [4] Mohamed Elbanhawi and Milan Simic.  Sampling-based robot motion planning: A review.  Ieee access, 2:56–77, 2014.
  * [5] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu.  Dual attention network for scene segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3146–3154, 2019.
  * [6] Jonathan D Gammell, Siddhartha S Srinivasa, and Timothy D Barfoot.  Informed rrt: Optimal sampling-based path planning focused via direct sampling of an admissible ellipsoidal heuristic.  In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 2997–3004. IEEE, 2014.
  * [7] Yi Gan, Bin Zhang, Chao Ke, Xiaofeng Zhu, Weiming He, and Tohru Ihara.  Research on robot motion planning based on rrt algorithm with nonholonomic constraints.  Neural Processing Letters, 53:3011–3029, 2021.
  * [8] Ian Goodfellow, Yoshua Bengio, and Aaron Courville.  Deep learning.  MIT press, 2016.
  * [9] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Advances in neural information processing systems, pages 2672–2680, 2014.
  * [10] Jie Hu, Li Shen, and Gang Sun.  Squeeze-and-excitation networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132–7141, 2018.
  * [11] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros.  Image-to-image translation with conditional adversarial networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1125–1134, 2017.
  * [12] In-Bae Jeong, Seung-Jae Lee, and Jong-Hwan Kim.  Rrt*-quick: A motion planning algorithm with faster convergence rate.  In Robot Intelligence Technology and Applications 3: Results from the 3rd International Conference on Robot Intelligence Technology and Applications, pages 67–76. Springer, 2015.
  * [13] In-Bae Jeong, Seung-Jae Lee, and Jong-Hwan Kim.  Quick-rrt*: Triangular inequality-based implementation of rrt* with improved initial solution and convergence rate.  Expert Systems with Applications, 123:82–90, 2019.
  * [14] Sertac Karaman and Emilio Frazzoli.  Sampling-based algorithms for optimal motion planning.  The international journal of robotics research, 30(7):846–894, 2011\.
  * [15] Sertac Karaman, Matthew R Walter, Alejandro Perez, Emilio Frazzoli, and Seth Teller.  Anytime motion planning using the rrt.  In 2011 IEEE international conference on robotics and automation, pages 1478–1483. IEEE, 2011.
  * [16] James J Kuffner and Steven M LaValle.  Rrt-connect: An efficient approach to single-query path planning.  In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), volume 2, pages 995–1001. IEEE, 2000.
  * [17] Yoshiaki Kuwata, Gaston A Fiore, Justin Teo, Emilio Frazzoli, and Jonathan P How.  Motion planning for urban driving using rrt.  In 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 1681–1686. IEEE, 2008.
  * [18] Steven M LaValle.  Planning algorithms.  Cambridge university press, 2006.
  * [19] Steven M LaValle, Michael S Branicky, and Stephen R Lindemann.  On the relationship between classical grid search and probabilistic roadmaps.  The International Journal of Robotics Research, 23(7-8):673–692, 2004.
  * [20] Steven M LaValle et al.  Rapidly-exploring random trees: A new tool for path planning.  1998\.
  * [21] Steven M LaValle and James J Kuffner.  Rapidly-exploring random trees: Progress and prospects: Steven m. lavalle, iowa state university, a james j. kuffner, jr., university of tokyo, tokyo, japan.  Algorithmic and computational robotics, pages 303–307, 2001.
  * [22] Steven M LaValle and James J Kuffner Jr.  Randomized kinodynamic planning.  The international journal of robotics research, 20(5):378–400, 2001\.
  * [23] Xiang Li, Xiaolin Hu, and Jian Yang.  Spatial group-wise enhance: Improving semantic feature learning in convolutional networks.  arXiv preprint arXiv:1905.09646, 2019.
  * [24] Xiang Li, Wenhai Wang, Xiaolin Hu, and Jian Yang.  Selective kernel networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 510–519, 2019.
  * [25] Yang Li, Rongxin Cui, Zhijun Li, and Demin Xu.  Neural network approximation based near-optimal motion planning with kinodynamic constraints using rrt.  IEEE Transactions on Industrial Electronics, 65(11):8718–8729, 2018\.
  * [26] Yehao Li, Ting Yao, Yingwei Pan, and Tao Mei.  Contextual transformer networks for visual recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022\.
  * [27] Han Ma, Chenming Li, Jianbang Liu, Jiankun Wang, and Max Q-H Meng.  Enhance connectivity of promising regions for sampling-based path planning.  IEEE Transactions on Automation Science and Engineering, 2022.
  * [28] Liang Ma, Jianru Xue, Kuniaki Kawabata, Jihua Zhu, Chao Ma, and Nanning Zheng.  Efficient sampling-based motion planning for on-road autonomous driving.  IEEE Transactions on Intelligent Transportation Systems, 16(4):1961–1976, 2015.
  * [29] Nachuan Ma, Jiankun Wang, Jianbang Liu, and Max Q-H Meng.  Conditional generative adversarial networks for optimal path planning.  IEEE Transactions on Cognitive and Developmental Systems, 14(2):662–671, 2021.
  * [30] Troy McMahon, Aravind Sivaramakrishnan, Edgar Granados, Kostas E Bekris, et al.  A survey on the integration of machine learning with sampling-based motion planning.  Foundations and Trends® in Robotics, 9(4):266–327, 2022.
  * [31] Diganta Misra, Trikay Nalamada, Ajay Uppili Arasanipalai, and Qibin Hou.  Rotate to attend: Convolutional triplet attention module.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 3139–3148, 2021.
  * [32] Kourosh Naderi, Joose Rajamäki, and Perttu Hämäläinen.  Rt-rrt* a real-time path planning algorithm based on rrt.  In Proceedings of the 8th ACM SIGGRAPH Conference on Motion in Games, pages 113–118, 2015.
  * [33] Jauwairia Nasir, Fahad Islam, Usman Malik, Yasar Ayaz, Osman Hasan, Mushtaq Khan, and Mannan Saeed Muhammad.  Rrt*-smart: A rapid convergence implementation of rrt.  International Journal of Advanced Robotic Systems, 10(7):299, 2013\.
  * [34] Iram Noreen, Amna Khan, and Zulfiqar Habib.  A comparison of rrt, rrt* and rrt*-smart path planning algorithms.  International Journal of Computer Science and Network Security (IJCSNS), 16(10):20, 2016.
  * [35] Iram Noreen, Amna Khan, and Zulfiqar Habib.  Optimal path planning using rrt* based approaches: a survey and future directions.  International Journal of Advanced Computer Science and Applications, 7(11), 2016.
  * [36] Alejandro Perez, Robert Platt, George Konidaris, Leslie Kaelbling, and Tomas Lozano-Perez.  Lqr-rrt*: Optimal sampling-based motion planning with automatically derived extension heuristics.  In 2012 IEEE International Conference on Robotics and Automation, pages 2537–2542. IEEE, 2012.
  * [37] Jie Qi, Hui Yang, and Haixin Sun.  Mod-rrt*: A sampling-based algorithm for robot path planning in dynamic environment.  IEEE Transactions on Industrial Electronics, 68(8):7244–7251, 2020\.
  * [38] Oren Salzman and Dan Halperin.  Asymptotically near-optimal rrt for fast, high-quality motion planning.  IEEE Transactions on Robotics, 32(3):473–483, 2016.
  * [39] Marlin P Strub and Jonathan D Gammell.  Adaptively informed trees (ait*) and effort informed trees (eit*): Asymmetric bidirectional sampling-based path planning.  The International Journal of Robotics Research, 41(4):390–417, 2022\.
  * [40] Zaid Tahir, Ahmed H Qureshi, Yasar Ayaz, and Raheel Nawaz.  Potentially guided bidirectionalized rrt* for fast optimal path planning in cluttered environments.  Robotics and Autonomous Systems, 108:13–27, 2018.
  * [41] Konstantinos I Tsianos, Ioan A Sucan, and Lydia E Kavraki.  Sampling-based robot motion planning: Towards realistic applications.  Computer Science Review, 1(1):2–11, 2007.
  * [42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [43] Jiankun Wang, Xiao Jia, Tianyi Zhang, Nachuan Ma, and Max Q-H Meng.  Deep neural network enhanced sampling-based path planning in 3d space.  IEEE Transactions on Automation Science and Engineering, 19(4):3434–3443, 2021.
  * [44] Jiankun Wang, Tingguang Li, Baopu Li, and Max Q-H Meng.  Gmr-rrt*: Sampling-based path planning using gaussian mixture regression.  IEEE Transactions on Intelligent Vehicles, 7(3):690–700, 2022.
  * [45] Qilong Wang, Banggu Wu, Pengfei Zhu, Peihua Li, Wangmeng Zuo, and Qinghua Hu.  Eca-net: Efficient channel attention for deep convolutional neural networks.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11534–11542, 2020.
  * [46] Dustin J Webb and Jur Van Den Berg.  Kinodynamic rrt*: Asymptotically optimal motion planning for robots with linear dynamics.  In 2013 IEEE international conference on robotics and automation, pages 5054–5061. IEEE, 2013.
  * [47] Kun Wei and Bingyin Ren.  A method on dynamic path planning for robotic manipulator autonomous obstacle avoidance based on an improved rrt algorithm.  Sensors, 18(2):571, 2018.
  * [48] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon.  Cbam: Convolutional block attention module.  In Proceedings of the European conference on computer vision (ECCV), pages 3–19, 2018.
  * [49] Zhenping Wu, Zhijun Meng, Wenlong Zhao, and Zhe Wu.  Fast-rrt: A rrt-based optimal path finding method.  Applied Sciences, 11(24):11777, 2021.
  * [50] Wang Xinyu, Li Xiaojuan, Guan Yong, Song Jiadong, and Wang Rui.  Bidirectional potential guided rrt* for motion planning.  IEEE Access, 7:95046–95057, 2019.
  * [51] Chengke Xiong, Hexiong Zhou, Di Lu, Zheng Zeng, Lian Lian, and Caoyang Yu.  Rapidly-exploring adaptive sampling tree*: a sample-based path-planning algorithm for unmanned marine vehicles information gathering in variable ocean environments.  Sensors, 20(9):2515, 2020.
  * [52] Chenning Yu and Sicun Gao.  Reducing collision checking for sampling-based motion planning using graph neural networks.  Advances in Neural Information Processing Systems, 34:4274–4289, 2021.
  * [53] Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena.  Self-attention generative adversarial networks.  In International conference on machine learning, pages 7354–7363. PMLR, 2019.
  * [54] Qing-Long Zhang and Yu-Bin Yang.  Sa-net: Shuffle attention for deep convolutional neural networks.  In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 2235–2239. IEEE, 2021.
  * [55] Tianyi Zhang, Jiankun Wang, and Max Q-H Meng.  Generative adversarial network based heuristics for sampling-based path planning.  IEEE/CAA Journal of Automatica Sinica, 9(1):64–74, 2021.