xx xx June

DOI: 10.1109/JPHOT.2009.XXXXXXX
1943-0655/$25.00 ©2009 IEEE

# Wavelength Conversion Efficiency Enhancement in Modal Phase Matched
$\chi^{(2)}$ Nonlinear Waveguides

Dongpeng Kang  1 Member, IEEE  Weiqi Zhang  1 Amr S. Helmy  2 Senior Member,
IEEE  Siyuan Yu  1 Liying Tan  1 and Jing Ma1 1\. School of Astronautics and
National Key Laboratory of Science and Technology on Tunable Laser, Harbin
Institute of Technology, 92 West Dazhi Street, Harbin, 150001, China
2\. The Edward S. Rogers Department of Electrical and Computer Engineering,
Centre for Quantum Information and Quantum Control, University of Toronto, 10
King’s College Road, Toronto, Ontario M5S 3G4, Canada

(2009)

###### Abstract

Modal phase matching (MPM) is a widely used phase matching technique in
AlxGa1-xAs and other $\chi^{(2)}$ nonlinear waveguides for efficient
wavelength conversions. The use of a non-fundamental spatial mode compensates
the material dispersion but also reduces the spatial overlap of the three
interacting waves and therefore limits the conversion efficiency. In this
work, we develop a technique to increase the nonlinear overlap by modifying
the material nonlinearity, instead of the traditional method of optimizing the
modal field profiles. This could eliminate the limiting factor of low spatial
overlap inherent to MPM and significantly enhance the conversion efficiency.
Among the design examples provided, this technique could increase the
conversion efficiency by a factor of up to $\sim$290 in an AlxGa1-xAs
waveguide. We further show that this technique is applicable to all
$\chi^{(2)}$ material systems that utilize MPM for wavelength conversion.

Manuscript received March 3, 2008; revised November 10, 2008. First published
December 10, 2008. Current version published February 25, 2009. This research
was sponsored by the National Natural Science Foundation of China (61705053),
China Postdoctoral Science Foundation (2016M600249), Heilongjiang Postdoctoral
Special Funds, Fundamental Research Funds for the Central Universities, and
Natural Sciences and Engineering Research Council of Canada. Corresponding
authors: Dongpeng Kang, Jing Ma (e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS>
Wavelength conversion, second harmonic generation, difference frequency
generation, modal phase matching, compound semiconductor, Aluminum Gallium
Arsenide.

##  1 Introduction

Wavelength conversion in nonlinear materials plays an important role in a
broad range of classical and quantum optical applications, including optical
parametric amplification[1], supercontinuum generation[2], all-optical signal
processing[3], sensing[4], photon pair generation[5, 6], and squeezed state
generation[7], etc. Compact and efficient wavelength conversion can be
achieved in nonlinear waveguides, in which the interacting fields are tightly
confined for a long interaction length comparing to their bulk crystal
counterparts. Waveguide based wavelength converters also make chip-scale
integration with various active and passive components possible, enabling
practical devices with new functionalities. In addition, waveguide dispersion
provides an additional tuning knob in certain applications of wavelength
conversion such as frequency comb generation[8] and pure heralded single
photon generation[9, 10].

Among popular $\chi^{(2)}$ nonlinear materials such as ferroelectrics (mainly
lithium niobate), AlN, GaP, etc, compound semiconductor AlxGa1-xAs is uniquely
attractive due to one of the highest $\chi^{(2)}$ coefficients
($d_{14}$$\approx$119 pm/V for GaAs), large transparency window from 0.9
$\mu$m to 17 $\mu$m, mature fabrication techniques, and perhaps in particular
the ability of monolithic integration with active components due to its direct
bandgap nature. However, phase matching in AlxGa1-xAs is challenging due to
the formidable material dispersion near its bandgap and its non-ferroelectric
nature. Several phase matching techniques have been developed for AlxGa1-xAs
waveguides, including form-birefringence phase matching using low index AlOx
layers[11, 12], quasi-phase matching (QPM) using etch and regrowth process
[13, 14, 15] or quantum-well intermixing (QWI) [16, 17, 18], modal phase
matching (MPM) using either a higher order spatial mode[19, 20] or a mode
guided by Bragg reflections[21, 22]. More recently, QPM in suspended whisper-
gallering-mode micro-disks[23] and snake-shaped curved waveguides[24] have
been demonstrated. In addition, high index contrast thin film AlGaAs-on-
insulator waveguides enabled by wafer-bonding have shown the capabilities of
birefringence phase matching[25] as well as QPM in resonant micro-
cavities[26].

While each of the above mentioned techniques shows certain advantages, the
most efficient wavelength conversion to date, to the best of our knowledge, in
any non-resonant monolithic platform was demonstrated in Bragg reflection
waveguides (BRWs) employing MPM[21]. Yet, among all phase matching techniques,
MPM in general has the lowest modal field overlap due to the oscillating
feature of one of the interacting fields. Although this limiting factor can be
alleviated to some extent through epi-structure engineering which reduces the
amount of negative electric field in the associated mode[19, 21], further
improving the modal overlap, and thus the conversion efficiency, has not been
reported in any AlxGa1-xAs waveguide using MPM.

In this work, we develop a technique to improve the conversion efficiency in a
MPM waveguide by increasing its nonlinear modal overlap. Instead of attempting
to optimize the shape of modal fields, we modify the material nonlinearity
according to the sign of the modal electric field involved. Conversion
efficiency enhancements are verified through a few design examples. Among
them, this technique could improve the conversion efficiency by a factor of
over 200 in the best case scenario. In addition, this technique is not limited
to AlxGa1-xAs but can also be applied to all $\chi^{(2)}$ material systems
which utilize MPM for wavelength conversion.

##  2 Theory and method

We consider a generic three-wave mixing process, in which the pump (p) and
signal ($s$) beams are mixed to produce an idler (i). For collinear
propagation in the waveguide along the $z$-axis, the spatial-temporal
dependence of the electric field of each interaction mode is given by

$E_{j}(x,y,z,t)=A_{j}(z)E_{j}(x,y)\exp[-j(\beta_{j}z-\omega_{j}t)],$ (1)

where $j\in\\{p,s,i\\}$, and $A_{j}(z)$ and $E_{j}(x,y)$ are the slowly
varying amplitude and spatial modal profile, respectively; $\omega_{j}$ is the
angular frequency; $\beta_{j}$ is the propagation constant given by
$\beta_{j}=2\pi n_{j}/\lambda_{j}$, where $n_{j}$ and $\lambda_{j}$ are the
corresponding effective modal index and wavelength respectively.

In the case of continuous wave excitation, we can neglect effects of group
velocity mismatch and third-order nonlinearities. The couple-mode equations
are given by[27]

$\frac{\mathrm{d}A_{p}(z)}{\mathrm{d}z}=-j\rho_{p}\xi
A_{s}^{\ast}A_{i}\exp[-j\Delta\beta z]-\frac{\alpha_{p}}{2}A_{p},$ (2)
$\frac{\mathrm{d}A_{s}(z)}{\mathrm{d}z}=-j\rho_{s}\xi
A_{p}^{\ast}A_{i}\exp[-j\Delta\beta z]-\frac{\alpha_{s}}{2}A_{s},$ (3)
$\frac{\mathrm{d}A_{i}(z)}{\mathrm{d}z}=-j\rho_{i}\xi
A_{p}A_{s}\exp[j\Delta\beta z]-\frac{\alpha_{i}}{2}A_{i},$ (4)

where $\alpha_{j}$ is the linear loss coefficient and
$\Delta\beta=\beta_{i}-\beta_{s}-\beta_{p}$ is the wave number mismatch. Here
we define coefficients $\rho_{j}$ and the nonlinear overlap factor $\xi$ as

$\rho_{j}=(\frac{8\pi^{2}}{n_{p}n_{s}n_{i}c\epsilon_{0}\lambda_{j}^{2}})^{1/2},$
(5)
$\xi=\frac{\iint{d(x,y)E_{p}^{\ast}(x,y)E_{s}(x,y)E_{i}(x,y)}\mathrm{d}x\mathrm{d}y}{(\iint|E_{p}^{\ast}(x,y)|^{2}\mathrm{d}x\mathrm{d}y\iint|E_{s}^{\ast}(x,y)|^{2}\mathrm{d}x\mathrm{d}y\iint|E_{i}^{\ast}(x,y)|^{2}\mathrm{d}x\mathrm{d}y)^{1/2}},$
(6)

where $c$ and $\epsilon_{0}$ are the speed of light and vacuum permittivity,
respectively, while $d(x,y)$ is the spatial distribution of the effective
second order nonlinearity, after taking into account all tensor elements
involved. In the case of $\omega_{p}=\omega_{s}$, Eqs. (2)–(4) describe a
second harmonic generation (SHG) process, otherwise a sum frequency generation
(SFG) process. The coupled-mode equations for difference frequency generation
(DFG) can be modified accordingly.

In the absence of propagation losses, the normalized conversion efficiency,
defined by $\eta=P_{i}/(P_{p}P_{s}L^{2})$, is given by

$\eta=\rho_{i}^{2}\lvert\xi\rvert^{2}\operatorname{sinc}^{2}{(\Delta\beta
L/2)},$ (7)

where $L$ is the nonlinear waveguide length. According to Eq. (7), the maximum
conversion efficiency in a lossless waveguide is uniquely determined by its
nonlinear overlap factor given by Eq. (6).

It should be noted that, the nonlinear overlap factor $\xi$ is not determined
by the modal overlap of the interacting fields themselves, but instead the
overlap of material nonlinearity and the modal fields involved. To illustrate
this point, we show schematically in Fig. 1 (a) a slab waveguide using MPM to
achieve SHG, where two photons in a fundamental mode are converted to one
photon of half the wavelength in a third order mode. Ideally, the two modes
need to have the same spatial distribution to obtain the maximum conversion
efficiency. However, the third order mode has both positive and negative
electric field components, as shown in Fig. 1 (a), which cancel out the
contribution of each other in the overlap integral in Eq. (6). This leads to a
decreased nonlinear overlap factor and thus severely limits the conversion
efficiency of MPM in general.

To reduce this limitation, the conventional technique is to minimize the
negative components of the modal field involved in MPM through epi-structure
engineering, such as the third order mode in a optimized M-waveguide[19], or
the Bragg mode in a BRW[21]. Nevertheless, the nonlinear overlap factors in
such cases are still low, due to the unavoidable existence of negative field
components. Here we propose an alternative solution which can overcome this
limitation. Without even attempting to optimize the field profiles, instead,
we modify the spatial distribution of the second order nonlinearity $d(x,y)$
such that it has negative values wherever the high order mode has negative
modal components. As such, the negative parts of the modal field will not
cancel out the positive parts in the overlap integral in Eq. (6).

Figure 1: (a) Schematic of a three-layer slab waveguide using MPM with a
fundamental mode (red) and a third order mode (green). The waveguide core
consists of regions 2, 3, and 4, while the claddings are regions 1 and 5. (b)
Material nonlinearity distributions for the original waveguide (red solid
line) and modified waveguides with negative nonlinearity (green dashed line)
and reduced nonlinearity (blue dotted line). The claddings are assumed to have
zero nonlinearity.

Taking the structure shown in Fig. 1 (a) for example, the nonlinear overlap
factor, and thus the conversion efficiency, can be maximized if the material
nonlinearity in regions 2 and 4 has a opposite sign to that of region 3, as
shown in Fig. 1 (b). This could be achieved by alternating the crystal
orientation if the nonlinear coefficient is dependent on it, such as the case
of compound semiconductors. If reversing the sign of nonlinearty is not
achievable, the conversion efficiency could also be improved by reducing the
magnitude of nonlinearty in regions 2 and 4, or region 3, as also shown in
Fig. 1 (b).

##  3 Design examples

###  3.1 Reversing the sign of $d(x,y)$

To illustrate the technique presented above, we first provide an comparative
example of AlxGa1-xAs waveguide utilizing MPM, as schematically shown in Fig.
2. The structure is grown on a GaAs [001] substrate with a detailed epi-
structure given in Fig. 2(b), and ridge waveguides are patterned along [110]
direction. For a ridge width of 0.63 $\mu$m in a deeply etched waveguide,
type-II phase matching is achieved between the TE00 and TM00 modes at 1550 nm
and the TE20 mode at 775 nm, with the modal electric field profiles shown in
Figs. 2(c)-(e), respectively. The nonlinear overlap factor, and thus the
conversion efficiency is limited the coexistence of both positive and negative
electric field components in the second harmonic (SH).

Figure 2: (a) Schematic of the AlxGa1-xAs ridge waveguide; (b) Waveguide
cross-section with a ridge width of 630 nm for MPM; (c)-(e) The electric field
profiles of the TE ($E_{x}$) and TM ($E_{y}$) fundamental modes at 1550 nm and
that of the third-order TE20 ($E_{x}$) mode at 775 nm, respectively.

A new design that achieves the maximal conversion efficiency requires the
material nonlinearity $d(x,y)$ to be negative wherever the high order mode
(TE20 in this case) has negative electric field components. In AlxGa1-xAs, the
effective value of $d(x,y)$ depends on the modal polarizations and the
propagation direction. Its dependence on the propagation direction scales as
$\cos{(2\theta)}$, with $\theta$ being the angle between propagation direction
and the crystalline direction of [110]. As such, the sign of $d(x,y)$ is
reversed if the waves propagate along [$\bar{1}10$]. This property has been
applied in a variety of QPM techniques, including orientation-patterned QPM
(OP-QPM)[15], micro-ring and micro-disks[23], as well as zig-zag[28] and
”snake-shaped” waveguides[24]. By utilizing this property, we design a new
structure with domain inversion, in which the crystal is rotated 90∘ in the
regions where the TE20 mode at SH has negative field components, as shown in
Fig. 3(a). Such waveguides could be fabricated by similar techniques used by
orientation-patterned waveguides, except that the ridges are etched along,
rather than perpendicular to the interfaces of alternating domains.

Figure 3: (a) Schematic of a domain inverted AlxGa1-xAs ridge waveguide
superimposed with the electric field profile of the TE20 mode at 775 nm, whose
positive and negative components reside in domains of orthogonal orientations;
(b) Schematics of an AlxGa1-xAs ridge waveguide employing QWI to selectively
reduce the magnitude of nonlinearity in regions where the TE20 mode at 775 nm
has negative field components.

Since the material is optically isotropic, the modal profiles, effective
indices and the phase matching condition in the new design are unchanged in
the ideal case. With $d(x,y)$ having negative values where the TE20 mode field
components are also negative, the nonlinear overlap is increased by a factor
of 17, leading to an increase of the conversion efficiency by a factor of
$\sim$290 if the waveguide is lossless.

In an actual waveguide, the conversion efficiency is also affected by
propagation losses. We assumed typical losses of $\alpha_{\text{FH}}=4.2$ cm-1
(18.3 dB/cm) for the fundamentals at 1550 nm and $\alpha_{\text{SH}}=73.7$
cm-1 (320 dB/cm) for the SH[20], adopted from similar samples in the
literatures, and solve the couple-mode equations given by Eqs. (2)–(4) for SHG
with an input power of 1 mW in each polarization. The dependence of SH powers
on the propagation distance are given by Fig. 4(a) for waveguides without and
with domain inversion. Maximal SH powers of 0.97 $\mu$W and 105.1 $\mu$W are
produced with optimal propagation distances of 0.68 mm and 0.56 mm,
respectively, showing an increase of the normalized conversion efficiency
$\eta$ by a factor of $\sim$160, from $5.2\times 10^{3}$ %W-1cm-2 to
$8.4\times 10^{5}$ %W-1cm-2.

In the structure with domain inversion, the losses are likely higher than
those in the reference design due to two additional interfaces. Therefore we
investigated the dependence of maximal SH power on the losses, as shown in
Fig. 4(b). The maximal SH power decreases as the increase of fundamental
and/or SH losses, as expected. However, an enhancement of the conversion
efficiency could still be obtained as long as neither of the losses is
increased by over 10 folds. As a comparison, the additional losses due to back
scattering from hundreds of interfaces between inverted domains can be limited
to a few dB/cm in the case of OP-QPM with the state of the art fabrication[14,
15]. This value could be a reasonable expectation for the loss increase in our
design.

Figure 4: (a) SH output powers as functions of propagation distance in
waveguides without and with domain inversion; (b) Maximal SH power as a
function of fundamental frequency and SH propagation losses.

Notice that fabricating ridge waveguides of such high aspect ratio and
precision control is possible but challenging. In the above example, ridges as
narrow as of $\sim$200 nm wide are required to be etched in the orientation-
patterning process. This fabrication requirement can be eased significantly
for longer phase matching wavelengths due to reduced material dispersions. For
instance, a similar ridge waveguide can be designed to generate mid-infrared
radiation at an output idler wavelength around 7.5 $\mu$m in its TE00 mode via
a DFG process with the input pump at around 1550 nm in its TE20 mode and
signal at 1950 nm in the TM00 mode[29]. The ridge width required for phase
matching is increased by a factor of 3.5 to 2.2 $\mu$m. With domain inversion,
the nonlinear overlap factor and the conversion efficiency are increased by a
factor of 11 and 113, respectively, in a lossless waveguide. With
representative loss values of 42.6 cm-1 (185 dB/cm), 5.4 cm-1 (23 dB/cm) and
18.6 cm-1 (81 dB/cm) for pump, signal and idler, respectively, the normalized
conversion efficiency is increased by a factor of 56 from 0.90 %W-1cm-2 to
50.8 %W-1cm-2.

###  3.2 Reducing the magnitude of $d(x,y)$

In cases that reversing the sign of $d(x,y)$ is impractical, the conversion
efficiency could still be improved as long as the magnitude of $d(x,y)$ is
reduced in the corresponding regions. In AlxGa1-xAs waveguides, the magnitude
of $d(x,y)$ can be reduced by the technique of QWI if a quantum-well
superlattice is embedded in the waveguide core[16, 17]. This technique has
been commonly used in domain-disordered QPM (DD-QPM), where the nonlinearity
is periodically reduced along the direction of propagation. Comparing to OP-
QPM, DD-QPM using QWI does not required any etch-and-regrowth fabrication
process, and thus the waveguide losses are not considerably increased.

Here we consider QWI being used in previous design examples to reduce the
magnitude of $d(x,y)$ instead of reversing its sign by orientation patterning.
The superlattice core is selectively intermixed such that the magnitude of
$d(x,y)$ is reduced in regions where the TE20 mode has negative modal field
components, as shown in Fig. 3(b). Assuming that $d(x,y)$ is reduced by a
theoretical maximal value of 50 pm/V [30] while the waveguide losses are
unchanged, the normalized conversion efficiencies for corresponding SHG and
DFG processes are increased by factors of 4 and 9, to $1.9\times 10^{4}$
%W-1cm-2 and 7.8 %W-1cm-2, respectively.

The realization of these structures using QWI in quantum confined
heterostructures has been reported before by some of the authors. For example,
having features on the order of 1 $\mu$m has been reported before, and the
transition between intermixed and un-intermixed regions is on the sub-micron
scale[17]. After intermixing, plasma etching can be used to define the final
width of the total waveguide width with an un-intermixed core and two lateral,
intermixed layers on either side of that core. Nevertheless, the fabrication
requirement can be lowered at a longer phase matching wavelength, where the
material dispersion is smaller.

Note that this strategy of improving the nonlinear overlap factor does not
only apply for MPM in compound semiconductors but also in all applicable
material systems. The reduction of material nonlinearity can also be achieved
by incorporating materials with smaller or no nonlinearity in a multi-layer
waveguide core. Here we offer an example based on newly developed aluminum
nitride (AlN)-on-insulator platform, which has been demonstrated for
applications such as electro-optic modulation[31], nonlinear wavelength
conversion[32], as well as photon pair generation[33]. MPM between the TM00
mode at 1550 nm and the TM20 mode at 775 nm in AlN waveguides can be achieved
by choosing an appropriate waveguide dimension, similar to previously shown
AlxGa1-xAs waveguide designs. The c axis of polycrystalline AlN is
perpendicular to the wafer in order to exploit the largest component of AlN’s
$\chi^{(2)}$ tensor ($d_{33}$)[31].

Figure 5: Schematic of the modified AlN-on-insulator ridge waveguide
superimposed with field profile of the TM20 mode at 775 nm.

In the modified design, the AlN ridge is sandwiched by a pair of Ta2O5
dielectric such that the positive part of the TM20 mode at 775 nm locates in
AlN while the negative parts locate in Ta2O5, as shown in Fig. 5. The
fabrication of these structures can be easily achieved using a two-step
lithography process, where initial core ridges are fabricated using
lithography and plasma etching. A subsequent process of conformal deposition
of the outer regions of the core is carried out. The deposition time and
conformal profile of this process can be tuned to suite the required
dimensions of the outer layers of the core. Finally, a lithography along with
plasma etching process can then follow to achieve the profile and dimensions
discussed in this work. Here Ta2O5 is chosen because it has an index of
refraction close to that of AlN, and it has near-zero second order
nonlinearity. This leads to an increase of the nonlinear overlap factor by a
factor of $\sim$12 and the SHG efficiency by 144 without waveguide losses.
With realistic losses of 2.6 cm-1 (11.3 dB/cm) and 86 cm-1 (373 dB/cm) for the
fundamental and SH[31], the normalized efficiency is increased by a factor of
12 from 0.11 %W-1cm-2 to 1.27 %W-1cm-2.

##  4 Discussion and conclusion

Design examples provided above demonstrate the effectiveness of the proposed
technique. In the best case scenario, we observe up to $\sim$290 folds
enhancement in the conversion efficiency assuming the waveguide does not
include any losses. With the existence of losses, the conversion efficiency
can still be improved significantly. The results are summarized in Table 1. It
should be emphasized that the technique proposed here applies to $\chi^{(2)}$
nonlinear waveguides of all material systems that adopt MPM in principle,
despite it was mainly motivated for compound semiconductors.

Table 1: Summary of normalized conversion efficiencies for all design examples provided in Sec. 3. Here $\eta_{\text{ref}}$ represents the normalized conversion efficiency in the reference design, while $\eta_{\text{mod}}$ represents that of the modified design using the proposed technique. Waveguide losses are all included in the calculations. example |  | frequency
---
conversion process
| $\eta_{\text{ref}}$
---
(%W-1cm-2)
| $\eta_{\text{mod}}$
---
(%W-1cm-2)
$\eta_{\text{mod}}/\eta_{\text{ref}}$
AlxGa1-xAs inversion | $\omega_{1550}+\omega_{1550}\rightarrow\omega_{775}$ | $5.2\times 10^{3}$ | $8.4\times 10^{5}$ | 160
$\omega_{1550}-\omega_{1950}\rightarrow\omega_{7556}$ | 0.90 | 50.8 | 56
AlxGa1-xAs QWI | $\omega_{1550}+\omega_{1550}\rightarrow\omega_{775}$ | $5.2\times 10^{3}$ | $1.9\times 10^{4}$ | 4
$\omega_{1550}-\omega_{1950}\rightarrow\omega_{7556}$ | 0.90 | 7.8 | 9
AlN | $\omega_{1550}+\omega_{1550}\rightarrow\omega_{775}$ | 0.11 | 1.27 | 12

The use of specific spatial modes in these designs poses strict constraints on
the waveguide width in each etch process. Deviation from the ideal value
causes the shift of phase matching wavelength and the decrease of nonlinear
overlap. Fabrication of these structures is certainly nontrivial but still
practical, as discussed above. After all, techniques including orientation-
patterning and QWI have been developed in domain-reversal and domain-
suppressed QPM respectively, and high aspect ratio waveguides with low losses
and precision etch control have also been achieved[34, 35]. In addition, the
use of a higher order mode may first seem intimidating as it often has no easy
access. However, waveguide couplers can be implemented to selectively couple
light from a neighboring single mode waveguide into the higher order mode, and
vice versa, as demonstrated in [32, 33].

Although we use TE20 and TM20 modes in the design examples, this technique
could be applied to MPM waveguides using any other spatial mode provided the
material nonlinearity can be modified accordingly. In above designs, material
nonlinearities are altered horizontally (parallel to the wafer surface), and
horizontal higher order modes (TE20 and TM20) are defined lithographically to
achieve phase matching. Instead, if material nonlinearities can be modified in
the vertical direction (perpendicular to the wafer surface), and the thickness
of each layer can be accurately controlled, vertical higher order modes, such
as TE02 and TM02, can be used. In such cases, requirements on ridge width and
etch depth can be significantly relaxed. For instance, in AlxGa1-xAs, quantum-
well superlattices could be embedded in the epi-stricture of a M-shaped
waveguide[19] or a BRW[21, 22, 29] followed by intermixing to reduce the
material nonlinearity in certain regions. As for AlN, it can be deposited on a
layer of Ta2O5 on silica, followed by a cap Ta2O5 layer. In this case, the
TM02 mode is used for the SH, while the ridge width is non-essential. During
the preparation of this manuscript, we noticed a recent work based on thin
film lithium niobate waveguides which took a strategy similar to this AlN
example to increase the conversion efficiency[36].

This technique could be used in conjunction with resonant structures to
further enhance the conversion efficiency. For example, the AlxGa1-xAs
waveguides given above can be embedded in Fabry-Pérot type cavities, and the
AlN waveguides can be lithographically defined as ring resonators[32].

In summary, we have developed a technique to increase the nonlinear overlap
factor and therefore the wavelength conversion efficiency in $\chi^{(2)}$
waveguides utilizing MPM. In contrary to the conventional method of optimizing
modal field overlap, we modify the material nonlinearity by either reversing
the sign or reducing the magnitude in regions which coincide with the negative
parts of the high frequency mode. This could eliminate the limiting factor of
low nonlinear overlap inherent to MPM and thus fully harness the material
nonlinearity for efficient wavelength conversions.

## References

  * [1] Y. Lin, Y. Nabekawa, and K. Midorikawa, “Optical parametric amplification of sub-cycle shortwave infrared pulses,” _Nature Communications_ , vol. 11, p. 3413, 2020.
  * [2] K. Werner, M. G. Hastings, A. Schweinsberg, B. L. Wilmer, D. Austin, C. M. Wolfe, M. Kolesik, T. R. Ensley, L. Vanderhoef, A. Valenzuela, and E. Chowdhury, “Ultrafast mid-infrared high harmonic and supercontinuum generation with $n_{2}$ characterization in zinc selenide,” _Opt. Express_ , vol. 27, no. 3, pp. 2867–2885, Feb 2019. [Online]. Available: http://www.opticsexpress.org/abstract.cfm?URI=oe-27-3-2867
  * [3] A. E. Willner, S. Khaleghi, M. R. Chitgarha, and O. F. Yilmaz, “All-optical signal processing,” _J. Lightwave Technol._ , vol. 32, no. 4, pp. 660–680, Feb 2014. [Online]. Available: http://jlt.osa.org/abstract.cfm?URI=jlt-32-4-660
  * [4] M. L. Weichman, P. B. Changala, J. Ye, Z. Chen, M. Yan, and N. Picqué, “Broadband molecular spectroscopy with optical frequency combs,” _Journal of Molecular Spectroscopy_ , vol. 355, pp. 66–78, 2019. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0022285218302881
  * [5] J. P. Torres, K. Banaszek, and I. Walmsley, “Chapter 5 - engineering nonlinear optic sources of photonic entanglement,” ser. Progress in Optics, E. Wolf, Ed. Elsevier, 2011, vol. 56, pp. 227–331. [Online]. Available: http://www.sciencedirect.com/science/article/pii/B9780444538864000058
  * [6] L. Caspani, C. Xiong, B. J. Eggleton, D. Bajoni, M. Liscidini, M. Galli, R. Morandotti, and D. J. Moss, “Integrated sources of photon quantum states based on nonlinear optics,” _Light: Science & Applications_, vol. 6, p. e17100, 2017.
  * [7] A. Dutt, K. Luke, S. Manipatruni, A. L. Gaeta, P. Nussenzveig, and M. Lipson, “On-chip optical squeezing,” _Phys. Rev. Applied_ , vol. 3, p. 044005, Apr 2015. [Online]. Available: https://link.aps.org/doi/10.1103/PhysRevApplied.3.044005
  * [8] M. Kues, C. Reimer, J. M. Lukens, W. J. Munro, A. M. Weiner, D. J. Moss, and R. Morandotti, “Quantum optical microcombs,” _Nature Photonics_ , vol. 13, pp. 170–179, 2019.
  * [9] D. Kang, A. Pang, Y. Zhao, and A. S. Helmy, “Two-photon quantum state engineering in nonlinear photonic nanowires,” _J. Opt. Soc. Am. B_ , vol. 31, no. 7, pp. 1581–1589, Jul 2014. [Online]. Available: http://josab.osa.org/abstract.cfm?URI=josab-31-7-1581
  * [10] R. P. Marchildon and A. S. Helmy, “Dispersion-enabled quantum state control in integrated photonics,” _Optica_ , vol. 3, no. 3, pp. 243–251, Mar 2016. [Online]. Available: http://www.osapublishing.org/optica/abstract.cfm?URI=optica-3-3-243
  * [11] L. Scaccabarozzi, M. M. Fejer, Y. Huo, S. Fan, X. Yu, and J. S. Harris, “Enhanced second-harmonic generation in AlGaAs/AlxOy tightly confining waveguides and resonant cavities,” _Opt. Lett._ , vol. 31, no. 24, pp. 3626–3628, Dec 2006. [Online]. Available: http://ol.osa.org/abstract.cfm?URI=ol-31-24-3626
  * [12] M. Savanier, A. Andronico, A. Lemaître, C. Manquest, I. Favero, S. Ducci, and G. Leo, “Nearly-degenerate three-wave mixing at 1.55 $\mu$m in oxidized AlGaAs waveguides,” _Opt. Express_ , vol. 19, no. 23, pp. 22 582–22 587, Nov 2011. [Online]. Available: http://www.opticsexpress.org/abstract.cfm?URI=oe-19-23-22582
  * [13] P. S. Kuo, K. L. Vodopyanov, M. M. Fejer, D. M. Simanovskii, X. Yu, J. S. Harris, D. Bliss, and D. Weyburne, “Optical parametric generation of a mid-infrared continuum in orientation-patterned GaAs,” _Opt. Lett._ , vol. 31, no. 1, pp. 71–73, Jan 2006. [Online]. Available: http://ol.osa.org/abstract.cfm?URI=ol-31-1-71
  * [14] X. Yu, L. Scaccabarozzi, A. C. Lin, M. M. Fejer, and J. S. Harris, “Growth of GaAs with orientation-patterned structures for nonlinear optics,” _Journal of Crystal Growth_ , vol. 301-302, no. 0, pp. 163 – 167, 2007. [Online]. Available: http://www.sciencedirect.com/science/article/pii/S0022024806014345
  * [15] K. Fedorova, A. McRobbie, G. Sokolovskii, P. Schunemann, and E. Rafailov, “Second harmonic generation in a low-loss orientation-patterned GaAs waveguide,” _Opt. Express_ , vol. 21, no. 14, pp. 16 424–16 430, 2013\.
  * [16] A. S. Helmy, D. C. Hutchings, T. C. Kleckner, J. H. Marsh, A. C. Bryce, J. M. Arnold, C. R. Stanley, J. S. Aitchison, C. T. A. Brown, K. Moutzouris, and M. Ebrahimzadeh, “Quasi phase matching in GaAs-AlAs superlattice waveguides through bandgap tuning by use of quantum-well intermixing,” _Opt. Lett._ , vol. 25, no. 18, pp. 1370–1372, Sep 2000. [Online]. Available: http://ol.osa.org/abstract.cfm?URI=ol-25-18-1370
  * [17] A. S. Helmy, A. C. Bryce, D. C. Hutchings, J. S. Aitchison, and J. H. Marsh, “Band gap gratings using quantum well intermixing for quasi-phase-matching,” _Journal of Applied Physics_ , vol. 100, no. 12, p. 123107, 2006. [Online]. Available: https://doi.org/10.1063/1.2402034
  * [18] P. Sarrafi, E. Y. Zhu, K. Dolgaleva, B. M. Holmes, D. C. Hutchings, J. S. Aitchison, and L. Qian, “Continuous-wave quasi-phase-matched waveguide correlated photon pair source on a III–V chip,” _Applied Physics Letters_ , vol. 103, no. 25, p. 251115, 2013.
  * [19] S. Ducci, L. Lanco, V. Berger, A. D. Rossi, V. Ortiz, and M. Calligaro, “Continuous-wave second-harmonic generation in modal phase matched semiconductor waveguides,” _Applied Physics Letters_ , vol. 84, no. 16, pp. 2974–2976, 2004. [Online]. Available: http://link.aip.org/link/?APL/84/2974/1
  * [20] D. Duchesne, K. A. Rutkowska, M. Volatier, F. Légaré, S. Delprat, M. Chaker, D. Modotto, A. Locatelli, C. D. Angelis, M. Sorel, D. N. Christodoulides, G. Salamo, R. Arès, V. Aimez, and R. Morandotti, “Second harmonic generation in AlGaAs photonic wires using low power continuous wave light,” _Opt. Express_ , vol. 19, no. 13, pp. 12 408–12 417, Jun 2011. [Online]. Available: http://www.opticsexpress.org/abstract.cfm?URI=oe-19-13-12408
  * [21] P. Abolghasem, J. Han, B. J. Bijlani, A. Arjmand, and A. S. Helmy, “Highly efficient second-harmonic generation in monolithic matching layer enhanced AlxGa1-xAs waveguides,” _Photonics Technology Letters, IEEE_ , vol. 21, no. 19, pp. 1462–1464, 2009.
  * [22] P. Abolghasem, J.-B. Han, D. Kang, B. J. Bijlani, and A. S. Helmy, “Monolithic photonics using second-order optical nonlinearities in multilayer-core Bragg reflection waveguides,” _Selected Topics in Quantum Electronics, IEEE Journal of_ , vol. 18, no. 2, pp. 812–825, 2012.
  * [23] P. S. Kuo, J. Bravo-Abad, and G. S. Solomon, “Second-harmonic generation using quasi-phasematching in a GaAs whispering-gallery-mode microcavity,” _Nature Communications_ , vol. 5, p. 3109, 2014.
  * [24] N. Morais, I. Roland, M. Ravaro, W. Hease, A. Lemaître, C. Gomez, S. Wabnitz, M. D. Rosa, I. Favero, and G. Leo, “Directionally induced quasi-phase matching in homogeneous AlGaAs waveguides,” _Opt. Lett._ , vol. 42, no. 21, pp. 4287–4290, Nov 2017. [Online]. Available: http://ol.osa.org/abstract.cfm?URI=ol-42-21-4287
  * [25] S. May, M. Kues, M. Clerici, and M. Sorel, “Second-harmonic generation in AlGaAs-on-insulator waveguides,” _Opt. Lett._ , vol. 44, no. 6, pp. 1339–1342, Mar 2019. [Online]. Available: http://ol.osa.org/abstract.cfm?URI=ol-44-6-1339
  * [26] L. Chang, A. Boes, P. Pintus, J. D. Peters, M. Kennedy, X.-W. Guo, N. Volet, S.-P. Yu, S. B. Papp, and J. E. Bowers, “Strong frequency conversion in heterogeneously integrated GaAs resonators,” _APL Photonics_ , vol. 4, no. 3, p. 036103, 2019. [Online]. Available: https://doi.org/10.1063/1.5065533
  * [27] T. Suhara and M. Fujimura, _Waveguide nonlinear-optic devices_. Springer, 2003.
  * [28] R. T. Horn and G. Weihs, “Directional quasi-phase matching in curved waveguides,” _arXiv preprint arXiv:1008.2190_ , 2010.
  * [29] D. F. Logan, M. Giguere, A. Villeneuve, and A. S. Helmy, “Widely tunable mid-infrared generation via frequency conversion in semiconductor waveguides,” _Opt. Lett._ , vol. 38, no. 21, pp. 4457–4460, Nov 2013. [Online]. Available: http://ol.osa.org/abstract.cfm?URI=ol-38-21-4457
  * [30] D. C. Hutchings, “Theory of ultrafast nonlinear refraction in semiconductor superlattices,” _IEEE Journal of Selected Topics in Quantum Electronics_ , vol. 10, no. 5, pp. 1124–1132, Sep. 2004.
  * [31] C. Xiong, W. H. P. Pernice, and H. X. Tang, “Low-loss, silicon integrated, aluminum nitride photonic circuits and their use for electro-optic signal processing,” _Nano Letters_ , vol. 12, no. 7, pp. 3562–3568, 2012, pMID: 22663299. [Online]. Available: https://doi.org/10.1021/nl3011885
  * [32] X. Guo, C.-L. Zou, and H. X. Tang, “Second-harmonic generation in aluminum nitride microrings with 2500%/W conversion efficiency,” _Optica_ , vol. 3, no. 10, pp. 1126–1131, Oct. 2016.
  * [33] X. Guo, C.-L. Zou, C. Schuck, H. Jung, and H. X. Tang, “Parametric down-conversion photon-pair source on a nanophotonic chip,” _Light: Science & Applications_, vol. 6, p. e16249, May 2017.
  * [34] G. A. Porkolab, P. Apiratikul, B. Wang, S. H. Guo, and C. J. K. Richardson, “Low propagation loss AlGaAs waveguides fabricated with plasma-assisted photoresist reflow,” _Opt. Express_ , vol. 22, no. 7, pp. 7733–7743, Apr 2014. [Online]. Available: http://www.opticsexpress.org/abstract.cfm?URI=oe-22-7-7733
  * [35] Z. Liao and J. S. Aitchison, “Precision etching for multi-level AlGaAs waveguides,” _Opt. Mater. Express_ , vol. 7, no. 3, pp. 895–903, Mar 2017\. [Online]. Available: http://www.osapublishing.org/ome/abstract.cfm?URI=ome-7-3-895
  * [36] R. Luo, Y. He, H. Liang, M. Li, and Q. Lin, “Semi-nonlinear nanophotonic waveguides for highly efficient second-harmonic generation,” _Laser & Photonics Reviews_, vol. 13, no. 3, p. 1800288, 2019. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/lpor.201800288


# Descriptor Distillation for Efficient Multi-Robot SLAM

Xiyue Guo1, Junjie Hu2, Hujun Bao1 and Guofeng Zhang1†
1State Key Lab of CAD&CG, Zhejiang University 2Chinese University of Hong
Kong, Shenzhen †Corresponding author. Email<EMAIL_ADDRESS>work
was partially supported by NSF of China (No. 61932003).

###### Abstract

Performing accurate localization while maintaining the low-level communication
bandwidth is an essential challenge of multi-robot simultaneous localization
and mapping (MR-SLAM). In this paper, we tackle this problem by generating a
compact yet discriminative feature descriptor with minimum inference time. We
propose descriptor distillation that formulates the descriptor generation into
a learning problem under the teacher-student framework. To achieve real-time
descriptor generation, we design a compact student network and learn it by
transferring the knowledge from a pre-trained large teacher model. To reduce
the descriptor dimensions from the teacher to the student, we propose a novel
loss function that enables the knowledge transfer between two different
dimensional descriptors. The experimental results demonstrate that our model
is 30% lighter than the state-of-the-art model and produces better descriptors
in patch matching. Moreover, we build a MR-SLAM system based on the proposed
method and show that our descriptor distillation can achieve higher
localization performance for MR-SLAM with lower bandwidth.

##  I INTRODUCTION

Multi-robot simultaneous localization and mapping (MR-SLAM) aims at perceiving
environments by utilizing a set of cooperative robots [1, 2, 3, 4]. It is an
extended approach to single-robot SLAM (SR-SLAM) and has attracted significant
attention due to its clear advantage in exploring large-scale environments.
However, a multi-robot system brings one additional restriction: the
limitation of the communication bandwidth. In a MR-SLAM system, to integrate
all the trajectories from the whole team, every robot in the team should share
its keyframe data (including keyframe pose and observed feature points) to
process the inter-robot loop-closure and global localization. This type of
data exchange occupies a high communication capacity, which is very likely to
degrade real-time performance. Some engineering solutions, such as reducing
the communication frequency, and declining the keypoints number, are able to
reduce the bandwidth. On the other hand, these approaches cause another
problem: the localization results in lower accuracy.

In order to maintain high localization performance while reducing the
bandwidth, producing more compact features is a reasonable and universal
solution. To this end, some previous approaches [5] [6] have attempted to
reduce the dimension of the handcraft descriptors. However, this kind of
methods leads to poor matching performance.

Figure 1: A visualization comparison of generated descriptors between previous
methods and our method. As seen, our method extracts significantly compact
descriptors and achieves more accurate matching performance.

Recently, convolutional neural network (CNN) approaches [7, 8, 9, 10] have
shown their superior performance against handcraft descriptors. However, these
methods are specially designed and evaluated only for SR-SLAM and have no
guarantee to be applied to MR-SLAM. Typically, there are two difficulties in
applying CNN-based descriptors to MR-SLAM. First, most state-of-the-art
approaches are designed to produce high-dimensional (more than 128 dimensions)
descriptors. Second, it is hard for most current CNN models to perform real-
time descriptor generation in the mobile platform since they tend to use large
and complex networks. The high model complexity and descriptor dimensions are
two major obstacles that hinder the deployment of learning-based methods to
MR-SLAM.

Figure 2: The diagram of the proposed teacher-student framework for descriptor
distillation. The teacher model is firstly trained with common losses, and its
parameters are fixed during student learning. Then, the high-dimensional
descriptors predicted by the teacher are utilized to guide the student to
produce low-dimensional descriptors with the proposed distillation loss.
Therefore, the loss function of the student model consists of several common
losses as well as the distillation loss.

In this paper, we aim to design a compact network that can produce low-
dimensional descriptors with a small number of parameters. Most importantly,
the generated descriptors have to outperform or at least be comparable to
those predicted by using larger models. Towards this goal, we propose a
knowledge distillation (KD) based framework for descriptor learning. Our
framework is a classic teacher-student combination, where the student is the
compact model which we aim to learn, and the teacher is a pre-trained larger
network whose predictions are used for supervising the student learning.
However, KD requires the same dimensional predictions from the teacher and the
student to calculate a loss function [11, 12, 13]. In our problem, predicted
descriptors between the teacher and the student have different dimensions,
making KD incompatible with descriptor distillation.

To address this problem, we propose a novel distillation loss function that
can distill feature descriptors between the two models even if they yield
different dimensional outputs. To do this, we compute the similarity between
the negative pairs (unmatched pair) distances across the teacher and student
models. Based on this distillation approach, our network is able to alleviate
the performance drop while reducing both model parameters and descriptor
dimension. Fig. 1 shows an example comparison between previous traditional
methods and our method. We fairly test our models on public patch matching
datasets. Moreover, based on our descriptor model, we develop a MR-SLAM system
to quantify the performance of our descriptors on the real-world SLAM task.

To summarize, we present the following contributions:

  * •

We design a teacher-student model to generate the compact binary descriptor.
Our model outperforms state-of-the-art methods in both matching performance
and computational cost.

  * •

We propose a novel distance-based distillation loss, which allows knowledge
transfer between models with different output dimensions.

  * •

We develop a MR-SLAM system based on the new descriptor model as the
evaluation platform. The results show that our model demonstrates good
performance for MR-SLAM task in public EuRoC dataset [14].

##  II RELATED WORK

###  II-A Bandwidth narrowing approaches

Recently, most MR-SLAM systems share all keyframe data between robots or
servers. This data exchange mode will easily exceed the bandwidth limitation
during real-world tasks. Moreover, simply reducing the containing information
or sending frequency of the keyframes will bring poor localization
performance. To deal with the difficulty, Some works only share the compact
global descriptors for loop-closure [3] [15] [16] in normal times. They
exchange the whole keyframe data only when the loop is detected. However, this
type of approaches is not suitable for all multi-robot architectures. Some
other works use high-level information such as semantics to generate object-
level landmarks for inter-robot localization [17] [18]. Since the object-level
landmarks are much less than local feature points, these approaches can
significantly reduce communication bandwidth. However, generating such high-
level landmarks requires specific environments that contain rich semantic
information. Some approaches try to deal with local features to maintain the
generalization while reducing the bandwidth. [5] uses visual BoW indexes to
replace the normal feature descriptors. In our paper, instead of compressing
the existing descriptors, we explore how to generate the compact descriptors
directly.

Figure 3: The detailed network architecture of the teacher model and the
student model. The teacher model contains 7 convolution layers, while our
student model has 4 depthwise separate convolution layers and 2 convolution
layers (the first and last layers)

###  II-B Traditional descriptors

The goal of descriptors works is embedding the patch information into a vector
space so that the corresponding patches have similar descriptors and vice
versa. Early handcraft descriptors are mainly extracted from low-level
information, such as gradient or intensity. SIFT [19] describes the patches by
gradient histograms. It has good matching performance but contains a large
data size. BRIEF [20] has a more compact size due to its structure of binary
vectors. However, its performance is decreased compared with SIFT. In order to
obtain higher quality and more compact descriptors, several learning-based
methods have been proposed. PCA-SIFT [21] exploits principal components
analysis (PCA) to generate the dimensional reduction version of SIFT. LDAhash
[22] utilizes the linear discriminant analysis (LDA) method to binarize the
real-valued descriptor. BinBoost [23] learns 64-bits binary descriptors
through the boost hash function. However, these learning methods are still
based on low-level information; hence huge patch information is lost during
the generation process.

###  II-C CNN-based descriptors

CNN-based approaches are end-to-end methods that generate the descriptors
directly from the patches. In most works, Siamese architectures and triplet
loss are utilized to learn descriptors [24]. During the training process,
distances between matched pairs are shortened, and unmatched pairs are
lengthened. L2Net [7] first generates the descriptors that can be matched by
Euclidean space. Binary descriptors can be obtained by taking the sign of the
real-valued vectors. To further improve the matching quality, HardNet [8] only
takes several closet unmatched pairs for training. SOSNet [9] adds a second-
order-similarity as a regularization term into the training. Based on previous
methods, HyNet [10] proposed a hybrid similarity measurement method to measure
the distance between descriptors during the training. On the other hand, some
works also provide unsupervised models for descriptor learning [25, 26, 27].
However, the matching performances of these works are much less than the
supervised methods. Although the recent CNN methods achieve excellent
performance, the models and descriptors sizes of them are still too large for
multi-robot systems. Hence, In our paper, we follow the state-of-the-art
supervised CNN methods and provide a more lightweight model to produce more
compact descriptors for MR-SLAM.

##  III Methodology

In this section, we present the proposed method in detail. Specifically, we
first introduce preliminary knowledge of learning-based descriptor generation.
Then, we provide our teacher and student architecture. Based on that, we
present the proposed distillation loss, which is the key to the success of
training the student. Finally, we give a detailed learning objective to train
the student. The framework of our descriptor distillation is shown in Fig. 2.

###  III-A Preliminary

####  III-A1 Siamese architecture

Currently, mainstream CNN methods learn descriptors through the Siamese
architectures. It contains twin models that share the same weight during the
training. In the Siamese framework, two batches are prepared on every epoch of
training. Each patch in a batch is negative to the other. On the other hand,
the two batches in one epoch are one-to-one correspondence. There are two
major losses used for training as follows.

####  III-A2 Triplet loss

Triplet loss $L_{T}$ has been widely used in descriptor generating. It
enforces the matched pairs are as close as possible while unmatched pairs are
as far as possible. Most recent methods follow the HardNet strategy [8], which
only extracts the negative pair that contains the smallest distance for
training:

$L_{T}=\sum_{i}^{N}\max\left(0,t+dis\left(R_{i},R_{p}\right)-dis\left(R_{i},R_{n}^{*}\right)\right),$
(1)

In the equation, $R_{i}$ is the current real-valued descriptor, $R_{p}$ is the
corresponding positive descriptor, $R_{n}^{*}$ is the closest negative
descriptor, $N$ is the batch size. $dis$ calculates the $L_{2}$ distance
between two descriptors.

####  III-A3 Binarization loss

Binarization loss term $L_{B}$ contributes to reducing the information loss
when binarizing the real-valued descriptors. It aims to minimize the
difference between the real-valued descriptor and the corresponding binary
descriptor, which is represented as:

$L_{B}=\sum_{i}^{N}\sum_{k}^{D}\frac{1}{D}\sqrt{\left(\left(R_{i}(k)\right)-B_{i}(k)\right)^{2}},$
(2)

where $B(k)$ is the $k$-th value of the binary descriptor $B$, and the $R(k)$
is the $k$-th value of the real-valued descriptor $R$. In addition, $D$ is the
dimension of the descriptors.

###  III-B The Teacher-Student Distillation Framework

Following the typical KD strategy [11, 28, 29, 13], we exploit two siamese
networks to form our teacher-student framework. The architecture of our
network is given by Fig. 3. Our teacher architecture is adopted from HyNet
[10]. It consists of 7 convolutional layers and outputs 128-dimensional
descriptors. Except for the final one, all convolutional layers are followed
by Filter Response Normalisation (FRN) and Thresholded Linear Unit (TLU).
Batch Normalization (BN) and L2-Normalization are placed after the final
convolutional layers.

For our student model, the output dimension is 64. We employ a more shallow
network (6 convolutional layers) to reduce the time consumption. Furthermore,
following the strategy of MobileNet [30], we replace the standard
convolutional layers with depthwise separate convolution layers (DSC), except
for the first and last one. The basic DSC layer consists of two parts. The
first part is a depthwise convolution, it applies one convolution filter for
each input channel. The second part is a 1$\times$1 convolution called
pointwise convolution.

Based on the architectures, each model is able to embed the image patches into
the real-valued descriptors. To obtain the binary descriptors, we use the
following equations to binarize the real-valued descriptors:

$B(k)=\begin{cases}-1&\text{ if }R(k)\leq 0\\\ 1&\text{ if
}R(k)>0\end{cases},$ (3)

where $B(k)$ is the $k-th$ value of the binary descriptor, and $R(k)$ is the
$k-th$ value of the real-valued descriptor that is output from the model.

###  III-C Distillation Loss Function

Due to the difficulty of transferring knowledge between two models with
different dimensional outputs, we design a distance-based distillation loss
function. It consists of a real-valued term and a binary-valued term, which
enforce the descriptor batch from the student model to have a similar
distribution to the teacher model on both real-valued and binary spaces,
respectively. To be specific, it minimizes the difference in distance
estimation between the same pairs by the teacher and student models. In
addition, since the distances of the positive pairs are already extremely
small, the differences between them are not worthy of reduction. Hence, we
only minimize the negative pairs during the training.

As mentioned above, the real-valued term minimizes two models’ difference in
the real-valued space, which can be formulated as:

$L_{Real}=\sum_{i}^{N}\sum_{n}^{N_{n}}\frac{1}{N_{n}}\sqrt{(\lambda_{r}dis(R_{i}^{t},R_{n}^{t})-dis(R_{i}^{s},R_{n}^{s}))^{2}}$
(4)

where $N_{n}$ is the total number of negative descriptors relative to the
current descriptor. $R_{i}^{t}$ and $R_{i}^{s}$ are the current real-valued
descriptors that produced by the teacher and student, respectively, while the
$R_{n}^{t}$ and $R_{n}^{s}$ are the corresponding negative descriptors.
Moreover, due to the different dimensions between the teacher and student, we
add a coefficient $\lambda_{r}$ that is set to 0.95  to adjust the scale of
teacher-side distance.

Binary-valued term transfers the knowledge in binary space. It can be
formulated as follows:

$L_{BIN}=\sum_{i}^{N}\sum_{n}^{N_{n}}\frac{1}{N_{n}}\sqrt{(\lambda_{b}(B_{i}^{t}\cdot
B_{n}^{t})-((B_{i}^{s}\cdot B_{n}^{s}))^{2}},$ (5)

where $\lambda_{b}$ is a scaling coefficient that is usually set to the ratio
between the output dimensions of the two models: ${D_{s}}/{D_{t}}$. Note that
in the training stage, we obtain the binary descriptor $B_{i}$ with the
following representation:

$B_{i}=\frac{R_{i}}{abs(R_{i})+\epsilon},$ (6)

where $\epsilon$ is a coefficient that prevents division by zero and is set to
1e-5 in our experiments.

Finally, we combine the two loss terms as our distillation loss function,
which is represented as:

$L_{distillation}=L_{Real}+\gamma L_{Bin},$ (7)

where the $\gamma$ is weighting coefficient of the binary-valued term.

###  III-D The Student Training

Our training framework consists of two stages. In the first stage, we train
the teacher model using the ADAM optimizer with a learning rate of 0.01. This
process enables the model to generate high-quality 128-dimensional
descriptors. The optimization of this stage relies on the two loss terms:
triplet loss $L_{T}$ and binarization loss $L_{B}$, which formulate the basic
loss function as follows:

$L_{basic}=L_{T}+\alpha_{B}L_{B},$ (8)

where $\alpha_{B}$ are weighting coefficients that control the binarization
loss $L_{B}$.

In the second stage, we train the student model under the teacher’s
supervision, the learning rate is set to 0.01. Therefore, the objective
function includes both basic loss $L_{basic}$ and distillation loss
$L_{distillation}$, it can be represented as:

$L_{train}=L_{basic}+\beta L_{distillation},$ (9)

where $\beta$ is the weighting coefficient of distillation loss , which is set
to 2 during the training.

##  IV Experiment

TABLE I: The quantitative comparisons of different descriptors on UBC dataset. The values are FPR95. Train |  Feature Size | Notredame | Yosemite | Liberty | Yosemite | Liberty | Notredame | Mean
---|---|---|---|---|---|---|---|---
Test | Liberty | Notredame | Yosemite
Traditional Methods
SIFT | 4096-bit | 29.84 | 22.53 | 27.29 | 26.55
BRIEF | 256-bit | 59.15 | 54.57 | 54.57 | 56.23
BinBoost | 64-bit | 20.49 | 21.67 | 16.90 | 14.54 | 22.88 | 18.97 | 19.24
LDAHash | 128-bit | 49.66 | 49.66 | 51.58 | 51.58 | 52.95 | 52.95 | 51.40
CNN Methods
HardNet | 64-bit | 11.89 | 16.40 | 7.60 | 9.93 | 14.50 | 13.20 | 12.25
SOSNet | 64-bit | 11.26 | 15.95 | 7.04 | 9.24 | 13.64 | 11.72 | 11.48
HyNet | 64-bit | 11.72 | 16.16 | 9.94 | 9.03 | 13.14 | 11.87 | 11.97
Ours | 64-bit | 11.84 | 14.68 | 10.14 | 8.39 | 14.24 | 11.58 | 11.81
Ours with distillation | 64-bit | 10.74 | 13.40 | 6.66 | 8.26 | 11.95 | 11.29 | 10.38
CNN Methods with Data Augmentation
HardNet | 64-bit | 11.38 | 16.03 | 6.89 | 8.79 | 14.13 | 14.78 | 12.00
SOSNet | 64-bit | 11.15 | 14.73 | 6.14 | 7.85 | 11.52 | 11.85 | 10.54
HyNet | 64-bit | 9.39 | 13.00 | 5.47 | 7.21 | 11.41 | 10.56 | 9.50
Ours | 64-bit | 9.62 | 12.52 | 8.43 | 7.12 | 11.56 | 11.07 | 10.05
Ours with distillation | 64-bit | 9.15 | 10.87 | 5.24 | 6.50 | 10.57 | 10.23 | 8.76
TABLE II:  The efficiency results of CNN methods. | Parameters | Runtime (ms)
---|---|---
HardNet | 810272 | 20.50
SOSNet | 812067 | 25.64
HyNet | 812067 | 25.62
Ours | 578531 | 18.29

We perform two types of experiments to validate the proposed method. First, we
fairly compare the matching performance of our method and previous methods on
the public UBC dataset [31]. Second, to demonstrate the application of our
method to the MR-SLAM task, we build a MR-SLAM system based on VINS-Mono. With
the system, we then conduct a MR-SLAM experiment on EuROC dataset [14].

###  IV-A Patch matching

####  IV-A1 Dataset and implementation Details

UBC dataset is the most widely used for patch matching [31]. It consists of
three subsets: _Liberty_ , _NotreDame_ , and _Yosemite_. The total dataset
contains around 400k 64 $\times$ 64 patches with labels. Following the
evaluation protocol, we train the models on one subset and test them on the
other two subsets. Specifically, for every test subset, we pick 100k patches
as the test set. Same as previous works, we show the matching performance by
reporting the false positive rate at 95% recall (FPR95).

Several methods are taken as the baseline methods. The first type of methods
are CNN-based methods, including HardNet [8], SOSNet [9], and HyNet [10]. To
ensure fair comparisons, we set the output dimensions of our approach and all
other methods to 64, by adjusting the number of output channels in their last
layers. We report the matching results with and without data augmentation.
Following the previous works, we augment the data by rotating part original
patches before training. The second type of methods are some traditional
methods, including SIFT [19], BRIEF [20], LDAHash [22], and BinBoost [23].  We
also report the parameter number and time consumption (of generating 500
descriptors) of each CNN method to visualize descriptor generation efficiency.
All efficiency experiments are conducted on a desktop PC with an NVIDIA TITAN
XP GPU. Furthermore, to test the generalization ability, we evaluate the
performance of CNN methods trained on the HPatches dataset [32] and tested on
the UBC dataset.

####  IV-A2 Experimental results

The comparison results of FPR95 are shown in Table I. It is clear that the
CNN-based methods are significantly better than the traditional methods. On
the other hand, matching results show that our method, even without
distillation, is comparable to most CNN-based methods. Furthermore, with the
assistance of the KD method, our network is able to outperform all baseline
methods. The average FPR95 values of HardNet, SOSNet, HyNet, and our method
(with KD) are 12.00, 10.54, 9.50, and 8.76, respectively. The lowest matching
error is due to the consistency of our descriptors’ distribution with high
dimensional descriptors. The network efficiency results are shown in Table II.
The results show that our model is around 30% lighter than the baseline CNN
models, and the running speed is also faster than the baseline methods.  Table
III further demonstrates our method’s generalization ability, since it
achieves comparable performance to CNN methods despite being trained on a
different dataset.

TABLE III: Generalization performance (FPR95) of CNN methods ( trained on the HPatches dataset) on the UBC dataset. | UBC (Mean)
---|---
HardNet | 14.36
SOSNet | 13.94
HyNet | 14.21
Ours | 13.21

###  IV-B MR-SLAM Task

####  IV-B1 SLAM architecture

To fairly test the descriptors’ performance on the MR-SLAM task, we design a
MR-SLAM system as the evaluation platform. To this end, we take VINS-Mono as
our basic SLAM system and expand it into a multi-robots version, which we call
the system MR-VINS. The detailed framework of MR-VINS is shown in Fig. 4. The
whole system has a centralized architecture, i.e., the MR-SLAM system consists
of several robots and a central server. We keep the visual-inertial odometry
module on the robots while placing the back-end modules on the server. On the
basis of this framework, we add the CNN descriptor models on the robots’ side.
Since the VINS-Mono exploits optical flow to track the features, the
descriptors are not needed on the visual-inertial odometry module. Hence, in
this experiment, we mainly apply the descriptors for loop closure and global
localization. We place the descriptor models before the robot communication
module so that the descriptors can be integrated into the keyframe message
sent to the server.

####  IV-B2 Dataset and implementation Details

For SLAM evaluation, we use the public EuRoC dataset [14], which collects data
from the Micro Aerial Vehicle (MAV). In our experiments, we evaluate the
methods on the Machine Hall scene. Similar to the benchmark MR-SLAM’s setting
[1],  we first conduct the experiment for SR-SLAM on sequences MH1, MH2, and
MH3, respectively. Then, to test the performance under the MR-SLAM task, we
take the experiment where three robots are run on these sequences (MH1-3)
simultaneously.

The EuRoC dataset contains stereo images and IMU measurements. And in our
experiments, we pick the left images and IMU measurements as the input of our
SLAM system. We take two types of comparisons in the experiments. The first
type is the comparison between different descriptor methods. We compare our
method with previous CNN methods, including HardNet [8], SOSNet [9], and HyNet
[10]. To have a fair comparison, we only change the descriptor extraction
models in our MR-SLAM system. In addition, all the models are trained on
HPatches dataset [32]. The second type is the comparison between our method
and benchmark MR-SLAM systems, including original MR-VINS (without CNN models)
and CCM-SLAM [1] . For evaluation, we take two measures: localization accuracy
and communication bandwidth.

####  IV-B3 Experimental results

Figure 4: The framework of the MR-VINS system. All robots first perform VIO,
which involves pure visual-based pose estimation and local bundle adjustment
with IMU factors. Next, feature descriptors are generated and integrated into
keyframes, which are transmitted to the server. Upon receiving the keyframe
messages, the server executes BoW-based loop closure to detect loops in the
robot trajectories. When loops are detected, the server performs global
localization using PnP-RANSAC. Pose graph optimization is then carried out
based on two factors: VIO and loop factors. Finally, the drift correction is
sent back to the robots to correct real-time pose.

The quantitative results of the SLAM accuracy are shown in Table IV. Compared
to the original MR-VINS, as seen, CNN-based methods are able to improve the
localization performance, especially for MR-SLAM. As the better descriptors
detect more loops, particularly the “hard” inter-robot loops. Moreover, our
method generally achieves comparable accuracy to the CNN methods on both
single and multi-robots tasks. However, since the SLAM performance depends on
more than image matching. The advantages of our localization accuracy are not
obvious as patch matching results.

While comparing our method against CCM-SLAM, we can see, CCM-SLAM achieves
better performance on SR-SLAM tasks since it uses the bundle adjustment (BA)
optimization method, which is able to produce more accurate results than the
pose-graph method we use. On the other hand, our method has a more accurate
result for MR-SLAM due to the much more detected inter-robot loops.
Furthermore, low keyframe sending frequency also limits the accuracy of CCM-
SLAM.

Table V shows the bandwidth of different SLAM systems. Due to the same
dimension of the CNN-based descriptors, the bandwidth of them should be
similar. Hence, we do not report the bandwidth of other CNN methods. As shown
by the results, our method has a much lower bandwidth than the original MR-
VINS and CCM-SLAM. As our message frequency is 3 times more than CCM-SLAM, the
size of our keyframe massage is around 10 times smaller than CCM-SLAM. Narrow
bandwidth is attributed to the extremely compact size of our descriptors (1/4
of normal BRIEF descriptors). In addition, due to the compact keyframe size,
we do not need to reduce the keyframe frequency during the MR-SLAM task
specifically.

TABLE IV: The Trajectory error (RMSE in cm) of Different methods. | SR-SLAM | MR-SLAM
---|---|---
| MH1 | MH2 | MH3 | Mean | MH1-3
CCM-SLAM | 6.10 | 8.10 | 4.80 | 6.33 | 7.70
MR-VINS | 6.15 | 5.88 | 11.58 | 7.87 | 9.91
MR-VINS+HardNet | 7.20 | 5.91 | 9.15 | 7.42 | 7.18
MR-VINS+SOSNet | 6.59 | 4.94 | 9.06 | 6.83 | 7.19
MR-VINS+HyNet | 6.48 | 5.44 | 8.43 | 6.78 | 7.08
MR-VINS+Ours | 6.26 | 4.14 | 9.41 | 6.60 | 6.94
TABLE V: The Bandwidth of Different methods. Methods | Frequency (Hz) | Bandwidth (kb/s)
---|---|---
CCM-SLAM | 2 | 350
MR-VINS | 7 | 224
MR-VINS+Ours | 7 | 119

##  V CONCLUSIONS

In this paper, we explore the problem of how to learn compact descriptors for
MR-SLAM. We argue that there are two challenges. The first is how to achieve
good matching when the descriptors are extremely compact. The second is the
descriptor needs to be generated in real-time. To tackle these challenges, we
propose a teacher-student framework that leverages a compact student model to
estimate low-dimensional descriptors. As the output dimensions between the
teacher and the student are different, we propose a distance-based
distillation loss function that enables the knowledge distillation between
different dimensional descriptors. Based on that, we are able to produce
extremely compact descriptors with high performance. We fairly test our model
on the public UBC dataset. As a result, our method outperforms baseline
methods in both accuracy and efficiency. Moreover, we have developed a MR-SLAM
system based on the proposed descriptor generation method. The results on the
EuRoC dataset show that our method performs well in both accuracy and
bandwidth on MR-SLAM tasks.

## References

  * [1] P. Schmuck and M. Chli, “Multi-uav collaborative monocular slam,” in _Proc. IEEE Int. Conf. Robot. Automat._ , 2017, pp. 3863–3870.
  * [2] P. Schmuck, T. Ziegler, M. Karrer, J. Perraudin, and M. Chli, “Covins: Visual-inertial slam for centralized collaboration,” in _2021 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct)_. IEEE, 2021, pp. 171–176.
  * [3] P.-Y. Lajoie, B. Ramtoula, Y. Chang, L. Carlone, and G. Beltrame, “Door-slam: Distributed, online, and outlier resilient slam for robotic teams,” _IEEE Robotics and Automation Letters_ , vol. 5, no. 2, pp. 1656–1663, 2020\.
  * [4] X. Guo, J. Hu, J. Chen, F. Deng, and T. L. Lam, “Semantic histogram based graph matching for real-time multi-robot global localization in large scale environment,” _IEEE Robotics and Automation Letters_ , vol. 6, no. 4, pp. 8349–8356, 2021.
  * [5] D. Tardioli, E. Montijano, and A. R. Mosteo, “Visual data association in narrow-bandwidth networks,” in _2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, 2015, pp. 2572–2577.
  * [6] D. Van Opdenbosch and E. Steinbach, “Collaborative visual slam using compressed feature exchange,” _IEEE Robotics and Automation Letters_ , vol. 4, no. 1, pp. 57–64, 2018.
  * [7] Y. Tian, B. Fan, and F. Wu, “L2-Net: Deep learning of discriminative patch descriptor in euclidean space,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 661–669.
  * [8] A. Mishchuk, D. Mishkin, F. Radenovic, and J. Matas, “Working hard to know your neighbor’s margins: Local descriptor learning loss,” _Advances in Neural Information Processing Systems_ , vol. 30, 2017.
  * [9] Y. Tian, X. Yu, B. Fan, F. Wu, H. Heijnen, and V. Balntas, “Sosnet: Second order similarity regularization for local descriptor learning,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 11 016–11 025.
  * [10] Y. Tian, A. Barroso Laguna, T. Ng, V. Balntas, and K. Mikolajczyk, “Hynet: Learning local descriptor with hybrid similarity measure and triplet loss,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 7401–7412, 2020.
  * [11] G. E. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” _arXiv preprint arXiv:1503.02531_ , 2015.
  * [12] J. Hu, C. Fan, H. Jiang, X. Guo, Y. Gao, X. Lu, and T. L. Lam, “Boosting light-weight depth estimation via knowledge distillation,” _arXiv preprint arXiv:2105.06143_ , 2021.
  * [13] L. Wang and K.-J. Yoon, “Knowledge distillation and student-teacher learning for visual intelligence: A review and new outlooks,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 44, no. 6, pp. 3048–3068, 2021.
  * [14] M. Burri, J. Nikolic, P. Gohl, T. Schneider, J. Rehder, S. Omari, M. W. Achtelik, and R. Siegwart, “The euroc micro aerial vehicle datasets,” _The International Journal of Robotics Research_ , vol. 35, no. 10, pp. 1157–1163, 2016.
  * [15] M. Giamou, K. Khosoussi, and J. P. How, “Talk resource-efficiently to me: Optimal communication planning for distributed loop closure detection,” in _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, 2018, pp. 3841–3848.
  * [16] Y. Tian, K. Khosoussi, M. Giamou, J. P. How, and J. Kelly, “Near-optimal budgeted data exchange for distributed loop closure detection,” _arXiv preprint arXiv:1806.00188_ , 2018.
  * [17] J. McCormac, R. Clark, M. Bloesch, A. Davison, and S. Leutenegger, “Fusion++: Volumetric object-level slam,” in _2018 international conference on 3D vision (3DV)_. IEEE, 2018, pp. 32–41.
  * [18] A. Gawel, C. Del Don, R. Siegwart, J. Nieto, and C. Cadena, “X-view: Graph-based semantic multi-view localization,” _IEEE Robotics and Automation Letters_ , vol. 3, no. 3, pp. 1687–1694, 2018.
  * [19] D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” _International journal of computer vision_ , vol. 60, no. 2, pp. 91–110, 2004\.
  * [20] M. Calonder, V. Lepetit, M. Ozuysal, T. Trzcinski, C. Strecha, and P. Fua, “Brief: Computing a local binary descriptor very fast,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 34, no. 7, pp. 1281–1298, 2011.
  * [21] Y. Ke and R. Sukthankar, “Pca-sift: A more distinctive representation for local image descriptors,” in _Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004._ , vol. 2. IEEE, 2004, pp. II–II.
  * [22] C. Strecha, A. Bronstein, M. Bronstein, and P. Fua, “Ldahash: Improved matching with smaller descriptors,” _IEEE transactions on Pattern Analysis and Machine Intelligence_ , vol. 34, no. 1, pp. 66–78, 2011.
  * [23] T. Trzcinski, M. Christoudias, P. Fua, and V. Lepetit, “Boosting binary keypoint descriptors,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2013, pp. 2874–2881.
  * [24] X. Han, T. Leung, Y. Jia, R. Sukthankar, and A. C. Berg, “Matchnet: Unifying feature and metric learning for patch-based matching,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 3279–3286.
  * [25] K. Lin, J. Lu, C.-S. Chen, and J. Zhou, “Learning compact binary descriptors with unsupervised deep neural networks,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 1183–1192.
  * [26] Y. Duan, Z. Wang, J. Lu, X. Lin, and J. Zhou, “Graphbit: Bitwise interaction mining via deep reinforcement learning,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 8270–8279.
  * [27] M. Zieba, P. Semberecki, T. El-Gaaly, and T. Trzcinski, “Bingan: Learning compact binary descriptors with a regularized gan,” _Advances in Neural Information Processing Systems_ , vol. 31, 2018.
  * [28] J. Hu, C. Fan, M. Ozay, H. Jiang, and T. L. Lam, “Data-free dense depth distillation,” _arXiv preprint arXiv:2208.12464_ , 2022.
  * [29] J. Hu, C. Fan, M. Ozay, H. Feng, Y. Gao, and T. L. Lam, “Progressive self-distillation for ground-to-aerial perception knowledge transfer,” _arXiv preprint arXiv:2208.13404_ , 2022.
  * [30] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam, “Mobilenets: Efficient convolutional neural networks for mobile vision applications,” _arXiv preprint arXiv:1704.04861_ , 2017.
  * [31] M. Brown, G. Hua, and S. Winder, “Discriminative learning of local image descriptors,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 33, no. 1, pp. 43–57, 2010.
  * [32] V. Balntas, K. Lenc, A. Vedaldi, and K. Mikolajczyk, “Hpatches: A benchmark and evaluation of handcrafted and learned local descriptors,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2017, pp. 5173–5182.