# Conversion efficiency in Kerr microresonator optical parametric oscillators:
From three modes to many modes

Jordan R. Stone<EMAIL_ADDRESS>Joint Quantum Institute, NIST/University of
Maryland, College Park, MD 20742  National Institute for Standards and
Technology, Gaithersburg, MD 20899  Gregory Moille  Joint Quantum Institute,
NIST/University of Maryland, College Park, MD 20742  National Institute for
Standards and Technology, Gaithersburg, MD 20899  Xiyuan Lu  National
Institute for Standards and Technology, Gaithersburg, MD 20899  Institute for
Research in Electronics and Applied Physics and Maryland NanoCenter,
University of Maryland, College Park, MD 20742, USA  Kartik Srinivasan  Joint
Quantum Institute, NIST/University of Maryland, College Park, MD 20742
National Institute for Standards and Technology, Gaithersburg, MD 20899

###### Abstract

We study optical parametric oscillations in Kerr-nonlinear microresonators,
revealing an intricate solution space – parameterized by the pump-to-signal
conversion efficiency – that arises from an interplay of nonlinear processes.
Using a three-mode approximation, we derive an efficiency-maximizing relation
between pump power and frequency mismatch. To move beyond a three-mode
approximation, a necessity for geometries such as integrated microring
resonators, we numerically simulate the Lugiato-Lefever Equation that accounts
for the full spectrum of nonlinearly-coupled resonator modes. We observe and
characterize two nonlinear phenomena linked to parametric oscillations in
multi-mode resonators: Mode competition and cross phase modulation-induced
modulation instability. Both processes may impact conversion efficiency.
Finally, we show how to increase the conversion efficiency by tuning the
microresonator loss rates. Our analysis will guide microresonator designs that
aim for high conversion efficiency and output power.

##  I Introduction

Integrated photonics offers scalable options for generating, processing, and
routing optical signals within classical and quantum networks Liang and Bowers
(2010); Agrell  _et al._ (2016); Sipahigil  _et al._ (2016); Elshaari  _et
al._ (2020); Jin  _et al._ (2021). In general, optical processors apply linear
and/or nonlinear operations to light. A notable case is the optical
microresonator, whose small size and large quality factor ($Q$) work to
intensify circulating light and promote efficient nonlinear interactions
Vahala (2003); Yang  _et al._ (2018). Indeed, microresonators host a veritable
zoo of nonlinear eigenstates, including soliton frequency combs Kippenberg
_et al._ (2018), Raman frequency combs Liu  _et al._ (2018), Hz-linewidth
lasers based on stimulated Brilluoin scattering Loh  _et al._ (2015),
$\chi^{(2)}$ and $\chi^{(3)}$-type parametric oscillators Bruch  _et al._
(2019); Lu  _et al._ (2019), and more for applications in communications,
timekeeping, and sensing Spencer  _et al._ (2018); Marin-Palomo  _et al._
(2017); Lai  _et al._ (2020); Newman  _et al._ (2019).

Many of the experiments cited above were motivated by a high demand for
coherent light sources on a chip. One important type of coherent source is the
optical parametric oscillator (OPO), which is often employed to reach
wavelengths not directly accessible by conventional laser gain Vodopyanov  _et
al._ (2000); Mieth  _et al._ (2014). Optical parametric oscillations occur in
$\chi^{(3)}$-nonlinear media when vacuum fluctuations are amplified by
stimulated four wave mixing (FWM), if the FWM gain exceeds the resonator
losses Boyd (2020). Degenerately-pumped OPOs are a special case in which two
frequency-degenerate pump photons are converted into one higher-frequency
signal photon and one lower-frequency idler photon. In principle, a
degenerately-pumped OPO can generate coherent light within the frequency range
$\textrm{DC}-2\omega_{\rm{p}}$, where $\omega_{\rm{p}}$ is the pump laser
frequency Lu and Srinivasan (2021). Hence, a chip-scale, degenerately-pumped
OPO could offer superior scalability, higher efficiencies, and broader
spectral coverage than alternatives. It would be readily implemented in
miniaturized technologies, from optical clocks in which the OPO could be tuned
to clock-type or cooling-type atomic transitions, to quantum processors in
which the OPO could be tuned to qubit frequencies.

Figure 1: Introduction to the microresonator-based, degenerately-pumped
optical parametric oscillator ($\mu$OPO). (a) Schematic of a microring
resonator coupled to an access waveguide that carries the input and output
fields. (b) Energy diagram for the degenerate four wave mixing (FWM) process
that drives parametric oscillation. (c) Depictions of mode spectra, nonlinear
couplings, and frequency shifts in both a three-mode approximation (TMA) and
multi-moded model. Dashed lines correspond to zero frequency mismatch. Red
arrows indicate mode frequency shifts induced by Kerr nonlinearity. The TMA
considers FWM between only the pump, signal, and idler modes, while multi-
moded models account for the nonlinear couplings (orange, hollow arrows)
between all mode sets conducive to FWM.

Recently, several experiments have been reported that advance the
microresonator-based, degenerately-pumped OPO ($\mu$OPO) and make real headway
towards a chip-scale, wavelength-by-design light source. Achievements include
sub-milliwatt oscillation thresholds Lu  _et al._ (2019), octave-spanning and
tunable spectra Sayson  _et al._ (2019); Tang  _et al._ (2020), visible-light
generation spanning red to green Lu  _et al._ (2020); Domeneguetti  _et al._
(2021), and a $\mu$OPO that uses a 2D photonic crystal cavity Marty  _et al._
(2021). Nonetheless, the reported pump-to-signal conversion efficiencies are
typically (except in instances involving narrow spectral bandwidths) $<0.1$% –
a nonstarter for applications Lu  _et al._ (2020); Sayson  _et al._ (2019);
Domeneguetti  _et al._ (2021); Tang  _et al._ (2020). Indeed, while the demand
for efficiency calls for a deeper understanding of the underlying nonlinear
physics, experiments have so far relied on a simplified theoretical framework
for $\mu$OPOs. For instance, frequency matching is considered in either a
cold-cavity limit Lu  _et al._ (2019), or otherwise only accounts for a
populated pump mode Sayson  _et al._ (2019). A more accurate description of
frequency matching should account for the exact distribution of intraresonator
photons. Moreover, analyses have relied on a three-mode approximation (TMA),
in which only the pump, signal, and idler modes interact through Kerr
nonlinearity Sayson  _et al._ (2019); Hansson  _et al._ (2013). Of course,
real microresonators comprise a more complex spectrum of modes that are
nonlinearly coupled together. As a result, there is presently a gap between
theoretical and experimental progress revolving around $\mu$OPOs; ultimately,
there is little theoretical basis on which to design microresonators to meet
end-user demands.

Here, we construct a generalized $\mu$OPO solution space; thereby, we reveal
connections between the $\mu$OPO state and experimental parameters, and we
identify processes that limit conversion efficiency. We adopt a model based on
the Lugiato-Lefever Equation (LLE) Coen  _et al._ (2013); Chembo and Menyuk
(2013) and support our main numerical results with theoretical analyses. In
the next section, we explain our modeling and present simulation results using
a TMA. Then, we expand the model to include a spectrum of nonlinearly-coupled
resonator modes. We demonstrate two nonlinear phenomena that cannot be
explained within a TMA. In the first, a mode competition takes place between
multiple signal and idler mode pairs. In the second, modulation instability
induced by cross-phase modulation constrains the $\mu$OPO conversion
efficiency. Finally, we propose two strategies for increasing the $\mu$OPO
conversion efficiency and output power. Surprisingly, when comparing
microresonators with different loss rates but identical geometries, we find
that the resonator with greater losses will, in some cases, promote higher
efficiency.

##  II Modeling the $\mu$OPO: The Lugiato-Lefever Equation, three-mode
approximation, and dispersion

To study $\mu$OPOs, we consider a microring resonator coupled to an optical
access waveguide and pumped by a continuous-wave (CW) laser, as depicted in
Fig. 1a. This structure supports whispering gallery modes, in which azimuthal
modes are grouped into families sharing a transverse spatial mode profile.
Modes within a family are spaced (in the frequency domain) by a free-spectral
range (FSR) that is inversely proportional to the ring circumference, $L$. In
our model, we consider a single mode family and denote its resonant
frequencies as $\omega_{\rm{\mu}}$, where $\mu$ is the azimuthal mode number
shifted to make $\omega_{\rm{0}}$ the frequency of the pumped mode. The pump
laser has frequency $\omega_{\rm{p}}$ and waveguide power $P_{\rm{in}}$, and
the intraresonator field, $a$, obeys the Lugiato-Lefever Equation (LLE) Chembo
and Menyuk (2013):

${}\frac{da}{dt}=\sqrt{\frac{\kappa_{\rm{c}}(0)}{\hbar\omega_{\rm{p}}}P_{\rm{in}}}-\left(\frac{\kappa_{i}}{2}+i\frac{\kappa(0)}{2}\alpha-
ig_{0}|a|^{2}\right)a-i\mathcal{D}(\mu)\tilde{a},$ (1)

where $|a|^{2}$ gives the intraresonator energy in units of photon number,
$\kappa_{\rm{c}}(\mu)$ is the mode-dependent coupling rate to the access
waveguide, $\kappa_{\rm{i}}$ is the mode-independent intrinsic loss rate,
$\kappa(\mu)=\kappa_{\rm{i}}+\kappa_{\rm{c}}(\mu)$ is the mode-dependent total
loss rate, $\alpha=\frac{\omega_{\rm{0}}-\omega_{\rm{p}}}{\kappa(0)/2}$ is the
normalized pump-resonator frequency detuning,
$g_{\rm{0}}=\frac{n_{\rm{2}}c\hbar\omega_{\rm{0}}^{2}}{n^{2}V}$ is the
nonlinear gain per photon, $n_{\rm{2}}$ is the Kerr index, $c$ is the speed of
light in vacuum, $n$ is the refractive index, $V$ is the mode volume, and
$\mathcal{D}(\mu)=\omega_{\rm{\mu}}-(\omega_{\rm{0}}+\mu
D_{\rm{1}})+i\kappa_{\rm{c}}(\mu)$, where $D_{\rm{1}}=2\pi\times\textrm{FSR}$;
$\tilde{a}$ indicates that operations to $a$ are performed in the frequency
domain. Notably, the integrated dispersion, $D_{\rm{int}}$, is contained in
Eq. 1 as $D_{\rm{int}}=Re(\mathcal{D})/\kappa(0)$. For concreteness, we use
$n_{\rm{2}}=2.4\times 10^{-19}$ m2/W and $n=1.9$, which are typical values for
silicon nitride (SiN) microrings, and $\omega_{\rm{p}}\approx 2\pi\times 384$
THz ($\approx$ 780 nm wavelength). Unless otherwise stated, we use
$\kappa_{\rm{i}}=2\pi\times 200$ MHz and $\kappa_{\rm{c}}=\kappa_{\rm{i}}$
(i.e., critical coupling).

Figure 2: Study of the $\mu$OPO conversion efficiency using a three-mode
approximation. (a) Conversion efficiency, $CE$ (gray), and effective mismatch,
$\Delta_{s}$ (green), versus the pump-resonator frequency detuning, $\alpha$,
for a pump laser power $P_{\rm{in}}=20~{}\rm{mW}$. From top to bottom, the
mismatch values are $\delta=1.25,2,4$, and $7$, respectively. Dashed sections
indicate unstable or oscillatory solutions. (b) A generalized $CE$ map showing
the relationship between maximum $CE$, normalized pump power, $X$, and
$\delta$. The white dashed line is expressed as $X=8\delta$ and closely
follows the contour of highest maximum $CE$. (c) $CE$ (gray) and effective
pump-resonator detuning, $\Delta_{\rm{p}}$ (green), versus $\alpha$. (d)
$\Delta_{\rm{p}}$ versus $\delta$ at values of $\alpha$ that maximize $CE$ for
$P_{\rm{in}}=15$ mW (red, $X\approx 5$) and $P_{\rm{in}}=25$ mW (orange,
$X\approx 8$).

Figure 1c introduces some key concepts and illustrates differences between the
TMA and the multi-mode LLE (here, ’multi-mode’ refers to the inclusion of many
longitudinal modes from the same spatial mode family). The main difference is
that a multi-mode model accounts for many more modes that can be coupled
together by the Kerr nonlinearity. In contrast, a TMA only considers nonlinear
interactions (e.g., FWM) between the pump, signal, and idler modes.
Importantly, both models account for imperfect frequency matching. In general,
phase-matched mode pairs (i.e., with azimuthal numbers $\pm\mu$) are not
frequency matched – the associated FWM process does not conserve energy.
Imperfect frequency matching is often quantified by the frequency mismatch
parameter

${}\delta_{\rm{\mu}}=\frac{\omega_{\mu}+\omega_{-\mu}-2\omega_{\rm{0}}}{\kappa(0)}.$
(2)

Throughout, we use $\delta_{\rm{\mu}}$ to refer to the dispersive mismatch
spectrum and $\delta$ to refer to the value of $\delta_{\rm{\mu(-\mu)}}$ at
the targeted signal (idler) mode. A degenerate FWM process only conserves
energy and momentum if $\delta$ is compensated by nonlinear frequency shifts
of $\omega_{\rm{p(s,i)}}$, which we denote as $N_{\rm{p(s,i)}}$ for the pump
(signal, idler) mode. Nonlinear shifts arise from self- and cross-phase
modulation (SPM and XPM, respectively) and are related to the intraresonator
intensity spectrum. Indeed, an occupied resonator necessarily implies
$N_{p(s,i)}<0$; therefore, $\delta=0$ is not conducive to parametric
oscillation.

Using a TMA, we investigate how changes in $\delta$ impact the $\mu$OPO, and
then we introduce other modes into our simulations. To ensure the consistency
of our numerical methods, we use the split-step Fourier method to simulate Eq.
1 for both the TMA and multi-mode LLE. Our primary goal is to understand the
efficiency with which pump photons are converted into signal or idler photons.
Accordingly, we define the conversion efficiency in units of photon flux as

${}CE=\frac{\omega_{\rm{p}}}{\omega_{\rm{s(i)}}}\frac{P_{\rm{s(i)}}}{P_{\rm{in}}},$
(3)

where $\omega_{\rm{s(i)}}$ is the signal (idler) frequency and $P_{\rm{s(i)}}$
is the signal (idler) output power in the access waveguide. Importantly, the
underlying symmetry of the FWM process implies that the signal and idler
fields have equal $CE$ values and that $N_{\rm{s}}=N_{\rm{i}}$ for a
critically-coupled resonator. Throughout, we present $CE$ for the signal; the
same results apply to the idler.

In Fig. 2, we present simulation results obtained using a TMA. In a single
simulation, $P_{\rm{in}}$ and $\delta$ are held fixed; we vary $\alpha$ to
simulate a $\omega_{\rm{p}}$ scan across resonance from blue to red detuning,
as shown in Fig. 2a. During the simulation, we record various data, including
the $CE$, $N_{\rm{p(s,i)}}$, and optical spectra (see Supplemental Material
for details). In Fig. 2a, each panel corresponds to $P_{\rm{in}}=20$ mW, but
$\delta$ is increased from top to bottom. Our results indicate a highest
obtainable $CE$ of $12.5$ % for critically-coupled resonators, in agreement
with Ref. Sayson  _et al._ (2019). Additionally, each panel depicts the
effective signal detuning, defined as
$\Delta_{\rm{s}}=\delta+\frac{\alpha}{2}-N_{\rm{s}}$. Notably, when $CE>0$,
$\Delta_{\rm{s}}\approx 0$, which indicates that dispersion is perfectly
compensated by detuning and nonlinearity in the $\mu$OPO state.

To fully characterize the relationship between $CE$, $P_{\rm{in}}$, and
$\delta$, we construct the universal $CE$ map shown in Fig. 2b. The parameter
space is defined by $P_{\rm{in}}$ and $\delta$; in Fig. 2b, we normalize
$P_{\rm{in}}$ as

${}X=\frac{P_{\rm{in}}}{P_{\rm{th}}},$ (4)

where
$P_{\rm{th}}=\frac{\hbar\omega_{\rm{0}}\kappa^{3}(0)}{8g_{\rm{0}}\kappa_{\rm{c}}(0)}$
is the oscillation threshold power Kippenberg  _et al._ (2004). Each data
pixel in the $CE$ map represents the maximum $CE$ value taken from the
corresponding LLE simulation, as indicated by the dashed line connecting Figs.
2a and 2b. The $CE$ map has a few notable features. First, we do not observe
parametric oscillation for any values of $P_{\rm{in}}$ when $\delta\leq 0$.
This indicates that nonlinear frequency shifts of $\omega_{\rm{\mu}}$ inhibit
frequency matching even when $X\approx 1$. Second, $CE$ contours follow clear
trends through the parameter space. In particular, to maintain $CE$ at larger
values of $X$, $\delta$ must be increased, apparently to compensate for larger
$N_{s(p,i)}$. Remarkably, we can derive an analytical expression for the
contour of highest $CE$. We provide the derivation in the Supplemental
Material; here, we present the final result, $X=8\delta$, and indicate it with
the white dashed line in Fig. 2b. Clearly, to maximize $CE$ for a given
$P_{\rm{in}}$, the microresonator dispersion (i.e., $\delta$) should be
designed appropriately.

Figure 3: Depictions of microresonator dispersion. (a) $\delta_{\rm{\mu}}$
(green circles), and integrated dispersion, $D_{\rm{int}}$ (purple stripe),
for the TE1 mode family calculated using finite-element method eigenfrequency
simulations. The microresonator has ring radius $RR=15$ $\mu$m, ring width
$RW=800$ nm, height $H=600$ nm, and pump frequency $\omega_{\rm{p}}=2\pi\times
388$ THz. Inset: Illustration of the test device cross section. (b) Group
velocity refractive index, $n_{\rm{g}}$, versus $\omega/2\pi$ for the same
device as in (a). The dashed lines illustrate how a turning point in
$\delta_{\rm{\mu}}$ occurs when $n_{\rm{g}}(-\mu)=n_{\rm{g}}(\mu)$.

In Fig. 2a, it is perhaps remarkable that $CE$ values tend to increase
monotonically until an abrupt cutoff. To elucidate this observation, we
monitor the effective pump detuning,
$\Delta_{\rm{p}}=\frac{\alpha}{2}-N_{\rm{p}}$, during our simulations and
present a sample result in Fig. 2c. When $CE$ is greatest, $\Delta_{\rm{p}}$
reaches a minimum value near zero. Physically, this condition implies that
$\omega_{\rm{p}}$ is nearly resonant; intuitively, this is necessary to
maximize the dropped power and, in turn, the nonlinear gain. The cutoff
results from Kerr bistability; i.e., when further increases in $\alpha$ are no
longer compensated by $N_{\rm{p}}$, the intraresonator field abruptly
transitions to the CW state. To characterize the universality of this feature,
we record $\Delta_{\rm{p}}$ (evaluated for the highest $CE$) versus $\delta$
for two different powers, as shown in Fig. 2d. Apparently, realizing high $CE$
requires $\Delta_{\rm{p}}\approx 0$.

As a first step towards transitioning from a TMA to a multi-mode LLE, we
briefly discuss the microresonator dispersion and its different
representations. In Fig. 3a, we present $\delta_{\rm{\mu}}$ and $D_{\rm{int}}$
spectra for the fundamental transverse electric (TE) mode family, labeled TE1,
of a SiN microring resonator (hereafter referred to as our test device) with
ring radius $RR=15$ $\mu$m, ring width $RW=800$ nm, and height $H=600$ nm. We
extract $\delta_{\rm{\mu}}$ and $D_{\rm{int}}$ from the mode spectrum,
$\omega_{\rm{\mu}}$, that we calculate from finite-element method
eigenfrequency simulations 111Finite element simulations performed using
Comsol Multiphysics, which is identified here to foster understanding, without
implying recommendation or endorsement by NIST.. It is easy to show that
$\delta_{\rm{\mu}}=2\times\mathcal{S}(D_{\rm{int}}(\mu))$, where
$\mathcal{S}(D_{\rm{int}}(\mu))$ indicates the symmetric part of
$D_{\rm{int}}$; i.e., the even orders of $D_{\rm{int}}$ when expressed as a
Taylor expansion around $\mu=0$. We have assessed that analyzing
$\delta_{\rm{\mu}}$ is sufficient to understand the $\mu$OPO dynamics, which
are not sensitive to odd orders of $D_{\rm{int}}$ Sayson  _et al._ (2019).
This is a notable departure from other nonlinear microresonator eigenstates,
e.g., dissipative Kerr solitons.

There are two defining features of the $\delta_{\rm{\mu}}$ spectrum shown in
Fig. 3a. First, its negative curvature around $\mu=0$ indicates normal
dispersion, which is required to suppress comb formation. Second, to overcome
the normal dispersion and achieve frequency matching, the $D_{\rm{int}}$
expansion must contain higher order (even) terms such that $\delta_{\rm{\mu}}$
turns and becomes positive. Physically, this means that an anomalous-to-normal
dispersion transition is necessary. Indeed, the dashed lines connecting Figs.
3a and 3b illustrate how $n_{\rm{g}}(\mu)=n_{\rm{g}}(-\mu)$ corresponds to a
$\delta_{\rm{\mu}}$ turning point, where $n_{\rm{g}}(\mu)$ is the dispersive
group velocity refractive index.

Figure 4: Transition from a TMA to multi-mode LLE simulations. (a) Depiction
of the four wave mixing (FWM) processes studied in this work. Mode competition
(orange arrows) suppresses FWM to the target signal and idler mode pair (with
mode numbers $\pm\mu$) in favor of spectrally-adjacent mode pairs (with mode
numbers $\pm(\mu\pm 1)$), and modulation instability induced by cross-phase
modulation (XPM-MI, purple arrows) reroutes energy in the pump mode to its
spectrally-nearest neighbors. (b) $CE$ map for mode $\mu=46$; the resonator
parameters are $RR=15\mu$m, $RW=800$ nm, $H=600$ nm, and
$\omega_{\rm{p}}=2\pi\times 382$ THz. (c) Difference in $CE$ between
simulations based on a TMA and the $CE$ map in part (b).

Next, we preview the differences between $CE$ maps calculated from multi-mode
and TMA models. We observe two FWM processes, depicted in Fig. 4a, that
require the multi-mode model. In one case, we observe mode competition between
mode pairs with consecutive $\mu$ values. In the second case, we observe
modulation instability (MI) in the normally-dispersive spectral region around
$\omega_{\rm{p}}$, and we explain it as arising from XPM between the pump,
signal, and idler fields. We mostly constrain our multi-mode LLE simulations
to $P_{\rm{in}}\leq 25$ mW ($X\leq 8$); at higher values of $P_{\rm{in}}$, the
$\mu$OPO dynamics become more complex.

Figure 5: Mode competition and switching. (a) Illustration of mode competition
in a $\mu$OPO. Oscillations on the target signal and idler mode pair (with
mode numbers $\pm\mu$) are suppressed in favor of spectrally-adjacent mode
pairs (with mode numbers $\pm(\mu\pm 1)$). (b) $\delta_{\rm{\mu}}$ for the
test device with $\omega_{\rm{p}}=2\pi\times 382$ THz. The pale orange stripe
overlaps modes that are predicted to oscillate when $P_{\rm{in}}=20$ mW. (c)
$CE$ maps for modes $\mu=46$, $\mu=45$, and $\mu=44$. The maps indicate a mode
competition in which the mode pair with smallest $\delta$ is favored for
oscillation. (d) $CE$ versus $\alpha$ for modes $\mu=46$ (blue) and $\mu=45$
(green), where $P_{\rm{in}}=20$ mW and $\beta=5\times 10^{-4}$. The pale
stripes come from a TMA. (e) Optical spectra that correspond to different
values of $\alpha$ in (d). The blue, orange, green, and red spectra correspond
to $\alpha=1,1.5,2.5$, and $3.5$, respectively.

Figure 4b shows a $CE$ map for our test device pumped near
$\omega_{\rm{p}}=2\pi\times 382$ THz. To make a straightforward comparison
between these data and Fig. 2b, we calculate the difference in $CE$ between
the two $CE$ maps, as shown in Fig. 4c. In general, XPM-induced MI (XPM-MI)
explains $CE$ differences for small $\delta$, while mode competition is
responsible for the abrupt cutoff in $CE$ (marked by the sharp transition to
zero $CE$ in Fig. 4b or the bold yellow stripe in Fig. 4c) that indicates a
different mode pair is oscillating. In the following sections, we explore mode
competition and XPM-MI in detail.

##  III Mode competition and switching

Mode competition is ubiquitous in laser systems with multi-mode resonators
Narducci  _et al._ (1986); Gong  _et al._ (2007). In general, mode competition
occurs when several resonator modes experience amplification simultaneously;
hence, modes compete for gain and become coupled. In this section, we present
the results of multi-mode LLE simulations in which several mode pairs are
simultaneously nearly frequency matched. We use our findings to establish
general principles for mode competition that will inform future microresonator
designs.

In Fig. 5, we present simulation results for our test device pumped near
$\omega_{\rm{p}}=2\pi\times 382$ THz. The FWM process related to mode
competition is illustrated in Fig. 5a. Modes that are spectrally adjacent to
the targeted signal and idler pair may be nearly frequency matched; hence,
light in these modes may become amplified through FWM. To explore this
phenomenon, we consider the $\delta_{\rm{\mu}}$ spectrum shown in Fig. 5b.
According to the TMA, any mode pairs with $\delta_{\rm{\mu}}>0$ may oscillate,
provided $P_{\rm{in}}$ is large enough. In Fig. 5b, the data points covered by
the pale gold stripe indicate mode pairs that would oscillate when
$P_{\rm{in}}=20$ mW, if they (along with the pump mode) were the only modes in
the system (i.e., in a TMA). Hence, we endeavour to understand how a mode pair
(or pairs) is chosen for oscillation over its spectral neighbors. To study
mode competition in our test device, we perform multi-mode LLE simulations,
from which we construct $CE$ maps for the modes corresponding to $\mu=46$,
$\mu=45$, and $\mu=44$, as shown in Fig. 5c. To vary $\delta_{\rm{\mu}}$
(which is fixed for a given geometry), we apply a quadratic dispersion to
$\omega_{\rm{\mu}}$, such that
$\omega_{\rm{\mu}}\rightarrow\omega_{\rm{\mu}}+\beta\kappa(0)\mu^{2}$, where
$\beta$ quantifies the added dispersion. We choose this approach in order to
maintain the overall shape of $\delta_{\rm{\mu}}$. Note that
$\delta_{\rm{45}}$ and $\delta_{\rm{44}}$ are both negative in Fig. 5b, but
they will become positive as $\beta$ is increased.

Figure 6: Characterization of XPM-MI in microresonators with normal
dispersion. (a) Illustration of the XPM-MI FWM process (purple arrows), in
which XPM between the pump, signal, and idler fields induces MI that
distributes pump energy into spectrally-nearby sidebands. (b)
$\delta_{\rm{\mu}}$ for $\beta=0.0625$ (green circles), $\beta=1.25$ (cyan
upward-pointing triangles), and $\beta=5$ (blue downward-pointing triangles).
One mode pair ($\mu=\pm 50$) is frequency shifted to be frequency matched to
the pump mode. (c) $CE$ versus $\alpha$ for the $\delta_{\rm{\mu}}$ spectra in
(b), with $P_{\rm{in}}=20$ mW and $\delta=3$. (d) Optical spectrum associated
with part (c), with $\alpha=4$ and $\beta=0.0625$. (e) $CE$ maps for
microresonators with the $\delta_{\rm{\mu}}$ spectra in (b), ordered from
least to greatest $\beta$.

The $CE$ maps in Fig. 5c present clear evidence for mode competition, and we
identify three notable features in them. First, the $CE$ map for mode $\mu=46$
resembles that of the TMA for small $\delta_{\rm{46}}$. Second, as $\beta$ is
increased, the maximum $CE$ for mode $\mu=46$ declines sharply, and this
coincides with $\delta_{\rm{45}}>0$ and oscillation on mode $\mu=45$. This
pattern repeats as $\beta$ is increased further – the oscillating mode changes
from $\mu=45$ to $\mu=44$, and so on. Hence, we assess that the mode pair with
smallest positive $\delta$ is favored for oscillation. Finally, there are
small regions of parameter space where multiple mode pairs seem to oscillate
simultaneously – we explore this phenomenon in Figs. 5d and 5e.

Figure 5d shows $CE$ for modes $\mu=46$ and $\mu=45$ during a
$\omega_{\rm{p}}$ scan (i.e., varying $\alpha$) with $\beta=5\times 10^{-4}$
($\delta_{\rm{46}}\approx 3$) and $P_{\rm{in}}=20$ mW. During the scan, mode
$\mu=46$ begins to oscillate first; its $CE$ closely follows predictions made
using a TMA (pale blue stripe) and reaches a maximum when $\alpha\approx 1.3$.
In this regime, the $\mu$OPO spectrum is tri-modal, as depicted by the blue
spectrum in Fig. 5e (the spectral bandwidth in Fig. 5e only spans
$\omega_{\rm{p}}$ and $\omega_{\rm{s}}$; we have confirmed the spectrum is
symmetric around $\omega_{\rm{p}}$). Beyond $\alpha\approx 1.3$, mode $\mu=45$
begins to oscillate, and $CE$ for mode $\mu=46$ deviates from its TMA
counterpart. Between $\alpha\approx 1.3$ and $\alpha\approx 2.2$, both modes
oscillate simultaneously, but their respective $CE$ values are not predicted
by a TMA. Moreover, in this region the $\mu$OPO spectrum is not tri-modal;
rather, it is strongly multi-moded with intensity peaks occurring near
$\omega_{\rm{p}}$, $\omega_{\rm{s}}$, and $\omega_{\rm{i}}$, as shown by the
orange spectrum in Fig. 5e. Beyond $\alpha\approx 2.2$, $CE$ for mode $\mu=46$
is zero, and $CE$ for mode $\mu=45$ follows its TMA counterpart. We term this
phenomenon, wherein $CE$ values for different mode pairs conform to their TMA
counterparts for different portions of a $\omega_{\rm{p}}$ scan, mode
switching. Finally, at $\alpha\geq 3$, the $\mu$OPO decays in favor of MI.
Remarkably, the MI state is supported in the normally-dispersive region around
$\mu=0$ and without strong signal or idler fields. The MI spectrum is shown by
the shaded red curve in Fig. 5e. In the next section, we explore MI and its
impact on $CE$.

##  IV XPM-MI: Characterization and Theory

It is clear from Fig. 5 that mode competition is not the only process that
differentiates the multi-mode LLE and TMA. Specifically, Figs. 5d and 5e
establish that MI can suppress or extinguish the signal and idler fields, and
this occurs in regions of parameter space where a TMA predicts efficient
parametric oscillation. In this section, we isolate the MI process in our
simulations by considering a special dispersion profile that eliminates mode
competition, and we develop a theory for MI as arising from XPM between the
pump, signal, and idler fields. We term this process XPM-MI to link it to
previous investigations Agrawal (1987).

The XPM-MI FWM process is illustrated by the purple arrows in Fig. 6a. Energy
from the pump laser is distributed to sidebands in a narrow spectral bandwidth
around $\omega_{\rm{p}}$. If we use a multi-mode LLE to simulate our test
device, the effects of mode competition can obfuscate the impact of XPM-MI.
Therefore, we contrive the heuristically useful $\delta_{\rm{\mu}}$ spectra
depicted in Fig. 6b. Here, we define $\delta_{\rm{\mu}}=-2\beta\mu^{2}$ for
all $\mu\neq\pm 50$. The modes $\mu=\pm 50$ are designated for parametric
oscillation and assigned a frequency mismatch value $\delta$ that can be
manipulated apart from $\beta$.

Figure 7: Theory of XPM-MI. (a) MI gain parameter, $\lambda$, versus $\alpha$
for $\mu=1$. Orange and gold curves correspond to $P_{\rm{in}}=15$ mW,
$\delta=1.5$, $\beta=0.25$, and $P_{\rm{in}}=25$ mW, $\delta=2.5$,
$\beta=0.25$, respectively. The red curves all use $P_{\rm{in}}=20$ mW, with
bold, dotted, and dashed curves corresponding to $\delta=2.5$ and $\beta=1$,
$\delta=5$ and $\beta=0.25$, and $\delta=2$ and $\beta=0.25$, respectively.
(b) The value of $\alpha$ at which MI first appears, $\alpha_{\rm{ON}}$,
versus $\beta$ for $P_{\rm{in}}=20$ mW and $\delta=2.5$. Light blue circles
indicate theoretical values using Eq. 5, while dark blue diamonds mark values
predicted by LLE simulations. (c) $\lambda$ map for $\mu=1$ and $\beta=0.25$.

In Fig. 6c, we present LLE simulations of $CE$ versus $\alpha$ for
microresonators with the $\delta_{\rm{\mu}}$ spectra in Fig. 6b; these data
overlay the corresponding simulation using a TMA (pale gray stripe). They are
representative of the entire ($P_{\rm{in}},\delta)$ parameter space, and they
are noteworthy for two reasons. First, in all cases $CE$ follows the gray
stripe until $\alpha\geq\alpha_{\rm{ON}}$, where $\alpha_{\rm{ON}}$
corresponds to the onset of XPM-MI. An example XPM-MI spectrum is shown in
Fig. 6d. Unlike the XPM-MI spectrum shown in Fig. 5e, here the XPM-MI and
$\mu$OPO states co-exist, albeit with suppressed $CE$. Second,
$\alpha_{\rm{ON}}$ increases with increasing $\beta$. As a result, $CE$ fully
converges to its TMA counterpart in the high-$\beta$ limit. To investigate
this convergence, we construct the $CE$ maps for different values of $\beta$,
and we present the results in Fig. 6e. We observe steady convergence to the
TMA $CE$ map as $\beta$ is increased. Clearly, it is crucial to realize strong
normal dispersion near $\omega_{\rm{p}}$ to observe efficient parametric
oscillation when $X>>1$.

Next, we present an expression for the XPM-MI gain that we derive from a set
of coupled mode equations (see Supplemental Material for details), and we
analyze this expression to support our conclusion that XPM underlies the
observed MI states. As noted in Ref. Hansson  _et al._ (2013), MI may occur in
normally-dispersive microresonators; however, it usually requires a hard
excitation, i.e., the MI sidebands are not amplified from vacuum fluctuations.
This fact explains how the MI state can sometimes persist after parametric
oscillations decay. Still, in our simulations we have never observed MI emerge
before parametric oscillations. Indeed, it was predicted in Ref. Agrawal
(1987) that XPM between two waves can make them modulationally unstable, even
when one or both waves propagates in normally-dispersive media. Motivated by
this study, we analyze a set of coupled mode equations that assume the pump,
signal, and idler modes are occupied, and we derive the MI gain,
$\lambda_{\rm{\mu}}$, as

${}\lambda_{\rm{\mu}}=-1+\sqrt{I_{\rm{0}}^{2}+I_{\rm{s}}^{2}+2I_{\rm{0}}I_{\rm{s}}\rm{cos}(\Delta\phi)-\textit{k}_{\rm{\mu}}^{2}},$
(5)

where $I_{\rm{0}}$ and $I_{\rm{s}}$ are the photon numbers for the pump and
signal modes, respectively,
$\Delta\phi=2\phi_{\rm{p}}-\phi_{\rm{s}}-\phi_{\rm{i}}$ is the relative phase
mismatch between the pump, signal, and idler fields, which are expressed as
$\tilde{a}_{\rm{p(s,i)}}=\sqrt{I_{\rm{p(s,i)}}}e^{-i\phi_{\rm{p(s,i)}}}$, and
$k_{\rm{\mu}}=2I_{\rm{0}}+4I_{\rm{s}}-\beta\mu^{2}-\alpha$. MI occurs when
$\lambda_{\rm{\mu}}>0$, for any $\mu$. Note from these definitions that
$\beta\mu^{2}$ is the frequency mismatch parameter for the MI sidebands, with
mode numbers $\pm\mu$, and not the frequency mismatch for the $\mu$OPO mode
pair, which we denote as $\delta$. In the limit $I_{\rm{s}}\rightarrow 0$, Eq.
5 is equivalent to the MI gain expression derived in Ref. Hansson  _et al._
(2013). Moreover, it is clear that $\lambda_{\rm{\mu}}$ is maximized when the
pump, signal, and idler fields are in phase (i.e. when $\Delta\phi=0$), which
corresponds to the in-phase addition of two FWM processes: the degenerate FWM
process $2\omega_{\rm{p}}\rightarrow\omega_{\rm{\mu}}+\omega_{\rm{-\mu}}$ and
the non-degenerate process
$\omega_{\rm{s}}+\omega_{\rm{i}}\rightarrow\omega_{\rm{\mu}}+\omega_{\rm{-\mu}}$.

To analyze XPM-MI using Eq. 5, we perform LLE simulations, using a TMA, for
various values of $P_{\rm{in}}$ and $\delta$ and extract values for
$I_{\rm{0}},I_{\rm{s}}$, and $\Delta\phi$ that we use to calculate
$\lambda_{\rm{\mu}}$. When $\lambda_{\rm{\mu}}>0$, the mode pair with mode
numbers $\pm\mu$ will be amplified and steal energy from the $\mu$OPO. In
general, a large normal dispersion leads to the sideband pair with $\mu=\pm 1$
having the largest gain; therefore, in what follows we drop the $\mu$
subscript and define $\lambda$ as the MI gain for this pair. We have confirmed
that $\lambda_{\rm{\mu}}<0$ whenever $I_{\rm{s}}=0$ and $\beta>0$; physically,
this means that XPM between the pump, sigal, and idler fields is required to
initiate MI for microresonators with normal dispersion. Hence, we term this
process XPM-MI. Figure 7a presents $\lambda$ calculations for various values
of $P_{\rm{in}}$, $\delta$, and $\beta$. They indicate that $\lambda$ grows
with increasing $\alpha$, which is explained by the stronger XPM that results
from more powerful $\mu$OPO sidebands at large $\alpha$. To compare our XPM-MI
theory (i.e., Eq. 5) to multi-mode LLE simulations, we calculate
$\alpha_{\rm{ON}}$ in both cases, for different values of $\beta$. Overall, we
observe good agreement, but Eq. 5 generally predicts higher values of
$\alpha_{\rm{ON}}$ than the multi-mode LLE. A sample comparison of this type
is presented in Fig. 7b.

Figure 8: Strategies to achieve high $CE$ and increase power output from a
$\mu$OPO. (a) Depiction of a microring resonator and its loss rates. (b)
Simulated mode transmission vs $\omega_{\rm{p}}$ for
$\kappa_{\rm{i}}=2\pi\times 125$ MHz (red), $\kappa_{\rm{i}}=2\pi\times 200$
MHz (orange), and $\kappa_{\rm{i}}=2\pi\times 275$ MHz (gold). In each case,
$\kappa_{\rm{c}}=\kappa_{\rm{i}}$. (c) $CE$ maps corresponding to (b), for a
resonator with $RR=15~{}\mu$m, $RW=800$ nm, $H=600$ nm, and
$\omega_{\rm{p}}=2\pi\times 385$ THz. (d) Simulated mode transmissions for
$\kappa_{\rm{c}}=\kappa_{\rm{i}}$ (green) and
$\kappa_{\rm{c}}=10\times\kappa_{\rm{i}}$ (blue), where
$\kappa_{\rm{i}}=2\pi\times 200$ MHz. (e) Maximum $CE$ versus
$\kappa_{\rm{c}}/\kappa_{\rm{i}}$ and $\delta$, where $\kappa_{\rm{c}}$ is the
signal mode coupling rate. The pump and idler modes are critically coupled.

To more comprehensively compare our theory with multi-mode LLE simulations, we
calculate the maximum gain, $\lambda_{\rm{max}}$, for $\beta=0.25$ and
different values of $P_{\rm{in}}$ and $\delta$. We use these data to construct
the $\lambda_{\rm{max}}$ map presented in Fig. 7c. It accurately predicts the
threshold $P_{\rm{in}}$ values for XPM-MI, and it has the overall trend that
$\lambda_{\rm{max}}$ grows with increasing $P_{\rm{in}}$ and $\delta$. This
trend is consistent with the $CE$ maps presented in Fig. 6e, which deviate
from the TMA as $P_{\rm{in}}$ and $\delta$ are increased. Importantly, our
observations may explain prior experimental results, which consistently report
that increasing $P_{\rm{in}}$ leads to the formation of undesired sidebands Lu
_et al._ (2019); Tang  _et al._ (2020); Fujii  _et al._ (2019). Overall, our
theory supports the hypothesis that XPM drives MI in a $\mu$OPO. Moreover, we
can now form a concise description of parasitic FWM in $\mu$OPOs: Mode
competition dictates which mode pair oscillates, while XPM-MI constrains $CE$.

##  V Towards efficient and high-power $\mu$OPO

Our results suggest several design rules for avoiding mode competition and
mitigating XPM-MI through dispersion engineering. For example, Fig. 6
demonstrates that larger normal dispersion around $\omega_{\rm{p}}$ will
suppress XPM-MI. Moreover, increasing $\frac{d\delta_{\rm{\mu}}}{d\mu}$ will
help prevent mode competitions. This can be accomplished by using smaller
resonators or in resonators with greater dispersion. Still, in practice it is
nontrivial to fabricate devices with ideal dispersion characteristics.
Therefore, in this section we describe two strategies to increase $CE$ by
tuning the microresonator loss rates, $\kappa_{\rm{c}}(\mu)$ and
$\kappa_{\rm{i}}$. We assume that users of real-world $\mu$OPO devices will
value high output power; therefore, we focus on ways to increase $CE$ for
fixed $P_{\rm{in}}$.

Figure 8a depicts the coupling and intrinsic loss rates that are quantified by
$\kappa_{\rm{c}}(\mu)$ and $\kappa_{\rm{i}}$, respectively. In practice,
$\kappa_{\rm{c}}(\mu)$ is controllable by various design parameters, including
the waveguide-resonator separation and waveguide-resonator coupling length,
for example, in a ‘pulley’ configuration Moille  _et al._ (2019). Moreover,
$\kappa_{\rm{i}}$ can be reduced within some spectral bands by annealing
Spencer  _et al._ (2014); Graziosi  _et al._ (2018); hence, one gains some
control over $\kappa_{\rm{i}}$ by making a suitable choice for the annealing
time or temperature (also, increasing $\kappa_{\rm{i}}$ can be realized
through, for example, intentionally-introduced surface roughness).

We consider two ways to increase $CE$. First, by increasing $P_{\rm{th}}$, one
increases the $P_{\rm{in}}$ values for which XPM-MI occurs. To demonstrate
this approach, we perform multi-mode LLE simulations of our test device for
three different values of $\kappa$. In every simulation,
$\kappa_{\rm{c}}(\mu)=\kappa_{\rm{i}}$ and $\omega_{\rm{p}}=2\pi\times 385$
THz. The simulated modal lineshapes for these resonators are shown in Fig. 8b,
and the corresponding $CE$ maps are shown in Fig. 8c. In Fig. 8c, $\kappa$ is
increased from the left-most panel to the right-most panel. As $\kappa$ is
increased, the region of highest $CE$ is shifted towards higher values of
$P_{\rm{in}}$. Moreover, this region is broadened along both axes, indicating
that devices with larger $\kappa$ will have a greater tolerance for design
errors in $\delta$. While increases in $\kappa$ prevent high $CE$ at low
$P_{\rm{in}}$, the obtainable output power generated with high $P_{\rm{in}}$
has grown; for instance, with $P_{\rm{in}}=30$ mW, the maximum output power
increases from $P_{\rm{sig}}\approx 2.25$ mW at $\kappa/2\pi=250$ MHz (Fig.
8c, left-most panel) to $P_{\rm{sig}}\approx 3.75$ mW at $\kappa/2\pi=550$ MHz
(Fig. 8c, right-most panel).

Finally, we consider the relationship between the signal mode coupling rate,
$\kappa_{\rm{c}}$, and $CE$. This relationship was explored in Ref. Sayson
_et al._ (2019), with the result that $CE$ may be as large as $25$%
(overcoupling the pump mode may further increase $CE$). Here, we reiterate
this result and explore it within the $\mu$OPO parameter space. Figure 8d
shows simulated signal mode lineshapes for two values of $\kappa_{\rm{c}}$,
and Fig. 8e shows the $CE$ map for $P_{\rm{in}}=20$ mW in a parameter space
defined by the coupling ratio, $\kappa_{\rm{c}}/\kappa_{\rm{i}}$, and
$\delta$. In our simulations, we use a TMA and keep the pump and idler modes
critically coupled. We not only observe an increase in the highest obtainable
$CE$ to nearly $25$%, but we also observe that the region of highest $CE$ is
broadened for increasing $\kappa_{\rm{c}}/\kappa_{\rm{i}}$. Broadening occurs
until $\kappa_{\rm{c}}/\kappa_{\rm{i}}\approx 30$, at which point the
advantages of overcoupling are overcome by the corresponding increases in
$P_{\rm{th}}$.

##  VI Discussion

In conclusion, we have established a foundation of simulation results for
$\mu$OPOs that moves beyond a TMA and will help guide experimental efforts to
realize the high conversion efficiencies predicted by the simplified theory.
We introduced a $CE$ map that encapsulates the $\mu$OPO solution space, and
through multi-mode LLE simulations we reveal nonlinear dynamics not
predictable from a TMA. In particular, we identified mode competition and
demonstrated how it determines the oscillating mode pair. Meanwhile, the range
of parameter space over which high $CE$ can be obtained is constrained by XPM-
MI. Mode competition and XPM-MI both depend on the microresonator dispersion,
$\delta_{\rm{\mu}}$, and suitable dispersion engineering may circumvent these
processes. Still, optimizing $\delta_{\rm{\mu}}$ may be nontrivial in
practice; therefore, we have proposed two strategies to increase $CE$ apart
from dispersion engineering. Ultimately, suitable control of both resonator
loss (including waveguide coupling) and dispersion makes it possible to tailor
microresonator geometries for high $CE$ at a targeted pump power, that is, to
produce a useful amount of output power. Such engineering will be crucial in
the development of compact, coherent light sources that take advantage of the
enormous wavelength flexibility inherent to $\chi^{(3)}$ OPOs.

###### Acknowledgements.

We thank Travis Briles and Edgar Perez for a careful reading of the paper.
This project is funded by the DARPA LUMOS program.

Supplemental Material

##  I Calculations of $\mu$OPO variables

In this section, we describe mathematical formulae that relate LLE variables
to the data (e.g. $CE$ and optical spectra) presented in the main text.
Simulations of the Lugiato-Lefever Equation (LLE) yield solutions for the
complex intraresonator field, $a$; we denote the intraresonator field spectrum
$\tilde{a}=\mathcal{F}(a)$, where $\mathcal{F}$ denotes the Fourier transform,
and $a$ is normalized such that $|\tilde{a}|^{2}_{\omega=\omega_{\mu}}$ gives
the number of intraresonator photons in the mode $\mu$.

Table 1 lists the important variables, along with their physical descriptions
and how they are calculated from $a$, $\tilde{a}$, and simulation parameters.
Notably, the expressions for $\phi_{\rm{p}}$ and $N_{\rm{p}}$ also apply to
the signal and idler modes when the expressions are evaluated at the
appropriate value of $\omega$. We follow Ref. Yu  _et al._ (2021) in our
calculation of the nonlinear mode frequency shifts.

Variable | Description (units) | LLE Calculation
---|---|---
$P_{\rm{sig}}$ | Output signal power (W) | $\kappa_{\rm{c}}\hbar\omega_{\rm{s}}|\tilde{a}|^{2}_{\omega=\omega_{\rm{s}}}$
$\phi_{\rm{p}}$ | Pump phase (rad) | $\angle\tilde{a}_{\omega=\omega_{\rm{0}}}$
$N_{\rm{p}}$ | Pump mode nonlinear frequency shift ($\kappa(0)/2\pi$) | $\frac{1}{\kappa(0)}\left(\frac{\mathcal{F}(g_{\rm{0}}|a|^{2}a)}{\tilde{a}}\right)_{\omega=\omega_{\rm{0}}}$
$P_{\rm{out}}$ | Pump laser output power ($W$) | $\hbar\omega_{\rm{p}}|\sqrt{\kappa_{\rm{c}}(0)}\tilde{a}_{\omega=\omega_{\rm{0}}}-\sqrt{\frac{P_{\rm{in}}}{\hbar\omega_{\rm{p}}}}|^{2}$
Table 1: List and descriptions of important variables derived from the LLE.

##  II Derivation of highest maximum $CE$ contour

Next, we seek to derive the result - stated in the main text - that the
contour of highest maximum $CE$ in the TMA is given by $X=8\delta$. Our
starting point is the following set of coupled equations Sayson  _et al._
(2019):

$\displaystyle\frac{dA_{\rm{p}}}{dt}$
$\displaystyle=-(1+i\alpha)A_{\rm{p}}+i(|A_{\rm{p}}|^{2}+2|A_{\rm{s}}|^{2}+2|A_{\rm{i}}|^{2})A_{\rm{p}}+2A_{\rm{p}}^{*}A_{\rm{s}}A_{\rm{i}}+\sqrt{X}$
(6) $\displaystyle\frac{dA_{\rm{s}}}{dt}$
$\displaystyle=-(1+i\alpha+i\delta)A_{\rm{s}}+i(|A_{\rm{s}}|^{2}+2|A_{\rm{p}}|^{2}+2|A_{\rm{i}}|^{2})A_{\rm{s}}+2A_{\rm{s}}^{*}A_{\rm{p}}A_{\rm{i}}$
(7) $\displaystyle\frac{dA_{\rm{i}}}{dt}$
$\displaystyle=-(1+i\alpha+i\delta)A_{\rm{i}}+i(|A_{\rm{i}}|^{2}+2|A_{\rm{p}}|^{2}+2|A_{\rm{s}}|^{2})A_{\rm{i}}+2A_{\rm{i}}^{*}A_{\rm{p}}A_{\rm{s}},$
(8)

where $A_{\rm{p(s,i)}}=\sqrt{I_{\rm{p(s,i)}}}e^{-i\phi_{\rm{p(s,i)}}}$ denotes
the pump (signal, idler) field, and $I_{\rm{p(s,i)}}=|A_{\rm{p(s,i)}}|^{2}$ is
proportional to the pump (signal, idler) intraresonator photon number. If we
substitute the field expressions into the above equations, we obtain six
coupled equations for the real variables $I_{\rm{p(s,i)}}$ and
$\phi_{\rm{p(s,i)}}$,

$\displaystyle\frac{dI_{\rm{p}}}{dt}$
$\displaystyle=-2I_{\rm{p}}-4I_{\rm{p}}\sqrt{I_{\rm{s}}I_{\rm{i}}}\textrm{sin}(\Delta\phi)+2\sqrt{I_{\rm{p}}X}\textrm{cos}(\phi_{\rm{p}})$
(9) $\displaystyle\frac{dI_{\rm{s}}}{dt}$
$\displaystyle=-2I_{\rm{s}}+2I_{\rm{p}}\sqrt{I_{\rm{s}}I_{\rm{i}}}\textrm{sin}(\Delta\phi)$
(10) $\displaystyle\frac{dI_{\rm{i}}}{dt}$
$\displaystyle=-2I_{\rm{i}}+2I_{\rm{p}}\sqrt{I_{\rm{s}}I_{\rm{i}}}\textrm{sin}(\Delta\phi)$
(11) $\displaystyle\frac{d\phi_{\rm{p}}}{dt}$
$\displaystyle=I_{\rm{p}}+2I_{\rm{s}}+2I_{\rm{i}}+2\sqrt{I_{\rm{s}}I_{\rm{i}}}\textrm{cos}(\Delta\phi)-\alpha-\sqrt{\frac{X}{I_{\rm{p}}}}\textrm{sin}(\phi_{\rm{p}})$
(12) $\displaystyle\frac{d\phi_{\rm{s}}}{dt}$
$\displaystyle=I_{\rm{s}}+2I_{\rm{p}}+2I_{\rm{i}}+I_{\rm{p}}\sqrt{\frac{I_{\rm{i}}}{I_{\rm{s}}}}\textrm{cos}(\Delta\phi)-\alpha-\delta$
(13) $\displaystyle\frac{d\phi_{\rm{i}}}{dt}$
$\displaystyle=I_{\rm{i}}+2I_{\rm{p}}+2I_{\rm{s}}+I_{\rm{p}}\sqrt{\frac{I_{\rm{s}}}{I_{\rm{i}}}}\textrm{cos}(\Delta\phi)-\alpha-\delta,$
(14)

where $\Delta\phi=2\phi_{\rm{p}}-\phi_{\rm{s}}-\phi_{\rm{i}}$. We next reduce
this system to four equations by first noting that, from the symmetry of Eqs.
7 and 8, $I_{\rm{s}}=I_{\rm{i}}$. Therefore, we define
$M=I_{\rm{s}}=I_{\rm{i}}$. Moreover, momentum conservation implies that
$\frac{d}{dt}(\phi_{\rm{s}}+\phi_{\rm{i}})=0$. Hence, for steady state
conditions we obtain

$\displaystyle I_{\rm{p}}$
$\displaystyle=-2I_{\rm{p}}M\textrm{sin}(\Delta\phi)+\sqrt{I_{\rm{p}}X}\textrm{cos}(\phi_{\rm{p}})$
(15) $\displaystyle 1$ $\displaystyle=I_{\rm{p}}\textrm{sin}(\Delta\phi)$ (16)
$\displaystyle\sqrt{\frac{X}{I_{\rm{p}}}}\textrm{sin}(\phi_{\rm{p}})$
$\displaystyle=I_{\rm{p}}+4M+2M\textrm{cos}(\Delta\phi)-\alpha$ (17)
$\displaystyle I_{\rm{p}}\textrm{cos}(\Delta\phi)$
$\displaystyle=\alpha-3M-2I_{\rm{p}}+\delta.$ (18)

Equations 15-18 are still sufficiently general to study the stationary
solutions. However, we proceed to simplify them via two ansatzes associated
with a maximally efficient $\mu$OPO. Specifically, we set $\phi_{\rm{p}}=0$,
which corresponds to the pump laser being on resonance, and
$M=\frac{I_{\rm{p}}}{2}$. The validity of these assumptions is confirmed by
our numerical results, but they are also physically intuitive. For instance,
because the FWM process that drives $\mu$OPO
($2\omega_{\rm{p}}\rightarrow\omega_{\rm{s}}+\omega_{\rm{i}}$) is reversible,
one expects the intraresonator photons to be evenly distributed between the
pump mode and sideband pair.

After inserting our two ansatzes into Eqs. 15-18, we combine Eqs. 15 and 16 to
obtain $I_{\rm{p}}=X/4$. Then, Eqs. 17 and 18 are combined to obtain
$\frac{I_{\rm{p}}}{2}=\delta$. Insertion of the former into the latter yields
the desired result, $X=8\delta$.

##  III Derivation of XPM-MI gain

Finally, in this section we derive Eq. 4 from the main text. We follow the
classical procedure of linearizing a set of coupled equations around a steady-
state solution and then allowing plane-wave perturbations to grow
exponentially. We start from the equations of motion for the fields
$A_{\rm{\mu}}$ and $A_{\rm{-\mu}}$ when the pump, signal, and idler modes are
occupied,

$\displaystyle\frac{dA_{\rm{\mu}}}{dt}$
$\displaystyle=i\beta\mu^{2}+i(2I_{\rm{p}}+2I_{\rm{s}}+2I_{\rm{i}}+2I_{\rm{-\mu}}+I_{\rm{\mu}})A_{\rm{\mu}}-(1+i\alpha)A_{\rm{\mu}}+iA_{\rm{\mu}}^{*}A_{\rm{p}}^{2}+iA_{\rm{\mu}}^{*}A_{\rm{s}}A_{\rm{i}}$
(19) $\displaystyle\frac{dA_{\rm{-\mu}}}{dt}$
$\displaystyle=i\beta\mu^{2}+i(2I_{\rm{p}}+2I_{\rm{s}}+2I_{\rm{i}}+I_{\rm{-\mu}}+2I_{\rm{\mu}})A_{\rm{-\mu}}-(1+i\alpha)A_{\rm{-\mu}}+iA_{\rm{-\mu}}^{*}A_{\rm{p}}^{2}+iA_{\rm{-\mu}}^{*}A_{\rm{s}}A_{\rm{i}},$
(20)

where $\beta$ quantifies the dispersion. Next, we introduce the perturbation
$\delta f_{\rm{\mu}}(t)$, so that the equations of motion for the
perturbations, after simplifying, read

$\displaystyle\frac{d\delta f_{\rm{\mu}}}{dt}$
$\displaystyle=(ik_{\rm{\mu}}-1)\delta
f_{\rm{\mu}}+i(A_{\rm{p}}^{2}+A_{\rm{s}}A_{\rm{i}})\delta f_{\rm{\mu}}^{*}$
(21) $\displaystyle\frac{d\delta f_{\rm{-\mu}}^{*}}{dt}$
$\displaystyle=-(ik_{\rm{\mu}}+1)\delta
f_{\rm{-\mu}}^{*}-i(A_{\rm{p}}^{*2}+A_{\rm{s}}^{*}A_{\rm{i}}^{*})\delta
f_{\rm{\mu}},$ (22)

where $k_{\rm{\mu}}=\beta\mu^{2}+2I_{\rm{p}}+4I_{\rm{s}}-\alpha$, and we have
again assumed $I_{\rm{s}}=I_{\rm{i}}$. If we set $\delta
f_{\rm{\mu}}(t)=ae^{\lambda_{\rm{\mu}}t}$, then we obtain a set of linear
homogenous equations with eigenvalue $\lambda_{\rm{\mu}}$. Solving the
eigenvalue problem yields

$\lambda_{\rm{\mu}}=-1\pm\sqrt{I_{\rm{p}}^{2}+I_{\rm{s}}^{2}+2I_{\rm{p}}I_{\rm{s}}\textrm{cos}(\Delta\phi)-k_{\rm{\mu}}^{2}},$
(23)

which is the desired result. Notably, the original set of coupled equations
are approximations in several ways. For instance, we only consider one
sideband pair, but the full XPM-MI dynamics involves the collective
interactions of multiple sideband pairs simultaneously. In addition, we
neglect FWM processes that do not involve both MI sidebands (e.g. the
interaction $A_{\rm{\mu}}^{*}A_{\rm{s}}^{*}A_{\rm{p}}A_{\rm{s+\mu}}$). These
approximations may explain the small discrepancies between our theory and LLE
simulations.

## References

  * Liang and Bowers (2010) D. Liang and J. E. Bowers, Nature Photonics 4, 511 (2010).
  * Agrell  _et al._ (2016) E. Agrell, M. Karlsson, A. Chraplyvy, D. J. Richardson, P. M. Krummrich, P. Winzer, K. Roberts, J. K. Fischer, S. J. Savory, B. J. Eggleton, _et al._ , Journal of Optics 18, 063002 (2016).
  * Sipahigil  _et al._ (2016) A. Sipahigil, R. E. Evans, D. D. Sukachev, M. J. Burek, J. Borregaard, M. K. Bhaskar, C. T. Nguyen, J. L. Pacheco, H. A. Atikian, C. Meuwly, _et al._ , Science 354, 847 (2016).
  * Elshaari  _et al._ (2020) A. W. Elshaari, W. Pernice, K. Srinivasan, O. Benson, and V. Zwiller, Nature Photonics 14, 285 (2020).
  * Jin  _et al._ (2021) W. Jin, Q.-F. Yang, L. Chang, B. Shen, H. Wang, M. A. Leal, L. Wu, M. Gao, A. Feshali, M. Paniccia, _et al._ , Nature Photonics 15, 346 (2021).
  * Vahala (2003) K. J. Vahala, Nature 424, 839 (2003).
  * Yang  _et al._ (2018) K. Y. Yang, D. Y. Oh, S. H. Lee, Q.-F. Yang, X. Yi, B. Shen, H. Wang, and K. Vahala, Nature Photonics 12, 297 (2018).
  * Kippenberg  _et al._ (2018) T. J. Kippenberg, A. L. Gaeta, M. Lipson, and M. L. Gorodetsky, Science 361 (2018).
  * Liu  _et al._ (2018) X. Liu, C. Sun, B. Xiong, L. Wang, J. Wang, Y. Han, Z. Hao, H. Li, Y. Luo, J. Yan, _et al._ , ACS Photonics 5, 1943 (2018).
  * Loh  _et al._ (2015) W. Loh, A. A. Green, F. N. Baynes, D. C. Cole, F. J. Quinlan, H. Lee, K. J. Vahala, S. B. Papp, and S. A. Diddams, Optica 2, 225 (2015).
  * Bruch  _et al._ (2019) A. W. Bruch, X. Liu, J. B. Surya, C.-L. Zou, and H. X. Tang, Optica 6, 1361 (2019).
  * Lu  _et al._ (2019) X. Lu, G. Moille, A. Singh, Q. Li, D. A. Westly, A. Rao, S.-P. Yu, T. C. Briles, S. B. Papp, and K. Srinivasan, Optica 6, 1535 (2019).
  * Spencer  _et al._ (2018) D. T. Spencer, T. Drake, T. C. Briles, J. Stone, L. C. Sinclair, C. Fredrick, Q. Li, D. Westly, B. R. Ilic, A. Bluestone, _et al._ , Nature 557, 81 (2018).
  * Marin-Palomo  _et al._ (2017) P. Marin-Palomo, J. N. Kemal, M. Karpov, A. Kordts, J. Pfeifle, M. H. Pfeiffer, P. Trocha, S. Wolf, V. Brasch, M. H. Anderson, _et al._ , Nature 546, 274 (2017).
  * Lai  _et al._ (2020) Y.-H. Lai, M.-G. Suh, Y.-K. Lu, B. Shen, Q.-F. Yang, H. Wang, J. Li, S. H. Lee, K. Y. Yang, and K. Vahala, Nature Photonics 14, 345 (2020).
  * Newman  _et al._ (2019) Z. L. Newman, V. Maurice, T. Drake, J. R. Stone, T. C. Briles, D. T. Spencer, C. Fredrick, Q. Li, D. Westly, B. R. Ilic, _et al._ , Optica 6, 680 (2019).
  * Vodopyanov  _et al._ (2000) K. Vodopyanov, F. Ganikhanov, J. Maffetone, I. Zwieback, and W. Ruderman, Optics Letters 25, 841 (2000).
  * Mieth  _et al._ (2014) S. Mieth, A. Henderson, and T. Halfmann, Optics express 22, 11182 (2014).
  * Boyd (2020) R. W. Boyd,  _Nonlinear optics_ (Academic press, 2020).
  * Lu and Srinivasan (2021) X. Lu and K. Srinivasan, Physical Review Applied 16, 014027 (2021).
  * Sayson  _et al._ (2019) N. L. B. Sayson, T. Bi, V. Ng, H. Pham, L. S. Trainor, H. G. Schwefel, S. Coen, M. Erkintalo, and S. G. Murdoch, Nature Photonics 13, 701 (2019).
  * Tang  _et al._ (2020) Y. Tang, Z. Gong, X. Liu, and H. X. Tang, Optics Letters 45, 1124 (2020).
  * Lu  _et al._ (2020) X. Lu, G. Moille, A. Rao, D. A. Westly, and K. Srinivasan, Optica 7, 1417 (2020).
  * Domeneguetti  _et al._ (2021) R. R. Domeneguetti, Y. Zhao, X. Ji, M. Martinelli, M. Lipson, A. L. Gaeta, and P. Nussenzveig, Optica 8, 316 (2021).
  * Marty  _et al._ (2021) G. Marty, S. Combrié, F. Raineri, and A. De Rossi, Nature Photonics 15, 53 (2021).
  * Hansson  _et al._ (2013) T. Hansson, D. Modotto, and S. Wabnitz, Physical Review A 88, 023819 (2013).
  * Coen  _et al._ (2013) S. Coen, H. G. Randle, T. Sylvestre, and M. Erkintalo, Optics Letters 38, 37 (2013).
  * Chembo and Menyuk (2013) Y. K. Chembo and C. R. Menyuk, Physical Review A 87, 053852 (2013).
  * Kippenberg  _et al._ (2004) T. Kippenberg, S. Spillane, and K. Vahala, Physical review letters 93, 083904 (2004).
  * Note (1) Finite element simulations performed using Comsol Multiphysics, which is identified here to foster understanding, without implying recommendation or endorsement by NIST.
  * Narducci  _et al._ (1986) L. Narducci, J. Tredicce, L. Lugiato, N. Abraham, and D. Bandy, Physical Review A 33, 1842 (1986).
  * Gong  _et al._ (2007) M. Gong, Y. Yuan, C. Li, P. Yan, H. Zhang, and S. Liao, Optics Express 15, 3236 (2007).
  * Agrawal (1987) G. P. Agrawal, Physical Review Letters 59, 880 (1987).
  * Fujii  _et al._ (2019) S. Fujii, S. Tanaka, M. Fuchida, H. Amano, Y. Hayama, R. Suzuki, Y. Kakinuma, and T. Tanabe, Optics letters 44, 3146 (2019).
  * Moille  _et al._ (2019) G. Moille, Q. Li, T. C. Briles, S.-P. Yu, T. Drake, X. Lu, A. Rao, D. Westly, S. B. Papp, and K. Srinivasan, Optics Letters 44, 4737 (2019).
  * Spencer  _et al._ (2014) D. T. Spencer, J. F. Bauters, M. J. Heck, and J. E. Bowers, Optica 1, 153 (2014).
  * Graziosi  _et al._ (2018) T. Graziosi, S. Mi, M. Kiss, and N. Quack, in  _2018 International Conference on Optical MEMS and Nanophotonics (OMN)_ (IEEE, 2018) pp. 1–5.
  * Yu  _et al._ (2021) S.-P. Yu, D. C. Cole, H. Jung, G. T. Moille, K. Srinivasan, and S. B. Papp, Nature Photonics 15, 461 (2021).


# AutoProtoNet: Interpretability for Prototypical Networks

Pedro Sandoval-Segura
Department of Computer Science
University of Maryland
<EMAIL_ADDRESS>
Wallace Lawson
Navy Center for Applied Research in Artificial Intelligence
Naval Research Laboratory
<EMAIL_ADDRESS>

###### Abstract

In meta-learning approaches, it is difficult for a practitioner to make sense
of what kind of representations the model employs. Without this ability, it
can be difficult to both understand what the model knows as well as to make
meaningful corrections. To address these challenges, we introduce
AutoProtoNet, which builds interpretability into Prototypical Networks by
training an embedding space suitable for reconstructing inputs, while
remaining convenient for few-shot learning. We demonstrate how points in this
embedding space can be visualized and used to understand class
representations. We also devise a prototype refinement method, which allows a
human to debug inadequate classification parameters. We use this debugging
technique on a custom classification task and find that it leads to accuracy
improvements on a validation set consisting of in-the-wild images. We advocate
for interpretability in meta-learning approaches and show that there are
interactive ways for a human to enhance meta-learning algorithms.

##  1 Introduction

††DISTRIBUTION STATEMENT A. Approved for public release: distribution
unlimited.

It is expensive and time-consuming to collect data to train current state-of-
the-art image classification systems [13]. When a classification algorithm is
deployed, new classes or labels cannot be easily added without incurring new
costs related to re-training the model [1][2]. Meta-learning approaches for
few-shot learning solve both these problems by training networks that learn
quickly from little data with computationally inexpensive fine-tuning
[23][20][15]. Despite these methods performing well on benchmark few-shot
image classification tasks, these methods are not interpretable; a human may
have no way of knowing why a certain classification decision was made.
Additionally, the lack of interpretability limits any kind of debugging of
network representations. In this work, we take a step toward the development
of a meta-learning algorithm which can learn in a few-shot setting, can handle
new classes at test time, is interpretable enough for a human to understand
how the model makes decisions, and which can be debugged in a simple way.

We revisit Prototypical Networks (ProtoNets) [20] as the focus of our study.
ProtoNets are based on a simple idea: there exists an embedding space where
images cluster around a single “prototype” for each class. Given the
simplicity of this few-shot learning approach, it makes sense to ask: what
does a prototype look like? And, have we learned an adequate prototype
representation?

The outcomes of our study can be summarized as follows:

  * •

We introduce AutoProtoNet, which merges ideas from autoencoders and
Prototypical Networks, to perform few-shot image classification and prototype
reconstruction.

  * •

We use AutoProtoNet to visualize prototypes and find that they are comparable
in quality to those of an autoencoder. AutoProtoNet also remains accurate on
few-shot image classification benchmarks.

  * •

We devise a prototype refinement method, which can be used to debug inadequate
prototypes, and we validate the performance of the resulting model using a
novel validation set of in-the-wild images.

Our goal in this work is to elucidate the benefits of learning embeddings that
can be visualized and interpreted by humans. To the best of our knowledge,
there is no meta-learning approach that allows for a human to play a role in
the fine-tuning of the base model.

##  2 Related Work

###  2.1 Meta-learning and Prototypical Networks

Before meta-learning, transfer learning was used to handle few-shot problems.
In transfer learning, a feature extractor is trained on a large dataset, then
fine-tuned for new tasks [2]. However, transfer learning has some drawbacks.
For example, adding a new class may require re-training the model and, in the
few-shot setting, overfitting few example images is possible.

Meta-learning algorithms aim to learn a “base” model that can be quickly fine-
tuned for a new task. The base model is trained using a set of training tasks
$\\{\mathcal{T}_{i}\\}$, sampled from some task distribution. Each task
consists of _support_ data, $\mathcal{T}_{i}^{s}$, and _query_ data,
$\mathcal{T}_{i}^{q}$. Support data is used to fine-tune the model, while
query data is used to evaluate the resulting model. Practically speaking, each
task is an image classification problem involving only a small number of
classes. The number of examples per class in the support set is called the
_shot_ , and the number of classes is called the _way_. For example, in 5-way
1-shot learning, we are given 1 example for each of the 5 classes to use for
fine-tuning.

Following the meta-learning framework presented in [8], Algorithm 1 can be
used as a general way to understand both metric-learning methods [23] [20] and
gradient-based methods like MAML [6].

Algorithm 1 The meta-learning framework

Input: Base model, $F_{\theta}$
Input: Fine-tuning algorithm, $A$
Input: Learning rate, $\gamma$
Input: Distribution over tasks, $p(\mathcal{T})$

1:Initialize $\theta$, the weights of $F$

2:while not done do

3: Sample batch of tasks $\\{\mathcal{T}_{i}\\}_{i=1}^{n}$, where
$\mathcal{T}_{i}\sim p(\mathcal{T})$ and
$\mathcal{T}_{i}=(\mathcal{T}_{i}^{s},\mathcal{T}_{i}^{q})$

4: for i=1,…,n do

5: $\theta_{i}\leftarrow A(\theta,\mathcal{T}_{i}^{s})$ $\triangleright$ Fine-
tune model on $\mathcal{T}_{i}^{s}$ (inner loop)

6:
$g_{i}\leftarrow\nabla_{\theta}\mathcal{L}(F_{\theta_{i}},\mathcal{T}_{i}^{q})$

7: end for

8: $\theta\leftarrow\theta-\frac{\gamma}{n}\sum_{i}g_{i}$ $\triangleright$
Update base model parameters (outer loop)

9:end while

For ProtoNets [20], the base model
$F_{\theta}:\mathbb{R}^{D}\rightarrow\mathbb{R}^{M}$ is an embedding network
which takes an image $x\in\mathbb{R}^{D}$ as input and outputs an embedding
vector of dimension $M$. Suppose, for example, we have a $K$-way task
$\mathcal{T}_{i}=(\mathcal{T}_{i}^{s},\mathcal{T}_{i}^{q})$ where
$\mathcal{T}_{i}^{s}=\\{(x_{i,1},y_{i,1}),(x_{i,2},y_{i,2}),...,(x_{i,N},y_{i,N})\\}$,
and where $y_{i,j}\in\\{1,...,K\\}$. Additionally, let
$S_{k}\subset\mathcal{T}_{i}^{s}$ denote the set of support examples of class
$k$. Then, a prototypical network computes a prototype $p_{k}$ for each class
$k$ by computing a class-wise mean of embedded support examples:

$p_{k}=\frac{1}{|S_{k}|}\sum_{(x,y)\in S_{k}}F_{\theta}(x)$ (1)

Thus, in the case of ProtoNets, the fine-tuning algorithm $A$ does not update
model parameters $\theta$, but instead it computes a set of prototypes which
the base model will use to classify query data. We can think of $A$ as a
function taking both embedding network parameters $\theta$ and support data
$\mathcal{T}_{i}^{s}$ and returning a tuple $\theta_{i}$ consisting of a set
of prototypes and an unchanged set of model parameters; i.e.,
$A(\theta,\mathcal{T}_{i}^{s})=(\\{p_{k}\\}_{i=0}^{k},\theta)=\theta_{i}$. In
this way, $F_{\theta_{i}}$ in Algorithm 1 refers to using the base model
parameters $\theta$ and the set of prototypes $\\{p_{k}\\}_{i=0}^{k}$ during
inference. Given a distance function
$d:\mathbb{R}^{M}\times\mathbb{R}^{M}\rightarrow[0,\infty)$ and a query point
$x$, a ProtoNet produces a distribution over classes based on a softmax over
distances to the prototypes in embedding space:

$p_{\theta}(y=k|x)=\frac{\exp(-d(F_{\theta}(x),p_{k}))}{\sum_{k^{\prime}}\exp(-d(F_{\theta}(x),p_{k^{\prime}}))}$
(2)

Training proceeds by minimizing the negative log-likelihood
$\mathcal{L}(\theta)=-\log p_{\theta}(y=k|x)$ of the true class $k$ using SGD.
Unfortunately, ProtoNet does not provide a way to understand the embedding
space or visualize $p_{k}$ – a problem we directly address in this work.

###  2.2 Understanding Meta-learning Approaches

Investigating the ability of meta-learning methods to adapt to new tasks has
been the subject of numerous studies. The success of meta-learning approaches
certainly seems to suggest that the representations learned by meta-learning
must be different than those learned through standard training [9]. Goldblum
et al. [9] find that meta-learned feature extractors outperform classically
trained models of the same architecture and suggest that meta-learned features
are qualitatively different from conventional features. While work has been
done to understand how the meta-learning networks train [10][7], there has
been little to no focus on developing tools to interpret the meta-learned
models.

###  2.3 Interpretability in Convolutional Models

In safety or security-critical applications, understanding why a
classification system made a certain prediction is important. Just because a
classification system is highly accurate, does not mean the network has
learned the right kinds of features [11]. We believe that a system that can
demonstrate its logic semantically or visually is more likely to be trusted
and used. Being that a ProtoNet is primarily a convolutional neural network,
it is appropriate to understand progress on interpretability of convolutional
neural networks (CNN).

There are many research branches within the umbrella of CNN interpretability
including visualizations of intermediate network layers [25][16][19][21],
diagnosis of CNN representations [27][26], and building explainable models
[28]. In contrast to works which focus their attention on CNN layers and
activations, we take a more specific approach in visualizing embedding space
for ProtoNets.

Zhang et al. [28] propose a compelling method of modifying convolutional
layers so that each filter learns to represent a particular object part, thus
allowing for each filter to correspond to a semantically meaningful image
feature. We believe there could be interesting work incorporating this
technique into meta-learning approaches, but is not appropriate for a shallow
embedding network like the one we employ for ProtoNets.

###  2.4 Generative Models

Work on Variational Prototyping Encoder (VPE) [12] is most similar to ours in
that a meta-task is used to learn an embedding space suitable for both few-
shot learning and unseen data representation. In contrast, we do not focus on
the image translation task from real images to prototypes and instead focus
our attention on visualizing prototypes for interpretability and refinement.

There are also a number of works which investigate connections between
autoencoder architectures and meta-learning, but which are not directly
applicable for interpretability of few-shot image classification. For example,
Wu et al. [24] propose the Meta-Learning Autoencoder (MeLA) framework which
learns a recognition and generative model to transform a single-task model
into one that can quickly adapt to new tasks using few examples. However,
their framework is meant for the more general understanding of _tasks_ like
physical state estimation and video prediction, as opposed to the image
classification tasks which we focus on. Similarly, Epstein et al. [5] develop
a meta-learning framework consisting of joint autoencoders for the purpose of
learning multiple tasks simultaneously, but this approach is tailored more for
the field of multi-task learning.

##  3 Algorithm

Our interpretability algorithm takes advantage of the simplicity of the
ProtoNet classification method. In particular, a ProtoNet classifies query
data according to the class of the prototype which the query data’s embedding
is nearest to, typically in Euclidean space. This classification method raises
an obvious question: what does a prototype look like? To answer this question,
we extend ProtoNets with a decoder to reconstruct images from embeddings.

###  3.1 Data

The CIFAR-FS dataset [3] is a recent few-shot image classification benchmark
consisting of all $100$ classes from CIFAR-100 [14]. Classes are randomly
split into 64, 16, and 20 for meta-training, meta-validation, and meta-testing
respectively. Every class contains $600$ images of size $32\times 32$.

The _mini_ ImageNet dataset [23] is another standard benchmark for few-shot
image classification. It consists of 100 randomly chosen classes from ILSVRC
2012 [4], which are split into 64, 16, and 20 classes for meta-training, meta-
validation, and meta-testing respectively. For every class, there are $600$
images of size $84\times 84$. We adopt the commonly-used Ravi and Larochelle
split proposed in [18].

###  3.2 Architecture

AutoProtoNet consists of an encoder-decoder architecture which compresses the
input to produce an embedding which must be reconstructed by the decoder.
There 4 sequential convolution blocks for the encoder and 4 sequential
transpose convolution blocks for the decoder. The details of these blocks can
be found in Table 2 of Appendix B. A forward pass through the model is shown
in Figure 1.

Output padding is used in the second transpose convolution block of the
decoder to ensure that the output size of the final transpose convolution
block matches the input $84\times 84$ dimensions of _mini_ ImageNet images,
but no output padding modifications are necessary for CIFAR-FS images.

Our architectural design choices imply that a $84\times 84$ _mini_ ImageNet
image is embedded as $1600$-dimensional vector, while a $32\times 32$ CIFAR-FS
image is embedded as $256$-dimensional vector.

Figure 1: Visualization of the forward pass through AutoProtoNet.

###  3.3 Training

Algorithm 2 AutoProtoNet Meta-Learning

Input: Encoder and decoder networks, $F_{\theta}$ and $G_{\phi}$, where
$\psi=[\theta;\phi]$
Input: Fine-tuning algorithm, $A$
Input: Reconstruction loss weight, $\lambda$
Input: Learning rate, $\gamma$
Input: Distribution over tasks, $p(\mathcal{T})$

1:Initialize $\theta,\phi$, the weights of encoder and decoder

2:while not done do

3: Sample batch of tasks $\\{\mathcal{T}_{i}\\}_{i=1}^{n}$, where
$\mathcal{T}_{i}\sim p(\mathcal{T})$ and
$\mathcal{T}_{i}=(\mathcal{T}_{i}^{s},\mathcal{T}_{i}^{q})$

4: for i=1,…,n do

5: $\hat{\mathcal{T}_{i}}\leftarrow G_{\phi}(F_{\theta}(\mathcal{T}_{i}))$
$\triangleright$ Reconstruct task data

6:
$\mathcal{L}_{R}\leftarrow\mathrm{MSE}(\mathcal{T}_{i},\hat{\mathcal{T}_{i}})$
$\triangleright$ Compute reconstruction loss

7: $\theta_{i}\leftarrow A(\theta,\mathcal{T}_{i}^{s})$ $\triangleright$
Compute prototypes (inner loop)

8: $\mathcal{L}_{C}\leftarrow\mathrm{NLL}(F_{\theta_{i}},\mathcal{T}_{i}^{q})$
$\triangleright$ Compute classification loss

9: $\mathcal{L}\leftarrow\mathcal{L}_{C}+\lambda\mathcal{L}_{R}$

10: $g_{i}\leftarrow\nabla_{\psi}\mathcal{L}$

11: end for

12: $\psi\leftarrow\psi-\frac{\gamma}{n}\sum_{i}g_{i}$ $\triangleright$ Update
base model parameters (outer loop)

13:end while

Training AutoProtoNet is not much different from training a ProtoNet. The main
difference is that we augment the meta-training loop with a reconstruction
loss to regularize the embedding space and make it suitable for image
reconstruction. We display the forward pass through AutoProtoNet in Figure 1
and adapt the meta-learning framwork from Section 2.1 to describe the meta-
training of AutoProtoNet in Algorithm 2.

Our “base” model now consists of parameters $\psi$ which is a concatenation of
encoder network parameters $\theta$ and decoder network parameters $\phi$. In
Line 5 of Algorithm 2, we pass both support and query data from the current
task $\mathcal{T}_{i}$ through the encoder and decoder to produce a
reconstruction $\hat{\mathcal{T}_{i}}$. This reconstruction is then compared
to the original data using mean squared error (MSE) loss. The finetuning
algorithm in Line 7 of Algorithm 2 is identical to the description in Section
2.1, where $\theta_{i}=(\\{p_{k}\\}_{i=0}^{k},\theta)$ is a tuple consisting
of a set of prototypes for every class and the encoder network’s model
parameters. Both of these are used to compute the likelihood of the true
labels of our query data as in Equation 2, which is maximized by minimizing
the negative log-likelihood (NLL). Finally, the classification loss
$\mathcal{L}_{C}$ and the reconstruction loss $\mathcal{L}_{R}$ are summed so
they can be jointly optimized.

We meta-train ProtoNet and AutoProtoNet on both _mini_ ImageNet and CIFAR-FS.
To create a prototype reconstruction baseline, we also train two models which
make use of ILSVRC 2012 [4], which we refer to as ImageNet Autoencoder and
ImageNet AutoProtoNet. Note that because _mini_ ImageNet is a subset of ILSVRC
2012, the ImageNet models also provide insight into whether more data during
pretraining offers any benefit for meta-learning or prototype reconstructions.
All training was performed on a single NVIDIA Quadro P6000 from our internal
cluster. Training details for each model used in this work are described
below.

#### ProtoNet

Using Algorithm 1, we meta-train a standard ProtoNet for 30 epochs using SGD.
Our SGD optimizer uses Nesterov momentum of $0.9$, weight decay of $5\times
10^{-4}$, and a learning rate of $0.1$, which we decrease to $0.06$ after $20$
epochs.

#### AutoProtoNet

Using Algorithm 2, we meta-train an AutoProtoNet for 30 epochs using SGD. We
use the same SGD settings as in ProtoNet training. We use a reconstruction
loss weight $\lambda=1$. Following [20], both ProtoNet and AutoProtoNet models
were trained using 20-way 5-shot episodes, where each class contains $15$
query points per episode, for $30$ epochs.

#### ImageNet Autoencoder

We train an autoencoder of the same architecture as AutoProtoNet using only
mean squared error (MSE) loss on ILSVRC 2012 [4] for 20 epochs. We use the SGD
optimizer with Nesterov momentum of $0.9$, weight decay of $5\times 10^{-4}$,
and a learning rate of $0.1$, which we decrease by a factor of $10$ every $5$
epochs. To evaluate this model’s performance on benchmark few-shot image
classification datasets, we make use of the only the encoder to produce
embeddings and produce classification labels using the standard ProtoNet
classification rule.

#### ImageNet AutoProtoNet

We use the encoder and decoder weights from the ImageNet Autoencoder as a
starting point for the weights of an AutoProtoNet. All other training details
are identical to that of AutoProtoNet, which we meta-train using Algorithm 2.

The 5-way 5-shot test set accuracies of all models used in this work are shown
in Table 1. AutoProtoNet is able to maintain the same level of few-shot image
classification accuracy on benchmark datasets as a standard ProtoNet. While we
expected AutoProtoNet to have an advantage due to having to incorporate
features useful for reconstruction into embeddings, our results suggest that
these reconstruction features are not always useful. Given the additional
ILSVRC 2012 [4] data during pretraining, we also expected that ImageNet
AutoProtoNet would outperform all other models, but our test results
demonstrate that representations learned for image reconstruction are not too
helpful for few-shot image classification. Test set accuracies for ImageNet
Autoencoder underscore the point that an embedding space trained for only
reconstruction is by no means competitive for few-shot classification, though
it does achieve better than chance accuracy.

##  4 Experiments

###  4.1 Prototype Visualization

While a standard ProtoNet employs an intuitive nearest-neighbor classification
rule for query points, there is no intuitive way for a user to understand what
a prototype embedding represents. Prototypical embeddings are crucial to
understanding the decision boundaries of ProtoNets. The idea is that a
ProtoNet embeds similar images nearby in embedding space, but without a way to
visualize these embeddings, we argue that a human practitioner would be unable
to debug or improve their deployed model. AutoProtoNet addresses this issue by
learning an embedding space that is suitable for image reconstruction.

Figure 2 displays prototype visualizations given a validation support set from
_mini_ ImageNet and CIFAR-FS. The ImageNet Autoencoder (IA) and ImageNet
AutoProtoNet (IAP) were both pretrained on all of ILSVRC 2012 [4], and so
classes present in this validation support set are not novel classes because
_mini_ ImageNet is a subset of ILSVRC 2012. However, in the case of the
AutoProtoNet (AP), the classes in this validation support set are novel and
the synthesized prototype images remain qualitatively on-par with the models
trained with more data (such as ImageNet Autoencoder), suggesting that meta-
tasks during training were sufficient to regularize an embedding space
suitable for image synthesis. Analyzing the prototype reconstructions from
CIFAR-FS in Figure 2(b), we see that prototype visualizations are generally
too blurry to help a human determine whether the model has learned a
sufficient representation of a class. We believe part of the problem is the
low resolution and size of CIFAR-FS images.

(a) _mini_ ImageNet

(b) CIFAR-FS

Figure 2: Support sets for a 5-way 5-shot validation task of _mini_ ImageNet (a) and CIFAR-FS (b). The embeddings of every image within a class are averaged to form a prototype embedding which is then synthesized as an image by using the decoder of an ImageNet Autoencoder (IA), an ImageNet AutoProtoNet (IAP), and an AutoProtoNet (AP).  Table 1: 5-way 5-shot test set accuracies with 95% confidence intervals. Model |  _mini_ ImageNet | CIFAR-FS
---|---|---
ImageNet Autoencoder |  $36.83\pm 0.48$% |  $46.08\pm 0.58$%
ImageNet AutoProtoNet |  $70.76\pm 0.51$% |  $79.65\pm 0.52$%
ProtoNet |  $70.20\pm 0.52$% |  $80.31\pm 0.51$%
AutoProtoNet |  $70.61\pm 0.52$% |  $80.16\pm 0.52$%

###  4.2 Human-guided Prototype Refinement

(a) Support set and prototype visualizations

(b) New image and corresponding embedding

(c) Interpolating 10 steps from initial prototype to new image embedding

(d) New set of prototypes

Figure 3: Steps for human-guided prototype selection in a 5-way 1-shot task.
Step (a): a human chooses an initial prototype to refine. Step (b): a human
captures one additional image to guide prototype refinement. Step (c):
Interpolations between the initial prototype and the new image embedding
(index 9) are shown to the human and a new prototype selection is made. Step
(d): A new set of prototypes is set, with class 2 having been refined.

To highlight the benefits of an embedding space suitable for image
reconstruction, we designed an experiement to demonstrate how a human can
guide prototype selection at test-time using AutoProtoNet. Assuming the user
knows the kinds of images the model will encounter at inference time and given
the ability to capture one more image, could we refine an initial prototype to
achieve higher accuracy on the validation set?

#### Data Collection

Based on objects we had around the house, we chose to formulate a 5-way 1-shot
classification problem between “door knob”, “frying pan”, “light switch”,
“orange”, and “water bottle”. Note that “orange” and “frying pan” are classes
in the _mini_ ImageNet training split, but all other classes are novel.
Because we sought to demonstrate how one might use an AutoProtoNet in a real-
world setting, all $55$ images in this task are novel, in-the-wild images,
captured using an iPhone 12. Our support set consists of $5$ images ($1$ image
per class). Our validation set consists of $50$ images ($10$ images per class)
and can be found in Figure 4 of Appendix A.

#### Prototype Refinement

Prototype refinement is a debugging technique meant for cases in which a human
believes prototype visualization may not be representative of the class. To
exaggerate the idea of prototype refinement, we purposefully choose the back-
side of a frying pan as a support image for class 1 (“frying pan”) so that the
prototype visualization has undesirable image features. Generally, a prototype
for an arbitrary object of a novel class is likely to be visually ambiguous if
the embedding network did not train on a suitable dataset, so this setup is
conceivable in the real-world.

For our classification model, we make use of the AutoProtoNet described in
Section 3.3. To apply AutoProtoNet to this new classification task, we “fine-
tine” AutoProtoNet by providing a support set shown in Figure 3(a). After
meta-learning, an AutoProtoNet’s only changeable parameters are its prototypes
which, by design, can be reconstructed into images using the decoder. By
visually understanding an AutoProtoNet’s embedding space, a user can choose to
change image features of a prototype reconstruction, thus changing the
prototype itself. In contrast, a standard ProtoNet performs inference using
its support data, which is visually inaccessible and uninterpretable.

Using a newly captured image $x\in\mathbb{R}^{d}$, we use the encoder
$F_{\theta}$ to generate an embedding $p=F_{\theta}(x)$. Given an initial
prototype $p_{k}$ for class $k$, we use the decoder $G_{\phi}$ to synthesize
images $\hat{x}_{i}\in\mathbb{R}^{d}$ for interpolations between $p_{k}$ and
$p$ as follows:

$\hat{x}_{i}=G_{\phi}((1-\alpha)p_{k}+\alpha p)\qquad\alpha\in[0,1]$ (3)

#### Results

Using the initial prototypes from Figure 3(a), AutoProtoNet achieves $80\%$
accuracy on the validation set consisting of $50$ images from all $5$ classes.
The $10$ misclassified images are all of the “frying pan” class. After
debugging the “frying pan” prototype by capturing an additional image of a
correctly-oriented frying pan and choosing an interpolation, the resulting
embedding is used as the new support as shown in Figure 3(d). Under the new
human-guided prototypes, AutoProtoNet achieves an accuracy of $98\%$ on the
validation set, where the single misclassified image is of the “door knob”
class.

The novelty of our method lies in the ability for a human to fine-tune the
model in an interactive way, leading to a performance increase in validation
set accuracy. In this example, AutoProtoNet’s decoder allowed for the
visualization of the prototype embedding, which we found to be visually
incorrect. Thus, we captured an additional, more representative image to
designate the direction in which to move the initial prototype to fit a human-
designated criteria.

##  5 Conclusion

With AutoProtoNet, we present a step toward meta-learning approaches capable
of giving some insight into their learned parameters. We argue that if meta-
learning approaches are to be useful in practice, there should be ways for a
human to glean some insight into why a classification might have been made.
Through prototype visualizations and a prototype refinement method, we
highlight the benefits of AutoProtoNet and take steps to improve a simple few-
shot classification algorithm by making it more interpretable while
maintaining the same degree of accuracy as a standard ProtoNet.

Our proposed method could likely be extended to Relation Networks [22],
MetaOptNet [15], or R2D2 [3], with a decoder network to visualize embeddings.
It may also be possible to meta-train a variational autoencoder to learn a
latent space more suitable for detailed image synthesis. We believe generative
models can play a larger role in interpretability of meta-learning algorithms.

To confirm the effectiveness of our interpretability results, we intend to
perform a human subjects study where a human determines whether prototype
visualizations help in understanding classification results. We also recognize
the limits of using a small dataset to evaluate the performance of our
prototype refinement method. We leave the creation of a larger, more diverse
validation set to future work.

## Acknowledgments and Disclosure of Funding

This work was supported by the Office of Naval Research (WL). The views and
conclusions contained in this document are those of the authors and should not
be interpreted as necessarily representing the official policies, either
expressed or implied, of the U.S. Navy.

## References

  * Altae-Tran et al. [2016] H. Altae-Tran, B. Ramsundar, A. S. Pappu, and V. S. Pande.  Low data drug discovery with one-shot learning.  _CoRR_ , abs/1611.03199, 2016.  URL http://arxiv.org/abs/1611.03199.
  * Bengio [2012] Y. Bengio.  Deep learning of representations for unsupervised and transfer learning.  In _Proceedings of ICML workshop on unsupervised and transfer learning_ , pages 17–36. JMLR Workshop and Conference Proceedings, 2012.
  * Bertinetto et al. [2018] L. Bertinetto, J. F. Henriques, P. H. S. Torr, and A. Vedaldi.  Meta-learning with differentiable closed-form solvers.  _CoRR_ , abs/1805.08136, 2018.  URL http://arxiv.org/abs/1805.08136.
  * Deng et al. [2009] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE Conference on Computer Vision and Pattern Recognition_ , pages 248–255, 2009.  doi: 10.1109/CVPR.2009.5206848.
  * Epstein et al. [2019] B. Epstein, R. Meir, and T. Michaeli.  Joint autoencoders: A flexible meta-learning framework.  In M. Berlingerio, F. Bonchi, T. Gärtner, N. Hurley, and G. Ifrim, editors, _Machine Learning and Knowledge Discovery in Databases_ , pages 494–509, Cham, 2019. Springer International Publishing.  ISBN 978-3-030-10925-7.
  * Finn et al. [2017] C. Finn, P. Abbeel, and S. Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  _CoRR_ , abs/1703.03400, 2017.  URL http://arxiv.org/abs/1703.03400.
  * Frosst et al. [2019] N. Frosst, N. Papernot, and G. Hinton.  Analyzing and improving representations with the soft nearest neighbor loss, 2019.
  * Goldblum et al. [2019] M. Goldblum, L. Fowl, and T. Goldstein.  Robust few-shot learning with adversarially queried meta-learners.  _CoRR_ , abs/1910.00982, 2019.  URL http://arxiv.org/abs/1910.00982.
  * Goldblum et al. [2020] M. Goldblum, S. Reich, L. Fowl, R. Ni, V. Cherepanova, and T. Goldstein.  Unraveling meta-learning: Understanding feature representations for few-shot tasks.  _CoRR_ , abs/2002.06753, 2020.  URL https://arxiv.org/abs/2002.06753.
  * Huang et al. [2019] W. R. Huang, Z. Emam, M. Goldblum, L. Fowl, J. K. Terry, F. Huang, and T. Goldstein.  Understanding generalization through visualizations.  _CoRR_ , abs/1906.03291, 2019.  URL http://arxiv.org/abs/1906.03291.
  * Ilyas et al. [2019] A. Ilyas, S. Santurkar, D. Tsipras, L. Engstrom, B. Tran, and A. Madry.  Adversarial examples are not bugs, they are features, 2019.
  * Kim et al. [2019] J. Kim, T. Oh, S. Lee, F. Pan, and I. S. Kweon.  Variational prototyping-encoder: One-shot learning with prototypical images.  _CoRR_ , abs/1904.08482, 2019.  URL http://arxiv.org/abs/1904.08482.
  * Kolesnikov et al. [2019] A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver, J. Yung, S. Gelly, and N. Houlsby.  Large scale learning of general visual representations for transfer.  _CoRR_ , abs/1912.11370, 2019.  URL http://arxiv.org/abs/1912.11370.
  * Krizhevsky [2009] A. Krizhevsky.  Learning multiple layers of features from tiny images.  2009\.
  * Lee et al. [2019] K. Lee, S. Maji, A. Ravichandran, and S. Soatto.  Meta-learning with differentiable convex optimization.  _CoRR_ , abs/1904.03758, 2019.  URL http://arxiv.org/abs/1904.03758.
  * Mahendran and Vedaldi [2014] A. Mahendran and A. Vedaldi.  Understanding deep image representations by inverting them.  _CoRR_ , abs/1412.0035, 2014.  URL http://arxiv.org/abs/1412.0035.
  * Paszke et al. [2019] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala.  Pytorch: An imperative style, high-performance deep learning library.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems 32_ , pages 8024–8035. Curran Associates, Inc., 2019.  URL http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf.
  * Ravi and Larochelle [2017] S. Ravi and H. Larochelle.  Optimization as a model for few-shot learning.  In _ICLR_ , 2017.
  * Simonyan et al. [2014] K. Simonyan, A. Vedaldi, and A. Zisserman.  Deep inside convolutional networks: Visualising image classification models and saliency maps, 2014.
  * Snell et al. [2017] J. Snell, K. Swersky, and R. S. Zemel.  Prototypical networks for few-shot learning.  _CoRR_ , abs/1703.05175, 2017.  URL http://arxiv.org/abs/1703.05175.
  * Springenberg et al. [2015] J. T. Springenberg, A. Dosovitskiy, T. Brox, and M. Riedmiller.  Striving for simplicity: The all convolutional net, 2015.
  * Sung et al. [2017] F. Sung, Y. Yang, L. Zhang, T. Xiang, P. H. S. Torr, and T. M. Hospedales.  Learning to compare: Relation network for few-shot learning.  _CoRR_ , abs/1711.06025, 2017.  URL http://arxiv.org/abs/1711.06025.
  * Vinyals et al. [2016] O. Vinyals, C. Blundell, T. P. Lillicrap, K. Kavukcuoglu, and D. Wierstra.  Matching networks for one shot learning.  _CoRR_ , abs/1606.04080, 2016.  URL http://arxiv.org/abs/1606.04080.
  * Wu et al. [2018] T. Wu, J. Peurifoy, I. L. Chuang, and M. Tegmark.  Meta-learning autoencoders for few-shot prediction, 2018.
  * Zeiler and Fergus [2013] M. D. Zeiler and R. Fergus.  Visualizing and understanding convolutional networks.  _CoRR_ , abs/1311.2901, 2013.  URL http://arxiv.org/abs/1311.2901.
  * Zhang et al. [2017a] Q. Zhang, R. Cao, F. Shi, Y. N. Wu, and S.-C. Zhu.  Interpreting cnn knowledge via an explanatory graph, 2017a.
  * Zhang et al. [2017b] Q. Zhang, R. Cao, Y. N. Wu, and S.-C. Zhu.  Growing interpretable part graphs on convnets via multi-shot learning, 2017b.
  * Zhang et al. [2018] Q. Zhang, Y. N. Wu, and S.-C. Zhu.  Interpretable convolutional neural networks.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 8827–8836, 2018.

##  Appendix A Validation Set for Custom Classification Task

In Figure 4, we display the $50$ images of our custom $5$-way validation set.
The images from the “light switch” and “door knob” classes are diverse in
terms of shape, pose, and lighting condition.

Figure 4: Validation set for experiment described in Section 4.2

##  Appendix B Architecture Details

In our description of the AutoProtoNet architecture in Table 2, we display
output sizes for the first Conv Block of the encoder and the first Conv
Transpose Block of the decoder, assuming an $84\times 84$ _mini_ ImageNet
image is used as input.

Table 2: AutoProtoNet Architecture Components Conv Block |  | Conv Transpose Block
---|---|---
Layer | Parameters | Output Size |  | Layer | Parameters | Output Size
Conv |  $3\times 3$, $64$ | $64\times 84\times 84$ |  | Conv Transpose |  $2\times 2$, $*2$ | $64\times 10\times 10$
Batch Norm |  |  |  | Batch Norm |  |
Max Pool |  $3\times 3$, /2 | $64\times 42\times 42$ |  | Conv |  $3\times 3$, $64$ | $64\times 10\times 10$

##  Appendix C Implementation Details

We use PyTorch [17] and work on a fork of code used for [8], which uses the
MIT License. Our fork can be used to reproduce experiments and is available
here: https://github.com/psandovalsegura/AdversarialQuerying.