# Prediction Calibration for Generalized Few-shot Semantic Segmentation

Zhihe Lu, Sen He, Da Li, Yi-Zhe Song, , Tao Xiang Manuscript received April
19, 2021; revised August 16, 2021.Zhihe Lu (e-mail:
zhihelu.academic@gmail.com), Yi-Zhe Song and Tao Xiang are with the Centre for
Vision Speech and Signal Processing (CVSSP), University of Surrey, and also
with iFlyTek-Surrey Joint Research Center on Artificial Intelligence,
University of Surrey, Guildford GU2 7XH, United Kingdom.Sen He is with Meta
AI, London, United Kingdom.Da Li is with Samsung AI center, Cambridge CB1 2JH,
United Kingdom.

###### Abstract

Generalized Few-shot Semantic Segmentation (GFSS) aims to segment each image
pixel into either base classes with abundant training examples or novel
classes with only a handful of (e.g., 1-5) training images per class. Compared
to the widely studied Few-shot Semantic Segmentation (FSS), which is limited
to segmenting novel classes only, GFSS is much under-studied despite being
more practical. Existing approach to GFSS is based on classifier parameter
fusion whereby a newly trained novel class classifier and a pre-trained base
class classifier are combined to form a new classifier. As the training data
is dominated by base classes, this approach is inevitably biased towards the
base classes. In this work, we propose a novel Prediction Calibration Network
(PCN) to address this problem. Instead of fusing the classifier parameters, we
fuse the scores produced separately by the base and novel classifiers. To
ensure that the fused scores are not biased to either the base or novel
classes, a new Transformer-based calibration module is introduced. It is known
that the lower-level features are useful of detecting edge information in an
input image than higher-level features. Thus, we build a cross-attention
module that guides the classifier’s final prediction using the fused multi-
level features. However, transformers are computationally demanding.
Crucially, to make the proposed cross-attention module training tractable at
the pixel level, this module is designed based on feature-score cross-
covariance and episodically trained to be generalizable at inference time.
Extensive experiments on PASCAL-$5^{i}$ and COCO-$20^{i}$ show that our PCN
outperforms the state-the-the-art alternatives by large margins.

###### Index Terms:

Generalized Few-shot Semantic Segmentation, Prediction Calibration, Normalized
Score Fusion, Feature-score Cross-covariance Transformer.

##  I Introduction

Image semantic segmentation has been studied for decades. In the past 8-10
years, it has made arguably the largest advancements, thanks to the
availability of large-scale annotated datasets and advanced deep segmentation
models [1, 2, 3]. However, existing semantic segmentation methods are still
limited in real-world applications because of the small number of object
classes they can recognize. This is due to the fact that data annotation for
semantic segmentation is at the pixel level and thus particularly laborious
and expensive. For example, it took more than 70,000 worker hours to build the
popular COCO [4] segmentation dataset with merely 80 common object classes. It
is thus infeasible to scale the efforts to the millions of object classes
existing in the natural world.

Few-shot Semantic Segmentation (FSS) offers a solution to the scalabilty
problem and has been studied extensively in recent years [5, 6, 7, 8, 9, 10,
11, 12]. With FSS, a new object class can be segmented with only a handful of
training examples. To achieve this, existing methods follow the meta-learning
paradigm that learns to learn a model by simulating the few-shot scenarios on
a large-scale annotated dataset. The learned model can then adapt quickly to
novel classes using only a few support examples per class. However, the
existing FSS setting is still unsuited for real-world deployments. This is
because it focuses only on segmenting novel classes while ignoring the base
classes that typically co-exist with the novel ones in the same scene.

In contrast, a Generalized Few-shot Semantic Segmentation (GFSS) setting is
more practical and thus the focus of this work. A GFSS model, once trained on
abundant base class examples and a few novel class ones, is able to recognize
both types of classes during inference time. Though it is clearly more
practical, GFSS has received very little attention with [13, 14] being only
two existing works as far as we know. This is because GFSS is also much more
challenging than FSS due to a base class bias problem. More specifically, a
straightforward solution for GFSS, adopted in [13, 14], is classifier
parameter fusion. That is, one first learns a base classifier on the base
training set, and then learns a novel classifier on the support set of novel
classes via existing FSS methods such as prototypical learning [15, 16, 6].
The two classifiers are then fused into one to segment both base and novel
classes. However, since the base classes have much more classes as well as
annotated training examples per class, the fused classifier would be
inevitably biased towards them (see results in Table IV and Figure 5).

Figure 1: Illustrating the overconfidence problem under our Normalized Score
Fusion (NSF). From left to right: support image, query image, base class
classifier’s prediction score heat map ($y_{base}$ HM) for chosen class (e.g.,
bus, cow and person), novel class classifier’s prediction score heat map
($y_{novel}$ HM), NSF’s result and our final model PCN’s result. Each row
contains one task (airplane, boat, bottle and bike as novel respectively). The
red circles on NSF’s result highlight the incorrect predictions caused by the
overconfidence of NSF on novel classes – base class pixels are wrongly
classified as novel.

In this paper, we propose a novel Prediction Calibration Network (PCN) for
GFSS to address the base class bias problem. Instead of classifier parameter
fusion, we resort to classifier score fusion with base-novel score
calibration. Specifically, PCN consists of two different calibration modules:
Normalized Score Fusion (NSF) and Transformer based prediction calibration.
NSF is designed to prevent base class bias. Concretely, two sets of prediction
scores produced separately by the base and novel classifiers are normalized
(by softmax) and then fused. We show empirically that by simply switching to
score-level fusion, we can indeed effectively remove the base class bias (see
Table IV) and achieve the SOTA performance (see Table III). However, another
type of bias now appears, namely novel class bias (see Figure 1). This is
because typically the number of novel classes is much smaller than base. A
softmax based normalization for them is thus prone to overshooting their
prediction confidence. A more principled approach to avoid both types of bias
is thus needed.

To that end, we propose to exploit the cross attention between the model
predictions (scores) and input features to enforce feature-score consistency,
i.e., two pixels with similar features should be assigned to the same class.
Especially, the features are designed to contain both detailed and semantic
knowledge, thereby benefiting the calibration process by enriching the few-
shot learned poor representation. The calibration is implemented by a
Transformer but it must be specially designed. This is because a standard
token (each pixel/patch regarded as a token) based cross attention is not
applicable to our case. First, the pixel number is too large leading to an
intractable computational complexity. Also the varying class numbers between
the base training and final test require different projection heads.
Therefore, inspired by the latest efficient Transformer design, cross-
covariance Transformer [17], we propose a novel covariance based Transformer
to tackle our problem. More specifically, we treat each class (score) and
feature channel as a token. After being embedded by the query and key
projection heads separately, their cross-covariance is calculated to transform
the value embeddings, which are mapped from the feature channel tokens.
Subsequently, the transformed values are mapped by a calibration head to
calibrate the original prediction scores. Our Transformer is episodically
trained so that it can generalize during inference time.

Our contributions are as follows.

  * •

For the first time, we found and investigated the core issue of existing GFSS
methods. i.e., base class bias issue, due to directly combining two types of
classifiers’ parameters, and proposed a simple yet efficient baseline –
Normalized Score Fusion (NSF). NSF alone can prevent the base-class bias
problem caused by classifier-parameter-level fusion, but is prone to novel-
class bias.

  * •

To address the novel-class bias, a novel feature-score cross-covariance based
Transformer is further proposed and meta-learned for prediction calibration.
Crucially, we introduce the multi-level fused feature for the few-shot learned
poor representation enrichment.

  * •

Extensive experiments on PASCAL-$5^{i}$ and COCO-$20^{i}$ demonstrate that our
PCN (NSF and Transformer combined) outperforms the state-of-the-art
alternatives by significant margins.

Figure 2: An overview of our proposed pipeline. A feature extractor $F$ and a
base classifier $G_{base}$ are first trained by standard segmentation training
on base classes (not shown in this figure) and frozen in subsequent training
stages. In the meta-training stage, we first learn a novel classifier
$G_{novel}$ on a support set and fix its parameters. With $G_{base}$ and
$G_{novel}$, we can now obtain two prediction scores $y_{base}$ and
$y_{novel}$ on a query set. Normalized Score Fusion (NSF) is then applied on
$y_{base}$ and $y_{novel}$ to form the initial prediction scores $y_{all}$ for
all classes. Finally, $y_{all}$ is calibrated by a Transformer-based
calibration module for the final prediction. During inference, given a new
task, only a novel classifier is needed to be trained on a support set while
other modules are frozen. Note that the prediction scores shown in the figure
are examples for a single pixel, 3 base classes and 1 novel class for
simplicity. Dashed boxes represent the frozen modules.

##  II Related Work

###  II-A Few-shot Learning

Few-Shot Learning (FSL) aims at learning a model that can recognize novel
classes with limited samples. There are two main streams for existing FSL
methods: optimization based [18, 19, 20] and metric-learning based [21, 22,
23, 24, 25]. Optimization based methods learn a model with good initialization
that can adapt to novel classes fast. For example, MAML [19] proposed the
model-agnostic meta-learning that trains a model on a variety of learning
tasks for a fast adaptation on unseen classes. Each task in MAML contains a
small amount of training data and the model is trained on it for a handful of
steps so that the model is generalized for a new task. [18] proposed to
combine a memory bank and meta-learning the network’s parameters for unlabeled
images for one-shot learning. Typically, a contextual learner with sequential
inputs from the memory bank slots is able to predict the parameters for
unlabeled data. Further, LEO [20] focused on the unbalanced problem: high-
dimensional parameter spaces vs. extreme low-data regimes, by introducing
latent embedding optimization. In contrast, the purpose of metric-learning
based methods is to learn a good embedding function which can generate a
robust prototype for each novel class. For instance, the popular Prototypical
Networks [21] meta-learned a prototype for each novel class by averaging the
latent embeddings in the support set. The predicted class of a sample is
determined by the closest prototype to its output feature. The closeness is
computed by Euclidean distance. Instead of using existing distance metrics,
[22] proposed to learn to learn a deep distance metric that compares the
similarity between support and query samples.

Generalized Few-Shot Learning (GFSL) [26] extends FSL to simultaneously
recognize both base and novel classes. Generally, two steps are needed for a
GFSL model: many-shot learning on base classes and few-shot learning on novel
classes. [27] introduced an attention based few-shot classification weight
generator to generate a classification weight vector for novel class, which is
then combined with the trained base class classifier weights for GFSL. ACASTLE
[28] introduced a neural dictionary based Transformer to adjust the combined
classifier weights. The common characteristic for these works is to do
classifier parameter (weights) fusion. In this work, we study Generalized Few-
shot Semantic Segmentation (GFSS), a similar but more challenging task
compared to GFSL. Instead of fusing the base and novel classifiers at the
parameter-level, we propose to fuse at score level and design a Transformer-
based score calibration method to avoid fusion bias.

###  II-B Few-shot Semantic Segmentation

Few-shot Semantic Segmentation (FSS) targets at the same few-shot challenge as
FSL but focuses on semantic segmentation task. Existing FSS methods generally
build relationships between query images and support images in two ways:
prototypical learning [15, 16, 6] and feature concatenation [29, 9, 10, 30].
PL [15] introduced prototypical learning [21] into FSS. PANet [6] further
proposed a prototype alignment regularization. Recently, part-aware prototypes
have gained much attention as investigated in PPNet [16]. In contrast, feature
concatenation based methods first combine the query features and support
prototypes together and obtain support-conditioned query features as the input
of the subsequent segmentation head. For instance, PFENet [30] proposed
feature enrichment network for the combination of the query features and
support prototypes. But the main limitation of FSS methods is that the trained
model is only able to segment few novel classes. Apart from FSS, there are
also other related semantic segmentation tasks. Among them, Generalized Few-
shot Semantic Segmentation (GFSS) is the most challenging setting. For better
understanding, we show the differences among them in Table I.

TABLE I: Setting comparisons among Semantic Segmentation (SS), Few-shot Semantic Segmentation (FSS), Continual/Incremental Semantic Segmentation (CSS/ISS) and Generalized Few-shot Semantic Segmentation (GFSS). Setting | SS | FSS | CSS/ISS | GFSS
---|---|---|---|---
Training with base classes | $\surd$ | $\surd$ | $\surd$ | $\surd$
Training with novel classes | $\times$ | $\times$ | $\surd$ | $\times$
Sufficient samples per novel class | $\times$ | $\times$ | $\surd$ | $\times$
Test on base classes | $\surd$ | $\times$ | $\surd$ | $\surd$
Test on novel classes | $\times$ | $\surd$ | $\surd$ | $\surd$
Using base classes during testing | $\times$ | $\times$ | $\times$ | $\times$
Using novel classes during testing | $\times$ | $\surd$ | $\times$ | $\surd$

GFSS [13] generalizes FSS to segment both base and novel classes. DENet [13]
proposed a dynamic extension network which used a learning strategy to avoid
bias towards base classes. CAPL [14] proposed to combine base-class trained
classifier’s weights with novel ones obtained by averaging novel features.
However, their learned models are found to be poor on base classes [13] or
novel classes [14] (Table II). In this work, we propose a novel framework for
GFSS based on prediction score fusion, along with a Transformer based
prediction calibration module, maintaining the good performance on both base
and novel classes.

###  II-C Prediction Calibration

Prediction calibration is a common technique used in open-set recognition. It
relies on a threshold to classify the novel classes, i.e., its prediction
score from base classifier is lower than the threshold [31]. However, deep
models are usually overconfident to their inputs [32, 33], regardless of base
or novel classes. Therefore, threshold-based calibration methods are typically
poor at detecting novel classes. Different from the goal of prediction
calibration for open-set recognition, we propose a prediction calibration
network for GFSS using a novel cross-covariance based Transformer.

###  II-D Transformers

Recently, inspired by the great success of self-attention in NLP [34, 35],
Transformer-based architectures have been explored in vision tasks, such as
object detection [36, 37], semantic segmentation [38], and image
classification [39, 40]. Our calibration Transformer is inspired by the recent
cross-covariance based vision Transformer [17] with a shared goal for
efficiency but different applications and formulations.

##  III Methodology

###  III-A Problem setup

The ultimate goal of GFSS is to learn a model that can segment images of both
base and novel classes. A GFSS model needs to be trained in three stages. 1)
_Base classifier training_ : a base classifier is trained on the large amount
of labeled data $D_{base}=\\{x,y\\}_{i}$ of $N_{base}$ base classes
($C_{base}$). 2) _Novel classifier training_ : a novel classifier will be
trained on a support set (1-5 shots per class) from “fake” novel111The “fake”
novel classes are sampled from $C_{base}$ for episodic training./novel classes
($C_{novel}|C_{base}\cap C_{novel}=\emptyset$) during training/testing. 3)
_Base + novel class fusion_ : the trained base and novel classifiers are fused
and tested on a test set $D_{test}$ including images of both base and novel
classes. Note that samples from base and novel classes do not co-exist in any
stages.

###  III-B Overview

Let us denote the base and the novel models as $G_{base}\circ F_{base}$ and
$G_{novel}\circ F_{novel}$ respectively, each comprising a feature extractor
and a classifier. We assume the base and novel classifiers share the same
feature extractor, i.e., $F_{base}=F_{novel}=F$. Note that $F$ extracts multi-
level fused features from Pyramid Pooling Module (PPM) of PSPNet [2] (our
backbone model). Besides the main task models, we also have a Normalized Score
Fusion (NSF) module and a Transformer-based calibration module denoted as
$\mathcal{T}$, which is episodically trained on $D_{base}$. The overall
pipeline of our PCN model is shown in Figure 2.

###  III-C Base Classifier Training

We first train our model $G_{base}\circ F_{base}$ on the base dataset
$D_{base}$. The feature extractor $F$ and classifier $G_{base}$ are optimized
using the cross entropy loss as follows:

$\underset{F,G_{base}}{\arg\min}\quad\frac{1}{|D_{base}|}\sum_{x,y\sim
D_{base}}\frac{1}{H\times W}\sum_{i=1}^{H\times
W}\ell_{ce}(y_{i},\hat{y}_{i}),\\\ $ (1)

where $\hat{y}=S(G_{base}\circ F(x))$ and $(H,W)$ is the shape of input $x$
and mask $y$. $S$ is the softmax function. After training, both the feature
extractor and the base classifier are frozen in the next stages.

###  III-D Support-set based Novel Classifier Training

With a pre-trained feature extractor $F$, we train a novel classifier
$G_{novel}$ using a support set $D_{novel}$ with just a few samples per novel
class. The optimization objective is as follows,

$\underset{G_{novel}}{\arg\min}\quad\frac{1}{|D_{novel}|}\sum_{x,y\sim
D_{novel}}\frac{1}{H\times W}\sum_{i=1}^{H\times
W}\ell_{ce}(y_{i},\hat{y}_{i}),$ (2)

where $\hat{y}=S(G_{novel}\circ F(x))$. Once trained, the novel classifier
needs to be fused with the base classifier to segment images of both base and
novel classes.

###  III-E Prediction Calibration

Existing approach to GFSS is based on parameter fusion. The prediction is
formulated as

$\hat{y}=S((G_{base}\cup G_{novel})\circ F(x)).$

However, with much more base classes and training samples, the fused
classifier will be biased towards the base classes. Instead of parameter-level
fusion, we adopt a score-level fusion approach, i.e., for each pixel, the two
classifiers generate two sets of scores separately. These two sets of scores
are then fused with calibration to avoid bias towards either the base or novel
classes. To that end, two modules are applied sequentially: Normalized Score
Fusion (NSF) and Transformer-based prediction calibration.

#### Normalized Score Fusion

With NSF, we normalize the predicted scores of two classifiers separately to
promote the saliency of the novel classes as

$\hat{y}_{nsf}=S(G_{base}\circ F(x))\cup S(G_{novel}\circ F(x)).$ (3)

Empirically, we find that this simple technique mitigates the prediction bias
towards base classes and already yields the state of the art performance.
However, the prediction tends to be overconfident on novel classes as shown in
Figure 1.

#### Transformer-based prediction calibration

In order to further improve the prediction calibration, we propose a feature-
score cross-covariance based Transformer $\mathcal{T}$ for prediction
calibration. Concretely, as per standard, our Transformer also has three
projection heads: Query head $\Omega:\mathbb{R}^{c\times
hw}\rightarrow\mathbb{R}^{c\times d}$, Key head $\Gamma:\mathbb{R}^{m\times
hw}\rightarrow\mathbb{R}^{m\times d}$ and Value head
$\Lambda:\mathbb{R}^{m\times hw}\rightarrow\mathbb{R}^{m\times d}$ ($c$: the
class number, $m$: feature dimension, $d$: projected dimension, $h,w$: spatial
dimension). Different from the conventional design, which treats each
pixel/patch embedding as a token, we instead treat each score and feature
channel as our token and compute the cross covariance between them as

$\sigma=\Omega(\hat{y})\cdot(\Gamma(F(x)))^{T},\sigma\in\mathbb{R}^{c\times
m}.$ (4)

This significantly reduces the Transformer computational cost and makes it
tractable for our pixel-level global attention task. Then, the value embedding
is transformed as

$v=\frac{S(\sigma)}{\sqrt{d}}\cdot\Lambda(F(x)).$ (5)

After that, a calibration head $\Delta:\mathbb{R}^{c\times
d}\rightarrow\mathbb{R}^{c\times hw}$ is followed to predict the prediction
offset to calibrate the vanilla model predictions as,

$\displaystyle\hat{y}_{\Delta}$ $\displaystyle=\Delta(v)$ (6)
$\displaystyle\hat{y}_{calib}$ $\displaystyle=\hat{y}_{nsf}+\hat{y}_{\Delta}.$

#### Episodic Training

As we have no access to the unseen novel classes during training, to simulate
the testing scenarios, in each episode we randomly sample $N_{fn}$ “fake”
novel classes ($C_{fn}$) from base dataset $D_{base}$ as per [27, 28]. For
each “fake” novel class, we sample $K$ support images as well as $Q$ query
images. We sample the same number of query images from the remaining $N_{rb}$
base classes ($C_{rb}$) to balance the training ($N_{fn}+N_{rb}=N_{base}$, and
$C_{fn}\cap C_{rb}=\emptyset$ ). The sampled support and query sets can be
denoted as:

$\displaystyle\mathcal{S}=\\{(x_{i},y_{i})~{}|~{}y_{i}\in
C_{fn}\\}^{N_{fn}K}_{i=1}$ (7)
$\displaystyle\mathcal{Q}=\\{(x_{i},y_{i})~{}|~{}y_{i}\in
C_{fn}\\}^{N_{fn}Q}_{i=1}\ \cup$ $\displaystyle\ \ \ \ \ \ \
\\{(x_{i},y_{i})~{}|~{}y_{i}\in C_{rb}\\}^{N_{fn}Q}_{i=1}.$

In each episode, we first activate the classifier weights $G_{rb}$ for the
real base classes from the pre-trained base classifier $G_{base}$. Meanwhile,
we use the support set of the novel classes to train a “fake” novel classifier
as

$\displaystyle G_{fn}$
$\displaystyle=\underset{G_{fn}}{\arg\min}\quad\mathcal{L}_{ce}(y_{i},\mathcal{S}).$
(8)

Then, we sample the “fake” test data from the query set $\mathcal{Q}$ and
optimize our proposed calibration module. Given its NSF predictions
$\hat{y}_{fnsf}$ of the real base and “fake” novel classifiers $G_{rb}$ and
$G_{fn}$, we can get the calibrated predictions $\hat{y}_{fcalib}$ through our
Transformer-based calibration module as
$\hat{y}_{fcalib}=\mathcal{T}_{(\Omega,\Gamma,\Lambda,\Delta)}(F(x),\hat{y}_{fnsf})$.
Then, we can compute the cross entropy loss per pixel and optimize the
calibration module as

$\displaystyle\underset{\Omega,\Gamma,\Lambda,\Delta}{\arg\min}\quad\frac{1}{|\mathcal{Q}|}\sum_{x,y\sim\mathcal{Q}}\frac{1}{H\times
W}\sum_{i=1}^{H\times W}\ell_{ce}(y_{i},\hat{y}_{fcalib}).$ (9)

The overall training algorithm could be found in Algorithm 1.

Data: Support set $\mathcal{S}$ and query set $\mathcal{Q}$ from $C_{base}$;

Initialize Transformer module $\mathcal{T}$ with ${\theta}_{t}$;

Set learning rate $\alpha_{t}$ for $\mathcal{T}$;

for  _each iteration_ do

Activate $G_{rb}$ with $G_{base}$;

Obtain $\\{(x_{i}^{S},y_{i}^{S})|y_{i}^{S}\in C_{fn}\\}$ from $\mathcal{S}$;

Initialize a novel classifier $G_{fn}$ with ${\theta}_{n}$;

Set learning rate $\alpha_{n}$ for $G_{fn}$;

while  _train a novel classifier_ do

Predict $\hat{y}_{i}^{S}$ by $G_{fn}\circ F(x_{i}^{S})$;

Compute the loss $\mathcal{L}_{n}$ via Ep. 8;

Update $G_{fn}$ by
${\theta}_{n}\leftarrow{\theta}_{n}-\alpha_{n}{\bigtriangledown}_{{\theta}_{n}}\mathcal{L}_{n}$;

end while

Obtain $\\{(x^{Q},y^{Q})|y^{Q}\in C_{fn}\cup C_{rb}\\}$ from $\mathcal{Q}$;

Predict $\hat{y}_{rb}^{Q}$ by $G_{rb}\circ F(x^{Q})$;

Predict $\hat{y}_{fn}^{Q}$ by $G_{fn}\circ F(x^{Q})$;

Normalized Score Fusion via Ep. 3 $\rightarrow$ $\hat{y}_{fnsf}^{Q}$;

Compute the cross covariance $\sigma$ via Ep. 4;

Value transformation via Ep. 5;

Compute prediction offset via Ep. 6 ;

Obtain calibrated prediction score: $\hat{y}_{fcalib}^{Q}$;

Compute the loss $\mathcal{L}_{t}$ via Ep. 9;

Update $\mathcal{T}$ by
$\theta_{t}\leftarrow\theta_{t}-\alpha_{t}{\bigtriangledown}_{{\theta}_{t}}\mathcal{L}_{t}$;

end for

Algorithm 1 Episodic Training for PCN

###  III-F Inference

Given a pre-trained base model $G_{base}\circ F$, a support set trained novel
model $G_{novel}\circ F$ and a learned calibration module $\mathcal{T}$, our
model can now segment any unseen data from $D_{test}$ with both the base and
novel classes. The overall inference formulation is

$\displaystyle\hat{y}=\mathcal{T}(F(x),\hat{y}_{nsf}),x\sim D_{test},$ (10)

where $\hat{y}_{nsf}$ is computed using Eq. 3.

##  IV Experiments

###  IV-A Datasets

We conduct experiments on two widely used datasets in FSS, i.e.,
PASCAL-$5^{i}$ and COCO-$20^{i}$. PASCAL-$5^{i}$ is the PASCAL-VOC 2012
dataset [41] with augmented annotations from [42], and has 5,953/1,449 images
in train/val set and 20 classes in each set. As per [5], 4 splits
$i\in\\{0,1,2,3\\}$ are separated with 5 classes in each one for cross-
validation. In each experiment, we use 3 splits as the base class training set
without having access to another split from novel classes. During testing on
the val set, the classes unseen in training construct the novel dataset while
seen classes are used as the base query. In total, four experiments are
conducted on this dataset. COCO-$20^{i}$ is a relatively larger dataset, with
82,081/40,137 images in 80 classes in the train/val set respectively.
Following [7], it was split into four folds. Each fold has 20 classes.
Experiments are conducted in a cross-validation manner as PASCAL-$5^{i}$.

TABLE II: Generalized few-shot semantic segmentation results on
PASCAL-$5^{i}$. The results of Base Classifier show the upper bound
performance on base classes. NSF: normalized score fusion only.

Backbone Method Shot s-0 s-1 s-2 s-3 Average Base Novel Base Novel Base Novel
Base Novel Base Novel mIoU $H_{mean}$ ResNet-50 Base Classifier - 63.15 -
64.17 - 63.87 - 65.47 - 64.17 - - - DENet (MM20) 1 29.91 56.66 18.20 65.68
20.75 60.48 22.45 47.02 22.83 57.46 31.49 32.67 PFENet (PAMI20) 58.24 13.00
59.14 9.99 54.87 6.08 58.61 5.13 57.72 8.55 45.42 14.89 ACASTLE (IJCV21) 59.46
34.82 55.12 37.64 59.69 40.45 61.09 32.08 58.84 36.25 53.19 44.86 CAPL
(CVPR22) - - - - - - - - 65.48 18.85 54.38 29.16 NSF (Ours) 29.38 51.86 36.61
55.41 44.73 50.63 42.30 36.93 38.26 48.71 40.87 42.85 PCN (Ours) 59.43 47.98
61.31 51.27 61.44 51.25 62.93 41.34 61.28 47.96 57.95 53.81 DENet (MM20) 5
30.15 58.04 18.58 66.10 21.21 61.00 22.84 47.73 23.20 58.22 31.95 33.17 PFENet
(PAMI20) 57.81 13.38 58.96 9.80 54.72 6.17 58.68 5.01 57.54 8.59 45.30 14.95
ACASTLE (IJCV21) 60.56 45.64 62.87 53.87 62.91 42.41 63.59 42.68 62.48 46.15
58.40 53.09 CAPL (CVPR22) - - - - - - - - 66.14 22.41 55.72 33.48 NSF (Ours)
29.90 59.07 44.99 64.01 49.60 62.74 48.25 48.36 43.19 58.55 47.03 49.71 PCN
(Ours) 59.05 53.03 60.59 58.03 60.60 61.67 63.15 51.60 60.85 56.08 59.66 58.37
ResNet-101 Base Classifier - 67.42 - 66.08 - 64.93 - 67.70 - 66.53 - - -
PFENet (PAMI20) 1 58.64 20.42 59.72 6.00 56.85 12.62 55.06 13.31 57.57 13.09
46.45 21.33 ACASTLE (IJCV21) 62.30 39.61 56.60 38.12 59.43 32.17 55.84 30.34
58.54 35.06 52.67 43.86 NSF (Ours) 27.65 52.64 34.42 61.93 37.75 58.72 41.63
41.56 35.36 53.71 39.95 42.65 PCN (Ours) 60.30 51.38 60.04 44.03 61.16 39.91
61.29 33.09 60.70 42.10 56.05 49.72 PFENet (PAMI20) 5 58.56 20.65 59.70 6.48
57.02 12.20 54.71 13.17 57.50 13.13 46.40 21.37 ACASTLE (IJCV21) 62.98 51.75
63.26 47.61 64.04 41.07 62.82 36.28 63.28 44.18 58.50 52.03 NSF (Ours) 27.41
59.97 41.54 67.39 43.44 65.69 47.96 50.47 40.09 60.88 45.29 48.34 PCN (Ours)
59.10 56.35 59.18 49.21 60.25 48.41 60.88 43.47 59.85 49.36 57.23 54.10

TABLE III: Generalized few-shot semantic segmentation results on
COCO-$20^{i}$. The results of Base Classifier show the upper bound performance
on base classes. NSF: normalized score fusion only.

Backbone Method Shot s-0 s-1 s-2 s-3 Average Base Novel Base Novel Base Novel
Base Novel Base Novel mIoU $H_{mean}$ ResNet-50 Base Classifier - 44.29 -
34.14 - 36.29 - 37.47 - 38.05 - - - DENet (MM20) 1 11.79 37.37 10.42 38.23
12.82 33.68 11.33 34.67 11.59 35.99 17.69 17.53 PFENet (PAMI20) 39.88 4.22
28.40 6.17 31.23 3.88 31.75 2.13 32.82 4.10 25.64 7.29 ACASTLE (IJCV21) 23.42
25.57 21.93 21.64 25.17 20.40 25.90 18.43 24.11 21.51 23.46 22.73 CAPL
(CVPR22) - - - - - - - - 44.61 7.05 35.46 12.18 NSF (Ours) 30.99 32.26 22.85
32.12 25.47 29.32 27.59 23.69 26.73 29.35 27.38 27.97 PCN (Ours) 39.83 31.93
33.60 28.63 35.33 27.53 36.51 29.39 36.32 29.37 34.58 32.48 DENet (MM20) 5
12.30 38.78 10.81 39.73 13.13 34.77 11.43 35.10 11.92 37.10 18.21 18.04 PFENet
(PAMI20) 39.83 4.38 28.18 6.00 31.20 3.77 31.49 2.23 32.68 4.10 25.53 7.28
ACASTLE (IJCV21) 42.03 23.78 22.32 26.11 35.22 17.52 35.60 23.63 33.79 22.76
31.03 27.20 CAPL (CVPR22) - - - - - - - - 45.24 11.05 36.80 17.76 NSF (Ours)
33.44 41.78 24.22 39.85 26.43 36.12 29.61 36.03 28.43 38.45 30.93 32.68 PCN
(Ours) 40.16 40.82 33.42 35.76 34.86 33.04 36.31 38.36 36.19 37.00 36.39 36.59
ResNet-101 Base Classifier - 37.49 - 41.72 - 37.28 - 41.85 - 39.59 - - -
PFENet (PAMI20) 1 34.44 2.81 38.35 2.55 34.83 5.07 38.38 3.90 36.50 3.58 28.27
6.52 ACASTLE (IJCV21) 21.70 25.90 28.53 23.57 20.21 18.75 29.83 21.99 25.07
22.55 24.44 23.74 NSF (Ours) 25.08 32.66 31.74 33.33 27.70 30.50 32.49 30.32
29.25 31.70 29.87 30.43 PCN (Ours) 35.26 27.54 39.08 30.32 36.07 24.66 39.46
26.37 37.47 27.22 34.91 31.53 PFENet (PAMI20) 5 34.52 2.53 38.32 2.76 34.69
6.09 38.72 3.64 36.56 3.76 28.36 6.81 ACASTLE (IJCV21) 35.33 20.72 41.22 23.75
34.01 22.77 40.97 19.34 37.88 21.65 33.82 27.55 NSF (Ours) 27.18 40.74 34.24
41.94 29.95 37.16 34.69 39.23 31.52 39.77 33.58 35.16 PCN (Ours) 35.09 35.84
40.05 38.11 35.75 31.01 40.50 33.73 37.85 34.67 37.05 36.19

###  IV-B Implementation Details

In stage one, we train the feature extraction network $F$ and the base
classifier $G_{base}$ on the base dataset. We adopt PSPNet [2] as our feature
extraction backbone with two versions, i.e., ResNet-50 and ResNet-101. The
training epochs are 100 and 20 for PASCAL-$5^{i}$ and COCO-$20^{i}$,
respectively. The batch size is 12 and the input image size is 417. The latent
feature map’s spatial dimension $h=w=53$. We use momentum SGD to train our
model. The initial learning rate is $2.5e-3$, decayed by a cosine learning
rate scheduler during training.

In the second episodic-training stage, in each episode we first use SGD
optimizer to train a novel class classifier $G_{novel}$ (a 3 $\times$ 3 Conv
layer $\rightarrow$ a ReLU activation $\rightarrow$ a 1 $\times$ 1 Conv layer)
on a support set with 50 iterations. The learning rate is $1e-1$ and fixed.
Weighted cross-entropy is used to stabilize the training. After that, we meta-
learn our PCN within the same episode. We train our models for 10,000/30,000
iterations with batch size 8 on PASCAL-$5^{i}$/COCO-$20^{i}$. The initial
learning rate for PCN is set to $1e-2$, which is also decayed by a cosine
learning rate scheduler. We set $N_{fn}=1$ and $Q=1$ during training.

###  IV-C Evaluation Metrics

According to generalized few shot recognition setting [27, 28], we adopt
harmonic mean Intersection over Union (mIoU) to evaluate our model.
Specifically, in each testing task, we sample $N_{base}$222The total number of
base classes, e.g., 15/60 for PASCAL-$5^{i}$/COCO-$20^{i}$. pairs of query
images. Each pair has two images, one from novel class and the other one from
the $j^{th}\in\\{1,\dots,N_{base}\\}$ base class. We separately compute the
mIoU over the base classes ($mIoU_{base}$) and novel classes ($mIoU_{novel}$)
and then compute their harmonic mean ($H_{mean}$) by:

$H_{mean}=\frac{2\cdot mIoU_{base}\cdot
mIoU_{novel}}{mIoU_{base}+mIoU_{novel}}.$ (11)

We also report mIoU over all classes ($mIoU$). On each testing split, we
evaluate our model with 1,000 tasks for both PASCAL-$5^{i}$ and COCO-$20^{i}$.

###  IV-D Compared Methods

We compare our method with DENet [13], CAPL [14], PFENet [30] and ACASTLE
[28]. DENet and CAPL are two state-of-the-art GFSS methods. PFENet is a state-
of-the-art FSS model which is naturally extendable to GFSS. ACASTLE is a
state-of-the-art generalized few-shot recognition model, which can also be re-
purposed for GFSS. All results are produced based on their officially released
codes except CAPL [14] that does not release code (citing results from their
paper). In addition, a baseline using normalized score fusion only is also
compared.

###  IV-E PASCAL-$5^{i}$ Results

Figure 3: Qualitative results comparing our method against DENet, PFENet,
ACASTLE and Normalized Score Fusion only (NSF). Setting: PASCAL-50 in the
1-shot case with ResNet-50 backbone for feature extraction. Each row contains
an inference task with one novel class to be added to the segmentation.

The results on PASCAL-$5^{i}$ for 1-shot/5-shot case with ResNet-50/ResNet-101
are shown in Table II. Overall, our method achieves a new state-of-the-art
performance for all cases, with significant improvements over other compared
methods. Specifically, our method surpasses the best competitor by 8.95%/5.86%
in 1-shot case and 5.28%/2.07% in 5-shot case, with ResNet-50/ResNet-101
respectively. A number of other observations can be made. First, the state-of-
the-art FSS method PFENet [30] has poor performance on novel classes when used
for GFSS. The potential reason is that PFENet is learned on base classes and
tested on novel classes only for FSS. When the base and novel classes are
tested together for GFSS, PFENet is more confident on the base classes it was
trained on and less confident on the unseen novel classes, yielding poor
performance on novel classes. Second, other baselines including our NSF
usually have an unbalanced performance. For example, DENet [13] generally has
a much higher mIoU on novel classes while performs poorly on base classes. In
contrast, our PCN has a much more stable and balanced performance on both base
and novel classes in most cases. Third, except DENet and PFENet, all other
methods benefit from more examples in the support set, cf. 1 shot vs 5 shot,
on novel classes . At last, our PCN also wins in most cases with the mIoU over
all classes. Some qualitative results can be found in Figure 3. Overall, they
are consistent with the quantitative observations. For example, PFENet
perfectly segments all objects of base classes in the $1^{st},5^{th}$ and
$6^{th}$ rows while mis-segments the novel class Bike/Airplane/Bottle/Boat in
the $2^{nd}/4^{th}/5^{th}/6^{th}$ row. In general, we can see our method can
segment both base and novel classes more accurately in different scenarios.

TABLE IV: Comparisons among score fusion, Normalized Parameter Fusion (NPF) and our NSF. Dataset: PASCAL-50. Backbone: ResNet-50. Method | Shot | s-0
---|---|---
Base | Novel | mIoU | $H_{mean}$
Score Fusion | 1 | 59.37 | 16.74 | 48.71 | 26.12
NPF | 62.81 | 16.00 | 51.11 | 25.51
NSF | 29.38 | 51.86 | 35.00 | 37.51
Figure 4: Visualization of the class prediction heatmap from base classifier
and novel classifier before and after calibration. Note that for $y_{base}$ we
only show one base class heatmap when multiple base classes exist. The main
purpose of this figure is to show how the novel classifier’s overconfidence on
base classes is calibrated, which can be observed by comparing the $4^{th}$
and the last column.

###  IV-F COCO-$20^{i}$ Results

Table III shows the comparative results on COCO-20i. Interestingly, our method
is more superior over other baselines in this more challenging setup.
Specifically, our method achieves the overall best performance on base class
mIoU, all class mIoU and $H_{mean}$. Apart from the similar observations on
PASCAL-$5^{i}$, there are some new findings. (1) NSF only already outperforms
all other competitors. This indicates that our proposed NSF is a promising
strategy for mitigating the prediction bias. (2) Our PCN can further boost the
performance by addressing novel classifier’s overconfidence problem. (3) We
also notice that the performance gap is less significant with ResNet-101 as
feature extraction backbone. The possible reason is that the stronger backbone
ResNet-101 might have better intrinsic novel class generalization ability,
thus resulting in less improvements brought by our Transformer.

###  IV-G Further Analysis

#### The Effectiveness of Normalized Score Fusion

We also conduct experiments to analyze the effectiveness of our proposed
normalized score fusion. From the results in Table IV, we can see that without
normalization the naive score fusion can only achieve $26.12$ $H_{mean}$,
showing its poor prediction calibration. We also experiment with an
alternative Normalized Parameter Fusion (NPF), which normalizes the fused
classifier parameters (weights) on feature channel before calculating the
scores. However, we can see that this alternative performs as bad as the naive
score fusion. Our normalized score fusion achieves the best among them with
over $12.0$ $H_{mean}$ improvement. We visualize the results of NPF and NSF in
Figure 5, which suggests that NPF performs poorly on novel classes while NSF
are worse at base classes.

Figure 5: Visual comparisons between Normalized Parameter Fusion (NPF) and
Normalized Score Fusion (NSF). From the results, we can see that NPF often
mis-segments novel classes into base classes or does not detect them (in the
$3^{rd}$ row) while NSF tends to segment base classes into novel ones.

#### The Effectiveness of Our Transformer $\mathcal{T}$

In this experiment, we first incorporate our proposed Transformer
$\mathcal{T}$ with score fusion, NPF and our NSF. From the results in Table V,
we can see that it improves them noticeably with $\sim 18.0$, $\sim 17.0$ and
$\sim 16.0$ $H_{mean}$ margins compared with Table IV. This demonstrates both
the efficacy and generality of our proposed Transformer module.

Furthermore, we investigate the effectiveness of $\mathcal{T}$ on existing
methods as shown in Table VI. Interestingly, the proposed $\mathcal{T}$ can
always improve their performance on either base class or novel class so as to
achieve the better overall mIoU and $H_{mIoU}$. For the method that has the
largest unbalanced performance gap, i.e., PFENet, $\mathcal{T}$ boosts its
performance considerably by 22.11% on the harmonic mIoU.

TABLE V: The effectiveness of our proposed Transformer. Dataset: PASCAL-50. Backbone: ResNet-50. Method | Shot | s-0
---|---|---
Base | Novel | mIoU | $H_{mean}$
Score Fusion + $\mathcal{T}$ | 1 | 59.31 | 35.68 | 53.40 | 44.55
NPF + $\mathcal{T}$ | 46.85 | 39.21 | 44.94 | 42.69
NSF + $\mathcal{T}$ | 59.43 | 47.98 | 56.57 | 53.09
TABLE VI: Evaluation for previous methods equipped with the proposed Transformer $T$. Dataset: PASCAL-$5^{0}$. Backbone: ResNet-50. CAPL [29] did not release source code, so was not compared. Method | Shot | s-0
---|---|---
Base | Novel | mIoU | $H_{mean}$
DENet | 1 | 29.91 | 56.66 | 36.60 | 39.15
DENet + $T$ | 39.54 | 52.98 | 42.90 | 45.28
PFENet | 58.24 | 13.00 | 46.93 | 21.26
PFENet + $T$ | 51.35 | 37.44 | 48.01 | 43.37
ACASTLE | 59.46 | 34.82 | 53.30 | 43.92
ACASTLE + $T$ | 60.02 | 40.04 | 55.03 | 48.04

#### Qualitative Results of Calibration

The heatmaps of the class predictions from base classifier and novel
classifier are visualized in Figure 4. The middle two and rightmost two are
generated from NSF and our full model respectively. We can see that the novel
classifier has high confidence on base classes, e.g., Person/Horse/Sheep/Dog
in the $4^{th}$ column. This will result in mis-segmentation of those base
classes into novel classes. After the calibration by our cross-covariance
Transformer, the novel classifier’s overconfidence on base classes is clearly
depressed.

#### Ablation Study of Our Transformer $\mathcal{T}$

In this experiment, we compare our cross-covariance Transformer with other
calibration variants that are linear layer based and self-attention based.
Specifically, linear layer based calibration replaces our cross-covariance
Transformer with one fully-connected layer; while the self-attention based
calibration uses [$y_{all}$, $y_{all}$, $y_{all}$] as inputs for self-
attention. The comparison is shown in Table VII. From the results we can see
that linear layer based variants are not comparable to NSF, indicating the
difficulty in calibrating the prediction with a simple fully-connected layer.
Meanwhile, we can see the residual connection is helpful in prediction
calibration. Self attention also works well in this setup showing a
Transformer architecture is beneficial. But it is much inferior to our
proposed cross-covariance Transformer, showing the prediction and feature
channel covariance is helpful in the prediction calibration in GFSS.

Figure 6: Failure cases. Dataset: PASCAL-$5^{0}$. Backbone: ResNet-$50$. TABLE VII: Ablation study of our proposed Transformer. Dataset: PASCAL-50. Backbone: ResNet-50. ${\dagger}$ means without residual connection. Calibration method | Shot | s-0
---|---|---
Base | Novel | mIoU | $H_{mean}$
NSF | 1 | 29.38 | 51.86 | 35.00 | 37.51
Linear layer† | 19.78 | 15.71 | 18.76 | 17.51
Linear layer | 22.53 | 18.27 | 21.47 | 20.18
Self-attention | 61.96 | 35.54 | 55.36 | 45.17
PCN (Ours) | 59.43 | 47.98 | 56.57 | 53.09

#### Comparisons among Varying-level Features Used for Cross-attention

Table VIII shows the results using varying-level features for cross-attention.
Note that the high-level feature is produced by Pyramid Pooling Module (PPM)
(bins [1,2,3,6]) from PSPNet [2] and concatenation without combining low-level
features. The first observation is that in most cases low-level features work
better than high-level features. This is reasonable as low-level features
contain more detailed information useful for segmentation. Second, the
combined features by both low-level and high-level features outperform all
other variants in that it benefits from both detailed structure and abstract
semantics.

TABLE VIII: Comparisons among varying-level features used for cross-attention.
Dataset: PASCAL-50. Backbone: ResNet-50.

Feature Shot s-0 Base Novel mIoU $H_{mean}$ $Layer_{2}$ (Low-level) 1 60.80
40.29 55.68 48.47 $Layer_{3}$ (Low-level) 60.94 37.42 55.06 46.37 $Layer_{4}$
(Low-level) 57.28 46.46 54.57 51.30 High-level 61.51 39.57 56.03 48.16
$Layer_{4}$ \+ High-level (Ours) 59.43 47.98 56.57 53.09

#### Comparisons of Computational Time

Table IX shows the comparisons on training time and memory cost. It seems the
proposed method has comparable training time and memory cost which are merely
inferior to the best competitors, but a much better $H_{mIoU}$ performance
with 9.17% gain. Note that after training, there are no significant
differences of the inference time of all compared methods.

TABLE IX: Comparing computational time and memory cost. Dataset: PASCAL-$5^{0}$. Backbone: ResNet-$50$. Batch size: $8$. ${\dagger}$: exclude the base classifier pre-training time. The best results are in bold while the second best is underlined. Results were obtained on one single NVIDIA GeForce RTX $2080$ Ti GPU. Method | Shot | s-0
---|---|---
Training time† $\downarrow$ | Memory cost $\downarrow$ | $H_{mIoU}\uparrow$
DENet | 1 | 31.4h | 4.2G | 39.15
PFENet | 70.5h | 8.6G | 21.26
ACASTLE | 4.5h | 5.7G | 43.92
PCN (Ours) | 6.8h | 4.7G | 53.09

#### Failure Cases

Figure 6 shows some failure cases. This can help better understand the
limitation of the proposed method and offer insights on future works.
Specifically, we observed that failure mainly occurs in three scenarios.
First, when only part of the object is shown in the figure, e.g., $Chair$ in
the $1^{st}$ row, the proposed PCN tends to mis-segment that part to a class
with a similar appearance. Second, when huge intra-class variances exist
between support and query sample as shown in $2^{nd}$ and $4^{th}$ rows, the
target objects are often missed. This is because the large intra-class
variance makes the model hard to recognize them as the same class. Finally,
the ambiguous annotations cause wrong segmentations. For example, the $Sofa$
in $3^{rd}$ row is visually more similar to the $Chair$ objects so that it is
reasonable for PCN to segment it to $Chair$ class. For the first two factors,
we can solve them by modeling the relationship between the part and the
global, and closing the representations within the same class. In contrast,
the ambiguous annotations are needed to rectify.

##  V Conclusion

We have proposed a novel Prediction Calibration Networks (PCN) for generalized
few-shot semantic segmentation. It consists of two key modules: Normalized
Score Fusion (NSF) and Transformer based prediction calibration. NSF alone
serves as a strong baseline that can mitigate the prediction bias towards base
classes, but is prone to novel class bias. A new feature-score cross-
covariance based Transformer is proposed to further calibrate prediction score
and episodically learn to avoid any bias. Extensive experiments on two
standard benchmarks show our proposed framework achieves the state-of-the-art
performance. We also conducted comprehensive further analysis for better
understanding how each component contributes to the final performance.

## References

  * [1] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs,” _TPAMI_ , 2017.
  * [2] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, “Pyramid scene parsing network,” in _CVPR_ , 2017.
  * [3] L.-C. Chen, Y. Zhu, G. Papandreou, F. Schroff, and H. Adam, “Encoder-decoder with atrous separable convolution for semantic image segmentation,” in _ECCV_ , 2018.
  * [4] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, “Microsoft COCO: Common objects in context,” in _ECCV_ , 2014.
  * [5] A. Shaban, S. Bansal, Z. Liu, I. Essa, and B. Boots, “One-shot learning for semantic segmentation,” in _BMVC_ , 2017.
  * [6] K. Wang, J. H. Liew, Y. Zou, D. Zhou, and J. Feng, “PANet: Few-shot image semantic segmentation with prototype alignment,” in _ICCV_ , 2019.
  * [7] K. Nguyen and S. Todorovic, “Feature weighting and boosting for few-shot segmentation,” in _ICCV_ , 2019.
  * [8] H. Wang, X. Zhang, Y. Hu, Y. Yang, X. Cao, and X. Zhen, “Few-shot semantic segmentation with democratic attention networks,” in _ECCV_ , 2020.
  * [9] B. Yang, C. Liu, B. Li, J. Jiao, and Q. Ye, “Prototype mixture models for few-shot semantic segmentation,” in _ECCV_ , 2020.
  * [10] C. Zhang, G. Lin, F. Liu, R. Yao, and C. Shen, “CANet: Class-agnostic segmentation networks with iterative refinement and attentive few-shot learning,” in _CVPR_ , 2019.
  * [11] X. Zhang, Y. Wei, Y. Yang, and T. S. Huang, “SG-one: Similarity guidance network for one-shot semantic segmentation,” _TC_ , 2020.
  * [12] Z. Lu, S. He, X. Zhu, L. Zhang, Y.-Z. Song, and T. Xiang, “Simpler is better: few-shot semantic segmentation with classifier weight transformer,” in _ICCV_ , 2021.
  * [13] L. Liu, J. Cao, M. Liu, Y. Guo, Q. Chen, and M. Tan, “Dynamic extension nets for few-shot semantic segmentation,” in _MM_ , 2020.
  * [14] Z. Tian, X. Lai, L. Jiang, M. Shu, H. Zhao, and J. Jia, “Generalized few-shot semantic segmentation,” in _CVPR_ , 2022.
  * [15] N. Dong and E. P. Xing, “Few-shot semantic segmentation with prototype learning.” in _BMVC_ , 2018.
  * [16] Y. Liu, X. Zhang, S. Zhang, and X. He, “Part-aware prototype network for few-shot semantic segmentation,” in _ECCV_ , 2020.
  * [17] A. El-Nouby, H. Touvron, M. Caron, P. Bojanowski, M. Douze, A. Joulin, I. Laptev, N. Neverova, G. Synnaeve, J. Verbeek _et al._ , “XCiT: Cross-covariance image transformers,” _arXiv preprint arXiv:2106.09681_ , 2021.
  * [18] Q. Cai, Y. Pan, T. Yao, C. Yan, and T. Mei, “Memory matching networks for one-shot image recognition,” in _CVPR_ , 2018.
  * [19] C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” in _ICML_ , 2017.
  * [20] A. A. Rusu, D. Rao, J. Sygnowski, O. Vinyals, R. Pascanu, S. Osindero, and R. Hadsell, “Meta-learning with latent embedding optimization,” in _ICLR_ , 2019.
  * [21] J. Snell, K. Swersky, and R. S. Zemel, “Prototypical networks for few-shot learning,” in _NeurIPS_ , 2017.
  * [22] F. Sung, Y. Yang, L. Zhang, T. Xiang, P. H. Torr, and T. M. Hospedales, “Learning to compare: Relation network for few-shot learning,” in _CVPR_ , 2018.
  * [23] O. Vinyals, C. Blundell, T. Lillicrap, K. Kavukcuoglu, and D. Wierstra, “Matching networks for one shot learning,” in _NeurIPS_ , 2016.
  * [24] C. Zhang, Y. Cai, G. Lin, and C. Shen, “DeepEMD: Few-shot image classification with differentiable earth mover’s distance and structured classifiers,” in _CVPR_ , 2020.
  * [25] H. Qi, M. Brown, and D. G. Lowe, “Low-shot learning with imprinted weights,” in _CVPR_ , 2018.
  * [26] B. Hariharan and R. Girshick, “Low-shot visual recognition by shrinking and hallucinating features,” in _ICCV_ , 2017.
  * [27] S. Gidaris and N. Komodakis, “Dynamic few-shot visual learning without forgetting,” in _CVPR_ , 2018.
  * [28] H.-J. Ye, H. Hu, and D.-C. Zhan, “Learning adaptive classifiers synthesis for generalized few-shot learning,” _IJCV_ , 2021.
  * [29] X. Cao and X. Zhen, “Few-shot semantic segmentation with democratic attention networks,” in _ECCV_ , 2020.
  * [30] Z. Tian, H. Zhao, M. Shu, Z. Yang, R. Li, and J. Jia, “Prior guided feature enrichment network for few-shot segmentation,” _TPAMI_ , 2020.
  * [31] D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-of-distribution examples in neural networks,” in _ICLR_ , 2017.
  * [32] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in _ICML_ , 2017.
  * [33] M. Hein, M. Andriushchenko, and J. Bitterwolf, “Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem,” in _CVPR_ , 2019.
  * [34] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in _NeurIPS_ , 2017.
  * [35] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in _NAACL-HLT_ , 2019\.
  * [36] H. Hu, J. Gu, Z. Zhang, J. Dai, and Y. Wei, “Relation networks for object detection,” in _CVPR_ , 2018.
  * [37] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in _ECCV_ , 2020.
  * [38] S. Zheng, J. Lu, H. Zhao, X. Zhu, Z. Luo, Y. Wang, Y. Fu, J. Feng, T. Xiang, P. H. Torr _et al._ , “Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers,” in _CVPR_ , 2021.
  * [39] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly _et al._ , “An image is worth 16x16 words: Transformers for image recognition at scale,” in _ICLR_ , 2021.
  * [40] B. Wu, C. Xu, X. Dai, A. Wan, P. Zhang, Z. Yan, M. Tomizuka, J. Gonzalez, K. Keutzer, and P. Vajda, “Visual transformers: Token-based image representation and processing for computer vision,” _arXiv preprint arXiv:2006.03677_ , 2020.
  * [41] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, “The pascal visual object classes (voc) challenge,” _IJCV_ , 2010.
  * [42] B. Hariharan, P. Arbeláez, R. Girshick, and J. Malik, “Simultaneous detection and segmentation,” in _ECCV_ , 2014.

|  Zhihe Lu is now a Research Fellow at National University of Singapore. He
received the Ph.D. degree at Centre for Vision, Speech and Signal Processing
(CVSSP), University of Surrey in 2022, and the master degree at Chinese
Academy of Sciences, Institute of Automation (CASIA) in 2019. His research
interest centers around deep learning with limited annotated data. Recently,
he focuses on two applications including domain adaptation and few-shot
learning.
---|---
|  Sen He received the Ph.D. degree at University of Exeter. He is now a
Research Scientist at Meta AI, London. Previously, he was a Postdoctoral
researcher at University of Surrey. He is broadly interested in the field of
Computer Vision and Deep Learning. His current research focuses on Generative
Models.
---|---
|  Da Li is a Senior Research Scientist within the Machine Learning and Data
Intelligence group in Samsung AI Centre Cambridge and a Visiting Researcher of
the Machine Intelligence Research group at the University of Edinburgh. His
research interests span transfer learning, meta-learning and semi-supervised
learning. He serves as regular reviewers for flagship venues, e.g. CVPR, ICML,
NeurIPS and journals, e.g. TPAMI, JMLR, ML, and has served as a Senior Program
Committee (area chair) of AAAI 2022.
---|---
|  Yi-Zhe Song (Senior Member, IEEE) received the bachelor’s degree (Hons.)
from the University of Bath in 2003, the M.Sc. degree from the University of
Cambridge in 2004, and the Ph.D. degree in computer vision and machine
learning from the University of Bath in 2008. He is currently a Professor of
computer vision and machine learning and the Director of the SketchX
Laboratory, Centre for Vision Speech and Signal Processing (CVSSP), University
of Surrey. Previously, he was a Senior Lecturer at the Queen Mary University
of London and a Research and Teaching Fellow at the University of Bath. He is
a fellow of the Higher Education Academy, as well as a full member of the
EPSRC Review College, the UK’s main agency for funding research in engineering
and the physical sciences. He received the Best Dissertation Award for his
M.Sc. degree. He is the Program Chair of the British Machine Vision Conference
(BMVC) in 2021 and regularly serves as the Area Chair (AC) for flagship
computer vision and machine learning conferences, most recently at ICCV’21 and
BMVC’20.
---|---
|  Tao Xiang received the Ph.D. degree in electrical and computer engineering
from the National University of Singapore in 2001. He is currently a Professor
with the Department of Electrical and Electronic Engineering, University of
Surrey, and a Research Scientist at Facebook AI. He has published over 200
papers in international journals and conferences. His research interests
include computer vision, machine learning, and data mining.
---|---


# Waveguide and cavity quantum electrodynamics with topological bowtie modes

Anastasiia V. Vladimirova  Department of Electrical and Photonics Engineering,
DTU Electro, Technical University of Denmark, Building 343, DK-2800 Kgs.
Lyngby, Denmark.  NanoPhoton - Center for Nanophotonics, Technical University
of Denmark, Ørsteds Plads 345A, DK-2800 Kgs. Lyngby, Denmark.  Guillermo
Arregui  Department of Electrical and Photonics Engineering, DTU Electro,
Technical University of Denmark, Building 343, DK-2800 Kgs. Lyngby, Denmark.
Sergei Lepeshov  Department of Electrical and Photonics Engineering, DTU
Electro, Technical University of Denmark, Building 343, DK-2800 Kgs. Lyngby,
Denmark.  NanoPhoton - Center for Nanophotonics, Technical University of
Denmark, Ørsteds Plads 345A, DK-2800 Kgs. Lyngby, Denmark.  Christian Anker
Rosiek  Department of Electrical and Photonics Engineering, DTU Electro,
Technical University of Denmark, Building 343, DK-2800 Kgs. Lyngby, Denmark.
Babak Vosoughi Lahijani  Department of Electrical and Photonics Engineering,
DTU Electro, Technical University of Denmark, Building 343, DK-2800 Kgs.
Lyngby, Denmark.  NanoPhoton - Center for Nanophotonics, Technical University
of Denmark, Ørsteds Plads 345A, DK-2800 Kgs. Lyngby, Denmark.  Søren Stobbe
Department of Electrical and Photonics Engineering, DTU Electro, Technical
University of Denmark, Building 343, DK-2800 Kgs. Lyngby, Denmark.  NanoPhoton
- Center for Nanophotonics, Technical University of Denmark, Ørsteds Plads
345A, DK-2800 Kgs. Lyngby, Denmark.

###### Abstract

We present a theoretical study on photonic topological crystals whose symmetry
is governed by quantum valley-Hall topological insulators and whose
propagating edge modes are strongly confined due to bowtie geometries.
Dielectric bowtie structures exploit the field discontinuities at boundaries
between materials with different refractive indices, and here bowties emerge
at the topological interface due to the close proximity of two triangular
features in the underlying crystal. The topological bowtie mode features a
unit-cell mode volume down to $8\times 10^{-4}$ cubic wavelengths at the
center of the bowtie bridge of width $10\text{\,}\mathrm{nm}$, and we show
that it is possible to use perturbed versions of the unit cells as building
blocks for topological heterostructure bowtie cavities with quality factors
exceeding $10^{7}$. Due to the tightly confined bowtie mode, this implies a
strongly enhanced light-matter interaction as quantified by a Purcell factor
of $3\times 10^{6}$.

††preprint: APS/123-QED

##  I Introduction

Controlling the interaction between light and matter is fundamental to
scientific and technological developments in sensing and imaging [1, 2, 3],
quantum optics and quantum information processing [4], and nanoscale light
sources [5]. To this end, photonic crystals have proven to be a highly
versatile platform because they strongly modify the local density of optical
states (LDOS) [4]. By varying the constitutive material, lattice parameters,
and structural symmetries, photonic crystals can be used to engineer the
dispersion of light [6], to either enhance or suppress light-matter
interaction, and more generally to realize fundamental components for
experiments on quantum electrodynamics, such as cavities and waveguides.

Photonic topological insulators (PTIs) are a particular class of photonic
crystals, which have recently attracted significant attention [7, 8, 9]. In
dielectric PTIs, which are governed by time-reversal symmetry, topological
edge modes may be created by exploiting the valley degree of freedom [10],
which results in a photonic analog of the quantum valley-Hall (QVH) effect.
Although the origin of QVH interface states and conventional line-defect
waveguide modes differ, they both exhibit propagating modes [7, 8], slow light
[11] and chiral light-matter interaction [12, 13]. Besides, both are predicted
to be prone to strong backscattering induced by fabrication imperfections
[14]. Two recent experiments found clear signatures of Anderson localization
[15, 16] but reached different conclusions on whether or not PTIs may offer a
small advantage in terms of reduced backscattering.

Figure 1: Strong light-matter interaction in dielectric topological bowtie
waveguides. The interface between two QVH PTIs rotated $180^{\circ}$ relative
to each other creates a topological interface with opposing tips that form
bowties at the interface. The structure features three symmetry planes, and
$1/8$ of the structure has been cut out. A cutaway shows the electric field
amplitude, $|\mathbf{E}|$, on cross-sections along the three symmetry planes
of the structure. The normalized electromagnetic energy (white curve),
$\varepsilon|\mathbf{E}|^{2}$, in the center of the structure and
perpendicular to the waveguide axis shows a strongly enhanced intensity in the
bowties, which greatly enhances the interaction with integrated quantum
emitters (zoom-in). Figure 2:  Design of the topological bowtie waveguide. (a)
Band structure of the bulk crystal before (purple) and after (orange) breaking
of inversion symmetry by reducing the radius,
$R:$50\text{\,}\mathrm{n}\mathrm{m}$\rightarrow$20\text{\,}\mathrm{n}\mathrm{m}$$,
of one circular hole and replacing the other with an equilateral triangle with
the rounded corners and the sidelength $L=$173\text{\,}\mathrm{n}\mathrm{m}$$,
measured from end to end of the radius of each rounded corner. The degeneracy
lifting at the $K$ point manifests the topological transition between energy
levels. Red-shaded areas indicate the light cone and the opened gap. The top
and bottom insets show the irreducible Brillouin zone and the unit cells
geometries, respectively. (b) Interface waveguide geometry with lattice
constant $a=$398\text{\,}\mathrm{n}\mathrm{m}$$. The unit cell of the top
crystal (pink dashed line) is related to the unit cell of the bottom crystal
(blue dashed line) by the mirror reflection. The triangle sidelength, $L$, the
void radius of curvature at the triangle corners,
$r=$10\text{\,}\mathrm{n}\mathrm{m}$$, and $a$, determine the bridge width,
$W=$10\text{\,}\mathrm{n}\mathrm{m}$$. (c) Dispersion diagram and group index
for the waveguide modes. (d) Energy density at the edge of the Brillouin zone
$(k_{x}=\pi/a)$ for the waveguide modes. The zoom-in shows the hot-spot
formation in the bowtie bridges for the lower-frequency mode. The arrows show
the local field polarization.

The LDOS may also be enhanced without periodic structures but solely relying
on the electromagnetic boundary conditions across material interfaces. In
particular, bowtie geometries have been predicted to enhance both linear and
nonlinear optical interactions [17, 18, 19, 20, 21] due to their ability to
localize the optical energy in a very small volume. This confinement regime
was recently demonstrated experimentally and constitutes a new avenue of
nanocavity research, in part because such tight confinement of light was
previously believed to be inaccessible without plasmonic effects and in part
because the roles of disorder and critical dimension are very different
compared to conventional dielectric cavities [22, 23]. Despite the recent
progress, the physics and applications of bowtie cavities remains almost
entirely unexplored. To date, no experiments have demonstrated modified light-
matter interaction in bowtie cavities, and theoretical works have so far not
considered the implications of bowtie geometries on waveguide quantum
electrodynamics, let alone topological photonics.

In the present work, we explore the advantages of both aforementioned
approaches and propose a QVH waveguide design with bowtie features at the
interface. The proposed platform enhances light-matter interaction in
topological states of light by forming optical hot-spots at dielectric bowtie
bridges, as shown in Fig. 1, and, at the same time, by slowing down light. We
evaluate the potential of this structure for applications in quantum
nanophotonics by calculating the Purcell factor [24], and we show that the
topological waveguide mode can be modified to form an adiabatic
heterostructure cavity with a theoretical quality factor of $Q\sim 10^{7}$ and
a mode volume of $V_{\text{mod}}=6\times 10^{-3}\lambda_{\text{res}}^{3}$.

Figure 3: Purcell factor and mode volume. (a) Spatial dependence of the
Purcell enhancement for a point-source emitter with dipole moment along the
$x$-axis. The evaluation is done at a fixed $n_{\text{g}}=30$ using the
geometry in Fig. 2(b). (b) Same as (a) for a dipole along the $y$-axis. (c)
Evaluation of $P_{x}/n_{\text{g}}$ along the BZ for an emitter at the center
of the bowtie (dark red) and in the air at a height
$0.3\text{\,}\mathrm{n}\mathrm{m}$ above the slab on top of the dielectric
bridge center (light red). (d) Purcell factor $P_{x}/n_{\text{g}}$, and mode
volume, $V_{\text{mod}}^{\text{UC}}$, for a fixed $n_{\text{g}}=30$ and as a
function of the bowtie width, $W$, which is modified by changing the triangle
side-lengths everywhere. This causes the guided-mode wavelength,
$\lambda_{\text{wg}}$, to shift as indicated by the top axis. Dark and light
red curves represent the same as in (c). (e) Purcell factor as a function of
the group index at $k_{x}\in[0.75,1]$. The blue color corresponds to the W1
waveguide with a lattice constant $a=$346\text{\,}\mathrm{nm}$$, and a hole
radius $r=0.3\times a$ maintaining the same slab thickness as utilized in the
modeling of PTI. The numerical simulation results are depicted by dots, while
the line represents the linear fit to the data.

We aim to establish building blocks for PTI bowtie structures realized in
dielectric slabs of height $h=$220\text{\,}\mathrm{n}\mathrm{m}$$ and
refractive index $n=3.48$, which corresponds to silicon in the telecom C-band.
We consider honeycomb crystals with a lattice constant of
$a=$398\text{\,}\mathrm{n}\mathrm{m}$$ and unit cells (UCs) as shown in Fig.
2(a). For the UC with two circles, the crystal has $C_{6\text{v}}$ symmetry,
and the band structure exhibits a Dirac-cone degeneracy at the $K$ point in
the Brillouin zone (BZ). It is well known [10, 25, 7] that this photonic
graphene turns into a time-reversal-invariant PTI upon reducing the symmetry
to $C_{3\text{v}}$. We adhere to this procedure by replacing one of the
circular holes with an equilateral triangular hole with rounded corners. The
rounding is necessary to avoid lightning-rod effects leading to non-physical
mode volumes [26]. Contrary to previous QVH waveguide realizations that
employed triangles with edges parallel to the UC boundaries [27, 11], we
consider triangles whose corners face the UC boundaries. This particular UC
geometry implies that a zigzag interface [7] between one such topological
crystal (denoted $I$) and the mirror-symmetric version of it (denoted $II$)
results in a waveguide UC with a bowtie geometry at the center, as shown in
Fig. 2(b). The length of the straight segment, $L$, and the radius of
curvature, $r$, define the bowtie width, $W$. The radius of the complementary
circular hole, $R$, serves as an additional degree of freedom that allows
tailoring the dispersion relation of the waveguide mode.

The proposed waveguide supports two guided modes, both exhibiting diverging
group indices, $n_{\text{g}}$, at the BZ edge, facilitating a large LDOS [4],
as shown in Fig. 2(c). The energy density of the two modes in the slab mid-
plane ($z=0$) for $k_{x}=\pi/a$ is shown in Fig. 2(d). The low-frequency mode,
Mode $1$, has an antinode in the bowtie bridge, and its local polarization is
along the $x$-axis, i.e., such that the high-index bowtie acts as a field
concentrator [26]. Mode $2$ does not exhibit highly confined energy in the
bowtie bridges, and we will henceforth concentrate solely on Mode $1$.

To evaluate the strength of the light-matter interaction, we calculate the
ratio of the spontaneous-emission decay rate in the QVH waveguide to that in
the corresponding homogeneous medium (bulk silicon), i.e., the Purcell factor
[24]. In the dipole approximation, the Purcell factor for a point-like quantum
emitter at a position $\mathbf{r}_{\text{d}}$, oriented along the unit vector
$\mathbf{\hat{n}}_{\text{d}}$, and emitting at the waveguide mode frequency,
$\omega$, can be derived analytically [28],

$P_{\mathbf{\mathbf{\hat{n}}_{\text{d}}}}(\mathbf{r}_{\text{d}},\omega)=\frac{3\pi
c^{2}an_{\text{g}}}{n(\mathbf{r}_{\text{d}})\omega^{2}}\frac{|\mathbf{E}(\mathbf{r}_{\text{d}})\cdot\mathbf{\hat{n}}_{\text{d}}|^{2}}{V_{\text{mod}}^{\text{UC}}(\mathbf{r}_{\text{d}})\varepsilon(\mathbf{r}_{\text{d}})|\mathbf{E}(\mathbf{r}_{\text{d}})|^{2}}.$
(1)

Here, $\mathbf{E}(\mathbf{r}_{\text{d}})$ and
$V_{\text{mod}}^{\text{UC}}(\mathbf{r})$ are the electric field and the mode
volume of the Bloch mode, respectively. The latter is given by [29]

$V_{\text{mod}}^{\text{UC}}(\mathbf{r})=\frac{\int_{\Omega_{\text{UC}}}d\mathbf{r^{\prime}}\varepsilon(\mathbf{r^{\prime}})\mathbf{E}(\mathbf{r^{\prime}})\cdot\mathbf{E}^{*}(\mathbf{r^{\prime}})}{\varepsilon(\mathbf{r})\mathbf{E}(\mathbf{r})\cdot\mathbf{E}^{*}(\mathbf{r})},$
(2)

where $\Omega_{\text{UC}}$ is the volume of the waveguide UC. Using Eq. (1),
we analyze the spatial dependence of the Purcell factors, $P_{x}$ and $P_{y}$,
corresponding to $x$\- and $y$-polarized dipole emitters, respectively. This
is shown in Fig. 3(a) and (b) for emitters at the mid-plane of the slab
($z=0$) and at a fixed point in the dispersion relation for which
$n_{\text{g}}=30$. We consider this value of $n_{\text{g}}$ in order to limit
the discussion to a regime where backscattering losses do not become
prohibitively high for practically relevant device lengths [15, 30]. We find a
Purcell factor for $x$-polarized emitters placed in the center of the bowtie
of $55$. Also, high Purcell factors up to $82$ are predicted for emitters
located near the air tips (see the zoom-in in Fig. 3(a)), although this may
likely be of limited experimental significance due to the immediate proximity
of surfaces and their associated roughness leading to significant experimental
and numerical uncertainties [31, 26, 13].

We now consider how the Purcell factor varies in momentum space. To separate
the well-known slow-light effect on the Purcell factor from the spatial
confinement of interest here, we evaluate
$P_{x}(\mathbf{r}_{\text{c}},\omega)/n_{\text{g}}$. We consider the dependence
on the wavevector, $k_{x}$, for two different points located at the center of
the bowtie at the heights $z=0$ and $z=h/2+$0.3\text{\,}\mathrm{n}\mathrm{m}$$
as shown in Fig. 3(c). These two planes correspond to the decay-rate
enhancement for emitters embedded in the center of the membrane, e.g., erbium
ions [32, 33, 34] and confined excitons in monolayer $2$D-materials deposited
above the patterned slab [35]. Notably,
$P_{x}(\mathbf{r}_{\text{c}},\omega)/n_{\text{g}}$ for our design exhibits
only a weak dependence on $k_{x}$.

The Purcell enhancement can be substantially improved by reducing the width of
the bowtie bridge [26]. Figures 3(d) and (e) illustrate the group-index-
normalized Purcell factor and mode volume as a function of the width, $W$, for
a fixed $n_{\text{g}}=30$. We vary the bowtie width from
$1\text{\,}\mathrm{n}\mathrm{m}$ to $30\text{\,}\mathrm{n}\mathrm{m}$ to
illustrate the span between extreme bowtie structures, potentially enabled by
self-assembly [23], and bowties achievable in several semiconductor platforms.
Such geometrical modification allows covering the full telecom C-band as shown
in the top $x$-axis of Fig. 3(d, e). We observe that $P_{x}/n_{\text{g}}$
drops with increasing $W$, which results from the associated increase in
$V_{\text{mod}}^{\text{UC}}$. For state-of-the-art silicon top-down
nanofabrication, i.e., $W=$8\text{\,}\mathrm{n}\mathrm{m}$$ [22], we find
$V_{\text{mod}}^{\text{UC}}|_{z=0}=0.63\times
10^{-3}(\lambda_{\text{wg}}^{3})$ and
$V_{\text{mod}}^{\text{UC}}|_{z=h/2+$0.3\text{\,}\mathrm{n}\mathrm{m}$}=13\times
10^{-3}(\lambda_{\text{wg}}^{3})$. This leads to high Purcell factors up to
$P_{x}^{z=0}=70$ and $P_{x}^{z=h/2+$0.3\text{\,}\mathrm{n}\mathrm{m}$}=143$.
Remarkably, these substantial enhancement factors are attained in a
propagating waveguide mode, i.e., without a cavity.

In order to investigate the benefits of the topological bowtie waveguides, we
now compare their performance to conventional W1 waveguides, which have been
studied extensively for waveguide quantum electrodynamics [4]. Our waveguides
and the W1 waveguides can be directly compared as they can be realized in the
same materials at the same wavelengths, only it is important to note that the
Purcell factor in the case of W1 is larger for $P_{y}$-oriented dipoles, so
this is what we consider in Fig. 3(f), where we assume a realistic value of
$W=$10\text{\,}\mathrm{nm}$$. Remarkably, while the slope of $P_{y}$ is $0.2$
for the W1 waveguide, the slope of $P_{x}$ is $2.0$ for the topological bowtie
waveguide. This twenty-fold enhancement relative to W1 waveguides at the same
group index is a direct consequence of the bowtie field enhancement. In
comparison to conventional W1 waveguides, which achieve high Purcell factors
only due to the slow light effect, our structures additionally take advantage
of the bowtie field confinement, spanning across the whole BZ, cf. Fig. 3(c).

Figure 4: Heterostructure bowtie crystal cavity (a) Band diagram of the
central waveguide UC with $R=$20\text{\,}\mathrm{n}\mathrm{m}$$ (solid lines)
plotted on top of the band diagram of the mirror waveguide UC with
$R=$32\text{\,}\mathrm{n}\mathrm{m}$$ (dashed lines). (b) A sketch of the
cavity design. The different radii of the circles in adjacent waveguide UCs
are shown with different colors spanning from
$R=$20\text{\,}\mathrm{n}\mathrm{m}$$ to $R=$32\text{\,}\mathrm{n}\mathrm{m}$$
with a $1\text{\,}\mathrm{n}\mathrm{m}$ step. (c) Electric field amplitude
distribution of the fundamental mode of the cavity in the center of the slab
($z=0$) obtained from a $3$D simulation. (d) Energy density distribution along
the waveguide interface and the Gaussian envelop fitting the peaks. In
subplots (b), (c), and (d), the $x$-coordinate spans a consistent range.

Finally, we utilize the topological waveguide mode to engineer a mode-gap
cavity [36]. Previous studies of optical cavities based on topological
structures have focused on trapped topological $0$D states at the interface of
1D PTI structures [37], higher-order topological corner states between
adjacent $2$D PTIs [38, 39, 40], or topological ring cavities formed through
1D topological edge states that exhibit robust transport across sharp bends
[41, 42, 43, 44, 45, 46, 47]. Here, we use the bowtie QVH waveguide as a
building block to form an ultra-high-$Q$ multi-step heterostructure cavity
following the approach proposed in Ref. 48. We introduce tapered mirrors by
varying the geometrical parameters of the waveguide along the propagation
direction such that the waveguide mode frequency at the BZ edge forms a smooth
confinement potential. In our case, the cavity is created by changing $R$ from
$20\text{\,}\mathrm{n}\mathrm{m}$ in the cavity center to
$32\text{\,}\mathrm{n}\mathrm{m}$ at the cavity edges in steps of
$1\text{\,}\mathrm{n}\mathrm{m}$, resulting in tapered mirrors distributed
along $N=12$ UCs. The dispersion diagrams of the UCs with
$R=$20\text{\,}\mathrm{n}\mathrm{m}$$ and
$R=$32\text{\,}\mathrm{n}\mathrm{m}$$ are shown in Fig. 4(a), evidencing the
creation of a $2.7\text{\,}\mathrm{T}\mathrm{H}\mathrm{z}$-deep potential at
the BZ edge, and the tapering scheme is depicted in Fig. 4(b). The variations
across the tapering section comply with conventional minimum shot pitches [22]
in direct-write electron-beam lithography systems while providing sufficient
adiabaticity to achieve ultra-low-loss cavities. To enhance the $Q$-factor,
additional mirror sections with fixed $R=$32\text{\,}\mathrm{n}\mathrm{m}$$
are added at both cavity edges. Typically, the $Q$-factor of such cavities
rapidly increases with the number of mirrors, $M$, before eventually
saturating. In our case, the $Q$-factor of the fundamental mode reaches its
maximum value at $M=15$. The electric field amplitude distribution of this
mode, occurring at $\lambda=$1550\text{\,}\mathrm{n}\mathrm{m}$$, is shown in
Fig. 4(c). The field exhibits hot spots in the waveguide defect region, the
intensity of which drops from the center toward the edges of the tapered
mirrors. Figure 4(d) illustrates the energy-density distribution of the
fundamental mode along the axis of the waveguide and a Gaussian envelope,
$f(x)$, fitted to the peak values. Given the slow variation of the envelope
function at the scale of the lattice period and its Bloch-mode nature [49],
the cavity mode volume can be approximated as

${V_{\text{mod}}}(\mathbf{r}_{\text{d}})=\frac{\int_{\text{L}}dx|f(x)|^{2}}{a|f(x_{\text{d}})|^{2}}V_{\text{mod}}^{\text{UC}}(\mathbf{r}_{\text{d}}),$
(3)

establishing a direct connection between the mode volume of the waveguide and
the cavity [30]. Employing Eq. (3) alongside a direct numerical evaluation of
Eq. (2) at the hot spot of the central bowtie, we achieve
$V_{\text{mod}}=6\times 10^{-3}\lambda_{\text{res}}^{3}$, in good agreement
with the value obtained by direct evaluation of the cavity version of Eq. (2)
using the full cavity field, i.e., $V_{\text{mod}}=5.9\times
10^{-3}\lambda_{\text{res}}^{3}$. Formally, $V_{\text{mod}}$ in Eq. (2) has an
additional surface term in case of the cavity due to its non-Hermitian
character, but we disregard this term as its impact is negligible for cavities
with high $Q$ factors [50]. We evaluate the Purcell factor for a resonant
emitter polarized along the fundamental cavity mode and at its center,

$P(\mathbf{r}_{\text{d}})=\frac{3}{4\pi^{2}}\left(\frac{\lambda}{n(\mathbf{r}_{\text{d}})}\right)^{3}\frac{Q}{V_{\text{mod}}(\mathbf{r}_{\text{d}})}$
(4)

which results in $P=3\times 10^{6}$ owing to the large $Q$-factor and small
$V_{\text{mod}}$. Compared to previously studied QVH cavity designs, which are
ring cavities and exhibit a broad spatial field distribution extending over a
significant area, our heterostructure design enables the confinement of the
field to a much smaller region. This confinement leads to significantly
reduced mode volume and, at the same time, a greatly increased $Q$-factor,
which is three orders of magnitude higher than those calculated for ring
cavities [41, 42, 43, 44, 45, 46, 47].

##  II  Conclusion

In conclusion, the QVH PTI bowtie waveguide design presented here exhibits
promising characteristics for enhancing light-matter interaction, achieving
$P_{x}=70$ at $n_{\text{g}}=30$ in the mid-plane of the slab. Compared to
previous approaches [4] to waveguide quantum electrodynamics, the attained
high Purcell factor is not only achieved by a slow-light effect, i.e., a high
$n_{\text{g}}$. Our work, therefore, shows a path to enhancing light-matter
interaction beyond what is possible in conventional waveguides. Notably, both
the slow-light effect and the bowtie effect are inherently broadband. However,
the field concentration at the bowtie surfaces may also induce additional
scattering, which calls for experimental investigations of propagation losses
[15, 16]. In addition, the proposed waveguide design is multimodal, which may
induce intramodal scattering, but the interplay between disorder and Anderson
localization in bowtie waveguides has not yet been explored and would be an
exciting avenue for future research. Further investigations could improve the
dispersion by applying the topology-optimization techniques [51]. Yet another
important line of research should consider time-reversal-symmetry-broken
systems, including magneto-optic materials [9], to achieve edge-mode non-
reciprocity and, thus, ensure topological protection.

We acknowledge financial support from the Villum Foundation Young Investigator
Program (Grant No. 13170), Innovation Fund Denmark (Grant No. 0175-00022 -
NEXUS and Grant No. 2054-00008 - SCALE), Independent Research Fund Denmark
(Grant No. 0135-00315 - VAFL), the European Research Council (Grant No.
101045396 - SPOTLIGHT), and the Horizon Europe Research and Innovation
Programme (Grant No. 101098961 - NEUROPIC and Grant no. 101067606 - TOPEX).

## References

  * Zhen  _et al._ [2013] B. Zhen, S.-L. Chua, J. Lee, A. W. Rodriguez, X. Liang, S. G. Johnson, J. D. Joannopoulos, M. Soljačić, and O. Shapira, Enabling enhanced emission and low-threshold lasing of organic molecules using special fano resonances of macroscopic photonic crystals, PNAS 110, 13711 (2013).
  * Altug  _et al._ [2022] H. Altug, S.-H. Oh, S. A. Maier, and J. Homola, Advances and applications of nanophotonic biosensors, Nat. Nanotechnol. 17, 5 (2022).
  * Beliaev  _et al._ [2022] L. Y. Beliaev, P. G. Stounbjerg, G. Finco, A.-I. Bunea, R. Malureanu, L. R. Lindvold, O. Takayama, P. E. Andersen, and A. V. Lavrinenko, Pedestal high-contrast gratings for biosensing, Nanomater. 12, 1748 (2022).
  * Lodahl  _et al._ [2015] P. Lodahl, S. Mahmoodian, and S. Stobbe, Interfacing single photons and single quantum dots with photonic nanostructures, Rev. Mod. Phys. 87, 347 (2015).
  * Romeira and Fiore [2018] B. Romeira and A. Fiore, Purcell effect in the stimulated and spontaneous emission rates of nanoscale semiconductor lasers, IEEE J. Quantum Electron. 54, 1 (2018).
  * Lopez [2003] C. Lopez, Materials aspects of photonic crystals, Adv. Mater. 15, 1679 (2003).
  * Ozawa  _et al._ [2019] T. Ozawa, H. M. Price, A. Amo, N. Goldman, M. Hafezi, L. Lu, M. C. Rechtsman, D. Schuster, J. Simon, O. Zilberberg, and I. Carusotto, Topological photonics, Rev. Mod. Phys. 91, 015006 (2019).
  * Kim  _et al._ [2020] M. Kim, Z. Jacob, and J. Rho, Recent advances in 2d, 3d and higher-order topological photonics, Light Sci. Appl. 9, 130 (2020).
  * Ni  _et al._ [2023] X. Ni, S. Yves, A. Krasnok, and A. Alu, Topological metamaterials, Chem. Rev. 123, 7585 (2023).
  * Ma and Shvets [2016] T. Ma and G. Shvets, All-Si valley-Hall photonic topological insulator, New J. Phys. 18, 025012 (2016).
  * Yoshimi  _et al._ [2020] H. Yoshimi, T. Yamaguchi, Y. Ota, Y. Arakawa, and S. Iwamoto, Slow light waveguides in topological valley photonic crystals, Opt. Lett. 45, 2648 (2020).
  * Lodahl  _et al._ [2017] P. Lodahl, S. Mahmoodian, S. Stobbe, A. Rauschenbeutel, P. Schneeweiss, J. Volz, H. Pichler, and P. Zoller, Chiral quantum optics, Nature 541, 473 (2017).
  * Hauff  _et al._ [2022] N. V. Hauff, H. Le Jeannic, P. Lodahl, S. Hughes, and N. Rotenberg, Chiral quantum optics in broken-symmetry and topological photonic crystal waveguides, Phys. Rev. Res. 4, 023082 (2022).
  * Arregui  _et al._ [2021] G. Arregui, J. Gomis-Bresco, C. M. Sotomayor-Torres, and P. D. Garcia, Quantifying the robustness of topological slow light, Phys. Rev. Lett. 126, 027403 (2021).
  * Rosiek  _et al._ [2023] C. A. Rosiek, G. Arregui, A. Vladimirova, M. Albrechtsen, B. Vongughi Lahijani, R. E. Christiansen, and S. Stobbe, Observation of strong backscattering in valley-Hall topological interface modes, Nat. Photon. 17, 386 (2023).
  * Arora  _et al._ [2023] S. Arora, T. Bauer, R. Barczyk, E. Verhagen, and L. Kuipers, Multiple backscattering in trivial and non-trivial topological photonic crystal edge states with controlled disorder (2023), 2310.02978v2 [physics.optics] .
  * Gondarenko  _et al._ [2006] A. Gondarenko, S. Preble, J. Robinson, L. Chen, H. Lipson, and M. Lipson, Spontaneous emergence of periodic patterns in a biologically inspired simulation of photonic structures, Phys. Rev. Lett. 96, 143904 (2006).
  * Hu and Weiss [2016] S. Hu and S. M. Weiss, Design of photonic crystal cavities for extreme light concentration, ACS Photon. 3, 1647 (2016).
  * Choi  _et al._ [2017] H. Choi, M. Heuck, and D. Englund, Self-similar nanocavity design with ultrasmall mode volume for single-photon nonlinearities, Phys. Rev. Lett. 118, 223605 (2017).
  * Wang  _et al._ [2018] F. Wang, R. E. Christiansen, Y. Yu, J. Mørk, and O. Sigmund, Maximizing the quality factor to mode volume ratio for ultra-small photonic crystal cavities, Appl. Phys. Lett. 113 (2018).
  * Christiansen  _et al._ [2023] R. E. Christiansen, P. T. Kristensen, J. Mørk, and O. Sigmund, Impact of figures of merit in photonic inverse design, Opt. Express 31, 8363 (2023).
  * Albrechtsen  _et al._ [2022a] M. Albrechtsen, B. Vosoughi Lahijani, R. E. Christiansen, V. T. H. Nguyen, L. N. Casses, S. E. Hansen, N. Stenger, O. Sigmund, H. Jansen, J. Mørk, and S. Stobbe, Nanometer-scale photon confinement in topology-optimized dielectric cavities, Nat. Commun. 13, 6281 (2022a).
  * Babar  _et al._ [2023] A. N. Babar, T. Weis, K. Tsoukalas, S. Kadkhodazadeh, G. Arregui, B. Vosoughi Lahijani, and S. Stobbe, Self-assembled photonic cavities with atomic-scale confinement, Nature 624, 57 (2023).
  * Purcell [1995] E. M. Purcell, Spontaneous emission probabilities at radio frequencies, in  _Confined Electrons and Photons: New Physics and Applications_ (1995) pp. 839–839.
  * [25] H. Xue, Y. Yang, and B. Zhang, Topological valley photonics: Physics and device applications, Adv. Photonics Res. 2, 2100013.
  * Albrechtsen  _et al._ [2022b] M. Albrechtsen, B. Vosoughi Lahijani, and S. Stobbe, Two regimes of confinement in photonic nanocavities: bulk confinement versus lightning rods, Opt. Express 30, 15458 (2022b).
  * Shalaev  _et al._ [2019] M. I. Shalaev, W. Walasik, A. Tsukernik, Y. Xu, and N. M. Litchinitser, Robust topologically protected transport in photonic crystals at telecommunication wavelengths, Nat. Nanotechnol. 14, 31 (2019).
  * Hughes [2004] S. Hughes, Enhanced single-photon emission from quantum dots in photonic crystal waveguides and nanocavities, Opt. Lett. 29, 2659 (2004).
  * Rao and Hughes [2007] V. M. Rao and S. Hughes, Single quantum-dot purcell factor and $\beta$ factor in a photonic crystal waveguide, Phys. Rev. B 75, 205437 (2007).
  * Arregui  _et al._ [2023] G. Arregui, R. C. Ng, M. Albrechtsen, S. Stobbe, C. Sotomayor-Torres, and P. D. García, Cavity optomechanics with anderson-localized optical modes, Phys. Rev. Lett. 130, 043802 (2023).
  * García and Lodahl [2017] P. D. García and P. Lodahl, Physics of quantum light emitters in disordered photonic nanostructures, Ann. Phys. 529, 1600351 (2017).
  * Polman [1997] A. Polman, Erbium implanted thin film photonic materials, J. Appl. Phys. 82, 1 (1997).
  * Weiss  _et al._ [2021] L. Weiss, A. Gritsch, B. Merkel, and A. Reiserer, Erbium dopants in nanophotonic silicon waveguides, Optica 8, 40 (2021).
  * Gritsch  _et al._ [2022] A. Gritsch, L. Weiss, J. Früh, S. Rinner, and A. Reiserer, Narrow optical transitions in erbium-implanted silicon waveguides, Phys. Rev. X 12, 041009 (2022).
  * Denning  _et al._ [2022] E. V. Denning, M. Wubs, N. Stenger, J. Mørk, and P. T. Kristensen, Quantum theory of two-dimensional materials coupled to electromagnetic resonators, Phys. Rev. B 105, 085306 (2022).
  * Akahane  _et al._ [2003] Y. Akahane, T. Asano, B.-S. Song, and S. Noda, High-q photonic nanocavity in a two-dimensional photonic crystal, Nature 425, 944 (2003).
  * Ota  _et al._ [2018] Y. Ota, R. Katsumi, K. Watanabe, S. Iwamoto, and Y. Arakawa, Topological photonic crystal nanocavity laser, Commun. Phys. 1, 86 (2018).
  * Ota  _et al._ [2019] Y. Ota, F. Liu, R. Katsumi, K. Watanabe, K. Wakabayashi, Y. Arakawa, and S. Iwamoto, Photonic crystal nanocavity based on a topological corner state, Optica 6, 786 (2019).
  * Zhang  _et al._ [2020] W. Zhang, X. Xie, H. Hao, J. Dang, S. Xiao, S. Shi, H. Ni, Z. Niu, C. Wang, K. Jin, _et al._ , Low-threshold topological nanolasers based on the second-order corner state, Light Sci. Appl. 9, 109 (2020).
  * Medina-Vázquez  _et al._ [2022] J. A. Medina-Vázquez, J. G. Murillo-Ramírez, E. Y. González-Ramírez, and S. F. Olive-Méndez, Weak and strong coupling regimes in a topological photonic crystal bowtie cavity, J. Appl. Phys. 132 (2022).
  * Smirnova  _et al._ [2020] D. Smirnova, A. Tripathi, S. Kruk, M.-S. Hwang, H.-R. Kim, H.-G. Park, and Y. Kivshar, Room-temperature lasing from nanophotonic topological cavities, Light Sci. Appl. 9, 127 (2020).
  * Noh  _et al._ [2020] W. Noh, H. Nasari, H.-M. Kim, Q. Le-Van, Z. Jia, C.-H. Huang, and B. Kanté, Experimental demonstration of single-mode topological valley-hall lasing at telecommunication wavelength controlled by the degree of asymmetry, Opt. Lett. 45, 4108 (2020).
  * Zeng  _et al._ [2020] Y. Zeng, U. Chattopadhyay, B. Zhu, B. Qiang, J. Li, Y. Jin, L. Li, A. G. Davies, E. H. Linfield, B. Zhang, _et al._ , Electrically pumped topological laser with valley edge modes, Nature 578, 246 (2020).
  * Mehrabad  _et al._ [2020] M. J. Mehrabad, A. P. Foster, R. Dost, E. Clarke, P. K. Patil, A. M. Fox, M. S. Skolnick, and L. R. Wilson, Chiral topological photonics with an embedded quantum emitter, Optica 7, 1690 (2020).
  * Gu  _et al._ [2021] L. Gu, Q. Yuan, Q. Zhao, Y. Ji, Z. Liu, L. Fang, X. Gan, and J. Zhao, A topological photonic ring-resonator for on-chip channel filters, J. Light. Technol. 39, 5069 (2021).
  * Gong  _et al._ [2020] Y. Gong, S. Wong, A. J. Bennett, D. L. Huffaker, and S. S. Oh, Topological insulator laser using valley-hall photonic crystals, Acs Photonics 7, 2089 (2020).
  * Liu  _et al._ [2022] X. Liu, L. Zhao, D. Zhang, and S. Gao, Topological cavity laser with valley edge states, Opt. Express 30, 4965 (2022).
  * Song  _et al._ [2005] B.-S. Song, S. Noda, T. Asano, and Y. Akahane, Ultra-high-q photonic double-heterostructure nanocavity, Nat. Mater. 4, 207 (2005).
  * Charbonneau-Lefort  _et al._ [2002] M. Charbonneau-Lefort, E. Istrate, M. Allard, J. Poon, and E. H. Sargent, Photonic crystal heterostructures: Waveguiding phenomena and methods of solution in an envelope function picture, Phys. Rev. B 65, 125318 (2002).
  * Kristensen  _et al._ [2020] P. T. Kristensen, K. Herrmann, F. Intravaia, and K. Busch, Modeling electromagnetic resonators using quasinormal modes, Adv. Opt. Photonics 12, 612 (2020).
  * Christiansen  _et al._ [2019] R. E. Christiansen, F. Wang, O. Sigmund, and S. Stobbe, Designing photonic topological insulators with quantum-spin-hall edge states using topology optimization, NanoPhoton. 8, 1363 (2019).