# E2EC: An End-to-End Contour-based Method for High-Quality High-Speed
Instance Segmentation

Tao Zhang∗, Shiqing Wei, Shunping Ji
Wuhan University, China
{zhang_tao, wei_sq<EMAIL_ADDRESS>: Equally contributed. :
Corresponding author.

###### Abstract

Contour-based instance segmentation methods have developed rapidly recently
but feature rough and handcrafted front-end contour initialization, which
restricts the model performance, and an empirical and fixed backend predicted-
label vertex pairing, which contributes to the learning difficulty. In this
paper, we introduce a novel contour-based method, named E2EC, for high-quality
instance segmentation. Firstly, E2EC applies a novel learnable contour
initialization architecture instead of handcrafted contour initialization.
This consists of a contour initialization module for constructing more
explicit learning goals and a global contour deformation module for taking
advantage of all of the vertices’ features better. Secondly, we propose a
novel label sampling scheme, named multi-direction alignment, to reduce the
learning difficulty. Thirdly, to improve the quality of the boundary details,
we dynamically match the most appropriate predicted-ground truth vertex pairs
and propose the corresponding loss function named dynamic matching loss. The
experiments showed that E2EC can achieve a state-of-the-art performance on the
KITTI INStance (KINS) dataset, the Semantic Boundaries Dataset (SBD), the
Cityscapes and the COCO dataset. E2EC is also efficient for use in real-time
applications, with an inference speed of 36 fps for 512×512 images on an
NVIDIA A6000 GPU. Code will be released at https://github.com/zhang-tao-
whu/e2ec.

##  1 Introduction

Instance segmentation is a fundamental computer vision task and the
cornerstone of many downstream computer vision applications, such as
autonomous driving and robotic grasping. The classic instance segmentation
methods are based on a two-stage pipeline, where the bounding boxes (bboxes)
of the instances are first generated, and then pixel-wise segmentation is
performed within the bboxes. Typical examples are methods such as Mask R-CNN
[14] and PANet [23]. These methods can achieve an excellent accuracy, but are
inefficient, which restricts their application in real-time tasks. With the
rapid development of one-stage detectors [39, 29], many one-stage mask-based
instance segmentation methods have now been proposed, such as YOLACT [2],
BlendMask [3], TensorMask [4], and CenterMask [17]. However, these one-stage
methods consume a lot of storage, require costly post-processing, and hardly
perform in real time. The quality of the instance boundary prediction is also
unsatisfactory as these methods usually use limited feature information (for
example, Mask R-CNN only segments instances in a 28×28 feature map).

Figure 1: The ideal deformation paths of several contour-based methods. White
boundaries and points are the initial contours, blue lines are the deformation
paths, and black points are the alignment points.

The contour-based methods have recently received renewed attention and have
shown great potential. Examples of such methods are Curve GCN [21], Deep Snake
[27], Point-Set Anchors [31], DANCE [25], PolarMask [33], and LSNet [9]. The
contour-based methods treat the instance segmentation as a regression task,
i.e., regressing the vertex coordinates of a contour represented by a series
of discrete vertices. A contour composed of $N$ (e.g., $N=128$) vertices is
sufficient to describe most of the instances well [27]. Compared with the
mask-based methods, which require intensive processing of each pixel, the
contour-based methods are simpler and require less calculation. The contour-
based methods can also directly obtain the boundaries of the instances,
without any complicated post-processing.

However, the existing contour-based methods still have many obvious
shortcomings. First, all of the existing multi-stage methods adopt a manually
designed shape for the initial contour. As shown in Figure 1, the difference
between the manually designed initial contour and the ground-truth instance
boundary can lead to many unreasonable deformation paths (the route from
initial to ground-truth vertex) and huge training difficulty. It is also
impossible to sample the manually designed initial contour to achieve a
uniform angle and uniform vertex spacing at the same time. The Point-Set
Anchors and DANCE methods attempt to address this problem by changing the
intuitive vertex pairing method [31, 25], but the results are not
satisfactory.

Second, local or limited information is popularly applied in contour
adjustment. For example, the one-stage PolarMask [33] and LSNet [9] methods
directly regress the coordinates of the contour vertices based on only the
limited features at the instance center, resulting in the loss of the
predicted contour details. The multi-stage methods iteratively adjust the
initial contour based on the features of the contour vertices to obtain a more
refined segmentation result. However, Curve GCN and Deep Snake utilize a local
information aggregation mechanism that propagates the features of the local
adjacent contour vertices to refine the contour, which can fail in correcting
large prediction errors. Moreover, the local aggregation has to be
inefficiently repeated to access global information. Instead, we propose a
global contour deformation method based on the features of all the contour
vertices.

Third, the pairing of the ground-truth and predicted vertices in the current
contour-based methods is fixed, regardless of the continuous position
adjustment of a predicted vertex (e.g., it is already on the ground-truth
boundary or close to another ground-truth vertex, but far from the given one).
Hence, the pre-fixed vertex pairing is not optimal, and can result in a slower
convergence speed, and even wrong predictions.

In this paper, we propose a multi-stage and highly efficient end-to-end
contour-based instance segmentation model named E2EC, which can completely
overcome these shortcomings. E2EC incorporates three novel components: 1) a
learnable contour initialization architecture; 2) multi-direction alignment
(MDA); and 3) a dynamic matching loss (DML) function.

E2EC replaces the manually designed initial contour with a learnable contour
initialization architecture, which handles the first and second problem. This
architecture contains two novel modules: 1) a contour initialization module;
and 2) a global contour deformation module. The contour initialization module
directly regresses the complete initial contour based on the center point
features, which differs from regressing lengths along given fixed rays [33].
The global contour deformation module then refines the initial contour based
on all of the features of the initial contour vertices and center point
instead of using features of local vertices. As shown in Figure 1, the
learnable initial contour architecture does not rely on a manually designed
initial contour (e.g., the ellipse of Curve GCN or the octagon of Deep Snake),
and directly deforms from the midpoint of the object instance to the contour
with more reasonable paths.

Figure 2: Overview of E2EC. E2EC consists of a learnable contour
initialization architecture including a contour initialization and a global
deformation module that produces the coarse contour, and a contour refinement
module that produces the final contour with the supervision of DML.

The difficulty in predicted-label vertex pairing roots in the fact that no
simple differentiable calculation can measure the distance between the
predicted and ground-truth boundaries. To address the third problem, on the
one hand, we propose multi-direction alignment (MDA), which fixes the
directions of the selected multiple contour vertices with respect to the
center point (the black points in Figure 1 (E2EC)), and then uniformly samples
between the fixed vertices to generate ground-truth vertices. MDA
appropriately restricts the possible vertex pairing and deformation paths, and
greatly reduces the difficulty of learning while ensuring the upper bound of
the performance. The combination of the learnable initial contour architecture
and MDA eliminates the unreasonable deformation paths that commonly exist in
the current contour-based methods.

On the other hand, we propose a matching strategy that dynamically matches the
predicted vertices and the most appropriate label vertices instead of fixed
pairing, and the corresponding dynamic matching loss (DML) function. DML
eliminates the problems of an over-smooth boundary and poor fitting of the
inflection points in the contour-based methods, and greatly improves the
quality of the predicted boundary details.

In the experiments conducted in this study, E2EC exhibited a state-of-the-art
performance on the KITTI INStance (KINS) dataset [28], the Semantic Boundaries
Dataset (SBD) [13] and the Cityscapes [6] dataset. For 512×512 images, E2EC
achieved a 36 fps inference speed on an NVIDIA A6000 GPU. If the iterative
deformation module is disabled, E2EC can reach a speed of 50 fps, with an
accuracy comparable to that of Deep Snake.

##  2 Related Work

Mask-based instance segmentation methods. The classic mask-based instance
segmentation methods, such as Mask R-CNN [14] and PANet [23], include a bbox
extraction stage and a mask segmentation stage. These methods can achieve a
good performance, but with a very slow speed. In recent years, one-stage
methods such as CenterMask [17], YOLACT [2], SOLO [30], and BlendMask [3] that
follow the above process have developed rapidly, and have greatly improved in
speed. However, dense pixel-wise classification requires a huge amount of
calculation. Although these methods try to sacrifice performance and perform
segmentation on the down-sampled feature maps, to reduce the amount of
calculation, they still cannot meet the requirement of real-time performance.
The methods proposed in [12, 26] follow another pipeline, where they first
perform semantic segmentation and then cluster the pixels to generate
instances. However, these methods require complex post-processing and cannot
be applied to amodal instance segmentation tasks.

Figure 3: Global deformation (b) vs. circular convolution [27] (a). The green
points represent the features of the contour vertices, the yellow points
represent the local kernel function of the circular convolution, and the blue
points represent the offsets of the contour vertices, and red is MLP.

Figure 4: Multi-direction alignment. M is the number of vertices fixed in the
direction with respect to the center point. When M increases, the learning
difficulty of the task gradually decreases, but the unevenness of the vertex
distribution also gradually increases.

Contour-based instance segmentation methods. Compared with the mask-based
methods, the contour-based methods have an absolute advantage in speed.
PolarMask [33] and LSNet [9] directly regress the coordinates of the instance
vertices based on the features at the center point, and can reach a speed that
is almost equivalent to that of the detector; however, the corresponding
segmentation quality is quite rough. Curve GCN [21], Deep Snake [27], Point-
Set Anchors [31], and DANCE [25] use the vertex features of the contour for
the boundary regression, which greatly improves the performance. These methods
first initialize the contour, and then iteratively deform the initial contour
to obtain the final instance contour. However, the initial contour shapes of
these methods are all manually designed, such as the ellipse of Curve GCN, the
octagonal of Deep Snake, and the rectangle of Point-Set Anchors and DANCE. The
huge difference between the manually designed initial contour and the ground-
truth contour leads to many inappropriate vertex pairs, as shown in Figure 1.
For example, there are many intersections between the deformation paths of
Curve GCN and Deep Snake at different vertices that confuse the training
process. The segment-wise matching strategy proposed by DANCE slightly
alleviates the above problem, but the intersections still exist. The
deformation paths of Point-Set Anchors seem reasonable, but its vertex paring
strategy seriously reduces the upper bound of the performance. The proposed
E2EC method eliminates the unreasonable deformation path of the contour-based
methods and does not reduce the upper bound of the performance.

Other instance segmentation methods. Dense RepPoints [37] uses a discrete
point set to model the instance, but the instance representation of Dense
RepPoints cannot be directly converted to a mask or contour representation,
and requires complex post-processing. Polytransform [19] combines a mask-based
method and a contour-based method. Poly Transform first generates the mask
representation of the instance, then converts the mask into a contour through
post-processing, and finally refines the contour by a deformation module.
However, Poly Transform cannot be trained end-to-end, and the speed is too
slow for it to be applied to real-time scenes.

Figure 5: Dynamic matching loss. The yellow points are the predicted contour
vertices, the green points are the label vertices, the red points are the key
label vertices, and the arrows represent the deformation path (the
relationship of the pairing). (a) The first part of DML, where each prediction
point is adjusted to the nearest point on the ground-truth boundary. (b) The
second part of DML, where the key label point pulls the nearest prediction
point toward its position.

##  3 The proposed E2EC method

In this section, we describe the three main parts of the proposed end-to-end
contour-based (E2EC) instance segmentation method, i.e., the learnable contour
initialization architecture, the multi-direction alignment (MDA), and the
dynamic matching loss (DML) function. The workflow of E2EC is shown in Figure
2. E2EC first generates a heatmap to locate the instance centers, and then
learns the initial contour by regressing the initial offsets based on the
center point features. The initial contour is first deformed by a global
deformation module, and evolves to the coarse contour. The deformation modules
[27] then deform the coarse contour twice to the final contour.

###  3.1 Learnable contour initialization architecture

The learnable contour initialization architecture includes a contour
initialization module and a global deformation module.

Initial contour. Unlike the manually designed initialization used in the
existing contour-based methods, it is not necessary to specify the shape of
the initial contour as this is learned by the network. Inspired by Dense
RepPoints [37], the offsets of each initial contour vertex are directly
regressed with respect to the center point, based on the center point
features, which is denoted as $\\{(\Delta x_{init}^{i},\Delta
y_{init}^{i})|i=1,2,...,N\\}$, where $N$ is the number of vertices of the
initial contour. The initial contour vertices are computed by adding the
center point coordinates and the offsets, which is denoted as
$\\{(x_{init}^{i},y_{init}^{i})|i=1,2,...,N\\}$. Dense RepPoints regresses an
unordered point set and then converts the point set into a contour or mask
representation through complex post-processing. In contrast, E2EC directly
regresses the contour (an ordered point set) without any requirement for post-
processing. Compared with the other manually designed initial contours (e.g.,
ellipse or octagon), the learnable initial contour is closer to the ground-
truth contour. In addition, the direction of the deformation path of the
learnable initial contour is from the center point to the contour vertex (as
shown in Figure 1), guaranteeing that no unfavorable intersections between the
deformation paths to impact the convergence.

Global deformation. It is challenging to directly regress the contour vertices
with only the center point features. Meanwhile, it is also difficult to
effectively deform the contour based only on the local features of a single
contour vertex or several adjacent ones. The circular convolution proposed in
Deep Snake uses a local aggregation mechanism to supplement the global
information. However, the circular convolution operating on the local adjacent
vertices needs to be repeated multiple times to aggregate the global
information, and it cannot effectively correct large errors in the contour. We
propose a simple but more effective global aggregation mechanism named global
deformation to deform the initial contour based on both the center point
features and all of the contour vertex features. As illustrated in Figure
3(b), the features of the $N$ initial contour vertices and the center point
are first concatenated into a vector of length $(N+1)\times C$ (where $C$ is
the channel number of the vertex feature). The vector is then input into the
MLP module (channels of hidden layer and output layer is $N\times 2$) to
obtain the offset predictions of the contour vertices (a vector of length
$N\times 2$, denoted as $\\{(\Delta x_{coarse}^{i},\Delta
y_{coarse}^{i})|i=1,2,...,N\\}$). The offsets and the initial contour
coordinates are summed to obtain the adjusted coarse instance contour, which
is denoted as $\\{(x_{coarse}^{i},y_{coarse}^{i})|i=1,2,...,N\\}$. In our
experiments, we set $N=128$ and $C=64$.

Figure 6: The actual deformation paths with different numbers (M) of fixed
vertices. The blue line is the initial contour, the green line is the final
contour, and the black lines are the deformation paths.

###  3.2 Multi-direction alignment (MDA)

Due to the challenge of the contour initialization and predicted-label vertex
pairing, deviation may exist between the actual vertex deformation path and
the ideal deformation path, which leads to the adjustment of some vertices
tending toward the along-contour direction with a slower convergence speed,
and even wrong predictions. MDA addresses this problem by fixing the direction
of several selected vertices with respect to the center point, and then
samples the ground truth uniformly between the fixed vertices. The sampling
results for different numbers of alignment vertices are shown in Figure 4. MDA
can effectively reduce the learning difficulty of the contour adjustment,
without reducing the upper bound of the performance. Interestingly, PolarMask
and LSNet are two extreme cases of MDA. If we suppose that the number of
contour vertices is $N$ and the number of alignment vertices is $M$, when
$M=N$, the strategy degenerates to PolarMask, which has the lowest learning
difficulty but the lowest upper bound of the performance. When $M=0$, the
strategy degenerates to LSNet, which is the most difficult case to learn, with
a high upper bound of the performance. We experimentally found that $M=4$
obtains the best performance. When $M=4$, the learning difficulty is
significantly reduced, but the upper bound of the performance is not reduced.

###  3.3 Dynamic matching loss (DML)

As the pre-fixed vertex pairing used in previous studies is not optimal and
can cause learning difficulty, we propose DML, which dynamically adjusts the
relationship of the vertex pairing to supervise output of the last deformation
module [27] as shown in Figure 2. The loss consists of two parts: 1) the
predicted vertex points toward the nearest points on the label boundary, as
shown in Figure 5(a), and then 2) the key label vertex pulls the nearest
predicted vertex toward its position, as shown in Figure 5(b). The full
details of DML are described below.

In the best case, the vertices should be adjusted to the target contour with
the least cost. For each predicted vertex, it is a complicated process to
dynamically find the nearest correspondence in the label contour line.
Firstly, in order to simplify the calculation, the adjacent ground-truth
vertices are split into 10 equal sub-segments. The problem is then transformed
into discovering the nearest interpolated ground-truth contour vertex.
Equation (1) describes the process of matching the nearest interpolated
ground-truth vertex ($gt^{ipt}$) for each predicted contour vertex by
minimizing the L2 distance of predicted $i$-$th$ point and $x$-$th$
($0<x<N+1$) label point. Equation (2) is the corresponding and first component
of DML. Secondly, the closest predicted vertex is dynamically matched to the
key vertices (obtained by the Douglas-Peucker algorithm [8]) of the label
contour to preserve the details of the predicted contour. Equation (3)
describes the process of matching each key vertex to the nearest predicted
vertex to best preserve the details of a boundary. Equation (4) is the
corresponding and second component of DML. DML is the average of the above two
components, as shown in Equation (5). DML can greatly improve the quality of
the predicted boundary and address the over-smoothing problem found in Deep
Snake and DANCE.

$x_{i}^{*}=\mathop{\arg\min_{x}}\ \|pred_{i}^{in}-gt_{x}^{ipt}\|_{2}$ (1)
$L_{1}(pred,gt)=\frac{1}{N}\sum_{i=1}^{N}\
\|pred^{out}_{i}-gt^{ipt}_{x_{i}^{*}}\|_{1}$ (2)
$y_{i}^{*}=\mathop{\arg\min_{y}}\ \|pred_{y}^{in}-gt_{i}^{key}\|_{2}$ (3)
$L_{2}(pred,gt)=\frac{1}{n_{key}}\sum_{i=1}^{n_{key}}\
\|pred^{out}_{y_{i}^{*}}-gt^{key}_{i}\|_{1}$ (4)
$L(pred,gt)=\frac{L_{1}(pred,gt)+L_{2}(pred,gt)}{2}$ (5)

##  4 Implementation details

Detector. E2EC can be constructed based on any detector, and it is only
necessary to change the output size of the bbox branch from $H\times W\times
2$ to $H\times W\times(N\times 2)$ to directly regress the initial contour
with $N$ vertices. In the experiments conducted in this study, for a fair
comparison with the other methods, CenterNet [39] was used as the detector for
E2EC.

Loss function. Smooth L1 loss is used to supervise the contour initialization
branch, the global deformation branch and the first refinement deformation
module. The losses are defined as:

$L_{init}=\frac{1}{N}\sum_{i=1}^{N}\ smooth\
l1(\tilde{x}_{i}^{init}-x_{i}^{gt})$ (6)
$L_{coarse}=\frac{1}{N}\sum_{i=1}^{N}\ smooth\
l1(\tilde{x}_{i}^{coarse}-x_{i}^{gt})$ (7)
$L_{iter1}=\frac{1}{N}\sum_{i=1}^{N}\ smooth\
l1(\tilde{x}_{i}^{iter1},x_{i}^{gt})$ (8)

Where $N$ is the number of contour vertices, $\tilde{x}_{i}^{init}$ is the
predicted initial contour vertex, $\tilde{x}_{i}^{coarse}$ is the predicted
coarse contour vertex, $x_{i}^{gt}$ is the label contour vertex, and
$\tilde{x}_{i}^{iter1}$ is the contour vertex after being deformed with the
first deformation module in the refinement step.

The DML function is used to supervise the last deformation module, as shown in
equation (9), where $\tilde{x}_{i}^{iter2}$ is the contour vertex after being
deformed with the second deformation module. The loss of refinement
deformation modules is then defined as equation (10).

$\vspace{-3mm}L_{iter2}=L_{DML}(\tilde{x}_{i}^{iter2},x_{i}^{gt})$ (9)
$L_{iter}=L_{iter1}+L_{iter2}$ (10)

The overall loss is as follows:

$\vspace{-3mm}L_{overall}=L_{det}+\alpha L_{init}+\beta L_{coarse}+L_{iter}$
(11)

Both $\alpha$ and $\beta$ are set to 0.1. $L_{det}$ is the loss of the center
point detection.

##  5 Experiments

Method |  APmsk |  AP${}_{50}^{msk}$ |  AP${}_{70}^{msk}$ |  APbdy |  AP${}_{50}^{bdy}$ |  AP${}_{75}^{bdy}$
---|---|---|---|---|---|---
Baseline | 54.4 | 62.1 | 48.3 | 10.8 | 35.3 | 2.6
+Arch | 57.5 | 64.3 | 52.2 | 17.0 | 44.9 | 9.4
+MDA | 58.8 | 65.4 | 53.9 | 18.0 | 46.8 | 10.4
+DML | 59.2 | 65.8 | 54.1 | 19.1 | 47.9 | 11.7
(a) Ablation experiments on the SBD dataset. Baseline method is Deep Snake
[27]. Arch denotes the learnable contour initialization architecture. MDA
denotes multi-direction alignment. DML denotes dynamic matching loss.

M |  APmsk |  AP${}_{50}^{msk}$ |  AP${}_{70}^{msk}$ |  APbdy |  AP${}_{50}^{bdy}$ |  AP${}_{75}^{bdy}$
---|---|---|---|---|---|---
1 | 57.5 | 64.3 | 52.2 | 17.0 | 44.9 | 9.4
2 | 58.9 | 65.8 | 54.1 | _17.1_ | _45.8_ | _9.3_
4 | _58.8_ | _65.4_ | _53.9_ | 18.0 | 46.8 | 10.4
8 | 58.4 | 65.3 | 53.8 | 17.2 | 45.7 | 9.3
(b) Results with different alignment numbers (M). The highest accuracy is
bolded and the second-hignest accuracy is italicized.

Loss |  APmsk |  AP${}_{50}^{msk}$ |  AP${}_{70}^{msk}$ |  APbdy |  AP${}_{50}^{bdy}$ |  AP${}_{75}^{bdy}$
---|---|---|---|---|---|---
smooth l1 | 58.8 | 65.4 | 53.9 | 18.0 | 46.8 | 10.4
Chamfer | 58.2 | 65.0 | 52.9 | 18.4 | 47.4 | 11.1
DML | 59.2 | 65.8 | 54.1 | 19.1 | 47.9 | 11.7
(c) The results of the last deformation module being supervised by different
loss functions. The proposed DML outperforms smooth L1 loss and chamfer loss
in terms of both the mask and boundary quality.

Stage |  APmsk |  AP${}_{50}^{msk}$ |  AP${}_{70}^{msk}$ |  APbdy |  AP${}_{50}^{bdy}$ |  AP${}_{75}^{bdy}$ | FPS
---|---|---|---|---|---|---|---
Initial | 49.8 | 61.3 | 34.4 | 1.7 | 7.5 | 0.2 | 58
Coarse | 55.7 | 64.6 | 49.5 | 8.7 | 29.4 | 2.6 | 56
Final | 59.2 | 65.8 | 54.1 | 19.1 | 47.9 | 11.7 | 36
(d) Accuracy/speed trade-off. Figure 2 shows how E2EC generates the contours
at these different stages.

Table 1: Ablation experiments for E2EC. All models are trained on SBD train
set and tested on SBD val set, using DLA-34 backbone.

Method |  APmsk |  AP${}_{50}^{msk}$ |  AP${}_{70}^{msk}$
---|---|---|---
MNC [7] | × | 63.5 | 41.5
FCIS [18] | × | 65.7 | 52.1
STS [16] | 29.0 | 30.0 | 6.5
ESE-20 [36] | 35.3 | 40.7 | 12.1
Deep Snake [27] | 54.4 | 62.1 | 48.3
DANCE [25] | 56.2 | 63.6 | 50.4
E2EC | 59.2 | 65.8 | 54.1
Table 2: Results on SBD val set.

Method | backbone | AP |  AP50 |  AP75 | FPS
---|---|---|---|---|---
E2EC | DLA-34 | 33.8 | 52.9 | 35.9 | 30.1
E2EC∗ | DLA-34 | 31.7 | 52.2 | 32.8 | 54.3
Deep Snake [27] | DLA-34 | 30.3 | - | - | 27.3
PolarMask [33] | ResNet-101-FPN | 30.4 | 51.9 | 31.0 | 15.0
PolarMask++ [34] | ResNet-50-FPN | 30.2 | 52.6 | 30.8 | 20.5
YOLACT [2] | ResNet-101-FPN | 29.8 | 48.5 | 31.2 | 33.5
Table 3: Results obtained on the COCO $test$-$dev$. ∗ means the two
deformation modules are removed.

Datasets | Method |  APbdy |  AP${}_{50}^{bdy}$ |  AP${}_{75}^{bdy}$
---|---|---|---|---
Kins | Deep Snake | 30.2 | 53.0 | 31.2
E2EC | 33.3 | 54.9 | 35.6
SBD | Deep Snake | 10.8 | 35.3 | 2.6
E2EC | 19.1 | 47.9 | 11.7
Cityscapes | Deep Snake | 34.5 | 62.9 | 32.1
E2EC | 36.3 | 62.3 | 36.5
Table 4: Comparison of the boundary quality for the different datasets.

Method | Training data | backbone | fps |  APval | AP |  AP50 | preson | rider | car | truck | bus | train | mcycle | bicycle
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
_mask-based_ |  |  |  |  |  |  |  |  |  |  |  |  |  |
SGN [22] | Fine+coarse | VGG16 | 0.6 | 29.2 | 25.0 | 44.9 | 21.8 | 20.1 | 39.4 | 24.8 | 33.2 | 30.8 | 17.7 | 12.4
Mask R-CNN [14] | Fine | ResNet50 | 2.2 | 31.5 | 26.2 | 49.9 | 30.5 | 23.7 | 46.9 | 22.8 | 32.2 | 18.6 | 19.1 | 16.0
GMIS [24] | Fine+coarse | ResNet101 | - | - | 27.6 | 49.6 | 29.3 | 24.1 | 42.7 | 25.4 | 37.2 | 32.9 | 17.6 | 11.9
Spatial [26] | Fine | ERFNet | 11 | - | 27.6 | 50.9 | 34.5 | 26.1 | 52.4 | 21.7 | 31.2 | 16.4 | 20.1 | 18.9
PANet [23] | Fine | ResNet50 |  $<$1 | 36.5 | 31.8 | 57.1 | 36.8 | 30.4 | 54.8 | 27.0 | 36.3 | 25.5 | 22.6 | 20.8
UPSNet [35] | Fine+COCO | ResNet50 | 4.4 | 37.8 | 33.0 | 59.6 | 35.9 | 27.4 | 51.8 | 31.7 | 43.0 | 31.3 | 23.7 | 19.0
_contour-based_ |  |  |  |  |  |  |  |  |  |  |  |  |  |
Polygon RNN++ [1] | Fine | ResNet50 | - | - | 25.5 | 45.5 | 29.4 | 21.8 | 48.3 | 21.1 | 32.3 | 23.7 | 13.6 | 13.6
Deep Snake [27] | Fine | DLA-34 | - | - | 28.2 | - | - | - | - | - | - | - | - | -
E2EC | Fine | DLA-34 | 6.2 | 34.0 | 30.3 | 54.0 | 40.7 | 27.9 | 55.4 | 28.4 | 35.8 | 20.1 | 20.9 | 13.2
Deep Snake* [27] | Fine | DLA-34 | 4.6 | 37.4 | 31.7 | 58.4 | 37.2 | 27.0 | 56.0 | 29.5 | 40.5 | 28.2 | 19.0 | 16.4
E2EC* | Fine | DLA-34 | 4.9 | 39.0 | 32.9 | 59.2 | 39.0 | 27.8 | 56.0 | 29.5 | 41.2 | 29.1 | 21.3 | 19.6
Table 5: Results obtained on the Cityscapes test set. * means that the multi-
component detection is used to integrate several components into a complete
instance. The proposed E2EC method outperforms Deep Snake [27] in all the
categories.

Method |  APdet |  APmsk |
---|---|---|---
_mask-based_ |  |  |
MNC [7] | 20.9 | 18.5 |
FCIS [18] | 25.6 | 23.5 |
ORCNN [11] | 30.9 | 29.0 |
Mask R-CNN [14] | 31.3 | 29.3 |
Mask R-CNN* [28] | 32.7 | 31.1 |
PANet [23] | 32.3 | 30.4 |
PANet* [28] | 33.4 | 32.2 |
VRS&SP [32] | × | 32.1 |
ARCNN [38] | × | 32.9 |
_contour-based_ |  |  |
Deep Snake [27] | 32.8 | 31.3 |
E2EC | 36.5 | 34.0 |
Table 6: Results obtained on the KINS test set. * denotes with ASN proposed by
[28].

###  5.1 Datasets and metrics

Datasets. The KINS [28], SBD [13], Cityscapes [6], and COCO [20] datasets were
used in the experiments. The KINS dataset is used for amodal instance
segmentation, and has seven instance classes, with 7,474 training images and
7,517 testing images. The SBD dataset has 20 instance classes and is split
into 5,623 training images and 5,732 testing images. The SBD dataset is made
up of 11,355 reannotated images from the PASCAL VOC [10] dataset, with
instance-level boundaries. The Cityscapes dataset has eight instance classes
and contains 2,975 training, 500 validation, and 1,525 testing images with
high-quality annotations. The COCO dataset has eighty instance classes and
contains 115k training, 5k validation, and 20k testing images.

Metrics. In this paper, the mask quality is evaluated in terms of the standard
AP metric. To distinguish the standard AP metric from other metrics, it is
denoted as APmsk. For all the datasets, all the settings of APmsk were the
same as for Deep Snake.

The boundary quality is evaluated in terms of the boundary AP metric proposed
by [5]. This is denoted as APbdy and focuses on the boundary quality.

###  5.2 Ablation experiments

To quantitatively analyze the effect of each component of the proposed E2EC
method and verify the design details, we conducted ablation experiments on the
SBD dataset, with Deep Snake as the baseline. The network was trained end-to-
end for 150 epochs with multi-scale data augmentation. The learning rate
started from 1e-4 and was decayed by 0.5 at 80 and 120 epochs. The results of
the ablation experiments are given in Table 1.

The learnable contour initialization architecture. When the learnable contour
initialization architecture is used to replace the contour initialization
method of the baseline method, this yields improvements of 3.1 APmsk, 2.2
AP${}_{50}^{msk}$ and 3.9 AP${}_{70}^{msk}$. For the APbdy metric, which is
sensitive to the boundary quality, the learnable contour initialization
architecture brings improvements of 6.2 APbdy, 9.6 AP${}_{50}^{bdy}$, and 6.8
AP${}_{75}^{bdy}$. Such a huge improvement in AP fully proves the importance
of a reasonable initial deformation path for the contour-based method.

Multi-direction alignment. When the uniform label sampling scheme used by the
baseline method is replaced with the MDA strategy, this yields improvements of
1.3 APmsk, 1.1 AP${}_{50}^{msk}$ and 1.7 AP${}_{70}^{msk}$ over the baseline
with the learnable contour initialization architecture. For the APbdy metric,
also results in improvements of 1.0 APbdy, 1.1 AP${}_{50}^{bdy}$ and 1.3
AP${}_{75}^{bdy}$. It can be seen in Figure 6 that, as the number of
alignments increases, the deviation between the actual and ideal deformation
paths becomes smaller and smaller, and the actual deformation path becomes
more optimal. We also conducted quantitative experiments with different
alignment numbers, to explore the most appropriate number of alignment
directions. The ablation results are shown in Table 1(b). The most appropriate
alignment number is 4, which achieves the highest APbdy and the second-highest
APmsk. As the number increases further, the difficulty of the learning
decreases, but too many alignment operations also affects the upper bound of
the performance and exacerbates the unevenness of the vertex distribution, as
shown in Figure 4. The proposed MDA label sampling scheme makes a good balance
between the learning difficulty and the upper bound of the performance.

Figure 7: The prediction contours at different stages. The initial contour of
E2EC fits the instance better than the octagonal initialization of Deep Snake.
The quality of the coarse contour predicted by E2EC is comparable to the final
contour predicted by Deep Snake. The final contour predicted by E2EC with DML
perfectly outlines the motorcycle.

Dynamic matching loss. Smooth L1 loss, chamfer loss [15], and DML were used to
supervise the last deformation module. The results obtained with the different
loss functions are shown in Table 1(c). Replacing smooth L1 loss with chamfer
loss brings about a deterioration of 0.6 APmsk and an improvement of 0.4
APbdy. Chamfer loss improves the quality of the predicted boundary, but also
reduces the quality of the mask, because the order of the contour vertices has
been destroyed. The proposed DML function can dynamically match all the
predicted vertices with the most appropriate label vertices. DML yields
improvements of 0.4 APmsk and 1.1 APbdy. The prediction results obtained with
and without DML are also illustrated in Figure 7, where it can be seen that
DML significantly improves the fitting degree between the predicted contour
and the instance (such as the motorcycle helmet). As shown in Figure 8, the
model supervised by DML predicts the best contour, without destroying the
order of the vertices.

Figure 8: The prediction results supervised by different loss functions.
Chamfer loss slightly improved the quality of boundary details but produced a
serious jagged phenomenon (especially the red and purple instances). Dynamic
matching loss heavily improved the quality of boundary details and did not
cause any harm.

Speed $vs.$ accuracy. The accuracy and inference speed of the contours at
different stages are listed in Table 1(d), and Figure 7 illustrates the
predicted contour at different stages. E2EC can generate an initial contour at
a high speed, but the quality of the result is rough for instance
segmentation. The coarse contour is obtained from the global deformation of
the initial contour. The coarse contour predicted by E2EC has the same
accuracy as the final output of Deep Snake. The inference speed of 56 fps on a
512×512 image far exceeds the speed of Deep Snake (33 fps). The complete E2EC
achieves an inference speed of 36 fps, which is slightly higher than Deep
Snake, and the final contour predicted by E2EC has a huge advantage in terms
of both the mask and boundary quality. The experimental results show that E2EC
predicts the coarse contour 55.5% faster than the final contour. However,
APmsk only reduced by 3.5. Therefore, in the case of the requirement for
extremely high speed, removing the deformation components can be a good
choice.

Figure 9: The qualitative results obtained by E2EC. The first row shows some
examples of the prediction results for the SBD dataset, the second row shows
some examples of the prediction results for the KINS dataset, and the third
row shows some examples of the prediction results for the Cityscapes dataset.

###  5.3 Comparison with the state-of-the-art methods

Performance on the KINS dataset. The KINS dataset is used for amodal instance
segmentation, and is annotated with inference completion information for the
occluded parts of the instances. E2EC was trained for 150 epochs with the Adam
optimizer, and the learning rate started from 1e-4 and was decayed by 0.5 at
80 and 120 epochs. Multi-scale training was conducted, and the models were
tested at a single resolution of 768×2496.

In Table 6, the performance of the different instance segmentation methods on
the KINS dataset is compared. E2EC does not use the bbox branch to generate
the initial contour, so the detection result is calculated from the predicted
contour. E2EC achieves a state-of-the-art performance in both the detection
and segmentation tasks. E2EC achieves 36.5 APdet, 34.0 APmsk and 33.3 APbdy
(see 4), and outperforms Deep Snake by 3.7 APdet, 2.7 APmsk and 3.1 APbdy.
Figure 9 shows some representative results of E2EC obtained on the KINS
dataset.

Performance on the SBD dataset. For the SBD dataset, the details of the model
training were the same as for the KINS dataset. Multi-scale training was
conducted and the models were tested at a single resolution of 512×512.

The SBD dataset has more instance categories and more complicated instance
contours than the KINS dataset, so that the advantage of the proposed E2EC
method with more optimal deformation paths is more obvious. E2EC outperforms
Deep Snake by 4.8 APmsk (see Table 4) and 8.3 APbdy (see Table 4). The contour
details predicted by E2EC are remarkable, and AP${}_{75}^{msk}$ is 9.1 higher
than Deep Snake. E2EC uses the lighter DLA-34 as the backbone, and it
outperforms the fully convolutional instance-aware semantic segmentation
(FCIS) method, which uses ResNet-101 as the backbone, by 2.0
AP${}_{75}^{msk}$. Figure 9 shows some examples of the results of E2EC on the
SBD dataset. It can be seen that instances with complex contours, such as legs
and chairs, are well segmented by the proposed method, but they cannot be well
segmented by Deep Snake.

Performance on the Cityscapes dataset. The details of the model training for
the Cityscapes dataset were the same as for the previous two datasets. It is
worth mentioning that it is not necessary to train the detector separately
with E2EC, but it is necessary to train the detector alone for 140 epochs and
then train the whole network end-to-end for 200 epochs with Deep Snake. Multi-
scale training was conducted, and the models were tested at a single
resolution of 1216×2432.

Table 6 compares the results of the proposed E2EC method with those of the
other state-of-the-art methods on the Cityscapes validation and test sets.
E2EC achieves 30.3 APmsk on the test set, without the multi-component
detection used in [27], outperforming Deep Snake by 2.1 APmsk. With the multi-
component detection strategy, the proposed method achieves 39.0 APmsk on the
validation set and 32.9 APmsk on the test set, outperforming Deep Snake by 1.6
APmsk and 1.2 APmsk respectively. E2EC also outperforms the classic mask-based
method of PANet by 2.5 APmsk on the validation set and 1.1 APmsk on the test
set, and is almost five times faster. The performance of the proposed method
is on a par with UPSNet [36]; however, the latter was trained on the
additional large Common Objects in Context (COCO) dataset [20].

Table 4 compares the boundary quality of the results obtained by the proposed
E2EC method and Deep Snake. E2EC achieves 36.3 APbdy and 36.5
AP${}_{75}^{bdy}$ on the validation set, outperforming Deep Snake by 1.8 APbdy
and 4.4 AP${}_{75}^{bdy}$. Some of the results obtained by E2EC are shown in
Figure 9, where it can be seen that the proposed method segments the details
of the instances well (such as the rearview mirror and tires), which cannot be
achieved by Deep Snake.

Performance on the COCO dataset. For the COCO dataset, the model is trained
end-to-end for 140 epochs with Adam optimizer and 20 epochs with SGD
optimizer. The learning rate starts from 1e-4 and drops by half at 80 and 120
epochs, respectively. The model tested at the original image resolution
without any tricks.

As shown in Table 4, E2EC achieves 33.8 APmsk with 30.1 fps, which outperforms
Deep Snake by 3.5 APmsk. When the deformation modules are removed, E2EC∗
achieves 31.7 APmsk with 54.3 fps, which is almost 2$\times$ faster than Deep
Snake and outperforms Deep Snake by 1.4 APmsk.

##  6 Conclusion

In this paper, we have proposed an end-to-end contour-based instance
segmentation method named E2EC. E2EC introduces three novel components: 1) the
learnable contour initialization architecture; 2) the multi-direction
alignment (MDA) label sampling scheme; and 3) the dynamic matching loss (DML)
function. E2EC greatly improves the contour extraction quality of the contour-
based instance segmentation. In this study, E2EC achieved state-of-the-art
results on the KINS, SBD, and Cityscapes datasets, with a beyond real-time
performance. We also introduced a faster variant, where, by only retaining the
learnable contour initialization architecture, the accuracy can be comparable
to that of Deep Snake, and the speed is almost as fast as that of the backbone
one-stage detector CenterNet. The modules proposed in E2EC could be easily
applied to other contour-based instance segmentation methods. We hope that
E2EC will serve as a fundamental and strong baseline for contour-based
instance segmentation research.

## References

  * [1] David Acuna, Huan Ling, Amlan Kar, and Sanja Fidler.  Efficient interactive annotation of segmentation datasets with polygon-rnn++.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 859–868, 2018.
  * [2] Daniel Bolya, Chong Zhou, Fanyi Xiao, and Yong Jae Lee.  Yolact: Real-time instance segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9157–9166, 2019.
  * [3] Hao Chen, Kunyang Sun, Zhi Tian, Chunhua Shen, Yongming Huang, and Youliang Yan.  Blendmask: Top-down meets bottom-up for instance segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8573–8581, 2020.
  * [4] Xinlei Chen, Ross Girshick, Kaiming He, and Piotr Dollár.  Tensormask: A foundation for dense object segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2061–2069, 2019.
  * [5] Bowen Cheng, Ross Girshick, Piotr Dollár, Alexander C Berg, and Alexander Kirillov.  Boundary iou: Improving object-centric image segmentation evaluation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15334–15342, 2021.
  * [6] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213–3223, 2016.
  * [7] Jifeng Dai, Kaiming He, and Jian Sun.  Instance-aware semantic segmentation via multi-task network cascades.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3150–3158, 2016.
  * [8] David H Douglas and Thomas K Peucker.  Algorithms for the reduction of the number of points required to represent a digitized line or its caricature.  Cartographica: the international journal for geographic information and geovisualization, 10(2):112–122, 1973.
  * [9] Kaiwen Duan, Lingxi Xie, Honggang Qi, Song Bai, Qingming Huang, and Qi Tian.  Location-sensitive visual recognition with cross-iou loss.  arXiv preprint arXiv:2104.04899, 2021.
  * [10] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  International journal of computer vision, 88(2):303–338, 2010.
  * [11] Patrick Follmann, Rebecca König, Philipp Härtinger, Michael Klostermann, and Tobias Böttger.  Learning to see the invisible: End-to-end trainable amodal instance segmentation.  In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1328–1336. IEEE, 2019.
  * [12] Naiyu Gao, Yanhu Shan, Yupei Wang, Xin Zhao, Yinan Yu, Ming Yang, and Kaiqi Huang.  Ssap: Single-shot instance segmentation with affinity pyramid.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 642–651, 2019.
  * [13] Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik.  Semantic contours from inverse detectors.  In 2011 International Conference on Computer Vision, pages 991–998. IEEE, 2011.
  * [14] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.
  * [15] Namdar Homayounfar, Wei-Chiu Ma, Shrinidhi Kowshika Lakshmikanth, and Raquel Urtasun.  Hierarchical recurrent attention networks for structured online maps.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3417–3426, 2018.
  * [16] Saumya Jetley, Michael Sapienza, Stuart Golodetz, and Philip HS Torr.  Straight to shapes: real-time detection of encoded shapes.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6550–6559, 2017.
  * [17] Youngwan Lee and Jongyoul Park.  Centermask: Real-time anchor-free instance segmentation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13906–13915, 2020.
  * [18] Yi Li, Haozhi Qi, Jifeng Dai, Xiangyang Ji, and Yichen Wei.  Fully convolutional instance-aware semantic segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2359–2367, 2017.
  * [19] Justin Liang, Namdar Homayounfar, Wei-Chiu Ma, Yuwen Xiong, Rui Hu, and Raquel Urtasun.  Polytransform: Deep polygon transformer for instance segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9131–9140, 2020.
  * [20] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [21] Huan Ling, Jun Gao, Amlan Kar, Wenzheng Chen, and Sanja Fidler.  Fast interactive object annotation with curve-gcn.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5257–5266, 2019.
  * [22] Shu Liu, Jiaya Jia, Sanja Fidler, and Raquel Urtasun.  Sgn: Sequential grouping networks for instance segmentation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3496–3504, 2017.
  * [23] Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia.  Path aggregation network for instance segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8759–8768, 2018.
  * [24] Yiding Liu, Siyu Yang, Bin Li, Wengang Zhou, Jizheng Xu, Houqiang Li, and Yan Lu.  Affinity derivation and graph merge for instance segmentation.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 686–703, 2018.
  * [25] Zichen Liu, Jun Hao Liew, Xiangyu Chen, and Jiashi Feng.  Dance: A deep attentive contour model for efficient instance segmentation.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 345–354, 2021.
  * [26] Davy Neven, Bert De Brabandere, Marc Proesmans, and Luc Van Gool.  Instance segmentation by jointly optimizing spatial embeddings and clustering bandwidth.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8837–8845, 2019.
  * [27] Sida Peng, Wen Jiang, Huaijin Pi, Xiuli Li, Hujun Bao, and Xiaowei Zhou.  Deep snake for real-time instance segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8533–8542, 2020.
  * [28] Lu Qi, Li Jiang, Shu Liu, Xiaoyong Shen, and Jiaya Jia.  Amodal instance segmentation with kins dataset.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3014–3023, 2019.
  * [29] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He.  Fcos: Fully convolutional one-stage object detection.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 9627–9636, 2019.
  * [30] Xinlong Wang, Tao Kong, Chunhua Shen, Yuning Jiang, and Lei Li.  Solo: Segmenting objects by locations.  In European Conference on Computer Vision, pages 649–665. Springer, 2020.
  * [31] Fangyun Wei, Xiao Sun, Hongyang Li, Jingdong Wang, and Stephen Lin.  Point-set anchors for object detection, instance segmentation and pose estimation.  In European Conference on Computer Vision, pages 527–544. Springer, 2020.
  * [32] Yuting Xiao, Yanyu Xu, Ziming Zhong, Weixin Luo, Jiawei Li, and Shenghua Gao.  Amodal segmentation based on visible region segmentation and shape prior.  arXiv preprint arXiv:2012.05598, 2020.
  * [33] Enze Xie, Peize Sun, Xiaoge Song, Wenhai Wang, Xuebo Liu, Ding Liang, Chunhua Shen, and Ping Luo.  Polarmask: Single shot instance segmentation with polar representation.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12193–12202, 2020.
  * [34] Enze Xie, Wenhai Wang, Mingyu Ding, Ruimao Zhang, and Ping Luo.  Polarmask++: Enhanced polar representation for single-shot instance segmentation and beyond.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021\.
  * [35] Yuwen Xiong, Renjie Liao, Hengshuang Zhao, Rui Hu, Min Bai, Ersin Yumer, and Raquel Urtasun.  Upsnet: A unified panoptic segmentation network.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8818–8826, 2019.
  * [36] Wenqiang Xu, Haiyang Wang, Fubo Qi, and Cewu Lu.  Explicit shape encoding for real-time instance segmentation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5168–5177, 2019.
  * [37] Ze Yang, Yinghao Xu, Han Xue, Zheng Zhang, Raquel Urtasun, Liwei Wang, Stephen Lin, and Han Hu.  Dense reppoints: Representing visual objects with dense point sets.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16, pages 227–244. Springer, 2020.
  * [38] Xunli Zeng and Jianqin Yin.  Amodal segmentation just like doing a jigsaw.  arXiv preprint arXiv:2107.07464, 2021.
  * [39] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl.  Objects as points.  arXiv preprint arXiv:1904.07850, 2019.


$c$-numerical ranges. For example, if $c=(c_{1},\dots,c_{k})$ has distinct
entries, then $W_{c}({\mathbf{A}})$ has a conical boundary point
$(a_{1},\dots,a_{m})$ will imply that there is a unitary
$U\in{{\mathcal{B}}({\mathcal{H}})}$ such that $U^{*}A_{j}U=D_{j}\oplus B_{j}$
for $j=1,\dots,m$, where $D_{j}\in M_{k}$ is a diagonal matrix satisfying
$({\rm tr}\,(CD_{1}),\dots,{\rm tr}\,(CD_{m}))=(a_{1},\dots,a_{m})$ with
$C={\rm diag}\,(c_{1},\dots,c_{k})$. It would be interesting to extend other
results to $W_{c}({\mathbf{A}})$. For example, it was shown in [24] for
$c=(c_{1},\dots,c_{n})$ with entries arranged and descending order and
$c_{k}>c_{k+1}$, where $|n/2-k|\leq 1$, then $A\in{\mathbf{M}}_{n}$ satisfies
$W_{c}(A)$ is polyhedral if and only if $A$ is normal. One can readily show
that $W_{c}({\mathbf{A}})$ is polyhedral for $(c_{1},\dots,c_{n})$ satisfying
the above condition and ${\mathbf{A}}=(A_{1},\dots,A_{m})\in M_{n}^{m}$ if and
only if $\\{A_{1},\dots,A_{m}\\}$ is a commuting family of normal matrices; it
follows that $W_{d}({\mathbf{A}})$ is polyhedral for all
$d=(d_{1},\dots,d_{n})\in{\mathbb{R}}^{n}$. It would be interesting to extend
the results to $W_{c}({\mathbf{A}})$ for
${\mathbf{A}}\in{{\mathcal{B}}({\mathcal{H}})}^{m}$ or
${\mathbf{A}}\in{{\mathcal{K}}({\mathcal{H}})}^{m}$.

In [10], the authors studied special boundary points of $W({\mathbf{A}})$ such
as conical points, bare points, or points with some extreme norm properties.
We have extended their result for conical points to $W_{k}({\mathbf{A}})$. It
is interesting to extend other results in the paper to $W_{k}({\mathbf{A}})$
or $W_{c}({\mathbf{A}})$.

## Acknowledgement

Li is an affiliate member of the Institute for Quantum Computing, University
of Waterloo; his research was partially supported by the Simons Foundation
Grant 851334.

## References

  * [1] Y.H. Au-Yeung and Y.T. Poon, A remark on the convexity and positive definiteness concerning Hermitian matrices, Southeast Asian Bull. Math. 3 (1979), 85–92.
  * [2] Y.H. Au-Yueng and N.K. Tsing, An extension of the Hausdorff-Toeplitz theorem on the numerical range, Proc. Amer. Math. Soc. 89 (1983), 215–218.
  * [3] G. De Barra, J.R. Giles and B. Sims, On the numerical range of compact operators on Hilbert spaces, J. London Math. Soc. 5 (1972), 704-706.
  * [4] P. Binding, D. R. Farenick, and C.K. Li, A dilation and norm in several variable operator theory, Canad. J. Math. 47 (1995), 449-461.
  * [5] J.C. Bourin, Compressions and pinchings, J. of Operator Theory, 50 (2003), 211–220.
  * [6] J.J. Buoni and B.L. Wadwa, On joint numerical ranges, Pacific J. Math. 77 (1978), 303-306.
  * [7] J.T. Chan, A note on the boundary of the joint numerical range, Linear Multilinear Algebra 66 (2018), 821–826.
  * [8] J.T. Chan, C.K. Li and Y.T. Poon, Closedness of the $k$-numerical range, Linear Multilinear Algebra (2020), https://doi.org/10.1080/03081087.2020.1790483.
  * [9] M. Cho, Joint spectra of commuting normal operators on Banach spaces, Glasgow Math. J. 30 (1988) 339-345.
  * [10] M. Cho and M. Takaguchi, Boundary points of joint numerical ranges, Pacific. J. Math. 95 (1981), 27-35.
  * [11] W.D. Cook and R.J. Webster, Carathéodory Theorem, Canad. Math. Bulletin 15 (1972), 293.
  * [12] G. Greiner and W.J. Ricker, Commutativity of compact selfadjoint operators, Studia Math. 112 (1995), 109-125.
  * [13] K. Fan and G. Pall, Imbedding conditions for Hermitian and normal matrices. Canadian J. Math. 9 (1957), 298–304.
  * [14] P.A. Fillmore, J.G. Stampfli, and J.P. Williams, On the essential numerical range, the essential spectrum and a problem of Halmos, Acta Sci. Math. (Szeged), 33 (1973), 172–192.
  * [15] P.R. Halmos, A Hilbert Space Problem Book, 2nd ed., Graduate Texts in Mathematics, 19, Springer-Verlag, New-York, 1982.
  * [16] R.A. Horn and C.R. Johnson, Topics in Matrix Analysis, Cambridge University Press, Cambridge, 1991.
  * [17] P. Juneja, On extreme points of the joint numerical ranges of commuting normal operators, Pacific J. Math. 67 (1976), 473-476.
  * [18] R.V. Kadison, The Pythagorean Theorem: II. The infinite discrete case, PNAS 99 (2002), 5217-5222.
  * [19] J.S. Lancaster, The boundary of the numerical range, Proc. Amer. Math. Soc. 49 (1975), 393-398.
  * [20] C.K. Li and Y.T. Poon, Some results on the $c$-numerical range, in Five decades as a mathematician and educator, 247–258, World Sci. Publ., River Edge, NJ, 1995.
  * [21] C.K. Li and Y.T. Poon, Convexity of the joint numerical range, SIAM J. Matrix Analysis Appl. 21 (2000), 668-678.
  * [22] C.K. Li and Y.T. Poon, The joint essential numerical range of operators: convexity and related results, Studia Math. 194 (2009), 91-104.
  * [23] C.K. Li, Y.T. Poon and Y. Wang, Joint numerical ranges and commutativity of matrices, J. Math. Anal. Appl. 491 (2020), no. 1, 124310.
  * [24] C.K. Li, C.H. Sung, and N.K. Tsing, $c$-Convex matrices: Characterizations, inclusion relations and normality, Linear Multilinear Algebra 25 (1989), 275-287.
  * [25] T. Mao, Y. Qiao and P. Wang, Comutativity of normal compact operators via projective spectrum, Proc. Amer. Math. Soc. 146 (2018), 1165-1172.
  * [26] M. Marcus and I. Filippenko, Nondifferentiable boundary points of the higher numerical range, Linear Algebra Appl. 21 (1978), 217–232.
  * [27] M. Marcus, M. Moyls and I. Filippenko, Normality and the higher numerical range, Canad. J. Math. 30 (1978), 419-430.
  * [28] A. W. Marshall and I. Olkin, Inequalities: theory of majorization and its applications, volume 143. Springer, 1979.
  * [29] V. Müller and Y. Tomilov, Joint numerical ranges and compressions of powers of operators, J. London Math. Soc. 99 (2019), 127-152.

[Chan]<EMAIL_ADDRESS>
[Li] Department of Mathematics, The College of William & Mary, Williamsburg,
VA 13185, USA<EMAIL_ADDRESS>
[Poon] Department of Mathematics, Iowa State University, Ames, IA 50011, USA.
<EMAIL_ADDRESS>


# Quantum confined Rydberg excitons in Cu2O nanoparticles

Konstantinos Orfanakis  Sai Kiran Rajendran  Hamid Ohadi  ho35@st-
andrews.ac.uk SUPA, School of Physics and Astronomy, University of St Andrews,
St Andrews, KY16 9SS, United Kingdom  Sylwia Zielińska-Raczyńska  Gerard
Czajkowski  Karol Karpiński  David Ziemkiewicz  UTP University of Science and
Technology, Bydgoszcz, Poland

(August 27, 2024)

###### Abstract

The quantum confinement of Rydberg excitons is an important step towards
exploiting their large nonlinearities for quantum applications. We observe
Rydberg excitons in natural nanoparticles of Cu2O. We resolve up to the
principal quantum number $n=12$ in a bulk Cu2O crystal and up to $n=6$ in
nanoparticles extracted from the same crystal. The exciton transitions in
nanoparticles are broadened and their oscillator strengths decrease as
$\propto n^{-4}$ compared to those in the bulk (decreasing as $\propto
n^{-3}$). We explain our results by including the effect of quantum
confinement of exciton states in the nanoparticles. Our results provide an
understanding of the physics of Cu2O Rydberg excitons in confined dimensions.

##  I Introduction

Solid-state quantum systems provide unprecedented capabilities for the
realization of novel devices owing to their robustness, miniaturization
capability and scalability Keyes (1988); Awschalom  _et al._ (2013); Devoret
and Schoelkopf (2013); Loredo  _et al._ (2016); Mak and Shan (2016); Gonzalez-
Zalba (2018). The operation of such devices requires developing means to
efficiently produce, control and detect strongly interacting particles.
Excitons, elementary excitations in semiconductors consisting of a Coulomb-
bound pair of an electron and a hole, are considered major candidates towards
this direction. An exciton represents a solid-state analog of the hydrogen
atom and hence excited states can be observed as a hydrogen-like discrete
series at energies $R_{y}/n^{2}$ below the bandgap, with $R_{y}$ the Rydberg
energy and $n$ the principal quantum number Fox (2011). Excitons in cuprous
oxide (Cu2O) were observed as early as 1952 Hayashi and Katsuki (1952), and
their various physical properties have been studied since then Meyer  _et al._
(2013). However only energy levels up to $n=8$ in Cu2O were observed for many
decades until the Rydberg spectrum was extended to $n=12$ in 1996 Matsumoto
_et al._ (1996).

Among semiconductors, Cu2O has the advantage of a large Rydberg energy which
allows access to much higher excited states. In a recent high-resolution laser
absorption study Kazimierczuk  _et al._ (2014), it was shown that Cu2O hosts
Rydberg excitons up to $n=25$. This demonstration opened the portal to the
field of giant Rydberg excitons in solid-state Thewes  _et al._ (2015);
Grünwald  _et al._ (2016); Heckötter  _et al._ (2017a, b); Takahata and Naka
(2018); Krüger and Scheel (2018); Lynch  _et al._ (2018); Mund  _et al._
(2019); Zielińska-Raczyńska  _et al._ (2019), in close analogy with their
highly excited counterparts in atomic Physics Jones  _et al._ (2017). Owing to
their giant microscopic dimensions (up to $\sim
1\text{\,}\mathrm{\SIUnitSymbolMicro m}$) leading to the onset of exciton
blockade, Rydberg excitons in Cu2O exhibit enhanced optical nonlinearities at
much smaller densities compared with other traditional semiconductors
Kazimierczuk  _et al._ (2014); Walther  _et al._ (2018). These nonlinearities
can be harnessed by quantum confinement of the excitons in semiconductor low-
dimensional structures such as quantum wells and quantum dots Konzelmann  _et
al._ (2020). Studying these excitons in confined dimensions is a crucial step
towards harnessing these nonlinearities for applications. Nanoparticles Smith
and Nie (2010) are naturally a suitable system for quantum confinement and an
interesting platform for realising quantum technologies with Rydberg excitons,
potentially underpinning future devices such as single-photon sourcesKhazali
_et al._ (2017) and single-photon switchesBaur  _et al._ (2014).

In this work, we report the observation of Rydberg excitons in nanoparticles
of Cu2O. We resolve the yellow $p$-exciton states in natural nanoparticles up
to $n=6$, while we observe up to $n=12$ in the bulk crystal that the particles
were extracted from. We show that the reduction of the crystal structure from
bulk to nanoparticles leads to broadened linewidths and an apparent reduction
of the oscillator strength of the excitonic peaks compared to that in the
bulk. We describe both effects by the quantum confinement of the Rydberg
states and the size distribution of our nanoparticles. Our results are the
first demonstration of the effect of quantum confinement in linewidth and
oscillator strength of Rydberg excitons.

##  II Experimental Methods

Figure 1: (a) SEM image of a cluster of particles. (b) Particle size
distribution. The fitted distribution is depicted as a blue solid line. The
mean diameter is calculated as $\simeq 1.1$ $\mu$m, while the median as
$\simeq 0.7$ $\mu$m.

Cu2O nanoparticles were extracted from the residual powder from polishing a
natural crystal mined in Tsumeb, Namibia. After forming a suspension of Cu2O
powder in water, a droplet of this suspension is deposited onto a CaF2
substrate. Scanning electron microscopy (SEM) reveals that Cu2O particles tend
to aggregate and form a thin layer due to Van der Waals forces as the
particles redistribute during water evaporation [Fig. 1(a)]. The particles are
of various sizes and shapes with a average and median diameter size of $\simeq
1.1$ and $\simeq 0.7$ nm [Fig. 1(b)]. We refer to these particles as natural
nanoparticles (NNPs). Nanoparticles are compared to a thin slab of natural
bulk crystal that is cut and mechanically polished down to a thickness of
$\leavevmode\nobreak\ \sim 60\text{\,}\mathrm{\SIUnitSymbolMicro m}$.

We perform broadband transmission spectroscopy (see Supplementary Information
1 for setup). The excitation source is a green-yellow light-emitting diode
(LED) with a center wavelength of $554\text{\,}\mathrm{nm}$ (Thorlabs MINTF4).
The resulting signal is collected, dispersed and analyzed in a spectrometer
(Andor Shamrock 750) coupled to a CCD camera. For transmission spectroscopy,
an objective lens (20$\times$ Mitutoyo Plan Apo, NA = 0.42) focuses the
excitation light to a spot on our sample ($\sim$100 $\mu$m diameter) and a
second objective lens (same NA as excitation) collimates the transmitted light
before it enters the spectrometer. The sample is maintained at
$4\text{\,}\mathrm{K}$ using a liquid-helium flow cryostat. We use CaF2
substrates because of their excellent thermal conductivity at cryogenic
temperatures and transparency in the visible spectrum.

##  III Experimental results

Figure 2: Transmission spectrum for a natural crystal of Cu2O at
$4\text{\,}\mathrm{K}$, where excitons up to $n=12$ are resolved (red line)
and for natural nanoparticles (blue line).

The transmission spectrum of the bulk crystal reveals a series of absorption
lines corresponding to the excited states of Rydberg excitons in Cu2O [red
line in Fig. 2(b)]. These states are labelled by their principal quantum
number, $n$. We can reliably identify Rydberg states up to $n=12$ overlaid on
a continuous phonon background Baumeister (1961); Jolk and Klingshirn (1998).
The reduced number of exciton lines observed in our experiment compared to
previous works Kazimierczuk  _et al._ (2014) is primarily attributed to the
small diameter and broadband spectrum of our excitation source. The asymmetric
lineshape stems from the Fano interference between the discrete excitonic
states and the absorption background originating from phonon-assisted
absorption of the 1$s$ exciton Toyozawa (1964). By fitting the exciton
resonance energies to the Rydberg formula, we extract the bandgap energy
$E_{g}=$2.173\text{\,}\mathrm{eV}$$ and the Rydberg energy
$R_{y}=$97.1\text{\,}\mathrm{meV}$$, which agree with those reported in the
literature Agekyan  _et al._ (1974); Matsumoto  _et al._ (1996).

The transmission spectra of the nanoparticles covering a large area ($\sim$100
$\mu$m diameter) on the substrate show clear exciton peaks [blue line in Fig.
2(b)]. Exciton resonances are evident as discrete asymmetric dips on top of
the phonon background. A noticeable difference between the spectra of clusters
and that of the thinned bulk crystal is that, in the former, is the sharp
decrease in the peak absorption of the excitons as $n$ increase, such that we
can only resolve resonances up to $n=6$.

Figure 3: Peak areas (a) and linewidths (b) calculated from experimental
(dots) and theoretical results (lines). Peak areas are normalized to the value
of $n$ = 2 in each case.

Repeating the measurements on synthetic nanoparticles (SNPs) of comparable
sizes can show if the reduced number of observed excitonic resonances in NNPs
could be due to the polishing procedure. The transmission spectrum for SNPs,
exhibits the same number of resonances as the one for NNPs (see Supplementary
Information 2). Therefore, we rule out polishing as the reason for the
observation of the reduced number of transitions. We note that resonances with
the same $n$ are slightly blueshifted in the bulk crystal compared to those in
nanoparticles due to a small temperature variation in each experiment (see
Supplementary Fig. 3).

The comparison of oscillator strength (peak area) of the excitons in bulk and
NPs shows (see Fig. 3a) that the relative peak area of excitons in
nanoparticles of Cu2O decreases as $\propto n^{-4}$ compared to that in the
bulk (decreasing as $\propto n^{-3}$, which is the theoretical dependence).
Individual peaks were fitted with an asymmetric Lorentzian Kazimierczuk  _et
al._ (2014) to extract their linewidth. The linewidth of NPs is approximately
double the linewidth of the bulk thin crystal for the first three observed
resonances. The broadening is more pronounced for $n=5$ as the linewidth
increases by nearly three folds for NPs (Fig. 3b).

Previously, the broadening of excitons in nanoparticles of CuCl
($<15\text{\,}\mathrm{nm}$ diameter) was found to be due to quantum
confinement and size distribution Ekimov  _et al._ (1985, 1986). Here, we show
that quantum confinement results in the broadening of the transitions in a
similar process, however, due to the narrow linewidth of the transitions the
effect is visible for larger nanoparticle sizes.

##  IV Theoretical description

Figure 4: Experimental (a),(c) and theoretical (b),(d) absorption spectra of
bulk Cu2O NPs in the full energy range and in the vicinity of $n=5$ state.

From the point of view of the classification of low-dimensional semiconductor
structures, the considered nano-crystals are quantum dots (QDs). The SEM
images (Fig. 1) suggest that the best model shape is a spherical quantum dot,
with a median diameter of $\sim$700 nm. Assuming that the interaction between
dots is small enough, we may consider interband transitions in an isolated QD,
and then average the results over a size distribution of QDs. As it has been
shown previouslyZiemkiewicz  _et al._ (2020), the optical properties of QDs
can be studied with the real density matrix approach (RDMA). In this method,
the optical response (absorption, reflection, transmission) is obtained by
solving a set of equations for the excitonic amplitude
$Y(\textbf{r}_{e},\textbf{r}_{h},t)$ and electric field vector
$\textbf{E}(\textbf{R},t)$ of the wave propagating in the nanostructure. The
quantities $\textbf{r}_{e},\textbf{r}_{h}$ are the coordinates of the electron
and the hole, and R is the center-of-mass coordinate of the electron-hole
pair. The basic equations of RDMA have the form Ziemkiewicz  _et al._ (2020)

$-i(\hbar\partial_{t}+{\mit\Gamma})Y(\textbf{r}_{e},\textbf{r}_{h},t)+H_{eh}Y(\textbf{r}_{e},\textbf{r}_{h})={\bf
M}{\bf E},$ (1)

where $\mit\Gamma$ is a phenomenological damping coefficient, ${\bf
M}(\textbf{r})$ is a smeared-out transition dipole density, $E_{g}$ is the
fundamental gap, and $\textbf{r}=\textbf{r}_{e}-\textbf{r}_{h}$ is the
relative electron-hole distance. The operator $H_{eh}$ stands for the two-band
effective mass Hamiltonian, which includes the electron and hole kinetic
energy, the electron-hole interaction potential and the confinement
potentials. In consequence, the Hamiltonian $H_{eh}$ is given by

$\displaystyle H_{eh}=E_{g}+\frac{{\bf p}_{h}^{2}}{2m_{h}}$
$\displaystyle+\frac{{\bf
p}_{e}^{2}}{2m_{e}}+V_{eh}(\textbf{r}_{e},\textbf{r}_{h})+V_{h}(\textbf{r}_{h})+V_{e}(\textbf{r}_{e}),$
(2)

where the second and the third terms on the r.h.s. are the electron and the
hole kinetic energy operators with appropriate effective masses, the fourth
term is the electron-hole attraction, and the two last terms are the surface
confinement potentials for the electron and hole. The total polarization of
the medium is related to the coherent amplitude by

${\bf P}({\bf R})=2\hbox{Re}\int d^{3}{r}\,{\bf M}({\bf r})Y({\bf R},{\bf
r}).$ (3)

This, in turn, is used in Maxwell’s field equation

$c^{2}\hbox{\boldmath$\nabla$}^{2}{\bf E(R)}-\epsilon_{b}\ddot{\bf
E}=\frac{1}{\epsilon_{0}}{\bf\ddot{P}(R)},$ (4)

where $\epsilon_{b}$=7.5 is the QD material dielectric constant. Equations
(1)-(4) form a system of coupled integro-differential equations in
6-dimensional configuration space $(\textbf{r}_{e},\textbf{r}_{h})$ The
optical properties of spherical QDs can be described by means of the exciton
center-of-mass (COM) quantization methodZiemkiewicz  _et al._ (2020). One
assumes that the COM is confined within a sphere of radius
$R_{\scriptsize\rm{max}}$, which will give the confinement states. Those
states will overlap with the 3-dimensional exciton states. The effective QD
susceptibility in this limit is given by the formula (see supplementary
information)

$\displaystyle\overline{\chi}_{QD}=\epsilon_{b}\sum\limits_{n=2}^{n_{max}}\sum\limits_{N=1}^{N_{max}}\frac{f_{n1}f_{N}\Delta^{(2)}_{LT}/R_{y}}{\left(E_{Tn}-E-i{\mit\Gamma}\right)/R_{y}+\frac{\mu}{M_{tot}}\left(\frac{x_{\frac{3}{2},N}}{R}\right)^{2}},$
$\displaystyle
f_{N}=\frac{1}{6\pi}\left|j_{2}(x_{\frac{3}{2},N})\right|^{-2}\,x_{\frac{3}{2},N}^{2}\left[\int\limits_{0}^{1}u^{3}du\,\prod\limits_{s=1}^{\infty}\left(1-\frac{x_{\frac{3}{2},N}^{2}}{x^{2}_{\frac{3}{2},s}}u^{2}\right)\right]^{2},$
$\displaystyle R=\frac{R_{\scriptsize\rm{max}}}{a^{*}},$ (5) $\displaystyle
f_{n1}=\frac{32(n^{2}-1)}{3n^{5}}\left[\frac{nr_{0}(r_{0}+2a^{*})}{2r_{0}(r_{0}+na^{*})}\right]^{6},$

where $j_{N}(x)$ are the spherical Bessel functions ($N=1,2,\ldots$),
$x_{N,s}$ are roots of the equation $j_{N}(x)=0$, $s=1,2,\ldots$, $a^{*}=1.1$
nm is the Rydberg radius and $E_{Tn}$ are energies of excitonic levels.

Fig. 4 shows a comparison of experimental and calculated absorption spectra.
The bulk crystal spectrum is also calculated from Eq. (IV) by taking a very
large NP radius $R>$$10\text{\,}\mathrm{\SIUnitSymbolMicro m}$. Due to the
relatively large size of considered NPs and the corresponding confinement
energy shifts on the order of $\sim$ $10\text{\,}\mathrm{\SIUnitSymbolMicro
eV}$, the direct observation of them can be challenging. To get a clearer
picture of the results, the absorption coefficient obtained for bulk and
nanoparticle systems has been normalized and the NP spectrum was shifted to
obtain an exact match of $n=2$ lines, removing the influence of different
experimental conditions such as temperature mentioned before. In such a case,
we observe a difference of $\Delta E\sim$
$25\text{\,}\mathrm{\SIUnitSymbolMicro eV}$ between bulk and NP lines for
$n=5$ state (vertical lines on Fig. 4 (c) and (d)), both in the experimental
spectra and calculation results. This amounts to the difference between
confinement energy of $n=5$ and $n=2$ state, the latter one being negligible.
This result is a close match to the theoretical predictions of Konzelmann _et
al._ Konzelmann  _et al._ (2020).

Our calculations show an excellent match between the measured oscillator
strength and the theoretical estimates when the size distribution of the
nanoparticles in considered [blue line in Fig. 3(a)]. The calculation for
single NP size ($R$=350 nm) fails to fully explain the oscillator strength
reduction [orange line in Fig. 3(a)], further confirming that quantum dot size
distribution is crucial to the understanding of the observed effects.
Moreover, upper confinement states ($N>1$) contribute to $\sim$23% of the peak
area for $n=5$, with a smaller effect for $n<5$. The calculated linewidths
match the experimental data for $n=3,4,5$, while slightly underestimating the
linewidth of $n=2$ state [blue line in Fig. 3(b)]. Again, we find that it is
crucial to take into account the size distribution of the NPs, further
evidencing the effect of NP size.

##  V Discussion

Figure 5: Imaginary part of susceptibility calculated for $n=5$ and various
$N$ values and dot sizes. For each $N$, a sample of three individual dot sizes
is shown (dashed lines) along with an average (solid lines). The normalized
sum over all confinement states and dot sizes is depicted as a black solid
line.

Previous studies Kazimierczuk  _et al._ (2014) have shown that the power-
dependent optical bleaching (reduction of peak area) only becomes visible for
$n=12$ for a pump intensity of $\sim$1 mW/mm2. Moreover, the intensity
required to see optical bleaching scales as $n^{10}$. Therefore, to notice any
nonlinearities at $n=6$ one needs laser power on the order of 1 W/mm2. The
intensity of the illuminating light in our experiment ($\sim 1$ mW/mm2) is
sufficiently low that the nonlinear effects observed in bulk can be neglected
for the observed excitonic states. We note that in nanoparticles, the blockade
diameter can become comparable to particle size for low $n$ (see Supplementary
Fig. S4). The intensities in our experiment, however, are 3 orders of
magnitude weaker than that required to reach the blockade effect.

The reduction in oscillator strength due to damage to crystal structure can be
ruled out since we observe this reduction in natural as well as synthetic NPs
(see Supplementary Information 2). Strain on Cu2O crystals can cause a change
in the absorption strength due to change from isotropic to anisotropic $np$
states Agekyan (1977). The substrates can exert stress on Cu2O crystals and
affect the exciton transitions especially if the crystal is glued onto the
cold finger or strongly sandwiched between two substrates. However, the
nanoparticles here are held in place only by the van der Waals forces.

Electron-hole plasma generated due to the incident broadband light can affect
the excitons. The collision of excitons with electron-hole plasma results in
the reduction of exciton lifetime, broadening of the transitions as well as
reducing the exciton oscillator strength. However, this effect was observed
Heckötter  _et al._ (2018) to be significant only for $n$ levels higher than
$n=10$.

The quantum confinement, however, is the dominant effect here. Theoretically,
for $n=5$ excitons in Cu2O three dimensional confinement to
$700\text{\,}\mathrm{nm}$ diameter, the lowest confinement state ($N$=1) would
exhibit a blueshift of $\sim$ $20\text{\,}\mathrm{\SIUnitSymbolMicro eV}$
Konzelmann  _et al._ (2020), which is an order of magnitude smaller than the
linewidth of the $n=5$ transition. However, the states $N=2,3,4...$ provide a
non-trivial contribution to the total area of the observed excitonic line,
which is a measure of oscillator strength. This is shown on the Fig. 5, where
$N=1,2,3$ states are marked by red, blue and green lines respectively. Since
the energy shift of those states is proportional to $N$, it can reach values
of over $200\text{\,}\mathrm{\SIUnitSymbolMicro eV}$ for $N>2$. Moreover, for
the given dot radius $R$, the energy shift is approximately proportional to
$R^{-2}$ and a relatively large energy shift on the order of
$0.5\text{\,}\mathrm{meV}$ can be expected at the lower end of the obtained NP
sizes. This is clearly seen in Fig. 5, where strongly blueshifted peaks
corresponding to small NPs are visible. All these factors contribute to the
shape of the total line (black curve) and result in an apparent reduction of
the oscillator strength; every observed excitonic line is an overlap of the
primary confinement state ($N=1$) and multiple blueshifted higher states which
are too close to each other to discern them on the spectrum. The contribution
of these states is twofold. 1- They make the transitions more symmetric. Since
they have higher energy than the $N=1$ state, their effect is more pronounced
on the right-hand side of the absorption peak; this is visible in Fig. 4(c)
and (d) where the asymmetric lineshape of the bulk crystal is transformed into
Gaussian shape. The effect is also visible for lower $n$ states [Fig. 4(a) and
(b)], but it is much less pronounced due to smaller energy shifts. 2- They
significantly increase the background absorption. Since the oscillator
strength of confinement states quickly decreases with $N$, their contribution
affects mostly the base of the absorption line. The widened bases overlap,
forming a strong absorptive background. This greatly reduces the area of the
peak visible above that background resulting in an apparent reduction of the
oscillator strengths.

In conclusion, we successfully observed Rydberg excitons in nanoparticles of
Cu2O. Through optical spectroscopy, we showed that reducing the size of the
system leads to a subsequent reduction in the oscillator strength and an
apparent linewidth broadening. We explained our observations through the
quantum confinement of the excitons in the nanoparticles. Our work paves the
way for exploiting Cu2O Rydberg excitons in the nanoscale for their large
nonlinearities.

## acknowledgements

We acknowledge EPSRC Grant No. EP/S014403/1 and The Royal Society
RGS\R2\192174. K.O. acknowledges EPSRC for PhD studentship support through
grant no. EP/L015110/1. We thank Michael Huang, Matthew Jones, Stephen Lynch,
Stefan Scheel and Mikhail M. Glazov for fruitful discussions.

## References

  * Keyes (1988) R. W. Keyes, IBM Journal of Research and Development 32, 84 (1988).
  * Awschalom  _et al._ (2013) D. D. Awschalom, L. C. Bassett, A. S. Dzurak, E. L. Hu, and J. R. Petta, Science 339, 1174 (2013).
  * Devoret and Schoelkopf (2013) M. H. Devoret and R. J. Schoelkopf, Science 339, 1169 (2013).
  * Loredo  _et al._ (2016) J. C. Loredo, N. A. Zakaria, N. Somaschi, C. Anton, L. de Santis, V. Giesz, T. Grange, M. A. Broome, O. Gazzano, G. Coppola, I. Sagnes, A. Lemaitre, A. Auffeves, P. Senellart, M. P. Almeida, and A. G. White, Optica 3, 433 (2016).
  * Mak and Shan (2016) K. F. Mak and J. Shan, Nature Photonics 10, 216 (2016).
  * Gonzalez-Zalba (2018) M. F. Gonzalez-Zalba, arXiv:1801.06722 [cond-mat] (2018), arXiv: 1801.06722.
  * Fox (2011) M. Fox,  _Optical properties of solids_ , 2nd ed., Oxford master series in condensed matter physics No. 3 (Oxford Univ. Press, Oxford, 2011) oCLC: 772954632.
  * Hayashi and Katsuki (1952) M. Hayashi and K. Katsuki, Journal of the Physical Society of Japan 7, 599 (1952).
  * Meyer  _et al._ (2013) B. K. Meyer, A. Polity, D. Reppin, M. Becker, P. Hering, B. Kramm, P. J. Klar, T. Sander, C. Reindl, C. Heiliger, M. Heinemann, C. Müller, and C. Ronning,  _Oxide Semiconductors_, edited by B. G. Svensson, S. J. Pearton, and C. Jagadish, Semiconductors and Semimetals, Vol. 88 (Elsevier, 2013) pp. 201 – 226.
  * Matsumoto  _et al._ (1996) H. Matsumoto, K. Saito, M. Hasuo, S. Kono, and N. Nagasawa, Solid. Stat. Comm. 97, 5 (1996).
  * Kazimierczuk  _et al._ (2014) T. Kazimierczuk, D. Fröhlich, S. Scheel, H. Stolz, and M. Bayer, Nature 514, 343 (2014).
  * Thewes  _et al._ (2015) J. Thewes, J. Heckötter, T. Kazimierczuk, M. Aßmann, D. Fröhlich, M. Bayer, M. Semina, and M. Glazov, Physical Review Letters 115, 027402 (2015).
  * Grünwald  _et al._ (2016) P. Grünwald, M. Aßmann, J. Heckötter, D. Fröhlich, M. Bayer, H. Stolz, and S. Scheel, Physical Review Letters 117, 133003 (2016).
  * Heckötter  _et al._ (2017a) J. Heckötter, M. Freitag, D. Fröhlich, M. Aßmann, M. Bayer, M. A. Semina, and M. M. Glazov, Physical Review B 96, 125142 (2017a).
  * Heckötter  _et al._ (2017b) J. Heckötter, M. Freitag, D. Fröhlich, M. Aßmann, M. Bayer, M. A. Semina, and M. M. Glazov, Physical Review B 95, 035210 (2017b).
  * Takahata and Naka (2018) M. Takahata and N. Naka, Physical Review B 98, 195205 (2018).
  * Krüger and Scheel (2018) S. O. Krüger and S. Scheel, Physical Review B 97, 205208 (2018).
  * Lynch  _et al._ (2018) S. A. Lynch, C. Hodges, W. W. Langbein, L. Gallagher, J. Rogers, C. S. Adams, M. P. A. Jones, and R. P. Singh, in  _2018 20th International Conference on Transparent Optical Networks (ICTON)_ (IEEE, Bucharest, 2018) pp. 1–4.
  * Mund  _et al._ (2019) J. Mund, C. Uihlein, D. Fröhlich, D. R. Yakovlev, and M. Bayer, Physical Review B 99, 195204 (2019).
  * Zielińska-Raczyńska  _et al._ (2019) S. Zielińska-Raczyńska, G. Czajkowski, K. Karpiński, and D. Ziemkiewicz, Physical Review B 99, 245206 (2019).
  * Jones  _et al._ (2017) M. P. A. Jones, L. G. Marcassa, and J. P. Shaffer, Journal of Physics B: Atomic, Molecular and Optical Physics 50, 060202 (2017).
  * Walther  _et al._ (2018) V. Walther, R. Johne, and T. Pohl, Nature Communications 9, 1309 (2018).
  * Konzelmann  _et al._ (2020) A. Konzelmann, B. Frank, and H. Giessen, Journal of Physics B: Atomic, Molecular and Optical Physics 53, 024001 (2020).
  * Smith and Nie (2010) A. M. Smith and S. Nie, Accounts of Chemical Research 43, 190 (2010).
  * Khazali  _et al._ (2017) M. Khazali, K. Heshami, and C. Simon, Journal of Physics B: Atomic, Molecular and Optical Physics 50, 215301 (2017).
  * Baur  _et al._ (2014) S. Baur, D. Tiarks, G. Rempe, and S. Dürr, Physical Review Letters 112, 073901 (2014).
  * Baumeister (1961) P. W. Baumeister, Physical Review 121, 359 (1961).
  * Jolk and Klingshirn (1998) A. Jolk and C. F. Klingshirn, physica status solidi (b) 206, 841 (1998).
  * Toyozawa (1964) Y. Toyozawa, Journal of Physics and Chemistry of Solids 25, 59 (1964).
  * Agekyan  _et al._ (1974) V. T. Agekyan, B. S. Monozon, and I. P. Shiryapov, physica status solidi (b) 66, 359 (1974).
  * Ekimov  _et al._ (1985) A. Ekimov, A. Onushchenko, A. Plyukhin, and A. L. Efros, Journal of Experimental and Theoretical Physics 61, 891 (1985).
  * Ekimov  _et al._ (1986) A. Ekimov, A. Onushchenko, M. Raikh, and A. L. Efros, J. Exp. Theor. Phys. 63, 1054 (1986).
  * Ziemkiewicz  _et al._ (2020) D. Ziemkiewicz, K. Karpiński, S. Zielińska-Raczyńska, and G. Czajkowski, Physical Review B 101, 205202 (2020).
  * Agekyan (1977) V. T. Agekyan, physica status solidi (a) 43, 11 (1977).
  * Heckötter  _et al._ (2018) J. Heckötter, M. Freitag, D. Fröhlich, M. Aßmann, M. Bayer, P. Grünwald, F. Schöne, D. Semkat, H. Stolz, and S. Scheel, Physical Review Letters 121, 097401 (2018).