(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: School of Computer Science and Technology,
Beijing Institute of Technology, Beijing, China
11email<EMAIL_ADDRESS>
11email<EMAIL_ADDRESS>
# Background Adaptation with Residual Modeling for Exemplar-Free Class-
Incremental Semantic Segmentation

Anqi Zhang  0000-0002-2112-8199 Guangyu Gao Corresponding Author.
0000-0002-0083-3016

###### Abstract

Class Incremental Semantic Segmentation (CISS), within Incremental Learning
for semantic segmentation, targets segmenting new categories while reducing
the catastrophic forgetting on the old categories. Besides, background
shifting, where the background category changes constantly in each step, is a
special challenge for CISS. Current methods with a shared background
classifier struggle to keep up with these changes, leading to decreased
stability in background predictions and reduced accuracy of segmentation. For
this special challenge, we designed a novel background adaptation mechanism,
which explicitly models the background residual rather than the background
itself in each step, and aggregates these residuals to represent the evolving
background. Therefore, the background adaptation mechanism ensures the
stability of previous background classifiers, while enabling the model to
concentrate on the easy-learned residuals from the additional channel, which
enhances background discernment for better prediction of novel categories. To
precisely optimize the background adaptation mechanism, we propose Pseudo
Background Binary Cross-Entropy loss and Background Adaptation losses, which
amplify the adaptation effect. Group Knowledge Distillation and Background
Feature Distillation strategies are designed to prevent forgetting old
categories. Our approach, evaluated across various incremental scenarios on
Pascal VOC 2012 and ADE20K datasets, outperforms prior exemplar-free state-of-
the-art methods with mIoU of 3.0% in VOC 10-1 and 2.0% in ADE 100-5, notably
enhancing the accuracy of new classes while mitigating catastrophic
forgetting. Code is available in https://andyzaq.github.io/barmsite/.

##  1 Introduction

With the development of deep learning and the support of convolutional neural
networks (CNNs) [31, 30, 51, 27] and Transformers [54, 20, 33, 53], current
semantic segmentation networks [34, 46, 10, 11, 12, 13, 16, 15, 24] have
achieved remarkable results on standard supervised learning tasks. However,
real-world data, arriving as an unstable continuous stream, poses the
challenge of retaining prior knowledge while accommodating new concepts,
termed as catastrophic forgetting [39, 38].

To tackle this problem, incremental learning is proposed to adapt to changing
data streams for new concepts, but also avoid forgetting old knowledge,
especially for the classification task, _i.e_., Class-Incremental Learning
(CIL) [28, 32, 14, 59, 65]. Cermelli et al. [6] introduced CIL into Semantic
Segmentation as Class Incremental Semantic Segmentation (CISS). Since the
semantic segmentation tasks involve dense predictions, the issue of
catastrophic forgetting typically becomes more challenging. Most recent works
[6, 8, 21] have struggled to mitigate catastrophic forgetting through
knowledge distillation [21, 40, 3] or parameter freezing [8, 61]. Although
these strategies effectively alleviate catastrophic forgetting, it comes at
the cost of the model’s plasticity, making it challenging to learn novel
classes. In current state-of-the-art approaches [61, 3], the performance of
established old classes approaches the upper limit of joint training, yet
there still remains a significant gap in mIoU for novel classes. However, in
the context of lifelong learning scenarios featuring an infinite influx of
novel classes, the incremental learner’s plasticity becomes increasingly
crucial. Therefore, the optimal solution should be fine-tuning more parameters
for novel classes, while catastrophic forgetting needs to be handled properly.

Figure 1: 3D Visualization of Background Adaptation results. The background
logits for step $t$ combine those from step $t-1$ with the adaptation logits
learned in step $t$, preventing disorderly adjustments and focusing on
residuals. The current step background logits are processed using the Sigmoid
function. Note that colors range from red (small values) to blue (large
values).

Nevertheless, background shifting, where the background category changes
constantly in each step, is another challenge for CISS. Upon further
investigation, it was observed that previous methods within CISS often utilize
a shared background classifier across each step. However, novel classes are
typically considered as the background in previous learning steps, indicating
that the background class undergoes continuous changes throughout incremental
learning. This way results in an incessantly shifting background in each step,
leading to disruption in background prediction and inaccurate learning of
novel classes. Some approaches [8, 9, 41] store old exemplars in extra memory
space to prevent background shifting, while long-term storage of such data may
not be feasible in real-world scenarios. To enhance the incremental nature of
the CISS method and reduce background instability, we revise the
representation of the background class and propose an exemplar-free background
adaptation mechanism. In particular, the background adaptation mechanism
explicitly models the background residual rather than the background itself in
each incremental step. The learned residuals of the additional channel are
aggregated with the previously learned background logits to represent the
evolving background, as illustrated in Fig. 1. By employing the background
adaptation mechanism, we only need to optimize the additional channel of
residuals for constructing the background during the incremental steps. This
approach eliminates the need for repeatedly learning background classifiers
associated with old classes, thereby avoiding the instability of optimizing a
shared background classifier.

Moreover, we customize our training approaches in order to fit the training
process of the background adaptation structure and promote its performance. We
first introduce the widely used pseudo-labeling [21, 8] to generate a pseudo-
background, and then we design a PB-BCE loss to focus on optimizing the
classifier of novel classes and related background adaptation residual.
Besides, for the Background Adaptation channel, the objective is to minimize
the values in the regions of novel categories, while ensuring that the values
in other regions remain above 0. Thus, two auxiliary Background Adaptation
losses are specifically applied to the background adaptation channel for
precise adjustment. Given our pursuit of plasticity on novel categories
without external memory, we have devised more comprehensive exemplar-free
knowledge distillation strategies to combat catastrophic forgetting. We
further adopt knowledge distillation strategies on the logit maps of old
categories and the intermediate features, namely the Group Knowledge
Distillation and Background Feature Distillation, respectively. Finally, our
approach outperforms previous methods in multiple benchmarks, especially in
realistic and challenging long-term scenarios such as 10-1 of VOC2012 and
100-5 of ADE20K. Our approach achieves significant performance gains of 4.5%
and 5.2% on novel classes, respectively, compared to the previous state-of-
the-art.

Overall, our contributions are summarized as follows:

  * •

We propose a Background Adaptation strategy in incremental steps by optimizing
an additional channel for residual modeling instead of the shared background
classifier, which avoids unstable optimization and prevents chaos in
background logits.

  * •

We design PB-BCE loss and Background Adaptation losses to enhance the
effectiveness of Background Adaptation, while two distillation strategies
ensure maintaining knowledge of old categories.

  * •

Experiments validate the effectiveness of our proposed method, showcasing a
new state-of-the-art performance on CISS benchmarks.

##  2 Related Works

###  2.1 Class Incremental Learning

Class Incremental Learning (CIL) breaks the constraints of the standard
training process in an incremental form on new classes. CIL task aims to
continually learn the new classes without retraining on the data of old
classes, which significantly reduces the training cost. As a compromise, the
CIL task has to effectively solve the problem of catastrophic forgetting [39,
38]. Current CIL methods are divided into several main categories. Replay-
based methods recall the data of old classes during incremental training. Some
of the approaches [35, 2, 4] reserve memory buffers and temporarily store
representative past data. Others [49, 55, 42] introduce powerful generative
models to generate pseudo-data for rehearsal. Regularization-based methods
introduce regularization terms to prevent forgetting the old classes. These
methods utilize loss functions as penalties [29, 45, 47, 1] or knowledge
distillation from the previous model [32, 44, 19]. Representation-based
methods attempt to transfer representations to certain categories, which apply
adaptation on a fixed pre-trained backbone [60, 50, 5, 22] or self-supervised
learning [36, 7, 43].

###  2.2 Class Incremental Semantic Segmentation

The achievements in CIL tasks encourage the Incremental Learning pipeline to
develop in other dense prediction tasks, including Semantic Segmentation.
Class Incremental Semantic Segmentation (CISS) task is first introduced in MiB
[6], which focused on reconstructing the background region under the
restriction from knowledge distillation. Since then, various methods [57, 63]
have been introduced in the CISS task. PLOP [21] defined the pseudo-labeling
and multi-scale local distillation to remember the old categories. SDR [40]
introduced category-wise prototype matching and contrastive learning methods
to enlarge the difference between categories. RCIL [58] and EWF [56] explored
methods for integrating parameters of new and old models. DKD [3] decomposed
the class logits into positive and negative logits for better knowledge
distillation and applied threshold to generate background regions. Incrementer
[48] borrows prompt learning from ViT [20] and applies the structure of
Segmenter [52] instead of DeeplabV3 [12], which is the basic structure of
other methods. Besides, several approaches utilize external memory [66] or
models [37, 62] for better performance. SSUL [8] introduced a Saliency-map
detector to identify the saliency region and store some of the previous
instances in memory. MicroSeg [61] utilized proposals from pretrained
Mask2Former [15] as a guidance. ALIFE [41] memorized category features and
adapted for remembering the old classes.

##  3 Methods

(a) The overall pipeline of our approach.  (b) BgA losses.

Figure 2: Overview of our framework that consists of a Background Adaptation
mechanism and proposed losses. (a) The Background Adaptation mechanism uses
the negative part of the adaptation channel to rectify the background logits.
(b) Two Background Adaptation losses separately optimize the regions w/ and
w/o novel categories to enhance the performance of the Background Adaptation.

###  3.1 Preliminaries

The Class-Incremental Semantic Segmentation (CISS) task performs training in
$T$ steps. During step $t$, the training data $D^{t}$ containing the labels of
the novel classes $C^{t}$ are applied for training, while the labels of old
classes $C^{1:t-1}$ are expunged. In particular, each sample of the training
data $D^{t}$ consists of an RGB image $x^{t}\in\mathbb{R}^{3\times H\times W}$
and a corresponding ground truth label $y^{t}$ on novel classes $C^{t}$.
Besides, the ground truth label $y^{t}$ replaces the labels of the $C^{1:t-1}$
with the background label $c_{b}$ as the old classes will not appear in
$C^{t}$ to ensure $C^{1:t-1}\cap C^{t}=\varnothing$. Therefore, the model
$f^{t}$ being trained at step $t$ not only needs to learn to segment novel
categories but also has to retain the capability to segment old categories.
However, while learning the novel categories, the model $f^{t}$ tends to
overfit the novel categories $C^{t}$ as $f^{t}$ cannot contact with samples of
$C^{1:t-1}$, causing severe catastrophic forgetting. Moreover, the novel
categories can only appear in the background region during the previous steps.
We are more likely to classify them from the previous background prediction
region, which motivates us to focus on background adaptation.

###  3.2 Background Adaptation

Previous studies often apply only one background classifier. They have to
readjust the parameters of the background classifier whenever the model learns
novel categories, which causes catastrophic forgotten even if the classifier
of the old categories remains the same probability output. The unstable
background classifier could be a thorny constraint factor. To tackle the
problems above, we design a Background Adaptation mechanism. Different from
the previous distinct background classifier, we introduce an additional
background adaptation channel for each classifier of the incremental process.

During the training process of incremental step $t$, the current model $f^{t}$
inherits the parameters from the previous model $f^{t-1}$ and adds a new
classifier $f_{C^{t}}^{t}$ for parallel classification of categories in
$C^{t}$ alongside the previous classifiers. The encoder $f_{e}^{t}$ and the
decoder $f_{d}^{t}$ transform the input image $x^{t}$ into features map
$F_{d}^{t}\in\mathbb{R}^{h\times w\times c}$:

$F^{t}_{d}=f_{d}^{t}(f_{e}^{t}(x^{t})).$ (1)

Compared to the standard classifier of the previous methods [21, 8, 41], our
classifier of each incremental step has an additional channel for background
adaptation. Therefore, for each group of categories $C^{i}$ from step $i$, the
classifier $f_{C^{i}}^{t}$ of the model $f^{t}$ generates the corresponding
classification logit maps
$\mu_{C^{i}\cup\\{b_{i}\\}}^{t}\in\mathbb{R}^{h\times w\times(|C^{i}|+1)}$:

$\mu_{C^{i}\cup\\{b_{i}\\}}^{t}=f_{C^{i}}^{t}(F^{t}_{d}),$ (2)

where $i\in\\{1,2,\ldots,t\\}$. We define $b^{1}$ to represent the background
class of the initial step, and $b^{i}$ denotes the background adaptation for
subsequent incremental steps when $i>=2$. The symbol $b$ refers to the adapted
background class. Thus, $\mu_{C^{i}\cup\\{b_{i}\\}}^{t}$ contains the
probability maps $\mu_{C^{i}}$ of $|C^{i}|$ categories from step $i$ and an
additional channel $\mu_{b^{i}}^{t}$ for constructing the background logits
$\mu_{b}^{t}$. Considering that the region of the novel categories $C^{t}$ is
more likely inherited from the background region of the previous step, the
classifier $f^{t}_{C^{t}}$ adjusts the background logits through
$\mu_{b^{t}}^{t}$, which decreases the probability values in the region
$\mathcal{R}_{C^{t}}$ of $C^{t}$. The parameters of previous classifiers are
not required to continuously optimize for new background regions, thereby
avoiding instability in the classification of old categories.

The background adaptation channel $\mu_{b^{t}}^{t}$ can correct the background
logits $\mu_{b}^{t}\in\mathbb{R}^{h\times w\times 1}$ in
$\mathcal{R}_{C^{t}}$, yet the noise in other regions of $\mu_{b^{t}}^{t}$ may
interfere with the background prediction. Particularly, when old categories
are present in the image, there is a risk that these regions could be
incorrectly covered by the adapted background. Hence, we design a Filter
operation that replaces positive values of $\mu_{b^{i}}^{t}$ with $0$. As
shown in Fig. 2(a), with the Filter operation applied, the residual modeling
of background logits during the inference process is as follows:

$\mu_{b}^{t}=\mu_{b^{1}}^{t}+\sum_{i=2}^{t}Filter(\mu_{b^{i}}^{t}),$ (3)

where the background logits from the initial step $\mu_{b^{1}}^{t}$ stay the
same and the background adaptation logits from the incremental steps
$\mu_{b^{i}}^{t}$ only remain the negative values via the Filter. In
particular, to make $\mu_{b}^{t}$ more friendly for gradient backpropagation,
we abandoned the Filter on the background adaptation logits of novel classes
$\mu_{b^{t}}^{t}$ during the training process:

$\mu_{b}^{t}=\mu_{b^{1}}^{t}+\sum_{i=2}^{t-1}Filter(\mu_{b^{i}}^{t})+\mu_{b^{t}}^{t}.$
(4)

###  3.3 Incremental Training Strategy

####  3.3.1 Pseudo Labeling.

During the incremental steps, even if objects of old categories $C^{1:t-1}$
exist in the image, we cannot obtain corresponding labels from the ground
truth. In order to simulate a ground truth label that contains the categories
we have learned, we follow [21, 8, 61] to introduce the prediction probability
of the previous step ${\varphi}^{t-1}\in\mathbb{R}^{h\times
w\times|C^{1:t-1}|}$ from $f^{t-1}(x^{t})$ with Sigmoid function and combine
it with the current ground truth label $y^{t}$:

$\hat{y}^{t}=\begin{cases}y^{t},&(y^{t}\in C^{t})\vee\\{(y^{t}\in
c_{b})\wedge(max({\varphi}^{t-1})<\tau)\\}\\\ \tilde{y}^{t-1},&(y^{t}\in
c_{b})\wedge(max({\varphi}^{t-1})\geq\tau)\end{cases}$ (5)

where $\hat{y}^{t}\in\mathbb{R}^{h\times w}$ denotes the generated pseudo
label and $\tilde{y}^{t-1}=argmax({\varphi}^{t-1})$ represents the final
prediction from $f^{t-1}(x^{t})$.

####  3.3.2 PB-BCE Loss.

Binary Cross-Entropy (BCE) loss is extensively employed for learning to
segment objects of novel categories. We follow the usage of the Sigmoid
function and activate $\mu^{t}$ as ${\varphi}^{t}$. Some of the CISS methods
[21, 8, 61] utilize the pseudo mask as the ground truth for supervision, which
effectively replaces the background label in the regions of the old
categories. However, the ambiguous supervision of the old categories could
improperly optimize the classifiers of $C^{1:t-1}$ and destroy their
identification capability. Therefore, we design a Pseudo Background BCE (PB-
BCE) loss which cuts off the backpropagation of old categories, meaning that
we only take the labels of current categories $\hat{y}^{t}_{C^{t}}$ and
pseudo-background $\hat{y}^{t}_{c_{b}}$ for optimization:

$\mathcal{L}_{PB-BCE}=-\frac{1}{hw}\sum_{k=1}^{hw}\sum_{c\in\\{c_{b}\\}\cap
C^{t}}CE({\varphi}^{t}_{c,k},\hat{\psi}^{t}_{c,k}),$ (6)

where $k$ represents the index of pixel and $\hat{\psi}^{t}_{c}$ denotes the
binary map of class c in pseudo map $\hat{y}^{t}$. $CE$ is the cross-entropy
calculation:

$CE({\varphi}^{t}_{*},\hat{\psi}^{t}_{*})=\hat{\psi}^{t}_{*}\cdot
log{\varphi}^{t}_{*}+(1-\hat{\psi}^{t}_{*})\cdot log(1-{\varphi}^{t}_{*}).$
(7)

Although the whole background prediction logits $\mu_{b}^{t}$ are involved in
the loss calculation, only the background adaptation channel for current
categories $\mu_{b^{t}}^{t}$ serves for optimization. Previous background
logits are detached to prevent disorderly evolving on parameters of the old
classifiers.

####  3.3.3 Background Adaptation Losses.

The Background Adaptation (BgA) losses, as shown in Fig. 2(b), are designed
for further refined adjustments on the background adaptation channel
$\mu_{b^{t}}^{t}$. We propose two losses for two different regions of
$\mu_{b^{t}}^{t}$ according to the target effect. For regions
$\mathcal{R}_{C^{t}}$ belonging to the current classes $C^{t}$, the ideal
target is to have negative values as small as possible. To achieve this, we
introduce cross-entropy loss $\mathcal{L}_{BgA^{+}}$ for supervision:

$\mathcal{L}_{BgA^{+}}=-\frac{1}{|\mathcal{R}_{C^{t}}|}\sum_{k\in\mathcal{R}_{C^{t}}}log\frac{exp(-\mu_{b^{t},k}^{t})}{1+exp(-\mu_{b^{t},k}^{t})},$
(8)

where $|\mathcal{R}_{C^{t}}|$ represents the number of pixels in
$\mathcal{R}_{C^{t}}$. Besides, as for the regions of the background that
might contain old classes, we hope the value of $\mu_{b^{t}}^{t}$ could be 0
for maintaining the historical background prediction result. However,
achieving precise control over the whole region to be a precise value is
challenging and may be impractical. Therefore, we set the positive logits in
the adaptation channel of the old categories to 0 in Eq. 4 to avoid the noise
and apply a Triplet loss $\mathcal{L}_{BgA^{-}}$ to supervise the background
region:

$\displaystyle\mathcal{L}_{BgA^{-}}=\frac{1}{hw-|\mathcal{R}_{C^{t}}|}\sum_{k\notin\mathcal{R}_{C^{t}}}max[0,(1-\varphi_{b^{t},k}^{t})^{2}-(\varphi_{b^{t},k}^{t}-0)^{2}].$
(9)

We first activate the background adaptation channel as $\varphi_{b^{t}}^{t}$
via Sigmoid function, then the logits are mapped to the range from 0 to 1. We
aim for the logits $\mu_{b^{t}}^{t}$ of this region to be greater than 0 while
not overfitting to the objects in the background, meaning that values of
$\varphi_{b^{t}}^{t}$ lie within the range of 0.5 to 1 and not close to 1. In
particular, The Triplet loss $\mathcal{L}_{BgA^{-}}$ respectively set 0 and 1
as negative and positive anchors. We simultaneously constrain the loss not to
go below 0 by setting a lower bound. Therefore, once the value of the pixel
exceeds 0.5, _i.e_., the middle of the positive and negative anchors, there
will be no more gradient on the pixel for backpropagation, which mitigates
overfitting to the objects in the background.

####  3.3.4 Knowledge Distillation Strategies.

In the training process of incremental steps for CISS tasks, substantial
alterations to the existing parameters of $f^{t}$ frequently occur, giving
rise to situations of catastrophic forgetting. Guided by the previous methods,
we have devised a series of knowledge distillation strategies tailored to our
background adaptation mechanism.

  * •

Group Knowledge Distillation. In Sec. 3.2, we have introduced the
classification logit map $\mu_{C^{i}\cup\\{b_{i}\\}}^{t}$ for each group of
categories $C^{i}$, which contains an additional channel for background
adaptation. We introduce a Group Knowledge Distillation loss
$\mathcal{L}_{GKD}$ to ensure the stability of the old category logits, as
well as the previous background adaptation channel:

$\mathcal{L}_{GKD}=-\frac{1}{hw}\sum_{k=1}^{hw}\sum_{i=1}^{t-1}\sum_{c\in\\{c_{b}^{i}\\}\cup
C^{i}}CE(\varphi_{c,k}^{t},\varphi_{c,k}^{t-1})$ (10)

where $\varphi_{*}^{t-1}$ and $\varphi_{*}^{t}$ denote the activated logits of
$\mu_{*}^{t-1}$ and $\mu_{*}^{t}$, respectively.

  * •

Background Feature Distillation. When the network learning novel categories,
not only the parameters of the novel classifier $f^{t}_{C^{t}}$ are optimized,
but the encoder $f^{t}_{e}$ and the decoder $f^{t}_{d}$ are affected as well.
Merely utilizing Group Knowledge Distillation loss $\mathcal{L}_{GKD}$ is
insufficient in effectively mitigating catastrophic forgetting as it primarily
focuses on constraining classifiers of the old categories. However,
excessively constraining parameter optimization can impede learning for novel
categories. Taking all these factors into account, we designed a Background
Feature Distillation (BFD) method that only acts on the features out of the
regions of current categories $\mathcal{R}_{C^{t}}$:

$\mathcal{L}_{BFD}=\sum_{i=1}^{t-1}\|\psi^{t}_{b}\cdot
F^{t}_{m^{i}}-\psi^{t}_{b}\cdot F^{t-1}_{m^{i}}\|_{2}^{2},$ (11)

where $F^{t}_{m^{i}}$ denotes the intermediate features in $f^{t}_{C^{i}}$,
$\psi^{t}_{b}$ represents the binary mask out of $\mathcal{R}_{C^{t}}$, and
$\|\cdot\|_{2}$ calculates the Euclidean distance between the two matrix. The
Background Feature Distillation allows other losses to supervise the region of
novel categories $\mathcal{R}_{C^{t}}$ while restricting the negative effect
outside $\mathcal{R}_{C^{t}}$, where old categories $C^{1:t-1}$ could appear.

###  3.4 Objective Function

With our proposed losses above, we construct our objective function as
follows:

$\displaystyle\mathcal{L}=\mathcal{L}_{PB-
BCE}+\lambda_{1}\cdot\mathcal{L}_{BgA^{+}}+\lambda_{2}\cdot\mathcal{L}_{BgA^{-}}+\lambda_{3}\cdot\mathcal{L}_{GKD}+\lambda_{4}\cdot\mathcal{L}_{BFD},$
(12)

where $\mathcal{L}_{PB-BCE}$ supervises the learning of novel classes,
$\mathcal{L}_{PB-BCE}$ and $\mathcal{L}_{BgA^{-}}$ enhance the effect of
Background Adaptation, $\mathcal{L}_{GKD}$ and $\mathcal{L}_{BFD}$ mitigate
forgetting old classes. Hyperparameters
$\lambda_{1},\lambda_{2},\lambda_{3},\lambda_{4}$ balance these constraints.

##  4 Experiments

###  4.1 Experimental Setup

Datasets. We follow the settings of most previous work and evaluate our
proposed methods on Pascal VOC 2012 [23, 26] and ADE20K [64] datasets. Pascal
VOC 2012 is composed of 20 different classes with 10,582 images for training
and 1,449 images for validation. ADE20K has 150 classes that contain 100 thing
classes and 50 stuff classes. There are 20,210 images for training and 2,000
images for validation in ADE20K.

Protocols. Earlier work [6, 21] has defined two types of training settings,
_i.e_. overlapping and disjoint. The overlapping setting contains future
classes in the previous steps, whereas the disjoint setting forbids future
classes until they belong to the training classes. We follow recent work [3,
41, 63, 56] and adopt the overlapping setting for evaluating as it is more
realistic. Besides, following the previous training protocols [56, 6, 21], we
set each sub-task from the dataset as $N_{ini}-N_{inc}$, where $N_{ini}$ and
$N_{inc}$ denotes the quantity of the categories in the initial step and a
single incremental step, respectively. For example, the 15-1 sub-task of
Pascal VOC 2012 has 15 classes for the initial training step. During each
incremental step, one novel class is applied for incremental training without
previous classes. Specifically, we define 4 sub-tasks on Pascal VOC 2012,
including 19-1 (2 steps), 15-1 (6 steps), 10-1 (11 steps), and 5-3 (6 steps).
On ADE20K, we define 4 sub-tasks containing 100-50 (2 steps), 100-10 (6
steps), 50-50 (3 steps), and 100-5 (11 steps).

Evaluation metrics. We evaluate our proposed method with widely-used mean
Intersection-over-Union (mIoU). The Intersection-over-Union (IoU) is defined
as $IoU=\frac{TP}{TP+FP+FN}$, where $TP$, $FP$, and $FN$ represent the
quantity of true-positive, false-positive, and false-negative prediction
pixels. The mIoU calculates the mean value of IoU corresponding to each
category. To effectively measure the performance of CISS tasks, we compute the
mIoU values of the initial classes $C^{0}$, the incremental classes $C^{1:t}$,
and all classes $C^{0:t}$, respectively.

Implementation details. Following the previous studies [6, 40, 8], we apply
Deeplab-v3 [12] as the segmentation network with a ResNet-101 [27] for feature
extraction. The ResNet-101 is initially pretrained on ImageNet-1K [18]. We use
SGD optimizer to optimize the network. The initial learning rate of the
initial step is set to $10^{-2}$, while in the incremental steps, the initial
learning rate is set to $10^{-3}$ for Pascal VOC 2012 and $10^{-2}$ for
ADE20K. The momentum value is 0.9 and the weight decay is $10^{-4}$ in all
steps. We apply the poly learning rate schedule following [12, 3, 8, 21].
During the initial step, the training epoch is set to 50 for Pascal VOC 2012
and 60 for ADE20K, while for each incremental step, we train the network for
20 epochs on Pascal VOC and 100 epochs on ADE20K. The batch size is 16 for
Pascal VOC 2012 and 8 for ADE20K in all steps. The data augmentation follows
the standard of previous work, which is composed of random scaling, random
flipping, and random crop. The coefficients $\lambda_{1}$, $\lambda_{2}$,
$\lambda_{3}$, $\lambda_{4}$, of the objective function are set to 1, 5, 1, 4.
The threshold of pseudo labeling $\tau=0.7$ is set to all settings. The
implementation is built on Pytorch. The experiments are conducted on NVIDIA
RTX2080Ti GPUs.

###  4.2 Quantitative Results

Comparison on Pascal VOC 2012. We evaluate our proposed method with current
exemplar-free DeeplabV3-based methods on the Pascal VOC 2012 dataset. The sub-
tasks of Pascal VOC 2012 have distinct characteristics. For example, the sub-
task 19-1 has more categories in the initial step, while 5-3 has fewer classes
for the initial training and more incremental steps, which is much more
challenging. The results in Tab. 1 show that our proposed method outperforms
other methods on novel classes and overall mIoU results of all sub-tasks.
Particularly, our approach has $1.7\%$ and $3.0\%$ promotion of mIoU in more
challenging sub-tasks of 10-1 and 5-3, respectively. Except for DKD [3] which
refine the background class, our approach achieves at least 3.0% and 7.0% of
promotion on mIoU Compared to the methods with a standard shared background
classifier, demonstrating the effectiveness of the Background Adaptation
mechanism. Besides, our method achieve an impressive advantage on the novel
classes, with significant improvements of 4.5% and 3.5% compared to the
previous state-of-the-art method. Many previous methods emphasize restricting
catastrophic forgetting by boosting the mIoU of larger-proportion old classes
in specific tasks (_e.g_. 15-1) to make the overall mIoU appear to be higher.
However, our approach not only retains the ability to segment old classes but
also significantly improves the accuracy of novel classes, aligning more
closely with the incremental objective emphasized in CISS.

Table 1: Quantitative comparison with state-of-the-art exemplar-free methods on Pascal VOC 2012 in mIoU. Scores of novel classes and all classes in bold are the best while underlined are the second best.  Method |  15-1 (6 steps) |  10-1 (11 steps) |  5-3 (6 steps) |  19-1 (1 step)
---|---|---|---|---
0-15 | 16-20 | all | 0-10 | 11-20 | all | 0-5 | 6-20 | all | 0-19 | 20 | all
MiB [CVPR20] [6] | 38.0 | 13.5 | 32.2 | 12.2 | 13.1 | 12.6 | 57.1 | 42.5 | 46.7 | 71.2 | 22.1 | 68.9
SDR [CVPR21] [40] | 47.3 | 14.7 | 39.5 | 32.4 | 17.1 | 25.1 | - | - | - | 69.1 | 32.6 | 67.4
PLOP [CVPR21] [21] | 65.1 | 21.1 | 54.6 | 44.0 | 15.5 | 30.5 | 25.7 | 30.0 | 28.7 | 75.4 | 37.3 | 73.5
RCIL [CVPR22] [58] | 70.6 | 23.7 | 59.4 | 55.4 | 15.1 | 34.3 | 63.1 | 34.6 | 42.8 | 77.0 | 31.5 | 74.7
EWF [CVPR23] [56] | 77.7 | 32.7 | 67.0 | 71.5 | 30.3 | 51.9 | 61.7 | 42.2 | 47.7 | 77.9 | 6.7 | 74.5
AWT [WACV23] [25] | 59.1 | 17.2 | 49.1 | 33.2 | 18.0 | 26.0 | 61.8 | 45.9 | 50.4 | - | - | -
IDEC [TPAMI23] [63] | 77.0 | 36.5 | 67.3 | 70.7 | 46.3 | 59.1 | 67.1 | 49.0 | 54.1 | - | - | -
DKD [NeurIPS22] [3] | 78.1 | 42.7 | 69.7 | 73.1 | 46.5 | 60.4 | 69.6 | 53.5 | 58.1 | 77.8 | 41.5 | 76.0
Ours | 77.6 | 45.9 | 70.0 | 72.2 | 51.0 | 62.1 | 71.3 | 57.0 | 61.1 | 78.2 | 42.2 | 76.4
Joint | 79.8 | 72.4 | 77.4 | 78.4 | 76.4 | 77.4 | 76.9 | 77.6 | 77.4 | 77.5 | 77.0 | 77.4

(a) Task 15-1 of VOC2012.  (b) Task 10-1 of VOC2012.

Figure 3: The step-wise mIoU comparison of our approach with previous methods under the sub-tasks 10-1 and 15-1 of Pascal VOC 2012.  Table 2: Quantitative comparison with state-of-the-art exemplar-free methods on ADE20K in mIoU. Scores of novel classes and all classes in bold are the best while underlined are the second best.  Method |  100-50 (2 steps) |  100-10 (6 steps) |  50-50 (3 steps) |  100-5 (11 steps)
---|---|---|---|---
0-100 | 101-150 | all | 0-100 | 101-150 | all | 0-50 | 51-150 | all | 0-100 | 101-150 | all
MiB [6] | 40.5 | 17.2 | 32.8 | 38.2 | 11.1 | 29.2 | 45.6 | 21.0 | 29.3 | 36.0 | 5.7 | 26.0
PLOP [21] | 41.9 | 14.9 | 32.9 | 40.5 | 13.6 | 31.6 | 48.8 | 21.0 | 30.4 | 39.1 | 7.8 | 28.8
RCIL [58] | 42.3 | 18.8 | 34.5 | 39.3 | 17.6 | 32.1 | 48.3 | 25.0 | 32.5 | 38.5 | 11.5 | 29.6
EWF [56] | 41.2 | 21.3 | 34.6 | 41.5 | 16.6 | 33.2 | - | - | - | 41.4 | 13.4 | 32.1
AWT [25] | 40.9 | 24.7 | 35.6 | 39.1 | 21.4 | 33.2 | 46.6 | 27.0 | 33.5 | 38.6 | 16.0 | 31.1
GSC [17] | 42.4 | 19.2 | 34.8 | 40.8 | 16.2 | 32.6 | 46.2 | 26.4 | 33.0 | - | - | -
IDEC [63] | 42.0 | 18.2 | 34.1 | 40.3 | 17.6 | 32.7 | 47.4 | 26.0 | 33.1 | 39.2 | 14.6 | 31.0
DKD [3] | 42.4 | 22.9 | 36.0 | 41.6 | 19.5 | 34.3 | 48.8 | 26.3 | 33.9 | - | - | -
Ours | 42.0 | 23.0 | 35.7 | 41.1 | 23.1 | 35.2 | 47.9 | 26.5 | 33.7 | 40.5 | 21.2 | 34.1
Joint | 44.3 | 28.2 | 38.9 | 44.3 | 28.2 | 38.9 | 51.1 | 33.3 | 38.9 | 44.3 | 28.2 | 38.9

Comparison on ADE20K. We evaluate our method on the sub-tasks of more
challenging ADE20K, including 100-50, 100-10, 100-5, and 50-50, as shown in
Tab. 2. Although our method is slightly lower than the state-of-the-art method
in short-term sub-tasks, we are remarkably ahead of other methods in more
challenging long-term 100-10 and 100-5 tasks. The performance in the novel
classes is still outstanding, which demonstrates the universality of our
method.

###  4.3 Ablation Studies

####  4.3.1 Component Ablations.

In this section, we analyze our proposed components, including the Background
Adaptation mechanism, Background Feature Distillation loss
$\mathcal{L}_{BFD}$, Group Knowledge Distillation loss $\mathcal{L}_{GKD}$,
Background Adaptation losses $\mathcal{L}_{BgA^{-}}$ and
$\mathcal{L}_{BgA^{+}}$. The experiments are carried out on the representative
sub-task 15-1 of Pascal VOC 2012. We separately evaluate different
combinations of components and present the results in Tab. 3. The baseline
method applies pseudo-labeling, $\mathcal{L}_{PB-BCE}$, and freezes the whole
network except the classifiers of background and novel classes. Under the same
experimental settings, the Background Adaptation mechanism significantly
promotes 5.2% of overall mIoU, demonstrating its effectiveness in retaining
the knowledge of old classes, and more notably, improving the performance of
novel classes. The usage of distillation strategies with $\mathcal{L}_{BFD}$
and $\mathcal{L}_{GKD}$ preserves the performance of old classes while further
boosting mIoU by 6.0% on novel classes. Moreover, the precise supervision of
the Background Adaptation using $\mathcal{L}_{BgA^{-}}$ and
$\mathcal{L}_{BgA^{+}}$ once again enhances the segmentation of novel classes
with an additional 9.4% of mIoU. Subsequently, there is a growth of 2.4% on
the overall mIoU, which proves that additional precise supervision is
necessary. The results above validate the effectiveness of the Background
Adaptation mechanism and other related losses. There is a notable improvement,
particularly in the performance of novel classes, exceeding twice that of the
baseline.

Table 3: Ablation Study on the components of our proposed method on 15-1. BgA
denotes Background Adaptation. In particular, the experiments w/o distillation
losses (_i.e_. $\mathcal{L}_{BFD}$ and $\mathcal{L}_{GKD}$) freeze the whole
network except the classifier $f^{t}$ following the settings in [8]. All
experiments contain $\mathcal{L}_{PB-BCE}$.

BgA $\mathcal{L}_{BFD}$ $\mathcal{L}_{GKD}$ $\mathcal{L}_{BgA^{-}}$
$\mathcal{L}_{BgA^{+}}$ mIoUini mIoUinc mIoU 73.4 21.8 61.1 ✓ 77.4 30.5 66.3 ✓
✓ 76.7 37.1 67.3 ✓ ✓ ✓ 77.3 36.5 67.6 ✓ ✓ ✓ ✓ 77.1 43.4 69.0 ✓ ✓ ✓ ✓ 76.6 34.1
66.5 ✓ ✓ ✓ ✓ ✓ 77.6 45.9 70.0

Table 4: Ablation study on schemes of Background Adaptation.

Filter mIoUini mIoUinc mIoU MSE-0 ✗ 74.2 41.8 66.5 ✓ 74.6 42.2 66.9 BCE-1 ✓
77.4 44.7 69.6 Ours ✗ 65.7 45.4 60.8 ✓ 77.6 45.9 70.0

Table 5: Ablation Study on the feature distillation (FD) strategies.

mIoUini mIoUinc mIoU w/o FD 56.9 33.2 51.2 $\mathcal{L}_{KD}$ 70.1 38.3 62.6
$\mathcal{L}_{MSE}$ 77.5 44.3 69.6 $\mathcal{L}_{BFD}$ 77.6 45.9 70.0

####  4.3.2 Background Adaptation Schemes.

Sec. 4.3.1 has presented the importance of the Background Adaptation losses.
We have devised multiple schemes to achieve more optimal results of Background
Adaptation, as shown in Tab. 5. Initially, we aim to have the Background
Adaptation value for the current ground truth background region equal to 0,
using Mean Square Error (MSE) loss to minimize the deviation from 0 (MSE-0).
However, MSE-0 couldn’t restrict the background value to absolute 0, leading
to noise accumulation and a lower mIoU compared to Background Adaptation w/
knowledge distillation. Later, we incorporate considerations for the current
class region and introduce a Binary Cross-Entropy (BCE) loss (BCE-1). The
expectation is to have the value in the current class region as small as
possible, while the value in the background class region is as large as
possible. Besides, we apply the Filter to remove the noise above 0. The BCE-1
scheme with Filter improved mIoU by $2\%$, but supervision in the background
region tends to overfit to background objects. Therefore, we retain the BCE
loss in the novel class region and redesign a triplet loss as in Sec. 3.3.3.
The final Background Adaptation losses further enhance the segmentation
capability for novel classes, resulting in a $1.2\%$ improvement.

####  4.3.3 Background Feature Distillation.

To validate the effectiveness of Background Feature Distillation loss
$\mathcal{L}_{BFD}$ in Sec. 3.3.4, we compare $\mathcal{L}_{BFD}$ with the
method w/o feature distillation (w/o FD), w/ standard knowledge distillation
($\mathcal{L}_{KD}$), and w/ mean square error ($\mathcal{L}_{MSE}$). As shown
in Tab. 5, our $\mathcal{L}_{BFD}$ outperforms other representative
distillation methods. Particularly, the Background distillation with MSE,
_i.e_., $\mathcal{L}_{BFD}$, shows a 1.6% promotion of mIoU on novel classes
with no decline in old classes compared to $\mathcal{L}_{MSE}$. This
demonstrates the necessity of preserving a tunnel for learning novel classes.

###  4.4 Qualitative Analysis

We’ve conducted a qualitative analysis of our approach with Baseline and Ours
w/o BgA losses in the $1^{st}$ row and the $4^{th}$ row of Tab. 3. The results
shown in Fig. 4 are visualized from the 15-1 sub-task of VOC 2012. Our
approach in the last three columns reduces the misclassification (_e.g_.,
train in the first two columns of each method) and enhances the precision
(_e.g_., sofa in the last column of each method) of the novel classes, while
maintaining segmentation ability on the old classes. The results showcase the
stability and plasticity of our approach.

Figure 4: Qualitative analysis on 15-1 of Pascal VOC 2012. The Baseline and
Ours w/o BgA losses are in the $1^{st}$ row and the $4^{th}$ row of Tab. 3.
$Ours$ applies additional Background Adaptation losses, compared to the Ours
w/o BgA losses.

##  5 Conclusions

We have introduced a novel CISS method featuring a Background Adaptation
mechanism, designed to prevent the continual shifting of a shared background
class and promote the learning of novel classes. The mechanism explicitly
models the background residual rather than the background at each step and
aggregates these residuals to represent the evolving background. To further
enhance the performance of the Background Adaptation, we customized
corresponding losses on the adaptation channel and improved the distillation
scheme. Experiments on our proposed components verify their effectiveness
under the Background Adaptation mechanism. Moreover, our method surpasses
previous state-of-the-art methods in most sub-tasks, notably excelling in
performance of novel classes.

## Acknowledgement

This work was supported by the National Natural Science Foundation of China
(No. 61972036) and the Industry-University-Institute Cooperation Foundation of
the Eighth Research Institute of China Aerospace Science and Technology
Corporation (No. SAST2022-049).

## References

  * [1] Ahn, H., Cha, S., Lee, D., Moon, T.: Uncertainty-based continual learning with adaptive regularization. NeurIPS 32 (2019)
  * [2] Aljundi, R., Lin, M., Goujaud, B., Bengio, Y.: Gradient based sample selection for online continual learning. NeurIPS 32 (2019)
  * [3] Baek, D., Oh, Y., Lee, S., Lee, J., Ham, B.: Decomposed knowledge distillation for class-incremental semantic segmentation. NeurIPS 35, 10380–10392 (2022)
  * [4] Borsos, Z., Mutny, M., Krause, A.: Coresets via bilevel optimization for continual learning and streaming. NeurIPS 33, 14879–14890 (2020)
  * [5] Boschini, M., Bonicelli, L., Porrello, A., Bellitto, G., Pennisi, M., Palazzo, S., Spampinato, C., Calderara, S.: Transfer without forgetting. In: ECCV. pp. 692–709 (2022)
  * [6] Cermelli, F., Mancini, M., Bulo, S.R., Ricci, E., Caputo, B.: Modeling the background for incremental learning in semantic segmentation. In: CVPR. pp. 9233–9242 (2020)
  * [7] Cha, H., Lee, J., Shin, J.: Co2l: Contrastive continual learning. In: ICCV. pp. 9516–9525 (2021)
  * [8] Cha, S., Yoo, Y., Moon, T., et al.: Ssul: Semantic segmentation with unknown label for exemplar-based class-incremental learning. NeurIPS 34, 10919–10930 (2021)
  * [9] Chen, J., Cong, R., Luo, Y., Ip, H., Kwong, S.: Saving 100x storage: Prototype replay for reconstructing training sample distribution in class-incremental semantic segmentation. Advances in Neural Information Processing Systems 36 (2024)
  * [10] Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Semantic image segmentation with deep convolutional nets and fully connected crfs. arXiv preprint arXiv:1412.7062 (2014)
  * [11] Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE TPAMI 40(4), 834–848 (2017)
  * [12] Chen, L.C., Papandreou, G., Schroff, F., Adam, H.: Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.05587 (2017)
  * [13] Chen, L.C., Zhu, Y., Papandreou, G., Schroff, F., Adam, H.: Encoder-decoder with atrous separable convolution for semantic image segmentation. In: ECCV. pp. 801–818 (2018)
  * [14] Chen, Z., Liu, B.: Lifelong machine learning, vol. 1. Springer (2018)
  * [15] Cheng, B., Misra, I., Schwing, A.G., Kirillov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmentation. In: CVPR. pp. 1290–1299 (2022)
  * [16] Cheng, B., Schwing, A., Kirillov, A.: Per-pixel classification is not all you need for semantic segmentation. NeurIPS 34, 17864–17875 (2021)
  * [17] Cong, W., Cong, Y., Dong, J., Sun, G., Ding, H.: Gradient-semantic compensation for incremental semantic segmentation. arXiv preprint arXiv:2307.10822 (2023)
  * [18] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: CVPR. pp. 248–255 (2009)
  * [19] Dhar, P., Singh, R.V., Peng, K.C., Wu, Z., Chellappa, R.: Learning without memorizing. In: CVPR. pp. 5138–5146 (2019)
  * [20] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. In: ICLR (2021)
  * [21] Douillard, A., Chen, Y., Dapogny, A., Cord, M.: Plop: Learning without forgetting for continual semantic segmentation. In: CVPR. pp. 4040–4050 (2021)
  * [22] Ermis, B., Zappella, G., Wistuba, M., Rawal, A., Archambeau, C.: Memory efficient continual learning with transformers. NeurIPS 35, 10629–10642 (2022)
  * [23] Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge. IJCV 88, 303–338 (2010)
  * [24] Fang, Y., Zhu, F., Cheng, B., Liu, L., Zhao, Y., Wei, Y.: Locating noise is halfway denoising for semi-supervised segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 16612–16622 (2023)
  * [25] Goswami, D., Schuster, R., van de Weijer, J., Stricker, D.: Attribution-aware weight transfer: A warm-start initialization for class-incremental semantic segmentation. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 3195–3204 (2023)
  * [26] Hariharan, B., Arbeláez, P., Bourdev, L., Maji, S., Malik, J.: Semantic contours from inverse detectors. In: ICCV. pp. 991–998 (2011)
  * [27] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR. pp. 770–778 (2016)
  * [28] Jung, H., Ju, J., Jung, M., Kim, J.: Less-forgetting learning in deep neural networks. arXiv preprint arXiv:1607.00122 (2016)
  * [29] Kirkpatrick, J., Pascanu, R., Rabinowitz, N., Veness, J., et al.: Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences 114(13), 3521–3526 (2017)
  * [30] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. NeurIPS 25 (2012)
  * [31] LeCun, Y., Bottou, L., Bengio, Y., Haffner, P.: Gradient-based learning applied to document recognition. Proceedings of the IEEE 86(11), 2278–2324 (1998)
  * [32] Li, Z., Hoiem, D.: Learning without forgetting. IEEE TPAMI 40(12), 2935–2947 (2017)
  * [33] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: ICCV. pp. 10012–10022 (2021)
  * [34] Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semantic segmentation. In: CVPR. pp. 3431–3440 (2015)
  * [35] Lopez-Paz, D., Ranzato, M.: Gradient episodic memory for continual learning. NeurIPS 30 (2017)
  * [36] Madaan, D., Yoon, J., Li, Y., Liu, Y., Hwang, S.J.: Representational continuity for unsupervised continual learning. In: ICLR (2022)
  * [37] Maracani, A., Michieli, U., Toldo, M., Zanuttigh, P.: Recall: Replay-based continual learning in semantic segmentation. In: ICCV. pp. 7026–7035 (2021)
  * [38] McClelland, J.L., McNaughton, B.L., O’Reilly, R.C.: Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review 102(3), 419 (1995)
  * [39] McCloskey, M., Cohen, N.J.: Catastrophic interference in connectionist networks: The sequential learning problem. In: Psychology of learning and motivation, vol. 24, pp. 109–165. Elsevier (1989)
  * [40] Michieli, U., Zanuttigh, P.: Continual semantic segmentation via repulsion-attraction of sparse and disentangled latent representations. In: CVPR. pp. 1114–1124 (2021)
  * [41] Oh, Y., Baek, D., Ham, B.: Alife: Adaptive logit regularizer and feature replay for incremental semantic segmentation. NeurIPS 35, 14516–14528 (2022)
  * [42] Ostapenko, O., Puscas, M., Klein, T., Jahnichen, P., Nabi, M.: Learning to remember: A synaptic plasticity driven framework for continual learning. In: CVPR. pp. 11321–11329 (2019)
  * [43] Pham, Q., Liu, C., Hoi, S.: Dualnet: Continual learning, fast and slow. NeurIPS 34, 16131–16144 (2021)
  * [44] Rebuffi, S.A., Kolesnikov, A., Sperl, G., Lampert, C.H.: icarl: Incremental classifier and representation learning. In: CVPR. pp. 2001–2010 (2017)
  * [45] Ritter, H., Botev, A., Barber, D.: Online structured laplace approximations for overcoming catastrophic forgetting. NeurIPS 31 (2018)
  * [46] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: Proc. International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 234–241 (2015)
  * [47] Schwarz, J., Czarnecki, W., Luketina, J., Grabska-Barwinska, A., Teh, Y.W., Pascanu, R., Hadsell, R.: Progress & compress: A scalable framework for continual learning. In: Proc. International Conference on Machine Learning. pp. 4528–4537. PMLR (2018)
  * [48] Shang, C., Li, H., Meng, F., Wu, Q., Qiu, H., Wang, L.: Incrementer: Transformer for class-incremental semantic segmentation with knowledge distillation focusing on old class. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7214–7224 (2023)
  * [49] Shin, H., Lee, J.K., Kim, J., Kim, J.: Continual learning with deep generative replay. NeurIPS 30 (2017)
  * [50] Shon, H., Lee, J., Kim, S.H., Kim, J.: Dlcft: Deep linear continual fine-tuning for general incremental learning. In: ECCV. pp. 513–529 (2022)
  * [51] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2015)
  * [52] Strudel, R., Garcia, R., Laptev, I., Schmid, C.: Segmenter: Transformer for semantic segmentation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 7262–7272 (2021)
  * [53] Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jégou, H.: Training data-efficient image transformers & distillation through attention. In: Proc. International Conference on Machine Learning. pp. 10347–10357 (2021)
  * [54] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. NeurIPS 30 (2017)
  * [55] Wu, C., Herranz, L., Liu, X., Van De Weijer, J., Raducanu, B., et al.: Memory replay gans: Learning to generate new categories without forgetting. NeurIPS 31 (2018)
  * [56] Xiao, J.W., Zhang, C.B., Feng, J., Liu, X., van de Weijer, J., Cheng, M.M.: Endpoints weight fusion for class incremental semantic segmentation. In: CVPR. pp. 7204–7213 (2023)
  * [57] Yang, G., Fini, E., Xu, D., Rota, P., Ding, M., Nabi, M., Alameda-Pineda, X., Ricci, E.: Uncertainty-aware contrastive distillation for incremental semantic segmentation. IEEE TPAMI 45(2), 2567–2581 (2022)
  * [58] Zhang, C.B., Xiao, J.W., Liu, X., Chen, Y.C., Cheng, M.M.: Representation compensation networks for continual semantic segmentation. In: CVPR. pp. 7053–7064 (2022)
  * [59] Zhang, G., Wang, L., Kang, G., Chen, L., Wei, Y.: Slca: Slow learner with classifier alignment for continual learning on a pre-trained model. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 19148–19158 (2023)
  * [60] Zhang, J.O., Sax, A., Zamir, A., Guibas, L., Malik, J.: Side-tuning: a baseline for network adaptation via additive side networks. In: ECCV. pp. 698–714 (2020)
  * [61] Zhang, Z., Gao, G., Fang, Z., Jiao, J., Wei, Y.: Mining unseen classes via regional objectness: A simple baseline for incremental segmentation. NeurIPS 35, 24340–24353 (2022)
  * [62] Zhang, Z., Gao, G., Jiao, J., Liu, C.H., Wei, Y.: Coinseg: Contrast inter-and intra-class representations for incremental segmentation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 843–853 (2023)
  * [63] Zhao, D., Yuan, B., Shi, Z.: Inherit with distillation and evolve with contrast: Exploring class incremental semantic segmentation without exemplar memory. IEEE TPAMI (2023)
  * [64] Zhou, B., Zhao, H., Puig, X., Fidler, S., Barriuso, A., Torralba, A.: Scene parsing through ade20k dataset. In: CVPR. pp. 633–641 (2017)
  * [65] Zhu, H., Wei, Y., Liang, X., Zhang, C., Zhao, Y.: Ctp: Towards vision-language continual pretraining via compatible momentum contrast and topology preservation. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 22257–22267 (2023)
  * [66] Zhu, L., Chen, T., Yin, J., See, S., Liu, J.: Continual semantic segmentation with automatic memory sample selection. In: CVPR. pp. 3082–3092 (2023)


# Morphology and structural properties of thin rubrene crystallites grown on
graphite

Moha Naeimi a, b, Katharina Engster a, b, Ingo Barke a, b, Sylvia Speller a, b
a Institute of Physics, University of Rostock, Albert-Einstein-Str. 23, 18059
Rostock, Germany.
b Department “Life, Light and Matter”, University of Rostock, Albert-Einstein-
Str. 25, 18059 Rostock, Germany.
<EMAIL_ADDRESS>

###### Abstract

Crystallization of rubrene, progressing from an amorphous phase to a triclinic
meta-stable and ultimately to the orthorhombic stable phase, offers broad
applications not only in organic electronic devices but also for in-depth
studies of optical and electronic properties, including exciton distribution
and dynamics. We investigate the crystallization of rubrene on highly oriented
pyrolytic graphite (HOPG), aiming at the growth of the preferred rubrene
orthorhombic phase, which has been reported to have one of the highest charge
mobilities in organic semiconductors. This is achieved through controlled
heating and enhanced partial pressure. Through precise control of the initial
deposition on the substrate, we investigate the growth habit of rubrene
crystals by high-rate heat treatment beyond the second crystallization
temperature. Furthermore, this work addresses thermal stability and
photodegradation across various morphologies.

##  1 Introduction

Rubrene crystalline structures have been subject of intense research in the
context of organic electronics [1], e.g. for field-effect transistors [2, 3]
and organic light emitting diodes [4, 5]. Rubrene is a small, organic molecule
which can be thought to be based on tetracene with 4 additional phenyl rings.
It shows various interesting electronic properties such as high charge
mobility [6, 7, 8], efficient singlet fission [9, 10] with long life time and
long diffusion length [11], triplet-triplet annihilation and fusion [12, 13]
and triplet energy transport [12]. Due to these properties, rubrene is a
promising candidate for energy and exciton transfer both in lateral and
vertical direction with respect to the surface. Rubrene singlet fission has
been widely studied in the past decades. It is well known that singlet fission
in rubrene is more efficient in crystalline stable phases [14] while in
amorphous structures it is significantly lower [15]. Hence, crystalline
structures are necessary to provide a playground to extract its unique
electronic properties.

Growth of rubrene crystalline domains both in platelets and spherulites [16,
17] has been the subject of several studies aiming for large lateral
dimensions by tailored heating treatment. However, preparing extended rubrene
crystals directly on a substrate is not simple because the structures have the
tendency to become trapped in a metastable state of low order or in an
unfavorable crystal structure during the preparation process [18]. The
challenge essentially lies in the fact that providing sufficient thermal
energy to overcome this metastable state easily results in massive desorption,
eventually evaporating all of the available material on the substrate. To
overcome this, a variety of methods have been proposed, which are based on
temperature treatments or epitaxial methods [19, 20, 21, 22, 23]. A key idea
is to reach the crystallization temperature while preventing desorption, by
providing a high rate heat transfer to amorphous rubrene in the local
environment of the substrate.

Most of the studies use silicon substrates with a native oxide layer and
different adlayers. The aim of this work is to control the structure and
morphology of highly crystalline, thin and flat rubrene crystallites on bare
HOPG. In particular, well separated crystals with laterally extended single
domains are sought. Special focus is on simplicity, reproducibility, purity
and flexibility. Such samples shall be suitable for exciton transfer studies
using surface science techniques such as local photoelectron spectroscopy
[24]. Important is to prevent optical waveguiding [25], i.e. along at least
one dimension crystal should measure only a few 10 nm.

Among the crystal phases of rubrene, the well-known orthorhombic phase had
been reported to have a very high charge mobility of 43
$cm\textsuperscript{2}/Vs$ [26], one of the highest charge mobilities reported
in organic semiconductors. Rubrene molecules are arranged in this phase in a
way such that their intermolecular interactions are minimized by their phenyl
rings. Although the orthorhombic crystal phase of rubrene is known to be the
thermodynamically stable phase, in contrast to the triclinic phase which is
metastable, it has been a challenge to grow in thin, yet ultra-smooth crystals
directly on a surface.

In the present paper, we investigate the control of the growth of rubrene
crystalline structures on highly oriented pyrolytic graphite (HOPG) using the
method of abrupt-heating [27]. We use HOPG as a substrate, because it has a
well-defined, flat surface on the atomic scale, it is conductive, and it is
inert which simplifies sample handling. Apart from rubrene, no other co-
materials such as buffer layers are used throughout this work to aid growth
[28, 29, 16, 30]. We also address thermal stability and occurrence of photo-
oxidation processes.

Figure 1: (a - c) Schematic representations of rubrene orthorhombic and
triclinic unit cell along with rubrene molecule, respectively. In the
stacking, only the tetracene backbone is shown for simplicity. (d) Bright-
field image with a magnified image of a single crystal prepared by spin-
coating and high-rate heating (e) AFM image of the crystal in (d). (f) AFM
image showing step heights of $\approx$1.3 nm (g) Line profile at the location
indicated in (e).

##  2 Experimental section

Rubrene powder with a purity of $99.999,/\%$ (Sigma-Aldrich) was used as
received and the container was kept in a desiccator to prevent oxidation. For
preparation using the high-rate heating approach, also referred to as abrupt
heating, the sample is directly covered with rubrene, either using spin-
coating or high-vacuum evaporation (see below for details). Then the sample
was placed onto a pre-heated hot plate. The temperature of the sample was
carefully controlled and verified by a K-type thermocouple to anneal the
amorphous phase at the second crystallization stage of 170∘C for different
heating times from 1 min to 1 hour. For spin-coating, we used toluene and
chloroform as base solvents with different solute concentrations ranging from
$10^{-3}$ to $10^{-5}$ molar. For high-vacuum evaporation, the rubrene powder
was placed inside a heated glass vessel, whose target temperature was changed
between 170 to 200∘C to control the evaporation rate. The substrate was kept
at room temperature. The base pressure of the evaporation chamber was of the
order of $10^{-7}$ mbars. HOPG was cleaved in air with adhesive tape and no
further annealing was done.

After preparation, the crystals were investigated by optical microscopy (Zeiss
Axio scope.A1), fluorescence microscopy (Olympus IX73 inverted fluorescence
microscope), polarization optical microscopy (POM) (Zeiss Axiolab.A5) and
atomic force microscopy (AFM)(Park Systems NX20 and XE100).

For photoemission electron microscopy (PEEM) we introduced the samples to a
PEEM (Focus IS-PEEM) with the base pressure of $10^{-10}$ mbar and used the
3rd harmonics of a tunable Ti:Sa femtosecond laser (Mira 900F) yielding
photons with energies equal to 4.6 eV (266 nm). The light angle of incident
was 67 degrees.

Visualisation and analysis was done by Gwyddion [31]and Igor Pro
(Wavemetrics). The evaluation of the area coverage was done by estimation of
the apparently covered area, marked manually, using ImageJ software.

##  3 Results and discussion

###  3.1 Spin coating and platelets

Among the three different crystal phases of rubrene, there is a competition
between triclinic and orthorhombic phases during heating. The monoclinic phase
has the weakest intermolecular binding, and is not regarded in the present
work [32] since the growth does not happen on surfaces but only in solution
[33]. The triclinic phase is a result of low-rate heat transfer or heating at
the first crystallization temperature (140∘C) [34], while a high-rate heat
transfer to amorphous rubrene or heating at the second crystallization
temperature (170∘C) yields the formation of the orthorhombic phase. Schemes of
the rubrene molecule, the triclinic, and the orthorhombic structure are
depicted in figure 1.a to c, along with the respective main crystal axes.
Figure 1.d shows an overview brightfield image of the well separated crystals
prepared by spin-coating 5 $\mu$L of a 1 mMol rubrene solution onto HOPG,
using toluene as a solvent. Subsequently, the sample underwent a high-rate
heating step for one minute on a preheated hot plate at a temperature of
170∘C, within a nitrogen-filled container. The inset of Figure 1.d is the
magnified brightfield image of one of these crystals. As shown in Figure 1.e,
which is an AFM image of the very same crystal in figure 1.d, the leaf-shape
platelets are quite thin, their height typically measures a few 10 nm. The
line profile in 1.e is indicated in Figure 1.f. Figure 1.f shows a high
resolution AFM image and a line profile at the top of the crystal revealing
single-layer steps. Step heights are about $\approx$1.3 nm, implying an
orthorhombic phase with the c-axis of the unit cell oriented perpendicular to
the surface [35].

A competition between single- and polycrystalline orthorhombic, and the
polycrystalline triclinic phase was observed, which can be controlled by
increasing the volume of spin-coated solution. Figure 2.a and 2.b show
overview brightfield and fluorescence images of an exemplary region with local
high (bottom right) and low (top left) coverage, respectively, prepared by
spin coating of 20 uL rubrene-toluene solution, subsequently subjected to
high-rate heating for one minute at 170∘C. It shows coexisting orthorhombic,
triclinic and amorphous domains. As indicated in figure 2.c, spin coating of
more solution onto HOPG favours the polycrystalline triclinic phase, while
orthorhombic single crystallites are the result of low solution coverage. A
reasoning why more solution, i.e., more molecules, reduce the emergence of the
orthorhombic phase, can be done considering the balance between flux or
delivery rate and diffusion. In spin-coating the concentration and volume of
the molecules and solution determines the rate of molecules delivered. If the
delivery is fast, many nucleation centers occur at high density in a short
time period. Upon further growth of these nuclei domain boundaries will result
because the different islands / grains cannot be monolithically connected.
This favors phases with lower order, such as the triclinic polymorph.

Figure 2: Sample prepared by spin-coating and high-rate heating. (a and b)
Bright-field and fluorescence images showing three different morphologies of
rubrene formed on HOPG. The triclinic domains (marked blue) show high
fluorescence intensity and form in areas with high rubrene coverage in
contrast to orthorhombic single crystallites (marked red) which grow in areas
with low rubrene coverage. (c) A semi-quantitative analysis of rubrene
polymorphs formed after high-rate heating versus solution coverage spin-coated
on HOPG.

One factor that determines the thickness of crystals is the total number of
high-rate heating steps. Figure 3 shows a series of brightfield images of a
crystal prepared by spin coating of 0.5 uL rubrene-toluene solution on HOPG
and treated consecutively with high-rate heating, each for 60 seconds at
170∘C. After the first heat treatment, the crystal thickness, based on
brightfield-AFM correlations, is estimated to be 40 nm, while the second heat
treatment changed the crystal thickness to a couple of hundred nm while the
lateral size slightly increased. A third high-rate heat treatment was
performed and resulted a sub-10 nm thickness change and for the fourth heating
treatment, no change in thickness was observed anymore. The final platelets
show step heights that match the orthorhombic crystalline phase of rubrene.
Figure 3.d shows the average growth of the single crystals by measuring the
thickness of various samples with AFM, through the first and second high-rate
heating step. We do not observe indications of oxidation for these crystals
(see below).

To summarize this section, we obtain well-separated single orthorhombic
platelets with exceptionally flat surfaces when starting from a solution which
is deposited on HOPG through spin coating with low coverage, followed by high-
rate heat treatment. The same preparation conducted with elevated coverage
yields an increased fraction of triclinic crystals. Interestingly, the same
preparation on silicon with a native oxide layer results in amorphous,
droplet-like structures, with occasional triclinic polycrystalline phases,
independent of the coverage. We attribute this observation to the fact that
HOPG has an atomically well-defined surface, while the oxide layer on silicon
is amorphous, hence impeding the initial ordered growth of large orthorhombic
domains.

Figure 3: Sample prepared by spin-coating and consecutive high-rate heating.
(a) - (c) A series of bright-field images showing the growth evolution through
consecutive heating (d) Average growth in thickness from different samples
with AFM images of a sample before and after 2nd high-rate heating. The error
bars indicate the standard deviation of the thickness of a selection of $>$ 20
measured samples for each heating stage.

###  3.2 Photo-oxidation assessment

The fluorescence intensity of rubrene crystalline phases strongly depends on
their structure and orientation [36]. We introduced a large number of our
crystals into a fluorescence microscope and compared their intensity and
monitored their photo bleaching behavior. As we observed, the fluorescence
intensity is significantly higher for the triclinic phase compared to the flat
platelets belonging to the orthorhombic phase, which can be used for quick and
easy assignment for crystalline phases (along with supporting AFM
investigations). Since the triclinic phase of rubrene has a polycrystalline
nature by itself [37] and the orthorhombic phase usually tends to grow in a
single crystalline domain, it is interesting to compare the photo-degradation
behaviour between those two. Figure 4.a and 4.b show fluorescence images of a
polycrystalline rubrene assembly prepared by spin coating 5 uL of rubrene-
toluene solution onto HOPG, followed by high-rate heating at 170∘C for one
minute. One can see that the triclinic part (red marked) yields high
fluorescence intensity (4.a) but after three hours of light exposure (4.b), it
degrades to almost zero. Figure 4.d shows the intensity of the marked parts
versus exposure time. After two hours of illumination, the orthorhombic part
seems to stay stable with no sign of further degradation. Interestingly, the
triclinic phase shows an inflection point, meaning the degradation of
fluorescence intensity is most rapid between one and two hours after start of
exposure by the microscope (100 Watt Halogen lamp).

The difference in rate of photo-degradation between these two phases could be
explained by the defect density [38, 39] which is much larger for triclinic
structures. Insets of figures 4.a and 4.b show the AFM height image of a
related orthorhombic crystal indicating that photo-oxidation predominantly
occurs at the step edges and defect spots on the surface. Previous reports
have indicated that this process can also penetrate into the bulk [38, 39,
40]. The AFM phase image is also suitable to demonstrate this oxidation
behaviour, because the AFM tip experiences variations in adhesion and friction
forces for different chemical compositions [41]. These differences in
interaction result in phase shifts that could be observed in the phase image
in figure S1.

Figure 4: Photo-bleaching of rubrene polycrystalline structure prepared by
spin-coating and high-rate heating. (a and b) Fluorescence image of the
rubrene polycrystal before and after light exposure. The insets are high
resolution AFM images of the orthorhombic part (marked blue) of the
polycrystal showing the photo-oxidation occurring at the step edges. For the
triclinic part, no change of morphology is visible by AFM. d: Fluorescence
degradation profile of two different crystal phases of rubrene during 3 hours
of light exposure (excitation wavelength: 540 nm).

The transition dipole moment of rubrene is along the short flat axis of the
molecule (i.e. horizontally within the plane of figure 1a). Due to the
polycrystalline nature of the triclinic phase of rubrene, the transition
dipole moments of the molecules have random orientations and the chance of
matching these and the polarization of the light (which is in-plane in our
case) is significantly higher than for the orthorhombic phase whose dipole
moment is oriented perpendicular to the polarization of our excitation light.
Also, distinct polymorphs have much larger effect in luminescence yield
compared to correlation between thickness and luminescence. In contrast, in
orthorhombic single crystals, the crystal thickness and luminescence are found
to be correlated (Figure S2).

Figure 5: a: Polarization optical microscope image of rubrene crystal colonies
grown with almost radial symmetry. The orthorhombic platelets are embedded in
a bath of needle-shape crystals forming triclinic spherulites. The inset of
image a is a PEEM image of the marked platelet showing different photoemission
yield for different crystal domains. b and c: Fluorescence images of the
triclinic spherulites. The polarization of the emitted light is filtered along
0 (horizontally) and 90 (vertically), respectively. d: POM image of the marked
platelet in a. The sample is rotated 45 degrees clockwise with the same
polarization orientation in a. e: AFM image of the same platelet in d. f: AFM
image of the triclinic needle-shape crystal colonies marked in c with dashed
square.

###  3.3 Vacuum deposition and spherulites

The initial rubrene coverage required for high-rate heating preparation can be
provided by various ways. Apart from spin-coating, as described above, vacuum
evaporation of a thin layer of amorphous rubrene on HOPG is another option.
This results in structures summarized in figure5 showing polarization optical
microscopy, fluorescence, AFM, and PEEM images. Large and relatively rough
crystalline domains occur, where the two most dominant structures are
spherulite-like crystal colonies (figure 5.a) and platelets (figure 5.d and
5.e). Figure 5.a is a polarization optical microscope image of rubrene
orthorhombic platelets embedded in needle-shape crystal colonies prepared by
evaporating a 20 nm layer of rubrene on HOPG in high vacuum and subsequently
high-rate heat treated in complete darkness. These structures which are known
as triclinic spherulites [37] are colonies of single crystals formed around
the center of nucleation in an almost radially symmetric way. The inset of
image 5.a shows the PEEM image of the platelet (marked red) revealing, that
the domains of the platelets have different photoemission yields, most likely
caused by different absorption efficiencies of the laser light. As shown in
figure 5.b and c, by rotating the orientation of the emission polarizer of the
fluorescence microscope, only crystals with matching fluorescence polarization
appear bright.

Figure 6: (a) Bright-field image of the area of a large rubrene coverage
gradient, created using a shadow mask (left region covered, right region
exposed). The growth of ultra-thin (thickness $<$10 nm) and ultra-flat (RMS
roughness less than 100 pm) single crystals in the shadowed area is a result
of surface diffusion during the evaporation phase. (b) Coverage of different
possible structures versus the evaporated layer thickness on various samples.

We analyse the spherulites by means of polarization optical microscopy (in
reflection, with entrance and exit polarizers 90° crossed) and fluorescence
microscopy (in reflection, with only the luminescence light passing a
polarizer). The imaging contrast with POM (figure 5.a) is clearly along radial
axes of the spherulites. Since the polarizers were crossed, the detected light
will be dominated by structures which turn the electric field of the light by
90°, which is unlikely in absence of luminescence or birefringence. If the
sample is turned 90° contrast swaps. Since the excitation light is from a LED
with largely white light, we tend to attribute the contrast to a mixture of
both. The imaging contrast of the fluorescence images (figures 5.b and 5.c) is
however exclusively from the emitted, i.e., luminescent light. The
illumination light has no preferred polarization. It reflects the radial
symmetry of the spherulites containing small grains with triclinic structure,
as has been found by x-ray diffraction [[37]].

Figure 5.d is the magnified POM image of the platelet marked in 5.a. The
sample is rotated 45 degrees clockwise with the same polarization orientation
in 5.a which results in changing the color and brightness of different
domains. The AFM image of the platelet in figure 5.e visualizes the growth
mode from a center of nucleation to the edges of platelet with a relatively
segmented rim. This is in contrast to platelets prepared by spin coating which
yield ultra flat and smooth surfaces (compare figure 1.b and 5.f). Figure 5.f
is an AFM image of the edge of a triclinic spherulite, revealing that these
spherulites are consisting of small needle-shaped crystals having almost 100
nm of thickness. The step heights of both structures are $\approx$1.3 nm and
reveal their orthorhombic nature.

As an intermediate conclusion, evaporation with subsequent high-rate heat
treatment yields vastly different morphologies, and hence fluorescence
properties, compared to solution coating as the first step. Evaporated rubrene
results in a wide range of crystal morphologies, including the orthorhombic
large crystalline domains known as spherulites and platelets that were
repeatedly reported previously [28, 29, 42, 16, 30] using more complex
preparation schemes. Often, considerable efforts have been spent to grow these
domains such that they interconnect and fully cover the surface, e.g., by
adding underlayers such as TPTPA and TPD [42], PMMA [16] and ITO [29]. On bare
HOPG, these orthorhombic platelets (figure 5.d) are rather embedded in a bath
of needle shape crystal colonies as large triclinic spherulite domains
(figures 5.a, 5.b and 5.c). The orthorhombic platelets appear with different
intensity and color in POM images due to their crystal order and
orientation[27]. Since in the orthorhombic phase the transition dipole moment
is perpendicular to the surface, its fluorescence is completely suppressed
which we indeed observe for the platelets (figure S3), such as those in figure
5.d, at the same time revealing their perfect crystal ordering with the c axis
perpendicular to the surface. [36].

Figure 7: a and b: brightfield and AFM image of an orthorhombic platelet
formed at the first high-rate heating process for 60 seconds. c and d:
brightfield and AFM image of a platelet after performing the seconds high-rate
heating showing the transform of the platelet to the separated bulk crystal.
The step heights reveal the orthorhombic phase of rubrene crystalline
structures.

Based on the step heights on platelets which could be found all over the
surface, these crystals belong to the orthorhombic phase with line and point
defects. Although the orthorhombic phase is dominant when one heats the
substrate at the second crystallization temperature which is around 170∘C,
there is always the coexisting pathway of a transformation into the triclinic
phase. The triclinic growth of rubrene, which is, as previously mentioned, a
meta-stable phase, is particularly common for low-rate heating i.e., gradually
heating the substrate from room temperature close to the first crystallization
temperature around 140∘C. By gradually heating the substrate, the molecules
have enough time to randomly attach to each other and form diffusion limited
aggregates, which results in a dendritic, yet crystalline growth with a rough
surface. In order to gain more insight into the competition between these
phases, we now investigate the coverage dependency of their appearance, by
means of a semi quantitative analysis [34].

One way to qualitatively analyze the coverage dependence is to inspect
locations of large local coverage gradients, because a coverage dependency
will be well visible as an inhomogeneous distribution of grown structures. We
created such inhomogeneities by applying a shadow mask during vacuum
evaporation, combined with an increased amount of rubrene (thickness:
$\approx$50 nm). Figure 6.a shows an overview brightfield image after high-
rate heat treatment at 170∘C for one minute. The corresponding crystal
structures were identified by comparing optical microscopy and AFM
investigations (see also Section 3.1). A well defined transition line between
orthorhombic and triclinic structures emerges at the shadowed boarder: in
figure 6.a orthorhombic ultra-flat and thin single crystals, indicated by
arrows, are grown in the shadowed area where the rubrene coverage is extremely
low due to low rubrene-on-HOPG diffusivity. But with an increase in rubrene
coverage toward the border, orthorhombic platelets grow in the region marked
by a dashed rectangle. They exhibit higher surface roughness which is even
higher at escalated local rubrene coverage, leading to the growth of triclinic
dendritic crystals. In the region right from the dashed rectangle, the growth
is triclinic. Figure 6.b shows a graph of the area covered by the different
crystal species after one minute of high-rate heating as a function of rubrene
layer thickness. In conclusion, for both, spin-coating and vacuum evaporation,
higher coverage of rubrene favors the triclinic phase over orthorhombic one.

In view of the case for spin-coated samples, the question is whether crystal
morphology and thickness is affected by the number of heating steps in a
similar fashion. Figure 7.a shoes a bright field image of a platelet prepared
by evaporation of a 20 nm layer rubrene on HOPG and then high-rate heat
treated for one minute at 170∘C. An AFM image of a related platelet in figure
7.a is shown in Figure 7.b. Figure 7.c shows the same sample after second,
identical heat treatment with figure 7.d showing the related AFM image. These
indicate that the thickness of the crystals increase by roughly 100 nm (see
figure 7.b and 7.d) and the domains transform to single bulk crystalline
packing upon a second high-rate heating step. The first heating step for one
minute obviously does not provide enough energy to the system to let all the
deposited rubrene contribute to the crystallization and the remaining material
can then boost the thickness during the second heat treatment, a similar
behavior as observed for spin-coating preparation (figure 3).

It was previously reported, for other substrates, that by heating the
evaporated rubrene, platelets and spherulites start to grow preferentially in
lateral size [34] rather than in thickness. Upon further heating steps, we
observe here that both the crystal morphology and thickness change for
platelets and spherulites (see figure 7). This means there must be a reservoir
(e.g. consisting of hardly visible few or monolayer rubrene) of rubrene which
can reach the crystallites via transport across the surface, related to the
process of Ostwald ripening [43]. It could be related to the better
diffusivity on HOPG compared to other substrates.

##  4 Conclusion

Rubrene crystalline structures were prepared on bare HOPG using high-rate
heating methods. The morphology of such molecular assemblies could be
controlled by the amount and method of prior rubrene deposition, as well as by
the duration, number and rate of heating treatments. Orthorhombic crystal
growth is the result of low coverage while high coverage favours triclinic
growth. The fluorescence intensities for these structures were investigated
and we showed that the fluorescence response is a solid method to determine
the quality and internal orientation of rubrene crystals. The orthorhombic
crystal phase is much more resistant to photo-degradation, in contrast to the
triclinic phase.

##  5 Acknowledgement

Funding by the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation) within project number 441234705, SFB 1477 "Light-Matter
Interactions at Interfaces" and project number 299150580, SFB 1270
"Electrically Active Implants" is acknowledged. We thank Prof. Stefan
Lochbrunner and Tim Völzer for fruitful discussion and advice for the
manuscript.

## References

  * [1] Kathryn A. McGarry, Wei Xie, Christopher Sutton, Chad Risko, Yanfei Wu, Victor G. Young, Jean-Luc Brédas, C. Daniel Frisbie, and Christopher J. Douglas.  Rubrene-based single-crystal organic semiconductors: Synthesis, electronic structure, and charge-transport properties.  Chemistry of Materials, 25(11):2254–2263, May 2013.
  * [2] Kihyun Kim, Min Ki Kim, Han Saem Kang, Mi Yeon Cho, Jinsoo Joo, Ju Hee Kim, Kyung Hwan Kim, Chang Seop Hong, and Dong Hoon Choi.  New growth method of rubrene single crystal for organic field-effect transistor.  Synthetic Metals, 157(10-12):481–484, June 2007.
  * [3] Yiwei Duan, Bowen Zhang, Shizan Zou, Chuqi Fang, Qijing Wang, Yi Shi, and Yun Li.  Low-power-consumption organic field-effect transistors.  Journal of Physics: Materials, 3(1):014009, January 2020.
  * [4] Shu-Jen Wang, Anton Kirch, Michael Sawatzki, Tim Achenbach, Hans Kleemann, Sebastian Reineke, and Karl Leo.  Highly crystalline rubrene light-emitting diodes with epitaxial growth.  Advanced Functional Materials, 33(14), January 2023.
  * [5] Zhaokui Wang, Hiroyuki Okada, and Shigeki Naka.  Evaluation of reliability in rubrene-based organic light emitting devices with a mixed single layer.  Japanese Journal of Applied Physics, 49(1):01AA02, January 2010.
  * [6] Arie van der Lee, Maurizio Polentarutti, Gilles H. Roche, Olivier J. Dautel, Guillaume Wantz, Frédéric Castet, and Luca Muccioli.  Temperature-dependent structural phase transition in rubrene single crystals: The missing piece from the charge mobility puzzle?  The Journal of Physical Chemistry Letters, 13(1):406–411, January 2022.
  * [7] Keke K. Zhang, Kejie Tan, Changji Zou, Magnus Wikberg, Laurie E. McNeil, Subodh G. Mhaisalkar, and C. Kloc.  Control of charge mobility in single-crystal rubrene through surface chemistry.  Organic Electronics, 11(12):1928–1934, December 2010.
  * [8] Jun Nitta, Kazumoto Miwa, Naoki Komiya, Emilia Annese, Jun Fujii, Shimpei Ono, and Kazuyuki Sakamoto.  The actual electronic band structure of a rubrene single crystal.  Scientific Reports, 9(1), July 2019.
  * [9] Tong Wu, Wenjun Ni, Gagik G. Gurzadyan, and Licheng Sun.  Singlet fission from upper excited singlet states and polaron formation in rubrene film.  RSC Advances, 11(8):4639–4645, 2021.
  * [10] Ilana Breen, Roel Tempelaar, Laurie A. Bizimana, Benedikt Kloss, David R. Reichman, and Daniel B. Turner.  Triplet separation drives singlet fission after femtosecond correlated triplet pair production in rubrene.  Journal of the American Chemical Society, 139(34):11745–11751, August 2017.
  * [11] Drew M. Finton, Eric A. Wolf, Vincent S. Zoutenbier, Kebra A. Ward, and Ivan Biaggio.  Routes to singlet exciton fission in rubrene crystals and amorphous films.  AIP Advances, 9(9), September 2019.
  * [12] P. Baronas, G. Kreiza, L. Naimovičius, E. Radiunas, K. Kazlauskas, E. Orentas, and S. Juršėnas.  Sweet spot of intermolecular coupling in crystalline rubrene: Intermolecular separation to minimize singlet fission and retain triplet–triplet annihilation.  The Journal of Physical Chemistry C, 126(36):15327–15335, August 2022.
  * [13] Sebastian Engmann, Adam J. Barito, Emily G. Bittle, Noel C. Giebink, Lee J. Richter, and David J. Gundlach.  Higher order effects in organic LEDs with sub-bandgap turn-on.  Nature Communications, 10(1), January 2019.
  * [14] Lin Ma, Keke Zhang, Christian Kloc, Handong Sun, Maria E. Michel-Beyerle, and Gagik G. Gurzadyan.  Singlet fission in rubrene single crystal: direct observation by femtosecond pump–probe spectroscopy.  Physical Chemistry Chemical Physics, 14(23):8307, 2012.
  * [15] Chia-Hsun Chen, Pin-Hao Sher, Cheng-Pin Chen, Wing-Kit Choi, Christopher J. Bardeen, Tien-Lung Chiu, Juen-Kai Wang, and Jiun-Haw Lee.  Thickness-dependent exciton dynamics in thermally evaporated rubrene thin films.  The Journal of Physical Chemistry C, 124(47):25729–25737, November 2020.
  * [16] Michael A. Fusella, Siyu Yang, Kevin Abbasi, Hyun Ho Choi, Zhuozhi Yao, Vitaly Podzorov, Amir Avishai, and Barry P. Rand.  Use of an underlayer for large area crystallization of rubrene thin films.  Chemistry of Materials, 29(16):6666–6673, July 2017.
  * [17] Shohei Horike, Yasuko Koshiba, Masahiro Misaki, and Kenji Ishida.  Crystal growth of rubrene in ionic liquids by vacuum vapor deposition.  Japanese Journal of Applied Physics, 53(5S1):05FT03, April 2014.
  * [18] Yajun Su, Jiangang Liu, Lidong Zheng, Zicheng Ding, and Yanchun Han.  Polymer assisted solution-processing of rubrene spherulites via solvent vapor annealing.  RSC Advances, 2(13):5779, 2012.
  * [19] Hao Chang, Weili Li, Hongkun Tian, Yanhou Geng, Haibo Wang, Donghang Yan, and Tong Wang.  High performance of rubrene thin film transistor by weak epitaxy growth method.  Organic Electronics, 20:43–48, May 2015.
  * [20] Duygu Akin Kara, Edmund K. Burnett, Koray Kara, Ozlem Usluer, Benjamin P. Cherniawski, Edward J. Barron, Burak Gultekin, Mahmut Kus, and Alejandro L. Briseno.  Rubrene single crystal solar cells and the effect of crystallinity on interfacial recombination.  Physical Chemistry Chemical Physics, 24(18):10869–10876, 2022.
  * [21] Y. Luo, M. Brun, P. Rannou, and B. Grevin.  Growth of rubrene thin film, spherulites and nanowires on SiO2.  physica status solidi (a), 204(6):1851–1855, May 2007.
  * [22] Se-W. Park, S. H. Jeong, Jeong-M. Choi, Jung Min Hwang, Jae Hoon Kim, and Seongil Im.  Rubrene polycrystalline transistor channel achieved through in situ vacuum annealing.  Applied Physics Letters, 91(3), July 2007.
  * [23] M Kratzer, A Matkovic, and C Teichert.  Adsorption and epitaxial growth of small organic semiconductors on hexagonal boron nitride.  Journal of Physics D: Applied Physics, 52(38):383001, July 2019.
  * [24] Yasuo Nakayama, Yuki Uragami, Shin’ichi Machida, Kaveenga Rasika Koswattage, Daisuke Yoshimura, Hiroyuki Setoyama, Toshihiro Okajima, Kazuhiko Mase, and Hisao Ishii.  Full picture of valence band structure of rubrene single crystals probed by angle-resolved and excitation-energy-dependent photoelectron spectroscopy.  Applied Physics Express, 5(11):111601, October 2012.
  * [25] Yingxin Ma, Yi Zong, Huiling Yin, Hongtao Lin, Shuhai Chen, and Xue-Dong Wang.  2d optical waveguides based on hierarchical organic semiconductor single crystals.  Advanced Optical Materials, 9(22), September 2021.
  * [26] A. Saeki, S. Seki, T. Takenobu, Y. Iwasa, and S. Tagawa.  Mobility and dynamics of charge carriers in rubrene single crystals studied by flash-photolysis microwave conductivity and optical spectroscopy.  Advanced Materials, 20(5):920–923, March 2008.
  * [27] Hyeok Moo Lee, Hanul Moon, Hyo-Sik Kim, Yong Nam Kim, Sung-Min Choi, Seunghyup Yoo, and Sung Oh Cho.  Abrupt heating-induced high-quality crystalline rubrene thin films for organic thin-film transistors.  Organic Electronics, 12(8):1446–1453, August 2011.
  * [28] Bregt Verreet, Paul Heremans, Andre Stesmans, and Barry P. Rand.  Microcrystalline organic thin-film solar cells.  Advanced Materials, 25(38):5504–5507, August 2013.
  * [29] Jenna A. Tan, Jordan T. Dull, Steven E. Zeltmann, Jakhangirkhodja A. Tulyagankhodjaev, Holly M. Johnson, Alex Liebman-Peláez, Brendan D. Folie, Sven A. Dönges, Omar Khatib, Jonathan G. Raybin, Trevor D. Roberts, Leo M. Hamerlynck, Christian P. N. Tanner, Jina Lee, Colin Ophus, Karen C. Bustillo, Markus B. Raschke, Hendrik Ohldag, Andrew M. Minor, Barry P. Rand, and Naomi S. Ginsberg.  Multimodal characterization of crystal structure and formation in rubrene thin films reveals erasure of orientational discontinuities.  Advanced Functional Materials, 33(13), January 2023.
  * [30] Alexandre L. Foggiatto, Yasuo Takeichi, Kanta Ono, Hiroki Suga, Yoshio Takahashi, Michael A. Fusella, Jordan T. Dull, Barry P. Rand, Kentaro Kutsukake, and Takeaki Sakurai.  Study of local structure at crystalline rubrene grain boundaries via scanning transmission x-ray microscopy.  Organic Electronics, 74:315–320, November 2019.
  * [31] David Necas and Petr Klapetek.  Gwyddion: an open-source software for SPM data analysis.  Central European Journal of Physics, 10:181–188, 2012.
  * [32] Yanping Liu, Xuexiao Yang, Lei Ye, Haibo Ma, and Haiming Zhu.  Molecular stacking controlling coherent and incoherent singlet fission in polymorph rubrene single crystals.  Aggregate, 4(5), April 2023.
  * [33] Liwei Huang, Qing Liao, Qiang Shi, Hongbing Fu, Jinshi Ma, and Jiannian Yao.  Rubrene micro-crystals from solution routes: their crystallography, morphology and optical properties.  J. Mater. Chem., 20(1):159–166, 2010.
  * [34] Thomas R. Fielitz and Russell J. Holmes.  Crystal morphology and growth in annealed rubrene thin films.  Crystal Growth & Design, 16(8):4720–4726, July 2016.
  * [35] Mira El Helou, Olaf Medenbach, and Gregor Witte.  Rubrene microcrystals: A route to investigate surface morphology and bulk anisotropies of organic semiconductors.  Crystal Growth & Design, 10(8):3496–3501, June 2010.
  * [36] Pavel Irkhin, Aleksandr Ryasnyanskiy, Marlus Koehler, and Ivan Biaggio.  Absorption and photoluminescence spectroscopy of rubrene single crystals.  Physical Review B, 86(8), August 2012.
  * [37] Julie Euvrard, Oki Gunawan, Antoine Kahn, and Barry P. Rand.  From amorphous to polycrystalline rubrene: Charge transport in organic semiconductors paralleled with silicon.  Advanced Functional Materials, 32(49), October 2022.
  * [38] M. Kytka, A. Gerlach, F. Schreiber, and J. Kováč.  Real-time observation of oxidation and photo-oxidation of rubrene thin films by spectroscopic ellipsometry.  Applied Physics Letters, 90(13), March 2007.
  * [39] Daniel D. T. Mastrogiovanni, Jeff Mayer, Alan S. Wan, Aleksey Vishnyakov, Alexander V. Neimark, Vitaly Podzorov, Leonard C. Feldman, and Eric Garfunkel.  Oxygen incorporation in rubrene single crystals.  Scientific Reports, 4(1), May 2014.
  * [40] Enrico Fumagalli, Luisa Raimondo, Leonardo Silvestri, Massimo Moret, Adele Sassella, and Marcello Campione.  Oxidation dynamics of epitaxial rubrene ultrathin films.  Chemistry of Materials, 23(13):3246–3253, June 2011.
  * [41] Enrico Werner, Uwe Güth, Bennet Brockhagen, Christoph Döpke, and Andrea Ehrmann.  Examination of polymer blends by AFM phase images.  Technologies, 11(2):56, April 2023.
  * [42] Michael A. Fusella, Frank Schreiber, Kevin Abbasi, Jae Joon Kim, Alejandro L. Briseno, and Barry P. Rand.  Homoepitaxy of crystalline rubrene thin films.  Nano Letters, 17(5):3040–3046, April 2017.
  * [43] Primož Rebernik Ribič and Gvido Bratina.  Ripening of rubrene islands.  The Journal of Physical Chemistry C, 111(50):18558–18562, November 2007.