# Inconsistency-Based Data-Centric Active Open-Set Annotation

Ruiyu Mao, Ouyang Xu, Yunhui Guo

###### Abstract

Active learning, a method to reduce labeling effort for training deep neural
networks, is often limited by the assumption that all unlabeled data belong to
known classes. This closed-world assumption fails in practical scenarios with
unknown classes in the data, leading to active open-set annotation challenges.
Existing methods struggle with this uncertainty. We introduce Neat, a novel,
computationally efficient, data-centric active learning approach for open-set
data. Neat differentiates and labels known classes from a mix of known and
unknown classes, using a clusterability criterion and a consistency measure
that detects inconsistencies between model predictions and feature
distribution. In contrast to recent learning-centric solutions, Neat shows
superior performance in active open-set annotation, as our experiments
confirm. Our implementations for the metrics and datasets are publicly
available at: https://github.com/RuiyuM/Active-OpenSet-NEAT.

## Introduction

The remarkable performance of modern deep neural networks owes much to the
availability of large-scale datasets such as ImageNet (Deng et al. 2009).
However, creating such datasets is a challenging task that requires a
significant amount of effort to annotate data points (Sorokin and Forsyth
2008; Snow et al. 2008; Raykar et al. 2010; Welinder et al. 2010).
Fortunately, active learning offers a solution by enabling us to label only
the most _significant_ samples (Settles 2009; Cohn, Ghahramani, and Jordan
1996; Settles 2011; Beygelzimer, Dasgupta, and Langford 2009; Gal, Islam, and
Ghahramani 2017). In active learning, a small set of labeled samples from
known classes is combined with unlabeled samples, with the goal of selecting
specific samples to label from the pool to enhance model training. Typical
active learning methods for training deep neural networks involve selecting
samples that have high levels of uncertainty (Settles 2009; Cohn, Atlas, and
Ladner 1994; Balcan, Beygelzimer, and Langford 2006), are in close proximity
to the classification boundary (Ducoffe and Precioso 2018), or reveal cluster
structures from the data (Sener and Savarese 2017; Ash et al. 2019).

Despite the considerable body of research on active learning, its practical
implementation in an _open-world_ context remains relatively unexplored (Ning
et al. 2022). Unlike in closed-world active learning settings, where the
unlabeled data pool is assumed to consist only of known classes, the open
world introduces unknown classes into the unlabeled data pool, making active
open-set annotations a challenge. For example, when collecting data to train a
classifier to distinguish between different dog breeds as known classes, a
large number of unlabeled images collected from online sources may include
images from unknown classes such as wolves and coyotes. The task is to
identify images with known classes and select informative samples for
labeling, while avoiding samples from unknown classes. Existing active
learning methods face a significant obstacle in this task, as they may tend to
select samples from unknown classes for labeling due to their high uncertainty
(Ning et al. 2022), which is undesirable.

Figure 1: Dataset consists of color images as known dog class and gray-scale
images as unknown wolf class. Prior work using learning-based approach may
identify some unknown classes as known classes. Our work focusing on local
feature distribution can find known classes more accurately.

Addressing the active open-set annotation problem requires specialized
methods, and one such approach, called LfOSA was recently proposed by Ning et
al. (2022). LfOSA is a learning-based active open-set annotation as it
involves training a detector network with an additional output for unknown
classes. The predictions of the detector network on unlabeled data are used
for identifying known classes. While this approach can achieve impressive
performance, it has two limitations: 1) training the additional detector
network is costly, 2) and it is difficult to identify informative samples from
the known classes as there is a contradiction in that while excluding unknown
classes is necessary, it is also easy to exclude informative samples from the
known classes.

In this paper, we propose a novel inconsistency-based data-centric active
learning method to actively annotate informative samples in an open world,
which not only reduces the computational cost but also improves the
performance of active open set annotation (Figure 1). Rather than using a
learning-based approach to differentiate known and unknown classes, we suggest
a data-centric perspective that naturally separates them by label
clusterability, eliminating the need for an additional detector network. In
addition, our method involves selecting informative samples from known classes
by estimating the inconsistency between the model prediction and the local
feature distribution. For example, suppose the model predicts an unlabeled
sample as a wolf, but the majority of nearby samples are actually dogs. In
that case, we would choose to label this unlabeled sample. The proposed
inconsistency-based active learning approach shares a similar spirit to the
version-space-based approach (Cohn, Atlas, and Ladner 1994; Balcan,
Beygelzimer, and Langford 2006; Dasgupta 2005). However, there are two key
differences. Firstly, our hypothesis class consists of deep neural networks.
Secondly, the version-space-based approach identifies uncertain examples by
analyzing the consistencies among multiple models, our approach leverages a
fixed model and estimates the consistencies between the model prediction and
the local feature distribution.

The contributions of this paper are summarized below,

  * •

We introduce Neat, a novel and efficient inconsistency-based data-centric
active learning method. This method is designed to select informative samples
from known classes within a pool containing both known and unknown classes. To
the best of our knowledge, our work stands as the pioneering effort in
utilizing large language models for active open-set annotation.

  * •

Compared with the learning-based method, the proposed data-centric active
learning method is more computationally efficient and can effectively identify
informative samples from the known classes.

  * •

Extensive experiments show that Neat achieves much better results compared to
standard active learning methods and the method specifically designed for
active open set annotation. In particular, Neat achieves an average accuracy
improvement of 9% on CIFAR10, CIFAR100 and Tiny-ImageNet compared with
existing active open-set annotation method given the same labeling budget.

## Related Work

Active learning, a topic which is extensively studied in machine learning
(Lewis and Gale 2001; Settles 2009; Cohn, Ghahramani, and Jordan 1996; Settles
2011; Beygelzimer, Dasgupta, and Langford 2009), focuses on selecting the most
uncertain samples for labeling. Uncertainty-based active learning methods
(Lewis and Gale 2001) employ measures such as entropy (Shannon 1948), least
confident (Settles 2009), and margin sampling (Scheffer, Decomain, and Wrobel
2001). Another widely used approach is the version-space-based method (Cohn,
Atlas, and Ladner 1994; Balcan, Beygelzimer, and Langford 2006; Dasgupta 2005,
2011) which maintains multiple models consistent with labeled samples. If
these models make inconsistent predictions on an unlabeled sample, a label
query is prompted.

For deep neural networks, specific active learning methods have been
developed. For instance, Core-Set (Sener and Savarese 2017) advocates for an
active learning method that selects samples which are representative of the
whole data distribution. Badge (Ash et al. 2019) selects samples based on
predictive uncertainty and sample diversity. In particular, Badge leverages
$k$-Means++ to select a set of samples which have diverse gradient magnitudes.
The approach BGADL (Tran et al. 2019) integrates active learning and data
augmentation. It leverages Bayesian inference in the generative model to
create new training samples from selected existing samples. These generated
samples are then utilized to enhance the classification accuracy of deep
neural networks. CEAL (Wang et al. 2016) combines pseudo-labeling of clearly
classified samples and actively labeled informative samples to train deep
neural networks. For deep object detection, active learning is introduced in
(Brust, Käding, and Denzler 2018), utilizing prediction margin to identify
valuable instances for labeling. DFAL (Ducoffe and Precioso 2018) is an
adversarial active learning method for deep neural networks that selects
samples based on their distance to the decision boundary, approximated using
adversarial examples. The samples closest to the boundary are chosen for
classifier training. Recently, LfOSA (Ning et al. 2022) is proposed as the
first active learning method for active open-set annotation. LfOSA trains a
detector network to identify known classes and selects samples based on model
confidence. MQNet (Park et al. 2022) is an open-set active learning method
that addresses the Purity-Informativeness Dilemma and dynamically solves it
using a Meta-learning MLP (Multilayer Perceptron) network, based on the
samples’ purity and informativeness scores.

## Preliminary

### Active Open-Set Annotation

Assume the sample-label pair $(X,Y)$ has the joint distribution $P_{XY}$. For
a given sample $\mathbf{x}$, the label $y$ can be determined via the
conditional expectation,

$\eta(\mathbf{x})=\mathbb{E}[Y|X=\mathbf{x}]$ (1)

We consider a pool-based active learning setup. We begin by randomly sampling
a small labeled set denoted as $L=\\{\mathbf{x}_{i},y_{i}\\}_{i=1}^{N}$ from
the joint distribution $P_{XY}$. We also have access to a large set of
unlabeled samples, denoted as $U$. Given a deep neural network $f_{\theta}$
parameterized by $\theta$, the expected risk of the network is computed as
$R=\mathbb{E}[\ell(y,f_{\theta}(\mathbf{x}))]=\int
l(y,f_{\theta}(\mathbf{x}))dP_{XY}$, where $\ell(y,f_{\theta}(\mathbf{x}))$
represents the classification error. We conduct $T$ query rounds, where in
each round $t$, we are allotted a fixed labeling budget $B$ to identify $B$
informative samples from $U$, denoted as $U_{B}$. The queried samples $U_{B}$
are given labels and added to the initial labeled set, with the aim of
minimizing the sum of cross-entropy losses $\sum_{(\mathbf{x},y)\in L\bigcup
U_{B}}\ell_{\textnormal{CE}}(f_{\theta}(\textbf{x}),y)$ and reducing the
expected risk of the model $f_{\theta}$ (Settles 2009). The term
$\ell_{\textnormal{CE}}(f_{\theta}(\mathbf{x}),y)$ represents the cross-
entropy loss between the predicted and true labels for a sample
$(\mathbf{x},y)$ (Bishop 2006).

In the context of active open-set annotation (Ning et al. 2022), the set of
unlabeled samples $U$ comprises data from both known classes and unknown
classes, denoted as $S_{known}$ and $S_{unknown}$, respectively. Importantly,
$S_{known}$ represents the classes observed during the initial labeling
process, and it is ensured that $S_{known}\cap S_{unknown}=\emptyset$. In each
active query round, we label the samples from known classes with their true
labels and label the samples from unknown classes as _invalid_. Active open-
set annotation presents a greater challenge than standard active learning, as
it requires the active learning method 1) to distinguish between known and
unknown classes among the unlabeled samples 2) and to select informative
samples exclusively from the known classes.

### Learning-Based Active Open-Set Annotation

The study of active open-set annotation is currently limited. A recent paper
(Ning et al. 2022) proposed a new learning-based method called LfOSA which is
specially designed for active open-set annotation. This approach combines a
model, $f_{\theta}$, trained on known classes with a detector network that
identifies unknown class samples. Suppose there are $C$ known classes, the
detector network has $C+1$ outputs, similar to OpenMax(Bendale and Boult
2016), which allow for the classification of unlabeled data into known and
unknown classes. During an active query, LfOSA focuses only on unlabeled data
classified as known classes, and then uses a Gaussian Mixture Model to cluster
the activation values of detected known classes data into two clusters. The
data closer to the larger cluster mean is selected for labeling. This
learning-based approach has shown significant improvement over traditional
active learning methods for active open-set annotation. However, there are
still challenges that need to be addressed, such as the additional computation
cost required to train the detector network and the difficulty of identifying
informative samples from the known classes.

Algorithm 1 Neat: Inconsistency-Based Data-Centric Active Open-Set annotation.

1:A deep neural network $f_{\theta}$, initial labeled set $L$, a set of known
class $Y_{\textnormal{Known}}$ , unlabeled labeled set $U$, number of query
rounds $T$, number of examples in each query batch $B$, a pre-trained model
$M$, number of neighbors $K$.

2:Use the pre-trained model $M$ for extracting features on $L$ and $U$.

3:for $t\leftarrow 1$ to $T$ do

4: Train the model $f_{\theta}$ on $L$ by minimizing $\sum_{(x,y)\in
L}\ell_{\textnormal{CE}}(f_{\theta}(x),y)$

5: $S\leftarrow\\{\\}$.

6: For each sample $x\in U$:

7: Compute the output of the softmax function as $P_{x}$.

8: Find the $K$-nearest neighbors $\\{N_{i}(\mathbf{x})\\}_{i=1}^{K}$ of
$\mathbf{x}$ in $L$ based on the extracted features.

9: If all the labels from $\\{N_{i}(\mathbf{x})\\}_{i=1}^{K}$ belong to known
classes $Y_{\textnormal{Known}}$, then $S\leftarrow S\bigcup\\{\mathbf{x}\\}$.

10: Compute the score $I(x)$ using Eq. 5 for each sample $\mathbf{x}\in S$.

11: Rank the samples based on $I(\mathbf{x})$ and denote the B samples which
have the largest scores as $U_{B}$.

12: Query the labels of each sample in $U_{B}$.

13: $U\leftarrow U\setminus U_{B}$, $L\leftarrow L\bigcup U_{B}$.

14:end for

## Neat

Algorithm 1 describes the detailed algorithm of Neat. Initially, Neat begins
with a randomly drawn labeled set $L$ from known classes. In each query round,
Neat performs two main steps. First, it identifies unlabeled samples whose
neighbors in the labeled set belong to known classes. Secondly, Neat selects a
batch of unlabeled samples for labeling by estimating the inconsistency
between the model’s prediction and the local feature distribution. Unlike the
learning-based method discussed in (Ning et al. 2022), Neat is more
computationally efficient. Furthermore, it effectively decouples the detection
of known classes from the identification of informative samples, enabling it
to identify informative samples even from the known classes, which is a
challenge for existing active learning methods.

### Data-Centric Known Class Detection

In the learning-based method (Ning et al. 2022), an additional detector
network is trained to differentiate known and unknown classes. However, the
training cost is high. In contrast, Neat adopts a data-centric perspective for
finding known classes samples based on feature similarity. In particular, Neat
relies on label clusterability to identify known classes from the unlabeled
pool.

Label clusterability. We leverage the intuition that samples with similar
features should belong to the same class (Gao, Yang, and Zhou 2016; Zhu, Song,
and Liu 2021; Zhu, Dong, and Liu 2022). This intuition can be formally defined
as follows,

######  Definition 0.1.

(($K$, $\sigma_{K}$) label clusterability). A dataset $D$ satisfies
$(K,\sigma_{K})$ label clusterability if for all $\mathbf{x}\in D$, the sample
$\mathbf{x}$ and its $K$-Nearest-Neighbors ($K$-NN)
$\\{\mathbf{x}_{1},\mathbf{x}_{2},...,\mathbf{x}_{K}\\}$ belong to the same
label with probability at least $1-\sigma_{K}$.

When $\sigma_{K}$ = 0, then it is called $K$-NN label clusterability (Zhu,
Song, and Liu 2021). Recent methods for noisy label learning (Zhu, Song, and
Liu 2021; Zhu, Dong, and Liu 2022) leverage label clusterability for detecting
examples with noisy labels.

Feature extraction. To utilize the clusterability of labels for identifying
known classes, we need to extract features from unlabeled inputs that group
semantically similar samples in the feature space. Additionally, the quality
of these features will inevitably impact the clusterability of the labels.
Instead of developing a separate detector, as demonstrated in Ning et al.
(2022), we suggest leveraging pre-trained large language models for feature
extraction, which have been demonstrated to possess exceptional zero-shot
learning ability (Radford et al. 2021). In particular, we leverage CLIP
(Radford et al. 2021) to extract features for both the labeled and unlabeled
data, providing high-quality features for calculating feature similarity.

Known class detection.  By utilizing the features extracted by CLIP from both
the labeled set $L$ and the unlabeled set $U$, we can identify the $K$-nearest
neighbors $\\{N_{1}(\mathbf{x}),N_{2}(\mathbf{x}),...,N_{K}(\mathbf{x})\\}$ in
the labeled set $L$ for each sample $\mathbf{x}\in U$, using cosine distance.
Each $N_{k}(\mathbf{x})\in L$ represents the $k$-th closest samples in $L$ to
the unlabeled sample $\mathbf{x}$. Subsequently, we compute the count of
neighbors with known and unknown classes for each unlabeled sample
$\mathbf{x}$, assuming label clusterability. If an unlabeled sample is in
proximity to numerous known class samples, it is more likely to belong to a
known class. Therefore, we classify an unlabeled sample as belonging to a
known class if all of its neighboring samples are from known classes.

Theoretical analysis. Here we give an upper bound on the detection error of
the proposed known class detection. The first step is to understand in what
condition our method will make a mistake. Given the sample $\mathbf{x}$,
although all of its $K$-nearest neighbors
$\\{N_{1}(\mathbf{x}),N_{2}(\mathbf{x}),...,N_{K}(\mathbf{x})\\}$ belong to
known classes, it is still possible that the sample is from the unknown
classes. This can caused by the randomness in the labeling process and the
quality of the features. Instead of using the definition 0.1 which is too
coarse for our analysis, we introduce the following assumptions,

######  Assumption 0.1.

There exists a constant $C>0$ and $0<\alpha<1$ such that for any $\mathbf{x}$
and $\mathbf{x}^{\prime}$,

$\mathbb{P}(y_{true}(\mathbf{x})\neq y_{true}(\mathbf{x}^{\prime}))\leq
C\rho(\mathbf{x},\mathbf{x}^{\prime})^{\alpha}$ (2)

Here, $\rho(\mathbf{x},\mathbf{x}^{\prime})$ represents the distance between
samples $\mathbf{x}$ and $\mathbf{x}^{\prime}$, while $y_{true}(\mathbf{x})$
and $y_{true}(\mathbf{x}^{\prime})$ correspond to the true labels of
$\mathbf{x}$ and $\mathbf{x}^{\prime}$, respectively. We use
$r_{K}(\mathbf{x})$ to denote the radius of the ball centered at $\mathbf{x}$
such that $\forall$ $\mathbf{x}^{\prime}$ in the $K$-nearest neighbors of
$\mathbf{x}$, $\rho(\mathbf{x},\mathbf{x}^{\prime})\leq r_{K}(\mathbf{x})$.

######  Assumption 0.2.

For any $\mathbf{x}$, the labeling error is upper-bounded by a small constant
$e$,

$\mathbb{P}(y(\mathbf{x})\neq y_{true}(\mathbf{x}))\leq e$ (3)

Then, we can establish an upper bound for the detection error as follows,

######  Theorem 0.1.

Given the assumption 0.1. and 0.2. and the number of neighbors $K$, the
probability of making a detection error is upper-bounded as,

$\displaystyle\begin{split}\mathbb{P}(\textnormal{Error}|&K)\leq\sum_{k=\lceil\frac{K-1}{2}\rceil+1}^{K}\binom{K}{k}e^{k}(1-e)^{K-k}C^{k}r_{K}(\mathbf{x})^{\alpha
k}\\\
&+\sum_{k=0}^{\lfloor\frac{K+1}{2}\rfloor-1}\binom{K}{k}e^{k}(1-e)^{K-k}C^{(K-k)}r_{K}(\mathbf{x})^{\alpha(K-k)}\end{split}$
(4)

The proof of the theorem will be given in the Appendix. From the theorem, we
can conclude that a good feature will decrease the detection error as we
expected.

### Inconsistency-Based Active Learning

To improve accuracy with a fixed labeling budget, it is crucial to select
informative samples for labeling. One active learning strategy, motivated by
theory, is the version-space-based approach (Cohn, Atlas, and Ladner 1994;
Balcan, Beygelzimer, and Langford 2006; Dasgupta 2005, 2011). This approach
involves maintaining a set of models that are consistent with the current
labeled data, and an unlabeled sample is selected for labeling if two models
produce different predictions. However, implementing this approach for deep
neural networks is challenging due to the computational cost of training
multiple models (Ash et al. 2019). To address this issue, we propose an
inconsistency-based active learning method that does not require training
multiple models and naturally leverages features produced by CLIP.

Given an unlabeled sample $\mathbf{x}\in U$, the model $f_{\theta}$’s
prediction for $\mathbf{x}$, denoted as $P_{\mathbf{x}}\in\mathbb{R}^{C}$,
represents a probability vector where $P_{\mathbf{x}}[c]$ is the model’s
confidence that $\mathbf{x}$ belongs to class $c$. Since we lack ground-truth
labels, measuring prediction accuracy is impossible. To address this, we
propose evaluating the importance of the sample for improving model training
by assessing whether the model prediction aligns with local feature
similarity. For example, if the model predicts an unlabeled sample as a dog
with low probability but the majority of the sample’s neighbors belong to the
dog class, then either the model’s prediction is incorrect or the sample is
near the decision boundary between the dog class and the true class. In either
case, the sample can be labeled to improve model training.

Given the $K$-nearest neighbors $\\{N_{i}(\mathbf{x})\\}_{i=1}^{K}$ of the
example $\mathbf{x}$, we first construct a vector
$V_{\mathbf{x}}\in\mathbb{R}^{C}$ with
$V_{\mathbf{x}}[c]=\sum_{k}\mathbf{1}(Y_{k}(\mathbf{x})=c)$, where
$Y_{k}(\mathbf{x})$ is the label of $k$-th nearest neighbor of $\mathbf{x}$
and $\mathbf{1}(\cdot)$ is an indicator function. Then the vector
$V_{\mathbf{x}}$ is normalized via the softmax function to be a probabilistic
vector $\tilde{V}_{\mathbf{x}}$. The inconsistency between the model
prediction and local feature similarity is computed using cross-entropy as,

$I(\mathbf{x})=-\sum_{c=1}^{C}P_{\mathbf{x}}[c]\log\tilde{V}_{\mathbf{x}}[c].$
(5)

A large $I(\mathbf{x})$ indicates that the model prediction is inconsistent
with local feature distribution. Similar to version-space-based approach, the
unlabeled sample is selected for labeling. In each query round, we rank all
the identified known classes samples in the first stage using $I(\mathbf{x})$
and select the top $B$ samples for labeling.

## Experiments

### Experimental Settings and Evaluation Protocol

Datasets and models. We consider CIFAR10 (Krizhevsky and Hinton 2009),
CIFAR100 (Krizhevsky and Hinton 2009), and Tiny-Imagenet (Le and Yang 2015),
to evaluate the performance of our proposed method. Similar to the existing
methods for active open-set annotations (Ning et al. 2022), we leverage a
ResNet-18 (He et al. 2016) architecture to train the classifier for the known
classes. For the proposed method, we leverage CLIP (Radford et al. 2021) to
extract the features for both known and unknown classes.

Active open-set annotation. In accordance with (Ning et al. 2022), the
experiment randomly selects 40 classes, 20 classes and 2 classes from Tiny-
Imagenet, CIFAR100, and CIFAR10, respectively, as known classes, while the
remaining classes are treated as unknown classes. To begin with, following
(Ning et al. 2022), 8% of the data from the known classes in Tiny-ImageNet and
CIFAR100, and 1% of the data from the known classes in CIFAR10 are randomly
selected to form an initial labeled set. The rest of the known class data and
the unknown class data are combined to form the unlabeled data pool.

Baseline methods. We consider the following active learning methods as
baselines,

  1. 1.

Random: The naive baseline which randomly selects samples for annotation;

  2. 2.

Uncertainty (Settles 2009): A commonly used active learning method which
selects samples with the highest degree of uncertainty as measured by entropy;

  3. 3.

Certainty (Settles 2009): A common baseline for active learning which selects
samples with the highest degree of certainty as measured by entropy;

  4. 4.

Coreset (Sener and Savarese 2017): This method identifies a compact,
representative subset of training data for annotation;

  5. 5.

Bgadl (Tran et al. 2019): A Bayesian active learning method which leverages
generative to select informative samples;

  6. 6.

OpenMax (Bendale and Boult 2016): A representative open-set classification
method which can differentiate between known classes and unknown classes;

  7. 7.

Badge (Ash et al. 2019): An active learning method designed for deep neural
networks which select a batch of samples with diverse gradient magnitudes;

  8. 8.

MQNet (Park et al. 2022): An open-set active learning method employs a Meta-
learning MLP network to dynamically select samples based on their purity and
informativeness scores.

  9. 9.

LfOSA (Ning et al. 2022): A learning-based active open-set annotation method
which selects samples based on maximum activation value (MAV) modeled by a
Gaussian Mixture Model;

  10. 10.

Neat (Passive): We also consider a baseline that is the passive version of
Neat. In Neat (Passive), we do not leverage the proposed inconsistency-based
active learning methods for selecting samples for labeling but instead
randomly sample from the identified known classes samples.

|  |
---|---|---
|  |
|  |
Figure 2: Neat achieves higher precision, recall and accuracy compared with
existing active learning methods for active open-set annotation. We evaluated
Neat and the baseline active learning methods on CIFAR10, CIFAR100 and Tiny-
ImageNet based on accuracy, precision and recall.

Metrics. We evaluate various active learning methods based on three key
metrics: accuracy, precision, and recall. Accuracy reflects how accurately the
model makes predictions on the test set. To measure recall, we use the ratio
of selected known class samples to the total number of known class samples
present in the unlabeled pool, denoted as $N^{t}_{\textnormal{known}}$ and
$N^{\textnormal{total}}_{\textnormal{known}}$, respectively, in the query
round $t$. Precision, on the other hand, measures the proportion of true known
class samples among the selected samples in each query round,

$\textnormal{precision}=\frac{N^{t}_{\textnormal{known}}}{B}\quad\quad\textnormal{recall}=\frac{\sum_{t=1}^{T}N^{t}_{\textnormal{known}}}{N^{\textnormal{total}}_{\textnormal{known}}}$
(6)

Implementations details. The classification model, specifically ResNet-18, was
trained for 100 epochs using Stochastic Gradient Descent (SGD) with an initial
learning rate of 0.01. The learning rate decayed by 0.5 every 20 epochs, and
the training batch size is 128. There were 9 query rounds with a query batch
size of 400. All experiments were repeated three times with different random
seeds and the average results and standard deviations were reported. The
experiments were conducted on four A5000 NVIDIA GPUs.

### Results

####  Neat VS. Baselines

We evaluate the performance of Neat and other methods by plotting curves as
the number of queries increases (Figure 5). It is evident that regardless of
the datasets, Neat consistently surpasses other methods in all cases. In
particular, Neat achieves much higher selection recall and precision compared
to existing active learning methods which demonstrates the effectiveness of
the data-centric known class detection. 1) In terms of recall, Neat
consistently outperforms other active learning methods by a significant
margin. Notably, on CIFAR10, CIFAR100, and Tiny-ImageNet, Neat achieves
improvements of 4%, 12%, and 7%, respectively, compared to LfOSA (Ning et al.
2022), which is a learning-based active open-set annotation method. 2) In
terms of precision, Neat consistently maintains a higher selection precision
than other baselines, with a noticeable gap. Importantly, Neat’s ability to
differentiate between known and unknown classes is significantly improved by
adding examples from unknown classes in the first query round. 3) In terms of
accuracy, Neat consistently outperforms LfOSA (Ning et al. 2022) across all
datasets, including CIFAR10, CIFAR100, and Tiny-ImageNet, with improvements of
6%, 11%, and 10%, respectively. These results indicate that the proposed Neat
method effectively addresses the open-set annotation (OSA) problem. It is
worth noting that Neat (Passive) achieves slightly higher recall and precision
than Neat. The possible reason is that by selecting informative samples from
known classes, it is also more likely to include samples from unknown classes
with high uncertainty. Despite having a slightly lower recall, Neat achieves a
higher accuracy in all data sets compared to Neat (Passive). This demonstrates
that Neat can effectively identify informative samples from known classes to
train the model.

#### The Use of Pre-Trained Models for Active Open-Set Annotation

Unlike the majority of existing active learning methods, Neat incorporates a
pre-trained model as an component of the method. Although there was some
initial works that utilized large language models for natural language
processing (NLP) tasks (Bai et al. 2020; Seo et al. 2022), to our knowledge,
our work represents the first instance of leveraging a large language model
for active open-set annotation.

The utilization of a pre-trained model inevitably introduces prior knowledge.
Although this pre-trained model has not been directly trained on the target
dataset, it is essential to examine its potential impact on other active
learning methods. To this end, we conducted experiments involving the pre-
trained model module across various baseline methods. In particular, we adopt
a hybrid approach that combines data-centric known class detection to
initially identify potential known classes, followed by the application of
different active learning methods using the data from these detected known
classes. The empirical results indeed show the influence of the pre-trained
model on classification accuracy. However, our proposed Neat method still
surpasses other active learning techniques in terms of accuracy on the Tiny-
ImageNet, as shown in Figure 3.

|  |
---|---|---

Figure 3: Neat is effective compared with other active learning methods for deep neural networks.  Dataset | Model | Accuracy (avg ± std)
---|---|---
Tiny-Imagenet | ResNet-50 | $50.53\pm 1.15$
ResNet-34 | $50.17\pm 0.20$
ResNet-18 | $48.98\pm 0.42$
CLIP | $49.01\pm 1.60$
CIFAR100 | ResNet-50 | $62.00\pm 1.26$
ResNet-34 | $62.73\pm 0.45$
ResNet-18 | $60.75\pm 0.58$
CLIP | $63.68\pm 0.67$
CIFAR10 | ResNet-50 | $97.53\pm 0.24$
ResNet-34 | $97.48\pm 0.19$
ResNet-18 | $97.48\pm 0.26$
CLIP | $98.15\pm 0.14$
Table 1: Neat is robust to the choices of pre-trained models. Dataset | # of Neighbors | Accuracy (avg ± std)
---|---|---
Tiny-Imagenet | 5 | $47.23\pm 1.21$
10 | $49.01\pm 1.60$
15 | $49.02\pm 0.61$
20 | $50.73\pm 0.29$
CIFAR100 | 5 | $62.63\pm 0.66$
10 | $63.68\pm 0.67$
15 | $62.77\pm 0.84$
20 | $60.85\pm 1.70$
CIFAR10 | 5 | $97.95\pm 0.27$
10 | $98.15\pm 0.14$
15 | $97.75\pm 0.14$
20 | $97.85\pm 0.21$
Table 2: The impact of number of neighbors on the final performance of Neat.

#### Computational Efficiency

Our approach is more computationally efficient compared to learning-based
open-set active learning methods, such as LfOSA. This efficiency is attributed
to the fact that learning-based methods like LfOSA require the additional
training of a detection network to discern known and unknown classes.
Consequently, the training time for our method on the Tiny-ImageNet is 88
minutes on an NVIDIA A5000 GPU, while the LfOSA method requires 156 minutes.
For the CIFAR100, our method takes 21 minutes, compared to LFOSA’s 59 minutes,
and for the CIFAR10, our method requires 17 minutes, while LFOSA needs 25
minutes. Overall, we can observe that Neat is much faster than LFOSA.

### Ablation Studies

Impact of feature quality. We investigate the impact of different pre-trained
models for feature extraction. The quality of the features is a crucial factor
that may affect the label clusterability of the dataset. Specifically, we
consider pre-trained ResNet-18, ResNet-34, and ResNet-50 on ImageNet, in
addition to CLIP. Figure 5 demonstrates that Neat achieves significantly
better accuracy than the baseline active learning methods, regardless of the
pre-trained models used. This highlights the robustness of Neat in detecting
known classes. It’s worth noting that while CLIP features achieve better
accuracy on CIFAR-10 and CIFAR-100 than ResNets, the accuracy on Tiny-ImageNet
is lower. This could be because the ResNets are pre-trained on ImageNet, and
their features are better suited for Tiny-ImageNet. However, large language
models like CLIP can generally provide high-quality features that are useful
across different datasets.

Influence of number of neighbors. We further investigate how the number of
neighbors impacts the detection of data-centric known classes. We consider
different values of $K$ (5, 10, 15, 20) and present the results in Table 2. We
observe that although a smaller value of $K$ (e.g., $K=5$) leads to slightly
worse performance, other choices of $K$ yield similar results. This suggests
that a smaller $K$ only captures the local feature distribution of the target
sample, which not provide a good characterization of the underlying feature
space. Ablation studies of query batch size and the impact of different
classification models are included in the Appendix.

Figure 4: Neat can accurately identify known classes from the unlabeled pool.

Visualization. We used t-SNE (Van der Maaten and Hinton 2008) to visualize the
features produced by CLIP on CIFAR-10, focusing on the samples selected by
Neat and LfOSA (Ning et al. 2022). We randomly select a query round for
plotting. Our results show LfOSA selects many samples from unknown classes,
explaining its low precision. Alternatively, nearly all samples Neat selects
are from known classes, demonstrating its effectiveness in the active open-set
annotation problem.

## Conclusion

In this paper, we propose a solution to the practical challenge of maintaining
high recall when identifying examples of known classes for target model
training from a massive unlabeled open-set. To address this challenge, we
introduce a data-centric active learning method called Neat, which utilizes
existing large language models to identify known classes in the unlabeled
pool. Our proposed method offers several advantages over traditional active
learning methods. Specifically, Neat uses the CLIP model to extract features,
which eliminates the need to train a separate detector. Additionally, our
approach achieves improved results with low query numbers, resulting in cost
savings on labeling.

## Acknowledgments

This work was supported by a startup funding by the University of Texas at
Dallas.

## References

  * Ash et al. (2019) Ash, J. T.; Zhang, C.; Krishnamurthy, A.; Langford, J.; and Agarwal, A. 2019.  Deep batch active learning by diverse, uncertain gradient lower bounds.  _arXiv preprint arXiv:1906.03671_.
  * Bai et al. (2020) Bai, G.; He, S.; Liu, K.; Zhao, J.; and Nie, Z. 2020.  Pre-trained language model based active learning for sentence matching.  _arXiv preprint arXiv:2010.05522_.
  * Balcan, Beygelzimer, and Langford (2006) Balcan, M.-F.; Beygelzimer, A.; and Langford, J. 2006.  Agnostic active learning.  In _Proceedings of the 23rd international conference on Machine learning_ , 65–72.
  * Bendale and Boult (2016) Bendale, A.; and Boult, T. E. 2016.  Towards open set deep networks.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 1563–1572.
  * Beygelzimer, Dasgupta, and Langford (2009) Beygelzimer, A.; Dasgupta, S.; and Langford, J. 2009.  Importance weighted active learning.  In _Proceedings of the 26th annual international conference on machine learning_ , 49–56.
  * Bishop (2006) Bishop, C. M. 2006.  _Pattern Recognition and Machine Learning_ , volume 4 of _Information science and statistics_.  Springer.  ISBN 9780387310732.
  * Brust, Käding, and Denzler (2018) Brust, C.-A.; Käding, C.; and Denzler, J. 2018.  Active learning for deep object detection.  _arXiv preprint arXiv:1809.09875_.
  * Cohn, Atlas, and Ladner (1994) Cohn, D.; Atlas, L.; and Ladner, R. 1994.  Improving generalization with active learning.  _Machine learning_ , 15: 201–221.
  * Cohn, Ghahramani, and Jordan (1996) Cohn, D. A.; Ghahramani, Z.; and Jordan, M. I. 1996.  Active learning with statistical models.  _Journal of artificial intelligence research_ , 4: 129–145.
  * Dasgupta (2005) Dasgupta, S. 2005.  Coarse sample complexity bounds for active learning.  _Advances in neural information processing systems_ , 18.
  * Dasgupta (2011) Dasgupta, S. 2011.  Two faces of active learning.  _Theoretical computer science_ , 412(19): 1767–1781.
  * Deng et al. (2009) Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , 248–255. Ieee.
  * Ducoffe and Precioso (2018) Ducoffe, M.; and Precioso, F. 2018.  Adversarial active learning for deep networks: a margin based approach.  _arXiv preprint arXiv:1802.09841_.
  * Gal, Islam, and Ghahramani (2017) Gal, Y.; Islam, R.; and Ghahramani, Z. 2017.  Deep bayesian active learning with image data.  In _International conference on machine learning_ , 1183–1192. PMLR.
  * Gao, Yang, and Zhou (2016) Gao, W.; Yang, B.-B.; and Zhou, Z.-H. 2016.  On the resistance of nearest neighbor to random noisy labels.  _arXiv preprint arXiv:1607.07526_.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * Krizhevsky and Hinton (2009) Krizhevsky, A.; and Hinton, G. 2009.  Learning multiple layers of features from tiny images.  Technical Report 0, University of Toronto, Toronto, Ontario.
  * Le and Yang (2015) Le, Y.; and Yang, X. 2015.  Tiny imagenet visual recognition challenge.  _CS 231N_ , 7(7): 3.
  * Lewis and Gale (2001) Lewis, D.; and Gale, W. 2001.  A Sequential Algorithm for Training Text Classifiers.  _Proceedings of the 17th annual international ACM SIGIR conference on Research and development in Information Retrieval_ , 29.
  * Ning et al. (2022) Ning, K.-P.; Zhao, X.; Li, Y.; and Huang, S.-J. 2022.  Active learning for open-set annotation.  In _CVPR_ , 41–49.
  * Park et al. (2022) Park, D.; Shin, Y.; Bang, J.; Lee, Y.; Song, H.; and Lee, J.-G. 2022.  Meta-Query-Net: Resolving Purity-Informativeness Dilemma in Open-set Active Learning.  In _Proceedings of the 35th Neural Information Processing Systems (NeurIPS) Conference_.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , 8748–8763. PMLR.
  * Raykar et al. (2010) Raykar, V. C.; Yu, S.; Zhao, L. H.; Valadez, G. H.; Florin, C.; Bogoni, L.; and Moy, L. 2010.  Learning from crowds.  _Journal of machine learning research_ , 11(4).
  * Scheffer, Decomain, and Wrobel (2001) Scheffer, T.; Decomain, C.; and Wrobel, S. 2001.  Active hidden markov models for information extraction.  In _Advances in Intelligent Data Analysis: 4th International Conference, IDA 2001 Cascais, Portugal, September 13–15, 2001 Proceedings 4_ , 309–318. Springer.
  * Sener and Savarese (2017) Sener, O.; and Savarese, S. 2017.  Active learning for convolutional neural networks: A core-set approach.  _arXiv preprint arXiv:1708.00489_.
  * Seo et al. (2022) Seo, S.; Kim, D.; Ahn, Y.; and Lee, K.-H. 2022.  Active learning on pre-trained language model with task-independent triplet loss.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 36, 11276–11284.
  * Settles (2009) Settles, B. 2009.  Active Learning Literature Survey.  Computer Sciences Technical Report 1648, University of Wisconsin–Madison.
  * Settles (2011) Settles, B. 2011.  From theories to queries: Active learning in practice.  In _Active learning and experimental design workshop in conjunction with AISTATS 2010_ , 1–18. JMLR Workshop and Conference Proceedings.
  * Shannon (1948) Shannon, C. E. 1948.  A mathematical theory of communication.  _The Bell system technical journal_ , 27(3): 379–423.
  * Snow et al. (2008) Snow, R.; O’connor, B.; Jurafsky, D.; and Ng, A. Y. 2008.  Cheap and fast–but is it good? evaluating non-expert annotations for natural language tasks.  In _Proceedings of the 2008 conference on empirical methods in natural language processing_ , 254–263.
  * Sorokin and Forsyth (2008) Sorokin, A.; and Forsyth, D. 2008.  Utility data annotation with amazon mechanical turk.  In _2008 IEEE computer society conference on computer vision and pattern recognition workshops_ , 1–8. IEEE.
  * Tran et al. (2019) Tran, T.; Do, T.-T.; Reid, I.; and Carneiro, G. 2019.  Bayesian generative active deep learning.  In _International Conference on Machine Learning_ , 6295–6304. PMLR.
  * Van der Maaten and Hinton (2008) Van der Maaten, L.; and Hinton, G. 2008.  Visualizing data using t-SNE.  _Journal of machine learning research_ , 9(11).
  * Wang et al. (2016) Wang, K.; Zhang, D.; Li, Y.; Zhang, R.; and Lin, L. 2016.  Cost-effective active learning for deep image classification.  _IEEE Transactions on Circuits and Systems for Video Technology_ , 27(12): 2591–2600.
  * Welinder et al. (2010) Welinder, P.; Branson, S.; Perona, P.; and Belongie, S. 2010.  The multidimensional wisdom of crowds.  _Advances in neural information processing systems_ , 23.
  * Zhu, Dong, and Liu (2022) Zhu, Z.; Dong, Z.; and Liu, Y. 2022.  Detecting corrupted labels without training a model to predict.  In _International Conference on Machine Learning_ , 27412–27427. PMLR.
  * Zhu, Song, and Liu (2021) Zhu, Z.; Song, Y.; and Liu, Y. 2021.  Clusterability as an alternative to anchor points when learning with noisy labels.  In _International Conference on Machine Learning_ , 12912–12923. PMLR.

## Theoretical analysis

We give an upper bound on the detection error of the proposed known class
detection. The first step is to understand in what condition our method will
make a mistake. Given the sample $\mathbf{x}$, although all of its $K$-nearest
neighbors $\\{N_{1}(\mathbf{x}),N_{2}(\mathbf{x}),...,N_{K}(\mathbf{x})\\}$
belong to known classes, it is still possible that the sample is from the
unknown classes. This can caused by the randomness in the labeling process and
the quality of the features.

To capture the randomness caused by the quality of the features, we introduce
the following assumption,

######  Assumption 0.3.

There exists a constant $C>0$ and $0<\alpha<1$ such that for any $\mathbf{x}$
and $\mathbf{x}^{\prime}$,

$\mathbb{P}(y_{true}(\mathbf{x})\neq y_{true}(\mathbf{x}^{\prime}))\leq
C\rho(\mathbf{x},\mathbf{x}^{\prime})^{\alpha}$ (7)

Here, $\rho(\mathbf{x},\mathbf{x}^{\prime})$ represents the distance between
samples $\mathbf{x}$ and $\mathbf{x}^{\prime}$, while $y_{true}(\mathbf{x})$
and $y_{true}(\mathbf{x}^{\prime})$ correspond to the true labels of
$\mathbf{x}$ and $\mathbf{x}^{\prime}$, respectively. We use
$r_{K}(\mathbf{x})$ to denote the radius of the ball centered at $\mathbf{x}$
such that $\forall$ $\mathbf{x}^{\prime}$ in the $K$-nearest neighbors of
$\mathbf{x}$, $\rho(\mathbf{x},\mathbf{x}^{\prime})\leq r_{K}(\mathbf{x})$.

To capture the randomness caused by the labeling process, we introduce the
following assumption,

######  Assumption 0.4.

For any $\mathbf{x}$, the labeling error is upper-bounded by a small constant
$e$,

$\mathbb{P}(y(\mathbf{x})\neq y_{true}(\mathbf{x}))\leq e$ (8)

Then, we can establish an upper bound for the detection error as follows,

######  Theorem 0.2.

Given the assumption 0.1. and 0.2. and the number of neighbors $K$, the
probability of making a detection error is upper-bounded as,

$\displaystyle\begin{split}\mathbb{P}(\textnormal{Error}|&K)\leq\sum_{k=\lceil\frac{K-1}{2}\rceil+1}^{K}\binom{K}{k}e^{k}(1-e)^{K-k}C^{k}r_{K}(\mathbf{x})^{\alpha
k}\\\
&+\sum_{k=0}^{\lfloor\frac{K+1}{2}\rfloor-1}\binom{K}{k}e^{k}(1-e)^{K-k}C^{(K-k)}r_{K}(\mathbf{x})^{\alpha(K-k)}\end{split}$
(9)

Proof: There are two scenarios in which the detection method may produce
errors: 1) When a majority of the neighboring instances are assigned incorrect
labels and 2) when a majority of the neighboring instances possess accurate
labels. However, due to the feature quality, the possibility remains that the
label of the example $\mathbf{x}$ originates from unknown classes. In the
first scenario, we account for the likelihood of encountering a minimum of
$\lceil\frac{K-1}{2}\rceil+1$ neighbors with wrong labels. Additionally, even
if a significant number of neighbors belong to unknown classes, the chance
persists that the sample originates from known classes. In the second
scenario, we consider the possibility of having a maximum of
$\lfloor\frac{K+1}{2}\rfloor-1$ neighbors with incorrect labels. Similarly,
even if most neighbors are associated with known classes, the potential
remains that the sample comes from unknown classes.

## Detailed Active Open-Set Annotation Setup

### Datasets

In our experiment, we use the CIFAR10, CIFAR100, and Tiny-ImageNet to test our
method. The CIFAR10 consists of 60,000 images in total, each being a 32x32
pixels images. This dataset is divided into 10 classes, with 6,000 images per
class. For the purpose of our experiments, we randomly select 2 out of these
10 classes as known classes, which include a variety of categories such as
airplanes, cars, and cats. Out of the total 60,000 images, 50,000 are used as
a training dataset while the remaining 10,000 serve as a test dataset. To
initialize our experiment, we randomly select 1 percent of the known classes’
training dataset’s images, resulting in 100 images used for the classifier
model’s training.

The CIFAR100, similar to the CIFAR10, consists of 60,000 32x32 pixels images.
However, it has a different configuration which has 100 classes, each
containing 600 images which has 500 for training and 100 for testing. As same
as the CIFAR10, 50,000 images in the CIFAR100 dataset are allocated for
training purposes and the remaining 10,000 are used for testing. Furthermore,
the CIFAR100 introduces an additional layer of categorization; the 100 classes
are grouped into 20 superclasses, known as ’coarse labels’. Each coarse label
corresponds to 5 classes. For our experiment, we randomly select 8% of the
images from the known class as our starting point, which equates to 80 images.

The Tiny-ImageNet comprises 20 classes, with each class containing 500
training images, 50 validation images, and 50 test images. Each image is 64x64
pixels, which in total has 100,000 images making this dataset significantly
larger than either CIFAR10 or CIFAR100 in terms of image number and
resolution. Tiny-ImageNet is a subset of the ILSVRC-2012 classification
dataset, representing a scaled-down version of the larger ImageNet dataset,
which has over 14 million images. For our experiment, we randomly select 8% of
the images from the known class to initialize our test, which also equals to
80 images. Due to the absence of labels in the Tiny-ImageNet test set, we
utilize the validation set as our test set.

### Active Open-Set Annotation

After the completion of the initialization stage, the subsequent stage is the
query stage. During this stage, the images belonging to the remaining
unselected known classes after the initialization, along with those from
unknown classes, are combined to form a new, unlabeled dataset. The next
process is different for learning-based methods and non-learning-based
methods. For the learning-based active learning method, which has an
additional detector network, the images randomly selected at the
initialization stage will be used to train. Then the newly trained detector
network will be used to predict the images from the unlabeled dataset because
the features or activation values will be used for the known and unknown class
selection. For the non-learning-based method, the unlabeled images will go
through their own individual selection process.

At the query stage, both active learning methods are required to select K
number of images, which each method predicted to have the highest possibility
from known classes. Next is the auto-annotation process, where the selected
images will be categorized into known classes and unknown classes based on
their true label. Besides the index of selected images, the query step will
also output the selection precision and recall.

The final process of the query stage is also divided into different pipelines
regarding whether the current active learning method is learning-based or not.
For the learning-based active learning method, the selected images that are
truly from known classes will be added to the previous labeled dataset, and
the images from the unknown classes will also be added to the labeled dataset,
except their label will be changed to the number of known classes plus one to
indicate an unknown class. For the non-learning-based method, only images from
known classes will be added to the labeled dataset. The above querying process
will repeat a Q number of times, and the last query’s classifier’s performance
will be used to evaluate the current active learning method’s effectiveness.

## Further Ablation Studies

### Query batch size

In addition to the experiments detailed in our main paper, where we used a
query batch size of 400 images, we also present our experimental results for
query batch sizes of 600 and 800. Our observations reveal that as the batch
size increases, the accuracy of all active learning methods improves. However,
our proposed method, Neat, outperforms other methods in terms of accuracy,
precision, and recall across all cases. Particularly at smaller query batch
sizes, our method achieves significantly higher accuracy compared to
alternative approaches, especially at a batch size of 400. Furthermore,
utilizing a batch size of 400 images reduces annotation costs by almost half
compared to a batch size of 800. The reduction in batch size is crucial in
open-set active learning because it reduces the annotation cost and
computation cost at the same time. The outcomes can be observed in Figures 5
and 6.

### Network architectures

We also employed ResNet-34, ResNet-50, and VGG-16 for training NEAT. However,
ResNet-18 outperformed the others in terms of classification accuracy.
Moreover, ResNet-18 required the least computational power among these models,
making it a more efficient choice. The outcomes can be observed in Figures 7.
In the subsequent experiment beside ResNet famliy we conducted experiments
using the base Vision Transformer (ViT) and present ImageNet results in Figure
8. The framework of NEAT can be viewed in Figures 9.

|  |
---|---|---
|  |
|  |
Figure 5: Neat achieves higher precision, recall and accuracy compared with
existing active learning methods for active open-set annotation. We evaluate
Neat and the baseline active learning methods on CIFAR10, CIFAR100 and Tiny-
ImageNet based on accuracy, precision and recall.

|  |
---|---|---
|  |
|  |
Figure 6: Neat achieves higher precision, recall and accuracy compared with existing active learning methods for active open-set annotation. We evaluate Neat and the baseline active learning methods on CIFAR10, CIFAR100 and Tiny-ImageNet based on accuracy, precision and recall.  |  |
---|---|---
|  |
Figure 7: Neat achieves higher precision, recall, and accuracy compared with LFOSA. We evaluate Neat and the LFOSA on CIFAR10, CIFAR100 based on accuracy, precision, and recall. Specifically the solid line represents the result of Neat and the dashed line represents the result of LfOSA. |
---|---
a) Accuracy | b) Recall

Figure 8: We use Vision Transformer to train Neat, Red for the proposed method
Neat and Blue for LFOSA.
Figure 9: The architecture of Neat includes a classifier model and a local
feature distribution extractor, designed to identify both known and unknown
samples.


# Automatic Polyp Segmentation using U-Net-ResNet50

Saruar Alam1, Nikhil Kumar Tomar2, Aarati Thakur3, Debesh Jha2,4, Ashish
Rauniyar5,6 1University of Bergen, Norway
2SimulaMet, Norway
3Nepal Medical College, Kathmandu University, Nepal
4UiT The Arctic University of Norway
5University of Oslo, Norway
6Oslo Metropolitan University, Norway
<EMAIL_ADDRESS>
###### Abstract.

Polyps are the predecessors to colorectal cancer which is considered as one of
the leading causes of cancer-related deaths worldwide. Colonoscopy is the
standard procedure for the identification, localization, and removal of
colorectal polyps. Due to variability in shape, size, and surrounding tissue
similarity, colorectal polyps are often missed by the clinicians during
colonoscopy. With the use of an automatic, accurate, and fast polyp
segmentation method during the colonoscopy, many colorectal polyps can be
easily detected and removed. The “Medico automatic polyp segmentation
challenge” provides an opportunity to study polyp segmentation and build an
efficient and accurate segmentation algorithm. We use the U-Net with pre-
trained ResNet50 as the encoder for the polyp segmentation. The model is
trained on Kvasir-SEG dataset provided for the challenge and tested on the
organizer’s dataset and achieves a dice coefficient of 0.8154, Jaccard of
0.7396, recall of 0.8533, precision of 0.8532, accuracy of 0.9506, and F2
score of 0.8272, demonstrating the generalization ability of our model.

††doi: ††isbn: ††conference: Multimedia Evaluation Workshop; December 14-15
2020; Online

##  1\. Introduction

Identification and removal of polyps during colonoscopy have become a standard
procedure. It is often challenging to detect polyps, as they are often hard to
differentiate from surrounding normal tissue. These polyps are usually covered
with stool, mucosa, and other materials that can obscure the correct
diagnosis. This is especially true for the small, flat, and sessile polyps
that are typically not visible during colonoscopy. Moreover, this increases
the miss-rate of polyps up-to 25% (Kumar et al., 2017) and increases the risk
of colorectal cancer in the affected patient. An increase in the 1% adenoma
detection rate leads to a 3% decrease in the risk of colorectal cancer (Corley
et al., 2014). Recently, deep learning techniques have been developed to
overcome these challenges and improve polyp detection accuracy during
colonoscopy. Polyp segmentation based deep learning methods has been
successfully applied for automatic polyp detection in a real-time.

The automatic polyp segmentation plays an important role in the identification
and localization of the polyps in the affected regions. It helps in analyzing
the images or even video frames and classify each pixel into polyp or non-
polyp class instances. This allows the clinician in easy, fast, and more
accurate identification of the polyp in the affected region. The automated
polyp segmentation can help in the development of a Computer-Aided Diagnosis
(CADx) system, which is specially designed for colonoscopy procedures.

The “Medico Automatic Polyp Segmentation Challenge” (Jha et al., 2020a)
consists of two tasks. The first task is “Polyp segmentation task” and the
second is “Algorithm efficiency task”. We have submitted our model in task 1
only.

Figure 1. The proposed U-Net-ResNet50 architecture

##  2\. Related Works

For semantic segmentation task, encoder-decoder networks like FCN (Long et
al., 2015), U-Net (Ronneberger et al., 2015), etc are mostly preferred over
other approaches. U-Net and its variants are used for both natural image
segmentation and biomedical image segmentation. In general, the encoder uses
multiple convolutions to learn and capture the essential semantic features
ranging from low-level to high-level. These upscaled features are then
concatenated with the features from the encoder using the skip connections and
then followed by convolution layers to generate the final output in the form
of a binary mask.

The encoder acts as a feature extractor, where the decoder uses features
extracted from the input to produce to desired segmentation mask. The encoder
can be replaced by a pre-trained network such as VGG16 (Simonyan and
Zisserman, 2014), VGG19 (Simonyan and Zisserman, 2014), etc. These pre-trained
networks are already trained on the ImageNet (Russakovsky et al., 2015)
dataset and have the necessary feature extraction capabilities. Architectures
like SegNet (Badrinarayanan et al., 2017) and TernausNet (Iglovikov and
Shvets, 2018) use pre-trained VGG16 and VGG11 respectively for segmentation
task.

With the success of the residual network (He et al., 2016), ResNet50 is one of
the commonly used architecture for any transfer learning task. The residual
network uses two $3\times 3$ convolutional layers and an identity mapping.
Each convolution layer is followed by a batch normalization layer and a
Rectified Linear Unit (ReLU) activation function. The identity mapping is the
shortcut connection connecting the input and output of the convolutional
layer. The identity mapping helps in building a deeper neural network by
eliminating the problem of vanishing gradients and exploding gradients.

##  3\. Approach

Figure 1 shows an overview of the proposed U-Net-ResNet50 architecture. It is
an encoder-decoder based architecture, where ResNet50 trained on ImageNet
dataset (Russakovsky et al., 2015) is used . The use of a pre-trained encoder
helps the model to converge easily. The input image is fed into the pre-
trained ResNet50 encoder, consisting of a series of residual blocks as their
main component. These residual blocks help the encoder extract the important
features from the input image, which are then passed to the decoder. The
decoder starts a transpose convolution that upscales the incoming feature maps
into the desired shape. Next, these upscaled feature maps are concatenated
with the specific shape feature maps from the pre-trained encoder via skip
connections. These skip connections help the model to get all the low-level
semantic information from the encoder, which allows the decoder to generate
the desired feature maps. After that, it is followed by the two $3\times 3$
convolution layer, where each layer is followed by a batch normalization layer
and a ReLU non-linearity. The last decoder block’s output is passed to a
$1\times 1$ convolution layer, which is further passed to a sigmoid activation
function, finally generating the desired binary mask.
The FastAI (version 2.0) library (doc, 2020) is used to train and evaluate our
model. We have employed resizing, flipping, rotating, zooming, lightning,
warping, and normalizing intensity based on the ImageNet dataset to augment
the input images for training. The model uses Adam optimizer with an initial
learning rate of $10^{-2}$, and cross-entropy loss as its loss function. We
have employed the one-cycle policy where the learning rate changes during
training and achieves super-convergence (Smith and Topin, 2019). We have run
just 50 epochs for training, and the model has converged.

##  4\. Results and analysis

The Medico Automatic Polyp Segmentation challenge (Jha et al., 2020a) provides
an opportunity to study the potential and challenges of automated polyp
segmentation. This study aims at building a model that performs well on the
organizer’s dataset while training on a separate Kvasir-SEG dataset (Jha et
al., 2020b).
Table 1 shows the overall results of the U-Net-ResNet50 architecture on the
Kvasir-SEG test dataset and the organizer’s test dataset provided for the
final evaluation of the model. For the evaluation of the model, the Jaccard
index, Sørensen-Dice coefficient (DSC), recall, precision (Prec.), accuracy
(Acc.), and the F2 are used as the evaluation metrics. Our trained U-Net-
ResNet50 model achieved a dice coefficient of 0.8154, Jaccard of 0.7396,
recall of 0.8533, precision of 0.8532, accuracy of 0.9506 and F2 score of
0.827 on the organiser’s test dataset which can be seen from the table 1.
These results demonstrate the generalization ability of our model. Moreover
Table 1 also shows that the recall value of the organizer’s test dataset is
1.00% higher than the Kvasir-SEG test dataset. This shows that the model is
not overfitting.

Table 1. Quantitative Results on Kvasir-SEG and Test Set (Challenge) Dataset for Task 1. Dataset | Jaccard | DSC | Recall | Prec. | Acc. | F2
---|---|---|---|---|---|---
Kvasir-SEG | 0.7871 | 0.8926 | 0.8433 | 0.9207 | 0.9639 | 0.8585
Test Set | 0.7396 | 0.8154 | 0.8533 | 0.8532 | 0.9506 | 0.8272

##  5\. Conclusion & Future Work

With our U-Net-ResNet50, we achieved competitive performance on the
organizer’s dataset with a dice coefficient of 0.8154. By replacing the U-Net
encoder with a pre-trained ResNet50 and employing a one-cycle policy during
training, we are able to converge the model in a short time. Thus, it helps in
reducing the training time as the encoder weights are not initialized from
scratch. This is an important step towards faster convergence, which would be
useful when the availability of high-performance computing resources is
limited.
In the future, we would like to experiment with more than one pre-trained
encoder by fusing their feature maps and using them for training our model.

###### Acknowledgements.

The computations in this paper were performed on the equipment provided by the
Experimental Infrastructure for Exploration of Exascale Computing (eX3), which
is financially supported by the Research Council of Norway under the contract
270053.
The authors would also like to thank the machine learning group of Mohn
Medical Imaging and Visualization (MMIV) Centre, Norway, for providing the
computing infrastructure for the experiments.

## References

  * (1)
  * doc (2020) 2020\.  FastAI Library.  (2020).  https://docs.fast.ai/.
  * Badrinarayanan et al. (2017) Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla. 2017\.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation.  IEEE transactions on pattern analysis and machine intelligence 39, 12 (2017), 2481–2495.
  * Corley et al. (2014) Douglas A Corley, Christopher D Jensen, Amy R Marks, Wei K Zhao, Jeffrey K Lee, Chyke A Doubeni, Ann G Zauber, Jolanda de Boer, Bruce H Fireman, Joanne E Schottinger, and others. 2014\.  Adenoma detection rate and risk of colorectal cancer and death.  New england journal of medicine 370, 14 (2014), 1298–1306.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770–778.
  * Iglovikov and Shvets (2018) Vladimir Iglovikov and Alexey Shvets. 2018.  Ternausnet: U-net with vgg11 encoder pre-trained on imagenet for image segmentation.  arXiv preprint arXiv:1801.05746 (2018).
  * Jha et al. (2020a) Debesh Jha, Steven A. Hicks, Krister Emanuelsen, Håvard D. Johansen, Dag Johansen, Thomas de Lange, Michael A. Riegler, and Pål Halvorsen. 2020a.  Medico Multimedia Task at MediaEval 2020:Automatic Polyp Segmentation. In Proc. of MediaEval 2020 CEUR Workshop.
  * Jha et al. (2020b) Debesh Jha, Pia H Smedsrud, Michael A Riegler, Pål Halvorsen, Thomas de Lange, Dag Johansen, and Håvard D Johansen. 2020b.  Kvasir-SEG: A Segmented Polyp Dataset. In Proc. of International Conference on Multimedia Modeling (MMM). 451–462.
  * Kumar et al. (2017) Sheila Kumar, Nirav Thosani, Uri Ladabaum, Shai Friedland, Ann M Chen, Rajan Kochar, and Subhas Banerjee. 2017.  Adenoma miss rates associated with a 3-minute versus 6-minute colonoscopy withdrawal time: a prospective, randomized trial.  Gastrointestinal endoscopy 85, 6 (2017), 1273–1280.
  * Long et al. (2015) Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015\.  Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 3431–3440.
  * Ronneberger et al. (2015) Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015\.  U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234–241.
  * Russakovsky et al. (2015) Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, and others. 2015.  Imagenet large scale visual recognition challenge.  International journal of computer vision 115, 3 (2015), 211–252.
  * Simonyan and Zisserman (2014) Karen Simonyan and Andrew Zisserman. 2014.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556 (2014).
  * Smith and Topin (2019) Leslie N Smith and Nicholay Topin. 2019.  Super-convergence: Very fast training of neural networks using large learning rates. In Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications, Vol. 11006. International Society for Optics and Photonics, 1100612\.