# Adaptive Negative Evidential Deep Learning
for Open-set Semi-supervised Learning

Yang Yu  The Chinese University of Hong KongHong KongChina
<EMAIL_ADDRESS>,  Danruo Deng  The Chinese University of Hong KongHong
KongChina<EMAIL_ADDRESS>,  Furui Liu  Zhejiang
LabHangzhouZhejiangChina<EMAIL_ADDRESS>,  Yueming Jin  National
University of SingaporeSingapore<EMAIL_ADDRESS>,  Qi Dou  The Chinese
University of Hong KongHong KongChina<EMAIL_ADDRESS>,  Guangyong Chen
Zhejiang LabHangzhouZhejiangChina<EMAIL_ADDRESS>and  Pheng-Ann Heng
The Chinese University of Hong KongHong KongChina<EMAIL_ADDRESS>
(2018)

###### Abstract.

Semi-supervised learning (SSL) methods assume that labeled data, unlabeled
data and test data are from the same distribution. Open-set semi-supervised
learning (Open-set SSL) considers a more practical scenario, where unlabeled
data and test data contain new categories (outliers) not observed in labeled
data (inliers). Most previous works focused on outlier detection via binary
classifiers, which suffer from insufficient scalability and inability to
distinguish different types of uncertainty. In this paper, we propose a novel
framework, Adaptive Negative Evidential Deep Learning (ANEDL) to tackle these
limitations. Concretely, we first introduce evidential deep learning (EDL) as
an outlier detector to quantify different types of uncertainty, and design
different uncertainty metrics for self-training and inference. Furthermore, we
propose a novel adaptive negative optimization strategy, making EDL more
tailored to the unlabeled dataset containing both inliers and outliers. As
demonstrated empirically, our proposed method outperforms existing state-of-
the-art methods across four datasets. Our code will be released once this
paper is accepted.

Open-set Semi-supervised Learning, Evidential Deep Learning, Adaptive Negative
Optimization

††copyright: acmcopyright††journalyear: 2018††doi:
XXXXXXX.XXXXXXX††conference: Make sure to enter the correct conference title
from your rights confirmation emai; June 03–05, 2018; Woodstock, NY††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Computing methodologies Object
recognition††ccs: Computing methodologies Object recognition

##  1\. Introduction

Figure 1. The EDL detector we introduce can quantify different types of
uncertainty, including epistemic uncertainty due to lack of knowledge, and
aleatoric uncertainty from the complexity of samples in distribution. Based on
these uncertainties, we design different metrics for self-training and
inference. (Images with blue and yellow backgrounds indicate inliers and
outliers, respectively.)

Semi-supervised learning (SSL) has recently witnessed significant progress by
propagating the label information from labeled data to unlabeled data
(Berthelot et al., 2019; Xu et al., 2021; Wang et al., 2022a; Zheng et al.,
2022). Despite the success, SSL methods are deeply rooted in the closed-set
assumption that labeled data, unlabeled data and test data share the same
predefined label set. In reality (Yu et al., 2020), such an assumption may not
always hold as we can only accurately control the label set of labeled data,
while unlabeled and test data may include outliers that belong to the novel
classes that are not seen in labeled data. During self-training, imprecisely
propagating label information to these outliers shall interfere with model
learning and lead to performance degradation. To this end, open-set semi-
supervised learning (Open-set SSL) (Yu et al., 2020) has been emerging to
tackle the problem. It first detects out-of-distribution (OOD) samples as
outliers and then recognizes in-distribution (ID) samples as belonging to the
classes from the predefined label set.

Existing Open-set SSL methods (Guo et al., 2020; Chen et al., 2020) generally
consist of an outlier detector and a classifier. To avoid propagating wrong
label information to outliers during SSL training of the classifier, they
first detect outliers and then optimize SSL loss only for unlabeled instances
taken as inliers. To detect outliers, they aim to estimate confidence score of
a sample being an inlier (reversely, uncertainty score for outliers). For
example, T2T (Huang et al., 2021) proposes to detect outliers by cross-modal
matching over binary detector. OpenMatch (Saito et al., 2021) deploys a one-
vs-all (OVA) classifier (Saito and Saenko, 2021) as outlier detector to output
the confidence score of considering a sample as an inlier and reject outliers
with confidence score below a threshold for SSL training. With outliers
rejected, they improve recognition accuracy of SSL methods.

We identify two main limitations in current state-of-the-art literature to
tackle this task. First, most previous works estimate confidence scores based
on binary likelihoods generated from binary detectors activated by Softmax
function. Such softmax-based networks only provide a point estimation of
predicted class probabilities and cannot quantify whether the uncertainty
stems from the lack of knowledge of outliers (epistemic uncertainty) or from
the complexity of the samples within the distribution (aleatoric uncertainty)
(Sensoy et al., 2018; Malinin and Gales, 2018; parkactive). Moreover, when we
tackle a K-way classification problem with a large K, the binary detectors are
less robust to identify outliers from such a complex dataset that contains
multi-class information (Carbonneau et al., 2018).

One advanced method, evidential deep learning (EDL) (Sensoy et al., 2018) can
explicitly quantify the classification uncertainty corresponding to the
unknown class, by treating the network’s output as evidence for parameterizing
the Dirichlet distribution according to subjective logic (Jøsang, 2016).
Compared with Softmax which only provides probabilistic comparison values
between classes, EDL models second-order probability and uncertainty for each
class.

In this work, we take the _first_ step to introduce EDL into the outlier
detector. As shown in Figure 1, EDL enables a multinomial detector to directly
output uncertainty and evidence for each class, which can alleviate the
insufficient scalability of existing binary detectors. Moreover, different
types of uncertainties can be distinguished by concentration parameters, so we
propose to design different uncertainty metrics for self-training and
inference. Note that though EDL shows an impressive performance in uncertainty
quantification, we empirically find its deficiency in learning representation
preferred by classification tasks and consequent low performance in
classification accuracy. Therefore, we propose to take the advantage of both
Softmax and EDL, by keeping Softmax for representation learning.

Figure 2. Overview of our proposed Adaptive Negative Evidential Deep Learning
(ANEDL). It consists of a shared feature extractor, a Softmax head and an EDL
head. EDL head is used to detect outliers, while Softmax head is used to
classify. We introduce Negative Optimization to explicitly regularize our EDL
detector to output low evidence values for uncertain classes, while proposing
an Adaptive loss weight to encourage the detector to pay more attention to
these uncertain classes. Unlabeled samples with confident predictions are used
jointly with labeled samples to train the classification head.

The other limitation of prior methods is that they do not effectively leverage
the information contained in unlabeled data. The detector should output low
confidence scores for outliers, but this information is underutilized for
model regularization, which may result in suboptimal performance (Malinin and
Gales, 2018). In this regard, we propose _adaptive negative optimization_
tailored for unlabeled data, where the Negative optimization is to explicitly
regulate our EDL detector to output K low evidence values for a given outlier
sample. However, negative optimization can not directly apply to inliers in
unlabeled data. Therefore, we derive an adaptive loss weight for encouraging
the model to separately treat inliers and outliers in the unlabeled data in
the learning process. It can largely avoid the interference between the ID and
OOD features as well as the leading problem of uncertainty availability
reduction. Specifically, we achieve this by using the Fisher information
matrix (FIM) introduced by $\mathcal{I}$-EDL (Deng et al., 2023) to identify
the amount of information for each class of unlabeled samples, while imposing
smaller constraints on the confident classes with high evidence. Therefore,
such optimization can sufficiently compress evidence values of unlabeled
outliers without interfering with the learning of unlabeled inliers.

To our knowledge, we are the first to exploit evidential deep learning in the
Open-set SSL task, and propose a novel Adaptive Negative Evidential Deep
Learning (ANEDL) framework. Below, we summarize the major contributions of
this work:

  * •

We introduce EDL as an outlier detector for Open-set SSL, which enables
quantification of different types of uncertainty and design different
uncertainty metrics for self-training and inference.

  * •

We propose a novel adaptive negative optimization strategy, which tailors EDL
on the unlabeled dataset containing both inliers and outliers.

  * •

Extensive experiments on four public datasets, i.e., CIFAR-10, CIFAR-100,
ImageNet-30, and Mini-ImageNet show that our method outperforms the state-of-
the-art methods in different settings.

##  2\. Related Works

###  2.1. Semi-Supervised Learning

Semi-supervised learning (SSL) aims to ease deep network’s reliance on massive
and expensive labeled data by utilizing limited labeled data together with a
large amount of unlabeled data. Existing works can be generally categorized
into two paradigms: consistency regularization and pseudo labeling (Lee et
al., 2013). Consistency regularization assumes that the network should be
invariant to different perturbations of the same instance. Pseudo-labeling
takes confidently pseudo-labeled unlabeled data as labeled data to train the
network. And many pseudo labeling methods like FixMatch (Sohn et al., 2020),
FlexMatch (Zhang et al., 2021) and AdaMatch (Berthelot et al., 2021) focus on
how to select confidently pseudo-labeled data. Although pseudo-labeling
methods have achieved tremendous success in SSL, they are deeply rooted in the
close-set assumption and can not be directly applied to open-set SSL (Yu et
al., 2020; Saito et al., 2021). Because noisy pseudo-labeled outliers may
degrade the performance of self-supervised training. On the other hand,
OpenMatch (Saito et al., 2021) proposes that soft consistency regularization
by minimizing the distance between its predictions on two augmentations of the
same image can help SSL methods get rid of the close-set assumption can be
kept for open-set SSL. Following OpenMatch, our method also applies soft
consistency regularization to the outlier detector.

###  2.2. Open-set Semi-Supervised Learning

Open-set SSL (Guo et al., 2020; Chen et al., 2020) aims to detect outliers
while categorizing inliers into correct close-set classes. Existing open-set
SSL methods recognize the weakness of SSL methods in terms of detecting
outliers and they develop an extra outlier detector to exclude outliers from
self-supervised training. Their outlier detectors are based on binary
classifier activated by softmax function. For example, MTC (Yu et al., 2020)
deploys a binary classifier to predict the probability of the instance
belonging to outliers and updates the network parameters and the outlier score
alternately. OpenMatch (Saito et al., 2021) takes one-vs-all (OVA) classifier
as its outlier detector and proposes to apply soft consistency regularization
to the outlier detector. An OVA classifier is composed of K binary classifiers
and each classifier predicts the probability of the sample belonging to a
specific class. However, softmax-based networks are notorious for inflating
the probabilities of predicted classes (Szegedy et al., 2016; Guo et al.,
2017; Wilson and Izmailov, 2020), and binary detectors perform poorly on tasks
with a large number of classes (Carbonneau et al., 2018). To address these
limitations, we adopt evidential deep learning (EDL) (Sensoy et al., 2018) as
the outlier detector for Open-set SSL.

###  2.3. Evidential Deep Learning

Evidential neural networks (Sensoy et al., 2018) model the outputs as evidence
to quantify belief masses and uncertainty based on the Dempster-Shafer Theory
of Evidence (DST) (Sentz and Ferson, 2002) and subjective logic (Jøsang,
2016). Similar to this work, (Malinin and Gales, 2018) propose Prior Networks
that explicitly consider the distributional uncertainty to quantify the
mismatch of ID and OOD distribution. Compared with the standard neural network
classifiers direct output the probability distribution of each sample, EDL
obtains the density of classification probability assignments by
parameterizing the Dirichlet distribution. Therefore, EDL can use the
properties of Dirichlet distribution to distinguish different types of
uncertainties, which shows an impressive performance in uncertainty
quantification and is widely used in various applications. For example, (Zhao
et al., 2020) proposes a multi-source uncertainty framework combined with DST
for semi-supervised node classification with GNNs. (Soleimany et al., 2021)
introduces evidential priors over the original Gaussian likelihood function to
model the uncertainty of regression networks. (Bao et al., 2021, 2022) propose
a general framework based on EDL for Open Set Recognition (OSR) and Open Set
Temporal Action Localization (OSTAL), respectively. Compared with previous
efforts, our work is the first to exploit EDL to Open-set SSL.

##  3\. Method

###  3.1. Problem Setting

Open-set semi-supervised learning (Open-set SSL) aims to learn a classifier by
using a set of labeled and unlabeled datasets. Unlike traditional semi-
supervised learning (SSL), the unlabeled dataset of Open-set SSL contains both
ID and OOD samples, i.e., inliers and outliers. More specifically, for a K-way
classification problem, let
$\mathcal{D}_{l}=\\{(\boldsymbol{x}^{l}_{j},y^{l}_{j})\\}_{j=1}^{N_{l}}$ be
the labeled dataset, containing $N_{l}$ labeled samples randomly generated
from a latent distribution $\mathcal{P}$. The labeled example
$\boldsymbol{x}^{l}_{j}$ can be classified into one of K classes, meaning that
its corresponding label $y^{l}_{j}\in\\{c_{1},\cdots,c_{K}\\}$. Let
$\mathcal{D}_{u}=\\{\boldsymbol{x}^{i}_{j}\\}_{j=1}^{N_{i}}\cup\\{\boldsymbol{x}^{o}_{j}\\}_{j=1}^{N_{o}}$
be the unlabeled dataset consisting of $N_{i}+N_{o}$ examples with labels
either from the K known classes (inliers) or never been seen (outliers). The
goal of Open-set SSL is to detect outliers while classifying inliers into the
correct class.

###  3.2. Adaptive Negative Evidential Deep Learning

Approach Overview. As shown in Figure 2, our proposed framework contains three
components: (1) an EDL head $D(\cdot)$ that quantifies uncertainty by
outputting K evidence values to detect outliers in unlabeled data; (2) a
Softmax head $C(\cdot)$ classifies all inlier samples by outputting K-way
probabilities, including labeled data and inliers in unlabeled data; (3) a
feature extractor $F(\cdot)$ shared by outlier detector and classifier. To
adopt our framework to Open-set SSL, we propose an Adaptive Negative
Optimization strategy to train EDL head by using all labeled and unlabeled
samples. We introduce negative optimization to explicitly regularize our EDL
detector to output low evidence values for uncertain classes, while deriving
an adaptive loss weight to encourage the detector to pay more attention to
these uncertain classes. For unlabeled data with confident prediction, we will
use them jointly to train the Softmax classifier with labeled data.

Evidential Outlier Detector. Traditional outlier detectors for Open-set SSL
use Softmax activation function to predict confidence scores and achieve
impressive performance. However, such softmax-based detectors only provide a
point estimation of outlier probabilities, unable to quantify different types
of uncertainty. To tackle the limitations of Softmax, we introduce evidential
deep learning (EDL) (Sensoy et al., 2018) to our framework as an outlier
detection module. EDL proposes to integrate deep classification networks into
the evidence framework of subjective logic (Jøsang, 2016), to jointly predict
classification probabilities and quantify uncertainty. Specifically, for K-way
classification, EDL uses a network $g(\boldsymbol{\theta})$ to calculate its
evidence vector $\boldsymbol{e}=g(\boldsymbol{x}|\boldsymbol{\theta})$. Note
that an activation function, e.g. Relu or Softplus, is on top of
$g(\boldsymbol{\theta})$ to guarantee non-negative evidence. Then EDL
correlates its derived evidence vector $\boldsymbol{e}\in\mathbb{R}_{+}^{K}$
to the concentration parameter $\boldsymbol{\alpha}\in\mathbb{R}_{+}^{K}$ of
Dirichlet distribution $D(\boldsymbol{p}|\boldsymbol{\alpha})$ with the
equality ${\alpha}_{i}=e_{i}+1$. Dirichlet distribution parameterized by
$\boldsymbol{\alpha}$ models the density of classification probability
assignments and uncertainty. The expected probability of the $k^{th}$ class
and uncertainty can be derived by

${\hat{p}}_{k}=\frac{{\alpha}_{k}}{S}\quad\text{and}\quad
u=\frac{K}{\alpha_{0}},$

where $S$ is the precision of Dirichlet distribution and defined as
$S=\sum_{k=1}^{K}{\alpha}_{k}$. Higher values of $S$ lead to sharper, more
confident distributions. Given a sample, ${\alpha}_{k}$ is incremented to
update the Dirichlet distribution of the sample when the evidence of $k^{th}$
class is observed.

Joint Optimization with Softmax and EDL. Although EDL-related works (Sensoy et
al., 2018; Deng et al., 2023) are capable of jointly classifying and
quantifying uncertainty, we empirically find its deficiency in optimizing a
network to learn representation for classification. That is, the
classification performance of the evidential network is lower than that of a
network activated by Softmax and optimized with only cross-entropy loss. In
this way, we propose to take the strength of both Softmax and EDL and jointly
optimize their losses. In our proposed framework, Softmax head takes charge of
learning representation and predicting classification probabilities, while EDL
is responsible for quantifying uncertainty. During training, with EDL
excluding outliers, we adopt an SSL method (i.e., FixMatch following OpenMatch
(Saito et al., 2021)) to our Softmax head to enhance representation quality
and classification accuracy. We also utilize counterfactual reasoning and
adaptive margins proposed in DebiasPL (Wang et al., 2022b) to remove the bias
of pseudo label in FixMatch.

Adaptive Negative Optimization. The core challenge of Open-set SSL is that
unlabeled data contains novel categories that are not seen in labeled training
data, i.e., outliers. To enhance the separation between inliers and outliers,
detectors should output low confidence scores for outliers, but traditional
binary detectors do not sufficiently leverage this information for model
regularization, which may result in suboptimal performance (Malinin and Gales,
2018). Inspired by negative learning (Ishida et al., 2017), we propose
Adaptive Negative Optimization (ANO) to avoid providing misinformation to the
EDL detector by focusing on uncertain classes. In particular, we introduce the
negative optimization to explicitly regulate our detector to output low
evidence values for uncertain classes. Meanwhile, adaptive loss weights are
proposed to encourage EDL detectors to pay more attention to uncertain classes
during the learning process.

More specifically, we adopt the Fisher information matrix (FIM) introduced by
$\mathcal{I}$-EDL (Deng et al., 2023) to identify the amount of information
contained in each class of each sample, while imposing explicit constraints on
the informative uncertainty classes. $\mathcal{I}$-EDL proves that the
evidence of a class exhibits a negative correlation with its Fisher
information, i.e., the class with more evidence corresponds to less Fisher
information, and shows an impressive performance in uncertainty quantification
(Deng et al., 2023). Therefore, we propose to use FIM as an indicator to
distinguish inliers from outliers to adaptively regulate our model to pay more
attention to uncertain classes. For those uncertain classes with less evidence
in unlabeled samples, we impose explicit constraints to output K low evidence
values through the Kullback-Leibler (KL) divergence term. The objective
function for unlabeled data is defined as

(1) $\displaystyle\min_{\boldsymbol{\theta}}\
\mathbb{E}_{\boldsymbol{x}\sim\mathcal{D}_{u}}\left[D_{KL}(\mathcal{U}\|\mathcal{N}(\mathbb{E}_{Dir(\boldsymbol{\alpha})}[\boldsymbol{p}],\sigma^{2}\mathcal{I}(\boldsymbol{\alpha})^{-1})\right]$

where $\mathcal{U}=\mathcal{N}(1/K,\lambda^{2})$,
$\boldsymbol{\alpha}=g(\boldsymbol{x}|\boldsymbol{\theta})+\boldsymbol{1}$,
and $\mathcal{I}(\boldsymbol{\alpha})$ denotes the FIM of
$Dir(\boldsymbol{\alpha})$ (the closed-form expression is provided in
Appendix). According to the definition of KL divergence, the loss function for
an unlabeled sample $\boldsymbol{x}_{j}\in\mathcal{D}_{u}$ in Eq. (1) can be
simplified as

(2)
$\displaystyle\mathcal{L}_{j}^{\text{N-EDL}}=\sum_{k=1}^{K}(\frac{1}{K}-\frac{\alpha_{jk}}{\alpha_{j0}})^{2}\psi^{(1)}(\alpha_{jk})-\lambda_{1}\log|\mathcal{I}(\boldsymbol{\alpha}_{j})|,$

where

$\log|\mathcal{I}(\boldsymbol{\alpha}_{j})|=\sum_{k=1}^{K}\log\psi^{(1)}(\alpha_{jk})+\log(1-\sum_{k=1}^{K}\frac{\psi^{(1)}(\alpha_{j0})}{\psi^{(1)}(\alpha_{jk})}),$

${\alpha}_{j0}=\sum_{k=1}^{K}{\alpha}_{jk}$, and $\psi^{(1)}(\cdot)$
represents the trigamma function, defined as
$\psi^{(1)}(x)=d\psi(x)/dx=d^{2}\ln\Gamma(x)/dx^{2}$ (See Appendix for the
detailed derivation process). We name the above weighted loss function as
adaptive negative EDL loss. Since $\psi^{(1)}(x)$ is a monotonically
decreasing function when $x>0$, class labels with less evidence would be
subject to greater penalties to achieve a flatter output. Conversely, once a
certain class of evidence is learned, the weight will be reduced (see red
number in Firgure 2), so that the inlier features in the unlabeled dataset do
not interfere too much with the learning of outlier features. We also use
labeled dataset to enhance the learning of inlier features. We follow
$\mathcal{I}$-EDL (Deng et al., 2023) to impose constraints on the uncertain
class of labeled samples, outputting a sharp distribution that fits the ground
truth. The loss function for an labeled sample
$(\boldsymbol{x}^{l}_{j},\boldsymbol{y}^{l}_{j})\in\mathcal{D}_{l}$ can be
expressed as:

(3) $\displaystyle\mathcal{L}_{j}^{\text{P-EDL}}=$
$\displaystyle\sum_{k=1}^{K}\left((y_{jk}-\frac{\alpha_{jk}}{\alpha_{j0}})^{2}+\frac{\alpha_{jk}(\alpha_{j0}-\alpha_{jk})}{\alpha_{j0}^{2}(\alpha_{j0}+1)}\right)\psi^{(1)}(\alpha_{jk})$
$\displaystyle-\lambda_{2}\log|\mathcal{I}(\boldsymbol{\alpha}_{j})|,$

where $\boldsymbol{y}^{l}_{j}$ denotes one-hot encoded ground-truth of
$\boldsymbol{x}^{l}_{j}$.

Strengthened KL Loss. KL divergence loss $\mathcal{L}_{j}^{\text{KL-ORI}}$ as
part of classical EDL loss, aims to penalize evidence for misleading classes
that a sample does not belong to:

(4) $\mathcal{L}_{j}^{\text{KL-
ORI}}=D_{KL}(Dir(\boldsymbol{p}_{j}|\hat{\boldsymbol{\alpha}_{j}})\|Dir(\boldsymbol{p}_{j}|\boldsymbol{1})).$

where
$\hat{\boldsymbol{\alpha}}_{j}=\boldsymbol{\alpha}_{j}\odot(1-\boldsymbol{y}^{l}_{j})+\boldsymbol{y}^{l}_{j}$.
However, $\mathcal{L}_{j}^{\text{KL-ORI}}$ only shrinks misleading evidence
but leaves non-misleading evidence ignored. To help enhance non-misleading
evidence, we propose to utilize a strengthened KL-divergence to force our
predicted Dirichlet distribution
$Dir(\boldsymbol{p}_{j}|{\boldsymbol{\alpha}}_{j})$ to approach a new target
Dirichlet distribution $Dir(\boldsymbol{p}_{j}|\boldsymbol{\beta})$. In
particular, we introduce a new KL divergence term as follows,

(5) $\displaystyle\mathcal{L}_{j}^{\text{KL}}=$ $\displaystyle
D_{KL}(Dir(\boldsymbol{p}_{j}|\boldsymbol{\alpha}_{j})\|Dir(\boldsymbol{p}_{j}|\boldsymbol{\beta}))$
$\displaystyle=$
$\displaystyle\log\Gamma(\alpha_{j0})-\sum_{k=1}^{K}\log\Gamma(\alpha_{jk})-\log\Gamma(\beta_{j0})+\sum_{k=1}^{K}\log\Gamma(\beta_{jk})$
$\displaystyle+\sum_{k=1}^{K}(\alpha_{jk}-\beta_{k})\left[\psi(\alpha_{jk})-\psi(\alpha_{j0})\right].$

For labeled data, we set
$\boldsymbol{\beta}=[1,\cdots,P,\cdots,1]\in\mathbb{R}^{K}_{+}$, where $P$ is
a large target evidence value for ground-truth, e.g. 100. Our strengthened KL-
Divergence also applies to unlabeled data by using
$\boldsymbol{\beta}=\boldsymbol{1}$ as the target distribution. The detailed
closed-form expression of $\mathcal{L}_{j}^{\text{KL}}$ can be found in
Appendix.

Overall Loss Function. For the EDL outlier detector, except Eq.
(LABEL:obj_nedl), (3), (5), we add a consistency loss
$\mathcal{L}_{\text{CON}}$ to enhance the smoothness of detector, which is
defined as

(6)
$\displaystyle\mathcal{L}_{\text{CON}}(\mathcal{D}_{u})=\frac{1}{N_{i}+N_{o}}\sum_{j=1}^{N_{i}+N_{o}}\|\boldsymbol{\alpha}_{j}^{s}-\boldsymbol{\alpha}_{j}^{w}\|^{2}_{2},$

where $\boldsymbol{\alpha}_{j}^{s},\boldsymbol{\alpha}_{j}^{w}$ represent the
predicted evidence vector of strong and weak augmentations of the same
unlabeled sample. For the Softmax classifier, we compute the standard cross-
entropy loss $\mathcal{L}_{\text{CE}}$ for labeled data and the FixMatch loss
$\mathcal{L}_{\text{FM}}$ for the high-certainty inliers $\mathcal{I}_{u}$
identified by the EDL detector. Therefore, the overall loss function of our
proposed ANEDL can be expressed as

(7) $\displaystyle\mathcal{L}_{\text{ANEDL}}=$
$\displaystyle\mathcal{L}_{\text{CE}}(\mathcal{D}_{l})+\mathcal{L}_{\text{FM}}(\mathcal{I}_{u})$
$\displaystyle+$
$\displaystyle\mathcal{L}_{\text{ANO}}(\mathcal{D}_{l},\mathcal{D}_{u})+\lambda_{\text{CON}}\mathcal{L}_{\text{CON}}(\mathcal{D}_{u}),$

where

(8) $\displaystyle\mathcal{L}_{\text{ANO}}(\mathcal{D}_{l},\mathcal{D}_{u})=$
$\displaystyle\lambda_{\text{P-EDL}}\frac{1}{N_{l}}\sum_{j=1}^{N_{l}}(\mathcal{L}_{j}^{\text{P-EDL}}+\mathcal{L}_{j}^{\text{KL}})$
$\displaystyle+$
$\displaystyle\lambda_{\text{N-EDL}}\frac{1}{N_{i}+N_{o}}\sum_{j=1}^{N_{i}+N_{o}}(\mathcal{L}_{j}^{\text{N-EDL}}+\mathcal{L}_{j}^{\text{KL}}),$

$\lambda_{\text{P-EDL}},\lambda_{\text{N-EDL}}$ and $\lambda_{\text{CON}}$ are
the hyperparameters used to control the trade-off for each objective.

To minimize our overall loss, we train our model for two stages. In the first
stage, we pre-train our model with
$\mathcal{L}=\mathcal{L}_{\text{CE}}(\mathcal{D}_{l})+\mathcal{L}_{\text{ANO}}(\mathcal{D}_{l},\mathcal{D}_{u})+\lambda_{\text{CON}}\mathcal{L}_{\text{CON}}(\mathcal{D}_{u})$
which drops $\mathcal{L}_{\text{FM}}$ for $E_{FM}$ epochs. In second stage, we
start self-training of classifier and use $\mathcal{L}_{\text{ANEDL}}$ as loss
function. Before every self-training epoch, we calibrate softmax head and EDL
head to calculate uncertainty metric to select the inliers for softmax head to
conduct self-training.

Dataset |  | CIFAR-10 |  | CIFAR-100 |  | CIFAR-100 |  | ImageNet-30
---|---|---|---|---|---|---|---|---
Inlier/Outlier Classes |  | 6/4 |  | 55/45 |  | 80/20 |  | 20/10
No. of labeled samples |  | 50 | 100 | 400 |  | 50 | 100 |  | 50 | 100 |  | 10%
FixMatch (Sohn et al., 2020) |  | 56.1$\pm$0.6 | 60.4$\pm$0.4 | 71.8$\pm$0.4 |  | 72.0$\pm$1.3 | 79.9$\pm$0.9 |  | 64.3$\pm$1.0 | 66.1$\pm$0.5 |  | 88.6$\pm$0.5
MTC (Yu et al., 2020) |  | 96.6$\pm$0.6 | 98.2$\pm$0.9 | 98.9$\pm$ 0.1 |  | 81.2$\pm$3.4 | 80.7$\pm$4.6 |  | 79.4$\pm$1.0 | 73.2$\pm$3.5 |  | 93.8$\pm$0.8
OpenMatch (Saito et al., 2021) |  | 99.3$\pm$0.9 | 99.7$\pm$0.2 | 99.3$\pm$0.2 |  | 87.0$\pm$1.1 | 86.5$\pm$2.1 |  | 86.2$\pm$0.6 | 86.8$\pm$1.4 |  | 96.4$\pm$0.7
Ours |  | 98.5$\pm$0.2 | 99.0$\pm$0.2 | 99.4$\pm$0.1 |  | 90.3$\pm$0.1 | 91.1$\pm$0.4 |  | 89.2$\pm$0.5 | 91.1$\pm$1.0 |  | 96.6$\pm$0.3

Table 1. Mean and standard deviation of AUROC (%) on CIFAR-10, CIFAR-100 and
ImageNet-30. Higher is better. For CIFAR-10 and CIFAR-100, the three runs
correspond to three different folds. For ImageNet-30, the three runs are
conducted on the same folder but with different random seeds. The best results
are in bold.

Dataset |  | CIFAR-10 |  | CIFAR-100 |  | CIFAR-100 |  | ImageNet-30
---|---|---|---|---|---|---|---|---
Inlier/Outlier Classes |  | 6/4 |  | 55/45 |  | 80/20 |  | 20/10
No. of labeled samples |  | 50 | 100 | 400 |  | 50 | 100 |  | 50 | 100 |  | 10%
FixMatch (Sohn et al., 2020) |  | 43.2$\pm$1.2 | 29.8$\pm$0.6 | 16.3$\pm$0.5 |  | 35.4$\pm$0.7 | 27.3$\pm$0.8 |  | 41.2$\pm$0.7 | 34.1$\pm$0.4 |  | 12.9$\pm$0.4
MTC (Yu et al., 2020) |  | 20.3$\pm$0.9 | 13.7$\pm$0.9 | 9.0$\pm$0.5 |  | 33.5$\pm$1.2 | 27.9$\pm$0.5 |  | 40.1$\pm$0.8 | 33.6$\pm$0.3 |  | 13.6$\pm$0.7
OpenMatch (Saito et al., 2021) |  | 10.4$\pm$0.9 | 7.1$\pm$0.5 | 5.9$\pm$0.5 |  | 27.7$\pm$0.4 | 24.1$\pm$0.6 |  | 33.4$\pm$0.2 | 29.5$\pm$0.3 |  | 10.4$\pm$1.0
Ours |  | 9.6$\pm$0.4 | 7.0$\pm$0.4 | 5.5$\pm$0.2 |  | 26.7$\pm$0.2 | 23.2$\pm$0.6 |  | 32.3$\pm$0.3 | 28.5$\pm$0.2 |  | 11.3$\pm$0.4

Table 2. Error rate (%) corresponding to Table 1. Lower is better. The best
results are in bold.

###  3.3. Uncertainty Metric for Open-set SSL

Inlier Selection in Self-training. To help better enhance recognition
performance with unlabeled data, we aim to select inliers with accurate pseudo
labels from unlabeled data for self-training. In FixMatch, pseudo label is
generated by Softmax head and will also be used to softmax head as a
supervision signal. Therefore, we propose to leverage EDL head to quantify the
uncertainty of the class predicted by softmax head and select inliers with
confident prediction which are more likely to be accurately pseudo-labeled.
Specifically, given an unlabeled sample $\boldsymbol{x}_{u}$, the Softmax head
first predicts pseudo label $\hat{k}=\arg\max_{k}C(F(\boldsymbol{x}_{u}))$.
Then we use the $\hat{k}^{th}$ evidence value of $\boldsymbol{\alpha}_{u}$
predicted by the EDL head as uncertainty metric and select top-O most certain
unlabeled samples as inliers. Here, O is a hyperparameter.

Outlier Detection for Inference. Detecting outliers in test data is also one
of the tasks of Open-set SSL. Unlike selecting inliers from unlabeled data,
detected inliers in test data do not have to consider calibrating with
Softmax. Therefore, the uncertainty metric for inference is to accurately
quantify epistemic uncertainty for better separation between inliers and
outliers. Based on the evidence vector $\boldsymbol{\alpha}$ predicted by EDL
head, we can derive various uncertainty metrics, i.e. mutual information,
differential entropy, and total evidence. Previous EDL methods demonstrate
total evidence ${\alpha}_{0}=\sum_{k=1}^{K}{\alpha}_{k}$ achieves the best
outlier detection performance (Sensoy et al., 2018; Deng et al., 2023).
However, we empirically find that classes with small evidence values may form
a long-tail effect because the EDL detector assigns a non-negative evidence
value to each class. Although these evidence values are small, their total
amount is not negligible, especially when dealing with tasks with a large
number of classes. To avoid these long-tailed evidence values disturbing the
separation between inliers and outliers, we propose to sort the evidence
vectors in descending order and use only the sum of the top-M evidence values
as our uncertainty metric for inference.

##  4\. Experiments

In this section, we conduct extensive experiments to compare the performance
of our method with previous SOTA methods and analyze several key components of
our method.

###  4.1. Datasets

We evaluate our method on four public datasets. Given a dataset, we first
split its class set into inlier and outlier class set. Next, we randomly
select inliers to form labeled, unlabeled, validation and test dataset as in
SSL. Then all outliers will be randomly arranged into unlabeled dataset and
test dataset. With four datasets, we set several experimental settings with
different numbers of labeled samples and different amounts of inliers/outliers
classes.

CIFAR-10 (Krizhevsky et al., 2009) has 10 categories and each category has
6,000 images of size $32\times 32$. 1,000 images of each category are used for
test. We choose 6/4 classes as inlier/outlier classes. The amount of labeled
images of each inlier class varies in $\\{50,100,400\\}$. And the number of
validation images of each inlier class is 50.

CIFAR-100 (Krizhevsky et al., 2009) is composed of 60,000 images of size
$32\times 32$ from 100 classes. 100 images of each class are used for test. We
choose 55/80 classes as inlier classes, resulting in 45/20 outliers classes.
The amount of labeled images of each inlier class varies in $\\{50,100\\}$.
And the number of validation images of each inlier class is 50.

ImageNet-30 (Hendrycks and Gimpel, 2016) is a subset of ImageNet (Deng et al.,
2009). It has 30 classes and each class contains 1,300 images of size
$224\times 224$. 100 images of each class are used for test. We choose 20
classes as inlier classes. The amounts of labeled and validation images of
each inlier class are both 130.

Mini-ImageNet (Vinyals et al., 2016) is also a subset of ImageNet. It has 100
classes and each class contains 600 images of size $84\times 84$. 100 images
of each class are used for test. The number of inlier classes varies in
$\\{50,60,80\\}$. The amounts of labeled and validation images of each inlier
class are both 50.

###  4.2. Implementation Details

We employ a randomly initialized Wide ResNet-28-2 (Zagoruyko and Komodakis,
2016) to conduct experiments on CIFAR-10 and CIFAR-100, and a randomly
initialized ResNet-18 (He et al., 2016) for experiments on ImageNet-30 and
Mini-ImageNet. The Softmax head is a linear layer and the EDL head is non-
linear, which is composed of 4 MLP layers. For the pre-training stage, we set
its length $E_{FM}$ as 10 for all experiments. We set the length of an epoch
as 1024 steps, which means we select a new set of inliers every 1024 steps
during self-training. For hyper-parameters of FixMatch, we set them the same
as in OpenMatch (Saito et al., 2021). We report the hyper-parameters of our
loss function in the Appendix. We adopt SGD optimizer with 0.0001 weight decay
and 0.9 momentum to train our model by setting initial learning rate as 0.03
with the cosine decrease policy. We set the batch size of labeled and
unlabeled samples as 64 and 128 respectively for all experiments. Experiments
on ImageNet-30 are conducted with a single 24-GB GPU and other experiments are
conducted with a single 12-GB GPU. All experiments in this work are conducted
over 3 runs with different random seeds, and we report their mean and standard
derivation.

Figure 3. Comparison with OpenMatch on Mini-ImageNet. We use the same fold to
conduct three runs and report the mean of three runs at the top of each bar.

###  4.3. Comparison with Other Methods

To illustrate the effectiveness of our method on Open-set SSL, we compare it
against several baseline methods, including FixMatch (Sohn et al., 2020), MTC
(Yu et al., 2020) and OpenMatch (Saito et al., 2021). We use classification
error rate to evaluate the performance of classifier in inliers and AUROC to
evaluate the performance of outlier detection. AUROC is the general evaluation
metric for novelty detection (Hendrycks and Gimpel, 2016). Table 1 and Table 2
show the AUROC and error rate on the CIFAR-10, CIFAR-100, and ImageNet-30,
respectively, where the results of other methods are quoted from OpenMatch
(Saito et al., 2021). Figure 3 reports the experimental results on Mini-
ImageNet, where the results of OpenMatch are obtained based on the author’s
own implementation.

As shown in Table 1, we can see that our method outperforms the state-of-the-
art method, i.e., OpenMatch, in most experiment settings with respect to
AUROC. In Table 2, our proposed method also achieves state-of-the-art
performance in terms of error rate in 7 out of 8 settings. These results
illustrate that our method is more advanced in epistemic uncertainty
quantification and has superior performance in outlier detection as well as
inlier classification. Notably, in the more challenging settings with a large
number of classes, i.e., settings on CIFAR-100, our method significantly
outperforms OpenMatch in both error rate and AUROC, with improvements of
$\sim\textbf{1\%}$ and $\sim\textbf{3\%}$. It shows our multinomial detector
is more robust than the binary detector in identifying outliers from complex
modalities when dealing with K-way classification tasks with large K. To
further demonstrate our superiority in settings with a large number of inlier
classes, we compare our method with OpenMatch on Mini-ImageNet, which is even
more difficult than CIFAR-100 with larger image size and more complex
patterns. As shown in Figure 3, our method consistently outperforms OpenMatch
with respect to both AUROC and error rate.

###  4.4. Ablation Studies

We next conduct ablation studies to evaluate each component of our method. All
experiments are conducted under the setting of 55 inlier classes and 50
labeled samples per class on CIFAR-100. We report the mean and standard
deviation of three runs on three different folds.

Component analysis of ANO. Our ANO loss consists of two components, i.e.
negative optimization and adaptive loss weight. For “without NO and Adaptive”,
we directly replace $\mathcal{L}_{\text{ANO}}$ to classical EDL loss, which
means we apply no explicit constraints to outliers in unlabeled data. Since
the information contained in unlabeled data keeps underutilized, both
classification and outlier detection performance are suboptimal as shown in
Table 3. For “without Adaptive”, we remove our adaptive loss weight from
$\mathcal{L}_{\text{ANO}}$, which means not using the FIM to identify the
amount of information for each class of unlabeled samples. As a result, all
unlabeled samples including inliers are equally regulated to output K-low
evidence values. Although negative optimization compresses evidence values of
unlabeled outliers, it simultaneously depresses the evidence of unlabeled
inliers, making inlier features and outlier features interfere with each
other. Therefore, its improvement compared to “without NO and Adaptive” is
still limited. Our proposed $\mathcal{L}_{\text{ANO}}$, i.e., “with NO and
Adaptive”, peaks the best performance by sufficiently compressing evidence
values of unlabeled outliers without interfering with the learning of
unlabeled inliers.

NO | Adaptive | AUROC | Error rate
---|---|---|---
|  | 88.4$\pm$0.4 | 27.0$\pm$0.3
$\checkmark$ |  | 89.0$\pm$0.3 | 26.9$\pm$0.3
$\checkmark$ | $\checkmark$ | 90.3$\pm$0.1 | 26.7$\pm$0.2
Table 3. Component analysis of Adaptive Negative Optimization (ANO). KL divergence | AUROC | Error rate
---|---|---
$D_{KL}(D(\boldsymbol{p}_{i}|\hat{\boldsymbol{\alpha}}_{i})\|D(\boldsymbol{p}_{i}|\boldsymbol{1}))$ | 89.9$\pm$0.3 | 27.0$\pm$0.2
$D_{KL}(D(\boldsymbol{p}_{i}|\boldsymbol{\alpha}_{i})\|D(\boldsymbol{p}_{i}|\boldsymbol{{\beta}_{100}}))$ | 90.3$\pm$0.1 | 26.7$\pm$0.2
$D_{KL}(D(\boldsymbol{p}_{i}|\boldsymbol{\alpha}_{i})\|D(\boldsymbol{p}_{i}|\boldsymbol{{\beta}_{200}}))$ | 89.9$\pm$0.9 | 26.9$\pm$0.2
Table 4. Evaluation on KL divergence.

Evaluation on KL divergence. Besides shrinking misleading evidence, we propose
to boost non-misleading evidence with KL divergence loss to improve the
separation of inliers and outliers. In Eq. (5), we keep
$\boldsymbol{\alpha}_{i}$ unchanged and setting target Dirichlet Distribution
to $Dir(\boldsymbol{p}_{i}|\boldsymbol{\beta})$, where
$\boldsymbol{\beta}=[1,\cdots,P,\cdots,1]$. $\boldsymbol{{\beta}_{100}}$ and
$\boldsymbol{{\beta}_{200}}$ denote $P=100$ and $P=200$, respectively. Table 4
shows that our strengthen KL divergence can effectively improve the separation
of inliers and outliers by boosting non-misleading evidence. However, a too
large $P$ may cause an unstable training process, resulting in suboptimal
performance.

Evaluation of uncertainty metric during inference. Previous EDL methods
propose to use the sum of all K evidence values as a metric to measure
epistemic uncertainty, while we propose to use the sum of top-M evidence
values as the metric. For each experiment, we use the same trained model to
evaluate performance and only change how uncertainty metric is calculated. We
also compare the configuration of the uncertainty metric used for inlier
selection in self-training, denoted as “Calibration”. Since uncertainty metric
during inference influences outlier detection more, we report AUROC in Figure
4. Results show that the sum of top $M=25$ evidence values is a better metric
to quantify epistemic uncertainty, compared with the sum of all $K=55$
evidence values. Note that the improvements steadily appear in each trained
model. Comparison between $M=25$ and $M=3$ (or $1$) also shows that a small M
misses useful evidence values and is insufficient to measure epistemic
uncertainty.

Figure 4. Evaluation of uncertainty metric during inference. Only AUROC is reported. We show the mean of three runs at the top of each box. M denotes the sum of top-M evidence values. Calibration denotes uncertainty metric used in ST. Uncertainty Metric in ST | AUROC | Error rate
---|---|---
w/o calibration | 89.8$\pm$0.2 | 27.8$\pm$0.2
w calibration | 90.3$\pm$0.1 | 26.7$\pm$0.2
Table 5. Evaluation of uncertainty metric in ST. “w/o calibration” denotes the
sum of top-25 evidence values (M=25) as uncertainty metric, which is used for
inference.

Evaluation of uncertainty metric in self-training. The choice of uncertainty
metric is important to inliers selection in self-training (ST). To illustrate
the effectiveness of our proposed uncertainty metric for ST (denoted as “with
calibration”) , we compare it with the sum of the top 25 evidence values
(denoted as “without calibration”) which achieves the best outlier detection
performance during inference (see Figure 4). We use these two different
metrics for self-training respectively and conduct inference with the latter
one. Although “without calibration” shows superior performance during
inference, it performs poorly if directly used in ST as shown in Table 5.
Whereas our proposed “with calibration”, which quantifies epistemic
uncertainty of the class predicted by Softmax head, shows clear advantages
over ”without calibration”. The superiority indicates that for inlier
selection in self-training, we should calibrate Softmax head and EDL head to
quantify epistemic uncertainty of predicted class, instead of the whole sample
estimated by the sum of top-25 evidence values.

Evaluation of DebiasPL. As mentioned in “Joint Optimization with Softmax and
EDL” of Section 3.2, we utilize counterfactual reasoning and adaptive margins
proposed in DebiasPL (Wang et al., 2022b) to remove the bias of pseudo label
in Fixmatch. In Table 6, we study the influence of DebiasPL. The results of
OpenMatch come from Table 1 and 2, and OpenMatch is trained without DebiasPL.
Therefore we remove DebiasPL from our method for a fair comparison. We can see
that without DebiasPL, our method still outperforms OpenMatch with respect to
both AUROC and error rate, because we select better inliers for self-training
with proper uncertainty metric. Also, the comparison between “with” and
“without DebiasPL” proves that DebiasPL can help further improves our
performance with bias in pseudo label alleviated.

Method | AUROC | Error rate
---|---|---
OpenMatch | 87.0$\pm$1.1 | 27.7$\pm$0.4
Ours(w/o DebiasPL) | 90.0$\pm$0.5 | 27.0$\pm$0.2
Ours(w DebiasPL) | 90.3$\pm$0.1 | 26.7$\pm$0.2
Table 6. Evaluation on DebiasPL.

##  5\. Conclusion

In this work, we propose a novel framework, adaptive negative evidential deep
learning (ANEDL), for open-set semi-supervised learning (Open-set SSL). ANEDL
adopts evidential deep learning (EDL) to Open-set SSL for the first time and
designs novel adaptive negative optimization method. In particular, EDL, as an
advanced uncertainty quantification method, is deployed to estimate different
types of uncertainty for inliers selection in self-training and outlier
detection in inference. Furthermore, to enhance the separation between inliers
and outliers, we propose adaptive negative optimization to explicitly compress
the evidence value of outliers in unlabeled data and avoid interfering with
the learning of inliers in unlabeled data with adaptive loss weight. Our
extensive experiments on four datasets demonstrate that our method is superior
to other state-of-the-art methods.

## References

  * (1)
  * Bao et al. (2021) Wentao Bao, Qi Yu, and Yu Kong. 2021.  Evidential deep learning for open set action recognition. In _Proceedings of the ieee/cvf international conference on computer vision_. 13349–13358.
  * Bao et al. (2022) Wentao Bao, Qi Yu, and Yu Kong. 2022.  OpenTAL: Towards Open Set Temporal Action Localization. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2979–2989.
  * Berthelot et al. (2019) David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. 2019.  Mixmatch: A holistic approach to semi-supervised learning.  _Advances in neural information processing systems_ 32 (2019).
  * Berthelot et al. (2021) David Berthelot, Rebecca Roelofs, Kihyuk Sohn, Nicholas Carlini, and Alex Kurakin. 2021.  Adamatch: A unified approach to semi-supervised learning and domain adaptation.  _arXiv preprint arXiv:2106.04732_ (2021).
  * Carbonneau et al. (2018) Marc-André Carbonneau, Veronika Cheplygina, Eric Granger, and Ghyslain Gagnon. 2018.  Multiple instance learning: A survey of problem characteristics and applications.  _Pattern Recognition_ 77 (2018), 329–353.
  * Chen et al. (2020) Yanbei Chen, Xiatian Zhu, Wei Li, and Shaogang Gong. 2020\.  Semi-supervised learning under class distribution mismatch. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 3569–3576.
  * Deng et al. (2023) Danruo Deng, Guangyong Chen, Yang Yu, Furui Liu, and Pheng-Ann Heng. 2023.  uncertainty estimation by fisher information-based evidential deep learning.  _arXiv preprint arXiv:0712.0248_ (2023).
  * Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009\.  Imagenet: A large-scale hierarchical image database. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 248–255.
  * Guo et al. (2017) Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. 2017\.  On calibration of modern neural networks. In _International conference on machine learning_. PMLR, 1321–1330.
  * Guo et al. (2020) Lan-Zhe Guo, Zhen-Yu Zhang, Yuan Jiang, Yu-Feng Li, and Zhi-Hua Zhou. 2020.  Safe deep semi-supervised learning for unseen-class unlabeled data. In _International conference on machine learning_. PMLR, 3897–3906.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 770–778.
  * Hendrycks and Gimpel (2016) Dan Hendrycks and Kevin Gimpel. 2016.  A baseline for detecting misclassified and out-of-distribution examples in neural networks.  _arXiv preprint arXiv:1610.02136_ (2016).
  * Huang et al. (2021) Junkai Huang, Chaowei Fang, Weikai Chen, Zhenhua Chai, Xiaolin Wei, Pengxu Wei, Liang Lin, and Guanbin Li. 2021\.  Trash to treasure: harvesting OOD data with cross-modal matching for open-set semi-supervised learning. In _Proceedings of the ieee/cvf international conference on computer vision_. 8310–8319.
  * Ishida et al. (2017) Takashi Ishida, Gang Niu, Weihua Hu, and Masashi Sugiyama. 2017\.  Learning from complementary labels.  _Advances in neural information processing systems_ 30 (2017).
  * Jøsang (2016) Audun Jøsang. 2016\.  _Subjective logic_. Vol. 3.  Springer.
  * Krizhevsky et al. (2009) Alex Krizhevsky, Geoffrey Hinton, et al. 2009\.  Learning multiple layers of features from tiny images.  (2009).
  * Lee et al. (2013) Dong-Hyun Lee et al. 2013\.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In _Workshop on challenges in representation learning, ICML_ , Vol. 3. 896.
  * Malinin and Gales (2018) Andrey Malinin and Mark Gales. 2018.  Predictive uncertainty estimation via prior networks.  _Advances in neural information processing systems_ 31 (2018).
  * Park et al. ([n. d.) ]parkactive Younghyun Park, Soyeong Kim, Wonjeong Choi, Dong-Jun Han, and Jaekyun Moon. [n. d.].  Active Learning for Object Detection with Evidential Deep Learning and Hierarchical Uncertainty Aggregation. In _International Conference on Learning Representations_.
  * Saito et al. (2021) Kuniaki Saito, Donghyun Kim, and Kate Saenko. 2021.  Openmatch: Open-set semi-supervised learning with open-set consistency regularization.  _Advances in neural information processing systems_ 34 (2021), 25956–25967.
  * Saito and Saenko (2021) Kuniaki Saito and Kate Saenko. 2021.  Ovanet: One-vs-all network for universal domain adaptation. In _Proceedings of the ieee/cvf international conference on computer vision_. 9000–9009.
  * Sensoy et al. (2018) Murat Sensoy, Lance Kaplan, and Melih Kandemir. 2018\.  Evidential deep learning to quantify classification uncertainty.  _Advances in neural information processing systems_ 31 (2018).
  * Sentz and Ferson (2002) Kari Sentz and Scott Ferson. 2002.  Combination of evidence in Dempster-Shafer theory.  (2002).
  * Sohn et al. (2020) Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. 2020.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  _Advances in neural information processing systems_ 33 (2020), 596–608.
  * Soleimany et al. (2021) Ava P Soleimany, Alexander Amini, Samuel Goldman, Daniela Rus, Sangeeta N Bhatia, and Connor W Coley. 2021\.  Evidential deep learning for guided molecular property prediction and discovery.  _ACS central science_ 7, 8 (2021), 1356–1367.
  * Szegedy et al. (2016) Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. 2016.  Rethinking the inception architecture for computer vision. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 2818–2826.
  * Vinyals et al. (2016) Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. 2016\.  Matching networks for one shot learning.  _Advances in neural information processing systems_ 29 (2016).
  * Wang et al. (2022b) Xudong Wang, Zhirong Wu, Long Lian, and Stella X Yu. 2022b.  Debiased learning from naturally imbalanced pseudo-labels. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 14647–14657.
  * Wang et al. (2022a) Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, Zhen Wu, and Jindong Wang. 2022a.  Freematch: Self-adaptive thresholding for semi-supervised learning.  _arXiv preprint arXiv:2205.07246_ (2022).
  * Wilson and Izmailov (2020) Andrew G Wilson and Pavel Izmailov. 2020.  Bayesian deep learning and a probabilistic perspective of generalization.  _Advances in neural information processing systems_ 33 (2020), 4697–4708.
  * Xu et al. (2021) Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. 2021\.  Dash: Semi-supervised learning with dynamic thresholding. In _International conference on machine learning_. PMLR, 11525–11536.
  * Yu et al. (2020) Qing Yu, Daiki Ikami, Go Irie, and Kiyoharu Aizawa. 2020\.  Multi-task curriculum framework for open-set semi-supervised learning. In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XII 16_. Springer, 438–454.
  * Zagoruyko and Komodakis (2016) Sergey Zagoruyko and Nikos Komodakis. 2016.  Wide residual networks.  _arXiv preprint arXiv:1605.07146_ (2016).
  * Zhang et al. (2021) Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. 2021.  Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling.  _Advances in neural information processing systems_ 34 (2021), 18408–18419.
  * Zhao et al. (2020) Xujiang Zhao, Feng Chen, Shu Hu, and Jin-Hee Cho. 2020\.  Uncertainty aware semi-supervised learning on graph data.  _Advances in neural information processing systems_ 33 (2020), 12827–12836.
  * Zheng et al. (2022) Mingkai Zheng, Shan You, Lang Huang, Fei Wang, Chen Qian, and Chang Xu. 2022\.  Simmatch: Semi-supervised learning with similarity matching. In _Proceedings of the IEEE conference on computer vision and pattern recognition_. 14471–14481.


# Keep It Simple: CNN Model Complexity Studies for Interference Classification
Tasks

Taiwo Oyedare⋆, Vijay K. Shah†, Daniel J. Jakubisin⋆‡, Jeffrey H. Reed⋆ ⋆
Bradley Dept. of Electrical and Computer Engineering, Virginia Tech,
Blacksburg, USA
† Dept. of Cybersecurity Engineering, George Mason University, Fairfax, USA
‡ Virginia Tech National Security Institute, Blacksburg, VA, USA
Email<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The growing number of devices using the wireless spectrum makes it important
to find ways to minimize interference and optimize the use of the spectrum.
Deep learning models, such as convolutional neural networks (CNNs), have been
widely utilized to identify, classify, or mitigate interference due to their
ability to learn from the data directly. However, there have been limited
research on the complexity of such deep learning models. The major focus of
deep learning-based wireless classification literature has been on improving
classification accuracy, often at the expense of model complexity. This may
not be practical for many wireless devices, such as, internet of things (IoT)
devices, which usually have very limited computational resources and cannot
handle very complex models. Thus, it becomes important to account for model
complexity when designing deep learning based models for interference
classification. To address this, we conduct an analysis of CNN based wireless
classification that explores the trade-off amongst dataset size, CNN model
complexity, and classification accuracy under various levels of classification
difficulty: namely, interference classification, heterogeneous transmitter
classification, and homogeneous transmitter classification. Our study, based
on three wireless datasets, shows that a simpler CNN model with fewer
parameters can perform just as well as a more complex model, providing
important insights into the use of CNNs in computationally constrained
applications.

###### Index Terms:

Interference Classification, Convolutional Neural Networks, Model Complexity.

##  I Introduction

The proliferation of internet of things (IoT), 5G devices, and other wireless
technologies has led to an increase in the number of wireless devices that
interfere with each other, either intentionally or unintentionally.
Classifying interference is essential for ensuring good communication quality
and adhering to spectrum sharing policies. The concept of wireless
interference, where a signal of interest is disrupted by another signal
sharing the same channel, has been studied extensively in the field of
wireless communication [1, 2, 3]. This type of interference can significantly
degrade the signal-to-noise-plus-interference ratio (SINR) and disrupt
communication between a transmitter and receiver.

Traditionally, many interference classification techniques have relied on
rule-based approaches that are not effective when multiple coexisting
technologies are in use. As a result, alternative classification algorithms
that use feature detection or extraction techniques, such as cyclostationary
feature detection [4], have been explored. However, these techniques require
domain expertise and can result in a complicated solution that is oftentimes
not scalable. In recent years, researchers have turned to deep learning
techniques to reduce the need for domain expertise [5, 6, 3]. Interference
suppression applications have also widely used deep learning [7].

Convolutional neural networks (CNNs), a model-free deep learning approach,
have been shown to be effective in various domains such as image
classification and natural language processing. CNNs have been used for
various classification tasks, including protocol/interference classification
[6, 5, 8, 3, 9, 10], transmitter classification [11], and modulation
classification [12]. In using CNNs or other deep learning models for
classification tasks in wireless communication applications, researchers need
to ensure that they have access to high quality datasets and efficient models.
While earlier works [6, 9, 5] have focused on improving the classification
performance of different deep learning models, the relationship between the
size of a dataset, the complexity of the CNN models used, and the difficulty
of classification is often overlooked or assumed. Even though many researchers
would tune hyper-parameters of their CNN during the training process, an
insight into other factors (such as filter size, number of nodes in the hidden
layer, etc) that affect model complexity has not been typically investigated.
In resource-constrained applications like the IoT, CNN models that are too
complex may not be feasible for classification in real-world situations. This
is because IoT is marked by its limited processing power and storage
capabilities, which can lead to challenges in terms of performance, security,
privacy, and reliability [13, 14]. It is essential to understand the processes
that lead to the selection of hyper-parameters in relation to model
complexity, the size of the dataset and the difficulty of classification.

A typical CNN architecture consists of a series of feed-forward layers that
apply convolutional filters and pooling operations, followed by fully-
connected layers that convert the 2D feature maps produced by the earlier
layers into 1D vectors for classification [15]. While CNNs do not require a
separate feature extraction step before being used, they can be time-consuming
and difficult to train from scratch because they require a large labeled
dataset for building and training the model [16]. The complexity of deep
learning models can be influenced by various factors, such as the number of
layers, number of filters, size of the filters, and number of nodes in the
hidden layer. Researchers in the field of deep learning often aim to improve
the performance of their models by hyper-parameter tuning and other
optimization techniques. In the literature, there has been a focus on
improving classification performance through these methods. Although it is
important to optimize deep learning models for performance, there has been
limited attention in the wireless communication literature on thoroughly
analyzing the factors that influence model complexity. This paper aims to fill
this gap by studying the relationship between model complexity, dataset size,
and classification difficulty in a thorough and empirical manner. To the best
of our knowledge, this is one of the first studies to examine this
relationship.

Our contributions are as follows:

  * •

We thoroughly analyze the complexity of three different CNN architectures
(simple, medium, and complex) in relation to dataset size and classification
difficulty.

  * •

We show, empirically, that the performance of a simple CNN model with fewer
parameters is comparable to that of a more complex CNN model. This is
important because resource-constrained devices, which have limited processing
power and storage capabilities, can benefit from using simpler CNN models.

##  II Overview of Classification Tasks

In this paper, our interference classification task is performed at levels of
difficulty as shown in Fig. 1. At the interference or protocol level, a CNN
can be used to classify different protocols or interference sources. At the
heterogeneous level, a CNN can be used to classify different transmitter
categories. Finally, at the homogeneous level, a CNN can be used to classify
specific emitters (homogeneous) categories, such as transmitters from the same
model or manufacturer. The outermost layer is the easiest classification while
the innermost layer is the most difficult.

Figure 1: Overview of Interference Classification

###  II-A Transmitter Categorization

Most transmitters have features that are peculiar to each of them. For
instance, when run at high power, the power amplifiers used in many wireless
devices sometimes display non-linearities [17][18]. These non-linearities can
be used to group the transmitters into different categories. We briefly
discuss the features of the transmitters used for our classification tasks.

####  II-A1 Category A (USRP 2921)

These transmitters utilize more reliable linear power amplifiers and finer
filters than other transmitter categories.

####  II-A2 Category B (USRP B200)

In comparison to those in category A, the components in the transmitters in
this category are less reliable. They were designed with low-cost
experimentation in mind. They employ a single chain of the $AD9364$, which is
frequently utilized to decrease hardware and software complexity.

####  II-A3 Category C (Adalm Pluto SDR)

This category’s transmitters are far less capable than the other two
categories’. Considering their small size and low cost, the Pluto SDR is
capable of a wide range of useful SDR applications.

###  II-B Levels of Classification

In this section we discuss the three levels of classification experiments
carried out for the model complexity study.

####  II-B1 Protocol or Interference Classification

The protocol or interference level of classification is easier than the other
two discussed in this section. This is because, there are enough
distinguishing features at this level. For instance, in the radio frequency
interference dataset used in this paper, there is a marked difference between
the three types of jammers described . For instance, the MCWI, which combines
the SoI and a two-tone CW is structurally different from the CI [19].

####  II-B2 Heterogeneous Classification

One transmitter from each of the three categories listed above in Section II-A
is used in this study. While two of the three transmitters (USRP 2921 and USRP
B200) were produced by the same company (National Instruments), all three are
of distinct models. In level of difficulty, this classification is easier than
homogeneous classification (discussed in Section II-B3) but harder than the
protocol or interference classification discussed in Section II-B1

####  II-B3 Homogeneous Classification

In this classification task, we seek to distinguish transmitters within
categories A, B, or C. The transmitters are identical (same manufacturer and
model), making this the most challenging classification problem of all three
levels. This is because the classification algorithm must identify slight
variations in transmitters which have the same architecture and hardware
components. The same OFDM waveform is sent by all of the devices,
significantly complicating categorization.

##  III Implementation Details of the CNN

###  III-A CNN Parameters for Model Complexity

The CNN algorithm utilized was a modified _Tensorflow_ CNN model that was used
to categorize handwritten digits from the MNIST dataset. We created three
levels of complexity for our CNN models by varying the number of nodes in the
hidden layer, the number of filters, and the size of the training dataset.
These factors all contribute to the number of parameters in the model, and as
a general rule, the complexity of a CNN algorithm increases with the number of
parameters. As a CNN architecture becomes more complex, it is generally
expected that the performance of the CNN algorithm will improve. However, one
potential downside is that the algorithm may begin to overfit to the training
data, resulting in a higher training accuracy compared to the test accuracy.

Our network contains three convolutional layers, max pooling layers and a
fully connected layer. A $3\times 3$ filter is applied to the input matrix by
the convolutional layers. Convolution operations are carried out in the
resulting sub-region to produce a single value in the respective output
feature map. To integrate non-linearities into the model, the scale-invariant
rectified linear unit (ReLU) activation function is applied to the feature map
values.

The data collected by the convolutional layer is down-sampled using the
pooling layer. We utilized the maximum pooling function of $2\times 2$. This
indicates that the most important features of the signals are kept while
others are deleted [20], which facilitates transmitter classification. Table I
lists specifics of the parameters for all three designs.

TABLE I: CNN Parameters for Model Complexity Parameters | Simple | Medium | Complex
---|---|---|---
Nodes in the hidden layer | 0.26k | 1.04k | 8k
Total number of parameters for the whole network | 6.1k | 48k | 276k
Filter size (No. of Filters) | 3*3(16) | 3*3(32)
Batch size | 16
Number of layers | 3
Strides | 2
Number of classes |  | 4 (for homogeneous devices)
---
3 (for heterogeneous devices)
Learning rate | 0.0002
Max pooling size | 2 * 2
Dropout Probability | 50%
Input matrix size | 38*100
Activation function | ReLU
Optimizer | Adam optimizer with cross entropy loss
Loss | Cross entropy loss
Training sizes | 16,64,256,1024,4096,8192,16384
Test sizes | 2k
TABLE II: CNN Parameters for Interference Classification Parameters | RESNet
---|---
Batch size | 64
Number of layers | 18
Learning rate | 0.0001
Maximum pooling dimension | $2\times 2$
Activation function | ReLU
Training/test size | 80%/20%
Dropout Probability | 80%
TABLE III: Comparison of Training Accuracy and Testing Accuracy for all Model Complexity and Device Category | Homogeneous Classification (USRP 2921) | Homogeneous Classification (USRP B200)
---|---|---
| Simple CNN | Medium CNN | Complex CNN | Simple CNN | Medium CNN | Complex CNN
| Dataset
---
Size
| Training
---
Acc
| Test
---
Acc
| Training
---
Acc
| Test
---
Acc
| Training
---
Acc
| Test
---
Acc
| Training
---
Acc
| Test
---
Acc
| Training
---
Acc
| Test
---
Acc
| Training
---
Acc
| Test
---
Acc
16 | 1.00 | 0.407 | 1.00 | 0.406 | 1.00 | 0.408 | 1.00 | 0.502 | 1.00 | 0.492 | 1.00 | 0.544
64 | 0.996 | 0.545 | 1.00 | 0.563 | 1.00 | 0.568 | 0.988 | 0.567 | 1.00 | 0.608 | 1.00 | 0.591
256 | 1.00 | 0.660 | 1.00 | 0.688 | 1.00 | 0.694 | 0.996 | 0.623 | 1.00 | 0.658 | 1.00 | 0.652
1024 | 0.971 | 0.744 | 1.00 | 0.784 | 1.00 | 0.788 | 0.955 | 0.663 | 1.00 | 0.709 | 1.00 | 0.722
4096 | 0.947 | 0.807 | 0.99 | 0.836 | 1.00 | 0.829 | 0.900 | 0.806 | 0.995 | 0.821 | 1.00 | 0.826
8192 | 0.899 | 0.838 | 0.986 | 0.856 | 0.99 | 0.842 | 0.864 | 0.830 | 0.984 | 0.837 | 0.99 | 0.858
16384 | 0.88 | 0.860 | 0.975 | 0.862 | 0.99 | 0.876 | 0.856 | 0.843 | 0.971 | 0.860 | 0.97 | 0.870
| Homogeneous Classification (Adalm Pluto SDR) | Heterogeneous Classification
| Simple CNN | Medium CNN | Complex CNN | Simple CNN | Medium CNN | Complex CNN
| Dataset
---
Size
| Training
---
Acc
| Test
---
Acc
| Training
---
Acc
| Test
---
Acc
| Training
---
Acc
| Test
---
Acc
| Training
---
Acc
| Test
---
Acc
| Training
---
Acc
| Test
---
Acc
| Training
---
Acc
| Test
---
Acc
16 | 0.859 | 0.369 | 1.00 | 0.437 | 1.00 | 0.407 | 0.96 | 0.762 | 1.00 | 0.842 | 1.00 | 0.816
64 | 0.988 | 0.512 | 1.00 | 0.548 | 1.00 | 0.450 | 1.00 | 0.836 | 1.00 | 0.844 | 1.00 | 0.841
256 | 0.939 | 0.641 | 1.00 | 0.633 | 1.00 | 0.579 | 0.997 | 0.883 | 1.00 | 0.885 | 1.00 | 0.895
1024 | 0.862 | 0.700 | 0.993 | 0.689 | 1.00 | 0.673 | 0.99 | 0.913 | 1.00 | 0.920 | 1.00 | 0.919
4096 | 0.820 | 0.718 | 0.959 | 0.699 | 0.99 | 0.706 | 0.97 | 0.937 | 0.99 | 0.954 | 1.00 | 0.952
8192 | 0.775 | 0.736 | 0.901 | 0.738 | 0.96 | 0.718 | 0.98 | 0.955 | 0.986 | 0.958 | 1.00 | 0.968
16384 | 0.772 | 0.747 | 0.865 | 0.753 | 0.85 | 0.754 | 0.98 | 0.97 | 0.975 | 0.970 | 1.00 | 0.972

###  III-B CNN Architecture for Interference Classification

In this section, we describe the architecture of the CNN model and the
training parameters used for the interference classification task. The CNN
model is a pre-trained ResNet18 model described in [21]. The architecture is
summarized in Table II. The model consists of many convolutional layers, two
fully-connected layers, and one output layer, in that order. Leaky ReLU
($\text{Leaky ReLU}(x)=\max\\{\alpha x,x\\}$, where $\alpha\in(0,1)$ is a
preset parameter). All convolutional layers and fully linked layers are
subjected to an activation function with $\alpha=0.2$. The resulting (output)
layer then has the softmax function applied to it. All convolutional layers
are subject to batch normalization [22], however the output layer and fully-
connected layers are excluded. Additionally, we use $\text{stride}=2$ in the
convolutional layers rather than $2\times 2$ pooling layers for down-sampling.
Such changes enhance performance and lower the variance of the results across
various training epochs. x is normalized as $x^{\prime}=x/x_{\max}$, where
$x_{\max}$ is the largest input value allowed in $x$. We utilize the Adam
optimizer with the suggested default values in [23]. The learning rate is
$1\times 10^{-4}$, and the batch size is $64$.

##  IV Experimental Setup

In this section, we describe the datasets used for the investigation of
dataset size, model complexity and level of classification. It is worth noting
that we only used publicly available datasets for interference classification,
while we used our own testbed to generate datasets for the model complexity
studies presented in this paper. This is because there are some limitation
with using public dataset, for instance, we were not able to control the types
of transmitters used to generate the interference. Also, the channel used for
transmission cannot be changed since we are using the dataset as is.

###  IV-A Dataset Generated For Model Complexity

The dataset generation process used for the assessment of model complexity is
similar to the one used in [11]. The only difference is that more training
data was added when compared to the work in [11]. We define the process of
creating a baseband waveform, transmitting it over a channel, and receiving it
as the transmitter-receiver chain. It is important to note that the details of
the transmitter-receiver chain can be found in [11].

####  IV-A1 Baseband Waveform Generation

The hardware devices transmit OFDM packets created using GNU Radio Companion
signal processing blocks. This is done by generating a stream of bits (0s and
1s) from a random source. We used an OFDM waveform that receives 10,000 data
bits from a random source, which are mapped to the OFDM waveform using a QPSK
modulation scheme with an FFT length of 512 and occupying the center 200
subcarriers with a cyclic prefix of 128.

####  IV-A2 RF Transmission

During transmission, GNU Radio and the transmitter hardware are connected
through the USRP hardware driver (UHD) hardware support package. All USRP
devices are managed and communicated with using a library called UHD. This is
done using a GNU Radio block that takes as inputs the sampling rate, RF
bandwidth, buffer size, center frequency, physical address of the device, and
attenuation. The OFDM waveform is then up-converted to 2.45 GHz as the
transmission center frequency and transmitted through the hardware’s antenna.
The sets of transmitters include four USRP 2921s, four USRP B200s, and four
Adalm Pluto software defined radios (SDRs).

####  IV-A3 RF Reception

On the receiver side, the signal is received when the antenna and the
transmitter are on the same channel. We assume that the receiver knows the
center frequency and bandwidth of the transmitter and corrects for frequency
offset at the receiver. The transmitted signal’s center frequency and sample
rate are stored in the UHD USRP Source block, which is used by a computer
running GNU Radio Companion to down-convert the signal to baseband frequency.
After being delivered to a low noise amplifier, the signal is separated into
in-phase and quadrature components at baseband. It is then low-pass filtered
and transferred to an analog-to-digital converter (ADC). When the ADC process
is completed, the digital samples are clocked into an FPGA. After being
digitally down-converted using precision frequency tuning, a series of filters
are used to decimate the FPGA image. The raw samples are then transmitted to a
host computer through the host interface (using an Ethernet gigabit link, in
this case) following decimation, made possible by the UHD. The complex samples
are collected by the host computer using GNU Radio, and the IQ data is saved
to a file and transferred to MATLAB for further processing.

####  IV-A4 Continuous Wavelet Transform (CWT) Signal Preprocessing

The features in the received signal that can be employed in the classification
process are highlighted by the CWT. For CWT, a $M\times N$ matrix of
coefficients is generated from $N$ samples for a $N$-length signal, where $M$
represents the number of scales [24]. The scales are determined automatically
using the wavelength’s energy distribution in frequency and time [24].

The CWT MATLAB function is used to pre-process the signals in order to enhance
key transmitter properties in the received signal. After the signal has been
transformed, the resultant 2D matrix of size $M\times N$ for each sample is
stacked together and sent to the CNN classifier as a three dimensional matrix.
The dimension of the output of the CWT process for this project is a $38\times
100$ matrix of coefficient whereas the input to the CWT is $2\times 128$.

###  IV-B Datasets for Interference Classification

####  IV-B1 Radio Frequency Interference Dataset

In this work, we use publicly available wireless interference datasets to
evaluate our approach. The RFI dataset used in this study was created by the
authors of [19]. They created the dataset by combining a signal of interest
(SOI) with three types of jammers (continuous-wave interference (CWI), multi-
continuous-wave interference (MCWI), and chirp interference (CI)) at different
signal-to-noise ratios (SNRs).

####  IV-B2 CRAWDAD Dataset

The second dataset used in this paper was obtained from the Community Resource
for Archiving Wireless Data at Dartmouth (CRAWDAD) website. This dataset,
provided by Schmidt et al. [3], includes traces of IEEE 802.11b/g, IEEE
802.15.4, and Bluetooth packet transmissions with different SNRs in the
baseband [25], as well as frequency offsets in the baseband [9]. There are a
total of 15 distinct classes, 10 of which are IEEE 802.15.1 devices, 3 of
which are IEEE 802.11 devices, and the remaining 2 are IEEE 802.15.4 devices.

(a) Confusion matrix for the RFI Dataset.(Classification Accuracy: $97.8\%$)

(b) Confusion Matrix for the CRAWDAD Dataset (Classification Accuracy: $80\%$)

Figure 2: Confusion Matrix for CNN classification of the RFI and CRAWDAD
Datasets.

##  V Results

In this section we discuss the results from the model complexity and
interference classification studies. Model complexity studies are important
since they help to understand the relationship between classification
performance and the number of parameters used by a deep learning algorithm
(CNN used in this paper). It is important to note that generating our own
dataset to study model complexity helps us to vary different conditions in the
data generating process which can not be done with publicly available
datasets.

###  V-A Model Complexity

This section examines the performance of the various CNN models across
different device categories. When the test accuracy is significantly lower
than the training accuracy, it suggests that the algorithm performs well on
the training set but poorly on the test set, indicating overfitting. In order
for a deep learning algorithm to perform well on new, unseen data, it is
important for the training and test accuracies to be similar. This indicates
that the algorithm has learned to generalize well.

Table III compares the training accuracy to the test accuracy for all the
classification tasks. At smaller dataset sizes, all the models tend to overfit
significantly. However, as the dataset size increases, the algorithms tend to
overcome overfitting issues. For most of the classification categories, the
overfitting problem is significantly reduced, with the difference between the
training and test accuracy being within $5\%$. Overfitting often occurs when
the CNN is complex, with multiple layers and many nodes in the hidden layer.
To prevent overfitting, regularization techniques such as dropout [26] and
early stopping [27] can be used.

Figure 3 demonstrates that, for most transmitters, the simple CNN performs
similarly to the medium and complex CNNs, despite having significantly fewer
parameters. In fact, Figure 3d shows that the performance of the simple CNN is
comparable to that of the complex CNNs for heterogeneous transmitters after a
dataset size of 100. While more complex or sophisticated CNN models may offer
some benefits, there must be a balance between the network’s generalization
ability and its complexity. _These findings suggest that, it is important for
the deep learning model to be both simple and robust, this is especially true
for resource-constrained applications_.

(a) USRP 2921

(b) USRP B200

(c) Pluto SDR

(d) Heterogeneous

Figure 3: Comparison of Test Accuracy for Homogeneous and Heterogeneous
Transmitter Classes

###  V-B Interference Classification

Accurate classification of interference sources is crucial for interference
suppression or mitigation. Using a pre-trained ResNet18 model, we achieved a
$97.8\%$ accuracy on the RFI dataset, as shown in Table II. Figure 2a shows
the confusion matrix for the interference classes, indicating excellent
performance on this comparatively easy classification task. For the CRAWDAD
dataset, the classification performance was about $80\%$, as shown in the
confusion matrix in Figure 2b, after 25 epochs.

As discussed in Section V-A, while more complex CNN models tend to perform
better in classification tasks, they are typically prone to overfitting.
Overfitting occurs when a deep learning model memorizes the training dataset,
leading to high training accuracy but low test or validation accuracy. This
can be seen in the results for the Pluto SDR in both Table III and Figure 3c.
These results further emphasize the importance of our findings from the model
complexity study discussed in Section V-A. As previously mentioned, a less
complex model can often perform as well as a more complex one without the risk
of overfitting. Therefore, it is important to use models that are neither too
complex nor too simple. The right level of complexity can be determined by
using just enough 2D convolution layers and filters to achieve good
performance, starting with a simpler model and gradually increasing complexity
as needed, this helps to prevent both under-fitting and overfitting issues.

##  VI Conclusion

In conclusion, this paper has thoroughly examined the use of CNN for
interference classification. Our results demonstrate that the CNN model is
capable of accurately classifying different interference sources, as shown in
the two datasets we used. Our study found that, while medium and complex CNN
classifiers performed slightly better than the simple classifier, the
difference in performance was not significant. This is an important finding
since resource-constrained devices can easily leverage the simpler models.
When designing deep learning models, the goal is to create models that can be
applied to new data. Simple models are more likely to be able to do this
because they are less prone to overfitting, which is a common issue with
complex models. Different classification tasks, such as homogeneous and
heterogeneous classification, may require different amounts of data and
different levels of model complexity. In our study, the training dataset size,
number of nodes in the hidden layer has had the greatest impact on CNN model
performance. While the size of the dataset is important, we also note that the
quality of the training dataset can also significantly impact the performance
of CNN classifiers.

## Acknowledgment

This work was supported in part by the U.S. Air Force Research Laboratory
(AFRL) under Grant FA8750-20-2-0504, in part by the Lockheed Martin
Corporation under Grant M16-005-RPP010, and in part by the National Science
Foundation under Grant CNS-1564148.

## References

  * [1] L. Qiu, Y. Zhang, F. Wang, M. K. Han, and R. Mahajan, “A general model of wireless interference,” in Proceedings of the 13th annual ACM international conference on Mobile computing and networking, pp. 171–182, 2007\.
  * [2] J. Bruno, “Interference reduction in wireless networks,” Computing Research Topics, Computing Sciences Department, Villanova University, 2007.
  * [3] M. Schmidt, D. Block, and U. Meier, “Wireless interference identification with convolutional neural networks,” in 2017 IEEE 15th International Conference on Industrial Informatics (INDIN), pp. 180–185, IEEE, 2017.
  * [4] K. Kim, I. A. Akbar, K. K. Bae, J.-S. Um, C. M. Spooner, and J. H. Reed, “Cyclostationary approaches to signal detection and classification in cognitive radio,” in 2007 2nd ieee international symposium on new frontiers in dynamic spectrum access networks, pp. 212–215, IEEE, 2007.
  * [5] J. Yu, M. Alhassoun, and R. M. Buehrer, “Interference classification using deep neural networks,” arXiv preprint arXiv:2002.00533, 2020.
  * [6] J. Kim, S. Lee, Y.-H. Kim, and S.-C. Kim, “Classification of interference signal for automotive radar systems with convolutional neural network,” IEEE Access, vol. 8, pp. 176717–176727, 2020.
  * [7] T. Oyedare, V. K. Shah, D. J. Jakubisin, and J. H. Reed, “Interference suppression using deep learning: Current approaches and open challenges,” IEEE Access, 2022.
  * [8] Y. Zhao, W.-C. Wong, H. K. Garg, T. Feng, Z. Zhang, and L. Tang, “Indoor position recognition and interference classification with a nested lstm network,” in 2019 IEEE Global Communications Conference (GLOBECOM), pp. 1–6, IEEE, 2019.
  * [9] S. Grunau, D. Block, and U. Meier, “Multi-label wireless interference identification with convolutional neural networks,” arXiv preprint arXiv:1804.04395, 2018.
  * [10] K. Sankhe, M. Belgiovine, F. Zhou, S. Riyaz, S. Ioannidis, and K. Chowdhury, “Oracle: Optimized radio classification through convolutional neural networks,” in IEEE INFOCOM 2019-IEEE Conference on Computer Communications, pp. 370–378, IEEE, 2019.
  * [11] T. Oyedare and J.-M. J. Park, “Estimating the required training dataset size for transmitter classification using deep learning,” in 2019 IEEE International Symposium on Dynamic Spectrum Access Networks (DySPAN), pp. 1–10, IEEE, 2019.
  * [12] T. J. O’Shea, J. Corgan, and T. C. Clancy, “Convolutional radio modulation recognition networks,” in International conference on engineering applications of neural networks, pp. 213–226, Springer, 2016.
  * [13] H. F. Atlam, R. J. Walters, and G. B. Wills, “Fog computing and the internet of things: A review,” big data and cognitive computing, vol. 2, no. 2, p. 10, 2018.
  * [14] F. Pereira, R. Correia, P. Pinho, S. I. Lopes, and N. B. Carvalho, “Challenges in resource-constrained iot devices: Energy and communication as critical success factors for future iot deployment,” Sensors, vol. 20, no. 22, p. 6420, 2020.
  * [15] D. Ravì, C. Wong, F. Deligianni, M. Berthelot, J. Andreu-Perez, B. Lo, and G.-Z. Yang, “Deep learning for health informatics,” IEEE journal of biomedical and health informatics, vol. 21, no. 1, pp. 4–21, 2016.
  * [16] H. Mohsen, E.-S. A. El-Dahshan, E.-S. M. El-Horbaty, and A.-B. M. Salem, “Classification using deep learning neural networks for brain tumors,” Future Computing and Informatics Journal, vol. 3, no. 1, pp. 68–71, 2018.
  * [17] F. Edalat, Effect of power amplifier nonlinearity on system performance metric, bit-error-rate (BER).  PhD thesis, Massachusetts Institute of Technology, 2003.
  * [18] A. C. Polak, S. Dolatshahi, and D. L. Goeckel, “Identifying wireless users via transmitter imperfections,” IEEE Journal on selected areas in communications, vol. 29, no. 7, pp. 1469–1479, 2011.
  * [19] S. Ujan, N. Navidi, and R. Jr Landry, “An efficient radio frequency interference (rfi) recognition and characterization using end-to-end transfer learning,” Applied Sciences, vol. 10, no. 19, p. 6885, 2020.
  * [20] W. Wang, Y. Ning, H. Rangwala, and N. Ramakrishnan, “A multiple instance learning framework for identifying key sentences and detecting events,” in Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pp. 509–518, 2016.
  * [21] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778, 2016.
  * [22] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” arXiv preprint arXiv:1502.03167, 2015.
  * [23] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  * [24] “CWT-based time-frequency analysis.” https://www.mathworks.com/help/wavelet/examples/cwt-based-time-frequency-analysis.html.
  * [25] M. Schmidt, D. Block, and U. Meier, “CRAWDAD dataset OWL/interference (v. 2019-02-12).” Downloaded from https://crawdad.org/owl/interference/20190212, Feb. 2019.
  * [26] L. Xie, J. Wang, Z. Wei, M. Wang, and Q. Tian, “Disturblabel: Regularizing CNN on the loss layer,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4753–4762, 2016.
  * [27] X. Ying, “An overview of overfitting and its solutions,” in Journal of physics: Conference series, vol. 1168, p. 022022, IOP Publishing, 2019.