# CDMAD: Class-Distribution-Mismatch-Aware Debiasing
for Class-Imbalanced Semi-Supervised Learning

Hyuck Lee Heeyoung Kim
Department of Industrial and Systems Engineering, KAIST
Daejeon 34141, Republic of Korea
{dlgur0921<EMAIL_ADDRESS>

###### Abstract

Pseudo-label-based semi-supervised learning (SSL) algorithms trained on a
class-imbalanced set face two cascading challenges: 1) Classifiers tend to be
biased towards majority classes, and 2) Biased pseudo-labels are used for
training. It is difficult to appropriately re-balance the classifiers in SSL
because the class distribution of an unlabeled set is often unknown and could
be mismatched with that of a labeled set. We propose a novel class-imbalanced
SSL algorithm called class-distribution-mismatch-aware debiasing (CDMAD). For
each iteration of training, CDMAD first assesses the classifier’s biased
degree towards each class by calculating the logits on an image without any
patterns (e.g., solid color image), which can be considered irrelevant to the
training set. CDMAD then refines biased pseudo-labels of the base SSL
algorithm by ensuring the classifier’s neutrality. CDMAD uses these refined
pseudo-labels during the training of the base SSL algorithm to improve the
quality of the representations. In the test phase, CDMAD similarly refines
biased class predictions on test samples. CDMAD can be seen as an extension of
post-hoc logit adjustment to address a challenge of incorporating the unknown
class distribution of the unlabeled set for re-balancing the biased classifier
under class distribution mismatch. CDMAD ensures Fisher consistency for the
balanced error. Extensive experiments verify the effectiveness of CDMAD.

##  1 Introduction

Classifiers trained on a class-imbalanced set suffer from being biased toward
the majority classes. Under semi-supervised learning (SSL) settings,
classifiers of pseudo-label-based algorithms tend to be further biased because
of the use of biased pseudo-labels for training. The use of biased pseudo-
labels also decreases the quality of representations. This problem becomes
more serious when the class distributions of the labeled and unlabeled sets
differ significantly. In fact, recent SSL algorithms, such as ReMixMatch [3]
and CoMatch [39], rely on the assumption that the class distribution of the
unlabeled set is the same as that of the labeled set and cannot consider a
potential class distribution mismatch between the labeled and unlabeled sets.

Recently, many class imbalanced SSL (CISSL) algorithms [59, 37, 14, 28, 45,
34] have been proposed. However, Wei et al. [59], Lee et al. [37], Fan et al.
[14] assumed that the class distribution of the unlabeled set is known and the
same as that of the labeled set, although the class distribution of the
unlabeled set can be unknown in practice (e.g., STL-10 [8]) and training sets
comprising labeled and unlabeled sets collected from different periods are
likely to have a class distribution mismatch. Kim et al. [28], Oh et al. [45],
Lai et al. [34] did not make an assumption of the same class distributions for
labeled and unlabeled sets in the main training stage. However, after the main
training stage, they additionally used the re-balancing technique of
Classifier Re-training (cRT) [27] or post-hoc logit-adjustment (LA) [42],
which were proposed for fully supervised class-imbalanced learning. When using
cRT for CISSL, there are disadvantages that the classifier cannot be learned
interactively with representations, and only the labeled set is used for
training the classifier [37]. Using LA for CISSL may not re-balance the
classifier to an appropriate degree when the class distribution of the
unlabeled set is unknown and differs from that of the labeled set, because LA
can not consider the unknown class distribution of the unlabeled set.

We propose a CISSL algorithm, class-distribution-mismatch-aware debiasing
(CDMAD), which effectively mitigates class imbalance in SSL even under severe
class distribution mismatch between labeled and unlabeled sets. The key idea
of CDMAD is to consider the classifier’s biased degree towards each class for
refining both the biased pseudo-labels of the base SSL algorithm and class
predictions on test samples. To measure the classifier’s biased degree, we
utilize the class prediction on an input that is reasonably assumed to be
irrelevant to the training set.

|
---|---
(a) FixMatch |  (b) ReMixMatch
Figure 1: Class probabilities on an image without any patterns.

In general, a trained classifier predicts a class of a new sample based on the
learned features. Therefore, for an image irrelevant to the learned features,
the predicted class probabilities are expected to be uniform across classes.
However, this may not be true when the training set is class-imbalanced
because the classifier tends to be biased towards the majority classes. Fig. 1
illustrates the class probabilities predicted on an image without any patterns
(white image) using FixMatch [49] and ReMixMatch [3], base SSL algorithms of
the recent CISSL studies, trained on CIFAR-10 under $\gamma_{l}=\gamma_{u}=1$
(class-balanced set), $\gamma_{l}=100$ and $\gamma_{u}=1$ (class-imbalanced
set), and $\gamma_{l}=\gamma_{u}=100$ (class-imbalanced set), where
$\gamma_{l}$ and $\gamma_{u}$ denote the class imbalanced ratios for the
labeled and unlabeled sets (formally defined in Sec. 3.1), respectively. The
classifiers trained on the class-imbalanced sets produced highly nonuniform
class probabilities for the white image, whereas they produced nearly uniform
class probabilities for the same input when trained on the class-balanced set.
Here, it may be reasonable to assume that the solid color image does not have
the features learned from the training set. Then, the class probabilities for
the solid color image can be thought of as predicted based solely on the
classifier’s biased degree towards each class, regardless of the learned
features.

Motivated by the above finding, CDMAD measures the classifier’s biased degree
by calculating logits on a solid color image for each iteration of training.
Then, CDMAD refines the biased pseudo-labels of a base SSL algorithm by
adjusting for the measured bias of the classifier in the logits for unlabeled
samples. The refined pseudo-labels are used to train the base SSL algorithm,
which leads to the mitigation of class imbalance and improved quality of the
representations. After training is completed, CDMAD similarly refines the
biased class predictions on test samples by adjusting for the measured bias of
the classifier in the logits for test samples. CDMAD can appropriately re-
balance the classifier even under severe class distribution mismatch between
labeled and unlabeled sets because the class distributions of both labeled and
unlabeled sets can be implicitly considered when measuring the classifier’s
biased degree. In Sec. 3.4, we analyze that CDMAD can be viewed as an
extension of LA, incorporating awareness of class distribution mismatch.
Similar to LA, CDMAD is Fisher consistent for minimizing the balanced error
[41].

Experimental results on four benchmark datasets verify that CDMAD outperforms
baseline CISSL algorithms in both scenarios where the class distributions of
the labeled and unlabeled sets either match or mismatch. Furthermore, through
qualitative analysis and an ablation study, we demonstrate the effectiveness
of each component of CDMAD. Unlike previous CISSL studies, CDMAD does not
require additional parameters or training stages in comparison to the base SSL
algorithm. Additionally, it can be implemented by simply adding a few lines of
code into the existing code of the base SSL algorithms as presented in
Appendix A. The code for the CDMAD is available at
https://github.com/LeeHyuck/CDMAD.

##  2 Related Works

CReST [59] uses unlabeled samples predicted as the minority classes more
frequently than those predicted as the majority classes for iterative self-
training. ABC [37] and CoSSL [14] use an auxiliary classifier and train the
classifier to be balanced. CoSSL generates pseudo-labels for base SSL
algorithms using the balanced classifier. These studies assume that the class
distribution of the unlabeled set is known and same as that of the labeled
set. DARP [28] and DASO [45] refine biased pseudo-labels by iteratively
solving a convex optimization problem and blending semantic pseudo-labels and
linear pseudo-labels, respectively. SAW [34] mitigates class imbalance using
smoothed reweighting based on the number of pseudo-labels belonging to each
class. These studies additionally use CIL techniques, such as cRT [27] and LA
[42], after the main training stage. Adsh [17] and InPL [63] use pseudo-labels
based on class-dependent confidence thresholds and energy score threshold,
respectively. DebiasPL [56] debiases pseudo-labels by mitigating the
classifier response bias based on counterfactual reasoning. UDAL [35] unifies
distribution alignment technique [3] and logit-adjusted loss [42] to
progressively mitigate class-imbalance. L2AC [54] trains a bias adaptive
classifier composed of a bias attractor and a linear classifier with bi-level
optimization. ACR [60] dynamically refines pseudo-labels using an adaptive
consistency regularizer that estimates the true class distribution of
unlabeled set.

##  3 Methodology

###  3.1 Problem setup

Suppose that we have a training set with labeled set
$\mathcal{X}=\left\\{\left(x_{n},y_{n}\right):n\in\left(1,\ldots,N\right)\right\\}$
and unlabeled set
$\mathcal{U}=\left\\{\left(u_{m}\right):m\in\left(1,\ldots,M\right)\right\\}$,
where $x_{n}\in\mathbb{R}^{d}$ and $y_{n}\in[C]=\left\\{1,\ldots,C\right\\}$
denote the $n$th labeled sample and corresponding label, respectively, and
$u_{m}\in\mathbb{R}^{d}$ denotes the $m$th unlabeled sample. We denote the
number of labeled and unlabeled samples of class $c$ as $N_{c}$ and $M_{c}$,
respectively, i.e., $\sum_{c=1}^{C}N_{c}=N$ and $\sum_{c=1}^{C}M_{c}=M$, where
$M_{c}$ is challenging to know in a realistic scenario. The $C$ classes are
sorted in descending order according to the cardinality of labeled samples,
i.e., $N_{1}\geq\cdots\geq N_{C}$. The ratio of the class imbalance of labeled
and unlabeled sets are $\gamma_{l}=\frac{N_{1}}{N_{C}}$ and
$\gamma_{u}=\frac{M_{1}}{M_{C}}$, respectively, where $\gamma_{l}\gg 1$ or
$\gamma_{u}\gg 1$ in the class-imbalanced training set. When $M_{c}$ is
unknown, $\gamma_{u}$ will be also unknown and can differ from $\gamma_{l}$.
That is, the class distribution of the unlabeled set can be mismatched with
that of the labeled set. For each iteration of training, we sample minibatches
$\mathcal{MX}=\left\\{\left(x^{m}_{b},y^{m}_{b}\right):b\in\left(1,\ldots,B\right)\right\\}\subset\mathcal{X}$
and $\mathcal{MU}=\left\\{\left(u^{m}_{b}\right):b\in\left(1,\ldots,\mu
B\right)\right\\}\subset\mathcal{U}$ from the training set, where $B$ denotes
the minibatch size and $\mu$ denotes the relative size of $\mathcal{MU}$ to
$\mathcal{MX}$. Using $\mathcal{MX}$ and $\mathcal{MU}$ for training, we aim
to learn a classifier
$f_{\theta}:\mathbb{R}^{d}\rightarrow\left\\{1,\ldots,C\right\\}$ that
effectively classifies samples in a test set
$\mathcal{X}^{test}=\left\\{\left(x^{test}_{k},y^{test}_{k}\right):k\in\left(1,\ldots,K\right)\right\\}$,
where $\theta$ denotes parameters of base SSL algorithm. We denote the output
logits of $f_{\theta}$ on an input as
$g_{\theta}\left(\cdot\right)\in\mathbb{R}^{C}$, i.e,
$f_{\theta}\left(\cdot\right)=\operatorname{arg\,max}_{c}g_{\theta}\left(\cdot\right)_{c}$,
where $\left(\cdot\right)_{c}$ denotes the $c$th element.

###  3.2 Base SSL algorithms

The proposed algorithm uses FixMatch [49] or ReMixMatch [3] as its base SSL
algorithm, following other CISSL studies. FixMatch and ReMixMatch use hard or
sharpened pseudo-labels for entropy minimization and strong data augmentation
techniques [13, 10] for consistency regularization. Specifically, FixMatch
first predicts the class probability of weakly augmented unlabeled data point
$\alpha\left(u^{m}_{b}\right)$ as
$q_{b}=P_{\theta}\left(y|\alpha\left(u^{m}_{b}\right)\right)$ and then
generates hard pseudo-label
$\hat{q_{b}}$=$\operatorname{arg\,max}_{c}\left(q_{b,c}\right)$, where
$P_{\theta}\left(y|\cdot\right)=\phi\left(g_{\theta}\left(\cdot\right)\right)$
for softmax activation function $\phi$. For consistency regularization,
FixMatch uses hard pseudo-label $\hat{q_{b}}$ only when
$\max_{c}\left(q_{b,c}\right)\geq\tau$, where $\tau$ denotes a predefined
confidence threshold, to improve the quality of the pseudo-labels used for
training.

ReMixMatch similarly produces $q_{b}$ and aligns the distribution of $q_{b}$
to the class distribution of the labeled set $P_{l}\left(y\right)$ as
$\tilde{q_{b}}=Normalize\left(q_{b}\times
P_{l}\left(y\right)/q\left(y\right)\right)$, where
$Normalize\left(x\right)_{i}=x_{i}/\sum_{j}x_{j}$ and $q\left(y\right)$
denotes the moving average of the class probabilities predicted over the last
128 unlabeled minibatches. Then, ReMixMatch sharpens the pseudo-label as
$\bar{q_{b}}=Normalize\left(\tilde{q_{b}}^{1/T}\right)$, where $1/T$ is the
sharpening temperature, $0<T<1$. With the sharpened pseudo-label
$\bar{q_{b}}$, ReMixMatch conducts consistency regularization by encouraging
the class prediction on $\mathcal{A}\left(u^{m}_{b}\right)$ to be consistent
with $\bar{q_{b}}$. ReMixMatch also conducts Mixup regularization and self-
supervised learning by rotating unlabeled samples [31]. Data augmentation
techniques $\alpha\left(\cdot\right)$ and $\mathcal{A}\left(\cdot\right)$ are
described in detail in Appendix C. We express the training losses of FixMatch
$loss_{F}$ and ReMixMatch $loss_{R}$ on $\mathcal{MX}$ and $\mathcal{MU}$ as:

$\displaystyle
loss_{F}\left(\mathcal{MX},\mathcal{MU},\hat{q},\tau;\theta\right),$ (1)
$\displaystyle loss_{R}\left(\mathcal{MX},\mathcal{MU},\bar{q};\theta\right),$
(2)

where $\hat{q}$ and $\bar{q}$ are concatenations of $\hat{q_{b}}$ and
$\bar{q_{b}}$, $b=1,\ldots,$ $\mu B$, respectively. The losses are detailed in
Appendix D.

The proposed algorithm uses FixMatch or ReMixMatch as its base SSL algorithm
with some modifications as follows: 1) The proposed algorithm does not use
hard or sharpened pseudo-labels because entropy minimization of class
predictions may cause the classifier to be biased towards certain classes
[37]. 2) The proposed algorithm does not use confidence threshold $\tau$ for
FixMatch, enabling the utilization of all unlabeled samples. A potential
limitation of utilizing inaccurate pseudo-labels can be alleviated by refining
them, as discussed in Sec. 3.3. 3) The proposed algorithm does not employ the
distribution alignment technique for ReMixMatch when the class distribution of
the unlabeled set is unknown. This is because the labeled and unlabeled sets
can potentially have different class distributions while the distribution
alignment technique aligns the distribution of pseudo-labels with the class
distribution of the labeled set. This modification helps prevent the
generation of low-quality pseudo-labels in situations where there is a severe
class distribution mismatch between the labeled and unlabeled sets, as
discussed in Sec. 4.2.

###  3.3 CDMAD

Refinement of pseudo-labels during training

To refine a pseudo-label $q_{b}$ generated by FixMatch or ReMixMatch, CDMAD
first calculates the logits on a weakly augmented unlabeled sample,
$g_{\theta}\left(\alpha\left(u_{b}^{m}\right)\right)$, and the logits on an
image without any patterns $g_{\theta}\left(\mathcal{I}\right)$, where
$\mathcal{I}$ denotes an image without any patterns (solid color image). The
logits on a solid color image $g_{\theta}\left(I\right)$ is considered the
classifier’s biased degree towards each class regardless of the learned
features, as discussed in Sec. 1. Then, CDMAD adjusts for the classifier’s
biased degree, $g_{\theta}\left(\mathcal{I}\right)$, in the logits
$g_{\theta}\left(\alpha\left(u_{b}^{m}\right)\right)$, by simple subtraction
as follows:

$g_{\theta}^{*}\left(\alpha\left(u_{b}^{m}\right)\right)=g_{\theta}\left(\alpha\left(u_{b}^{m}\right)\right)-g_{\theta}\left(\mathcal{I}\right),$
(3)

where $g_{\theta}^{*}\left(\cdot\right)$ denotes the refined logits, which are
considered to be calculated based only on the learned features. With
$g_{\theta}^{*}\left(\alpha\left(u_{b}^{m}\right)\right)$, the refined pseudo-
label $q_{b}^{*}$ is obtained as:

$q_{b}^{*}=\phi\left(g_{\theta}^{*}\left(\alpha\left(u_{b}^{m}\right)\right)\right).$
(4)

As noted in Sec. 3.2, CDMAD does not use the distribution alignment technique
for ReMixMatch. Instead, CDMAD adds the supervised loss for weakly augmented
labeled sample $Sup(\mathcal{MX};\theta)$ into the training loss of ReMixMatch
to enhance the classifier’s familiarity with labeled samples. This can
effectively improve the quality of pseudo-labels when the class distributions
of labeled and unlabeled sets mismatch, as discussed in Sec. 4.2. The training
losses for FixMatch and ReMixMatch with CDMAD, denoted by $loss_{F}^{*}$ and
$loss_{R}^{*}$, respectively, are expressed as:

$loss_{F}^{*}=loss_{F}\left(\mathcal{MX},\mathcal{MU},q^{*},0;\theta\right),$
(5)
$loss_{R}^{*}=loss_{R}\left(\mathcal{MX},\mathcal{MU},q^{*};\theta\right)+Sup(\mathcal{MX};\theta),$
(6)

where $loss_{F}$ and $loss_{R}$ are from Eq. 1 and Eq. 2, and $q^{*}$ is the
concatenation of the $q^{*}_{b}$. Fig. 2 illustrates the pseudo-label
refinement process. By using the refined pseudo-labels during the training of
the base SSL algorithm, the quality of representations is improved.

Figure 2: Pseudo-label refinement process using CDMAD.

CDMAD can effectively refine the pseudo-labels even under severe class
distribution mismatch between the labeled and unlabeled sets because
classifier’s biased degree $g_{\theta}\left(\mathcal{I}\right)$ is affected by
the class distributions of both sets. Unlike the previous CISSL studies, CDMAD
does not require additional parameters for an auxiliary classifier or
additional training stages. CDMAD can be implemented by adding a few lines of
code as presented in Appendix A.

Refinement of biased class predictions during testing

Even with all biased pseudo-labels perfectly refined during the training
process, biased predictions may still be produced for the test samples because
the training set is class-imbalanced. To refine biased class predictions on
test samples, CDMAD also adjusts the biased logits on $x^{test}_{k}$, for
$k=1,\ldots,K$. CDMAD first calculates the classifier’s biased degree
$g_{\theta}\left(\mathcal{I}\right)$. Then, similar to Eq. 5, the logits for
test samples, $g_{\theta}\left(x^{test}_{k}\right)$, for $k=1,\ldots,K$, are
adjusted as:

$g_{\theta}^{*}\left(x^{test}_{k}\right)=g_{\theta}\left(x^{test}_{k}\right)-g_{\theta}\left(\mathcal{I}\right).$
(7)

With the adjusted logits $g_{\theta}^{*}\left(x^{test}_{k}\right)$, the
refined class prediction $f_{\theta}^{*}\left(x^{test}_{k}\right)$ is obtained
as follows:

$\displaystyle
f_{\theta}^{*}\left(x^{test}_{k}\right)=\operatorname{arg\,max}_{c}g_{\theta}^{*}\left(x^{test}_{k}\right)_{c}$
(8)
$\displaystyle=\operatorname{arg\,max}_{y\in[C]}P_{\theta}\left(y|x^{test}_{k}\right)/P_{\theta}\left(y|\mathcal{I}\right).$

We illustrate the test process in Appendix E and pseudo code of the proposed
algorithm in Appendix F.

###  3.4 CDMAD as a CISSL extension of post-hoc logit-adjustment (LA)

CDMAD can be viewed as a CISSL extension of LA [42] to take into account class
distribution mismatch between labeled and unlabeled sets, where LA was
originally introduced to re-balance a biased classifier in CIL. To re-balance
the classifier, LA post-adjusts the logits on test samples
$g_{\theta}\left(x^{test}_{k}\right)$ by simply subtracting the log of the
estimate of the underlying class prior $P(y)$, denoted by $\pi$ (e.g., each
class frequency on the training set), as follows:

$g_{\theta}^{*}\left(x^{test}_{k}\right)=g_{\theta}\left(x^{test}_{k}\right)-\log\pi.$
(9)

The adjustment in Eq. 9 was proven to be Fisher consistent for minimizing the
balanced error rate (BER),

$\mbox{BER}\left(f^{*}_{\theta}\right)=\frac{1}{C}\sum_{y\in[C]}P_{x|y}\left(y\neq
f_{\theta}^{*}\left(x\right)\right).\vspace{-0.05in}$ (10)

In addition, empirical evidence has demonstrated that LA can effectively
enhance classification performance across various class-imbalanced learning
scenarios.

However, in CISSL, where the class distribution of the unlabeled set is
unknown and may substantially differ from that of the labeled set, LA may
result in the classifier being re-balanced to an inappropriate degree, thereby
leading to a decrease in classification performance. This limitation arises
from the challenge of incorporating the unknown class distribution of the
unlabeled set for re-balancing. Specifically, the class prior $P(y)$, which is
often approximated as the class distribution of the labeled set
$P_{l}\left(y\right)$, cannot consider the class distribution of the unlabeled
set.

By comparing Eq. 7 and Eq. 9, we can observe that CDMAD shares a similar form
with LA. Specifically, given that
$g_{\theta}\left(\mathcal{I}\right)+constant=\log
P_{\theta}\left(y|\mathcal{I}\right)$, CDMAD can be seen as replacing the
class frequencies $\pi$ in Eq. 9 by $P_{\theta}\left(y\right|\mathcal{I})$,
which can be considered an estimate of the classifier’s prior $P_{\theta}(y)$.
(With this interpretation, CDMAD can be viewed as allowing the classifier to
predict class probabilities based solely on the given input, without being
affected by the classifier’s prior $P_{\theta}(y)$.) This replacement allows
CDMAD to implicitly incorporate the class distributions of both labeled and
unlabeled sets, facilitating its awareness of class distribution mismatch
between the two sets, as we will further discuss in Sec. 4.3. Furthermore,
whereas LA is solely employed to refine biased class predictions on the test
set, CDMAD is also employed to refine the biased pseudo-labels. Similar to LA,
CDMAD ensures Fisher consistency for the balanced error.

######  Proposition 1.

Given a solid color image $\mathcal{I}$ independent of class labels $y$, the
refinement by CDMAD in Eq. 8 is Fisher consistent for minimizing the BER in
Eq. 10.

###### Proof.

Due to the universal approximation theorem [20, 68],
$P_{\theta}\left(y\right)$ becomes $P\left(y\right)$, and
$P_{\theta}\left(y|I\right)$ becomes $P\left(y|I\right)$, under the population
setting. By the assumption of Proposition 1, i.e.,
$P\left(y\right)=P\left(y|I\right)$, it follows that
$P_{\theta}\left(y\right)=P_{\theta}\left(y|I\right)$. The refined class
prediction for input $x$,
$f_{\theta}^{*}\left(x\right)=\operatorname{arg\,max}_{y\in[C]}P_{\theta}\left(y|x\right)/P_{\theta}\left(y|\mathcal{I}\right)=\operatorname{arg\,max}_{y\in[C]}P_{\theta}\left(y|x\right)$
$/P_{\theta}\left(y\right)$. If the network is trained with the entire
population, $P_{\theta}\left(y|x\right)$ becomes $P\left(y|x\right)$, and
$P_{\theta}\left(y\right)=\int P_{\theta}\left(y|x\right)P\left(x\right)dx$
becomes $\int P\left(y|x\right)P\left(x\right)dx=P\left(y\right)$, due to the
universal approximation theorem. Then, $f_{\theta}^{*}\left(x\right)$
$=\operatorname{arg\,max}_{y\in[C]}P_{\theta}\left(y|x\right)/P_{\theta}\left(y\right)$
becomes $\operatorname{arg\,max}_{y\in[C]}$
$P\left(y|x\right)/P\left(y\right)=\operatorname{arg\,max}_{y\in[C]}P\left(x|y\right)$,
which minimizes the BER [42, 9]. ∎

Fisher consistency is a desirable property for an estimator and implies that
in the entire population setting, optimizing the estimator yields the best
result. In our case, we prove that in the population setting, CDMAD minimizes
the BER. However, in practice, the entire population is not available, and
thus the theory cannot be directly applied. Nevertheless, Proposition 1 makes
us to be optimistic about that the algorithm is learned in the direction to
minimize the BER.

##  4 Experiments

###  4.1 Experimental setup

We conducted experiments on CIFAR-10-LT, CIFAR-100-LT [12], STL-10-LT [28],
and Small-ImageNet-127 [14] under the settings considered in previous studies
[34, 14]. We used the balanced accuracy (bACC) [21] and geometric mean (GM)
[33] to evaluate the classification performance on CIFAR-10-LT and STL-10-LT
and only bACC to evaluate classification performance on CIFAR-100-LT and
Small-ImageNet-127, following Fan et al. [14]. For CIFAR-10-LT, CIFAR-100-LT,
and STL-10-LT, we repeated the experiments three times and report the average
and standard error for the performance measures. We used a white image to
measure the $g_{\theta}\left(y\right)$. Performance measures, description of
datasets and experimental setup, and baseline algorithms are detailed in
Appendices G, H, and I, respectively.

###  4.2 Experimental results

Table 1: Comparison of bACC/GM on CIFAR-10-LT

CIFAR-10-LT ($\gamma=\gamma_{l}=\gamma_{u}$, $\gamma_{u}$ is assumed to be
known)
---
Algorithm | $\gamma=50$ | $\gamma=100$ | $\gamma=150$
Vanilla |  $65.2$$\pm 0.05$ / $61.1$$\pm 0.09$ |  $58.8$$\pm 0.13$ / $58.2$$\pm 0.11$ |  $55.6$$\pm 0.43$ / $44.0$$\pm 0.98$
Re-sampling |  $64.3$$\pm 0.48$ / $60.6$$\pm 0.67$ |  $55.8$$\pm 0.47$ / $45.1$$\pm 0.30$ |  $52.2$$\pm 0.05$ / $38.2$$\pm 1.49$
LDAM-DRW |  $68.9$$\pm 0.07$ / $67.0$$\pm 0.08$ |  $62.8$$\pm 0.17$ / $58.9$$\pm 0.60$ |  $57.9$$\pm 0.20$ / $50.4$$\pm 0.30$
cRT |  $67.8$$\pm 0.13$ / $66.3$$\pm 0.15$ |  $63.2$$\pm 0.45$ / $59.9$$\pm 0.40$ |  $59.3$$\pm 0.10$ / $54.6$$\pm 0.72$
FixMatch |  $79.2$$\pm 0.33$ / $77.8$$\pm 0.36$ |  $71.5$$\pm 0.72$ / $66.8$$\pm 1.51$ |  $68.4$$\pm 0.15$ / $59.9$$\pm 0.43$
/++DARP+cRT |  $85.8$$\pm 0.43$ / $85.6$$\pm 0.56$ |  $82.4$$\pm 0.26$ / $81.8$$\pm 0.17$ |  $79.6$$\pm 0.42$ / $78.9$$\pm 0.35$
/+CReST+LA |  $85.6$$\pm 0.36$ / $81.9$$\pm 0.45$ |  $81.2$$\pm 0.70$ / $74.5$$\pm 0.99$ |  $71.9$$\pm 2.24$ / $64.4$$\pm 1.75$
/+ABC |  $85.6$$\pm 0.26$ / $85.2$$\pm 0.29$ |  $81.1$$\pm 1.14$ / $80.3$$\pm 1.29$ |  $77.3$$\pm 1.25$ / $75.6$$\pm 1.65$
/+CoSSL |  $86.8$$\pm 0.30$ / $86.6$$\pm 0.25$ |  $83.2$$\pm 0.49$ / $82.7$$\pm 0.60$ |  $80.3$$\pm 0.55$ / $79.6$$\pm 0.57$
/+SAW+LA |  $86.2$$\pm 0.15$ / $83.9$$\pm 0.35$ |  $80.7$$\pm 0.15$ / $77.5$$\pm 0.21$ |  $73.7$$\pm 0.06$ / $71.2$$\pm 0.17$
/+Adsh |  $83.4$$\pm 0.06$/ - |  $76.5$$\pm 0.35$/ - |  $71.5$$\pm 0.30$/ -
/+DebiasPL | -/ - |  $80.6$$\pm 0.50$/ - | -/ -
/+UDAL |  $86.5$$\pm 0.29$ / - |  $81.4$$\pm 0.39$ / - |  $77.9$$\pm 0.33$ / -
/+L2AC | -/ - |  $82.1$$\pm 0.57$ / $81.5$$\pm 0.64$ |  $77.6$$\pm 0.53$ / $75.8$$\pm 0.71$
/+CDMAD | 87.3$\pm 0.12$ / 87.0$\pm 0.15$ | 83.6$\pm 0.46$ / 83.1$\pm 0.57$ | 80.8$\pm 0.86$ / 79.9$\pm 1.07$
ReMixMatch |  $81.5$$\pm 0.26$ / $80.2$$\pm 0.32$ |  $73.8$$\pm 0.38$ / $69.5$$\pm 0.84$ |  $69.9$$\pm 0.47$ / $62.5$$\pm 0.35$
/+DARP+cRT |  $87.3$$\pm 0.61$ / $87.0$$\pm 0.11$ |  $83.5$$\pm 0.07$ / $83.1$$\pm 0.09$ |  $79.7$$\pm 0.54$ / $78.9$$\pm 0.49$
/+CReST+LA |  $84.2$$\pm 0.11$ / - |  $81.3$$\pm 0.34$ / - |  $79.2$$\pm 0.31$ / -
/+ABC |  $87.9$$\pm 0.47$ / $87.6$$\pm 0.51$ |  $84.5$$\pm 0.32$ / $84.1$$\pm 0.36$ |  $80.5$$\pm 1.18$ / $79.5$$\pm 1.36$
/+CoSSL |  $87.7$$\pm 0.21$ / $87.6$$\pm 0.25$ |  $84.1$$\pm 0.56$ / $83.7$$\pm 0.66$ |  $81.3$$\pm 0.83$ / $80.5$$\pm 0.76$
/+SAW+cRT |  $87.6$$\pm 0.21$ / $87.4$$\pm 0.26$ | 85.4$\pm 0.32$ / $83.9$$\pm 0.21$ |  $79.9$$\pm 0.15$ / $79.9$$\pm 0.12$
/+CDMAD | 88.3$\pm 0.35$ / 88.1$\pm 0.35$ | 85.5$\pm 0.46$ / 85.3$\pm 0.44$ | 82.5$\pm 0.23$ / 82.0$\pm 0.30$

Table 2: Comparison of bACC/GM on CIFAR-10-LT and STL-10-LT under
$\gamma_{l}\neq\gamma_{u}$ ($\gamma_{u}$ is assumed to be unknown).
ReMixMatch* denotes ReMixMatch with the estimated class distribution of the
unlabeled set [28].

|  | CIFAR-10-LT ($\gamma_{l}=100$, $\gamma_{u}$ is assumed to be unknown) | STL-10-LT ($\gamma_{u}=$Unknown)
---|---|---|---
Algorithm | $\gamma_{u}=1$ | $\gamma_{u}=50$ | $\gamma_{u}=150$ | $\gamma_{l}=10$ | $\gamma_{l}=20$
FixMatch |  $68.9$$\pm 1.95$ / $42.8$$\pm 8.11$ |  $73.9$$\pm 0.25$ / $70.5$$\pm 0.52$ |  $69.6$$\pm 0.60$ / $62.6$$\pm 1.11$ |  $72.9$$\pm 0.09$/ $69.6$$\pm 0.01$ |  $63.4$$\pm 0.21$/ $52.6$$\pm 0.09$
FixMatch+DARP |  $85.4$$\pm 0.55$ / $85.0$$\pm 0.65$ |  $77.3$$\pm 0.17$ / $75.5$$\pm 0.21$ |  $72.9$$\pm 0.24$ / $69.5$$\pm 0.18$ |  $77.8$$\pm 0.33$/ $76.5$$\pm 0.40$ |  $69.9$$\pm 1.77$/ $65.4$$\pm 3.07$
FixMatch+DARP+LA |  $86.6$$\pm 1.11$ / $86.2$$\pm 1.15$ |  $82.3$$\pm 0.32$ / $81.5$$\pm 0.29$ |  $78.9$$\pm 0.23$ / $77.7$$\pm 0.06$ |  $78.6$$\pm 0.30$/ $77.4$$\pm 0.40$ |  $71.9$$\pm 0.49$/ $68.7$$\pm 0.51$
FixMatch+DARP+cRT |  $87.0$$\pm 0.70$ / $86.8$$\pm 0.67$ |  $82.7$$\pm 0.21$ / $82.3$$\pm 0.25$ |  $80.7$$\pm 0.44$ / $80.2$$\pm 0.61$ |  $79.3$$\pm 0.23$/ $78.7$$\pm 0.21$ |  $74.1$$\pm 0.61$/ $73.1$$\pm 1.21$
FixMatch+ABC |  $82.7$$\pm 0.49$ / $81.9$$\pm 0.68$ |  $82.7$$\pm 0.64$ / $82.0$$\pm 0.76$ |  $78.4$$\pm 0.87$ / $77.2$$\pm 1.07$ |  $79.1$$\pm 0.46$/ $78.1$$\pm 0.57$ |  $73.8$$\pm 0.15$/ $72.1$$\pm 0.15$
FixMatch+SAW |  $81.2$$\pm 0.68$ / $80.2$$\pm 0.91$ |  $79.8$$\pm 0.25$ / $79.1$$\pm 0.32$ |  $74.5$$\pm 0.97$/ $72.5$$\pm 1.37$ | -/- | -/-
FixMatch+SAW+LA |  $84.5$$\pm 0.68$ / $84.1$$\pm 0.78$ |  $82.9$$\pm 0.38$ / $82.6$$\pm 0.38$ |  $79.1$$\pm 0.81$ / $78.6$$\pm 0.91$ | -/- | -/-
FixMatch+SAW+cRT |  $84.6$$\pm 0.23$ / $84.4$$\pm 0.26$ |  $81.6$$\pm 0.38$ / $81.3$$\pm 0.32$ |  $77.6$$\pm 0.40$ / $77.1$$\pm 0.41$ | -/- | -/-
FixMatch+CDMAD | 87.5$\pm 0.46$ / 87.1$\pm 0.50$ | 85.7$\pm 0.36$ / 85.3$\pm 0.38$ | 82.3$\pm 0.23$ / 81.8$\pm 0.29$ | 79.9$\pm 0.23$/ 78.9$\pm 0.38$ | 75.2$\pm 0.40$/ 73.5$\pm 0.31$
ReMixMatch |  $48.3$$\pm 0.14$ / $19.5$$\pm 0.85$ |  $75.1$$\pm 0.43$ / $71.9$$\pm 0.77$ |  $72.5$$\pm 0.10$ / $68.2$$\pm 0.32$ |  $67.8$$\pm 0.45$/ $61.1$$\pm 0.92$ |  $60.1$$\pm 1.18$/ $44.9$$\pm 1.52$
ReMixMatch* |  $85.0$$\pm 1.35$ / $84.3$$\pm 1.55$ |  $77.0$$\pm 0.12$ / $74.7$$\pm 0.04$ |  $72.8$$\pm 0.10$ / $68.8$$\pm 0.21$ |  $76.7$$\pm 0.15$ / $73.9$$\pm 0.32$ |  $67.7$$\pm 0.46$ / $60.3$$\pm 0.76$
ReMixMatch*+DARP |  $86.9$$\pm 0.10$ / $86.4$$\pm 0.15$ |  $77.4$$\pm 0.22$ / $75.0$$\pm 0.25$ |  $73.2$$\pm 0.11$ / $69.2$$\pm 0.31$ |  $79.4$$\pm 0.07$/ $78.2$$\pm 0.10$ |  $70.9$$\pm 0.44$/ $67.0$$\pm 1.62$
ReMixMatch*+DARP+LA |  $81.8$$\pm 0.45$ / $80.9$$\pm 0.40$ |  $83.9$$\pm 0.42$ / $83.4$$\pm 0.45$ |  $81.1$$\pm 0.20$ / $80.3$$\pm 0.26$ |  $80.6$$\pm 0.45$/ $79.6$$\pm 0.55$ |  $76.8$$\pm 0.60$/ $74.8$$\pm 0.68$
ReMixMatch*+DARP+cRT |  $88.7$$\pm 0.25$ / $88.5$$\pm 0.25$ |  $83.5$$\pm 0.53$ / $83.1$$\pm 0.51$ |  $80.9$$\pm 0.25$ / $80.3$$\pm 0.31$ |  $80.9$$\pm 0.53$/ $80.0$$\pm 0.46$ |  $76.7$$\pm 0.50$/ $74.9$$\pm 0.70$
ReMixMatch+ABC |  $76.4$$\pm 5.34$ / $74.8$$\pm 6.05$ |  $85.2$$\pm 0.20$ / $84.7$$\pm 0.25$ |  $80.4$$\pm 0.40$ / $80.0$$\pm 0.44$ |  $76.8$$\pm 0.52$/ $74.8$$\pm 0.64$ |  $71.2$$\pm 1.37$/ $67.4$$\pm 1.89$
ReMixMatch*+SAW |  $87.0$$\pm 0.75$ / $86.4$$\pm 0.85$ |  $80.6$$\pm 1.57$ / $79.2$$\pm 2.19$ |  $77.6$$\pm 0.76$ / $76.0$$\pm 0.93$ | -/- | -/-
ReMixMatch*+SAW+LA |  $74.2$$\pm 1.49$ / $65.1$$\pm 2.36$ |  $84.8$$\pm 1.07$ / $82.4$$\pm 2.32$ |  $81.3$$\pm 2.42$ / $80.9$$\pm 2.47$ | -/- | -/-
ReMixMatch*+SAW+cRT |  $88.8$$\pm 0.79$ / $88.6$$\pm 0.83$ |  $84.5$$\pm 0.78$ / $83.6$$\pm 1.27$ |  $82.4$$\pm 0.10$ / $82.0$$\pm 0.10$ | -/- | -/-
ReMixMatch+CDMAD | 89.9$\pm 0.45$ / 89.6$\pm 0.46$ | 86.9$\pm 0.21$ / 86.7$\pm 0.17$ | 83.1$\pm 0.46$ / 82.7$\pm 0.50$ | 83.0$\pm 0.38$/ 82.1$\pm 0.35$ | 81.9$\pm 0.32$/ 80.9$\pm 0.44$

Tab. 1 summarizes bACC and GM of the baseline algorithms and proposed
algorithm on CIFAR-10-LT when $\gamma_{u}$ is assumed to be known and equal to
$\gamma_{l}$. We can first observe that the vanilla algorithm (Deep CNN
trained with cross-entropy loss) performed the worst. CIL (Re-sampling, LDAM-
DRW, and cRT) mitigated class imbalance but did not significantly improve the
classification performance compared to the vanilla algorithm. These results
demonstrate the importance of using the unlabeled set. Compared to the vanilla
algorithm, SSL algorithms (FixMatch and ReMixMatch) significantly improved the
classification performance. However, their lower performance than that of the
CISSL algorithms highlights the importance of mitigating class imbalance. By
mitigating class imbalance and leveraging unlabeled data, CISSL algorithms
achieved higher performance than the other algorithms. Overall, the proposed
algorithm outperformed all other algorithms. This may be because CDMAD
effectively refined biased pseudo-labels and class predictions on the test set
by considering the classifier’s biased degree.

Table 3: Comparison of bACC/GM under $\gamma_{l}=\gamma_{u}=100$ (reversed).

CIFAR-10-LT, $\gamma_{l}=100$, $\gamma_{u}=100$ (reversed)
---
Algorithm | ABC | SAW | SAW+LA | SAW+cRT | CDMAD
FixMatch+ | 69.5/66.8 | 72.3/68.7 | 74.1/72.0 | 75.5/73.9 |  77.1/75.4
ReMixMatch+ | 63.6/60.5 | 79.5/78.5 | 50.2/14.8 | 80.8/79.9 |  81.7/81.0

Table 4: Comparison of bACC on CIFAR-100-LT.

CIFAR-100-LT ($\gamma=\gamma_{l}=\gamma_{u}$, $\gamma_{u}$ is assumed to be
known)
---
Algorithm | $\gamma=20$ | $\gamma=50$ | $\gamma=100$
FixMatch |  $49.6$$\pm 0.78$ |  $42.1$$\pm 0.33$ |  $37.6$$\pm 0.48$
FixMatch+DARP |  $50.8$$\pm 0.77$ |  $43.1$$\pm 0.54$ |  $38.3$$\pm 0.47$
FixMatch+DARP+cRT |  $51.4$$\pm 0.68$ |  $44.9$$\pm 0.54$ |  $40.4$$\pm 0.78$
FixMatch+CReST |  $51.8$$\pm 0.12$ |  $44.9$$\pm 0.50$ |  $40.1$$\pm 0.65$
FixMatch+CReST+LA |  $52.9$$\pm 0.07$ |  $47.3$$\pm 0.17$ |  $42.7$$\pm 0.70$
FixMatch+ABC |  $53.3$$\pm 0.79$ |  $46.7$$\pm 0.26$ |  $41.2$$\pm 0.06$
FixMatch+CoSSL |  $53.9$$\pm 0.78$ |  $47.6$$\pm 0.57$ |  $43.0$$\pm 0.61$
FixMatch+UDAL | - |  $48.0$$\pm 0.56$ |  $43.7$$\pm 0.41$
FixMatch+CDMAD | 54.3$\pm 0.44$ | 48.8$\pm 0.75$ | 44.1$\pm 0.29$
ReMixMatch |  $51.6$$\pm 0.43$ |  $44.2$$\pm 0.59$ |  $39.3$$\pm 0.43$
ReMixMatch+DARP |  $51.9$$\pm 0.35$ |  $44.7$$\pm 0.66$ |  $39.8$$\pm 0.53$
ReMixMatch+DARP+cRT |  $54.5$$\pm 0.42$ |  $48.5$$\pm 0.91$ |  $43.7$$\pm 0.81$
ReMixMatch+CReST |  $51.3$$\pm 0.34$ |  $45.5$$\pm 0.76$ |  $41.0$$\pm 0.78$
ReMixMatch+CReST+LA |  $51.9$$\pm 0.60$ |  $46.6$$\pm 1.14$ |  $41.7$$\pm 0.69$
ReMixMatch+ABC |  $55.6$$\pm 0.35$ |  $47.9$$\pm 0.10$ |  $42.2$$\pm 0.12$
ReMixMatch+CoSSL |  $55.8$$\pm 0.62$ |  $48.9$$\pm 0.61$ |  $44.1$$\pm 0.59$
ReMixMatch+CDMAD | 57.0$\pm 0.32$ | 51.1$\pm 0.46$ | 44.9$\pm 0.42$

Tab. 2 summarizes bACC and GM of the baseline algorithms and proposed
algorithm on CIFAR-10-LT and STL-10-LT when $\gamma_{u}$ is unknown and
different from $\gamma_{l}$. ReMixMatch performed poorly when $\gamma_{l}$ and
$\gamma_{u}$ differed significantly, probably because the distribution
alignment technique employed in ReMixMatch significantly degraded the quality
of pseudo-labels. By aligning the class distribution of the pseudo-labels with
that of the unlabeled set estimated as in Kim et al. [28], ReMixMatch*
significantly improved the classification performance. However, the estimation
of the class distribution of the unlabeled set becomes more time-consuming as
the amount of unlabeled data increases. Furthermore, the estimation process
requires more than 10 labeled samples for each class, making it unsuitable for
datasets with a very small number of labeled samples, such as CIFAR-100-LT. In
contrast, CDMAD does not rely on the estimated class distribution of the
unlabeled set, making it more effective than baseline algorithms combined with
ReMixMatch* for real-world scenarios. We can also observe that the LA
decreased the performance of ReMixMatch*+DARP and ReMixMatch*+SAW when
$\gamma_{l}$ and $\gamma_{u}$ differed significantly. This may be because the
LA considers only the class distribution of the labeled set for re-balancing
the classifier when the class distribution of the unlabeled set is unknown. In
Appendix K, we present further comparisons of LA and CDMAD under the settings
that the class distributions of labeled and unlabeled sets mismatch.

We also conducted experiments under the setting that the class distribution of
the unlabeled set is imbalanced in the opposite direction to the labeled set.
From Tab. 3, we can observe that CDMAD outperforms the baseline algorithms.

Table 5: Comparison of bACC on Small-ImageNet-127 (size $32\times 32$ and
$64\times 64$, $\gamma_{u}$ is assumed to be known)

Small-ImageNet-127 ($\gamma=\gamma_{l}=\gamma_{u}$, $\gamma_{u}$ is assumed to
be known)
---
Algorithm |  |  | $32\times 32$ | $64\times 64$
FixMatch |  |  | $29.7$ | $42.3$
FixMatch+DARP |  |  | $30.5$ | $42.5$
FixMatch+DARP+cRT |  |  | $39.7$ | $51.0$
FixMatch+CReST |  |  | $32.5$ | $44.7$
FixMatch+CReST+LA |  |  | $40.9$ | $55.9$
FixMatch+ABC |  |  | $46.9$ | $56.1$
FixMatch+CoSSL |  |  | $43.7$ | $53.8$
FixMatch+CDMAD |  |  | 48.4 | 59.3

Tab. 4 summarizes bACC of the baseline algorithms and the proposed algorithm
on CIFAR-100-LT. The proposed algorithm outperformed baseline algorithms.
These results demonstrate that the proposed algorithm is well-suited for CISSL
on datasets with a large number of classes. Moreover, given that several
minority classes in the training set have only one labeled sample when
$\gamma=100$, the results indicate that the proposed algorithm may outperform
the baseline CISSL algorithms when the number of labeled samples from minority
classes is extremely limited. This may be because CDMAD effectively
compensates for the lack of labeled samples by well refining the biased
pseudo-labels compared to the baseline algorithms.

Tab. 5 summarizes bACC of the baseline algorithms on Small-ImageNet-127. For
both sizes of Small-ImageNet-127, CDMAD outperformed the baseline algorithms
by a large margin. The effective use of unlabeled samples through appropriate
refinement of the pseudo-labels may allow the proposed algorithm suitable for
CISSL on large-scale datasets. Given that the test set of Small-ImageNet-127
is class-imbalanced, the results also show that CDMAD can be suitable for
CISSL with a class-imbalanced test set.

Table 6: Comparison of bACC/GM on CIFAR-10-LT with FreeMatch as the base SSL
algorithm

CIFAR-10-LT
---
Algorithm | $\gamma_{l}=\gamma_{u}=100$ |  $\gamma_{l}=100$, $\gamma_{u}=1$
FreeMatch | 75.4/72.9 | 74.2/69.5
FreeMatch+SAW+cRT | 82.8/82.3 | 86.4/86.2
FreeMatch+CDMAD | 84.8/84.4 | 89.0/88.7

To verify that CDMAD can be also effectively combined with recent SSL
algorithms, we conducted experiments by setting FreeMatch [57] as the base SSL
algorithm. From Tab. 6, we can observe that CDMAD outperforms FreeMatch and
FreeMatch+SAW+cRT.

We also compared the classification performance of CDMAD with ACR [60], a
recent CISSL algorithm. From Tab. 7, we can observe that CDMAD outperforms
ACR.

Table 7: Comparison of bACC/GM on CIFAR-10-LT

Algorithm/ CIFAR-10-LT | $\gamma_{l}=\gamma_{u}=100$ |  $\gamma_{l}=100$, $\gamma_{u}=1$
---|---|---
FixMatch+ACR | 81.8/81.4 | 85.6/85.3
FixMatch+CDMAD | 83.6/83.1 | 87.5/87.1

|  |  |
---|---|---|---
(a) FixMatch+CDMAD | (b) ReMixMatch+CDMAD | (c) ReMixMatch | (d) ReMixMatch+CDMAD
Figure 3: (a) and (b) present the class probabilities predicted on a white
image using the proposed algorithm. (c) and (d) present the confusion matrices
of the class predictions on test samples. Table 8: Ablation study for the
proposed algorithm on CIFAR-10-LT under $\gamma_{l}=100$ and $\gamma_{u}=1$

Ablation study ($\gamma_{l}=100$, $\gamma_{u}=1$) | bACC/GM |  | bACC/GM
---|---|---|---
FixMatch+CDMAD | $\mathbf{87.5/87.1}$ | ReMixMatch+CDMAD | $\mathbf{89.9/89.6}$
Without CDMAD for refining pseudo-labels | $78.2/75.8$ | Without CDMAD for refining pseudo-labels | $72.3/65.9$
Without CDMAD for test phase | $84.9/84.1$ | Without CDMAD for test phase | $88.2/87.7$
With the use of hard pseudo-labels | $86.7/86.3$ | With the use of sharpened pseudo-labels | $88.9/88.6$
With the use of confidence threshold $\tau=0.95$ | $86.8/86.3$ | With the use of distribution alignment technique | $80.4/78.5$

We present additional experimental results in Appendix. Specifically, fine
grained results (many/medium/few group performance) are summarized in Appendix
L. In Appendix M, we compare the classification performance of CDMAD with DASO
[45] whose classification performance was measured under different settings
with the settings of ours.

###  4.3 Qualitative analyses

We argue that the CDMAD can implicitly consider the class distributions of
both labeled and unlabeled sets when measuring the classifier’s biased degree.
To verify this argument, in Fig. 3 (a) and (b), we analyze the class
probabilities predicted on a white image,
$P_{\theta}\left(y|\mathcal{I}\right)$, using FixMatch+CDMAD and
ReMixMatch+CDMAD trained on CIFAR-10-LT under the three settings: 1)
$\gamma_{l}=\gamma_{u}=100$, 2) $\gamma_{l}=100$ and $\gamma_{u}=1$, and 3)
$\gamma_{l}=\gamma_{u}=1$.

We can observe that both FixMatch+CDMAD and ReMixMatch+CDMAD produced highly
nonuniform class probabilities when they were trained under $\gamma_{l}=100$
and $\gamma_{u}=100$. In contrast, when trained with $\gamma_{l}=100$ and
$\gamma_{u}=1$, both algorithms produced significantly more balanced class
probabilities. These results show that the classifier’s biased degree,
$g_{\theta}\left(\mathcal{I}\right)$, depends on the class distribution of the
unlabeled set. Moreover, the comparison of nearly uniform class probabilities
produced under $\gamma_{l}=\gamma_{u}=1$ and the results under
$\gamma_{l}=100$ and $\gamma_{u}=1$ shows that
$g_{\theta}\left(\mathcal{I}\right)$ also depends on the class distribution of
the labeled set. Based on the above findings, CDMAD can be considered as
measuring the classifier’s biased degree by implicitly incorporating the class
distributions of both labeled and unlabeled sets. It is worth noting that
under $\gamma_{l}=100$ and $\gamma_{u}=1$, FixMatch+CDMAD and ReMixMatch+CDMAD
produced significantly more balanced class probabilities compared to FixMatch
and ReMixMatch in Fig. 1. This may be because the use of biased pseudo-labels
generated by FixMatch and ReMixMatch for training exacerbated class imbalance,
whereas CDMAD effectively refined the biased pseudo-labels, as discussed in
Figure 6 of Appendix J.

We also argue that the ability of CDMAD to implicitly incorporate the class
distributions of both labeled and unlabeled sets enables it to effectively
mitigate class imbalance even under severe class distribution mismatch. To
verify this argument, we present the confusion matrices of the class
predictions on the test set of CIFAR-10 using ReMixMatch and ReMixMatch+CDMAD
trained on CIFAR-10-LT under $\gamma_{l}=100$ and $\gamma_{u}=1$ in Fig. 3 (c)
and (d). The value in the $i$th row and $j$th column represents the proportion
of the $i$th class samples classified as the $j$th class. We can observe that
the class predictions of ReMixMatch in Fig. 3 (c) are biased towards the
majority classes. Specifically, the data points in the minority classes (e.g.,
classes 7, 8 and 9) are often misclassified into the majority classes (e.g.,
classes 0 and 1). In contrast, ReMixMatch+CDMAD in Fig. 3 (d) made nearly
balanced class predictions. Further qualitative analyses are presented in
Appendix J.

###  4.4 Ablation study

To investigate the effectiveness of each element of CDMAD, we conducted an
ablation study using CIFAR-10-LT ($\gamma_{l}=100$ and $\gamma_{u}=1$,
$\gamma_{u}$ is assumed to be unknown). Each row in Tab. 8 represents the
proposed algorithm under the condition specified in that row. The results are
as follows: 1) Without the refinement of the biased pseudo-labels using CDMAD
in the training phase, the classification performance significantly decreased.
2) Without the refinement of the biased class predictions on the test set
using CDMAD, the classification performance decreased. 3) With entropy
minimization (using hard pseudo-labels and sharpened pseudo-labels) of the
class predictions during training, the classification performance slightly
decreased. 4) For FixMatch+CDMAD, the use of confidence threshold $\tau=0.95$
slightly decreased the classification performance. 5) For ReMixMatch+CDMAD,
the classification performance significantly decreased by using the
distribution alignment technique instead of adding the supervised loss on
$\alpha\left(x^{m}_{b}\right)$ for training. These results indicate that every
element of CDMAD enhances performance.

Table 9: Experiments with the replacement of $\mathcal{I}$ by other inputs

ReMixMatch+CDMAD | CIFAR-10-LT
---|---
Input | $\gamma_{l}=\gamma_{u}=100$ |  $\gamma_{l}=100$, $\gamma_{u}=1$
Uniform |  $81.3$/ $80.7$ |  $85.3$/ $84.2$
Bernoulli |  $82.5$/ $82.0$ |  $83.6$/ $82.8$
Normal |  $78.4$/ $77.5$ |  $84.0$/ $83.2$
Black |  $84.8$/ $84.5$ |  $89.3$/ $89.0$
Red |  $84.8$/ $84.6$ |  $90.1$/ $89.9$
Green |  $84.9$/ $84.6$ |  $89.3$/ $88.9$
Blue |  $84.9$/ $84.7$ |  $90.2$/ $89.9$
Gray |  $85.1$/ $84.9$ |  $89.6$/ $89.3$
White |  $85.5$/ $85.3$ |  $89.9$/ $89.6$

To explore whether the classifier’s biased degree can be measured using other
images rather than the white image, we conducted experiments by replacing
$\mathcal{I}$ with other solid color images or an image consisting of random
pixel values that are generated from uniform, Bernoulli, and normal
distributions. Experimental results are summarized in Tab. 9. From the table,
we can observe that the classification performance of ReMixMatch+CDMAD
decreased when $\mathcal{I}$ was replaced by images consisting of random
pixels. This may be because the parameters of the distributions (e.g., mean
and standard deviation of a normal distribution) used to generate random
pixels may be related to specific classes. In contrast, the classification
performance of ReMixMatch+CDMAD did not significantly change when
$\mathcal{I}$ was replaced by images of other solid color images. These
results show that other solid color images can also be used to measure the
classifier’s biased degree.

Table 10: Experiments with replacing $\mathcal{I}$ by non-image input

Algorithm | CIFAR-10-LT | $\gamma_{l}=\gamma_{u}=100$ |  $\gamma_{l}=100$, $\gamma_{u}=1$
---|---|---|---
FixMatch+CDMAD | White image | 83.6/83.1 | 87.5/87.1
Non-image | 84.0/83.6 | 87.4/87.0
ReMixMatch+CDMAD | White image | 85.5/85.3 | 89.9/89.6
Non-image | 85.6/85.4 | 89.8/89.6

However, the assumption that a solid color image is non-informative for the
class labels may fail when the classification of images is related to their
color. To address this concern, we additionally considered an image with pixel
values that are outside the range [0, 255] (actually, non-image input) to
replace the solid color image $\mathcal{I}$. For example, in the case of
CIFAR-10-LT, the maximum value of each (R,G,B) channel becomes (2.06, 2.12,
2.11) after input normalization. In this case, if we generate an input with
every pixel’s (R,G,B) values set to (3, 3, 3), the input would not be
associated with a specific class because it is actually not an image. By
replacing $\mathcal{I}$ with this non-image input, we conducted experiments
and presented the results in Tab. 10. We observe that CDMAD with the non-image
input performs comparably with the white image. These results verify that the
non-image input can be effectively used for measuring the classifier’s biased
degree, overcoming the challenge of finding data that are non-informative for
the class labels.

##  5 Conclusion

We proposed CDMAD, which considers the classifier’s biased degree towards each
class to appropriately mitigate the class imbalance in SSL even under severe
class distribution mismatch between the labeled and unlabeled sets. Using
CDMAD, we refined biased pseudo-labels as well as biased class predictions on
test samples. Experiments on four benchmark datasets show that the proposed
algorithm outperforms the existing CISSL algorithms. Moreover, the qualitative
analysis and ablation study on the proposed algorithm demonstrate the
effectiveness of each component of CDMAD. In this paper, we used a solid color
image to measure the classifier’s biased degree, which lacks a firm
theoretical basis. In future research, we plan to establish a theoretical
foundation for utilizing the solid color image to measure the classifier’s
biased degree.

Acknowledgements This research was supported by the National Research
Foundation of Korea (NRF) grant funded by the Korea government (MSIT)
(2023R1A2C2005453, RS-2023-00218913).

## References

  * Barandela et al. [2003] Ricardo Barandela, E Rangel, José Salvador Sánchez, and Francesc J Ferri.  Restricted decontamination for the imbalanced training sample problem.  In _Iberoamerican congress on pattern recognition_ , pages 424–431. Springer, 2003.
  * Berthelot et al. [2019] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel.  Mixmatch: A holistic approach to semi-supervised learning.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2019.
  * Berthelot et al. [2020] David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel.  Remixmatch: Semi-supervised learning with distribution matching and augmentation anchoring.  In _International Conference on Learning Representations_ , 2020.
  * Cai et al. [2021] Jiarui Cai, Yizhou Wang, and Jenq-Neng Hwang.  Ace: Ally complementary experts for solving long-tailed recognition in one-shot.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 112–121, 2021.
  * Cao et al. [2019] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma.  Learning imbalanced datasets with label-distribution-aware margin loss.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2019.
  * Chawla et al. [2002] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer.  Smote: synthetic minority over-sampling technique.  _Journal of artificial intelligence research_ , 16:321–357, 2002.
  * Chen et al. [2023] H Chen, R Tao, Yue Fan, Y Wang, M Savvides, J Wang, B Raj, X Xie, and Bernt Schiele.  Softmatch: Addressing the quantity-quality tradeoff in semi-supervised learning.  In _Eleventh International Conference on Learning Representations_. OpenReview. net, 2023.
  * Coates et al. [2011] Adam Coates, Andrew Ng, and Honglak Lee.  An analysis of single-layer networks in unsupervised feature learning.  In _Proceedings of the fourteenth international conference on artificial intelligence and statistics_ , pages 215–223. JMLR Workshop and Conference Proceedings, 2011.
  * Collell et al. [2016] Guillem Collell, Drazen Prelec, and Kaustubh Patil.  Reviving threshold-moving: a simple plug-in bagging ensemble for binary and multiclass imbalanced data.  _arXiv preprint arXiv:1606.08698_ , 2016.
  * Cubuk et al. [2020] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le.  Randaugment: Practical automated data augmentation with a reduced search space.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , pages 702–703, 2020.
  * Cui et al. [2021] Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia.  Parametric contrastive learning.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 715–724, 2021.
  * Cui et al. [2019] Yin Cui, Menglin Jia, Tsung-Yi Lin, Yang Song, and Serge Belongie.  Class-balanced loss based on effective number of samples.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9268–9277, 2019.
  * DeVries and Taylor [2017] Terrance DeVries and Graham W Taylor.  Improved regularization of convolutional neural networks with cutout.  _arXiv preprint arXiv:1708.04552_ , 2017.
  * Fan et al. [2022] Yue Fan, Dengxin Dai, Anna Kukleva, and Bernt Schiele.  Cossl: Co-learning of representation and classifier for imbalanced semi-supervised learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14574–14584, 2022.
  * Gidaris et al. [2018] Spyros Gidaris, Praveer Singh, and Nikos Komodakis.  Unsupervised representation learning by predicting image rotations.  In _International Conference on Learning Representations_ , 2018.
  * Grandvalet and Bengio [2005] Yves Grandvalet and Yoshua Bengio.  Semi-supervised learning by entropy minimization.  In _Advances in Neural Information Processing Systems_. MIT Press, 2005.
  * Guo and Li [2022] Lan-Zhe Guo and Yu-Feng Li.  Class-imbalanced semi-supervised learning with adaptive thresholding.  In _International Conference on Machine Learning_ , pages 8082–8094. PMLR, 2022.
  * He and Garcia [2009] Haibo He and Edwardo A Garcia.  Learning from imbalanced data.  _IEEE Transactions on knowledge and data engineering_ , 21(9):1263–1284, 2009.
  * He et al. [2016] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Hornik et al. [1989] Kurt Hornik, Maxwell Stinchcombe, and Halbert White.  Multilayer feedforward networks are universal approximators.  _Neural networks_ , 2(5):359–366, 1989.
  * Huang et al. [2016] Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang.  Learning deep representation for imbalanced classification.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 5375–5384, 2016.
  * Huh et al. [2016] Minyoung Huh, Pulkit Agrawal, and Alexei A Efros.  What makes imagenet good for transfer learning?  _arXiv preprint arXiv:1608.08614_ , 2016.
  * Jamal et al. [2020] Muhammad Abdullah Jamal, Matthew Brown, Ming-Hsuan Yang, Liqiang Wang, and Boqing Gong.  Rethinking class-balanced methods for long-tailed visual recognition from a domain adaptation perspective.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7610–7619, 2020.
  * JAPKOWICZ [2000] N JAPKOWICZ.  The class imbalance problem: Significance and strategies.  In _Proc. 2000 International Conference on Artificial Intelligence_ , pages 111–117, 2000.
  * Jiang et al. [2021] Ziyu Jiang, Tianlong Chen, Bobak J Mortazavi, and Zhangyang Wang.  Self-damaging contrastive learning.  In _International Conference on Machine Learning_ , pages 4927–4939. PMLR, 2021.
  * Kang et al. [2020a] Bingyi Kang, Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng.  Exploring balanced feature spaces for representation learning.  In _International Conference on Learning Representations_ , 2020a.
  * Kang et al. [2020b] Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis.  Decoupling representation and classifier for long-tailed recognition.  In _International Conference on Learning Representations_ , 2020b.
  * Kim et al. [2020a] Jaehyung Kim, Youngbum Hur, Sejun Park, Eunho Yang, Sung Ju Hwang, and Jinwoo Shin.  Distribution aligning refinery of pseudo-label for imbalanced semi-supervised learning.  In _Advances in Neural Information Processing Systems_ , pages 14567–14579. Curran Associates, Inc., 2020a.
  * Kim et al. [2020b] Jaehyung Kim, Jongheon Jeong, and Jinwoo Shin.  M2m: Imbalanced classification via major-to-minor translation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 13896–13905, 2020b.
  * Kingma and Ba [2015] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In _ICLR, 2015_ , 2015.
  * Komodakis and Gidaris [2018] Nikos Komodakis and Spyros Gidaris.  Unsupervised representation learning by predicting image rotations.  In _International conference on learning representations (ICLR)_ , 2018.
  * Krizhevsky [2009] A Krizhevsky.  Learning multiple layers of features from tiny images.  _Technical report, Department of Computer Science, University of Toronto_ , 2009.
  * Kubat et al. [1997] Miroslav Kubat, Stan Matwin, et al.  Addressing the curse of imbalanced training sets: one-sided selection.  In _Icml_ , page 179. Citeseer, 1997.
  * Lai et al. [2022] Zhengfeng Lai, Chao Wang, Henrry Gunawan, Sen-Ching S Cheung, and Chen-Nee Chuah.  Smoothed adaptive weighting for imbalanced semi-supervised learning: Improve reliability against unknown distribution data.  In _International Conference on Machine Learning_ , pages 11828–11843. PMLR, 2022.
  * Lazarow et al. [2023] Justin Lazarow, Kihyuk Sohn, Chen-Yu Lee, Chun-Liang Li, Zizhao Zhang, and Tomas Pfister.  Unifying distribution alignment as a loss for imbalanced semi-supervised learning.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 5644–5653, 2023.
  * Lee et al. [2013] Dong-Hyun Lee et al.  Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks.  In _Workshop on challenges in representation learning, ICML_ , 2013.
  * Lee et al. [2021] Hyuck Lee, Seungjae Shin, and Heeyoung Kim.  Abc: Auxiliary balanced classifier for class-imbalanced semi-supervised learning.  _Advances in Neural Information Processing Systems_ , 34:7082–7094, 2021.
  * Li et al. [2022a] Bolian Li, Zongbo Han, Haining Li, Huazhu Fu, and Changqing Zhang.  Trustworthy long-tailed classification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6970–6979, 2022a.
  * Li et al. [2021] Junnan Li, Caiming Xiong, and Steven CH Hoi.  Comatch: Semi-supervised learning with contrastive graph regularization.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9475–9484, 2021.
  * Li et al. [2022b] Tianhong Li, Peng Cao, Yuan Yuan, Lijie Fan, Yuzhe Yang, Rogerio S Feris, Piotr Indyk, and Dina Katabi.  Targeted supervised contrastive learning for long-tailed recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6918–6928, 2022b.
  * Menon et al. [2013] Aditya Menon, Harikrishna Narasimhan, Shivani Agarwal, and Sanjay Chawla.  On the statistical consistency of algorithms for binary classification under class imbalance.  In _International Conference on Machine Learning_ , pages 603–611. PMLR, 2013.
  * Menon et al. [2020] Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar.  Long-tail learning via logit adjustment.  In _International Conference on Learning Representations_ , 2020.
  * Mikolov et al. [2013] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  Distributed representations of words and phrases and their compositionality.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2013.
  * Miyato et al. [2018] Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii.  Virtual adversarial training: a regularization method for supervised and semi-supervised learning.  _IEEE transactions on pattern analysis and machine intelligence_ , 41(8):1979–1993, 2018.
  * Oh et al. [2022] Youngtaek Oh, Dong-Jin Kim, and In So Kweon.  Daso: Distribution-aware semantics-oriented pseudo-label for imbalanced semi-supervised learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9786–9796, 2022.
  * Park et al. [2018] Sungrae Park, JunKeon Park, Su-Jin Shin, and Il-Chul Moon.  Adversarial dropout for supervised and semi-supervised learning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2018.
  * Ren et al. [2020] Jiawei Ren, Cunjun Yu, shunan sheng, Xiao Ma, Haiyu Zhao, Shuai Yi, and hongsheng Li.  Balanced meta-softmax for long-tailed visual recognition.  In _Advances in Neural Information Processing Systems_ , pages 4175–4186. Curran Associates, Inc., 2020.
  * Russakovsky et al. [2015] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  _International journal of computer vision_ , 115(3):211–252, 2015.
  * Sohn et al. [2020] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li.  Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  _Advances in Neural Information Processing Systems_ , 33, 2020.
  * Tarvainen and Valpola [2017] Antti Tarvainen and Harri Valpola.  Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2017.
  * Van der Maaten and Hinton [2008] Laurens Van der Maaten and Geoffrey Hinton.  Visualizing data using t-sne.  _Journal of machine learning research_ , 9(11), 2008.
  * Verma et al. [2019] Vikas Verma, Alex Lamb, Juho Kannala, Yoshua Bengio, and David Lopez-Paz.  Interpolation consistency training for semi-supervised learning.  In _Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19_ , pages 3635–3641. International Joint Conferences on Artificial Intelligence Organization, 2019.
  * Wang et al. [2021a] Peng Wang, Kai Han, Xiu-Shen Wei, Lei Zhang, and Lei Wang.  Contrastive learning based hybrid networks for long-tailed image classification.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 943–952, 2021a.
  * Wang et al. [2023a] Renzhen Wang, Xixi Jia, Quanziang Wang, Yichen Wu, and Deyu Meng.  Imbalanced semi-supervised learning with bias adaptive classifier.  In _The Eleventh International Conference on Learning Representations_ , 2023a.
  * Wang et al. [2021b] Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella Yu.  Long-tailed recognition by routing diverse distribution-aware experts.  In _International Conference on Learning Representations_ , 2021b.
  * Wang et al. [2022] Xudong Wang, Zhirong Wu, Long Lian, and Stella X Yu.  Debiased learning from naturally imbalanced pseudo-labels.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14647–14657, 2022.
  * Wang et al. [2023b] Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, Bernt Schiele, and Xing Xie.  Freematch: Self-adaptive thresholding for semi-supervised learning.  In _The Eleventh International Conference on Learning Representations_ , 2023b.
  * Wang et al. [2017] Yu-Xiong Wang, Deva Ramanan, and Martial Hebert.  Learning to model the tail.  In _Advances in Neural Information Processing Systems_. Curran Associates, Inc., 2017.
  * Wei et al. [2021] Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, and Fan Yang.  Crest: A class-rebalancing self-training framework for imbalanced semi-supervised learning.  _arXiv preprint arXiv:2102.09559_ , 2021.
  * Wei and Gan [2023] Tong Wei and Kai Gan.  Towards realistic long-tailed semi-supervised learning: Consistency is all you need.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 3469–3478, 2023.
  * Xiang et al. [2020] Liuyu Xiang, Guiguang Ding, and Jungong Han.  Learning from multiple experts: Self-paced knowledge distillation for long-tailed classification.  In _European Conference on Computer Vision_ , pages 247–263. Springer, 2020.
  * Yin et al. [2018] Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker.  Feature transfer learning for deep face recognition with under-represented data.  _arXiv e-prints_ , pages arXiv–1803, 2018.
  * Yu et al. [2023] Zhuoran Yu, Yin Li, and Yong Jae Lee.  InPL: Pseudo-labeling the inliers first for imbalanced semi-supervised learning.  In _The Eleventh International Conference on Learning Representations_ , 2023.
  * Zagoruyko and Komodakis [2016] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks.  _arXiv preprint arXiv:1605.07146_ , 2016.
  * Zhang et al. [2021] Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki.  Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling.  _Advances in Neural Information Processing Systems_ , 34:18408–18419, 2021.
  * Zhang et al. [2022] Yifan Zhang, Bryan Hooi, Lanqing Hong, and Jiashi Feng.  Self-supervised aggregation of diverse experts for test-agnostic long-tailed recognition.  _Advances in Neural Information Processing Systems_ , 35:34077–34090, 2022.
  * Zhou et al. [2020] Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen.  Bbn: Bilateral-branch network with cumulative learning for long-tailed visual recognition.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 9719–9728, 2020.
  * Zhou [2020] Ding-Xuan Zhou.  Universality of deep convolutional neural networks.  _Applied and computational harmonic analysis_ , 48(2):787–794, 2020.

##  Appendix A Core part of the code for CDMAD

Figure 4: Code for refining pseudo-labels using CDMAD

Fig. 4 presents a core part of the code for CDMAD to refine the biased pseudo-
labels of the base SSL algorithm. As we can see in Fig. 4, CDMAD is very easy
to implement. We simply need to calculate the logits for an image without any
patterns (solid color image) and then subtract them from the logits for
unlabeled samples. Biased class predictions on test samples are refined in a
similar way.

##  Appendix B Further related works

Semi-supervised learning (SSL) algorithms use unlabeled data for training when
labeled samples are insufficient. Entropy minimization [16] encourages the
class predictions on unlabeled samples to be confident by directly minimizing
entropy or using pseudo-labels [36]. Consistency regularization [46, 44, 50]
encourages the class predictions on two augmented versions of an unlabeled
sample to be consistent. FixMatch [49] and ReMixMatch [3] conduct entropy
minimization and consistency regularization using strong data augmentation
techniques [13, 10]. ReMixMatch also conducts Mixup regularization [52, 2] and
self-supervised learning with rotation [15]. CoMatch [39] proposed graph-based
contrastive learning using embedding and pseudo-label graphs. Recently,
curriculum pseudo-labeling that considers the learning status for each class
was proposed by FlexMatch [65] and extended in Adsh [17], SoftMatch [7] and
FreeMatch [57].

Class-imbalanced learning (CIL) algorithms mitigate class imbalance to improve
classification performance for minority classes. Resampling techniques [1, 6,
24, 18] balance the number of each class samples, and reweighting techniques
[43, 58, 21, 12, 23] balance the loss for each class. Cao et al. [5] and Ren
et al. [47] proposed losses that minimize a generalization error bound, and
Kim et al. [29], Yin et al. [62] transferred knowledge from the data of the
majority classes to the data of minority classes. Kang et al. [27] decoupled
representation and classifier learning. Menon et al. [42] proposed post-hoc
logit-adjustment and loss, which is Fisher consistent for minimizing the
balanced error. Recently, CIL algorithms based on contrastive learning [26,
26, 25, 53, 11, 40] and multi-expert learning [67, 61, 4, 55, 38, 66] received
considerable attention.

##  Appendix C Data augmentation techniques

|  |  |
---|---|---|---
(a) Random horizontal flipping 1 | (b) Random horizontal flipping 2 | (c) Random cropping 1 | (d) Random cropping 2
|  |  |
(e) Cutout 1 | (f) Cutout 2 | (g) RandomAugment 1 | (h) RandomAugment 2
Figure 5: Example images augmented using each data augmentation technique

CDMAD uses data augmentation techniques utilized in FixMatch, ReMixMatch, and
previous CISSL algorithms. Specifically, CDMAD uses random horizontal flipping
and random cropping as weak data augmentation techniques and uses Cutout [13]
and RandomAugment [10] as strong data augmantation techniques. Random
horizontal flipping and cropping flips and crops images, respectively. We
implemented these weak data augmentation techniques using
torchvision.transforms library. Cutout randomly masks out the square region of
the image during training, which prevents the network from focusing on non-
general features. The purpose of RandomAugment is to teach the network
invariances. RandomAugment is a data augmentation technique that automatically
searches for improved augmentation policies, where the search space of the
policy consists of many sub-policies, one of which is randomly chosen for each
data point at each iteration. A sub-policy is composed of basic data-
augmentation techniques, such as shearing, rotation, and translation. We
implemented Cutout and RandomAugment using the code from
https://github.com/ildoonet/pytorch-randaugment. Example images augmented
using each data augmentation technique are presented in Fig. 5.

##  Appendix D Training losses of FixMatch [49] and ReMixMatch [3]

Training losses of FixMatch [49] and ReMixMatch [3] on a minibatch for labeled
set $\mathcal{MX}$ and a minibatch for unlabeled set $\mathcal{MU}$ can be
expressed as follows:

$\displaystyle
loss_{F}\left(\mathcal{MX},\mathcal{MU},\hat{q},\tau;\theta\right)=Con(\mathcal{MU},\hat{q},\tau;\theta)$
(11) $\displaystyle+Sup(\mathcal{MX};\theta),$ $\displaystyle
loss_{R}\left(\mathcal{MX},\mathcal{MU},\bar{q};\theta\right)=Mix(\mathcal{MX},\mathcal{MU},\bar{q};\theta)$
(12)
$\displaystyle+Con(\mathcal{MU},\bar{q};\theta)+Rot(\mathcal{MU},r;\theta),$

where $\hat{q}$ and $\bar{q}$ denote the concatenations of $\hat{q_{b}}$ and
$\bar{q_{b}}$, $b=1,\ldots,\mu B$, respectively,
$Con(\mathcal{MU},\hat{q},\tau;\theta)$ and $Con(\mathcal{MU},\bar{q};\theta)$
denote the consistency regularization loss with and without the confidence
threshold $\tau$, respectively, $Sup(\mathcal{MX};\theta)$ denotes the
supervised loss for weakly augmented labeled data points,
$Mix(\mathcal{MX},\mathcal{MU},\bar{q};\theta)$ denotes the mix-up
regularization loss, and $Rot(\mathcal{MU},r;\theta)$ denotes the rotation
loss with the rotated degree $r$.

Each loss term in Eq (1) and (2) of the main paper is detailed as follows:

$\displaystyle Con(\mathcal{MU},\hat{q},\tau;\theta)=$ (13)
$\displaystyle\frac{1}{\mu
B}\sum_{u^{m}_{b}\in\mathcal{MU}}{\mathbf{I}(\max(\hat{q_{b}})\geq\tau)}\mathbf{H}(P_{\theta}(y|\mathcal{A}(u^{m}_{b})),\hat{q_{b}}),$
$\displaystyle Con(\mathcal{MU},\bar{q};\theta)=\frac{1}{\mu
B}\sum_{u^{m}_{b}\in\mathcal{MU}}\mathbf{H}(P_{\theta}(y|\mathcal{A}(u^{m}_{b})),\bar{q_{b}}),$
(14)
$Sup(\mathcal{MX};\theta)=\frac{1}{B}\sum_{x^{m}_{b}\in\mathcal{MX}}\mathbf{H}(P_{\theta}(y|\alpha(x^{m}_{b})),p^{m}_{b}),$
(15) $\displaystyle
Mix(\mathcal{MX},\mathcal{MU},\bar{q};\theta)=\frac{1}{B}\sum_{mx^{m}_{b}\in\mathcal{MX^{\prime}}}\mathbf{H}(P_{\theta}(y|mx^{m}_{b}),mp^{m}_{b})+$
(16) $\displaystyle\frac{1}{\mu
B}\sum_{mu^{m}_{b}\in\mathcal{MU^{\prime}}}\mathbf{H}(P_{\theta}(y|mu^{m}_{b}),\bar{mq_{b}}),$
$Rot(\mathcal{MU},r;\theta)=\frac{1}{\mu
B}\sum_{u^{m}_{b}\in\mathcal{MU}}\mathbf{H}(P_{\theta^{\prime}}(d|\mathcal{R}(u^{m}_{b},r)),r),$
(17)

where $\mathbf{H}(\cdot,\cdot)$ denotes the cross-entropy loss, $p^{m}_{b}$ is
one-hot encoded $y_{b}^{m}$, $\mathcal{MX^{\prime}}$ and
$\mathcal{MU^{\prime}}$ are generated by mixup operation with strongly
augmented $\mathcal{MX}$ and $\mathcal{MU}$, respectively, $mx^{m}_{b}$
denotes a mixed-labeled image, $mp^{m}_{b}$ denotes a mixed label,
$mu^{m}_{b}$ denotes a mixed-unlabeled image, $\bar{mq_{b}}$ denotes a mixed
pseudo-label, $\mathcal{R}(u^{m}_{b},r)$ denotes the rotated $u^{m}_{b}$ with
degree $r$, and $P_{\theta^{\prime}}(\hat{r}|\mathcal{R}(u^{m}_{b},r))$
denotes the prediction of rotated degree $r$ using network parameters
$\theta^{\prime}$ that mostly overlap with $\theta$.

##  Appendix E Illustration of refining biased class predictions on test
samples using CDMAD

Figure 6: Refinement of biased class predictions on test samples using CDMAD

Fig. 6 presents refinement process of the biased class predictions on test
samples using the CDMAD.

##  Appendix F Pseudo code of the proposed algorithm

The pseudo code that describes both training and test phases of the proposed
algorithm is presented in Algorithm 1.

Algorithm 1 Pseudo code of the proposed algorithm

Input: Labeled set $\mathcal{X}$, unlabeled set $\mathcal{U}$, test set
$\mathcal{X}^{test}$, network parameters $\theta$

Output: Refined class predictions on test samples
$f^{*}_{\theta}\left(x^{test}_{k}\right)$ for $k=1,\ldots,K$

while training do

Generate minibatches
$\mathcal{MX}=\left\\{\left(x^{m}_{b},y^{m}_{b}\right):b\in\left(1,\ldots,B\right)\right\\}\subset\mathcal{X}$
and $\mathcal{MU}=\left\\{\left(u^{m}_{b}\right):b\in\left(1,\ldots,\mu
B\right)\right\\}\subset\mathcal{U}$

Produce logits for a solid color image $g_{\theta}\left(\mathcal{I}\right)$

Produce logits for weakly augmented unlabeled samples
$g_{\theta}\left(\alpha\left(u^{m}_{b}\right)\right)$ for $b=1,\ldots,\mu B$

Obtain refined logits
$g^{*}_{\theta}\left(\alpha\left(u^{m}_{b}\right)\right)=g_{\theta}\left(\alpha\left(u^{m}_{b}\right)\right)-g_{\theta}\left(\mathcal{I}\right)$
for $b=1,\ldots,\mu B$

Obtain refined pseudo-labels
$q_{b}^{*}=\phi\left(g^{*}_{\theta}\left(\alpha\left(u^{m}_{b}\right)\right)\right)$
for $b=1,\ldots,\mu B$

if Base SSL==‘FixMatch’ then

$loss_{F}^{*}=loss_{F}\left(\mathcal{MX},\mathcal{MU},q^{*},0;\theta\right)$

$\Delta\boldsymbol{\theta}\propto\nabla_{\boldsymbol{{\theta}}}loss_{F}^{*}$,
$\quad\boldsymbol{\theta}\leftarrow\boldsymbol{\theta}+\Delta\boldsymbol{\theta}$

end if

if Base SSL==‘ReMixMatch’ then

Produce class probabilities on wealy augmented labeled samples
$P_{\theta}\left(y|\alpha\left(x^{m}_{b}\right)\right)$ for $b=1,\ldots,B$

$CEloss=CrossEntropy\left(p^{m}_{b},P_{\theta}\left(y|\alpha\left(x^{m}_{b}\right)\right)\right)$

$loss_{R}^{*}=loss_{R}\left(\mathcal{MX},\mathcal{MU},q^{*};\theta\right)+CEloss$

$\Delta\boldsymbol{\theta}\propto\nabla_{\boldsymbol{{\theta}}}loss_{R}^{*}$,
$\quad\boldsymbol{\theta}\leftarrow\boldsymbol{\theta}+\Delta\boldsymbol{\theta}$

end if

end while

Produce logits for a solid color image $g_{\theta}\left(\mathcal{I}\right)$

Produce logits for test samples $g_{\theta}\left(x^{test}_{k}\right)$ for
$k=1,\ldots,K$

Obtain refined logits
$g^{*}_{\theta}\left(x^{test}_{k}\right)=g_{\theta}\left(x^{test}_{k}\right)-g_{\theta}\left(\mathcal{I}\right)$
for $k=1,\ldots,K$

Obtain refined class predictions
$f^{*}_{\theta}\left(x^{test}_{k}\right)=\operatorname{arg\,max}_{c}g^{*}_{\theta}\left(x^{test}_{k}\right)_{c}$
for $k=1,\ldots,K$

##  Appendix G Performance measures

Following previous CISSL studies, we used balanced accuracy (bACC) [21],
geometric mean (GM) [33] as performance measures for the experiments in
Section 4.2. Each performance measure is detailed as follows. Balanced
accuracy (bACC) is the average of per-class accuracy. When the test set is
class-balanced, bACC equals to the overall test accuracy. bACC is also
referred to as the averaged class recall in previous CISSL studies [59] and
[14]. Geometric mean (GM) is obtained by multiplying the $C$th root of per-
class accuracy, where $C$ denotes the number of classes. GM equals to the
overall test accuracy when all classes have the same per-class accuracy.

##  Appendix H Further details about datasets and experimental setup

CIFAR-10-LT and CIFAR-100-LT are long-tailed datasets artificially generated
from CIFAR-10 and CIFAR-100 [32], respectively, with
$N_{k}=N_{1}\times\left(N_{C}/N_{1}\right)^{\frac{k-1}{C-1}}$ and
$M_{k}=M_{1}\times\left(M_{C}/M_{1}\right)^{\frac{k-1}{C-1}}$. For
CIFAR-10-LT, we assumed that $\gamma_{u}$ is known and equal to $\gamma_{l}$
while varying both $\gamma_{l}$ and $\gamma_{u}$ as $50$, $100$ and $150$. We
then assumed that $\gamma_{u}$ is unknown and different from $\gamma_{l}$
while setting $\gamma_{l}$ to $100$ and varying $\gamma_{u}$ as $1$, $50$ and
$150$. We set $N_{1}=1500$ and $M_{1}=3000$. For CIFAR-100-LT, we assumed that
$\gamma_{u}$ is known and equal to $\gamma_{l}$ while varying both
$\gamma_{l}$ and $\gamma_{u}$ as $20$, $50$ and $100$. We set $N_{1}=150$ and
STL-10-LT is a long-tailed dataset created from STL-10 [8], where the number
of labeled samples exponentially decreases from $N_{1}$ to $N_{C}$. We
conducted experiments with unknown $\gamma_{u}$ while varying $\gamma_{l}$ as
$10$ and $20$. We set $N_{1}$ to $450$ and used all $100,000$ unlabeled
samples. Small-ImageNet-127 is a down-sampled version of ImageNet-127 [22],
created by grouping ImageNet [48] into 127 classes based on WordNet hierarchy.
The training set of ImageNet-127 consists of a total of 1,281,167 images and
is imbalanced with the class imbalanced ratio of 286. Fan et al. [14] created
two versions of this dataset by down-sampling the images to 32$\times$32 and
64$\times$64, and randomly selected 10% of the training samples of each class
as a labeled set and used the remaining as an unlabeled set. We conducted
experiments on both versions under the assumption that $\gamma_{u}$ is known
and equal to $\gamma_{l}$. Similar to Wei et al. [59], Fan et al. [14], we
conducted experiments using only FixMatch because of an excessive training
cost. The test set of Small-ImageNet-127 is also class-imbalanced.

We used the Adam optimizer [30]. We used the exponential moving average (EMA)
of the network parameters for each iteration to evaluate the classification
performance. We used Wide ResNet-28-2 [64] as a deep CNN for CIFAR-10-LT,
CIFAR-100-LT, and STL-10-LT, and ResNet-50 [19] for Small-ImageNet-127.

For the experiments using FixMatch, we set the minibatch size $B$ to 32,
relative size of the unlabeled to labeled minibatches $\mu$ to 2, and learning
rate of the optimizer to $1.5*10^{-3}$. We trained FixMatch for 500 epochs,
where 1 epoch= 500 iterations. For the experiments using ReMixMatch, we set
the minibatch size $B$ to $64$, relative size of the unlabeled to labeled
minibatches $\mu$ to 2, and learning rate of the optimizer to $2*10^{-3}$. We
trained ReMixMatch for 300 epochs. For the experiments on CIFAR-100, we set
the weight decay parameter of L2 regularization (for EMA parameters) to 0.08
because CIFAR-100 has significantly many classes compared to the total number
of training samples. For the experiments on CIFAR-10, STL-10, and Small-
ImageNet-127, we set the weight decay parameter of L2 regularization to 0.04
when the number of training samples is smaller than $3*10^{4}$, while we set
it to 0.01 and 0.02 for FixMatch and ReMixMatch, respectively, when the number
of training samples is larger than $3*10^{4}$, because L2 regularization
becomes ineffective as the number of training samples increases. We confirmed
that the training of the proposed algorithm took less time than the baseline
CISSL algorithms. We used random cropping and horizontal flipping for weak
data augmentation and Cutout [13] and RandomAugment [10] for strong data
augmentation. These augmentation techniques are detailed in Appendix C. To use
CDMAD after network parameters are stabilized, we trained naive ReMixMatch and
FixMatch for first 100 epochs, and subsequently used CDMAD to refine pseudo-
labels, similar to DARP [28]. We conducted experiments using the GPU server
Nvidia Tesla-V100 and 3090ti and used the Python library PyTorch 1.11.0 and
1.12.1. Our experiment results can be reproduced using the code in the
supplementary material.

##  Appendix I Description of baseline algorithms

The classification performance of the CDMAD was compared with those of the
following algorithms: 1\. vanilla algorithm \- Deep CNN trained with cross-
entropy loss, 2\. CIL algorithms \- Re-sampling [24], LDAM-DRW [5], and cRT
[27], 3\. SSL algorithms \- FixMatch [49] and ReMixMatch [3], and 4\. CISSL
algorithms \- DARP, DARP+LA, DARP+cRT [28], CReST, CReST+LA [59], ABC [37],
CoSSL [14], DASO [45], SAW, SAW+LA and SAW+cRT [34] combined with FixMatch and
ReMixMatch. Adsh [17], DebiasPL [56], UDAL [35] and L2AC [54] combined with
FixMatch. We report the performance of the baseline algorithms reported in
Tables of Lai et al. [34] and Fan et al. [14] when it is reproducible; the
performance measured using the uploaded code was reported otherwise.

##  Appendix J Further qualitative analysis

###  J.1 Case of $\gamma_{l}=\gamma_{u}$

In Table 1 of Section 4.2, CDMAD performed better than the baseline CISSL
algorithms when the class distributions of the labeled and unlabeled sets are
assumed to be the same. To verify whether the pseudo-labels and class
predictions on test samples refined by CDMAD contributed to its superior
performance, we conducted two types of comparison: 1) pseudo-labels refined by
CDMAD vs. true labels of unlabeled samples, and 2) class predictions refined
by CDMAD vs. true labels of test samples. These results are also compared to
those from FixMatch and ReMixMatch.

First, Fig. 7 compares the confusion matrices of pseudo-labels generated by
(a) FixMatch, (b) FixMatch+CDMAD, (c) ReMixMatch, and (d) ReMixMatch+CDMAD
trained on CIFAR-10-LT under $\gamma_{l}=100$ and $\gamma_{u}=100$. The value
in the $i$th row and $j$th column represents the proportion of the $i$th class
samples classified as the $j$th class. We can observe that the pseudo-labels
of FixMatch and ReMixMatch are biased toward the majority classes.
Specifically, the data points in the minority classes (e.g., classes 8 and 9)
are often misclassified into the majority classes (e.g. classes 0 and 1). In
contrast, Fig. 7 (b) and Fig. 7 (d) show that FixMatch+CDMAD and
ReMixMatch+CDMAD made nearly balanced class predictions.

|  |  |
---|---|---|---
(a) FixMatch |  (b) FixMatch+CDMAD | (c) ReMixMatch | (d) ReMixMatch+CDMAD
Figure 7: Confusion matrices of pseudo-labels generated by (a) FixMatch, (b) FixMatch+CDMAD, (c) ReMixMatch, and (d) ReMixMatch+CDMAD trained on CIFAR-10-LT under $\gamma_{l}=100$ and $\gamma_{u}=100$. |  |  |
---|---|---|---
(a) FixMatch | (b) FixMatch+CDMAD | (c) ReMixMatch | (d) ReMixMatch+CDMAD
Figure 8: Confusion matrices of the class predictions on the test set of
CIFAR-10 using (a) FixMatch, (b) FixMatch+CDMAD, (c) ReMixMatch, and (d)
ReMixMatch+CDMAD trained on CIFAR-10-LT under $\gamma_{l}=100$ and
$\gamma_{u}=100$.

Second, Fig. 8 compares the confusion matrices of the class predictions on the
test set of CIFAR-10 using (a) FixMatch, (b) FixMatch+CDMAD, (c) ReMixMatch,
and (d) ReMixMatch+CDMAD trained on CIFAR-10-LT under $\gamma_{l}=100$ and
$\gamma_{u}=100$. Similar to Fig. 7, FixMatch+CDMAD and ReMixMatch+CDMAD made
more balanced predictions across classes.

###  J.2 Case of $\gamma_{l}\neq\gamma_{u}$

In Table 2 of Section 4.2, the proposed algorithm performed better than the
baseline algorithms when the class distribution of the unlabeled set is
assumed to be unknown and actually differs with that of the labeled set. To
verify whether the pseudo-labels and class predictions refined by CDMAD
contributed to its superior performance, we conducted three types of
comparison: 1) pseudo-labels refined by CDMAD vs. true labels of unlabeled
samples, 2) representations learned with unrefined pseudo-labels vs.
representations learned with pseudo-labels refined by CDMAD, and 3) class
predictions refined by CDMAD vs. true labels of test samples. These results
are also compared to those from FixMatch and ReMixMatch.

First, Fig. 9 compares the confusion matrices of pseudo-labels generated by
(a) FixMatch, (b) FixMatch+CDMAD, (c) ReMixMatch, and (d) ReMixMatch+CDMAD
trained on CIFAR-10-LT under $\gamma_{l}=100$ and $\gamma_{u}=1$. The value in
the $i$th row and $j$th column represents the proportion of the $i$th class
samples classified as the $j$th class. We can observe that the pseudo-labels
of FixMatch and ReMixMatch are biased toward the majority classes.
Specifically, the data points in the minority classes (e.g., classes 7, 8 and
9) are often misclassified into the majority classes (e.g. classes 0 and 1).
In contrast, Fig. 9 (b) and Fig. 9 (d) show that FixMatch+CDMAD and
ReMixMatch+CDMAD made nearly balanced class predictions.

|  |  |
---|---|---|---
(a) FixMatch |  (b) FixMatch+CDMAD | (c) ReMixMatch | (d) ReMixMatch+CDMAD
Figure 9: Confusion matrices of pseudo-labels generated by (a) FixMatch, (b) FixMatch+CDMAD, (c) ReMixMatch, and (d) ReMixMatch+CDMAD trained on CIFAR-10-LT under $\gamma_{l}=100$ and $\gamma_{u}=1$. |  |  |
---|---|---|---
(a) FixMatch | (b) FixMatch+CDMAD | (c) ReMixMatch | (d) ReMixMatch+CDMAD
Figure 10: t-SNE of representations obtained for the test set of CIFAR-10
using (a) FixMatch, (b) FixMatch+CDMAD, (c) ReMixMatch, and (d)
ReMixMatch+CDMAD trained on CIFAR-10-LT under $\gamma_{l}=100$ and
$\gamma_{u}=1$.

Second, Fig. 10 compares t-distributed stochastic neighbor embedding (t-SNE)
[51] of representations obtained for the test set of CIFAR-10 using FixMatch,
FixMatch+CDMAD, ReMixMatch, and ReMixMatch+CDMAD trained on CIFAR-10 with
$\gamma_{l}=100$ and $\gamma_{u}=1$ (unknown $\gamma_{u}$), where different
colors indicate different classes in CIFAR-10. We can observe that the
representations obtained using FixMatch+CDMAD and ReMixMatch+CDMAD are
separated into classes with clearer boundaries compared the those from
FixMatch and ReMixMatchFrom in Fig. 10 (a) and Fig. 10 (c). This is probably
because CDMAD appropriately refined the biased pseudo-labels and used them for
training, whereas FixMatch and ReMixMatch failed to learn the representations
properly because they used the biased pseudo-labels for training. These
results demonstrate that the quality of representations can be improved by
using well refined pseudo-labels ( Fig. 9 (b) and Fig. 9 (d)) for training.

Third, Fig. 11 compares the confusion matrices of the class predictions on the
test set of CIFAR-10 using (a) FixMatch and (b) FixMatch+CDMAD trained on
CIFAR-10-LT under $\gamma_{l}=100$ and $\gamma_{u}=1$. Similar to Fig. 9,
FixMatch+CDMAD made more balanced predictions across classes compared to the
other algorithms. (Note that the results using ReMixMatch and ReMixMatch+CDMAD
are presented in Section 4.3.)

|
---|---
(a) FixMatch |  (b) FixMatch+CDMAD
Figure 11: Confusion matrices of the class predictions on the test set of
CIFAR-10 using (a) FixMatch and (b) FixMatch+CDMAD trained on CIFAR-10-LT
under $\gamma_{l}=100$ and $\gamma_{u}=1$.

##  Appendix K Further comparison with LA

Because CDMAD can be viewed as an extension of LA for incorporating awareness
of class distribution mismatch, we compared the classification performance of
LA and CDMAD for CISSL under the settings that the class distributions of the
labeled and unlabeled sets mismatch. To use LA for CISSL, we refined pseudo-
labels and class predictions on test samples by LA similar to CDMAD.
Experimental results are presented in Tab. 11. ReMixMatch+LA adjusts the
logits on inputs by the log of the class distribution of the labeled set by
assuming that the class distribution of the unlabeled set is the same as that
of the labeled set. ReMixMatch+LA* adjusts the logits on inputs by the log of
the class distribution of the whole training set by assuming that the class
distribution of the unlabeled set is known, although it differs from that of
the labeled set. From Tab. 11, we can observe that ReMixMatch+CDMAD performed
significantly better than both ReMixMatch+LA and ReMixMatch+LA*. This may be
because CDMAD refined the biased pseudo-labels and class predictions on test
samples more effectively than ReMixMatch+LA and ReMixMatch+LA* by
incorporating awareness of class distribution mismatch. It should be noted
that LA* cannot re-balance the classifier to an appropriate degree even if the
class distribution of the unlabeled set is known under the class distribution
mismatch setting. This may be because in SSL, each labeled data point is
typically used more frequently and importantly than each unlabeled data point.
Consequently, the classifier may become biased towards the class distribution
of the labeled set to a greater degree than the class distribution of the
entire training set, while still being affected by the class distribution of
the unlabeled set.

Table 11: bACC/GM on CIFAR-10-LT under $\gamma_{l}\neq\gamma_{u}$.

|  | CIFAR-10-LT ($\gamma_{l}=100$)
---|---|---
Algorithm | $\gamma_{u}=1$ | $\gamma_{u}=50$ | $\gamma_{u}=150$
ReMixMatch+LA |  $76.6$/ $66.8$ |  $69.9$/ $52.6$ |  $70.5$/ $42.7$
ReMixMatch+LA* |  $69.2$/ $54.0$ |  $73.7$/ $70.8$ |  $58.3$/ $27.4$
ReMixMatch+CDMAD | 89.9/ 89.6 | 86.9/ 86.7 | 83.1/ 82.7

##  Appendix L Fine grained experimental results

To verify that CDMAD improves classification performance for minority classes,
we performed experiments using FixMatch/ ReMixMatch and
FixMatch/ReMixMatch+CDMAD on CIFAR-10-LT and measured the accuracy for
Many/Medium/Few groups separately (for CIFAR-10-LT, we set the first three
classes as many shot groups, then next four classes as medium shot groups, and
then last three classes as few shot groups). We also measured the fine grained
classification performance of FixMatch/ReMixMatch+CoSSL [14] on CIFAR-10-LT
and compared them with those of CDMAD for comparison with a recent CISSL
algorithm. The results are summarized in Tab. 12, Tab. 13, and Tab. 14. We can
observe that FixMatch+CDMAD and ReMixMatch+CDMAD greatly improved accuracy for
few shot groups with only slightly decreased accuracy for many shot groups
compared to FixMatch and ReMixMatch. We can also observe that FixMatch/
ReMixMatch+CDMAD achieved better medium and few shot classification accuracies
than FixMatch/ ReMixMatch+COSSL. These results demonstrate that CDMAD
effectively relieves class imbalance.

Table 12: Fine grained experimental results under $\gamma_{l}=\gamma_{u}=100$.

CIFAR-10-LT ($\gamma_{l}=\gamma_{u}=100$)
---
Algorithm | Overall | Many | Medium | Few
FixMatch | $72.5$ | $95.0$ | $74.6$ | $47.3$
FixMatch+CDMAD | $83.6$ | $91.9$ | $82.2$ | $77.2$
ReMixMatch | $74.3$ | $96.7$ | $77.8$ | $47.2$
ReMixMatch+CDMAD | $85.5$ | $90.1$ | $84.8$ | $81.8$

Table 13: Fine grained experimental results under $\gamma_{l}=100$, and
$\gamma_{u}=1$.

CIFAR-10-LT ($\gamma_{l}=100$, $\gamma_{u}=1$)
---
Algorithm | Overall | Many | Medium | Few
FixMatch | $70.2$ | $96.3$ | $77.7$ | $34.0$
FixMatch+CDMAD | $87.5$ | $95.6$ | $86.4$ | $80.9$
ReMixMatch | $65.4$ | $96.6$ | $70.8$ | $27.0$
ReMixMatch+CDMAD | $89.9$ | $96.5$ | $87.8$ | $86.0$

Table 14: Fine grained experimental results under $\gamma_{l}=\gamma_{u}=100$.

CIFAR-10-LT ($\gamma_{l}=\gamma_{u}=100$)
---
Algorithm | Overall | Many | Medium | Few
FixMatch+CoSSL | $83.2$ | $93.4$ | $81.1$ | $75.8$
FixMatch+CDMAD | $83.6$ | $91.9$ | $82.2$ | $77.2$
ReMixMatch+CoSSL | $84.1$ | $91.7$ | $82.1$ | $79.1$
ReMixMatch+CDMAD | $85.5$ | $90.1$ | $84.8$ | $81.8$

##  Appendix M Comparing CDMAD with DASO

Table 15: bACC/GM on CIFAR-10-LT under $\gamma=\gamma_{l}=\gamma_{u}$.

CIFAR-10-LT ($\gamma=\gamma_{l}=\gamma_{u}$)
---
Algorithm | $\gamma=50$ | $\gamma=100$ | $\gamma=150$
FixMatch+DASO |  $81.8$/ $81.0$ |  $75.7$/ $74.0$ |  $72.0$/ $68.9$
FixMatch+DASO+LA |  $84.1$/ $83.7$ |  $79.4$/ $78.8$ |  $76.5$/ $75.5$
FixMatch+CDMAD | 87.3/87.0 | 83.6/83.1 | 80.8/79.9
ReMixMatch+DASO |  $82.5$/ $81.9$ |  $76.0$/ $73.9$ |  $70.8$/ $66.5$
ReMixMatch+DASO+LA |  $85.9$/ $85.7$ |  $82.8$/ $82.4$ |  $79.0$/ $78.4$
ReMixMatch+CDMAD | 88.3/ 88.1 | 85.5/ 85.3 | 82.5/ 82.0

Table 16: Comparison of bACC/GM on CIFAR-10-LT and STL-10-LT under
$\gamma_{l}\neq\gamma_{u}$.

|  | CIFAR-10-LT ($\gamma_{l}=100$) | STL-10-LT ($\gamma_{u}=$Unknown)
---|---|---|---
Algorithm | $\gamma_{u}=1$ | $\gamma_{u}=50$ | $\gamma_{u}=150$ | $\gamma_{l}=10$ | $\gamma_{l}=20$
FixMatch+DASO |  $86.4$/ $86.0$ |  $79.1$/ $78.2$ |  $74.2$/ $71.6$ |  $68.4$/ $65.3$ |  $62.1$/ $58.9$
FixMatch+DASO+LA |  $86.2$/ $85.8$ |  $81.7$/ $81.2$ |  $78.0$/ $77.0$ |  $68.9$/ $66.3$ |  $66.0$/ $64.6$
FixMatch+CDMAD | 87.5/ 87.1 | 85.7/ 85.3 | 82.3/ 81.8 | 79.9/ 78.9 | 75.2/ 73.5
ReMixMatch+DASO |  $89.6$/ $89.3$ |  $79.6$/ $77.8$ |  $72.3$/ $69.0$ |  $75.1$/ $73.6$ |  $66.8$/ $61.8$
ReMixMatch+DASO+LA |  $80.6$/ $77.7$ |  $84.8$/ $84.5$ |  $79.7$/ $79.2$ |  $78.1$/ $77.3$ |  $75.3$/ $74.0$
ReMixMatch+CDMAD | 89.9/ 89.6 | 86.9/ 86.7 | 83.1/ 82.7 | 83.0/82.1 | 81.9/80.9

Table 17: Comparison of bACC on CIFAR-100-LT.

CIFAR-100-LT ($\gamma=\gamma_{l}=\gamma_{u}$)
---
Algorithm | $\gamma=20$ | $\gamma=50$ | $\gamma=100$
FixMatch+DASO | $45.8$ | $39.2$ | $33.9$
FixMatch+DASO+LA | $46.2$ | $39.9$ | $34.5$
FixMatch+CDMAD | 54.3 | 48.8 | 44.1
ReMixMatch+DASO | $51.5$ | $43.0$ | $38.2$
ReMixMatch+DASO+LA | $52.8$ | $45.5$ | $40.3$
ReMixMatch+CDMAD | 57.0 | 51.1 | 44.9

Because classification performance of DASO were measured under slightly
different settings from ours, it was difficult to fairly compare their
classification performance with that of CDMAD in the main paper. Nevertheless,
in the case of DASO, we conducted experiments in the same setting as ours
using the official code in github. The classification performance of DASO is
summarized in Tab. 15, Tab. 16, and Tab. 17. From Tab. 15, Tab. 16, and Tab.
17, we can observe that the proposed algorithm outperforms DASO. From Tab. 16,
we can also observe that combining DASO with LA degrades the classification
performance when the class distributions of the labeled and unlabeled sets
severely differ. This may be because the LA considers only the class
distribution of the labeled set when the class distribution of the unlabeled
set is unknown. These results show the importance of re-balancing the
classifier by considering the class distribution of the unlabeled set. These
results demonstrate the effectiveness of CDMAD.