# Do not trust what you trust: Miscalibration in Semi-supervised Learning

###### Abstract

State-of-the-art semi-supervised learning (SSL) approaches rely on highly
confident predictions to serve as pseudo-labels that guide the training on
unlabeled samples. An inherent drawback of this strategy stems from the
quality of the uncertainty estimates, as pseudo-labels are filtered only based
on their degree of uncertainty, regardless of the correctness of their
predictions. Thus, assessing and enhancing the uncertainty of network
predictions is of paramount importance in the pseudo-labeling process. In this
work, we empirically demonstrate that SSL methods based on pseudo-labels are
significantly miscalibrated, and formally demonstrate the minimization of the
min-entropy, a lower bound of the Shannon entropy, as a potential cause for
miscalibration. To alleviate this issue, we integrate a simple penalty term,
which enforces the logit distances of the predictions on unlabeled samples to
remain low, preventing the network predictions to become overconfident.
Comprehensive experiments on a variety of SSL image classification benchmarks
demonstrate that the proposed solution systematically improves the calibration
performance of relevant SSL models, while also enhancing their discriminative
power, being an appealing addition to tackle SSL tasks. The code is publicly
available at https://github.com/ShambhaviCodes/miscalibration-ssl.

###### Keywords:

Semi-supervised learning Calibration Uncertainty

##  1 Introduction

Deep learning models have significantly advanced the state-of-the-art across a
myriad of tasks [25, 26]. Nonetheless, their success has been often contingent
on the availability of large amounts of labeled data. Having access to curated
large training datasets, however, is not easy, and often involves a tremendous
human labor, particularly in those domains where labeling data samples
requires expertise, hindering the progress to address a broader span of real-
world problems.

Semi-supervised learning (SSL) [4] mitigates the need for large labeled
datasets by providing means of efficiently leveraging unlabeled samples, which
are easier to obtain. This learning paradigm has led to a plethora of
approaches, which can be mainly categorized into consistency regularization
[1, 18] and pseudo-labeling [20, 46] methods. Indeed, state-of-the-art SSL
approaches [44, 49, 39, 6, 50] combine both strategies, obtaining promising
results. The underlying idea of these approaches follows the low-density and
smoothness assumptions in SSL [5]. In particular, incorporating pseudo-labels
from unlabeled data points into the training process aids the decision
boundary to lie in low density regions. Furthermore, consistency
regularization assumes that the same unlabeled data point should yield the
same pseudo-label regardless of the perturbations applied, implicitly
capturing the underlying data manifold. As the model can produce very
uncertain predictions for strongly perturbed samples, these techniques
incorporate a threshold, either fixed [39] or adaptive [44, 49], to only
integrate very confident samples in the training loss. Thus, all samples whose
predicted probabilities are highly confident are trusted by these methods as
supervisory signals for subsequent steps, even when their predictions are
wrong.

Despite being a standard practice, recent evidence [6] suggests that the
amount of incorrect pseudo-labels integrated into the training is not
negligible, potentially undermining the optimization process. Hence, given
that the generated pseudo-labels play a significant role in the training of
SSL models, producing accurate uncertainty estimates is of pivotal importance.
Nevertheless, while we have observed a remarkable progress in their
discriminative performance, little attention has been paid to studying, and
improving, the calibration of SSL approaches. Motivated by these findings, in
this work we address the critical yet under-explored issue of miscalibration
in SSL, particularly for those methods based on pseudo-labeling. To this end,
we select a set of relevant and recent strategies that build on pseudo-labels
and consistency regularization [39, 44, 49] and empirically demonstrate that
they are poorly calibrated. Furthermore, we explore the underlying causes of
this issue and shed light about the potential reasons that produce
overconfident pseudo-labeling SSL models. Last, inspired by these observations
we propose a simple solution to tackle miscalibration in these models. Our
contributions can be therefore summarized as follows:

  1. 1.

We empirically demonstrate that state-of-the-art SSL approaches based on
pseudo-labels are significantly miscalibrated. Through our analysis, we
formally show that the cause of miscalibration is the minimization of a min-
entropy term, a specific case of the Rényi family of entropies, on a
considerably large proportion of unlabeled samples, which forces the model to
yield overconfident predictions. Indeed, the ensuing gradients from this term
strongly push the unlabeled samples to be highly confident since the beginning
of the training, even if their class predictions are incorrect. This results
in large logit magnitudes, a phenomenon known to cause miscalibration.

  2. 2.

Based on our observations, we propose to use a simple solution that refrains
the model from pushing unlabeled samples towards very unconfident regions,
improving the calibration of pseudo-labeling SSL methods. More concretely, we
add a penalty term on a dominant set of unlabeled samples, which enforces
logit distances to remain low, alleviating the miscalibration issue.

  3. 3.

Through a comprehensive set of experiments, we empirically demonstrate that
the proposed approach consistently improves the uncertainty estimates of a set
of very relevant and recent state-of-the-art pseudo-labeling SSL approaches on
popular benchmarks, in both standard and long-tailed classification tasks. In
addition, in most cases, the proposed solution further improves their
discriminative performance.

##  2 Related work

###  2.1 Semi-supervised learning

Prevailing semi-supervised learning (SSL) approaches heavily rely on the
concept of pseudo-labels [20, 38] and consistency regularization [1, 18, 37,
41, 28], where labels are dynamically generated for unlabeled data throughout
the training process. Essentially, these methods exploit the role of
perturbations, by stochastically perturbing the unlabeled images and enforcing
consistency across their predictions. This consistency is achieved by a
pseudo-supervised loss, where the predictions over the strong perturbations
are supervised by the pseudo-labels obtained from the weak perturbations [45,
44, 49, 6, 50, 48, 47, 50]. This paradigm to use artificial labels facilitates
the integration of unlabeled data into the learning process, thereby
augmenting the training set and improving the model ability to generalize. To
avoid introducing noise in the pseudo-supervision process, these approaches
retain a given pseudo-label only if the model assigns a high probability to
one of the possible classes. This strategy effectively harnesses the
information from unlabeled data by leveraging the network confidence in
assigning pseudo labels, enabling the model to access the valuable knowledge
encapsulated within these unlabeled samples. Thus, the main differences across
the different approaches based on pseudo-labels lie on the mechanism
introduced to select confident samples. For example, FixMatch [39] and
ShrinkMatch [48] employ a fixed threshold, whereas Dash [47] proposes a
dynamically growing threshold. Other approaches, such as FlexMatch [49] and
FreeMatch [44], integrate class-adaptative thresholds, considering a larger
amount of unlabeled data which is otherwise ignored, especially at the early
stage of the training process.

Limitations of pseudo-labeling SSL from a calibration standpoint. Although
these methods enhance the discriminative power of deep models, their
calibration has been significantly overlooked, lacking of principled
strategies to simultaneously improve the classification performance while
maintaining the quality of the uncertainty estimates. As pseudo-labeling
state-of-the-art SSL approaches trust highly confident artificial labels
derived from unlabeled samples, understanding how this confidence is assigned,
and ensure its accuracy, is of paramount importance. Very recently, only BAM
[24] studied miscalibration in SSL, and proposed to replace the last layer of
a neural network by a Bayesian layer. Nevertheless, the source of
miscalibration was not explored, and in-depth empirical results were not
reported.

###  2.2 Calibration

Recent evidence [15, 30, 29] has shown that deep networks are prone to make
overconfident predictions due to miscalibrated output probabilities. This
emerges as a byproduct of minimizing the prevalent cross-entropy loss, which
occurs when the softmax predictions for all training samples fully match the
ground-truth labels, and thus the entropy of output probabilities is
encouraged to be zero. To mitigate the miscalibration issue, and to better
estimate the predictive uncertainty of deterministic models, two main families
of approaches have emerged recently: post-processing [15, 10, 42] and learning
[34, 30, 29, 22, 7, 23, 32, 31, 19, 33] approaches. Among the post-processing
strategies, Temperature Scaling (TS) [15] has been a popular alternative,
which manipulates logit outputs monotonely, by applying a single scalar
temperature parameter. This idea is further extended in [10], where a local TS
per pixel is provided by a regression neural network. Nevertheless, despite
the simplicity of these methods, learning approaches have arisen as a more
powerful choice, as in this scenario the model adapts to calibration
requirements alongside its primary learning objectives, optimizing both
aspects simultaneously. Initial attempts integrated learning objectives that
maximize the entropy of the network softmax predictions either explicitly [34,
19], or implicitly [29, 30, 7]. In order to alleviate the non-informative
nature of simply maximizing the entropy of the softmax predictions, recent
work [22, 23] has presented a generalized inequality constraint, which
penalizes logits distances larger than a pre-defined margin.

##  3 Semi-supervised learning and calibration

###  3.1 Problem statement

In the semi-supervised learning scenario, the training dataset is composed of
labeled and unlabeled data points. In this setting, let
$\mathcal{D}_{L}=\\{(\mathbf{x}_{i},\mathbf{y}_{i})\\}_{i}^{N_{L}}$ be the
labeled dataset and $\mathcal{D}_{U}=\\{\mathbf{x}_{i}\\}_{i}^{N_{U}}$ the
unlabeled dataset, where $N_{L}$ and $N_{U}$ represent the number of labeled
and unlabeled samples, respectively, and $N_{L}<<N_{U}$. Furthermore,
$\mathbf{x}_{i}\in^{d}$ is a $d$-dimensional training sample, with
$\mathbf{y}_{i}~{}\in~{}\\{0,1\\}^{K}$ its associated ground truth (only for
labeled data points, i.e., $\mathbf{x}_{i}\in\mathcal{D}_{L}$) that assigns
one of the $K$ classes to the sample. The objective is, given a batch of
labeled and unlabeled samples, to find an optimal set of parameters of a
deterministic function, e.g., a neural network, parameterized by
$\boldsymbol{\theta}$, by using a compounded loss including a labeled and an
unlabeled term. Given an input image $\mathbf{x}_{i}$, the network will
generate a vector of logits
$f_{\boldsymbol{\theta}}(\mathbf{x}_{i})=\mathbf{l}_{i}\in^{K}$, which can be
converted to probabilities with the softmax function.

Supervised loss. The supervised objective is typically formulated as a
standard cross-entropy $\mathcal{H}$ between the one-hot encoded labels
$\mathbf{y}_{i}$ and the corresponding softmax predictions
$\mathbf{p(y|x_{i})}\in[0,1]^{K}$ of labeled samples:

$\displaystyle\mathcal{L}_{S}=\mathcal{H}(\mathbf{y}_{i},\mathbf{p(y|x_{i})})=-\sum_{i\in\mathcal{D}_{L}}\mathbf{y}_{i}\log\mathbf{p(y|x_{i})}$
(1)

Unsupervised loss. Most modern SSL approaches adopt a consistency
regularization strategy based on pseudo-labeling for the unsupervised
objective. To this end, the same image $\mathbf{x}_{i}$ follows a set of weak
and strong augmentations, denoted as $\omega(\cdot)$ and $\Omega(\cdot)$,
respectively. Thus, a pseudo-cross-entropy term on the unlabeled training
dataset can be formulated as:

$\displaystyle\mathcal{L}_{U}=-\sum_{i\in\mathcal{D}_{U}}\tilde{\mathbf{y}}_{i}\log\mathbf{p}(\mathbf{y}|\Omega(\mathbf{x}_{i}))$
(2)

where $\tilde{\mathbf{y}}_{i}$ is the one-hot encoding of the $\arg\max$ of
the softmax probabilities for the weak augmented version, i.e.,
$\arg\max(\mathbf{p}(\mathbf{y}|\omega(\mathbf{x}_{i}))$. To avoid that
samples with high uncertainty, and possibly incorrect predictions, intervene
in the optimization of the term in Eq. 2, a common strategy is to retain only
discrete pseudo-labels whose largest class probability fall above a predefined
threshold [20, 44, 39, 49]. This results in the following objective:

$\displaystyle\mathcal{L}_{U}=-\sum_{i\in\mathcal{D}_{U}}\mathbbm{1}(\max(\mathbf{p}(\mathbf{y}|\omega(\mathbf{x}_{i})))\geq\tau)\tilde{\mathbf{y}}_{i}\log\mathbf{p}(\mathbf{y}|\Omega(\mathbf{x}_{i}))$
(3)

where $\tau$ is the predefined threshold. Note that all loss terms are
normalized by the cardinality of each set, which we omit for simplicity.

###  3.2 Revisiting the calibration of semi-supervised models

We now introduce a series of observations revealing several intrinsic
properties of semi-supervised methods built up on pseudo-labels generation,
which allows us to motivate a calibration technique tailored for pseudo-label
based SSL.

Observation 1. Semi-supervised learning degrades the calibration performance.
An important body of literature on SSL relies on pseudo-labeling to leverage
the large amount of unlabeled samples. To achieve this, a very common strategy
is to generate weak augmentations of each unlabeled image, whose predictions
serve as supervision for their strong augmentation counterpart, as presented
in Eq. 3. Pseudo-labeling [20] is indeed closely related to entropy
regularization [14], which favors a low-density separation between classes, a
commonly assumed prior for semi-supervised learning. While minimizing the
entropy of the predictions can actually improve the discriminative performance
of neural networks, it inherently favours overconfident predictions, which is
one of the main causes of miscalibration. Figure 1 brings empirical evidence
about this observation, where we can observe that despite bringing performance
gains, in terms of accuracy, pseudo-label based SSL methods naturally degrade
the calibration properties of a supervised baseline trained with a few labeled
samples.

Figure 1: Observation 1. Reliability plots for a baseline supervised model
(trained with Eq. 1) and three representative SSL approaches (trained with Eq.
4) on CIFAR-100. These plots empirically highlight the calibration degradation
observed when training with the standard unsupervised loss, despite the gains
achieved in discrimination.

Observation 2. Pseudo-labeling in SSL indeed minimizes min-entropy on
unlabeled points. The use of a hard label makes pseudo-labeling closely
related to entropy minimization [14]. Nevertheless, the different
transformations that unlabeled images follow in modern SSL methods produce
different probability distributions for the weak and strong versions of the
same image, where the predictions of the former are used to correct the
predictions of the later. Note that this is slightly different from the
traditional pseudo-labeling approaches, where the same image is used for
assigning the pseudo-label and updating the predictive model. Thus, albeit
they are related, the minimization of entropy cannot be attributed as the
cause of miscalibration. Motivated by this, we explore in this section the
implications of the standard SSL unlabeled loss based on pseudo-labels and its
effect on network calibration. The common learning objective is composed of
two terms: the first one is the standard cross-entropy (CE) on labeled samples
(Eq. 1), while the second term is a CE between pseudo-labels obtained from
weak augmentations and the predictions of their strong augmented counterparts
(Eq. 3):

$\displaystyle\mathcal{L}_{T}=\underbrace{-\sum_{i\in\mathcal{D}_{L}}\mathbf{y}_{i}\log\mathbf{s}_{i}}_{\textrm{CE
on labeled
samples}}\underbrace{-\sum_{i\in\mathcal{D}_{U}}\tilde{\mathbf{y}}^{w}_{i}\log\mathbf{s}^{s}_{i}}_{\textrm{Pseudo-
CE on $\mathcal{D}_{U}$}},$ (4)

where $\mathbf{s}_{i}=\mathbf{p(y|x_{i})}$ is used for simplicity, and the
superscripts $w$ and $s$ denote weak and strong transformations, respectively.
We now split the unlabeled dataset into $\mathcal{D_{U^{\prime}}}$, which
contains the unlabeled samples whose predicted class from weak and strong
augmentations are different, i.e.,
$\arg\max(\mathbf{s}^{w}_{i})\neq\arg\max(\mathbf{s}^{s}_{i})$ and
$\mathcal{D_{U^{\prime\prime}}}$, containing the samples whose predicted class
from weak and strong augmentations are the same. Thus, we can decompose the
right-hand term in Eq. 4 into two terms, one acting over
$\mathcal{D_{U^{\prime}}}$ and one over $\mathcal{D_{U^{\prime\prime}}}$:

$\displaystyle\mathcal{L}_{T}=\underbrace{-\sum_{i\in\mathcal{D}_{L}}\mathbf{y}_{i}\log\mathbf{s}_{i}}_{\textrm{CE
on labeled
samples}}\underbrace{-\sum_{i\in\mathcal{D}_{U^{\prime}}}\tilde{\mathbf{y}}^{w}_{i}\log\mathbf{s}^{s}_{i}}_{\textrm{Pseudo-
CE on
$\mathcal{D}_{U^{\prime}}$}}-\sum_{i\in\mathcal{D}_{U^{\prime\prime}}}\tilde{\mathbf{y}}^{w}_{i}\log\mathbf{s}^{s}_{i}$
(5)

In the above equation, the second term can be seen as a pseudo cross-entropy
on $\mathcal{D}_{U^{\prime}}$. Furthermore, as $\tilde{\mathbf{y}}^{w}_{i}$ is
equal to the one-hot vector from $\arg\max(\mathbf{s}^{s}_{i})$ on samples
from $\mathcal{D}_{U^{\prime\prime}}$, the last term is equivalent to the min-
entropy111Pseudo-label $\hat{y}_{i,c}^{w}=1$ if
$\text{s}^{s}_{i,c}=\max_{k}\text{s}^{s}_{i,k}$, and 0 otherwise.:

$\displaystyle\mathcal{L}_{T}=\underbrace{-\sum_{i\in\mathcal{D}_{L}}\mathbf{y}_{i}\log\mathbf{s}_{i}}_{\textrm{CE
on labeled
samples}}\underbrace{-\sum_{i\in\mathcal{D}_{U^{\prime}}}\tilde{\mathbf{y}}^{w}_{i}\log\mathbf{s}^{s}_{i}}_{\textrm{Pseudo-
CE on
$\mathcal{D}_{U^{\prime}}$}}\underbrace{-\sum_{i\in\mathcal{D}_{U^{\prime\prime}}}\log(\max_{k}\text{s}^{s}_{i,k})}_{\textrm{min-
entropy on $\mathcal{D}_{U^{\prime\prime}}$}}$ (6)

As shown in Figure 2 in the case of a two-class distribution $(p,1-p)$, min-
entropy is a lower bound of the Shannon Entropy, which has several
implications in network miscalibration. In particular, while both Shannon
Entropy and min-entropy reach their minimum at the vertices of the simplex,
i.e., when $p~{}=~{}0$ or $p~{}=~{}1$ (left), the dynamics of the gradients
are different (middle). More concretely, in the case of the Shannon entropy,
the gradients of low-confidence predictions at the middle of the simplex are
small and, therefore, dominated by the other terms at the beginning of
training. In contrast, by minimizing the min-entropy, the inaccuracies
resulting from uncertain predictions are reinforced, i.e., pushed towards the
simplex vertices, yielding early errors in the predictions, which are hardly
recoverable, and potentially misleading the training process.

Figure 2: Observation 2. (Left)The unsupervised term in pseudo-label SSL is
(approximately) equivalent to min-entropy, a lower bound of the Shannon
Entropy. (Middle) Compared to the Shannon Entropy, the min-entropy is more
aggressive in the gradient dynamics, particularly at the beginning of the
training, when most predictions are uncertain. (Right) Ratio of samples with
same hard prediction for weak and strong augmentations that were above the
selection threshold of three relevant SSL methods.

Furthermore, we empirically observed that the amount of samples where the
$\arg\max$ of the predictions from weak and strongly augmented versions was
the same, i.e., $\mathcal{D}_{U^{\prime\prime}}$, was significantly larger
than those with different predictions, i.e., $\mathcal{D}_{U^{\prime}}$
(Figure 2, right). In particular, as shown in this figure, and after some
iterations, more than 80% of unlabeled samples included in the training share
the same pseudo-label between weak and strong annotations, regardless of the
approach analyzed. Thus, based on our observations we argue that the training
of SSL methods based on pseudo-labels can be approximated by a supervised term
coupled with a regularization loss that minimizes the min-entropy of unlabeled
samples. This means that, while implicitly, or explicitly, minimizing the
Shannon entropy on the network softmax predictions is known to cause
miscalibration [15], employing the min-entropy aggravates the problem, which
explains why SSL methods based on pseudo-labels are not well calibrated,
particularly compared to a simple supervised baseline (Figure 1).

Observation 3. Pseudo-label SSL techniques produce highly overlapped logit
distributions, with large logit magnitudes and distances. A direct implication
of Observation 2 is that softmax predictions in pseudo-labeling SSL methods
become highly confident, which translates into larger logit values compared to
a supervised baseline. As a result, even when a predicted category is
incorrect, the network will still express a high level of certainty in its
prediction. Take for instance the example shown in Figure 3, which depicts the
logit value distributions for the samples belonging to class 5. We can observe
that, in the case of the supervised baseline, the maximum logit values for
incorrect predicted classes is at $\approx$ 7.5 (classes 6 and 7) and around
6.5 (class 8). In contrast, when adding unlabeled samples in the form of
pseudo-labels (e.g., in FixMatch [39]), the maximum logit values for incorrect
classes increase to nearly 12.5. This means that, while both models yield
incorrect predictions, these are highly confident in the SSL methods.
Furthermore, the logit range in the supervised baseline goes from around -7 to
10, whereas FreeMatch produces logits in the range from -10 to 15, which will
result in highest probability scores for the predicted category. From these
observations, we argue that a well-calibrated SSL model should decrease the
magnitude of logits associated with incorrect classes, as well as their total
logits range, thereby decreasing confidence in erroneous predictions, while
simultaneously preserving high values in logits corresponding to the target
class.

Figure 3: Observation 3. These plots depict the Kernel Density Estimation of
the logit distributions obtained by (left) the supervised baseline trained
with Eq. 1 and (right) FreeMatch on STL-10, for the samples belonging to class
5. We can observe that, even for non-target classes (k$\neq$5), the logit
magnitudes in FreeMatch are larger, which translates to higher overconfidence
in both correct and incorrect predictions. We select STL-10 due to its number
of classes (10 vs. 100 in CIFAR-100).

##  4 Our solution

Based on our findings, especially in Observation 3, it becomes evident that an
effective strategy for addressing miscalibration within the SSL scenario
involves controlling the magnitude of predicted logits for unlabeled samples.
Furthermore, from Observation 2 we can derive that the underlying mechanism
magnifying the miscalibration issue stems from the hidden min-entropy term on
the data points in $\mathcal{D}_{U^{\prime\prime}}$, which represents the
majority of unlabeled samples. Thus, we resort to an inequality constraint
that imposes a controllable margin on the logit distances of predictions in
samples from $\mathcal{D}_{U^{\prime\prime}}$. This constraint, which draws
inspiration from [22], takes the following form
$\mathbf{d}(\mathbf{l})\leq\mathbf{m}$, where
$\mathbf{d}(\mathbf{l})=(\max_{j}(l_{j})-l_{k})_{1\leq k\leq
K}\in\mathbb{R}^{K}$ represents the vector of logit distances between the
winner class and the rest (only for samples in
$\mathcal{D}_{U^{\prime\prime}}$) and m a $K$-dimensional vector defining the
margin values, with all elements equal to $m\in_{++}$. Note that while [22]
was proposed in the fully supervised learning scenario, in this work we only
enforce the constraint on a subset of samples, which is motivated by the
observations presented in Sec. 3.2. Furthermore, this choice is also supported
empirically in the ablation studies presented in the experiments.

By integrating this inequality constraint on the logit distances, training
becomes a constrained problem, whose objective can formally defined as:

$\begin{gathered}\text{minimize}\quad\mathcal{L}_{\text{T}}\\\ \text{subject
to}\quad\mathbf{d}(\mathbf{l})\leq\mathbf{m}\qquad\mathbf{m}\in_{++},\forall\mathbf{x}_{i}\in\mathcal{D}_{U^{\prime\prime}}.\end{gathered}$
(7)

The above constrained problem in Eq. 7 can be approximated by penalty-based
optimization method, transforming the formulation into an unconstrained
problem by using a simple ReLU function:

$\displaystyle\min_{\boldsymbol{\theta}}\quad$
$\displaystyle\mathcal{L}_{\text{T}}+\lambda\sum_{i\in\mathcal{D}_{U^{\prime\prime}}}\sum_{k}\max(0,\max_{j}(l_{i,j})-l_{i,k}-m_{k})$
(8)

where the second term, i.e., the non-linear ReLU penalty, prevents logit
distances from exceeding a predetermined margin $m$, and $\lambda\in_{+}$ is a
blending hyperparameter which controls the contribution of the CE loss and the
corresponding penalty. The intuition behind this penalty term is simple. For
the winner logits where the distance with the remaining logits is above the
margin $m$, a gradient will be back-propagated to enforce those values to
decrease. As a result, the whole logit magnitudes will decrease, potentially
alleviating the miscalibration issue in the set of unlabeled samples
$\mathcal{L}_{U^{\prime\prime}}$, which dominate the SSL training.

##  5 Experiments

Datasets. We resort to the recent Unified Semi-supervised Learning Benchmark
for Classification (USB) [43], which compiles a diverse and challenging
benchmark across several datasets. In particular, we focus on three popular
datasets: CIFAR-100 [17], which has significant value as a standard for fine-
grained image classification due to its wide range of classes and detailed
object distinctions; STL-10 [8], which is widely recognized for its limited
sample size and extensive collection of unlabeled data, rendering it a
challenging scenario of special significance in the context of SSL; and
EuroSAT [16], containing 10 unique fine-grained categories related to earth
observation and satellite imagery analysis, and important challenges such as
high variability and imbalance classes. Last, we also conduct further
experiments in the long-tailed version of CIFAR-100.

Architectures. We have prioritized Vision Transformers (ViT) over
Convolutional Neural Networks (CNNs), for three main reasons related to
discriminative performance, quality of uncertainty estimates, generalization
and transfer learning capabilities. First, the emergence of ViTs has proven
these models to outperform their CNNs counterparts [36, 3]. Second, from a
calibration standpoint, ViTs have also shown to be better calibrated than CNNs
[27, 35]. Hence, due to their superior discriminative and calibration
performance, they pose a more challenging scenario to evaluate the
effectiveness of the proposed strategy. And last, the fine-tuning capabilities
of ViTs enable effective transfer learning across diverse visual tasks and
datasets. This capability is particularly advantageous in scenarios where
labeled data is scarce, as it allows leveraging pre-trained representations
learned from large-scale datasets, significantly minimizing the amount of
training iterations while maintaining consistent performance222Training
FreeMatch on CIFAR100 with 400 labeled samples goes from 12 days (WideResNet
from scratch) to 10 hours (ViTSmall) in an NVIDIA V100-32G GPU.. More
concretely, we employ a ViTSmall [13] with a patch size of 2 and an image size
of 32 for CIFAR-100 and EuroSAT, in accordance with the standard in USB, and a
ViT-Small with an image size of 96 for STL10. We use same settings for all
models and benchmarks to provide a fair comparison.

Training, evaluation protocol and metrics. While several works use different
amounts of labeled data in their experiments, we perform due diligence and
follow the settings proposed in USB [43] for training. For each method and
configuration, we perform three runs with different seeds, select the best
checkpoint and report their mean and standard deviation, following the
literature. We report error rates for the accuracy performance and the
expected calibration error (ECE), following the literature in calibration of
supervised models. Implementation details are discussed in Appendix.

###  5.1 Results

Main results. The proposed strategy is model agnostic, and can be integrated
on top of any SSL approach based on pseudo-labels, enabling substantial
flexibility. For the empirical evaluation, we selected three popular and
relevant approaches that resort to hard pseudo-labels: FixMatch [39],
FlexMatch [49] and FreeMatch [44] and assess the impact of adding our simple
solution during training. Discriminative performance (Tab. 1): we observe that
in 16 out of the 18 different settings, adding the penalty in Eq. 8 brings
improvement gains compared to the original versions of each method, which are
only trained with Eq. 4. Note that these gains are sometimes substantial,
improving the original method by up to 4% (e.g., FixMatch in CIFAR-100(200)
and EuroSAT(20) or FlexMatch in EuroSAT(20)). Furthermore, the three
approaches combined with our penalty achieve very competitive performance
compared to existing SSL literature, typically yielding state-of-the-art
results. Calibration performance (Tab. 2): Similarly, including the penalty
term systematically enhances the calibration performance of the three analyzed
approaches, whose improvements are typically significant (up to 6-7% in
several cases). An interesting observation is that, MixMatch [2], a
consistency regularization based approach, yields surprisingly well calibrated
models. Indeed, MixMatch has several components that have shown to improve
calibration, such as ensembling predictions and MixUp, which may explain the
obtained values. Nevertheless, it is noteworthy to mention that the
discriminative performance compared to our modified versions of SSL methods is
strikingly lower, with differences typically going from 10% to 24%, failing to
achieve a good compromise between accuracy and calibration.

Table 1: Classification performance (error rate (%)). Arrows indicate whether our modified version improves ($\downarrow$) or deteriorates ($\uparrow$) the performance. Best overall performance in bold and best across pseudo-label SSL approaches underlined. Dataset | CIFAR-100 | EuroSAT | STL-10
---|---|---|---
# Labeled samples |  200 |  400 |  20 |  40 |  40 |  100
Only consistency regularization
MixMatch${}_{\text{NeurIPS'19}}$ |  37.68±2.66 |  26.84±1.06 |  28.77±10.40 |  14.88±2.07 |  25.19±2.05 |  11.37±1.49
Dash${}_{\text{ ICML'21}}$ |  28.51±2.91 |  19.54±1.20 |  10.05±8.15 |  6.83±3.24 |  18.30±4.58 |  8.74±2.13
AdaMatch${}_{\text{ ICLR'22}}$ |  19.26±1.83 |  17.13±0.92 |  12.01±4.16 |  6.07±2.26 |  13.31±3.75 |  8.14±1.48
DeFixMatch${}_{\text{ ICLR'23}}$ |  30.44±0.82 |  20.93±1.42 |  14.27±9.05 |  5.42±2.69 |  25.36±4.40 |  10.97±1.75
Pseudo-labeling
FixMatch${}_{\text{NeurIPS'20}}$ |  31.28±1.58 |  19.42±1.56 |  11.88±6.32 |  6.64±5.03 |  16.13±2.36 |  8.06±2.15
FixMatch + Ours |  27.57±1.49$\downarrow$ |  18.48±1.65$\downarrow$ |  7.19 ±4.83$\downarrow$ |  5.02±2.24$\downarrow$ |  17.55±4.00$\uparrow$ |  7.96±1.64$\downarrow$
FlexMatch${}_{\text{NeurIPS'21}}$ |  28.27±0.59 |  17.61±0.51 |  7.89±3.06 |  7.13±1.23 |  13.34±1.63 |  8.35±1.24
FlexMatch + Ours |  26.49±0.52$\downarrow$ |  18.15±0.47$\uparrow$ |  3.69±0.81$\downarrow$ |  5.00±0.98$\downarrow$ |  12.87±4.32$\downarrow$ |  7.53±1.32$\downarrow$
FreeMatch${}_{\text{ICLR'23}}$ |  23.92±2.02 |  16.18±0.38 |  4.74±1.77 |  4.48±0.73 |  14.88±0.72 |  8.83±0.14
FreeMatch + Ours |  21.36±1.62$\downarrow$ |  16.09±0.80$\downarrow$ |  4.30±1.46$\downarrow$ |  3.50±0.70$\downarrow$ |  13.18±1.61$\downarrow$ |  8.57±1.05$\downarrow$
Table 2: Calibration performance (ECE). Arrows indicate whether our modified version improves ($\downarrow$) or deteriorates ($\uparrow$) the performance. Best overall performance in bold, whereas best across pseudo-label SSL approaches is underlined. Dataset | CIFAR-100 | EuroSAT | STL-10
---|---|---|---
# Labeled samples |  200 |  400 |  20 |  40 |  40 |  100
Only consistency regularization
MixMatch${}_{\text{NeurIPS'19}}$ |  8.13±2.16 |  7.19±2.31 |  9.27±3.68 |  3.75±3.30 |  3.42±1.77 |  5.89±1.51
Dash${}_{\text{ ICML'21}}$ |  22.23±2.85 |  13.20±1.07 |  7.09±6.65 |  4.24±2.22 |  11.23±2.33 |  5.23±1.87
AdaMatch${}_{\text{ ICLR'22}}$ |  12.96±1.76 |  11.17±0.72 |  8.55±4.83 |  2.66±0.67 |  8.80±3.01 |  4.96±1.25
DeFixMatch${}_{\text{ ICLR'23}}$ |  24.54±1.37 |  14.89±0.97 |  10.46±8.71 |  2.90±1.29 |  12.89±1.73 |  6.62±1.87
Pseudo-labeling
FixMatch${}_{\text{NeurIPS'20}}$ |  27.77±1.49 |  13.45±1.45 |  8.36±5.29 |  4.72±4.45 |  10.27±2.4 |  5.83±2.25
FixMatch + Ours |  21.56±1.32$\downarrow$ |  12.12±1.70$\downarrow$ |  4.66±2.49$\downarrow$ |  3.84±1.91$\downarrow$ |  7.83±4.23$\downarrow$ |  5.64±1.40$\downarrow$
FlexMatch${}_{\text{NeurIPS'21}}$ |  21.95±0.57 |  11.95±0.30 |  5.42±2.95 |  4.50±2.60 |  9.72±1.63 |  5.85±0.98
FlexMatch + Ours |  19.74±0.50$\downarrow$ |  11.61±0.29$\downarrow$ |  2.26±1.13$\downarrow$ |  3.21±1.79$\downarrow$ |  8.89±3.39$\downarrow$ |  4.97±1.19$\downarrow$
FreeMatch${}_{\text{ICLR'23}}$ |  18.27±1.60 |  11.56±0.44 |  3.49±1.39 |  3.22±0.55 |  10.49±1.87 |  5.24±0.90
FreeMatch + Ours |  14.86±1.22$\downarrow$ |  10.35±0.68$\downarrow$ |  2.82±0.81$\downarrow$ |  2.63±0.70$\downarrow$ |  3.74±0.90$\downarrow$ |  3.50±1.02$\downarrow$

We would like to stress that our goal is not to propose a novel state-of-the-
art SSL approach, but to shed light about an important issue in prevalent
pseudo-labeling based SSL methods, and present a simple yet efficient solution
that can improve their performance. These results demonstrate that integrating
the penalty in Eq. 8 during training appears as an appealing strategy to
improve both accuracy and calibration performance of pseudo-label SSL
approaches.

Last, we follow [43] and employ the Friedman rank [11, 12] to fairly compare
the performance of different methods across various settings. This metric can
be defined as
$\operatorname{rank}_{F}=\frac{1}{m}\sum_{i=1}^{m}\operatorname{rank}_{i}$,
with $m$ being the number of evaluation settings ($m=12$ in our case, 2
metrics $\times$ 6 datasets), and $\operatorname{rank}_{i}$ the rank of a
method in the i-th setting. Hence, the lower the rank obtained, the better the
method. These rankings, which are depicted in Fig. 4, show that our modified
versions provide competitive performances considering accuracy (Error), ECE
and both (All), with our FreeMatch and FlexMatch versions presenting the best
alternatives across overall methods.

Figure 4: Friedman Rank for the methods analyzed in Tables 1 and 2 following
[43]: a) MixMatch, b) Dash, c) AdaMatch, d) DeFixMatch, e) Fixmatch, f)
FixMatch + Ours, g) FlexMatch, h) FlexMatch + Ours, i) FreeMatch and j)
FreeMatch + Ours.

On the impact on the logits. We now analyze in more detail the impact of
incorporating the constraint during training, particularly on the logit
distribution. In Fig. 5, we depict the kernel density estimation of the logits
distribution, which was negatively affected by SSL methods compared to a
fully-supervised baseline (Observation 3). From these figures, we can observe
two interesting findings that will have a positive effect in calibration: i)
the logit magnitude of the incorrect predicted classes is decreased, and ii)
the whole logit range also decreases. This means that, even for incorrect
predictions, the reduced logit values will lead to less confident predictions,
which contrasts with the potentially more confidence scores obtained by SSL
methods. We argue that the reduction of the logit magnitudes is a byproduct of
the penalty preventing the logit differences to be large. This limits the
effect of the min-entropy pushing hard towards the vertex of the simplex,
which is minimized with either 0 or 1 softmax predictions. Thus, following our
hypothesis from Observation 3, we can advocate that our strategy provides
well-calibrated SSL methods, as both the total range and logit magnitude of
incorrect predictions are significantly decreased.

Figure 5: Impact of the proposed solution in the logits, which plots the
Kernel density estimation of the logits distribution (per-class) for target
class 5 for the supervised baseline (left), original FreeMatch (middle) and
our version (right).

Comparison to other calibration methods. In this section, we compare our
approach to relevant calibration approaches in the fully-supervised scenario.
In particular, we evaluate the impact of adding label smoothing (LS) [40] and
focal loss (FL) [21] to FreeMatch, as it is the most recent studied method.
Results from these experiments, depicted in Fig. 6, confirm that our approach
consistently yields the best discriminative-calibration trade-off across
datasets and settings, which can be measured by the largest gap between the
accuracy (bright yellow) and ECE (dark yellow). Furthermore, we compare to BAM
[24], up to our knowledge the only concurrent work that tackles calibration
work on SSL. This comparison (Tab. 3) demonstrates that our simple solution
outperforms the recent BAM in terms of error rate and ECE, emerging as a
promising choice.

Figure 6: Radar plots for different calibration approaches based on FreeMatch. The range of the radar plot for ECE is changed for better visualization (CIFAR-100: [1, 50], EuroSAT: [1, 20], STL-10: [1, 20]) Table 3: Discriminative and calibration performance compared to BAM [24]. Method | CIFAR100 (200) | CIFAR100 (400)
---|---|---
|  Error |  ECE |  Error |  ECE
FixMatch + BAM |  28.46±1.74 |  25.04±1.43 |  $19.32_{\pm 0.94}$ |  16.75±0.96
FixMatch + Ours |  $\textbf{27.57}_{\pm 1.49}$ |  $\textbf{21.56}_{\pm 1.32}$ |  $\textbf{18.42}_{\pm 1.65}$ |  $\textbf{12.12}_{\pm 1.70}$

In which samples to impose the penalty? A natural question that arises from
our analysis is whether constraining the logit distances should also be
applied to the samples in $\mathcal{D}_{U^{\prime}}$. We argue that, indeed,
it is only beneficial to the samples in $\mathcal{D}_{U^{\prime\prime}}$.
While we formally exposed that the third term in Eq. 8 minimizes the min-
entropy, the second term has some sort of corrective effect, where pseudo-
labels from weak augmentations correct the predictions obtained with the
strong augmented versions. As labels are different, enforcing a penalty on the
logit distances might indeed have a counterproductive effect. More concretely,
trying to satisfy the constraint in samples from $\mathcal{D}_{U^{\prime}}$,
where hard predictions are from different classes, may actually impede the
network to learn semantically meaningful features that can bring
discriminative capabilities to the model. To support these arguments
empirically, we report in Tab. 4 the performance of the three SSL methods -
FixMatch, FlexMatch and FreeMatch when the constraint is enforced across
different scenarios. We can clearly see that, regardless of the configuration
or model used, including the penalty on the samples from
$\mathcal{D}_{U^{\prime}}$ has a detrimental effect, with discriminative and
calibration results suffering a substantial degradation.

Table 4: Impact of the penalty term in the different unlabeled subsets, $\mathcal{D_{U^{\prime}}}$ and $\mathcal{D_{U^{\prime\prime}}}$ for the three relevant pseudo-labeling SSL methods studied in this work. The proposed approach is shadowed in gray and best result in bold.  |  |  | CIFAR-100 (200) | CIFAR-100 (400)
---|---|---|---|---
| $\mathcal{D}_{U^{\prime}}$ | $\mathcal{D}_{U^{\prime\prime}}$ | Error | ECE | Error | ECE
FixMatch |  |  |  27.77±1.49 |  31.28±1.58 |  15.76±0.39 |  19.73±0.56
| ✓ |  21.56±1.32 |  27.57±1.49 |  13.49±0.80 | 18.57±0.77
✓ | ✓ | 19.30±2.52 | 25.72±2.36 | 13.00±1.79 | 19.20±1.63
FlexMatch |  |  |  21.95±0.57 |  28.27±0.59 |  11.95±0.30 | 17.61±0.51
| ✓ | 19.74±0.50 | 26.49±0.52 | 11.61±0.29 | 18.15±0.47
✓ | ✓ |  20.98±3.05 |  27.57±3.37 |  13.41±0.51 |  19.62±0.44
FreeMatch |  |  |  23.92±2.02 |  18.27±1.95 |  16.18±0.38 |  11.56±0.53
| ✓ | 21.36±1.62 | 14.86±1.48 | 16.09±0.80 | 10.35±0.83
✓ | ✓ |  27.44±1.08 |  20.86±1.48 |  18.82±0.67 |  12.39±0.78

Long-tailed experiments. In Tab. 5 we report error rate and ECE on CIFAR100-LT
to asses the effect of the proposed calibration when the class population is
imbalanced. Across all cases both the method accuracy and the ECE improve
consistently. It is noteworthy to mention that for all these experiments we
use the same margin as in CIFAR-100, without any adaptation to the specific
setting.

Table 5: Qualitative performance on long-tailed classification (CIFAR-100-LT). Method |  $\gamma_{l}$ = 10, $\gamma_{u}$ = -10 |  $\gamma_{l}$ = 10, $\gamma_{u}$ = 10 |  $\gamma_{l}$ = 15, $\gamma_{u}$ = 15
---|---|---|---
|  Error |  ECE |  Error |  ECE |  Error |  ECE
FixMatch |  $15.05_{\pm 0.13}$ |  $8.58_{\pm 0.82}$ |  $15.01_{\pm 0.18}$ |  $10.51_{\pm 1.82}$ |  $16.45_{\pm 0.11}$ |  $10.22_{\pm 2.01}$
FixMatch + Ours |  $\textbf{14.73}_{\pm 0.12}$ |  $\textbf{7.22}_{\pm 1.20}$ |  $\textbf{14.37}_{\pm 0.21}$ |  $\textbf{9.42}_{\pm 0.29}$ |  $\textbf{15.81}_{\pm 0.31}$ |  $\textbf{8.44}_{\pm 0.44}$
FlexMatch |  $14.79_{\pm 0.28}$ |  $7.70_{\pm 1.03}$ |  $14.98_{\pm 0.21}$ |  $9.49_{\pm 0.47}$ |  $16.19_{\pm 0.32}$ |  $9.25_{\pm 0.68}$
FlexMatch + Ours |  $\textbf{14.72}_{\pm 0.23}$ |  $\textbf{7.33}_{\pm 1.49}$ |  $\textbf{14.6}_{\pm 0.05}$ |  $\textbf{7.44}_{\pm 0.40}$ |  $\textbf{16.08}_{\pm 0.49}$ |  $\textbf{8.85}_{\pm 0.47}$
FreeMatch |  $14.93_{\pm 0.20}$ |  $8.52_{\pm 0.95}$ |  $15.04_{\pm 0.09}$ |  $9.99_{\pm 0.52}$ |  $16.26_{\pm 0.35}$ |  $8.68_{\pm 0.84}$
FreeMatch + Ours |  $\textbf{14.67}_{\pm 0.28}$ |  $\textbf{6.74}_{\pm 0.98}$ |  $\textbf{14.77}_{\pm 0.15}$ |  $\textbf{9.05}_{\pm 0.84}$ |  $\textbf{15.87}_{\pm 0.69}$ |  $\textbf{8.53}_{\pm 0.60}$

Additional experiments with detailed analysis across methods, comparison to
additional approaches and additional settings are reported in Appendix.

##  6 Conclusion

In this work we have raised awareness of the miscalibratiion problem induced
by pseudo-label training, which is one of the most popular approaches for SSL.
We demonstrated that the unsupervised loss used by pseudo-label methods is
dominated by the min-entropy term, a lower bound of the Shannon entropy, and
identified it as a potential source of miscalibration. We then proposed a
simple solution based on enforcing a fixed margin constraint between the
winner class and its contenders. Our solution on popular SSL datasets and
methods yields consistent calibration improvements, whereas we also found
consistent gains in terms of predictive accuracy, typically outperforming the
state-of-the-art for SSL.

## Acknowledgments

This work was funded by the Natural Sciences and Engineering Research Council
of Canada (NSERC). We also thank Calcul Quebec and Compute Canada.

## References

  * [1] Bachman, P., Alsharif, O., Precup, D.: Learning with pseudo-ensembles. Advances in neural information processing systems 27 (2014)
  * [2] Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., Raffel, C.A.: Mixmatch: A holistic approach to semi-supervised learning. Advances in Neural Information Processing Systems 32 (2019)
  * [3] Cai, Z., Ravichandran, A., Favaro, P., Wang, M., Modolo, D., Bhotika, R., Tu, Z., Soatto, S.: Semi-supervised vision transformers at scale. In: Oh, A.H., Agarwal, A., Belgrave, D., Cho, K. (eds.) Advances in Neural Information Processing Systems (2022)
  * [4] Chapelle, O., Schölkopf, B., Zien, A. (eds.): Semi-Supervised Learning. The MIT Press (2006)
  * [5] Chapelle, O., Zien, A.: Semi-supervised classification by low density separation. In: International workshop on artificial intelligence and statistics. pp. 57–64. PMLR (2005)
  * [6] Chen, H., Tao, R., Fan, Y., Wang, Y., Wang, J., Schiele, B., Xie, X., Raj, B., Savvides, M.: Softmatch: Addressing the quantity-quality tradeoff in semi-supervised learning. In: The Eleventh International Conference on Learning Representations (2023)
  * [7] Cheng, J., Vasconcelos, N.: Calibrating deep neural networks by pairwise constraints. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13709–13718 (2022)
  * [8] Coates, A., Ng, A., Lee, H.: An analysis of single-layer networks in unsupervised feature learning. In: Proceedings of the fourteenth international conference on artificial intelligence and statistics. pp. 215–223. JMLR Workshop and Conference Proceedings (2011)
  * [9] Cubuk, E.D., Zoph, B., Shlens, J., Le, Q.V.: Randaugment: Practical automated data augmentation with a reduced search space. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. pp. 702–703 (2020)
  * [10] Ding, Z., Han, X., Liu, P., Niethammer, M.: Local temperature scaling for probability calibration. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 6889–6899 (2021)
  * [11] Friedman, M.: The use of ranks to avoid the assumption of normality implicit in the analysis of variance. Journal of the american statistical association 32(200), 675–701 (1937)
  * [12] Friedman, M.: A comparison of alternative tests of significance for the problem of m rankings. The annals of mathematical statistics 11(1), 86–92 (1940)
  * [13] Gani, H., Naseer, M., Yaqub, M.: How to train vision transformer on small-scale datasets? In: 33rd British Machine Vision Conference 2022, BMVC 2022, London, UK, November 21-24, 2022. BMVA Press (2022)
  * [14] Grandvalet, Y., Bengio, Y.: Semi-supervised learning by entropy minimization. Advances in neural information processing systems 17 (2004)
  * [15] Guo, C., Pleiss, G., Sun, Y., Weinberger, K.Q.: On calibration of modern neural networks. In: International Conference on Machine Learning. pp. 1321–1330. PMLR (2017)
  * [16] Helber, P., Bischke, B., Dengel, A., Borth, D.: Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 12(7), 2217–2226 (2019)
  * [17] Krizhevsky, A., Hinton, G.: Learning multiple layers of features from tiny images. Technical report, University of Toronto (2010)
  * [18] Laine, S., Aila, T.: Temporal ensembling for semi-supervised learning. In: International Conference on Learning Representations (2017)
  * [19] Larrazabal, A.J., Martínez, C., Dolz, J., Ferrante, E.: Maximum entropy on erroneous predictions: Improving model calibration for medical image segmentation. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 273–283. Springer (2023)
  * [20] Lee, D.H., et al.: Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In: Workshop on challenges in representation learning, ICML. vol. 3, p. 896. Atlanta (2013)
  * [21] Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision. pp. 2980–2988 (2017)
  * [22] Liu, B., Ben Ayed, I., Galdran, A., Dolz, J.: The devil is in the margin: Margin-based label smoothing for network calibration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 80–88 (2022)
  * [23] Liu, B., Rony, J., Galdran, A., Dolz, J., Ben Ayed, I.: Class adaptive network calibration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16070–16079 (2023)
  * [24] Loh, C., Dangovski, R., Sudalairaj, S., Han, S., Han, L., Karlinsky, L., Soljacic, M., Srivastava, A.: Mitigating confirmation bias in semi-supervised learning via efficient bayesian model averaging. Transactions on Machine Learning Research (2023)
  * [25] Masana, M., Liu, X., Twardowski, B., Menta, M., Bagdanov, A.D., Van De Weijer, J.: Class-incremental learning: survey and performance evaluation on image classification. IEEE Transactions on Pattern Analysis and Machine Intelligence 45(5), 5513–5533 (2022)
  * [26] Minaee, S., Boykov, Y., Porikli, F., Plaza, A., Kehtarnavaz, N., Terzopoulos, D.: Image segmentation using deep learning: A survey. IEEE transactions on pattern analysis and machine intelligence 44(7), 3523–3542 (2021)
  * [27] Minderer, M., Djolonga, J., Romijnders, R., Hubis, F., Zhai, X., Houlsby, N., Tran, D., Lucic, M.: Revisiting the calibration of modern neural networks. Advances in Neural Information Processing Systems 34, 15682–15694 (2021)
  * [28] Miyato, T., Maeda, S.i., Koyama, M., Ishii, S.: Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence 41(8), 1979–1993 (2018)
  * [29] Mukhoti, J., Kulharia, V., Sanyal, A., Golodetz, S., Torr, P., Dokania, P.: Calibrating deep neural networks using focal loss. Advances in Neural Information Processing Systems 33, 15288–15299 (2020)
  * [30] Müller, R., Kornblith, S., Hinton, G.E.: When does label smoothing help? Advances in neural information processing systems 32 (2019)
  * [31] Murugesan, B., Adiga Vasudeva, S., Liu, B., Lombaert, H., Ben Ayed, I., Dolz, J.: Trust your neighbours: Penalty-based constraints for model calibration. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 572–581. Springer (2023)
  * [32] Noh, J., Park, H., Lee, J., Ham, B.: Rankmixup: Ranking-based mixup training for network calibration. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1358–1368 (2023)
  * [33] Park, H., Noh, J., Oh, Y., Baek, D., Ham, B.: ACLS: Adaptive and conditional label smoothing for network calibration. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3936–3945 (2023)
  * [34] Pereyra, G., Tucker, G., Chorowski, J., Kaiser, L., Hinton, G.: Regularizing neural networks by penalizing confident output distributions. In: International Conference on Learning Representations (ICLR) (2017)
  * [35] Pinto, F., Torr, P., Dokania, P.K.: Are vision transformers always more robust than convolutional neural networks? In: NeurIPS 2021 Workshop on Distribution Shifts: Connecting Methods and Applications (2021)
  * [36] Raghu, M., Unterthiner, T., Kornblith, S., Zhang, C., Dosovitskiy, A.: Do vision transformers see like convolutional neural networks? In: Beygelzimer, A., Dauphin, Y., Liang, P., Vaughan, J.W. (eds.) Advances in Neural Information Processing Systems (2021)
  * [37] Sajjadi, M., Javanmardi, M., Tasdizen, T.: Regularization with stochastic transformations and perturbations for deep semi-supervised learning. Advances in neural information processing systems 29 (2016)
  * [38] Shi, W., Gong, Y., Ding, C., Tao, Z.M., Zheng, N.: Transductive semi-supervised deep learning using min-max features. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 299–315 (2018)
  * [39] Sohn, K., Berthelot, D., Carlini, N., Zhang, Z., Zhang, H., Raffel, C.A., Cubuk, E.D., Kurakin, A., Li, C.L.: Fixmatch: Simplifying semi-supervised learning with consistency and confidence. Advances in Neural Information Processing Systems 33 (2020)
  * [40] Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2818–2826 (2016)
  * [41] Tarvainen, A., Valpola, H.: Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems 30 (2017)
  * [42] Tomani, C., Gruber, S., Erdem, M.E., Cremers, D., Buettner, F.: Post-hoc uncertainty calibration for domain drift scenarios. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10124–10132 (2021)
  * [43] Wang, Y., Chen, H., Fan, Y., Sun, W., Tao, R., Hou, W., Wang, R., Yang, L., Zhou, Z., Guo, L.Z., et al.: USB: A unified semi-supervised learning benchmark for classification. Advances in Neural Information Processing Systems 35, 3938–3961 (2022)
  * [44] Wang, Y., Chen, H., Heng, Q., Hou, W., Fan, Y., Wu, Z., Wang, J., Savvides, M., Shinozaki, T., Raj, B., Schiele, B., Xie, X.: Freematch: Self-adaptive thresholding for semi-supervised learning. In: The Eleventh International Conference on Learning Representations (2023)
  * [45] Xie, Q., Dai, Z., Hovy, E., Luong, T., Le, Q.: Unsupervised data augmentation for consistency training. Advances in Neural Information Processing Systems 33 (2020)
  * [46] Xie, Q., Luong, M.T., Hovy, E., Le, Q.V.: Self-training with noisy student improves imagenet classification. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 10687–10698 (2020)
  * [47] Xu, Y., Shang, L., Ye, J., Qian, Q., Li, Y.F., Sun, B., Li, H., Jin, R.: Dash: Semi-supervised learning with dynamic thresholding. In: International Conference on Machine Learning. pp. 11525–11536. PMLR (2021)
  * [48] Yang, L., Zhao, Z., Qi, L., Qiao, Y., Shi, Y., Zhao, H.: Shrinking class space for enhanced certainty in semi-supervised learning. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 16187–16196 (2023)
  * [49] Zhang, B., Wang, Y., Hou, W., Wu, H., Wang, J., Okumura, M., Shinozaki, T.: Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems 34 (2021)
  * [50] Zheng, M., You, S., Huang, L., Wang, F., Qian, C., Xu, C.: Simmatch: Semi-supervised learning with similarity matching. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14471–14481 (2022)

## Supplementary Material for
Do not trust what you trust:
Miscalibration in Semi-supervised Learning

##  1 Additional discriminative results

Table 6: Comparison of error rate (%) for different SSL methods across various labeled settings and datasets (CIFAR-100, EuroSAT, and STL-10).$\dagger$ indicates that the results are reported from [43]. Best results in bold, whereas second-best are underlined. Dataset | CIFAR-100 | EuroSAT | STL-10
---|---|---|---
# Labeled samples |  200 |  400 |  20 |  40 |  40 |  100
$\pi$-Model${}_{\text{ NeurIPS'15}}$ $\dagger$ |  36.06±0.15 |  26.52±0.41 |  21.82±1.22 |  12.09±2.27 |  42.76±15.94 |  19.85±13.02
Mean-Teacher${}_{\text{ NeurIPS'17}}$ |  35.47±0.40 |  26.03±0.30 |  26.83±1.46 |  15.85±1.66 |  18.67±2.66 |  24.19±10.15
VAT${}_{\text{ TPAMI'18}}$ $\dagger$ |  31.49±1.33 |  21.34±0.50 |  18.45±1.47 |  26.16±0.96 |  10.09±0.94 |  10.69±0.51
MixMatch${}_{\text{ NeurIPS'19}}$ |  37.68±2.66 |  26.84±1.06 |  28.77±10.40 |  14.88±2.07 |  25.19±2.05 |  11.37±1.49
ReMixMatch${}_{\text{ ICLR'20}}$$\dagger$ |  22.21±2.21 |  16.86±0.57 |  5.05±1.05 |  5.07±0.56 |  13.08±3.34 |  7.21±0.39
UDA${}_{\text{ NeurIPS'20}}$ |  28.80±0.61 |  19.00±0.79 |  9.83±2.15 |  6.22±1.36 |  15.58±3.16 |  7.65±1.11
CRMatch${}_{\text{ GCPR'21}}$$\dagger$ |  25.70±1.75 |  18.03±0.20 |  13.24±1.69 |  8.35±1.71 |  10.17±0.00 |  –
CoMatch${}_{\text{ ICCV'21}}$$\dagger$ |  35.08±0.69 |  25.35±0.50 |  5.75±0.43 |  4.81±1.05 |  15.12±1.88 |  9.56±1.35
Dash${}_{\text{ ICML'21}}$ |  28.51±2.91 |  19.54±1.20 |  10.05±8.15 |  6.83±3.24 |  18.30±4.58 |  8.74±2.13
AdaMatch${}_{\text{ ICLR'22}}$ |  19.26±1.83 |  17.13±0.92 |  12.01±4.16 |  6.07±2.26 |  13.31±3.75 |  8.14±1.48
SimMatch${}_{\text{ CVPR'22}}$ $\dagger$ |  23.78±1.08 |  17.06±0.78 |  7.66±0.60 |  5.27±0.89 |  11.77±3.20 |  7.55±1.86
SoftMatch${}_{\text{ ICLR'23}}$ $\dagger$ |  22.67±1.32 |  16.84±0.66 |  5.75±0.62 |  5.90±1.42 |  13.55±3.16 |  7.84±1.72
DeFixMatch${}_{\text{ ICLR'23}}$ |  30.44±0.82 |  20.93±1.42 |  14.27±9.05 |  5.42±2.69 |  25.36±4.40 |  10.97±1.75
FixMatch${}_{\text{ NeurIPS'20}}$ |  31.28±1.58 |  19.73±0.56 |  11.88±6.32 |  6.64±5.03 |  16.13±2.36 |  8.06±2.15
FixMatch + Ours |  27.57±1.49 |  18.57±0.77 |  7.19 ±4.83 |  5.02±2.24 |  17.55±4.00 |  7.96±1.64
FlexMatch${}_{\text{ NeurIPS'21}}$ |  28.27±0.59 |  17.61±0.51 |  7.89±3.06 |  7.13±1.23 |  13.34±1.63 |  8.35±1.24
FlexMatch + Ours |  26.49±0.52 |  18.15±0.47 |  3.69±0.81 |  5.00±0.98 |  12.87±4.32 |  7.53±1.32
FreeMatch${}_{\text{ ICLR'23}}$ |  23.92±2.02 |  16.18±0.38 |  4.74±1.77 |  4.48±0.73 |  14.88±0.72 |  8.83±0.14
FreeMatch + Ours |  21.36±1.62 |  16.09±0.80 |  4.30±1.46 |  3.50±0.70 |  13.18±1.61 |  8.57±1.05

In the main paper, we included several relevant SSL methods to compare both
error and calibration performance. We did not include a considerably large set
of approaches because we need to run three times each method in each setting
(so that each evaluated method means 3$\times$6=18 runs, with an average of 10
hours per run). Furthermore, we would like to stress again that our goal in
this work is not to provide a novel state-of-the-art SSL method, but
investigate the miscalibration issue of pseudo-labeling SSL, identify the
source of the problem, and provide a solution that can enhance the calibration
of this family of methods. Having said this, the discriminative performance of
additional methods is provided in [43], which is used to complement the
discriminative comparison in Table 1, whose results can be found in Table 6 of
this appendix.

##  2 Can we add our calibration strategy to other methods?

In the main paper we have evaluated the impact of incorporating the proposed
term into relevant SSL approaches that resort to hard pseudo-labels during
training. Indeed, our motivation stems from the observations that under this
scenario (i.e., the standard pseudo-labeling process in SSL), there is a
hidden min-entropy objective that dominates the training. While we have
demonstrated that our approach effectively improves both the discriminative
and calibration performance of these models, we further assess its impact on
approaches that use soft pseudo-labels, i.e., pseudo-labels on weak
augmentations are not converted to one-hot encoded vectors. For these
approaches, the reasoning in Eq. 5 and Eq. 6 does not entirely hold in these
cases. More concretely, the recent work SimMatch [50] uses this
strategy333Note that most pseudo-labeling SSL approaches use a hard pseudo-
label strategy, where the softmax predictions of weak augmentations are
transformed into one-hot encoded pseudo-labels., where the direct softmax
prediction over the weak augmented samples is used as supervisory signal for
its strong augmented counterpart. The results from this experiment are
reported in Table 7, which show that adding the proposed penalty has a
positive effect favouring better calibrated models even in methods that use
soft pseudo-labels, particularly as the number of labeled samples increases.
Note that, as our main findings, observations, and motivations, do not
entirely apply for soft pseudo-labels approaches, such as SimMatch, we do not
make any claim regarding the benefits of our strategy in these approaches. In
contrast, we simply show empirically that even in these cases, our method can
still bring performance benefits.

Table 7: Comparison to a related approach, SimMatch, which uses soft pseudo-labels, instead of hard pseudo-labels (as FixMatch, FlexMatch and FreeMatch).  | Error rate (%) | ECE
---|---|---
| CIFAR-100
# Labeled samples |  200 |  400 |  1000 |  200 |  400 |  1000
SimMatch |  22.49±0.26 |  19.45±0.08 |  15.46±0.48 |  3.17±0.28 |  6.54±1.92 |  9.60±0.72
SimMatch + Ours |  23.00±0.07$\uparrow$ |  19.10±0.31$\downarrow$ |  15.36±0.41$\downarrow$ |  3.31±0.59$\uparrow$ |  5.55±0.13$\downarrow$ |  9.18±0.17$\downarrow$

##  3 Dataset details

We provide detailed insights into the datasets utilized in our experiments,
whose configuration is strongly inspired by the recent USB (A Unified Semi-
supervised Learning Benchmark for Classification) benchmark [43]. For
CIFAR-100, a renowned benchmark for fine-grained image classification, we
considered two label settings: 2 labeled samples and 4 labeled samples per
class for each of the 100 classes,resulting in a total of 50,000 training
samples and 10,000 samples for testing. Each image in CIFAR-100 is sized at
32×32 pixels. STL-10, known for its limited sample size and extensive
unlabeled data, offers a unique challenge. We employed two label settings as
well: 4 labeled samples and 10 labeled samples per class for all 10 classes,
and an additional 100,000 unlabeled samples for training, along with 8,000
samples for testing. Each STL-10 image measures 96×96 pixels. Lastly, EuroSAT,
based on Sentinel-2 satellite images, features two label settings: 2 labeled
samples per class and 4 samples per class for 10 classes. With a total of
16,200 training samples, including labeled and unlabeled images and 5,400
testing samples, EuroSAT images are sized at 64×64. While literature includes
SVHN and CIFAR-10 datasets, we do not use them for evaluation, as state-of-
the-art SSL methods already achieve a performance comparable to that of fully
supervised training on these datasets. The selected datasets offer diverse
challenges and settings, allowing for a comprehensive evaluation of semi-
supervised learning methods across various domains.

For the long-tailed CIFAR-100 experiments, we investigate two subsets of the
dataset, with $N1$ representing the number of samples in the minority class
and $M$ denoting the number of samples in the majority class. The imbalance
ratio is controlled by $\gamma_{l}$ and $\gamma_{u}$, where both parameters
are set to -10, 10 or 15, ensuring a consistent level of label imbalance
across categories. In the first setting, we set the imbalance ratio of labeled
samples $\gamma_{l}$ to 10 and the imbalance ratio of unlabeled samples
$\gamma_{u}$ to 10, with the number of labeled samples (N1) set to 150 and the
number of unlabeled samples (M) set to 300. In the second setting, we maintain
the same label imbalance ratio ($\gamma_{l}$ = 10) and number of labeled
samples (N1 = 150), but we set the unlabeled sample imbalance ratio
$\gamma_{u}$ to -10. Finally, in the third setting, we increase the
$\gamma_{l}$ to 15 while keeping the $\gamma_{u}$ at 15, with N1 and M set to
150 and 300, respectively. This configuration introduces a higher degree of
label imbalance, with the minority class having 15 times fewer samples than
the majority class. We follow the same setup as authors in [43] for the
classic setting using WideResnet. These settings allow us to comprehensively
evaluate the performance of machine learning models under varying degrees of
label imbalance and class distribution scenarios within the CIFAR100-LT
dataset.

##  4 Implementation details

Method-dependent hyperparemeters. In the context of pseudo-labeling methods,
while FixMatch and FlexMatch rely on a single threshold hyperparameter for
pseudo-label selection, FreeMatch introduces additional hyperparameters such
as a pre-defined threshold $\tau$, unlabeled batch ratio $\mu$, unsupervised
loss weight $w_{u}$, fairness loss weight $w_{f}$, and EMA decay $\lambda$. To
avoid unfair comparisons across methods, we use the default values for all
these hyperparameters, which are reported in their respective papers.

Specific-case: BAM. In our comparison with BAM [24], we encountered
difficulties in hyperparameter tuning. BAM employs the quantile $Q$ over the
batch to determine the threshold for pseudo-label selection, which serves as a
key hyperparameter. In the cited paper, $Q=0.75$ for the CIFAR-100 benchmark
and $Q=0.95$ for the CIFAR-10 benchmark were chosen. Additionally, BAM
incorporates a separate Adam optimizer for the Bayesian Neural Network (BNN)
layer, with a fixed learning rate of 0.01. Notably, for the sharpening
temperature parameter in BaM-UDA, BAM opts for $t=0.9$ as opposed to $t=0.4$
utilized in UDA. These hyperparameter selections are specific to both the
dataset and the method employed, thus presenting a challenge when evaluating
the method across different datasets. For our experiments, we chose to focus
on CIFAR-100 due to the inherent complexity involved in conducting extensive
hyperparameter searches, not only concerning the dataset or method, but also
regarding the optimizer and learning rate for the introduced BNN layer.

Margin Selection with Limited Fine-tuning: We stress that in the majority of
cases, we did not extensively fine-tune the margin hyperparameter. Across
various experiments, settings and datasets, the chosen margin values remained
consistent without significant adjustments. Specifically, for the CIFAR-100
and EuroSAT datasets, the margins were uniformly set without requiring further
refinement, with values of 10 and 8 across methods, respectively. Given its
complexity, we needed to perform a hyperparameter search in a few cases in the
STL-10 dataset. In particular, most settings employed a margin equal to 10,
similar to the CIFAR-100 dataset. Nevertheless, we observe that, particularly
for FreeMatch, this margin did not yield the best performance across the two
settings of labeled data (i.e., 40 and 100 labeled samples). After fine-tuning
the margin value, we found that for STL-10 (40) and STL-10 (100) FreeMatch
worked best with a margin of 4 and 6, respectively. Furthermore, for all the
remaining experiments with CIFAR-100, as well as with long-tailed CIFAR-100,
we kept the margin fixed ($m=10$) across experiments and methods.

Training hyperparemeters. Regarding algorithm-independent hyperparameters, we
adhered to the settings outlined in [43]. Specifically, the learning rate was
set to $5\times 10^{-4}$ for CIFAR-100, $10^{-4}$ for STL-10, and $5\times
10^{-5}$ for EuroSAT. During training, the batch size was fixed at 8, while
for evaluation, it was set to 16. Additionally, the layer decay rate varied
across datasets: 0.5 for CIFAR-100, 0.95 for STL-10, and 1.0 for EuroSAT. Weak
augmentation techniques employed included random crop and random horizontal
flip, while strong augmentation utilized RandAugment [9]. The cosine annealing
scheduler was utilized with a total of 204,800 steps and a warm-up period of
5,120 steps. Both labeled and unlabeled batch sizes were set to 16.
Furthermore, we follow [43, 49] to report the best number of all checkpoints
to avoid unfair comparisons caused by different convergence speeds.

##  5 Model choices

In this work we have selected three relevant SSL approaches based on hard
pseudo-labels, as our findings are closely related to these approaches. More
concretely, we demonstrate empirically our observations in FixMatch [39],
FlexMatch [49] and FreeMatch [44], as they are popular SSL methods, some of
them published recently (i.e., FreeMatch has been published in 2023). Thus,
for the Table 1 and Table 2, we have shown the effect of adding the proposed
term in all the three approaches. Similarly, the impact of our term over the
three methods is evaluated in Table 5 (i.e., CIFAR-100-LT), as we consider
that long-tailed classification represents an important task, and assessing
the performance of the three approaches will undoubtedly strengthen the
message that our solution can improve both classification and calibration
performance across a general family of SSL approaches. Regarding the ablation
studies to evaluate the impact of several choices, we have used FreeMatch, the
most recent approach among the three, in all the ablations (Fig 6 and Table
4), except when comparing to the concurrent work in BAM [24]. The reason is
that authors in [24] only consider FixMatch in their experiments, despite BAM
being a work published in 2023. Thus, the
code444https://github.com/clott3/BaM-SSL is all constrained to the use of
FixMatch and CIFAR-100, making it difficult to evaluate with other SSL
approaches and datasets. In fact, we attempted to integrate FreeMatch on the
BAM framework, but the results obtained were suboptimal. We believe that, the
sensitivity to hyperparameters (e.g., just the choice of the backbone for the
Bayesian layer in BAM requires different learning rates) could be the main
cause for the low performance of BAM in SSL approaches other than FixMatch.
Therefore, in order to avoid an unfair comparison (due to the suboptimal
performance of BAM in other SSL methods) we decided to compare to it based
only on FixMatch, following their original work.

##  6 Numerical values for results in radar plots

For the radar plots shown in the main paper, we present here (Table 8) the
individual quantitative results, for a more detailed comparison across
methods. As a reminder, FreeMatch [44] is used as a baseline strategy, and the
different calibration strategies (i.e., Focal Loss [21], Label Smoothing [40],
and ours) are applied over the $\mathcal{D}_{U^{\prime\prime}}$ subset.

Table 8: Numerical values of radar plots depicted in Figure 6 of the main paper. Method | Setting | Error | ECE | CECE | AECE
---|---|---|---|---|---
Baseline | CIFAR 100 (200) |  23.92±2.02 |  18.27±1.95 |  0.41±0.04 |  18.27±1.95
Focal Loss |  28.81±0.75 |  24.42±0.85 |  0.53$\pm_{0.01}$ |  24.42±0.85
Label Smoothing |  29.14±2.46 |  23.55±2.33 |  0.52±0.04 |  23.54±2.34
Ours | 21.36±1.62 | 14.86±1.48 | 0.35±0.03 | 14.81±1.48
Baseline | CIFAR 100 (400) |  16.18±0.38 |  11.56±0.53 |  0.27±0.01 |  11.54±0.52
Focal Loss |  20.35±0.35 |  15.63±0.54 |  0.35±0.008 |  15.60±0.54
Label Smoothing |  19.01±1.90 |  13.03±1.39 |  0.31±0.02 |  13.02±1.39
Ours | 16.09±0.80 | 10.35±0.83 | 0.26±0.01 | 10.33±0.83
Baseline | EuroSAT (20) |  4.74±1.82 |  3.50±1.70 |  0.82±0.32 |  3.38±1.82
Focal Loss |  10.25±8.48 |  8.54±8.29 |  1.88$\pm_{1.70}$ |  8.52±8.31
Label Smoothing |  5.94±4.21 |  3.57±2.83 |  0.95±0.67 |  3.45±2.81
Ours | 4.30±1.46 | 2.82±1.00 | 0.70±0.23 | 2.78±0.98
Baseline | EuroSAT (40) |  4.48±0.73 |  3.22±0.55 |  0.74±0.11 |  3.15±0.61
Focal Loss |  4.37±0.30 |  3.10±0.10 |  0.74$\pm_{0.04}$ |  3.01±0.08
Label Smoothing |  15.84±3.57 |  4.43±3.38 |  1.02±0.70 |  4.37±3.31
Ours | 3.50±0.70 | 2.63±0.70 | 0.60±0.12 | 2.58±0.64
Baseline | STL 10 (40) |  14.88±0.72 |  10.49±1.87 |  2.45±0.40 |  9.40±1.81
Focal Loss |  14.73±2.77 |  11.16±3.13 |  2.39$\pm{0.64}$ |  11.15±3.13
Label Smoothing | 15.59±2.74 |  10.09±1.64 | 2.20±0.28 |  10.04±1.58
Ours | 13.18±1.61 | 3.74±5.64 | 2.39±0.68 | 8.43±4.92
Baseline | STL 10 (100) |  7.62±1.25 |  5.24±1.09 |  1.15±0.21 |  5.19±1.03
Focal Loss |  8.75±1.59 |  6.07±1.40 |  1.35${}_{\pm}{0.31}$ |  6.02±1.35
Label Smoothing | 8.25±2.05 |  4.94±1.79 |  1.13±0.35 |  4.94±1.79
Ours | 8.57±1.90 | 3.50±1.54 | 0.94±0.27 | 4.00±1.21

##  7 Detailed comparison between original and our versions

We include additional comparisons between our proposed method and the original
SSL approaches.

Convergence analysis. First, as depicted in Fig. 7, we assess the impact of
adding our approach to FreeMatch in terms of convergence. Notably, our method,
denoted as FreeMatch (Ours), demonstrates superior performance in terms of
accuracy, outperforming the baseline FreeMatch method within the initial
$50,000$ iterations. In contrast, the original method reaches convergence
around $170,000$ iterations, making it considerably slower if one looks at
discriminative performance. Additionally, our approach consistently achieves
lower values of Expected Calibration Error (ECE) throughout the iterative
process, highlighting its potential to enhance the calibration performance.
This contrasts with the original FreeMatch, whose predictions seem to be
better calibrated at the first iteration, and these are degraded (i.e., ECE
increases) with the training. Thus, adding our method exhibits accelerated
convergence rates compared to the original approach, reaching competitive
performance levels in a fraction of the time.

Figure 7: Comparison of convergence of Accuracy (left) and ECE (right) over
iterations for STL10 dataset with 40 labeled samples.

Logits distribution. We now provide additional insights into the effects of
our approach across the logit distribution. More concretely, in Fig. 8 and
Fig. 9 we present the kernel density estimation of the logits distribution for
each class on the STL10 dataset for FreeMatch and FreeMatch plus our method
(denoted as Ours). This analysis expands on Observation 3 from the main paper,
highlighting the impact of SSL on the calibration of the model across a
diverse range of classes. Specifically, while the original FreeMatch provides
larger logit ranges, as well as larger logit values even for incorrect
classes, adding our approach limits these logit increases across all the
classes. As the logit range decreases, the resulting softmax probabilities for
the predicted classes will be smaller, alleviating the problem of
overconfident predictions. This is particularly important for wrong
predictions, as we expect a well-calibrated model to not be overconfident when
incorrect classes are predicted.

Figure 8: Comparison of convergence of Accuracy (left) and ECE (right) over
iterations for STL10 dataset with 40 labeled samples. Figure 9: Comparison of
convergence of Accuracy (left) and ECE (right) over iterations for STL10
dataset with 40 labeled samples.

Reliability plots. Last, we depict several reliability plots for both the
semi-supervised learning (SSL) methods that were discussed in the main paper
and our modified version, to highlight the calibration improvements brought by
our approach (Fig. 10 and Fig. 11).

Figure 10: Reliability plots on CIFAR-100(200) for the original (top) and our
(bottom) versions of the three pseudo-label SSL methods selected. Figure 11:
Reliability plots on CIFAR-100(400) for the original (top) and our (bottom)
versions of the three pseudo-label SSL methods selected.


Let $u_{4}$ be the cohomology class $\gamma_{2,1}\cdot z_{1}$. Then, we have
an isomorphism

$H^{*}(\overline{\mathrm{Fl}}_{4}(\mathbb{C});\mathbb{F}_{2})\cong\frac{H^{*}(BN(4);\mathbb{F}_{2})}{\big{(}f_{4}^{*}(c_{1}),f_{4}^{*}(c_{2}),f_{4}^{*}(c_{3}),f_{4}^{*}(c_{4})\big{)}}\oplus\mathbb{F}_{2}\\{u_{4},u_{4}\gamma_{1,2},u_{4}\gamma_{2,1},u_{4}\gamma_{1,2}\gamma_{2,1}\\}.$

On the $E_{\infty}$ page of the spectral sequence, the classes
$u_{4},u_{4}\gamma_{1,2},u_{4}\gamma_{2,1}$, and
$u_{4}\gamma_{1,2}\gamma_{2,1}$ survive due to the non-injectivity of the
differential $d_{2}$. However, in the corresponding spectral sequence for
stable cohomology, all differentials are injective on the corresponding ideals
generated by the generators $z_{2k-1}$, and therefore these four classes are
not restrictions of any stable class. We refer to such classes as “unstable
classes”.

######  Corollary 6.4.

The Poincaré series of the mod $2$ cohomology ring of
$\overline{\mathrm{Fl}}_{4}$ is

$\Pi_{\overline{\mathrm{Fl}}_{4}}(t)=1+t+2t^{2}+3t^{3}+4t^{4}+4t^{5}+5t^{6}+4t^{7}+4t^{8}+3t^{9}+2t^{10}+t^{11}+t^{12}.$

###### Proof.

We can check this directly. Recall that the cohomological dimensions of the
classes that generate $H^{*}(BN(4);\mathbb{F}_{2})$ are as follows:

$\displaystyle|\gamma_{1,1}\odot 1_{2}|$
$\displaystyle=1,\quad|\gamma_{1,2}|=2,\quad|\gamma_{2,1}|=3,$
$\displaystyle|c_{[2]}\odot\gamma_{1,1}|$
$\displaystyle=5,\quad\text{and}\quad|c_{[k]}\odot 1_{4-k}|=2k\quad(1\leq
k\leq 4).$

We also have the following relations between the generators:

$\displaystyle\big{(}(\gamma_{1,1}\odot
1_{2})\cdot(c_{[2]}+1_{2})+(c_{[2]}\odot\gamma_{1,1})\big{)}\cdot(c\odot
1_{3})+(\gamma_{1,2}\odot 1_{2})\cdot(c_{[3]}\odot 1_{1})$ $\displaystyle=0,$
$\displaystyle\text{for all }x\in\\{\gamma_{1,1}\odot
1_{2},c_{[2]}\odot\gamma_{1,1},c\odot 1_{3},c_{[2]}\odot 1_{2},c_{[3]}\odot
1_{1}\\},\quad x\cdot\gamma_{2,1}$ $\displaystyle=0,$
$\displaystyle(\gamma_{1,1}\odot 1_{2})\cdot(c_{[3]}\odot
1_{1})=0,\quad(c_{[2]}\odot\gamma_{1,1})\cdot(c_{[3]}\odot 1_{1})$
$\displaystyle=0,$ $\displaystyle\gamma_{1,2}\cdot(c\odot
1_{3})=0,\quad\gamma_{1,2}\cdot(c_{[3]}\odot 1_{1})$ $\displaystyle=0.$

Note that these relations between the generators are not exhaustive. From the
description of $H^{*}(\overline{\mathrm{Fl}}_{4};\mathbb{F}_{2})$ in Theorem
6.3 and the above relations, we get the following relations in
$H^{*}(\overline{\mathrm{Fl}}_{4};\mathbb{F}_{2})$ between the Hopf ring
generators. By identifying $(\gamma_{1,1}\odot 1_{2})^{2}=c\odot 1_{3}$, we
have

$\displaystyle(\gamma_{1,1}\odot
1_{2})^{2}\cdot\gamma_{1,2}=0,\quad(\gamma_{1,1}\odot
1_{2})^{2}\cdot\gamma_{2,1}$ $\displaystyle=0,$
$\displaystyle(\gamma_{1,1}\odot
1_{2})^{3}\cdot(c_{[2]}+1_{2})+(c_{[2]}\odot\gamma_{1,1})\cdot(\gamma_{1,1}\odot
1_{2})^{2}$ $\displaystyle+(\gamma_{1,2}\odot 1_{2})\cdot(c_{[3]}\odot
1_{1})=0.$

By identifying $c_{[2]}\odot 1_{2}=c\odot\gamma_{1,1}^{2}\odot
1_{1}+\gamma_{1,2}^{2}=(\gamma_{1,1}\odot 1_{2})^{4}+\gamma_{1,2}^{2}$ and
$c_{[3]}\odot
1_{1}=\gamma_{1,2}^{2}+\gamma_{2,1}^{2}+(c_{[2]}\odot\gamma_{1,1})\cdot(\gamma_{1,1}\odot
1_{2})$ we have

$\displaystyle\gamma_{1,2}^{2}\cdot\gamma_{2,1}=0,\quad(\gamma_{1,1}\odot
1_{2})^{7}+(\gamma_{1,1}\odot 1_{2})\cdot\gamma_{1,2}^{3}=0.$

With the information we have so far, we can explicitly write down all
generators of the cohomology groups
$H^{k}(\overline{\mathrm{Fl}}_{4};\mathbb{F}_{2})$ for $0\leq k\leq 6$.

$\displaystyle H^{0}(\overline{\mathrm{Fl}}_{4};\mathbb{F}_{2})$
$\displaystyle\cong\mathbb{F}_{2},$ $\displaystyle
H^{1}(\overline{\mathrm{Fl}}_{4};\mathbb{F}_{2})$
$\displaystyle\cong\mathbb{F}_{2}\\{(\gamma_{1,1}\odot 1_{2})\\},$
$\displaystyle H^{2}(\overline{\mathrm{Fl}}_{4};\mathbb{F}_{2})$
$\displaystyle\cong\mathbb{F}_{2}\\{(\gamma_{1,1}\odot
1_{2})^{2},\gamma_{1,2}\\},$ $\displaystyle
H^{3}(\overline{\mathrm{Fl}}_{4};\mathbb{F}_{2})$
$\displaystyle\cong\mathbb{F}_{2}\\{(\gamma_{1,1}\odot
1_{2})^{3},(\gamma_{1,1}\odot 1_{2})\gamma_{1,2},\gamma_{2,1}\\},$
$\displaystyle H^{4}(\overline{\mathrm{Fl}}_{4};\mathbb{F}_{2})$
$\displaystyle\cong\mathbb{F}_{2}\\{(\gamma_{1,1}\odot
1_{2})^{4},(\gamma_{1,1}\odot 1_{2})\gamma_{2,1},\gamma_{1,2}^{2},u_{4}\\},$
$\displaystyle H^{5}(\overline{\mathrm{Fl}}_{4};\mathbb{F}_{2})$
$\displaystyle\cong\mathbb{F}_{2}\\{(\gamma_{1,1}\odot
1_{2})^{5},(\gamma_{1,1}\odot
1_{2})\gamma_{1,2}^{2},\gamma_{1,2}\gamma_{2,1},c_{[2]}\odot\gamma_{1,1}\\},$
$\displaystyle H^{6}(\overline{\mathrm{Fl}}_{4};\mathbb{F}_{2})$
$\displaystyle\cong\mathbb{F}_{2}\\{(\gamma_{1,1}\odot
1_{2})^{6},(\gamma_{1,1}\odot
1_{2})(c_{[2]}\odot\gamma_{1,1}),\gamma_{1,2}^{3},\gamma_{2,1}^{2},u_{4}\gamma_{1,2}\\}.$

Although we haven’t identified the specific generators of
$H^{k}(\overline{\mathrm{Fl}}_{4};\mathbb{F}_{2})$ for $k>6$, we can still
determine its Poincar’e series. Note that $\overline{\mathrm{Fl}}_{4}$ is a
$12$-dimensional manifold and that with mod $2$ coefficients, Poincar’e
duality holds without needing the orientability assumption. Hence, the
corollary follows. ∎

###  6.3. Mod $2$ Cohomology of $\overline{\mathrm{Fl}}_{5}(\mathbb{C})$

From (12),
$H^{*}(U(5);\mathbb{F}_{2})\cong\Lambda_{\mathbb{F}_{2}}[z_{1},z_{3},z_{5},z_{7},z_{9}]$.
Also, from Lemma 5.1 and Example 5.3 the differentials in the spectral
sequence $E$ associated with
$U(5)\rightarrow\overline{\mathrm{Fl}}_{5}(\mathbb{C})\rightarrow BN(5)$ are
given by

$\displaystyle d_{2}(z_{1})$ $\displaystyle=c\odot 1_{4}+\gamma_{1,1}^{2}\odot
1_{3},$ $\displaystyle d_{4}(z_{3})$ $\displaystyle=c_{[2]}\odot
1_{3}+c\odot\gamma_{1,1}^{2}\odot 1_{2}+\gamma_{1,2}^{2}\odot 1_{1},$
$\displaystyle d_{6}(z_{5})$ $\displaystyle=c_{[3]}\odot
1_{2}+c_{[2]}\odot\gamma_{1,1}^{2}\odot
1_{1}+c\odot\gamma_{1,2}^{2}+\gamma_{2,1}^{2}\odot 1_{1},$ $\displaystyle
d_{8}(z_{7})$ $\displaystyle=c_{[4]}\odot
1_{1}+c_{[3]}\odot\gamma_{1,1}^{2}+c\odot\gamma_{2,1}^{2},$ $\displaystyle
d_{10}(z_{9})$ $\displaystyle=c_{[5]}.$

From the Hopf ring structure of $A_{BU(1)}$, we deduce that
$H^{*}(BN(5);\mathbb{F}_{2})$ is generated by

$\\{c\odot 1_{4},c_{[2]}\odot 1_{3},c_{[3]}\odot 1_{2},c_{[4]}\odot
1_{1},c_{[5]},\gamma_{1,1}\odot 1_{3},c_{[2]}\odot\gamma_{1,1}\odot
1_{1},\gamma_{1,2}\odot 1_{1},\gamma_{2,1}\odot 1_{1}\\}$

with the relations determined by Theorem 2.3 along with Hopf ring
distributivity. Again, we check the injectivity of the differentials on the
rank zero Hopf monomials multiplied by the corresponding $z_{2k-1}$’s. On the
$E_{2}$-page we have

$\displaystyle d_{2}((\gamma_{1,1}\odot 1_{3})\cdot z_{1})$
$\displaystyle=(\gamma_{1,1}\odot 1_{3})\cdot
d_{2}(z_{1})=c\odot\gamma_{1,1}\odot 1_{2}+\gamma_{1,1}^{3}\odot
1_{3}+\cdots,$ $\displaystyle d_{2}((c_{[2]}\odot\gamma_{1,1}\odot 1_{1})\cdot
z_{1})$ $\displaystyle=(c_{[2]}\odot\gamma_{1,1}\odot 1_{1})\cdot
d_{2}(z_{1})=c_{[3]}\odot\gamma_{1,1}+\cdots,$ $\displaystyle
d_{2}((\gamma_{1,2}\odot 1_{1})\cdot z_{1})$ $\displaystyle=(\gamma_{1,2}\odot
1_{1})\cdot
d_{2}(z_{1})=c\odot\gamma_{1,2}+\gamma_{1,1}^{3}\odot\gamma_{1,1}\odot 1_{1},$
$\displaystyle d_{2}((\gamma_{2,1}\odot 1_{1})\cdot z_{1})$
$\displaystyle=(\gamma_{2,1}\odot
1_{1})\cdot(d_{2}(z_{1})=c\odot\gamma_{2,1}.$

The above formulas along with Lemma 5.4 show that $d_{2}$ is injective on the
ideal generated by $z_{1}$. So, $E_{4}\equiv E_{3}$ is given by

$E_{4}^{*,*}\cong\frac{H^{*}(BN(5);\mathbb{F}_{2})}{\big{(}f_{5}^{*}(c_{1})\big{)}}.$

Similarly as before it can be checked that the differentials $d_{4}$, $d_{6}$,
and $d_{8}$ are all injective on the ideals generated by $z_{3}$, $z_{5}$, and
$z_{7}$ respectively. Thus the $E_{10}$-page is given by

$E_{10}^{*,*}\cong\frac{H^{*}(BN(5);\mathbb{F}_{2})}{\big{(}f_{5}^{*}(c_{1}),f_{5}^{*}(c_{2}),f_{5}^{*}(c_{3}),f_{5}^{*}(c_{4})\big{)}}$

Finally, the differential $d_{10}$ on the rank zero Hopf monomial on the
$E_{10}$-page is given by

$\displaystyle d_{10}((\gamma_{1,1}\odot 1_{3})\cdot z_{9})$
$\displaystyle=(\gamma_{1,1}\odot 1_{3})\cdot c_{[5]}=c_{[2]}\gamma_{1,1}\odot
c_{[3]},$ $\displaystyle d_{10}((c_{[2]}\odot\gamma_{1,1}\odot 1_{1})\cdot
z_{9})$ $\displaystyle=(c_{[2]}\odot\gamma_{1,1}\odot 1_{1})\cdot
c_{[5]}=c_{[2]}^{2}\odot c_{[2]}\gamma_{1,1}\odot c+\cdots,$ $\displaystyle
d_{10}((\gamma_{1,2}\odot 1_{1})\cdot z_{9})$
$\displaystyle=(\gamma_{1,2}\odot 1_{1})\cdot c_{[5]}=c_{[4]}\gamma_{1,2}\odot
c,$ $\displaystyle d_{10}((\gamma_{2,1}\odot 1_{1})\cdot z_{9})$
$\displaystyle=(\gamma_{2,1}\odot 1_{1})\cdot c_{[5]}=c_{[4]}\gamma_{2,1}\odot
c$ $\displaystyle=(c_{[4]}\gamma_{2,1}\odot 1_{1})\cdot(c\cdot
1_{4}+\gamma_{1,1}^{2}\odot 1_{3})=0.$

We see that $d_{10}$ is not injective with
$\mathrm{ker}(d_{10})=\langle(\gamma_{2,1}\odot 1_{1})\cdot z_{9}\rangle$.
Note that

$\displaystyle d_{4}((\gamma_{2,1}\odot 1_{1})\cdot z_{3}z_{9})$
$\displaystyle=(\gamma_{1,2}^{2}\gamma_{2,1}\odot 1_{1})\cdot z_{9},$
$\displaystyle d_{6}((\gamma_{2,1}\odot 1_{1})\cdot z_{5}z_{9})$
$\displaystyle=(\gamma_{2,1}^{3}\odot 1_{1})\cdot z_{9},$ $\displaystyle
d_{8}((\gamma_{2,1}\odot 1_{1})\cdot z_{7}z_{9})$
$\displaystyle=(c_{[4]}\gamma_{2,1}\odot 1_{1})\cdot z_{9}.$

All higher differentials are zero and therefore the spectral sequence
collapses at the $E_{11}$-page. The $E_{\infty}\equiv E_{11}$-page is given by

$E_{\infty}^{*,*}\cong\frac{H^{*}(BN(5);\mathbb{F}_{2})}{\big{(}f_{5}^{*}(c_{1}),f_{5}^{*}(c_{2}),f_{5}^{*}(c_{3}),f_{5}^{*}(c_{4}),f_{5}^{*}(c_{5})\big{)}}\oplus\mathcal{U}$

where $\mathcal{U}=\mathbb{F}_{2}\\{(\gamma_{2,1}\odot 1_{1})\cdot
z_{9},(\gamma_{1,2}\gamma_{2,1}\odot 1_{1})\cdot z_{9},(\gamma_{2,1}^{2}\odot
1_{1})\cdot z_{9},(\gamma_{1,2}\gamma_{2,1}^{2}\odot 1_{1})\cdot z_{9}\\}$. We
summarize the result of our computations in this subsection as the following
theorem.

######  Theorem 6.5.

Let us denote the cohomology class $(\gamma_{2,1}\odot 1_{1})\cdot z_{9}$ by
$u_{12}$. Then, we have an isomorphism

$H^{*}(\overline{\mathrm{Fl}}_{5}(\mathbb{C});\mathbb{F}_{2})\cong\frac{H^{*}(BN(5);\mathbb{F}_{2})}{\big{(}f_{5}^{*}(c_{1}),\dots,f_{5}^{*}(c_{5})\big{)}}\oplus\mathbb{F}_{2}\\{u_{12},u_{12}\gamma_{1,2},u_{12}\gamma_{2,1},u_{12}\gamma_{1,2}\gamma_{2,1}\\}.$

###  6.4. Mod $p$ Cohomology of $\overline{\mathrm{Fl}}_{p}(\mathbb{C})$ for
$p>2$

In this subsection, we present a computation for
$H^{*}(\overline{\mathrm{Fl}}_{p}(\mathbb{C});\mathbb{F}_{p})$ for all odd
prime $p$. From Theorem 2.4, we have that $H^{*}(BN(p);\mathbb{F}_{p})$ is
generated by $\gamma_{1,1}$, $\alpha_{1,1}$ and $c_{[k]}\odot 1_{p-k}$ for
$k=1,2,\dots,p$. Our strategy will be to study the Serre spectral sequence
associated with (11) with $n=p$.

We introduce some notations that will be helpful later in describing the
cohomology of $\overline{\mathrm{Fl}}_{p}(\mathbb{C})$. Recall from (12) that,

(13)
$H^{*}(U(p);\mathbb{F}_{p})=\Lambda_{\mathbb{F}_{p}}[z_{1},z_{3},\dots,z_{2p-1}]$

######  Definition 6.6.

Let $S=\\{s_{1},\dots,s_{k}\\}$ be a subset of $\\{1,2,\dots,p\\}$ such that
$s_{1}<\cdots<s_{k}$. We define $z_{S}$ to be the cohomology class

$z_{S}:=\begin{cases}z_{2s_{1}-1}\cdots z_{2s_{k}-1}&\text{if
}S\neq\emptyset\\\ 1_{U(p)}&\text{if }S=\emptyset\end{cases}$

The following are immediate from Theorem 4.11 and Proposition 5.2.

######  Corollary 6.7.

Let $f_{p}:BN(p)\rightarrow BU(p)$ as before. Then
$f_{p}^{*}(c_{k})=c_{[k]}\odot 1_{p-k}$ for $k=1,\dots,p-2$,
$f_{p}^{*}(c_{p-1})=c_{[p-1]}\odot 1_{1}+\gamma_{1,1}$, and
$f_{p}^{*}(c_{p})=c_{[p]}$.

Note that $\alpha_{1,1}$ never shows up in the pullback formulas for
$f_{p}^{*}$ as it has an odd cohomological dimension whereas
$f_{p}^{*}(c_{k})$ are even-dimensional for all $1\leq k\leq p$.

######  Theorem 6.8.

Let $\gamma_{S}:=z_{S}\cdot\gamma_{1,1}$ and
$\alpha_{S}:=z_{S}\cdot\alpha_{1,1}$, where $z_{S}$ is as in Definition 6.6.
Then,

$H^{*}(\overline{\mathrm{Fl}}_{p}(\mathbb{C});\mathbb{F}_{p})=\frac{\mathbb{F}_{p}[\gamma_{S},\alpha_{S}|S\subset\\{1,2,\dots,p-2\\}]}{(\gamma_{S}^{2},\alpha_{S}^{2},\gamma_{S}\cdot\alpha_{S})}$

###### Proof.

From the Hopf ring structure of $A_{BU(1)}$ in Theorem 2.3, we deduce that
from

$H^{*}(BN(p);\mathbb{F}_{p})\cong\frac{\mathbb{F}_{p}[\gamma_{1,1},\alpha_{1,1},c_{[k]}\odot
1_{p-k}\mid
k=1,\dots,p]}{\big{(}\alpha_{1,1}^{2},\gamma_{1,1}\cdot(c_{[k]}\odot
1_{p-k}),\alpha_{1,1}\cdot(c_{[k]}\odot 1_{p-k})\mid k=1,\dots,p-1\big{)}}$

From Lemma 5.1, the differential $d_{2k}$ is given by

$d_{2k}:z_{2k-1}\longmapsto f_{p}^{*}(c_{k}).$

Using the product structure on the $E_{2i}$-page of the spectral sequence, we
deduce the following:

$\displaystyle\gamma_{1,1}\cdot(c_{[k]}\odot 1_{[p-k]})=0\quad$
$\displaystyle\text{and}\quad\alpha_{1,1}\cdot(c_{[k]}\odot
1_{[p-k]})=0\quad\text{for all }k=1,\dots,p-1$ $\displaystyle\gamma_{1,1}\cdot
c_{[p]}\neq 0\quad$ $\displaystyle\text{and}\quad\alpha_{1,1}\cdot c_{[p]}\neq
0.$

Hence, $d_{2k}(z_{2k-1}\cdot\gamma_{1,1})=0$ and
$d_{2k}(z_{2k-1}\cdot\alpha_{1,1})=0$ for all $k=1,2,\dots,p-2$. Also, using
the multiplicative structure of the spectral sequence and the formulas from
Corollary 6.7, we have

$\displaystyle d_{2(p-1)}(z_{2p-3}\cdot\gamma_{1,1})=\gamma_{1,1}^{2}$
$\displaystyle,\quad
d_{2(p-1)}(z_{2p-3}\cdot\alpha_{1,1})=\gamma_{1,1}\cdot\alpha_{1,1}$
$\displaystyle d_{2p}(z_{2p-1}\cdot\gamma_{1,1})=\gamma_{1,1}\cdot c_{[p]}$
$\displaystyle,\quad d_{2p}(z_{2p-1}\cdot\alpha_{1,1})=\alpha_{1,1}\cdot
c_{[p]}.$

As all higher differentials $d_{k}$ for $k>2p$ are zero, the spectral sequence
collapses at the $E_{2p}$-page and the $E_{\infty}$-page is given by

$E_{\infty}^{*,*}\cong\frac{H^{*}(BN(p);\mathbb{F}_{p})}{\big{(}f_{p}^{*}(c_{1}),\dots,f_{p}^{*}(c_{p})\big{)}}\oplus\big{(}\mathbb{F}_{p}\\{\gamma_{1,1},\alpha_{1,1}\\}\otimes\Lambda_{\mathbb{F}_{p}}[z_{1},\dots,z_{2p-5}]\big{)}.$

This proves the theorem. ∎

######  Corollary 6.9.

The mod $p$ Poincaré series of $\overline{\mathrm{Fl}}_{p}(\mathbb{C})$ is
given by

$\Pi_{\overline{\mathrm{Fl}}_{p}}^{p}(t)=1+(t^{2p-3}+t^{2p-2})\prod_{k=1}^{p-2}(1+t^{2k-1}).$

###  6.5. Mod $p$ cohomology of $\overline{\operatorname{Fl}}_{p}(\mathbb{R})$
for $p>2$

We conclude this section by computing the mod $p$ cohomology of
$\overline{\operatorname{Fl}}_{p}(\mathbb{R})$ for all odd prime $p$. From
Theorem 3.7 and Corollary 3.8, there is an isomorphism

$H^{*}(B\operatorname{B}_{p}^{+};\mathbb{F}_{p})\cong\frac{\mathbb{F}_{p}[\gamma_{1,1},\alpha_{1,1}]}{(\alpha_{1,1}^{2})},$

where the degrees of $\gamma_{1,1}$ and $\alpha_{1,1}$ are $2p-2$ and $2p-3$,
respectively. Similarly

$H^{*}(SO(p);\mathbb{F}_{p})\cong\Lambda(\\{a_{1},\dots,a_{2p-3}).$

The differential $d_{4r}$ of the $4r$-th page of the associated spectral
sequence maps $a_{4r-1}$ to the pullback of the Pontrjagin class
$\mathcal{p}_{r}$, and all the other differentials are $0$. By Proposition
5.5, $d_{2p-2}(a_{2p-3})=\gamma_{1,1}$, and $d_{r}=0$ if $r\not=2p-2$. We
conclude that $E_{2}^{*,*}\cong E_{2p-2}^{*,*}$ and that

$E_{\infty}^{*,*}\cong
E_{2p-1}^{*,*}\cong\Lambda(\\{a_{3},\dots,2p-7\\})\otimes\frac{\mathbb{F}_{p}[\alpha_{1,1}]}{(\alpha_{1,1}^{2})}.$

We deduce the following results.

######  Theorem 6.10.

Let $p>2$ be an odd prime. Then there is an isomoprhism of graded algebras

$H^{*}(\overline{\operatorname{Fl}}_{p}(\mathbb{R});\mathbb{F}_{p})\cong\Lambda(\\{a_{3},\dots,a_{2p-7}\\})\otimes\frac{\mathbb{F}_{p}[\alpha_{1,1}]}{(\alpha_{1,1}^{2})}.$

######  Corollary 6.11.

The mod $p$ Poincarè series of $\overline{\operatorname{Fl}}_{p}(\mathbb{R})$
is given by

$\Pi^{p}_{\overline{\operatorname{Fl}}_{p}(\mathbb{R})}(t)=(1+t^{3})(1+t^{7})\cdots(1+t^{2p-7})(1+t^{2p-3}).$

## References

  * [1] A. Adem and J. Gómez.  A classifying space for commutativity in Lie groups.  Algebraic & Geometric Topology, 15(1):493 – 535, 2015.
  * [2] M. F. Atiyah.  The geometry of classical particles.  Surveys in differential geometry, 7:1–15, 2002.
  * [3] M. F. Atiyah and I. G. MacDonald.  Introduction to commutative algebra. Addison-Wesley-Longman, 1969.
  * [4] E. H. Brown, Jr.  The cohomology of $B{\rm SO}_{n}$ and $B{\rm O}_{n}$ with integer coefficients.  Proc. Amer. Math. Soc., 85(2):283–288, 1982.
  * [5] F. R. Cohen, T. J. Lada, and J. P. May.  The homology of iterated loop spaces.  Lecture Notes in Mathematics, Vol. 533. Springer-Verlag, Berlin-New York, 1976.
  * [6] D. Eisenbud.  Commutative algebra. With a view toward algebraic geometry. Berlin: Springer-Verlag, 0 edition, 2 1999.
  * [7] R. H. Fox.  On the Lusternik–Schnirelmann category.  Annals of Mathematics, 42(2):333–370, 1941.
  * [8] C. Giusti, P. Salvatore, and D. Sinha.  The mod-2 cohomology rings of symmetric groups.  J. Topol., 5(1):169–198, 2012.
  * [9] C. Giusti and D. Sinha.  Mod-two cohomology rings of alternating groups.  J. Reine Angew. Math., 772:1–51, 2021.
  * [10] L. Guerra.  Hopf ring structure on the mod $p$ cohomology of symmetric groups.  Algebr. Geom. Topol., 17(2):957–982, 2017.
  * [11] L. Guerra.  The mod 2 cohomology of the infinite families of coxeter groups of type $\mathrm{B}$ and $\mathrm{D}$ as almost hopf rings.  Algebr. Geom. Topol., 2023.
  * [12] L. Guerra and S. Jana.  The mod-2 cohomology of the alternating subgroups of the coxeter groups of type $\mathrm{B}$, (in preparation).
  * [13] L. Guerra, S. Jana, and A. Maiti.  Mod-2 cohomology of unordered flag varieties in lower orders and Auerbach bases, 2023.
  * [14] L. Guerra, P. Salvatore, and D. Sinha.  Cohomology rings of extended powers and free infinite loop spaces, 2023\.
  * [15] A. Hatcher.  Algebraic topology.  Cambridge University Press, Cambridge, 2002.
  * [16] J. E. Humphreys.  Reflection groups and Coxeter groups, volume 29 of Cambridge Studies in Advanced Mathematics.  Cambridge University Press, Cambridge, 1990.
  * [17] S. O. Kochman.  Homology of the classical groups over the Dyer-Lashof algebra.  Trans. Amer. Math. Soc., 185:83–136, 1973.
  * [18] J. McCleary.  A User’s Guide to Spectral Sequences.  Cambridge University Press, Cambridge, 2001.
  * [19] J. W. Milnor and J. D. Stasheff.  Characteristic classes.  Annals of Mathematics Studies, No. 76. Princeton University Press, Princeton, N. J.; University of Tokyo Press, Tokyo, 1974.
  * [20] A. Weber and M. Wojciechowski.  On the Pełczyński conjecture on Auerbach bases.  Communications in Contemporary Mathematics, 19(06):1750016, 2017\.


CHAPTER: INDEX

We have used the following acronyms, abbreviations and terminology

1PIone-particle irreducible
ac[ac]Anti-commuting $\gamma_5$ scheme
bm[bm]Breitenlohner-Maison scheme
BRbranching ratio
BSMBeyond the Standard Model
cdr[cdr]Conventional dimensional regularisation
dred[dred]Dimensional reduction
dreg[dreg]Dimensional regularisation
EFTeffective field theory
fdf[fdf]Four-dimensional formulation of fdh
fdh[fdh]Four-dimensional helicity scheme
fdu[fdu]Four-dimensional unsubtraction
FKS2[$\mathrm{FKS}^2$]FKS double soft
FKSFrixione-Kunszt-Signer subtraction scheme
GPLgeneralised polylogarithm
HPLharmonic polylogarithm
HVPhadronic vacuum polarisation
hv[hv]'t Hooft-Veltman scheme
LFVlepton-flavour violating
LHCLarge Hadron Collider
LLleading logarithm
LOLeading order
LSZLehmann-Symanzik-Zimmermann reduction formula
MMCT core team
msbar[$\MS$]modified minimal subtraction
n3lo[N$^3$LO]Next-to-next-to-next-to-leading order
NLLNext-to-leading logarithm
NLONext-to-leading order
NNLONext-to-next-to-leading order
PCSpseudo-collinear singularity
PIDparticle identification
PDFparton distribution function
PSIPaul Scherrer Institut
PSparton shower
QCDQuantum chromodynamics
QEDQuantum electron dynamics
QFTQuantum field theory
RGERenormalisation-group equation
RNGRandom number generator
rs[rs]Regularisation scheme
SCETSoft-collinear effective theory
SHA1Secure Hashing Algorithm 1
SMStandard Model
VPvacuum polarisation

active scales
analytic regularisation
anomalous dimension
auxiliary contributions
Casimir scaling
colour factors
config file
corner integral
corner region
cusp angle
cusp anomalous dimension
decoupling transformation
dimension-six operator
effective field theory
eikonal factor
eikonal subtraction
electronic corrections
factorisation anomaly
factorisation scale
Feynman parameters
Feynman parametrisation
Fierz identities
full period
fully differential
generic pieces
generic processes
graph polynomials
integrated eikonal
IR safe
lexicographic ordering
loop diagrams
loop integrals
master integral
matching calculation
measurement function
Mellin transform
menu files
method of regions
Michel decay
mixed corrections
MSlikesubtracted[$\MS$-like subtracted]
muonic corrections
non-trivial scheme dependence
process groups
radiative muon decay
Ramanujan's master theorem
random seed
rare muon decay
reducible scalar integrals
renormalisation constants
renormalisation scale
renormalisation scheme
seed identity
soft cut
submission script
subtraction scheme
Symanzik polynomials
tree-level diagrams
trivial scheme dependence
two-loop ready
Ward identity
Wilson coefficient