#  Diversified Ensemble of Independent Sub-Networks for
Robust Self-Supervised Representation Learning

Amirhossein Vahidi1,2, Lisa Wimmer1,2, Hüseyin Anil Gündüz1,2, Bernd
Bischl1,2, Eyke Hüllermeier3, Mina Rezaei1,2

###### Abstract

Ensembling a neural network is a widely recognized approach to enhance model
performance, estimate uncertainty, and improve robustness in deep supervised
learning. However, deep ensembles often come with high computational costs and
memory demands. In addition, the efficiency of a deep ensemble is related to
diversity among the ensemble members which is challenging for large, over-
parameterized deep neural networks. Moreover, ensemble learning has not yet
seen such widespread adoption, and it remains a challenging endeavor for self-
supervised or unsupervised representation learning. Motivated by these
challenges, we present a novel self-supervised training regime that leverages
an ensemble of independent sub-networks, complemented by a new loss function
designed to encourage diversity. Our method efficiently builds a sub-model
ensemble with high diversity, leading to well-calibrated estimates of model
uncertainty, all achieved with minimal computational overhead compared to
traditional deep self-supervised ensembles. To evaluate the effectiveness of
our approach, we conducted extensive experiments across various tasks,
including in-distribution generalization, out-of-distribution detection,
dataset corruption, and semi-supervised settings. The results demonstrate that
our method significantly improves prediction reliability. Our approach not
only achieves excellent accuracy but also enhances calibration, surpassing
baseline performance across a wide range of self-supervised architectures in
computer vision, natural language processing, and genomics data.

## Introduction

Ensemble learning has become a potent strategy for enhancing model performance
in deep learning (Hansen and Salamon 1990; Dietterich 2000; Lakshminarayanan,
Pritzel, and Blundell 2017). This method involves combining the outputs of
multiple independently-trained neural networks, all using the same
architecture and same training dataset but differing in the randomness of
their initialization and/or training. Despite its remarkable effectiveness,
training deep ensemble models poses several challenges: i) The high
performance achieved by deep ensembles comes with a significant increase in
computational costs. Running multiple neural networks independently demands
more resources and time. ii) Maintaining diversity among ensemble members – a
property often critical to success – becomes progressively difficult for
large, over-parameterized deep neural networks (Ramé and Cord 2021; Dabouei et
al. 2020) in which the main source of diversity comes from random weight
initialization. iii) Most of the existing literature focuses on deep ensembles
for supervised models. Adapting these approaches to unsupervised and self-
supervised models requires careful consideration and evaluation to ensure
comparable performance.

In recent years, self-supervised learning methods have achieved cutting-edge
performance across a wide range of tasks in natural language processing (NLP;
(Devlin et al. 2018; Brown et al. 2020), computer vision (Chen et al. 2020b;
Bardes, Ponce, and LeCun 2021; Grill et al. 2020; Rezaei et al. 2023; Lienen,
Demir, and Hüllermeier 2022), multimodal learning (Radford et al. 2021; Li et
al. 2022; Shi et al. 2022), and bioinformatics (Gündüz et al. 2021). In
contrast to supervised techniques, these models learn representations of the
data without relying on costly human annotation. Despite remarkable progress
in recent years, self-supervised models do not allow practitioners to inspect
the model’s confidence. This problem is non-trivial given the degree to which
critical applications rely on self-supervised methods. As recently discussed
by LeCun111https://ai.facebook.com/blog/self-supervised-learning-the-dark-
matter-of-intelligence/, representing predictive uncertainty is particularly
difficult in self-supervised contrastive learning for computer vision.
Therefore, quantifying the predictive uncertainty of self-supervised models is
critical to more reliable downstream tasks. Here, we follow the definition of
reliability as described by Plex (Tran et al. 2022), in which the ability of a
model to work consistently across many tasks is assessed. In particular, Tran
et al. (2022) introduce three general desiderata of reliable machine learning
systems: a model should generalize robustly to _new tasks_ , as well as _new
datasets_ , and represent the associated _uncertainty_ in a faithful manner.

In this paper, we introduce a novel, robust, and scalable framework for
ensembling _self-supervised learning_ while _preserving performance_ with a
negligible increase in computational cost and _encouraging diversity among the
ensemble of sub-networks_.

Our contributions can be summarized as follows:

  * •

We propose a novel scalable ensemble of self-supervised learning to be robust,
efficient, and enhance the model performance in various downstream tasks.

  * •

We develop a complementary loss function to enforce diversity among the
independent sub-networks.

  * •

We perform extensive empirical analyses to highlight the benefits of our
approach. We demonstrate that this inexpensive modification achieves very
competitive (in most cases, better) predictive performance: 1) on in-
distribution (IND) and out-of-distribution (OOD) tasks; 2) in semi-supervised
settings; 3) learns a better predictive performance-uncertainty trade-off than
compared baselines (i.e., exhibits high predictive performance and low
uncertainty on IND datasets as well as high predictive performance and high
uncertainty on OOD datasets).

Figure 1:  Illustration of our proposed method. Given a batch $\bm{X}$ of
input samples, two different views $\tilde{\bm{x}}$ and
$\tilde{\bm{x}}^{\prime}$ are produced for each sample, which is then encoded
into representations by the encoder network $f_{\bm{\theta}^{\prime}}$. The
representations are projected to the ensemble of independent sub-networks
$g_{m}$, where each sub-network produces embedding vectors $\bm{z}$ and
$\bm{z}^{\prime}$. The mean value of these embeddings is passed to the self-
supervised loss, while their standard deviation is used for the diversity
loss. Finally, the total loss is computed by a combination of the two loss
components.

## Related Work

Self-supervised learning  For most large-scale modeling problems, learning
under full supervision is severely inhibited by the scarcity of annotated
samples. Self-supervised learning techniques, which solve _pretext tasks_
(Devlin et al. 2018) to generate labels from (typically abundant) unlabeled
data, have proven to be a powerful remedy to this bottleneck. The learned
feature maps can serve as a starting point for _downstream_ supervised tasks,
such as classification, object detection, or sentiment analysis, with a
substantially reduced need for labeled examples (Jaiswal et al. 2020).
Alternatively, the downstream application may directly use the extracted
representation for problems such as anomaly OOD detection. While there have
been attempts to make pretraining more robust by preventing embedding collapse
(Bardes, Ponce, and LeCun 2021; Rezaei et al. 2023) or boosting performance in
OOD detection (Winkens et al. 2020; Sehwag, Chiang, and Mittal 2021; Rezaei et
al. 2022b; Tran et al. 2022), the aspect of _uncertainty-awareness_ has been
studied to a lesser extent in the self-supervised context. Motivated by this,
we present a simple way to make self-supervised learning robust during
pretext-task learning.

Ensemble learning  Deep Ensembles (Lakshminarayanan, Pritzel, and Blundell
2017) comprise a set of $M$ neural networks that independently train on the
same data using random initialization. Deep ensembles often outperform other
approaches in terms of calibration and predictive accuracy (Ovadia et al.
2019; Gustafsson, Danelljan, and Schon 2020; Rezaei et al. 2022a; Ashukha et
al. 2020; Hüllermeier and Waegeman 2021), but their naive application incurs
high computational complexity, as training, memory, and inference cost
multiplies with the number of base learners. BatchEnsemble (Wen, Tran, and Ba
2020) introduces multiple low-rank matrices with little training and storage
demand, whose Hadamard products with a shared global weight matrix mimic an
ensemble of models. Masksensemble (Durasov et al. 2021) builds upon Monte
Carlo dropout (Gal and Ghahramani 2016) and proposes a learnable (rather than
a random) selection of masks used to drop specific network neurons. MIMO
(Havasi et al. 2021) uses ensembles of sub-networks diverging only at the
beginning and end of the parent architecture – thus sharing the vast majority
of weights – in order to obtain multiple predictions with a single forward
pass. At test time, several copies of each sample are fed to the enlarged
input layer, and the multi-head last layer returns an according number of
predictions. Although these methods reduce the inference time and
computational resources required at training, the benefits are limited to the
larger pretraining model that is used in self-supervised learning.

Diversity in ensembles: Diversity is a crucial component for successful
ensembles. Ramé and Cord (2021) classify existing approaches for encouraging
diversity among ensemble members into three groups: i) methods that force
_diversity in gradients_ with adaptive diversity in prediction (Pang et al.
2019), or using joint gradient phase and magnitude regularization (GPMR)
between ensemble members (Dabouei et al. 2020), ii) methods focusing on
_diversity in logits_ , improving diversity with regularization and estimating
the uncertainty of out-of-domain samples (Liang, Li, and Srikant 2018), or by
bounding the Lipschitz constant of networks and limiting the variety of
predictions against slight input changes (Cisse et al. 2017; Tsuzuku, Sato,
and Sugiyama 2018), iii) methods promoting diversity in features that increase
diversity with adversarial loss (Chen et al. 2020a) for conditional redundancy
(Ramé and Cord 2021), information bottleneck (Sinha et al. 2021; Fischer
2020), or $f1$-divergences (Chen et al. 2020a). Our method belongs to this
last category, where our loss function encourages the diversity of feature
maps.

## Method

We propose a simple principle to 1) make self-supervised pretraining robust
with an ensemble of diverse sub-networks, 2) improve predictive performance
during pretraining of self-supervised deep learning, 3) while keeping an
efficient training pipeline.

As depicted in Figure 1, our proposed method can be readily applied to the
most recent trends in self-supervised learning (Caron et al. 2021; Grill et
al. 2020; Chen et al. 2020b; Devlin et al. 2019; Gündüz et al. 2021; Klein and
Nabi 2022) and is based on a joint embedding architecture. In the following
sections, we first describe our proposed ensemble model, followed by the
diversity loss, and then a discussion on diversity, and computational cost.

### Robust Self-Supervised Learning via Independent Sub-Networks

#### Setting.

Given a randomly sampled mini-batch of data
$\bm{X}=\\{\bm{x}_{k}\\}_{k=1}^{N}\subset\mathcal{X}\subseteq\mathbb{R}^{p}$,
the transformer function derives two augmented views
$\tilde{\bm{x}}=\tau(\bm{x}),\tilde{\bm{x}}^{\prime}=\tau^{\prime}(\bm{x})$
for each sample in $\bm{X}$. The augmented views are obtained by sampling
$\tau,\tau^{\prime}$ from a distribution over suitable data augmentations,
such as masking parts of sequences (Baevski et al. 2022; Devlin et al. 2019),
partially masking image patches (He et al. 2022), or applying image
augmentation techniques (Chen et al. 2020b).

The two augmented views $\tilde{\bm{x}}$ and $\tilde{\bm{x}}^{\prime}$ are
then fed to an encoder network $f_{\bm{\theta}}$ with trainable parameters
$\bm{\theta}\subseteq\mathbb{R}^{d}$. The encoder (e.g., ResNet-50 (He et al.
2016), ViT (Dosovitskiy et al. 2021b)) maps the distorted samples to a set of
corresponding features. We call the output of the encoder the
_representation_. Afterward, the representation features are transformed by
$M$ independent sub-networks $\\{g_{\bm{\phi_{m}}}\\}_{m=1}^{M}$ with
trainable parameters $\bm{\phi}_{m}$ to improve the feature learning of the
encoder network. The ensemble constructs from the representation $M$ different
$q$-dimensional _embedding_ vectors $\\{\bm{z}_{m}\\}_{m=1}^{M}$,
$\\{\bm{z}_{m}^{\prime}\\}_{m=1}^{M}$, respectively, for $\tilde{\bm{x}}$ and
$\tilde{\bm{x}}^{\prime}$. We modify the conventional self-supervised loss and
replace the usual $\bm{z}_{m}$ by the mean value
$\bar{\bm{z}}=(\bm{z}_{1}+\ldots+\bm{z}_{M})/M$, and similarly
$\bm{z}_{m}^{\prime}$ by $\bar{\bm{z}}^{\prime}$. Averaging over the
embeddings generated by the $M$ sub-networks is likely to increase robustness,
which in turn may help to improve predictive performance in downstream tasks

#### Self-supervised loss.

In the case of contrastive learning (Chen et al. 2020b), the self-supervised
loss $\ell_{\text{ssl}}$ with temperature $t>0$ and cosine similarity
$\mathrm{sim}(\cdot,\cdot)$ is computed as:

$\ell_{\text{ssl}}\left(\tilde{\bm{x}}_{k},\tilde{\bm{x}}^{\prime}_{k}\right)=-\log~{}\frac{\exp(\mathrm{sim}(\bar{\bm{z}}_{k},\bar{\bm{z}}^{\prime}_{k})/t)}{\sum_{i=1}^{2N}\mathbb{I}_{[k\neq
i]}\exp(\mathrm{sim}(\bm{\bar{}}{\bm{z}}_{k},\bar{\bm{z}}_{i})/t)}.$ (1)

#### Diversity loss.

Since diversity is a key component of successful model ensembles (Fort, Hu,
and Lakshminarayanan 2019), we design a new loss function for encouraging
diversity during the training of the sub-networks. We define the diversity
regularization term $\ell_{\text{div}}$ as a hinge loss over the difference of
the standard deviation across the embedding vectors
$\\{\bm{z}_{k,m}\\}_{m=1}^{M}$, $\\{\bm{z}_{k,m}^{\prime}\\}_{m=1}^{M}$ to a
minimum diversity of $\alpha>0$. The standard deviation is the square root of
the element-wise variance $\\{\sigma_{k,o}^{2}\\}_{o=1}^{q}$:

$~{}{\sigma}_{k,o}^{2}=\tfrac{1}{M-1}\textstyle\sum_{m=1}^{M}(z_{k,m,o}-\bar{z}_{k,o})^{2}+\epsilon\,,$

where we add a small scalar $\epsilon>0$ to prevent numerical instabilities.
The diversity regularization function is then given by:

$\displaystyle\ell_{\text{div}}\left(\tilde{\bm{x}}_{k},\tilde{\bm{x}}_{k}^{\prime}\right)=\textstyle\sum_{o=1}^{q}$
$\displaystyle\operatorname{max}\left(0,\alpha-{\sigma_{k,o}}\right)$ (2)
$\displaystyle\,+\operatorname{max}(0,\alpha-{\sigma_{k,o}^{\prime}})\,,$

where $\sigma$ and $\sigma^{\prime}$ indicate standard deviation for the input
sample and augmented views, respectively.

Figure 2:  Total Standard Deviation: sum of all standard deviations between
independent sub-networks during training. Training with diversity loss (Eq. 2)
increases the standard deviation and improves the diversity between
independent sub-networks.

#### Total loss.

The objective of the diversity loss is to encourage disagreement among sub-
networks by enforcing the element-wise standard deviations to be close to
$\alpha>0$ and to thus prevent the embeddings from collapsing to the same
vector. Figure 2 underlines the importance of the diversity loss on the total
sum of standard deviations between different sub-networks, which increases by
adding this loss. The total loss is calculated by combining the self-
supervised loss (Eq. 1) and the diversity loss (Eq. 2), where the degree of
regularization is controlled by a tunable hyperparameter $\lambda\geq 0$:

$\ell\left(\tilde{\bm{x}}_{k},\tilde{\bm{x}}_{k}^{\prime}\right)=\ell_{\text{ssl}}\left(\tilde{\bm{x}}_{k},\tilde{\bm{x}}_{k}^{\prime}\right)+\lambda\cdot\ell_{\text{div}}\left(\tilde{\bm{x}}_{k},\tilde{\bm{x}}_{k}^{\prime}\right).$
(3)

Finally, the total loss is aggregated over all the pairs in minibatch
$\bm{X}$:

$\mathcal{L}_{\text{total}}=\tfrac{1}{N}{\textstyle\sum_{k=1}^{N}\ell\left(\tilde{\bm{x}}_{k},\tilde{\bm{x}}_{k}^{\prime}\right)}.$
(4)

#### Gradients.

Consider the output of the encoder $f_{\bm{\theta}}(\bm{x})=b$ and the output
of the $m$-th linear sub-network $\bm{z}_{m}=g_{m}(b)=w_{m}\cdot b$. The
weight $w_{m}$ is updated by two components during backpropagation, the first
of which depends on the self-supervised loss and is the same for the entire
ensemble, while the second term depends on the diversity loss and is different
for each sub-network. Given Eq. Diversity loss., we simplify the equation by
vector-wise multiplication since the sub-networks are linear; furthermore, we
omit the numerical stability term since it does not have an effect on the
derivative. The element-wise standard deviation can be computed as follows:

${\sigma}_{k,o}=\left(\tfrac{1}{M-1}\textstyle\sum_{m=1}^{M}(\bm{z}_{k,m,o}-\bar{\bm{z}}_{k,o})^{2}\right)^{\tfrac{1}{2}}.$
(5)

Consider Eq. 2 for aggregating the element-wise standard deviations for one
observation ($\bm{x}$) and assume ${\sigma}_{k}<\alpha$; otherwise, the
diversity loss is zero when ${\alpha\leq\sigma}_{k}$. The derivative of the
loss with respect to $\bm{z}_{k,\hat{m},o}$, $\hat{m}\in{1,\ldots,M}$, is then
given as follows:

${\frac{\partial\left(\ell_{\text{div}}\right)}{\partial\bm{z}_{k,\hat{m},o}}}=\frac{-A}{M-1}\cdot(\bm{z}_{k,\hat{m},o}-\bar{\bm{z}}_{k,o}),$
(6)

where
$A:=\tfrac{1}{M-1}\textstyle\sum_{m=1}^{M}(\bm{z}_{k,m,o}-\bar{\bm{z}}_{k,o})^{2})$.
The proof is provided in the appendix (see Theoretical Supplement).

In the optimization step of stochastic gradient descent (SGD), the weight of
sub-network $\hat{m}$ is updated by:

$\eta\cdot\nabla_{w_{\hat{m},o}}\ell_{\text{div}}=-C\cdot(\bm{z}_{k,\hat{m},o}-\bar{\bm{z}}_{k,o}),$
(7)

where $\eta>0$ is the learning rate, and $C$ is constant with respect to
$w_{\hat{m},o}$, which depends on the learning rate, number of sub-networks,
$A$, and $b$. The proof is provided in Appendix (see Theoretical Supplement).

Eq. 7 shows the updating step in backpropagation. Hyperparameter $\alpha$
prevents ${\bm{z}}_{k,\hat{m},o}$ from collapsing to the a single point.
Hence, $w_{\hat{m},o}$ is updated in the opposite direction of
$\bar{\bm{z}}_{k,o}$, so the diversity loss prevents weights in the sub-
networks from converging to the same values.

Figure 3: Diversity Analysis: prediction diversity disagreement vs. achieved
accuracy on test set of CIFAR-10. Diversity analysis encompasses the
comparative assessment of two distinct models that have been trained on test
datasets, with a focus on quantifying the dissimilarity in their respective
predictions. This evaluation entails computing the fraction of test data
points for which predictions of models disagree, the diversity, and
normalizing it by the model’s error rate. Our method with $10$ subnetworks is
on par with the deep self-supervised ensemble with $10$ members in terms of
both accuracy and diversity disagreement. Models in the top right corner are
better.

### Empirical Analysis of Diversity

Diversity of ensemble members is an important feature for powerful model
ensembles and reflects the degree of independence among its members (Zhang and
Ma 2012; Ortega, Cabañas, and Masegosa 2022). We follow Fort, Hu, and
Lakshminarayanan (2019) to quantify the diversities among the ensemble of sub-
networks. Specifically, we report the diversities in terms of _disagreement
score_ between the members’ predictive distributions and a baseline. Diversity
disagreement is defined as distance disagreement divided by $1-$ accuracy,
where the distance disagreement between two classification models $h_{i}$ and
$h_{j}$ is calculated as
$\frac{1}{N}\sum_{k=1}^{N}\big{[}h_{i}(\mathbf{x}_{k})\neq
h_{j}(\mathbf{x}_{k})\big{]},$ with $N$ denoting the number of samples. Figure
3 compares the diversity disagreement between our method with $10$-sub-
networks, a deep ensemble with $10$ members, and the single-network baseline.
The results clearly indicate that our proposed method achieves comparable
results with deep self-supervised ensembles in terms of both accuracy and
diversity disagreement.

### Computational Cost and Efficiency Analysis

We analyze the efficiency of our proposed method in Table. 1. SSL-Ensemble
increases memory and computational requirements compared to the baseline by
200% and 900% for 3 and 10 members, respectively. The increase in the number
of parameters is 32% and 143%, and the increase in computational requirement
is $\sim 0-6\%$ for our method. A more detailed description of the relative
cost and the reason for the difference between the increase in memory and the
computational requirements of our method is provided in Appendix (see
Computation Cost Analysis).

Table 1: Computational cost in 4 DGX-A100 40G GPUs (PyTorch) on CIFAR 10.

Method | Members | Parameters(M) | Memory / GPU | Time / 800-ep.
---|---|---|---|---
Baseline (SSL) | 1 | 28 | 9 G | 3.6 (h)
SSL-Ensemble | 3 | 3$\times$28 | 3$\times$9 G | 3$\times$ 3.6 (h)
SSL-Ensemble | 10 | 10$\times$28 | 10$\times$9 G | 10$\times$3.6 (h)
Our method | 3 | 37 | 9.2 G | 3.6 (h)
Our method | 10 | 68.1 | 10 G | 3.8 (h)

## Experimental Setup

We perform several experiments with a variety of self-supervised methods to
examine our hypothesis for robustness during both pretext-task learning and
downstream tasks (fine-tuning).

Deep self-supervised network architecture  Our proposed approach builds on two
recent popular self-supervised models in computer vision: i) SimCLR (Chen et
al. 2020b) is a contrastive learning framework that learns representations by
maximizing agreement on two different augmentations of the same image,
employing a contrastive loss in the latent embedding space of a convolutional
network architecture (e.g., ResNet-50 (He et al. 2016)), and ii) DINO (Caron
et al. 2021) is a self-distillation framework in which a student vision
transformer (ViT; (Dosovitskiy et al. 2021a)) learns to predict global
features from local image patches supervised by the cross-entropy loss from a
momentum teacher ViT’s embeddings. Furthermore, we study the impact of our
approach in NLP and modify SCD (Klein and Nabi 2022), which applies the
bidirectional training of transformers to language modeling. Here, the
objective is self-supervised contrastive divergence loss. Lastly, we examine
our approach on Self-GenomeNet (Gündüz et al. 2021), a contrastive self-
supervised learning algorithm for learning representations of genome
sequences. More detailed descriptions of the employed configurations are
provided in Appendix (see Implementation Details)

Deep independent sub-networks  We implement $M$ independent sub-networks on
top of the encoder, for which many possible architectures are conceivable. For
our experiments on computer vision datasets, we consider an ensemble of sub-
network architecture where each network includes a multi-layer perceptron
(MLP) with two layers of 2048 and 128 neurons, respectively, with ReLU as a
non-linearity and followed by batch normalization (Ioffe 2017). Each sub-
network has its own independent set of weights and learning parameters. For
the NLP dataset, the projector MLP contains three layers of 4096 neurons each,
also using ReLU activation’s as well as batch normalization. For the genomics
dataset, our ensemble of sub-networks includes one fully connected layer with
an embedding size of 256.

Optimization  For all experiments on image datasets based on DINO and SimCLR,
we follow the suggested hyperparameters and configurations by the paper (Caron
et al. 2021; Chen et al. 2020b). Implementation details for pretraining with
DINO on the 1000-classes ImagetNet dataset without labels are as follows:
coefficients $\epsilon$, $\alpha$, and $\lambda$ are respectively set to
$0.0001,0.15,$ and $2$ in Eq.Diversity loss., 2, and 3. We provide more
details in ablation studies (Section Ablation Study) on the number of sub-
networks and the coefficients $\lambda$ and $\alpha$ used in the loss
function. The encoder network $f_{\bm{\theta}}$ is either a ResNet-50 (He et
al. 2016) with 2048 output units when the baseline is SimCLR (Chen et al.
2020b) or ViT-s (Dosovitskiy et al. 2021b) with 384 output units when the
baseline is DINO (Caron et al. 2021). The best prediction and calibration
performance is achieved when the number of sub-networks is 5. We followed the
training protocol and settings suggested by (Caron et al. 2021).

Datasets  We use the following datasets in our experiments: CIFAR-10/100
(Krizhevsky 2009) are subsets of the tiny images dataset. Both datasets
include 50,000 images for training and 10,000 validation images of size
$32\times 32$ with 10 and 100 classes, respectively. SVH (Netzer et al. 2011)
is a digit classification benchmark dataset that contains 600,000 $32\times
32$ RGB images of printed digits (from 0 to 9) cropped from pictures of house
number plates. ImageNet (Deng et al. 2009), contains 1,000 classes, with 1.28
million training images and 50,000 validation images. For the NLP task, we
train on a dataset of 1 million randomly sampled sentences from Wikipedia
articles (Huggingface 2021) and evaluate our models on 7 different semantic
textual similarity datasets from the SentEval benchmark suite (Conneau and
Kiela 2018): MR (movie reviews), CR (product reviews), SUBJ (subjectivity
status), MPQA (opinion-polarity), SST-2 (sentiment analysis), TREC (question-
type classification), and MRPC (paraphrase detection). The T6SS effector
protein dataset is a public real-world bacteria dataset (SecReT6, (Li et al.
2015)) with actual label scarcity. The sequence length of the genome sample is
1000nt in all experiments.

Tasks  We examine and benchmark a model’s performance on different tasks
considering evaluation protocols by self-supervised learning (Chen et al.
2020b) and Plex’s benchmarking tasks (Tran et al. 2022). Specifically, we
evaluate our model on the basis of uncertainty-aware IND generalization, OOD
detection, semi-supervised learning, corrupted dataset evaluation (see Section
Results and Discussion), and transfer learning to other datasets and tasks
(see Appendix: Transfer to Other Tasks and Datasets )

Evaluation metrics  We report prediction/calibration performance with the
following metrics, where upward arrows indicate that higher values are
desirable, et vice versa. Top-1 accuracy $\uparrow$: share of test
observations for which the correct class is predicted. AUROC $\uparrow$: area
under the ROC curve arising from different combinations of false-positive and
false-negative rates (here: with positive and negative classes referring to
being in and out of distribution, respectively) for a gradually increasing
classification threshold. Negative log-likelihood (NLL) $\downarrow$: negative
log-likelihood of test observations under the estimated parameters. Expected
calibration error (ECE);(Naeini, Cooper, and Hauskrecht 2015) $\downarrow$:
mean absolute difference between accuracy and confidence (highest posterior
probability among predicted classes) across equally-spaced confidence bins,
weighted by relative number of samples per bin. Thresholded adaptive
calibration error (TACE); (Nixon et al. 2019)) $\downarrow$: modified ECE with
bins of equal sample size, rather than equal interval width, and omitting
predictions with posterior probabilities falling below a certain threshold
(here: 0.01) that often dominate the calibration in tasks with many classes.

Compared methods  We compare our method to the following contenders. Baseline:
self-supervised architectures (i.e., SimCLR, DINO, SCD, or Self-GenomeNet,
depending on the task). SSL-Ensemble: deep ensemble comprising a multiple of
the aforementioned baseline networks. Monte Carlo (MC) dropout: (Gal and
Ghahramani 2016) baseline networks with dropout regularization applied during
pretraining of baseline encoder. BatchEnsemble: baseline encoder with
BatchEnsemble applied during pretraining.

## Results and Discussion

In-distribution generalization  IND generalization (or prediction calibration)
quantifies how well model confidence aligns with model accuracy. We perform
several experiments on small and large image datasets as well as the genomics
sequence dataset to evaluate and compare the predictive performance of our
proposed model in IND generalization. Here, the base encoder $f_{\bm{\theta}}$
is frozen after unsupervised pretraining, and the model is trained on a
supervised linear classifier. The linear classifier is a fully connected layer
followed by softmax, which is placed on top of $f_{\bm{\theta}}$ after
removing the ensemble of sub-networks. High predictive scores and low
uncertainty scores are desired.

Figure 4 illustrates the predictive probability of correctness for our model
on CIFAR-10, CIFAR-100, ImageNet, and T6SS datasets in terms of Top-1
accuracy, ECE, and NLL, respectively. Based on Figure 4, our method achieves
better calibration (ECE and NLL) than the deep ensemble of self-supervised
models. The discrepancy in performance between our model and the deep ensemble
can be explained by various factors, including differences in uncertainty
modeling, complexity, and robustness. While the deep ensemble excels in top-1
accuracy, our model’s superior ECE and NLL scores indicate better-calibrated
and more reliable predictions, which are essential for safety-critical
applications and decision-making under uncertainty. More detailed descriptions
are provided in Appendix (see Additional Results) (Tables 4, 5, 6, and 7).

Figure 4: IND generalization in terms of (a) Top-1 Accuracy (b) ECE (c) NLL
averaged over in-distribution on test samples of CIFAR-10/100, ImageNet, T6SS
datasets. Here, we compare our method with the ensemble of deep self-
supervised networks (SSL-Ens), as well as the baseline. Detailed descriptions
of IND generalization for each dataset and other competitors are presented in
Appendix (see Additional Results) (Tables 4, 5, 6, and 7). Table 2:  OOD
detection. Results reported using AUROC show our method enhances the baseline
up to 6%.

IND | OOD | Baseline | SSL-Ensemble | Our method
---|---|---|---|---
CIFAR-100 | SVHN | 84.22 | 84.95 | 88.00
Uniform | 91.65 | 90.53 | 97.57
Gaussian | 90.00 | 89.42 | 94.10
CIFAR-10 | 74.71 | 74.80 | 75.18
CIFAR-10 | SVHN | 95.03 | 96.68 | 97.07
Uniform | 96.73 | 91.64 | 99.05
Gaussian | 96.39 | 93.24 | 99.24
CIFAR-100 | 91.79 | 91.59 | 91.87

Out-of-distribution detection  OOD detection shows how well a model can
recognize test samples from the classes that are unseen during training (Geng,
Huang, and Chen 2020). We perform several experiments to compare the model
generalization from IND to OOD datasets and to predict the uncertainty of the
models on OOD datasets. Evaluation is performed directly after unsupervised
pretraining without a fine-tuning step. Table 2 shows the AUROC on different
OOD sets for our model, baseline, and deep self-supervised ensemble. Our
approach improves overall compared to other methods.

Table 3: Semi-supervised evaluation: Top-1 accuracy (ACC), ECE, and NLL for
semi-supervised CIFAR-10/100 classification using 1% and 10% training
examples.

Method | CIFAR-10 (1%) | CIFAR-10 (10%) | CIFAR-100 (1%) | CIFAR-100 (10%)
---|---|---|---|---
| ACC | ECE | NLL | ACC | ECE | NLL | ACC | ECE | NLL | ACC | ECE | NLL
Baseline | 89.1 | 0.075 | 0.364 | 91.1 | 0.039 | 0.274 | 56.2 | 0.097 | 2.01 | 59.5 | 0.086 | 1.79
SSL-Ensemble | 90.1 | 0.056 | 0.334 | 92.2 | 0.050 | 0.257 | 59.7 | 0.081 | 1.86 | 62.6 | 0.053 | 1.48
Our method | 90.4 | 0.018 | 0.296 | 92.6 | 0.016 | 0.249 | 59.3 | 0.060 | 1.71 | 62.4 | 0.042 | 1.56

Semi-supervised evaluation  We explore and compare the performance of our
proposed method in the low-data regime. Again, the encoder $f_{\bm{\theta}}$
is frozen after self-supervised pretraining, and the model is trained on a
supervised linear classifier using 1% and 10% of the dataset. The linear
classifier is a fully connected layer followed by softmax. Table 3 shows the
result in terms of top-1 accuracy, ECE, and NLL. The results indicate that our
method outperforms other methods in the low-data regime – in terms of
calibration.

Figure 5: Performance under dataset corruption (CIFAR-10/100 with five levels
of increasing perturbation), evaluation in terms of AUROC and TACE for several
types of corruption (vertical spread).

Corrupted dataset evaluation  Another important component of model robustness
is its ability to make accurate predictions when the test data distribution
changes. Here, we evaluate model robustness under _covariate shift_. We employ
a configuration similar to the one found in (Tran et al. 2022). Figure 5
summarizes the improved performance across metrics of interest. The results
confirm that our method outperforms the baseline and achieves comparable
predictive performance as a deep self-supervised ensemble – both in terms of
calibration (TACE) and AUROC.

## Ablation Study

Figure 6: We compare the feature diversity for different subnetworks and
ensemble members. The top images are for different sub-networks, and the
bottom images are for different ensemble members. We used Grad-CAM (Selvaraju
et al. 2017) for visualization.

In order to build intuition around the behavior and the observed performance
of the proposed method, we further investigate the following aspects of our
approach in multiple ablation studies exploring: (1) the number $M$ of sub-
networks, (2) the role of each component of the proposed loss, and (3)
analysis of diversity with visualization of the gradients of subnetworks. We
also present more results on (4) the impact of our approach during pretraining
vs. at the finetuning step, (5) the size of sub-networks, and (6) the impact
of model parameters in the Appendix (see Additional Ablation Analysis) .

Number of sub-networks  We train $M$ individual deep neural networks on top of
the representation layer. The networks receive the same inputs but are
parameterized with different weights and biases. Here, we provide more details
regarding our experiments on IND generalization by considering varying $M$.
Fig. 7a compares the performance in terms of top-1 accuracy, ECE, and NLL for
CIFAR-10 and CIFAR-100. Based on the quantitative results depicted in Fig. 7a,
the predictive performance improves in both datasets when increasing the
number of sub-networks ($M$) until a certain point. For example, in the case
of CIFAR-10, when $M=3$, our performance is $91.9\%$; increasing $M$ to 10
levels top-1 accuracy up to $92.6\%$, while the ECE and NLL decrease from
$0.026$ and $0.249$ to $0.023$ and $0.222$, respectively. These findings
underline that training our sub-networks with a suitable number of heads can
lead to a better representation of the data and better calibration. Recently
(Wen and Li 2022; Tian, Chen, and Ganguli 2021) provided a theoretical
statement as well as experimental results that projection heads help with
faster convergence.

(a)

(b)

(c)

Figure 7: Ablation study on number of $M$ sub-networks (a), hyperparameters of
our proposed loss (b) $\lambda$ and (c) $\alpha$.

Analysis of loss  The total loss (Eq. 3) is calculated by the combination of
self-supervised loss (Eq. 1) and diversity loss (Eq. 2), where the mean value
of the embeddings across the ensemble of sub-networks is fed to the self-
supervised loss, and the corresponding standard deviation is used for the
diversity loss. First, we note that the use of our diversity regularizer
indeed improves calibration and provides better uncertainty prediction. The
results in Fig. 4 show the impact of our loss function in relation to the
baseline. By comparing the first and fifth rows of Table 4, it can be inferred
that our proposed loss function results in a much lower ECE ($0.016$) than the
network trained by SimCLR (baseline) with $0.039$ on the CIFAR-10 dataset.
Similarly, the first and third rows of Table 6 compare the predictive
probability of correctness of DINO (baseline) and our model on ImageNet.

Second, we explore different hyperparameter configurations to find the optimal
values for $\alpha$ and $\lambda$ in Fig. 7b, 7c. Note that, in practice,
$\alpha$ and $\lambda$ must be optimized jointly. The best top-1 accuracy in
our case is achieved when $\alpha$ and $\lambda$ are set to 0.08 and 1.5,
respectively, on the CIFAR-10 dataset.

Analysis of diversity  In addition to quantitative results for diversity
analysis provided in Figure 3, we visualize the activation map for the last
convolution layer in the encoder for each ensemble member and each subnetwork
to motivate the effect of subnetworks on the encoder. As illustrated in Fig.
6, different subnetworks have more feature diversity compared to the deep
ensemble as we expected.

## Conclusion

In this paper, we presented a novel diversified ensemble of self-supervised
framework. We achieved high predictive performance and good calibration using
a simple yet effective idea – an ensemble of independent sub-networks. We
introduced a new loss function to encourage diversity among different sub-
networks. It is straightforward to add our method to many existing self-
supervised learning frameworks during pretraining. Our extensive experimental
results show that our proposed method outperforms, or is on par with, an
ensemble of self-supervised baseline methods in many different experimental
settings.

## References

  * Ashukha et al. (2020) Ashukha, A.; Lyzhov, A.; Molchanov, D.; and Vetrov, D. 2020.  Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning.  In _International Conference on Learning Representations (ICLR)_.
  * Baevski et al. (2022) Baevski, A.; Hsu, W.; Xu, Q.; Babu, A.; Gu, J.; and Auli, M. 2022.  data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language.  In Chaudhuri, K.; Jegelka, S.; Song, L.; Szepesvári, C.; Niu, G.; and Sabato, S., eds., _International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA_ , volume 162 of _Proceedings of Machine Learning Research_ , 1298–1312. PMLR.
  * Bardes, Ponce, and LeCun (2021) Bardes, A.; Ponce, J.; and LeCun, Y. 2021.  Vicreg: Variance-invariance-covariance regularization for self-supervised learning.  _International Conference on Learning Representations (ICLR)_.
  * Brown et al. (2020) Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33: 1877–1901.
  * Caron et al. (2021) Caron, M.; Touvron, H.; Misra, I.; Jégou, H.; Mairal, J.; Bojanowski, P.; and Joulin, A. 2021.  Emerging properties in self-supervised vision transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 9650–9660.
  * Chen et al. (2020a) Chen, C.; Sun, X.; Hua, Y.; Dong, J.; and Xv, H. 2020a.  Learning deep relations to promote saliency detection.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 10510–10517.
  * Chen et al. (2020b) Chen, T.; Kornblith, S.; Norouzi, M.; and Hinton, G. 2020b.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , 1597–1607. PMLR.
  * Cisse et al. (2017) Cisse, M.; Bojanowski, P.; Grave, E.; Dauphin, Y.; and Usunier, N. 2017.  Parseval networks: Improving robustness to adversarial examples.  In _International Conference on Machine Learning_ , 854–863. PMLR.
  * Conneau and Kiela (2018) Conneau, A.; and Kiela, D. 2018.  SentEval: An Evaluation Toolkit for Universal Sentence Representations.  In Calzolari, N.; Choukri, K.; Cieri, C.; Declerck, T.; Goggi, S.; Hasida, K.; Isahara, H.; Maegaard, B.; Mariani, J.; Mazo, H.; Moreno, A.; Odijk, J.; Piperidis, S.; and Tokunaga, T., eds., _Proceedings of the Eleventh International Conference on Language Resources and Evaluation, LREC 2018, Miyazaki, Japan, May 7-12, 2018_. European Language Resources Association (ELRA).
  * Dabouei et al. (2020) Dabouei, A.; Soleymani, S.; Taherkhani, F.; Dawson, J.; and Nasrabadi, N. M. 2020\.  Exploiting joint robustness to adversarial perturbations.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 1122–1131.
  * Deng et al. (2009) Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , 248–255. Ieee.
  * Devlin et al. (2018) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _ACL_.
  * Devlin et al. (2019) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , 4171–4186. Minneapolis, Minnesota: Association for Computational Linguistics.
  * Dietterich (2000) Dietterich, T. G. 2000.  Ensemble methods in machine learning.  In _International workshop on multiple classifier systems_ , 1–15. Springer.
  * Dosovitskiy et al. (2021a) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; Uszkoreit, J.; and Houlsby, N. 2021a.  An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale.  In _Proceedings of the 9th International Conference on Learning Representations (ICLR)_.
  * Dosovitskiy et al. (2021b) Dosovitskiy, A.; Beyer, L.; Kolesnikov, A.; Weissenborn, D.; Zhai, X.; Unterthiner, T.; Dehghani, M.; Minderer, M.; Heigold, G.; Gelly, S.; et al. 2021b.  An image is worth 16x16 words: Transformers for image recognition at scale.
  * Durasov et al. (2021) Durasov, N.; Bagautdinov, T.; Baque, P.; and Fua, P. 2021.  Masksembles for uncertainty estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 13539–13548.
  * Fischer (2020) Fischer, I. 2020.  The conditional entropy bottleneck.  _Entropy_ , 22(9): 999.
  * Fort, Hu, and Lakshminarayanan (2019) Fort, S.; Hu, H.; and Lakshminarayanan, B. 2019.  Deep ensembles: A loss landscape perspective.  _arXiv preprint arXiv:1912.02757_.
  * Gal and Ghahramani (2016) Gal, Y.; and Ghahramani, Z. 2016.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In _international conference on machine learning_ , 1050–1059. PMLR.
  * Geng, Huang, and Chen (2020) Geng, C.; Huang, S.-j.; and Chen, S. 2020.  Recent advances in open set recognition: A survey.  _IEEE transactions on pattern analysis and machine intelligence_ , 43(10): 3614–3631.
  * Grill et al. (2020) Grill, J.-B.; Strub, F.; Altché, F.; Tallec, C.; Richemond, P.; Buchatskaya, E.; Doersch, C.; Avila Pires, B.; Guo, Z.; Gheshlaghi Azar, M.; et al. 2020.  Bootstrap your own latent-a new approach to self-supervised learning.  _Advances in neural information processing systems_ , 33: 21271–21284.
  * Gündüz et al. (2021) Gündüz, H. A.; Binder, M.; To, X.-Y.; Mreches, R.; Münch, P. C.; McHardy, A. C.; Bischl, B.; and Rezaei, M. 2021.  Self-GenomeNet: Self-supervised Learning with Reverse-Complement Context Prediction for Nucleotide-level Genomics Data.
  * Gustafsson, Danelljan, and Schon (2020) Gustafsson, F. K.; Danelljan, M.; and Schon, T. B. 2020.  Evaluating scalable bayesian deep learning methods for robust computer vision.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_.
  * Hansen and Salamon (1990) Hansen, L. K.; and Salamon, P. 1990.  Neural network ensembles.  _IEEE transactions on pattern analysis and machine intelligence_ , 12(10): 993–1001.
  * Havasi et al. (2021) Havasi, M.; Jenatton, R.; Fort, S.; Liu, J. Z.; Snoek, J.; Lakshminarayanan, B.; Dai, A. M.; and Tran, D. 2021.  Training independent subnetworks for robust prediction.  In _International Conference on Learning Representations (ICLR)_.
  * He et al. (2022) He, K.; Chen, X.; Xie, S.; Li, Y.; Dollár, P.; and Girshick, R. 2022.  Masked autoencoders are scalable vision learners.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 16000–16009.
  * He et al. (2016) He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778.
  * Huggingface (2021) Huggingface. 2021.  wiki1m_for_simcse.txt.
  * Hüllermeier and Waegeman (2021) Hüllermeier, E.; and Waegeman, W. 2021.  Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods.  _Machine Learning_ , 110: 457–506.
  * Ioffe (2017) Ioffe, S. 2017.  Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_ , 1945–1953.
  * Jaiswal et al. (2020) Jaiswal, A.; Babu, A. R.; Zadeh, M. Z.; Banerjee, D.; and Makedon, F. 2020.  A Survey on Contrastive Self-Supervised Learning.  _Technologies_ , 9(1).
  * Klein and Nabi (2022) Klein, T.; and Nabi, M. 2022.  SCD: Self-Contrastive Decorrelation for Sentence Embeddings.  _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Krizhevsky (2009) Krizhevsky, A. 2009.  Learning multiple layers of features from tiny images.  Technical report, University of Toronto.
  * Lakshminarayanan, Pritzel, and Blundell (2017) Lakshminarayanan, B.; Pritzel, A.; and Blundell, C. 2017.  Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles.  In _Advances in Neural Information Processing Systems_ , volume 30.
  * Li et al. (2015) Li, J.; Yao, Y.; Xu, H. H.; Hao, L.; Deng, Z.; Rajakumar, K.; and Ou, H.-Y. 2015\.  SecReT6: a web-based resource for type VI secretion systems found in bacteria.  _Environmental microbiology_ , 17(7): 2196–2202.
  * Li et al. (2022) Li, M.; Xu, R.; Wang, S.; Zhou, L.; Lin, X.; Zhu, C.; Zeng, M.; Ji, H.; and Chang, S.-F. 2022.  Clip-event: Connecting text and images with event structures.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 16420–16429.
  * Liang, Li, and Srikant (2018) Liang, S.; Li, Y.; and Srikant, R. 2018.  Enhancing the reliability of out-of-distribution image detection in neural networks.  _International Conference on Learning Representations (ICLR)_.
  * Lienen, Demir, and Hüllermeier (2022) Lienen, J.; Demir, C.; and Hüllermeier, E. 2022.  Conformal credal self-supervised learning.  _arXiv preprint arXiv:2205.15239_.
  * Naeini, Cooper, and Hauskrecht (2015) Naeini, M. P.; Cooper, G. F.; and Hauskrecht, M. 2015.  Obtaining Well Calibrated Probabilities Using Bayesian Binning.  In _Proceedings of AAAI’15_ , 2901–2907. AAAI Press.
  * Netzer et al. (2011) Netzer, Y.; Wang, T.; Coates, A.; Bissacco, A.; Wu, B.; and Ng, A. Y. 2011.  Reading digits in natural images with unsupervised feature learning.
  * Nixon et al. (2019) Nixon, J.; Dusenberry, M.; Jerfel, G.; Nguyen, T.; Liu, J.; Zhang, L.; and Tran, D. 2019.  Measuring Calibration in Deep Learning.
  * Ortega, Cabañas, and Masegosa (2022) Ortega, L. A.; Cabañas, R.; and Masegosa, A. 2022.  Diversity and generalization in neural network ensembles.  In _International Conference on Artificial Intelligence and Statistics_ , 11720–11743. PMLR.
  * Ovadia et al. (2019) Ovadia, Y.; Fertig, E.; Ren, J.; Nado, Z.; Sculley, D.; Nowozin, S.; Dillon, J.; Lakshminarayanan, B.; and Snoek, J. 2019.  Can you trust your model's uncertainty? Evaluating predictive uncertainty under dataset shift.  In _Advances in Neural Information Processing Systems_ , volume 32.
  * Pang et al. (2019) Pang, T.; Xu, K.; Du, C.; Chen, N.; and Zhu, J. 2019.  Improving adversarial robustness via promoting ensemble diversity.  In _International Conference on Machine Learning_ , 4970–4979. PMLR.
  * Radford et al. (2021) Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; et al. 2021.  Learning transferable visual models from natural language supervision.  In _International Conference on Machine Learning_ , 8748–8763. PMLR.
  * Ramé and Cord (2021) Ramé, A.; and Cord, M. 2021.  DICE: Diversity in Deep Ensembles via Conditional Redundancy Adversarial Estimation.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Rezaei et al. (2022a) Rezaei, M.; Näppi, J.; Bischl, B.; and Yoshida, H. 2022a.  Deep mutual GANs: representation learning from multiple experts.  In _Medical Imaging 2022: Imaging Informatics for Healthcare, Research, and Applications_ , volume 12037, 191–197. SPIE.
  * Rezaei et al. (2022b) Rezaei, M.; Näppi, J. J.; Bischl, B.; and Yoshida, H. 2022b.  Bayesian uncertainty estimation for detection of long-tail and unseen conditions in abdominal images.  In _Medical Imaging 2022: Computer-Aided Diagnosis_ , volume 12033, 270–276. SPIE.
  * Rezaei et al. (2023) Rezaei, M.; Soleymani, F.; Bischl, B.; and Azizi, S. 2023.  Deep Bregman divergence for self-supervised representations learning.  _Computer Vision and Image Understanding_ , 103801.
  * Sehwag, Chiang, and Mittal (2021) Sehwag, V.; Chiang, M.; and Mittal, P. 2021.  SSD: A Unified Framework for Self-Supervised Outlier Detection.  _CoRR_ , abs/2103.12051.
  * Selvaraju et al. (2017) Selvaraju, R. R.; Cogswell, M.; Das, A.; Vedantam, R.; Parikh, D.; and Batra, D. 2017.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In _Proceedings of the IEEE international conference on computer vision_ , 618–626.
  * Shi et al. (2022) Shi, B.; Hsu, W.-N.; Lakhotia, K.; and Mohamed, A. 2022.  Learning audio-visual speech representation by masked multimodal cluster prediction.  _International Conference on Learning Representations (ICLR)_.
  * Sinha et al. (2021) Sinha, S.; Bharadhwaj, H.; Goyal, A.; Larochelle, H.; Garg, A.; and Shkurti, F. 2021\.  Dibs: Diversity inducing information bottleneck in model ensembles.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , 9666–9674.
  * Tian, Chen, and Ganguli (2021) Tian, Y.; Chen, X.; and Ganguli, S. 2021.  Understanding self-supervised learning dynamics without contrastive pairs.  In _International Conference on Machine Learning_ , 10268–10278. PMLR.
  * Tran et al. (2022) Tran, D.; Liu, J.; Dusenberry, M. W.; Phan, D.; Collier, M.; Ren, J.; Han, K.; Wang, Z.; Mariet, Z.; Hu, H.; et al. 2022.  Plex: Towards reliability using pretrained large model extensions.  _arXiv preprint arXiv:2207.07411_.
  * Tsuzuku, Sato, and Sugiyama (2018) Tsuzuku, Y.; Sato, I.; and Sugiyama, M. 2018.  Lipschitz-margin training: Scalable certification of perturbation invariance for deep neural networks.  _Advances in neural information processing systems_ , 31.
  * Wen, Tran, and Ba (2020) Wen, Y.; Tran, D.; and Ba, J. 2020.  Batchensemble: an alternative approach to efficient ensemble and lifelong learning.  In _International Conference on Learning Representations (ICLR)_.
  * Wen and Li (2022) Wen, Z.; and Li, Y. 2022.  The Mechanism of Prediction Head in Non-contrastive Self-supervised Learning.  _Advances in Neural Information Processing Systems_.
  * Winkens et al. (2020) Winkens, J.; Bunel, R.; Roy, A. G.; Stanforth, R.; Natarajan, V.; Ledsam, J. R.; MacWilliams, P.; Kohli, P.; Karthikesalingam, A.; Kohl, S.; Cemgil, A. T.; Eslami, S. M. A.; and Ronneberger, O. 2020.  Contrastive Training for Improved Out-of-Distribution Detection.  _CoRR_ , abs/2007.05566.
  * Zbontar et al. (2021) Zbontar, J.; Jing, L.; Misra, I.; LeCun, Y.; and Deny, S. 2021.  Barlow twins: Self-supervised learning via redundancy reduction.  In _International Conference on Machine Learning_ , 12310–12320. PMLR.
  * Zhang and Ma (2012) Zhang, C.; and Ma, Y. 2012.  _Ensemble Machine Learning: Methods and Applications_.  Springer Publishing Company, Incorporated.  ISBN 1441993258.

## Implementation Details

### Computation Cost Analysis

Figure 8 illustrates relative computation cost – as compared to the baseline –
in terms of the number of parameters, computation time, and memory required
between our model and a deep self-supervised ensemble.

Figure 8: The test time cost (purple, dotted) and memory cost (purple, dashed)
of our model w.r.t. the ensemble size. The figures are relative to the cost
incurred by a single model (green). The inference time cost and memory cost of
a deep self-supervised ensemble are plotted in blue.

### Computational Cost Analysis

As we mentioned in Section Method the increase in the number of parameters is
32% and 143%, and the increase in computational requirement is negligible and
6% for our method compared to the baseline when there exist 3 and 10 ensemble
members, respectively. We would like to explain the reason as follows:

While the encoder networks used in the baseline methods (and our method)
contain many convolutional layers, the additional parameters introduced by our
method are in the projection head, and they are a few linear layers. Although
these additional linear layers increase the number of parameters to some
extent, the computational burden introduced by them is much more limited
compared to the convolutional layers that exist in both baselines and our
method. That is because convolutional layers typically contain fewer
parameters compared to fully connected layers due to parameter-sharing but
have a much higher computational burden since their outputs are evaluated over
the whole image. A toy example to understand this would be the comparison of
the two settings below:

  1. 1.

Consider a 100 $\times$ 100 image fed into a convolutional filter with (kernel
size= 3 $\times$ 3, stride= 1 $\times$ 1, padding= ”same”, bias= False). The
convolutional filter has 9 parameters but needs to do 100 $\times$ 100
$\times$ 9 = 90000 multiplications to evaluate its output.

  2. 2.

Consider a vector of 1000 that is fed into a fully connected layer (without
bias) to produce 1 output value. The fully connected layer contains 1000
parameters and the number of multiplications needed to evaluate its output is
also 1000.

Comparing these two settings, the convolutional layer needs $\sim$100 times
more computational burden (convolutional: 90000 vs dense: 1000
multiplications) to evaluate its outcome, although it has $\sim$100 times
fewer parameters (convolutional: 9 vs dense: 1000) compared to the fully
connected layer.

Similarly, the increase in memory requirements is low for our method compared
to the SSL-Ensemble, but the increase in computational requirements is much
lower and even negligable.

### Data Augmentation for Computer Vision Datasets

We define a random transformation function $\bm{T}$ that applies a combination
of crop, horizontal flip, color jitter, and grayscale. Similar to (Chen et al.
2020b), we perform crops with a random size from $0.2$ to $1.0$ of the
original area and a random aspect ratio from $3/4$ to $4/3$ of the original
aspect ratio. We also apply horizontal mirroring with a probability of $0.5$.
Then, we apply grayscale with a probability of $0.2$ as well as color
jittering with a probability of $0.8$ and a configuration of
$(0.4,0.4,0.4,0.1)$. However, for ImageNet, we define augmentation based on
the original DINO from their official repository. In all experiments, at the
testing phase, we apply only resize and center crop.

### Hyperparameters for Self-supervised Network Architectures

SimCLR (Chen et al. 2020b): we use ResNet-50 as a backbone, a loss temperature
of 0.07, batch size 512, and a cosine-annealing learning rate scheduler. The
embedding size is 2048, and we train for 800 epochs during pretraining. DINO
(Caron et al. 2021): we use ViT-small as a backbone, patch size 16, batch size
1024, and a cosine-annealing learning rate scheduler. The embedding size is
384/1536, and we train for 100 epochs during pretraining.

## Additional Results

### Robustness of representation: IND- Generalization

Tables 4, 5, and 6 present results for the predictive performance and
calibration of our model on CIFAR-10, CIFAR-100, and ImageNet respectively.
Based on Table 4, our method achieves better calibration than the deep
ensemble of self-supervised networks, MC-Dropout, and BatchEnsemble, with
significant margins at large ensemble sizes. In order to have multiple batches
for BatchEnsemble, we decreased the initial batch size because of memory, so
we ended up with a smaller batch size to which the self-supervised model
(i.e., SimCLR) is sensitive. Also, each time we have more positive samples
than the original.

In the case of dropouts, we again face the same problem with positive and
negative samples. Dropouts also count as data regularization, but when applied
randomly to all data in contrastive learning, it degrades the idea of positive
and negative. For example, in NLP, dropouts are used to produce different
augmentations.

Table 4:  IND Generalization: Top-1 accuracy, ECE and NLL averaged over in-
distribution on test samples of the CIFAR-10 dataset over three random seeds.
The best score for each metric is shown in bold, and the second-best is
underlined.

Method | Top-1 Acc (%) ($\uparrow$) | ECE ($\downarrow$) | NLL ($\downarrow$)
---|---|---|---
# member ($M$) | 3 | 5 | 10 | 3 | 5 | 10 | 3 | 5 | 10
Baseline | 92.8 $\pm$ 0.4 | 0.039 $\pm$ 0.002 | 0.233 $\pm$ 0.011
SSL-Ensemble | $92.8\pm 0.1$ | $93.0\pm 0.2$ | 94.2 $\pm$ 0.3 | $0.043\pm 0.02$ | $0.033\pm 0.01$ | 0.029 $\pm$ 0.02 | $0.221\pm 0.011$ | $0.226\pm 0.009$ | $0.199\pm 0.004$
MC Dropout | $65.7\pm 0.2$ | $66.3\pm 0.2$ | $66.4\pm 0.2$ | $0.083\pm 0.014$ | $0.077\pm 0.009$ | $0.075\pm 0.005$ | $0.66\pm 0.012$ | $0.637\pm 0.002$ | $0.593\pm 0.006$
BatchEnsemble | $69.1\pm 04$ | $72.1\pm 0.3$ | $71.9\pm 0.2$ | $0.064\pm 0.011$ | $0.061\pm 0.008$ | $0.062\pm 0.005$ | $0.613\pm xx$ | $0.58\pm 0.007$ | $0.551\pm 0.004$
Our method | $92.6\pm 0.2$ | $92.9\pm 0.1$ | $93.6\pm 0.1$ | $0.021\pm 0.004$ | $0.019\pm 0.002$ | 0.016 $\pm$ 0.001 | $0.241\pm 0.010$ | $0.221\pm 0.005$ | 0.193 $\pm$ 0.003

Table 5:  IND Generalization: Top-1 accuracy, ECE and NLL averaged over in-
distribution on test samples of the CIFAR-100 dataset over three random seeds.
The best score for each metric is shown in bold, and the second-best is
underlined.

Method | Top-1 Acc (%) ($\uparrow$) | ECE ($\downarrow$) | NLL ($\downarrow$)
---|---|---|---
# member ($M$) | 3 | 5 | 10 | 3 | 5 | 10 | 3 | 5 | 10
Baseline | 68.9 $\pm$ 0.3 | 0.086 $\pm$ 0.014 | 1.28 $\pm$ 0.05
SSL-Ensemble | $70.6\pm 0.12$ | $71.4\pm 0.5$ | 72.0 $\pm$ 0.2 | $0.12\pm 0.01$ | $0.122\pm 0.01$ | $0.119\pm 0.04$ | $1.09\pm 0.01$ | $1.12\pm 0.01$ | $1.06\pm 0.02$
MC Dropout | $46.3\pm 0.1$ | $45.2\pm 0.4$ | $48.2\pm 0.1$ | $0.077\pm 0.012$ | $0.081\pm 0.002$ | $0.071\pm 0.002$ | $2.66\pm 0.11$ | $2.37\pm 0.02$ | $2.43\pm 0.06$
BatchEnsemble | $44.1\pm 03$ | $45.2\pm 0.2$ | $46.1\pm 0.1$ | $0.073\pm 0.01$ | $0.071\pm 0.08$ | 0.068 $\pm$ 0.001 | $2.43\pm 0.03$ | $2.64\pm 0.007$ | $2.51\pm 0.004$
Our method | $67.7\pm 0.1$ | 68.8 $\pm$ 0.1 | $70.1\pm 0.0$ | $0.067\pm 0.001$ | $0.063\pm 0.001$ | 0.048 $\pm$ 0.000 | $0.114\pm 0.005$ | $0.116\pm 0.0002$ | 1.06 $\pm$ 0.001

Table 6:  IND Generalization: Top-1 accuracy, ECE and NLL averaged over in-
distribution on test samples of the ImageNet dataset over three random seeds.
The best score for each metric is shown in bold, and the second-best is
underlined.

Method | Top-1 Acc (%) ($\uparrow$) | ECE ($\downarrow$) | NLL ($\downarrow$)
---|---|---|---
Baseline | 73.8 $\pm$ 0.3 | 0.013 $\pm$ 0.015 | 1.05 $\pm$ 0.01
SSL-Ensemble | 75.1 $\pm$ 0.1 | 0.014 $\pm$ 0.000 | 0.98 $\pm$ 0.01
Our method | 74.0 $\pm$ 0.0 | 0.010 $\pm$ 0.000 | 1.03 $\pm$ 0.01

Table 7: IND Generalization: Top-1 accuracy, ECE and NLL averaged over in-
distribution on test samples of the T6SS Identification dataset over three
random seeds. The best score for each metric is shown in bold, and the second-
best is underlined.

Method | Top-1 Acc (%) ($\uparrow$) | ECE ($\downarrow$) | NLL ($\downarrow$)
---|---|---|---
Baseline | 75.9 $\pm$ 2.0 | 0.100 $\pm$ 0.006 | 0.502 $\pm$ 0.020
SSL-Ensemble | 80.2 $\pm$ 0.7 | 0.099 $\pm$ 0.014 | 0.471 $\pm$ 0.011
Our method | 76.7 $\pm$ 2.3 | 0.108 $\pm$ 0.006 | 0.492 $\pm$ 0.024

We also performed experiments on a dataset of 1-dimensional genomic sequences
– the T6SS identification of effector proteins– to demonstrate that
uncertainty-aware subnetworks can also be readily combined with existing
models for 1-dimensional datasets and models. Based on Table 7, our method
improves the accuracy and the calibration compared to the baseline.

### Transfer to Other Tasks and Datasets

We further assess the generalization capacity of the learned representation on
learning a new task in NLP. We train our model without any labels on a dataset
of sentences from Wikipedia (Huggingface 2021) and fine-tune the pretrained
representation on seven different semantic textual similarity datasets from
the SentEval benchmark suite (Conneau and Kiela 2018): MR (movie reviews), CR
(product reviews), SUBJ (subjectivity status), MPQA (opinion-polarity), SST-2
(sentiment analysis), TREC (question-type classification), and MRPC
(paraphrase detection). Then, we evaluate the test set of each dataset. Figure
9 provides a comparison of the transfer learning performance of our self-
supervised approach for different tasks. Our results in Figure 9 indicate that
our approach performs comparably to or better than the baseline method.

Figure 9: Transfer to other dataset and tasks: Comparision of Sentence
embedding performance on semantic textual similarity tasks.

We test the performance of the trained model on ImageNet (Deng et al. 2009) on
CIFAR-10 (Krizhevsky 2009) dataset where the model is trained for 100 epochs.

Table 8: Transfer to other dataset: Expected calibration error averaged over
uncertainty-aware evaluation on CIFAR-10 datasets.

Method | ACC (%) ($\uparrow$) | ECE ($\downarrow$) | NLL ($\downarrow$) | TACE ($\downarrow$)
---|---|---|---|---
Baseline | 73.5 | 0.038 | 0.78 | 0.20
Our method | 73.9 | 0.030 | 0.75 | 0.18

## Additional Ablation Analysis

Efficient ensemble of sub-networks at pretraining vs. finetuning We performed
additional experiments to study the efficiency of proposed loss and
independent sub-networks (InSub) i) during pretraining, ii) during finetuning,
and iii) during both pretraining and finetuning. As shown in Table 9,
pretraining with an ensemble of sub-networks is beneficial, and additional
fine-tuning with multiple heads can further improve performance.

Table 9: Pretraining vs. Finetuning: Expected calibration error averaged over
uncertainty-aware evaluation on CIFAR-10 datasets. InSub refers to training
with our proposed Independent Subnetworks

Method | ACC (%) ($\uparrow$) | ECE ($\downarrow$) | NLL ($\downarrow$) | TACE ($\downarrow$)
---|---|---|---|---
Baseline | 92.5 | 0.039 | 0.238 | 0.133
Pretrain-InSub | 92.6 | 0.032 | 0.226 | 0.131
Finetune-InSub | 92.6 | 0.021 | 0.222 | 0.103
Pretrain-InSub + Finetune-InSub | 92.8 | 0.023 | 0.227 | 0.115

Table 10: Sub-Network Size: Expected calibration error averaged over
uncertainty-aware evaluation on CIFAR-10 datasets.

Method | ACC (%) ($\uparrow$) | ECE ($\downarrow$) | NLL ($\downarrow$)
---|---|---|---
Our method with 5 sub-network (100%) | 92.9 | 0.019 | 0.221
With 25 percent of sub-network size | 92.3 | 0.026 | 0.231
With 50 percent of sub-network size | 92.6 | 0.021 | 0.226
With 75 percent of sub-network size | 92.6 | 0.019 | 0.221

Table 11: Large variant encoder: Expected calibration error averaged over
uncertainty-aware evaluation on CIFAR-10 datasets.

Method | ACC (%) ($\uparrow$) | ECE ($\downarrow$) | NLL ($\downarrow$) | Number of parameters (M)
---|---|---|---|---
Our method with ResNet50 as a encoder with 5 sub-networks | 92.9 | 0.019 | 0.221 | 45.79
Baseline with ResNet101 as a encoder | 93.2 | 0.027 | 0.202 | 46.95

Table 12: Different encoder (medium size): Expected calibration error averaged
over uncertainty-aware evaluation on CIFAR-10 datasets.

Method | ACC (%) ($\uparrow$) | ECE ($\downarrow$) | NLL ($\downarrow$) | Number of parameters (M)
---|---|---|---|---
Our method with ResNet34 as a encoder with 20 sub-networks | 92.5 | 0.016 | 0.23 | 27.84
Baseline with ResNet50 as a encoder | 92.8 | 0.039 | 0.233 | 27.89

### Analysis of Size of Sub-Networks

We perform several experiments to study the different sizes of sub-network. As
shown in Table 10, the dimension of projection heads does not change the top-1
accuracy. Recent self-supervised models such as SimCLR (Chen et al. 2020b),
BarlowTwins (Zbontar et al. 2021) also reach the same results with different
projection head sizes.

### Impact of Model Parameters

Our project aims to improve the predictive uncertainty of the baseline without
losing predictive performance by mimicking the ensembles of self-supervised
models with much lower computational costs. According to the results shown in
Table 11, a bigger encoder can potentially improve the predictive performance,
but it does not necessarily improve the predictive uncertainty of the results.
We used ResNet101 as a baseline with more parameters in the encoder. To have a
fair comparison, we compare it with our model with five heads. Our model
performs better in ECE and NLL and has comparable accuracy.

Also, we used ResNet34 as a baseline with fewer parameters in the encoder with
twenty heads and compared it with baseline ResNet50 with one head. According
to results obtained in Table 12, our model performs better in terms of ECE and
NLL and has on-par accuracy.

### Source Code

Please find the source code in the supplementary material.

## Theoretical Supplement

### Proof for Eq. 6

$\displaystyle{\frac{\partial\left(\ell_{div}\right)}{\partial\bm{z}_{k,\hat{m},o}}}=\tfrac{-1}{2}\underbrace{\left(\tfrac{1}{M-1}\textstyle\sum_{m=1}^{M}(\bm{z}_{k,m,o}-\bar{\bm{z}}_{k,o})^{2}\right)}_{A}\cdot{\frac{\partial\left(\tfrac{1}{M-1}\textstyle\sum_{m=1}^{M}(\bm{z}_{k,m,o}-\bar{\bm{z}}_{k,o})^{2}\right)}{\partial\bm{z}_{k,\hat{m},o}}}$
$\displaystyle=$ (8)
$\displaystyle\tfrac{-A}{2}\cdot\tfrac{1}{M-1}\left[2\cdot\left[(\bm{z}_{k,\hat{m},o}-\bar{\bm{z}}_{k,o})\cdot\left({\frac{\partial\
\bm{z}_{k,\hat{m},o}}{\partial\bm{z}_{k,\hat{m},o}}}-{\frac{\partial\bar{\bm{z}}_{k,o}}{\partial\bm{z}_{k,\hat{m},o}}}\right)+\textstyle\sum_{m=1}^{M}\mathbb{I}_{[m\neq\hat{m}]}(\bm{z}_{k,m,o}-\bar{\bm{z}}_{k,o})\cdot\left({\frac{\partial\bm{z}_{k,m,o}}{\partial\bm{z}_{k,\hat{m},o}}}-{\frac{\partial\bar{\bm{z}}_{k,o}}{\partial\bm{z}_{k,\hat{m},o}}}\right)\right]\right]$
$\displaystyle=$
$\displaystyle\tfrac{-A}{2}\cdot\tfrac{1}{M-1}\left[2\cdot\left[(\bm{z}_{k,\hat{m},o}-\bar{\bm{z}}_{k,o})\cdot\left(1-\tfrac{1}{M}\right)+\textstyle\sum_{m=1}^{M}\mathbb{I}_{[m\neq\hat{m}]}(\bm{z}_{k,m,o}-\bar{\bm{z}}_{k,o})\cdot\left(\tfrac{-1}{M}\right)\right]\right]$
$\displaystyle=$
$\displaystyle\tfrac{-A}{M-1}\cdot\left[(\bm{z}_{k,\hat{m},o}-\bar{\bm{z}}_{k,o})\cdot\tfrac{M-1}{M}+\left(\left(\tfrac{\bm{z}_{k,\hat{m},o}}{M}\right)-\bar{\bm{z}}_{k,o}+\tfrac{M-1}{M}\cdot\bar{\bm{z}}_{k,o}\right)\right]$
$\displaystyle=$
$\displaystyle\tfrac{-A}{M-1}\cdot(\bm{z}_{k,\hat{m},o}-\bar{\bm{z}}_{k,o})$

### Proof for Eq. 7

$\eta\cdot\nabla_{w_{\hat{m},o}}\ell_{div}=\eta\cdot{\frac{\partial\ell_{div}}{\partial\bm{z}_{k,\hat{m},o}}}\cdot{\frac{\partial\bm{z}_{k,\hat{m},o}}{\partial
w_{\hat{m},o}}}=\eta\cdot\frac{-A}{M-1}\cdot(\bm{z}_{k,\hat{m},o}-\bar{\bm{z}}_{k,o})\cdot
b$