Unsupervised Learning on a DIET: Datum IndEx as Target
Free of Self-Supervision, Reconstruction, Projector Head

Randall Balestrieroa

aMeta AI, FAIR, NYC, USA

Machine Learning, ICML

Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale  Use footnote for providing further information
about author (webpage, alternative address)—not for acknowledging
funding agencies.  Funding acknowledgements go at the end of the paper.

Department of Computer Science

Cranberry-Lemon University

Pittsburgh, PA 15213, USA

Ji Q. Ren & Yevgeny LeNet

Department of Computational Neuroscience

University of the Witwatersrand

Joburg, South Africa

Costly, noisy, and over-specialized, labels are to be set aside in favor of unsupervised learning if we hope to learn cheap, reliable, and transferable models. To that end, spectral embedding, self-supervised learning, or generative modeling have offered competitive solutions. Those methods however come with numerous challenges e.g. estimating geodesic distances, specifying projector architectures and anti-collapse losses, or specifying decoder architectures and reconstruction losses. In contrast, we introduce a simple explainable alternative —coined DIET— to learn representations from unlabeled data, free of those challenges. DIET is blatantly simple: take one's favorite classification setup and use the Datum IndEx as its Target class, i.e. each sample is its own class, no further changes needed. DIET works without a decoder/projector network, is not based on positive pairs nor reconstruction, introduces no hyper-parameters, and works out-of-the-box across datasets and architectures. Despite DIET's simplicity, the learned representations are of high-quality and often on-par with the state-of-the-art e.g. using a linear classifier on top of DIET's learned representation reaches $71.4\%$ on CIFAR100 with a Resnet101, $52.5\%$ on TinyImagenet with a Resnext50.

§ INTRODUCTION

Unsupervised learning of a model $f_{\vtheta}$, governed by some parameter $\vtheta$, has always been and still is one of the most challenging and rewarding task in deep learning [Bengio, 2012]. In fact, supervised learning which learns to produce predictions from known input-output pairs can be considered solved in contrast to unsupervised learning which aims to produce descriptive and intelligible representations from inputs only [Hastie et al., 2009, Goodfellow et al., 2016].

Self-Supervised Learning (SSL) [Chen et al., 2020, Misra & Maaten, 2020] has recently demonstrated that one could learn without labels  highly non-trivial Deep Networks (DNs) whose representations are as descriptive as supervised ones. In particular, SSL differs from reconstruction-based methods such as (denoising, variational) Autoencoders [Vincent et al., 2008, Vincent et al., 2010, Kingma & Welling, 2013] and their cross-variants by removing the need for a decoder DN and an input-space reconstruction loss, both being difficult to design [Wang et al., 2004, Grimes & Rao, 2005, Larsen et al., 2016, Cosentino et al., 2022]. SSL's recent developments have also led to outperforming reconstruction-free methods e.g. Noise Contrastive Estimation and its variants [Hyvärinen & Dayan, 2005, Hinton, 2002, Song & Ermon, 2019, Rhodes et al., 2020].
Nonetheless, SSL which is the current state-of-the-art unsupervised learning solution, comes with many moving pieces e.g. (i) artificially constructed positive pairs, commonly obtained by applying two a priori known and tailored Data-Augmentations (DAs) to each datum, (ii) a carefully designed projector DN $g_{\vgamma}$ to perform SSL training with the composition $g_{\vgamma} \circ f_{\vtheta}$ and throwing away the projector $g_{\vgamma}$ afterwards [Chen et al., 2020], or (iii) advanced anti-collapse techniques involving moving average teacher models [Grill et al., 2020, Caron et al., 2021], representation normalization [Chen & He, 2021, Zbontar et al., 2021], or Entropy estimation [Chen et al., 2020, Li et al., 2021]. An incorrect pick of any of those moving pieces results in a drastic drop in performances [Cosentino et al., 2022, Bordes et al., 2022]. This design sensitivity poses a real challenge as SSL is computationally demanding therefore preventing cross-validation to be employed at every change in the DN's architecture and/or dataset. Even more limiting, SSL's cross-validation relies on assessing the quality of the produced DN through the dataset's labels and test accuracy e.g. from a (supervised) linear probe. This supervised quality assessment is required simply because the actual values of current SSL losses fail to convey any qualitative information about the representation being learned [Ghosh et al., 2022, Garrido et al., 2022].

Instead of further refining of existing SSL methods, and thus inheriting most of their limitations, we propose a stripped down unsupervised learning solution, free of all those challenges —coined DIET. Unsupervised learning on a DIET consists in exploiting the most stable and understood setting: supervised learning but removing the need for labels by instead using the Datum IndEx as its Target label. Three striking observations will emerge. First, this simple DIET learns high-quality representations that are surprisingly on-par with much more sophisticated state-of-the-art methods. Second, the DIET performs equally well regardless of the dataset and architecture being considered. This is in contrast with most existing methods that are often architecture specific. Lastly, and perhaps of most importance, the DIET can be employed with low-resources e.g. most of our experiments employ a single GPU, and DIET's training loss is informative of downstream performances. Again, since DIET is a supervised learning method with the datum index as target, its training is as stable as the current state-of-the-art supervised methods and any progress made within the supervised learning realm is directly transferable to the proposed DIET. We hope that our method provides a novel go-to solution for many practitioners interested in learning high-quality representations, in a stable and minimalist way.

We summarize our contributions below:

* We introduce a DIET for unsupervised learning in <ref> (summarized in <ref>), a competitive and minimalist strategy bloating a few key benefits…
* Stable and Out-of-the-box: we validate the DIET on 16 official DN architectures including ConvNexts or ViTs, and on 10 datasets; the same setup (<ref>) is successful across all cases (<ref>)

* No hyper-parameter and single-GPU: moving away from decoders/positive pairs/projectors/…allows us to propose a DIET introducing no hyper-parameter and requiring only a few lines of code (<ref>) and we perform sensitivity analysis on batch size, data-augmentation, training time in <ref>

* Informative training loss: the DIET's training loss strongly correlates with the test accuracy across architectures and datasets (<ref>) enabling informed cross-validation to unsupervised learning

Code provided in <ref>.

[inner sep=0pt] (a) at (0,0) ;
[inner sep=0pt,above=.1cm of a]  $\texttt{datum}_{\texttt{1}}$;

[inner sep=0pt,right=.2cm of a] (b) ;
[inner sep=0pt,above=.1cm of b]  $\texttt{datum}_{\texttt{2}}$;
[inner sep=0pt,above right=.4cm and -2cm of b]  Training dataset;

[inner sep=0pt,right=.2cm of b] (c) 0.7cm…;

[inner sep=0pt,right=.2cm of c] (d) ;
[inner sep=0pt,above=.1cm of d]  $\texttt{datum}_{\texttt{N}}$;

[black,thick,rounded corners] ($(a.north west)+(-0.1,0.8)$)  rectangle ($(d.south east)+(0.1,-0.1)$);

[inner sep=0pt,below right=-.8cm and 0.5cm of d] (node2) 2cm$\mathcal{T}(\texttt{datum}_{\texttt{n}})$
(with DA $\mathcal{T}$);
[inner sep=0pt,above=2.2cm of node2] sample a datum index ();

every pin edge=[<-,shorten <=1pt]
neuron=[circle,fill=black!25,minimum size=17pt,inner sep=0pt]
input neuron=[neuron, fill=black!60];
output neuron=[neuron, fill=red];
hidden neuron=[neuron, fill=blue];

[input neuron,right=3.2cm of d] (I-2);
[input neuron,above=0.2cm of I-2] (I-1);
[input neuron,below=0.2cm of I-2] (I-3);

[hidden neuron,right=0.4cm of I-2] (H-3) ;
[hidden neuron,above=0.3cm of H-3] (H-4) ;
[hidden neuron,above=0.3cm of H-4] (H-5) ;
[hidden neuron,below=0.3cm of H-3] (H-2) ;
[hidden neuron,below=0.3cm of H-2] (H-1) ;

[hidden neuron,below right=0.1cm and 0.6cm of H-4] (h-2) ;
[hidden neuron,below=0.3cm of h-2] (h-1) ;

[output neuron, above right=2.5cm and 1.cm of h-1] (O1) ;

in 1,...,3
in 1,...,5
[blue!70] (I-) edge (H-);
in 1,...,5
[blue!70] (H-) edge (h-1);
[blue!70] (H-) edge (h-2);

in 1,...,2
[red!70] (h-) edge (O1);
/in 1/2,2/3,3/4,4/5,5/6,6/7
[output neuron, below=0.2cm of O] (O) ;
in 1,...,2
[red!70] (h-) edge (O);

[pen colour=red,thick,decorate,decoration=calligraphic brace,amplitude=2mm, raise=1.5mm] ($(O1.north east)+(-1.4,0.)$) – (O1.north east);
[red,above right=0.3cm and -2.1cm of O1] 2.5cm-output linear
layer (logits);

[pen colour=blue,thick,decorate,decoration=calligraphic brace,amplitude=2mm, raise=1.5mm,mirror] ($(H-1.south west)-(0.7,0)$) – ($(H-1.south west)+(1.6,0)$);
[blue,below right=0.3cm and -1.1cm of H-1] Deep Network;

[inner sep=0pt,right=1cm of O4] (TO)$\vy_n$;
[->,red!70,line width=0.8mm] ($(O4.east)+(0.2,0)$) edge ($(TO.west)+(-0.1,0)$);
[inner sep=0pt,below=.5cm of TO] (TO2) $(\texttt{n},$$\vy_\texttt{n}$$)$;
[->,black!70,very thick] (TO) edge (TO2);
[->,black!70,line width=0.8mm] ($(d.east)+(0.3,0.)$) edge ($(I-2)+(-0.5,0)$);
[->,black!70,very thick] (TO) edge (TO2);
[black!70,line width=0.8mm,rounded corners=0.3cm] ($(d.east)+(0.8,1.9)$) – node[midway]  ($(d.east)+(0.8,0.0)$) – node[midway] ($(d.east)+(1.4,0.0)$);

[above right=2.5cm and -2.1cm of a] 9.2cmDatum IndEx as Target (DIET);

[above left=1.1cm and -9cm of a] (top1)

* no siamese/teacher-student/projector DN

* no representation collapse

* informative training loss

* out-of-the-box across architectures/datasets

DIET uses the datum index () as the class-target –effectively turning unsupervised learning into a supervised learning problem. In our case, we employ the cross-entropy loss (), no extra care needed to handle different dataset or architectures. As opposed to current SOTA, we do not rely on a projector nor positive views i.e no change need to be done to any existing supervised pipeline to obtain the DIET. As highlighted in <ref>, DIET's training loss is even informative of downstream test performances, and as ablated in <ref> there is no degradation of performance with longer training, even for very small datasets (<ref>).

§ WHY UNSUPERVISED LEARNING NEEDS A DIET

The current state of unsupervised learning consists in complicated methods combining numerous moving pieces that need re-tweaking for each DN architecture and dataset. As a result reproducibility, scalability, and explainability are hindered.

Spectral embedding is untractable. Spectral embedding takes many forms but can be summarized into estimating geodesic distances [Meng et al., 2008, Thomas Fletcher, 2013] between all or some pairs of training samples to then learn a non-parametric [Roweis & Saul, 2000, Belkin & Niyogi, 2001, Balasubramanian & Schwartz, 2002, Brand & Huang, 2003], or parametric [Bengio et al., 2003, Pfau et al., 2018] mapping that produces embeddings whose pairwise $\ell_2$ distance matches the estimated geodesic ones. As such, spectral embedding heavily relies on the estimation of the geodesic distances which is a challenging problems [Lantuéjoul & Beucher, 1981, Lantuéjoul & Maisonneuve, 1984, Revaud et al., 2015], especially for images and videos [Donoho & Grimes, 2005, Wakin et al., 2005]. This limitation fueled the development of alternative methods e.g. Self-Supervised Learning (SSL), that often employ similar losses than spectral embedding [HaoChen et al., 2021, Balestriero & LeCun, 2022, Cabannes et al., 2022] but manage to move away from geodesic distance estimation through the explicit generation of positive pairs i.e. samples with known geodesic distances.

Self-Supervised Learning is unintelligible. Despite flamboyant performance reporting and well motivated first principles, SSL –as of today– falls back to combining numerous hacks driven by supervised performances. In fact, SSL has evolved to a point where novel methods are architecture specific. A few challenges that limit SSL to be widely adopted are (i) loss values which are uninformative of the DN's quality [Reed et al., 2021, Garrido et al., 2022], partly explained by the fact that SSL composes the DN of interest $f_{\theta}$ with a projector DN $ g_{\gamma}$ appended to it during training and thrown away afterward, (ii) too many per-loss and per-projector hyper-parameters whose impact on the DN's performances are hard to control or predict [Grill et al., 2020, Tian et al., 2021, He & Ozay, 2022], and which are even widely inconsistent across datasets and architectures Zhai et al., 2019, Cosentino et al., 2022, (iii) lack of theoretical guarantees as all existing studies have derived optimality conditions at the projector's output [Wang & Isola, 2020, Tian et al., 2020, Jing et al., 2021, Huang et al., 2021, HaoChen et al., 2021, Dubois et al., 2022, Zhang & Stratos, 2021, Wang & Liu, 2021] which is not the output of interest since the projector is thrown away after SSL training, and it is known that the DN's output and the projector's output greatly differ, see e.g. Tab. 3, Tab. 1, Tab. 1 and Fig. 1 of [Chen et al., 2020, Chen et al., 2020, Cosentino et al., 2022, Bordes et al., 2022] respectively.
From a more practical standpoint, SSL requires to generate positive pairs making it much more costly and resource hungry than standard supervised learning.

Reconstruction-based learning is unstable.
Reconstruction without careful tuning of the loss has been known to be sub-optimal for long [Bishop, 1994, Graves, 2013] and new studies keep reminding us of that [LeCun, 2022]. The argument is simple, suppose one aims to minimize a reconstruction metric $R$ for some input $\vx$

\begin{align}
\end{align}

where $e_{\eta}$ and $d_{\gamma}$ are parametrized learnable encoder and decoder networks respectively; $e_{\eta}(\vx)$ is the representation of interest to be used after training. In practice, as soon as some noise $\epsilon$ is present in the data, i.e. we observe $\vx + \epsilon$ and not $\vx$, that noise $\epsilon$ must be encoded by $e_{\eta}$ to minimize the loss from <ref> unless one carefully designs $R$ so that
$R(\vx+\epsilon,\vx) = 0$. However, designing such a noise invariant $R$ has been attempted for decades [Park, 1995, Simoncelli, 1996, Fienup, 1997, Grimes & Rao, 2005, Wang & Simoncelli, 2005] and remains a challenging open problem. Hence, many solutions rely on learning $R$ e.g. in VAE-GANs [Larsen et al., 2016] bringing even further instabilities and training challenges. Other alternatives carefully tweak $R$ per dataset and architectures e.g. to only compute the reconstruction loss on parts of the data as with BERT Devlin et al., 2018 or MAEs He et al., 2022. Lastly, the quality of the encoder's representation depends on its architecture but also on the decoder's one [Yang et al., 2017, Xu et al., 2021] making cross-validation more costly and unstable [Antun et al., 2020].

In numerous scenarios one finds themselves in a position where none of the existing state-of-the-art's limitations can be overcome –motivating the development of our DIET– an overly simple yet highly effective unsupervised learning strategy that inherits none of the aforementioned challenges.

§ UNSUPERVISED LEARNING ON A DIET

We first present in <ref> the DIET enabling competitive yet simple unsupervised learning; in particular, <ref> will demonstrate how that DIET competes and sometimes outperforms SSL on a variety of small to medium scale datasets while working out-of-the-box across architectures. <ref> will then demonstrate how to scale DIET to large datasets such as Imagenet and INaturalist.

§.§ The DIET: Datum IndEx as Target

The goal of this section is to introduce the proposed DIET, focusing on its simplicity and ease of implementation. Empirical validation of the method is deferred to the subsequent <ref> as summarized as the end of this section.

Algorithm: As well indicated by its name, Datum IndEx as Target (DIET) proposes to perform unsupervised learning by employing the datum index as its target class. That is, given a dataset of $N$ samples $\{\vx_1,\dots,\vx_N\}$, define the class of sample $\vx_n$ with $n \in \{1,\dots,N\}$ to be $n$ itself, leading to the DIET's loss $\mathcal{L}_{\rm\small DIET}$ for a datum to be

\begin{align}
\mathcal{L}_{\rm\small DIET}(\vx_n)={\rm ClassificationLoss}({\color{red}\mW}{\color{blue}f_{\vtheta}}(\vx_n),n),\label{eq:DIET}
\end{align}

given a sample $\vx_n \in \mathbb{R}^{D}$, a DN ${\color{blue}f_{\vtheta}:\mathbb{R}^{D} \mapsto \mathbb{R}^{K}}$, DIET's linear classifier ${\color{red}\mW \in \mathbb{R}^{K \times N}}$, and one's preferred classification loss. Unless one expects to sample some data more than others, there is no reason to add a bias vector to DIET's linear classifier. As such, DIET performs unsupervised learning through a supervised scheme meaning that any progress made within the supervised learning realm can be ported as-is to DIET. Throughout our study, we will be employing the cross-entropy loss, denoted as . We summarize DIET in <ref> and propose its pseudo-code in <ref> as well as the code to obtain a data loader providing the user with the indices ($n$) in <ref>.

# take any preferred DN e.g. resnet50
# see
f = torchvision.models.resnet50() #

# f comes with a classifier, let's remove it
K = f.fc.in_features # =
f.fc = nn.Identity() # remove it so f=

# define DIET's linear classifier and X-Ent
W = nn.Linear(K, N, bias=False) #
diet_loss = nn.CrossEntropyLoss(label_smoothing=0.8)

# start DIET training
for x, n in train_loader: # see
preds = f(t(x)) # t applies the DA
loss = diet_loss(W(preds), n) #
# proceed with backprop/optimizer/scheduler

DIET's algorithm, minimal code refactoring is required to employ the DIET given any already built deep learning pipeline, to obtain a dataset that provides the indices ($n$), see <ref>, ( stands for , Pytorch used for illustration).

from torch.utils.data import Dataset, DataLoader

class DatasetWithIndices(Dataset):
def __init__(self, dataset):
self.dataset = dataset
def __getitem__(self, n):
# if true class label is given, disregard it
x, _ = self.dataset[n]
return x, n
def __len__(self):
return len(self.dataset)

# example of train_loader with CIFAR100
CIFAR100 = torchvision.datasets.CIFAR100(root)
CIFAR100_w_ind = DatasetWithIndices(CIFAR100)
train_loader = DataLoader(C100_w_ind)

Custom loader ( in <ref>) to obtain the indices ($n$) in addition to the inputs $\vx_n$ and (optionally) the labels $y_n$ (Pytorch used for illustration).

Benefits: We ought to highlight three key benefits of DIET's <ref>. First, the amount of code refactoring is minimal (recall <ref>): there is no change to be done into the data loading pipelines (recall <ref>) as opposed to SSL which requires positive pairs, no need to specify teacher-student architectures, and no need to design a projector/predictor DN. Second, DIET's implementation is not architecture specific as we will validate on Resne(x)ts, ConvNe(x)ts, Vision Transformers and their variants. Third, DIET does not introduce any additional hyper-parameters in addition to the ones already present in one's favorite ClassificationLoss used in <ref>, all while providing a training loss that is informative of test time performances, as we depict in <ref>.

Relation to previous methods: Despite DIET's simplicity, we could not find an existing method that considered it perhaps due to the common –but erroneous– belief that dealing with hundreds of thousands of classes (N in <ref>, the training set size) would not produce successful training. As such, the closest method to ours is Exemplar CNN [Alexey et al., 2015] which extracts a few patches from a given image dataset, and treat each of them as their own class; this way the number of classes is the number of extracted patches, which is made independent from N. Performances are however far below SSL methods. A more recent method, Instance Discrimination [Wu et al., 2018] extends Exemplar CNN by introducing inter-sample discrimination. However, they do so using a non-parametric softmax i.e. by defining a learnable bank of centroids to cluster training samples; for successful training those centroids must be regularized to prevent representation collapse. As we will compare in <ref>, DIET outperforms Instance Discrimination while being simpler. Lastly, methods such as Noise as Targets [Bojanowski & Joulin, 2017] and DeepCluster [Caron et al., 2018] are quite far from DIET as (i) they perform clustering and use the datum's cluster as its class i.e. greatly reducing the dependency on N, and (ii) they perform such clustering in the output space of the model  $f_{\vtheta}$ being learned which brings multiple collapsed solutions requiring those methods to employ complicated mechanisms to ensure training to learn non-trivial representations.

Empirical validation roadmap: To support the different claims we have made above, we propose to scatter our empirical validation in a few subsequent sections. First, we will explore small and medium scale datasets in <ref> that include the eponymous CIFAR100 but also other datasets such as Food101 which have been challenging for SSL. In fact, whenever the number of training samples is small, most SSL methods favor Imagenet pre-training and few-shot transfer learning. In <ref> we will also consider TinyImagenet and Imagenet-100. After having validated the ability of DIET to match and sometimes outperform SSL methods, we will spend <ref> to probe the few hyper-parameters that govern DIET, in our case the label smoothing of the  loss, and the training time. We will see that without label smoothing, DIET is often as slow as SSL methods to converge, and sometimes slower –but that high values of label smoothing greatly speed up convergence. Lastly, we dedicate <ref> to scaling up DIET to large dataset such as Imagenet and INaturalist. In fact, recall from <ref> that DIET's N-output classifier becomes a memory bottleneck when $N$ is large in which case a slightly different treatment is required to employ DIET. We will see that even the most naive solutions e.g. subsampling of a large dataset enables to apply DIET as-is all while producing highly competitive performances.

DIET's experimental setup:

* Official Torchvision architectures  (no changes in init./arch.), only swapping the classification layer with DIET's one  (right of <ref>), no projector DN

* Same DA pipeline ($\mathcal{T}$ in <ref>) across datasets/architectures with batch size of 256 to fit on 1GPU

* AdamW optimizer with linear warmup (10 epochs) and cosine annealing learning rate schedule,  loss (right of <ref>) with label smoothing of $0.8$

* Learning rate/weight-decay of $0.001/0.05$ for non transformer architectures and $0.0002/0.01$ for transformers

In underlined are the design choices directly ported from standard supervised learning (not cross-validated for DIET), in italic are the design choices cross-validated for DIET but held constant across this study unless specified otherwise. Batch-size sensitivity analysis is reported in <ref> showing that performances do not vary when taking values from $32$ to $4096$. 's label smoothing parameter plays a role into DIET's convergence speed, and is cross-validated in <ref>; we also report DA ablation in <ref>.

Throughout our subsequent empirical validation, we will religiously follow the experimental setup described in <ref>, unless stated otherwise. Our goal in adopting the same setup across experiments is to highlight the stability of DIET to dataset and architectural changes; careful tuning of those design choices should naturally lead to greater performance if desired.

§.§ The DIET Competes with the State-Of-The-Art

We start the empirical validation of DIET on the eponymous CIFAR100 dataset; following that, we will consider other common medium scale datasets e.g. TinyImagenet, and in particular we will consider datasets such as Food101, Flowers102 for which current SSL does not provide working solutions and for which the common strategy consists in transfer learning. We will see in those cases that applying DIET as-is on each dataset is able to produce high-quality representations for a large set of DN architectures.

CIFAR100/linear-probe/single-GPU: with the settings of <ref> and optionally longer training (LT) or different label-smoothing (ls) specified, notice the consistent progression of the performance through architectures which is not easily achieved in SSL. In particular, we recall that the alternative methods all employ (except MoCo) a projector network (recall <ref>).
Benchmarks taken from
$\dagger:$Ho & Nvasconcelos, 2020;
$\ddagger:$Peng et al., 2022;
$\ast$:Zhang et al., 2022;
$\bullet$:Pham et al., 2022;
$\diamond$:da Costa et al., 2022;
$\star$:Yeh et al., 2022;
$\triangleleft$:Ren et al., 2022;
$\triangleright$:Yang et al., 2022;
$\Box$:Huang et al., 2022.

MoCoV2    53.28$^\ast$

SimSiam    53.66$^\bullet$

SimCLR   53.79$^\dagger$

SimMoCo    54.11$^\ast$

ReSSL   54.66$^\bullet$

SimCLR+adv     55.51$^\dagger$

MoCo    56.10$^\ddagger$

SimCLR   56.30$^\star$

MoCo+CC    57.65$^\ddagger$

SimCLR   57.81$^\triangleright$

DINO    58.12$^\bullet$

SimCO    58.35$^\ast$

SimCLR+DCL     58.50$^\dagger$

SimCLR   60.30$^\ddagger$

SimCLR   60.45$^\bullet$

W-MSE    61.33$^\diamond$

SimCLR+CC   61.91$^\ddagger$

BYOL    62.01$^\bullet$

MoCoV2    62.34$^\bullet$

red!15DIET    red!1562.93

BYOL    63.75$^\ddagger$

red!15DIET (LT)    red!1563.77

BYOL+CC    64.62$^\ddagger$

SimSiam    64.79$^\ddagger$

SwAV    64.88$^\diamond$

SimCLR   65.78$^\diamond$

SimSiam+CC    65.82$^\ddagger$

SimCLR    52.04$^\dagger$

MoCoV2    53.44$^\ast$

SimMoCo    54.64$^\ast$

SimCLR+adv    57.71$^\dagger$

SimCO    58.48$^\ast$

SimCLR    61.10$^\star$

SimCLR+DCL    62.20$^\star$

red!15DIET   red!1568.96

MoCoV3   69.00$^\triangleleft$

red!15DIET (ls:0.95)   red!1569.56

red!15DIET (LT)   red!1569.91

SimCLR    52.28$^\dagger$

SimCLR+adv    59.02$^\dagger$

MoCoV3    68.50$^\triangleleft$

red!15DIET   red!1570.29

red!15DIET (ls:0.95)   red!1571.09

red!15DIET (LT)   red!1571.39

SplitBrain    39.00$^\Box$

InstDisc    39.40$^\Box$

DeepCluster    41.90$^\Box$

AND    47.90$^\Box$

red!15DIET   red!1548.25

SeLa   57.40$^\Box$

Depiction of the DIET's training loss (y-axis) against the online test linear probe accuracy (x-axis) for all the models and hyper-parameters corresponding to the experiments of <ref> for CIFAR100 (left column), and <ref> for IN100 and TinyImagenet (middle and left columns). We colorize (yellow to blue) the points based on the strength of the label smoothing parameter (recall that it plays a role in DIET's convergence speed from <ref>). We clearly identify that for a given label smoothing parameter, there exists a strong relationship between DIET's training loss and the test accuracy showing that model selectin can be performed this way. Therefore,  even without labels, DIET's loss can be used as a quantitative quality assessment measure of one's model. The shift observed for different values of label smoothing can be accounted for to re-calibrate all the experiment if needed using the known relationship between that increasing the label smoothing parameter decreases the X-Ent loss, everything else equal.

Depiction of DIET's performances ( red) against the supervised ( blue) using a controlled training set size (x-axis), subsampled from the original training set and identical between methods; evaluation is performed over the original full evaluation set and conducted on various dataset (rows) and architectures (columns). We see that for small training set size, DIET is able to match and sometimes even outperform the supervised benchmark. See <ref> for additional datasets, in the Appendix.

TinyImagenet+IN100/linear-probe/single-GPU: with the settings of <ref>, as per <ref>, notice the consistent progression of the performance through architectures. We observe that DIET comes on par (higher-end for TinyImagenet and lower-end for IN100) with SSL methods.
Benchmarks are taken from $1$:[Dubois et al., 2022], $2:$[Ozsoy et al., 2022]

SimSiam      44.54 $^\ddagger$

red!15DIET         red!1545.07

SimCLR       46.21$^\ddagger$

BYOL         47.23$^\ddagger$

MoCo         47.98 $^\ddagger$

SimClR       48.70 $^1$

DINO         49.20 $^1$

SimCLr    48.12 $^2$

SimSiam    46.76 $^2$

Spectral    49.86 $^2$

red!15DIET    red!1551.66

CorInfoMax    54.86 $^2$

2cred!15DIET (other archs.)

blue!45 resnet34     red!15 47.04

blue!45 resnet101     red!15 51.86

blue!45 wide_resnet50_2     red!15 50.03

blue!45 resnext50_32x4d     red!15 52.45

blue!45 densenet121     red!15 49.38

blue!45 convnext_tiny     red!15 50.88

blue!45 convnext_small     red!15 50.05

blue!45 MLPMixer     red!15 39.32

blue!45 swin_t     red!15 50.80

blue!45 vit_b_16     red!15 48.38

2cImagenet-100 (IN100)

SimMoCo    58.20$^\ast$

MocoV2     60.52$^\ast$

SimCo      61.28 $^\ast$

red!15DIET         red!1564.31

W-MSE2     69.06 $^2$

ReSSL      74.02$^\bullet$

DINO       74.16$^\bullet$

MoCoV2     76.48$^\bullet$

BYOL       76.60$^\bullet$

SimCLR     77.04$^2$

SimCLR     78.72$^2$

MocoV2     79.28$^2$

VICReg     79.40$^2$

Barlow     80.38$^2$

red!15DIET         red!1573.50

MoCo+Hyper.        75.60 $^\star$

MoCo+DCL           76.80 $^\star$

MoCoV2 + Hyper.    77.70 $^\star$

BYOL               78.76 $^2$

MoCoV2 + DCL       80.50 $^\star$

SimCLR             80.70 $^\star$

SimSiam            81.60$^2$

SimCLR + DCL       83.10 $^\star$

2cred!15DIET (other archs.)

blue!45 wide_resnet50_2     red!15 71.92

blue!45 resnext50_32x4d     red!15 73.07

blue!45 densenet121     red!15 67.46

blue!45 convnext_tiny     red!15 69.77

blue!45 convnext_small     red!15 71.06

blue!45 MLPMixer     red!15 56.46

blue!45 swin_t     red!15 67.02

blue!45 vit_b_16     red!15 62.63

CIFAR100: Let's first consider CIFAR-100 [Krizhevsky et al., 2009] with a few variations of Resnet [He et al., 2016] and AlexNet [Krizhevsky, 2014]. To accomodate with the $32 \times 32$ resolution, we follow the standard procedure to slightly modify the ResNet architecture: the first convolution layer sees its kernel size go from 7$\times$7 to 3 $\times$ 3 and its stride reduce from 2 to 1; the max pooling layer following it is removed (details in <ref>). On Alexnet, a few non-SSL baselines are available, and we thus compare with SplitBrain [Zhang et al., 2017], DeepCluster [Caron et al., 2018], InstDisc [Wu et al., 2018] (closest to ours, see <ref>), AND [Huang et al., 2019], SeLa [Asano et al., 2019], ReSSL [Zheng et al., 2021]. The models are trained and linear evaluation is employed to judge the quality of the learned model to solve the original classification task; results are reported in <ref>. We observe that DIET is able to match and often slightly exceed current SSL methods. In particular, even though CIFAR100 is a relatively small dataset, increasing the DN capacity i.e. going from Resnet18 to Resnet101 does not exhibit any overfitting.

TinyImagenet and IN-100: We continue our empirical validation by now considering the more challenging Imagenet100 (IN100) [Tian et al., 2020] dataset which consists of 100 classes of the full Imagenet-1k dataset, the list of classes can be found online[<https://github.com/HobbitLong/CMC/blob/master/imagenet100.txt>] and the TinyImagenet [Le & Yang, 2015] dataset. Thanks to the higher resolution images present in those datasets, $224 \times 224$ and $64 \times 64$ respectively, we broaden the range of architecture we consider to include the Resnet variants of the previous section, SwinTransformer [Liu et al., 2021], VisionTransform [Dosovitskiy et al., 2020], Densenet [Huang et al., 2017], ConvNext [Liu et al., 2022], WideResnet [Zagoruyko & Komodakis, 2016], ResNext [Xie et al., 2017], and the MLPMixer [Tolstikhin et al., 2021]. We report DIET and benchmark results in <ref> where we now see that while on TinyImagenet, DIET is able to again strongly match SSL methods, DIET fall on the lower end of the spectrum on Imagenet100. Again we recall that as opposed to competing methods, DIET does not employ any projector DN.

Depiction of in-distribution and out-distribution (transfer) performances of DIET on a variety of datasets. They key observation is that even on small datasets such as Cars, Aircraft, DIET is able without any pretraining to come on-par with SSL performances obtained through Imagenet-1k pretraining. In addition, for medium scale dataset such as Food101, we see how DIET is able to significantly outperform SSL methods in both cases where we use DIET without pre-training, or when DIET is used to pretrain the architecture on another dataset. We also report performances for a ViT based architecture (SwinTiny) to demonstrate the ability of DIET to handle different models out-of-the-box, as per <ref>, we only adapted the learning rate and weight decay parameters. Every DIET run in this table is done on 1GPU and using a batch-size of 128. $\dagger$:[Yang et al., 2022], +:[Ericsson et al., 2021]

3*Arch.    3*Pretrained     3*Frozen     Method/Dataset     Aircraft       DTD           Pets          Flower        CUB-200     Food101    StanfordCars

N=   6667   1880   2940   1020   11788   68175   6509

C=   100   47   37   102   200   101   196
4*Resnet18   3*IN100$^\dagger$   Yes   SimCLR            24.19             54.35         46.46       75.00        16.73    -   -

SimCLR + CLAE     25.87             52.12         43.55       76.82        17.58    -   -

SimCLR + IDAA     26.02             54.97         46.76       77.99        18.15    -   -
None   No   DIET     red!1537.29     red!1550.62     red!1564.06     red!1572.01    red!15 33.03    red!15 62.00    red!15 42.55
17*Resnet50   13*IN-1k$^+$   13*Yes   InsDis      36.87     68.46     68.78     83.44     -     63.39     28.98

MoCo        35.55     68.83     69.84     82.10     -     62.10     27.99

PCL.        21.61     62.87     75.34     64.73     -     48.02     12.93

PIRL        37.08     68.99     71.36     83.60     -     64.65     28.72

PCLv2       37.03     70.59     82.79     85.34     -     64.88     30.51

SimCLR      44.90     74.20     83.33     90.87     -     67.47     43.73

MoCov2      41.79     73.88     83.30     90.07     -     68.95     39.31

SimCLRv2    46.38     76.38     84.72     92.90     -     73.08     50.37

SeLav2      37.29     74.15     83.22     90.22     -     71.08     36.86

InfoMin     38.58     74.73     86.24     87.18     -     69.53     41.01

BYOL        53.87     76.91     89.10     94.50     -     73.01     56.40

DeepClusterv2    54.49     78.62     89.36     94.72     -     77.94     58.60

Swav        54.04     77.02     87.60     94.62     -     76.62     54.06
None   No   DIET    red!1544.81     red!1551.75     red!1567.08     red!1573.32     red!1541.03     red!1571.58     red!1555.82
Tiny   Yes   DIET    red!1532.77     red!1549.34     red!1554.56     red!1567.89     red!1524.98     red!1580.43     red!1534.45
IN100   Yes   DIET    red!1536.20     red!1553.85     red!1559.77    red!15 77.22     red!1530.31     red!1580.92     red!1539.96
IN-1k   Yes   DIET    red!1535.58     red!1553.48     red!1560.81    red!15 77.16     red!1532.11     red!1581.07     red!1541.12
1*SwinTiny   None   No   DIET    red!1533.15     red!1551.88     red!1558.06     red!1570.78     red!1532.11     red!1568.86    red!15 47.12
1*ConvnextSmall   None   No   DIET    red!1543.13     red!1549.52     red!1561.72     red!1567.72     red!1531.44     red!1569.84    red!15 40.63

Small datasets with and without pre-training: We conclude the first part of our empirical validation by considering datasets that are commonly handled by SSL through transfer learning: Aircraft [Maji et al., 2013], DTD [Cimpoi et al., 2014], Pets [Parkhi et al., 2012], Flowers [Nilsback & Zisserman, 2008], CUB200 [Wah et al., 2011], Food101 [Bossard et al., 2014], Cars [Krause et al., 2013], where the numbers of training samples and classes are given in <ref>. The goal is to apply DIET directly on those dataset, without any pre-training, a feast that as far as we are aware, SSL was not able to perform successfully. We thus report those performances in <ref>. We see that DIET competes with SSL pre-trained models in most of the cases, and arrives not far behind for very small dataset which are DTD, Pets, Flower which contain 1880, 2940 and 1020 training samples respectively. Even then, we see that IN100 pre-training and Resnet18 is near DIET's performances, and that a stronger gap only appears using SSL pre-training on imagenet-1k with a Resnet50.

We additionally propose in <ref> the direct comparison of DIET with supervised learning on a variety of models and dataset but with controlled training size. We clearly observe that for small dataset i.e. for which we only use a small part of the original training set, DIET is matching supervised performances, which can be considered as ideal since in this setting the data dataset and task is used for evaluation (on the full evaluation dataset).

§.§ DIET's Dependency on Data-Augmentation, Training Time and Optimizer

We hope in this section to better inform practitioners on the role of Data-Augmentations (DA), training time, and label smoothing into DIET's performances; as well as sensitivity to mini-batch size, which is crucial for single-GPU training.

TinyImagenet/Resnet18/single-GPU batch-size sensitivity: top1 test accuracy after a fixed number of epochs (3k) and a single learning rate which is adjusted for each batch-size using $lr=0.001\frac{\texttt{bs}}{256}$; therefore performing per batch-size cross-validation will improve the reported numbers. We observe that DIET is able to perform well across the spectrum of batch sizes and will a slight drop in performance for very small batch size (8, 16) which is by batch-normalization as [Ioffe, 2017] identified $32$ to be the minimal working batch-size. Due to large batch sizes being faster (see <ref> for training times), this should be the preferred solution unless hardware constraint is present.

bs     8       16       32       64       128       256       512       1024       2048
acc.   32.9     37.9     42.7     43.4      43.3      43.7      43.7       42.6       41.2

Batch-size does not impact performances. One important question when it comes to training a method with low resources is the ability to employ (very) small mini-batch sizes. This is in fact one reason hindering at the deployment of SSL methods which require quite large mini-batch sizes to work (256 is a strict minimum in most cases). We thus propose in <ref> a small sensitivity analysis where we vary the mini-batch size from $8$ to $2048$ –without any tuning of the hyper-parameters– we use the standard learning rate scaling used in supervised learning: $lr=0.001\frac{\texttt{bs}}{256}$. We observe small fluctuations of performances (due to a sub-optimal learning rate) but no significant drop in performance, even for mini-batch size of $32$. When going to $16$ and $8$, we observe slightly lower performances which we believe to emerge from batch-normalization [Ioffe & Szegedy, 2015] which is known to behave erratically below a mini-batch size of $32$ [Ioffe, 2017].

Data-Augmentations sensitivity is similar to SSL. We observed in the previous <ref> that when using DA, the proposed DIET was able to almost match high engineered state-of-the-art methods, which should reassure the reader on the usefulness of the method. Yet, knowing which DA to employ is not trivial e.g. many data modalities have no know DA. One natural question is thus around the sensitivity of DIET's performance to the employed DA. To that end, we propose  three DA regimen, one only consistent of random crop and horizontal flips (strength:1), which could be consider minimal in computer vision, one which adds color jittering and random grayscale (strength:2), and one last which further adds Gaussian blur and random erasing [Zhong et al., 2020] (strength:3); the exact parameters for those transformations are given in <ref>. We observe on TinyImagenet and with a Resnet34 the following performances 32.93$\pm$ 0.6, 45.60$\pm$ 0.2, and  45.75$\pm$ 0.1 respectively over 5 independent runs, details and additional architectures provided in <ref> in the Appendix. We thus observe that while DIET greatly benefit from richer DA (strength:1 $\mapsto$ 2), it however does not require heavier transformation such as random erasing.

Convergence is slower than SSL but label smoothing helps. One important difference in training behavior between supervised learning and SSL is in the number of epochs required to see the quality of the representation plateau. Due to the supervised loss used in DIET, one might wonder how is the training behavior in our case. We observe that the convergence speed of DIET sometimes on-par but often slower than that of SSL in term of number of epochs required to reach a plateau –at least without using label smoothing. In fact, we surprisingly observe that by enabling large values of label smoothing, e.g. $0.8$, it was possible to obtain faster convergence. We provide a sensitivity analysis in <ref> in the Appendix. We believe that convergence speed could be improved by designing improved update mechanism for DIET's linear classifier. In fact, one should recall that within a single epoch, only one of each datum/class is observed, making the convergence speed of the classifier's $\mW$ matrix the main limitation; we hope to explore improved training strategies in the future as discussed in <ref>.

§.§ Pushing the DIET to Large Models and Datasets

Given DIET's formulation of considering each datum as its own class, it is natural to ask ourselves how scalable is such a method. Although we saw that on small and medium scale dataset, DIET's was able to come on-par with most current SSL methods, it is not clear at all it remains true for larger datasets. In this section we briefly describe what can be done to employ DIET on datasets such as Imagenet and INaturalist.

The first dataset we consider is INaturalist which contains slightly more than $500K$ training samples for its mini version (the one commonly employed, see e.g. [Zbontar et al., 2021]). It contains almost $10K$ actual classes and most SSL methods focus on transfer learning e.g. transferring with a Resnet50 from Imagenet-1k lead to SimCLR's 37.2$\%$, MoCoV2's 38.6, BYOL's 47.6 and BarlowTwins'46.5. However training on INaturalist directly produces lower performances reaching only 29.1 with MSN and a ViT. Using DIET is possible out-of-the-box with Resnet18 and ViT variants as their embedding is of dimension 512 and 762 respectively making $\mW$ fit in memory. We obtain 22.81 with a convnext small, and 21.6 with a ViT.

The second dataset we consider is the full Imagenet-1k dataset which contains more than 1 million training samples and 1000 actual classes. In this case, it is not possible to directly hold $\mW$ in-memory. We however tried a simple strategy which simply consists of sub-sampling the training set to a more reasonable size. This means that although we are putting aside many training images, we enable single GPU Imagenet training with DIET. With a training size of $400K$, we able to reach 44.05 with a convnext small, 43.78 with a SwinTiny, and 44.89 with a ViT/B/16. A standard SSL pipeline has performances ranging between $64\%$ and $72\%$. From those experiments, it is clear that DIET's main limitation comes from very large training set sizes. Although the above simple strategy offers a workable solution, it is clearly not sufficient to match with existing unsupervised learning method and thus should require further consideration. As highlighted in <ref> below, this is one key avenue for future work.

§ CONCLUSIONS AND FUTURE WORK

We presented a simple unsupervised learning method coined DIET, for Datum IndEx as Target, which simply casts the task of descriptive representation learning with Deep Networks (DNs) into a supervised problem of instance discrimination. Despite its simplicity, DIET is able to learn competitive representations that are often on-par with current state-of-the-art methods e.g. Self-Supervised Learning (SSL). We believe that DIET provides an out-of-the-box solution for many situations since (i) its training loss functions is informative of the downstream task test accuracy, (ii) it does not introduce any additional hyper-parameters, and training works seamlessly across architectures, and (iii) its implementation is requiring nearly no code refactoring from already built supervised pipelines, which contrast with e.g. SSL or autoencoders for which complicated data pipelines and additional DN specification. That being said, DIET suffers from one main limitations: the computational and memory complexity grows linearly with the dataset size, opening a few avenues.

To speed up training, a smarter initialization of the N-output classifier could be envisioned, along with a possibly different learning schedule for this large matrix and for the rest of the DN.

Unsupervised learning of a model $f_{\vtheta}$, governed by some parameter $\vtheta$, has always been and still is one of the most challenging and rewarding task in deep learning [Bengio, 2012]. In fact, supervised learning which learns to produce predictions from known input-output pairs can be considered solved in contrast to unsupervised learning which aims to produce descriptive and intelligible representations from inputs only [Hastie et al., 2009, Goodfellow et al., 2016].

Self-Supervised Learning (SSL) [Chen et al., 2020, Misra & Maaten, 2020] has recently demonstrated that one could learn without labels  highly non-trivial Deep Networks (DNs) whose representations are as descriptive as supervised ones. In particular, SSL differs from reconstruction-based methods such as (denoising, variational) Autoencoders [Vincent et al., 2008, Vincent et al., 2010, Kingma & Welling, 2013] and their cross-variants by removing the need for a decoder DN and an input-space reconstruction loss, both being difficult to design [Wang et al., 2004, Grimes & Rao, 2005, Larsen et al., 2016, Cosentino et al., 2022]. SSL's recent developments have also led to outperforming reconstruction-free methods e.g. Noise Contrastive Estimation and its variants [Hyvärinen & Dayan, 2005, Hinton, 2002, Song & Ermon, 2019, Rhodes et al., 2020].
Nonetheless, SSL which is the current state-of-the-art unsupervised learning solution, comes with many moving pieces e.g. (i) artificially constructed positive pairs, commonly obtained by applying two a priori known and tailored Data-Augmentations (DAs) to each datum, (ii) a carefully designed projector DN $g_{\vgamma}$ to perform SSL training with the composition $g_{\vgamma} \circ f_{\vtheta}$ and throwing away the projector $g_{\vgamma}$ afterwards [Chen et al., 2020], or (iii) advanced anti-collapse techniques involving moving average teacher models [Grill et al., 2020, Caron et al., 2021], representation normalization [Chen & He, 2021, Zbontar et al., 2021], or Entropy estimation [Chen et al., 2020, Li et al., 2021]. An incorrect pick of any of those moving pieces results in a drastic drop in performances [Cosentino et al., 2022, Bordes et al., 2022]. This design sensitivity poses a real challenge as SSL is computationally demanding therefore preventing cross-validation to be employed at every change in the DN's architecture and/or dataset. Even more limiting, SSL's cross-validation relies on assessing the quality of the produced DN through the dataset's labels and test accuracy e.g. from a (supervised) linear probe. This supervised quality assessment is required simply because the actual values of current SSL losses fail to convey any qualitative information about the representation being learned [Ghosh et al., 2022, Garrido et al., 2022].

Instead of further refining of existing SSL methods, and thus inheriting most of their limitations, we propose a stripped down unsupervised learning solution, free of all those challenges —coined DIET. Unsupervised learning on a DIET consists in exploiting the most stable and understood setting: supervised learning but removing the need for labels by instead using the Datum IndEx as its Target label. Three striking observations will emerge. First, this simple DIET learns high-quality representations that are surprisingly on-par with much more sophisticated state-of-the-art methods. Second, the DIET performs equally well regardless of the dataset and architecture being considered. This is in contrast with most existing methods that are often architecture specific. Lastly, and perhaps of most importance, the DIET can be employed with low-resources e.g. most of our experiments employ a single GPU, and DIET's training loss is informative of downstream performances. Again, since DIET is a supervised learning method with the datum index as target, its training is as stable as the current state-of-the-art supervised methods and any progress made within the supervised learning realm is directly transferable to the proposed DIET. We hope that our method provides a novel go-to solution for many practitioners interested in learning high-quality representations, in a stable and minimalist way.

We summarize our contributions below:

* We introduce a DIET for unsupervised learning in <ref> (summarized in <ref>), a competitive and minimalist strategy bloating a few key benefits…
* Stable and Out-of-the-box: we validate the DIET on 16 official DN architectures including ConvNexts or ViTs, and on 10 datasets; the same setup (<ref>) is successful across all cases (<ref>)

* No hyper-parameter and single-GPU: moving away from decoders/positive pairs/projectors/…allows us to propose a DIET introducing no hyper-parameter and requiring only a few lines of code (<ref>) and we perform sensitivity analysis on batch size, data-augmentation, training time in <ref>

* Informative training loss: the DIET's training loss strongly correlates with the test accuracy across architectures and datasets (<ref>) enabling informed cross-validation to unsupervised learning

Code provided in <ref>.

[inner sep=0pt] (a) at (0,0) ;
[inner sep=0pt,above=.1cm of a]  $\texttt{datum}_{\texttt{1}}$;

[inner sep=0pt,right=.2cm of a] (b) ;
[inner sep=0pt,above=.1cm of b]  $\texttt{datum}_{\texttt{2}}$;
[inner sep=0pt,above right=.4cm and -2cm of b]  Training dataset;

[inner sep=0pt,right=.2cm of b] (c) 0.7cm…;

[inner sep=0pt,right=.2cm of c] (d) ;
[inner sep=0pt,above=.1cm of d]  $\texttt{datum}_{\texttt{N}}$;

[black,thick,rounded corners] ($(a.north west)+(-0.1,0.8)$)  rectangle ($(d.south east)+(0.1,-0.1)$);

[inner sep=0pt,below right=-.8cm and 0.5cm of d] (node2) 2cm$\mathcal{T}(\texttt{datum}_{\texttt{n}})$
(with DA $\mathcal{T}$);
[inner sep=0pt,above=2.2cm of node2] sample a datum index ();

every pin edge=[<-,shorten <=1pt]
neuron=[circle,fill=black!25,minimum size=17pt,inner sep=0pt]
input neuron=[neuron, fill=black!60];
output neuron=[neuron, fill=red];
hidden neuron=[neuron, fill=blue];

[input neuron,right=3.2cm of d] (I-2);
[input neuron,above=0.2cm of I-2] (I-1);
[input neuron,below=0.2cm of I-2] (I-3);

[hidden neuron,right=0.4cm of I-2] (H-3) ;
[hidden neuron,above=0.3cm of H-3] (H-4) ;
[hidden neuron,above=0.3cm of H-4] (H-5) ;
[hidden neuron,below=0.3cm of H-3] (H-2) ;
[hidden neuron,below=0.3cm of H-2] (H-1) ;

[hidden neuron,below right=0.1cm and 0.6cm of H-4] (h-2) ;
[hidden neuron,below=0.3cm of h-2] (h-1) ;

[output neuron, above right=2.5cm and 1.cm of h-1] (O1) ;

in 1,...,3
in 1,...,5
[blue!70] (I-) edge (H-);
in 1,...,5
[blue!70] (H-) edge (h-1);
[blue!70] (H-) edge (h-2);

in 1,...,2
[red!70] (h-) edge (O1);
/in 1/2,2/3,3/4,4/5,5/6,6/7
[output neuron, below=0.2cm of O] (O) ;
in 1,...,2
[red!70] (h-) edge (O);

[pen colour=red,thick,decorate,decoration=calligraphic brace,amplitude=2mm, raise=1.5mm] ($(O1.north east)+(-1.4,0.)$) – (O1.north east);
[red,above right=0.3cm and -2.1cm of O1] 2.5cm-output linear
layer (logits);

[pen colour=blue,thick,decorate,decoration=calligraphic brace,amplitude=2mm, raise=1.5mm,mirror] ($(H-1.south west)-(0.7,0)$) – ($(H-1.south west)+(1.6,0)$);
[blue,below right=0.3cm and -1.1cm of H-1] Deep Network;

[inner sep=0pt,right=1cm of O4] (TO)$\vy_n$;
[->,red!70,line width=0.8mm] ($(O4.east)+(0.2,0)$) edge ($(TO.west)+(-0.1,0)$);
[inner sep=0pt,below=.5cm of TO] (TO2) $(\texttt{n},$$\vy_\texttt{n}$$)$;
[->,black!70,very thick] (TO) edge (TO2);
[->,black!70,line width=0.8mm] ($(d.east)+(0.3,0.)$) edge ($(I-2)+(-0.5,0)$);
[->,black!70,very thick] (TO) edge (TO2);
[black!70,line width=0.8mm,rounded corners=0.3cm] ($(d.east)+(0.8,1.9)$) – node[midway]  ($(d.east)+(0.8,0.0)$) – node[midway] ($(d.east)+(1.4,0.0)$);

[above right=2.5cm and -2.1cm of a] 9.2cmDatum IndEx as Target (DIET);

[above left=1.1cm and -9cm of a] (top1)

* no siamese/teacher-student/projector DN

* no representation collapse

* informative training loss

* out-of-the-box across architectures/datasets

DIET uses the datum index () as the class-target –effectively turning unsupervised learning into a supervised learning problem. In our case, we employ the cross-entropy loss (), no extra care needed to handle different dataset or architectures. As opposed to current SOTA, we do not rely on a projector nor positive views i.e no change need to be done to any existing supervised pipeline to obtain the DIET. As highlighted in <ref>, DIET's training loss is even informative of downstream test performances, and as ablated in <ref> there is no degradation of performance with longer training, even for very small datasets (<ref>).

§ WHY UNSUPERVISED LEARNING NEEDS A DIET

The current state of unsupervised learning consists in complicated methods combining numerous moving pieces that need re-tweaking for each DN architecture and dataset. As a result reproducibility, scalability, and explainability are hindered.

Spectral embedding is untractable. Spectral embedding takes many forms but can be summarized into estimating geodesic distances [Meng et al., 2008, Thomas Fletcher, 2013] between all or some pairs of training samples to then learn a non-parametric [Roweis & Saul, 2000, Belkin & Niyogi, 2001, Balasubramanian & Schwartz, 2002, Brand & Huang, 2003], or parametric [Bengio et al., 2003, Pfau et al., 2018] mapping that produces embeddings whose pairwise $\ell_2$ distance matches the estimated geodesic ones. As such, spectral embedding heavily relies on the estimation of the geodesic distances which is a challenging problems [Lantuéjoul & Beucher, 1981, Lantuéjoul & Maisonneuve, 1984, Revaud et al., 2015], especially for images and videos [Donoho & Grimes, 2005, Wakin et al., 2005]. This limitation fueled the development of alternative methods e.g. Self-Supervised Learning (SSL), that often employ similar losses than spectral embedding [HaoChen et al., 2021, Balestriero & LeCun, 2022, Cabannes et al., 2022] but manage to move away from geodesic distance estimation through the explicit generation of positive pairs i.e. samples with known geodesic distances.

Self-Supervised Learning is unintelligible. Despite flamboyant performance reporting and well motivated first principles, SSL –as of today– falls back to combining numerous hacks driven by supervised performances. In fact, SSL has evolved to a point where novel methods are architecture specific. A few challenges that limit SSL to be widely adopted are (i) loss values which are uninformative of the DN's quality [Reed et al., 2021, Garrido et al., 2022], partly explained by the fact that SSL composes the DN of interest $f_{\theta}$ with a projector DN $ g_{\gamma}$ appended to it during training and thrown away afterward, (ii) too many per-loss and per-projector hyper-parameters whose impact on the DN's performances are hard to control or predict [Grill et al., 2020, Tian et al., 2021, He & Ozay, 2022], and which are even widely inconsistent across datasets and architectures Zhai et al., 2019, Cosentino et al., 2022, (iii) lack of theoretical guarantees as all existing studies have derived optimality conditions at the projector's output [Wang & Isola, 2020, Tian et al., 2020, Jing et al., 2021, Huang et al., 2021, HaoChen et al., 2021, Dubois et al., 2022, Zhang & Stratos, 2021, Wang & Liu, 2021] which is not the output of interest since the projector is thrown away after SSL training, and it is known that the DN's output and the projector's output greatly differ, see e.g. Tab. 3, Tab. 1, Tab. 1 and Fig. 1 of [Chen et al., 2020, Chen et al., 2020, Cosentino et al., 2022, Bordes et al., 2022] respectively.
From a more practical standpoint, SSL requires to generate positive pairs making it much more costly and resource hungry than standard supervised learning.

Reconstruction-based learning is unstable.
Reconstruction without careful tuning of the loss has been known to be sub-optimal for long [Bishop, 1994, Graves, 2013] and new studies keep reminding us of that [LeCun, 2022]. The argument is simple, suppose one aims to minimize a reconstruction metric $R$ for some input $\vx$

\begin{align}
\end{align}

where $e_{\eta}$ and $d_{\gamma}$ are parametrized learnable encoder and decoder networks respectively; $e_{\eta}(\vx)$ is the representation of interest to be used after training. In practice, as soon as some noise $\epsilon$ is present in the data, i.e. we observe $\vx + \epsilon$ and not $\vx$, that noise $\epsilon$ must be encoded by $e_{\eta}$ to minimize the loss from <ref> unless one carefully designs $R$ so that
$R(\vx+\epsilon,\vx) = 0$. However, designing such a noise invariant $R$ has been attempted for decades [Park, 1995, Simoncelli, 1996, Fienup, 1997, Grimes & Rao, 2005, Wang & Simoncelli, 2005] and remains a challenging open problem. Hence, many solutions rely on learning $R$ e.g. in VAE-GANs [Larsen et al., 2016] bringing even further instabilities and training challenges. Other alternatives carefully tweak $R$ per dataset and architectures e.g. to only compute the reconstruction loss on parts of the data as with BERT Devlin et al., 2018 or MAEs He et al., 2022. Lastly, the quality of the encoder's representation depends on its architecture but also on the decoder's one [Yang et al., 2017, Xu et al., 2021] making cross-validation more costly and unstable [Antun et al., 2020].

In numerous scenarios one finds themselves in a position where none of the existing state-of-the-art's limitations can be overcome –motivating the development of our DIET– an overly simple yet highly effective unsupervised learning strategy that inherits none of the aforementioned challenges.

We presented a simple unsupervised learning method coined DIET, for Datum IndEx as Target, which simply casts the task of descriptive representation learning with Deep Networks (DNs) into a supervised problem of instance discrimination. Despite its simplicity, DIET is able to learn competitive representations that are often on-par with current state-of-the-art methods e.g. Self-Supervised Learning (SSL). We believe that DIET provides an out-of-the-box solution for many situations since (i) its training loss functions is informative of the downstream task test accuracy, (ii) it does not introduce any additional hyper-parameters, and training works seamlessly across architectures, and (iii) its implementation is requiring nearly no code refactoring from already built supervised pipelines, which contrast with e.g. SSL or autoencoders for which complicated data pipelines and additional DN specification. That being said, DIET suffers from one main limitations: the computational and memory complexity grows linearly with the dataset size, opening a few avenues.

To speed up training, a smarter initialization of the N-output classifier could be envisioned, along with a possibly different learning schedule for this large matrix and for the rest of the DN.

[Alexey et al., 2015]
Alexey, D., Fischer, P., Tobias, J., Springenberg, M. R., and Brox, T.
Discriminative unsupervised feature learning with exemplar
convolutional neural networks.
IEEE Trans. Pattern Analysis and Machine Intelligence, 99,

[Antun et al., 2020]
Antun, V., Renna, F., Poon, C., Adcock, B., and Hansen, A. C.
On instabilities of deep learning in image reconstruction and the
potential costs of ai.
Proceedings of the National Academy of Sciences, 1170
(48):0 30088–30095, 2020.

[Asano et al., 2019]
Asano, Y. M., Rupprecht, C., and Vedaldi, A.
Self-labelling via simultaneous clustering and representation
arXiv preprint arXiv:1911.05371, 2019.

[Balasubramanian & Schwartz, 2002]
Balasubramanian, M. and Schwartz, E. L.
The isomap algorithm and topological stability.
Science, 2950 (5552):0 7–7, 2002.

[Balestriero & LeCun, 2022]
Balestriero, R. and LeCun, Y.
Contrastive and non-contrastive self-supervised learning recover
global and local spectral embedding methods.
arXiv preprint arXiv:2205.11508, 2022.

[Belkin & Niyogi, 2001]
Belkin, M. and Niyogi, P.
Laplacian eigenmaps and spectral techniques for embedding and
Advances in neural information processing systems, 14, 2001.

[Bengio, 2012]
Bengio, Y.
Deep learning of representations for unsupervised and transfer
In Proceedings of ICML workshop on unsupervised and transfer
learning, pp.  17–36. JMLR Workshop and Conference Proceedings, 2012.

[Bengio et al., 2003]
Bengio, Y., Paiement, J.-f., Vincent, P., Delalleau, O., Roux, N., and Ouimet,
Out-of-sample extensions for lle, isomap, mds, eigenmaps, and
spectral clustering.
Advances in neural information processing systems, 16, 2003.

[Bishop, 1994]
Bishop, C. M.
Mixture density networks.

[Bojanowski & Joulin, 2017]
Bojanowski, P. and Joulin, A.
Unsupervised learning by predicting noise.
In International Conference on Machine Learning, pp.  517–526. PMLR, 2017.

[Bordes et al., 2022]
Bordes, F., Balestriero, R., Garrido, Q., Bardes, A., and Vincent, P.
Guillotine regularization: Improving deep networks generalization by
removing their head.
arXiv preprint arXiv:2206.13378, 2022.

[Bossard et al., 2014]
Bossard, L., Guillaumin, M., and Van Gool, L.
Food-101 – mining discriminative components with random forests.
In European Conference on Computer Vision, 2014.

[Brand & Huang, 2003]
Brand, M. and Huang, K.
A unifying theorem for spectral embedding and clustering.
In International Workshop on Artificial Intelligence and
Statistics, pp.  41–48. PMLR, 2003.

[Cabannes et al., 2022]
Cabannes, V., Bietti, A., and Balestriero, R.
On minimal variations for unsupervised representation learning.
arXiv preprint arXiv:2211.03782, 2022.

[Caron et al., 2018]
Caron, M., Bojanowski, P., Joulin, A., and Douze, M.
Deep clustering for unsupervised learning of visual features.
In Proceedings of the European conference on computer vision
(ECCV), pp.  132–149, 2018.

[Caron et al., 2021]
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P.,
and Joulin, A.
Emerging properties in self-supervised vision transformers.
In Proceedings of the IEEE/CVF International Conference on
Computer Vision, pp.  9650–9660, 2021.

[Chen et al., 2020]
Chen, T., Kornblith, S., Norouzi, M., and Hinton, G.
A simple framework for contrastive learning of visual
In International conference on machine learning, pp.  1597–1607. PMLR, 2020a.

[Chen & He, 2021]
Chen, X. and He, K.
Exploring simple siamese representation learning.
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp.  15750–15758, 2021.

[Chen et al., 2020]
Chen, X., Fan, H., Girshick, R., and He, K.
Improved baselines with momentum contrastive learning.
arXiv preprint arXiv:2003.04297, 2020b.

[Cimpoi et al., 2014]
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., , and Vedaldi, A.
Describing textures in the wild.
In Proceedings of the IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2014.

[Cosentino et al., 2022]
Cosentino, R., Balestriero, R., Bahroun, Y., Sengupta, A., Baraniuk, R., and
Aazhang, B.
Spatial transformer k-means.
arXiv preprint arXiv:2202.07829, 2022a.

[Cosentino et al., 2022]
Cosentino, R., Sengupta, A., Avestimehr, S., Soltanolkotabi, M., Ortega, A.,
Willke, T., and Tepper, M.
Toward a geometrical understanding of self-supervised contrastive
arXiv preprint arXiv:2205.06926, 2022b.

[Cosentino et al., 2022]
Cosentino, R., Shekkizhar, S., Soltanolkotabi, M., Avestimehr, S., and Ortega,
The geometry of self-supervised learning models and its impact on
transfer learning.
arXiv preprint arXiv:2209.08622, 2022c.

[da Costa et al., 2022]
da Costa, V. G. T., Fini, E., Nabi, M., Sebe, N., and Ricci, E.
solo-learn: A library of self-supervised methods for visual
representation learning.
J. Mach. Learn. Res., 23:0 56–1, 2022.

[Devlin et al., 2018]
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K.
Bert: Pre-training of deep bidirectional transformers for language
arXiv preprint arXiv:1810.04805, 2018.

[Donoho & Grimes, 2005]
Donoho, D. L. and Grimes, C.
Image manifolds which are isometric to euclidean space.
Journal of mathematical imaging and vision, 230
(1):0 5–24, 2005.

[Dosovitskiy et al., 2020]
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X.,
Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.
An image is worth 16x16 words: Transformers for image recognition at
arXiv preprint arXiv:2010.11929, 2020.

[Dubois et al., 2022]
Dubois, Y., Hashimoto, T., Ermon, S., and Liang, P.
Improving self-supervised learning by characterizing idealized
arXiv preprint arXiv:2209.06235, 2022.

[Ericsson et al., 2021]
Ericsson, L., Gouk, H., and Hospedales, T. M.
How well do self-supervised models transfer?
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp.  5414–5423, 2021.

[Fienup, 1997]
Fienup, J. R.
Invariant error metrics for image reconstruction.
Applied optics, 360 (32):0 8352–8357, 1997.

[Garrido et al., 2022]
Garrido, Q., Balestriero, R., Najman, L., and Lecun, Y.
Rankme: Assessing the downstream performance of pretrained
self-supervised representations by their rank.
arXiv preprint arXiv:2210.02885, 2022.

[Ghosh et al., 2022]
Ghosh, A., Mondal, A. K., Agrawal, K. K., and Richards, B.
Investigating power laws in deep representation learning.
arXiv preprint arXiv:2202.05808, 2022.

[Goodfellow et al., 2016]
Goodfellow, I., Bengio, Y., Courville, A., and Bengio, Y.
Deep learning, volume 1.
MIT Press, 2016.

[Graves, 2013]
Graves, A.
Generating sequences with recurrent neural networks.
arXiv preprint arXiv:1308.0850, 2013.

[Grill et al., 2020]
Grill, J.-B., Strub, F., Altché, F., Tallec, C., Richemond, P.,
Buchatskaya, E., Doersch, C., Avila Pires, B., Guo, Z., Gheshlaghi Azar, M.,
et al.
Bootstrap your own latent-a new approach to self-supervised learning.
Advances in neural information processing systems,
33:0 21271–21284, 2020.

[Grimes & Rao, 2005]
Grimes, D. B. and Rao, R. P.
Bilinear sparse coding for invariant vision.
Neural computation, 170 (1):0 47–73, 2005.

[HaoChen et al., 2021]
HaoChen, J. Z., Wei, C., Gaidon, A., and Ma, T.
Provable guarantees for self-supervised deep learning with spectral
contrastive loss.
Advances in Neural Information Processing Systems,
34:0 5000–5011, 2021.

[Hastie et al., 2009]
Hastie, T., Tibshirani, R., and Friedman, J.
Overview of supervised learning.
In The elements of statistical learning, pp.  9–41.
Springer, 2009.

[He & Ozay, 2022]
He, B. and Ozay, M.
Exploring the gap between collapsed & whitened features in
self-supervised learning.
In International Conference on Machine Learning, pp.  8613–8634. PMLR, 2022.

[He et al., 2016]
He, K., Zhang, X., Ren, S., and Sun, J.
Deep residual learning for image recognition.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pp.  770–778, 2016.

[He et al., 2022]
He, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R.
Masked autoencoders are scalable vision learners.
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp.  16000–16009, 2022.

[Hinton, 2002]
Hinton, G. E.
Training products of experts by minimizing contrastive divergence.
Neural computation, 140 (8):0 1771–1800,

[Ho & Nvasconcelos, 2020]
Ho, C.-H. and Nvasconcelos, N.
Contrastive learning with adversarial examples.
Advances in Neural Information Processing Systems,
33:0 17081–17093, 2020.

[Huang et al., 2017]
Huang, G., Liu, Z., Van Der Maaten, L., and Weinberger, K. Q.
Densely connected convolutional networks.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pp.  4700–4708, 2017.

[Huang et al., 2019]
Huang, J., Dong, Q., Gong, S., and Zhu, X.
Unsupervised deep learning by neighbourhood discovery.
In International Conference on Machine Learning, pp.  2849–2858. PMLR, 2019.

[Huang et al., 2022]
Huang, L., Zhang, C., and Zhang, H.
Self-adaptive training: Bridging supervised and self-supervised
IEEE Transactions on Pattern Analysis and Machine
Intelligence, 2022.

[Huang et al., 2021]
Huang, W., Yi, M., and Zhao, X.
Towards the generalization of contrastive self-supervised learning.
arXiv preprint arXiv:2111.00743, 2021.

[Hyvärinen & Dayan, 2005]
Hyvärinen, A. and Dayan, P.
Estimation of non-normalized statistical models by score matching.
Journal of Machine Learning Research, 60 (4), 2005.

[Ioffe, 2017]
Ioffe, S.
Batch renormalization: Towards reducing minibatch dependence in
batch-normalized models.
Advances in neural information processing systems, 30, 2017.

[Ioffe & Szegedy, 2015]
Ioffe, S. and Szegedy, C.
Batch normalization: Accelerating deep network training by reducing
internal covariate shift.
In International conference on machine learning, pp.  448–456. PMLR, 2015.

[Jing et al., 2021]
Jing, L., Vincent, P., LeCun, Y., and Tian, Y.
Understanding dimensional collapse in contrastive self-supervised
arXiv preprint arXiv:2110.09348, 2021.

[Kingma & Welling, 2013]
Kingma, D. P. and Welling, M.
Auto-encoding variational bayes.
arXiv preprint arXiv:1312.6114, 2013.

[Krause et al., 2013]
Krause, J., Stark, M., Deng, J., and Fei-Fei, L.
3d object representations for fine-grained categorization.
In 4th International IEEE Workshop on 3D Representation and
Recognition (3dRR-13), Sydney, Australia, 2013.

[Krizhevsky, 2014]
Krizhevsky, A.
One weird trick for parallelizing convolutional neural networks.
arXiv preprint arXiv:1404.5997, 2014.

[Krizhevsky et al., 2009]
Krizhevsky, A., Hinton, G., et al.
Learning multiple layers of features from tiny images.

[Lantuéjoul & Beucher, 1981]
Lantuéjoul, C. and Beucher, S.
On the use of the geodesic metric in image analysis.
Journal of Microscopy, 1210 (1):0 39–49,

[Lantuéjoul & Maisonneuve, 1984]
Lantuéjoul, C. and Maisonneuve, F.
Geodesic methods in quantitative image analysis.
Pattern recognition, 170 (2):0 177–187, 1984.

[Larsen et al., 2016]
Larsen, A. B. L., Sønderby, S. K., Larochelle, H., and Winther, O.
Autoencoding beyond pixels using a learned similarity metric.
In International conference on machine learning, pp.  1558–1566. PMLR, 2016.

[Le & Yang, 2015]
Le, Y. and Yang, X.
Tiny imagenet visual recognition challenge.
CS 231N, 70 (7):0 3, 2015.

[LeCun, 2022]
LeCun, Y.
A path towards autonomous machine intelligence.
preprint posted on openreview, 2022.

[Li et al., 2021]
Li, Y., Pogodin, R., Sutherland, D. J., and Gretton, A.
Self-supervised learning with kernel dependence maximization.
Advances in Neural Information Processing Systems,
34:0 15543–15556, 2021.

[Liu et al., 2021]
Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B.
Swin transformer: Hierarchical vision transformer using shifted
In Proceedings of the IEEE/CVF International Conference on
Computer Vision, pp.  10012–10022, 2021.

[Liu et al., 2022]
Liu, Z., Mao, H., Wu, C.-Y., Feichtenhofer, C., Darrell, T., and Xie, S.
A convnet for the 2020s.
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp.  11976–11986, 2022.

[Maji et al., 2013]
Maji, S., Kannala, J., Rahtu, E., Blaschko, M., and Vedaldi, A.
Fine-grained visual classification of aircraft.
Technical report, 2013.

[Meng et al., 2008]
Meng, D., Leung, Y., Xu, Z., Fung, T., and Zhang, Q.
Improving geodesic distance estimation based on locally linear
Pattern Recognition Letters, 290 (7):0
862–870, 2008.

[Misra & Maaten, 2020]
Misra, I. and Maaten, L. v. d.
Self-supervised learning of pretext-invariant representations.
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp.  6707–6717, 2020.

[Nilsback & Zisserman, 2008]
Nilsback, M.-E. and Zisserman, A.
Automated flower classification over a large number of classes.
In 2008 Sixth Indian Conference on Computer Vision, Graphics &
Image Processing, pp.  722–729. IEEE, 2008.

[Ozsoy et al., 2022]
Ozsoy, S., Hamdan, S., Arik, S. Ö., Yuret, D., and Erdogan, A. T.
Self-supervised learning with an information maximization criterion.
arXiv preprint arXiv:2209.07999, 2022.

[Park, 1995]
Park, F. C.
Distance metrics on the rigid-body motions with applications to
mechanism design.

[Parkhi et al., 2012]
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, C.
Cats and dogs.
In 2012 IEEE conference on computer vision and pattern
recognition, pp.  3498–3505. IEEE, 2012.

[Peng et al., 2022]
Peng, X., Wang, K., Zhu, Z., Wang, M., and You, Y.
Crafting better contrastive views for siamese representation
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp.  16031–16040, 2022.

[Pfau et al., 2018]
Pfau, D., Petersen, S., Agarwal, A., Barrett, D. G., and Stachenfeld, K. L.
Spectral inference networks: Unifying deep and spectral learning.
arXiv preprint arXiv:1806.02215, 2018.

[Pham et al., 2022]
Pham, T., Zhang, C., Niu, A., Zhang, K., and Yoo, C. D.
On the pros and cons of momentum encoder in self-supervised visual
representation learning.
arXiv preprint arXiv:2208.05744, 2022.

[Reed et al., 2021]
Reed, C. J., Metzger, S., Srinivas, A., Darrell, T., and Keutzer, K.
Selfaugment: Automatic augmentation policies for self-supervised
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp.  2674–2683, 2021.

[Ren et al., 2022]
Ren, S., Wang, H., Gao, Z., He, S., Yuille, A., Zhou, Y., and Xie, C.
A simple data mixing prior for improving self-supervised learning.
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp.  14595–14604, 2022.

[Revaud et al., 2015]
Revaud, J., Weinzaepfel, P., Harchaoui, Z., and Schmid, C.
Epicflow: Edge-preserving interpolation of correspondences for
optical flow.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pp.  1164–1172, 2015.

[Rhodes et al., 2020]
Rhodes, B., Xu, K., and Gutmann, M. U.
Telescoping density-ratio estimation.
Advances in neural information processing systems,
33:0 4905–4916, 2020.

[Roweis & Saul, 2000]
Roweis, S. T. and Saul, L. K.
Nonlinear dimensionality reduction by locally linear embedding.
science, 2900 (5500):0 2323–2326, 2000.

[Simoncelli, 1996]
Simoncelli, E.
A rotation invariant pattern signature.
In Proceedings of 3rd IEEE International Conference on Image
Processing, volume 3, pp.  185–188. IEEE, 1996.

[Song & Ermon, 2019]
Song, Y. and Ermon, S.
Generative modeling by estimating gradients of the data distribution.
Advances in Neural Information Processing Systems, 32, 2019.

[Thomas Fletcher, 2013]
Thomas Fletcher, P.
Geodesic regression and the theory of least squares on riemannian
International journal of computer vision, 1050
(2):0 171–185, 2013.

[Tian et al., 2020]
Tian, Y., Krishnan, D., and Isola, P.
Contrastive multiview coding.
In European conference on computer vision, pp.  776–794.
Springer, 2020a.

[Tian et al., 2020]
Tian, Y., Sun, C., Poole, B., Krishnan, D., Schmid, C., and Isola, P.
What makes for good views for contrastive learning?
Advances in Neural Information Processing Systems,
33:0 6827–6839, 2020b.

[Tian et al., 2021]
Tian, Y., Chen, X., and Ganguli, S.
Understanding self-supervised learning dynamics without contrastive
In International Conference on Machine Learning, pp.  10268–10278. PMLR, 2021.

[Tolstikhin et al., 2021]
Tolstikhin, I. O., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X.,
Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., et al.
Mlp-mixer: An all-mlp architecture for vision.
Advances in Neural Information Processing Systems,
34:0 24261–24272, 2021.

[Vincent et al., 2008]
Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P.-A.
Extracting and composing robust features with denoising autoencoders.
In Proceedings of the 25th international conference on Machine
learning, pp.  1096–1103, 2008.

[Vincent et al., 2010]
Vincent, P., Larochelle, H., Lajoie, I., Bengio, Y., Manzagol, P.-A., and
Bottou, L.
Stacked denoising autoencoders: Learning useful representations in a
deep network with a local denoising criterion.
Journal of machine learning research, 110 (12), 2010.

[Wah et al., 2011]
Wah, C., Branson, S., Welinder, P., Perona, P., and Belongie, S.
Cub200 dataset.
Technical Report CNS-TR-2011-001, California Institute of Technology,

[Wakin et al., 2005]
Wakin, M. B., Donoho, D. L., Choi, H., and Baraniuk, R. G.
High-resolution navigation on non-differentiable image manifolds.
In Proceedings.(ICASSP'05). IEEE International Conference on
Acoustics, Speech, and Signal Processing, 2005., volume 5, pp.  v–1073.
IEEE, 2005.

[Wang & Liu, 2021]
Wang, F. and Liu, H.
Understanding the behaviour of contrastive loss.
In Proceedings of the IEEE/CVF conference on computer vision
and pattern recognition, pp.  2495–2504, 2021.

[Wang & Isola, 2020]
Wang, T. and Isola, P.
Understanding contrastive representation learning through alignment
and uniformity on the hypersphere.
In International Conference on Machine Learning, pp.  9929–9939. PMLR, 2020.

[Wang & Simoncelli, 2005]
Wang, Z. and Simoncelli, E. P.
Translation insensitive image similarity in complex wavelet domain.
In Proceedings.(ICASSP'05). IEEE International Conference on
Acoustics, Speech, and Signal Processing, 2005., volume 2, pp.  ii–573.
IEEE, 2005.

[Wang et al., 2004]
Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P.
Image quality assessment: from error visibility to structural
IEEE transactions on image processing, 130
(4):0 600–612, 2004.

[Wu et al., 2018]
Wu, Z., Xiong, Y., Yu, S. X., and Lin, D.
Unsupervised feature learning via non-parametric instance
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pp.  3733–3742, 2018.

[Xie et al., 2017]
Xie, S., Girshick, R., Dollár, P., Tu, Z., and He, K.
Aggregated residual transformations for deep neural networks.
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pp.  1492–1500, 2017.

[Xu et al., 2021]
Xu, W., Jang-Jaccard, J., Singh, A., Wei, Y., and Sabrina, F.
Improving performance of autoencoder-based network anomaly detection
on nsl-kdd dataset.
IEEE Access, 9:0 140136–140146, 2021.

[Yang et al., 2022]
Yang, K., Zhou, T., Tian, X., and Tao, D.
Identity-disentangled adversarial augmentation for self-supervised
In International Conference on Machine Learning, pp.  25364–25381. PMLR, 2022.

[Yang et al., 2017]
Yang, Z., Hu, Z., Salakhutdinov, R., and Berg-Kirkpatrick, T.
Improved variational autoencoders for text modeling using dilated
In International conference on machine learning, pp.  3881–3890. PMLR, 2017.

[Yeh et al., 2022]
Yeh, C.-H., Hong, C.-Y., Hsu, Y.-C., Liu, T.-L., Chen, Y., and LeCun, Y.
Decoupled contrastive learning.
In European Conference on Computer Vision, pp.  668–684.
Springer, 2022.

[Zagoruyko & Komodakis, 2016]
Zagoruyko, S. and Komodakis, N.
Wide residual networks.
arXiv preprint arXiv:1605.07146, 2016.

[Zbontar et al., 2021]
Zbontar, J., Jing, L., Misra, I., LeCun, Y., and Deny, S.
Barlow twins: Self-supervised learning via redundancy reduction.
arXiv preprint arXiv:2103.03230, 2021.

[Zhai et al., 2019]
Zhai, X., Puigcerver, J., Kolesnikov, A., Ruyssen, P., Riquelme, C., Lucic, M.,
Djolonga, J., Pinto, A. S., Neumann, M., Dosovitskiy, A., et al.
The visual task adaptation benchmark.

[Zhang et al., 2022]
Zhang, C., Zhang, K., Pham, T. X., Niu, A., Qiao, Z., Yoo, C. D., and Kweon,
I. S.
Dual temperature helps contrastive learning without many negative
samples: Towards understanding and simplifying moco.
In Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition, pp.  14441–14450, 2022.

[Zhang et al., 2017]
Zhang, R., Isola, P., and Efros, A. A.
Split-brain autoencoders: Unsupervised learning by cross-channel
In Proceedings of the IEEE conference on computer vision and
pattern recognition, pp.  1058–1067, 2017.

[Zhang & Stratos, 2021]
Zhang, W. and Stratos, K.
Understanding hard negatives in noise contrastive estimation.
arXiv preprint arXiv:2104.06245, 2021.

[Zheng et al., 2021]
Zheng, M., You, S., Wang, F., Qian, C., Zhang, C., Wang, X., and Xu, C.
Ressl: Relational self-supervised learning with weak augmentation.
Advances in Neural Information Processing Systems,
34:0 2543–2555, 2021.

[Zhong et al., 2020]
Zhong, Z., Zheng, L., Kang, G., Li, S., and Yang, Y.
Random erasing data augmentation.
In Proceedings of the AAAI conference on artificial
intelligence, volume 34, pp.  13001–13008, 2020.

Supplementary Materials

The supplementary materials is providing the proofs of the main's paper formal results. We also provide as much background results and references as possible throughout to ensure that all the derivations are self-contained.
Some of the below derivation do not belong to formal statements but are included to help the curious readers get additional insights into current SSL methods.

§ CODE

model = torchvision.models.__dict__[architecture]()

# CIFAR procedure to adjust to the lower image resolution
if is_cifar and "resnet" in architecture:
model.conv1 = torch.nn.Conv2d(3, 64, kernel_size=3, stride=1, padding=2, bias=False)
model.maxpool = torch.nn.Identity()

# for each architecture, remove the classifier and get the output dim. (K)
if "alexnet" in architecture:
K = model.classifier[6].in_features
model.classifier[6] = torch.nn.Identity()
elif "convnext" in architecture:
K = model.classifier[2].in_features
model.classifier[2] = torch.nn.Identity()
elif "convnext" in architecture:
K = model.classifier[2].in_features
model.classifier[2] = torch.nn.Identity()
elif "resnet" in architecture or "resnext" in architecture or "regnet" in architecture:
K = model.fc.in_features
model.fc = torch.nn.Identity()
elif "densenet" in architecture:
K = model.classifier.in_features
model.classifier = torch.nn.Identity()
elif "mobile" in architecture:
K = model.classifier[-1].in_features
model.classifier[-1] = torch.nn.Identity()
elif "vit" in architecture:
K = model.heads.head.in_features
model.heads.head = torch.nn.Identity()
elif "swin" in architecture:
K = model.head.in_features
model.head = torch.nn.Identity()

Get the output dimension and remove the linear classifier from a given torchvision model (Pytorch used for illustration).

Reprise of <ref> but on additional datasets.

§ IMPACT OF TRAINING TIME AND LABEL SMOOTHING

Depiction of the evolution of linear top1 accuracy throughout epochs on CIFAR100 with three Resnet variants and three label smoothing parameters represented by the different shades of blue going from light to dark shades with values of $0.1$, $0.4$, and $0.8$ respectively. We clearly observe that higher value of label smoothing speeds up convergence, although in this setting all cases greatly benefit from longer training schedules; final linear probe performances are reported in <ref>.

Final top1 test accuracy values on CIFAR100 with the models and training schedules depicted in <ref>, only the label smoothing value with the best accuracy is reported.

Epochs        50        100        200        500        1000        5000        10000

resnet18      33.46      42.94      48.24      54.54       58.81       62.63        63.29

resnet50      37.71      47.86      54.04      60.23       64.24       69.51        69.91

resnet101     34.03      46.59      54.3       60.8        64.71       70.56        71.39

§ IMPACT OF MINI-BATCH SIZE

TinyImagenet with fixed number of epochs and a single learning rate which is adjusted for each case using the LARS rule therefore per batch-size learning cross-validation can only improve performances, see <ref>, , the per-epoch time includes training, testing, and checkpointing.

§ IMPACT OF DATA-AUGMENTATION

TinyImagenet, see <ref> for table of results, and the specific DAs can be found in <ref>.

TinyImagenet top1 test accuracy with a faster training schedule than the <ref> (2000 epochs) with average and std over $5$ runs with 3 different DA augmentation pipelines and $4$ different architectures, the per-epoch linear probe evaluation performances are reported in <ref>, DAs can be found in <ref>.

strength: 1        strength: 2        strength: 3

resnet18    31.48$\pm$ 0.3     43.62$\pm$ 0.2     43.88$\pm$ 0.3

resnet34    32.93$\pm$ 0.6     45.60$\pm$ 0.2      45.75$\pm$ 0.1

resnet50    40.24$\pm$ 0.7     48.80$\pm$ 0.6      50.81$\pm$ 0.8

resnet101    40.07$\pm$ 0.4     49.74$\pm$ 0.5     50.76$\pm$ 0.5

transforms = [
RandomResizedCropRGBImageDecoder((size, size)),
if strength > 1:
torch.nn.ModuleList([T.ColorJitter(0.4, 0.4, 0.4, 0.2)]), p=0.3
if strength > 2:
torch.nn.ModuleList([T.GaussianBlur((3, 3), (1.0, 2.0))]), p=0.2

Custom dataset to obtain the indices ($n$) in addition to inputs $\vx_n$ and (optionally) the labels $y_n$ to obtain  used in <ref> (Pytorch used for illustration).

§.§ How DIET Relates to Existing Methods

Possible original of noise label formulation <cit.>

those techniques  and yet often reaches, and sometimes surpasses, final performances, leading us to the conclusion that existing methods are overly complicated i.e. for only a marginal performance gain, the amount of complication increases exponentially.
Looking at noise distribution over labels start from a very natural observation when labels are obtained form annotators: noisy labels can appear during the labeling process and can even be more and more common as the sample get near the true underlying decision boundary <cit.>.

This led to novel losses e.g. in kernel regime <cit.>, precise statistical guarantees <cit.>.

learns first good features <cit.>

Deep learning robust to noise<cit.>

early stopping is provably robust <cit.>

could be related to spectral bias <cit.>

we will see more later, but overfitting is actually good (as in you preserve as much information as possible about the input) i.e. no need for early stopping and the likes.

§ OUTPERFORMING SELF-SUPERVISED METHODS BY EMPLOYING RANDOM LABELS AND ROBUST SUPERVISED LOSSES

Difference with SSL:
In the paradigm we consider, all the data has noisy labels, with an unknown fraction being trust- worthy. We do not assume the availability of any clean labels, e.g. those provided by a human. This contrasts with semi-supervised learning (SSL) <cit.>, where some fraction of the data has high quality labels but the rest are either unlabeled or have unreliable labels. Although closely related, in fact the two approaches are complementary to one another.

§.§ Single image per class can be used for trusted labels

take one image per class but a lot of classes to avoid degenerate case, but not too many either?

and then apply the SOTA method <cit.>

led to combining label noise robust method with SSL [Huang et al., 2022]

§ SSL HAS BEEN DOING RANDOM LABEL CLASSIFICATION ALL ALONG

The notations will be crucial to understand all the variants of the existing losses. Let's use the following

\begin{align}
\hat{p}_{y|\vx}(y=c|\vx;\vtheta)=\softmax(h_{\vtheta}(\vx))_c\triangleq f_{\vtheta}(\vx)_c,
\end{align}

with $\vx \in \mathbb{R}^{D}$ and $y$ our categorical variable taking values in $\{1,\dots,C\}$. In our case, we will observe $N$ samples, flattened and stacked into the $N \times D$ matrix $\mX\triangleq [\vx_1,\dots,\vx_N]^\intercal$. For each $\vx_n$, we also observe a noisy label $y_{\epsilon,n}$, we stack all of them into the $N \times C$ matrix $\mY_{\epsilon}$ where each line is a one-hot vector $(\mY_{\epsilon})_{n,c}=1_{\{y_n=c\}}$; this will help us lighten our notations throughout the study. For example one could assume that $\mY_{\epsilon}=\mY \mP$ where $\mY$ was the true label matrix of same type as $\mY_{\epsilon}$, and $\mP$ is a $C \times C$ permutation matrix; more complicated forms could be used as well. When denoting $h_{\vtheta}(\mX)$ or $f_{\vtheta}(\mX)$ it will represent the row-wise application of the operators.

<cit.> introduce the concept of critical parameters based on the lottery ticket hypothesis. The idea is that although DNs memorize clean label signals first, the noisy labels still impact the over DN's parameters. By actively picking which parameters are updated, it is possible to perform robust early-learning. This is done by using the loss only on a subset of parameters; let's denote the use of any supervised loss $\mathcal{L}_{\rm sup}$ and a regularizer term $\mathcal{R}$, then the method sparsifies the gradient signals coming from the complete loss $\mathcal{L}=\mathcal{L}_{\rm sup}+\lambda \mathcal{R}$ via

\begin{align*}
\frac{\partial \mathcal{L}}{\partial \theta_i}=\begin{cases}
\gamma \frac{\partial \mathcal{R}}{\partial \theta_i}(\vx_n),&\iff |\frac{\partial \mathcal{L}_{\rm sup}}{\partial \theta_i}(\vx_n,\vy_n)\theta_i|<\kappa\\
(1-\tau)\frac{\partial \mathcal{L}_{\rm sup}}{\partial \theta_i}(\vx_n,\vy_n)+\gamma\frac{\partial   \mathcal{R}}{\partial \theta_i}(\vx_n),&\text{ else}\\
\end{cases}
\end{align*}

with $\kappa$ the  $(1-\tau)\text{Card}(\vtheta)^{\rm th}$ order statistic of $\nabla_{\vtheta} \mathcal{L}_{\rm sup}(\vx_n,\vy_n)$, and $\tau$ the noise level. Nore on gradient clipping and self-paced learning in <cit.>.

reweighting samples <cit.> but unfortunately uses the true validation set with clean labels.

§.§ Noise Adaptation Layer Recover SSL's Projector Network

Methods in this category utilize the noise transition matrix, denoting the probabilities that clean labels flip into noisy labels, to build consistent algorithms.

The explicit noise modeling layer was derived from a robust logistic regression loss in  <cit.> also concurrently found in <cit.> and all extending the outlier-robust loss modeling from <cit.>

\begin{align}
-\log\left(\sum_{c'=1}^{C}\underbrace{p(y=c|\tilde{y}=c')}_{\text{noise resampling layer}}\underbrace{\hat{p}(\tilde{y}=c'|\vx)}_{\text{$f(\vx)_{c'}$ if using softmax}}\right),\label{eq:noise_model}
\end{align}

at test time only $p(\tilde{y}=c'|f(\vx))$ is employed for prediction.

Noise Adaptation Layer (NAL). Going to the multi-class setting and assuming that one employs the cross-entropy loss $\mathcal{L}_{\rm CE}$ for training, <cit.> proposes to learn the transition matrix $\mT$ while adding a trace regularization for training as in leading to the loss

\begin{align}
\mathcal{L}_{\rm NALtr}(\vtheta,\mT)=\mathcal{L}_{\rm CE}(\mY_{\epsilon},\softmax(\mT^\intercal)^\intercal f_{\vtheta}(\mX))+\lambda \Tr(\softmax(\mT^\intercal)),\label{eq:NALtr}
\end{align}

where one can employ a row-wise $\softmax$ to maintain a row-stochastic matrix as in <ref>, or can use another method.

Dropout NAL. Building on this model, <cit.> proposes to model $\mT$ from a linear layer with softmax output activation, and to prevent possible noise-label overfitting by adding dropout leading to the loss

\begin{align}
\mathcal{L}_{\rm NALdrop}(\vtheta,\mW)=\mathcal{L}_{\rm CE}(\mY_{\epsilon},\softmax(\mW \dropout(f_{\vtheta}(\mX)))),
\end{align}

Expectation-Maximization NAL. Another extension <cit.> considers the true label to be latent variables and perform EM-based inference to obtain
an $(N \times C)$ assignment matrix $\mC$ leading to

\begin{align}
&\mC \leftarrow \softmax(\mY_{\epsilon}\log(\mT)+ h_{\vtheta}(\mX)),\label{eq:C_update}\\
&\mT \leftarrow \diag(\mC^\intercal\mathbf{1}_{N})^{-1}\mC^\intercal \mY_{\epsilon},\label{eq:T_update}\\
&\mathcal{L}_{\rm NALem}(\vtheta)=\mathcal{L}_{\rm CE}(\mC,f_{\vtheta}(\mX)),
\end{align}

where <ref> are applied at each step during training and correspond to the E-step inference of the true labels, and the M-step update of the transition matrix parameters.

Forward/Backward Correction. <cit.> considers a mapping that can be expressed as $f_{\vtheta}=\phi^{-1} \circ h_{\vtheta}$ with the invertible link function $\phi:\Delta_{C-1} \mapsto \mathbb{R}^C$ <cit.>, e.g. $f_{\vtheta}$ contained the softmax output and now $h_{\vtheta}$ is the same mapping but without softmax, and $\phi^{-1}$ is the softmax. Notice how in this case the considered supervised loss function can be expressed as $\mathcal{L}(\vy,f_{\vtheta}(\vx))=\mathcal{L}(\vy,\phi^{-1}(h_{\vtheta}(\vx)))$. If in addition one has
\argmin_{h}\mathbb{E}_{(\vx,\vy)}\left[\mathcal{L}(\vy,\phi^{-1}(h(\vx)))\right]=\phi \circ p_{(\vy|\vx)}$,
then one has $
\mathbb{E}_{\vx,\vy_{\epsilon}}\left[\mathcal{L}(\vy,T^{T}f_{\vtheta}(\vx))\right]=\mathbb{E}_{\vx,\vy}\left[\mathcal{L}(\vy,f_{\vtheta}(\vx))\right]$

and the estimator for $\mT$ is the multiclass version of the one proposed for binary classes from <cit.> and is defined as

\begin{align}
T_{i,j} = f_{\vtheta}(\argmax_{\vx \in \sX}f_{\vtheta}(\vx)_i)_j,\label{eq:T_estimate}
\end{align}

T-revision <cit.> proposes a slightly different formulation based on the observation that a weighted loss can also provided a consistent estimator

\begin{align*}
\mathbb{E}_{\vx,\vy}\left[\mathcal{L}(\vy,f_{\vtheta}(\vx))\right]=&\int \sum_{c=1}^{C} p_{\vx,\vy}(\vx,\vy)\mathcal{L}(\vy,f_{\vtheta}(\vx)) d\vx\\
=&\int \sum_{c=1}^{C} p_{\vx,\vy_{\epsilon}}(\vx,\vy)\frac{p_{\vy|\vx}(\vy|\vx)}{p_{\vy_{\epsilon}|\vx}(\vy|\vx)}\mathcal{L}(\vy,f_{\vtheta}(\vx)) d\vx.
\end{align*}

By using $T^Tf_{\vtheta}(\vx)$ as the approximation of $p_{\vy_{\epsilon}|\vx}$ as already done in the noise modeling layer studies, leaving $f_{\vtheta}(\vx)$ as the approximation of $p_{\vy|\vx}$. The T-revision method then consists of first fitting the unweighted loss, approximate $T$ e.g. from <ref> and then optimize

\begin{align*}
\mathcal{L}_{\rm Trev}(\vtheta,\Delta \mT)=\frac{1}{N}\sum_{n=1}^{N}\frac{f_{\vtheta}(\vx_n)_{y_{\epsilon,n}}}{((\mT+\Delta \mT)^Tf_{\vtheta}(\vx_n))_{y_{\epsilon,n}}}\mathcal{L}_{\rm CE}(\vy_{\epsilon,n}, f_{\vtheta}(\vx_n)).
\end{align*}

Although $\mT + \Delta \mT$ should be a row-stochastic matrix, the authors did not impose the constraint during tuning [<https://github.com/xiaoboxia/T-Revision>].

The main issue is that even with the real confusion matrix i.e. the entries of $p(y=c|\tilde{y}=c')$ no optimality guarantee is known.

§.§ Going From Inconsistent to Consistent Losses is equivalent to going from contrastive to non-contrastive methods

More generally <cit.> presented a way to modify any given surrogate loss function for binary classification to achieve noise robustness.

A loss function $\mathcal{L}$ is called symmetric if
$\exists \kappa \in \mathbb{R} : \sum_{c=1}^{C}\mathcal{L}(f(\vx),c)=\kappa, \forall f,\forall \vx \in \sX$.

A crucial benefit of symmetric loss functions is that minimizing their empirical risk produce a noise-tolerant mapping under uniform label noise, even with class dependent noise <cit.>; this holds as long as the noise rate is lesser than $\frac{C-1}{C}$, see <cit.> for a theoretical study. From this, we also see that CE is sensitive to label noise.

Mean Absolute Error. <cit.> thus proposed the use of the mean absolute error

\begin{align}
\mathcal{L}_{\rm MAE}(\vtheta)=\frac{\left\| \mY_{\epsilon}-f(\mX)\right\|_1}{N}=2-2\frac{\Tr\left(\mY^\intercal_{\epsilon}f_{\vtheta}(\mX)\right)}{N},
\end{align}

is symmetric and thus preferable; note that it is bounded since there is a softmax activation at the output of the DN, otherwise some form of regularization e.g. weight-decay must be employed to prevent divergence. Notice how this also corresponds to the unhinged loss also found to be crucial for label noise robustness <cit.>.

Label Smoothing.
<cit.> found that label-smoothing

\begin{align}
\mathcal{L}_{\rm LS}(\vtheta)=\mathcal{L}_{\rm CE}(\mY_{\epsilon}-\alpha,f_{\vtheta}(\mX)),
\end{align}

when distilling models from noisy data is akin backward correction and helps. In particular when doing distillation, label smoothing of the teacher is beneficial.  In distillation, a trained “teacher” model’s logits are used to augment (or replace) the one-hot labels used to train a “student” model <cit.>.

Generalized Cross-Entropy. The MAE loss is however slow to converge, and thus <cit.> propose to use the Generalized Cross Entropy (GCE) loss introduced as a mean to perform robust maximum likelihood estimation in <cit.>. This loss is obtained by applying a Box-Cox transformation to probabilities (power law function of probability with exponent $q$)

\begin{align}
\mathcal{L}_{\rm GCE}(\vtheta)=\sum_{n=1}^N 1_{\{f(\vx_n)_{y_n}>k\}}\frac{1-f(\vx_n)_{y_n}^q}{q},
\end{align}

and can behave like a weighted MAE and CE; commonly one picks $k \approx0.5$ and $q\approx 0.7$. If $k$ is set so that the indicator function is not always true, then we refer to this loss as the truncated one denoted as $\mathcal{L}_{\rm GCEtrunc}$. Notice how for $q=0$ and $q=1$ the loss $\mathcal{L}_{\rm GCE}$ recovers the cross-entropy $\mathcal{L}_{\rm CE}$ and the MAE $\mathcal{L}_{\rm MAE}$ losses respectively.

Symmetric Cross-Entropy. This was further extended by <cit.> introduce the Symmetric cross-entropy (SCE)

\begin{align*}
\mathcal{L}_{\rm SCE}(\vtheta)=\mathcal{L}_{\rm CE}(\mY_{\epsilon},f_{\vtheta}(\mX))+\alpha\mathcal{L}_{\rm CE}(f_{\vtheta}(\mX),\mY_{\epsilon}+(1-\mY_{\epsilon})\epsilon),
\end{align*}

where $\epsilon$ is used to prevent instabilities; although notice that $\ve_c+(1-\ve_c)\epsilon$ no longer sums to $1$; yet with this formulation, it is possible to recover the mean absolute error setting $\epsilon = \exp(-2)$. Usually one sets $\epsilon\approx 0.0001$[<https://github.com/YisenWang/symmetric_cross_entropy_for_noisy_labels>].

Determinant Mutual Information. Moving away from distance based losses between classifier’s outputs and the labels, an information theoretic loss robust to noisy labels was proposed in <cit.>. In particular, Determinant based Mutual Information (DMI) <cit.>, a generalization of Shannon's Mutual information (MI) <cit.> is employed.

\begin{align}
\mathcal{L}_{\rm DMI}(\vtheta)=-\log \left(|\det\left( \mY_{\epsilon}^\intercal f_{\vtheta}(\mX)\right)|+\epsilon\right),
\end{align}

with $\epsilon$ usually being set to $0.001$[<https://github.com/Newbeeer/L_DMI/blob/master/CIFAR-10/DMI.py>]

Meta-Classes. <cit.> Define $M$ meta-class tasks corresponding to different grouping of the original classes, need to be a partition. Then train a classifier for each of the $M$ classification tasks

\begin{align*}
\mathcal{L}_{\rm COMB}(\vtheta)=\sum_{m=1}^{M}\mathcal{L}_{\rm CE}(\mC^{(m)},g_{\veta^{(m)}}(f_{\vtheta}(\mX)))
\end{align*}

and then despite predicting different classification tasks, it is possible to predict the original class although not relevant for our study, seepredict the true class (if needed, irrelevant to use) with

§.§ Agreement Constraints and Self-Labeling Recover Joint-Embedding SSL

based on multiple networks akin to teacher-student

cotraining <https://www.cs.cmu.edu/ avrim/Papers/cotrain.pdf>

mentornet <https://arxiv.org/abs/1712.05055>

coteaching <https://arxiv.org/abs/1804.06872>

co teaching + <https://proceedings.mlr.press/v97/yu19b/yu19b.pdf>

SURVEY <cit.>

JoCor <cit.> uses joint network, each with its loss + agreement loss! same as MSN if you add a few tweaks.

decoupling when and where <cit.> extends M-Net.

self-ensemble label filtering (SELF) <cit.> using the idea of mean teacher <cit.> i.e. ensemble of a model at different training times.

SAVAGE LOSS <cit.>

Another approach is to alter the target to enforce consistency from the current state of the model. The motivation is to leverage the model's improvement over training to mitigate the noisy labels by combining them. To do so, <cit.> introduce “bootstrapping”[the authors justify this nomination from the meaning“pulling oneself up by one’s own bootstraps” and due to inspiration to <cit.>] by updating the targets from the model's prediction in one of two ways

\begin{align*}
\mathcal{L}_{\rm boot}(\vy_n,f(\vx_n);\beta)=\begin{cases}\mathcal{L}_{\rm CE}(\beta \vy_n + (1-\beta)f(\vx_n),f(\vx_n)),&\text{(soft-assignment)}\\
\mathcal{L}_{\rm CE}(\beta \vy_n + (1-\beta)\ve_{\argmax_cf(\vx_n)_c},f(\vx_n)),&\text{(hard-assignment)}
\end{cases}
\end{align*}

where interestingly the soft-assignment setting falls back to

\begin{align*}
\mathcal{L}_{\rm boot}(\vy_n,f(\vx_n);\beta)=\beta\mathcal{L}_{\rm CE}(\vy_n,f(\vx_n))+(1-\beta)H(f(\vx)),
\end{align*}

recovering Entropy regularization used e.g. in semi-supervised learning <cit.>. One limitation of $\mathcal{L}_{\rm boot}$ lies in the fact that the original noisy labels are kept throughout training. Hence, <cit.> propose a more drastic strategy which is to continuously replace the current label values from the model's prediction as in

\begin{align*}
\vy_n \leftarrow \begin{cases}f(\vx_n),&\text{(soft-assignment)}\\
\ve_{\argmax_cf(\vx_n)_c},&\text{(hard-assignment)}
\end{cases}
\end{align*}

notice how training recovers the case of <cit.> by setting $\beta=1$ and stopping the gradient on the labels. Due to this update, two challenges emerge: (i) nothing prevents a trivial solution to occur e.g. all $\vy_n$ are the same, and (ii) with the soft-label update the training gets stuck to a local minimum rapidly since the new target is the model's prediction which is a global minimizer of the CE. Hence, two regularizers are introduced to prevent those cases: (i) a KL-divergence between a target prior $\vp$ and the empirical class distribution $\overline{f}\triangleq\frac{1}{|\sB|}\sum_{n \in \sB}f(\vx_n)$, and (ii) an Entropy regularization on the predicted posterior probabilities $f(\vx_n)$ leading to the complete loss

\begin{align}
\mathcal{L}_{\rm CE}(f(\vx_n),\vy_n)+KL(\vp || \overline{f})+\frac{1}{|\sB|}\sum_{n=1}^{N}H(f(\vx_n))
\end{align}