# Generative reconstruction of 3D volume elements for Ti-6Al-4V basketweave
microstructure by optimization of CNN-based microstructural descriptors

Vincent Blümer
Chair of Nonlinear Solid Mechanics
University of Twente
Enschede, The Netherlands
&Celal Soyarslan
Chair of Nonlinear Solid Mechanics
Fraunhofer Innovation Platform
University of Twente
Enschede, The Netherlands
&Ton van den Boogaard
Chair of Nonlinear Solid Mechanics
University of Twente
Enschede, The Netherlands

###### Abstract

We present a methodology for the generative reconstruction of 3D Volume
Elements (VE) for numerical multiscale analysis of Ti-6Al-4V processed by
Additive Manufacturing (AM). The basketweave morphology, which is typically
dominant in AM-processed Ti-6Al-4V, is analyzed in conventional Electron
Backscatter Diffusion (EBSD) micrographs. Prior $\beta$-grain reconstruction
is performed to obtain the out-of-plane orientation of the observed grains
leveraging Burgers orientation relationship. Convolutional Neural Network
(CNN) - based microstructure descriptors are extracted from the 2D data, and
used for cross-section-based optimization of pixel values on orthogonal planes
in 3D, using the Microstructure Characterization and Reconstruction (MCR)
implementation MCRpy [16]. In order to utilize MCRpy, which performs best for
binary systems, the basketweave microstructure, which consists of up to twelve
distinct grain orientations, is decomposed into several separate two-phase
systems. Our reconstructions capture key characteristics of the titanium
basketweave morphology and show qualitative resemblance to experimentally
obtained 3D data. The preservation of volume fraction during assembly of the
reconstruction remains an unadressed challenge at this stage.

_Keywords_ Machine Learning $\cdot$ Image Reconstruction $\cdot$ Convolution
Neural Network $\cdot$ Titanium $\cdot$ Multiscale

##  1 Introduction

The accuracy of computed macroscale properties of any full-field multiscale
analysis depends on the representativeness of the microscale solution domain.
A volume element is representative, when it captures at least the minimum
amount of microscale characteristics required to compute converged macroscale
(or effective) properties. In other words, when the size of a Representative
Volume Element (RVE) is increased, and therefore more information is captured,
the computed macroscale properties do not change. For an extended discussion,
see for example [8, 15, 17, 9].

Computational multiscale studies of metals are usually based on EBSD
micrographs, as they capture information about the material phase and crystal
lattice orientation at the observed points in the micrograph domain. The
single crystals are represented by elasticity with the applicable symmetry
class and slip-system-based plasticity, a concept usually referred to as
crystal plasticity.

While 2D EBSD data can usually be acquired with reasonable effort, the
generation of 3D data becomes tedious and often infeasible. Unlike other
techniques like Computer Tomography (CT), it cannot penetrate the material and
needs to be performed on free surfaces. Therefore, the only way to generate 3D
data is the sequential removal of material, surface preparation, and data
recording. 3D EBSD studies that are especially noteworthy in this context are
[5, 21, 4].

An alternative is the synthetic generation of RVEs by computational means and
geometrical assumptions. This approach has long been common practice for
example in steel microstructures that show a strong resemblance to Voronoi
tessellations. The approximation error of the tessellation relative to the
actual EBSD data is tolerated for the gain of 3D volumetric data. Due to the
greatly reduced expense compared to the acquisition of 3D EBSD data through
sequential slicing, it is a common choice for microstructures of low to
moderate complexity. However, as the complexity of the microstructure
increases, more details of the composition are lost.

The advent of Machine Learning (ML) in image recognition, sophisticated open-
source modules for parallel optimization, and specialized computing hardware
allows for new approaches to dimensionality expansion of microstructural data.
In this contribution we develop a methodology for the generative
reconstruction of 3D microstructural data for polycrystalline materials.

##  2 Crystallography of Titanium

As a basis for subsequent chapters, a brief introduction into the
crystallographic properties of Titanium alloys is given.

###  2.1 Phases and transformations

At high temperatures, titanium generally occurs as $\beta$-titanium with body-
centered-cubic (bcc) atomic lattice. At low/room temperature,
$\alpha$-titanium with hexagonal-closest-packed (hcp) lattice is the dominant
phase. Alloying elements can be categorized into either $\alpha$\- or
$\beta$-stabilizers, with Al being one of the former and V one of the latter.
The alloy composition also influences the $\beta$-transus temperature. The
widely used Ti-6Al-4V alloy transforms from $\beta$ to $\alpha$ phase during
cooling at $995^{\circ}$C [11]. It occurs in a wide range of room-temperature
morphologies depending on processing history.

The most commonly observed morphologies are globular (or primary) alpha,
transformed $\alpha+\beta$ (in the form of colonies or basketweave), bi-modal
(when globular alpha is surrounded by transformed $\alpha+\beta$), and
martensite [14]. In wrought Ti-6Al-4V, bi-modal microstructure is typically
observed. However, when processed by additive manufacturing, Ti-6Al-4V
consists almost exclusively of transformed $\alpha+\beta$.

The formation of the transformed $\alpha+\beta$ microstructure is governed by
Burgers orientation relationship, which allows for twelve distinct variants of
the transformation of the bcc- into the hcp-lattice. The emerging
$\alpha$-grains have high aspect ratios and expand in so-called habit planes
[20]. In the formation of basketweave, all twelve variants of the $\beta$ to
$\alpha$ transformation are active. In colonies, entire prior $\beta$ grains
transform along one of the variants, leading to a larger region of stacked
$\alpha$-platelets with the same plate normal and crystal orientation [12].

Titanium martensite can occur in the form of distorted-hcp $\alpha^{\prime}$\-
and orthorhombic $\alpha^{\prime\prime}$-martensite. In Ti-6Al-4V, martensite
creation is caused by the supersaturation of V in the hcp lattice. It should
be noted that titanium martensite does not suffer from the characteristic loss
of ductility as steel martensite, due to absence of carbon atoms. In the Ti-V
system, a concentration of $0-10$ at.$\%$ V leads to the formation of
$\alpha^{\prime}$, while a concentration of $>10$ at.$\%$ invokes
$\alpha^{\prime\prime}$ martensite [13]. The latter is not commonly observed
in Ti-6Al-4V.

###  2.2 Microstructural evolution during the AM-process

The Powder Bed Fusion (PBF) process is characterized by rapid, repeated
heating and subsequent cooling. The molten powder solidifies as $\beta$ in
columnar grains that extend along the build direction. Upon cooling past the
$\beta$-transus temperature, the $\beta$-titanium undergoes a martensitic
transformation into $\alpha^{\prime}$-martensite which is supersaturated with
V and arranged into plates governed by Burgers orientation relationship. V is
ejected to the boundaries of the $\alpha^{\prime}$ grains which decompose into
$\alpha+\beta$. $\beta$ rods form on the boundary of the platelets [19]. It is
important to note that the transformed $\alpha+\beta$ microstructure obtains
its morphology from a martensitic transformation, but generally does not
consist of martensite. Tan et al. [19] observed martensite phase only in very
thin printed samples.

Apart from a thin boundary layer where colonies are observed, the basketweave
morphology is dominant. Formation of the basketweave microstructure can be
attributed to the stabilizing temperature profile and coarsening
$\beta$-grains [12]. Higher cooling rates lead to finer basketweave and
therefore increased hardness [3].

AM-processed Ti-6Al-4V is characterized by large, columnar
prior-$\beta$-grains, that extend along the print direction. Among these
grains, the preferred orientation is the alignment of the
$[001]_{\beta}$-direction with the build direction. In the proximity of free
surfaces of a printed part, prior-$\beta$-grains tend to be inclined towards
the build direction at an angle and have distributed texture [1].

##  3 Generative reconstruction of 3D data

In the following sections, the necessary steps for the proposed methodology
are presented. More detailed insights into Burgers orientation relationship as
well as CNN-based microstructural descriptors are given.

###  3.1 Acquired data

Point of departure for the 3D reconstruction is a EBSD micrograph taken from a
tensile test specimen manufactured by Laser Powder Bed Fusion (L-PBF). The
physical dimensions of the micrograph are $77.86\,\mu\textup{m}\times
58.09\,\mu\textup{m}$ and the resolution is $1072\times 800$.

Figure 1: EBSD micrograph of an additively manufactured tensile test specimen
built from Ti-6Al-4V. The vertical axis is aligned with the build direction.
The rectangular region is cropped out for further study.

The build direction is aligned with the vertical axis of the image.
Basketweave microstructure can be observed everywhere in the micrograph. The
separation of the region into elongated parent grains is apparent to the eye.
The alignment convention of the crystal reference frame is
$\boldsymbol{X}\parallel\boldsymbol{a}^{*},\,\,\boldsymbol{Y}\parallel\boldsymbol{b},\,\,\boldsymbol{Z}\parallel\boldsymbol{c}^{*}$,
where $\boldsymbol{X},\boldsymbol{Y},\boldsymbol{Z}$ are the orthogonal axes
of the reference coordinate system and
$\boldsymbol{a},\boldsymbol{b},\boldsymbol{c}$ are the crystal axes of the hcp
lattice, where $\boldsymbol{a}$ and $\boldsymbol{b}$ are inclined at a
$120^{\circ}$ angle. The asterisk denotes the reciprocal of an axis. They are
computed as

$\boldsymbol{a}^{*}=2\pi\frac{\boldsymbol{b}\times\boldsymbol{c}}{\boldsymbol{a}\cdot\left(\boldsymbol{b}\times\boldsymbol{c}\right)},\quad\boldsymbol{b}^{*}=2\pi\frac{\boldsymbol{c}\times\boldsymbol{a}}{\boldsymbol{a}\cdot\left(\boldsymbol{b}\times\boldsymbol{c}\right)},\quad\boldsymbol{c}^{*}=2\pi\frac{\boldsymbol{a}\times\boldsymbol{b}}{\boldsymbol{a}\cdot\left(\boldsymbol{b}\times\boldsymbol{c}\right)}$
(1)

respectively.

###  3.2 Parent grain reconstruction

Parent grain reconstruction is performed using Matlab MTEX [2]. The
reconstruction is based on Burgers orientation relationship, according to
which each $\alpha$-grain should match one of twelve misorientations to the
parent grain from which it emerged. The twelve variants for alignment are
given in Table 1.

Table 1: The twelve variants of Burgers orientation relationship [6]. Each row
indicates one possible axis-alignment of the emerging $\alpha$-grain to the
parent grains axes.

Variant ID | $\boldsymbol{Z}_{\alpha}$ | $\boldsymbol{Y}_{\alpha}$ | $\boldsymbol{X}_{\alpha}$
---|---|---|---
V1 | $(110)_{\beta}\parallel(0001)_{\alpha}$ | $[\overline{1}1\overline{1}]_{\beta}\parallel[11\overline{2}0]_{\alpha}$ | $[\overline{1}12]_{\beta}\parallel[\overline{1}100]_{\alpha}$
V2 | $(110)_{\beta}\parallel(0001)_{\alpha}$ | $[1\overline{1}\overline{1}]_{\beta}\parallel[11\overline{2}0]_{\alpha}$ | $[\overline{1}1\overline{2}]_{\beta}\parallel[\overline{1}100]_{\alpha}$
V3 | $(1\overline{1}0)_{\beta}\parallel(0001)_{\alpha}$ | $[11\overline{1}]_{\beta}\parallel[11\overline{2}0]_{\alpha}$ | $[112]_{\beta}\parallel[\overline{1}100]_{\alpha}$
V4 | $(1\overline{1}0)_{\beta}\parallel(0001)_{\alpha}$ | $[111]_{\beta}\parallel[11\overline{2}0]_{\alpha}$ | $[\overline{1}\overline{1}2]_{\beta}\parallel[\overline{1}100]_{\alpha}$
V5 | $(011)_{\beta}\parallel(0001)_{\alpha}$ | $[11\overline{1}]_{\beta}\parallel[11\overline{2}0]_{\alpha}$ | $[\overline{2}1\overline{1}]_{\beta}\parallel[\overline{1}100]_{\alpha}$
V6 | $(011)_{\beta}\parallel(0001)_{\alpha}$ | $[1\overline{1}1]_{\beta}\parallel[11\overline{2}0]_{\alpha}$ | $[21\overline{1}]_{\beta}\parallel[\overline{1}100]_{\alpha}$
V7 | $(01\overline{1})_{\beta}\parallel(0001)_{\alpha}$ | $[\overline{1}11]_{\beta}\parallel[11\overline{2}0]_{\alpha}$ | $[211]_{\beta}\parallel[\overline{1}100]_{\alpha}$
V8 | $(01\overline{1})_{\beta}\parallel(0001)_{\alpha}$ | $[111]_{\beta}\parallel[11\overline{2}0]_{\alpha}$ | $[2\overline{1}\overline{1}]_{\beta}\parallel[\overline{1}100]_{\alpha}$
V9 | $(101)_{\beta}\parallel(0001)_{\alpha}$ | $[\overline{1}11]_{\beta}\parallel[11\overline{2}0]_{\alpha}$ | $[\overline{1}\overline{2}1]_{\beta}\parallel[\overline{1}100]_{\alpha}$
V10 | $(101)_{\beta}\parallel(0001)_{\alpha}$ | $[11\overline{1}]_{\beta}\parallel[11\overline{2}0]_{\alpha}$ | $[\overline{1}21]_{\beta}\parallel[\overline{1}100]_{\alpha}$
V11 | $(\overline{1}01)_{\beta}\parallel(0001)_{\alpha}$ | $[1\overline{1}1]_{\beta}\parallel[11\overline{2}0]_{\alpha}$ | $[121]_{\beta}\parallel[\overline{1}100]_{\alpha}$
V12 | $(\overline{1}01)_{\beta}\parallel(0001)_{\alpha}$ | $[111]_{\beta}\parallel[11\overline{2}0]_{\alpha}$ | $[\overline{1}2\overline{1}]_{\beta}\parallel[\overline{1}100]_{\alpha}$

For each $\alpha$-grain, its association to one parent-grain, as well as the
matching variant is computed. The orientation of the parent grain is also
reconstructed.

Figure 2: Computed orientations of parent grains after prior-$\beta$ grain
reconstruction is performed in Matlab MTEX.

Figure 2 illustrates the association of the $\alpha$-grains to their
respective parent grains and confirms the first visual impression. The
rectangular region shown in Figure 1 is used for further study. This is done
to isolate the microstructural information of one parent grain from blending
with the statistics of other regions of the EBSD.

Figure 3: During the prior-$\beta$ grain reconstruction, each $\alpha$ grain
is matched to one of the twelve variants of Burgers orientation relationship.

Figure 3 shows the computed misorientation variants of the $\alpha$ grains. It
is evident that all grains that belong to one of the misorientation variants
(hereafter called grain group) have almost identical grain axis orientations
in the 2D observation plane (in addition to their identical crystal lattice
orientations). The preferred growth direction of an $\alpha$-grain is called
the habit plane, which is normal to one of the family of
$[1100]_{\alpha}$-directions of the $\alpha$-crystal after transformation
[20]. Therefore, the grain axis orientation in the out of plane dimension can
be reconstructed up to the ambiguity caused by the rotational symmetry of the
hcp lattice.

Applying Burgers orientation relationship yields three vectors that
potentially represent a grain group’s habit plane normal. From these vectors,
the one that is closest to in-plane orthogonal to the in-plane grain axis is
selected as the most plausible habit plane normal. Results of this process are
shown in Figure 4. The habit plane normals are later used to assemble the
microstructural descriptors in orientations consistent with the 3D
orientations of grains.

Figure 4: All $\alpha$-grain groups are extracted from the EBSD and their
habit plane normals are computed. This process is shown for three of the
twelve groups.

###  3.3 Microstructural Descriptor

MCRpy is an open-source software platform used for characterization and
reconstruction of microstructures. Reconstructions are obtained through
optimization of pixel values in a reconstruction domain to minimize a loss
function. The choice of descriptor is flexible and ranges from $n$-point
statistics to more novel choices like Gram matrices [7], which have lead to
promising results on recent years [10]. Gram matrices are calculated from the
activations of a pre-trained convolutional neural network when exposed to an
image. MCRpy uses the general image recognition network VGG-19 [18] with
ImageNet weights as a reference [16].

###  3.4 Reconstruction of grain groups

A characteristic property of titanium basketweave microstructure is that all
$\alpha$-grains emerging from the same parent grain have one out of twelve
distinct orientations. Not all of the continuous space of possible
orientations is observed in the microstructure, which allows the
representation of the microstructure as a discrete set of twelve phases. MCRpy
is primarily built for the reconstruction of two-phase microstructures, by
varying pixel values in a continuous range from 0 to 1. Although the applied
concepts extend to higher number of phases, and multiphase reconstruction is
implemented in MCRpy, the addition of a third phase already leads to a steep
increase in computational time. A potential circumvention of this limitation
is the separation of the microstructure into twelve separate reconstruction
problems and subsequent superposition. Each of the grain groups is isolated
from the EBSD and represented as phase 1 against a background of phase 0.
Reconstructions are performed over $100\times 100\times 100$ grid domains
where the out-of-plane descriptors of the phases are assembled consistent with
the orientation of the grain group’s habit plane normal. A selection of
results is shown in the following figures. For a visual comparison to
experimentally obtained data, the reader is referred to [5].

(a) The 7th grain group with a volume fraction of 8.35%

(b) The 12th grain group with a volume fraction of 8.47%

(c) The 6th grain group with a volume fraction of 10.34%

Figure 5: Three-dimensional reconstructions of three distinct groups from a
set of twelve categorized grain types.

###  3.5 Superposition of reconstructions

The complete reconstruction of the microstructure is assembled from the
reconstructions of the separate subsystems. At this stage, no specific
hierarchy is governing the superposition. The resulting microstructure is
depicted in Figure 6.

Figure 6: Superposition of all twelve 3D reconstructions with a resulting
volume fraction of 52.78%

In the reconstructed microstructure, key characteristics of the titanium
basketweave microstructure, like high-aspect-ratio, intersecting grains, are
captured and qualitative resemblance to experimentally obtained 3D data [5] is
observed. A problem that becomes evident quickly is the relatively low volume
fraction of filled space in the reconstruction domain. While the sum of volume
fractions of the separate reconstructions is 73.11%, the superposition of
reconstructions only fills 52.78% of the space in the reconstruction domain.
The volume fraction is preserved in the 2D to 3D reconstructions, but
substantial loss is observed in the superposition of reconstructions, as the
separate reconstructions do not take into account the space that is to be
filled by another grain group.

##  4 Conclusion

We present a methodology for the generative reconstruction of 3D
reconstructions of polycrystalline microstructures from conventional EBSD
data, leveraging Burgers orientation relationship and cross-section-based
optimization based on CNN-based descriptors using the MCRpy implementation.
Circumventions to a few of a the limitations of the approach are proposed. A
number of open challenges remain like the treatment of secondary martensite,
ensuring a representative distribution of volume fractions, and quantitative
evaluation of the reconstructed microstructures.

## Acknowledgements

We would like to thank Ali Reza Safi and Dr. Benjamin Klusemann from
Helmholtz-Zentrum Hereon for the in-depth discussions regarding microstructure
reconstruction and MCRpy usage, and Sikander Naseem from the Fraunhofer
Innovation Platform for Advanced Manufacturing at the University of Twente for
supplying the EBSD micrographs.

## References

  * [1] A. A. Antonysamy, J. Meyer, and P. Prangnell.  Effect of build geometry on the $\beta$-grain structure and texture in additive manufacture of Ti-6Al-4V by selective electron beam melting.  Materials characterization, 84:153–168, 2013.
  * [2] F. Bachmann, R. Hielscher, and H. Schaeben.  Texture analysis with MTEX–free and open source software toolbox.  Solid state phenomena, 160:63–68, 2010.
  * [3] E. Brandl and D. Greitemeier.  Microstructure of additive layer manufactured Ti-6Al-4V after exceptional post heat treatments.  Materials Letters, 81:84–87, 2012.
  * [4] M. G. Chapman, M. N. Shah, S. P. Donegan, J. M. Scott, P. A. Shade, D. Menasche, and M. D. Uchic.  AFRL additive manufacturing modeling series: challenge 4, 3D reconstruction of an IN625 high-energy diffraction microscopy sample using multi-modal serial sectioning.  Integrating Materials and Manufacturing Innovation, 10:129–141, 2021.
  * [5] R. DeMott, P. Collins, C. Kong, X. Liao, S. Ringer, and S. Primig.  3D electron backscatter diffraction study of $\alpha$ lath morphology in additively manufactured Ti-6Al-4V.  Ultramicroscopy, 218:113073, 2020.
  * [6] T. Furuhara, H. Nakamori, and T. Maki.  Crystallography of $\alpha$ phase precipitated on dislocations and deformation twin boundaries in a $\beta$ titanium alloy.  Materials Transactions, JIM, 33(6):585–595, 1992.
  * [7] L. A. Gatys, A. S. Ecker, and M. Bethge.  Image style transfer using convolutional neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2414–2423, 2016.
  * [8] R. Hill.  Elastic properties of reinforced solids: Some theoretical principles.  Journal of the Mechanics and Physics of Solids, 11(5):357–372, 1963\.
  * [9] T. Kanit, S. Forest, I. Galliet, V. Mounoury, and D. Jeulin.  Determination of the size of the representative volume element for random composites: statistical and numerical approach.  International Journal of Solids and Structures, 40(13-14):3647–3679, 2003.
  * [10] X. Li, Y. Zhang, H. Zhao, C. Burkhart, L. C. Brinson, and W. Chen.  A transfer learning approach for microstructure reconstruction and structure-property predictions.  Scientific reports, 8(1):13461, 2018.
  * [11] G. Lütjering and J. C. Williams.  Titanium.  Springer, 2007.
  * [12] K. Makiewicz, S. Babu, M. Keller, and A. Chaudhary.  Microstructure evolution during laser additive manufacturing of Ti-6Al-4V alloys.  In Proceedings of the International Conference. Trends Weld. Res. Chicago, IL, 2012.
  * [13] W. Mei, J. Sun, and Y. Wen.  Martensitic transformation from $\beta$ to $\alpha^{\prime}$ and $\alpha^{\prime\prime}$ phases in Ti-V alloys: A first-principles study.  Journal of Materials Research, 32(16):3183–3190, 2017.
  * [14] R. Pederson.  Microstructure and phase transformation of Ti-6Al-4V.  PhD thesis, Luleå Tekniska Universitet, 2002.
  * [15] K. SAB.  On the homogenization and the simulation of random materials.  European journal of mechanics. A. Solids, 1992.
  * [16] P. Seibert, A. Raßloff, K. Kalina, M. Ambati, and M. Kästner.  Microstructure characterization and reconstruction in Python: MCRpy.  Integrating Materials and Manufacturing Innovation, 11(3):450–466, 2022.
  * [17] H. Shen and L. C. Brinson.  A numerical investigation of the effect of boundary conditions and representative volume element size for porous titanium.  Journal of Mechanics of Materials and Structures, 1:1179–1204, 2006\.
  * [18] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [19] X. Tan, Y. Kok, W. Q. Toh, Y. J. Tan, M. Descoins, D. Mangelinck, S. B. Tor, K. F. Leong, and C. K. Chua.  Revealing martensitic transformation and $\alpha$/$\beta$ interface evolution in electron beam melting three-dimensional-printed Ti-6Al-4V.  Scientific reports, 6(1):26039, 2016.
  * [20] V. Tong, S. Joseph, A. K. Ackerman, D. Dye, and T. B. Britton.  Using transmission Kikuchi diffraction to characterise $\alpha$ variants in an $\alpha$\+ $\beta$ titanium alloy.  Journal of microscopy, 267(3):318–329, 2017.
  * [21] S.-P. Tsai, P. J. Konijnenberg, I. Gonzalez, S. Hartke, T. A. Griffiths, M. Herbig, K. Kawano-Miyata, A. Taniyama, N. Sano, and S. Zaefferer.  Development of a new, fully automated system for electron backscatter diffraction (EBSD)-based large volume three-dimensional microstructure mapping using serial sectioning by mechanical polishing, and its application to the analysis of special boundaries in 316L stainless steel.  Review of Scientific Instruments, 93(9), 2022.


Date of publication 11 January 2023, date of current version 11 January 2023.
10.1109/ACCESS.2023.3236087

† Equal contribution.
This work was supported by the Institute for Information & communications
Technology Promotion (IITP) grant funded by the Korea government (MSIT) (No.
2021-0-01381, Development of Causal AI through Video Understanding and
Reinforcement Learning, and Its Applications to Real Environments and
No.2022-0-00184, Development and Study of AI Technologies to Inexpensively
Conform to Evolving Policy on Ethics).

Corresponding author: Chang D. Yoo (e-mail: cd_yoo@kaist.ac.kr).

# DimCL: Dimensional Contrastive Learning For Improving Self-Supervised
Learning

THANH NGUYEN $\mathbf{\dagger}$1  TRUNG PHAM$\mathbf{\dagger}$1  CHAONING
ZHANG1  TUNG LUU1  THANG VU1 and CHANG D. YOO 1  School of Electrical
Engineering, Korea Advanced Institute of Science and Technology, Daejeon
34141, Republic of Korea

###### Abstract

Self-supervised learning (SSL) has gained remarkable success, for which
contrastive learning (CL) plays a key role. However, the recent development of
new non-CL frameworks has achieved comparable or better performance with high
improvement potential, prompting researchers to enhance these frameworks
further. Assimilating CL into non-CL frameworks has been thought to be
beneficial, but empirical evidence indicates no visible improvements. In view
of that, this paper proposes a strategy of performing CL along the dimensional
direction instead of along the batch direction as done in conventional
contrastive learning, named Dimensional Contrastive Learning (DimCL). DimCL
aims to enhance the feature diversity, and it can serve as a regularizer to
prior SSL frameworks. DimCL has been found to be effective, and the hardness-
aware property is identified as a critical reason for its success. Extensive
experimental results reveal that assimilating DimCL into SSL frameworks leads
to performance improvement by a non-trivial margin on various datasets and
backbone architectures.

###### Index Terms:

Self-supervise learning, Computer Vision, Contrastive Learning, Deep Learning,
Transfer Learning.

=-15pt

##  I Introduction

The success of self-supervised learning (SSL) has been demonstrated in a wide
range of applications, ranging from early attempts in natural language
processing [40, 51, 19, 55, 48] to more recent computer vision tasks [42, 13,
21]. To be more specific, in contrast to supervised learning which requires a
huge amount of labeled data [18, 73, 63, 63], SSL learns the representations
without the need for labeled ones. Thus, it significantly reduces the human-
label cost and enables machine learning to learn from a massive amount of
available unlabeled data leading to benefits for many real-world applications
in various fields: teaching robots to work from raw pixel images [41, 47, 45,
44], training medical diagnosis systems from un-labels checkups results [37,
8], enhance 3D face reconstruction using images in the wild [60], …

Without using human annotation labels, SSL methods seek to learn an encoder
with augmentation-invariant representation [3, 29, 9, 7, 25]. A common
approach is to minimize the distance between two representations of positive
samples, i.e. two augmented views of the same image. Based on this simple
approach, the past few years have witnessed the development of various SSL
frameworks, which can be roughly divided into two categories: CL-based and
non-CL frameworks. The CL-based frameworks [50, 33, 70, 75, 3, 31, 58, 9, 29,
65, 68] have achieved remarkable developments and greatly contributed to the
progress of SSL. Recently, multiple works [12, 25, 22, 71, 5] have also
demonstrated successful attempts with the non-CL frameworks, among which BYOL
[12] and SimSiam [9] are the two representatives.

Figure 1: Dimensional contrastive learning (DimCL). As the term suggests,
existing BCL performs CL along the batch direction to encourage diversity of
representations, while our proposed DimCL performs CL along the dimensional
direction to encourage diversity among elements within a representation
(termed feature diversity). Our DimCL can be used as a plug-and-play
regularization method to improve non-CL (and CL-based) SSL frameworks.

Compared with the CL-based frameworks, the non-CL ones [12, 9] have a unique
advantage: they propose simpler frameworks without using the negative samples,
yet achieve comparable or even superior performance on benchmark datasets
(like ImageNet-1K and CIFAR-10/100). Thus, there is a trend to shift from CL
to non-CL frameworks. Recognizing the significance of CL in the development of
SSL, this work attempts to distill beneficial properties of CL to push the
frontiers of non-CL frameworks further. However, naively assimilating CL to
non-CL does not show visible improvement, as pointed out in BYOL [25]. This
can be attributed to the fact that the frameworks mentioned above focus on the
same inter-instance level of constraints and mainly pursue the same objective
(augmentation invariant). In essence, existing CL encourages representation
diversity among the instances in the batch. In this paper, CL is utilized to
encourage diversity among the representation elements in obtaining “feature
diversity”, referred to as Dimensional Contrastive Learning. To avoid any
confusion between batch contrastive learning and dimensional contrastive
learning, we denote each as BCL and DimCL, respectively. The difference
between BCL and DimCL is depicted in Fig. 2.

A prudent variation in BCL led to a separate SSL framework, while the proposed
DimCL (as illustrated in Fig. 1) is designed as a regularizer for feature
diversity enhancement to support other frameworks. Even though DimCL is
originally motivated to boost non-CL frameworks, empirically, DimCL is found
to also enhance the performance of existing CL-based frameworks and can be
generalized to other domains (e.g,. supervised learning). This implies that
feature diversity is necessary for good representations.

Our contributions are as follows:

  * •

Recognizing the significance of CL in the development of self-supervised
learning, we are the first to apply DimCL to push the frontiers of non-CL
frameworks. In contrast to existing BCL, our proposed DimCL performs CL along
the dimensional direction and can be used as a regularizer for boosting the
performance of non-CL (and CL-based) frameworks.

  * •

We perform extensive experiments on various frameworks with different backbone
architectures on diverse datasets to validate the effectiveness of our
proposed DimCL. We also investigate the reason for the benefit brought by
DimCL and identify the hardness-aware property as an essential factor.

The rest of this paper is organized as follows. Section II summarizes the
related works. Section III describes the background of Batch Contrastive
Learning. Section IV presents the proposed method DimCL. Section V provides
the experiment setup and results. Section VI shows the ablation study on
important hyper-parameters. Section VII provides some discussions about DimCL.
Finally, Section VII concludes this work.

##  II Related Work

Contrastive Learning. Contrastive learning (CL) is one of the prominent
keystones of self-supervised learning. It fosters discriminability in the
representation [53, 67, 54, 46, 23]. Early works have studied margin-based
contrastive losses [27, 67, 32]. After the advent of [70, 50], NCE-based loss
has become the standard loss in CL. Inspired by this success, CL has been
extensively studied for SSL pretext training [70, 50, 3, 31, 33, 57, 75, 9].
SimCLR [9] proposes a simple yet effective method to train the unsupervised
model. They show that more negative samples (4096, for instance) are
beneficial for performance. However, such a massive number of negative samples
require a huge batch size for training to achieve the desired performance.

MoCo v1 [29] has attracted significant attention by demonstrating superior
performance over supervised pre-training counterparts in downstream tasks
while making use of large negative samples, decoupling the need for batch size
by introducing a dynamic dictionary. Inspired by [9], MoCo v2 [10] applies
stronger augmentations and an additional MLP projector, which shows
significant performance improvement over the first version of MoCo. [14] has
empirically shown that the predictor from the non-CL frameworks [12, 25] helps
to gain performance boost for MoCo variants with ViT structures [20].

Several works explain the key properties that lead to the success of CL. It is
noticeable that momentum update [9] and large negative samples play an
important role in preventing collapse. InfoNCE loss was identified to have the
hardness-aware property, which is critical for optimization [64] and
preventing collapse by instance de-correlation [1]. [49, 62, 66, 36, 15, 34,
69] have demonstrated that hard negative samples mining strategies can be
beneficial for better performance over the baselines. Notably, [65] identified
CL form alignment and uniformity of feature space which benefits downstream
tasks.

Most of the contrastive learning frameworks adopt the instance discrimination
task which inevitably causes class collision problems [74] where the
representations of the same class images are forced to be different. The
problem can hurt the quality of the learned representation. Different from the
above methods, which perform CL along the batch direction, DimCL performs the
CL along the dimensional direction in order to encourage diversity among
representation elements instead of representation vectors. This approach never
faces class collision problems.

Non-Contrastive Learning. Non-contrastive learning focuses on making
augmentation invariant without using negative samples. With the absence of
negative samples, training the simple siamese network using the cosine
similarity loss leads to complete collapse [1, 25]. BYOL [25] and Simsiam [12]
demonstrated that using a careful architecture design to break the
architecture symmetry can avoid collapse. Specifically, a special ‘predictor’
network is added in conjunction with the exponential moving average update
(BYOL) or with a stop gradient in one branch (Simsiam). Besides, several works
have attempted to demystify the success of BYOL [25]. A recent work [24] has
suggested that batch normalization (BN) plays a critical role in the success
of BYOL; however, another work [52] refutes that claim by showing BYOL works
without the need for BN.

Recognizing the strong points of CL in the development process, this work
tries to distill beneficial properties of CL in a novel manner and use it as a
regularizer to boost the performance of non-CL (and CL) based frameworks.
Moreover, most non-CL frameworks aim to learn augmentation invariant
representation which training often leads to trivial constant solutions (i.e.,
collapse) [6]. DimCL naturally avoids collapse as it encourages diversity in
the solution which is a great complement to non-CL.

##  III Background

Figure 2:  The difference between (a) Batch Contrastive Learning (BCL) and (b)
Dimensional Contrastive Learning (DimCL) . BCL performs along the batch
direction to encourage representation diversity whereas DimCL performs along
the dimensional direction to encourage feature diversity. N is the batch size,
and D is the feature dimension.

Conventional contrastive learning, i.e. BCL, aims to make the representations
similar if they are from different augmented versions of the same image and
dissimilar if they are from different images. Or shortly, it aims to make
meaningful discriminative representations. To be more specific, in BCL, there
are query, positive, and negative samples. The considered image is called the
query sample. The augmentation views of the query image are called positive
samples. The other images in the sample batch and their augmentation views are
called negative samples. The loss of CL-based frameworks basically makes the
query representation to be near the positive sample presentation and far apart
from the negative sample representation. Mathematically, given an encoder $f$,
an input image is augmented and encoded as a query $q\in\mathbb{R}^{D}$ or
positive key $k^{+}\in\mathbb{R}^{D}$, which are often $l_{2}$-normalized to
avoid scale ambiguity [28, 9]. Processing a mini-batch of N images will form a
set of queries $\mathbb{Q}=\\{q_{1},q_{2},...,q_{N}\\}$ and positive keys
$\mathbb{K}^{+}=\\{k_{1}^{+},k_{2}^{+},...,k_{N}^{+}\\}$. Consider a query
$q_{i}$, the corresponding negative keys are defined
$\mathbb{K}^{-}_{i}=\mathbb{Q}\cup\mathbb{K}^{+}\setminus\\{q_{i},k^{+}_{i}\\}=\\{k_{1}^{-},k_{2}^{-},...,k_{2N-2}^{-}\\}$
[9]. With similarity measured by dot product, BCL can be achieved by the
simple CL loss below [64]:

$\displaystyle\mathcal{L}$
$\displaystyle=\frac{1}{N}\sum_{i=1}^{N}\mathcal{L}_{i}$ (1)
$\displaystyle\mathcal{L}_{i}$
$\displaystyle=-q_{i}{\cdot}k_{i}^{+}+\frac{1}{2N-2}\sum_{j=1}^{2N-2}q_{i}{\cdot}k_{j}^{-}.$

The gradient of $\mathcal{L}_{i}$ w.r.t $q_{i}$ is derived as:

$\frac{\partial\mathcal{L}_{i}}{\partial
q_{i}}=-k_{i}^{+}+\frac{1}{2N-2}\sum_{j=1}^{2N-2}k_{j}^{-}.$ (2)

The above equation treats all negative keys equally. Based on this, [64]
proved that the superficial loss in Eq. 1 performs poorly in practice.

The NCE-based loss [26, 50] has been independently developed with various
motivations in multiple popular works [54, 70] and it has become the standard
loss for BCL. Following [50, 29, 1], we term it InfoNCE for consistency. The
InfoNCE is formulated as follows [28]:

$\displaystyle\mathcal{L}^{BCL}$
$\displaystyle=\frac{1}{N}\sum_{i=1}^{N}\mathcal{L}^{BCL}_{i}$ (3)
$\displaystyle\mathcal{L}_{{i}}^{BCL}$
$\displaystyle=-\log\frac{\exp(q_{i}{\cdot}k_{i}^{+}/\tau)}{\exp(q_{i}{\cdot}k_{i}^{+}/\tau)+\sum_{j=1}^{2N-2}\exp(q_{i}{\cdot}k_{j}^{-}/\tau)},$

with $\tau$ denoting the temperature. The InfoNCE has been identified to
outperform the above simple loss Eq. 1 due to its hardness-aware property,
which puts more weight on optimizing hard negative pairs (where the query is
close to negative keys) as shown in [64].

##  IV Methodology

Dimensional Contrastive Learning (DimCL) explores a new way of using InfoNCE
compared to BCL. As shown in Fig. 2, BCL aims to make meaningful
discriminative representations by applying InfoNCE along the batch direction.
The keys and queries are the representation vectors. By contrast, DimCL
encourages each representation element to contain a piece of distinct
information to maximize the amount of information contained in the overall
representation, which is feature diversity enhancement 111Note that feature
diversity is defined as the independence among the elements of a
representation. It should not be related to diversity among representation
vectors. To this end, the DimCL make the elements of the representation vector
orthogonal to each other in term of information by minimizing the empirical
correlation among column vectors. A novel form of InfoNCE along the
dimensional direction is proposed as the loss to achieve this objective.
Therein, the corresponding queries and keys are column vectors, each of which
is formed from the same-index representation elements within a batch, as
highlighted in Fig. 2.

Mathematically, similar to BCL, given a mini-batch of N images, we have a set
of queries $\mathbb{G}=\\{g_{1},g_{2},...,g_{D}\\}$ and positive keys
$\mathbb{H}^{+}=\\{h_{1}^{+},h_{2}^{+},...,h_{D}^{+}\\}$. Note that
$g,h\in\mathbb{R}^{N}$ are column vectors. Considering a query $g_{i}$, the
corresponding negative keys are defined as
$\mathbb{H}^{-}_{i}=\mathbb{G}\cup\mathbb{H}^{+}\setminus\\{g_{i},h^{+}_{i}\\}=\\{h_{1}^{-},h_{2}^{-},...,h_{2D-2}^{-}\\}$.
In order to maximize the feature diversity, the considered query $g_{i}$
should be orthogonal with all negative keys $\mathbb{H}^{-}_{i}$. The
corresponding objective is:

$\displaystyle\mathcal{L}^{AbsCL}$
$\displaystyle=\frac{1}{D}\sum_{i=1}^{D}\mathcal{L}_{i}^{AbsCL}$ (4)
$\displaystyle\mathcal{L}_{{i}}^{AbsCL}$
$\displaystyle=-\log\frac{\exp(g_{i}{\cdot}h_{i}^{+}/\tau)}{\exp(g_{i}{\cdot}h_{i}^{+}/\tau)+\sum_{j=1}^{2D-2}\exp(|g_{i}{\cdot}h_{j}^{-}|/\tau)}.$

Empirically, we observe that the original InfoNCE is sufficient to achieve the
objective without any modification (e.g., adding the absolute) (evidence is
provided in the discussion). This can be explained by considering the $\exp$
term and the effect of temperature $\tau$. With small $\tau$, the
$\exp(x/\tau)$ has high weight on pushing positive value x toward zero with a
corresponding high gradient but has almost no consideration on negative value
x with the same magnitude due to its much smaller gradient. For simplicity, we
adopt the following loss as the DimCL optimization target:

$\displaystyle\mathcal{L}^{DimCL}$
$\displaystyle=\frac{1}{D}\sum_{i=1}^{D}\mathcal{L}_{i}^{DimCL}$ (5)
$\displaystyle\mathcal{L}_{{i}}^{DimCL}$
$\displaystyle=-\log\frac{\exp(g_{i}{\cdot}h_{i}^{+}/\tau)}{\exp(g_{i}{\cdot}h_{i}^{+}/\tau)+\sum_{j=1}^{2D-2}\exp(g_{i}{\cdot}h_{j}^{-}/\tau)}.$

Note that, in DimCL each query $g_{i}$ has total $2D-2$ negative keys instead
of $2N-2$ as in BCL. And each of collumn vector $g,h$ are $l_{2}$-normalized
along the batch direction instead of dimensional direction as in BCL.
Furthermore, the proposed DimCL inherits the hardness-aware property of the
traditional BCL for which we provide more detail in the discussion part.

Contrary to BCL, which works as an independent SSL framework, DimCL serves as
a regularizer to benefit existing SSL frameworks. We denote
$\mathcal{L}^{BASE}$ as the loss of the SSL baseline. DimCL can be simply
assimilated into the baseline by a linear combination to form a final loss as:

$\mathcal{L}=\lambda\mathcal{L}^{DimCL}+(1-\lambda)\mathcal{L}^{BASE},$ (6)

where $\lambda\in[0,1]$ is a weight factor to balance the two loss components.
We perform a grid search and find that $\lambda=0.1$ works well in most cases
and recommend this value as a starting point for more fine-grained tuning. The
pseudo algorithm is provided in Algorithm. 1

Algorithm 1 Pytorch-style pseudocode for DimCL

$f$: encoder network, $f^{\prime}$: target encoder

$N$: batch size, $D$: dimension

$\tau$: temperature, $\lambda$: balance weight

$L_{base}$: baseline loss, $Optim$: optimizer

for $x$ in loader($N$) do $\triangleright$ Load batches with N samples

$y_{a},y_{b}=augment(x)$ $\triangleright$ Augmentations of x $\triangleright$
Compute representations

$z_{a}=f(y_{a})$ $\triangleright$ N*D

$z_{b}=f^{\prime}(y_{b})$ $\triangleright$ N*D $\triangleright$ Get queries
and positives

$G=\left[z_{a}[:,i]\text{ for i {in} range($D$)}\right]$

$H^{+}=\left[z_{b}[:,i]\text{ for i {in} range($D$)}\right]$

$L_{dimCL}=0$

for $i$ in range($D$) do

$H^{-}=G\cup H^{+}\setminus\\{G[i],H[i]\\}$ $\triangleright$ 2D-2 elements

$L_{dimCL}=L_{dimCL}+\mathcal{L}_{{i}}^{DimCL}$ $\triangleright$ Equation 5

end for

$L_{dimCL}=L_{dimCL}/D$

$Loss=\lambda L_{dimCL}+(1-\lambda)L_{base}(z_{a},z_{b})$ $\triangleright$
Optimization step

$Loss$.backward()

$Optim$.step()

end for

##  V Experiments

Datasets | Method | Type | ResNet-18 | ResNet-50
---|---|---|---|---
BASE | \+ DimCL | $\Delta_{acc}$ | BASE | \+ DimCL | $\Delta_{acc}$
CIFAR-10 |  MoCo v2 [28]* | CL | 89.55 | 89.59 | \+ 0.04 | 90.60 | 91.12 | \+ 0.60
SimCLR [9] | 86.01 | 88.32 | \+ 2.31 | 86.71 | 89.67 | \+ 2.96
BYOL [25] | Non-CL | 88.51 | 90.57 | \+ 2.06 | 88.0 | 89.98 | \+ 1.98
SimSiam [12] | 83.33 | 88.22 | \+ 4.89 | 84.60 | 89.33 | \+ 4.73
CIFAR-100 |  MoCo v2 [28]* | CL | 62.79 | 64.04 | \+ 1.25 | 64.68 | 66.24 | \+ 1.56
SimCLR [9] | 58.21 | 61.75 | \+ 3.54 | 60.81 | 65.27 | \+ 4.46
BYOL [25] | Non-CL | 62.36 | 67.85 | \+ 5.49 | 64.71 | 70.94 | \+ 6.23
SimSiam [12] | 51.67 | 62.49 | \+ 10.82 | 54.00 | 65.40 | \+ 11.4
STL-10 |  MoCo v2 [28]* | CL | 85.96 | 86.34 | \+ 0.38 | 88.16 | 88.40 | \+ 0.24
SimCLR [9] | 82.35 | 82.73 | \+ 0.48 | 84.33 | 86.28 | \+ 1.95
BYOL [25] | Non-CL | 83.36 | 84.94 | \+ 1.58 | 83.83 | 87.89 | \+ 4.05
SimSiam [12] | 84.24 | 84.35 | \+ 0.11 | 86.13 | 87.14 | \+ 1.01
ImageNet-100 |  MoCo v2 [28]* | CL | 76.02 | 78.38 | \+ 2.36 | 82.36 | 83.18 | \+ 0.82
SimCLR [9] | 75.96 | 76.52 | \+ 0.56 | 80.86 | 81.78 | \+ 1.12
BYOL [25] | Non-CL | 77.30 | 80.72 | \+ 3.42 | 81.74 | 84.80 | \+ 3.06
SimSiam [12] | 70.64 | 76.08 | \+ 5.42 | 72.98 | 80.20 | \+ 7.22

TABLE I:  The top-1 classification test accuracy (%) of the BASEs (the
baseline frameworks) + DimCL (the baseline with DimCL regularization) amongst
various datasets, and backbones. All models are trained for 200 epochs
Classification is performed with a linear classifier trained on top of the
frozen pre-trained encoder (output of the evaluated framework). “*” denotes an
improved version of MoCo v2 with symmetric loss.

###  V-A Experiment Setup

To show its effectiveness, we evaluate DimCL by assimilating it to state-of-
the-art non-CL and CL-based frameworks. Five widely used benchmark datasets
are considered including CIFAR-10 [38], CIFAR-100 [38], STL-10 [16],
ImagetNet-100 [57], and ImageNet-1K (1000 classes) [39]. Different encoders
(ResNet-18, ResNet-50) are also considered. The performance is bench-marked
with linear classification evaluation and transfer learning with object
detection following the common evaluation protocol in [25, 29, 12]. To be more
specific, the encoder is pre-trained in an unsupervised manner on the training
set of the selected dataset without labels [39]. For the linear classification
evaluation, the pre-trained frozen encoder is evaluated by training an
additional linear classifier and tested on the corresponding test set. For
object detection evaluation, the pre-trained frozen encoder is evaluated by a
Faster R-CNN detector (C4-backbone) with the object detection datasets (i.e.,
VOC object detection). In this paper, the Faster R-CNN detector (C4-backbone)
is finetuned on the VOC train-val 07+12 set with standard 2x schedule and
tested on the VOC test2007 set [68, 29]. More details regarding the two
evaluation methods are provided in Appendix

###  V-B Implementation Details

For a simple implementation, DimCL directly uses the InfoNCE loss [9] but
transposes the input. BCL framework implementations are based on the open
library solo-learn [61]. Setups of the SSL baseline framework for training are
described below.

Image augmentations. The paper follows the setting in previous approach [25,
9]. Concretely, a patch of the image is sampled and resized to 224 × 224.
Random horizontal flips and color distortion are applied in turn. The color
distortion is a random sequence of saturation, contrast, brightness, hue
adjustments, and an optional grayscale conversion. Gaussian blur and
solarization are applied to the patches at the final.

Training. We use stochastic gradient descent (SGD) as the optimizer. The SGD
weight decay is set to 1e-5, and the SGD momentum is 0.9 as BYOL [25]. We use
a batch size of 256, and a single GPU for all methods except in benchmark
ImageNet-1K for which a mini-batch size of $64\times 8$ to train on an 8-GPUs
machine (NVIDIA Titan Xp) is used. As a standard practice, the learning rate
is decayed using the cosine scheduler with ten epochs warm-up at the beginning
[43]. For baselines, we use the optimal set of hyperparameters tuned by [61].
We re-train all baselines in the same environment for a fair comparison. The
balance weight factor $\lambda$ is set to 0.1. The temperature $\tau$ is set
to 0.1 for all experiments.

###  V-C Experimental Results

This session reports the results with four settings to prove the efficacy of
DimCL: (1) Compatibility and generalization (experiments are performed with
200 epochs across non-CL (and CL) frameworks, datasets, and backbones) (2)
Large-scale dataset (experiments are performed with 100 epochs on ImageNet-1K)
(3) Longer Training (experiments are conducted with 1000 epochs on CIFAR-100,
ImageNet-100), and (4) Transfer Learning on Object Detection.

####  V-C1 Compatibility and generalization

To show the compatibility with various SSL frameworks and generalization
across datasets and backbones, we provide the extensive result as shown in
Tab. I. The results demonstrate that assimilating DimCL consistently improves
the performance by a large margin for all frameworks (MoCo v2, SimCLR, BYOL,
SimSiam), datasets (CIFAR-10, CIFAR-100, STL-10, ImageNet-100), and backbones
(ResNet-18, ResNet-50).

For example, on CIFAR-100 with Resnet-50, DimCL enhances the baseline MoCo v2
and SimCLR with a performance boost of 1.56% and +4.46%, respectively. A more
significant performance boost can be observed for BYOL (+6.63%), and SimSiam
(+11.4%). In addition, during the experiment, the BASEs are highly tuned to
get the best performance, and BASEs+DimCL does not. With a fine-tuned
parameter search, a higher gain might be possible. Overall, the result
indicates DimCL is compatible with both CL and non-CL SSL frameworks with a
non-trivial performance gain. Furthermore, it also has good generalization
across various datasets and backbones.

When evaluating the performance of DimCL under different metrics, the result
suggests the same conclusion. To be more specific, an experiment is conducted
on CIFAR100 with Resnet-18 backbone. The pre-trained models of the baselines
and DimCL are evaluated on the classification task with different performance
metrics: Top-1 Accuracy, Top-5 Accuracy, Top-1 KNN, and Top-5 KNN. The result,
shown in Tab. II, suggests that DimCL consistently improves the baseline under
various performance metrics.

Metrics | Top-1 Acc | Top-5 Acc | Top-1 KNN | Top-5 KNN
---|---|---|---|---
Methods | BASE | +DimCL | BASE | +DimCL | BASE | +DimCL | BASE | +DimCL
MoCo V2 | 62.79 | 64.04 | 88.75 | 89.3 | 57.16 | 57.82 | 80.81 | 81.61
SimCLR | 58.21 | 61.75 | 84.97 | 87.72 | 51.7 | 54.85 | 76.61 | 77.82
BYOL | 62.36 | 67.85 | 88.51 | 90.87 | 56.2 | 59.91 | 80.27 | 81.94
SimSiam | 51.67 | 62.49 | 81.65 | 88.17 | 50.11 | 55.12 | 76.19 | 79.65
TABLE II: Performance evaluated with different metrics. The methods are
trained on the CIFAR-100 dataset with 200 epochs and use Resnet-18 as the
backbone. Classification is performed with a linear classifier trained on top
of the frozen pre-trained encoder. The test accuracy is reported with various
performance metrics: Top-1 accuracy, Top-5 accuracy, Top-1 KNN, and Top-5 KNN.

####  V-C2 Large-scale dataset

Method | Top-1 (%) | Top-5 (%)
---|---|---
MoCo v2 [11]${\dagger}$ | 67.4 | -
BYOL [25]${\dagger}$ | 66.5 | -
BYOL (Reproduce) | 67.3 | 88.0
BYOL + DimCL | 69.3 | 89.0

TABLE III: Imagenet-1K classification. All frameworks are trained without
labels on the training set for 100 epochs. Evaluation is on a single crop
$224\times 224$. “${\dagger}$” denotes the results employed from [12].

For the large-scale dataset, Imagenet-1K is chosen, and BYOL is selected as
the baseline. Due to the resource constraint, BYOL and BYOL+DimCL are pre-
trained for 100 epochs without labels. The results are reported in Tab. III.
The results show that on the large-scale dataset, DimCL improves the BYOL
baseline with a performance boost of $+2.0\%$ and outperforms all other
frameworks. The performance is consistent with the results in Tab. I,
verifying the generalization and effectiveness of DimCL.

####  V-C3 Longer Training

(a) 200 epochs

(b) 1000 epochs

Figure 3:  Top-1 classification accuracy learning curve on the test set of
CIFAR-100 of (a) 200 epochs and (b) 1000 epochs. The figure shows the
consistent result between long and short training. Note that at the same
epoch, the Top-1 accuracy of two settings is not necessarily the same due to
using the cosine learning rate scheduler.

To demonstrate the results are consistent between short training (200 epochs)
and long training (1000 epochs), we conduct experiments on CIFAR-100 and
ImageNet-100 with BYOL as the baseline framework [25]. Top-1 classification
accuracies are reported in Tab. IV.

We observe that DimCL also has a consistent performance boost for the long
training. Specifically, incorporating DimCL helps to significantly boost the
top-1 accuracy of BYOL from 70.54% to 71.94% (+1.4%) for CIFAR-100, and
further improves BYOL from 81.24% to 82.51% for ImageNet-100. It is reasonable
that the performance boost margin can be relatively smaller in the setup of
the long training compared to the short training.

Fig. 3 shows the learning curve in two different settings: 200 epochs (a) and
1000 epochs (b). The results demonstrate that our method does not vanish but
further improves BYOL in long training. There is a high correlation in
performance improvement between short and long training. It proves that the
200 epochs setting is reasonably adequate to evaluate the performance gain.

Dataset | Epoch | SimCLR | MoCo v2 | BYOL | BYOL+DimCL
---|---|---|---|---|---
CIFAR-100 | 200 | 58.21 | 62.79 | 62.36 | 67.85
1000 | 65.85 | 69.39 | 70.54 | 71.94
ImageNet-100 | 200 | 75.96 | 76.02 | 77.30 | 80.72
1000 | 78.76 | 79.98 | 81.24 | 82.51

TABLE IV: Long training with 1000 epochs. Linear classification accuracy (%)
on the test set of CIFAR-100. All models are pre-trained on the training set
without labels before evaluation. Note that MoCo v2+ is the improved version
of MoCo v2 with symmetric loss [17].

####  V-C4 Transfer Learning on Object Detection

Method | Epoch | AP | AP50 | AP75
---|---|---|---|---
SimCLR [9] † | 200 | 51.5 | 79.4 | 55.6
BYOL [25] † | 200 | 51.9 | 81.0 | 56.5
BYOL [25] | 100 | 50.3 | 79.8 | 54.2
BYOL + DimCL | 100 | 55.6 | 81.9 | 61.4

TABLE V:  Transfer learning on detection task VOC07. The † denotes the results
from [68].

Tab. V shows Objection detection evaluation on pre-trained frozen backbone
ResNet-50 in Tab. III. DimCL significantly boosts BYOL in the object detection
task by a large margin. Specifically, in 100 epoch pre-training on ImageNet-1K
with BYOL [25], the encoder gives 50.3, 79.8, 54.2 in AP, AP50, AP75,
respectively. Doubling pre-training epochs with BYOL, i.e. 200 epochs, the
encoder shows a slight improvement. By contrast, in 100 epoch pre-training
with BYOL+DimCL, the encoder can strongly outperform BYOL for the AP, AP50,
AP75 with 55.6, 81.9, 61.4, and even surpass the performance of the baseline
200 epoch pre-training with BYOL for all metrics. It demonstrates the
effectiveness of the proposed DimCL. Overall, the results demonstrate DimCL is
effective for both downstream tasks: classification and object detection.

##  VI Ablation study

In this section, we provide ablation for important hyper-parameters of DimCL:
the temperature $\tau$, the weight factor $\lambda$, and the dimensionality
$D$.

###  VI-A The effect of the temperature $\tau$

(a)  Feature diversity

(b)  Top-1 linear accuracy

Figure 4:  Feature diversity (a) and performance (b) with respect to $\tau$ on
the test set of CIFAR-100. Our hypothesis emphasizes the importance of
increasing the feature diversity or decreasing the correlation to remove the
residual information of the feature representation.

We monitor the changes in feature diversity and performance when assimilating
DimCL to BYOL with various $\tau$ values. The experiment runs on CIFAR-100 for
200 epochs. The results in Fig. 4 suggest that selecting a reasonable $\tau$
leads to high feature diversity (and performance). $\tau=1$ does not lead to
good feature diversity. The value of $\tau$ that supports gaining the best
performance is around 0.1. This result coincides with the $\tau$ used in
conventional BCL frameworks [9, 29]. There is a drop in performance when using
too large or too small $\tau$ in DimCL.

###  VI-B The effect of the weight factor $\lambda$

(a) Top-1 linear accuracy

(b)  KNN accuracy

Figure 5:  Top-1 classification accuracy and top-1 KNN with respect to
$\lambda$ on the test set of CIFAR-100. Note that, the performance at
$\lambda=0$, corresponding to the performance of the baseline BYOL, is much
lower than the case when incorporating BYOL with our loss.

The balance weight factor between DimCL and the baseline plays an important
role in gaining performance. We conduct the experiments with a range of [0, 1]
for $\lambda$ in Eq. 6. All other parameters are kept unchanged.

The results in Fig. 5 show that with all $\lambda$ in the range of (0, 0.7),
our method consistently outperforms the baseline BYOL (corresponding to
$\lambda=0$) in both two measures: top-1 classification accuracy and top-1 KNN
accuracy. $\lambda=0.1$ is found to be the optimal value to boost performance
when plugging DimCL into BYOL. $\lambda$ usually depends on the baselines and
dataset. However, we empirically find that setting $\lambda$ to 0.1 often
gives the best performance for the most recent SSL frameworks in datasets. It
is recommended to use this value at the beginning of the tuning process when
using our DimCL regularization.

###  VI-C The effect of the dimensionality $D$

As the DimCL targets to address dimension-wise diversity, dimensionality
should be a key fact that needs to be considered. We provide ablation studies
on the effects of dimensionality. Tab. VI shows the top 1 accuracy on CIFAR
100 with 200 epochs of BYOL and BYOL+DimCL.

The result shows that for the small dimensionality, DimCL provides a large
improvement over the baseline. For bigger dimensionality, the improvement
tends to reduce. It is understandable since DimCL aims to maximize the useful
information (or in other words, minimize the redundancy) contained in a low
dimensionality. For bigger dimensionality, there is plenty of space for
storing information which reduces the importance of DimCL. It is also
noticeable that for very small dimensionality, the performance starts to drop
for both BYOL and BYOL+DimCL (e.g: under 256) since there is not much space
for storing information.

Dimensionality | 64 | 128 | 256 | 512 | 1024 | 2048 | 8192
---|---|---|---|---|---|---|---
BYOL | 59.85 | 60.72 | 62.36 | 62.62 | 62.44 | 62.02 | 62.99
BYOL+DimCL | 66.47 | 66.84 | 67.85 | 67.41 | 67.18 | 67.41 | 67.33
Improvement | 6.62 | 6.12 | 5.49 | 4.79 | 4.74 | 5.39 | 4.34

TABLE VI: The effects of DimCL on dimensionality. The table shows the top 1
accuracy on CIFAR 100 with 200 epochs of BYOL and BYOL+DimCL

##  VII discussion

Method | Feature Diversity | Accuracy Top-1 | Accuracy KNN
---|---|---|---
BASE | \+ DimCL | BASE | \+ DimCL | BASE | \+ DimCL
BYOL [25] | 0.83 |  0.88 (+0.05) | 62.36 |  67.85 (+5.49) | 56.20 |  59.91 (+3.71)
SimSiam [12] | 0.75 |  0.92 (+0.17) | 51.67 |  62.49 (+10.82) | 50.11 |  55.12 (+5.01 )

TABLE VII:  Comparison of Feature diversity and performance in CIFAR-100
dataset for both BASE (baseline) and +DimCL (baseline with DimCL
regularization). All frameworks are pre-trained with 200 epochs on ResNet-18
backbone.

###  VII-A Feature Diversity Enhancement

Our proposed DimCL is motivated to enhance feature diversity which is defined
as the independence among the elements of a representation. In other words,
good feature diversity means each element of representation should carry a
piece of distinct information about the input image. In this view, feature
diversity can be evaluated by considering correlation among all pairs of
negative column vectors. Given a tensor with size $N\times D$, the feature
diversity measure is defined as:

$\textit{feature diversity}=1-\frac{1}{D(D-1)}\sum_{i}^{D}\sum_{j\neq
i}^{D}|sim(g_{i}{\cdot}h_{j})|.$ (7)

(a)  Performance

(b)  Feature diversity

Figure 6: Relation between feature diversity and performance during training
on CIFAR-100. a) the top-1 test classification accuracy. b) the corresponding
feature diversity. The higher feature diversity leads to higher performance.

Here, $g,h\in\mathbb{R}^{N}$ are column vectors. $sim(.)$ is the cosine
similarity measure. The range for the feature diversity measure is within
$[0,1]$. The optimum value of feature diversity is 1 which means all elements
of representation are mutually independent.

To prove the enhancement of feature diversity, we take BYOL [25] and SimSiam
[12] into account where the encoders are designed to learn the representation,
which is invariant to augmentation without considering feature diversity. We
assimilate DimCL to BYOL, and Simsiam then observe changes in feature
diversity and accuracy. Results are reported in Tab. VII.

Interestingly, the BASEs generate embedding, which already has high feature
diversity. Adding DimCL to BASEs has a strong effect on further increasing the
feature diversity, which impacts performance improvement. Specifically, DimCL
makes an improvement 0.05 (5% in percentage) feature diversity with
corresponding 5.49% accuracy on BYOL and 0.17 (17% in percentage) feature
diversity with corresponding 10.82% accuracy on SimSiam. The more feature
diversity improvement, the better performance gain. The relation between
feature diversity is shown clearly in Fig. 6.

From the perspective of information theory, improving feature diversity can be
classified as the Information Bottleneck objective [59] which forces a
representation that conserves as much information about the sample as
possible. It is mentioned to be beneficial in various research [2, 35, 59].
Our result is one of the empirical pieces of evidence proving the benefit of
feature diversity.

###  VII-B Hardness-Aware Property in DimCL

The hardness-aware property plays a key role in BCL controlling the
uniformity-tolerance dilemma [65] leading to its success. In the view of
optimization, the hardness-aware property puts more weight into optimizing
negative pairs that have high similarities. This way is influenced by hard
examples mining and has proven to be effective [49, 72, 62, 66, 4, 36].

The interpretation of Harness-aware in the DimCL can be understood via
gradient analysis of loss function. Let’s consider loss for query $g_{i}$:

$\mathcal{L}_{i}^{DimCL}=-\log\frac{\exp(g_{i}{\cdot}h^{+}_{i}/\tau)}{\exp(g_{i}{\cdot}h^{+}_{i}/\tau)+\sum_{j}\exp(g_{i}{\cdot}h^{-}_{j}/\tau)}$
(8)

where $g$ and $h$ are the $l_{2}$-normalized column vectors. The gradient of
$\mathcal{L}_{i}^{DimCL}$ w.r.t query $g_{i}$ is derived as:

$\displaystyle\frac{\partial\mathcal{L}_{i}^{DimCL}}{\partial g_{i}}=-$
$\displaystyle\frac{1}{\tau}(1-\frac{\exp(g_{i}{\cdot}h^{+}_{i}/\tau)}{\exp(g_{i}{\cdot}h^{+}_{i}/\tau)+\sum_{j}\exp(g_{i}{\cdot}h^{-}_{j}/\tau)}).h^{+}_{i}$
(9) $\displaystyle+$
$\displaystyle\frac{1}{\tau}\frac{\sum_{j}\exp(g_{i}{\cdot}h^{-}_{j}/\tau).h^{-}_{j}}{\exp(g_{i}{\cdot}h^{+}_{i}/\tau)+\sum_{j}\exp(g_{i}{\cdot}h^{-}_{j}/\tau)}$
(10) $\displaystyle=-$
$\displaystyle\frac{1}{\tau}(1-\alpha_{i}^{\prime}).h^{+}_{i}+\frac{1}{\tau}\sum_{j}\alpha_{j}.h^{-}_{j},$
(11)

where
$\alpha_{i}^{\prime}=\frac{\exp(g_{i}{\cdot}h^{+}_{i}/\tau)}{\exp(g_{i}{\cdot}h^{+}_{i}/\tau)+\sum_{j}\exp(g_{i}{\cdot}h^{-}_{j}/\tau)}$
can be interpreted as the probability of $g_{i}$ being recognized as the
positive colunm vector $h^{+}_{i}$. Similarly,
$\alpha_{j}=\frac{\exp(g_{i}{\cdot}h^{-}_{j}/\tau)}{\exp(g_{i}{\cdot}h^{+}_{i}/\tau)+\sum_{j}\exp(g_{i}{\cdot}h^{-}_{j}/\tau)}$
can be interpreted as the probability of $g_{i}$ being recognized as the
negative vector $h^{-}_{j}$. We can easily see that
$\alpha_{i}^{\prime}+\sum_{j}\alpha_{j}=1$ and all $\alpha>0$.

The Eq. 11 reveals how DimCL makes the query similar to the positive key and
dissimilar from negative keys. Concretely, if $g_{i}$ and $h^{+}_{i}$ are very
close, the gradient of $g_{i}$ is very small because
$1-\alpha_{i}^{\prime}\approx 0$ and $\sum_{j}\alpha_{j}\approx 0$ (because
$1-\alpha_{i}^{\prime}\approx 0$ and
$\alpha_{i}^{\prime}+\sum_{j}\alpha_{j}=1$) . Thus, the optimizer does not
update the query $g_{i}$. By contrast, if $g_{i}$ and $h^{-}_{j}$ are very
close, the weight $\alpha_{j}$ is big, encouraging the optimizer to push the
query far away from the corresponding negative keys.

Regarding the ability to differently treat negative keys, the gradient weight
w.r.t negative keys are proportional to the exponential
$\exp(\frac{g_{i}.h^{-}_{j}}{\tau})$. It shows that hard column pairs, where
query $g_{i}$ is far with negative keys, are penalized more with larger
$\alpha_{j}$. In other words, the optimizer will pay more attention to
optimizing hard column pairs, which leads to better optimization results than
treating them equally. This phenomenon is the hardness-awareness property of
the loss 8. The effect of the hardness-aware property in DimCL in relation to
feature diversity can be empirically seen clearly in the ablation study Fig. 4

###  VII-C Beyond CL and non-CL.

Datasets | ResNet-18 | ResNet-50
---|---|---
BT | \+ DimCL | BT | \+ DimCL
CIFAR-10 | 88.45 | 89.21 | 88.91 | 90.28
CIFAR-100 | 65.61 | 66.42 | 66.35 | 66.88
STL-10 | 82.26 | 82.66 | 84.99 | 85.34
IMAGENET-100 | 78.50 | 78.72 | 82.44 | 82.78

TABLE VIII:  DimCL for improving Barlow Twins (BT) [71]. Frameworks are trained for 200 epochs with ResNet-18 and ResNet-50 backbone on the 4 datasets. We report top-1 linear classification (%) accuracy.  Dataset | BYOL+DimCL | BYOL+Barlow Twins
---|---|---
CIFAR-100 | 67.85 | 66.55
TABLE IX:  Comparison between DimCL and Barlow Twins on top of baseline BYOL.
Models are trained for 200 epochs with ResNet-18 on the CIFAR-100. We report
top-1 linear classification (%) accuracy.

Previous results show that DimCL is most beneficial in boosting the
performance of CL and non-CL frameworks with a non-trivial margin. Here, we
also investigate the recent work that designed an explicit term for
decorrelation, Barlow Twins (BT) [71].

We experiment by adding the correlation-reduction loss of BT to the previous
baseline BYOL and comparing it against DimCL. The result in Tab. IX shows that
BYOL+DimCL strongly outperforms BYOL+Barlow. Furthermore, as shown in Tab.
VIII, When incorporate into BT, DimCL can also improve BT.

This empirical result recommends that DimCL provides better performance than
BT.

###  VII-D DimCL for supervised learning.

Since DimCL works as a regularizer enhancing the feature diversity, it is
expected to benefit other fields beyond self-supervised learning (e.g.
supervised learning (SL)). This experiment utilizes DimCL to boost SL on
CIFAR-100 and CIFAR-10 datasets. We use the solo-learn library [61] to train
the supervised model with backbone ResNet-18 [30].

Dataset | Supervised | \+ DimCL (%)
---|---|---
CIFAR-100 | 70.27 | 71.68
CIFAR-10 | 93.29 | 93.35
TABLE X:  DimCL for improving supervised learning. Models are trained for 200
epochs with ResNet-18 and ResNet-50 backbone on the 4 datasets. We report
top-1 linear classification (%) accuracy.

DimCL is assimilated with cross-entropy loss for training the model
simultaneously. Tab. X shows the top-1 classification accuracy on the test
set. For CIFAR-10 DimCL shows slight improvement, while CIFAR-100 shows the
DimCL supports to boost the conventional supervised learning from 70.27% to
71.68% ($+$1.4%), demonstrating the benefit of DimCL for SL.

###  VII-E DimCL versus AbsCL

In order to maximize the feature diversity, the considered query $g_{i}$
should be orthogonal with all negative keys $\mathbb{H}^{-}_{i}$. The
corresponding objective is:

$\displaystyle\mathcal{L}^{AbsCL}$
$\displaystyle=\frac{1}{D}\sum_{i=1}^{D}\mathcal{L}_{i}^{AbsCL}\quad$ (12)
$\displaystyle\mathcal{L}_{{i}}^{AbsCL}$
$\displaystyle=-\log\frac{\exp(g_{i}{\cdot}h_{i}^{+}/\tau)}{\exp(g_{i}{\cdot}h_{i}^{+}/\tau)+\sum_{j=1}^{2D-2}\exp(|g_{i}{\cdot}h_{j}^{-}|/\tau)}.$

Datasets | BYOL | SimSiam
---|---|---
Baseline | $\tau$ | DimCL | AbsCL | Baseline | $\tau$ | DimCL | AbsCL
CIFAR-10 | 88.51 | 1 | 88.92 | 89.81 | 83.33 | 1 | 86.07 | 86.59
0.1 | 90.57 | 90.83 | 0.1 | 88.22 | 87.67
CIFAR-100 | 62.34 | 1 | 62.77 | 66.02 | 51.67 | 1 | 54.67 | 57.56
0.1 | 67.85 | 67.87 | 0.1 | 62.49 | 62.96

TABLE XI: DimCL versus AbsCL. We report top-1 linear test accuracy (%) on
CIFAR-10 and CIFAR-100. All methods are trained for 200 epochs. For
$\tau=0.1$, all methods DimCL and AbsCL perform best and performance is almost
similar.

Empirically, Tab. XI shows that the original InfoNCE is sufficient to achieve
the objective without any modification (e.g., adding the absolute). It is
important to note that without $\tau$, DimCL and AbsCL can outperform the
baseline. However, to achieve the best performance, $\tau$ is needed to
present. At the optimal $\tau=0.1$, the performance of DimCL is nearly the
same as AbsCL. This phenomenon can be explained by considering the $\exp$ term
and the effect of temperature $\tau$. With small $\tau$, the $\exp(x/\tau)$
has a high weight on pushing positive value x toward zero with a corresponding
high gradient but has almost no consideration on negative value x with the
same magnitude due to its much smaller gradient.

###  VII-F Visualization of Representation.

Visualization of representation via t-SNE is reported to see the effect of DCL
on representation space. Fig. 7 and Fig. 8 show the representation of BYOL
baseline and our method on the 2D space. The experiment is conducted on
CIFAR-10 with 10 classes. The results clearly show that our method in Fig. 8
gives more separable representations. More specifically, airplane, auto, ship,
and truck are almost separable among them and also from other animal classes.
All classes are scatted in the more compact clusters compared to the baseline
in Fig. 7.

Figure 7:  t-SNE plot of ten classes for data trained by the BYOL baseline in
200 epochs with accuracy = 88.51% in CIFAR-10 with 10,000 samples of the test
set.

Figure 8:  t-SNE plot of ten classes for data trained by the BYOL + DimCL in 200 epochs with accuracy = 90.57% in CIFAR-10 with 10,000 samples of the test set. | Intra-class distance $\downarrow$ | Inter-class distance $\uparrow$
---|---|---
BYOL | 27.0 | 82.3
BYOL+DCL | 24.1 | 85.9
TABLE XII: Inter-class distance and Inter-class distance on CIFAR-10 test set.

To show the difference between the two representation spaces quantitatively,
intra-class distance and inter-class distance [56] are calculated and provided
in Tab. XII. The quantitative result agrees that BYOL+DCL forms the more
compact clusters while maintaining a higher separation among different
clusters compared to BYOL.

##  VIII Conclusion

This paper introduces Dimensional Contrastive Learning (DimCL), a new way of
applying CL. DimCL works as a regularization that can assimilate with non-CL
(and CL) based frameworks to boost performance on downstream tasks such as
classification and object detection. DimCL enhances feature diversity among
elements within a representation. DimCL has high compatibility and
generalization across datasets, frameworks, and backbone architectures. We
believe that feature diversity is a key indispensable ingredient for learning
representation. This paper focuses on images and provides mostly empirical
evidence but DimCL can be generalized to other modalities (e.g. audio, video,
text) and proven with theoretical results. We let it for future work.

## References

  * [1] Anonymous.  How does simsiam avoid collapse without negative samples? towards a unified understanding of progress in SSL.  In Submitted to The Tenth International Conference on Learning Representations, 2022.  under review.
  * [2] Babajide O Ayinde, Tamer Inanc, and Jacek M Zurada.  Regularizing deep neural networks by enhancing diversity in feature extraction.  IEEE transactions on neural networks and learning systems, 30(9):2650–2661, 2019.
  * [3] Philip Bachman, R Devon Hjelm, and William Buchwalter.  Learning representations by maximizing mutual information across views.  arXiv preprint arXiv:1906.00910, 2019.
  * [4] Yingbin Bai and Tongliang Liu.  Me-momentum: Extracting hard confident examples from noisily labeled data.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 9312–9321, October 2021.
  * [5] Adrien Bardes, Jean Ponce, and Yann LeCun.  Vicreg: Variance-invariance-covariance regularization for self-supervised learning.  arXiv preprint arXiv:2105.04906, 2021.
  * [6] Horace Barlow.  Redundancy reduction revisited.  Network: computation in neural systems, 12(3):241, 2001.
  * [7] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments.  arXiv preprint arXiv:2006.09882, 2020.
  * [8] Liang Chen, Paul Bentley, Kensaku Mori, Kazunari Misawa, Michitaka Fujiwara, and Daniel Rueckert.  Self-supervised learning for medical image analysis using image context restoration.  Medical image analysis, 58:101539, 2019.
  * [9] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In ICML, 2020.
  * [10] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [11] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning.  arXiv preprint arXiv:2003.04297, 2020.
  * [12] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In CVPR, 2021.
  * [13] Xinlei Chen, Saining Xie, and Kaiming He.  An empirical study of training self-supervised vision transformers.  ICCV, 2021.
  * [14] Xinlei Chen, Saining Xie, and Kaiming He.  An empirical study of training self-supervised vision transformers.  ICCV, 2021.
  * [15] Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka.  Debiased contrastive learning.  In NeurIPS, 2020.
  * [16] Adam Coates, Andrew Ng, and Honglak Lee.  An analysis of single-layer networks in unsupervised feature learning.  In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215–223. JMLR Workshop and Conference Proceedings, 2011.
  * [17] Victor G. Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci.  Solo-learn: A library of self-supervised methods for visual representation learning, 2021.
  * [18] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
  * [19] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [20] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In ICLR, 2021.
  * [21] Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al.  Xcit: Cross-covariance image transformers.  arXiv preprint arXiv:2106.09681, 2021.
  * [22] Aleksandr Ermolov, Aliaksandr Siarohin, Enver Sangineto, and Nicu Sebe.  Whitening for self-supervised representation learning.  In ICML. PMLR, 2021.
  * [23] Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata.  Learning robust representations via multi-view information bottleneck.  In International Conference on Learning Representations, 2020.
  * [24] Abe Fetterman and Josh Albrecht.  Understanding self-supervised and contrastive learning with ”bootstrap your own latent” (byol), 2020.
https://untitled-ai.github.io/
understanding-self-supervised-
contrastive-learning.html.

  * [25] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al.  Bootstrap your own latent: A new approach to self-supervised learning.  arXiv preprint arXiv:2006.07733, 2020.
  * [26] Michael Gutmann and Aapo Hyvärinen.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.  In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 297–304. JMLR Workshop and Conference Proceedings, 2010.
  * [27] Raia Hadsell, Sumit Chopra, and Yann LeCun.  Dimensionality reduction by learning an invariant mapping.  In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 2, pages 1735–1742. IEEE, 2006.
  * [28] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  arXiv preprint arXiv:1911.05722, 2019.
  * [29] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In CVPR, 2020.
  * [30] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [31] Olivier Henaff.  Data-efficient image recognition with contrastive predictive coding.  In International Conference on Machine Learning, pages 4182–4192. PMLR, 2020.
  * [32] Alexander Hermans, Lucas Beyer, and Bastian Leibe.  In defense of the triplet loss for person re-identification.  arXiv preprint arXiv:1703.07737, 2017.
  * [33] R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio.  Learning deep representations by mutual information estimation and maximization.  arXiv preprint arXiv:1808.06670, 2018.
  * [34] Chih-Hui Ho and Nuno Vasconcelos.  Contrastive learning with adversarial examples.  arXiv preprint arXiv:2010.12050, 2020.
  * [35] Tianyu Hua, Wenxiao Wang, Zihui Xue, Sucheng Ren, Yue Wang, and Hang Zhao.  On feature decorrelation in self-supervised learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9598–9608, 2021.
  * [36] Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondřej Chum.  Mining on manifolds: Metric learning without labels.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7642–7651, 2018.
  * [37] Rayan Krishnan, Pranav Rajpurkar, and Eric J Topol.  Self-supervised learning in medicine and healthcare.  Nature Biomedical Engineering, pages 1–7, 2022.
  * [38] Alex Krizhevsky et al.  Learning multiple layers of features from tiny images.  ., 2009.
  * [39] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  ImageNet classification with deep convolutional neural networks.  In NeurIPS, 2012.
  * [40] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut.  Albert: A lite bert for self-supervised learning of language representations.  In International Conference on Learning Representations, 2020.
  * [41] Michael Laskin, Aravind Srinivas, and Pieter Abbeel.  Curl: Contrastive unsupervised representations for reinforcement learning.  In International Conference on Machine Learning, pages 5639–5650. PMLR, 2020.
  * [42] Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao.  Efficient self-supervised vision transformers for representation learning.  arXiv preprint arXiv:2106.09785, 2021.
  * [43] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  In ICVLR, 2017.
  * [44] Tung M Luu, Thanh Nguyen, Thang Vu, and Chang D Yoo.  Utilizing skipped frames in action repeats for improving sample efficiency in reinforcement learning.  IEEE Access, 2022.
  * [45] Tung M Luu, Thang Vu, Thanh Nguyen, and Chang D Yoo.  Visual pretraining via contrastive predictive model for pixel-based reinforcement learning.  Sensors, 22(17):6504, 2022.
  * [46] Ishan Misra, C Lawrence Zitnick, and Martial Hebert.  Shuffle and learn: unsupervised learning using temporal order verification.  In European Conference on Computer Vision, pages 527–544. Springer, 2016.
  * [47] Thanh Nguyen, Tung M Luu, Thang Vu, and Chang D Yoo.  Sample-efficient reinforcement learning representation learning with curiosity contrastive forward dynamics model.  In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3471–3477. IEEE, 2021.
  * [48] Ping Nie, Yuyu Zhang, Xiubo Geng, Arun Ramamurthy, Le Song, and Daxin Jiang.  Dc-bert: Decoupling question and document for efficient contextual encoding.  In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1829–1832, 2020.
  * [49] Kento Nozawa and Issei Sato.  Understanding negative samples in instance discriminative self-supervised representation learning.  arXiv preprint arXiv:2102.06866, 2021.
  * [50] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748, 2018.
  * [51] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  OpenAI blog, 2019.
  * [52] Pierre H Richemond, Jean-Bastien Grill, Florent Altché, Corentin Tallec, Florian Strub, Andrew Brock, Samuel Smith, Soham De, Razvan Pascanu, Bilal Piot, et al.  Byol works even without batch statistics.  arXiv preprint arXiv:2010.10241, 2020.
  * [53] Florian Schroff, Dmitry Kalenichenko, and James Philbin.  Facenet: A unified embedding for face recognition and clustering.  2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 815–823, 2015.
  * [54] Kihyuk Sohn.  Improved deep metric learning with multi-class n-pair loss objective.  In NeurIPS, 2016.
  * [55] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai.  {VL}-{bert}: Pre-training of generic visual-linguistic representations.  In International Conference on Learning Representations, 2020.
  * [56] Afirah Taufik and Sharifah Sakinah Syed Ahmad.  A comparative study of fuzzy c-means and k-means clustering techniques.  In Malays. Tech. Univ. Conf. Eng. Technol. 8th MUCET, volume 1, pages 10–11, 2014.
  * [57] Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive multiview coding.  arXiv preprint arXiv:1906.05849, 2019.
  * [58] Yonglong Tian, Dilip Krishnan, and Phillip Isola.  Contrastive multiview coding.  In ECCV 2020, pages 776–794. Springer, 2020.
  * [59] Naftali Tishby and Noga Zaslavsky.  Deep learning and the information bottleneck principle.  In 2015 IEEE Information Theory Workshop (ITW), pages 1–5. IEEE, 2015.
  * [60] Xiaoguang Tu, Jian Zhao, Mei Xie, Zihang Jiang, Akshaya Balamurugan, Yao Luo, Yang Zhao, Lingxiao He, Zheng Ma, and Jiashi Feng.  3d face reconstruction from a single image assisted by 2d face images in the wild.  IEEE Transactions on Multimedia, 23:1160–1172, 2020.
  * [61] Victor G Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci.  Solo-learn: A library of self-supervised methods for visual representation learning.  arXiv e-prints, pages arXiv–2108, 2021.
  * [62] Bhavya Vasudeva, Puneesh Deora, Saumik Bhattacharya, Umapada Pal, and Sukalpa Chanda.  Loop: Looking for optimal hard negative embeddings for deep metric learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 10634–10643, October 2021.
  * [63] Thang Vu, Kookhoi Kim, Tung M Luu, Xuan Thanh Nguyen, and Chang D Yoo.  Softgroup for 3d instance segmentation on point clouds.  In CVPR, 2022.
  * [64] Feng Wang and Huaping Liu.  Understanding the behaviour of contrastive loss.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2495–2504, June 2021.
  * [65] Tongzhou Wang and Phillip Isola.  Understanding contrastive representation learning through alignment and uniformity on the hypersphere.  In ICML, 2020.
  * [66] Weilun Wang, Wengang Zhou, Jianmin Bao, Dong Chen, and Houqiang Li.  Instance-wise hard negative example generation for contrastive learning in unpaired image-to-image translation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 14020–14029, October 2021.
  * [67] Xiaolong Wang and Abhinav Gupta.  Unsupervised learning of visual representations using videos.  In Proceedings of the IEEE international conference on computer vision, pages 2794–2802, 2015.
  * [68] Xinlong Wang, Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li.  Dense contrastive learning for self-supervised visual pre-training.  In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2021.
  * [69] Mike Wu, Milan Mosse, Chengxu Zhuang, Daniel Yamins, and Noah Goodman.  Conditional negative sampling for contrastive learning of visual representations.  arXiv preprint arXiv:2010.02037, 2020.
  * [70] Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin.  Unsupervised feature learning via non-parametric instance discrimination.  In CVPR, 2018.
  * [71] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.  Barlow twins: Self-supervised learning via redundancy reduction.  ICML, 2021.
  * [72] Kaiwei Zeng, Munan Ning, Yaohua Wang, and Yang Guo.  Hierarchical clustering with hard-batch triplet loss for person re-identification.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [73] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer.  Scaling vision transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12104–12113, 2022.
  * [74] Mingkai Zheng, Fei Wang, Shan You, Chen Qian, Changshui Zhang, Xiaogang Wang, and Chang Xu.  Weakly supervised contrastive learning.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10042–10051, 2021.
  * [75] Chengxu Zhuang, Alex Lin Zhai, and Daniel Yamins.  Local aggregation for unsupervised learning of visual embeddings.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6002–6012, 2019.