# Self-Supervised Learning Based Handwriting Verification

Mihir Chauhan Mohammad Abuzar Shaikh Bina Ramamurthy
Mingchen Gao Siwei Lyu Sargur Srihari

Department of Computer Science and Engineering
The State University of New York, Buffalo, NY, USA
{mihirhem, mshaikh2, bina, mgao8, siweilyu<EMAIL_ADDRESS>
###### Abstract

We present SSL-HV: Self-Supervised Learning approaches applied to the task of
Handwriting Verification. This task involves determining whether a given pair
of handwritten images originate from the same or different writer
distribution. We have compared the performance of multiple generative,
contrastive SSL approaches against handcrafted feature extractors and
supervised learning on CEDAR AND dataset. We show that ResNet based
Variational Auto-Encoder (VAE) outperforms other generative approaches
achieving 76.3% accuracy, while ResNet-18 fine-tuned using Variance-
Invariance-Covariance Regularization (VICReg) outperforms other contrastive
approaches achieving 78% accuracy. Using a pre-trained VAE and VICReg for the
downstream task of writer verification we observed a relative improvement in
accuracy of 6.7% and 9% over ResNet-18 supervised baseline with 10% writer
labels. Our code is available at https://github.com/Mihir2/ssl-hv.

##  1 Introduction

Figure 1: The overall framework for the SSL-HV for generating representation
for the task of Handwriting Verification.

Handwriting Verification is the process of comparing questioned writing and
known writing [59]. It is a critical task in various domains including
forensics, banking, and legal proceedings. Traditional approaches to
handwriting verification [65] includes using global handwriting features to
determine between-writer and within-writer variations do not capture the full
complexity and variability in handwriting. With the advent of artificial
neural networks, deep networks like Convolutional Neural Networks (CNNs) and
Vision Transformers (ViT) are used to generate hierarchical representation
from images. Such networks have shown promising results on variety of vision
tasks and there has been increasing amount of research being done to apply
these deep learning techniques to the downstream tasks of Handwriting
Identification, Writer-Retrieval and Recognition. Supervised approaches
$f_{\theta_{s}}(x,y)$ [24] [64] [16] using deep networks heavily rely on
supervised writer labels $y$ during it’s training process. Collecting a
diverse dataset $X(x_{q},x_{k},y)$ with known $x_{k}$, questioned $x_{q}$
handwritten samples and corresponding writer labels $y$ is expensive and time-
consuming. The dependency on labeled dataset limits the scalability of the
supervised methods because of data collection and labeling efforts.

Self-Supervised Learning (SSL) [6] provides an alternative approach to learn
meaningful representations from input $X$ by leveraging the intrinsic patterns
and structures from input $X$ without the need of explicit supervised labels
$y$. This helps to reduce the burden of data collection and allows the
utilization of large amount of untapped unlabeled or partially labeled data
that is available. Although, SSL has been employed in various domains within
Computer Vision, but the application of SSL has been limited within
handwriting domain.

Some examples of handwritten features generated using SSL are: SURDS [15]
propose a two-staged SSL framework for writer independent Offline Signature
Verification by using a dual triplet loss base fine tuning. The authors in
[51] use a SSL for handwriting identification for medieval by finetuning a
ResNet18 [43] architecture on a set of unlabeled manuscripts using Triplet
Loss. POSM [55] uses SSL for pretraining models to extract representations
from online handwriting in English and Chinese languages. The pretrained POSM
models were capable of achieving good results on diverse set of handwriting
tasks such as writer identification, handedness classification etc. The
authors in [58] uses SSL approach with Vision Transformers (ViT) for writer
retrieval task based on knowledge distillation. The authors also showcased the
attention feature maps which elaborated different parts of the handwriting
like loops, characters which enhanced the explainability for writer retrieval.
More recently, CSSL-RHA [75] used Contrastive SSL for Handwriting
Authentication. The authors use a hybrid CNN-Vit network for pre-training with
momentum-based paradigm followed by projection head while minimizing a InfoNCE
loss [73].

Motivated by the lack of SSL application to the domain of handwritten document
representation, we apply SSL approaches $f_{\theta_{SSL}}(x)$ to generate
representations $h_{ssl}$ for the handwritten images $(x_{q},x_{k})$ as shown
in Figure 1. Our contributions: (1) We create a baseline for handwriting
verification task using handcrafted features and supervised learning approach
using ResNet-18 [43] and ViT [32]. (2) We pre-train using four Generative SSL
(GSSL-HV) approaches for learning representations $h_{ssl}$ for the downstream
task of Handwriting Verification using Auto-Regressive Image Modeling [34],
Flow based model [29] [30] [45], Masked AutoEncoder [41], ResNet based
Variational AutoEncoders (VAE) [48] and Bi-Directional Generative Adversarial
Network (Bi-GAN) [31]. (3) We pre-train eight Contrastive SSL (CSSL-HV)
approaches using ResNet-18 [43] as encoder networks $f_{\theta}(x)$ to learn
representations from handwritten images $x$. The contrastive approaches are
MoCo[42], SimCLR[18], SimSiam [21], FastSiam [60], DINO[14], BarlowTwins [82]
and VicReg[8]. (4) Lastly, we fine-tune a MLP $f_{\theta_{MLP}}(h_{SSL},y)$
for the downstream task of handwriting verification on CEDAR AND dataset.

Figure 2: Examples of “AND” image fragments extracted from CEDAR Letter
dataset.

##  2 Dataset

CEDAR AND dataset is used for pre-training and fine-tuning on downstream task
of verification. CEDAR AND dataset is derived from CEDAR Letter dataset
wherein 1567 writers have writer a letter manuscript three times. Each
manuscript had upto five occurrences of the word “AND”. The manuscripts were
passed to transcript-mapping tool of CEDAR-FOX [46] to extract image fragments
of the word “AND”. In total, the tool was able to extract 15,518 image
fragments of the word “AND”. Some examples of the snippets are shown in Figure
2. Each image was resized to 64x64 keeping uniform padding and aspect ratio.

Handcrafted features were derived for each “AND” image fragment. Gradient
Structural Concavity (GSC) [35] features are used as micro features for
forensic verification by CEDAR-FOX tool, a state-of-the-art handwriting
analysis tool [66], developed at Center of Excellence for Document Analysis
and Recognition, University at Buffalo. GSC features were extracted for the
binarized “AND” images using a C-code. The GSC features are in 512 dimensions.
CEDAR-FOX uses the GSC features to verify the log likelihood ratio (LLR)
between the known and questioned handwritten sample. We use OpenCV
HOGDescriptor for generating Histogram of Oriented Gradients (HOGS) [26]
features for each AND image fragment. The HOGS features are 1764 dimensional
vectors.

We use unseen writer data partitioning for supervised fine-tuning for
downstream verification task. Hence, $W_{train}\bigcap W_{test}=\emptyset$,
where $W_{train}$ represents train writers and $W_{test}$ represents test
writers. For both pre-training and downstream fine-tuning, writer ids $w_{i}$
up-to 1200 were used for training $i_{train}\in\\{{1,2,..,1200}\\}$ and rest
were used as test. For fine-tuning, we generate equal number of same and
different writer sample pairs. We have two setups for fine-tuning with 10% and
100% of train writers resulting in 13,232 and 129,602 pairs of known and
questioned “AND” samples. The test is fixed for both setups with all the test
writers $i_{test}>1200$.

##  3 Learning Representation using Self-Supervised Learning based Pre-
Training

Self-supervised learning (SSL) provides an opportunity to generate good
representation for handwritten images without the need of supervised labels
(writer ids). SSL has been used as a pre-text task for pre-training a network
$f_{\theta_{SSL}}$ to generate representations $h_{ssl}$ only using input data
$x$ without explicit writer labels $y$. SSL approaches either maximize the
likelihood of unlabeled data $p(x)$ by reconstructing the input $x$ from $h$
or use a discriminative approach to learn representations by exploiting rich
similarities between parts of input data. Hence, SSL approaches are mainly
classified into Generative and Contrastive as summarized by Liu et al. in
[53].

###  3.1 Generative SSL (GSSL)

Given input pairs $(X,Y)$ where $x_{i}$ is the $i$-th input example and
$y_{i}$ is $i$-th target/class label for $x_{i}$, generative approaches in
statistics learns parameters $\theta$ from underlying probability distribution
of $p(X)$ by maximizing likelihood of a generative objective function $L$. The
parameters $\theta$ are optimized based on Maximium Likelihood Estimation
(MLE), Bayesian Inference or adversarial training. SSL leverages the
generative approaches to pre-train a network $f_{\theta}(x)$ which learns to
generate hidden latent representation $h$ by learning to fit on $P(X)$ by
optimizing $\theta$ on $L$. The representations $h$ learned during pre-
training phase are used to train the downstream tasks with fewer labels $Y$.

Earliest work in the field of generative approach to learning representations
and dimensionality reduction $h$ are Deep Belief Networks [10], RBM [44] which
uses a deep network whose parameters are updated using a reconstruction loss
$L_{recon}$. Recently, significant advancements have been made in generative
SSL approaches, leveraging techniques such as Auto-Regressive (AR) models,
Flow-based models, Auto-Encoding models, and Generative Adversarial Networks
(GANs).

AR Models can be considered directed probabilistic graphical models which
models input data distribution $p(X)=p(x_{1},...,x_{n})$ as product of
conditionals $\prod_{i=1}^{n}P(x_{i}|(x_{1},..,x_{i-1}))$ where $n$ is the
input dimensionality. Many approaches have been proposed in the research to
model the input image $X$ distribution using product of pixel conditionals
like NADE [70], RIDE [69], PixelRNN [71] and Gated PixelCNN [72]. We primarily
use state-of-the art AR model named Auto Regressive Image Modeling (AIM) which
uses AutoRegressive loss function $L_{AR}$ using Vision Transformer (ViT) [32]
architecture. Given input data $X$ the AR objective $L_{AR}$ was to minimize
the negative log likelihood of input distribution which is a density function
modeled using product of $n$ image patch conditionals. The prediction is a
normalized pixel-level regression loss as shown in Equation 1 below.

$\begin{split}L_{AR}&=\underset{x\sim
X}{\mathbb{E}}[-\log\prod_{i=1}^{n}P(x_{i}|(x_{1},..,x_{i-1}))]\\\
\mathcal{L}_{recon}&=\frac{1}{n}\sum_{i=1}^{n}\|\hat{x}_{i}-x_{i})\|^{2}\end{split}$
(1)

Flow based models explicitly learns the true data distribution $p(x)$ by a
sequence of invertible transformation functions $f(x)$ to map input $x$ to
latent representation $z$. Since the transformation are invertible
$x=f^{-1}(z)$ is true. Multiple flow based models like NICE [29], RealNVP
[30],Glow [47] aim to provide a tractable and flexible solution to computing
density function containing Jacobian determinant of the transformation. The
loss is a negative log-likelihood over the input distribution dataset as shown
in the Equation 2 below:

$\log p_{x}(\mathbf{x})=\log
p_{z}(f(\mathbf{x}))+\log{}\left|\det\frac{df(\mathbf{x})}{d\mathbf{x}}\right|$
(2)

Layers used in Flow based models shown in Figure 3 are variational
dequantization to handle discrete pixel values in images to continuous values
using uniform noise to each pixel, coupling layers helps to ensure
inveribility of the forward and inverse transformation, squeeze and split
operation helps in reducing the spatial resolution for efficiently training
flow based models.

Figure 3: Flow based model architecture.

AutoEncoding Models AutoEncoder [7] transforms input $x$ to latent
representation $z$ using an encoder feed forward neural network
$z=f_{enc}(x)$. The latent representations $z$ are then reconstructed using a
decoder feed forward decoder network $x^{\prime}=f_{dec}(z)$. The objective of
the model is to regenerate $x^{\prime}$ which should be as close to $x$ as
possible from latent representations $z$. The loss is optimized using a
reconstruction error $\mathcal{L}_{recon}(x,x^{\prime})$ which could be mean
squared error (MSE) as shown in Equation 3. The reconstruction loss could also
be binary/categorical cross-entropy depending on the input data $x$ and
modality.

(a) Original

(b) Transformed

(c) Masked

Figure 4: (a) Original AND image from writer 1471 first sample B with 64x64
size (b) Shows transformed image sample resized to 224x224 and normalized to
ImageNet mean and std. deviation (b) Shows masked transformed image with 32x32
masked patches with masking ratio set to 0.2

MAE Masked AutoEncoder [41] applies random mask patches $M$ with high masking
ratio on the input image $x$. Encoder $f_{enc}(x^{m},p)$ is a Vision
Transformer (ViT) [32] which takes as input visible parts of the image $x^{m}$
and positional embeddings $p$ to generates latent representation $z^{m}$. A
lightweight decoder $f_{dec}(z^{m},p,M)$ reconstructs the entire image $x$
including the missing patches using the encoded latent representations $z^{m}$
of the masked image $x^{m}$, positional embeddings $p$ and tokens from masked
patches $M$. The loss is computed on the masked tokens patches only and is
computed using Mean Squared Error (MSE) as shown in Equation 3.

$\mathcal{L}_{recon}=\frac{1}{N^{2}}\sum_{i=1}^{N^{2}}\|x_{i}-f_{dec}(z^{m}_{i},p_{i},M_{i})\|^{2}$
(3)

VAE Variational Auto Encoder belongs to class of Latent variable models (LVM).
The goal of a latent generative model is to generate samples $z$ from which we
generate the most probable value of $x$ according to the distribution
$p(x|z)$. We first sample a value of $z$ from some prior distribution $p(z)$
and then generate a sample from $p(x|z)$. Hence, our goal is to maximize the
probability of $x$. The key to compute $p(x)$ is to attempt to sample values
of $z$ that are likely to have produced $x$ using posterior probability
$p(z|x)$. VAE minimizes the Reconstruction and Latent loss as shown in
Equation 4. We can perform gradient-ascent on $\boldsymbol{L}_{vae}$ to update
the generative and variational parameters.

$\boldsymbol{L}_{vae}=-E_{z\sim q_{\phi}(z|x)}[\log p_{\theta}(x|z)]+\\\
D_{KL}[q_{\phi}(z|x)||p_{\theta}(z)]$ (4)

###  3.2 Contrastive SSL (CSSL)

Contrastive learning uses discriminative approach to learn representations $h$
by maximizing the agreement between similar (positive) images and minimize the
agreement between dissimilar (negative) images $P(Y|X=x)$. These
discriminative model learns representation using Noise Contrastive Estimation
(NCE) [39], InfoNCE [73] whose aim is to compare and learn the objective
function as shown in Eqn. 5 below:

$\mathcal{L}_{nce}=-\log\frac{\exp\left[h_{a}^{T}\cdot
h_{+}/\tau\right]}{\exp\left[h_{a}^{T}\cdot
h_{+}/\tau\right]+\exp\left[h_{a}^{T}\cdot h_{-}/\tau\right]}$ (5)

where, $h=f(x)$ are features and $f$ is a function to embeding input $x$.
$x^{+}$ is similar to input image used an an anchor $x^{a}$, $x^{-}$ is
dissimilar to $x$ and $f$ is a function to embed input image $x$ to features
$h$. Most contrastive SSL methods augmented views of the anchor image $x^{a}$
as positive $x^{+}$ whereas all other images are used as negatives $x^{-}$.

Momentum Contrast (MoCo v1[42], MoCo v2[20], MoCo v3[23]) maintains a
dictionary of positive $k_{+}$ and negative $k_{i}$ encoded-samples (keys)
which is compared with the anchor (query) $q$. MoCo uses ResNet [43] as query
$f_{q}$ encoder and key $f_{k}$ encoder parameterized by weights $\theta_{f}$
and $\theta_{k}$ with last Fully-Connected (FC) layer having 128-D fixed
embeddings which are generated for each all Query and Keys. Further, InfoNCE
[73] based loss function is used to measure the similarity between the query
and key embeddings as shown in Equation 6 below:

$\mathcal{L}_{moco}=-\log\frac{\exp\left(q\cdot
k_{+}/\tau\right)}{\sum_{i=0}^{K}\exp\left(q\cdot k_{i}/\tau\right)}$ (6)

The parameters of the query $f_{q}$ and key encoder $f_{k}$ are update with a
momentum parameters as: $\theta_{\mathrm{k}}\leftarrow
m\theta_{\mathrm{k}}+(1-m)\theta_{\mathrm{q}}$ with $m$ as momentum parameter.
MoCo proposed v2 version by using using a MLP projection head and more data
augmentation following the work of SimCLR [18].

Simple Contrastive Learning (SimCLR v1[18], SimCLR v2[19]) also learns
representation by maximizing agreement between two differently augmented views
$x_{i}$ and $x_{j}$ of the same example $x$. Specific augmentations types used
were random cropping, color distrocutions and Gaurssian blur. Similar to MoCo,
SimCLR also uses ReNet as the base encoder $h=f(x)$ where $h$ is the average
pooling layer from ResNet. Additionally, an MLP with 1 hidden layer and RELU
activation function is used $z=g(h)$. The output of MLP are the latent
embeddings $z$. The contrastive loss function for a pair of positive latent
embeddings used is the normalized temperature-scaled cross entropy loss (NT-
Xent) as shown in Equation 7 below:

$\ell_{i,j}=-\log\frac{\exp\left(\operatorname{sim}\left(\boldsymbol{z}_{i},\boldsymbol{z}_{j}\right)/\tau\right)}{\sum_{k=1}^{2N}\mathbbm{1}_{[k\neq
i]}\exp\left(\operatorname{sim}\left(\boldsymbol{z}_{i},\boldsymbol{z}_{k}\right)/\tau\right)}$
(7)

where, $\mathbbm{1}_{[k\neq i]}\in{0,1}$, $\tau$ is temperature parameter and
$sim(u,v)$ is the dot product between the $l_{2}$ normalized $u$ and $v$.

(a) Original

(b) Crop+R

(c) Center

(d) V. Flip

(e) H. Flip

(f) Rotate

(g) Persp.

(h) Blur

(i) Jitter

(j) Invert

Figure 5: Data Augmentation Views from an example original image of word “AND”
extracted from writer ID 1471-Sample-B1 from CEDAR Dataset.

Bootstrap Your Own Latent (BYOL) [38] was removes the dependency on negative
examples thereby being robust to batch size and memory constraints. BYOL uses
an online and target network. Input to online network is first augmented view
$u$ of input. The online network consists of an encoder
$h_{\theta}=f_{\theta}(u_{\theta})$, MLP projector
$z_{\theta}=g_{\theta}(h_{\theta})$ and predictor $q_{\theta}(z_{\theta})$.
The target network parameters $\xi$ has the same architecture but uses moving
average weights of the online parameters $\theta$. After each training step,
the weights are updated using $\xi\leftarrow\tau\xi+(1-\tau)\theta$. The loss
is mean squared error between the $l_{2}$ normalized predictions from the
first augmented input-online network and the target network projections as
shown in Equation 8:

$\mathcal{L}_{\theta,\xi}\triangleq\left\|\overline{q_{\theta}}\left(z_{\theta}\right)-\bar{z}_{\xi}^{\prime}\right\|_{2}^{2}=2-2\cdot\frac{\left\langle
q_{\theta}\left(z_{\theta}\right),z_{\xi}^{\prime}\right\rangle}{\left\|q_{\theta}\left(z_{\theta}\right)\right\|_{2}\cdot\left\|z_{\xi}^{\prime}\right\|_{2}}$
(8)

Simple Siamese (SimSiam) [21] simplifies BYOL [38] by maximizing contrastive
between two augmented views $x_{1}$, $x_{2}$ of an example image $x$ without
negative pairs and without momentum encoder. The two views are processed by
using ResNet [43] as an encoder $f$. Both the encoder share the same weights
along with a MLP head $p$. $x_{1}$ is passed through the encoder $f$ and MLP
predictor $p$ to get $p1=p(f(x_{1}))$. Wherease, $x_{2}$ is only passed
through $f$ to get $z_{2}=f(x_{2}$. The loss is calculated as shown in the
Equation 9 below:

$\mathcal{L}_{SimSIAM}=\frac{1}{2}\mathcal{D}\left(p_{1},z_{2}\right)+\frac{1}{2}\mathcal{D}\left(p_{2},z_{1}\right)$
(9)

where,

$\mathcal{D}\left(p_{1},z_{2}\right)=-\frac{p_{1}}{\left\|p_{1}\right\|_{2}}\cdot\frac{z_{2}}{\left\|z_{2}\right\|_{2}}$
(10)

Fast Siamese (FastSiam) [60] makes efficient use of SimSiam [21] approach by
using multiple views of a single image, which allows for faster convergence
and reduces amount of pre-training time.

Distillation with No Labels (DINO v1, v2) [14] [56] approach is similar to
BYOL [38] and SimSIAM [21] where the idea is to use knowledge distillation.
DINO trains a student network $g_{\theta_{s}}$ to imitate the output of the
teacher network $g_{\theta_{t}}$. A pair ($x_{1}$, $x_{2}$) of randomly
augmented views of an image $x$ are passed to student
$s1=g_{\theta_{s}}(x_{1})$ and teacher $t1=g_{\theta_{t}}(x_{1})$. DINO uses
Vision Transformers [32] or ResNet [43] for network $g$ and 3-layer MLP
followed by $l_{2}$ normalization and a weight normalized FC layer with $K$
dimensions. The resultant of passing the augmented images through network $g$
followed by MLP is a output probability distributions over $K$ dimensions as
shown in the Equation 11 below:

$P_{s}(x)^{(i)}=\frac{\exp\left(g_{\theta_{s}}(x)^{(i)}/\tau_{s}\right)}{\sum_{k=1}^{K}\exp\left(g_{\theta_{s}}(x)^{(k)}/\tau_{s}\right)}$
(11)

In the above Equation 11 $\tau_{s}$ is a temperature parameter. DINO minimizes
cross-entropy loss between probability distributions $P_{s}(x)$ and $P_{t}(x)$
generated by student and teacher networks as shown in the Equation 12 below:

$\mathcal{L}_{DINO}=\min_{\theta_{s}}\sum_{x\in\left\\{x_{1}^{g},x_{2}^{g}\right\\}}\sum_{\begin{subarray}{c}x^{\prime}\in
V\\\ x^{\prime}\neq
x\end{subarray}}H\left(P_{t}(x),P_{s}\left(x^{\prime}\right)\right)$ (12)

Where, $H(a,b)=-a\log b$. Furthermore, DINO uses Exponential Moving Averages
on the student weights
$\theta_{\mathrm{t}}\leftarrow\lambda\theta_{\mathrm{t}}+(1-\lambda)\theta_{\mathrm{s}}$
which is similar to momentum encoder ideas in MoCo [42].

Barlow Twins [82] makes use of cross-correlation matrix as shown in Equation
13 on a batch of mean-centered embeddings $Z^{A}$ and $Z^{B}$ generated by a
network function $f_{\theta}$ from two augmented views $Y^{A}$ $Y^{B}$ of the
image $X$.

$\mathcal{C}_{ij}\triangleq\frac{\sum_{b}z_{b,i}^{A}z_{b,j}^{B}}{\sqrt{\sum_{b}\left(z_{b,i}^{A}\right)^{2}}\sqrt{\sum_{b}\left(z_{b,j}^{B}\right)^{2}}}$
(13)

The encoder network uses ResNet [43] (without classification layer) followed
by 3 layer MLP with 8192 output units. The goal is to make the cross-
correlations between the outputs $Z^{A}$ and $Z^{B}$ closer to the identity
matrix as shown in Loss Equation 14.

$\mathcal{L}_{\mathcal{B}\mathcal{T}}\triangleq\underbrace{\sum_{i}\left(1-\mathcal{C}_{ii}\right)^{2}}_{\text{invariance
term }}+\lambda\underbrace{\sum_{i}\sum_{j\neq
i}\mathcal{C}_{ij}{}^{2}}_{\text{redundancy reduction term }}$ (14)

Variance-Invariance-Covariance Regularization (VICReg, VICRegL) [8] [9]:
tackles the model collapse problem - which happens when the model produces
same representations irrespective of input. Similar to other approaches
mentioned above, given an input image $i$, two augmented views $x$ and
$x^{\prime}$ are generated. Then using encoder network $f_{\theta}$
representations are generated. $y=f(x)$ and $y=f(x^{\prime})$ which is then
passed to expander to generate embeddings $z=h_{\phi}(x)$ and
$z^{\prime}=h_{\phi}(x^{\prime})$. The loss between the embeddings $z$ and
$z^{\prime}$ is a weighted average of three terms in loss function as shown in
Equation 15.

$\ell\left(Z,Z^{\prime}\right)=\lambda\underbrace{s\left(Z,Z^{\prime}\right)}_{Invariance}+\mu\underbrace{\left[v(Z)+v\left(Z^{\prime}\right)\right]}_{Variance}+\\\
\nu\underbrace{\left[c(Z)+c\left(Z^{\prime}\right)\right]}_{Covariance}$ (15)

Variance is a reqularization term acting as a hinge function on standard
deviation of the embeddings $Z$. For covariance, VICReg uses covariance matrix
term similar to BarlowTwin [82] and finally invariance criterion is a mean-
squared euclidean distance between each pairs of vectors without
normalization.

Model Accuracy Precision Recall F1-Score GSC [35] 0.71 / 0.78 0.69 / 0.81 0.72
/ 0.77 0.69 / 0.79 ResNet-18 [43] 0.72 / 0.84 0.70 / 0.86 0.73 / 0.82 0.72 /
0.84 ViT [32] 0.65 / 0.79 0.68 / 0.80 0.64 / 0.78 0.66 / 0.79

Table 1: Performance Metrics on Test Writer ($w_{i}$ where $i>=1200$) set for
Supervised Baselines with 10% & 100% of Train Writers.

##  4 Experiments and Implementation Details

###  4.1 Supervised Baseline

Feature Extractors: Supervised baseline is performed using two handcrafted
features (GSC, HOGS), CNN based ResNet-18 and Vision Transformer based ViT
architecture on CEDAR AND dataset for 10% and 100% of train writers. ResNet-18
[43] CNN architecture having 11.2 M parameters is fine-tuned on the training
pairs for each setup. With ResNet-18 we update the first convolutional layer
to accept 3 channel input and set the last FC within ResNet-18 to be Identity
and add a supervised classification head as elaborated in the next section.
MaskedCausalVisionTransformer is used with configurations same as AIM which
will be described in detail within the following sections on applying AIM to
downstream verification task. ViT has 88.2M parameters.

Classification Head: The output of the feature extractors is fed into 2 fully-
connected (FC) layers. FC1 and FC2 has 256 and 128 hidden neurons with ReLU
activations. The final layer has 2 output neurons whose softmax activations
represent similarity of samples with a one hot vector representation. We use
categorical cross entropy loss given one-hot encoded logits compared to the
target which is binary (0 or 1).

Training: Batch Size for training was 256, Learning rate 1e-3, Adam Optimizer
and Early stopping with F1 score stagnating with patience 5 and delta 0.001
for all baselines GSC, ViT and ResNet-18. The results from the supervised
training with on 10% and 100% train writers are tabulated in Table 1.

Pre-Training Metric: In Table 2 shows multiple SSL methods and pre-training
performance using the mean of intra-writer $\text{COS}_{intra}$ and inter-
writer cosine $\text{COS}_{inter}$ similarity on a validation set.

$\text{COS}(\mathbf{h}_{k},\mathbf{h}_{q})=\frac{\mathbf{h}_{k}\cdot\mathbf{h}_{q}}{\|\mathbf{h}_{k}\|\|\mathbf{h}_{q}\|}$
(16)

In the Equation 16 above, $\mathbf{h}_{q}$ represents the features of the
handwritten sample from known writer and $\mathbf{h}_{q}$ represents the
features of the handwritten samples form questioned writer.
$\text{COS}_{intra}$ is when writer of Known and Questioned sample is same.
$\text{COS}_{intra}$ is when Know and Questioned sample is from different
writers. This intra and inter cosine similarity seperation evaluation metric
was set to measure the separation between $\text{COS}_{intra}$ and
$\text{COS}_{inter}$ during the pre-training phase in order to track SSL model
capability to differentiate samples between writers. The seperation provides a
proxy metric to check for representation collapse and track the progress of
pre-training. Intra-Nd and Inter-Nd shows the cosine similarity between and
amongst test writers with writer ids greater than 1200. Intra-2d and Inter-2d
are two dimensional representation obtained using TSNE dimensionality
reduction. We then calculate the cosine similarity amongst test writers.

Downstream Verification Metric: We compare the downstream model performance
using classification metrics such as acccuracy to compare pre-trained SSL-HV
model performance against handcrafted features and supervised models.

###  4.2 Generative Self-Supervised Learning for Handwriting Verification
(GSSL-HV)

AIM [34] uses random masking to sequentially learn masked patches of the
handwritten word AND using an autoregressive objective function. The input
handwritten image of AND is divided into non-overlapping patches with 32
patch_size as shown in Figure 4(c). AIM works well with simple data
augmentations during training hence we used Resize by Torch [3] with size
224x224 because the ViT was trained on 224x224 image sizes. We normalize the
image with mean (0.485, 0.456, 0.406) and standard deviation (0.229, 0.224,
0.225) which is same as used in the ImageNet dataset. For testing, same
transformation are used as train. Output of the image transformations is shown
in the Figure 4. We use CausalVisionTransformer as the backbone with masked
causal attention based on AIM [34] at input image resolution 224x224. The
sequence length for CausalVisionTransformer was 49 since the patch size was 32
with no [CLS] classification token. The embedding dimension of the encoder is
768, depth and number of attention heads 12. The encoder has 88.2M parameters.
Projection head consists of a Linear FC layer followed by
AIMPredictionHeadBlock, LayerNorm and another Linear Layer. The projection
head contains 41.4M parameters. The loss for AIM is Mean Square Error Loss
which is same as MAE loss as shown in Equation 3. Optimizer is AdamW with
learning rate 1.5e-4. The batch_size was set to 1024 which takes 22.3GiB out
of the total 24GiB of GPU memory.

Normalizing Flow model trained using a series of flow transformation to
estimate the density of $p(x)$ using the latent representation $z$ with
$p(z)$. The input handwritten AND image is inverted with and converted to
gray-scale with a single channel and pixel value between [0,255]. Figure 3
shows the flow based architecture. We use a single variational de-quantization
layer to quantize discrete pixel values as samples from continuous
distribution which helps improve diversity and quality of generated samples.
This is followed by 2 affine coupling layers [29] with a single channel
checkerboard mask throughout the network. A Gated CNN using two-layer
convolutional ResNet block with input gate is used similar to Flow++ [45]. We
use multi-scale architecture as proposed by RealNVP[30] using Squeeze and
Split layers. The batch_size for train and test is set to 128, Adam optimizer
and learning rate set to 1e-3. The objective function is a negative log-
likelihood function. The model is evaluated based using bits per dimension
(bpd) for the train and validation set. Train and Val bpd was 0.8355 and 0.841
respectively. Total numbers of parameters are 1.7M and the 16GiB of GPU memory
during training.

MAE is applied using random masking to reconstruct masked patches of the
handwritten word AND. The masking ratio of removed patches was set to 20%. The
input handwritten image of AND is divided into non-overlapping patches with 32
patch_size as shown in Figure 4(c). Similar to AIM, MAE uses simple resizing
to 224x224 and normalization with mean (0.485, 0.456, 0.406) and standard
deviation (0.229, 0.224, 0.225) as used in the ImageNet dataset. For testing,
we use the same transformation as train. Output of the image transformations
is shown in the Figure 4. We use vit-base-patch32-224-in21k [77] as the
backbone for MAE which is a Vision Transformer (ViT) trained on ImageNet-21k
[27] at resolution 224x224. The embedding dimension of the encoder is 768. The
encoder has 88M parameters. Decoder contains a linear layer which takes in the
768 dimensional embedding from the encoder and outputs 512 dimensional. The
output of the linear layer is fed into a single Vision Transformer followed by
normalization and linear layer. The final linear layer reconstructs the masked
patches with outputs size as (patch_size, patch_size, num_channels). The
decoder has 5.1M parameters. The loss for MAE is Mean Square Error Loss as
shown in Equation 3. Optimizer is AdamW with learning rate 1e-3. The
batch_size was set to 1024 which takes 16GiB out of the total 24GiB of GPU
memory.

VAE taskes as input an inverted image with size 64x64x3. The encoder $f_{enc}$
and decoder $f_{dec}$ used in VAE is the ResNet Encoder and Decoder by Pytorch
Bolt [11]. The ResNet encoder $f_{enc}$ output dimension 512 and has 11.2M
trainable parameters while the $f_{dec}$ has 8.6M parameters. $f_{enc}$ is
followed by two fully-connected layers $FC_{\mu}$ and $FC_{\sigma}$ with 131k
parameters as $f_{enc}$ output dimensions are 512 and latent dimensions $z$
are 256. The loss was computed using ELBO as described in the VAE section. The
learning rate was set to 1e-4 with Adam optimizer.

BiGan [31] is trained using an Encoder $f_{enc}$, Generator $f_{G}$ and
Discriminator $f_{D}$ network. The $f_{enc}$ network consists of 5 blocks,
starting with 1024 hidden units and consequently layers have hidden units
divided by 2. $f_{enc}$ network which takes as input raw flattened image and
outputs latent representation $z$ with dimensionality as 100. Each block has a
LeakyRelu activation and normalization except the first layer. The final
activation layer has tanh activation to keep the value between -1 and 1. The
$f_{G}$ network takes as input $z$ latent representation and also network
structure which is opposite to the $f_{enc}$ in order to re-generate back the
output dimensions 64x64 with 3 channels. The discriminator takes as input
image and latent representation and minimizes the binary cross-entropy loss
between the fake and valid combination samples of latent representation and
input image. We use pytorch lightning multi-optimizer function to update the
gradients of generator and discriminator in an alternating fashion. The total
number of parameters in generator and encoder are 26.6M while the
discriminator has 6.9M parameters. Adam optimizer with learning rate set to
2e-4.

###  4.3 Contrastive Self-Supervised Learning for Handwriting Verification
(CSSL-HV)

Data Augmentation regularizes the model and helps it to learn from patterns
within different parts of the input image. There exists a wide variety of
augmentation techniques which can be used to get different forms of
invariances. Here, invariance is a property of the representation learning
model to generate similar image embeddings/representations irrespective of the
position, rotation, scale etc. of the image. Depending on the type of
invariance requirement of the downstream task, we can apply these transforms
(invariances) to the pre-training network. For example, since our domain is of
handwriting verification pen and background color can be invariant. However,
we should not randomize on aspect-ratio transformation or bluring in cases of
handwriting because it will lead to underfitting and performance degradation.
Some examples of invariances as mentioned and illustrated by SimCLR [18] are:
Shape Invariances like Random Cropping, Random Horizontal/Vertical Flip,
Rotation etc, Texture Invariances - Gaussian Blur and Color Invariance. The
authors in [61] have shown the importance of data augmentation techniques in
context of coarse-grained, fine-grained and few-shot downstream tasks. It is
important to note that, for certain downstream fine-grained downstream tasks
like that of differentiating between animal of same species like birds, we
should not use color distortions like jitter, contrast etc. since the
embeddings generated would underfit on the fine-grained downstream task as
shown in [78]. In our case, we have used Pytorch Transforms [3] implementation
for data augmentation with Lightly SSL framework [67].

Pre-Training, Projection Network and Loss Function For pre-training network,
we use a ResNet-18 [43] model with stochastic gradient descent with custom
loss functions. We use 3x3 convolution kernel in the first few layers of
ResNet-18 instead of originally proposed 7x7 kernel size for CEDAR AND dataset
because it is well suited for small input images as in our case 64x64.
ResNet-18 has 11.2M parameters. For CEDAR AND dataset we chose a larger kernel
size 7x7 variant of ResNet-18 since the Crop size from augmentation is
224x224. The variant has also has 11.2M parameters. For the projection
head/network and loss function, each CSSL approach described will have
different number of neurons and layers in the projection network. MoCo [42]
strategy was applied for pre-training a ResNet-18 [43] backbone network
$f_{\theta}$ with 11.2M frozen weights $\theta$. The backbone output is 512
dimensions which serves as the hidden representations for the down stream
task. We then have projection head (328K weights) with momentum as described
by MoCo [42]. The network minimizes the NTXent Loss as shown in Equation 6. We
use transforms described by MoCo v2 [20] except turn off the RandomGaussian
blur since the transform is too strong and underfits on the smaller resolution
of the image. The batch size is set to 1024, memory bank size is 4096 and 200
number of epochs. With the given batch size and resized image to 45x45 20Gb
out of 23Gb is consumed. The total pre-training time is 54 mins. For
inference, 3GB memory GPU memory is consumed. SimCLR [18] was used for pre-
training ResNet-18 [43] backbone. The output of backbone were 512 features.
The backbone was followed by a preojection head with (328k) weights and the
loss was NTXent Loss. The transforms used were RandomResize (size 45x45)
RandomHorizontalFlip, RandomVerticalFlip (0.5), RandomRotation(90 degrees),
RandomGrayScale and GaussianBlur was not applied for the same reason as
mentioned above. The batch size was 8000 to make sure that we are utilizing
23Gb of GPU memory during pretraining. During inference only 1.5Gb of memory
was used. The total pretraining took 1h 7mins. BYOL [38] pretrains ResNet-18
to generate 512 features. The BYOLProjection head contains 788k parameters and
the loss criterion is NegativeCosineSimilarity. VicReg [8], BarlowTwins [82]
and BYOL uses similar trasforms as SimCLR along with RandomSolarization
applied. The batchsize was set to 1024 for 200 epochs. Similar to BYOL,
FastSiam [60] and SimSiam [21] uses the same transforms except
RandomSolarization and the batch size was set to 256 for both. DINO [14]
global crop size was set to 45 rest of the transforms were the same as BYOL.
All the experiments were conducted using Lightly SSL Python package [67] on
AWS notebook instance with ml.g5.2xlarge which has 1 Nvidia A10G (24GB) GPU.

Model Intra-Nd Inter-Nd Intra-2d Inter-2d Accuracy Raw Pixels 0.96 0.95 0.07
-0.02 0.63 HOGS [26] 0.57 0.02 0.63 0.11 0.72 GSC [35] 0.92 0.67 0.86 0.56
0.71 AIM [34] 0.32 -0.05 0.78 0.75 0.73 Flow [29] [30] [45] 0.12 0.08 0.12
0.01 0.66 MAE [41] 0.18 0.02 0.82 0.77 0.71 VAE [49] 0.24 0.06 0.38 0.30 0.75
BiGAN [31] 0.35 0.30 0.27 0.25 0.68 MoCo [42] 0.89 0.78 0.92 0.73 0.73 SimClr
[18] 0.89 0.87 0.87 0.85 0.72 BYOL [38] 0.88 0.84 0.91 0.97 0.73 SimSiam [21]
0.87 0.81 0.94 0.84 0.75 FastSiam [60] 0.83 0.75 0.83 0.75 0.71 DINO [14] 0.88
0.85 0.78 0.74 0.68 BarlowTwins [82] 0.87 0.79 0.66 0.38 0.76 VicReg [8] 0.69
0.48 0.65 0.60 0.78

Table 2: Performance comparison of GSSL-HV and CSSL-HV approaches against
handcrafted feature baselines on CEDAR AND Dataset with 10% train writers.

##  5 Results

Table 2 shows the performance of the SSL and baseline approaches. We observe
that higher the separation between writers (Intra distance - Inter distance)
leads to higher test accuracy on a small training dataset. In the experiments
performed we observe VAE to be best performing with a good separation of 0.18
between the intra-inter distance between and amongst the writers during the
pre-training phase which lead to 6.7% relative increase in the accuracy when
compared to the best performing supervised ResNet-18 baseline with accuracy
72% accuracy on 10% train writers. Auto-Regressive AIM model and MAE
outperformed it’s supervised counterpart ViT on 10% train writers but had a
lower precision compared to VAE which is contributed to the difference in the
feature extraction process. Within VAE we used ResNet-18 architecture whereas
AIM and MAE uses a ViT architecture whose baseline metrics under performed
when compared to ResNet-18 as shown in Table 1. Flow based models performed
similar to the baselines but underperformed when compared to VAE and AIM. This
is contributed to the fact that flow based models do not support sparsity in
feature representation and the type of invertible transformations are not
suitable for granular variations within handwritten styles. GANs also
performed similar to baselines, this may be primarily due to the fact that
GANs are primarily used for data generation and do not naturally include an
encoder to map data back to the latent space. We observed that GSC and HOGS
features have maximum separation between $\text{COS}_{intra}$ and
$\text{COS}_{inter}$ whereas using raw pixels the separability is very low.
From the table 2 maximum separation of 0.28 is obtained using VicReg on the
CEDAR AND Dataset leading to 9% relative improvement in accuracy over best
performing supervised ResNet-18 baseline.

##  6 Conclusion

In conclusion, self-supervised learning provides a pathway to generating
robust handwritten features which helps improve downstream task of handwriting
verification with limited amount of training labels. In this paper, we
evaluated AutoRegressive, Flow Based, AutoEncoding and GANs as part of the
GSSL-HV framework. We also compared performance of eight CSSL-HV approaches.
VAE outperformed other generative self-supervised feature extraction
approaches, achieving a relative gain of 6.73% in accuracy whereas VICReg was
outperformed all the generative and contrastive approaches with a relative
accuracy gain of 9% over the baselines. Future research can aim to enhance the
feature extraction capabilities using multiple unlabeled handwritten datasets
such as IAM handwriting dataset and comparing similar and different
handwritten content using state-of-the-art self-supervised approaches.

## References

  * [1] Alexander A. Alemi, Ben Poole, Ian Fischer, Joshua V. Dillon, Rif A. Saurous, and Kevin Murphy.  Fixing a broken elbo, 2018.
  * [2] Alec Radford Jeffrey Wu Rewon Child David Luan Dario Amodei and Ilya Sutskever.  Language models are unsupervised multitask learners. technical report., 2019.
  * [3] Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano, Yanbo Liang, Jason Liang, Yinghai Lu, CK Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk, Michael Suo, Phil Tillet, Eikan Wang, Xiaodong Wang, William Wen, Shunting Zhang, Xu Zhao, Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala.  PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation.  In 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS ’24). ACM, Apr. 2024.
  * [4] Mahmoud Assran, Randall Balestriero, Quentin Duval, Florian Bordes, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, and Nicolas Ballas.  The hidden uniform cluster prior in self-supervised learning, 2022.
  * [5] Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael Rabbat, and Nicolas Ballas.  Masked siamese networks for label-efficient learning, 2022.
  * [6] Randall Balestriero, Mark Ibrahim, Vlad Sobal, Ari Morcos, Shashank Shekhar, Tom Goldstein, Florian Bordes, Adrien Bardes, Gregoire Mialon, Yuandong Tian, Avi Schwarzschild, Andrew Gordon Wilson, Jonas Geiping, Quentin Garrido, Pierre Fernandez, Amir Bar, Hamed Pirsiavash, Yann LeCun, and Micah Goldblum.  A cookbook of self-supervised learning, 2023.
  * [7] Dana H. Ballard.  Modular learning in neural networks.  In Proceedings of the Sixth National Conference on Artificial Intelligence - Volume 1, AAAI’87, page 279–284. AAAI Press, 1987.
  * [8] Adrien Bardes, Jean Ponce, and Yann LeCun.  Vicreg: Variance-invariance-covariance regularization for self-supervised learning, 2022.
  * [9] Adrien Bardes, Jean Ponce, and Yann LeCun.  Vicregl: Self-supervised learning of local visual features, 2022.
  * [10] Yoshua Bengio, Pascal Lamblin, Dan Popovici, and Hugo Larochelle.  Greedy layer-wise training of deep networks.  In B. Schölkopf, J. Platt, and T. Hoffman, editors, Advances in Neural Information Processing Systems, volume 19. MIT Press, 2006.
  * [11] Jirka Borovec, William Falcon, Akihiro Nitta, Ananya Harsh Jha, otaj, Annika Brundyn, Donal Byrne, Nathan Raw, Shion Matsumoto, Teddy Koker, Brian Ko, Aditya Oke, Sidhant Sundrani, Baruch, Christoph Clement, Clément POIRET, Rohit Gupta, Haswanth Aekula, Adrian Wälchli, Atharva Phatak, Ido Kessler, Jason Wang, JongMok Lee, Shivam Mehta, Zhengyu Yang, Garry O’Donnell, and zlapp.  Lightning-ai/lightning-bolts: Minor patch release, dec 2022.
  * [12] Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, and Samy Bengio.  Generating sentences from a continuous space, 2016.
  * [13] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin.  Unsupervised learning of visual features by contrasting cluster assignments, 2021.
  * [14] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers, 2021.
  * [15] Soumitri Chattopadhyay, Siladittya Manna, Saumik Bhattacharya, and Umapada Pal.  Surds: Self-supervised attention-guided reconstruction and dual triplet loss for writer independent offline signature verification, 2022.
  * [16] Mihir Chauhan, Mohammad Abuzar Shaikh, and Sargur N. Srihari.  Explanation based handwriting verification, 2019.
  * [17] Mark Chen, Alec Radford, Jeff Wu, Heewoo Jun, Prafulla Dhariwal, David Luan, and Ilya Sutskever.  Generative pretraining from pixels.  In International Conference on Machine Learning, 2020.
  * [18] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations, 2020.
  * [19] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton.  Big self-supervised models are strong semi-supervised learners, 2020.
  * [20] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He.  Improved baselines with momentum contrastive learning, 2020.
  * [21] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning, 2020.
  * [22] Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel.  Pixelsnail: An improved autoregressive generative model, 2017.
  * [23] Xinlei Chen, Saining Xie, and Kaiming He.  An empirical study of training self-supervised vision transformers, 2021.
  * [24] Jun Chu, Mohammad Abuzar Shaikh, Mihir Chauhan, Lu Meng, and Sargur Srihari.  Writer verification using cnn feature extraction.  In 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR), pages 181–186, 2018.
  * [25] Elijah Cole, Xuan Yang, Kimberly Wilber, Oisin Mac Aodha, and Serge Belongie.  When does contrastive visual representation learning work?, 2022.
  * [26] N. Dalal and B. Triggs.  Histograms of oriented gradients for human detection.  In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), volume 1, pages 886–893 vol. 1, 2005.
  * [27] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009.
  * [28] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding, 2019.
  * [29] Laurent Dinh, David Krueger, and Yoshua Bengio.  Nice: Non-linear independent components estimation, 2015.
  * [30] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio.  Density estimation using real nvp, 2017.
  * [31] Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell.  Adversarial feature learning, 2017.
  * [32] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
  * [33] Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman.  With a little help from my friends: Nearest-neighbor contrastive learning of visual representations, 2021.
  * [34] Alaaeldin El-Nouby, Michal Klein, Shuangfei Zhai, Miguel Angel Bautista, Alexander Toshev, Vaishaal Shankar, Joshua M Susskind, and Armand Joulin.  Scalable pre-training of large autoregressive image models, 2024.
  * [35] John T. Favata and Geetha Srikantan.  A multiple feature/resolution approach to handprinted digit and character recognition.  International Journal of Imaging Systems and Technology, 7(4):304–311, 1996.
  * [36] Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks, 2014.
  * [37] Alex Graves and Jürgen Schmidhuber.  Offline handwriting recognition with multidimensional recurrent neural networks.  In D. Koller, D. Schuurmans, Y. Bengio, and L. Bottou, editors, Advances in Neural Information Processing Systems, volume 21. Curran Associates, Inc., 2008.
  * [38] Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko.  Bootstrap your own latent: A new approach to self-supervised learning, 2020.
  * [39] Michael U. Gutmann and Aapo Hyvärinen.  Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics.  J. Mach. Learn. Res., 13(null):307–361, feb 2012.
  * [40] David Ha and Douglas Eck.  A neural representation of sketch drawings, 2017.
  * [41] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners, 2021.
  * [42] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning, 2020.
  * [43] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  CoRR, abs/1512.03385, 2015.
  * [44] Y. W. Hinton G. E. Osindero S. & Teh.  A fast learning algorithm for deep belief nets., 2006.
  * [45] Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel.  Flow++: Improving flow-based generative models with variational dequantization and architecture design, 2019.
  * [46] Chen Huang and Sargur N. Srihari.  Mapping Transcripts to Handwritten Text.  In Guy Lorette, editor, Tenth International Workshop on Frontiers in Handwriting Recognition, La Baule (France), Oct. 2006. Université de Rennes 1, Suvisoft.  http://www.suvisoft.com.
  * [47] Diederik P. Kingma and Prafulla Dhariwal.  Glow: Generative flow with invertible 1x1 convolutions, 2018.
  * [48] Diederik P. Kingma and Max Welling.  Auto-encoding variational bayes.  CoRR, abs/1312.6114, 2014.
  * [49] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes, 2022.
  * [50] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton.  Imagenet classification with deep convolutional neural networks.  In Proceedings of the 25th International Conference on Neural Information Processing Systems - Volume 1, NIPS’12, pages 1097–1105, USA, 2012. Curran Associates Inc.
  * [51] Lorenzo Lastilla, Serena Ammirati, Donatella Firmani, Nikos Komodakis, Paolo Merialdo, and Simone Scardapane.  Self-supervised learning for medieval handwriting identification: A case study from the Vatican Apostolic Library.  Information Processing & Management, 59(3):102875, 2022.
  * [52] Phillip Lippe.  UvA Deep Learning Tutorials.  https://uvadlc-notebooks.readthedocs.io/en/latest/, 2024.
  * [53] Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang.  Self-supervised learning: Generative or contrastive.  IEEE Transactions on Knowledge and Data Engineering, page 1–1, 2021.
  * [54] Alejandro López-Cifuentes, Marcos Escudero-Viñolo, Jesús Bescós, and Álvaro García-Martín.  Semantic-aware scene recognition.  Pattern Recognition, 102:107256, June 2020.
  * [55] Pouya Mehralian, Bagher BabaAli, and Ashena Gorgan Mohammadi.  Self-supervised representation learning for online handwriting text classification, 2023.
  * [56] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski.  Dinov2: Learning robust visual features without supervision, 2024.
  * [57] Bo Pang, Yifan Zhang, Yaoyi Li, Jia Cai, and Cewu Lu.  Unsupervised visual representation learning by synchronous momentum grouping, 2022.
  * [58] Marco Peer, Florian Kleber, and Robert Sablatnig.  Self-supervised vision transformers with data augmentation strategies using morphological operations for writer retrieval.  In Frontiers in Handwriting Recognition: 18th International Conference, ICFHR 2022, Hyderabad, India, December 4–7, 2022, Proceedings, page 122–136, Berlin, Heidelberg, 2022. Springer-Verlag.
  * [59] R. Plamondon and S.N. Srihari.  Online and off-line handwriting recognition: a comprehensive survey.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(1):63–84, 2000.
  * [60] Daniel Pototzky, Azhar Sultan, and Lars Schmidt-Thieme.  Fastsiam: Resource-efficient self-supervised learning on a single gpu.  In Björn Andres, Florian Bernard, Daniel Cremers, Simone Frintrop, Bastian Goldlücke, and Ivo Ihrke, editors, Pattern Recognition, pages 53–67, Cham, 2022. Springer International Publishing.
  * [61] Senthil Purushwalkam and Abhinav Gupta.  Demystifying contrastive self-supervised learning: Invariances, augmentations and dataset biases, 2020.
  * [62] Ali Razavi, Aaron van den Oord, and Oriol Vinyals.  Generating diverse high-fidelity images with vq-vae-2, 2019.
  * [63] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma.  Pixelcnn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications, 2017.
  * [64] Mohammad Abuzar Shaikh, Mihir Chauhan, Jun Chu, and Sargur Srihari.  Hybrid feature learning for handwriting verification.  In 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR), pages 187–192, 2018.
  * [65] S.N. Srihari, Sung-Hyuk Cha, H. Arora, and Sangjik Lee.  Individuality of handwriting: a validation study.  In Proceedings of Sixth International Conference on Document Analysis and Recognition, pages 106–109, 2001.
  * [66] Sargur N. Srihari, Barish Srinivasan, and Kartik Desai.  Questioned document examination using cedar-fox.  Journal of Forensic Document Examination, 28:15–26, Dec. 2018.
  * [67] Igor Susmelj, Matthias Heller, Philipp Wirth, Jeremy Prescott, and Malte Ebner et al.  Lightly.  GitHub. Note: https://github.com/lightly-ai/lightly, 2020.
  * [68] Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther.  Ladder variational autoencoders, 2016.
  * [69] Lucas Theis and Matthias Bethge.  Generative image modeling using spatial lstms.  In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 2, NIPS’15, page 1927–1935, Cambridge, MA, USA, 2015. MIT Press.
  * [70] Benigno Uria, Marc-Alexandre Côté, Karol Gregor, Iain Murray, and Hugo Larochelle.  Neural autoregressive distribution estimation, 2016.
  * [71] Aaron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu.  Pixel recurrent neural networks, 2016.
  * [72] Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu.  Conditional image generation with pixelcnn decoders, 2016.
  * [73] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding, 2019.
  * [74] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu.  Neural discrete representation learning, 2018.
  * [75] Jingyao Wang, Luntian Mou, Changwen Zheng, and Wen Gao.  Cssl-rha: Contrastive self-supervised learning for robust handwriting authentication, 2023.
  * [76] Liwei Wang, Alexander G. Schwing, and Svetlana Lazebnik.  Diverse and accurate image description using a variational auto-encoder with an additive gaussian encoding space, 2017.
  * [77] Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Zhicheng Yan, Masayoshi Tomizuka, Joseph Gonzalez, Kurt Keutzer, and Peter Vajda.  Visual transformers: Token-based image representation and processing for computer vision, 2020.
  * [78] Tete Xiao, Xiaolong Wang, Alexei A. Efros, and Trevor Darrell.  What should not be contrastive in contrastive learning, 2021.
  * [79] Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatrick.  Improved variational autoencoders for text modeling using dilated convolutions, 2017.
  * [80] Chun-Hsiao Yeh, Cheng-Yao Hong, Yen-Chi Hsu, Tyng-Luh Liu, Yubei Chen, and Yann LeCun.  Decoupled contrastive learning, 2022.
  * [81] Thomas Yerxa, Yilun Kuang, Eero Simoncelli, and SueYeon Chung.  Learning efficient coding of natural images with maximum manifold capacity representations, 2023.
  * [82] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny.  Barlow twins: Self-supervised learning via redundancy reduction, 2021.
  * [83] Jiachen Zhu, Rafael M. Moraes, Serkan Karakulak, Vlad Sobol, Alfredo Canziani, and Yann LeCun.  Tico: Transformation invariance and covariance contrast for self-supervised visual representation learning, 2022.

##  Appendix A Appendix Section

###  A.1 Conditions for Good Representation

The methods for Self-Supervised Learning highlighted above have shown great
potential in generating useful visual representation $h$ using data
augmentation on input $x$ and then using contrastive loss for comparison. Most
of these methods have been benchmarked using standard academic datasets like
ImageNet [50] (1.3M images with 1k classes) and Places365 [54] (1.8M images
with 365 classes). The practical applicability of contrastive self-supervised
learning are mentioned in [25] where the authors have observed four conditions
for which self-supervised contrastive learning methods produce good feature
representations $h$. Below we have elaborated the four conditions and how they
relate to the downstream application of Handwriting Verification:

  1. 1.

Data Quantity: Limited pre-training data with smaller networks allows for
faster and cheaper pre-training since the impact on downstream application is
within tolerance. In context of handwriting verification, despite the
abundance of unlabeled handwritten images the model should be able to learn
good representation $h$ even from a smaller subset of handwritten dataset $x$.

  2. 2.

Pre-Training Domain: Representations learned during pre-training benefits from
having same domain type as the downstream application. Adding representations
from other domain generalizes the features space $h$. In the case of
handwriting verification task, we can use CEDAR Letter Handwriting Dataset for
feature extraction (pre-training) $f(x)$ since the features generated are
within-domain and relevant to the handwritten text. Generalizing feature space
$h$ with other non-handwritten text like images from ImageNet [50] which
broaden the feature space leading to high variability from unrelated input
domains.

  3. 3.

Data Augmentation Quality: With Data Augmentation the input is altered
$x^{\prime}$ to create contrastive examples which helps in learning self-
supervised representations. While certain augmentation techniques help enhance
generalization within specific domains, it is important to maintain
consistency and natural variability for the inherent patterns observed in the
handwritten styles. Certain transformations like changing aspect ratio,
elastic distortions could alter the handwritten style. Hence, we have
emperically evaluated the data augmentation techniques for handwriting
verification in context of contrastive self-supervised pre-training.

  4. 4.

Downstream Task Granularity: Self-Supervised Contrastive methods have shown
better applicability with coarse-grained downstream tasks involving higher-
level attributes as compared to fine-grained tasks. Handwriting verification,
benefits from both coarse-grained and fine-grained features. Coarse-grained
features are responsinable in extracting high-level handwriting characterstics
such as overall shape, slantness, curvation and spatial layout. Whereas, fine-
grained features capture details such as characters shapes (e.g. staff of a),
stoke thickness, order, character variations and so on. Hence, in this chapter
we show a contrastive self-supervised approach for handwriting verification.
This methods is pre-trained on unlabeled handwritten dataset $f(x)$ to
generate both coarse grained and fine-grained features which represents the
individuality of writers $p(y|x1,x2)$ within the handwritten domain.

Figure 6: Original, Masked and Decoded handwritten image samples from multiple
writers 1471 first sample B with 64x64 size with 32x32 masked patches and
masking ratio set to 0.2

###  A.2 Related Generative AR Models

PixelRNN [71] uses Long Short Term Memory (LSTM [37]) trained with masked
convolutions and multinomial loss function to predict raw pixel values (0-255)
as categories. PixelCNN [71] was proposed to make the PixeRNN efficient and
capture a fixed size receptive field using convolutional layers.

Gated PixelCNN [72] efficiently uses masked convolutions to model conditional
pixel distribution. The masked convolutions replaces the ReLu activation using
gated activation function as shown in Equation 17. This allows complex
interactions between pixel $x_{i}$ to be modeled simile to PixelRNN [71].

$\mathbf{y}=\tanh\left(\mathbf{W}_{f}\mathbf{x}\right)\odot\sigma\left(\mathbf{W}_{g}\mathbf{x}\right)$
(17)

where, $\sigma$ is sigmoid activation for $k^{th}$ layer and $\odot$ is
element wise multiplication between the result of the convolutions between the
two receptive fields. PixelCNN++ [63] and PixelSNAIL [22] improves Gated
PixelCNN [72], with improvements like using output mean and standard
deviations for the logistics instead of softmax in the mixture. PixelCNN have
slow sampling rate since product of conditional is a time expensive operation
because $n^{2}$ forward model propagation is needed. AR models do not
explicitly learn the latent representations $h$ like Flow based models or
AutoEncoders (AE) but we can still apply feature aggregation techniques to
derive hidden representations $h$.

iGPT [17] more recently has explored the use of auto-regressive objectives
$L_{AR}$ (e.g. PixelCNN objective [71]) and BERT [28] $L_{BERT}$ objective
using transformer architecture. $L_{AR}$ and $L_{BERT}$ have worked well with
Large Language Models for natural language (e.g. GPT2 [2] and similar) which
uses masks word tokens in its pre-training regime which has shown great
generalization and improvement for the downstream task with fewer labels.
Given input data $X$ the AR objective $L_{AR}$ was to minimize the negative
log likelihood $\underset{x\sim X}{\mathbb{E}}[-\log p(x)]$ of input
distribution $p(x)$ which is a density function modeled using product of pixel
conditionals $\prod_{i=1}^{n}P(x_{i}|(x_{1},..,x_{i-1}))$. iGPT also considers
the use of BERT [28] objective $L_{BERT}$ where the model is trained to
minimize the negative log-likelihood of $p(x)$ to learn predicting masked part
of the input $x_{m}$ using the unmasked part of the input $x_{[1,n]\backslash
M}$ as shown in the Equation shown in Equation 18 below:

$L_{BERT}=\underset{x\sim X}{\mathbb{E}}\underset{M}{\mathbb{E}}\sum_{i\in
M}\left[-\log p\left(x_{i}\mid x_{[1,n]\backslash M}\right)\right]$ (18)

###  A.3 Related Generative Flow based Models

Flow based generative models explicitly learns the true data distribution
$p(x)$ by a sequence of invertible transformation functions $f(x)$ to map
input $x$ to latent representation $z$. Since the transformation are
invertible $x=f^{-1}(z)$ is true. [52] provides an intuitive understanding of
how flows model input density function $p(x)$ using the Equations 19 below:

$\begin{split}\begin{split}\begin{split}\int p_{x}(x)dx&=\int
p_{z}(z)dz=1\hskip 23.33147pt\text{. . Probability Definition}\\\
p_{x}(x)&=p_{h}(z)\left|\frac{dz}{dx}\right|=p_{z}(f(x))\left|\frac{df(x)}{dx}\right|\hskip
2.84544pt\text{. . Derivative}\\\ \log p_{x}(\mathbf{x})&=\log
p_{z}(f(\mathbf{x}))+\log{}\left|\det\frac{df(\mathbf{x})}{d\mathbf{x}}\right|\hskip
2.84544pt\text{. . Logarithm}\end{split}\end{split}\end{split}$ (19)

It is difficult to find $f^{-1}$ and
$\log{}\left|\det\frac{df(\mathbf{x})}{d\mathbf{x}}\right|$ of complex $f$
hence flow models use sequence of invertible functions $f_{1,..,K}$.

NICE [29] Non-Linear Independent Component estimates $p(x)$ uses an easy
invertible function $f^{-1}$ and easy jacobian
$\log{}\left|\det\frac{df(\mathbf{x})}{d\mathbf{x}}\right|$ computation. NICE
uses dequantized version of input images $x$ by adding uniform noise $u$ and
re-scaling input data between $[-1,1]$ to prevent degenerate solution because
of infinite high likelihood placed on discrete points. NICE seperates
dequantized input space into odd $x^{(i)}_{1}$ and even $x^{(i)}_{2}$
dimensions. Coupling function is then applied as shown in Equation 20 below:

$\begin{split}\begin{split}z^{(i)}_{1}&=x^{(i)}_{1}\\\
z^{(i)}_{2}&=x^{(i)}_{2}+m^{(i)}x^{(i)}_{1}\\\ z&=\text{exp}(s)\odot
z^{(4)}\end{split}\end{split}$ (20)

Each coupling layers $m^{(i)}$ ($i$= 1 to 4) have rectified linear units. The
output of $m^{4}$ coupling layer is element-wise multiplied by scaling matrix
$s$ to preserve the relative importance across dimensions in $x$. The loss is
the negative log likelihood of Equation 2.

RealNVP [30] uses real-valued non-volume preserving transformations $f$ which
are stable, invertable and learnable for computing exact $p(x)$. Similar to
NICE [29], RealNVP aims to provide a tractable and flexible solution to
computing Jacobian determinant of the transformation. RealNVP uses affine
coupling layer on input $x$ with dimensionality $D$ as shown in Equation 21
below:

$\displaystyle y_{1:d}$ $\displaystyle=x_{1:d}$ (21) $\displaystyle y_{d+1:D}$
$\displaystyle=x_{d+1:D}\odot\exp\left(s\left(x_{1:d}\right)\right)+t\left(x_{1:d}\right)$

where, $s$ and $t$ stands for scale and translation. The Jacobian of the
affine coupling is the product of diagonals which are efficiently computed to
be independent of scale $s$ and translation $t$. RealNVP uses deep
convolutional neural networks to make complex $s$ and $t$. It uses masked
convolutions to exploit the local correlation within input image and couples
layers in an alternating pattern to overcome stagnant output post
transformation. It also uses multi-scale architecture to overcome the problem
of high-compute needed to generate high dimensional latent space $z$. It
applies squeezing and split operation to reduce the dimensionality of the
image by certain factor as needed. Similar to NICE [29], the loss function is
simply the negative log likelihood of Equation 2. Similar, to Nice and
RealNVP, Glow [47] proposes to use invertible 1x1 convolution within coupling
function. Within handwritten verification domain, a flow based model helps in
capturing characteristics of the handwriting, such as stroke patterns,
curvature and other writing styles which can be used as latent features by a
discriminator network for generating a similarity metric.

###  A.4 Related VAE based Models

Within VAE, latent representations $z$ are sampled from an approximate
posterior $q_{\phi}(z|x)=\mathcal{N}(\mu,\sigma^{2})$. The encoder represents
input $x$ using $z$ with help of a normal distribution with mean function
$\mu$ and standard deviation as $\sigma^{2}$. Posterior collapse happens when
$\mu$ and $\sigma$ collapse to a constant values $C_{\mu}$ and $C_{\sigma}$.
Hence, approximate posterior distribution $q_{\phi}(z|x)$ is only dependent on
prior $q_{\phi}(z)$ which is fixed to a normal distribution of
$\mathcal{N}(C_{\mu},C_{\sigma^{2}})$. The result of the collapse, is that the
latent representation $z$ becomes independent of $x$ thereby the decoder only
samples noisy $z$ samples from a incorrect approximate posterior
$q_{\phi}(z)=\mathcal{N}(C_{\mu},C_{\sigma}^{2})$. This makes the decoder
impossible to use the information $x$ from the latent representation $z$. Many
approaches have been proposed in the literature to tackle the problem of
posterior collapse using KL annealing for task-specific representation
learning: Generating sentences [12], sketch-rnn[40], dilated convolutions in
VAW [79], AG-CVAE[76]. More recetly, Ladder VAE [68], fixing ELBO [1], VQ-VAE
[74] have been proposed to help tackle the collapse problem with VAE.

VQ-VAE [74] is one of the notable mentioned within the VAE family of models.
VQ-VAE is a discrete latent VAE model. VQ-VAE merges auto-regressive
distributions within the decoder. VQ-VAE defines a codebook of latent
embdedding space $e$ with size $K$ and dimensionality $D$. Input $x$ is passed
into encoder to generate latent representation $z_{e}=t_{enc}(x)$. The
representation $z_{e}$ is descritized $z_{q}$ using the codebook $e$ by
mapping $z_{e}$ to the nearest element in embedding $e_{j}$ using l2 distance
$z_{q}$=argmin${}_{j}\|z_{e}-e_{j}\|_{2}$. During forward propagation nearest
$z_{q}(x)$ is passed to the decoder$f_{dec}(z_{q})$. While ile in backward
pass the gradient the gradients are unaltered and passed to encoder. VQ-VAE
loss is as shown in the Equation 22 below:

$L=\log p\left(x\mid
z_{q}(x)\right)+\left\|\operatorname{sg}\left[z_{e}(x)\right]-e\right\|_{2}^{2}+\\\
\beta\left\|z_{e}(x)-\operatorname{sg}[e]\right\|_{2}^{2}$ (22)

First term in Equation 22 is the reconstruction loss used by the encoder and
decoder to reconstruct input $x$ from $z_{q}(x)$. The second term is the
$l_{2}$ error to update the embedding vector $e_{i}$ closer to $z_{e}(x)$
which is used to optimize the embeddings and the last term is a commitment
loss to make sure that the encoder output $z_{e}(x)$ grows in same proportion
as the codebook embeddings $e$. $\operatorname{sg}$ stop gradient is identity
at forward compute and zero derivatives during gradient computation. VQ-VAE2
[62] was proposed as an improvement to generate synthetic samples of higher
coherence and fidelity using multi-scale hierarchical organization of VQ-VAE
with priors over latent variables which generated samples that rival
Generative Adversarial Networks [36] (GANs) on ImageNet [50] datasets.

###  A.5 Related SimCLR and Contrastive based Models

Although, SimCLR got state-of-the-art SSL performance on standard datasets
like ImageNet there are some weaknesses like sensitivity to augmentation
techniques and need for large batch size with lot of negative examples to
improve performance. Building on top of SimCLR are many approaches are:
Decoupled Contrastive Learning (DCL) [80] which defines a DCL loss by removing
the positive pairs from the denominator of the InfoNCE [73] loss. Nearest
Neighbor Contrastive Learning (NNCLR) [33] also builds upon the idea from
SimCLR [18] and have proposesed to use nearest-neighbours to get diverse set
of positive pairs. Other interesting approaches to contrastive SSL are:
Swapping Assignment with multiple Views (SWAV) [13], Masked Siamese Networks
(MSN) [5], Prior Matching for Siamese Networks (PMSN) [4], Synchronous
Momentum Grouping (SMoG) [57], Transformation Invariance and Covariance
Contrast (TiCo) [83], Self-Supervised Learning of Maximum Manifold Capacity
Representations (MMCR) [81].