# On the Pros and Cons of Momentum Encoder in Self-Supervised Visual
Representation Learning

Trung Pham<EMAIL_ADDRESS>Chaoning Zhang  Axi Niu  Kang Zhang  Chang D.
Yoo<EMAIL_ADDRESS>
###### Abstract

Exponential Moving Average (EMA or momentum) is widely used in modern self-
supervised learning (SSL) approaches, such as MoCo, for enhancing performance.
We demonstrate that such momentum can also be plugged into momentum-free SSL
frameworks, such as SimCLR, for a performance boost. Despite its wide use as a
basic component in the modern SSL frameworks, the benefit caused by momentum
is not well understood. We find that its success can be at least partly
attributed to the stability effect. In the first attempt, we analyze how EMA
affects each part of the encoder and reveal that the portion near the
encoder’s input plays an insignificant role while the latter parts have much
more influence. By monitoring the gradient of the overall loss with respect to
the output of each block in the encoder, we observe that the final layers tend
to fluctuate much more than other layers during backpropagation, i.e. less
stability. Interestingly, we show that using EMA to the final part of the SSL
encoder, i.e. projector, instead of the whole deep network encoder can give
comparable or preferable performance. Our proposed projector-only momentum
helps maintain the benefit of EMA but avoids the double forward computation.

###### keywords:

Self-Supervised Learning , Exponential Moving Average , Momentum Encoder ,
Representations Learning, Partial Update

††journal: PR for Review

##  1 Introduction

Self-supervised learning (SSL) is known for its capability to learn
representation with unlabeled data. SSL has been widely applied in NLP [1, 2,
3, 4, 5] and computer vision [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16]. We find
that multiple seminal SSL frameworks, MoCo [10, 17, 18], BYOL [19], DINO [20],
and ReSSL [21] all use momentum to form a teacher-student paradigm where the
teacher encoder is updated from the student model with exponential moving
average (EMA). To avoid any confusion, we use either “EMA” or “momentum”
interchangeably when mentioning the EMA [22].

In the literature, EMA encoder has been motivated mainly for two purposes: (a)
to generate consistent representations for negative pairs [10, 17, 21]; and
(b) avoid collapse in negative-free SSL frameworks [20, 19]. These motivations
have been challenged by the claims in follow-up works [15, 11]. For example, a
large number of negative samples might not be necessary, rendering the EMA
unnecessary from the perspective of negative samples [15]. SimSiam [11] also
shows that EMA is not necessary for preventing a collapse in the negative-free
SSL frameworks. This motivates a need for understanding EMA from a new
perspective for explaining its empirical benefit in boosting performance.

We investigate and find that such a performance boost can be partly explained
by the stability effect. In practice, unfortunately, this benefit of the
stability effect does not come free in the sense that it doubles the forward
computation resources. Note that the core of SSL lies in learning an
augmentation-invariant representation, thus the optimization goal is often
designed to make two images augmented from the same image have a similar
representation. Specifically, the representation of another image augmented
from the same image is used as the target for optimizing the network. Applying
momentum on the teacher encoder requires additional forward computation
resources as well as memory for obtaining the target representation.

To this end, we investigate the influence of momentum on different parts of
the encoder. Specifically, we disentangle the momentum in the teacher network
encoder into different parts and investigate the effect of momentum for each
stage (block) of the whole encoder network. We find that the benefit of
momentum differs massively at different stages of the encoder. The EMA in the
early stages (close to the input) has an insignificant role, while that in the
latter stages (close to the output) is much more significant. We find that
this can be attributed to much higher yet fluctuating gradient values which
cause more instability.

Interestingly, we find that only applying EMA to the final projector of the
encoder is sufficient enough to yield a comparable or sometimes preferable
performance, especially when its momentum coefficient is set to be more
aggressive. A merit of applying EMA only on the projector is that the forward
propagation on the backbone can be shared, which only causes double forward
propagation on the light projector.

Compared with the forward computation on the backbone, the computation
overhead on the projector is negligible. Therefore, in this work, we suggest
projector-only momentum as Fig. 1b) for maintaining the benefit of EMA but
avoiding the double forward computation overhead. To the best of our
knowledge, we are the first to investigate the impact of the momentum encoder
given the recent progress of SSL.

Overall, the contributions of our work are summarized as follows:

  1. 1.

Recognizing that the existing motivations for introducing momentum into the
SSL frameworks are refuted by follow-up works, we investigate with experiments
and show that its empirical benefit for boosting performance can be partly
attributed to the stability effect.

  2. 2.

We explore and reveal an intriguing observation that the benefit of applying
momentum in the later stages of the encoder is significantly higher than that
in the early stage. Moreover, we provide a gradient analysis to justify this
phenomenon.

  3. 3.

We note that the benefit of momentum comes at the cost of causing double
forward computation. Motivated by the above observation, we propose a
projector-only momentum for maintaining the pros of momentum while mitigating
its cons.

(a) EMA with whole network (BYOL [19]) (b) EMA with projector-only (Ours)

Figure 1: Comparing traditional EMA updates to the whole target network (a)
and EMA updates on a final part of the target network (b). We investigate the
EMA effect on each separate part of the entire encoder. Fig. b) shows the case
when EMA is applied on the projector, all other parts are kept sharing for
online and target network. Here, $x_{1}$ and $x_{2}$ are two random
augmentations of the same image.

##  2 Background

###  2.1 Full EMA in Self-Supervised Learning

Given an unlabeled image $\mathbf{x}^{i}$, two random augmentations
$\mathbf{x}^{i}_{1}$ and $\mathbf{x}^{i}_{2}$ are generated and formed as a
positive pair. We consider an encoder $E_{\theta}$ which includes backbone
$b(.)$ and an MLP projector $g(.)$ for SSL frameworks in general [11, 20, 10,
23, 24, 21], and plus an additional MLP predictor $h(.)$ for BYOL [19] or
SimSiam [23] as illustrated in Fig. 1a). With the input image $\mathbf{x}_{1}$
and $\mathbf{x}_{2}$, the representations after backbone $b(.)$ are as
follows:

$f_{1}=b(x_{1}),f_{2}=b(x_{2}).$ (1)

And outputs for the momentum backbone $b_{m}(.)$ are computed by:

$f_{1m}=b_{m}(x_{1}),f_{2m}=b_{m}(x_{2}).$ (2)

Note that, subscript ‘m’ denotes the momentum version. Similarly, we feed
$f_{1},f_{2},f_{1m},f_{2m}$ to projector $g(.),g_{m}(.)$ to have the projected
outputs:

$z_{1}=g(f_{1}),z_{2}=g(f_{2}),z_{1m}=g_{m}(f_{1m}),z_{2m}=g_{m}(f_{2m}).$ (3)

Finally, the predictor is applied to the projections of the online network
$h(.)$ only, i.e. no momentum version for the predictor:

$p_{1}=h(z_{1}),p_{2}=h(z_{2}).$ (4)

For other SSL frameworks such as Moco, DINO, and ReSSL the projections
$z_{1},z_{2}$ will be used for the objective function. Here BYOL loss is taken
into account for example, where additional predictor outputs are used for
computing the objective loss. With the distance between the anchor $p^{i}_{1}$
(or $p^{i}_{2}$) and the target $z^{i}_{2m}$ (or $z^{i}_{1m}$) measured by
cosine similarity with operator ($\cdot$), the objective function of BYOL [19]
within a mini-batch $N$ samples is defined as follows:

$\mathcal{L}=-\frac{1}{2}(\frac{1}{N}\sum_{i=1}^{N}(p^{i}_{1}\cdot
z^{i}_{2m})+\frac{1}{N}\sum_{i=1}^{N}(p^{i}_{2}\cdot z^{i}_{1m})),$ (5)

where $p^{i}_{1}$ and $z^{i}_{1m}$ denote the output of the predictor $h(.)$
and output at the momentum projector $g(.)$ of the augmented image
$\mathbf{x}^{i}_{1}$, respectively. In the same way, $p^{i}_{2}$ and
$z^{i}_{2m}$ denote the notations for the second augmented image
$\mathbf{x}^{i}_{2}$. Note that here each augmented image is fed into the full
momentum encoder, i.e. both momentum backbone ($b_{m}$) and projector
($g_{m}$).

It is also worth noticing that all $p_{1}^{i},p_{2}^{i},z^{i}_{1m},z^{i}_{2m}$
are $l_{2}$-normalized, and BYOL uses a symmetric loss as default. It is also
worth to note that $z^{i}_{1m}$ and $z^{i}_{2m}$ here are the outputs of the
momentum encoder in BYOL (or stop gradient in SimSiam [23]), the subscript $m$
is put beside index $i$ to indicate it is momentum output.

Obviously, the BYOL loss only allows the gradient through $p^{i}_{1}$ and
$p^{i}_{2}$, not $z^{i}_{1}$ and $z^{i}_{2}$. Follow [19], we term online
network for the branch that allows the gradient update [25] and target network
for the branch that is updated with EMA. It can also be referred to as student
and teacher network [20], respectively. The online network is specified by the
parameters $\theta$ which allows updating gradient with back-propagation
during training [25], and the target network is modeled by the parameters
$\xi$. The momentum update rule for the parameters $\xi$ as follows [19]:

$\xi\leftarrow\beta\xi+(1-\beta)\theta,$ (6)

where a constant $\beta\in[0,1]$ is the momentum coefficient, which is often
chosen with value of 0.99 for short training (200 epochs). In longer training,
i.e. 1000 epochs, SSL methods often use the higher momentum value, i.e.
$\beta=0.996$ for BYOL [19, 23] and DINO [20], or $\beta=0.999$ for MoCo [10,
17].

###  2.2 EMA Projector in Self-Supervised Learning

In this new setting, instead of using backbone momentum outputs
$f_{1m},f_{2m}$, the outputs of the online backbone $f_{1},f_{2}$ are used to
feed into the momentum projector $g_{m}(.)$:

$z_{1m}=g_{m}(f_{1}),z_{2m}=g_{m}(f_{2}).$ (7)

The final objective loss formula is kept the same as the baselines, i.e. Eq.
5. For the other frameworks, the EMA projector is performed the same as
described above.

##  3 Towards Understanding Momentum in SSL

###  3.1 Momentum Brings Performance Boost

We bring four widely used momentum-based SSL methods including BYOL [19],
ReSSL [21], MoCo v2 [17], and DINO [20] to conduct comprehensive experiments
to analyse the EMA encoder effect. Furthermore, to generalize our findings, we
also inject the momentum on two non-momentum-based frameworks, SimCLR [11] and
SimSiam [23].

We conduct two experiments to compare the influence of momentum on the quality
of the representations learned by the SSL encoders. First, the experiments are
conducted for each method that uses the momentum with optimal parameters, and
second, we remove the momentum by setting the momentum coefficient $\beta=0$
given stop gradient [23] which means sharing the weight between online and
target network.

We measure the performance using top-1 accuracy (%) on the test set as the
common protocol in SSL, i.e. BYOL [19]. Tab. 1 shows that without momentum
encoder, all momentum-based methods dramatically drop performance (BYOL, DINO,
MoCo v2, and ReSSL). Besides, non-momentum-based frameworks, i.e. SimSiam and
SimCLR get improvement with the existence of momentum.

Specifically, when the momentum of the target network is removed, i.e.
$\beta=0$, MoCo v2, DINO, and ReSSL are not able to train, their accuracy is
trivial ($\sim 2\%$, not better than a random guess) while BYOL can be able to
train but with the visible degradation on accuracy (43.48%) comparing to the
baseline that uses EMA (62.01%).

Table 1: Effect of EMA in various SSL frameworks. We use ResNet-18 on
CIFAR-100, train for 200 epochs. Plugging momentum consistently improves the
performance compared to the version without it. We report top-1 accuracy.
Notations, CE: cross-entropy, CL: contrastive learning.

Method | Architecture / Loss | Momentum | Coefficient $\beta$ | Remarks | Acc (%) |  $\Delta_{\text{acc}}$ (%)
---|---|---|---|---|---|---
BYOL [19] | Asymmetric / Non-CL | ✓ | $\beta=0.99$ | Baseline | 62.01 | -
- | $\times$ | $\beta=0$ | Remove | 43.48 | $-18.53$
SimSiam [23] | Asymmetric / Non-CL | $\times$ | $\beta=0$ | Baseline | 51.41 | -
- | ✓ | $\beta=0.99$ | Add | 53.66 | $+2.25$
DINO [20] | Symmetric / CE | ✓ | $\beta=0.99$ | Baseline | 58.12 | -
- | $\times$ | $\beta=0$ | Remove | 1.08 | $-57.04$
ReSSL [21] | Symmetric / CL | ✓ | $\beta=0.99$ | Baseline | 54.66 | -
- | $\times$ | $\beta=0$ | Remove | 1.56 | $-53.10$
MoCo v2 [17] | Symmetric / CL | ✓ | $\beta=0.99$ | Baseline | 62.34 | -
- | $\times$ | $\beta=0$ | Remove | 2.22 | $-60.12$
SimCLR [11] | Symmetric / CL | $\times$ | $\beta=0$ | Baseline | 60.45 | -
- | ✓ | $\beta=0.90$ | Add | 61.97 | $+1.52$

Note that, BYOL without momentum can be considered same as SimSiam framework
given some different configurations in projector and predictor [23]. With a
careful design, SimSiam [23] presented a special case of BYOL where $\beta=0$
that works fine however it cannot be comparable with BYOL in performance [19,
23]. We ran SimSiam using [26] and found its accuracy is 51.41% which
significantly outperforms the version without momentum in the BYOL framework
whose accuracy is 43.48% (Tab. 1). However, compared to the BYOL baseline that
uses momentum (accuracy 62.01%), SimSiam is far lower to catch up. By default,
SimSiam [23] and SimCLR [11] do not have EMA in its architecture. We performed
experiments by adding EMA to their target encoder and find that EMA boosts
SimSiam from 51.41% to 53.66% ($+2.25$%) and improves SimCLR from 60.45% to
61.97% ($+1.52$%) as shown in Tab. 1.

The analysis in this section demonstrates the significance of EMA in
guaranteeing learning and boosting the performance of a wide range of network
architectures. But, how can EMA help in SSL?

###  3.2 How Does Momentum Improve Performance?

Prior motivations. The use of EMA in the CL-based frameworks [10, 17, 21] is
for creating consistent negative-pairs representations, and avoiding
collapsing in the Non-CL-based frameworks [20, 19]. Their motivations have
been refuted by the follow-up works [15, 11] as the earlier discussion. This
motivates us with a new perspective for the empirical benefit of EMA as the
hypothesis below.

Hypothesis. Momentum helps the encoder learn from the past to make the
training more stable.

Empirical Analysis. To demystify the hypothesis that momentum can help to
stabilize the training by learning from the past, we monitor and visualize the
student and teacher weight when using momentum compared to the weight that is
not trained with momentum. BYOL [19] is chosen for experiments. We use the
ResNet-18 backbone, CIFAR-100, to train in 200 epochs. By default, with BYOL
is implemented in [26], the projector in the encoder has two FC layers:
FC1(512,4096), BN, ReLU, FC2(4096,256). To be able to visualize the weight, we
design the final FC layer of the encoder in BYOL to have two dimensions. To
this end, we change the hidden layer from 4096 to 2, i.e. FC1(512,2), BN,
ReLU, FC2(2,256), and all other settings are kept intact.

(a) Filter 60 (b) Filter 103 (c) Filter 133

Figure 2: Effect of the momentum to the weights (filters) of FC2. Filters are
visualized in 3D space. Without momentum, we observe the “abnormal point” at
step around 10k out of 40k (green line). The axis y, and z are weight 1 and
weight 2, respectively.

(a) Linear train accuracy (b) Linear test accuracy (c) KNN-1 test accuracy

Figure 3: Performance with EMA effects on CIFAR-100. The abnormal accuracy
around epoch 50 out of 200 (w/o momentum) happens when weights (filters) have
suddenly fluctuated at step 10k in Fig. Fig. 2. With momentum, this
instability is successfully prevented (black and blue lines).

At first, we conduct experiment without momentum ($\beta=0$). Second, we
perform EMA ($\beta=0.9$, and $\beta=0.99$) to only FC2 of the target
projector (Fig. 1b). Our goal is to observe what happens to the network weight
w/ and w/o momentum in the part that can be visualized. Note that, FC2(2,256)
in pytorch [25] means $input\\_dim=2$, $output\\_dim=256$, and weight $w$ has
shape of $256\times 2$. In other words, FC2 has a total of 256 filters with
$dim=2$ that can be visualized.

Tab. 2 shows that without momentum, the top-1 linear and KNN-1 test accuracy
are 20.01% and 10.01%, respectively. With momentum, it can significantly boost
the top-1 linear accuracy to 25.76% and KNN-1 to 14.17%. Interestingly, it
reveals that momentum actually has an effect even though EMA is applied to a
tiny part (FC2) of the target encoder. How can it help?

Table 2: The EMA is applied to the FC2.

Setup | Top-1 (%) | KNN-1 (%)
---|---|---
Fixed random encoder | 15.21 | 6.85
w/o momentum | 20.01 | 10.01
w/ momentum, $\beta=0.9$ | 25.76 | 14.17
w/ momentum, $\beta=0.99$ | 22.63 | 12.56

We find that with EMA ($\beta=0.9$), the weight of the online network is much
more stable than training w/o EMA as clearly shown in Fig. 2. Here we plot
five lines, green line denotes no momentum $\beta=0$; blue and red lines show
the student and teacher with momentum $\beta=0.9$; black and orange lines
illustrate the student and teacher with momentum $\beta=0.99$.

We visualize FC2 weights in 3D Fig. 2 where these weights are recorded in 200
epochs with a total of 40k steps. Fig. 3 shows the corresponding top-1 linear
accuracy with no momentum, momentum $\beta=0.9$, and momentum $\beta=0.99$. We
observe a sudden change of the weight in Fig. 2 which strongly corresponds to
the accuracy drop in Fig. 3.

More visualizations of filters (weights) are provided in Appendix. These
experiments show that using EMA into FC2 (shape $256\times 2$) whose weights
have only two dimensions can have a great influence on the stability in
training and thus increases performance by a large margin ($+5.75\%$). Note
that in optimal setting [26], FC2 has $4096$ dimensions ($256\times 4096$)
which is much more powerful than that in our visualization experiments.

This phenomenon motivates us to investigate the EMA effect in the other parts
of the encoder which have different capacities and parameters. To this end, we
dig into the main concern of the paper with two questions: 1) How does the EMA
affect each part of the network encoder? and 2) Does the EMA update to the
whole network or EMA update in some certain parts of the network perform the
best?

##  4 Impact of Partial EMA Update on Different Encoder Stages

Partial EMA Update. ResNet-18 [27] is a convolutional neural network with 18
layers whose design has four blocks. Note that, in ResNet-18, there has a
convolutional layer before the four residual blocks as illustrated in Fig.
1a). We inject EMA to each separate block to see how it influences the final
performance. BYOL [19] is chosen to investigate. All these experiments are
conducted in CIFAR-100 with 200 epochs and all evaluation protocols are
followed as [19].

The whole network of BYOL includes 1) backbone (ResNet-18), 2) Projector, and
3) Predictor. By default, the target encoder network of BYOL is updated EMA
for entire encoder, i.e. both backbone and projector.

We split backbone into different blocks as its design: Conv1, Block1, Block2,
Block3, and Block4 as shown in Fig. 1a). We decompose the parameters set
$\theta$ of the whole online network into a set of consequential parameters
$\theta_{1},\theta_{2},\theta_{3},\theta_{4},\theta_{5},\theta_{6},\theta_{7}$
and $\xi$ in the target network to separate parameters
$\xi_{1},\xi_{2},\xi_{3},\xi_{4},\xi_{5},\xi_{6}$ for different parts of the
encoder in Fig. 1a).

Note that each block contains four convolutional layers followed by a batch
normalization layer and ReLU [27]. We use EMA with $\beta=0.99$ as default
library [26] for each block of the target network while keeping all other
parts sharing weight with the corresponding parts of the online network, i.e.
$\beta=0$.

We also consider a case of applying EMA to a very small part at the beginning
of the whole network, here we choose the first convolutional layer of backbone
namely Conv1. Interestingly, we find that the portion near the input of the
encoder gives the least impact on performance, while the latter parts of the
encoder have much more influence.

As shown in Fig. 4 a), EMA in Conv1, the first part of the network gives an
accuracy of $43.07\%$ which is almost similar to the case not applying EMA
with the accuracy of $43.48\%$.

(a) Momentum in separate part (b) Momentum in each combination

Figure 4:  Effects of momentum in each individual part in the whole network
encoder BYOL. We report top-1 accuracy and train 200 epochs on CIFAR-100. Here
backbone is the ResNet-18 which has 4 blocks in its architecture. Conv1 means
that we take the first convolutional layer in the backbone. When applying
momentum to each part, all other parts will be sharing weight (i.e.
$\beta=0$).

The impact of EMA keeps increasing to the final layers. Here, applying EMA to
the final layer, i.e. projector which has only two FC layers can bring
significant improvement from $43.48\%$ to $59.61\%$ which is nearly comparable
to baseline which updates EMA in the whole deep encoder with $62.01\%$.

This phenomenon can be attributed to the fact that the earlier parts of the
network share the weight with the online network which updates faster to help
learn the latest information from the input and the latter final parts need to
slow down the network update for training more stable. We elaborate on this
phenomenon by analyzing the gradient for each block below. We also investigate
the EMA influence of each combination of some parts of the encoder. Applying
EMA on Block4+projector works comparable to EMA on projector. Combining Conv1
and four blocks (equal to backbone) can give the comparable accuracy to the
baseline (whole Network), i.e. $62.17\%$ vs. $62.01\%$, respectively, Fig. 4
b).

Overall, momentum/EMA on the final part of the SSLs encoder (i.e. projector)
has the most important influence on the performance in the classification
downstream task. For the SSL methods such as DINO, MoCo v2, and ReSSL, EMA in
the final part of the encoder is even more critical compared to the BYOL as
shown in Tab. 3 and Fig. 4.

A natural question is raised: why does EMA apply in the final parts has more
influence compared to the earlier parts of the encoder?

Gradient Analysis. Amongst SSL frameworks, we take a negative-free method,
such as BYOL [19] to perform gradient analysis for simplicity. We monitor the
gradient to each part of the network encoder to find some clues to elaborate
on why EMA has no or minor effect on the earlier but latter parts. For visual
observation, we record the $l_{\infty}-$norm gradient magnitude of the overall
loss with respect to the output of each block for the case shown in Fig. 2
with $\beta=0$ where the abnormal point shows the strong instability during
the training.

Fig. 5 clearly shows that the magnitude gradient in the final part of the
encoder i.e. projector has the most significance and is most fluctuated. The
earlier part such as block 4 has a much smaller gradient magnitude and is more
stable. And, the very beginning blocks such as block 1, block 2, and block 3
have the smallest magnitude gradients and are most stable.

This empirical observation explains why applying EMA (Eq. 6) to the final part
of the network encoder has the most influence compared to the previous stages
of the network encoder. In other words, making the weight in the final stages
learned from the past to stabilize the training can have the most effects on
the entire deep network encoder.

Figure 5: Gradient magnitude of each layer. It shows the gradients of latter
parts such as block 4 and projector of the encoder are more fluctuated.

Based on the above observation, we can also use additionally 1) a simple
technique such as BN [28] to the output of the projector and block 4 or 2)
freezing the target projector to stabilize the training to improve performance
[18]. We show the results when using EMA in the final stage projector (without
EMA to the deep backbone) with BN in the output of projector and/or block 4
can match the baseline BYOL [19] in the Appendix.

##  5 Projector-only Momentum in SSL

Double Propagation of Momentum. The benefit of momentum comes at the cost of
causing double forward computation [19, 10, 20, 21] compared to non-momentum-
based frameworks [11, 23]. Motivated by our previous analysis, we propose the
projector-only momentum for SSL frameworks which applies EMA only to the
projector, while sharing parameters for deep backbone as an example in Fig. 1
b) for BYOL. Our method causes only double propagation on the light projector
whose computation is negligible compared to that of the backbone (i.e.
ResNet-18 and ResNet-50).

Implementation Details. The implementation is based on the open library solo-
learn for SSL [26]. To investigate the momentum/EMA, we consider different SSL
methods, datasets, and ResNet backbone architectures. Three widely used
benchmark datasets are considered including: CIFAR-100 [29], ImagetNet-100
[30], and ImageNet-1K [31]. Different deep backbones (ResNet-18, ResNet-50)
are considered. We follow the common protocol for evaluation the SSL [19].

Training and Evaluation. The encoder is pre-trained in an unsupervised manner
on the training set of each dataset without labels. The transfer capability is
measured by the classification downstream task [19, 10]. To be specific, we
evaluate the pre-trained representations by training a linear classifier on
frozen representations with the corresponding test set [26]. Classification
accuracy is used as the performance metric. More details and setups for SSL
are provided in the Appendix.

Table 3: The EMA is applied to either backbone or projector for each SSL
framework. We report top-1 accuracy with ResNet-18 backbone on CIFAR-100. “-”
denotes sharing weight ($\beta=0$).

Method | EMA Backbone | EMA Projector | Acc (%)
---|---|---|---
BYOL [19] (baseline) | ✓ | ✓ | 62.01
| ✓ | - | 62.17
| - | ✓ | 59.61
DINO [20] (baseline) | ✓ | ✓ | 58.12
| ✓ | - | 8.11
| - | ✓ | 58.14
ReSSL [21] (baseline) | ✓ | ✓ | 54.66
| ✓ | - | 29.79
| - | ✓ | 53.51
MoCo v2 [17] (baseline) | ✓ | ✓ | 62.34
| ✓ | - | 45.84
| - | ✓ | 60.39

Projector-only Momentum SSLs. We investigate the effect of EMA to each part of
the EMA-based frameworks including DINO [20], MoCo v2 [17], ReSSL [21], and
BYOL [19]. Tab. 3 shows that, unlike BYOL, all other frameworks perform poorly
if EMA updates on the backbone only. By contrast, Tab. 3 reveals that EMA
applying on the final part of the target network, i.e. projector, all SSL
frameworks perform competitively with baselines.

Interestingly, with DINO, performing EMA only to the projector (3 layers) can
give the same performance as the baseline which needs to update EMA for every
layer in the whole deep encoder, 21 and 53 layers for ResNet-18, ResNet-50,
respectively. It suggests that EMA to the final part of the DINO encoder is
sufficient and it helps to reduce half of the parameter needed to store in the
memory for the deep momentum backbone and save half forwards while still
exploiting the benefit of the momentum via a shallow projector (3 layers).
Different from BYOL [19], EMA updates only to backbone of the other SSLs
witnesses the significant drop on performance as shown in Tab. 3.

Table 4: DINO with ImageNet-1K, 200 epochs using ResNet-50 backbone. Here “ep”
stands for the epoch, and “Fwd” denotes the number of forward to ResNet-50 of
each image.

Method | # EMA layers | Params (MB) | #Fwd | Time (min/ep) | Top-1 (%) | Top-5 (%)
---|---|---|---|---|---|---
DINO [20] | 53 | 47.0 | 2 | 56.5 | 67.9 | 88.3
EMA projector | 3 | 23.5 | 1 | 38.5 | 67.8 | 88.2

Large-Scale Dataset. We conduct the experiments of partial update EMA in the
larger scale dataset, i.e. ImageNet-100 [32] (Tab. 6) and ImageNet-1K [31]
(Tab. 4) whose image size is $224\times 224$. All methods are trained in 200
epochs. ResNet-18 and ResNet-50 are used for ImageNet-100 and ImageNet-1K,
respectively. Tab. 6 shows that results are consistent as the ones in
CIFAR-100 dataset. Specifically, we observe that for the large-scale dataset
of ImageNet-100, with $\beta=0.99$ applying EMA to only projector gives
slightly a decrease in accuracy while keeping competitive performance with
$\beta=0.9995$ for all SSL frameworks.

Table 5: Partial update EMA for SSLs frameworks with ResNet-50. We train 200
epochs on CIFAR-100. All methods is set with $\beta=0.99$. We report top-1
accuracy, “-” denotes sharing weight.

Method | EMA Backbone | EMA Projector | Acc (%)
---|---|---|---
BYOL [19] | ✓ | ✓ | 63.71
| - | ✓ | 59.88
MoCo v2 [17] | ✓ | ✓ | 65.12
| - | ✓ | 61.76
ReSSL [21] | ✓ | ✓ | 57.07
| - | ✓ | 56.10
DINO [20] | ✓ | ✓ | 53.04
| - | ✓ | 56.54

Applying EMA to the projector in DINO keeps the competitive performance of
$74.22\%$ compared to the baseline of $74.16\%$. Consistently for the larger
scale dataset, with highly aggressive EMA, i.e. $\beta=0.9995$, all methods
witness a dramatic drop whereas EMA projector-only has a slight drop and still
keeps the competitive performance. Result on ImageNet-1K is consistent for
DINO whose accuracy is maintained without EMA on deep backbone (Tab. 4). This
suggests that for DINO with ResNet, EMA on the projector is sufficient.

Table 6: The EMA is applied to whole encoder or projector for each SSL
framework with baseline $\beta=0.99$ and more aggressive update with
$\beta=0.9995$. We report top-1 accuracy (%). All SSL frameworks are run for
200 epochs with ResNet-18 on ImageNet-100, “-” denotes sharing weight. Our
method is less sensitive to the change of $\beta$ compared to baselines.

Method | EMA Backbone | EMA Projector |  $\beta=0.99$ (%) |  $\beta=0.9995$ (%) |  $\Delta_{\text{acc}}$ (%)
---|---|---|---|---|---
BYOL [19] | ✓ | ✓ | 76.60 | 71.18 | $\color[rgb]{1,0,0}-5.42$
| - | ✓ | 75.02 | 75.52 | $\color[rgb]{0,0,1}+0.50$
DINO [20] | ✓ | ✓ | 74.16 | 63.78 | $\color[rgb]{1,0,0}-10.38$
| - | ✓ | 74.22 | 72.02 | $\color[rgb]{0,0,1}-2.20$
ReSSL [21] | ✓ | ✓ | 74.02 | 61.48 | $\color[rgb]{1,0,0}-12.54$
| - | ✓ | 69.12 | 65.81 | $\color[rgb]{0,0,1}-3.31$
MoCo v2 [17] | ✓ | ✓ | 76.48 | 69.86 | $\color[rgb]{1,0,0}-6.62$
| - | ✓ | 74.31 | 72.06 | $\color[rgb]{0,0,1}-2.25$

Based on the learning curve, if training longer, i.e. 1000 epochs for
ImageNet-1K, the gap between DINO baseline and DINO with EMA on projector
reaches zero. We provide the training trend SSL frameworks in Appendix.

Deeper Backbone with ResNet-50. We keep the same projector and consider a
deeper backbone, i.e. ResNet-50 (3 times deeper than ResNet-18). Two datasets
are used: CIFAR-100, and ImageNet-1K. All methods are trained with 200 epochs,
$\beta=0.99$. With ImageNet-1K, we conduct experiments for DINO [20]. As shown
in Tab. 5 for a deeper backbone (ResNet-50), SSL frameworks such as BYOL [19],
MoCo v2 [17], and ReSSL [21] show the consistent results as the backbone
ResNet-18 which are investigated above where EMA to the projector can give the
competitive performance no matter how deep backbone is (18 and 50 layers).

Interestingly, for DINO [20] in CIFAR-100, baseline updates EMA to whole deep
encoder (backbone + projector) performs inferior than the case updates EMA
just to the projector, i.e. $53.04\%$ vs. $56.54\%$. In ImageNet-1K, Tab. 4
shows their performance are almost comparable ($67.9\%$ vs. $67.8\%$ in top-1
accuracy) where DINO needs to update EMA for total of 53 layers (backbone +
projector) compared to EMA on projector with only three layers in depth.

##  6 Computation and Processing Time

Amongst SSL frameworks, we find that DINO is the most stable method when
training with the EMA projector. Therefore, we focus on DINO in the setting
with ImageNet-1K, and the common use backbone is ResNet-50. We measure the
processing time per epoch based on two GPUs NVIDIA RTX A6000.

As shown in Tab. 4, our method EMA projector just needs half of the parameters
for the backbone (just reduce half forward computational overhead) and shows
significant efficiency in terms of process time per epoch while achieving the
same performance compared to DINO [20].

Concretely, the proposed EMA projector updates EMA only for 3 layers of the
projector and gets rid of the need for the forward to the backbone i.e.
ResNet-50 (with 50 layers in depth) which is an advantage compared to the
traditional DINO. Note that the core of SSL lies in learning representation
with multiple augmentations, and the total processing time may quickly become
a burden. Our analysis suggests a potential paradigm for future research in
self-supervised learning to solve the computational bottleneck, especially for
the labs with low hardware resources.

##  7 Discussion

Momentum Coefficient. It is expected that SSL frameworks give undesired
performance if the EMA update is very slow. However, we show that even with
the extremely slow update, EMA still can be maximally exploited to keep
competitive performance.

(a) $\beta=0.99$ (b) $\beta=0.9995$

Figure 6: Compare EMA updating to the whole encoder (baseline) and partial
update EMA in two cases: (a) with the optimal parameter of momentum
coefficient $\beta$ it shows that EMA in the projector can give the
competitive accuracy compared to baseline, and (b) with a more aggressive
update, only updating EMA to the projector (ours) performs much better than
baselines of all the considered frameworks, especially ReSSL and DINO give a
big gap lower than our scheme.

We consider two scenarios: 1) EMA updates for the whole network in a
conventional way, and 2) EMA partially updates to the only final part of the
network encoder, i.e. projector. And we set $\beta=0.9995$ for all SSL
baselines, which is slow enough to observe the differences. We find that with
the highly aggressive update of EMA, all SSL frameworks drop dramatically in
performance as shown in Fig. 6a), and Fig. 6b). Specifically, ReSSL [21]
degrades $12.20$% (from $54.66\%$ to $42.46\%$), DINO [20] impairs $19.63$%
compares to baseline (from $58.12\%$ to $38.49\%$), BYOL [19] decreases
$7.20$% (from $62.01\%$ to $54.81\%$), and MoCo v2 [17] drops $9.30$% (from
$62.34\%$ to $53.04\%$).

By contrast, the proposed partial EMA update witnesses the slightly drop on
the final accuracy such as ReSSL, DINO, BYOL, and MoCo v2 only decrease
$3.33$%, $4.36$%, $2.07$%, and $4.78$%, respectively. The dramatically dropped
performance of the conventional EMA because of the fact that a very high
$\beta$ causes the network to learn very slowly so that the encoder cannot
retrieve the new information from the new input augmentations promptly.

By contrast, with the new scheme, EMA updates only to the projector, while
keeping the backbone of the teacher’s weight the same as student which can
leverage the encoder to learn quickly in the first stages while making sure
the entire target encoder not updated too fast which helps to stabilize the
final part to improve performance.

Note that, given the highly aggressive EMA only to projector, the accuracy
slightly decreases from baseline but still is much higher than the cases using
EMA to the whole network. This shows that with the new scheme, EMA is still
exploited to get its benefit in extreme cases which the traditional EMA
approach cannot.

These experiments also show that the EMA projector is less sensitive to the
hyper-parameters of momentum coefficient compared to the traditional SSL
approaches.

Table 7: Freezing the target projector $g(.)$, we report top-1 accuracy with
ResNet-18 backbone on CIFAR-100 trained with 200 epochs.

Method | W/ predictor
---|---
$\eta=1.0$ | $\eta=4.0$
BYOL [19] | 62.01 | 58.01
Fixed $g(.)$ | 60.27 | 62.32

Learning Rate. The SSL encoders work with the appropriate small learning rate,
i.e. BYOL [19] uses $\eta=1.0$ as the optimal parameter in setting of [26].
With larger learning rate, i.e. $\eta=4.0$, BYOL’s performance drops from
baseline 62.01% to 58.01% ($\color[rgb]{1,0,0}-4\%$) as shown in Tab. 7. With
our analysis above, the most instability in the encoder comes from the final
part, i.e. the projector. We keep the projector of the target network freeze
during training, i.e. $\beta=1.0$, while sharing parameters in backbone. We
find that the encoder can learn well with a much larger learning rate. As
shown in Tab. 7, with $\eta=4.0$ keeping projector fixed can boost accuracy
from 60.27% to 62.32% ($\color[rgb]{0,0,1}+2.05\%$) which is slightly better
than BYOL. Note that BYOL requires a cost of double forward propagation to the
backbone momentum compared to ours that shares backbone but fixed projector.

##  8 Related Works

Exponential Moving Average. EMA/Momentum has been studied deeply for smoothing
the original sequence signal [33, 34, 35]. It becomes the widely used
technique in practices for most of fields ranging from optimization [36, 37,
38], reinforcement learning [39, 40, 41], knowledge distillation [30, 42],
recent semi-supervised learning frameworks [22, 43, 44, 45], and self-
supervised learning methods [19, 10, 20, 46]. Momentum is interpreted as an
average of consecutive $q$-functions in reinforcement learning [39], or is
used in SSL frameworks preventing model collapse [20, 19, 47].

Self-Supervised Learning. Recently, SSL has been extensively studied in many
filed applications [48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 13, 58, 59, 60,
61, 62, 63, 64]. SSL recently becomes an emerging paradigm and key research
field thanks to its unique advantages over the traditional supervised learning
approaches that does not require any human manual supervision and costly
labelling process while showing the superior capability in learning
representations for downstream tasks such as classification, segmentation, and
object detection [11, 65, 10, 17, 18, 20, 19, 66, 24, 67].

Amongst those seminal works, MoCo [17, 18], DINO [20], ReSSL [21], and BYOL
[19] are the state-of-the-art SSL frameworks that use the EMA technique in the
target encoder. There are several SSL frameworks such as SimCLR [11], SimSiam
[23], and Barlow Twin [68] do not use EMA in its architecture.

Although those non-momentum-based methods exhibit the inferior performance in
classification i.e. in ImageNet-1K [31] or other downstream tasks compared to
the momentum-based approaches [17, 65, 20], they are still considered in
practice for certain cases due to their simpler design, i.e. without the need
for storing parameters and forward of the momentum encoder [12].

##  9 Conclusion

In this paper, we investigate a new perspective on the benefit of momentum and
show that its empirical benefit for boosting performance can be at least
partly attributed to the stability effect. We reveal an intriguing observation
that the benefit of applying momentum in the later stages of the SSL encoder
is significantly higher than that in the early stage and provide a gradient
analysis to justify the phenomenon. We note that the benefit of momentum comes
at the cost of causing double forward computation. Motivated by the above
observation, we propose a projector-only momentum method for maintaining the
pros of momentum while mitigating its cons in the SSL frameworks, which is a
potential direction for future research in solving the bottleneck of SSL in
terms of computational cost.

Nevertheless, in this work, we focus on analyzing the behaviour of momentum
with the ResNet backbones for image classification. The other complicated
backbones such as ViT as well as other modalities such as text, audio, and
video are also interesting to investigate. We let it for future work.

##  Appendix A Experimental Details

We directly clone the official code from the open library for SSL [26] and
keep everything as default settings such as augmentation settings, projectors,
predictors, and hyper-parameters in the bash files.

For datasets CIFAR-10, and CIFAR-100, we use a single GPU of NVIDIA TITAN Xp
with batch size 256. For ImageNet-100, we use two GPUs of NVIDIA TITAN Xp with
batch size 128 for each GPU. For ImageNet-1K, we use two GPUs of NVIDIA RTX
A6000, batch size 128 for each GPU. Except for other notices, all experiments
are conducted for 200 epochs.

We use a cosine learning rate schedule with the warm-up in 10 first epochs
[19]. All methods are trained without using labels. The representation quality
is evaluated by training an online linear classifier with the label on the
train set (frozen on backbone features) and evaluated with the label on the
test set. As shown in the solo-learn [26] frameworks, the performance gap
between online and offline linear evaluation is not significant. For the
convenience to avoid the need of retraining a linear classifier after the
encoder pre-training, we directly report top-1 accuracy (%) on the test
dataset with the online linear evaluation. This also facilitates observing the
performance during the training.

For one experiment with ImageNet-1K [31], it took 7 days 16h 56m 36s to train
DINO [20] with ResNet-50 backbone for 200 epochs (2 NVIDIA RTX A6000 GPUs).
For ImageNet-100, it took 19h 33m 58s to train DINO with ResNet-18 backbone
for 200 epochs (2 NVIDIA TITAN Xp GPUs).

##  Appendix B Code for Momentum-only Projector

For all SSL methods, instead of using the momentum output of the momentum
backbone as baselines, we use the output from the online backbone (with stop
gradient). It helps to reduce half of the forward propagation to the backbone
which has more computation overhead compared to a light projector. Therefore,
our proposed projector-only momentum help maintain the pros of momentum while
mitigating its cons. Below we attach a PyTorch [25] pseudo-code for BYOL [19]
with our projector-only momentum, in Alg. 1. Other SSL methods can be done in
the same way.

Algorithm 1 Pytorch-like Pseudo-code: Projector-only Momentum

⬇

# f: backbone (ResNet)

# g, g_m: online and target projector

# h: predictor

for x in loader: # load a minibatch x with n samples

x_1, x_2 = aug(x), aug(x) # augmentations

f_1, f_2 = f(x_1), f(x_2) # backbone features

z_1, z_2 = g(f_1), g(f_2) # online projections

z_1m, z_2m = g_m(f_1), g_m(f_2) # momentum projections

p_1, p_2 = h(z_1), h(z_2) # predictions

L = D(p_1, z_2m)/2 + D(p_2, z_1m)/2 # loss

L.backward() # back-propagate

update(f, g, h) # SGD update

ema_update(g_m) # update parameters of the momentum projector, g_m := beta*g_m
+ (1-beta)*g

def D(p, z):

z = z.detach() # stop gradient

p = normalize(p, dim=1) # l2-normalize

z = normalize(z, dim=1) # l2-normalize

return -(p*z).sum(dim=1).mean() # negative cosine similarity

##  Appendix C Learning Curves

Table 8: Linear classification accuracy in CIFAR-100. All methods are trained
in 200 epochs using ResNet-18 as the backbone. BN1, BN2 denote the BN applying
to the output of block 4 and projector, respectively.

Method | Top-1 (%) | KNN-1 (%)
---|---|---
BYOL [19] | 62.01 | 54.84
w/o momentum | 43.48 | 31.58
w/o momentum + BN1 + BN2 | 49.59 | 42.92
EMA projector | 59.61 | 54.02
EMA projector + BN1 | 60.75 | 53.32
EMA projector + BN2 | 61.97 | 55.42
EMA projector + BN1 + BN2 | 62.48 | 55.66

Learning Curves of DINO on ImageNet-1K. For ImageNet-1K, we train DINO With
200 epochs, using ResNet-50 as the backbone. We ran the DINO baseline that
uses EMA for the entire target encoder and ran DINO with EMA only to the
projector similar to Alg. 1. We report the top-1 and top-5 test accuracy for
the val set. For both cases, EMA applied on the projector (3 layers) is almost
comparable.

Due to the resource constraint, we did not run for 1000 epochs, however, we
believe, the gap for the longer training is almost zero as the trend in Fig. 7
shows that using EMA in the deep backbone is unnecessary, but EMA on the
projector is sufficient.

(a) Linear train accuracy (b) Linear test accuracy

Figure 7: ImageNet-1K classification accuracy. Linear accuracy (top-1 %) for
DINO and projector-only EMA in DINO. The accuracy gap is almost close to zero.
It is best viewed in color.

Learning Curves of BYOL on CIFAR-100.

We show the case when applying projector-only momentum in the BYOL framework.
As the analysis in the main paper, the final parts of the encoder have more
fluctuation and less stability. We can use additionally a technique as BN to
the output of block 4 or projector to help training more stable. As shown in
Fig. 8, with no momentum (orange line) there always has the unstable point and
showing the worst accuracy. Using BN to the final layers (block 4 + projector)
can help to stabilize the training as the green line that improves 6.7%
compared to the version without BN and EMA.

Our proposed projector-only EMA can help to stabilize and boost performance to
be competitive with baseline BYOL to a smaller gap. Adding BN to the output of
block 4 and the output of the projector can make projector-only EMA (purple
line) almost match the BYOL baseline as clearly shown in Fig. 8. In this
figure, EMA projector + BN means that BN is applied to both block 4 and the
projector.

Tab. 8 shows a more detail about the important of making the block 4 or
projector stable with BN. It demonstrates that our proposed projector-only EMA
can match the BYOL baseline while avoiding the double forward computation
overhead to the deep backbone. It shows more evidence for our analysis in the
main paper that the final layers (block 4, projector) have much more
fluctuation, i.e. less stability. Making the final layers learn from the past
(with EMA) to stabilize the training is crucial, and a simple technique such
as BN can further help.

(a) Linear train accuracy (b) Linear test accuracy

Figure 8: Increasing stability with BN. Linear accuracy (top-1 %) for BYOL and
projector-only EMA in CIFAR-100. Here BN is applied to both outputs of block 4
and projector. It is best viewed in color.

##  Appendix D More Results on CIFAR-10

We show more results on CIFAR-10 where BYOL is compared with our projector-
only momentum. Tab. 9 shows that our method can also match baseline without
using EMA to the deep backbone that can maintain the benefit of EMA while
avoiding the double computation overhead to the backbone as the traditional
method.

Table 9: Linear classification accuracy on CIFAR-10. We train all SSL methods
for 200 epochs, with the ResNet-18 backbone. “-” denotes sharing weight. BN1,
BN2 denote the BN applying to the output of block 4 and projector,
respectively. We report top-1 accuracy on the test set.

Method | Backbone Momentum (18 layers) | Projector Momentum (2 layers) | Acc (%)
---|---|---|---
BYOL (baseline) | ✓ | ✓ | 88.37
w/o momentum | - | - | 80.34
EMA Projector | - | ✓ | 87.73
EMA Projector + BN1 | - | ✓ | 87.91
EMA Projector + BN2 | - | ✓ | 88.03
EMA Projector + BN1 + BN2 | - | ✓ | 88.45

##  Appendix E More Visualizations of Filters

Beyond the filters that are visualized in the main paper, we also presented 16
more filters in 2D and 3D visualizations out of 256 filters (weights) for the
experiments in section 3.2 where EMA is applied only to the final FC layer of
the target encoder. Here, start point, end in the figures meaning that are the
beginning and final point of the training process (a total of 40k steps),
respectively.

###  E.1 3D visualization

We show more 3D visualizations where green line denotes no momentum $\beta=0$;
blue and red lines show the student and teacher with momentum $\beta=0.9$;
black and orange lines illustrate the student and teacher with momentum
$\beta=0.99$. We can see several filters are stable for both curves and there
are filters that have the unstable point (at step around 10k) for the no
momentum line, such as filter 001, 002, 005, 008, 011, 013, 014 (Fig. 3 - Fig.
10). We also provide 2D plots (Fig. 11 - Fig. 18) below which can show more
clearly than some cases of 3D plots.

We observe that if using momentum, all filters show a mostly stable curve,
i.e. no ‘spike’ or abnormal point suddenly changes as the case no momentum.

(a) Filter 001 (b) Filter 002

Figure 9:

(a) Filter 003 (b) Filter 004

Figure 10:

(a) Filter 007 (b) Filter 008

Figure 11:

(a) Filter 011 (b) Filter 012

Figure 12:

(a) Filter 013 (b) Filter 014

Figure 13:

###  E.2 2D visualization

We also plot filters in 2D where the time dimension (training step) is omitted
for the more simple view as shown in Fig. 11 - Fig. 18 below. The 2D plots
also show clearly that the green line of no momentum exhibits the unstable
learning curves, such as Filter 001, 002, 007, 010, 011, 014. By contrast,
learning with EMA shows much more stable trends for every filter.

(a) Filter 001 (b) Filter 002

Figure 14:

(a) Filter 005 (b) Filter 006

Figure 15:

(a) Filter 007 (b) Filter 008

Figure 16:

(a) Filter 009 (b) Filter 010

Figure 17:

(a) Filter 011 (b) Filter 012

Figure 18:

(a) Filter 013 (b) Filter 014

Figure 19:

(a) Filter 015 (b) Filter 016

Figure 20:

## References

  * [1] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in neural information processing systems 33 (2020) 1877–1901.
  * [2] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, R. Soricut, Albert: A lite bert for self-supervised learning of language representations, in: International Conference on Learning Representations, 2020.
  * [3] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, Language models are unsupervised multitask learners, OpenAI blog (2019).
  * [4] W. Su, X. Zhu, Y. Cao, B. Li, L. Lu, F. Wei, J. Dai, {VL}-{bert}: Pre-training of generic visual-linguistic representations, in: International Conference on Learning Representations, 2020.
  * [5] P. Nie, Y. Zhang, X. Geng, A. Ramamurthy, L. Song, D. Jiang, Dc-bert: Decoupling question and document for efficient contextual encoding, in: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, 2020, pp. 1829–1832.
  * [6] J. Liu, Z. Qi, B. Wang, Y. Tian, Y. Shi, Self-llp: Self-supervised learning from label proportions with self-ensemble, Pattern Recognition 129 (2022) 108767\.  doi:https://doi.org/10.1016/j.patcog.2022.108767.
  * [7] X. Zhang, J. Mu, X. Zhang, H. Liu, L. Zong, Y. Li, Deep anomaly detection with self-supervised learning and adversarial training, Pattern Recognition 121 (2022) 108234.  doi:https://doi.org/10.1016/j.patcog.2021.108234.
  * [8] N. Komodakis, S. Gidaris, Unsupervised representation learning by predicting image rotations, in: International Conference on Learning Representations (ICLR), 2018.
  * [9] A. v. d. Oord, Y. Li, O. Vinyals, Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.03748 (2018).
  * [10] K. He, H. Fan, Y. Wu, S. Xie, R. Girshick, Momentum contrast for unsupervised visual representation learning, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [11] T. Chen, S. Kornblith, M. Norouzi, G. Hinton, A simple framework for contrastive learning of visual representations, in: International Conference on Machine Learning, 2020.
  * [12] D. Madaan, J. Yoon, Y. Li, Y. Liu, S. J. Hwang, Representational continuity for unsupervised continual learning, in: International Conference on Learning Representations, 2022.
URL https://openreview.net/forum?id=9Hrka5PA7LW

  * [13] B. Vasudeva, P. Deora, S. Bhattacharya, U. Pal, S. Chanda, Loop: Looking for optimal hard negative embeddings for deep metric learning, in: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 10634–10643.
  * [14] C. Zhang, K. Zhang, C. Zhang, T. X. Pham, C. D. Yoo, I. S. Kweon, How does simsiam avoid collapse without negative samples? a unified understanding with self-supervised contrastive learning, in: International Conference on Learning Representations, 2022.
  * [15] C. Zhang, K. Zhang, T. X. Pham, A. Niu, Z. Qiao, C. D. Yoo, I. S. Kweon, Dual temperature helps contrastive learning without many negative samples: Towards understanding and simplifying moco, arXiv preprint arXiv:2203.17248 (2022).
  * [16] G. Baykal, F. Ozcelik, G. Unal, Exploring deshufflegans in self-supervised generative adversarial networks, Pattern Recognition 122 (2022) 108244.  doi:https://doi.org/10.1016/j.patcog.2021.108244.
  * [17] X. Chen, H. Fan, R. Girshick, K. He, Improved baselines with momentum contrastive learning, moco v2, arXiv preprint arXiv:2003.04297 (2020).
  * [18] X. Chen, S. Xie, K. He, An empirical study of training self-supervised vision transformers, Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) (2021).
  * [19] J.-B. Grill, F. Strub, F. Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, Z. Guo, M. Gheshlaghi Azar, et al., Bootstrap your own latent-a new approach to self-supervised learning, Advances in Neural Information Processing Systems 33 (2020) 21271–21284.
  * [20] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, A. Joulin, Emerging properties in self-supervised vision transformers, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 9650–9660.
  * [21] M. Zheng, S. You, F. Wang, C. Qian, C. Zhang, X. Wang, C. Xu, ReSSL: Relational self-supervised learning with weak augmentation, in: Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
  * [22] A. Tarvainen, H. Valpola, Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results, Advances in neural information processing systems 30 (2017).
  * [23] X. Chen, K. He, Exploring simple siamese representation learning, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [24] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, A. Joulin, Unsupervised learning of visual features by contrasting cluster assignments, Advances in Neural Information Processing Systems 33 (2020) 9912–9924.
  * [25] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, S. Chintala, Pytorch: An imperative style, high-performance deep learning library, in: H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, R. Garnett (Eds.), Advances in Neural Information Processing Systems 32, Curran Associates, Inc., 2019, pp. 8024–8035.
  * [26] V. G. T. da Costa, E. Fini, M. Nabi, N. Sebe, E. Ricci, solo-learn: A library of self-supervised methods for visual representation learning, Journal of Machine Learning Research 23 (56) (2022) 1–6.
URL http://jmlr.org/papers/v23/21-1155.html

  * [27] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016\.
  * [28] S. Santurkar, D. Tsipras, A. Ilyas, A. Madry, How does batch normalization help optimization?, in: NeurIPS, 2018.
  * [29] A. Krizhevsky, et al., Learning multiple layers of features from tiny images, . (2009).
  * [30] Y. Tian, D. Krishnan, P. Isola, Contrastive representation distillation, in: International Conference on Learning Representations, 2019.
  * [31] A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, in: Advances in Neural Information Processing Systems, 2012.
  * [32] Y. Tian, D. Krishnan, P. Isola, Contrastive multiview coding, in: Proceedings of the European conference on computer vision (ECCV), Springer, 2020, pp. 776–794.
  * [33] F. Klinker, Exponential moving average versus moving exponential average, Mathematische Semesterberichte 58 (1) (2011) 97–107.
  * [34] G. Appel, Technical analysis: power tools for active investors, FT Press, 2005.
  * [35] M. J. Pring, Technical analysis explained: The successful investor’s guide to spotting investment trends and turning points, McGraw-Hill Professional, 2002\.
  * [36] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, in: International Conference on Learning Representations, 2015.
  * [37] I. Sutskever, J. Martens, G. Dahl, G. Hinton, On the importance of initialization and momentum in deep learning, in: International conference on machine learning, PMLR, 2013, pp. 1139–1147.
  * [38] J. Ma, D. Yarats, Quasi-hyperbolic momentum and adam for deep learning, in: International Conference on Learning Representations, 2018.
  * [39] N. Vieillard, B. Scherrer, O. Pietquin, M. Geist, Momentum in reinforcement learning, in: International Conference on Artificial Intelligence and Statistics, PMLR, 2020, pp. 2529–2538.
  * [40] E. Korkmaz, Nesterov momentum adversarial perturbations in the deep reinforcement learning domain, in: International Conference on Machine Learning, ICML 2020, Inductive Biases, Invariances and Generalization in Reinforcement Learning Workshop, Vol. 1, 2020.
  * [41] T. Haarnoja, A. Zhou, P. Abbeel, S. Levine, Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, in: International conference on machine learning, PMLR, 2018, pp. 1861–1870.
  * [42] K. Lee, Prototypical contrastive predictive coding, in: International Conference on Learning Representations, 2022.
URL https://openreview.net/forum?id=8la28hZOwug

  * [43] T. Chen, S. Kornblith, K. Swersky, M. Norouzi, G. E. Hinton, Big self-supervised models are strong semi-supervised learners, Advances in neural information processing systems 33 (2020) 22243–22255.
  * [44] Z. Cai, A. Ravichandran, S. Maji, C. Fowlkes, Z. Tu, S. Soatto, Exponential moving average normalization for self-supervised and semi-supervised learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 194–203.
  * [45] Z. Li, S. Liu, J. Sun, Momentum^ 2 teacher: Momentum teacher with momentum statistics for self-supervised learning, arXiv preprint arXiv:2101.07525 (2021).
  * [46] X. Wang, H. Fan, Y. Tian, D. Kihara, X. Chen, On the importance of asymmetry for siamese representation learning, arXiv preprint arXiv:2204.00613 (2022).
  * [47] J. Von Kügelgen, Y. Sharma, L. Gresele, W. Brendel, B. Schölkopf, M. Besserve, F. Locatello, Self-supervised learning with data augmentations provably isolates content from style, Advances in Neural Information Processing Systems 34 (2021).
  * [48] T. M. Paixão, R. F. Berriel, M. C. Boeres, A. L. Koerich, C. Badue, A. F. De Souza, T. Oliveira-Santos, Self-supervised deep reconstruction of mixed strip-shredded text documents, Pattern Recognition 107 (2020) 107535.  doi:https://doi.org/10.1016/j.patcog.2020.107535.
  * [49] H. Eun, J. Moon, J. Park, C. Jung, C. Kim, Learning to discriminate information for online action detection, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [50] C. Zhuang, T. She, A. Andonian, M. S. Mark, D. Yamins, Unsupervised learning from video with deep neural embeddings, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [51] H. Wu, Y. Qu, S. Lin, J. Zhou, R. Qiao, Z. Zhang, Y. Xie, L. Ma, Contrastive learning for compact single image dehazing, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 10551–10560.
  * [52] T. Pan, Y. Song, T. Yang, W. Jiang, W. Liu, Videomoco: Contrastive video representation learning with temporally adversarial examples, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 11205–11214.
  * [53] F. Wang, H. Liu, Understanding the behaviour of contrastive loss, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 2495–2504.
  * [54] H. Hu, J. Cui, L. Wang, Region-aware contrastive learning for semantic segmentation, in: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021, pp. 16291–16301.
  * [55] A. Diba, V. Sharma, R. Safdari, D. Lotfi, S. Sarfraz, R. Stiefelhagen, L. Van Gool, Vi2clr: Video and image for visual contrastive learning of representation, in: ICCV, 2021.
  * [56] K. Nozawa, I. Sato, Understanding negative samples in instance discriminative self-supervised representation learning, arXiv preprint arXiv:2102.06866 (2021).
  * [57] K. Zeng, M. Ning, Y. Wang, Y. Guo, Hierarchical clustering with hard-batch triplet loss for person re-identification, in: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [58] Z. Zhang, J. Sun, Y. Dai, D. Zhou, X. Song, M. He, Self-supervised rigid transformation equivariance for accurate 3d point cloud registration, Pattern Recognition 130 (2022) 108784.  doi:https://doi.org/10.1016/j.patcog.2022.108784.
  * [59] A. Iscen, G. Tolias, Y. Avrithis, O. Chum, Mining on manifolds: Metric learning without labels, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 7642–7651.
  * [60] C.-Y. Chuang, J. Robinson, Y.-C. Lin, A. Torralba, S. Jegelka, Debiased contrastive learning, in: NeurIPS, 2020.
  * [61] M. Wu, M. Mosse, C. Zhuang, D. Yamins, N. Goodman, Conditional negative sampling for contrastive learning of visual representations, in: International Conference on Learning Representations, 2020.
  * [62] Y. Kalantidis, M. B. Sariyildiz, N. Pion, P. Weinzaepfel, D. Larlus, Hard negative mixing for contrastive learning, in: H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, H. Lin (Eds.), Advances in Neural Information Processing Systems, Vol. 33, Curran Associates, Inc., 2020, pp. 21798–21809.
  * [63] J. D. Robinson, C.-Y. Chuang, S. Sra, S. Jegelka, Contrastive learning with hard negative samples, in: International Conference on Learning Representations, 2021.
  * [64] A. Kaku, S. Upadhya, N. Razavian, Intermediate layers matter in momentum contrastive self supervised learning, Advances in Neural Information Processing Systems 34 (2021).
  * [65] X. Wang, R. Zhang, C. Shen, T. Kong, L. Li, Dense contrastive learning for self-supervised visual pre-training, in: Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2021.
  * [66] Y. Tian, O. J. Henaff, A. van den Oord, Divide and contrast: Self-supervised learning from uncurated data, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 10063–10074.
  * [67] A. Ermolov, A. Siarohin, E. Sangineto, N. Sebe, Whitening for self-supervised representation learning, in: International Conference on Machine Learning, PMLR, 2021.
  * [68] J. Zbontar, L. Jing, I. Misra, Y. LeCun, S. Deny, Barlow twins: Self-supervised learning via redundancy reduction, International Conference on Machine Learning (2021).


# Towards Optimizing OCR for Accessibility

Peya Mowar Tanuja Ganu Saikat Guha
Microsoft Research, India
{t-peyamowar, tanuja.ganu<EMAIL_ADDRESS>
###### Abstract

Visual cues such as structure, emphasis, and icons play an important role in
efficient information foraging by sighted individuals and make for a
pleasurable reading experience. Blind, low-vision and other print-disabled
individuals miss out on these cues since current OCR and text-to-speech
software ignore them, resulting in a tedious reading experience. We identify
four semantic goals for an enjoyable listening experience, and identify
syntactic visual cues that help make progress towards these goals.
Empirically, we find that preserving even one or two visual cues in aural form
significantly enhances the experience for listening to print content.

##  1 Introduction

Worldwide, there are over 250 million blind and low-vision (BLV), or otherwise
print-disabled individuals [11]. Accessible long-form content, such as
textbooks, magazines, newspapers, and literature, is a key enabler for their
intellectual enrichment and high quality of life. Braille and audio books are
scarce or expensive, while human-mediated solutions like having friends and
family read out content is an imposition reserved only for the most critically
needed content.

Computer vision based apps [8] that combine optical character recognition
(OCR) and text-to-speech (TTS) tend to be used for micro-content, such as
labels on medicines and packaged goods, not for long-form content. Figure 1
illustrates why. On left is an article from a daily newspaper, and on the
right is the OCR output of the same from a popular OCR app. The OCR output
lacks all the visual cues (e.g., structure, emphasis, icons) that sighted
users depend on to get a quick overview of the content before selectively
drilling down. The lack of visual cues combined with the linear sequential
nature of audio ruins the enjoyability of automatically converted long-form
content.

Figure 1: A news article and it’s OCR output.

OCR was never optimized for the accessibility use case. Driven primarily by
the information retrieval use-case, whether searching in books [4] or
recognizing handwritten postal codes [10], OCR entirely ignores visual cues as
evidenced by the output of popular OCR libraries and services [9, 5, 7]. At
the same time, recognizing all possible visual cues across all languages is a
significant undertaking with questionable utility for the accessibility use
case given the constraints placed by the linearity and limited aural cues
(e.g. voice, tempo) of the eventual audio output.

This work argues for more research in preserving visual cues in aural form and
takes the first steps in that direction. Specifically, we  i) define the OCR
for accessibility problem and propose metrics for glanceability, readability,
listenability, and satisfiability of long-form content,  ii) identify the
simplest visual cues that, when preserved as aural cues, significantly enhance
the accessibility of long-form content, and  iii) apply our approach to daily
newspaper content and provide initial results.  Overall, we believe a
concerted effort into optimizing OCR and TTS for accessibility of long-form
content is possible and holds significant promise in enriching the lives of
print-disabled individuals.

##  2 Background

###  2.1 Accessibility Research

Accessibility research has majorly concentrated on increasing digital access
for BLV users[18]. This includes improving the narrative experience for
digital mediums such as IDEs[24, 21], collaborative documents[16] and
scientific reports[28]. The focal point of the research community has been
generating non-visual representations for visual content. This is done either
in the form of rich and descriptive alt-text[19] or using auditory cues such
as earcons and their derivatives (spearcons, lyricons, etc.)[12].

These non-visual representations are generated to substitute “visual
information scents” that help users easily navigate to information of their
interest. This process is referred to as information foraging and has been
studied extensively for web access to BLV users[26].

###  2.2 Text Recognition and OCRs

Optimal character recognition (popularly known as OCR) systems and more
broadly text recognition systems extract text from images. They are mostly
utilized as a form of data entry for digitizing print content such as bank
statements, legal records, healthcare reports, and invoices. They are also
employed in applications such as machine translation[6] and cognitive
computing[1].

Table 1 summarizes the output functionalities of some state-of-the-art OCR
APIs. The research in this domain contributes in improving these OCRs in one
of three ways:  i) scene-text, handwriting, cursive text recognition (in the
wild) [31, 29, 14, 22, 23, 27] ii) developing models for Indic languages[13]
and  iii) extracting text in the correct reading order[25, 30, 15].

OCR APIs | Output Functionalities
---|---
PyTesseract | Recognised characters, box boundaries, confidences, orientation, script
Azure Cognitive Services Read API | Recognised characters, words, sentences, box boundaries, confidences, orientation, script, whether handwriting
Google Cloud OCR | Recognised words, phrases, polygonal boundaries, confidences, script

Table 1: Output functionalities of state-of-the-art OCR APIs

###  2.3 Research Gaps

Prior accessibility research has not focused on print-impaired users (that
include BLV, illiterate, and users with learning and physical disabilities),
and more specifically on print accessibility. More work has been done for
digital access, where text metadata is more commonly available in the form of
HTML tags and alternate texts. Non-profits such as the DAISY Forum [3] in
India have addressed this need, however, they rely on manual intervention due
to the lack of important metadata in the OCR text outputs. This leads to a
huge restraint in the speed and costs associated with digitization for the
print-impaired.

The required metadata is nothing else but the proxy for the visual information
scents present in printed content. These visual cues or “syntaxes” include but
are not limited to bold, italics, underline, highlighting, font style, size,
color, ordered and unordered lists, shapes such as boxes and lines, spacing,
indentation, alignment, shadow, background color. Since current OCR systems
lack in providing this information, a print-disabled reader is unable to
leverage these cues resulting in a higher time cost to gain value from the
same content.

Thus, there is a need for text recognition systems that can preserve these
visual cues for narration. However, not all syntaxes may be equally relevant
to optimize the users’ information gain, and thus, it is vital to understand
the degree of abstraction necessary to alleviate the accessibility challenges
presented by their underlying semantics[17].

##  3 Methodology

Our research goal is to identify the visual cues, which when presented to the
users, result in an optimal rate of information gain. To do this, we consider
the underlying semantics of these cues, the lack of which pose certain
accessibility challenges. We classify these challenges in long-form text
content into four broad categories:  Figure 2: Visual cues in a page from a
magazine. 1. Glanceability: In order to emphasise key content for the users,
long-form texts such as newspapers, books and magazines contain content with
varying grades of emphasis. For example, breaking news is often centered on
the first page, occupying a significant amount of area to grab the reader’s
attention; and similarly, headings and subheadings in books or magazines often
have bolden text and a larger font size than the remaining text as shown in
Figure 2. This relies on the users’ ability to glance at the page, view the
emphasized content and get the gist of the whole content, and is a challenge
for print-impaired users who do not get access to the emphasized content from
an OCR.  2. Readability: This is the ease with which the users can comprehend
the content. Sighted users can understand the organization and structure of
the content due to the text formatting such as a breakout box depicted using a
colored shape in the background or list elements marked by bullet points or
enlarged first alphabets as shown in Figure 2. On the other hand, print-
impaired users listen to the same content with no context being provided about
the structure of the content, which is innately visual in nature and thus lose
out on the comprehensibility of the content.  3. Listenability: The listening
experience very often differs from the reading experience of content due to
the lack of spatial information which can be randomly accessed. Revealing too
many visual cues in the narration may become overwhelming for the listener.
Thus, along with readability, it is also fitting to take listenability of the
content into consideration. This is done by considering the ability of the
users to listen, understand and enjoy the narrative version of a long-form
text content.  4. Satisfiability: Often in print media, a plethora of colors,
icons, shapes, and font styles are used, as shown in Figure 2, in order to
attract and retain the readers, improve their overall reading experience and
most importantly, make it fun and create appropriate psychological impact.
Thus, when we digitize the same print content for print-impaired users, it
becomes imperative to take these fun visual elements into consideration and
make the narration a rewarding and enjoyable experience for them. This is what
we refer to as satisfiability \- when the content meets the user expectations.

##  4 Experiments and Learnings

We conducted two sets of experiments - an interview study and a survey to find
the least degree of abstractive formatting required that can address these
challenges defined in Section 3. We perform our initial set of experiments on
8 articles extracted from the front page of The Hindu, a popular English daily
newspaper in India. The articles contain 250-500 words and a diverse set of
visual cues such as breakout boxes, list elements and figures.

###  4.1 Interview Study

The objective of this experiment was to observe patterns in the reading
behaviour of sighted users and derive insights on glanceability, readability
and satisfiability.

####  4.1.1 Experiment Setup

We conducted task-based, semi-structured user interviews. We recruited 8
participants for reading the content. All participants were above 18 years of
age, sighted and proficient in the English language. The interviews were
conducted over video conferencing and were 30 minutes long in duration.

For each article, we extracted the text using the Azure Read API [7] and
generated four stages of visual formatting, as explained in Table 2. We
created four decks - each containing the 8 articles, 2 articles per a format
style. The participants were randomly allotted one deck to perform their tasks
on.

Visual Formats |  Description
---|---
F0 |  Plain text extracted from the article using Azure Cognitive Services OCR in ‘natural reading order’
F1 |  To F0, added emphasis by boldening the headline, subheading, byline, breakout box as well as metadata
F2 |  To F1, changing font colors for all the text present in bounding boxes to provide a sense of structuring
F3 |  Original article image with all formatting intact
Auditory format |  Description
A0 |  F0 narrated as is by the voice ‘Neerja – English Indian’
A1 |  F1 narrated where the bolden text narrated by the voice ‘Prabhat – English Indian’ and the plain article text narrated by ‘Neerja – English Indian’

Table 2: Considered visual and auditory formats

####  4.1.2 Evaluation

1. Glanceability: We evaluated glanceability by noting the time taken by the users in identifying the key content from a given text content. In the interview, we requested the participants to find and locate all the subheadings present in the news article. We then noted the the accuracy and the time that the participants took in finding these subheadings.  2. Readability: We measured the time taken by the users in identifying different structural elements to evaluate readability of the content. In our interviews, the participants were asked to identify if the news article contains a breakout box (which typically contains a summarization, timeline, quote or statistics) in the article. We noted the time and accuracy with which the participants responded.  3. Satisfiability: The metric we used to evaluate satisfiability was the users’ self-assessment of their reading experience based on the User Engagement Scale [20]. We asked the users to rate their responses to the following two statements on a Likert Scale ranging from Strongly Disagree to Strongly Agree (1-5):  i) The reading experience was rewarding.  ii) The reading experience was enjoyable.

####  4.1.3 Quantitative Results

1. Glanceability: Accuracy \- Without any visual cues (F0), the participants were able to correctly identify the subheadings only 18.75% of the time. Simply by boldening the important text (F1), the accuracy raised to 68.75%. Time Cost \- On an average, the participants spent 43 seconds to identify the headline in unformatted text (F0). In fact, 43.75% of the time, the users spent less than 30 seconds and gave up. Moreover, of the users who gave accurate responses for the task, the median time spent to find the subheading in unformatted text was 80 seconds, significantly higher than any of the other formats, as is evident from Figure 3(a).  2. Readability: Accuracy \- The accuracy of responses positively correlated with the presence of visual cues, rising from 25% when no cues were given (F0) to 81.25% when text was emphasised and structured (F2). Time Cost \- The median time spent by the participants on unformatted text was 91 seconds, and this time dropped significantly (to 20 seconds) with a single grade increase in formatting (F1). This is illustrated in Figure 3(b).

(a) Glanceability.

(b) Readability.

(c) Satisfiability.

(d) Listenability.

Figure 3: Visualisations of the quantitative results. 3. Satisfiability: On an
average, we found that participants found the content to be both more
rewarding and enjoyable with more visual cues. This is illustrated in Figure
3(c).

###  4.2 Survey

The objective of this experiment was to garner insights on the relative
listenability of two audios and validate if inserting the simplest aural cues
in the narration improves the overall listening experience of the users.

####  4.2.1 Experiment Setup

We distributed the survey to our friends and family and ended up receiving 12
responses. All participants were above 18 years of age, sighted and proficient
in the English language. The survey took an average time of 40 minutes for
completion.

We leveraged the Azure Speech Studio [2] for converting two of the previously
mentioned formats to audio narrations. The audio formats are explained further
in detail in Table 2.

####  4.2.2 Evaluation and Quantitative Results

1. Metrics: We evaluated listenability by asking the users to compare between two audio clips of different formats and answer whether one was ‘much better’, ‘better’, ‘same’, ‘worse’ or ‘much worse’ than the other. Moreover, we also took the listener’s qualitative inputs on their listening experience into account.  2. Results: We discovered that simply by emphasizing (A1), the narration became 31% better (than A0) for the listeners. Figure 3(d) illustrates the results of the survey.

###  4.3 Qualitative Results

1. Heavy reliance on bolden text: We observed that some participants completely relied on bolden text to find the subheadings. P2, when presented with unformatted text (F0), simply gave up the search: ”This article doesn’t have any subheadings. There’s nothing in bold”. Moreover, some participants declared that everything bolden in F1 (headline, meta-data, byline, etc.) was the subheading, without actually going through the content.  2. Cases and Punctuation as substitutes: During the interviews, participants heavily depended on cases and punctuation (visual cues picked up by the OCR) for finding elements. Text in quotes were declared as subheadings, and a continuous stream of uppercase letters as breakout boxes. This had a negative effect. For instance, an article had a quote inside a breakout box. No participant was able to correctly identify that breakout box.  3. Emphasis provided a break in monotony: The survey participants pointed out that usage of two different voices helped break the monotonous flow of the audio, provided a better outline of the article and improved the overall listening experience - ”The variation ( the use of a different voice to speak out, what mostly appeared to be heading), helping alleviating the monotonicity of the spoken content.”.  4. Lack of visual appeal affected reading experience: Some participants were severely underjoyed to see unformatted text (F0) and rated the reading experience without actually going through it in entirety. This points to how appeal plays a major role in attracting a reader and making it a rewarding experience. When in the deck, the original article (F3) showed up, the reaction was of pure bliss and comfort.

##  5 Conclusion

In this paper, we articulated the need for preserving visual cues in aural
form in order to enhance the accessibility of printed content for blind, low-
vision, and other print-disabled individuals. We found that existing OCR
services and current research in OCR largely ignore visual cues that are
necessary for information foraging. We proposed metrics for glanceability,
readability, listenability and overall satisfiability, and characterised how
adding even one or two visual cues in aural form significantly improve the
narration experience for print-disabled individuals.

## References

  * [1] Azure cognitive services.  https://azure.microsoft.com/en-in/services/cognitive-services/.
  * [2] Azure speech studio.  https://speech.microsoft.com/portal.
  * [3] Daisy forum of india.  https://daisyindia.org/.
  * [4] Google books.  https://books.google.co.in/.
  * [5] Google cloud vision api.  https://cloud.google.com/vision/docs/ocr.
  * [6] Google lens.  https://lens.google/.
  * [7] Microsoft azure cognitive services read api.  https://docs.microsoft.com/en-us/azure/cognitive-services/computer-vision/overview-ocr#read-api.
  * [8] Microsoft lens.  https://play.google.com/store/apps/details?id=com.microsoft.office.officelens&gl=US.
  * [9] Pytesseract.  https://pypi.org/project/pytesseract/.
  * [10] The mnist database of handwritten digits.  http://yann.lecun.com/exdb/mnist/.
  * [11] World blindness and visual impairment: despite many successes, the problem is growing.  https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5820628/#:~:text=In%202015%2C%20there%20were%20an%20estimated%20253%20million,whom%200.49%25%20are%20blind%20and%202.95%25%20have%20MSVI.
  * [12] Stephen A. Brewster, Peter C. Wright, and Alistair D. N. Edwards.  An evaluation of earcons for use in auditory human-computer interfaces.  In Proceedings of the INTERACT ’93 and CHI ’93 Conference on Human Factors in Computing Systems, CHI ’93, page 222–227, New York, NY, USA, 1993. Association for Computing Machinery.
  * [13] Agam Dwivedi, Rohit Saluja, and Ravi Kiran Sarvadevabhatla.  An ocr for classical indic documents containing arbitrarily long words.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2020.
  * [14] Sharon Fogel, Hadar Averbuch-Elor, Sarel Cohen, Shai Mazor, and Roee Litman.  Scrabblegan: Semi-supervised varying length handwritten text generation.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4323–4332, 2020.
  * [15] Chen-Yu Lee, Chun-Liang Li, Chu Wang, Renshen Wang, Yasuhisa Fujii, Siyang Qin, Ashok Popat, and Tomas Pfister.  ROPE: Reading order equivariant positional encoding for graph-based document information extraction.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 314–321, Online, Aug. 2021. Association for Computational Linguistics.
  * [16] Cheuk Yin Phipson Lee, Zhuohao Zhang, Jaylin Herskovitz, JooYoung Seo, and Anhong Guo.  Collabally: Accessible collaboration awareness in document editing.  In CHI Conference on Human Factors in Computing Systems, CHI ’22, New York, NY, USA, 2022. Association for Computing Machinery.
  * [17] Anastassia Loukina, Van Rynald T. Liceralde, and Beata Beigman Klebanov.  Towards understanding text factors in oral reading.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2143–2154, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
  * [18] Kelly Mack, Emma McDonnell, Dhruv Jain, Lucy Lu Wang, Jon E. Froehlich, and Leah Findlater.  What do we mean by “accessibility research”? a literature survey of accessibility papers in chi and assets from 1994 to 2019.  In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI ’21, New York, NY, USA, 2021. Association for Computing Machinery.
  * [19] Meredith Ringel Morris, Jazette Johnson, Cynthia L. Bennett, and Edward Cutrell.  Rich representations of visual content for screen reader users.  In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, page 1–11, New York, NY, USA, 2018. Association for Computing Machinery.
  * [20] Heather L. O’Brien, Paul Cairns, and Mark Hall.  A practical approach to measuring user engagement with the refined user engagement scale (ues) and new ues short form.  International Journal of Human-Computer Studies, 112:28–39, 2018\.
  * [21] Venkatesh Potluri, Priyan Vaithilingam, Suresh Iyengar, Y. Vidya, Manohar Swaminathan, and Gopal Srinivasa.  Codetalk: Improving programming environment accessibility for visually impaired developers.  In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, page 1–11, New York, NY, USA, 2018. Association for Computing Machinery.
  * [22] Liang Qiao, Sanli Tang, Zhanzhan Cheng, Yunlu Xu, Yi Niu, Shiliang Pu, and Fei Wu.  Text perceptron: Towards end-to-end arbitrary-shaped text spotting.  Proceedings of the AAAI Conference on Artificial Intelligence, 34(07):11899–11907, Apr. 2020.
  * [23] Prasun Roy, Saumik Bhattacharya, Subhankar Ghosh, and Umapada Pal.  Stefann: Scene text editor using font adaptive neural network.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13225–13234, 2020.
  * [24] Harini Sampath, Alice Merrick, and Andrew Macvean.  Accessibility of command line interfaces.  In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI ’21, New York, NY, USA, 2021. Association for Computing Machinery.
  * [25] Hongyin Tang, Xingwu Sun, Beihong Jin, and Fuzheng Zhang.  A bidirectional multi-paragraph reading model for zero-shot entity linking.  Proceedings of the AAAI Conference on Artificial Intelligence, 35(15):13889–13897, May 2021.
  * [26] Pooja Upadhyay.  Comparing non-visual and visual information foraging on the web.  In Extended Abstracts of the 2020 CHI Conference on Human Factors in Computing Systems, CHI EA ’20, page 1–8, New York, NY, USA, 2020\. Association for Computing Machinery.
  * [27] Zhaoyi Wan, Minghang He, Haoran Chen, Xiang Bai, and Cong Yao.  Textscanner: Reading characters in order for robust scene text recognition.  Proceedings of the AAAI Conference on Artificial Intelligence, 34(07):12120–12127, Apr. 2020.
  * [28] Lucy Lu Wang, Isabel Cachola, Jonathan Bragg, Evie Yu-Yen Cheng, Chelsea Haupt, Matt Latzke, Bailey Kuehl, Madeleine van Zuylen, Linda Wagner, and Daniel S. Weld.  Improving the accessibility of scientific documents: Current state, user needs, and a system solution to enhance scientific pdf accessibility for blind and low vision users, 2021.
  * [29] Wenhai Wang, Enze Xie, Xiang Li, Wenbo Hou, Tong Lu, Gang Yu, and Shuai Shao.  Shape robust text detection with progressive scale expansion network.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9328–9337, 2019.
  * [30] Zilong Wang, Yiheng Xu, Lei Cui, Jingbo Shang, and Furu Wei.  Layoutreader: Pre-training of text and layout for reading order detection.  In EMNLP 2021, November 2021.
  * [31] Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, and Jiajun Liang.  East: An efficient and accurate scene text detector.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2642–2651, 2017.