# Enhanced Performance of Pre-Trained Networks by Matched Augmentation
Distributions

Touqeer Ahmad, Mohsen Jafarzadeh, Akshay Raj Dhamija, Ryan Rabinowitz,
Steve Cruz, Chunchun Li, Terrance E. Boult {touqeer, mjafarzadeh, adhamija,
rrabinow, cli, scruz<EMAIL_ADDRESS>Vision and Security Technology
Lab, University of Colorado at Colorado Springs, USA

###### Abstract

There exists a distribution discrepancy between training and testing, in the
way images are fed to modern CNNs. Recent work tried to bridge this gap either
by fine-tuning or re-training the network at different resolutions. However
re-training a network is rarely cheap and not always viable. To this end, we
propose a simple solution to address the train-test distributional shift and
enhance the performance of pre-trained models – which commonly ship as a
package with deep learning platforms e.g., PyTorch. Specifically, we
demonstrate that running inference on the center crop of an image is not
always the best as important discriminatory information may be cropped-off.
Instead we propose to combine results for multiple random crops for a test
image. This not only matches the train time augmentation but also provides the
full coverage of the input image. We explore combining representation of
random crops through averaging at different levels i.e., deep feature level,
logit level, and softmax level. We demonstrate that, for various families of
modern deep networks, such averaging results in better validation accuracy
compared to using a single central crop per image. The softmax averaging
results in the best performance for various pre-trained networks without
requiring any re-training or fine-tuning whatsoever. On modern GPUs with batch
processing, the paper’s approach to inference of pre-trained networks, is
essentially free as all images in a batch can all be processed at once.

Figure 1: Examples from ILSVRC-2012 [1] val split where a center-crop (red
square) may miss discriminatory information e.g., rows one & two, or at best
provides partial coverage of object of interest e.g., row three (daisy,
mushroom, kangaroo). In some of such cases, a non-central random crop (green
square) may provide good coverage of the image and that of underlying object
of interest. Whereas in other cases a single random or central crop may not be
optimal and an average of several random crops may serve best. Each image
above is resized to $256$ respecting aspect ratio and central $224\times 224$
crop identified by red square which is the conventional input to standard
networks (e.g., ResNet-50) at inference time.

##  I Introduction

Boosting the performance of established deep networks is an active research
area where methods including custom training [2], employing additional
training sets [3], ensemble teacher-student paradigms [4, 5], architecture
modifications, complex learning schedules and data augmentation [6, 7, 8, 9]
strategies have been investigated. However re-training or fine-tuning an
existing model with custom settings is not always a possible or preferred
solution.

This paper’s approach improves performance using a pre-trained network by
addressing the discrepancies that exist between training and inference of deep
neural networks. For example, during training data augmentation with random
crops are generated from the images on which the loss is minimized, whereas,
during inference it is conventional to resize an image to a fixed resolution
maintaining aspect ratio and then take the central crop to forward pass it
through the trained network. In most cases the central crop provides a
reasonable coverage of the image and underlying object of interest. However in
some cases centrally cropping an image may discard the discriminatory
information essential for a good recognition. Fig 1 shows several examples
from ILSVRC-2012 [1] validation set where a central crop may result into
insufficient coverage of the resized image and especially of the object of
interest.

One can easily identify two types of categories of images which are at
disadvantage due to central cropping: (i) images where the captured object of
interest is not in the center of the frame, and (ii) images where longer
dimension is significantly larger than the shorter one (more than 2x). In
first category, since the object of interest is already at a non-central
location, cropping centrally further removes or reduces the discriminatory
information essential for the recognition. For example in Fig 1, non-centrally
captured objects of interest [row3: mushroom, kangaroo] are either partially
covered by the central crop or fully cropped off in more severe cases [row1:
roaster, row3: daisy]. In second category, the object of interest is generally
laid out along the longer dimension of the image, central copping only
captures part of the object and may discard important discriminatory
information. For example the heads and tails of the animals imaged in row1
(right) and row2 of Fig 1 are discarded as a result of center cropping and
characteristics of these parts could be the discriminatory features that the
deep network learned during training.

To enhance the coverage of the underlying object of interest, some networks
used multiple fixed crops at inference time [10, 11], e.g., the original
AlexNet [12], used 5 fixed crops (center and 4 corners), plus their horizontal
mirrored version for 10 crops at inference. While AlexNet did use multiple
crops at inference, it still has a significant difference between training,
which has 2048 random variations, and test time which has 5 fixed crop
locations. Through an ablation study we demonstrate that while AlexNet’s 5/10
crop strategy may be sufficient for most images, it can still benefit when
more random or mirrored random crops are additionally added.

Recent work such as FixRes [2], show that a performance gain can be obtained
by adjusting resolution so the distribution of object sizes is better matched
between training and inference. They accomplish this size-distribution gap by
retraining the network. In this paper we content that the size mismatch is
only part of the distributional mismatch problem, and that object coverage via
crops is also an issue. To remedy the recognition performance, we propose
Matched Inference Distributions (MID) which approximates the distribution of
training sampling at inference time, and show this improves performance
without the need for any retraining. In training, over the multiple epochs the
system effectively averages the different sample augmentations. Hence MID
combines results by averaging from several augmentations instead of forward
passing a single central crop per image.

The benefit of using random crops is three fold: (i) it better aligns with
augmentation employed during the training of deep networks, (ii) provides good
coverage of underlying object of interest, and (iii) it addresses some of the
resolution issues identified in FixRes [2]. The representations generated for
several random crops of an image can be combined at different levels of a deep
network. To this end we explore averaging at the levels of deep features,
logits and the softmax layer. We found that averaging after deep features and
logits results in identical performance whereas averaging at softmax layers
results in the best performance. We have investigated our averaging of random
crops approach for various modern network families including ResNet [13],
EfficientNet [14], and NFNet [15] with up to 2.0% boost in ImageNet-2012 Top-1
validation accuracy. We should note that the performance gain is achieved
without requiring any custom training, re-training or fine-tuning of the
networks. Since methods focusing on better training strategies [5, 4, 3, 2]
still run inference using a single central crop, they can further benefit from
proposed averaging strategy. We should note that batch processing is common in
modern deep learning platforms and forward passing multiple crops can by
achieved without any significant overhead. For a non-batched processing,
forward passing multiple crops directly results in enhanced computations. To
address this we suggest an adaptive strategy where the number of random crops
are pre-determined as function of input resolution as center cropping may
still be sufficient for majority of the test images.

Our Contributions

  * •

Demonstrating that central or simple fixed crops at inference are sub-optimal
and that our novel Matched Inference Distribution (MID) approach which uses
the same sampling process for data augmentation in both training and
inference, results in better performance.

  * •

Thorough evaluation of several families of modern deep networks to quantify
their achievable best performance without any custom training or fine-tuning.

Figure 2: One variant of our proposed inference scheme where we advocate using
averaging of several random crops instead of just a single central one. A
sample image from ILSVRC-2012 [1] validation split, resized to 256 respecting
aspect ratio with several $224\times 224$ random crops marked in different
colors. Adopting PyTorch’s preferred NCHW tensor format, the generated random
crops are concatenated in the batch (N) dimension and passed through the
feature extractor of a pre-trained CNN. The generated deep feature vectors are
then averaged and subsequently passed through the classification head to
predict the classification label. Softmax layer is omitted for brevity.

##  II Related Work

Squeezing the best performance out of established network architectures for
image recognition is an active research area with supreme practical importance
of real-world deployment especially of smaller networks like ResNet-18, and
MobileNet etc. Latest attempts have been focused on training, re-training or
fine-tuning the core networks with custom strategies using default or
additional data.

In popular FixRes [2], authors emphasized the existence of train-test object
size discrepancy and argued for using smaller resolutions during training.
Through extensive experiments they demonstrated that training on smaller
resolutions not only compensates for the distributional shift but also reduces
the training time. They further showed that fine-tuning pre-trained networks
for higher test-time resolution results in enhanced performance. Using
ResNet-50 as the core architecture they explored various train-test resolution
combinations and demonstrated that even higher performance can be achieved by
employing bigger networks like ResNeXt-101 [16] and leveraging Billion-scale
training data [3]. In an extended version [17] of their work, authors explored
FixRes to train EfficientNet architectures [14] where they further integrated
label smoothing. They focused on two best performing EfficientNet versions
trained with adversarial examples [18] and noisy student in weakly-supervised
fashion using 300 million unlabeled images [19]. The resultant FixEfficientNet
models are demonstrated to outperform these two versions on ImageNet [1],
ImageNet-V2 [20], and ImageNet-Real [21].

CutMix [6] belongs to the category of regional dropout augmentation methods
where random regions in images are removed to enhance the generalization
performance of deep networks. However unlike other regional dropout approaches
[8, 9], CutMix fills the removed regions with patches from other training
examples, effectively achieving not only better generalization but also better
detection and localization performance. Mixup [7] is another augmentation
strategy where training images and ground truth labels are linearly
interpolated to synthesize samples to enhance the generalization capability of
the underlying network. Mixing features instead of the images [22] and other
mixup variants [23, 24] have also been explored.

In [5], authors explored knowledge distillation [25] and adversarial learning
[26] to train a student network using the predictions from an ensemble of
teacher networks. Instead of using one-hot ground truth labels, the predicted
probability vectors out of the teacher networks are used to distill the
knowledge where L1, L2, and KL-divergence losses are explored. Additionally
discriminator networks are incorporated at several levels to distinguish
between the feature representations generated by student and the teacher. At
each iteration a teacher is randomly selected out of the teacher zoo and its
predicted probability vector provides the supervisory signal. For ImageNet,
they used ResNet-50 as the student network and (i) VGG-19 w/BN and ResNet-50,
or (ii) ResNet-101 and ResNet-152 as the two teacher ensemble variants. In
extended version of their work i.e., MEAL-V2 [4] they trained a student
network using an ensemble of teachers where average of the softmax scores of
the ensemble is used as soft supervisory signal instead of one-hot hard labels
or selecting one teacher at random for every iteration. In this variant, the
student network is trained using KL-divergence loss to match the probability
distribution of the averaged ensemble. Like [5], a discriminator is further
employed to distinguish whether the input features are generated from teacher
ensemble or student network. Authors used their framework to train ResNet-50,
EfficientNet-B0, and three size variants of MobileNet-V3 where they used
senet154 and resnet152_v1s as teachers. For training the student on a larger
resolution e.g., $380\times 380$, larger teachers like efficientnet_b4_ns and
efficientnet_b4 are employed.

As evidenced by summarized research above, an enhanced performance can be
achieved by smartly training the existing networks and/or leveraging the
additional billion-scale data. However, training a network may not be a
preferred or viable solution for vendors deploying deep models for real-world
applications. To this end, in this paper we focus on squeezing best achievable
performance for several popular pre-trained deep networks without any training
or fine-tuning. We demonstrate that for several larger networks, averaging of
random crops results in comparable or better performance than employing custom
training strategies and billion-scale data rendering such strategies
questionable.

##  III Problem Formulation

We operate in the conventional image recognition setting where we assume a
pre-trained CNN model trained on train split of ILSVRC-2012 [1] is available.
We further assume that the deep model can be decomposed into feature extractor
$f_{e}$ and the classification head $f_{c}$. In the conventional single
central-crop inference, a resized query image $x$ can be passed through the
feature extractor $f_{e}$ to generate its deep representation:

$R_{x^{c}}=f_{e}(x^{c})$ (1)

which can be subsequently passed through the classification head $f_{c}$ and
the softmax layer to generate the predicted label:

$s_{i}=softmax(f_{c}(R_{x^{c}}))$ (2)

where, $x^{c}$ refers to the central crop generated from the query image $x$.
As demonstrated in Fig 1, a central-crop may not be ideal for all images, so
we propose to use several random crops per image. Given the resized query
image $x$, we generate $N$ random crops
$x^{r_{1}},x^{r_{2}},\cdots,x^{r_{N}}$; each of which is passed through the
feature extractor to generate respective representation $R_{x^{rj}}$. The
representations for these random crops are then averaged:

$R_{x^{avg}}=\frac{1}{N}\sum_{j=1}^{N}{R_{x^{rj}}},$ (3)

and passed through Eq. 2 to get the predicted label. Practically random crops
can be concatenated to generate a batch which can be forward passed thought
the feature extractor at once, and averaging can be accomplished by using the
conventional AvgPool layer on the batch dimension. Fig 2 provides a
visualization for this formulation. Although the formulation has been
described for averaging the representations at the deep feature level, similar
averaging can be accomplished using the logits or softmax scores of random
crops and have been explored in our experiments.

TABLE I: Comparison of Top-1 validation accuracy on ImageNet dataset for
ResNet [13] family. The performance gain in each case is noted in parentheses.
† indicates the numbers reported on the PyTorch page [27] and verified on our
end. Whereas ‡ indicates the numbers taken from [6]. It should be noted that
the numbers reported for baseline ResNets in [6] are higher than PyTorch and
have been noted below. The numbers reported for all other approaches are taken
from their respective papers. For consistent comparison we report the accuracy
for all approaches where train and test resolution is fixed to $224\times 224$
and specifically note the variants where test resolution deviates. Averaging
the softmax (SM) scores of random crops (RCs) results in better performance
than that of averaging at the deep feature vector (FV) level. We also provide
the numbers when 5 five fixed crops (FCs) suggested by AlexNet [12] are
employed or additionally when mirrored versions of these fixed crops (i.e.,
MFCs) are also used. Including the mirrored random crops (MRCs) in the
averaging results in better performance. We conduct an ablation study (Tab
VIII) demonstrating combining RCs, MRCs, FCs and MFCs results in the best
achievable performance. The performance gain is high for smaller networks and
low for larger ones. An increase in the number of crops results in enhanced
performance, however, performance gain saturates after a certain number of
crops as evidenced in Fig 3.

Approach | Custom Training | ResNet-18 | ResNet-34 | ResNet-50 | ResNet-101 | ResNet-152
---|---|---|---|---|---|---
Image Resized To | 256
Input Crop Size | $224\times 224$
1.0 - (Crop to Image Ratio) | 0.125
Feature Dimension | 512 | 2048
|  Center Crop† | - | 69.76 | 73.31 | 76.13 | 77.37 | 78.31
MID (RCs) | (Average of FV of 10 RCs) | ✗ | $71.42^{\textbf{(+1.66)}}$ | $74.74^{\textbf{(+1.43)}}$ | $77.22^{\textbf{(+1.09)}}$ | $78.46^{\textbf{(+1.09)}}$ | $79.39^{\textbf{(+1.08)}}$
(Average of FV of 20 RCs) | ✗ | $71.56^{\textbf{(+1.80)}}$ | $74.93^{\textbf{(+1.62)}}$ | $77.30^{\textbf{(+1.17)}}$ | $78.65^{\textbf{(+1.28)}}$ | $79.54^{\textbf{(+1.23)}}$
(Average of SM of 10 RCs) | ✗ | $71.64^{\textbf{(+1.88)}}$ | $74.88^{\textbf{(+1.57)}}$ | $77.44^{\textbf{(+1.31)}}$ | $78.67^{\textbf{(+1.30)}}$ | $79.58^{\textbf{(+1.27)}}$
(Average of SM of 20 RCs) | ✗ | $71.83^{\textbf{(+2.07)}}$ | $75.14^{\textbf{(+1.83)}}$ | $77.49^{\textbf{(+1.36)}}$ | $78.85^{\textbf{(+1.48)}}$ | $79.79^{\textbf{(+1.48)}}$
MID (RCs + MRCs) | (Average of FV of 5 RCs + 5 MRCs) | ✗ | $71.65^{\textbf{(+1.89)}}$ | $75.01^{\textbf{(+1.70)}}$ | $77.43^{\textbf{(+1.30)}}$ | $78.61^{\textbf{(+1.24)}}$ | $79.64^{\textbf{(+1.33)}}$
(Average of FV of 10 RCs + 10 MRCs) | ✗ | $71.85^{\textbf{(+2.09)}}$ | $75.15^{\textbf{(+1.84)}}$ | $77.50^{\textbf{(+1.37)}}$ | $78.79^{\textbf{(+1.42)}}$ | $79.75^{\textbf{(+1.44)}}$
(Average of FV of 15 RCs + 15 MRCs) | ✗ | $71.94^{\textbf{(+2.18)}}$ | $75.24^{\textbf{(+1.93)}}$ | $77.56^{\textbf{(+1.43)}}$ | $78.85^{\textbf{(+1.48)}}$ | $79.76^{\textbf{(+1.45)}}$
(Average of FV of 20 RCs + 20 MRCs) | ✗ | $71.91^{\textbf{(+2.15)}}$ | $75.30^{\textbf{(+1.99)}}$ | $77.57^{\textbf{(+1.44)}}$ | $78.88^{\textbf{(+1.51)}}$ | $79.82^{\textbf{(+1.51)}}$
(Average of SM of 5 RCs + 5 MRCs) | ✗ | $71.90^{\textbf{(+2.14)}}$ | $75.28^{\textbf{(+1.97)}}$ | $77.64^{\textbf{(+1.51)}}$ | $78.77^{\textbf{(+1.40)}}$ | $79.84^{\textbf{(+1.53)}}$
(Average of SM of 10 RCs + 10 MRCs) | ✗ | $72.12^{\textbf{(+2.36)}}$ | $75.42^{\textbf{(+2.11)}}$ | $77.78^{\textbf{(+1.65)}}$ | $78.93^{\textbf{(+1.56)}}$ | $79.97^{\textbf{(+1.66)}}$
(Average of SM of 15 RCs + 15 MRCs) | ✗ | $72.19^{\textbf{(+2.43)}}$ | $75.47^{\textbf{(+2.16)}}$ | $77.78^{\textbf{(+1.65)}}$ | $79.02^{\textbf{(+1.65)}}$ | $79.99^{\textbf{(+1.68)}}$
(Average of SM of 20 RCs + 20 MRCs) | ✗ | $72.24^{\textbf{(+2.48)}}$ | $75.49^{\textbf{(+2.18)}}$ | $77.79^{\textbf{(+1.66)}}$ | $79.03^{\textbf{(+1.66)}}$ | $79.99^{\textbf{(+1.68)}}$
FCs + MFCs | (Average of FV of 5 FCs) | ✗ | $71.31^{\textbf{(+1.55)}}$ | $74.78^{\textbf{(+1.47)}}$ | $77.12^{\textbf{(+0.99)}}$ | $78.66^{\textbf{(+1.29)}}$ | $79.40^{\textbf{(+1.09)}}$
(Average of FV of 5 FCs + 5 MFCs) | ✗ | $71.85^{\textbf{(+2.09)}}$ | $75.27^{\textbf{(+1.96)}}$ | $77.44^{\textbf{(+1.31)}}$ | $78.93^{\textbf{(+1.56)}}$ | $79.73^{\textbf{(+1.42)}}$
(Average of SM of 5 FCs) | ✗ | $71.70^{\textbf{(+1.94)}}$ | $75.09^{\textbf{(+1.78)}}$ | $77.35^{\textbf{(+1.22)}}$ | $78.84^{\textbf{(+1.47)}}$ | $79.69^{\textbf{(+1.38)}}$
(Average of SM of 5 FCs + 5 MFCs) | ✗ | $72.23^{\textbf{(+2.47)}}$ | $75.63^{\textbf{(+2.32)}}$ | $77.66^{\textbf{(+1.53)}}$ | $79.15^{\textbf{(+1.78)}}$ | $80.01^{\textbf{(+1.70)}}$
|  MEAL [5] | Training Required | - | - | 76.42 | - | -
|  MEAL Plus [5] | Training Required | - | - | 78.21 | - | -
|  MEAL V2 [4] | Training Required | 73.19 | - | 80.67 | - | -
|  Baseline‡ | - | N/A | 76.32 | 78.13 | N/A
|  Cutout‡ [8] | Training Required | 77.07 | 79.28
|  Mixup‡ [7] | Training Required | 77.42 | 79.48
|  Manifold Mixup‡ [22] | Training Required | 77.50 | -
|  CutMix‡ [6] | Training Required | 78.60 | 79.83
FixRes | Base FixRes | Training Required | - | - | 77.0 | - | -
$(\text{adaptation + augmentation})$ | Training Required | - | - | 77.1 | - | -
$(\text{adaptation + augmentation + @ 384})$ | Training Required | - | - | 79.1 | - | -
$(\text{adaptation + augmentation + @ 320 + Billion-scale})$ | Training Required | - | - | 82.5 | - | -

TABLE II: Comparison of Top-1 validation accuracy on ImageNet dataset for
MobileNet family [28, 29]. The performance gain in each case is noted in
parentheses. $\dagger$ indicates the numbers reported on the PyTorch page and
verified on our end. Whereas ‡ indicates the numbers taken from [4]. The
numbers reported for MobileNet baselines in [4] do not match with PyTorch and
explicitly copied here.

Approach | Custom Training | MobileNet V2 | MobileNet V3-Small | MobileNet V3-Large
---|---|---|---|---
Feature Dimension | - | 1280 | 576 | 960
Image Resized To | - | 256 | 256 | 256
Input Crop Size | - | $224\times 224$ | $224\times 224$ | $224\times 224$
1.0 - (Crop to Image Ratio) | - | 0.125 | 0.125 | 0.125
Center Crop $\dagger$ | - | 71.88 | 67.67 | 74.04
MID (Average of FV of 10 RCs) | ✗ | $73.55^{\textbf{(+1.67)}}$ | $69.79^{\textbf{(+2.12)}}$ | $75.38^{\textbf{(+1.34)}}$
MID (Average of FV of 20 RCs) | ✗ | $73.66^{\textbf{(+1.78)}}$ | $69.87^{\textbf{(+2.20)}}$ | $75.40^{\textbf{(+1.36)}}$
MID (Average of SM of 10 RCs) | ✗ | $73.76^{\textbf{(+1.88)}}$ | $70.06^{\textbf{(+2.39)}}$ | $75.52^{\textbf{(+1.48)}}$
MID (Average of SM of 20 RCs) | ✗ | $73.88^{\textbf{(+2.00)}}$ | $70.27^{\textbf{(+2.60)}}$ | $75.66^{\textbf{(+1.62)}}$
Baseline‡ | - | - | 67.40 | 75.20
MEAL V2‡ [4] | Training Required | - | 69.65 | 76.92

TABLE III: Comparison of Top-1 validation accuracy on ImageNet dataset for
EfficientNet [14] family (tf_efficientnet_bx_ap variant). The performance gain
in each case is noted in parentheses. $\dagger$ indicates the numbers reported
on the Timm’s page [30] and verified on our end. Numbers reported with ‡ are
taken from [17]. Since, FixEfficientNet re-trains networks for different
resolutions; these test resolutions are noted in the last row.

Approach | Custom Training | B0 | B1 | B2 | B3 | B4 | B5 | B6 | B7 | B8
---|---|---|---|---|---|---|---|---|---|---
Feature Dimension | - | 1280 | 1280 | 1408 | 1536 | 1792 | 2048 | 2304 | 2560 | 2816
Image Resized To | - | 256 | 272 | 292 | 332 | 412 | 488 | 562 | 632 | 704
Input Crop Size | - | $224\times 224$ | $240\times 240$ | $260\times 260$ | $300\times 300$ | $380\times 380$ | $456\times 456$ | $528\times 528$ | $600\times 600$ | $672\times 672$
1.0 - (Crop to Image Ratio) | - | 0.125 | 0.118 | 0.110 | 0.096 | 0.078 | 0.066 | 0.060 | 0.051 | 0.046
Center Crop $\dagger$ | - | 77.09 | 79.28 | 80.30 | 81.82 | 83.25 | 84.25 | 84.79 | 85.12 | 85.37
MID (Average of FV of 10 RCs) | ✗ | $78.24^{\textbf{(+1.15)}}$ | $80.10^{\textbf{(+0.82)}}$ | $80.93^{\textbf{(+0.63)}}$ | $82.32^{\textbf{(+0.50)}}$ | $83.62^{\textbf{(+0.37)}}$ | $84.52^{\textbf{(+0.27)}}$ | $84.96^{\textbf{(+0.17)}}$ | $85.25^{\textbf{(+0.13)}}$ | $85.49^{\textbf{(+0.12)}}$
MID (Average of FV of 20 RCs) | ✗ | $78.32^{\textbf{(+1.23)}}$ | $80.15^{\textbf{(+0.87)}}$ | $80.97^{\textbf{(+0.67)}}$ | $82.37^{\textbf{(+0.55)}}$ | $83.68^{\textbf{(+0.43)}}$ | $84.57^{\textbf{(+0.32)}}$ | $84.96^{\textbf{(+0.17)}}$ | $85.27^{\textbf{(+0.15)}}$ | $85.50^{\textbf{(+0.13)}}$
MID (Average of SM of 10 RCs) | ✗ | $78.40^{\textbf{(+1.31)}}$ | $80.24^{\textbf{(+0.96)}}$ | $81.10^{\textbf{(+0.80)}}$ | $82.41^{\textbf{(+0.59)}}$ | $83.75^{\textbf{(+0.50)}}$ | $84.62^{\textbf{(+0.37)}}$ | $85.03^{\textbf{(+0.22)}}$ | $85.32^{\textbf{(+0.20)}}$ | $85.55^{\textbf{(+0.18)}}$
MID (Average of SM of 20 RCs) | ✗ | $78.43^{\textbf{(+1.34)}}$ | $80.34^{\textbf{(+1.06)}}$ | $81.16^{\textbf{(+0.86)}}$ | $82.47^{\textbf{(+0.65)}}$ | $83.79^{\textbf{(+0.54)}}$ | $84.69^{\textbf{(+0.44)}}$ | $85.06^{\textbf{(+0.25)}}$ | $85.34^{\textbf{(+0.22)}}$ | $85.58^{\textbf{(+0.21)}}$
EfficientNet AdvProp‡ [18] | - | 77.6 | 79.6 | 80.5 | 81.9 | 83.3 | 84.3 | 84.8 | 85.2 | 85.5
FixEfficientNet AdvProp‡ [17] | Training Required | 79.3 | 81.3 | 82.0 | 83.0 | 84.0 | 84.7 | 84.9 | 85.3 | 85.7
FixEfficientNet Test Res‡ [17] | - | 320 | 384 | 420 | 472 | 512 | 576 | 576 | 632 | 800

##  IV Experimental Settings

###  IV-A Data Set

We conduct experiments on ILSVRC-2012 benchmark for classification [1] where
models have been trained to discriminate among 1,000 classes using 1.2 million
images in the training split. Since, we do not employ any training, we use
50,000 images in the validation split for evaluation.

###  IV-B Networks

We evaluated four different families of modern deep networks for object
recognition. EfficientNet and NFNet are the best performing state-of-the-art
models in supervised setting. Whereas ResNet and MobileNet architectures are
of practical deployment importance due to their smaller memory footprint. For
each model, we evaluated averaging at deep feature and softmax levels.
Averaging at the logit level results in numbers identical to the ones
resulting from averaging of feature vectors and not reported for brevity. For
ResNet models (Tab I), we have investigated fixed crops (FCs), mirrored fixed
crops (MFCs), random crops (RCs), mirrored random crops (MRCs), and
additionally combining RCs and MRCs (Tab VIII). For other models we
constrained the evaluation to 10 or 20 random crops per image, and
demonstrated performance gains on par with expensive training-based
approaches.

ResNet [13] We use the pre-trained models from PyTorch [27], resize the input
image to $256$ respecting the aspect ratio and then take $224\times 224$
central or random crop. For resizing, bilinear interpolation is employed.
Results available in Tab I.

MobileNet [28, 29] For MobileNet V2 [28] and V3 [29], we use the pre-trained
models from PyTorch [27]. For V3, both small and large models are evaluated.
The input size, crops size, and the interpolation method stay the same as that
for ResNet. Results are listed in Tab II.

EfficientNet [14] For EfficientNet architecture, we use the pre-trained models
from Timm library [31] where weights have been converted from TensorFlow to
PyTorch. Timm library provides the original pre-trained models [14] as well as
the NoisyStudent [19] and AdvProp [18] variants. For our experiments we used
four EfficientNet variants from Timm; identified by notation: efficientnet_bx,
tf_efficientnet_bx, tf_efficientnet_bx_ns, and tf_efficientnet_bx_ap where x
can range from 0 to 8; tf, ns, and ap refers to TensorFlow, NoisyStudent, and
AdvProp respectively. For efficientnet_bx, only five pre-trained models (b0
through b4) are available. For each EfficientNet architecture, the input image
is resized to the specific input size and then specific central or random crop
is taken. Specific input and crop sizes are noted in respective tables. To
comply with Timm [31], we use bicubic interpolation. Results for
tf_efficientnet_bx_ap, tf_efficientnet_bx_ns, efficientnet_bx, and
tf_efficientnet_bx are respectively available in Tabs III, V, VI, and VII.

NFNet [15] NFNet is the current state-of-the-art architecture for image
recognition. Again we used pre-trained models from Timm [31]. Results for
NFNet variants are available in Tab IV with respective input and crop size
information. Bicubic interpolation is used for resizing.

##  V Results

###  V-A Comparison against SOTA

ResNet Results for ResNet family are shown in Tab I. We report Top-1 accuracy
for each ResNet model as reported on PyTorch page [27] for single central-crop
evaluation. For each ResNet model, we report the performance gain due to
averaging of the feature vectors (FV) and softmax scores (SM), and use 10 or
20 random crops in each evaluation. We see a consistent improvement for each
model with performance gain in Top-1 accuracy ranging from 1.08% to 2.07%. The
performance improves with increasing the number of random crops being averaged
regardless of the underlying model. Additionally averaging the softmax scores
results in better performance than averaging of feature vectors. We should
further note that performance gain for smaller models (e.g., ResNet-18 and
ResNet-34) is higher than the larger ones (e.g., ResNet-50, ResNet-101, and
ResNet-152). We provide the comparison against several recent training-based
approaches [5, 4, 2, 6] and list whichever numbers are available for any of
the five ResNet models. It should be noted that many such approaches chose to
focus on ResNet-50. It is interesting to see that averaging of random crops
can achieve comparable performance to some of the training-based methods. For
examples, for ResNet-50, averaging is either outperforming (MEAL, Cutout,
Mixup, FixRes, FixRes (adaptation + augmentation)) or comparable (Manifold
Mixup) to training-based approaches which require re-training the model and/or
exploiting datasets beyond conventional ImageNet-2012 train split.

Supplementing random crops with mirrored versions of additional random crops
as well as using fixed crops and their flipped versions as originally
suggested by AlexNet further boosts the performance. We conduct an ablation
demonstrating that combining fixed and random crops results in the best
achievable performance.

Figure 3: Top-1 accuracy of ResNet-18/ResNet-50 as the function of number of
RCs being averaged. When only a single crop is used per image, on average
using central crop (CC) is better than RC. However as the number of RCs being
averaged increases (even average of two RCs), the performance increases. The
performance is saturated after about 20 RCs. We also demonstrate that
including CC in averaging helps improving the performance when number of RCs
per image is less than 5. In general using even a single RC in addition to the
CC results in better performance than using just the central one.

MobileNet We document results for MobileNet V2 [28], and V3 [29] in Tab II
where we first list the numbers from PyTorch page [27] for V2 and two variants
of V3. Averaging of random crops results in consistent improvement for all
three MobileNet models with better performance gains for larger number of
crops and softamx averaging outperforming feature averaging. The performance
gain in Top-1 accuracy for MobileNet family ranges from 1.34% to 2.60%.
Similar to ResNet, performance gain for smaller model (V3-Small) is more than
that for larger ones (V2 and V3-Larger). In Tab II, we also report the numbers
for one of the training-based approach i.e., MEAL V2 [4]. The performance
improvement for MEAL V2 over the baseline V3-Small and V3-Large is 2.25% and
1.72% respectively, whereas, averaging of random crops results in 2.60% and
1.62% gain in Top-1 accuracy i.e., outperforming MEAL V2 for V3-Small and
comparable for V3-Large.

EfficientNet Results for EfficientNet AdvProp are available in Tab III & Fig
5. The results for other EfficientNet variants can be found in Tabs VI, VII,
and V. For AdvProp variant, the performance gain ranges from 0.12% to 1.34%.
We notice a larger gain for smaller models, and consistent with earlier
results that larger number of crops and averaging at sofmax level helps. We
provide a comparison against FixEfficientNet [17] which re-trains the
underlying networks on test resolutions and employs additional data beyond
ImageNet-2012 training set. It is interesting to note that for larger networks
(e.g., B5 – B8), averaging of random crops is comparable or outperforming
FixEfficientNet, which raises the question if training such big models with
additional million/billion scale data is worth the effort? Additionally as the
crop to image ratio becomes larger for such bigger networks, the gains due to
several random crop averaging diminishes as well, we explicitly discuss this
phenomenon in the next subsection.

NFNet We report the results for NFNet family in Tab IV where we see the Top-1
performance gain ranging from 0.05% to 0.63%. The performance gain for NFNet
is low compared to EfficientNet. This is partly due to larger crop to image
ratio as there is relatively smaller area around the center crop from which a
random crop can be chosen.

###  V-B Additional Results & Analysis

Number of Random Crops The number of random crops being averaged plays an
important role in squeezing the performance out of the pre-trained models.
Generally a larger number of random crops per image results in better
performance, as seen in almost all results where irrespective of the
underlying architecture, averaging of 20 crops resulted in better performance
than averaging of 10. In Fig 3, we study the performance gain as a function of
number of crops being averaged for ResNet-18/ResNet-50 and a similar figure
(Fig 7) for the complete ResNet family is also provided. It is apparent that
including more crops in averaging results in better performance, however,
performance saturates after about 20 random crops.

Importance of Central Crop In general, central crop of a resized image
provides a reasonable coverage of the object of interest. To study the
importance of center crop, we included its representation in the average. As
clear from Fig 3, including central crop in averaging definitely helps when
the number of random crops is less than 5. As the number of random crops
increases, the importance of including center crop in averaging diminishes. It
is also interesting to note that even including one extra random crop per
image in addition to the conventional center crop results in better
performance than just using the central crop.

Averaging of Feature Vectors vs SoftMax Scores We have investigated averaging
the representation of random crops at feature vector, logit, and softmax score
levels. Averaging at the logit level results into identical performance as
that of averaging at the deep feature level. Whereas averaging at the softmax
level results in better performance than that of feature averaging as can be
seen throughout the results (Tab. I – VII). It is worth noting that averaging
of softmax has also been investigated by others [4] as a soft supervisory
label to train better student networks.

Smaller vs Larger Models Overall the performance gain is higher for smaller
models than for the larger ones. This is true regardless of the network
family. This is partially due to the reason that larger input is used for
larger networks (e.g., B8 compared to B0). For ResNet family where the input
crop size stays the same, we notice a larger gain for small models (e.g.,
ResNet-18) than larger ones (e.g., ResNet-152). Higher performance gain for
smaller models and lower for larger ones is also consistent with training
based methods e.g., [17] (Tabs III, V).

TABLE IV: Comparison of Top-1 validation accuracy on ImageNet dataset for
NFNet [15] family. The performance gain in each case is noted in parentheses.
$\dagger$ indicates the numbers reported on the Timm’s page [30] and verified
on our end. Softmax (SM) scores averaging is better than feature (FV)
averaging. The performance gain is high for smaller networks and low for
larger ones.

Approach | Custom Training | F0 | F1 | F2 | F3 | F4 | F5 | F6
---|---|---|---|---|---|---|---|---
Feature Dimension | - | 3072 | 3072 | 3072 | 3072 | 3072 | 3072 | 3072
Image Resized To | - | 284 | 351 | 382 | 443 | 538 | 570 | 602
Input Crop Size | - | $256\times 256$ | $320\times 320$ | $352\times 352$ | $416\times 416$ | $512\times 512$ | $544\times 544$ | $576\times 576$
1.0 - (Crop to Image Ratio) | - | 0.099 | 0.088 | 0.079 | 0.061 | 0.048 | 0.046 | 0.043
Center Crop $\dagger$ | - | 83.34 | 84.60 | 84.99 | 85.56 | 85.66 | 85.71 | 86.30
MID (Average of FV of 10 RCs) | ✗ | $83.67^{\textbf{(+0.33)}}$ | $84.79^{\textbf{(+0.19)}}$ | $85.13^{\textbf{(+0.14)}}$ | $85.79^{\textbf{(+0.23)}}$ | $85.84^{\textbf{(+0.18)}}$ | $85.91^{\textbf{(+0.20)}}$ | $86.35^{\textbf{(+0.05)}}$
MID (Average of FV of 20 RCs) | ✗ | $83.73^{\textbf{(+0.39)}}$ | $84.81^{\textbf{(+0.21)}}$ | $85.13^{\textbf{(+0.14)}}$ | $85.82^{\textbf{(+0.26)}}$ | $85.85^{\textbf{(+0.19)}}$ | $85.94^{\textbf{(+0.23)}}$ | $86.33^{\textbf{(+0.03)}}$
MID (Average of SM of 10 RCs) | ✗ | $83.77^{\textbf{(+0.53)}}$ | $84.90^{\textbf{(+0.30)}}$ | $85.22^{\textbf{(+0.23)}}$ | $85.87^{\textbf{(+0.31)}}$ | $85.91^{\textbf{(+0.25)}}$ | $85.95^{\textbf{(+0.24)}}$ | $86.43^{\textbf{(+0.13)}}$
MID (Average of SM of 20 RCs) | ✗ | $83.87^{\textbf{(+0.63)}}$ | $84.92^{\textbf{(+0.32)}}$ | $85.20^{\textbf{(+0.21)}}$ | $85.89^{\textbf{(+0.33)}}$ | $85.90^{\textbf{(+0.24)}}$ | $86.02^{\textbf{(+0.31)}}$ | $86.41^{\textbf{(+0.11)}}$

Importance of Crop to Image Ratio Another reason behind better performance for
smaller models is due to crop to image size ratio which we note for each model
and all the network families in their respective tables. When this ratio is
smaller, there is more space for the selection of random crop, and selecting
more random crops provides better coverage of the underlying object. This can
be noticed for EfficientNet (Tabs III, V, VI, VII) and NFNet (Tab IV) where
smaller models like B0 and F0 have smaller crop to image ratio than larger
ones like B7/B8 and F6. Comparing the performance gains for NFNet (Tab IV)
against EfficientNet (Tab III), we can see comparatively larger gains for
EfficientNet due to relatively smaller crop to image ratios e.g., 0.875 for B0
compared to 0.901 for F0. Similarly, comparing efficientnet_bx variant (Tab
VI) against other variants specifically tf_efficientnet_bx_ns (Tab V), and
tf_efficientnet_bx_ap (Tab III), we notice smaller gains especially for B1 –
B4 as the images are resized to the actual crop size and wiggle room to choose
random crops is only available in one dimension. Another related issue is the
size of the underlying image after resizing as can be seen in Fig 4 where
number of images in ILSVRC-2012 val split having longer dimension larger than
500 pixels are relatively fewer. That is why models that resize an image to
more than 500 pixels (e.g., B6 – B8, F4 – F6, and L2) do not gain much due to
averaging of random crops as there is not much room.

Figure 4: Frequency distribution of longer side of images in ILSVRC-2012 val
split when shorter side is resized to 256 pixels. Noticeably many images are
larger than 256 pixels, and have a good chance of benefiting from averaging of
RCs. We should further note that after resizing, fewer images have longer
sides greater than 400 pixels, so a strategy to choose number of crops as a
function of image size is effective both computationally and performance-wise.

Results for Other EfficientNet Variants The results for other EfficientNet
variants i.e., efficientnet_bx, tf_efficientnet_bx, and tf_efficientnet_bx_ns
are available in Tabs VI, VII, and V respectively. For best performing
EfficientNet variants i.e., NoisyStudent [19] and AdvProp [18], we are able to
achieve comparable performance to that of FixEfficientNet [17] without
requiring expensive million scale training. The visual performance comparison
for AdvProp and NoisyStudent variants against FixEfficientNet versions is
available in Figs 5 and 6 respectively. We should note for efficientnet_bx
only four pre-trained models (B0 – B4) are available from Timm [31]. For
tf_efficientnet_bx variant, only B0 comparison (i.e., MEAL V2 [4]) is
available and has been listed in the respective table (Tab VII).

Figure 5: Performance comparison of EfficienNet AdvProp [18] compared to
baselines (blue curve) reported by Timm [30]. Our performance (red curve) is
comparable to training-based FixEfficientNet [17] (green curve). Figure 6:
Performance comparison of EfficienNet NoisyStudent [19] compared to baselines
(blue curve) reported by Timm [30]. Our performance (red curve) is comparable
to training-based FixEfficientNet [17] (green curve). TABLE V: Comparison of
Top-1 validation accuracy on ImageNet dataset for EfficientNet [14] family
(tf_efficientnet_bx_ns variant). The performance gain in each case is noted in
parentheses. $\dagger$ indicates the numbers reported on the Timm’s page [30]
and verified on our end. Numbers reported with ‡ are taken from [17]. Since,
FixEfficientNet re-trains networks for different resolutions; these test
resolutions are noted in the last row. Performing averaging of the softmax
(SM) scores of random crops (RCs) results in better performance than that of
averaging at the deep feature vector (FV) level. The performance gain is high
for smaller networks and low for larger ones.

Approach | Custom Training | B0 | B1 | B2 | B3 | B4 | B5 | B6 | B7 | L2
---|---|---|---|---|---|---|---|---|---|---
Feature Dimension | - | 1280 | 1280 | 1408 | 1536 | 1792 | 2048 | 2304 | 2560 | 5504
Image Resized To | - | 256 | 272 | 292 | 332 | 412 | 488 | 562 | 632 | 833
Input Crop Size | - | $224\times 224$ | $240\times 240$ | $260\times 260$ | $300\times 300$ | $380\times 380$ | $456\times 456$ | $528\times 528$ | $600\times 600$ | $800\times 800$
1.0 - (Crop to Image Ratio) | - | 0.125 | 0.118 | 0.110 | 0.096 | 0.078 | 0.066 | 0.060 | 0.051 | 0.038
Center Crop $\dagger$ | - | 78.66 | 81.39 | 82.38 | 84.05 | 85.16 | 86.09 | 86.45 | 86.84 | 88.35
Ours (Average of FV of 10 RCs) | ✗ | $79.55^{\textbf{(+0.89)}}$ | $82.05^{\textbf{(+0.66)}}$ | $82.92^{\textbf{(+0.54)}}$ | $84.52^{\textbf{(+0.47)}}$ | $85.41^{\textbf{(+0.25)}}$ | $86.27^{\textbf{(+0.18)}}$ | $86.62^{\textbf{(+0.17)}}$ | $86.92^{\textbf{(+0.08)}}$ | $88.35^{\textbf{(+0.00)}}$
Ours (Average of FV of 20 RCs) | ✗ | $79.65^{\textbf{(+0.99)}}$ | $82.16^{\textbf{(+0.77)}}$ | $83.05^{\textbf{(+0.67)}}$ | $84.58^{\textbf{(+0.53)}}$ | $85.48^{\textbf{(+0.32)}}$ | $86.26^{\textbf{(+0.17)}}$ | $86.60^{\textbf{(+0.15)}}$ | $86.96^{\textbf{(+0.12)}}$ | $88.40^{\textbf{(+0.05)}}$
Ours (Average of SM of 10 RCs) | ✗ | $79.76^{\textbf{(+1.10)}}$ | $82.26^{\textbf{(+0.87)}}$ | $83.10^{\textbf{(+0.72)}}$ | $84.68^{\textbf{(+0.63)}}$ | $85.53^{\textbf{(+0.37)}}$ | $86.33^{\textbf{(+0.24)}}$ | $86.67^{\textbf{(+0.20)}}$ | $87.0^{\textbf{(+0.16)}}$ | $88.39^{\textbf{(+0.04)}}$
Ours (Average of SM of 20 RCs) | ✗ | $79.84^{\textbf{(+1.18)}}$ | $82.36^{\textbf{(+0.97)}}$ | $83.21^{\textbf{(+0.83)}}$ | $84.69^{\textbf{(+0.64)}}$ | $85.59^{\textbf{(+0.43)}}$ | $86.36^{\textbf{(+0.27)}}$ | $86.68^{\textbf{(+0.21)}}$ | $87.04^{\textbf{(+0.20)}}$ | $88.42^{\textbf{(+0.07)}}$
EfficientNet NoisyStudent‡ [19] | - | 78.8 | 81.5 | 82.4 | 84.1 | 85.3 | 86.1 | 86.4 | 86.9 | 88.4
FixEfficientNet NoisyStudent‡ [17] | Training Required | 80.2 | 82.6 | 83.6 | 85.0 | 85.9 | 86.4 | 86.7 | 87.1 | 88.5
FixEfficientNet Test Res‡ [17] | - | 320 | 384 | 420 | 472 | 472 | 576 | 680 | 632 | 600

TABLE VI: Comparison of Top-1 validation accuracy on ImageNet dataset for
EfficientNet [14] family (efficientnet_bx variant). It should be noted only
five pre-trained models (B0 – B4) are available for this variant. The
performance gain in each case is noted in parentheses. $\dagger$ indicates the
numbers reported on the Timm’s page [30] and verified on our end. Performing
averaging of the softmax (SM) scores of random crops (RCs) results in better
performance than that of averaging at the deep feature vector (FV) level. The
performance gain is high for smaller networks and low for larger ones.

Approach | Custom Training | B0 | B1 | B2 | B3 | B4
---|---|---|---|---|---|---
Feature Dimension | - | 1280 | 1280 | 1408 | 1536 | 1792
Image Resized To | - | 256 | 256 | 288 | 320 | 384
Input Crop Size | - | $224\times 224$ | $256\times 256$ | $288\times 288$ | $320\times 320$ | $384\times 384$
1.0 - (Crop to Image Ratio) | - | 0.125 | 0.000 | 0.000 | 0.000 | 0.000
Center Crop $\dagger$ | - | 77.70 | 78.80 | 80.61 | 82.24 | 83.43
Ours (Average of FV of 10 RCs) | ✗ | $78.59^{\textbf{(+0.89)}}$ | $79.37^{\textbf{(+0.57)}}$ | $81.01^{\textbf{(+0.40)}}$ | $82.49^{\textbf{(+0.25)}}$ | $83.54^{\textbf{(+0.11)}}$
Ours (Average of FV of 20 RCs) | ✗ | $78.68^{\textbf{(+0.98)}}$ | $79.34^{\textbf{(+0.54)}}$ | $81.05^{\textbf{(+0.44)}}$ | $82.49^{\textbf{(+0.25)}}$ | $83.63^{\textbf{(+0.20)}}$
Ours (Average of SM of 10 RCs) | ✗ | $78.88^{\textbf{(+1.18)}}$ | $79.46^{\textbf{(+0.66)}}$ | $81.12^{\textbf{(+0.51)}}$ | $82.60^{\textbf{(+0.36)}}$ | $83.62^{\textbf{(+0.19)}}$
Ours (Average of SM of 20 RCs) | ✗ | $78.99^{\textbf{(+1.29)}}$ | $79.46^{\textbf{(+0.66)}}$ | $81.19^{\textbf{(+0.58)}}$ | $82.58^{\textbf{(+0.34)}}$ | $83.70^{\textbf{(+0.27)}}$

TABLE VII: Comparison of Top-1 validation accuracy on ImageNet dataset for
EfficientNet [14] family (tf_efficientnet_bx variant). The performance gain in
each case is noted in parentheses. $\dagger$ indicates the numbers reported on
the Timm’s page [30] and verified on our end. Numbers reported for other
approaches are taken from their respective papers. Performing averaging of the
softmax (SM) scores of random crops (RCs) results in better performance than
that of averaging at the deep feature vector (FV) level. The performance gain
is high for smaller networks and low for larger ones.

Approach | Custom Training | B0 | B1 | B2 | B3 | B4 | B5 | B6 | B7 | B8
---|---|---|---|---|---|---|---|---|---|---
Feature Dimension | - | 1280 | 1280 | 1408 | 1536 | 1792 | 2048 | 2304 | 2560 | 2816
Image Resized To | - | 256 | 272 | 292 | 332 | 412 | 488 | 562 | 632 | 704
Input Crop Size | - | $224\times 224$ | $240\times 240$ | $260\times 260$ | $300\times 300$ | $380\times 380$ | $456\times 456$ | $528\times 528$ | $600\times 600$ | $672\times 672$
1.0 - (Crop to Image Ratio) | - | 0.125 | 0.118 | 0.110 | 0.096 | 0.078 | 0.066 | 0.060 | 0.051 | 0.046
Center Crop $\dagger$ | - | 76.85 | 78.83 | 80.09 | 81.64 | 83.02 | 83.81 | 84.11 | 84.94 | 85.37
Ours (Average of FV of 10 RCs) | ✗ | $77.78^{\textbf{(+0.93)}}$ | $79.56^{\textbf{(+0.73)}}$ | $80.75^{\textbf{(+0.66)}}$ | $82.14^{\textbf{(+0.50)}}$ | $83.29^{\textbf{(+0.27)}}$ | $84.03^{\textbf{(+0.22)}}$ | $84.27^{\textbf{(+0.16)}}$ | $85.06^{\textbf{(+0.12)}}$ | $85.52^{\textbf{(+0.15)}}$
Ours (Average of FV of 20 RCs) | ✗ | $77.83^{\textbf{(+0.98)}}$ | $79.72^{\textbf{(+0.89)}}$ | $80.87^{\textbf{(+0.78)}}$ | $82.16^{\textbf{(+0.52)}}$ | $83.34^{\textbf{(+0.32)}}$ | $84.11^{\textbf{(+0.30)}}$ | $84.33^{\textbf{(+0.22)}}$ | $85.10^{\textbf{(+0.16)}}$ | $85.53^{\textbf{(+0.16)}}$
Ours (Average of SM of 10 RCs) | ✗ | $77.98^{\textbf{(+1.13)}}$ | $79.73^{\textbf{(+0.90)}}$ | $80.91^{\textbf{(+0.82)}}$ | $82.23^{\textbf{(+0.59)}}$ | $83.42^{\textbf{(+0.40)}}$ | $84.14^{\textbf{(+0.33)}}$ | $84.38^{\textbf{(+0.27)}}$ | $85.14^{\textbf{(+0.20)}}$ | $85.60^{\textbf{(+0.23)}}$
Ours (Average of SM of 20 RCs) | ✗ | $78.03^{\textbf{(+1.18)}}$ | $79.89^{\textbf{(+1.06)}}$ | $81.05^{\textbf{(+0.96)}}$ | $82.29^{\textbf{(+0.65)}}$ | $83.45^{\textbf{(+0.43)}}$ | $84.20^{\textbf{(+0.39)}}$ | $84.45^{\textbf{(+0.34)}}$ | $85.21^{\textbf{(+0.27)}}$ | $85.58^{\textbf{(+0.21)}}$
MEAL V2 [4] | Training Required | 78.29 | - | - | - | - | - | - | - | -

Ablation: Mirrored & Fixed Crops Using ResNet models as the base, in Tab VIII
we have investigated various additional settings. As reported in Tab I; in
addition to using just the random crops (RCs) only, we have explored using
mirrored random crops (MRCs). While keeping the number of total crops fixed to
10 or 20, and comparing (i) using only RCs versus (ii) using a combination of
RCs and MRCs, the lateral always results in better performance regardless of
averaging at softmax or feature level. Focusing on ResNet-18 feature averaging
and having the number of crops fixed to 10 and 20, a mixture of RCs and MRCs
results in better performance (71.65 & 71.85) compared to using RCs only
(71.42 & 71.56). This directly confirms our hypothesis of matching the
inference-time distribution with that of training; as during training not only
the RCs are employed but generally MRCs are also incorporated with a
probability of 0.5.

AlexNet [12] originally suggested to use 5 fixed crops (FCs) – 1 central and 4
corners as an attempt to match the train-test distribution. Although FCs might
be sufficient for majority of square images; they may not work for elongated
ones. We combined 5 FCs with an increased number of RCs (5, 10, 15, and 20)
which resulted into consistent better performance (Tab VIII). Continuing with
ResNet-18 feature averaging examples, we can see that including 5 & 10 RCs in
the average resulted into Top-1 accuracy of 71.49 and 71.59 compared to using
only 5 FCs (71.31). This trend holds for larger ResNet models and averaging at
the softmax layer.

To further bridge the distributional gap, we also investigated using the
mirrored versions of the fixed crops (MFCs) and additional mirrored random
crops. We should note that AlexNet suggested to use the flipped versions of
the same fixed crops whereas the flipped versions in our approach are not
necessarily of the same random crops and hence even better align with the
train-time distribution. We have identified the best achieved Top-1
performance for all ResNet models in bold font in VIII where we can see an
average of fixed, random, mirrored-fixed, and mirrored-random crops have
resulted in the best numbers.

TABLE VIII:  Comparison of Top-1 validation accuracy on ImageNet dataset for
ResNet [13] family. † indicates the numbers reported on the PyTorch page [27]
and verified on our end. Compared to fixed crops (FCs) and mirrored fixed
crops (MFCs), adding RCs and MRCs into the average results in enhanced
performance. The best achieved performance (highlighted in bold) results when
a mix of FCs, MFCs, RCs and MRCs is used. This is due to the fact that fixed
crops may be sufficient for square images but the elongated images can further
benefit from RCs and MRCs that provide additional coverage of the underlying
object. Additionally since the MRCs are not necessarily the flipped versions
of the RCs, rather independent, so they further bridge the train-test
distributional gap.

Approach | Custom Training | ResNet-18 | ResNet-34 | ResNet-50 | ResNet-101 | ResNet-152
---|---|---|---|---|---|---
Image Resized To | 256
Input Crop Size | $224\times 224$
1.0 - (Crop to Image Ratio) | 0.125
Feature Dimension | 512 | 2048
|  Center Crop† | - | 69.76 | 73.31 | 76.13 | 77.37 | 78.31
AlexNet | (Average of FV of 5 FCs) | ✗ | $71.31$ | $74.78$ | $77.12$ | $78.66$ | $79.40$
(Average of FV of 5 FCs + 5 MFCs) | ✗ | $71.85$ | $75.27$ | $77.44$ | $78.93$ | $79.73$
(Average of SM of 5 FCs) | ✗ | $71.70$ | $75.09$ | $77.35$ | $78.84$ | $79.69$
(Average of SM of 5 FCs + 5 MFCs) | ✗ | $72.23$ | $75.63$ | $77.66$ | $79.15$ | $80.01$
AlexNet + Ours | (Average of FV of 5 FCs + 5 RCs) | ✗ | $71.49$ | $74.99$ | $77.25$ | $78.69$ | $79.56$
(Average of FV of 5 FCs + 10 RCs) | ✗ | $71.59$ | $75.04$ | $77.33$ | $78.71$ | $79.63$
(Average of FV of 5 FCs + 15 RCs) | ✗ | $71.63$ | $75.00$ | $77.41$ | $78.78$ | $79.63$
(Average of FV of 5 FCs + 20 RCs) | ✗ | $71.64$ | $75.02$ | $77.40$ | $78.80$ | $79.62$
(Average of FV of 5 FCs + 5 MFCs + 5 RCs + 5 MRCs) | ✗ | $71.97$ | $75.32$ | $77.67$ | $78.98$ | $79.84$
(Average of FV of 5 FCs + 5 MFCs + 10 RCs + 10 MRCs) | ✗ | $71.99$ | $75.34$ | $77.67$ | $78.97$ | $79.89$
(Average of FV of 5 FCs + 5 MFCs + 15 RCs + 15 MRCs) | ✗ | $71.97$ | $75.33$ | $77.60$ | $79.02$ | $79.87$
(Average of FV of 5 FCs + 5 MFCs + 20 RCs + 20 MRCs) | ✗ | $71.96$ | $75.37$ | $77.66$ | $78.98$ | $79.83$
(Average of SM of 5 FCs + 5 RCs) | ✗ | $71.87$ | $75.28$ | $77.45$ | $78.92$ | $79.81$
(Average of SM of 5 FCs + 10 RCs) | ✗ | $71.96$ | $75.31$ | $77.55$ | $78.96$ | $79.85$
(Average of SM of 5 FCs + 15 RCs) | ✗ | $72.01$ | $75.27$ | $77.64$ | $79.02$ | $79.86$
(Average of SM of 5 FCs + 20 RCs) | ✗ | $71.98$ | $75.28$ | $77.63$ | $79.04$ | $79.89$
(Average of SM of 5 FCs + 5 MFCs + 5 RCs + 5 MRCs) | ✗ | $72.36$ | 75.66 | $77.83$ | $79.13$ | $80.14$
(Average of SM of 5 FCs + 5 MFCs + 10 RCs + 10 MRCs) | ✗ | $72.36$ | $75.60$ | $77.87$ | 79.18 | 80.14
(Average of SM of 5 FCs + 5 MFCs + 15 RCs + 15 MRCs) | ✗ | 72.40 | $75.60$ | $77.86$ | $79.16$ | $80.13$
(Average of SM of 5 FCs + 5 MFCs + 20 RCs + 20 MRCs) | ✗ | $72.33$ | $75.61$ | 77.88 | $79.17$ | $80.11$

Figure 7: Top-1 accuracy of ResNet family as the function of number of random
crops (RCs) being averaged. We demonstrate when only a single crop is used per
image, on average using central crop (CC) is better than random crop (RC).
However as the number of random crops being averaged increases (even average
of two random crops), the performance increases. The performance is saturated
after about twenty RCs irrespective of the ResNet model. We also demonstrate
that including central crop in averaging helps in improving the performance
when number of random crops per image is less than five. In general using even
a single random crop in addition to the central crop results in better
performance than using just the central one. Figure 8: Performance comparison
of NFNet [15] compared to baselines (blue curve) reported by Timm [30]. Our
performance (red curve) for smaller model like F3 is comparable to larger ones
like F4/F5. A vendor can deploy a smaller model (fewer parameters) and still
achieve performance on par with larger ones.

##  VI Discussion

Through extensive evaluations, we have demonstrated that using central crop
for inference is not optimal. Matching the train-time augmentations by using
even a small number of random or fixed crops can provide performance better or
comparable to approaches that require custom training and additionally
leverage billion/million scale data. This is especially important for vendors
from a practical deployment view point. In such practical settings, does a
vendor really want to invoke a training that requires huge amounts of data and
computation to train existing models just to squeeze mere $1$ – $2$ % extra
performance? For example for NFNet [15], a smaller model like F1 or F3 can be
used to squeeze the performance equivalent to a larger model like F2 or F4/F5
(Fig 8). To this end, even approaches like Meal [5] or Meal-V2 [4] that do not
rely on extra data beyond ImageNet, still require performing inference of 1.2
million training images through several teacher networks to get the softmax
soft labels. A resource-constrained vendor can directly deploy the available
pre-trained models with rare extra processing required for non-central or
long-sided images. It is worth noting that performance gains through these
inference-time augmentation are mostly beneficial for smaller networks like
ResNet-18, MobileNet, and B0 etc., so additional inference does not cost much
especially in modern GPU era.

##  VII Conclusions

We have demonstrated that using central crop at inference-time is sub-optimal
and one can achieve the same performance out of pre-trained models with a
minor overhead due to employing multiple crops per image at inference time.
The evaluation conducted on various families of modern deep networks render
the performance gains due to several custom training strategies questionable.
While the experiments herein have shown that MID using just random crops
improves performance, future work should examine full distributional matching
including other augmentations used in training of the pre-trained networks
e.g., color-jittering.

## References

  * [1] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and F.-F. Li, “Imagenet: A large-scale hierarchical image database,” in Computer Vision and Pattern Recognition, 2009.
  * [2] H. Touvron, A. Vedaldi, M. Douze, and H. Jegou, “Fixing the train-test resolution discrepancy,” in Advances in Neural Information Processing Systems, 2019.
  * [3] I. Z. Yalniz, H. Jégou, K. Chen, M. Paluri, and D. Mahajan, “Billion-scale semi-supervised learning for image classification,” ArXiv, vol. arXiv:1905.00546, 2019.
  * [4] Z. Shen and M. Savvides, “Meal v2: Boosting vanilla resnet-50 to 80%+ top-1 accuracy on imagenet without tricks,” ArXiv, vol. arXiv:2009.08453v2, 2021\.
  * [5] Z. Shen, Z. He, and X. Xue, “Meal: Multi-model ensemble via adversarial learning,” in AAAI Conference on Artificial Intelligence, 2019.
  * [6] S. Yun, D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo, “Cutmix: Regularization strategy to train strong classifiers with localizable features,” in International Conference on Computer Vision, 2019.
  * [7] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz, “mixup: Beyond empirical risk minimization,” in International Conference on Learning Representations, 2018.
  * [8] T. DeVries and G. W. Taylor, “Improved regularization of convolutional neural networks with cutout,” ArXiv, vol. arXiv:1708.04552v2, 2017.
  * [9] Z. Zhong, L. Zheng, G. Kang, S. Li, and Y. Yang, “Random erasing data augmentation,” in AAAI Conference on Artificial Intelligence, 2020.
  * [10] A. G. Howard, “Some improvements on deep convolutional neural network based image classification,” in International Conference on Learning Representations, 2014.
  * [11] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich, “Going deeper with convolutions,” in Computer Vision and Pattern Recognition, 2015.
  * [12] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Advances in neural information processing systems, vol. 25, pp. 1097–1105, 2012.
  * [13] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Computer Vision and Pattern Recognition, 2016.
  * [14] M. Tan and Q. V. Le, “Efficientnet: Rethinking model scaling for convolutional neural networks,” in International Conference on Machine Learning,, 2019\.
  * [15] A. Brock, S. De, S. L. Smith, and K. Simonyan, “High-performance large-scale image recognition without normalization,” ArXiv, vol. arXiv:2102.06171v1, 2021.
  * [16] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He, “Aggregated residual transformations for deep neural networks,” in Computer Vision and Pattern Recognition, 2017.
  * [17] H. Touvron, A. Vedaldi, M. Douze, and H. Jégou, “Fixing the train-test resolution discrepancy: Fixefficientnet,” ArXiv, vol. arXiv:2003.08237v5, 2020.
  * [18] C. Xie, M. Tan, B. Gong, J. Wang, A. Yuille, and Q. V. Le, “Adversarial examples improve image recognition,” in Computer Vision and Pattern Recognition, 2020.
  * [19] Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le, “Self-training with noisy student improves imagenet classification,” in Computer Vision and Pattern Recognition, 2020.
  * [20] B. Recht, R. Roelofs, L. Schmidt, and V. Shankar, “Do imagenet classifiers generalize to imagenet?,” in International Conference on Machine Learning, 2019.
  * [21] L. Beyer, O. J. Hénaff, A. Kolesnikov, X. Zhai, and A. v. d. Oord, “Are we done with imagenet?,” ArXiv, vol. arXiv:2006.07159v1, 2020.
  * [22] V. Verma, A. Lamb, C. Beckham, A. Najafi, I. Mitliagkas, A. Courville, D. Lopez-Paz, and Y. Bengio, “Manifold mixup: Better representations by interpolating hidden states,” in International Conference on Machine Learning, 2019.
  * [23] C. Summers and M. J. Dinneen, “Improved mixed-example data augmentation,” in Winter Conference on Applications of Computer Vision, 2019.
  * [24] H. Guo, Y. Mao, and R. Zhang, “Mixup as locally linear out-of-manifold regularization,” in AAAI Conference on Artificial Intelligence, 2019.
  * [25] G. Hinton, O. Vinyals, and J. Dean, “Distilling the knowledge in a neural network,” ArXiv, vol. arXiv:1503.02531v1, 2015.
  * [26] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial networks,” in Advances in Neural Information Processing Systems, 2014.
  * [27] Pytorch. https://pytorch.org/vision/stable/models.html, 2019.
  * [28] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen, “Mobilenetv2: Inverted residuals and linear bottlenecks,” in Computer Vision and Pattern Recognition, 2018.
  * [29] A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, V. Vasudevan, Q. V. Le, and H. Adam, “Searching for mobilenetv3,” in International Conference on Computer Vision, 2019.
  * [30] R. Wightman, “Pytorch image models – results-imagenet.csv.” https://github.com/rwightman/pytorch-image-models/blob/master/results/results-imagenet.csv, 2019\.
  * [31] R. Wightman, “Pytorch image models.” https://github.com/rwightman/pytorch-image-models, 2019.


We introduce a new approach for circuit anonymous communication based on Lizama's non-invertible Key Exchange Protocol (ni-KEP) which has been conceived to work in the quantum era. Lizama's protocol has the smallest key size when compared to main post-quantum schemes thus it becomes a promising alternative for the quantum era. Circuit-based communication can be scaled to support the Hidden Service Protocol (HSP) as well as cross-domain digital certificates that promise greater computing security, speed and efficiency.

The Onion Routing (TOR) protocol provides non-traceability to the users of data networks. Non-traceability also called anonymity, complements other well established data security services as confidentiality, integrity and authentication. On the one hand, non-traceability is imperative for activists and informants, but on the other it does not allow to identify the origin where cybernetic crimes could be committed.

Unfortunately, the security of TOR's underlying algorithms is based on computational problems whose security has been threatened by the imminent development of quantum computers since Peter Shor conceived a useful quantum algorithm to solve in polynomial time the integer factoring problem  running on a hypothetical quantum computer <cit.>. Worse still, the vast majority of the public key cryptography most used today, among which we can mention RSA, Diffie-Hellman (DH) and elliptic curve cryptography (ECC), will become useless in the near future because quantum computers will be able to break it [2]. Because of this, the National Institute of Standards and Technology (NIST) began in 2015 a process of evaluation of post-quantum algorithms to select the most suitable methods for cryptography in the quantum era. Currently, this process is in the third round of evaluation  [3, 4].

Cryptography in the quantum era can be separated into two main approaches: quantum and post-quantum cryptography. A detailed review of these fields is beyond the purpose of this article. However, let us simply describe that quantum cryptography is based on the principles of quantum physics which are used to establish a secret key between two previously authenticated remote parties <cit.>. In this scheme, the Heisenberg uncertainty principle guarantees that an attacker is unable to control quantum communication because it generates detectable noise. Recently, research results have been published demonstrating the ability to resist quantum attacks [6, 7, 8, 9].

Post-quantum cryptography encompasses those methods designed to be immune to the computational power of quantum computers [10, 4]. Several algorithms based on computational problems have been invented, the difficulty of which goes beyond the theoretical capacity of quantum computers. Among the most prominent techniques, we can include code-based cryptography [11, 12], lattice based key exchange [13, 14], supersingular elliptic curve isogeny [15], hash based cryptography [16, 17, 18, 19], zero knowledge [20, 21] and multivariate cryptography [22, 23].

As a final remark, we emphasize that the security of current technologies widely used on the internet as bitcoin and blockchain would be seriously threatened by quantum computers. TOR would be in the same disadvantage, whose basic anonymizer mechanism is supported on RSA, Diffie-Hellman and Elliptic Curve Cryptography. In this work, we extend a novel cryptographic approach introduced by Lizama in [24, 25] to achieve anonymous communication. In Appendix <ref> can be found a detailed description of this protocol. The rest of the paper is organized as follows: in Section 2 we describe the fundamentals of TOR and circuit operation, we briefly discuss the hidden service protocol. In Section 3 we explain some security  issues of Lizama's ni-KEP and how it can be used to achieve non-traceability circuit communication. Finally, in Section 4 we emphasize the main advantages of our approach in terms of the key size, scalability and interoperability.

§ NON-TRACEABLE CIRCUIT-BASED COMMUNICATION

Today, the Onion Routing Protocol (TOR) has become the most widespread technology for achieving non-traceability and anonymous web browsing  [26]. Non-traceability can be achieved thanks to a fundamental approach known as circuit-based communication between network nodes. In the following paragraphs we will discuss circuit-based communication and the hidden service protocol.

§.§ Circuit Initialization

Circuit-based communication between a user Alice that we will denote here as A and a web server that we represent as S, is built along at least three nodes of the network that we write as B, C, D: the input node is denoted here as B, the intermediate node is C and the exit node is represented as D.
The purpose of the circuit is to establish a secret key between Alice and each of the three nodes of the network hiding Alice's identity, at least from the intermediate node and the exit node. Alice establishes the secret key $k_{ab}$ with B, who in turn allows Alice to set $k_{ac}$ with C but as a result of this process input node B cannot know this key. Nodes B and C will allow Alice to derive $k_{ad}$ with D, but again they cannot know the secret key. Using these keys, Alice prepares an onion data packet (see Figure fig:Fig.3) so that input node B knows the request coming from Alice but ignores the final destination of the packet. Intermediate node C knows the identity of the input node and the next node in the packet path, but neither of them correspond to Alice or the final web server's identity. Finally, exit node D knows where is the web server but is unaware that Alice sent the request.

When using Diffie-Hellman (DH) as the key exchange algorithm and RSA as the authentication protocol, we see the circuit initialization in Figs. <ref> and <ref> where $g^{x_i} \ \text{mod} \ p$ represents the DH constructor sent by user $i$ and ${(g^{x_i} \ \text{mod} \ p)}^{e_j} \ \text{mod} n_j$ means that the constructor is encrypted with the public key $(e_j, n_j)$ that belongs to the target user $j$.

[0]ACreate B, $(g^{x_a}$ mod ${p)^{e_b}}$ mod $n_b$BR
[0]BCreated B, $g^{x_b}$ mod $p$, $H_{k_{ab}}$AL$k_{ab}$$k_{ab}$
[0]ARelay B, [Extend C,$ $ $(g^{y_a}$ mod ${p)^{e_c}}$ mod $n_c]_{k_{ab}}$BR
[0]BCreate C,$ $ $(g^{y_a}$ mod ${p)^{e_c}}$CR
[0]CCreated C, $g^{y_c}$ mod $p$, $H_{k_{ac}}$BL$k_{ac}$
[0]BRelay B, [Extended C, $g^{y_c}$ mod $p$, $H_{k_{ac}}]_{k_{ab}}$AL$k_{ac}$

Alice sends B a packet with a Create B tag which means she must execute the key exchange protocol with A. Then, B sends a Created B confirmation response to Alice. Now, Alice sends B a packet labeled Relay B containing the Extend C command encrypted with $k_{ab}$. B decrypts it and forwards it to C. When B receives the Created C packet from C, she sends it back to Alice as a Relay B packet encrypted with $k_{ab}$. The hash value of the key $H_{k}$ is appended to the packets for Alice to authenticate the source of the keys.

[0]BRelay C, [Extend D, $ $ $(g^{z_a}$ mod ${p)^{e_d}}$ mod $n_d]_{k_{ac}}$CR
[0]CCreate D,$ $ $(g^{z_a}$ mod ${p)^{e_d}}$ mod $n_d$DR
[0]DCreated D, $g^{z_d}$ mod $p$, $H_{k_{ad}}$CL$k_{ad}$
[0]CRelay C, [Extended D, $g^{z_d}$ mod $p$, $H_{k_{ad}}]_{k_{ac}}$BL

When B receives a packet labeled Relay B from Alice, it forwards it to C as a Relay C command. Then, C decrypts the message with $k_{ac} $ and forwards it to D as a Create D command. In the opposite direction, when C receives a packet from D labeled Created D, forwards it to B encrypted with $k_{ac} $ and labeled Relay C. Then B delivers it to Alice as a Relay B packet. Alice decrypts it with $k_{ac} $ and she gets $k_{ad}$, the secret key between Alice and output node D.

In Figure fig:Fig.3 the onion routing protocol on the created circuits is shown. The notation $[m]_k$ denotes that the message $m$ is encrypted with $k$. Therefore, in the final phase of the protocol, the output node D gets $m$ and delivers the request to the server. Then node D returns the server's response to C encrypted with $k_{ad}$ by running the onion routing protocol in reverse.

[0]A$[$ $[$ $[ m ]_{k_{ad}} ]_{k_{ac}}]_ {k_{ab}}$BR
[0]B$[$ $[ m ]_{k_{ad}}]_{k_{ac}}$CR
[0]C$[ m ]_{k_{ad}}$DR

Basic functionality of the circuit-based protocol. Node D retrieves $m$ and delivers it to the web server.

§.§ The Hidden Service Protocol

In the above discussion, circuit-based communication gives Alice (A) anonymity, however, it is still possible to identify the location of the Web Server (S) on the network. To avoid being traced, the server must run the Hidden Service Protocol (HSP) which is based on basic circuit function and the Onion Routing Protocol. HSP is represented in Figure fig:Fig.4 where the use of the circuits is observed. Instead of revealing its location on the network (that is, the IP address) by means of a directory service (DS), the web server publishes three nodes called Introduction Points (IP) that act as representatives of S and are connected to S through the three usual nodes of circuit-based communication. Alice randomly chooses an IP to inform S the chosen meeting point (RP) where the service will be achieved as shown in Figure fig:Fig.4. The Hidden Service Protocol demonstrates that the operation of the basic circuit makes it possible to keep the location of users and services on the network anonymous.

Simplified representation of the Hidden Service Protocol (HSP).

§ NON-TRACEABILITY WITH LIZAMA'S NON-INVERTIBLE KEY EXCHANGE PROTOCOL

A detailed description of Lizama's non-invertible Key Exchange Protocol (ni-KEP) can be found in [24, 25]. An overview of the protocol has been written in Appendix <ref>. In this section we detail a method to construct a circuit using Lizama's algorithm. A previous attempt was presented in [27], however the protocol is vulnerable to a MITM attack. First of all, let us introduce the mathematical notation we will use to describe the protocol and some important security issues.

§.§ Mathematical notation

We use the symbol $(P_i, Q_i)$ to represent the public key of the user $i$. Here, $P_i=p^{2x_i} k_i$ and $Q_i=q^{y_i} k_i$ where $(x_i, k_i)$ constitute the private key of the user $i$ provided that $x_i + y_i = \phi(n) +1$. User $j$ raises the public key of $i$ to the power given by his private key numbers. Then $j$ returns to $i$ the number $[k_{i,j}] \ k_i$ where $[k_{i,j}] = {p^{2x_i x_j}}q^{y_i y_j}$ and $k_i$ is a component of the private key of the user $i$, then he applies the inverse of $k_i$ in order to derive the shared secret key $k_{i,j}$. The same procedure is applied in the opposite direction, so the user $i$ sends to $j$ the number $[k_{i,j}] \ k_j$ to get the same secret number $k_{i,j}$ (see Table <ref>).

Mathematical representation of the public keys. All operations are performed module $n$ where $n=pqr$.

Short notation      Math operation

$(P_i, Q_i)$       $P_i=p^{2x_i} k_i$, $Q_i=q^{y_i} k_i$

${P_i}^{x_j} \cdot {Q_i}^{y_j}$       $\left({p^{2x_i} k_i}\right)^{x_{j}} \cdot \left({q^{y_i} k_i }\right)^{y_j}$

$[k_{i,j}]$ $k_i$       ${p^{2x_i x_j}}q^{y_i y_j} k_i$

§.§ Prefix attack

Lizama's ni-KEP works as a cryptosystem as described in Appendix <ref>. Let's just highlight some of the most important security properties:

1. Encryption in Lizama's algorithm is performed multiplying by the encryption key, so we denote that a message $m$ is encrypted with the key $k$ as $[w]_{k}$.

2. The Lizama cryptosystem is homomorphic under the multiplication operation. Therefore, if we consider the encryption $[w]_{k_a k_b}$ and assuming that we have the appropriate decryption key we can write:

$[w]_{k_a  k_b  {k_a}^{-1}} = [w]_{k_a}$

$[w]_{k_a k_b {k_a}^{-1}} = [w]_{k_b}$

where ${k_i}^{-1}$ represents the multiplicative inverse of $k_i$ in $\mathbb{Z}_{n}$ and $n=pqr$. Special attention must be paid to the messages exchanged through the public channel due to the rules of modular multiplication. When an eavesdropper captures let's say $[w]_{k_m}$ which is a prefix of another that we can write as $[w]_{k_m k_n}$, then she derives $k_n$ because Eve first calculates the inverse of the prefix, it is say ${[w]_{k_m}}^{-1}$ and then she factors it from the second number. However, in Lizama's algorithm, $k_n$ and $[w]_{k_m}$ are non-invertible integers in $\mathbb{Z}_{n}$. Despite this, the attacker could multiply them by ${2}^{-2}$ which changes the modulus from $4r$ to $r$, e.g. $p=q=2$ and $n=4r$. Suppose that Eve has captured from the public channel the integers $[4x]_{k_m}$ and $[4x]_{k_m k_n}$ both in the module $4r$, and then she divides each of them by $2^{-2}$, then Eve gets $k_n$ because $({[x]_{k_m}}) ^{-1} \cdot ({[x]_{k_m k_n}}) = k_n \ \text{mod} \ r$. To avoid a prefix attack, $k_n$ should be chosen to be greater than the prime integer $r$.

§.§ Non-traceability with Lizama's ni-KEP

Lizama's non-traceable protocol is depicted in Figures <ref> and <ref>. The general idea is that node $i$ can establish a secret key with node $j$ but preserving anonymity of node $i$ because $i$ uses the public constructor $(P_j, Q_j)$ to send $[k_{jw}]_{k_j}$ to $j$ along with the temporary public constructor $(P_w, Q_w)$. Let us describe the steps of the protocol:

1. Using B's public key $(P_b, Q_b)$, Alice computes $[k_{bx}]_{k_b}$ and sends it to B along the constructor $(P_x, Q_x)$ labeled as a Create command.

2. B derives $k_{bx}$ and he takes $(P_x, Q_x)$ to compute $[k_{bx}]_{k_x}$, then he sends it back to Alice inside a Created command who derives $k_{bx}$, the secret key between A and B.

3. Using C's public key $(P_c, Q_c)$, Alice computes and sends $[k_{cy}]_{k_c}$ to B along the constructor $(P_y, Q_y)$ inside a Relay command and encrypted with $k_{bx}$.

4. B decrypts the message with $k_{bx}$ so he gets $[k_{cy}]_{k_c}$, now he knows the identity of the next node and the public constructor $(P_y, Q_y)$, then B forwards them to C inside a Create C command.

5. C derives $k_{cy}$ and he responds with $[k_{cy}]_{k_y}$ to B inside a Created C command, then C encrypts the packet with $k_{bx}$ and forwards it to A as a Relay B command. Alice decrypts the command and she derives $k_{cy}$, the secret key between A and C.

6. Using D's public key $(P_d, Q_d)$, Alice computes $[k_{dz}]_{k_d}$ and sends them to B along the constructor $(P_z, Q_z)$ encrypted with $k_{cy}$ inside a Relay B command. B forwards the packet to C as a Relay C command, then C decrypts the message using $k_{cy}$.

7. C gets $[k_{dz}]_{k_d}$, the name of D and the public constructor $(P_z, Q_z)$, then he forwards them to D inside a Create D command.
8. D derives $k_{dz}$ and responds with $[k_{dz}]_{k_z}$ to C as a Created D command. Then C encrypts it with $k_{cy}$ and forwards it to B as a Relay C command, then C forwards it to A as a Relay B command.

9. Alice decrypts the packet and derives $k_{dz}$, the secret key between A and D.

In the protocol has not been appended the hash value of the shared key into the packet response. We incorporates this functionality in the reverse packets depicted in Fig. <ref>.

[0]ACreate B, $[k_{bx}]_{k_b}$,$ $ ${(P_x, Q_x)}$BR$k_{bx}$
[0]BCreated B, $[k_{bx}]_{k_x}$AL$k_{bx}$
[0]ARelay B, [Extend C, $[k_{cy}]_{k_c}$, ${(P_y, Q_y)}]_{k_{bx}}$BR
[0]BCreate C, $[k_{cy}]_{k_c}$, ${(P_y, Q_y)}$CR$k_{cy}$
[0]CCreated C, $[k_{cy}]_{k_y}$BR
[0]BRelay B, [Extended C, $[k_{cy}]_{k_y}]_{k_{bx}}$AL$k_{cy}$
[0]ARelay B, [Extend D, $[k_{dz}]_{k_d}$, ${(P_z, Q_z)}]_{k_{cy}}$BR
[0]BRelay C, [Extend D, $[k_{dz}]_{k_d}$, ${(P_z, Q_z)}]_{k_{cy}}$CR
[0]CRelay C, [Extended D, $[k_{dz}]_{k_z}]_{k_{cy}}$BL
[0]BRelay B, [Extended D, $[k_{dz}]_{k_z}]_{k_{cy}}$AL$k_{dz}$

Circuit initialization protocol using ni-KEP.

[0]BRelay C, [Extend D, $[k_{dz}]_{k_d}$, ${(P_z, Q_z)}]_{k_{cy}}$CR
[0]CCreate D,$ $ $[k_{dz}]_{k_d}$, ${(P_z, Q_z)}$DR$k_{dz}$
[0]DCreated D, $[k_{dz}]_{k_z}$CR
[0]CRelay C, [Extended D, $[k_{dz}]_{k_z}]_{k_{cy}}$BL

Circuit initialization (cont).

[0]BCreated B, $[k_{bx}]_{k_x}, H_{k_{bx}}$AL$k_{bx}$$k_{bx}$
[0]C$[$ Created C, $[k_{cy}]_{k_y}]_{k_{bx}},H_{k_{cy}}$AL$k_{cy}$$k_{cy}$
[0]D$[$ Created D, $[k_{dz}]_{k_z}]_{k_{cy}}, H_{k_{dz}}$AL$k_{dz}$$k_{dz}$

Hash integrity check. The packets in the opposite direction contain the Hash value of the shared key.

§ DISCUSSION

After the third round of evaluation has been completed, NIST has chosen seven algorithms and eight alternative methods. Four of them are public key encryption and key-establishment systems. Three algorithms correspond to digital signature. In the first category are CRYSTALS-KYBER, NTRU-HPS, SABER which are lattice-based, while Classic McEliece is a public key encryption system based on code theory. In relation to digital signature schemes, CRYSTALS-DILITHIUM and FALCON are lattice-based and Rainbow is a multivariate-based method. Since our approach falls into the first category, we found that public keys size in Lizama's protocol has the smallest size: 0.256 kilobytes when $n$ reaches 1024 bits (see Table <ref>).

Even more in [25] has been demonstrated that Lizama's protocol can be scaled to support digital certificates and interoperability across different certification domains. For the reasons discussed above, we consider our approach to be very promising to achieve user anonymity in data networks in the pre-quantum and quantum era, since the size of the keys is highly competitive leading to greater computing speed and efficiency.

A comparison of Lizama's protocol in relation to the National Institute of Standards and Technology (NIST) Round 3 finalist algorithms is shown in the categories of public key encryption and key-establishment methods [28].

Scheme     System     Public Key (KB)     Private Key (KB)     Signature (KB)

4*[l]Public Key/
KEM     LIZAMA'S KEP     $0.256-0.512$     $0.192-0.384$     –

Classic McEliece     $261,120 - 1,357,824$     $6,492 - 14,120$     –

CRYSTALS-KYBER     $1.632 - 3.168$      $0.8 - 1.568$     –

NTRU-HPS     $0.931 - 1.230$     $1.235 - 1.592$     –

SABER     $0.672-1.312$     $1.568-3.040$     –

Algorithms     CRYSTALS-DILITHIUM     $1.312 - 2.592$     –     $2.420 - 4.595$

FALCON     $0.897 - 1.793$     –     $0.666 - 0.280$

Rainbow     $157.8 - 1,885.4$     $101.2 - 1,375.7$      $0.066 - 0.212$

Lizama's ni-KEP     $\sim 0.2$

Lizama's certified     $\sim 0.3$

ECDSA     $\sim 0.1$

RSA     $\sim 0.5$

Code based     190

Lattice based     11

Multivariate     99

SS Isogenies     122

§ CONCLUSIONS

In this research we have discussed a new approach to circuit-based communication to achieve user anonymity through Lizama's non-invertible key exchange protocol. Since the circuit communication is the foundation of the onion routing protocol, it can be properly enhanced to anonymize web services as well.

The non-invertible key exchange algorithm has been conceived on the basis of perfect secrecy, thus the approach presented here can be properly used in the pre-quantum and quantum era since Lizama's protocol has the smallest key size when compared to main post-quantum schemes. Furthermore Lizama's protocol can be scaled to support digital certificates and interoperability across different certification domains.

§ LIZAMA'S KEY EXCHANGE PROTOCOL

Lizama's key exchange protocol was introduced in [24, 25] and is illustrated in Fig. <ref>. The public key of user $i$ ($a$ for Alice, $b$ for Bob) has two components $(P_i, Q_i)$ where $P_i = p^{2x_i} k_i \ \text{mod} \ n$ and $Q_i = q^{y_i} k_i \ \text{mod} \ n$. The value $x_i$ is chosen randomly while $y_i$ is computed according to the relation $y_i=\phi(n)-x_i+1$. The module $n$ is the product of three public integer primes, so that $n= p \cdot q \cdot r$ where $p$ and $q$ are small prime numbers and $r$ is a big integer prime. To achieve indistinguishability $p$ and $q$ are suggested to be 2, since 2 is a primitive root module $r$ [24]. The exponent is chosen to be $2x_i$ instead of $x_i$ to avoid a multiplication attack. The $x_i$ value constitutes along $k_i$ the private key of user $i$ where $k_i$ is an invertible integer in the ring $\mathbb{Z}_{n}$. Users exchange their public keys $(P_i, Q_i)$ as well as the integer module $n$. The steps of the protocols are summarized as follows:

1. Once public keys have been exchanged, the users perform two operations over the numbers received: exponentiation and multiplication as indicated in Tab. <ref>.

Exponentiation and multiplication are performed by users after their public keys have been exchanged.

User 		    Operation      Result
2*Alice     	    2*$ \ \ {\left( {p}^{2x_b} \cdot k_b \ \text{mod} \ n \right)}^{x_a} \cdot {\left(q^{y_b} \cdot k_b\ \text{mod} \ n \right)}^{y_a} = $      2* $p^{2 x_b x_a} q^{y_b y_a} \cdot k_b \ \text{mod} \ n $

2*Bob   	    2*$ \ \ {\left( p^{2x_a} \cdot k_a \ \text{mod} \ n \right)}^{x_b} \cdot {\left(q^{y_a} \cdot k_a\ \text{mod} \ n \right)}^{y_b} = $      2* $p^{2 x_a x_b} q^{y_a y_b} \cdot k_a \ \text{mod} \ n $

2. To derive the results in the right column of Table <ref>, Euler's theorem is applied in $\mathbb{Z}_{n}$. The theorem is written in Eq.<ref> where $n=pqr$ and $\phi(n)= (p-1)(q-1)(r-1)$. Here, $k$ and $n$ are relative prime each other, so $k$ is an invertible integer in $\mathbb{Z}_{n}$. Thus, according to Eq.<ref> we have ${k}^{\phi(n)+1} = {k}^{\phi(n)} \cdot {k}^{1} = k$.

\begin{equation}
\label{eq:Eq.1}
{k}^{\phi(n)} \equiv 1 \ \text{mod} \ n
\end{equation}

3. Users exchange the resulting value $p^{2 x_a x_b} q^{y_a y_b} k_i \ \text{mod} \ n$, which is multiplied by the corresponding inverse ${k_i}^{-1}$ at each side to derive the secret shared key $p^{2x_a x_b} q^{y_a y_b} \allowbreak \ \text{mod} \ n$ as depicted in Fig. <ref>.

[0]AAL${\left(p^{2x_b} k_b \right)}^{x_a}\cdot{\left(q^{y_b} k_b\right)}^{y_a}$
[0]AAR$p^{2 x_a x_b} q^{y_a y_b} {k_b}^{\phi(n)+1}$
[0]AAR$p^{2 x_a x_b} q^{y_a y_b} {k_b}$
[0]A$p^{2 x_a x_b} q^{y_a y_b} k_b$BR$ $ $ $ $p^{2 x_a x_b} q^{y_a y_b} k_b {k_b}^{-1}$
[0]BBL$ $ $ $ $k_s = p^{2 x_a x_b} q^{y_a y_b}$
[0]BBL$ $ $ $ ${\left(p^{2x_a} k_a\right)}^{x_b}\cdot{\left(q^{y_a} k_a\right)}^{y_b}$
[0]BBL$ $ $ $ $p^{2x_a x_b} q^{y_a y_b} {k_a}^{\phi(n)+1}$
[0]B$ $ $ $ $p^{2x_a x_b} q^{y_a y_b} k_a$AL$p^{2x_a x_b} q^{y_a y_b} {k_a}$$p^{2x_a x_b} q^{y_a y_b} k_a {k_a}^{-1}$
[0]AAR$k_s = p^{2x_a x_b} q^{y_a y_b}$

Lizama's non-invertible KEP [24]. All operations are modulo $n$ where $n=pqr$. According to Euler's theorem ${k}^{\phi(n)+1} \ \text{mod} \ n = k$ because $k$ is an invertible integer in $\mathbb{Z}_{n}$.

§.§ Encryption-system

In Fig. <ref>, the secret shared key $k_s$ is a non-invertible number in $\mathbb{Z}_n$, thus a convenient method to achieve a cipher-system and secret communication is to divide $k_s = p^{2x_a x_b} q^{y_a y_b} \ \text{mod} \ n$ by $pq$. Now, Alice and Bob can compute its multiplicative inverse ${k_r}^{-1}$. The enciphered message is obtained as $c = m \cdot k_r \ \text{mod} \ r$ and the original plaintext is recovered as $m = c \cdot {k_r}^{-1} \ \text{mod} \ r$ because $m = m \cdot k_r {k_r}^{-1} \ \text{mod} \ r$. To send a message encoded as an integer in $\mathbb{Z}_{r}$, the number $m$ must be less than $r$.

Encryption/decryption mathematical relations.

Message      Mathematical relation

Encryption       $c = m \cdot k_{r} \ \ \text{mod} \ \ r$

Decryption       $m = c \cdot {k_{r}}^{-1} \ \ \text{mod} \ \ r$

[1]
Peter W Shor.
Algorithms for quantum computation: Discrete logarithms and
In Foundations of Computer Science, 1994 Proceedings., 35th
Annual Symposium on, pages 124–134. IEEE, 1994.

[2]
Marco A Barreno.
The future of cryptography under quantum computers.

[3]
Information Technology Laboratory.
PQC Standardization Process: Third Round Candidate Announcement.
[Online; accessed 08-October-2020].

[4]
Lily Chen, Lily Chen, Stephen Jordan, Yi-Kai Liu, Dustin Moody, Rene Peralta,
Ray Perlner, and Daniel Smith-Tone.
Report on post-quantum cryptography, volume 12.
US Department of Commerce, National Institute of Standards and
Technology, 2016.

[5]
H Bennett Ch and G Brassard.
Quantum cryptography: public key distribution and coin tossing int.
In Conf. on Computers, Systems and Signal Processing (Bangalore,
India, Dec. 1984), pages 175–9, 1984.

[6]
Luis A Lizama-Pérez, J Mauricio López, Eduardo De Carlos-López, and
Salvador E Venegas-Andraca.
Quantum flows for secret key distribution in the presence of the
photon number splitting attack.
Entropy, 16(6):3121–3135, 2014.

[7]
Luis Adrian Lizama-Pérez, José Mauricio López, and Eduardo
De Carlos López.
Quantum key distribution in the presence of the intercept-resend with
faked states attack.
Entropy, 19(1):4, 2016.

[8]
López JM Lizama-Perez LA.
Quantum key distillation using binary frames.
Symmetry, 12(6):1053, 2020.

[9]
Luis Adrián Lizama-Pérez, Emmanuel H Samperio, et al.
Beyond the limits of shannon’s information in quantum key
Entropy, 23(2):229, 2021.

[10]
Daniel J Bernstein and Tanja Lange.
Post-quantum cryptography.
Nature, 549(7671):188–194, 2017.

[11]
Robert J McEliece.
A public-key cryptosystem based on algebraic.
Coding Thv, 4244:114–116, 1978.

[12]
David Ott, Christopher Peikert, et al.
Identifying research challenges in post quantum cryptography
migration and cryptographic agility.
arXiv preprint arXiv:1909.07353, 2019.

[13]
ShanBiao Wang, Yan Zhu, Di Ma, and RongQuan Feng.
Lattice-based key exchange on small integer solution problem.
Science China Information Sciences, 57(11):1–12, 2014.

[14]
Criptoanálisis del protocolo de intercambio de claves basado en celosía de
wang et al.
Perspectives in Science.

[15]
David Jao and Luca De Feo.
Towards quantum-resistant cryptosystems from supersingular elliptic
curve isogenies.
In International Workshop on Post-Quantum Cryptography, pages
19–34. Springer, 2011.

[16]
Leslie Lamport.
Constructing digital signatures from a one-way function.
Technical report, Technical Report CSL-98, SRI International Palo
Alto, 1979.

[17]
Ralph C Merkle.
Method of providing digital signatures, January 5 1982.
US Patent 4,309,569.

[18]
Luis A Lizama-Perez.
Digital signatures over hash-entangled chains.
SN Applied Sciences, 1(12):1568, 2019.

[19]
Luis Adrián Lizama-Pérez, Leonardo Javier Montiel-Arrieta, Flor Seleyda
Hernández-Mendoza, Luis Adrián Lizama-Servín, and
Simancas-Acevedo Eric.
Public hash signature for mobile network devices.
Ingeniería, Investigación y Tecnología,
XX(2):1–10, 2019.

[20]
Shafi Goldwasser, Silvio Micali, and Charles Rackoff.
The knowledge complexity of interactive proof systems.
SIAM Journal on computing, 18(1):186–208, 1989.

[21]
Eli Ben-Sasson, Iddo Bentov, Yinon Horesh, and Michael Riabzev.
Scalable, transparent, and post-quantum secure computational
IACR Cryptology ePrint Archive, 2018:46, 2018.

[22]
Tsutomu Matsumoto and Hideki Imai.
Public quadratic polynomial-tuples for efficient
signature-verification and message-encryption.
In Workshop on the Theory and Application of of Cryptographic
Techniques, pages 419–453. Springer, 1988.

[23]
Jintai Ding and Dieter Schmidt.
Rainbow, a new multivariable polynomial signature scheme.
In International Conference on Applied Cryptography and Network
Security, pages 164–175. Springer, 2005.

[24]
Luis Adrian Lizama-Perez.
Non-invertible key exchange protocol.
SN Applied Sciences, 2:1083, 2020.

[25]
Luis Adrián Lizama-Perez, J Mauricio López R, et al.
Non-invertible public key certificates.
Entropy, 23(2):226, 2021.

[26]
Paul Syverson, D Goldschlag, and M Reed.
Onion routing for anonymous and private internet connections.
Communications of the ACM, 42(2):5, 1999.

[27]
I.N. Lucio-Morales and L.A. Lizama-Pérez.
Sistema de autenticación anónima por medio de criptografía no
In Simposio Iberoamericano Multidisciplinario de Ciencias e
Ingenierías, volume 8, pages 25–34. Universidad Politécnica de Pachuca,

[28]
Edoardo Persichetti.
NIST Round 3 finalists.
<https://pqc-wiki.fau.edu/w/Special:DatabaseHome>, 2020.
[Online; accessed August 27, 2024].