(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: ARTORG, University of Bern, Switzerland
11email<EMAIL_ADDRESS>

# Tune without Validation: Searching for Learning Rate and Weight Decay on
Training Sets

Lorenzo Brigato  Stavroula Mougiakakou

###### Abstract

We introduce Tune without Validation (Twin), a pipeline for tuning learning
rate and weight decay without validation sets. We leverage a recent
theoretical framework concerning learning phases in hypothesis space to devise
a heuristic that predicts what hyper-parameter (HP) combinations yield better
generalization. Twin performs a grid search of trials according to an
early-/non-early-stopping scheduler and then segments the region that provides
the best results in terms of training loss. Among these trials, the weight
norm strongly correlates with predicting generalization. To assess the
effectiveness of Twin, we run extensive experiments on 20 image classification
datasets and train several families of deep networks, including convolutional,
transformer, and feed-forward models. We demonstrate proper HP selection when
training from scratch and fine-tuning, emphasizing small-sample scenarios.

##  1 Introduction

Figure 1: Overview. While traditional pipelines need a validation set to tune
learning rate and weight decay, Twin performs the search directly on the
training set, simplifying the process or saving additional data-collection
costs.

Like most machine learning models, deep networks are configured by a set of
hyper-parameters (HPs) whose values must be carefully chosen and which often
considerably impact the final outcome [26, 17, 60]. Setting up wrong
configurations translates into bad performance, particularly in the most
difficult optimization scenarios, e.g., large models that overfit on small
datasets [35, 4, 5].

Traditionally, HP search is performed in two ways, as exemplified in Fig. 1
(top). Although more comprehensive methodologies, such as multi-fold or multi-
round cross-validation [44], exist, they are scarcely employed in training
deep networks due to their significant computational overhead. When no
validation set is available, the training set is split into two unbalanced
subsets to perform the HP search. Then, the “optimal” HP configuration
initializes the final training on the original collection. In contrast, if the
validation set is accessible, ML professionals can expedite the HP search and
simplify the model selection process since the two-step training pipeline is
avoided. However, this comes at the expense of collecting an additional 10-30%
of samples. Use cases where acquiring additional data is either expensive or
logistically unfeasible, such as medical imaging [50] or federated learning
[37], challenge the traditional HP selection pipeline. HP optimization on
small validation sets may have inherent noise [35, 7].

Motivated by these challenges, we introduce Tune without Validation (Twin), an
innovative HP selection approach inspired by a theoretical framework which
aims at explaining representation learning across HPs from phase diagrams
[33]. Twin obviates the need for validation sets when tuning optimizer
parameters. In particular, Twin enables practitioners to directly select the
learning rate (LR) and weight decay (WD) from the training set, as sketched in
Fig. 1 (bottom). Twin performs a grid search over an hypothesis space using an
early-/non-early-stopping scheduler and successfully predicts generalizing HP
configurations by monitoring only the training loss, as a proxy for task
performance, and the weight norm, to measure regularization strength.

We perform an extensive empirical analysis to demonstrate Twin’s practical
versatility by training 4,000+ deep models involving 20 datasets and several
architectures. On a suite of 34 different dataset-architecture configurations
with networks trained from scratch and without early stopping, Twin scores an
MAE of 1.3% against an Oracle pipeline - the ideal unrealistic scenario - that
directly selects HPs from testing sets. We summarize the contributions of our
paper:

  * •

We introduce Twin, a simple but effective HP selection pipeline which
optimizes LR and WD, directly from training sets (Sec. 3).

  * •

We showcase the effectiveness of Twin across a wide spectrum of experimental
scenarios (Sec. 4.1, Sec. 4.2, Sec. 4.3, Sec. 4.4), encompassing datasets from
different domains (e.g., natural, medical imagery) and scales (hundreds to
thousands of samples) as well as models with various architecture (e.g.,
ResNet, MLP, CVT) and size (from $\sim$0.2M to $\sim$90M params.).

  * •

We ablate on the different components and parameters of Twin to provide
additional insights regarding the working mechanisms of our pipeline (Sec.
4.5).

##  2 Related Work

#### Image classification.

Since the introduction of AlexNet [26], image classification has witnessed
remarkable advances, propelled by the development of novel neural
architectures (e.g., ResNet [16], Vision Transformer (ViT) [11]), and large
datasets [42]. Large-scale pre-training has favored the application of
transfer learning to tackle small-sample scenarios [54, 24]. More recent work
provides insights regarding the training of deep models from scratch on
limited datasets [56, 1, 6, 28, 5, 8]. Motivated by the medium-to-small size
of datasets explored in this work, we mostly train convolutional networks
(ConvNets) but also experiment with ViTs and feed-forward networks.

#### Hyper-parameter tuning.

There is a vast literature tackling the problem of HP tuning for deep networks
[60], including works on implicit differentiation [35], data augmentation [10,
31], neural-architecture search [12], invariance learning [55, 3, 21], and
general-purpose schedulers [30, 29]. Concerning optimization-related HPs, the
seminal work of Goyal et al. [14] popularized the linear scaling rule for
learning rate and batch size. Yang et al., [58] proposed a parameterization to
zero-shot transfer LRs to larger model sizes [58]. Recent work studied HP
selection as data scales by exploiting SGD symmetries [61, 62]. However, only
a few studies explore HP optimization without employing validation sets,
mainly focusing on learning invariances. When employing Bayesian inference,
methods either fail to scale to relatively simple tasks (e.g., CIFAR-10) [43]
or larger network sizes (e.g., ResNet-14) [21]. Benton et al. [3] make strong
assumptions about knowing what HPs help learning invariances in advance. A
recent method improves scalability issues but still introduces complexity by
needing data and model partitioning and an additional backward-forward pass
[38]. Unlike such methods, Twin focuses on LR and WD, easily scales to
increased model and data sizes, and simplifies the HP optimization pipeline.

##  3 Tune without Validation

###  3.1 Preliminaries

#### Problem settings.

Image classification tasks present a training set
$\mathcal{D}_{train}=\\{x_{i},y_{i}\\}$ and a testing set
$\mathcal{D}_{test}=\\{x_{i},y_{i}\\}$ sampled from a distribution $P(X,Y)$.
The learners, in our case, deep neural networks $f_{\theta}$ parameterized by
parameters $\theta$, are trained via SGD optimization to minimize the cross-
entropy loss over the training set,
$\min_{\theta}(\mathcal{L}_{\theta}=\mathcal{L}(f_{\theta},\mathcal{D}_{train}))$.
A popular regularization technique to avoid over-fitting and improve
generalization is $L_{2}$ regularization, i.e.,
$\min_{\theta}\mathcal{\hat{L}}_{\theta}$ with
$\mathcal{\hat{L}}_{\theta}=\mathcal{L}(f_{\theta},\mathcal{D}_{train})+\lambda\cdot\frac{{||\theta||}^{2}}{2}$
that features a penalty over the norm of the weights controlled by the
parameter $\lambda$, widely known as WD. For modern scale-invariant
architectures111Up to 90% of the parameters in ResNets tend to be scale-
invariant, primarily due to the influence of Batch Normalization [20]., WD
does not reduce the complexity of the model but rather increases the effective
learning rate by reducing the weight norm [48, 64], and hence can indirectly
exert a regularizing effect by means of larger gradient noise [39, 22, 32].
When optimized with momentum SGD, the parameters follow the update rule
$\theta_{t+1}=\theta_{t}-\mu
v_{t}+\alpha_{t}(\nabla\mathcal{L}_{\theta}+\lambda\theta_{t})$ where
$\alpha_{t}$ being the LR adjusted at each iteration according to a LR
schedule and $\mu$ the momentum coefficient.

#### Cross-validation.

To estimate the HPs controlling the model, specifically the regularization
parameter $\lambda$ and the learning speed $\alpha$, we ideally need a
surrogate of the test set, which identifies the right model complexity that
minimizes the test error. As anticipated previously, the simplest (and most
popular) operation is to split the training set into two sub-datasets to
deliver a smaller training set and a validation set
$\mathcal{D}_{val}=\\{x_{i},y_{i}\\}$, as shown in Fig. 1 (top). The
cardinality of the sub-sampled training and validation sets are respectively
$|\hat{D}_{train}|=n-m$ and $|\hat{D}_{val}|=m$. Now, we define the expected
value and variance of the prediction error over unseen points respectively as
$\delta=\mathrm{E}[\mathcal{L}(f_{\theta}(x),y]$ and
$\sigma^{2}=\mathrm{Var}[\mathcal{L}(f_{\theta}(x),y]$. We also refer to the
loss computed over the validation set as
$\mathcal{L}(f_{\theta},\mathcal{D}_{val})=\sum^{m}_{i=1}\mathcal{L}(f_{\theta}(x_{i}),y_{i})$.
By applying the linearity property of the expectation and the previously
defined relationships, we derive that
$\mathrm{E}[\mathcal{L}(f_{\theta},\mathcal{D}_{val})]=\delta$,
$\mathrm{Var}[\mathcal{L}(f_{\theta},\mathcal{D}_{val})]=\frac{\sigma^{2}}{m}$,
and $\mathrm{Std}=\mathcal{O}(\frac{1}{\sqrt{m}})$. Finally, we can express
that the expected error on the validation set is proportional to the error
scored over an unseen test sample plus a term depending on the number of
samples in the validation split, or more formally
$\mathrm{E}[\mathcal{L}(f_{\theta},\mathcal{D}_{val})]=\delta\pm\mathcal{O}(\frac{1}{\sqrt{m}})$.
The straightforward consequence is that a small validation set does not
provide a good estimate of the test error; hence, it is unreliable for HP
selection.

#### Motivation.

In classification problems where the independent and identically distributed
(IID) assumption holds, the search for HPs is less challenging since the
overlapping among training and testing sets makes the prediction of
generalization relatively easy. In other words, if $\mathcal{D}_{train}$, and
$\mathcal{D}_{test}$ are sampled from the same distribution, the expected
prediction error $\delta$ on unseen test points is going to be proportional to
the prediction error over the training set, i.e.,
$\delta\approx\mathcal{L}(f_{\theta},\mathcal{D}_{train})$. In section Sec.
4.1, we further discuss this claim and indeed empirically validate it in Fig.
3. On the contrary, distribution shifts, possibly caused by a plethora of
factors such as corruptions [19] or lack of data [52], induce the so-called
out-of-distribution (OOD) learning problems. In this work, we focus on OOD
scenarios caused by sample scarcity. As demonstrated in the previous
paragraph, the number of samples available in the validation set strongly
impacts the expected prediction error and can bias the search for proper HPs.
The significance of this observation is the main motivation guiding our work,
which aims to eliminate the dependency on validation sets and make the search
for LR and WD more robust. Our paper aims to derive a robust recipe for
practically predicting HP generalization in IID and OOD settings.

###  3.2 Working Principle

#### Phases of learning.

We rely on a recently introduced theoretical framework that explains
representation learning and the recently observed behavior of grokking [41], a
phenomenon where models generalize long after overfitting their training set.
Precisely, Liu et al. [33] observe, via a macroscopic analysis of phase
diagrams describing learning performance across HPs, four learning phases: i)
comprehension, ii) grokking, iii) memorization, and iv) confusion. The last
two phases correspond to the well-known overfitting and underfitting
scenarios, while comprehension to standard representation learning and
grokking to delayed generalization.222Note that grokking is mostly observed in
algorithmic tasks, but can also be induced for image classification problems
[33, 34] We leverage the following observations to design Twin [33]:

  * •

Observation 1 (O1): The comprehension, grokking, and memorization phases all
share the learner to reach a low enough training error.

  * •

Observation 2 (O2): Out of the three phases of O1, only the comprehension and
grokking configurations manage to reach a low-enough testing error.

  * •

Observation 3 (O3): Representation learning (comprehension and grokking)
occurs only in a “Goldilocks zone” between memorization and confusion.

#### Intuition.

Our goal is to predict generalizing configurations of LR and WD from a defined
hypothesis space. Configurations that lie in the comprehension or grokking
areas provide the best generalization performance. However, the definitions to
classify learning phases, as proposed in [33], leverage validation error, a
metric that we are willing to avoid. According to O1, low training error
excludes the phase with confusion (underfitting). Monitoring the training loss
can hence identify configurations that underfit the training objectives.
However, O2 predicts that only comprehension and grokking reach low testing
errors. Reasonably, the training loss alone can not discern overfitting from
generalizing solutions. To identify memorization within the hypothesis space,
we leverage O3 and recognize that the norm of the network’s parameters
provides a suitable metric for assessing the transition from confusion to
memorization, passing through comprehension. High WD strongly penalizes the
parameter’s norm, leading to confusion, while low WD causes memorization since
the model is poorly regularized. In summary, we employ the training loss as a
proxy to identify underfitting configurations. Out of the remaining models, we
expect the generalizing configurations to be the ones with the lowest norm of
the parameters. Fig. 4 strongly supports our hypothesis and shows the
predictive power of the parameter norm related to model generalization. We
next describe Twin’s pipeline in practice.

Figure 2: Twin overview. Twin employs a gradient-based optimizer and a trial
scheduler to perform a grid search across the LR-WD space. Twin logs train-
loss and parameter-norm matrices to identify the network with the lowest norm
within the fitting region. The parameter norm within this region is a good
predictor of generalization (right plot). In this figure, we show as an
example a WRN-16-10 trained on ciFAIR-10.

###  3.3 Pipeline

We show the overview of Twin in Fig. 2. Twin performs a grid search over the
LR-WD space and optimizes deep networks via gradient-based methods. Trials are
scheduled according to a first-input-first-output scheduler (no early-
stopping) or successive-halving-based scheduler that reduces the computational
burden. Then, a simple cascade of segmentation and filtering modules
identifies the training loss region where generalization or overfitting
happens. Within this area, the network with the smallest parameter norm is
selected.

#### Grid search and trial scheduler.

Twin runs $N_{\alpha}\cdot N_{\lambda}$ trials sampled by default from a grid
of equally spaced points in logarithmic space for both LR and WD. However,
Twin also supports different grid densities as ablated in Sec. 4.5.

We experiment with two types of trial schedulers: 1) a default first-input-
first-output (FIFO) without any form of automated trial stopping and 2)
HyperBand (HB) [30]. Other trial schedulers (e.g., median stopping rule [13])
and search strategies (e.g., random search) are left for future exploration.
Default HP search over validation sets usually terminates a few HP
configurations, out of which the optimal is picked. Conversely, as anticipated
in Sec. 3.2, Twin needs to detect the region in the LR-WD space with low
training loss. Trial stopping too aggressively would make the loss landscape
noisy and challenging to segment. To this end, we adapted the HB scheduler to
our needs. More precisely, we run the original HB algorithm until only a
certain percentage $X$ of trials is still alive and continue such trials
without further stopping. In other words, we systematically pass the most
promising trials to the bottom rung. The asynchronous version of HB,
Asynchronous Successive Halving Algorithm (ASHA) [29], employs a similar
mechanism to exploit parallelism. We opt for this solution rather than
increasing the minimum resources per trial to save compute resources. We refer
to our adapted HB with HBX% when $X\%$ of the trials are terminated. In Sec.
4.5, we ablate on the impact of different levels of early stopping, i.e.,
$X\%\in\\{25\%,12\%\\}$, and show that Twin supports aggressive stopping well.

#### Region segmentation and selection.

The grid search guided by the trial scheduler yields two log matrices needed
to find the optimal configuration: training losses and parameter norms. We
refer to the first matrix as $\Psi$ and the second as $\Theta$. Note that the
matrix elements are logged at different epochs when early stopping is
employed. In the case of the training losses, we average the values of the
last five training epochs to have a more reliable estimate. To identify the
area of the LR-WD space where the best fitting happens, we treat the loss
matrix as an image and apply the popular Quickshift image segmentation
algorithm [51]. Quickshift proved to be more robust than hand-crafted
thresholds that fail to generalize across diverse datasets and model
configurations. Quickshift needs two input parameters, namely kernel_size and
max_dist, which control the segmentation granularity. We set both parameters
to the squared root of the largest grid side to have a more fine-grained
segmentation. We ablate on this choice in Sec. 4.5. Practically, we first
filter possible outliers, e.g., exorbitant loss values or NaN, stemming from
instabilities during training caused by extreme parameter configurations. We
consider outliers all points having a z-score larger than two. We then scale
the values of $\Psi$ to 0-1 range by minmax normalization, make the lowest
loss the highest value ($|1-\texttt{minmax}(\Psi)|$), and run Quickshift
segmentation. We finally take the mean for each predicted region and apply the
argmax to retrieve the best cluster index.

#### Output configuration.

The selected grid region is converted into a binary mask and applied to the
logged $\Theta$ matrix. Out of this sub-region, the argmin function returns
the final output configuration with the lowest norm.

Figure 3: Overview of quantitative results. Twin scores an overall 1.3% MAE
against the Oracle pipeline across 34 different dataset-model configurations
when using a FIFO scheduler. Twin closely matches the Oracle in IID and OOD
scenarios, while SelTS fails to correctly predict HPs that generalize in OOD
cases.

Lossflowerscubcifair10clammeurosatisic2018retinamnistcifar100organsmnistcifar10pathmnistR.
Segm.R. Sel.$\displaystyle||\theta||$Test loss

Figure 4: Qualitative results. Visualization of the various steps of Twin in
the LR-WD space (first four rows) and the relationship between the selected
parameter norms and test loss (bottom row). The dashed green line represents
the lowest achievable test loss.

##  4 Experiments

###  4.1 Overview

#### Experimented domains.

In selecting datasets for experimentation, we aimed to cover diverse domains
to thoroughly assess Twin’s capabilities. Firstly, our evaluation encompasses
small datasets, a scenario particularly suitable for Twin, given that
traditional pipelines struggle with HP selection due to the limited dimensions
of validation sets, as explained in the paragraph dedicated to cross-
validation of Sec. 3.1. Additionally, we explore Twin’s potential in the
medical field, where its ability to mitigate the need for validation sets is
particularly valuable, considering the complexities and regulations inherent
in healthcare settings. Finally, we examine Twin’s versatility in dealing with
natural images, arguably the most widely studied domain within the computer
vision community.

#### Baselines.

To assess the relative effectiveness of Twin, we introduce three different
baselines. The Selection from Training Set (SelTS) baseline selects the HP
configuration, which scores the lowest loss on the training set. The Selection
from Validation Set (SelVS) is the more traditional reference point, where HP
optimization is conducted exclusively on the validation set. The validation
set can either be subsampled from the training set, as is going to be the case
for small and natural image datasets, or collected externally, like the case
of medical data. These two cases indeed correspond to the upper two schemes of
Fig. 1. SelVS can also be performed with different trial schedulers, e.g.,
with early stopping. Lastly, the Oracle represents the ideal but unrealistic
scenario of selecting HPs directly from the test set. The Oracle always runs a
FIFO scheduler. When Twin performs at its absolute best, it yields an MAE of
0% vs the Oracle. All baselines select the HPs according to the relevant last-
epoch metric with FIFO schedulers and the average of the last 5 with early
stopping.

Method | Model | CUB | ISIC 2018 | EuroSAT | CLaMM | Average
---|---|---|---|---|---|---
Oracle | EN-B0 | 67.2 | 66.8 | 91.0 | 65.3 | 72.6
SelTS | EN-B0 | 62.4 | 64.0 | 79.8 | 56.8 | 65.8
SelVS | EN-B0 | 67.0 | 65.1 | 86.6 | 58.0 | 69.2
Twin | EN-B0 | 66.2 | 66.4 | 89.8 | 62.6 | 71.3
Oracle | RN50 | 72.0 | 69.4 | 90.2 | 74.6 | 76.6
SelTS | RN50 | 57.3 | 65.4 | 83.8 | 68.2 | 68.7
SelVS† | RN50 | 70.8 | 64.5 | 90.6 | 70.2 | 74.0
Twin | RN50 | 70.1 | 68.8 | 89.2 | 73.8 | 75.5
Oracle | RNX101 | 73.0 | 66.7 | 88.6 | 75.2 | 75.9
SelTS | RNX101 | 54.2 | 59.8 | 86.8 | 62.5 | 65.8
SelVS | RNX101 | 72.1 | 62.4 | 90.0 | 70.1 | 73.7
Twin | RNX101 | 73.0 | 65.8 | 87.6 | 75.2 | 75.4

Table 1: Small datasets. The evaluation metric is the balanced test accuracy
(%) [5]. We allocate 100 and 36 trials for EN-B0/RN50 and RNX101,
respectively. Twin and SelVS respectively employ the HB25% and ASHA
schedulers. †Values are taken from [5].

#### Quantitative results.

To provide an overview of the quantitative results, we compare Twin against
our lower (SelTS) and upper (Oracle) bounds in Fig. 3. In particular, we show
the performance per dataset with FIFO schedulers averaged across all
architectures. The order along the x-axis represents the number of images per
class per dataset, which increases from left to right. As anticipated in Sec.
3.1, the training loss alone as an HP-selection metric (SelTS) falls short as
the alignment between training and testing distributions decreases, i.e., when
the IID assumption does not hold because the training set does not fully
represent the broader testing set [49, 52]. On the other hand, we find nearly
optimal LR-WD configurations across all dataset scales by considering the
regularization strength and the task-learning performance, as done with Twin
(Sec. 3.2).

#### Qualitative results.

In Fig. 4, we present qualitative results and empirical evidence supporting
Twin’s ability to predict generalizing HP configurations. We demonstrate
Twin’s consistency across 11 cases involving different models (ConvNets, MLPs,
and transformers), optimizers (SGD, Adam), and grid densities (100 and 49
trials). In Fig. 4, the first row displays the training loss at the last
epoch. Region segmentation and selection steps using Quickshift are shown in
the second and third rows, respectively. The fourth row illustrates the mask
application to the parameter norm matrix, while the last row depicts the
relationship between the parameter norm of the selected sub-region and the
test loss. The dashed green line represents the lowest loss achieved by the
Oracle baseline. Indeed, despite the best-fitting region showcasing various
patterns and positions in the LR-WD space depending on the dataset-
architecture configuration, Twin can identify it robustly. Furthermore, it is
visible that a strong (almost linear) relationship exists between the
parameter norms extracted from the identified region and the test loss.

###  4.2 Small Datasets

#### Datasets.

We select the benchmark introduced in [5], which contains five different
datasets spanning various domains and data types. In particular, the benchmark
contains sub-sampled versions of ciFAIR-10 [2], EuroSAT [18], CLaMM [45], all
with 50 samples per class, and ISIC 2018, with 80 samples per class [9]. Also,
the widely known CUB dataset with 30 images per category is included [53]. The
spanned image domains of this benchmark hence include RGB natural images
(ciFAIR-10, CUB), multi-spectral satellite data (EuroSAT), RGB skin medical
imagery (ISIC 2018), and grayscale hand-written documents (CLaMM). For
EuroSAT, an RGB version is also available. Finally, we include the Oxford
Flowers dataset in our setup, which comprises 102 categories with 20 images
[40].

#### Implementation details.

Along with the popular ResNet-50 (RN50), which was originally evaluated on the
benchmark [5], we also employ EfficientNet-B0 (EN-B0) [46] and ResNeXt-101 (32
$\times$ 8d) (RNX101) [57] to cover three classes of model scales,
respectively tiny, small, and base [47], with 5.3M, 25.6M, and 88.8M
parameters. For the low-resolution images of ciFAIR-10, we employ a Wide
ResNet 16-10 (WRN-16-10). We refer the reader to the Appendix for all the
details regarding training-related parameters. We perform squared grid
searches of 100 trials for RN50 and EN-B0 and 36 trials for RNX101. We set the
LRs and WDs intervals for the grid search to $[5\cdot 10^{-5},5\cdot 10^{-1}]$
to span four orders of magnitude. When training from scratch, we report
results for Twin and SelVS with early stopping, which respectively employ
HB25% and ASHA as schedulers, with the same number of trials. For the
parameters, we follow [5] and keep a halving rate of two and a grace period of
5% of the total epoch budget.

TransfercubScratchflowerseurosat-rgbisic2018clamm0.00.20.40.60.81.0Oracle (B.
Acc.)

cubflowerseurosat-rgbisic2018clamm012410MAE vs Oracle
($\displaystyle\downarrow$)Twin (Transfer)FIFOHB25%HB12%

Figure 5: Transfer learning. (Left) Normalized balanced accuracy of the Oracle
with ImageNet pre-trained (top) or from-scratch RN50 (bottom). Feature overlap
makes the best generalization appear with lower regularization, and Twin (with
EN-B0, RN50, RNX101) plus early stopping identifies this region by scoring a
low MAE (right).

#### Results from scratch.

As visible in Tab. 1, Twin nearly matches the Oracle balanced accuracy by
scoring an MAE of less than 1.5% across different datasets and networks. Twin
outperforms the traditional HP selection from the validation sets (SelVS) by
scoring 71.3% versus 69.2% with EN-B0, 75.5% vs 74% with RN50, and 75.4 vs
73.7 with RNX101 when averaging performance across the CUB, ISIC 2018,
EuroSAT, and CLaMM datasets. Indeed, SelVS relies on a small validation set,
which may lead to sub-optimal HPs given the higher variability of the
prediction error. Furthermore, despite aggressive trial stopping making the
optimal region-segmentation step more challenging, Twin still finds semi-
optimal LR-WD configurations and hence is scalable to computationally heavy
search tasks that would be prohibitive without early stopping strategies.

#### Results with transfer learning.

When dealing with small datasets, it is common practice to start from a
network pre-trained on a larger amount of data (e.g., ImageNet [42]).
Therefore, we also experiment with transfer learning and repeat the
optimization runs with checkpoints initialized from ImageNet. In Fig. 5
(left), we notice that the generalization of networks as a function of the LR-
WD space may differ from when training from scratch, and the main cause
regards the overlapping between the source and target domains. Expectedly,
with a strong class (CUB and Flowers) or feature (EuroSAT RGB, ISIC 2018)
overlap, the best comprehension region shifts towards smaller regularization.
To this end, Twin struggles, as visible from the higher MAE ($\sim$5%) in the
cases of CUB, EuroSAT RGB, and ISIC 2018. As a solutions, we employ early
stopping to terminate the mostly regularized trials whose training loss has a
slower decay rate. In Fig. 5 (right), it is indeed visible that Twin with
HB12% reduces the MAE vs the Oracle to $\leq$ 1% for CUB, EuroSAT RGB, and
ISIC 2018. Conversely, in the case of CLaMM, which has no class and poor
feature overlap (hand-written documents), the pre-trained checkpoints do not
alter the LR-WD landscape and enable Twin to find good HP configurations (< 2%
MAE) with the FIFO and HB25% schedulers. In summary, when applying transfer
learning, it is critical to consider the level of domain overlap to select the
more suitable Twin configuration.

###  4.3 Medical Images

Method | Path | Derma | OCT | Pneum. | Retina | Breast | Blood | Tissue | OrganA | OrganC | OrganS | Avg.
---|---|---|---|---|---|---|---|---|---|---|---|---
Oracle | 91.9 | 80.8 | 79.8 | 92.8 | 52.5 | 89.7 | 97.8 | 73.2 | 95.9 | 94.5 | 84.4 | 84.8
SelTS | 91.9 | 79.7 | 77.3 | 90.7 | 47.8 | 85.9 | 97.2 | 72.8 | 95.0 | 94.0 | 82.8 | 83.2
SelVS | 90.5 | 80.3 | 78.0 | 92.5 | 46.0 | 85.3 | 96.9 | 72.8 | 94.9 | 94.4 | 83.5 | 83.2
Twin | 88.5 | 80.8 | 78.6 | 88.9 | 46.7 | 86.5 | 97.3 | 72.7 | 95.3 | 93.7 | 83.4 | 82.9

Table 2: Medical images. The performance is the test accuracy (%). We allocate
a 100-trial budget per dataset. Both Twin and SelVS employ the FIFO scheduler.

#### Datasets.

We leverage the MedMNIST v2 benchmark [59] to test Twin on medical imaging
tasks. We focus on 2D classification and select 11 out of 12 binary/multi-
class or ordinal regression tasks of the MedMNIST2D sub-collection, which
covers primary data modalities (e.g., X-ray, OCT, Ultrasound, CT, Electron
Microscope) and data scales (from 800 to 100,000 samples). The MedMNIST2D
benchmark provides held-out validation sets to allow HP tuning. The data
diversity of this benchmark presents a significant challenge. We select the
testbed with the images pre-processed to $28\times 28$ resolution out of the
full benchmark to maintain the total computational load under a reasonable
budget.

#### Implementation details.

We use the ResNet-18 (RN18) originally employed in the benchmark [59], which
consists of four stages, as the version developed for ImageNet classification
[16], but with a modified stem more suitable for low-resolution images. We
keep the same Twin configurations tested on small datasets (Sec. 4.2), except
for the trial schedulers that we default to FIFO for both Twin and SelVS.
Refer to the Appendix for additional details on the implementation.

#### Results.

We summarize the empirical results over MedMNIST2D in Tab. 2. The Oracle
scores an upper bound 84.8% test accuracy averaged across the 11 tasks. Twin
is comparable to the traditional SelVS (82.9% vs 83.2%). Note also that Twin
slightly improves its performance in this domain when early stopping is
employed (Sec. 4.5). Twin finds proper HPs and simultaneously leads to a cost-
effective solution by reducing data collection and labeling expenses
associated with the $\sim$10% of samples per dataset originally allocated for
validation.

###  4.4 Natural Images

Dataset | Model | # Trials | Aug. | Oracle | SelTS | SelVS | Twin
---|---|---|---|---|---|---|---
C10 | MLP-4-256 | 100 | + | 66.1 | 65.1 | 65.9 | 65.4
C10 | CCT-2/3×2 | 49 | +++ | 87.3 | 87.3 | 87.3 | 86.7
C10 | RNX11 | 100 | + | 90.7 | 90.2 | 90.6 | 89.8
C10 | RN20 | 100 | + | 92.7 | 91.8 | 92.4 | 90.5
C10 | WRN-40-2 | 49 | ++ | 94.0 | 93.7 | 93.3 | 93.6
C100 | MLP-4-512 | 100 | ++ | 35.4 | 35.0 | 35.1 | 34.9
C100 | CCT-2/3×2 | 49 | +++ | 65.0 | 64.0 | 65.0 | 65.0
C100 | RNX11 | 100 | + | 68.8 | 67.7 | 68.6 | 66.8
C100 | RN20 | 100 | + | 69.8 | 67.6 | 69.0 | 68.2
C100 | WRN-40-2 | 49 | ++ | 74.2 | 74.2 | 72.8 | 72.8
TinyIN | CVT-7/8 | 36 | +++ | 58.0 | 58.0 | 58.0 | 58.0
TinyIN | WRN-16-4 | 49 | ++ | 61.8 | 60.8 | 61.8 | 61.3
Average |  |  |  | 72.0 | 71.3 | 71.6 | 71.1

Table 3: Natural Images. The performance is reported in test accuracy (%).
FIFO scheduler is employed. For the strength of data augmentation, refer to
the Appendix.

#### Datasets.

Finally, we test Twin on popular natural-image datasets such as CIFAR-10/100
[25] and Tiny Imagenet [27]. These datasets contain 50,000/100,000 training
samples from 10 to 200 classes, with an image resolution of $32\times 32$ for
CIFAR and $64\times 64$ for TinyImagenet.

#### Implementation details.

For CIFAR datasets, we employ ConvNets, transformers, and feed-forward
networks. As ConvNets, we select ResNet-20 (RN20) [16], ResNeXt-11 (4 x 16d)
(RNX11) [57], and a Wide ResNet of depth 40 and width two (WRN-40-2) [63]. As
transformers, we train architectures specifically designed for CIFAR, such as
the Compact Convolutional Transformer with two encoder and convolutional-stem
layers (CCT-2/3$\times$2) [15]. We employ a Multi-Layer Perceptron (MLP) with
batch normalization, ReLU activations, and hidden layers of constant width as
feed-forward networks. We set the depth of the MLP to four layers and the
width to 256 units for CIFAR-10 and 512 units for CIFAR-100 (MLP-4-256 and
MLP-4-512). On TinyImagenet, we train a WRN-16-4 [63] and a Compact Vision
Transformer [15] with seven encoder layers and a patch size of 8 (CVT-7/8). We
vary the data augmentation strength from base to medium to strong {+, ++,
+++}. Refer to the Appendix for additional details.

#### Results.

As shown in Tab. 3, Twin is, on average, comparable to SelVS (71.1% vs 71.6%),
despite not having access to the validation set, and SelTS (71.1% vs 71.3%),
while considering the regularization strength along with the training loss.
Remarkably, we also notice that Twin works properly for transformers and MLPs,
confirming that the intuition behind Twin translates well to various network
architectures. Similarly, Twin is agnostic to the data-augmentation strength.

kernel_size | max_dist | MAE
---|---|---
$N_{\alpha}$ | $N_{\alpha}$ | 9.4
$\sqrt{N_{\alpha}}$ | $N_{\alpha}$ | 4.2
$N_{\alpha}$ | $\sqrt{N_{\alpha}}$ | 1.4
$\sqrt{N_{\alpha}}$ | $\sqrt{N_{\alpha}}$ | 1.3

(a) Quickshift

$\alpha$ | $\lambda$ |
---|---|---
[::1] | [::2] | [::3] | [::1] | [::2] | [::3] | MAE
✓ |  |  | ✓ |  |  | 1.3
✓ |  |  |  | ✓ |  | 1.4
| ✓ |  | ✓ |  |  | 1.8
| ✓ |  |  | ✓ |  | 1.5
✓ |  |  |  |  | ✓ | 1.5
|  | ✓ | ✓ |  |  | 1.2
|  | ✓ |  |  | ✓ | 1.2

(b) Grid density

Table 4: Ablation studies concerning: (a) Quickshift controlling the
segmentation density and (b) the robustness of Twin against the grid density.
The MAE $(\downarrow)$ is computed against the Oracle baseline.

|  Small Datasets |  Medical Datasets
---|---|---
Scheduler |  EN-B0 |  RN50 |  RNX101 |  RN18
FIFO |  71.3 |  75.2 |  75.4 |  82.9
HB25% |  71.3 |  75.5 |  75.4 |  83.8
HB12% |  71.5 |  74.9 |  69.8 |  83.2

Table 5: Ablation on early stopping for Twin. The performance is the average
balanced test accuracy (%) on small datasets and test accuracy (%) on medical
datasets.

###  4.5 Ablations

#### Quickshift.

We ablate on the impact of kernel_size and max_dist from Quickshift in Tab.
4(a). In this analysis, we assume a squared grid ($N_{\alpha}=N_{\lambda}$)
and hence report the parameters as a function of $N_{\alpha}$ only. We
consider max($N_{\alpha},$ $N_{\lambda}$) when the grid is not squared. In
Tab. 4(a), it is visible that setting max_dist as large as the size of the
grid side (i.e., $N_{\alpha}$) leads to poor results because Quickshift tends
to segment the loss matrix into an insufficient number of regions. By fixing
kernel_size and max_dist to $\sqrt{N_{\alpha}}$, we increase the segmentation
density which is necessary to identify the best region in terms of task
performance.

#### Grid density.

We systematically test the robustness of Twin against different grid
densities. We slice the log-spaced intervals of LR ($\alpha$) and/or WD
($\lambda$) by sampling values every one, two, or three steps and refer to
such operations with the python slicing notation [::x]. We average the error
of Twin (FIFO scheduler) against the Oracle across 34 different configurations
(as Fig. 3) and show the analysis in Tab. 4(b). The MAE remains almost
unaffected and close to 1.3% obtained with default settings to show Twin’s
support to various grid intervals.

#### Early stopping.

In Tab. 5, we ablate concerning the impact of the early stopping scheduler. As
visible, Twin effectively accommodates HB25% or HB12%. Practitioners could
safely default to either of the two, with HB25% slightly ahead. The drop in
performance for RNX101 with HB12% is due to the small $7\times 7$ grid
employed. We refer to the Appendix for additional comments and guidance.

#### Optimizers and schedulers.

In all experiments throughout the paper, we employed SGD with momentum (SGDM)
and cosine scheduler as standard practice in deep learning. In this paragraph,
we ablate on the possibility of using different optimization setups. In
particular, we test plain SGD in six configurations involving RN20 on
CIFAR-10/100, WRN-16-10 on ciFAIR-10, and RN50 on EusoSAT, ISIC 2018, and
CLaMM. We also test RN20 with a piece-wise LR scheduler. Finally, we train
ConvNets, MLPs, and transformers with either Adam [23] or AdamW [36], two
popular choices when training such models. In Tab. 6, we notably observe that
Twin also closely follows the Oracle in terms of (mean) absolute error (M)AE
in such alternative optimization setups.

Dataset | Model | Optim. Setup | Performance | (M)AE
---|---|---|---|---
{C10, C100, ES, I2018, c10, CM} | {WRN-16-10, RN20, RN50} | SGD | 75.0 | 1.8
SGDM | 75.6 | 1.2
C100 | CVT-7/8 | AdamW | 68.8 | 0.0
C10 | MLP-4-256 | Adam | 65.5 | 0.6
c10 | WRN-16-10 | Adam | 54.5 | 0.6
C10 | RN20 | SGDM (piece-wise) | 91.0 | 1.7
SGDM (cosine) | 90.5 | 2.2

Table 6: Ablation concerning the usage of different optimization setups.
Performance is the (balanced) test accuracy. The (M)AE $(\downarrow)$ is
computed against the Oracle baseline.

##  5 Conclusions

We introduced Twin, a simple yet effective HP tuning approach that reliably
predicts learning rate and weight decay without using validation sets. Twin is
not only beneficial in practice for simplifying model selection pipelines but
also provides additional insights into the predictability of generalization
for deep networks. Twin showed robust performance from a broad suite of
experimental scenarios, including varying dataset sizes, imaging domains,
architectures, model scales, and training setups. In this paper, we
exclusively focused on image classification problems. However, future work
could explore the application of Twin to computer vision tasks and beyond.
Moreover, there is potential for extending Twin to alternative regularization
strategies beyond L2 penalty.

## References

  * [1] Barz, B., Denzler, J.: Deep learning on small datasets without pre-training using cosine loss. In: IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) (2020)
  * [2] Barz, B., Denzler, J.: Do we train on test data? purging CIFAR of near-duplicates. Journal of Imaging 6(6) (2020). https://doi.org/10.3390/jimaging6060041, https://www.mdpi.com/2313-433X/6/6/41
  * [3] Benton, G., Finzi, M., Izmailov, P., Wilson, A.G.: Learning invariances in neural networks from training data. Advances in neural information processing systems (2020)
  * [4] Brigato, L., Barz, B., Iocchi, L., Denzler, J.: Tune it or don’t use it: Benchmarking data-efficient image classification. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1071–1080 (2021)
  * [5] Brigato, L., Barz, B., Iocchi, L., Denzler, J.: Image classification with small datasets: overview and benchmark. IEEE Access (2022)
  * [6] Brigato, L., Iocchi, L.: A close look at deep learning with small data. In: 2020 25th International Conference on Pattern Recognition (ICPR) (2021)
  * [7] Brigato, L., Mougiakakou, S.: No data augmentation? alternative regularizations for effective training on small datasets. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops (2023)
  * [8] Bruintjes, R.J., Lengyel, A., Rios, M.B., Kayhan, O.S., Zambrano, D., Tomen, N., van Gemert, J.: Vipriors 3: Visual inductive priors for data-efficient deep learning challenges. arXiv preprint arXiv:2305.19688 (2023)
  * [9] Codella, N., Rotemberg, V., Tschandl, P., Celebi, M.E., Dusza, S., Gutman, D., Helba, B., Kalloo, A., Liopyris, K., Marchetti, M., et al.: Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (ISIC). arXiv preprint arXiv:1902.03368 (2019)
  * [10] Cubuk, E.D., Zoph, B., Mane, D., Vasudevan, V., Le, Q.V.: Autoaugment: Learning augmentation strategies from data. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2019)
  * [11] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)
  * [12] Elsken, T., Metzen, J.H., Hutter, F.: Neural architecture search: A survey. The Journal of Machine Learning Research (2019)
  * [13] Golovin, D., Solnik, B., Moitra, S., Kochanski, G., Karro, J.E., Sculley, D. (eds.): Google Vizier: A Service for Black-Box Optimization (2017)
  * [14] Goyal, P., Dollár, P., Girshick, R., Noordhuis, P., Wesolowski, L., Kyrola, A., Tulloch, A., Jia, Y., He, K.: Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv:1706.02677 (2017)
  * [15] Hassani, A., Walton, S., Shah, N., Abuduweili, A., Li, J., Shi, H.: Escaping the big data paradigm with compact transformers. arXiv preprint arXiv:2104.05704 (2021)
  * [16] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
  * [17] He, T., Zhang, Z., Zhang, H., Zhang, Z., Xie, J., Li, M.: Bag of tricks for image classification with convolutional neural networks. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (2019)
  * [18] Helber, P., Bischke, B., Dengel, A., Borth, D.: EuroSAT: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 12(7), 2217–2226 (2019). https://doi.org/10.1109/JSTARS.2019.2918242
  * [19] Hendrycks, D., Dietterich, T.G.: Benchmarking neural network robustness to common corruptions and perturbations. In: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 (2019)
  * [20] Heo, B., Chun, S., Oh, S.J., Han, D., Yun, S., Kim, G., Uh, Y., Ha, J.W.: Adamp: Slowing down the slowdown for momentum optimizers on scale-invariant weights. arXiv preprint arXiv:2006.08217 (2020)
  * [21] Immer, A., van der Ouderaa, T., Rätsch, G., Fortuin, V., van der Wilk, M.: Invariance learning in deep neural networks with differentiable laplace approximations. Advances in Neural Information Processing Systems (2022)
  * [22] Keskar, N.S., Mudigere, D., Nocedal, J., Smelyanskiy, M., Tang, P.T.P.: On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609.04836 (2016)
  * [23] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [24] Kornblith, S., Shlens, J., Le, Q.V.: Do better imagenet models transfer better? In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2661–2671 (2019)
  * [25] Krizhevsky, A.: Learning multiple layers of features from tiny images (2009)
  * [26] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25 (2012)
  * [27] Le, Y., Yang, X.: Tiny imagenet visual recognition challenge. CS 231N (2015)
  * [28] Lengyel, A., Bruintjes, R.J., Rios, M.B., Kayhan, O.S., Zambrano, D., Tomen, N., van Gemert, J.: Vipriors 2: visual inductive priors for data-efficient deep learning challenges. arXiv preprint arXiv:2201.08625 (2022)
  * [29] Li, L., Jamieson, K., Rostamizadeh, A., Gonina, E., Ben-tzur, J., Hardt, M., Recht, B., Talwalkar, A.: A system for massively parallel hyperparameter tuning. Conference of Machine Learning and Systems (2020)
  * [30] Li, L., Jamieson, K., DeSalvo, G., Rostamizadeh, A., Talwalkar, A.: Hyperband: A novel bandit-based approach to hyperparameter optimization. The journal of machine learning research (2017)
  * [31] Li, Y., Hu, G., Wang, Y., Hospedales, T., Robertson, N.M., Yang, Y.: Dada: Differentiable automatic data augmentation. arXiv preprint arXiv:2003.03780 (2020)
  * [32] Li, Y., Wei, C., Ma, T.: Towards explaining the regularization effect of initial large learning rate in training neural networks. Advances in Neural Information Processing Systems (2019)
  * [33] Liu, Z., Kitouni, O., Nolte, N.S., Michaud, E., Tegmark, M., Williams, M.: Towards understanding grokking: An effective theory of representation learning. Advances in Neural Information Processing Systems 35, 34651–34663 (2022)
  * [34] Liu, Z., Michaud, E.J., Tegmark, M.: Omnigrok: Grokking beyond algorithmic data. In: The Eleventh International Conference on Learning Representations, ICLR 2023 (2023)
  * [35] Lorraine, J., Vicol, P., Duvenaud, D.: Optimizing millions of hyperparameters by implicit differentiation. In: International conference on artificial intelligence and statistics (2020)
  * [36] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
  * [37] McMahan, B., Moore, E., Ramage, D., Hampson, S., y Arcas, B.A.: Communication-efficient learning of deep networks from decentralized data. In: Artificial intelligence and statistics (2017)
  * [38] Mlodozeniec, B., Reisser, M., Louizos, C.: Hyperparameter optimization through neural network partitioning. arXiv preprint arXiv:2304.14766 (2023)
  * [39] Neelakantan, A., Vilnis, L., Le, Q.V., Sutskever, I., Kaiser, L., Kurach, K., Martens, J.: Adding gradient noise improves learning for very deep networks. arXiv preprint arXiv:1511.06807 (2015)
  * [40] Nilsback, M.E., Zisserman, A.: Automated flower classification over a large number of classes. In: 2008 Sixth Indian conference on computer vision, graphics & image processing. IEEE (2008)
  * [41] Power, A., Burda, Y., Edwards, H., Babuschkin, I., Misra, V.: Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177 (2022)
  * [42] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. International Journal of Computer Vision (IJCV) 115(3), 211–252 (2015)
  * [43] Schwöbel, P., Jørgensen, M., Ober, S.W., Van Der Wilk, M.: Last layer marginal likelihood for invariance learning. In: International Conference on Artificial Intelligence and Statistics (2022)
  * [44] Stone, M.: Cross-validatory choice and assessment of statistical predictions. Journal of the royal statistical society: Series B (Methodological) (1974)
  * [45] Stutzmann, D.: Clustering of medieval scripts through computer image analysis: towards an evaluation protocol. Digital Medievalist 10 (2016)
  * [46] Tan, M., Le, Q.: Efficientnet: Rethinking model scaling for convolutional neural networks. In: International conference on machine learning (2019)
  * [47] Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., Jégou, H.: Training data-efficient image transformers & distillation through attention. In: International conference on machine learning (2021)
  * [48] Van Laarhoven, T.: L2 regularization versus batch and weight normalization. arXiv preprint arXiv:1706.05350 (2017)
  * [49] Vapnik, V.: Principles of risk minimization for learning theory. Advances in neural information processing systems (1991)
  * [50] Varoquaux, G., Cheplygina, V.: Machine learning for medical imaging: methodological failures and recommendations for the future. NPJ digital medicine (2022)
  * [51] Vedaldi, A., Soatto, S.: Quick shift and kernel methods for mode seeking. In: 10th European Conference on Computer Vision, Marseille, France, October 12-18, 2008, Proceedings, Part IV 10 (2008)
  * [52] Wad, T., Sun, Q., Pranata, S., Jayashree, K., Zhang, H.: Equivariance and invariance inductive bias for learning from insufficient data. In: Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XI (2022)
  * [53] Wah, C., Branson, S., Welinder, P., Perona, P., Belongie, S.: The Caltech-UCSD Birds-200-2011 Dataset. Tech. Rep. CNS-TR-2011-001, California Institute of Technology (2011)
  * [54] Weiss, K., Khoshgoftaar, T.M., Wang, D.: A survey of transfer learning. Journal of Big data 3(1), 1–40 (2016)
  * [55] van der Wilk, M., Bauer, M., John, S., Hensman, J.: Learning invariances using the marginal likelihood. Advances in Neural Information Processing Systems (2018)
  * [56] Worrall, D.E., Garbin, S.J., Turmukhambetov, D., Brostow, G.J.: Harmonic networks: Deep translation and rotation equivariance. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)
  * [57] Xie, S., Girshick, R., Dollár, P., Tu, Z., He, K.: Aggregated residual transformations for deep neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (2017)
  * [58] Yang, G., Hu, E., Babuschkin, I., Sidor, S., Liu, X., Farhi, D., Ryder, N., Pachocki, J., Chen, W., Gao, J.: Tuning large neural networks via zero-shot hyperparameter transfer. Advances in Neural Information Processing Systems 34, 17084–17097 (2021)
  * [59] Yang, J., Shi, R., Wei, D., Liu, Z., Zhao, L., Ke, B., Pfister, H., Ni, B.: Medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. Scientific Data (2023)
  * [60] Yu, T., Zhu, H.: Hyper-parameter optimization: A review of algorithms and applications. arXiv preprint arXiv:2003.05689 (2020)
  * [61] Yun, J., Kim, B., Kim, J.: Weight decay scheduling and knowledge distillation for active learning. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXVI 16 (2020)
  * [62] Yun, J., Lee, J., Shon, H., Yi, E., Kim, S.H., Kim, J.: On the angular update and hyperparameter tuning of a scale-invariant network. In: European Conference on Computer Vision (2022)
  * [63] Zagoruyko, S., Komodakis, N.: Wide residual networks. In: Richard C. Wilson, E.R.H., Smith, W.A.P. (eds.) British Machine Vision Conference (BMVC). pp. 87.1–87.12. BMVA Press (September 2016). https://doi.org/10.5244/C.30.87
  * [64] Zhang, G., Wang, C., Xu, B., Grosse, R.B.: Three mechanisms of weight decay regularization. In: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 (2019)


# Extension of the correspondence principle in relativistic quantum mechanics

K. G. Hernández<EMAIL_ADDRESS>Facultad de Ciencias Naturales y
Matemática, Universidad de El Salvador, Ciudad Universitaria, Av. Mártires y
Héroes del 30 julio, San Salvador, El Salvador, América Central.  S. E.
Aguilar<EMAIL_ADDRESS>Facultad de Ciencias Naturales y Matemática,
Universidad de El Salvador, Ciudad Universitaria, Av. Mártires y Héroes del 30
julio, San Salvador, El Salvador, América Central.  Instituto de Física
Teórica, Universidade Estadual Paulista (UNESP), R. Dr. Bento Teobaldo Ferraz
271, Barra Funda São Paulo- SP, 01140-070, Brazil.  J. Bernal
<EMAIL_ADDRESS>División Académica De Ciencias Básicas,
Universidad Juárez Autónoma de Tabasco, Carretera Cunduacán-Jalpa Km 1,
Cunduacán, Tabasco, México. A.P. 24 C.P. 86690

(August 27, 2024)

###### Abstract

In this paper we apply the Bohr’s correspondence principle to analyze the
asymptotic behavior of the Klein-Gordon and Dirac probability densities. It is
found that in the non-relativistic limit, the densities reduce to their
respective classical single-particle probability distributions plus a series
of quantum corrections. The procedure is applied in two basic problems, the
relativistic quantum oscillator and the relativistic particle in a box. The
particle and antiparticle solutions are found to yield the same classical
distribution plus quantum correction terms for the proposed limit. In the
quantum oscillator case, a $\kappa$ parameter modifies the probability
distribution. Its origin is briefly discussed in terms of energy.

Keywords: Correspondence principle, Klein-Gordon oscillator, Dirac oscillator,
Particle in a box, Non-relativistic limit.

##  1 Introduction

The classical limit of quantum mechanics is a fundamental problem. There are
different methodologies that have been proposed to derive classical
observables from quantum observables involving constraints. The limit of
quantum mechanics as the Planck constant approaches zero, $\hbar\to 0$, is one
of the first methods that explore the classical limit of quantum physics [1].
However, the limit $\hbar\to 0$ is misleading since Planck’s constant in a
nonzero universal constant, and thus, this limit should be understood when
$\hbar$ is negligible with respect to the other physical parameters, like when
the Raileigh-Jean’s law is obtained from the Planck’s Law. A second link
between quantum and classical mechanics is the Ehrenferest’s theorem, which
states that the quantum mechanical expectation values of the position and
momentum operators satisfy the classical equations of motion [2, 3]. However,
the applicability of this theorem is neither necessary nor sufficient, since
the classical limit of a quantum system is an ensemble of classical orbits,
where its mean position $\expectationvalue{x}$ does not necessarily follow
from the corresponding classical orbit [4]. Moreover, this implies that when
Ehrenfest’s theorem is not applicable, a quantum system may have a classic
behavior.

Another example is Wigner’s distribution function, which presents congruent
results when studying the quantum corrections of classical statistical
mechanics [5, 6]. Nevertheless, the Wigner’s function needs restrictions in
order to be interpreted as a probability distribution [1]. On the other hand,
Bohm’s potential formulation of quantum mechanics states that the Hamilton-
Jacobi equation can be recovered from the Schrödinger equation, resulting in
the WKB method (semiclassical regime), which presents congruent results
subject to constraints [7, 8].

The partition function of a system of quantum particles can be used to derive
the classical partition function of the system and its equations of state [9].
Nonetheless, this correspondence should also be studied from quantum to
classical single-particle configurations. Chaotic and regular motions in the
Henon-Heiles model of quantum and classical probability distributions has been
studied in phase-space, the difference in the centroids of the respective
quantum and classical distributions has been calculated and compared with the
prediction by the Ehrenfest’s theorem [10]. Consistency between classical and
quantum models of a localized spin driven by a polarization requires the
correspondence of the classical and quantum auto-correlation functions of the
spin components [11]. Some authors compare classical and quantum probability
densities of single particles [12, 13], but they do not establish a
correspondence between their distributions.

Niels Bohr established a correspondence principle where a classical behavior
is recovered when the quantum principal number, $n$, is large. Bohr applied
this correspondence for frequencies and orbits of quantum systems [1, 14], and
it has been applied successfully in atomic physics [15, 16, 17]. However,
Ketchum contradicts the notion that classical variables can only be obtained
from large principal quantum numbers, suggesting that classical frequencies
can be recovered from small quantum numbers [18]. An example of ”breakdown” of
the Bohr’s correspondence principle has been found in the semiclassical regime
[19], although another study argued that the conclusions do not hold since the
regime of the WKB approximation is limited [20].

Previous studies have been conducted on the correspondence between classical
mechanics and quantum field theory. Earlier studies, applied the Planck’s
limit to quantum field theory with arbitrary interactions, demonstrated that
semiclassical limits are always reached [21]. A recent paper by Staunton and
Browne suggest that the Poisson brackets in quantum states can be used to
derive a classical trajectory [22]. Shin and Rafelski discussed the classical
limit of the relativistic quantum transport theory [23]. Brown and Goble
explored the correspondence between quantum and classical electrodynamics and
concluded that the bremsstrahlung amplitude has a correspondence with the
classical radiation [24]. It has been proven that vacuum classical radiation
occurs in the limit of low frequencies and high photon density [25]. The
classical limit of the quantum electrodynamics was found when Dente used the
path integral formulation in order to eliminate photon coordinates [26].

Refs. [1, 27, 28] propose a simple mathematical method to derive the classical
probability distribution of a quantum system from its corresponding quantum
mechanical analogue. This method has been successfully applied to the harmonic
oscillator, the infinite square well and the Kepler’s problem. Additionally,
the authors show that the Bohr’s correspondence principle is applicable to the
quantum mechanical probability densities, and its application can be performed
for periodic systems (such as atomic orbits).

Here, we propose that the Bohr’s correspondence principle shown in Bernal and
others [1] can also be applied and extended to relativistic quantum systems.
Our results show that, in the high energy regime (large principal quantum
numbers), the quantum distribution can be written as a power series in
$\hbar$, so that the zeroth order corresponds exactly with the classical
single-particle probability distribution. In this paper, we focus on
particular relativistic quantum systems as described by the Klein-Gordon and
the Dirac equations, namely, the infinite square well and the harmonic
oscillator.

##  2 General procedure

The methodology of our research is based on the one described in Refs. [1, 27,
28], but we include the non-relativistic limit. The formulation of this
article is purely in first quantization and we only consider single
eigenstates of the 1-dimensional Klein-Gordon and Dirac equations.

The relativistic probability density of the Dirac spinor that corresponds to
the $n$-th energy state is given by
$\rho_{n}^{RQM}(x)=\psi_{n}^{\dagger}(x)\psi_{n}(x)$ where $\psi_{n}$ is the
four component Dirac spinor that corresponds to the $n$-th energy state and
the symbol $\dagger$ stands for conjugate transpose. In the Klein-Gordon case,
the probability density is given by
$\rho^{RQM}_{n}(x)=\frac{E_{n}}{m}\phi_{n}^{*}(x)\phi_{n}(x)$, where
$\phi_{n}(x)$ is the wavefunction of the $n$-th eigenstate with energy
$E_{n}$, which follows from
$\rho(x,\>t)=\frac{i\hbar}{2mc^{2}}\left[\phi^{*}(x,\>t)\frac{\partial}{\partial
t}\phi(x,\>t)-\phi(x,\>t)\frac{\partial}{\partial t}\phi^{*}(x,\>t)\right]$
[29] when the time-dependent wavefunction is written as
$\phi(x,\>t)=\phi_{n}(x)e^{-\frac{iE_{n}t}{\hbar}}$. The procedure can be
applied for either the particle or antiparticle solutions, with $E_{n}>0$ or
$E_{n}<0$ respectively. We denote the Fourier transform of the probability
density $\rho_{n}^{RQM}(x)$ from position to its Fourier transform in momentum
space, which we denote $f_{n}^{RQM}(p)$.

The extension of the correspondence principle requires the calculation of the
Fourier coefficient $f_{n}^{RQM}(p)$, such that by inverse Fourier
transforming this result we obtain, in the zeroth order of approximation, the
probability density in coordinate representation. The non-relativistic limit
(where the speed of light is larger than the typical velocities of the system)
fixes the relativistic quantum energy $E_{n}^{QRM}$ to a classical value, and
also affects the resulting probability distribution, denoted by
$\rho^{QM}_{n}(x)$, so that it becomes into a classical probability density
plus a series of quantum correction terms, which appear in a power series in
terms of the $\hbar$ constant, as it will be seen in the relativistic quantum
oscillator cases.

##  3 Examples

###  3.1 Klein-Gordon oscillator

According to Ref. [30], the wave function of the one-dimensional Klein-Gordon
oscillator is exactly the same as that of the one-dimensional Schrödinger
oscillator; however, the energy spectrum is modified by the rest energy of the
system. For either particle or antiparticle, the probability density for a
linear combination of particle and antiparticle stationary states is given by

$\rho^{RQM}_{n}(x)=\frac{E_{n}}{mc^{2}}\sqrt{\frac{\alpha}{\pi}}\frac{1}{2^{n}n!}H^{2}_{n}(\sqrt{\alpha}x)e^{-\alpha
x^{2}},$ (1)

where $H_{n}(x)$ are the Hermite polynomials, $n$ is a non-negative integer,
and $\alpha\equiv\frac{m\omega}{\hbar}$, where, $m$ stands for the mass of the
particle or antiparticle, $\omega$ the frequency of the oscillator, and the
energies are given by
$E_{n}^{2}=m^{2}c^{4}+2\quantity(n+\frac{1}{2})mc^{2}\hbar\omega$ [30]. The
resulting Fourier transform of Eq. (1) can be found in the literature [31]

$f^{RQM}_{n}(p)=\frac{E_{n}}{mc^{2}}e^{-\frac{p^{2}}{4m\omega\hbar}}L_{n}\quantity(\frac{p}{2m\omega\hbar}),$
(2)

where $L_{n}(x)$ is a Laguerre polynomial of degree $n$. The asymptotic limit
of the Fourier transform can be evaluated in a similar way to the non-
relativistic case by means of Bessel functions [1]. The inverse Fourier
transform in the asymptotic limit is

$\displaystyle\rho^{RQM}_{n}(x)\sim$
$\displaystyle\frac{E_{n}}{mc^{2}}\frac{1}{\pi}\frac{1}{\sqrt{\kappa_{n}^{2}-x^{2}}}$
(3)
$\displaystyle+\frac{E_{n}}{mc^{2}}{2\pi\kappa_{n}}\sum_{j=1}^{\infty}\quantity(\frac{-\hbar^{2}}{S_{n}^{2}})^{j}i_{j}(x,\kappa_{n}),$

where

$\kappa_{n}\equiv\sqrt{\frac{2\hbar(n+\frac{1}{2})}{m\omega}}=\sqrt{\frac{E_{n}^{2}-m^{2}c^{4}}{m^{2}\omega^{2}c^{2}}},$
(4)

is a parameter found in the relativistic quantum mechanical case;
$S_{n}=4\sqrt{2\pi}m\omega\kappa_{n}^{2}$ and $i_{j}(x,\kappa_{n})$ is the
$j$-th dimensionless integral, defined in Ref. [1]. The factor
$\frac{E_{n}}{mc^{2}}$ is a dilation factor which comes from the inverse of
length proportionality of the probability density.

According to Ref. [1], the principal quantum number is fixed by the classical
energy of the system, and this is achieved by equating the expressions for the
classical and quantum energies. This allows us to express the value of the
principal quantum number in terms of the classical amplitude of the
oscillator. In the present case, this procedure yields to
$\absolutevalue{E_{n}}\rightarrow mc^{2}+\frac{1}{2}m\omega^{2}x_{0}^{2}$,
where $x_{0}$ is the amplitude of the oscillator.

It is easy to verify that in the non-relativistic case $\kappa_{n}\rightarrow
x_{0}$, and therefore

$\displaystyle\rho^{QM}_{n}(x)\sim$
$\displaystyle\frac{1}{\pi}\frac{1}{\sqrt{x_{0}^{2}-x^{2}}}$ (5)
$\displaystyle+\frac{1}{2\pi
x_{0}}\sum_{j=1}^{\infty}\quantity(\frac{-\hbar^{2}}{S^{2}})^{j}i_{j}(x,x_{0}),$

where $S=4\sqrt{2\pi}m\omega x_{0}^{2}$ is the classical action of the
particle up to a constant factor.

As we can see in Eq. (5), the zeroth order result, which is
$\hbar$-independent, corresponds to the classical probability distribution of
a single harmonic oscillator, while the higher order terms can be interpreted
as quantum corrections. This is the same result derived in non-relativistic
quantum mechanics [1].

Note that we have not specified whether the wavefunction is for particles or
antiparticles, it follows that stationary states of particles or antiparticles
yield to the classical probability distribution for the limit that have been
considered.

###  3.2 Klein-Gordon particle in a box

Consider a Klein-Gordon stationary state for either particle or antiparticle
trapped in an infinite square well of length $0\leq x\leq L$. The probability
density is almost the same as the one that would be found from the Schrödinger
equation [29]

$\rho^{RQM}_{n}(x)=\frac{E_{n}}{mc^{2}}\frac{2}{L}\sin^{2}\quantity(\frac{n\pi
x}{L}).$ (6)

A difference with respect to the non-relativistic case is the energy spectrum,
which is given by

$\displaystyle E_{n}^{2}=m^{2}c^{4}+c^{2}\hbar^{2}\frac{n^{2}\pi^{2}}{L^{2}}.$
(7)

According to Ref. [1], by writing the probability density as a Fourier
expansion, the asymptotic behavior of the corresponding Fourier coefficients
is

$f^{RQM}_{n}(p)\sim\frac{E_{n}}{mc^{2}}\frac{i\hbar}{pL}\quantity(e^{-i\frac{Lp}{\hbar}}-1).$
(8)

such that by inverse Fourier transforming we find

$\rho^{RQM}_{n}(x)\sim\frac{E_{n}}{mc^{2}}\frac{1}{L}\quantity[H(L-x)-H(-x)],$
(9)

where $H(x)$ is the Heaviside step function.

We observe that the Klein-Gordon solution yields the same classical limit as
that obtained by using the nonrelativistic Schrödinger equation up to a
Lorentz dilatation factor given by the ratio $\frac{E_{n}}{mc^{2}}$, which is
expected by the dependence $1/L$ in the probability density. Comparing the
lack of quantum correction terms with the result of the Klein-Gordon
oscillator, this happens because the wavefunctions and the asymptotic
condition are the same as the non-relativistic case of the particle in a box
[1], where there is no need of explicitly fixing the energies to the classical
value. The leading term of the probability density becomes independent of
relativistic quantum corrections (which is later contrasted with the Dirac
version of this problem).

###  3.3 Dirac oscillator

The Dirac oscillator was originally proposed by Moshinsky and Szczepaniak
[32], and its eigenfunctions in the 1-dimensional case can be found in Ref.
[33]. For either particles or antiparticles, the probability density for
single eigenstates (either spin up or down), has the form

$\displaystyle\rho^{RQM}_{n}(x)=$ $\displaystyle e^{-\alpha
x^{2}}\absolutevalue{a_{n}}^{2}H_{n}^{2}(\sqrt{\alpha}x)$ (10)
$\displaystyle+e^{-\alpha
x^{2}}\absolutevalue{a^{\prime}_{n}}^{2}H_{n-1}^{2}(\sqrt{\alpha}x),$

where $\alpha\equiv\frac{m\omega}{\hbar}$,
$\absolutevalue{a_{n}}^{2}=\frac{\sqrt{\alpha}(E_{n}+mc^{2})}{2^{n+1}n!E_{n}\sqrt{\pi}}$,
$\absolutevalue{a^{\prime}_{n}}^{2}=\frac{\sqrt{\alpha}(E_{n}-mc^{2})}{2^{n}(n-1)!E_{n}\sqrt{\pi}}$
and $E_{n}^{2}=m^{2}c^{4}+2n\hbar\omega mc^{2}$ are the energy values for this
model, denoting $E_{n}>0$ for the particle solution, $E_{n}<0$ for the
antiparticle solution.

The Fourier transforms for the relevant terms of the probability density can
be found analogously to the Klein-Gordon case, which leads to the expression

$\displaystyle f^{RQM}_{n}(p)\sim$ $\displaystyle
e^{-\frac{p^{2}}{4m\omega\hbar}}\absolutevalue{\frac{a_{n}}{A_{n}}}^{2}L_{n}\quantity(\frac{p}{2m\omega\hbar})$
(11)
$\displaystyle+e^{-\frac{p^{2}}{4m\omega\hbar}}\absolutevalue{\frac{a^{\prime}_{n}}{A_{n-1}}}^{2}L_{n-1}\quantity(\frac{p}{2m\omega\hbar}),$

where $\absolutevalue{A_{n}}^{2}=\sqrt{\frac{\alpha}{\pi}}\frac{1}{2^{n}n!}$.

As in the Klein-Gordon case, now we compute the Fourier transform of the
probability density Eq. (11) and next we calculate its asymptotic behavior.
The result is

$\displaystyle\rho^{RQM}_{n}(x)\sim$
$\displaystyle\absolutevalue{\frac{a^{\prime}_{n}}{A_{n-1}}}^{2}\frac{1}{2\pi\kappa_{n-1}}\sum_{j=1}^{\infty}\quantity(\frac{-\hbar^{2}}{S_{n-1}^{2}})^{j}i_{j}(x,\kappa_{n-1})$
(12)
$\displaystyle+\absolutevalue{\frac{a_{n}}{A_{n}}}^{2}\frac{1}{2\pi\kappa_{n}}\sum_{j=1}^{\infty}\quantity(\frac{-\hbar^{2}}{S_{n}^{2}})^{j}i_{j}(x,\kappa_{n})$
$\displaystyle+\frac{1}{\pi}\absolutevalue{\frac{a^{\prime}_{n}}{A_{n-1}}}^{2}\frac{1}{\sqrt{\kappa_{n-1}^{2}-x^{2}}}$
$\displaystyle+\frac{1}{\pi}\absolutevalue{\frac{a_{n}}{A_{n}}}^{2}\frac{1}{\sqrt{\kappa_{n}^{2}-x^{2}}},$

where

$\kappa_{n}=\sqrt{\frac{2\hbar(n+\frac{1}{2})}{m\omega}},\quad
S_{n}=4\sqrt{2\pi}m\omega\kappa_{n}^{2}.$

The non-relativistic case with quantum corrections, as in the Klein Gordon
case, are found by fixing
$E_{N}\rightarrow\pm\quantity(mc^{2}+\frac{1}{2}m\omega^{2}x_{0}^{2})$ and
choosing $N=n\pm\frac{1}{2}$, where the plus and minus sign stands for
particles and antiparticles respectively. In the particle case we find that
$\absolutevalue{a_{n}}^{2}\rightarrow\absolutevalue{A_{n}}^{2}$,
$a^{\prime}_{n}\rightarrow 0$ and $\kappa_{n}\rightarrow x_{0}$. In the
antiparticle case $a_{n}\rightarrow 0$,
$\absolutevalue{a^{\prime}_{n}}^{2}\rightarrow\absolutevalue{A_{n-1}}^{2}$ and
$\kappa_{n-1}\rightarrow x_{0}$. Thus for either case Eq. (12) becomes

$\rho^{QM}_{n}(x)\sim\frac{1}{\pi}\frac{1}{\sqrt{x_{0}^{2}-x^{2}}}+\frac{1}{2\pi
x_{0}}\sum_{j=1}^{\infty}\quantity(\frac{-\hbar^{2}}{S^{2}})^{j}i_{j}(x,x_{0}),$
(13)

where $S=4\sqrt{2\pi}m\omega x_{0}^{2}$. It should be emphasized that the
system can be in an up or down spin state, or a linear combination of both,
but in either case the probability density adopts the same form shown in Eq.
(10), which corresponds to the classical expressions dictated by the
correspondence principle. We also observe that in the fermionic case, two
$\kappa_{n}$ parameters appear, and both reduce in the classical limit to the
same amplitude $x_{0}$, although only one of them might contribute to the
probability density depending on whether the state we consider is for
particles or antiparticles, or both in the case a superposition of states.

Another remark is that the energy was fixed to $E_{N}$, with
$N=n\pm\frac{1}{2}$ for particles or antiparticles, because of a particular
feature of the Moshinsky model for the Dirac oscillator. The Moshinsky model
does not reproduce the non-relativistic energy values that would be found in
quantum harmonic oscillator from the Schrödinger equation, because of how the
harmonic term is added in the Hamiltonian [32], thus the non-relativistic
limit of the energy spectrum creates a factor $n\hbar\omega$ instead of
$(n+\frac{1}{2})\hbar\omega$ [33].

###  3.4 Dirac particle in a box

Let’s consider the Dirac particle wavefunction, $\psi^{(+)}_{k}(x)$, for a
1-dimensional infinite square Lorentz scalar potential, which does not require
considering the Klein paradox, found explicitly in Ref. [34]. The
corresponding antiparticle solution, $\psi^{(-)}_{k}(x)$, can be calculated as
$\psi^{(-)}_{k}(x)=\gamma^{5}\psi^{(+)}_{-k}(-x)$, where we follow the gamma
matrix representation of Ref. [29] where
$\gamma^{5}=i\gamma^{0}\gamma^{1}\gamma^{2}\gamma^{3}$. Either particle or
antiparticle, having spin up or down, produce a probability density of the
form

$\displaystyle\rho^{RQM}_{k}(x)=$
$\displaystyle\absolutevalue{B_{k}}^{2}\cos^{2}\quantity(kx-\frac{\delta_{k}}{2})$
(14)
$\displaystyle+\absolutevalue{B_{k}}^{2}\Phi_{k}^{2}\sin^{2}\quantity(kx-\frac{\delta_{k}}{2}),$

for the interval $0\leq x\leq L$, and the probability density vanishes outside
this interval. We denote
$\absolutevalue{B_{k}}^{2}\equiv\quantity[\frac{\Phi_{k}^{2}-1}{4k}(2kL-\sin(kL+\delta_{k})-\sin\delta_{k})+L]^{-1}$,
$k\equiv\frac{1}{\hbar}\sqrt{\frac{E_{k}^{2}}{c^{2}}-m^{2}c^{2}}$,
$\Phi_{k}\equiv\frac{\hbar kc}{E_{k}+mc^{2}}$ and
$\delta_{k}\equiv\arctan\quantity(\frac{2\Phi_{k}}{\Phi_{k}^{2}-1})$. The
energy of the particle $E_{k}$ can be found from the condition
$\tan(kL)=-\frac{\hbar k}{mc^{2}}$ [34]. As in Eq. (8) for infinite well from
Klein-Gordon, the asymptotic limit the Fourier transform of the probability
density reduces to

$f^{RQM}_{k}(p)\sim(1+\Phi_{k}^{2})\absolutevalue{B_{k}}^{2}\frac{i\hbar}{2p}\quantity(e^{-i\frac{Lp}{\hbar}}-1).$
(15)

The inverse Fourier transform results into

$\rho^{RQM}_{k}(x)\sim\frac{1+\Phi_{k}^{2}}{2}\absolutevalue{B_{k}}^{2}\quantity[H(L-x)-H(-x)].$
(16)

It should be noticed that the terms $\Phi_{k}^{2}$ and
$\absolutevalue{B_{k}}^{2}$ of the probability density appear as fermionic
quantum parameters, given that Eq. (9) for the Klein Gordon system does not
contain similar factors.

The non-relativistic limit can be stated as the condition such that $\hbar
k\ll mc$, which implies that $\Phi_{k}\rightarrow 0$ and
$\absolutevalue{B_{k}}^{2}\rightarrow 2/L$. The probability density becomes

$\rho^{QM}_{k}(x)\simeq\frac{1}{L}\quantity[H(L-x)-H(-x)].$ (17)

The resulting probability distribution shows invariance under speed boost of
the particle, as it was already discussed for the Klein-Gordon particle in a
box. It should be observed that the non-relativistic asymptotic probability
density of the Klein-Gordon and Dirac particles do not yield a series of
quantum corrections because the asymptotic term has no dependence on $\hbar$,
which is also found in the non-relativistic case of Ref. [1]. As a remark, the
analytic solutions of the Dirac equation for a box potential, that is time
component of a vector, and which considers the Klein paradox problem, are
given in Ref. [35], and can be also be treated with the previous procedure.

##  4 General remarks

The proposed extension of the Bohr’s correspondence principle allowed us to
derive particular classical distributions from relativistic quantum mechanical
ones. The procedure applied to the Klein-Gordon equation (a relativistic
theory) was very similar to Ref. [1] where the classical probability
distribution is recovered through the Schrödinger equation with infinite well
and the quantum harmonic oscillator potentials, with the difference lying on
the energy spectrum, which in turn modifies the probability distribution when
it has dependence on the particle’s velocity. Our results show that the
mathematical procedure proposed in this letter is applicable to the Dirac
equation, and that the big component of the Dirac spinor is the only
contribution that leads to the non-relativistic probability density after
implementing the respective limit.

Whether we considered particle or antiparticle solutions, the same classical
single particle picture plus a series of quantum corrections is found after
applying the non-relativistic limit, with results that coincide with those in
[1]. However if this limit is not considered, the relativistic asymptotic
probability density shows manifest difference between the particle and
antiparticle probability distributions, and the dependence on the $c$ constant
leads to relativistic corrections, which should be experimentally verified in
a future research.

We thank Mauricio Paulin and Alberto Ruiz for their useful comments and
suggestions. S.E. Aguilar acknowledges the financial support by FAPESP under
funding Grant No. 2018/10788-8.

## References

  * [1] J. Bernal, A. Martín-Ruíz, and J. C. García-Melgarejo, “A simple mathematical formulation of the correspondence principle,” Journal of Modern Physics, vol. 4, no. 1, pp. 108–112, 2013.
  * [2] A. J. Makowski and K. J. Górska, “Bohr’s correspondence principle: The cases for which it is exact,” Phys. Rev. A, vol. 66, p. 062103, Dec 2002.
  * [3] L. E. Ballentine, “Quantum-to-classical limit in a hamiltonian system,” Phys. Rev. A, vol. 70, p. 032111, Sep 2004.
  * [4] L. E. Ballentine, Y. Yang, and J. P. Zibin, “Inadequacy of ehrenfest’s theorem to characterize the classical regime,” Phys. Rev. A, vol. 50, pp. 2854–2859, Oct 1994.
  * [5] X. Y. Huang, “Correspondence between quantum and classical descriptions for free particles,” Phys. Rev. A, vol. 78, p. 022109, Aug 2008.
  * [6] A. J. Bracken and J. G. Wood, “Semiquantum versus semiclassical mechanics for simple nonlinear systems,” Phys. Rev. A, vol. 73, p. 012104, Jan 2006.
  * [7] A. J. Makowski, “Exact classical limit of quantum mechanics: Central potentials and specific states,” Phys. Rev. A, vol. 65, p. 032103, Jan 2002\.
  * [8] A. J. Makowski, “Exact classical limit of quantum mechanics: Noncentral potentials and ermakov-type invariants,” Phys. Rev. A, vol. 68, p. 022102, Aug 2003.
  * [9] R. W. Zwanzig, “Transition from quantum to ”classical” partition function,” Phys. Rev., vol. 106, pp. 13–15, Apr 1957.
  * [10] L. E. Ballentine and S. M. McRae, “Moment equations for probability distributions in classical and quantum mechanics,” Phys. Rev. A, vol. 58, pp. 1799–1809, Sep 1998.
  * [11] L. E. Ballentine, “Quantum-to-classical limit of a dynamically driven spin,” Phys. Rev. A, vol. 47, pp. 2592–2600, Apr 1993.
  * [12] M. A. Doncheski and R. W. Robinett, “Comparing classical and quantum probability distributions for an asymmetric infinite well,” European Journal of Physics, vol. 21, no. 3, pp. 227–251, 2000.
  * [13] M. N. Berberan-Santos, E. N. Bodunov, and L. Pogliani, “Classical and quantum study of the motion of a particle in a gravitational field,” Journal of Mathematical Chemistry, vol. 37, no. 2, pp. 101–115, 2005.
  * [14] R. L. Liboff, “The correspondence principle revisited,” Physics Today, vol. 37, no. 2, pp. 50–55, 1984.
  * [15] J. H. van Vleck, “The absorption of radiation by multiply periodic orbits, and its relation to the correspondence principle and the rayleigh-jeans law. part ii. calculation of absorption by multiply periodic orbits,” Phys. Rev., vol. 24, pp. 347–365, Oct 1924.
  * [16] F. C. Hoyt, “Application of the correspondence principle to relative intensities in series spectra,” Phys. Rev., vol. 26, pp. 749–760, Dec 1925\.
  * [17] J. A. West, Z. D. Gaeta, and C. R. Stroud, “Classical limit states of the helium atom,” Phys. Rev. A, vol. 58, pp. 186–195, Jul 1998.
  * [18] P. W. Ketchum, “An extension of bohr’s correspondence principle to apply to small quantum numbers,” Phys. Rev., vol. 24, pp. 463–465, Nov 1924.
  * [19] B. Gao, “Breakdown of bohr’s correspondence principle,” Phys. Rev. Lett., vol. 83, pp. 4225–4228, Nov 1999.
  * [20] C. Eltschka, H. Friedrich, and M. J. Moritz, “Comment on “breakdown of bohr’s correspondence principle”,” Phys. Rev. Lett., vol. 86, pp. 2693–2693, Mar 2001.
  * [21] L. V. Prokhorov, “Limit $\hbar\rightarrow 0$ in quantum field theory,” Phys. Rev., vol. 183, pp. 1515–1517, Jul 1969.
  * [22] L. P. Staunton and S. Browne, “Classical limit of relativistic positive-energy theories with intrinsic spin,” Phys. Rev. D, vol. 12, pp. 1026–1037, Aug 1975.
  * [23] G. R. Shin and J. Rafelski, “Relativistic classical limit of quantum theory,” Phys. Rev. A, vol. 48, pp. 1869–1874, Sep 1993.
  * [24] L. S. Brown and R. L. Goble, “Soft photons and the classical limit,” Phys. Rev., vol. 173, pp. 1505–1516, Sep 1968.
  * [25] P. Stehle and P. G. DeBaryshe, “Quantum electrodynamics and the correspondence principle,” Phys. Rev., vol. 152, pp. 1135–1139, Dec 1966.
  * [26] G. C. Dente, “Classical limit of quantum electrodynamics,” Phys. Rev. D, vol. 12, pp. 1733–1738, Sep 1975.
  * [27] A. Martín-Ruíz, J. Bernal, A. Frank, and A. Carbajal-Domínguez, “The classical limit of the quantum kepler problem,” Journal of Modern Physics, vol. 4, no. 6, pp. 818–822, 2013.
  * [28] A. Martín-Ruíz, J. Bernal, and A. Carbajal-Domínguez, “Macroscopic quantum behavior of periodic quantum systems,” Journal of Modern Physics, vol. 5, no. 1, pp. 44–50, 2014.
  * [29] J. J. Sakurai and J. Napolitano, Modern quantum mechanics.  Pearson Harlow, 2014.
  * [30] N. A. Rao and B. A. Kagali, “Energy profile of the one-dimensional klein–gordon oscillator,” Physica Scripta, vol. 77, no. 1, p. 015003, 2008.
  * [31] I. S. Gradshteyn and I. M. Ryzhik, Table of Integrals, Series and Products.  Academic Press, 8 ed., 2015.
  * [32] M. Moshinsky and A. Szczepaniak, “The dirac oscillator,” Journal of Physics A, vol. 22, no. 17, pp. 817–819, 1989.
  * [33] R. Szmytkowski and M. Gruchowski, “Completeness of the dirac oscillator eigenfunctions,” Journal of Physics A: Mathematical and General, vol. 34, no. 23, pp. 4991–4997, 2001.
  * [34] P. Alberto, C. Fiolhais, and V. M. S. Gil, “Relativistic particle in a box,” European Journal of Physics, vol. 17, no. 1, p. 19, 1996.
  * [35] A. D. Alhaidari, “Dirac particle in a square well and in a box,” AIP Conf. Proc., vol. 1370, no. 1, pp. 21–25, 2011.