#  FerKD: Surgical Label Adaptation for Efficient Distillation

Zhiqiang Shen
Mohamed bin Zayed University of AI
<EMAIL_ADDRESS>
###### Abstract

We present FerKD, a novel efficient knowledge distillation framework that
incorporates partial soft-hard label adaptation coupled with a region-
calibration mechanism. Our approach stems from the observation and intuition
that standard data augmentations, such as RandomResizedCrop, tend to transform
inputs into diverse conditions: easy positives, hard positives, or hard
negatives. In traditional distillation frameworks, these transformed samples
are utilized equally through their predictive probabilities derived from
pretrained teacher models. However, merely relying on prediction values from a
pretrained teacher, a common practice in prior studies, neglects the
reliability of these soft label predictions. To address this, we propose a new
scheme that calibrates the less-confident regions to be the context using
softened hard groundtruth labels. Our approach involves the processes of hard
regions mining + calibration. We demonstrate empirically that this method can
dramatically improve the convergence speed and final accuracy. Additionally,
we find that a consistent mixing strategy can stabilize the distributions of
soft supervision, taking advantage of the soft labels. As a result, we
introduce a stabilized SelfMix augmentation that weakens the variation of the
mixed images and corresponding soft labels through mixing similar regions
within the same image. FerKD is an intuitive and well-designed learning system
that eliminates several heuristics and hyperparameters in former FKD solution
[37]. More importantly, it achieves remarkable improvement on ImageNet-1K and
downstream tasks. For instance, FerKD achieves 81.2% on ImageNet-1K with
ResNet-50, outperforming FKD and FunMatch by remarkable margins. Leveraging
better pre-trained weights and larger architectures, our finetuned ViT-G14
even achieves 89.9%. Our code is available at
https://github.com/szq0214/FKD/tree/main/FerKD.

Figure 1: Illustration of motivation for FerKD. The left figure depicts the
original input, and the middle figure shows the center points of bounding
boxes generated using RandomResizedCrop. The radius of each circle corresponds
to the area of the bounding box. It can be observed that the center points of
the bounding boxes are concentrated in the center of the image, and their area
increases as they approach the center. The right figure displays several top
and bottom confident bounding boxes and their corresponding predictive
probabilities from a pre-trained teacher or teachers ensemble. The proposed
hard region calibration strategy is established based on these predictions.

##  1 Introduction

Knowledge Distillation (KD) [13] has achieved impressive results in various
visual domains, including image classification [52, 36, 5, 37], object
detection [6, 46, 11, 8, 53] and semantic segmentation [22, 15, 16]. However,
KD methods are often computationally expensive and inefficient due to the
additional computational burden imposed by the teacher models. The primary
advantage of KD that motivates its usage is its ability to generate precise
soft labels that convey more informative details about the input examples. It
differs from other label softening techniques, such as label smoothing [43],
Mixup [58], and CutMix [56], mainly in two aspects: (1) KD generates soft
labels dynamically in each iteration, which is more informative than fixed
smoothing patterns used in label smoothing; (2) Mixup and CutMix techniques
essentially combine hard labels with coefficients, while KD produces soft
labels that are highly correlated with the input sample. This allows KD’s soft
labels to become more accurate when different data augmentations, such as
RandomResizedCrop, flipping and rotation, color jittering, etc., are applied.
In general, mixing-based label softening methods cannot monitor such changes
in input content, but KD can address them effortlessly.

To overcome the computational inefficiency of traditional knowledge
distillation, FKD [37] was developed to generate region-level soft labels in
advance and reuse them across multiple training cycles to eliminate redundant
computation. This approach only requires the preparation of soft labels once
at the beginning and they can be reused indefinitely. However, this approach
overlooks certain critical issues. One is the quality of the soft labels. When
using RandomResizedCrop to generate regions, some may be cropped from
background areas, and the teacher model will still produce a soft label for
them based on their similarity to the dataset classes. However, in some cases,
these areas may contain irrelevant noise, compensatory information, or context
information for the class, and the soft labels may not accurately reflect the
context of information they carry. To address this problem, this work proposes
to recalibrate these soft labels by incorporating context information from
hard ground-truth labels with smoothing.

Furthermore, due to the random nature of the sampling process, a certain
proportion of crops that are either excessively easy or difficult do not
contribute to the model’s learning capacity. As demonstrated in Fig. 2 and
Table 1, these samples can be discarded to expedite the convergence process.
The pre-generated soft labels can be utilized as useful indicators to select
these specific samples. In our adaptation of surgical soft labeling, we
categorize the soft labels into four distinct groups: extreme hard (negative),
moderate hard (background or context), hard positive (partial object), and
easy positive. Each of these categories is subject to different treatment
methodologies.

Figure 2: Statistics of soft label max-probability for crops on ImageNet-1K.
The soft label is from FKD [37]. In each image, 500 regions are randomly
cropped.

The Role of Background. The role of the background in images is essential, as
it provides critical context and spatial information that aids the model in
accurately identifying objects of interest within the scene. Backgrounds can
vary in complexity and structure, ranging from simple monochromatic
backgrounds to highly cluttered and detailed ones. Soft labels in background
areas are typically low, and therefore, it is crucial to handle the background
carefully with precise supervision to achieve higher model capability within
our surgical label calibration framework.

Hard Regions Mining and Calibration. Hard Regions Mining involves the
identification and isolation of challenging or complex regions within an image
that the model struggles to identify accurately. These regions can include
objects with complex shapes, occlusions, or those with low contrast. By
identifying these regions, the model can focus on learning the features and
characteristics of these regions, resulting in improved performance.
Calibration, on the other hand, involves adjusting the confidence levels of
the model’s predictions in challenging regions. The model’s predictions may be
less reliable in hard regions, leading to lower confidence scores. Calibrating
the predictions can improve the model’s accuracy in these regions by adjusting
the confidence levels of the predictions. We found that carefully discarding a
portion of negative crops and selecting those hard positive crops by
calibrating their labels, can force the training process more efficient and
effective.

Stable Training on Soft Labels. Mixture-based augmentations, such as Mixup and
CutMix have seen widespread use for training models under hard supervision,
where each image is labeled with a single class label. However, in the soft
label scenario, we have made a different observation: when employed together
with pre-generated soft labels on a typical ResNet, Mixup and CutMix tend to
be overly strong, which, conversely, leads to decreased performance. To
mitigate label fluctuations and achieve more stable training, we propose a
SelfMix scheme, which is particularly suitable for cases where data
augmentation should not be so strong, such as in finetuning distillation,
where mixture-based augmentation is usually disabled. On the other hand, when
training ViT models from scratch, stronger data augmentation can yield better
results [45, 42], which is consistent with the larger capacity perspective of
this type of network.

range ($P$) | ratio | range ($P$) | agg. ratio
---|---|---|---
$[0.0,0.1)$ | 0.43% | $[0,0.1)$ | 0.43%
$[0.1,0.2)$ | 0.89% | $[0,0.2)$ | 1.32%
$[0.2,0.3)$ | 1.29% | $[0,0.3)$ | 2.61%
$[0.3,0.4)$ | 2.03% | $[0,0.4)$ | 4.64%
$[0.4,0.5)$ | 3.66% | $[0,0.5)$ | 8.31%
$[0.5,0.6)$ | 4.35% | $[0,0.6)$ | 12.65%
$[0.6,0.7)$ | 5.04% | $[0,0.7)$ | 17.69%
$[0.7,0.8)$ | 7.76% | $[0,0.8)$ | 25.45%
$[0.8,0.85)$ | 8.14% | $[0,0.85)$ | 33.59%
$[0.85,0.9)$ | 28.73% | $[0,0.9)$ | 62.32%
$[0.9,0.95)$ | 37.34% | $[0,0.95)$ | 99.67%
$[0.95,1.0)$ | 0.33% | $[0,1.0)$ | 100%

Table 1: Detailed statistics of soft labels. “range” indicates max-probability
of crops, “ratio” indicates the percentage in the whole crops. “agg. ratio” is
the aggregated ratio.

In summary, our contributions of this work are:

\- We present FerKD, a sample-calibration framework for Faster Knowledge
Distillation that achieves state-of-the-art performance. We conduct extensive
analysis, ablation, and discussion on the impact of hard and easy samples.

\- We make two key observations in the pre-generated soft label training
framework. Firstly, we observe that the few most challenging and simplest
crops obtained through the  RandomResizedCrop operation do not contribute
significantly to the model’s learning and can therefore be removed. Secondly,
we find that moderately hard crops can provide crucial contextual information
that improves the model’s ability to learn robust representations.

\- We perform extensive experiments on ImageNet-1K and downstream tasks. On
ImageNet-1K, FerKD achieves an accuracy of 81.2% using the ResNet-50. When
leveraging self-supervised pre-trained weights, our larger model fine-tuned
using ViT-G/14 achieves an accuracy of 89.9%.

##  2 Related Work

Knowledge Distillation and Fast Knowledge Distillation. Knowledge Distillation
[13] is a learning method in which a “student” model is trained to imitate the
predictions of a larger, more complex “teacher” model. A key advantage of this
approach is that the teacher model can provide soft supervision that contains
more information regarding the input data than traditional one-hot human
annotated labels, particularly when the input data is subject to data
augmentation. There have been many recent variants and extensions of knowledge
distillation [31, 26, 47, 59, 25, 27, 35, 7, 41, 50, 45, 18, 51], including
approaches that use internal feature representations [7], adversarial training
with discriminators [33], transfer learning techniques [54], fast distillation
[37] via instance label preparation, and methods that prioritize patient and
consistent learning [5].

Hard Sample Mining. The aim of hard sample mining [23, 38] is to enhance the
performance of learning models by selectively focusing on challenging examples
that are typically difficult to classify. By prioritizing hard samples during
training, models can be trained to better handle a wider range of real-world
scenarios and improve their overall performance. One approach [38] to achieve
this for object detection is to use Online Hard Example Mining (OHEM) which
employs a strategy of selecting challenging examples during training of
region-based ConvNet detectors. The motivation behind this approach is that
detection datasets typically comprise an overwhelming number of easy examples
and a small number of hard examples. Automatic selection of these hard
examples can make training more effective and efficient.

Some other techniques that are close to hard sample mining: (1) Curriculum
learning [3, 48, 40]: it trains a model on easy examples first and then
gradually increasing the difficulty of the examples over time. (2) Active
learning [32, 24, 28]: it selects the most informative or uncertain samples
for labeling by a human annotator. By focusing on the samples that the model
is most uncertain about, the model can learn to better generalize and improve
its performance on difficult samples. (3) Loss functions: Attentive loss
functions can be used to emphasize the importance of hard samples during
training. For example, focal loss [20] places more weight on the difficult
examples during training, helping the model to learn to handle them better.

Data Augmentations. Several studies have incorporated data augmentations into
distillation frameworks to improve performance. For instance, FunMatch [5]
employed Mixup, and FKD [37] utilized CutMix. Both techniques achieved
competitive accuracy on large-scale ImageNet-1K dataset. In this work, we
investigate the impact of data augmentation intensity on soft labels. We
discover that different network architectures require unique data augmentation
levels. Specifically, ResNet necessitates mild data augmentation, while ViTs
require stronger data augmentation. However, even for ViTs, finetuning
distillation requires a reduction in the intensity of data augmentation,
especially for mixture-based methods. Motivated by this observation, we
propose a mild SelfMix approach for ResNet and finetuning distillation
scenarios.

##  3 Approach

The proposed FerKD is a soft label calibration framework for fast and
efficient knowledge distillation training. In this section, we aim to provide
an elaborated overview of our method, starting with an in-depth analysis of
the roles of hard and soft labels in the distillation process. We then present
the key components of our approach, which include a region selection strategy
and a soft label calibration scheme. Additionally, we explore the data
augmentation requirements for soft labels and introduce a simple label
ensemble technique to enhance the quality of the soft labels.

###  3.1 Revisiting Hard and Soft Label in Distillation

The utilization of hard and soft labels is dependent on the comprehension,
problem scenarios, and underlying objectives. Various arguments exist
regarding their effectiveness. The vanilla knowledge distillation method [13]
employs both hard and soft labels to maximize the benefits of both. However,
recent researches [36, 35, 57, 37] suggest that the use of hard labels is not
necessary in distillation on large-scale datasets as strong teachers can
provide more precise soft supervision. Incorporating hard labels may introduce
erroneous supervisory signals, ultimately hampering student performance. Other
than the above views, this work presents a novel perspective beyond them by
acknowledging that both hard and soft labels offer unique advantages,
highlights the proper practice of usage that is necessary, and finally
proposes an elegant solution to reap benefits derived from both sources of
hard and soft labels.

Combination of Soft and Hard Labels. In vanilla KD design [13], for each
training example, it will minimize two loss terms for both hard and soft
labels. The final objective $\mathcal{L}_{\text{\em VKD}}$ can be formulated
as:

$\displaystyle\mathcal{L}_{\text{\em
VKD}}\\!=\\!\frac{1}{N}\sum_{x}(\alpha\\!*\\!\underbrace{\mathcal{L}_{h}(p_{\theta}(x),y_{h}(x))}_{\textbf{CE
loss with hard
label}}\\!+(1\\!-\\!\alpha)\\!*\\!\underbrace{\mathcal{L}_{s}(p_{\theta}(x),y_{s}(x))}_{\textbf{KL
loss with soft label}})$ (1)

where $\alpha$ is the coefficient to balance the loss signal intensity from
soft and hard labels. $p_{\theta}(x)$ is the logits prediction from the
student model and $\theta$ is its parameters. $y_{h}$ is the hard label and
$y_{s}$ is the soft label from a pre-trained teacher. $N$ is the total number
of training samples $x$.

Full Soft Label Training. Fast KD [37] proposes to use the soft label solely
since the prediction from strong teachers is precise enough. Thus, the loss
objective $\mathcal{L}_{\text{\em FKD}}$ is:

$\mathcal{L}_{\text{\em
FKD}}=\frac{1}{N}\sum_{x}\underbrace{\hat{\mathcal{L}}_{s}(p_{\theta}(x),y_{s}(x))}_{\textbf{SCE
loss with soft label}}$ (2)

where “SCE” is the soft version of cross-entropy loss.

Figure 3: Illustration of region calibration according to their predictive
probabilities in FerKD. Left is the input image with RandomResizedCrop.
Bounding box is with high probability and bounding box is with low
probability.

###  3.2 FerKD: Surgical Label Calibration Distillation

Surgical/Partial Soft and Hard Label Adaptive Training. Different from VKD,
the proposed FerKD strategy will only involve one loss term but also unlike
FKD, FerKD will employ both hard and soft labels in a single objective term
and exploit the additional information derived from both sources. The solution
is that we only keep their original soft labels for positive regions since
they contain fine-grained information regarding the crops, for those
background or context regions, we will re-calibrate them by the human-
annotated ground-truth labels to avoid misinformation from the soft labels.
Hence, the loss function will be:

$\mathcal{L}_{\text{\em
FerKD}}=\frac{1}{N}\sum_{x}\underbrace{\mathcal{L}_{{\texttt{adap}:\ \text{h}\
\text{or}\ \text{s}?}}(p_{\theta}(x),y_{a}(x))}_{\textbf{SCE loss with
hard/soft label}}$ (3)

where “adap:” indicates$\ \\{{hard}\\}\ \text{or}\ \\{{soft}\\}$ used for
individual training samples. $y_{a}$ is the calibrated soft labels. As
illustrated in Fig. 3, we will calibrate regions’ soft labels $y_{a}$ using
the following rule:

$y_{a}=\left\\{\begin{array}[]{ll}\textbf{UR}:\text{discarded}&\text{ if
}y_{s}<\mathcal{T}_{L}\ or\ y_{s}>\mathcal{T}_{T}\\\
\textbf{HR}:\bm{1.0-\varepsilon}&\text{ if
}\mathcal{T}_{L}<y_{s}<\theta_{M}\\\ \textbf{IR}:y_{s}&\text{ otherwise
}\end{array}\right.$ (4)

where $\mathcal{T}_{L}$, $\mathcal{T}_{M}$, and $\mathcal{T}_{T}$ are
thresholds at low, middle, and top boundaries. “UR” represents the
uninformative regions such as black or white blocks in an image that will be
discarded during training. As shown in Fig. 4, “HR” represents the hard
regions with a smoothing value $\bm{\varepsilon}$ and “IR” represents the
important regions. Thus, the key goal in FerKD becomes to identify the
positive or background regions. Thanks to FKD [37], we can have access to all
crops’ individual predictions. A quick exploration is performed for
verification and the result is shown in Fig. 5. It is clear that by discarding
a certain ratio of samples (hardest and easiest), the performance is
consistently improved.

Figure 4: Illustration of region calibration according to their predictive
probabilities in FerKD. Left is the input image with RandomResizedCrop. Right
is the rule for calibrating the probabilities of regions.

Figure 5: Minimal and maximal probability. The upper figure indicates that
only regions having the max probability in $[\text{minimal},1.0]$ will be
trained, and baseline indicates that the model is trained with all randomly
sampled regions. The bottom figure indicates that only regions having the max
probability in $[0.3,\text{maximal}]$ will be trained, and baseline indicates
that the model is trained with regions in $[0.3,1.0]$.

###  3.3 Training Speed of FerKD

When training a model, easy examples allow the model to quickly learn the
patterns in the data and update its parameters in a way that minimizes the
loss function. This means that the model will converge faster and require
fewer iterations to reach a satisfactory level of accuracy. On the other hand,
hard examples can slow down the convergence speed but they can force models to
learn more robust classify boundaries. The speed of model convergence can be
impacted significantly by the sampling strategy of hard and easy examples. In
our FerKD framework, “hard” and “easy” examples refer to the level of
difficulty a particular sample presents to a model. This difficulty can be
quantified by the probability assigned to the correct label by the teacher. An
“easy” sample is one where the probability assigned to the correct label is
high, indicating that the teacher is confident in its prediction. On the other
hand, a “hard” sample is one where the probability assigned to the correct
label is low, indicating that the teacher is uncertain about its prediction.
The threshold for what constitutes an “easy” or “hard” sample can vary
depending on the specific task and model being used, which is the key for
exploring in FerKD.

Figure 6: Illustration of the testing accuracy curves. “300” represents the
training with the full budget. “50, 100, 150, 200” are the training with
reduced budgets.

Our sampling strategy involves the removal of the easiest and hardest training
examples to reduce computational costs incurred during uninformative training
steps. This strategy offers a clear benefit, as shown in Fig. 6, which
presents the test accuracy curves for different training budgets. The results
show that, with a training budget reduced by $\frac{2}{3}$ (200 epochs), the
achieved accuracy is comparable to that of full-budget training. Although
further reducing the budget slightly affects accuracy, our proposed FerKD
method is shown to be robust across different training budgets. Moreover, our
calibration process will not involve additional training cost since it can be
done offline in advance.

###  3.4 SelfMix: A Mild and Stable Data Augmentation for Soft Labels

Figure 7: Illustration of the high-level outline for SelfMix to make mixed
soft label consistent, reducing the variance. Practically, each label will be
a soft distribution for the input instead of the hard label as illustrated,
thus it is more moderate on supervision in training. Empirically, it is
observed that this strategy is crucial for ConvNet like ResNet but ViT needs
intense data augmentations as introduced in [45].

Soft label Calibration with SelfMix. The current prevailing data augmentation
techniques are designed for hard labels or smoothed hard labels. However, in
the dynamic soft label scenario, a different approach is necessary to meet the
unique attributes of pre-generated soft labels. Soft labels themselves can
mitigate overfitting, making it imperative to develop tailored data
augmentation techniques that account for soft labels to achieve improved
accuracy. The proposed SelfMix is based on the empirical observations
presented in Table 2, which indicate that strong data augmentation does not
necessarily improve accuracy if the network is already saturated and may even
hurt performance. Consequently, we aim to reduce the intensity of data
augmentation while still benefiting from its effects. To achieve this goal, we
redesign the data augmentation using a self-mixing approach, as illustrated in
Fig. 7.

The specific steps involved in the SelfMix process are illustrated in Fig. 8.
Mixing operations are exclusively conducted within each individual image to
minimize variations between mixed images and their corresponding mixed soft
labels. This straightforward constraint yields significant improvements in the
performance of the ResNet backbone and in finetuning distillation.

Figure 8: Illustration of the detailed SelfMix augmentation for FerKD. In this
strategy, mixture operation only happens within the same image in a mini-batch
and cross-image mixing is disabled to preserve stability of soft labels.

Ensemble Supervisions. It has been demonstrated that ensembling more teachers
[13, 33] can enhance the performance of distilled students. In this work, we
apply this approach to improve the quality of pre-existing soft labels. As
these soft labels are quantized for efficient storage [37], they must be
recovered to their full dimension prior to calibration and averaging for final
supervision. Specifically, we define the ensemble soft labels $y_{\text{\em
en}}=\frac{1}{M}\sum_{t\in\text{teachers}}{\hat{y}^{t}_{a}(x)}$, where
$\hat{y}^{t}_{a}$ represents the recovered soft label with calibration from
teacher $t$, $x$ is the input and $M$ is the number of teachers.

Mixup | CutMix | SelfMix (Ours) | ResNet50 | ViT-S/16
---|---|---|---|---
✓ |  |  | 78.94 | 80.32
| ✓ |  | 79.52 | 80.95
|  | ✓ | 80.47 | 79.29
✓ | ✓ |  | 79.92 | 81.16
✓ | ✓ | ✓ | 80.26 | 80.69

Table 2: Top-1 accuracy of data augmentations on different backbones. On ViT,
the soft label can benefit from combinations of more mixture operations. The
backbones are ResNet-50 and ViT-S/16 and the teacher model is
EfficientNet_L2_475. We run three trials and report the means.

##  4 ImageNet Experiments

We conduct training on the ImageNet-1K (IN1K) [9] training set. We report
top-1 validation accuracy of a single 224$\times$224 crop. The default
training budget is 300 epochs, and the temperature for both teacher and
student is 1.0. The finetuning distillation settings follow their individual
designs. All our soft label generation and model training are performed on the
A100-GPU High-Flyer cluster with 80GB on each. More details are provided in
Appendix.

Baselines: FKD [37] and FunMatch [5]. Regions are randomly sampled in these
two approaches. We use ResNet-50 and ViT-S/16 as the backbones for the
ablation study. FKD + Curriculum Sampling: (i) we sample regions for training
from easier ones and gradually increase the level of difficulty. (ii) In
contrast, we sample regions from hard ones and gradually decrease the level of
difficulty.

Teacher |  $\text{T}_{\text{Top-1}}$ (%) |  $\text{S}_{\text{ResNet50}}$ (%) |  $\text{S}_{\text{ViT-S/16}}$ (%)
---|---|---|---
Effi_L2_475 [52] | 88.14 | 80.23 | 81.16
Effi_L2_800 [52] | 88.39 | 80.16 | 81.30
RegY_128GF_384 [39] | 88.24 | 80.34 | 81.42
ViT_L16_512 [39] | 88.07 | 80.29 | 81.43
ViT_H14_518 [39] | 88.55 | 80.18 | 81.41
BEIT_L_224 [2] | 87.52 | 80.03 | 81.16
BEIT_L_384 [2] | 88.40 | 80.06 | 81.11
BEIT_L_512 [2] | 88.60 | 80.09 | 81.07
ViT_G14_336_30M [10] | 89.59 | 79.03 | 79.62
ViT_G14_336_CLIP [10] | 89.38 | 79.59 | 79.48

Table 3: Top-1 accuracy of distillation on ImageNet-1K using a singe ConvNet
or Vision Transformer teacher. Note that Mixup [58] and CutMix [56] are used
for training ViT-S/16. We run three trials and report the means.

###  4.1 Soft Label from Different Teachers

The soft labels produced by distinct teacher models for the same input image
may vary owing to differences in their distinctive features, architectures,
and training strategies. In this section, our objective is to determine which
teacher model has the greatest capacity to distill a student. We evaluate two
types of student models: ResNet-50 and ViT-S/16. The results are shown in
Table 3, where RegY_128GF_384 and ViT_L16_512 achieve the highest student
accuracy individually, despite not being the best on their own.

calibr. range | [0, 0.2] | [0, 0.3] | [0, 0.4] | [0, 0.5] | [0,0.6]
---|---|---|---|---|---
Top-1 | 80.31 | 80.42 | 80.14 | 79.84 | 79.66

Table 4: Ablation of calibration for different probability ranges. The base
model is the single teacher FerKD using ResNet-50 without SelfMix data
augmentation.

Pre-train | Top-1
---|---
vanilla | 80.23
+calibration&selfmix |  80.68+0.45
+multi-teacher ensemble |  81.15+0.47
+more epochs |  81.44+0.29

Table 5: Ablation results using ResNet-50 on ImageNet-1K.

Effi_L2_475 | Effi_L2_800 | RegY_128GF_384 | ViT_L16_512 | ViT_H14_518 | BEIT_224 | BEIT_384 | BEIT_512 | Student Acc.
---|---|---|---|---|---|---|---|---
✓ | ✓ |  |  |  |  |  |  | 80.23
✓ |  | ✓ |  |  |  |  |  | 80.52
✓ | ✓ | ✓ |  |  |  |  |  | 80.49
|  |  |  | ✓ |  |  | ✓ | 80.29
|  |  | ✓ | ✓ |  |  | ✓ | 80.53
|  |  | ✓ | ✓ | ✓ | ✓ | ✓ | 80.48
✓ |  |  | ✓ |  |  |  |  | 80.35
✓ |  |  |  | ✓ |  |  |  | 80.51
✓ |  |  |  |  |  |  | ✓ | 80.35
| ✓ |  |  |  |  |  | ✓ | 80.38
| ✓ |  |  | ✓ |  |  |  | 80.51
|  | ✓ | ✓ |  |  |  |  | 80.53
|  | ✓ | ✓ | ✓ |  |  |  | 80.52
✓ |  | ✓ | ✓ |  |  |  |  | 80.62
✓ |  | ✓ | ✓ | ✓ |  |  |  | 80.74

Table 6: Ablation top-1 accuracy of teacher ensemble on ImageNet-1K with
ConvNet, Vision Transformer or hybrid teachers. The left group is ConvNet
teachers, the middle is the ViT teachers and the right group is the
corresponding student ResNet-50 accuracy. Note that surgical calibration and
SelfMix are not used here.

###  4.2 Ablations

Ablation on Calibration. The results for different calibration ranges are
shown in Table 4, it can be observed that [0, 0.3] achieves the best accuracy.
In practice, we discard examples in [0, 0.15) and (0.95, 1.0], meanwhile,
calibrate examples in [0.15, 0.3] for the final strategy. More visualization
of the hardest and easiest regions is shown in Fig. 9.

Ablation on teacher ensemble and training budget. The results are in Table 5,
showing that each design has the consistent improvement. Our final results are
shown in Table 8, FerKD performs the best over other SOTA methods.

###  4.3 Curriculum Distillation

In curriculum distillation, the student model learns from the teachers’
knowledge in a sequential manner, starting from easier examples (high
probability regions) and gradually increasing the level of difficulty. This
allows the student model to learn from easier to more complex regions. By
presenting samples in a curriculum, the student model can learn from the
easier samples and build a strong foundation before being exposed to more
challenging samples. However, our experimental results, as shown in Table 7,
demonstrate that this curriculum learning approach is inferior to our surgical
label adaptation strategy, FerKD. Our approach outperforms curriculum
distillation by 0.8%. We attribute this performance improvement to the strong
ability of soft labels to mitigate overfitting and improve generalization in
the initial stages of training. As a result, the advantage of the curriculum
learning approach is not as apparent in this scenario. We also notice that
“e-to-h” performs slightly better than “h-to-e” demonstrating the
effectiveness of curriculum strategy, while both of them are inferior to the
random sampling baseline.

Method | sampling | Top-1
---|---|---
Random (FKD [37]) | random | 80.2
curriculum distillation |  h-to-e | 79.7
curriculum distillation |  e-to-h | 79.9
FerKD (Ours) | surgical | 80.7

Table 7: Curriculum and surgical (no ensemble) distillation. “e-to-h” refers
to curriculum sampling from easy to hard regions. “h-to-e” refers to sampling
from hard to easy.

Model | Epoch | Label | Top-1
---|---|---|---
Timm [49] | 600 | Hard${}_{\text{LS}}$ | 80.4
Pytorch (advanced) [44] | 600 | Hard${}_{\text{LS}}$ | 80.9
MEAL [33] |  100‡ | Soft | 78.2
MEAL V2 [36] |  180‡ | Soft | 80.7
MEAL V2 [36]${}_{\text{w/ CutMix}}$ |  180‡ | Soft | 81.0
ReLabel [57]${}_{\text{w/ CutMix}}$ | 300 | Soft | 80.2
FunMatch [5]${}_{\text{w/ Mixup}}$ | 300 | Soft | 80.5
FKD [37] | 300 | Soft | 80.5
FerKD (Ours) | 300 | Adap |  81.2+0.7
FerKD (Ours) | 600 | Adap |  81.4+0.9

Table 8: Top-1 accuracy on ImageNet-1K dataset. The backbone network in this
table is ResNet-50. ${\ddagger}$ indicates that the model was fine-tuned from
hard-label pre-trained weights, resulting in a total training epoch of “300 +
$\times$”.

###  4.4 Finetuning Distillation

Finetuning distillation [36] has been demonstrated as an effective approach to
improve the accuracy of knowledge distillation (KD) frameworks. In the case of
hard labels, multiple finetuning schemes have been proposed to adapt the model
parameters to fit the target dataset, including partial finetuning on selected
intermediate layers with varying learning rates [34, 1, 17, 30] or on the last
few layers [55]. In this work, we adopt the MEAL V2 [36] protocol by
finetuning the entire network from the pre-trained weights to evaluate the
effectiveness of distillation with surgical soft label calibration as the
objective. Specifically, we employ three types of pre-trained models:

(1) Supervised pre-train: ResNet-50 on Timm [49];

(2) Weakly-supervised pre-train: RegNetY-128GF from SWAG [39];

(3) Self-supervised pre-train: ViT-G14 from EVA [10];

We verify whether FerKD can continue improving fine-tuning distillation. As
shown in Table 9, FerKD achieves consistent improvement across different
architectures.

Pre-train | FT | Top-1
---|---|---
Timm [49] | Hard${}_{\text{LS}}$ | 80.38
Timm [49] | FKD [37]* | 80.62
Timm [49] | FerKD* (Ours) | 81.06
SWAG [39] | Hard${}_{\text{LS}}$ | 87.22
SWAG [39] | FKD [37]† | 87.42
SWAG [39] | FerKD† (Ours) | 87.76
EVA_MIM [10] | Hard${}_{\text{LS}}$ | 89.59
EVA_MIM [10] | FKD [37] | 89.67
EVA_MIM [10] | FerKD (Ours) | 89.86

Table 9: Fine-tuning distillation results using pre-trained ResNet-50 (Timm),
RegNetY-128GF (SWAG) and ViT-G14 (EVA) on ImageNet-1K. † We use the same
recipe of EVA for SWAG finetuning since SWAG did not provide the complete
fine-tuning details. * On ResNet-50, we finetune with 150 epochs for both FKD
and FerKD. Figure 9: Illustration of the location points from
RandomResizedCrop and the identified crops by teacher model for hard and easy
samples. Note that we do not involve any localization information, but
teacher’s probability can reflect it automatically.

###  4.5 Single Teacher vs. Teacher Ensemble

The soft labels generated by different teachers for the same input image can
be different due to their unique features, architectures, and training
strategies. By combining these soft labels from different teachers, the
resulting label becomes more informative and can help the student model learn
better representations. This approach can be particularly useful in scenarios
where the teacher models have complementary strengths or when the input data
is challenging and requires multiple perspectives to be accurately labeled.
Furthermore, the use of multiple teachers can also help mitigate the effects
of overfitting and improve the generalization performance of the student
model. The results are shown in Table 6, ensembling four hybrid teachers of
Effi_L2_475, RegY_128GF_384, ViT_L16_512, and ViT_H14_518 achieves the best
accuracy.

##  5 Transfer Learning Experiments

Object detection and segmentation. We conducted evaluations to investigate
whether the improvement achieved by FerKD on ImageNet-1K can be transferred to
various downstream tasks. Specifically, Table 10 presents the results of
object detection and segmentation on COCO dataset [21] using models pre-
trained on ImageNet-1K with FerKD. We utilize Mask RCNN [12] with FPN [19]
following FKD for the experiment. Our FerKD pre-trained weight consistently
outperforms both the baselines ReLabel and FKD on the downstream tasks.

Method | Label | $\text{AP}^{\text{box}}$ | $\text{AP}^{\text{mask}}$
---|---|---|---
ReLabel [57] | LM | 39.1 | 35.2
FKD [37] | Soft | 39.7 | 35.9
FerKD (Ours) | Adap | 40.2 | 36.3

Table 10: COCO object detection and segmentation using a Mask-RCNN with FPN
baseline. “LM” represents label map during pretraining. The backbone is
ResNet-50-300ep.

Method | iNat 2019224 | iNat 2019336 | Places365224
---|---|---|---
EVA_MIM [10] | 79.9 | 86.6 | 61.0
FerKD (Ours) |  80.3+0.4 |  87.1+0.5 |  61.4+0.4

Table 11: Transfer learning accuracy on various classification datasets. The
input sizes are 224$\times$224 and 336$\times$336.

Method | IN1K | ReaL | ImageNetV2 | ImageNetV2 | ImageNetV2
---|---|---|---|---|---
Top-images | Matched-freq | Threshold-0.7
ResNet-50:
ReLabel [57] | 78.9 | 85.0 | 80.5 | 67.3 | 76.0
FKD [37] | 80.1 | 85.8 | 81.2 | 68.2 | 76.9
FerKD | 81.2 |  86.4+0.6 |  82.1+0.9 |  69.5+1.3 |  77.8+0.9
ViT-G14-336:
EVA [10] | 89.6 | 90.8 | 89.0 | 81.9 | 86.7
FerKD | 89.9 |  91.3+0.5 |  89.4+0.4 |  82.4+0.5 |  87.1+0.4

Table 12: Results of FerKD on ImageNet ReaL [4] and ImageNetV2 [29] with
ResNet-50 and ViT-G14 backbones.

Classification tasks. Table 11 shows the transfer learning result on
iNaturalists [14] and Places [60] datasets. On both of these two datasets, our
results surpass the baseline EVA pretrained model by significant margins.

##  6 Robustness

We provide comparisons on ImageNet ReaL [4] and ImageNetV2 [29] datasets to
examine the robustness of FerKD trained models. On ImageNetV2 [29], we verify
our FerKD models on three metrics “Top-Images”, “Matched Frequency”, and
“Threshold 0.7” following FKD [37]. We perform experiments on two network
structures: ResNet-50 and ViT-G14. The results are shown in Table 12, we
achieve consistent improvement over ReLabel and FKD on ResNet-50
(224$\times$224) and better accuracy than EVA on ViT-G14 (336$\times$336).

##  7 Conclusion

In this work, we have presented a new paradigm of faster knowledge
distillation (FerKD), which employs label adaptation on randomly cropped
regions. The proposed method outperforms existing state-of-the-art
distillation approaches in terms of both training speed and convergence.
Additionally, we make two key observations that could be leveraged in future
studies. Firstly, we notice that the most challenging and easiest few crops
obtained through the RandomResizedCrop operation do not contribute to the
model’s learning and can thus be discarded. Secondly, we find that moderate
hardness crops can provide crucial context information that helps calibrate
the model to learn more robust representations, which in turn benefit
downstream tasks.

## References

  * [1] Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta.  Better fine-tuning by reducing representational collapse.  arXiv preprint arXiv:2008.03156, 2020.
  * [2] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei.  Beit: Bert pre-training of image transformers.  In ICLR, 2022.
  * [3] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston.  Curriculum learning.  In ICML, 2009.
  * [4] Lucas Beyer, Olivier J Hénaff, Alexander Kolesnikov, Xiaohua Zhai, and Aäron van den Oord.  Are we done with imagenet?  arXiv preprint arXiv:2006.07159, 2020.
  * [5] Lucas Beyer, Xiaohua Zhai, Amélie Royer, Larisa Markeeva, Rohan Anil, and Alexander Kolesnikov.  Knowledge distillation: A good teacher is patient and consistent.  In CVPR, 2022.
  * [6] Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker.  Learning efficient object detection models with knowledge distillation.  NeurIPS, 30, 2017.
  * [7] Inseop Chung, SeongUk Park, Jangho Kim, and Nojun Kwak.  Feature-map-level online adversarial knowledge distillation.  In ICML, 2020.
  * [8] Xing Dai, Zeren Jiang, Zhao Wu, Yiping Bao, Zhicheng Wang, Si Liu, and Erjin Zhou.  General instance distillation for object detection.  In CVPR, 2021.
  * [9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [10] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao.  Eva: Exploring the limits of masked visual representation learning at scale.  arXiv preprint arXiv:2211.07636, 2022.
  * [11] Jianyuan Guo, Kai Han, Yunhe Wang, Han Wu, Xinghao Chen, Chunjing Xu, and Chang Xu.  Distilling object detectors via decoupled features.  In CVPR, 2021.
  * [12] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In ICCV, 2017.
  * [13] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [14] Grant Van Horn, Oisin Mac Aodha, Yang Song, Alexander Shepard, Hartwig Adam, Pietro Perona, and Serge J. Belongie.  The inaturalist species classification and detection dataset.  In CVPR, 2018.
  * [15] Yuenan Hou, Xinge Zhu, Yuexin Ma, Chen Change Loy, and Yikang Li.  Point-to-voxel knowledge distillation for lidar semantic segmentation.  In CVPR, 2022.
  * [16] Deyi Ji, Haoran Wang, Mingyuan Tao, Jianqiang Huang, Xian-Sheng Hua, and Hongtao Lu.  Structural and statistical texture knowledge distillation for semantic segmentation.  In CVPR, 2022.
  * [17] Jaejun Lee, Raphael Tang, and Jimmy Lin.  What would elsa do? freezing layers during transformer fine-tuning.  arXiv preprint arXiv:1911.03090, 2019.
  * [18] Lujun Li.  Self-regulated feature learning via teacher-free feature distillation.  In European Conference on Computer Vision, pages 347–363. Springer, 2022.
  * [19] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In CVPR, 2017.
  * [20] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In ICCV, 2017.
  * [21] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In ECCV, 2014.
  * [22] Yifan Liu, Ke Chen, Chris Liu, Zengchang Qin, Zhenbo Luo, and Jingdong Wang.  Structured knowledge distillation for semantic segmentation.  In CVPR, 2019.
  * [23] Ilya Loshchilov and Frank Hutter.  Online batch selection for faster training of neural networks.  arXiv preprint arXiv:1511.06343, 2015.
  * [24] Joel Michael.  Where’s the evidence that active learning works?  Advances in physiology education, 2006.
  * [25] Rafael Müller, Simon Kornblith, and Geoffrey Hinton.  When does label smoothing help?  In NeurIPS, 2019.
  * [26] Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami.  Distillation as a defense to adversarial perturbations against deep neural networks.  In IEEE symposium on security and privacy (SP), 2016.
  * [27] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho.  Relational knowledge distillation.  In CVPR, 2019.
  * [28] Michael Prince.  Does active learning work? a review of the research.  Journal of engineering education, 93(3):223–231, 2004.
  * [29] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar.  Do imagenet classifiers generalize to imagenet?  In ICML, 2019.
  * [30] Youngmin Ro and Jin Young Choi.  Autolr: Layer-wise pruning and auto-tuning of learning rates in fine-tuning of deep networks.  In AAAI, 2021.
  * [31] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio.  Fitnets: Hints for thin deep nets.  arXiv preprint arXiv:1412.6550, 2014.
  * [32] Burr Settles.  Active learning literature survey.  2009\.
  * [33] Zhiqiang Shen, Zhankui He, and Xiangyang Xue.  Meal: Multi-model ensemble via adversarial learning.  In AAAI, 2019.
  * [34] Zhiqiang Shen, Zechun Liu, Jie Qin, Marios Savvides, and Kwang-Ting Cheng.  Partial is better than all: Revisiting fine-tuning strategy for few-shot learning.  In AAAI, 2021.
  * [35] Zhiqiang Shen, Zechun Liu, Dejia Xu, Zitian Chen, Kwang-Ting Cheng, and Marios Savvides.  Is label smoothing truly incompatible with knowledge distillation: An empirical study.  In ICLR, 2021.
  * [36] Zhiqiang Shen and Marios Savvides.  Meal v2: Boosting vanilla resnet-50 to 80%+ top-1 accuracy on imagenet without tricks.  arXiv preprint arXiv:2009.08453, 2020.
  * [37] Zhiqiang Shen and Eric Xing.  A fast knowledge distillation framework for visual recognition.  In ECCV, 2022.
  * [38] Abhinav Shrivastava, Abhinav Gupta, and Ross Girshick.  Training region-based object detectors with online hard example mining.  In CVPR, 2016.
  * [39] Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra Gedik, Raj Prateek Kosaraju, Dhruv Mahajan, Ross Girshick, Piotr Dollár, and Laurens van der Maaten.  Revisiting weakly supervised pre-training of visual perception models.  In CVPR, 2022.
  * [40] Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe.  Curriculum learning: A survey.  International Journal of Computer Vision, 130(6):1526–1565, 2022\.
  * [41] Samuel Stanton, Pavel Izmailov, Polina Kirichenko, Alexander A Alemi, and Andrew Gordon Wilson.  Does knowledge distillation really work?  arXiv preprint arXiv:2106.05945, 2021.
  * [42] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer.  How to train your vit? data, augmentation, and regularization in vision transformers.  arXiv preprint arXiv:2106.10270, 2021.
  * [43] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna.  Rethinking the inception architecture for computer vision.  In CVPR, 2016.
  * [44] PyTorch TorchVision IMAGENET1K_V2.  https://pytorch.org/vision/stable/models/generated/torchvision.models.resnet50.html#torchvision.models.ResNet50_Weights.  2022\.
  * [45] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou.  Training data-efficient image transformers & distillation through attention.  In ICML, 2021.
  * [46] Tao Wang, Li Yuan, Xiaopeng Zhang, and Jiashi Feng.  Distilling object detectors with fine-grained feature imitation.  In CVPR, 2019.
  * [47] Tongzhou Wang, Jun-Yan Zhu, Antonio Torralba, and Alexei A Efros.  Dataset distillation.  arXiv preprint arXiv:1811.10959, 2018.
  * [48] Xin Wang, Yudong Chen, and Wenwu Zhu.  A survey on curriculum learning.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(9):4555–4576, 2021.
  * [49] Ross Wightman, Hugo Touvron, and Hervé Jégou.  Resnet strikes back: An improved training procedure in timm.  arXiv preprint arXiv:2110.00476, 2021.
  * [50] Kan Wu, Jinnian Zhang, Houwen Peng, Mengchen Liu, Bin Xiao, Jianlong Fu, and Lu Yuan.  Tinyvit: Fast pretraining distillation for small vision transformers.  In ECCV, 2022.
  * [51] Liu Xiaolong, Li Lujun, Li Chao, and Anbang Yao.  Norm: Knowledge distillation via n-to-one representation matching.  In ICLR, 2023.
  * [52] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le.  Self-training with noisy student improves imagenet classification.  In CVPR, 2020.
  * [53] Zhendong Yang, Zhe Li, Xiaohu Jiang, Yuan Gong, Zehuan Yuan, Danpei Zhao, and Chun Yuan.  Focal and global knowledge distillation for detectors.  In CVPR, 2022.
  * [54] Junho Yim, Donggyu Joo, Jihoon Bae, and Junmo Kim.  A gift from knowledge distillation: Fast optimization, network minimization and transfer learning.  In CVPR, 2017.
  * [55] Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson.  How transferable are features in deep neural networks?  NeurIPS, 27, 2014.
  * [56] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo.  Cutmix: Regularization strategy to train strong classifiers with localizable features.  In ICCV, 2019.
  * [57] Sangdoo Yun, Seong Joon Oh, Byeongho Heo, Dongyoon Han, Junsuk Choe, and Sanghyuk Chun.  Re-labeling imagenet: from single to multi-labels, from global to localized labels.  In CVPR, 2021.
  * [58] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz.  mixup: Beyond empirical risk minimization.  In ICLR, 2018.
  * [59] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma.  Be your own teacher: Improve the performance of convolutional neural networks via self distillation.  In ICCV, 2019.
  * [60] Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva.  Learning deep features for scene recognition using places database.  NeurIPS, 2014.

## Appendix

In the appendix, we provide more details omitted in the main paper, including:

• Section A: Implementation details.

• Section B: More visualization of identified crops.

Backbone | ResNet-50 | ViT-S/16
---|---|---
Epoch | 300 | 300
Batch size | 1,024 | 1,024
Optimizer | AdamW | AdamW
Init. lr | 0.002 | 0.002
lr scheduler | cosine | cosine
Weight decay | 0.05 | 0.05
Warmup epochs | 5 | 5
Num crops | 4 | 4
Label smoothing | ✗ | ✗
Dropout | ✗ | ✗
Stoch. Depth | ✗ | 0.1
Repeated Aug | ✗ | ✗
Gradient Clip. | ✗ | ✗
Rand Augment | ✗ | ✗
Mixup prob. | ✗ | 0.8
Cutmix prob. | ✗ | 1.0
SelfMix prob. | 1.0 | ✗
Random erasing | ✗ | ✗

Table 13: Pre-training setting for ImageNet-1K.

Backbone |  ViT-G/14 [10] $|$ RegY-128GF [39]
---|---
Peak learning rate | 3e-5
Optimizer | AdamW
Optimizer hyper-parameters |  $\beta_{1}$, $\beta_{2}$, $\epsilon$ = 0.9, 0.999, 1e-8
Layer-wise lr decay | 0.95
Learning rate schedule | cosine decay
Weight decay | 0.05
Input resolution | 336
Batch size | 512
Warmup epochs | 2
Training epochs | 15
Num crops | 2
Drop path |  0.4 $|$ 0.0
Augmentation | RandAug (9, 0.5)
Label smoothing | ✗
Cutmix | ✗
Mixup | ✗
Random erasing | ✗
SelfMix prob. | 1.0
Random resized crop | (0.08, 1)
Ema | 0.9999
Test crop ratio | 1.0

Table 14: Fine-tuning setting for ImageNet-1K. Figure 10: Illustration of the
identified crops by teacher model for hard (background) and easy (foreground)
samples. The teacher’s probability can reflect object and background areas
visually based on their magnitudes.

##  Appendix A Implementation Details

Training details for ResNet-50 and ViT-S/16 in the main text. We elaborate the
detailed training settings and hyper-parameters of FerKD for pre-training from
scratch on ImageNet-1K with ResNet-50 and ViT-S/16 backbones, as provided in
Table 13. Generally, the training protocal follows FKD [37]’s training
strategy on ViT, DeiT and SReT. We employ SelfMix for ResNet-50, Mixup and
CutMix for ViT-S/16 separately. We also use 4 as the number of crops in each
image, batch size $=1,024$ during training.

Training details for finetuning ViT-G/14 and RegY-128GF in the main text. The
finetuning settings and hyper-parameters of FerKD with ViT-G/14 [10] and
RegY-128GF [39] backbones are provided in Table 14, which are similar to the
training protocol in EVA [10]. We employ SelfMix for both of the two
pretrained backbones.

Data augmentation details for Mixup, Cutmix and SelfMix. The data augmentation
configurations adopted in training are: for Mixup, we use probability 0.8 to
generate the Beta distribution, and 1.0 for CutMix and SelfMix.

##  Appendix B More Visualization

Fig. 10 illustrates the identified crops by teacher for hard and easy samples.
We do not involve any localization information, but the teacher’s probability
can reflect object and background areas automatically based on their
magnitudes.


# The Hardy-Littlewood-Pólya inequality of majorization in the context of
$\mathbf{\omega}$-$\mathbf{m}$-star-convex functions

Geanina Maria Lăchescu  University of Craiova, Department of Mathematics, A.I.
Cuza Street 13, Craiova 200585, ROMANIA<EMAIL_ADDRESS>and  Ionel
Rovenţa  University of Craiova, Department of Mathematics, A.I. Cuza Street
13, Craiova 200585, ROMANIA<EMAIL_ADDRESS>
(Date: January 24, 2021)

###### Abstract.

The Hardy-Littlewood-Pólya inequality of majorization is extended for the
$\mathbf{\omega}$-$\mathbf{m}$-star-convex functions to the framework of
ordered Banach spaces. Several open problems which seem of interest for
further extensions of the Hardy-Littlewood-Pólya inequality are also included.

###### Key words and phrases:

$\mathbf{\omega}$-$\mathbf{m}$-star-convex function, majorization theory,
ordered Banach space, isotone operator

###### 2000 Mathematics Subject Classification:

Primary 26B25; Secondary 26D10, 46B40, 47B60, 47H07

##  1\. Introduction

The Hardy-Littlewood-Pólya theorem of majorization is an important result in
convex analysis that lies at the core of majorization theory, a subject that
attracted a great deal of attention due to its numerous applications in
mathematics, statistics, economics, quantum information etc. See [8], [9],
[19], [20], [23], [24] and [26] to cite just a few books treating this topic.

The relation of majorization was initially formulated as a relation between
the pairs of vectors with real entries rearranged downward, but nowadays
prevails its formulation as a preordering of probability measures.

For the reader’s convenience we briefly recall here the most basic facts
concerning the theory of majorization.

Given two discrete probability measures
$\mu=\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{x}_{k}}$ and
$\nu=\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{y}_{k}},$ supported by a compact
interval $[a,b],$ we say that $\mu$ is _majorized_ by $\mathbf{\nu}$
$($denoted $\mu\prec\nu)$ if the following three conditions are fulfilled:

$\quad\begin{array}[c]{ll}(M1)&\mathbf{x}_{1}\geq\mathbf{x}_{2}\geq\cdots\geq\mathbf{x}_{N}\\\
(M2)&\sum_{i=1}^{k}\lambda_{i}\mathbf{x}_{i}\leq\sum_{i=1}^{k}\lambda_{i}\mathbf{y}_{i}\quad\text{for
}k=1,\dots,N;\text{ and}\\\
(M3)&\sum_{i=1}^{N}\lambda_{i}\mathbf{x}_{i}=\sum_{i=1}^{N}\lambda_{i}\mathbf{y}_{i}.\end{array}$

When only conditions $(M1)$ and $(M2)$ occur, we say that $\mu$ is _weakly_
_majorized_ by $\mathbf{\nu}$ $($denoted $\mu\prec_{w}\nu).$

Hardy, Littlewood and Pólya [5] used a stronger formulation of $(M1),$ by
asking also that
$\mathbf{y}_{1}\geq\mathbf{y}_{2}\geq\cdots\geq\mathbf{y}_{N}$. Later, their
result was improved by Maligranda, Pečarić and Persson [7] who were able to
prove that

(HLP) $\mu\prec\nu\text{
implies}\int_{a}^{b}f~{}\mathrm{d}\mu=\sum_{k=1}^{N}f(\mathbf{x}_{k})\leq\int_{a}^{b}f~{}\mathrm{d}\nu=\sum_{k=1}^{N}f(\mathbf{y}_{k}),$

for all continuous convex functions $f:[a,b]\rightarrow\mathbb{R}.$ Moreover,
the same conclusion holds in the case of weak majorization and the convex and
nondecreasing functions.

Nowadays the inequality HLP is known as the Hardy-Littlewood-Pólya inequality
of majorization.

In the early 1950s, the Hardy-Littlewood-Pólya inequality was extended by
Sherman [27] to the case of continuous convex functions of a vector variable
by using a much larger concept of majorization, based on matrices stochastic
on lines. The full details can be found in [19], Theorem 4.7.3, p. 219. Over
the years, many other generalizations in the same vein were published. See,
for example, [3], [14], [15], [16], [17], [18] and [21].

As was noticed in [12] and [13], the Hardy-Littlewood-Pólya inequality of
majorization can be extended to the framework of convex functions defined on
ordered Banach spaces alongside the conditions $(M1)-(M3)$. The aim of the
present paper is to prove that the same works for the larger class of
$\omega$-$m$-star-convex functions.

The main features of these functions make the objective of Section 2.

In Section 3 we present different types of majorization relations in ordered
Banach spaces. The corresponding extensions of the Hardy-Littlewood-Pólya
inequality constitute the objective of Section 4. The paper ends by mentioning
several open problems which seem of interest for further extensions of the
Hardy-Littlewood-Pólya inequality.

##  2\. Preliminaries on $\omega$-$m$-star-convex functions

Throughout this paper $E$ is a Banach space and $C$ is a convex subset of it.

######  Definition 1.

Let $m$ be a real parameter belonging to the interval $(0,1].$ A function
$\Phi:C\rightarrow\mathbb{R}$ is said to be a perturbed $m$-star-convex
function with modulus
$\omega:\mathbb{[}0,\infty\mathbb{)}\rightarrow\mathbb{R}$ $($abbreviated,
$\omega$-$m$-star-convex function$)$ if it verifies an estimate of the form

$\Phi((1-\lambda)\mathbf{x}+\lambda
m\mathbf{y})\leq(1-\lambda)\Phi(\mathbf{x})+m\lambda\Phi(\mathbf{y})-m\lambda(1-\lambda)\omega\left(\left\|\mathbf{x}-\mathbf{y}\right\|\right),\text{\quad}$

for all $\mathbf{x},\mathbf{y}\in C$ and $\lambda\in(0,1).$

The $\omega$-$m$-star-convex functions associated to an identically zero
modulus will be called $m$-star-convex. They verify the inequality

$\Phi((1-\lambda)\mathbf{x}+\lambda
m\mathbf{y})\leq(1-\lambda)\Phi(\mathbf{x})+m\lambda\Phi(\mathbf{y}),$

for all $\mathbf{x},\mathbf{y}\in C$ and $\lambda$ $\in$ $(0,1)$.

Notice that the usual convex functions represent the particular case of
$m$-star-convex functions where $m=1.$ On the other hand every convex function
is $m$-star-convex (for every $m\in(0,1])$ if $\mathbf{0}\in C$ and
$\Phi(\mathbf{0})\leq 0.$ Indeed, we have

$\displaystyle\Phi((1-\lambda)\mathbf{x}+\lambda m\mathbf{y})$
$\displaystyle=\Phi((1-\lambda)\mathbf{x}+\lambda
m\mathbf{y+(\lambda-\lambda}m\mathbf{)0})$
$\displaystyle\leq(1-\lambda)\Phi(\mathbf{x})+m\lambda\Phi(\mathbf{y})+\mathbf{(\lambda-\lambda}m\mathbf{)\Phi(0})$
$\displaystyle=(1-\lambda)\Phi(\mathbf{x})+m\lambda\Phi(\mathbf{y}).$

Every $\omega$-$m$-star-convex function associated to a modulus __ $\omega\geq
0$ is necessarily $m$-star-convex. The $\omega$-$m$-star-convex functions
whose moduli $\omega$ are strictly positive except at the origin (where
$\omega(0)=0)$ are usually called _uniformly_ $m$_-star-convex_.__ In their
case the definitory inequality is strict whenever $\mathbf{x}\neq\mathbf{y}$
and $\lambda\in(0,1).$

By reversing the inequalities, one obtain the notions of $\omega$-$m$-_star_
-_concave function_ and __ of _uniformly_ $m$-_star_ -_concave function._

The theory of $m$-star-convex functions was initiated by Toader [28], who
considered only the case of functions defined on real intervals. For
additional results in the same setting see [11] and the references therein.

A simple example of $(16/17)$-star-convex function which is not convex is

(2.1)
$f:[0,\infty)\rightarrow\mathbb{R},\text{\quad}f(x)=x^{4}-5x^{3}+9x^{2}-5x.$

See [11], Example 2. Note that if $\Phi:C\rightarrow\mathbb{R}$ and
$\Psi:C\rightarrow\mathbb{R}$ are $\omega$-$m$-star-convex functions and
$\alpha,\beta\in\mathbb{R}_{+},$ then

$\alpha\Phi+\beta\Psi\,\text{and }\sup\left\\{\Phi,\Psi\right\\}$

are functions of the same nature. So is

$\Phi\times\Psi:C\times
C\rightarrow\mathbb{R},\text{\quad}\left(\Phi\times\Psi\right)\left(\mathbf{x},\mathbf{y}\right)=\Phi(\mathbf{x})+\Psi(\mathbf{y}).$

The class of $\omega$-$m$-star-convex functions is also stable under pointwise
convergence (when this exists).

Assuming $C\subset E$ is a convex cone with vertex at the origin, the
_perspective_ of a function $f:C\rightarrow\mathbb{R}$ is the positively
homogeneous function

$\widetilde{f}:C\times(0,\infty)\rightarrow\mathbb{R},\text{\quad}\widetilde{f}(\mathbf{x},t)=tf\left(\frac{\mathbf{x}}{t}\right).$

######  Lemma 1.

The perspective of every $m$-star-convex/concave function is a function of the
same nature.

###### Proof.

Indeed, assuming (to make a choice) that $f$ is $\omega$-$m$-star-convex, then
for all $(\mathbf{x},s),(\mathbf{y},t)\in C\times(0,\infty)$ and
$\lambda\in[0,1]$ we have

$\displaystyle f\left(\frac{(1-\lambda)\mathbf{x}+\lambda
m\mathbf{y}}{(1-\lambda)s+\lambda mt}\right)$
$\displaystyle=f\left(\frac{(1-\lambda)s}{(1-\lambda)s+\lambda
mt}\cdot\frac{\mathbf{x}}{s}+\frac{\lambda mt}{(1-\lambda)s+\lambda
mt}\cdot\frac{\mathbf{y}}{t}\right)$
$\displaystyle\leq\frac{(1-\lambda)s}{(1-\lambda)s+\lambda
mt}f\left(\frac{\mathbf{x}}{s}\right)+\frac{\lambda mt}{(1-\lambda)s+\lambda
mt}f\left(\frac{y}{t}\right)$

that is,

$\widetilde{f}((1-\lambda)\mathbf{x}+\lambda m\mathbf{y},(1-\lambda)s+\lambda
mt)\leq(1-\lambda)\widetilde{f}(\mathbf{x},s)+\lambda
m\widetilde{f}(\mathbf{y},t).$

∎

Lemma 1, allows us easily to produce nontrivial examples of $m$-star-convex
functions of several variables with some nice properties. For example,
starting (2.1), we conclude that

$\Phi(x,t)=\frac{x^{4}-5x^{3}t+9x^{2}t^{2}-5xt^{3}}{t^{3}}$

is a $(16/17)$-star-convex function on $[0,\infty)\times(0,\infty).$

Under the presence of Gâteaux differentiability, the $\omega$-$m$-star-convex
functions generate specific gradient inequalities that play a prominent role
in our generalization of the Hardy-Littlewood-Pólya inequality of
majorization.

######  Lemma 2.

Suppose also that $C$ is an open convex subset of the Banach space $E$ and
$\Phi:C\rightarrow\mathbb{R}$ is a function both Gâteaux differentiable and
$\omega$-$m$-star-convex. Then __

(2.2)
$m\Phi(\mathbf{y})\geq\Phi(\mathbf{x})+d\Phi(\mathbf{x})(m\mathbf{y}-\mathbf{x})+m\omega\left(\left\|\mathbf{x}-\mathbf{y}\right\|\right),$

for all points $\mathbf{x,y}\in C.$

###### Proof.

Indeed, we have

$\frac{\Phi((1-\lambda)\mathbf{x}+m\lambda\mathbf{y})-\Phi(\mathbf{x})}{\lambda}\leq-\Phi(\mathbf{x})+m\Phi(\mathbf{y})-m(1-\lambda)\omega\left(\left\|\mathbf{x}-\mathbf{y}\right\|\right)$

and the proof ends by passing to the limit as $\lambda\rightarrow 0+.$ ∎

######  Remark 1.

Lemma 2 shows that the critical points $\mathbf{x}$ of the differentiable
$\omega$-$m$-star–convex functions for which $\omega\geq 0$ verify the
property

$m\inf_{\mathbf{y}\in C}\Phi(\mathbf{y})\geq\Phi(\mathbf{x}).$

Unlike the case of convex functions of one real variable, when the isotonicity
of the differential is automatic, for several variables, while this is not
necessarily true in the case of a differentiable convex function of a vector
variable. See [12], Remark 4.

In this paper we deal with functions defined on ordered Banach spaces, that
is, on real Banach spaces endowed with order relations $\leq$ that make them
ordered vector spaces such that the positive cones are closed and

$\mathbf{0}\leq\mathbf{x}\leq\mathbf{y}\text{ implies
}\left\|\mathbf{x}\right\|\leq\left\|\mathbf{y}\right\|.$

The Euclidean $N$-dimensional space $\mathbb{R}^{N}$ has a natural structure
of ordered Banach space associated to the coordinatewise ordering. The usual
sequence spaces $c_{0},c,\ell^{p}$ (for $p\in[1,\infty])$ and the function
spaces $C(K)$ (for $K$ a compact Hausdorff space) and $L^{p}\left(\mu\right)$
(for $1\leq p\leq\infty$ and $\mu$ a $\sigma$-additive positive measure$)$ are
also examples of ordered Banach spaces (with respect to the
coordinatewise/pointwise ordering and the natural norms).

A map $T:E\rightarrow F$ between two ordered vector spaces is called _isotone_
(or _order preserving_) if

$\mathbf{x}\leq\mathbf{y}\text{ in }E\text{ implies }T(\mathbf{x})\leq
T(\mathbf{y})\text{ in }F$

and _antitone_(or _order reversing_) if $-T$ is isotone. When $T$ is a linear
operator, $T$ is isotone if and only if $T$ maps positive elements into
positive elements (abbreviated, $T\geq 0).$

For basic information on ordered Banach spaces see [13]. The interested reader
may also consult the classical books of Aliprantis and Tourky [1] and Meyer-
Nieberg [10].

As was noticed by Amann [2], Proposition 3.2, p. 184, the Gâteaux
differentiability offers a convenient way to recognize the property of
isotonicity of functions acting on ordered Banach spaces: the positivity of
the differential. We state here his result (following the version given in
[12], Lemma 4):

######  Lemma 3.

Suppose that $E$ and $F$ are two ordered Banach space, $C$ is a convex subset
of $E$ with nonempty interior $\operatorname{int}C$ and $\Phi:C\rightarrow F$
is a __ convex function, continuous on $C$ and Gâteaux differentiable on
$\operatorname{int}C.$ Then $\Phi$ is isotone on $C$ if and only if
$\Phi^{\prime}(\mathbf{a})\geq 0$ for all $\mathbf{a}\in\operatorname{int}C.$

######  Remark 2.

If the ordered Banach space $E$ has finite dimension, then the statement of
Lemma _3_ remains valid by replacing the interior of $C$ by the relative
interior of $C$. See _[19]_, Exercise $6$, p. $81$.

As was noticed in [11], Example $7$, the function

$\gamma:(-\infty,1]\rightarrow\mathbb{R},\quad\gamma(x)=-2x^{3}+5x^{2}+6x.$

is convex on $(-\infty,5/6]$, concave on $[5/6,1]$, and $m$-star-convex on
$(-\infty,1],$ with $m=27/28.$ The last assertion follows from a formula due
to Mocanu,

$m=\inf\Big{\\{}\frac{x\gamma^{\prime}(x)-\gamma(x)}{y\gamma^{\prime}(x)-\gamma(y)}:y\gamma^{\prime}(x)-\gamma(y),\,x,y\in
I\Big{\\}},$

mentioned at the bottom of page 72 in [11].

Proceeding like in Lemma 1, one can prove that the function associated to
$\gamma,$

$\Upsilon:(-\infty,1]\times[1,\infty)\rightarrow\mathbb{R},\text{\quad}\Upsilon(x,y)=-\frac{2x^{3}}{y^{2}}+\frac{5x^{2}}{y}+6x,$

is $27/28$-star-convex. The function $\Upsilon$ is also Gateaux
differentiable, with

$(x,y)=\allowbreak\left(\frac{1}{y^{2}}\left(-6x^{2}+10xy+6y^{2}\right),\allowbreak\frac{x^{2}}{y^{3}}\left(4x-5y\right)\right).$

According to Lemma 3, the map

$d\Upsilon:(-\infty,1]\times[1,\infty)\subset\mathbb{R}^{2}\rightarrow\mathbb{R}^{2}$

is isotone on the domain where $d^{2}\Upsilon=d(d\Upsilon)$ is positive, that
is, where the Hessian of $\Upsilon,$

$\left(\begin{array}[c]{cc}-\frac{2}{y^{2}}\left(6x-5y\right)&2\frac{x}{y^{3}}\left(6x-5y\right)\\\
2\frac{x}{y^{3}}\left(6x-5y\right)&\allowbreak-2\frac{x^{2}}{y^{4}}\left(6x-5y\right)\end{array}\right),$

has all entries nonnegative. Therefore $d\Upsilon$ is isotone on
$(-\infty,1]\times[1,\infty).$

##  3\. The majorization relation on ordered Banach spaces

In this section we discuss the concept of majorization into the framework of
ordered Banach spaces. Since in an ordered Banach space not every string of
elements admits a decreasing rearrangement, in this paper we will concentrate
to the case of pairs of discrete probability measures of which at least one of
them is supported by a monotone string of points. The case where the support
of the left measure consists of a decreasing string is defined as follows.

######  Definition 2.

Suppose that $\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{x}_{k}}$ and
$\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{y}_{k}}$ are two discrete Borel
probability measures that act on the ordered Banach space $E$ and $m\in(0,1]$
is a parameter. We say that $\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{x}_{k}}$
is weakly $mL^{\downarrow}$-majorized by
$\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{y}_{k}}$ _(_ denoted
$\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{x}_{k}}\prec_{wmL^{\downarrow}}\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{y}_{k}}$_)_
if the left hand measure is supported by a decreasing string of points

(3.1) $\mathbf{x}_{1}\geq\cdots\geq\mathbf{x}_{N}$

and

(3.2)
$\sum_{k=1}^{n}\lambda_{k}\mathbf{x}_{k}\leq\sum_{k=1}^{n}\lambda_{k}m\mathbf{y}_{k}\quad\text{for
all }n\in\\{1,\dots,N\\}.$

We say that $\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{x}_{k}}$ is
$mL^{\downarrow}$-majorized by
$\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{y}_{k}}$ $($denoted
$\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{x}_{k}}\prec_{mL^{\downarrow}}\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{y}_{k}})$
if in addition

(3.3)
$\sum_{k=1}^{N}\lambda_{k}\mathbf{x}_{k}=\sum_{k=1}^{N}\lambda_{k}m\mathbf{y}_{k}.$

Notice that the context of Definition 2 makes necessary that all weights
$\lambda_{1},...,\lambda_{N}$ belong to $(0,1]$ and
$\sum_{k=1}^{N}\lambda_{k}=1.$

The three conditions (3.1), (3.2) and (3.3) imply
$m\mathbf{y}_{N}\leq\mathbf{x}_{N}\leq\mathbf{x}_{1}\leq$ $m\mathbf{y}_{1}$
but not the ordering $\mathbf{y}_{1}\geq\cdots\geq\mathbf{y}_{N}.$ For
example, when $N=3,$ one may consider the case where

$m=1,\text{\ }\lambda_{1}=\lambda_{2}=\lambda_{3}=1/3,\text{
}\mathbf{x}_{1}=\mathbf{x}_{2}=\mathbf{x}_{3}=\mathbf{x}$

and

$\mathbf{y}_{1}=\mathbf{x},\text{ }\mathbf{y}_{2}=\mathbf{x+z,}\text{
}\mathbf{y}_{3}=\mathbf{x-z,}$

$\mathbf{z}$ being any positive element.

Under these circumstances it is natural to introduce the following companion
to Definition 2, involving the ascending strings of elements as support for
the right hand measure.

######  Definition 3.

The relation of _weak_ $mR^{\uparrow}$-_majorization_ ,

$\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{x}_{k}}\prec_{wmR^{\uparrow}}\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{y}_{k}},$

between two discrete Borel probability measures means the fulfillment of the
condition $($_3.2_ $)$ under the presence of the ordering

(3.4) $\mathbf{y}_{1}\leq\cdots\leq\mathbf{y}_{N};$

assuming in addition the condition $($_3.3_ $)$_,_ we say that
$\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{x}_{k}}$ is
$mR^{\uparrow}$-majorized by
$\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{y}_{k}}$ $($denoted
$\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{x}_{k}}\prec_{mR^{\uparrow}}\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{y}_{k}})$.

When every element of $E$ is the difference of two positive elements, the weak
majorization relations $\prec_{mL^{\downarrow}}$and $\prec_{mR^{\uparrow}}$
can be augmented so to obtain majorization relations.

##  4\. The extension of the Hardy-Littlewood-Polya inequality of majorization

The objective of this section is to consider the corresponding extensions of
the Hardy-Littlewood-Pólya inequality of majorization for
$\prec_{wmL^{\downarrow}},\prec_{mL^{\downarrow}},\prec_{wmR^{\uparrow}}$and
$\prec_{mR^{\uparrow}}$. Moreover, we present also a Sherman’s type
inequality.

The proof of the following theorem is inspired by the techniques succesfully
used in [7] and [12].

######  Theorem 1.

Suppose that $\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{x}_{k}}$ and
$\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{y}_{k}}$ are two discrete
probability measures whose supports are included in an open convex subset $C$
of the ordered Banach space $E$. If
$\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{x}_{k}}\prec_{mL^{\downarrow}}\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{y}_{k}},$
then

(4.1)
$m\sum_{k=1}^{N}\lambda_{k}\Phi(\mathbf{y}_{k})\geq\sum_{k=1}^{N}\lambda_{k}\Phi(\mathbf{x}_{k})+\sum_{k=1}^{N}\lambda_{k}\omega(\left\|\mathbf{x}_{k}-\mathbf{y}_{k}\right\|),$

for every Gâteaux differentiable $\omega$-$m$-star-convex function
$\Phi:C\rightarrow F$ whose differential is isotone and verifies the
hypotheses of Lemma 2.

The conclusion $(\ref{Cons1})$ still works under the weaker hypothesis
$\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{x}_{k}}\prec_{wmL^{\downarrow}}\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{y}_{k}},$
provided that $\Phi$ is also an isotone function.

###### Proof.

According to the gradient inequality (2.2), we have

$m\sum_{k=1}^{N}\lambda_{k}\Phi(\mathbf{y}_{k})-\sum_{k=1}^{N}\lambda_{k}\Phi(\mathbf{x}_{k})=\sum_{k=1}^{N}\lambda_{k}\left(m\Phi(\mathbf{y}_{k})-\Phi(\mathbf{x}_{k})\right)\\\
\geq\sum_{k=1}^{N}\Phi^{\prime}(\mathbf{x}_{k})(\lambda_{k}m\mathbf{y}_{k}-\lambda_{k}\mathbf{x}_{k})+\sum_{k=1}^{N}\lambda_{k}\omega\left(\left\|\mathbf{x}_{k}-\mathbf{y}_{k}\right\|\right),$

whence, by using Abel’s trick of interchanging the order of summation ([19],
Theorem 1.9.5, p. 57), one obtains

$\sum_{k=1}^{N}\lambda_{k}m\Phi(\mathbf{y}_{k})-\sum_{k=1}^{N}\lambda_{k}\Phi(\mathbf{x}_{k})-\sum_{k=1}^{N}\lambda_{k}\omega\left(\left\|\mathbf{x}_{k}-\mathbf{y}_{k}\right\|\right)\\\
\geq\Phi^{\prime}(\mathbf{x}_{1})(\lambda_{1}m\mathbf{y}_{1}-\lambda_{1}\mathbf{x}_{1})+\sum_{m=2}^{N}\Phi^{\prime}(\mathbf{x}_{m})\Bigl{[}\sum_{k=1}^{m}(\lambda_{k}\mathbf{y}_{k}-\lambda_{k}\mathbf{x}_{k})-\sum_{k=1}^{m-1}(\lambda_{k}\mathbf{y}_{k}-\lambda_{k}\mathbf{x}_{k})\Bigr{]}\\\
=\sum_{m=1}^{N-1}\Bigl{[}(\Phi^{\prime}(\mathbf{x}_{m})-\Phi^{\prime}(\mathbf{x}_{m+1}))\sum_{k=1}^{m}(\lambda_{k}m\mathbf{y}_{k}-\lambda_{k}\mathbf{x}_{k})\Bigr{]}+\Phi^{\prime}(\mathbf{x}_{N})\left(\sum_{k=1}^{N}(\lambda_{k}m\mathbf{y}_{k}-\lambda_{k}\mathbf{x}_{k})\right).$

When
$\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{x}_{k}}\prec_{mL^{\downarrow}}\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{y}_{k}},$
the last term vanishes and the fact that $D\geq 0$ is a consequence of the
isotonicity of $\Phi^{\prime}.$ When
$\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{x}_{k}}\prec_{wmL^{\downarrow}}\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{y}_{k}}$
and $\Phi$ is isotone, one applies Lemma 3 $(a)$ to infer that

$\Phi^{\prime}(\mathbf{x}_{N})\left(\sum_{k=1}^{N}(\lambda_{k}m\mathbf{y}_{k}-\lambda_{k}\mathbf{x}_{k})\right)\geq
0.$

The other cases can be treated in a similar way. ∎

######  Remark 3.

Even in the context of usual convex functions, the isotonicity of the
differential is not only sufficient but also _necessary_ for the validity of
Theorem 1. See [12], Remark 5.

We leave to the reader as an exercise the problem of formulating the variant
of Theorem 1 in the case of relations $\prec_{wmR^{\uparrow}}$and
$\prec_{mR^{\uparrow}}.$

##  5\. Further results and open problems

The aim of this section is to mention some open problems which might be of
interest for further research on .

Notice first that any perturbation of an $\omega$-$m$-star-convex function
$\Phi$ satisfying the hypotheses of Theorem 1 by a bounded function $\Pi$
verify an inequality of majorization very close to (4.1). Precisely, if
$\left|\Pi\right|\leq\delta$ and
$\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{x}_{k}}\prec_{mL^{\downarrow}}\sum_{k=1}^{N}\lambda_{k}\delta_{\mathbf{y}_{k}},$
then $\Psi=\Phi+\Pi$ will verify the relation

$m\sum_{k=1}^{N}\lambda_{k}\Phi(\mathbf{y}_{k})\geq\sum_{k=1}^{N}\lambda_{k}\Phi(\mathbf{x}_{k})+\sum_{k=1}^{N}\lambda_{k}\omega(\left\|\mathbf{x}_{k}-\mathbf{y}_{k}\right\|)-(1+m)\delta.$

This call the attention to the following class of _approximately_
$\omega$-$m$-star-convex functions:

######  Definition 4.

A function $\Phi:C\rightarrow\mathbb{R}$ is said to be
$\delta$-$\omega$-$m$-star-convex function if it verifies an estimate of the
form

$\Phi((1-\lambda)\mathbf{x}+\lambda
m\mathbf{y})\leq(1-\lambda)\Phi(\mathbf{x})+m\lambda\Phi(\mathbf{y})-m\lambda(1-\lambda)\omega\left(\left\|\mathbf{x}-\mathbf{y}\right\|\right)+\delta,\text{\quad}$

for some $\delta\geq 0$ and all $\mathbf{x},\mathbf{y}\in C$ and
$\lambda\in(0,1).$

The above definition extends (for $\omega=0$ and $m=1)$ the concept of
$\delta$-convex function, first considered by Hyers and Ulam [6] in a paper
dedicated to the stability of convex functions. It is natural to rise the
problem wheather their result extends to the framework of
$\delta$-$\omega$-$m$-star-convex functions:

######  Problem 1.

Suppose that $C$ is a convex subset of $\mathbb{R}^{N}.$ Is that true that
every $\delta$-$\omega$-$m$-star-convex function $\Phi:C\rightarrow\mathbb{R}$
can be written as $\Phi=\Psi+\Pi$, where $\Psi$ is an $\omega$-$m$-star-convex
function and $\Pi$ is a bounded function whose supremum norm is not larger
than $k_{N}\delta$ , where the positive constant $k_{N}$ depends only on the
dimension $N$ of the underlying space?

Of some interest seems to be the concept of local approximate $m$-star-
convexity suggested by [4], Definition 1, which clearly yields new extensions
of the majorization inequality:

######  Definition 5.

A function $\Phi:C\rightarrow\mathbb{R}$ is called locally approximately
$m$-star-convex if for every $x_{0}\in C$, and every $\varepsilon>0$ there
exists $\delta>0$ such that for all $x,y$ in the open ball of center $x_{0}$
and radius $\delta$ and all $\lambda\in(0,1),$

$\Phi((1-\lambda)x+m\lambda
y)\leq(1-\lambda)\Phi(x)+m\lambda\Phi(y)+\rule{7.22743pt}{7.22743pt}t(1-t)\left\|x-y\right\|.$

The whole discussion above can be placed in the more general context of
$M_{p}$-convexity.

Recall that the weighted $M_{p}$-mean is defined for every pair of positive
numbers $a,b$ by the formula

$M_{p}(a,b;1-\lambda,\lambda)=\left\\{\begin{array}[c]{cl}((1-\lambda)a^{p}+\lambda
b^{p})^{1/p},&\text{if }p\in\mathbb{R}\backslash\\{0\\}\\\
a^{1-\lambda}b^{\lambda},&\text{if }p=0\\\ \max\\{a,b\\},&\text{if
}p=\infty,\end{array}\right.$

where $\lambda\in[0,1].$ If $p>0,$ then it is usual to extend $M_{p}$ to all
pairs of nonnegative numbers.

######  Definition 6.

A function $\Phi:C\rightarrow\mathbb{R}$ is called $\omega$-$m$-$M_{p}$-star-
convex if there exist a number $p\in\mathbb{R}$ and a modulus
$\omega:\mathbb{[}0,\infty\mathbb{)}\rightarrow\mathbb{R}$ such that

$\Phi\left(\left(1-\lambda\right)\mathbf{x}+\lambda\mathbf{y}\right)\leq((1-\lambda)\Phi(\mathbf{x})^{p}+m\lambda\Phi(\mathbf{y})^{p})^{1/p}-m\lambda(1-\lambda)\omega\left(\left\|\mathbf{x}-\mathbf{y}\right\|\right),\text{\quad}$

for all $\mathbf{x},\mathbf{y}\in C$ and $\lambda\in(0,1).$

Reversing the inequality one obtain the concept of $\omega$-$m$-$M_{p}$-star-
concave functions.

The usual $M_{p}$-convex/$M_{p}$-concave functions represent the particular
case where $m=1$ and $\omega=0.$

It is worth noticing that the $M_{p}$-convex ($M_{p}$-concave) functions for
$p\neq 0$ are precisely the functions $\Phi$ such that $\Phi^{p}$ is convex
(concave), while the $M_{0}$-convex ($M_{0}$-concave) functions are nothing
but the $\log$-convex ($\log$-concave) functions. Notice also that the
$M_{\infty}$-convex ($M_{-\infty}$-concave) functions are precisely the quasi-
convex (quasi-concave) functions.

The next result represents the extension of Lemma 2 to the case of
$\omega$-$m$-$M_{p}$-star-convex functions.

######  Lemma 4.

Suppose that $C$ is an open convex subset of the Banach space $E$ and
$\Phi:C\rightarrow\mathbb{R}_{+}$ is a function both Gâteaux differentiable
and $\omega$-$m$-$M_{p}$-star-convex. If $p\neq 0,$ then $\Phi$ verifies the
inequality __

$\Phi^{p}(\mathbf{y})\geq\Phi^{p}(\mathbf{x})+p\Phi(\mathbf{x})^{p-1}d\Phi(\mathbf{x})(\mathbf{y}-\mathbf{x})+m\omega\left(\left\|\mathbf{x}-\mathbf{y}\right\|\right),$

for all $\mathbf{x,y}\in C.$

The analogue of this result for $p=0$ and $\omega=0$ requires the strict
positivity of the function $\Phi$ and can be stated as

$\log\Phi(\mathbf{y})-\log\Phi(\mathbf{x})\geq\frac{d\Phi(\mathbf{x})(\mathbf{y}-\mathbf{x})}{\Phi(\mathbf{x})},$

for all $\mathbf{x,y}\in C.$ The last two inequalities work in the reverse
direction in the case of $\omega$-$m$-$M_{p}$-star-concave functions.

While it is clear that Lemma 4 allows us to prove Hardy-Littlewood-Pólya type
inequalities more general than those provided by Theorem 1, the exploration of
the world of $\omega$-$m$-$M_{p}$-star-convex/concave functions for
$\omega\neq 0$ and $m\in(0,1)$ is just at the beginning..

Data availability statement: The authors declare that data supporting the
findings of this study are available within the article and its supplementary
information files.

## References

  * [1] C.D. Aliprantis, R. Tourky, Cones and Duality, Graduate Studies in Mathematics 84, American Mathematical Society, Providence, RI, 2007.
  * [2] H. Amann, Multiple positive fixed points of asymptotically linear maps, J. Functional Analysis 17 (2) (1974), 174-213.
  * [3] S. Ivelić Bradanović, N. Latif, Pečarić, and J. Pečarić, Sherman’s and related inequalities with applications in information theory, J Inequal Appl. 2018, (2018) (1), article 98.
  * [4] A. Daniilidis, P. Georgiev. Approximate convexity and submonotonicity. J. Math. Anal. Appl. 291 (1) (2004), 292-301.
  * [5] G.H. Hardy, J. E. Littlewood and G. Pólya, Inequalities, 2nd ed., Cambridge University Press, 1952. Reprinted 1988.
  * [6] D. H. Hyers and S. M. Ulam, Approximately convex functions, Proc. Amer. Math. Soc. 3 (1952), 821-828.
  * [7] L. Maligranda, J. Pečarić and L.-E. Persson, Weighted Favard’s and Berwald’s inequalities, J. Math. Anal. Appl. 190 (1) (1995), 248-262.
  * [8] A.W. Marshall, I. Olkin, Inequalities: theory of majorization and its applications. Vol. 143. New York, Academic Press, 1979.
  * [9] A.W. Marshall, I. Olkin and B. Arnold, Inequalities: Theory of majorization and its applications, 2nd ed., Springer Series in Statistics, Springer, New York, 2011. star-c0nvex
  * [10] P. Meyer-Nieberg, Banach Lattices, Springer-Verlag, Berlin, 1991.
  * [11] P.T. Mocanu, I. Serb, and G. Toader, Real star-convex functions. Stud. Univ. Babes-Bolyai, Math 43 (1997), 65-80.
  * [12] C.P. Niculescu, A new look at the Hardy-Littlewood-Polya inequality of majorization, J. Math. Anal. Appl. 501 (2021) (2), article 125211.
  * [13] C.P. Niculescu, O. Olteanu, From the Hahn-Banach extension theorem to the isotonicity of convex functions and the majorization theory, Rev. R. Acad. Cienc. Exactas Fís. Nat. Ser. A Math. RACSAM 114 (4) (2020), 1-19.
  * [14] C.P. Niculescu, F. Popovici, The extension of majorization inequalities within the framework of relative convexity, Journal of Inequalities in Pure and Applied Mathematics (JIPAM) 7 (2006) Article No. 27, 6 pp. (Electronic only)
  * [15] C.P. Niculescu, I. Rovenţa, An approach of majorization in spaces with a curved geometry, J. Math. Anal. Appl. 411 (2014), 119-128.
  * [16] C.P. Niculescu, I. Rovenţa, Relative convexity and its applications, Aequationes Math. 89 (2015), 1389-1400.
  * [17] C.P. Niculescu, I. Rovenţa, Relative Schur convexity on global NPC spaces, Math. Inequal. Appl. 18 (2015), 1111-1119.
  * [18] C.P. Niculescu, I. Rovenţa, Hardy-Littlewood-Pólya theorem of majorization in the framework of generalized convexity, Carpathian Journal of Mathematics 33 (2017), 87-95.
  * [19] C.P. Niculescu, L.-E. Persson, Convex Functions and Their Applications. A Contemporary Approach, 2nd ed., CMS Books in Mathematics vol. 23, Springer-Verlag, New York, 2018.
  * [20] F. Nielsen and R. Bhatia, eds., Matrix information geometry. Heidelberg: Springer, 2013.
  * [21] M. Niezgoda, Nonlinear Sherman-type inequalities, Adv. Nonlinear Anal. 9 (2020), 168-175.
  * [22] Z. Páles, On approximately convex functions, Proceedings of the American Mathematical Society 131 (1) (2003), 243-252.
  * [23] J.E., Pečarić, F. Proschan, Y.C. Tong, Convex Functions, Partial Orderings and Statistical Applications. Academic Press Inc, San Diego (1992).
  * [24] D. Petz, Quantum information theory and quantum statistics. Springer Science & Business Media, 2007.
  * [25] T. Popoviciu, Sur certaines inégalités qui caractérisent les fonctions convexes, An. Ştiinţ. Univ. Al. I. Cuza Iaşi. Secţ. Mat. 11 (1965), 155-164.
  * [26] L. Rüschendorf, Mathematical risk analysis, Springer Ser. Oper. Res. Financ. Eng. Springer, Heidelberg 2013.
  * [27] S. Sherman, On a theorem of Hardy, Littlewood, Polya, and Blackwell, Proc. Natl. Acad. Sci. USA 37 (1951), 826-831.
  * [28] G. Toader, Some generalizations of the convexity, Proceedings of the Colloquium on Approximation and Optimization, Univ. Cluj-Napoca, Cluj-Napoca, 1985, 329-338.