2021

[1]Binghui Lu

1]School of Instrumentation Science and Engineering, Harbin Institute of
Techonoloy, Harbin, 150001, China

2]Shanghai Spaceflight Precision Machinery Institute, Shanghai, 201600, China

# SAL-CNN: Estimate the Remaining Useful Life of Bearings Using Time-frequency
Information

Bingguo Liu<EMAIL_ADDRESS>Zhuo Gao<EMAIL_ADDRESS><EMAIL_ADDRESS>Hangcheng Dong<EMAIL_ADDRESS>Zeru An
<EMAIL_ADDRESS>[  [

###### Abstract

In modern industrial production, the prediction ability of remaining useful
life (RUL) of bearings directly affects the safety and stability of the
system. Traditional methods require rigorous physical modeling and perform
poorly for complex systems. In this paper, an end-to-end RUL prediction method
is proposed, which uses short-time Fourier transform (STFT) as preprocessing.
Considering the time correlation of signal sequences, a long and short-term
memory network is designed in CNN, incorporating the convolutional block
attention module, and understanding the decision-making process of the network
from the interpretability level. Experiments were carried out on the 2012PHM
dataset and compared with other methods, and the results proved the
effectiveness of the method.

###### keywords:

Bearing, STFT, RUL, Deep learning

##  1 Introduction

In the operation of equipment, a large number of faults are caused by bearing
failure. Prediction of remaining useful life (RUL) of bearings has become a
key technology to ensure mechanical work safety. Existing bearing RUL
prediction methods include two types, statistical life model prediction and
data-driven prediction. Model-based methods include particle filteringbib1 ,
Eyring modelbib2 , Weibull distributionbib3 , etc. These methods need a large
number of statistical data as the basis to have certain reliability, but it is
difficult to establish an accurate and general mathematical model for complex
equipment.

The data-driven method uses the end-to-end training strategy and uses the
state monitoring data to predict. A.Soualhi et al. proposed a rolling bearing
condition monitoring method combining Hilbert-Huang transform, support vector
machine, and support vector regressionbib4 . S.A. Aye et al. proposed an
optimal Gaussian process regression (GPR) for low-speed bearing RULbib5 .
Zhang Z et al. converted these signals into the frequency domain using wavelet
packet decomposition and fast Fourier transform and trained artificial neural
network (ANN)bib6 .

In the field of CNN, G.S. Babu et al. and L. Ren et al. respectively used fast
Fourier transform and wavelet transform as pretreatments and used CNN to
predict RULbib7 bib8 . Zhu et al. used the original vibration signal as the
input of CNN for training and testingbib9 . A.Z. Hinchi et al. using CNN and
LSTM to predict RULbib10 . Biao Wang et al. remember information by feedback
connection that feeds back the output of the convolutional layer as inputbib11
. Xiang Li Wei et al. adopt the convolutional neural network of multi-scale
feature fusion to gradually extract high-level features and discard low-level
features at the same time. Features of different levels are fused, which
improves the characteristic of one-way step-by-step transmission of feature
information in the traditional convolutional networkbib12 . The difference
between this paper and other methods is that the time-frequency diagram of
vibration signals is taken as the network input, and CNN integrates LSTM and
CBAM.

Focusing on RUL prediction, STFT pretreatment attention and memory based CNN
(SAL-CNN) is proposed in this paper. Based on CNN, the short-time Fourier
transform (STFT) is used as the vibration signal preprocessing, and long
short-term memory(LSTM) is integrated to make up for the deficiency that
traditional CNN cannot consider the time correlation between vibration signal
sequences, and convolutional block attention module (CBAM) is used to achieve
accurate acquisition of fault features and improve the accuracy of network
life prediction.

The main contributions of this paper are as follows: A CNN framework (SAL-CNN)
is proposed, which takes STFT transformation as pretreatment and integrates
CBAM and LSTM; visualizing the output of CBAM to explain the framework,
discuss the relationship between RUL and frequency, and enhance experimental
credibility.

This paper introduces the basic theory about LSTM and CBAM in Section 2, the
preprocessing algorithm and network SAL-CNN are detailed in Section 3, the
experimental situation and comparative analysis are introduced in Section 4,
and the interpretability of the network is subsequently studied. Finally, the
summary is made in Section 5.

##  2 Related work

###  2.1 Long short-term memory

The recurrent neural network (RNN)bib13  takes both the output characteristics
of the last moment and the data of the current moment as input. But gradient
disappearance and gradient explosion will occur with the increase of cycle
layers. LSTMbib14  is a special RNN, it introduces the concepts of gating unit
and cell state. The input gate controls the output information from the
previous layer, and the memory information from the previous moment is
controlled by the forgetting gate. The most common LSTM structures are shown
in Fig.1.

Figure 1: LSTM structures

The expression of the forgetting gate is:

$f_{t}=\sigma\left(W_{f}\cdot\left[h_{t-1},x_{t}\right]+b_{f}\right)$ (1)

The expression of the input gate is:

$i_{t}=\sigma\left(W_{i}\cdot\left[h_{t-1},x_{t}\right]+b_{i}\right)$ (2)
$\widetilde{C_{t}}=tanh{\left(W_{C}\cdot\left[h_{t-1},x_{t}\right]+b_{C}\right)}$
(3)

The expression of the output gate is:

$o_{t}=\sigma\left(W_{o}\cdot\left[h_{t-1},x_{t}\right]+b_{o}\right)$ (4)
$h_{t}=o_{t}\cdot tanh{\left(C_{t}\right)}$ (5)

Where $W_{f},W_{i},Wc$ and $W_{o}$ represent the weight matrix respectively,
$\sigma$ represents the sigmoid activation function, and act on after passing
through the forgetting gate and the input gate. The expression is:

$C_{t}=f_{t}\cdot C_{t-1}+i_{t}\cdot{\widetilde{C}}_{t-1}$ (6)

###  2.2 Convolutional block attention module

Convolutional block attention module (CBAM)bib15  is a simple and effective
injection module for feedforward convolutional neural network, its structure
is shown in the Fig.2. As a lightweight module, CBAM can be placed behind any
feature graph as required. For feature map $F\in R_{c}\cdot h\cdot w$ of an
intermediate layer, CBAM will deduce the 1-dimensional channel attention map
$M_{c}\in R_{c}\cdot 1\cdot 1$ and 2-dimensional spatial attention map
$M_{s}\in R1\cdot H\cdot W$, as shown below :

$F’=M_{c}(F)\otimes F$ (7) $F^{\prime\prime}=M_{s}(F^{\prime})\otimes
F^{\prime}$ (8)

Where $\otimes$ is element-wise multiplication, and the expressions of the
channel attention module Mc(F) and space attention module Ms(F) are as follows
:

$M_{C}\left(F\right)=\sigma\left(\mathrm{MLP}\left(\mathrm{AvgPool}\left(F\right)\right)+\mathrm{MLP}\left(\mathrm{MaxPool}\left(F\right)\right)\right)$
(9)
$M_{S}\left(F\right)=\sigma\left(f\left(\left[\mathrm{AvgPool}\left(F\right),\mathrm{MaxPool}\left(F\right)\right]\right)\right)$
(10)

Where $\sigma$ is Sigmoid activation function, and $f$ is convolution
operation.

Figure 2: CBAM structures

##  3 Framework

Aiming at the problems that existing models cannot accurately predict RUL and
work in real-time, an end-to-end detection framework, SAL-CNN, is proposed.
The flow diagram is shown in Fig.3. The method includes two parts: vibration
signal preprocessing and convolution neural network. CNN is designed to
integrate convolutional block attention module and long and short term memory
network.

Figure 3: Flow diagram

###  3.1 Vibration signal preprocessing

Since vibration signals have many spectral components and are non-stationary
signals, short-time Fourier transformbib16  is used to preprocessing the input
data to obtain the time-frequency graph diagram. Short-time Fourier transform
is a kind of joint time-frequency analysis method. Based on Fourier transform,
the whole segment of signal is processed by window segmentation, and each
small segment is considered as a stationary signal. The window function slips
along the time axis, and the Fourier transform is carried out in the
neighborhood at any time.

For non-stationary signals $x\left(t\right)\in L^{2}\left(R\right)$, the
short-time Fourier transform of $x\left(t\right)$ is expressed as:

$STFT_{x}\left(t,f\right)=\int_{-\infty}^{\infty}x\left(\tau\right)h\left(\tau-t\right)e^{-j2\pi
ft}d\tau$ (11)

Where $h\left(\tau-t\right)$ is window function.

Take the Fourier transform of $h\left(t\right)$, the energy is concentrated at
the low frequency range, so it’s usually thought of a low-pass filter. As can
be seen from the equation(11), window function $h\left(t\right)$ moves along
the time axis and conducts segmented processing on $h\left(t\right)$ in STFT
transformation, the expression is:

$x_{t}\left(\tau\right)=x\left(\tau\right)h\left(\tau-t\right)$ (12)

Later, take the Fourier transform of $x_{t}\left(\tau\right)$.

After windowing the signal, all the features of the signal covered by the
window function will be displayed. The selection of the window function
directly affects the results after STFT transformation. The frequency
resolution is mainly determined by the main lobe width, so the window function
with the narrowest main lobe and the smallest sidelobe peak value should be
selected. The common window functions are rectangular window, triangular
window, Hamming window, Hamming window, Blackman window, etc., and hamming
window is selected as the window function comprehensively.

Fig.4 shows the vibration signal of bearing 1-1 in the whole life cycle and
the time-frequency diagram after STFT transformation in the initial, middle
and final stages, the first row is the vibration signal diagram, and the
second row is the time-frequency diagram after STFT transformation. The X-axis
represents the time axis, and the Y-axis represents the vibration frequency.
The more brightly colored part represents the more concentrated energy of the
vibration frequency in the same time series. The third row is a 3D display of
the time-frequency graph, where the z-axis represents the frequency occurrence
times. As can be seen from the figure, with the degradation of bearings and
the increase of vibration signal amplitude, signals gradually concentrated in
the low-frequency range. By this time, the high-frequency part was mainly an
external noise signal, which made little contribution to life prediction.
Therefore, the frequency of the upper part of the vibration signal was
filtered out, and the remaining time-frequency information was used to predict
RUL.

Figure 4: Three stages of vibration signal

###  3.2 Network model structure

Considering that the input data is time-dependent, it is proposed to integrate
a convolutional neural network and long and short term memory network. On this
basis, a convolutional block attention module is introduced to improve the
learning performance of the network, then visualizing the output of this
module. The structure of the proposed neural network is shown in Fig.5.

Figure 5: Structure of SAL-CNN

Firstly, the vibration signal is preprocessed by STFT transformation, the size
of time-frequency diagram is 11$\times$129, after preprocessing, enters into a
six-layer convolution operation, and then into CBAM. After that, a one-layer
convolution operation with a convolution kernel is used to compress the
information, followed by LSTM, the last layer is full connection layer and
outputs a predicted life value, Table 1 describes specific network parameters.

Table 1: Network parameters of SAL-CNN

Layer | Parameter | Value
---|---|---
STFT | window | Hamming
fs | 25600 Hz
Nperseg | 512
Conv1 | In channels | 1
Kernel size | 3×3
Out channels | 5
Conv2 | In channels | 5
Kernel size | 3×3
Out channels | 5
Conv3 | In channels | 5
Kernel size | 3×3
Out channels | 1
LSTM | Input size | 1408
Hidden size | 1408
Num layers | 2
|  |

##  4 Experiments and discussions

###  4.1 Data sets and evaluation indicators

In order to verify the effectiveness of the method, the data set of the
2012PHM Challengebib17  was used as the validation data set. The data set was
provided by the FEMTO-ST research institute in France and was obtained by
building an experimental platform in a laboratory environment. Three working
conditions were used for experiments: (1)rotate speed:1800 r/min, radial
force: 4000 N; (2)rotate speed: 1650 r/min, radial force: 4200 N; (3)rotate
speed: 1500 r/min, radial force: 5000 N. The experimental platform collects
vibration signals of the whole life cycle of bearings, and a total of 17
bearings are tested and collected vibration signals,Table 2 is the
distribution of bearings.

Table 2: distribution of bearings in 2012PHM Challenge operating mode | radial force(N) | rotate speed(rpm) | Bearing
---|---|---|---
1 | 4000 | 1800 | Bearing1-1,Bearing1-2
Bearing1-3,Bearing1-4
|  |  | Bearing1-5,Bearing1-6,Bearing1-7
2 | 4200 | 1650 | Bearing2-1,Bearing2-2
Bearing2-3,Bearing2-4
|  |  | Bearing2-5,Bearing2-6,Bearing2-7
3 | 5000 | 1500 | Bearing3-1,Bearing3-2,Bearing3-3
|  |  |

In order to quantitatively evaluate the network structure, MAE is used as the
evaluation index. As shown by expression 13, the more accurate the prediction
results are, the lower MAE is:

$MAE=\frac{1}{s}\sum_{i=1}^{s}\left({\rm ActRUL}_{i}-{\rm PreRUL}_{i}\right)$
(13)

where S is the number of testing samples, ActRULi is the actual RUL values
corresponding to the i-th testing sample, and PreRULi is the predictive RUL
values corresponding to it.

###  4.2 Experimental setup

All bearings were used as prediction bearings in turn, and the remaining 16
bearings were used as training sets in each experiment to obtain cross-
validation results.

The training set is input into SAL-CNN for training, SAL-CNN is implemented by
PyTorch with Nvidia GeForce GTX 2080ti GPU, the iteration training is
terminated after 150 epochs. The loss function is L1 loss and the optimizer is
Adam, the learning rate is set to 0.001 and batch size is 32, dropout rate is
0.1, The number of LSTM cycle layers is 1.

###  4.3 Analysis

Fig.6 shows the prediction results of some bearings. The red curve is the real
RUL and the yellow curve is the predicted RUL. It can be seen that the yellow
curve basically fits the true value.

Figure 6: RUL prediction results

This paper conducted comparative experiments with 4 methods on the 2012PHM
Challenge data set. Table 3 shows MAE indexes compared with other methods. DNN
is also known as multi-layer perceptron (MLP), Single Scale-Low (SSL), and
Single Scale-High (SSH) use low-scale features and high-level features
respectively for RUL estimation without feature concatenation, and multi-scale
method integrates all scale featurebib18 .

The comparison result indicates that the MAE obtained by the proposed
algorithm is higher than others. Among the 17 bearings, the prediction results
of 11 bearings are best than others, and 2/3 of the results are better and
evenly distributed under each working condition, indicating that this method
is effective for bearings under three working conditions. According to the
average value of the final MAE, it can be seen that the results in this paper
are improved by 2.8 compared with the multi-scale method.

Table 3: Comparison of life prediction results based on MAE index Bearing | DNN | SSL | SSH | multi-scalebib18  | SAL-CNN
---|---|---|---|---|---
Bearing1-1 | 30.4 | 24.3 | 24.4 | 21.4 | 16.9
Bearing1-2 | 28.5 | 17.4 | 22.4 | 14.7 | 13.0
Bearing1-3 | 16.3 | 9.5 | 9.3 | 7.8 | 10.3
Bearing1-4 | 32.1 | 21.3 | 24.5 | 21.8 | 23.0
Bearing1-5 | 28.7 | 21.4 | 19.5 | 18.5 | 11.8
Bearing1-6 | 32.4 | 24.5 | 20.4 | 20.3 | 21.4
Bearing1-7 | 16.6 | 9.6 | 11.7 | 8.3 | 11.9
Bearing2-1 | 28.6 | 24.2 | 25.2 | 21.5 | 18.5
Bearing2-2 | 26.4 | 15.3 | 16.5 | 16.2 | 8.8
Bearing2-3 | 40.5 | 35.2 | 31.5 | 32.8 | 14.8
Bearing2-4 | 22.3 | 11.6 | 9.9 | 6.1 | 24.2
Bearing2-5 | 31.4 | 27.3 | 25.2 | 23.6 | 23.3
Bearing2-6 | 29,4 | 23.2 | 19.8 | 20.7 | 8.6
Bearing2-7 | 38.0 | 28.2 | 28.4 | 27.4 | 27.1
Bearing3-1 | 28.3 | 28.5 | 26.3 | 19.6 | 25.2
Bcaring3-2 | 30.4 | 25.2 | 23.7 | 23.1 | 18.1
Bearing3-3 | 38.5 | 35.4 | 33.5 | 30.4 | 11.1
|  |  |  |  |

###  4.4 Feature information visualization

Feature visualization is a post-hoc interpretation of the pre-training model.
After training, the decision-making process of CNN is explained in the form of
images, which can intuitively reflect the key areas of input features that the
network focuses on.

Table 4 shows the time-frequency graph and heatmap of bearings 1-1, 1-3,1-7 in
the early, middle, and late stages. In time-frequency graph, the ordinate is
the frequency and the value range is [0,6400], and the abscissa is time and
the value range is [0,1]. In the trained network, visualizing the output of
the CBAM layer, and obtaining the heatmap, where the highlighted parts
represent the areas that the network is more concerned with. As time goes by,
the frequency concentrated range in the time-frequency graph is from
3000-4000Hz to 0-2000Hz, and the high-frequency part is basically no energy.
It can also be seen from the heatmap that the network pays more attention to
the information of the low-frequency part in the late stage, it means low-
frequency signal may become more import.

Table 4: Spectrogram and Heatmap of bearing in three stages bearing | Image type | Early stage | Middle stage | Late stage
---|---|---|---|---
1-1 |  | time-frequency
---
graph
|  |
Heatmap |  |  |
1-3 |  | time-frequency
---
graph
|  |
Heatmap |  |  |
1-5 |  | time-frequency
---
graph
|  |
Heatmap |  |  |
|  |  |  |

##  5 Summary

This paper proposes a long and short-term memory convolutional neural network
that introduces a convolutional block attention module. The improved algorithm
makes up for the lack of traditional convolutional networks that cannot
consider the time correlation of vibration signal sequences. At the same time,
the convolutional block attention module further improves the prediction
performance of the network. The test on the PHM2012 data set proves that the
average MAE is slightly higher than the best recent results.

In addition, we found that for predicting the RUL task, low-frequency
vibration signals could be more worthy of attention at a later stage, from the
time-frequency graph can directly see the most energy distribute in low-
frequency range at a later stage . Using heatmap visualization the output of
CBAM, also found that the network at the later stage pay more attention to the
low-frequency range, from the aspect of interpretability confirms this
conjecture.

## References

  *   * (1) Marine, Jouin, Rafael, Gouriveau, Daniel, Hissel, Marie-Cécile, Péra, Noureddine, Zerhouni: Particle filter-based prognostics: Review, discussion and perspectives. Mechanical Systems & Signal Processing (2016)
  * (2) Jouin, M., Gouriveau, R., Hissel, D., Pera, M.C., Zerhouni, N.: Degradations analysis and aging modeling for health assessment and prognostics of pemfc. Reliability Engineering & System Safety 148(APR.), 78–95 (2016)
  * (3) Ali, J.B., Chebel-Morello, B., Saidi, L., Malinowski, S., Fnaiech, F.: Accurate bearing remaining useful life prediction based on weibull distribution and artificial neural network. Mechanical Systems & Signal Processing 56-57(may), 150–172 (2015)
  * (4) Soualhi, A., Medjaher, K., Zerhouni, N.: Bearing health monitoring based on hilbert-huang transform, support vector machine and regression. IEEE Transactions on Instrumentation & Measurement 64(1), 52–62 (2014)
  * (5) Aye, S.A., Heyns, P.S.: An integrated gaussian process regression for prediction of remaining useful life of slow speed bearings based on acoustic emission. Mechanical Systems and Signal Processing (2017)
  * (6) Zhang, Z., Wang, Y., Wang, K.: Fault diagnosis and prognosis using wavelet packet decomposition, fourier transform and artificial neural network. Journal of Intelligent Manufacturing 24(6), 1213–1227 (2013)
  * (7) Babu, G.S., Zhao, P., Li, X.L.: Deep convolutional neural network based regression approach for estimation of remaining useful life. International Conference on Database Systems for Advanced Applications (2016)
  * (8) Ren, L., Sun, Y., Wang, H., Zhang, L.: Prediction of bearing remaining useful life with deep convolution neural network. IEEE Access, 13041–13049 (2018)
  * (9) Zhu, J., Nan, C., Peng, W.: Estimation of bearing remaining useful life based on multiscale convolutional neural network. IEEE Transactions on Industrial Electronics PP, 1–1 (2018)
  * (10) Hinchi, A.Z., Tkiouat, M.: Rolling element bearing remaining useful life estimation based on a convolutional long-short-term memory network. Procedia Computer Science 127, 123–132 (2018)
  * (11) Wang, B., Lei, Y., Yan, T., Li, N., Guo, L.: Recurrent convolutional neural network: A new framework for remaining useful life prediction of machinery. Neurocomputing 379, 117–129 (2020)
  * (12) Li, X., Zhang, W., Ding, Q.: Deep learning-based remaining useful life estimation of bearings using multi-scale feature extraction. Reliability Engineering & System Safety 182(FEB.), 208–218 (2019)
  * (13) Elman, J.L.: Finding structure in time. Cognitive Science 14(2), 179–211 (1990)
  * (14) Zaremba, W., Sutskever, I., Vinyals, O.: Recurrent neural network regularization. Eprint Arxiv (2014)
  * (15) Woo, S., Park, J., Lee, J.Y., Kweon, I.S.: Cbam: Convolutional block attention module. In: European Conference on Computer Vision (2018)
  * (16) Chui, C.K., Jiang, Q., Li, L., Lu, J.: Analysis of an adaptive short-time fourier transform-based multicomponent signal separation method derived from linear chirp local approximation. Journal of Computational and Applied Mathematics 396(1971), 113607 (2021)
  * (17) Sutrisno, E., Oh, H., Vasan, A.S.S., Pecht, M.: Estimation of remaining useful life of ball bearings using data driven methodologies. In: 2012 IEEE Conference on Prognostics and Health Management, pp. 1–7 (2012)
  * (18) Li, X., Zhang, W., Ding, Q.: Deep learning-based remaining useful life estimation of bearings using multi-scale feature extraction. Reliability Engineering & System Safety 182(FEB.), 208–218 (2019)


# Few-Shot Image Classification Along Sparse Graphs

Joseph F Comer  Philip L Jacobson  Heiko Hoffmann
HRL Laboratories, LLC
3011 Malibu Canyon Rd, Malibu, CA 90265
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Few-shot learning remains a challenging problem, with unsatisfactory 1-shot
accuracies for most real-world data. Here, we present a different perspective
for data distributions in the feature space of a deep network and show how to
exploit it for few-shot learning. First, we observe that nearest neighbors in
the feature space are with high probability members of the same class while
generally two random points from one class are not much closer to each other
than points from different classes. This observation suggests that classes in
feature space form sparse, loosely connected graphs instead of dense clusters.
To exploit this property, we propose using a small amount of label propagation
into the unlabeled space and then using a kernel PCA reconstruction error as
decision boundary for the feature-space data distribution of each class. Using
this method, which we call “K-Prop,” we demonstrate largely improved few-shot
learning performances (e.g., 83% accuracy for 1-shot 5-way classification on
the RESISC45 satellite-images dataset) for datasets for which a backbone
network can be trained with high within-class nearest-neighbor probabilities.
We demonstrate this relationship using six different datasets.

##  1 Introduction

Learning from few labeled examples, or “few-shot” learning, is needed for
applications where labels are expensive or hard to obtain or where adapting to
new data has to be fast. But few-shot learning remains a challenging problem.
Particularly, with only 1 to 5 labels per class, classification accuracies are
typically low on real-world data [13].

The simplest approach to few-shot learning is to adapt (fine-tune) a pre-
trained network to a new target dataset based on the small set of available
labeled data [5]. Either the weights of the entire network are adapted or only
the final classification layer. In the latter case, the network is typically
split into two parts: a backbone network, consisting, e.g., of multiple
convolutional and pooling layers and a head consisting of a multi-layer
perception or a linear mapping. Here, the backbone maps the input data into a
so-called feature space. For few-shot learning the weights of the backbone are
typically frozen while the weights of the head are adapted. With comprehensive
pre-training of the backbone (on a meta training set), training a linear
classifier in the feature space can provide decent results [22], though the
1-shot results are typically in the best case in the 60-70% accuracy regime
for 5-way classification.

Figure 1: Comparing the accuracy of our method, K-Prop, with ProtoNets on the
RESISC45 dataset for 5-way classification (mean $\pm$ SE).

As alternatives to a linear classifier baseline, various methods have been
suggested, broadly falling into three categories: initialization-based,
metric-based, and generative methods. Initialization-based methods, such as
MAML [8, 17], search for a good initialization of the model weights for which
good performance on downstream tasks can be achieved with few labeled examples
and in few training epochs. Metric-based methods work by replacing the linear
classifier with a more sophisticated way of comparing distances between unseen
data and the few labeled points. Some examples include ProtoNets [21], which
compute distances to prototypical references points for each class, and
Adaptive Subspaces [20], which compute distances to subspaces fitted to the
data distribution of a class in feature space (Fig. 2). These methods do not
necessarily provide an improvement over a linear classifier, which can, e.g.,
beat ProtoNets if having a well trained backbone [22]. Lastly, generative
methods work by using a generator to augment the original data with novel data
for training [25]. MetaGAN uses an existing few-shot learning method and
boosts its performance by generating additional data, though the reported
improvements have been only by a few percent [25].

Figure 2: Methods for classification in feature space: ProtoNets [21],
Subspaces [20], and kernel PCA.

Here, we provide an alternative to these approaches for classifying data in
the feature space. As a backbone network, we consider either a pretrained
network or a self-supervised trained network, which is trained on the
unlabeled target data. Here, we assume that we have access to a large amount
of unlabeled data in the target domain, while the labeled data are scant.

As a background, we observed that the pairwise distances between images in
feature space are fairly similar to each other (most are within $\pm 50$% of
the mean distance). That is, we cannot expect data points of one class be
clustered together in Euclidean space. In addition, we observed that for some
datasets, the nearest neighbors in feature space belong with high probability
($>90$%) to the same class. These two properties suggest a data distribution
of a class that resembles a loosely connected sparse graph, and the graphs of
different classes are tightly intermingled.

Based on these insights, we construct a few-shot learning method as follows:
given a few labeled points in feature space, we first propagate the labels to
nearby unlabeled points over nearest-neighbor links. Second, given the
resulting data-point distribution, we compute the kernel principal component
analysis (kernel PCA) reconstruction error [10] as a distance measure of a
test point to each class. The hypersurfaces of equal kernel PCA reconstruction
error have been shown to follow the shape of any data-point distribution [10].

Our experiments show that this new method outperforms state-of-the-art few-
shot learning methods, like ProtoNets and Adaptive Subspaces, particularly,
for 1-shot learning on certain data sets. We found that these datasets in
combination with a backbone network share a common property: in feature space,
the nearest neighbors are with high probability part of the same class.

Contributions. In summary, we make the following three main contributions:

  1. i

Provide a new perspective on how to interpret data in the feature space based
on our observation of pairwise data-point distances.

  2. ii

Introduce a new method, K-Prop, combining self-supervised learning, label
propagation, and kernel PCA for few-shot learning.

  3. iii

Show a relationship between high few-shot learning performance with our method
and data-point distances in feature space.

The remainder of this article is organized as follows: Section 2 describes
related work; Section 3 provides the background for our new method including
our observation of pairwise data-point distances and nearest-neighbor
relationships; Section 4 describes our proposed method; Section 5 our
experiments with the corresponding results in Section 6. Finally, Section 7
concludes with a summary, discussion of limits and potential risks to society,
and outlook for future work.

##  2 Related Work

For few-shot learning, a subset of methods deals with different means for
mapping from the feature space onto the image classes. This mapping is
generally much less complex than the mapping from the image space onto the
feature space, for which usually deep multi-layer convolutional neural
networks are used. This reduced complexity allows for adaptation to a target
dataset with only a few labels. Examples of this approach are ProtoNets [21]
and Adaptive Subspaces [20].

ProtoNets were introduced as an improvement over MatchingNets [23], which is a
popular few-shot learning technique, and Adaptive Subspaces were introduced as
an improvement over ProtoNets [20]. From a certain angle, our approach could
be viewed as an extension of Adaptive Subspaces to non-linear subspaces, which
we describe with kernel PCA. So, there is a natural progression of methods
from the reference points in ProtoNets over subspaces to kernel PCA (Fig. 2).

Label propagation has been suggested before. For example, Zhou et al. [26]
introduced label propagation through diffusion in a semi-supervised learning
setting if the data manifold is sufficiently smooth. More recently, authors
studied label propagation in feature space: Iscen et al. [12] construct a
k-nearest-neighborhood graph and propagate labels with a diffusion process.
Liu et al. [15] construct a neighborhood graph with a Gaussian similarity
matrix and learn the parameters for label propagation in a meta-learning
setting. Benato et al. [1] map from the feature space onto a t-SNE-generated
2-dimensional plane before propagating labels.

None of these works mentioned the property that we observed: that nearest
neighbors can be with high probability within the same class for certain
datasets and backbones.

##  3 Background

In this section, we provide the background for our new method: 1) self-
supervised learning of a backbone network, 2) our observation of pairwise
data-point distances, and 3) the kernel PCA reconstruction error.

###  3.1 Self-supervised learning

As unlabeled data are often less expensive to obtain, one common approach to
few-shot learning is to use so-called self-supervision, wherein a proxy task
is employed to pretrain a backbone network to produce features which can be
leveraged for the downstream few-shot classification task. Using self-
supervised learning to train a backbone network has been shown to rival
supervised training based on linear-classifier accuracy on the trained
features for certain datasets [4, 14].

Li et al. [14] train various architectures in the multistage vision
transformers (ViT) family using a self-supervision scheme which builds off of
DINO [2]. ViT architectures work by first splitting an image into a regular
grid of non-overlapping patches, flattening and (optionally) projecting the
patches, and then performing sparse multi-head attention on the collection of
patches.

In DINO, an exponential moving average ViT teacher network and a student
network of the same architecture are fed different augmentations (views) of
the full image, and the view-level features produced by each network are then
fed to a shared prediction head which maximizes agreement between the feature
representations of the two views. Li et al. [14] use an additional region-
level task which similarly enforces similarity of the top-level feature
representations of the various patches by first matching each student feature
to the most similar output feature of the same layer of the teacher network,
and then using the mean similarity of the resulting collection of feature
pairs as the loss (see [14] for full details).

###  3.2 Pairwise data-point distances

We investigated the geometric structure of images, as embedded in the high-
dimensional space where each pixel describes one dimension. For a selected
subset of Imagenet [7] classes, we computed pairwise distances between images
within each class and between classes. As a result, in the original image
space, the data points have almost the same distance to each other (Fig. 3).

Figure 3: Pairwise distances (within and between classes) in the original
image space for selected Imagenet classes.

Next, we mapped the images into the 512-dimensional feature space of a
ResNet18 network, which was pre-trained on Imagenet. Here, the pairwise
distances show more structure (Fig. 4), but a large part of the data still
have similar distances to each other. In addition, for the Imagenette dataset,
we computed the pairwise distances in the 512-dimensional feature space of an
Imagenet-pretrained ResNet18 network. For the intra-class pairwise distances,
we observed a mean of $23.25\pm 3.48$ SD ($n=4.5*10^{6}$). For the inter-class
distances, we observed $29.06\pm 3.15$ (mean $\pm$ SD, $n=4*10^{7}$). That is,
the difference between inter and intra distances is relatively small, which
matches the qualitative observation in Fig. 4.

Figure 4: Pairwise distances in the 512-dimensional feature space of an
Imagenet-pretrained Resnet18.

In addition, we evaluated the probability, $p_{NN}$, of a nearest neighbor in
feature space being in the same class. We computed this probability for the
Imagenet-pretrained ResNet18 features (Tab. 1) as well as for the EsViT
features (Tab. 2) for six different datasets (RESISC45, CUB, Imagenette,
EuroSat, CropDisease, and Fungi - see also Experiments). For RESISC45,
Imagenette, EuroSat, and CropDisease, we computed 100 trials, in each picking
either 5 or 10 classes at random. In each trial, we computed the pairwise
distances in feature space between all training data of the selected classes.
For CUB and Fungi, we computed $1,000$ trials.

As a result, in most cases, these probabilities, $p_{NN}$, were in the 90s%.
An outlier is CUB, for which $p_{NN}$ was low ($<60$%) for the EsViT features.

# of Classes | RESISC | CUB | Imagenette
---|---|---|---
5 | $94.5\pm 5.1$ | $90.0\pm 11.2$ | $98.1\pm 1.5$
10 | $89.5\pm 8.2$ | $82.6\pm 14.6$ | $96.7\pm 2.5$

# of Classes | EuroSat | Crop | Fungi
---|---|---|---
5 | $92.5\pm 6.5$ | $97.7\pm 4.6$ | $80.0\pm 19.0$
10 | $86.4\pm 9.8$ | $95.8\pm 6.5$ | $70.9\pm 20.7$

Table 1: Probability that nearest neighbor is within the same class for the
ImageNet-pretrained Resnet18 features (mean $\pm$ SD).

# of Classes | RESISC | CUB | Imagenette
---|---|---|---
5 | $98.7\pm 2.1$ | $59.7\pm 19.0$ | $92.0\pm 4.0$
10 | $97.1\pm 3.3$ | $43.8\pm 20.4$ | $87.8\pm 6.8$

# of Classes | EuroSat | Crop | Fungi
---|---|---|---
5 | $98.0\pm 1.5$ | $99.3\pm 1.6$ | $88.7\pm 13.3$
10 | $96.3\pm 2.2$ | $98.5\pm 2.6$ | $83.2\pm 15.5$

Table 2: Probability that nearest neighbor is within the same class for the
self-supervised trained EsViT features (mean $\pm$ SD).

Figure 5a shows a common illustration of the data-point distribution in
feature space. Based on our observations, we found this illustration to be
misleading because the points of one class are not locally clustered together.
This lack of clustering is evidenced by the small difference between intra-
class and inter-class distances. So, instead, we propose that the data are
distribution along sparse graphs, and the graphs between classes are
intermingled (Fig. 5b).

Figure 5: a) Common illustration of data point distributions in feature space,
b) our interpretation: intermingled sparse graphs. Each color denotes a
different class.

###  3.3 Kernel Principle Component Analysis

Kernel Principal Component Analysis is a kernelized version of the PCA
algorithm, essentially, expanding the linear method to non-linear data
distributions [19]. Kernel PCA uses the so-called “kernel trick,” i.e., the
PCA is computed in a high-dimensional (potentially, infinitely dimensional)
space, into which all data points, $\\{\bf{x}_{i}\\}$, are mapped, without
actually carrying out the mapping, ${\bf\Phi}(\bf{x}_{i})$, into this space
because the mapping appears only inside scalar products, and so, the scalar
product can be replaced with a kernel function in the original space. A common
kernel function is a Gaussian function,

$k({\bf x}_{i},{\bf x}_{j})=\exp(-||{\bf x}_{i}-{\bf
x}_{j}||^{2}/(2\sigma^{2}))\,.$ (1)

For this function, the corresponding mapping would actually be one into an
infinite-dimensional space. Practically, however, the dimensionality is
limited by the number of data points.

Computing kernel PCA involves computing the kernel matrix $K_{ij}=k({\bf
x}_{i},{\bf x}_{j})$, transforming $K$ to account for the non-zero mean of
$\\{{\bf\Phi}(\bf{x}_{i})\\}$, and extracting a number, $q$, of eigenvectors
corresponding to the $q$-largest eigenvalues [19, 10]. Here, for few-shot
learning, we are dealing with a low-dimensional kernel matrix, so the
computational cost of the eigenvalue extraction is negligible.

When computing kernel PCA for a non-linear data distribution, the
corresponding reconstruction error (analogue to the reconstruction error in
PCA) was introduced as a novelty-detection measure [10]. For Gaussian kernel
functions, it turned out that the equipotential curves/surfaces of the
reconstruction error describe well the non-linear shape of a data-point
distribution [10]. Thus, we use this reconstruction error to compare between
different classes in features space.

Let $\bf{z}$ be a vector in feature space and $\\{\bf{x}_{i}\\}$ be a
distribution of vectors in feature space. Then, the reconstruction error of
$\bf{z}$ is computed as follows [10],

$\displaystyle L_{RE}({\bf z})$ $\displaystyle=$ $\displaystyle k({\bf z},{\bf
z})-\frac{2}{n}\sum_{i=1}^{n}k({\bf z},{\bf x}_{i})$ (2) $\displaystyle+$
$\displaystyle\frac{1}{n^{2}}\sum_{i,j=1}^{n}k({\bf x}_{i},{\bf
x}_{j})-\sum_{l=1}^{q}f_{l}({\bf z})^{2}\enspace,$

where $f_{l}$ are the projections onto the principal components,

$\displaystyle f_{l}({\bf z})$ $\displaystyle=$
$\displaystyle\sum_{i=1}^{n}\alpha_{i}^{l}\,\Bigg{[}k({\bf z},{\bf
x}_{i})-\frac{1}{n}\sum_{r=1}^{n}k({\bf x}_{i},{\bf x}_{r})$ (3)
$\displaystyle-$ $\displaystyle\frac{1}{n}\sum_{r=1}^{n}k({\bf z},{\bf
x}_{r})+\frac{1}{n^{2}}\sum_{r,s=1}^{n}k({\bf x}_{r},{\bf x}_{s})\Bigg{]}\,,$

and $\alpha_{i}^{l}$ are the eigenvectors of $K$. When using a Gaussian
kernel, we have two hyperparameters, the width, $\sigma$, and the number of
principal components, $q$.

##  4 Proposed Method

Based on the above background, we propose a new few-shot learning method,
K-Prop, using self-supervised learning, label propagation, and the kernel PCA
reconstruction error (Fig. 6). First, we exploit the fact that for many
datasets, a good backbone, and thus a good mapping onto a feature space, can
be obtained with self-supervised pre-training. Second, we exploit the nearest-
neighbor within-class connections to artificially expand the number of labels
with label propagation, and third, we exploit that the reconstruction error
for kernel PCA can describe the sparse graph-like distribution of the few
labeled points. In the following, we describe those three elements in more
detail.

Figure 6: Process flow of our new method, K-Prop.

1) For self-supervised learning (SSL), we suggest the EsViT method [14]. We
use SSL on the target training data (but not on the test data) to avoid cross-
domain transfer loss. After training, we freeze the backbone parameters and
weights.

2) For label propagation, we iteratively add a fixed number of extra labels.
So, we propagate only into the neighborhood of the given labels, instead of
diffusing into the entire unlabeled set. In each iteration step, we add only
one unlabeled data point: in feature space, we find the point ${\bf x}_{j}$
with the smallest Euclidean distance to any of the points ${\bf x}_{i}$ in the
set of labeled points, i.e.,

$j=\mbox{argmin}_{j}\mbox{min}_{i}||{\bf x}_{i}-{\bf x}_{j}||\,.$ (4)

The resulting ${\bf x}_{j}$ is then added to the set of labeled points, and
the iteration continues until a given number of points is added to the number
of originally labeled points. Figure 7 illustrates one example of this
iterative process. Here, adding a nearest neighbor is shown as a link in a
graph. For 1-shot learning, this process results in a sparse graph, and when
starting with more labels, we generally end up with a multitude of graphs.

Figure 7: Label propagation: we iteratively add one unlabeled nearest neighbor
at a time (a to d) building a graph of labeled points for each class.

3) We compute kernel PCA for each set of labeled feature points $\\{{\bf
x}_{i}\\}$ for each class separately. Here, the labeled data contain the extra
labels from the label propagation. When presenting a new test image, we first
compute its mapping into the feature space, ${\bf z}$, and then compute the
reconstruction error, $L_{RE}^{c}({\bf z})$ for each class, $c$. We classify
the test image based on the smallest reconstruction error,
$\mbox{argmin}_{c}L_{RE}^{c}({\bf z})$.

RESISC
# of Shots ProtoNets MatchingNets Subspaces Ours (EsViT) Ours (Resnet18) 1
$60.66\pm 0.9$ $60.21\pm 0.9$ $60.55\pm 0.8$ $\mathbf{83.18\pm 0.6}$ $71.15\pm
0.7$ 2 $72.18\pm 0.7$ $69.59\pm 0.7$ $69.17\pm 0.7$ $\mathbf{87.06\pm 0.5}$
$76.52\pm 0.6$ 5 $81.21\pm 0.5$ $75.75\pm 0.5$ $79.16\pm 0.5$
$\mathbf{92.08\pm 0.4}$ $84.84\pm 0.5$
Crop
# of Shots ProtoNets MatchingNets Subspaces Ours (EsViT) Ours (Resnet18) 1
$77.40\pm 0.8$ $\mathbf{79.33\pm 0.8}$ $75.87\pm 0.8$ $78.11\pm 0.8$ $78.53\pm
0.7$ 2 $\mathbf{89.08\pm 0.6}$ $85.21\pm 0.7$ $81.41\pm 0.6$ $85.20\pm 0.6$
$83.56\pm 0.6$ 5 $91.38\pm 0.4$ $90.76\pm 0.4$ $91.95\pm 0.5$
$\mathbf{92.82\pm 0.4}$ $91.20\pm 0.4$
EuroSat
# of Shots ProtoNets MatchingNets Subspaces Ours (EsViT) Ours (Resnet18) 1
$39.93\pm 0.7$ $37.40\pm 0.7$ $27.55\pm 0.6$ $\mathbf{77.20\pm 0.5}$ $67.90\pm
0.7$ 2 $44.97\pm 0.7$ $39.57\pm 0.5$ $36.85\pm 0.6$ $\mathbf{83.40\pm 0.4}$
$74.54\pm 0.6$ 5 $55.08\pm 0.5$ $44.86\pm 0.6$ $40.42\pm 0.5$
$\mathbf{90.44\pm 0.3}$ $82.86\pm 0.4$
Imagenette
# of Shots ProtoNets MatchingNets Subspaces Ours (EsViT) Ours (Resnet18) 1
$31.57\pm 0.6$ $28.95\pm 0.6$ - $\mathbf{74.73\pm 0.6}$ $91.47\pm 0.4$ 2
$30.99\pm 0.5$ $30.64\pm 0.5$ - $\mathbf{81.12\pm 0.4}$ $94.77\pm 0.2$ 5
$37.76\pm 0.5$ $36.98\pm 0.5$ - $\mathbf{84.90\pm 0.4}$ $97.28\pm 0.1$
CUB
# of Shots ProtoNets MatchingNets Subspaces Ours (EsViT) Ours (Resnet18) 1
$71.88\pm 0.9$ $72.36\pm 0.9$ $63.30\pm 0.9$ $40.79\pm 0.7$ $\mathbf{81.08\pm
0.8}$ 2 $81.44\pm 0.6$ $80.18\pm 0.7$ $68.38\pm 0.8$ $46.44\pm 0.9$
$\mathbf{83.77\pm 0.7}$ 5 $87.42\pm 0.5$ $83.64\pm 0.6$ $78.25\pm 0.6$
$53.59\pm 1.0$ $\mathbf{89.39\pm 0.6}$
Fungi
# of Shots ProtoNets MatchingNets Subspaces Ours (EsViT) Ours (Resnet18) 1
$58.87\pm 0.9$ $\mathbf{60.79\pm 0.9}$ $35.34\pm 0.8$ $47.26\pm 1.1$ $42.87\pm
1.1$ 2 $\mathbf{74.65\pm 0.8}$ $74.21\pm 0.7$ $39.12\pm 0.8$ $68.18\pm 1.3$
$56.93\pm 1.2$ 5 $\mathbf{82.13\pm 0.6}$ $80.64\pm 0.8$ $63.41\pm 0.7$
$74.29\pm 1.2$ $64.21\pm 1.2$

Table 3: Comparing the performance of our method (with either EsViT or
Resnet18 backbone) with ProtoNets, MatchingNets, and Adaptive Subspaces on the
RESISC, CropDisease, Eurosat and Imagenette datasets (mean $\pm$ SE). We
disregard the Imagenette results with Imagenet-pretrained Resnet18 when
comparing with other methods because of unfair advantage. The Adpative
Subspaces method failed to converge when training on Imagenette, and so the
results are missing.

##  5 Experiments

We evaluate our method on six datasets and compare it against three other
state-of-the-art methods and in ablation studies replacing key elements of
K-Prop. Moreover, we consider features produced from backbones trained either
by fully-supervised pretraining on a dissimilar source domain or by self-
supervision with zero labels on the target domain. In the former case, we used
a Resnet18 network with frozen weights, which has been pretrained on
Imagenet-1k. In the latter case, we trained a multistage transformer
architecture with the self-supervision scheme proposed for EsViT [14].

For EsViT, we used the tiny sliding window architecture (Swin-T [14]). Images
were divided into non-overlapping $16\times 16$ pixel patches and two
additional (global) random crops of size $224\times 224$, all of which were
subject to random transformations (augmentations) as described in [3]. We used
a base learning rate of 0.0005 and cosine annealing, with a weight decay
scaling linearly from 0.04 to 0.4 over 300 epochs. The network was trained for
300 epochs or until loss convergence.

For label propagation, we added 4 extra labels for 1-shot learning, 3 extra
labels for 2-shot learning, and 2 extra labels for 5- and 10-shot learning.
For 1-shot learning, the accuracies started to saturate by the 4th extra
label, and too many extra labels will hurt because of pollution with wrong
labels (see Appendix, Fig. 10).

For kernel PCA, we used a Gaussian kernel with width $\sigma=16$ and
$q=\lfloor 2k/3+1\rfloor$ principal components, where $k$ is the number of
labels per class (before label propagation).

We compare our method against ProtoNets [21], MatchingNets [23], and Adaptive
Subspaces [20]. Each of the methods was implemented with a Resnet18 backbone,
while otherwise following the training and augmentation routines in [5, 20].
Since these methods require meta learning to adapt parameters, we trained all
three models using the labels for half of the classes in the dataset, while
the other half was used to evaluate the few-shot learning performance. In
contrast, using EsViT, our method did not use any labels (apart from few-shot)
but used all classes of the unlabeled training set.

For our target datasets, we used Imagenette [11], RESISC45 [6], CropDisease
[16], EuroSat [9], Fungi [18], and CUB [24]. Imagenette is distributed under
the Apache license, CropDisease under Creative Commons 1.0 Universal, Fungi
under MIT, and CUB under Attribution 4.0 International. The licenses for
RESISC45 and EuroSat are unknown. For all datasets, we used the default
training/test data split.

For each dataset and for each pre-training scheme, we evaluate 5-way, k-shot
performance using $k=1,2,5$. For each $k$, we randomly generate $1,000$ tasks
by sampling, for each task, 5 uniformly random classes from the test set and,
then average the classification accuracies over all tasks. The backbone
weights were frozen after pretraining. For each task, we evaluate our method
using either the EsViT or Resnet18 backbone, compare against other methods and
do ablation studies with our method. For these ablation studies, we replaced
kernel PCA with a linear classifier and tested the linear classifier either
with or without label propagation.

##  6 Results

On several datasets, our new method, K-Prop, outperformed state-of-the-art
methods (Tab. 3). For example, for 1-shot learning on RESISC45, K-Prop had a
83% accuracy compared to 61% for ProtoNets (Fig. 1). As another comparison,
the best known 1-shot 5-way accuracy reported in the literature for RESISC45
with a Resnet18 backbone is $64.6$% [13].

Interestingly, on this dataset, the nearest-neighbors in feature space were
with high probability ($>98$%) part of the same class if trained with EsViT
and higher compared to the Resnet18 backbone. Generally, we found that high
$p_{NN}$ for a dataset and backbone corresponded to a high few-shot
classification accuracy (Fig. 8).

In addition, our method outperformed the linear classifier on the RESISC45,
EuroSat, Imagenette, CUB, and CropDisease datasets for 1-, 2-, and 5-shot
learning (Fig. 9). For low $p_{NN}$ ($<90$%, Fungi and CUB with EsViT), we
found that label propagation hurt the performance and a standard linear
classifier was better. Moreover, our ablation studies showed that using the
kernel PCA reconstruction error provided a boost over a linear classifier in
most settings (Fig. 9, see Appendix for the corresponding numerical values).

Figure 8: Comparison of 1-shot accuracy boost for EsViT relative to Resnet18
as function of difference in probability, $p_{NN}$, that nearest-neighbors are
in the same class. Each data point corresponds to a different dataset (mean
$\pm$ SE).

Figure 9: Ablation study with a linear classifier (Linear), label propagation
followed by a linear classifier (Prop-Linear), and label propagation followed
by kernel PCA (Prop-KPCA) for self-supervised features (EsViT) and features
from a Resnet18 backbone pretrained on Imagenette (mean $\pm$ SE).

##  7 Conclusions

We provided a new perspective for looking at data distributions in feature
space and showed how to exploit it for few-shot learning. Based on our
observations, data in feature space tend to be loosely connected through
nearest-neighbor connections, and the resulting sparse graphs are intermingled
between different classes. Therefore, moderate label propagation followed by
classification based on the kernel PCA reconstruction error showed promising
results for few-shot learning. Moreover, we observed that a high probability
of nearest neighbors being in the same class was indicative of a high
classification accuracy.

The comparison with other methods was not exactly equal: ProtoNets,
MatchingNets, and Adaptive Subspaces had additional labels for pretraining; on
the other hand, they lacked full access to all classes of the unlabeled data,
which could bias the results in one or the other way (see Appendix for
additional comparisons). Moreover, using an Imagenet-pretrained Resnet18 is an
unfair advantage for datasets that share similar images (mainly, Imagenette
but also CUB to some extent), but we included those because of the interesting
relationship regarding the above nearest-neighbor same-class probability.

Limitations: The above relationship shows that the strategy of using SSL
before our label propagation is limited to datasets in which SSL can move
same-class data points close together in feature space. This strategy will
fail if the inter-class difference is small: e.g., the CUB dataset consists of
birds of similar shape, which belong to different classes due to relatively
small differences in texture. On the other hand, we found that we can still
boost 1-shot learning performance if we use label propagation and kernel PCA
on an Imagenet-pretrained network. This alternative in turn is limited to
datasets that share similarities in features to Imagenet. Finally, despite the
large improvements that we have seen for some datasets, the 1-shot accuracies
are still too low for applications requiring reliable automated decision
making and instead are more suitable for applications with a human in the loop
or with multiple redundancies.

Potential negative societal impact: Given the above limitations, the risk to
society is low. Mainly, we provide an insight into feature-space
distributions. In terms of applications, the most promising would be to update
a target tracking system with only a few labels, where the system
automatically filters input data (e.g., a video stream) for later analysis by
a human operator. Like any tool, this system could potentially be abused by a
bad actor, who would gain increased situational awareness. As mitigation,
since the accuracies are still not high, human verification is likely needed,
limiting the potential abuse. In terms of environmental impact, this work has
a positive contribution, reducing the required computational time for updating
a model to new data because only kernel PCA has to be recomputed with a frozen
backbone.

Future work: This work could inspire future work in the geometric properties
of feature-space distributions leading to a better understanding of such
distributions under various learning paradigms (unsupervised, self-supervised,
and supervised). As part of that work, it would be interesting to find means
to estimate $p_{NN}$ without requiring labeled data. In addition, we plan to
explore new ways to further improve the few-shot learning performance.

## Acknowledgments

We thank Drs Soheil Kolouri and Navid Naderializadeh for discussions related
to this work and Dr Kolouri for his help getting funding for this effort. This
material is based upon work supported by the United States Air Force under
Contract No. FA8750-19-C-0098. Any opinions, findings, and conclusions or
recommendations expressed in this material are those of the author(s) and do
not necessarily reflect the views of the United States Air Force and DARPA.

## References

  * [1] Barbara Caroline Benato, Jancarlo Ferreira Gomes, Alexandru Cristian Telea, and Alexandre Xavier Falcão.  Semi-supervised deep learning based on label propagation in a 2D embedded space.  CoRR, abs/2008.00558, 2020.
  * [2] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  2021\.
  * [3] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin.  Emerging properties in self-supervised vision transformers.  CoRR, abs/2104.14294, 2021.
  * [4] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey E. Hinton.  A simple framework for contrastive learning of visual representations.  CoRR, abs/2002.05709, 2020.
  * [5] Wei-Yu Chen, Yen-Cheng Liu, Zsolt Kira, Yu-Chiang Frank Wang, and Jia-Bin Huang.  A closer look at few-shot classification.  CoRR, abs/1904.04232, 2019.
  * [6] Gong Cheng, Junwei Han, and Xiaoqiang Lu.  Remote sensing image scene classification: Benchmark and state of the art.  Proceedings of the IEEE, 105(10):1865–1883, Oct 2017.
  * [7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 248–255. Ieee, 2009.
  * [8] Chelsea Finn, Pieter Abbeel, and Sergey Levine.  Model-agnostic meta-learning for fast adaptation of deep networks.  In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 1126–1135. JMLR.org, 2017.
  * [9] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth.  Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.  IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019.
  * [10] Heiko Hoffmann.  Kernel PCA for novelty detection.  Pattern Recognition, 40(3):863–874, 2007.
  * [11] Jeremy Howard.  Imagenette.
  * [12] Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondrej Chum.  Label propagation for deep semi-supervised learning.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
  * [13] Ashraful Islam, Chun-Fu Chen, Rameswar Panda, Leonid Karlinsky, Rogério Feris, and Richard J. Radke.  Dynamic distillation network for cross-domain few-shot recognition with unlabeled data.  CoRR, abs/2106.07807, 2021.
  * [14] Chunyuan Li, Jianwei Yang, Pengchuan Zhang, Mei Gao, Bin Xiao, Xiyang Dai, Lu Yuan, and Jianfeng Gao.  Efficient self-supervised vision transformers for representation learning, 2021.
  * [15] Yanbin Liu, Juho Lee, Minseop Park, Saehoon Kim, and Yi Yang.  Transductive propagation network for few-shot learning.  In International Conference on Learning Representations, 2019.
  * [16] Sharada P. Mohanty, David P. Hughes, and Marcel Salathé.  Using deep learning for image-based plant disease detection.  Frontiers in Plant Science, 7:1419, 2016.
  * [17] Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine.  Meta-learning with implicit gradients.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d’ Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
  * [18] Brigit Schroeder and Yin Cui.  FGVCx fungi classification challenge, 2018.
  * [19] Bernhard Schölkopf, Alexander Smola, and Klaus-Robert Müller.  Nonlinear component analysis as a kernel eigenvalue problem.  Neural Computation, 10:1299–1319, 1998.
  * [20] Christian Simon, Piotr Koniusz, Richard Nock, and Mehrtash Harandi.  Adaptive subspaces for few-shot learning.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4135–4144, 2020.
  * [21] Jake Snell, Kevin Swersky, and Richard S. Zemel.  Prototypical networks for few-shot learning, 2017.
  * [22] Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B. Tenenbaum, and Phillip Isola.  Rethinking few-shot image classification: a good embedding is all you need?  CoRR, abs/2003.11539, 2020.
  * [23] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Koray Kavukcuoglu, and Daan Wierstra.  Matching networks for one shot learning.  In D. Lee, M. Sugiyama, U. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc., 2016.
  * [24] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie.  The Caltech-UCSD Birds-200-2011 Dataset.  Technical Report CNS-TR-2011-001, California Institute of Technology, 2011\.
  * [25] Ruixiang Zhang, Tong Che, Zoubin Ghahramani, Yoshua Bengio, and Yangqiu Song.  MetaGAN: An adversarial approach to few-shot learning.  NIPS’18, page 2371–2380, Red Hook, NY, USA, 2018. Curran Associates Inc.
  * [26] Dengyong Zhou, Olivier Bousquet, Thomas Lal, Jason Weston, and Bernhard Schölkopf.  Learning with local and global consistency.  In S. Thrun, L. Saul, and B. Schölkopf, editors, Advances in Neural Information Processing Systems, volume 16. MIT Press, 2004.

##  Appendix A Appendix

In this Appendix, we present 1) the dependence of K-Prop on the number of
propagated labels for 1-shot learning, 2) the relationship between the 1-shot
accuracy and the probability that nearest neighbors are in the same class, 3)
additional comparisons between K-Prop and the three other methods from the
literature: MatchingNets, ProtoNets, and Adaptive Subspaces, and 4) numerical
values for our ablation studies.

###  A.1 Dependence on number of extra labels

Figure 10 shows the dependence of the 1-shot accuracy for K-Prop (with EsViT
backbone) on the number, $M$, of labels added by our label propagation method.
For each number of labels added, we sampled one hundred random 5-way tasks and
plot the mean. We evaluated each value of $M$ from $1$ to $10$, and
subsequently $M=15$, $20$, … , $100$. The maximum accuracy depended on the
dataset used, e.g., for RESISC45, the peak was at $M=45$, while it was at a
lower value for CropDisease and a higher value for EuroSat. In our main
experiments, we chose $M=4$ for all datasets for simplicity and computational
speed (since we did $1,000$ trials in each setting), but for EuroSat we could
have gotten significantly better results using $M=95$ instead of $4$.

Figure 10: Dependence of 5-way, 1-shot accuracy on number of labels added
using our label propagation.

###  A.2 Accuracy vs nearest-neighbor probability

Figure 11 shows the absolute values for the relationship between 1-shot
learning accuracy and the probability that nearest-neighbors are in the same
class, $p_{NN}$. For each dataset, we evaluated three different backbone
networks: Resnet18 trained on Imagenet1k, EsViT trained on Imagenet1k, and
EsViT trained on the target data.

Figure 11: Accuracy for 1-shot learning vs probability that nearest-neighbors
are in the same class, $p_{NN}$ (mean $\pm$ SE). Each data point corresponds
to a different dataset and backbone.

###  A.3 Comparison with other methods

We present additional results comparing K-Prop to other methods, using again
ProtoNets, MatchingNets, and Subspaces for comparison. Here, we used a fixed
backbone (either Resnet18 or EsViT) pretrained on Imagenet1k for each method.
For each of the meta-learning algorithms, we carried out meta-training using
mini-Imagenet. We then evaluated each method on the RESISC45, CropDisease,
EuroSat, CUB, and Fungi datasets.

For K-Prop with EsViT, no label information from Imagenet1k or mini-Imagenet
is used, while for K-Prop with Resnet18, we used the same Imagenet1k
pretrained weights as with the other methods, but no additional training using
mini-Imagenet. The comparisons using a Resnet18 backbone are shown in Tab. 5,
while the comparisons using an EsViT backbone are shown in Tab. 5. Despite
K-Prop with EsViT being at a disadvantage compared to ProtoNets and
MatchingNets, it still outperformed both in most cases. For this comparison
with the Imagenet-pretrained EsViT, we omitted Subspaces due to computation-
time limits.

RESISC
# of Shots ProtoNets MatchingNets Subspaces Ours 1 $49.47\pm 0.8$ $40.45\pm
0.7$ $49.14\pm 0.8$ $\mathbf{71.15\pm 0.7}$ 2 $57.45\pm 0.7$ $53.50\pm 0.7$
$55.57\pm 0.7$ $\mathbf{76.52\pm 0.6}$ 5 $69.9\pm 0.6$ $64.03\pm 0.6$
$67.12\pm 0.6$ $\mathbf{84.84\pm 0.5}$
CropDisease
# of Shots ProtoNets MatchingNets Subspaces Ours 1 $57.16\pm 0.9$ $46.36\pm
0.9$ $51.01\pm 0.9$ $\mathbf{78.53\pm 0.7}$ 2 $69.76\pm 0.8$ $53.75\pm 0.9$
$55.28\pm 0.8$ $\mathbf{83.56\pm 0.6}$ 5 $80.77\pm 0.7$ $62.38\pm 0.9$
$70.37\pm 0.7$ $\mathbf{91.20\pm 0.4}$
EuroSat
# of Shots ProtoNets MatchingNets Subspaces Ours 1 $60.58\pm 0.9$ $48.04\pm
0.8$ $45.94\pm 0.8$ $\mathbf{67.90\pm 0.7}$ 2 $67.38\pm 0.8$ $60.66\pm 0.8$
$53.61\pm 0.8$ $\mathbf{74.54\pm 0.6}$ 5 $77.67\pm 0.7$ $68.50\pm 0.8$
$69.79\pm 0.7$ $\mathbf{82.86\pm 0.4}$
CUB
# of Shots ProtoNets MatchingNets Subspaces Ours 1 $38.95\pm 0.8$ $37.42\pm
0.8$ $35.60\pm 0.7$ $\mathbf{81.08\pm 0.8}$ 2 $46.02\pm 0.8$ $42.36\pm 0.7$
$40.24\pm 0.7$ $\mathbf{83.77\pm 0.7}$ 5 $56.62\pm 0.8$ $49.19\pm 0.8$
$48.92\pm 0.7$ $\mathbf{89.39\pm 0.6}$
Fungi
# of Shots ProtoNets MatchingNets Subspaces Ours 1 $36.19\pm 0.7$ $34.17\pm
0.7$ $31.76\pm 0.7$ $\mathbf{42.87\pm 1.1}$ 2 $42.25\pm 0.8$ $38.00\pm 0.7$
$33.92\pm 0.7$ $\mathbf{56.93\pm 1.2}$ 5 $52.05\pm 0.7$ $44.93\pm 0.7$
$41.65\pm 0.7$ $\mathbf{64.21\pm 1.2}$

Table 4: Comparing the performance of our method with ProtoNets, MatchingNets,
and Adaptive Subspaces, all using Resnet18 backbones pre-trained on
Imagenet1k.

RESISC
# of Shots ProtoNets MatchingNets Ours 1 $72.82\pm 0.8$ $72.56\pm 0.9$
$\mathbf{78.60\pm 1.0}$ 2 $82.08\pm 0.7$ $78.56\pm 0.8$ $\mathbf{82.80\pm
0.9}$ 5 $\mathbf{89.48\pm 0.5}$ $85.55\pm 0.6$ $89.28\pm 0.9$
CropDisease
# of Shots ProtoNets MatchingNets Ours 1 $81.97\pm 0.8$ $81.94\pm 0.8$
$\mathbf{86.23\pm 0.6}$ 2 $89.54\pm 0.6$ $87.37\pm 0.7$ $\mathbf{90.06\pm
0.5}$ 5 $94.61\pm 0.4$ $92.05\pm 0.6$ $\mathbf{95.43\pm 0.3}$
EuroSat
# of Shots ProtoNets MatchingNets Ours 1 $65.17\pm 0.8$ $64.77\pm 0.9$
$\mathbf{70.85\pm 0.8}$ 2 $76.22\pm 0.7$ $70.90\pm 0.8$ $\mathbf{77.16\pm
0.6}$ 5 $84.34\pm 0.5$ $77.36\pm 0.6$ $\mathbf{84.73\pm 0.4}$
CUB
# of Shots ProtoNets MatchingNets Ours 1 $71.00\pm 1.0$ $71.77\pm 1.0$
$\mathbf{80.00\pm 0.8}$ 2 $82.23\pm 0.9$ $77.68\pm 0.9$ $\mathbf{84.04\pm
0.6}$ 5 $89.47\pm 0.7$ $85.54\pm 0.8$ $\mathbf{90.51\pm 0.6}$
Fungi
# of Shots ProtoNets MatchingNets Ours 1 $57.26\pm 1.3$ $\mathbf{57.97\pm
1.3}$ $50.24\pm 1.1$ 2 $\mathbf{68.11\pm 1.1}$ $66.64\pm 1.2$ $65.98\pm 1.1$ 5
$\mathbf{77.78\pm 1.1}$ $76.18\pm 1.0$ $71.49\pm 1.2$

Table 5: Comparing the performance of our method with ProtoNets and
MatchingNets all using EsViT backbones pre-trained on Imagenet1k.

###  A.4 Ablation study

We provide the results of Fig. 9 in tabular form in Tables 8, 8, and 8.

Dataset Pretraining Linear LP+Linear LP+KPCA NWPU-RESISC45 EsViT 75.93$\pm$
0.6 80.31$\pm$ 0.6 83.18$\pm$ 0.6 Pretrained ResNet18 63.29$\pm$ 0.6 71.4$\pm$
0.7 72.09$\pm$ 0.7 EuroSat EsViT 69.13$\pm$ 0.5 72.62$\pm$ 0.5 77.2$\pm$ 0.5
Pretrained ResNet18 56.78$\pm$ 0.6 66.79$\pm$ 0.7 68.77$\pm$ 0.7 CropDisease
EsViT 74.9$\pm$ 0.8 75.77$\pm$ 0.8 78.11$\pm$ 0.8 Pretrained ResNet18
72.56$\pm$ 0.7 80.96$\pm$ 0.7 81.12$\pm$ 0.7 Fungi EsViT 57.11$\pm$ 1.0
50.61$\pm$ 1.0 47.26$\pm$ 1.1 Pretrained ResNet18 48.76$\pm$ 0.9 43.41$\pm$
1.0 42.87$\pm$ 1.1 Imagenette EsViT 71.78$\pm$ 0.7 72.69$\pm$ 0.7 74.73$\pm$
0.6 Pretrained ResNet18 83.68$\pm$ 0.4 89.74$\pm$ 0.4 88.54$\pm$ 0.4 CUB EsViT
41.63$\pm$ 0.6 39.79$\pm$ 0.7 40.8$\pm$ 0.7 Pretrained ResNet18 68.65$\pm$ 0.7
78.93$\pm$ 0.8 81.08$\pm$ 0.8

Table 6: 5-way, 1-shot performance of linear fine-tuning (Linear), label
propagation + linear fine-tuning (LP+Linear), and label propagation + KPCA
(LP+KPCA) (ours).

Dataset Pretraining Linear LP+Linear LP+KPCA NWPU-RESISC45 EsViT 83.57$\pm$
0.5 85.12$\pm$ 0.4 87.06$\pm$ 0.5 Pretrained ResNet18 73.39$\pm$ 0.5
76.84$\pm$ 0.6 77.1$\pm$ 0.6 EuroSat EsViT 78.01$\pm$ 0.4 79.1$\pm$ 0.4
83.4$\pm$ 0.4 Pretrained ResNet18 69.68$\pm$ 0.5 73.52$\pm$ 0.5 74.53$\pm$ 0.6
CropDisease EsViT 83.1$\pm$ 0.7 83.41$\pm$ 0.7 85.2$\pm$ 0.7 Pretrained
ResNet18 82.81$\pm$ 0.5 85.71$\pm$ 0.5 86.0$\pm$ 0.6 Fungi EsViT 66.25$\pm$
0.9 64.44$\pm$ 1.1 68.18$\pm$ 1.4 Pretrained ResNet18 58.24$\pm$ 1.0
53.22$\pm$ 1.0 56.93$\pm$ 1.3 Imagenette EsViT 79.05$\pm$ 0.5 80.15$\pm$ 0.5
81.12$\pm$ 0.4 Pretrained ResNet18 92.03$\pm$ 0.3 93.13$\pm$ 0.2 92.29$\pm$
0.3 CUB EsViT 45.87$\pm$ 0.6 45.59$\pm$ 0.7 46.44$\pm$ 0.9 Pretrained ResNet18
79.01$\pm$ 0.6 83.34$\pm$ 0.6 83.77$\pm$ 0.7

Table 7: 5-way, 2-shot performance.

Dataset Pretraining Linear LP+Linear LP+KPCA NWPU-RESISC45 EsViT 90.6$\pm$ 0.3
90.37$\pm$ 0.3 92.07$\pm$ 0.4 Pretrained ResNet18 83.9$\pm$ 0.4 84.45$\pm$ 0.4
84.82$\pm$ 0.5 EuroSat EsViT 88.13$\pm$ 0.3 87.88$\pm$ 0.3 90.44$\pm$ 0.3
Pretrained ResNet18 80.96$\pm$ 0.4 81.3$\pm$ 0.4 82.54$\pm$ 0.4 CropDisease
EsViT 91.28$\pm$ 0.4 91.4$\pm$ 0.4 92.82$\pm$ 0.4 Pretrained ResNet18
91.43$\pm$ 0.4 91.65$\pm$ 0.4 92.79$\pm$ 0.4 Fungi EsViT 77.85$\pm$ 0.8
75.22$\pm$ 1.0 74.29$\pm$ 1.3 Pretrained ResNet18 69.79$\pm$ 0.9 63.77$\pm$
1.0 64.21$\pm$ 1.3 Imagenette EsViT 84.89$\pm$ 0.3 85.12$\pm$ 0.3 84.9$\pm$
0.4 Pretrained ResNet18 96.15$\pm$ 0.1 96.2$\pm$ 0.1 95.97$\pm$ 0.2 CUB EsViT
54.05$\pm$ 0.6 53.62$\pm$ 0.6 53.59$\pm$ 1.0 Pretrained ResNet18 88.25$\pm$
0.4 88.66$\pm$ 0.4 89.39$\pm$ 0.7

Table 8: 5-way, 5-shot performance.