Assessing the Value of Transfer Learning Metrics for RF Domain Adaptation

Lauren J. Wong

National Security Institute, Virginia Tech

Intel AI Lab
Santa Clara

Sean McPherson

Intel AI Lab
Santa Clara

Alan J. Michaels

National Security Institute, Virginia Tech

The use of TL techniques has become common practice in fields such as CV and NLP.
Leveraging prior knowledge gained from data with different distributions, TL offers higher performance and reduced training time, but has yet to be fully utilized in applications of ML and DL techniques to applications related to wireless communications, a field loosely termed RFML.
This work begins this examination by evaluating the how RF domain changes encourage or prevent the transfer of features learned by CNN-based automatic modulation classifiers.
Additionally, we examine existing transferability metrics, LEEP and LogME, as a means to both select source models for RF domain adaptation and predict post-transfer accuracy without further training.

<concept_desc>General and reference General literature</concept_desc>
<concept_desc>Computing methodologies Learning paradigms</concept_desc>
<concept_desc>Computing methodologies Learning settings</concept_desc>

[300]General and reference General literature
[500]Computing methodologies Learning paradigms
[500]Computing methodologies Learning settings

§ INTRODUCTION

Following the release of the DARPA RFMLS program in 2017 <cit.>, research into the application of ML and DL techniques to wireless communications problems has risen significantly.
The field of RFML continues to be the topic of many government research programs and academic conferences including the IARPA SCISRS program <cit.> and the DySPAN <cit.>.
As a result, advances in RFML have yielded increased performance and flexibility in applications such as spectrum awareness, cognitive radio, and networking while reducing the need for expert-defined pre-processing and feature extraction techniques  <cit.>.

Traditional Machine Learning

Transfer Learning

In traditional ML (Fig. <ref>), a new model is trained from random initialization for each domain/task pairing. TL (Fig. <ref>) utilizes prior knowledge learned on one domain/task, in the form of a pre-trained model, to improve performance on a second domain and/or task. A concrete example for environmental adaptation to SNR is given in blue.

Current state-of-the-art RFML techniques rely upon supervised learning techniques trained from random initialization, and thereby assume the availability of a large corpus of labeled training data (synthetic, captured, or augmented <cit.>), which is representative of the anticipated deployed environment.
Over time, this assumption inevitably breaks down as a result of changing hardware and channel conditions, and as a consequence, performance degrades significantly <cit.>.
TL techniques can be used to mitigate these performance degradations by using prior knowledge obtained from a source domain and task, in the form of learned representations, to improve performance on a “similar" target domain and task using less data, as depicted in Fig. <ref>.

A system overview of the RF hardware and channel environment simulated in this work with the parameters/variables ($\alpha[t]$, $\omega[t]$, $\theta[t]$, $\nu[t]$, $\omega_\Delta[t]$) that each component of the system has the most significant impact on.

Though TL techniques have demonstrated significant benefits in fields such as CV and NLP <cit.>, including higher performing models, significantly less training time, and far fewer training samples <cit.>, <cit.> showed that the use of TL in RFML is currently lacking through the construction of an RFML specific TL taxonomy.

This work addresses current limitations in understanding how the training domain in particular, characterized by the RF hardware and the channel environment <cit.> depicted in Fig. <ref>, impacts learned behavior and therefore facilitates or prevents successful transfer.

The contributions of this work are three fold:
First, this work systematically evaluates RF domain adaptation performance as a function of several parameters of interest for an AMC use-case <cit.>:
@ SNR, which represents a change in the RF channel environment (i.e., an increase/decrease in the additive interference, $\nu[t]$, of the channel) and/or transmitting devices (i.e., an increases/decrease in the magnitude, $\alpha[t]$, of the transmitted signal) and an environment adaptation problem,
@ FO, which represents a change in the transmitting and/or receiving devices (i.e., an increase/decrease in $\omega_\Delta[t]$ due to hardware imperfections or a lack of synchronization) and a platform adaptation problem, and
@ Both SNR and FO, representing a change in both the RF channel environment and the transmitting/receiving devices and an environment platform co-adaptation problem.

These three parameter sweeps address each type of RF domain adaptation discussed in the RFML TL taxonomy <cit.>, and resulted in the construction of 82 training sets, 82 validation sets, and 82 test sets and the training and evaluation of 4442 models.
From these experiments, we identify a number of practical takeaways for how best to perform RF domain adaptation including how changes in SNR and FO impact the difficulty of AMC, an intuitive notion of source/target domain similarity induced by these parameter sweeps, and the impact these have on transfer performance, as well as a comparison of head re-training versus fine-tuning for RF TL.

The second contribution of this work regards how transferability is measured herein.
Intuitively, post-transfer top-1 accuracy provides the ground truth measure of transferability, as a measure of performance after transfer learning has occurred through head re-training or fine-tuning of the source model.
However, in the scenario where many source models are available for transfer to an alternate domain, evaluating the post-transfer top-1 accuracy for each source model may be too time consuming and computationally expensive.
Therefore, in addition to evaluating the transferability of pre-trained source models using post-transfer top-1 accuracy, we examine two existing transferability metrics not known to be used in the RF domain prior to this work: LEEP <cit.> and LogME <cit.>.
Transferability metrics, such as LEEP and LogME, provide a measure of how well a source model will transfer to a target dataset, and are evaluated without performing transfer learning, using only a single forward pass through the source model.
Though LEEP and LogME are designed to be modality independent, we confirm that they are suitable for use in RFML by showing that LEEP and LogME strongly correlate with post-transfer top-1 accuracy, as well as with each other.

Third, we present a method for using transferability metrics such as these to predict post-transfer accuracy, within a confidence interval, and without further training.
More specifically, given a labelled target raw IQ dataset and a selection of pre-trained source models, we show that transferability metrics such as LEEP and/or LogME can be used to provide a lower and upper bound on how well each source model will perform once transferred to the target dataset, without performing head re-training or fine-tuning.

In total, this work addresses a number of key research questions, including:
@ When and how is RF domain adaptation most successful? – Through exhaustive experimentation, this work provides generalized and practical guidelines for using TL for RF domain adaptation that are intuitive and consistent with the general theory of TL and domain adaptation.
@ Can transferability metrics, such as LEEP and LogME, be used to effectively select models for RF domain adaptation? – This work shows that both LEEP and LogME strongly correlate with post-transfer top-1 accuracy in the context of this AMC use-case, and that results are consistent with those shown in the original publications.
@ Can transferability metrics, such as LEEP and LogME, predict post-transfer accuracy? – This work presents such an approach, with no additional re-training requirements and confidence intervals provided.

This paper is organized as follows:
Section <ref> provides requisite background knowledge, and discusses related and prior works in TL for RFML, as well as transferability metrics and transfer accuracy prediction in other modalities such as CV and NLP.
In Section <ref>, each of the key methods and systems used and developed for this work are described in detail, including the simulation environment and dataset creation, the model architecture and training, and the transferability metrics.
Section <ref> presents experimental results and analysis, addressing the key research questions described above, and the proposed post-transfer accuracy prediction method.
Section <ref> highlights several directions for future work including extensions of this work performed herein using alternative transferability metrics and captured and/or augmented data, generalizations of this work to inductive TL settings, and developing more robust or RF-specific transferability metrics.
Finally, Section <ref> offers conclusions about the effectiveness of TL and existing transferability metrics for RFML and next steps for incorporating and extending TL techniques in RFML-based research.
A list of the acronyms used in this work is provided in the appendix for reference.

§ BACKGROUND & RELATED WORK

The recent RFML and TL taxonomies and surveys  <cit.> highlight the limited existing works that successfully use sequential TL techniques for both domain adaptation and inductive transfer including transferring pre-trained models across channel environments <cit.>, across wireless protocols <cit.>, and from synthetic data to real data <cit.>, as well as to add or remove output classes <cit.>, for tasks such as signal detection, AMC, and SEI.
However, outside of observing a lack of direct transfer <cit.>, little-to-no work has examined what characteristics within RF data facilitate or restrict transfer <cit.>.
Without such knowledge, TL algorithms for RFML are generally restricted to those borrowed from other modalities, such as CV and NLP.
While correlations can be drawn between the vision or language spaces and the RF space, these parallels do not always align, and therefore algorithms designed for CV and NLP may not always be appropriate for use in RFML.
In this work, post-transfer top-1 accuracy is paired with existing transferability metrics, LEEP and LogME, to identify how changes in the RF domain impact transferability, and how LEEP and LogME can be of value when performing RF domain adaptation in practice, namely for model selection and post-transfer accuracy prediction.

§.§ Transferability Metrics

As discussed previously, TL techniques use prior knowledge obtained from a source domain/task to improve performance on a similar target domain/task.
More specifically, TL techniques aim to further refine a pre-trained source model using a target dataset and specialized training techniques.
However, not all pre-trained source models will transfer well to a given target dataset.
Though it is generally understood that TL is successful when the source and target domains/tasks are “similar" <cit.>, this notion of source/target similarity is ill-defined.
The goal of a transferability metric is to quantify how well a given pre-trained source model will transfer to a target dataset.
While the area of transferability metrics is growing increasingly popular, to our knowledge, no prior works have examined these metrics in the context of RFML.
Transferability metrics developed and examined in the context of other modalities can broadly be categorized in one of two ways: those requiring partial re-training and those that do not.

Partial re-training methods such as Taskonomy <cit.> and Task2Vec <cit.> require some amount of training to occur, whether that be the initial stages of TL, full TL, or the training of an additional probe network, in order to quantify transferability.
Partial re-training methods are typically used to identify relationships between source and target tasks and are useful in meta-learning settings, but are not well suited to settings where time and/or computational resources are limited.
Though the computational complexity of partial re-training methods varies, it vastly exceeds the computational complexity of methods that do not require any additional training, such as those used in this work.

This work focuses on methods that do not require additional training, which typically use a single forward pass through a pre-trained model to ascertain transferability.
Methods such as these are often used to select a pre-trained model from a model library for transfer to a target dataset, a problem known as source model selection.
LEEP <cit.> and LogME <cit.> were chosen for this work having outperformed similar metrics such as NCE <cit.> and H-scores <cit.> in CV and NLP-based experiments, and for their modality agnostic design.
However, several new transferability metrics developed concurrently with this work also show promise including OTCE <cit.> and JC-NCE <cit.>, TransRate <cit.>, and GBC <cit.>, and may be examined as follow on work.

Related works examine source model ranking or selection procedures <cit.>, which either rank a set of models by transferability or select the model(s) most likely to provide successful transfer.
However, source model ranking or selection methods are less flexible than transferability metrics in online or active learning scenarios.
More specifically, source model ranking or selection methods are unable to identify how a new source model compares to the already ranked/selected source models without performing the ranking/selection procedure again.
Related works also include methods for selecting the best data to use for pre-training <cit.> or during the transfer phase <cit.>, and approaches to measuring domain, task, and/or dataset similarity <cit.>.

§.§ Predicting Transfer Accuracy

The problem of predicting transfer accuracy is still an open field.
To the best of our knowledge, no prior works have examined predicting transfer accuracy specifically for RFML, but approaches have been developed for other modalities.
Most similar to our work is the approach given in <cit.>, where the authors showed linear correlation between several domain similarity metrics and transfer accuracy, using statistical inference to derive performance predictions for NLP tools.
Similarly, work in <cit.> used domain similarity metrics to predict performance drops as a result in domain shift.
More recently, <cit.> proposed using a simple MLP to determine how well a source dataset will transfer to a target dataset, again in an NLP setting.

Signal types included in this work and generation parameters.

Name      1cc]@c@Parameter
BPSK     c]@l@Symbol Order {2}
RRC Pulse Shape
Excess Bandwidth {0.35, 0.5}
Symbol Overlap $\in$ [3, 5]

QPSK     c]@l@Symbol Order {4}
RRC Pulse Shape
Excess Bandwidth {0.35, 0.5}
Symbol Overlap $\in$ [3, 5]

PSK8      c]@l@Symbol Order {8}
RRC Pulse Shape
Excess Bandwidth {0.35, 0.5}
Symbol Overlap $\in$ [3, 5]

PSK16     c]@l@Symbol Order {16}
RRC Pulse Shape
Excess Bandwidth {0.35, 0.5}
Symbol Overlap $\in$ [3, 5]

OQPSK     c]@l@Symbol Order {4}
RRC Pulse Shape
Excess Bandwidth {0.35, 0.5}
Symbol Overlap $\in$ [3, 5]

QAM16     c]@l@Symbol Order {16}
RRC Pulse Shape
Excess Bandwidth {0.35, 0.5}
Symbol Overlap $\in$ [3, 5]

QAM32     c]@l@Symbol Order {32}
RRC Pulse Shape
Excess Bandwidth {0.35, 0.5}
Symbol Overlap $\in$ [3, 5]

QAM64     c]@l@Symbol Order {64}
RRC Pulse Shape
Excess Bandwidth {0.35, 0.5}
Symbol Overlap $\in$ [3, 5]

APSK16     c]@l@Symbol Order {16}
RRC Pulse Shape
Excess Bandwidth {0.35, 0.5}
Symbol Overlap $\in$ [3, 5]

Name     1cc]@c@Parameter
APSK32     c]@l@Symbol Order {32}
RRC Pulse Shape
Excess Bandwidth {0.35, 0.5}
Symbol Overlap $\in$ [3, 5]

FSK5k     c]@l@Carrier Spacing {5kHz}
Rect Phase Shape
Symbol Overlap {1}

FSK75k     c]@l@Carrier Spacing {75kHz}
Rect Phase Shape
Symbol Overlap {1}

GFSK5k     c]@l@Carrier Spacing {5kHz}
Gaussian Phase Shape
Symbol Overlap {2, 3, 4}
Beta $\in$ [0.3, 0.5]

GFSK75k     c]@l@Carrier Spacing {75kHz}
Gaussian Phase Shape
Symbol Overlap {2, 3, 4 }
Beta $\in$ [0.3, 0.5]

MSK     c]@l@Carrier Spacing {2.5kHz}
Rect Phase Shape
Symbol Overlap {1}

GMSK     c]@l@Carrier Spacing {2.5kHz}
Gaussian Phase Shape
Symbol Overlap {2, 3, 4}
Beta $\in$ [0.3, 0.5]

FM-NB     Modulation Index $\in$ [0.05, 0.4]

FM-WB     Modulation Index $\in$ [0.825, 1.88]

AM-DSB     Modulation Index $\in$ [0.5, 0.9]

AM-DSBSC     Modulation Index $\in$ [0.5, 0.9]

AM-LSB     Modulation Index $\in$ [0.5, 0.9]

AM-USB      Modulation Index $\in$ [0.5, 0.9]

§ METHODOLOGY

The training dataset generation process, repeated to create the validation and test datasets.

The process for model pre-training and TL.

The process for model evaluation and metric calculation.

A system overview of the (a) dataset creation, (b) model pre-training and TL, and (c) model evaluation and transferability metric calculation processes used in this work.

This section presents the experimental setup used in this work, shown in Fig. <ref>, which includes the data and dataset creation process, the model architecture and training, and the transferability metrics.
These three key components and processes are each described in detail in the following subsections.

§.§ Dataset Creation

§.§ Model Architecture and Training

§.§ Transferability Metrics

§ EXPERIMENTAL RESULTS & ANALYSIS

The product of the experiments performed herein is 82 data subsets, each with distinct RF domains, 82 source models trained from random initialization, and 4360 transfer learned models, half transferred using head re-training and the remaining half transferred using fine-tuning.
Associated with each of the 4360 transfer learned models is a top-1 accuracy value, a LEEP score, and a LogME score.
Given the careful curation of the signal parameters contained within each data subset, as well as the breadth of signal parameters observed, generalized conclusions can be drawn regarding TL performance as function changes in the propagation environment (SNR) and transmitter/receiver hardware (FO).
However, it should be noted that further experiments using captured data are required in order to draw more concrete guidelines for using RF TL in the field <cit.>, and is left for future work.
The following subsections present the results obtained from the experiments performed, and discuss how well LEEP and LogME perform in the RF modality, how to use transferability metrics to predict post-transfer performance, as well as some insights and practical takeaways that can be gleaned from the results given including a preliminary understanding of when and how to use TL for RF domain adaptation.

§.§ When and how is RF domain adaptation most successful?

The post-transfer top-1 accuracy for each source/target dataset pair constructed for the sweep over SNR using head re-training to perform domain adaptation. When fine-tuning is used to perform domain adaptation, the same trends are apparent.

The post-transfer top-1 accuracy for each source/target dataset pair constructed for the sweep over FO using head re-training to perform domain adaptation. When fine-tuning is used to perform domain adaptation, the same trends are apparent. Note the scale differs from Figs. <ref> and <ref>.

The post-transfer top-1 accuracy for each source/target dataset pair constructed for the sweep over both SNR and FO using head re-training to perform domain adaptation. When fine-tuning is used to perform domain adaptation, the same trends are apparent.

§.§.§ Impact of Source/Target Domain Similarity and “Difficulty" on Transfer Performance

The heatmaps in Figs. <ref>-<ref> show the post-transfer top-1 accuracy achieved with each of the source/target pairs.
Note that the post-transfer top-1 accuracy results shown in Figs. <ref>-<ref> are from the models that used head re-training to transfer from the source to target domains/datasets.
However, the accuracy results from the models that used fine-tuning for transfer show the same trends.

Figs. <ref>-<ref> show that highest post-transfer performance is achieved along the diagonal of the heatmap, where the source and target domains are most similar.
This behavior is expected, as models trained on similar domains likely learn similar features, and is consistent with the general theory of TL <cit.>, as well as existing works in modalities outside of RF <cit.>.
While the notion of domain similarity is ill-defined in general, for the purposes of this work, we are able to say that domains are more similar when the difference between the source and target SNR and/or FO ranges is smaller, as all other data generation parameters are held constant.

Additionally, Figs. <ref>-<ref> show that transfer across changes in FO is approximately symmetric, while transfer across changes in SNR are not.
This behavior is also expected, and can be attributed to changes in the relative “difficulty" between the source and target domains.
More specifically, changing the source/target SNR inherently changes the difficulty of the problem, as performing AMC in lower SNR channel environments is more challenging than performing AMC in high SNR channel environments.
Therefore, the source models trained on the lower SNR ranges will transfer to the higher SNR ranges, though may not perform optimally, while the source models trained on the higher SNR ranges will fail to transfer to the lower SNR ranges, as shown in Figs. <ref> and <ref>.
In contrast, changing the source/target FO does not make performing AMC any more or less difficult, but may require modifications to the learned features to accommodate this change.
Moreover, small changes in FO, $\omega_\Delta[t]$, in either the positive and negative direction, are expected to perform similarly.
As a result, transfer occurs in either direction off of the diagonal, with best performance closest to the diagonal, as discussed previously.

Practically, these trends indicate that the effectiveness of RF domain adaptation increases as the source and target domains become more similar, and, when applicable, RF domain adaptation is more often successful when transferring from harder to easier domains.
For example, transferring from [-5dB, 0dB] to [0dB, 5dB] SNR is likely more effective than transferring from [5dB, 10dB] to [0dB, 5dB] SNR, and transferring from a FO range of [-9%, -4%] of sample rate to [-8%, -3%] of sample rate is likely more effective than transferring from a FO range of [-10%, -5%] of sample rate to [-8%, -3%] of sample rate.

§.§.§ Environment Adaptation vs. Platform Adaptation

The LEEP versus LogME scores for the sweep over SNR, FO, and both SNR and FO. The dashed lines present the linear fit.

Recalling that the sweep over SNR can be regarded as an environment adaptation experiment and the sweep over FO can be regarded as a platform adaptation experiment, more general conclusions can be drawn regarding the challenges that environment and platform adaptation present.

From the discussion in the previous subsection, it follows that changes in FO should be easier to overcome than changes in SNR.
That is environment adaptation is more difficult to achieve than platform adaptation, and changes in transmitter/receiver hardware are likely easier to overcome using TL techniques than changes in the channel environment.
This trend is indirectly shown through the range of accuracies achieved in Figs. <ref>-<ref>, which is smaller for the FO sweep than the SNR sweep and SNR $+$ FO sweep, and is more directly shown in Fig. <ref>.
(It should be noted that the scale in Fig. <ref> differs from Figs. <ref> and <ref>.)
Fig. <ref> presents the LogME scores as a function of the LEEP scores for each of the parameter sweeps performed, showing both the LEEP and LogME scores are significantly higher for the FO sweep than the SNR sweep or SNR and FO sweep indicating better transferability.
(Of course, this conclusion is dependent upon the results presented in Section <ref> which show that LEEP and LogME correlate with post-transfer accuracy.)
Therefore, in practice, one should consider the similarity of the source/target channel environment before the similarity of the source/target platform, as changes in transmitter/receiver pair are more easily overcome during TL.

§.§.§ Head Re-Training vs. Fine-Tuning

The LEEP (a) and LogME (b) scores versus post-transfer top-1 accuracy for the sweep over SNR. The dashed lines present the linear fits for all target domains.

The LEEP (a) and LogME (b) scores versus post-transfer top-1 accuracy for the sweep over FO. The dashed lines present the linear fits for all target domains.

The LEEP (a) and LogME (b) scores versus post-transfer top-1 accuracy for the sweep over both SNR and FO. The dashed lines present the linear fits for all target domains.

In Figs. <ref>-<ref>, post-transfer top-1 accuracy is shown as a function of either LEEP or LogME.
These figures indicate that when considering post-transfer top-1 accuracy as the sole performance metric, head re-training is as effective as fine-tuning in most settings.
The only setting observed herein in which the fine-tuned models markedly outperformed head re-trained models is in the sweep over FO, especially when the LEEP and LogME scores were low.
A low LEEP/LogME score indicates a significant change between the source and target domains, in this case a large change in FO.
As a result, new features are needed to discern between modulation types, and modifications to the earlier layers of the pre-trained source model, where feature learning occurs, are needed in order to best adapt to the new target domain.

As previously discussed, head re-training is more time efficient and less computationally expensive than fine-tuning, making is a strong case for using head re-training over fine-tuning for RF domain adaptation.
The computational complexity of using head re-training versus fine-tuning is architecture and training algorithm dependent, but as an example, for the CNN architecture used in this work and shown in Tab. <ref>, the number of trainable parameters for head re-training and fine-tuning is 1,518 and 7,434,243  respectively.
Finally, the correlation coefficients in Figs. <ref>-<ref> show that head re-training is more consistent with LEEP and LogME scores than fine-tuning, leading to the next discussion of whether LEEP and LogME scores can be used to select models for RF domain adaptation.

§.§ Can transferability metrics, such as LEEP and LogME, be used to select models for RF domain adaptation?

When evaluating whether a transferability metric is accurate, the primary consideration is how well the metric reflects or correlates with the performance metric(s) used.
Therefore, to identify whether LEEP and/or LogME can be used to select models for RF domain adaptation is to identify how well LEEP and LogME correlate with post-transfer top-1 accuracy.
To this end, Figs. <ref>-<ref> show LEEP and LogME versus the achieved transfer accuracy for each of the parameter sweeps described in Section <ref>.
These figures qualitatively show that both LEEP and LogME correlate well with top-1 accuracy after transfer learning, whether through head re-training or fine-tuning for all domain adaptation settings studied.

To quantify whether or not the metrics are useful, two correlation measures are also examined – the Pearson correlation coefficient <cit.> and the weighted $\tau$ <cit.> – specified in the shaded boxes of Figs. <ref>-<ref>.
The Pearson correlation coefficient, or Pearson's $r$, is a measure of linear correlation between two variables used in a wide variety of works, including the original LEEP paper.
However, Pearson's $r$ makes a number of assumptions about the data, some of which may not be met by this data.
Most notably, Pearson's r assumes that both variables (LEEP/LogME and post-transfer top-1 accuracy, herein) are normally distributed and have a linear relationship.
Alternatively, weighted $\tau$, a weighted version of the Kendall rank correlation coefficient (Kendall $\tau$), is used in the original LogME work.
Weighted $\tau$ is a measure of correspondence between pairwise rankings, where higher performing/scoring models receive higher weight, and only assumes the variables (LEEP/LogME and post-transfer top-1 accuracy, herein) are continuous.
Both Pearson's $r$ and weighted $\tau$ have a range of $[-1,1]$.
These correlation coefficients confirm the results discussed above.

From these figures and metrics it can be concluded that both LEEP and LogME are strong measures for selecting models for RF domain adaptation.
However, as alluded to in the previous subsection, head re-training is more consistent with LEEP and LogME scores than fine-tuning, as evidenced by higher correlation coefficients.
Therefore, when using LEEP or LogME for model selection, using head re-training as a TL method would be more reliable than using fine-tuning.
In contrast, fine-tuning, while less reliable than head re-training when used in conjunction with LEEP or LogME for model selection, offers potential for small performance gains over head re-training.
In practice, this indicates that unless top performance is of more value than reliability, head re-training should be used for TL when using LEEP or LogME for model selection.
In the setting where model accuracy is of the utmost importance, it may be advantageous to try both head re-training and fine-tuning.

It should also be noted that the results shown in Figs. <ref>-<ref> are consistent with the results presented in the original LEEP and LogME publications where the metrics were tested in CV and NLP settings, supporting the claim that these metrics are truly modality agnostic.
Therefore, other modality agnostic metrics seem likely to perform well in RFML settings as well, and may be examined as follow on work.

§.§ How can  LEEP/LogME be used to predict post-transfer accuracy?

Having confirmed that LEEP and LogME can be used to select models for RF domain adaptation, what follows is an approach to not only select models for RF domain adaptation, but also to predict the post-transfer top-1 accuracy without any further training.
The approach is time and resource intensive to initialize, but once initialized, is fast and relatively inexpensive to compute and shows the predictive capabilities of these metrics.

Given $n$ known domains and assuming a single model architecture, to initialize the approach:
@ Run baseline simulations for all $n$ known domains including pre-training source models on all domains, and using head re-training and/or fine-tuning to transfer each source model to the remaining known domains
@ Compute LEEP/LogME scores using all pre-trained source models and the remaining known domains.
@ Compute post-transfer top-1 accuracy for all transfer-learned models, constructing datapoints like those displayed in Figs. <ref>-<ref>.
@ Fit a function of the desired form (i.e. linear, logarithmic, etc.) to the LEEP/LogME scores and post-transfer top-1 accuracies. For example, a linear fit of the form $y = \beta_0 x + \beta_1$ is shown in Figs. <ref>-<ref> such that $x$ is the transferability score and $y$ is the post-transfer top-1 accuracy.
@ Compute the margin of error by first calculating the mean difference between the true post-transfer top-1 accuracy and the predicted post-transfer top-1 accuracy (using the linear fit), and then multiplying this mean by the appropriate z-score(s) for the desired confidence interval(s) <cit.>.

Then, during deployment, given a new labelled target dataset:
@ Compute LEEP/LogME scores for all pre-trained source models and new target dataset.
@ Select the pre-trained source model yielding the highest LEEP/LogME score for TL.
@ Use the fitted linear function to estimate post-transfer accuracy, given the highest LEEP/LogME score, and add/subtract the margin of error to construct the confidence interval.

Optionally, after transferring to the new labelled target dataset, add this dataset to the list of known domains, and update the linear fit and margin of error, as needed.

The error in the predicted post-transfer accuracy using a linear fit to the LEEP scores (x-axis) and LogME scores (y-axis) for the sweep over SNR.

The error in the predicted post-transfer accuracy using a linear fit to the LEEP scores (x-axis) and LogME scores (y-axis) for the sweep over FO. Note the change in scale compared to Figs. <ref> and <ref>.

The error in the predicted post-transfer accuracy using a linear fit to the LEEP scores (x-axis) and LogME scores (y-axis) for the sweep over both SNR and FO.

The error in the predicted post-transfer accuracy using the proposed method is shown in Figs. <ref>-<ref>.
These plots show that not only are LEEP/LogME highly correlated with post-transfer top-1 accuracy (as shown in Figs. <ref>-<ref>), but the error in the predicted post-transfer top-1 accuracy using a linear fit to the LEEP and LogME scores respectively is also highly correlated.
More specifically, when the proposed method constructed using LEEP predicts a lower/higher post-transfer accuracy than ground truth, the proposed method constructed using LogME will do the same with the frequencies shown in Tab. <ref>.
This indicates that these scores could be combined to create a more robust transferability metric and more robust post-transfer accuracy prediction with relative ease, which is left for future work.

The frequency with which the proposed method constructed using LEEP and LogME agree in over/under predicting post-transfer accuracy.

SNR Sweep     FO Sweep     SNR + FO Sweep
Head Re-Training     0.6175     0.7856     0.7258

Fine-Tuning     0.7496     0.8803     0.7468

§ FUTURE WORK

As previously mentioned, several new transferability metrics were developed concurrently with this work, and are suitable as replacements for LEEP and LogME in any of the above experiments.
Therefore, the first direction for future work is replicating this work using alternative metrics such as OTCE <cit.>, JC-NCE <cit.>, TransRate <cit.>, and GBC <cit.>, to identify if these metrics are also suitable for use in the context of RFML and if these metrics might outperform those used herein.
Given that this work supports the claim that LEEP and LogME are modality agnostic, it seems likely that additional transferability metrics that are also modality agnostic by design will also follow this trend.
Additionally, the concept of transferability metrics and the experiments performed herein should be extended to inductive TL settings including multi-task learning and sequential learning settings in which the source and target tasks differ (i.e. adding/removing output classes), as this work only considered RF domain adaptation.

Another direction for future work is the development of new transferability metrics that are more robust than LEEP or LogME alone or are RFML-specific.
Most apparently, results discussed previously in Section <ref> indicate that LEEP and LogME could be combined to create a more robust transferability metric and more robust post-transfer accuracy prediction with relative ease.
However, while modality agnostic metrics such as LEEP and LogME are shown herein to be suitable for use in RFML, a transferability metric purpose-built for the RF space would likely be more widely accepted amongst traditional RF engineers <cit.>.

With or without the use of transferability metrics, this work provides generalized conclusions about how best to use TL in the context of RFML.
Provided future verification and refinement of these results and guidelines using captured and augmented data <cit.>, this work can be used in future RFML systems to construct the highest performing models for a given target domain when data is limited.
More specifically, these guidelines begin a discussion regarding how best to continually update RFML models once deployed, in an online or incremental fashion, to overcome the highly fluid nature of modern communication systems <cit.>.

§ CONCLUSION

TL has yielded tremendous performance benefits in CV and NLP, and as a result, TL is all but commonplace in these fields.
However, the benefits of TL have yet to be fully demonstrated and integrated in RFML.
To begin to address this deficit, this work systematically evaluated RF domain adaptation performance as a function of several parameters-of-interest, including SNR and FO, using post-transfer top-1 accuracy and existing transferability metrics, LEEP and LogME.
This work demonstrated that LEEP and LogME correlate well with post-transfer accuracy, and can therefore be used for model selection successfully in the context of RF domain adaptation.
Further, an approach was presented for predicting post-transfer accuracy using these metrics, within a confidence interval, and without further training.

Through this exhaustive study, a number of guidelines have been identified for when and how to use TL for RF domain adaptation successfully.
More specifically, results indicate:
@ Using source models trained on the most similar domain to the target yields highest performance
@ Transferring from a more challenging domain than the target, is preferred to transferring from an easier domain
@ Selecting source models based on this similarity of the source/target channel environment is more important than the similarity of the source/target platform(s)
@ Head re-training is more reliable, faster, and less computationally expensive than fine-tuning for RF domain adaptation
@ TL via both head re-training and fine-tuning should be attempted, when top performance is of greater value than time and/or computational efficiency

These takeaways can be used in future RFML systems to construct higher performing models when limited data is available, and can be used to develop to online and active RFML approaches, a critical need for practical and deployable RFML.

#1     #1#1#1     #1     #1     #1     #1#1          #1#1

titleCross Entropy Loss.

[Achille et al(2019)]
authorpersonAlessandro Achille,
personMichael Lam, personRahul Tewari,
personAvinash Ravichandran, personSubhransu Maji,
personCharless C Fowlkes, personStefano Soatto, and
personPietro Perona. year2019.
Task2Vec: Task embedding for meta-learning. In
booktitleProceedings of the IEEE/CVF International
Conference on Computer Vision. pages6430–6439.

[Bao et al(2019)]
authorpersonYajie Bao, personYang Li,
personShao-Lun Huang, personLin Zhang,
personLizhong Zheng, personAmir Zamir, and
personLeonidas Guibas. year2019.
An Information-Theoretic Approach to
Transferability in Task Transfer Learning. In booktitle2019
IEEE International Conference on Image Processing (ICIP).

[Bhattacharjee et al(2020)]
authorpersonBishwaranjan Bhattacharjee,
personJohn R Kender, personMatthew Hill,
personParijat Dube, personSiyu Huo,
personMichael R Glass, personBrian Belgodere,
personSharath Pankanti, personNoel Codella, and
personPatrick Watson. year2020.
P2L: Predicting transfer learning for images and
semantic relations. In booktitleProceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition Workshops.

[Chen et al(2019)]
authorpersonS. Chen, personS. Zheng,
personL. Yang, and personX. Yang.
Deep Learning for Large-Scale Real-World ACARS
and ADS-B Radio Signal Classification.
journalIEEE Access  volume7
(year2019), pages89256–89264.

[Clark et al(2019)]
authorpersonBill Clark, personZach Leffke,
personChris Headley, and personAlan Michaels.
booktitleCyborg Phase II Final Report.
typeTechnical Report. institutionTed and
Karyn Hume Center for National Security and Technology.

[Clark et al(2019)]
authorpersonW. H. Clark, personV.
Arndorfer, personB. Tamir, personD. Kim,
personC. Vives, personH. Morris,
personL. Wong, and personW. C. Headley.
Developing RFML Intuition: An Automatic
Modulation Classification Architecture Case Study. In
booktitle2019 IEEE Military Comm. Conference (MILCOM).

[Clark IV et al(2020)]
authorpersonWilliam H Clark IV, personSteven
Hauser, personWilliam C Headley, and personAlan J
Michaels. year2020.
Training data augmentation for deep learning radio
frequency systems.
journalThe Journal of Defense Modeling and
Simulation (year2020).

[Conference, IEEE Communications Society(2021)]
authorpersonConference, IEEE Communications Society.
titleDySPAN 2021: 2020 IEEE International Symposium on
Dynamic Spectrum Access Networks, 13–15 December 2021.

[Dörner et al(2018)]
authorpersonS. Dörner, personS.
Cammerer, personJ. Hoydis, and personS. t.
Brink. year2018.
Deep Learning Based Communication Over the Air.
journalIEEE Journal of Selected Topics in Signal
Processing volume12, number1
(year2018), pages132–143.

[Elsahar and Gallé(2019)]
authorpersonHady Elsahar and
personMatthias Gallé. year2019.
To annotate or not? Predicting performance drop
under domain shift. In booktitleProceedings of the 2019
Conference on Empirical Methods in Natural Language Processing and the 9th
International Joint Conference on Natural Language Processing
(EMNLP-IJCNLP). pages2163–2173.

authorpersonJoseph Gaeddert.

authorpersonSteven Charles Hauser.
titleReal-World Considerations for Deep Learning in
Spectrum Sensing.
thesistypeMaster's thesis. schoolVirginia

authorpersonAvijit Hazra.
Using the confidence interval confidently.
journalJournal of thoracic disease
volume9, number10 (year2017),

[Hilburn et al(2018)]
authorpersonBen Hilburn, personNathan West,
personTim O'Shea, and personTamoghna Roy.
SigMF: the signal metadata format. In
booktitleProceedings of the GNU Radio Conference,
Vol. volume3.

[Huang et al(2021)]
authorpersonLong-Kai Huang, personYing Wei,
personYu Rong, personQiang Yang, and
personJunzhou Huang. year2021.
Frustratingly Easy Transferability Estimation.
journalarXiv preprint arXiv:2106.09362

[Kashyap et al(2020)]
authorpersonAbhinav Ramesh Kashyap,
personDevamanyu Hazarika, personMin-Yen Kan, and
personRoger Zimmermann. year2020.
Domain divergences: a survey and empirical
journalarXiv preprint arXiv:2010.12198

authorpersonMaurice G Kendall.
A new measure of rank correlation.
journalBiometrika volume30,
number1/2 (year1938), pages81–93.

[Kingma and Ba(2014)]
authorpersonDiederik P Kingma and
personJimmy Ba. year2014.
Adam: A method for stochastic optimization.
journalarXiv preprint arXiv:1412.6980

authorpersonPaul Kolb.
titleSecuring Compartmented Information with Smart Radio
Systems (SCISRS).

[Kuzdeba et al(2021)]
authorpersonScott Kuzdeba, personJosh
Robinson, and personJoseph Carmack.
Transfer Learning with Radio Frequency Signals. In
booktitle2021 IEEE 18th Annual Consumer Communications
Networking Conference (CCNC). pages1–9.

[Li et al(2021)]
authorpersonYandong Li, personXuhui Jia,
personRuoxin Sang, personYukun Zhu,
personBradley Green, personLiqiang Wang, and
personBoqing Gong. year2021.
Ranking neural checkpoints. In
booktitleProceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition. pages2663–2673.

authorpersonKevin Merchant.
titleDeep Neural Networks for Radio Frequency
thesistypePh. D. Dissertation.

[Nguyen et al(2020)]
authorpersonCuong Nguyen, personTal
Hassner, personMatthias Seeger, and personCedric
Archambeau. year2020.
LEEP: A new measure to evaluate transferability
of learned representations. In booktitleInternational
Conference on Machine Learning. PMLR, pages7294–7305.

[Olivas et al(2009)]
authorpersonEmilio Soria Olivas, personJos
David Mart Guerrero, personMarcelino Martinez-Sober,
personJose Rafael Magdalena-Benedito, personL
Serrano, et al year2009.
booktitleHandbook of research on machine learning
applications and trends: Algorithms, methods, and techniques.
publisherIGI Global.

[O’Shea et al(2018)]
authorpersonT. J. O’Shea, personT.
Roy, and personT. C. Clancy.
Over-the-Air Deep Learning Based Radio Signal
journalIEEE Journal of Selected Topics in Signal
Processing volume12, number1
(year2018), pages168–179.

[Pan and Yang(2010)]
authorpersonS. J. Pan and personQ.
Yang. year2010.
A Survey on Transfer Learning.
journalIEEE Trans. on Knowledge and Data Eng.
volume22, number10 (year2010),

[Pándy et al(2021)]
authorpersonMichal Pándy, personAndrea
Agostinelli, personJasper Uijlings, personVittorio
Ferrari, and personThomas Mensink.
Transferability Estimation using Bhattacharyya
Class Separability.
journalarXiv preprint arXiv:2111.12780

[Paszke et al(2019)]
authorpersonAdam Paszke, personSam Gross,
personFrancisco Massa, personAdam Lerer,
personJames Bradbury, personGregory Chanan,
personTrevor Killeen, personZeming Lin,
personNatalia Gimelshein, personLuca Antiga,
et al year2019.
PyTorch: An imperative style, high-performance
deep learning library.
journalAdvances in neural information processing
systems  volume32 (year2019),

[Pati et al(2020)]
authorpersonB. M. Pati, personM.
Kaneko, and personA. Taparugssanagorn.
A Deep Convolutional Neural Network Based Transfer
Learning Method for Non-Cooperative Spectrum Sensing.
journalIEEE Access  volume8
(year2020), pages164529–164545.

[Peng et al(2020)]
authorpersonQ. Peng, personA. Gilman,
personN. Vasconcelos, personP. C. Cosman, and
personL. B. Milstein. year2020.
Robust Deep Sensing Through Transfer Learning in
Cognitive Radio.
journalIEEE Wireless Comm. Letters
volume9, number1 (year2020),

[Pogrebnyakov and Shaghaghian(2021)]
authorpersonNicolai Pogrebnyakov and
personShohreh Shaghaghian. year2021.
Predicting the Success of Domain Adaptation in Text
journalarXiv preprint arXiv:2106.04641

[Renggli et al(2020)]
authorpersonCedric Renggli,
personAndré Susano Pinto, personLuka Rimanic,
personJoan Puigcerver, personCarlos Riquelme,
personCe Zhang, and personMario Lucic.
Which model to transfer? Finding the needle in
the growing haystack.
journalarXiv preprint arXiv:2010.06402

[Robinson and Kuzdeba(2021)]
authorpersonJosh Robinson and personScott
Kuzdeba. year2021.
RiftNet: Radio Frequency Classification for Large
Populations. In booktitle2021 IEEE 18th Annual Consumer
Communications Networking Conference (CCNC). pages1–6.

authorpersonTom Rondeau.
titleRadio Frequency Machine Learning Systems (RFMLS).

[Rosenstein et al(2005)]
authorpersonMichael T Rosenstein,
personZvika Marx, personLeslie Pack Kaelbling, and
personThomas G Dietterich. year2005.
To transfer or not to transfer. In
booktitleNIPS 2005 workshop on transfer learning,
Vol. volume898. pages1–4.

authorpersonSebastian Ruder.
titleNeural transfer learning for natural language
thesistypePh. D. Dissertation. schoolNUI

[Ruder and Plank(2017)]
authorpersonSebastian Ruder and
personBarbara Plank. year2017.
Learning to select data for transfer learning with
bayesian optimization.
journalarXiv preprint arXiv:1707.05246

[Sankhe et al(2019)]
authorpersonKunal Sankhe, personMauro
Belgiovine, personFan Zhou, personShamnaz Riyaz,
personStratis Ioannidis, and personKaushik
Chowdhury. year2019.
ORACLE: Optimized Radio Classification through
Convolutional Neural Networks. In booktitleIEEE INFOCOM
2019-IEEE Conference on Computer Communications. IEEE,

[Schober et al(2018)]
authorpersonPatrick Schober, personChrista
Boer, and personLothar A Schwarte.
Correlation coefficients: appropriate use and
journalAnesthesia & Analgesia
volume126, number5 (year2018),

[Tan et al(2021a)]
authorpersonYang Tan, personYang Li, and
personShao-Lun Huang. year2021a.
OTCE: A Transferability Metric for Cross-Domain
Cross-Task Representations. In booktitleProceedings of the
IEEE/CVF Conference on Computer Vision and Pattern Recognition.

[Tan et al(2021b)]
authorpersonYang Tan, personYang Li, and
personShao-Lun Huang. year2021b.
Practical Transferability Estimation for Image
Classification Tasks.
journalarXiv preprint arXiv:2106.10479

[Tran et al(2019)]
authorpersonAnh T Tran, personCuong V
Nguyen, and personTal Hassner.
Transferability and hardness of supervised
classification tasks. In booktitleProceedings of the
IEEE/CVF International Conference on Computer Vision.

[Van Asch and Daelemans(2010)]
authorpersonVincent Van Asch and
personWalter Daelemans. year2010.
Using domain similarity for performance
estimation. In booktitleProceedings of the 2010 Workshop on
Domain Adaptation for Natural Language Processing. pages31–36.

[West and O'Shea(2017)]
authorpersonN. E. West and personT.
O'Shea. year2017.
Deep architectures for modulation recognition. In
booktitle2017 IEEE Int. Symp. on Dynamic Spectrum Access
Networks (DySPAN). pages1–6.

[Wong et al(2022)]
authorpersonLauren Wong, personSean
McPherson, and personAlan Michaels.
titleTransfer Learning for RF Domain Adaptation –
Synthetic Dataset.

[Wong et al(2021)]
authorpersonLauren J. Wong, personWilliam H.
Clark, personBryse Flowers, personR. Michael
Buehrer, personWilliam C. Headley, and
personAlan J. Michaels. year2021.
An RFML Ecosystem: Considerations for the
Application of Deep Learning to Spectrum Situational Awareness.
journalIEEE Open Journal of the Communications
Society  volume2 (year2021),

[Wong and McPherson(2021)]
authorpersonLauren J. Wong and personSean
McPherson. year2021.
Explainable Neural Network-based Modulation
Classification via Concept Bottleneck Models. In
booktitle2021 IEEE 11th Annual Computing and Communication
Workshop and Conference (CCWC). pages0191–0196.

[Wong and Michaels(2022)]
authorpersonLauren J. Wong and
personAlan J. Michaels. year2022.
Transfer Learning for Radio Frequency Machine
Learning: A Taxonomy and Survey.
journalSensors volume22,
number4 (year2022).

[You et al(2021)]
authorpersonKaichao You, personYong Liu,
personMingsheng Long, and personJianmin Wang.
LogME: Practical Assessment of Pre-trained Models
for Transfer Learning.
journalarXiv preprint arXiv:2102.11005

[Zamir et al(2018)]
authorpersonAmir R Zamir, personAlexander
Sax, personWilliam Shen, personLeonidas J Guibas,
personJitendra Malik, and personSilvio Savarese.
Taskonomy: Disentangling task transfer learning.
In booktitleProceedings of the IEEE conference on computer
vision and pattern recognition. pages3712–3722.

[Zheng et al(2020)]
authorpersonS. Zheng, personS. Chen,
personP. Qi, personH. Zhou, and
personX. Yang. year2020.
Spectrum sensing based on deep learning
classification for cognitive radios.
journalChina Comm. volume17,
number2 (year2020), pages138–148.