# Automatic Mass Detection in Breast Using Deep Convolutional Neural Network
and SVM Classifier

Md. Kamrul Hasan  Erasmus Joint Master in Medical Imaging & Applications
(MAIA), University of Girona, Girona, Spain  md-
<EMAIL_ADDRESS>Tajwar Abrar Aleef  Erasmus Joint Master in
Medical Imaging & Applications (MAIA), University of Girona, Girona, Spain

###### Abstract

Mammography is the most widely used gold standard for screening breast cancer,
where, mass detection is considered as the prominent step. Detecting mass in
the breast is however an arduous problem as they usually have large variations
between them in terms of shape, size, boundary, and texture. In this
literature, the process of mass detection is automated with the use of
transfer learning techniques of Deep Convolutional Neural Networks (DCNN).
Pre-trained VGG19 network is used to extract features which are then followed
by bagged decision tree for features selection and then a Support Vector
Machine (SVM) classifier is trained and used for classifying between the mass
and non-mass. Area Under ROC Curve (AUC) is chosen as the performance metric,
which is then maximized during classifier selection and hyper-parameter
tuning. The robustness of the two selected type of classifiers, C-SVM and
$\upsilon$-SVM, are investigated with extensive experiments before selecting
the best performing classifier. All experiments in this paper were conducted
using the INbreast dataset. The best AUC obtained from the experimental
results is 0.994 +/- 0.003 i.e. [0.991, 0.997]. Our results conclude that by
using pre-trained VGG19 network, high-level distinctive features can be
extracted from Mammograms which when used with the proposed SVM classifier is
able to robustly distinguish between the mass and non-mass present in breast.

###### keywords:

Deep Convolutional Neural Networks (DCNN), SVM, Breast Cancer, Mass
Classification

## Introduction

Cancer is the foremost worldwide public health problem and it is considered to
be the second leading cause of death with an estimated 9.6 million deaths in
2018 [1]. Approximately, 70 % of cancer-related death occurs in low and
middle-income countries. Cancer is a generic term for a large group of
diseases that can affect any organ and can be defined as rapid growth of
abnormal cells [2]. Breast cancer contributes to the second most cause of
death arising due to cancer for women [3]. A tumor in the breast can be
defined as an uncontrolled growth of cells which can generally be of two types
e.g. non-cancerous or ‘benign’, and cancerous or ‘malignant’. The term “breast
cancer” refers to the presence of malignant tumor in the breast as shown in
Fig. 1.

Figure 1: Presence of malignant tumor in the breast [4].

There are several imaging techniques available that help clinicians to analyze
and pinpoint suspicious regions. Modalities such as X-ray (Mammography,
Digital breast tomosynthesis, Xeromammography, Galactography), MRI, CT, PET,
Ultrasound, and Scintimammography are some of the non-invasive techniques used
to detect mass in the breast. Among all these methods, mammography is more
commonly used and it is considered as the gold standard for detecting early
stage breast cancer before the lesions become clinically palpable [5]. In
mammography, X-rays are used to produce images of the breast that provide
information about breast morphology, normal anatomy, and gross pathology.
Mammography is used primarily to detect and diagnose breast cancer and to
evaluate palpable and non-palpable breast lesions. After mammographic imaging,
one of the foremost challenges for a radiologist or a CAD system is to
accurately distinguish between the mass and non-mass regions of the breast
[6].

There are plenty of methods used in CAD systems in order to make this
differentiation. In ref[7], K-means clustering, Template-matching, Simpson’s
Diversity Index, and finally SVM are used which resulted in an overall
accuracy of the pipeline at 83.94 %. In ref[8], for each mass, eight shape
parameters and ten enhancement texture features were calculated and then an
Artificial Neural Network (ANN) was used to build the diagnostic model. The
average AUC reported by the system was 0.76. In ref[9], tissue identification
results were obtained by multivariate statistical analysis of mass
spectrometer data with a sensitivity of 90.9% and specificity of 98.8%. In
ref[10], transfer learning using AlexNet with 1656 mammography images
($454\times 454$) was implemented where the system had an accuracy of 85.35 %.
Ref[11],[12],[13],[14] initially detected mass candidates or regions of
interest (ROI) followed by feature vector extraction based on special
knowledge and then classification was done using the feature vectors. In
ref[15], authors used SVMs with ConvNets to detect mass on mammograms, where
the reported accuracy was 98.44%, which is superior to the baseline (ConvNets)
by 8 %. In ref[16], authors used DCNN along with SVM and were able to achieve
an AUC of 0.94. In ref[17], authors proposed a computational methodology,
where, pre-processing was done initially to improve the quality of mammogram
images as well as to remove regions outside the breast and hence reducing
noise and highlighting internal structures of the breast. Next, cellular
neural networks were used to segment the regions and to extract shape
descriptors (eccentricity, circularity, density, circular disproportion, and
circular density), followed by a SVM classifier. They reported a sensitivity
of 80% and AUC of 0.87. In ref[18], pre-trained ResNet-50 architecture and
Class Activation Map (CAM) technique were employed in breast cancer
classification and localization resulting in an AUC of 0.96.

It is challenging to have a reliable comparison between the published methods
as they are not using the same dataset. However, a general trend observed in
the literature includes less focus on the computational complexity, robustness
and also reporting low accuracy on unseen test set. The novelty of this
research lies in the: pipeline that uses transfer learning for feature
extraction, optimization process of hyper-parameters that gives the best AUC
using SVM (C-SVM or $\upsilon$-SVM) classifiers, and its ability to run
seamlessly on CPUs. The optimal penalization parameters C and $\upsilon$ for
C-SVM and $\upsilon$-SVM were found using extensive grid search methods that
maximize AUC. The proposed method in this paper uses a pre-trained VGG19 model
to extract features from patches ($454\times 454$) of mammogram. The features
are selected using bagged decision tree to avoid redundant features in order
to reduce over-fitting and the curse of dimensionality. Selected features are
then given as an input to the optimized SVM classifier (C-SVM and
$\upsilon$-SVM) for classification. Robustness of the classifier has been
validated using 5-fold cross-validation.

The remaining sections of this paper are organized as follows: section II is
dedicated to discussing data set. Section III discusses the overall pipeline
followed in this research. Section IV is about obtained results and
discussions using the proposed pipeline and finally, Section V concludes and
summarizes the literature.

## Mammography Database

INbreast [19] database was used in this literature which was acquired at a
Breast Centre located in a University Hospital (Hospital de São João, Breast
Centre, Porto, Portugal) and was made public for research use under the
permission of both the Hospital’s Ethics Committee and the National Committee
of Data Protection. INbreast mammographic images were collected using a
MammoNovation Siemens FFDM with a solid-state detector of amorphous selenium.
The dataset contains image dimensions of $3328\times 4084$ pixels or
$2560\times 3328$ pixels with a pixel size of 70 microns along with a 14-bit
contrast resolution. There are total of 410 images and 115 cases of which 90
of them have mediolateral oblique (MLO) view and the others have craniocaudal
(CC) view. Example of the two main views, MLO (visible pectoral muscle) and
CC, are shown in Fig. 2.

(a) MLO view

(b) CC view

Figure 2: Two major views of Mammograms available in INbreast database.

## Methods

### Pipeline

The overall pipeline used for this research is shown in a block diagram as
shown in Fig. 3. The patch extraction step is followed by feature extraction
using VGG19 network. And then, feature selection step is followed by the SVM
classifier which finally classifies between the mass and non-mass tissues.
Each block of the proposed pipeline is elaborately described below step by
step.

https://www.overleaf.com/project/5c7440defb0def2fb94c57a2
https://www.overleaf.com/project/5c7440defb0def2fb94c57a2

Figure 3: Overall pipeline for the proposed breast mass-detection system.

### Patch Extraction

Patch-based classification increases the robustness of the classifier by
increasing the number of training samples [20] and reducing the computational
complexity. In this study, patches having dimensions of $454\times 454$ pixels
are extracted from the original mammograms. To create the patches, a stride of
300 pixels was used which lowers the probability of redundant information
between two consecutive patches. A total of 1000 positive patches and 1000
negative patches were extracted for training, validation, and testing. Two
geometric data augmentations techniques, Flipping and Rotation, were employed
to avoid over-fitting as well as the curse of dimensionality. OpenCV’s python
API was used for flipping the patches in x-axis and also for performing 2D
rotations around row/2 and col/2 axis by $None$. The following pseudocode in
Algorithm 1 shows how the patches were extracted for training.

Algorithm 1 Pseudocode for Patch Extraction

for i=1:1:rows do

$Final\\_row=initial\\_row+patch\\_height$

if $Final\\_row<rows$ then

for j=1:1:columns do

$Final\\_column=initial\\_column+patch\\_width$

if $Final\\_column<columns$ then

$patch=Croped(Original\ Image)$

$Saved$

end if

$initial\\_column=initial\\_column+stride$

end for

$initial\\_column=1$

end if

$initial\\_row=initial\\_row+stride$

end for

### Feature Extraction using VGG19

After extracting the patches, feature extraction was conducted by passing the
patches through the pre-trained VGG19 network[21]. The pre-trained VGG19 model
has a depth of 19 layers and was trained using the ImageNet dataset[22]. VGG19
network can be characterized by its simplicity that uses only $3\times 3$
convolutional layers stacked on top of each other in increasing depths. To
reduce dimensionality, down-sampling of input representation (image, hidden-
layer output matrix, etc) is used in this network. Two fully-connected layers
(FC1 and FC2), each with 4096 nodes are then followed by a softmax layer which
in together forms the classifier. The adopted VGG19 [21] network with 19
layers is shown in detail in Fig. 4.

Figure 4: Details of the 19 layers of VGG19 network [21] used for feature
extraction.

Patches were normalized with zero mean and unit variance before feeding them
into the network for feature extraction. Two sets of features were extracted
from the VGG19 model, first one after the fully connected layer 2 (FC2) that
gives 4096 features and second one after the flatten layer (FL) that gives
25088 features. These two sets of features were later used to compare the
effect of the dimension of features vectors on the robustness and
computational complexity of final classifier.

### Feature Selection

Not all extracted features are relevant, reducing redundant and irrelevant
features benefits the classifier in a couple of ways. It i) reduces over-
fitting, ii) improves accuracy, iii) reduces training time, and iv) decreases
the complexity of the classifier. To select the optimum set of features,
bagged decision trees like Random Forest and Extra Trees were used. The bagged
algorithm [23] implements a meta estimator that fits several randomized
decision trees (also known as extra-trees) on various sub-samples of the
dataset. In this literature, Random Forest and Extra Trees are used to
estimate the importance of feature and then the features were selected
depending on their rank. To implement features selector, ExtraTreesClassifier
class in the scikit-learn API was used. To select the features from the
feature importance, 95% importance was considered as mentioned in En. 1. The
working flow diagram of the implemented feature selection technique is shown
in Fig. 5.

Figure 5: Flow diagram of the implemented feature selector.

$\centering Selected\;Feature\;importance=0.95\times
Total\;importance\@add@centering$ (1)

### Train, Validation and Test Data Selection

After selecting the optimum set of features, this feature vector was split
into train, validation and test data. To do so, firstly, whole data was
divided into five (5-fold cross validation [24]) equal parts as shown in Fig.
6. Cross-fold validation was done in order to get a more generalized model
that can provide results which can be expected from unseen test data. $N$ is
the total number of observations in both classes (mass and non-mass) and $M$
is the number of selected features where data was balanced in both the mass
and non-mass classes. Data having an equal number of classes was chosen
randomly: 60% for training, 20 % for validation and 20 % for testing.

Figure 6: Mass and Non-mass data split for the cross validation.

From Fig. 6, it is seen that there are 5 possible cases of train, validation
and test data split as given in Table 1. For each case, AUC for test data was
calculated after tuning the classifier’s hyper-parameters from the validation
stage.

Table 1: Training, validation and test selection Possible Cases | Train Set (60%) | Validation Set (20%) | Test Set (20%)
---|---|---|---
Mass class | Non-mass class | Mass class | Non-mass class | Mass class | Non-mass class
Case I | Group: A, B, C | Group: A, B, C | Group: D | Group: D | Group: E | Group: E
Case II | Group: B, C, D | Group: B, C, D | Group: E | Group: E | Group: A | Group: A
Case III | Group: C, D, E | Group: C, D, E | Group: A | Group: A | Group: B | Group: B
Case IV | Group: D, E, A | Group: D, E, A | Group: B | Group: B | Group: C | Group: C
Case V | Group: E, A, B | Group: E, A, B | Group: C | Group: C | Group: D | Group: D

### Performance Metric for SVM

In statistics, a Receiver Operating Characteristic (ROC)[25] curve is a
graphical plot that illustrates the diagnostic (classification) ability of a
binary classifier system as its discrimination threshold is varied which is
considered as a fundamental tool for diagnostic test evaluation. In a ROC
curve, the True Positive Rate (TPR) i.e. Sensitivity is plotted in function of
the False Positive Rate (FPR) (100-Specificity) for different cut-off points.
Each point on the ROC curve represents a sensitivity/ specificity pair.
Lowering the classification threshold classifies more items as positive, thus
increasing both False Positives (FP) and True Positives (TP). The AUC[25] is a
measure of how well a parameter can distinguish between two diagnostic class
which ranges between $0\sim 1$. Alternately, AUC provides an aggregate measure
of performance across all possible classification thresholds. In this paper,
standard deviation was used to measure the robustness of the classifier,
where, a small value of standard deviation indicates that classifier is more
generalized and robust on independent unseen data. Robustness of the
classifier means it is possible to reproduce the output class posterior
probability with a minimum level of discrepancy.

### Classification using Support Vector Machine (SVM)

Support vector machines (SVMs), originally proposed by V. Vapnik, have been
applied to many classification problems in medical imaging. However, classes
having larger training samples have a bias on the decision boundary [26]. This
unwanted class bias can be penalized using C-SVM and $\upsilon$-SVM by
introducing penalty parameter C and $\upsilon$ respectively. In C-SVM and
$\upsilon$-SVM, C and $\upsilon$ both are regularization parameters which
support to implement a penalty on the miss-classifications. After splitting
features vector in train, validation and test set, SVM [27] (C-SVM and
$\upsilon$-SVM) classifiers are trained and compared in order to select the
best classifier for the prediction of mass and non-mass patches.

To select the best hyper-parameters e.g. C and $\upsilon$ for C-SVM and
$\upsilon$-SVM, cross-validation was used. To do hyper-parameters tuning
(optimization), grid search [28] was used which can be defined simply as an
exhaustive searching through a manually specified subset of the hyper-
parameter space of a learning algorithm. A grid search algorithm should be
steered by some performance metric which was the average AUC in our case. The
overall workflow for getting best hyper-parameters and test AUC is shown in
Fig. 7.

Figure 7: Block diagram for tuning the best hyper-parameters and predicting
results on test set.

From Fig. 7, it is seen that from the set of penalization parameters (C and
$\upsilon$), grid search automatically return the best C and $\upsilon$ values
for C-SVM and $\upsilon$-SVM that maximizes AUC based on the validation sets.
After selecting the best values for C and $\upsilon$, the network was
evaluated using the corresponding test data. The five different values of AUC
is then obtained for the five different cases. The average AUC with standard
deviation was calculated and compared from the 5 fold cross-validation sets.
All experiments were performed in a Windows machine (Intel (R) Core (TN)
i5-7200U CPU @ 2.5 GHz 2.71 GHz). System type was 64-bit x64-based processor
with 8 GB of RAM.

## Results & Discussions

In all the experiments with C-SVM and $\upsilon$-SVM, only results with radial
basis function (RBF)[29] kernel are reported in this paper as it had the best
performance compared to using polynomial, linear or sigmoid kernel after
calibration. The research results described in this section is for
C=$10^{-3}\sim 10^{4}$ and $\upsilon$=$0.001\sim 0.9$ and these values of C
and $\upsilon$ were optimized in the validation stage by maximizing AUC. For
all these experiments, average AUC with standard deviation for test data and
total computation time (features extraction, training, penalization parameters
optimization, and testing of SVM model) were recorded. Some extracted patches
and their corresponding augmented (flipping and rotation) images are given in
Table 2. The experiments performed in this research for obtaining the best
classifier are as follows:

Table 2: Example of extracted Patches with corresponding geometric augmentation Positive patch | Flipped Positive | Rotated Positive | Negative Patch | Flipped Negative | Rotated Negative
---|---|---|---|---|---
|  |  |  |  |
|  |  |  |  |

### Experiment 1

In this experiment, the features come from fully connected layer 2 (FC2) of
VGG19 network that has 4096 features and using 2000 instances of observations
for both classes. Among 4096 features, 90 features that satisfy the En. 1 are
considered in this experiment. From validation set, it is found that maximum
AUC is at C = 100 and $\upsilon$ = 0.01. Using those values, the best model
was selected to examine on the unseen test data. Test results for C-SVM and
$\upsilon$-SVM are given in Table 3.

Table 3: Test results for C-SVM (Left) and $\upsilon$-SVM (Right) from Experiment 1 Test Data | C-SVM |  $\upsilon$-SVM
---|---|---
AUC | Average AUC | AUC | Average AUC
Case I | 0.95905 | 0.989 (+/-0.015) | 0.95800 | 0.988 (+/-0.016)
Case II | 0.99002 | 0.99025
Case III | 0.99900 | 0.99905
Case IV | 0.99495 | 0.99482
Case V | 1.00000 | 0.99995

From Table 3, it is seen that average AUC is 0.989 +/- 0.015 i.e. [0.974,
1.00] for C-SVM and 0.988 +/- 0.016 i.e. [0.972, 1.00] for $\upsilon$-SVM
respectively. The ROC curves for C-SVM and $\upsilon$-SVM are shown in Fig. 8
(left) and Fig. 8 (right). Total computation time which includes the feature
extraction, feature selection and classification stage was 11 mins using the
CPU machine. Although, both the penalization parameters, C and $\upsilon$ are
obtained from the grid search optimizer using the same set of validation data,
from Table 3 and Fig. 8, it is noticed that C-SVM performs better than the
$\upsilon$-SVM.

(a) For C-SVM

(b) For $\upsilon$-SVM

Figure 8: ROC curves for test data of Experiment 1.

### Experiment 2

In this experiment, the features were obtained from the flatten layer (FL) of
VGG19 network that has 25088 features and using 2000 instances of the
observations. Total of 189 features have been selected in this experiment.
From the validation set, it is found that maximum AUC is at C = 10 and
$\upsilon$ = 0.1. Best SVM model was taken using those values of C and
$\upsilon$ for test data. The obtained test results for C-SVM and
$\upsilon$-SVM are given in Table 4.

Table 4: Test results for C-SVM (Left) and $\upsilon$-SVM (Right) for Experiment 2 Test Data | C-SVM |  $\upsilon$-SVM
---|---|---
AUC | Average AUC | AUC | Average AUC
Case I | 0.916975 | 0.976 (+/-0.031) | 0.91697 | 0.976 (+/-0.031)
Case II | 0.971325 | 0.97132
Case III | 0.999224 | 0.99922
Case IV | 0.995400 | 0.99540
Case V | 0.997675 | 0.99767

It is seen from Table 4 that average AUC is 0.976 +/-0.031 i.e. [0.945, 1.00]
and 0.976 +/-0.031 i.e. [0.945, 1.00] for C-SVM and $\upsilon$-SVM
respectively. The ROC curves for C-SVM and $\upsilon$-SVM are shown in Fig. 9
(left) and Fig. 9 (right). Total computation time now was 40 mins. From
Experiment 2, it can be seen that both the C-SVM and $\upsilon$ are giving
similar results. The performance of Experiment 1 is slightly better than the
performance of Experiment 2 even though both the experiments have the same
number of observations. This can be explained by the fact that Experiment 2
uses a higher number of features which can result in overfitting.

(a) For C-SVM

(b) For $\upsilon$-SVM

Figure 9: ROC curves for test data of Experiment 2.

### Experiment 3

In this experiment, two geometric augmentations (flipping and rotation) was
added on extracted patches and the features were extracted from fully
connected layer 2 (FC2) of VGG19 network which gave 4096 features and now with
augmentation, there were 6000 instances of the observations for this
experiment. The selected feature number is 97 in this experiment. From the
validation test, it is seen that maximum AUC is at C= 100 and
$\upsilon$=0.005. Using these values, the best SVM model was selected to
examine the test data. Test results for C-SVM and $\upsilon$-SVM are given in
Table 5. Average AUC is 0.994 +/-0.003 i.e. [0.991, 0.997] and 0.994 +/-0.003
i.e. [0.991, 0.997] for C-SVM and $\upsilon$-SVM respectively. The ROC curves
for C-SVM and $\upsilon$-SVM are shown in Fig. 10 (left) and Fig. 10 (right).
Total computation time for this experiment was 3 hrs18 mins. Although the
computation time here is more than the first two experiments, the performance
is much better than the previous two experiments. From Fig. 10 and Table 5, it
can clearly be seen that for both C-SVM and $\upsilon$-SVM and for all the
test cases, the ROC curves and AUC’s are quite similar. This shows the
robustness of this method compared to the first two experiments.

Table 5: Test results for C-SVM (Left) and $\upsilon$-SVM (Right) for Experiment 3 Test Data | C-SVM |  $\upsilon$-SVM
---|---|---
AUC | Average AUC | AUC | Average AUC
Case I | 0.99521 | 0.994 (+/-0.003) | 0.99537 | 0.994 (+/-0.003))
Case II | 0.99453 | 0.99441
Case III | 0.99079 | 0.99129
Case IV | 0.99893 | 0.99906
Case V | 0.99109 | 0.99129

(a) For C-SVM

(b) For $\upsilon$-SVM

Figure 10: ROC curves for test data of Experiment 3.

### Experiment 4

In this experiment, again the same two geometric augmentations (flipping and
rotation) were executed on extracted patches and the features were extracted
from the flatten layer (FL) of the network which provides 25088 features. In
this experiment, a total of 205 features were selected. From the validation
test, it is seen that maximum AUC is at C=10 and $\upsilon$=0.001. Using these
values of the hyper-parameters, best model was selected to validate the test
data. Test results for C-SVM and $\upsilon$-SVM are given in Table 6. Now, the
average AUC is 0.988 +/- 0.009 i.e. [0.991, 0.997] and 0.988 +/- 0.009 i.e.
[0.991, 0.997] for C-SVM and $\upsilon$-SVM respectively.

Table 6: Test results for C-SVM (Left) and $\upsilon$-SVM (Right) for Experiment 4 Test Data | C-SVM |  $\upsilon$-SVM
---|---|---
AUC | Average AUC | AUC | Average AUC
Case I | 0.98625 | 0.988 (+/-0.009) | 0.98636 | 0.988 (+/-0.009)
Case II | 0.97621 | 0.97550
Case III | 0.98049 | 0.98117
Case IV | 0.99823 | 0.99820
Case V | 0.99704 | 0.99688

The ROC curves for C-SVM and $\upsilon$-SVM are shown in Fig. 11 (left) and
Fig. 11 (right). Total computation time for this experiment was 12 hrs. From
Fig. 11, it is observed that the ROC behavior of all the cases of the test
data in this approach is better than the first two experiments with a
significant increase in the computation time.

(a) For C-SVM

(b) For $\upsilon$-SVM

Figure 11: ROC curves for test data of Experiment 4.

## Conclusions

In this literature, the robustness of the VGG19 along with SVM (C-SVM and
$\upsilon$-SVM) for the breast mass and non-mass classification was analyzed.
All the parameters and hyperparameters of the C-SVM and $\upsilon$-SVM (except
the penalization parameters), the number of observations, and the dimensions
of the features vector were kept the same during each of the experiment. For
each experiment, the test results were obtained using the best C value for
C-SVM and best $\upsilon$ value for $\upsilon$-SVM using extensive grid search
algorithm. From the experimental results, it is worth mentioning that
Experiment 4 gives higher results than the first two experiments. The observed
reason can be pointed as Experiment 4 used a higher number of the observations
which is three times more than the first two experiments. But,
computationally, 12 hrs is excessively higher compared to all the other
experiments. On the other-hand, Experiment 3 has the best results than all the
other experiments with an AUC of 0.994 +/-0.003 i.e. [0.991, 0.997] and
computation time of 3 hrs 18 mins. From the ROC curves of all the experiments,
qualitatively it can be seen that Experiment 3 is more robust than others due
to having almost similar ROC for all the test cases (for each fold). From the
experimental tables, quantitatively, it is seen that Experiment 3 has less
value of standard deviation which also quantitatively proves that this method
is more robust than others. Same performance (AUC) for both C-SVM and
$\upsilon$-SVM are achieved in experiment 3 using grid search algorithm for
the hyper-parameters optimization which resulted in values for C and
$\upsilon$ at100 and 0.005. Either one of the C-SVM and $\upsilon$-SVM method
can be selected for the future applications for mass and non-mass breast
tissue classification as they are expected to give similar results using the
trained classifier from experiment 3. We conclude that even with a small
training set, it is possible to obtain a robust classifier for the mass and
non-mass tissue classification in the breast using our proposed pipeline.

## References

  * [1] Bray, F. & et. al.  Global cancer statistics 2018: Globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries.  _CA Cancer J Clin._ 68, 394–424, DOI: 10.3322/caac.21492 (2018).
  * [2] World Health Organization: Cancer.  Available at: http://www.who.int/news-room/fact-sheets/detail/cancer.
  * [3] Hou, L. _et al._ Cancer statistics.  _Wiley Online Library_ 68, 7–30, DOI: https://doi.org/10.3322/caac.21551 (2018).
  * [4] Breast cancer.  Available at: https://www.kpwomenshealth.org/breast_health_breast_cancer.asp.
  * [5] Sree, S. V., Ng, E. Y., Acharya, R. U. & Fauste, O.  Breast imaging: A survey.  _World Journal of Clinical Oncology_ 2, 171–178, DOI: 10.5306/wjco.v2.i4.171 (2011).
  * [6] Uematsu, T.  Non-mass-like lesions on breast ultrasonography: a systematic review.  _Breast Cancer_ 19, 295–301, DOI: 10.1007/s12282-012-0364-z (2012).
  * [7] Nunes, A., Silva, A. & Paiva, A.  Detection of masses in mammographic images using geometry, simpson’s diversity index and svm.  _International Journal of Signal and Imaging Systems Engineering (IJSISE)_ 3, DOI: 10.1504/IJSISE.2010.034631 (2010).
  * [8] Newell, D. _et al._ Selection of diagnostic features on breast mri to differentiate between malignant and benign lesions using computer-aided diagnosis: differences in lesions presenting as mass and non-mass-like enhancement.  _European Radiology_ 20, 771–781, DOI: 10.1007/s00330-009-1616-y (2010).
  * [9] John, E. R. S. & et. al.  Rapid evaporative ionisation mass spectrometry of electrosurgical vapours for the identification of breast pathology: towards an intelligent knife for breast cancer surgery.  _Breast Cancer Research_ 19, DOI: https://doi.org/10.1186/s13058-017-0845-2 (2017).
  * [10] Suzuki, S. & et. al.  Mass detection using deep convolutional neural network for mammographic computer-aided diagnosis.  _Annual Conference of the Society of Instrument and Control Engineers of Japan (SICE)_ 1382–1386, DOI: 10.1109/SICE.2016.7749265 (2016).
  * [11] Wang, Y. & et al.  Computer-aided classification of mammographic masses using visually sensitive image features.  _Journal of X-Ray Science and Technology_ 25, 1–16, DOI: 10.3233/XST-16212 (2016).
  * [12] Oliveira, F. S. S. D. & et. al.  Classification of breast regions as mass and non-mass based on digital mammograms using taxonomic indexes and svm.  _Breast Cancer_ 57, 42–53, DOI: https://doi.org/10.1016/j.compbiomed.2014.11.016 (2015).
  * [13] Varela, C. & et. al.  Non-mass-like lesions on breast ultrasonography: a systematic review.  _Physics in Medicine & Biology_ 51, DOI: 10.1088/0031-9155/51/2/016 (2006).
  * [14] Wei, J. & et. al.  Computer aided detection of breast masses on full field digital mammograms.  _Med. Phys._ 32, 2827–2837, DOI: 0.1118/1.1997327 (2005).
  * [15] Wichakam, I.  Combining deep convolutional networks and svms for mass detection on digital mammograms.  _International Conference on Knowledge and Smart Technology (KST)_ 239–244, DOI: 10.1109/KST.2016.7440527 (2016).
  * [16] Ragab, D. A. & et. al.  Breast cancer detection using deep convolutional neural networks and support vector machines.  _PeerJ_ DOI: http://doi.org/10.7717/peerj.6201 (2016).
  * [17] Sampaio, W. B. & et. al.  Detection of masses in mammogram images using cnn, geostatistic functions and svm.  _Comput Biol Med_ 653–664, DOI: 10.1016/j.compbiomed.2011.05.017 (2011).
  * [18] Wael, E. F. & et. al.  A deep learning approach for breast cancer mass detection.  _International Journal of Advanced Computer Science and Applications_ 10 (2019).
  * [19] Moreira, I. C. & et. al.  Inbreast: toward a full-field digital mammographic database.  _Academia Radiology_ 19, 236–248, DOI: 10.1016/j.acra.2011.09.014 (2012).
  * [20] Seigel, R. L., Miller, D., K. & Jemal, A.  Patch-based convolutional neural network for whole slide tissue image classification.  _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ DOI: 10.1109/CVPR.2016.266 (2016).
  * [21] Simonyan, K. & et. al.  Very deep convolutional networks for large-scale image recognition.  _Computer Vision and Pattern Recognition (cs.CV)_ DOI: arXiv:1409.1556 (2014).
  * [22] Deng, J. & et. al.  Imagenet: A large-scale hierarchical image database.  _IEEE Conference on Computer Vision and Pattern Recognition_ DOI: 10.1109/CVPR.2009.5206848 (2009).
  * [23] Guan, D. & et. al.  A review of ensemble learning based feature selection.  _IETE Technical Review_ 31, 190–198, DOI: https://doi.org/10.1080/02564602.2014.906859 (2014).
  * [24] Wong, T. & et. al.  Dependency analysis of accuracy estimates in k-fold cross validation.  _IEEE Transactions on Knowledge and Data Engineering_ 29, 2417–2427, DOI: 10.1109/TKDE.2017.2740926 (2017).
  * [25] Marzban, C. & et. al.  The roc curve and the area under it as performance measures.  _Weather and Forecasting_ 19, 1106, DOI: https://doi.org/10.1175/825.1 (2004).
  * [26] Liu, S. & et. al.  A new weighted support vector machine with ga-based parameter selection.  _International Conference on Machine Learning and Cybernetics_ DOI: 10.1109/ICMLC.2005.1527703 (2005).
  * [27] Karatzoglou, A. & et. al.  Support vector machines in r.  _Journal of Statistical Software_ 15 (2006).
  * [28] Diaz, G. & et. al.  An effective algorithm for hyperparameter optimization of neural networks.  _Ibm Journal of Research and Development_ 61, DOI: 10.1147/JRD.2017.2709578 (2017).
  * [29] Kuo, B. & et. al.  A kernel-based feature selection method for svm with rbf kernel for hyperspectral image classification.  _IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing_ 7, 317–326, DOI: 10.1109/JSTARS.2013.2262926 (2014).


# Let the Poem Hit the Rhythm: Using a Byte-Based Transformer for Beat-Aligned
Poetry Generation

Mohamad Elzohbi, Richard Zhao
Department of Computer Science
University of Calgary
Calgary, Alberta, Canada, T2N 1N4
{melzohbi<EMAIL_ADDRESS>

###### Abstract

> The intersection between poetry and music provides an interesting case for
> computational creativity, yet remains relatively unexplored. This paper
> explores the integration of poetry and music through the lens of beat
> patterns, investigating whether a byte-based language model can generate
> words that fit specific beat patterns within the context of poetry. Drawing
> on earlier studies, we developed a method to train a byte-based transformer
> model, ByT5, to align poems with beat patterns. The results demonstrate a
> high level of beat alignment while maintaining semantic coherence. Future
> work will aim to improve the model’s ability to create complete beat-aligned
> poems.

## Introduction

The integration of poetry and music has a significant impact on the aesthetic
reception of both art forms. This intersection is evident in the shared
terminology, such as “rhythm”, which plays a vital role in cultural
transmission and emotional expression. Despite this similarity, limited
comparative studies have explored rhythms in music as opposed to rhythms in
poetry. This scarcity can mainly be attributed to the complexity of the
involvement of experts in both domains (?). The patterns and sounds of words
are generally overlooked when studying poetry, whereas studies tend to
concentrate on other language-related aspects (?).

Songs are great examples of creative language meeting music. They often stick
to lyrics that fit well with the musical tones and beats forming patterns that
not only convey figurative meaning but also align smoothly with the music.
Linguistic stress is comparable to musical accents, indicating points of
emphasis. ? (?) explored the relationship between the sounds of English poetry
and music, focusing on how stress patterns and pitch levels in poetry and
music are related. In a later study, ? (?) used the Beatles’ song “Yesterday”
as a key example to demonstrate these connections. The challenge in generating
lyrics is in balancing coherence with fitting into a steady musical rhythm.
While many studies focus on rhyme, they often overlook the essential rhythmic
component of the lyrics (?). However, some research has explored generating
lyrics of various languages that align with music from stress and pitch
perspective (?; ?; ?).

Additionally, beat rhythm is another point of comparison between language and
music, though whether the language itself is rhythmic has been a subject of
debate and research has led to controversial findings (?). The African talking
drums provide a fascinating case study of the convergence of sound and
language. These drums are used to mimic the tonal and rhythmic patterns of
spoken language, acting as acoustic speech surrogates illustrating the
relationship between rhythmic patterns and verbal communication (?).
Similarly, Arabic poetry has deep ties to drum music, serving as the
foundation for Arabic prosody. It is said that the rhythmic pounding of
hammers in the street of braziers in al-Baṣrah inspired al-Khalīl al-Farāhīdī
to establish the rules of Arabic prosody based on the alignment he found
between the clanging patterns and the speech rhythm in poetry (?).

? (?) studied the perceptual beat locations in words taken from spontaneous
English conversations. The study involved several experiments including having
participants tap their fingers along with perceived beats in the words,
confirming that rhythmic beats often align with vowel onsets, i.e. the
transitions from consonants to vowels, and were more accurately detected in
stressed syllables. ? (?) recently revisited the topic and studied the
validity of sensorimotor synchronization to perceive speech rhythms where
similar tapping experiments showed that people tend to identify perceptual
beats at vowel onsets in English sentences. These studies mainly focus on
English, but a similar concept can be observed in other languages such as
Arabic, where scansion places rhythmic beats on vocalized consonants (those
followed by short vowels), which characterize the meter of the poem (?).

Figure 1: The process of converting a four-syllable phrase “I believe in” from
its phones to a beat pattern.

This raises several research questions: Can a large language model effectively
learn beat patterns perceived from natural language? How accurately can such a
model generate textual content that conforms to predetermined rhythmic
constraints? And, importantly, does this capability extend to maintaining
semantic coherence while adhering to these constraints?

Building on these questions, this short paper seeks to determine if a large
language model can be trained to select words that align with a given beat
pattern. Our focus is on generating words in the context of poems, aiming to
make them resemble lyrics used in music. To achieve this, we develop a model
that maps graphemes to beat patterns, building on insights from earlier
research. We then utilize a byte-based transformer model to replace or insert
words that align with a specified beat pattern, all without exposing the model
to phonemes or audio-based data, but instead by interpreting symbolic beat
patterns derived from the arrangement of consonants and vowels. To measure the
model’s quality, we employ automated evaluation metrics.

## Methodology

In this study, we selected the ByT5 model (?), a byte-level transformer model
derived from the T5 architecture which is an encoder-decoder transformer
developed by Google (?). Unlike models that break text into subword-level
units, ByT5 operates at the character level, providing the precision needed to
handle text with granular control over character-level patterns.

### Task Formalization

The task involves inserting a set of words
$W^{\prime}=(w^{\prime}_{1},w^{\prime}_{2},\ldots,w^{\prime}_{i})$ within a
given poetry verse $S=(w_{1},w_{2},\ldots,w_{n})$, $i<n$, that exhibit a given
beat, $B(W^{\prime})$. The task may also be formalized as replacing a set of
words $W\subset S$ with an undesirable beat $B(W)$ with a set of words
$W^{\prime}$ of desired beat $B(W^{\prime})$. This task requires a poetry
dataset as well as a graphemes-to-beat transformation $B(.)$.

### Dataset

The dataset used in this study is derived from the English-labeled subset of a
poetry corpus compiled by ? (?). To ensure the dataset contained only English
poems, we applied additional cleaning using an XLM-RoBERTa-based language
identification model trained on a language identification dataset111The model
is available at: https://huggingface.co/papluca. This step filtered out non-
English poetry verses, leaving only the relevant poetry verses for our
analysis. We used only the poetry verses from the dataset ignoring non-
relevant information.

Figure 2: Grapheme-to-Beat pattern model. Figure 3: An example input/output of
the ByT5-B in green and ByT5-CV in orange.

### Grapheme-to-Beat Transformation

To identify the rhythmic beats in English words, we follow the findings
outlined in past studies (?; ?; ?). That is, we place the beat attack on the
transition from a consonant to a vowel. A syllable comprises an onset, a
nucleus, and a coda. The nucleus is typically a vowel- either a short vowel,
long vowel, or diphthong- while the onset and coda can include zero or more
consonants. It is possible for a syllable in English to have a null onset or
coda, but if there is no onset, a glottal stop generally precedes the vowel
(?), which can still be considered a vowel onset. Any other sounds — such as
consonants not followed by a vowel, vowel elongations, or the second vowel in
a diphthong — are treated as non-beat units (or rests) (Figure 1).

English script is orthographic deep, where there is little one-to-one
correspondence between letters and sounds. To get the consonant-vowel pattern
$CV(w)$ for a given word $w$, we first need to extract its phonemes from the
English script. To do this, we used a pre-trained grapheme-to-phoneme
transformer model222https://github.com/as-ideas/DeepPhonemizer. This model
transforms text into phonemes, which we then convert to consonants and vowels
with one-to-one mapping. Long vowels and diphthongs are mapped to double
vowels “VV”, while consonants are denoted by “C”. We then convert these CV
patterns into beat patterns: a “1” represents a beat where there is a vowel
onset, and a “0” represents a non-beat unit (or rest) (Figure 2).

### Model Training

We fine-tuned the pre-trained ByT5 model on the processed dataset to generate
words that align with a specified rhythmic beat pattern. During training, we
used a masking strategy to simulate the task’s objective. For each verse, let
us say $S=(l_{1},l_{2},...,l_{n})$ where $l$ represents an individual letter
in the verse, we randomly masked a set of words
$W=(l_{i},l_{i+1},...,l_{i+j-1})$ where $i$ is the position of the first
letter and $j$ is the total letter count in $W$.

To indicate where the masking occurred in $S$, we used special tokens ($E_{0}$
and $E_{1}$) so that the masked version of $S$ looks like this:
$S^{\prime}=(l_{1},...,E_{0},B(W),E_{1},...,l_{n})$. $B(W)$ represents the
beat pattern of the original words. We then appended a special token $E_{2}$
followed by $W$ to prompt the model to align the words with their
corresponding beats (Figure 3). For a comparative analysis, we used a similar
strategy to fine-tune a model that focused on consonant-vowel patterns with
$CV(W)$ replacing $B(W)$ in $S^{\prime}$.

To ensure the model could handle spans of varying lengths, we applied a span-
masking strategy inspired by SpanBERT (?). This strategy involved sampling
span lengths from a geometric distribution with a probability parameter of
$0.2$. A maximum threshold was set at $25\%$ of verse words’ count, but the
geometric distribution favours shorter spans.

## Experimental Setup

### Dataset Split

Our dataset consists of $1,038,743$ verses obtained after data preprocessing.
We excluded potentially garbled lines, as was done in the Quatrain dataset
(?). We then split the dataset into two random subsets: a training set
comprising $1,033,549$ examples and an evaluation set with $5,194$ examples.

### Training Setup

We fine-tuned two ByT5-base models on the specified tasks: ByT5-B was trained
with a focus on beat patterns, while ByT5-CV focused on consonant-vowel
patterns. We also fine-tuned two ByT5-base models ByT5-CV-base and ByT5-B-base
where attention masks were set to zero in the pattern positions, ensuring that
the model did not attend to these patterns, using this configuration as a
baseline. All models were trained for four epochs on an NVIDIA A100 GPU. The
learning rate was set to $3e-4$ with a cosine scheduler and a weight decay of
$0.1$. The training batch size was set to $128$, and the evaluation batch size
was set to $16$.

###  GPT-4 Baseline

To compare our results against the current state-of-the-art large language
models, we used GPT-4 (version: gpt-4-0613), which is OpenAI’s most capable
model at the time of this research. We randomly selected $1000$ examples from
our dataset and masked them as described. We prompted GPT-4 with a short
explanation of the problem and a few-shot examples to evaluate its performance
on reconstructing beat patterns333The source code, the dataset, and GPT-4
prompts used are available at: https://github.com/melzohbi/poem-rhythm. At the
time of this research, we did not have access to any fine-tuning capabilities
of GPT-4. However, the purpose of this comparison was to assess whether GPT-4,
in its current state, could understand and perform this task effectively,
given its proficiency in various downstream tasks. We acknowledge that fine-
tuning is important for a fairer comparison, and we plan to compare our model
with a fine-tuned version of GPT-4 in the future if such capabilities become
available.

### Automated Evaluation Metrics

The task’s primary objective is to generate words that match a beat pattern
while maintaining semantic coherence within the context of a poetry verse.
Using automated evaluation, we measure Coherence and Beat Alignment (while
human-based evaluations will be sought in future work).

#### Coherence Score

measures how well the predicted words fit with the surrounding context. To
evaluate coherence, we utilized the original subword-level T5 model (using the
base-size version) to derive a log-perplexity score (through calculating the
cross-entropy loss). We inserted a special token at the location of the masked
span in the verse, so the sequence became
$S=(w_{1},\ldots,E_{0},\ldots,w_{n})$. We then prompted the model to predict
the tokens replacing the special $E_{0}$ token, computing the cross-entropy
loss for the prediction. This pre-trained model, serving as an external
scoring mechanism, was independent of our dataset. This setup allows us to
verify whether or not our training process impacts the coherence of the model.

#### Alignment Scores

evaluate the alignment of generated words with the required beat rhythm. We
employed two metrics:

  * •

Exact Alignment Accuracy: This metric checks whether the generated word
precisely aligns with the expected beat rhythm, resulting in a binary outcome
(0 for non-alignment, 1 for exact alignment).

  * •

Levenshtein distance: This metric assesses alignment by calculating the
normalized Levenshtein distance between the generated word and the expected
beat rhythm. This measure accounts for cases where the alignment may not be
exact but is reasonably close, providing additional flexibility in evaluation.

## Results

This section explores how well the fine-tuned ByT5 models meet the task’s
objectives. We evaluate their performance in terms of Coherence and Beat
Alignment. In the following section, we discuss the implications of these
results for further research.

#### Coherence:

The results in Table 1 show that both models achieved a comparable log
perplexity with the baseline model, with the ByT5-CV version slightly ahead
with a log-perplexity score of $7.48$, compared to $7.494$ for the ByT5-B
version. The baseline models score between $7.442$ and $7.432$. These outcomes
suggest that the custom fine-tuning process didn’t significantly impact the
model’s semantic coherence, indicating that it adapted well to the rhythmic
task without sacrificing fluency. We did not evaluate GPT-4 output for
coherence, given that T5 exhibits lower fluency compared to GPT-4.

#### Alignment Scores:

Both of our models showed a high level of beat alignment accuracy. The ByT5-CV
model achieved an Exact Alignment Accuracy score of $98.88\%$, while the
ByT5-B variant achieved $98.31\%$. The Levenshtein distance, which allows for
some flexibility, indicated near-perfect scores of $99.83\%$ and $99.63\%$
respectively, suggesting that the models generally aligned with the rhythm,
with only minor deviations. This demonstrates that the models understand
rhythmic patterns even when they do not exactly match. In contrast, the
baseline ByT5 models, including GPT-4 on the $1000$ examples, had much lower
scores for both exact alignment and Levenshtein distance, indicating that
these patterns are challenging to learn without proper training.

These findings confirm that our fine-tuned ByT5 models can generate words that
align with beat patterns while maintaining coherence. This sets the stage for
further research into generating entire verses or poems with consistent
rhythmic structures.

Model | Coherence | Accuracy | Levenshtein
---|---|---|---
ByT5-Base-CV | 7.442 | 0.3512 | 0.7670
ByT5-Base-B | 7.432 | 0.5575 | 0.8323
GPT-4 | - | 0.2550 | 0.6105
ByT5-CV | 7.480 | 0.9888 | 0.9983
ByT5-B | 7.494 | 0.9831 | 0.9963
Table 1: Performance comparison between our consonant-vowel model, beat model
and the baselines. For Coherence, lower is better. For Exact Alignment
Accuracy and Levenshtein distance, higher is better.

## Conclusion and Future Work

In this study, we investigated the capabilities of ByT5, a byte-level language
model, particularly its aptitude for generating words that conform to specific
consonant-vowel patterns and rhythmic beats. Our methodology focused on fine-
tuning multiple ByT5-based models on a conditional mask-predict objective to
reconstruct words with predetermined consonant-vowel patterns and rhythmic
beats. The results demonstrated that our models were able to generate words
that align with specified patterns while maintaining semantic coherence. Our
models showed high rhythmic alignment accuracy indicating their effectiveness
in this task without adversely sacrificing the models’ fluency.

The models have potential applications in various co-creative frameworks,
including songwriting, rap lyric generation, and in the process of rhythmic
poetry creation. In future work, we aim to expand the model’s capabilities to
not only insert individual phrases but also generate complete verses or poems
that maintain rhythmic coherence throughout. We will also explore the impact
of unstressed syllables on beat strength and the precise locations and
durations of beats backed by human evaluation. By advancing these aspects, we
intend to enhance the model’s utility furthering the integration between
poetry and music.

## Acknowledgements

We thank the anonymous reviewers for their valuable feedback and members of
the Serious Games Research Group at the University of Calgary for their
support. We appreciate the scholarship provided by Elsevier’s Artificial
Intelligence Journal through the ICCC program committee.

## References

  * [Allen 1972] Allen, G. D.  1972\.  The location of rhythmic stress beats in english: an experimental study i.  Language and Speech 15(1):72–100.  PMID: 5073939.
  * [Belouadi and Eger 2023] Belouadi, J., and Eger, S.  2023\.  Bygpt5: End-to-end style-conditioned poetry generation with token-free language models.  In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 7364–7381.
  * [Chen and Teufel 2024] Chen, Y., and Teufel, S.  2024\.  Scansion-based lyrics generation.  In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 14370–14381.
  * [Conneau et al. 2019] Conneau, A.; Khandelwal, K.; Goyal, N.; Chaudhary, V.; Wenzek, G.; Guzmán, F.; Grave, E.; Ott, M.; Zettlemoyer, L.; and Stoyanov, V.  2019\.  Unsupervised cross-lingual representation learning at scale.  arXiv preprint arXiv:1911.02116.
  * [Durojaye et al. 2021] Durojaye, C.; Knowles, K. L.; Patten, K. J.; Garcia, M. J.; and McBeath, M. K.  2021\.  When music speaks: An acoustic study of the speech surrogacy of the nigerian dùndùn talking drum.  Frontiers in Communication 6:652690.
  * [Frolov 2000] Frolov, D.  2000\.  Classical Arabic Verse: History and Theory of ‘arūd, volume 21.  Brill.
  * [Haider 2021] Haider, T.  2021\.  Metrical tagging in the wild: Building and annotating poetry corpora with rhythmic features.  In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, 3715–3725.  Online: Association for Computational Linguistics.
  * [Hermes 2023] Hermes, D. J.  2023\.  Beat detection.  In The Perceptual Structure of Sound. Cham, Switzerland: Springer International Publishing.  225–259.
  * [Hirjee and Brown 2009] Hirjee, H., and Brown, D. G.  2009\.  Automatic detection of internal and imperfect rhymes in rap lyrics.  In Proceedings of the 10th International Society for Music Information Retrieval Conference (ISMIR), 711–716.
  * [Ibn al Mu’tazz 1976] Ibn al Mu’tazz, A.  1976\.  Tabaqāt al-Shu‘arā’ al-Muḥdathīn.  Transcription, Dar Al-Ma‘arif.
  * [Joshi et al. 2020] Joshi, M.; Chen, D.; Liu, Y.; Weld, D. S.; Zettlemoyer, L.; and Levy, O.  2020\.  Spanbert: Improving pre-training by representing and predicting spans.  Transactions of the association for computational linguistics 8:64–77.
  * [Lerdahl 2001] Lerdahl, F.  2001\.  The sounds of poetry viewed as music.  Annals of the New York Academy of Sciences 930(1):337–354.
  * [Lerdahl 2013] Lerdahl, F.  2013\.  Musical syntax and its relation to linguistic syntax.  Language, Music and the Brain, ed M. A. Arbib 257–272.
  * [Mesaros and Virtanen 2008] Mesaros, A., and Virtanen, T.  2008\.  Automatic alignment of music audio and lyrics.  In Proceedings of the 11th Int. Conference on Digital Audio Effects (DAFx-08).
  * [Oliveira, Cardoso, and Pereira 2007] Oliveira, H. R. G.; Cardoso, F. A.; and Pereira, F. C.  2007\.  Tra-la-lyrics: An approach to generate text based on rhythm.  In Proceedings of the 4th. International Joint Workshop on Computational Creativity.  A. Cardoso and G. Wiggins.
  * [Ong 1977] Ong, W. J.  1977\.  African talking drums and oral noetics.  New Literary History 8(3):411–429.
  * [Patel 2010] Patel, A. D.  2010\.  Music, language, and the Brain.  Oxford University Press.
  * [Raffel et al. 2020] Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J.  2020\.  Exploring the limits of transfer learning with a unified text-to-text transformer.  Journal of machine learning research 21(140):1–67.
  * [Rathcke et al. 2021] Rathcke, T.; Lin, C.-y.; Falk, S.; and Dalla Bella, S.  2021\.  Tapping into linguistic rhythm.  Laboratory Phonology 12(1):1–32.
  * [Sheng et al. 2021] Sheng, Z.; Song, K.; Tan, X.; Ren, Y.; Ye, W.; Zhang, S.; and Qin, T.  2021\.  Songmass: Automatic song writing with pre-training and alignment constraint.  In Proceedings of the AAAI Conference on Artificial Intelligence, 13798–13805.
  * [Xue et al. 2021] Xue, L.; Song, K.; Wu, D.; Tan, X.; Zhang, N. L.; Qin, T.; Zhang, W.-Q.; and Liu, T.-Y.  2021\.  Deeprapper: Neural rap generation with rhyme and rhythm modeling.  In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP), 69–81.
  * [Xue et al. 2022] Xue, L.; Barua, A.; Constant, N.; Al-Rfou, R.; Narang, S.; Kale, M.; Roberts, A.; and Raffel, C.  2022\.  Byt5: Towards a token-free future with pre-trained byte-to-byte models.  Transactions of the Association for Computational Linguistics 10:291–306.


# Game-theoretic Learning Anti-jamming Approaches in Wireless Networks

Luliang Jia, Nan Qi, Feihuang Chu, Shengliang Fang, Ximing Wang, Shuli Ma, and
Shuo Feng

###### Abstract

In this article, the anti-jamming communication problem is investigated from a
game-theoretic learning perspective. By exploring and analyzing intelligent
anti-jamming communication, we present the characteristics of jammers and the
requirements of an intelligent anti-jamming approach. Such approach is
required of self-sensing, self-decision making, self-coordination, self-
evaluation, and learning ability. Then, a game-theoretic learning anti-jamming
(GTLAJ) paradigm is proposed, and its framework and challenges of GTLAJ are
introduced. Moreover, through three cases, i.e., Stackelberg anti-jamming
game, Markov anti-jamming game and hypergraph-based anti-jamming game,
different anti-jamming game models and applications are discussed, and some
future directions are presented.

##  I Introduction

Due to the sharing and open nature, wireless communications are vulnerable to
jamming attack, which has been a critical and major threat that captured
growing attention in the past decade [1]-[4]. Various anti-jamming approaches
have been proposed to cope with jamming attacks, which can be divided into
five categories: confrontation, avoidance, elimination, hide, and deceit [5].
Many common anti-jamming countermeasures are adopted, such as frequency
hopping spread spectrum (FHSS) and direct sequence spread spectrum (DSSS).
However, traditional anti-jamming approaches are in low spectral efficiency,
and sacrifice resource utilization efficiency for transmission reliability.
Moreover, they employ fixed or preset patterns that lack of intelligent
decision-making ability. Thus, it is of great importance to develop effective
anti-jamming approaches to fight against jamming attacks, especially for
scenarios with limited resources and intelligent jammers.

With the rapid development of cognitive radio and artificial intelligence
technology, the anti-jamming problem has been endowed with new content. On the
one hand, jamming attacks can utilize these emerging technologies to produce
more complex and effective jamming patterns. On the other hand, these new
technologies also provide new ideas to improve the anti-jamming ability for
wireless communication systems. Thus, it is a trend to learn from cognitive
radio and artificial intelligence technologies. In particular, the game-
theoretic learning method is an extremely promising technology, since it has
the following advantages in anti-jamming field:

  * •

A jammer aims to intentionally disrupt the transmission of legitimate users,
making their communication ability reduced or lost. However, legitimate users
pursue eliminating the impacts of various jamming attacks as much as possible
to achieve reliable information transmission. Due to the natural confrontation
characteristic, there are interactions between legitimate users and jammers.
Game theory stands out as a well-developed tool to formulate the mutual
interactions among players, and it can adequately formulate and analyze
interactions between legitimate users and jammers [4]-[12].

  * •

It is difficult to obtain accurate information of the opponents for both
legitimate users and jammers. Moreover, the jamming environment is dynamic
because of the jamming activities and time-varying electromagnetic
environment. Thus, the anti-jamming problem needs to deal with dynamic and
incomplete (DI) information constraints. Learning is an effective method that
can cope with the DI information constraints through trial and error
interaction with the jamming environment [9].

Game theory has provided insights to improve anti-jamming performance. In [6],
an anti-jamming game was formulated, and a coordination-learning approach was
proposed. In [7], a bimatrix game was employed to analyze and model the
interaction between the legitimate link and the jammer. In [8], a Stackelberg
power game with observation errors was investigated. In [9]-[12], the authors
investigated some preliminary exploration on game-theoretic learning anti-
jamming approach. These existing works are summarized in Table I.

TABLE I: Summary of game theory-based anti-jamming approaches.

Ref |  Models |  Techniques |  Situations |  Advantages
---|---|---|---|---
[6] |  Correlated equilibrium game |  Coordination-learning |  Interference and jamming |  Taking jamming signals as coordination signals.
[7] |  Bimatrix game |  Linear programs |  Jamming |  Describing confrontation relationship.
[8], [9] |  Stackelberg game |  Convex optimization; Hierarchical learning |  Jamming [8];Interference and jamming [9] |  Capturing hierarchical behaviors; Analyzing competitive interactions at different levels.
[10] |  Hypergraph game |  Stochastic learning |  Interference and jamming |  Capturing accurate interference relationship.
[11] |  Markov game |  Multi-agent Q-learning |  Interference and jamming |  Describing the dynamic characteristic of jamming environment.

Motivated by the above, we propose a new anti-jamming paradigm called game-
theoretic learning anti-jamming (GTLAJ) communication, which integrates game
theory into solving the anti-jamming problem. Note that the GTLAJ is a new
paradigm to design effective anti-jamming communication approaches. The main
contributions of this article are:

  * •

A framework of intelligent anti-jamming communication is introduced. Then,
characteristics of jammers and requirements of such intelligent anti-jamming
communication approach are analyzed and outlined.

  * •

A game-theoretic learning anti-jamming paradigm is proposed, and different
anti-jamming games are analyzed.

  * •

Three cases, i.e., Stackelberg anti-jamming game, Markov anti-jamming game,
and hypergraph-based anti-jamming game, are given, and future directions are
illustrated.

##  II Requirements and Summary of Intelligent Anti-jamming Communication

To achieve reliable transmission in the increasingly complex and harsh
electromagnetic environment, the intelligent anti-jamming communication
technology is promising. A compositional architecture of an intelligent anti-
jamming communication system is shown in Fig.1. Similar to a cognitive dynamic
system [14], an intelligent system should have five distinctive
characteristics: perception-action cycle (PAC), memory, attention,
intelligence, and language. The PAC begins with the observation of the radio
environment. Then, legitimate user acts according to the feedback information.
Given a radio environment, memory means storage and updating of acquired
knowledge, which is essential to learn from the jamming environment. Attention
is another mechanism, which determines the priority of available resources
allocation. Different priorities indicate different degrees of importance, and
high priority users need to be satisfied first. Intelligence is the most
important property that based on PAC, attention, and memory. According to a
large amount of feedback information, it promotes the intelligence level of
the system, and then makes intelligent decision. Similar to the role in human
brain, language is an important tool to connect different parts of a single
intelligent system or among multiple intelligent systems.

Inspired by the PAC in cognition cycle, in this article, we focus on the PAC
in anti-jamming communication cycle, which plays a significant role in an
intelligent anti-jamming system. PAC observes and evaluates the jamming
environment, and acts according to the collected perception information.

Figure 1: The compositional architecture of an intelligent anti-jamming
communication system.

###  II-A Discussion of the characteristics of jammers

With the improvement of jamming intelligence level, wireless communication is
facing increasingly complex and harsh electromagnetic environment. According
to the analysis in [4], an intelligent jammer is of sensing ability, learning
ability, and reconfigurability. It should be noted that ordinary jamming
pattern can be regarded as a special case of intelligent jammer.

By sensing, it observes or measures the radio environment, and obtains
historical knowledge. Based on sensing results, the learning ability means
that the jammer mines the communication rule to release more effective jamming
patterns. The reconfigurability represents that the jammer can reconfigure its
jamming pattern, which can make it more flexible. Moreover, with increasing
demands of wireless services, the number of wireless devices increases
rapidly. Accordingly, the spectrum resources become very rare, and the
corresponding inter-user mutual interference becomes serious. The anti-jamming
communication is faced with intense external confrontation and severe internal
conflict. Therefore, there is a complex constitution that inter-user mutual
interference and external malicious jamming will coexist in the new jamming
environment. In addition, the mutual interference problem will become more
prominent in dense wireless networks. In this article, the mutual interference
refers to co-channel interference.

###  II-B The requirements of intelligent anti-jamming approaches

Generally speaking, the development of anti-jamming communication technologies
has gone through three stages: fixed spectrum anti-jamming, adaptive spectrum
anti-jamming, and intelligent anti-jamming. The fixed spectrum anti-jamming
approach means that the spectrum resources do not change according to the
external jamming environment, but depends on the anti-jamming ability of the
technology itself, such as DSSS and FHSS. As a contrast, adaptive anti-jamming
approach is able to sense the external jamming environment and adjust some
communication parameters in quasi-real time according to the jamming
activities, such as adaptive FHSS, and adaptive DSSS. Intelligent anti-jamming
is an inevitable trend of anti-jamming technology. It can recognize and learn
the external jamming environment, and make decisions autonomously. For an
intelligent anti-jamming system, it should meet the following requirements.

Effective resource utilization: The spread-spectrum-based anti-jamming
technologies are common, and widely used in military and civil communication
systems. They improve the anti-jamming performance at the cost of spectrum
resource utilization efficiency, making the contradiction between anti-jamming
performance and spectrum resource scarcity increasingly acute. It is an
important criterion of anti-jamming design to obtain better anti-jamming
performance with minimum resource cost. Thus, it is necessary to develop
optimal resource allocation based anti-jamming schemes to cope with the
complicated electromagnetic environment, as such higher resource utilization
efficiency can be expected.

Dynamic spectrum anti-jamming: To deal with the dynamic characteristic of the
jamming environment, it is necessary to consider the dynamic mechanism and
weakness of jammers. Based on the idea of dynamic spectrum access in cognitive
radio, the dynamic strategy is adopted to resist and avoid jamming attacks.
Different from traditional fixed or preset anti-jamming patterns, it aims to
flexibly utilize spectrum resources, and can adapt to varying spectrum
environment. Moreover, it is more difficult for jamming attacks to master the
communication rule.

Collaborative anti-jamming: To enhance the anti-jamming performance, it is
necessary to deeply explore a multi-level collaborative mode and design a
collaborative anti-jamming paradigm, such as information exchange, local
interaction and jamming utilization. Based on collaboration mechanism, it can
realize the transformation of anti-jamming ideas from independent
countermeasures to collaborative countermeasures. In [11], the information
exchange among users is adopted. In [10], local interaction is employed, and
each user considers the impact on neighbor users. In [6], jamming signals are
regarded as coordination signals to guide anti-jamming spectrum access.

Anti-intelligent jamming: With the high integration of artificial intelligence
technology and communication jamming technology, intelligent jamming attacks
become serious threats that deteriorate information transmission. As jamming
environment becomes more dynamic, and the jamming rule is more difficult to
mine. Anti-intelligent jamming problem becomes important in anti-jamming
field, and will be an inevitable trend of intelligent anti-intelligent jamming
attacks.

###  II-C The capability elements of intelligent anti-jamming approaches

Based on the PAC, an intelligent anti-jamming system should be equipped with
the following abilities.

Self-sensing ability: It is the base for an intelligent anti-jamming system
that aims to autonomously observe the jamming environment and detect jammers’
activities.

Self-decision making ability: It is the core of an intelligent anti-jamming
system. Based on sensing results, it autonomously makes decision and obtains
the desirable anti-jamming strategies.

Self-coordination ability: With the increasing of wireless devices, mutual
interference becomes a serious problem. Thus, it is necessary to coordinate
different users for interference mitigation. Moreover, the multi-level
collaboration mode, i.e., information exchange, local interaction, and relay
enhancement, can be employed to improve the anti-jamming ability.

Self-evaluation ability: To estimate the effectiveness of anti-jamming
approaches, anti-jamming evaluation is indispensable. Through online real-time
evaluation, the effectiveness of anti-jamming approaches can be measured, and
anti-jamming schemes can be adjusted according to the evaluation in real-time
to further improve the anti-jamming performance.

Learning ability: Learning is the critical ability of an intelligent anti-
jamming system that aims to acquire experiences, mines jamming rules, and
accumulates historical knowledge. The higher goal is to predict jamming rules
and discover jamming knowledge according to the historical information.

##  III Game-theoretic learning anti-jamming communications: framework and
challenges

###  III-A Discussion of framework

A framework of GTLAJ is designed and illustrated in Fig. 2, and it mainly
includes two parts: anti-jamming game formulation, and design of anti-jamming
learning algorithms.

Figure 2: Framework of game-theoretic learning anti-jamming communication.

Anti-jamming game formulation: The anti-jamming game model focuses on
theoretical analysis and provides a theoretical framework for analyzing anti-
jamming problem. In the anti-jamming field, many game models can be employed,
such as Stackelberg game, graphical game, and Markov game. A suitable game
model can be selected according to specific anti-jamming scenarios:
Stackelberg game can model the sequential interactions between legitimate
users and jammers; Markov game is a suitable tool to characterize the dynamic
activities of jammers; the graphical/hypergraphical game can accurately model
the mutual interference among legitimate users.

According to whether the jammer is a game player or not, anti-jamming game
models can be divided into two types.

Jammer as a player: Stackelberg game is a suitable tool to analyze and model
the anti-jamming problem when the jammer acts as a player [9]. A Stackelberg
anti-jamming game is an ideal game framework that includes two types of
players with different attributes, namely, jammers and legitimate users. They
are both game players that aim to optimize their own utility functions. The
game can capture the sequential interactions between legitimate users and
jammers. Moreover, it can not only characterize the adversarial relationship
between legitimate users and jammers, but also describe the competitive
behavior among legitimate users.

Jammer as environment: From the perspective of engineering implementation, it
is difficult to obtain accurate jammer information in some scenarios, in which
jammers are often regarded as part of the environment. As an important game
model, Markov game stands out to formulate the anti-jamming problem that can
describe the dynamic characteristics of jammers in [11]. There are also other
game models that can be employed in the anti-jamming problem. For example, in
[15], graphical game can be adopted to characterize the mutual interference
among users. Furthermore, to represent the mutual interference more
accurately, hypergraph game is adopted, which can simultaneously capture both
strong interference relationship and cumulative weak interference relationship
in [10], especially in ultra-dense networks.

In an anti-jamming game model, it first needs to characterize two kinds of
relationship: one is the adversarial relationship between jammers and
legitimate users, and the other is the mutual interference relationship among
users. Second, it is necessary to formulate an anti-jamming game that is
generally non-cooperative. Game models pursue steady solutions, while people
usually want to get the optimal solutions in practical scenarios.
Unfortunately, steady solutions are usually not the optimal ones. Therefore,
the first and core problem is how to make these steady solutions equal or
close to the optimal ones in the anti-jamming game formulation. It should be
noted that the properties of anti-jamming game models are closely related to
the utility function, which determines the performance of game models.

Design of anti-jamming learning algorithms: Learning algorithms are dedicated
to designing effective anti-jamming algorithms, and can converge to a stable
solution (i.e., Nash equilibrium, _NE_) with good performance. To acquire
solutions of an anti-jamming game, learning algorithms need to be designed to
attain desirable anti-jamming strategies through some behavior updating rules
[13]. Some learning algorithms, such as spatial adaptive play, multi-agent
learning, and stochastic learning automata, have been presented in existing
works. It should be noted that proving the validity and existence of game
solutions is important, though direct proof is difficult. The usual way is to
prove the formulated anti-jamming game as a specific type of game, and then
guarantee the existence and effectiveness of the solution according to the
properties of the specific game model. For example, when the formulated game
is an exact potential game, it means that it has at least one pure strategy
NE, and global or local optimal solution of the potential function is a pure
strategy NE. Therefore, the important problem is how to design the potential
function so that the objective function of the problem is linearly related to
the potential function.

###  III-B Discussion of technical challenges

Although GTLAJ has many advantages, there are still some technical challenges.
In this section, we summarize the main challenges from the perspective of
network characteristic.

High dynamic: Due to the time-varying electromagnetic spectrum environment,
being dynamic is its inherent attribute in wireless networks, and the channel
state may change from time to time. Moreover, the dynamic can be greatly
significant considering the dynamic characteristics of jammers, such as
jamming channel, jamming power, and jamming pattern. In addition, the dynamic
characteristics of the wireless network itself should be considered, such as
traffic demands and network topology relationship.

Strong confrontation: Confrontation is the most important feature due to the
adversarial and non-cooperative relationship between legitimate users and
jammers. Legitimate users aim to achieve reliable transmission, while jammers
aim to disrupt such communication. Besides, such confrontation becomes more
intense as the intelligent level of jammers improves.

Ultra density: Various wireless devices are deployed in some wireless
networks, and both mutual interference among users and malicious jamming
should be considered. The malicious jamming is the most immediate threat that
deteriorates the transmission of legitimate users on the one hand, and on the
other hand, the mutual interference is an important factor that restricts the
system performance. This will be increasingly prominent in ultra-dense
wireless networks, such as small cell networks and unmanned aerial vehicles
networks. In addition, it should be noted that there are not only many strong
interference relations, but also many weak interference relations in ultra-
dense networks.

##  IV Case studies

###  IV-A Stackelberg anti-jamming game

Stackelberg game is an extension of the non-cooperative game, and contains two
types of game players, the leaders and the followers. The leaders take actions
first, and the followers play their strategies according to the leaders’
announced strategies. In a Stackelberg anti-jamming game, it has two types of
players, namely, malicious jammers and legitimate users. In this game, both
legitimate users and jammers have their own utility functions, and aim to
maximize their own utility functions. The Stackelberg anti-jamming game has
two advantages in the anti-jamming field. First, it can model and analyze the
common hierarchical behaviors between legitimate users and jammers in an anti-
jamming problem. For example, legitimate users need to detect the jammers’
activities in some scenarios. Moreover, intelligent jammers can learn
legitimate users’ actions. Second, the Stackelberg game can characterize the
competitive interactions at different levels. The competition exists between
leader players and follower players, that is, between legitimate users and
jammers. Besides, there are also mutual competitions among legitimate users at
the same level.

In [9], there are _N_ legitimate users and one jammer in a system, and a
Stackelberg anti-jamming game with one leader and multiple followers is
investigated. Specifically, the jammer is the leader, and the legitimate users
are followers. Then, to obtain the desirable anti-jamming strategies, a
hierarchical learning framework is formulated, and a hierarchical learning
algorithm is designed. For comparison, four methods are considered: the
proposed algorithm, the best NE, the worst NE, and the random selection
method. For the best and the worst NE, the best response algorithm is
employed. 200 independent trials are simulated, and then the best one and the
worst one are respectively taken. As can be known in Fig. 3, the best and the
worst NE are respectively the upper and lower bounds of the proposed
algorithm. Compared with the random selection approach, the proposed algorithm
has a higher expected achievable rate. The reason is that each legitimate user
randomly chooses one channel in each time slot for random selection approach,
and the proposed algorithm can converge to a desirable anti-jamming strategy.
In addition, the expected achievable rate increases with lower jamming power
_J_.

Figure 3: Performance comparison of different algorithms in the Stackelberg
anti-jamming game.

###  IV-B Markov anti-jamming game

Markov game is an extension of the Markov decision problem with many agents.
This game has many advantages. First, the jamming activities are usually
dynamic, and the Markov game can describe the dynamic characteristics of a
jamming environment, which provides a mathematical framework for decision
optimization in dynamic scenarios. Second, Markov game can represent the
relationship among players, and describe collaboration and competition among
players according to their utility functions. Moreover, the multi-agent
reinforcement learning approach is a suitable tool to obtain desirable anti-
jamming strategies through "action-feedback-action" approach, in which each
legitimate user acts as an agent.

In [11], a Markov anti-jamming channel selection game is formulated. Then, a
collaborative multi-agent reinforcement learning anti-jamming algorithm is
proposed. The coordination among users is considered through information
exchange (e.g., Q table), which can be realized by a common control channel.
Based on collaborative learning, it can simultaneously tackle the jamming and
mutual interference among legitimate users. Fig. 4 shows the performance
comparison of the average rate in sweep jamming and comb jamming scenarios.
Compared with the sensing-based method, the independent _Q_ -learning and the
random selection method, the proposed algorithm has the highest average rate
in the sweep jamming scenario. The reason is that each legitimate user
independently employs a standard _Q_ -learning for the independent _Q_
-learning method, without considering the coordination; for the sensing-based
method, it makes the channel selection depends only on sensing results; in
random selection scheme, each legitimate user randomly chooses a channel, and
it lacks of learning ability and collaboration mechanism. Moreover, the
proposed algorithm also has good performance in the comb jamming scenario.

Figure 4: Performance comparison of different algorithms in the Markov anti-
jamming game.

###  IV-C Hypergraph-based anti-jamming game

In ultra-dense wireless networks, anti-jamming defense is more challenging,
and mutual interference becomes very prominent. First, the strong interference
relationship between two neighboring legitimate users needs to be considered.
Second, there is also some accumulative weak interference relationship that
may be equivalent to a strong one when the accumulative weak interference from
multiple users reaches a certain threshold. However, only the strong
interference relationship is considered in traditional graph model. Similar to
[10], the strong interference relationship is represented by lines, while the
weak accumulative interference relationship is denoted by circles. The lines
and circles together form hyperedges. A strong interference relationship
occurs between two neighboring legitimate users when they choose the same
channel, while a weak accumulative interference relationship is generated
among three or more legitimate users. Specifically, only when three or more
legitimate users employ the same channel simultaneously, a weak accumulative
interference relationship can be generated. Otherwise, there won’t be an
interference relationship in an accumulative interference relationship
hyperedge. For a hypergraph-based anti-jamming game, it has two advantages.
First, it can accurately describe the interference relationship among
legitimate users, considering both the strong and accumulative weak
interference relationships. Second, the anti-jamming problem can be formulated
as a hypergraph game, and transformed into a generalized interference and
jamming minimization problem.

In [10], a system with _N_ users and _K_ jammers is considered, and the number
of available channels is _M_. Then, a hypergraph-based anti-jamming game is
formulated, and an anti-jamming channel selection algorithm is proposed to
achieve the desirable anti-jamming strategies. In the proposed algorithm, each
user updates its strategies according to the learning rule based on stochastic
learning automata, and a common control channel is needed to realize the
information exchange. Fig 5 shows the performance comparison of the expected
normalized network capacity. To evaluate the performance of the proposed
algorithm, the performance of the graph-based method, and the random selection
method are presented for comparison. It shows that the performance of the
proposed hypergraph-based algorithm is the best. Besides, the performance
improvement is more significant with the increases of hyperedges for the weak
accumulative interference (HWI) relationship when the hyperedges of strong
interference relationship is fixed. The reason is that the graph-based method
ignores the weak accumulative interference relationship, and the random
selection scheme is an instinctive method. In addition, growing channels
results in the increase of expected normalized network capacity. The reason is
that the more the channels, the more available resources for the users,
leading to less mutual interference among users. It is also noted that the
expected normalized network capacity decrease with lower active probability
_p_. The reason is that active probability means a user competes for channels
with certain probability, that is to say, a user is active with probability
_p_ , and the lower active probability brings less user communication.

Figure 5: Performance comparison of different algorithms in the hypergraph-
based anti-jamming game.

##  V Future research directions

Although the GTLAJ approaches attained much attention in the past years, there
are still many problems to be discussed.

  * •

Jamming prediction: Although jammers’ activities are dynamic, the jamming
spectrum information may have some correlations at different time slots.
According to the observed historical information of jammers, useful jamming
knowledge and jamming rules can be mined. Then, based on the machine learning
method (i.e., long-short-term-memory), the law of jammers’ actions can be
learned so that jamming behaviors can be avoided in advance. Thus, the
improvement from passive anti-jamming to active anti-jamming can be realized.

  * •

Integrated design of jamming attack and anti-jamming: It is necessary to
design the integration of attack and defense. On the one hand, it is essential
to ensure communication reliability, and on the other hand, the offensive
capability is also required. By utilizing offense capability to suppress
opponent transmissions, one user may gain advantages in information
transmission by exerting jamming on opponent users. If the opponent user is
attacked, it releases the jammed channel due to the friendly jammer, which may
be helpful to strive for more channel resources for the users.

  * •

Knowledge transfer: In an unknown jamming environment, learning technology is
important to obtain desirable anti-jamming strategies through continuous
trial-and-error interactions. To speed up the convergence speed of learning
algorithm, knowledge transfer (i.e., hotbooting technique) is important. When
encountering unknown jamming environment, the similarity analysis can be
carried out first, and the acquired experiences can be transferred to the new
scene. Therefore, it realizes the accumulation of jamming experiences and
improves the speed of learning.

  * •

Jamming location: Jamming location is helpful to obtain accurate jamming
information, and facilitates the anti-jamming method design. To obtain
effective anti-jamming strategies, it is necessary to estimate jamming
parameters, such as jamming power, and jamming channel gain. The inaccurate
estimation is a main challenge in jammer localization due to the confrontation
and non-cooperative relationships between legitimate users and jammers. In the
process of designing anti-jamming approaches, jamming localization-assisted
anti-jamming methods may effectively improve the anti-jamming performance.

##  VI Conclusions

In this article, the game-theoretic learning anti-jamming (GTLAJ), a new
paradigm for intelligent anti-jamming communication, was investigated. First,
jammers’ characteristics, requirements and capability elements of intelligent
anti-jamming approaches were presented. Then, the framework and challenges for
GTLAJ were introduced, and different anti-jamming game models were analyzed.
Finally, three cases were studied, and future research directions were
discussed.

##  VII Acknowledgements

This work was supported in part by the National Science Foundation of China
under Grant 61901523.

## References

  * [1] Y. E. Sagduyu, R. A. Berry, and A. Ephremides, “Jamming games in wireless networks with incomplete information," _IEEE Commun. Mag._ , vol. 49, no. 8, pp. 112-118, Aug. 2011.
  * [2] K. Pelechrinis, M. Iliofotou, and S. V. Krishnamurthy, “Denial of service attacks in wireless networks: the case of jammers," _IEEE Commun. Surveys Tutorials_ , vol. 13, no. 2, pp. 245-257, Second Quarter, 2011.
  * [3] K. Grover, A. Lim, and Q. Yang, “Jamming and anti-jamming techniques in wireless networks: a survey," _Int. J. Ad Hoc and Ubiquitous Comput._ , vol. 17, no. 4, pp. 197-215, Dec. 2014.
  * [4] M. Aref, S. Jayaweera, _et al._ , “Survey on cognitive anti-jamming communications," _IET Commun._ , vol. 14, no. 18, pp. 3110-3127, Nov. 2020.
  * [5] X. Wang, J. Wang, _et al._ ,“Dynamic spectrum anti-jamming communications: challenges and opportunities," _IEEE Commun. Mag._ , vol. 58, no. 2, pp. 79-85, Feb. 2020.
  * [6] Y. Xu, Y. Xu, X. Dong, _et al._ , “Convert harm into benefit: a coordination-learning based dynamic spectrum anti-jamming approach," _IEEE Trans. Veh. Technol._ , vol. 69, no. 11, pp. 13018-13032, Aug. 2020.
  * [7] Y. Gao, Y. Xiao, M. Wu, _et al._ , “Game theory-based anti-jamming strategies for frequency hopping wireless communications," _IEEE Trans. Wireless Commun._ , vol. 17, no. 8, pp. 5314-5326, Aug. 2018.
  * [8] L. Xiao, T. Chen, J. Liu, and H. Dai, “Anti-jamming transmission stackelberg game with observation errors," _IEEE Commun. Lett._ , vol. 19, no. 6, pp. 949-952, Jun. 2015.
  * [9] L. Jia, Y. Xu, _et al._ ,“Stackelberg game approaches for anti-jamming defence in wireless networks," _IEEE Wireless Commun. Mag._ , vol. 25, no. 6, pp. 120-128, Dec. 2018.
  * [10] L. Jia, Y. Xu, Y. Sun, _et al._ ,“A game-theoretical learning approach for anti-jamming dynamic spectrum access in dense wireless networks," _IEEE Trans. Veh. Technol._ , vol. 68, no. 2, pp. 1646-1656, Feb. 2019.
  * [11] F. Yao, and L. Jia,“A collaborative multi-agent reinforcement learning anti-jamming algorithm in wireless networks," _IEEE Commun. Lett._ , vol. 20, no. 10, pp. 1991-1994, Oct. 2016.
  * [12] N. Qi, W. Wang, M. Xiao, _et al._ , “A learning-based spectrum access Stackelberg game: friendly jammer-assisted communication confrontation," _IEEE Trans. Veh. Technol._ , vol. 70, no. 1, pp. 700-713, Jan. 2021.
  * [13] Y. Xu, J. Wang, Q. Wu, _et al._ , “A game-theoretic perspective on self-organizing optimization for cognitive small cells," _IEEE Commun. Mag._ , vol. 53, no. 7, pp. 100-108, Jul. 2015.
  * [14] S. Feng, P. Setoodeh, and S. Haykin, “Smart home: cognitive interactive people-centric internet of things," _IEEE Commun. Mag._ , vol. 55, no. 2, pp. 34-39, Feb. 2017.
  * [15] Y. Xu, Q. Wu, L Shen, _et al._ , “Opportunistic spectrum access with spatial reuse: graphical game and uncoupled learning solutons," _IEEE Trans. Wireless Commun._ , vol. 12, no. 10, pp. 4814-4826, Oct. 2013.

_Luliang Jia_<EMAIL_ADDRESS>received his M.S. and Ph.D. degree
from Army Engineering University of PLA, Nanjing, China, in 2014 and 2018,
respectively. Currently, he is an Assistant Professor at Space Engineering
University, Beijing, China. His research interests include game theory,
learning theory, and communication anti-jamming technology.

_Nan Qi_<EMAIL_ADDRESS>received the B.Sc. and Ph.D. degrees from
Northwestern Polytechnical University (NPU), China, in 2011 and 2017,
respectively. She is currently an assistant professor in Nanjing University of
Aeronautics and Astronautics, China. Her research interests include anti-
jamming, UAV communications, and relaying networks.

_Feihuang Chu_<EMAIL_ADDRESS>received his M.S. and Ph.D. degree
from the Electronic Engineering Institute of PLA, China, in 1997 and 2003,
respectively. He is currently working with Space Engineering University,
Beijing, China. His research interests include learning theory, and jamming
and anti-jamming technology.

_Shengliang Fang_<EMAIL_ADDRESS>received his M.S. and Ph.D.
degree from Electronic Engineering Institute of PLA, China, in 1998 and 2004,
respectively. He is currently working with Space Engineering University,
Beijing, China. His research interests include learning theory, and anti-
jamming technology.

_Ximing Wang_<EMAIL_ADDRESS>received his Ph.D. degree from Army
Engineering University of PLA, China, in 2020. He is currently working with
the National University of Defense Technology. His current research interests
include anti-jamming, and machine learning.

_Shuli Ma_<EMAIL_ADDRESS>received his Ph.D. degree from Beijing
Institute of Technology in 2020. She is currently the lecture in Space
Engineering University. Her main research interests include image
reconstruction, and anti-jamming.

_Shuo Feng_<EMAIL_ADDRESS>received his Ph.D. degree from McMaster
University, Canada, in 2019. He is currently a Research Scientist with the
Nanjing University of Aeronautics and Astronautics. His research interests
include cognitive dynamic systems, and machine learning.