# Survey: Image Mixing and Deleting for Data Augmentation

Humza Naveed<EMAIL_ADDRESS>
###### Abstract

Data augmentation has been widely used to improve deep neural networks
performance. Numerous approaches are suggested, for example, dropout,
regularization, and image augmentation, to avoid over-fitting and enhancing
the generalization of neural networks. One of the sub-area within data
augmentation is image mixing and deleting. This specific type of augmentation
either mixes two images or deletes image regions to hide or make particular
characteristics of images confusing for the network to force it to emphasize
on the overall structure of the object in an image. The model trained with
this approach proves to perform and generalize well as compared to one trained
without image mixing or deleting. An additional benefit achieved with this
method of training is robustness against image corruption. Due to its low
compute cost and success in the recent past, many techniques of image mixing
and deleting are proposed. This paper provides a detailed review of these
devised approaches, dividing augmentation strategies into three main
categories cut and delete, cut and mix, and mixup. The second part of the
paper empirically evaluates these approaches for image classification, fine-
grained image recognition, and object detection where it is shown that this
category of data augmentation improves the overall performance for deep neural
networks.

###### keywords:

Image Augmentation , Regularization , Overfitting , Data Mixing

##  1 Introduction

Deep neural networks have shown tremendous success in image classification
[59, 35], object detection [58, 57], and semantic segmentation [8, 86]. These
networks are data hungry with millions of parameters that make them prone to
overfitting [42, 81, 55]. To enhance model generalization, many approaches
have been suggested, for example, regularization [41], dropout [67], and data
augmentations [63] including image rotation, random cropping, jittering and
etc. In addition to these techniques, a relatively new data augmentation
strategy that mixes and deletes different image regions has recently gained
popularity. The augmented images generated by this approach are present in the
vicinity of original data distribution. The model trained with this new data
generalizes well against data variations, avoids overfitting, and achieves
robustness against data corruptions. Overall, the model learns to focus on the
complete structure of the object under consideration resulting in improved
generalization.
Mainly, image mixing and deleting are divided into three categories, cut and
delete, cut and mix, and mixup. The taxonomy of image augmentation can be seen
in Figure 1. Cut and delete [15, 65, 9, 87] erases the selected region, cut
and mix [82, 76, 71, 56, 74, 28] replaces the selected region with some region
of other image, whereas mixup [83, 75, 37, 31, 30] mixes the pixel intensity
values of two images. The first two categories are region-level augmentation,
whereas the third category is image-level augmentation. Within these
categories, many approaches rely completely on random data augmentation
strategies, whereas others consider salient information either keeping it or
deleting it in the generated image.

Figure 1: The categorical division of image mixing and deleting data
augmentation

Although image mixing and deleting improves the overall performance by adding
noise and ambiguity in training data and labels, it is also likely to reduce
the performance if performed naively. Therefore, carrying out image mixing
carefully is important as it should not reduce or delete the information in an
image beyond a certain limit. Because in this scenario, the model does not
have enough information to learn and distinguish among classes, and the label
of the image becomes completely irrelevant and noisy for the model. To avoid
this situation, many approaches are suggested in the literature that
emphasizes keeping salient information in the augmented image.
The generalization power of image mixing or deleting is not limited to image
classification only. It has been proven to increase accuracy for fine-grained
image recognition and object detection tasks as well. The augmentation
approaches suggested for image classification are transferrable to object
detection without any modifications and shown to improve accuracy over the
baseline models. Whereas, for fine-grained image recognition, particular
augmentation strategies by image mixing are proposed. Apart from this, image
mixing or deleting can easily be used along with regularization [41], dropout
[67] or conventional image augmentations [63] based on affine transformations
to further enhance the performance.
The rest of the paper contains the review of the various image augmentation
techniques in section 2, performance comparison in section 5 followed by
conclusion in section 6. The review is focused entirely on the proposed hand-
crafted image mixing and deletion data augmentation techniques. It does not
discuss any of the approaches that perform neural augmentation search [13,
47].

##  2 Image Mixing and Deleting

We divide image mixing and deleting-based augmentation schemes into three main
categories. This includes 1) erasing image patches, 2) cut image region and
replacing with some other image region, and 3) mixing multiple images. All the
proposed approaches within these categories are discussed in detail in this
section.

###  2.1 Cut and Delete

This section discusses data augmentation by deleting image patches randomly or
semantically. The purpose of erasing image patches is to allow the network to
learn in case of occlusions. This way model is forced to understand and focus
on the partially visible image or object properties. This kind of dropout is
different from conventional dropout because it drops contiguous image regions,
whereas values in traditional dropout work at non-contiguous locations. Here,
much of the image region is provided to the network to build connections
semantically among various parts of the image. Below, we discuss proposed
approaches falling in this category.

Figure 2: Comparison of different cut and delete approaches, cutout first
column, random erasing second column, hide and seek third column and gridmask
last column, courtsey [9]

####  2.1.1 Cutout

Cutout [15] removes constant size square patches randomly by replacing them
with any constant value. The selection of region is performed by selecting a
pixel value randomly and placing a square around it. In this procedure, cutout
region lying outside the borders is allowed and proven to improve performance.
Patch deletion is applied on all the training images where optimal patch size
is selected by performing grid search on training data. An example is shown in
figure 2.

####  2.1.2 Random Erasing

Random Erasing [87] deletes contiguous rectangular image regions similar to
cutout [15] with minor differences in region selection procedure. Opposite to
cutout, where deletion is applied on all the images, here it is performed with
a probability of either applying it or not. In every iteration, region size is
defined randomly with upper and lower limits on region area and aspect ratio.
The algorithm also takes care of the region to be inside the image boundaries.
Additional to this, random erasing provides region-aware deletion for object
detection and person identification tasks. Regions inside the object bounding
boxes are randomly erased to generate occlusions. Figure 2 shows augmented
images.

####  2.1.3 Hide and Seek

Hide and seek [65] divides an image into a specified number of grids and turns
on and off each grid with an assigned probability. This way various image
regions are deleted, and they can be connected or disconnected from each
other, as shown in figure 2. Values in turned-off regions are replaced with
the average of all the pixel values in the entire dataset.

####  2.1.4 GridMask

Another simple approach is GridMask [9]. The algorithm tries to overcome
drawbacks of Cutout, Random Erasing, and Hide and Seek that are prone to
deleting important information entirely or leaving it untouched without making
it harder for the algorithm to learn. To handle this, GridMask creates
multiple blacked-out regions in evenly spaced grids to maintain a good balance
between deletion and retention of critical information. The number of masking
grids and their sizes are tuneable. The paper provides a comparison between
erasing a single region vs multiple regions, showing GridMask is less likely
to hide out foreground information completely. Even with the large box size,
it retains a sufficient amount of important information for the model to
learn. Gridmask generated examples are shown in figure 2.

Figure 3: The figure shows attentive cutmix image augmentation procedure,
attentive regions from one image are overlayed on other image, image is taken
from [76]

####  2.1.5 Cut and Delete for Object Detection

This section describes cut and delete specifically designed for object
detection.

#####  2.1.5.1 Adversarial Spatial Dropout for Occlusion

This technique [78] drops region pixels to generate hard positives for object
detection by learning key image regions. Within the proposed region only 1/3
pixels are dropped after sorting based on magnitudes. Dropped values are non-
contiguous here as compared to previously discussed methods. Salient image
regions are learned using a separate network. The mask generator is penalized
based on the performance of the detector similar to generative adversarial
networks [21].

###  2.2 Cut and Mix

This section discusses data augmentation strategies where instead of deleting
a patch, the patch is replaced with some other image region. By this approach,
an image shares multiple class labels, whereas the major class label belongs
to the original class label. Hence, the model learns to differentiate between
two classes within a single image.

####  2.2.1 CutMix

In CutMix [82] images are augmented by sampling patch coordinates, $x,y,h,w$
from a uniform distribution. The selected patch is replaced at the
corresponding location with a patch from the other randomly picked image from
the current mini-batch during training. CutMix updates image labels according
to replaced patch proportion as defined in equation 1, where $M$ is the image
mask, $x_{a}$ and $x_{b}$ are images, $\lambda$ is the proportion of label,
and $y_{a}$ and $y_{b}$ are the labels of images.

$\displaystyle x_{new}=M.x_{a}+(1-M).x_{b}$ (1) $\displaystyle
y_{new}=\lambda.y_{a}+(1-\lambda).y_{b}$

####  2.2.2 Attentive CutMix

Attentive CutMix [76] builds up on CutMix. Instead of random pasting, it
identifies attentive patches for cutout and pastes them at the same location
in the other image. This avoids the problem of selecting a background region
not important for the network and updating the label information accordingly,
as in equation 1. A separate pre-trained network is employed to extract
attentive regions. The attention output is mapped back onto the original
image. Image is divided into a grid of patches where highly activated six
responses are pasted onto the training image. These image pairs are selected
randomly in every training iteration. This procedure is shown in figure 3.

####  2.2.3 Attribute Mix

Attribute Mix [44] augments images based on semantically extracted image
attributes. It trains an attribute classifier by extracting $k$ attributes
(e.g., leg, head, and wings of a bird) from each image. The attribute mining
procedure for every image is performed repetitively $k$ times, whereas for
each iteration, an attribute is masked out from the original image based on
the most discriminative region in the attention map. With these images, an
attribute-level classifier is trained to generate new images for the actual
classification model. For the given two images, the attribute-level classifier
identifies attributes masks, these masks are randomly picked to create a new
training image. The procedure of label update is the same as in equation 1.

####  2.2.4 RICAP

Random image cropping and patching (RICAP) [71] performs data augmentation by
cropping four regions from randomly sampled images and augmenting them to
create a new image. The generated image has mixed labels proportional to the
pasted area. The area of cropped regions in the output image is determined by
sampling through uniform distribution. The paper suggests three methods for
common boundary points in the augmented image: anywhere-RICAP (origin can be
anywhere), center-RICAP (origin can only be in the middle of the image), and
corner-RICAP (origin can only be in corners). Corner-RICAP has shown the best
performance because a larger region of one image is visible to the network to
learn. An example of RICAP strategy is shown in figure 4.

Figure 4: An example of RICAP image augmentation procedure, courtesy [71]

####  2.2.5 Mixed Example

This method [68] experimented with 14 different types of augmentation
approaches. The output image is generated using the following techniques:
vertical concatenation, horizontal concatenation, mixed concatenation, random
2x2, VH-mixup (vertical concatenation, horizontal concatenation, and mixup),
VH-BC+ (vertical concatenation, horizontal concatenation, and between-class),
random square, random column interval, random row interval, random rows,
random columns, random pixels, random elements, and noisy mixup. From all
these approached, VHmixup has the best performance.

####  2.2.6 CowMask

CowMask [19] is used in semi-supervised learning where original and augmented
images are brought closer during training. CowMask suggests two types of
mixing approaches 1) erasing and 2) mixing two images similar to cutmix. The
mask here is of irregular shape rather than rectangular and generated by
masking or keeping a proportion of image pixels through thresholding. Gaussian
filter is applied to remove noise before thresholding. Pixel values below the
threshold are either erased or replaced by the pixel values of the randomly
selected image at the corresponding locations.

####  2.2.7 ResizeMix

ResizeMix [56] performs random image cropping and pasting. It empirically
proves to perform better than salient and other non-salient image mixing
methods, as shown in table 1. ResizeMix solves the random region cropping
problem that misallocates the output image label in certain cases where the
pasted region does not contain any object information. The labels of the
output image are updated as per the equation 1. ResizeMix handles this by
scaling down (scale rate is sampled from a uniform distribution) the selected
image completely and pasting it randomly on the target image, as shown in
Figure 5. This way, modified labels of the output image are always accurate
and proportionate to the mixing.

Figure 5: An example of augmented image generated using resizemix, figure
taken from [56]

####  2.2.8 SaliencyMix

Instead of random cut and mix, SaliencyMix [74] extracts salient regions and
pastes them on the corresponding location in the target image. The salient
region is extracted around the maximum intensity pixel location in the
saliency map. Augmented image labels are updated as per the equation 1.

####  2.2.9 Intra-Class Part Swapping

Intra-Class Part Swapping [84] replaces most attentive regions of one image by
the other. Attentive regions are extracted using a classification activation
map (CAM), thresholded for the most prominent region. The attentive region in
the source image is scaled and translated according to the attentive region of
the target image for region replacement. The label information of the output
is similar to the target image as this approach relies on augmenting similar
class images.

####  2.2.10 SnapMix

SnapMix [28] augments training images by extracting and merging random image
regions of different sizes, where the region size is drawn through the beta
distribution for both the images. Generated image label is assigned based on
semantic composition from normalized (sum to one) CAMs. The procedure of label
generation is similar to equation 1. However, the summation of label
coefficients can exceed beyond one depending on the semantic composition of
the output image.

####  2.2.11 KeepAugment

Similar to SaliencyMix and attentive cutmix, KeepAugment [20] performs
augmentation based on salient region. However, this approach does not use two
images for the augmentation. KeepAugment identifies the salient area in an
image and assures the image generated by the augmentation strategies, for
example, Cutout, RandAugment [14], CutMix [82] or AutoAugment [13], contains
salient region in it.

Figure 6: Augmix [27] data augmentation procedure is shown in image, multiple
transformations are applied randomly on image in separate branches that are
later combined to generate one image, example is from [27]

####  2.2.12 Cut and Mix for Object Detection

This section discusses cut and mix approaches specifically designed for object
detection.

#####  2.2.12.1 Visual Context Augmentation

This augmentation strategy [16] learns to place object instances at an image
location depending on the surrounding context. A neural network is trained for
this purpose. The training data is prepared to generate a context image with
the masked-out object inside it. From an image, 200 context sub-images are
generated surrounding the blacked-out bounding box. The neural network learns
to predict the category (object or background) in masked pixels. During
testing, the context network performs the identification of plausible object
bounding boxes. The object instances are placed inside the selected boxes to
generate a new training image. These newly created images are used as training
data for the other network, for example, object detectors.

#####  2.2.12.2 Cut, Paste and Learn

This simple approach [17] generates new data by extracting object instances
and pasting them on randomly selected background images. Instances are blended
with various blending approaches, for example, gaussian blurring and poison
blending, to reduce pixel artifacts around the augmented object boundaries.
Added instances are also rotated, occluded, and truncated to make the learning
algorithm robust. This simple data mixing approach improves object detection
performance.

###  2.3 MixUp

This type of data mixing is different from Cutout and CutMix such that it
mixes pixel values of two images instead of cropping or erasing image regions.
MixUp regularizes neural networks by forcing them to learn linear
interpolations between training images. The network trained with MixUp has
shown robustness to adversarial attacks with improved performance on the test
set.

Figure 7: An example of network training using smoothmix strategy, courtsey
[37]

####  2.3.1 MixUp

MixUp [83] generates a new image by mixing pixel values of two randomly
selected images. The proportion of pixel strength during data mixing is
decided by the mixing factor $\lambda$ $\in$ [0-1] as shown in equation 2. The
values for $\lambda$ are drawn from the beta distribution.

$\displaystyle x_{new}=\lambda.x_{a}+(1-\lambda).x_{b}$ (2) $\displaystyle
y_{new}=\lambda.y_{a}+(1-\lambda).y_{b}$

####  2.3.2 Manifold MixUp

Opposite to mixup, manifold mixup [75] mixes feature values generated from
intermediate neural network layers. For two random images, inputs are
feedforward up to k layer of the network where the output feature maps are
mixed based on equation 2. The mixed feature maps are given input to the next
layer and forward propagated up to the last layer. After the forward
propagation, backward propagation is performed in the standard way with
updated labels (similar to mixup).

####  2.3.3 AugMix

Augmix [27] performs data mixing using the input image itself. It transforms
(translate, shear, rotate and etc) the input image and mixes it with the
original image. Image transformation involves series of randomly selected
augmentation operations applied with three parallel augmentation chains. Each
chain has a composition of operations that could involve applying, for
example, translation on input image followed by shear and so on. The output of
these three chains is three images mixed to form a new image. An example is
shown in figure 6. This new image is later mixed with the original image to
generate the final augmented output image, as per equation 2. The
transformations in AugMix are selected from AutoAugment [13].

Figure 8: Image augmentation performed by co-mixup is shown in figure,
courtesy [31]

####  2.3.4 SmoothMix

SmoothMix [37] is a mask-based approach, matching closely with the Cutout and
CutMix techniques. However, it has few differences: (1) the mask has soft
edges with gradually decreasing intensity, and (2) the mixing strategy is the
same as in equation 2. The augmented image has mixed pixel values depending on
the strength of the mask, as shown in figure 7. The paper has suggested
various square and circular masks containing smooth edges. Frome these
circular masks are the best performing, generated using Gaussian distribution.
The value of $\lambda$ is decided by the following equation 3:

$\lambda=\frac{\sum_{i=1}^{W}\sum_{j=1}^{H}G_{ij}}{WH}$ (3)

where $G_{ij}$ is the pixel value of mask $G$, $H$ is height, and $W$ is
width. With the given mask, the equation to mix two images becomes:

$\displaystyle x_{new}=G.x_{a}+(1-G).x_{b}$ (4) $\displaystyle
y_{new}=\lambda.y_{a}+(1-\lambda).y_{b}$

####  2.3.5 Co-Mixup

Co-Mixup [31] performs salient image mixing on a batch of input images to
generate a batch of augmented images. This technique maximizes saliency in
output images by penalizations to ensure local data smoothness and diverse
image regions. An example of generated images by Co-Mixup is shown in figure
8.

####  2.3.6 Sample Pairing

This simple technique [30] merges two images by averaging their pixel
intensities. The new image has the same training image label opposite to MixUp
and other approaches where labels are updated according to the proportion of
image mixing.

####  2.3.7 Puzzle Mix

Puzzlemix [32] learns to augment two images optimally based on saliency. The
algorithm employs the following procedure 1) Images are divided into regions
for the mixup, 2) The algorithm learns to transport the salient region of one
image such that the output image has the maximized saliency from both images.
An example of a transported salient region in an augmented image is shown in
figure 9.

Figure 9: Example of puzzlemix, depicting how maximum saliency in output image
is attained with region transport, courtesy [32]

####  2.3.8 ReMix

Remix [12] addresses the issue of class imbalance by generating mixed images
for minority classes. The process of remix is identical to equation 2 for data
mixing, but in the case of label assignment, it sets the label of the output
image to the minority class.

####  2.3.9 Mixup for Object Detection

Similar to Cutout and CutMix, MixUp is also used to enhance the performance of
object detection algorithms. One of the approaches in [6] MixUp image data to
update object labels and locations. Instead of employing the fundamental MixUp
strategy, this algorithm identifies object locations, mixes object pixels at
corresponding locations in two images, and updates the object box-coordinates
in output image depending on the prominent object and the image label.

##  3 Why Image Mixing and Deleting performs well?

Understanding the underlying cause of why these techniques improve the
performance of baselines is important. This builds the foundation to improve
the existing approaches. The various methods are adopted to draw analysis
between standard training procedures and training with augmentations. Cutout
[15] and CutMix [82] argues that hindering image regions enforces the
classifier to learn from the partially visible objects and understand the
overall structure. CutMix verifies this argument by showing enhanced focus
towards the target class in CAMs, shown in figure 10. Opposite to this, MixUp
has shown to improve classifier’s calibration and reduced prediction
uncertainity in [73]. To prove this, authors draw a comparison between mean of
predictions vs accuracy where the confidence distribution for MixUp trained
model is evenly distributed against the standard model whose distribution is
towards higher conficence i.e. over-confidence. Similarly, the loss contours
obtained for a network trained with MixUp are smooth as compared to sharp
contours in standarad training [46]. This proves [73, 46] the better
generalization and robustness of MixUp against adversarial attacks.

##  4 Applications

The benefits of image mixing and deleting are not limited to only image
classification or object detection, for which they were designed initially.
These algorithms, when used in conjunction, have been shown to enhance the
performance of other systems. A few examples are semi-supervised learning,
unsupervised learning, adversarial training, privacy mixing, etc. The
subsequent sub-sections provide a brief overview, how the incorporation of
image mixing and deleting with minor tweaks have improved the performance in
other applications.

Figure 10: Class activation maps (CAMs) for the augmented images, image is
taken from [82]

###  4.1 Semi-Supervised

Semi-supervised learning (SSl) leverages unlabelled data to train strong
classifiers. One of the semisupervised learning approaches, named as MixMatch
[2] suggested weighted MixUp, given in equation 5, to ensure augmented image
is closer to the original image. This approach is successfully employed in
other SSL techniques [1, 45, 77]. Another SSL approach [66] make use of Cutout
in association with RandAugment [14]14 naming it control theory augment
(CTAugment).

$\begin{gathered}\lambda\sim\beta(\alpha,\alpha)\\\
\acute{\lambda}=max(\lambda,1-\lambda)\\\
x_{new}=\acute{\lambda}.x_{a}+(1-\acute{\lambda}).x_{b}\\\
y_{new}=\acute{\lambda}.y_{a}+(1-\acute{\lambda}).y_{b}\end{gathered}$ (5)

###  4.2 Unsupervised Learning

Unsupervised learning aims at learning better data representations without
requiring human annotations. Recent success in self-supervised and contrastive
learning [11, 10, 22], that learns to bring positive samples closer and push
away negative samples, has attracted researchers to investigate image mixing
and deleting for better representation learning in the unsupervised domain.
The literature shows unsupervised algorithms using CutMix in [43] and MixUp in
[38] improve the vanilla model performances simply by additional image mixing
data augmentation. Another approach Unmix [61]61], employs MixUp for
prediction smoothness in an unsupervised setting. Instead of keeping the
distance of positive pairs equal to zero, unmix generates a mini-batch of
mixtures of positive pair augmented images. The distance between pairs is
lowered and equaled to the mixing factor either $\lambda$ or $1-\lambda$ based
on the mixing order in a mini-batch.
Unsupervised domain adaptation (UDA), a sub-area of unsupervised learning
utilizes MixUp to introduce linear prediction behavior across domains. For
efficient domain adaptation, from the source domain (has data labeled) to
target domain (has data unlabelled), approaches
citefixbi,virtualmix,domainadaptation mixes images from both the domains. For
UDA, images and labels $x_{a}$ and $y_{a}$ are collected from the source
domain, $x_{b}$ and $y_{b}$ (pseudo label) are taken from the target domain.

###  4.3 Adversarial Training

Adversarial training is about training the neural networks for robustness
against adversarial examples. However, to achieve adversarial robustness
neural networks performance becomes worse for standard classification. To
reduce the impact of this phenomenon, approaches are proposed to incorporate
adversarial training with MixUp. Adversarial vertex mixup [40] trains the
network with the MixUp of clean image x and adversarial vertex image $x_{av}$
generated by the equation 6, where perturbations $\delta_{x}$ are generated by
Projected Gradient Descent (PGD) [51]. Similarly, interpolated adversarial
training [36], performs the MixUp of clean and perturbed images generated by
PGD for enhanced robustness.

$x_{av}=x+\gamma\delta_{x}$ (6)

####  4.3.1 Backdoor and Targeted Attacks

Backdoor attacks poison data by inserting a trigger patch (generally a small
noise patch), whereas targeted attacks modify the image such that the network
labels the input image as the targeted label. These attacks lose their
effectiveness when the model is trained with the CutMix and MixUp data
augmentations as in [4, 5].

###  4.4 Privacy Preserving

Training on edge devices produces less accurate models due to data scarcity.
Additional to this, the edge devices have insufficient computing resources
both for training and inference. To overcome these problems, private data has
to be transferred to the cloud. The cloud is prone to be malicious where the
attacker can exploit the user’s private data. This inspired the researchers to
devise solutions to preserve data privacy. Within the proposed approaches, few
employ MixUp to achieve data privacy [29, 50, 62]. InstaHide [29] mixes
private images with public data k times that are later passed through random
pixel sign flipping. DataMix [50] is a privacy preserving inference algorithm.
This algorithm mixes two private images multiple times and transmits the data
to the cloud for inference. The predictions are transferred back to the edge
device that performs demix operation and generates true prediction
probabilities. The mixing and demixing coefficients are only known to the edge
devices. XOR MixUp [62] employs iterative MixUp of private and dummy images
followed by XOR for data encoding. The data is decoded on the server-side
using dummy data present on the server by performing another XOR operation.
The model is trained using decoded data whose parameters are shared with all
the devices after model convergence.

###  4.5 Text Classification

The applications of MixUp are also found in text classification. Due to the
discrete nature of the text, mixing is not performed on the input, but words
or sentence embeddings [7, 85, 24]. The performance of transformers for text
classification improves significantly with MixUp on transformers generated
embeddings [69]. Another approach [23] suggests non-linear mixup on embeddings
space of words, where every dimension of word embedding (multi-dimensional
embedding) has a different mixing parameter rather than the global value as in
equation 2.

| Code | Method |  | Cifar-10
---
Accuracy (%)
| Cifar-10
---
(Best Model)
| Cifar-100
---
Accuracy (%)
| Cifar-100
---
(Best Model)
| ImageNet
---
Accuracy (%)
| ImageNet
---
(Best Model)
Cutout | y | Random | 97.04 | WRN-28-10 | 81.59 | WRN-28-10 | 77.1 | R-50
| Random
---
Erasing
y | Random | 96.92 | WRN-28-10 | 82.27 | WRN-28-10 | - | -
| Hide and
---
Seek
y | Random | 96.94 | WRN-28-10 | 78.13 | R-110 | 77.20 | R-50
GridMask | y | Random | 97.24 | WRN-28-10 | - | - | 77.9 | R-50
Cutmix | y | Random | 97.10 | WRN-28-10 | 83.40 | WRN-28-10 | 78.6 | R-50
| Attentive
---
Cutmix
n | Salient | 95.86 | ENet-B7 | 78.52 | ENet-B7 | - | -
RICAP | y | Random | 97.18 | WRN-28-10 | 82.56 | WRN-28-10 | 78.62 | WRN-50-2
| Mixed
---
Example
y | Random | 96.2 | ResNet-18 | 80.3 | R-18 | - | -
CowMask | y | Random | 97.44 |  | WRN-28-96
---
x2d
84.27 |  | WRN-28-96
---
x2d
73.94 | R-152
| Resize-
---
Mix
n | Random | 97.60 | WRN-28-10 | 84.31 | WRN-28-10 | 79.00 | R-50
| Saliency-
---
Mix
y | Salient | 97.24 | WRN-28-10 | 83.44 | WRN-28-10 | 78.74 | R-50
| Keep-
---
Augment
n | Salient | 97.8 | WRN-28-10 | - | - | 79.1 | R-50
MixUp | y | Random | 97.3 | WRN-28-10 | 82.5 | WRN-28-10 | 77.9 | R-50
| Manifold
---
Mixup
y | Random | 97.45 | WRN-28-18 | 81.96 | WRN-28-10 | 78.7 | R-50
| Smooth-
---
Mix
y | Random | 97.02 | PNet-200 | 85.53 | PNet-200 | 77.66 | R-50
| Sample
---
Pairing
n | Random | 93.07 | GNet | 72.1 | GNet | 70.99 | GNet
AugMix | y | Random | - | - | - | - | 77.6 | R-50
PuzzleMix | y | Salient | - | - | 84.05 | WRN-28-10 | 77.51 | R-50
Co-Mixup | y | Salient | - | - | 80.85 | WRN-16-8 | 77.61 | R-50
Table 1: Performance comparison for various image augmentation approaches for
image classification problem. In table, R refers to ResNet [26], PNet is
PyramidNet [25], GNet is GoogleNet [70], and ENet is EfficientNet [72]

##  5 Performance Comparison

This section compares the augmentation approaches performance for main-stream
tasks, image classification, object detection, and fine-grained image
recognition on publicly available datasets.

###  5.1 Image Classification

Performance for image classification is evaluated on three datasets, Cifar-10
[34], Cifar-100 [34] and ImageNet [59]. For consistency of reported results
among different approaches, we try to collect results for a similar base model
from respective publications. The WideResnet-28-10 is largely common for the
Cifar-10 and Cifar-100, and ResNet-50 for ImageNet. We can conclude about best
performing approaches even for the evaluations with other models based on
performance and number of parameters of these models. From the Table 1, we can
conclude ResizeMix [56], SaliencyMix [74] and KeepAugment [20] performs best
for Cifar-10, Cifar-100 and ImageNet respectively. Overall, the performance
gap between salient and random mixing techniques is not very high, and one can
prefer random mixing approaches for data augmentation due to its low
computation requirements.

|  | Faster-RCNN
---
Backbone
| Baseline
---
(mAP)
| Augmented
---
(mAP)
Train Set | Test Set |  | Experimentation
---
Design
Cutout | R-50 | 76.71 | 77.17 | VOC 07+12 | VOC 07 |  | Backbone is trained on ImageNet using cutout
---
augmentation strategy. The Faster-RCNN backbone
is initialized with this newly trained model
and fine-tuned on train set as in [58]
CutMix | R-50 | 76.71 | 78.31 | VOC 07+12 | VOC 07 | Same as above
MixUp | R-50 | 76.71 | 77.98 | VOC 07+12 | VOC 07 | Same as above
SaliencyMix | R-50 | 76.71 | 78.38 | VOC 07+12 | VOC 07 | Same as above
GridMask | R-50-FPN | 37.4 | 38.3 | COCO 17 | COCO 17 |  | Backbone trained on ImageNet is used for
---
initialization. During object detector training,
gridmask is applied on input after baseline
augmentation strategy as in [58]
KeepAugment | R-50 | 38.4 | 39.5 | COCO 17 | COCO 17 |  | Pre-trained backbone on ImageNet with
---
keepaugment is used. The object detector
is fine-tuned using strategy as in [58]
Random Erasing | VGG-16 | 74.8 | 76.2 | VOC 07+12 | VOC 07 |  | Pre-trained backbone with ImageNet weights.
---
The detector is trained with combined image
and object random erasing strategies keeping
everything similar to [58]
ResizeMix | R-50 | 38.1 | 38.4 | COCO 17 | COCO 17 | Same as above

Table 2: The table shows the improvements in accuracy for object detection
using various image mixing and deleting augmentation techniques. In table, R
refers to ResNet [26] and FPN is feature pyramid network [48], whereas VGG is
from [64]

###  5.2 Object Detection

Backbones trained with these data mixing techniques are also evaluated for
object detection. The evaluation procedure involves replacing the existing
backbone with a newly trained model and fine-tune it with the object detection
task, as per the standard methods mentioned in the literature.
The performance comparison for this task is shown in Table 2. The methods in
Table 2 are evaluated for the faster-RCNN [58] object detection. Every
approach shows the comparison between the standard backbone and the backbone
trained with data augmentation techniques. Here, SaliencyMix outperforms for
VOC 07 [18] dataset, whereas KeepAugment is the best among all for the COCO 17
[49] dataset.

| Code | Model | CUB (%) | Cars (%) | Aircraft (%)
---|---|---|---|---|---
| Attribute
---
Mix
n | R-50 | 88.4 | 94.9 | 92.0
| Intra-Class
---
Part Swapping
n | R-50 | 87.56 | 94.59 | 92.65
SnapMix | y | R-50 | 87.75 | 94.30 | 92.80

Table 3: Comparison of data augmentation strategies designed
for fine-grained image classification. In table, abbreviation of
ResNet is R

###  5.3 Fine-Grained Image Recognition

Fine-grained image classification emphasizes classifying images based on sub-
class categories. For example, the class bird is further split into multiple
categories based on species. More information can be seen for Caltech-UCSD
Birds 200 (CUB), Cars and aircraft in [79], [33] and [52] respectively.
Table 3 shows the performance comparison for a few of the specific image
augmentation techniques designed for fine-grained image recognition. All of
these approaches are designed with saliency information. AttributeMix is well
ahead for the CUB and Cars dataset, whereas SnapMix wins for aircraft.

##  6 Conclusion

Image augmentation with mixing and deletion has shown promising results,
improving the accuracy of CNN’s over baselines. The improvement is not limited
to few tasks but also extended and proven for other tasks, for example, object
detection, semisupervised learning, adversarial training, etc. This paper has
reviewed recently introduced techniques in image mixing and deleting. The
performance comparison among various approaches for different tasks is shown
in tables. The comparative analysis will help researchers in selecting
suitable data augmentation techniques in their research to improve
performance. Overall, the margin of improvement lies within 1-3% range with
these augmentation techniques.

## References

  * [1] Berthelot, D., Carlini, N., Cubuk, E.D., Kurakin, A., Sohn, K., Zhang, H., Raffel, C.: Remixmatch: Semi-supervised learning with distribution alignment and augmentation anchoring.  arXiv preprint arXiv:1911.09785 (2019)
  * [2] Berthelot, D., Carlini, N., Goodfellow, I., Papernot, N., Oliver, A., Raffel, C.: Mixmatch: A holistic approach to semi-supervised learning.  arXiv preprint arXiv:1905.02249 (2019)
  * [3] Berthelot, D., Carlini, N., Goodfellow, I.J., Papernot, N., Oliver, A., Raffel, C.: Mixmatch: A holistic approach to semi-supervised learning.  In: NeurIPS, pp. 5050–5060 (2019)
  * [4] Borgnia, E., Cherepanova, V., Fowl, L., Ghiasi, A., Geiping, J., Goldblum, M., Goldstein, T., Gupta, A.: Strong data augmentation sanitizes poisoning and backdoor attacks without an accuracy tradeoff.  In: ICASSP, pp. 3855–3859. IEEE (2021)
  * [5] Borgnia, E., Geiping, J., Cherepanova, V., Fowl, L., Gupta, A., Ghiasi, A., Huang, F., Goldblum, M., Goldstein, T.: Dp-instahide: Provably defusing poisoning and backdoor attacks with differentially private data augmentations.  arXiv preprint arXiv:2103.02079 (2021)
  * [6] Bouabid, S., Delaitre, V.: Mixup regularization for region proposal based object detectors.  arXiv preprint arXiv:2003.02065 (2020)
  * [7] Chen, J., Yang, Z., Yang, D.: Mixtext: Linguistically-informed interpolation of hidden space for semi-supervised text classification.  In: ACL, pp. 2147–2157. Association for Computational Linguistics (2020)
  * [8] Chen, L.C., Papandreou, G., Kokkinos, I., Murphy, K., Yuille, A.L.: Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  IEEE transactions on pattern analysis and machine intelligence 40(4), 834–848 (2017)
  * [9] Chen, P.: Gridmask data augmentation.  arXiv preprint arXiv:2001.04086 (2020)
  * [10] Chen, T., Kornblith, S., Swersky, K., Norouzi, M., Hinton, G.E.: Big self-supervised models are strong semi-supervised learners.  In: NeurIPS (2020)
  * [11] Chen, X., Fan, H., Girshick, R.B., He, K.: Improved baselines with momentum contrastive learning.  CoRR abs/2003.04297 (2020)
  * [12] Chou, H.P., Chang, S.C., Pan, J.Y., Wei, W., Juan, D.C.: Remix: Rebalanced mixup.  In: European Conference on Computer Vision, pp. 95–110. Springer (2020)
  * [13] Cubuk, E.D., Zoph, B., Mane, D., Vasudevan, V., Le, Q.V.: Autoaugment: Learning augmentation strategies from data.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 113–123 (2019)
  * [14] Cubuk, E.D., Zoph, B., Shlens, J., Le, Q.V.: Randaugment: Practical automated data augmentation with a reduced search space.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702–703 (2020)
  * [15] DeVries, T., Taylor, G.W.: Improved regularization of convolutional neural networks with cutout.  arXiv preprint arXiv:1708.04552 (2017)
  * [16] Dvornik, N., Mairal, J., Schmid, C.: Modeling visual context is key to augmenting object detection datasets.  In: Proceedings of the European Conference on Computer Vision (ECCV), pp. 364–380 (2018)
  * [17] Dwibedi, D., Misra, I., Hebert, M.: Cut, paste and learn: Surprisingly easy synthesis for instance detection.  In: Proceedings of the IEEE International Conference on Computer Vision, pp. 1301–1310 (2017)
  * [18] Everingham, M., Van Gool, L., Williams, C.K., Winn, J., Zisserman, A.: The pascal visual object classes (voc) challenge.  International journal of computer vision 88(2), 303–338 (2010)
  * [19] French, G., Oliver, A., Salimans, T.: Milking cowmask for semi-supervised image classification.  arXiv preprint arXiv:2003.12022 (2020)
  * [20] Gong, C., Wang, D., Li, M., Chandra, V., Liu, Q.: Keepaugment: A simple information-preserving data augmentation approach.  arXiv preprint arXiv:2011.11778 (2020)
  * [21] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks.  Communications of the ACM 63(11), 139–144 (2020)
  * [22] Grill, J., Strub, F., Altché, F., Tallec, C., Richemond, P.H., Buchatskaya, E., Doersch, C., Pires, B.Á., Guo, Z., Azar, M.G., Piot, B., Kavukcuoglu, K., Munos, R., Valko, M.: Bootstrap your own latent - A new approach to self-supervised learning.  In: NeurIPS (2020)
  * [23] Guo, H.: Nonlinear mixup: Out-of-manifold data augmentation for text classification.  In: AAAI, pp. 4044–4051. AAAI Press (2020)
  * [24] Guo, H., Mao, Y., Zhang, R.: Augmenting data with mixup for sentence classification: An empirical study.  arXiv preprint arXiv:1905.08941 (2019)
  * [25] Han, D., Kim, J., Kim, J.: Deep pyramidal residual networks.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5927–5935 (2017)
  * [26] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778 (2016)
  * [27] Hendrycks, D., Mu, N., Cubuk, E.D., Zoph, B., Gilmer, J., Lakshminarayanan, B.: Augmix: A simple data processing method to improve robustness and uncertainty.  arXiv preprint arXiv:1912.02781 (2019)
  * [28] Huang, S., Wang, X., Tao, D.: Snapmix: Semantically proportional mixing for augmenting fine-grained data.  arXiv preprint arXiv:2012.04846 (2020)
  * [29] Huang, Y., Song, Z., Li, K., Arora, S.: Instahide: Instance-hiding schemes for private distributed learning.  In: ICML, _Proceedings of Machine Learning Research_ , vol. 119, pp. 4507–4518. PMLR (2020)
  * [30] Inoue, H.: Data augmentation by pairing samples for images classification.  arXiv preprint arXiv:1801.02929 (2018)
  * [31] Kim, J.H., Choo, W., Jeong, H., Song, H.O.: Co-mixup: Saliency guided joint mixup with supermodular diversity.  arXiv preprint arXiv:2102.03065 (2021)
  * [32] Kim, J.H., Choo, W., Song, H.O.: Puzzle mix: Exploiting saliency and local statistics for optimal mixup.  In: International Conference on Machine Learning, pp. 5275–5285. PMLR (2020)
  * [33] Krause, J., Stark, M., Deng, J., Fei-Fei, L.: 3d object representations for fine-grained categorization.  In: Proceedings of the IEEE international conference on computer vision workshops, pp. 554–561 (2013)
  * [34] Krizhevsky, A., Hinton, G., et al.: Learning multiple layers of features from tiny images (2009)
  * [35] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks.  Advances in neural information processing systems 25, 1097–1105 (2012)
  * [36] Lamb, A., Verma, V., Kannala, J., Bengio, Y.: Interpolated adversarial training: Achieving robust neural networks without sacrificing too much accuracy.  In: AISec@CCS, pp. 95–103. ACM (2019)
  * [37] Lee, J.H., Zaigham Zaheer, M., Astrid, M., Lee, S.I.: Smoothmix: A simple yet effective data augmentation to train robust classifiers.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 756–757 (2020)
  * [38] Lee, K., Zhu, Y., Sohn, K., Li, C., Shin, J., Lee, H.: i-mix: A strategy for regularizing contrastive representation learning.  CoRR abs/2010.08887 (2020)
  * [39] Lee, K., Zhu, Y., Sohn, K., Li, C.L., Shin, J., Lee, H.: i-mix: A strategy for regularizing contrastive representation learning.  arXiv preprint arXiv:2010.08887 (2020)
  * [40] Lee, S., Lee, H., Yoon, S.: Adversarial vertex mixup: Toward better adversarially robust generalization.  In: CVPR, pp. 269–278. Computer Vision Foundation / IEEE (2020)
  * [41] Lever, J., Krzywinski, M., Altman, N.: Regularization (2016)
  * [42] Li, H., Li, J., Guan, X., Liang, B., Lai, Y., Luo, X.: Research on overfitting of deep learning.  In: 2019 15th International Conference on Computational Intelligence and Security (CIS), pp. 78–81. IEEE (2019)
  * [43] Li, H., Zhang, X., Sun, R., Xiong, H., Tian, Q.: Center-wise local image mixture for contrastive representation learning.  CoRR abs/2011.02697 (2020)
  * [44] Li, H., Zhang, X., Tian, Q., Xiong, H.: Attribute mix: semantic data augmentation for fine grained recognition.  In: 2020 IEEE International Conference on Visual Communications and Image Processing (VCIP), pp. 243–246. IEEE (2020)
  * [45] Li, J., Socher, R., Hoi, S.C.H.: Dividemix: Learning with noisy labels as semi-supervised learning.  In: ICLR. OpenReview.net (2020)
  * [46] Liang, D., Yang, F., Zhang, T., Yang, P.: Understanding mixup training methods.  IEEE Access 6, 58774–58783 (2018)
  * [47] Lim, S., Kim, I., Kim, T., Kim, C., Kim, S.: Fast autoaugment.  arXiv preprint arXiv:1905.00397 (2019)
  * [48] Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks for object detection.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2117–2125 (2017)
  * [49] Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft coco: Common objects in context.  In: European conference on computer vision, pp. 740–755. Springer (2014)
  * [50] Liu, Z., Wu, Z., Gan, C., Zhu, L., Han, S.: Datamix: Efficient privacy-preserving edge-cloud inference.  In: ECCV (11), _Lecture Notes in Computer Science_ , vol. 12356, pp. 578–595. Springer (2020)
  * [51] Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards deep learning models resistant to adversarial attacks.  In: ICLR (Poster). OpenReview.net (2018)
  * [52] Maji, S., Rahtu, E., Kannala, J., Blaschko, M., Vedaldi, A.: Fine-grained visual classification of aircraft.  arXiv preprint arXiv:1306.5151 (2013)
  * [53] Mao, X., Ma, Y., Yang, Z., Chen, Y., Li, Q.: Virtual mixup training for unsupervised domain adaptation.  arXiv preprint arXiv:1905.04215 (2019)
  * [54] Na, J., Jung, H., Chang, H.J., Hwang, W.: Fixbi: Bridging domain spaces for unsupervised domain adaptation.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1094–1103 (2021)
  * [55] Naveed, H., Jafri, F., Javed, K., Babri, H.A.: Driver activity recognition by learning spatiotemporal features of pose and human object interaction.  Journal of Visual Communication and Image Representation 77, 103135 (2021)
  * [56] Qin, J., Fang, J., Zhang, Q., Liu, W., Wang, X., Wang, X.: Resizemix: Mixing data with preserved object information and true labels.  arXiv preprint arXiv:2012.11101 (2020)
  * [57] Redmon, J., Farhadi, A.: Yolov3: An incremental improvement.  arXiv preprint arXiv:1804.02767 (2018)
  * [58] Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks.  arXiv preprint arXiv:1506.01497 (2015)
  * [59] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge.  International journal of computer vision 115(3), 211–252 (2015)
  * [60] Shen, Z., Liu, Z., Liu, Z., Savvides, M., Darrell, T.: Rethinking image mixture for unsupervised visual representation learning.  arXiv preprint arXiv:2003.05438 (2020)
  * [61] Shen, Z., Liu, Z., Liu, Z., Savvides, M., Darrell, T., Xing, E.: Un-mix: Rethinking image mixtures for unsupervised visual representation learning.  arXiv preprint arXiv:2003.05438 (2020)
  * [62] Shin, M., Hwang, C., Kim, J., Park, J., Bennis, M., Kim, S.L.: Xor mixup: Privacy-preserving data augmentation for one-shot federated learning.  arXiv preprint arXiv:2006.05148 (2020)
  * [63] Shorten, C., Khoshgoftaar, T.M.: A survey on image data augmentation for deep learning.  Journal of Big Data 6(1), 1–48 (2019)
  * [64] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556 (2014)
  * [65] Singh, K.K., Yu, H., Sarmasi, A., Pradeep, G., Lee, Y.J.: Hide-and-seek: A data augmentation technique for weakly-supervised localization and beyond.  arXiv preprint arXiv:1811.02545 (2018)
  * [66] Sohn, K., Berthelot, D., Carlini, N., Zhang, Z., Zhang, H., Raffel, C., Cubuk, E.D., Kurakin, A., Li, C.: Fixmatch: Simplifying semi-supervised learning with consistency and confidence.  In: NeurIPS (2020)
  * [67] Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R.: Dropout: a simple way to prevent neural networks from overfitting.  The journal of machine learning research 15(1), 1929–1958 (2014)
  * [68] Summers, C., Dinneen, M.J.: Improved mixed-example data augmentation.  In: 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1262–1270. IEEE (2019)
  * [69] Sun, L., Xia, C., Yin, W., Liang, T., Yu, P.S., He, L.: Mixup-transformer: Dynamic data augmentation for NLP tasks.  In: COLING, pp. 3436–3440. International Committee on Computational Linguistics (2020)
  * [70] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1–9 (2015)
  * [71] Takahashi, R., Matsubara, T., Uehara, K.: Ricap: Random image cropping and patching data augmentation for deep cnns.  In: Asian Conference on Machine Learning, pp. 786–798 (2018)
  * [72] Tan, M., Le, Q.: Efficientnet: Rethinking model scaling for convolutional neural networks.  In: International Conference on Machine Learning, pp. 6105–6114. PMLR (2019)
  * [73] Thulasidasan, S., Chennupati, G., Bilmes, J.A., Bhattacharya, T., Michalak, S.: On mixup training: Improved calibration and predictive uncertainty for deep neural networks.  In: NeurIPS, pp. 13888–13899 (2019)
  * [74] Uddin, A., Monira, M., Shin, W., Chung, T., Bae, S.H., et al.: Saliencymix: A saliency guided data augmentation strategy for better regularization.  arXiv preprint arXiv:2006.01791 (2020)
  * [75] Verma, V., Lamb, A., Beckham, C., Najafi, A., Mitliagkas, I., Lopez-Paz, D., Bengio, Y.: Manifold mixup: Better representations by interpolating hidden states.  In: International Conference on Machine Learning, pp. 6438–6447. PMLR (2019)
  * [76] Walawalkar, D., Shen, Z., Liu, Z., Savvides, M.: Attentive cutmix: An enhanced data augmentation approach for deep learning based image classification.  In: ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3642–3646. IEEE (2020)
  * [77] Wang, D., Zhang, Y., Zhang, K., Wang, L.: Focalmix: Semi-supervised learning for 3d medical image detection.  In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3951–3960 (2020)
  * [78] Wang, X., Shrivastava, A., Gupta, A.: A-fast-rcnn: Hard positive generation via adversary for object detection.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2606–2615 (2017)
  * [79] Welinder, P., Branson, S., Mita, T., Wah, C., Schroff, F., Belongie, S., Perona, P.: Caltech-ucsd birds 200 (2010)
  * [80] Yan, S., Song, H., Li, N., Zou, L., Ren, L.: Improve unsupervised domain adaptation with mixup training.  arXiv preprint arXiv:2001.00677 (2020)
  * [81] Ying, X.: An overview of overfitting and its solutions.  In: Journal of Physics: Conference Series, vol. 1168, p. 022022. IOP Publishing (2019)
  * [82] Yun, S., Han, D., Oh, S.J., Chun, S., Choe, J., Yoo, Y.: Cutmix: Regularization strategy to train strong classifiers with localizable features.  In: Proceedings of the IEEE International Conference on Computer Vision, pp. 6023–6032 (2019)
  * [83] Zhang, H., Cisse, M., Dauphin, Y.N., Lopez-Paz, D.: mixup: Beyond empirical risk minimization.  arXiv preprint arXiv:1710.09412 (2017)
  * [84] Zhang, L., Huang, S., Liu, W.: Intra-class part swapping for fine-grained image classification.  In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3209–3218 (2021)
  * [85] Zhang, R., Yu, Y., Zhang, C.: Seqmix: Augmenting active sequence labeling via sequence mixup.  arXiv preprint arXiv:2010.02322 (2020)
  * [86] Zhang, Y., Qiu, Z., Yao, T., Liu, D., Mei, T.: Fully convolutional adaptation networks for semantic segmentation.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6810–6818 (2018)
  * [87] Zhong, Z., Zheng, L., Kang, G., Li, S., Yang, Y.: Random erasing data augmentation.  In: AAAI, pp. 13001–13008 (2020)


∎

e2e-mail<EMAIL_ADDRESS>
11institutetext: School of Mathematics, Physics and Statistics, Shanghai
University of Engineering Science, Shanghai 201620, China 22institutetext:
Center of Application and Research of Computational Physics, Shanghai
University of Engineering Science, Shanghai 201620, China 33institutetext:
Guangxi Key Laboratory for Relativistic Astrophysics, Guangxi University,
Nanning 530004, China

# Applying explicit symplectic integrator to study chaos of charged particles
around magnetized Kerr black hole

Wei Sun1,2 Ying Wang1,2 Fuyao Liu1 Xin Wu1,2,3

(Received: date / Accepted: date)

###### Abstract

In a recent work of Wu, Wang, Sun and Liu, a second-order explicit symplectic
integrator was proposed for the integrable Kerr spacetime geometry. It is
still suited for simulating the nonintegrable dynamics of charged particles
moving around the Kerr black hole embedded in an external magnetic field. Its
successful construction is due to the contribution of a time transformation.
The algorithm exhibits a good long-term numerical performance in stable
Hamiltonian errors and computational efficiency. As its application, the
dynamics of order and chaos of charged particles is surveyed. In some
circumstances, an increase of the dragging effects of the spacetime seems to
weaken the extent of chaos from the global phase-space structure on Poincaré
sections. However, an increase of the magnetic parameter strengthens the
chaotic properties. On the other hand, fast Lyapunov indicators show that
there is no universal rule for the dependence of the transition between
different dynamical regimes on the black hole spin. The dragging effects of
the spacetime do not always weaken the extent of chaos from a local point of
view.

††journal: EPJC

##  1 Introduction

The Kerr spacetime [1] as well as the Schwarzschild spacetime is integrable
because the Carter constant [2] appears as a fourth constant of motion.
However, a test charge motion in a magnetic field around the Kerr black hole
or the Schwarzschild black hole is nonintegrable due to the absence of the
fourth constant of motion. In some cases, the electromagnetic perturbation
induces the onset of chaos and islands of regularity [3-10]. In particular,
Takahashi $\&$ Koyama [11] found the connection between chaoticness of the
motion and the black hole spin. That is, the chaotic properties are weakened
with an increase of the black hole spin.

It is worth pointing out that the detection of the chaotical behavior needs
reliable results on the trajectories. Usually, it needs long-time
calculations, especially in the case of the computation of Lyapunov exponents.
Thus, the adopted computational scheme is required to have good stability,
high precision and small cost of computational time. Because these general
relativistic curved spacetimes correspond to Hamiltonian systems with the
symplectic nature, a low order symplectic integration scheme which respects
the symplectic nature of Hamiltonian dynamics [12-14] is naturally regarded to
as the most appropriate solver. The Hamiltonian systems for these curved
spacetimes are not separable to the phase-space variables, therefore, no
explicit symplectic integrators but implicit (or implicit and explicit
combined) symplectic integrators [15-22] can be available in general. However,
the implicit methods are more computationally demanding than the explicit ones
at same order. To solve this problem, Wang et al. [23-25] split the
Hamiltonians of Schwarzschild, Reissner-Nordström and Reissner-Nordström
-(anti)-de Sitter spacetime geometries into four, five and six integrable
separable parts, whose analytical solutions are explicit functions of proper
time. In this way, explicit symplectic integrators were designed for the
spacetime geometries. Unfortunately, the dragging effects of the spacetime by
a rotating black hole make the Hamiltonian of Kerr spacetime have no desired
splitting form similar to the Hamiltonian splitting form of Schwarzschild
spacetime, and then lead to the difficulty in the construction of explicit
symplectic integrators. This obstacle was successfully overcame by the
introduction of time transformations of Mikkola [26] to the Hamiltonian of
Kerr spacetime geometry. The obtained time-transformed Hamiltonian exists a
desired splitting, and explicit symplectic integrators were established by Wu
et al. [27]. At this point, a note is worthwhile on such time-transformed
explicit symplectic integrators. They are constant time-steps in the new
coordinate time and the algorithmic symplecticity can be maintained. Although
they are variant time-steps in the proper coordinate time, the varying proper
time-steps have only small differences and are approximately constant when the
new integration time is not very long. In other words, the time transformation
plays a main role in obtaining separable time-transformed Hamiltonian for the
use of explicit symplectic integrators rather than adaptive time step control.

One of the main purposes in this paper is to explore whether the proposed
explicit symplectic integrators for the Kerr spacetime geometry work well in
the simulation of the chaotic motion of charged particles near the Kerr black
hole immersed in a weak, asymptotically uniform external magnetic field. As
another important purpose, we apply the established second-order explicit
symplectic method to check the result of Takahashi $\&$ Koyama [11] on the
dragging effects weakening the chaotic properties, and to survey how a small
change of the magnetic parameter affects a dynamical transition from order to
chaos. Besides the Poincaré map method, the methods of Lyapunov exponents [28,
29] and fast Lyapunov indicators [30-32] are employed to distinguish between
regular and chaotic orbits and to quantify the dependence of transition from
regular to chaotic dynamics on a certain parameter.

For the sake of our purposes, we introduce a Hamiltonian system for the
description of charged particles moving around the Kerr black hole surrounded
with an external magnetic field in Sect. 2. A second-order time-transformed
explicit symplectic algorithm is constructed and used to quantify the
transition from regular to chaotic dynamics by means of different methods
finding chaos in Sect. 3. Finally, the main results are concluded in Sect. 4.

##  2 Kerr black hole with external magnetic field

Let us consider a particle with charge $q$ moving around the Kerr black hole,
surrounded by an external asymptotically uniform magnetic field with strength
$B$. The particle motion is described by the following Hamiltonian formalism
[33]

$H=\frac{1}{2}g^{\mu\nu}(p_{\mu}-qA_{\mu})(p_{\nu}-qA_{\nu}).$ (1)

Contravariant Kerr metric $g^{\mu\nu}$ has nonzero components

$\displaystyle
g^{tt}=-\frac{D}{\Lambda\Sigma},~{}~{}g^{t\phi}=-\frac{2ar}{\Lambda\Sigma}=g^{\phi
t},$ $\displaystyle
g^{rr}=\frac{\Lambda}{\Sigma},~{}~{}~{}~{}~{}~{}g^{\theta\theta}=\frac{1}{\Sigma},$
$\displaystyle g^{\phi\phi}=\frac{\Sigma-2r}{\Lambda\Sigma\sin^{2}\theta};$
$\displaystyle\Sigma=r^{2}+a^{2}\cos^{2}\theta,~{}~{}~{}~{}\Lambda=r^{2}+a^{2}-2r,$
$\displaystyle D=(r^{2}+a^{2})^{2}-\Lambda a^{2}\sin^{2}\theta.$

$A^{\mu}$ is an electromagnetic four-vector potential [33, 34]

$A^{\mu}=aB\xi^{\mu}_{(t)}+\frac{B}{2}\xi^{\mu}_{(\phi)},$ (2)

where $\xi^{\mu}_{(t)}=(1,0,0,0)$ and $\xi^{\mu}_{(\phi)}=(0,0,0,1)$ are
timelike and spacelike axial Killing vectors. Obviously, the four-vector
potential has two nonzero covariant components

$\displaystyle A_{t}$ $\displaystyle=$ $\displaystyle
aBg_{tt}+\frac{B}{2}g_{t\phi},$ (3) $\displaystyle=$ $\displaystyle-
aB[1+\frac{r}{\Sigma}(\sin^{2}\theta-2)]$ $\displaystyle A_{\phi}$
$\displaystyle=$ $\displaystyle aBg_{t\phi}+\frac{B}{2}g_{\phi\phi}$ (4)
$\displaystyle=$ $\displaystyle
B\sin^{2}\theta[\frac{r^{2}+a^{2}}{2}+\frac{a^{2}r}{\Sigma}(\sin^{2}\theta-2)].$

The above covariant metric’s components are

$\displaystyle
g_{tt}=-(1-\frac{2r}{\Sigma}),~{}~{}g_{t\phi}=-\frac{2ar\sin^{2}\theta}{\Sigma}=g_{\phi
t},$ $\displaystyle
g_{\phi\phi}=(r^{2}+a^{2}+\frac{2ra^{2}}{\Sigma}\sin^{2}\theta)\sin^{2}\theta.$

$p_{\mu}$ represents a covariant generalized four-momentum, determined by the
relation

$\dot{x}^{\mu}=\frac{\partial H}{\partial
p_{\mu}}=g^{\mu\nu}(p_{\nu}-qA_{\nu}).$ (5)

Note that $\dot{x}^{\mu}$ is a derivative of coordinate
$x^{\mu}=(t,r,\theta,\phi)$ with respect to proper time $\tau$, called as a
4-velocity. Eq. (5) is rewritten as

$p_{\mu}=g_{\mu\nu}\dot{x}^{\nu}+qA_{\mu}.$ (6)

Because $\dot{p}_{\mu}=-\partial H/\partial x^{\mu}$, $p_{t}$ and $p_{\phi}$
are constants of motion. They are the particle’s energy and angular momentum:

$\displaystyle-E$ $\displaystyle=$ $\displaystyle
p_{t}=g_{tt}\dot{t}+g_{t\phi}\dot{\phi}+qA_{t},$ (7) $\displaystyle L$
$\displaystyle=$ $\displaystyle
p_{\phi}=g_{t\phi}\dot{t}+g_{\phi\phi}\dot{\phi}+qA_{\phi}.$ (8)

They have equivalent expressions

$\displaystyle\dot{t}$ $\displaystyle=$ $\displaystyle-
f_{1}(E+qA_{t})-f_{2}(L-qA_{\phi}),$ (9) $\displaystyle\dot{\phi}$
$\displaystyle=$ $\displaystyle f_{2}(E+qA_{t})+f_{3}(L-qA_{\phi}),$ (10)

where

$\displaystyle f_{1}$ $\displaystyle=$
$\displaystyle\frac{g_{\phi\phi}}{g_{tt}g_{\phi\phi}-g^{2}_{t\phi}},$ (11)
$\displaystyle f_{2}$ $\displaystyle=$
$\displaystyle\frac{g_{t\phi}}{g_{tt}g_{\phi\phi}-g^{2}_{t\phi}},$ (12)
$\displaystyle f_{3}$ $\displaystyle=$
$\displaystyle\frac{g_{tt}}{g_{tt}g_{\phi\phi}-g^{2}_{t\phi}}.$ (13)

Now, Eq. (1) is rewritten as

$\displaystyle H$ $\displaystyle=$
$\displaystyle\frac{1}{2}g^{\mu\nu}(p_{\mu}-qA_{\mu})(p_{\nu}-qA_{\nu})$ (14)
$\displaystyle=$ $\displaystyle
F+\frac{1}{2}\frac{\Lambda}{\Sigma}p^{2}_{r}+\frac{1}{2}\frac{p^{2}_{\theta}}{\Sigma},$

where $F$ is a function of $r$ and $\theta$ as follows:

$\displaystyle F$ $\displaystyle=$
$\displaystyle\frac{1}{2}[g^{tt}(E+qA_{t})^{2}+g^{\phi\phi}(L-qA_{\phi})^{2}]$
(15) $\displaystyle-g^{t\phi}(E+qA_{t})(L-qA_{\phi})$ $\displaystyle=$
$\displaystyle\frac{1}{2}[f_{1}(E+qA_{t})^{2}+f_{3}(L-qA_{\phi})^{2}]$
$\displaystyle+f_{2}(E+qA_{t})(L-qA_{\phi})$ $\displaystyle=$
$\displaystyle-\frac{D}{2\Lambda\Sigma}(E+qA_{t})^{2}+\frac{\Sigma-2r}{2\Lambda\Sigma\sin^{2}\theta}(L-qA_{\phi})^{2}$
$\displaystyle+\frac{2ar}{\Lambda\Sigma}(E+qA_{t})(L-qA_{\phi}).$

In Eq. (14), we take the speed of light $c$ and the gravitational constant $G$
as geometrized units, $c=G=1$. The black hole’s mass $M$ is also one unit,
$M=1$. In practice, dimensionless operations are given to the related
quantities via a series of scale transformations. That is, $r\rightarrow rM$,
$t\rightarrow tM$, $\tau\rightarrow\tau M$, $a\rightarrow aM$, $E\rightarrow
Em$, $p_{r}\rightarrow mp_{r}$, $L\rightarrow mML$, $p_{\theta}\rightarrow
mMp_{\theta}$, $q\rightarrow mq$, $B\rightarrow B/M$ and $H\rightarrow
m^{2}H$, where $m$ denotes the particle’s mass. Hereafter, we take $\beta=qB$.
For any rotating black hole in general relativity, its spin angular momentum
always satisfies $|a|\leq 1$.

Because the timelike 4-velocity (5) always satisfies the relation
$\dot{x}^{\mu}\dot{x}_{\mu}=-1$, the Hamiltonian (14) itself is identical to a
given constant

$H=-\frac{1}{2}.$ (16)

Besides this value and $E$ and $L$, other constants do not exist in the
system. Thus, the Hamiltonian is non-integrable.

##  3 Numerical investigations

Following the work of Wu et al. [27], we establish a second-order explicit
symplectic integrator for a time-transformed Hamiltonian of the system (14).
For comparison, a second-order implicit and explicit mixed symplectic method
is applied to solve the Hamiltonian (14) in Sect. 3.1. Then, the explicit
symplectic integrator is used to investigate the dependence of the regular and
chaotic dynamics of the time-transformed Hamiltonian on a certain parameter by
means of several methods finding chaos in Sect. 3.2.

###  3.1 Numerical integration scheme

Clearly, the Hamiltonian (14) is inseparable to the variables. Implicit
symplectic methods can be applied to it without doubt. The first term $F$ in
Eq. (14) is solved analytically, and the sum of the second and third terms is
solved numerically by the second-order implicit midpoint symplectic method IM2
[15]. The explicit and implicit solutions compose a second-order implicit and
explicit mixed symplectic integrator for $H$, labeled as IE2. Such a mixed
integrators was discussed by several authors [17-22]. It can save labour
compared with the midpoint rule IM2 directly acting on the whole Hamiltonian
$H$.

On the other hand, Wu et al. [27] used a time transformation function

$d\tau=g(r,\theta)dw,~{}~{}g(r,\theta)=\frac{\Sigma}{r^{2}}$ (17)

to obtain a time-transformed Hamiltonian to the Kerr geometry. The time-
transformed Hamiltonian can be solved by explicit symplectic integrators. In
the present problem, the time-transformed Hamiltonian is

$K=g(H+p_{0})=\frac{\Sigma}{r^{2}}(F+p_{0})+\frac{\Lambda}{2r^{2}}p^{2}_{r}+\frac{1}{2r^{2}}p^{2}_{\theta}.$
(18)

$p_{0}=-H=1/2$ is a momentum with respect to coordinate $q_{0}=\tau$. Thus,
$K$ is always identical to zero, $K=0$. The time-transformed Hamiltonian is
split into five parts

$K=K_{1}+K_{2}+K_{3}+K_{4}+K_{5},$ (19)

where the sub-Hamiltonians read

$\displaystyle K_{1}$ $\displaystyle=$
$\displaystyle\frac{\Sigma}{r^{2}}(F+p_{0}),$ (20) $\displaystyle K_{2}$
$\displaystyle=$ $\displaystyle\frac{1}{2}p^{2}_{r},$ (21) $\displaystyle
K_{3}$ $\displaystyle=$ $\displaystyle-\frac{1}{r}p^{2}_{r},$ (22)
$\displaystyle K_{4}$ $\displaystyle=$
$\displaystyle\frac{a^{2}}{2r^{2}}p^{2}_{r},$ (23) $\displaystyle K_{5}$
$\displaystyle=$ $\displaystyle\frac{1}{2r^{2}}p^{2}_{\theta}.$ (24)

$K_{1}$ seems to be the same as that of Wu et al. [27] from the expressional
form, but is unlike that of Wu et al. $K_{2}$, $K_{3}$, $K_{4}$ and $K_{5}$
are the same as those of Wu et al. The five sub-Hamiltonians have analytical
solutions that are explicit functions of the new coordinate time $w$. Their
solutions correspond to operators $\widetilde{K}_{1}$, $\widetilde{K}_{2}$,
$\widetilde{K}_{3}$, $\widetilde{K}_{4}$ and $\widetilde{K}_{5}$. According to
the idea of Wu et al., these operators can compose second-order explicit
symplectic integrator for $K$:

$\displaystyle S2(h)$ $\displaystyle=$
$\displaystyle\widetilde{K}_{5}(\frac{h}{2})\circ\widetilde{K}_{4}(\frac{h}{2})\circ\widetilde{K}_{3}(\frac{h}{2})\circ\widetilde{K}_{2}(\frac{h}{2})\circ\widetilde{K}_{1}(h)$
(25)
$\displaystyle\circ\widetilde{K}_{2}(\frac{h}{2})\circ\widetilde{K}_{3}(\frac{h}{2})\circ\widetilde{K}_{4}(\frac{h}{2})\circ\widetilde{K}_{5}(\frac{h}{2}),$

where $h$ represents a new coordinate time-step. The use of fixed new
coordinate time-step maintains the symplecticity of the algorithms for the
time-transformed Hamiltonian $K$. However, the proper time $\tau$ takes
varying step-sizes. Such symplectic integrators are adaptive time-steps [26,
35, 36]. However, the adaptive proper time step control is almost absent in
the present problem because the time transformation function $1<g\leq
1+1/r^{2}\approx 1$ for $r\gg 1$. This indicates that the time transformation
function mainly provides the desirable separable time-transformed Hamiltonian.

Based on the numerical results of Wu et al., the second-order explicit
symplectic method S2 performs good computational efficiency and stabilizing
error behavior, compared with the fourth-order explicit symplectic method S4.
Thus, we focus on the application of S2 to the system (19). For comparison,
the second-order implicit and explicit mixed symplectic integrator IE2 is
applied to $H$. Let us take the time-step $h=1$. Note that $h=1$ denotes a
proper time-step for IE2 and a new coordinate time-step for S2. The parameters
are given by $E=0.995$, $L=4.6$, $a=0.5$ and $\beta=0.001$. The initial
conditions are $\theta=\pi/2$ and $p_{r}=0$. The initial separations
$r_{0}=11$ and 75 are respectively given to Orbits 1 and 2. The initial value
of $p_{\theta}>0$ is determined by Eqs. (14) and (16). It is shown in Fig.
1(a) that Hamiltonian errors $\Delta=K$ for the explicit symplectic algorithm
S2 and $\Delta=-H-1/2$ for the second-order implicit and explicit mixed
symplectic method IE2 remain stable and bounded. Both algorithms give no
explicit error differences to the same orbit. The errors have the same order
for the two orbits. If IE2 is applied to $K$, the results are almost the same
as those for the application of IE2 to $H$. Although S2 is not explicitly
better than IE2 in numerical accuracy, it is in computational efficiency, as
claimed in the work of Wu et al.

It is worth pointing out that IE2 and S2 work in different time systems. The
integration time for IE2 is the proper time $\tau=10^{8}$. However, the
integration time for S2 is the new coordinate time $w=10^{8}$. The new
coordinate time $w=10^{8}$ and its corresponding real proper time have no
large differences. This result is shown clearly in Table 1. In fact,
differences between the solutions of $H$ and those of $K$ are negligible
before the integration time arrives at $10^{5}$. So are the differences
between the proper times of $K$ and the coordinate times of $K$. These facts
confirm that the time transformation does not provide adaptive time-steps but
a separable time-transformed Hamiltonian. Of course, these differences between
the solutions of the two systems get slowly large as the integration time
spans $10^{5}$ and increases. In this case, the solutions of $K$ at a given
proper time are obtained with the aid of some interpolation method.

In fact, the tested orbits have different dynamical behaviors, as shown on
Poincaré sections at the plane $\theta=\pi/2$ and $p_{\theta}>0$ in Fig. 1(b).
The phase-space structure of Orbit 1 is a torus, regarded as the
characteristic of an ordered orbit. However, the phase-space structure of
Orbit 2 has many discrete points that are randomly filled in an area, regarded
as the characteristic of a chaotic orbit. It can be seen from Fig. 1(a) that
the performance of S2 (with IE2) is independent of the regularity and
chaoticity of orbits. The phase-space structures in Fig. 1(b) are described by
S2. They are also given by IE2.

###  3.2 Dynamical transition with parameters varying

In what follows, we use the algorithm S2 to trace the orbital dynamics of the
system $K$. Meantime, methods of Poincaré sections, Lyapunov exponents and
fast Lyapunov indicators are employed to find chaos.

####  3.2.1 Poincaré sections

It is worth pointing out that the chaoticity of Orbit 2 is due to the charged
particle suffered from the electromagnetic field interaction. As claimed in
the Introduction, the Kerr spacetime without the electromagnetic field
interaction holds the Carter constant as the fourth integral of motion and
therefore is integrable and nonchaotic. If the electromagnetic field is
included in the Kerr spacetime, then the Carter constant is no longer present
and the dynamics of charged particles becomes nonintegrable. This
nonintegrability is an insufficient but necessary condition for the occurrence
of chaos. When the electromagnetic field interaction acts as a relatively
small perturbation, the gravitational force from the black hole is a dominant
force and the dynamical system is nearly integrable. In this case, the Kerr
spacetime with the electromagnetic field interaction can exhibit the similar
dynamical features of the Kerr spacetime without the electromagnetic field
interaction. That is to say, no chaos occurs and all orbits are regular
Kolmogorov-Arnold-Moser (KAM) tori. This result is suitable for the magnetic
parameter $\beta=4\times 10^{-4}$ in Fig. 2(a). The tori in the perturbed case
unlike those in the unperturbed case are twisted in the shapes; e.g., Orbit 1
corresponds to a peculiar torus, and Orbit 4 with initial separation
$r_{0}=110$ yields a triangle torus. When $\beta=6\times 10^{-4}$ in Fig.
2(b), the tori are still present, but are to a large degree different from
those for $\beta=4\times 10^{-4}$ in the shapes. There are typical differences
of Orbits 1, 2 and 3 between Figs. 2 (a) and (b). In particular, Orbit 4 in
Fig. 2(a) is evolved to a twisted figure-eight orbit with a hyperbolic point
in Fig. 2(b) due to the electromagnetic field interaction. Such a hyperbolic
point has a stable direction and another unstable direction, and easily
induces chaos. As the electromagnetic field interaction increases and can
generally match with the black hole gravitational force, chaos occurs. The
hyperbolic point causes Orbit 4 to be chaotic for $\beta=7\times 10^{-4}$ in
Fig. 2(c). When the magnetic field strength is further enhanced, e.g.,
$\beta=8\times 10^{-4}$ in Fig. 2(d), $\beta=1\times 10^{-3}$ in Fig. 1(b) and
$\beta=1.2\times 10^{-3}$ in Fig. 2(e), chaos becomes stronger and stronger.
Particular for $\beta=3\times 10^{-3}$ in Fig. 2(f), chaos is existent almost
everywhere in the whole phase space. In a word, it can be concluded clearly
from Figs. 1(b) and 2 that an increase of the magnetic parameter is easier to
induce chaos and enhances the strength of chaos from the global phase-space
structure.

On the other hand, we are also interested in knowing what the dynamical
transition with an increase of the black hole angular momentum $a$ is. To
answer this question, we consider some values of $a$. For $a=0$ corresponding
to the Schwarzschild case in Fig. 3(a), chaos densely exists almost everywhere
in the whole phase space. This does not mean that any regular orbits are ruled
out. In fact, regular Orbit 2 and black orbit with initial separation
$r_{0}=25$ in Fig. 1(b) are still ordered in Fig. 3(a). Given $a=0.2$, chaotic
Orbits 1 and 3 in Fig. 3(a) become regular in Fig. 3(b), whereas ordered Orbit
2 in Fig. 3(a) becomes chaotic in Fig. 3(b). As the spin frequently increases
[e.g., $a=0.5$ in Fig. 1(b), $a=0.8$ in Fig. 3(c) and $a=1$ in Fig. 3(d)], the
regular region seems to gradually increase and the chaotic region seems to
decrease. In other words, an increase of $a$ seems to weaken the extent of
chaos from the global phase-space structure. This seems to support the result
of [11] about the black hole spin weakening the chaotic properties. However,
this does not mean that some individual orbits must be more chaotic as the
dragging effects of the spacetime by a rotating black hole decrease. For
example, Orbit 2 is regular for $a$=0, 1 but chaotic for $a$=0.2, 0.5, 0.8.

####  3.2.2 Fast Lyapunov indicators

Apart from the Poincaré map method, the technique of Lyapunov exponents is
very efficient to detect the onset of chaos and provides a quantitative
measure to chaos. An average exponential deviation of two nearby orbits in the
system $K$ is measured by the largest Lyapunov exponent

$\lambda=\lim_{w\rightarrow\infty}\frac{1}{w}\ln\frac{d(w)}{d(0)},$ (26)

where $d(0)$ and $d(w)$ represent the separations between the two nearby
orbits at times 0 and $w$, respectively. Precisely speaking, $d(w)$ is written
as

$\displaystyle d(w)$ $\displaystyle=$
$\displaystyle[(r_{2}-r_{1})^{2}+(\theta_{2}-\theta_{1})^{2}+(p_{r2}-p_{r1})^{2}$
(27) $\displaystyle+(p_{\theta 2}-p_{\theta 1})^{2}]^{1/2},$

where $(r_{1},\theta_{1},p_{r1},p_{\theta 1})$ is a set of phase-space
variables of one orbit at time $w$ and $(r_{2},\theta_{2},p_{r2},p_{\theta
2})$ is a set of phase-space variables of its nearby orbit. Three points
should be noted. (i) The initial distance $d(0)$ had better be $10^{-8}$ in
the double-precision case. (ii) Appropriate renormalizations to the distance
$d(w)$ are required from time to time. (iii) Although the Lyapunov exponent
depends on the choice of spacetime coordinates, it is very approximate to an
invariant Lyapunov exponent because the new coordinate time $w$ and its
corresponding proper time $\tau$ are approximately equal, and the distance
$d(w)$ and its corresponding proper distance are, too. More details about the
invariant Lyapunov exponent were given by Wu $\&$ Huang [29] and Wu et al.
[32]. A positive Lyapunov exponent indicates the chaoticity of a bounded
orbit, and zero Lyapunov exponent describes the regularity of a bounded orbit.
This allows us to detect chaos from order. It is clearly seen from the
Lyapunov exponents in Fig. 4(a) that Orbits 1 and 2 in Fig. 1 are ordered and
chaotic, respectively. Orbit 3 in Fig. 2(d) seems to be regular till
$w=10^{7}$ because its Lyapunov exponent like Orbit 1’s Lyapunov exponent
decreases and does not tend to a stabilizing value. However, the Lyapunov
exponent of Orbit 3 unlike that of Orbit 1 tends to a stabilizing positive
value when the integration time spans from $10^{7}$ to $10^{8}$; therefore,
Orbit 3 is chaotic. Because the Lyapunov exponent of Orbit 3 is smaller than
that of Orbit 2, the chaoticity of Orbit 3 is weaker than that of Orbit 2. The
result is consistent with that described by the method of Poincaré sections in
Figs. 1(b) and 2(d).

In general, a long enough integration time is necessary to obtain a
stabilizing value of Lyapunov exponent. In particular, the chaoticity of Orbit
3 cannot be distinguished from the regularity of Orbit 1 in terms of Lyapunov
exponents until $w=10^{8}$. Compared with the method of Lyapunov exponents,
the fast Lyapunov indicator (FLI) of Froeschlé $\&$ Lega [31] is based on
computations of tangent vectors and is regarded as a quicker technique to
separate chaotic orbits from regular ones. Wu et al. [32] modified the FLI
with tangent vectors as the FLI of two nearby orbits:

$FLI=\log_{10}\frac{d(w)}{d(0)}.$ (28)

Here, $d(0)=10^{-9}$ is admissible in the double-precision case, and a few but
not many renormalizations to the distance $d(w)$ are still necessary to avoid
the saturation of orbits. Regular and chaotic orbits can be identified
according to completely different time rates of growth of FLIs. An exponential
increase of FLI of a bounded with time $\log_{10}w$ indicates the chaoticity
of the orbit; an algebraical increase of FLI of a bounded with time
$\log_{10}w$ indicates the regularity of the orbit. In other words, if the FLI
of a bounded orbit is much larger than that of another bounded orbit for a
given time, the former orbit is chaotic, but the latter orbit is regular. On
the basis of this point, the properties of orbits in Fig. 4(b) can be known
easily. Without doubt, Orbit 2 is strongly chaotic because its FLI arrives at
12 when the integration time $w=6\times 10^{5}$, and is larger than 200 when
$w=10^{7}$. Unlike the Lyapunov exponents, the FLIs are easy to distinguish
between the regularity of Orbit 1 and the chaoticity of Orbit 3 when
$w=10^{7}$ because the FLI of Orbit 1 is smaller than 6, but the FLI of Orbit
3 is 13.5.

As claimed in Ref. [32], the FLI is a good tool to allow us to identify the
transition between different dynamical regimes with a variation of a certain
dynamical parameter. Letting $a=0.5$ be fixed and $\beta$ run from 0 to
$1.5\times 10^{-3}$, we trace the dependence of the dynamical transition on
the magnetic parameter. For a given value $\beta$, the integration lasts to
the time $w=10^{7}$, but it ends if the FLI of some orbit is 20. There are
three areas: regular area with FLIs smaller than 6, weak chaos area with FLIs
no less than 6 but no more than 20 and strong chaos area with FLIs larger than
20. When $\beta<0.7\times 10^{-3}$, no chaos occurs in Figs. 5 (a) and (b).
This is because the magnetic field force to charged particles is smaller than
the black hole gravity to charged particles, as is mentioned above.
$\beta<0.7\times 10^{-3}$ is a regular area of the magnetic parameter. Chaos
will get easier to a large degree as $\beta$ increases. When $\beta>0.8\times
10^{-3}$, many values of $\beta$ correspond to the onset of chaos and many
other values indicate the presence of order in Fig. 5(a). In particular, chaos
and order are present in the neighbourhood of $\beta=1.5\times 10^{-3}$.
Unlike in Fig. 5(a), all values of $\beta>1.1\times 10^{-3}$ can induce strong
chaos in Fig. 5(b). Of course, few values of the magnetic parameter correspond
to weak chaos in the two panels. In short, the technique of FLIs and the
method of Poincaré sections consistently show that an increase of the magnetic
parameter leads to a strong probability for inducing the occurrence of chaos
and strengthening the extent of chaos.

Figs. 6 (a)-(d) plot the dependence of FLIs on the black hole spin $a$ when
$\beta=0.001$ is fixed. Strong chaos occurs as $a\rightarrow 0$ but no chaos
appears as $a\rightarrow 1$ for the initial separation $r_{0}=40$ in Fig.
6(a). For the most values of $a\in[0.25,0.65]$, chaos is absent, too. The
transition has some differences for the initial separation $r_{0}=60$ in Fig.
6(b). Chaos exists when $a\rightarrow 1$. However, there is no chaos at the
left end $a=0$ and the right end $a=1$ in Figs. 6 (c) and (d). These facts
show that different combinations of initial conditions and other dynamical
parameters affect the dependence of the dynamical transition on the black hole
spin. Thus, no universal rule can be given to the relation between the
dynamical transition and the black hole spin.

A notable point is that the aforementioned results are completely based on
mathematical calculations. To justify the mathematical correctness of the
obtained results, we employ an eighth- and ninth-order Runge-Kutta-Fehlberg
integrator (RKF89) with adaptive step sizes to integrate the system (14) or
(19). This algorithm can provide a machine precision to the Hamiltonians if
roundoff errors are ignored. Of course, the higher-order method is more
expensive in computations than the second-order explicit scheme S2.
Consequently, the results obtained from RKF89 are consistent with those given
by S2. It is very perfect and idealized to make a comparison of the obtained
theoretical results with real observational results in the relativistic
astrophysics. However, the known observational results have not sufficiently
supported the study of chaotic behavior of black holes. There are two main
reasons. On one hand, it costs a long enough time span to distinguish between
the regular and chaotic two cases. The time for the determination of chaos is
$10^{7}M$ or $10^{8}M$ from the numerical computations. Nevertheless, it is
impossibly available from the observations. On the other hand, it costs
massive data to detect chaos from order. This fact can be seen clearly from
the numerical simulations. However, only a small amount of observational data
are given. Because of these reasons, the study of chaos and islands of
regularity of black holes in all the known publications such as [3-11, 23-25,
33 and 34] is restricted to the theoretically numerical results rather than
the real observational results.

##  4 Conclusion

Because of the dragging effects of the spacetime by a rotating black hole, the
Kerr geometry is complicated. In spite of this, it is integrable and
nonchaotic. When an external magnetic field is included, the dynamics of
charged particles becomes more complicated. In fact, it is nonintegrable and
possibly chaotic. We find that the second-order explicit symplectic integrator
for the integrable Kerr spacetime proposed in the work of Wu et al. [27] is
still suited for the present nonintegrable problem. The successful application
of the explicit symplectic integrator is owing to the contribution of the time
transformation given in Eq. (17). The theoretical analysis and numerical
results show that such a time transformation does not mainly bring adaptive
proper time steps but a separable time-transformed Hamiltonian, which
satisfies a need for the use of explicit symplectic integrator. The explicit
symplectic algorithm performs good performance in numerical accuracy, stable
error behaviour and computational cost regardless of whether tested orbits are
regular or chaotic.

The explicit symplectic integrator is very suitable for studying the long-term
qualitative evolution of orbits of charged particles moving around the
magnetized Kerr black hole. The electromagnetic force interactions are mainly
responsible for the nonintegrability and chaoticity of charged particle
dynamics. In some circumstances, charged particle motions can be chaotic. An
increase of the dragging effects of the spacetime seems to weaken the extent
of chaos from the global phase-space structure on Poincaré sections. However,
an increase of the magnetic parameter results in strengthening the chaotic
properties. On the other hand, the fast Lyapunov indicators by scanning the
black hole spin parameter show that no universal rule can be given to the
dependence of the transition between different dynamical regimes on the black
hole spin. This is because the transition is dependent on not only the spin
parameter but also different combinations of initial conditions and other
parameters. Unlike an increase of the black hole spin, that of the magnetic
parameter may much easily induce chaos.

Acknowledgements: The authors are very grateful to two referees for valuable
comments and suggestions. This research has been supported by the National
Natural Science Foundation of China [Grant Nos. 11973020 (C0035736), 11533004,
11803020, 41807437, U2031145], and the Natural Science Foundation of Guangxi
(Grant Nos. 2019JJD110006 and 2018GXNSFGA281007).

## References

  * Kerr (1963) R. P. Kerr, Phy. Rev. Lett. 11, 237 (1963)
  * Carter (1968) B. Carter, Phy. Rev. 174, 1559 (1968)
  * Nakamura & Ishizuka (1968) Y. Nakamura, T. Ishizuka, Astrophys. Space Science 210, 105 (1993)
  * Kopáček et al. (2010) O. Kopáček, V. Karas, J. Kovář, Z. Stuchlík, Astrophys. J. 722, 1240 (2010)
  * Kopáček & Karas (2014) O. Kopáček, V. Karas, Astrophys. J. 787, 117 (2014)
  * Kopáček & Karas (2018) O. Kopáček, V. Karas, Astrophys. J. 853, 53 (2018)
  * Frolov & Shoom (2010) V. P. Frolov, A. A. Shoom, Phys. Rev. D 82, 084034 (2010)
  * Kopáček et al. (2017) M. Kološ, A. Tursunov, Z. Stuchlík, Eur. Phys. J. C 77, 860 (2017)
  * Pánis et al. (2019) R. Pánis, M. Kološ, Z. Stuchlík, Eur. Phys. J. C 79, 479 (2019)
  * Stuchlík et al. (2020) Z. Stuchlík, M. Kološ, J. Kovář, P. Slaný, A. Tursunov, Universe 6, 26 (2020)
  * Takahashi & Koyama (2009) M. Takahashi, H. Koyama, Astrophys. J. 693, 472 (2009)
  * Wisdom (1982) J. Wisdom, Astron. J. 87, 577 (1982)
  * Ruth (1983) R. D. Ruth, IEEE Trans. Nucl. Sci. NS 30, 2669 (1983)
  * Wisdom & Holman (1991) J. Wisdom J., M. Holman, Astron. J. 102, 1528 (1991)
  * Feng (1986) K. Feng, Journal of Computational Mathematics 44, 279 (1986)
  * Brown (2006) J. D. Brown, Phys. Rev. D 73, 024001 (2006)
  * Liao (2006) X. H. Liao, Celest. Mech. Dyn. Astron. 66, 243 (1997)
  * Preto (2009) M. Preto, P. Saha, Astrophys. J. 703, 1743 (2009)
  * Lubich & Walther (2010) C. Lubich, B. Walther, B. Brügmann, Phys. Rev. D 81, 104025 (2010)
  * Zhong et al. (2010) S. Y. Zhong, X. Wu, S. Q. Liu, X. F. Deng, Phys. Rev. D 82, 124040 (2010)
  * Mei et al. (2013a) L. Mei, X. Wu, F. Liu, Eur. Phys. J. C 73, 2413 (2013)
  * Mei et al. (2013b) L. Mei, M. Ju, X. Wu, S. Liu, Mon. Not. R. Astron. Soc. 435, 2246 (2013)
  * Wang et al. (2021a) Y. Wang, W. Sun, F. Liu, X. Wu, Astrophys. J. 907, 66 (2021) (Paper I)
  * Wang et al. (2021b) Y. Wang, W. Sun, F. Liu, X. Wu, Astrophys. J. 909, 22 (2021) (Paper II)
  * Wang et al. (2021c) Y. Wang, W. Sun, F. Liu, X. Wu, Astrophys. J. Supp. Series 254, 8 (2021) (Paper III)
  * Mikkola (1997) S. Mikkola, Celest. Mech. Dyn. Ast. 67, 145 (1997)
  * Wu et al. (2021) X. Wu, Y. Wang, W. Sun, F. Liu, Astrophys. J. 914, 63 (2021) (Paper IV)
  * Tancredi et al. (2001) G. Tancredi, A. Sánchez, F. Roig, Astron. J. 121, 1171 (2001)
  * Wu & Huang (2003) X. Wu, T.-Y. Huang, Phys. Lett. A 313, 77 (2003)
  * Froeschlé et al. (1997) C. Froeschlé, E. Lega, R. Gonczi, Celest. Mech. Dyn. Astron. 67, 41 (1997)
  * Froeschlé & Lega (1997) C. Froeschlé, E. Lega, Celest. Mech. Dyn. Astron. 78, 167 (2000)
  * Wu et al. (2006) X. Wu, T.-Y. Huang, H. Zhang, Phys. Rev. D 74, 083001 (2006)
  * Stuchlík & Kološ (2016) Z. Stuchlík, M. Kološ, Eur. Phys. J. C 76, 32 (2016)
  * Tursunov et al. (2016) A. Tursunov, Z. Stuchlík, M. Kološ, Phys. Rev D 93, 084012 (2016)
  * Mikkola & Tanikawa (1999) S. Mikkola, K. Tanikawa, Celest. Mech. Dyn. Ast. 74, 287 (1999)
  * Preto & Tremaine (1999) M. Preto, S. Tremaine, Astron. J. 118, 2532 (1999)

Table 1: Solutions for the methods S2 and IE2 solving Orbit 1 in Fig. 1(a). The two systems $H$ and $K$ almost have the same solutions in an integration time of $\tau=10000$ or $w=10000$.  | $w$ | $\tau$ | $r$ | $\theta$ | $p_{r}$ | $p_{\theta}$
---|---|---|---|---|---|---
$H$ | / | 1 | 11.0031 | 1.5867 | 7.6213e-3 | 1.9255
$K$ | 1 | 1 | 11.0031 | 1.5867 | 7.6203e-3 | 1.9255
$H$ | / | 10 | 11.3074 | 1.7235 | 7.3287e-2 | 1.7922
$K$ | 10 | 10.0002 | 11.3075 | 1.7235 | 7.3281e-2 | 1.7921
$H$ | / | $10^{2}$ | 26.2463 | 1.9136 | 0.2041 | -1.0078
$K$ | $10^{2}$ | 100.0094 | 26.2473 | 1.9136 | 0.2041 | -1.0079
$H$ | / | $10^{3}$ | 126.4088 | 1.5798 | 6.3895e-2 | -1.8227
$K$ | $10^{3}$ | 1000.0118 | 126.3912 | 1.5797 | 6.3868e-2 | -1.8226
$H$ | / | $10^{4}$ | 132.8861 | 1.5449 | -5.1566e-2 | -3.0532
$K$ | $10^{4}$ | 10000.1079 | 132.7308 | 1.5441 | -5.1730e-2 | -3.0572
$H$ | / | $10^{5}$ | 37.3004 | 1.1427 | -0.1763 | -1.4168
$K$ | $10^{5}$ | 100001.0807 | 33.7413 | 1.1251 | -0.1827 | -1.2180
$H$ | / | $10^{6}$ | 150.9389 | 1.6606 | 1.2518e-2 | -2.0503
$K$ | $10^{6}$ | 1000011.0250 | 151.8362 | 1.6616 | -2.7765e-3 | -2.0940
$H$ | / | $10^{7}$ | 142.0132 | 1.5745 | -3.5099e-2 | -3.0676
$K$ | $10^{7}$ | 10000110.1522 | 110.0560 | 1.6668 | 8.0709e-2 | -3.0845

Figure 1: (a) Hamiltonian errors $\Delta=K$ for the explicit symplectic
algorithm S2 and $\Delta=-H-1/2$ for the second-order implicit and explicit
mixed symplectic method IE2. The parameters are $E=0.995$, $L=4.6$, $a=0.5$
and $\beta=0.001$. Orbits 1 and 2 have their initial separations $r_{0}=11$
and 75, respectively. The other initial conditions are $\theta=\pi/2$,
$p_{r}=0$ and $p_{\theta}>0$ given by Eqs. (14) and (16). $h=1$ is a proper
time step for IE2 and a new coordinate time step for S2. Both algorithms make
the errors have no secular drift and remain of the same order for Orbits 1 and
2. (b) Poincaré sections on the plane $\theta=\pi/2$ and $p_{\theta}>0$, given
by S2. It is clear that Orbit 1 is regular, whereas Orbit 2 is chaotic.

Figure 2: Same as Fig. 1(b), but for different values of the magnetic
parameter $\beta$. An increase of $\beta$ results in strengthening the extent
of chaos. Orbit 3 with the initial separation $r_{0}=60$ in panel (d) is
weakly chaotic.

Figure 3: Same as Fig. 1(b), but for different values of the black hole’s
angular momentum $a$. Here, the magnetic parameter $\beta=1\times 10^{-3}$. An
increase of $a$ seems to weaken the extent of chaos from the global phase
space structure.

Figure 4: (a) Lyapunov exponents of three orbits including Orbits 1 and 2 in
Fig. 1 and Orbit 3 in Fig. 2(d). The Lyapunov exponent (colored red) of Orbit
2 calculated in the new coordinate time $w$ coincides with that (colored
black) of Orbit 2 computed in the proper time $\tau$. Orbit 1 is regular,
whereas Orbits 2 and 3 show the onset of chaos. (b) Fast Lyapunov indicators
(FLIs) of the three orbits. The FLI of Orbit 2 reaches 12 when the integration
time $w=6\times 10^{5}$, but is larger than 200 for the integration time
$w=10^{7}$. Thus, this orbit is strongly chaotic. Weak chaos of Orbit 3 can be
distinguished from regular Orbit 1 in terms of the growth of FLIs of the two
orbits during the integration time $w=10^{7}$.

Figure 5: Dependence of the values of FLIs on the magnetic parameter $\beta$.
The other parameters are $E=0.995$, $L=4.6$ and $a=0.5$. The initial
conditions $\theta=\pi/2$ and $p_{r}=0$ are the same in the two panels, but
the initial values $r_{0}$ and $p_{\theta}>0$ are different. For the strong
chaotic case, the integration ends when the FLI reaches 20, whereas the
integration time $w=10^{7}$ for the two other cases.

Figure 6: Same as Fig. 5, but the dependence of the values of FLIs on the
black hole spin $a$. Here, $\beta=0.001$.