11institutetext:  University College London, UK
22institutetext: NAAMII, Nepal 33institutetext: Redev Technology, UK
33email<EMAIL_ADDRESS>
#  Histogram of Oriented Gradients Meet Deep Learning: A Novel Multi-task Deep
Network for Medical Image Semantic Segmentation

Binod Bhattarai 11 Ronast Subedi∗ 22 Rebati Raman Gaire∗ 22 Eduard Vazquez 33
Danail Stoyanov 11

###### Abstract

We present our novel deep multi-task learning method for medical image
segmentation. Existing multi-task methods demand ground truth annotations for
both the primary and auxiliary tasks. Contrary to it, we propose to generate
the pseudo-labels of an auxiliary task in an unsupervised manner. To generate
the pseudo-labels, we leverage Histogram of Oriented Gradients (HOGs), one of
the most widely used and powerful hand-crafted features for detection.
Together with the ground truth semantic segmentation masks for the primary
task and pseudo-labels for the auxiliary task, we learn the parameters of the
deep network to minimise the loss of both the primary task and the auxiliary
task jointly. We employed our method on two powerful and widely used semantic
segmentation networks: UNet and U2Net to train in a multi-task setup. To
validate our hypothesis, we performed experiments on two different medical
image segmentation data sets. From the extensive quantitative and qualitative
results, we observe that our method consistently improves the performance
compared to the counter-part method. Moreover, our method is the winner of
FetReg Endovis Sub-challenge on Semantic Segmentation organised in conjunction
with MICCAI 2021. For the code and implementation details, please click here.

**footnotetext: Equal contributions

##  1 Introduction

Figure 1:  This Figure shows an input image (left) and its ground truth
semantic segmentation map (left) for the primary task and the Histogram of
Oriented Gradients of the input image (right). In the HOG map, we can observe
the boundary between the organs and the instruments that belong to different
semantic categories. Zoom in for a better view.

Medical image segmentation [18, 19, 31, 24] is an important and active
research problem. The usage of semantic segmentation in several biomedical
applications such as computer-assisted diagnosis [41], robotic surgery [5],
radiotherapy planning and follow-ups [20], etc., is growing day by day. Due to
this reason, the research community has witnessed an unprecedented growth of
research interest in this domain. There are several types of semantic
segmentation problems in medical imaging. Broadly, the existing semantic
segmentation tasks can be grouped into four major categories viz. organ
segmentation [13], robotic-instrument segmentation [21, 32], vessels
segmentation [7], and cellar and sub-cellular segmentation [28], etc.

After the seminal work of [15] on large-scale image classification using deep
convolutional neural networks, the use of deep architectures has not been
limited only on computer vision [33, 36, 11]; it is equally popular in medical
image analysis [35, 16]. With the usage of deep learning algorithms, the
accuracy of computer vision tasks such as classification, segmentation, and
detection is improving significantly [27]. A similar trend has been observed
on medical image analysis too [2]. We obtain the performance gain at the cost
of many annotated examples (e.g. Imagenet consists of 1M annotated examples).
It is evident that deep learning algorithms are data voracious and demand
millions of training examples. Collecting data, in general, is time-consuming,
needs experts and is also expensive. Moreover, in medical imaging, it is not
only about collecting annotations as they come from highly trained experts,
e.g. radiologists (e.g., MRI or CT scanner), but due to growing concerns on
privacy, it is difficult to get the unlabelled examples [23].

To improve the generalisation of a model from a fixed amount of training
examples, sharing the parameters between main task and auxiliary tasks [3] is
popular for a long time. MaskRCNN [12] , one of the most popular networks in
recent time, shares the parameters between detection and segmentation
networks. Similarly, [37] proposed to predict contour as an auxiliary task
while training a network for semantic segmentation as the primary task. The
major drawback of these methods is a need of annotated examples for both the
primary and the auxiliary tasks. Collecting such a heterogeneously labelled
set of training examples is even more challenging in the medical image domain.

To tackle the problem of collecting training examples with the heterogeneous
set of labels, we propose to generate pseudo-labels for the auxiliary task
from the hand-crafted features instead. As one can extract hand-crafted
features in an unsupervised manner, generating pseudo-labels of any type of
images for an auxiliary task can be done easily. To this end, we leverage the
Histogram of Oriented Gradients (HOGs) [6] to generate pseudo-labels.
Demarcation of the organs and surgical instruments parts belonging to a common
category from unrelated ones would play a significant role in their accurate
segmentation. Auxiliary tasks focusing on such aspects would help the network
to learn the robust representation for semantic segmentation. Thus, we chose
HOGs to generate pseudo-labels for the auxiliary task as these features are
carefully designed state-of-the-art hand-crafted features for object detection
[6]. However, any other type of hand-crafted features can be employed in our
pipeline to extract the pseudo-labels. Figure 1 shows the HOGs map of eye
anatomy and surgical instrument. In the Figure, we can see the demarcation of
a surgical instrument from eye anatomies made by the map of the Histogram of
Oriented Gradients. Once, we extract the HOGs, we consider these
representations as annotations of the auxiliary task and the ground truth
semantic map as annotations of the primary task. We extended existing popular
architectures for semantic segmentation: UNet [29] and U2Net [25] to minimise
the loss of both the auxiliary and primary tasks and train the network in a
multi-task manner.

Use of image feature representations as a pseudo-label is growing these days.
Recently, [8] trained a deep network to predict Bag of Visual Words (BoWs) for
image classification. Unlike ours, this method relied on the learned features
extracted from a network trained to minimize the image rotation angle loss. In
medical imaging, organs such as the eye bulb, pupils, colons, etc., are either
hollow and cylindrical or rotationally invariant. Hence, the pipeline is not
directly applicable in medical imaging. In addition, they trained their method
to minimise the objective function of a single task, whereas we train our
pipeline in a multi-task set-up. We summerise our contributions in the
following points:

  * •

We investigated the Histogram of Oriented Gradients to generate pseudo-labels
of images and exploited these representations as labels of an auxiliary task.

  * •

We extended existing semantic segmentation networks to train in a multi-task
framework.

  * •

We applied our method on two challenging medical semantic segmentation data
sets. Our extensive experiments demonstrate that our pipeline consistently
outperforms the counter-part single task networks.

##  2 Related Works

Our work falls into the category of deep multi-task learning with pseudo
labels, self-supervised learning. In this Section, we summarise some of the
important past works closely related to our method.

Deep Multi-task Learning for Semantic Segmentation: UNet [29] is one of the
earliest and the most widely used deep architectures for medical image
segmentation. This architecture is a supervised learning architecture and can
handle only semantic maps as the ground-truth annotations. Another work on
pancreas segmentation [30] trains deep learning architecture in a multi-stage
manner. It predicts the bounding box to localise the pancreas followed by
fine-tuned semantic segmentation. Unlike our approach, this method uses ground
truth annotations on both stages. In contrast, we rely on HOGs computed
unsupervised and trained the model to minimise the losses jointly. Another
work on brain lesion segmentation [14] employs 3D Convolutional Neural Network
with a fully connected Conditional Random Field. Similarly, [17] employ self
co-attention to improve the performance of anatomy segmentation in whole
breast ultrasound. However, these methods consider only semantic segmentation
maps for ground truth. One of the recent works on tumours segmentation in 3D
breast ultrasound images [42] proposed to train CNN in multitasking fashion.
[38] modified UNet architecture to jointly minimise the segmentation and
classification loss in ultra-sound images. [39] trained multi-stage multitask
learning framework for breast tumour segmentation in ultrasound images. [34]
learns the parameters of network to minimize the loss for skin lesion
detection, classification, and segmentation. [4] trained a multi-task learning
CNN for semantic segmentation and image level glaucoma classification. Another
work on histopathology image analysis [26] trained a multi-task network for
nucleus classification and segmentation. All of these methods need ground
truth annotations for both the main task (semantic segmentation) and auxiliary
tasks. Whereas, in our case, we have annotations for the primary task and
generate pseudo-labels for the auxiliary task.

Self-supervised Learning: In Self-supervised learning, the annotations for the
pre-text tasks are generated in an unsupervised manner. In general, the
parameters of a CNN are learned to minimise the loss of pre-text tasks
followed by fine-tuning of the parameters for the downstream tasks. Several
different ways are investigated in the past years to generate the annotations
of pre-text tasks. These includes, image rotation angle [9], colorization
[40], image-patch context [22], in-painting [22], etc. These methods mostly
pivot on the geometric transformations of the images. What kind of pre-text
task is going to be the most useful for the end-task is still an open research
problem. Recently, [8] proposed to learn the representations by predicting the
visual Bag of Words (BoW). This method, closest to ours, rely on visual
features to generate the pseudo-labels. As we mentioned before, they compute
BoWs from the visual representations extracted from model trained to minimise
the rotation angle of an image. Thus, this approach is not directly applicable
to our applications as most of the organs such as eyes, eye-bulb exhibit
rotationally invariant shape. Unlike most of the self-supervised pipeline, we
propose to minimise the loss of end-task and pre-text task jointly.

##  3 Proposed Method

In this Section, we present our pipeline in detail. We start with the
description of HOGs followed by the generation of pseudo-labels for the
auxiliary task. Afterwards, we explain our approach to extend a single-task
semantic segmentation network to a multi-task network. Finally, we explain the
overall objectives.

We have a scenario $\mathcal{X}\times\mathcal{Y}$ where $\mathcal{X}$
represents input image space and $\mathcal{Y}$ represents output semantic map
space. Our goal is to learn a function $f:\mathcal{X}\to\mathcal{Y}$ with a
given training examples
$T=\\{(x_{1},y_{1}),(x_{2},y_{2})\dots(x_{i},y_{i})\dots(x_{N},y_{N})\\}\subset\mathcal{X}\times\mathcal{Y}$.
In the training set $T$, $N$ is total number of training examples,
$x_{i}\in\mathbb{R}^{(W\times H\times C)}$ , $y_{i}\in\mathbb{R}^{(W\times
H)}$, where, $W,H,C$ represents width, height, and total number of channels in
an image respectively. Our contribution lies in generating extra annotations
of the images in an unsupervised way and extending the single task semantic
segmentation network to train in a multi-task manner to improve the
performance of semantic segmentation. We make use of HOGs to extract the
pseudo-annotations of an image.

###  3.1 Histogram of Oriented Gradients as Pseudo Labels

It is proven that the HOGs [6] were one of the most powerful hand-features on
computer vision and medical image analysis especially for detection before the
advent of data driven feature extraction methods such Alexnet [15], ResNet
[11], and UNet [29]. In this paper, we use HOGs for a novel cause i.e. to
extract the pseudo-labels of the images. To compute HOGs from an image, first
of all, we crop and resize the images to the desired dimensions of width, $W$
and height, $H$. We further divide the images into a non-overlapping image
patches of width $w$, and height $h$, resulting the total number of patches of
$\lfloor W/w\rfloor\times\lfloor H/h\rfloor$. For each of the patches, we run
1-D discrete derivative masks centred around a pixel in both the horizontal
and vertical directions. $d_{x}=[1,0,-1]$ and $d_{y}=[1,0,-1]^{T}$ are
horizontal and vertical filtering kernels respectively. We run these filters
on all the pixels of every image patches as shown in Figure 3.

Figure 2:  This diagram shows the overall proposed framework. In the Figure,
the main network corresponds to semantic segmentation network (e.g. U2Net),
while the auxiliary network is our contribution to extend the single task
network to a multi-task network. Training examples in triplet, i.e. input
image, ground truth semantic map and pseudo-label computed from HOGs, are fed
into the network and train the network jointly.

After applying the kernels centred on every pixels, we compute the histogram
of gradients for all the patches and append them together. Gradients are
computed as $\arctan(\frac{d_{y}}{d_{x}})$, and the gradients are assigned to
the nearest bin. The histogram can have $k$ number of bins with angle ranging
from 0 to 180 degrees. The magnitude of the gradient is computed as
$\sqrt{d_{x}^{2}+d_{y}^{2}}$. This magnitude of the gradients encodes the
frequency of a bin of the gradient taken into consideration. In this manner,
we estimate the histogram of oriented gradients in every patch. The number of
the bins and the patches determine the dimension of the HOGs and are the
hyper-parameters in our study. We present their studies in Experimental
Section in depth. We concatenate the HOGs for all the patches of an image, and
the final representations of HOGs are the pseudo-label, $y^{pl}$ of the image.
We augment the pseudo-label on the given training set. Thus, the training set
with augmented pseudo-labels become
$\\{(x_{i},y_{i},y_{i}^{pl})\\}_{i=1}^{i=N}$ which we use to train the
semantic segmentation network in multi-task setup.

###  3.2 Multi-task semantic segmentation with pseudo labels

For an input image $x$ with the ground truth semantic segmentation map $y$ and
its pseudo-label $y^{pl}$, we train a semantic segmentation network in a
multi-task learning fashion. The primary task for us is to predict the
semantic map and the secondary task is to regress the Histogram of Oriented
Gradients (HOGs). To predict the semantic map we employ categorical cross-
entropy loss and minimise mean squared loss to predict the HOGs. As mentioned
before, UNet and U2Net are two most popular and the powerful semantic
segmentation networks in medical imaging. However, these networks are
originally designed to support semantic map as only ground truth. Thus, these
networks can not readily handle our heterogeneously labelled training
examples. To enable them to handle pseudo-labels and share the parameters
between these tasks, we proposed to add a regression unit with two
convolutional layers and a fully connected layer on every layers of the
decoder side on U2Net as shown in the Figure 2. On UNet, we added only one
such unit on bottleneck. It is because, UNet has relatively less parameters
compared to U2Net. In the Figure 2, the left hand block depicts the U2Net
architecture and the right hand side block shows the regression units we
introduced in the architecture. The regression units learn the parameters
predicts HOGs correctly. In the similar manner, we plugged in regression units
on UNet. Compared to UNet, U2Net is also an hourglass architecture where each
layer consists of a UNet. We learn the parameters of the the whole
architecture to minimize the following objective.

Figure 3: Diagram showing the pipeline to extract the Histogram of Oriented Gradients (HOGs). Zoom in for better view. Input Shape | Operations
---|---
($3,h,h$) | Conv(3,3,1), ReLU(), MaxPool2d(2,2)
(3, $\frac{h}{2},\frac{h}{2}$) | Conv(3,3,1), ReLU(), MaxPool2d(2,2)
(3, $\frac{h}{4},\frac{h}{4}$) | Flatten()
($3\times\frac{h}{4}\times\frac{h}{4}$) | Linear(504)
Table 1: Architecture of the Auxiliary Task Network to Regress HOGs.

$L=\frac{1}{N}\sum_{i=1}^{i=N}\alpha L_{ce}(x_{i},y_{i})+\beta
L_{HOG}(x_{i},y_{i}^{pl})$ (1)

In Equation 1, $L_{ce}$ is the primary task loss i’e minimization of cross-
entropy loss to predict the ground truth mask correctly. Whereas, $L_{HOG}$ is
loss of secondary task to predict the HOGs of the input image. We minimize the
mean squared error between the predicted and ground truth HOGs. $\alpha$ and
$\beta$ are two hyper-parameters to weight the contributions of each of the
losses to best generalise the model parameters on unseen data for semantic
segmentation. We fine-tune these parameters by doing cross-valiation on
validation set. The details are on Section 4.

##  4 Experiments

Data sets: We evaluated our methods on two different publicly available
challenging data sets with diverse characterstics. CaDIS data set [10] was
released in MICCAI 2020 in one of the EndoVis challenges. It consists of 25
surgical videos. Each video frame is annotated broadly into eye anatomies,
surgical instruments, and miscellaneous categories. Based on the granularity
of the segments, [10] designed the challenge into three different tasks. Task
1 consists of 8 different segments: four eye anatomies, three misc objects,
and one instrument category. In Task 2, the instrument category is further
split into nine classes, resulting in 17 different categories. Finally, in
Task 3, there is an increase in granularity on the handles of the surgical
instrument. This further increase in granularity resulted in 25 different
categories to segment. There are 3,550 annotated frames in train set, 534 in
validation set, and 586 are in test set.

Another data set on which we evaluated our method is Robotic Instrument
Segmentation [1]. This data set is publicly available for research since
MICCAI 2017 challenge. The main task on this data set is to segment surgical
instruments from the background. Based on the granularity of segmentation of
the parts of the surgical instruments, three tasks were designed in the
challenge. Task 1 is to segment the instruments as a whole from the rest of
the background. Similarly, the challenge in Task 2 is to segment the
instruments into wrist, jaw, and shaft and distinguish the instrument from the
background. Finally, Task 3 further segments the instrument into seven parts
and segregates it from the background. There are 10 different folds of videos
in total. Following the evaluation protocol described on [1], we report
performance on folds 9 and 10 and train on rest of the videos.

Baselines Architectures: We took UNet [29] and U2Net [25], two representative
architectures, for semantic segmentation and employed our method on these two
architectures. Since our method is generic in nature, we can easily extend to
other architectures. UNet is one of the most widely used architectures in
medical image segmentation. It is a lightweight architectures consisting of
encoder and decoder. Encoder consists of convolutional and pooling layers that
map high-dimensional images into low-dimensional latent space. Decoder feeds
in the latent representations of the image and learns the parameters to
predict the correct semantic maps. There are skip connections from encoder
layers to decoder layers.

U2Net is another recently proposed architecture with state-of-the-art
performance on multiple computer vision semantic segmentation benchmarks.
Similar to UNet, this is an hourglass architecture with skip connections
between the encoder and decoder layers. Compared to UNet, U2Net consists of
UNet like structures in every layer of encoders and decoders and also known as
UNet inside UNet. Thus, the learning parameters in this architecture are much
higher than UNet.

Evaluation Metrics: We used mean Intersection of Union (mIoU) to compare the
quantitative performance. Intersection of Union (IoU) is computed as follows:

$\displaystyle\text{IoU}=\frac{\text{true positive}}{\text{true positive +
false positive + false negative}}$

In addition to this, we also present extensive qualitative analysis to make
the comparisons.

Implementation Details: We implemented our algorithms on PyTorch framework.
For optimization, we employ Adam Optimizer. We set the initial learning rate
to 2e-4 and scaled it by a factor of 0.5 in every 50k iteration. We train our
algorithms for 150K iterations and validate every 1k iterations. We save the
best performing checkpoint on the validation set and report the performance on
the test set.

Task | # Classes | Validation set mIOU | Test set mIOU
---|---|---|---
MICCAI’21 | U2Net | +HOG (Ours) | MICCAI’21 | U2Net | +HOG (Ours)
1 | 8 | 86.7 | 84.9 | 85.5 | 83.7 | 80.2 | 81.4
2 | 18 | 72.7 | 83.8 | 84.1 | 70.6 | 77.8 | 80.2
3 | 26 | 66.6 | 82.1 | 83.0 | 59.2 | 78.2 | 78.4
Table 2: Summary of quantitative performance comparison on CaDIS data set. Task | # Classes | mIOU on test Video 9 | mIOU on test Video 10
---|---|---|---
MICCAI’17 | U2Net | +HOG (Ours) | MICCAI’17 | +HOG (Ours) | Ours
1 | 2 | 87.7 | 94.2 | 95.6 | 91.7 | 96.0 | 96.2
2 | 4 | 73.6 | 70.8 | 75.8 | 80.7 | 84.1 | 84.4
3 | 8 | 35.7 | 57.9 | 65.4 | 79.1 | 89.4 | 91.3
Table 3: Summary of quantitative performance comparison on Robotic Instrument
Segmentation data set.

Hyper-parameter Selection: There are two critical sets of hyper-parameters in
our proposed pipeline. The first one is the weights of the primary loss
($\alpha$) and the secondary loss ($\beta$) as shown in Equation 1. Another
hyper-parameter is the dimension of HOGs. We estimated the values of these
hyper-parameters by doing cross-validation on Validation Set. Table 4
summarises the cross-validation for weighing the contributions of the proposed
losses on CaDIS validation set. We observed that setting equal contribution to
the losses gives us optimal performance. We observed a similar trend on
another benchmark too. This outcome also highlights the significance of the
proposed auxiliary loss in our pipeline. We set the values of $\alpha$ and
$\beta$ equal to 1 in the rest of the experiments. Similarly, Figure 5 shows
the performance on CaDIS Validation Set with varying the dimension of the
HOGs. We can see the highest performance with the dimension of 502, which we
set for the rest of the experiments.

Weight of losses | mIOU
---|---
$\alpha$ | $\beta$ |
0.01 | 1.0 | 81.2
0.1 | 1.0 | 82.1
1.0 | 1.0 | 82.3
1.0 | 0.1 | 81.7
1.0 | 0.01 | 82.0
Table 4: Ablation study on weights of losses. Figure 4:  A performance
comparison with varying sizes of training data. Figure 5: A performance
comparison with varying dimensions of HOGs. Figure 6: Qualitative comparison
between the proposed method with its counter-part architecture U2Net on three
different tasks. First two rows represent examples from Task 1, the middle two
rows, and the last two rows are examples from Task 2 and Task 3 respectively.
Figure 7: Qualitative comparison between before and after applying our method
on U2Net in the Task 1 of robotic instrument segmentation challenge held in
MICCAI 2017. Figure 8: Qualitative comparison between before and after
applying our method on U2Net in the Task 2 of robotic instrument segmentation
challenge held in MICCAI 2017.  Figure 9: Qualitative comparison between
before and after applying our method on U2Net in the Task 3 of robotic
instrument segmentation challenge held in MICCAI 2017.

Quantitative Evaluations: Here, we present the outcomes from our extensive
experiments on two different data sets: CaDIS and Robotic Instrument
Segmentation. As mentioned before, each of the benchmarks consists of three
tasks resulting in six different tasks from two data sets. We extended our
method on two popular baseline architectures: UNet and U2Net. We evaluate the
empirical performance on the mean Intersection of Union (mIOU).

Compared to U2Net, UNet is more efficient but is less accurate. We evaluate
both the architectures on CaDIS Task 2. We choose this task due to the good
trade-off of granularity and the number of training examples per category. In
this experiment, UNet and U2Net obtained 81.9% and 83.75% mIOU, respectively.
We also took a different proportion of training examples and compared the
performance of UNet with/out the auxiliary task to predict HOGs. Figure 4
summerizes our experiments. Our technique to extend UNet to a multi-task
network improves the performance consistently. This gain in performance also
shows that our method equally generalizes on varying sizes of training
examples. For experiments on the remaining tasks from both the data sets, we
decided to choose U2Net as our baseline architecture as its performance is
clearly superior to UNet.

Table 2 summarises the performances of three different tasks on CaDIS data
set. We have compared our performance with the winner of the MICCAI 2021
challenge and U2Net. From the Table, we can see that our method consistently
outperforms the U2Net on both the validation set and the set. Out of 6
different scenarios, our method obtained the highest mIOU on 4 cases, slightly
lagging behind the winner of MICCAI’21 challenge on Task 1. Compared to Task1,
on Task 2 and Task 3, the mIOU of the winning method on MICCAI’21 dropped by a
large margin (-20%). In contrast, our cases have a slight drop in performance
(-2.0%). This shows the robustness of the proposed pipeline over the increase
in granularity of the segmentation tasks.

Similarly, Table 3 details the performance comparison on Robotic Instrument
Segmentation. We followed the evaluation protocol presented on the challenge
paper and compared our performance with the winner model. In every task, our
method obtained the highest mIOU surpassing the winning team’s performance and
our baseline U2Net by a large margin. With the increase in the granularity in
the segmentation task, the mIOU of the winner method drops by up to -50%. At
the same time, the drop in our method is only up to -30.2%. Again, this is yet
another evidence for our method being robust compared to the contemporary
methods.

Qualitative Evaluations: We did not limit our experiments to quantitative
evaluations only. To deeper understand our method’s role in improving the
performance of existing architecture such as U2Net, we performed an extensive
qualitative analysis. Figure 6 shows the qualitative comparisons of Task 1 ,
Task 2, and Task 3 on CaDIS data set. The bounding boxes locate some of the
representative regions on the eye and the surgical instrument where U2Net
fails, but our method correctly segments it. From these locations, we can see
that the characteristics of HOGs to identify the organs and tools boundary
play a crucial role in correctly segmenting the organs and the semantic parts
of the surgical tools.

Similarly, Figure 7, Figure 8, Figure 9 show the qualitative comparison of
Task 1, Task 2, and Task 3 on robotic instrument segmentation. In these
qualitative analyses, we observe the similar trends that were seen on CaDIS
data set. As we can see from these analysis, U2Net struggles quite a lot on
boundary regions. Our method enables correct segmentation on such regions that
we can see in our qualitative comparisons. The red bounding boxes on the
Figures locates the failed cases by the baseline, whereas the green bounding
boxes show the correction made by our method.

##  5 Conclusions and Future Works

In this paper, we present a novel multi-task deep learning framework for
medical image segmentation. We generate the annotations of the auxiliary task
in an unsupervised manner. We leverage Histogram of Oriented Gradients of
images as their labels. We train the deep network jointly to minimise the
losses of both the primary task, which is semantic segmentation and the
auxiliary task. From our extensive qualitative and quantitative experiments on
two challenging medical image segmentation benchmarks, we observe the proposed
pipeline’s performance superior to its counter-part single task network. In
the future work, we plan to explore the higher-order statistics of hand-
crafted features such as Fisher Vectors as annotation of images to train the
multi-task deep semantic network.

##  6 Acknowledgement

This project is funded by the EndoMapper project by Horizon 2020 FET (GA
863146). For the purpose of open access, the author has applied a CC BY public
copyright licence to any author accepted manuscript version arising from this
submission.

## References

  * [1] Allan, M., Shvets, A., Kurmann, T., Zhang, Z., Duggal, R., Su, Y.H., Rieke, N., Laina, I., Kalavakonda, N., Bodenstedt, S., et al.: 2017 robotic instrument segmentation challenge. arXiv preprint arXiv:1902.06426 (2019)
  * [2] Anwar, S.M., Majid, M., Qayyum, A., Awais, M., Alnowami, M., Khan, M.K.: Medical image analysis using convolutional neural networks: a review. Journal of medical systems 42(11), 1–13 (2018)
  * [3] Caruana, R.: Multitask learning. Machine learning 28(1), 41–75 (1997)
  * [4] Chakravarty, A., Sivswamy, J.: A deep learning based joint segmentation and classification framework for glaucoma assesment in retinal color fundus images. arXiv preprint arXiv:1808.01355 (2018)
  * [5] Colleoni, E., Edwards, P., Stoyanov, D.: Synthetic and real inputs for tool segmentation in robotic surgery. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 700–710. Springer (2020)
  * [6] Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: CVPR (2005)
  * [7] Fraz, M.M., Remagnino, P., Hoppe, A., Uyyanonvara, B., Rudnicka, A.R., Owen, C.G., Barman, S.A.: Blood vessel segmentation methodologies in retinal images–a survey. Computer methods and programs in biomedicine 108(1), 407–433 (2012)
  * [8] Gidaris, S., Bursuc, A., Komodakis, N., Pérez, P., Cord, M.: Learning representations by predicting bags of visual words. In: CVPR. pp. 6928–6938 (2020)
  * [9] Gidaris, S., Singh, P., Komodakis, N.: Unsupervised representation learning by predicting image rotations. In: ICLR (2018)
  * [10] Grammatikopoulou, M., Flouty, E., Kadkhodamohammadi, A., Quellec, G., Chow, A., Nehme, J., Luengo, I., Stoyanov, D.: Cadis: Cataract dataset for surgical rgb-image segmentation. Medical Image Analysis (2021)
  * [11] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR (2016)
  * [12] He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: ICCV (2017)
  * [13] Hu, P., Wu, F., Peng, J., Bao, Y., Chen, F., Kong, D.: Automatic abdominal multi-organ segmentation using deep convolutional neural network and time-implicit level sets. International journal of computer assisted radiology and surgery 12(3), 399–411 (2017)
  * [14] Kamnitsas, K., Ledig, C., Newcombe, V.F., Simpson, J.P., Kane, A.D., Menon, D.K., Rueckert, D., Glocker, B.: Efficient multi-scale 3d cnn with fully connected crf for accurate brain lesion segmentation. Medical image analysis 36, 61–78 (2017)
  * [15] Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. In: NIPS (2012)
  * [16] Lee, J.G., Jun, S., Cho, Y.W., Lee, H., Kim, G.B., Seo, J.B., Kim, N.: Deep learning in medical imaging: general overview. Korean journal of radiology 18(4), 570–584 (2017)
  * [17] Lei, B., Huang, S., Li, H., Li, R., Bian, C., Chou, Y.H., Qin, J., Zhou, P., Gong, X., Cheng, J.Z.: Self-co-attention neural network for anatomy segmentation in whole breast ultrasound. Medical image analysis 64, 101753 (2020)
  * [18] Lei, T., Wang, R., Wan, Y., Du, X., Meng, H., Nandi, A.K.: Medical image segmentation using deep learning: A survey. arXiv e-prints (2020)
  * [19] Milletari, F., Navab, N., Ahmadi, S.A.: V-net: Fully convolutional neural networks for volumetric medical image segmentation. In: 2016 fourth international conference on 3D vision (3DV). pp. 565–571. IEEE (2016)
  * [20] Nemoto, T., Futakami, N., Yagi, M., Kunieda, E., Akiba, T., Takeda, A., Shigematsu, N.: Simple low-cost approaches to semantic segmentation in radiation therapy planning for prostate cancer using deep learning with non-contrast planning ct images. Physica Medica 78, 93–100 (2020)
  * [21] Pakhomov, D., Premachandran, V., Allan, M., Azizian, M., Navab, N.: Deep residual learning for instrument segmentation in robotic surgery. In: International Workshop on Machine Learning in Medical Imaging. pp. 566–573. Springer (2019)
  * [22] Pathak, D., Krahenbuhl, P., Donahue, J., Darrell, T., Efros, A.A.: Context encoders: Feature learning by inpainting. In: CVPR (2016)
  * [23] Peng, J., Wang, P., Desrosiers, C., Pedersoli, M.: Self-paced contrastive learning for semi-supervised medical image segmentation with meta-labels. In: NeurIPS (2021)
  * [24] Pham, D.L., Xu, C., Prince, J.L.: Current methods in medical image segmentation. Annual review of biomedical engineering 2(1), 315–337 (2000)
  * [25] Qin, X., Zhang, Z., Huang, C., Dehghan, M., Zaiane, O.R., Jagersand, M.: U2-net: Going deeper with nested u-structure for salient object detection. Pattern Recognition 106, 107404 (2020)
  * [26] Qu, H., Riedlinger, G., Wu, P., Huang, Q., Yi, J., De, S., Metaxas, D.: Joint segmentation and fine-grained classification of nuclei in histopathology images. In: 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019). pp. 900–904 (2019)
  * [27] Rawat, W., Wang, Z.: Deep convolutional neural networks for image classification: A comprehensive review. Neural computation 29(9), 2352–2449 (2017)
  * [28] Rizk, A., Paul, G., Incardona, P., Bugarski, M., Mansouri, M., Niemann, A., Ziegler, U., Berger, P., Sbalzarini, I.F.: Segmentation and quantification of subcellular structures in fluorescence microscopy images using squassh. Nature protocols 9(3), 586–596 (2014)
  * [29] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: MICCAI (2015)
  * [30] Roth, H.R., Lu, L., Lay, N., Harrison, A.P., Farag, A., Sohn, A., Summers, R.M.: Spatial aggregation of holistically-nested convolutional neural networks for automated pancreas localization and segmentation. Medical image analysis 45, 94–107 (2018)
  * [31] Sharma, N., Aggarwal, L.M.: Automated medical image segmentation techniques. Journal of medical physics/Association of Medical Physicists of India 35(1), 3 (2010)
  * [32] Shvets, A.A., Rakhlin, A., Kalinin, A.A., Iglovikov, V.I.: Automatic instrument segmentation in robot-assisted surgery using deep learning. In: 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA). pp. 624–628. IEEE (2018)
  * [33] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. In: ICLR (2015)
  * [34] Song, L., Lin, J., Wang, Z.J., Wang, H.: An end-to-end multi-task deep learning framework for skin lesion analysis. IEEE journal of biomedical and health informatics (2020)
  * [35] Suzuki, K.: Overview of deep learning in medical imaging. Radiological physics and technology 10(3), 257–273 (2017)
  * [36] Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., Rabinovich, A.: Going deeper with convolutions. In: CVPR (2015)
  * [37] Takikawa, T., Acuna, D., Jampani, V., Fidler, S.: Gated-scnn: Gated shape cnns for semantic segmentation. In: ICCV (2019)
  * [38] Wang, P., Patel, V.M., Hacihaliloglu, I.: Simultaneous segmentation and classification of bone surfaces from ultrasound using a multi-feature guided cnn. In: MICCAI (2018)
  * [39] Xie, X., Shi, F., Niu, J., Tang, X.: Breast ultrasound image classification and segmentation using convolutional neural networks. In: Pacific rim conference on multimedia (2018)
  * [40] Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: ECCV (2016)
  * [41] Zhao, Y.Y., Xue, D.X., Wang, Y.L., Zhang, R., Sun, B., Cai, Y.P., Feng, H., Cai, Y., Xu, J.M.: Computer-assisted diagnosis of early esophageal squamous cell carcinoma using narrow-band imaging magnifying endoscopy. Endoscopy 51(04), 333–341 (2019)
  * [42] Zhou, Y., Chen, H., Li, Y., Liu, Q., Xu, X., Wang, S., Yap, P.T., Shen, D.: Multi-task learning for segmentation and classification of tumors in 3d automated breast ultrasound images. Medical Image Analysis 70, 101918 (2021)


# Nonlinear Topological Magnon Spin Hall Effect

Zhejunyu Jin1 Xianglong Yao1 Zhenyu Wang1 H. Y. Yuan2 Zhaozhuo Zeng1 Yunshan
Cao1 Peng Yan1<EMAIL_ADDRESS>1School of Electronic Science and Engineering
and State Key Laboratory of Electronic Thin Films and Integrated Devices,
University of Electronic Science and Technology of China, Chengdu 610054,
China
2Institute for Theoretical Physics, Utrecht University, 3584 CC Utrecht, The
Netherlands

###### Abstract

When a magnon passes through two-dimensional magnetic textures, it will
experience a fictitious magnetic field originating from the $3\times 3$ skew-
symmetric gauge fields. To date, only one of the three independent components
of the gauge fields has been found to play a role in generating the fictitious
magnetic field while the rest two are perfectly hidden. In this work, we show
that they are concealed in the nonlinear magnon transport in magnetic
textures. Without loss of generality, we theoretically study the nonlinear
magnon-skyrmion interaction in antiferromagnets. By analyzing the scattering
features of three-magnon processes between the circularly-polarized incident
magnon and breathing skyrmion, we predict a giant Hall angle of both the
confluence and splitting modes. Furthermore, we find that the Hall angle
reverses its sign when one switches the handedness of the incident magnons. We
dub it nonlinear topological magnon spin Hall effect. Our findings are deeply
rooted in the bosonic nature of magnons that the particle number is not
conserved, which has no counterpart in low-energy fermionic systems, and may
open the door for probing gauge fields by nonlinear means.

Introduction.—Topology dictates the particle or wave transport in many
branches of physics, ranging from solid state physics to geophysics and
astrophysics Delplace2017 ; Parker2020 . One outstanding example in condensed
matter physics is the intrinsic spin Hall effect which originates from the
momentum-space topology, i.e., the Berry curvature of the band structure
Kato2004 ; Wunderlich2005 ; Sinova2015 ; Kavokin2005 ; Leyder2007 ; Onose2010
; Sheng2020 ; Sinova2004 ; Guo2008 . On the other hand, non-collinear spin
textures, such as the magnetic vortex, meron, and skyrmion, can give rise to
the real-space topology. When a spinful particle propagates through the
topological spin texture, it will experience an effective Lorentz force,
resulting in the so-called topological (spin-) Hall effect Yin2015 ; Mook2017
; Akosa2018 ; Daniels2019 ; Kim2019 ; Du2022 .

Magnons, quanta of spin waves, are the collective excitations of ordered
magnets Yan2011 ; Yuan2022 . Very recently, magnon-based spintronics has
attracted enormous interest due to peculiar advantages of magnons, such as the
long-distance transport and low-energy consumption. Magnons carry spin angular
momentum as well, so that they can experience an effective Lorentz force from
the spin texture, leading to the topological magnon Hall effect Dugaev2005 ;
Kovalev2012 ; Hoogdalem2013 ; rainbow ; Lan2021 . In antiferromagnets, magnons
have two degenerate modes with opposite spins, i.e., right- and left-handed
magnons Wangst2022 . Therefore, when a magnon passes through the
antiferromagnetic (AFM) skyrmion, for instance, it will experience a spin-
dependent Lorentz force, resulting in the topological magnon spin Hall effect
Daniels2019 ; Jin2021 ; Liu2022 . These topological magnon Hall effects
originate from the gauge fields in transforming the non-collinear magnetic
texture to the collinear state. The gauge transformation generates the
covariant form of the differential operator $\partial_{\mu}+\mathcal{A}_{\mu}$
with $\mu=x,y$. Here, the $3\times 3$ skew-symmetric matrix
$\mathcal{A}_{\mu}=\mathcal{R}^{-1}\partial_{\mu}\mathcal{R}$ with the
rotation matrix $\mathcal{R}$ contains three independent gauge fields Tan2020
; Tatara2019 . So far, only one of the three elements, i.e.,
$\mathcal{A}_{\mu,12}$, has been identified to play a role in the Hall
transport of magnons while the rest two ($\mathcal{A}_{\mu,13}$ and
$\mathcal{A}_{\mu,23}$) are concealed from the community.

In the past few years, the nonlinear Hall effect due to the momentum-space
topology, e.g., Berry curvature dipole, has attracted much attention
Sodemann2015 ; Ma2019 ; Kang2019 ; Du2018 ; He2019 ; Shao2020 ; Lai2021 ;
Du2021 ; Duan2022 ; Itahashi2022 ; Mook2018 ; Kondo2022 . However, its
counterpart induced by the real-space topology has not been reported till now.
It is well known that the geometric phase derived from the adiabatic evolution
is crucial for the Hall transport. Notably, in the three-wave mixing process,
the accumulation of adiabatic geometric phase takes place not only on incident
waves but also on nonlinear ones Tymchenko2015 ; Li2015 ; Li2020 . One thus
expects that magnons generated in the nonlinear three-magnon process in spin
textures Wang2021 ; Wang2022 ; Schultheiss2009 ; Iwasaki2014 ; Zhang2018 ;
Schultheiss2019 ; Korber2020  may also experience a topological Hall effect
subject to the conventional gauge field, but it is not clear whether the rest
two gauge fields play any role.

In this Letter, we aim to reveal the concealed gauge fields by addressing the
nonlinear Hall transport of magnons in spin textures. To this end, we
theoretical study the nonlinear interaction between polarized magnons and
magnetic skyrmions in antiferromagnets. We show that the two long-sought gauge
fields are actually hidden in the nonlinear magnon transport. By analyzing the
“bunny ears” scattering pattern of three-magnon processes between the
circularly-polarized magnon and breathing skyrmion in an antiferromagnet, we
discover a giant Hall angle of both confluence and splitting modes. The Hall
angle reverses its sign when one switches the handedness of incident AFM
magnons. We dub it nonlinear topological magnon spin Hall effect. Our findings
are deeply connected to both the nonconservation of magnon number and the
spin-texture-induced Berry curvature in real space, as shown in Fig. 1.

Figure 1: Schematic illustration of the nonlinear topological magnon spin Hall
effect in magnon-AFM skyrmion scattering. Circles with arrows indicate the
handedness of AFM magnons. Incident, skyrmion breathing, sum-frequency, and
difference-frequency modes are denoted by black, green, red, and blue colors,
respectively. ${\bf v}_{s,p,q}$ represent the velocity of three propagating
magnon wavepackets. The bottom panel shows the magnon splitting (left) and
confluence (right) processes. It is noted that magnons with opposite
handedness will experience magnitude-equal but opposite Lorentz forces,
resulting in the opposite transverse displacement (not shown).

Model.—Let us consider a chiral antiferromagnet described by the following
Lagrangian Kim2019

$\displaystyle\mathcal{L}=\int(\partial_{t}{\bf l})^{2}d{\bf r}-\mathcal{H},$
(1)

where ${\bf l}$ is the normalized N$\rm{\acute{e}}el$ vector and
$\mathcal{H}=\int\big{[}J(\nabla{\bf l})^{2}+D{\bf l}\cdot(\nabla\times{\bf
l})-Kl_{z}^{2}\big{]}d{\bf r}$ is the system Hamiltonian including the
exchange energy, Dzyaloshinskii-Moriya interaction (DMI), and magnetic
anisotropy, with $J$, $D$, and $K$ being the exchange stiffness, DMI strength,
and anisotropy coefficient, respectively. To facilitate the analysis, we use
the $3\times 3$ matrix $\mathcal{R}=$exp$(\phi L_{z})$exp$(\theta L_{y})$ to
rotate the ${z}-$axis to the equilibrium direction of the stagger vector ${\bf
l}_{0}$, i.e., $\mathcal{R}{\bf e}_{z}={\bf
l}_{0}=(\sin\theta\cos\phi,\sin\theta\sin\phi,\cos\theta)$ with the polar
angle $\theta$ and azimuthal angle $\phi$. Here, $L_{z}$ and $L_{y}$ are
generators of the three-dimensional rotations about the $z$ and $y$ axis,
respectively. To investigate the magnon excitation and transport in spin
textures, we introduce the magnon creation ($a^{\dagger}$) and annihilation
($a$) operators by the Holstein-Primakoff transformation on the vector ${\bf
n}=\mathcal{R}^{-1}{\bf l}$ HP . We expand the bosonic operator as
$a=a_{s}e^{i{\bf k}_{s}\cdot{\bf r}}+a_{p}e^{i{\bf k}_{p}\cdot{\bf
r}}+a_{q}e^{i{\bf k}_{q}\cdot{\bf r}}+a_{r}\psi_{r}$, where
$a_{s},a_{p},a_{q},$ and $a_{r}$ are operators of incident magnon, sum-
frequency, difference-frequency, and the skyrmion breathing modes
Mochizuki2012 , respectively. ${\bf k}_{s},$ ${\bf k}_{p},$ and ${\bf k}_{q}$
are the corresponding wave vectors of three propagating modes in the far-field
region, and $\psi_{r}$ is the wavefunction of the localized breathing mode.
Furthermore, we assume that the magnon excitation is in the form of a wave
packet, which has a fixed shape and can be described by its central position
${\bf r}_{i}(t)$, with $i=s,p,q$. In terms of these collective coordinates
Thiele , the Lagrangian can be simplified as a function of the position ${\bf
r}_{i}$ and the group velocity ${\bf v}_{i}=\dot{\bf r}_{i}$ of the magnon
wavepacket Lan2021 . Keeping up to third-order terms, the total Hamiltonian
can be expressed as $\mathcal{H}=\mathcal{H}_{2}+\mathcal{H}_{3}$. Here, the
quadratic Hamiltonian is

$\mathcal{H}_{2}=\sum_{i=s,p,q}2a_{i}^{\dagger}a_{i}\int\Big{[}{\frac{1}{J}\omega_{i}^{2}{\bf
v}_{i}^{2}}-(2{\bf A}_{12}+\frac{D{\bf l}_{0}}{J})\cdot\omega_{i}{\bf
v}_{i}\Big{]}d{\bf r},\\\ $ (2)

which determines the magnon dispersion relation. The cubic Hamiltonian
$\mathcal{H}_{3}=\mathcal{H}_{3s}+\mathcal{H}_{3p}+\mathcal{H}_{3q}$ includes
contributions from the incident term $\mathcal{H}_{3s}$, sum-frequency term
$\mathcal{H}_{3p}$, and difference-frequency term $\mathcal{H}_{3q}$

$\displaystyle\mathcal{H}_{3s}=$ $\displaystyle\int\omega_{s}{\bf
v}_{s}\cdot\bigg{\\{}-\frac{1}{\sqrt{2}}\big{[}(i{\bf A}_{13}+{\bf
A}_{23})+\frac{D}{2J}(i{\bf e}_{\phi}+{\bf e}_{\theta})\big{]}$ (3)
$\displaystyle\big{[}3a_{q}a_{r}a_{s}^{\dagger}e^{i(-{\bf k}_{s}+{\bf
k}_{q})\cdot{\bf r}}+a_{p}^{\dagger}a_{r}a_{s}e^{i({\bf k}_{s}-{\bf
k}_{p})\cdot{\bf r}}\big{]}\psi_{r}+{\rm H.c.}\bigg{\\}}d{\bf r},$
$\displaystyle\mathcal{H}_{3p}=$ $\displaystyle\int\omega_{p}{\bf
v}_{p}\cdot\bigg{\\{}-\frac{3}{\sqrt{2}}\big{[}(i{\bf A}_{13}+{\bf
A}_{23})+\frac{D}{2J}(i{\bf e}_{\phi}+{\bf e}_{\theta})\big{]}$ $\displaystyle
a_{s}a_{r}a_{p}^{\dagger}e^{i({\bf k}_{s}-{\bf k}_{p})\cdot{\bf
r}}\psi_{r}+{\rm H.c.}\bigg{\\}}d{\bf r},$ $\displaystyle\mathcal{H}_{3q}=$
$\displaystyle\int\omega_{q}{\bf
v}_{q}\cdot\bigg{\\{}-\frac{1}{\sqrt{2}}\big{[}(i{\bf A}_{13}+{\bf
A}_{23})+\frac{D}{2J}(i{\bf e}_{\phi}+{\bf e}_{\theta})\big{]}$ $\displaystyle
a_{s}a_{r}^{\dagger}a_{q}^{\dagger}e^{i({\bf k}_{s}-{\bf k}_{q})\cdot{\bf
r}}\psi_{r}^{*}+{\rm H.c.}\bigg{\\}}d{\bf r},$

where ${\bf A}_{\nu\nu^{\prime}}=\mathcal{A}_{x,\nu\nu^{\prime}}{\bf
e}_{x}+\mathcal{A}_{y,\nu\nu^{\prime}}{\bf e}_{y}$ are the gauge fields
($\nu,\nu^{\prime}=1,2,3$), ${\bf e}_{\theta}$ and ${\bf e}_{\phi}$ are two
unit vectors in spherical coordinates, and $\omega_{s},$ $\omega_{p},$ and
$\omega_{q}$ are, respectively, the frequencies of the incident, confluence,
and splitting magnons meeting law of energy conservation, i.e.,
$\omega_{p(q)}=\omega_{s}\pm\omega_{r}$ with $\omega_{r}$ the skyrmion
breathing frequency, see the bottom panel of Fig. 1. Equations (2) and (3)
show that the conventional gauge field ${\bf A}_{12}$ only appears in the
quadratic term, while gauge fields ${\bf A}_{13}$ and ${\bf A}_{23}$ emerge in
the nonlinear three-magnon processes. To reveal their role in the magnon
transport, we employ the Euler-Lagrangian formula to derive equations of
motion of magnon wavepackets

$\displaystyle a_{i}^{\dagger}a_{i}\frac{\hbar\omega_{i}^{2}}{eJ}\dot{{\bf
v}_{i}}-a_{i}^{\dagger}a_{i}\omega_{i}{\bf v}_{i}\times{\bf B}-{\bf
F}_{i}^{{\rm cubic}}=0,(i=s,p,q),$ (4)

where ${\bf B}=B_{z}{\bf e}_{z}$ with
$B_{z}=\frac{\hbar}{e}\big{[}{\bf\nabla}\times({\bf A}_{12}+\frac{D{\bf
l}_{0}}{2J})\big{]}_{z}=\frac{\hbar}{e}\big{[}{\bf
l}_{0}\cdot(\partial_{x}{\bf l}_{0}\times\partial_{y}{\bf
l}_{0})+({\bf\nabla}\times\frac{D{\bf l}_{0}}{2J})_{z}\big{]}$ is the
conventional fictitious magnetic field for the linear magnon transport with
the reduced Planck constant $\hbar$ and elementary charge $e$. See
Supplemental Material SM  for the derivation of (4). It is noted that we have
ignored the effective electric field associated with the skyrmion static
energy due to its negligible role in the magnon Hall effect. The extra force
${\bf F}_{i}^{{\rm cubic}}$ originates from the three-magnon process and the
newfound gauge fields, with the following expression

$\displaystyle{\bf F}_{i}^{{\rm cubic}}=c_{i}{\bf v}_{i}\times{\bf
B}^{\prime},(i=s,p,q),$ (5)

where ${\bf B}^{\prime}=B_{z}^{\prime}{\bf e}_{z}$ with
$B_{z}^{\prime}=\frac{\hbar}{e}({\bf\nabla}\times{\bf A}_{23})_{z}+\frac{\hbar
D}{2Je}({\bf\nabla}\times{\bf
e}_{\theta})_{z}=\frac{\hbar}{e}\big{[}\partial_{y}{\bf
l}_{0}\cdot\partial_{x}({\bf n}\times\frac{{\bf
e}_{z}}{\sin\theta})-\partial_{x}{\bf l}_{0}\cdot\partial_{y}({\bf
n}\times\frac{{\bf e}_{z}}{\sin\theta})\big{]}+\frac{\hbar
D}{2Je}(\nabla\times\frac{{\bf e}_{z}+\cos\theta{\bf l}_{0}}{\sin\theta})_{z}$
represents the new fictitious magnetic field playing a role merely when the
nonlinear three-magnon process occurs. Due to the circular symmetry of
skyrmion, the ${\bf A}_{13}$ component is absent. Here,
$c_{s}=\frac{\omega_{s}}{4}(g_{p}a_{p}^{\dagger}a_{r}a_{s}+3g_{q}a_{q}^{\dagger}a_{r}^{\dagger}a_{s}+{\rm
H.c.})$, $c_{p}=\frac{3\omega_{p}}{4}(g_{p}a_{s}a_{r}a_{p}^{\dagger}+{\rm
H.c.})$, and
$c_{q}=\frac{\omega_{q}}{4}(g_{p}a_{s}a_{r}^{\dagger}a_{q}^{\dagger}+{\rm
H.c.})$, with overlap integrals $g_{p}=\frac{1}{\sqrt{2}V}\int e^{i({\bf
k}_{s}-{\bf k}_{p})\cdot{\bf r}}\psi_{r}d{\bf r}$,
$g_{q}=\frac{1}{\sqrt{2}V}\int e^{i({\bf k}_{s}-{\bf k}_{q})\cdot{\bf
r}}\psi_{r}^{*}d{\bf r}$, and $V$ being the system volume. As shown in Eq.
(4), the spin-wave packet can be regarded as a particle-like object moving in
its own parameter space Sundaram1999  subject to fictitious magnetic fields
(${\bf B}$ and ${\bf B}^{\prime}$). The first term on the left-hand side of
Eq. (4) characterizes the acceleration of magnons. The second term represents
the effective Lorentz force from the quadratic Hamilton $\mathcal{H}_{2}$,
resulting in the conventional topological magnon Hall effect. Interestingly
enough, the third term induces an extra Lorentz force on the wavepacket,
leading to the nonlinear topological magnon Hall effect. The spatial
distributions of the dimensionless magnetic fields $B_{z}/B_{0}$ and
$B^{\prime}_{z}/B_{0}$ are shown in Figs. 2(a) and 2(b), respectively, where
$B_{0}=\hbar/{a^{2}e}$ with $a$ being the lattice constant. It is noted that
$B_{0}\approx 660$ T for $a=1$ nm. Due to the rotational symmetry of the Bloch
skyrmion, both magnetic fields ${\bf B}/B_{0}$ and ${\bf B}^{\prime}/B_{0}$
have the circular symmetry. For ${\bf B}/B_{0}$, its main origin comes from
the topological charge density of skyrmion, and the total magnetic flux is
4$\pi$ Volovik1987 . The spatial distribution of ${\bf B}^{\prime}/B_{0}$,
however, is similar to the fictitious magnetic field distribution of the
target skyrmion Tang2021 ; Ndiaye2017  with a vanishing total flux but a
singularity at the skyrmion core.

Revealing the concealed fictitious magnetic field.—In nonlinear magnon-
skyrmion scatterings, the time-evolution of populations of confluence and
splitting modes is governed by the coupled Heisenberg equations:
$i\dot{a}_{p}=(\Delta_{p}-i\alpha\omega_{p})a_{p}+\tilde{g}_{p}a_{s}a_{r}$ and
$i\dot{a}_{q}=(\Delta_{q}-i\alpha\omega_{q})a_{q}+\tilde{g}_{q}a_{s}a_{r}^{\dagger}$.
Here, the detuning parameter $\Delta_{p(q)}=\omega_{p(q)}-\omega_{0}$ with the
driving microwave frequency $\omega_{0}$,
$\tilde{g}_{p}=\int\big{[}-2g_{1,\mu}ik_{p,\mu}\psi_{r}+g_{2,\mu}^{*}(\partial_{\mu}\psi_{r}+ik_{s,\mu})-\frac{5}{\sqrt{2}}K\sin\theta\cos\theta\big{]}e^{i(\bf{k}_{s}-\bf{k}_{p})\cdot\bf{r}}d\bf{r}$
and
$\tilde{g}_{q}=\int\big{[}g_{2,\mu}(\partial_{\mu}\psi_{r}^{*}+ik_{q,\mu})+2g_{1,\mu}^{*}ik_{s,\mu}\psi_{r}^{*}-\frac{5}{\sqrt{2}}K\sin\theta\cos\theta\big{]}e^{i(\bf{k}_{s}-\bf{k}_{q})\cdot\bf{r}}d\bf{r}$
where the Einstein summation rule is applied, coefficients
$g_{1,\mu}=\frac{3J}{2\sqrt{2}}(\mathcal{A}_{\mu,13}-i\mathcal{A}_{\mu,23})+\frac{3D}{4\sqrt{2}}(e_{\phi,\mu}-ie_{\theta,\mu})$
and
$g_{2,\mu}=\frac{J}{\sqrt{2}}(-\mathcal{A}_{\mu,13}-i\mathcal{A}_{\mu,23})+\frac{D}{2\sqrt{2}}(-e_{\phi,\mu}-ie_{\theta,\mu})$
denote the strength of the three-magnon confluence and splitting,
respectively, and $\alpha$ is the Gilbert damping constant. Then, one can
analytically derive the steady-state magnon populations as
$a_{p}=\frac{ga_{s}a_{r}}{\epsilon+i\alpha(\omega_{s}+\omega_{r})}$ and
$a_{q}=\frac{ga_{s}a_{r}^{\dagger}}{\epsilon-i\alpha(\omega_{s}-\omega_{r})}$
with $\epsilon=\omega_{s}-\omega_{r}$. Here, we have adopted the approximation
$\tilde{g}_{p}\approx\tilde{g}_{q}\approx g$ which is justified by the small
difference between confluence and splitting frequencies since
$\omega_{r}\ll\omega_{s}$. We therefore obtain

Figure 2: Spatial distribution of dimensionless field $B_{z}/B_{0}$ (a) and
$B^{\prime}_{z}/B_{0}$ (b). Spin wave trajectories in real space under
fictitious magnetic field ${\bf B}$ (c) and ${\bf B}+{\bf B}^{\prime}$ (d),
where different black curves represent trajectories of magnon wavepackets with
different impact parameters, the red curve indicates the averaged trajectory
of 51 magnon wavepackets, and the dashed green circle labels skyrmion’s wall
center ($l_{z}=0$).

$\displaystyle m_{{\rm sw},\,i}\dot{\bf v}_{i}-e{\bf v}_{i}\times\sigma({\bf
B}+\lambda_{i}{\bf B}^{\prime})=0,(i=s,p,q),$ (6)

which is the main result of this work (see Supplemental Material SM  for
detailed derivations). Here, $m_{{\rm sw},i}=\hbar\omega_{i}/J$ is the
effective mass of the spin-wave packet in antiferromagnets, $\sigma=\mp 1$
represents the left/right-hand magnon polarizations,
$\lambda_{s}=n_{r}(\frac{gg_{p}}{4\epsilon}+\frac{3gg_{q}}{4\epsilon}+{\rm
H.c.})$, $\lambda_{p}=\frac{3}{4}(\frac{\epsilon g_{p}}{g}+{\rm H.c.})$, and
$\lambda_{q}=\frac{1}{4}(\frac{\epsilon g_{q}}{g}+{\rm H.c.})$ with the
particle number of skyrmion breathing mode $n_{r}=\langle
a_{r}^{\dagger}a_{r}\rangle$. Equation (6) shows that the extra effective
Lorentz force $e\lambda_{i}{\bf v}_{i}\times\sigma{\bf B}^{\prime}$
$(i=s,p,q)$ is mode-dependent. For incident magnons, the extra Lorentz force
is proportional to the product of the skyrmion breathing number $n_{r}\ (\ll
1)$, the coupling parameter $g/\epsilon$, and the overlap integral $g_{p,q}$.
In general, magnon populations of confluence and splitting modes are far less
than the incident one. It implies $g/\epsilon,\ g_{p,q}\ll 1$. The effect of
${\bf B}^{\prime}$ on incident magnons is thus negligible. However, for the
confluence and splitting modes, parameters $\lambda_{p,q}$ are inversely
proportional to $g/\epsilon$, the additional effective Lorentz force is
therefore expected to bring an enormous effect.

To explore the role of fictitious magnetic fields on the magnon transport, we
numerically solve Eq. (6) both without and with the new fictitious magnetic
field ${\bf B}^{\prime}$. In calculations, we consider a right-handed magnon
wavepacket ($\sigma=1$), and set the incident-magnon’s initial velocity
$v_{s}(t=0)=2.65$ (with unit $J/a\omega$) along $x$ direction, magnon mass
$m_{\rm sw}=0.31$ (with unit $\hbar\omega/J$), and coefficient
$\lambda_{s,p,q}=1$. It is observed that the effective magnetic field ${\bf
B}^{\prime}$ significantly enhances the magnon Hall effect, as displayed in
Figs. 2(c) and 2(d). Due to the singularity of the fictitious magnetic field
${\bf B}^{\prime}$, we note anomalous magnon trajectories near the skyrmion
center. Below, we verify our theoretical predictions by full micromagnetic
simulations using MUMAX3 package Vansteenkiste2014 .

We consider an AFM thin film of dimension $1000$ $\times 1000$ $\times 1$ nm3,
hosting a Bloch-type skyrmion at the film center Binz2009 ; Yu2010 ; Huang2013
. Magnetic parameters of ${\rm KMnF_{3}}$ Barker2016 : $J=6.59$ pJ/m,
$K=1.16\times 10^{5}$ J/m3, and $D=1$ mJ/m2 are used in the simulations, which
gives rise to a skyrmion radius $\approx 11$ nm (defineds as the radius of
circle $l_{z}=0$). The Gilbert damping is set as $\alpha=0.001$. To
efficiently generate polarized magnons and the three-wave mixing, we apply a
microwave field ${\bf h}_{\rm RH/LH}(t)=h_{0}[\cos(\omega_{s}t){\bf
e}_{x}\mp\sin(\omega_{s}t){\bf e}_{y}]$ with amplitude $h_{0}=50$ mT and
frequency $\omega_{s}/2\pi=1.205$ THz (generating the incident magnon of
wavelength $\approx 15.2$ nm) on one sublattice in a narrow region: $-401$
nm$\leq x\leq-399$ nm and a local field ${\bf
h}_{r}(t)=h_{r}\sin(\omega_{r}t){\bf e}_{z}$ over the skyrmion with amplitude
$h_{r}=5$ mT and $\omega_{r}/2\pi=0.095$ THz (the skyrmion breathing
frequency) SM . Here, RH and LH represent the abbreviation of microwave with
right and left handedness, respectively. Absorbing boundary conditions are
adopted to eliminate the spin-wave reflection by film edges Venkat2018 .

Figure 3: (a) MFC in the nonlinear scattering between the incident magnon and
AFM skyrmion. (b) Isoline maps for righted-handed (top panel) and left-handed
(bottom panel) magnons scattered by the skrymion at the origin. In each panel,
modes from left to right correspond to splitting, incident, and confluence
magnons, respectively.

To analyze the magnon spectrum in the skyrmion area, we implement fast Fourier
transform of local magnetic moments. Figure 3(a) shows the emerging magnon
frequency comb (MFC) Hula2022  in the terahertz region, where the mode spacing
of the comb is exactly the skyrmion breathing frequency. Furthermore, we plot
the isoline map of incident, confluence, and splitting modes to analyze the
Hall angle of each mode, as shown in Fig. 3(b). We observe an interesting
“bunny ears” pattern of magnons scattering off the AFM skyrmion, with red and
blue lines denoting the propagation direction of two branches. Here, the Hall
angle is defined as the included angle between each branch and the horizontal
line. Compared with the incident mode, the Hall angle of nonlinear modes
nearly doubles (quintuples) for the main (secondary) branch of the “bunny
ears” [see Fig. 3(b)], where the major (secondary) branch is referred to as
the one with a large (small) Hall angle. More importantly, by flipping the
chirality of incident magnons, we observe an opposite magnon Hall motion
[comparing the top and bottom panels in Fig. 3(b)]. The small difference of
Hall angles between right-handed and left-handed magnons results from the
dipolar field SM . Micromagnetic simulations thus offer solid evidences for
the nonlinear topological magnon spin Hall effect as we predicted above.

Furthermore, we derive the frequency-dependent Hall angle by fitting the flow
direction of the main branch of the isosurface of each mode. Figure 4(a) plots
the quantitative comparison between theoretical calculations and micromagnetic
simulations for incident (black), confluence (blue), and splitting (red)
modes. It shows that the Hall angle monotonically decreases with the increase
of the mode frequency. Simulation results can be well explained by the
analytical model (6) with parameters $n_{r}=0$, $g=49$ MHz, and
$g_{p}=\frac{1}{3}g_{q}=9.4\times 10^{-6}$. A vanishing mode number of
skyrmion breathing is justified by its small wave amplitude. The coupling
coefficient $g$ is independently obtained by numerically solving the coupled
Heisenberg equations. Acceptable deviations could be attributed to the
simplified wavepacket treatment in the present formalism and the neglected
topological electric-field component of the gauge fields. Figure 4(b) shows
the Hall angle of nonlinear magnons over a broad frequencies
$\omega_{s}+m\omega_{r}$ in the MFC with integer $m$ labeling the order of the
spectrum line. Their “bunny ears” scattering patterns are plotted in
Supplemental Material SM . It is found that the noninear Hall angle increases
linearly with $|m|$. This monotonic dependence is reminiscent of the
refraction process of light waves through multilayer media Multiple , where
the refraction angle accumulates upon each scattering layer. It is noted that
the slope of the linear trendline decreases as the incident magnon’s frequency
$\omega_{s}$ increases.

Figure 4: (a) The Hall angle of the main branch of “bunny ears” as a function
of the driving frequency for the incident (black), confluence (blue), and
splitting (red) modes. Symbols are micromagnetic simulations and curves are
analytical fitting by solving Eq. (6). (b) The Hall angle as a function of the
mode index $m$ for different incident magnon frequencies. Symbols and lines
represent micromagnetic simulations and linear fittings, respectively.

Discussion.—In the above calculations, we have considered rotationally
symmetric spin textures, the ${\bf A}_{13}$ gauge field thus vanishes.
However, the curl of gauge field induced by DMI becomes finite when the
rotational symmetry is broken. We then envision contributions from ${\bf
A}_{13}$ in generating the fictitious magnetic field for elliptical skyrmions
Jena2020 .

To summarize, we revealed the long-sought gauge fields concealed in the
nonlinear magnon transport. By investigating the three-wave mixing between
propagating magnons and breathing skyrmions, we found giant Hall angles
emerging for each nonlinear spectrum line of the MFC. We further identified
that the sign of the Hall angle is reversed by switching the chirality of
incident magnons, and we dub it nonlinear topological magnon spin Hall effect.
Our findings are intimately connected to the particle number nonconservation
of magnons and thus applicable to generic bosons, which does not have the low-
energy fermionic counterpart. Our results significantly advance the
understanding of the nonlinear Hall effect and pave the way to probing the
gauge field by frequency comb.

###### Acknowledgements.

This work was funded by the National Key Research Development Program under
Contract No. 2022YFA1402802 and the National Natural Science Foundation of
China (NSFC) (Grant No. 12074057). Z.W. acknowledges financial support from
the NSFC (Grant No. 12204089) and the China Postdoctoral Science Foundation
under Grant No. 2019M653063. H.Y.Y. acknowledges the European Union’s Horizon
2020 research and innovation programme under Marie Skłodowska-Curie Grant
Agreement SPINCAT No. 101018193.

## References

  * (1) P. Delplace, J. B. Marston, and A. Venaille, Topological origin of equatorial waves, Science 358, 1075 (2017).
  * (2) J. B. Parker, J. B. Marston, S. M. Tobias, and Z. Zhu, Topological Gaseous Plasmon Polariton in Realistic Plasma, Phys. Rev. Lett. 124, 195001 (2020).
  * (3) Y. K. Kato, R. C. Myers, A. C. Gossard, and D. D. Awschalom, Observation of the spin Hall effect in semiconductors, Science 306, 1910 (2004).
  * (4) J. Wunderlich, B. Kaestner, J. Sinova, and T. Jungwirth, Experimental Observation of the Spin-Hall Effect in a Two-Dimensional Spin-Orbit Coupled Semiconductor System, Phys. Rev. Lett. 94, 047204 (2005).
  * (5) J. Sinova, S. O. Valenzuela, J. Wunderlich, C. H. Back, and T. Jungwirth, Spin Hall effects, Rev. Mod. Phys. 87, 1213 (2015).
  * (6) A. Kavokin, G. Malpuech, and M. Glazov, Optical Spin Hall Effect, Phys. Rev. Lett. 95, 136601 (2005).
  * (7) C. Leyder, M. Romanelli, J. Ph. Karr, E. Giacobino, T. C. H. Liew, M. M. Glazov, A. V. Kavokin, G. Malpuech, and A. Bramati, Observation of the optical spin Hall effect, Nat. Phys. 3, 628 (2007).
  * (8) Y. Onose, T. Ideue, H. Katsura, Y. Shiomi, N. Nagaosa, and Y. Tokura, Observation of the magnon Hall effect, Science 329, 297 (2010).
  * (9) K. Shen, Magnon Spin Relaxation and Spin Hall Effect Due to the Dipolar Interaction in Antiferromagnetic Insulators,  Phys. Rev. Lett. 124, 077201 (2020).
  * (10) J. Sinova, D. Culcer, Q. Niu, N. A. Sinitsyn, T. Jungwirth, and A. H. MacDonald, Universal Intrinsic Spin Hall Effect, Phys. Rev. Lett. 92, 126603 (2004).
  * (11) G. Y. Guo, S. Murakami, T.-W. Chen, and N. Nagaosa, Intrinsic Spin Hall Effect in Platinum: First-Principles Calculations, Phys. Rev. Lett. 100, 096401 (2008).
  * (12) G. Yin, Y. Liu, Y. Barlas, J. Zang, and R. K. Lake, Topological spin Hall effect resulting from magnetic skyrmions, Phys. Rev. B 92, 024411 (2015).
  * (13) B. Göbel, A. Mook, J. Henk, and I. Mertig, Antiferromagnetic skyrmion crystals: Generation, topological Hall, and topological spin Hall effect, Phys. Rev. B 96, 060406 (2017).
  * (14) C. A. Akosa, O. A. Tretiakov, G. Tatara, and A. Manchon, Theory of the Topological Spin Hall Effect in Antiferromagnetic Skyrmions: Impact on Current-Induced Motion, Phys. Rev. Lett. 121, 097204 (2018).
  * (15) M. W. Daniels, W. Yu, R. Cheng, J. Xiao, and D. Xiao, Topological spin Hall effects and tunable skyrmion Hall effects in uniaxial antiferromagnetic insulators, Phys. Rev. B 99, 224433 (2019).
  * (16) S. K. Kim, K. Nakata, D. Loss, and Y. Tserkovnyak, Tunable Magnonic Thermal Hall Effect in Skyrmion Crystal Phases of Ferrimagnets, Phys. Rev. Lett. 122, 057204 (2019).
  * (17) Q. Du, Z. Hu, M. G. Han, F. Camino, Y. Zhu, and C. Petrovic, Topological Hall Effect Anisotropy in Kagome Bilayer Metal $\rm{Fe_{3}Sn_{2}}$, Phys. Rev. Lett. 129, 236601 (2022).
  * (18) P. Yan, X. S. Wang, and X. R. Wang, All-Magnonic Spin-Transfer Torque and Domain Wall Propagation, Phys. Rev. Lett. 107, 177207 (2011).
  * (19) H. Y. Yuan, Y. Cao, A. Kamra, R. A. Duine, and P. Yan, Quantum magnonics: When magnon spintronics meets quantum information science, Phys. Rep. 965, 1 (2022).
  * (20) V. K. Dugaev, P. Bruno, B. Canals, and C. Lacroix, Berry phase of magnons in textured ferromagnets, Phys. Rev. B 72, 024456 (2005).
  * (21) A. A. Kovalev and Y. Tserkovnyak, Thermomagnonic spin transfer and Peltier effects in insulating magnets, EPL 97, 67002 (2012).
  * (22) K. A. van Hoogdalem, Y. Tserkovnyak, and D. Loss, Magnetic texture-induced thermal Hall effects Phys. Rev. B 87, 024402 (2013).
  * (23) C. Schütte and M. Garst, Magnon-skyrmion scattering in chiral magnets, Phys. Rev. B 90, 094423 (2014).
  * (24) J. Lan and J. Xiao, Skew scattering and side jump of spin wave across magnetic texture, Phys. Rev. B 103, 054428 (2021).
  * (25) Z. Wang, W. Bao, Y. Cao, and P. Yan, All-magnonic Stern-Gerlach effect in antiferromagnets, Appl. Phys. Lett. 120, 242403 (2022).
  * (26) Z. Jin, C. Y. Meng, T. T. Liu, D. Y. Chen, Z. Fan, M. Zeng, X. B. Lu, X. S. Gao, M. H. Qin, and J.-M. Liu, Magnon-driven skyrmion dynamics in antiferromagnets: Effect of magnon polarization, Phys. Rev. B 104, 054419 (2021).
  * (27) Y. Liu, T. T. Liu, Z. Jin, Z. P. Hou, D. Y. Chen, Z. Fan, M. Zeng, X. B. Lu, X. S. Gao, M. H. Qin, and J.-M. Liu, Spin-wave-driven skyrmion dynamics in ferrimagnets: Effect of net angular momentum, Phys. Rev. B 106, 064424 (2022).
  * (28) S. G. Tan, S.-H. Chen, C. S. Ho, C.-C. Huang, M. B. A. Jalil, C. R. Chang, and S. Murakami, Yang-Mills physics in spintronics, Phys. Rep. 882, 1 (2020).
  * (29) G. Tatara, Effective gauge field theory of spintronics, Physica E 106, 208 (2019).
  * (30) I. Sodemann and L. Fu, Quantum Nonlinear Hall Effect Induced by Berry Curvature Dipole in Time-Reversal Invariant Materials, Phys. Rev. Lett. 115, 216806 (2015).
  * (31) Q. Ma, S.-Y. Xu, H. Shen, D. MacNeill, V. Fatemi, T.-R. Chang, A. M. M. Valdivia, S. Wu, Z. Du, C.-H. Hsu, S. Fang, Q. D. Gibson, K. Watanabe, T. Taniguchi, R. J. Cava, E. Kaxiras, H.-Z. Lu, H. Lin, L. Fu, N. Gedik, and P. J. Herrero, Observation of the nonlinear Hall effect under time-reversal-symmetric conditions, Nature (London) 565, 337 (2019).
  * (32) K. Kang, T. Li, E. Sohn, J. Shan, and K. F. Mak, Nonlinear anomalous Hall effect in few-layer WTe2, Nat. Mater. 18, 324 (2019).
  * (33) Z. Z. Du, C. M. Wang, H.-Z. Lu, and X. C. Xie, Band Signatures for Strong Nonlinear Hall Effect in Bilayer WTe2, Phys. Rev. Lett. 121, 266601 (2018).
  * (34) P. He, S. S.-L. Zhang, D. Zhu, S. Shi, O. G. Heinonen, G. Vignale, and H. Yang, Nonlinear Planar Hall Effect, Phys. Rev. Lett. 123, 016801 (2019).
  * (35) D.-F. Shao, S.-H. Zhang, G. Gurung, W. Yang, and E. Y. Tsymbal, Nonlinear Anomalous Hall Effect for Néel Vector Detection, Phys. Rev. Lett. 124, 067203 (2020).
  * (36) S. Lai, H. Liu, Z. Zhang, J. Zhao, X. Feng, N. Wang, C. Tang, Y. Liu, K. S. Novoselov, S. A. Yang, and W.-B. Gao, Third-order nonlinear Hall effect induced by the Berry-connection polarizability tensor, Nat. Nano. 16, 869 (2021).
  * (37) J. Duan, Y. Jian, Y. Gao, H. Peng, J. Zhong, Q. Feng, J. Mao, and Y. Yao, Giant Second-Order Nonlinear Hall Effect in Twisted Bilayer Graphene, Phys. Rev. Lett. 129, 186801 (2022).
  * (38) Y. M. Itahashi, T. Ideue, S. Hoshino, C. Goto, H. Namiki, T. Sasagawa, and Y. Iwasa, Giant second harmonic transport under time-reversal symmetry in a trigonal superconductor, Nat. Commun. 13, 1659 (2022).
  * (39) Z. Z. Du, H. Z. Lu, and X. C. Xie, Nonlinear Hall effects, Nat. Rev. Phys 3, 744 (2021).
  * (40) A. Mook, B. Göbel, J. Henk, and I. Mertig, Taking an electron-magnon duality shortcut from electron to magnon transport, Phys. Rev. B 97, 140401 (2018).
  * (41) H. Kondo and Y. Akagi, Nonlinear magnon spin Nernst effect in antiferromagnets and strain-tunable pure spin current, Phys. Rev. Research 4, 013186 (2022).
  * (42) M. Tymchenko, J. S. Gomez-Diaz, J. Lee, N. Nookala, M. A. Belkin, and A. Alù, Gradient Nonlinear Pancharatnam-Berry Metasurfaces, Phys. Rev. Lett. 115, 207403 (2015).
  * (43) G. Li, S. Chen, N. Pholchai, B. Reineke, P. W. H. Wong, E. Y. B. Pun, K. W. Cheah, T. Zentgraf, and S. Zhang, Continuous control of the nonlinearity phase for harmonic generations, Nat. Mat. 14, 607 (2015).
  * (44) Y. Li, O. Yesharim, I. Hurvitz, A. Karnieli, S. Fu, G. Porat, and A. Arie, Adiabatic geometric phase in fully nonlinear three-wave mixing, Phys. Rev. A 101, 033807 (2020).
  * (45) Z. Wang, H. Y. Yuan, Y. Cao, Z.-X. Li, R. A. Duine, and P. Yan, Magnonic Frequency Comb through Nonlinear Magnon-Skyrmion Scattering, Phys. Rev. Lett. 127, 037202 (2021).
  * (46) Z. Wang, H. Y. Yuan, Y. Cao, and P. Yan, Twisted Magnon Frequency Comb and Penrose Superradiance, Phys. Rev. Lett. 129, 107203 (2022).
  * (47) H. Schultheiss, X. Janssens, M. van Kampen, F. Ciubotaru, S. J. Hermsdoerfer, B. Obry, A. Laraoui, A. A. Serga, L. Lagae, A. N. Slavin, B. Leven, and B. Hillebrands, Direct Current Control of Three Magnon Scattering Processes in Spin-Valve Nanocontacts, Phys. Rev. Lett. 103, 157202 (2009).
  * (48) J. Iwasaki, A. J. Beekman, and N. Nagaosa, Theory of magnon-skyrmion scattering in chiral magnets, Phys. Rev. B 89, 064412 (2014).
  * (49) B. Zhang, Z. Wang, Y. Cao, P. Yan, and X. R. Wang, Eavesdropping on spin waves inside the domain-wall nanochannel via three-magnon processes, Phys. Rev. B 97, 094421 (2018).
  * (50) K. Schultheiss, R. Verba, F. Wehrmann, K. Wagner, L. Körber, T. Hula, T. Hache, A. K$\rm{\acute{a}}$kay, A. A. Awad, V. Tiberkevich, A. N. Slavin, J. Fassbender, and H. Schultheiss, Excitation of Whispering Gallery Magnons in a Magnetic Vortex, Phys. Rev. Lett. 122, 097202 (2019).
  * (51) L. Körber, K. Schultheiss, T. Hula, R. Verba, J. Fassbender, A. Kákay, and H. Schultheiss, Nonlocal Stimulation of Three-Magnon Splitting in a Magnetic Vortex, Phys. Rev. Lett. 125, 207203 (2020).
  * (52) T. Hula, K. Schultheiss, F. J. T. Goncalves, L. Körber, M. Bejarano, M. Copus, L. Flacke, L. Liensberger, A. Buzdakov, A. Kákay, M. Weiler, R. Camley, J. Fassbender, and H. Schultheiß, Spin-wave frequency combs, Appl. Phys. Lett. 121, 112404 (2022).
  * (53) T. Holstein and H. Primakoff, Field dependence of the intrinsic domain magnetization of a ferromagnet, Phys. Rev. 58, 1098 (1940).
  * (54) M. Mochizuki, Spin-Wave Modes and Their Intense Excitation Effects in Skyrmion Crystals, Phys. Rev. Lett. 108, 017061 (2012).
  * (55) A. A. Thiele, Steady-State Motion of Magnetic Domains, Phys. Rev. Lett. 30, 230 (1973).
  * (56) See Supplemental Material at http://link.aps.org/supplemental/ for the derivation of equations of motion for magnon wavepackets, the numerical verification of the skyrmion breathing and its frequency, the analysis of magnon polarizations on their nonlinear topological Hall transport, and the isolines of incident magnons scattered by skyrmions with and without breathing, which includes Refs. Kim2019 ; Wang2021 ; HP .
  * (57) G. Sundaram and Q. Niu, Wave-packet dynamics in slowly perturbed crystals: Gradient corrections and Berry-phase effects, Phys. Rev. B 59, 14915 (1999).
  * (58) G. E. Volovik, Linear momentum in ferromagnets, J. Phys. C 20, L83 (1987).
  * (59) J. Tang, Y. Wu, W. Wang, L. Kong, B. Lv, W. Wei, J. Zang, M. Tian, and H. Du, Magnetic skyrmion bundles and their current-driven dynamics, Nat. Nanotechnol. 16, 1086 (2021).
  * (60) P. B. Ndiaye, C. A. Akosa, and A. Manchon, Topological Hall and spin Hall effects in disordered skyrmionic textures, Phys. Rev. B 95, 064426 (2017).
  * (61) A. Vansteenkiste, J. Leliaert, M. Dvornik, M. Helsen, F. Garcia-Sanchez, and B. V. Waeyenberge, The design and verification of MUMAX3, AIP Adv. 4, 107133 (2014).
  * (62) S. Mühlbauer, B. Binz, F. Jonietz, C. Pfleiderer, A. Rosch, A. Neubauer, R. Georgii, and P. Böni, Skyrmion Lattice in a Chiral Magnet, Science 323, 915 (2009).
  * (63) X. Z. Yu, Y. Onose, N. Kanazawa, J. H. Park, J. H. Han, Y. Matsui, N. Nagaosa, and Y. Tokura, Real-space observation of a two-dimensional skyrmion crystal, Nature (London) 465, 901 (2010).
  * (64) S. X. Huang and C. L. Chien, Extended Skyrmion Phase in Epitaxial FeGe (111) Thin Films, Phys. Rev. Lett. 108, 267201 (2012).
  * (65) J. Barker and O. A. Tretiakov, Static and Dynamical Properties of Antiferromagnetic Skyrmions in the Presence of Applied Current and Temperature, Phys. Rev. Lett. 116, 147203 (2016).
  * (66) G. Venkat, H. Fangohr, and A. Prabhakar, Absorbing boundary layers for spin wave micromagnetics, J. Magn. Magn. Mater. 450, 34 (2018).
  * (67) M. Lax, Multiple Scattering of Waves, Rev. Mod. Phys. 23, 287 (1951).
  * (68) J. Jena, B. Göbel, T. Ma, V. Kumar, R. Saha, I. Mertig, C. Felser, and S. S. P. Parkin, Elliptical Bloch skyrmion chiral twins in an antiskyrmion system, Nat. Commun. 11, 1115 (2020).