#  Quest: A Retrieval Dataset of Entity-Seeking Queries
with Implicit Set Operations

Chaitanya Malaviya1, Peter Shaw2, Ming-Wei Chang2, Kenton Lee2, Kristina
Toutanova2

1University of Pennsylvania 2Google DeepMind
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>Work done during an
internship at Google.

###### Abstract

Formulating selective information needs results in queries that implicitly
specify set operations, such as intersection, union, and difference. For
instance, one might search for "shorebirds that are not sandpipers" or
"science-fiction films shot in England". To study the ability of retrieval
systems to meet such information needs, we construct Quest, a dataset of 3357
natural language queries with implicit set operations, that map to a set of
entities corresponding to Wikipedia documents. The dataset challenges models
to match multiple constraints mentioned in queries with corresponding evidence
in documents and correctly perform various set operations. The dataset is
constructed semi-automatically using Wikipedia category names. Queries are
automatically composed from individual categories, then paraphrased and
further validated for naturalness and fluency by crowdworkers. Crowdworkers
also assess the relevance of entities based on their documents and highlight
attribution of query constraints to spans of document text. We analyze several
modern retrieval systems, finding that they often struggle on such queries.
Queries involving negation and conjunction are particularly challenging and
systems are further challenged with combinations of these operations.111The
dataset is available at https://github.com/google-
research/language/tree/master/language/quest.

##  1 Introduction

Figure 1: The dataset construction process for Quest. First, (1) we sample
Wikipedia category names and find their corresponding set of relevant
entities. (2) Then, we compose a query with set operations and have this query
paraphrased by crowdworkers. (3) These queries are then validated for fluency
and naturalness. (4) Finally, crowdworkers mark the entities’ relevance by
highlighting attributable spans in their documents.

People often express their information needs with multiple preferences or
constraints. Queries corresponding to such needs typically implicitly express
set operations such as intersection, difference, and union. For example, a
movie-goer might be looking for a science-fiction film from the 90s which does
not feature aliens and a reader might be interested in a historical fiction
novel set in France. Similarly, a botanist attempting to identify a species
based on their recollection might search for shrubs that are evergreen and
found in Panama. Further, if the set of entities that satisfy the constraints
is relatively small, a reader may like to see and explore an exhaustive list
of these entities. In addition, to verify and trust a system’s
recommendations, users benefit from being shown evidence from trusted sources
Lamm et al. (2021).

Addressing such queries has been primarily studied in the context of question
answering with structured knowledge bases (KBs), where query constraints are
grounded to predefined predicates and symbolically executed. However, KBs can
be incomplete and expensive to curate and maintain. Meanwhile, advances in
information retrieval may enable developing systems that can address such
queries without relying on structured KBs, by matching query constraints
directly to supporting evidence in text documents. However, queries that
combine multiple constraints with implicit set operations are not well
represented in existing retrieval benchmarks such as MSMarco Nguyen et al.
(2016) and Natural Questions Kwiatkowski et al. (2019). Also, such datasets do
not focus on retrieving an exhaustive document set, instead limiting
annotation to the top few results of a baseline information retrieval system.

To analyze retrieval system performance on such queries, we present Quest, a
dataset with natural language queries from four domains, that are mapped to
relatively comprehensive sets of entities corresponding to Wikipedia pages. We
use categories and their mapping to entities in Wikipedia as a building block
for our dataset construction approach, but do not allow access to this semi-
structured data source at inference time, to simulate text-based retrieval.
Wikipedia categories represent a broad set of natural language descriptions of
entity properties and often correspond to selective information need queries
that could be plausibly issued by a search engine user. The relationship
between property names and document text is often subtle and requires
sophisticated reasoning to determine, representing the natural language
inference challenge inherent in the task.

Our dataset construction process is outlined in Figure 1. The base queries are
semi-automatically generated using Wikipedia category names. To construct
complex queries, we sample category names and compose them by using pre-
defined templates (for example, $A\cap B\setminus C$). Next, we ask
crowdworkers to paraphrase these automatically generated queries, while
ensuring that the paraphrased queries are fluent and clearly describe what a
user could be looking for. These are then validated for naturalness and
fluency by a different set of crowdworkers, and filtered according to those
criteria. Finally, for a large subset of the data, we collect scalar relevance
labels based on the entity documents and fine-grained textual attributions
mapping query constraints to spans of document text. Such annotation could aid
the development of systems that can make precise inferences from trusted
sources.

Performing well on this dataset requires systems that can match query
constraints with corresponding evidence in documents and handle set operations
implicitly specified by the query (see Figure 2), while also efficiently
scaling to large collections of entities. We evaluate several retrieval
systems by finetuning pretrained models on our dataset. Systems are trained to
retrieve multi-document sets given a query. We find that current dual encoder
and cross-attention models up to the size of T5-Large Raffel et al. (2020) are
largely not effective at performing retrieval for queries with set operations.
Queries with conjunctions and negations prove to be especially challenging for
models and systems are further challenged with combinations of set operations.
Our error analysis reveals that non-relevant false positive entities are often
caused by the model ignoring negated constraints, or ignoring the conjunctive
constraints in a query.

Figure 2: An example of a query and relevant entity from Quest. The
attribution for different query constraints can come from different parts of
the document.

##  2 Related Work

Previous work in question answering and information retrieval has focused on
QA over knowledge bases as well as open-domain QA and retrieval over a set of
entities or documents. We highlight how these relate to our work below.

##### Knowledge Base QA

Several datasets have been proposed for question answering over knowledge
bases (Berant et al., 2013; Yih et al., 2016; Talmor and Berant, 2018; Keysers
et al., 2020; Gu et al., 2021, inter alia). These benchmarks require retrieval
of a set of entities that exist as nodes or relations in an accompanying
knowledge base. Questions are optionally supplemented with logical forms. Lan
et al. (2021) provide a comprehensive survey of complex KBQA datasets.

Previous work has simultaneously noted that large curated KBs are incomplete
Watanabe et al. (2017). Notably, KBQA systems operate over a constrained
answer schema, which limits the types of queries they can handle. Further,
these schema are expensive to construct and maintain. For this reason, our
work focuses on a setting where we do not assume access to a KB. We note that
KBQA datasets have also been adapted to settings where a KB is incomplete or
unavailable Watanabe et al. (2017); Sun et al. (2019). This was done by either
removing some subset of the data from the KB or ignoring the KB entirely. A
key difference from these datasets is also that we do not focus on multi-hop
reasoning over multiple documents. Instead, the relevance of an entity can be
determined solely based on its document.

##### Open-Domain QA and Retrieval

Many open-domain QA benchmarks, which consider QA over unstructured text
corpora, have been proposed in prior work. Some of these, such as TREC
Craswell et al. (2020), MSMarco Nguyen et al. (2016) and Natural Questions
Kwiatkowski et al. (2019) are constructed using "found data", using real user
queries on search engines. Thakur et al. (2021) present a benchmark where they
consider many such existing datasets. Datasets such as HotpotQA Yang et al.
(2018), and MultiRC Khashabi et al. (2018) have focused on multi-hop question
answering. Other work has explored e-commerce datasets (for example, Kong et
al. (2022)), but these have not been released publicly. Notably, the focus of
these datasets differs from ours as we focus on queries that contain implicit
set operations over exhaustive answer sets. Such queries are not well
represented in existing datasets because they occur in the tail of the query
distributions considered.

##### Multi-Answer Retrieval

Related work Min et al. (2021); Amouyal et al. (2022) also studies the problem
of _multi-answer retrieval_ , where systems are required to predict multiple
distinct answers for a query. Min et al. (2021) adapt existing datasets (for
example, WebQuestionsSP Yih et al. (2016)) to study this setting and propose a
new metric, MRecall@K, to evaluate exhaustive recall of multiple answers. We
also consider the problem of multi-answer set retrieval, but consider queries
that implicitly contain set constraints.

In concurrent work, RomQA Zhong et al. (2022) proposes an open-domain QA
dataset, focusing on combinations of constraints extracted from Wikidata.
RomQA shares our motivation to enable answering queries with multiple
constraints, which have possibly large answer sets. To make attribution to
evidence feasible without human annotation, RomQA focuses on questions whose
component constraints can be verified from single entity-linked sentences from
Wikipedia abstracts, annotated with relations automatically through distant
supervision, with high precision but possibly low recall (T-Rex corpus). In
Quest, we broaden the scope of query-evidence matching operations by allowing
for attribution through more global, document-level inference. To make human
annotation for attribution feasible, we limit the answer set size and the
evidence for an answer to a single document.

##  3 Dataset Generation

Domain | Template | Example | Num. Queries
---|---|---|---
| $A$ | Biographical Italian bandits films | 125
| $A\cup B$ | Dutch crime comedy or romantic comedy films | 135
| $A\cap B$ | Italian crime films set in the 1970’s | 143
Films | $A\setminus B$ | Indian sport films that are not about cricket | 126
| $A\cup B\cup C$ | Dutch or Swiss war films, or war films from 1945 | 122
| $A\cap B\cap C$ | 2020’s drama films shot in cleveland | 124
| $A\cap B\setminus C$ | Epic films about Christianity not set in Israel | 121
| $A$ | 2004 German novels | 125
| $A\cup B$ | 1925 Russian novels or Novels by Ivan Bunin | 125
| $A\cap B$ | 1991 Novels set in Iceland | 133
Books | $A\setminus B$ | Novels set in the 1900s not based on real events | 123
| $A\cup B\cup C$ | Novels set in Nanjing, Hebei, or Jiangsu | 125
| $A\cap B\cap C$ | English language Harper & Brothers Children’s fiction books | 124
| $A\cap B\setminus C$ | Novels that take place in Vietnam that aren’t about war | 115
| $A$ | plants only from Gabon | 115
| $A\cup B$ | Trees of Manitoba or Subarctic America | 125
| $A\cap B$ | Shrubs used in traditional Native American medicine | 135
Plants | $A\setminus B$ | Trees from the Northwestern US that can’t be found in Canada | 61
| $A\cup B\cup C$ | Moths or Insects or Arthropods of Guadeloupe | 121
| $A\cap B\cap C$ | Plants the Arctic, the United Kingdom, and the Caucasus have in common | 123
| $A\cap B\setminus C$ | Orchids of Indonesia and Malaysia but not Thailand | 122
| $A$ | what are the Rodents of Cambodia | 115
| $A\cup B$ | Animals from Cuba or Jamaica that are extinct | 121
| $A\cap B$ | Neogene mammals of Africa that are Odd-toed ungulates | 111
Animals | $A\setminus B$ | Non-Palearctic birds of Mongolia | 110
| $A\cup B\cup C$ | Cenozoic birds of Asia or Africa or Paleogene birds of Asia | 114
| $A\cap B\cap C$ | Birds of Chile that are also Birds of Peru and Fauna of the Guianas | 104
| $A\cap B\setminus C$ | mammals found in the Atlantic Ocean and Colombia, but not in Brazil | 114

Table 1: Templates used for construction of queries with set operations and
examples from the four domains considered, along with the count of examples
per each domain and template.

Quest consists of 3357 queries paired with up to 20 corresponding entities.
Each entity has an associated document derived from its Wikipedia page. The
dataset is divided into 1307 queries for training, 323 for validation, and
1727 for testing.

The task for a system is to return the correct set of entities for a given
query. Additionally, as the collection contains 325,505 entities, the task
requires retrieval systems that can scale efficiently. We do not allow systems
to access additional information outside of the text descriptions of entities
at inference time. Category labels are omitted from all entity documents.

###  3.1 Atomic Queries

The base atomic queries (i.e., queries without any introduced set operations)
in our dataset are derived from Wikipedia category names222We use the
Wikipedia version from 06/01/2022.. These are hand-curated natural language
labels assigned to groups of related documents in Wikipedia333Note that these
category labels can sometimes be conjunctive themselves, potentially
increasing complexity.. Category assignments to documents allow us to
automatically determine the set of answer entities for queries with high
precision and relatively high recall. We compute transitive closures of all
relevant categories to determine their answer sets.

However, repurposing these categories for constructing queries poses
challenges: 1) lack of evidence in documents: documents may not contain
sufficient evidence for judging their relevance to a category, potentially
providing noisy signal for relevance attributable to the document text, 2) low
recall: entities may be missing from categories to which they belong. For
about half of the dataset, we crowdsource relevance labels and attribution
based on document text, and investigate recall through manual error analysis
(§5).

We select four domains to represent some diversity in queries: films, books,
animals and plants. Focusing on four rather than all possible domains enables
higher quality control. The former two model a general search scenario, while
the latter two model a scientific search scenario.

| Films | Books | Plants | Animals | All
---|---|---|---|---|---
Num. Queries | 896 | 870 | 802 | 789 | 3357
Num. Entities | 146368 | 50784 | 83672 | 44681 | 325505
Avg. Query Len. | 8.68 | 7.93 | 8.94 | 9.09 | 8.64
Avg. Doc. Len. | 532.2 | 655.3 | 258.1 | 293.1 | 452.2
Avg. Ans. Set Size | 8.8 | 8.6 | 12.2 | 12.6 | 10.5

Table 2: Statistics of examples in Quest across different domains.

###  3.2 Introducing set operations

To construct queries with set operations, we define templates that represent
plausible combinations of atomic queries. Denoting atomic queries as A, B and
C, our templates and corresponding examples from different domains are listed
in Table 1. Templates were constructed by composing three basic set operations
(intersection, union and difference). They were chosen to ensure unambiguous
interpretations of resulting queries by omitting those combinations of set
operations that are non-associative.

Below we describe the logic behind sampling atomic queries (i.e., $A$, $B$,
$C$) for composing complex queries, with different set operations. In all
cases, we ensure that answer sets contain between 2-20 entities so that
crowdsourcing relevance judgements is feasible. We sample 200 queries per
template and domain, for a total of 4200 initial queries. The dataset is split
into train + validation (80-20 split) and testing equally. In each of these
sets, we sampled an equal number of queries per template.

##### Intersection.

The intersection operation for a template $A\cap B$ is particularly
interesting and potentially challenging when both $A$ and $B$ have large
answer sets but their intersection is small. We require the minimum answer set
sizes of each $A$ and $B$ to be fairly large (>50 entities), while their
intersection to be small (2-20 entities).

##### Difference.

Similar to intersection, we require the answer sets for both $A$ and $B$ to be
substantial (>50 entities), but also place maximum size constraints on both
$A$ (<200 entities) and $B$ (<10000 entities) as very large categories tend to
suffer from recall issues in Wikipedia. We also limit the intersection of $A$
and $B$ (see reasoning in Appendix B).

##### Union.

For the union operation, we require both $A$ and $B$ to be well-represented
through the entities in the answer set for their union $A\cup B$. Hence, we
require both $A$ and $B$ to have at least 3 entities. Further, we require
their intersection to be non-zero but less than 1/3rd of their union. This is
so that $A$ and $B$ are somewhat related queries.

For all other templates that contain compositions of the above set operations,
we apply the same constraints recursively. For example, for $A\cap B\setminus
C$, we sample atomic queries $A$ and $B$ for the intersection operation, then
sample $C$ based on the relationship between $A\cap B$ and $C$.

###  3.3 Annotation Tasks

Automatically generating queries based on templates results in queries that
are not always fluent and coherent. Further, entities mapped to a query may
not actually be relevant and don’t always have attributable evidence for
judging their relevance. We conduct crowdsourcing to tackle these issues. The
annotation tasks aim at ensuring that 1) queries are fluent, unambiguous and
contain diverse natural language logical connectives, (2) entities are
verified as being relevant or non-relevant and (3) relevance judgements are
attributed to document text for each relevant entity. Crowdsourcing is
performed in three stages, described below. More annotation details and the
annotation interfaces can be found in Appendix C.

####  3.3.1 Paraphrasing

Crowdworkers were asked to paraphrase a templatically generated query so that
the paraphrased query is fluent, expresses all constraints in the original
query, and clearly describes what a user could be looking for. This annotation
was done by one worker per query.

####  3.3.2 Validation

This stage is aimed at validating the queries we obtain from the paraphrasing
stage. Crowdworkers were given queries from the first stage and asked to label
whether the query is 1) fluent, 2) equivalent to the original templatic query
in meaning, and 3) rate its naturalness (how likely it is to be issued by a
real user). This annotation was done by 3 workers per query. We excluded those
queries which were rated as not fluent, unnatural or having a different
meaning than the original query, based on a majority vote. Based on the
validation, we removed around around 11% of the queries from stage 1.

Figure 3: We compare several systems consisting of a retriever for efficiently
selecting a set of candidates from the document corpus and a document
relevance classifier for determining the final predicted document set.

####  3.3.3 Relevance Labeling

Next, crowdworkers were asked to provide relevance judgements for the
automatically determined answer sets of queries. Specifically, they were given
a query and associated entities/documents, and asked to label their relevance
on a scale of 0-3 (definitely not relevant, likely not relevant, likely
relevant, definitely relevant). They were asked to ensure that relevance
should mostly be inferred from the document, but they could use some
background knowledge and do minimal research.

We also asked them to provide attributions for document relevance.
Specifically, we ask them to first label whether the document provides
sufficient evidence for the relevance of the entity (complete/partial/no).
Then, for different phrases in the query (determined by the annotator), we ask
them to mark sentence(s) in the document that indicate its relevance. The
attribution annotation is broadly inspired by Rashkin et al. (2021). For
negated constraints, we ask annotators to mark attributable sentences if they
provide counter-evidence. Since this annotation was time-intensive, we
collected these annotations for two domains (films and books). We found that
relevance labeling was especially difficult for the plants and animals
domains, as they required more specialized scientific knowledge. In our pilot
study prior to larger scale data collection, we collected 3 relevance ratings
from different annotators for 905 query and document pairs from the films
domain. In 61.4% of cases, all 3 raters judged the document to be “Definitely
relevant” or “Likely relevant” or all 3 raters judged the document to be
“Definitely not relevant” or “Likely not relevant”. The Fleiss’ kappa metric
on this data was found to be K=0.43. We excluded all entities which were
marked as likely or definitely not relevant to a query based on the document
text from its answer set. Around 23.7% of query-document pairs from stage 2
were excluded.

###  3.4 Dataset Statistics

Basic dataset statistics are reported in Table 2. The dataset contains more
entities from the films domain, because this domain is more populated in
Wikipedia. The average length of queries is 8.6 words and the average document
length is 452 words. Documents from the films and books domains are longer on
average, as they often contain plots and storylines. Around $\sim$69% of
entities have complete evidence and $\sim$30% have partial evidence. Evidence
was labeled as partial when not all phrases in the query had explicit evidence
in the document (i.e., they may require background knowledge or reasoning).
There are on average 33.2 words attributed for each entity with the maximum
attribution text span ranging up to length 1837 words. Finally, the average
answer set size is 10.5 entities.

###  3.5 Additional Training Examples

Beyond the annotated data, we generated additional synthetic examples for
training. We found including such examples improved model performance, and we
include these examples for the experiments in §4. To generate these examples,
we sample 5000 atomic queries from all domains, ensuring that they do not
already appear as sub-queries in any of the queries in Quest and use their
corresponding entities in Wikipedia as their relevant entity set.

##  4 Experimental Setup

We evaluate modern retrieval systems to establish baseline performances. We
also perform extensive error analysis to understand patterns of model errors
and the quality of the labels in Quest.

###  4.1 Task Definition

We consider a corpus, $\mathcal{E}$, that contains entities across all domains
in the dataset. Each entity is accompanied with a document based on its
Wikipedia page. An example in our dataset consists of a query, $x$, and an
annotated set of relevant entities, $y\subset\mathcal{E}$. As described in §3,
for all examples $|y|<20$. Our task is to develop a system that, given
$\mathcal{E}$ and a query $x$, predicts a set of relevant entities,
$\hat{y}\subset\mathcal{E}$.

###  4.2 Evaluation

Our primary evaluation metric is average $F_{1}$, which averages per-example
$F_{1}$ scores. We compute $F_{1}$ for each example by comparing the predicted
set of entities, $\hat{y}$, with the annotated set, $y$.

Retriever (K=100) | Classifier | Avg. Precision | Avg. Recall | Avg. F1
---|---|---|---|---
BM25 | T5-Base | 0.168 | 0.160 | 0.141
BM25 | T5-Large | 0.178 | 0.168 | 0.150
T5-Large DE | T5-Base | 0.153 | 0.354 | 0.176
T5-Large DE | T5-Large | 0.165 | 0.368 | 0.192

Table 3: Average Precision, Recall, and F1 of baseline systems evaluated on
the test set.

| Avg. Recall@K | MRecall@K
---|---|---
Retriever | 20 | 50 | 100 | 1000 | 20 | 50 | 100 | 1000
BM25 | 0.104 | 0.153 | 0.197 | 0.395 | 0.020 | 0.030 | 0.037 | 0.087
T5-Base DE | 0.255 | 0.372 | 0.455 | 0.726 | 0.045 | 0.088 | 0.127 | 0.360
T5-Large DE | 0.265 | 0.386 | 0.476 | 0.757 | 0.047 | 0.100 | 0.142 | 0.408

Table 4: Average Recall and MRecall of various retrievers.

###  4.3 Baseline Systems

We evaluated several combinations of retrievers and classifiers, as shown in
Figure 3. For the retriever component, we consider a sparse BM25 retriever
Robertson et al. (2009) and a dense dual encoder retriever (denoted DE).
Following Ni et al. (2022), we initialize our dual encoder from a T5 Raffel et
al. (2020) encoder and train with an in-batch sampled softmax loss Henderson
et al. (2017). Once we have a candidate set, we need to determine a set of
relevant entities. To classify relevance of each candidate document for the
given query, we consider a cross-attention model which consists of a T5
encoder and decoder.444Scores from BM25 and dual encoders trained with a
softmax loss are not normalized to provide relevance probabilities for
documents. We found that naively applying a global threshold to these scores
to produce answer sets did not perform as well as using a classifier trained
with a binary cross-entropy loss to predict document relevance. We train the
cross-attention classifier using a binary cross-entropy loss with negative
examples based on non-relevant documents in top 1,000 documents retrieved by
BM25 and random non-relevant documents (similarly to Nogueira and Cho (2019)).
As cross-attention classification for a large number of candidates is
computationally expensive, we restrict BM25 and the dual encoder to retrieve
100 candidates which are then considered by the cross-attention classifier. As
our T5-based dual encoder can only efficiently accommodate up to 512 tokens,
we truncate document text. We discuss the impact of this and alternatives in
§5. Further, since T5 was pre-trained on Wikipedia, we investigate the impact
of memorization in Appendix D. Additional details and hyperparameter settings
are in Appendix A.

###  4.4 Manual Error Annotation

For the best overall system, we sampled errors and manually annotated 1145
query-document pairs from the validation set. For the retriever, we sampled
relevant documents not included in the top-100 candidate set and non-relevant
documents ranked higher than relevant ones. For the classifier, we sampled
false positive and false negative errors made in the top-100 candidate set.
This annotation process included judgements of document relevance (to assess
agreement with the annotations in the dataset) and whether the document (and
the truncated version considered by the dual encoder or classifier) contained
sufficient evidence to reasonably determine relevance. We also annotated
relevance for each constraint within a query. We discuss these results in §5.

##  5 Results and Analysis

We report the performance of our baseline systems on the test set in Table 3.
In this section, we summarize the key findings from our analysis of these
results and the error annotation described in §4.4.

##### Dual encoders outperform BM25.

As shown in Table 3, the best overall system uses a T5-Large Dual Encoder
instead of BM25 for retrieval. The performance difference is even more
significant when comparing recall of Dual Encoders and BM25 directly. We
report average recall (average per-example recall of the full set of relevant
documents) and MRecall Min et al. (2021) (the percentage of examples where the
candidate set contains all relevant documents), over various candidate set
sizes in Table 4.

##### Retrieval and classification are both challenging.

As we consider only the top-100 candidates from the retriever, the retriever’s
recall@100 sets an upper bound on the recall of the overall system. Recall@100
is only 0.476 for the T5-Large Dual Encoder, and the overall recall is further
reduced by the T5-Large classifier to 0.368, despite achieving only 0.165
precision. This suggests that there is room for improvement from both stages
to improve overall scores. As performance improves for larger T5 sizes for
both retrieval and classification, further model scaling could be beneficial.

##### Models struggle with intersection and difference.

We also analyzed results across different templates and domains, as shown in
Table 5. Different constraints lead to varying distributions over answer set
sizes and the atomic categories used. Therefore, it can be difficult to
interpret differences in F1 scores across templates. Nevertheless, we found
the queries with set union have the highest average F1 scores. Queries with
set intersection have the lowest average F1 scores, and queries with set
difference also appear to be challenging.

To analyze why queries with conjunction and negation are challenging, we
labeled the relevance of individual query constraints (§4.4), where a system
incorrectly judges relevance of a non-relevant document. The results are
summarized in Table 6. For a majority of false positive errors involving
intersection, at least one constraint is satisfied. This could be interpreted
as models incorrectly treating intersection as union when determining
relevance. Similarly, for a majority of examples with set difference, the
negated constraint is not satisfied. This suggests that the systems are not
sufficiently sensitive to negations.

Template | Films | Books | Plants | Animals | All
---|---|---|---|---|---
$A$ | 0.231 | 0.436 | 0.209 | 0.214 | 0.274
$A\cup B$ | 0.264 | 0.366 | 0.229 | 0.271 | 0.282
$A\cap B$ | 0.115 | 0.138 | 0.049 | 0.063 | 0.092
$A\setminus B$ | 0.177 | 0.188 | 0.216 | 0.204 | 0.193
$A\cup B\cup C$ | 0.200 | 0.348 | 0.306 | 0.294 | 0.287
$A\cap B\cap C$ | 0.086 | 0.121 | 0.07 | 0.065 | 0.086
$A\cap B\setminus C$ | 0.119 | 0.112 | 0.121 | 0.136 | 0.122
All | 0.171 | 0.248 | 0.165 | 0.182 | 0.192

Table 5: F1 of our strongest baseline (T5-Large DE + T5-Large Classifier)
across templates and domains.

##### There is significant headroom to improve both precision and recall.

As part of our manual error analysis (§4.4), we made our own judgements of
relevance and measured agreement with the relevance annotations in Quest. As
this analysis focused on cases where our best system disagreed with the
relevance labels in the dataset, we would expect agreement on these cases to
be significantly lower than on randomly selected query-document pairs in the
dataset. Therefore, it provides a focused way to judge the headroom and
annotation quality of the dataset.

For false negative errors, we judged 91.1% of the entities to be relevant for
the films and books domains, and 81.4% for plants and animals. Notably, we
collected relevance labels for the films and books domains and removed some
entities based on these labels, as described in §3, which likely explains the
higher agreement for false negatives from these domains. This indicates
significant headroom for improving recall as defined by Quest, especially for
the domains where we collected relevance labels.

For false positive errors, we judged 28.8% of the entities to be relevant,
showing a larger disagreement with the relevance labels in the dataset. This
is primarily due to entities not included in the entity sets derived from the
Wikipedia category taxonomy (97.7%), rather than entities removed due to
relevance labeling. This is a difficult issue to fully resolve, as it is not
feasible to exhaustively label relevance for all entities to correct for
recall issues in the Wikipedia category taxonomy. Future work can use pooling
to continually grow the set of relevant documents Sparck Jones and Van
Rijsbergen (1975). Despite this, our analysis suggests there is significant
headroom for improving precision, as we judged a large majority of the false
positive predictions to be non-relevant.

| # Constraints |
---|---|---
| 1 | 2 | 3 | Neg.
Retriever |  |  |  |
$A\cap B$ | 63.5 | 36.5 | — | —
$A\cap B\cap C$ | 56.5 | 37.0 | 6.5 | —
$A\setminus B$ | 80.3 | 19.7 | — | 59.1
$A\cap B\setminus C$ | 47.6 | 40.5 | 11.9 | 26.2
Classifier |  |  |  |
$A\cap B$ | 83.3 | 16.7 | — | —
$A\cap B\cap C$ | 73.2 | 22.0 | 4.9 | —
$A\setminus B$ | 81.0 | 19.1 | — | 38.1
$A\cap B\setminus C$ | 95.5 | 4.6 | 0.0 | 68.2

Table 6: Analysis of false positive errors from the T5-Large classifier and
cases where a non-relevant document was ranked ahead of a relevant one for the
T5-Large dual encoder. For queries with conjunction, we determined the
percentage of cases where 1, 2, or 3 constraints in the template were not
satisfied by the predicted document (# Constraints). For queries with
negation, we measured the percentage of cases where the negated constraint
(Neg.) was not satisfied.

##### Truncating document text usually provides sufficient context.

In our experiments, we truncate document text to 512 tokens for the dual
encoder, and 384 tokens for the classifier to allow for the document and query
to be concatenated. Based on our error analysis (§4.4), out of the documents
with sufficient evidence to judge relevance, evidence occurred in this
truncated context 93.2% of the time for the dual encoder, and 96.1% of the
time for the classifier. This may explain the relative success of this simple
baseline for handling long documents. We also evaluated alternative strategies
but these performed worse in preliminary experiments555For the dual encoder,
we split documents into overlapping chunks of 512 tokens, and aggregated
scores at inference Dai and Callan (2019). For the cross-attention model, we
evaluated using BM25 to select the top-3 passages of length 128.. Future work
can evaluate efficient transformer variants Guo et al. (2022); Beltagy et al.
(2020).

##  6 Conclusion

We present Quest, a new benchmark of queries which contain implicit set
operations with corresponding sets of relevant entity documents. Our
experiments indicate that such queries present a challenge for modern
retrieval systems. Future work could consider approaches that have better
inductive biases for handling set operations in natural language expressions
(for example, Vilnis et al. (2018)). The attributions in Quest can be
leveraged for building systems that can provide fine-grained attributions at
inference time. The potential of pretrained generative LMs and multi-evidence
aggregation methods to answer set-seeking selective queries, while providing
attribution to sources, can also be investigated.

##  7 Limitations

##### Naturalness.

Since our dataset relies on the Wikipedia category names and semi-
automatically generated compositions, it does not represent an unbiased sample
from a natural distribution of real search queries that contain implicit set
operations. Further, we limit attention to non-ambiguous queries and do not
address the additional challenges that arise due to ambiguity in real search
scenarios. However, the queries in our dataset were judged to plausibly
correspond to real user search needs and system improvements measured on Quest
should correlate with improvements on at least a fraction of natural search
engine queries with set operations.

##### Recall.

We also note that because Wikipedia categories have imperfect recall of all
relevant entities (that contain sufficient evidence in their documents),
systems may be incorrectly penalised for predicted relevant entities assessed
as false positive. We quantify this in section 5. We have also limited the
trusted source for an entity to its Wikipedia document but entities with
insufficient textual evidence in their documents may still be relevant.
Ideally, multiple trusted sources could be taken into account and evidence
could be aggregated to make relevance decisions. RomQA Zhong et al. (2022)
takes a step in this latter direction although the evidence attribution is not
manually verified.

##### Answer Set Sizes.

To ensure that relevance labels are correct and verifiable, we seek the help
of crowdworkers. However, this meant that we needed to restrict the answer set
sizes to 20 for the queries in our dataset, to make annotation feasible. On
one hand, this is realistic for a search scenario because users may only be
interested in a limited set of results. On the other hand, our dataset does
not model a scenario where the answer set sizes are much larger.

## Acknowledgements

We would like to thank Isabel Kraus-Liang, Mahesh Maddinala, Andrew Smith,
Daphne Domansi, and all the annotators for their work. We would also like to
thank Mark Yatskar, Dan Roth, Zhuyun Dai, Jianmo Ni, William Cohen, Andrew
McCallum, Shib Sankar Dasgupta and Nicholas Fitzgerald for useful discussions.

## References

  * Amouyal et al. (2022) Samuel Joseph Amouyal, Ohad Rubin, Ori Yoran, Tomer Wolfson, Jonathan Herzig, and Jonathan Berant. 2022.  Qampari:: An open-domain question answering benchmark for questions with many answers from multiple paragraphs.  _ArXiv, abs/2205.12665_.
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.  _arXiv:2004.05150_.
  * Berant et al. (2013) Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013.  Semantic parsing on Freebase from question-answer pairs.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 1533–1544, Seattle, Washington, USA. Association for Computational Linguistics.
  * Craswell et al. (2020) Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. 2020.  Overview of the TREC 2019 deep learning track.  _arXiv preprint arXiv:2003.07820_.
  * Dai and Callan (2019) Zhuyun Dai and Jamie Callan. 2019.  Deeper text understanding for ir with contextual neural language modeling.  In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 985–988.
  * Gu et al. (2021) Yu Gu, Sue Kase, Michelle Vanni, Brian Sadler, Percy Liang, Xifeng Yan, and Yu Su. 2021.  Beyond iid: three levels of generalization for question answering on knowledge bases.  In _Proceedings of the Web Conference 2021_ , pages 3477–3488.
  * Guo et al. (2022) Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2022.  LongT5: Efficient text-to-text transformer for long sequences.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 724–736, Seattle, United States. Association for Computational Linguistics.
  * Henderson et al. (2017) Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-Hsuan Sung, László Lukács, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017.  Efficient natural language response suggestion for smart reply.  _arXiv preprint arXiv:1705.00652_.
  * Keysers et al. (2020) Daniel Keysers, Nathanael Schärli, Nathan Scales, Hylke Buisman, Daniel Furrer, Sergii Kashubin, Nikola Momchev, Danila Sinopalnikov, Lukasz Stafiniak, Tibor Tihon, Dmitry Tsarkov, Xiao Wang, Marc van Zee, and Olivier Bousquet. 2020.  Measuring compositional generalization: A comprehensive method on realistic data.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Khashabi et al. (2018) Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018.  Looking beyond the surface: A challenge set for reading comprehension over multiple sentences.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 252–262, New Orleans, Louisiana. Association for Computational Linguistics.
  * Kong et al. (2022) Weize Kong, Swaraj Khadanga, Cheng Li, Shaleen Gupta, Mingyang Zhang, Wensong Xu, and Mike Bendersky. 2022.  Multi-aspect dense retrieval.  In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining_.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Lamm et al. (2021) Matthew Lamm, Jennimaria Palomaki, Chris Alberti, Daniel Andor, Eunsol Choi, Livio Baldini Soares, and Michael Collins. 2021.  QED: A Framework and Dataset for Explanations in Question Answering.  _Transactions of the Association for Computational Linguistics_ , 9:790–806.
  * Lan et al. (2021) Yunshi Lan, Gaole He, Jinhao Jiang, Jing Jiang, Wayne Xin Zhao, and Ji-Rong Wen. 2021.  A survey on complex knowledge base question answering: Methods, challenges and solutions.  _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI-21)_.
  * Min et al. (2021) Sewon Min, Kenton Lee, Ming-Wei Chang, Kristina Toutanova, and Hannaneh Hajishirzi. 2021.  Joint passage ranking for diverse multi-answer retrieval.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6997–7008, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Nguyen et al. (2016) Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  Ms marco: A human generated machine reading comprehension dataset.  In _CoCo@ NIPs_.
  * Ni et al. (2022) Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and Yinfei Yang. 2022.  Large dual encoders are generalizable retrievers.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9844–9855, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Nogueira and Cho (2019) Rodrigo Nogueira and Kyunghyun Cho. 2019.  Passage re-ranking with bert.  _arXiv preprint arXiv:1901.04085_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21:1–67.
  * Rashkin et al. (2021) Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. 2021\.  Measuring attribution in natural language generation models.
  * Robertson et al. (2009) Stephen Robertson, Hugo Zaragoza, et al. 2009.  The probabilistic relevance framework: Bm25 and beyond.  _Foundations and Trends® in Information Retrieval_ , 3(4):333–389.
  * Sparck Jones and Van Rijsbergen (1975) K. Sparck Jones and C. J. Van Rijsbergen. 1975.  Report on the need for and provision of an ideal information retrieval test collection.
  * Sun et al. (2019) Haitian Sun, Tania Bedrax-Weiss, and William Cohen. 2019.  PullNet: Open domain question answering with iterative retrieval on knowledge bases and text.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2380–2390, Hong Kong, China. Association for Computational Linguistics.
  * Talmor and Berant (2018) Alon Talmor and Jonathan Berant. 2018.  The web as a knowledge-base for answering complex questions.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 641–651, New Orleans, Louisiana. Association for Computational Linguistics.
  * Thakur et al. (2021) Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021.  BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Vilnis et al. (2018) Luke Vilnis, Xiang Li, Shikhar Murty, and Andrew McCallum. 2018.  Probabilistic embedding of knowledge graphs with box lattice measures.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 263–272, Melbourne, Australia. Association for Computational Linguistics.
  * Watanabe et al. (2017) Yusuke Watanabe, Bhuwan Dhingra, and Ruslan Salakhutdinov. 2017.  Question answering from unstructured text by retrieval and comprehension.  _arXiv preprint arXiv:1703.08885_.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics.
  * Yih et al. (2016) Wen-tau Yih, Matthew Richardson, Chris Meek, Ming-Wei Chang, and Jina Suh. 2016\.  The value of semantic parse labeling for knowledge base question answering.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 201–206, Berlin, Germany. Association for Computational Linguistics.
  * Zhong et al. (2022) Victor Zhong, Weijia Shi, Wen-tau Yih, and Luke Zettlemoyer. 2022.  RoMQA: A benchmark for robust, multi-evidence, multi-answer question answering.  _arXiv preprint arXiv:2210.14353_.

##  Appendix A Experimental Details and Hyperparameters

All models were fine-tuned starting from T5 1.1 checkpoints
666https://github.com/google-research/t5x/blob/main/docs/models.md. We fine-
tune T5 models on 32 Cloud TPU v3 cores777https://cloud.google.com/tpu/. Fine-
tuning takes less than 8 hours for all models.

##### Dual Encoder.

We used the t5x_retrieval library 888https://github.com/google-
research/t5x_retrieval for implementing dual encoder models. We tuned some
parameters based on results on the validation set. Relevant hyperparameters
for training the dual encoder are:

  * •

Learning Rate: 1e-3

  * •

Warmup Steps: 1500

  * •

Finetuning Steps: 15000

  * •

Batch Size: 512

  * •

Max Query Length: 64

  * •

Max Candidate Length: 512

##### Classifier.

For negative examples, we sampled 250 random non-relevant documents and
sampled 250 non-relevant documents from the top-1000 documents retrieved by
BM25. We also replicated each positive example 50 times. We found an
approximately even number of positive and negative examples lead to better
performance than training with a large class imbalance. We found a combination
of random negatives and negatives from BM25 performed better than using only
either individual type of negative examples. Additionally, selecting negative
examples from BM25 performed better than selecting negative examples from the
T5-Large dual encoder.

For the T5 input we concatenated the query and truncated document text. The T5
output is the string “relevant” or “not relevant”. To classify document
relevance at inference time, we applied a threshold to the probability
assigned to the “relevant” label, which we tuned on the validation set. When
classifying BM25 candidates we used a threshold of $0.9$ and when classifying
the dual encoder candidates we used a threshold of $0.95$.

Other relevant hyperparameters for training the classifier are:

  * •

Learning Rate: 1e-3

  * •

Warmup Steps: 1000

  * •

Finetuning Steps: 10000

  * •

Batch Size: 1024

  * •

Max Source Length: 512

  * •

Max Target Length: 16

##  Appendix B Set Difference and Recall

##### Notation and Assumptions

Let us assume we have two sets derived from the Wikipedia category graph,
$\hat{A}$ and $\hat{B}$. The Wikipedia category graph can be missing some
relevant entities, such that $\hat{A}\subset A$ and $\hat{B}\subset B$, where
$A$ and $B$ are interpreted as the hypothetical sets containing _all_ relevant
entities. We quantify the degree of missing entities by denoting recall as
$r_{A}$ and $r_{B}$, such that $|\hat{A}|=r_{A}*|A|$ and
$|\hat{B}|=r_{B}*|B|$. We quantify the fraction of elements in $A$ that are
also in $B$ as $r_{\cap}$, such that $|A\cap B|=r_{\cap}*|A|$. For simplicity,
we also assume that the overlap between $\hat{A}$ and $\hat{B}$ is such that
$|\hat{A}\cap B|=r_{A}*|A\cap B|$ and $|\hat{A}\cap\hat{B}|=r_{A}*r_{B}*|A\cap
B|$.

##### Derivation

_What is the recall ( $r$) and precision ($p$) of $\hat{A}\setminus\hat{B}$
relative to $A\setminus B$ as a function of $r_{A}$, $r_{B}$, and $r_{\cap}$?_

First, we derive this function for recall:999We note some useful properties of
pairs of sets $X$ and $Y$: $X\setminus Y$ = $X\cap Y^{c}$, $|X\setminus
Y|=|X|-|X\cap Y|$, if $X\subset Y$ then $X\cap Y=X$, and if $X\subset Y$ then
$Y^{c}\subset X^{c}$.

$r=\frac{|(A\setminus B)\cap(\hat{A}\setminus\hat{B})|}{|(A\setminus B)|}$
$r=\frac{|(\hat{A}\setminus B)|}{|(A\setminus B)|}$

$r=\frac{|\hat{A}|-|\hat{A}\cap B|}{|A|-|A\cap B|}$
$r=\frac{r_{A}*|A|-r_{A}*r_{\cap}*|A|}{|A|-(r_{\cap}*|A|)}$

$r=\frac{r_{A}*(1-r_{\cap})*|A|}{(1-r_{\cap})*|A|}$ $\boxed{r=r_{A}}$

And for precision:

$p=\frac{|(A\setminus
B)\cap(\hat{A}\setminus\hat{B})|}{|(\hat{A}\setminus\hat{B})|}$
$p=\frac{|(\hat{A}\setminus B)|}{|(\hat{A}\setminus\hat{B})|}$

$p=\frac{|\hat{A}|-|\hat{A}\cap B|}{|\hat{A}|-|\hat{A}\cap\hat{B}|}$

$p=\frac{r_{A}*|A|-r_{A}*r_{\cap}*|A|}{r_{A}*|A|-r_{A}*r_{B}*r_{\cap}*|A|}$

$p=\frac{r_{A}*(1-r_{\cap})*|A|}{r_{A}*(1-r_{B}*r_{\cap})*|A|}$
$\boxed{p=\frac{(1-r_{\cap})}{(1-r_{B}*r_{\cap})}}$

##### Discussion

While recall is simply equal to $r_{A}$, precision is a more complicated
function of $r_{B}$ and $r_{\cap}$, and can be very low for large values of
$r_{\cap}$. Intuitively, if subtracting $\hat{B}$ from $\hat{A}$ removes most
of $\hat{A}$, then the precision of the resulting set will be dominated by the
relevant entities missing from $\hat{B}$. This motivates limiting the
intersection of the two sets used to construct queries involving set
intersection. For example, if $r_{B}=0.95$, then with $r_{\cap}<0.8$, we can
ensure $p>0.83$.

##  Appendix C Annotation Details

The annotation tasks in Quest were carried out by participants who were paid
contractors. They are based in Austin, TX and either have a bachelor’s degree
(55%) or equivalent work experience (45%). They were paid by the hour for
their work and were recruited from a vendor who screened them for knowledge of
US English. They were informed of how their work would be used and could opt
out. They received a standard contracted wage, which complies with living wage
laws in their country of employment. The annotation interfaces presented to
the annotators are shown in Figures 4, 5 and 6.

Figure 4: Annotation interface for the paraphrasing stage. Figure 5:
Annotation interface for the validation stage. Figure 6: Annotation interface
for the relevance labeling stage.

##  Appendix D Impact of Memorization of Pre-training Data

Since the T5 checkpoints we use to initialize our models were pre-trained on
the C4 corpus (which includes Wikipedia), we investigate whether these models
have memorized aspects of the Wikipedia category graph. We compare recall of
the T5-based dual encoder model for Wikipedia documents that were created
prior to the pre-training date of the T5 checkpoint compared with documents
that were added after pre-training. We report these in Table 7, along with the
recalls for the same sets of documents with a BM25 retriever, for a baseline
comparison. We note that the ratio of scores between the documents added
before pre-training to documents added after pre-training is similar for both
systems, which suggests factors other than memorization may explain the
difference. For example, the documents created before vs. after the pre-
training date have average lengths of 759.7 vs. 441.2 words, respectively.

| Avg. Recall@100
---|---
Retriever | Before | After
BM25 | 0.183 | 0.050
T5-Large DE | 0.466 | 0.171

Table 7: Average recall@100 on the subsets of documents created before vs
after T5 pre-training.


# Skyrmion Dynamics in the Presence of Deformation

Zehan Chen  Department of Physics, The Hong Kong University of Science and
Technology, Hong Kong, China  Xichao Zhang  Department of Electrical and
Computer Engineering, Shinshu University, 4-17-1 Wakasato, Nagano 380-8553,
Japan  Yan Zhou  School of Science and Engineering, The Chinese University of
Hong Kong, Shenzhen, Guangdong 518172, China  Qiming Shao<EMAIL_ADDRESS>Department of Physics, The Hong Kong University of Science and Technology,
Hong Kong, China  Department of Electronic and Computer Engineering, The Hong
Kong University of Science and Technology, Hong Kong, China  Guangdong-Hong
Kong-Macao Joint Laboratory for Intelligent Micro-Nano Optoelectronic
Technology, The Hong Kong University of Science and Technology, Hong Kong,
China

###### Abstract

Magnetic skyrmions are topological spin textures promising for future high-
density and non-volatile memory. It is crucial to understand the current-
driven skyrmion dynamics in the presence of deformation, of which an
analytical model, however, remains elusive. Here we extend Thiele’s model by
considering both the radial and tangential forces. Our model attributes the
skyrmion deformation to the current-induced rotational symmetry breaking,
which includes magnetization canting and domain wall width variation. Our
predictions of skyrmion radius and nonlinear dynamics are consistent with
micromagnetic simulation results. Besides, we show that by applying an in-
plane magnetic field, the deformation of a skyrmion can be suppressed, and
even the compression of a skyrmion can be achieved. Our model provides a
generic way to analyze the skyrmion deformation and may inspire applications
based on nonlinear skyrmion dynamics.

Introduction.— Skyrmion is a topological soliton first proposed by Skyrme to
solve the nonlinear sigma model [1]. Recently, magnetic skyrmions were
generated and observed in B20 chiral magnets and ferromagnet/heavy metal
bilayer systems with Dzyaloshinskii-Moriya interaction (DMI) [2, 3, 4, 5, 6,
7, 8]. Due to its non-trivial topological charge, skyrmion is topologically
protected and can be driven by spin-transfer torque (STT) at ultralow current
densities [9, 10, 11], promising for information carrier in non-volatile
memory and other spintronic devices [12, 13, 14, 15].

Manipulation of skyrmions can be achieved by current injection in a
ferromagnet/heavy metal bilayer [16, 17, 18, 19]. The charge current in the
heavy metal layer with strong spin-orbit coupling induces a spin current in
out-of-plane direction, exerting spin-orbit torque (SOT) on the ferromagnetic
layer to drive the skyrmion. For high-speed skyrmion applications, high drive
current densities are needed. Experimental results have suggested that
skyrmion deformation can occur during the large current-driven motion [19, 18,
20]. Theoretically, STT- or SOT-induced skyrmion deformation has been studied
using micromagnetic simulations and semianalytical approach [21, 22, 23].
Semianalytical means that micromagnetic simulation is still needed in
analytical modeling.

Although the Thiele’s approach [24] is very successful in explaining the
current-driven skyrmion motion at low drive currents [25, 11, 26], and it has
been extended to include antiferromagnetic spin textures [27, 28], magnon
effect [29], etc., there is no pure analytical model describing the
deformation and nonlinear dynamics of current-driven skyrmions. It is highly
desired to extend Thiele’s approach to better understand the skyrmion dynamics
in the presence of high current-induced deformation [30]. In this work, we
establish an analytical model including current-induced radial and tangential
forces and successfully predict skyrmion radius and critical current density
without the need of micromagnetic simulation. Exceeding this critical current
density, the skyrmion breaks down. Induced by tangential Thiele’s force
density, the magnetization canting contributes the most to the symmetry
breaking. We also find that the domain wall width depends on the skyrmion
radius. Unlike most studies that focus on the translational part integration
of Thiele’s equation leading to skyrmion motion velocity, we calculate the
integration of the radial part of Thiele’s equation and obtain skyrmion
radius. With the obtained skyrmion radius, we can accurately describe
nonlinear skyrmion dynamics. Moreover, we propose a method to suppress the
deformation by applying an in-plane magnetic field.

Model and Methods.— The dynamics of a ferromagnetic system is governed by the
Landau–Lifshitz–Gilbert (LLG) equation [31] with a damping-like SOT term,

$\frac{\partial\bm{m}}{\partial t}=-\gamma_{0}\bm{m}\times\bm{H}^{\rm
eff}+\alpha\bm{m}\times\frac{\partial\bm{m}}{\partial t}-\tau_{\rm
ad}\bm{m}\times(\bm{m}\times\bm{p}),$ (1)

where $\gamma_{0}$ is the gyromagnetic ratio, $\bm{H}^{\rm eff}$ is the
effective field, $\alpha$ is the Gilbert damping constant, $\tau_{\rm ad}$ is
the damping-like spin orbit torque and $\bm{p}$ is the polarization direction.
Assume that $\bm{p}$ is along $+y$ direction, i.e. $\bm{p}=\bm{\hat{\jmath}}$
and electron current along $+x$ direction, i.e.
$\bm{j}_{e}=j_{x}\bm{\hat{\imath}}$, where
$\bm{\hat{\imath}},\bm{\hat{\jmath}}$ and $\bm{\hat{k}}$ are unit vectors in
Cartesian coordinates.

Using Thiele’s approach [24, 32], we can substitute each term in the LLG
equation with an equivalent field, and define corresponding force densities,
$f_{i}^{\alpha}=-\mu_{0}M_{S}\left(\partial_{i}\bm{m}\cdot\bm{H}^{\alpha}\right)$,
$i=x,y$, $\alpha$ runs $g,e,d,c$.
$\bm{H}^{g},\bm{H}^{e},\bm{H}^{d},\bm{H}^{c}$ are the effective fields
corresponding to the terms in Eq.(1), and the exact definitions are given in
Eqs.(S5)-(S8) in supplemental material [33].

A skyrmion in steady-state motion must satisfy
$\bm{f}^{g}+\bm{f}^{e}+\bm{f}^{d}+\bm{f}^{c}=\bm{0}$ everywhere, where
$\bm{f}^{g}$ is gyroscopic force density, $\bm{f}^{e}$ is effective-field
force density, $\bm{f}^{c}$ is current-induced SOT force density, and
$\bm{f}^{d}$ is dissipative force density. Moreover, these forces can be
decomposed into radial force and tangential force,
$\bm{f}=f_{r}\bm{\hat{r}}+f_{\varphi}\bm{\hat{\varphi}}$, where $\bm{\hat{r}}$
is the radial unit vector and $\bm{\hat{\varphi}}$ is the tangential unit
vector.

For simplicity, we define radial force, which is the integral of radial force
density, as,

$F_{r}^{\alpha}=\int_{\Sigma}\bm{f}^{\alpha}\cdot\bm{\hat{r}}{\rm
d}\bm{S}=\int_{0}^{2\pi}\int_{0}^{\infty}\bm{f}^{\alpha}\cdot\bm{\hat{r}}{\rm
d}r{\rm d}\varphi,$ (2)

where $\Sigma$ is the whole plane and $\bm{\hat{r}}$ is the unit radial vector
in the polar coordinates centered at the skyrmion center (the center of mass).
Since all the force densities balance each other, their radial force density
integrals or radial forces must balance as well, that
$F_{r}^{g}+F_{r}^{e}+F_{r}^{d}+F_{r}^{c}=0$. $F_{r}^{g}$, $F_{r}^{d}$ and
$F_{r}^{c}$ are related to current and skyrmion motion, which cause the
deformation of skyrmion. So we define the expanding force to be the sum of
these three radial forces, $F_{r}^{\rm exp}=F_{r}^{g}+F_{r}^{d}+F_{r}^{c}$. To
the contrary, $F_{r}^{e}$ is the restoring force, and we let $F_{r}^{\rm
res}=F_{r}^{e}$. It will be shown later that both $F_{r}^{\rm exp}$ and
$F_{r}^{\rm res}$ depend on skyrmion radius $R$ and domain wall width
parameter $w$. By solving

$F_{r}^{\rm exp}+F_{r}^{\rm res}=0,$ (3)

we are able to find $R$ and $w$. How to accurately solve Eq.(3) without using
micromagnetic simulation is the major contribution of this work.

Figure 1: (a) The expanding force density $\bm{f}^{\rm
exp}=\bm{f}^{g}+\bm{f}^{c}+\bm{f}^{d}$ and the restoring force density
$\bm{f}^{\rm res}$ balancing each other in steady-state motion. (b) The radial
parts and (c) the tangential parts of the expanding force density. The radial
parts relate to skyrmion radius variation and the tangential parts relate to
the magnetization canting.

The restoring force $F_{r}^{\rm res}=F_{r}^{e}$ is determined by effective
field $\bm{H}^{\rm eff}$. $\bm{H}^{\rm eff}$ contains four components, i.e.,
$\bm{H}^{\rm eff}=\bm{H}^{\rm exch}+\bm{H}^{\rm DMI}+\bm{H}^{\rm
anis}+\bm{H}^{\rm ext}$, which correspond to Heisenberg exchange interaction,
DMI, effective anisotropy, and external magnetic field. $\bm{H}^{\rm
exch}=(2A/\mu_{0}M_{S})\nabla^{2}\bm{m}$, $\bm{H}^{\rm
anis}=(2K/\mu_{0}M_{S})(\bm{m}\cdot\bm{\hat{k}})\bm{\hat{k}}$, where $A$ is
the exchange stiffness, $K$ is the effective anisotropy constant. Effective
anisotropy contains magnetocrystalline anisotropy and demagnetization field in
ultrathin film with perpendicular magnetization [34], that
$K=K_{u}-\frac{1}{2}\mu_{0}M_{S}^{2}$, where $K_{u}$ is the uniaxial
anisotropy constant. For Néel-type skyrmion stabilized by interfacial DMI,
$\bm{H}^{\rm
DMI}=(-2D/\mu_{0}M_{S})[(\nabla\bm{m})\bm{\hat{k}}-\nabla(\bm{m}\cdot\bm{\hat{k}})]$,
and for Bloch-type skyrmion stabilized by bulk DMI, $\bm{H}^{\rm
DMI}=(-2D/\mu_{0}M_{S})\nabla\times\bm{m}$ [26, 34].

Rotational Symmetry Breaking of Skyrmion.— If a steady-state motion skyrmion
takes a rotation-symmetric magnetization distribution similar to a ground
state skyrmion, the expanding force $F_{r}^{\rm exp}$ will be zero. SOT breaks
the rotational symmetry of skyrmion, and induces a non-trivial $F_{r}^{\rm
exp}$. We will show that the breaking of rotational symmetry comes from two
parts, i.e., magnetization canting and domain wall compression and expansion.

Here, we discuss the Néel-type skyrmion case, where skyrmion domain wall
profile is
$m_{z}=\cos\Theta,m_{r}=\sin\Theta,\Theta(r)=2\arctan[\sinh(R/w)/\sinh(r/w)]$.
If no in-plane external field or current is applied, the direction of
magnetization at each point of a ground state skyrmion will have no tangential
component. However, a current will induce magnetization canting as shown in
Fig.2(a). The effect of an in-plane magnetic field is similar to that of a
current [21].

Figure 2: (a) Skyrmion canting configuration. In ground state, any
magnetization should align with unit radial vector $\bm{\hat{r}}$, but canting
occurs after applying current or in-plane field. A characteristic vector
$\bm{\delta}$ is used to denote the amplitude and direction of canting, which
is related to the amplitude and phase in (b). (b) Canting angle $\Delta\theta$
as a function of $\varphi$ under different current densities. The triangular
dots are simulation results, and the solid lines are fitted sinusoidal curves.
$\Delta\theta$ is defined as the angle between the radial unit vector
$\bm{\hat{r}}$ and the in-plane $\bm{m}$ where $m_{z}=0$, as illustrated in
(a). (c) Schematic of an intersection of moving and static skyrmion. $+x$
direction current injection will cause the left domain wall to compress and
the right domain wall to expand. (d) $w$ as a function of $\varphi$ under
different current densities. The triangular dots are simulation results, and
the solid lines are fitted sinusoidal curves. $w$ is uniformly distributed
only in ground state.

To quantitatively describe the canting, we use polar coordinates centered at
skyrmion center and let
$\bm{m}=m_{r}\bm{\hat{r}}+m_{\varphi}\bm{\hat{\varphi}}+m_{z}\bm{\hat{k}}$.
Note that the canting is equal to the tangential deviation of $\bm{m}$, i.e.
$m_{\varphi}$. We show below that the canting is determined by tangential
Thiele’s force densities, which sum up to be zero as

$f_{\varphi}^{c}+f_{\varphi}^{g}+f_{\varphi}^{d}+f_{\varphi}^{\rm
DMI}+f_{\varphi}^{\rm exch}+f_{\varphi}^{\rm anis}=0.$ (4)

Although Eq.(4) cannot be solved analytically, we can still estimate
$m_{\varphi}$. We ignore insignificant terms and obtain
$m_{\varphi}=m_{r}\left(\bm{\delta}\cdot\bm{\hat{\varphi}}\right)$ [33], where

$\bm{\delta}=\bm{\delta}_{\rm
current}=\frac{1}{2D}\frac{M_{S}}{\gamma}\left(\bm{v}\times\bm{\hat{k}}\right)$
(5)

is the characteristic vector of canting shown in Fig.2(a),
$\bm{v}=v_{x}\bm{\hat{\imath}}+v_{y}\bm{\hat{\jmath}}$ being the skyrmion
velocity. It suggests that the canting is proportional to the skyrmion
velocity. The canting vector $\bm{\delta}$, perpendicular to the velocity,
characterizes the amplitude of the asymmetrical deformation of the skyrmion
spin configuration.

With this estimation of $m_{\varphi}$, it can be shown that terms in the
expanding force $F_{r}^{\rm exp}$ are [33],

$\displaystyle F_{r}^{c}$ $\displaystyle=\pi^{2}R\tau_{\rm
ad}\frac{M_{S}}{\gamma}(\bm{\hat{\imath}}\cdot\bm{\delta}),$ (6a)
$\displaystyle F_{r}^{g}$
$\displaystyle=2\pi\frac{M_{S}}{\gamma}(\bm{v}\times\bm{\hat{k}})\cdot\bm{\delta},$
(6b) $\displaystyle F_{r}^{d}$ $\displaystyle=0.$ (6c)

The skyrmion velocity $\bm{v}$ can be computed by [23, 32],

$\displaystyle v_{x}$ $\displaystyle=\tau_{\rm
ad}\Lambda_{21}\frac{\alpha\Gamma_{11}}{(4\pi
Q)^{2}+(\alpha\Gamma_{11})^{2}},$ (7a) $\displaystyle v_{y}$
$\displaystyle=\tau_{\rm ad}\Lambda_{21}\frac{4\pi Q}{(4\pi
Q)^{2}+(\alpha\Gamma_{11})^{2}},$ (7b) $\displaystyle Q=\pm 1$
$\displaystyle,\quad\Lambda_{21}=\pi^{2}R,\quad\Gamma_{11}=2\pi\left(\frac{R}{w}+\frac{w}{R}\right).$
(7c)

Here $Q$ is the topological charge of skyrmion defined by
$Q=\frac{1}{4\pi}\int_{\Sigma}\bm{m}\cdot\left(\frac{\partial\bm{m}}{\partial
x}\times\frac{\partial\bm{m}}{\partial y}\right){\rm d}\bm{S}$ [15],
$\Lambda_{21}$ and $\Gamma_{11}$ are the driving factor and damping factor in
Thiele’s equation respectively. Eq.(7c) is derived from a rigid skyrmion as
shown in Ref.[23].

For current-driven skyrmion, $w$ is different from $w$ of ground state
skyrmion. Also, $w$ along different azimuthal direction is not the same, i.e.
$w=w(\varphi)$ is a function of polar angle $\varphi$, as shown in
Figs.2(c)(d). Without loss of validity, we can use average $w$, i.e.,
$w=\frac{1}{2\pi}\int_{0}^{2\pi}w(\varphi){\rm d}\varphi$.

The total Hamiltonian of a skyrmion includes exchange interaction, DMI,
magnetocrystalline anisotropy and Zeeman energy due to external field. For
ground state skyrmion without an external field, the skyrmion radius $R_{0}$
is given by $R_{0}=\pi D\sqrt{A/(16AK^{2}-\pi^{2}D^{2}K)}$ [35]. The steady-
state of skyrmion motion is a quasi-static state and the skyrmion should
minimize its total energy under the constraint of Thiele’s equation. By
minimizing the total energy, we have the relation between $R$ and $w$ for a
moving skyrmion [33],

$w=\sqrt{\frac{1}{\frac{1}{R^{2}}+\frac{K}{A}}}.$ (8)

Unlike expectations in most papers that the domain wall is almost rigid [23],
we found that $w$ actually increases with $R$. This is also an improvement for
normal ground state domain wall width parameter equation $w=\sqrt{A/K}$ [34].

We then consider the effect of in-plane external field $\bm{B}^{\rm
ext}=\mu_{0}\bm{H}^{\rm ext}$. It induces another canting vector
$\bm{\delta}_{\rm ext}=wM_{s}\bm{B}_{\rm ext}/2D$ [33]. Together with the
current-induced canting, the net canting vector is $\bm{\delta}_{\rm
net}=\bm{\delta}_{\rm current}+\bm{\delta}_{\rm ext}$. If $-x$ direction in-
plane field is applied, two canting vectors will partially cancel, and the
expanding forces in Eq.(6) will consequently be reduced and even be reversed.

In Eq.(1), we only consider damping-like SOT. Field-like SOT acts like an in-
plane external field perpendicular to the current direction [18]. Therefore,
we can use an equivalent in-plane field to describe field-like SOT.

Figure 3: (a) Schematic of solving Eq.(3). Solution exists for smaller
$j_{x}$, but not for larger $j_{x}$. In the approximation, $F_{r}^{\rm res}$
and $F_{r}^{\rm exp}$ are treated as linear and quadratic respectively. (b)
The skyrmion radii change with time in 15ns. Above critical current, skyrmion
cannot have a saturation radius. Figure 4: (a) Comparison of predicted radius
and simulation results under different current densities. The red dot is the
critical current and radius given by simulation. The blue dot is the critical
point predicted by our model. The characteristics of the curve are captured by
Eq.(9). (b)-(d) Comparison between critical current density $J_{\rm crit}$
predicted by our model and the simulation results. $D=1.4\rm mJm^{-2}$ in (c)
and (d). (e) The saturation radius as a function of a $-x$ direction in-plane
external field $B_{x}$ under current density $j_{x}=40\rm GA/m^{2}$. The red
dashed line represents the skyrmion radius with no current or in-plane field.
(f) The change of skyrmion radius with respect to time when different in-plane
external fields and same current density are applied.

To better understand the model, we can make approximations for $F_{r}^{\rm
res}$ and $F_{r}^{\rm exp}$ and analytically solve Eq.(3). The expanding force
is approximately a quadratic function of both $R$ and $j_{x}$, $F_{r}^{\rm
exp}=\beta j_{x}^{2}R^{2}$, where $\beta>0$. The restoring force $F_{r}^{\rm
res}$ is a linear function of $R$, $F_{r}^{\rm res}=-\alpha_{1}(R/R_{0}-1)$,
where $\alpha_{1}>0$. Both $\alpha_{1}$ and $\beta$ can be analytically
determined [33], but their closed forms are too complicated to be written
down. Approximately, $\alpha_{1}$ and $\beta D$ are constant [33]. Eq.(3) is
therefore reduced to a quadratic equation, of which the physical solution is,

$R=\frac{\alpha_{1}-\sqrt{\alpha_{1}^{2}-4\alpha_{1}\beta
j_{x}^{2}R_{0}^{2}}}{2\beta j_{x}^{2}R_{0}}.$ (9)

The solution for $R$ only exists for $j_{x}$ smaller than some critical
current density $j_{\rm crit}$, and $j_{\rm
crit}=\sqrt{\alpha_{1}/4\beta}R_{0}^{-1}$. The radius at $j_{\rm crit}$ is
$R_{\rm crit}=2R_{0}$.

In the presence of $x$-direction in-plane external field, the expanding force
is attached with an additional term, $F_{r}^{\rm exp}=\beta
j_{x}^{2}R^{2}+\lambda j_{x}B_{x}R$ [33]. We can expand the effect of external
field to the first order. In such case, $j_{\rm
crit}\approx\sqrt{\alpha_{1}/4\beta}R_{0}^{-1}-\lambda B_{x}/4\beta R_{0}$ and
$R\approx R_{c}+\lambda j_{x}B_{x}R_{0}^{2}/\alpha_{1}$ [33], where $R_{c}$ is
the skyrmion radius without external field, determined by Eq.(9). These
approximations reveal that the critical current increases and the skyrmion
radius can decrease if a sufficiently large $-x$ direction in-plane field is
applied.

Verification Using Micromagnetic Simulation.— Since for fixed material
parameters and $j_{x}$, $F_{r}^{\rm res}$ and $F_{r}^{\rm exp}$ only depend on
$R,w$ and $\bm{v}$. $w$ is determined by Eq.(8), and $\bm{v}$ can also be
calculated by parameters, $R$ and $w$ by Eq.(7). As a result, numerically
solving Eq.(3) leads to $R$, the radius of the skyrmion in steady-state
motion, called saturation radius. However, when $j_{x}$ is lager than a
threshold current density, skyrmion has no steady-state motion, as shown in
Fig.3(a). We can find the critical current density by solving Eq.(3). In
Fig.3(b), we show two special cases, where one case with a drive current
smaller than $j_{\rm crit}$ has a saturation radius and the other one with a
drive current larger than $j_{\rm crit}$ does not.

We use the Mumax3 software to perform micromagnetic simulations [36]. The
default parameters are $A=20{\rm pJm^{-1}},K=0.1{\rm MJm^{-3}},D=1.6{\rm
mJm^{-2}},M_{S}=1.42{\rm MAm^{-1}},\alpha=0.39$, thickness $d=0.9\rm nm$ [18,
23]. We apply and adjust the $x$-direction current density, $j_{x}$, and
record skyrmion saturation radius. Also, the existence of saturation radius is
determined by testing whether the skyrmion radius diverges with time, as
illustrated in Fig.3(b). We compare the simulation results of current-
dependent saturation radius with what our model predicts, and we achieve a
good agreement, as shown in Fig.4(a).

We then verify our prediction for the critical current density $j_{\rm crit}$.
$j_{\rm crit}$ is determined by the maximum $j_{x}$ such that solution still
exists for Eq.(3). For each parameter setting, we use binary search to find
the critical current using micromagnetic simulation. We test for $D$, $A$ and
$K$ respectively. The results are shown in Figs.4(b)-(d) and are consistent
with the predictions given by the analytical model. We can further understand
the result from the approximation model, Eq.(9). $j_{\rm
crit}=\sqrt{\alpha_{1}/4\beta}R_{0}^{-1}=\sqrt{\alpha_{1}/4\beta
D}\left(\pi\sqrt{AD/\left(16AK^{2}-\pi^{2}D^{2}K\right)}\right)^{-1}$, where
$\sqrt{\alpha_{1}/4\beta D}$ is constant [33]. The dependence of $j_{\rm
crit}$ on $D$, $A$ and $K$ can be easily understood using the approximation
model now.

As predicted by the analytical model and the approximation model, applying an
$-x$ direction in-plane field can suppress the deformation. Our predicted
saturation radii and the micromagnetic simulation results in the presence of
the external field are shown in Figs.4(e)(f), which show a quantitative
agreement. Our approximation model provides the insight that in-plane external
field can cancel the magnetization canting caused by current, suppress
skyrmion deformation, and even cause skyrmion to contract under current
injection.

Figure 5: (a) Velocities ($v_{x}$ and $v_{y}$) and skyrmion Hall angle as
functions of current density. Square dots are simulation results. Solid curves
are prediction of our model. Dashed lines are normal estimations of skyrmion
velocity and Hall angle assuming no deformation. The nonlinear response to a
large current is well predicted by our model. (b) Ground state skyrmion, (c)
skyrmion in linear regime, (d) skyrmion in nonlinear regime and (e) skyrmion
in breakdown regime. Periodic boundary condition is used here. Skyrmion size
continues to increase and the skyrmion eventually collapses. See supplemental
movies S1-S3 for (c)-(e) [33]. (f) Skyrmion mobility (defined as the skyrmion
velocity divided by the current density) as a function of skyrmion radius. The
mobility increases as the radius increases. (g) Skyrmion velocity as a
function of the length of the canting vector $\bm{\delta}$. The circles on the
theoretical curves in (f) and (g) have the same current density with those on
the simulation curves.

At last, we describe the nonlinear velocity of skyrmion in the presence of
deformation, which goes beyond Thiele’s equation in the linear regime. The
change in skyrmion radius induces nonlinear response of skyrmion velocity by
Eq.(7), as shown in Fig.5(a). Our analytical model provides a much better
predictions about velocity and skyrmion Hall angle than the linear model,
where only translational parts of forces are considered. We also show
snapshots of skyrmion dynamics at different regimes in Figs.5(b)-(e). We can
see that our predicted velocity and skyrmion Hall angle are consistent with
the micromagnetic simulation. From the simulation results in Fig.5(f), we find
that the mobility of the skyrmion, defined as $v/j_{x}$, increases as the
skyrmion radius increases. This is consistent with our analytical model and
Eqs.(7). To investigate how rotational symmetry breaking affects the skyrmion
dynamics, we show the relation between the skyrmion velocity and the
magnetization canting, which is the asymmetric deformation of the skyrmion
spin configuration in Fig.5(g). The velocity increases quasi-linearly with the
magnetization canting, which is consistent with the analytical model and
Eq.(5). The quantitative discrepancy between the simulated and the theoretical
canting is expected to resolve by further improving the assumptions from
Eq.(4) to Eq.(5).

In supplemental material [33], an empirical correction of the model is
proposed, leading to a significant improvement in the quantitative agreement.
Besides, we generalize our model to cases that involve out-of-plane external
field and deformation dynamics of Bloch-type skyrmions, and show our model
works very well for these cases.

In conclusion, our analytical and nonlinear model accurately describes the
deformation and velocity of SOT-driven skyrmion. It captures the essential
physics of skyrmion deformation, that is the rotational symmetry breaking of
SOT-driven skyrmion. From this foothold we are able to suppress skyrmion
deformation by applying in-plane field, which also has a sizeable effect on
skyrmion symmetry. Our results suggest that the spatial symmetry breaking in
topological spin textures may significantly affect their dynamics, which can
be potentially utilized to build nonlinear functional devices [37]. Moreover,
we anticipate that our model can be extended to describe nonlinear dynamics of
other topological spin textures such as antiferromagnetic skyrmions and
bimeron.

We appreciate fruitful discussions with Motohiko Ezawa and Se Kwon Kim. Z.C.
acknowledges the support from Zheyu Ren and Cheuk Pan Fong. The authors at
HKUST acknowledge funding support from the Shenzhen-Hong Kong-Macau Science
and Technology Program (Category C, Grant No. SGDX2020110309460000), Research
Grant Council—Early Career Scheme (Grant No. 26200520), and the Research Fund
of Guangdong-Hong Kong-Macao Joint Laboratory for Intelligent Micro-Nano
Optoelectronic Technology (Grant No. 2020B1212030010). Y.Z. acknowledges the
support by Guangdong Special Support Project (2019BT02X030), Shenzhen
Fundamental Research Fund (Grant No. JCYJ20210324120213037), Shenzhen Peacock
Group Plan (KQTD20180413181702403), Pearl River Recruitment Program of Talents
(2017GC010293) and National Natural Science Foundation of China (11974298,
61961136006).X.Z. was an International Research Fellow of the Japan Society
for the Promotion of Science (JSPS). X.Z. was supported by JSPS KAKENHI (Grant
No. JP20F20363).

## References

  * Skyrme [1962] T. Skyrme, A unified field theory of mesons and baryons, Nuclear Physics 31, 556 (1962).
  * Rößler  _et al._ [2006] U. K. Rößler, A. N. Bogdanov, and C. Pfleiderer, Spontaneous skyrmion ground states in magnetic metals, Nature 442, 797 (2006).
  * Fert  _et al._ [2013] A. Fert, V. Cros, and J. Sampaio, Skyrmions on the track, Nature Nanotechnology 8, 152 (2013).
  * Nagaosa and Tokura [2013] N. Nagaosa and Y. Tokura, Topological properties and dynamics of magnetic skyrmions, Nature Nanotechnology 8, 899 (2013).
  * Woo  _et al._ [2017] S. Woo, K. M. Song, H.-S. Han, M.-S. Jung, M.-Y. Im, K.-S. Lee, K. S. Song, P. Fischer, J.-I. Hong, J. W. Choi, B.-C. Min, H. C. Koo, and J. Chang, Spin-orbit torque-driven skyrmion dynamics revealed by time-resolved x-ray microscopy, Nature Communications 8, 15573 (2017).
  * Jiang  _et al._ [2015] W. Jiang, P. Upadhyaya, W. Zhang, G. Yu, M. B. Jungfleisch, F. Y. Fradin, J. E. Pearson, Y. Tserkovnyak, K. L. Wang, O. Heinonen, S. G. E. te Velthuis, and A. Hoffmann, Blowing magnetic skyrmion bubbles, Science 349, 283 (2015).
  * Yu  _et al._ [2010] X. Z. Yu, Y. Onose, N. Kanazawa, J. H. Park, J. H. Han, Y. Matsui, N. Nagaosa, and Y. Tokura, Real-space observation of a two-dimensional skyrmion crystal, Nature 465, 901 (2010).
  * Mühlbauer  _et al._ [2009] S. Mühlbauer, B. Binz, F. Jonietz, C. Pfleiderer, A. Rosch, A. Neubauer, R. Georgii, and P. Böni, Skyrmion lattice in a chiral magnet, Science 323, 915 (2009).
  * Büttner  _et al._ [2018] F. Büttner, I. Lemesh, and G. S. D. Beach, Theory of isolated magnetic skyrmions: From fundamentals to room temperature applications, Scientific Reports 8, 4464 (2018).
  * Jonietz  _et al._ [2010] F. Jonietz, S. Mühlbauer, C. Pfleiderer, A. Neubauer, W. Münzer, A. Bauer, T. Adams, R. Georgii, P. Böni, R. A. Duine, K. Everschor, M. Garst, and A. Rosch, Spin transfer torques in MnSi at ultralow current densities, Science 330, 1648 (2010).
  * Iwasaki  _et al._ [2013] J. Iwasaki, M. Mochizuki, and N. Nagaosa, Universal current-velocity relation of skyrmion motion in chiral magnets, Nature Communications 4, 1463 (2013).
  * Fert  _et al._ [2017] A. Fert, N. Reyren, and V. Cros, Magnetic skyrmions: advances in physics and potential applications, Nature Reviews Materials 2, 17031 (2017).
  * Yu  _et al._ [2017] G. Yu, P. Upadhyaya, Q. Shao, H. Wu, G. Yin, X. Li, C. He, W. Jiang, X. Han, P. K. Amiri, and K. L. Wang, Room-temperature skyrmion shift device for memory application, Nano Letters 17, 261 (2017), pMID: 27966987\.
  * Song  _et al._ [2020] K. M. Song, J.-S. Jeong, B. Pan, X. Zhang, J. Xia, S. Cha, T.-E. Park, K. Kim, S. Finizio, J. Raabe, J. Chang, Y. Zhou, W. Zhao, W. Kang, H. Ju, and S. Woo, Skyrmion-based artificial synapses for neuromorphic computing, Nature Electronics 3, 148 (2020).
  * Zhang  _et al._ [2020] X. Zhang, Y. Zhou, K. M. Song, T.-E. Park, J. Xia, M. Ezawa, X. Liu, W. Zhao, G. Zhao, and S. Woo, Skyrmion-electronics: writing, deleting, reading and processing magnetic skyrmions toward spintronic applications, Journal of Physics: Condensed Matter 32, 143001 (2020).
  * Jiang  _et al._ [2017] W. Jiang, X. Zhang, G. Yu, W. Zhang, X. Wang, M. Benjamin Jungfleisch, J. E. Pearson, X. Cheng, O. Heinonen, K. L. Wang, Y. Zhou, A. Hoffmann, and S. G. E. te Velthuis, Direct observation of the skyrmion Hall effect, Nature Physics 13, 162 (2017).
  * Litzius  _et al._ [2017] K. Litzius, I. Lemesh, B. Krüger, P. Bassirian, L. Caretta, K. Richter, F. Büttner, K. Sato, O. A. Tretiakov, J. Förster, R. M. Reeve, M. Weigand, I. Bykova, H. Stoll, G. Schütz, G. S. D. Beach, and M. Kläui, Skyrmion Hall effect revealed by direct time-resolved X-ray microscopy, Nature Physics 13, 170 (2017).
  * Juge  _et al._ [2019] R. Juge, S.-G. Je, D. d. S. Chaves, L. D. Buda-Prejbeanu, J. Peña Garcia, J. Nath, I. M. Miron, K. G. Rana, L. Aballe, M. Foerster, F. Genuzio, T. O. Menteş, A. Locatelli, F. Maccherozzi, S. S. Dhesi, M. Belmeguenai, Y. Roussigné, S. Auffret, S. Pizzini, G. Gaudin, J. Vogel, and O. Boulle, Current-driven skyrmion dynamics and drive-dependent skyrmion Hall effect in an ultrathin film, Phys. Rev. Applied 12, 044007 (2019).
  * Litzius  _et al._ [2020] K. Litzius, J. Leliaert, P. Bassirian, D. Rodrigues, S. Kromin, I. Lemesh, J. Zazvorka, K.-J. Lee, J. Mulkers, N. Kerber, D. Heinze, N. Keil, R. M. Reeve, M. Weigand, B. Van Waeyenberge, G. Schütz, K. Everschor-Sitte, G. S. D. Beach, and M. Kläui, The role of temperature and drive current in skyrmion dynamics, Nature Electronics 3, 30 (2020).
  * Okuyama  _et al._ [2019] D. Okuyama, M. Bleuel, J. S. White, Q. Ye, J. Krzywon, G. Nagy, Z. Q. Im, I. Živković, M. Bartkowiak, H. M. Rønnow, S. Hoshino, J. Iwasaki, N. Nagaosa, A. Kikkawa, Y. Taguchi, Y. Tokura, D. Higashi, J. D. Reim, Y. Nambu, and T. J. Sato, Deformation of the moving magnetic skyrmion lattice in MnSi under electric current flow, Communications Physics 2, 79 (2019).
  * Masell  _et al._ [2020] J. Masell, D. R. Rodrigues, B. F. McKeever, and K. Everschor-Sitte, Spin-transfer torque driven motion, deformation, and instabilities of magnetic skyrmions at high currents, Phys. Rev. B 101, 214428 (2020).
  * Weißenhofer and Nowak [2019] M. Weißenhofer and U. Nowak, Orientation-dependent current-induced motion of skyrmions with various topologies, Phys. Rev. B 99, 224430 (2019).
  * Liu  _et al._ [2020] L. Liu, W. Chen, and Y. Zheng, Current-driven skyrmion motion beyond linear regime: Interplay between skyrmion transport and deformation, Phys. Rev. Applied 14, 024077 (2020).
  * Thiele [1973] A. A. Thiele, Steady-state motion of magnetic domains, Phys. Rev. Lett. 30, 230 (1973).
  * Sampaio  _et al._ [2013] J. Sampaio, V. Cros, S. Rohart, A. Thiaville, and A. Fert, Nucleation, stability and current-induced motion of isolated magnetic skyrmions in nanostructures, Nature Nanotechnology 8, 839 (2013).
  * Tomasello  _et al._ [2014] R. Tomasello, E. Martinez, R. Zivieri, L. Torres, M. Carpentieri, and G. Finocchio, A strategy for the design of skyrmion racetrack memories, Scientific Reports 4, 6784 (2014).
  * Zhang  _et al._ [2016] X. Zhang, Y. Zhou, and M. Ezawa, Magnetic bilayer-skyrmions without skyrmion Hall effect, Nature Communications 7, 10293 (2016).
  * Barker and Tretiakov [2016] J. Barker and O. A. Tretiakov, Static and dynamical properties of antiferromagnetic skyrmions in the presence of applied current and temperature, Phys. Rev. Lett. 116, 147203 (2016).
  * Weißenhofer  _et al._ [2021] M. Weißenhofer, L. Rózsa, and U. Nowak, Skyrmion dynamics at finite temperatures: Beyond Thiele’s equation, Phys. Rev. Lett. 127, 047203 (2021).
  * Shao [2020] Q. Shao, Skyrmions get pushed beyond the limit, Nature Electronics 3, 16 (2020).
  * Gilbert [2004] T. Gilbert, A phenomenological theory of damping in ferromagnetic materials, IEEE Transactions on Magnetics 40, 3443 (2004).
  * Wang  _et al._ [2019] Z. Wang, X. Zhang, J. Xia, L. Zhao, K. Wu, G. Yu, K. L. Wang, X. Liu, S. G. E. te Velthuis, A. Hoffmann, Y. Zhou, and W. Jiang, Generation and Hall effect of skyrmions enabled using nonmagnetic point contacts, Phys. Rev. B 100, 184426 (2019).
  * [33] See Supplemental Material at  for the details of the derivation of our model and an empirical correction to the model, which includes Refs.[24,26,31,32,35].
  * Rohart and Thiaville [2013] S. Rohart and A. Thiaville, Skyrmion confinement in ultrathin film nanostructures in the presence of Dzyaloshinskii-Moriya interaction, Phys. Rev. B 88, 184422 (2013).
  * Wang  _et al._ [2018] X. S. Wang, H. Y. Yuan, and X. R. Wang, A theory on skyrmion size, Communications Physics 1, 31 (2018).
  * Vansteenkiste  _et al._ [2014] A. Vansteenkiste, J. Leliaert, M. Dvornik, F. Garcia-Sanchez, and B. Waeyenberge, The design and verification of mumax3, AIP Advances 4 (2014).
  * Shen  _et al._ [2020] L. Shen, J. Xia, X. Zhang, M. Ezawa, O. A. Tretiakov, X. Liu, G. Zhao, and Y. Zhou, Current-induced dynamics and chaos of antiferromagnetic bimerons, Phys. Rev. Lett. 124, 037202 (2020).