#

Victor Zhong ⋄† , Weijia Shi⋄† , Wen-tau Yih† Corresponding author
<EMAIL_ADDRESS>Luke Zettlemoyer⋄†

⋄University of Washington
†Meta AI

###### Abstract

We introduce RoMQA, the first benchmark for robust, multi-evidence, multi-
answer question answering (QA). RoMQA contains clusters of questions that are
derived from related constraints mined from the Wikidata knowledge graph.
RoMQA evaluates robustness of QA models to varying constraints by measuring
worst-case performance within each question cluster. Compared to prior QA
datasets, RoMQA has more human-written questions that require reasoning over
more evidence text and have, on average, many more correct answers. In
addition, human annotators rate RoMQA questions as more natural or likely to
be asked by people. We evaluate state-of-the-art large language models in
zero-shot, few-shot, and fine-tuning settings, and find that RoMQA is
challenging: zero-shot and few-shot models perform similarly to naive
baselines, while supervised retrieval methods perform well below gold evidence
upper bounds. Moreover, existing models are not robust to variations in
question constraints, but can be made more robust by tuning on clusters of
related questions. Our results show that RoMQA is a challenging benchmark for
large language models, and provides a quantifiable test to build more robust
QA methods.

Figure 1: A cluster of related questions, implicit constraints, evidence text,
and answers from RoMQA. Within a RoMQA cluster, related questions differ in
implicit constraints. In addition to evaluating model performance across
questions, RoMQA evaluates robustness to variations in question constraints by
scoring worst-case performance among related questions. RoMQA
---
A film composed by S. Thaman and produced by Ganesh Babu.
Who did not play for the Carolina Panthers but was a linebacker and was on the
Chicago Bears?
Which members of the Royal Society received the Order of Australia, but were
not employed by the University of Oxford?
Sub-orbital spaceflight that launched from Cape Canaveral Air Force Station
Launch Complex 5. Launched by Mercury-Redstone Launch Vehicle
Who is an athlete who participated in diving, and was born in Stockholm?
HotpotQA
Are Random House Tower and 888 7th Avenue both used for real estate?
Which American singer and songwriter has a mezzo-soprano vocal range, Tim
Armstrong or Tori Amos?
WFMT FM radio transmits from the second tallest building in the United States,
which is located where?
Who was the recipient of a prize also given to a player for Chinese club
Tianjin Quanjian?
Which of Tara Strong major voice role in animated series is an American
animated television series based on the DC Comics fictional superhero team,
the "Teen Titans"?
ComplexWebQuestions
What university has more than 15,835 undergraduates and is the university
Derek Fisher attended?
Who influenced Whitman’s poetry who was the public speaker who spoke about the
American Civil War?
What is the main train station called in the governmental jurisdiction where
the government includes the position Mayor of San Francisco?
Which country that borders Russia has the smallest ISO?
What country that’s a neighbor of Russia is a governmental jurisdiction where
Erik Asanbayev holds a governmental office?
QAMParI
Where did a Roman Catholic archbishop of San Francisco attend school?
At what institution did a Bishop of Derby receive their education?
For which movie did Mani Ratnam work on the script and serve as producer?
What Type VII C/41 and Type VII ships was in both the vessel classes of
German?
Philip Kaufman was responsible for both writing and directing of which movie?
Table 1:  Randomly sampled examples from RoMQA and other compositional QA
datasets. Human evaluations show that people are more likely to ask RoMQA
questions than those from other compositional QA datasets. Qualitatively,
RoMQA questions exhibit fewer artifacts such as overly precise constraints
(e.g. 15,835 undergraduates), overly numerous references (e.g. is an American
animated…based on…the “Teen Titans”), and unusual attribute comparisons (e.g.
smallest ISO).

##  1 Introduction

A high quality compositional question answering (QA) model should be robust to
small variations in the underlying meaning of input questions. Consider the
question “which pianists born in Paris play Western classical music?” To show
robust understanding, a QA model should not only be able to correctly answer
this direct question, but also a wide range of related queries that different
in only a few constraints (e.g. who was a pianist born in Paris?, who was a
Western classical pianist, not born in Paris?). Prior compositional QA
datasets do not evaluate the robustness of QA models to variations in question
constraints.

We introduce RoMQA, a benchmark for Robust, Multi-evidence, multi-answer QA,
which explicitly evaluates for robustness to small question perturbations.
Figure 1 shows examples from RoMQA. RoMQA differs from previous work in a
number of ways.

#### Evaluates robustness to constraint variations.

RoMQA contains clusters of related questions that are used to measure
robustness to varying implicit question constraints. For each cluster, we
compute a robustness score that is the the minimum score over the questions it
contains. In order to perform well on RoMQA robustness evaluation, a model
must be able to understand many different combinations of the implicit
constraints that define the cluster, such as what it means to be a pianist, to
be born in Paris, and to play Western classical music. To our knowledge, RoMQA
is the first QA benchmark that evaluates this type of robustness.

#### More complex questions.

Human questions often have many answers and cannot be answered from a single
text. When compared to existing datasets, RoMQA questions have more answers
(mean 108.6, median 11), cover more diverse topics, and require more pieces of
evidence text (mean 41.6, median 24). RoMQA also contains entity-linked,
relation-extracted text that provide provenance for the constraints, showing
the questions are answerable with multi-evidence reasoning from the text
corpus.

#### More natural human written questions.

Compared to prior multi-answer compositional QA datasets, RoMQA provides an
order of magnitude more human-written questions. Human evaluations show that
these questions are more natural, as gauged by how likely a person is to ask
the question. Qualitatively, RoMQA questions are less likely to contain overly
precise constraints, unusual attribute comparisons, or overly large numbers of
referential hops.

We evaluate state-of-the-art large language models (LMs) on RoMQA in zero-shot
prompting, few-shot in-context learning, and supervised learning settings. On
the closed setting where the model selects among 100 candidate entities, we
find that zero-shot and few-shot LMs perform on par (e.g. 38.5 F1 by 8-shot
OPT-175B, Zhang et al. (2022)) with simple baselines such as predicting the
entire set of candidate entities (33.5 F1). RoMQA also remains very
challenging to state-of-the-art supervised methods, with the best retrieve-
then-classify model achieving 51.6 F1 compared to a gold-evidence upper bound
of 95.0 F1. The open setting, where no candidates are given, is even more
challenging to existing methods — the state-of-the-art Instruct-GPT3 (`text-
davinci-002`, Ouyang et al. (2022)) obtains 12.6 Pr@10 (precision at 10) while
supervised retrieve-then-generate obtains 52.1 Pr@10.

Finally, no test model is robust to variations in question constraints. The
best performing retrieval method obtains a worse-case related question test
score of 24.1 F1 in the closed setting — a 27.5 F1 absolute drop compared to
evaluating questions independently. Training on clusters of related questions,
such RoMQA clusters, improves model robustness over training on unrelated
questions. However the robustness gap remains large — closing this gap will
likely require significant advances in natural language understanding.

##  2 RoMQA

We describe RoMQA construction and how it differs from prior compositional QA
datasets.

###  2.1 Dataset construction

RoMQA construction has three goals. First, we want a diverse selection of
question topics. Second, these questions should require reasoning over
multiple pieces of evidence. Third, we need to understand what implicit
constraints the questions contain in order to evaluate robustness to varying
constraints. At a high level, RoMQA construction involves 1) sampling
constraints from knowledge base (KB) triples, 2) clustering related
constraints, 3) sampling implicit constraints that form logical queries, and
4) annotating language questions.

#### Sampling constraints from a KB.

We create RoMQA questions from Wikidata (Vrandečić and Krötzsch, 2014) that
are answerable given entity-linked and relation-extracted text (Elsahar et
al., 2018). Wikidata consists of subject-proposition-object triples such as
`Gilbert_Amy` `occupation` `pianist`. We convert these triples into entity-
relation constraints. For instance, the previous example is decomposed into
constraints `Gilbert_Amy` `occupation` `obj` and `pianist` `occupation`
`subj`.

#### Clustering related constraints.

A cluster of related constraints share at least two answer entities. For
example, `occupation` `pianist` `subj` and `place_of_birth` `Paris` `subj` are
in the same cluster because they share the same answers `Gilbert_Amy` and
`Claude_Helffer` (Paris-born pianists). As Wikidata has a skewed proposition
distribution, we resample cluster constraints with probability inversely
proportional to their proposition frequency in the KB (Appendix A). This down-
samples over-represented propositions such as `country`. We keep clusters with
$\geq$3 constraints to be able to generate many related questions from each
cluster. We discard clusters of potentially spuriously related constraints
with a single shared answer. 10k clusters are randomly chosen for training and
15k clusters for evaluation.

Dataset | Train | Dev |  Test |  | Human
---
written
| Multi
---
answer
| Gold
---
evidence
| Robustness
---
evaluation
RoMQA (Ours) | 11k | 7k |  11k |  Yes |  Yes |  Yes |  Yes
HotpotQA | 90k | 7k |  7k |  Yes |  No |  Yes |  No
CWQ | 28k | 3k |  3k |  Yes |  Yes |  No |  No
QAMParI | 64k | 1k |  1k |  Eval only |  Yes |  Yes |  No
Table 2:  Dataset size and question complexity.  Figure 2:  Dataset comparison
over question, evidence, and answer size distributions.

#### Sampling constraints to form logical queries.

We generate up to 5 logical queries using each cluster. For each logical
query, we copy the cluster and remove constraints with probability 0.1 and
negate with 0.1. We negate sparingly because initial trials showed that a
large number of negative constraints resulted in unnatural questions. We
further remove redundant constraints (e.g. American presidents born in the
US), and uniformly subsample up to 4 constraints. This constitutes a logical
query with multiple conjunctions and subtractions. For instance, the cluster
{`occupation` `pianist` `subj`, `born_in` `Paris` `subj`} can form a logical
query `occupation` `pianist` `subj` `AND` `born_in` `Paris` `subj`. We discard
overly general queries with $\geq$5000 answers.

#### Creating natural language questions.

Mechanical Turk crowd-workers annotate logical queries marked with Wikidata
titles, descriptions, and aliases into questions. Appendix B Figure 11 shows
the interface. Two more annotators verify each annotation to confirm that it
matches the logical query. We keep only annotations with 100% agreement,
resulting in 11% being discarded. After verification, we additionally discard
clusters with $\leq$2 questions.

###  2.2 Dataset analyses and comparison

We compare RoMQA to prior compositional QA datasets: HotpotQA (Yang et al.,
2018), ComplexWebQuestions (CWQ; Talmor and Berant, 2018), and QAMParI
(Amouyal et al., 2022).

#### Dataset size and question complexity

Table 2 shows that only RoMQA evaluates robustness to input variations.
Moreover, only RoMQA and QAMParI are human annotated with multiple answers and
gold evidence. However, QAMParI provides 2,000 human-written questions for
evaluation while RoMQA provides 11k for training and 17k for evaluation.

Figure 2 shows the distribution of answer, evidence, and question sizes.
First, RoMQA questions requires finding many answers. On average, RoMQA
questions have 108 answers — at least 10x larger than others. Second, RoMQA
requires reasoning over a much more evidence documents. On average, entities
in the RoMQA answer set combine for a total of 52 evidence sentences. Third,
RoMQA questions are longer with more words. Figure 3 shows that in a random
sample of 500 questions, RoMQA refers to more unique noun phrases apart from
HotpotQA.

Figure 3:  Question diversity as measured by the number of unique noun-phrases
in 500 randomly sampled questions from the development set of each dataset.
The batches are randomly sampled 4 times to compute standard deviation.
Figure 4:  The distribution of questions naturalness ratings by 3 annotators
on 1,000 randomly sampled questions from the development set of each dataset.
Each annotator rates four questions shuffled in random order, one from each
dataset. he annotator is asked whether they would ask the question themselves,
and if they think someone else would ask the question.

#### Naturalness human evaluation

Prior work sometimes sacrifice question naturalness in pursuit of complexity.
Table 1 illustrates artifacts in randomly sampled questions from HotpotQA,
ComplexWebQuestions, and QAMParI. They include unusual constraints such as IDs
(e.g. …has the smallest ISO?111ISO codes are 2-3 character-long codes that
represent names of countries and their subdivisions.), overly precise
constraints (e.g. …is an American animated television series based on the DC
Comics fictional superhero team, the “Teen Titans”) and an excessive number of
referential expressions (e.g. …in the governmental jurisdiction where the
government includes the position Mayor of San Francisco). We compare the
naturalness of 1,000 randomly sampled human written questions from each
dataset. Each annotator is shown a randomly sampled question from each
dataset, shuffled in random order. The annotator is asked “how likely would
you ask this question?” and “how likely do you think another person would ask
this question?” Each question is annotated by 3 crowdworkers. The breakdown of
ratings across questions is shown for each dataset in Figure 4. On average,
annotators consider RoMQA to be significantly more natural than HotpotQA and
ComplexWebQuestions.

Model class | Setting |  Input format
---|---|---
Zero-shot | Closed |  Gilbert Amy [SEP] Who was a pianist born in Paris?
| Open |  Who was a pianist born in Paris?
Few-shot | Closed |  Katie Bell [SEP] Who is an athlete who participated in diving, and was born in Stockholm? [SEP] False [newline] …Gilbert Amy [SEP] Who was a pianist born in Paris? [SEP] True
| Open |  Who is an athlete who participated in diving, and was born in Stockholm? [SEP] Johan Jansson …[newline] …Who was a pianist born in Paris? [SEP]
Supervised | Closed |  Gilbert Amy [SEP] Who was a pianist born in Paris?
| Open |  Who was a pianist born in Paris?
Sup+evidence | Closed |  Gilbert Amy [SEP] Who was a pianist born in Paris? [SEP] Gilbert Amy (born 29 August 1936) is a French composer and conductor …
| Open |  Who was a pianist born in Paris? [SEP] Gilbert Amy (born 29 August 1936) is a French composer and conductor …
Table 3: Input format given the question “Who was a pianist born in Paris”.
For closed setting, the candidate “Gilbert Amy” is used as an example.
Evidence includes retrieved sentences for the retrieval model or gold evidence
for the upperbound model.

##  3 Experiments

How do existing QA systems perform on RoMQA? Are they robust to variations in
question constraints? We answer these questions by experimenting with state-
of-the-art models in zero-shot, few-shot, and supervised learning settings.

###  3.1 Evaluation

#### Open vs closed settings

Given a question in RoMQA, a QA system should produce a set of answer
entities. In the closed setting, the system is given 100 candidate entities
and must identify which ones answer the question. Negative candidates are
potentially difficult for a model because they can match any constraint in the
question. In the open setting, candidates are not given.

#### Evaluation metrics

RoMQA questions have many answers. For the closed setting, we evaluate
predictions using $\mathrm{F}_{1}$ and accuracy. $\mathrm{F}_{1}$ measures set
overlap between predicted and gold answers, while accuracy measures whether
they match exactly. For the open setting, we evaluate precision@K
(${\mathrm{P}_{10}}$) for two reasons. First, precision gives partial credit
when it is too hard to enumerate the full set. Second, the user may ask a
question with many answers (e.g. hundreds or thousands) with the intent of
only seeing some examples (e.g. list British footballers). For each score, we
additionally have a robustness variant. Let $Q=\\{q_{1},q_{2}\dots q_{n}\\}$
denote a cluster of $n$ related questions. The question $q_{i}$ has the
corresponding predicted answer set $p_{i}$ and gold answer set $g_{i}$. A
robustness score is the worst-case score across the cluster. For instance, the
robust $\mathrm{F}_{1}$ is
${\mathrm{F}_{1}}^{R}(Q)=\min_{i}\left(\mathrm{F}_{1}(p_{i},g_{i})\right)$. We
compute similar robustness scores for accuracy and precision@K.

###  3.2 Models

We evaluate three classes of models. The input format for each class is shown
in Table 3.

Model | Dev | Test
---|---|---
$\mathrm{F}_{1}$ | ${\mathrm{F}_{1}}^{R}$ | ${\mathrm{acc}}$ | ${{\mathrm{acc}}}^{R}$ | $\mathrm{F}_{1}$ | ${\mathrm{F}_{1}}^{R}$ | ${\mathrm{acc}}$ | ${{\mathrm{acc}}}^{R}$
Zero-shot |  |  |  |  |  |  |  |
predict all | 34.1 | 13.1 | 0.0 | 0.0 | 33.5 | 12.8 | 0.0 | 0.0
tk-instruct-3b-0shot | 34.5 | 13.2 | 0.0 | 0.0 | 34.0 | 12.8 | 0.0 | 0.0
opt-instruct-0shot | 36.0 | 14.0 | 0.0 | 0.0 | 36.0 | 14.0 | 0.0 | 0.0
Few-shot |  |  |  |  |  |  |  |
tk-instruct-3b-4shot | 33.5 | 12.9 | 0.0 | 0.0 | 33.1 | 12.5 | 0.0 | 0.0
opt-8shot | 38.9 | 16.1 | 0.0 | 0.0 | 38.5 | 15.5 | 0.0 | 0.0
Supervised |  |  |  |  |  |  |  |
binary | 35.9±0.7 | 10.9±0.6 | 2.3±0.1 | 0.3±0.1 | 35.5±1.8 | 10.2±1.3 | 2.5±0.3 | 0.3±0.1
binary+retrieval | 63.4±0.6 | 36.4±0.9 | 7.2±0.2 | 0.3±0.0 | 51.6±6.6 | 24.1±6.4 | 3.9±2.0 | 0.2±0.3
binary+gold evidence | 95.3±0.3 | 83.5±0.8 | 72.3±1.8 | 39.2±2.6 | 95.0±0.3 | 83.4±0.9 | 71.5±0.9 | 38.2±1.2
Table 4:  Model performance on closed setting RoMQA. Metrics are set
$\mathrm{F}_{1}$, set accuracy, and their robustness counterparts (i.e. worst
case measure over cluster of related questions). Each model is given 100
candidate entities and must decide which entity belongs to the answer set. The
retrieval model additionally observes sentences retrieved via BM25 followed by
DPR. Zero-shot and few-shot are binary-classifiers calibrated with channel
calibration. Supervised models fine-tune BART-large on the training data to
classify the answer set on a per-entity basis.

#### Zero-shot.

We consider a naive closed setting baseline that predicts all candidates as
answers. We also include state-of-the-art prompting models `tk-instruct-3B`
(Wang et al., 2022) and `opt-instruct-175B` (Zhang et al., 2022). In the
closed setting, they generate yes/no given the question and a candidate. In
the open setting, they generate answers given the question.

#### Few-shot in-context learning.

We evaluate `tk-instruct-3B` (Wang et al., 2022), `opt-instruct-175B` (Zhang
et al., 2022), and GPT3 (`text-davinci-002`; Brown et al. (2020)) with as many
in-context examples as model context allows (4, 8, and 8 respectively). Input
format is similar to that of the zero-shot setting, with the addition of in-
context examples. In the closed setting, the context includes an equal number
of randomly sampled positive and negative examples. We compare the scores for
the candidate answering vs. not answering the current question. These scores
are calibrated using channel calibration (Min et al., 2022). In the open
setting, the model context includes $\leq$10 subsampled answers for each
example.

#### Supervised learning.

We tune `BART-large` with and without retrieved evidence (Lewis et al., 2020)
and show standard deviation across 5 random seeds. For the closed setting
which considers a candidate entity, we use a two-stage hybrid retrieval
because dense retrievers under-perform sparse retrievers on rare, precise
entities (Sciavolino et al., 2021). We first retrieve documents with BM25
using entity title as the query. We then use DPR (Karpukhin et al., 2020) to
retrieve document sentences whose cosine similarity with the question exceed a
threshold (0.65) tuned on the training set. Finally, we fine-tune to classify
whether each candidate belongs to the answer set.

In the open setting, we do not use classification models because it must
decide over all possible (2.9M) entities per question and is computationally
prohibitive.222For reference, inference over the entire test set with 100
candidate entities per example using the classification model requires 10
hours on a Volta 32GB GPU. Instead, we directly retrieve evidence with DPR and
fine-tune the model to generate the answer set as a 1024-token sequence.

Model | Dev | Test
---|---|---
$\mathrm{F}_{1}$ | ${\mathrm{F}_{1}}^{R}$ | ${\mathrm{P}_{10}}$ | ${{\mathrm{P}_{10}}}^{R}$ | $\mathrm{F}_{1}$ | ${\mathrm{F}_{1}}^{R}$ | ${\mathrm{P}_{10}}$ | ${{\mathrm{P}_{10}}}^{R}$
Zero-shot |  |  |  |  |  |  |  |
tk-instruct-3b-0shot | 0.2 | 0.0 | 1.7 | 0.0 | 0.3 | 0.0 | 1.8 | 0.0
opt-instruct-0shot | 1.6 | 0.0 | 5.0 | 0.1 | 1.6 | 0.1 | 5.1 | 0.2
Few-shot |  |  |  |  |  |  |  |
tk-instruct-3b-4shot | 0.3 | 0.0 | 1.9 | 0.0 | 0.4 | 0.0 | 2.0 | 0.0
opt-8shot | 2.1 | 0.0 | 5.5 | 0.2 | 2.2 | 0.1 | 5.6 | 0.1
gpt3-8shot | 4.3 | 0.4 | 13.3 | 1.2 | 4.4 | 0.4 | 12.6 | 1.0
Supervised |  |  |  |  |  |  |  |
seq2seq | 32.2±0.9 | 11.2±0.5 | 47.3±1.2 | 20.3±1.2 | 32.6±1.0 | 11.5±0.7 | 45.7±0.8 | 19.7±1.1
seq2seq+retrieval | 37.1±0.5 | 15.0±0.4 | 53.1±0.9 | 28.1±1.1 | 37.6±0.6 | 15.4±0.3 | 52.1±1.6 | 28.3±1.4
seq2seq+gold evidence | 39.5±0.5 | 16.9±0.3 | 55.9±0.9 | 31.6±0.7 | 39.9±0.4 | 17.4±0.3 | 54.9±1.0 | 31.2±1.3
Table 5:  Model performance on open setting RoMQA. Metrics are set
$\mathrm{F}_{1}$, precision at 10, and their robustness counterparts (i.e.
worst case measure over cluster of logically related questions). Each model is
given the question and must generate the answer set as a sequence. The
retrieval model additionally observes sentences retrieved by DPR. Supervised
models fine-tune BART-large on the training data. All models generate the
answer set as a sequence. The upperbound gold evidence method still struggles
because its context cannot fit all gold evidence.  Figure 5: Correlation with
model performance ($\mathrm{F}_{1}$) on the closed setting. Imprecise
questions with many answers are easier to answer (higher $\mathrm{F}_{1}$).
Questions based on general propositions that co-occur with many different
entities are easier to answer. Questions with more constraints are more
difficult to answer.

#### Upper bound with gold evidence.

We provide a performance upper bound by training supervised models with gold
evidence — sentences that provide provenance to an implicit question
constraint. For instance, consider the example in Figure 1. Claude Hellfer is
an answer to the question “Who was a pianist born in Paris?”, whereas David
Fray is not. In this case, the gold evidence includes “Claude Helffer… was a
French pianist”, “Helffer was born in Paris”, and “David Fray… is a French
classical pianist”. Because the gold evidence only contains sentences that
provide provenance to an implicit constraint, it does not contain the sentence
“David Fray was born in Tarbes, near the Pyrenees.” In other words, given gold
evidence, the QA model does not need to filter out candidates (e.g. David
Fray) because of evidence that conflict with implicit constraints (e.g. born
in Tarbes instead of Paris). Instead, it only needs to verify that the
evidence references all implicit constraints. Consequently, the gold evidence
setting is overly optimistic in that part of the reasoning is completed by a
perfect retriever. While no such retriever currently exists, this setting
nevertheless provides an upper bound estimate for RoMQA.

###  3.3 Results

Table 4 and Table 5 show performance on RoMQA closed and open settings. RoMQA
is challenging for state-of-the-art large-scale models. Moreover, these models
are not robust to variations in question constraints. The best models
significantly trail the gold evidence upper bound, showing there is
significant room future work.

#### Zero-shot and few-shot models perform similarly to naive predict-all
baseline.

In the closed setting, each system is given a set of 100 candidate entities
and must identify which entity belong to the answer set. We find that state-
of-the-art pretrained instruction prompting models perform on par with the
naive baseline of simply predicting that every candidate belongs to the answer
set. This occurs both with instructing prompting and in-context learning
models, and suggests that they can not effectively reason about the types of
compositional questions found in RoMQA.

#### Both closed and open settings remain challenging with supervised
training.

When given 11k annotated examples, large retrieval models perform better than
zero-shot and few-shot LMs. However, supervised performance also trails the
gold evidence upper bound. This suggests that there is significant room for
modeling improvements that retrieve and compose evidence in order to answer
compositional questions.

What types of questions do the best-performing supervised systems struggle
with? Figure 5 plots Pearson correlation with $\mathrm{F}_{1}$ in the closed
setting, and shows that systems generally struggles with more precise
questions.333Pearson correlation is a measure of linear correlation. A Pearson
coefficient of 1 or -1 implies positive or negative linear correlation, while
0 implies no linear dependency. First, when the question has many answers, the
model has an easier time producing some correct answers). Second, the model
performs better on more general propositions that co-occur with many different
unique entities. Third, the model struggles with questions with more implicit
constraints.

#### Methods are not robust to question constraint variations.

All methods drop in performance when considering the worst-case performance
among clusters of related questions. This suggests that large LM-based QA
systems are not robust to variations in question constraints. Figure 6 shows
what types of questions result in robustness drops. Compared to other
questions in the same cluster, a question is easier if it contains more
answers, and harder if it specifies more implicit constraints.

Training on clusters of related questions (e.g. RoMQA clusters) is one way to
improve model robustness. Given clusters of questions with related implicit
constraints, in the first setting we train on unrelated questions — one
question from each cluster for a total of K training examples. In the second
setting, we train on related questions — K consecutive examples from entire
clusters. Table 6 shows that while the diversity from training on unrelated
questions marginally improves overall performance, training on clusters of
related questions results in more robust models. Nevertheless, the robustness
drops remain significant. Considering that variations in RoMQA questions are
reasonable questions humans would write, as opposed to artificially created
adversarial questions, our findings suggests that there is a practical need
for developing more robust QA systems.

Figure 6:  Correlation with robustness drop ($\mathrm{F}_{1}$ \- cluster mean
$\mathrm{F}_{1}$) on the closed setting. The axes denote deviation from the
cluster means. Among a cluster of related questions, a more precise question
with more constraints or fewer answers tend to be harder for the model than
related questions with more answers or less constraints.

#### Building context for open setting is very challenging.

While the closed setting RoMQA challenges current state-of-the-art models, the
open setting remains an even greater challenge. One core difficulty associated
with open setting RoMQA is that it is difficult to compute the evidence set
required the answer the question. Consider Figure 1’s question “Who was a
Western classical music pianist, not born in Paris”. The obvious way a human
would answer this question is substracting the set of people born in Paris
from the set of Western classical music pianists. However, both of these sets
are very large. Our results show that an end-to-end large language model
struggles in reasoning over such large sets.

##  4 Related Work

#### Question answering datasets

Existing QA datasets focus on answering from a single passage (Rajpurkar et
al., 2016; Joshi et al., 2017; Kwiatkowski et al., 2019; Sciavolino et al.,
2021) to answering over multiple pieces of evidence (Yang et al., 2018; Welbl
et al., 2018; Thorne et al., 2018). Recent datasets further emphasize
answering questions that have multiple answers (Min et al., 2020; Amouyal et
al., 2022). RoMQA combines the latter two settings in that it requires
answering questions over multiple pieces of evidence to provide multiple
answers. Compared to prior datasets, RoMQA questions require robust reasoning
over more pieces of evidence to provide more answers.

Training | Closed | Open
---|---|---
questions | $\mathrm{F}_{1}$ | ${\mathrm{F}_{1}}^{R}$ | $\Delta\mathrm{F}_{1}$ | ${\mathrm{P}_{10}}$ | ${{\mathrm{P}_{10}}}^{R}$ | $\Delta{\mathrm{P}_{10}}$
Unrelated | 56.2 | 28.2 | -28.0 | 27.0 | 1.6 | -25.4
Related | 55.7 | 28.6 | -27.1 | 26.3 | 11.7 | -14.6
Table 6:  Training supervised retrieval models on related vs unrelated
questions. For unrelated questions training, one question is taken from each
cluster for a total of 2695 questions. For related questions training, entire
clusters of related questions are included until there are 2695 questions.
While training on a more diverse set of unrelated questions results in
marginally higher overall performance, training on related questions results
in more robust models.

#### Robustness evaluation

NLP systems have previous been show to lack robustness. They are susceptible
to character based attacks that comprise of both nonsensical inputs (Jia and
Liang, 2017), random sentence/word triggers (Jia and Liang, 2017; Wallace et
al., 2019), and semantically equivalent inputs adversarially selected to
disrupt system behaviour (Ribeiro et al., 2018; Zhao et al., 2018; Iyyer et
al., 2018). In contrast, the questions in RoMQA are not adversarial — they are
written with reasonable information-seeking intent.

#### Zero-shot/few-shot learning

Recent work has also shown that large pretrained LMs can perform zero-shot and
few-shot inference (Brown et al., 2020; Wang et al., 2022; Zhang et al.,
2022). For the former, the LM performs inference given a prompt or an
instruction. For the latter, the LM is additionally given a sample of training
examples as demonstration. We use both settings as baselines for RoMQA, and
find that there is significant room for improvement in large-scale pretraining
to capture compositional reasoning over multiple pieces of evidence text.

##  5 Conclusion

In order to build effective NLP models, we must move towards evaluations that
test model robustness to variations in the input. We presented RoMQA, the
first benchmark for robust, multi-evidence, multi-answer QA. RoMQA evaluates
robustness of models to varying question constraints by testing for worst-case
performance among clusters of related questions. Compared to prior QA
datasets, RoMQA has more natural human-written questions that require
reasoning over more evidence text to more answers. RoMQA is challenging for
state-of-the-art large LMs in zero-shot, few-shot, and supervised settings,
and provides a quantifiable test to build more robust QA methods.

## References

  * Amouyal et al. (2022) Samuel Joseph Amouyal, Ohad Rubin, Ori Yoran, Tomer Wolfson, Jonathan Herzig, and Jonathan Berant. 2022.  QAMPARI: : An open-domain question answering benchmark for questions with many answers from multiple paragraphs.  _arXiv 2205.12665_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _NeurIPS_.
  * Elsahar et al. (2018) Hady Elsahar, Pavlos Vougiouklis, Arslen Remaci, Christophe Gravier, Jonathon Hare, Frederique Laforest, and Elena Simperl. 2018.  T-REx: A large scale alignment of natural language with knowledge base triples.  In _LREC 2018_.
  * Iyyer et al. (2018) Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018.  Adversarial example generation with syntactically controlled paraphrase networks.  In _NAACL_.
  * Jia and Liang (2017) Robin Jia and Percy Liang. 2017.  Adversarial examples for evaluating reading comprehension systems.  In _EMNLP_.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  In _ACL_.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _EMNLP_.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _TACL_.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _ACL_.
  * Min et al. (2022) Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022.  Noisy channel language model prompting for few-shot text classification.  In _ACL_.
  * Min et al. (2020) Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020.  AmbigQA: Answering ambiguous open-domain questions.  In _EMNLP_.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  _arXiv 2203.02155_.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _EMNLP_.
  * Ribeiro et al. (2018) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018.  Semantically equivalent adversarial rules for debugging NLP models.  In _ACL_.
  * Sciavolino et al. (2021) Christopher Sciavolino, Zexuan Zhong, Jinhyuk Lee, and Danqi Chen. 2021.  Simple entity-centric questions challenge dense retrievers.  In _EMNLP_.
  * Talmor and Berant (2018) Alon Talmor and Jonathan Berant. 2018.  The web as a knowledge-base for answering complex questions.  In _NAACL_.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018\.  FEVER: a large-scale dataset for fact extraction and VERification.  In _NAACL_.
  * Vrandečić and Krötzsch (2014) Denny Vrandečić and Markus Krötzsch. 2014.  Wikidata: A free collaborative knowledge base.  _Communications of the ACM_.
  * Wallace et al. (2019) Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019.  Universal adversarial triggers for attacking and analyzing NLP.  In _EMNLP_.
  * Wang et al. (2022) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, et al. 2022\.  Benchmarking generalization via in-context instructions on 1,600+ language tasks.  _arXiv 2204.07705_.
  * Welbl et al. (2018) Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018.  Constructing datasets for multi-hop reading comprehension across documents.  _TACL_.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  In _EMNLP_.
  * Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022.  Opt: Open pre-trained transformer language models.  _arXiv 2205.01068_.
  * Zhao et al. (2018) Zhengli Zhao, Dheeru Dua, and Sameer Singh. 2018.  Generating natural adversarial examples.  In _ICLR_.

##  Appendix A Subsampling propositions

Figure 7: Number of constraints per cluster.

We want questions that cover diverse topics, however Wikidata has a very
skewed proposition distribution, with a long tail of rare propositions. Hence,
we down-sample frequent propositions. Let ${P_{\mathrm{prop}}}({x})$ denote
the percentage of triples that contain the proposition ${x}$. We define the
average proposition probability as
${{{P_{\mathrm{prop}}}^{\prime}}}=\frac{1}{|{\mathit{X}}|}\sum_{x}{P_{\mathrm{prop}}}({x})$.
Given a constraint with proposition ${x}$, we remove it with probability
${r}=1-\min\left(1,\frac{{{{P_{\mathrm{prop}}}^{\prime}}}}{{P_{\mathrm{prop}}}(x)}\right)^{\frac{1}{2}}$.
In particular, those with below average frequency are not removed, and those
with above average frequency are removed with increasing likelihood. After
removing constraints based on propositions, we randomly sample up to 10
constraints using a distribution over their inverse proposition probabilities
$\frac{1}{{P_{\mathrm{prop}}}}$. Figure 7 shows the distribution over cluster
sizes after resampling. Figure 10 shows that resampling results in a more
diverse set of questions by emphasizing rarer propositions in the knowledge
graph.

##  Appendix B Annotation

RoMQA questions are annotated by crowd-workers on Amazon Mechanical Turk from
the US, Canada, UK, Australia, and New Zealand. We require that annotators
have $\geq$95% approval rating and have done a minimum of 5000 HITs. Questions
are submitted for annotation in batches of 500. For each batch, a sample of 10
questions from each worker is inspected by the authors. If $\geq$2 of
annotations in the sample are incorrect, then response from the worker in that
batch are marked for re-annotation. The final set of annotations are
additionally verified by 2 more crowd-workers to confirm that they correspond
to constraints. We keep only examples with 100% agreement, which corresponds
to 89% of the annotated data.

##  Appendix C Dataset Statistics

Figure 8: Answer set size vs constraint count. Figure 9: Positive vs negative
constraint count.

#### Cluster sizes.

Figure 7 shows the distribution of cluster sizes in RoMQA. During the sampling
procedure, we remove small clusters of $\leq$3 questions and avoid large
clusters of $\geq$7 questions.

Figure 10: Most common propositions, before and after subsampling. Subsampling
downsamples overly represented propositions in the knowledge graph and results
in a more diverse set of propositions and question topics. Figure 11:
Mechanical Turk annotation interface for question writing. The annotator is
shown a collection of positive and negative constraints in random order. Each
constraints consists of an entity, a proposition, and a direction. Entities
and propositions are expanded with descriptions and aliases from Wikidata. A
subset of answer entities is listed to disambiguate answer types.

#### Implicit constraint distribution.

Figure 9 shows the distribution of positive and negative implicit constraints
in RoMQA questions. Most questions have 2 positive constraints, and 0-1
negative constraints. We limit questions to 7 constraints during sampling. In
practice, nonsensical questions with too many constraints are filtered out
during verification. Figure 8 shows the distribution of implicit constraints
vs. the size of the answer set. More precise questions with more implicit
constraints typically have fewer answers. In general, RoMQA questions may have
more than 1000 answers, though the vast majority contain less than 1000
answers. The outlier questions with more than 1000 answers are not shown in
the figure.

##  Appendix D Experiment setup

For zero-shot and few-shot models, we use API services provided by the
original authors. For supervised models, we fine-tune `BART-large` models with
learning rate $1e-6$ on V100 GPUs. For classification in the closed setting,
we train with batch size 100 and evaluate with batch size 1000. For generation
in the open setting, we train and evaluate with batch size 2 and decode with
beam size 3.


# Higher-order topological phases in tunable $C_{3}$-symmetric photonic
crystals

Hai-Xiao Wang<EMAIL_ADDRESS>School of Physical Science and Technology,
Guangxi Normal University, Guilin 541004, China  Li Liang  School of Physical
Science and Technology, Guangxi Normal University, Guilin 541004, China  Bin
Jiang  School of Physical Science and Technology, & Collaborative Innovation
Center of Suzhou Nano Science and Technology, Soochow University, 1 Shizi
Street, Suzhou 215006, China  Junhui Hu  School of Physical Science and
Technology, Guangxi Normal University, Guilin 541004, China  Xiancong Lu
Department of Physics, Xiamen University, Xiamen 361005, China  Jian-Hua Jiang
<EMAIL_ADDRESS>School of Physical Science and Technology, &
Collaborative Innovation Center of Suzhou Nano Science and Technology, Soochow
University, 1 Shizi Street, Suzhou 215006, China

###### Abstract

We demonstrate that multiple higher-order topological transitions can be
triggered via the continuous change of the geometry in kagome photonic
crystals composed of three dielectric rods. By tuning a single geometry
parameter, the photonic corner and edge states emerge or disappear with the
higher-order topological transitions. Two distinct higher-order topological
insulator phases and a normal insulator phase are revealed. Their topological
indices are obtained from symmetry representations. A photonic analog of
fractional corner charge is introduced to distinguish the two higher-order
topological insulator phases. Our predictions can be readily realized and
verified in configurable dielectric photonic crystals.

##  I Introduction

Topological phases and phase transitions have been extensively studied in
electronic Hasan2010 ; Qi2011 , photonic Ozawa2019  and acoustic Ma2019 ;
Zhangxj2018  systems in the past decades. Recently, a new class of topological
insulators, higher-order topological insulators (HOTIs), which are
characterized by higher-order bulk-boundary (e.g., bulk-corner or bulk-hinge)
correspondence, were discovered Hughes2017Sci ; Hughes2017prb ; Langbehn2017 ;
Song2017 ; Schindler2018SCI ; Schindler2018Nat ; Huber2018 ; Bahl2018 ;
Imhof2018 ; AQTI ; Hafezi2019 ; Bernevig2020qudrupole1 ; Zhenbo2020quadrupole2
; Xuyong2019qudrupole3 ; JJH2020quadrupole ; JJH2020polariton ;
Meng2020quadrupole ; Ezawa2018 ; Zhangbl2018Kagome ; Khanikaev2018kagome ;
LuMH2018dipole ; JJH2019Natphys ; Hassan2019 ; Zhangshuang2019 ;
Iwamoto2019dipole ; Christensen2019 ; Xiabz2019elastic ; DongJW2019dipole ;
LuMH2019dipole ; JJH2020surfacewave . Prototype HOTIs include quadrupole and
octupole topological insulators Hughes2017Sci ; Hughes2017prb ; AQTI ;
Huber2018 ; Bahl2018 ; Imhof2018 ; Hafezi2019 ; JJH2020quadrupole ;
JJH2020polariton ; Zhenbo2020quadrupole2 ; Meng2020quadrupole ;
ABK2020octupole ; blzhang2020octupole , three-dimensional HOTIs in electronic
systems with topological hinge states Langbehn2017 ; Song2017 ;
Schindler2018SCI ; Schindler2018Nat , and HOTIs with quantized Wannier centers
Ezawa2018 ; Noh2018 ; Zhangbl2018Kagome ; Khanikaev2018kagome ; LuMH2018dipole
; JJH2019Natphys ; Hassan2019 ; Zhangshuang2019 ; Iwamoto2019dipole ;
Christensen2019 ; Xiabz2019elastic ; DongJW2019dipole ; LuMH2019dipole ;
blzhang2020thirdorderTI ; ABK2020thirdorderTI ; JJH2020surfacewave . For
example, quadrupole topological insulators, featured with a fractional bulk
quadrupole moment, host gapped edge states with fractional dipole polarization
and in-gap corner states accompanying a fractional corner charge. Another
example is the $C_{3}$-symmetric two-dimensional (2D) crystals which exhibit
quantized bulk polarization, gapped edge sates and corner states Ezawa2018 ;
Zhangbl2018Kagome ; Khanikaev2018kagome ; Hassan2019 ; ABKphoton . In these
and other types of HOTIs, the crystalline symmetry plays a key role in the
underlying physics. HOTIs set up examples with multidimensional topological
physics going beyond the bulk-edge correspondence in conventional topological
insulators and semimetals and thus offer novel applications in photonics and
phononics.

Despite the extensive studies on HOTIs, 2D photonic crystals (PhCs) with
$C_{3}$ rotation symmetries are rarely studied ABKphoton ; Chen2019 . In
particular, such PhCs have rich higher-order topological phenomena which have
not yet been revealed. Here, we show that by moving the dielectric rods
continuously, the $C_{3}$-symmetric PhCs can switch between triangule and
kagome lattice configurations, leading to rich higher-order topological phases
and phase transitions. Accompanying with such phase transitions, the corner
and edge states emerge or disappear, while the corner charge changes between 0
and $\frac{1}{3}$. The topological indices for various phases are deduced from
the symmetry indicators which are closely related to the fractional corner
charge Hughes2019corcharge . We also discuss the physical meaning of the
fractional corner charge in the photonic context. The richness of the higher-
order topological phases and their evolutions provide intriguing photonic
phenomena and potential applications in topological photonics which can be
readily realized in genuine materials.

Figure 1: (Color online) Geometric transitions in the 2D PhCs with $C_{3}$
symmetry. The primitive cells are indicated by hexagonal dotted lines with the
lattice constant $a$ and the side length $l$. A tunable parameter $d$ with a
range of $0$ to $3l$ (the parameter $d$ is modulo $3l$) is employed to
illustrate the geometric transitions between triangular, kagome and breathing
kagome configurations. By tuning the geometric parameter $d$, the $C_{3}$
symmetry is preserved, while various configurations can be generated,
including (a) triangular I with $d=0$, (b) kagome I with $d=0.5l$, (c)
triangular II with $d=l$, (d) kagome II with $d=1.5l$, (e) triangular III with
$d=2l$, (f) kagome III with $d=2.5l$. Each primitive cell consists of three
dielectric rods (possibly overlaping with each other) with identical radii
$r=0.1a$ and permittivity $\epsilon=15$. Figure 2: (Color online) Photonic
band structures of 2D PhCs with $C_{3}$ symmetry for (a) $d=0,l,2l$ (i.e.,
triangular I, II and III lattices), (b) $d=0.5l,1.5l,2.5l$ (i.e., kagome I, II
and III lattices), (c) $d=0.25l,1.25l,2.25l$ (i.e., breathing kagome
lattices). (d) The eigen-frequencies of the first and second photonic bands at
the $K$ point as functions of $d$. Band gaps of distinct topology are painted
with different colors. The topological index $\chi$ is labeled for each
region. The Wannier center for each region is depicted as well. Insets
illustrate the phase distributions of the eigenstates of the first photonic
band at the $K$ point for various $d$’s.

##  II Higher-order topological phases in tunable $C_{3}$-symmetric photonic
crystals

We study 2D hexagonal PhCs of $C_{3}$ rotation symmetry as illustrated in Fig.
1. The lattice vectors are denoted as $\vec{a}_{1}=(a,0)$ and
$\vec{a}_{2}=(\frac{a}{2},\frac{\sqrt{3}a}{2})$ where $a$ is the lattice
constant. The side length of the unit cell is denoted as $l=a/\sqrt{3}$. The
simplest configuration is the triangular lattice with a dielectric rod at the
center of each unit-cell [Fig. 1(a)] which can be regarded as the special case
where three identical dielectric rods overlap with each other. By moving the
three dielectric rods along the three symmetry lines, as indicated by the
arrows in Fig. 1(b), the PhC undergoes a continuous geometry transformation
which includes three triangular lattice configurations (denoted as triangular
I, II and III) and three kagome lattice configurations (denoted as kagome I,
II and III) [see Figs. 1(b)-1(f)]. The configurations between these six
special cases are the breathing kagome lattices. The whole cycle of the
continuous deformation encompasses $d$ from 0 to $3l$ (see Fig. 1). The kagome
lattices are characterized by $d=(n+\frac{1}{2})l$ with $n=0,1,2$, while the
triangular lattices are characterized by $d=nl$ with $n=0,1,2$.

Intuitively, as the dielectric rods move, the Wannier center changes. We
consider the band gap between the first and the second photonic bands.
Therefore, there is only one Wannier center in the unit-cell which can locate
at the center ($\alpha$) or the corner of the unit-cell ($\beta$ or $\gamma$)
[see Fig. 1(g)]. Unlike the positions of the dielectric rods, the Wannier
center is constrained by the crystalline symmetry and thus cannot change
continuously. The change of the Wannier center is nonadiabatic which has to be
achieved by closing and reopening of the band gap, as shown in details below.

We first provide the photonic band structures for nine prototype cases in
Figs. 2(a-c) where we use $c/a$ as the frequency unit ($c$ is the speed of
light in vacuum). Throughout this work, we focus on the low-lying photonic
bands due to the transverse magnetic (TM) harmonic modes. All the numerical
simulation results are carried out by using the finite element numerical
solver COMSOL Multiphysics. The photonic bands for the triangular I, II and
III configurations are shown in Fig. 2(a) which indicates that the three
triangular configurations have identical band structure. This phenomenon is
because the three triangular configurations differ only by a partial lattice
translation, while the structure of the 2D array of the dielectric rods remain
the same. However, the symmetry representations of the photonic bands are
distinct for the three triangular configurations. Hence, their topological
properties are different. In particular, the location of the Wannier center is
distinct for the three triangular configurations, as revealed below.

Similarly, the photonic band structures for the kagome I, II and III
configurations are identical because they can be related to each other by
partial lattice translations [Fig. 2(b)]. Such translations change the
location of the Wannier center as well as the symmetry representations of the
Bloch bands and their topological properties.

Furthermore, as shown in Fig. 2(d), the photonic band structure is identical,
if two configurations differe by an integer times of $l$ in the geometry
parameter $d$. Since the lattice periodicity for the translation along the
black arrows in Fig. 1(b) is $3l$, there are three different configurations
with the same photonic band structure, where the geometry parameter $d$
differs by an integer times of $l$. The translation of $l$ along the black
arrows in Fig. 1(b) shifts the unit-cell center to the unit-cell corner
without changing the pattern of the 2D array of the dielectric rods. The
photonic band structure is insensitive to such global shifts. Therefore,
configurations differ by an integer times of $l$ in the geometry parameter $d$
have the same photonic band structure. To further demonstrate such a
periodicity of the photonic band structure, we present the photonic bands for
three breathing kagome configurations with $d=0.25l$, $1.25l$, and $d=2.25l$
in Fig. 2(c). It is seen that their photonic band structures are identical
with each other.

The evolution of the first two photonic bands at the $K$ point [i.e. ${\bf
k}=(\frac{4\pi}{3},0)$] with the geometry parameter $d$ is systematically
summarized in Fig. 2(d). There are three topologically distinct photonic band
gaps (regions painted by different colors) which are characterized by three
different locations of the Wannier center, as indicated in the figure. The
band gap between the first two bands experiences closing and reopening during
the change of the parameter $d$. We find that the band gap closes at the
kagome I, II and III configurations where $d=(n+\frac{1}{2})l$ with $n=0,1,2$,
separately. These three $d$’s separate the whole region $d\in[0,3l]$ (bearing
in mind that the parameter $d$ is modulo $3l$, since $3l$ corresponds to a
lattice periodic translation) into three topologically distinct phases:
$d\in(-0.5l,0.5l)$ where the Wannier center is at $\alpha$, $d\in(0.5l,1.5l)$
where the Wannier center is at $\beta$, $d\in(1.5l,2.5l)$ where the Wannier
center is at $\gamma$.

The symmetry representation of the first photonic band at the $K$ point is
depicted by the phase profile of the electric field $E_{z}$ which are shown in
Fig. 2(d) for several $d$’s. We find that the $C_{3}$ symmetry eigenvalue does
not change within the same phase. Upon the topological phase transitions,
i.e., the band gap closing and reopening, the symmetry eigenvalue changes
abruptly. We use the symmetry indicators to characterize the bulk band
topology. Following Ref. Hughes2019corcharge, , the topological crystalline
index can be expressed by the full set of the $C_{3}$ eigenvalues at the high-
symmetry points (HSPs). For a HSP denoted by the symbol $\Pi$, the $C_{3}$
eigenvalues can only be $\Pi_{n}=e^{2\pi(n-1)/3}$ with $n=1,2,3$. Here, the
HSPs include the $\Gamma$, $K$ and $K^{\prime}$ points. The full set of
$C_{3}$ eigenvalues at the HSPs are redundant due to the time-reversal
symmetry and the conservation of the number of bands below the band gap. The
minimum set of indices that describe the band topology are given by
Hughes2019corcharge ,

$[K]_{n}=\\#K_{n}-\\#\Gamma_{n},\quad n=1,2$ (1)

where $\\#K_{n}$ ($\\#\Gamma_{n}$) is the number of bands below the band gap
with the $C_{3}$ symmetry eigenvalue $K_{n}$ ($\Gamma_{n}$) at the $K$
($\Gamma$) point. In this scheme, the $\Gamma$ point is taken as the reference
point to get rid of the redundance. For the trivial atomic insulators (i.e.,
band gap formed by uncoupled atoms), all the HSPs have exactly the same
symmetry eigenvalues. Therefore, the trivial atomic insulators have
$[\Pi_{n}]=0$ for all the HSPs. In constrast, any nonzero $[\Pi_{n}]$
indicates a topological band gap that is adiabatically disconnected from the
trivial atomic insulator.

For the $C_{3}$-symmetric PhCs, the topological indices can be written in a
compact form as

$\chi=[[K_{1}],[K_{2}]].$ (2)

We find that for all $d$’s $\\#\Gamma_{1}=1$ and $\\#\Gamma_{2}=0$.
Furthermore, from the $C_{3}$ eigenvalue at the $K$ point as indicated in Fig.
2(d), we find that the topological index for the three parameter regions are:
$\chi=[0,0]$ for $d\in(-0.5l,0.5l)$, $\chi=[-1,1]$ for $d\in(0.5l,1.5l)$, and
$\chi=[-1,0]$ for $d\in(1.5l,2.5l)$.

Figure 3: (Color online) (a) Schematic illustration of the large triangular
supercells with two types of PhCs. The outer PhC has $d=0.25l$, while the
inner PhC has variable $d$. Several cases with different $d$’s are shown in
(a). (b) Eigen-frequencies of the photons as functions of the geometry
parameter $d$. The green-gray regions represent the bulk states, the green
regions represent the edge states, while the purple and blue curves represent
the type-I and type-II corner states, respectively. (c) Electric field
patterns of corner states and edge states with different $d$. Throughout this
paper, the electric field patterns of the corner states are given by the
superpostion of $|E_{z}|$ of the three degenerate corner states. In the
calculation, the side length of the supercell is $10a$, while the inside
structure has a side length of $4a$. Figure 4: (Color online) (a) Schematic
illustration of the large triangular supercells with two types of PhCs. The
outer PhC has the displacement $d$, while the inner PhC has the displacement
$3l-d$. Several cases with different $d$’s are shown in (a). (b) Eigen-
frequencies of the photons as functions of the geometry parameter $d$. The
green-gray regions represent the bulk states, the green regions represent the
edge states, while the purple and blue curves represent the type-I and type-II
corner states, respectively. (c) Electric field patterns of corner states and
edge states with different $d$. Throughout this paper, the electric field
patterns of the corner states are given by the superpostion of $|E_{z}|$ of
the three degenerate corner states. In the calculation, the side length of the
supercell is $10a$, while the inside structure has a side length of $4a$.

##  III Emergence and evolution of the corner and edge states

Both the phase with $\chi=[-1,1]$ and that with $\chi=[-1,0]$ are higher-order
topological phases that hosting gapped edge states and in-gap corner states.
In contrast, the phase with $\chi=[0,0]$ is the trivial phase. To demonstrate
the higher-order topology, we construct a large triangular supercell which is
schematically shown in Fig. 3(a). In the supercell, the inside is the phase
that we study, while the outside is the trivial band gap phase with $d=0.25l$.
The side length of the supercell is $10a$, while the inside structure has a
side length of $4a$. The whole structure is surrounded by the PEC (i.e.,
perfect electric conductor) boundary condition which is physically a hard-wall
boundary for photons.

We study the evolution of the edge and corner states when the parameter $d$ of
the inside PhC structure goes from 0 to $3l$. A number of prototype geometry
is shown in Fig. 3(a). The results are presented systematically in Fig. 3(b).
Fig. 3(c) gives the electric field $|E_{z}|$ distributions of the eigenstates.
Throughout this paper, the electric field patterns of the corner states are
given by the superpostion of $|E_{z}|$ of the three degenerate corner states.
From the figure, it is seen that the edge and corner states emerge only in the
region with $0.5l<d<2.5l$, i.e., the two higher-order topological phases. In
particular, in the region with $0.5l<d<1.5l$, there are two types of edge
states emerging, as revealed previously in Ref. ABKphoton, : type-I corner
states [denoted by the purple curve in Fig. 3(b), two examples (“A” and “C”)
are shown in Fig. 3(c)] due to the nearest neighbor couplings and type-II
corner states [denoted as the blue curve in Fig. 3(b), two examples (“B” and
“D”) are shown in Fig. 3(c)] due to long-range couplings. As the common band
gap between the inside and outside structures become smaller, at $d=l$, the
type-II corner states are much less localized and become edge states alike
[see “D” in Fig. 3(c)]. In contrast, the type-I corner states remain well-
localized and distinguishable from the edge states [denoted as green in Fig.
3(b)] and bulk states [denoted as green-gray in Fig. 3(b)]. This indicates
that the type-I corner states are due to the bulk topology, while the type-II
corner states may originate from long range couplings between the adjacent
edge states.

The region with $1.5l<d<2.5l$ has not yet been studied in the literature. We
find that in this region the type-II corner states are hardly seen. Meanwhile,
there are two sets of type-I corner states. Each set has three degenerate
corner states. One set has frequency higher than the edge states, while the
other set has frequency lower than the edge states. In both sets, the
wavefunctions of the corner states are well-localized around the corners,
distinghuisable from the edge states (the green band) and the bulk states (the
green-gray bands). Examples of the corner and edge wavefunctions at $d=1.9l$
and $d=2.25l$ are shown in Fig. 3(c). In some cases, for instance, $d=2.25l$,
type-II corner states can be found. However, the wavefunctions are not well-
localized at the corners.

We now explore the corner and edge states in another type of supercell. We
design the supercell in such a way that the inner structure is a PhC with the
parameter $d$ while the outer structure is the PhC with the parameter $3l-d$.
That is, we consider the edge and corner boundaries between complimentary PhC
structures. Such a supercell architecture will induce intriguing edge and
corner boundaries [e.g., various zigzag edge boundaries as depicted
schematically in Fig. 4(a)]. The evolution of the bulk, edge and corner states
are systematically summarized in Fig. 4(b) and 4(c).

In this type of supercells, the edge and corner states emerge only in the two
topological regions, $0.5l<d<1.5l$ and $1.5l<d<2.5l$, as shown in Fig. 4(b).
In the region with $0.5l<d<1.5l$, the inner PhC has the topological index
$\chi=[-1,1]$ while the outer PhC has the topological index $\chi=[-1,0]$. In
the other region with $1.5l<d<2.5l$, the topological indices of the inner and
outer PhCs switch, i.e., the inner PhC has $\chi=[-1,0]$ and the outer PhC has
$\chi=[-1,1]$. The emergence of the edge and corner states, which has not yet
been discovered in the literature, reveals that higher-order topological
phenomena can appear at the boundaries between two topologically distinct
higher-order phases. This is consistent with the topological band theory
Hughes2019corcharge ; JJH2020invsym  and the Wannier center picture.

From Fig. 4(b), the bulk band gap closing is clearly seen at the phase
transition points, $d=0.5l$, $1.5l$ and $2.5l$. Type-II corner states can be
found only in the higher-order phase with $\chi=[-1,1]$ (i.e., the phase
studied in Ref. ABKphoton, ; here $0.5l<d<1.5l$), but not in the higher-order
phase with $\chi=[-1,0]$ (i.e., $1.5l<d<2.5l$). For all cases in the region
$0.5l<d<2.5l$, the edge states are clearly visible [see Fig. 4(c)]. The
bandwidth of the edge states is considerably larger in this type of supercells
compared with the supercell studied in Fig. 3. As a consequence, the corner
states are less localized, particularly in the region with $1.5l<d<2.5l$ where
the corner states live in the small band gap between the edge and bulk states.

Figure 5: Fractional “charges” in the triangular supercell with perfect
electric conductor boundary conditions. Only the charges of the corner unit-
cells and the bulk unit-cells are shown in the figure, as indicated by the
blue areas. Four cases are considered (a) $d=0.25l$, (b) $d=0.75l$, (c)
$d=2.75l$ and (d) $d=2.25l$.

##  IV fractional corner charge

We now show that the higher-order band topology can also be manifested in the
fractional corner charge. Even though we are considering photonic bands and
photonic states in this work, it is possible to define an analog of “charge”
through the local density of states (LDOS), $\rho_{e}({\bf r},E)$. In
electronic systems, the charge contributed by the filling of the valence bands
in the $j$-th unit-cell is given by

$Q_{j,e}=e\int^{E_{\rm gap}}dE\int_{j}d{\bf r}\rho_{e}({\bf r},E).$ (3)

Here, $e$ is the charge of an electron, $E_{\rm gap}$ is an energy in the
topological band gap which is below the eigen-energies of the edge and corner
states. In the above equation, the integration over the position ${\bf r}$ is
defined within the $j$-th unit-cell. The filling of all the valence bands
below the topological band gap contributes a fractional charge in the corner
region. It was predicted Hughes2019corcharge  that the fractional corner
charge $eQ_{c}$ is completely determined by the topological indices of the
bulk bands as follows,

$Q_{c}=-\frac{1}{3}([K_{1}]+[K_{2}])\mod 1.$ (4)

The actual size of the corner region depends on the specific model. However,
one can often choose one or a few unit-cells around the corner boundary to
converge the fractional corner charge.

We then check the theoretical prediction in photonic system by calculating the
following quantity which is the analog of the “charge” in the $j$-th unit-cell
in the photonic system,

$Q_{j}=\int_{0}^{f_{\rm gap}}df\int_{j}d{\bf r}\rho_{p}({\bf r},f).$ (5)

Here, we omitted the elementary charge $e$ which does not have a physical
meaning in photonics. The integration over frequency is from 0 to a frequency
in the band gap $f_{\rm gap}$ which is below the eigen-frequencies of the edge
and corner states. The photonic LDOS is calculated through the following
spectral decomposition of all the photonic eigenstates of the valence bulk
bands

$\rho_{p}({\bf r},f)=\sum_{n}\frac{\Gamma}{\pi}\frac{\epsilon({\bf
r})|E_{z}^{(n)}({\bf r})|^{2}}{(f-f_{n})^{2}+\Gamma^{2}}.$ (6)

Here, $n$ labels the photonic eigenstates of the valence bulk bands, and
$\Gamma\to 0$ is a sufficiently small number that converges the calculation.
$E_{z}^{(n)}({\bf r})$ is the scaled electric field distribution of the $n$-th
photonic eigenstate which satisfies the following normalization condition

$1=\int d{\bf r}\epsilon({\bf r})|E_{z}^{(n)}({\bf r})|^{2}$ (7)

where the integral of ${\bf r}$ is over the whole photonic system and
$\epsilon({\bf r})$ is the position-dependent relative permittivity.

The photonic “charge” defined above does have a physical meaning. It
represents the number of the photonic modes contributed from the $j$-th unit-
cell from the valence bulk bands. We calculate the photonic “charge” for each
unit-cell and present the results in Fig. 5 for various configurations.

For all the four cases considered in Fig. 5, the calculated charge for the
bulk unit-cells are close to 1. This is consistent with the fact that there is
only one band below the band gap, i.e., each unit-cell contributes a single
charge (mode) to the bulk band. Figs. 5(a) and 5(c) show that for both
$d=0.25l$ and $d=2.75l$ (i.e., $\chi=[0,0]$), the fractional corner charge is
zero, since the corner unit-cell has a charge very close to 1. Fig. 5(b) shows
that for $d=0.75l$ (i.e., $\chi=[-1,1]$), the fractional corner charge is
$Q_{c}=0$ which is indicated by that the corner unit-cell has a charge very
close to $0$. In this case, despite that the band gap carries higher-order
topology and the resultant corner states, the corner charge vanishes, which is
consistent with the theoretical prediction given in Eq. (4). Fig. 5(d) shows
that for $d=2.25l$ (i.e., $\chi=[-1,0]$), the fractional corner charge is
$Q_{c}=1/3$ which is manifested by that the corner unit-cell has a charge very
close to $1/3$. These photonic charges can be measured through the classical
or quantum versions of the Purcell effect, as indicated by Ref. disclination,
.

##  V Conclusion

In conclusion, we demonstrate that rich higher-order topological phases and
multiple phase transitions can be obtained in $C_{3}$ symmetric PhCs by tuning
a single geometry parameter $d$. These higher-order topological phases yield
intriguing multidimensional topological phenomena where the corner and edge
states can be tuned in versatile ways. Our study shows that continuously
configurable dielectric PhCs Zhu2018  can be useful in generating topological
photonic circuits with tunable edge and corner states. The emergent fractional
photonic charge indicates that photonic systems can be powerful in revealing
the fundamental properties of topological bands.

###### Acknowledgements.

H.-X. Wang and L. Liang contributed equally to this work. This work is
supported by the National Natural Science Foundation of China under Grant No.
11904060, 12074281. J.-H. Jiang is supported by the Jiangsu specially-
appointed professor funding, and a project funded by the Priority Academic
Program Development of Jiangsu Higher Education Institutions (PAPD).

## References

  * (1) M. Z. Hasan and C. L. Kane, Colloquium: Topological insulators, Rev. Mod. Phys. 82, 3045 (2010). https://journals.aps.org/rmp/abstract/10.1103/RevModPhys.82.3045
  * (2) X.-L. Qi and S.-C. Zhang, Topological insulators and superconductors, Rev. Mod. Phys. 83, 1057 (2011).https://journals.aps.org/rmp/abstract/10.1103/RevModPhys.83.1057
  * (3) T. Ozawa, et al., Topological photonics, Rev. Mod. Phys. 91, 015006 (2019).https://journals.aps.org/rmp/abstract/10.1103/RevModPhys.91.015006
  * (4) G. Ma, M. Xiao, and C. T. Chan, Topological phases in acoustic and mechanical systems, Nat. Rev. Phys. 1, 281 (2019).https://www.nature.com/articles/s42254-019-0030-x
  * (5) X. Zhang, M. Xiao, Y. Cheng, M.-H. Lu, and J. Christensen, Topological sound, Commun. Phys. 1, 97 (2018).https://www.nature.com/articles/s42005-018-0094-4
  * (6) W. A. Benalcazar, B. A. Bernevig, and T. L. Hughes, Quantized electric multipole insulators, Science 357, 61 (2017).https://science.sciencemag.org/content/357/6346/61.abstract
  * (7) W. A. Benalcazar, B. A. Bernevig, and T. L. Hughes, Phys. Rev. B 96, 245115 (2017).https://journals.aps.org/prb/abstract/10.1103/PhysRevB.96.245115
  * (8) S. Mittal, V.V. Orre, G. Zhu, M. A. Gorlach, A. Poddubny, and M. Hafezi, Photonic quadrupole topological phases, Nat. Photon. 13, 692 (2019).https://www.nature.com/articles/s41566-019-0452-0
  * (9) M. Serra-Garcia, V. Peri, R. Susstrunk, O. R. Bilal, T. Larsen, L. G. Villanueva, and S. D. Huber, Observation of a phononic quadrupole topological insulator, Nature 555, 342 (2018).https://www.nature.com/articles/nature25156
  * (10) C. W. Peterson, W. A. Benalcazar, T. L. Hughes, and G. Bahl, A quantized microwave quadrupole insulator with topologically protected corner states, Nature 555, 346 (2018).https://www.nature.com/articles/nature25777
  * (11) S. Imhof, et al., Topolectrical-circuit realization of topological corner modes, Nat. Phys. 14, 925 (2018).https://www.nature.com/articles/s41567-018-0246-1
  * (12) S. Franca, J. van den Brink, and I. C. Fulga, An anomalous higher-order topological insulator, Phys. Rev. B 98, 201114(R) (2018).https://journals.aps.org/prb/abstract/10.1103/PhysRevB.98.201114
  * (13) X. Zhang, Z.-K. Lin, H.-X. Wang, Z. Xiong, Y. Tian, M.-H. Lu, Y.-F. Chen, and J.-H. Jiang, Symmetry-protected hierarchy of anomalous multipole topological band gaps in nonsymmorphic metacrystals, Nat. Commun. 11, 65 (2020). https://www.nature.com/articles/s41467-019-13861-4
  * (14) L. He, Z. Addison, E.J. Mele, and B. Zhen, Quadrupole topological photonic crystals, Nat. Commun. 11, 3119 (2020).https://www.nature.com/articles/s41467-020-16916-z
  * (15) Y. Qi, C. Qiu, M. Xiao, H. He, M. Ke, and Z. Liu, Acoustic Realization of Quadrupole Topological Insulators, Phys. Rev. Lett. 124, 206601 (2020).https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.124.206601
  * (16) Y. Chen, Z.-K. Lin, H. Chen, and J.-H. Jiang, Plasmon-polaritonic quadrupole topological insulators, Phys. Rev. B 101, 041109(R) (2020).https://journals.aps.org/prb/abstract/10.1103/PhysRevB.101.041109
  * (17) J. Langbehn, Y. Peng, L. Trifunovic, F. v. Oppen, and P. W. Brouwer, Reflection-Symmetric Second-Order Topological Insulators and Superconductors, Phys. Rev. Lett. 119, 246401 (2017).https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.119.246401
  * (18) Z. Song, Z. Fang, and C. Fang, $(d-2)$-Dimensional Edge States of Rotation Symmetry Protected Topological States, Phys. Rev. Lett. 119, 246402 (2017).https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.119.246402
  * (19) F. Schindler, A. M. Cook, M. G. Vergniory, Z. Wang, S. S. P. Parkin, B. A. Bernevig, and T. Neupert, Higher-order topological insulators, Sci. Adv. 4, eaat0346 (2018).https://advances.sciencemag.org/content/4/6/eaat0346.abstract
  * (20) F. Schindler, Z. J. Wang, M. G. Vergniory, A. M. Cook, A. Murani, S. Sengupta, A. Y. Kasumov, R. Deblock, S. Jeon, I. Drozdov, H. Bouchiat, S. Gueron, A. Yazdani, B. A. Bernevig, and T. Neupert, Higher-order topology in bismuth, Nat. Phys. 14, 918 (2018).https://www.nature.com/articles/s41567-018-0224-7
  * (21) M. Ezawa, Higher-Order Topological Insulators and Semimetals on the Breathing Kagome and Pyrochlore Lattices, Phys. Rev. Lett. 120, 026801 (2018).https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.120.026801
  * (22) A. E. Hassan, F. K. Kunst, A. Moritz, G. Andler, E. J. Bergholtz, and M. Bourennane, Corner states of light in photonic waveguides, Nat. Photon. 13, 697 (2019).https://www.nature.com/articles/s41566-019-0519-y
  * (23) H. Xue, Y. Yang, F. Gao, Y. Chong, and B. Zhang, Acoustic higher-order topological insulator on a kagome lattice, Nat. Mater. 18, 108 (2019).https://www.nature.com/articles/s41563-018-0251-x
  * (24) X. Ni, M.Weiner, A. Alu, and A. B. Khanikaev, Observation of higher-order topological acoustic states protected by generalized chiral symmetry, Nat. Mater. 18, 113 (2019).https://www.nature.com/articles/s41563-018-0252-9
  * (25) B.-Y. Xie, H.-F. Wang, H.-X. Wang, X.-Y. Zhu, J.-H. Jiang, M.-H. Lu, and Y.-F. Chen, Second-order photonic topological insulator with corner states, Phys. Rev. B 98, 205147 (2018).https://journals.aps.org/prb/abstract/10.1103/PhysRevB.98.205147
  * (26) X. Zhang, H.-X. Wang, Z.-K. Lin, Y. Tian, B. Xie, M.-H. Lu, Y.-F. Chen, and J.-H. Jiang, Second-order topology and multidimensional topological transitions in sonic crystals, Nat. Phys. 15, 582 (2019).https://www.nature.com/articles/s41567-019-0472-1
  * (27) S. Liu, W. Gao, Q. Zhang, S. Ma, L. Zhang, C. Liu, Y. J. Xiang, T. J. Cui, and S. Zhang, Topologically Protected Edge State in Two-Dimensional Su–Schrieffer–Heeger Circuit, Research 2019, 8609875 (2019).https://spj.sciencemag.org/research/2019/8609875
  * (28) Y. Ota, F. Liu, R. Katsumi, K. Watanabe, K. Wakabayashi,Y. Arakawa, and S. Iwamoto, Photonic crystal nanocavity based on a topological corner state, Optica 6, 786 (2019).https://www.osapublishing.org/optica/abstract.cfm?uri=optica-6-6-786
  * (29) Z. Zhang, H. Long, C. Liu, C. Shao, Y. Cheng, X. Liu, and J. Christensen, Deep‐Subwavelength Holey Acoustic Second‐Order Topological Insulators, Adv. Mater. 31, 1904682 (2019).https://onlinelibrary.wiley.com/doi/full/10.1002/adma.201904682
  * (30) H. Fan, B. Xia, L. Tong, S. Zheng, and D. Yu, Elastic Higher-Order Topological Insulator with Topologically Protected Corner States, Phys. Rev. Lett. 122, 204301 (2019).https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.122.204301
  * (31) X.-D. Chen, W.-M. Deng, F.-L. Shi, F.-L. Zhao, M. Chen, and J.-W. Dong, Direct Observation of Corner States in Second-Order Topological Photonic Crystal Slabs, Phys. Rev. Lett. 122, 233902 (2019).https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.122.233902
  * (32) B.-Y. Xie, G.-X. Su, H.-F. Wang, H. Su, X.-P. Shen, P. Zhan, M.-H. Lu, Z.-L. Wang, and Y.-F. Chen, Visualization of Higher-Order Topological Insulating Phases in Two-Dimensional Dielectric Photonic Crystals, Phys. Rev. Lett. 122, 233903 (2019).https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.122.233903
  * (33) L. Zhang, Y. Yang, P. Qin, Q. Chen, F. Gao, E. Li, J.-H. Jiang, B. Zhang, and H. Chen, Higher‐Order Topological States in Surface‐Wave Photonic Crystals, Adv. Sci. 7, 1902724 (2020).https://onlinelibrary.wiley.com/doi/full/10.1002/advs.201902724
  * (34) B. J. Wieder, Z. Wang, J. Cano, X. Dai, L. M. Schoop, B. Bradlyn, and B. A. Bernevig, Strong and fragile topological Dirac semimetals with higher-order Fermi arcs, Nat. Commun. 11, 627 (2020).https://www.nature.com/articles/s41467-020-14443-5
  * (35) Q.-B. Zeng, Y.-B. Yang, and Y. Xu, Higher-order topological insulators and semimetals in generalized Aubry-André-Harper models, Phys. Rev. B 101, 241104(R) (2020).https://journals.aps.org/prb/abstract/10.1103/PhysRevB.101.241104
  * (36) X. Ni, M. Li, M.Weiner, A. Alu, and A. B. Khanikaev, Demonstration of a quantized acoustic octupole topological insulator, Nat. Commun. 11, 2108 (2020).https://www.nature.com/articles/s41467-020-15705-y
  * (37) H. Xue, Y. Ge, H.-X. Sun, Q. Wang, D. Jia, Y.-J. Guan, S.-Q. Yuan, Y. Chong, and B. Zhang, Observation of an acoustic octupole topological insulator, Nat. Commun. 11, 2442 (2020).https://www.nature.com/articles/s41467-020-16350-1
  * (38) J. Noh, W. A. Benalcazar, S. Huang, M. J. Collins, K. P. Chen, T. L. Hughes, and M. C. Rechtsman, Topological protection of photonic mid-gap defect modes, Nat. Photon. 12, 408 (2018).https://www.nature.com/articles/s41566-018-0179-3
  * (39) H. Xue, Y. Yang, G. Liu, F. Gao, Y. Chong, and B. Zhang, Realization of an Acoustic Third-Order Topological Insulator, Phys. Rev. Lett. 122, 244301 (2019).https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.122.244301
  * (40) M. Weiner, X. Ni, M. Y. Li, A. Alu, and A. B. Khanikaev, Demonstration of a 3rd order hierarchy of higher order topological states in a three-dimensional acoustic metamaterial, Sci. Adv. 6, eaay4166 (2020).https://advances.sciencemag.org/content/6/13/eaay4166
  * (41) M. Li, D. Zhirihin, M. Gorlach, X. Ni, D. Filonov, A. Slobozhanyuk, A. Alu, and A. B. Khanikaev, Higher-order topological states in photonic kagome crystals with long-range interactions, Nat. Photonics 14, 89-94 (2020).https://www.nature.com/articles/s41566-019-0561-9
  * (42) Y. Chen, X. Lu, and H. Chen, Effect of truncation on photonic corner states in a Kagome lattice, Opt. Lett. 44 4251 (2019).https://www.osapublishing.org/ol/abstract.cfm?uri=ol-44-17-4251
  * (43) W. A. Beanalcazar, T. Li, and T. L. Hughes, Quantization of fractional corner charge in $C_{n}$-symmetric higher-order topological crystalline insulators, Phys. Rev. B 99, 245151 (2019).https://journals.aps.org/prb/abstract/10.1103/PhysRevB.99.245151
  * (44) Z. Xiong, Z.-K. Lin, H.-X. Wang, X. Zhang, M.-H. Lu, Y.-F. Chen, and J.-H. Jiang, Corner states and topological transitions in two-dimensional higher-order topological sonic crystals wit inversion symmetry, Phy. Rev. B 102, 125144 (2020).https://journals.aps.org/prb/abstract/10.1103/PhysRevB.102.125144
  * (45) Y. Liu, S. Leung, F.-F. Li, Z.-K. Lin, X. Tao, Y. Poo, and J.-H. Jiang, Experimental discovery of bulk-disclination correspondence. Nature 589, 381-385 (2021).
  * (46) X. Zhu, H.-X. Wang, C. Xu, J.-H. Jiang, and S. John, Topological transitions in continuously deformed photonic crystals, Phy. Rev. B 97, 085148 (2018).https://doi.org/10.1103/PhysRevB.97.085148


#  Neutron skin thickness of 208Pb, 116,120,124Sn, and 40Ca
determined from reaction cross sections of 4He scattering

Masayuki Matsuzaki  Department of Physics, Fukuoka University of Education,
Munakata, Fukuoka 811-4192, Japan  Shingo Tagami  Department of Physics,
Kyushu University, Fukuoka 819-0395, Japan  Masanobu Yahiro
<EMAIL_ADDRESS>Department of Physics, Kyushu University, Fukuoka
819-0395, Japan

###### Abstract

* Background

We constructed the Kyushu chiral $g$-matrix and confirmed its reliability at
$30~{}\,$ E_in $\stackrel{{\scriptstyle<}}{{\widetilde{}}}$100 $~{}MeVand$250
$\stackrel{{\scriptstyle<}}{{\widetilde{}}}$E_in
$\stackrel{{\scriptstyle<}}{{\widetilde{}}}$400$~{}MeVfor$^12$Cscattering.Reactioncrosssectiondataof$^4$Hescatteringareavailableforsomenuclidesincluding$^208$Pb.PREXIIcollaborationreportedathickneutronskinfor$^208$Pb.\par\description@item@[Purpose]Ourpurposeistodeduceneutronskinthicknessesof$^208$Pbandsomeothernuclidesfromreactioncrosssectionscalculatedintermsofthedoublefoldingmodelwiththe$g$-matrix.\par\description<EMAIL_ADDRESS>constrainednessoftheneutronsector,werenormalizedensitiessoastoreproducetheobservedcrosssections.\par\description@item@[Results]Wefoundthata3.4~{}$%$renormalizationisnecessaryfor$^208$Pb.Theneutrondensityobtainedfromrenormalizationresultsin$R_skin=$0.416$±$0.146fmbyconfrontingtheprecisionprotonradius.\par\description@item@[Conclusions]OurresultisconsistentwithPREXIIandthereforesupportslargerslopeparameter$L$.Resultsfor$^40$Caand$^124$Snarealsoconsistentwith$R_skin$deducedfromotherexperiments.For$^116,120$Snthepresentmethodgivesthickerskins.\par$


* Kubo, Toda, and Hashitsume (1998) R. Kubo, M. Toda, and N. Hashitsume,  _Statistical Physics II_ (Springer Berlin Heidelberg, 1998).
  * Loring, Yan, and Mukamel (1987) R. F. Loring, Y. J. Yan, and S. Mukamel, J. Chem. Phys. 87, 5840 (1987).
  * Sparpaglione and Mukamel (1988) M. Sparpaglione and S. Mukamel, J. Chem. Phys. 88, 3263 (1988).
  * Hu and Mukamel (1989) Y. Hu and S. Mukamel, J. Chem. Phys. 91, 6973 (1989).
  * Borgis and Hynes (1991) D. Borgis and J. T. Hynes, J.Chem. Phys. 94, 3619 (1991).
  * Islampour and Lin (1991a) R. Islampour and S. H. Lin, J. Phys. Chem. 95, 10261 (1991a).
  * Islampour and Lin (1991b) R. Islampour and S. H. Lin, Chem. Phys. Lett. 179, 147 (1991b).
  * Islampour  _et al._ (1993) R. Islampour, R. G. Alden, G. Y. C. Wu, and S. H. Lin, J. Phys. Chem. 97, 6793 (1993).
  * Cho and Silbey (1995) M. Cho and R. J. Silbey, J. Chem. Phys. 103, 595 (1995).
  * Georgievskii, Hsu, and Marcus (1999) Y. Georgievskii, C.-P. Hsu, and R. A. Marcus, J. Chem. Phys. 110, 5307 (1999).
  * Soudackov, Hatcher, and Hammes-Schiffer (2005) A. Soudackov, E. Hatcher, and S. Hammes-Schiffer, J. Chem. Phys. 122, 014505 (2005).
  * Renger and Marcus (2003) T. Renger and R. A. Marcus, J. Phys. Chem. A 107, 8404 (2003).
  * Zhu  _et al._ (2009) C. Zhu, K. K. Liang, M. Hayashi, and S. H. Lin, Chem. Phys. 358, 137 (2009).
  * Ma and Cao (2015) J. Ma and J. Cao, J. Chem. Phys. 142, 094106 (2015).
  * Soudackov and Hammes-Schiffer (2015) A. V. Soudackov and S. Hammes-Schiffer, J. Chem. Phys. 143, 194101 (2015).
  * Soudackov and Hammes-Schiffer (2016) A. V. Soudackov and S. Hammes-Schiffer, Faraday Discuss. 195, 171 (2016).
  * Note (7) We can understand this behaviour from a comparison with instanton theory, whose value of $\tau$ gives a simple measure of the relative importance of the reactant and product dynamics in the calculation. For $\varepsilon=\Lambda$, $\tau=0$ which implies that all dynamics take place on the reactant, but at $\varepsilon=0$, $\tau$ approaches $\beta\hbar/2$ (which is only strictly true for a symmetric system) so both are approximately equally important. The cumulant expansion treats the reactant state on a higher level than the product state and is thus expected to work best near the activationless regime ($\varepsilon=\Lambda$). Hence, it is a good method for predicting optical lineshapes which are largest at this point, but not in general for rate calculations.
  * Chandler (1987) D. Chandler,  _Introduction to Modern Statistical Mechanics_ (Oxford University Press, New York, 1987).
  * Lax (1952) M. Lax, J. Chem. Phys. 20, 1752 (1952).
  * Curie (1963) D. Curie,  _Luminescence in Crystals_ (Wiley, New York, 1963) pp. 47–52.
  * Hopfield (1974) J. J. Hopfield, Proc. Natl. Acad. Sci. 71, 3640 (1974).
  * Note (8) This lineshape function is related to the absorption spectrum calculated in Ref. GoldenInverted with an excitation frequency corresponding to $\hbar\omega_{\text{ex}}\equiv\varepsilon-\Delta V^{\text{b}}$.
  * Kleinert (2009) H. Kleinert,  _Path Integrals in Quantum Mechanics, Statistics, Polymer Physics and Financial Markets_ , 5th ed. (World Scientific, Singapore, 2009).
  * Note (9) There is also no problem that the predicted energy is lower than the zero-point energy. In fact, semiclassical trajectories predict the exact partition function of the harmonic oscillator at any temperature despite having an energy of 0 Richardson (2018b).
  * Mündel and Domcke (1984) C. Mündel and W. Domcke, J. Phys. B: At. Mol. Phys. 17, 3593 (1984).
  * Bunkin and Tugov (1973) F. Bunkin and I. Tugov, Phys. Rev. A 8, 601 (1973).
  * Marcus (1982b) R. A. Marcus, in  _Oxidases and Related Redox Systems_, edited by T. E. King, M. Morrison, and H. S. Mason (Pergamon, New York, 1982) pp. 3–19.