#  Tasty Burgers, Soggy Fries: Probing Aspect Robustness in
Aspect-Based Sentiment Analysis

Xiaoyu Xing1∗ Zhijing Jin2 Di Jin3 Bingning Wang4 Qi Zhang1 Xuanjing Huang1
1Fudan University 2Max Planck Institute, 3CSAIL, MIT, 4Sogou Inc.
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>Equal Contribution. Please email
(or cc’) both first authors for correspondence.

###### Abstract

Aspect-based sentiment analysis (ABSA) aims to predict the sentiment towards a
specific aspect in the text. However, existing ABSA test sets cannot be used
to probe whether a model can distinguish the sentiment of the target aspect
from the non-target aspects. To solve this problem, we develop a simple but
effective approach to enrich ABSA test sets. Specifically, we generate new
examples to disentangle the confounding sentiments of the non-target aspects
from the target aspect’s sentiment. Based on the SemEval 2014 dataset, we
construct the Aspect Robustness Test Set (ARTS) as a comprehensive probe of
the aspect robustness of ABSA models. Over 92% data of ARTS show high fluency
and desired sentiment on all aspects by human evaluation. Using ARTS, we
analyze the robustness of nine ABSA models, and observe, surprisingly, that
their accuracy drops by up to 69.73%. We explore several ways to improve
aspect robustness, and find that adversarial training can improve models’
performance on ARTS by up to 32.85%.111Our code and new test set are available
at https://github.com/zhijing-jin/ARTS_TestSet.

##  1 Introduction

Aspect-based sentiment analysis (ABSA) is an advanced sentiment analysis task
that aims to classify the sentiment towards a specific aspect (e.g., burgers
or fries in the review “Tasty burgers, and crispy fries.”). The key to a
strong ABSA model is it is sensitive to only the sentiment words of the target
aspect, and therefore not be interfered by the sentiment of any non-target
aspect. Although state-of-the-art models have shown high accuracy on existing
test sets, we still question their robustness. Specifically, given the
prerequisite that a model outputs correct sentiment polarity for the test
example, we have the following questions:

  1. (Q1)

If we reverse the sentiment polarity of the target aspect, can the model
change its prediction accordingly?

  2. (Q2)

If the sentiments of all non-target aspects become opposite to the target one,
can the model still make the correct prediction?

  3. (Q3)

If we add more non-target aspects with sentiments opposite to the target one,
can the model still make the correct prediction?

A robust ABSA model should both meet the prerequisite and have affirmative
answers to all the questions above. For example, if a model makes the correct
sentiment classification (i.e., positive) for burgers in the original sentence
“Tasty burgers, and crispy fries”, it should flip its prediction (to negative)
when seeing the new context “Terrible burgers, but crispy fries”. Hence, these
questions together form a probe to verify if an ABSA model has high aspect
robustness.

SubQ. | Generation Strategy | Example
---|---|---
Prereq. |  Source: The original sample from the test set |  Tasty burgers, and crispy fries. (Tgt: burgers)
Q1 |  RevTgt: Reverse the sentiment of the target aspect |  Terrible burgers, but crispy fries.
Q2 |  RevNon: Reverse the sentiment of the non-target aspects with originally the same sentiment as target |  Tasty burgers, but soggy fries.
Q3 |  AddDiff: Add aspects with the opposite sentiment from the target aspect |  Tasty burgers, crispy fries, but poorest service ever!

Table 1: The generation strategies and examples of the prerequisite (Prereq)
and three questions (Q1)-(Q3). Each example are annotated with the target
aspect (Tgt), and altered sentence parts.

Unfortunately, existing ABSA datasets have very limited capability to probe
the aspect robustness. For example, the Twitter dataset Dong et al. (2014) has
only one aspect per sentence, so the model does not need to discriminate
against non-target aspects. In the most widely used SemEval 2014 Laptop and
Restaurant datasets Pontiki et al. (2014), for 83.9% and 79.6% instances in
the test sets, the sentiments of the target aspect, and all non-target aspects
are all the same. Hence, we cannot decide whether models that make correct
classifications attend only to the target aspect, because they may also
wrongly look at the non-target aspects, which are confounding factors. Only a
small portion of the test set can be used to answer our target questions
proposed in the beginning. Moreover, when we test on the subset of the test
set (59 instances in Laptop, and 122 instances in Restaurant) where the target
aspect sentiment differs from all non-target aspect sentiments (so that the
confounding factor is disentangled), the best model Xu et al. (2019a) drops
from 78.53% to 59.32% on Laptop and from 86.70% to 63.93% on Restaurant. This
implies that the success of previous models may over-rely on the confounding
non-target aspects, but not necessarily on the target aspect only. However, no
datasets can be used to analyze the aspect robustness more in depth.

We develop an automatic generation framework that takes as input the original
test instances from SemEval 2014, and applies three generation strategies
showed in Table 1. New test instances generated by RevTgt, RevNon, and AddDiff
can be used to answer the questions (Q1)-(Q3), respectively. The generated new
instances largely overlap with the content and aspect terms of the original
instances, but manage to disentangle the confounding sentiment polarity of
non-target aspects from the target, as showed in the examples in Table 1. In
this way, we produce an “all-rounded” test set that can test whether a model
robustly captures the target sentiment instead of other irrelevant clues.

We enriched the laptop dataset by 294% from 638 to 1,877 instances and the
restaurant dataset by 315% from 1,120 to 3,530 instances. By human evaluation,
more than 92% of the new aspect robustness test set (ARTS) shows high fluency,
and desired sentiment on all aspects. Our ARTS test set is in line with other
recent works on NLP challenge sets McCoy et al. (2019); Gardner et al. (2020).
Using our new test set, we analyze the aspect robustness of nine existing
models. Experiment results show that their performance degrades by up to
55.64% on Laptop and 69.73% on Restaurant. We also use our generation strategy
to conduct adversarial training and find it improves aspect robustness by at
least 11.87% and at most 35.37% across various models.

The contributions of our paper are as follows:

  1. 1.

We develop simple but effective automatic generation methods that generate new
test instances (with over 92% accuracy by human evaluation) to challenge the
aspect robustness.

  2. 2.

We construct ARTS, a new test set targeting at aspect robustness for ABSA
models, and propose a new metric, Aspect Robustness Score.

  3. 3.

We probe the aspect robustness of nine models, and reveal up to 69.73%
performance drop on ARTS compared with the original test set.

  4. 4.

We provide several solutions to enhance aspect robustness for ABSA models,
including adversarial training detailed in Section 5.4.

##  2 Data Generation

As shown in Table 1, we aim to build a systematic method to generate all
possible aspect-related alternations, in order to remove the confounding
factors in the existing ABSA data. In the following, we will introduce three
different ways to disentangle the sentiment of the target aspect from
sentiments of non-target aspects.

###  2.1 RevTgt

The first strategy is to generate sentences that reverse the original
sentiment of the target aspect. The word spans of each aspect’s sentiment of
SemEval 2014 data are provided by Fan et al. (2019a). We design two methods to
reverse the sentiment, and one additional step of conjunction adjustment on
top of the two methods to polish the resulting sentence.

Strategy | Example
---|---
Flip Opinion |  It’s light and easy to transport.
$\rightarrow$ It’s heavy and difficult to transport.
Add Negation |  The menu changes seasonally.
$\rightarrow$ The menu does not change seasonally.
Adjust |  The food is good, and the decor is nice.
Conjunctions |  $\rightarrow$ The food is good, but the decor is nasty.
Table 2: Three strategies and examples of RevTgt. Strategy | Example
---|---
Original sentence & sentiment | It has great food and a reasonable price, but the service is poor.
(Tgt) food:$+$ |  price:$+$ |  service:$-$ |  overall:$\Circle$
RevNon |  |  |  |
Flip same-sentiment non-target aspects (and adjust conjunctions) | It has great food but an unreasonable price, and the service is poor.
Exaggerate opposite-sentiment | It has great food but an unreasonable price, and the service is extremely poor.
non-target aspects |  (Tgt) food:$+$ |  price:$-$ |  service:$--$ |  overall:$-$
Table 3: The generation process of RevNon. The target aspect (Tgt), and
sentiments of all aspects are annotated.

#### Flip Opinion Words

Suppose we have the sentence “Tasty burgers and crispy fries,” where the
sentiment term for the target aspect is Tasty. We aim to generate a new
sentence that flips the sentiment Tasty. A baseline approach is antonym
replacement by looking up WordNet Miller (1995). However, due to polysemy, the
simple lookup is very likely to derive an inappropriate antonym and cause
incompatibility with the context. Among the retrieved set of antonyms, we only
keep words with the same Part-of-Speech (POS) tag as original, using the
stanza package222https://stanfordnlp.github.io/stanza/ which takes the context
into account by the state-of-the-art neural network-based model.333For the
candidate filter, we do not use GPT-2 perplexity because its low accuracy,
e.g., 38.4% on a random sample set. And its output is also less interprettable
than the POS filter. Lastly, in the case of multiple antonyms, we prioritize
the words that are already in the existing vocabulary, and then randomly
select an antonym from the candidate set.

#### Add Negation

As the above strategy of flipping by the antonym is constrained by whether
appropriate antonyms are available. For those cases without suitable antonyms,
including long phrases, we add negation according to the linguistic features.
In most cases, the sentiment expression is an adjective or verb term, so we
simply add negation (i.e., “not”) in front of it. If the sentiment term is not
an adjective or verb, we add negation to its closest verb. For example, in
Table 2, there are no available antonyms for “change” in the original example
“The menu changes seasonally.”, so we simply negate it as “The menu does not
change seasonally.”

#### Adjust Conjunctions

As pinpointed in Section 1, 79.6$\sim$83.9% of the original test data of
SemEval 2014 Pontiki et al. (2014) have the same sentiment for all aspects. A
possible result of reverting one aspect’s sentiment is that the other aspects’
sentiments will be opposite to the altered one. So we need to adjust the
conjunctions for language fluency. If the two closest surrounding sentiments
of a conjunction word have the same polarity, then cumulative conjunctions
such as “and” should be applied; otherwise, we should adopt adversative
conjunctions such as “but.” In the example in Table 2, after flipping the
sentiment, we derive the example “The food is good, and the decor is nasty”
which is very unnatural, so we replace the conjunction “and” with “but,” and
thus generate the example “The food is good, but the decor is nasty.”

###  2.2 RevNon

Changing the target sentiment by RevTgt can test if a model is sensitive
enough towards the target-aspect sentiment, but we need to further complement
this probe by perturbing the sentiments of the non-target aspects (RevNon). As
showed in Table 3, for all the non-target aspects with the same sentiment as
the target aspect’s, we reverse their sentiments using the same method as
RevTgt. And for all the remaining non-target aspects, whose sentiments are
already opposite from the target sentiment, we exaggerate the extent by
randomly adding an adverb (e.g., “very”, “really” and “extremely”) from a
dictionary of adverbs of degree that is collected based on the training set.
The resulting test example will be a solid proof of the ABSA quality, because
only the target aspect has the desired sentiment, and all non-target aspects
have been flipped to or exaggerated with the opposite sentiment.

###  2.3 AddDiff

The first two strategies, RevTgt and RevNon, have explored how the sentiment
changes of existing aspects will challenge an ABSA model, and AddDiff further
investigate if adding more non-target aspects can confuse the model. Moreover,
the existing SemEval 2014 test sets have only on average 2 aspects per
sentence, but the real-world applications can have more aspects. With these
motivations, we develop AddDiff as follows.

We first form a set of aspect expressions $\mathrm{AspectSet}$444The full
$\mathrm{AspectSet}$ is available on our GitHub. by extracting all aspect
expressions from the entire dataset. Specifically, for each example in the
dataset, we first identify each sentiment term (e.g., “reasonable” in “Food at
a reasonable price”) and then extract its linguistic branch as the aspect
expression (e.g., “at a reasonable price”) by pretrained constituency parsing
Joshi et al. (2018). Table 4 shows several examples of $\mathrm{AspectSet}$ in
the restaurant domain.

Sentiment | Aspect Expression
---|---
Positive |  staff is friendly and knowledgeable
desserts are out of this world
texture is a velvety
Negative |  service is severely slow
dining experience is miserable
tables are uncomfortably close

Table 4: Example aspect expressions from $\mathrm{AspectSet}$ of the
restaurant domain.

Using the $\mathrm{AspectSet}$, we randomly sample 1-3 aspects that are not
mentioned in the original test sample and whose sentiments are different from
the target aspect’s, and then append these to the end of the original example.
For example, “Great food and best of all GREAT beer!”
$\xrightarrow{\textsc{AddDiff}}$ “Great food and best of all GREAT beer, but
management is less than accommodating, music is too heavy, and service is
severely slow.” In this way, AddDiff enables the advanced testing of whether
the model will be confused when there are more irrelevant aspects with
opposite sentiments.

##  3 ARTS Dataset

###  3.1 Overview

Our source data is the most555We surveyed deep learning-based ABSA papers from
2015 to 2020 at top conferences (ACL, EMNLP, NAACL, NeurIPS, ICLR, ICML, AAAI,
IJCAI). Among the 63 ABSA papers, 50 use SemEval 2014 Laptop and Restaurant,
which is the top 1 widely used dataset. widely used ABSA dataset, SemEval 2014
Laptop and Restaurant Reviews Pontiki et al.
(2014).666http://alt.qcri.org/semeval2014/task4/ We follow Wang et al. (2016);
Ma et al. (2017); Xu et al. (2019a) to remove instances with conflicting
polarity and only keep positive, negative, and neutral labels. We use the
train-dev split as in Xu et al. (2019a). The resulting Laptop dataset has
2,163 training, 150 validation, and 638 test instances, and Restaurant has
3,452 training, 150 validation, and 1,120 test instances.

Building upon the original SemEval 2014 data, we generate enriched test sets
of 1,877 instances (294% of the original size) in the laptop domain, and 3,530
instances (315%) in the restaurant domain using generation method introduced
in Section 2. The statistics of our ARTS test set are in Table 5. (A more
detailed explanations of the number of instances generated by each strategy is
in Appendix A.1.)

| Laptop | Restaurant
---|---|---
Original Test Set | 638 | 1,120
Enriched Test Set | 1,877 | 3,530
Relative Size | 294.20% | 315.17%
Fluency Check
Accepted Instances | 1,732 | 3,260
Fixed Instances | 145 | 270
Acceptance Rate | 92.27% | 92.35 %
Inter-Agreement | 91.10% | 92.69%
Sentiment Check
Accepted Instances | 1,763 | 3,362
Fixed Instances | 114 | 168
Acceptance Rate | 93.93% | 95.24%
Inter-Agreement | 94.14% | 95.61%

Table 5: Overall statistics of the ARTS test set and results of fluency and
sentiment checks.

###  3.2 Quality Inspection

We conduct human evaluation to validate the generation quality of our ARTS
dataset on two criteria:

  1. 1.

Fluency: Does the generated sentence maintain the fluency of the source
sentence?

  2. 2.

Sentiment Correctness: Does the sentiment of each aspect have the desired
polarity?

  * •

RevTgt: Is the target sentiment reversed?

  * •

RevNon: For non-target aspects with originally the same sentiment as the
target, is it reversed? For the rest, are they exaggerated?

  * •

AddDiff: Is the target sentiment unchanged?

Each task is completed by two native-speaker judges. We first calculate the
inter-agreement rate of the human annotators, and then resolve the divergent
opinions on samples that they disagree with. We accept the samples that both
judges considered as correct or are resolved to be correct after our check.
Finally, we ask the annotators to fix the rejected samples by minimal edit
which does not change the aspect term or the sentence meaning, but satisfies
both criteria.

#### Fluency Check

The evaluation results on fluency are showed in Table 5. Most samples (92.27%
of Laptop and 92.35% of Restaurant test sets) are accepted as fluent text. The
inter-agreement rate between the two human judges is also high, 91.10% and
92.69% on the two datasets.

#### Sentiment Check

We also evaluate the sentiment correctness of the generated text. Note that
for RevNon, we count the samples with all “yes” answers as accepted samples.
Overall, the acceptance rate of the generated samples is 93.93% on Laptop and
95.24% on Restaurant, along with inter-agreement rates of over 94.14% on both
datasets.

###  3.3 Dataset Analysis

After checking the quality of our enriched ARTS test set, we analyze the
dataset characteristics and make comparisons with the original test sets.

| Laptop | Restaurant
---|---|---
| Ori | ARTS | Ori | ARTS
#Words/Sent | 18.56 | 22.27 | 19.37 | 23.15
Vocab Size | 1565 | 1746 | 2197 | 2451
Labels |  |
Positive | 341 | 883 | 728 | 1953
Negative | 128 | 587 | 196 | 1104
Neutral | 169 | 407 | 196 | 473
#Positive/#Negative | 2.66 | 1.5 | 3.71 | 1.77
Aspect-Related Challenge |  |
#Aspects/Sent | 2.05 | 2.75 | 2.57 | 3.28
Opp. Nontgt $\geq$ 1 | 16% | 59% | 20% | 67%
Opp. Nontgt $=$ All | 9% | 38% | 11% | 42%
#Opp. Nontgt/Sent | 0.23 | 1.16 | 0.27 | 1.39

Table 6: Characteristics of the ARTS test sets in comparison to the Original
(“Ori”) Laptop and Restaurant test sets.

For general statistics, we can see from Table 6 that the sentence length in
the new test set is on average 4 words more than the original, and the
vocabulary size is also larger by around two hundred. For the label
distribution, we can see that the new test set has an increasing number of all
labels, and especially balances the ratio of positive-to-negative labels from
the original 2.66 to 1.5 on Laptop, and from 3.71 to 1.77 on Restaurant.

For the aspect-related challenge in the test set, the new test set, first of
all, has a larger number of aspects per sentence than the original. Our test
set also features the higher disentanglement of the target aspect from the
non-target aspects that have the same sentiment as the target: the portion of
samples with at least one non-target aspects of sentiments different from the
target is 59$\sim$67%, and on average 45% higher than the original test sets.
And the portion of the most challenging samples where all non-target aspects
have sentiments different from the target one on the new test set is on
average 30% more than that of the original test set. The average number of
non-target aspects with opposite sentiments per sample in the new test set is
on average 5 times that of the original set.

###  3.4 Aspect Robustness Score (ARS)

As mentioned in Section 1, a model is considered to have high aspect
robustness if it satisfies both the prerequisite and all three questions
(Q1)-(Q3). So we propose a novel metric, Aspect Robustness Score (ARS), that
counts the correct classification of the source example and all its variations
(RevTgt, RevNon, and AddDiff) as one unit of correctness. Then we apply the
standard calculation of accuracy. Note that the three variations correspond to
questions (Q1)-(Q3), respectively.

##  4 Evaluating ABSA Models

We use our enriched test set as a comprehensive test on the aspect robustness
of ABSA models.

###  4.1 Models

For a comprehensive overview of the ABSA field, we conduct extensive
experiments on models with a variety of neural network architectures.

TD-LSTM: Tang et al. (2016a) uses two Long Short-Term Memory Networks (LSTM)
to encode the preceding and following contexts of the target aspect
(inclusive) and concatenate the last hidden states of the two LSTMs to make
the sentiment classification.

AttLSTM: Wang et al. (2016) apply an Attention-based LSTM on the
concatenatation of the aspect and word embeddings of each token.

GatedCNN: Xue and Li (2018) use a Gated Convolutional Neural Networks (CNN)
that applies a Tanh-ReLU gating mechanism to the CNN-encoded text with aspect
embeddings.

MemNet: Tang et al. (2016b) use memory networks to store the sentence as
external memory and calculate the attention with the target aspect.

GCN: Aspect-specific Graph Convolutional Networks (GCN) Zhang et al. (2019a)
first applies GCN over the syntax tree of the sentence and then imposes an
aspect-specific masking layer on its top.

BERT: Xu et al. (2019a) uses a BERT-based baseline Devlin et al. (2019) and
takes as input the concatenation of the aspect term and the sentence.

BERT-PT: Xu et al. (2019a) post-train BERT on other review datasets such as
Amazon laptop reviews He and McAuley (2016) and Yelp Dataset Challenge
reviews, and finetune on ABSA tasks.

CapsBERT: Jiang et al. (2019) encode the sentence and the aspect term with
BERT, and then feed it into Capsule Networks to predict the polarity.

BERT-Sent: For more in-depth analysis, we also implement a sentence
classification baseline. BERT-Sent takes as input sentences without aspect
information, and predicts the “global” sentiment. We use it because if other
ABSA models fails to pay attention to aspects, they will degenerate to a
sentence classifier. If so, they will resemble BERT-Sent, which performs well
on original tests and badly on ARTS. So BERT-Sent is a reference to check
degenerated aspect-level models.

###  4.2 Implementation Details

For all existing models, we use the authors’ official implementation. For our
self-proposed BERT-Sent, we use Adam optimizer with a learning rate of 5e-5,
weight decay of 0.01, and batch size of 32. We apply the $l_{2}$
regularization with $\lambda=10^{-4}$, and train 50 epochs. Note that the
tokenization of the ASTS dataset is the same as the original SemEval 2014, as
we prepared the new test set by inverting the NLTK tokenization rules we used
when applying the generation strategies.777We used the metanl package to
detokenize: https://github.com/commonsense/metanl.

###  4.3 Results on ARTS

We list the accuracy888For ABSA, accuracy is the standard metric to be
reported Wang et al. (2016); Xue and Li (2018); Tang et al. (2016b). of the
nine models on the

Model | Entire Test |  RevTgt Subset |  RevNon Subset |  AddDiff Subset
---|---|---|---|---
| Ori $\rightarrow$ New (Change) | Ori $\rightarrow$ New (Change) | Ori $\rightarrow$ New (Change) | Ori $\rightarrow$ New (Change)
Laptop Dataset
MemNet | 64.42 $\rightarrow$ 16.93 ($\downarrow$47.49)⋆ | 72.10 $\rightarrow$ 28.33 ($\downarrow$43.77)⋆ | 82.22 $\rightarrow$ 79.26 ($\downarrow$02.96) | 64.42 $\rightarrow$ 56.58 ($\downarrow$07.84)⋆
GatedCNN | 65.67 $\rightarrow$ 10.34 ($\downarrow$55.33)⋆ | 75.11 $\rightarrow$ 24.03 ($\downarrow$51.08)⋆ | 83.70 $\rightarrow$ 78.52 ($\downarrow$05.18) | 65.67 $\rightarrow$ 45.14 ($\downarrow$20.53)⋆
AttLSTM | 67.55 $\rightarrow$ 09.87 ($\downarrow$57.68)⋆ | 72.96 $\rightarrow$ 27.04 ($\downarrow$45.92)⋆ | 85.93 $\rightarrow$ 75.56 ($\downarrow$10.37)⋆ | 67.55 $\rightarrow$ 39.66 ($\downarrow$27.89)⋆
TD-LSTM | 68.03 $\rightarrow$ 22.57 ($\downarrow$45.46)⋆ | 73.39 $\rightarrow$ 29.83 ($\downarrow$43.56)⋆ | 83.70 $\rightarrow$ 77.04 ($\downarrow$06.66) | 68.03 $\rightarrow$ 60.66 ($\downarrow$07.37)⋆
GCN | 72.41 $\rightarrow$ 19.91 ($\downarrow$52.50)⋆ | 78.33 $\rightarrow$ 35.62 ($\downarrow$42.71)⋆ | 88.89 $\rightarrow$ 74.81 ($\downarrow$14.08)⋆ | 72.41 $\rightarrow$ 52.51 ($\downarrow$19.90)⋆
BERT-Sent | 73.04 $\rightarrow$ 17.40 ($\downarrow$55.64)⋆ | 78.76 $\rightarrow$ 59.44 ($\downarrow$19.32)⋆ | 88.15 $\rightarrow$ 42.22 ($\downarrow$45.93)⋆ | 73.04 $\rightarrow$ 34.64 ($\downarrow$38.40)⋆
CapsBERT | 77.12 $\rightarrow$ 25.86999CapsBERT hand-crafted the Capsule Guided Routing specifically for the restaurant domain, so it fails significantly. ($\downarrow$51.26)⋆ | 80.69 $\rightarrow$ 57.73 ($\downarrow$22.96)⋆ | 88.89 $\rightarrow$ 49.63 ($\downarrow$39.26)⋆ | 77.12 $\rightarrow$ 45.14 ($\downarrow$31.98)⋆
BERT | 77.59 $\rightarrow$ 50.94 ($\downarrow$26.65)⋆ | 83.05 $\rightarrow$ 65.02 ($\downarrow$18.03)⋆ | 93.33 $\rightarrow$ 71.85 ($\downarrow$21.48)⋆ | 77.59 $\rightarrow$ 71.00 ($\downarrow$06.59)⋆
BERT-PT | 78.53 $\rightarrow$ 53.29 ($\downarrow$25.24)⋆ | 82.40 $\rightarrow$ 60.09 ($\downarrow$22.31)⋆ | 93.33 $\rightarrow$ 83.70 ($\downarrow$09.63)⋆ | 78.53 $\rightarrow$ 75.71 ($\downarrow$02.82)
Average | 71.60 $\rightarrow$ 25.23 ($\downarrow$46.37)⋆ | 77.42 $\rightarrow$ 43.01 ($\downarrow$34.41)⋆ | 87.57 $\rightarrow$ 70.29 ($\downarrow$17.28)⋆ | 71.60 $\rightarrow$ 53.45 ($\downarrow$18.15)⋆
Restaurant Dataset
MemNet | 75.18 $\rightarrow$ 21.52 ($\downarrow$53.66)⋆ | 80.73 $\rightarrow$ 27.54 ($\downarrow$53.19)⋆ | 84.46 $\rightarrow$ 73.65 ($\downarrow$10.81)⋆ | 75.18 $\rightarrow$ 60.71 ($\downarrow$14.47)⋆
GatedCNN | 76.96 $\rightarrow$ 13.12 ($\downarrow$63.84)⋆ | 85.11 $\rightarrow$ 23.17 ($\downarrow$61.94)⋆ | 88.06 $\rightarrow$ 72.97 ($\downarrow$15.09)⋆ | 76.96 $\rightarrow$ 54.91 ($\downarrow$22.05)⋆
AttLSTM | 75.98 $\rightarrow$ 14.64 ($\downarrow$61.34)⋆ | 82.98 $\rightarrow$ 28.96 ($\downarrow$54.02)⋆ | 86.26 $\rightarrow$ 61.26 ($\downarrow$25.00)⋆ | 75.98 $\rightarrow$ 52.32 ($\downarrow$23.66)⋆
TD-LSTM | 78.12 $\rightarrow$ 30.18 ($\downarrow$47.94)⋆ | 85.34 $\rightarrow$ 34.99 ($\downarrow$50.35)⋆ | 88.51 $\rightarrow$ 75.68 ($\downarrow$12.83)⋆ | 78.12 $\rightarrow$ 70.18 ($\downarrow$07.94)⋆
GCN | 77.86 $\rightarrow$ 24.73 ($\downarrow$53.13)⋆ | 86.76 $\rightarrow$ 35.58 ($\downarrow$51.18)⋆ | 88.51 $\rightarrow$ 79.50 ($\downarrow$09.01)⋆ | 77.86 $\rightarrow$ 65.00 ($\downarrow$12.86)⋆
BERT-Sent | 80.62 $\rightarrow$ 10.89 ($\downarrow$69.73)⋆ | 89.60 $\rightarrow$ 44.80 ($\downarrow$44.80)⋆ | 89.86 $\rightarrow$ 57.21 ($\downarrow$32.65)⋆ | 80.62 $\rightarrow$ 30.89 ($\downarrow$49.73)⋆
CapsBERT | 83.48 $\rightarrow$ 55.36 ($\downarrow$28.12)⋆ | 89.48 $\rightarrow$ 71.87 ($\downarrow$17.61)⋆ | 90.99 $\rightarrow$ 74.55 ($\downarrow$16.44)⋆ | 83.48 $\rightarrow$ 77.86 ($\downarrow$05.62)⋆
BERT | 83.04 $\rightarrow$ 54.82 ($\downarrow$28.22)⋆ | 90.07 $\rightarrow$ 63.00 ($\downarrow$27.07)⋆ | 91.44 $\rightarrow$ 83.33 ($\downarrow$08.11)⋆ | 83.04 $\rightarrow$ 79.20 ($\downarrow$03.84)⋆
BERT-PT | 86.70 $\rightarrow$ 59.29 ($\downarrow$27.41)⋆ | 92.20 $\rightarrow$ 72.81 ($\downarrow$19.39)⋆ | 92.57 $\rightarrow$ 81.76 ($\downarrow$10.81)⋆ | 86.70 $\rightarrow$ 80.27 ($\downarrow$06.43)⋆
Average | 79.77 $\rightarrow$ 31.62 ($\downarrow$48.15)⋆ | 86.92 $\rightarrow$ 44.75 ($\downarrow$42.17)⋆ | 88.96 $\rightarrow$ 73.32 ($\downarrow$15.64)⋆ | 79.77 $\rightarrow$ 63.48 ($\downarrow$16.29)⋆

Table 7: Model accuracy on Laptop and Restaurant data. We compare the accuracy
on the Original and our New test sets (Ori $\rightarrow$ New), and calculate
the change of accuracy. Besides the Entire Test Set, we also list accuracy on
subsets where the generation strategies RevTgt, RevNon and AddDiff can be
applied. The accuracy of Entire Test-New is calculated using ARS. $\star$
indicates whether the performance drop is statistically significant (with
p-value $\leq 0.05$ by Welch’s $t$-test).

Laptop and Restaurant test sets in Table 7. For Entire Test-New in Table 7,
accuracy is calculated using ARS. Supplementary to ARS, Table 7 also
decomposes ARS into single-strategy scores (the right three columns) by
splitting the entire ARTS test set into three subsets according to the
corresponding data generation techniques. Each of the single-strategy scores
explains from one perspective the reason for large performance drop in ARS,
which will be elaborated later.

#### Overall Performance

On the entire test set, we can see that the accuracy of all models on the
original test set is very high, achieving up to 78.53% on Laptop and 86.70% on
Restaurant, but it drops drastically ($\downarrow$69%$\sim\downarrow$25%) on
our new test sets.

#### Performance of Different Models

From the overall performance on our new test set, we can see that BERT models
on average are more robust to the aspect-targeted challenges that our new test
set poses. The most effective model BERT-PT scores the best on both original
accuracy and robustness. It has 53.29% ARS on Laptop and 59.29% on Restaurant.
However, the accuracy of non-BERT models on average drops drastically to under
30% by over $\downarrow$50%.

#### Performance on Different Subsets

We list in detail the performance of each model on the three subsets of our
new test set: RevTgt, RevNon, and AddDiff. They correspond to the three
questions (Q1)-(Q3). RevTgt on average induces the most performance drop, as
it requires the model to pay precise attention to the target sentiment words.
RevNon makes the performance of the sentence classifier BERT-Sent drops the
most by up to $\downarrow$45.93%, and the model CapsBERT also drops by up to
$\downarrow$39.26%. The last subset AddDiff causes most non-BERT models to
drop significantly, indicating that these models are not robust enough against
an increased number of non-target aspects, which should have been irrelevant.

#### Laptop vs. Restaurant

The performance drop on Restaurant is higher than that on Laptop. There are
two possible reasons: (1) the original performance on restaurant is higher,
and (2) the new test set is more challenging in the Restaurant domain. We
verify this by calculating the relative drop
($\frac{\text{new}-\text{old}}{\text{old}}$) in addition to the reported
absolute values of the change. The relative drop on Laptop is 64.76%, which is
higher than 60.36% on Restaurant. For the laptop dataset, both the lower
original performance and the larger relative decrease of performance might be
due to the nature of the dataset. For example, Laptop restaurant has far fewer
training data than Restaurant, which makes the models less accurate originally
and weaker on ARTS.

##  5 Analysis

###  5.1 Variations of Generation Strategies

#### Combining Multiple Strategies

Each sample in the ARTS test set is generated by one of the three strategies.
However, it is also worth exploring whether combining several strategies can
make a more challenging probe on the aspect robustness of ABSA models. As a
case study, we analyze the model robustness against test samples generated by
the combination of RevNon+AddDiff.

Model | Laptop | Restaurant
---|---|---
| Ori $\rightarrow$ New (Change) | Ori $\rightarrow$ New (Change)
MemNet | 82.22 $\rightarrow$ 72.59 ($\downarrow$09.63) | 84.46 $\rightarrow$ 50.90 ($\downarrow$33.56)⋆
GatedCNN | 84.44 $\rightarrow$ 59.26 ($\downarrow$25.18)⋆ | 87.84 $\rightarrow$ 53.83 ($\downarrow$34.01)⋆
AttLSTM | 85.93 $\rightarrow$ 51.85 ($\downarrow$34.08)⋆ | 86.26 $\rightarrow$ 38.06 ($\downarrow$48.20)⋆
TD-LSTM | 83.70 $\rightarrow$ 68.89 ($\downarrow$14.81)⋆ | 88.51 $\rightarrow$ 65.99 ($\downarrow$22.52)⋆
GCN | 88.89 $\rightarrow$ 60.74 ($\downarrow$28.15)⋆ | 88.51 $\rightarrow$ 72.52 ($\downarrow$15.99)⋆
BERT-Sent | 88.15 $\rightarrow$ 11.85 ($\downarrow$76.30)⋆ | 89.86 $\rightarrow$ 11.94 ($\downarrow$77.92)⋆
CapsBERT | 90.37 $\rightarrow$ 24.44 ($\downarrow$65.93)⋆ | 90.99 $\rightarrow$ 66.89 ($\downarrow$24.10)⋆
BERT | 93.33 $\rightarrow$ 68.15 ($\downarrow$25.18)⋆ | 91.44 $\rightarrow$ 76.58 ($\downarrow$14.86)⋆
BERT-PT | 93.33 $\rightarrow$ 78.52 ($\downarrow$14.81)⋆ | 92.57 $\rightarrow$ 78.60 ($\downarrow$13.97)⋆
Average | 87.57 $\rightarrow$ 55.14 ($\downarrow$32.43) ⋆ | 88.96 $\rightarrow$ 57.26 ($\downarrow$31.70)⋆

Table 8: The accuracy of each model on the original test set and the new test
set generated by RevNon+AddDiff in laptop and restaurant domains.

By comparing the performance decrease caused by RevNon+AddDiff in Table 8 and
by only RevNon and AddDiff in Table 7, we can see that the accuracy of each
model decreases by a much larger extent on RevNon+AddDiff than either of
RevNon and AddDiff. The performance drop by the RevNon+AddDiff subset is
almost the sum of RevNon and AddDiff in most cases, and sometimes larger than
the sum, for example, in the case of BERT-PT on Laptop where the sum of the
performance drop by the two separate strategies 9.63%+2.82% is smaller than
the combined strategy’s 14.81%.

####  AddDiff with More Aspects

Some strategies such as AddDiff can be parameterized by $k$, where $k$ is the
number of additional non-target aspects to be added. We select three models
(the best, the worst, and an average-performing one), and plot their accuracy
on test samples generated by AddDiff$(k)$ on Laptop in Figure 1. As $k$ gets
larger, the test samples become more difficult. The sentence classification
baseline BERT-Sent drops drastically, BERT-PT remains high, and GCN lies in
the middle.

$0$$1$$2$$3$$4$$5$$0$$20$$40$$60$$80$Number of Instances Added by AddDiff
($k$)Test Accuracy (%)BERT-PTGCNBERT-Sent

Figure 1: Accuracy of BERT-PT, GCN, and BERT-Sent on the test samples in the
laptop domain generated by AddDiff$(k)$ where $k$ varies from 1 to 5.

###  5.2 How to Effectively Model the Aspect?

An important usage of our ARTS is to understand what model components are key
to aspect robustness. We list the aspect-specific mechanisms of all models
according to the ascending order of their ARS on Laptop dataset in Table 9. We
can see that for BERT-based models, BERT-PT, which is further trained on large
review corpora, gets the best accuracy and aspect robustness. More complicated
structures like CapsBERT underperforms the basic BERT by 25.08%.

Model | ARS | Asp+W Emb | Posi-Aware | Asp Att
---|---|---|---|---
AttLSTM | 9.87 | ✓ | ✗ | ✓
GatedCNN | 10.34 | ✓ | ✗ | ✓
MemNet | 16.93 | ✗ | ✗ | ✓
GCN | 19.91 | ✗ | ✓ | ✓
TD-LSTM | 22.57 | ✗ | ✓ | ✗
CapsBERT | 25.86 | ✗ | ✗ | ✓
BERT | 50.94 | ✗ | ✗ | ✗
BERT-PT | 53.29 | ✗ | ✗ | ✗

Table 9: Models in the ascending order of their ARS on Laptop. We list their
aspect-specific mechanisms, including concatenating the aspect and word
embeddings (Asp+W Emb), position-aware mechanism for aspects (Posi-Aware), and
attention using the aspect (Asp Att). We highlight ✓ for Posi-Aware as it is
the most related to aspect robustness for non-BERT models.

Among the non-BERT models, the aspect position-aware models TD-LSTM and GCN
are the most robust, as they have a stronger sense of the location of the
target aspect in a sentence. On the contrary, the other models with poorer
robustness (9.87%$\sim$16.93% in Table 9) only use mechanisms such as aspect-
based attention, or concatenating the aspect embedding to the word embedding.

To summarize, the main takeaways are

  * •

For BERT models, additional pretraining is the most effective.

  * •

For non-BERT models, explicit position-aware designs lead to more aspect
robustness.

Model | MAMS
---|---
O$\rightarrow$O | O$\rightarrow$N
MemNet | 70.51 | 37.80
GatedCNN | 66.02 | 32.93
AttLSTM | 67.14 | 39.67
TD-LSTM | 77.62 | 49.25
GCN | 76.95 | 47.98
BERT-Sent | 49.25 | 10.48
CapsBERT | 83.38 | 60.18
BERT | 84.51 | 61.38
BERT-PT | 85.10 | 64.37
(a) Accuracy of each model trained on the MAMS Original training data and
evaluated on the Original test data (O$\rightarrow$O), as well as the New test
set generated by our models (O$\rightarrow$N).

Restaurant | Laptop
---|---
O$\rightarrow$O | O$\rightarrow$N | MAMS$\rightarrow$N | Adv$\rightarrow$N | O$\rightarrow$O | O$\rightarrow$N | Adv$\rightarrow$N
75.18 | 21.52 | 24.02 | 37.95 | 64.42 | 16.93 | 31.82
76.96 | 13.13 | 18.48 | 37.50 | 65.67 | 10.34 | 41.85
75.98 | 14.64 | 22.32 | 48.66 | 67.55 | 9.87 | 42.63
78.12 | 30.18 | 41.60 | 62.76 | 68.03 | 22.57 | 54.86
77.86 | 24.73 | 46.51 | 61.52 | 72.41 | 19.91 | 56.43
80.62 | 10.89 | 12.95 | 45.80 | 73.04 | 17.40 | 53.92
83.66 | 55.36 | 61.43 | 75.80 | 76.80 | 25.86 | 61.23
83.04 | 54.82 | 62.77 | 74.82 | 77.59 | 50.94 | 65.67
86.70 | 59.29 | 62.77 | 74.64 | 78.53 | 53.29 | 66.93
(b) Accuracy of each model trained on the Original data and evaluated on the
Original test set (O$\rightarrow$O), and the New test set (O$\rightarrow$N),
as well as that trained on the Adversarial data and evaluated on the New test
set (Adv$\rightarrow$N). For Restaurant, we also test models trained on MAMS
dataset and tested on the New test set of Restaurant (MAMS$\rightarrow$N).

Table 10: Improvements on the new test set using different training data.

###  5.3 Does a More Diverse Training Set Help?

A recent dataset, Multi-Aspect Multi-Sentiment (MAMS) Jiang et al. (2019), is
collected from the same data source as the SemEval 2014 Restaurant dataset
Ganu et al. (2009). However, its sentences are more complicated, each having
at least two aspects with different sentiment polarities.

We use this dataset to inspect whether a stronger training set can help
improve aspect robustness.

#### Training and Testing on MAMS

Table 10(a) checks the aspect robustness of models trained on MAMS using the
original MAMS test set (O$\rightarrow$O) and the new test set that we produced
by applying the same generation strategies to its test set (O$\rightarrow$N).
Models trained and tested on MAMS have a smaller decrease rate than those on
the Restaurant dataset. This shows that a more challenging training set can
make models more robust.

#### Training on MAMS and Testing on Restaurant

As MAMS and Restaurant are collected from the same source data, we test
whether MAMS-trained models perform well on the new test set of Restaurant (in
the column “MAMS$\rightarrow$N” of Table 10(b)). We can see that all models
trained on MAMS are more robust than those trained on the Restaurant dataset.
For example, the accuracy of BERT and BERT-PT on the new test set is lifted up
to 62.77%.

###  5.4 Does Adversarial Training Help?

Although the MAMS described in Section 5.3 provides a training set with
diversity, it remains difficult to improve aspect robustness for other
domains, or future new datasets. Therefore, we propose a flexible method,
adversarial training, for aspect robustness, which is applicable to any given
dataset.

We conducted adversarial training on the Laptop and Restaurant datasets, and
analyze its effect in Table 10(b). Specifically, for the column
“Adv$\rightarrow$N”, we generated an additional training set by applying the
three proposed strategies on training data, then trained models on the
augmented data obtained by combining the original training data and this newly
generated data, and finally evaluated on the ARTS test data. This practice
follows Table 7 of Zhang et al. (2019b) which is a similar stream of work as
ours for the paraphrasing domain.

In both Restaurant and Laptop domains, adversarial training
(Adv$\rightarrow$N) leads to significant performance improvement than only
training on the original datasets (O$\rightarrow$N). On the Restaurant
datasets, adversarial training is even more effective than training on MAMS,
because our generated data instances comprehensively covered all possible
perturbations of the non-target aspects, and naturally collected datasets
might not be comparable.

##  6 Error Analysis for Data Generation

We analyze the error types in the subset of ARTS that was fixed by human
judges. Two most significant error types are wrong antonyms ($\sim$2%), such
as “the weight of the laptop is light$\rightarrow$dark”, and negation which
causes grammatical errors ($\sim$1.1%). In future work, we can fix the latter
by applying a grammatical error correction system on top of our generation.
Also, RevTgt and RevNon cannot be applied to 1.4$\sim$6.6% instances with
complicated sentiment expressions which rely on commonsense. For example, “a
2-hour wait” is negative bust too difficult to alter in our current generation
framework. It needs more advanced models such as text style transfer Shen et
al. (2017); Jin et al. (2019b).

##  7 Related Work

#### Robustness in NLP

Robustness in NLP has attracted extensive attention in recent works Hsieh et
al. (2019); Li et al. (2016). As a popular method to probe the robustness of
models, adversarial text generation becomes an emerging research field in NLP.
Techniques include adding extraneous text to the input Jia and Liang (2016),
character-level noise Belinkov and Bisk (2018); Ebrahimi et al. (2018), and
word replacement Alzantot et al. (2018); Jin et al. (2019a). Using the
adversarial generation techniques, new adversarial test sets are proposed for
several tasks such as paraphrasing Zhang et al. (2019b) and entailment
Glockner et al. (2018); McCoy et al. (2019).

#### Aspect-Based Sentiment Analysis

ABSA has emerged as an active research area recently. Early works hand-craft
sentiment lexicons and syntactic features for rule-based classifiers Vo and
Zhang (2015); Kiritchenko et al. (2014). Recent neural network-based models
use architectures such as LSTM Tang et al. (2016a), CNN Xue and Li (2018),
Attention mechanisms Wang et al. (2016), Capsule Network Jiang et al. (2019),
and the pretrained model BERT Xu et al. (2019a). Similar to the motivation in
our paper, some work shows preliminary speculation that the current ABSA
datasets might be downgraded to sentence-level sentiment classification Xu et
al. (2019b).

##  8 Conclusion

In this paper, we proposed a simple but effective mechanism to generate test
instances to probe the aspect robustness of the models. We enhanced the
original SemEval 2014 test sets by 294% and 315% in laptop and restaurant
domains. Using our new test set, we probed the aspect robustness of nine ABSA
models, and discussed model designs and better training that can improve the
robustness.

## Acknowledgments

We appreciate Professor Rada Mihalcea for her insights that helped us plan
this research, Pengfei Liu for valuable suggestions on writing, and Yuchun Dai
for helping to code some functions in our annotation tool. We also want to
convey special thanks to Mahi Shafiullah and Osmond Wang for brilliant
suggestions on the wording of the title.

## References

  * Alzantot et al. (2018) Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani B. Srivastava, and Kai-Wei Chang. 2018.  Generating natural language adversarial examples.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018_ , pages 2890–2896. Association for Computational Linguistics.
  * Belinkov and Bisk (2018) Yonatan Belinkov and Yonatan Bisk. 2018.  Synthetic and natural noise both break neural machine translation.  In _6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 - May 3, 2018, Conference Track Proceedings_. OpenReview.net.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 4171–4186. Association for Computational Linguistics.
  * Dong et al. (2014) Li Dong, Furu Wei, Chuanqi Tan, Duyu Tang, Ming Zhou, and Ke Xu. 2014.  Adaptive recursive neural network for target-dependent twitter sentiment classification.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 49–54, Baltimore, Maryland. Association for Computational Linguistics.
  * Ebrahimi et al. (2018) Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018.  Hotflip: White-box adversarial examples for text classification.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers_ , pages 31–36. Association for Computational Linguistics.
  * Fan et al. (2019a) Zhifang Fan, Zhen Wu, Xin-Yu Dai, Shujian Huang, and Jiajun Chen. 2019a.  Target-oriented opinion words extraction with target-fused neural sequence labeling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 2509–2518. Association for Computational Linguistics.
  * Fan et al. (2019b) Zhifang Fan, Zhen Wu, Xin-Yu Dai, Shujian Huang, and Jiajun Chen. 2019b.  Target-oriented opinion words extraction with target-fused neural sequence labeling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 2509–2518. Association for Computational Linguistics.
  * Ganu et al. (2009) Gayatree Ganu, Noemie Elhadad, and Amélie Marian. 2009.  Beyond the stars: Improving rating predictions using review text content.  In _12th International Workshop on the Web and Databases, WebDB 2009, Providence, Rhode Island, USA, June 28, 2009_.
  * Gardner et al. (2020) Matt Gardner, Yoav Artzi, Victoria Basmova, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hanna Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020.  Evaluating NLP models via contrast sets.  _CoRR_ , abs/2004.02709.
  * Glockner et al. (2018) Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018.  Breaking NLI systems with sentences that require simple lexical inferences.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers_ , pages 650–655. Association for Computational Linguistics.
  * He and McAuley (2016) Ruining He and Julian J. McAuley. 2016.  Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering.  In _Proceedings of the 25th International Conference on World Wide Web, WWW 2016, Montreal, Canada, April 11 - 15, 2016_ , pages 507–517. ACM.
  * Hsieh et al. (2019) Yu-Lun Hsieh, Minhao Cheng, Da-Cheng Juan, Wei Wei, Wen-Lian Hsu, and Cho-Jui Hsieh. 2019.  On the robustness of self-attentive models.  In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 1520–1529. Association for Computational Linguistics.
  * Jia and Liang (2016) Robin Jia and Percy Liang. 2016.  Data recombination for neural semantic parsing.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 12–22, Berlin, Germany. Association for Computational Linguistics.
  * Jiang et al. (2019) Qingnan Jiang, Lei Chen, Ruifeng Xu, Xiang Ao, and Min Yang. 2019.  A challenge dataset and effective models for aspect-based sentiment analysis.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 6279–6284. Association for Computational Linguistics.
  * Jin et al. (2019a) Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2019a.  Is BERT really robust? natural language attack on text classification and entailment.  _CoRR_ , abs/1907.11932.
  * Jin et al. (2019b) Zhijing Jin, Di Jin, Jonas Mueller, Nicholas Matthews, and Enrico Santus. 2019b.  Imat: Unsupervised text attribute transfer via iterative matching and translation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 3095–3107. Association for Computational Linguistics.
  * Joshi et al. (2018) Vidur Joshi, Matthew E. Peters, and Mark Hopkins. 2018.  Extending a parser to distant domains using a few dozen partially annotated examples.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers_ , pages 1190–1199. Association for Computational Linguistics.
  * Kiritchenko et al. (2014) Svetlana Kiritchenko, Xiaodan Zhu, Colin Cherry, and Saif Mohammad. 2014.  Nrc-canada-2014: Detecting aspects and sentiment in customer reviews.  In _Proceedings of the 8th International Workshop on Semantic Evaluation, SemEval@COLING 2014, Dublin, Ireland, August 23-24, 2014_ , pages 437–442. The Association for Computer Linguistics.
  * Li et al. (2016) Jiwei Li, Will Monroe, and Dan Jurafsky. 2016.  Understanding neural networks through representation erasure.  _CoRR_ , abs/1612.08220.
  * Ma et al. (2017) Dehong Ma, Sujian Li, Xiaodong Zhang, and Houfeng Wang. 2017.  Interactive attention networks for aspect-level sentiment classification.  In _Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI 2017, Melbourne, Australia, August 19-25, 2017_ , pages 4068–4074. ijcai.org.
  * McCoy et al. (2019) Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019.  Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.  In _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pages 3428–3448. Association for Computational Linguistics.
  * Miller (1995) George A. Miller. 1995.  Wordnet: A lexical database for english.  _Commun. ACM_ , 38(11):39–41.
  * Pontiki et al. (2014) Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014.  SemEval-2014 task 4: Aspect based sentiment analysis.  In _Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014)_ , pages 27–35, Dublin, Ireland. Association for Computational Linguistics.
  * Shen et al. (2017) Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi S. Jaakkola. 2017.  Style transfer from non-parallel text by cross-alignment.  In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, 4-9 December 2017, Long Beach, CA, USA_ , pages 6830–6841.
  * Tang et al. (2016a) Duyu Tang, Bing Qin, Xiaocheng Feng, and Ting Liu. 2016a.  Effective lstms for target-dependent sentiment classification.  In _COLING 2016, 26th International Conference on Computational Linguistics, Proceedings of the Conference: Technical Papers, December 11-16, 2016, Osaka, Japan_ , pages 3298–3307. ACL.
  * Tang et al. (2016b) Duyu Tang, Bing Qin, and Ting Liu. 2016b.  Aspect level sentiment classification with deep memory network.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016_ , pages 214–224. The Association for Computational Linguistics.
  * Vo and Zhang (2015) Duy-Tin Vo and Yue Zhang. 2015.  Target-dependent twitter sentiment classification with rich automatic features.  In _Proceedings of the Twenty-Fourth International Joint Conference on Artificial Intelligence, IJCAI 2015, Buenos Aires, Argentina, July 25-31, 2015_ , pages 1347–1353. AAAI Press.
  * Wang et al. (2016) Yequan Wang, Minlie Huang, Xiaoyan Zhu, and Li Zhao. 2016.  Attention-based LSTM for aspect-level sentiment classification.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, EMNLP 2016, Austin, Texas, USA, November 1-4, 2016_ , pages 606–615. The Association for Computational Linguistics.
  * Xu et al. (2019a) Hu Xu, Bing Liu, Lei Shu, and Philip S. Yu. 2019a.  BERT post-training for review reading comprehension and aspect-based sentiment analysis.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 2324–2335. Association for Computational Linguistics.
  * Xu et al. (2019b) Hu Xu, Bing Liu, Lei Shu, and Philip S. Yu. 2019b.  A failure of aspect sentiment classifiers and an adaptive re-weighting solution.  _CoRR_ , abs/1911.01460.
  * Xue and Li (2018) Wei Xue and Tao Li. 2018.  Aspect based sentiment analysis with gated convolutional networks.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers_ , pages 2514–2523. Association for Computational Linguistics.
  * Zhang et al. (2019a) Chen Zhang, Qiuchi Li, and Dawei Song. 2019a.  Aspect-based sentiment classification with aspect-specific graph convolutional networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019_ , pages 4567–4577. Association for Computational Linguistics.
  * Zhang et al. (2019b) Yuan Zhang, Jason Baldridge, and Luheng He. 2019b.  PAWS: paraphrase adversaries from word scrambling.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 1298–1308. Association for Computational Linguistics.

##  Appendix A Appendices

###  A.1 Test Set Generation Details

In our main paper, we mentioned that the size of our enriched ARTS test set is
294% of the original Laptop data size, and 315% of the original Restaurant
test data size. These two ratios should ideally be both 400%, because there
are three generation strategies, plus one original sentence. However, this gap
is because not every original test sentence can qualify for every generation
strategy. The number of instances generated by each strategy is shown in Table
11.

Dataset | Ori Test | RevTgt | RevNon | AddDiff
---|---|---|---|---
Laptop | 638 | 466 | 135 | 638
Restaurant | 1120 | 846 | 444 | 1120
MAMS | 1336 | 789 | 402 | 1336

Table 11: The statistics of dataset.

Although AddDiff can apply to all test cases, the other two strategies, RevTgt
and RevNon, cannot apply to all instances. For RevTgt and RevNon, we need to
flip opinion words, so we can only apply these two strategies on instances
with explicit opinion words. The number of opinion words is the main
bottleneck for RevTgt. Specifically, Fan et al. (2019b) provide opinion words
for 466 instances of Laptop, and 846 instances of Restaurant. Since RevTgt is
applicable to these instances, the number of new test instances in ARTS
generated by RevTgt are the same as the number of opinion word-available
instances.

In addition to the opinion word constraint, RevNon has some further
requirements. We filter out instances which have only one aspect (224 in
Laptop, and 263 in Restaurant), as well as instances where the opinion words
of the target aspect are overlapped with the opinion words of the non-target
aspect (102 in Laptop and 132 in Restaurant). Also we did not consider
instances of which all the sentiment of non-target aspects are neutral (4 in
Laptop, and 7 in Restaurant).


* [39] W. V. Quine.  Concatenation as a basis for arithmetic.  The Journal of Symbolic Logic, 11(4):105–114, 1946.
  * [40] K. U. Schulz.  Makanin’s algorithm for word equations-two improvements and a generalization.  In International Workshop on Word Equations and Related Topics, pages 85–150. Springer, 1990.
  * [41] R.E. Stearns.  A regularity test for pushdown machines.  Information and Control, 11(3):323–340, 1967.
  * [42] Minh-Thai Trinh, Duc-Hiep Chu, and Joxan Jaffar.  Progressive reasoning over recursively-defined strings.  In International Conference on Computer Aided Verification, pages 218–240. Springer, 2016.
  * [43] L. G. Valiant.  Regularity and related problems for deterministic pushdown automata.  Journal of the ACM (JACM), 22(1):1–10, 1975.
  * [44] Detlef Wotschke.  The boolean closures of the deterministic and nondeterministic context-free languages.  In GI Gesellschaft für Informatik e. V., pages 113–121. Springer, 1973.


# Multi-Tone Microwave Frequency Locking to a Noisy Cavity via Real-Time
Feedback

J.P. van Soest<EMAIL_ADDRESS>Kavli Institute of NanoScience, Delft
University of Technology, PO Box 5046, 2600 GA Delft, Netherlands  C.A. Potts
Kavli Institute of NanoScience, Delft University of Technology, PO Box 5046,
2600 GA Delft, Netherlands  S. Peiter  Kavli Institute of NanoScience, Delft
University of Technology, PO Box 5046, 2600 GA Delft, Netherlands  A. Sanz
Mora  Kavli Institute of NanoScience, Delft University of Technology, PO Box
5046, 2600 GA Delft, Netherlands  G.A. Steele<EMAIL_ADDRESS>Kavli
Institute of NanoScience, Delft University of Technology, PO Box 5046, 2600 GA
Delft, Netherlands

###### Abstract

Microwave cavities are commonly used in many experiments, including
optomechanics, magnetic field sensing, magnomechanics and circuit quantum
electrodynamics. Noise, such as variations in magnetic field or mechanical
vibrations, can cause fluctuations of the cavity’s natural frequency, creating
challenges in operating them in experiments. To overcome these challenges, we
demonstrate a dynamic feedback system implemented by the locking of a
microwave drive to the noisy cavity. A homodyne interferometer scheme monitors
the cavity resonance fluctuations due to low-frequency noise, which is
mitigated by frequency-modulating the microwave generator. The feedback has a
bandwidth of $400$ Hz, with a reduction of cavity fluctuations by $85\%$
integrating up to a bandwidth of $2$ kHz. Moreover, the cavity resonance
frequency fluctuations are reduced by $73\%$. This scheme can be scaled to
enable multi-tone experiments locked to the same feedback signal. As a
demonstration, we apply the feedback to an optomechanical experiment and
implement a cavity-locked, multi-tone mechanical measurement. As low-frequency
cavity frequency noise can be a limiting factor in many experiments, the
multi-tone microwave locking technique presented here is expected to be
relevant for a wide range of microwave cavity experiments.

††preprint: APS/123-QED

##  I Introduction

Low-frequency noise is often the limiting factor in mechanical and magnetic
experiments [1, 2, 3, 4, 5, 6, 7, 8, 9], arising, for example, from variations
in magnetic field or vibrations in cryogenic experiments. These sources of
noise can cause fluctuations in a microwave cavity’s resonance. There are two
common ways to counteract this noise: the first is to dampen the noise before
it reaches the device under test. While a mass-on-a-spring low-pass filter can
suppress high-frequency vibrations for mechanical experiments [10, 11], the
lowest frequencies can often introduce considerable noise. In this sense, the
mechanical filter is not optimal: it is effective at blocking high frequencies
but not low frequencies. The second method actively counters noise by applying
dynamic feedback to the system [12, 13]. As a feedback system is limited by
its feedback bandwidth and response time, it acts as a high-pass filter: it
cannot react at high frequencies but is very effective at low frequencies.
Combining these two complementary techniques, therefore, gives an opportunity
for noise reduction over the entire spectrum.
In this work, we present an active feedback system based on the cavity
stabilization method introduced by Pound in Ref. [14]. Here, the inverse of
the Pound-Drever-Hall technique [15, 16, 17, 18] is applied using active
feedback to adjust a microwave generator frequency, locking it to a cavity
whose resonance frequency fluctuates in time. This scheme not only locks a
single microwave signal to the cavity, but also enables the active locking of
multiple tones using the same feedback signal, demonstrated in Section. V.

Figure 1: A microwave cavity with resonance frequency fluctuations. (a) Lumped
element circuit schematic of the device under test. (b) Optical image of the
flip-chip device. The metallized membrane can be seen as a white square in the
phononic shield. (c) Optical micrograph of the RF circuit. The meander
inductor $L$ and capacitor $C_{\rm 0}$ form the microwave cavity. (d) $500$
individual normalized $240$ µs VNA traces show a distribution of the cavity
resonance frequency $\omega_{\rm c}$. Figure 2: The feedback system is used to
reduce the cavity resonance frequency fluctuations. (a) Schematic of the
feedback circuit, consisting of a balanced microwave homodyne interferometer,
of which one path goes to the device under test. The reference arm is path
length matched, downconverting the signal at a mixer. The resulting voltage
$V_{\rm h}$ is led to a Red Pitaya (RP), which creates a control voltage
$V_{\rm c}$ to modulate the frequency of the local oscillator $\omega_{\rm
LO}$. (b) Homodyne response as the control voltage $V_{\rm c}$ sweeps the
local oscillator frequency over the cavity resonance. Within the linewidth of
the cavity, the response is linear. Low-frequency noise can be seen outside
the linear regime. (c) Time trace of the homodyne voltage $V_{\rm h}$. The
feedback is turned on at $t=19$ seconds.

Since any source of low-frequency cavity noise can be mitigated, this
technique has applications in various fields. It can be used to modulate
magnetic flux to stabilize SQUID cavities [13, 19], and magnomechanical
experiments [20, 21, 22, 23], which are also known to suffer from magnetic
field noise. As we demonstrate in Section. VI, our technique can be directly
applied to an optomechanical system experiencing mechanical noise [24]. As
identified in previous work, cavity frequency noise from vibrations can be a
limiting factor in ground state cooling of mechanical resonators [25].

##  II Experimental Setup

To demonstrate the feedback system, we apply it to a radio frequency (RF)
circuit in an optomechanical flip-chip setup, similar to Ref. [26, 27]. The
microwave cavity consists of a meander inductor $L$ and a pair of capacitor
plates $C_{\rm 0}$ etched into a NbTiN film, onto which a mechanically
compliant capacitor $C_{\rm m}$ is positioned$\sim\\!1$ micron above. A
circuit schematic of the device and optical images are shown in Fig. 1(a-c).
The top capacitor plate consists of aluminum deposited on a thin silicon
nitride membrane, embedded within a patterned silicon substrate etched to form
a phononic shield; for example, see Ref. [27, 28]. The primary purpose of the
phononic crystal structure is to reduce unwanted mechanical noise at $1$ MHz
and to suppress the radiative loss of mechanical energy through the substrate,
as shown in Ref. [25]. However, the phononic crystal has the unwanted side
effect of increasing the mechanical susceptibility of the top chip to low-
frequency ($\sim$kHz) vibrations in the setup [29].
The resonance frequency of the flip-chip cavity is given by $\omega_{\rm
c}(t)=[L(C_{\rm 0}+C_{\rm m}(t))]^{-1/2}\approx 2\pi\times 6.28$ GHz.
Vibrations drive low-frequency modes of the top chip, resulting in a time-
dependent capacitance $C_{\rm m}(t)$. Therefore, these low-frequency
mechanical vibrations of the top chip modulate the cavity resonance frequency
$\omega_{\rm c}$. The fluctuations in the cavity resonance frequency can be
seen in Fig. 1(d), where many repeated VNA sweeps demonstrate the noise
imparted on the microwave resonator. These measurements were performed on the
base plate of a dilution refrigerator held at $\sim\\!20$ mK. The pulse tube
cooler is the primary source of vibrations within a dry dilution refrigerator;
therefore, the pulse tube was switched off during measurements to reduce
vibrations within the experimental setup. However, we still observed that the
resonance frequency fluctuations are on the order of the linewidth of the
cavity. To compensate for these fluctuations, we implemented an active
feedback system to track the cavity’s resonance frequency.

##  III Feedback System

Figure 3: Noise reduction of the homodyne voltage $V_{\rm h}$ depending on the
feedback gain. (a) The power spectral density $S_{f_{\rm 0}f_{\rm 0}}$ of the
homodyne voltage $V_{\rm h}$ with the feedback off (dark blue) and on (red).
The unlocked data has been calibrated as described in Appendix B. Noise is
reduced up to $30$ dB over a bandwidth of $400$ Hz. (b) Integrated power
spectral density for unlocked (dark blue) and locked cavity for two PI gains
(light blue and red). Asymptotes up to $2$ kHz (dashed gray) show a noise
reduction of $85\%$. (c) Integrated power spectral density up to $2$ kHz for
different gains of the feedback loop, using the same color coding. The solid
black line is a polynomial fit to guide the eye.

To compensate for the fluctuating cavity frequency, we have developed a
feedback system similar to the Pound-Drever-Hall technique [15, 16, 17, 18].
While the Pound-Drever-Hall technique is usually used to stabilize a noisy
laser by locking it to a stable optical cavity, here we apply its inverse
role: we lock a stable generator to the resonance frequency of a noisy cavity.
In our system, the microwave frequency of the drive tone is locked to the
fluctuating resonance frequency of the cavity. Locking the drive tone is
accomplished by producing an error signal depending on the time-dependent
resonance frequency of the cavity. The error signal is then used in an active
feedback loop to frequency-modulate (FM) the microwave generator’s output,
producing a drive tone that tracks the shifting cavity resonance frequency. A
schematic diagram of the feedback system is shown in Fig. 2(a).
The scheme starts by splitting the local oscillator signal into two paths: one
of which goes directly to the microwave cavity, then it is directed into the
RF port of a mixer. The other path goes through a path length matched
reference arm to the LO port of the same mixer, making the system a balanced
homodyne microwave interferometer. Small fluctuations in the resonance
frequency of the microwave cavity result in a linear response at the output of
the mixer, as seen in Fig. 2(b). This downconverted homodyne signal, which we
call the homodyne voltage $V_{\rm h}$, depends on the detuning of the cavity
resonance frequency relative to the current microwave drive frequency. When
the system is unlocked, the homodyne voltage fluctuations follow the
fluctuations in the cavity’s resonance frequency. In this experiment, the
cavity fluctuations are larger than the linear region of the homodyne
downconversion. Thus, the local oscillator frequency is updated in real-time
by modulation using a control voltage $V_{\rm c}$ to track the cavity
resonance frequency.
Sweeping the local oscillator’s frequency $\omega_{\rm LO}$, a linear
dependency between the control voltage $V_{\rm c}$ and the measured homodyne
voltage $V_{\rm h}$ was observed around the cavity resonance, as seen in Fig.
2(b). Notice that if an unbalanced homodyne detection scheme were used, the
path length difference would result in phase-winding fringes in the homodyne
voltage signal, which could result in a loss of cavity lock. Thus, using a
balanced homodyne scheme ensures a frequency-independent homodyne voltage
signal regardless of the instantaneous local oscillator frequency, see
Appendix A.
To maintain constant power arriving at the LO port of the mixer while having
dynamic control over the power incident on the microwave resonator under test,
a variable attenuator was added to the device arm. Moreover, the phase
difference between the local oscillator and device arm was set to $90^{\circ}$
out of phase, so the quadrature signal was measured. The homodyne voltage is
then low-pass filtered and led to a proportional-integral (PI) controller
programmed in the field-programmable gate array (FPGA) of a Red Pitaya [30].
The output control voltage of the PI controller $V_{\rm c}$ continuously
modulates the local oscillator’s frequency $\omega_{\rm LO}$ to remain
resonant with the cavity frequency $\omega_{\rm c}$. Ideally, the homodyne
voltage is zero at resonance; however, a slight DC offset was present in the
experiment, which determined the setpoint in the PI controller.
To optimize our feedback system, the proportional and integral gain of the PI
controller needed to be tuned for optimal noise suppression. As the gain is
increased, low-frequency noise is suppressed; however, a too-aggressive PI
setting results in added noise at higher frequencies. The optimal value for
all experimental implementations of our feedback system depends on the
specific configuration; therefore, each system must be calibrated to find the
optimal PI gains. Turning on the feedback system stabilizes the homodyne
voltage, as can be seen in Fig. 2(c). It should be noted that the feedback did
not lose the lock during the measurements over a time of approximately $7$
minutes (the period that the pulse tube could be turned off).

##  IV Noise Reduction

Figure 4: Characterization of the locked cavity response measured with the
extended feedback circuit. (a) Schematic of the extended feedback circuit to
lock multiple signals to the same feedback. The gray box corresponds with that
in Fig. 2. The local oscillator is now up- and downconverted with a lock tone.
The components in the orange box are included to add a probe tone to the input
of the device. (b) Normalized VNA traces with the cavity unlocked (blue) and
locked (red). Two separate measurements are performed for the locked cavity.
(c) Histogram of the fitted resonance frequency of $376$ traces unlocked and
locked, using the same color coding. The locked histogram has been shifted for
clarity. The standard deviation of the locked cavity resonance frequency is
$\sigma_{\rm locked}=21$ kHz, which is a factor $4$ smaller than the unlocked
$\sigma_{\rm unlocked}=77$ kHz.

We can quantitatively analyze the suppressed frequency noise in the reference
frame of the cavity by considering the power spectral density of frequency
fluctuations $S_{f_{\rm 0}f_{\rm 0}}(f)$ (Hz2/Hz) [29]. The power spectral
density can be calculated by taking the Fourier transform of the measured
homodyne voltage $V_{\rm h}$ and is shown in Fig. 3(a). The power spectral
density of the locked homodyne voltage $S_{V_{\rm h}V_{\rm h}}(f)$ (V2/Hz) was
calibrated using the slope of the homodyne voltage versus control voltage,
shown in Fig. 2(b). However, the unlocked homodyne signal’s power spectral
density cannot be calibrated this way, since the region of linear response is
narrower than the fluctuations of the cavity resonance frequency. Therefore,
the detected homodyne signal does not contain information of large frequency
fluctuations. An extra calibration has been applied to the unlocked spectra,
as discussed in Appendix B.
We find that when we turn the feedback on, even with a small gain, the large
frequency shifts of the cavity are compensated for, and the cavity
fluctuations remain within the linear regime of the homodyne signal. This
indicates that low-frequency fluctuations dominate the cavity noise. We
identified that for our system the optimal proportional and integral gain
ratio is $\rm P\,\colon I=6\ \colon 10$. Figure 3(a) shows the power spectral
density of the cavity unlocked and locked with the optimal feedback loop gain.
It can be seen that for low frequencies, the noise is suppressed up to $30$
dB. The spectra cross around$400$ Hz, indicating the feedback bandwidth. Above
this frequency, the system adds noise to the measurement, due to feedback of
the added noise by the amplification chain used. At $2$ kHz and above, the two
spectra overlap, suggesting no additional high-frequency noise is being added
by our feedback system. Taking the square root of the integrated power
spectral densities $S_{f_{\rm 0}f_{\rm 0}}(f)$, we obtain the root mean square
(rms) value for the noise power in units of Hz. Even though there is added
noise at intermediate frequencies, in Fig. 3(b) it can be seen that the total
frequency noise is reduced. The integrated spectra show a noise reduction of
$85\%$ at $2$ kHz, as indicated by the asymptotes shown in Fig. 3(b). Again,
this suggests that low-frequency fluctuations were most detrimental in our
system. Since the feedback loop acts as a high-pass filter, these low
frequencies are compensated most efficiently.
Finally, the total gain of the feedback loop was varied by increasing the PI
gain while keeping their ratio the same. Increasing the gain initially results
in additional noise suppression until the feedback increases the total noise
power, as shown in Fig. 3(c). This provides an optimal feedback loop gain for
this system.

##  V Multi-Tone Locking

Thus far, all measurements were done using a single tone on cavity resonance.
However, many experiments require multiple tones or, in fact, frequency
sweeps. Since the local oscillator follows the cavity’s resonance frequency,
the feedback circuit can easily be scaled up by adding additional generators
to the same feedback setup. For example, an extra stage of up- and
downconversion was added to the circuit, as shown in Fig. 4(a). The gray box
corresponds with that shown in Fig. 2(a). The local oscillator signal is now
split into three paths: two paths in parallel are upconverted, to a lock
signal $\omega_{\rm lock}=\omega_{\rm c}\>-\>\omega_{\rm LO}$ and a probe
signal $\omega_{\rm probe}$ generated by a VNA; the third path goes to the
reference arm as before. The signal coming from the cavity is first
downconverted using $\omega_{\rm LO}$, producing two tones at $\omega_{\rm
lock}$ and $\omega_{\rm probe}$. The signal at $\omega_{\rm probe}$ can be
measured by the VNA as long as $\omega_{\rm lock}$ is not within the IF
bandwidth of the measurement. The other path (at $\omega_{\rm lock}$) is
downconverted a second time to DC, resulting, as before, in a voltage
containing the frequencies of the cavity resonance fluctuations. This signal
will also contain fluctuations at the frequency $\delta=\omega_{\rm
probe}-\omega_{\rm lock}$. If $\delta$ is within the feedback bandwidth of the
cavity lock, it will need to be filtered out not to modulate the local
oscillator with this frequency. Again, the downconverted signal is then led to
the Red Pitaya to generate an error signal to frequency modulate the local
oscillator.
Figure 4(b) shows VNA measurements with the feedback off and on, where the
same number of traces were used. Note that these measurements were performed
with an IFBW of $600$ and $1$ kHz, respectively, and with feedback gains $\rm
P=480$, $\rm I=800$. The locked measurement was performed on the left and
right sides of the cavity separately to avoid the lock tone entering the VNA
measurement, as discussed above. However, if a measurement requires a VNA
sweep on cavity resonance, it is also possible to lock off-resonant, see
Appendix A.
The individual traces have been fit using a Lorentzian function to determine
the resonance frequency for each trace. The extracted resonance frequency of
the unlocked and locked traces are compared in Fig. 4(c). The left and right
datasets are fit separately, using a fixed asymmetry due to the Fano effect
[31] extracted from the unlocked measurement. Here ${\it\Delta}$ is the
deviation from the average value of the fit resonance frequency. The fits give
skewed Gaussian distributions with a standard deviation $\sigma_{\rm
unlocked}=77$ kHz and $\sigma_{\rm locked}=21$ kHz, which is a decrease of
$73\%$ for the locked measurement.

##  VI Optomechanical Experimental Validation

Figure 5: Demonstration of a cavity-locked multi-tone optomechanical
measurement. (a) Illustration of the driving scheme. The upconverted lock tone
is set on the cavity resonance. The pump tone is applied $\omega_{\rm m}$ away
from the center of the probe tone of the VNA, which in turn is detuned by
$\delta$ from resonance. (b) The mechanical resonance is visible as an OMIT
peak, centered around ${\it\Delta}_{\rm p}=\omega_{\rm probe}-\omega_{\rm
pump}$, when the local oscillator is locked to the cavity, with $\delta\approx
150$ kHz. Normalized data is shown in orange with a fit in black.

As a demonstration of the applicability of the feedback system, it is used in
an optomechanical experiment [24]. The metal film of the capacitor $C_{\rm m}$
is located on a high-tensile stress silicon nitride membrane, parametrically
coupling the membrane to the cavity [26]. The membrane is isolated from its
surroundings as it is located in a phononic shield [25]. However, implementing
this lattice structure also poses difficulties, as low-frequency modes are
excited by vibrations in the dilution refrigerator. Although the energy of
these lattice vibrations will mostly not be transferred to the vibrational
mode of the membrane, it does mean that the membrane is located on a moving
structure. As these modes alter the distance between the two chips by an
amount larger than the amplitude of the vibrational mode of the membrane, this
noise will change $C_{\rm m}$, and thus also $\omega_{\rm c}$, significantly.
As introduced in Section V, multiple microwave tones can be sent to the device
while locked to the same local oscillator. Simply by multiplexing generators
and mixers after the local oscillator, more drive frequencies can be added.
Repeating the elements inside the orange box Fig. 4(a). offers continual
scaling.
Figure 5(a) shows the scheme for a multi-tone measurement. By applying a drive
tone detuned from the red sideband of the cavity $\omega_{\rm
pump}=\omega_{\rm c}-\,\omega_{\rm m}-\delta$ and probing close to resonance
$\omega_{\rm probe}=\omega_{\rm c}-\delta$ the mechanical resonance appears
from optomechanically induced transparency (OMIT) [32, 33, 34]. While having
the local oscillator locked, a measurement of the mechanical resonance is
presented in Fig. 5(b). Note that the asymmetry of the OMIT peak is due to the
detuning $\delta\approx 150$ kHz of the pump and probe with respect to the
lock tone. However, it is equally possible to instead apply the detuning to
the lock tone by adjusting the setpoint of the feedback circuit, see Appendix
A.

##  VII Conclusions

In conclusion, we have presented a dynamic feedback system locking multiple
microwave tones to an unstable microwave cavity. A local oscillator’s
frequency was modulated to follow the cavity resonance frequency by
calibrating the homodyne signal to a control voltage. The power spectral
density of the homodyne voltage has been analyzed, revealing a noise reduction
of $85\%$ at $2$ kHz. Optimization of the feedback loop and the PI controller
resulted in an optimal response for our system with a feedback bandwidth of
$400$ Hz. Furthermore, we extended the feedback system to enable multi-tone
experiments. Additional microwave tones were implemented by upconversion with
the same locked tone of the local oscillator. Moreover, VNA measurements
probing the cavity showed that fluctuations in the cavity resonance frequency
when locked had a standard deviation of $\sigma_{\rm locked}=21$ kHz, as
opposed to an unlocked measurement with $\sigma_{\rm unlocked}=77$ kHz, which
is an improvement of $73\%$.
Finally, we demonstrated the applicability of the feedback system by
performing an optomechanical experiment. We observed OMIT using another locked
tone from a third generator without losing the cavity lock. Our feedback
system provides continual scaling for adding tones to the same lock by
multiplexing generators and mixers after the local oscillator. Besides
optomechanics [35, 36], the feedback system presented here has applications in
various fields, as any source of low-frequency cavity noise can be reduced.
For example, other potential applications include reducing TLS noise [37],
locking to coplanar waveguide resonators [38], SQUID cavities [19],
gravitational wave detectors [6], axion dark-matter detectors [39], and
magnomechanical experiments [20, 21, 22, 23].

###### Acknowledgements.

This work was supported by the European Union’s Horizon 2020 research and
innovation programme under grant agreements 681476 - QOM3D and the Dutch
Research Council (NWO) through a Vici award from the NWO Talent Programme.
C.A. Potts acknowledges the support of the Natural Sciences and Engineering
Research Council of Canada (NSERC).

##  Appendix A Homodyne Phase Matching

It is worth discussing the effect of the balanced interferometer. As mentioned
in Section III, the path length of the reference arm of the interferometer is
matched to that of the signal arm. In contrast to the most common use of an
interferometer, where the absolute phase difference of the two signals is not
important, here we need to match the phases exactly. Because the noise in the
system fluctuates the resonance frequency of the cavity, our feedback system
continually changes the frequency of the drive tone and, thereby, its
wavelength. Therefore, the signals arriving at the mixer will have a
frequency-dependent phase difference unless the two path lengths are
identical.
Figure 6 shows the calculated homodyne voltage $V_{\rm h}$ for a balanced and
a ${\it\Delta L}=10$ m unbalanced interferometer. In both cases, the incoming
signals are set to be $90^{\circ}$ out of phase to measure the quadrature
signal. It is clearly visible that in the unbalanced case, there is a
sinusoidal profile superimposing the frequency response of the cavity, known
as phase-winding fringes. The linear regime has decreased significantly,
increasing the chance for the lock to fringe hop to the next slope. This would
result in an unstable feedback signal.
Moreover, it is possible to send the lock tone off-resonant from the cavity by
adjusting the setpoint as long as the tone is within the linear regime of the
homodyne response. However, one advantage of the homodyne voltage locking, in
contrast to the conventional Pound-Drever-Hall technique [15, 16, 17, 18], is
that one can also choose to lock at other points in the cavity. Doing so will
sacrifice some stability, which will reduce the quality of the lock, but
choosing the right phase difference at the mixer allows one lock on the other
quadrature, as demonstrated in Figure 7. The dashed lines show examples of
setpoints that allow locking on- and off-resonant, while still remaining on an
unidirectional slope, using the other quadrature. Note that moving the
setpoint further from resonance can decrease the stability of the lock.

Figure 6: Homodyne response for different interferometer path lengths. (a)
Calculations for a balanced interferometer. (b) Calculations for an unbalanced
by ${\it\Delta L}=10$ m interferometer. Figure 7: Calculated homodyne response
for two quadratures, as defined by the phase difference of the two signals
coming into the mixer. Locking on resonance can be achieved using the ”phase”
quadrature (blue) with a tone resonant with the nominal cavity frequency and a
feedback setpoint indicated by the blue circle. Locking off resonance can be
achieved by choosing the orthogonal ”amplitude” quadrature (green) and a
setpoint indicated by the green square.

##  Appendix B Spectral Density Calibration

Figure 8: Calibration of the unlocked power spectral density. (a) Three
measured homodyne power spectral density peaks, produced by an addition
modulation tone applied to the local oscillator, for the unlocked (light blue)
and locked (red) cavity. The calibration tones were provided at different
frequencies inside and outside the feedback bandwidth. (b) The integrated
power of the measured calibration tones $P_{\rm cal}$, using the same color
coding. The dashed lines show the frequencies of the peaks in (a). The
unlocked trace is corrected (dark blue) with an offset of the calibration
factor.

The fluctuations of the unlocked cavity resonance frequency were greater than
the region of linear response of the homodyne voltage, as shown in Fig. 2(b).
As a result, the measured homodyne voltage did contain information about large
frequency fluctuations. To compare the unlocked and locked noise spectra,
separate calibration measurements had to be performed. With the Red Pitaya, an
additional modulation was applied to the local oscillator at different
frequencies for the cavity unlocked and locked. The amplitude of this tone
caused a frequency fluctuation of the drive tone that was well within the
linear regime of the locked homodyne response and resulted in a sharp peak in
the measured power spectral density at the modulation frequency. For the
locked cavity, when the frequency of this tone was within the feedback
bandwidth, the feedback would create a control voltage to compensate for this
artificially added noise. However, for frequencies greater than the feedback
bandwidth, the feedback could not compensate this. Therefore, for the tones
that were well outside the feedback bandwidth, the integrated area of the
calibration tonne is equal for the locked and unlocked spectra. By comparing
the integrated areas of these measured peaks, we found the correction factor
to scale the unlocked power spectral density accurately. Figure 8(a) shows
three added modulation peaks of the measured homodyne voltage, both for the
unlocked and locked cavity: one within, one just outside, and one well outside
the feedback bandwidth. Figure 8(b) shows the total measured power $P_{\rm
cal}$ of the calibration tones applied. It can be seen that the feedback
produces a high-pass filter behavior as expected. For the locked cavity, the
powers of the calibration tones well outside the feedback bandwidth show a
flat relation, indicating that the feedback did not reduce their noise power.
The correction factor was then determined by the difference of the powers at
the highest frequency, and the corrected unlocked trace was shown. Amending
the unlocked data with this calibration enabled us to compare the unlocked and
locked measurements fairly.

## References

  * Den Haan  _et al._ [2014] A. M. Den Haan, G. H. Wijts, F. Galli, O. Usenko, G. J. Van Baarle, D. J. Van Der Zalm, and T. H. Oosterkamp, Atomic resolution scanning tunneling microscopy in a cryogen free dilution refrigerator at 15 mK, Rev. Sci. Instrum. 85, 035112 (2014).
  * Niepce  _et al._ [2021] D. Niepce, J. J. Burnett, M. Kudra, J. H. Cole, and J. Bylander, Stability of superconducting resonators: Motional narrowing and the role of landau-zener driving of two-level defects, Sci. Adv. 7, eabh0462 (2021).
  * Learn  _et al._ [2022] R. T. Learn, E. Varga, V. Vadakkumbatt, and J. P. Davis, Precision measurements of the zero-temperature dielectric constant and density of liquid ${}^{4}\mathrm{He}$, Phys. Rev. B 106, 214509 (2022).
  * Brock  _et al._ [2020] B. L. Brock, M. P. Blencowe, and A. J. Rimberg, Frequency fluctuations in tunable and nonlinear microwave cavities, Phys. Rev. Appl. 14, 054026 (2020).
  * Müller  _et al._ [2015] C. Müller, J. Lisenfeld, A. Shnirman, and S. Poletto, Interacting two-level defects as sources of fluctuating high-frequency noise in superconducting circuits, Phys. Rev. B 92, 035442 (2015).
  * Vadakkumbatt  _et al._ [2021] V. Vadakkumbatt, M. Hirschel, J. Manley, T. J. Clark, S. Singh, and J. P. Davis, Prototype superfluid gravitational wave detector, Phys. Rev. D 104, 082001 (2021).
  * Buters  _et al._ [2017] F. M. Buters, K. Heeck, H. J. Eerkens, M. J. Weaver, F. Luna, S. De Man, and D. Bouwmeester, High-Q nested resonator in an actively stabilized optomechanical cavity, Appl. Phys. Lett. 110, 104104 (2017), 1701.04212 .
  * Regal  _et al._ [2008] C. A. Regal, J. D. Teufel, and K. W. Lehnert, Measuring nanomechanical motion with a microwave cavity interferometer, Nat. Phys. 4, 555 (2008), arXiv:0801.1827 .
  * Clark  _et al._ [2018] T. J. Clark, V. Vadakkumbatt, F. Souris, H. Ramp, and J. P. Davis, Cryogenic microwave filter cavity with a tunability greater than 5 ghz, Rev. Sci. Instrum. 89, 114704 (2018).
  * De Wit  _et al._ [2019] M. De Wit, G. Welker, K. Heeck, F. M. Buters, H. J. Eerkens, G. Koning, H. Van der Mee, D. Bouwmeester, and T. H. Oosterkamp, Vibration isolation with high thermal conductance for a cryogen-free dilution refrigerator, Rev. Sci. Instrum. 90, 015112 (2019).
  * Lee  _et al._ [2018] C. Lee, H. Jo, C. S. Kang, G. B. Kim, I. Kim, Y. H. Kim, H. J. Lee, and J. So, Vibration mitigation for a cryogen-free dilution refrigerator for the amore-pilot experiment, J. Low Temp. Phys. 193, 786 (2018).
  * Kanhirathingal  _et al._ [2022] S. Kanhirathingal, B. Thyagarajan, B. L. Brock, J. Li, E. Jeffrey, M. P. Blencowe, J. Y. Mutus, and A. J. Rimberg, Feedback stabilization of the resonant frequency in a tunable microwave cavity with single-photon occupancy, Phys. Rev. Appl. 18, 064033 (2022).
  * Vepsäläinen  _et al._ [2022] A. Vepsäläinen, R. Winik, A. H. Karamlou, J. Braumüller, A. D. Paolo, Y. Sung, B. Kannan, M. Kjaergaard, D. K. Kim, A. J. Melville, _et al._ , Improving qubit coherence using closed-loop feedback, Nat. Commun 13, 1932 (2022).
  * Pound [1946] R. V. Pound, Electronic frequency stabilization of microwave oscillators, Rev. Sci. Instrum. 17, 490 (1946).
  * Drever  _et al._ [1983] R. W. P. Drever, J. L. Hall, F. V. Kowalski, J. Hough, G. M. Ford, A. J. Munley, and H. Ward, Laser phase and frequency stabilization using an optical resonator, Appl. Phys. B 31, 97 (1983).
  * Bjorklund  _et al._ [1983] G. C. Bjorklund, M. D. Levenson, W. Lenth, and C. Ortiz, Frequency modulation (fm) spectroscopy: theory of lineshapes and signal-to-noise analysis, Appl. Phys. B 32, 145 (1983).
  * Black [2001] E. D. Black, An introduction to Pound–Drever–Hall laser frequency stabilization, Am. J Phys. 69, 79 (2001).
  * Carmon  _et al._ [2005] T. Carmon, T. J. Kippenberg, L. Yang, H. Rokhsari, S. Spillane, and K. J. Vahala, Feedback control of ultra-high-Q microcavities: application to micro-Raman lasers and microparametric oscillators, Opt. Express 13, 3558 (2005).
  * Bothner  _et al._ [2021] D. Bothner, I. C. Rodrigues, and G. A. Steele, Photon-pressure strong coupling between two superconducting circuits, Nat. Phys. 17, 85 (2021).
  * Potts  _et al._ [2021] C. A. Potts, E. Varga, V. A. S. V. Bittencourt, S. Viola-Kusminskiy, and J. P. Davis, Dynamical backaction magnomechanics, Phys. Rev. X 11, 031053 (2021).
  * Potts  _et al._ [2022] C. A. Potts, Y. Huang, V. A. S. V. Bittencourt, S. Viola-Kusminskiy, and J. P. Davis, Dynamical backaction evading magnomechanics, arXiv preprint arXiv:2211.13766 (2022).
  * Zhang  _et al._ [2016] X. Zhang, C.-L. Zou, L. Jiang, and H. X. Tang, Cavity magnomechanics, Sci. Adv. 2, e1501286 (2016).
  * Shen  _et al._ [2022] R.-C. Shen, J. Li, Z.-Y. Fan, Y.-P. Wang, and J. Q. You, Mechanical bistability in kerr-modified cavity magnomechanics, Phys. Rev. Lett. 129, 123601 (2022).
  * Aspelmeyer  _et al._ [2014] M. Aspelmeyer, T. J. Kippenberg, and F. Marquardt, Cavity optomechanics, Rev. Mod. Phys. 86, 1391 (2014).
  * Peiter [2022] S. R. Peiter,  _Delft University of Technology_ , Ph.D. thesis (2022).
  * Noguchi  _et al._ [2016] A. Noguchi, R. Yamazaki, M. Ataka, H. Fujita, Y. Tabuchi, T. Ishikawa, K. Usami, and Y. Nakamura, Ground state cooling of a quantum electromechanical system with a silicon nitride membrane in a 3d loop-gap cavity, New J. Phys. 18, 103036 (2016).
  * Brubaker  _et al._ [2022] B. M. Brubaker, J. M. Kindem, M. D. Urmey, S. Mittal, R. D. Delaney, P. S. Burns, M. R. Vissers, K. W. Lehnert, and C. A. Regal, Optomechanical ground-state cooling in a continuous and efficient electro-optic transducer, Phys. Rev. X 12, 021062 (2022).
  * Delaney  _et al._ [2022] R. D. Delaney, M. D. Urmey, S. Mittal, B. M. Brubaker, J. M. Kindem, P. S. Burns, C. A. Regal, and K. W. Lehnert, Superconducting-qubit readout via low-backaction electro-optic transduction, Nature 606, 489 (2022).
  * Fedorov  _et al._ [2020] S. A. Fedorov, A. Beccari, A. Arabmoheghi, D. J. Wilson, N. J. Engelsen, and T. J. Kippenberg, Thermal intermodulation noise in cavity-based measurements, Optica 7, 1609 (2020).
  * Red [2023] See https://redpitaya.com/ for information. (2023), accessed: 2023-03-09.
  * Rieger  _et al._ [2022] D. Rieger, S. Günzler, M. Spiecker, A. Nambisan, W. Wernsdorfer, and I. M. Pop, Fano interference in microwave resonator measurements, arXiv preprint arXiv:2209.03036 (2022).
  * Weis  _et al._ [2010] S. Weis, R. Rivière, S. Deléglise, E. Gavartin, O. Arcizet, A. Schliesser, and T. J. Kippenberg, Optomechanically induced transparency, Science 330, 1520 (2010).
  * Safavi-Naeini  _et al._ [2011] A. H. Safavi-Naeini, T. M. Alegre, J. Chan, M. Eichenfield, M. Winger, Q. Lin, J. T. Hill, D. E. Chang, and O. Painter, Electromagnetically induced transparency and slow light with optomechanics, Nature 472, 69 (2011).
  * Teufel  _et al._ [2011] J. D. Teufel, D. Li, M. Allman, K. Cicak, A. J. Sirois, J. D. Whittaker, and R. W. Simmonds, Circuit cavity electromechanics in the strong-coupling regime, Nature 471, 204 (2011).
  * Teufel  _et al._ [2016] J. D. Teufel, F. Lecocq, and R. W. Simmonds, Overwhelming Thermomechanical Motion with Microwave Radiation Pressure Shot Noise, Phys. Rev. Lett. 116, 013602 (2016).
  * Suh  _et al._ [2013] J. Suh, A. J. Weinstein, and K. C. Schwab, Optomechanical effects of two-level systems in a back-action evading measurement of micro-mechanical motion, Appl. Phys. Lett. 103, 052604 (2013).
  * Lindström  _et al._ [2011] T. Lindström, J. Burnett, M. Oxborrow, and A. Y. Tzalenchuk, Pound-locking for characterization of superconducting microresonators, Review of Scientific Instruments 82, 10.1063/1.3648134 (2011), arXiv:1106.5396 .
  * Gao  _et al._ [2007] J. Gao, J. Zmuidzinas, B. A. Mazin, H. G. Leduc, and P. K. Day, Noise properties of superconducting coplanar waveguide microwave resonators, Applied Physics Letters 90, 10.1063/1.2711770 (2007).
  * Khatiwada  _et al._ [2021] R. Khatiwada, D. Bowring, A. S. Chou, A. Sonnenschein, W. Wester, D. V. Mitchell, T. Braine, C. Bartram, R. Cervantes, N. Crisosto, _et al._ , Axion dark matter experiment: Detailed design and operations, Rev. Sci. Instrum. 92, 124502 (2021).