# How Vulnerable Are Automatic Fake News Detection Methods to Adversarial
Attacks?

Camille Koenders
Technische Universität Berlin
<EMAIL_ADDRESS>
Johannes Filla
Technische Universität Berlin
<EMAIL_ADDRESS>
Nicolai Schneider
Technische Universität Berlin
<EMAIL_ADDRESS>
Vinicius Woloszyn
Technische Universität Berlin
<EMAIL_ADDRESS>

###### Abstract

As the spread of false information on the internet has increased dramatically
in recent years, more and more attention is being paid to automated fake news
detection. Some fake news detection methods are already quite successful.
Nevertheless, there are still many vulnerabilities in the detection
algorithms. The reason for this is that fake news publishers can structure and
formulate their texts in such a way that a detection algorithm does not expose
this text as fake news. This paper shows that it is possible to automatically
attack state-of-the-art models that have been trained to detect Fake News,
making these vulnerable. For this purpose, corresponding models were first
trained based on a dataset. Then, using Text-Attack, an attempt was made to
manipulate the trained models in such a way that previously correctly
identified fake news was classified as true news. The results show that it is
possible to automatically bypass Fake News detection mechanisms, leading to
implications concerning existing policy initiatives.

##  1 Introduction

The spreading of disinformation throughout the web has become a serious
problem for a democratic society. The dissemination of fake news has become a
profitable business and a common practice among politicians and content
producers. A recent study entitled “Regulating disinformation with artificial
intelligence” Marsden and Meyer (2019), examines the trade-offs involved in
using automated technology to limit the spread of disinformation online. Based
on this study, this paper discusses the social and technical problems of
Automatic Content Moderation (ACM) poses to freedom of expression.

Although AI and Natural Language Generation have evolved tremendously in the
last decade, there are still concerns regarding the potential implications of
automatically using AI to moderate content. One problem is that automatic
moderation of content on social networks will accelerate a race in which AI
will be created to counter-attack AI.
Adversarial machine learning is an technique that attempts to fool models by
exploiting vulnerabilities and compromising the results. For example, by
changing particular words - e.g., from “Barack” (Obama) to “b4r4ck” - it is
possible to mislead classifiers and overpass automatic detection filters.
Recent works Zhou et al. (2019) show the state-of-the-art machine learning
models are vulnerable to these attacks.

This study relies on state-of-the-art techniques to attack and dive deep into
the fake news detection vulnerabilities. The goal is to experiment with
adversarial attacks to discover and compute the vulnerabilities of fake news
classifiers. Therefore, this work aims to answer the following research
question: _How vulnerable is fake news detection to adversarial attacks?_

The remainder of this paper is organized as follows. Section 2 discusses the
background and previous related works. Section 4 describes the design of our
experiments, and Section 5 presents the results and discussion. Section 6
summarizes our conclusions and presents future research directions.

##  2 Related Work

###  2.1 Fake News Detection

Fake news are used to manipulate general opinions of readers about a certain
topic Zhou and Zafarani (2019). Unlike typical “clickbait” articles, which use
misleading and eye-catching headlines, fake news are usually quite long and
wordy, consisting of inaccurate or invented plots Chakraborty et al. (2016).
This gives rise to the assumption of a well-researched and factually correct
article. The reader, thus, does not notice how their personal opinion about a
certain topic is deliberately manipulated.

Fake news detection refers to any kind of identification of such fake news.
Due to the speed at which digital news is produced today, effective, automated
fake news detection requires the use of machine learning tools. Previous
research has mainly focused on fake news in social media and fake news in
online news articles Ghanem et al. (2021). There are various models of the
machine detection of fake news, which are based on different heuristics.

For example, Ghanem et al. Ghanem et al. (2021) discuss the effectiveness of
the “FakeFlow” model, which incorporates both word embedding and affective
information such as emotions, moods or hyperbolic words, based on four
different datasets. The model receives several small text segments as input
instead of an entire article. The result of the study was that this model is
more effective than most state-of-the-art models. It generated similar results
with less resources.

Another study Woloszyn et al. (2021) investigates the possibility of using
artificial intelligence for the automatic generation of Claim-Review. Claim-
Review is the web markup introduced in 2015 that allows search engines to
access fact-checked articles. The basic idea of the so-called “fact check” is
that journalists and fact checkers identify misinformation and prevent it from
spreading. Accordingly, it is important that fact-checked articles are
highlighted and shared by users.

Furthermore, research is currently looking at when and why a news article is
identified as fake news and when it is not Shu et al. (2019). This research on
“explainable fake news detection” aims to improve the detection performance of
the algorithms. For this purpose, both news content and user comments are used
as data input.

###  2.2 Adversarial Attack

Adversarial attacks are part of adversarial machine learning, which has become
increasingly important in the field of applied artificial intelligence in
recent years. In an adversarial attack, the input data of a neural network is
intentionally manipulated to test how resistant it is to deliver the same
outputs. These manipulated input data is called “adversarial examples”.
Zhixuan Zhou and Hsu (2021) Such a neural network is described as a “fake news
detector” in the context of this paper. The reason for research in the field
of adversarial attacks is that more and more attempts are being made to
outsmart fake news detectors on the internet. This can be done, for example,
by changing the spelling of a word so that the word remains easily
interpretable for humans but not for an algorithm (from “Barack” to “B4r4ck”).
For this reason, the input data is always manipulated in such a way that a
human hardly notices any differences from the non-manipulated input data. In
the area of object recognition, for example, only individual pixels are
slightly changed, which the human eye can hardly perceive.

Examples of adversarial examples in the field of fake news detection are:

  * •

Fact distortion: Here, some words are changed or exaggerated. It can be about
people, time or places.

  * •

Subject-object-exchange: By exchanging subject and object in a sentence, the
reader is confused as to who is the executor and who is the recipient.

  * •

Cause confounding: Here, either false causal connections are made between
events or certain passages of a text are omitted.

Up to now, vulnerabilities to adversarial attacks have been identified in all
application areas of neural networks. Especially in the context of
increasingly safety-critical tasks for neural networks (e.g., autonomous
driving), methods for detecting false or distorted input data are becoming
more and more relevant. Morris et al. (2020).

###  2.3 Regulating disinformation with artificial intelligence

According to the study “Regulating disinformation with artificial
intelligence” Marsden and Meyer (2019), disinformation is defined as ’false,
inaccurate, or misleading information designed, presented and promoted to
intentionally cause public harm or for profit. This definition is based on the
definition of “Final report of the High Level Expert Group on Fake News and
Online Disinformation” Comission (2018), which additionally specifies that the
term “disinformation” does not include fundamentally illegal content such as
hate speech or incitement to violence. Nor does it include misinformation that
is clearly not misleading, such as satire or parody. As a further
delimitation, the paper defines the term “misinformation”, which refers to any
misinformation that is unintentionally or accidentally false or inaccurate.

Marsden and Meyer explain the causes of disinformation in the online context
and the responses that have been formulated from a technological perspective.
Furthermore they analyze the impact of AI-disinformation initiatives on
freedom of expression, media pluralism and democracy.

The issue of disinformation is a very long-term historical problem with human
society. It has just got a global effect through automation, through the
internet and new technologies. Even in the past, people faced the challenges
of filtering out false information or illegal content from newly emerging
media such as newspapers, radio or television. So, on the one hand there is
currently a desire within the European Commission to take action against
illegal and unwanted content through online intermediaries. On the other hand,
the intervention and regulation of content on the internet is also seen
critically, because this is against the basic concept of the internet with its
freedom of expression. However, to stop the global spread of disinformation, a
restriction of freedom of expression is necessary. Accordingly, for measures
to filter fake news, there are three principles that must exist when
restricting freedom of expression Marsden and Meyer (2019):

  * •

Measures must be established by law

  * •

Measures must be legitimate and shown to be necessary

  * •

Measures must be the least restrictive method of pursuing the goal

There are two different ways to detect and remove disinformation: The human
and the technical moderation by AI. Over time, AI solutions have become
increasingly effective in detecting and removing illegal or unwanted content,
but they also raise the question of who is the judge in deciding what is legal
or illegal and what is wanted or unwanted in society. The problem here is that
neither law nor technology can be truly neutral. Both reflect the values and
priorities of those who designed them. This principal is called “garbage in –
garbage out”. According to expert opinions, AI can help to make an initial
filtering, especially for texts and articles, which is then checked by humans.
If AI is wrong, there is always the possibility to reverse the decision. To do
this, companies also hire cheap subcontractors in remote countries to remove
content.

Policy initiatives in the past have focused on making internet intermediaries
more responsible for reducing disinformation on their platforms. While actual
content creators were responsible for their content in the past, now the
platforms have to take more and more responsibility for the content of the
individual actors. The reactions of intermediary sites such as Facebook and
YouTube are technological initiatives that identify certain content and remove
it in different ways or do not publish it at all. The three most popular
initiatives are filtering, blocking, deprioritisation:

Filtering is the most effective method. There are 2 different types of
filtering: ex-ante and ex-post. This means filtering before a content goes
online and filtering after it has already been published. With the exception
of obviously illegal content (e.g. child abuse images or terrorist content) or
disruptive content (e.g. spam, viruses), the ex post-removal is preferable.
The reason for this is that there is better legal justification for this. One
example of filtering is YouTube Content ID. With YouTube Content ID, uploaded
files are matched against databases of works provided by copyright holders. If
a match is found, copyright owners can decide whether to block, monetise or
track a video containing their work.

Blocking is probably the most widespread method. It is used by users, email
providers, search engines, social media platforms and network and internet
access providers. Similar to filtering, blocking can take place ex-ante or ex-
post, i.e. after knowledge, request or order. Blocking means that the content
is not completely removed but blocks certain users from accessing the content.
For this reason, it provides one significant advantage: Content can be blocked
depending on the provider’s terms of use or the laws of a particular region.
For example, content may be blocked in certain places but may be available in
others, e.g. if some countries allow certain content by law while others do
not.

The last option is deprioritisation. In the context of disinformation,
deprioritisation means that content is de-emphasized in users’ feeds. This
takes place when correct content from certain providers is displayed side by
side with incorrect content. Then the wrong information is identified and
deprioritised so that it is displayed further down and not so prominently
anymore.

##  3 Data Set

The data used in this paper was collected by a project called Untrue.News: A
search engine designed to find fake stories on the internet Woloszyn et al.
(2020). It uses an open-source web crawler for searching fake news. This web
crawler is connected to a natural language processing pipeline that uses
automatic and semi-automatic strategies for data enrichment. In this way
approximately 30.000 documents have been retrieved, dating back to the year
1995 (English documents). For our research we used 25.886 English sentences of
these documents. More languages, that were not used for our project can be
found at untrue.news. Woloszyn et al. (2020) use four categories to classify
their documents:

  * •

TRUE: completely accurate statements

  * •

FALSE: completely false statement

  * •

MIXED: partially accurate statements with some elements of falsity

  * •

OTHER: special articles that do not provide a clear verdict or do not match
any other categories

These can be found in Tchechmedjiev et al. (2019) and are less than those
found in the schema.org/ClaimReview markup. For performing the text attacks,
we only used the TRUE and FALSE statements. In Table 2 classification results
can be found for the models trained with all four categories.

##  4 Experiment Design

To understand the vulnerability of models trained to detect fake news we split
our experiment into two steps. First training state-of-the-art machine
learning models using the dataset described in Section 3 and second applying
adverserial attacks on the dataset using TextAttack to manipulate the trained
models to classify fake news as TRUE news.

###  4.1 Fake News Detection as a Classification Problem

Two types of classifiers were used. Bin a binary classifier, in which a
positive class represents true news, and the negative not being true news and
Mult a multiclass classifier, where each document is classified as either
being true news, fake news, untrue news or other.

###  4.2 Pre-trained Models

In this subsection, we present the applied pre-trained models and BERT Devlin
et al. (2019) which two of the models are based on. In this context a token
describes a single word of a given text and a segments describes a sequence of
tokens. The trained models will later be attacked by the Text Attack recipes
described in Section 4.4.

####  4.2.1 BERT

The BERT (Bidirectional Encoder Representations from Transformers) model was
originally trained on two datasets: BookCoorpus Zhu et al. (2015) and English
Wikipedia.
The way BERT analyzes the text is by taking concatenations of two segments.
The total length of the concatenation is bound by a parameter $T$ and is
computed as one input with particular tokens in between describing the
beginning and end of the sentence, as well as the separation point of the two
segments.
During pretraining, BERT applies a Masked Language Model (MLM) and a Next
Sentence Prediction (NSP). For MLM BERT selects 12% of input tokens and
replaces them with a [MASK] token and another 1.5% with a random vocabulary
token. It then proceeds with predicting the selected [MASK] tokens. With NSP
BERT makes a binary prediction on whether two segments in a text are adjacent.

####  4.2.2 RoBERTa

The pre-trained RoBERTa (Robustly optimized BERT approach) Liu et al. (2019)
is an optimized version of the original BERT model. The authors use a total
length of concatenation $T=512$ (-longer than previously used) and expand the
used datasets to: BookCoorpus, CC-News Nagel (2016), OpenWebText Gokaslan and
Cohen (2019) and Stories Trinh and Le (2019). Furthermore, RoBERTa trains
using a bigger batch size, removes NSP and changes the the masking for MLM
each time a new input is given to the model therefore avoiding constant
masking during different training epochs.

####  4.2.3 BERTweet

BERTweet Nguyen et al. (2020) is a pre-trained model that uses elements from
both BERT and RoBERTa. For instance, it’s architecture is taken from BERT,
while the pre-training procedure is ”copied” from RoBERTa. The main difference
is the used dataset: the only used dataset consists of 850 English tweets
together accumulating 80GB of memory.

####  4.2.4 Flair Embeddings

The main difference of Flair Embeddings Akbik et al. (2018) to other models is
their capturing of words which considers token as a sequence of characters.
Furthermore, Flair Embeddings, also called contextual string embeddings, take
contextual words into consideration, i.e. words that would appear consequently
or previously. A word would therefore be embedded depending on the sequence of
words. For our training we use the embeddings news-forward and news-backward,
which were both trained on a 1 billion word corpus.

###  4.3 Parameterization

The training of each model runs for $E=15$ Epochs each, since at the small
amount of training samples all models start converging after approximately 7-8
epochs. The mini batch size is set to $b=32$. For BERTweet and RoBERTa we
specify the learning rate to $lr=3-e5$, while for Flair Embeddings to
$lr=0.1$. Finally, for Flair Embeddings we also set the anneal factor
(describing the factor by which the learning rate is annealed) to $a_{f}=0.5$
and the patience (the number of epochs without improvement until the learning
rate would be annealed) to $p=5$. $tp$ is the number of positive instances
correctly classified as positive, $tn$ number of negative instances correctly
classified as negative, $fp$ negative instances wrongly classified as
positive, and $fn$ is the number of positive instances wrongly classified as
negative. We defined positive instances as fake news websites and negative
instances as reliable news websites.

###  4.4 Text Attack Recipes

The following attack recipes from TextAttack Morris et al. (2020) were applied
on the dataset (described in Section 3) to manipulate the three introduced
models to classify FALSE statements as TRUE, therefore misinterpreting fake
news as true news.

  * •

DeepWordBug: Generates small text perturbations in a black-box setting. It
uses different types of character swaps (swapping, substituting, deleting and
inserting) with greedy replace-1 scoring Gao et al. (2018).

  * •

Pruthi: Simulates common typos with a concentration on QWERTY keyboard. Uses
character insertion, deletion and swapping Pruthi et al. (2019).

  * •

TextBugger: These attacks were optimized to perform with real world
applications. They use space insertions, character deletion and swapping.
Additionally they substitute characters with similar looking letters (ex. o
with 0) and replace words with top nearest neighbor in context-aware word
vector space Li et al. (2019).

  * •

PSOZang: Word level attack using a sememe-based word substitution strategy as
well as particle swarm optimization Zang et al. (2020).

  * •

PWWSRen2019: These attacks focus on maintaining lexical correctness,
grammatical correctness as well as semantic similarity by using synonym swap.
Words for swap are prioritized on a combination of there saliency score and
maximum word-swap effectiveness Ren et al. (2019).

  * •

TextFoolerJin2019: Word swap with their 50 closest embedding nearest
neighbors. Optimized on BERT Jin et al. (2020).

  * •

IGAWang: Implemented as an adversarial attack defense method. Uses counter-
fitted word embedding swap Wang et al. (2021).

  * •

BAEGarg2019: Uses a BERT masked language model transformation. It uses the
language model for token replacement to best fit overall context Garg and
Ramakrishnan (2020).

  * •

CheckList2020: Inspired by the principles of behavioral testing. Uses changes
in names, numbers and locations as well as contraction and extension Ribeiro
et al. (2020).

  * •

InputReductionFeng: This attack concentrates on the least important words in a
sentence. It iteratively removes the world with the lowest importance value
until the model changes its prediction. The importance is measured by looking
at the change in confidence of the original prediction when removing the word
from the original sentence Feng et al. (2018)

With exception of InputReductionFeng each recipe has three possible results
for the attack on each sentence. Success means the text attack resulted in a
wrong classification. Skipped mean that the model classified the sentence
wrongly to begin with, therefore the sentence doesn‘t need to be manipulated.
Fail means the model still classified the sentence correctly.
InputReductionFeng uses Maximized to indicate that the model uncertainty was
maximized. A rubbish example is classified as correct with a higher accuracy
than the original valid input. Skipped is used when the model classified the
sentence wrongly to begin with, therefore the sentence doesn‘t need to be
manipulated.

##  5 Results and Discussion

The results discussed in this section can also be found in our Github
repository Schneider (2021) including the respective implementations.

###  5.1 Training the Models

The first step of our experiment is to train state-of-the-art machine learning
models to detect fake news. Since our dataset includes classes beyond TRUE and
FALSE, we trained RoBERTa, BERTweet and Flair Embeddings both as binary and
multi class classifier.

####  5.1.1 Binary Classification

For the binary classification RoBERTa, BERTweet and FlairEmbeddings were only
trained on TRUE and FALSE statements. Their precision, recall and F1-scores,
depicted in Table 1, demonstrate that the BERT based models performed best
with scores of $>80\%$. Overall, BERTweet had the best results, however, the
difference to RoBERTa is minimal even though the models were pre-trained on
entirely different datasets. With an F1-score of 70% FlairEmbeddings received
the worst score which could be traced back to the small size of the dataset.

Model | P | R | F1
---|---|---|---
RoBERTa | 0,827 | 0,8264 | 0,8260
BERTweet | 0,8382 | 0,8376 | 0,8373
FlairEmbeddings | 0.7066 | 0.7064 | 0.7061
Table 1: These models were trained only with FALSE and TRUE statements. The
scores indicate the Precision, the Recall and the F1-Score

####  5.1.2 Multiclass Classification

We also trained all three models on the four existing classes. Due to the
higher classification complexity all validation scores (shown in Table 2)
resulted about 10-20% worse than for the binary classification. It is worth to
mention, that RoBERTa performed slightly better than BERTweet, contrary to the
less complex classification in Section 5.1.1. FlairEmbeddings’ scores were
affected the most and dropped to 50%.

Model | P | R | F1
---|---|---|---
RoBERTa | 0,7186 | 0,7141 | 0,7154
BERTweet | 0,7149 | 0,7140 | 0,7119
FlairEmbeddings | 0,5298 | 0,5055 | 0,5026
Table 2: These models were trained only with all existing classes. The scores
indicate the Precision, the Recall and the F1-Score

###  5.2 Adversarial Attacks

The next step is to apply adverserial attacks on the dataset using TextAttack.
Approximately 40 FALSE statements were sampled from the dataset and attacked
using the TextAttack recipes. Due to the poor multiclass classification
performance we applied all attacks on the binary-trained models. The results
can be found in Table 3 for BERTweet, in Table 4 for RoBERTa and in Table 5
for Flair. The tables contain the percentage of sentences predicted as
Successful, Failed or Skipped. Table 6 shows the percentages obtained for
InputReductionFeng for all three models and the mean value of score
improvement over all sentences for each model. To see the distribution of word
level and character level attacks, Table 7 was generated. It contains the mean
value of the Success percentage that can be found in tables 3, 4 and 5.

Recipe | Success | Fail | Skipped
---|---|---|---
| (%) | (%) | (%)
IGAWang | 90 | 0 | 10
TextFoolerJin2019 | 85 | 0 | 15
DeepWordBug | 77.5 | 7.5 | 15
PWWSRen2019 | 77.5 | 7.5 | 15
PSOZang | 67.5 | 17.5 | 15
BAEGarg2019 | 67.5 | 30 | 2.5
TextBugger | 57 | 28 | 15
Pruthi | 28 | 57 | 15
CheckList2020 | 20 | 75 | 5
Table 3: Attack results for BERTweet.  Recipe | Success | Fail | Skipped
---|---|---|---
| (%) | (%) | (%)
TextFoolerJin2019 | 92.5 | 5 | 2.5
DeepWordBug | 87.5 | 10 | 2.5
PWWSRen2019 | 77.5 | 20 | 2.5
TextBugger | 75 | 22.5 | 2.5
IGAWang | 70 | 10 | 20
PSOZang | 70 | 20 | 10
BAEGarg2019 | 60 | 37.5 | 2.5
Pruthi | 37.5 | 60 | 2.5
CheckList2020 | 7.5 | 90 | 2.5
Table 4: Attack results for Roberta.

BERTweet and RoBERTa both use document embeddings. They show similar results.
For BERTweet the word level attack IGAWang had the highest success rate with
90%. CheckList2020 was the least successful with 20%. For RoBERTa the word
level attack TextFoolerJin2019 was the most successful with 92.5% and
Checklist2020 the least successful with 7.5%. With the exception of IGAWang
the recipes show the same order in success rates. The character level attack
DeepWordBug seems to be very successful for these two model, ranking place 2
for RoBERTa and 3 for BERTweet. This is surprising since in the overall
ranking in Table 7 it only achieved place 7. This might be an indication that
document embeddings are more vulnerable to character level attacks than word
embeddings.

Recipe | Success | Fail | Skipped
---|---|---|---
| (%) | (%) | (%)
TextFoolerJin2019 | 66 | 6 | 28
BAEGarg2019 | 60 | 18 | 22
IGAWang | 50 | 20 | 30
PWWSRen2019 | 50 | 22 | 28
PSOZang | 46 | 26 | 28
TextBugger | 37 | 26 | 37
DeepWordBug | 24 | 48 | 28
Pruthi | 18 | 54 | 28
CheckList2020 | 2 | 76 | 22
Table 5: Attack results for Flair Embeddings.

In comparison to BERT-based models Flair classified a lot more inputs wrongly,
which is depicted in the higher percentage of skipped statements in Table 5.
Nevertheless, it proved to be a lot less vulnerable towards adverserial
attacks considering that the best performing recipe TextFoolerJin2019 only
reached a success rate of 66% (vs 90% and 92.5%). Similar to the previous
models, CheckList2020 performed worse, but this time with a success rate of
only 2%. Both pure character level attack recipes failed most of their attacks
with 48% (DeepWordBug) and 54% (Pruthi) fail rate. Overall, it appears that
the contextualization used by Flair makes the model a lot robuster towards the
used word and character level based TextAttack recipes.

Model | Maximized | Skipped | Score
---|---|---|---
| (%) | (%) | ↑$\phi$
BERTweet | 83 | 17 | +0.65
Roberta | 81 | 19 | +0.63
Flair | 75 | 25 | +0.64
Table 6: Attack results for InputReductionFeng. The result outputs of this
attack are described in Section 4.4.

BERTweet seems to be the most vulnerable to InputReductionFeng attacks but the
difference in score increasing is not very high (+-0.01). Overall it seems
that these attacks show similar results over all models.

Recipe | Success | Level
---|---|---
| (%) |
TextFoolerJin2019 | 81.17 | word
IGAWang | 70 | word
PWWSRen2019 | 77.5 | word
PSOZang | 68.33 | word
BAEGarg2019 | 67.5 | word & character
DeepWordBug | 63 | character
TextBugger | 62.5 | word & character
Pruthi | 27.83 | character
CheckList2020 | 9.83 | word
Table 7: Mean Success Rates (all models).

Table 8 shows that the 4 top ranking attacks are word level attacks. It seems
that the models are more vulnerable to word level attacks than to character
level or mixed attacks (character & word).

Finally the Success Rate, for every model was calculated using formula 1:

$S_{r}={\sum_{i=1}^{a}{s_{i}\over s_{i}+f_{i}}\over a}$ (1)

with $S_{r}$ being the success rate $s$ being the number of successful
attacks, $f$ being the number of failed attacks and $a$ being number of
attacks recipes.

Model | $S_{r}$
---|---
| (%)
BERTweet | 72,45
Roberta | 68,22
Flair | 54,77
Total Average |  65,15%
Table 8: Success Rates.

The skipped statements were not taken into the calculation, as they are
dependent on the model training and not on TextAttack. The skipped values show
which statements were not predicted correctly by the model in the first place.
Thus, they are depended on the model and reflect models accuracy. For the
success rates we wanted to see TextAttack’s efficiency on statements that the
model would correctly classify as fake news. Our results for the Success Rates
underline the results seen in tables 3, 4 and 5, showing that Flair is less
vulnerable to the attacks with Sr=54,77% than the other two models with Sr=
$\sim$70%. As an conclusion this gives us an Total Success Rate of 65,15% for
adversarial attacks on fake news detection using TextAttack.

##  6 Conclusion

This paper aimed to answers the question: how vulnerable are automatic fake
news detection to adversarial attacks? We tested this by checking if automated
augmentation of fake news sentences (FALSE statements) will lead to TRUE
classifications. This would allow them to bypass the fake news detection
mechanisms. Our results show that using the python library TextAttack allows
automated changing of classification for 65,15% of the sentences. Flair, the
only model using word-level embedding (contextual string embeddings), seems
less vulnerable to attacks with a Success Rate of 54,77%. The other two models
using document embeddings show 72,45% (BERTweet) and 68,22% (Roberta).
Furthermore, word-level swaps seem to be slightly more successful with an
average of 76,87% compared to character level or mixed swaps with average
55,21%. Consequently, the models are more vulnerable to attacks using
semantically correct sentences with changed meaning than to attacks using
typos. Overall it seems that it is possible to bypass the classifier with
these attacks. However, our results do not consider that a human will be able
to see obvious spelling mistakes in sentences. Furthermore, a human will have
a higher accuracy of recognizing unfitting words in sentences. Looking at the
augmented sentences we think that many of them will be recognized as FALSE
(TrOmp hsut down American airportA on 4 Jul 201B or Hollywood Action Star
Christelle Chan Dead).
As a conclusion of these results we think that using the policy initiatives
blocking and deprioritization, should be avoided if possible, as these methods
don’t completely remove the fake news statements from the users feeds. This
makes it easier to target these statements with automated attacks as shown in
our research. A scenario would be to attack these sentences until they are
unblocked or re-prioritized in a users feed. This would lead to dangerous
spreading of Fake News in social networks.

## Acknowledgments

We would like to thank Dr. Vinicius Woloszyn for his support and supervision
of the project.

## References

  * Akbik et al. (2018) Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018.  Contextual string embeddings for sequence labeling.  In _COLING 2018, 27th International Conference on Computational Linguistics_ , pages 1638–1649.
  * Chakraborty et al. (2016) Abhijnan Chakraborty, Bhargavi Paranjape, Sourya Kakarla, and Niloy Ganguly. 2016\.  Stop clickbait: Detecting and preventing clickbaits in online news media.
  * Comission (2018) European Comission. 2018.  _Final report of the High Level Expert Group on Fake News and Online Disinformation_.  Publications Office of the European Union.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.
  * Feng et al. (2018) Shi Feng, Eric Wallace, Alvin Grissom II au2, Mohit Iyyer, Pedro Rodriguez, and Jordan Boyd-Graber. 2018.  Pathologies of neural models make interpretations difficult.
  * Gao et al. (2018) Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. 2018.  Black-box generation of adversarial text sequences to evade deep learning classifiers.
  * Garg and Ramakrishnan (2020) Siddhant Garg and Goutham Ramakrishnan. 2020.  Bae: Bert-based adversarial examples for text classification.
  * Ghanem et al. (2021) Bilal Ghanem, Simone Paolo Ponzetto, Paolo Rosso, and Francisco Rangel. 2021.  Fakeflow: Fake news detection by modeling the flow of affective information.  _arXiv preprint arXiv:2101.09810_.
  * Gokaslan and Cohen (2019) Aaron Gokaslan and Vanya Cohen. 2019.  Openwebtext corpus.  http://Skylion007.github.io/OpenWebTextCorpus.
  * Jin et al. (2020) Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020.  Is bert really robust? a strong baseline for natural language attack on text classification and entailment.
  * Li et al. (2019) Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. 2019.  Textbugger: Generating adversarial text against real-world applications.  _Proceedings 2019 Network and Distributed System Security Symposium_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.
  * Marsden and Meyer (2019) Chris Marsden and Trisha Meyer. 2019.  _Regulating disinformation with artificial intelligence: effects of disinformation initiatives on freedom of expression and media pluralism_.  European Parliament.
  * Morris et al. (2020) John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020\.  Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 119–126.
  * Nagel (2016) Sebastian Nagel. 2016.  Cc-news.  https://commoncrawl.org/2016/10/news-dataset-available/.
  * Nguyen et al. (2020) Dat Quoc Nguyen, Thanh Vu, and Anh Tuan Nguyen. 2020.  BERTweet: A pre-trained language model for English Tweets.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_.
  * Pruthi et al. (2019) Danish Pruthi, Bhuwan Dhingra, and Zachary C. Lipton. 2019.  Combating adversarial misspellings with robust word recognition.
  * Ren et al. (2019) Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019.  Generating natural language adversarial examples through probability weighted word saliency.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1085–1097, Florence, Italy. Association for Computational Linguistics.
  * Ribeiro et al. (2020) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020.  Beyond accuracy: Behavioral testing of nlp models with checklist.
  * Schneider (2021) Filla Schneider, Koenders. 2021.  Code.  https://github.com/nicolaischneider/FakeNewsDetectionVulnerability.
  * Shu et al. (2019) Kai Shu, Limeng Cui, Suhang Wang, Dongwon Lee, and Huan Liu. 2019.  defend: Explainable fake news detection.  In _Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining_, pages 395–405.
  * Tchechmedjiev et al. (2019) Andon Tchechmedjiev, Pavlos Fafalios, Katarina Boland, Malo Gasquet, Matthäus Zloch, Benjamin Zapilko, Stefan Dietze, and Konstantin Todorov. 2019\.  Claimskg: a knowledge graph of fact-checked claims.  In _International Semantic Web Conference_ , pages 309–324. Springer.
  * Trinh and Le (2019) Trieu H. Trinh and Quoc V. Le. 2019.  A simple method for commonsense reasoning.
  * Wang et al. (2021) Xiaosen Wang, Hao Jin, Yichen Yang, and Kun He. 2021.  Natural language adversarial defense through synonym encoding.
  * Woloszyn et al. (2021) Vinicius Woloszyn, Eduardo G Cortes, Rafael Amantea, Vera Schmitt, Dante AC Barone, and Sebastian Möller. 2021.  Towards a novel benchmark for automatic generation of claimreview markup.  In _13th ACM Web Science Conference 2021_ , pages 29–35.
  * Woloszyn et al. (2020) Vinicius Woloszyn, Felipe Schaeffer, Beliza Boniatti, Eduardo Cortes, Salar Mohtaj, and Sebastian Möller. 2020.  Untrue.news: A new search engine for fake stories.  _arXiv preprint arXiv:2002.06585_.
  * Zang et al. (2020) Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. 2020.  Word-level textual adversarial attacking as combinatorial optimization.  _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Zhixuan Zhou and Hsu (2021) Meghana Moorthy Bhat Zhixuan Zhou, Huankang Guan and Justin Hsu. 2021.  Fake news detection via nlp is vulnerable to adversarial attacks.  _arXiv preprint arXiv:2101.09810_.
  * Zhou and Zafarani (2019) Xinyi Zhou and Reza Zafarani. 2019.  _Fake news: A survey of research, detection methods, and opportunities_.  arXiv preprint arXiv:1901.09657.
  * Zhou et al. (2019) Zhixuan Zhou, Huankang Guan, Meghana Moorthy Bhat, and Justin Hsu. 2019.  Fake news detection via nlp is vulnerable to adversarial attacks.  _arXiv preprint arXiv:1901.09657_.
  * Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.


# Nonlinear filtering of an optical pulse train using dissipative Kerr
solitons

Victor Brasch  CSEM, CH-2000, Neuchatel, Switzerland  Ewelina Obrzud  CSEM,
CH-2000, Neuchatel, Switzerland  University of Geneva, Department of
Astronomy, CH-1290, Versoix, Switzerland  Steve Lecomte  CSEM, CH-2000,
Neuchatel, Switzerland  Tobias Herr<EMAIL_ADDRESS>CSEM, CH-2000,
Neuchatel, Switzerland

###### Abstract

The capability to store light for extended periods of time enables optical
cavities to act as narrow-band optical filters, whose linewidth corresponds to
the cavity’s inverse energy storage time. Here, we report on nonlinear
filtering of an optical pulse train based on temporal dissipative Kerr
solitons in microresonators. Our experimental results in combination with
analytical and numerical modelling show that soliton dynamics enables storing
information about the system’s physical state longer than the cavity’s energy
storage time, thereby giving rise to a filter width that can be more than an
order of magnitude below the cavity’s intrinsic linewidth. Such nonlinear
optical filtering can find immediate applications in optical metrology, low-
timing jitter ultra-short optical pulse generation and potentially opens new
avenues for microwave photonics.

††preprint: ArXiv

##  I Introduction

Cavities are omnipresent in optics. For instance, lasers rely on the feedback
by their cavity Maiman (1960) or can be stabilized to a cavity Drever  _et
al._ (1983). Ultra-stable resonators have enabled sub-Hz linewidth lasers and
represent the fly-wheels for optical atomic clocks Ludlow  _et al._ (2015). In
particular, filter cavities have been used to reduce the timing jitter of
pulsed laser sources and optical signals, which is of high importance to
microwave (MW) photonics and optical precision metrology Capmany and Novak
(2007); Beha  _et al._ (2017). One of the key characteristics of optical
cavities is their linewidth $\kappa$ and the related cavity decay time
$1/\kappa$. Any dynamics faster than $1/\kappa$ (i.e. Fourier-frequencies in
excess of $\kappa$) are suppressed via the linear superposition and temporal
averaging of the light fields inside the cavity. Beyond linear optics, the
strong light intensity in optical enhancement cavities has enabled efficient
nonlinear optical frequency conversion and the advent of high-Q dielectric
optical microresonators Vahala (2003) has given rise to several new areas of
research in nonlinear optics.

Figure 1: Nonlinear filtering of an optical pulse train. a The Fabry-Pérot
filter reduces the timing jitter of the pulse train. b Photograph of a fiber-
based Fabry-Pérot microresonator as it was used in this work. The highly
reflective dielectric mirror appears pink. Sketched into the image is a
dissipative Kerr soliton (DKS) circulating inside the cavity on top of a
driving pulse with a round-trip time of $1/f_{\mathrm{rep}}$. c Linear (L) and
nonlinear (NL) filtering effects in the frequency domain. The linear filtering
effect is given by the Lorentzian lineshape and the linewidth of the cavity
(grey). The nonlinear filtering effectively provides a substantially narrower
filter (purple).

One of these areas of research, which has gained a lot of attention over the
last years, is the one of temporal dissipative Kerr solitons (DKS) in
microresonators Herr  _et al._ (2013) that are coherently driven by a
continuous wave (CW) laser or by synchronous optical pulse trains Obrzud,
Lecomte, and Herr (2017). DKS are pulses of light, which retain their shape
while propagating in the cavity as long as the driving laser is present, owing
to a balance between dispersion and the focusing part of the nonlinearity as
well as between the optical losses and nonlinear gain Akhmediev and Ankiewicz
(2008). Since their first experimental observation in long, fiber-based
cavities Leo  _et al._ (2010) and crystalline microresonators Herr  _et al._
(2013) as well as microresonators of other types Brasch  _et al._ (2016); Yi
_et al._ (2015), DKS have proven to be a breakthrough in particular for the
field of microresonator-based Kerr frequency combs Del’Haye  _et al._ (2007);
Weiner (2017); Kippenberg  _et al._ (2018); Gaeta, Lipson, and Kippenberg
(2019). Important for experimental systems, DKS have demonstrated to be stable
and robust against small changes of their environment and to tolerate
perturbations of the ideal system Herr  _et al._ (2014); Sahoo and Roy (2019).
It is moreover expected that DKS benefit from being generated inside a cavity,
which filters noise that may be present in the driving laser. Interestingly,
and so far unexplained, it has been observed in a weakly phase-modulated CW-
driven microresonator Weng  _et al._ (2019) that DKS can lead to a reduction
of microwave modulation phase noise at noise frequencies below $\kappa$. In
addition, sub-linewidth suppression of amplitude noise was observed for a
synchronously-driven DKS in free-space enhancement cavities Lilienfein  _et
al._ (2019).

In this work, we generate DKS in fiber-based Fabry-Pérot microresonators (Fig.
1b) via synchronous pulsed driving. Such pulsed driving has enabled
unprecedentedly high conversion efficiency, deterministic operation and,
importantly, all-optical stabilization of the DKS via slow synchronization to
the driving pulses with metrological precision Obrzud  _et al._ (2019). A
major open question, however, is how timing jitter in the input pulse train
translates into DKS timing jitter (Fig. 1a). Here, besides the expected linear
cavity filtering effect, we find that DKS lead to an additional drastic
reduction of timing fluctuations already well within the cavity’s bandwidth
$\kappa$. Despite the differences in the investigated DKS systems,
phenomenologically this observation is similar to the observations mentioned
above Weng  _et al._ (2019); Lilienfein  _et al._ (2019), suggesting a similar
physical origin. In order to explain the observations, we show analytically,
based on soliton physics, that DKS give rise to nonlinear optical filtering
equivalent to an intrinsic second filter cavity with an effective linewidth
$\kappa_{\mathrm{NL}}\ll\kappa$ (Fig. 1c).

Combining both, linear and nonlinear filtering, into a single model, we are
able to understand the experimentally obtained data and explain the observed
filter characteristics. Moreover, numerical simulations confirm the model and
allow us to relate different measured $\kappa_{\mathrm{NL}}$ and features in
the optical spectra to the relative temporal position of the DKS with respect
to the intracavity driving pulse. As such, we establish a novel connection
between the nonlinear optical physics of DKS and the use of cavities as
optical filters.

Figure 2: Nonlinear filter characterization. a Setup used to characterize the
linear and nonlinear filtering effects. AM: electro-optic amplitude modulator;
Att: RF attenuator; CFBG: chirped fiber Bragg grating; ECDL: external-cavity
diode laser; EDFA: erbium-doped fiber amplifier; Microresonator: fiber-based
Fabry-Pérot microresonator; NA: network analyzer; OSA: optical spectrum
analyzer; PD: photodetector; PLL: phase-locked loop controller; PM: electro-
optic phase modulator; RF: MW synthesizer. b Lorentzian lineshape on a linear
scale and on a double-logarithmic scale. c Phase modulation (PM) response of
the cavity in the linear regime for three different laser detunings. The
traces are measured with the NA and are then normalized. The dashed-dotted
line is the fitted full linear model $Z_{\mathrm{mod}}^{\mathrm{L}}(\Omega)$.
The dotted and dashed lines show the individual contributions of
$Z^{\mathrm{L}}_{\mathrm{PM}}$ and $Z^{\mathrm{L}}_{\mathrm{XM}}$ respectively
as explained in the text. d Nonlinear filter response with a DKS state present
in the cavity for three different detunings (blue, green, red). Purple lines
represent the respective best fit of $Z^{\mathrm{NL}}$ to the low frequency
part of the modulation response. The resulting values of
$\kappa_{\mathrm{NL}}$ are given in g. e Optical spectra of the input (grey)
and the output after the microresonator in a DKS state (purple). f Envelope of
the DKS spectrum around the input spectrum for different detunings
corresponding to d. g Fitted values for $\kappa_{\mathrm{NL}}$ for different
detunings. The x-axis is not a quantified detuning but represents different
measurements for different driving laser positions as defined by a change in
the laser lock point. This monotonically changes the detuning. Colors
represent the same detunings as in d and f.

##  II Experimental Setup

For the experiments we use a fiber-based, anomalous-dispersion Fabry-Pérot
microresonator (Fig. 1b) Obrzud, Lecomte, and Herr (2017) with a linewidth of
$\kappa/2\pi\approx 7$ MHz and a free-spectral range (FSR) of 9.8 GHz. This
microresonator is synchronously-driven by an input pulse train whose pulse
repetition rate $f_{\mathrm{rep}}$ is approximately matched to the cavity’s
FSR such that a DKS is generated when the laser is tuned into resonance
Obrzud, Lecomte, and Herr (2017). Experimentally, a small repetition rate
mismatch $\Delta=f_{\mathrm{rep}}-\mathrm{FSR}$ is usually present and, in
particular, dependent on the pump power and the laser detuning $\delta$ as
different intracavity power levels thermally induce a change of the FSR.

The driving pulse train is synthesized via electro-optic modulation from a
continuous wave laser Kobayashi  _et al._ (1988). To this end, the CW
external-cavity diode laser (ECDL) is fed into a cascade of one electro-optic
amplitude modulator and two electro-optic phase modulators (Fig.2a), driven by
the same MW source, with suitable phase delays. A chirped fiber Bragg grating
and a spool of single-mode fiber are used for pulse compression. Before
coupling the light to the cavity, the pulse train is amplified to an average
power of around 100 to 500 mW using an erbium-doped fiber amplifier (EDFA).
Using the Fabry-Pérot microresonator in transmission allows us to readily
measure the intra-cavity field, rejecting light-components that are not
coupled to the cavity (corresponding to a drop port in ring-type
microresonators Wang  _et al._ (2013)). After the cavity the signal is split
and in one part detected with an optical spectrum analyzer; the other part is
amplified with a second EDFA (not shown in Fig. 2a), detected on a fast
photodetector and further amplified in the MW domain.

In our experiment, we aim at characterizing the response of the DKS to timing
jitter in the input pulse train. In order to avoid contamination from noise
sources in the experiment we measure the response in a coherent manner. An
artificial timing modulation is introduced in the input pulse train and the
response of the out-coupled pulse train is measured. As the timing modulation
in the optical domain is identical to a phase modulation in the MW domain, we
practically generate the timing modulation by imprinting a phase modulation
onto the MW signal driving the electro-optic modulators. This phase modulation
(PM) is accomplished via a MW Mach-Zehnder interferometer where the second arm
is strongly attenuated and in quadrature with the first arm. A mixer in the
second arm is used to electrically vary its transmission and thereby to
modulate the phase of the MW signal after the interferometer. The mixer’s
modulation input is connected to the output of a network analyzer (NA). The
input of the NA is connected to the demodulated MW pulse repetition rate
signal after the cavity. As slow thermal drifts in the optical setup impact
the phase of the microwave signal after the photodetector, the demodulation is
accomplished via a second reference MW synthesizer that is slowly phase-locked
to the detected and amplified MW signal.

In order to derive the modulation response $Z_{\mathrm{meas}}(\Omega)$ at
modulation frequency $\Omega$ from the NA data, the raw traces are normalized
by a reference trace taken with the same optical and MW paths but without the
optical cavity and with a matching photo-current on the photodetector. In
addition, because the internal amplitude referencing of the NA is not
meaningful in our measurements, we normalize the measured response functions
to 0 dB at the near-DC end of the measurement.

##  III Experimental Results

In a first step we characterize the linear filtering characteristics of the
cavity when the input pulses are at low optical power, such that no nonlinear
effects are observed. Using the transmitted power as an error signal, the
pulsed driving laser is locked at different detunings. Here, we define the
detuning $\delta=\omega_{0}-\omega_{\mathrm{p}}$ as the difference between the
central spectral frequency component $\omega_{\mathrm{p}}$ of the driving
pulses and the closest resonance frequency $\omega_{0}$ of the cavity.

The simplest case is the one of zero detuning, $\delta=0$, where the expected
measured modulation response $Z_{\mathrm{meas}}^{\mathrm{L}}$ is a Lorentzian-
shaped filter with a cut-off frequency given by the linewidth $\kappa$ as
shown in Fig. 2b. This is also what we observe experimentally for
$\delta\approx 0$ (dark blue trace in Fig. 2c). For larger detunings,
$Z_{\mathrm{meas}}^{\mathrm{L}}$ does not exhibit the simple Lorentzian
lineshape anymore (Fig. 2c, green and golden traces). Instead, the modulation
response shows a peak at around the detuning frequency caused by different
effects, which can be accurately described by an analytic model (dashed-dotted
line) as we show below.

In a second step, we repeat the previous measurements in the nonlinear optical
regime when a single DKS pulse is present in the cavity (Fig. 2d).

Once created, the DKS pulse is stable over a certain range of pump power
levels, laser detunings $\delta$ and repetition rate mismatches $\Delta$
Obrzud, Lecomte, and Herr (2017). In order to create reproducible experimental
conditions, we adjust the repetition rate $f_{\mathrm{rep}}$ such that the DKS
spectrum is as symmetric as possible for a medium pump laser detuning
$\delta$.

Markedly different from the linear optical case, we see a substantial
suppression at frequencies well below the cavity’s linear linewidth $\kappa$
(golden-colored marks in Fig. 2c, d). This suppression can reach values of
more than 10 dB for frequencies ten times lower than $\kappa$. The strength
and the frequency onset of this suppression depend on the detuning $\delta$ as
shown exemplary for three cases (red, green, blue) in Figure 2d.

Similar to the linear case we also see a local enhancement of the modulation
response for values around the detuning, albeit significantly weaker than in
the linear case and with a double-peak structure, which is only resolved for
certain detunings.

Simultaneously to measuring the transfer of the timing modulation, we record
the intracavity optical spectra at the output port. In the central portion of
these spectra, where the DKS and the input pulse spectrum overlap (purple and
grey in Fig. 2e respectively), we observe a striking correlation between the
spectrum and the strength of the nonlinear filtering. This is shown in Figure
2f where the same colors in panels d and f correspond to the same DKS state
with the same detuning. For the lowest frequency filtering onset (green
color), the envelope of the central portion of the spectrum is almost flat
whereas it is tilted towards longer or shorter wavelengths otherwise.
Experimentally, the soliton state can be adiabatically moved between those
states by changing the detuning or by slight changes of the input pulse
repetition rate. We note that a perfectly flat central part of the spectral
envelope is typically not observed and a switching behavior between long and
short wavelength tilt (and vice versa) is commonly the case.

Figure 3: Nonlinear filter model. a The linear filter is given by the
Lorentzian lineshape (grey) of the resonator with linewidth $\kappa$ taking
into account the detuning (gold) resulting in a more complicated lineshape
shown in b. The nonlinear filter represents a much narrower Lorentzian
(purple) with linewidth $\kappa_{\mathrm{NL}}$ which remains centered on the
optical input frequencies for all detunings. b Shape of the different linear
and nonlinear contributions. c Time domain picture of the soliton on top of
the driving pulse and the resulting dynamic for the soliton. d Diagram showing
the different modulation signal paths through the system and how they
contribute. e Fit of the full model $Z_{\mathrm{mod}}^{\mathrm{NL}}$ (green)
to the experimental data (solid grey, same data as green in Fig. 2d). Dashed
lines represent the individual contributions as shown in d. Grey dashed is the
pure linear response function. f Phase modulation response function derived
from numerical simulations (grey) and fitted model (blue). Dashed purple is
again the ideal soliton PM response.

##  IV Analytical model

To understand the observations and the physics of our system, we develop an
analytical model starting out from the linear case and later extending it to
the nonlinear case with DKS.

For a linear optical cavity the standard quantum Langevin approach Gardiner
and Collett (1985) can be used to derive the timing jitter or microwave phase
modulation transfer

$Z_{\mathrm{PM}}^{\mathrm{L}}(\Omega,\delta)=\frac{\kappa^{2}\left(4\Omega^{2}+8\delta^{2}+\kappa^{2}\right)+16\delta^{4}}{\left(4\Omega^{2}+4\delta^{2}+\kappa^{2}\right)^{2}-64\Omega^{2}\delta^{2}}$
(1)

which is equivalent to the absolute optical phase modulation transfer of a
single optical line. For completeness we note that the non-zero group-velocity
dispersion (GVD) of the cavity and a small potential mismatch between the
cavity’s FSR and the driving pulse repetition rate $f_{\mathrm{rep}}$ implies
that the detuning $\delta$ between cavity modes and driving laser lines is
sightly dependent on their distance from the central modes $\omega_{p}$ and
$\omega_{0}$, respectively. However, in our model we neglect this fact.

The characteristic form of $Z_{\mathrm{PM}}^{\mathrm{L}}$ for
$\delta=1.7\kappa$ is shown in Fig. 3b, left. It might seem counterintuitive
that $Z_{\mathrm{PM}}^{\mathrm{L}}$ exceeds 0 dB for $\Omega\approx\delta$ but
this can be understood by realizing that for a detuned, off-resonant line its
modulation sideband can be resonant and gain relative power.

In an ideal experimental system timing modulation or phase modulation (PM) of
the input pulse train will not translate into amplitude modulation (AM) in the
intracaviy pulse. Indeed the optical frequency components of the input pulse
train will ‘breath’ symmetrically around the pulse’s center frequency such
that a resulting intracavity amplitude modulation in a single sideband with
relative mode index $+\mu$ (with regard to the central mode) will be
compensated by the amplitude change in the sideband with mode index $-\mu$.
However, in the case of inevitable imperfections such as a mismatch between
the driving pulse repetition rate $f_{\mathrm{rep}}$ and the cavity’s FSR
(Fig. 3a), non-zero higher order dispersion and asymmetric driving spectra,
even a pure timing modulation will result in some amplitude modulation.
Moreover, AM-to-PM conversion caused by chirped driving pulses as well as AM-
to-PM conversion in the photodetector and in the MW mixers can add an
additional component to the measured signal. While it is difficult to
quantitatively capture all these nonideal effects, their functional shape is
distinctively different from $Z_{\mathrm{PM}}^{\mathrm{L}}$ and described by
the PM-to-AM transfer of a single optical line (again derived using the
standard quantum Langevin approach)

$Z_{\mathrm{XM}}^{\mathrm{L}}(\Omega,\delta)=\frac{1024\cdot
D_{1}^{2}\kappa^{2}\Omega^{2}\delta^{2}}{(4\delta^{2}+\kappa^{2})^{2}\cdot\left((4\Omega^{2}+4\delta^{2}+\kappa^{2})^{2}-(8\Omega\delta)^{2}\right)}$
(2)

where $D_{1}=2\pi\cdot\mathrm{FSR}$. The characteristic form of
$Z_{\mathrm{XM}}^{\mathrm{L}}$ is shown in Fig. 3b, middle, again for
$\delta=1.7\kappa$. Based on equations 1 and 2, we build a model that does not
only include the ideal timing fluctuation transfer characteristics represented
by $Z_{\mathrm{PM}}^{\mathrm{L}}$ but in addition a nonideal component with
the functional form of $Z_{\mathrm{XM}}^{\mathrm{L}}$:

$Z_{\mathrm{mod}}^{\mathrm{L}}(\Omega,\delta)=Z_{\mathrm{PM}}^{\mathrm{L}}(\Omega,\delta)+\alpha
Z_{\mathrm{XM}}^{\mathrm{L}}(\Omega,\delta)$ (3)

where $\alpha$ is an empirical positive and unit-free parameter describing the
effects resulting from experimental imperfections as discussed above.

Taking $\alpha$ and $\delta$ as the only free fitting parameters
($\kappa/2\pi=7.0$ MHz, $D_{1}/2\pi=9.8$ GHz), we can model the measured trace
$Z_{\mathrm{meas}}^{\mathrm{L}}$ (Fig. 2c, golden trace) very well for
$\alpha=3.2\cdot 10^{-4}$ and $\delta=1.7\kappa$ (dashed-dotted trace). Based
on this model, we can also plot the ideal modulation response
$Z_{\mathrm{PM}}^{\mathrm{L}}$ (dotted grey line in Fig. 2c).

In order to extend the model to the nonlinear case, we first focus on the most
striking feature in the nonlinear response, the suppression of modulations
below the linear linewidth $\kappa$. This is surprising as, given the ultra-
fast nature of the Kerr-nonlinear effects, the cavity decay time $1/\kappa$
can be assumed to be the slowest relevant time scale. On the other hand, it is
known that DKS can exhibit slow dynamics resulting from weak soliton-soliton
interaction Jang  _et al._ (2013), interaction with a chirped background Jang
_et al._ (2015) or from interaction with a driving pulse Obrzud, Lecomte, and
Herr (2017); Hendry  _et al._ (2018, 2019).

Recent studies have shown that while in a stable equilibrium the DKS is bound
to the driving pulse, it may drift with a certain relative velocity $v$
relative to the driving pulse if the system is perturbed Hendry  _et al._
(2018, 2019) (Fig. 3c):

$v=\frac{\mathrm{d}x}{\mathrm{d}t}=a(S,\delta)\frac{\mathrm{d}S}{\mathrm{d}x}+d$
(4)

where $t$ denotes time, $x(t)$ is the relative position of the soliton with
regard to its bound equilibrium position on the driving pulse, $a$ is a
coefficient that depends on the detuning $\delta$ and the local envelope of
the driving field $S$, and $d$ is the drift coefficient due to the mismatch
between the driving pulse repetition rate and the intrinsic repetition rate of
the soliton state. The computation of $a$ can be performed numerically Hendry
_et al._ (2018).

Here we assume that the disturbances of the bound state due to the induced
modulations or other perturbations are small such that the response of the
system can be expanded and that it will be dominated by the linear part with
coefficient $A=-\frac{\mathrm{d}v}{\mathrm{d}x}|_{x=0}$, where for a stable
equilibrium $A>0$. The phase modulation introduced in our experiment is
equivalent to a small timing modulation of the driving pulses, which we can
describe as a harmonic perturbation of the driving pulse position
$p(t)=\varepsilon\mathrm{e}^{i\Omega t}$, where $\Omega$ and $\varepsilon$ are
the frequency and amplitude of the perturbation. The dynamics of the soliton
are then described by the following differential equation

$\frac{\mathrm{d}x}{\mathrm{d}t}=-A(x(t)-\varepsilon\mathrm{e}^{i\Omega t})\,$
(5)

which corresponds to a Lorentzian low-pass filter

$Z^{\mathrm{NL}}(\Omega)=\frac{A^{2}}{\Omega^{2}+A^{2}}$ (6)

with a cut-off frequency of $\kappa_{\mathrm{NL}}=2A$. This explains the
observed nonlinear filter characteristics and shows that effectively, the
nonlinear filtering can be interpreted as an equivalent additional dispersion-
free, zero-detuned linear cavity filter with linewidth
$\kappa_{\mathrm{NL}}<\kappa$. The cut-off frequency is determined by the
parameter $A$ that depends on the laser detuning $\delta$, the repetition rate
mismatch and the derivative of the driving pulse profile $\frac{dS}{dx}$
Hendry  _et al._ (2018, 2019).

One interpretation of this narrower filter bandwidth is, that effectively the
nonlinear optical DKS dynamics can store information about the system’s
physical state much longer than the intrinsic cavity decay time $1/\kappa$
thereby enabling a filter bandwidth significantly below $\kappa$.

The purple dashed lines in Figure 2d represent the best fits of the Lorentzian
filter behavior $Z^{\mathrm{NL}}$ for low frequencies where linear cavity
filtering effects of $Z_{\mathrm{PM}}^{\mathrm{L}}$ and
$Z_{\mathrm{AM}}^{\mathrm{L}}$ are negligible. As expected from eq. 4,
$\kappa_{\mathrm{NL}}$ is detuning dependent (Fig. 2d), where a change in the
detuning also impacts the repetition rate mismatch $\Delta$ and thereby the
drift coefficient $d$. For a fixed pulse repetition rate $f_{\mathrm{rep}}$,
the lowest value of $\kappa_{\mathrm{NL}}$, measured while changing the
detuning, is more than 20 times below the cavity linewidth $\kappa$. This
minimum corresponds to the green traces in Figure 2d and f and in particular
to the case where the envelope of the central portion of the spectrum is close
to horizontal.

In order to describe the full measured nonlinear trace also for higher
modulation frequencies $\Omega$, we extend our model for the linear case (eq.
3) to take into account the DKS with the effects described above as well as
the driving pulses as summarized in Figure 3d. First, we need to differentiate
between the contribution of the residual driving pulses and the contribution
of the DKS. Because of the large difference in intensity, the two parts
experience different effective detunings $\delta_{\mathcal{C}}$ and
$\delta_{\mathcal{S}}$ respectively. Here, our nomenclature adopts the
convention established for CW driven DKS, where the different Kerr-shifts lead
to distinctly detuned CW $\mathcal{C}$ and soliton $\mathcal{S}$ resonances
Guo  _et al._ (2016).

For the fraction $\mathcal{C}$ of the transmitted input light, we use the
linear model developed above. For the soliton fraction $\mathcal{S}$ we
combine the nonlinear model and the linear model by multiplying
$Z_{\mathrm{PM}}^{\mathrm{L}}$ with $Z^{\mathrm{NL}}$ to obtain the full
nonlinear PM response function
$Z_{\mathrm{PM}}^{\mathrm{NL}}(\Omega,\delta_{\mathcal{S}})=Z^{\mathrm{NL}}(\Omega)Z_{\mathrm{PM}}^{\mathrm{L}}(\Omega,\delta_{\mathcal{S}})$
as shown in Fig.3b, right. As for the linear case (eq. 2), the response
resulting from imperfections is given as $\alpha\cdot
Z_{\mathrm{XM}}^{\mathrm{L}}$. Combining the linear with the nonlinear part
the full model reads:

$\begin{split}Z_{\mathrm{mod}}^{\mathrm{NL}}(\Omega,\delta_{\mathcal{S}},\delta_{\mathcal{C}})=\mathcal{S}\cdot\left(Z^{\mathrm{NL}}_{\mathrm{PM}}(\Omega,\delta_{\mathcal{S}})+\alpha
Z_{\mathrm{XM}}^{\mathrm{L}}(\Omega,\delta_{\mathcal{S}})\right)\\\
+\mathcal{C}\cdot\left(Z_{\mathrm{PM}}^{\mathrm{L}}(\Omega,\delta_{\mathcal{C}})+\alpha
Z_{\mathrm{XM}}^{\mathrm{L}}(\Omega,\delta_{\mathcal{C}})\right)\end{split}$
(7)

where $\mathcal{S}$ and $\mathcal{C}=1-\mathcal{S}$ are the relative
contributions of the soliton and non-soliton fractions. Figure 3e shows a fit
of this nonlinear modulation transfer model (green) to the experimental trace
(grey) where the following fit parameters were found:
$\delta_{\mathcal{C}}/2\pi=20.5$ MHz, $\delta_{\mathcal{S}}/2\pi=30.0$ MHz,
$\kappa_{\mathrm{NL}}/2\pi=280$ kHz, $\alpha=2.8\cdot 10^{-5}$ and
$\mathcal{S}=0.978$. Despite the simple assumptions the model for
$Z_{\mathrm{mod}}^{\mathrm{NL}}$ shows very good agreement with the
experimental data, suggesting that it captures the essential underlying
physics. The model therefore allows estimating the nonlinear filter
characteristics for an ideal DKS based filter by making $\alpha\rightarrow 0$
and therefore rejecting effects related to imperfections as discussed above.
The resulting trace is shown in Figure 3e in blue. Comparing this trace to the
ideal linear filter characteristics ($\mathcal{C}=1$ and $\alpha=0$, dashed
grey line) the strong modulation suppression by the DKS of several orders of
magnitude is apparent in particular at low offset frequencies, where most
oscillators are inherently noisy. In addition, we can set $\mathcal{S}=1$ and
$\mathcal{C}=0$ corresponding to a photodetection of only the soliton
component (e.g. by perfectly suppressing the driving portion of the spectrum),
resulting in even stronger timing modulation suppression (purple).

##  V Numerical simulations

In order to further support our experimental and analytical results we perform
numerical simulations that are based on the coupled mode model for
synchronously driven microresonators Chembo and Yu (2010); Hansson, Modotto,
and Wabnitz (2014); Obrzud, Lecomte, and Herr (2017), with parameters close to
those in the experiment. The experiment, whose results are shown in Figure 2d,
is repeated numerically by creating a DKS state and injecting an artificial
timing modulation. Numerical demodulation leads to the modulation response
shown in Fig. 3f. All characteristic features of the experiment are reproduced
on a qualitative level demonstrating that also the numerical simulation
captures the relevant physics. As the numerical simulation implements an
idealized experiment there is good agreement of the simulated modulation
response and the idealized model $Z_{\mathrm{mod}}^{\mathrm{NL}}$ with
$\alpha=0$ (Fig. 3f, blue trace). The purple trace in Figure 3f shows
$Z_{\mathrm{mod}}^{\mathrm{NL}}$ with $\alpha=0$ and $\mathcal{C}=0$.

Numerically, creating a number of different soliton states, we find that the
tilt of the spectral envelope as observed in Figure 2f is also reproduced in
the simulations. Closer inspection shows that this tilt is directly related to
the relative temporal position of the DKS with regard to the intracavity
driving pulse. As shown in Figure 4, a perfectly flat spectral envelope (green
trace in Fig. 4a and b) corresponds to the DKS being symmetrically centered on
the underlying driving pulse (Fig.4c, driving pulse in blue, DKS in purple),
tilted spectral envelopes are observed for off-centered DKS (red and blue
traces), where the tilt increases for larger asymmetries. This, together with
the insights from Fig. 2f and g, indicates a link between a more centered DKS
position on the driving pulse and lower values of $\kappa_{\mathrm{NL}}$.

In order to better understand the effects of repetition rate mismatch,
detuning as well as pump power on the DKS position and $\kappa_{\mathrm{NL}}$,
we perform numerical simulations wherein each parameter is swept and studied
separately free of thermally induced cross-coupling.

Figure 4: Soliton dynamics in numerical simulations. a Zoom into the central
part of the full simulated spectrum shown in b. c Soliton position on top of
the driving pulse for the different cases shown in a. The position of the
soliton in the middle of the driving pulse represents the symmetric spectrum
(both in green). d Time domain intracavity field with respect to the driving
pulse’s temporal position. As the repetition rate mismatch is numerically
swept from –5 kHz to 5 kHz the soliton (high intensity, yellow) moves over the
center of the pump pulse (white line at 0 ps). Other pump parameters:
$\delta=$ 3.5 $\kappa$, $P_{\mathrm{p}}=$ 0.02 W. e Symmetric dependence of
$\kappa_{\mathrm{NL}}$ on the soliton’s temporal position relative to the
driving pulse. f Sweep similar to a but with changing detuning ($\Delta=$ 0
kHz, $P_{\mathrm{p}}=$ 0.02 W). g As d and f but with changing pump power
($\Delta=$ 0 kHz, $\delta=$ 3.5 $\kappa$).

As shown in Figure 4d the DKS position can be tuned continuously via the
repetition rate mismatch $\Delta$ across both sides of the driving pulse.
Figure 4e confirms that more centered soliton positions are indeed linked to a
lower $\kappa_{\mathrm{NL}}$. Also, the detuning $\delta$ impacts the DKS
position as shown in Figure 4f, where larger detunings (closer to the ‘edge’
of the soliton step), result in more centered DKS. Finally, Figure 4g shows
the dependence of the DKS position on the pump power, where a lower pump power
leads to a more centered DKS position.

However, while perfectly centered DKS can be generated numerically, these
states are found to be long-term stable only for low pump power levels and
otherwise will assume a slightly off-center position (unless additional chirp
is present in the driving pulses). This is in agreement with the
experimentally observed switching behavior mentioned above and also consistent
with previous analysis by Hendry et al. Hendry  _et al._ (2019) where for
higher pump power, two symmetric stable off-center positions are identified.
Which of those symmetric position is assumed is related to temporal symmetry
breaking Hendry  _et al._ (2018); Xu and Coen (2014); Copie  _et al._ (2019)
and may be influenced via the repetition rate mismatch.

We conclude that a position of the DKS close to the center of the driving
pulse results in lower values for $\kappa_{\mathrm{NL}}$ and that the DKS
position itself depends on the combination of all pump parameters. For
instance, in case of high pump power or small detuning a non-zero repetition
rate mismatch may be used to obtain a more centered DKS position. An example
for this are the results shown in Figure 2g, where the lowest
$\kappa_{\mathrm{NL}}$ is obtained for a medium detuning for which the
effective repetition rate mismatch $\Delta$ is optimal. In the same
measurement, the increase of $\kappa_{\mathrm{NL}}$ for smaller (larger)
detunings can be attributed to a thermally induced decreasing (increasing)
mismatch $\Delta$, departing from the optimal condition.

Figure 5: Absolute phase noise filtering. a The intrinsic phase noise of a
compact RF synthesizer is shown in blue as the phase noise of the input pulse
train. The phase noise after nonlinear optical filtering is shown in purple.
The grey, dashed trace is derived from the blue input trace by multiplying the
data with $Z^{\mathrm{NL}}$ (eq. 6) with $\kappa_{\mathrm{NL}}/2\pi=750$ kHz.
b Dependence of $\kappa_{\mathrm{NL}}$ on the detuning. As in Fig. 2g, the
detuning is not quantified but simply represents different driving laser
detuning steps as fixed by different laser lock points.

##  VI Application

It has been shown before that solitons driven by continuous wave pump lasers
Yi  _et al._ (2015); Liang  _et al._ (2015); Lucas  _et al._ (2019) can have
very low phase noise at larger offset frequencies and, moreover, can be
disciplined to a MW reference via weakly phase-modulated CW lasers without
losing this purity Weng  _et al._ (2019). Here we show that using nonlinear
filtering we can achieve corresponding phase noise reductions for a pulse
train.

In this measurement, a compact, low-cost microwave synthesizer is used to
drive the electro-optic modulators that generate the pulse train. No
additional timing modulation is injected and instead we measure directly the
absolute phase noise of the pulse train before and after the cavity using a
commercial, high-performance signal analyzer with cross-correlation function.
We detect the entire output spectrum and do not compensate for any of the
nonideal contributions discussed above (therefore $\mathcal{S}<1$ and
$\alpha>0$). The measured phase noise after the microresonator and the DKS
(purple trace in Fig. 5a) shows a clear suppression with a maximum of 15 dB at
around 3 MHz compared to the input trace (blue), implying that the DKS acts as
an effective clean-up oscillator. In order to quantify the nonlinear filtering
behavior we fit $Z^{\mathrm{NL}}$ to the lower offset frequencies. For the
data in Figure 5a we obtain a nonlinear linewidth $\kappa_{\mathrm{NL}}/2\pi$
of 750 kHz, 9$\times$ below the measured linear linewidth $\kappa/2\pi$ of the
resonator.

Similarly to our previous measurement (Fig. 2g) and our numerical simulations,
we find again a detuning dependence of $\kappa_{\mathrm{NL}}$ with a minimum
of approximately 400 kHz (Fig.5b.)

We note that the decrease in phase noise suppression for higher offset
frequencies (around 17 MHz in Fig. 5a) is related to the detuning and the
different nonideal effects discussed above. The filter effect could be further
increased by amplifying and detecting only light at wavelengths outside the
input pulses’ spectral coverage, which in equation 7 is equivalent to
$\mathcal{S}\approx 1$ as transmitted residual input light is rejected.

##  VII Conclusion

In summary, we present a nonlinear optical filter based on dissipative Kerr
soliton dynamics in a nonlinear optical microresonator, which reduces the
timing jitter of an optical input pulse train. We show that the tunable
bandwidth of the filter can be more than one order of magnitude lower than the
cavity’s intrinsic optical linewidth. In an example application we demonstrate
that the soliton can act as a clean-up oscillator, leading to a phase noise,
or equivalently timing jitter, reduction of an electro-optically generated
pulse train, where the suppression can reach 15 dB for some offset
frequencies.

These surprising effects are accurately described by an analytic model of the
soliton dynamics and are fundamentally related to the solitons capability of
storing information about the system’s physical state longer than the cavity’s
light storage time. These insights can likely also explain previously made
observations of sub-linewidth noise suppression, where a similar underlying
mechanism might be in place. Our findings are of immediate relevance to
optical precision metrology, ultra-fast optical measurements, resonant
supercontinuum generation in microresonators Obrzud, Lecomte, and Herr (2017);
Anderson  _et al._ (2019) as well as microwave photonics including ultra-
narrowband optical filters for radar or signal processing. The fact that DKS
states can preserve information longer than the cavity’s dissipation time
scale may open new avenues for fundamental research at the interface of
nonlinear optics and information theory.

## Funding Information

This work was supported by the Swiss National Science Foundation and
Innosuisse via the Bridge Discovery Grant 20B2-1_176563 as well as the Canton
of Neuchâtel.

## Acknowledgments

We thank the Laboratoire Temps-Fréquence (LTF) at the University of Neuchâtel
for support in performing the absolute phase noise measurements and Miro
Erkintalo for helpful discussions. Moreover, we thank many members of the Open
Source Software community for their great work as several different open
source projects have been used in the course of the described work.

## References

  * Maiman (1960) T. H. Maiman, Nature 187, 493 (1960).
  * Drever  _et al._ (1983) R. W. P. Drever, J. L. Hall, F. V. Kowalski, J. Hough, G. M. Ford, A. J. Munley, and H. Ward, Applied Physics B 31, 97 (1983).
  * Ludlow  _et al._ (2015) A. D. Ludlow, M. M. Boyd, J. Ye, E. Peik, and P. Schmidt, Reviews of Modern Physics 87, 637 (2015).
  * Capmany and Novak (2007) J. Capmany and D. Novak, Nature Photonics 1, 319 (2007).
  * Beha  _et al._ (2017) K. Beha, D. C. Cole, P. Del’Haye, A. Coillet, S. A. Diddams, and S. B. Papp, Optica 4, 406 (2017).
  * Vahala (2003) K. J. Vahala, Nature 424, 839 (2003).
  * Herr  _et al._ (2013) T. Herr, V. Brasch, J. D. Jost, C. Y. Wang, N. M. Kondratiev, M. L. Gorodetsky, and T. J. Kippenberg, Nature Photonics 8, 145 (2013).
  * Obrzud, Lecomte, and Herr (2017) E. Obrzud, S. Lecomte, and T. Herr, Nature Photonics 11, 600 (2017).
  * Akhmediev and Ankiewicz (2008) N. Akhmediev and A. Ankiewicz, Springer-Verlag (2008).
  * Leo  _et al._ (2010) F. Leo, S. Coen, P. Kockaert, S.-P. Gorza, P. Emplit, and M. Haelterman, Nature Photonics 4, 471 (2010).
  * Brasch  _et al._ (2016) V. Brasch, M. Geiselmann, T. Herr, G. Lihachev, M. H. P. Pfeiffer, M. L. Gorodetsky, and T. J. Kippenberg, Science 351, 357 (2016).
  * Yi  _et al._ (2015) X. Yi, Q.-F. Yang, K. Y. Yang, M.-G. Suh, and K. Vahala, Optica 2, 1078 (2015).
  * Del’Haye  _et al._ (2007) P. Del’Haye, A. Schliesser, O. Arcizet, T. Wilken, R. Holzwarth, and T. J. Kippenberg, Nature 450, 1214 (2007).
  * Weiner (2017) A. M. Weiner, Nature Photonics 11, 533 (2017).
  * Kippenberg  _et al._ (2018) T. J. Kippenberg, A. L. Gaeta, M. Lipson, and M. L. Gorodetsky, Science 361, eaan8083 (2018).
  * Gaeta, Lipson, and Kippenberg (2019) A. L. Gaeta, M. Lipson, and T. J. Kippenberg, Nature Photonics 13, 158 (2019).
  * Herr  _et al._ (2014) T. Herr, V. Brasch, J. Jost, I. Mirgorodskiy, G. Lihachev, M. Gorodetsky, and T. Kippenberg, Physical Review Letters 113 (2014), 10.1103/PhysRevLett.113.123901.
  * Sahoo and Roy (2019) A. Sahoo and S. Roy, arXiv:1905.05960 [nlin, physics:physics] (2019), arXiv: 1905.05960.
  * Weng  _et al._ (2019) W. Weng, E. Lucas, G. Lihachev, V. E. Lobanov, H. Guo, M. L. Gorodetsky, and T. J. Kippenberg, Physical Review Letters 122, 013902 (2019).
  * Lilienfein  _et al._ (2019) N. Lilienfein, C. Hofer, M. Högner, T. Saule, M. Trubetskov, V. Pervak, E. Fill, C. Riek, A. Leitenstorfer, J. Limpert, F. Krausz, and I. Pupeza, Nature Photonics , 1 (2019).
  * Obrzud  _et al._ (2019) E. Obrzud, M. Rainer, A. Harutyunyan, M. H. Anderson, J. Liu, M. Geiselmann, B. Chazelas, S. Kundermann, S. Lecomte, M. Cecconi, A. Ghedina, E. Molinari, F. Pepe, F. Wildi, F. Bouchy, T. J. Kippenberg, and T. Herr, Nature Photonics 13, 31 (2019).
  * Kobayashi  _et al._ (1988) T. Kobayashi, H. Yao, K. Amano, Y. Fukushima, A. Morimoto, and T. Sueta, IEEE Journal of Quantum Electronics 24, 382 (1988).
  * Wang  _et al._ (2013) P.-H. Wang, Y. Xuan, L. Fan, L. T. Varghese, J. Wang, Y. Liu, X. Xue, D. E. Leaird, M. Qi, and A. M. Weiner, Optics Express 21, 22441 (2013).
  * Gardiner and Collett (1985) C. W. Gardiner and M. J. Collett, Physical Review A 31, 3761 (1985).
  * Jang  _et al._ (2013) J. K. Jang, M. Erkintalo, S. G. Murdoch, and S. Coen, Nature Photonics 7, 657 (2013).
  * Jang  _et al._ (2015) J. K. Jang, M. Erkintalo, S. Coen, and S. G. Murdoch, Nature Communications 6, 7370 (2015).
  * Hendry  _et al._ (2018) I. Hendry, W. Chen, Y. Wang, B. Garbin, J. Javaloyes, G.-L. Oppo, S. Coen, S. G. Murdoch, and M. Erkintalo, Physical Review A 97, 053834 (2018).
  * Hendry  _et al._ (2019) I. Hendry, B. Garbin, S. G. Murdoch, S. Coen, and M. Erkintalo, arXiv:1905.09810 [physics] (2019), arXiv: 1905.09810.
  * Guo  _et al._ (2016) H. Guo, M. Karpov, E. Lucas, A. Kordts, M. H. P. Pfeiffer, V. Brasch, G. Lihachev, V. E. Lobanov, M. L. Gorodetsky, and T. J. Kippenberg, Nature Physics 13, nphys3893 (2016).
  * Chembo and Yu (2010) Y. K. Chembo and N. Yu, Physical Review A 82 (2010), 10.1103/PhysRevA.82.033801.
  * Hansson, Modotto, and Wabnitz (2014) T. Hansson, D. Modotto, and S. Wabnitz, Optics Communications 312, 134 (2014).
  * Xu and Coen (2014) Y. Xu and S. Coen, Optics Letters 39, 3492 (2014).
  * Copie  _et al._ (2019) F. Copie, M. T. Woodley, L. Del Bino, J. M. Silver, S. Zhang, and P. Del’Haye, Physical Review Letters 122, 013905 (2019).
  * Liang  _et al._ (2015) W. Liang, D. Eliyahu, V. S. Ilchenko, A. A. Savchenkov, A. B. Matsko, D. Seidel, and L. Maleki, Nature Communications 6, 7957 (2015).
  * Lucas  _et al._ (2019) E. Lucas, P. Brochard, R. Bouchand, S. Schilt, T. Südmeyer, and T. J. Kippenberg, arXiv:1903.01213 [physics] (2019), arXiv: 1903.01213.
  * Anderson  _et al._ (2019) M. H. Anderson, R. Bouchand, J. Liu, W. Weng, E. Obrzud, T. Herr, and T. J. Kippenberg, arXiv:1909.00022 [physics] (2019).