#  KoBEST: Korean Balanced Evaluation of Significant Tasks

Dohyeong Kim1 Myeongjun Jang2 Deuk Sin Kwon1 Eric Davis1
1Language Super Intelligence Labs, SK Telecom, South Korea
{bruno, brian.k<EMAIL_ADDRESS>
2Department of Computer Science, University of Oxford, UK
<EMAIL_ADDRESS>

###### Abstract

A well-formulated benchmark plays a critical role in spurring advancements in
the natural language processing (NLP) field, as it allows objective and
precise evaluation of diverse models. As modern language models have become
more elaborate and sophisticated, more difficult benchmarks that require
linguistic knowledge and reasoning have been proposed. However, most of these
benchmarks only support English, and great effort is necessary to construct
benchmarks for other low resource languages. To this end, we propose a new
benchmark named Korean balanced evaluation of significant tasks (KoBEST),
which consists of five Korean-language downstream tasks. Professional Korean
linguists designed the tasks that require advanced Korean linguistic
knowledge. Moreover, our data is purely annotated by humans and thoroughly
reviewed to guarantee high data quality. We also provide baseline model and
human performance results. Our dataset is available on the Huggingface
111https://huggingface.co/datasets/skt/kobest_v1.

##  1 Introduction

The NLP field is now facing unprecedented rapid development. A major factor
propelling the progress is the existence of unified benchmark datasets like
GLUE Wang et al. (2018), which are designed to assess models’ language
understanding capabilities. Such benchmark datasets, enabled modern pre-
trained language models, such as BERT Devlin et al. (2019) and GPT-2 and GPT-3
Radford et al. (2019); Brown et al. (2020), to be assessed in objective and
multifaceted manners. The success of GLUE also lead to similar benchmark
datasets in a variety of other languages. Xu et al. (2020); Le et al. (2020);
Wilie et al. (2020); Park et al. (2021).

However, many recent studies reveal that the outstanding performance of PLMs
on such benchmark datasets seems plausible but not probable. These studies
have found that datasets may contain many spurious artefacts, and the
performance of PLMs is enhanced by excessive usage of said artefacts Habernal
et al. (2018); Niven and Kao (2019); McCoy et al. (2019); Bender and Koller
(2020). Another line of work observed that many PLMs, which showed promising
results in GLUE, fall short of expectations for more difficult tasks that
require linguistic knowledge Bhatt et al. (2021) or logical reasoning Tian et
al. (2021). As a result, the importance of well-designed evaluation datasets
with higher difficulty-level has been highlighted, and new datasets, such as
CheckList Ribeiro et al. (2020), and LogicNLI Tian et al. (2021), have been
proposed. Most of them only support specific languages like English, and it
requires large efforts to build higher difficulty-level language evaluation
suits for other low resource languages, however.

When it comes to the Korean language, two benchmarks are widely used: Korean-
NLI & STS Ham et al. (2020) and KLUE Park et al. (2021). The former is
machine- and human-translated from English natural language inference (NLI)
and semantic textual similarity (STS) datasets, which hardly reflect the
characteristics of the Korean language. The latter is a Korean version GLUE
benchmark which supports eight tasks, such as NLI, STS, named entity
recognition (NER), and relation extraction (RE). Although these tasks are
useful for assessing general language ability, it is difficult to ascertain
whether a model is able to reason based on more complicated knowledge beyond
text form (e.g., passage of time, meaning of text, causality). To this end, we
aim to construct a new benchmark dataset in Korean named KoBEST, which
consists of five downstream tasks that require advanced knowledge of Korean.
We carefully constructed the data based on the following design principles:

  * •

Human-driven data annotation: Our data is purely annotated by humans to
prevent incorrect and ambiguous data instances caused by automatic data
annotation approach.

  * •

Leveraging professional linguistic knowledge: As a result of our collaboration
with professional Korean linguists, we re able to collect grammatically
correct data with rich vocabulary and expressions.

  * •

Availability to public: As a benchmark dataset, it is important to ensure
public accessibility. We guarantee that our data is free to use and
redistribute.

  * •

High data quality: Our data passed thorough reviews driven by both models and
humans to deliver high quality data without superficial cues and heuristic
artefacts.

  * •

Avoiding AI ethical issues: Our data does not contain any toxic content,
social biases, or personal information.

Next, we evaluated widely used Korean PLMs on the KoBEST dataset.
Specifically, we conducted fine-tuning, zero-shot, one-shot, and few-shot
experiments. The experimental results can serve as a baseline for performance
on KoBEST. Participants also provided human performance baselines for all of
our tasks. Our results suggest that modern PLMs and a large-size generative
language model (GLM) are far from reaching human-level language ability.

Tasks | |Train| | |Dev| | |Test| | Metrics | Text Source
---|---|---|---|---|---
KB-BoolQ | 3.665 | 700 | 1,404 | F1 | Wikipedia
KB-COPA | 3,076 | 1,000 | 1,000 | F1 | N.A
KB-WiC | 3,318 | 1,260 | 1,260 | F1 | Korean Dictionary
KB-HellaSwag | 2,029 | 500 | 500 | F1 |  Wikipedia, YouTube
KB-SentiNeg | 3.649 | 400 | 397 | F1 | Product reviews
Table 1: The number of data instances for each downstream task.

##  2 KoBEST Downstream Tasks

###  2.1 Overview

The KoBEST benchmark consists of the following five downstream tasks:

  1. 1.

KoBEST-BoolQ (KB-BoolQ): identify whether a given question is true or false
considering a paragraph.

  2. 2.

KoBEST-COPA (KB-COPA): select an alternative which is a cause/effect of a
given premise.

  3. 3.

KoBEST-WiC (KB-WiC): identify whether the meaning of a target word is the same
or different in two given contexts.

  4. 4.

KoBEST-HellaSwag (KB-HellaSwag): select a correct sentence among four
candidates that is likely to appear after a given context.

  5. 5.

KoBEST-SentiNeg (KB-SentiNeg): predict the polarity of a negated sentence.

The number of training/development/test data points is illustrated in Table 1.

KB-BoolQ |  Paragraph: 구한말, 통영 안뒤산 기슭 간창골에 김봉제 형제가 살았다. 김봉제는 관약국을 경영하며 부를 누렸는데, 선비적 성품을 지녔던 형과 반대로 막냇동생 김봉룡은 성질이 포악했다. 어느 날 봉룡은 아내였던 숙정을 사모하던 나그네를 살해하였고, 숙정은 누명을 벗으려고 비상을 먹고 자살한다. (At the end of the Joseon Dynasty, the Kim Bong-je brothers lived in Ganchanggol at the foot of Andui Mountain in Tongyeong. Kim Bong-je enjoyed wealth while running a government office, but his younger brother Kim Bong-ryong had a violent temper, contrary to his brother who had a scholarly character. One day, Bongryong killed a traveler who adored his wife Sukjeong, and Sukjeong commits suicide to clear his name.) Question: 봉룡은 숙정을 죽였는가? (Did Bongryong kill Sukjeong?) Answer: False
---|---
KB-COPA |  Premise: 전쟁이 시작되었다. (The war had begun.) Question: 결과 (Effect) Alternative 1: 병사들이 집으로 돌아왔다. (Soldiers returned home.) Alternative 2: 병사들이 전투에 파견되었다. (Soldiers were sent to battle.) Correct Alternative: 2
KB-WiC |  Context 1: 망가진 엔진은 수리가 불가능하다. (It is impossible to repair a broken engine.) Context 2: 이 배는 수리에 들어간 지 일주일이 됐다. (The ship has been under repair for a week.) Target Word: 수리 (repair) Answer: True
KB-HellaSwag |  Context: 양궁 선수들이 경기장으로 입장한다. 관중들이 함성을 지르고 응원한다. 선수들이 상대팀과 악수하고 자리로 돌아온다. 코치가 전략을 설명하고 화이팅을 외친다. (Archery players enter the stadium. The crowd shouts and cheers. The players shake hands with the opposing team and return to their seats. The coach explains the strategy and shouts "Go for it.") Ending 1: 활이 과녁에 적중했다. (The arrow hits the target.) Ending 2: 선수가 심호흡을 하고 활을 쏜다. (The player takes a deep breath and shoots an arrow.) Ending 3: 선수가 활을 들어 과녁을 조준한다. (The player raises his bow and aims at the target.) Ending 4: 선수들이 각자 자리에 서서 활을 꺼낸다. (The players stand in their own positions and take out their bows.) Correct Ending: 4
KB-SentiNeg |  Sentence 1: 뚜껑이 잘 열려요! (The lid opens well!) Label 1: 긍정 (Positive) Sentence 2: 뚜껑이 잘 안열려요! (The lid does not open well!) Label 2: 부정 (Negative)
Table 2: Examples of development set from the KoBEST tasks. The variables of
each task are highlighted in bold. Text written in parenthesis is the English
translated version of the original data points.

###  2.2 KoBEST-BoolQ

##### Data/Task Description

We built the KB-BoolQ dataset by referencing boolean questions (BoolQ) task
Clark et al. (2019). A data point consists of a paragraph, question, and
label. The task aims to evaluate models’ understanding of the paragraph by
asking a yes/no question. An example is presented in Table 2.

We extracted paragraphs from Korean
Wikipedia222https://ko.wikipedia.org/wiki/. To cover diverse topics, such as
Science/Technology and Art/Culture, we defined keywords for each topic and
selected documents containing enough information regarding the keyword. Next,
we extracted paragraphs for each document and generated corresponding
questions that could be answered as yes/no based on the paragraph.

##### Guidelines

Annotators were instructed to construct the KB-BoolQ dataset following the
guidelines described below.

  1. 1.

Paragraphs should be evenly extracted from various domains and topics to
minimise bias.

  2. 2.

Questions should be answered only with the information presented in the
paragraph. We set this guide to exclude the impact of pre-trained commonsense
knowledge for decision making. Also, annotators have different viewpoints
regarding the boundary of commonsense knowledge, which can cause uneven task
difficulty.

  3. 3.

Questions should be written in clear, unambiguous, easy-to-understand
language. A true/false judgement should be obvious from a human perspective.

###  2.3 KoBEST-COPA

##### Data/Task Description

We referenced choice of plausible alternatives (COPA) Roemmele et al. (2011)
to construct the KB-COPA dataset. The data has four variables: premise, two
alternatives, and a question that asks a model to decide the cause or effect
of the premise from the two alternatives. An example is available in Table 2.

##### Guidelines

We provided the following guidelines to annotators for generating data
instances.

  1. 1.

The alternatives should belong to a similar area, e.g., states and actions.
This rule is introduced to preclude systems from making decisions based on
situational difference, not the meaning of alternatives.

  2. 2.

The alternatives should contain a keyword related to that of premise. For
instance, in the example presented in Table 2, the keyword of the premise is
“전쟁 (war)”, and both alternatives contain the related same keyword “병사
(soldier)”. We introduce this guideline to increase the task’s difficulty by
making the alternatives belong to the same category.

  3. 3.

All the premises and alternatives should be written concisely so that the
content can be understood intuitively. Therefore, using proper noun, slang,
and redundant expressions should be avoided.

  4. 4.

All sentences should be written in the past tense. In the Korean language,
simple present can cause confusion because it has indication of tense and
sometimes can imply present progressive. On the other hand, the past tense is
morphologically clear and is able to convey meaning without confusion.

  5. 5.

All sentences must include a subject. Although the subject is frequently
omitted in Korean, it is difficult to infer the cause or effect without a
subject because all the premises and alternatives are quite short. Therefore,
even though such sentences are slightly unnatural in Korean, we guide
annotators to insert a subject.

###  2.4 KoBEST-WiC

##### Data/Task Description

KB-WiC is a task that determines whether a word has the same connotation in
different contexts. We referenced words in context (WiC) Pilehvar and Camacho-
Collados (2019) when building the dataset. An instance is composed of a target
homonym and two different contexts that contain the target word. Table 2
provides an example for the KB-WiC task. Unlike the original WiC dataset that
includes various word forms, we only used words with the same form, so as to
focus more on recognising a word’s meaning without the distracton of variouus
forms.

##### Guidelines

To construct the KB-WiC dataset, we instructed annotators to follow these
guidelines.

  1. 1.

A target word should be listed in the National Institute of the Korean
Language Basic Korean Dictionary333https://stdict.korean.go.kr/main/main.do or
Korea University Korean Dictionary Hong and Kim (2009). We exclude words not
registered in the dictionaries because they can cause ambiguous criteria for
determining an answer. This is despite the fact that they are generally used
in daily life.

  2. 2.

For generating a data point where an answer is False, only a homonym should be
used as a target word because a polysemy makes the task considerably more
challenging, even for native speakers.

  3. 3.

The part of speech (PoS) tag of a target word should be a noun, pronoun,
numeral, or dependent noun444A noun that cannot be used without the help of
other words in Korean.. We introduce this guideline because the four PoS tags
have a fixed form and distinct meaning in Korean.

  4. 4.

The contexts should be extracted from example sentences in the dictionaries to
make it possible to clearly understand the sense of a target word only using
the given context.

###  2.5 KoBEST-HellaSwag

##### Data/Task Description

This task evaluates whether a system can utilize passage of time and order to
complete the last sentence in a series of sentences. We referenced the
HellaSwag dataset Zellers et al. (2019) to build our version but modified the
task to consider specific characteristics of the Korean language.

The original HellaSwag benchmark was designed to ascertain whether a LM can
generate a plausible ending sentence given a relevant subject and context. In
Korean, however, subjects are typically omitted. As a result, if the ending
sentence is generated from the subject, the sentence becomes awkward and
barely resembles a plausible Korean sentence. Evaluating such unnatural
sentences is not in line with the purpose of KoBEST, so we modify the task to
predict the most plausible final sentence among four candidates. An example
instance is available in Table 2.

##### Guidelines

We instruct annotators to build the data based on the following guidelines.

  1. 1.

The annotators should generate or modify free-text descriptions of YouTube
videos and Wikepedia documents that progress with the passage of time.

  2. 2.

At least three sentences should be included in the context. A system should
have as much context as possible to generate a plausible ending sentence.

  3. 3.

All the candidate-ending sentences should be thematically related to the
context. The answer should only be able to be found by analysing the passage
of time among the sentences, not via the topic or keywords. This guideline is
introduced to prevent low task difficulty by generating alternative endings
that simply contradict the correct ending.

###  2.6 KoBEST-SentiNeg

##### Data/Task Description

Many studies have revealed that PLMs lack understanding of negation
expressions Hossain et al. (2020); Ettinger (2020); Kassner and Schütze
(2020); Hosseini et al. (2021). Inspired by the Negation capability test of
Ribeiro et al. (2020), we designed a similar but enhanced task by utilizing
negation to create sentences opposite in meaning. Specifically, we created a
two-class sentiment analysis task by generating product reviews based on real
product reviews available on e-commerce websites. 555The real product reviews
are only used as references and our data is newly generated by human
annotators. We then used the training and development sets to train a
sentiment classification model. Next, we extracted candidates from training
data where the polarity switched when transformed into a sentence with the
opposite meaning. Finally, we converted each candidate to a sentence with its
opposite meaning and reversed the label. The modified candidate is then added
to the final test set. We used the following three methods to generate the
sentences with opposite meanings.

  1. 1.

Adding/removing negation expressions: We add or remove Korean negation
expressions (e.g., “안”, “못”, “ 지 않다”).

  2. 2.

Antonym replacement: A word is replaced with its antonym.

  3. 3.

Using both method 1 and 2 or idiom: Both methods described above are used. If
a sentence includes an idiom, we replace it with its corresponding opposite
meaning idiom.

##### Guidelines

We instructed the annotators to comply with the following guidelines to
generate data points for the KB-SentiNeg task.

  1. 1.

The sentence should not include the name of specific brands or products. This
guideline is meant to avoid any possible legal issues.

  2. 2.

To generate a new sentence resembling a real product review, typos and spacing
errors that frequently occur in Korean spoken language should be included
occasionally.

###  2.7 Evaluation Metrics

All of our downstream tasks have discrete labels. Therefore, we use the $F1$
score as a base criterion to evaluate models’ performance.

##  3 Design Principles

In this section, we provide detailed descriptions about how we attempted to
achieve the design principles illustrated in Section 1.

###  3.1 Human-driven data annotation

Automatic data generation using meta-information, such as a review score and
news article category (e.g., Naver Sentiment Movie
Corpus666https://github.com/e9t/nsmc, is a widely used approach to rapidly
collect a large amount of labelled data. While it is an efficient approach,
there exists a high risk of the dataset containing incorrect and ambiguous
data points. Such noisy data points are a major issue for evaluation datasets
because they can lead to spurious performance increases (and or degradations)
in the performance of LMs.

Our data is created purely by human annotations to produce the highest quality
dataset with the lowest amount of incorrect and ambiguous data points
possible. We hired four annotators who are Korean native speakers and major in
Korean Language Education or Korean Language in Literature. Also, our Korean
linguists trained the annotators before the data annotation process to avoid
generating possible grammatical errors and unethical expressions.

###  3.2 Leveraging linguistic knowledge

Korean benchmark datasets, translated from English datasets (e.g., Kornli and
Korsts Ham et al. (2020)), might include incorrect translations and
grammatical errors, particularly if they are machine-translated. Moreover,
since the original examples come from English, such benchmark datasets are
unlikely to assess properly assess Korean-specific knowledge or language
intricacies.

Relying on our in-house Linguistic team, allowed us to mitigate and resolve
issues with automatically generated datasets. First, our linguists trained the
annotators to generate natural and grammatical Korean sentences and performed
thorough reviews of the data. Thanks to their efforts, we have created a
highly grammatical and natural. Secondly, the linguists designed tasks and
data generation processes that considered the Korean language’s
characteristics. This is illustrated in guidelines for each task in Section 2.
Such guidelines and processes enabled us to create an accurate Korean
evaluation dataset with expressive vocabulary and colloquial usage.

###  3.3 Availability to public

Our data is free from copyright issues. All sentences and answers, except for
the paragraphs in KB-BoolQ task, are generated by our annotators from scratch
by referencing publicly available sources. Also, the paragraphs in KB-BoolQ
were extracted from Wikipedia, which is free from copyright issues. Therefore,
researchers are free to use, modify and redistribute the KoBEST dataset.

Example #1
---
P: 날씨가 추워졌다. (The weather has become colder.)
Q: 원인 (Cause)
A1: 겨울이 되었다. (Winter has come.)
A2: 여름이 되었다. (Summer has come.)
Example #2
P: 겨울이 되었다. (Winter has come.)
Q: 결과 (Effect)
A1: 날씨가 추워졌다. (The weather has become colder.)
A2: 날씨가 더워졌다. (The weather became hot.)
Table 3: Examples of the KB-COPA data where the causality is interlocked. P,
Q, A1 and A2 denote a premise, question, and alternatives, respectively. The
two sentences highlighted in red and blue colours are swapped by changing the
question from Cause to Effect. Figure 1: The overall process of model-driven
review for removing artefacts in training data.

###  3.4 High data quality

As a benchmark suite, accomplishing high data quality is important to
accurately evaluate various LMs. We achieve this purpose through two review
phases: human-driven and model-driven reviews.

##### Human-driven review

After collecting data, our linguists reviewed all data instances and found two
major issues in the KB-COPA and KB-HellaSwag datasets.

For the KB-COPA task, we observed many cases with high correlation between
between data instances. Examples corresponding to this case are presented in
Table 3. We conjecture this is because it was easier for the annotators to
collect data by simply swapping the premise and answer along with changing the
question. We removed or modified such instances because they are near
duplicates and harm data diversity.

In the KB-HellaSwag task, we found several critical cases where predicting a
correct final sentence is quite difficult due to the omission of detailed
delineations in the context. This issue occurs because the source that the
annotators referenced for generating contexts lacks detailed information
occasionally. For such ambiguous instances, we appended additional clues to
the context to allow inferring the meaning between the context and the final
sentence.

##### Model-driven review

Artefacts existing in training data can lead a model to learn spurious
inductive biases, resulting in distorted evaluation results Gururangan et al.
(2018); McCoy et al. (2019); Hossain et al. (2020). Therefore, we conduct a
model-driven review process to find and remove such unwanted artefacts. The
overall process is illustrated in Figure 1. First, we trained an ad-hoc model
with the initial dataset for each task. Next, we generated predictions for the
development and test datasets and analysed the results to ascertain whether
specific words/patterns/phrases were strongly correlated with labels. Finally,
our linguists analyzed the issues and updated the datasets accordingly. We
repeated the whole process up to three times for each dataset. Through this
process, we observed serious artefacts, especially in the KB-WiC and KB-COPA
datasets. More than 70% of questions containing number-related representations
had False as a label. Also, the label distributions of data instances
containing specific phrases (e.g., " 덥다/춥다 (hot/cold)" or 했다/하지 않았다 (did/did
not)) were highly skewed towards the False label. All such artefacts were
successfully removed and modified by our linguists.

###  3.5 Avoiding AI ethical issues

Social biases embedded in training data can lead to unethical behaviour of
language models Nangia et al. (2020). To mitigate such issues, we made efforts
to remove unethical expressions, such as toxic content (e.g., insults, slang,
sexual harassment) and social bias (e.g., gender, race, religion). Our
linguists clearly instructed the annotators to avoid unethical expressions
when generating sentences and extracting paragraphs from Wikipedia for the KB-
BoolQ task. Also, linguists reviewed the data for potential ethical issues, as
described in Section 3.4

Model | KB-BoolQ | KB-COPA | KB-WiC | KB-HellaSwag | KB-SentiNeg | Average
---|---|---|---|---|---|---
$\mathcal{F}_{test}$ | $\mathcal{F}_{test}$ | $\mathcal{F}_{test}$ | $\mathcal{F}_{test}$ | $\mathcal{F}_{test}$ | $\mathcal{F}_{test}$
KoBERT (FT) | 62.9$\pm$3.0 | 74.6$\pm$0.8 | 77.3$\pm$0.8 | 74.4$\pm$0.4 | 86.8$\pm$2.0 | 75.2
KoElectra (FT) | 75.1$\pm$1.0 | 81.5$\pm$0.4 | 79.7$\pm$1.8 | 74.7$\pm$0.8 | 91.9$\pm$1.1 | 80.6
KoGPT3-1.2B (FT) | 73.5$\pm$1.6 | 79.3$\pm$0.6 | 68.4$\pm$2.2 | 73.8$\pm$1.0 | 89.5$\pm$3.3 | 77.0
KoBART (FT) | 60.6$\pm$2.9 | 56.9$\pm$3.2 | 60.4$\pm$4.9 | 51.4$\pm$1.3 | 88.6$\pm$1.2 | 63.6
KoGPT3-39B ($k=0$) | 33.1 | 76.8 | 34.7 | 59.8 | 57.7 | 52.8
KoGPT3-39B ($k=1$) | 50.2 | 78.3 | 51.8 | 60.2 | 74.2 | 66.3
KoGPT3-39B ($k=10$) | 46.9 | 80.9 | 52.2 | 58.7 | 91.6 | 70.6
Human | 95.1 | 98.1 | 96.6 | 92.4 | 99.0 | 96.2
Table 4: The performance of Korean LMs on the KoBEST downstream tasks.
$\mathcal{F}_{test}$ refer to the test F1 score. The first and second blocks
are the experimental results of fine-tuning and zero/few-shot learning,
respectively. For the fine-tuning experiments, we repeat each experiment five
times and report the average and standard deviation. The best values for each
task are written in bold.

##  4 Experiments

In this section, we provide model and human performance baseline results.

###  4.1 Fine-tuning Experiments

####  4.1.1 Experimental Design

##### Model Candidates

We used the following four pre-trained Korean language models to benchmark our
KoBEST dataset:

  * •

Encoder models: KoBERT 777https://huggingface.co/monologg/kobert, KoElectra
888https://huggingface.co/monologg/koelectra-base-v2-discriminator

  * •

Decoder models: KoGPT3-1.2B 999https://huggingface.co/skt/ko-gpt-
trinity-1.2B-v0.5

  * •

Encoder-Decoder models: KoBART 101010https://huggingface.co/hyunwoongko/kobart

For KoBART, we applied the text-to-text multi-task training technique to train
the model. We transformed inputs of each text to a free-text input by
referencing the transformation formats used in the T5 model Raffel et al.
(2020).

| BoolQ | COPA | WiC | HellaSwag | SentiNeg
---|---|---|---|---|---
b-size | 8 | 16 | 16 | 8 | 16
s-len | 256 | 128 | 256 | 256 | 128
lr | $5e^{-6}$ | $5e^{-6}$ | $1e^{-5}$ | $2e^{-5}$ | $1e^{-5}$
Table 5: Batch-size (b-size), maximum input length (s-len), and learning rates
(lr) used for the KoBEST benchmark experiments.

##### Training Details

We used AdamW optimiser Loshchilov and Hutter (2017) for training with a
linear learning rate scheduler decaying from 1e-2. We trained all models for
10 epochs, and used the early stopping method during the training. Different
batch size and learning rate were used across the tasks, and detailed training
hyperparamters are presented in Table 5.

####  4.1.2 Results and Discussion

The fine-tuning results can be seen in the first block of Table 4. Overall,
KoElectra showed the best results, followed by KoGPT-1B, suggesting that model
size is not necessarily a requisite for better performance.

Model | $\mathcal{F}_{val}$ | $\mathcal{F}_{test}$ | $\triangle$
---|---|---|---
KoBERT | 99.1 | 86.8 | 12.3
KoElectra | 99.4 | 91.9 | 7.5
KoGPT3-1B | 99.8 | 89.5 | 10.3
KoBART | 98.7 | 88.6 | 10.1
Table 6: The average performance gap between validation and test sets of KB-
SentiNeg task.

##### Do models understand the opposite in meaning?

Table 6 shows the validation and test performance of fine-tuned models on the
SentiNeg task. Interestingly, all models show a large performance gap between
the validation and test performance in the SentiNeg task. The results suggest
that PLMs are vulnerable to a simple negation and antonym-replacement
perturbation, even though the data points all originated from the training
data. Our results are aligned with previous studies on English data that
showed PLMs are incapable of understanding negation expressions Hossain et al.
(2020); Kassner and Schütze (2020); Ettinger (2020); Hosseini et al. (2021),
suggesting that the issue stems from the PLM, not from language itself.

##### Multi-task training is not always beneficial

Unlike the other three models, KoBART was trained in a multi-task fashion.
However, contrary to the common belief that multi-task training is beneficial
in improving performance on benchmark suites (e.g., GLUE Liu et al. (2019)),
in our case, multi-task training produced the worst performance by a large
margin. We conjecture that a leading cause is a misalignment between tasks.
All the downstream tasks in KoBEST are independent of each other. However, in
the GLUE benchmark, for instance, the sub-tasks are well aligned, containing
multiple datasets that share a common objective, e.g., NLI and STS, and it is
well studied that the misalignment between task data can cause poor results Wu
et al. (2020).

###  4.2 Zero/Few Shot Experiments

####  4.2.1 Experimental Design

The advent of extremely large size GLMs like GPT3 Brown et al. (2020) has
allowed in-context learning (providing the model with a few or no samples) to
apply the model to downstream tasks. To this end, we conducted zero-, one- and
few-shot experiments by using a Korean GPT3 model Language Superintelligence
Labs trained with 39 billion parameters and trained with 132 billion tokens.
We then referenced the work of EleutherAI
111111https://github.com/EleutherAI/lm-evaluation-harness to design prompts
for our zero, one, and few-shot experiments. Several prompt examples are
available in Table 7 in the appendix. For multiple-choice problems like KB-
COPA and KB-HellaSwag, we selected the candidate having the lowest perplexity
as the prediction.

####  4.2.2 Results and Discussion

##### Fine-tuned models are still best.

The results, presented in the second block of Table 4, reveal that the fine-
tuned models, apart from KoBART, outperform in-context learning methods. Our
results are aligned with the work of Brown et al. (2020) that showed GPT3
performance based on few-shot learning is behind that of fine-tuned SOTA in
many tasks, including all downstream tasks of SuperGLUE Wang et al. (2019).
Although it is interesting that a large GLM can achieve decent performance
with only a few training examples, results suggest that we should be judicious
using large GLM in practical applications; especially when considering
performance compared to excessively high training costs (i.e., time and
resources).

##### Increasing $k$ is not always beneficial.

Few-shot learning approaches with more examples increase performance in
general, but merely increasing $k$ does not always lead to better performance.
Specifically, in the case of KoBEST, the performance is slightly worse on the
KB-BoolQ and KB-HellaSwag tasks. We believe that the length of the input
document is a leading cause of this phenomenon. Since the model’s max input
length plays a critical role in deciding the maximum number of examples ($n$)
in the prompt Yang et al. (2021), the available $n$ decreases as the length of
prompts increases. However, as we can see in Table 2, the data points of the
KB-BoolQ and KB-HellaSwag tasks have longer inputs than the other tasks. As a
result, the prompts for these tasks become very long and likely to exceed the
model’s maximum input length. This would result in a sliced prompt that may
lack key information the model needs to make a successful prediction.

###  4.3 Human performance

We asked volunteers to evaluate the dataset to provide human-level performance
metrics for KoBEST. Specifically, 10 native Korean evaluators evaluated 100
randomly sampled examples for each downstream task. The results are summarised
in the last row of Table 4. The human evaluators outperformed all the PLMs by
a large margin, suggesting that modern PLMs need further improvements to
achieve human-level language ability.

##  5 Conclusion

A well-designed benchmark dataset is crucial for an objective and precise
evaluation of LMs. Following the GLUE benchmark, more challenging benchmarks
have been proposed as modern LMs become more elaborate and sophisticated.
However, most of these benchmarks only support English or originate from
English (e.g., translation), which hardly captures important characteristics
of a specific language.

To this end, we propose a new Korean benchmark suite named KoBEST, which
consists of five challenging downstream tasks. To overcome the disadvantages
of the previous Korean benchmarks, we focused on 1) employing Korean-specific
knowledge, 2) achieving high data quality and 3) removing superficial cues. To
achieve these goals, we worked with professional Korean linguists and
collected data manually and not automatically. We also conducted human- and
model-driven review processes to eliminate superficial cues from our dataset.
Moreover, we were extra cautious to avoid using unethical expressions.

Finally, we evaluated various PLMs on our new benchmark and provide baseline
model and human performance metrics. Our experimental results show that
current LMs need further improvements to attain human-level language ability.
We hope our new benchmark can contribute to advancements in the Korean NLP
field.

LM

    language model
MLM

    masked language modelling
NLI

    natural language inference
NLP

    natural language processing
NLU

    natural language understanding
STS

    semantic textual similarity
NER

    named entity recognition
RE

    relation extraction
GLM

    generative language model
PLM

    pre-trained language model
KoBEST

     Korean balanced evaluation of significant tasks
BoolQ

    boolean questions
COPA

    choice of plausible alternatives
WiC

    words in context
QA

    question answering
PoS

    part of speech
KLUE

    Korean Language Understanding Evaluation

## Acknowledgements

We would like to express our great appreciation to Sunwoo Lee, Seo-jin Lee,
and Seokyoung Hong for their valuable assistance and feedback.

## References

  * Bender and Koller (2020) Emily M. Bender and Alexander Koller. 2020.  Climbing towards NLU: On meaning, form, and understanding in the age of data.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5185–5198, Online. Association for Computational Linguistics.
  * Bhatt et al. (2021) Shaily Bhatt, Rahul Jain, Sandipan Dandapat, and Sunayana Sitaram. 2021.  A case study of efficacy and challenges in practical human-in-loop evaluation of NLP systems using checklist.  In _Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval)_ , pages 120–130, Online. Association for Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019.  BoolQ: Exploring the surprising difficulty of natural yes/no questions.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2924–2936, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ettinger (2020) Allyson Ettinger. 2020.  What bert is not: Lessons from a new suite of psycholinguistic diagnostics for language models.  _Transactions of the Association for Computational Linguistics_ , 8:34–48.
  * Gururangan et al. (2018) Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R Bowman, and Noah A Smith. 2018.  Annotation artifacts in natural language inference data.  In _NAACL-HLT (2)_.
  * Habernal et al. (2018) Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018.  The argument reasoning comprehension task: Identification and reconstruction of implicit warrants.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1930–1940, New Orleans, Louisiana. Association for Computational Linguistics.
  * Ham et al. (2020) Jiyeon Ham, Yo Joong Choe, Kyubyong Park, Ilji Choi, and Hyungjoon Soh. 2020.  Kornli and korsts: New benchmark datasets for korean natural language understanding.  _arXiv preprint arXiv:2004.03289_.
  * Hong and Kim (2009) IS Hong and HG Kim. 2009.  Korea university korean dictionary.  _Research Institute of Korean Studies, Korea University, Korea_.
  * Hossain et al. (2020) Md Mosharaf Hossain, Venelin Kovatchev, Pranoy Dutta, Tiffany Kao, Elizabeth Wei, and Eduardo Blanco. 2020.  An analysis of natural language inference benchmarks through the lens of negation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 9106–9118, Online. Association for Computational Linguistics.
  * Hosseini et al. (2021) Arian Hosseini, Siva Reddy, Dzmitry Bahdanau, R Devon Hjelm, Alessandro Sordoni, and Aaron Courville. 2021.  Understanding by understanding not: Modeling negation in language models.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1301–1312, Online. Association for Computational Linguistics.
  * Kassner and Schütze (2020) Nora Kassner and Hinrich Schütze. 2020.  Negated and misprimed probes for pretrained language models: Birds can talk, but cannot fly.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7811–7818, Online. Association for Computational Linguistics.
  * Le et al. (2020) Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoit Crabbé, Laurent Besacier, and Didier Schwab. 2020.  FlauBERT: Unsupervised language model pre-training for French.  In _Proceedings of the 12th Language Resources and Evaluation Conference_ , pages 2479–2490, Marseille, France. European Language Resources Association.
  * Liu et al. (2019) Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019.  Multi-task deep neural networks for natural language understanding.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4487–4496, Florence, Italy. Association for Computational Linguistics.
  * Loshchilov and Hutter (2017) I. Loshchilov and F. Hutter. 2017.  Fixing weight decay regularization in adam.  _ArXiv_ , abs/1711.05101.
  * McCoy et al. (2019) Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019.  Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3428–3448, Florence, Italy. Association for Computational Linguistics.
  * Nangia et al. (2020) Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel R. Bowman. 2020.  CrowS-pairs: A challenge dataset for measuring social biases in masked language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1953–1967, Online. Association for Computational Linguistics.
  * Niven and Kao (2019) Timothy Niven and Hung-Yu Kao. 2019.  Probing neural network comprehension of natural language arguments.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4658–4664, Florence, Italy. Association for Computational Linguistics.
  * Park et al. (2021) Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Jiyoon Han, Jangwon Park, Chisung Song, Junseong Kim, Yongsook Song, Taehwan Oh, et al. 2021.  Klue: Korean language understanding evaluation.  _arXiv preprint arXiv:2105.09680_.
  * Pilehvar and Camacho-Collados (2019) Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019.  Wic: the word-in-context dataset for evaluating context-sensitive meaning representations.  In _Proceedings of the 2020 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT)_ , Minneapolis, Minnesota. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research_ , 21:1–67.
  * Ribeiro et al. (2020) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020.  Beyond accuracy: Behavioral testing of NLP models with CheckList.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4902–4912, Online. Association for Computational Linguistics.
  * Roemmele et al. (2011) Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011.  Choice of plausible alternatives: An evaluation of commonsense causal reasoning.  In _2011 AAAI Spring Symposium Series_.
  * Tian et al. (2021) Jidong Tian, Yitian Li, Wenqing Chen, Liqiang Xiao, Hao He, and Yaohui Jin. 2021\.  Diagnosing the first-order logical reasoning ability through LogicNLI.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3738–3747, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019.  Superglue: A stickier benchmark for general-purpose language understanding systems.  _arXiv preprint arXiv:1905.00537_.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Wilie et al. (2020) Bryan Wilie, Karissa Vincentio, Genta Indra Winata, Samuel Cahyawijaya, Xiaohong Li, Zhi Yuan Lim, Sidik Soleman, Rahmad Mahendra, Pascale Fung, Syafri Bahar, and Ayu Purwarianti. 2020.  IndoNLU: Benchmark and resources for evaluating Indonesian natural language understanding.  In _Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing_ , pages 843–857, Suzhou, China. Association for Computational Linguistics.
  * Wu et al. (2020) Sen Wu, Hongyang R Zhang, and Christopher Ré. 2020.  Understanding and improving information transfer in multi-task learning.  In _International Conference on Learning Representations_.
  * Xu et al. (2020) Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, Yin Tian, Qianqian Dong, Weitang Liu, Bo Shi, Yiming Cui, Junyi Li, Jun Zeng, Rongzhao Wang, Weijian Xie, Yanting Li, Yina Patterson, Zuoyu Tian, Yiwen Zhang, He Zhou, Shaoweihua Liu, Zhe Zhao, Qipeng Zhao, Cong Yue, Xinrui Zhang, Zhengliang Yang, Kyle Richardson, and Zhenzhong Lan. 2020.  CLUE: A Chinese language understanding evaluation benchmark.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pages 4762–4772, Barcelona, Spain (Online). International Committee on Computational Linguistics.
  * Yang et al. (2021) Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2021.  An empirical study of gpt-3 for few-shot knowledge-based vqa.  _arXiv preprint arXiv:2109.05014_.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  HellaSwag: Can a machine really finish your sentence? In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4791–4800, Florence, Italy. Association for Computational Linguistics.

##  Appendix A Appendix

KB-BoolQ |  Inputs: Paragraph: 구한말, 통영 안뒤산 기슭 간창골에 김봉제 형제가 살았다. 김봉제는 관약국을 경영하며 부를 누렸는데, 선비적 성품을 지녔던 형과 반대로 막냇동생 김봉룡은 성질이 포악했다. 어느 날 봉룡은 아내였던 숙정을 사모하던 나그네를 살해하였고, 숙정은 누명을 벗으려고 비상을 먹고 자살한다. Question: 봉룡은 숙정을 죽였는가? Answer: False Prmopt Design: Answer: “예” if Answer is True else “아니오” Format: “{Paragraph} 질문: {Question} 답변: {Answer}.” Example: 구한말, 통영 안뒤산 기슭 간창골에 김봉제 형제가 살았다. 김봉제는 관약국을 경영하며 부를 누렸는데, 선비적 성품을 지녔던 형과 반대로 막냇동생 김봉룡은 성질이 포악했다. 어느 날 봉룡은 아내였던 숙정을 사모하던 나그네를 살해하였고, 숙정은 누명을 벗으려고 비상을 먹고 자살한다. 질문: 봉룡은 숙정을 죽였는가? 답변: 아니요.
---|---
KB-COPA |  Inputs: Premise: 전쟁이 시작되었다. Question: 결과 Answer Alternative: 병사들이 전투에 파견되었다. Prompt Design: Connector: “왜냐하면” if Question is “원인” else “그래서” Format: “{Premise} {Connector} {Answer Alternative}” Example: 전쟁이 시작되었다. 그래서 (“왜냐하면” if question is 원인) 병사들이 전투에 파견되었다.
KB-WiC |  Inputs: Context 1: 망가진 엔진은 수리가 불가능하다. Context 2: 이 배는 수리에 들어간 지 일주일이 됐다. Target Word: 수리 Answer: True Prompt Design: Answer: “예” if Answer is True else “아니오” Format: “문장1: {Context1} 문장2: {Context2} 두 문장에서 {Target Word}가 같은 뜻으로 쓰였나? {Answer}” Example: 문장1: 망가진 엔진은 수리가 불가능하다. 문장2: 이 배는 수리에 들어간 지 일주일이 됐다. 두 문장에서 수리가 같은 뜻으로 쓰였나? 예.
KB-HellaSwag |  Inputs: Context: 양궁 선수들이 경기장으로 입장한다. 관중들이 함성을 지르고 응원한다. 선수들이 상대팀과 악수하고 자리로 돌아온다. 코치가 전략을 설명하고 화이팅을 외친다. Correct Ending: 선수들이 각자 자리에 서서 활을 꺼낸다. Prompt Design: 문장: {Context} {Correct Ending} Example: 문장: 양궁 선수들이 경기장으로 입장한다. 관중들이 함성을 지르고 응원한다. 선수들이 상대팀과 악수하고 자리로 돌아온다. 코치가 전략을 설명하고 화이팅을 외친다. 선수들이 각자 자리에 서서 활을 꺼낸다.
KB-SentiNeg |  Inputs: Sentence: 뚜껑이 잘 안열려요! Answer: 부정 Prompt Design: Format: “문장: {Sentence} 긍부정: {Answer}” Example: 문장: 뚜껑이 잘 안열려요! 긍부정: 부정
Table 7: Prompt designs for each task used in in-context learning experiments.
Example data points presented in Table 2 are used.

  *[NLP]: natural language processing
  *[KoBEST]: Korean balanced evaluation of significant tasks
  *[PLMs]: pre-trained language model
  *[NLI]: natural language inference
  *[STS]: semantic textual similarity
  *[NER]: named entity recognition
  *[RE]: relation extraction
  *[GLM]: generative language model
  *[BoolQ]: boolean questions
  *[COPA]: choice of plausible alternatives
  *[WiC]: words in context
  *[PoS]: part of speech
  *[LM]: language model
  *[LMs]: language model
  *[PLM]: pre-trained language model
  *[GLMs]: generative language model


# Interference of the scattered vector light fields from two optically
levitated nanoparticles

Yuanbin Jin  State Key Laboratory of Quantum Optics and Quantum Optics
Devices, Institute of Opto-Electronics, Synergetic Innovation Center of
Extreme Optics, Shanxi University, Taiyuan 030006, P.R.China  Jiangwei Yan
State Key Laboratory of Quantum Optics and Quantum Optics Devices, Institute
of Opto-Electronics, Synergetic Innovation Center of Extreme Optics, Shanxi
University, Taiyuan 030006, P.R.China  Shah Jee Rahman  State Key Laboratory
of Quantum Optics and Quantum Optics Devices, Institute of Opto-Electronics,
Synergetic Innovation Center of Extreme Optics, Shanxi University, Taiyuan
030006, P.R.China  Xudong Yu  State Key Laboratory of Quantum Optics and
Quantum Optics Devices, Institute of Opto-Electronics, Synergetic Innovation
Center of Extreme Optics, Shanxi University, Taiyuan 030006, P.R.China  Jing
Zhang<EMAIL_ADDRESS><EMAIL_ADDRESS>State Key Laboratory of Quantum
Optics and Quantum Optics Devices, Institute of Opto-Electronics, Synergetic
Innovation Center of Extreme Optics, Shanxi University, Taiyuan 030006,
P.R.China

###### Abstract

We experimentally study the interference of dipole scattered light from two
optically levitated nanoparticles in vacuum, which present an environment free
of particle-substrate interactions. We illuminate the two trapped
nanoparticles with a linearly polarized probe beam orthogonal to the
propagation of the trapping laser beams. The scattered light from the
nanoparticles are collected by a high numerical aperture (NA) objective lens
and imaged. The interference fringes from the scattered vector light for the
different dipole orientations in image and Fourier space are observed.
Especially, the interference fringes of two scattered light fields with
polarization vortex show the $\pi$ shift of the interference fringes between
inside and outside the center region of the two nanoparticles in the image
space. As far as we know, this is the first experimental observation of the
interference of scattered vector light fields from two dipoles in free space.
This work also provides a simple and direct method to determine the spatial
scales between optically levitated nanoparticles by the interference fringes.

##  I Introduction

Interference plays an important role in exploring the nature of light, i.e.,
the wave-particle duality [1, 2] and has extensive applications such as in
precise measurement [3, 4, 5, 6]. Interference of light scattered from
particles has received much attention because it has provided information on
the degree of localization of particles in the trap [7, 8, 9, 10]. As the
techniques for trapping atoms and ions developed, the interference of faint
scattered light from the atoms and ions have been reported [7, 8, 9, 10]. The
previous studies usually considered the scalar fields (or fields with the same
polarization). In recent years, the coherence of vector light fields in
quantum and classical regimes have been a new focus [11, 12, 13, 14, 15], in
which the polarization is taken into account in the coherence theorem and
complementarity principle. The Young’s double-slit interference illuminated by
a vector field has been explored [7].

In recent years, the optically levitated micro- and nano-particles in vacuum
have attracted much attention and became an important platform for the precise
measurement and fundamental physics studies [16, 17, 18, 19, 3, 20, 21, 22].
Cooling the center-of-mass motion of the trapped nanoparticle to reach quantum
ground state has been realized recently [23, 24, 25]. The optically levitated
nanoparticle has been used to measure the Casimir force [26], torque [27, 6]
and realize GHz scale rotation [27, 6, 28, 29]. In this system, the micro- and
nano- particles are free from particle-substrate compared to other systems
which have particle-environment interactions [30, 31, 32, 33, 34]. Recently
our group reported the imaging of the dipole scattering orientations of an
optically levitated silica nanoparticle in the image and Fourier space
(k-space) [35]. This system has several distinguished features. First, the
optically levitated nanoparticle provides the possibility to be illuminated
with an excitation laser beam orthogonally to the optical axis of objective
lens. Thus this scheme provides a dark background and high signal-noise ratio
for detecting the dipole scattering. Secondly, vector light field has been
known in the context of classical optics, such as conical diffraction [36] and
propagation through anisotropic crystals [37], singular optics [38],
spin–orbit coupling [39], and more recently quantum states [40, 41]. Vector
light field refers to the light field with spatially varying polarization and
in particular polarization vortex beam refers to the light field which reveal
an azimuthal change of polarization orientation in its transverse plane. In
this dipole scattering system, the scattered vector light fields can be
produced, and in particular polarization vortex field with radial polarization
is created when the dipole orientation of nanoparticle is aligned exactly
along the optical axis of objective lens. Consequently, this system provides
an important platform for studying the dynamical properties and the scattering
anisotropy. Here, the imaging of dipole scattering orientations of an
optically levitated nanoparticle is similar to 3D dipole orientations of a
single-molecule, which have been extensively studied by the polarization
analysis of emitted fluorescence [42, 43, 44], or by the recording of the
defocused [45, 46, 47] or aberrated [48] or k-space [49, 50, 51, 52]
fluorescence image of a molecule. Compared with the single molecule image in
which the excitation light is illuminated confocally along the optical axis of
objective lens, in our work the illuminating light on the nanoparticle is
perpendicular to the optical axis of objective lens.

In this paper, we experimentally investigate the interference of the scattered
vector light fields from two optically levitated nanoparticles by employing
the distinct advantage of this system. The two nanoparticles are trapped in
vacuum using two 1064 nm laser beams with the frequency difference of 110 MHz
and orthogonal polarization, strongly focused by a high NA objective lens. We
shine a 532 nm laser beam on the two trapped nanoparticles in the direction
perpendicular to the propagation of the trapping beams. The scattered 532 nm
light is collected by the same high NA objective lens which is used to trap
nanoparticles, and imaged on a CCD. The interference fringes of the scattered
light in the image and Fourier space are observed. We experimentally
investigate the interference of the scattered vector light fields produced for
the special dipole orientation from two free-space dipoles and show the
distinct interference pattern in intensity and polarization distributions. The
interference fringes are measured through an imaging system and can be used as
a precise ruler to calibrate the magnification of the image system, so
determine the distance between the two nanoparticles.

##  II Experimental setup and theoretical analyzation

###  II.1 Experimental setup

Figure 1: Schematic diagram of the experimental setup and the imaging system
of dipole scattering. The two trapping beams with slight misalignment are
strongly focused by a high NA objective lens (NA=0.95) to simultaneously trap
two silica nanoparticles. The 532 nm laser beam is used to illuminate the
nanoparticles along the orthogonal direction of the trapped beams ($z$-axis).
The scattered light is collected by the same objective lens into a camera. The
dipolar scattering is measured in the image space (solid box) and Fourier
space (dashed box), respectively. The inset is the diagram of spatial angles
of the scattered light. AOM: acousto-optic modulator; PBS: polarization beam
splitter; DBS: dichroic beam splitter; HW1-3: half-wave plate. OB: objective
lens.

Fig. 1 represents the schematic diagram of the trapping nanoparticles and the
imaging system for measuring the scattered light by the nanoparticles. We have
used two 1064 nm laser beams from a diode-pumped single-frequency laser in the
experiment. The frequency of one beam is shifted by a mount of 110 MHz using
an acousto-optic modulator (AOM), and then it is combined with the other laser
beam (the frequency is not shifted) on a PBS to avoid the interference of the
two trapping beams. Then the two beams are tightly focused by a high NA
objective lens (Nikon CF Plan 100X, NA=0.95) in the vacuum system [53, 54].
The directivity of one beam is precisely controlled by a motor-driving mirror.
Thus, the two focus points can be adjusted arbitrarily in the radial direction
and form two potential wells for trapping two nanoparticles. Simultaneously,
the distance of two wells in the axial direction can be changed by adjusting
the positions of a collimating lens of the trapping beams. At last, we
precisely control the positions of the double wells and keep them on the same
object plane of the objective lens.

In the experiment, we utilize silica nanoparticles (Tianjing BaseLine, Unibead
silica nanoparticle, diameters are around 100 nm, refractive index = 1.4),
which have low dispersion. The commercially available hydrosoluble silica
nanoparticles are first dissolved in ultra-pure anhydrous ethanol and then
sonicated for 15 minutes. Next the suspension is diluted and poured into an
ultrasonic nebulizer. Finally, the droplets containing nanoparticles are
dispersed through a narrow tube into the vacuum chamber by the nebulizer. In
order to simultaneously load two nanoparticles, in the beginning we have kept
the separation between the two wells about 5 $\mu$m. When a nanoparticle gets
trapped by one potential well, then we need to slow down the pouring speed to
avoid losing the trapped nanoparticle due to air flow. Once two nanoparticles
are trapped in the two wells separately, we turn on a vacuum pump to evacuate
the chamber. Here, we choose the proper distance (larger than 1.5 $\mu$m) of
the two wells in the radial direction and a proper trapping power (both 150
mW) to avoid the nanoparticles jumping forth and back between the two wells.
We can stably trap nanoparticle and measured at the pressure from 1 to
$10^{5}$ Pa. Here, all measurements are made at the pressure of 1 kPa in the
vacuum chamber.

A linearly polarized 532 nm laser beam focused by a lens ($f=175$ mm)
illuminates the trapped nanoparticles. The direction of linear polarization of
the illuminating light can be adjusted by a half wave-plate (HW3) before the
vacuum chamber. The propagation direction of illuminating beam is along the
$y$-axis orthogonal to the trapping beams direction ($z$-axis). The waist of
the illuminating beam is about 50 $\mu m$ at the nanoparticle position which
is much larger than the distance between the two nanoparticles. The power of
the illuminating beam is about 20 $\mu W$, which is weak enough to neglect the
influence on the motions of the nanoparticles. The size of the nanoparticles
(100 nm diameter) is much smaller than the wavelength of the illuminating
light (532 nm). Therefore the two nanoparticles can be approximated as two
dipoles. The scattered light from the nanoparticles is collected by the same
objective lens, which is used to trap the nanoparticles, and reflected by a
dichroic mirror (which is high reflective for 532 nm and anti-reflective for
1064 nm) and imaged on a high resolution CCD camera (Andor Zyla 4.2 sCOMS,
6.5$\mu m$ pixels, 2048 $\times$ 2048).

###  II.2 Theoretical analyzation

We first consider the case of a single nanoparticle briefly. We approximate
the nanoparticles as dipoles due to their small size (100 nm diameter) in
relation to the wavelength (532 nm) of the illuminating light. In Cartesian
coordinates, the dipole orientation induced by the linear polarization of the
incident laser is denoted by ($\alpha$, $\beta$), in which $\alpha$ is the
zenith angle relative to $z$-axis and $\beta$ is the azimuthal angle in
$xy$-plane relative to $x$-axis. Similarly, the k-vector of the scattered
field is denoted by ($\theta$, $\varphi$). The intensity distribution in the
backaperture plane of the objective lens, which is the Fourier space (k-space)
distribution of the scattered light, can be written as [49, 35]

$\displaystyle{I^{F}}\left({\theta,\varphi;\alpha,\beta}\right)\propto\frac{1}{{\cos\theta}}\left({{{\left|{{\bf{q}}\cdot{{\bf{e}}_{p}}}\right|}^{2}}+{{\left|{{\bf{q}}\cdot{{\bf{e}}_{s}}}\right|}^{2}}}\right),$
(1)

where $\bf{q}$ is the dipole orientation unit vector, i.e.
${\bf{q}}=\left({\sin\alpha\cos\beta,\sin\alpha\sin\beta,\cos\alpha}\right)$.
The two mutual orthogonal unit electric field vectors ${\bf{e}}_{p}$ and
${\bf{e}}_{s}$, which are all orthogonal to the wave k-vector of the scattered
field, are defined as
${{\bf{e}}_{p}}=\left({\cos\theta\cos\varphi,\cos\theta\sin\varphi,\sin\theta}\right)$,
${{\bf{e}}_{s}}=\left({\sin\varphi,-\cos\varphi,0}\right)$. The NA of the
objective lens determines the range of angle $\theta$. The polarization
direction of the scattered light field in the Fourier space can be written as
${\bf{p}}=({\bf{q}}\cdot{{\bf{e}}_{p}}\cos\varphi+{\bf{q}}\cdot{{\bf{e}}_{s}}\sin\varphi,{\bf{q}}\cdot{{\bf{e}}_{p}}\sin\varphi-{\bf{q}}\cdot{{\bf{e}}_{s}}\cos\varphi)$,
which is spatial-dependent and corresponds to the vector light field.

In order to measure the distribution in image space, the scattered light
collected by the objective lens is focused through a lens and measured by a
CCD camera in the focal region. The distribution of the scattered light in the
plane of the dipole is non-paraxial regime of light, however, the observed
distribution in image space is the paraxial light of the dipole determined by
NA of objective lens. The intensity distribution in the focus region of the
lens, which corresponds to the distribution of the scattered light in the
image space, is given by the z component of the Poynting vector, [55, 56]

$\displaystyle{I^{I}}=\frac{c}{{8\pi}}{{\bf{e}}_{z}}\cdot\left({{\bf{E}}\times{\bf{B}}^{*}}\right),$
(2)

the electric field E and the magnetic field B are defined as

$\displaystyle\begin{array}[]{l}{\bf{E}}=\int{\int{m\sqrt{\frac{{\cos\theta^{\prime}}}{{\cos\theta}}}}}\left[{{{\bf{e}}_{p}}^{\prime}\left({{\bf{q}}\cdot{{\bf{e}}_{p}}}\right)+{{\bf{e}}_{s}}\left({{\bf{q}}\cdot{{\bf{e}}_{s}}}\right)}\right]{e^{i{\bf{k^{\prime}}}\cdot{\bf{r^{\prime}}}}}d\Omega\\\
{\bf{B}}=\int{\int{m\sqrt{\frac{{\cos\theta^{\prime}}}{{\cos\theta}}}}}\left[{{{\bf{e}}_{s}}\left({{\bf{q}}\cdot{{\bf{e}}_{p}}}\right)-{{\bf{e}}_{p}}^{\prime}\left({{\bf{q}}\cdot{{\bf{e}}_{s}}}\right)}\right]{e^{i{\bf{k^{\prime}}}\cdot{\bf{r^{\prime}}}}}d\Omega\end{array}$
(5)

where $d\Omega=\sin\theta^{\prime}d\theta^{\prime}d\varphi$, $\bf{k}^{\prime}$
and $\bf{r}^{\prime}$ are the wave vector and the position vector of a target
point in image space, $\theta^{\prime}$ is the intersection angle between
$\bf{k}^{\prime}$ and $z$-axis. The relationship between the $\theta$ and
$\theta^{\prime}$ is given by Abbe’s sine condition,
$\sin\theta=m\sin\theta^{\prime}$, $m$ is the magnification of the imaging
system. The wave vector $\bf{k}^{\prime}$ and unit vector
${{\bf{e}}_{p}}^{\prime}$ can be define as
$\bf{k}^{\prime}=2\pi/\lambda\left({-\sin\theta^{\prime}\cos\varphi,-\sin\theta^{\prime}\sin\varphi,-\cos\theta^{\prime}}\right)$,
${{\bf{e}}_{p}}^{\prime}=\left({\cos\theta^{\prime}\cos\varphi,\cos\theta^{\prime}\sin\varphi,-\sin\theta^{\prime}}\right)$,
respectively. We can theoretically calculate the intensity and polarization
distributions of scattered light of a single nanoparticle in the image and
Fourier space according to the Eq. (1)- (3).

Figure 2: Theoretical calculation of the intensity and polarization
distributions of the scattered light. The patterns of the scattered light of
single nanoparticle in the image space are shown in (a1)-(a4) and the patterns
in the Fourier space are shown in (b1)-(b4) with the polarization of the
illuminating light in $xz$-plane. The interference pattern of the scattered
light of the two nanoparticles in the image space are shown in (c1)-(c4) and
the patterns in the Fourier space are shown in (d1)-(d4) with the different
polarization of the illuminating light. (e1) and (e2) ((e3) and (e4)) are the
intensity and polarization distribution of the partial area (black dashed box)
in (c1) ((c4)). The black lines, red and green ellipses represent the linear
polarization, right and left-handed elliptic polarization, respectively.

Here we consider three cases of the polarization of the illuminating light
along $x$\- and $z$-axis, which correspond to the dipole orientations of Case
1: $(\alpha,\beta)=(-\pi/2,0)$, Case 2: $(\alpha,\beta)=(0,0)$, and Case 3 as
the intermediate cases between case 1 and case 2:
$(\alpha,\beta)=(-\pi/2<\alpha<0$ and $0<\alpha<\pi/2,0)$, respectively. The
intensity and polarization distributions of the scattered light are shown in
Figs. 2(a1)-(a4) and 2(b1)-(b4). The intensity distribution has a symmetric
elliptical profile in the image space (Fig. 2(a1)) and saddle-like shape in
the Fourier space (Fig. 2(b1)) for the case 1. Simultaneously, the
polarization in the image and Fourier space is linear, aligned along $x$-axis.
Case 3 for the arbitrary polarization of the illuminating light is discussed
in our previous work [35]. The dark area in the image and Fourier space will
appear on the right and gradually move towards the center when the
polarization of the illuminating light rotates from $-\pi/2$ to $0$ (Figs.
2(a2)-(b2) for $-\pi/3$ and Figs. 2(a3)-(b3) for $-\pi/6$), and the
polarization is also changed. For the symmetry, the dark area in the image and
Fourier space will gradually move from the center to the left when the
polarization of the illuminating light rotates from $0$ to $\pi/2$. In
contrast, for the case 2, the intensity distribution profile of the scattered
light changes into a donut hole in the image space (Fig. 2(a4)) and Fourier
space (Fig. 2(b4)), while polarization distribution shows a polarization
vortex with radial polarization.

Now we consider the interference of the scattered light by two nanoparticles.
We assume that the two trapped nanoparticles are aligned along $x$-axis, i.e.
$x_{1}=-2\lambda$, $x_{2}=2\lambda$, as shown in Fig. 2(b). We calculate the
intensity distribution of the scattered light interference of the two dipoles
in the Fourier space. The interference intensity distribution of the two
linearly polarized light fields can be written as

$\displaystyle I_{{\mathop{\rm
int}}}^{F}\left({{x^{F}},{y^{F}}}\right)=I_{1}^{F}+I_{2}^{F}+2\sqrt{I_{1}^{F}I_{2}^{F}}\cos\phi\cos\eta,$
(6)

where $I_{j}^{F}$ is the intensity of the scattered light of single
nanoparticle in the Fourier space, $j=1,2$ correspond to the two
nanoparticles, $\phi$ is the phase difference and $\eta$ is the polarization
intersection angle between the two light fields at same point. The $\phi$ and
$\eta$ are spatial dependent in the Fourier space.

In the image space, we can calculate the interference distribution obtained
from Eq. (3). However, the calculation is very complex. Here we adopt the
paraxial approximation for the dipole scattering (regard as two spot light
source) to obtain the simple expressions as below. The interference
distribution in image space is determined only by $E_{x}$ and $E_{y}$
(paraxial approximation; $E_{z}$ negligible). The complex amplitude form of
the electric field at the image space (x,y) can be written as
${{\bf{E}}_{j}}(x,y)=\frac{1}{{{r_{j}}}}{{\bf{p}}_{j}}{e^{ik{r_{j}}}}$, where
$r_{j}$ is the distance between the field (x,y) and the position of the
nanoparticle in the image space. The polarization of the scattered light
fields ${{\bf{p}}_{j}}=\left({{p_{xj}},{p_{yj}}}\right)$ can be resolved into
x and y components. Thus the superposition of the two light fields in the
image space (x,y) can be written as
${E_{x}}=\frac{1}{{{r_{1}}}}{p_{x1}}{e^{ik{r_{1}}}}+\frac{1}{{{r_{2}}}}{p_{x2}}{e^{ik{r_{2}}}}$,
${E_{y}}=\frac{1}{{{r_{1}}}}{p_{y1}}{e^{ik{r_{1}}}}+\frac{1}{{{r_{2}}}}{p_{y2}}{e^{ik{r_{2}}}}$,
where
$\left({{p_{xj}},{p_{yj}}}\right)=\left({\frac{{x-{x_{j}}}}{{{r_{j}}}}\cos\alpha+\sin\alpha,\frac{y}{{{r_{j}}}}\cos\alpha}\right)$.
The intensity and polarization distributions of the interference of two
scattered light fields need to be considered simultaneously. According to the
Stokes parameters,
${S_{0}}={\left|{{E_{x}}}\right|^{2}}+{\left|{{E_{y}}}\right|^{2}}$,
${S_{1}}={\left|{{E_{x}}}\right|^{2}}-{\left|{{E_{y}}}\right|^{2}}$,
${S_{2}}={E_{x}}E_{y}^{*}+{E_{y}}E_{x}^{*}$,
${S_{3}}=i\left({{E_{x}}E_{y}^{*}-{E_{y}}E_{x}^{*}}\right)$, we can calculate
the intensity distribution of interference and give the polarization
distribution in the image space.

We plot the intensity distributions with the interference fringes in the image
and Fourier space for the three cases as shown in Figs. 2(c1)-(c4) and
(d1)-(d4). For case 1, the interference fringes in the image space presents
hyperbolic-like lineshape (Figs. 2(c1)) and in the Fourier space present the
vertical stripes (Figs. 2(d1)) which are superimposed on the original saddle-
like intensity distribution. The polarization distribution in the image space
(Fig. 2(e2)) and Fourier space show the homogeneous linear polarization along
$x$-axis. In comparison, the intensity distributions in the image and Fourier
space for the case 2 are shown in Figs. 2(c4) and (d4), respectively. The
interference fringes in the Fourier space also presents the vertical stripes
which are superimposed on the original doughnut intensity distribution.
However, the interference fringes in the image space are quite different from
the case 1, and show the $\pi$ shift of interference fringes between inside
and outside the circular demarcation line, where the circular demarcation line
is defined as the half of the distance between the two nanoparticles from the
center of the two nanoparticles. On the circular demarcation line, the
polarization of scattered light of two nanoparticles is orthogonal. The
polarization intersection angle is less than $\pi/2$ outside the circular
demarcation line (far region). However, the polarization intersection angle is
more than $\pi/2$ inside the circular demarcation line (center region). So the
polarization of scattered light of two nanoparticles in the center region
almost is aligned the opposite direction, which looks like that a $\pi$ global
phase is introduced in the center region. This induces relative $\pi$ shift of
interference fringes between inside and outside the circular demarcation line.
Therefore, the bright and dark interference fringes in the central region
between the two nanoparticles in the image space are inverse and almost same
in the far region as case 1 compares with the case 2. Moreover, the
polarization distribution in the image space presents the periodic variations
of the ellipticity of polarization as shown in Fig. 2(e4). The interference
fringes for Case 3 are studied as shown in Figs. 2(c2)-(c3) and (d2)-(d3) for
$-\pi/3$ and $-\pi/6$ respectively. The interference fringes in the image and
Fourier space are also superimposed on the corresponding intensity
distribution of the single nanoparticle scattering (Fig. 2(a2)-(b2) and
(a3)-(b3). The bright and dark interference fringes in the central region
between the two nanoparticles for the image space are gradually shifted to
inverse state. These results show the distinct features of the interference
pattern for the scattered vector light fields.

##  III Results

###  III.1 Rotation of the input polarization of illuminating light

Figure 3: The interference image of the scattered light from the two
nanoparticles versus the linear polarization of the illuminating light in the
image and Fourier space. (a1)-(g1) The polarization of the illuminating light
in $xz$-plane. (a2)-(g2) and (a3)-(g3) are the measured results in the image
space ($xy$-plane) and Fourier space ($xy$-plane) in the case of the two
nanoparticles along $x$-axis. (a4)-(g4) and (a5)-(g5) are the measured results
in the image space ($xy$-plane) and Fourier space ($xy$-plane) in the case of
the two nanoparticles along $y$-axis.The line plots below each image of
(a2)-(g2) and (a4)-(g4) is the intensity distribution from cross-sectional
view through the center of the two nanoparticles. The line plot below each
image of (a3)-(g3) and (a5)-(g5) is the intensity distribution from the center
cross-sectional view of the images alone the direction of the two
nanoparticles. A few specks in (a3)-(g3) and (a5)-(g5) is caused by some dusts
on the microscope objective. The Rayleigh diffraction rings centering on the
position of nanoparticles caused by the diffraction of the nanaparticle are
also observed in the images of the image space (a2)-(g2) and (a4)-(g4).

Now we give the experimental results on the interference of the scattered
light of two nanoparticles in the image and Fourier space. Firstly, we discuss
the interference pattern of the light scattered by the two nanoparticles along
the $x$-axis for the different input linear polarization directions of the
illuminating light as shown in Fig. 3. When the linear polarization of the
illuminating light is aligned along the $x$-axis (orthogonal to optical axis
of the objective lens, as shown in Fig. 3(a1)), i.e., $\alpha=-\pi/2$, the
interference pattern in the image space shows several interference fringes,
which present hyperbolic-like lineshape and become shallower close to the
middle as shown in Fig. 3(a2). In parallel, the interference pattern in the
Fourier space is similar with the double-pinhole interference [15, 57] as
shown in Fig. 3(a3), which is consistent with the theoretical simulation of
Fig. 2(d1). When the polarization of the illuminating light is rotated anti-
clockwise in the $xz$-plane by turning the HW3, the images of the scattered
light for every $\pi/6$ radians of the polarization angle are recorded in the
image space (Figs. 3(a2)-(g2)) and Fourier space (Figs. 3(a3)-(g3))
respectively. The total intensity of the scattered light gradually becomes
weaker (stronger) as we scan the polarization from $-\pi/2$ to $0$ ($0$ to
$\pi/2$), which can be seen from the line plots in Fig. 3(a2)-(g2). The bright
and dark interference fringes in the central region in the image space are
gradually shifted to inverse state as shown in Figs. 3(b3) and (c3) for the
polarization values of $-\pi/3$ and $-\pi/6$ and Fig. 3(e3) and (f3) for
$\pi/6$ and $\pi/3$ because of the change of polarization distribution. When
the dipole orientation is aligned along $z$-axis, the scattered light
represents a polarization vortex field with radial polarization (the intensity
distribution profile of scattered light presents a donut hole) and the
interference fringes in the image space show the special feature with the
$\pi$ shift of the interference fringes (Fig. 3(d2)) in the central region,
which is in good agreement with the theoretical prediction.

Then we change the relative positions of two trapped nanoparticles to along
$y$-axis by titling the propagation directions of the two trapping beams to
shift the focus point in the radial direction. Fig3. (a4)-(g4) and (a5)-(g5)
are the measured results in the image and Fourier space in the case of the two
nanoparticles along $y$-axis, respectively. The interference patterns are
similar with the case of the two nanoparticles along $x$-axis. Note that in
our experiment, the intensity of the lower part of the images is stronger than
the intensity of upper part, which means that the scattered light in forward
direction of the illuminating light ($y$-axis negative direction) is a little
stronger than the backward direction ($y$-axis positive direction), which can
be seen in Fig. 3.

###  III.2 Change distance between two nanoparticles

Figure 4: The interference images of scattered light versus the distance
between the two nanoparticles in the image space. (a1)-(d1) and (a2)-(d2) are
the measured results at different distances with the polarization of
illuminating light along $x$\- and $z$-axis respectively. (a3)-(d3) are the
intensity distributions on the connecting line of the two nanoparticles.
(a4)-(d4) are the corresponding partial enlargement (black dashed boxes) of
(a3)-(d3).

Measuring the position of a dipolar scatter in a focus light field was studied
theoretically [58]. Here, we give a new method to determine the exact distance
between the two nanoparticles by using the interference fringes of the
scattered light. In the experiment, we carefully control the distance between
the two nanoparticles and record the interference fringes pattern for the case
1 and 2. The experimental results are plotted in Fig. 4 (Figs. 4(a1)-(d1) for
the case 1 and Figs. 4(a2)-(d2) for the case 2). We can see the different
number of the interference fringes for the different distance between the two
nanoparticles. Furthermore, we plot the intensity distributions at the line
connecting the positions of the two nanoparticles on the images for the two
cases as shown in Figs. 4(a3)-(d3). The red and blue lines correspond to the
case 1 and 2, respectively. Figs. 4(a4)-(d4) are the corresponding partial
enlargement (black dashed boxes) of Figs. 4(a3)-(d3), in which we can clearly
see the bright and dark interference fringes are inverse for the two different
case 1 and 2 due to the $\pi$ shift of the interference fringes. Moreover, we
measure and control the distance between the two nanoparticles. Every pixel of
our camera is 6.5 $\mu$m. The mean distance of the two adjacent bring (or
dark) fringes is 23.85 pixel in the image of our experimental results, which
equal to half of the wavelength of the illuminating light (532/2 nm) in the
actual space. This is a very accurate ruler to determine the spatial distance.
Thus, the magnification of our imaging system is about 583 times. Then we can
control and calibrate the distance between the two nanoparticles by measuring
the distance of the images of two nanoparticles. It can be obtained that the
actual distance of the two nanoparticles are
$d=3.02\lambda,3.97\lambda,5.02\lambda,5.99\lambda$ from Fig. 4(a1)-(d1). The
precision is limited by the size of the camera pixel and the magnification of
the imaging system. It can be reached to 10 nm for our system.

##  IV Conclusion

We have experimentally observed the interference of two scattered vector
fields from two optically levitated nanoparticles in vacuum. We employ the
distinguished features of this system including an environment free of the
particle-substrate interactions, providing a dark background and high signal-
noise ratio for detecting the dipole scattering, producing the scattered
vector light fields. The distinct interference patterns in intensity and
polarization distributions are observed for the different dipole orientations.
Especially, the interference fringes in the image space for two scattered
vector light fields show the $\pi$ shift of the interference fringes in the
central region between the two nanoparticles, in which the bright and dark
interference fringes are inverse and almost same in the far region for the
case 1 and case 2. Furthermore, this work provides a precise tool to measure
the spatial scale of the optically levitated system.

##  Appendix A Appendixes

Intensity distribution of the scattered light of nanoparticle in the image
space. The intensity distribution of the scattered light in the image space is
given by the Eq. (3). The position vector of a target point in focus plane is
${\bf{r^{\prime}}}=\left({\rho\cos\varphi^{\prime},\rho\sin\varphi^{\prime},z^{\prime}}\right)$.
Thus, we can write the x and y components of electric field E as

$\displaystyle\begin{array}[]{l}\left({\begin{array}[]{*{20}{c}}{{E_{x}}}\\\
{{E_{y}}}\end{array}}\right)=\int{\int{m\sqrt{\frac{{\cos\theta^{\prime}}}{{\cos\theta}}}}}{e^{-2i\pi/\lambda\left({\rho\sin\theta^{\prime}\cos\left({\varphi-\varphi^{\prime}}\right)+z^{\prime}\cos\theta^{\prime}}\right)}}\sin\theta^{\prime}d\theta^{\prime}d\varphi\times\\\
\left\\{{\left({\begin{array}[]{*{20}{c}}{\cos\theta^{\prime}\cos\varphi}\\\
{\cos\theta^{\prime}\sin\varphi}\end{array}}\right)\left[{\sin\alpha\cos\theta\cos\left({\varphi-\beta}\right)+\cos\alpha\sin\theta}\right]+\left({\begin{array}[]{*{20}{c}}{\sin\varphi}\\\
{-\cos\varphi}\end{array}}\right)\sin\alpha\sin\left({\varphi-\beta}\right)}\right\\}\end{array},$
(15)

and the x and y components of magnetic field B as

$\displaystyle\begin{array}[]{l}\left({\begin{array}[]{*{20}{c}}{{B_{x}}}\\\
{{B_{y}}}\end{array}}\right)=\int{\int{m\sqrt{\frac{{\cos\theta^{\prime}}}{{\cos\theta}}}}}{e^{-2i\pi/\lambda\left({\rho\sin\theta^{\prime}\cos\left({\varphi-\varphi^{\prime}}\right)+z^{\prime}\cos\theta^{\prime}}\right)}}\sin\theta^{\prime}d\theta^{\prime}d\varphi\times\\\
\left\\{{\left({\begin{array}[]{*{20}{c}}{\sin\varphi}\\\
{-\cos\varphi}\end{array}}\right)\left[{\sin\alpha\cos\theta\cos\left({\varphi-\beta}\right)+\cos\alpha\sin\theta}\right]-\left({\begin{array}[]{*{20}{c}}{\cos\theta^{\prime}\cos\varphi}\\\
{\cos\theta^{\prime}\sin\varphi}\end{array}}\right)\sin\alpha\sin\left({\varphi-\beta}\right)}\right\\}\end{array}.$
(24)

###### Acknowledgements.

This research was supported by the National Natural Science Foundation of
China (Grant Nos. 12034011 and 61975101), the Shanxi “1331 Project” Key
Subjects Construction, and the Xplorer Prize.

## References

  * Wootters and Zurek [1979] W. K. Wootters and W. H. Zurek, Phys. Rev. D 19, 473 (1979).
  * Englert [1996] B.-G. Englert, Phys. Rev. Lett. 77, 2154 (1996).
  * Millen  _et al._ [2014] J. Millen, T. Deesuwan, P. Barker, and J. Anders, Nat. Nanotech. 9, 425 (2014).
  * Zhang  _et al._ [2017] J. Zhang, T. Zhang, and J. Li, Phys. Rev. A 95, 012141 (2017).
  * Hebestreit  _et al._ [2018] E. Hebestreit, M. Frimmer, R. Reimann, and L. Novotny, Phys. Rev. Lett. 121, 063602 (2018).
  * Ahn  _et al._ [2020] J. Ahn, Z. Xu, J. Bang, P. Ju, X. Gao, and T. Li, Nat. Nanotech. 15, 89 (2020).
  * Grangier  _et al._ [1985] P. Grangier, A. Aspect, and J. Vigue, Phys. Rev. Lett. 54, 418 (1985).
  * Eichmann  _et al._ [1993] U. Eichmann, J. C. Bergquist, J. J. Bollinger, J. M. Gilligan, W. M. Itano, D. J. Wineland, and M. G. Raizen, Phys. Rev. Lett. 70, 2359 (1993).
  * Wolf  _et al._ [2016] S. Wolf, J. Wechs, J. von Zanthier, and F. Schmidt-Kaler, Phys. Rev. Lett. 116, 183002 (2016).
  * Araneda  _et al._ [2018] G. Araneda, D. B. Higginbottom, L. Slodička, Y. Colombe, and R. Blatt, Phys. Rev. Lett. 120, 193603 (2018).
  * Gori  _et al._ [2006] F. Gori, M. Santarsiero, and R. Borghi, Opt. Lett. 31, 858 (2006).
  * Eberly  _et al._ [2017] J. H. Eberly, X.-F. Qian, and A. N. Vamivakas, Optica 4, 1113 (2017).
  * Norrman  _et al._ [2017] A. Norrman, K. Blomstedt, T. Setälä, and A. T. Friberg, Phys. Rev. Lett. 119, 040401 (2017).
  * Qian and Agarwal [2020] X.-F. Qian and G. S. Agarwal, Phys. Rev. Research 2, 012031(R) (2020).
  * Norrman  _et al._ [2020] A. Norrman, A. T. Friberg, and G. Leuchs, Optica 7, 93 (2020).
  * Li  _et al._ [2010] T. Li, S. Kheifets, D. Medellin, and M. G. Raizen, Science 328, 1673 (2010).
  * Aspelmeyer  _et al._ [2014] M. Aspelmeyer, T. J. Kippenberg, and F. Marquardt, Rev. Mod. Phys. 86, 1391 (2014).
  * Arndt and Hornberger [2014] M. Arndt and K. Hornberger, Nat. Phys. 10, 271 (2014).
  * Gieseler  _et al._ [2014] J. Gieseler, R. Quidant, C. Dellago, and L. Novotny, Nat. Nanotech. 9, 358 (2014).
  * Ranjit  _et al._ [2015] G. Ranjit, D. P. Atherton, J. H. Stutz, M. Cunningham, and A. A. Geraci, Phys. Rev. A 91, 051805(R) (2015).
  * Jain  _et al._ [2016] V. Jain, J. Gieseler, C. Moritz, C. Dellago, R. Quidant, and L. Novotny, Phys. Rev. Lett. 116, 243601 (2016).
  * Hoang  _et al._ [2018] T. M. Hoang, R. Pan, J. Ahn, J. Bang, H. T. Quan, and T. Li, Phys. Rev. Lett. 120, 080602 (2018).
  * Delić  _et al._ [2020] U. Delić, M. Reisenbauer, K. Dare, D. Grass, V. Vuletić, N. Kiesel, and M. Aspelmeyer, Science 367, 892 (2020).
  * Magrini  _et al._ [2021] L. Magrini, P. Rosenzweig, C. Bach, A. Deutschmann-Olek, S. G. Hofer, S. Hong, N. Kiesel, A. Kugi, and M. Aspelmeyer, Nature 595, 373 (2021).
  * Tebbenjohanns  _et al._ [2021] F. Tebbenjohanns, M. L. Mattana, M. Rossi, M. Frimmer, and L. Novotny, Nature 595, 378 (2021).
  * Xu and Li [2017] Z. Xu and T. Li, Phys. Rev. A 96, 033843 (2017).
  * Ahn  _et al._ [2018] J. Ahn, Z. Xu, J. Bang, Y.-H. Deng, T. M. Hoang, Q. Han, R.-M. Ma, and T. Li, Phys. Rev. Lett. 121, 033603 (2018).
  * Reimann  _et al._ [2018] R. Reimann, M. Doderer, E. Hebestreit, R. Diehl, M. Frimmer, D. Windey, F. Tebbenjohanns, and L. Novotny, Phys. Rev. Lett. 121, 033602 (2018).
  * Jin  _et al._ [2021a] Y. Jin, J. Yan, S. J. Rahman, J. Li, X. Yu, and J. Zhang, Photon. Res. 9, 1344 (2021a).
  * Sick  _et al._ [2000] B. Sick, B. Hecht, and L. Novotny, Phys. Rev. Lett. 85, 4482 (2000).
  * Monici [2005] M. Monici (Elsevier, 2005) pp. 227–256.
  * Dyla  _et al._ [2017] M. Dyla, D. S. Terry, M. Kjaergaard, T. L.-M. Sørensen, J. Lauwring Andersen, J. P. Andersen, C. Rohde Knudsen, R. B. Altman, P. Nissen, and S. C. Blanchard, Nature 551, 346 (2017).
  * Lerner  _et al._ [2018] E. Lerner, T. Cordes, A. Ingargiola, Y. Alhadid, S. Chung, X. Michalet, and S. Weiss, Science 359, eaan1133 (2018).
  * Zhou  _et al._ [2019] H. Zhou, C. Qin, R. Chen, Y. Liu, W. Zhou, G. Zhang, Y. Gao, L. Xiao, and S. Jia, J. Phys. Chem. Lett. 10, 223 (2019).
  * Jin  _et al._ [2021b] Y. Jin, J. Yan, S. J. Rahman, X. Yu, and J. Zhang, Appl. Phys. Lett. 119, 021106 (2021b).
  * Berry and Jeffrey [2007] M. Berry and M. Jeffrey (Elsevier, 2007) pp. 13–50.
  * Oron  _et al._ [2000] R. Oron, S. Blit, N. Davidson, A. A. Friesem, Z. Bomzon, and E. Hasman, Applied Physics Letters 77, 3322 (2000).
  * Soskin and Vasnetsov [2001] M. Soskin and M. Vasnetsov, in  _Progress in Optics_, Progress in Optics, Vol. 42, edited by E. Wolf (Elsevier, 2001) pp. 219–276.
  * Cardano and Marrucci [2015] F. Cardano and L. Marrucci, Nat. Photon. 9, 776 (2015).
  * Spreeuw [1998] R. J. C. Spreeuw, Found. Phy. 28, 361 (1998).
  * Rosales-Guzmán  _et al._ [2018] C. Rosales-Guzmán, B. Ndagano, and A. Forbes, Journal of Optics 20, 123001 (2018).
  * Fourkas [2001] J. T. Fourkas, Opt. Lett. 26, 211 (2001).
  * Zhanghao  _et al._ [2019] K. Zhanghao, X. Chen, W. Liu, M. Li, Y. Liu, Y. Wang, S. Luo, X. Wang, C. Shan, H. Xie, J. Gao, X. Chen, D. Jin, X. Li, Y. Zhang, Q. Dai, and P. Xi, Nat. Commun. 10, 4694 (2019).
  * Lethiec  _et al._ [2014] C. Lethiec, J. Laverdant, H. Vallon, C. Javaux, B. Dubertret, J.-M. Frigerio, C. Schwob, L. Coolen, and A. Maître, Phys. Rev. X 4, 021037 (2014).
  * Sepioł  _et al._ [1997] J. Sepioł, J. Jasny, J. Keller, and U. P. Wild, Chem. Phys. Lett. 273, 444 (1997).
  * Bartko and Dickson [1999a] A. P. Bartko and R. M. Dickson, J. Phys. Chem. B 103, 11237 (1999a).
  * Karedla  _et al._ [2015] N. Karedla, S. C. Stein, D. Hähnel, I. Gregor, A. Chizhik, and J. Enderlein, Phys. Rev. Lett. 115, 173002 (2015).
  * Bartko and Dickson [1999b] A. P. Bartko and R. M. Dickson, J. Phys. Chem. B 103, 3053 (1999b).
  * Lieb  _et al._ [2004] M. A. Lieb, J. M. Zavislan, and L. Novotny, J. Opt. Soc. Am. B 21, 1210 (2004).
  * Dodson  _et al._ [2014] C. M. Dodson, J. A. Kurvits, D. Li, and R. Zia, Opt. Lett. 39, 3927 (2014).
  * Osorio  _et al._ [2015] C. I. Osorio, A. Mohtashami, and A. F. Koenderink, Sci. Rep. 5, 9966 (2015).
  * Curcio  _et al._ [2020] V. Curcio, L. A. Alemán-Castañeda, T. G. Brown, S. Brasselet, and M. A. Alonso, Nat. Commun. 11, 5307 (2020).
  * Jin  _et al._ [2018] Y. Jin, X. Yu, and J. Zhang, Sci. China-Phys., Mech. Astron. 61, 114221 (2018).
  * Jin  _et al._ [2019] Y. Jin, X. Yu, and J. Zhang, J. Opt. Soc. Am. B 36, 2369 (2019).
  * Enderlein [2000] J. Enderlein, Opt. Lett. 25, 634 (2000).
  * Böhmer and Enderlein [2003] M. Böhmer and J. Enderlein, J. Opt. Soc. Am. B 20, 554 (2003).
  * Castañeda  _et al._ [2021] R. Castañeda, P. Bedoya, and G. Matteucci, Physica Scripta 96, 125036 (2021).
  * Tebbenjohanns  _et al._ [2019] F. Tebbenjohanns, M. Frimmer, and L. Novotny, Phys. Rev. A 100, 043821 (2019).