# Broaden the Vision: Geo-Diverse Visual Commonsense Reasoning

Da Yin Liunian Harold Li Ziniu Hu Nanyun Peng Kai-Wei Chang
Computer Science Department, University of California, Los Angeles
<EMAIL_ADDRESS>
gd-vcr.github.io

###### Abstract

Commonsense is defined as the knowledge that is shared by everyone. However,
certain types of commonsense knowledge are correlated with culture and
geographic locations and they are only shared locally. For example, the
scenarios of wedding ceremonies vary across regions due to different customs
influenced by historical and religious factors. Such regional characteristics,
however, are generally omitted in prior work. In this paper, we construct a
Geo-Diverse Visual Commonsense Reasoning dataset (GD-VCR) to test vision-and-
language models’ ability to understand cultural and geo-location-specific
commonsense. In particular, we study two state-of-the-art Vision-and-Language
models, VisualBERT and ViLBERT trained on VCR, a standard multimodal
commonsense benchmark with images primarily from Western regions. We then
evaluate how well the trained models can generalize to answering the questions
in GD-VCR. We find that the performance of both models for non-Western regions
including East Asia, South Asia, and Africa is significantly lower than that
for Western region. We analyze the reasons behind the performance disparity
and find that the performance gap is larger on QA pairs that: 1) are concerned
with culture-related scenarios, e.g., weddings, religious activities, and
festivals; 2) require high-level geo-diverse commonsense reasoning rather than
low-order perception and recognition. Dataset and code are released at
https://github.com/WadeYin9712/GD-VCR.

##  1 Introduction

Commonsense reasoning endows machines with high-level reasoning ability to
understand situations with implicit commonsense knowledge. Suppose that there
is a scene where a woman is wearing a bridal gown at a party. An ideal AI
system with commonsense knowledge should be able to infer that this woman is
attending a wedding and likely to be the bride.

Recently, the field of commonsense reasoning is progressing with the
development of large-scale benchmark datasets Zellers et al. (2018); Talmor et
al. (2019), intended to cover a wide range of commonsense knowledge, such as
physical interactions Bisk et al. (2020), social conventions Sap et al.
(2019), and commonsense grounded in vision Zellers et al. (2019).

Figure 1: Examples in GD-VCR. The three images are all about weddings but from
different regions (left-to-right order: Western, East Asian, South Asian).
Current Vision-and-Language models perform well on answering questions about
Western weddings but often make mistakes when encountering wedding scenarios
in other regions.

However, existing benchmarks are often composed by data from sources in
certain regions (e.g., Western movies) and overlook the differences across
groups in different regions111Due to resource constraints, we use regions as a
proxy to evaluate commonsense among different groups. We note that groups of
individuals in the same region may have different beliefs, cultures, and
behaviors. Please see discussion in the section of Ethical Considerations. due
to factors including cultural differences. In the aforementioned wedding
example, while brides are usually in white in Western weddings, they often
wear red and their faces are covered with a red cloth in traditional Chinese
weddings. If a model is unaware of regional characteristics or incapable of
capturing the nuanced regional characteristics, it leads to a disparity in
performance across different regions Acharya et al. (2020). This motivates us
to study how well a model trained on existing commonsense annotations can
generalize to tasks requiring commonsense beyond Western regions.

In this paper, we mainly focus on regional commonsense with _visual scenes_.
As shown in Figure 1, the three images all describe a wedding but the dresses
of the grooms and brides are different, reflecting the regional
characteristics of the wedding scenario. In this paper, we introduce a new
_evaluation_ benchmark, Geo-Diverse Visual Commonsense Reasoning (GD-VCR),
following the settings of the visual commonsense reasoning (VCR) task Zellers
et al. (2019). VCR consists of multiple-choice questions paired with images
extracted from movies or TV series primarily in _Western_ regions. GD-VCR
includes 328 images, which are mainly sourced from movies and TV series in
East Asian, South Asian, African, and Western countries. The images are paired
with 886 QA pairs, which need to be answered with geo-diverse commonsense and
thorough understanding of the images. An example is given in Figure 1. GD-VCR
benchmark addresses geo-diverse commonsense, such as “What are [person1] and
[person2] participating?”. With the help of these questions, it can manifest
how models behave differently and reveal potential issues with geographical
bias in commonsense reasoning. GD-VCR is one of the first benchmarks to
evaluate model’s reasoning ability on the task which requires geo-specific
commonsense knowledge.

We first study _how well a model trained on VCR can generalize to questions
involving geo-specific commonsense_. Experimenting with two pre-trained
vision-and-language (V&L) models, VisualBERT Li et al. (2019) and ViLBERT Lu
et al. (2019), we observe that two models achieve 64%-68% accuracy over the QA
pairs on images from Western regions, while their accuracy on images from East
Asian region ranges around 45%-51%. The significant performance disparity
suggests that the commonsense learned in these models cannot be generalized
well across different regions.

We further investigate _the reasons why the model exhibits such disparity_
based on the results of VisualBERT. We first find that the performance gap on
the images from Western and non-Western regions is large on the scenarios
involving regional characteristics, such as weddings, religion and festivals.
We also discover that the disparity is related to the reasoning difficulty of
QA pairs. On the QA pairs only requiring basic visual recognition, e.g.,
“What’s [person3] wearing? [person3] is wearing a suit.”, the model achieves
relatively similar performance over the four regions; however, the gap
enlarges when the questions involve higher-level reasoning with commonsense
and rich visual contexts.

By presenting the GD-VCR benchmark, we call upon the researchers to empower AI
systems with geo-diverse commonsense such that they are capable of conducting
high-level reasoning on data from different regions.

##  2 Related Work

##### Commonsense Reasoning Benchmarks.

Recently, there has been an emergence of commonsense reasoning benchmarks
Zellers et al. (2019); Talmor et al. (2019); Sap et al. (2019); Zhou et al.
(2019); Huang et al. (2019); Bhagavatula et al. (2020); Bisk et al. (2020),
which cover a great variety of commonsense knowledge including visual, social,
physical, and temporal commonsense. However, these commonsense benchmarks are
mostly constructed by annotators from certain regions (e.g., the US and UK)
using specific languages (e.g., English). XCOPA Ponti et al. (2020) and X-CSR
Lin et al. (2021) are two multilingual benchmarks, but most samples in both
benchmarks are simply translated from English and cannot reflect the regional
characteristics. Different from previous benchmarks, GD-VCR focuses on geo-
diverse commonsense instead of viewing commonsense as a universal monolith.

##### Vision-and-Language Tasks.

A long line of research seeks to build vision-and-language datasets that test
a model’s ability to understand the visual world and how it is grounded in
natural language. The tasks take on various forms, such as phrase grounding
Kazemzadeh et al. (2014); Plummer et al. (2015), visual question answering
Antol et al. (2015); Goyal et al. (2017), and visual reasoning Zellers et al.
(2019); Suhr et al. (2019). To solve these tasks, a wide range of visual
grounding skills are required. However, in existing tasks, little
consideration is taken into reasoning on the images with regional
characteristics.

##### Geographic Bias.

Geographic bias is a serious issue that may cause harmful effects on certain
groups of people. In computer vision, researchers Shankar et al. (2017); de
Vries et al. (2019) find that most images from two large-scale image datasets,
ImageNet Deng et al. (2009) and OpenImages Krasin et al. (2017), are
amerocentric and eurocentric. When a model trained on these datasets is
applied to images from other regions, the performance will drop drastically.
There also exists geographic bias in language technology. For example, it
underlies natural language processing Blodgett et al. (2016); Jurgens et al.
(2017); Ghosh et al. (2021) and automatic speech recognition Tatman (2017);
Koenecke et al. (2020) models. Our work seeks to reveal and test the
geographic bias in the visual commonsense reasoning task and models.

##  3 Benchmark Construction

To build a geo-diverse visual commonsense reasoning benchmark, we design a
three-stage annotation pipeline, following the original VCR dataset. 1) We
first ask annotators to collect images from movies and TV series in Western,
East Asian, South Asian, and African countries. 2) We request annotators to
design questions and write down the right answers according to the collected
images. 3) We generate answer candidates automatically and formulate multiple-
choice questions. The overview of our pipeline is illustrated in Figure 2. We
elaborate on the three stages in the following.

Figure 2: Overall annotation pipeline. It is divided into three stages: Stage
1 is to collect images with regional characteristics; Stage 2 is to design QA
pairs based on the detected objects; Stage 3 is to generate answer candidates
to complete the dataset with the help of the answer choices in the original
VCR development set.

###  3.1 Image Collection

In the image collection stage, we request annotators to follow two principles:

##### Images with Regional Characteristics.

In our annotation instruction, we require that the collected images should
have representative scenarios containing cultural elements of the annotators’
regions. We further recommend annotators choose scenarios that are ubiquitous
but have specific characteristics across regions, e.g., wedding, funeral,
festival, religion events, etc. For the purpose of analysis, during image
collection, annotators are required to write down keywords on each of their
collected images for categorization. For example, the keywords of the middle
image in Figure 1 are labeled as “wedding, soldier, bride, groom, couple, war,
countryside”.

##### Sources of Images.

Follow the settings of the original VCR dataset, we ask annotators to select
diverse and informative images by taking screenshots from movies, TV series,
documentaries, and movie trailers from websites including
Youtube222www.youtube.com/, BiliBili333www.bilibili.com/,
IQIYI444www.iqiyi.com/, etc. These videos usually include various scenarios
and rich contents containing a large amount of actions and human interactions.
Note that our collected images from Western regions share the same
source555www.youtube.com/c/MOVIECLIPS/videos with those in the original VCR
development set. We use them as a control set in the experiments. Details are
in Appendix A.1.

###  3.2 QA Pair Annotation

We recruit another batch of annotators who are familiar with the culture in
one of the four regions to annotate QA pairs upon the collected images in
English. The annotation stage is divided into two parts: 1) designing
questions according to the image contents; 2) annotating the correct answers
of the questions.

Following the pre-processing of the original VCR dataset, we first apply the
Mask R-CNN object detector666github.com/facebookresearch/detectron2. COCO-
pretrained Mask R-CNN. to mark bounding boxes of objects in each image, and
the annotators can use the labels (e.g., person, car, bowl, etc.) to design QA
pairs.

####  3.2.1 Designing Questions

Annotators are asked to design questions based on the following three
instructions.

##### Usage of the Detected Objects.

Annotators are requested to choose the named objects in the bounding boxes to
construct questions. As shown in Figure 1, annotators can design questions
such as “What is the relationship between [person1] and [person2]?”. This
requirement is aligned with the question design in the original VCR dataset.

##### High-Order Cognitive Questions.

Following the original VCR dataset, we ask annotators to design _high-order_
cognitive questions which require geo-specific commonsense knowledge and
visual understanding to be answered. Take the rightmost image in Figure 1 as
an example. “Why is [person11] so happy?” is a _qualified_ question because
people have to observe the surroundings including [person1] and [person2]’s
wearing and others’ facial expression, and conclude that it is a wedding.
Moreover, [person1] is wearing a wedding dress and others are celebrating for
her. Thus, people can infer that [person11] is happy because it is [person1]
and [person2]’s wedding. Overall, to answer this question, we need to combine
the image context and commonsense knowledge, and reason with multiple turns. A
_disqualified_ example of question is “What is [person3] wearing?” in the left
image of Figure 1. It is defined as a _low-order_ cognitive question because
it can be directly answered by recognizing that the woman is wearing a suit.
This type of question does not need commonsense reasoning based on the context
information.

##### Question Templates.

Since models trained on the original VCR dataset will be evaluated on GD-VCR
dataset, we attempt to eliminate the discrepancy of questions used between the
original VCR dev set and GD-VCR to mitigate the effect of different question
formats. Hence, we ask annotators to design questions by referring to the
templates summarized from the original VCR development set. To generate the
templates, we first replace nouns, verbs, adjectives, and adverbs of the
questions in VCR development set with their POS tags (e.g., NN, VB, JJ, etc.)
labeled by NLTK777www.nltk.org, while keeping question words such as “what”,
“why” and “how”, and auxiliary verbs like “is”, “do” and “have”. In this way,
we remove the terms associated with specific questions, while keeping the
general patterns. We then apply K-Means (MacQueen et al., 1967) algorithm888We
concatenate the question words and the POS tags of all the other words (e.g.,
from “What is [person1] doing?” to “What VBZ NNP VBG?”). Then we use sentence
representations of the converted sentences as real-valued vectors in K-Means.
The representations are obtained from Sentence-Transformers Reimers and
Gurevych (2019) based on RoBERTa-base Liu et al. (2019). to the question
patterns, and manually summarize 17 templates, e.g., “What (n.) is sb. v.+ing
sth.?”, “What is sb.’s relationship with sb.?”. Details of the clustering
method and the list of question templates are in Appendix A.2.

####  3.2.2 Annotating Correct Answers

Annotators are required to write down _only_ the _right_ answer for the
questions they designed. This is to reduce annotation cost and avoid potential
annotation artifacts Zellers et al. (2019). We require that the right answers
to the questions should be consistent with the image contents. However, we
remind annotators to avoid writing answers that are too specific to the video
plots because the answers should be inferred without prior knowledge about the
plots. In addition, instead of writing named entities or proper names specific
to one region, annotators are required to use common expressions in English.
These instructions would help us maintain the difficulty of GD-VCR to a proper
extent. Finally, we invite 3 annotators from each QA pair’s corresponding
regions to validate the correctness of each question and its answer. If 2 of
them have an agreement to approve a certain QA pair, we keep it in the final
dataset.

###  3.3 Answer Candidate Generation

In this stage, for each question, we generate three wrong answer candidates
(i.e., wrong answer choices), to construct a 4-way multiple-choice QA example.
We follow the answer candidate generation algorithm in VCR Zellers et al.
(2019):

##### Answer Candidate Pool.

Instead of generating answer candidates from scratch by language models, we
leverage the right choices in the original VCR development set, and treat them
as an answer candidate pool. All the answer candidates of GD-VCR are derived
from this pool.

##### Answer Candidate Selection.

The principles for selecting answer candidates from the pool are two-fold: 1)
answer candidates should be relevant to questions; 2) they should be
dissimilar with the right choice and other selected answer candidates, so that
they would not be the right answer incidentally. Details of the candidate
selection algorithm are in Appendix 1.

Datasets | # Images | # QA Pairs | Avg. Len. of Ques. | Avg. Len. of Ans. | Avg. # Obj. | Avg. # Relevant Obj. | OOV Rate
---|---|---|---|---|---|---|---
Original VCR | 9929 | 26534 | 6.77 | 7.67 | 10.34 | 2.39 | 12.70%
GD-VCR | 328 | 886 | 7.38 | 7.68 | 10.18 | 2.38 | 6.75%
$\circ$ West | 100 | 275 | 7.36 | 7.19 | 11.10 | 2.28 | 3.44%
$\circ$ East Asia | 101 | 282 | 7.59 | 7.59 | 9.57 | 2.42 | 4.50%
$\circ$ South Asia | 87 | 221 | 6.85 | 8.00 | 10.29 | 2.12 | 5.49%
$\circ$ Africa | 40 | 108 | 7.98 | 8.54 | 9.29 | 3.03 | 7.34%

Table 1: Statistics of the GD-VCR benchmark. The top half of the table is the
overall statistics of GD-VCR and the original VCR development set. The bottom
half includes the subsets of each region in GD-VCR.

###  3.4 Dataset Statistics

Table 1 summarizes the statistics of the GD-VCR benchmark and the original VCR
development set.

##### Texts.

We observe that the average lengths of questions and answers in GD-VCR are
similar to those in the original VCR development set. Aside from the text
lengths, we also consider out-of-vocabulary (OOV) rate with respect to the
original VCR training set. This indicates how much unseen knowledge (e.g.,
entities specific to certain region) are involved in GD-VCR. As shown in Table
1, we find that the OOV rate of the entire GD-VCR dataset is 6.75%, while that
of the original VCR development set is 12.70%. This shows that GD-VCR has a
similar distribution of the vocabulary with the original VCR dataset and the
difficulty of GD-VCR does not come from the vocabulary gap.

##### Images.

The average number of the detected objects 10.18 is similar to that of the
original VCR development set. Moreover, since the objects mentioned in
questions and answers are directly relevant to the reasoning process on each
QA pair, we consider statistics of the average number of the relevant objects.
The average number of relevant objects in image in GD-VCR is 2.38, which
nearly equals that of the VCR development set.

##  4 Model Performance and Human Evaluation over GD-VCR

We are interested in the following questions: 1) Can a model trained on the
original VCR dataset (mostly Western scenarios) generalize well to solve
reasoning questions require commonsense specific to other regions? 2) Do
humans show similar trend when dealing with questions require regional
commonsense that they are not familiar with?

Our experiments are conducted with two Vision-and-Language models VisualBERT
Li et al. (2019) and ViLBERT Lu et al. (2019). We fine-tuned the two pre-
trained models on the original VCR training set and evaluate them on GD-VCR.
All the experimental results are the average of 3 runs with different seeds.
Implementation details are listed in Appendix C.

Datasets | Human | Text-only BERT | VisualBERT | ViLBERT
---|---|---|---|---
Acc. | Gap (West) | Acc. | Gap (West)
Original VCR | - | 53.8∗ | 70.10 | +5.73 | 69.84 | +2.57
GD-VCR | 88.84 | 35.33 | 53.95 | -10.42 | 59.99 | -7.28
$\circ$ West | 91.23 | 37.09 | 64.37 | 0.00 | 67.27 | 0.00
$\circ$ South Asia | 92.98 | 33.48 | 54.90 | -9.43 | 63.57 | -3.70
$\circ$ Africa | 87.93 | 34.26 | 47.53 | -16.84 | 59.73 | -7.54
$\circ$ East Asia | 83.05 | 35.46 | 45.51 | -18.86 | 50.18 | -17.09

Table 2: Accuracy (%) on the subset of each region in GD-VCR and the original
VCR development set. With regard to Western regions, two models’ performance
gap of the original VCR development set and other regions is shown. We also
report human’s accuracy (%) over each region subset. Annotators are from
United Kingdom and United States according to MTurk. ∗ denotes the reported
result in Zellers et al. (2019).

###  4.1 Model Performance

We apply the two models on GD-VCR benchmark to study how well the two models
can generalize. Results are shown in Table 2. Key observations are summarized
as follows:

##### Western vs. Non-Western Regions.

We find that the models perform significantly worse on the images from non-
Western regions. According to VisualBERT’s results, we observe that the gap
between Western and South Asian regions is 9.43%, while it greatly amplifies
to 16.84% and 18.86% when it comes to the comparison with African and East
Asian regions, respectively. These results reflect significant differences of
models’ reasoning ability on the examples from different regions.

##### VisualBERT vs. ViLBERT.

We find that ViLBERT outperforms VisualBERT by 6.04% on GD-VCR. We conjecture
that the higher performance of ViLBERT partly results from the pre-training
data: VisualBERT is pre-trained on COCO Captions which includes 80K images
Chen et al. (2015), while ViLBERT’s pre-training data are from Conceptual
Captions containing 3.3M images Sharma et al. (2018). The larger coverage of
image contents may help ViLBERT generalize to the images with regional
characteristics. It is also shown that the performance gap over the images
from Western and non-Western regions shrinks when applying ViLBERT. However,
the gap is still significant, ranging from 3.70% to 17.09%.

##### Western v.s. Original VCR Dataset.

We observe a performance gap around 2%-6% between images from Western and the
original VCR dataset. We speculate that the gap is caused by one main aspect:
the requirements in the image collection stage are slightly different. We
expect to collect images containing regional characteristics, including
cultural elements like customs. It may add to the complexity of the reasoning
process as cultural commonsense is needed. However, the gap is much smaller
compared with the gap between Western and other regions.

###  4.2 Human Evaluation

Apart from the model performance, we investigate how well human beings perform
on GD-VCR. We randomly select 40 QA pairs of each region, and there are 160 QA
pairs in total for evaluation. We recruit qualified annotators living in
United Kingdom and United States from MTurk999The annotators should complete
at least 1000 HITs, with an approval rate above 95%. to accomplish the
evaluation. Assuming them to be familiar with Western culture, we are
interested to see their performance on the examples from other regions.

Human evaluation results are shown in Table 2. We notice that human
performance is much better than models. More importantly, we observe that the
performance gap among regions is much smaller than that of two V&L models. For
example, annotators from Western can achieve 87.93% accuracy on East Asian
images, and the gap reduces to 8.18% from 18.86% and 17.09%. It implies that
human beings are more capable of applying their commonsense knowledge and
transferring it to the comprehension in geo-diverse settings, while models are
still far away from this level.

##  5 Analyses of Performance Disparity

As we observe large performance gaps between Western and non-Western data in
Section 4.1, in this section, we inspect the pre-trained V&L model to analyze
the reasons behind such performance disparity on two aspects, 1) _regional
differences of scenarios_ and 2) _reasoning level of QA pairs_. We analyze the
_VisualBERT_ model, since its performance gap is more evident.

###  5.1 Regional Differences of Scenarios

As shown in Figure 1, even the same scenarios such as wedding can take
different visual forms across regions. Motivated by this, we investigate how
large the performance gap is when we apply VisualBERT to the images of the
same scenarios happening in different regions.

We select the scenarios that frequently appear in the annotated keyword set of
GD-VCR. Specifically, we choose the scenarios which appear at least 10 times
in _not only_ Western images, _but also_ the images from any of the other
regions. We visualize each scenario’s performance gap between the images from
Western and non-Western regions in Figure 3. The scenarios whose gap is above
8% are colored in red; otherwise, they are labeled by blue.

Figure 3: Visualization of the performance gap on images of the same scenarios
in Western and non-Western regions. The larger the characters, the larger the
performance gap over the scenarios. The red and blue words are the scenarios
whose performance gap is _above_ and _below_ 8%, respectively. Detailed
accuracy on these scenarios is shown in Appendix D.

Regions | Low-order | High-order | $|Low-High|$
---|---|---|---
Acc. ($Low$) | Gap (West) | Acc. ($High$) | Gap (West)
West | 65.15 | 0.00 | 66.60 | 0.00 | 1.45
South Asia | 54.37 | -10.78 | 52.37 | -14.23 | 2.00
East Asia | 58.74 | -6.41 | 50.47 | -16.13 | 8.27
Africa | 56.06 | -9.09 | 40.35 | -26.25 | 15.71

Table 3: VisualBERT’s accuracy (%) on low-order and high-order cognitive QA
pairs. “Gap (West)” denotes performance gap over the QA pairs of images from
Western and non-Western regions. “$|Low-High|$” denotes the performance gap
between low-order and high-order cognitive QA pairs from the same regions.

As shown in Figure 3, we find that on the scenarios which often contain
regional characteristics (e.g., wedding, religion, festival), the performance
gap is much larger, from 8.28% to 23.69%. One interesting finding is that,
aside from festival, wedding and religion, which are generally considered to
be different across regions, the gap is considerably large over the scenarios
involving customers. We speculate that it is also due to regional
characteristics. As shown in Figure 5 of Appendix F, in East Asia and South
Asia, many customers would buy things from the local merchants along the
streets, while in Western regions, customers typically shop in supermarkets
and restaurants. The visual discrepancy may result in errors on the “customer”
scenarios in GD-VCR.

On the other hand, for the scenarios such as party, restaurant and student,
the gap is only 0.42%, 1.29% and 1.12%, respectively. We notice that these
scenarios are more similar across regions. For example, parties are related to
actions like drinking, dancing, and celebration, which are common and take on
similar visual forms across regions. Such similarity may contribute to model’s
high transfer performance on “party”.

###  5.2 Reasoning Level of QA Pairs

The QA pairs in GD-VCR are _high-order_ cognitive QA pairs, which require
several reasoning steps to be solved. For example, to infer that “[person1]
and [person2] are in a wedding” in the middle image of Figure 1, human beings
must first recognize basic facts such as [person1] is wearing in red and her
face is covered by a red cloth. Only by combining the recognized facts and
regional commonsense can human make correct predictions. Therefore, the
model’s failure on these high-order cognitive QA pairs from non-Western
regions may be attributed to two reasons: 1) the model fails to recognize the
basic facts from the image, 2) or the model succeeds on the basic facts but
fails eventually due to lack of geo-specific commonsense.

To determine at which stage the model fails to generalize, we aim to answer
the following two questions: Q1. _Can the model perform similarly on
recognizing basic visual information in the images from different regions?_
Q2. _Is the performance disparity attributed to the failure of understanding
more advanced or basic visual information?_

According to the standard of reasoning level discrimination mentioned in
Section 3.2.1, we categorize QA pairs into two types: _low-order_ and _high-
order_ cognitive QA pairs. _Low-order_ cognitive QA pairs correspond to the
inquiry on basic visual information, while _high-order_ QA pairs involve more
advanced information. Our analysis is mainly concerned with the two types of
QA pairs.

##### Q1. Can model perform similarly on understanding basic visual
information across regions?

We evaluate model’s performance on _low-order_ cognitive QA pairs to analyze
this aspect.

As mentioned in Section 3.2.1, GD-VCR is composed of high-order cognitive QA
pairs but without low-order pairs. Therefore, we additionally annotate low-
order cognitive QA pairs on the images of GD-VCR. Specifically, we randomly
select 30 images per region and design low-order QA pairs based on these
selected images. Finally, we collect 22 QA pairs on Western images, 26 on East
Asian images, 16 on South Asian images, and 22 on African images.

Results are shown in Table 3. We observe that the performance over the low-
order cognitive QA pairs is all around 60% for the four regions. Performance
over Western images is still the highest among the four regions. But note that
the performance gap between the images from Western and non-Western regions is
not so large as the overall gap shown in Table 2. For example, the overall
performance gap between East Asia and Western is around 19%, but it decreases
to 6.41% when the model deals with simpler situations. It demonstrates that,
when encountering the QA pairs focusing on simple recognition, VisualBERT can
narrow down the gap on the images from different regions. In other words,
VisualBERT shows more similar ability to process basic visual information, no
matter where the images are from.

##### Q2. Is the performance disparity attributed to understanding on more
advanced or basic visual information?

We analyze the performance over _low-order_ and _high-order_ cognitive QA
pairs. For a fair comparison, both types of QA pairs share _the same images_.

Results are shown in Table 3. We observe that VisualBERT’s performance over
low-order cognitive QA pairs is higher than that over high-order QA pairs on
images from East Asia, South Asia, and Africa. Especially, on the images from
African regions, the performance gap between these two types of QA pairs is
15.71%.

Furthermore, from Table 3, we notice that the performance gap between Western
and non-Western regions on high-order cognitive QA pairs is much larger than
that on low-order QA pairs. For the images from East Asian regions, the
performance gap with regard to Western regions on low-order pairs is 6.41%.
The gap amplifies to 16.13% when VisualBERT is applied to high-order QA pairs.
For African images, the gap changes rapidly from 9.09% to 26.25%. These
results show that VisualBERT trained on VCR lacks the ability to perform
complex reasoning on the scenarios in non-Western regions. We hope our
findings could inspire future work to model high-level reasoning process
better with geo-diverse commonsense knowledge in commonsense reasoning tasks.

##  6 Conclusion

We propose a new benchmark, GD-VCR, for evaluating V&L models’ reasoning
ability on the QA pairs involving geo-diverse commonsense knowledge.
Experimental results show that the V&L models cannot generalize well to the
images regarding the regional characteristics of non-Western regions. Based on
VisualBERT’s results, we find that 1) the scenarios such as wedding, religion
and festival, which require geo-diverse commonsense knowledge to be
understood, and 2) the reasoning difficulty of QA pairs are highly associated
with the performance disparity. For broader impact, we hope that the GD-VCR
benchmark could broaden researchers’ vision on the scope of commonsense
reasoning field and motivate researchers to build better commonsense reasoning
systems with more inclusive consideration.

## Acknowledgement

We thank Xiao Liu, Ming Zhong, Te-Lin Wu, Masoud Monajatipoor, Nuan Wen, and
other members of UCLANLP and UCLA PlusLab groups for their helpful comments.
We also greatly appreciate the help of anonymous annotators for their effort
into constructing the benchmark. This work was partially supported by DARPA
MCS program under Cooperative Agreement N66001-19-2-4032. The views expressed
are those of the authors and do not reflect the official policy or position of
the Department of Defense or the U.S. Government.

## Ethical Considerations

In this work, we propose a geo-diverse visual commonsense reasoning dataset
GD-VCR. Since the paper introduces new dataset, we discuss the potential
ethical issues about data collection.

##### Intellectual Property and Privacy Rights.

We ensure that intellectual property and privacy rights of the original
authors of videos and recruited annotators are respected during the dataset
construction process with permission of licence101010Fair use on YouTube.
support.google.com/youtube/answer/9783148?hl=en111111Copyright Law of the
People’s Republic of China (Article 22).
http://www.gov.cn/flfg/2010-02/26/content_1544458.htm.. We also claim that the
collected data would not be used commercially.

##### Compensation for Annotators.

We recruit annotators from Amazon Mechanical Turk platform121212www.mturk.com
and college departments of foreign languages and culture. In image collection
stage, we paid annotators $0.5-0.7 per collected image. In QA pair annotation
stage, the payment is $0.2 per QA pair. For validation and human evaluation,
we pay them $0.02-0.03 per QA pair. The pay rate is determined by a
preliminary annotation trial to ensure the average hourly pay rate is around
$12 per hour. The annotations on the images from East Asian regions are partly
done by the authors of this work.

##### Potential Problems.

Although we have considered the potential geographic bias in the benchmark
construction process, GD-VCR may still contain unwanted bias. First, due to
the resource constraints, GD-VCR dataset is unable to cover diverse regional
characteristics at once. For instance, we do not take Southeast Asian, Arabic
and Latin American regions into account. Moreover, even groups in the same
region may have different beliefs. For the regions like Africa, the regional
differences between West Africa, East Africa, and North Africa are evident.
However, in GD-VCR, the images from Africa are mainly sourced from East
Africa. It inevitably introduces geographic bias into our benchmark. More
fine-grained analysis should be conducted to scale up this study, especially
before the visual commonsense reasoning model is used in the commercial
product.

## References

  * Acharya et al. (2020) A. Acharya, Kartik Talamadupula, and Mark A. Finlayson. 2020.  An Atlas of Cultural Commonsense for Machine Reasoning.  _ArXiv_ , abs/2009.05664.
  * Antol et al. (2015) Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015.  VQA: Visual Question Answering.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2425–2433.
  * Bhagavatula et al. (2020) Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen tau Yih, and Yejin Choi. 2020.  Abductive Commonsense Reasoning.  In _International Conference on Learning Representations_.
  * Bisk et al. (2020) Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020.  PIQA: Reasoning about Physical Commonsense in Natural Language.  In _Thirty-Fourth AAAI Conference on Artificial Intelligence_.
  * Blodgett et al. (2016) Su Lin Blodgett, Lisa Green, and Brendan O’Connor. 2016.  Demographic Dialectal Variation in Social Media: A Case Study of African-American English.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1119–1130, Austin, Texas. Association for Computational Linguistics.
  * Chen et al. (2015) Xinlei Chen, H. Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. L. Zitnick. 2015.  Microsoft COCO Captions: Data Collection and Evaluation Server.  _ArXiv_ , abs/1504.00325.
  * de Vries et al. (2019) Terrance de Vries, Ishan Misra, Changhan Wang, and Laurens van der Maaten. 2019\.  Does Object Recognition Work for Everyone? In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , pages 52–59.
  * Deng et al. (2009) J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. 2009.  ImageNet: A Large-Scale Hierarchical Image Database.  In _CVPR09_.
  * Ghosh et al. (2021) Sayan Ghosh, Dylan Baker, David Jurgens, and Vinodkumar Prabhakaran. 2021.  Cross-geographic Bias Detection in Toxicity Modeling.  _NAACL 2021_.
  * Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017\.  Making the V in VQA matter: Elevating the Role of Image Understanding in Visual Question Answering.  In _CVPR_.
  * Huang et al. (2019) Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019.  Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2391–2401, Hong Kong, China. Association for Computational Linguistics.
  * Jurgens et al. (2017) David Jurgens, Yulia Tsvetkov, and Dan Jurafsky. 2017.  Incorporating Dialectal Variability for Socially Equitable Language Identification.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 51–57, Vancouver, Canada. Association for Computational Linguistics.
  * Kazemzadeh et al. (2014) Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014.  ReferItGame: Referring to Objects in Photographs of Natural Scenes.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 787–798, Doha, Qatar. Association for Computational Linguistics.
  * Koenecke et al. (2020) Allison Koenecke, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R Rickford, Dan Jurafsky, and Sharad Goel. 2020.  Racial Disparities in Automated Speech Recognition.  _Proceedings of the National Academy of Sciences_ , 117(14):7684–7689.
  * Krasin et al. (2017) Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng, Dhyanesh Narayanan, and Kevin Murphy. 2017.  OpenImages: A Public Dataset for Large-scale Multi-label and Multi-class Image Classification. _Dataset available from https://github.com/openimages_.
  * Li et al. (2019) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019\.  VisualBERT: A Simple and Performant Baseline for Vision and Language.  _ArXiv_.
  * Lin et al. (2021) Bill Yuchen Lin, Seyeon Lee, Xiaoyang Qiao, and Xiang Ren. 2021.  Common Sense Beyond English: Evaluating and Improving Multilingual Language Models for Commonsense Reasoning.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1274–1287, Online. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _CoRR_ , abs/1907.11692.
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019.  ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks.  In _NeurIPS_.
  * MacQueen et al. (1967) James MacQueen et al. 1967.  Some Methods for Classification and Analysis of Multivariate Observations.  In _Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability_ , volume 1, pages 281–297. Oakland, CA, USA.
  * Plummer et al. (2015) Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015.  Flickr30k Entities: Collecting Region-to-phrase Correspondences for Richer Image-to-sentence Models.  In _Proceedings of the IEEE international conference on computer vision_ , pages 2641–2649.
  * Ponti et al. (2020) Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020.  XCOPA: A Multilingual Dataset for Causal Commonsense Reasoning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2362–2376, Online. Association for Computational Linguistics.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
  * Sap et al. (2019) Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019.  Social IQa: Commonsense Reasoning about Social Interactions.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4463–4473, Hong Kong, China. Association for Computational Linguistics.
  * Shankar et al. (2017) Shreya Shankar, Yoni Halpern, Eric Breck, James Atwood, Jimbo Wilson, and D Sculley. 2017.  No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World.  _arXiv preprint arXiv:1711.08536_.
  * Sharma et al. (2018) Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018.  Conceptual Captions: A Cleaned, Hypernymed, Image Alt-text Dataset For Automatic Image Captioning.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2556–2565, Melbourne, Australia. Association for Computational Linguistics.
  * Suhr et al. (2019) Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019\.  A Corpus for Reasoning about Natural Language Grounded in Photographs.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6418–6428, Florence, Italy. Association for Computational Linguistics.
  * Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019.  CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Tatman (2017) Rachael Tatman. 2017.  Gender and Dialect Bias in YouTube’s Automatic Captions.  In _Proceedings of the First ACL Workshop on Ethics in Natural Language Processing_ , pages 53–59, Valencia, Spain. Association for Computational Linguistics.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-Art Natural Language Processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Zellers et al. (2019) Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  From Recognition to Cognition: Visual Commonsense Reasoning.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Zellers et al. (2018) Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018.  SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 93–104, Brussels, Belgium. Association for Computational Linguistics.
  * Zhou et al. (2019) Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019.  “Going on a vacation” takes longer than “Going for a walk”: A Study of Temporal Commonsense Understanding.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3363–3369, Hong Kong, China. Association for Computational Linguistics.

##  Appendix A Additional Details of Annotation Pipeline

###  A.1 Image Collection

In addition to the requirements mentioned in Section 3.1, we have additional
requirements on the contents, quality, and sources of images. The image should
have at least two people, and should not be grainy and blurry. We require
annotators to choose movies, TV series, documentaries and trailers which are
free to access and do not have copyright issues. Together with the images and
their keywords, we also collect video names, screenshot timestamps, and the
links of videos. It is to help the annotators in later stages better
understand the image contents with video contexts.

Regions | Wedding | Festival | Religion | Bride | Groom | Restaurant | Family | Student | Customer | Party
---|---|---|---|---|---|---|---|---|---|---
West | 62.22 | 68.89 | 58.33 | 66.67 | 69.14 | 61.90 | 59.26 | 61.54 | 66.67 | 55.83
Other Regions | 50.00 | 60.61 | 46.21 | 52.78 | 45.45 | 60.61 | 47.27 | 60.42 | 44.44 | 56.25

Table 4: Accuracy (%) on the images involving the same scenarios from
different regions.

Question Templates
---
1\. What did sb. (just) v.?
2\. What did sb do when/before/as CLAUSE?
3\. What (n.) is sb. v.+ing prep. PHRASE?
4\. What (n.) is sb. v.+ing?
5\. What is sb.’s job/occupation?
6\. What is sb.’s relationship with sb.?
7\. Why is sb. v.+ing sth. CLAUSE?
8\. Why is sb. adj.?
9\. Why is sb. here?
10\. Why is sb. acting adv.?
11\. How does sb. feel/look?
12\. Where are sb. (v.+ing)?
13\. What will sb v. next/is about to do?
14\. What will sb v. when/after CLAUSE?
15\. What will sb v. (if) CLAUSE?
16\. Where will sb. go?
17\. Where was sb. previously?

Table 5: Question template list summarized from the original VCR development
set.

###  A.2 QA Pair Annotation

##### Question Template List.

As mentioned in Section 3.2, we recommend annotators to design questions based
on question templates. The template list is shown in Table 5. For clustering
methods to summarize templates, we use K-Means algorithm to cluster similar
question patterns. Specifically, the maximum number of clusters is at most 20
clusters. The algorithm will automatically stop until the 200-th iteration.

##### Other Annotation Details.

To pursue diversity of question types, we require annotators to design
questions via different question templates. Besides, we ask annotators to
avoid annotating too simple answers, such as “yes” and “no”.

##  Appendix B Details of Answer Candidate Generation Algorithm

###  B.1 Relevance Model

Relevance model is to evaluate the relevance score between questions and
answers. Higher relevance scores indicate that the answers are more relevant
with the questions. We train the relevance model based on pre-trained BERT-
base parameters Wolf et al. (2020). Specifically, the training data are all
from the _original VCR training set_ and composed by relevant and irrelevant
QA pairs. The relevant QA pairs are the ones consisting of questions and their
corresponding right answers; the irrelevant pairs are the ones consisting of
questions and random answers sampled from the whole set of answer choices. We
build a binary classifier upon these training data to classify whether an
answer is relevant with a question or not. The relevance score is the
probability of being relevant pairs.

###  B.2 Pseudo Code of Answer Candidate Generation Algorithm

Algorithm 1 Answer Candidate Generation Alg.

Input: Question $Q=\\{q_{1},q_{2},...,q_{n}\\}$, the question’s right answer
$Corr=\\{c_{1},c_{2},...,c_{n}\\}$, answer candidate pool
$\mathbf{A}=\\{A_{1},A_{2},...,A_{m}\\}$, relevance model $\mathrm{Rel}$,
similarity model $\mathrm{Sim}$. $q_{i}$ and $c_{i}$ indicate tokens.
Output: The whole set of four answer choices $\mathbf{ansList}$ of the given
question $Q$, including one right choice $Corr$ and three answer candidates
$W_{1}$, $W_{2}$, and $W_{3}$.

1:Initialization: $\mathbf{ansList}$ $\leftarrow$ $\\{Corr\\}$.

2:for $t=1,2,3$ do

3: for each $A_{i}$ in
$\mathbf{A}_{\lfloor(t-1)\times\frac{m}{3}\rfloor+1:\lfloor
t\times\frac{m}{3}\rfloor}$ do

4: Initialization: $score$ $\leftarrow$ $0$, $minscore$ $\leftarrow$
$+\infty$.

5: if $\mathrm{Rel}(Q,A_{i})\geq 0.9$ then

6: for each ${ansList}_{i}$ in $\mathbf{ansList}$ do

7: $similarity\leftarrow$ $\mathrm{Sim}({ansList}_{i},A_{i})$

8: if $similarity\leq 0.2$ then

9: $score\leftarrow$ $score+similarity$

10: else

11: $score\leftarrow$ $score+10$

12: end if

13: end for

14: if $score\textless minscore$ then

15: $minscore\leftarrow score$

16: $W_{t}\leftarrow A_{i}$

17: end if

18: end if

19: end for

20: $\mathbf{ansList}$ $\leftarrow$ $\mathbf{ansList}\cup\\{W_{t}\\}$

21:end for

22:return $\mathbf{ansList}$

The pseudo code of the algorithm is shown in Algorithm 1. The two principles
for selecting answer candidates are as follows: for each QA pair, 1) they
should be relevant with the questions; 2) they should not be similar with the
right choices and the selected answer candidates. The model that computes
similarity is the “stsb-roberta-base” model Reimers and Gurevych (2019) from
github.com/UKPLab/sentence-transformers.

##  Appendix C Implementation Details of Fine-tuning VisualBERT and ViLBERT

Following VisualBERT (135.07M parameters) configuration on
VCR131313github.com/uclanlp/visualbert, we directly use the model pre-trained
on COCO Chen et al. (2015) and original VCR training set. The experiments of
ViLBERT141414github.com/jiasenlu/vilbert_beta (252.15M parameters) is based on
the model pre-trained on Conceptual Captions Sharma et al. (2018) and original
VCR training set. Both models are then fine-tuned for 8 epochs on 4 NVIDIA
GeForce GTX 1080 Ti GPUs, with learning rate $2e-5$. The batch size of
VisualBERT and ViLBERT is 32 and 16, and fine-tuning one epoch with VisualBERT
and ViLBERT costs 5.28 and 6.75 hours, respectively. For both models, we
choose the epoch which performs the best on the original VCR development set
among 8 epochs.

##  Appendix D Accuracy on the QA Pairs Involving Specific Scenarios

Table 4 shows VisualBERT’s accuracy of the QA pairs involving specific
scenarios depicted in Figure 3. Besides the study on GD-VCR, we also make
comparison between model performance on GD-VCR and the original VCR
development set. We select the scenarios frequently appearing in both GD-VCR
and the original VCR development set.

Results are shown in Table 6. We observe that on the images involving
scenarios such as funeral, VisualBERT’s performance gap is nearly 25%, which
is considerably large. The results further demonstrate that the model is still
incapable of tackling the QA pairs which are involving cultural differences
behind scenarios well.

Figure 4: Statistics of keyword occurrences. “Others” denotes the average
occurrences of the keywords appearing less than 20 times.

Regions/Datasets | Wedding | Funeral | Servant
---|---|---|---
Original VCR | 71.78 | 55.00 | 59.72
Other Regions | 50.00 | 30.25 | 48.81

Table 6: Accuracy (%) on the images involving the same scenarios from the
original VCR dataset and non-Western regions from GD-VCR dataset,
respectively.

##  Appendix E Keywords in GD-VCR Dataset

Figure 4 shows the overall statistics of keyword occurrences in GD-VCR
benchmark. There are 693 keywords in total, showing the diversity of the
scenarios covered by GD-VCR dataset. Besides, we observe that the keywords
whose corresponding scenarios have evident regional differences, such as
“wedding”, “religion”, “groom”, “bride”, appear frequently in GD-VCR.

##  Appendix F More Examples in GD-VCR Dataset

In this section, we showcase several examples of GD-VCR in detail. Aside from
the images about “wedding” in Figure 1, we manifest the images regarding to
“customer” and “funeral” from the four regions we study. In Figure 5 and
Figure 6, we can observe the regional characteristics from the selected
images. Furthermore, we visualize VisualBERT’s prediction on each QA pair.

Figure 5: Examples of the images regarding “customer”. Left-to-right order:
Western, South Asia, East Asia. We visualize the prediction of the VisualBERT
model fine-tuned on the original VCR training set. The blue blocks denote the
right answer choices. If red block appears, it means that VisualBERT wrongly
predict the answer. The rightmost value indicates the probability of the
corresponding choices being selected by VisualBERT. Figure 6: Examples of the
images regarding “funeral” or “death”. Left-to-right order in the first row:
Western, East Asia; Left-to-right order in the second row: South Asia, Africa.


# A novel machine learning technique to identify and categorize plasma waves
in spacecraft measurements

###### Abstract

The available magnetic field data from the terrestrial magnetosphere, solar
wind and planetary magnetospheres exceeds over $10^{6}$ hours. Identifying
plasma waves in these large data sets is a time consuming and tedious process.
In this Paper, we propose a solution to this problem. We demonstrate how Self-
Organizing Maps can be used for rapid data reduction and identification of
plasma waves in large data sets. We use 72,000 fluxgate and 110,000 search
coil magnetic field power spectra from the Magnetospheric Multiscale Mission
(MMS1) and show how the Self-Organizing Map sorts the power spectra into
groups based on their shape. Organizing the data in this way makes it very
straightforward to identify power spectra with similar properties and
therefore this technique greatly reduces the need for manual inspection of the
data. We suggest that Self-Organizing Maps offer a time effective and robust
technique, which can significantly accelerate the processing of magnetic field
data and discovery of new wave forms.

JGR: Space Physics - Technical Reports: Methods

Laboratory for Atmospheric and Space Physics, University of Colorado Boulder,
Boulder, CO, USA Department of Astrophysical and Planetary Sciences,
University of Colorado Boulder, Boulder, Colorado, USA

Daniel<EMAIL_ADDRESS>
We develop a robust method to classify large data sets of power spectra of
magnetic field

The technique significantly reduces the need for time consuming manual
inspection of the data and allows the discovery of new wave forms

The classification technique can be used to categorize plasma waves based on
frequency, amplitude and bandwidth

## Plain Language Summary

[ enter your Plain Language Summary here or delete this section]

##  1 Introduction

Space missions like Wind [Lepping . (1995), Bougeret . (1995)], Cassini
[Dougherty . (2004), Gurnett . (2004)], Van Allen Probes [Kletzing . (2013)]
and the Magnetospheric Multiscale Mission (MMS) [Russell . (2016), Le Contel .
(2016), Lindqvist . (2016), Ergun . (2016)] collected massive data sets of
electromagnetic field data in a very wide range of plasma environments. Given
the sheer volume of the available data, identifying plasma waves is a
difficult task since there is no straightforward method for automatic
identification of wave activity such as fast mode, ion cyclotron, whistler,
lower/upper hybrid and ion acoustic waves. In many cases the identification of
plasma waves is still done by manually inspecting measurements of electric and
magnetic fields. This is a time consuming and tedious process, which typically
focuses only on a small subset of all the available data. Manual inspection of
the data can certainly identify unique wave events (i.e. case studies) that
display unusual features and provide useful insights into the underlying
plasma physics. However, it is often difficult to design a search algorithm,
which processes the entire available data set and filters out similar wave
events to the ones, which were manually identified.

Another frequently used approach for wave event selection is setting arbitrary
thresholds on the data such as amplitude of magnetic and/or electric field
fluctuations at given frequencies [<]e.g.¿[]usanova2012themis, magnetic
compressibility
[<]e.g¿[]shi2017systematic,malaspina2017statistical,ala2019alfven, magnetic
helicity [<]e.g.¿[]woodham2019parallel and coherence between components of the
electric and/or magnetic fields [<]e.g.¿[]gao2014statistical. Although, this
technique is effective at processing large data sets, it may miss a
significant number of waves because they may not meet the amplitude or other
thresholds and the waves may occur at different frequency than expected.

In this Paper, we a present a solution to this problem: we use an unsupervised
machine learning technique called Self-Organizing Map (SOM) [Kohonen (1990)]
and show how it can be used for rapid data reduction, identification of plasma
waves from large data sets of spacecraft measurements and discovery of new
wave forms. This technique sorts power spectra of magnetic field into nodes
where power spectra belonging to the same node have similar properties: they
all display wave activity at approximately the same frequency with similar
amplitude, bandwidth and they have similar spectral indices and break
frequencies.

Various machine learning techniques have been developed for processing solar
radio bursts [S. Chen . (2017), Mohd . (2020)] and determining electron
density from plasma waves [Zhelavskaya . (2016)], however, to the best of our
knowledge, there has not been an attempt to process electromagnetic field data
with SOMs. The most similar approach to the one presented in our paper was
developed by the WAVES instrument team of the Wind spacecraft [Bougeret .
(1995)] who used an on-board neural network for the data selection of the
time-domain sampler and thermal noise receiver. This neural network was
trained to recognize event signatures, separate them from noise, rank them
depending on how similar they are to the reference signal and mark them for
download. Our technique is more advanced compared to this approach since there
is no pre-defined reference signal and the data is organized into groups based
on a similarity metric.

We demonstrate our approach by analyzing a data set of 72,000 fluxgate (FGM)
and 110,000 search coil (SCM) magnetic field power spectra from the
Magnetospheric Multiscale Mission (MMS1). We compute the power spectra of the
magnetic field for each 2-minute and 5-second interval for the FGM and SCM
data, respectively and use the SOM to assign these power spectra to nodes. We
show that this technique can classify power spectra remarkably well and makes
it very easy to identify intervals with significant wave activity. Our
technique is also suitable for classification of power spectra based on more
subtle features such as break frequency and spectral indices.

##  2 Self-Organizing Maps

In this Section we provide a summary of the basic principles of Self-
Organizing Maps based on kohonen2013essentials. Self-Organizing Map is an
unsupervised machine learning technique, which uses a finite set of models to
represent a distribution of input data. A key feature of SOM is that the
models are associated with the nodes of a regular (usually two-dimensional)
grid. Adjacent nodes in the grid are associated with more similar models, in
contrast, more distinct models are located farther away from each other in the
grid. This is essentially a similarity diagram of the models, which allows us
to understand the topographic relationships between high-dimensional input
data. Each model is characterized by a set of weights that match the size of
the input data items (i.e. for an input data set of 30 dimensional vectors,
each model is characterized with 30 weights whose initial values are randomly
chosen).

The central idea of the SOM learning process is the following: _”Every input
data item shall select the model that matches best with the input item, and
this model, as well as a subset of its spatial neighbors in the grid, shall be
modified for better matching.”_ The learning process is carried out with a
batch-type method: the entire input data set is presented to the algorithm and
all models and their weights are simultaneously updated. This process is
repeated a few dozens times until the models are stabilized.

Various metrics are used to quantify the similarity between the input data and
a model. In most applications including our study as well, the Euclidean
distance ($d(q,p)=\sqrt{\sum(q_{i}-p_{i}})^{2}$) is used to measure the
similarity. Therefore an input data item is assigned to a node, which has the
lowest $d$ value (i.e. the input data item is most similar to that node). The
determination of the number of nodes is carried out by a trial-and-error
approach and a compromise must be made between resolution (number of nodes)
and statistical accuracy (variability of input data assigned to the same
node). For most applications the number of nodes is typically between a few
dozens and a few hundreds.

Potentially the most important advantage of SOMs is that the resulting grid
clustering can be easily interpreted, therefore high dimensional input vectors
with similar properties can be identified. A possible disadvantage is that
SOMs require large volume of data in order to develop meaningful clusters.
Additionally, the method assumes that the input data items fall in a finite
number of classes and the different models can be made to correspond to these
classes [Kohonen (2013)], however, this may not be the case if the input data
is the affected by significant random noise.

##  3 Demonstration of the Method

We use 48000 burst mode intervals ($\approx$2400 hours of data) from MMS1
spacecraft from 2015 October 1 to 2020 December 22. For the FGM (128 Hz
cadence) data analysis, each burst mode interval is split into 2-min segments.
If the data quality flag (assigned by the instrument team) is 0 (”good data”)
for all data points in a 2-min segment, the magnetic power spectra of the X,
Y, Z field components are calculated separately and are summed up hence the
trace power spectra is obtained. For the calculation of the power spectra the
Welch method (Welch, 1967) is used with Hanning window (50% overlap). The FGM
power spectra are calculated on a logarithmic grid with 83 frequencies in the
range of 0.03 to 7 Hz. For the SCM data (8192 Hz cadence), we use 5000 burst
mode intervals ($\approx$155 hours of data) from MMS1 spacecraft from 2015
October 1 to 2016 September 1. The SCM data processing follows the same
methodology, however, the power spectra are calculated for each 5-second
interval between 10-3000 Hz range. For both the FGM and SCM data we chose 83
frequency bins to ensure that fine structures in the power spectra (such as
narrow band wave activity) are well preserved. We clip the FGM and SCM power
spectra at 7 and 3000 Hz frequencies, respectively to avoid the classification
of the spectra based on the frequency range, which is significantly affected
by noise. For the data selection no constraints are imposed on the spacecraft
location and both the FGM and SCM data sets include approximately 52% and 48%
power spectra from the dayside and nightside magnetosphere, respectively.

Since the background turbulence amplitude is highly variable, the power
spectra have to be normalized before the SOM training process. We shift (in
amplitude) the SCM and FGM power spectra with a constant factor so they are
all set to log101 nT2/Hz at the lowest frequency (0.03 and 10 Hz for the FGM
and SCM, respectively). This normalization means that the differences between
the power spectra are explained by differences in the spectral indices, break
frequencies, wave activity and therefore the effect of the varying turbulence
amplitude is minimized.

Our input data set includes 72,000 FGM and 110,000 SCM power spectra,
respectively. For the SOM training process, we use 100 nodes, the Euclidean
metric and 200 training iterations to ensure that the models are stabilized.
We have tested several different number of iterations (50, 100, 150, 200) and
found that after 150 iterations the models converged to a steady state and
between 150 and 200 iterations there was negligible change in the distribution
of the input data points among the nodes.

We have trained separate SOMs for the FGM and SCM data sets, respectively
using all the available power spectra. The distribution of the input data
points in the grid of nodes is shown in Figure 1. In Figure 1a and b the
hexagonal grid shows the 100 nodes and the numbers correspond to how many
power spectra were assigned to each node. The training process was repeated
five times to test the robustness of the models. Approximately 0.08% of the
input vectors showed variations, meaning that they were not assigned to the
same node in all iterations, therefore we suggest that the models are not
sensitive to the initial values of the nodes.

We used the distribution of the input data in the SOM grid (Figure 1) to
identify whether the number of nodes is sufficient. In the case of the FGM
data, the nodes with the least number of data points have approximately
$\approx$150 power spectra ($\approx$750 min of data). Given the size of the
input data (72,000 power spectra), this means that rare wave events that make
up only 0.2% (150 / 72,000) of the data can form a separate node and therefore
they can be easily identified. In the case of the SCM grid the ratio of the
minimum number of power spectra / total number of power spectra is also around
0.2%.

To understand how the SOM organized the input data among the 100 nodes, we
calculate the average power spectra in each node. Figure 2 shows 9 examples of
these and the error bars correspond to the standard deviation of the data in
each frequency bin. As a reference, the dotted line shows the spectral index
of -1.5. The spectral slopes are calculated over frequency windows, which have
a factor of 2.8 and 1.8 width for the FGM and SCM data, respectively, which is
the reason the beginning of the blue and brown lines in Figure 2 and 4 are not
aligned.

In Figure 2a-b-c there are three nodes that show signatures of ion-scale wave
activity, which can be seen an enhanced wave power in the frequency range of
approximately 0.1-1 Hz and the spectra becomes flat. Considering the frequency
range, these waves are consistent with ion cyclotron and fast mode waves
[<]e.g.¿[]gary1992mirror. The second row of Figure 2 show three nodes, which
have the most data points ($>$1500 power spectra in each node). These straight
power spectra are typically associated with the dissipation of kinetic Alfvén
waves and are often observed in the terrestrial magnetosheath (see Figure 4 in
[C. Chen Boldyrev (2017)]). In the third row of Figure 2 there are three power
spectra, which show steepening above 1 Hz. A possible explanation is ion
cyclotron/fast mode wave activity in the frequency range of 0.1-1 Hz or in the
absence of those waves the ion-scale break might be at an unusually high
frequency.

Figure 3 shows a one-hour interval of fluxgate magnetic field data, the
corresponding trace magnetic power spectra and the node number assigned to
each 2-min interval. It can be see that the node number changes nearly every
consecutive 2-min intervals, which indicates that the power spectra and
therefore the fluctuations of the magnetic field show substantial variations.
A major advantage of our technique is that this variability can be quantified
and intervals with similar properties (i.e. same node number) can be
qualitatively studied in terms of background plasma parameters (plasma beta,
temperature anisotropy etc.) and spatial locations. Similar quantification of
the power spectra with manual inspection is infeasible given the volume of the
data. The threshold technique based on magnetic helicity
[<]e.g.¿[]woodham2019parallel, compressibility
[<]e.g.¿[]malaspina2017statistical and other proxies might be extended to an
array of frequencies to select intervals based on multiple criteria (i.e
measuring magnetic helicity simultaneously at 0.1, 1, 3 Hz etc.). However, as
we use more and more variables to characterize the power spectra the
interpretation is getting difficult as well. An important advantage of our
technique is that despite the high-dimensional (83) input data, the SOM gives
a clear interpretation by assigning each spectra to a node.

Figure 4 has the same format as Figure 2 and shows nine examples from the SCM
nodes. We suggest that our technique is capable of distinguishing wave events
based on amplitude, frequency, and bandwidth. Figure 4a-b-c show examples of
nodes with minor wave activity. Figure 4d-e-f from left to the right show
waves with increasing frequency of the spectral break and finally Figure
4g-h-i show wave events with increasing bandwidths. Designing a search
algorithm, which takes into account the amplitude, frequency and bandwidth is
a very challenging problem and would be a tedious trail-and-error process to
fine tune the filtering parameters. In contrast, the SOM offers a simple
solution to distinguish wave events based on these characteristics.

Our technique is capable of identifying rare wave events. For example, 95% of
the power spectra assigned to node $\\#87$ (Figure 4i) occurred within a one
hour interval out of the 155 hours searched. A key feature of SOM is that no
prior assumptions are made about the underlying data. If certain wave events
occur rarely in the data set but are significantly different from the rest of
the data points, then they will be assigned to a separate node and therefore
they can be easily identified. The search process based on thresholds would be
inherently biased and would be based on assumptions about the expected wave
spectral shapes in the data set, therefore it would be difficult to identify
rare events.

Our technique is also suitable for identifying intervals with desirable
frequency features. For example, we identified intervals when the power
spectra moved from node $\\#42$ (Figure 5a) to node $\\#41$ (Figure 5b).
Figure 5c shows the time-frequency domain for these intervals: the first 5s
(node $\\#42$) shows signatures of wave activity starting from 30 Hz and
between 5-10s (node $\\#41$) the wave activity increases toward higher
frequencies and reaches nearly 200 Hz. For the design of a search algorithm
several free parameters (e.g. lowest frequency of the wave, rate of change of
the frequency) would have to be estimated with a reasonable accuracy to
identify large number of similar wave events. In the case of SOM analysis,
there is no need to estimate these parameters for the purpose of wave
identification since one can look for specific frequency features given the
nodes.

Figure 1: a)-b) Distribution of the input data in the grid of the FGM and SCM
self-organized maps. The 100 nodes are numbered from left to right the
following way: bottom left corner is node $\\#1$, bottom right corner is
$\\#10$. Top left and top right nodes are $\\#91$ and $\\#100$, respectively.
Figure 2: Averaged FGM power spectra and corresponding error bars from nine
nodes. Figure 3: a) Example of a one-hour long fluxgate magnetic field data
from 10 November 2015 02:00-03:00, (b) the corresponding trace power spectra
and (c) the assigned node numbers to each 2-min interval. In panel c) the red
circles are placed at the center of each 2-min interval. Figure 4: Averaged
SCM power spectra and corresponding error bars from nine nodes. Figure 5:
Example for a wave with time dependent frequency feature based on SCM data.
The first 5s of the interval was assigned to node $\\#$42 while the
consecutive interval to $\\#$41\. In the first interval the wave activity
starts at approximately 30 Hz and steadily increases toward 200 Hz in the
second interval.

##  4 Conclusion

In this Paper, we have described a novel machine learning technique to
classify the power spectra of magnetic field fluctuations. The technique uses
a high-dimensional input (magnetic turbulence amplitude in 83 frequency bins)
and organizes the power spectra into 100 nodes based on their shape. Similar
classification of the data is not possible with the threshold techniques or
with manual inspection of the data. We note that the technique does not
identify the wave mode itself, which would require careful analysis of the
wave polarization properties and dispersion relations. Despite this
limitation, the proposed technique can have huge implications to process
magnetic field measurements by significantly reducing the need for time
consuming manual inspection of the data and eliminating the need for arbitrary
thresholds on the data to filter out wave events. Here we summarize three main
areas where our technique will advance the processing of electric and magnetic
field data.

Identification of wave type: The proposed technique can greatly accelerate the
processing of electromagnetic field data by identifying and classifying
consistently occurring patterns in plasma wave activity independent from the
observational biases or expectations of a given researcher. In our example, 3
nodes of the FGM data showed significantly enhanced wave power and nearly flat
power spectra in the 0.1-1 Hz frequency range, which are consistent with ion
cyclotron and fast magnetosnic waves. For the SCM data we found large number
of nodes with intense wave activity and we showed that our technique can
classify them based on frequency, bandwidth, and normalized amplitude as well.

Discovery of new wave spectral shapes: a very important feature of the
proposed approach is that no assumptions are made about the shape of the power
spectra, which is a major advantage compared to the threshold techniques.
Regardless the underlying input data, the SOM automatically identifies wave
events that deviate significantly from the rest of the input data and assigns
them to nodes accordingly. This means that if the data set includes only a
handful of those wave events we are looking for, they can still be identified.
This aspect is particularly useful for studying less well understood space
plasmas such as the inner-heliosphere [Bale . (2016)] where the observable
wave forms might be substantially different from the expectations and using
arbitrary threshold could not identify those waves events. The technique could
lead to an improved understanding of magnetospheric physics as well since most
studies rely on arbitrary thresholds of frequency, amplitude, ellipticity for
wave identification [<]e.g.¿[]usanova2012themis, shi2017systematic and
therefore compelling wave events are hidden in the large data sets provided by
Van Allen Probes, Cluster, THEMIS and MMS.

Our technique is capable of classifying power spectra based on spectral
indices and break frequencies. Very large number of studies focused on the
ion-scale spectral break and its dependence on plasma parameters
[<]e.g¿[]bourouaine2012spectral, vech2018magnetic, duan2020radial. One
fundamental difficulty is that measuring the break frequency is challenging
especially in those cases when the spectral index shows only minor steepening
around ion scale. Our technique makes it possible to identify groups of power
spectra with similar break frequencies therefore the underlying plasma
properties can be investigated and quantified. Additionally, studying the
underlying plasma parameters for each of the 100 nodes can help to understand
the operation of the dissipation range turbulence cascade and damping
mechanisms since we could precisely quantify how the power spectra responds to
changes in plasma beta, ion and electron temperature anisotropy and other
parameters.

###### Acknowledgements.

D. V. was supported by NASA contract 80NSSC21K0454. D.M. was supported by NASA
contract 80NSSC19K0305. The data presented in this paper are the L2 data of
MMS, which can be accessed from https://lasp.colorado.edu/mms/sdc/public/)

## References

  * Ala-Lahti . (2019) ala2019alfvenAla-Lahti, M., Kilpua, EK., Souček, J., Pulkkinen, TI. Dimmock, AP.  2019\.  Alfvén ion cyclotron waves in sheath regions driven by interplanetary coronal mass ejections Alfvén ion cyclotron waves in sheath regions driven by interplanetary coronal mass ejections.  Journal of Geophysical Research: Space Physics12463893–3909.
  * Bale . (2016) bale2016fieldsBale, S., Goetz, K., Harvey, P., Turin, P., Bonnell, J., De Wit, TD.others  2016\.  The FIELDS instrument suite for solar probe plus The fields instrument suite for solar probe plus.  Space science reviews204149–82.
  * Bougeret . (1995) bougeret1995wavesBougeret, JL., Kaiser, ML., Kellogg, PJ., Manning, R., Goetz, K., Monson, S.others  1995\.  Waves: The radio and plasma wave investigation on the Wind spacecraft Waves: The radio and plasma wave investigation on the wind spacecraft.  Space Science Reviews711-4231–263.
  * Bourouaine . (2012) bourouaine2012spectralBourouaine, S., Alexandrova, O., Marsch, E. Maksimovic, M.  2012\.  On spectral breaks in the power spectra of magnetic fluctuations in fast solar wind between 0.3 and 0.9 AU On spectral breaks in the power spectra of magnetic fluctuations in fast solar wind between 0.3 and 0.9 au.  The Astrophysical Journal7492102.
  * C. Chen Boldyrev (2017) chen2017natureChen, C. Boldyrev, S.  2017\.  Nature of kinetic scale turbulence in the Earth’s magnetosheath Nature of kinetic scale turbulence in the earth’s magnetosheath.  The Astrophysical Journal8422122.
  * S. Chen . (2017) chen2017convolutionalChen, S., Xu, L., Ma, L., Zhang, W., Chen, Z. Yan, Y.  2017\.  Convolutional neural network for classification of solar radio spectrum Convolutional neural network for classification of solar radio spectrum.  2017 IEEE International Conference on Multimedia & Expo Workshops (ICMEW) 2017 ieee international conference on multimedia & expo workshops (icmew) ( 198–201).
  * Dougherty . (2004) dougherty2004cassiniDougherty, M., Kellock, S., Southwood, D., Balogh, A., Smith, E., Tsurutani, B.others  2004\.  The Cassini magnetic field investigation The cassini magnetic field investigation.  The cassini-huygens mission331–383.
  * Duan . (2020) duan2020radialDuan, D., Bowen, TA., Chen, CH., Mallet, A., He, J., Bale, SD.others  2020\.  The radial dependence of proton-scale magnetic spectral break in slow solar wind during PSP Encounter 2 The radial dependence of proton-scale magnetic spectral break in slow solar wind during psp encounter 2\.  The Astrophysical Journal Supplement Series246255.
  * Ergun . (2016) ergun2016axialErgun, R., Tucker, S., Westfall, J., Goodrich, K., Malaspina, D., Summers, D.others  2016\.  The axial double probe and fields signal processing for the MMS mission The axial double probe and fields signal processing for the mms mission.  Space Science Reviews1991-4167–188.
  * Gao . (2014) gao2014statisticalGao, X., Li, W., Thorne, R., Bortnik, J., Angelopoulos, V., Lu, Q.Wang, S.  2014\.  Statistical results describing the bandwidth and coherence coefficient of whistler mode waves using THEMIS waveform data Statistical results describing the bandwidth and coherence coefficient of whistler mode waves using themis waveform data.  Journal of Geophysical Research: Space Physics119118992–9003.
  * Gary (1992) gary1992mirrorGary, SP.  1992\.  The mirror and ion cyclotron anisotropy instabilities The mirror and ion cyclotron anisotropy instabilities.  Journal of Geophysical Research: Space Physics97A68519–8529.
  * Gurnett . (2004) gurnett2004cassiniGurnett, DA., Kurth, WS., Kirchner, D., Hospodarsky, G., Averkamp, T., Zarka, P.others  2004\.  The Cassini radio and plasma wave investigation The cassini radio and plasma wave investigation.  The Cassini-Huygens Mission395–463.
  * Kletzing . (2013) kletzing2013electricKletzing, C., Kurth, W., Acuna, M., MacDowall, R., Torbert, R., Averkamp, T.others  2013\.  The electric and magnetic field instrument suite and integrated science (EMFISIS) on RBSP The electric and magnetic field instrument suite and integrated science (emfisis) on rbsp.  Space Science Reviews1791-4127–181.
  * Kohonen (1990) kohonen1990selfKohonen, T.  1990\.  The self-organizing map The self-organizing map.  Proceedings of the IEEE7891464–1480.
  * Kohonen (2013) kohonen2013essentialsKohonen, T.  2013\.  Essentials of the self-organizing map Essentials of the self-organizing map.  Neural networks3752–65.
  * Le Contel . (2016) le2016searchLe Contel, O., Leroy, P., Roux, A., Coillot, C., Alison, D., Bouabdellah, A.others  2016\.  The search-coil magnetometer for MMS The search-coil magnetometer for mms.  Space Science Reviews1991-4257–282.
  * Lepping . (1995) lepping1995windLepping, R., Acũna, M., Burlaga, L., Farrell, W., Slavin, J., Schatten, K.others  1995\.  The WIND magnetic field investigation The wind magnetic field investigation.  Space Science Reviews711-4207–229.
  * Lindqvist . (2016) lindqvist2016spinLindqvist, PA., Olsson, G., Torbert, R., King, B., Granoff, M., Rau, D.others  2016\.  The spin-plane double probe electric field instrument for MMS The spin-plane double probe electric field instrument for mms.  Space Science Reviews1991-4137–165.
  * Malaspina . (2017) malaspina2017statisticalMalaspina, DM., Jaynes, AN., Hospodarsky, G., Bortnik, J., Ergun, RE. Wygant, J.  2017\.  Statistical properties of low-frequency plasmaspheric hiss Statistical properties of low-frequency plasmaspheric hiss.  Journal of Geophysical Research: Space Physics12288340–8352.
  * Mohd . (2020) mohd2020performanceMohd, MRS., Johari, J. Ruslan, FA.  2020\.  Performance Analysis of Artificial Neural Network Approach on Solar Radio Burst Detection Performance analysis of artificial neural network approach on solar radio burst detection.  2020 IEEE 10th International Conference on System Engineering and Technology (ICSET) 2020 ieee 10th international conference on system engineering and technology (icset) ( 55–60).
  * Russell . (2016) russell2016magnetosphericRussell, C., Anderson, B., Baumjohann, W., Bromund, K., Dearborn, D., Fischer, D.others  2016\.  The magnetospheric multiscale magnetometers The magnetospheric multiscale magnetometers.  Space Science Reviews1991-4189–256.
  * Shi . (2017) shi2017systematicShi, R., Li, W., Ma, Q., Reeves, GD., Kletzing, C., Kurth, W.others  2017\.  Systematic evaluation of low-frequency hiss and energetic electron injections Systematic evaluation of low-frequency hiss and energetic electron injections.  Journal of Geophysical Research: Space Physics1221010–263.
  * Usanova . (2012) usanova2012themisUsanova, M., Mann, I., Bortnik, J., Shao, L. Angelopoulos, V.  2012\.  THEMIS observations of electromagnetic ion cyclotron wave occurrence: Dependence on AE, SYMH, and solar wind dynamic pressure Themis observations of electromagnetic ion cyclotron wave occurrence: Dependence on ae, symh, and solar wind dynamic pressure.  Journal of Geophysical Research: Space Physics117A10.
  * Vech . (2018) vech2018magneticVech, D., Mallet, A., Klein, KG. Kasper, JC.  2018\.  Magnetic reconnection may control the ion-scale spectral break of solar wind turbulence Magnetic reconnection may control the ion-scale spectral break of solar wind turbulence.  The Astrophysical Journal Letters8552L27.
  * Woodham . (2019) woodham2019parallelWoodham, LD., Wicks, RT., Verscharen, D., Owen, CJ., Maruca, BA. Alterman, BL.  2019\.  Parallel-propagating Fluctuations at Proton-kinetic Scales in the Solar Wind Are Dominated By Kinetic Instabilities Parallel-propagating fluctuations at proton-kinetic scales in the solar wind are dominated by kinetic instabilities.  The Astrophysical Journal Letters8842L53.
  * Zhelavskaya . (2016) zhelavskaya2016automatedZhelavskaya, I., Spasojevic, M., Shprits, Y. Kurth, W.  2016\.  Automated determination of electron density from electric field measurements on the Van Allen Probes spacecraft Automated determination of electron density from electric field measurements on the van allen probes spacecraft.  Journal of Geophysical Research: Space Physics12154611–4625.