# Global Reactions to the Cambridge Analytica Scandal: A Cross-Language Social
Media Study

Felipe González  UTFSMSantiagoChile<EMAIL_ADDRESS>,  Yihan Yu
University of WashingtonSeattleUnited States<EMAIL_ADDRESS>,  Andrea Figueroa
UTFSMValparaísoChile<EMAIL_ADDRESS>,  Claudia López
UTFSMValparaísoChile<EMAIL_ADDRESS>and  Cecilia Aragon  University of
WashingtonSeattleUnited States<EMAIL_ADDRESS>
(2019)

###### Abstract.

Currently, there is a limited understanding of how data privacy concerns vary
across the world. The Cambridge Analytica scandal triggered a wide-ranging
discussion on social media about user data collection and use practices. We
conducted a cross-language study of this online conversation to compare how
people speaking different languages react to data privacy breaches. We
collected tweets about the scandal written in Spanish and English between
April and July 2018. We used the Meaning Extraction Method in both datasets to
identify their main topics. They reveal a similar emphasis on Zuckerberg’s
hearing in the US Congress and the scandal’s impact on political issues.
However, our analysis also shows that while English speakers tend to attribute
responsibilities to companies, Spanish speakers are more likely to connect
them to people. These findings show the potential of cross-language
comparisons of social media data to deepen the understanding of cultural
differences in data privacy perspectives.

Data privacy; cross-language comparisons; Twitter

††journalyear: 2019††copyright: iw3c2w3††conference: Companion Proceedings of
the 2019 World Wide Web Conference; May 13–17, 2019; SFO, CA, USA††booktitle:
Companion Proceedings of the 2019 World Wide Web Conference (WWW ’19
Companion), May 13–17, 2019, San Francisco, CA, USA††doi:
10.1145/3308560.3316456††isbn: 978-1-4503-6675-5/19/05††submissionid:
laweb07††ccs: Security and privacy Social aspects of security and
privacy††ccs: Information systems Document topic models

##  1\. Introduction

While there is evidence that concerns about privacy and its intricate
relationship with users’ decisions to use social media have been rising among
Americans (Smith et al., 1996), much less is known about these perspectives
across the world. So far, most of the literature about privacy concerns has
been focused on the United States and even though recently research has begun
to be carried out examining European privacy perspectives after the General
Data Protection Regulation (GDPR) implementation, privacy research in
international settings is still needed (Smith et al., 2011). A few survey
studies of people from different pairs of countries have been conducted to
address. This, and the results so far indicate that data privacy perspectives
vary significantly across countries. Belgians self-reported lower levels of
concerns about sensitive information leakage than people from the United
States. Harris et al. attributed this difference to the privacy laws of the
respondents’ countries of origin (Harris et al., 2003; Smith, 2001). A survey
of North American and Turkish freshmen living in similar residence hall
settings (Kaya and Weber, 2003) showed that Americans wished for more privacy
in their hall rooms than Turkish students. In the context of e-commerce,
another survey found that Italians tend to exhibit lower privacy concerns than
Americans (Dinev et al., 2006).

Unfortunately, relying solely on survey data for cross-cultural studies of
data privacy has various limitations. Most of them focus only on two
geographic regions and have limited sample size (Ur and Wang, 2013).
Additionally, most privacy surveys are only available in English and only a
few of them have been translated to other languages (Ur and Wang, 2013).
Surveys of a multinational or global nature that can mitigate these
limitations would be very costly, which makes it difficult to compare privacy
attitudes more broadly.

As the Cambridge Analytica scandal unfolded and people became aware that the
personal data of 87 million Facebook users were exposed without their consent
and used by Cambridge Analytica to support political campaigns (Lapaire,
2018), thousands of people in different parts of the world expressed on social
media their reactions to and reflections on the scandal, its relationship to
data privacy, and its broader implications. Indeed, a movement to delete
Facebook accounts emerged and the #deleteFacebook hashtag was trending for
several days (Mirchandani, 2018).

In this paper, we report on a study that observes Twitter activity about the
Cambridge Analytica scandal in Spanish and English and proposes a methodology
for cross-language comparison of social media text. We believe that our
approach offers an alternative or complementary method to conduct studies on
data privacy perspectives across speakers of different languages and may
provide a roadmap for future cross-cultural research. As Twitter allows people
to express themselves freely and spontaneously and in different languages, it
enables a unique opportunity to analyze multi-language large-scale data. These
characteristics allow researchers to address some of the limitations of the
survey-based methods described above, such as those related to language and
sample size (Wang et al., 2017). Our premise is that written communication can
be a “window into culture and an external reflection of cultural values” (Choi
et al., 2016); therefore, what people write about the scandal in their own
languages can reveal differences and similarities in data privacy concerns
across users worldwide.

We summarize prior work on cross-language comparisons of social media in
Section 2. Section 3 introduces our research question, Section 4 details our
research method, Section 5 reports on our findings, and Section 6 offers a
discussion of our results, its limitations, and future work. Finally, Section
7 provides our conclusions.

##  2\. Cross-language comparisons of user-generated written content

Prior research has used social media text in different languages to make
comparisons among people who speak these languages. An analysis of more than
62 million tweets compared the top 10 most common languages regarding the use
of features such as: URLs, hashtags, mentions, replies and re-tweets (Hong et
al., 2011). The findings show that German-speaking users tend to include more
URLs and hashtags in their tweets than other users, while Korean-speaking
users are prone to reply to each other more often than speakers of other
languages. Hong et al. argue that users of different languages use Twitter for
different purposes. The German community often uses this platform for
information sharing, while the Korean community employs it for conversational
purposes. Another study analyzes tweets written by Americans in English and
Japanese in their official language (Acar, 2013). As opposed to Americans,
Japanese people tweet more self-related messages and more messages about TV
programs. In turn, Americans tweet more about their peers, sports and news.

Previous work has also explored how user-generated content can reveal
different views of the same issues among people who write in different
languages. The Meaning Extraction Method (MEM) was applied to compare posts
from depression-related forums in Spanish and English (Ramirez-Esparza et al.,
2008). MEM is used to discover the main topics in a corpus. A comparison of
the resulting topics shows that English posts tend to use words that are more
concrete and descriptive and the main topics are related to medicinal
questions and concerns. Spanish posts use relatively more emotional words and
the main topics are associated with sharing and disclosing information about
relational concerns. Hecht and Gergle used the Explicit Semantic Analysis
(ESA) algorithm to analyze pairs of terms from ten different Wikipedia
language editions (Hecht and Gergle, 2010). ESA indicates a score of semantic
relatedness between two concepts. The findings reveal that “even when two
language editions cover the same concept, they may describe that concept
differently” (Hecht and Gergle, 2010). For example, consider the pair
“Germany” / “Saxony-Anhalt” (a state of Germany). In most languages this pair
receives a high ESA score, but the algorithm detects no relation at all in
Italian and Danish. This occurs because there are no articles that mention
“Germany” and “Saxony-Anhalt” together in these two languages. Analyses of
semantic networks and the salience of semantic concepts in articles about
China in the Chinese and English versions of Wikipedia found dissimilarities
in the semantic content of these two versions (Jiang et al., 2017). Articles
in the Chinese version are framed from the perspectives of respecting
authority, emphasizing harmony and patriotism. Articles in English are written
from the perspective of Western-societies’ core value of democracy. The
English version contains critical attitudes toward the authority of the
Chinese government and the Communist Party in terms of human rights and
territorial dispute. According to Jiang et al. cultures, values, interests,
situations and emotions of different language groups can explain these
dissimilarities. The latter studies provide evidence in support of what is
known as the Sapir-Whorf hypothesis, which indicates that the structure of
anyone’s native language influences the world-views she or he will acquire as
she or he learns the language (Kay and Kempton, 1984). Thus, speakers of
different languages could think, perceive reality and organize the world
around them in different ways (Hussein, 2012). Inspired by this hypothesis, we
seek to study whether people who speak a different language hold different
views of a data privacy scandal, such as the Cambridge Analytica case.

##  3\. Research Question

Given the relative lack of data privacy research in international settings
(Smith et al., 2011), our project aims to investigate the potential of social
media text written in different languages as a source to compare data privacy
views worldwide. Beyond the Sapir-Whorf hypothesis (explained in Section 2),
prior privacy research has argued that language and country of residence can
relate to diverse perspectives on privacy. Smith et al. (Smith et al., 2011)
noted that “many languages, including those in European countries (e.g
Russian, French, Italian), do not have a word for privacy and have adopted the
English word”. Belanger and Crossler argued that “individuals from different
countries can be expected to have different cultures, values and laws, which
may result in differences in their perceptions of information privacy and its
impacts” (Bélanger and Crossler, 2011).

As the Cambridge Analytica scandal sparked worldwide conversations (in diverse
languages) on Twitter about this particular misuse of user data, these online
public communications are useful sources to contrast views on the scandal
itself, its relation to data privacy, and its implications. To start
addressing our ultimate research goal, this paper focuses on the following
research question: “Which are the shared and unique topics that emerge from
the Twitter activity in Spanish and English about the Cambridge Analytica data
misuse scandal?”

##  4\. Data and methods

To answer our research question, we used Tweepy111http://www.tweepy.org/, a
Python library for accessing to the standard realtime streaming Twitter API.
Using this library we were able to capture tweets that include hashtags or
keywords related to the Cambridge Analytica scandal or data privacy, such as:
“#CambridgeAnalytica”, “#DeleteFacebook”, “Zuckerberg” and “Facebook privacy”.
The standard realtime streaming Twitter API returns a random sample of all
public tweets that match the search keywords. We collected tweets written in
Spanish and English between April 1st and July 10th, 2018. Overall, we
collected more than 7.4 million tweets written in English and more than
$470,000$ tweets in Spanish (see Table 4). The English tweets were generated
by about 1.8 million unique Twitter accounts while the Spanish tweets were
produced by approximately $220,000$ users. The difference between the number
of tweets and users collected in English and Spanish suggests that English-
speaking Twitter users tweeted more about this scandal using the selected
keywords than Spanish-speaking users, although this may be explained by the
greater volume of English tweets
overall222https://www.statista.com/statistics/267129/most-used-languages-on-
twitter/.

We cleaned our dataset in two ways. First, we removed all retweets to focus
our study on original user opinions and avoid analyzing duplicates. This step
downsized both datasets to approximately 20% of their original sizes. Second,
we attempted to eliminate tweets generated by automated accounts so our study
could indeed reflect people’s opinions. Unfortunately, there is not yet an
infallible mechanism to detect bots’ activity on Twitter. We chose to use
Botometer to identify potential bots (Davis et al., 2016). Botometer
implements a machine learning algorithm that has achieved high accuracy
($0.94$) in detecting both simple and sophisticated bots in prior work (Varol
et al., 2017). The algorithm has been trained to detect bots by analyzing
Twitter accounts’ metadata, their contacts’ metadata, tweets’ content and
sentiment, network patterns, and activity time series. The result is a score
that is based on how likely it is to be a bot. The score ranges from 0 to 1,
where lower scores indicate that the account behaves like a human and higher
scores signal bot-like behavior. Unfortunately, there is not yet agreement on
a threshold that can reliably distinguish bots from humans. To define
thresholds for our two datasets, we used the Ckmeans (Wang and Song, 2011)
algorithm to cluster the Botometer scores in each dataset into five groups,
with the first cluster including the accounts with the lowest Botometer scores
(more human-like) and the fifth group the users with the highest scores (more
bot-like). We reasoned that the fourth and fifth clusters in each dataset were
least likely to contain humans; therefore, we excluded them from our analysis.
Given that the Botometer analysis is time-consuming, we analyzed only a sample
of users. In this study, we focused on the users who contributed the highest
number of tweets in our datasets. In the future we plan to analyze users who
contribute less. Thus, we were able to classify $19,478$ accounts in the
Spanish dataset (40.6%) and 74,021 (12.9%) accounts in the English dataset.
Accounts with a Botometer score higher than $0.4745$ were labeled as bots in
the Spanish dataset. Those with a score higher than $0.4849$ were considered
bots in the English dataset. These users and their tweets were removed from
our analysis. As a result, our final Spanish dataset includes $15,531$ users
who tweeted $50,559$ times about the Cambridge Analytica scandal. The English
dataset comprises $60,491$ accounts that generated $446,462$ tweets about it.
Table 4 details these figures.

Table 1. Size of the Spanish and English datasets before and after data
cleaning

Dataset | Spanish | English
---|---|---
| #Tweets | #Users | #Tweets | #Users
Total | 472,363 | 222,352 | 7,476,988 | 1,846,542

Table 2. Size of the Spanish and English datasets before and after data
cleaning

Dataset | Spanish | English
---|---|---
| #Tweets | #Users | #Tweets | #Users
Total | 472,363 | 222,352 | 7,476,988 | 1,846,542
Without retweets | 106,656 | 47,951 | 1,572,371 | 574,452
Most active | 70,393 | 19,478 | 741,694 | 74,021
Humans | 50,559 | 15,531 | 446,462 | 60,491

Table 3. Size of the Spanish and English datasets before and after data
cleaning

Dataset | Spanish | English
---|---|---
| #Tweets | #Users | #Tweets | #Users
Total | 472,363 | 222,352 | 7,476,988 | 1,846,542
Without retweets | 106,656 | 47,951 | 1,572,371 | 574,452
Most active users | 70,393 | 19,478 | 741,694 | 74,021
Humans | 50,559 | 15,531 | 446,462 | 60,491

Table 4. Size of the Spanish and English datasets collected

Dataset | Spanish | English
---|---|---
| #Tweets | #Users | #Tweets | #Users
Total | 472,363 | 222,352 | 7,476,988 | 1,846,542

To identify key topics in the resulting Spanish and English datasets, we used
the Meaning Extraction Method (MEM) (Chung and Pennebaker, 2008). MEM is a
topic modeling technique that can infer “what words are being used together,
essentially resulting in a dictionary of word-to-category mappings from a
collection of texts”(Boyd, 2017). After applying principal component analysis
over this dictionary it is possible to identify words that can be grouped into
themes or topics. This method has been identified as well-suited for cross-
cultural and cross-language research (Boyd, 2017). MEM has been used to find
themes in different contexts, such as: mental health (Wolf et al., 2010;
Ramirez-Esparza et al., 2008); personality (Boyd and Pennebaker, 2015; Griffin
and Fingerman, 2018) and values (Wilson et al., 2016).

We employed the Meaning Extraction Helper (MEH) software (Boyd, 2014), tool
that can automate the majority of the MEM process (Boyd, 2017). The software
contains a default list of Spanish and English stopwords; all these words were
removed. Also, the tool allowed us to apply text segmentation by whitespace,
conduct lemmatization and run Twitter-aware tokenization. To assist in
lemmatization tasks, a conversion list was used to fix common misspellings
(e.g “hieght” to “height”) and convert “textisms” (e.g, “bf” to “boyfriend”).
No stemming algorithm was used. We computed the frequency of each unigram as
the percentage of tweets that contain it. The 300 most frequent unigrams were
kept. We obtained a csv file with values of 1 and 0 indicating the
corresponding unigrams’ presence or absence, respectively, for each tweet.

Principal component factor analysis (PCA) was run over the MEH results. PCA
was performed with varimax rotation to ensure that all resulting components
are independent from each other. We conducted PCA with 5, 8, 11, 30 and 100
components. In both the Spanish and English datasets, 11 components gave the
best results, with fit based upon diagonal scores of $0.55$ and $0.9$,
respectively. This metric is a goodness of fit statistic, where values closer
to 1 indicate better fit. The selected components accounted for $9\%$ of the
total variance of the Spanish data and $13\%$ of the total variance of the
English data.

To obtain the most representative words of each resulting component, we
selected the words with factor loadings above $0.1$, as recommended in (Boyd,
2017). The words were sorted according to their contribution to the component.
Additionally, a python script was used to identify the top-30 tweets most
related to each component. Using the most representative words and tweets by
component, two authors examined and conceptualized the theme represented by
each component, assigned a representative name and determined its relevance to
our research question.

Finally, we used GeoNames API333http://www.geonames.org/ to geo-locate all
human-like accounts (see Table 4). Table 5 reports the proportion of tweets
and users by the top-10 countries in each of our datasets. Most accounts could
not be geo-located. The remaining accounts reveal that Spain and the US
account for the majority of tweets and users in the Spanish and English
datasets, respectively.

Table 5. Top ten most frequent user location in the English and Spanish
datasets

Spanish | English
---|---
Country | % tweets | % users | Country | % tweets | % users
not found | 43.5% | 46.05% | not found | 41.6% | 44.9%
Spain | 17.5% | 16.43% | U.S | 33.6% | 31.1%
Mexico | 10.3% | 9.90% | U.K | 6.2% | 6.9%
Venezuela | 5.6% | 3.56% | India | 3.3% | 3.0%
Argentina | 5.4% | 5.41% | Canada | 2.4% | 2.3%
Colombia | 2.8% | 2.98% | Australia | 1.1% | 1.4%
U.S | 2.3% | 2.40% | France | 1.0% | 0.6%
Chile | 2.0% | 2.35% | Germany | 0.9% | 0.7%
Peru | 1.3% | 1.41% | U.A.E | 0.6% | 0.3%
Ecuador | 1.2% | 1.25% | Netherlands | 0.5% | 0.4%
Brazil | 0.9% | 0.34% | Ireland | 0.4% | 0.4%

##  5\. Results

The words that clustered together to form coherent themes in the English and
Spanish corpora are available online444https://github.com/gonzalezf/LA-WEB-
Paper. Tables 6 and 7 report the seven words with the highest loadings by
component in the English and Spanish datasets, respectively. The tables also
show the proportion explained (PE) by each of them according to the factor
analysis. This number is proportional to the number of tweets associated with
each theme. Table 9 presents the key themes in Twitter activity about the
Cambridge Analytica scandal in Spanish and English. The themes are ordered
according to their relevance to our research question.

Table 6. Terms by themes in the Spanish dataset #ID | PE (%) | Words with the highest loadings by component
---|---|---
word1 | word2 | word3 | word4 | word5 | word6 | word7
S1 | 13% | animals | lottery | predict | live | program | listen | win
S2 | 11% | cambridge | analytica | million | user | scandal | data | affect
S3 | 10% | congress | error | zuckerberg | sorry | mark | usa | senate
S4 | 9% | rgpd | protection | gdpr | data | regulation | privacy | dataprotection
S5 | 9% | marketing | digitalmarketing | analytic | publicity | digital | google | youtube
S6 | 9% | press | like | followers | platform | follow-us | come | digital
S7 | 9% | red | social | twitter | instagram | youtube | facebook | follow-us
S8 | 8% | parliament | european | zuckerberg | mark | ask | hearing | sorry
S9 | 8% | stop | ask | join | people | red | create | senate
S10 | 8% | communitymanager | socialmedia | blog | socialnetworks | news | work | facebook
S11 | 7% | message | messenger | user | send | million | remove | delete
Table 7. Terms by themes in the English dataset #ID |  PE (%) | Words with the highest loadings by component
---|---|---
word1 | word2 | word3 | word4 | word5 | word6 | word7
E1 | 24% | newyorkcity | newyork | nyc | ny | career | code | hire
E2 | 12% | rsi | btc | signal | min | eth | bitcoin | crypto
E3 | 10% | machinelearn | deeplearn | artificialintelligence | ml | robotic | dl | ai
E4 | 10% | chatbot | infosec | databreach | hack | cybersecurity | crypto | blockchain
E5 | 9% | iot | iiot | smartcity | digitaltransformation | innovation | infographic | startup
E6 | 7% | cambridge | analytica | election | campaign | voter | vote | brexit
E7 | 7% | data | user | facebook | privacy | access | personal | law
E8 | 6% | zuckerberg | mark | testify | ceo | congress | committee | senate
E9 | 6% | trump | president | donald | white | house | democrat | america
E10 | 5% | social | media | twitter | instagram | censorship | conservative | facebook
E11 | 5% | late | daily | thanks | bigdata | remove | social | ai
Table 8. Themes in the Spanish and English datasets Spanish | English
---|---
Theme | #ID | Theme | #ID
Cambridge Analytica’s impact on political issues | S2 |  Cambridge Analytica’s impact on political issues | E6
Mark Zuckerberg’s Senate hearing in the US | S3 |  Mark Zuckerberg’s Senate hearing in the US | E8
General Data Protection Regulation | S4 |  General Data Protection Regulation | E7
Zuckerberg in front of the European Parliament | S8 |  Facts and opinions about Donald Trump | E9
Opinions about Mark Zuckerberg | S9 |  Stop censorship on social media | E10
Facebook deletes Zuckerberg’s private messages | S11 |  Cryptocurrencies | E2
Digital marketing | S5 |  Artificial intelligence | E3
Promoting subscription to social platforms | S7 |  Blockchain | E4
Promoting likes in social platforms | S6 |  Internet of things | E5
Lottery results | S1 |  News about privacy on social media | E11
News and random facts | S10 |  Hiring tech jobs in New York | E1

Three key themes emerge in both languages. Spanish and English speakers talk
about:

  * •

“Cambridge Analytica’s impact on political issues”

  * •

“Mark Zuckerberg’s Senate hearing in the USA,” and

  * •

“General Data Protection Regulation.”

However, differences appear in how these themes are articulated. In regard to
the first topic, the scandal’s connection to Russia was much less relevant for
Spanish speakers than for English speakers. English tweets focus on how Russia
might have used Cambridge Analytica to intervene in the 2016 US elections and
the UK Brexit campaign. For example, this component includes the following
tweet: “@ianbremmer @billmaher @RealTimers You want to know how Brexit
happened and Trump got to win? Cambridge Analytica, Bannon, Mercers and the
Bot Farms in Russia. They started testing MAGA, Build the Wall, Lock Her Up,
Anti-Muslim sentiment, Anti-Immigrant propaganda. Putin worked hard at it
since 2012”. On the other hand, the token Russia does not appear as a
representative word of this theme in the Spanish dataset. Instead, Spanish
tweets are centered on Cambridge Analytica’s closure as a result of the
scandal. This behavior could be explained by the users’ country of residence
and its closeness to a salient political issue related to Cambridge Analytica.
The US accounts for the largest share of users who report their location in
our English dataset (see Table 5). It is well known that many people from this
country have apprehensions about Russia since the Cold War. Furthermore,
recent research has found evidence that Americans express “continued mistrust
of Russia and a majority think Russia tried to interfere in the 2016 election”
(Smeltz and Wojtowicz, 2018). We believe that this is a plausible reason to
explain this difference between the English and Spanish tweets.

While both datasets include a topic about “Mark Zuckerberg’s Senate hearing in
the US”, their tweets’ verb tenses differ. English-speaking users tend to
tweet about this topic in future or present tense. These tweets reflect either
certain level of anticipation of the event or live reports on how the event
was unfolding. An example of these tweets is: “Facebook CEO Mark Zuckerberg
will testify in front of a joint hearing of the Senate Judiciary and Commerce
Committees today. Senators will demand answers from Zuckerberg about Facebooks
failure to protect up to 87 million users’ private information
https://t.co/9qpdqxjM7F https://t.co/aLlZtMMVCC”. On the other hand, Spanish-
speaking users are more likely to discuss this issue in past tense. They
comment on sentences that Zuckerberg said during the hearing, putting special
emphasis on the moment when he assumed responsibility for what has happened.
For instance, translated Spanish tweets state: “From #Whoknows to
#ThroughMyFault, the change of attitude of #MarkZuckerberg. ‘It was my
mistake, and I’m sorry, I started #Facebook and I’m responsible for what
happens here’: Mark Zuckerberg before the Congress of #EEUU
https://t.co/UX2QwT7pFw”, and “Zuckerberg takes full blame for the abuse of
Cambridge Analytica before the US Senate: ‘It was my mistake, and I’m sorry’
https://t.co/VjH9ocdLcB”. The difference in tenses (English future/present and
Spanish past) may be explained by a delay in news reporting (volume of the
Spanish tweets in our dataset relating to this particular topic tended to peak
about 24 hours after the peak occurred in the English tweets) and translation
to another language as the event occurred in an English-speaking country.

In the case of the “General Data Protection Regulation” (GDPR) theme, a
tendency to attribute responsibility to companies for not attempting to comply
with the GDPR was present in the English data, but not in the Spanish tweets.
The English dataset includes accusatory tweets to companies such as Facebook
and Google for trying to dodge GDPR rules, e.g. “Facebook and Google are
pushing users to share private information by offering invasive and limited
default options despite new EU data protection laws aimed at giving users more
control and choice https://t.co/x0srYwZeUz”, “If a new European personal data
regulation (aka #GDPR) went into effect tomorrow, almost 1.9bn #Facebook users
around the world would be protected by it. The online social network is making
changes that ensure the number will be much smaller.https://t.co/UXonA0mTCs
#privacy https://t.co/gqFaMYq9Su”, and “Facebook moves billions of
international user accounts to California to avoid European privacy law
https://t.co/nC2ccf2ClR”. On the other hand, Spanish-speaking users tweet
about GDPR to describe it and inform local companies how to prepare for it.
Some translated tweets belonging to this component are: “ General #Data
Protection Regulation (#GPDR) is a new #law of the European Union and it will
enter into force on May 25th. Do you know what it is? Do you know its
characteristics? Is your #company ready? Get up to date by clicking on the
following link! https://t.co/AeKPR4w6Q9 https://t.co/rdlzbrAU1n”, and “On May
25th, the General Protection #Data Regulation (GDPR) of the EU, one of the
most modern regulations regarding personal data use by companies and
institutions, begins to be applied. There will be some repercussions in
#Chile. https://t.co/XNbmALUcdD”

Three other emerging themes are unique to Spanish speakers and are relevant to
our research question as they all refer to the founder of Facebook and its
role in different aspects of the scandal. These topics are:

  * •

“Zuckerberg in front of the European Parliament”

  * •

“Opinions about Mark Zuckerberg,” and

  * •

“Facebook deletes Mark Zuckerberg’s private messages.”

The first topic focused on Zuckerberg’s laments for the situation, e.g. “Mark
Zuckerberg apologized to the European Parliament for the data breach. The
founder of Facebook acknowledged on Tuesday that the tools of the social
network were used ‘to do harm’. https://t.co/aeDUVViUhq” We note that this
theme only emerges in the Spanish tweets. Again, we attribute this distinction
to the users’ country of residence. Both datasets include users whose self-
reported location is in Europe; however, they are the majority only in the
Spanish dataset where Spain is associated with more tweets than any other
country (see Table 5). Thus, the hearing in the EU Parliament is much more
prominent in the Spanish dataset.

The second topic contains supporting and accusatory tweets for Mark Zuckerberg
in relation to the Cambridge Analytica scandal. This theme includes tweets
such as the following: “what they do not forgive to Zuckerberg is that
Facebook has unwittingly helped Trump triumph (which I do not think), because
when Facebook censored pages of right-wing groups for nothing (the same Mark
Zuckerberg has said that he worries about the leftist prejudice of his staff),
there they said nothing.”

The third theme groups together tweets about the option to delete private
direct messages on Facebook. This functionality apparently was available to
Mark Zuckerberg in the past while no other user could use it. Spanish speakers
tweet about the special privileges of Zuckerberg as a Facebook user and about
the possibility that such functionality could become available to everyone.
Example translated tweets are: “Although you can not delete your messages sent
from another person’s inbox in Messenger, Facebook has done so in the case of
Mark Zuckerberg and other company executives. https://t.co/PkJByaesdT”, and
“It is clear that Mark Zuckerberg is the only one who can have the God Mode of
Facebook because only he can erase messages sent through Facebook Messenger, a
feature that will be released for everyone after this scandal.”

Two themes are unique to English and are considered slightly relevant to our
research question. They are conceptualized as:

  * •

“Stop censorship on social media,” and

  * •

“News about privacy on social media.”

First, English-speaking users protest about censorship on social media
companies such as Facebook, Twitter, Instagram and Google. They demand freedom
of speech. This component includes the following tweet: “@facebook Social
media censorship is tearing out our tongues! Is Facebook a neutral public
forum? STOP CENSORSHIP! Demand an #InternetBillOfRights #IBOR #MAGA
#InvasionOfPrivacy #HumanRights #DeleteFacebook #Censorship #Twitter
#Censorship #Google #FaceBook #Instagram @realDonaldTrump ”. The next topic
includes informative tweets (usually from electronic newspapers) that relate
to social media and privacy, e.g. “The latest The GDPR and Data Protection
Daily! https://t.co/GZBscXkxPl Thanks to @simpledatainc @Colt_Technology
@content_app #bigdata”.

Other key themes arise from both datasets, but we consider them not
sufficiently relevant to our research question. In the Spanish dataset, these
themes are: “Digital marketing”, “Promoting likes in social platforms”,
“Promoting subscription to social platforms”, “Lottery results”, and “News and
random facts”. In English, there are the following topics: “Cryptocurrencies”,
“Artificial Intelligence”, “Blockchain”, “Internet of Things”, “Facts and
opinions about Donald Trump,” and “Hiring tech jobs in New York.”

##  6\. Discussion

Table 9. Themes in the Spanish and English datasets

Spanish | English
---|---
Cambridge Analytica’s impact on political issues | Cambridge Analytica’s impact on political issues
Mark Zuckerberg’s Senate hearing in the US | Mark Zuckerberg’s Senate hearing in the US
General Data Protection Regulation | General Data Protection Regulation
Zuckerberg in front of the European Parliament | Stop censorship on social media
Opinions about Mark Zuckerberg | News about privacy on social media
Facebook deletes Zuckerberg’s private messages | Facts and opinions about Donald Trump
Digital marketing | Cryptocurrencies
Promoting subscription to social platforms | Artificial intelligence
Promoting likes in social platforms | Blockchain
Lottery results | Internet of things
News and random facts | Hiring tech jobs in New York

Our study of tweets in Spanish and English about the Cambridge Analytica data
misuse scandal allowed us to conduct a cross-language comparison of their main
emerging themes. Out of eleven topics, three are common to both languages.
However, they present meaningful differences in their articulation. Three
other relevant themes are unique to the Spanish data and two relevant topics
only arise from the English tweets. These findings show the potential of our
algorithm-based approach for cross-language comparisons of tweets to identify
similarities and (nuanced) differences on privacy-related views across people
from different countries.

We discuss here two underlying patterns that could explain the distinctions we
found between Spanish and English speakers.

First, we observe a tendency for a local perspective to appear in tweets about
this data privacy scandal. Tweets in English provide elaborate rationales
behind the Cambridge Analytica’s impact on political issues. They often relate
them to Russia’s actions. This is a common argument among Americans (Smeltz
and Wojtowicz, 2018), who are also the most active contributors of tweets in
our English dataset. This kind of rationale is much less visible in tweets in
the Spanish dataset, where most users are not located in the US. Furthermore,
there is a large component of tweets about Mark Zuckerberg’s hearing in front
of the US Senate in both languages; however, only the Spanish corpus has a
theme related to Zuckerberg’s audience in the European parliament. Again, we
relate this finding to the users’ country of residence. Our Spanish dataset
consists primarily of European users. This could explain why this event became
more salient only in this language. Another finding that can be explained by a
local perspective is the demand for freedom of speech in the English tweets.
This theme does not emerge at all in the Spanish data. Free speech is a
central value in the US (Alvarez and Kemmelmeier, 2018), but it is not as
prominent in Spanish-speaking countries.

Second, English tweets often attribute responsibilities to governments and
organizations while Spanish tweets tend to question individual actions. For
example, English-speaking users tend to tweet accusatory statements about big
companies such as Google and Facebook dodging GDPR. On the other hand, these
large companies are very rarely questioned about their acts in the Spanish
data. Instead, it is possible to find many tweets that support Mark Zuckerberg
and blame Facebook users for being irresponsible by failing to read Facebook’s
terms of service and neglecting to protect their personal data.

Research on cross-cultural differences could provide a framework to further
explore the second pattern. This research argues that national cultures could
be characterized by their scores on a small number of dimensions (Leidner and
Kayworth, 2006). Hofstede’s cultural dimensions (Hofstede, 1983) have been
widely used to study the relationship between people’s culture and technology
(Leidner and Kayworth, 2006). We hypothesize that one of these dimensions,
“power distance”, could explain differences between Spanish and English
speakers on responsibility attribution. This dimension is defined as: “the
extent to which the less powerful members of institutions and organizations
within a country expect and accept that power is distributed unequally”
(Hofstede, 2001). In countries with a high level of power distance, people
tend to accept hierarchies without further justification and authority is
hardly questioned. Most Anglo and Nordic countries are characterized by small
power distance scores. The opposite was found in Latin American countries.
These nations often score high in power distance (Hofstede, 1983). This
pattern could explain why many tweets that question authorities (e.g.,
governments and companies) occur in English while they are much less present
in Spanish. While this is a plausible path of reasoning, cross-cultural
research has its own limitations (Terlutter et al., 2006; Tung and Verbeke,
2010); therefore, additional work is needed to confirm or refute this
explanation.

In both the English and the Spanish dataset we found tweets from spammers and
self-promoters’ accounts. These accounts tend to use hashtags relevant to
popular online discussions (e.g: #Facebook, #privacy) to promote their
conversations and possibly share malicious links. These tweets account for the
themes that were considered irrelevant in our analysis.

###  6.1. Limitations and future work

As in any study, our research has limitations that need to be taken into
consideration. We collected data through the standard streaming Twitter API
and by using specific hashtags and keywords. Thus, we only had access to a
small sample of all the tweets about the scandal. Nevertheless, studies have
shown that this API is highly correlated with the full Twitter stream
(Leetaru, 2019). While we attempted to identify and remove bot activity from
our data, and the threshold we defined lies between the recommended range
[$0.43$-$0.49$] (Varol et al., 2017), our dataset contains both false
positives and negatives, which could affect the results. Also, difficulties
interpreting MEM results could occur due to the lack of contextual information
such as “information about valence and timing that is obscured in the
analysis” (Fitzpatrick and Armstrong, 2010). Further qualitative analysis may
help with future interpretation.

Future work includes investigating the English and Spanish corpora in order to
strengthen the analysis of shared themes. This could be done, for example, by
incorporating measurements of similarity between texts. Using bigrams and
trigrams as inputs of the MEM analysis could help to contextualize the topic
components. Also, it could be interesting to compare MEM results with other
topic modeling techniques such as LDA or LDA2Vec. Furthermore, exploring the
data over the time dimension should be considered because it may allow the
detection of topic changes over time.

##  7\. Conclusions

We proposed a methodology to conduct a cross-language study of Twitter
activity about a major data privacy leakage: the Cambridge Analytica scandal.
We collected tweets about it in English and Spanish during 100 days, processed
them to remove tweets generated by bots, and conducted MEM to identify the key
themes in each language. We conceptualized the themes and compared them. We
identified five topics that were unique to only one of the languages. We also
found three common topics: “Cambridge Analytica’s impact on political issues,”
“Mark Zuckerberg’s Senate hearing in the USA,” and “General Data Protection
Regulation.” Nevertheless, we detected dissimilarities in how these topics
were formulated in each language. We proposed two plausible patterns that may
explain these differences. First, reactions to a data privacy scandal reflect
the users’ local perspectives. Second, there is a tendency for English
speakers to assign responsibilities to governments and organizations while
Spanish speakers tend to attribute them to people.

The contributions of our work and findings are twofold. First, we proposed a
method to leverage social media data to investigate reactions to a data
privacy scandal in two languages that cover a multi-national audience. Second,
we found cross-language differences on privacy-related views that go beyond
measuring levels of privacy concerns, as defined in widely-adopted surveys
such as (Smith et al., 1996). We expect that future work can deepen the
understanding of these differences through diverse methods of research.

## Acknowledgments

This collaboration was possible thanks to the support of the Fulbright
Program, under a 2017-18 Fulbright Fellowship award. This work was also
partially funded by CONICYT Chile, under grant Conicyt-Fondecyt Iniciación
11161026. The first author acknowledges the support of the PIIC program from
Universidad Técnica Federico Santa María and CONICYT-
PFCHA/MagísterNacional/2019 - 22190332.

## References

  * (1)
  * Acar (2013) Adam Acar. 2013\.  Culture and social media usage: Analysis of Japanese Twitter users.  _International Journal of Electronic Commerce Studies_ 4, 1 (2013), 21–32.
  * Alvarez and Kemmelmeier (2018) Mauricio J Alvarez and Markus Kemmelmeier. 2018.  Free Speech as a Cultural Value in the United States.  _Journal of Social and Political Psychology_ 5, 2 (2018), 707–735.
  * Bélanger and Crossler (2011) France Bélanger and Robert E Crossler. 2011.  Privacy in the digital age: a review of information privacy research in information systems.  _MIS quarterly_ 35, 4 (2011), 1017–1042.
  * Boyd (2014) RL Boyd. 2014.  MEH: Meaning Extraction Helper (Version 1.0. 6)[Software].
  * Boyd (2017) Ryan L Boyd. 2017\.  Psychological text analysis in the digital humanities.  In _Data analytics in digital humanities_. Springer, 161–189.
  * Boyd and Pennebaker (2015) Ryan L Boyd and James W Pennebaker. 2015.  Did Shakespeare write Double Falsehood? Identifying individuals by creating psychological signatures with text analysis.  _Psychological science_ 26, 5 (2015), 570–582.
  * Choi et al. (2016) Kyungsub Stephen Choi, Il Im, and Gert Jan Hofstede. 2016\.  A cross-cultural comparative analysis of small group collaboration using mobile twitter.  _Computers in Human Behavior_ 65 (2016), 308–318.
  * Chung and Pennebaker (2008) Cindy K Chung and James W Pennebaker. 2008.  Revealing dimensions of thinking in open-ended self-descriptions: An automated meaning extraction method for natural language.  _Journal of research in personality_ 42, 1 (2008), 96–132.
  * Davis et al. (2016) Clayton Allen Davis, Onur Varol, Emilio Ferrara, Alessandro Flammini, and Filippo Menczer. 2016\.  Botornot: A system to evaluate social bots. In _Proceedings of the 25th International Conference Companion on World Wide Web_. International World Wide Web Conferences Steering Committee, 273–274.
  * Dinev et al. (2006) Tamara Dinev, Massimo Bellotto, Paul Hart, Vincenzo Russo, Ilaria Serra, and Christian Colautti. 2006\.  Privacy calculus model in e-commerce–a study of Italy and the United States.  _European Journal of Information Systems_ 15, 4 (2006), 389–402.
  * Fitzpatrick and Armstrong (2010) Marilyn R Fitzpatrick and Calli Renee Armstrong. 2010.  Beyond the tip of the iceberg: Exploring the potential of the meaning extraction method and aftercare e-mail themes.  _Psychotherapy research_ 20, 1 (2010), 86–89.
  * Griffin and Fingerman (2018) Eden M Griffin and Karen L Fingerman. 2018.  Online Dating Profile Content of Older Adults Seeking Same-and Cross-Sex Relationships.  _Journal of GLBT Family Studies_ 14, 5 (2018), 446–466.
  * Harris et al. (2003) Michael M Harris, Greet Van Hoye, and Filip Lievens. 2003\.  Privacy and attitudes towards internet-based selection systems: a cross-cultural comparison.  _International Journal of Selection and Assessment_ 11, 2-3 (2003), 230–236.
  * Hecht and Gergle (2010) Brent Hecht and Darren Gergle. 2010.  The tower of Babel meets web 2.0: user-generated content and its applications in a multilingual context. In _Proceedings of the SIGCHI conference on human factors in computing systems_. ACM, 291–300.
  * Hofstede (1983) Geert Hofstede. 1983\.  National cultures in four dimensions: A research-based theory of cultural differences among nations.  _International Studies of Management & Organization_ 13, 1-2 (1983), 46–74.
  * Hofstede (2001) Geert Hofstede. 2001\.  _Culture’s consequences: Comparing values, behaviors, institutions and organizations across nations_.  Sage publications.
  * Hong et al. (2011) Lichan Hong, Gregorio Convertino, and Ed H Chi. 2011\.  Language Matters In Twitter: A Large Scale Study. In _Proceedings of the Fifth International AAAI Conference on Weblogs and Social Media_. 518–521.
  * Hussein (2012) Basel Al-Sheikh Hussein. 2012\.  The Sapir-Whorf Hypothesis Today.  _Theory and Practice in Language Studies_ 2, 3 (2012), 642–646.
  * Jiang et al. (2017) Ke Jiang, Grace A. Benefield, Junfei Yang, and George Barnett. 2017.  Mapping Articles on China in Wikipedia An Inter-Language Semantic Network Analysis. In _Proceedings of the 50th Hawaii International Conference on System Sciences_. IEEE.
  * Kay and Kempton (1984) Paul Kay and Willett Kempton. 1984.  What is the Sapir-Whorf hypothesis?  _American anthropologist_ 86, 1 (1984), 65–79.
  * Kaya and Weber (2003) Naz Kaya and Margaret J Weber. 2003.  Cross-cultural differences in the perception of crowding and privacy regulation: American and Turkish students.  _Journal of environmental psychology_ 23, 3 (2003), 301–309.
  * Lapaire (2018) Jean-Rémi Lapaire. 2018\.  Why content matters. Zuckerberg, Vox Media and the Cambridge Analytica data leak.  _ANTARES: Letras e Humanidades_ 10, 20 (2018), 88–110.
  * Leetaru (2019) Kalev Leetaru. 2019\.  Is Twitter’s Spritzer Stream Really A Nearly Perfect 1% Sample Of Its Firehose?
  * Leidner and Kayworth (2006) Dorothy E Leidner and Timothy Kayworth. 2006.  A review of culture in information systems research: Toward a theory of information technology culture conflict.  _MIS quarterly_ 30, 2 (2006), 357–399.
  * Mirchandani (2018) Maya Mirchandani. 2018\.  To delete or not to# Delete Facebook, that is the question.
  * Ramirez-Esparza et al. (2008) Nairan Ramirez-Esparza, Cindy K Chung, Ewa Kacewicz, and James W Pennebaker. 2008.  The Psychology of Word Use in Depression Forums in English and in Spanish: Texting Two Text Analytic Approaches. In _Proceedings of the 2nd International Conference on Weblogs and Social Media_.
  * Smeltz and Wojtowicz (2018) Dina Smeltz and Lily Wojtowicz. 2018.  Public Opinion on US-Russian Relations in the Aftermath of the 2016 Election.  _SAIS Review of International Affairs_ 38, 1 (2018), 17–26.
  * Smith (2001) H Jeff Smith. 2001\.  Information privacy and marketing: What the US should (and shouldn’t) learn from Europe.  _California Management Review_ 43, 2 (2001), 8–33.
  * Smith et al. (2011) H Jeff Smith, Tamara Dinev, and Heng Xu. 2011.  Information privacy research: an interdisciplinary review.  _MIS quarterly_ 35, 4 (2011), 989–1016.
  * Smith et al. (1996) H Jeff Smith, Sandra J Milberg, and Sandra J Burke. 1996\.  Information privacy: measuring individuals’ concerns about organizational practices.  _MIS quarterly_ (1996), 167–196.
  * Terlutter et al. (2006) Ralf Terlutter, Sandra Diehl, and Barbara Mueller. 2006\.  The GLOBE study—applicability of a new typology of cultural dimensions for cross-cultural marketing and advertising research.  In _International advertising and communication_. Springer, 419–438.
  * Tung and Verbeke (2010) Rosalie L Tung and Alain Verbeke. 2010.  Beyond Hofstede and GLOBE: Improving the quality of cross-cultural research.  _Journal of International Business Studies_ 41, 8 (01 Oct 2010), 1259–1274.
  * Ur and Wang (2013) Blase Ur and Yang Wang. 2013.  A cross-cultural framework for protecting user privacy in online social media. In _Proceedings of the 22nd International Conference on World Wide Web_. ACM, 755–762.
  * Varol et al. (2017) Onur Varol, Emilio Ferrara, Clayton A Davis, Filippo Menczer, and Alessandro Flammini. 2017\.  Online human-bot interactions: Detection, estimation, and characterization.  _arXiv preprint arXiv:1703.03107_ (2017).
  * Wang and Song (2011) Haizhou Wang and Mingzhou Song. 2011.  Ckmeans. 1d. dp: optimal k-means clustering in one dimension by dynamic programming.  _The R journal_ 3, 2 (2011), 29.
  * Wang et al. (2017) Yuanyuan Wang, Muhammad Syafiq Mohd Pozi, Yukiko Kawai, Adam Jatowt, and Toyokazu Akiyama. 2017\.  Exploring Cross-cultural Crowd Sentiments on Twitter. In _Proceedings of the 28th ACM Conference on Hypertext and Social Media_. ACM, 321–322.
  * Wilson et al. (2016) Steven Wilson, Rada Mihalcea, Ryan Boyd, and James Pennebaker. 2016.  Disentangling Topic Models: A Cross-cultural Analysis of Personal Values through Words. In _Proceedings of the First Workshop on NLP and Computational Social Science_. 143–152.
  * Wolf et al. (2010) Markus Wolf, Cindy K Chung, and Hans Kordy. 2010.  Inpatient treatment to online aftercare: e-mailing themes as a function of therapeutic outcomes.  _Psychotherapy Research_ 20, 1 (2010), 71–85.


Lihua Xie, School of Electrical and Electronic Engineering, Nanyang
Technological University, 50 Nanyang Ave, Singapore, 639798

#  A demonstration of the LaTeX 2ε class file for SAGE Publications

Thien-Minh Nguyen11affiliationmark:  Shenghai Yuan11affiliationmark:  Muqing
Cao11affiliationmark:  Yang Lyu11affiliationmark:  Thien Hoang
Nguyen11affiliationmark:  Lihua Xie11affiliationmark:  11affiliationmark:
Nanyang Technological University, Singapore<EMAIL_ADDRESS>
# NTU VIRAL: A visual-inertial-ranging-lidar dataset, from an aerial vehicle
viewpoint

Thien-Minh Nguyen11affiliationmark:  Shenghai Yuan11affiliationmark:  Muqing
Cao11affiliationmark:  Yang Lyu11affiliationmark:  Thien Hoang
Nguyen11affiliationmark:  Lihua Xie11affiliationmark:  11affiliationmark:
Nanyang Technological University, Singapore<EMAIL_ADDRESS>
###### Abstract

In recent years, autonomous robots have become ubiquitous in research and
daily life. Among many factors, public datasets play an important role in the
progress of this field, as they waive the tall order of initial investment in
hardware and manpower. However, for research on autonomous aerial systems,
there appears to be a relative lack of public datasets on par with those used
for autonomous driving and ground robots. Thus, to fill in this gap, we
conduct a data collection exercise on an aerial platform equipped with an
extensive and unique set of sensors: two 3D lidars, two hardware-synchronized
global-shutter cameras, multiple Inertial Measurement Units (IMUs), and
especially, multiple Ultra-wideband (UWB) ranging units. The comprehensive
sensor suite resembles that of an autonomous driving car, but features
distinct and challenging characteristics of aerial operations. We record
multiple datasets in several challenging indoor and outdoor conditions.
Calibration results and ground truth from a high-accuracy laser tracker are
also included in each package. All resources can be accessed via our webpage
https://ntu-aris.github.io/ntu_viral_dataset/.

###### keywords:

Dataset, Aerial Robot, Autonomous System, Simultaneous Localization and
Mapping

##  1 Introduction

Over the years, autonomous systems have made significant progress. One of the
most crucial factors for these advancements can be attributed to the public
data suites. On one hand, these datasets can waive the inhibitive requirements
on budget and manpower, e.g. hardware development, calibration, field
operations, etc. for individual researchers. Hence, researchers can easily
investigate new navigation schemes and put them to test across a variety of
scenarios and environments. On the other hand, the benchmark tools also help
streamline the verification, evaluation and comparison between proposed
methods, thus allowing navigation methods to be fairly rated and ranked based
on a common basis.

Indeed, there are many datasets for a variety of tasks on autonomous vehicles,
e.g. object recognition, stereo depth perception, scene understanding, traffic
analysis, etc. In this paper we only focus on the datasets that can be used to
investigate navigation capability of aerial vehicles (AVs) in GPS-denied
environments, Fig. 1 illustrates a range of such environments where the
application of autonomous AV system is being investigated. We review some of
the datasets that are deemed most relevant to our targeted applications above.
Table 1 gives a summary of these datasets and their features, to the best of
our knowledge.

Figure 1: Typical inspection applications for autonomous AVs in GPS-denied
environments: (a) building complex, (b) cranes, (c) cargo ships (d) hangar

###  1.1 Related works: the ground-air dichotomy

Table 1: Notable public datasets divided by their ground and air focuses.

Dataset | Sensor used | Ground truth |  Mobile Platform
---|---|---|---
IMU | Camera | Lidar | UWB
| MIT DARPA,
---
Huang et al. (2010)
N/A |  | 4 Point Grey: 4×376×240 @10Hz
---
1 Point Grey: 752×480 @22.8Hz
| 3D Velodyne HDL-64E @15Hz
---
12 2D-SICK @75Hz
N/A | GPS/INS | Car
| Ford Campus,
---
Pandey et al. (2011)
acc/gyr @100Hz | LadyBug 3: 6×1600×600 @8Hz |  | 3D-Velodyne HDL-64E @10Hz
---
2 2D-Riegl LMS
N/A | GPS/INS | Car
| KITTI,
---
Geiger et al. (2013)
acc/gyr @10Hz |  | 2 Point Grey (gray): 2×1392×512 @10Hz
---
2 Point Grey (color): 2×1392×512 @10Hz
3D-Velodyne HDL-64E @10Hz | N/A | RTK GPS/INS | Car
| NCLT,
---
Carlevaris-Bianco et al. (2016)
acc/gyr @100Hz | LadyBug 3: 6×1600×1200 @5Hz |  | 3D-Velodyne HDL-32E @10Hz
---
2 2D-Hokuyo @10-40Hz
N/A |  | RTK-GPS
---
LIDAR SLAM
| Wheeled
---
Robot
| Oxford RobotCar,
---
Maddern et al. (2017)
acc/gyr @50Hz |  | BumbleBee: 2×1280×960 @16Hz
---
3 Grasshoper2: 3×1024×1024 @11.1Hz
| 2 2D-SICK @50Hz
---
3D-SICK @12.5Hz
N/A | GPS/INS | Car
| KAIST Urban,
---
Jeong et al. (2019)
| acc/gyr @200Hz
---
FOG @1000Hz
FLIR (color): 2×1280×560 @10Hz |  | 2 3D-Velodyne-16 @10Hz
---
2 2D-SICK @100Hz
N/A | SLAM | Car
| Newer College,
---
Ramezani et al. (2020)
acc/gyr @650Hz | D435i (Infrared): 2×848×480 @30Hz | 3D-Ouster-64 @10Hz | N/A | 6DOF ICP | Handheld
NTU VIRAL (Ours) |  | acc/gyr/mag
---
@385Hz
IDS (gray): 2×752×480@10Hz, | 2 3D-Ouster-16 @10Hz |  | 4 on UAV
---
3 anchors
| 3D Laser
---
Tracker
UAV
| UMA-VI,
---
Zuñiga-Noël et al. (2020)
acc/gyr @250Hz |  | BumbleBee: 2×1024×768 @12.5Hz
---
IDS (gray): 2×752×480 @25Hz,
N/A | N/A | SLAM | Handheld
| UZH FPV,
---
Delmerico et al. (2019)
| acc/gyr
---
@500/1000Hz
| Fisheye stereo: 2×640×480 @30Hz
---
mDAVIS: 346×260 + event @50Hz
N/A | N/A |  | 3D Laser
---
Tracker
UAV
| TUM VI,
---
Schubert et al. (2018)
acc/gyr @200Hz | IDS (gray): 2×1024×1024 @20Hz | N/A | N/A | 6DOF MoCap | Handheld
| Upenn Fast Flight,
---
Sun et al. (2018)
acc/gyr @200Hz | FLIR (gray): 2×960×800 @40Hz | N/A | N/A | GPS | UAV
| Zurich Urban MAV,
---
Majdik et al. (2017)
acc/gyr @10Hz | GoPro (color): 1920×1080 @30Hz | N/A | N/A |  | Aerial
---
Photogrammetry
UAV
| EuRoC,
---
Burri et al. (2016)
acc/gyr @200Hz | 2 MT9V034: 2×752×480 @20Hz | N/A | N/A |  | 6DOF MoCap /
---
3D Laser
Tracker
UAV

From Table 1, it is clear that there exist two distinct groups of datasets.
For those collected from cars or ground robots, a large number of sensors is
often present, especially lidar. For example, the MIT DARPA dataset is
comprised of 12 2D lidars and 1 64-channel lidar, along with a significant
number of cameras. Over the years, there appears to be a decline of interest
in 2D lidar (whose role has been gradually replaced by 3D lidar), along with a
reduced number of cameras. This reflects the trajectory of autonomous driving
research in the last decade, where in the beginning sensors had limited
capability and variety, thus quality was compensated by quantity. As the field
progresses, sensors have become much more compact and efficient, along with
better software and algorithms.

For aerial platforms, also referred to as Unmanned Aerial Vehicles (UAVs),
Micro Aerial Vehicles (MAVs), aerial robots or drones in the literature, we
can see a clear contrast to the ground-based datasets. In this case, due to
the payload constraint, 3D lidar is often left out of consideration. Instead,
the focus is often on high frame-rate camera systems for self-localization
capability at high speed and aggressive maneuvers, often under low lighting
conditions, with the goal of pushing the capability of Visual-Inertial
Odometry (VIO) and vision-based Simultaneous Localization and Mapping (SLAM)
systems to cover ever more extreme conditions.

It can be seen that the dataset we are presenting in this paper falls in the
middle of the two aforementioned classes. For those datasets using cars and
wheeled robots, the vehicle’s trajectory is often close to the ground plane,
without much abrupt translation or rotation motions. This is an advantage that
has been well exploited in many navigation systems, especially the lidar-
centric ones. However, they may no longer be effective with drastic changes in
roll, pitch and vertical motions commonly seen in a UAV dataset. Besides, due
to their long endurance, ground vehicle datasets can be conducted over a long
period of time and a large environment, and have access to GPS. In these cases
GPS data can be used for loop closure and/or ground truth, however it has
significant error and reduces the reliability and comparability of the
analysis. In contrast, the UAV datasets often have a shorter timespan and
cover a smaller environment, which make them less representative of real-world
scenarios. However, for semi-controlled environment, UAV datasets can feature
some artificial landmarks, i.e. visual markers or ranging anchors, to help
reduce localization drift, which is applicable in most targeted industry
inspection applications. Moreover, we can also employ high-accuracy laser
tracking methods to provide ground truth, which is desirable for a stringent
analysis and comparison of the localization methods.

###  1.2 A new dataset for autonomous drone.

Despite the aforementioned differences in the two classes of datasets, there
is no doubt on their usefulness for the intended applications. As mentioned
earlier, for the applications of aerial vehicles in our interest (Fig. 1),
there appears to be an absence of compatible public dataset for the relevant
scenarios. This motivates us to construct a novel sensor setup and offer a new
benchmark suite for advanced autonomous aerial systems that can be adopted by
the industry in the near future. Fig. 2 gives an overview of sensor setup.

Figure 2: The drone setup used for the data collection. A DJI M600 Hexacopter
is used to carry the payload comprising of 1 IMU, 2 lidars, 2 Cameras, 4 UWB
ranging nodes, and a crystal prism to be tracked by a total station to provide
ground truth.

Briefly speaking, we employ a high-frequecy IMU and a stereo camera rig
typical of UAV systems (Sec. 2.1, Sec. 2.3). Next, thanks to the current
advances in lidar hardware that have brought down its size and price, we find
that it is due time 3D lidar can be popularly used on aerial platforms, thanks
to many benefits of 3D lidar over a pure vision system. Hence, we integrate
two 3D lidars on our UAV platform. Different from previous datasets with 3D
lidar mounted on ground vehicles, ours features a much more complex motion in
3D space with frequent drastic rotational and translational moves. More
details on the lidars can be found in Sec. 2.2.

In addition to the aforementioned sensors, as highlighted in Table 1, our
dataset also features UWB range measurements. Indeed, it is a consensus that
drift is an intrinsic feature of onboard self-localization. However, our
recent works have shown that by using range measurements to some fixed
landmarks in an environment, one can reduce, or even eliminate such drift
Nguyen et al. (2020b, a). For semi-controlled environments such landmarks can
be easily deployed without much effort. More information on this sensor is
given in Sec. 2.4.

Finally, we employ a laser tracking system to provide high-
accuracy111https://github.com/ntu-aris/ntu_viral_dataset/blob/gh-
pages/docs/Leica_Nova_MS60_DS.pdf ground truth for the position estimate. The
experiments are conducted at various environments on the campus of Nanyang
Technological University (NTU), including both indoor and outdoor conditions.
The datasets are also recorded in rosbag format, which can be directly used on
Robot Operation System (ROS). We also include some codes that can be used for
the analysis of the localization information using our datasets. Several plug-
and-play examples of using state-of-the-art localization methods with our
datasets such as OpenVINS Geneva et al. (2020), VINS-Fusion Qin et al. (2018),
A-LOAM Zhang and Singh (2018), LIO-SAM Shan et al. (2020), M-LOAM Jiao et al.
(2020) are also provided. All of these resources can be found on our data
suite’s web page https://ntu-aris.github.io/ntu_viral_dataset/.

The rest of this paper is organized as follows: Sec. 2 presents our hardware
setup and the intended purpose for each sensor. Sec. 3 presents the
description of the environments and the flight tests. Sec. 4 describes the
organization, format, definitions and convention used in the datasets. Sec. 5
explains the calibration procedures. Sec. 6 describes our recommendation for
evaluation. Sec. 7 lists out the known issues in our implementation.

##  2 Sensor setup

A DJI M600 Pro222https://www.dji.com/sg/matrice600-pro hexacopter is used to
carry the sensor setup (Fig. 2). Table 2 provides a summary of the sensors and
their corresponding specifications. All of the messages are timestamped by
their publish time on ROS. Below, we provide a more detailed description of
these sensors.

Table 2: The sensors used in this dataset and their corresponding
specifications

No. | Sensor | Model | Topic name | Message Type | Rate (Hz)
---|---|---|---|---|---
1 |  IMU (Sec. 2.1) |  VectorNav (VN100) | /imu/imu | sensor_msgs/Imu | 385
/imu/magnetic_field | sensor_msgs/MagneticField | 385
/imu/temperature | sensor_msgs/Temperature | 385
2 |  Horizontal Lidar (Sec. 2.2) | OS1-16 Gen1 | /os1_cloud_node1/imu | sensor_msgs/Imu | 100
/os1_cloud_node1/points | sensor_msgs/PointCloud2 | 10
3 |  Vertical Lidar (Sec. 2.2) | OS1-16 Gen1 | /os1_cloud_node2/imu | sensor_msgs/Imu | 100
/os1_cloud_node2/points | sensor_msgs/PointCloud2 | 10
4 |  | Camera 1
---
(Sec. 2.3)
| uEye 1221 LE
---
(monochrome)
/left/image_raw | sensor_msgs/Image | 10
5 |  | Camera 2
---
(Sec. 2.3)
| uEye 1221 LE
---
(monochrome)
/right/image_raw | sensor_msgs/Image | 10
6 |  UWB Sensors (Sec. 2.4) | Humatics P440 | /uwb_endorange_info | uwb_driver/UwbRange | 68.571
/uwb_exorange_info | uwb_driver/UwbEcho | 5.714
/nodes_pos_sc | nav_msgs/Path | 5.714
7 |  | 3D Laser Tracker
---
(Sec. 2.5)
| MS60 Leica
---
TotalStation
/leica/pose/relative |  | geometry_msgs/PoseStamped
---
(orientation is not set)
20

###  2.1 IMU:

A VectorNav VN100333https://www.vectornav.com/products/VN-100 rugged IMU is
employed as the main inertial sensor in our system. It is also chosen to be
the center of the body frame where the extrinsics of other sensors are
referenced to. The sensor is configured to publish data at 400Hz, though the
effective rate is roughly 385Hz. Note that besides the angular velocity and
acceleration measurements, the topic /imu/imu also contains the orientation
estimate by the device’s internal Extended Kalman Filter, which also fuses
magnetometer with the aforementioned measurements.

###  2.2 3D lidars:

In this work, two 16-channel OS1 gen1444https://ouster.com/products/os1-lidar-
sensor lidars are configured so that the so-called horizontal lidar can scan
the objects in the front, back, left, and right sides of the UAV, while the
other so-called vertical lidar can scan the ground, front and back sides.
Thus, the two can complement each other well to maintain good observation on
the environment. Note that each lidar has an internal IMU that outputs angular
rate and acceleration measurements at 100Hz rate. Each lidar is loaded with
the latest V2 firmware for additional fields such as reflectivity, time, and
ambient. These quantities are also recorded in the bag files.

###  2.3 Stereo cameras

Two uEye 1221 LE555https://en.ids-imaging.com/store/ui-1221le-rev-2.html
monochrome-global-shutter cameras are mounted on the drone, facing directly
forward. The two are synchronized by an external trigger to capture and
publish image at almost the same time. Typically, the difference in the
timestamps of the images triggered at the same time is below 3 ms. This is
necessary if one needs to set a threshold to synchronize the images in the
buffer for stereo processing such as in VINS-Fusion Qin et al. (2018).

###  2.4 UWB ranging sensors

The UWB ranging network is setup in a similar manner with our previous works
Nguyen et al. (2020b, 2021a, 2018). Two Humatics P440 UWB
radios666https://humatics.com, given the IDs 200 and 201 in the network, are
mounted on the UAV, each has two antennae A and B extended to the four
corners. Hence we have a total of 4 ranging nodes on the UAV: 200.A, 200.B,
201.A, 201.B (see Fig. 2). Another three UWB nodes with IDs 100, 101, 102 are
used as anchors. As such in total we have 12 UAV-to-anchor ranging pairs.

The ranging sequence is programmed in a way such that all 12 ranging pairs are
executed equally. Each ranging step takes 25 ms. Hence with only one UAV
ranging node, ideally one can expect 40 Hz of ranging measurements on the
topic /uwb_endorange_info. However, in our case, there are two ranging nodes
(we program node 200 to range to one anchor while 201 ranges to another), thus
we can obtain twice the amount of range measurements, i.e. 80 Hz.
Nevertheless, we reserve some steps in the sequence to let the anchors range
among themselves and broadcast the measurements. Therefore the rate of the
UAV-to-anchor ranges, i.e. the /uwb_endorange_info topic is reduced to 68.571
Hz, while the rate of anchor-to-anchor ranges, published over the
/uwb_exorange_info topic, is 5.714 Hz, as shown in Tab. 2. Fig. 3 gives a
simple illustration of the ranging incidents over two time steps. Note that
due to the fact that the transmission power of the UWB depends on the relative
orientation between the antennae of the the two nodes, some ranging steps may
fail to return, or return an unreliable measurements. Thus the effective rate
of UWB measurement can vary through time, depending on the position of the UAV
relative to the anchors during flight.

Figure 3: Illustration of the ranging scheme and the external frame of
references. Refer to Sec. 2.4 for detailed descriptions.

The anchors are placed on tripods and they are adjusted to have the same
height from the ground. Thus, their nominal z coordinate is 1.5m. If we select
anchor 100 to be at the origin, and anchor 101 to be on the +x direction, and
+z direction is from the ground to the anchor nodes, then we can establish a
coordinate system using the right hand rule (the frame $\\{\texttt{W}\\}$ in
Fig. 3). Using this arrangement and the distances from the topic
/uwb_exorange_info we can easily estimate the coordinates $x_{1}^{*}$,
$x_{2}^{*}$, $y_{2}^{*}$. The coordinates of the anchor nodes are published in
the /nodes_pos_sc topic.

###  2.5 Ground truth

A Leica Nova MS60 MultiStation777https://Leica-geosystems.com/en-
sg/products/total-stations/multistation/Leica-nova-ms60 is used to track a
crystal prism mounted on the top of the UAV to provide ground truth for
position estimate. Note that the coordinate frame of this ground truth system
is aligned with gravity during the startup, and its z axis points at the
opposite direction of the gravity (Fig. 3). Moreover, since there is a
significant displacement between the prism and the body frame’s origin, the
accuracy analysis needs to take this into account. Sec. 6 discusses this issue
in details.

##  3 Dataset characteristics

The datasets are divided into three groups based on the environments, namely
the EEE, SBS and NYA sequences. The environments include both indoor and
outdoor locations on NTU campus. Fig. 4 presents some overviews of these
environments. Tab. 3 gives a brief summary of the sequences and their
statistics.

(a) EEE environment

(b) SBS environment

(c) NYA environment

Figure 4: Environments where the datasets are collected Table 3: Some statistics of each sequence Name | Time |  | Path
---
Length
| Average
---
Vel.
| Average
---
Ang. Vel
eee_01 | 398.7 s | 237.07 m | 0.677 m/s | 0.119 rad/s
eee_02 | 321.1 s | 171.07 m | 0.585 m/s | 0.101 rad/s
eee_03 | 181.4 s | 127.83 m | 0.800 m/s | 0.184 rad/s
nya_01 | 396.3 s | 160.30 m | 0.463 m/s | 0.150 rad/s
nya_02 | 428.7 s | 249.10 m | 0.657 m/s | 0.141 rad/s
nya_03 | 411.2 s | 315.46 m | 0.821 m/s | 0.125 rad/s
sbs_01 | 354.2 s | 202.30 m | 0.625 m/s | 0.101 rad/s
sbs_02 | 373.3 s | 183.57 m | 0.573 m/s | 0.145 rad/s
sbs_03 | 389.3 s | 198.54 m | 0.601 m/s | 0.120 rad/s

###  3.1 The EEE sequences

Three datasets are collected at the carpark in the center of the School of
EEE, NTU, hence named the EEE sequences. The area is surrounded by tall
building blocks on all side which can be conducive for lidar-based SLAM. Also,
reliable visual features can be detected on nearby objects such as trees, road
markings, and buildings (see Fig. 4(a)).

###  3.2 The SBS sequences

The SBS sequences are collected at an open square next to the School of
Biological Sciences, NTU. This area is surrounded by some low-rise buildings
with large glass surfaces. Also, visual features may only be detected on
objects far way, which can produce noisy depth. The drone’s trajectories in
these sequences are shown in Fig. 5.

Figure 5: Trajectories of the AV in the SBS sequences as recorded by the Leica
laser tracker.

###  3.3 The NYA sequences

The auditorium at NTU, shown in Figure 4(c), represents a challenging
environment for visual SLAM. These datasets feature some significant
challenges. For e.g. semi-transparent surfaces can be an issue for lidar SLAM,
while flight dynamics and low lighting conditions are difficult for visual
SLAM. Moreover, we also notice significant multi-path effects and signal loss
on the UWB measurements (Fig. 6).

(a) UWB ranges in sbs_03 dataset. The zoom-in plot shows a 5 cm fluctuation in
measurement when the drone was static.

(b) UWB ranges in nya_03 dataset, showing signal loss by UWB 200 mid-flight,
and multi-path effect on all ranging edges

Figure 6: UWB ranges in outdoor and indoor conditions

##  4 Dataset format

###  4.1 Files organization

The datasets are recorded into rosbag files during the flight, and saved on
the memory of the drone’s onboard computer (DJI Manifold 2C). The ground truth
data is recorded on a seperate computer, and is later temporally synchronized
and merged with the bag from the drone’s onboard computer, yielding a single
rosbag for each flight test.

Each rosbag file is accompanied by a set of files containing the calibration
parameters for each sensor. Fig. 7 illustrates the content of such a
calibration file. These files document the coefficients of the camera model,
sensor-to-body coordinate transforms, IMU noise, camera-IMU time delay, and
can be parsed using opencv and ROS APIs. In addition, some accessories to
utilize the pointclouds and UWB data are also provided with links on the data
suite’s web page.

Figure 7: Content of the calibration file camera_right.yaml of the eee_01
dataset.

###  4.2 Units and timestamps

All of the measurements are reported in SI units except the timestamps of the
messages are in ros/Time888http://wiki.ros.org/roscpp/Overview/Time format,
which can be found in the header.stamp field of the messages. The timestamps
are assigned by each sensor’s driver when the message is published over ROS.

In addition, the timestamp of each lidar pointcloud message matches with the
the end-time of the scan. Moreover, the pointcloud also contains the timestamp
of each point relative to the start-time of the scan. This will be useful if
one seeks to perform deskew operation on the pointcloud.

###  4.3 Message types

The message types for most sensors are standard ROS messages, however there
are some custom definitions for UWB. Specifically, for the UWB messages of
type uwb_driver/UwbRange, besides the distance measurement, the message also
contains other information such as signal over noise ratio, line-of-sight
self-diagnosis, antenna coordinates in the body frame, and the anchor
coordinates, which are fixed by using the first messages in /nodes_pos_sc. The
source code that defines these message can be found on our data suite’s web
page https://ntu-aris.github.io/ntu_viral_dataset.

###  4.4 Coordinate transformations

For common objects such as vectors, quaternions, rotation matrices recorded in
the rosbag files, the ROS conventions are followed. Here we shall explain the
extrinsics defined in the calibration files.

Let us take an instance to explain the convention used throughout the
datasets. In reference to Fig. 7, notice that the transform
${}^{\texttt{B}}_{\texttt{C}}\mathbf{T}\in SE(3)\subset\mathbb{R}^{4\times 4}$
is declared via the parameter T_Body_Cam (the ”body-cam” suffix follows a
parent-child order between the two frames of reference, following the
convention of ROS and the EuRoC dataset). This transform consists of the
rotation ${}^{\texttt{B}}_{\texttt{C}}\mathbf{R}\in
SO(3)\subset\mathbb{R}^{3\times 3}$ on the upper left corner and the
translation ${}^{\texttt{B}}_{\texttt{C}}\mathbf{t}\in\mathbb{R}^{3\times 1}$
on the upper right corner, i.e.
${}^{\texttt{B}}_{\texttt{C}}\mathbf{T}=\begin{bmatrix}{}^{\texttt{B}}_{\texttt{C}}\mathbf{R}&{}^{\texttt{B}}_{\texttt{C}}\mathbf{t}\\\
0&1\end{bmatrix}$.

For an object A observed in the camera frame $\\{\texttt{C}\\}$ with position
${}^{\texttt{C}}_{\texttt{A}}\mathbf{p}$ and orientation
${}^{\texttt{C}}_{\texttt{A}}\mathbf{R}$, its corresponding position and
orientation in the body frame B, denoted as
${}^{\texttt{B}}_{\texttt{A}}\mathbf{p}$ and
${}^{\texttt{B}}_{\texttt{A}}\mathbf{R}$, can be obtained by:

${}^{\texttt{B}}_{\texttt{A}}\mathbf{p}={}^{\texttt{B}}_{\texttt{C}}\mathbf{R}{}^{\texttt{C}}_{\texttt{A}}\mathbf{p}+{}^{\texttt{B}}_{\texttt{C}}\mathbf{t},\
\
{}^{\texttt{B}}_{\texttt{A}}\mathbf{R}={}^{\texttt{B}}_{\texttt{C}}\mathbf{R}{}^{\texttt{C}}_{\texttt{A}}\mathbf{R}.$
(1)

The object A in this case can be a 3D feature or an object that is being
tracked by the camera or the lidar, which in a typical SLAM algorithm needs to
be coupled with the extrinsics in the estimation process. Moreover, A can be
another coordinate system that we need to convert the coordinates from other
frames of references to.

##  5 Sensor calibration

Sensor fusion often requires prior knowledge of the spatial configuration of
multiple sensors. This section introduces our calibration pipelines, which is
divided into two parts, namely the rigid systems and flexible systems. The
cameras and IMU are mounted on the same rigid titanium-alloy-based 3D-printed
parts. Thus, their spatial relation has very small dynamic variance. The other
sensors are flexibly mounted as they are connected via some carbon fiber tubes
or dampers. These flexible parts will have large dynamic variance.

We first calibrate the rigid body system. The stereo cameras are calibrated
both intrinsically and extrinsically. We then find the IMU to camera transform
based on visual-inertial alignment. For the more flexible parts, the transform
may come with more considerable variance in the spatial relation. Therefore,
we mostly use the VICON system to calibrate between those sensors with some
help from other priors such as recognizable planes in the lab. All of the
calibration results can be verified using localization or mapping packages.

###  5.1 Stereo calibration

Our first goal is to find the intrinsic and extrinsic parameters of the stereo
cameras using an approach similar to Zhang (2000). We collect N well-
synchronized stereo images in front of a static calibration chessboard to
minimize the motion blur. Using pinhole camera and radial-tangential
distortion models, the intrinsics are calibrated using all of the image
sequences on the MATLAB calibration toolbox. We set a threshold for the
reprojection error and only use the smaller subset of images for the
extrinsics. We find that a large roll of chessboard pattern will enlarge the
reprojection error for this 120-degree FOV lens. Therefore, we keep a small
roll factor in calibration sequences. Finally, we verify the calibration
result by comparing the stereo-matching reprojected depth to lidar
measurement. The result can be well aligned with the lidar. The stereo
calibration image sequence is provided in the dataset portal.

###  5.2 Visual inertial calibration

We calibrate and refine the intrinsic and extrinsic parameters of IMU and
cameras by using the camera-IMU calibration
procedure999https://github.com/ethz-asl/kalibr/wiki/camera-imu-calibration of
the Kalibr package Furgale et al. (2013). In this procedure, the initial guess
of the camera intrinsic parameters is required and we obtain them from the
stereo calibration process in the previous part. The intrinsics (biases) of
IMU are calibrated beforehand using the manufacturer’s
software101010https://www.vectornav.com/resources/software, and the correction
is applied to the raw inertial measurement. At last, we collect the visual-
inertial sequence with full 6 DOF excitation and perform the visual-inertial
calibration. We have test the calibration result using VINS-Mono, VINS-Fusion,
OpenVINS, and we can obtain meaningful trajectories using the calibrated
parameters. A 20-ms time offset between IMU and camera timestamps is found by
the Kalibr package, which is also reported in the accompanying yaml file (Fig.
9). However this value is just for reference, and we do not make any
modification on sensor’s timestamps. The rosbag files used for calibration can
also be found at the website.

###  5.3 Other flexible parts compensation and ground truth

The dataset is captured from a payload bay mounted on our custom DJI M600 Pro
drone. Therefore, the triple GPS, compass, and prism on top of the drone are
not rigidly connected to the payload bay, i.e. their spatial relation can
change during flight. However, this is a necessary compromise to reduce
vibration effect on the quality of the lidar scan, camera image, and IMU. In
light of these conditions, we first use VICON to verify most of the other
extrinsic parameters by putting the VICON reflective ball at each sensor while
the drone is at rest on the floor. Then we exert some force on the dampened
payload bay to tilt it up to some maximum angle. The maximum displacement of
the prism between static configuration and maximum tilting angle is around
2cm. Based on our study, the drone only exhibits small control angles with a
minimized jerk during the flight tests. Therefore, the prism measurement is
relatively accurate to represent the ground truth. Note that we use hand-eye
calibration to align the ground truth with onboard altitude estimate obtained
from the drone’s internal barometer and IMU, which are independent from the
sensors on the payload.

To obtain a bound of the error in the temporal alignment between ground truth
and onboard sensors, we conduct several experiments with several state-of-the-
art localization methods and calculate their Absolute Trajectory Error (ATE)
with multiple versions of ground truth that are time-shifted by -0.5, -0.49,
-0.48,… 0.5 seconds. The results are reported in Fig. 8. We can see that
despite using different sensor combinations and methods, i.e. stereo camera-
IMU (VINS), lidar-IMU (LIO-SAM), and stereo camera-IMU-lidar-UWB (VIRAL SLAM),
all methods can achieve minimal ATE within the [-0.1, 0.1] interval. Thus, we
are confident that the temporal alignment error between ground truth and the
onboard sensors is within 0.1s in the current datasets. Table 4 presents the
ATE of the SLAM frameworks without time shift. A more comprehensive analysis
can be found in Nguyen et al. (2021b). Fig. 9 demonstrates a result of VIRAL
SLAM on one sequence.

Figure 8: The changes in ATE of different methods with respect to time shifts
of the ground truth. Note that except A-LOAM the selected methods all feature
loop closure and global optimization to eliminate the influence of drift. The
minimum of each curve is marked by a circle, and the time shift that minimizes
the ATE for each method is reported in the legend. It can be seen that all
visual-inertial, lidar-inertial, and visual-inertial-lidar-ranging methods can
achieve minimum error within the [-0.1, 0.1] interval.

Figure 9: VIRAL SLAM result using the eee_01 dataset. Ground truth is maked in red, the trajectory estimate in blue. The 3D map of the environment is visualized in blue and yellow points in the 3D visualisation below. Key frames are highlighted by green boxes. The ground truth and trajectory estimate are aligned using the 6DoF transformation calculated in (3). Table 4: ATE of state-of-the-art localization methods over NTU VIRAL datasets. The best odometry result is highlighted in bold, and the second best is underlined. All values are in meter. Dataset |  | VINS-
---
Fusion
| LIO-
---
SAM
| MLOAM
---
| VIRAL-
---
SLAM
eee_01 | 0.608 | 0.075 | 0.249 | 0.060
eee_02 | 0.506 | 0.069 | 0.166 | 0.058
eee_03 | 0.494 | 0.101 | 0.232 | 0.037
nya_01 | 0.397 | 0.076 | 0.123 | 0.051
nya_02 | 0.424 | 0.090 | 0.191 | 0.043
nya_03 | 0.787 | 0.137 | 0.226 | 0.032
sbs_01 | 0.508 | 0.089 | 0.173 | 0.048
sbs_02 | 0.564 | 0.083 | 0.147 | 0.062
sbs_03 | 0.878 | 0.140 | 0.153 | 0.054

##  6 Evaluation recommendations

From our experiments of state-of-the-art SLAM methods on NTU VIRAL, we believe
some issues regarding the evaluation process merit a detailed discussion. In
most cases, after applying a navigation method on our dataset, the user can
obtain a trajectory consisting of $N+1$ position estimates of the body frame,
denoted as
$\\{{}_{\texttt{B}}^{\texttt{W}}\hat{\mathbf{p}}_{t_{n}}\\}_{n=0}^{N}$, and a
trajectory of $M$ ground truth samples of the crystal prism
$\\{{}_{\texttt{P}}^{\texttt{L}}{\mathbf{p}}_{t_{m}}\\}_{m=0}^{M}$. From these
samples, the user would like to calculate the estimation accuracy of the
method based on some metrics. To facilitate the use of our dataset, MATLAB
scripts for this purpose are also included in the suite. We note that some
previous works such as Sturm et al. (2012); Zhang and Scaramuzza (2018) might
have briefly addressed some of the issues discussed below. However, we think a
more formal description can be of interest to some readers.

###  6.1 Are we tracking the same point?

Since the crystal prism has a displacement of almost 0.4 m from the body
frame’s origin, if we naively calculate the error between the body-centered
trajectory with the ground truth, there can be a bias of 0.4 m in the estimate
in the worst case scenario. Thus it is recommended that the user use the pose
estimate to calculate the estimated trajectory of the crystal prism, denoted
as $\\{{}_{\texttt{P}}^{\texttt{W}}\hat{\mathbf{p}}_{t_{n}}\\}_{n=0}^{N}$, for
comparison with ground truth. Specifically, using (1), we can obtain:

${}_{\texttt{P}}^{\texttt{W}}\hat{\mathbf{p}}_{t_{n}}={}_{\texttt{B}}^{\texttt{W}}\hat{\mathbf{p}}_{t_{n}}+{}_{\texttt{B}}^{\texttt{W}}\hat{\mathbf{R}}_{t_{n}}{}_{\texttt{P}}^{\texttt{B}}\mathbf{t}_{t_{n}},\
\forall n\in\\{0,1,\dots N\\},$ (2)

where ${}_{\texttt{B}}^{\texttt{W}}\hat{\mathbf{R}}_{t_{n}}$ is the
orientation estimate, and ${}_{\texttt{P}}^{\texttt{B}}\mathbf{t}_{t_{n}}$ is
the translation from the body origin to the prism reported in the dataset’s
calibration files. Hence, we can drop the subscript ${}_{\texttt{P}}$ as it is
implied in later parts.

###  6.2 Resampling

The goal of this task is to resample
$\\{{}^{\texttt{W}}\hat{\mathbf{p}}_{t_{n}}\\}_{n=0}^{N}$ and
$\\{{}^{\texttt{L}}{\mathbf{p}}_{t_{m}}\\}_{m=0}^{M}$ to obtain the sequences
$\\{{}^{\texttt{W}}\hat{\mathbf{p}}_{t_{k}}\\}_{k=0}^{K}$ and
$\\{{}^{\texttt{L}}{\mathbf{p}}_{t_{k}}\\}_{k=0}^{K}$ of the same length.
Specifically $\\{t_{k}\\}_{k=0}^{K}$ is a subset of $\\{t_{n}\\}_{n=0}^{N}$
that satisfies the following

$\displaystyle\exists\ t_{s}\in\\{t_{m}\\}_{m=1}^{M}\colon t_{s}\leq t_{k}\leq
t_{s+1},|t_{s}-t_{s+1}|<0.1,\forall t_{k}.$

Given the time $t_{s}$ above, the ground truth sample at time $t_{k}$ can be
linearly interpolated as
${}^{\texttt{L}}\mathbf{p}_{t_{k}}=\frac{t_{k}-t_{s}}{t_{s+1}-t_{s}}{}^{\texttt{L}}\mathbf{p}_{t_{s}}+\frac{t_{s+1}-t_{k}}{t_{s+1}-t_{s}}{}^{\texttt{L}}\mathbf{p}_{t_{s+1}}$.
We can further simplify the notation by denoting the trajectories as
$\\{{}^{\texttt{W}}\hat{\mathbf{p}}_{k}\\}_{k=0}^{K}$ and
$\\{{}^{\texttt{L}}{\mathbf{p}}_{k}\\}_{k=0}^{K}$.

###  6.3 Coordinates transform

At the final stage, we can see that
$\\{{}^{\texttt{W}}\hat{\mathbf{p}}_{k}\\}_{k=0}^{K}$ and
$\\{{}^{\texttt{L}}{\mathbf{p}}_{k}\\}_{k=0}^{K}$ are still w.r.t. different
coordinate frames. Hence they cannot be directly compared. Following the
common public benchmarking tools by Sturm et al. (2012) and Zhang and
Scaramuzza (2018), we adopt the following coordinate transform
$({}^{\texttt{L}}_{\texttt{W}}\mathbf{R},{}^{\texttt{L}}_{\texttt{W}}\mathbf{t})$
to bring the two trajectories to a common frame of reference:

$({}^{\texttt{L}}_{\texttt{W}}\mathbf{R},\
{}^{\texttt{L}}_{\texttt{W}}\mathbf{t})=\operatorname*{arg\,min}_{(\mathbf{R},\
\mathbf{t})}\left(\sum_{k=0}^{K}\left\lVert\mathbf{R}{}^{\texttt{W}}\hat{\mathbf{p}}_{k}+\mathbf{t}-\mathbf{p}_{k}\right\rVert^{2}\right).$
(3)

The above was shown to have a closed-form solution in Umeyama (1991), and the
value of the summation is referred to as ATE of the algorithm’s estimate over
the dataset. A MATLAB script is also provided in this data suite to perform
this task.

##  7 Known issues and limitations

Despite careful design and execution of the data collection experiments, we
are aware of several practical issues which pose some challenges and limit the
achievable accuracy. The specifics are given below

###  7.1 Camera exposure setting

Stereo camera exposure setting is a challenging topic for any global-shutter
stereo camera. Since the incoming photons to the CMOS sensor can be vastly
different at different poses, the obtained image quality can also vary quite
dramatically. However, stereo matching needs a set of images with consistent
brightness. This dataset uses a set of fixed global shutter exposure settings
for various indoor and outdoor locations, but each camera master gain is set
to be automatic. Our exposure setting is usually one-third of the default
auto-exposure setting for reducing the glare effect. This shorter exposure
time improves the image’s sharpness and reduces the motion blur at the cost of
a slightly darker appearance. The darker appearance can be a challenge for any
direct visual SLAM system. We think it is a necessary cost to get a sharp
accurate measurement of the environment.

###  7.2 Partial loss of information

In the SBS experiments, the Leica station lost track of the prism for some
short periods of time, due to its vantage point being much lower than that in
other environments. The evaluation method provided by us already takes this
into account in Sec. 6.2.

Besides, there was significant loss of UWB in some experiments due to low
received signal strength (RSS) at certain relative position between the
ranging node and the anchor. This in turn is due to the shape of the radiation
pattern (see Chen et al. (2018) for this pattern). This is not the ideal case
for a dataset, but it is a real-scenario that the researcher has to address
when developing new algorithms.

##  8 Conclusion and future works

In this paper we present the datasets collected from a sensor suite typical of
autonomous driving car, but equipped on a drone. The datasets are aimed at
boosting the investigations into autonomous navigation of UAVs using the up-
to-date technologies, which can facilitate progress in many important
industries.

Despite our best efforts, there remain some issues in the development of the
datasets. However, we do plan to keep our work continuously updated, and these
issues will be addressed in the future. For example, the loss of UWB can be
remedied by updating the UWB hardware, the tracking of ground truth can be
improved with a better vantage point. Moreover, temporal alignment can be
further improved by batch optimization methods. The camera hardware can also
be made up-to-date with other more popular RGBD-type sensors. Finally, more
datasets will be added to the suite in the future.

## Dataset access

The dataset can be accessed at our web page https://ntu-
aris.github.io/ntu_viral_dataset/ . Other details on calibration, evaluation
and updates are also presented at this site to facilitate the use of our
dataset on individual’s research.

## Funding acknowledgement

This work is supported in part by the Wallenberg AI, Autonomous Systems and
Software Program (WASP), funded by the Knut and Alice Wallenberg Foundation,
under the Grant Call 10013 - Wallenberg-NTU Presidential Postdoctoral
Fellowship 2020.

## References

  * Burri et al. (2016) Burri M, Nikolic J, Gohl P, Schneider T, Rehder J, Omari S, Achtelik MW and Siegwart R (2016) The euroc micro aerial vehicle datasets.  _The International Journal of Robotics Research_ 35(10): 1157–1163.
  * Carlevaris-Bianco et al. (2016) Carlevaris-Bianco N, Ushani AK and Eustice RM (2016) University of michigan north campus long-term vision and lidar dataset.  _The International Journal of Robotics Research_ 35(9): 1023–1035.
  * Chen et al. (2018) Chen J, Raye D, Khawaja W, Sinha P and Guvenc I (2018) Impact of 3d uwb antenna radiation pattern on air-to-ground drone connectivity.  In: _2018 IEEE 88th Vehicular Technology Conference (VTC-Fall)_. pp. 1–5.  10.1109/VTCFall.2018.8690726.
  * Delmerico et al. (2019) Delmerico J, Cieslewski T, Rebecq H, Faessler M and Scaramuzza D (2019) Are we ready for autonomous drone racing? the uzh-fpv drone racing dataset.  In: _2019 International Conference on Robotics and Automation (ICRA)_. IEEE, pp. 6713–6719.
  * Furgale et al. (2013) Furgale P, Rehder J and Siegwart R (2013) Unified temporal and spatial calibration for multi-sensor systems.  In: _2013 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, pp. 1280–1286.
  * Geiger et al. (2013) Geiger A, Lenz P, Stiller C and Urtasun R (2013) Vision meets robotics: The kitti dataset.  _International Journal of Robotics Research (IJRR)_ .
  * Geneva et al. (2020) Geneva P, Eckenhoff K, Lee W, Yang Y and Huang G (2020) Openvins: A research platform for visual-inertial estimation.  In: _2020 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, pp. 4666–4672.
  * Huang et al. (2010) Huang AS, Antone M, Olson E, Fletcher L, Moore D, Teller S and Leonard J (2010) A high-rate, heterogeneous data set from the darpa urban challenge.  _The International Journal of Robotics Research_ 29(13): 1595–1601.
  * Jeong et al. (2019) Jeong J, Cho Y, Shin YS, Roh H and Kim A (2019) Complex urban dataset with multi-level sensors from highly diverse urban environments.  _The International Journal of Robotics Research_ 38(6): 642–657.
  * Jiao et al. (2020) Jiao J, Ye H, Zhu Y and Liu M (2020) Robust odometry and mapping for multi-lidar systems with online extrinsic calibration.  _arXiv preprint arXiv:2010.14294_ .
  * Maddern et al. (2017) Maddern W, Pascoe G, Linegar C and Newman P (2017) 1 year, 1000 km: The oxford robotcar dataset.  _The International Journal of Robotics Research_ 36(1): 3–15.
  * Majdik et al. (2017) Majdik AL, Till C and Scaramuzza D (2017) The zurich urban micro aerial vehicle dataset.  _The International Journal of Robotics Research_ 36(3): 269–273.
  * Nguyen et al. (2020a) Nguyen TH, Nguyen TM and Xie L (2020a) Tightly-coupled ultra-wideband-aided monocular visual slam with degenerate anchor configurations.  _Autonomous Robots_ 44(8): 1519–1534.
  * Nguyen et al. (2020b) Nguyen TM, Cao M, Yuan S, Lyu Y, Nguyen TH and Xie L (2020b) Liro: Tightly coupled lidar-inertia-ranging odometry.  _2021 IEEE International Conference on Robotics and Automation (ICRA)_.
  * Nguyen et al. (2021a) Nguyen TM, Cao M, Yuan S, Lyu Y, Nguyen TH and Xie L (2021a) Viral-fusion: A visual-inertial-ranging-lidar sensor fusion approach.  _IEEE Transactions on Robotics_ 10.1109/TRO.2021.3094157.
  * Nguyen et al. (2021b) Nguyen TM, Yuan S, Cao M, Nguyen TH and Xie L (2021b) Viral slam: Tightly coupled camera-imu-uwb-lidar slam.  _Submitted to IEEE Robotics and Automation Letters_ URL https://arxiv.org/abs/2105.03296.
  * Nguyen et al. (2018) Nguyen TM, Zaini AH, Wang C, Guo K and Xie L (2018) Robust target-relative localization with ultra-wideband ranging and communication.  In: _2018 IEEE International Conference on Robotics and Automation (ICRA)_. IEEE, pp. 2312–2319.
  * Pandey et al. (2011) Pandey G, McBride JR and Eustice RM (2011) Ford campus vision and lidar data set.  _The International Journal of Robotics Research_ 30(13): 1543–1552.
  * Qin et al. (2018) Qin T, Li P and Shen S (2018) Vins-mono: A robust and versatile monocular visual-inertial state estimator.  _IEEE Transactions on Robotics_ 34(4): 1004–1020.
  * Ramezani et al. (2020) Ramezani M, Wang Y, Camurri M, Wisth D, Mattamala M and Fallon M (2020) The newer college dataset: Handheld lidar, inertial and vision with ground truth.  In: _2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. pp. 4353–4360.
  * Schubert et al. (2018) Schubert D, Goll T, Demmel N, Usenko V, Stückler J and Cremers D (2018) The tum vi benchmark for evaluating visual-inertial odometry.  In: _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, pp. 1680–1687.
  * Shan et al. (2020) Shan T, Englot B, Meyers D, Wang W, Ratti C and Daniela R (2020) Lio-sam: Tightly-coupled lidar inertial odometry via smoothing and mapping.  In: _IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, pp. 5135–5142.
  * Sturm et al. (2012) Sturm J, Engelhard N, Endres F, Burgard W and Cremers D (2012) A benchmark for the evaluation of rgb-d slam systems.  In: _2012 IEEE/RSJ International Conference on Intelligent Robots and Systems_. IEEE, pp. 573–580.
  * Sun et al. (2018) Sun K, Mohta K, Pfrommer B, Watterson M, Liu S, Mulgaonkar Y, Taylor CJ and Kumar V (2018) Robust stereo visual inertial odometry for fast autonomous flight.  _IEEE Robotics and Automation Letters_ 3(2): 965–972.
  * Umeyama (1991) Umeyama S (1991) Least-squares estimation of transformation parameters between two point patterns.  _IEEE Computer Architecture Letters_ 13(04): 376–380.
  * Zhang and Singh (2018) Zhang J and Singh S (2018) Laser–visual–inertial odometry and mapping with high robustness and low drift.  _Journal of Field Robotics_ 35(8): 1242–1264.
  * Zhang (2000) Zhang Z (2000) A flexible new technique for camera calibration.  _IEEE Transactions on pattern analysis and machine intelligence_ 22(11): 1330–1334.
  * Zhang and Scaramuzza (2018) Zhang Z and Scaramuzza D (2018) A tutorial on quantitative trajectory evaluation for visual (-inertial) odometry.  In: _2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)_. IEEE, pp. 7244–7251.
  * Zuñiga-Noël et al. (2020) Zuñiga-Noël D, Jaenal A, Gomez-Ojeda R and Gonzalez-Jimenez J (2020) The uma-vi dataset: Visual–inertial odometry in low-textured and dynamic illumination environments.  _The International Journal of Robotics Research_ 39(9): 1052–1060.