# Theme Analysis of Political Facebook Ads in the 2021 Dutch General Election

Joren Vrancken  Radboud UniversityNijmegenNL<EMAIL_ADDRESS>
###### Abstract.

Social media platforms have been trying to be more transparent about the
political ads they run on their platforms, because the Cambridge Analytica
scandal revealed that political campaigns are using social media on a large
scale. One such transparency effort is the Facebook Ad Library, a public
repository of all political ads run on Facebook and Instagram. This library
provides journalist and researchers with data to get a better understanding of
political advertising and microtargeting on Facebook’s platforms.
Unfortunately, the Facebook Ad Library only provides estimates and basic
information. This paper analyses political ads in more depth, by examining the
themes that ads are about. We provide a method to match themes to political
Facebook ads and we apply this method to analyse Facebook ad campaigns ran by
Dutch political parties during the 2021 Dutch general election.

Microtargeting, Political Advertising, Political Campaigns, Facebook Ad
Library, Elections, Dutch 2021 General Election

††ccs: Human-centered computing Social networking sites††ccs: Security and
privacy Social network security and privacy††ccs: Information systems Online
advertising

##  1\. Introduction

Social media has revolutionized the advertising business. Social media strive
to provide a unique user experience to each user. The content one sees is
tailored exactly to them. To accomplish this, social media platforms gather
data from their users. They analyse this data to show content that keeps their
users on the platform as long as possible. Social media platforms make money
by showing ads to their users (Facebook, 2021). Just as they try to tailor the
content a user sees to their preferences, they also tailor the ads a user sees
to their preferences. This way of delivering content and ads is called
_microtargeting_.

In traditional advertising, ads are seen by a large number of people (e.g. TV
commercials). Many people that see these ads are not interested in their
message and many others would have been interested anyway without seeing the
ads. Microtargeting solves this problem. Instead of advertising to everyone
that watches a particular TV channel or everybody that waits at a specific bus
stop, social media platforms target ads to the people with high probability of
being interested in them. Facebook provides advertisers with a large range of
selectors to specify the right people (for Business, 201). Facebook also helps
advertisers find their ideal target audience. Facebook tracks the performance
of ads to find the audiences that respond best to an advertiser’s ads. For
example, if an advertiser sells baby products, Facebook will let them target
ads to women between the ages of 30 and 35, live in Amsterdam and are
interested in products for young mothers.

However, microtargeting is controversial, as it has become a tool for
political campaigns to reach voters unnoticed. Borgesius et al. define
microtargeting for political purpose as “personalised communication that
involves collecting information about people, and using that information to
show them targeted political advertisements” (Zuiderveen Borgesius et al.,
2018). Political microtargeting has become part of mainstream public debate
after multiple media outlets reported that Cambridge Analytica, a political
consultancy company, used data of tens of millions of Facebook users to
microtarget political ads, without the users’ consent (Matthew Rosenberg and
Cadwalladr, 2018) (Kang and Frenkel, 2018). Microtargeting went unnoticed
because journalists and researchers cannot get a transparent view of who
political parties are targeting and what messages political parties are using,
as the ads are only seen by the target audience.

As a response to this controversy, or to limit the chance that lawmakers adopt
new strict laws, social media companies have amended their microtargeting
practices. Twitter has decided to ban political advertising altogether
(Twitter, 2020). Facebook has created a public repository of all political ads
that are run on their platforms (i.e. Facebook and Instagram), called the
_Facebook Ad Library_ (Facebook, 2019a). The library does not only provide the
content of ads, but also metadata about the ads (e.g. how much was spent on
the ad and when it was active). In theory, the goal of this library is to give
better insights into the advertisements political campaigns are running.
However, the library does have some transparency problems. For example, it
does not show how much an ad cost, but instead only shows a range (e.g.
between €2000 and €3000).

Political science scholars and political journalists are interested in the
messaging of political parties, because the messaging of a political party
shows which political themes a party is focusing on, what their stance is on
those themes and what they want voters to know about those themes. Ads are an
important part of this messaging. The Facebook Ad Library is a useful tool
that researchers and journalists can use to analyse the messaging in political
ads, however the Facebook Ad Library does not provide any metadata (e.g. what
an ad is about) on the content. This makes it hard to analyse the content of
the ads on a large scale. In this paper we present a technique to help
researchers analyse the content of political ads (in the Facebook Ad Library),
by matching themes to ads. We showcase this technique by analyzing the 2021
Dutch general elections.

First, we look at what data is available in the Facebook Ad library
(subsection 1.1). We then look at what Facebook ads look like and which
elements are important for our analysis (subsection 1.2). In section 2 we
discuss related research. In section 3 we describe the technique on how to
match themes to ads and walk through each step. In section 4 we apply this
technique to the 2021 Dutch general elections and compare the results with
existing voter research. We conclude in section 5 by discussing the
methodology and the results. Finally, in section 6, we look at some
interesting leads to continue this research.

###  1.1. The Facebook Ad Library

The Facebook Ad Library is a public repository of ads published on Facebook
and Instagram, released in 2019 (Shukla, 2019). It consists of two parts: a
public website and an API (Facebook, 2019b). The website can be used to
manually search for ads. The API can be used by applications to automate
searching and retrieving ads.

####  1.1.1. Information about ads

The Facebook Ad Library provides a lot of information about each ad. The most
important data the library provides for an ad is:

  * •

Start date: The date Facebook started showing an ad.

  * •

End date111Not all ads have an end date, as active ads are also available in
the Facebook Ad Library.: The date Facebook stopped showing the ad.

  * •

Spending: A range estimating the amount that is spent on the ad. The currency
that was used in the payment is also provided.

  * •

Impressions (Facebook, 2020b): Impressions measure how many times an ad was
shown to a user. Like spending, impressions are provided as a range estimating
the amount.

  * •

Estimated audience size (previously called potential reach) (Facebook, 2020a):
A range estimating how many users the ad could potentially be shown to.

  * •

Demographic distribution: Facebook provides some basic demographic information
of the distribution of impressions222The exact language Facebook uses is
“people reached by the ad”, we assume this means the demographic information
is about the impressions.. They provide three demographic characteristics:

    * –

Female/male ratio.

    * –

Age groups333Facebook requires all users to be at least 13 years old.: 13-17,
18-24, 25-34, 35-44, 45-54, 55-64, 65+.

    * –

Regions: Large regions within a country. For example, in the Netherlands the
regions are the twelve provinces.

  * •

Content: The actual text that appears in the ad. An ad has multiple text
elements, see subsection 1.2 for more information. It should be noted that the
images or videos of an ad are not directly accessible through the Facebook Ad
Library API, but the API does provide a link to the ad including any image or
video.

####  1.1.2. Spending Tracker

Besides information on ads, the library also provides information on the
organisations and political parties behind the Facebook pages (Facebook ads
are always linked to a Facebook page) that pay for the ads, in the so-called
Spending Tracker (Facebook, 2019c). This is a good resource to identify an
exhaustive list of Facebook pages that are used by political parties to run
ads.

####  1.1.3. Limitations

The Facebook Ad Library does have limitations (Leerssen et al., 2019):

  * •

No exact numbers are given. The numeric data is given in ranges. For example,
between €2000 and €3000 was spent on an ad or the ad has between 1000 and 2000
impressions.

  * •

It is not clear when an ad was most active. For example, if an ad gained
between 1000 and 2000 impressions and was active for 30 days, we do not know
whether the ad was mostly shown on one day or evenly over the 30 days.

  * •

There is a nuanced difference in the information that is available through the
website and the API. For example, Facebook bundles similar ads on the library
website to give a better estimation of the metrics. This information is not
available through the API.

  * •

It is not publicly known how Facebook decides who sees which ad. The Facebook
Ad Library only provides basic information about whom an ad was shown to. This
means that the Facebook Ad Library provides only half the picture. Because we
do not know how this decision is made, we do not know whether it is an
explicit decision by a political party to target certain people or whether it
is an automated choice by the Facebook algorithm, or a combination.

###  1.2. Anatomy of a Facebook Ad

A Facebook Ad contains the following elements:

  * •

The top of the ad shows an header that informs the user that they are being
shown an ad and which Facebook page has published and paid for the ad.

  * •

The creative body: The main text of the ad.

  * •

An image or video.

  * •

Below the creative body and image we find a call to action. This section
consists of the following elements:

    * –

The creative link caption: A link to a website of the advertiser.

    * –

The creative link title: A title that is shown above the creative link
caption.

    * –

The creative link description: A description of the creative link caption.

Not all of these elements need to be present in an ad (e.g. an ad can have an
image but no creative body). It is also possible for ads to have multiple
variants of one element (e.g. an ad with multiple creative bodies or multiple
images). In this case the Facebook algorithm will pick the best combination of
elements to show to a user.

##  2\. Related Work

The Facebook Ad Library has been used by researchers to get a better
understanding of political advertising on social media. In 2020 Fowler et al.
(FOWLER et al., 2021) used the Facebook Ad Library to compare political
Facebook ads to more traditional political advertising (e.g. television
commercials). Edelson et al. (Edelson et al., 2020) use the Facebook Ad
Library to find suspicious and malicious advertising practices. Schmøkel and
Bossetta (Schmøkel and Bossetta, 2021) provide tools to analyse the images in
Facebook ads.

The Facebook Ad Library is not the only dataset that has been used to analyse
ads on social media. ProPublica, in collaboration with other media outlets and
researchers, published a dataset of Facebook ads (both political and non-
political) that was gathered by volunteers installing a browser extension that
saves all ads (ProPublica, 2021). Ortega (López Ortega, 2021) uses this
dataset to analyse the use of negative campaigning with online political ads.
Levi et al. (Levi et al., 2020) use it to classify political ads.

Researchers have used other political texts (e.g. speeches) to analyse the
themes that political parties and specific politicians focus on. Many
researchers use topic modeling to extract themes from these political texts.
Topic modeling is a class of natural language processing models to cluster
words into groups of similar word, where each group should represent a theme.
Latent Dirichlet Allocation (LDA), presented by Blei et al. (Blei et al.,
2003), is a widely used topic model. Topic modeling work best if the analysed
documents are homogeneous. There are many models that implement topic
modeling, each suitable for a specific type of text. For example, GSDMM (Yin
et al. (Yin and Wang, 2014)) is suitable for short texts. Political ads,
however, come in many forms that range from just a few words to multi-
paragraph articles. This makes topic modeling less suited for analyzing
political ads. Instead of relying on topic models, we provide an alternative
approach to match themes to political ads.

##  3\. Methodology

The goal of our methodology is to match a theme (or multiple themes) to an ad.
In other words, we want to categorize what an ad is about using a list of
themes. To accomplish this we provide the following repetitive process:

  1. (1)

Create a list of themes (subsection 3.1).

  2. (2)

Obtain and pre-process the ad content (subsection 3.2);

  3. (3)

Create a list of relevant words for each theme (subsection 3.3);

  4. (4)

Match themes to ads using the word lists (subsection 3.4);

  5. (5)

Update the word lists by using the matched ads (subsection 3.5);

  6. (6)

Go to step 3 (subsection 3.6).

In this section we will go into detail about each step.

###  3.1. Themes

We first need a list of themes to match to ads. These themes should cover the
full public debate. We use existing research for this purpose. We base our
themes on the codebooks from the Comparative Agendas Project (CAP) (Bevan,
2019). These codebooks contain categories and sub-categories that cover every
topic in the public debate and a description for each category.

As some themes are less prevalent in the current public debate and elections,
it can be hard to distinguish between related themes. For example, it might be
hard to distinguish between international affairs and foreign trade in ads. To
solve this, we combine similar themes into one theme.

###  3.2. Obtaining and Pre-processing Ad texts

We use the Facebook Ad Library API to retrieve the ad data we need. We want to
analyse the textual content of the ads, we are interested in three elements:
the creative bodies, the creative link descriptions and the creative link
titles. We will refer to these combined elements as the _ad text_.

Before we can analyse the ad texts, we need to make sure they are all in the
same format. We use common natural language processing for this:

  * •

If an ad has multiple variants of an element (e.g. multiple possible creative
bodies), combine them into a single text.

  * •

Replace all non-ASCII characters (e.g. emojis and bold characters) with the
nearest equivalent ASCII characters.

  * •

Remove all words except for the nouns, proper nouns and adjectives.

  * •

Normalize all words to their root form (e.g. “cars” becomes “car” and “better”
becomes “good”). This is referred to as lemmatization.

To speed up the analysis, we also remove all duplicate texts, as political
parties tend to reuse text.

###  3.3. Creating Theme Word Lists

For each theme we use a list of words that are relevant to that theme. We
refer to these word lists as _theme word lists_. We use the descriptions in
the CAP codebooks for this purpose. For each theme, we (manually) add the
relevant words from the corresponding CAP codebook category description to a
list.

To make sure the theme word lists are all in the same format we pre-process
them like the ad texts in subsection 3.2.

###  3.4. Matching Themes to Ads

Once we have the themes and their corresponding theme word lists we use the
following algorithm to match themes to an ad:

  1. (1)

Compute the sizes of the intersections between each theme word list and the
words in the ad text.

  2. (2)

If the largest intersection contains at most one common word, no theme is
matched to the ad. This mean that ads do not have to correspond to a theme.
This is necessary, because political parties regularly advertise about non-
policy matters (e.g. events they are organizing and membership benefits).

  3. (3)

If the largest intersection contains more than one common word, match that
theme to the ad.

  4. (4)

If an intersection contains more than five common words, match that theme to
the ad (even if there is a larger intersection). This means that multiple
themes can be matched to an ad. This is necessary as some advertisements are
longer texts that cover multiple themes (e.g. a summary of the most important
points of a party’s policy plans).

###  3.5. Updating the Theme Word Lists

At this point we have ads that correspond to themes. We use these ads to
improve our theme word lists by updating the theme word lists for a theme with
words common in ads about that theme. For each theme, we compute the frequency
of each word in the ads about that theme. We (manually) check the most common
words for words that are relevant to the theme and (manually) add these to the
theme word list.

Whether a word is relevant to a theme is subjective. This is ultimately up to
the researcher creating the theme word list. We provide the following
guidelines to aid in this decision:

  * •

When computing the most commonly used words in a set of ads, we will find many
words that are commonly used in all ads (e.g. “vote” and “party” are common
words in election ads) and words that are common in general (e.g. “world” and
“year”). These need to be filtered out, as they do not correspond to a single
theme.

  * •

Ideally, a word should be in only one theme word list. If a word is relevant
to multiple themes, it is probably better to not add it to any list as its
meaning is too broad.

  * •

Reading political ads helps getting a feeling for the way political ads are
written and how words are used by political parties. This will help the
researcher decide whether a word that has different meanings in different
contexts should be added a theme word list.

###  3.6. Reiterating the Process

After an iteration of this process, we have improved theme word lists. We can
use these improved lists to create an improved matching between themes and ads
to find more relevant words for the theme word lists. Eventually, we come to a
point where we do not find any more new, relevant words. At this point the
process stops.

Some themes are more important in an election than others. As political
parties advertise more about the themes that voters find important, there will
be less unique, relevant words for some themes. This means that some theme
word lists will be finished after fewer iterations than others. This also
means that the theme word lists will not be of the same length.

##  4\. Analysis of 2021 Dutch General Election

In this section we will use the method described in section 3 to analyse the
themes used in political Facebook ads during the 2021 Dutch general election.
This election was held from 15 to 17 March. We look at ads that ran between 1
September 2020 and 1 September 2021444We choose a full year to get a good
picture of all ads leading up to the election and after the election. However,
the vast majority of ads ran in the month leading up to the election. and we
focus on three parties that are seen as the winners of the election (NOS,
2021) (Waarlo, 2021) (Telegraaf, 2021) (Wassens, 2021):

  * •

Democraten 66 (D66): The party that ended second in number of seats.

  * •

Forum voor Democratie (FvD): The party that gained the most seats relative to
their previous election results555FvD has since lost three of their eight
seats, because three members split off..

  * •

Volkspartij voor Vrijheid en Democratie (VVD): The party that won most seats.

These parties cover a broad political spectrum, D66 being a progressive,
social liberal party, VVD a centre-right, conservative liberal party and FvD
being a conservative, right-wing populist party (Parties and in Europe, 2021).

In subsection 4.1 we look at the themes we used during this analysis. In
subsection 4.2 we show the distribution of themes per party. We analyse these
distributions through three lenses: we compare our results with existing voter
research (subsection 4.3), we look at which parties got the most impressions
for each theme (subsection 4.4) and we look at the demographic make-up of the
audience (subsection 4.5).

We published more detailed results (e.g. the results of all parties) on our
website666 _REDACTED FOR ANONYMITY_. On this website we also published an
analysis of the metadata of political Facebook Ads.

###  4.1. Themes

As described in subsection 3.1 we create a list of themes using the CAP
codebook of the Netherlands (Arco Timmermans, 2020). This resulted in the
following list of themes. The categories from the CAP codebook on which the
themes are based are listed for each theme.

  * •

Agriculture: This theme covers all topics and debate relevant to agriculture,
farming and livestock. This theme is based on category 4 (Agriculture and
Fisheries).

  * •

Civil Rights: This theme covers all topics and debate relevant to rights of
citizens (e.g. discrimination and privacy). This theme is based on category 2
(Civil Rights). Category 2 also includes migration, which we split off into a
new theme.

  * •

Climate: This theme covers all topics and debate relevant to climate and the
environment. This theme is based on categories 7 (Environment), 8 (Energy) and
21 (Public Lands).

  * •

Defense: This theme covers all topics and debate relevant to defense and the
military. This theme is based on category 16 (Defense).

  * •

Economy: This theme covers all topics and debate relevant to the economy (e.g.
macroeconomic policies and commerce). This theme is based on categories 1
(Macroeconomics and taxes) and 15 (Domestic Commerce).

  * •

Education & Culture: This theme covers all topics and debate relevant to
education, culture and religion. This theme is based on categories 6
(Education) and 17 (Technology).

  * •

Foreign Affairs: This theme covers all topics and debate relevant to
international relations and foreign affairs. This theme is based on categories
18 (Foreign Trade) and 19 (International Affairs).

  * •

Government: This theme covers all topics and debate relevant to government
services, government operations and public service. This theme is based on
category 20 (Government Operations).

  * •

Healthcare: This theme covers all topics and debate relevant to healthcare.
This theme is based on category 3 (Healthcare).

  * •

Housing: This theme covers all topics and debate relevant to housing and city
planning. This theme is based on category 14 (Housing).

  * •

Law & Order: This theme covers all topics and debate relevant to general law,
crime and jurisdiction. This theme is based on category 12 (Law and Crime).

  * •

Migration: This theme covers all topics and debate relevant to immigration,
emigration and refugees. This theme is split from category 2 (Civil Rights).

  * •

Social Welfare: This theme covers all topics and debate relevant to social
welfare (e.g. low-Income assistance). This theme is based on categories 5
(Labor) and 13 (Social Welfare).

  * •

Transportation: This theme covers all topics and debate relevant to traffic,
transportation and infrastructure. This theme is based on category 10
(Transportation).

The theme word lists we created be found at _REDACTED FOR ANONYMITY_.

###  4.2. Distribution of Themes in Ads

Using the themes in subsection 4.1, we created theme word lists using our
methodology and matched themes to the ads of D66, FvD and VVD. Table 1 shows
the distribution of themes in number of ads and Table 2 shows the distribution
of themes of impressions777As Facebook does not provide a precise number of
impressions, these numbers are based on averages..

Table 1. The theme distribution by number of ads.

Theme (D66) | %
---|---
Education & Culture | 33.88%
Climate | 26.82%
Healthcare | 14.42%
Housing | 9.68%
Civil Rights | 6.68%
Economy | 2.42%
Social Welfare | 2.23%
Transportation | 1.84%
Government | 0.77%
Law & Order | 0.58%
Foreign Affairs | 0.39%
Agriculture | 0.29%
Defense | 0.0%
Migration | 0.0%

Theme (FvD) | %
---|---
Healthcare | 20.86%
Migration | 10.43%
Climate | 8.9%
Government | 8.9%
Economy | 7.67%
Civil Rights | 6.44%
Education & Culture | 6.44%
Defense | 5.83%
Foreign Affairs | 5.21%
Housing | 5.21%
Social Welfare | 4.91%
Agriculture | 4.29%
Law & Order | 3.07%
Transportation | 1.84%

Theme (VVD) | %
---|---
Housing | 26.15%
Economy | 16.43%
Climate | 12.24%
Civil Rights | 8.77%
Healthcare | 8.45%
Social Welfare | 7.66%
Law & Order | 6.95%
Education & Culture | 3.87%
Transportation | 3.48%
Government | 2.76%
Defense | 1.82%
Migration | 0.71%
Agriculture | 0.39%
Foreign Affairs | 0.32%

Table 2. The theme distribution by impressions.

Theme (D66) | %
---|---
Climate | 30.84%
Education & Culture | 27.31%
Healthcare | 15.66%
Housing | 11.44%
Civil Rights | 10.58%
Transportation | 1.32%
Economy | 1.09%
Social Welfare | 1.08%
Law & Order | 0.27%
Government | 0.24%
Foreign Affairs | 0.09%
Agriculture | 0.08%
Defense | 0.0%
Migration | 0.0%

Theme (FvD) | %
---|---
Healthcare | 40.56%
Economy | 12.09%
Social Welfare | 7.48%
Civil Rights | 5.91%
Defense | 4.89%
Foreign Affairs | 4.81%
Housing | 4.38%
Climate | 3.93%
Migration | 3.31%
Agriculture | 2.97%
Education & Culture | 2.96%
Government | 2.73%
Law & Order | 2.5%
Transportation | 1.49%

Theme (VVD) | %
---|---
Civil Rights | 20.58%
Climate | 17.25%
Housing | 15.99%
Healthcare | 10.18%
Economy | 8.91%
Foreign Affairs | 8.72%
Law & Order | 7.7%
Social Welfare | 5.01%
Education & Culture | 2.08%
Defense | 0.97%
Transportation | 0.81%
Migration | 0.77%
Government | 0.73%
Agriculture | 0.29%

####  4.2.1. Distribution of Matched and Non-matched Ads

The calculations of Table 1 and Table 2 only include ads that were matched to
a theme. Table 3 shows what percentage of ads did not match any theme.

Table 3. Which percentage of (the number of) ads is matched to at least one theme. Party | Number of Ads | Matched | Not matched
---|---|---|---
D66 | 2640 | 34.68% | 65.32%
FvD | 672 | 41.48% | 58.52%
VVD | 6503 | 16.74% | 83.26%

There are multiple explanations why the matched percentage is relatively low
for some parties:

  * •

Ads about organisational matters such as training sessions, conferences and
becoming a member.

  * •

Ads that focus on candidates or a broad message rather than on policy plans or
themes. For example, in the election the VVD focused their campaign on the
party leader and D66 focused on the message “it is time for new leadership”.
Other parties focus more on their policy plans. For example, 79.59% of ads by
the Dutch labor party, Partij van de Arbeid (PvdA), were matched to a theme.

  * •

Ads consisting of only a few words are harder to match to a specific theme.

###  4.3. Most Important Themes According to Voters

Before the election, Nieuwsuur888A Dutch current affairs television show.
commissioned Ipsos999A market research company that regularly performs
polling. to do research into which themes are important to the constituency of
each party (van Heck, 2021). The research includes the top five themes for
each party:

D66 • Climate & Sustainability • Healthcare • Education • Housing • Fighting COVID-19101010Healthcare and Fighting COVID-19 from the Ipsos research are combined in the theme Healthcare in our research. |  FvD • Immigration • Healthcare • Fighting COVID-19 • Norms and Values • Economic Inequality |  VVD • Healthcare • Fighting COVID-19 • Employment • Housing • Safety
---|---|---

If we compare the top five themes according to the voters of each party with
the top five themes in subsection 4.2, we see almost a 1-to-1 overlap.

  * •

D66: The top five themes according to voters correspond the top four themes in
ads. This shows us that D66 focuses its ads on themes that are important to
their voters.

  * •

FvD: Ads are highly focused on Healthcare, which their voters also care a lot
about. However, the theme Climate is second in % of ads, but not one of the
most important themes according to their voters. We can also see this in Table
2: Climate is in the 10th place of % of impressions, showing us that these ads
are (relatively) not performing well.

  * •

VVD: The top five themes according to voters correspond to the top six themes
in the ads. Like with FvD, the theme Climate is not one of the most important
themes to the voters, but is prevalent in VVD ads.

###  4.4. Most Impressions per Theme

Issue ownership for a political party is defined as having the best solution
for an issue according to voters. In this section we compare existing issue
ownership research with which parties got the most impressions for each theme,
to see whether the party with the best solutions is reaching the most voters
with their messaging.

Table 4 shows the three parties that got the most impressions (and what
percentage of the total impressions they got) for each theme. It should be
noted, that this has an inherent bias to right leaning parties as they tend to
advertise more than left leaning parties.

Table 4. The top three parties that got the most impressions for each theme. Theme | 1st | 2nd | 3rd
---|---|---|---
Foreign Affairs | JA21 (29.91%) | VVD (26.44%) | FvD (26.44%)
Civil Rights | CDA (26.02%) | PvdA (20.59%) | SP (17.21%)
Defense | FvD (64.93%) | CDA (11.03%) | GL (10.14%)
Economy | CDA (29.61%) | PvdA (22.96%) | FvD (15.71%)
Healthcare | FvD (44.44%) | GL (9.43%) | PvdA (8.95%)
Housing | PvdA (23.71%) | CDA (23.65%) | SP (11.89%)
Law & Order | JA21 (29.46%) | CDA (22.31%) | VVD (21.75%)
Climate | GL (23.36%) | D66 (18.54%) | PvdD (17.62%)
Agriculture | BBB (38.42%) | CDA (26.83%) | PvdD (20.17%)
Migration | JA21 (62.23%) | FvD (17.43%) | CDA (11.93%)
Education & Culture | D66 (28.05%) | CDA (20.85%) | PvdA (13.83%)
Government | CDA (50.13%) | SP (17.19%) | FvD (13.82%)
Social Welfare | CDA (42.21%) | PvdA (18.71%) | SP (14.28%)
Transportation | GL (31.58%) | FvD (16.11%) | BBB (9.27%)

On the 13th of February, I&O Research111111A Dutch research bureau researching
social issues., commissioned by the Volkskrant121212A Dutch newspaper.,
published voter research about the 2021 election(Peter Kanne, 2021). Section
2.3 of this report analyses issue ownership. In table 2.4 of the report, we
can see the following issues are owned by D66, Fvd and VVD:

D66 • Education • European Union |  FvD • Immigration • European Union |  VVD • Safety • Economy • Employment • Fighting COVID-19 • European Union • Public Finances • Counterterrorism
---|---|---

If we compare these issues with the data in Table 4 and subsection 4.2, we
make the following observations:

  * •

D66 owns Education and gets the most impressions for Education & Culture.

  * •

FvD advertises about issues that they do not own. For example, FvD advertises
a lot about Healthcare and Climate, but does not own these issues.

  * •

VVD owns multiple issues and focuses on most of them in their ads. However, if
we look at Table 4, we see that the VVD is only in the top three of
impressions for two themes.

  * •

The European Union issue from the I&O report (which corresponds to the Foreign
Affairs theme) does not get much attention in ads. For D66, FvD and VVD
Foreign Affairs falls in the bottom three in % of ads. However, the ads about
Foreign Affairs do relatively well when we look at the % of impressions.
Especially the VVD gets many impressions and is second in number of
impressions.

###  4.5. Demographic Groups

The demographic data that the Facebook Ad Library provides allows us to
compute the distribution of ads among demographic groups. Appendix A shows the
distribution of impressions (of all parties, not only D66, FvD and VVD) for
each theme. The data is split into the three demographic types that Facebook
provides: Female/male ratio, age groups and the twelve provinces. Appendix B
shows a similar table that splits the demographic data by party.

For comparison, the _Population_ column shows the population distribution of
the demographic groups, based on data of the Statistics Netherlands131313A
Dutch governmental institution that gathers statistical information about the
Netherlands. Centraal Bureau voor de Statistiek in Dutch. (voor de Statistiek,
2021a) (voor de Statistiek, 2021b).

Using these tables we can make some interesting observations about the
advertising strategy of the Dutch political parties:

  * •

There are some big swings in female/male distribution of the impressions. For
example, nearly 60% of all ads about migration are shown to men and nearly 60%
of all ads about social welfare are shown to women.

  * •

As people between the ages of 13 and 17 are not allowed to vote, they are
mostly excluded from the advertising.

  * •

65+ is the largest group by population, but this group is significantly under-
represented in impressions for almost all themes. This may be because senior
citizens are less active on social media.

  * •

The distribution of impressions in provinces seems to be mostly inline with
the population statistics. We only see a significant deviation from the
population statistics in specific themes (e.g. Agriculture).

##  5\. Discussion

In subsection 4.3 we see that themes that are important to voters are the same
themes that political parties advertise about. This shows that our technique
produces accurate results. It also shows that political parties prioritize
themes that their constituencies find important.

As the theme word lists are updated with common words in ads, the word lists
consist of commonly used words. This is advantageous, because political
parties tend to use specific terms to convey their opinions. For example, when
talking about climate change, a party that prioritizes the dangers of climate
change will use different language than a party that is skeptic towards
climate change. Only using pre-made lists of words or clustering similar words
into groups (i.e. using topic modeling) might not include relevant words (and
common misspellings), which will result in miss-matching themes to ads.

Different political parties use different strategies when it comes to writing
ads. Some parties use many short ads each covering a single topic, others use
large texts covering everything that is important to the party. This means
that any technique to analyse the themes in ads should be agnostic to the
length and format of the content of the ads. We accomplish this by only taking
the intersection between the ad and the theme word lists into account and not
looking at the length of the ad and the theme word list.

Our method does, however, have two inherent biases. Whether a word is relevant
to a theme is subjective. This is an inherent bias towards the opinion of the
researcher creating the theme word list. Whether a word in an ad is relevant
to a theme is also dependent on the time the ad is active. The problems that
are relevant to a certain theme are different each election, this means that
there is a bias towards the current political issues in the theme word lists.

As discussed in subsection 3.5, someone needs to decide whether a word
appearing in an ad is relevant to a theme or not. Unfortunately, this creates
a lot of manual work for the researcher. However, as we only look at words
that are common in ads, the method is still scalable. For example, we analysed
more than thirty thousand ads for this research.

Although the Facebook Ad Library gives a lot of insights into the ads that are
run on the platforms of Facebook, we still only see part of the picture.
Facebook only provides estimates of the performance of an ad. These estimates
make it possible to analyse the advertising performance of a theme or
political party. However, it does not help us answer a maybe more important
question: How are political parties using opaque online tools (such as
microtargeting) to reach voters?

##  6\. Future Work

  * •

In this research we focused on the text contained in political ads. However,
many ads also include an image or video. Some ads only consist of an image or
video and do not have any text. These images and videos should also be
analysed to improve the theme matching.

  * •

In section 4 we focused on the distribution of themes in ads by political
parties. It would be interesting to look more deeply into the way political
parties talk about theses themes, for example, sentiment analysis.

  * •

As Edelson et al. (Edelson et al., 2020) shows, we can see which seemingly
separate advertisers publish closely related (or identical) ads. We might be
able to find parties that are collaborating in their ad campaigns.

###### Acknowledgements.

We would like to sincerely thank Frederik Zuiderveen Borgesius and Tom Dobber.
They have provided invaluable guidance, feedback and advice during this
research. We would also like to sincerely thank everybody that proofread this
paper.

## References

  * (1)
  * Arco Timmermans (2020) Gerard Breeman Arco Timmermans. 2020\.  _The Netherlands Agendas Project studies the policy outputs of the national government._ Comparative Agendas Project.  https://www.comparativeagendas.net/netherlands
  * Bevan (2019) Shaun Bevan. 2019\.  _Comparitive Agendas Project Master Codebook_.  Comparative Agendas Project.  https://www.comparativeagendas.net/pages/master-codebook
  * Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003\.  Latent dirichlet allocation.  _the Journal of machine Learning research_ 3 (2003), 993–1022.
  * Edelson et al. (2020) Laura Edelson, Tobias Lauinger, and Damon McCoy. 2020\.  A security analysis of the facebook ad library. In _2020 IEEE Symposium on Security and Privacy (SP)_. IEEE, IEEE, Online, 661–678.
  * Facebook (2019a) Facebook. 2019a.  _Facebook Ad Library_.  Facebook.  https://www.facebook.com/ads/library/
  * Facebook (2019b) Facebook. 2019b.  _Facebook Ad Library API_.  Facebook.  https://www.facebook.com/ads/library/api
  * Facebook (2019c) Facebook. 2019c.  _Facebook Ad Library Report_.  Facebook.  https://www.facebook.com/ads/library/report
  * Facebook (2020a) Facebook. 2020a.  _About Estimated Audience Size_.  Facebook.  https://www.facebook.com/business/help/1665333080167380
  * Facebook (2020b) Facebook. 2020b.  _Impressions_.  Facebook.  https://www.facebook.com/business/help/675615482516035
  * Facebook (2021) Facebook. 2021.  _Facebook Reports Third Quarter 2021 Results_.  Facebook.  https://investor.fb.com/investor-news/press-release-details/2021/Facebook-Reports-Third-Quarter-2021-Results/default.aspx
  * for Business (201) Facebook for Business. 201\.  _Help your ads find the people who will love your business_.  Facebook for Business.  https://www.facebook.com/business/ads/ad-targeting
  * FOWLER et al. (2021) ERIKA FRANKLIN FOWLER, MICHAEL M. FRANZ, GREGORY J. MARTIN, ZACHARY PESKOWITZ, and TRAVIS N. RIDOUT. 2021.  Political Advertising Online and Offline.  _American Political Science Review_ 115, 1 (2021), 130–149.  https://doi.org/10.1017/S0003055420000696
  * Kang and Frenkel (2018) Cecilia Kang and Sheera Frenkel. 2018.  _Facebook Says Cambridge Analytica Harvested Data of Up to 87 Million Users_.  New York Times.  https://www.nytimes.com/2018/04/04/technology/mark-zuckerberg-testify-congress.html
  * Leerssen et al. (2019) P. Leerssen, J. Ausloos, B. Zarouali, N. Helberger, and C.H de Vreese. 2019.  Platform ad archives: promises and pitfalls.  _Internet Policy Review_ 8, 4 (9 October 2019).  https://doi.org/10.14763/2019.4.1421
  * Levi et al. (2020) Or Levi, Sardar Hamidian, and Pedram Hosseini. 2020.  Automatically Identifying Political Ads on Facebook: Towards Understanding of Manipulation via User Targeting. In _Disinformation in Open Online Media_ , Max van Duijn, Mike Preuss, Viktoria Spaiser, Frank Takes, and Suzan Verberne (Eds.). Springer International Publishing, Cham, 95–106.
  * López Ortega (2021) Alberto López Ortega. 2021\.  Are microtargeted campaign messages more negative and diverse? An analysis of Facebook ads in European election campaigns.  _European Political Science_ Online First (2021), 1–24.
  * Matthew Rosenberg and Cadwalladr (2018) Nicholas Confessore Matthew Rosenberg and Carole Cadwalladr. 2018.  _How Trump Consultants Exploited the Facebook Data of Millions_.  New York Times.  https://www.nytimes.com/2018/03/17/us/politics/cambridge-analytica-trump-campaign.html
  * NOS (2021) NOS. 2021.  _Winnaars VVD en D66 aan zet bij formatie, forse verliezen op links_.  NOS.  https://nos.nl/collectie/13860/artikel/2373089-winnaars-vvd-en-d66-aan-zet-bij-formatie-forse-verliezen-op-links
  * Parties and in Europe (2021) Parties and Elections in Europe. 2021.  _Netherlands_.  Parties and Elections in Europe.  http://www.parties-and-elections.eu/netherlands.html
  * Peter Kanne (2021) Milan Driessen Peter Kanne. 2021\.  _Ipsos & Nieuwsuur Kiezersonderzoek 2021_.  I&O Research.  https://www.ioresearch.nl/actueel/de-kiezer-gaat-vooralsnog-voor-zekerheid-niet-voor-de-inhoud/
  * ProPublica (2021) ProPublica. 2021\.  _Political Advertisements from Facebook_.  ProPublica.  https://www.propublica.org/datastore/dataset/political-advertisements-from-facebook
  * Schmøkel and Bossetta (2021) Rasmus Schmøkel and Michael Bossetta. 2021.  FBAdLibrarian and Pykognition: open science tools for the collection and emotion detection of images in Facebook political ads with computer vision.  _Journal of Information Technology & Politics_ 0, 0 (2021), 1–11.  https://doi.org/10.1080/19331681.2021.1928579
  * Shukla (2019) Satwik Shukla. 2019\.  _A Better Way to Learn About Ads on Facebook_.  Facebook.  https://about.fb.com/news/2019/03/a-better-way-to-learn-about-ads/
  * Telegraaf (2021) Telegraaf. 2021\.  _Rutte op weg om langstzittende premier te worden: “Energie voor 10 jaar!”_.  Telegraaf.  https://www.telegraaf.nl/nieuws/22405976/rutte-op-weg-om-langstzittende-premier-te-worden-energie-voor-10-jaar
  * Twitter (2020) Twitter. 2020.  _Political content_.  Twitter.  https://business.twitter.com/en/help/ads-policies/ads-content-policies/political-content.html
  * van Heck (2021) Sjoerd van Heck. 2021\.  _Ipsos & Nieuwsuur Kiezersonderzoek 2021_.  Ipsos.  https://www.ipsos.com/nl-nl/ipsos-nieuwsuur-kiezersonderzoek-2021
  * voor de Statistiek (2021a) Centraal Bureau voor de Statistiek. 2021a.  _Bevolkingspiramide_.  Centraal Bureau voor de Statistiek.  httpshttps://www.cbs.nl/nl-nl/visualisaties/dashboard-bevolking/bevolkingspiramide
  * voor de Statistiek (2021b) Centraal Bureau voor de Statistiek. 2021b.  _Regionale kerncijfers Nederland_.  Centraal Bureau voor de Statistiek.  https://opendata.cbs.nl/statline/#/CBS/nl/dataset/70072NED/table?fromstatweb
  * Waarlo (2021) Niels Waarlo. 2021\.  _Dit zijn de belangrijkste verkiezingsconclusies (tot nu toe)_.  Volkskrant.  https://www.volkskrant.nl/nieuws-achtergrond/dit-zijn-de-belangrijkste-verkiezingsconclusies-tot-nu-toe~baa28c7d/
  * Wassens (2021) Rik Wassens. 2021\.  _Nog niet eerder waren de linkse partijen zo klein_.  NRC.  https://www.nrc.nl/nieuws/2021/03/18/nog-niet-eerder-waren-de-linkse-partijen-zo-klein-2-a4036378
  * Yin and Wang (2014) Jianhua Yin and Jianyong Wang. 2014.  A dirichlet multinomial mixture model-based approach for short text clustering. In _Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining_. ACM, New York, 233–242.
  * Zuiderveen Borgesius et al. (2018) F.J. Zuiderveen Borgesius, J. Möller, S. Kruikemeier, R. Ó Fathaigh, K. Irion, T. Dobber, B. Bodo, and C. de Vreese. 2018.  Online Political Microtargeting: Promises and Threats for Democracy.  _Utrecht Law Review_ 14, 1 (2018), 82–96.  https://doi.org/10.18352/ulr.420

##  Appendix A Demographic Distribution Tables

Table 5. The demographic distribution of impressions per theme. Dem. | Pop. |  Agriculture |  | Civil
---
Rights

Climate |  Defense |  Economy |  | Education
---
& Culture

| Foreign
---
Affairs

Female | 50.29% | 47.08% | 58.99% | 50.31% | 41.09% | 54.09% | 54.76% | 45.77%
Male | 49.71% | 52.92% | 41.01% | 49.69% | 58.91% | 45.91% | 45.24% | 54.23%
13-17 | 6.47% | 0.09% | 0.22% | 0.06% | 0.01% | 0.33% | 0.14% | 0.04%
18-24 | 10.17% | 17.39% | 16.09% | 19.37% | 13.34% | 13.31% | 21.82% | 11.78%
25-34 | 14.92% | 25.24% | 19.45% | 24.21% | 21.60% | 20.27% | 21.83% | 20.23%
35-44 | 13.78% | 16.67% | 14.01% | 15.03% | 19.36% | 15.27% | 13.53% | 17.01%
45-54 | 15.96% | 15.86% | 14.77% | 19.55% | 18.94% | 17.75% | 14.62% | 18.65%
55-64 | 15.86% | 13.51% | 15.60% | 13.81% | 15.37% | 16.07% | 13.75% | 17.58%
65+ | 22.84% | 11.24% | 19.86% | 7.98% | 11.39% | 17.00% | 14.31% | 14.72%
Drenthe | 2.83% | 6.00% | 3.26% | 4.03% | 4.01% | 4.14% | 3.48% | 4.22%
Flevoland | 2.45% | 2.01% | 2.30% | 2.26% | 2.14% | 2.41% | 2.48% | 2.46%
Friesland | 3.73% | 6.04% | 3.76% | 4.74% | 4.69% | 4.55% | 3.94% | 4.82%
Gelderland | 12.00% | 14.20% | 11.45% | 12.33% | 11.90% | 11.90% | 13.46% | 12.14%
Groningen | 3.36% | 4.62% | 3.62% | 4.12% | 3.96% | 4.96% | 3.83% | 4.20%
Limburg | 6.39% | 7.18% | 8.31% | 6.54% | 7.84% | 7.71% | 6.36% | 7.82%
Noord-Brabant | 14.73% | 14.34% | 15.30% | 13.81% | 13.83% | 14.89% | 13.57% | 13.75%
Noord-Holland | 16.53% | 9.59% | 14.64% | 15.55% | 17.31% | 12.78% | 13.83% | 13.24%
Overijssel | 6.68% | 12.46% | 7.50% | 7.52% | 7.71% | 8.58% | 8.53% | 9.66%
Utrecht | 7.79% | 6.34% | 5.98% | 6.59% | 5.24% | 6.18% | 6.43% | 5.09%
Zeeland | 2.21% | 3.77% | 2.76% | 2.78% | 2.46% | 2.73% | 3.66% | 2.64%
Zuid-Holland | 21.32% | 13.46% | 21.11% | 19.72% | 18.92% | 19.17% | 20.44% | 19.95%
Table 6. The demographic distribution of impressions per theme. Dem. | Pop. |  Government |  Healthcare |  Housing |  Law & Order |  Migration |  Social Welfare |  Transportation
---|---|---|---|---|---|---|---|---
Female | 50.29% | 51.78% | 52.82% | 54.89% | 47.17% | 41.48% | 58.80% | 47.21%
Male | 49.71% | 48.22% | 47.18% | 45.11% | 52.83% | 58.52% | 41.20% | 52.79%
13-17 | 6.47% | 0.10% | 0.03% | 0.10% | 0.06% | 0.02% | 0.35% | 0.18%
18-24 | 10.17% | 9.55% | 16.15% | 17.02% | 17.93% | 9.88% | 11.56% | 13.25%
25-34 | 14.92% | 16.51% | 22.55% | 24.76% | 22.14% | 16.42% | 16.92% | 20.05%
35-44 | 13.78% | 13.88% | 15.81% | 14.75% | 14.37% | 14.12% | 13.26% | 14.01%
45-54 | 15.96% | 16.24% | 16.70% | 13.83% | 15.57% | 17.11% | 14.91% | 21.32%
55-64 | 15.86% | 19.79% | 15.38% | 14.02% | 15.30% | 20.01% | 18.39% | 17.84%
65+ | 22.84% | 23.93% | 13.37% | 15.53% | 14.62% | 22.44% | 24.62% | 13.36%
Drenthe | 2.83% | 4.07% | 3.99% | 3.49% | 3.83% | 4.26% | 3.65% | 3.74%
Flevoland | 2.45% | 2.35% | 2.45% | 2.04% | 2.31% | 2.47% | 2.41% | 2.13%
Friesland | 3.73% | 3.66% | 4.67% | 4.18% | 4.21% | 5.05% | 4.28% | 4.40%
Gelderland | 12.00% | 12.30% | 12.23% | 10.37% | 11.73% | 12.03% | 12.38% | 12.70%
Groningen | 3.36% | 4.02% | 4.17% | 4.23% | 3.84% | 4.32% | 3.81% | 3.57%
Limburg | 6.39% | 6.97% | 7.69% | 8.18% | 7.62% | 8.62% | 7.78% | 5.47%
Noord-Brabant | 14.73% | 13.50% | 14.00% | 13.97% | 14.42% | 13.86% | 15.74% | 9.80%
Noord-Holland | 16.53% | 11.82% | 13.83% | 13.48% | 13.02% | 12.95% | 13.21% | 11.45%
Overijssel | 6.68% | 7.28% | 8.12% | 9.53% | 9.59% | 8.63% | 7.98% | 13.40%
Utrecht | 7.79% | 6.76% | 5.78% | 6.70% | 6.05% | 5.30% | 6.36% | 13.93%
Zeeland | 2.21% | 5.36% | 2.92% | 2.77% | 2.88% | 2.49% | 3.66% | 3.53%
Zuid-Holland | 21.32% | 21.91% | 20.16% | 21.06% | 20.50% | 20.01% | 18.75% | 15.87%

##  Appendix B Distribution of Impressions per Demographic by Party

Table 7. The demographic distribution of impressions for D66, FvD and VVD. Demographic | Population | D66 | FvD | VVD
---|---|---|---|---
Female | 50.29% | 53.33% | 41.47% | 48.28%
Male | 49.71% | 46.67% | 58.53% | 51.72%
13-17 | 6.47% | 0.04% | 0.03% | 0.04%
18-24 | 10.17% | 22.49% | 19.31% | 16.29%
25-34 | 14.92% | 24.53% | 28.32% | 23.48%
35-44 | 13.78% | 14.69% | 15.03% | 18.32%
45-54 | 15.96% | 14.38% | 15.30% | 18.31%
55-64 | 15.86% | 12.68% | 12.71% | 13.79%
65+ | 22.84% | 11.19% | 9.31% | 9.79%
Drenthe | 2.83% | 3.54% | 4.21% | 3.82%
Flevoland | 2.45% | 2.38% | 2.33% | 2.02%
Friesland | 3.73% | 4.36% | 5.07% | 3.45%
Gelderland | 12.00% | 12.22% | 11.81% | 11.60%
Groningen | 3.36% | 4.11% | 4.20% | 3.10%
Limburg | 6.39% | 7.05% | 8.12% | 6.61%
Noord-Brabant | 14.73% | 14.95% | 15.41% | 16.84%
Noord-Holland | 16.53% | 14.38% | 12.51% | 15.36%
Overijssel | 6.68% | 7.52% | 8.50% | 7.16%
Utrecht | 7.79% | 6.71% | 4.83% | 7.06%
Zeeland | 2.21% | 2.40% | 2.74% | 2.09%
Zuid-Holland | 21.32% | 20.38% | 20.26% | 20.90%


# What Predicts Interpersonal Affect?
Preliminary Analyses from Retrospective Evaluations

Maria Teresa Parreira, Michael J. Sack, Malte Jung Cornell University

###### Abstract

While the field of affective computing has contributed to greatly improving
the seamlessness of human-robot interactions, the focus has primarily been on
the emotional processing of the self, rather than the perception of the other.
To address this gap, in a user study with 30 participant dyads, we collected
the users’ retrospective ratings of the interpersonal perception of the other
interactant, after a short interaction. We made use of CORAE, a novel web-
based open-source tool for COntinuous Retrospective Affect Evaluation. In this
work, we analyze how these interpersonal ratings correlate with different
aspects of the interaction, namely personality traits, participation balance,
and sentiment analysis. Notably, we discovered that conversational imbalance
has a significant effect on the retrospective ratings, among other findings.
By employing these analyses and methodologies, we lay the groundwork for
enhanced human-robot interactions, wherein affect is understood as a highly
dynamic and a context-dependent outcome of interaction history.

###### Index Terms:

affective computing, annotation tool, sentiment analysis

##  I Background

Research in human-robot interaction is often focused on measuring users’
external outcomes (e.g., improve group task performance [30]) or internal
individual states (e.g., sensing student engagement [2]). However, for robots
to both understand and shape interactions among humans, they require an
understanding of internal interpersonal states – that is, the perception of
the other through the user’s perspective [19].

Observable behavior and subjective experience are highly dynamic [20]. In
interactions, these aspects co-evolve over time for all the interactants [3],
interacting in ways that are yet to be fully explored. To study these
dynamics, continuous representations of affective states have been popularized
[14, 23], allowing for an understanding of how humans aggregate affect
information across time and unveiling regions of “emotional saliency”, which
may be pivotal to assessing the emotional experience [23].

A popular approach for collecting continuous affect data is retrospective
analysis [7, 8, 22, 21], which relies on the phenomenon that individuals often
re-experience emotions when reliving a situation [11]. In prior work [27], we
introduced CORAE, an intuitive tool for COntinuous Retrospective Affect
Evaluation. This tool enables researchers to collect continuous affect data
about interpersonal perceptions. Participants retrospectively rank how another
interactant came across immediately following an interaction, thus allowing us
to capture interpersonal affective perceptions rather than feelings or
affective state inferences. In other words, our system allows us to capture
data about how people perceive each other emotionally continuously over time.

While affect data has been collected across subjective experience [26, 9, 13],
and observable behavior [18, 6], we still lack continuous data about how
affective perceptions of others develop dynamically over time. Prior work
identified relational emotion metrics that take temporal patterns of emotional
expressions in interactions into account and that focus on relationally
relevant dimensions of emotion expression. [17, 18]. However, these metrics
often rely on manual coding of emotional expressions, while automated
approaches for relationally relevant emotion metrics remain to be developed.

Figure 1: Annotation dashboard for CORAE. Participants retrospectively
evaluate how the other person came across by reviewing the recording of the
interaction with only the video from the other participant and audio of both.

We ran an online user study where 30 participant dyads interacted while
completing a task. Following their discussion, participants retrospectively
annotated the interaction, evaluating how the other participant came across.
In the present work, we explored how different interaction variables –
personality of the interactants, conversational balance, emotional content of
participants’ speech – affected how users perceived each other.

Our perceptions of robots are yet to be fully understood [29], but authors
have recognized the value of robotic systems that account for the dynamic
nature of affect [5]. This preliminary analysis sheds light on aspects of
interactions that impact our interpersonal states, with the potential to
better inform the development of these systems.

##  II CORAE

We provide below a brief description of CORAE, to facilitate the understanding
of the interpersonal perception data used in this study. The CORAE platform
(related to the Latin word for “heart”) [27] enables individuals to
intuitively evaluate how a person’s behavior is interpreted emotionally during
interactions. It is an open-source tool that can be found in corae.org.

fnum@@desciitemDesign:

CORAE is intuitive and visually minimal (see Figure 1). The central focus is a
video of the other interactant. Brief instructions above the video player
describe the controls of the annotation dashboard (Spacebar to toggle playback
and Left and Right Arrows to control the slider), as well as a brief
description of the terms used for measuring interpersonal perception, which
can be personalized to the platform’s use case.

A progress bar is displayed below the video player to inform participants what
proportion remains of their evaluation. Finally, below the video player is
displayed the annotation slider. The annotation bar is bounded and discretized
(a total of 15 points, from $-7$ — Disagreeable — to $+7$ — Agreeable).
Participants may only change their rating during video playback and are
constrained by the platform to do it “continuously” (i.e., they cannot
instantaneously change the rating from Neutral ($0$) to Agreeable ($7$), but
rather adjust to each value in sequence).

fnum@@desciitemData Logging:

Data is logged for a session in two ways: (1) by default, the mode for data
logging is set to predetermined intervals of one second; and (2) whenever a
change in the rating occurs. Associated data points are the slider position
(rating), time code, and video frame (in the format “SliderNumericalPosition”:
“Hours:Minutes:Seconds:VideoFrame”), which are logged in a JSON file.

##  III User Study

We carried out a user study to collect dyadic interaction data and
retrospective interpersonal ratings through CORAE. A more detailed description
can be found in Sack et al. [27].

###  III-A Experimental Procedure

Participants were recruited through Prolific111https://www.prolific.co/.
Before scheduling their slot, each participant read and signed a consent form.
The study took place fully online, with interactions facilitated through
Zencastr222https://zencastr.com/, a video call platform that allows for the
recording of each video and audio stream separately. Participants read task
instructions, including a description of the discussion topic (Reasons for
Poverty task [28], detailed below). After this, participants were recorded
while interacting to solve the task. When they reached an agreement, or after
10 minutes of discussion, participants were asked to stop discussing and fill
out a survey. This survey collected demographic data, as well as measures of
interpersonal affect. Each participant was then distributed a URL that opened
an instance of CORAE’s annotation platform in their browser. Participants were
each presented with a video of their discussion partner, and the audio of
both, and were asked to continuously rate how their partner came across,
moment-by-moment. Once finished, participants completed an exit survey with
open comments. Participants were compensated for their participation with
US$14, through Prolific.

fnum@@desciiitemReasons for Poverty task:

We used a modified version of the Reasons for Poverty task [28] as a
discussion prompt. The task requires participants to agree on selecting 5
items from a list of “reasons for poverty”, and rank them according to their
“accuracy”. Some options in the 10-item list include “Poor people lack the
ability to manage money.”, or “The society lacks justice”. We aimed to elicit
an emotionally engaging interaction.

Participants were given a maximum of 10 minutes to discuss, to prevent
individuals from getting disengaged when reviewing their discussion on CORAE.

###  III-B Research Questions

In this preliminary analysis, we wanted to investigate if aspects of the
users’ identity, or conversational aspects such as the balance of the
participation of the users were impactful for how participants perceived each
other. We focused on the following research questions:

  * •

RQ1: Do personality traits or demographic aspects (age, gender) of a user
impact how they come across to the other interactant?

  * •

RQ2: Do dynamics of the user’s conversation (imbalance, duration of
conversational turns) impact the users’ interpersonal perceptions of each
other?

  * •

RQ3: Does the emotional content of the user’s speech impact how they come
across to the other interactant?

###  III-C Measures

To answer the research questions laid out above, we defined a set of measures
from the data collected during the interaction sessions.

fnum@@desciiiitemInterpersonal measures:

We evaluated the perception of the other through two measures. The
Interpersonal Agreeableness measure was operationalized by asking participants
“How did the other participant come across?” on a 7-point Likert scale (from
disagreeable to agreeable) in the post-interaction survey. Additionally,
Interpersonal Perception (IP) was extracted from the continuous interpersonal
rating data collected via CORAE. To calculate the IP, and in line with prior
work[18, 12], for each participant, we took the cumulative sum of the ratings
during the interaction and fitted a linear regression to that data. The
Interpersonal Perception measure is given by the slope of that regression,
providing an understanding of how the perception of the other interactant
evolved over the interaction (see Figure 2).

Figure 2: Dynamics of interpersonal ratings (IR) in one session sessions (top)
and respective cumulative sum of rated values (bottom). Interpersonal
Perception is calculated by adjusting a linear regression to the latter
curves, and extracting the slope of these lines (bottom, overlaid dashed
curves).

fnum@@desciiiitemDemographic and Personality measures:

In the post-interaction survey, we collected demographic information (age,
gender, nationality, race/ethnicity) and personality traits through the short-
version of the Big Five Inventory [25]. Participants were also asked to rate
their religiousness (not-at-all religious to very religious) and political
leaning (very liberal to very conservative) with 7-point Likert scales.

fnum@@desciiiitemConversational Balance measures:

To assess RQ2, we measured the users’ total number and average For duration of
conversational turns, as well as participation imbalance [30, 10], which is
given by:

$\text{Imbalance}=\sum_{i\in[1,2]}|\text{s}^{i}-\overline{\text{s}}|$ (1)

where $\text{s}^{i}$ is the amount of time that participant $i$ has spoken
over the total amount of speech. $\overline{\text{s}}$ is the mean of the
relative speech time of the two participants, which in the case of a dyadic
interaction is $0.5$. This measures how much the participation of the users
deviated from a perfectly balanced interaction. These measures were collected
from Voice Activation Detection (VAD).

fnum@@desciiiitemSpeech emotion content measures:

We transcribed the speech of the participants using the speech-to-text model
Whisper [24]. Also through Whisper, we analyzed the emotional “tone” of each
utterance of the participants in the dataset (either neutral, positive or
negative). Because the task contained prompts that were negative, we extracted
only the positive emotional tone ratio, i.e. the ratio of utterances that were
evaluated as positive (over total number of utterances). Additionally, we
collected the total number of agreement words within the speech of each user,
to obtain a measure of affirmed agreement from each participant.

###  III-D Participants

To potentially elicit disagreement during the interactions, participants were
selected according to their political leaning (one conservative- and one
liberal-leaning). Other recruitment criteria were proficiency in English and a
computer device with a functioning camera and microphone.

##  IV Results

We collected data from 30 interaction sessions. Due to connection problems or
data collection issues, we only used data from 27 interaction sessions (54
participants, a total of 3663 transcribed utterances). Participants’ age
ranged from $20-87$ years ($M\pm SD:42.33\pm 15.78$). Out of the 54
participants, 28 identified as female, 25 as male, one as non-binary.
Race/ethnicity was mostly Caucasian/White (41), followed by Asian/Asian
American (5), Hispanic/Latino (5), African/African American/Black (3), Middle
Eastern/North African (1) and American Indian (1) (participants could select
multiple). Most participants were native speakers of English (49), with 5
proficient users.

###  IV-A Personality and demographics

To investigate RQ1, we evaluated whether personality or demographic traits
impact how users are perceived with an ANCOVA. We examined the effects of each
personality trait (agreeableness, conscientiousness, extroversion,
neuroticism, openness) on both Interpersonal Perception (IP) and Interpersonal
Agreeableness (IA), while controlling for gender and age. The effect of
agreeableness on IP (the slope of the cumulative ratings curve, as evaluated
by each discussion partner) was significant ($F(1,53)=6.67,p=0.01$).
Agreeableness was also a predictor of IA (how the participant came across
during an interaction, reported via survey by their discussion partner),
$F(1,53)=4.48,p=0.04$, as well as neuroticism ($F(1,53)=4.65,p=0.04$).

###  IV-B Conversational balance

We considered how aspects of conversational balance can impact differences in
the perception of the interaction. For this, we considered whether the
difference in the IP values for the dyad ($|IP_{1}-IP_{2}|$), as well as the
IA value difference, were impacted by the imbalance in the conversation. We
also evaluated the effects of the difference in the total number and average
length of turns for both participants. An ANCOVA revealed that conversational
imbalance is a predictor of the IP difference ($F(1,26)=4.79,p=0.04$). The
interaction between these two variables can be seen in Fig. 3.

Figure 3: Conversational Imbalance impacts the alignment between the
Interpersonal Perceptions of the two participants. More imbalanced
conversations correlate with dissimilar retrospective ratings (higher IP
Difference). Imbalance was discretized for the purpose of visualization
($N=11$ for imbalance $<0.25$, $N=10$ for imbalance $<0.5$, $N=6$ for
imbalance $>0.5$).

###  IV-C Speech emotional content

Finally, to answer RQ3, we evaluated whether the content of the speech from
each participant could impact how users are perceived. We ran an ANCOVA to
evaluate the effects of the number of agreement words, as well as the positive
emotional tone ratio, on IP and IA. Our results show that the ratio of
positive tone utterances (in relation to all utterances from that participant)
has a significant effect on the IP value ($F(1,54)=8.55,p=0.005$).

##  V Discussion

Our study contributes to the broader understanding of affective computing by
emphasizing the significance of the perception of the other in shaping
affective experiences. It expands the existing knowledge by highlighting the
need to consider social distance as a key factor in human-robot interactions
[19, 4, 1]. By incorporating such considerations into the design and
development of affective computing systems, we can create more empathetic and
effective interactions between humans and robots.

We investigated if the personality traits of users impact how they are
perceived in an interaction, to answer RQ1. Indeed, we found that the
personality trait of agreeableness has an effect on how the participant is
rated by their discussion partner, both on the “static” measure of the
interpersonal dimension (Interpersonal Agreement), and on the continuous
retrospective rating of the interaction (Interpersonal Perception). This
indicates that the brief discussion task was sufficient to grant an alignment
between the perception of the self and of the other, in spite of some
literature reporting that the two may conflict [15].

In addition to the personality of each participant, we investigated if
conversational dynamics impacted how users rated each other. To answer RQ2, we
found that the imbalance in the speech time of each participant has an effect
on how aligned their perceptions of each other are. We found that more
imbalanced conversations lead to higher differences in the IP values of the
participants. Further analyses are needed to understand how exactly the
interpersonal perceptions get distorted by the amount of participation in the
discussion.

Finally, we answered RQ3 by investigating the emotional content of the speech
of each participant. Interestingly, our results indicate that the ratio of
utterances with positive emotional content impacts how the participant is
rated by the other user. Agreeableness is associated with the expression of
more positive emotions [31], which may explain why participants came across as
generally more agreeable.

In addition to improving our knowledge of interpersonal affect, our study and
follow-up work can also contribute to bettering human-robot interactions –
agreeableness of users has been found to play a role in rapport building with
robots [16]. By computing affect as a highly dynamic and context-dependent
phenomenon, we can develop strategies to improve the perceived social distance
between humans and robots. Understanding how different aspects of the
interaction influence affective perceptions allows for the design of more
seamless and responsive systems, fostering a sense of connection and
engagement.

## Acknowledgment

We thank Nawid Jamali and Hifza Javed for the collaboration on CORAE and user
study design. This work was supported by Honda Research Institute USA, Inc..

## References

  * Andersen and Guerrero [1996] P. A. Andersen and L. K. Guerrero.  Principles of communication and emotion in social interaction.  In _Handbook of communication and emotion_ , pages 49–96. Elsevier, 1996.
  * Bourguet et al. [2020] M.-L. Bourguet, Y. Jin, Y. Shi, Y. Chen, L. Rincon-Ardila, and G. Venture.  Social robots that can sense and improve student engagement.  In _2020 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE)_ , pages 127–134, 2020.  doi: 10.1109/TALE48869.2020.9368438.
  * Butler and Randall [2013] E. A. Butler and A. K. Randall.  Emotional coregulation in close relationships.  _Emotion Review_ , 5(2):202–210, 2013.
  * Caffi and Janney [1994] C. Caffi and R. W. Janney.  Toward a pragmatics of emotive communication.  _Journal of pragmatics_ , 22(3-4):325–373, 1994\.
  * Churamani et al. [2020] N. Churamani, S. Kalkan, and H. Gunes.  Continual learning for affective robotics: Why, what and how?  In _2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN)_ , pages 425–431, 2020.  doi: 10.1109/RO-MAN47096.2020.9223564.
  * Coan and Gottman [2007] J. A. Coan and J. M. Gottman.  The specific affect coding system (spaff).  _Handbook of emotion elicitation and assessment_ , 267:285, 2007.
  * Cowie et al. [2000] R. Cowie, E. Douglas-Cowie, S. Savvidou, E. McMahon, M. Sawey, and M. Schröder.  ’feeltrace’: An instrument for recording perceived emotion in real time.  01 2000.
  * Cowie et al. [2013] R. Cowie, M. Sawey, C. Doherty, J. Jaimovich, C. Fyans, and P. Stapleton.  Gtrace: General trace program compatible with emotionml.  In _2013 Humaine Association Conference on Affective Computing and Intelligent Interaction_ , pages 709–710, 2013.  doi: 10.1109/ACII.2013.126.
  * Csikszentmihalyi et al. [2014] M. Csikszentmihalyi, M. Csikszentmihalyi, and R. Larson.  Validity and reliability of the experience-sampling method.  _Flow and the foundations of positive psychology: The collected works of Mihaly Csikszentmihalyi_ , pages 35–54, 2014.
  * Gillet et al. [2021] S. Gillet, R. Cumbal, A. Pereira, J. Lopes, O. Engwall, and I. Leite.  Robot gaze can mediate participation imbalance in groups with different skill levels.  In _Proceedings of the 2021 ACM/IEEE International Conference on Human-Robot Interaction_ , HRI ’21, page 303–311, New York, NY, USA, 2021. Association for Computing Machinery.  ISBN 9781450382892.  doi: 10.1145/3434073.3444670.  URL https://doi.org/10.1145/3434073.3444670.
  * Gottman and Levenson [1985] J. M. Gottman and R. W. Levenson.  A valid procedure for obtaining self-report of affect in marital interaction.  _Journal of consulting and clinical psychology_ , 53(2):151, 1985.
  * Gottman and Levenson [1992] J. M. Gottman and R. W. Levenson.  Marital processes predictive of later dissolution: behavior, physiology, and health.  _Journal of personality and social psychology_ , 63(2):221, 1992.
  * Gottman and Levenson [2000] J. M. Gottman and R. W. Levenson.  The timing of divorce: Predicting when a couple will divorce over a 14-year period.  _Journal of Marriage and Family_ , 62(3):737–745, 2000.
  * Gunes and Schuller [2013] H. Gunes and B. Schuller.  Categorical and dimensional affect analysis in continuous input: Current trends and future directions.  _Image and Vision Computing_ , 31(2):120–136, 2013.  ISSN 0262-8856.  doi: https://doi.org/10.1016/j.imavis.2012.06.016.  URL https://www.sciencedirect.com/science/article/pii/S0262885612001084.  Affect Analysis In Continuous Input.
  * Herringer and Haws [1991] L. G. Herringer and S. C. Haws.  Perception of personality traits in oneself and others.  _The Journal of Psychology_ , 125(1):33–43, 1991\.  doi: 10.1080/00223980.1991.10543267.  URL https://doi.org/10.1080/00223980.1991.10543267.
  * Jeong et al. [2023] S. Jeong, L. Aymerich-Franch, K. Arias, S. Alghowinem, A. Lapedriza, R. Picard, H. W. Park, and C. Breazeal.  Deploying a robotic positive psychology coach to improve college students’ psychological well-being.  _User Modeling and User-Adapted Interaction_ , 33(2):571–615, Apr 2023.  ISSN 1573-1391.  doi: 10.1007/s11257-022-09337-8.  URL https://doi.org/10.1007/s11257-022-09337-8.
  * Jung et al. [2012] M. Jung, J. Chong, and L. Leifer.  Group hedonic balance and pair programming performance: Affective interaction dynamics as indicators of performance.  In _Proceedings of the SIGCHI Conference on Human Factors in Computing Systems_ , CHI ’12, page 829–838, New York, NY, USA, 2012. Association for Computing Machinery.  ISBN 9781450310154.  doi: 10.1145/2207676.2208523.  URL https://doi.org/10.1145/2207676.2208523.
  * Jung [2016] M. F. Jung.  Coupling interactions and performance: Predicting team performance from thin slices of conflict.  _ACM Trans. Comput.-Hum. Interact._ , 23(3), jun 2016.  ISSN 1073-0516.  doi: 10.1145/2753767.  URL https://doi.org/10.1145/2753767.
  * Jung [2017] M. F. Jung.  Affective grounding in human-robot interaction.  In _Proceedings of the 2017 ACM/IEEE International Conference on Human-Robot Interaction_ , pages 263–273, 2017.
  * Kuppens and Verduyn [2017] P. Kuppens and P. Verduyn.  Emotion dynamics.  _Current Opinion in Psychology_ , 17:22–26, 2017.
  * Lopes et al. [2017] P. Lopes, G. N. Yannakakis, and A. Liapis.  Ranktrace: Relative and unbounded affect annotation.  In _2017 Seventh International Conference on Affective Computing and Intelligent Interaction (ACII)_ , pages 158–163, 2017.  doi: 10.1109/ACII.2017.8273594.
  * Melhart et al. [2019] D. Melhart, A. Liapis, and G. N. Yannakakis.  Pagan: Video affect annotation made easy.  pages 130–136. IEEE, 9 2019.  ISBN 978-1-7281-3888-6.  doi: 10.1109/ACII.2019.8925434.  URL https://ieeexplore.ieee.org/document/8925434/.
  * Metallinou and Narayanan [2013] A. Metallinou and S. Narayanan.  Annotation and processing of continuous emotional attributes: Challenges and opportunities.  In _2013 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG)_ , pages 1–8, 2013.  doi: 10.1109/FG.2013.6553804.
  * Radford et al. [2022] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever.  Robust speech recognition via large-scale weak supervision, 2022.
  * Rammstedt and John [2007] B. Rammstedt and O. P. John.  Measuring personality in one minute or less: A 10-item short version of the big five inventory in english and german.  _Journal of Research in Personality_ , 41(1):203–212, 2007.  ISSN 0092-6566.  doi: https://doi.org/10.1016/j.jrp.2006.02.001.  URL https://www.sciencedirect.com/science/article/pii/S0092656606000195.
  * Ruef and Levenson [2007] A. M. Ruef and R. W. Levenson.  Continuous measurement of emotion.  _Handbook of emotion elicitation and assessment_ , pages 286–297, 2007.
  * Sack et al. [2023] M. J. Sack, M. T. Parreira, J. Fu, A. Lipman, H. Javed, N. Jamali, and M. Jung.  Corae: A tool for intuitive and continuous retrospective evaluation of interactions.  2023\.
  * Shek [2002] D. T. L. Shek.  Chinese adolescent’ explanations of poverty: the perceived causes of poverty scale.  _Adolescence_ , 37:789–804, 12 2002.  ISSN 00018449.
  * Stock-Homburg [2021] R. Stock-Homburg.  Survey of emotions in human–robot interactions: Perspectives from robotic psychology on 20 years of research.  _International Journal of Social Robotics_ , 14, 06 2021.  doi: 10.1007/s12369-021-00778-6.
  * Tennent et al. [2019] H. Tennent, S. Shen, and M. Jung.  Micbot: A Peripheral Robotic Object to Shape Conversational Dynamics and Team Performance.  _ACM/IEEE International Conference on Human-Robot Interaction_ , 2019-March:133–142, 2019.  ISSN 21672148.  doi: 10.1109/HRI.2019.8673013.
  * Whittaker et al. [2021] S. Whittaker, Y. Rogers, E. Petrovskaya, and H. Zhuang.  Designing personas for expressive robots: Personality in the new breed of moving, speaking, and colorful social home robots.  _J. Hum.-Robot Interact._ , 10(1), feb 2021.  doi: 10.1145/3424153.  URL https://doi.org/10.1145/3424153.