# Operationalizing Framing to Support Multiperspective Recommendations of
Opinion Pieces

Mats Mulder  Delft University of TechnologyDelftThe Netherlands
<EMAIL_ADDRESS>,  Oana Inel  Delft University of TechnologyDelftThe
Netherlands<EMAIL_ADDRESS>,  Jasper Oosterman  BlendleUtrechtThe
Netherlands<EMAIL_ADDRESS>and  Nava Tintarev  Maastricht
University Maastricht
The Netherlands<EMAIL_ADDRESS>
(2018)

###### Abstract.

Diversity in personalized news recommender systems is often defined as
dissimilarity, and based on topic diversity (_e.g._ , corona versus farmers
strike). Diversity in news media, however, is understood as multiperspectivity
(_e.g._ , different opinions on corona measures), and arguably a key
responsibility of the press in a democratic society. While viewpoint diversity
is often considered synonymous with source diversity in communication science
domain, in this paper, we take a computational view. We operationalize the
notion of framing, adopted from communication science. We apply this notion to
a re-ranking of topic-relevant recommended lists, to form the basis of a novel
viewpoint diversification method. Our offline evaluation indicates that the
proposed method is capable of enhancing the viewpoint diversity of
recommendation lists according to a diversity metric from literature. In an
online study, on the Blendle platform, a Dutch news aggregator platform, with
more than 2000 users, we found that users are willing to consume viewpoint
diverse news recommendations. We also found that presentation characteristics
significantly influence the reading behaviour of diverse recommendations.
These results suggest that future research on presentation aspects of
recommendations can be just as important as novel viewpoint diversification
methods to truly achieve multiperspectivity in online news environments.

recommender systems, viewpoint diversity, framing aspects

††copyright: acmcopyright††journalyear: 2018††doi:
10.1145/1122445.1122456††conference: FAccT ’21: ACM Conference on Fairness,
Accountability, and Transparency; March, 2021; ††booktitle: FAccT ’21: ACM
Conference on Fairness, Accountability, and Transparency, March 2021††price:
15.00††isbn: 978-1-4503-XXXX-X/18/06††ccs: Information systems Recommender
systems††ccs: Human-centered computing User studies††ccs: Human-centered
computing Empirical studies in HCI

##  1\. Introduction

In recent years, traditional news sources are increasingly using online news
platforms to distribute their content. Digital-born news websites and news
aggregators, which combine content from various sources in one service, are
also gaining ground (Newman et al., 2015). In 2015, 23% of survey respondents
reported online media as their primary news source, and 44% considered digital
and traditional sources equally relevant (Newman et al., 2015). This change
also induces a wide adoption of news recommender systems that automatically
provide personalized news recommendations to users.

Communication studies generally acknowledge two important roles of media in a
democratic society (Helberger, 2019). The first role is to inform citizens
about important societal and political issues. The second role is to foster a
diverse public sphere. Both roles are then related to multiple social-cultural
objectives of democracy, such as informed decision-making, cultural pluralism
and citizens welfare (McQuail, 1992; Strömbäck, 2005).

The role of news recommender systems in promoting these democratic values is
under heavy discussion in academic debate. For example, the term filter
bubbles received increasing awareness, suggesting that high levels of
personalisation would lock people up people in bubbles of what they already
know or think (Pariser, 2011). According to Helberger, the democratic role of
news recommender systems mainly depends on the democratic theory that is being
followed. In their conceptual framework, this role is being evaluated for the
most common theories: the liberal, the participatory and the deliberative
(Helberger, 2019). In particular in relation to the participatory and
deliberative model, the development of viewpoint diversification methods can
be motivated.

However, current diversification methods (Kunaver and Požrl, 2017; Ziegler et
al., 2005) do not address viewpoint diversity, but define diversity as
dissimilarity and operationalize it through topic diversity (_e.g._ , corona
versus farmers strike). Therefore, current diversification methods are not
applicable in the news domain, and novel viewpoint diversification methods are
needed to maintain and assure multiperspectivity in online news environments.
To truly enable _multiperspectivity_ , users should be willing to consume
viewpoint-diverse recommendations. Moreover, their behaviour should be studied
in real, online scenarios. Thus, we investigate the following research
questions:

R1: How is reading behaviour affected by viewpoint diverse news
recommendations?

R2: How is reading behaviour affected by presentation characteristics of
viewpoint diverse news recommendations?

To answer these questions, we propose a re-ranking approach for lists of
recommended articles based on aspects of news frames, a concept taken from
communication studies. In particular, a news frame describes how to identify a
view on an issue, in a given article (Entman, 1993). Thus, by bridging aspects
from the social and the computational domains, we aim to overcome the current
gap between the definition of diversity in recommender systems and news media.

During an offline evaluation, the proposed method increased the viewpoint
diversity of recommended lists of news articles on several topics. Further, we
measured the influence of the viewpoint diversification method on the reading
behaviour of more than 2000 users, which are likely to interact with the
recommended articles, in an online study on the Blendle platform, a Dutch news
aggregator platform. We found that reading behaviour of users that received
diverse recommendations was comparable with the reading behaviour of users
that received news articles optimized only for relevance. However, we did find
a positive influence of two presentation characteristics on the click-through
rate of recommendations, _i.e._ , news articles with thumbnails and news
articles with more hearts are more often read.

Therefore, we make the following contributions:

  * •

a novel method for viewpoint diversification using re-ranking of news
recommendation lists, based on framing aspects;

  * •

an online evaluation with more than 2000 users, on the Blendle platform, to
understand:

    1. (a)

how viewpoint-diverse recommendations affect the reading behaviour of users;
and

    2. (b)

how article’s presentation characteristics affect the reading behaviour of
users.

##  2\. Related Work

In this section, we first investigate how communication science understands
diversity. Then, we review current approaches for diversity in recommender
systems. These allow us to bridge the gap between the domains of communication
and computer science, by operationalizing framing aspects in a diversification
algorithm.

###  2.1. Diversity in News Media

In news media, diversity refers to multiperspectivity or a diversity of
viewpoints (Gans, 2003). In communication science, diversity is, in general, a
key measure for news quality (Porto, 2007; Choi, 2009; Masini et al., 2018),
thus fostering multiple democratic aspects, such as informed decision-making,
cultural pluralism and citizens welfare (Napoli, 1999; Voakes et al., 1996).
Two main approaches for assessing diversity can be distinguished: source and
content diversity (Napoli, 1999; Baden and Springer, 2017; Benson, 2009), with
most studies focusing on source diversity (Napoli, 1999; Baden and Springer,
2017; Voakes et al., 1996; Baden and Springer, 2014). When measuring source
diversity, most methods follow Bennett (1996)’s indexing theory, which assumes
that including non-official or non-elite sources corresponds to high levels of
diversity (Baden and Springer, 2017). Alternatively, Napoli (1999) approaches
the issue from a policymaker point of view and distinguishes three aspects of
source diversity: content ownership or programming, ownership of media
outlets, and the workforce within individual media outlets.

Critics, however, state that multiple sources can still foster the same point
of view and therefore, source diversity is not a direct measure for viewpoint
diversity (Voakes et al., 1996). Multiple studies also indicate that power
distributions in society, commercial pressure of news media and journalistic
norms and practices, significantly influence which sources gain media access
(Benson, 2009; Baden and Springer, 2017). Therefore, it is often argued that
viewpoint diversity can only be achieved by fostering content diversity
(Masini et al., 2018; Napoli, 1999; Choi, 2009; Gans, 2003; Baker, 2001;
Voakes et al., 1996). Content diversity is defined in (Van Cuilenburg, 1999)
as _“heterogeneity of media content in terms of one or more specified
characteristics”_. Baden and Springer (2017) identified six common approaches
to assess content diversity. The first three methods focus on the tone or
political position represented in the news, _i.e._ , the inclusion of non-
official positions, the diversity of political tone or analysis of political
slant. These methods, however, assume that political disagreement equals
viewpoint diversity (Baden and Springer, 2017). Another approach uses language
diversity to evaluate content diversity. However, this is again no direct
measure, since different language can describe the same perspective (Baden and
Springer, 2017).

The final two approaches use the concept of frames to assess content
diversity. Framing theory states that every communicative message selectively
emphasizes certain aspects of the complex reality (Baden and Springer, 2017).
Thereby, frames enable different interpretations of the same issue (Scheufele,
1999). Framing has been put forward by many scholars to enhance content
diversity. For example, Porto (2007) states that news environments need to be
evaluated by their ability to provide diverse frames. Baden and Springer
(2017) describe three frames’ aspects that are central to the role of
viewpoint diversity in democratic media. First, frames create different
interpretations of the same issue by selecting some aspects of the complex
reality (Gamson and Modigliani, 1989). Second, frames are not neutral but
suggest specific evaluations and courses of actions that serve some purpose
better than other (Entman, 1993). Third, frames are often strategically
constructed to advocate particular political views and agendas. Framing, thus,
can be a suitable conceptualization of viewpoint diversity.

###  2.2. Diversity in Recommender Systems

Traditionally, research on recommender systems focused on evaluating their
performance in terms of accuracy metrics (Ziegler et al., 2005). Such focus,
however, induced a problem which is known as over-fitting, _e.g._ , a model is
fitted so strongly to a user that it is unable to detect any other interests
(Kunaver and Požrl, 2017). Additionally, there is a need for a more user-
centric evaluation of recommender systems. Thus, diversity has become one of
the most prominent beyond-accuracy metrics for recommender systems (Ziegler et
al., 2005). In this context, diversity is generally defined as the opposite of
similarity (Kunaver and Požrl, 2017), and it is often based on topic diversity
(_e.g._ , corona versus farmers strike). For example, Ziegler et al. (2005)
proposed a topic diversification method based in the intra-list diversity
metric.

Current diversification methods for recommender systems, thus, do not focus on
viewpoint diversity and are not applicable in the news domain. To the best of
our knowledge, only one study for viewpoint diversification has been proposed
so far (Tintarev et al., 2018). Tintarev et al. (2018) propose a new distance
measure for viewpoint diversity based on linguistic representations of news
articles. This diversity measure was then applied in a post-processing re-
ranking algorithm (Carbonell and Goldstein, 1998) to a list of news articles.
These allowed optimizing for the balance between topic relevance and viewpoint
diversity. In a small scale user study (Tintarev et al., 2018), readers
indicated a lower intent to consume diversified content, motivating the need
to study behavioural measures for newsreaders on a larger scale. Thus, we
argue that more research is required to understand the relationship between
the metric and the influence on readers behaviour.

In this work, we aim to bridge the current gap between the notion of framing
in communication science and potential computational measure. Additionally, we
aim to study how viewpoint diversification affects the behaviour of
newsreaders in an applied setting. The next section justifies the
operationalization of _framing_ in the computational domain.

##  3\. Framing for viewpoint diversity

Framing is an extensively researched concept in different domains, including
psychology, communication and sociology, having its roots in the latter
domain. Bateson (1955) state that communication only gets meaning in its
context and by the way the message is constructed. Later, frame theory gained
increasing momentum and was generally understood as follows: every
communicative message selectively emphasizes certain aspects of a complex
reality (Baden and Springer, 2017). Thus, every news article (unintentionally)
comprises some form of framing (Baden and Springer, 2017). Frames are often
deliberately used to construct strategic, often political, views on a topic.
Consequently, frames enable different interpretations of the same issue (Baden
and Springer, 2017). However, every frame inevitably deselects other, equally
plausible and relevant frame (Baden and Springer, 2017).

When considering frames in news articles, multiple definitions exist (Giltin,
1980; Gamson and Modigliani, 1989; De Vreese, 2005). However, the definition
of Entman (1993) is the most commonly adopted in the literature. It states
that framing includes the selection of _“some aspects of perceived reality and
make the more salient in a communicating text, in such a way as to promote a
particular definition of a problem, causal interpretation, moral evaluation
and treatment recommendation for the item described”_. Within this definition,
the problem describes _four framing functions_ \- for which we also provide a
running example -, namely:

  1. (1)

Problem Definition : “what a causal agent is doing with what costs and
benefits”; _e.g., a second Coronavirus wave is approaching_ ;

  2. (2)

Causal Attribution : “identifying the forces creating the problem”; _e.g., (it
is due to the) government policy response_ ;

  3. (3)

Moral Evaluation : _“evaluate causal agents and their effects”_ ; _e.g._ ,
response to approaching second wave came too late (negative evaluation);

  4. (4)

Treatment Recommendation : “offer and justify treatments for the problems and
predict their likely effects”; _e.g., there must be predefined measures to be
deployed at a critical threshold of virus spread._

Additionally, Entman (1993) describes how to find frames at different levels
of analysis, including single sentences, paragraphs or articles as a whole.
Also, a frame may not necessarily include all the four functions.

Most framing analysis approaches focus on manual analysis of articles (Kroon
et al., 2016; Matthes and Kohring, 2008; Vliegenthart, 2012). Only recently,
some computer-assisted methods gained interest (Burscher et al., 2014; Vu and
Lynn, 2020; Greussing and Boomgaarden, 2017). As a result, the identification
of frames often falls into a methodological black box (Matthes and Kohring,
2008). Thereby, the main issue includes the ambiguity of _“which elements
should be present in an article or news story to signify the existence of a
frame”_ (Matthes and Kohring, 2008). To overcome this problem, some recent
studies (Matthes and Kohring, 2008; Vliegenthart, 2012; Baden and Springer,
2017) propose a novel identification method based on the extraction of the
four aforementioned framing aspects in the definition of Entman (1993).

###  3.1. Focus Group Setup

To guide the operationalization of the framing aspects, we started with a
qualitative analysis. Through a small focus group, we aimed to gain insights
into how the four framing functions of the main frame of an article manifest
in its content and how we can identify them computationally.

####  3.1.1. Participants

We invited three experts in the field of news article and framing analysis.
All experts had a background in journalism, communication, or news media. They
all had multiple years of relevant work experience.

####  3.1.2. Materials

As a basis for discussion during the focus group, we used opinion pieces on
the topic of _Dutch farmers protests_. Opinion pieces refer to news articles
that reflect the authors opinion and thus, do not claim to be objective. An
initial discussion with domain experts indicated that this type of news
article is the most suitable to identify framing functions.

####  3.1.3. Procedure

The focus group procedure consisted of two steps.

_1\. Annotation session:_ First, the participants were asked to perform
framing analysis on an opinion piece, using the four framing functions as
described by Entman (1993). In particular, the participants had to
individually highlight parts of the article, such as word clauses or
sentences, that can be related to one of the four framing functions of the
main frame of the news article.

_2\. Review session:_ Second, the results were discussed, together with some
general questions on news article analysis and framing. For every highlighted
part, we asked the participants to motivate why the highlighted part is
related to one of the four framing functions. Besides, we used the results as
input to a broader discussion on news article analysis and framing, such as:

  * •

What is the main heuristic that you used to analyze the article?

  * •

What procedure did you follow to analyze the framing functions of the article?

  * •

Can you derive any patterns in the way framing functions manifest in opinion
pieces?

###  3.2. Results of Framing Analysis

During the review session, all experts indicated that they used the article
structure as the main heuristic to find the framing functions regarding the
main frame. They also pointed out that opinion pieces are still strongly
shaped by journalistic values on how an article should be structured. We
further analyzed this heuristic according to the four framing functions:

  1. (1)

Problem Definition : In opinion pieces, the first part of the article often
presents the main problem that the author addresses and includes the title,
the lead, and the first x paragraphs. Work on manual frame analysis (Kroon et
al., 2016) supports this finding. The number of introductory paragraphs, x,
can be different per source, author, or article.

  2. (2)

Causal Attribution \+  Moral Evaluation : The body of an article is used to
analyze the main problem and usually contains different factors that
contribute to the problem under investigation and their evaluation. We can
match this with: a) the causal attribution of a frame (forces creating the
problem), and b) the moral judgements (evaluate the causal attribution and
their effect) (Entman, 1993).

  3. (3)

Treatment Recommendation : Treatment recommendations can be seen as
suggestions to improve or solve the issue described by the problem definition
of the main frame. They normally appear in the concluding paragraphs,
according to the focus group members.

Note, however, that this structure is only a heuristic and it only applies to
opinion pieces. Other types, such as interviews, are structured differently.

The results of the annotation session also indicate that each framing function
related to the main frame of an article can normally be found within one
paragraph. Additionally, a paragraph can include multiple framing functions,
but words, clauses, and sentences generally represent a single framing
function.

##  4\. Dataset

In this section, we describe the experimental dataset, which consists of
opinion pieces, in Dutch. The choice of article type is motivated by the focus
group session presented in Section 3, in which the structure of this article
type is put forward as the primary heuristic to find framing aspects. We
picked topics that we expected a) to be present on the Blendle platform at the
time when we performed the online user study; b) to contain different
viewpoints addressed in the news; and c) to balance issues that more current
versus long-standing. The dataset consists of four ongoing topics: _Black
Lives Matter_ , _Coronavirus_ , _U.S. Elections_ \- as more current topics,
and the dominance and privacy issues around _Big Tech_ \- as a long-standing
topic.

Table 1. Queries used (in Dutch) to retrieve news articles for the four topics
in our dataset.

Topic | Search Query |  | Start
---
Date
| Black Lives
---
Matter
| (’black lives matter’ OR ’racisme debat’ OR ’blm-demonstraties’ OR ’George
Floyd’ OR ’racisme-debat’) AND NOT (’belastingdienst’ OR ’corona’)
---
| June 15
---
2020
Coronavirus |  | ’corona’ OR ’covid-19’ OR ’mondkapjes’ OR ’mondkapje’ OR ’mondmasker’ OR ’mondkapjesplicht’ OR ’coronatest’ OR ’coronatesters’ OR ’rivm’
---
OR ’virus’ OR ’viroloog’ OR ’golf’ OR ’topviroloog’ OR ’uitbraak’ OR
’uitbraken’ OR ’coronaregels’ OR ’versoeplingen’ OR ’staatssteun’ OR ’vaccin’
| June 1
---
2020
U.S. Elections |  | ’Donald Trump’ AND (’presidentsverkiezingen’ OR ’Verkiezingen’ OR ’campagne’ OR ’verkiezingsstrijd’ OR ’verkiezingscampagne’ OR ’Joe biden’)
---
| June 1
---
2020
Big Tech |  | (’macht’ OR ’machtig’ OR ’privacy’ OR ’data’ OR ’privacyonderzoek’ OR privacy-schandaal’) AND (’big tech’ OR ’tech-bedrijven’ OR techbedrijven’)
---
2018

We collected our dataset from an archive containing more than 5 million Dutch
news articles. The archive is known to undergo checks for articles quality, to
remove undesirable content, such as the weather or short actualities. For each
topic, we used the search terms (queries) and restrictions shown in Table 1.
We provide the list of search terms in their original language, Dutch, because
we do not want to add additional bias through translation. Additionally, since
the proposed method heavily relies on the structure of the article, we set up
a filter for the minimum number of words to 450 and a filter for the minimum
number of paragraphs to 5.

Table 2 provides an overview of the dataset, per topic. While the length of
the articles varies across topics, they are usually far longer than the
450-word limit we chose. Four publishers are present for all topics: De
Volkskrant, De Standaard, Trouw and Het Algemeen Dagblad. Furthermore, De
Volkskrant is the most prominent publisher for all topics, except for the
_U.S. Elections_ topic. The inclusion of other, less frequent, publishers
varies per topic. Overall, our dataset covers a set of 15 unique publishers.

We also present some properties concerning the presentation characteristics of
the articles on the news aggregator website. We observe that the ratio of
articles that contains a thumbnail image depends on the topic. For the _Black
Lives Matter_ and _Coronavirus_ topics, more than half of the articles have a
thumbnail image, while the opposite holds for the other two topics. The number
of custom titles from the editorial team and the average title length also
differ considerable per topic. Only a few articles have an editorial title,
and they usually appear for the _Big Tech_ and _U.S. Elections_ topics.

Table 2. Overview of the experimental dataset, per topic.

Topic | Articles | Publishers |  | Avg
---
#Words
| With
---
thumb.
| With
---
ed. title
| Avg title
---
length
| Black Lives
---
Matter
69 | 10 | 697 | 39 | 1 | 6.3
Coronavirus | 52 | 7 | 608 | 27 | 4 | 5.2
U.S. Elections | 42 | 6 | 744 | 20 | 8 | 9.6
Big Tech | 51 | 10 | 761 | 17 | 10 | 8.1

##  5\. Viewpoint Diversity Methodology

We proposed a novel diversification method based on framing aspects, using the
insights from the focus group. First, we describe the extraction pipeline,
which supports the structure heuristic described in the results of the focus
group session (Section 3). The pipeline forms the basis for the generation of
recommendation lists that we use in the offline evaluation (Section 6) and the
online study (Section 7). We implemented the pipeline using methods employed
by the news aggregator platform and off-the-shelf natural language processing
toolkits, such as IBM-Watson. We chose to use state-of-the-art and off-the-
shelf methods used by the news aggregator platform to ensure output quality.
Then we describe the distance function, which combines the metadata related to
each framing aspect in a measure for viewpoint diversity for news articles.
Finally, we present the re-ranking algorithm based on this viewpoint diversity
measure. Our contribution, therefore, stands in the novelty of the overall
diversification framework, rather than the implementation of specific
components. Figure 1 shows an overview of the end-to-end pipeline.

Figure 1. Viewpoint diversification pipeline

###  5.1. Metadata Extraction

For each framing aspect, as described in the definition of Entman, we
implemented an extraction pipeline:

#####  Problem Definition

As described in Section 2, the problem definition can be understood as the
central issue or topic under investigation (Matthes and Kohring, 2008).
Therefore, we decided to use a topic model as the main extraction method for
this framing aspect. The model, provided by the research partner, included a
1000-topic latent Dirichlet allocation (LDA) model trained on 900k Dutch news
articles. Based on the conclusions from the focus group described in Section
3, the title and the first x paragraphs are used to retrieve metadata related
to this framing aspect. We also applied multiple pre-processing steps on the
content, including cleaning, chunking, tokenization, lemmatization and stop-
word removal.

#####  Causal Attribution \+  Moral Evaluation

According to Entman (1993), the causal attribution of a frame relates to the
forces creating the problem, while the moral judgements evaluate the causal
attribution and their effect. From the discussion of the focus group session,
described in Section 3, we concluded that the body of an article usually
elaborates on these aspects. Additionally, paragraph-level seems to be the
most suitable level of analysis. Therefore, a text-classification algorithm
was applied using the IBM Watson Natural Language Processing API. The service
returns a category for each paragraph according to a predefined five-level
taxonomy, from the most general category (_e.g._ level 1 - technology and
computing), to the most specific one (_e.g._ , level 5 - portable computer).
To extract information related to the evaluation of these attributions, we
also analyze the sentiment of these paragraphs, using the IBM Watson NLP API.
Thereby, it would be able to identify if two articles evaluate the same
aspects of a problem differently. The content of interest for this task
includes all paragraphs except the $x$ introductory and $y$ concluding
paragraphs. We optimize these variables during the offline evaluation.

#####  Treatment Recommendation

Following the definition of Entman (1993), a treatment recommendation suggests
remedies for problems and predicts their likely effect. The research domain of
suggestion mining, which involves the task of retrieving sentences that
contain advice, tips, warnings and recommendations from opinionated texts
(Negi, 2019), was found to be highly relevant for this framing aspect (Negi et
al., 2016). However, the state-of-the-art models are topic-specific (Negi et
al., 2016), and can not be easily applicable to our domain. Thus, only the
more naive rule-based approach could be applied for this study, being more
generally applicable. In a crowdsourcing task with domain experts, we
evaluated, and we optimized the generally applicable rules from the literature
on the news article content. Afterwards, we implemented the method to extract
sentences that contain suggestions from the article content. Then, to obtain
comparable information between the suggestions of two articles, the suggestion
sentences of each were classified using the same text-classification algorithm
that was used for the causal attribution framing aspect. Corresponding to the
conclusion of the focus group described in Section 3, the content of interest
for this framing aspect includes the $y$ concluding paragraphs of an article.
We optimize this variable in the offline evaluation.

###  5.2. Distance Functions

Having defined the extraction pipeline for each framing aspect, _i.e._ ,
problem definition, causal attribution, moral evaluation and treatment
recommendation (Entman, 1993), we now define our distance function. We compare
the extracted metadata for every pair of articles. Thus, we implement a
distance function for each framing aspect.

#####  Problem Definition

The metadata regarding the problem definition framing aspect involves a
probability distribution over 1000 topics. Thus, we need a statistical
distance measure. We chose the Kullback-Leibler divergence because it is one
of the most commonly used statistical distance measures for LDA-models, and it
is used in the comparable work (Tintarev et al., 2018) on viewpoint
diversification.

#####  Causal Attribution and  Moral Evaluation

We compare the five-level taxonomy categories extracted from the pipeline
described in the previous section, to obtain a distance measure for the causal
attribution framing function of the primary frame. Thus, we use the weighted
Jaccard index, which measures the similarity (or diversity) of two sets
(Jaccard, 1901). The index is calculated for each level of detail in the five-
level taxonomy, such that we apply weight factors per taxonomy level. Thereby,
overlap in higher levels of detail can contribute more to the overall
similarity score. In the offline evaluation, we compare different weight
factors per taxonomy-levels.

For the moral evaluation framing aspect, we implement the distance function by
multiplying the Jaccard distance and the absolute sentiment difference between
each paragraph combination of two articles. Thus, paragraphs with no
overlapping categories yield a value of zero, while highly similar paragraphs,
with different sentiment scores, lead to high levels of diversity related to
the moral evaluation framing aspect.

#####  Treatment Recommendation

For the treatment recommendation we used the five-level taxonomy
classification, _i.e._ , from the most general to the most specific category,
as returned by IBM Watson Natural Language Processing API, and the Jaccard
index.

###  5.3. Re-ranking

We implement the re-ranking of the input list of articles using the Maximal
Marginal Relevance (MMR) algorithm (Carbonell and Goldstein, 1998). In our
case, the re-ranking consists of ranking news articles that are more diverse
higher. First, we normalize the output of the distance functions related to
each framing aspect using a min-max normalization, and then we combine them in
a diversity score through a weighted sum. We optimize the weight factors
during the offline evaluation. We note here that we re-rank news articles that
are known to also be relevant for the given topic.

Where most re-ranking algorithms for recommender systems order lists only on
relevance, the MMR algorithm provides a linear combination between diversity,
in our case viewpoint diversity, and relevance, set by the parameter
$\lambda$. Thus, the re-ranking algorithm is defined as follows:

(1) $MMR\equiv max_{i\in R\setminus S}[\lambda(Rel(i)-(1-\lambda)max_{j\in
S}(1-Div(i||j))]$

Since this work proposes a measure for viewpoint diversity rather than a
relevance measure, we decided to implement the relevance score using a simple
frequency-inverse document frequency (TF-IDF) score.

##  6\. Offline Evaluation

In this section, we describe the offline evaluation of our viewpoint
diversity-driven approach for re-ranking lists of news articles.

###  6.1. Materials

For our offline experiment, we used the news dataset introduced in Section 4,
which covers 214 news articles on four topics.

###  6.2. Procedure

The experimental procedure consists of four main steps that we detail as
follows. First, we process and enrich all the news articles in our dataset
according to the four framing aspects as defined by Entman (1993): problem
definition, causal attribution, moral evaluation, and treatment
recommendations (for details see Section 5.1).

Second, we generate the diversity matrix by comparing all combinations of two
articles, based on the enrichment described in Section 5.1. Thus, using the
distance function defined in Section 5.2 we measure the dissimilarity of two
articles based on the framing aspects. Finally, since the MMR algorithm re-
ranks a list of news articles based on a linear combination between diversity
and relevance, we calculate the TF-IDF relevance matrix, including a relevance
score for each two article combination.

Third, we optimize the model variables and evaluate the performance using
cross-validation. For each article $i$ in the dataset, we calculate a set of
$s$ recommendations by re-ranking the remainder articles in the dataset. To
prevent over-fitting, we use cross-validation. Thus, we split the dataset into
$k$ distinct sets. We experimented with different values of $k={5,10,20}$ and
$s={3,6,9}$. For every set, we take the following steps:

  1. (1)

Grid search of model variables on training set: The training set contains the
$k-1$ subsets of articles. We obtain the optimal combination of the model
variables for the training set using a grid search. An overview of the model
variables can be found in Table 3 and in Section 6.2.1.

  2. (2)

Evaluation on test set: After the variables are trained on the $k-1$ subsets,
the model is evaluated on the test set for different values of $\lambda$,
between 0 and 1 with a step of 0.1. As described before, for each article in
the test set, a set of $s$ recommendations is calculated by re-ranking the
remaining articles in the dataset.

And finally, we combined the results of all $k$ cross-validations.

####  6.2.1. Model variables

Table 3 shows the model variables that we optimize during the offline
evaluation. We choose the variation of the weights for each framing aspect
such that no single framing aspect can have the majority. Additionally, a
step-size of 0.1 is assumed to bring enough variation. We consider two
variations for the taxonomy level weights: equal weights for each taxonomy
level or ascending weights. Finally, the number of introductory and concluding
paragraphs can be either $1$ or $2$.

Table 3. Overview of possible values of model variables

Variable | Values
---|---
Weight Framing function - Problem Definition | [0.1, 0.2, 0.3, 0,4]*
Weight Framing function - Causal Attribution | [0.1, 0.2, 0.3, 0,4]*
Weight Framing function - Moral Evaluation | [0.1, 0.2, 0.3, 0,4]*
Weight Framing function - Treatment Recommendation | [0,1, 0.2, 0.3, 0,4]*
Taxonomy level weight | [equal, ascending]
Number of introducing paragraphs | [1, 2]
Number of concluding paragraphs | [1, 2]
$\lambda$ | [0.0, 0.1, …, 0.9]

  * •

*Note that all framing function weight factors should sum up to 1

###  6.3. Evaluation Metrics

We assess the performance of the viewpoint diversification method using a
metric from literature (Tintarev et al., 2018). The metric is based on the
Intra-List Diversity metric (Zhang and Hurley, 2008; Ziegler et al., 2005;
Tintarev et al., 2018; Vargas and Castells, 2011) and it is defined as the
average distance between all pairs of articles $i$ and $j$, such that $i\neq
j$. Thereby, the distance between a pair is defined by the articles’ channels
(predefined taxonomy of 20 high-level topics) and the articles’ LDA topic-
distribution, as derived from the enrichment methods in Section 5:

(2) $Distance(i,j)=0.5\times Distance_{Channels}+0.5\times Distance_{LDA}$

The channel distance is calculated using the cosine distance, whereas the LDA
distance is computed using the Kullback-Leibler divergence.

####  6.3.1. Additional metrics

Besides the viewpoint diversity metric, we also measure the effectiveness of
the diversification model on other properties, as follows:

_Relevance_ : We measure the TF-IDF relevance for the recommendation lists,
such that we can measure the effectiveness of the viewpoint diversification
method.

_Kendall’s $\tau$_: We compute the Kendall’s $\tau$ rank correlation
coefficient (Kendall, 1948) to measure the similarity between two ranks of
recommended items.

_Average number of words_ : We compute the average number of words for the
recommended article lists as a measure of quality (_i.e._ , longer news
articles can be considered to be higher quality).

_Publisher Ratio_ : We measure the publisher ratio for the recommendation
lists because this could potentially provide insights on the effect of the
content diversity on the source diversity.

###  6.4. Baseline

To assess if the proposed diversification method can increase the viewpoint
diversity based on the presented metric, we compare it with a baseline,
consisting of a full relevance MMR, where $\lambda=1$, such that we rank the
recommendations purely on the TF-IDF relevance. We chose this baseline because
it has minimal effects on the recommendations in terms of viewpoint diversity.

(a) Topic: Black Lives Matter

(b) Topic: Coronavirus

(c) Topic: U.S. Elections

(d) Topic: Big Tech

Figure 2. Diversity and relevance scores for different values of $\lambda$ per
topic.

(a) Topic: Black Lives Matter

(b) Topic: Coronavirus

(c) Topic: U.S. Elections

(d) Topic: Big Tech

Figure 3. Average number of publishers in recommendation lists, normalised by
the input ratio, for all topics.

###  6.5. Results

In Figure 2, we show the performance of the model in terms of viewpoint
diversity and relevance for different values of $\lambda$, and the optimal
setting of the model variables. The red bars represent the results of the
viewpoint diversity metric, while the blue bars represent the relevance
scores. Variations of the cross-validation variable $k$ did not yield
significant differences between the results, and thus, we fixed $k=10$. The
list size $s$ did show to influence the number of publishers included in the
recommended list, but the results were not significant. Thus, we fixed the
list size to $s=3$, to better align the offline evaluation set up with the
online evaluation set up, where only 3 recommended news articles can be shown
at a time. Table 4 shows the optimal model variables values, per topic.

Across all topics, the proposed diversification method is capable of
increasing the viewpoint diversity of recommendation lists. According to the
metric, the viewpoint diversity increases on average from 0.55 to 0.79 between
$\lambda=1$ and $\lambda=0$. Additionally, the average relevance score
decreases from 0.58 to 0.27.

Table 4. Overview of model variables used during the offline and online
evaluation for each topic: cross validation folds ($k$), recommended list size
($s$), number of introductory paragraphs, number of concluding paragraphs,
general weights for the four framing aspects, category weights and $\lambda$.

Topic | $k$ | $s$ |  | intro.
---
par.
| concl.
---
par
| general
---
weight
| cat.
---
weight
$\lambda$
| Black Lives Matter
---
10 | 3 | 2 | 1 | [0.2, 0.4, 0.1, 0.3] | eq | 0
Coronavirus | 10 | 3 | 2 | 1 | [0.1, 0.4, 0.1, 0.4] | eq | 0
U.S. Elections | 10 | 3 | 1 | 2 | [0.1, 0.4, 0.1, 0.4] | eq | 0
Big Tech | 10 | 3 | 1 | 2 | [0.2, 0.4, 0.1, 0.3] | asc | 0

##### Kendall’s $\tau$

We computed the Kendall’s $\tau$ rank correlation to assess whether the
proposed diversification method is capable of providing different
recommendation lists compared to the baseline. We computed the coefficient
between the baseline ($\lambda=1$) and each other value of
$\lambda=[0.0,0.1,...,0.9]$. Overall, we observed that the re-ranking of the
set of recommendations based on viewpoint diversity results in different
recommendation lists compared to the baseline. The coefficient decreases for
smaller values of $\lambda$, but it is bounded around $\tau=0$ for decreasing
values of $\lambda$.

##### Average number of words

We observe no consistent pattern in the average number of words for different
values of $\lambda$ across topics. For the _Black Lives Matter_ and _Big Tech_
topics, the average number of words increases for larger values of $\lambda$,
for the _U.S. Elections_ topic the average decreases and for _Coronavirus_ the
average is stable.

##### Publisher ratio

Figure 3 shows the average number of articles in the recommended lists,
normalized by the input ratio, for each value of $\lambda$. For every topic,
the number of publishers increases for larger values of $\lambda$ and the
number of different publishers for the baseline recommendation list is larger
than the one in the diverse recommendation list. Thus, we observe that the
diversification method influences the publisher ratio. For small values of
$\lambda$, some publishers get amplified, while others are excluded. We see
this effect primarily for the topics of _U.S. Elections_ and _Big Tech_. The
topic of _Corona Virus_ seems to be the only exception.

##  7\. Online Study

We conducted a between-subjects online study on the Blendle platform to
compare the reading behaviour of users who receive news articles optimized
only for relevance, versus news articles that are also diverse on viewpoint.

###  7.1. Materials

In the online study, we used the articles collected in Section 4.

###  7.2. Participants

We selected 2076 active users of the news aggregator platform. These users
were assumed to most likely see and use the recommendation functionality. We
included only users who clicked at least four times on a recommended article
below any article read, in the last 14 days before the study. Groups for
baseline and diversified recommendations were created by randomly splitting
the users.

###  7.3. Independent Variables

In the between-subjects user study we manipulated the following conditions,
referring to the recommended list of news articles:

  * •

baseline recommendation: was implemented using a MMR that was based only on
relevance ($\lambda=1.0$)

  * •

diversified recommendation: was implemented using a MMR that maximized
viewpoint diversity ($\lambda=0.0$)

###  7.4. Procedure

During the two-week experiment, six days per week, we provided recommendations
for two articles featured on the selected users’ homepage. We provided sets of
three recommendations below the content on the reading page of the original
article. Every morning, we chose these two articles manually, to match any of
the topics that we selected (_Black Lives Matter_ , _Big Tech_ , _Coronavirus_
, and _U.S. Elections_). Afterwards, both the baseline and diversified
recommendation sets were calculated for both articles and included in the news
aggregator platform.

###  7.5. Dependent Variables

To analyze the reading behaviour of the two different user groups and answer
RQ1, we measure specific events on the news aggregator platform (_i.e._ ,
check whether the user opened the article and if the user finished reading the
article). Based on these available events, we observe multiple implicit
(click-through-rate per news article, click-through-rate per recommendation
set and completion rate of recommendation) and explicit (heart ratio) measures
of the reading behaviour. To answer RQ2, we look into presentation
characteristics of the recommended articles (_i.e._ , presence of editorial
title, presence of thumbnail and counting number of hearts).

_1\. Click-through rate per article:_ The number of clicks on a news article
is divided by the total number of users who finished one of the original news
articles for which that article was recommended. The completion of an original
news article is registered using a scroll-position.

_2\. Click-through rate per recommendation set:_ The total number of clicks on
either of the three news articles in the recommendation set is divided by the
number of users who finished the original news article (using scroll-position)
for which the recommendation set was presented.

_3\. Completion rate of recommendation:_ Is implemented as the number of users
that read the full recommended article (using scroll-position) divided by the
number of users who opened the news article. The completion rate is assumed to
be a measure for the user satisfaction with the recommendations. We can argue
that short news articles are more likely to be completed than long news
articles. Thus, we also analyze the completion rate of a news article in
relation to the number of words in the news article.

_4\. Favourite ratio:_ The news aggregator platform allows users to mark an
article as a favourite, illustrated by an icon of a heart. The users can click
this icon at the end of the article content. We implemented the measure as the
number of users of the user group (baseline or diverse) that clicked on the
icon, divided by the number of users in the same group that completed the
article. The metric is assumed to be a marker of user satisfaction with the
article.

_5\. Presentation characteristics:_ We measured three additional properties of
a recommended article during the experiment, which referred to the
presentation characteristics of recommended news articles. First, the
editorial team can replace the original title of a news article with a custom,
editorial title. In general, these custom titles are longer and more
explanatory than the original ones. Second, articles can be presented with or
without a thumbnail image. Third, the number of users who selected the article
as a favourite is visualised by a counting number of hearts in the left-upper
corner of an article banner. All three properties are assumed to potentially
influence the click-through rate and are, therefore, measured during the
experiment.

_6\. Source diversity:_ Finally, we also measured the influence of the source
diversity of the recommendation set on the click-through rate. As seen in
Section 6, higher levels of viewpoint diversity showed to influence the number
of times a publisher is included in the recommendation.

###  7.6. Results

The online study ran six days a week for two weeks. Thus, we provided
recommendations below 24 articles. During the experiment, the topic of
_Coronavirus_ became extremely prominent, so we provided recommendations below
18 out of 24 news articles on this topic. In contrast, the _Black Lives
Matter_ topic lost all actuality, resulting in no recommendations for this
topic. For the _U.S. Elections_ topic, we provided recommendations below four
articles, and for the _Big Tech_ topic, below two news articles.

##### Click-through rate per recommended article

The mean click -through rate per recommended article for the baseline
recommendations was 0.11 (stderr. = 0.011) while for the diversified
recommendations was 0.087 (stderr. = 0.0083) when looking at all topics.
Furthermore, according to the Mann-Whitney U test (U=570, p-val$>$0.05), we
did not find a significant difference between the two user groups in terms of
click-through rate per recommended article. The same result holds per topic.

##### Click-through rate per recommended set

The mean click-through rate per recommended set for the baseline
recommendations was 0.31 (stderr. = 0.016) while for the diversified
recommendations was 0.25 (stderr. = 0.016) when looking at all topics (Figure
4(a)). According to the Mann-Whitney U test (U=2.9, p-val$<$0.05), we find a
significant difference between the mean click-through rate per recommended
sets for the two user groups. Per topic, we find such difference significant
only for _Coronavirus_ , shown in Figure 4(b), with a click-through rate per
recommended set of 0.32 (stderr. = 0.018) for the baseline recommendations and
0.25 (stderr. = 0.018) for the diversified recommendations (U=80.0,
p-val$<$0.05). For the other topics, we found no significant difference
between the two user groups.

##### Completion rate

We found no significant difference in terms of completion rate for the two
user groups. We also applied the Spearman’s rank correlation to see whether
the completion rate is correlated with the length of the articles. However, we
did not find any correlation in either of the two conditions.

##### Heart ratio

We found no significant difference, for all topics and across topics, in terms
of heart ratio for the two user groups. This suggests that the quality of the
recommendations was comparable between the two conditions.

(a) Click-through rate per recommended set, for the two user groups.

(b) Click-through rate per recommended set and per topic, for the diversified
user group.

(c) Influence of the thumbnail image as presentation characteristic, for the
two user groups.

(d) Influence of the hearts as presentation characteristic, for the two user
groups.

Figure 4. Overview of significant results in the online study

####  7.6.1. Influence of presentation characteristics

We measured the influence of three factors, namely the presence of an
editorial title, the presence of a thumbnail and the number of users that
chose the article as a favourite on the click-through rate of an article.

##### Editorial title

Regarding the influence of the inclusion of an editorial title on the click-
through rate, no statistical significance was found for neither user groups.

##### Thumbnail image

We found no significant influence of the inclusion of a thumbnail image on the
click-through rate for baseline users. In contrast, recommendations with a
thumbnail are 3.1% more times opened than recommendations without a thumbnail
for diverse users, as seen in Figure 4(c), a difference that is also
statistically significant.

##### Favorite articles

We applied the Spearman’s rank correlation to see whether we find a
correlation between the click-through rate and the number of hearts. Figure
4(d) shows the distribution of click-through rates and the number of hearts.
We only found a moderate positive correlation of 0.57, also statistically
significant (p-val$<<$0.05) for the diversified user group.

####  7.6.2. Source diversity

As seen in the offline evaluation, higher levels of viewpoint diversity turned
out to have remarkable effects on the publisher ratio. Therefore, we also
evaluated the effect of the source diversity of a recommendation set on the
click-through rate. For each recommendation set, we computed the number of
different publishers and we found recommendation sets in which all articles
are from a different publisher and sets in which two articles are from the
same publisher. Afterwards, the click-through was calculated for each
category. The results for both the baseline users and diverse users show that
no statistically significant difference can be found in the click-through rate
between two or three different publishers in the recommendation set for
neither baseline nor diversified users.

##  8\. Discussion

We first discuss the results of the offline and online evaluation and then
provide an overview of the limitations of our approach.

###  8.1. Offline Evaluation

The offline evaluation indicated that the proposed method is capable of
increasing the viewpoint diversity of recommendation sets according to the
metric defined in (Tintarev et al., 2018). The average viewpoint diversity
scores across all topics increased from 0.55 to 0.79 for an increasing level
of diversity in the MMR algorithm. Simultaneously, the average relevance score
decreased from 0.58 to 0.27. Remarkably, the diversity score of 0.41 in
(Tintarev et al., 2018) is considerably smaller than the maximum average value
of 0.79 found in this work. A possible factor could be the fact that in
(Tintarev et al., 2018) the LDA topic model was excluded from the
diversification method to prevent any interference with the evaluation metric,
whereas the diversification method in this work still depends on an LDA topic-
model. Therefore, the difference in viewpoint diversity scores between the
methods can possibly appear due to the interference of metadata between the
viewpoint diversity metric and diversification method in this work.

A remarkable effect of the diversification algorithm that was found in the
offline evaluation includes the decreasing publisher ratio for larger
contributions of diversity in the MMR. After investigating the effect in more
detail, it was found that the maximum frequency an article is included in the
recommendation lists is around 2 to 4 times higher at $\lambda$ = 0, compared
to $\lambda$ = 1. Thus, for larger contributions of diversity, the algorithm
increasingly selects the same article for the recommendation lists. This could
be a possible explanation of the decreasing publisher ratio, suggesting that
some outliers in the dataset get amplified, thereby suppressing the inclusion
of different sources. To be able to study this effect thoroughly, the offline
evaluation could have benefited from a setup in which it was possible to
assess the contribution of individual framing aspects to the global viewpoint
diversity score per article.

We can conduct a broader discussion about the viewpoint diversity metric used.
Although approaches that use source diversity are more popular, scholars
generally agree that viewpoint diversity can only be achieved by fostering
content diversity, because, multiple sources can still refer to the same point
of view (Voakes et al., 1996). Based on these findings, this study used a
content-based approach. From the results of the offline evaluation, it became
clear that increasing levels of content diversity exclude multiple publishers
and thus, decreases source diversity. Moreover, some specific publishers got
amplified remarkably for high levels of content diversity. Therefore,
viewpoint diversification methods could benefit from considering both content
and source diversity.

###  8.2. Online Evaluation

No major influence of viewpoint diversification on the reading behaviour was
found, except for the click-through rate calculated per recommendation set,
which indicated a statistically significant difference between baseline and
diverse users of 6.5% (in favour for baseline recommendations). However, the
results of the click-through rate calculated per recommendation indicated no
significant difference between the two user groups. Likewise, the other two
measurements of the reading behaviour, including the completion rate of
recommendations and the ratio of users who selected a recommendation as a
favourite, showed no significant difference between baseline and diverse
users.

In reflection on the motivation of this study, the proposed diversification
for news media is capable of enhancing the viewpoint diversity of news
recommendation, while maintaining comparable measures of the reading behaviour
of users. The results thus suggest that recommender systems are capable of
preserving the quality standards of multiperspectivity in online news
environments. Thereby, situations of extreme low diversity, known as filter
bubbles, could also be mitigated.

These results are in contrast with the most comparable study, Tintarev et al.
(2018), who found a negative effect on intent to read diversified news
articles. The authors proposed a viewpoint diversification method based on the
MMR-algorithm with linguistic features, such as gravity, complexity and
emotional tone. During a user study, 15 participants were asked to make a
forced choice between a recommendation from the diverse set and a
recommendation from the baseline set, after reading an article on the same
topic. It was found that 66% of the participants chose the baseline article,
compared with 33% who chose the diverse article. However, in the current
study, we observed the reading behaviour of both user groups without them
being aware, and we argue that the present setup simulates the situation in a
more realistic way.

Additionally, the results shed light on the importance of how a recommendation
is presented. Multiple presentation properties, such as the inclusion of a
thumbnail image and the number of times an article is marked as favourite,
were shown to have a significant influence on the click-through rate of
recommendations. Future research, thus, should not only address the capability
of a model to enhance viewpoint diversity according to an offline metric but
also evaluate what presentation characteristics could impact the users’
willingness to read multiperspectival news. Related research on viewpoint-
aware interfaces, which aim to explain the recommendation choices to users,
can be seen as very valuable (Tintarev, 2017; Nagulendra and Vassileva, 2014).

###  8.3. Limitations

We further discuss the limitations of our approach.

Choice of participants in the online study. Only users who frequently followed
recommendations below articles were selected for the experiment. Thus, the
click-through rates presented in this study are higher than for average news
readers.

Limited number of topics and articles. For both the online and offline
evaluations, we used only opinion pieces. Furthermore, each evaluation had a
limited number of topics, namely four, as well as a limited number of news
articles. New topics could reveal additional results that hold across topics.

Missing user perceptions. While we were able to study user behavior at a
reasonable scale, a notable omission is users’ qualitative judgement of
viewpoint diversity in the resulting recommendations. We plan to continue
collaborating with the news aggregator platform to refine the proposed
framework, _i.e._ , to improve the viewpoints extraction.

Presentation characteristics. Some presentation characteristics, and in
particular the heart ratio, could also be markers of quality. Further
qualitative analysis is needed to _e.g._ , understand how much of user
behavior is directed by quality. We also saw that for some topics the presence
of thumbnail was more common than for other topics, and it would be relevant
to study whether this also interacted with user perceptions of relevance or
quality.

Relevance metric. The offline study could use a more sophisticated relevance
measure between the recommendation and the original article. The relevance
score was based on a simple TF-IDF score, limited to the terms in a
handcrafted search query.

Influence of $\lambda$. Given limited time for online testing, we only
compared against a maximum viewpoint diversity score.

Influence of publishers. In Figure 3 we see that, although 15 publishers are
represented in the datasets, three publishers are predominant. Due to the
limited number of articles and the unbalance in terms of publishers, the
inclusion of a wide variety of perspectives on a topic can be challenged.

##  9\. Conclusions

In this paper, we proposed a novel method for enhancing the diversity of
viewpoints in lists of news recommendations. Inspired by research in
communication science, we identified frames as the most suitable
conceptualization for news content diversity. We operationalized this concept
as a computational measure, and we applied it in a re-ranking of topic
relevant recommended lists, to form the basis of a novel viewpoint
diversification method.

In an offline evaluation, we found that the proposed method improved the
diversity of the recommended items considerably, according to a viewpoint
diversity metric from literature. We also conducted an online study with more
than 2000 users, on the Blendle platform, a Dutch news aggregator. The reading
behaviour of users receiving diversified recommendations was largely
comparable to those in the baseline. Besides, the results suggest that
presentation characteristics (thumbnail image, and the number of hearts) lead
to significant differences in reading behaviour. These results suggest that
research on presentation aspects for recommendations may be just as relevant
as novel viewpoint diversification methods, to achieve multiperspectivity in
automated online news environments.

As future work, we plan to investigate further the presentation
characteristics and how they influence user experience, in addition to
behaviour. In more controlled settings, we will study the relative effects of
actual (e.g., as judged by experts) versus perceived quality (e.g., number of
hearts in the interface) of recommended news items. Future work will also
focus on defining a better metric to measure viewpoint diversity, as opposed
to topic diversity, c.f., (Draws et al., [n.d.]). Additionally, we learnt that
contextual information, _i.e._ , general knowledge about a topic (_e.g._ , the
current measures in place to stop the spread of coronavirus) can also be
essential to reveal a specific frame. We hope that this work will encourage
further research on how framing can be defined, conceptualized, and evaluated
in the computational domain.

## References

  * (1)
  * Baden and Springer (2014) Christian Baden and Nina Springer. 2014.  Com (ple) menting the news on the financial crisis: The contribution of news users’ commentary to the diversity of viewpoints in the public debate.  _European journal of communication_ 29, 5 (2014), 529–548.
  * Baden and Springer (2017) Christian Baden and Nina Springer. 2017.  Conceptualizing viewpoint diversity in news discourse.  _Journalism_ 18, 2 (2017), 176–194.
  * Baker (2001) C Edwin Baker. 2001\.  _Media, markets, and democracy_.  Cambridge University Press.
  * Bateson (1955) Gregory Bateson. 1955\.  A theory of play and fantasy; a report on theoretical aspects of the project of study of the role of the paradoxes of abstraction in communication.  _Psychiatric research reports_ 2 (1955), 39–51.
  * Bennett (1996) W Lance Bennett. 1996\.  An introduction to journalism norms and representations of politics.  (1996).
  * Benson (2009) Rodney Benson. 2009\.  What makes news more multiperspectival? A field analysis.  _Poetics_ 37, 5-6 (2009), 402–418.
  * Burscher et al. (2014) Björn Burscher, Daan Odijk, Rens Vliegenthart, Maarten De Rijke, and Claes H De Vreese. 2014\.  Teaching the computer to code frames in news: Comparing two supervised machine learning approaches to frame analysis.  _Communication Methods and Measures_ 8, 3 (2014), 190–206.
  * Carbonell and Goldstein (1998) Jaime Carbonell and Jade Goldstein. 1998.  The use of MMR, diversity-based reranking for reordering documents and producing summaries. In _Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval_. 335–336.
  * Choi (2009) Jihyang Choi. 2009\.  Diversity in foreign news in US newspapers before and after the invasion of Iraq.  _International Communication Gazette_ 71, 6 (2009), 525–542.
  * De Vreese (2005) Claes H De Vreese. 2005\.  News framing: Theory and typology.  _Information design journal & document design_ 13, 1 (2005).
  * Draws et al. ([n.d.]) Tim Draws, Nava Tintarev, Ujwal Gadiraju, Alessandro Bozzon, and Benjamin Timmermans. [n.d.].  Assessing Viewpoint Diversity in Search Results Using Ranking Fairness Metrics. In _BIAS Workshop in association with ECMLPKDD’2020_.
  * Entman (1993) Robert M Entman. 1993\.  Framing: Toward clarification of a fractured paradigm.  _Journal of communication_ 43, 4 (1993), 51–58.
  * Gamson and Modigliani (1989) William A Gamson and Andre Modigliani. 1989.  Media discourse and public opinion on nuclear power: A constructionist approach.  _American journal of sociology_ 95, 1 (1989), 1–37.
  * Gans (2003) Herbert J Gans. 2003\.  _Democracy and the News_.  Oxford University Press.
  * Giltin (1980) Todd Giltin. 1980\.  _The whole world is watching: Mass media in the making and unmaking of the new left_.  McGraw-Hill.
  * Greussing and Boomgaarden (2017) Esther Greussing and Hajo G Boomgaarden. 2017.  Shifting the refugee narrative? An automated frame analysis of Europe’s 2015 refugee crisis.  _Journal of Ethnic and Migration Studies_ 43, 11 (2017), 1749–1774.
  * Helberger (2019) Natali Helberger. 2019\.  On the democratic role of news recommenders.  _Digital Journalism_ 7, 8 (2019), 993–1012.
  * Jaccard (1901) Paul Jaccard. 1901\.  Étude comparative de la distribution florale dans une portion des Alpes et des Jura.  _Bull Soc Vaudoise Sci Nat_ 37 (1901), 547–579.
  * Kendall (1948) Maurice George Kendall. 1948\.  Rank correlation methods.  (1948).
  * Kroon et al. (2016) Anne C Kroon, Alena Kluknavska, Rens Vliegenthart, and Hajo G Boomgaarden. 2016.  Victims or perpetrators? Explaining media framing of Roma across Europe.  _European Journal of Communication_ 31, 4 (2016), 375–392.
  * Kunaver and Požrl (2017) Matevž Kunaver and Tomaž Požrl. 2017.  Diversity in recommender systems–A survey.  _Knowledge-Based Systems_ 123 (2017), 154–162.
  * Masini et al. (2018) Andrea Masini, Peter Van Aelst, Thomas Zerback, Carsten Reinemann, Paolo Mancini, Marco Mazzoni, Marco Damiani, and Sharon Coen. 2018\.  Measuring and explaining the diversity of voices and viewpoints in the news: A comparative study on the determinants of content diversity of immigration news.  _Journalism Studies_ 19, 15 (2018), 2324–2343.
  * Matthes and Kohring (2008) Jörg Matthes and Matthias Kohring. 2008.  The content analysis of media frames: Toward improving reliability and validity.  _Journal of communication_ 58, 2 (2008), 258–279.
  * McQuail (1992) Denis McQuail. 1992\.  _Media performance: Mass communication and the public interest_. Vol. 144.  Sage London.
  * Nagulendra and Vassileva (2014) Sayooran Nagulendra and Julita Vassileva. 2014.  Understanding and controlling the filter bubble through interactive visualization: a user study. In _Proceedings of the 25th ACM conference on Hypertext and social media_. 107–115.
  * Napoli (1999) Philip M Napoli. 1999\.  Deconstructing the diversity principle.  _Journal of communication_ 49, 4 (1999), 7–34.
  * Negi (2019) Sapna Negi. 2019\.  _Suggestion mining from text_.  Ph.D. Dissertation. NUI Galway.
  * Negi et al. (2016) Sapna Negi, Kartik Asooja, Shubham Mehrotra, and Paul Buitelaar. 2016.  A study of suggestions in opinionated texts and their automatic detection. In _Proceedings of the Fifth Joint Conference on Lexical and Computational Semantics_. 170–178.
  * Newman et al. (2015) Nic Newman, David AL Levy, and Rasmus Kleis Nielsen. 2015\.  _Reuters Institute digital news report 2015: Tracking the future of news_.  Reuters Institute for the Study of Journalism.
  * Pariser (2011) Eli Pariser. 2011\.  _The filter bubble: How the new personalized web is changing what we read and how we think_.  Penguin.
  * Porto (2007) Mauro P Porto. 2007\.  Frame diversity and citizen competence: Towards a critical approach to news quality.  _Critical Studies in Media Communication_ 24, 4 (2007), 303–321.
  * Scheufele (1999) Dietram A Scheufele. 1999\.  Framing as a theory of media effects.  _Journal of communication_ 49, 1 (1999), 103–122.
  * Strömbäck (2005) Jesper Strömbäck. 2005\.  In search of a standard: Four models of democracy and their normative implications for journalism.  _Journalism studies_ 6, 3 (2005), 331–345.
  * Tintarev (2017) Nava Tintarev. 2017\.  Presenting diversity aware recommendations: Making challenging news acceptable.  (2017).
  * Tintarev et al. (2018) Nava Tintarev, Emily Sullivan, Dror Guldin, Sihang Qiu, and Daan Odjik. 2018.  Same, same, but different: algorithmic diversification of viewpoints in news. In _Adjunct Publication of the 26th Conference on User Modeling, Adaptation and Personalization_. 7–13.
  * Van Cuilenburg (1999) Jan Van Cuilenburg. 1999\.  On competition, access and diversity in media, old and new: Some remarks for communications policy in the information age.  _New media & society_ 1, 2 (1999), 183–207.
  * Vargas and Castells (2011) Saúl Vargas and Pablo Castells. 2011.  Rank and relevance in novelty and diversity metrics for recommender systems. In _Proceedings of the fifth ACM conference on Recommender systems_. ACM, 109–116.
  * Vliegenthart (2012) Rens Vliegenthart. 2012\.  Framing in mass communication research–An overview and assessment.  _Sociology Compass_ 6, 12 (2012), 937–948.
  * Voakes et al. (1996) Paul S Voakes, Jack Kapfer, David Kurpius, and David Shano-yeon Chern. 1996.  Diversity in the news: A conceptual and methodological framework.  _Journalism & Mass Communication Quarterly_ 73, 3 (1996), 582–593.
  * Vu and Lynn (2020) Hong Tien Vu and Nyan Lynn. 2020.  When the news takes sides: Automated framing analysis of news coverage of the Rohingya crisis by the elite press from three countries.  _Journalism Studies_ (2020), 1–21.
  * Zhang and Hurley (2008) Mi Zhang and Neil Hurley. 2008.  Avoiding monotony: improving the diversity of recommendation lists. In _Proceedings of the 2008 ACM conference on Recommender systems_. ACM, 123–130.
  * Ziegler et al. (2005) Cai-Nicolas Ziegler, Sean M McNee, Joseph A Konstan, and Georg Lausen. 2005.  Improving recommendation lists through topic diversification. In _Proceedings of the 14th international conference on World Wide Web_. ACM, 22–32.