# Mitigating Media Bias through Neutral Article Generation

Nayeon Lee Yejin Bang Andrea Madotto Pascale Fung
Center for Artificial Intelligence Research (CAiRE)
Hong Kong University of Science and Technology
<EMAIL_ADDRESS>

###### Abstract

Media bias can lead to increased political polarization, and thus, the need
for automatic mitigation methods is growing. Existing mitigation work displays
articles from multiple news outlets to provide diverse news coverage, but
without neutralizing the bias inherent in each of the displayed articles.
Therefore, we propose a new task, a single neutralized article generation out
of multiple biased articles, to facilitate more efficient access to balanced
and unbiased information. In this paper, we compile a new dataset (NeuWS),
define an automatic evaluation metric, and provide baselines and multiple
analyses to serve as a solid starting point for the proposed task. Lastly, we
obtain a human evaluation to demonstrate the alignment between our metric and
the human judgment.

##  1 Introduction

Media bias refers to the bias produced when journalists report about an event
in a prejudiced manner or with a slanted viewpoint Gentzkow and Shapiro
(2006). Since news media play a crucial role in shaping public opinion toward
various important issues De Vreese (2004); McCombs and Reynolds (2009); Perse
and Lambe (2016), bias in media could reinforce the problem of political
polarization. Due to its potential societal harm, this issue has been
extensively studied in social sciences, and there have been both journalistic
and computational efforts to detect and analyze media bias Entman (1993);
Groseclose and Milyo (2005); Recasens et al. (2013). However, computational
methods for media bias mitigation are still under-explored.

|  [Left] Dan Bishop’s narrow win suggests that
---
Republicans need to dial up their efforts to keep
the suburbs in 2020.
|  [Right] Republicans scored an important victory
---
in North Carolina Tuesday as President Trump
helped them hang on to a GOP congressional
seat in a closely-watched special election.
|  [Neutral] Republican Dan Bishop won an
---
special election in North Carolina on Tuesday.

Table 1: Illustration of news articles from different political ideologies
(Left vs. Right). The highlighted spans refer to lexical and informational
bias.

Currently, the main computational approach to mitigating media bias is
aggregation of multiple news articles to provide a comprehensive reporting
with the additional analysis of news outlets Park et al. (2009); Sides (2018);
Zhang et al. (2019). In this way, readers can access diverse news coverage,
however, at the cost of reading more articles than what they would normally
read. Moreover, since bias within each individual article is not neutralized,
it could still undesirably sway readers’ views. According to Bail et al.,
exposure to opposing political views can actually reinforce political
polarization. Therefore, presenting articles from different stances alone
cannot entirely solve the problem.

As a remedy, we take a step forward from the news aggregation approach and
propose a new task – generating a single neutralized article out of multiple
biased articles on the same event (multi-document neutralization111For ease of
writing, we refer to our task as “neutralization” in the rest of the paper).
The articles from conflicting news outlets still share the “same set of
underlying facts”, however, convey different impression of same event through
deliberate omission of certain fact and slanted choice of words Gentzkow and
Shapiro (2006). With an automatic method to extract and aggregate neutral
information from multiple articles, the public could easily access more
unbiased news information, leading to reduced risk of political polarization.

In this work, we formulate our new task by i) constructing a weakly-labeled
news neutralization dataset, which we call NeuWS, ii) defining a general model
setup, and iii) designing an automatic metric called the Neutral score to
evaluate the success of the neutralization. Then, we establish a solid
baselines by leveraging large pre-trained model for neutralization. Lastly, we
conduct human evaluation to examine how aligned our Neutral score is with
human judgement, and provide interesting insights from additional experimental
analysis that suggests potential directions for future work.

##  2 Related Work

#### Media Bias Detection and Prediction

Media bias has been studied extensively in various fields such as social
science, economics and political science, and various measures have been used
to analyze the political preference of news outlets Groseclose and Milyo
(2005); Miller and Riechert (2001); Park et al. (2011); Gentzkow and Shapiro
(2010); Haselmayer and Jenny (2017). For instance, Gentzkow and Shapiro count
the frequency of slanted words within articles. In natural language processing
(NLP), computational approaches for detecting media bias consider lexical
bias, which is linguistic cues that induce bias in political text Recasens et
al. (2013); Yano et al. (2010); Lee et al. (2019); Hamborg et al. (2019b).
While these methods specifically focus on the lexical aspect of media bias,
our work attempts to address media bias more comprehensively.

As highlighted by Fan et al., media bias also has an informational aspect due
to framing bias, which is selective reporting of an event to sway readers’
opinions – e.g., omission of crucial facts or choice of words Entman (1993,
2007); Gentzkow and Shapiro (2006). Efforts related to informational bias Park
et al. (2011); Fan et al. (2019) are constrained to detection tasks. In this
work, we attempt to tackle mitigation of media bias (both lexical and
informational bias).

#### Media Bias Mitigation

News aggregation, displaying articles from different news outlets on a
particular topic (e.g., Google News,222https://news.google.com/ Yahoo
News333https://news.yahoo.com/), is the most common approach in NLP to
mitigate media bias, but it still has limitations Hamborg et al. (2019a).
Thus, multiple approaches have proposed to provide additional information
Laban and Hearst (2017), such as automatically classified multiple view points
Park et al. (2009), multinational perspectives Hamborg et al. (2017), and
detailed media profiles Zhang et al. (2019).
Allsides.com444https://www.allsides.com/ provides bias ratings of each news
outlet alongside balanced political coverage. However, they focus on making
news consumers more aware of what they are reading. Thus far, there has been
no attempt to automatically aggregate biased articles to produce a single
neutralized article.

#### Controlled Text Generation

One line of work under controlled text generation tries to de-bias or
neutralize text Dathathri et al. (2019); Pryzant et al. (2020); Ma et al.
(2020). Specifically, efforts were made to reduce toxicity Dathathri et al.
(2019), implicit social bias Ma et al. (2020) or subjectivity Pryzant et al.
(2020) in generations. Our work differs from theirs in two ways: 1)
Neutralization in the previous works is close to revision of single input text
with a focus on style, while our work is neutralized aggregation of multiple
texts. 2) We focus on media bias. To the best of our knowledge, we are the
first to attempt politically neutralized news article generation.

#### Hallucination

Recent studies have shown that neural sequence models can suffer from
hallucination of additional content, not supported by the input, as result,
adding factual inaccuracy to the generation of abstractive summarization
models. To address this problem, many researchers proposed methods to measure
factual inconsistency Holtzman et al. (2019); Kryściński et al. (2019); Zhou
et al. (2020); Lux et al. (2020); Gabriel et al. (2020), and to correct them
Zhao et al. (2020); Cao et al. (2020); Dong et al. (2020). While these works
focus on the factual inaccuracy and inconsistency, we focus on the bias that
is not factually incorrect but can still pose a problem due to the way how it
affects the readers’ opinion.

Event | Democratic presidential candidates ask to see full Mueller report
---|---
Left |  | Democrats want access to special counsel Robert Mueller’s investigation into Russian interference in
---
the 2016 presidential election before President Donald Trump has a chance to
interfere. […] Sen. Mark
Warner said in a statement: “Any attempt by the Trump Administration to cover
up the results of this
investigation into Russia’s attack on our democracy would be unacceptable.”
Right |  | Democratic presidential candidates wasted no time Friday evening demanding the immediate public
---
release of the long-awaited report from Robert S. Mueller III. […] Several
candidates, in calling for
the swift release of the report, also sought to gather new supporters and
their email addresses by
putting out “petitions” calling for complete transparency from the Justice
Department.

Table 2: Illustration of informational bias spans and lexical bias spans from
BASIL dataset Fan et al. (2019)

##  3 Task Formulation

###  3.1 Task Definition

The main objective of this work is to neutralize biased news articles from two
different news outlets (left-winged and right-winged555For simplicity, we use
two biased articles, but this can be extended to more than two articles.) into
a single neutral article which (i) retains as much information as possible and
(ii) eliminates as much bias as possible from the input articles.

#### Media Bias Definition

We follow the categorization and definition of media bias from Fan et al.
There are two types of media bias: lexical bias, which refers to the writing
style or linguistic attributes that may mislead readers, and informational
bias, which refers to tangential or speculative information pieces to sway the
minds of readers. Such biases can make an article convey a different
impression of what actually happened Gentzkow and Shapiro (2006). Ideally, a
neutral article should avoid both types of bias by using a neutral tone and
including balanced information without preference towards any particular
stance or target.

| Left | Right | Center | Total
---|---|---|---|---
Train/Valid | 580 | 580 | 580 | 1,740
Test | 100 | 100 | 100 | 300
Table 3: NeuWS data statistics. Article triplet refers to the set of {left,
right, center} news for each events.

###  3.2 Neutralize NeWS Dataset (NeuWS)

The neutralization task requires a dataset that consists of politically
opposing source articles and neutral target article, reporting about same
event. We, therefore, build a weakly-labeled data from article triplets
consisting of articles from politically left, right and center publishers. The
dataset language is English, and mainly focuses on the U.S. political events
only.

We term our dataset to be weakly-labeled on the following basis. First, there
is no single answer to writing a neutral news, thus, there cannot be “the
gold” neutral article to optimize for. Second, since all news requires
editorial judgements on what is the “important” information to report, it is
possible that even news articles from the most neutral publisher contain some
bias. Note that, the literature still considers center publishers to be bias-
free, especially in comparison to other hyper-partisan publishers Baum and
Groeling (2008).

For the political orientations of the publishers, we rely on the Media Bias
Ratings666https://www.allsides.com/media-bias/media-bias-ratings, which uses
editorial reviews, blind bias surveys (10,000+ community participants),
independent reviews, and third party research to correctly judge the political
stance of various publishers.

#### Train/Valid Set

To construct this train/valid set, we first crawled the URLs of article
triplets from Allsides.com, which displays news coverage of events from left,
right, and center publishers. Then, we built custom news crawlers to obtain
the full article content from the collected URLs. In total, we collected 1,740
full articles which were compiled into 580 article triplets. The data
statistics are listed in Table 3.

#### Test Set

For test set, we utilize a subset of BASIL dataset Fan et al. (2019) which
contains sentence-level annotation of media bias within news articles. These
annotation of bias spans are key to the measurement of the neutralization
performance. This dataset is only used in testset because it is small (100
samples) to be split into train/val/test.

We extend the BASIL dataset by adding center articles. BASIL dataset
originally consists of article triplets from Huffington Post (left-wing), Fox
News (right-wing) and New York Times (left-wing). The New York Times is the
outlet closest to center among the three. However, its political leaning is
still considered pro-Liberal Puglisi (2011); Chiang and Knight (2011).
Therefore, we replace the New York Times articles with those from center
publishers (e.g. Reuters, BBC) that report on the same event. To ensure that
the newly collected center news articles are covering the same event as the
left/right articles, we manually confirmed the content and the publication
dates.

#### Notations

Here, we introduce the notations used throughout the paper. We denote the two
biased articles with $X^{(l)}=\\{x^{(l)}_{1},\cdots,x^{(l)}_{L}\\}$ and
$X^{(r)}=\\{x^{(r)}_{1},\cdots,x^{(r)}_{R}\\}$, and a center article as
$X^{(c)}=\\{x^{(c)}_{1},\cdots,x^{(c)}_{C}\\}$, where $x$ represents a token.
These three articles form one triplet of our dataset
$D=\\{(X_{i}^{(l)},X_{i}^{(r)},X_{i}^{(c)})\\}_{i=1}^{N}$. Then, we denote the
set of annotated bias span $B^{(l)}=\\{X^{(l)}_{i,j}|1\leq i\leq j\leq z\\}$
and $B^{(r)}=\\{X^{(r)}_{i,j}|1\leq i\leq j\leq m\\}$ as the bias sub-strings
in $X^{(l)}$ and $X^{(r)}$ respectively. An example of bias spans is shown in
Table 2.

###  3.3 General Model Architecture

#### Model

Following the current state-of-the-art in sequence-to-sequence modelling, we
propose to firstly encode the concatenation of the two articles and to then
generate a neutralized article token-by-token using a decoder. Hence, given
the two articles $X^{(l)}$ and $X^{(r)}$ as a single sequence of tokens, the
encoder processes the input as follows,:

$H=\mathrm{ENC}_{\theta}(X^{(l)};X^{(r)}),$ (1)

where $H\in\mathbb{R}^{z+n\times d}$ and $d$ is the hidden feature size. Note
that the conversion between tokens and embedding is done directly in the
encoder $(ENC)$. This hidden representation is then passed to the decoder,
which generates an article $\hat{X}^{(n)}$ token-by-token in an autoregressive
manner. More formally,

$\hat{X}^{(n)}=\mathrm{DEC}_{\theta}(H),$ (2)

In the experiment section (Section 4), we provide more details on how we train
this general model.

| Baseline | Rouge-1 | Bias | Neutral
---|---|---|---|---
A. Zero-shot | $\texttt{BART}_{cnn}$ | 16.93 | 30% | 11.85
$\texttt{BART}_{XSum}$ | 7.79 | 6% | 7.32
B. Fine-tuning | $\texttt{BART}_{ft}$ | 38.11 | 42% | 22.10
$\texttt{T5}_{ft}$ | 18.72 | 38% | 11.61
C. $\texttt{BART}_{ft}$\+ Disinfo | $\texttt{BART}_{ft+\mathcal{L}_{\texttt{FakeNews}}}$ | 47.90 | 54% | 22.03
$\texttt{BART}_{ft+\mathcal{L}_{\texttt{Prop}}}$ | 46.86 | 23% | 36.08

Table 4: Baseline experiment results using beam-4 decoding technique. For
Rouge-1 Recall and Neutral, higher number is better. For Bias, lower number is
better.

Figure 1: Plot of experiment results. The closer to the top-left, the better.

###  3.4 Neutralization metric

Two important goals for successful neutralization are to minimize bias and to
maximize information recall. Therefore, we introduce ways to measure each
aspect and combine the both to serve as the final neutralization score
(Neutral).

#### Bias Metric

The most important criterion for success is to assess whether the generated
article $\hat{X}^{(n)}$ manages to filter out bias spans ($B^{(l)}$ and
$B^{(r)}$) originally existing in the input articles.

To quantify this, we obtain generated neutralized-articles for the whole test
set ($\\{\hat{X}_{i}^{(n)}\\}_{i=1}^{N}$), and measure the ratio that still
contains at least one bias span. The lower the ratio, the better the
neutralization performance (i.e., less bias spans). Formally, we define the
bias score as follows:

$\displaystyle\textsc{Bias}=\frac{1}{N}\sum_{k=1}^{N}\textsc{BiasExists}(B_{k}^{(A)},\hat{X}_{k}^{(n)}),$
(3)

where $B^{(A)}=\\{B^{(l)}\cup B^{(r)}\\}$ refers to the union of the bias
spans from left-wing and right-wing articles, and BiasExists refers to a
function that identifies the existence of bias in a given generated article
($\hat{X}$):

$\textsc{BiasExists}(B,\hat{X})=\\\ \begin{cases}1&\exists X_{i,j}\in B\
\mathrm{and}\\\ &X_{i,j}\in\hat{X}\\\ 0&\text{Otherwise.}\end{cases}$ (4)

#### Information Recall

One of the easiest way of deceiving the Bias metric is to generate random text
that has nothing to do with the biased source articles. Therefore, it is
crucial to also ensure that key information is being retained while removing
the bias.

We adopt the Rouge-1 Lin (2004) recall score between generated neutralized-
article, $\hat{X}^{(n)}$, and center article, $X^{(c)}$ to measure the
information coverage. The higher the Rouge-1 score, the better the information
covered by the neutralized generation. We mainly report unigram-based Rouge-1
for simplicity, but Rouge-2 and Rouge-L are also reported in the appendix for
interested readers.

#### Neutralization Score (Neutral)

Since the ultimate goal is to optimize for both the information recall score
and bias score, we define a single neutralization score by combining the two:

$\textsc{Neutral}=\textsc{{Rouge-1}}~{}\times(1-\textsc{Bias}).$ (5)

Through multiplication, both scores get equal weighting in the final metric.

##  4 Models and Experiments

Since neutralized article generation is a new task with no baseline, we
propose three different baseline methods to serve as reference points.

###  4.1 Zero-shot Using Summarization Model

Summarization, the task of producing a shorter version of one or several
documents that preserves most of the input’s meaning, has a similar setup to
our proposed neutralization task. Therefore, we investigate the zero-shot
neutralization performance of two strong BART-based Lewis et al. (2019)
summarization models by utilizing their pre-trained weights to initialize our
encoder ($\mathrm{ENC}_{\theta}$) and decoder ($\mathrm{DEC}_{\theta}$). We
test with two version of BART trained on the CNN/DailyMail Hermann et al.
(2015) and XSum Narayan et al. (2018) dataset – $\texttt{BART}_{cnn}$ and
$\texttt{BART}_{XSum}$.

###  4.2 Fine-tuning Seq2Seq Model

Many Transformer-based Vaswani et al. (2017) pre-trained language models
Raffel et al. (2019); Lewis et al. (2019) achieved excellent performance in
downstream tasks through simple fine-tuning with, small, task-specific data.
Therefore, we fine-tune the encoder-decoder parameters of pre-trained Seq2seq
models by minimizing the negative log-likelihood over the training set $D$. We
experiment with the following two pre-trained models:

  * •

$\texttt{BART}_{ft}$: pre-trained BART-large model fine-tuned with NeuWS data.

  * •

$\texttt{T5}_{ft}$: pre-trained T5-base Raffel et al. (2019) model fine-tuned
with NeuWS data.

###  4.3 Incorporating Disinformation Loss

Next, we explore adding an additional loss when fine-tuning the Seq2Seq model.
We encourage the model to learn about hyper-partisanship writing and, as
result, learn to avoid generating text alike. This is done by incorporating an
additional classification head on top of the encoder ($\mathrm{ENC}_{\theta}$)
to jointly optimize for this classification cross-entropy loss with the
original negative log-likelihood loss from the decoder
($\mathrm{DEC}_{\theta}$). In our experiments, the following settings are
explored:

  * •

$\texttt{BART}_{ft+\mathcal{L}_{\texttt{FakeNews}}}$: pre-trained BART-large
jointly fine-tuned on NeuWS and fake news dataset Potthast et al. (2017).

  * •

$\texttt{BART}_{ft+\mathcal{L}_{\texttt{Prop}}}$: pre-trained BART-large
jointly fine-tuned on NeuWS and propagandistic sentence detection task Da San
Martino et al. (2019), which is to classify whether a given sentence contains
any propagandistic technique (e.g., “Name calling”, “Appeal to fear”).

###  4.4 Experimental Details

All our experimental codes are based on the HuggingFace library Wolf et al.
(2020). During training, and across models, we used the following hyper-
parameters: $10$ epoch size, $3e-5$ learning rate and a batch size of $8$. We
did not do hyper-parameters tuning since our objective is to provide various
baselines and analysis. Training run-time for all of our experiments are fast
($<6$hr). No pre-processing of the text was made, except for the concatenation
of left and right articles with special token [SEP] in the middle (i.e. “
left-article [SEP] right-article ”). When concatenating, we ensured the first
half to begin with left-articles and the other half to begin with right-
articles. This was done to avoid any unintended bias from the ordering of the
ideology in the input.

##  5 Results

All the experimental results are reported in Table 4.

#### A. Zero-shot

From Table 4, we can observe that the zero-shot performance of both
summarization models on the proposed neutralization task are poor in terms of
Neutral. One possible reason is due to the difference in the training data
distribution, especially regarding the characteristics of the target
generation. In fact, summarization tasks focus on obtaining concise and
representative “summary” whereas our task focuses on obtaining neutral and
representative article. Within the summarization models, we can observe that
different training datasets (CNN vs XSum) lead to different neutralization
performance ($11.85$ vs. $7.32$).

It is important to note that the final Neutral score of $\texttt{BART}_{XSum}$
is very low ($7.32$) despite having the lowest Bias score ($6\%$). The reason
behind $\texttt{BART}_{XSum}$’s low bias score is its extremely short
generation length ($1.01\pm 0.10$ sentence on average). By design, our Neutral
metric includes the Rouge-1 score as well, which serves as a counteractor in
such scenario. This illustrates the effectiveness of considering both the
comprehensiveness and neutrality of the generated article to avoid pitfalls.

#### B. Fine-tuning

We compare and report two fine-tuned Seq2Seq models. It is evident that the
choice of the base model (i.e., $\texttt{BART}_{ft}$ vs. $\texttt{T5}_{ft}$)
greatly affects the neutralization performance after the fine-tuning.
$\texttt{BART}_{ft}$ achieves double the Neutral score compared to the zero-
shot baselines, but $\texttt{T5}_{ft}$ hardly shows any improvement. A likely
explanation for this observation would be the difference in the parameter size
of these two base models. BART-large has 406M parameters when T5-base only has
220M, so $\texttt{BART}_{ft}$ has access to more rich features and bigger
model capacity.

From the zero-shot and fine-tuning experiments, we can observe a weak positive
correlation between the Rouge-1 scores and the Bias scores - i.e. lowest
Rouge-1 result ($7.79$) with the lowest Bias score ($6\%$) and highest Rouge-1
($38.11$) with the highest Bias score ($42\%$). This is because these baseline
models failed to select the important information in a neutral manner from
input articles. With this positively correlating pattern, achieving even the
highest performance in one metric would not lead to a good neutralization
score. To illustrate, achieving 100.00 Rouge-1 can still result in 0.00
Neutral score if Bias is also 100. Therefore, more sophisticated models are
required to correctly identify as much important information as possible while
avoiding selecting bias spans at the same time.

#### C. Incorporating Disinformation Loss

In this section, we analyze the impact of adding additional disinformation
loss to the $\texttt{BART}_{ft}$ model’s encoder. This experiment is based on
$\texttt{BART}_{ft}$ since it is the best performing model from the earlier
fine-tuning experiment. The most notable result is that the result of
$\texttt{BART}_{ft+\mathcal{L}_{\texttt{Prop}}}$ does not follow the
concerning pattern observed earlier. This is illustrated in Fig 1, where
$\texttt{BART}_{ft+\mathcal{L}_{\texttt{Prop}}}$ is leaning toward the left-
top. Both the “$\texttt{BART}_{ft+\mathcal{L}_{\texttt{FakeNews}}}$” and
“$\texttt{BART}_{ft+\mathcal{L}_{\texttt{Prop}}}$” experiments have similar
Rouge-1 scores with only 1.04 difference, but with a clear distinction in Bias
scores and, as result, different Neutral scores. Propaganda loss is
empirically shown to be more effective in reducing the media bias in the
generation, achieving the highest Neutral score ($36.08$).

Decoding Technique | Hparam | Rouge-1 | Bias | Neutral
---|---|---|---|---
Indiv. | Avg.
BEAM | B:4 | 46.86 | 23% | 36.08 | 35.61
B:5 | 46.85 | 25% | 35.14
Top-k | k:5,t:1.0 | 36.51 | 21% | 28.84 | 29.27
k:50,t:1.0 | 36.67 | 19% | 29.70
Top-p | p:0.8,t:1.0 | 44.93 | 17% | 37.29 | 36.63
p:0.9,t:1.0 | 43.33 | 17% | 35.96

Table 5: Effect of decoding technique choices on the Neutral. All results are
obtained from $\texttt{BART}_{ft+\mathcal{L}_{\texttt{Prop}}}$ . Event 1:
Obama willing to use executive orders on guns
---
[$\texttt{BART}_{ft}$]: President Obama unveiled a sweeping plan on Wednesday
to curb gun violence that includes a ban on assault weapons and a 10-round
limit on high-capacity magazines, as well as measures to improve school safety
and mental health. The president’s proposal, unveiled at the White House after
a month-long review by Vice President Joe Biden and other top officials, is
expected to face stiff opposition from gun-control advocates and Republican
lawmakers, who have vowed to fight it in Congress.
[$\texttt{BART}_{ft+\mathcal{L}_{\texttt{Prop}}}$]: President Obama said
Wednesday he will propose sweeping gun control measures to address a growing
gun violence crisis in the United States, including a ban on assault-style
weapons and requiring background checks for all gun purchases. The proposal
was unveiled in response to the latest deadly shootings in America, the latest
in which 20 people were killed by a gunman with an assault-type rifle.
Event 2: Netanyahu plans to deliver a speech to Congress
[$\texttt{BART}_{ft}$]: Susan Rice, President Obama’s national security
adviser, said on Tuesday that Israeli Prime Minister Benjamin Netanyahu’s
planned speech in front of a joint session of Congress will be “destructive”
to U.S.-Israel relations. “It’s destructive to the fabric of the
relationship,” Rice said on PBS’s “Charlie Rose” show. ”We need to keep it
that way. The American people want it to be bipartisan.”
[$\texttt{BART}_{ft+\mathcal{L}_{\texttt{Prop}}}$]: Israel Prime Minister
Benjamin Netanyahu will address a joint session of Congress next week on his
visit to Washington in what will be his first speech in almost a decade. He
will address lawmakers in an address that will come two weeks before he is
scheduled to visit the country ahead of his expected election. The event,
which will take place at the U.S. Capitol on March 3, comes amid tensions
between President Obama and Republicans in Congress over Iran’s nuclear
program and tensions between the United States and Israel ahead of next
month’s elections.
Table 6: Generation examples from $\texttt{BART}_{ft}$ and
$\texttt{BART}_{ft+\mathcal{L}_{\texttt{Prop}}}$ for two different events.
According to our Neutral metric,
$\texttt{BART}_{ft+\mathcal{L}_{\texttt{Prop}}}$ is more neutral than
$\texttt{BART}_{ft}$. We highlight neutral and biased parts of the
generations.

##  6 Analysis

###  6.1 Decoding Strategy

In generation tasks, various decoding strategies have been proposed and
studied to find the optimal method of decoding for different tasks. For
instance, sampling-based decoding techniques are normally adopted to encourage
a more diverse response instead of being generic. Thus, we also investigated
the impact of decoding strategy on the generation neutrality on the best
performing baseline ($\texttt{BART}_{ft+\mathcal{L}_{\texttt{Prop}}}$).

We explored with two focuses: i) whether different decoding strategies affect
the neutralization performance, and ii) if yes, what is the best or the worst
decoding strategy for the task of neutralization. Four decoding techniques
were explored $-$ BEAM Search, Top-k Sampling and Nucleus Sampling
(Top-p)Holtzman et al. (2019).

The results are shown in Table 5. For ease of comparison between the decoding
techniques, we report the average Neutral scores as well. We can observe that
BEAM and Top-p have rather indifferent average Neutral scores, although each
has its strength - BEAM produces better Rouge-1 scores, whereas Top-p achieves
lower Bias scores ($17\%$). The most notable observation is that the
performance drops shown in Top-k decoding results – they experience marginal
drops in Rouge-1 ($10$) while retaining a similar level of bias. It would be
an interesting future work to understand the root cause of this phenomenon and
devise a decoding technique that can better avoid generating media-bias.

Figure 2: Illustrating the breakdown of media-bias (lexical vs informational).
Original Input refers to biased articles before neutralization. The rest two
represent generated articles from $\texttt{BART}_{ft}$ and
$\texttt{BART}_{ft+\mathcal{L}_{\texttt{Prop}}}$ models.

###  6.2 Lexical Bias vs. Information Bias

In Fig 2, we visualize the breakdown ratio of bias types (lexical bias,
informational bias, or both) that exist in different versions of generation.
Through this visualization, we aim to investigate if any one type of bias is
harder to eliminate than another.

To begin with, we illustrate the ratio from “original input” that refers to
the bias breakdown of the biased input articles ($X^{(l)}+X^{(r)}$). This
gives a complete picture of the original breakdown of bias before the
neutralization attempt. Then, we visualize the bias breakdown of the
generation from $\texttt{BART}_{ft}$ model and
$\texttt{BART}_{ft+\mathcal{L}_{\texttt{Prop}}}$ model.

The most important insight is that majority of the eliminated bias is
informational bias. For $\texttt{BART}_{ft+\mathcal{L}_{\texttt{Prop}}}$ , the
“lexical bias only” ratio is relatively unchanged compared to the big drop in
“info bias only”. Similarly, $\texttt{BART}_{ft}$’s “lexical bias only” ratio
stays unchanged, clearly indicating the difficulty involved in neutralizing
the lexical bias. This is likely because lexical bias normally exists as a
very short phrase that is hidden within an otherwise neutral sentence, thus,
requiring more sophisticated mitigation.

On the contrary, informational bias is shown to be relatively easy to
mitigate. We conjecture this is due to the informational bias often being
additional piece of information to sway readers mind. There is a high chance
for non-overlapping information between politically conflicting articles to be
the informational bias – this serves as a good indicator for the
neutralization models. For instance, as shown in Table 2, Left article
contains informational bias span “before President Donald Trump has a chance
to interfere,” negatively targeting Donald Trump, which does not appear in the
Right article.

###  6.3 Generation Examples

We compare the generation outputs from two models with different Neutral
scores to qualitatively check if the difference in scores aligns with the
generated articles. In this analysis, we selected two models
$\texttt{BART}_{ft+\mathcal{L}_{\texttt{Prop}}}$, relatively more neutral
model, and $\texttt{BART}_{ft}$, which is the less neutral model.

The generation examples777We provide more examples in the appendix. from Table
6 illustrates that $\texttt{BART}_{ft+\mathcal{L}_{\texttt{Prop}}}$ indeed
generates more neutralized version. In the first example about gun control, we
can clearly notice a difference in the language used when describing the same
information - $\texttt{BART}_{ft+\mathcal{L}_{\texttt{Prop}}}$ writes in a
neutral manner (“address” gun violence”), but $\texttt{BART}_{ft}$ writes in
more sensational style (“curb” gun violence). Going further, we can also
observe the difference in the neutrality from the nature of additional
information being generated by each model.
$\texttt{BART}_{ft+\mathcal{L}_{\texttt{Prop}}}$ provides substantial detail
directly related to the event being discussed (i.e., the reason behind the gun
control proposal), but $\texttt{BART}_{ft}$ reports on the negative opposition
that “is expected” to be faced by those who have “vowed to fight it in
Congress”.

For the second example related to Netanyahu, $\texttt{BART}_{ft}$ generates
more provoking contents (i.e., the polarized relationship between US and
Israel) and lexicons (i.e., “destructive”), whereas
$\texttt{BART}_{ft+\mathcal{L}_{\texttt{Prop}}}$ generation focuses on the
factual information related to the actual event (i.e., event will take place
at the U.S. Capitol on March 3).

##  7 Human Evaluation – Neutral metric

We conduct A/B testing of neutrality between two articles to verify the
alignment between our Neutral metric and human judgment. The A/B pair-
comparison method is chosen over the scale-rating evaluations method because
it is shown to be more reliable in the literature Kiritchenko and Mohammad
(2017). We carry out a human evaluation on a data annotation platform,
Appen.com. Each annotator is provided two different model generations (i.e.,
$\texttt{BART}_{ft}$ vs. $\texttt{BART}_{ft+\mathcal{L}_{\texttt{Prop}}}$) and
is asked to select a less biased version. Our goal is to see if the model with
a higher Neutral score (i.e.,
$\texttt{BART}_{ft+\mathcal{L}_{\texttt{Prop}}}$) is also perceived to be more
neutral by humans as well. We obtained three annotations each for 50 random
samples of the generations. $74\%$ voted for
$\texttt{BART}_{ft+\mathcal{L}_{\texttt{Prop}}}$ to be more neutral with an
average sample-wise agreement of $78.14\%$, suggesting that our Neutral metric
aligns with the human judgment.

In addition, we also asked the annotators to vote for the generation that has
higher information overlap with the given neutral article. $60\%$ of the
annotators voted for $\texttt{BART}_{ft+\mathcal{L}_{\texttt{Prop}}}$ to have
higher information coverage as well, with the average sample-wise agreement of
$75.52\%$.

To ensure the quality of annotation, we did the following. First, we only
allow annotators that pass the qualification test step to participate in the
A/B testing. Second, we ask for the political orientation to only incorporate
the answers from annotators with center political orientation. This is done to
avoid as much political bias of annotators as possible. For more detail, refer
to the Appendix.

##  8 Conclusion

In this work, we presented a novel generation-based media bias mitigation
task. By providing a new dataset, evaluation metric and baselines, this work
serves as a solid benchmark for the proposed task. Through a human evaluation,
we showed that the proposed metric aligns with the human judgement.
Furthermore, our experimental results empirically proved that adding
additional disinformation related loss is helpful in neutralization. Lastly,
our analysis showed that lexical bias is harder to neutralize than
informational bias and that decoding techniques could affect the neutrality of
the generations. We hope our work initiates more research on automatic media-
bias mitigation to make a positive impact on society.

## References

  * Bail et al. (2018) Christopher A Bail, Lisa P Argyle, Taylor W Brown, John P Bumpus, Haohan Chen, MB Fallin Hunzaker, Jaemin Lee, Marcus Mann, Friedolin Merhout, and Alexander Volfovsky. 2018.  Exposure to opposing views on social media can increase political polarization.  _Proceedings of the National Academy of Sciences_ , 115(37):9216–9221.
  * Baum and Groeling (2008) Matthew A Baum and Tim Groeling. 2008.  New media and the polarization of american political discourse.  _Political Communication_ , 25(4):345–365.
  * Cao et al. (2020) Meng Cao, Yue Dong, Jiapeng Wu, and Jackie Chi Kit Cheung. 2020.  Factual error correction for abstractive summarization models.  _arXiv preprint arXiv:2010.08712_.
  * Chiang and Knight (2011) Chun-Fang Chiang and Brian Knight. 2011.  Media bias and influence: Evidence from newspaper endorsements.  _The Review of economic studies_ , 78(3):795–820.
  * Da San Martino et al. (2019) Giovanni Da San Martino, Seunghak Yu, Alberto Barrón-Cedeno, Rostislav Petrov, and Preslav Nakov. 2019.  Fine-grained analysis of propaganda in news article.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5640–5650.
  * Dathathri et al. (2019) Sumanth Dathathri, Andrea Madotto, Janice Lan, Jane Hung, Eric Frank, Piero Molino, Jason Yosinski, and Rosanne Liu. 2019.  Plug and play language models: a simple approach to controlled text generation.  _arXiv preprint arXiv:1912.02164_.
  * De Vreese (2004) Claes De Vreese. 2004.  The effects of strategic news on political cynicism, issue evaluations, and policy support: A two-wave experiment.  _Mass Communication & Society_, 7(2):191–214.
  * Dong et al. (2020) Yue Dong, Shuohang Wang, Zhe Gan, Yu Cheng, Jackie Chi Kit Cheung, and Jingjing Liu. 2020.  Multi-fact correction in abstractive text summarization.  _arXiv preprint arXiv:2010.02443_.
  * Entman (1993) Robert M Entman. 1993.  Framing: Toward clarification of a fractured paradigm.
  * Entman (2007) Robert M Entman. 2007.  Framing bias: Media in the distribution of power.  _Journal of communication_ , 57(1):163–173.
  * Fan et al. (2019) Lisa Fan, Marshall White, Eva Sharma, Ruisi Su, Prafulla Kumar Choubey, Ruihong Huang, and Lu Wang. 2019.  In plain sight: Media bias through the lens of factual reporting.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 6344–6350.
  * Gabriel et al. (2020) Saadia Gabriel, Asli Celikyilmaz, Rahul Jha, Yejin Choi, and Jianfeng Gao. 2020\.  Go figure! a meta evaluation of factuality in summarization.  _arXiv preprint arXiv:2010.12834_.
  * Gentzkow and Shapiro (2006) Matthew Gentzkow and Jesse M Shapiro. 2006.  Media bias and reputation.  _Journal of political Economy_ , 114(2):280–316.
  * Gentzkow and Shapiro (2010) Matthew Gentzkow and Jesse M Shapiro. 2010.  What drives media slant? evidence from us daily newspapers.  _Econometrica_ , 78(1):35–71.
  * Groseclose and Milyo (2005) Tim Groseclose and Jeffrey Milyo. 2005.  A measure of media bias.  _The Quarterly Journal of Economics_ , 120(4):1191–1237.
  * Hamborg et al. (2019a) Felix Hamborg, Karsten Donnay, and Bela Gipp. 2019a.  Automated identification of media bias in news articles: an interdisciplinary literature review.  _International Journal on Digital Libraries_ , 20(4):391–415.
  * Hamborg et al. (2017) Felix Hamborg, Norman Meuschke, and Bela Gipp. 2017.  Matrix-based news aggregation: exploring different news perspectives.  In _2017 ACM/IEEE Joint Conference on Digital Libraries (JCDL)_ , pages 1–10. IEEE.
  * Hamborg et al. (2019b) Felix Hamborg, Anastasia Zhukova, and Bela Gipp. 2019b.  Illegal aliens or undocumented immigrants? towards the automated identification of bias by word choice and labeling.  In _International Conference on Information_ , pages 179–187. Springer.
  * Haselmayer and Jenny (2017) Martin Haselmayer and Marcelo Jenny. 2017.  Sentiment analysis of political communication: combining a dictionary approach with crowdcoding.  _Quality & quantity_, 51(6):2623–2646.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  _Advances in neural information processing systems_ , 28:1693–1701.
  * Holtzman et al. (2019) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019.  The curious case of neural text degeneration.  _arXiv preprint arXiv:1904.09751_.
  * Kiritchenko and Mohammad (2017) Svetlana Kiritchenko and Saif Mohammad. 2017.  Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 465–470, Vancouver, Canada. Association for Computational Linguistics.
  * Kryściński et al. (2019) Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher. 2019\.  Evaluating the factual consistency of abstractive text summarization.  _arXiv preprint arXiv:1910.12840_.
  * Laban and Hearst (2017) Philippe Laban and Marti A Hearst. 2017.  newslens: building and visualizing long-ranging news stories.  In _Proceedings of the Events and Stories in the News Workshop_ , pages 1–9.
  * Lee et al. (2019) Nayeon Lee, Zihan Liu, and Pascale Fung. 2019.  Team yeon-zi at semeval-2019 task 4: Hyperpartisan news detection by de-noising weakly-labeled data.  In _Proceedings of the 13th International Workshop on Semantic Evaluation_ , pages 1052–1056.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_.
  * Lin (2004) Chin-Yew Lin. 2004.  Rouge: A package for automatic evaluation of summaries.  In _Text summarization branches out_ , pages 74–81.
  * Lux et al. (2020) Klaus-Michael Lux, Maya Sappelli, and Martha Larson. 2020.  Truth or error? towards systematic analysis of factual errors in abstractive summaries.  In _Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems_ , pages 1–10.
  * Ma et al. (2020) Xinyao Ma, Maarten Sap, Hannah Rashkin, and Yejin Choi. 2020.  Powertransformer: Unsupervised controllable revision for biased language correction.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7426–7441.
  * McCombs and Reynolds (2009) Maxwell McCombs and Amy Reynolds. 2009.  How the news shapes our civic agenda.  In _Media effects_ , pages 17–32. Routledge.
  * Miller and Riechert (2001) M Mark Miller and Bonnie Parnell Riechert. 2001.  The spiral of opportunity and frame resonance: Mapping the issue cycle in news and public discourse.  _Framing public life: Perspectives on media and our understanding of the social world_ , pages 107–121.
  * Narayan et al. (2018) Shashi Narayan, Shay B Cohen, and Mirella Lapata. 2018.  Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 1797–1807.
  * Park et al. (2009) Souneil Park, Seungwoo Kang, Sangyoung Chung, and Junehwa Song. 2009.  Newscube: delivering multiple aspects of news to mitigate media bias.  In _Proceedings of the SIGCHI conference on human factors in computing systems_ , pages 443–452.
  * Park et al. (2011) Souneil Park, Kyung-Soon Lee, and Junehwa Song. 2011.  Contrasting opposing views of news articles on contentious issues.  In _Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies_ , pages 340–349.
  * Perse and Lambe (2016) Elizabeth M Perse and Jennifer Lambe. 2016.  _Media effects and society_.  Routledge.
  * Potthast et al. (2017) Martin Potthast, Johannes Kiesel, Kevin Reinartz, Janek Bevendorff, and Benno Stein. 2017.  A stylometric inquiry into hyperpartisan and fake news.  _arXiv preprint arXiv:1702.05638_.
  * Pryzant et al. (2020) Reid Pryzant, Richard Diehl Martinez, Nathan Dass, Sadao Kurohashi, Dan Jurafsky, and Diyi Yang. 2020.  Automatically neutralizing subjective bias in text.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 480–489.
  * Puglisi (2011) Riccardo Puglisi. 2011.  Being the new york times: the political behaviour of a newspaper.  _The BE journal of economic analysis & policy_, 11(1).
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _arXiv preprint arXiv:1910.10683_.
  * Recasens et al. (2013) Marta Recasens, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2013.  Linguistic models for analyzing and detecting biased language.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1650–1659.
  * Sides (2018) All Sides. 2018.  Media bias ratings.  _Allsides.com_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Yano et al. (2010) Tae Yano, Philip Resnik, and Noah A Smith. 2010.  Shedding (a thousand points of) light on biased language.  In _Proceedings of the NAACL HLT 2010 Workshop on Creating Speech and Language Data with Amazon’s Mechanical Turk_ , pages 152–158.
  * Zhang et al. (2019) Yifan Zhang, Giovanni Da San Martino, Alberto Barrón-Cedeno, Salvatore Romeo, Jisun An, Haewoon Kwak, Todor Staykovski, Israa Jaradat, Georgi Karadzhov, Ramy Baly, et al. 2019.  Tanbih: Get to know what you are reading.  _EMNLP-IJCNLP 2019_ , page 223.
  * Zhao et al. (2020) Zheng Zhao, Shay B Cohen, and Bonnie Webber. 2020.  Reducing quantity hallucinations in abstractive summarization.  _arXiv preprint arXiv:2009.13312_.
  * Zhou et al. (2020) Chunting Zhou, Jiatao Gu, Mona Diab, Paco Guzman, Luke Zettlemoyer, and Marjan Ghazvininejad. 2020.  Detecting hallucinated content in conditional neural sequence generation.  _arXiv preprint arXiv:2011.02593_.