# Connecting the Dots in News Analysis: A Cross-Disciplinary Survey of Media
Bias and Framing

Gisela Vallejo1 Timothy Baldwin1,2 Lea Frermann1
1The University of Melbourne 2MBZUAI
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

The manifestation and effect of bias in news reporting have been central
topics in the social sciences for decades, and have received increasing
attention in the NLP community recently. While NLP can help to scale up
analyses or contribute automatic procedures to investigate the impact of
biased news in society, we argue that methodologies that are currently
dominant fall short of addressing the complex questions and effects addressed
in theoretical media studies. In this survey paper, we review social science
approaches and draw a comparison with typical task formulations, methods, and
evaluation metrics used in the analysis of media bias in NLP. We discuss open
questions and suggest possible directions to close identified gaps between
theory and predictive models, and their evaluation. These include model
transparency, considering document-external information, and cross-document
reasoning rather than single-label assignment.

##  1 Introduction

The depiction of complex issues in the media strongly impacts public opinion,
politics, and policies (Ghanem, 1997; Giles and Shaw, 2009). Because a handful
of global corporations own an increasing proportion of news outlets, the reach
and impact of biased reporting are amplified Hamborg (2020). Although perfect
neutrality is neither realistic nor desirable, media bias turns into an issue
when it becomes systematic. If the public is unaware of the presence of bias,
this can lead to dangerous consequences, including intolerance and ideological
segregation (Baly et al., 2020).

For decades, news analysis has been an active field of research in the social
sciences, and more recently, computational methods for framing and political
bias classification have gained considerable momentum. The increasing pace of
news reporting suggests a need to scale the process of media bias detection,
and there is evidence that exposing media bias promotes healthy public debate,
helps journalists to increase thoroughness and objectivity, and promotes
critical and conscious news consumption (Dallmann et al., 2015). In the
context of this paper, we see the role of NLP as helping to understand,
characterise and expose bias at scale.

Figure 1: Two articles about the same event written from different political
ideologies. Example taken from AllSides.com.

Figure 1 illustrates the concepts of ‘framing’ and ‘media bias’ adopted in
this paper, using the passing of the Respect for Marriage Act as an example.
Framing refers to the deliberate presentation or emphasis of selected facts
with the goal of eliciting a desired interpretation or reaction in the reader
(Entman, 2007). The left-leaning article in Figure 1 leads with an uplifiting
picture of a wedding and emphasizes bill support, evoking a positive framing
by emphasizing new opportunities for same-sex couples; while the right-leaning
article focuses on concerns and debates in both image and text, framing the
issue in the a more negative light. Political bias refers to partisan slanted
news stories, or the “tendency to deviate from an accurate, neutral, balanced,
and impartial representation of ‘reality’ of events and social world” (McQuail
and Deuze, 2020), which can be a result of a selected framing. In Figure 1,
each document was flagged as far-left and far-right ideological leaning,
respectively, as a result of the different attitudes and selected points of
emphasis chosen in the reporting. Political bias is typically deliberate
(Williams, 1975) while framing may be inadvertent as a result of focusing on
selective information due to external pressures such as space limitations.

In this paper, we survey work on framing and media bias prediction in NLP and
relate it to typical research questions and hypotheses in the social sciences.
We tease out disconnects across disciplines, and make concrete suggestions on
how social science approaches can improve NLP methodology, and how NLP methods
can more effectively aid both social science scholars in their analyses as
well as underpin tools and applications to raise awareness of media bias among
the general public.

##  2 Background

###  2.1 Framing and Media Bias

We focus on the widely-studied phenomena of framing and political bias, as
they support the detection of partisan-biased documents, and both framing and
media bias are strategies to promote a particular view about a specific topic.

A variety of definitions of framing exists in social science and communication
studies. Prevalent definitions include equivalence framing: presenting the
same logical information in different forms (Cacciatore et al., 2016) and
emphasis framing: highlighting particular aspects of an issue to promote a
particular interpretation (Entman, 2007). Additionally, framing has been
conceptualised as a process (de Vreese, 2005; Entman, 2007; Chong and
Druckman, 2007), a communication tool (Scheufele, 1999), and/or a political
strategy (Roy and Goldwasser, 2020). In order to identify and classify frames
automatically, it can be helpful to understand the ‘generative process’ of
frames. Frames can be conceptualised into different typologies, e.g. de Vreese
(2005) proposes issue-specific: only pertinent to a single matter, and issue-
generic: identifiable across several issues. While Scheufele (1999)
differentiates between media frames: embedded in the political discourse, and
audience frames: the reader’s interpretation of an issue. And Gross (2008)
defines episodic framing as portraying an issue with an individual example
compared to thematic framing, which takes a more broader context to describe
the same issue. In this manuscript we cover both issue-specific and issue-
generic frames and attach to de Vreese (2005)’s definition of a frame as “an
emphasis in salience of different aspects of a topic”.

Political bias refers to an explicit association of a news article or media
outlet with a specific political leaning. Although framing and political bias
are different phenomena, NLP researchers have attempted to address them
jointly, either by investigating political framing (Roy and Goldwasser, 2020)
or by identifying correlations between framing and partisan slanted articles
(Ziems and Yang, 2021). NLP studies have attempted automatic media bias
identification under several names, including: hyper-partisan news detection
(Kiesel et al., 2019), media bias detection (Spinde et al., 2021b; Lei et al.,
2022), identification of biased terms (Spinde et al., 2021a), and political
ideology detection (Iyyer et al., 2014; Kulkarni et al., 2018). Their common
goal is to detect and classify the bias of a data sample towards a particular
political ideology. Many of these approaches naturally relate to investigate
_how the story is told_ (framing).

###  2.2 Why is this Survey Relevant?

Hamborg et al. (2019) present a thorough overview of traditional and
computational approaches to media bias, including detailed definitions of bias
types and their emergence in the context of news production. We complement the
survey by providing a more in-depth review of research methodologies in NLP,
more recent computational approaches, and a unified focus on the phenomenon of
framing and its manifestation as media bias. A very recent survey by Ali and
Hassan (2022) reviews computational approaches to modelling framing providing
a detailed systematic analysis of NLP methods.

For an exhaustive list of NLP previous work we refer the reader to Ali and
Hassan (2022). In contrast, our survey takes a closer look at the overall NLP
pipeline: data, methodology, and evaluation; draw connections to social
science methodology; and pinpoint the gaps between the two disciplines. In
order to obtain a comprehensive body of literature which bridge the
domains,111Here, we do not follow the standard approach of selecting the top
$N$ results from Google Scholar or the ACL Anthology for a simple query, as
work in this space is published under many different names in a myriad of
venues. we departed from influential cross-disciplinary papers: (1) a review
of media bias and framing across disciplines, but with no focus on state-of-
the-art NLP (Hamborg et al., 2019); and (2) one of the first and most
influential NLP framing data sets, with a strong theoretical grounding (Card
et al., 2015). We identified other relevant work by following both papers’
citation graphs (both backwards and forwards).

##  3 Three Disconnects

To illustrate the disconnects between the social sciences and NLP, we use the
case study of Hernández (2018)’s study of the framing of domestic violence, in
which the author formulates two research questions:

  1. 1.

Framing functions: Are femicides recognized as a problem of domestic violence?
What are the causes of femicides? And what are the solutions proposed?

  2. 2.

Frame narratives: What are the main narratives of the SCMP222South China
Morning Post? And what are the sources used to report them?

The first research question considers the local aspects within each news
article. Specifically, it looks at the causes and solutions presented,
grounded in Entman (1993)’s conceptualisation of framing in terms of a
problem, its cause, and its solution. The second research question relates
these local aspects to a global view by contrasting narratives that present
domestic violence as isolated incidents with those that treat it as a societal
problem. They connect the news reports to extrinsic variables like the sources
used or the cultural context of the story e.g. whether the article refers the
role of women in the Chinese family or understands domestic violence through
the lens of the Confucian philosophy. Their study considers full articles over
an extended period, capturing the temporal development of framing of the
issue. In contrast, current NLP approaches to frame prediction: (a) typically
take a single-class prediction approach —with a few exceptions Akyürek et al.
(2020); Mendelsohn et al. (2021) — per unit of analysis (sentence or article),
rather than treating frames as more complex structures which could for
instance distinguish aspects such as cause vs. solution; and (b) treat units
of analysis as independent without explicitly drawing connections across
articles, or across time, or to document-external context.

We thus highlight three important aspects of framing that we could identify
while reviewing social science literature. These aspects emerge in theoretical
media studies, but cannot be modeled through (single-label) classification,
and consequently not attainable by most current NLP approaches:

#### Framing is local and global

It is local, because because a single document can contain several frames, and
it is global because to understand the general framing of an article it is
often necessary to (a) aggregate local frames and (b) link them to document-
external information such as cited (or omitted) sources, or the outlets’
political leaning.

#### Framing is dynamic

Frames change over time, across outlets, or across countries or communities.
Understanding the development of framing can shed light on the impacts of a
sustained exposure to biased reporting on readers’ opinions, and enables the
study of trends.

#### Framing as a comparative task

Media bias and framing often become most apparent when directly contrasting
articles from different perspectives, places or times (cf., Figure 1). We
propose to address bias and frame classification as a comparative task rather
than labeling documents in isolation. This can help inducing frames from data
by analyzing axes of largest variation; and can naturally support tools and
applications to raise readers’ bias awareness by exposing them to contrasting
perspectives on the same issue.

The remainder of this article reviews current practice in NLP, points out
disconnects to the social science principles introduced above, and suggests
steps towards bridging the gap between the disciplines.

##  4 A Critical Review of Current Practices in NLP and Social Science

In this section we review both sides of the field, NLP and social sciences,
especially communication studies. We look at three main aspects, which we
consider to be the most relevant criteria when conducting research: datasets,
methods, and evaluation and metrics. Here, the reader can find the
similarities and differences across both disciplines.

###  4.1 Datasets

Dataset | Categories | Size | Granularity | Task
---|---|---|---|---
Bitterlemons (Lin et al., 2006) | Israel vs. Palestine | 594 | Documents | Classification
Flipper (Chen et al., 2018) | Left, Centre, Right | 6,447 | Documents | Classification
BASIL (Fan et al., 2019) |  | Libe., Cons., Centre;
---
Pos, Neu, Neg
| 1.2k / 448
---
300
| Spans/Words
---
Documents
Classification
AllSides (Baly et al., 2020) | Left, Centre, Right | 34k | Documents | Classification
BiasedSents (Lim et al., 2020) | not-, slightly-, very-, biased | 966 | Sentences | Classification
BABE (Spinde et al., 2021b) | Biased, Non-biased | 3.7k | Sentences | Classification
BIGNEWSALIGN (Liu et al., 2022) | Left, Centre, Right | 1M | Documents | Classification
NeuS (Lee et al., 2022) | Left, Centre, Right | 10.6k | Documents |  | Cross-Doc
---
Summarisation
MFC (Card et al., 2015) | 15 Frames |  | 61.5k/
---
11.9k
| Sentences/
---
Documents
Classification
GVFC (Liu et al., 2019) | 9 Frames | 2.99k | Headlines | Classification
Multimodal GVFC (Tourni et al., 2021) | 9 Frames | 1.3k |  | Headlines
---
$+$ Images
Classification
PVFC (Ziems and Yang, 2021) |  | Entity frames &
---
Cons., Libe., none
82k | Documents |  | Entity frame
---
prediction
Table 1: Benchmark contributions in political bias (top) and framing (bottom)
mostly in American English. Categories for BASIL denote liberal, conservative,
and centre for partisan labels, and polarity classes represent positive,
neutral and negative.

Benchmark datasets dominate modern-day NLP research, and news analysis is no
exception. In this section, we review NLP datasets relating to framing and
political bias analysis in the news domain. In Table 1, we list relevant
datasets, along with the type of labels they provide, the size of the
collection, the associated tasks, and sample granularity, whether words,
sentences or documents.

For the media bias detection task at the sentence level, Lim et al. (2020)
used crowdsourcing to annotate sentences on 46 English-language news articles
about 4 different events with four levels of bias (not-biased, slightly
biased, biased, or very biased). Spinde et al. (2021b) released BABE (“Bias
Annotations By Experts”), a collection of sentences labelled by experts
according to binary categories: biased and non-biased, at the sentence and
word levels. Fan et al. (2019) contributed the BASIL (“Bias Annotation Spans
on the Informational Level”) data set which includes word and sentence (span)
level annotations of political leaning, as well as sentiment (stance) towards
the entities in the article.

At the document level, the Bitterlemons corpus (Lin et al., 2006), comprises
weekly issues about the Palestine–Israel conflict. Each issue contains
articles from Palestinian and Israeli perspectives written by the portal’s
editors and guest authors. Despite being intended for document classification,
this dataset can be employed to explore framing and political bias, given the
documents’ nature of strong bias towards one side of the conflict.

Additionally, the web portal AllSides333https://www.allsides.com/about
categorises articles into three political ideologies: right, centre, and left
(they also offer a finer-grained five-point scale annotation: left, lean left,
centre, lean right, right) with the aim to provide all political perspectives
on a given story (cf., Figure 1). Experts manually assigned categories at the
article level. Several research groups have contributed datasets scraped from
AllSides Chen et al. (2018); Baly et al. (2020); Liu et al. (2022); Lee et al.
(2022).

In the field of framing at the sentence (headline) level, Liu et al. (2019)
released the Gun Violence Frame Corpus (GVFC). It includes headlines about gun
violence in news articles from 2016 and 2018 in the U.S., labelled with frames
like politics, economics, and mental health. Tourni et al. (2021) released a
multi-modal version of the GVFC collection, including the main image
associated with each article, and annotations about relevance and framing at
the image level.

At the document level, there is what is probably the most extensive data
collection for investigating framing: the Media Frames Corpus (MFC, Card et
al., 2015). It includes articles from $13$ U.S. newspapers on three policy
issues: immigration, same-sex marriage, and smoking. This dataset is intended
to enable the analysis of policy issue framing, providing annotations at
document and span levels with frames like morality, economic, and cultural.
Ziems and Yang (2021) contribute a police violence news articles collection
(PVFC) that can be categorised in both domains, media bias and framing. They
provide annotations for political leaning: conservative, liberal or none and
also entity-centric frames, including the victim’s age, race, and gender. They
also include the code to extract those entity frames automatically using
regular expressions. It is pertinent to note that this survey is primarily
U.S.- and English-centred, in large part because currently-available datasets
and work predominantly focus on U.S. news sources. Diversifying research to
other countries, cultures, and languages is an important step for future work.

In Section 3, we propose three main aspects to investigate framing and media
bias: (1) conducting studies at a local and global level; (2) considering the
dynamics of framing; and (3) addressing the problem as a comparative task. We
suggest that despite being intended for document classification, benchmarks
like AllSides, MFC, and Bitterlemons can be redeployed to explore framing and
political bias in a different fashion. Instead of assigning frames or
political ideologies to documents, they could be used to examine framing and
political bias by extracting the most common expressions for each frame or
ideology, and investigating commonalities, which can be helpful to social
scientists for local and global analyses. Indicators that have been explored
by Roy and Goldwasser (2020) are point-wise mutual information (Church and
Hanks, 1990) over bigrams and trigrams, but this approach does not generalise
well. The MFC contains sentence-level annotations for exploring local framing,
however to the best of our knowledge no study has attempted to aggregate those
labels to a global level. Regarding datasets providing sentence-level (BABE)
and headline (GVFC) annotation, this can be considered as a local dimension.
However, this generalises from the headline to the entire document, which
ignores the subtle signals in the local dimension.

With respect to aspect (2), dynamics occur on many levels, some of which are
captured by current data sets: the MFC, BASIL, GVFC and BABE provide article
timestamps, supporting diachronic modeling of bias and framing. While some
studies exist in this domain (Kwak et al., 2020; Card et al., 2022), the
majority of NLP framing considers articles in isolation. Other dynamics, e.g.,
across countries, communities or media types (e.g., news vs. blogs) are of
central interest in communication studies but less achievable with existing
data sets. Modelling those dynamics is under-explored.

For addressing aspect (3), we propose that researchers explore cross-document
differences from various outlets, and their particular angle on a specific
issue. Several of the datasets obtained from AllSides include alignment at the
event level and hence enable comparison across documents on the
left–centre–right spectrum at a finer granularity. A cross-ideology analysis
at the event level facilitates the detection of local differences among the
three ideologies and allows global aggregation at the document level.

###  4.2 Methods

Researchers in NLP have attempted to tackle media bias as political ideology
detection or framing categorisation using different task formulations. The
first and most common strategy is single-label classification, i.e. assigning
a single label to each data point. At the word level, Recasens et al. (2013)
learn linguistic features from word removal edit-logs in Wikipedia. Spinde et
al. (2021a) compared the Euclidean distance of word embeddings to identify
biased words in articles from Huffington Post (left wing) and Breitbart News
(right wing). And Liu et al. (2021) experimented with identifying and
replacing bias-inducing words with neutral ones using salience scores over
word embeddings.

At the sentence level, Iyyer et al. (2014) used RNNs to identify political
ideology in sentences in congressional debate transcripts and articles from
the Ideological Book corpus. Using the BASIL corpus, Hartmann et al. (2019)
correlated sentence and document distributions using a Gaussian mixture model
(Reynolds, 2009) to identify biased sentences; Chen et al. (2020) classified
biased spans by calculating their probability distributions on news articles;
and Guo and Zhu (2022) applied contrastive learning and created sentence
graphs to categorise biased sentences. Other researchers translated keywords
from GVFC into several languages, and fine-tuned mBERT to classify frames in
news headlines in languages other than English (Akyürek et al., 2020; Aksenov
et al., 2021).

At the document level, there has been substantial work on assigned frames to
documents in the MFC corpus. The task has been approached with RNNs (Naderi
and Hirst, 2017), attention and discourse information (Ji and Smith, 2017),
and pre-trained models (Khanehzar et al., 2019). Baly et al. (2020) combined
adversarial adaptation and adapted triple loss with features like Twitter and
Wikipedia information about the readers and the outlet to classify the
political ideology of news articles. Scholars have performed similar tasks on
languages other than English, e.g. by translating English keywords in MFC to
Russian to investigate the U.S. framing in Russian media over 13 years (Field
et al., 2018).

The second formulation is multi-label classification. Researchers have
primarily used topic modelling (Tsur et al., 2015; Menini et al., 2017) or
clustering (Ajjour et al., 2019) to determine the frames present in a
document, in an unsupervised setting. Soft membership for topics or clusters
allows documents to be assigned to various clusters or topics. Most of this
work has been done over political speeches rather than news articles. In a
supervised manner, Mendelsohn et al. (2021) employ RoBERTa to classify
multiple framing typologies on immigration-related tweets. Similarly, Akyürek
et al. (2020) address multi-label framing over headlines using different
configurations of BERT. However, all of this work has been done over headlines
or documents with a maximum length of 280 characters, and no work has been
done at the level of full news articles.

Other related task formulations include entity framing. At the document level,
Ziems and Yang (2021) use regular expressions to identify entity
characteristics (gender, race, age, etc.); and Frermann et al. (2023) explore
the co-occurrence of narrative roles (entity pictured as villain, hero, or
victim) with frames on manually-annotated climate change data. Finally, NLP
researchers have also investigated bias mitigation. At the headline level,
Chen et al. (2018) used LSTMs to flip the leaning of a headline, for example,
from a right-leaning title to a left-leaning one, in an attempt to alleviate
bias. However, flipping the ideology does not entail the reduction of bias. At
the document level, Lee et al. (2022) aggregate all perspectives in one
document using multi-document summarisation. We argue that including all
biases does not necessarily reduce the impact of ideology bias. Aggregating
the most relevant aspects and presenting them comparatively, as depicted in
Figure 1, is more effective and has greater utility for social scientists.

In the social sciences, approaches tend to be manual, with fewer data samples.
One common approach is to reason across many documents from a high-level
perspective. For example, Chyi and McCombs (2004) design and evaluate a two-
dimensional framework (spatial and temporal) to investigate framing changes
over time in 170 news articles in American English about a U.S. school
shooting event. They manually annotated articles with the signals indicating
both of the frame typologies, quantified those annotations and draw
conclusions about the temporal and spatial framing behaviour in the inspected
articles. Muschert and Carr (2006) assessed the previously-proposed framework
based on 290 news documents, and confirmed that the present temporal dimension
frame still holds when using data from more than one school shooting.
Hernández (2018) analysed the framing of 124 news stories from the South China
Morning Post (SCMP) about femicides by manually coding the articles and
quantifying those observations. The author explored whether those cases were
portrayed as isolated cases or part of a systematic social problem, by
manually analysing signals like narratives, sources, and the role of the
entities.

In addition, communication science studies often correlate features of news
reports with extra-textual information to formulate or validate their
hypotheses (see also Hamborg et al., 2019). For example, McCarthy et al.
(2008) investigate whether the media is ideologically biased in reporting
about demonstration events. They track media coverage of protests during the
transition period from communism in Belarus by considering features like the
size of the protest, sponsors’ status, and number of arrests, and examine
their correlation with the event’s media coverage. Similarly, Gentzkow and
Shapiro (2010) investigate media bias by calculating citations of different
media outlets by think tanks, and correlating those statistics with the number
of times that members of the U.S. Congress mentioned the same groups. Here, we
see a stark disconnect between largely local frame modelling in NLP but a
strong dominance on dynamic and global questions raised in communication
studies. Social science research provides a lens through which to consider NLP
methodology, and its insistence on considering each sample in isolation. We
argue that learning from signals like the use of metaphoric or technical
(legal) language, the correlation with informative features like sources
integrated in the report, and the role of the audience and journalist’s
cultural background in the story all contribute to news framing and bias
analysis.

###  4.3 Metrics and Evaluation

We consider two levels of validation: validating data annotations, and
validating model predictions.

The former — validating the quality of labelled data — applies to both the
social sciences and NLP. In a typical social science study, the distribution
of manual labels is the main factor for accepting or rejecting hypotheses or
drawing larger conclusions. As such, measures for data quality such as inter-
coder reliability (ICR) are routinely reported and a core requisite of the
study. This validation ensures that the codebook was correctly conceptualised,
and coding often includes discussions and several iterations on trial data or
pilot studies (Hernández, 2018), leading to relatively high ICR scores from
carefully trained annotators, often with domain knowledge.

Social science studies are largely analytical (examining labelled data,
qualitatively based on manual analysis, and quantitatively based on
statistical tests).

NLP studies on framing are empirical, and evaluation (regrettably) often comes
down to numeric comparison of a newly proposed method against previous work,
by comparing the predictions of systems against the ground truth frame labels.
This does not provide fundamental insights into how well a model can capture
framing or political bias at a higher and more abstract level, or whether it
is better able to lead to fresh insights into the data. In other words,
current approaches fall short of providing inferences from explicit
information, i.e. assessing the objectivity of a story as well as measuring
the level of factuality by identifying whether a story adopts a recounting or
metaphoric style. These strategies are graded in nature (rather than binary)
and metrics like accuracy are deficient.

In order to address the above-mentioned issues, we propose that automatic
framing and bias analysis evaluation tackle three main points: (1) model
performance, (2) error analysis, and (3) measuring model certainty. Even
though overall model performance in terms of accuracy or F1 does not provide a
complete picture of its utility (Spinde et al., 2021b), we still need to
consider point (1) to gauge the overall capabilities of a model. However, we
can go deeper and also investigate the performance at the outlet level or look
into the most challenging frames for the model to predict. This leads to point
(2), error analysis, following previous work (Vilar et al., 2006; Kummerfeld
and Klein, 2013), we propose three key components: (a) error categorisation.
(b) Scrutinising the potential causes of these errors. (c) Going beyond
identification, extending to suggesting feasible strategies for improvement
based on the nature and origins of the errors. Finally, we see the role of NLP
as developing meaningful tools and methods that can support social science
scholars to enhance and scale the investigation of framing and political bias.
Therefore, a user should be able to access model confidence scores to assess
the reliability of model predictions, as per point (3).

##  5 Discussion

Having reviewed approaches in the social sciences and NLP, and enumerated
disconnects, we ask: What are practices in the social sciences that NLP can
adopt? NLP task formulations tend to focus on assigning a single label (e.g.,
a frame) to a unit of analysis, typically a document or sentence. Social
science studies annotate news excerpts at the local dimension and combine that
information with external signals to arrive at higher-level conclusions.
Recalling our introductory example on the framing of domestic violence in the
SCMP, Hernández (2018) considers the broader impact, incorporating other
victims included in the news story (local signals); the role of culture in the
article: whether women are portrayed from their role in the Chinese family or
the story mentions Confucianism concepts; the type of report: brief or news
story; the sources, whether the article is based on a police report (external
factors). She combines these signals and aggregates them to the document-level
(global perspective) to draw higher-level conclusions on the dominant
narrative framing of domestic violence as isolated instances or a societal
phenomenon across the entire collection of articles.

With regard to NLP, we argue that the standard practice of assigning a single
frame label to news documents is overly simplistic, given that a typical news
story comprises viewpoints, arguments, or aspects, which may individually have
different connotations or framing. We acknowledge that causes of simplifying
annotation relate to factors that affect scalability and automation like the
costs and the difficulty of achieving inter-annotator agreement. In these
cases researchers are overcoming these challenges by means of few-shot pre-
training models (He et al., 2023; Bansal and Sharma, 2023). In the context of
political debates, Ajjour et al. (2019) suggest breaking down debates into
arguments and identifying a frame for each idea. Similar strategies in a media
framing context could mitigate the simplifying assumption of one frame per
article. Khanehzar et al. (2021) also argue that the single, primary frame
annotation in the MFC is oversimplified and propose a model for multi-view
representations of news stories. To address this gap, we suggest a two-step
process: (1) split a news document into self-contained discourse units (such
as arguments or events); and (2) assign a frame label to each unit, and/or one
or more global frame-label(s) by aggregating across units. As reviewed in
Sections 4.1 and 4.2, NLP methods operate mostly on the sentence level (which
cannot capture longer arguments) or the document level. Analyzing frames on an
argument- or event level reflects the typical level of analysis in the media
studies. Rather than assigning the single most likely frame, researchers might
want to take the full distribution over labels into account.

Although research questions are the starting point in both disciplines, these
are distinct. In the social sciences, tasks address more complex issues, e.g.
correlating the coverage of protests with characteristics like event size and
number of arrests during a political transition period (McCarthy et al.,
2008), compared with identifying factuality in an article (Baly et al., 2018),
detecting whether a sentence is biased or not (Lim et al., 2020; Lei et al.,
2022; Spinde et al., 2021b), or categorising full news documents based on
their framing about a specific issue (Naderi and Hirst, 2017; Ji and Smith,
2017; Khanehzar et al., 2019). Social scientists often consider external
knowledge to draw conclusions. In contrast, most NLP work operates on the
individual article level, disregarding external information as well as other
articles in the collection. A few exceptions exist, including Baly et al.
(2020) who incorporate readership demographics from Twitter and publisher
information from Wikipedia; and Kulkarni et al. (2018) who incorporate article
link structure into their models. Still, they consider each news item in
isolation. We encourage the NLP community to ground frame predictions in
external signals – be it document-extraneous (such as readership and sources)
and/or cross-document (by explicitly contrasting the framing across articles
from different times, locations, or outlets). We envision as a result more
expressive frame conceptualizations; outputs and analyses that are more
aligned with typical questions in the media sciences; and a stepping stone
towards tools that can highlight contrastive framing of issues in the news to
a general readership.

More broadly, we advocate for a more cross-disciplinary perspective in NLP
research, involving domain experts in all steps of the process: from the
formulation of research questions, to model design with consideration to
transparency and robustness, and evaluation. While prior work has highlighted
the importance of expert annotation (Spinde et al., 2021b), we argue that in
order to develop useful assistive tools for scholars or applications for the
general public, a dialogue with domain experts over the whole process is
essential. Cross-disciplinary projects would guide NLP researchers to go back
to the basics of framing analysis and political bias prediction as in the
social sciences, and adopt back best practices in steps like annotation.

What could NLP contribute to the social sciences? NLP can support and scale up
social science analyses with powerful tools like pre-trained and generative
models accompanied with domain expertise on how to employ these tools safely
and responsibly. For example, Bhatia et al. (2021) supply a web platform for
semi-automatic data annotation and document classifier training in order to
support communication-science researchers without the resources and skills in
using automatic tools. The system generates LDA topics (Blei et al., 2003) in
the first step and allows researchers to tag the topics and annotate
documents, which are used as training data for document-level frame
prediction.

NLP has a strong culture of sharing code and annotated data sets to encourage
collaboration and reproducibility. This practice is less common in the
humanities. Sharing this data more explicitly through cross-disciplinary
dialogue could provide critical assessment and feedback from domain experts
and encourage innovation on how to combine large (and potentially noisier)
data into the small-scale (but high-quality) annotations, to address
increasingly complex questions on the emergence and effects of media biases
and framing.

##  6 Conclusion

This survey takes a critical look at recent work in NLP on framing and media
bias, and points out disconnects and synergies in datasets, methodologies, and
validation techniques to research practices in the social sciences. Despite
the opportunities for NLP to support and scale social science scholarship on
media bias, a current oversimplification in conceptualisation, modelling, and
evaluation models of framing and media bias hinders fertile collaboration. We
have teased out three disconnects and proposed directions for future work,
including: (1) analysing news articles from a local and global perspective,
incorporating external non-textual features; (2) taking into account the
dynamics of framing and bias across documents, cultures or over time; and (3)
tackling the issue of media bias as a comparative task, defining frames on the
basis of systematic differences between articles whose origins differ on pre-
defined characteristics. This would allow for a more complex characterisation
of bias than the currently dominant approach of single-label classification.

## Limitations

This survey focuses on framing to news articles. This constrains the scope of
our analysis to media rather than framing in a broader context. Additionally,
we are aware that regardless on the approach taken for sampling the previous
work included in this manuscript, there will be always bias present. With the
aim of mitigating this bias, we point the reader to complement our work with
previous surveys in this field i.e. Hamborg et al. (2019) and Ali and Hassan
(2022).

## Ethics Statement

Identifying framing and ideology bias in news articles is highly influenced by
social and structural bias. Datasets and technologies intending to tackle
these phenomena comprise the social bias of annotators and researchers
developing them in an environment lacking diversity, in addition to the
potential for dual use of models and benchmarks to promote polarisation and
misinformation. However, we see this paper as an opportunity to identify new
directions to diversify NLP methodologies and develop new datasets that help
to push the field further, and address authentic analytical goals in the
social sciences.

## References

  * Ajjour et al. (2019) Yamen Ajjour, Milad Alshomary, Henning Wachsmuth, and Benno Stein. 2019.  Modeling frames in argumentation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2922–2932, Hong Kong, China. Association for Computational Linguistics.
  * Aksenov et al. (2021) Dmitrii Aksenov, Peter Bourgonje, Karolina Zaczynska, Malte Ostendorff, Julian Moreno-Schneider, and Georg Rehm. 2021.  Fine-grained classification of political bias in German news: A data set and initial experiments.  In _Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021)_ , pages 121–131, Online. Association for Computational Linguistics.
  * Akyürek et al. (2020) Afra Feyza Akyürek, Lei Guo, Randa Elanwar, Prakash Ishwar, Margrit Betke, and Derry Tanti Wijaya. 2020.  Multi-label and multilingual news framing analysis.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8614–8624, Online. Association for Computational Linguistics.
  * Ali and Hassan (2022) Mohammad Ali and Naeemul Hassan. 2022.  A survey of computational framing analysis approaches.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9335–9348, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Baly et al. (2020) Ramy Baly, Giovanni Da San Martino, James Glass, and Preslav Nakov. 2020.  We can detect your bias: Predicting the political ideology of news articles.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4982–4991, Online. Association for Computational Linguistics.
  * Baly et al. (2018) Ramy Baly, Georgi Karadzhov, Dimitar Alexandrov, James Glass, and Preslav Nakov. 2018.  Predicting factuality of reporting and bias of news media sources.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3528–3539, Brussels, Belgium. Association for Computational Linguistics.
  * Bansal and Sharma (2023) Parikshit Bansal and Amit Sharma. 2023.  Large language models as annotators: Enhancing generalization of NLP models at minimal cost.  _CoRR_ , abs/2306.15766.
  * Bhatia et al. (2021) Vibhu Bhatia, Vidya Prasad Akavoor, Sejin Paik, Lei Guo, Mona Jalal, Alyssa Smith, David Assefa Tofu, Edward Edberg Halim, Yimeng Sun, Margrit Betke, Prakash Ishwar, and Derry Tanti Wijaya. 2021.  OpenFraming: Open-sourced tool for computational framing analysis of multilingual data.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 242–250, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Blei et al. (2003) David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003.  Latent dirichlet allocation.  _J. Mach. Learn. Res._ , 3:993–1022.
  * Cacciatore et al. (2016) Michael A Cacciatore, Dietram A Scheufele, and Shanto Iyengar. 2016.  The end of framing as we know it… and the future of media effects.  _Mass communication and society_ , 19(1):7–23.
  * Card et al. (2015) Dallas Card, Amber E. Boydstun, Justin H. Gross, Philip Resnik, and Noah A. Smith. 2015.  The media frames corpus: Annotations of frames across issues.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 438–444, Beijing, China. Association for Computational Linguistics.
  * Card et al. (2022) Dallas Card, Serina Chang, Chris Becker, Julia Mendelsohn, Rob Voigt, Leah Boustan, Ran Abramitzky, and Dan Jurafsky. 2022.  Computational analysis of 140 years of US political speeches reveals more positive but increasingly polarized framing of immigration.  _Proceedings of the National Academy of Sciences_ , 119(31):e2120510119.
  * Chen et al. (2020) Wei-Fan Chen, Khalid Al Khatib, Benno Stein, and Henning Wachsmuth. 2020.  Detecting media bias in news articles using Gaussian bias distributions.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 4290–4300, Online. Association for Computational Linguistics.
  * Chen et al. (2018) Wei-Fan Chen, Henning Wachsmuth, Khalid Al-Khatib, and Benno Stein. 2018.  Learning to flip the bias of news headlines.  In _Proceedings of the 11th International Conference on Natural Language Generation_ , pages 79–88, Tilburg University, The Netherlands. Association for Computational Linguistics.
  * Chong and Druckman (2007) Dennis Chong and James N. Druckman. 2007.  Framing theory.  _Annual Review of Political Science_ , 10(1):103–126.
  * Church and Hanks (1990) Kenneth Ward Church and Patrick Hanks. 1990.  Word association norms, mutual information, and lexicography.  _Computational Linguistics_ , 16(1):22–29.
  * Chyi and McCombs (2004) Hsiang Iris Chyi and Maxwell McCombs. 2004.  Media salience and the process of framing: Coverage of the Columbine school shootings.  _Journalism & Mass Communication Quarterly_, 81(1):22–35.
  * Dallmann et al. (2015) Alexander Dallmann, Florian Lemmerich, Daniel Zoller, and Andreas Hotho. 2015.  Media bias in German online newspapers.  In _Proceedings of the 26th ACM Conference on Hypertext $\&$ Social Media_, HT ’15, page 133–137, New York, NY, USA. Association for Computing Machinery.
  * de Vreese (2005) Claes H. de Vreese. 2005.  News framing: Theory and typology.  _Information Design Journal_ , 13(1):51–62.
  * Entman (1993) Robert M. Entman. 1993.  Framing: Toward clarification of a fractured paradigm.  _Journal of Communication_ , 43(4).
  * Entman (2007) Robert M. Entman. 2007.  Framing bias: Media in the distribution of power.  _Journal of Communication_ , 57(1):163–173.
  * Fan et al. (2019) Lisa Fan, Marshall White, Eva Sharma, Ruisi Su, Prafulla Kumar Choubey, Ruihong Huang, and Lu Wang. 2019.  In plain sight: Media bias through the lens of factual reporting.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 6343–6349, Hong Kong, China. Association for Computational Linguistics.
  * Field et al. (2018) Anjalie Field, Doron Kliger, Shuly Wintner, Jennifer Pan, Dan Jurafsky, and Yulia Tsvetkov. 2018.  Framing and agenda-setting in Russian news: a computational analysis of intricate political strategies.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3570–3580, Brussels, Belgium. Association for Computational Linguistics.
  * Frermann et al. (2023) Lea Frermann, Jiatong Li, Shima Khanehzar, and Gosia Mikolajczak. 2023.  Conflicts, villains, resolutions: Towards models of narrative media framing.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 8712–8732, Toronto, Canada. Association for Computational Linguistics.
  * Gentzkow and Shapiro (2010) Matthew Gentzkow and Jesse M Shapiro. 2010.  What drives media slant? evidence from U.S. daily newspapers.  _Econometrica_ , 78(1):35–71.
  * Ghanem (1997) Salma Ghanem. 1997.  _Filling in the Tapestry: The Second Level of Agenda Setting_, 1 edition, pages 3–14. Routledge, New York.
  * Giles and Shaw (2009) David Giles and Rachel L Shaw. 2009.  The psychology of news influence and the development of media framing analysis.  _Social and personality psychology compass_ , 3(4):375–393.
  * Gross (2008) Kimberly Gross. 2008.  Framing persuasive appeals: Episodic and thematic framing, emotional response, and policy opinion.  _Political Psychology_ , 29(2):169–192.
  * Guo and Zhu (2022) Shijia Guo and Kenny Q. Zhu. 2022.  Modeling multi-level context for informational bias detection by contrastive learning and sentential graph network.  _CoRR_ , abs/2201.10376.
  * Hamborg (2020) Felix Hamborg. 2020.  Media bias, the social sciences, and NLP: Automating frame analyses to identify bias by word choice and labeling.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop_ , pages 79–87, Online. Association for Computational Linguistics.
  * Hamborg et al. (2019) Felix Hamborg, Karsten Donnay, and Bela Gipp. 2019.  Automated identification of media bias in news articles: An interdisciplinary literature review.  _International Journal on Digital Libraries (IJDL)_ , 20(4):391–415.
  * Hartmann et al. (2019) Mareike Hartmann, Tallulah Jansen, Isabelle Augenstein, and Anders Søgaard. 2019\.  Issue framing in online discussion fora.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1401–1407, Minneapolis, Minnesota. Association for Computational Linguistics.
  * He et al. (2023) Xingwei He, Zhenghao Lin, Yeyun Gong, A-Long Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2023.  Annollm: Making large language models to be better crowdsourced annotators.  _CoRR_ , abs/2303.16854.
  * Hernández (2018) Miriam Hernández. 2018.  "Killed out of love": A frame analysis of domestic violence coverage in Hong Kong.  _Violence Against Women_ , 24(12):1454–1473.
  * Iyyer et al. (2014) Mohit Iyyer, Peter Enns, Jordan Boyd-Graber, and Philip Resnik. 2014.  Political ideology detection using recursive neural networks.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1113–1122, Baltimore, Maryland. Association for Computational Linguistics.
  * Ji and Smith (2017) Yangfeng Ji and Noah A. Smith. 2017.  Neural discourse structure for text categorization.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 996–1005, Vancouver, Canada. Association for Computational Linguistics.
  * Khanehzar et al. (2021) Shima Khanehzar, Trevor Cohn, Gosia Mikolajczak, Andrew Turpin, and Lea Frermann. 2021.  Framing unpacked: A semi-supervised interpretable multi-view model of media frames.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2154–2166, Online. Association for Computational Linguistics.
  * Khanehzar et al. (2019) Shima Khanehzar, Andrew Turpin, and Gosia Mikolajczak. 2019.  Modeling political framing across policy issues and contexts.  In _Proceedings of the The 17th Annual Workshop of the Australasian Language Technology Association_ , pages 61–66, Sydney, Australia. Australasian Language Technology Association.
  * Kiesel et al. (2019) Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel Vincent, Payam Adineh, David Corney, Benno Stein, and Martin Potthast. 2019.  SemEval-2019 task 4: Hyperpartisan news detection.  In _Proceedings of the 13th International Workshop on Semantic Evaluation_ , pages 829–839, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
  * Kulkarni et al. (2018) Vivek Kulkarni, Junting Ye, Steve Skiena, and William Yang Wang. 2018.  Multi-view models for political ideology detection of news articles.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3518–3527, Brussels, Belgium. Association for Computational Linguistics.
  * Kummerfeld and Klein (2013) Jonathan K. Kummerfeld and Dan Klein. 2013.  Error-driven analysis of challenges in coreference resolution.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_ , pages 265–277, Seattle, Washington, USA. Association for Computational Linguistics.
  * Kwak et al. (2020) Haewoon Kwak, Jisun An, and Yong-Yeol Ahn. 2020.  A systematic media frame analysis of 1.5 million new york times articles from 2000 to 2017.  In _12th ACM Conference on Web Science_ , page 305–314, Southampton United Kingdom. ACM.
  * Lee et al. (2022) Nayeon Lee, Yejin Bang, Tiezheng Yu, Andrea Madotto, and Pascale Fung. 2022.  NeuS: Neutral multi-news summarization for mitigating framing bias.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3131–3148, Seattle, United States. Association for Computational Linguistics.
  * Lei et al. (2022) Yuanyuan Lei, Ruihong Huang Lu Wang, and Nick Beauchamp. 2022.  Sentence-level media bias analysis informed by discourse structures.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 10040–10050, Abu Dhabi. Association for Computational Linguistics.
  * Lim et al. (2020) Sora Lim, Adam Jatowt, Michael Färber, and Masatoshi Yoshikawa. 2020.  Annotating and analyzing biased sentences in news articles using crowdsourcing.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pages 1478–1484, Marseille, France. European Language Resources Association.
  * Lin et al. (2006) Wei-Hao Lin, Theresa Wilson, Janyce Wiebe, and Alexander Hauptmann. 2006.  Which side are you on? identifying perspectives at the document and sentence levels.  In _Proceedings of the Tenth Conference on Computational Natural Language Learning (CoNLL-X)_ , pages 109–116, New York City. Association for Computational Linguistics.
  * Liu et al. (2021) Ruibo Liu, Lili Wang, Chenyan Jia, and Soroush Vosoughi. 2021.  Political depolarization of news articles using attribute-aware word embeddings.  In _Proceedings of the Fifteenth International AAAI Conference on Web and Social Media, ICWSM 2021, held virtually, June 7-10, 2021_ , pages 385–396. AAAI Press.
  * Liu et al. (2019) Siyi Liu, Lei Guo, Kate Mays, Margrit Betke, and Derry Tanti Wijaya. 2019.  Detecting frames in news headlines and its application to analyzing news framing trends surrounding U.S. gun violence.  In _Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL)_ , pages 504–514, Hong Kong, China. Association for Computational Linguistics.
  * Liu et al. (2022) Yujian Liu, Xinliang Frederick Zhang, David Wegsman, Nicholas Beauchamp, and Lu Wang. 2022.  POLITICS: Pretraining with same-story article comparison for ideology prediction and stance detection.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 1354–1374, Seattle, United States. Association for Computational Linguistics.
  * McCarthy et al. (2008) John McCarthy, Larissa Titarenko, Clark McPhail, Patrick Rafail, and Boguslaw Augustyn. 2008.  Assessing stability in the patterns of selection bias in newspaper coverage of protest during the transition from communism in Belarus.  _Mobilization: An International Quarterly_ , 13(2):127–146.
  * McQuail and Deuze (2020) Denis McQuail and Mark Deuze. 2020.  _Mcquail’s media and mass communication theory_, 7th ed edition.  SAGE Publications, Thousand Oaks.
  * Mendelsohn et al. (2021) Julia Mendelsohn, Ceren Budak, and David Jurgens. 2021.  Modeling framing in immigration discourse on social media.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2219–2263, Online. Association for Computational Linguistics.
  * Menini et al. (2017) Stefano Menini, Federico Nanni, Simone Paolo Ponzetto, and Sara Tonelli. 2017.  Topic-based agreement and disagreement in US electoral manifestos.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2938–2944, Copenhagen, Denmark. Association for Computational Linguistics.
  * Muschert and Carr (2006) Glenn W. Muschert and Dawn Carr. 2006.  Media salience and frame changing across events: Coverage of nine school shootings, 1997–2001.  _Journalism & Mass Communication Quarterly_, 83(4):747–766.
  * Naderi and Hirst (2017) Nona Naderi and Graeme Hirst. 2017.  Classifying frames at the sentence level in news articles.  In _Proceedings of the International Conference Recent Advances in Natural Language Processing, RANLP 2017_ , pages 536–542, Varna, Bulgaria. INCOMA Ltd.
  * Recasens et al. (2013) Marta Recasens, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2013.  Linguistic models for analyzing and detecting biased language.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1650–1659, Sofia, Bulgaria. Association for Computational Linguistics.
  * Reynolds (2009) Douglas A. Reynolds. 2009.  Gaussian mixture models.  In Stan Z. Li and Anil K. Jain, editors, _Encyclopedia of Biometrics_ , pages 659–663. Springer US.
  * Roy and Goldwasser (2020) Shamik Roy and Dan Goldwasser. 2020.  Weakly supervised learning of nuanced frames for analyzing polarization in news media.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7698–7716, Online. Association for Computational Linguistics.
  * Scheufele (1999) Dietram A. Scheufele. 1999.  Framing as a theory of media effects.  _Journal of Communication_ , 49(1):103–122.
  * Spinde et al. (2021a) Timo Spinde, Felix Hamborg, Lada Rudnitckaia, and Bela Gipp. 2021a.  Identification of biased terms in news articles by comparison of outlet-specific word embeddings.  In _Proceedings of the iConference 2021_.
  * Spinde et al. (2021b) Timo Spinde, Manuel Plank, Jan-David Krieger, Terry Ruas, Bela Gipp, and Akiko Aizawa. 2021b.  Neural media bias detection using distant supervision with BABE - bias annotations by experts.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 1166–1177, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Tourni et al. (2021) Isidora Tourni, Lei Guo, Taufiq Husada Daryanto, Fabian Zhafransyah, Edward Edberg Halim, Mona Jalal, Boqi Chen, Sha Lai, Hengchang Hu, Margrit Betke, Prakash Ishwar, and Derry Tanti Wijaya. 2021.  Detecting frames in news headlines and lead images in U.S. gun violence coverage.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 4037–4050, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Tsur et al. (2015) Oren Tsur, Dan Calacci, and David Lazer. 2015.  A frame of mind: Using statistical models for detection of framing and agenda setting campaigns.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 1629–1638, Beijing, China. Association for Computational Linguistics.
  * Vilar et al. (2006) David Vilar, Jia Xu, Luis Fernando D’Haro, and Hermann Ney. 2006.  Error analysis of statistical machine translation output.  In _Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06)_ , Genoa, Italy. European Language Resources Association (ELRA).
  * Williams (1975) Alden Williams. 1975.  Unbiased study of television news bias.  _Journal of Communication_ , 25(4):190–199.
  * Ziems and Yang (2021) Caleb Ziems and Diyi Yang. 2021.  To protect and to serve? analyzing entity-centric framing of police violence.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 957–976, Punta Cana, Dominican Republic. Association for Computational Linguistics.