# Learned Distributed Image Compression with Multi-Scale Patch Matching in
Feature Domain

Yujun Huang, Bin Chen111Corresponding Author, Shiyu Qin, Jiawei Li, Yaowei
Wang, Tao Dai, Shu-Tao Xia

###### Abstract

Beyond achieving higher compression efficiency over classical image
compression codecs, deep image compression is expected to be improved with
additional side information, e.g., another image from a different perspective
of the same scene. To better utilize the side information under the
distributed compression scenario, the existing method (Ayzik and Avidan 2020)
only implements patch matching at the image domain to solve the parallax
problem caused by the difference in viewing points. However, the patch
matching at the image domain is not robust to the variance of scale, shape,
and illumination caused by the different viewing angles, and can not make full
use of the rich texture information of the side information image. To resolve
this issue, we propose Multi-Scale Feature Domain Patch Matching (MSFDPM) to
fully utilizes side information at the decoder of the distributed image
compression model. Specifically, MSFDPM consists of a side information feature
extractor, a multi-scale feature domain patch matching module, and a multi-
scale feature fusion network. Furthermore, we reuse inter-patch correlation
from the shallow layer to accelerate the patch matching of the deep layer.
Finally, we ﬁnd that our patch matching in a multi-scale feature domain
further improves compression rate by about 20% compared with the patch
matching method at image domain (Ayzik and Avidan 2020).

## Introduction

SI

SI with weakened

brightness

  (a) 0.0384 bpp 0.9058 MS-SSIM (a’) 0.0384 bpp 0.8975 MS-SSIM (b) 0.0360 bpp
0.8829 MS-SSIM (b’) 0.0360 bpp 0.8709 MS-SSIM

Figure 1: Visual comparisons of decoded main images with different compression
methods when the brightness of the side information (SI) image is normal or
weakened. (a) and (b) are examples of path matching in the multi-scale feature
domain (proposed) and patch matching in the image domain with normal
brightness, respectively. (a’) and (b’) are the corresponding examples when
the brightness of SI image is $60\%$ of the original image. The visual
perception of (a) is similar to (a’). However, (b’) loses some color and
texture details compared to (b).

Distributed image compression is less studied than the commonly known single
image compression, as the inherent correlation among images taken from
different viewpoints is hard to capture. With the development of the stereo
camera, camera array, self-driving system, and multiple UAV/monitor camera
systems during the past decade, high-volume multi-view images become the
primary data source in large-scale digital communication. For example, it has
been reported that self-driving cars can capture 1G of data per second
(Mearian 2013). Since multi-view images are usually captured from different
angles of the same scene at the same time, there exist overlapping fields with
high correlation among these images. In order to improve the compression
efficiency for such distributed compression scenario, the inherent correlation
must be utilized to remove redundancy in data. This inspires the development
of distributed compression with side information, where the sensors encode
their data, independently, and the decoders recover the data with the help of
side information from another source only available at the decoder. The
Distributed Source Coding (DSC) theorem (Slepian and Wolf 1973; Cover 1975;
Wyner and Ziv 1976) reveals that the same compression rate can be
asymptotically achieved by using side information (SI) only at the decoding
end as when available at both encoder and decoder. However, there is still a
performance gap between these two coding schemes in practice.

Recently, with the success of deep learning, deep image compression (Ballé,
Laparra, and Simoncelli 2017; Ballé et al. 2018; Mentzer et al. 2018) has made
great improvement over classical codecs like JPEG. In a deep image compression
model, a variational autoencoder is adopted as a basic architecture for end-
to-end optimized encoding and decoding. For a given input image, the multiple
nonlinear layers extract its latent feature representation. And the entropy of
the quantized feature layer is estimated by an entropy model. Then the
quantized feature is input into a symmetric decoder to reconstruct the image.
By optimizing the rate-distortion (R-D) loss over the large-scale training
set, deep image compression achieves state-of-the-art compression ratios.

In recent years, some learning-based distributed image coding methods have
been proposed (Diao, Ding, and Tarokh 2020; Ayzik and Avidan 2020; Mital et
al. 2022). Ayzik et al. (Ayzik and Avidan 2020) proposed to align the side
information image with the main image by patch matching in the image domain.
However, operating only in the image domain can not make full use of the
abundant texture details in side information, and is not robust to the
variants of illumination and scale caused by different viewing points. As
shown in Fig. 1, the gain of the side information in the image domain achieves
a slight improvement when the brightness of the side image is weakened. By
contrast, our proposed multi-scale feature domain patch matching method still
makes good use of the correlative texture information within the side
information image. This significantly demonstrates the robustness and
efficiency of the multi-scale feature domain patch matching for side
information utilization.

We introduce a Multi-Scale Feature Domain Patch Matching (MSFDPM) method to
better utilize the side information image at the decoder of the deep image
compression model. Similarity matching in the multi-scale feature domain can
fully explore the correlative texture information in the side information
image and is robust to variants of illumination and scale. At the same time,
the complementary scale features can effectively recover the decoded main
image from fine-grained texture to coarse-grained structures simultaneously.
Specifically, MSFDPM consists of three modules: (1) A feature extractor
network to extract multi-scale side information features from side information
image; (2) Patch matching module to obtain aligned side information features,
in which the inter-patch correlation of the largest scale feature layer is
reused to later feature layers to accelerate the decoding process; (3) Feature
fusion network to concatenate the decoded multi-scale features from the main
image with that of the side information image to obtain high-quality
reconstruction.

To summarize, we make the following contributions.

  * •

We propose a new paradigm of multi-scale feature domain patch matching method
for deep distributed image compression.

  * •

We validate that the patch matching in the multi-scale feature domain is more
robust to variants of illumination and scale than patch matching in the image
domain.

  * •

We provide comprehensive experiments to show that our method outperforms the
state-of-the-art distributed image compression methods.

## Related Work

#### Deep Image Compression

Deep image compression has attracted more and more attention in recent years.
Thanks to the powerful representation capability of autoencoder, flexible
entropy model and end-to-end optimization architecture, deep image compression
methods have outperformed reigning image compression standards. Ballé et al.
first proposed an end-to-end autoencoder structure based on rate-distortion
optimization for image compression (Ballé, Laparra, and Simoncelli 2017).
Ballé et al. further proposed a hyperprior model to capture the spatial
correlation of latent feature map and transfer the captured information to the
decoder as side information (Ballé et al. 2018). Minnen et al. proposed an
autoregressive prior and a hierarchical prior to obtain the correlation
between neighboring components in latent feature map (Minnen, Ballé, and
Toderici 2018). Cheng et al. proposed to use the discretized mixture Gaussian
model to exactly model the distribution of latent feature map and introduced
the attention modules into the autoencoder (Cheng et al. 2020). Xie et al.
proposed to use Invertible Encoding Network instead of autoencoder structure
to mitigate information loss (Xie, Cheng, and Chen 2021). Kim et al. proposed
a novel entropy model to focus on long-range correlations in hidden layer
features through an attention mechanism (Kim, Heo, and Lee 2022).

Figure 2: The proposed MSFDPM architecture for side information based decoding
of deep image compression

#### Learned Distributed Source Coding

Recently, some papers use deep learning to implement distributed source
coding. Ding et al. proposed to use recurrent autoencoder for distributed
image coding, but without using side information images when decoding main
image (Diao, Ding, and Tarokh 2020). It only uses decoder model to learn the
correlation. Ayzik et al. proposed that the side information image could be
aligned with the main image by patch matching method when there is a large
overlapping field between them, and the aligned side information image could
be used to obtain better main image reconstruction (Ayzik and Avidan 2020).
However, the information available in image domain is limited and not robust
enough. Mital et al. proposed to extract the common information from the side
information image through a network and input the common information and the
latent feature map of main image into a decoder network to obtain an enhanced
reconstruction (Mital et al. 2022). However, the proposed method does not
align the side information image with main image, which will lead to the
degraded reconstruction quality when the parallax is large.

## The Proposed Method

The main idea of our proposed MSFDPM decoder is to transmit both high-level
semantic information and low-level texture information to the main image based
on multi-scale feature domain patch matching, so that useful information can
be screened out for reconstruction. Specifically, our method first uses a
feature extractor to obtain multi-scale side information features, aligns the
features with the main features by patch matching, and then fuses the aligned
side information features with main features to get a better reconstruction.
The proposed method is shown in Fig. 2.

### Multi-Scale Features of Main Image and Side Information Image

Before patch matching, we need the model to obtain multi-scale features of the
main image and side information image. First, the main image X is fed into the
encoder of the single-image compression model to obtain latent feature map
$\hat{\textbf{{Z}}}$. Because single-image decoding is an upsampling process,
the decoder could take the latent feature map $\hat{\textbf{{Z}}}$ and output
multi-scale decoded main features
$\left\\{\left.\textbf{{F}}_{\hat{\textbf{{X}}}}^{\textit{h}}\in\mathbb{R}^{\frac{H}{2^{h}}\times\frac{W}{2^{h}}\times
C}\right|h=1,2,3,4\right\\}$ and the first stage decoded main image
$\hat{\textbf{{X}}}_{1}\in\mathbb{R}^{H\times W\times C}$, where $H$ and $W$
are the height and width of the main image. We use smaller superscripts to
represent larger scale features, which facilitates the following description
of patch matching. With the same single-image autoencoder, the side
information image Y can be used to produce multi-scale decoded side
information features
$\left\\{\left.\textbf{{F}}_{\hat{\textbf{{Y}}}}^{\textit{h}}\in\mathbb{R}^{\frac{H}{2^{h}}\times\frac{W}{2^{h}}\times
C}\right|h=1,2,3,4\right\\}$. Because the decoded main features and the
decoded side information features are produced by the same autoencoder, they
have a similar property, which will improve matching accuracy. However, the
decoded side information features lose part of the side information because of
the quantization noise, lossless side information features are needed to
improve the main image reconstruction. A trainable feature extractor is
proposed to take the side information image and produce the multi-scale
lossless side information features
$\left\\{\left.\textbf{{F}}_{\textbf{{Y}}}^{\textit{h}}\in\mathbb{R}^{\frac{H}{2^{h}}\times\frac{W}{2^{h}}\times
C}\right|h=1,2,3,4\right\\}$. Our feature extractor uses the encoder structure
in (Cheng et al. 2020).

### Multi-Scale Feature Domain Patch Matching

Because of the disparity problem caused by different viewing points, the side
information image needs to be aligned with the main image to effectively use
the side information. In addition, compared to the image domain, which would
be more affected by variance of illumination and color caused by different
viewing points, the feature domain will lay more emphasis on the texture and
structure to be migrated. Moreover, because it is time-consuming to calculate
the inter-patch correlation on all feature layers, we choose to calculate only
this correlation on the first feature layer, which can be reused by other
layers. For the convenience of description, we define the set of small patches
sampled by a $B\times B$ window with stride $s$ on a $H\times W\times C$
feature map $T$ as:

$\displaystyle S^{T,B,s}=\\{p_{i,j}^{T,B,s}\mid i=0,\cdots,I,j=0,\cdots,J\\}$
(1)
$\displaystyle\text{where}\,\,I\triangleq\left\lfloor\frac{W-B}{s}\right\rfloor,J\triangleq\left\lfloor\frac{H-B}{s}\right\rfloor$

According to this definition, we can obtain the patch set of the first decoded
main feature layer $S^{\textbf{{F}}_{\hat{\textbf{{X}}}}^{1},B,B}$, the first
decoded side information feature layer
$S^{\textbf{{F}}_{\hat{\textbf{{Y}}}}^{1},B,1}$, and multi-scale lossless side
information
$\left\\{\left.S^{\textbf{{F}}_{\textbf{{Y}}}^{h},\frac{B}{2^{h-1}},1}\right|h=1,2,3,4\right\\}$.
It is worth noting that the patches of the decoded main feature are not
overlapping and cover all the regions of the feature map, while the patch
sampling of the side information feature is fine-grained because we need to
match the most similar patch on the side information feature map. We use the
pearson correlation coefficient to measure the inter-patch correlation between
patches of the first decoded main feature layer and the first decoded side
information feature layer:

$r_{(i,j),(k,l)}={\rm
Pr}(p_{i,j}^{\textbf{{F}}_{\hat{\textbf{{X}}}}^{1},B,B},p_{k,l}^{\textbf{{F}}_{\hat{\textbf{{Y}}}}^{1},B,1})*m_{(i,j),(k,l)},$
(2)

where $m_{(i,j),(k,l)}$ is the Gaussian mask proposed in (Ayzik and Avidan
2020), which is the prior to choose adjacent patches with higher probability.
The computationally expensive inner product operation in the pearson
correlation can be efficiently calculated by a series of convolutions, where
the convolution kernel is each patch of the decoded main feature, with the
decoded side information feature as input:

$p_{i,j}^{\textbf{{F}}_{\hat{\textbf{{X}}}}^{1},B,B}*\textbf{{F}}_{\hat{\textbf{{Y}}}}^{1},$
(3)

where $*$ denotes the convolution operation. Based on the pearson correlation
coefficient, we can find the location of the side information feature patch
that are most similar to each patch in the main feature:

$(k_{(i,j)}^{*},l_{(i,j)}^{*})=\mathop{\arg\max}\limits_{(k,l)}r_{(i,j),(k,l)}.$
(4)

Then we can get the first aligned side information feature layer
$\textbf{{F}}_{\textbf{{Y}};a}^{1}$ by putting the most similar lossless side
information feature patch into the corresponding position in the main feature:

$p_{i,j}^{\textbf{{F}}_{\textbf{{Y}};a}^{1},B,B}=p_{k_{(i,j)}^{*},l_{(i,j)}^{*}}^{\textbf{{F}}_{\textbf{{Y}}}^{1},B,1}.$
(5)

#### Reusing First Feature Layer Inter-Patch Correlation

We reduce the computational complexity and GPU memory footprint of patch
matching by reusing the above-mentioned first feature layer inter-patch
correlation. First, we correspond the patches of the second to the fourth
feature layer to the patches of the same position in the first feature layer.
A toy example is shown in Appendix A. We find that for the decoded main
features the position subscripts of patches corresponding to the same position
in different feature layers are the same. For the decoded side information
features, the position subscript of the patch in the previous layer is twice
that of the corresponding patch in the later layer. So we can use the
correlation between the corresponding patches of the first feature layer to
represent the inter-patch correlation of the following layer:

$\begin{split}R(p_{i,j}^{\textbf{{F}}_{\hat{\textbf{{X}}}}^{h},\frac{B}{2^{h-1}},\frac{B}{2^{h-1}}},p_{k,l}^{\textbf{{F}}_{\hat{\textbf{{Y}}}}^{h},\frac{B}{2^{h-1}},1})&=r_{(i,j),(2^{h-1}k,2^{h-1}l)},\\\
&\qquad\qquad h=2,3,4.\end{split}$ (6)

Then we can get the second to the fourth aligned side information feature
layers
$\left\\{\textbf{{F}}_{\textbf{{Y}};a}^{2},\textbf{{F}}_{\textbf{{Y}};a}^{3},\textbf{{F}}_{\textbf{{Y}};a}^{4}\right\\}$
by using the method similar to Eq. 5.

Figure 3: One iteration of feature fusion network.

### Feature Fusion Network

Feature fusion network is designed to iteratively fuse aligned side
information features and decoded main features from small scale to large
scale, as illustrated in the red box in Fig. 2. The effectiveness of feature
fusion at multiple scales is demonstrated in ablation experiments.
Specifically, in one iteration of the feature fusion network, the aligned side
information feature and the decoded main feature at scale $h$, as well as the
output feature of the previous iteration (if any) are concatenated and input
into two residual blocks to obtain the fused feature at scale $h$:

$\begin{split}\bm{\phi}^{4}&={\rm Res}^{4}_{2}\left({\rm
Res}^{4}_{1}\left(\textbf{{F}}_{\hat{\textbf{{X}}}}^{4},\textbf{{F}}_{\textbf{{Y}};a}^{4}\right)\right),\\\
\bm{\phi}^{h}&={\rm Res}^{h}_{2}\left({\rm
Res}^{h}_{1}\left(\textbf{{F}}_{\hat{\textbf{{X}}}}^{h},\textbf{{F}}_{\textbf{{Y}};a}^{h},\bm{\phi}^{h+1}\right)+\bm{\phi}^{h+1}\right),\\\
&\ \ \ \ \ \ \ \ \qquad\qquad\qquad\qquad\qquad h=1,2,3.\end{split}$ (7)

The architectures of the residual blocks refer to the decoder of single-image
compression model (Cheng et al. 2020). Fig. 3 illustrates in detail one
iteration of the feature fusion network. Finally, the second stage decoded
main image is obtained by adding the latest output of the feature fusion
network to the first stage decoded main image:

$\hat{\textbf{{X}}}_{2}=\bm{\phi}^{1}+\hat{\textbf{{X}}}_{1}$ (8)

### Loss Function

Lossy compression is a joint optimization problem of compression rate and
distortion. For the proposed model, the loss function consists of the entropy
of the latent feature map $\hat{\textbf{{Z}}}$, the distortion of the first
stage decoded main image and the second stage decoded main image:

$\mathcal{L}=H(\hat{\textbf{{Z}}})+\lambda((1-\alpha)d(\textbf{{X}},\hat{\textbf{{X}}}_{1})+\alpha
d(\textbf{{X}},\hat{\textbf{{X}}}_{2}))),$ (9)

where $\lambda$ is the weight that controls trade-off between the compression
rate and the distortions. $\alpha$ is the weight that controls the trade-off
between the two distortions.

Figure 4: The rate-distortion performance comparison of different methods.

## Experiments and Results

### Experimental Setup

#### Datasets

We conduct experiments on two datasets: _KITTI Stereo_ and _KITTI General_
proposed in (Ayzik and Avidan 2020). _KITTI Stereo_ contains 1578 training
pairs and 790 test pairs, which are paired stereo images from the KITTI Stereo
2012 (Geiger, Lenz, and Urtasun 2012) and KITTI Stereo 2015(Menze and Geiger
2015) dataset. _KITTI General_ has 174936 training pairs and 3609 test pairs.
The pairs of data are different not only in viewing points but also in time
steps.

#### Evaluation Metrics

We use bits per pixel (bpp) to measure the compression ratio. Both peak
signal-to-noise ratio (PSNR) and multi-scale structural similarity (MS-SSIM)
(Wang, Simoncelli, and Bovik 2003) are used to measure image quality/signal
distortion. Moreover, we use BD-Rate, negative numbers to indicate the
percentage of average bit savings at the same image quality across the rate-
distortion curve compared with some chosen baselines. Specifically, BD-
Rate${}_{\text{P}}$ and BD-Rate${}_{\text{M}}$ indicate the performance gain
under the same PSNR and MS-SSIM, respectively.

#### Baselines

We compare the MSFDPM with 4 baselines. Two classical image compression
baselines: JPEG 2000(Rabbani 2002) and BPG (Bellard 2014). And two learned
distributed image compression baselines: DSIN (Ayzik and Avidan 2020) and
NDIC-UCI (Mital et al. 2022). DSIN and NDIC-UCI are briefly introduced in
related work.

#### Implementation Details

The proposed MSFDPM is implemented with PyTorch (Paszke et al. 2019) and the
experiments are conducted on four Intel(R) Xeon(R) E5-2698 v4 CPUs and eight
NVIDIA Tesla V100 GPUs. We first train a single-image compression baseline
(Cheng et al. 2020). Then we train the full model with cropped $320\times 960$
image pair, where the parameters of the autoencoder are initialized with the
pretrained baseline. The number of epochs on _KITTI Stereo_ is 10 and the
number of epochs on _KITTI General_ is 1. We used a batch size of 1 and the
Adam optimizer (Kingma and Ba 2015) with $1\cdot 10^{-4}$ learning rate. Other
hyper-parameters are listed as follows: (i) The number of features, $C=128$.
(ii) The patch size, $B=16$. (iii) The weight for rate-distortion trade-off,
$\lambda\in\left\\{0.005,0.01,0.02,0.035,0.05,0.07,0.1\right\\}$. (iv) The
weight for two stages of distortions, $\alpha$ is equal to $0$ when training
the autoencoder baseline and $1$ when training the full model.

0.0391 bpp/0.9292 MS-SSIM

0.0367 bpp/0.8908 MS-SSIM

  0.0365 bpp/0.9298 MS-SSIM MSFDPM 0.0331 bpp/0.9053 MS-SSIM
MSFDPM${}_{\text{img}}$ 0.1094 bpp/0.9541 MS-SSIM 0.1047 bpp/0.9433 MS-SSIM
0.1362 bpp/0.9019 MS-SSIM 0.1303 bpp/0.8900 MS-SSIM

Figure 5: Visual comparisons of MSFDPM and MSFDPM${}_{\text{img}}$.

### Results and Analysis

#### Results

We report the rate-distortion results in Fig. 4, which shows that our MSFDPM
outperforms all comparison baselines across different compression ratios. The
increase in compression is significant. For example, in the KITTI Stereo
dataset, when the MS-SSIM is $0.90$, the bpp of MSFDPM is about $0.025$, that
of NDIC-UCI is about $0.048$, and that of DSIN is about $0.07$, that of BPG is
about $0.098$, and that of JPEG 2000 is about $0.134$. Compared with NDIC-UCI,
DSIN, BPG, and JPEG 2000, MSFDPM has a compression rate increase of $1.92$
times, $2.80$ times and $3.92$ times, $5.36$ times, respectively.

Table 1: The BD-Rate results of different MSFDPM variants. Baseline is single-
image compression model (Cheng et al. 2020).

Variant | _KITTI Stereo_ | _KITTI General_
---|---|---
BD-Rate${}_{\text{P}}$ | BD-Rate${}_{\text{M}}$ | BD-Rate${}_{\text{P}}$ | BD-Rate${}_{\text{M}}$
MSFDPM | -49.26% | -51.11% | -40.37% | -40.89%
MSFDPM${}_{\text{img}}$ | -27.81% | -24.54% | -18.72% | -13.53%
MSFDPM${}_{\text{f1}}$ | -41.29% | -42.02% | -31.15% | -29.92%
MSFDPM${}_{\text{f2}}$ | -45.99% | -49.13% | -38.77% | -38.50%
MSFDPM${}_{\text{f3}}$ | -25.43% | -33.25% | -32.72% | -35.12%
MSFDPM${}_{\text{f4}}$ | -24.80% | -33.22% | -24.41% | -27.89%
MSFDPM${}_{\text{w/o r}}$ | -47.01% | -49.89% | -40.06% | -40.41%

#### Ablation Experiments

To explore the effect of multi-scale feature patch matching and inter-patch
correlation reusing, we design 3 variants of MSFDPM: (i)
MSFDPM${}_{\text{img}}$ replaces multi-scale feature domain patch matching
with image domain patch matching. The aligned side information image and the
first stage decoded main image are input into a network containing four
residual blocks to obtain fine reconstruction. (ii) MSFDPM${}_{\text{f$h$}}$,
$h=1,2,3,4$, only fuses the $h$-th aligned side information feature layer in
the feature fusion network. (iii) MSFDPM${}_{\text{w/o r}}$ performs block
matching at each feature layer without reusing the inter-patch correlation of
the first feature layer.

  * •

Multi-Scale Feature Domain Patch Matching v.s. Image Domain Patch Matching.
MSFDPM outperforms MSFDPM${}_{\text{img}}$ by BD-Rate${}_{\text{P}}$ of
$-21.45\%$ and BD-Rate${}_{\text{M}}$ of $-26.57\%$ on _KITTI Stereo_ and BD-
Rate${}_{\text{P}}$ of $-31.65\%$ and BD-Rate${}_{\text{M}}$ of $-27.36\%$ on
_KITTI General_. This implies that the multi-scale feature domain can extract
more texture or structure information to help obtain higher quality
reconstruction compared to the image domain. Fig. 5 presents some visual
comparisons of MSFDPM and MSFDPM${}_{\text{img}}$. Benefiting from the multi-
scale feature patch matching, MSFDPM can provide finer texture structure and
richer color details than MSFDPM${}_{\text{img}}$.

  * •

Multi-Scale Feature Fusion v.s. Single-Scale Feature Fusion. MSFDPM
outperforms MSFDPM${}_{\text{f1}}$, MSFDPM${}_{\text{f2}}$,
MSFDPM${}_{\text{f3}}$ and MSFDPM${}_{\text{f4}}$ on both BD-
Rate${}_{\text{P}}$ and BD-Rate${}_{\text{M}}$. This indicates that multi-
scale feature fusion can improve the quality of reconstruction by using
complementary fine-grained texture information to coarse-grained structure
information. Specifically, MSFDPM outperforms MSFDPM${}_{\text{f1}}$,
MSFDPM${}_{\text{f2}}$, MSFDPM${}_{\text{f3}}$, MSFDPM${}_{\text{f4}}$ by an
average BD-Rate${}_{\text{P}}$ of $-8.59\%$, $-2.43\%$, $-15.74\%$, $-20.21\%$
and an average BD-Rate${}_{\text{M}}$ of $-10.03\%$, $-2.18\%$, $-11.82\%$,
$-15.45\%$. The BD-rate performance of MSFDPM${}_{\text{f2}}$ is the best
among all the single-scale feature fusion models. This may be because the
second feature layer has a better trade-off between high-frequency detailed
texture and low-frequency structural information. The performance of
MSFDPM${}_{\text{f3}}$ and MSFDPM${}_{\text{f4}}$ is worse than that of
MSFDPM${}_{\text{f1}}$ and MSFDPM${}_{\text{f2}}$, which indicates that
texture information is more useful to improve image quality than structure
information.

Figure 6: The performance reduction of MS-SSIM of MSFDPM and MSFDPMimg as side
information image brightness and scale change ($\lambda=0.035$).

SI with scale 1

SI with scale 0.75

SI

SI

  MSFDPM 0.0368/0.9498 MSFDPM 0.0368/0.9113/83.33% MSFDPM${}_{\text{img}}$
0.0356/0.9196 MSFDPM${}_{\text{img}}$ 0.0356/0.8995/125.62%

Figure 7: Visual comparisons of MSFDPM and MSFDPM${}_{\text{img}}$ with scale
1 and 0.75. Evaluation Metric is denoted by bpp/MS-SSIM/PR.

#### Robustness Experiments

In order to explore the robustness of the multi-scale feature domain and image
domain to illumination and scale change, we adjuest the brightness and scale
of the side information image during inference. We set the brightness and
scale of the original side information image are 1, and proportionally change
the brightness and scale of the original image. To evaluate the performance
reduction (PR), we define it as

$\text{PR}=1-\frac{\text{performance improvement after
change}}{\text{performance improvement before change}}$ (10)

As shown in Fig. 6, when the brightness is $0.8$, the numerical MS-SSIM
improvement of the model using side information is $0.015$ compared with the
model without using side information. When the brightness is $1$, the
numerical MS-SSIM improvement of the model using side information is $0.02$.
Then the performance reduction at $0.8$ brightness is
$1-\frac{0.015}{0.02}=25\%$.

MSFDPM outperforms MSFDPM${}_{\text{img}}$ by an average of performance
reduction of $-46.39\%$ and $-18.45\%$ with brightness change and scale
change, respectively. Besides, it can be seen that the performance reduction
of MSFDPM is consistently lower than that of MSFDPM${}_{\text{img}}$ across
different brightness and scales. Therefore, we can achieve the conclusion that
the multi-scale feature domain is more robust to the variants of illumination
and scale than the image domain. Besides, the performance reduction of the
multi-scale feature domain is much lower during brightness change. For
example, when the brightness is 0.75, the performance reduction of
MSFDPM${}_{\text{img}}$ is $86.44\%$ and the performance reduction of MSFDPM
is only $24.09\%$. This demonstrates that the feature domain can mitigate the
brightness variation and emphasize texture and structure.

Fig. 7 shows the visualization results when the scale of the side information
image changes. It can be seen that when the scale is $0.75$, the example of
MSFDPM${}_{\text{img}}$ shows a clustering block effect while the example of
MSFDPM does not. This may be because the change of scale causes the
corresponding patches cannot be accurately matched in the image domain.
Appendix B shows more visualization results.

Table 2: Decoding speed, generation speed of Gaussian mask, and GPU memory
usage of MSFDPM and MSFDPM${}_{\text{w/o r}}$.

Variant | Decoding | Guassian Masks | GPU Memory
---|---|---|---
MSFDPM | 0.5404s | 22.86s | 9908Mb
MSFDPM${}_{\text{w/o r}}$ | 0.5740s | 25.30s | 10791Mb

#### Efficiency Analysis

To verify the efficiency of reusing the first feature layer inter-patch
correlation, we compare the decoding speed, Gaussian mask generation speed,
and GPU memory usage of MSFDPM and MSFDPM${}_{\text{w/o r}}$, and the results
are shown in Table 2. The decoding speed and Gaussian mask generation speed of
MSFDPM are $6\%$ and $11\%$ faster than MSFDPM${}_{\text{w/o r}}$. And MSFDPM
has $8\%$ less GPU memory usage than MSFDPM${}_{\text{w/o r}}$. Besides, as
shown in Table 1, the BD-rate of MSFDPM is slightly lower than that of
MSFDPM${}_{\text{w/o r}}$. This may be because the later decoded feature
layers are closer to the quantization and are more likely inaccurately match
patches due to the quantization noise.

## Conclusions

In this paper, we propose multi-scale feature domain patch matching (MSFDPM)
for distributed image compression. Compared with the previous patch matching
in the image domain, MSFDPM is more robust to the variants of illumination and
scale caused by different viewing points, and the feature domain contains
richer texture information in the side information image. In addition, we
reuse the first feature layer inter-patch correlation to improve the decoding
efficiency. On the experimental side, we demonstrate the superiority of MSFDPM
over state-of-the-art baseline, while being more robust to variants of
illumination and scale.

## References

  * Ayzik and Avidan (2020) Ayzik, S.; and Avidan, S. 2020.  Deep image compression using decoder side information.  In _European Conference on Computer Vision_ , 699–714. Springer.
  * Ballé, Laparra, and Simoncelli (2017) Ballé, J.; Laparra, V.; and Simoncelli, E. 2017.  End-to-end optimized image compression.  5th International Conference on Learning Representations, ICLR 2017 ; Conference date: 24-04-2017 Through 26-04-2017.
  * Ballé et al. (2018) Ballé, J.; Minnen, D.; Singh, S.; Hwang, S. J.; and Johnston, N. 2018.  Variational Image Compression with a Scale Hyperprior.  In _6th Int. Conf. on Learning Representations (ICLR)_.
  * Bellard (2014) Bellard, F. 2014.  BPG Image format.  https://bellard.org/bpg/.
  * Cheng et al. (2020) Cheng, Z.; Sun, H.; Takeuchi, M.; and Katto, J. 2020.  Learned Image Compression With Discretized Gaussian Mixture Likelihoods and Attention Modules.  In _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 7936–7945.
  * Cover (1975) Cover, T. 1975.  A proof of the data compression theorem of Slepian and Wolf for ergodic sources (Corresp.).  _IEEE Transactions on Information Theory_ , 21(2): 226–228.
  * Diao, Ding, and Tarokh (2020) Diao, E.; Ding, J.; and Tarokh, V. 2020.  Drasic: Distributed recurrent autoencoder for scalable image compression.  In _2020 Data Compression Conference (DCC)_ , 3–12. IEEE.
  * Geiger, Lenz, and Urtasun (2012) Geiger, A.; Lenz, P.; and Urtasun, R. 2012.  Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Kim, Heo, and Lee (2022) Kim, J.-H.; Heo, B.; and Lee, J.-S. 2022.  Joint Global and Local Hierarchical Priors for Learned Image Compression.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , 5992–6001.
  * Kingma and Ba (2015) Kingma, D. P.; and Ba, J. 2015.  Adam: A Method for Stochastic Optimization.  In _ICLR (Poster)_.
  * Mearian (2013) Mearian, L. 2013.  Self-driving cars could create 1GB of data a second.  _Computerworld_ , 23.
  * Mentzer et al. (2018) Mentzer, F.; Agustsson, E.; Tschannen, M.; Timofte, R.; and Van Gool, L. 2018.  Conditional probability models for deep image compression.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 4394–4402.
  * Menze and Geiger (2015) Menze, M.; and Geiger, A. 2015.  Object Scene Flow for Autonomous Vehicles.  In _Conference on Computer Vision and Pattern Recognition (CVPR)_.
  * Minnen, Ballé, and Toderici (2018) Minnen, D.; Ballé, J.; and Toderici, G. D. 2018.  Joint autoregressive and hierarchical priors for learned image compression.  _Advances in neural information processing systems_ , 31.
  * Mital et al. (2022) Mital, N.; Özyılkan, E.; Garjani, A.; and Gündüz, D. 2022.  Neural distributed image compression using common information.  In _2022 Data Compression Conference (DCC)_ , 182–191. IEEE.
  * Paszke et al. (2019) Paszke, A.; Gross, S.; Massa, F.; Lerer, A.; Bradbury, J.; Chanan, G.; Killeen, T.; Lin, Z.; Gimelshein, N.; Antiga, L.; et al. 2019.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  _Advances in Neural Information Processing Systems_ , 32: 8026–8037.
  * Rabbani (2002) Rabbani, M. 2002.  JPEG2000: Image Compression Fundamentals, Standards and Practice.  _Journal of Electronic Imaging_ , 11(2): 286.
  * Slepian and Wolf (1973) Slepian, D.; and Wolf, J. 1973.  Noiseless coding of correlated information sources.  _IEEE Transactions on information Theory_ , 19(4): 471–480.
  * Wang, Simoncelli, and Bovik (2003) Wang, Z.; Simoncelli, E. P.; and Bovik, A. C. 2003.  Multiscale structural similarity for image quality assessment.  In _The Thrity-Seventh Asilomar Conference on Signals, Systems & Computers, 2003_, volume 2, 1398–1402. Ieee.
  * Wyner and Ziv (1976) Wyner, A.; and Ziv, J. 1976.  The rate-distortion function for source coding with side information at the decoder.  _IEEE Transactions on information Theory_ , 22(1): 1–10.
  * Xie, Cheng, and Chen (2021) Xie, Y.; Cheng, K. L.; and Chen, Q. 2021.  Enhanced invertible encoding for learned image compression.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , 162–170.


# Towards Corpus-Scale Discovery of Selection Biases in News Coverage:
Comparing What Sources Say About Entities as a Start

Sihao Chen William Bruno Dan Roth
University of Pennsylvania
{sihaoc, wwbruno<EMAIL_ADDRESS>
###### Abstract

News sources undergo the process of selecting newsworthy information when
covering a certain topic. The process inevitably exhibits _selection biases_ ,
i.e. news sources’ typical patterns of choosing what information to include in
news coverage, due to their agenda differences. To understand the magnitude
and implications of selection biases, one must first discover (1) on what
topics do sources typically have diverging definitions of "newsworthy"
information, and (2) do the content selection patterns correlate with certain
attributes of the news sources, e.g. ideological leaning, etc.

The goal of the paper is to investigate and discuss the challenges of building
scalable NLP systems for discovering patterns of media selection biases
directly from news content in massive-scale news corpora, without relying on
labeled data. To facilitate research in this domain, we propose and study a
conceptual framework, where we compare how sources typically mention certain
controversial entities, and use such as indicators for the sources’ content
selection preferences. We empirically show the capabilities of the framework
through a case study on NELA-2020, a corpus of $1.8$M news articles in English
from 519 news sources worldwide. We demonstrate an unsupervised representation
learning method to capture the selection preferences for how sources typically
mention controversial entities. Our experiments show that that distributional
divergence of such representations, when studied collectively across entities
and news sources, serve as good indicators for an individual source’s
ideological leaning. We hope our findings will provide insights for future
research on media selection biases.

Figure 1: Overview of our conceptual framework: We represent a sources’
content selection preferences by comparing the content of news coverage on
each fine-grained topic against other sources. We found that a source’s
selection preferences on different topics collectively serve as good
indicators for its ideological leaning ($\S$ 5.2). In our study, we choose to
represent the topic of a text segment by the entities it mentions.

##  1 Introduction

News media represents one of the most prominent information channels to our
eventful world. Although we are blessed with a wide variety of news sources,
each source naturally has to go through the process of sampling and condensing
information to include in news coverage. The editorial decisions of what is
newsworthy or not directly influence the perceived representation of the
reported topic or event by the audience D’Alessio and Allen (2000); Groeling
(2013). Such problem is generally referred to as media selection bias111As
_selection bias_ is a much overloaded term, we will define the precise scope
of our study in $\S$ 2.. Understanding the patterns of media selection biases
is crucial, yet intrinsically difficult. As the editorial decisions of various
sources have different motivations, and their content selection preferences
vary drastically by topic, it would require a holistic, massive scale of
content analysis before we can uncover the magnitude and form of selection
biases that each source exhibits.

In this paper, we discuss and investigate the challenges of developing
scalable computational methods for discovering patterns of selection biases
from news content directly. Our study revolves around two research questions.
First, how do we represent a sources’ selection preferences by each topic of
the news content they produce? And subsequently, can we try to explain such
selection preferences by quantifying their correlation with a set of the
source’s motivating attributes, such as ideological leaning?

We start off by discussing the challenges and limitations of existing methods
from the NLP perspective ($\S$ 2). Surrounding the two research questions, we
propose and sketch a conceptual framework which uses _named entities_ to
represent the _topic_ of a text segment. Figure 1 illustrates our idea with a
motivating example. From news articles published by different sources within
similar time frames, we extract and categorize text segments by which two222We
use two instead one, as we observe two entities in a sentence would provide
focused and much finer-grained topic guidance, as shown in Figure 1. of the
common named entities that they mention. When looking at such text segments
collectively and comparatively across news sources, the aspect or context in
which the two entities get mentioned would implicitly reflect the two sources’
similar or different content preferences on the topic. We formulate the task
of recognizing whether two text segments present similar context as an
unsupervised representation learning objective, as we will describe in $\S$
3.2. We observe that such entity-centric formulations provide a flexible way
of aligning different sources’ statements made on similar topics, and thus the
similarity between those statements would conceptually reflect the similarity
between the two sources’ content selection preferences with respect to the
topic.

We assess the utility of the idea by conducting a case study on the NELA-2020
corpus Horne et al. (2018); Gruppi et al. (2021), which consists of
$\tilde{~}{}1.8$M news articles in 2020 from 519 news sources worldwide. We
first identify a set of most frequent entities through entity linking and
corpus statistics as our topics of interest for the study ($\S$ 3.1). With
political bias ratings of the sources from media watchdog _AllSides_
333https://www.allsides.com/media-bias/media-bias-ratings, we discover
empirical evidence that the collective set of content selection preferences
w.r.t. topics guided by entities show positive correlation with the
ideological polarization among the sources. Interestingly, we also observe
that our methodology can be used to identify common grounds among sources,
where sources typically agree with each other regardless of their ideological
leaning. Following such observation, we describe a statistical way to define
and discover _archetypical entity pairs_ , for which sources with conflicting
ideological preferences display polarized content selection preferences. We
empirically show that archetypical entity pairs provide a straightforward way
of recognizing the ideological leaning of a source.

In summary, our contributions are three-fold:

  1. 1.

We investigate the challenges of discovering patterns of media selection bias
from massive-scale news corpora. We discuss and connect the research questions
with the goal of building scalable NLP systems for such.

  2. 2.

We propose a conceptual framework for the challenge, which uses _named
entities_ to represent the topic of a text segment. We propose an unsupervised
representation learning objective to capture the similar/different content
selection preferences from the set of text segments by different news sources.

  3. 3.

We conduct a case study on NELA-2020, a large-scale multi-national news
corpus. We empirically validate the utility of our apporach, and discover that
the derived representations of sources’ selection preferences can be used as
indicators for the ideological polarity among sources.

##  2 Background and Challenges

During news production, selection biases typically happen on the macro-level
of editorial decisions, and thus it is widely believed that selection biases
serve as good indicators for the implicit intent and motivating theme behind
news sources. However, recognizing selection biases from news content alone is
intrinsically difficult, as conceptually it requires knowing the _population_
of all the stories within a topic that news sources can cover, which D’Alessio
and Allen (2000) describe as “unknowable and unidentifiable”. For the exact
reason, past studies of media selection biases in communication research limit
their scope on a focused event, such as political protests, where the complete
set of newsworthy aspects can be reasonably estimated McCarthy et al. (1996);
Hocke (1996). However, the general applicability of such findings is fairly
limited, since sources do not typically conform to the same type of content
selection decisions for different topics, as we also observe in this study.

From a pragmatic point of view, there exist two ways to discover the existence
of selection bias from a source with respect to another. The first and most
obvious way is through observations such as: _Source A mentioned this fact or
event, while Source B did not._ Other than the difficulty in assessing the
truth value of such statements, such observations might often be attributed to
less significant factors such as the mismatching size or publication frequency
of the target news sources D’Alessio and Allen (2000). The second way, which
we adopt in this study, is to identify cases where _Source A and Source B
discuss the same topics within a relatively small time frame, but they choose
to mention them in very different context or aspects._ The right hand side of
Figure 1 shows an example of such. Although such observation on its own does
not reveal which side potentially exhibits selection bias, it represents the
diverging interests of the sources on the matter. Along this line, our
empirical results with such analysis across many sources reveal that the
interests can be partially explained by the diverging ideological preferences
of various sources.

#### Challenge: Representing Topic

The first challenge of this space lies within the notion of _topic_. The type
of analysis we target in this study requires a fine-grained yet generally
applicable definition of topic. The goal aligns well, at least conceptually,
with the broad scope of computational frame analysis Boydstun et al. (2013).
_Framing_ generally refers to a set of nuanced discourse strategies to
selectively report and highlight certain aspects or sub-topics over others
when discussing a topic Entman (1993). In practice, frame analysis usually
starts from a pre-defined typology of sub-topic frames with respect to a set
of controversial topics. Despite the recent efforts in extending the
typologies and resources to increase coverage on topics and domains Roy and
Goldwasser (2020), the general applicability of such analysis remains limited.

Our study aims to address the limitation of using a fixed frame typology by
considering _entities_ as an alternative to sub-topic frame. This is in-part
motivated by the fact that entities themselves usually provide strong topic
guidance Gabrilovich et al. (2007). This enables us to align statements made
by different sources on the same entity pairs, and study them comparatively.
We will refer to such statements as _context sentences_ for a pair of entities
throughout the rest of the paper.

#### Challenge: Recognizing Selection Preferences

The next piece of the puzzle is how we would represent the different content
selection preferences of sources, given that set of statements they made
containing the target pairs of entities.

Formally, given two entities of interest $e_{1}$ and $e_{2}$, we want to learn
a representation $g(C^{s}_{e_{1},e_{2}}|e_{1},e_{2})$ from a list of context
sentences $C^{s}_{e_{1},e_{2}}$ sampled from news articles by source $s\in S$.
Again, context sentences $C^{s}_{e_{1},e_{2}}$ are defined as all sentences
that contain mentions of both $e_{1}$ and $e_{2}$ from source $s$.
Collectively, $C^{s}_{e_{1},e_{2}}$ represent all the statements that source
$s$ made about $e_{1}$ and $e_{2}$ within the sample time frame.

Given two sources $s_{A}$ and $s_{B}$, the conceptual goal is to identify
whether $C^{s_{A}}_{e_{1},e_{2}}$ are topically different from
$C^{s_{B}}_{e_{1},e_{2}}$. This objective in part resemble the family of
conditional language modeling tasks Keskar et al. (2019), where the
distributional divergence of the two sets of statement can be used as distance
metric for their topical relatedness. In our case, we train a language model
with the learning objective of recognizing whether two context sentences (with
entity mentions masked) contain the same set of entities or not, with training
data sampled from Wikipedia, We observe that language models learned through
such objective is effective in capturing the high-level topical selection
differences reflected in the context sentences.

##  3 Model Overview

###  3.1 Extracting Entities As Topics

As a first step, we use entity linking and corpus statistics to identify and
disambiguate a set of frequently mentioned entities within the NELA corpus as
the topics of interest throughout our study. Due to the sheer scale of the
corpus, we use the small version of NER model in spaCy 444en_core_web_sm to
identify named entity mentions. We opt for a simple yet effective method for
entity linking Ratinov et al. (2011). In Wikipedia, we compute the fraction of
times a page of title $t$ is linked from another page with an anchor
text/mention $m$. At inference time, given a mention $m$, we take the title
with highest conditional probability, i.e. $\hat{t}=argmax_{t\in{T}}(P(t|m))$.

We then compute the frequency of the disambiguated entities in all NELA corpus
and take the $1,000$ most frequent as the topics of interest for the study.

Figure 2: PCA visualization of the source representations with respect to four
representative pairs of entities. We consider the subset of 65 sources in NELA
that are annotated by _AllSides_ with their political bias rating. Each dot in
the plots represent a source, and is colored with respect to their politcal
bias rating. $D(\mathcal{N}_{L}\|\mathcal{N}_{R})$ measures the _left_ vs
_right_ distributional divergence of the source embeddings. More details can
be found in § 5.1

###  3.2 Recognizing Topical Difference between Context Sentences

We now describe the process of learning the topical representation of context
sentence $c\in C^{s}_{e_{1},e_{2}}$. Conceptually, as data sparsity can be a
concern with entity pairs, we want the representation to capture the topical
context of $e_{1}$ and $e_{2}$ described in $c$, while being agnostic to the
topical information exposed by the surface form of $e_{1}$ and $e_{2}$
themselves.

Inspired by recent advances in neural knowledge base construction Verga et al.
(2021); Sun et al. (2021), we build an encoder model from pretrained
RoBERTabase model and parameters Liu et al. (2019). For each input context
sentence $c\in C^{s}_{e_{1},e_{2}}$, we use a special token ‘[ENT]’ to mask
the mention of $e_{1}$ and $e_{2}$. In addition, we add ‘[R1]’ and ‘[R2]’
after the two entity masks respectively. The token embeddings of ‘[R1]’ and
‘[R2]’ would jointly represent the context in which the two entities get
mentioned. The last hidden layer $h_{R1}$ and $h_{R2}$ of the two tokens are
then forwarded through a linear layer to get the embedding $r_{e_{1},e_{2}}$.

$r(c)=W^{T}[h_{R1},h_{R2}]$

To obtain training data for the language model, we sample all sentences with
two or more mentions of the $1,000$ entities of interest from Wikipedia. This
gives us about 3M sentences in total. We adopt triplet margin loss as our
training objective. Each training instance consists of three context sentences
– For each anchor sentence $c\in C^{s}_{e_{1},e_{2}}$, we randomly sample a
positive sentence $c^{+}\in C^{s}_{e_{1},e_{2}}$ and a negative sentence
$c^{-}\notin C^{s}_{e_{1},e_{2}}$ which mentions at least one other entity
other than $\\{e_{1},e_{2}\\}$. The objective is to maximize the relative
similarity between $r(c)$ and $r(c+)$, and we use L-2 norm as the distance
function.

$\displaystyle\mathcal{L}=max[d(c,c^{+})-d(c,c^{-})+1,0]$ $\displaystyle
d(c_{1},c_{2})=\|r(c_{1})-r(c_{2})\|_{2}$

Our model is trained on a single Nvidia TITAN RTX in 5 hours. We report the
hyperparameter settings for the RoBERTa model in Appendix A.

Given the set of news articles published by each source
$s\in\\{s_{1},...,s_{n}\\}$ in NELA, we identify all context sentences
$C^{s}_{e_{i},e_{j}}$ for each entity pair $(e_{i},e_{j})$. We run each
context sentence $c\in C^{s}_{e_{i},e_{j}}$ through the context sentence
encoder and mean-pool to get the result embedding. The embedding conceptually
represents the collective content selection choices made by the source with
respect to the target topic entities.

$g(C^{s}_{e_{i},e_{j}})=\frac{\sum_{c}r(c)}{|C^{s}_{e_{i},e_{j}}|}$

##  4 Experiments and Results

To assess the utility of our model framework, we investigate whether the
learned representation of the sources’ selection preferences correlate with
the respective ideological preferences among sources. The main hypothesis we
assess here is that sources with similar ideological preferences would discuss
certain topics in more similar ways compared to sources with diverging
interests. This way the distributional similarity/divergence of the source
representations can be used as a tool to identify polarity among sources.

For such, we focus on the 65 sources in NELA with 5-way political bias ratings
from _AllSides_ ({_left, lean left, center, lean right, right_}).

###  4.1 Source Embedding Visualization

We first demonstrate that our langauge model representations of content
selection preferences are indicative of the sources’ selection biases and
ideological preferences out-of-the-box. Figure 2 shows the visualization of
the first two principle components for the per-source embeddings of four
example pairs of entities. We observe that the embeddings capture the
following two factors of the typical context in which the two entities are
described.

#### Topic Variance

Since the embeddings are learned through the conditional similarity between
context sentences, naturally, the variance, or the “scatteredness” of the
embeddings is indicative of the variety of topics in which the two entities
are mentioned in. For example, as Figure 2 shows, U.S. House and Senate are
generally more likely mentioned in similar context than Russia and United
States, regardless of the political tendency of the sources.

#### Divergence among _left_ vs _right_ sources

We observe that the representations of selection preferences for certain
topics are directly indicative of the ideological divide between the _left_ vs
_right_ sources. From Figure 2, we see that U.S. House and Senate is a good
example of such. For Russia and United States, United Kingdom and United
States, despite the high topic variance, the PCA plot still suggests a clear
divide between how the two sides typically frame the topic. We also observe
counter-examples such as _Donald Trump_ vs. _Joe Biden_ , we suspect that as
news mentioning the two in 2020 mostly focus on election, the two entities are
more likely to be mentioned in similar context by different sources.

The findings suggest that we can recognize and characterize the polarity
between sources, using a set of _archetypical_ entity pairs, as we will
demonstrate in § 4.2 and § 5.2.

Model | Classifier | $P$ | $R$ | $F_{\beta=1}$
---|---|---|---|---
RoBERTa | Lasso Regression | 59.55 | 56.10 | 57.77
Ridge Regression | 68.90 | 56.10 | 61.84
Linear SVM | 76.51 | 56.10 | 64.73
RBF-Kernel SVM | 70.49 | 68.29 | 69.37
Ours | Lasso Regression | 70.22 | 60.98 | 65.27
Ridge Regression | 70.85 | 53.66 | 61.06
Linear SVM | 74.49 | 60.98 | 67.06
RBF-Kernel SVM | 80.49 | 75.61 | 77.97

Table 1:  Three-way polarity $\\{$left, center, right$\\}$ prediction results
on the MBFC test set, in weighted macro-averaged precision ($P$), recall ($R$)
and $F_{\beta=1}$. We use the embeddings from 20 most frequent entity pairs in
the NELA corpus as input features to the classifiers.

###  4.2 Source Polarity Prediction

We now use a modeling approach to evaluate whether our language model
representations of content selection preferences can be used as indicators for
a sources’ ideological preference. We formulate our task as a three-way
polarity prediction task: Given a set of $k$ entity pairs, we construct the
representation $g^{s_{i}}$ of each source $s_{i}$ by concatenating $s_{i}$’s
embeddings for all $k$ entity pairs. Then we train a classifier using
$g^{s_{i}}$ as input features, and predict the three-way $\\{$left, center,
right$\\}$ political bias rating of the source $s_{i}$.

Highest Divergence | $D(\mathcal{N}_{L}\|\mathcal{N}_{R})$ | Lowest Divergence | $D(\mathcal{N}_{L}\|\mathcal{N}_{R})$
---|---|---|---
U.S. House of Representatives and Senate | 1.87 |  Donald Trump and U.S. House | 0.07
Iran and United States | 1.42 |  United States and U.S. Senate | 0.07
Donald Trump and the F.B.I. | 1.40 |  Joe Biden and Kamala Harris | 0.07
C.D.C and United States | 1.36 |  United States and Mexico | 0.08
China and United States | 1.12 |  United States and White House | 0.08
Donald Trump and Russia | 1.00 |  United States and United States Congress | 0.09

Table 2:  Example entity pairs with the highest and lowest distributional
divergence (§ 5.1) among sources with respect to their political bias ratings
from _AllSides_.

To construct a test set with sources not labeled by AllSides, we collect the
5-way political bias ratings from Media Bias / Fact Check
(MBFC)555https://mediabiasfactcheck.com/. We aggregate the 5-way labels from
both AllSides and MBFC into 3-way $\\{$left, center, right$\\}$. After
removing the $54$ sources from MBFC that overlaps with AllSides, we use the
rest $41$ sources as the heldout test set. We verify the label agreement
between the two by evaluating the ground truth MBFC’s labels against AllSides’
using the overlap set, and get $81.99$ weighted macro-averaged $F_{1}$. More
details can be found in Appendix B.

For baseline comparison, we replace our context sentence encoder with
RoBERTabase without finetuning. As the main goal here is to evaluate the
descriptive power of the embeddings, we compare four types of popular
linear/non-linear classifiers with relatively small parameter size to avoid
overfitting, and freeze the embeddings from underlying embedding model during
classifier training. For all models, we set $k=20$, and use the 20 most
frequent entity pairs in the NELA corpus to construct the source
representation and input features $g^{s_{i}}$ for each source $s_{i}$.

Table 1 shows the evaluation results. We run 5-fold cross-validation on the
AllSide training set to select the hyper-parameters of the classifier models
individually. We report the hyperparamter settings in Appendix A. We observe
that the SVM model with RBF kernel, the only non-linear model, performs the
best on the MBFC test set. More importantly, across all methods, except for
Ridge Regression, our method achieved consistently better results than
RoBERTa, suggesting the benefit and advantage of our learning objectives on
modeling sources’ selection biases.

##  5 Analysis

###  5.1 Identifying Archetypical Entity Pairs for Ideological Divergence

In the previous section, we empirically show that, depending on the choice of
entity pairs, the content selection preferences can be indicative of the
divide between _left vs. right_ ideological preferences. Conceptually,
identifying such a _archetypical_ set of entity pairs would provide us an
interface to better recognize and characterize polarity among sources, and in
turn enable us to reason about their interests and patterns with respect to
the each topic.

We now describe a statistical procedure to automatically identify such
_Archetypical Entity Pairs (AEPs)_ by measuring the distributional divergence
of the embeddings from left vs right sources. Like § 4.2, we aggregate five-
way media bias ratings from AllSides into three-way $\\{$left, center,
right$\\}$ labels. For each entity pair $\\{e_{i},e_{j}\\}$, we use the
distributional divergence, measured by two-way Kullback–Leibler (KL)
divergence, of left- vs. right-leaning sources to measure whether
$\\{e_{i},e_{j}\\}$ is a good indicator for the divide between left vs. right
sources. For such, we fit multi-variate Gaussian distribution
$\mathcal{N}_{L}(\mu_{L},\Sigma_{L})$ and
$\mathcal{N}_{R}(\mu_{R},\Sigma_{R})$ to the $d$ dimensional embeddings of
left- vs. right-leaning sources respectively. We then compute the KL-
divergence between the two Gaussians.

$\displaystyle
D(\mathcal{N}_{L}\|\mathcal{N}_{R})=\frac{1}{2}(tr(\Sigma_{R}^{-1}\Sigma_{L})+ln\frac{|\Sigma_{R}|}{|\Sigma_{L}|}$
$\displaystyle+(\mu_{R}-\mu_{L})^{T}\Sigma_{R}^{-1}(\mu_{R}-\mu_{L})-d)$

We compute $D(\mathcal{N}_{L}\|\mathcal{N}_{R})$ for the 100 most frequent
entity pairs in the NELA corpus. The average
$D(\mathcal{N}_{L}\|\mathcal{N}_{R})$ is $0.386$, with standard deviation of
$0.394$. Table 2 shows the entity pairs with the highest and lowest divergence
respectively.

Conceptually, this provides us a systematic way to quantify which entity pairs
serve as better indicators for the divide between the content selection
preferences from groups of sources with different political tendencies.

###  5.2 Understanding Sources’ Selection Biases with Archetypical Entity
Pairs

In § 5.1, we demonstrated a way to construct a set of AEPs through
distributional divergence measurements. We now demonstrate, through a case
study, how a such set of AEPs can be used to characterize sources’ selection
biases.

We start by selecting five example sources of interest: Breitbart (Right), Fox
News (Lean Right), CNN (Center), The New York Times (Lean Left), and The
Huffington Post (Left). We select the 10 entity pairs with the highest
$D(\mathcal{N}_{L}\|\mathcal{N}_{R})$, that also have more than $10$ context
sentences present in the corpus. For each entity pair, we train an individual
linear SVM classifier666We choose not to use the best performing RBF-kernel
SVM here, as it is more sensitive to hyperparamter choices, which are more
difficult to tune for in this setting. to predict the three way political bias
rating of the sources using the AllSides training set, with the five sources
of interest held out from training. We use the individual SVM classifiers to
classify the political bias rating of the five sources with respect to each
entity pair.

Figure 3: Predicted polarity {left, center, right} of five held-out sources
(with most prolific publication rates in the NELA corpus) of different
political bias ratings with respect to ten archetypical entity pairs (AEPs)
with highest $D(\mathcal{N}_{L}\|\mathcal{N}_{R})$ values. We observe that the
distribution of predicted polarity on the AEPs is indicative of the sources’
political bias ratings. Some entity pairs shown in Table 2 are omitted from
this figure, due to lack of coverage by one or more of the five sources.

Figure 3 shows the prediction results. We observe that label distribution on
the 10 entity pairs indicates the political bias rating of the individual
sources. In the case of The New York Times and Breitbart, the predicted
polarity on all 10 entity pairs consistently reflect their political bias
ratings. Some disagreements can be observed with the other three sources.
Interestingly, in the case of CNN, the predicted polarity on the 10 entity
pairs are mixed, potentially indicating the source’s more centered or unbiased
position on the political bias spectrum.

With such analysis, we reach the interesting observation that sources don’t
conform to their _perceived_ ideological preference or political tendency on
every matter. Depending on the topic, the same source could very well exhibit
opposite attitude to their usual ideological preference. Similarly, sources
with different ideological preferences do not always disagree with each other,
and there exist common grounds. We argue that the key advantage of our
proposed methodology is that it can uncover such agreements/disagreements on a
fine-grained topic level. This allows us to jointly reason and characterize
the selection biases and (political) polarity in an interpretable way.

###  5.3 Probing with Subjective Predicates Templates

To understand what media attitude or sentiment that the context sentence
embeddings imply or entail, we design a probing method to our embeddings that
leverages the semantics of subjective predicates.

Figure 4: Sentiment polarity score (_positive_ $\rightarrow$1 and
negative$\rightarrow$-1) by a list of sources for three example pairs of
entities, inferred from the projected similarity between subjective verb
predicates (§ 5.3) and the context sentence embeddings by each source. The
sources are colored according to their political bias rating from _AllSides_.

Following similar strategies proposed by Iyyer et al. (2016) and Han et al.
(2019), we construct a list of subjective predicates and project our
embeddings as a distribution over the subjective predicates. For such, we
sample 400 verb predicates with its sentiment polarity labels (positive vs
negative) from the MPQA subjective lexicon Wilson et al. (2005), with the 200
most frequent positive and negative polarity verb predicates in the NELA
corpus respectively. With each verb predicate, we formulate a prompt in the
same formats as the inputs to our context sentence encoder (§ 3.2): We prepend
[ENT][R0] and append [ENT][R1]. to the verb predicate as a probing prompt to
the model. We encode the probing prompt for each individual verb predicate
using the context sentence encoder. Given an context sentence embedding from a
source, we compute its cosine similarity to the embeddings of all 400 verb
predicate prompts. We take the top 10 most similar verb predicates, and take
the weighted mean of the polarity label (positive $=1$ vs negative $=-1$) of
the verb predicates as the sentiment score for the context sentence. The
weight on each verb predicate is determined by the softmax probabilities over
the cosine similarity scores.

Figure 5 shows the projected sentiment score on three example entity pairs
from a list of sources with different political bias ratings. We observe that
in the cases of C.D.C and United States, as well as Afghanistan and United
States, sources from different sides mostly exhibit negative sentiment when
describing the entity pairs. In the case of Afghanistan and United States, the
overall negative sentiment is stronger, and more conservative sources seem to
exhibit more polarized attitude compared to the liberal sources. Similar
observations can be made on C.D.C and United States. In the case of Democratic
Party and Nancy Pelosi, we see a more mixed sentiment from most sources, with
only a few exceptions.

The probing result shows once again the nuanced difference in how sources
typically frame different set of topics, and how such difference does not
necessarily conform with their perceived political bias.

##  6 Related Work and Discussion

In this study, we propose and study a conceptual entity-focused approach to
represent and reason about the similarity/difference of various sources’
content selection preferences. We in turn discover that such representations
can be used as describe their ideological polarity.

Besides the use case in this study, named entities have many other
implications to discourse modeling and understanding in NLP. Choi et al.
(2016) propose a general ILP-based inference method for tracking the dynamic
sentiment relations between entities mentioned in the same document. Iyyer et
al. (2016) introduce a unsupervised learning method to project the relation
between fiction characters on to word embedding space. Han et al. (2019) adopt
a similar strategy to track the dynamic relationship between nations, as
depicted by news media. You and Goldwasser (2020) formulate a cloze style task
for predicting the relation between two fictional characters in the multiple-
choice question-answering format.

The goal of our study aligns in part with frame analysis. The goal of frame
analysis is typically identifying selection biases, manifested by frame usages
in articles, and study its connection to certain attributes of information
sources, such as ideological preferences. Frame analysis usually starts from a
pre-defined typology of sub-topic frames with respect to a set of
controversial topics. Boydstun et al. (2013) defines a set of 15 commonly used
frames by news media, and Card et al. (2015) introduces a large-scale corpus,
where documents of three topics are hand labeled with their sub-topic frame
usage. A series of studies has since used the two resources to develop
classifiers for predicting the primary frame of a document Card et al. (2016);
Ji and Smith (2017); Field et al. (2018). Our work instead looks to alleviate
the limits of a predefined typology, and instead use entities as an open-
domain way to represent topic.

It’s worth noting that the goal of this work is intrinsically different from
the political polarity prediction line of work Preoţiuc-Pietro et al. (2017);
Kulkarni et al. (2018), as our goal is not trying to predict political
polarity for the sources, but instead using polarity as an interface to
characterize the sources’ selection biases.

The motivation of our work is closest in spirit to a recent study by Pujari
and Goldwasser (2021), where they represent the political figures with their
statements with regards to different entities. However, their approach
requires a predefined set of political figures, news events, entities, which
limits scalable analysis on more timely news content. As we design our context
sentence encoder to be entity-agnostic, our method does not have such
requirements. Our framework instead leverages the coverage on entity pairs
that can be aligned across multiple sources. Although such formulation
requires no labeled data, data scarcity can be a concern due to the sparse
nature of entity pairs.

##  7 Conclusion

In this paper, we discuss and investigate the challenges of discovering
patterns of selection biases from a massive-scale news corpus. Through a case
study, we empirically validate the hypothesis that a source’s selection biases
have correlation with its ideological leaning.

With our empirical experimental results, we reach the interesting observation
that sources do not necessarily conform to the same expected way of framing
topics, and sources with different ideological preferences would still have
common ground when it comes to certain topics. Therefore, we argue that
understanding the selection biases in news media requires far more than
assigning labels on the source-level. Instead, a good, alternative strategy is
to comparatively analyze the statements made with respect to various topics
among different sources, and understanding the implications of such difference
in connection to the ideological/political preferences of the sources. Our
paper merely offers one speculative way of doing so, and We hope that our
proposed methodology and findings will facilitate future work on alone this
line.

## Limitations

Since the nature of the paper is provide initial speculative insights and
empirical validation of our conceptual model, our method, in its current form,
has many limitations. For one, even though our method requires no labeled
data, it indeed requires a large amount of unlabeled data (for enough entity
coverage). Specifically, our analysis method requires enough presence of
context sentences per entity pair in the corpus.

As mentioned in $\S~{}\ref{sec:background}$, our study only focus on comparing
the topics that both/all sources cover. We leave the more challenging cases of
partial source coverage for future work.

It’s worth noting that the analysis tool developed in this study only provides
speculations for studying selection biases and polarity of news sources. The
results alone do not stand as evidence for such.

## Acknowledgements

This work was supported by a Focused Award from Google, and by Contract
FA8750-19-2-1004 with the US Defense Advanced Research Projects Agency
(DARPA). Approved for Public Release, Distribution Unlimited. The views
expressed are those of the authors and do not reflect the official policy or
position of the Department of Defense or the U.S. Government.

## References

  * Boydstun et al. (2013) Amber E Boydstun, Justin H Gross, Philip Resnik, and Noah A Smith. 2013.  Identifying media frames and frame dynamics within and across policy issues.  In _New Directions in Analyzing Text as Data Workshop, London_.
  * Card et al. (2015) Dallas Card, Amber Boydstun, Justin H Gross, Philip Resnik, and Noah A Smith. 2015\.  The media frames corpus: Annotations of frames across issues.  In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 438–444.
  * Card et al. (2016) Dallas Card, Justin H Gross, Amber Boydstun, and Noah A Smith. 2016.  Analyzing framing through the casts of characters in the news.  In _Proceedings of the 2016 conference on empirical methods in natural language processing_ , pages 1410–1420.
  * Choi et al. (2016) Eunsol Choi, Hannah Rashkin, Luke Zettlemoyer, and Yejin Choi. 2016.  Document-level sentiment inference with social, faction, and discourse context.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 333–343.
  * D’Alessio and Allen (2000) Dave D’Alessio and Mike Allen. 2000.  Media bias in presidential elections: A meta-analysis.  _Journal of communication_ , 50(4):133–156.
  * Entman (1993) Robert M Entman. 1993.  Framing: Towards clarification of a fractured paradigm.  _McQuail’s reader in mass communication theory_ , pages 390–397.
  * Field et al. (2018) Anjalie Field, Doron Kliger, Shuly Wintner, Jennifer Pan, Dan Jurafsky, and Yulia Tsvetkov. 2018.  Framing and agenda-setting in russian news: a computational analysis of intricate political strategies.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3570–3580.
  * Gabrilovich et al. (2007) Evgeniy Gabrilovich, Shaul Markovitch, et al. 2007.  Computing semantic relatedness using wikipedia-based explicit semantic analysis.  In _IJcAI_ , volume 7, pages 1606–1611.
  * Groeling (2013) Tim Groeling. 2013.  Media bias by the numbers: Challenges and opportunities in the empirical study of partisan news.  _Annual Review of Political Science_ , 16.
  * Gruppi et al. (2021) Maurício Gruppi, Benjamin D Horne, and Sibel Adalı. 2021.  Nela-gt-2020: A large multi-labelled news dataset for the study of misinformation in news articles.  _arXiv preprint arXiv:2102.04567_.
  * Han et al. (2019) Xiaochuang Han, Eunsol Choi, and Chenhao Tan. 2019.  No permanent friends or enemies: Tracking relationships between nations from news.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 1660–1676.
  * Hocke (1996) Peter Hocke. 1996.  _Determining the selection bias in local and national newspaper reports on protest events_.  WZB.
  * Horne et al. (2018) Benjamin D Horne, Sara Khedr, and Sibel Adali. 2018.  Sampling the news producers: A large news and feature data set for the study of the complex media landscape.  In _Twelfth International AAAI Conference on Web and Social Media_.
  * Iyyer et al. (2016) Mohit Iyyer, Anupam Guha, Snigdha Chaturvedi, Jordan Boyd-Graber, and Hal Daumé III. 2016.  Feuding families and former friends: Unsupervised learning for dynamic fictional relationships.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 1534–1544.
  * Ji and Smith (2017) Yangfeng Ji and Noah A Smith. 2017.  Neural discourse structure for text categorization.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 996–1005.
  * Keskar et al. (2019) Nitish Shirish Keskar, Bryan McCann, Lav R Varshney, Caiming Xiong, and Richard Socher. 2019.  Ctrl: A conditional transformer language model for controllable generation.  _arXiv preprint arXiv:1909.05858_.
  * Kulkarni et al. (2018) Vivek Kulkarni, Junting Ye, Steven Skiena, and William Yang Wang. 2018.  Multi-view models for political ideology detection of news articles.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3518–3527.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * McCarthy et al. (1996) John D McCarthy, Clark McPhail, and Jackie Smith. 1996.  Images of protest: Dimensions of selection bias in media coverage of washington demonstrations, 1982 and 1991.  _American sociological review_ , pages 478–499.
  * Preoţiuc-Pietro et al. (2017) Daniel Preoţiuc-Pietro, Ye Liu, Daniel Hopkins, and Lyle Ungar. 2017.  Beyond binary labels: political ideology prediction of twitter users.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 729–740.
  * Pujari and Goldwasser (2021) Rajkumar Pujari and Dan Goldwasser. 2021.  Understanding politics via contextualized discourse processing.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1353–1367, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Ratinov et al. (2011) Lev Ratinov, Dan Roth, Doug Downey, and Mike Anderson. 2011.  Local and global algorithms for disambiguation to wikipedia.  In _Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies_ , pages 1375–1384.
  * Roy and Goldwasser (2020) Shamik Roy and Dan Goldwasser. 2020.  Weakly supervised learning of nuanced frames for analyzing polarization in news media.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7698–7716, Online. Association for Computational Linguistics.
  * Sun et al. (2021) Haitian Sun, Pat Verga, Bhuwan Dhingra, Ruslan Salakhutdinov, and William W. Cohen. 2021.  Reasoning over virtual knowledge bases with open predicate relations.  In _Proceedings of the 38th International Conference on Machine Learning_.
  * Verga et al. (2021) Pat Verga, Haitian Sun, Livio Baldini Soares, and William Cohen. 2021.  Adaptable and interpretable neural memoryover symbolic knowledge.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3678–3691.
  * Wilson et al. (2005) Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005.  Recognizing contextual polarity in phrase-level sentiment analysis.  In _Proceedings of human language technology conference and conference on empirical methods in natural language processing_ , pages 347–354.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * You and Goldwasser (2020) Keen You and Dan Goldwasser. 2020.  “where is this relationship going?”: Understanding relationship trajectories in narrative text.  In _Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics_ , pages 168–178.

##  Appendix A Models and Hyperparamter Settings

We implement our context sentence encoder (§ 3.2) using Huggingface’s
transformers library Wolf et al. (2020), under Apache 2.0 license. We use and
initialize the parameters of our models with pretrained roberta-base model.

We train the context sentence encoder with ADAM optimizer, with learning rate
$=1e-5$ and $3$ training epoch. Our model is trained on a single Nvidia TITAN
RTX in 5 hours.

For the classifiers, we use scikit-learn’s implementation of the four
classifier models. Below we describe the hyperparamter settings of each model
individually.

  1. 1.

Lasso: $C=1$

  2. 2.

Ridge: $\alpha=1.0$

  3. 3.

Linear SVM: $C=0.1$ with L-2 penalty and hinge loss

  4. 4.

RBF-Kernel SVM: $C=2$, with polynomial degree=3 for the kernel function.
$\gamma=0.01$

The weights for all classifiers are zero initialized.

##  Appendix B Label Agreement between AllSides and MBFC

MBFC has $54$ sources labeled with 5-way political bias ratings that overlaps
with AllSides annotations. We verify the label agreement between the two by
evaluating the ground truth MBFC’s labels against AllSides’ using the overlap
set, and get precision=$84.50$, recall=$79.63$, and $81.99$ weighted macro-
averaged $F_{1}$. We show the confusion matrix between the overlap set below.

Figure 5: Confusion matrix of label agreement between MBFC and AllSides on the
overlapping set of 54 sources.