# Analysis of the Global Banking Network by Random Matrix Theory

Ali Namaki<EMAIL_ADDRESS>Jamshid Ardalankia  Reza Raei  Leila
Hedayatifar  Ali Hosseiny  Emmanuel Haven<EMAIL_ADDRESS>G.Reza Jafari
<EMAIL_ADDRESS>Department of Finance, University of Tehran, Tehran, Iran
Iran Finance Association, Tehran, Iran  Department of Financial Management,
Shahid Beheshti University, G.C., Evin, Tehran, 19839, Iran  Center for
Complex Networks and Social Datascience, Department of Physics, Shahid
Beheshti University, G.C., Evin, Tehran, 19839, Iran  New England Complex
Systems Institute, NECSI HQ 277 Broadway, Cambridge, MA — 02139, United States
Faculty of Business Administration, Memorial University, St. John’s, Canada
and IQSCS, UK  Department of Physics, Shahid Beheshti University, G.C., Evin,
Tehran, 19839, Iran  Department of Network and Data Science, Central European
University, 1051 Budapest, Hungary

###### Abstract

Since 2008, the network analysis of financial systems is one of the most
important subjects in economics. In this paper, we have used the complexity
approach and Random Matrix Theory (RMT) for analyzing the global banking
network. By applying this method on a cross border lending network, it is
shown that the network has been denser and the connectivity between peripheral
nodes and the central section has risen. Also, by considering the collective
behavior of the system and comparing it with the shuffled one, we can see that
this network obtains a specific structure. By using the inverse participation
ratio concept, we can see that after 2000, the participation of different
modes to the network has increased and tends to the market mode of the system.
Although no important change in the total market share of trading occurs,
through the passage of time, the contribution of some countries in the network
structure has increased. The technique proposed in the paper can be useful for
analyzing different types of interaction networks between countries.

###### keywords:

Global Banking Network , Complex Systems , Random Matrix Theory , Financial
Contagion

##  1 Introduction

Since the recent global financial crisis, cross-border lending and financial
contagions have gained importance. This importance stems from the propagated
effects [1, 2] of financial crises on political and economic situations [3,
4]. This fact has prompted a lot of research on the systemic dependence of the
international banking sector [5, 6, 7, 8, 9, 10, 11].

One of the most recent approaches for analyzing this situation comes from the
notion of complexity [5, 12]. The purpose of complexity science in finance
focuses on the analysis of the structure and the dynamics of entangled
systems. Many scholars have applied complexity techniques for analyzing
financial contagion [13, 10, 9, 6, 14]. Their findings suggested that
connectivity of financial institutions is the source of potential contagions.

Random Matrix Theory is one of the useful methods for analyzing the behavior
of complex systems [15, 16, 17, 18, 19, 12, 20, 21, 22, 23]. This theory was
developed by researchers to describe the situation of energy levels of quantum
systems [24, 25].

The universality regime of the eigenvalue statistics is the success factor of
Random Matrix Theory [26, 27, 28]. Based on previous studies, it is shown that
when the size of the matrix is very large, the eigenvalue distribution tends
towards a specific distribution [28].

Random Matrix Theory has been applied to analyze the behavior of coupling
matrices [12]. This technique divides the contents of the coupling matrix into
noise and information parts. The noise part of the coupling matrix conforms to
the Random Matrix Theory findings and the information part deviates from them.
This concept stems from the idea of solving the problem of non-stationary
cross correlation and measurement noise, as a result of market conditions and
the finite length of time series [28, 26].

It is shown that the majority of their eigenvalues agree with the random
matrix predictions, but the largest eigenvalue has deviations from those
estimations [27, 26, 29, 22]. In essence, this eigenvalue develops an energy
gap that separates it from the other eigenvalues [17]. The largest eigenvalue
is related to a strongly delocalized eigenvector that presents the collective
evolution of the system, and this is called the market mode. From this
perspective, the largest eigenvalue’s magnitude reflects the coupling strength
of the system [17].

One of the systems which can be analyzed by the complexity approach, is the
global banking network [30]. In this paper, by applying Random Matrix Theory
as a useful technique from complexity science, we want to analyze the global
banking network.

Our paper is organized as follows. In Section 2 we present our methods and, in
section 3 we apply Random Matrix Theory on the global banking network and
present our findings. Then, in section 4 we conclude.

##  2 Methods

Primarily Random Matrix Theory has been presented by some scholars in nuclear
physics such as Mehta [24, 25], for analyzing the energy levels of complex
quantum systems. Subsequently, the mentioned method helped to address specific
issues in other fields, such as finance [17, 28, 27, 26].

Based on the perception from random matrix theory, the eigenvalues –in the
real matrix– which deviate from the range of the eigenvalues –in the random
matrix– possess relatively more complete information from the system [27, 23,
28].

Figure 1: The evolution of global banking network is demonstrated for the 5
snapshots of 1978-Q3, 1988-Q3, 1998-Q3, 2008-Q3 and 2018-Q3. Left column;
shows the evolution of trading matrices between countries. In order to extract
the structure of their communities, we have applied the dendrogram weighted
matrices. Right column; shows the evolution of network topology.

Figure 2: up) The evolution of the largest eigenvalue, $\lambda_{max}$, of the
global banking network and its shuffled, $\lambda_{max}^{sh}$, are depicted.
down) The evolution of total trading volume is demonstrated. Figure 3: It is
depicted that overtime $<IPR>$ has tended to $IPR_{\lambda_{max}}$. It implies
that the contribution of countries has generally increased. Figure 4: shows
the % participation of each country in the eigenvector – corresponding to the
largest eigenvalue – versus %($\frac{volume_{j}}{\Sigma_{i}^{N}Volume_{i}}$.)

In Random Matrix Theory, there is a parameter named as the Inverse
Participation Ratio ${IPR}$ which is based on the theory of Anderson’s
localization [31], and it computes the number of components which
significantly participate in each eigenvector. This notion shows the effect of
components of each eigenvector, and specifically how the largest eigenvalues
deviate from the bulk region which is densely occupied by eigenvalues of the
random matrix. Based on the previous papers [17, 32], $IPR$ can be applied as
an indicator for measuring the collective behavior of the networks. The
formula of this concept is as follows:

$IPR(k)=\frac{1}{\sum_{l=1}^{n}(u_{l}^{k})^{4}};$ (1)

where $l=1,\ldots,n$ and $u_{l}^{k}$ is the $l^{th}$ element of $k^{th}$
eigenvector ($lk$). To further clarify the concept, one may consider examples
below:

– In case all elements of a certain eigenvector are equal to
$\frac{1}{\sqrt{N}}$, $IPR$ will be equal to $N$. This implies that whole
elements are significantly influential on the systems’ behavior.

– On the other hand, if just a single element is equal to $1$ and the others
are equal to $0$, $IPR$ would be equal to $1$. This implies that only this
component is effective in the corresponding eigenvector.

Hence, one can perceive that $IPR$ clarifies the number of influential
elements in a certain eigenvector.

##  3 Analysis of Global Banking Network by Random Matrix Theory

The banking industry is one of the most important sectors in finance. In this
regard, one of the significant aspects of financial contagion is the emergence
and transmission of crisis throughout the banking network. In Fig. 1, the
evolution of the global banking network in 5 snapshots (1978-Q3, 1988-Q3,
1998-Q3, 2008-Q3 and 2018-Q3) has been depicted. The left panel in Fig. 1
shows the dendrogram structure of communities for trading weighted matrices.
Also, the right column shows the evolution of the network topology. As
depicted, the network has been denser over time. Not only the contributions
have risen, but also the peripheral nodes are arranged closer and connected to
the central section.

In this study, we apply Random Matrix Theory for the data of BIS bilateral
locational statistics provided by the Bank for International Settlements (BIS)
[33] from 1978 until 2019. This data includes all ‘core’ countries (the
qualifier ‘core’ is used by many researchers such as [30], for 31 countries
which regularly report their financial data to BIS).

We create a weighted and directed financial transaction network corresponding
to each quarter from 1978 until 2019. Each link corresponds to a loan given by
a certain country to another one. Previous studies specifically shed light
onto countries’ dependency network and showed an increase in the dependency
structure of the network of those countries during the passage of time [30].
As already discussed, Random Matrix Theory is a powerful approach for
analyzing complex systems. In this paper we apply this concept for the
analysis of the global banking network as a complex network. For this purpose,
we choose the shuffling technique for the construction of a random matrix. The
shuffling method which is applied in this research is randomization of
bilateral trading volume (or links) in the network. It means that the PDF
remains unchanged and the bilateral trading relations will be shuffled. The
Shuffled matrix is an indication of no information in the system.

The global banking network possesses an adjacency matrix. This matrix can
intrinsically be explained by the eigenvalue decomposition methods [34]. The
eigenvector corresponding to the largest eigenvalue, $\lambda_{max}$, is the
most significant and is the market mode of the network [35, 17, 27, 26].

In this regard, we assess the temporal behavior of the largest eigenvalue, as
shown in Fig. 2.

By evaluating the behavior of $\lambda_{max}$ and comparing it with the
$\lambda_{max}$ of the corresponding shuffled matrix in Fig. 2, one can
observe the information content of the market mode. As depicted in Fig. 2, the
temporal behavior of the largest eigenvalue in the banking interaction matrix,
is totally different from that of the largest eigenvalue in the shuffled
matrix. This phenomenon determines the existence of information content
embedded in the largest eigenvalue of the banking interaction matrix.

When it comes to Fig. 2, the behavior of the maximum eigenvalue has been
ascending. This issue – as stated before— has bilateral effects.

The reasoning behind this is that on the one side, it causes more strength and
stability in the network, whilst on the other side, it yields to a more agile
contagion throughout the network [7]. In the post-crisis era after 2008,
simultaneous to a decrease in the maximum eigenvalue, the collective behavior
of the system has reduced and accordingly, local identities have been more
significant.

Since the so-obtained eigenvalue does not describe all the details and
properties of the collective behavior, one should investigate other quantities
in the network.

It is observable that during the global financial crisis, a structural
emergence with an increase in the difference between $\lambda_{max}$ and
$\lambda_{max}^{sh}$, Fig. 2, has occurred.

However, after the crisis, a significant decrease in the behavior of the
largest eigenvalue of the banking matrix relating to that of the shuffled
matrix has emerged.

Based on the above concepts, one of the best approaches for analyzing the
global banking network is the Random Matrix Theory technique.

As already discussed, one should keep in mind that $IPR$ possesses the ability
of information extraction from the collective behaviors of the systems.

In Fig. 3, By comparing $<IPR>$ and $IPR_{\lambda_{max}}$, one is able to
distinguish the temporal evolution of participation in the network.

In Fig. 3, we investigate the inverse participation ratio (IPR) in a temporal
process. In this context, by focusing on the mean inverse participation ratio
, $<IPR>$, and also, the inverse participation ratio of the largest eigenvalue
corresponding to the largest eigenvector , we investigate banking behaviors of
the countries and their influences on the network structure and the market
trend. In Fig. 3, $IPR_{mean}$ implies the effectiveness of the banking system
of most countries on the global network. However, from the temporal behavior
of $IPR_{\lambda_{max}}$, we observe that over time, less participation from
those countries on the largest eigenvector emerges.

In Fig. 4, $\%Participation$ stands for the contribution percentage of each
country in the eigenvector corresponding to the largest eigenvalue. $\%Volume$
is the trading volume of a country divided by the total trading volume. Hence,
$\%Participation$ shows the contribution in the structure, and, $\%Volume$
shows the contribution in the total trading volume. Thereby, Fig. 4 visualizes
the contributions in the structure versus the contribution in trading volume
within each year. In 2018-Q3, for the US, while the percentage of contribution
in the structure has been approximately constant, the percentage of
contribution in trading volume decreased.

##  4 Conclusion

In this paper, by applying Random Matrix Theory, the global banking network is
investigated. For this purpose, we compute the matrix of interaction of
banking sectors of BIS countries, and then by using the Random Matrix Theory
approach, the behavior of the largest eigenvalue and Inverse Participation
Ratio of this eigenvalue, as the market mode of the system over time, has been
analyzed. The value of the largest eigenvalue increases during the passage of
time. By observing the behavior of trading volume, it is shown that these
increases stem from the expansion of the network to some extent. Also, by
comparing with the shuffled one, we can deduce that the system gets a specific
structure. Generally speaking, the global banking network, today, is more
dense and interconnected. Also, we can see that after the year 2000 the value
of the mean $IPR$ has dropped and converged to $IPR_{\lambda_{max}}$. It means
that more countries have become more influential on the global banking
network. Furthermore, despite small changes in the share of total trading
volume, some countries such as the UK, have become more important in the
network structure.

As a concluding remark, the identities of banking systems of BIS countries
stems from two parts, i.e. i) from their own identities individually and, ii)
from their interactions in the global banking network. As a suggestion for
further work, one can construct the interaction matrices of the countries
based on other variables such as commercial interactions and so on.

## References

  * [1] G. Iori, R. N. Mantegna, L. Marotta, S. Miccichè, J. Porter, M. Tumminello, Networked relationships in the e-MID interbank market: A trading model with memory, Journal of Economic Dynamics and Control 50 (2015) 98–116.  doi:10.1016/j.jedc.2014.08.016.
  * [2] A. G. Haldane, R. M. May, Systemic risk in banking ecosystems, Nature 469 (7330) (2011) 351–355.  doi:10.1038/nature09659.
  * [3] K. R. Carmen M. Reinhart, This Time Is Different: Eight Centuries of Financial Folly, 1st Edition, Princeton University Press, 2009.
  * [4] M. G. A. Contreras, G. Fagiolo, Propagation of economic shocks in input-output networks: A cross-country analysis, Physical Review E 90 (6).  doi:10.1103/physreve.90.062812.
  * [5] A. S. E. B. e. Cont, Rama; Moussa, Network structure and systemic risk in banking systems, SSRN Electronic Journaldoi:10.2139/ssrn.1733528.
  * [6] J. Etesami, A. Habibnia, N. Kiyavash, Econometric modeling of systemic risk: going beyond pairwise comparison and allowing for nonlinearity, LSE Research Online Documents on Economics 70769, London School of Economics and Political Science, LSE Library (Mar. 2017).
URL https://ideas.repec.org/p/ehl/lserod/70769.html

  * [7] S. Battiston, D. D. Gatti, M. Gallegati, B. Greenwald, J. E. Stiglitz, Liaisons dangereuses: Increasing connectivity, risk sharing, and systemic risk, Journal of Economic Dynamics and Control 36 (8) (2012) 1121–1141.  doi:10.1016/j.jedc.2012.04.001.
  * [8] F. Betz, N. Hautsch, T. A. Peltonen, M. Schienle, Systemic risk spillovers in the european banking and sovereign network, Journal of Financial Stability 25 (2016) 206–224.  doi:10.1016/j.jfs.2015.10.006.
URL https://doi.org/10.1016/j.jfs.2015.10.006

  * [9] M. D’Errico, S. Battiston, T. Peltonen, M. Scheicher, How does risk flow in the credit default swap market?, ECB Working Paper 2041 (2017).  doi:10.2866/086521.
  * [10] S. Battiston, G. Caldarelli, Systemic risk in financial networks, Journal of Financial Management, Markets and Institutions, Rivista semestrale on line (2/2013) (2013) 129–154.  doi:10.12831/75568.
URL https://www.rivisteweb.it/doi/10.12831/75568

  * [11] F. Atyabi, O. Buchel, L. Hedayatifar, Driver countries in global banking network.
  * [12] G. Jafari, A. H. Shirazi, A. Namaki, R. Raei, Coupled time series analysis: Methods and applications, Computing in Science & Engineering 13 (6) (2011) 84–89.  doi:10.1109/mcse.2011.102.
  * [13] P. Glasserman, H. P. Young, Contagion in financial networks, Journal of Economic Literature 54 (3) (2016) 779–831.  doi:10.1257/jel.20151228.
  * [14] M. Bardoscia, S. Battiston, F. Caccioli, G. Caldarelli, Pathways towards instability in financial networks, Nature Communications 8 (1).  doi:10.1038/ncomms14416.
  * [15] M. POTTERS, J.-P. BOUCHAUD, Financial applications of random matrix theory: a short review, arXiv preprint arXiv:0910.1205.
  * [16] X. F. Jiang, T. T. Chen, B. Zheng, Structure of local interactions in complex financial dynamics, Scientific Reports 4 (1).  doi:10.1038/srep05321.
  * [17] A. Namaki, A. Shirazi, R. Raei, G. Jafari, Network analysis of a financial market based on genuine correlation and threshold method, Physica A: Statistical Mechanics and its Applications 390 (21-22) (2011) 3835–3841.  doi:10.1016/j.physa.2011.06.033.
  * [18] M. MacMahon, D. Garlaschelli, Community detection for correlation matrices, Physical Review X 5 (2).  doi:10.1103/physrevx.5.021006.
  * [19] L. Sandoval, I. D. P. Franca, Correlation of financial markets in times of crisis, Physica A: Statistical Mechanics and its Applications 391 (1-2) (2012) 187–208.  doi:10.1016/j.physa.2011.07.023.
  * [20] A. H. Shirazi, G. R. Jafari, J. Davoudi, J. Peinke, M. R. R. Tabar, M. Sahimi, Mapping stochastic processes onto complex networks, Journal of Statistical Mechanics: Theory and Experiment 2009 (07) (2009) P07046.  doi:10.1088/1742-5468/2009/07/p07046.
  * [21] J. Jurczyk, T. Rehberg, A. Eckrot, I. Morgenstern, Measuring critical transitions in financial markets, Scientific Reports 7 (1).  doi:10.1038/s41598-017-11854-1.
  * [22] J. Kwapień, S. Drożdż, Physical approach to complex systems, Physics Reports 515 (3-4) (2012) 115–226.  doi:10.1016/j.physrep.2012.01.007.
  * [23] A. Namaki, G. Jafari, R. Raei, Comparing the structure of an emerging market with a mature one under global perturbation, Physica A: Statistical Mechanics and its Applications 390 (17) (2011) 3020–3025.  doi:10.1016/j.physa.2011.04.004.
  * [24] M. L. MEHTA, Random Matrices, Elsevier, 1991.  doi:10.1016/c2009-0-22297-5.
  * [25] M. L. MEHTA, Preface to the third edition, in: Random Matrices, Elsevier, 2004, pp. xiii–xiv.  doi:10.1016/s0079-8169(04)80088-6.
  * [26] V. Plerou, P. Gopikrishnan, B. Rosenow, L. A. N. Amaral, T. Guhr, H. E. Stanley, Random matrix approach to cross correlations in financial data, Physical Review E 65 (6).  doi:10.1103/physreve.65.066126.
  * [27] L. LALOUX, P. CIZEAU, M. POTTERS, J.-P. BOUCHAUD, RANDOM MATRIX THEORY AND FINANCIAL CORRELATIONS, International Journal of Theoretical and Applied Finance 03 (03) (2000) 391–397.  doi:10.1142/s0219024900000255.
  * [28] A. NAMAKI, R. RAEI, G. R. JAFARI, COMPARING TEHRAN STOCK EXCHANGE AS AN EMERGING MARKET WITH a MATURE MARKET BY RANDOM MATRIX APPROACH, International Journal of Modern Physics C 22 (04) (2011) 371–383.  doi:10.1142/s0129183111016300.
  * [29] G.-J. Wang, C. Xie, S. Chen, J.-J. Yang, M.-Y. Yang, Random matrix theory analysis of cross-correlations in the US stock market: Evidence from pearson’s correlation coefficient and detrended cross-correlation coefficient, Physica A: Statistical Mechanics and its Applications 392 (17) (2013) 3715–3730.  doi:10.1016/j.physa.2013.04.027.
  * [30] J. A. Reyes, C. Minoiu, and, A network analysis of global banking:1978-2009, IMF Working Papers 11 (74) (2011) 1.  doi:10.5089/9781455227051.001.
  * [31] G. Lim, S. Kim, J. Kim, P. Kim, Y. Kang, S. Park, I. Park, S.-B. Park, K. Kim, Structure of a financial cross-correlation matrix under attack, Physica A: Statistical Mechanics and its Applications 388 (18) (2009) 3851–3858.  doi:10.1016/j.physa.2009.05.018.
  * [32] M. Saeedian, T. Jamali, M. Kamali, H. Bayani, T. Yasseri, G. Jafari, Emergence of world-stock-market network, Physica A: Statistical Mechanics and its Applications 526 (2019) 120792.  doi:10.1016/j.physa.2019.04.028.
  * [33] Bank for international settlements (bis).
URL https://stats.bis.org/

  * [34] P. Barucca, M. Kieburg, A. Ossipov, Eigenvalue and eigenvector statistics in time series analysis, arXiv preprint arXiv:1904.05079.
  * [35] A. Utsugi, K. Ino, M. Oshikawa, Random matrix theory analysis of cross correlations in financial markets, Physical Review E 70 (2).  doi:10.1103/physreve.70.026110.


#  $Q_{bias}$ \- A Dataset on Media Bias in Search Queries and Query
Suggestions

Fabian Haak<EMAIL_ADDRESS>0000-0002-3392-7860 Technische Hochschule
KölnGustav-Heinemann-Ufer 54CologneGermany50678 and  Philipp Schaer
<EMAIL_ADDRESS>0000-0002-8817-4632 Technische Hochschule
KölnGustav-Heinemann-Ufer 54CologneGermany50678

(2023)

###### Abstract.

This publication describes the motivation and generation of $Q_{bias}$, a
large dataset of Google and Bing search queries, a scraping tool and dataset
for biased news articles, as well as language models for the investigation of
bias in online search. Web search engines are a major factor and trusted
source in information search, especially in the political domain. However,
biased information can influence opinion formation and lead to biased
opinions. To interact with search engines, users formulate search queries and
interact with search query suggestions provided by the search engines. A lack
of datasets on search queries inhibits research on the subject. We use
$Q_{bias}$ to evaluate different approaches to fine-tuning transformer-based
language models with the goal of producing models capable of biasing text with
left and right political stance. Additionally to this work we provided
datasets and language models for biasing texts that allow further research on
bias in online information search.

web search, dataset, bias, query suggestion, search queries, language models,
transformers

††copyright: acmcopyright††journalyear: 2023††copyright:
acmlicensed††conference: 15th ACM Web Science Conference 2023; April 30-May 1,
2023; Evanston, TX, USA††booktitle: 15th ACM Web Science Conference 2023
(WebSci ’23), April 30-May 1, 2023, Evanston, TX, USA††price: 15.00††doi:
10.1145/3578503.3583628††isbn: 979-8-4007-0089-7/23/04††copyright:
none††copyright: none††ccs: Information systems Web search engines††ccs:
Information systems Query suggestion††ccs: Information systems Query
reformulation††ccs: Computing methodologies Natural language generation

##  1\. Introduction

Search engines such as Google and Bing are seen as trustworthy sources of
information on many topics, including political news and information (Ray,
2020; Edelman, 2022). Further, search engines have proven to have a major
impact on the formation of political opinions (Epstein and Robertson, 2015).
To interact with search engines, users formulate search queries that are an
expression of their information need. Based on these queries search engines
usually provide a set of search query suggestions (Niu and Kelly, 2014).
Queries and the choices users make when interacting with query suggestions are
based on the information need they want to satisfy, which is often to support
their personal opinions or beliefs founded on previously encountered
information (Belkin et al., 2001). This interaction process and the results
presented to the users are prone to be biased (Introna and Nissenbaum, 2000;
Bolukbasi et al., 2016; Mitra et al., 2014) and therefore the high level of
trust can be seen as problematic.

However, the true effect of biased search queries and different types of
inherent biases on the actual list of search results has not sufficiently been
investigated. One of the main reasons for the infrequency of studies on bias
in search queries might be a lack of publicly available datasets. datasets
that include real-world user queries, query suggestions, and actual query
reformulations are rare. One of the reasons is that collecting search queries
from users is problematic due to privacy concerns. Although there are
techniques like pseudonymization, query logs enable the identification of
users (Barbaro and Zeller, 2006). Previously available datasets such as the
AOL query log dataset (Pass et al., 2006) are no longer available, and their
usage is morally debatable.

Using unpersonalized search query suggestions as proxies might solve this
issue. Query suggestions describe the list of predicted queries suggested to
users during the input of search queries by the search engine, sometimes also
called search predictions (Sullivan, 2018) or query auto completion (Cai and
de Rijke, 2016). While these search query suggestions can be generated locally
from the result set (Xu and Croft, 2000) or be taken from global knowledge
bases (Hienert et al., 2011), in web search suggestions are mostly based on
frequently issued queries by users (Sullivan, 2018). It can therefore be
assumed that in many cases query suggestions are popular related search
queries for the initial root query that represented the user’s interest in a
topic or entity.

The U.S. news domain is a popular domain for bias research due to its
sociological relevance and the two-party left-right spectrum that facilitates
bias analysis (Robertson et al., 2018). One of the benefits of our dataset is
to enable in-depth investigations of the correlation between bias in search
queries and search results, as search results for popular web search engines
can easily be collected using our provided dataset of search queries.
Therefore, one goal of this work is to provide a large dataset of search query
suggestions for the U.S. political news domain. Additionally, we provide a
transformer-based methodology for biasing text. Such a system can be a useful
tool in bias research, f.e. for generating biased derivatives of search
queries. Thus, we evaluate a range of fine-tuning scenarios to find settings,
that produce the most biased results. We provide the fine-tuned transformer-
based language models that are capable of inducing left and right political
stance bias in the form of lexical biases.

This publication describes the motivation and generation of $Q_{bias}$, a
large dataset of Google and Bing search queries, a scraping tool and dataset
for biased news articles, as well as language models for the investigation of
bias in online search.

The main contributions of $Q_{bias}$ and this paper are:

  * •

Two datasets:111The datasets can be found at Zenodo
(https://doi.org/10.5281/zenodo.7682914) (1) The (to the best of our
knowledge) largest labeled dataset of search query suggestions of a single
domain for Google and Bing. (2) A dataset of biased news articles of the U.S.
political news domain.

  * •

A scraping tool that allows researchers to easily retrieve an up-to-date
version of the biased news dataset.222https://github.com/irgroup/Qbias

  * •

A new approach for producing biased search queries, using intentionally biased
transformer-based language models capable of producing biased texts.

  * •

An evaluation of different fine-tuning settings to find the most capable setup
for producing bias-inducing language models.

##  2\. Background and Related Work

#### Definition of Bias is the News Domain

Due to the subjective nature of bias, few publications attempt to explicitly
define bias in news (D’Alessio and Allen, 2000; Chen et al., 2020; Recasens et
al., 2013; AllSides, 2021). ’Biased news’ generally describes non-neutral and
opinionated news, but both are fuzzy attributes (Recasens et al., 2013). In
the political domain, the term opinionated describes having a fixed political
opinion and agenda, aspect of a cognitive (author- or reader-sided) bias often
described as partisan bias (Gawronski, 2021) or (political) stance (Liu et
al., 2021). Partisanship manifests at different granularities. At publishing
or reporting level, partisanship is expressed by the selection of certain
topics called selection bias, and coverage bias, the selection of different
views on a topic (D’Alessio and Allen, 2000). At text level, statement bias
describes “members of the media interjecting their own opinions into the text”
(D’Alessio and Allen, 2000), manifesting in overall opinionated texts. This
can take various forms, ranging from phrasing bias, the use of non-neutral
language (Hube and Fetahu, 2019) to moral framing and ideological bias
(Mokhberian et al., 2020). Spin bias describes bias introduced either by
omitting necessary (neutral) information or adding unnecessary information
(Chen et al., 2020). At word and n-gram level, biases manifest as linguistic
or lexical biases (Chen et al., 2020). Most linguistic biases are highly
domain- and context-specific. For example, framing bias describes subjective
words, while epistemological bias can be attributed to words targeting the
credibility of a statement (Recasens et al., 2013). Since these biases can be
identified more objectively, most approaches to detecting biases rely on the
identification of linguistic biases (AllSides, 2021).

#### Research on Bias in Online Search and Search Queries

Few publications investigate bias in online search in aspects other than
search results: Robertson et al. (2018) investigate partisan bias and filter
bubble effects in political searches by auditing SERPS for a set of queries.
They did not find significant evidence, that unbiased search queries in real
search sessions performed by real users lead to filter bubble effects. The
unanswered question is, whether biased queries in general lead to biased
search results.

Few studies investigate bias in search queries. Due to the difficult
accessibility of biased search queries, most studies focus on bias in search
query suggestions: in most of those, the authors investigate topical group
biases in search query suggestions in the political domain (Bonart et al.,
2019; Haak and Schaer, 2021, 2022). Overall, they observe minor topical gender
biases for search queries consisting of names of politicians. Research on bias
in online search has shown, that “factors such as the topic of the query, the
phrasing of query and the time at which a query is issued also impact the bias
seen by the users”(Kulshrestha et al., 2019).

#### Datasets of Biased News and Search Queries

Baly et al. (2020) predict media bias using news articles collected from
AllSides balanced news.333https://www.allsides.com/unbiased-balanced-news
AllSides news is a popular source for balanced news (Baly et al., 2020; Chen
et al., 2020; Mokhberian et al., 2020), since AllSides has a high standard for
assigning bias labels (AllSides, 2021). Similarly, Chen et al. (2020) use
AllSides-labeled news articles and adfontes labels to analyze bias at
different granularities. Mokhberian et al. (2020) develop a framing bias
detection and quantification approach, using a collection of news articles
that they label according to news outlet bias labels provided by
AllSides.444https://www.kaggle.com/datasets/snapcrack/all-the-news datasets
for search queries and query suggestions are sparse: most are either not
available anymore (Pass et al., 2006) or focus on a narrow topic (Bonart et
al., 2019). Robertson et al. (2019) introduce recursive algorithm
interrogation, a technique for recursively retrieving query suggestions of a
root query and their consecutive suggestions. This technique was employed by
Haak and Schaer (2022) to investigate bias in the German political domain.
However, to the best of our knowledge, there currently is no large-scale
dataset on search queries.

##  3\. AllSides Scraper and Datasets

We present two novel datasets, that promote the investigation of bias in
online news search. Both datasets can be found on Zenodo as mentioned in
section 1. Further, we provide a web scraping tool for retrieving an up-to-
date version of the AllSides dataset we provide. This section describes the
content of the datasets as well as their creation process. Figure 1 shows how
we assembled the datasets and use them in our approach for creating biasing
language models.

Figure 1. Pipeline used for generating bias-inducing language models. Black
boxes represent datasets, language models (only available upon request), and
tools provided via Zenodo, and GitHub.

###  3.1. AllSides Scraper

As described in section 2, the AllSides platform and the provides news is a
frequently used source for high quality labeled biased news. We want to
provide an easy means of retrieving the news and all corresponding
information. Similar datasets have previously been produced, as mentioned in
section 2. However, compared to the currently most recent available version by
Baly et al. (2020), our version includes more than 20 percent of additional
articles. Furthermore, for many tasks, especially in the news domain, it is
relevant to have the most recent documents available. We provide a Python-
based scraper, that scrapes all available AllSides news articles and gathers
available information as described in section 3.2. By providing the scraper we
facilitate access to a recent version of the dataset for other researchers.

###  3.2. AllSides Biased News Dataset

The dataset contains 21,747 news articles collected from AllSides balanced
news headline roundups (AllSides, 2022) in November 2022. The AllSides
balanced news feature three expert-selected U.S. news articles from sources of
different political views (left, right, center), often featuring spin bias,
slant other forms of non-neutral reporting on political news (AllSides, 2021).
All articles are tagged with a bias label by four expert annotators based on
the expressed political partisanship, left, right, or neutral (AllSides,
2021). The AllSides balanced news aims to offer multiple political
perspectives on important news stories, educate users on biases, and provide
multiple viewpoints (AllSides, 2021). Collected data includes the headline,
news text, publishing date, topic tags (e.g., ”Republican party”,
”coronavirus”, ”federal jobs”), links to the article, and the publishing news
outlet. We also include AllSides’ neutral description of the topic of the
articles. Overall, the produced dataset contains 10,273 articled tagged as
left, 7,222 articles tagged as right, and 4,252 articles tagged as center. The
collected articles have been published between June 2012, and the date of the
data collection at the end of November 2022. We use the AllSides dataset for
developing our biasing language model, as described in section 4. To allow for
easier access to the most recent and complete version of the dataset for
future research, we provide the scraping tool described in section 3.1.

###  3.3. Search Queries in U.S. News Domain Dataset

The second dataset we provide consists of 671,669 search query suggestions for
root queries based on tags of the AllSides biased news dataset. We collected
search query suggestions from Google and Bing for the 1,431 topic tags, that
have been used for tagging AllSides news at least five times, approximately
half of the total amount of topics. The topic tags include names, a wide range
of political terms, agendas, and topics (e.g., ”communism”, ”libertarian
party”, ”same-sex marriage”), cultural and religious terms (e.g., ”Ramadan”,
”pope Francis”), locations and other news-relevant terms. On average, the
dataset contains 469 search queries for each topic. In total, 318,185
suggestions have been retrieved from Google and 353,484 from Bing.

Using a python implementation loosely adopting the framework provided by
Robertson et al. (2019), we scraped query suggestions for the topics as root
queries. Using Google Colab to run our scraper, we retrieved ten search query
suggestions provided by the Google and Bing search autocomplete systems for
the input of each of these root queries. Furthermore, we extended the root
queries by the letters a to z (e.g., ”democrats” (root term) $\rightarrow$
”democrats a” (query input) $\rightarrow$ ”democrats and recession” (query
suggestion)). The goal of this procedure is to simulate a user’s input during
information search. Retrieving the suggestions for the root query and the
extended queries generates a total of up to 270 query suggestions per topic
and search engine. The dataset we provide contains columns for root term,
query input, and query suggestion for each suggested query. The location from
which the search is performed is the location of the Google servers running
Colab, in our case Iowa in the United States of America, which is added to the
dataset. Since we perform the scrape on a blank browser for each of the
searches, personalization effects other than the location did not effect the
suggested search queries. Our scraping setup thus eliminates personalization
effects that could, in theory, cause echo chamber effects (Robertson et al.,
2018). Search engine providers describe that query suggestions are based on
other users’ searches (Sullivan, 2018). Successful attempts to influence query
suggestions have confirmed this claim (Wang et al., 2018). Thus, we deduce
that query suggestions reflect real, frequently used search queries and can be
used as proxies for search queries. Despite the lack of information on the
frequency of the collected search queries, our dataset contains the ranks of
suggestions, as well as the suggestions to the root term for approximating the
most frequent search queries. Assuming that the topical tags in the AllSides
news reflect popular topics, the produced dataset consists of real search
queries for news-relevant topics.

##  4\. Developing Biasing Language Models

Another contribution $Q_{bias}$ is to produce a pair of transformer-based
language models that are capable of inducing left or right partisanship as
linguistic biases. This can be done by leveraging the masking function on the
words of the target document.

###  4.1. Domain-Adopting DistilBERT

This section describes the methodological approach for developing transformer-
based language models capable of biasing texts. Usually, systems are developed
with the goal of producing and reproducing as little bias as possible or to
debias biased texts (Raza et al., 2022). We try to achieve the opposite,
biasing texts by reproducing biases inherent in biased fine-tuning datasets.
This would, for example, allow for simulating a user that searches for biased
terms potentially caused by exposure to biased news. With the goal of
capturing biased language in pretrained language models and using the
HuggingFace Transformers module(Wolf et al., 2020), we fine-tune the base
DistilBERT model, which was trained in Wikipedia articles and a large book
dataset.555https://huggingface.co/distilbert-base-uncased DistilBERT–(Sanh et
al., 2019) was chosen as a base model since it has proven to perform well with
comparably small datasets, performs better than BERT despite its smaller size
(Sanh et al., 2019), and for its aptness for adopting domain-specificity (Bai
et al., 2020; Büyüköz et al., 2020). We are aware that there are more
effective models, but to show the suitability of our approach, we chose
DistilBERT due to its efficiency and sufficient effectiveness.

We fine-tuned a range of pairs of models, each with one left model fine-tuned
on the part of the AllSides corpus tagged as left-biased news and one right
model fine-tuned with documents of the dataset labeled as right-biased news.
We produced 24 models, a left and a right model for 12 combinations of
parameters. Our goal is to find the ideal approach for capturing and
reproducing as much bias as possible while producing meaningful results. The
central aspect we varied in fine-tuning the models is the data used. The
models are developed using three different data configurations: (a) the
headline and news text of each of the left and right news articles of the
AllSides dataset, (b) only the news text, and (c) only the headline. Another
factor we evaluate is the use of padding or concatenation to generate a
consistent chunk size for fine-tuning. The chosen chunk size is the max length
of fine-tuning documents for the padding approach and 128 for the
concatenation approach. Lastly, we compare the effects of intentional
overfitting by raising the number of epochs to 20, compared to a more
reasonable 6 epochs, which proved to produce a good combination of training
loss and validation loss for all dataset configurations. In theory,
overfitting could be another possibility for reproducing biased formulations
from the texts used in fine-tuning. This is why we tried to intentionally
raise the number of epochs as a parameter (Mosbach et al., 2021).

###  4.2. Evaluating the Biasing Language Models

All 24 models are available upon request. We decided not to make the models
publicly available due to concerns about potential harms that could be caused
by misuse of the systems that we further elaborated in section 5. For
evaluating the models’ effectiveness, we choose a combination of manual
assessment and quantitative measures. The task of the models is to generate
biased output, however, biases are diverse and not easily measurable. Since we
have no way of identifying biased tokens in the dataset, perplexity is not a
useful measure for evaluating the model. Further, we cannot effectively
measure bias as a criterion for the effectiveness of the ability of the
systems to produce biased versions of queries. However, we can assume that
when masking words in search queries for topics of the political news domain
and letting the pairs of language models predict the words, the output of the
left and right models should differ from each other. We choose to evaluate
their performance on search queries since we want to use the models in future
research on bias in search results for biased and unbiased search queries.
Further, they represent texts of a different type but the same domain as the
texts used in training the models.

For each pair of models, we measure the difference between the two models’ 10
most probable non-punctuation predictions for 100 randomly selected queries
from the dataset by measuring the Rank Biased Overlap (RBO) with $p=0.9$. We
mask a random token of each query that is neither part of the original news
topic nor a stopword since we want to let the models generate meaning-carrying
tokens while keeping the query’s topic intact. Although nonsensical random
predictions or neologisms would lead to great RBO scores, we want to generate
queries, that humans could have formulated. To assure that the models generate
output that makes sense, we let the models generate next word predictions for
ten bias-provoking sentences (e.g., ”Hilary Clinton is a”, ”Covid Vaccines
should be”). Two domain experts (a professor and a postdoctoral researcher)
then label nonsensical and biased predictions. We measure the inter-annotator
agreement on the individual statements (independent of the models, that
produce the statements) with Cohen’s Kappa (Cohen, 1960). For nonsensical
predictions, we obtained a value of 0.18 (slight agreement), for biased
predictions the value is 0.84 (almost perfect). The bias labels are assigned
if the suggested word induces political stance bias.

Table 1 shows the results of the model evaluation process. The lower $h$,$t$,
or $ht$ describes if headline, text, or both have been used. $RBO$ is the rank
bias overlap between the left and the right model of each configuration.
$P_{nons}$ describes the average percentage of nonsensical predictions of both
models and $P_{bias}$ the percentage of biased predictions. Overall, the
models fine-tuned with only headlines show the best (lowest) RBO scores, as
well as the lowest $P_{nons}$ scores. Headlines and text and only text perform
more or less equally in terms of their RBO. Concatenating texts produces on
average better results than padding, although for fine-tuning with only
headlines, the effect is minimal. Intentional overfitting by raising the
amount of training epochs seems to worsen the RBO scores. The percentage of
nonsensical predictions does not differ much between different fine-tuning
setups, fine-tuning on texts only seems to be the only scenario that increases
the percentage of nonsensical predictions. Many of the models generate a high
percentage of biased suggestions, with the headline models having the highest
percentage of biased predictions.

Table 1. Evaluation results of the fine-tuned language models. The highlighted values are the best results for each metric. Model configuration | $RBO$ | $P_{nons}$ | $P_{bias}$
---|---|---|---
$DistilBERTconcat_{h}$ | 0.54 | 0.04 | 0.33
$DistilBERTconcat_{t}$ | 0.72 | 0.10 | 0.14
$DistilBERTconcat_{ht}$ | 0.68 | 0.08 | 0.15
$DistilBERTconcat_{h}+overfitting$ | 0.56 | 0.03 | 0.5
$DistilBERTconcat_{t}+overfitting$ | 0.73 | 0.11 | 0.14
$DistilBERTconcat_{ht}+overfitting$ | 0.68 | 0.07 | 0.23
$DistilBERTpadding_{h}$ | 0.54 | 0.04 | 0.38
$DistilBERTpadding_{t}$ | 0.74 | 0.05 | 0.28
$DistilBERTpadding_{ht}$ | 0.77 | 0.07 | 0.1
$DistilBERTpadding_{h}+overfitting$ | 0.58 | 0.03 | 0.43
$DistilBERTpadding_{t}+overfitting$ | 0.77 | 0.06 | 0.24
$DistilBERTpadding_{ht}+overfitting$ | 0.77 | 0.07 | 0.12

##  5\. Discussion

Our language models show, that by using small, high-quality datasets, it is
possible to fine-tune transformer-based language models to bias texts. In our
fine-tuning setup, models fine-tuned with headlines produce the overall best
results. As the main reason for that, we assume that the high amount of quotes
in news texts, which often are statements the authors of the articles disagree
with, might have induced noise in fine-tuning the models. Further, the
condensed nature of headlines, which aims to catch attention, might also
reflect in resulting language models. The output of left and right models
fine-tuned with headlines produce texts containing linguistic biases. For
example, for ”Donald Trump is a”, our model produces ”hero” as right-biased
and ”fraud” as left-biased next word predictions. The overall low percentage
of nonsensical predictions supports RBO as a suited evaluation metric. Despite
the overall good results, future research should investigate the findings with
other language models and compare the performance to our results.

Our scraping tool and the datasets not only allow us to effectively generate
the language models but enables the investigation of other research topics in
the political news information search domain such as bias classification or
sentiment analysis.

#### Remark on Moral Issues

We are aware, that building bias-inducing systems and providing datasets that
enable the reproduction of developing similar systems is problematic. We also
did consider that our work can inspire and help to develop transformer-based
language models capable of producing biased, toxic, or otherwise harmful
texts. The severity of openly published biased systems has been shown by other
such models, e.g. the gpt-4chan
model.666https://huggingface.co/ykilcher/gpt-4chan The publication of the
model incited a debate, that shows how problematic biased language made
available to a wide public audience are, despite explicitly stating the
intended use for research applications (Reddit, 2022; Gault, 2022).

Our models are biased primarily in terms of linguistic biases that reflect
political stances and views on political topics. Since this can in part
include objectively wrong and opinionated statements, hate speech, racism, and
other forms of despicable language and toxicity, our models can and will
reproduce text, that conveys these phenomena. To minimize the harmful effects
of our publication that could be caused by the misuse of the models, we
decided to not make our biased models publicly accessible. Despite that, we
provide access to the models upon request for research, if a reasonable
application use case is provided. However, we do so in the interest of
investigating biases and correlations of bias in online information search,
with the goal of increasing transparency and fairness. Raising awareness for
how easily, intentionally or not, biases can be reproduced and induced with AI
systems is part of this endeavor.

Although we have shown how the data we provide can be used to create models
that produce biased language we chose to publish the datasets and scraper.
This is mainly because of three reasons: (a) as stated in section 3 similar
datasets are available publicly, (b) we believe that the benefits of raising
awareness outweigh making already public biased data more accessible, and (c)
datasets on biased language are required for developing systems to detect and
inhibit bias. With our work, we hope to highlight the need to assure, that
data used for developing models is unbiased and raise awareness for how easily
transformer-based language models can be fine-tuned to produce biased
language.

##  6\. Outlook

This publication represents the first and foundational milestone of a larger-
scale investigation of bias in online information search. As a major next
contribution, we plan to use the presented datasets and models to investigate
the effects of biased and unbiased search queries on the search results of
different search engines for popular topics of the U.S. political news domain.
To accomplish this, we need to overcome the issue of subjectivity and lack of
effective methodological approaches to bias identification other than by
employing human annotation. We plan to introduce an ensemble of methods,
including bias-agnostic analysis of linguistic differences, lexical features,
and transformer-based approaches for bias classification. Further, we plan to
conduct a study using a simulation approach. By simulation different user
behaviors in terms of information need, formulating queries, and interacting
with query suggestions in an interactive information search simulation, we
plan to gain insights into echo chamber effects and bias formation in
information search. Additionally, we hope to identify which user properties
and behaviors increase and which lower the risk of encountering bias in
information search.

##  7\. Conclusion

This work presents $Q_{bias}$, a first milestone of our ongoing research on
bias in online search. We present two datasets: a biased news dataset and a
large dataset of biased and unbiased search queries for topics of the U.S.
political news domain. Further, we provide a scraping tool, that allows for
collecting bias-labeled news texts from AllSides. Lastly, we evaluate
approaches to fine-tuning DistilBERT transformer-based language models for
biasing texts and publish our models capable of inducing left and right
political stance bias in the form of lexical biases.

## References

  * (1)
  * AllSides (2021) AllSides. 2021.  How AllSides Creates Balanced News: A Step-by-Step Guide.  Retrieved Nov 30, 2022 from https://www.allsides.com/blog/how-does-allsides-create-balanced-news
  * AllSides (2022) AllSides. 2022.  Balanced News Headlines Roundup.  Retrieved Nov 30, 2022 from https://www.allsides.com/unbiased-balanced-news
  * Bai et al. (2020) Jing Bai, Rui Cao, Wen Ma, and Hiroyuki Shinnou. 2020\.  Construction of Domain-Specific DistilBERT Model by Using Fine-Tuning. In _TAAI_. 237–241.
  * Baly et al. (2020) Ramy Baly, Giovanni Da San Martino, James Glass, and Preslav Nakov. 2020.  We Can Detect Your Bias: Predicting the Political Ideology of News Articles. In _EMNLP_. 4982–4991.
  * Barbaro and Zeller (2006) Michael Barbaro and Tom Zeller. 2006.  A Face is exposed for AOL searcher no. 4417749.  _New York Times_ (01 2006).
  * Belkin et al. (2001) Nicholas Belkin, Colleen Cool, Diane Kelly, S.-J Lin, S.Y Park, Jose Perez-carballo, and Cynthia Sikora. 2001.  Iterative exploration, design and evaluation of support for query reformulation in interactive information retrieval.  _IPM_ 37 (05 2001), 403–434.
  * Bolukbasi et al. (2016) Tolga Bolukbasi, Kai-Wei Chang, James Zou, Venkatesh Saligrama, and Adam Kalai. 2016.  Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings. In _NIPS_. 4356–4364.
  * Bonart et al. (2019) Malte Bonart, Anastasiia Samokhina, Gernot Heisenberg, and Philipp Schaer. 2019.  An investigation of biases in web search engine query suggestions.  _OIR_ 44, 2 (2019), 365–381.
  * Büyüköz et al. (2020) Berfu Büyüköz, Ali Hürriyetoǧlu, and Arzucan Özgür. 2020.  Analyzing ELMo and DistilBERT on Socio-political News Classification. In _AESPEN_.
  * Cai and de Rijke (2016) Fei Cai and Maarten de Rijke. 2016.  A Survey of Query Auto Completion in Information Retrieval.  _FNTIR_ 10, 4 (2016), 273–363.
  * Chen et al. (2020) Wei-Fan Chen, Khalid Al Khatib, Henning Wachsmuth, and Benno Stein. 2020.  Analyzing Political Bias and Unfairness in News Articles at Different Levels of Granularity. In _NLPCSS_. 149–154.  https://doi.org/10.18653/v1/2020.nlpcss-1.16
  * Cohen (1960) Jacob Cohen. 1960\.  A Coefficient of Agreement for Nominal Scales.  _Educ Psychol Meas_ 20, 1 (1960), 37–46.
  * D’Alessio and Allen (2000) Dave D’Alessio and Mike Allen. 2000.  Media Bias in Presidential Elections: A Meta-Analysis.  _J. Commun._ 50, 4 (2000), 133–156.
  * Edelman (2022) Edelman. 2022.  2022 Edelman Trust Barometer.  Retrieved Nov 30, 2022 from https://www.edelman.com/trust/2022-trust-barometer
  * Epstein and Robertson (2015) Robert Epstein and Ronald E. Robertson. 2015.  The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections.  _PNAS_ 112, 33, E4512–E4521.  Publisher: National Academy of Sciences Section: PNAS Plus.
  * Gault (2022) Matthew Gault. 2022\.  AI Trained on 4Chan Becomes ‘Hate Speech Machine’.  Retrieved Feb 28, 2023 from https://www.vice.com/en/article/7k8zwx/ai-trained-on-4chan-becomes-hate-speech-machine
  * Gawronski (2021) Bertram Gawronski. 2021\.  Partisan bias in the identification of fake news.  _TiCS_ 25, 9 (2021), 723–724.
  * Haak and Schaer (2021) Fabian Haak and Philipp Schaer. 2021.  Perception-Aware Bias Detection for Query Suggestions. In _BIAS_. 130–142.
  * Haak and Schaer (2022) Fabian Haak and Philipp Schaer. 2022.  Auditing Search Query Suggestion Bias Through Recursive Algorithm Interrogation. In _WebSci_. 219–227.
  * Hienert et al. (2011) Daniel Hienert, Philipp Schaer, Johann Schaible, and Philipp Mayr. 2011.  A Novel Combined Term Suggestion Service for Domain-Specific Digital Libraries.. In _TPDL_ _(Lecture Notes in Computer Science, Vol. 6966)_ , Stefan Gradmann, Francesca Borri, Carlo Meghini, and Heiko Schuldt (Eds.). Springer, 192–203.  http://dblp.uni-trier.de/db/conf/ercimdl/tpdl2011.html#HienertSSM11
  * Hube and Fetahu (2019) Christoph Hube and Besnik Fetahu. 2019.  Neural Based Statement Classification for Biased Language. In _WSDM_. ACM.
  * Introna and Nissenbaum (2000) L. Introna and H. Nissenbaum. 2000.  Defining the Web: the politics of search engines.  _Computer_ 33, 1, 54–62.
  * Kulshrestha et al. (2019) Juhi Kulshrestha, Motahhare Eslami, Johnnatan Messias, Muhammad Bilal Zafar, Saptarshi Ghosh, Krishna P. Gummadi, and Karrie Karahalios. 2019.  Search bias quantification: investigating political bias in social media and web search.  _Inf. Retr. J._ 22, 1, 188–227.
  * Liu et al. (2021) Ruibo Liu, Chenyan Jia, and Soroush Vosoughi. 2021.  A Transformer-based Framework for Neutralizing and Reversing the Political Polarity of News Articles.  _Proc. ACM Hum.-Comput. Interact._ 5, 1–26.
  * Mitra et al. (2014) Bhaskar Mitra, Milad Shokouhi, Filip Radlinski, and Katja Hofmann. 2014.  On user interactions with query auto-completion. In _SIGIR_. 1055–1058.
  * Mokhberian et al. (2020) Negar Mokhberian, André s Abeliuk, Patrick Cummings, and Kristina Lerman. 2020.  Moral Framing and Ideological Bias of News.  In _SocInfo_. 206–219.
  * Mosbach et al. (2021) Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2021\.  On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines. In _ICLR_.  https://openreview.net/forum?id=nzpLWnVAyah
  * Niu and Kelly (2014) Xi Niu and Diane Kelly. 2014.  The use of query suggestions during information search.  _IPM_ 50, 1, 218–234.
  * Pass et al. (2006) Greg Pass, Abdur Chowdhury, and Cayley Torgeson. 2006\.  A Picture of Search. In _InfoScale_. 1–es.
  * Ray (2020) Lily Ray. 2020.  2020 Google Search Survey: How Much Do Users Trust Their Search Results?  Retrieved Nov 30, 2022 from https://moz.com/blog/2020-google-search-survey
  * Raza et al. (2022) Shaina Raza, Deepak John Reji, and Chen Ding. 2022.  Dbias: Detecting biases and ensuring Fairness in news articles.  _Int J Data Sci Anal_ (2022).
  * Recasens et al. (2013) Marta Recasens, Cristian Danescu-Niculescu-Mizil, and Dan Jurafsky. 2013\.  Linguistic Models for Analyzing and Detecting Biased Language. In _ACL_. 1650–1659.
  * Reddit (2022) Reddit. 2022.  This is the worst AI ever.  Retrieved Feb 28, 2023 from https://www.reddit.com/r/MachineLearning/comments/v42pej/p_this_is_the_worst_ai_ever_gpt4chan_model/
  * Robertson et al. (2018) Ronald E. Robertson, Shan Jiang, Kenneth Joseph, Lisa Friedland, David Lazer, and Christo Wilson. 2018\.  Auditing Partisan Audience Bias within Google Search.  _Proc. ACM Hum.-Comput. Interact._ 2, CSCW, Article 148 (2018).
  * Robertson et al. (2019) Ronald E. Robertson, Shan Jiang, David Lazer, and Christo Wilson. 2019.  Auditing Autocomplete: Suggestion Networks and Recursive Algorithm Interrogation. In _WebSci_. 235–244.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.  _CoRR_ abs/1910.01108 (2019).
  * Sullivan (2018) Danny Sullivan. 2018\.  How Google autocomplete works in Search.  Retrieved Nov 30, 2022 from https://blog.google/products/search/how-google-autocomplete-works-search/
  * Wang et al. (2018) Peng Wang, Xianghang mi, Xiaojing Liao, Xiaofeng Wang, Kan Yuan, Feng Qian, and Raheem Beyah. 2018.  Game of Missuggestions: Semantic Analysis of Search-Autocomplete Manipulations. In _NDSS_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020\.  Transformers: State-of-the-Art Natural Language Processing. In _EMNLP_. 38–45.
  * Xu and Croft (2000) Jinxi Xu and W. Bruce Croft. 2000.  Improving the Effectiveness of Information Retrieval with Local Context Analysis.  _ACM Trans. Inf. Syst._ 18, 1 (Jan. 2000), 79–112.  https://doi.org/10.1145/333135.333138