# Algorithmic Misjudgement in Google Search Results: Evidence from Auditing
the US Online Electoral Information Environment

Brooke Perreault<EMAIL_ADDRESS>,  Johanna Lee<EMAIL_ADDRESS>,
Ropafadzo Shava<EMAIL_ADDRESS>and  Eni Mustafaraj<EMAIL_ADDRESS>
(2024)

###### Abstract.

Google Search is an important way that people seek information about politics
(Dutton et al., 2017), and Google states that it is “committed to providing
timely and authoritative information on Google Search to help voters
understand, navigate, and participate in democratic
processes”.111https://elections.google/civics-in-search/ In this paper, we
interrogate the extent to which government-maintained web domains are
represented in the online environment of electoral information of the 2022 US
midterm elections, as captured through Google Search results in 3.45 million
SERPs for 786 locations across the United States between October and November
2022. Although we find that almost 40% of organic results are contributed by
the 40% of government domains, this proportional equilibrium hides the fact
that most results either belong to a small number of popular domains or are
mistargeted (at a rate of 71.18%) with respect to the location of the search.
We consider the frequent omission and mistargeting of non-federal websites a
form of algorithmic misjudgement that contributes to civic harm, by obscuring
the important role that these institutions play in the election information
environment.

algorithm auditing, problematic behavior, google search audit, elections,
algorithmic misjudgement

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX††conference:
ACM Conference on Fairness, Accountability, and Transparency; June 03–06,
2024; Rio de Janeiro, Brazil††isbn: 978-1-4503-XXXX-X/18/06

##  1\. Introduction

Electoral denialism (the belief that elections are rigged or that electoral
fraud is rampant) in the United States (but also other countries, such as
Brasil and Peru) is emerging as a significant threat to democracy (Cameron,
2023), and it has roots, among others in racial resentment and rising
conspiracism (Stewart III, 2023). Since the early 2000s, internet was regarded
as a medium “uniquely receptive to conspiracism” (James, 2000), but in the
past two decades, there is little doubt that it has contributed significantly
to conspiracism going mainstream (Rosenblum and Muirhead, 2019). Following the
2020 US Presidential Election, “election rigging” conspiracies that came to be
labeled as “The Big Lie” led among others to the violent attack to the US
Capitol on January 6 2021 (Fahey, 2023). While beliefs about electoral fraud
are somewhat persistent (Enders et al., 2021), the 2020 electoral denialism
movement was different because it was embraced by partisan state legislatures,
as reported by the Brennan Center for
Justice.222https://www.brennancenter.org/our-work/research-reports/voting-
laws-roundup-june-2023 As a result, a large number of changes to electoral
laws (some of them restricting and some of them expanding voting rights),
became operational ahead of the 2022 US midterm
election.333https://www.reuters.com/legal/how-new-us-laws-could-trip-up-
voters-midterm-elections-2022-11-01/

In an environment of election denialism on one hand and changing electoral
laws on the other, voters must seek accurate, up-to-date, and relevant
information about elections and voting. A survey by the Bipartisan Policy
Center in Fall 2022 found that voters are “most likely to look to their state
and local election officials, and search engines” for election
information,444https://bipartisanpolicy.org/blog/new-survey-data-election-
information/ aligning with previous research (Dutton et al., 2017). Election
officials similarly position themselves as authorities on election
information. For example, the National Association of Secretaries of State
launched a campaign ahead of the 2020 presidential elections called
#TrustedInfo2020 to “promote election officials as the trusted sources of
election information.” Operating as #TrustedInfo during non-federal election
years and now #TrustedInfo2024 ahead of the 2024 presidential elections, NASS
states, “By driving voters directly to election officials’ websites, social
media pages, and materials, they will be able to receive credible, timely
information on each step of the elections
process.”555https://www.nass.org/initiatives/trustedinfo

Likewise, search engines also position themselves as trustworthy
intermediaries for seeking election information. Google states that it is
“committed to providing timely and authoritative information on Google Search
to help voters understand, navigate, and participate in democratic
processes.”666https://elections.google/civics-in-search/ In an article about
their work to “support” the 2022 US midterm elections, Google’s Vice President
for Trust and Safety writes, “Our work is centered around connecting voters to
the latest election information …”777https://blog.google/outreach-
initiatives/civics/our-ongoing-work-to-support-the-2022-us-midterm-elections/
and highlights “features that show data from nonpartisan organizations” that
work to “connect voters with accurate information about voter registration and
how to vote,” referring to Google Search election information panels that
contain aggregated information from _Democracy Works_ 888Democracy Works
describes itself as “a nonpartisan, nonprofit organization that collaborates
with election officials, leading tech platforms, and world-class partners to
drive voter access and participation”. Source:
https://www.democracy.works/about. about where and how to vote.

Study Motivation: Given this background, three assumptions motivate the
current study: (1) voting-eligible US citizens who search about electoral
information using Google’s search engine have the right to receive accurate
and relevant search results, regardless of where they live, 2) similar to
Ballatore et al. (Andrea Ballatore and Sen, 2017), we acknowledge a spatial
dimension to searching and search results, in which users searching for
information related to elections and voting deserve results relevant to their
geographic area, and 3) given the size of the United States and the varying
voting laws per state (most of them updated ahead of the 2022 elections),
receiving information from authoritative, locality-specific sources is a
reasonable expectation. With these in mind, we proceed to audit Google Search
results to determine the extent to which government-maintained web domains, as
authoritative sources on election information, are represented in the online
environment of electoral information, during the 2022 US midterm elections.
Specifically, we analyze where, how often, and how accurately state, county,
and local government domains appear in Google’s organic results for election-
related queries collected in 786 locations across the United States between
October and November 2022, in a dataset of 3.45 million SERPs. Drawing
parallels to studies of geotargeting errors in advertisements (Bandy and
Hecht, 2021), we label the misalignment between the geographic area that a
non-federal government web domain serves (such as a state or county) and the
location for which it appears in search results as an instance of
mistargeting, in which the search algorithm targets locality-specific
information to a user in a different (incorrect) location. In this paper, we
answer two related research questions:

  * •

RQ1: To what extent are government web domains represented in the online
environment of electoral information about the 2022 US Midterm Elections, as
mediated by Google Search?

  * •

RQ2: To what extent are state, county, and local government web domains
correctly targeted to their respective constituents?

Findings and Contributions: We find that 40% of domains that appear among
organic results are government websites, constituting nearly 40% of all
organic results. However, in the 32.89% of organic results contributed by non-
federal government domains, there is a 71.18% mistargeting rate. Our detailed
analysis indicates a lack of consistency in the search algorithm’s treatment
of locality-specific government sources, leaving most government domains,
especially at the local level, appearing minimally, inconsistently, or missing
entirely from election-related search results. We frame the omission and
frequent mistargeting of non-federal government domains as a form of
algorithmic misjudgement that obscures the important role that these domains
play in providing official, locally-relevant information to voters. To the
best of our knowledge, this is the largest locality-specific Google Search
audit in the context of a political election.

##  2\. Literature Review

###  2.1. Algorithm Auditing and Problematic Behavior

Online platforms and other sociotechnical and algorithmic systems are neither
neutral nor unbiased entities. As Gillespie (Gillespie, 2010) argues, the
presentation of platforms as open or neutral parties obscures the consequences
of deliberate choices platforms make as to what content is published and how
content is shown. Through a systematic literature review, Bandy organizes
potential problematic behavior of algorithmic systems into four main
categories (a) discrimination based on identity or socioeconomic status; (b)
distortion or obscuration of reality, (c) exploitation of sensitive user
information, and (d) misjudgement and inaccurate classifications (Bandy,
2021). Potential consequences of algorithmic systems become more wide-reaching
when considering the interplay between the technical components of algorithmic
systems and the social systems in which they exist; one such theme of
sociotechnical harm is social systems harms, including how algorithmic systems
contribute to the spread of misinformation, disinformation, and malinformation
(informational harms) and how algorithmic systems impact governance,
democracy, and civil liberties (political and civic harms) (Shelby et al.,
2023).

Sandvig et al. (Sandvig et al., 2014) propose algorithm auditing as a way of
studying problematic behavior in Internet platforms and algorithmic systems,
particularly problematic behavior that is subtle or unobservable via one-off
instances. To date, search engines are among the most audited algorithmic
systems, often for discrimination or distortion problematic behavior. Prior
research has focused on personalization of search results, including
personalization based on logged-in status (Hannak et al., 2013), browsing
history (Le et al., 2019; Haim et al., 2018), and location (Kliman-Silver et
al., 2015); composition of search results, including the presence and content
of snippets (Hu et al., 2019), Top stories (Lurie and Mustafaraj, [n. d.];
Trielli and Diakopoulos, 2019), and other non-organic panels (Robertson et
al., 2018b); and prominence of different types of sources among organic
results (Fischer et al., 2020; Courtois et al., 2018; Metaxa et al., 2019).
The pervasive concern in such studies is that the algorithmic choices are
leading to a distorted view of reality. Meanwhile, studied to a lesser extent
is problematic behavior related to algorithmic misjudgement. As Bandy defines
it (Bandy, 2021), misjudgement occurs when an algorithm makes incorrect
predictions or classifications, and misjudgement often leads to other types of
problematic behavior. Algorithmic misjudgement has been studied in the
criminal justice context (eg. recidivism prediction (Duwe and Kim, 2017)), as
well as the advertising context, where algorithms incorrectly infer
information about users (Bandy, 2021). One implication of advertising
algorithm misjudgement is that purchased advertisements do not reach intended
audiences. For example, Bandy and Hecht (Bandy and Hecht, 2021) find that
geotargeting errors are common in Google Display Network, with users often
seeing advertisements not intended for their ZIP code. However, they find that
severity of such geotargeting errors were minor in the context of advertising
goals, with most users living in the intended county, designated market area,
or state (Bandy and Hecht, 2021). With this study, we contribute to the
literature on algorithmic misjudgement in the context of location-sensitive
electoral information.

###  2.2. Politics and Search Engines

Much of previous work on auditing search engines is rooted in a political
context, motivated by the potential informational, political, and civic harms
that could result from biased search. Epstein and Robertson (Epstein and
Robertson, 2015) hypothesized, for example, that biased search results can
impact electoral outcomes by shifting voter preferences based on manipulated
rankings, as studied in controlled, lab-based experiments (Epstein and
Robertson, 2015; Epstein et al., 2017). However, algorithm audits in the wild
find little evidence for this concern (Robertson et al., 2018a, b). Findings
that support the presence of filter bubbles in Google Search and Google News
are also limited (Robertson et al., 2018a; Puschmann, 2019; Courtois et al.,
2018; Haim et al., 2018; Trielli and Diakopoulos, 2022). Trielli and
Diakopoulos (Trielli and Diakopoulos, 2022) in particular find that while
members of different ideological groups use different queries to search for
political information, such partisan queries do not lead to a filter bubble
effect; rather, they observe a “mainstreaming effect”, where highly similar
results appear even for diverging partisan queries. Moreover, Robertson et al.
(Robertson et al., 2023) find that user choice plays a larger role in
engagement with partisan or unreliable news than algorithmic curation, further
evidence against the idea of partisan filter bubbles.

Given the focus on potential bias or filter bubbles, the query phrases used in
most political information audits are intended to capture such phenomena with
respect to either the candidates or issues central to an election. Examples
include queries related to Donald Trump’s inauguration (Robertson et al.,
2018b), candidate names (Metaxa et al., 2019), partisan terms about
immigration (Le et al., 2019), socio-political themes, (Courtois et al.,
2018), and crowd-sourced queries from users about senate candidates (Trielli
and Diakopoulos, 2022). Most of the audits that focus on candidate names are
conducted during election periods, including the 2016 US Congressional
Elections (Metaxas and Pruksachatkun, 2017), the 2018 US Midterms (Trielli and
Diakopoulos, 2022; Metaxa et al., 2019), and 2020 US Presidential Primaries
(Urman et al., 2022; Kawakami et al., 2020). Diakopoulos et al. (Diakopoulos
et al., 2018) and Lurie and Mulligan (Lurie and Mulligan, 2021) similarly
study various information panels that appear in Google search results when
searching for candidate or representative names, with the latter finding that
featured snippets are often likely to mislead searchers by presenting
incorrect or incomplete information about a representative. Lurie and Mulligan
contextualize this behavior as “algorithmic breakdown”, and identify place
name ambiguity as an important contributor to such breakdown, wherein the
algorithmic system is unable to discern a particular location (and the
respective locality-relevant information, like the representative of a
congressional district) from an ambiguous place name, such as a county and
city that share the same name. Diakopoulos et al. (Diakopoulos et al., 2018)
additionally study the composition of organic results for queries of Democrat
and Republican candidates ahead of the 2016 US presidential election, finding
that official sources, such as campaign websites, were clustered at the top of
organic results in high ranking positions. Most of these studies focus on
query phrases selected by researchers in various ways. Mustafaraj et al.
(Mustafaraj et al., 2020) discuss that voters have information needs not
previously captured in such audits. Building upon prior literature, we create
an election-relevant query phrases dataset composed of popular queries from
Google Trends and queries formulated by users as reported in (Mustafaraj et
al., 2020).

###  2.3. Source Geoprovenance

As Ballatore et al. (Andrea Ballatore and Sen, 2017) describe, search engines
“increasingly mediate not just information but also spatial knowledges and
experiences,” as users rely on search engines to inform them about
destinations, where to shop, or other ways to interact with their “lived
everyday geographies” - including participating in civic processes, such as
voting. Moreover, where information originates from, particularly if
information originates from local or non-local sources, can impact how people
learn about and interact with places near them, to the extent that local
community members and non-community members view their locality differently
(Sen et al., 2015). The geographic origin of information, or geoprovenance
(Sen et al., 2015), has largely been studied in places where volunteered
geographic information (e.g. geotagging) is accessible, such as Wikipedia (Sen
et al., 2015) and Twitter (Johnson et al., 2016). Ballatore et al. (Andrea
Ballatore and Sen, 2017) study the extent to which different country variants
of Google Search direct users to locally produced information about places,
the first such audit study of search engines. Using a geoprovenance inference
algorithm developed by Sen et al. (Sen et al., 2015), they identify the
geoprovenance of a URL at the country level and compare URL geoprovenance to
the country of search, finding a greater degree of local content for wealthy
and well-connected countries. This study uses the concept of geoprovenance of
a URL to study algorithmic misjudgement in search results. We audit not only
the extent to which government sources are present in search results, but also
the extent to which state, county, and local government sources are shown to
the geographic areas which they serve.

##  3\. Data Collection

An audit comprises a set of inputs and the corresponding outputs. Our inputs
are the query phrases and the geographical locations where Google searches
were performed. The outputs are the search engine results pages (SERPs)
generated for each query-location pair. We describe how we selected the
queries and locations and how we performed the audit.

###  3.1. Selection of Queries

In line with previous research about centering voters in algorithm audit
methodology (Mustafaraj et al., 2020), we developed a list of queries aimed at
capturing the diverse information needs of voters who use Google to search for
election information.

In early October 2022, we collected Top and Rising related terms from Google
Trends for the seed queries “midterm elections”, “voting near me”, and
“ballot”, as well as from the Google Trends category “United States election,
2022 - General election”. The location of Google Trends was the United States.
From the received results, we excluded terms that were irrelevant to the topic
of elections; terms that included the names of political figures or
candidates; and any terms that included specific states, state abbreviations,
or localities so that queries generalize across locations.

A review of the query list after these steps revealed a lack of general
queries about candidates and key voter issues, as well as a lack of biased
and/or partisan queries. To fill these gaps, we augmented our query list by
manually selecting queries formulated by eligible US voters included in the
Voter Searches dataset in (Mustafaraj et al., 2020). Lastly, a few additional
seed queries were added related to the topic of abortion, which was one of the
top voting issues999https://www.pewresearch.org/politics/2022/08/23/abortion-
rises-in-importance-as-a-voting-issue-driven-by-democrats/ in the 2022
election. These processes resulted in a final list of 161 queries, of which 74
were from Google Trends.101010The query phrases and their provenance
provenance information is available here: https://bit.ly/audit-queries.

Queries were reviewed qualitatively by one author to develop three major
themes of a) Voting Logistics (in which queries are primarily about logistical
information related to voting); b) Candidates and Issues (in which queries are
primarily about candidates, key voting issues, and candidate’s opinions,
stances, and/or leanings) and c) General Election Topics, (where queries are
primarily general inquiries about the midterm election, such as what midterms
are and when they occur). Three labelers then independently labeled all
queries into these three categories. The majority label was used as the final
label per query. Table 1 shows the query distribution by category and some
examples for each category.

Query Category | Example Queries | n
---|---|---
Voting Logistics | voting early near me, sample ballot for my county, what district am i in | 68
Candidates and Issues | who is on the ballot, candidate midterm info, candidates for my district | 50
General Election Topics | 2022 midterm, mid term elections, should i vote, | 43
Table 1. The audit used 161 queries that cover three major themes related to
logistics of voting, candidates and key voting issues, and general inquiries
about midterms. The query categories were assigned by three independent
labelers.

###  3.2. Selection of Locations

We selected locations for data collection to correspond to U.S states and
congressional districts whose House or Senate midterm races were classified as
toss-ups, leaning Democrat, and leaning Republican, according to the Cook
Political Report.111111 https://www.cookpolitical.com/ratings/senate-race-
ratings and https://www.cookpolitical.com/ratings/house-race-ratings We
identified 30 congressional districts with competitive House races across 23
states, as well as 10 states with competitive Senate races; 3 states had only
competitive Senate races, making for a total of 26 unique states.

For house races, because congressional district borders do not align with
county borders, we first manually compiled a list of counties within the
relevant congressional district, according to Wikipedia,121212Eg.
https://en.wikipedia.org/wiki/California's_27th_congressional_district
including only the counties that were described as having most or all of their
localities within the congressional district. Next, we identified the
associated Federal Information Processing System (FIPS) Codes for each
county131313https://transition.fcc.gov/oet/info/maps/census/fips/fips.txt and
used these codes to filter 2020 U.S. Census data141414Dataset titled
“Incorporated Places and Minor Civil Divisions Datasets: Subcounty Resident
Population Estimates: April 1, 2020 to July 1, 2021 (SUB-EST2021)” from
https://www.census.gov/data/tables/time-series/demo/popest/2020s-total-cities-
and-towns.html to include localities from the relevant counties only.
Additionally, we filtered locations to only include cities, towns, and
townships, as identified on the Census.

To sample locations, we identified the median population of possible
locations, and selected (up to) 15 locations below the median population and
(up to) 15 locations above the median population, as well as the location with
the highest population. Some states (for example, Nevada, which had only a
competitive Senate race) had fewer than 31 cities, towns, and townships from
the relevant counties. This process resulted in 872 locations across 26
states. Google Places API was used to get latitude and longitude coordinates
for each of the locations. This information is needed to change the location
of the Chrome browser to the desired location for the audit,151515Location
information is available here: https://bit.ly/audit-locations as shown next.

###  3.3. Scraping Audit and SERP Validation

We use the Python-scripted Selenium web driver to automate the process of
querying Google Search, changing the geolocation, and saving the SERPs. For
each query, our custom script opens a blank-slate of the Chrome browser,
enters the query, and scrolls to the bottom of the page to update the location
using latitude and longitude coordinates (Figure 1). The page reloads,
displaying the results for the new location, and the entire HTML page is
stored.

Figure 1. The script automatically updates the location with latitude and
longitude coordinates so that the search location is “from your device” rather
than IP address. Previous audits (Kliman-Silver et al., 2015) have tested and
confirmed that search results are personalized based on provided coordinates,
rather than the IP address.

SERPs were collected for 30 days, starting on October 18, 2022 and ending
November 18, 2022. Data were not collected on November 14 and 15. Data were
collected using 12 desktop computers, with 2 to 3 states collecting per
computer. Because of the large amount of locations, data collection started
each day at 2am EST and continued until the evening. The order of locations
was not changed daily, so SERPs per query were collected at approximately the
same time within each location, and locations were completed sequentially.
3,838,128 SERPs were collected in total.

To validate that locations were changed correctly, we extracted the text of
the location box on each SERP and compared it to the intended location. The
validation process revealed two types of errors: (1) the location of a SERP
failed to update properly for unknown reasons, and (2) the latitude and
longitude coordinates from the Google Places API did not exactly match the
actual intended location. To resolve these issues, we remove all 86 locations
that had errors and conduct analysis on 3,455,822 SERPs from the remaining 786
locations across 25 states.

##  4\. Methods

Since our research questions revolve around the presence of government-
maintained websites in SERPs, we describe in this section how we extracted
their occurrences; labeled them according to the government type; and defined
when they are mistargeted with respect to the search location.

###  4.1. Extracting domains and election information boxes

SERPs are composed of multiple components that supplement the organic search
results (the blue links) such as top stories, videos, images, the people also
ask panel, tweets, and more. During election seasons, Google includes
election-specific boxes on many SERPs with information about how to vote,
notable dates, or interactive tools to find your voting location. Thus, we
parsed SERPs for two types of information: a) the organic search results
links; and b) the election information boxes.

Organic search links. 22,750 unique links were extracted from 31,310,263
organic results. Once all such links were extracted from all SERPs, they were
cleaned to their root domain; however, given our focus on various government
sources, in some cases we maintain the subdomain and avoid aggregating to the
root, particularly if the subdomain is a county or town website. For example,
while clintonco.illinois.gov is a subdomain of the state government website
illinois.gov, we maintain the county government website. 4,556 unique domains
were identified.

Election Information Boxes. We identified seven types of election boxes in the
SERPs, which were displayed either at the top of the page (with a specific
banner) or on the right side. The content of these boxes came from Democracy
Works (for notable dates, where to vote, how to vote, etc.), Wikipedia
(general information about the election), and Associated Press (election
results).

###  4.2. Identifying Government Websites

To identify government websites, we make use of two repositories maintained by
the US government: (1) a list of nearly 10,000 .gov domains maintained by the
Cybersecurity and Infrastructure Security Agency
(CISA),161616https://github.com/cisagov/dotgov-data/blob/main/current-full.csv
and (2) a list of 9,200 government-maintained websites outside of the .gov and
.mil domains, maintained by the US General Services Administration
(GSA).171717https://github.com/GSA/govt-urls/blob/main/1_govt_urls_full.csv
Both datasets include information about the type of government domain, such as
state, county, or local, the agency that maintains the domain, and some
geographic information about the organization that the domain serves (such as
the state or county). 726 domains were found in the CISA repository and 716
domains from the GSA repository. Because of how links were cleaned and
aggregated, some .gov websites, particularly those corresponding to towns,
cities, or counties, were not included in these repositories because they were
subdomains of state root domains. We manually examined remaining websites
ending with .gov and included any that correspond to a town, city, or county
government, manually filling in relevant information about government type and
geographic area.

Additionally, we identified some county and city websites using the .us domain
that were not included in the GSA repository; because the .us domain is not
strictly governmental, we manually reviewed all websites with the .us domain
and included only those which corresponded to a county or city. 362 domains
were added by manual review. In total, 1,848 websites were identified, which
we call “government domains” or “government sources”. We are making the
information about these domains in our dataset publicly
available.181818https://bit.ly/FAccT24_gov_domains The distribution of domains
across government types is shown in Table 2.

Government Domain Type | Examples (geoprovenance) | n
---|---|---
Federal | whitehouse.gov (United States) | 44
State | az.gov (Arizona) | 160
County | cookcountyclerkil.gov (Cook County, Illinois) | 697
Local | cityofmaywood.com (Maywood, California) | 806
Representative | bishop.house.gov (Georgia, 2nd District) | 137
Senator | portman.senate.gov (Ohio) | 3
Native | nativefederation.org (Alaska) | 1
Table 2. 1,848 government sources were identified from the data set of 4,556
domains (40.6%). Government sources were labeled for type and geoprovenance.
We focus our analysis on state, county, and local government sources, as
senator, representative, and Native sources show up minimally in organic
results.

###  4.3. Quantifying Misjudgement

####  4.3.1. Domain level

Misjudgement can be studied to varying degrees of granularity. First, we
examine misjudgement with respect to government domains and where they appear.
To do so, we first identify how many government domains that appeared in
organic results are from states in which we did not collect data; these are
domains which are always mistargeted. Next, for domains originating in states
for which we collected data, we calculate the proportion of times the domain
appeared in the appropriate state and the proportion of times the domain was
mistargeted (at the state-level). This gives insight into whether there are
some locality-specific government domains that Google shows often across
locations and are frequently mistargeted.

####  4.3.2. Aggregate Organic Results

We also study misjudgement by looking at all organic results in aggregate, and
parsing differences in misjudgement by query categories, domain type, and
state. Every organic result is labeled as to whether it is a government
domain, and if so, we compare the domain’s geoprovenance (the geographic area
it serves, such as a state, county, or city) to the search location. This
gives us a more precise understanding of mistargeting than at the domain
level, in which we only compare states. Organic results are labeled as either
correct or mistargeted. Mistargeted results for county and city domains are
also labeled as either in-state, if they serve a different location in the
same state as the search location, or out-of-state, if they do not, which
functions as an indicator for the severity of mistargeting. For each query
category and state, as well as overall, we calculate the proportion of organic
results composed of all non-federal government sources and the proportion of
results composed of correctly targeted government results. Mistargeting rate
is calculated by subtracting the proportion of correctly targeted from the
total proportion, and dividing by the total proportion. We similarly calculate
the proportion of results that are mistargeted but in-state, which gives
insight into the nature of mistargeted results.

####  4.3.3. SERP Level

Lastly, we quantify misjudgement at the SERP level by calculating descriptive
statistics about the number of government sources that appear per SERP, as
well as descriptive statistics about the rank of correctly and incorrectly
targeted government results. Since users heavily rely on ranking of search
results (Pan et al., 2007), this informs the severity of misjudgement with
respect to a user.

##  5\. Results

###  5.1. RQ1: To what extent are government sources represented in the online
environment of electoral information?

Of the 4,556 domains that appeared in organic results, 1,848 were identified
as a government domain (40.6%). Overall, links belonging to these 1,848
government domains compose 39.97% of all organic results captured in the 3.45
million SERPs (31.3 million organic results) analyzed. Non-federal government
domains compose 32.89% of all organic results. However, the extent to which
government domains appear depends on the type of query, as well as the type of
government source as shown in Table 3. In particular, government sources
constitute 60.55% of organic results for Voting Logistics queries, with state
government sources composing half of such appearances. In contrast, government
sources are underrepresented among organic results for General Election Topics
and Candidates and Issues queries, composing just 26.36% and 23.15% of organic
results, respectively.

| Voting Logistics | General Election Topics | Candidates and Issues | All Queries
---|---|---|---|---
% Federal | 4.85 | 9.1 | 8.44 | 7.08
% State | 30.06 | 11.99 | 11.87 | 19.67
% County | 22.91 | 4.75 | 2.66 | 11.86
% Local | 2.62 | 0.52 | 0.18 | 1.31
% Total excl. Federal | 55.7 | 17.26 | 14.71 | 32.89
% Total | 60.55 | 26.36 | 23.15 | 39.97
Table 3. Government domains compose 39.97% of organic results overall, with
state and county government domains constituting the majority of government
domain appearances, while local government domains are minimally represented
in organic results. The degree to which government domains appear depends
largely on the query type: government sources compose 60% of organic results
for Voting Logistics queries but only around a quarter for General Election
Topics and Candidates and Issues.

Across all queries, local government sources are minimally represented among
organic results. The prominence of state-level government sources compared to
other government sources can be explained by the key roles that state-level
actors play in elections: while the composition of election administrations
vary by state, elections in each state are overseen by a chief election
officer (such as a secretary of state, a governor, or a board), and their
responsibilities include enforcing election laws, administering a statewide
voter registration database, and other duties related to election
logistics.191919https://www.ncsl.org/elections-and-campaigns/election-
administration-at-state-and-local-levels However, their dominance is striking.
While state government websites make up 3.5% of all websites (160 out of
4,556), they are shown in 30.06% of Voting Logistics results and in 19.67% of
overall results.

On a SERP level, Voting Logistics queries had a median of 6 government
results, with medians of 3 state-level and 2 county-level government sources,
aligning with the aggregate proportions in Table3. In contrast, General
Election Topics queries had a median of 2 government source per page and
Candidates and Issues queries had a median of 1. The fact that the median of
state-level websites per SERP is 3 for Voting Logistics queries naturally
raises the question of whether all these three websites are targeted correctly
to the search location.

###  5.2. RQ2: To what extent are state, county, and local government web
domains correctly targeted to their respective constituents?

####  5.2.1. Domains

At the domain level, 174 of the 1,804 (9.6%) non-federal government sources
represent a state for which we did not collect data; in other words, these
sources were always mistargeted. This includes 71 local-level, 53 county-level
and 48 state-level government domains, along with 2 senator websites.
Appearances from these domains constitute 13.65% of appearances of non-federal
government sources, or about 4.4% of all organic results. The remaining 1,630
non-federal government sources match, at the state level, the states for which
we collected SERPs. 1,268 of these domains always show up in the appropriate
state (although not necessarily in the appropriate county or locality, for
county and local sources). While this is the majority of non-federal
government domains (70.3%), only 4.96% of all organic appearances by non-
federal government sources can be attributed to these domains.

Domain | Type |  | State
---
Domain Serves
| % Appearances
---
Correct State
| % Appearances
---
Other States
| Collected data
---
in state?
pa.gov | State | Pennsylvania | 55.25 | 44.75 | Yes
ca.gov | State | California | 61.98 | 38.02 | Yes
sos.state.tx.us | State | Texas | 18.86 | 81.14 | Yes
wi.gov | State | Wisconsin | 26.30 | 73.69 | Yes
wakegov.com | County | North Carolina | 5.78 | 94.52 | Yes
lavote.gov | County | California | 25.79 | 74.21 | Yes
ncsbe.gov | State | North Carolina | 34.45 | 65.55 | Yes
georgia.gov | State | Georgia | 28.28 | 71.71 | Yes
maryland.gov | State | Maryland | 0 | 100.00 | No
votebrevard.gov | County | Florida | 5.83 | 94.17 | Yes
Table 4. These 10 domains constitute 35.96% of all non-federal government
organic results, which is 11.8% of all organic results. While collecting from
more locations in certain states (such as Pennsylvania and California)
increases the number of times these domains occur, all of these domains appear
often in states for which they do not serve, indicating frequent mistargeting.
maryland.gov is one of 174 non-federal sources that appear despite no data
being collected for its respective state.

Table 4 shows the 10 non-federal government domains which make up the highest
proportion of organic results, and together compose 35.96% of appearances for
non-federal domains, or 11.8% of organic results overall. Because the number
of locations for which data was collected varied per state, it is not
appropriate to directly compare the frequency with which these domains occur
to each other. However, it is clear that these domains suffer from high rates
of mistargeting, with many appearing more often in other states than the
states for which they serve. Notably, wakegov.com, a county government site
for Wake County, North Carolina, and votebrevard.gov, the Supervisor of
Elections site for Brevard County, Florida, are mistargeted over 94% of the
time. Overall, 84 domains were mistargeted more often than they were correctly
targeted, and they compose 44% of non-federal organic results (14.45% of all
organic results). In contrast, 245 domains are correctly targeted more often
than they are mistargeted, and they compose 41% of non-federal organic results
(13.5% of all organic results).

Thus, while the majority of non-federal government sources are always
correctly targeted at the state-level, these sources make up only a minority
of all organic results; a relatively small subset of domains are mistargeted
more often than they are correctly targeted, and these mistargeted results
show frequently among organic results.

####  5.2.2. Misjudgement Levels by Query Category and State

Table 5 shows the percent of all organic results that are correctly targeted
to the appropriate locality, and Table 6 shows the mistargeting rates for each
state, county, and local government sources for different query categories.
71% of all appearances of locality-specific government websites in organic
results were mistargeted, with the proportion of mistargeting highest for
Candidates and Issues queries. Voting Logistics queries, which see the highest
proportion of government results by far, have the lowest mistargeting rate of
the 3 query categories, albeit still 69.4%.

| Voting Logistics | General Election Topics | Candidates and Issues | All Queries
---|---|---|---|---
% State Correct | 14.5 | 4.1 | 3.0 | 8.2
% County Correct | 1.89 | 0.42 | 0.11 | 0.96
% Local Correct | 0.63 | 0.13 | 0.03 | 0.31
% Total Correct | 17.02 | 4.65 | 3.14 | 9.47
Table 5. Percent of all organic results that are correctly targeted to the appropriate locality for state, county, and local sources. While 32.89% of all organic results can be attributed to locality-specific government sources, only 9.47% of organic results are composed of correctly targeted government sources. | Voting Logistics | General Election Topics | Candidates and Issues | All Queries
---|---|---|---|---
% State Mistargeted | 51.75 | 65.8 | 74.73 | 58.29
% County Mistargeted | 91.73 | 91.12 | 95.7 | 91.94
% Local Mistargeted | 75.92 | 75.06 | 81.79 | 76.08
% Total Mistargeted | 69.4 | 73.05 | 78.61 | 71.18
Table 6. Proportion of government results that were mistargeted; the
denominator here is the total appearances of a source type, rather than all
organic results. Overall, 71% of locality-specific government organic results
were mistargeted, with the mistargeting rate being highest for county
government sources at nearly 92%.

Across all queries, county-level government sources suffer from the highest
levels of mistargeting, with 91.9% of these results mistargted. State-level
government domains, which appear the most among organic results, have the
lowest mistargeting rate of the locality-specific sources at about 58.3%.
Mistargeting of state-level government sources is lowest for Voting Logistics
queries at 51.75%, notable since state-level sources are the most common
government source in this query category. Lastly, local government sources
suffer from a mistargeting rate of 76%, unfortunate considering the minimal
appearances of local government sources in organic results.

There is some variation in the extent to which government domains appear in
different states, as well as mistargeting rates by state, as visualized in
Figure 2. Studying this variation is left to future work.

Figure 2. Variation in the extent to which government sources appear in
different states. much of the mistargeting for local government sources (and
county government sources, to a lesser extent) can be attributed to other
government sources within the respective state, indicating a lack of precision
with targeting locality-specific government domains to the appropriate
audience when relying only on geocoordinates to represent user location.

####  5.2.3. Mistargeting Severity

While mistargeting rates are high, particularly for county and local
government domains, it is useful to distinguish between error rate and error
severity. Error severity can be assessed in two ways: 1) the rank of
mistargeted organic results, 2) how close mistargeted results are to the
correct location.

With respect to rank, while only 28.8% of non-federal government sources are
correctly targeted, correctly targeted results generally appear in higher
ranks among organic results than mistargeted results (Figure 3). The median
rank of a correctly-targeted government result was 3, while the median rank of
a mistargeted government result was 7. Moreover, for the 43.7% of SERPs that
contained both correctly-targeted and mistargeted government results, the
first correctly-targeted result appeared above the first mistargeted result
83.9% of the time. A mistargeted government result appeared in the first
organic position for only 3.3% of SERPs analyzed, while a correctly-targeted
government result appeared in the first organic position for 19.3% of SERPs.
Given users tend to focus on search results in the top positions (Pan et al.,
2007), these findings suggest that while mistargeting rates are high, their
severity, within the context of impact on a user, may be low. Additionally,
these findings may indicate that the search algorithm prioritizes non-federal
government sources in ranked results only when there is a high likelihood that
the source geoprovenance matches the user’s locations; otherwise, non-federal
government sources appear in lower ranked positions, and other content (such
as a news or non-profit organization) is given the top ranks.

Figure 3. While only 28.8% of non-federal government domains are correctly
targeted, these results usually appear in higher ranks among organic results.
On the 43.7% of SERPs that have both correctly-targeted and mistargeted
results, the correctly targeted result appeared above the mistargeted result
83.9% of the time.

With respect to the nature of mistargeted results, a high percentage of
mistargeting for local government sources can be attributed to local
government sources for places in the same state as the search location. While
the overall mistargeting rate for local government sources is 76%, 64% of
local government results are for other localities in the correct state. Figure
2 visualizes this phenomenon; in most states, nearly all of the mistargeted
local government results are from other local government sources in the
respective state. A similar phenomenon with targeting county-level government
sources is observed, though to a lesser degree and with more variation between
states. These findings imply a lack of precision with targeting locality-
specific government sources to the appropriate audience when using general
queries that do not specify location information. While a similar lack of
precision has been observed with geotargeting of Ads on Google (Bandy and
Hecht, 2021), the consequences of this lack of precision is greater in the
context of “low information
voters”202020https://en.wikipedia.org/wiki/Low_information_voter searching for
election information as it relates to their locale, given that other county or
locality information is likely not be applicable to them.

Lastly, while we find no correlation between the number of local governments
and the percent that local government sources compose in organic results, we
find a moderate, positive correlation between the number of local governments
in a state and the mistargeting rate for local government sources (r=0.36),
and for the correlation between number of local governments and the rate of
in-state mistargeting (r=0.37), suggesting to some degree that a greater
number of local governments is associated with less precise targeting of local
government sources in organic search.

###  5.3. Presence of Election Information Boxes

Between Oct 18-24, only 39 queries showed an electoral box (across most
locations), though this number jumped to 73 on Oct 25 (two weeks before
election) and went up to 83 on election day. Notably, on election day, all
locations had at least one election box. Once the elections were over, the
number of queries with an election box went back in the 30s, but remained
consistent across most locations.

These results indicate that the presence of election information boxes is not
location-specific, but depends on the date and query phrase. Only 86 out of
161 query phrases (53%) had at least one box on at least one date.
Furthermore, 66% of these queries (57 out of 86) belonged to the list of
queries we collected from Google Trends. These queries made up 77% of all
queries from that source (57 out of 74). This suggests that Google might be
showing election information boxes for popular query phrases of a certain
type. However, a more detailed analysis (with a more expansive set of queries)
might be needed to understand what triggers these boxes.

##  6\. Discussion and Future Work

###  6.1. How Algorithmic Misjudgement Shapes the Electoral Information
Environment

The results of our first research question reveal that about 40.6% of all
websites are government sources and that combined they contribute 39.97% of
organic results. This might look like an excellent outcome, given the almost
perfect matching between the proportion of domains and the proportion of their
occurrences in the search results. However, our deeper analysis of the results
reveals several issues that are hidden, when one only considers proportions.

For one, the top ten non-federal government sources show up disproportionally
often, contributing to 11.8% of all organic results (see Table 4). As a
reference point, all 1,848 non-federal sources combined make up only 32.89% of
all results (i.e., about 1/3 of their combined results are accounted for by
these ten domains). This phenomenon is reminiscent of source concentration and
mainstreaming effects found in previous work (Trielli and Diakopoulos, 2022,
2019) for news sources. While it might not be harmful to an individual to read
national news from New York Times more than from Baltimore Sun (although,
collectively, source concentration has led to the so-called ’news desert’
phenomenon (Abernathy, 2018)), for elections, all out-of-state web domains
might be irrelevant to a voter, when it comes to their voting information. Our
results make it clear that Google Search is treating electoral information
needs as any other type of information and is populating the organic search
results with popular websites, instead of locally-relevant websites.

Further, while correctly-targeted government sources are usually ranked higher
than those that are mistargeted (Figure 3), indicating that Google is capable
of getting things right (19.3% of SERPs have a correctly targeted government
website at the first position, mostly a state government), 71.19% of
government website occurrences were mistargeted. Overall, our analysis
indicates a lack of consistency in the search algorithm’s treatment of
locality-specific government sources, leaving most government domains,
especially those of local governments, appearing minimally, inconsistently, or
missing entirely from election-related search results. We consider the case of
mistargeted websites and that of the absence of locality-specific websites as
instances of algorithmic misjudgement that need to be corrected.

Google Search is aware of the importance of authoritative information about
elections, as indicated by the consistent inclusion of “election information
panels” in prominent SERP positions across locations. Google should consider
that showing locality-specific web domains in the organic results might
contribute to strengthening the civic infrastructure of the election by
increasing the visibility of information from elected officials.

###  6.2. Limitations

There are limitations to our study. First, we focus exclusively on government
web domains in search results, but in reality there are other types of
information that users encounter in search results, including results from
other types of sources and curated information panels. Additionally, our audit
was conducted when there were around 10 organic results per SERP, while since
December 2022 (after our data collection), Google has moved to “continous
scrolling”, which loads up to 60 results at
once.212121https://www.theverge.com/2022/12/6/23495973/google-search-desktop-
continuous-scrolling Second, we rely on geocoordinates to serve as a proxy for
locations of real users and do not study how including location information in
the search query (eg. “voting near me Boston, MA”) impacts the composition and
mistargeting of government search results. While the queries come from Google
Trends and real users, we do not study how users actually interact with
government sources in search results. Moreover, differences in popularity of
search terms between those that come from Google Trends and those from real
users may impact the composition of search results (i.e. more popular queries
may have better results).Third, we analyze data from 786 locations across 25
states, a size which is sufficient for statistical analysis but does not
capture all of the United States. Finally, this study only focuses on the
United States, given that electoral laws and election authorities are
different state by state, making this kind of analysis meaningful. In other
countries, where there is a single election authority, the needs and
assumptions might be different.

###  6.3. Future work

One challenge that we encountered when analyzing our results was finding
accurate information about government websites (as discussed in Methods).
However, government websites were only 40.6% of the entire dataset of
websites. The rest was composed of news organizations, non-profit civic
organizations, companies, universities, individual blogs, and more. There are
no high-quality and freely available databases that provide detailed
information on web domains (their nature and what audiences they serve).
Nuanced audits of search engine results need such information; therefore,
methods to create and maintain such databases need to be developed. (We tried
to use some news related curated lists for other studies, such as (Fischer et
al., 2020), but their overlap with our dataset was not high.)

A scraping audit is useful for controlling the inputs to the algorithmic
system, so that we can systematically compare results, but it does not involve
real voters with real information needs, that is, it lacks ecological
validity. Currently, there are efforts underway like the National Internet
Observatory222222https://nationalinternetobservatory.org/ that will enable
researchers to carry out crowdsourced audits, similar to what was used in
(Robertson et al., 2023). Future audits can then address both issues that are
central to our audit: what queries do users formulate (and reformulate) to
find information about elections and with what results do they engage? For
example, do they click on Google’s election boxes, do they click on their own
local/county/state websites, or do they visit non-governmental sources?
Additionally, such audits might allow us to find out if Google Search is
bypassed entirely and users go directly to their trusted sources.

###  6.4. Ethical Considerations

Our study does not include any personal data, as Google Search result pages
are public data. Google handles circa 8.5 billion of searches per
day,232323https://blog.hubspot.com/marketing/google-search-statistics so our
scraping audit of 140 K daily searches provided a minimal burden. Finally, a
US Federal Court has ruled that audits such as ours are legal and do not
violate the Computer Fraud and Abuse Act.242424https://www.aclu.org/press-
releases/federal-court-rules-big-data-discrimination-studies-do-not-violate-
federal-anti

###### Acknowledgements.

We are grateful to members of our lab and funding from XXX.

## References

  * (1)
  * Abernathy (2018) Penelope Muse Abernathy. 2018.  _The expanding news desert_.  Center for Innovation and Sustainability in Local Media, School of Media and ….
  * Andrea Ballatore and Sen (2017) Mark Graham Andrea Ballatore and Shilad Sen. 2017.  Digital Hegemonies: The Localness of Search Engine Results.  _Annals of the American Association of Geographers_ 107, 5 (2017), 1194–1215.  https://doi.org/10.1080/24694452.2017.1308240 arXiv:https://doi.org/10.1080/24694452.2017.1308240
  * Bandy (2021) Jack Bandy. 2021.  Problematic Machine Behavior: A Systematic Literature Review of Algorithm Audits.  _Proc. ACM Hum.-Comput. Interact._ 5, CSCW1, Article 74 (apr 2021), 34 pages.  https://doi.org/10.1145/3449148
  * Bandy and Hecht (2021) Jack Bandy and Brent Hecht. 2021.  Errors in Geotargeted Display Advertising: Good News for Local Journalism?  _Proc. ACM Hum.-Comput. Interact._ 5, CSCW1, Article 92 (apr 2021), 19 pages.  https://doi.org/10.1145/3449166
  * Cameron (2023) Maxwell A Cameron. 2023.  Electoral Denialism in American Democracies.  (2023).
  * Courtois et al. (2018) Cédric Courtois, Laura Slechten, and Lennert Coenen. 2018.  Challenging Google Search filter bubbles in social and political information: Disconforming evidence from a digital methods case study.  _Telematics and Informatics_ 35, 7 (2018), 2006–2015.  https://doi.org/10.1016/j.tele.2018.07.004
  * Diakopoulos et al. (2018) Nicholas Diakopoulos, Daniel Trielli, Jennifer Stark, and Sean Mussenden. 2018.  I vote for—how search informs our choice of candidate.  _Digital Dominance: The Power of Google, Amazon, Facebook, and Apple, M. Moore and D. Tambini (Eds.)_ 22 (2018).
  * Dutton et al. (2017) William Dutton, Bianca Reisdorf, Elizabeth Dubois, and Grant Blank. 2017.  Search and Politics: The Uses and Impacts of Search in Britain, France, Germany, Italy, Poland, Spain, and the United States.  _SSRN Electronic Journal_ (01 2017).  https://doi.org/10.2139/ssrn.2960697
  * Duwe and Kim (2017) Grant Duwe and KiDeuk Kim. 2017.  Out With the Old and in With the New? An Empirical Comparison of Supervised Learning Algorithms to Predict Recidivism.  _Criminal Justice Policy Review_ 28, 6 (2017), 570–600.  https://doi.org/10.1177/0887403415604899 arXiv:https://doi.org/10.1177/0887403415604899
  * Enders et al. (2021) Adam M Enders, Joseph E Uscinski, Casey A Klofstad, Kamal Premaratne, Michelle I Seelig, Stefan Wuchty, Manohar N Murthi, and John R Funchion. 2021.  The 2020 presidential election and beliefs about fraud: Continuity or change?  _Electoral studies_ 72 (2021), 102366.
  * Epstein and Robertson (2015) Robert Epstein and Ronald E. Robertson. 2015.  The search engine manipulation effect (SEME) and its possible impact on the outcomes of elections.  _Proceedings of the National Academy of Sciences_ 112, 33 (2015), E4512–E4521.  https://doi.org/10.1073/pnas.1419828112 arXiv:https://www.pnas.org/doi/pdf/10.1073/pnas.1419828112
  * Epstein et al. (2017) Robert Epstein, Ronald E. Robertson, David Lazer, and Christo Wilson. 2017.  Suppressing the Search Engine Manipulation Effect (SEME).  _Proc. ACM Hum.-Comput. Interact._ 1, CSCW, Article 42 (dec 2017), 22 pages.  https://doi.org/10.1145/3134677
  * Fahey (2023) James J Fahey. 2023.  The big lie: Expressive responding and misperceptions in the United States.  _Journal of Experimental Political Science_ 10, 2 (2023), 267–278.
  * Fischer et al. (2020) Sean Fischer, Kokil Jaidka, and Yphtach Lelkes. 2020.  Auditing local news presence on Google News.  _Nature human behaviour_ 4, 12 (December 2020), 1236—1244.  https://doi.org/10.1038/s41562-020-00954-0
  * Gillespie (2010) Tarleton Gillespie. 2010.  The politics of ‘platforms’.  _New Media & Society_ 12, 3 (2010), 347–364.  https://doi.org/10.1177/1461444809342738 arXiv:https://doi.org/10.1177/1461444809342738
  * Haim et al. (2018) Mario Haim, Andreas Graefe, and Hans-Bernd Brosius. 2018.  Burst of the Filter Bubble?: Effects of personalization on the diversity of Google News.  _Digital Journalism_ 6 (03 2018), 330–343.  https://doi.org/10.1080/21670811.2017.1338145
  * Hannak et al. (2013) Aniko Hannak, Piotr Sapiezynski, Arash Molavi Kakhki, Balachander Krishnamurthy, David Lazer, Alan Mislove, and Christo Wilson. 2013.  Measuring Personalization of Web Search. In _Proceedings of the 22nd International Conference on World Wide Web_ (Rio de Janeiro, Brazil) _(WWW ’13)_. Association for Computing Machinery, New York, NY, USA, 527–538.  https://doi.org/10.1145/2488388.2488435
  * Hu et al. (2019) Desheng Hu, Shan Jiang, Ronald E. Robertson, and Christo Wilson. 2019.  Auditing the Partisanship of Google Search Snippets. In _The World Wide Web Conference_ (San Francisco, CA, USA) _(WWW ’19)_. Association for Computing Machinery, New York, NY, USA, 693–704.  https://doi.org/10.1145/3308558.3313654
  * James (2000) Nigel James. 2000.  Militias, the Patriot movement, and the internet: the ideology of conspiracism.  _The Sociological Review_ 48, 2_suppl (2000), 63–92.
  * Johnson et al. (2016) Isaac L. Johnson, Subhasree Sengupta, Johannes Schöning, and Brent Hecht. 2016.  The Geography and Importance of Localness in Geotagged Social Media. In _Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems_ (San Jose, California, USA) _(CHI ’16)_. Association for Computing Machinery, New York, NY, USA, 515–526.  https://doi.org/10.1145/2858036.2858122
  * Kawakami et al. (2020) Anna Kawakami, Khonzodakhon Umarova, and Eni Mustafaraj. 2020.  The Media Coverage of the 2020 US Presidential Election Candidates through the Lens of Google’s Top Stories. In _Proceedings of the International AAAI Conference on Web and Social Media_ , Vol. 14. 868–877.
  * Kliman-Silver et al. (2015) Chloe Kliman-Silver, Aniko Hannak, David Lazer, Christo Wilson, and Alan Mislove. 2015.  Location, Location, Location: The Impact of Geolocation on Web Search Personalization. In _Proceedings of the 2015 Internet Measurement Conference_ (Tokyo, Japan) _(IMC ’15)_. Association for Computing Machinery, New York, NY, USA, 121–127.  https://doi.org/10.1145/2815675.2815714
  * Le et al. (2019) Huyen Le, Raven Maragh, Brian Ekdale, Andrew High, Timothy Havens, and Zubair Shafiq. 2019.  Measuring Political Personalization of Google News Search. In _The World Wide Web Conference_ (San Francisco, CA, USA) _(WWW ’19)_. Association for Computing Machinery, New York, NY, USA, 2957–2963.  https://doi.org/10.1145/3308558.3313682
  * Lurie and Mulligan (2021) Emma Lurie and Deirdre K. Mulligan. 2021.  Searching for Representation: A sociotechnical audit of googling for members of U.S. Congress.  _CoRR_ abs/2109.07012 (2021).  arXiv:2109.07012 https://arxiv.org/abs/2109.07012
  * Lurie and Mustafaraj ([n. d.]) Emma Lurie and Eni Mustafaraj. [n. d.].  Opening Up the Black Box: Auditing Google’s Top Stories Algorithm.  _Proceedings of the … International Florida Artificial Intelligence Research Society Conference_ 32 ([n. d.]).  https://par.nsf.gov/biblio/10101277
  * Metaxa et al. (2019) Danaë Metaxa, Joon Sung Park, James A. Landay, and Jeff Hancock. 2019.  Search Media and Elections: A Longitudinal Investigation of Political Search Results.  _Proc. ACM Hum.-Comput. Interact._ 3, CSCW, Article 129 (nov 2019), 17 pages.  https://doi.org/10.1145/3359231
  * Metaxas and Pruksachatkun (2017) Panagiotis Takis Metaxas and Yada Pruksachatkun. 2017.  Manipulation of Search Engine Results during the 2016 US Congressional Elections.  https://api.semanticscholar.org/CorpusID:11867830
  * Mustafaraj et al. (2020) Eni Mustafaraj, Emma Lurie, and Claire Devine. 2020.  The Case for Voter-Centered Audits of Search Engines during Political Elections. In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ (Barcelona, Spain) _(FAT* ’20)_. Association for Computing Machinery, New York, NY, USA, 559–569.  https://doi.org/10.1145/3351095.3372835
  * Pan et al. (2007) Bing Pan, Helene Hembrooke, Thorsten Joachims, Lori Lorigo, Geri Gay, and Laura Granka. 2007.  In Google We Trust: Users’ Decisions on Rank, Position, and Relevance.  _Journal of Computer-Mediated Communication_ 12, 3 (2007), 801–823.  https://doi.org/10.1111/j.1083-6101.2007.00351.x arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1083-6101.2007.00351.x
  * Puschmann (2019) Cornelius Puschmann. 2019.  Beyond the Bubble: Assessing the Diversity of Political Search Results.  _Digital Journalism_ 7, 6 (2019), 824–843.  https://doi.org/10.1080/21670811.2018.1539626 arXiv:https://doi.org/10.1080/21670811.2018.1539626
  * Robertson et al. (2023) Ronald Robertson, Jon Green, Damian Ruck, Katherine Ognyanova, Christo Wilson, and David Lazer. 2023.  Users choose to engage with more partisan news than they are exposed to on Google Search.  _Nature_ 618 (05 2023), 1–7.  https://doi.org/10.1038/s41586-023-06078-5
  * Robertson et al. (2018a) Ronald E. Robertson, Shan Jiang, Kenneth Joseph, Lisa Friedland, David Lazer, and Christo Wilson. 2018a.  Auditing Partisan Audience Bias within Google Search.  _Proc. ACM Hum.-Comput. Interact._ 2, CSCW, Article 148 (nov 2018), 22 pages.  https://doi.org/10.1145/3274417
  * Robertson et al. (2018b) Ronald E. Robertson, David Lazer, and Christo Wilson. 2018b.  Auditing the Personalization and Composition of Politically-Related Search Engine Results Pages. In _Proceedings of the 2018 World Wide Web Conference_ (Lyon, France) _(WWW ’18)_. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 955–965.  https://doi.org/10.1145/3178876.3186143
  * Rosenblum and Muirhead (2019) Nancy L Rosenblum and Russell Muirhead. 2019.  _A lot of people are saying: The new conspiracism and the assault on democracy_.  Princeton University Press.
  * Sandvig et al. (2014) Christian Sandvig, Kevin Hamilton, Karrie Karahalios, and Cédric Langbort. 2014.  Auditing Algorithms : Research Methods for Detecting Discrimination on Internet Platforms.  https://api.semanticscholar.org/CorpusID:15686114
  * Sen et al. (2015) Shilad W. Sen, Heather Ford, David R. Musicant, Mark Graham, Os Keyes, and Brent Hecht. 2015.  Barriers to the Localness of Volunteered Geographic Information. In _Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems_ (Seoul, Republic of Korea) _(CHI ’15)_. Association for Computing Machinery, New York, NY, USA, 197–206.  https://doi.org/10.1145/2702123.2702170
  * Shelby et al. (2023) Renee Shelby, Shalaleh Rismani, Kathryn Henne, AJung Moon, Negar Rostamzadeh, Paul Nicholas, N’Mah Yilla-Akbari, Jess Gallegos, Andrew Smart, Emilio Garcia, and Gurleen Virk. 2023.  Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction. In _Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society_ (Montreal, Canada) _(AIES ’23)_. ACM, New York, NY, USA, 723–741.  https://doi.org/10.1145/3600211.3604673
  * Stewart III (2023) Charles Stewart III. 2023.  Public Opinion Roots of Election Denialism.  _SSRN_ (2023).  http://dx.doi.org/10.2139/ssrn.4318153
  * Trielli and Diakopoulos (2019) Daniel Trielli and Nicholas Diakopoulos. 2019.  Search as News Curator: The Role of Google in Shaping Attention to News Information. In _Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems_ (Glasgow, Scotland Uk) _(CHI ’19)_. Association for Computing Machinery, New York, NY, USA, 1–15.  https://doi.org/10.1145/3290605.3300683
  * Trielli and Diakopoulos (2022) Daniel Trielli and Nicholas Diakopoulos. 2022.  Partisan search behavior and Google results in the 2018 U.S. midterm elections.  _Information, Communication & Society_ 25, 1 (2022), 145–161.  https://doi.org/10.1080/1369118X.2020.1764605 arXiv:https://doi.org/10.1080/1369118X.2020.1764605
  * Urman et al. (2022) Aleksandra Urman, Mykola Makhortykh, and Roberto Ulloa. 2022.  The Matter of Chance: Auditing Web Search Results Related to the 2020 U.S. Presidential Primary Elections Across Six Search Engines.  _Social Science Computer Review_ 40, 5 (2022), 1323–1339.  https://doi.org/10.1177/08944393211006863 arXiv:https://doi.org/10.1177/08944393211006863