# Quantifying Geospatial in the Common Crawl Corpus

Ilya Ilyankou<EMAIL_ADDRESS>0009-0008-7082-7122 ,  Meihui Wang
<EMAIL_ADDRESS>0000-0001-8420-2141 ,  James Haworth
<EMAIL_ADDRESS>0000-0001-9506-4266 UCL SpaceTimeLabLondonUK and  Stefano
Cavazzi<EMAIL_ADDRESS>0000-0003-3575-0365 Ordnance
SurveySouthamptonUK

(2024; 31 May 2024)

###### Abstract.

Large language models (LLMs) exhibit emerging geospatial capabilities,
stemming from their pre-training on vast unlabelled text datasets that are
often derived from the Common Crawl corpus. However, the geospatial content
within CC remains largely unexplored, impacting our understanding of LLMs’
spatial reasoning. This paper investigates the prevalence of geospatial data
in recent Common Crawl releases using Gemini, a powerful language model. By
analyzing a sample of documents and manually revising the results, we estimate
that between 1 in 5 and 1 in 6 documents contain geospatial information such
as coordinates and street addresses. Our findings provide quantitative
insights into the nature and extent of geospatial data within Common Crawl,
and web crawl data in general. Furthermore, we formulate questions to guide
future investigations into the geospatial content of available web crawl
datasets and its influence on LLMs.

Common Crawl, geospatial

††copyright: acmlicensed††journalyear: 2024††doi: XXXXXXX.XXXXXXX††isbn:
978-1-4503-XXXX-X/18/06††ccs: Information systems Geographic information
systems††ccs: Information systems Web mining††ccs: Computing methodologies
Natural language processing

##  1\. Introduction

Recent studies have found that large language models (LLMs) may have some
knowledge and understanding of physical space. Gurnee and Tegmark (Gurnee and
Tegmark, 2024) demonstrated that Llama-2 learned linear representations of
space at various scales, such as landmarks and cities; the authors even
identified specific nodes in the model’s architecture that are responsible for
this behaviour. Bhandari et al. (Bhandari et al., 2023) probed Llama and
several smaller language models for geocoding abilities and geospatial
awareness, showing that larger and more sophisticated models have better
abilities to synthesise geospatial knowledge from pre-training texts. Mai et
al. (Mai et al., 2023) demonstrated superior performance of the GPT-family
LLMs on toponym recognition and location description recognition tasks, with
few-shot learning often performing on par with the specialised, state-of-the-
art named entity recognition (NER) models.

It is generally accepted that language models acquire most of their knowledge
during the unsupervised pre-training process, when large amounts of unlabelled
and unstructured text are fed into the model (Radford et al., 2018). The
Common Crawl corpus111https://commoncrawl.org/ (CC) is the largest free web
crawl dataset that powers many smaller, higher-quality datasets such as
OSCAR222https://oscar-project.org/ (Suárez et al., 2021),
CC-100333https://huggingface.co/datasets/cc100, Colossal Cleaned Crawl Corpus
(C4)444https://huggingface.co/datasets/c4 (Raffel et al., 2020), and
RefinedWeb555https://huggingface.co/datasets/tiiuae/falcon-refinedweb (Penedo
et al., 2023), which are often used in pre-training of LLMs.

Given the importance of CC in the field of generative AI and the growing
concerns over the biases and ethics of the generative AI models, it is not
surprising the corpus has been analysed for undesirable content such as hate
speech and sexually explicit content (Luccioni and Viviano, 2021; Raffel et
al., 2020), and its derivative datasets were analysed for general quality
(Kreutzer et al., 2022; Dodge et al., 2021; Birhane et al., 2021). While the
geospatial bias of LLMs has been subject to multiple investigations (Moayeri
and Feizi, 2024; Manvi et al., 2024; Fulman et al., 2024), to our knowledge,
no attempts were made to look into the geospatial contents of CC.

In this paper, we aim to fill in this gap by quantifying the number of
geospatial files found in recent CC releases, as well as estimating the share
of traditional web documents (for example, HTML and XML files) that contain
geospatial information, such as coordinates or street addresses. We believe
that any debate about the emergent geospatial capabilities of LLMs cannot be
complete without knowing the prevalence of geospatial content in the pre-
training text corpus.

In the following section, we will present an overview of existing literature
quantifying various types of content in CC and the derived datasets. We will
then suggest our method of quantifying geospatial content in CC, and present
our findings based on three CC releases spaced out over the last five years.

##  2\. Related work

In 2012, Mühleisen and Bizer (Mühleisen and Bizer, 2012) suggested a pipeline
to extract structured data, such as
Microformats666https://developer.mozilla.org/en-US/docs/Web/HTML/microformats,
RDFa777https://www.w3.org/TR/rdfa-primer/ and
Microdata888https://developer.mozilla.org/en-US/docs/Web/HTML/Microdata, from
CC. The authors estimated that 6% of the HTML pages found in the 2009/10
version of CC contained at least one type of structured data, increasing to
12% in the February 2012 version. Such a jump is not surprising given the
exploding popularity of structured data on the web since their creation in
mid-2000s (Ruth et al., 2013). According to (Mühleisen and Bizer, 2012), only
about 2% of RDFa and 8% of Microdata objects contained geographical
information.

To our knowledge, no other research focused on quantifying geospatial in CC
and its derivative datasets. When it comes to other types of content, in 2021,
Dodge et al. (Dodge et al., 2021) claimed to be some of the first researchers
to analyse the C4 dataset, which is a subset of CC (Raffel et al., 2020). They
identified a large number of patents, documents coming from the US military
websites, and machine-generated text, and concluded that the text from and
about minority individuals that is present in CC is disproportionately removed
from C4. In the same year, Luccioni and Viviano (Luccioni and Viviano, 2021)
randomly sampled 1% of the documents in a single CC release, and applied
n-grams, HateSonar999https://github.com/Hironsan/HateSonar, and DE-
LIMIT101010https://github.com/hate-alert/DE-LIMIT methods to estimate the
prevalence of hate speech in sampled documents at 4.02% to 6.38%, and the
prevalence of sexually explicit content at between 0.73% and 2.36%. In a
somewhat peculiar exploration in 2014, van Hague et al. (Van Hage et al.,
2014) identified the frequency of numbers found in the 2012 version of CC,
showing that the (overwhelmingly positive) numbers found across web pages do
follow the Power law distribution.

These works gave us insight into how researchers approach the problem of
analysing and quantifying phenomena in large text corpora, and we propose our
own method of quantifying geospatial in CC in the following section.

##  3\. Method

Our aim is to estimate what _share_ of documents in a typical CC release
contains _any_ geospatial information as defined in the next subsections. We
deliberately avoid labelling and then quantifying the _actual_ geospatial text
as our delineation criteria and judgement of relevance would get extremely
complex.

###  3.1. About Common Crawl

Common Crawl is a San-Francisco based non-profit organisation. Established in
2007, its goal is to ‘make wholesale extraction, transformation, and analysis
of open web data accessible to researchers’ (Crawl, 2024). The released
datasets comprise the world’s largest free corpus of scraped web data (Baack,
2024), estimated to contain about 250 billion pages, with between 3 and 5
billion added each month (Crawl, 2024).

The datasets are released every one-two months, and are freely available via
S3 and HTTP protocols111111https://data.commoncrawl.org. Each release is
divided into 300 Web Archive (WARC) chunks (files) for easier access, and
accompanied by the index with relevant metadata on each file, such as file’s
original URL, MIME-type, and content length. One can obtain _individual_
scraped web documents from the WARC chunks directly from the Common Crawl
servers, as we showed in our recent work (Ilyankou et al., 2024):

    requests.get(
        f’https://data.commoncrawl.org/{warc_file}’
        headers={
            ’Range’: f’bytes={offset}-{offset+length-1}’
        }
    )

In the code snippet above, we use requests Python
library121212https://docs.python-requests.org/en/latest/index.html to retrieve
an individual scraped web file from a CC warc_file using the Range HTTP
header, only reading the bytes of the relevant file by knowing its location
within the warc_file, including the offset from the start of the warc_file,
and its length in bytes.

In this work, we will focus on three CC releases spanning the last five years,
CC-MAIN-{2019-09,2021-39,2024-10}.

###  3.2. What constitutes geospatial

The vast majority of the files found in CC are standard web pages, such as
HTML and XML pages. In fact, the share of CC files that are exclusively
geospatial, such as .geojson or .kml, is only roughly 0.003-0.004%. Table 1
shows all geospatial MIME types that we were able to identify from the index
of the three relevant CC releases. Therefore, our aim is to estimate the
prevalence of geospatial content in predominantly non-geospatial files.

Table 1. Exclusively geospatial files found in selected CC releases. MIME-types for ‘application’ are abbreviated as ‘app.’ | CC Release
---|---
MIME file type | 2019-09 | 2021-39 | 2024-10
app./geo+json | 0 | 11 | 1
app./gml+xml | 232 | 80 | 333
app./gpx+xml | 6,071 | 13,144 | 16,951
app./tcx+xml | 12 | 0 | 0
app./vnd.garmin.gpx+xml | 1 | 0 | 1
app./vnd.garmin.tcx+xml | 4 | 0 | 0
app./vnd.geo+json | 1 | 0 | 0
app./vnd.google-earth.gpx+xml | 0 | 16 | 57
app./vnd.google-earth.kml+xml | 67,982 | 67,797 | 108,563
app./vnd.google-earth.kmz | 14,870 | 12,849 | 12,124
app./vnd.openstreetmap.data+xml | 0 | 172 | 70
app./vnd.topografix.gpx+xml | 2 | 0 | 49
app./x-shapefile | 512 | 219 | 961
text/gpx+xml | 2 | 2 | 15
text/x-gpx+xml | 0 | 0 | 1
text/xml+georss | 4 | 0 | 0
Total geo files | 89,693 | 94,290 | 139,126
All docs in release | 2914M | 2930M | 3107M
Geo files as % of all docs | 0.003 | 0.003 | 0.004

In the _Manual of Geospatial Science and Technology_ , Bossler defines
geospatial data as ‘features tied to the surface of the earth by coordinates,
addresses, or other means’ (Bossler, 2010). Thus, we focus on two elements:

  * •

_coordinates_ , such as latitude-longitude pairs or easting-northing of the
British National Grid, whether or not they are accompanied by an address or a
location description, and

  * •

_street addresses_ , specific enough so that they can be geocoded into a
point.

We do not count derivative location systems such as
what3words131313https://what3words.com or geohash141414https://geohash.org/ as
valid for our purposes. (In real life, these are typically accompanied by
traditional addresses or coordinates, so we are likely to still count the
document.) We exclude simple mentions of landmarks, such as ‘The White House’,
unless they are accompanied by an address (e.g., ‘The White House, 1600
Pennsylvania Avenue NW, Washington, DC’ or even simply ‘The White House,
Washington DC’). This is to ensure we don’t count documents that mention
landmarks in passing.

We do not discriminate against real and fictional addresses. For example, we
count ‘123 Main Street¡br/¿New York, NY 10001’ as a valid address even though
we are certain it is used as a filler. Similarly, we do not check if latitude-
longitude pairs are valid as long as it is clear the pair of numbers
represents physical coordinates. We do, however, exclude the coordinates of
the Null Island (0,0) as those are often used for map initialisation and are
not meaningful otherwise (Juhasz and Mooney, 2022), as well as astronomical
coordinates of celestial bodies where possible.

###  3.3. Identifying coordinates and addresses

In web documents, the coordinates most commonly appear inside Google Maps
links, inside JavaScript code snippets for map initialisation or marker
locations, or as part of <meta> tags or structured documents (e.g.,
Schema.org151515https://schema.org/):

    https://maps.google.co.jp/maps?q=35.6655165,139.7284844

    new google.maps.LatLng(38.674594,15.889066)

    <meta name="geo.position" content="48.85837;2.294481" />

    {
      "@type": "GeoCoordinates",
      "latitude": "35.935775",
      "longitude": "-75.612923"
    }

In general, identifying coordinates can be performed using regular
expressions, as those are typically expressed as pairs of numbers. For
example, latitudes and longitudes are typically expressed as precise numbers
between $(-90.0,90.0)$ and $(-180.0;180.0)$, respectively. However, any regex
pattern is likely to produce false positive matches, such as scientific
numbers ($\pi\approx 3.141592$) or unusually formatted phone numbers (e.g.,
$+1.234567890$). In addition, we occasionally see coordinates expressed in
minutes and seconds, such as ‘N 56° 59’ 55” E 009° 19’ 33,7”’, as well as
those expressed using various national grid systems, such as positive integers
representing easting and northing in the British National
Grid161616https://britishnationalgrid.uk/. Adding more regular expressions to
capture all these cases is likely to produce even more false positives in
different languages, which will be laborious to classify manually.

Identifying street addresses is much more challenging. Addresses may or may
not contain digits or punctuation, be expressed as a single string or over
multiple HTML tags or JSON fields. Below are some examples of addresses we’ve
encountered in the sampled CC documents:

    Montevideo 938, 3piso - CABA

    <meta itemprop="address" content="4/64-66 Castlereagh St.">

    "address": {
        "@type": "PostalAddress",
        "addressCountry": "RO",
        "addressLocality": "Rosu",
        "addressRegion": "Ilfov",
       "streetAddress": "Nuferilor 16a"
    }

    12417 Woodgreen St
    Los Angeles
    CA

    Pantile House | Newlands Drive | Witham | CM8 2AP | UK

    http://maps.google.ca/maps?...&q=800+Steeles+Ave+W
    +Vaughan,+York+Regional+Municipality,+Ontario&...

In addition to varied formats, many texts are in non-Latin scripts, such as
Cyrillic, Arabic, Greek, and Chinese/Japanese/Korean characters. Popular open-
source software packages such as pyap171717https://github.com/vladimarius/pyap
or libpostal181818https://github.com/openvenues/libpostal usually only work
with shorter strings and are typically limited to particular countries. In
2010, Chang and Li (Chang and Li, 2010) proposed MapMarker, a sequence
labelling algorithm coupled with pattern mining, to perform postal addresses
extraction from web texts, achieving the F-score of 91%. In 2013, Schmidt et
al. (Schmidt et al., 2013) suggested a hybrid approach using patterns and data
from gazetteers to extract addresses from web documents, showing high recall
and precision. However, in their approach HTML tags are removed from the
documents, which is not ideal given that geospatial information can be housed
inside <meta> and other tags, as well as inside JavaScript code snippets and
JSON files. In 2018, Efremova et al. (Efremova et al., 2018) introduced a geo-
tagging framework that employs n-grams, sliding windows, and a Support Vector
Machine classifier to extract addresses from CC documents, achieving precision
and recall of 91% and 93% respectively. Named entity recognition (NER) tools
such as spaCy191919https://spacy.io/ can also be trained to extract addresses,
yet this would require a labelled training set.

Inspired by the needle-in-a-haystack evaluation task for LLMs202020This
popular GitHub repository contains needle-in-a-haystack benchmark tests for
LLMs: https://github.com/gkamradt/LLMTest_NeedleInAHaystack, in which long-
context window LLMs are tasked to retrieve particular bits of information from
a long document, such as an article or a book, we decided to employ a language
model to retrieve both coordinates and addresses from web documents (if they
exist), and then manually review the results. LLMs are excellent zero-shot
learners and have demonstrated superior performance in named entity
recognition and text classification tasks (Xie et al., 2023; Zhou et al.,
2024; Wei et al., 2022; Team, 2024), so we believe they are better suited to
identify the presence of geospatial in multilingual CC documents than
traditional, language-and country-specific methods.

###  3.4. Gemini 1.5 for geospatial detection

Gemini 1.5 is one of the few LLMs currently capable of handling inputs of up
to 1 million tokens, and has shown to perform well in the needle-in-a-haystack
retrieval task (Team, 2024). The context window consideration is important
given the web documents found in CC can be lengthy. Figure 1 shows the
histogram of lengths of sampled documents that we used in our analysis. While
the median and means lengths range from ca. 17k to 55k tokens between the
releases, the length distribution is heavily skewed to the right, with some
documents reaching up to 1M tokens.

Figure 1. The distribution of sampled web documents’ length, in tokens. We
capped the x-axis at 200,000; some documents reach 1M tokens in length.

Web document length in tokens (includes query text)

Figure 2. The prevalence of geospatial data in select CC releases, estimated
within $\pm 0.5\%$ at $95\%$ confidence

The prevalence of geospatial data in CC documents

In this paper, we use the Gemini 1.5 Flash version, which has the most
generous free tier API usage allowance of 1,500 requests per day. At the time
of running our experiment, the pay-as-you-go tier with up to 10,000 requests
per day has been announced, but is not yet available212121According to
https://ai.google.dev/pricing, the pay-as-you-go pricing for Gemini 1.5 Flash
and Pro will become available on May 30, 2024.

After some instruction-tuning on sampled web documents from CC, we adapted the
following query:

Document: “‘{cc_web_document}”’ Question: Does the document enclosed in triple
quotes contain geographic coordinates, such as a latitude- longitude pair, or
a street address that can be used to identify a physical location such as a
building? If so, return a few examples. If not, just say ‘None’.

We assume the web documents do not contain adverse instructions that may
confuse the language model.

Initially, we wanted to use separate queries for coordinates and street
addresses; however, due to the API limitations, we decided to combine both
elements into a single query to half the execution time after confirming that
Gemini responds with both categories of examples (coordinates, addresses)
where both are present in the document.

Below is an example of a Gemini response to the query for the document that
contains both a pair of coordinates and a street address:

The document contains the following geographic information: * **Latitude-
Longitude Pair:** 45.3702375; -85.0128508 * **Street Address:** 1889 Petoskey
Harbor Road * **City, State, Zip Code:** Petoskey, MI 49770

###  3.5. Determining sample size

Previous studies that analysed CC have done so on many randomly sampled
documents (often an arbitrary, but large and round number). For example,
Luccioni and Viviano (Luccioni and Viviano, 2021) sampled 1% of the English
language documents in a single CC release, which amounted to just under 6
million documents. Dodge et al. (Dodge et al., 2021) randomly sampled and
clustered 100,000 documents from CC that were block-listed by (and hence did
not make it into) the C4 dataset.

However, in our task we are unable to utilise fast algorithms such as keyword
lookup, n-grams, or K-Means. We are limited by the number of API requests we
can make; in addition, each query takes Gemini between several seconds to
several minutes to be processed. We need to choose the smallest yet
scientifically sound sample size, and so we decided to use Cochran’s formula
(1) (Bartlett et al., 2001). The formula is used to calculate the ideal sample
size that estimates population with a desired level of precision and a
particular confidence interval. It is commonly used in social sciences to
estimate the number of people to survey. We use the version of Cochran’s
formula modified for very large (or infinite) population because a single CC
release contains billions of web pages:

(1) $x=\frac{Z^{2}\times p\times(1-p)}{e^{2}},$

where $e$ is the desired level of precision (we chose 0.005 to get the
estimate within $\pm 0.5\%$), $p$ is the estimated proportion of the
population that possesses a characteristic (in our case, this is what we are
trying to determine, so we leave it at $0.5$ as is customary in such
situations), and $Z$ is the z-value of our chosen confidence interval
($\approx 1.96$ for 95% confidence).

Therefore, according to the formula, we need to sample 38,416 documents to get
a prevalence estimate approximated at $\pm 0.5\%$ with a confidence interval
of 95%. We decided to sample 39,900 documents (133 random documents from each
of 300 parts) from each of 3 individual CC releases. The headroom of 1,400+
documents is to account for the fact that Gemini occasionally fails to process
queries, either due to an _Internal error_ (code 500) or by marking the query
as _Unsafe_. At the end, we queried the language model $3\times 133\times
300=119,700$ times, and then manually reviewed all responses.

###  3.6. Revising Gemini responses

We manually reviewed each Gemini response that is longer than 1 or 2 tokens
(_‘None’_ or _‘None.’_ , respectively) in order to check that we are satisfied
with the quality of the examples returned, as well as to distinguish whether
the document contains an address, or coordinates, or both.

To ensure Gemini did not hallucinate the examples, that is, made up random
coordinates or addresses that do not exist in the source document, we manually
checked a random sample of 100 documents for which the language model returned
geospatial examples. We were able to locate _all_ examples provided by Gemini
in the corresponding source documents.

In order to check for false negatives, that is, the documents for which Gemini
returned _None_ that may contain geospatial data, we manually reviewed a
random sample of 100 such documents. Using Google
Translate222222https://translate.google.com when needed, we were unable to
identify street addresses or coordinates in _all_ cases.

##  4\. Findings

Our estimate of the prevalence of geospatial data in CC documents is shown in
Figure 2. Overall, when analysing the three CC releases from between 2019 and
2024, we find that 18.3% of the documents contain either a street address or a
pair of coordinates (or both), 15.6% of the documents contain a street
address, 7.1% of the document contain a pair of coordinates, and 4.4% of the
documents have both an address and a pair of coordinates. These estimates are
fairly consistent between the releases.

Almost 28% of the web documents that contain an address also contain a pair of
coordinates, and almost 62% of the documents that contain coordinates also
contain an address. (It is worth noting that the coordinate and the address
found in the same document do not necessarily refer to the same location.)

The most pronounced single source of coordinates in web documents is Google
Maps URLs that encode latitude and longitude (occasionally with the address,
too). Examples of these can be found in the _Identifying coordinates and
addresses_ subsection.

On several occasions, Gemini judged that an IP address or a phone number or a
named landmark (such as a theatre) is sufficient to locate the place. In some
cases, it also returned the astronomical coordinates of celestial bodies.
However, these did not meet our definition of geospatial described in the
_What constitutes geospatial_ subsection, so we excluded them from the final
count.

##  5\. Conclusion

In this paper, we analysed the contents of CC for the presence of geospatial
data. Using Gemini and manual verification, we estimated that about 18.3% of
all documents in recent CC releases contain a street address or a pair of
coordinates. Thus, we conclude that geospatial data is fairly well represented
in mostly unstructured CC documents.

Therefore, it is not surprising that LLMs exhibit some geospatial
capabilities, such as the knowledge of distances and directions between
landmarks and other locations. We suggest further research into the accuracy
and quality of the geospatial data in CC documents is needed to better
understand the geospatial biases that LLMs exhibit. We also recognise that
coordinates and addresses, being specific point-level bits of information, are
not the only way LLMs learn the physical space; the context of the web pages
where geospatial data is found is also very important. This may include
textual descriptions of places, both objective and subjective, such as those
describing the surroundings or the experience of the visitors or the
activities happening in the area.

Given the scale of CC, which advertises itself as having the total of ‘over
250 billion documents’232323https://commoncrawl.org/, we can estimate that 45
billion pages are likely to contain geospatial data. In some cases, the
geospatial data will be context rich, making CC a potential goldmine for
domain-specific geospatial datasets that can be used to train or fine-tune
LLMs on geospatial tasks.

###### Acknowledgements.

This work was supported by the Ordnance Survey, and the Engineering and
Physical Sciences Research Council [grant no. EP/Y528651/1]. The authors would
also like to thank Chuang Liu, Zhengxiang Shi, Meihui Wang, Xiaowei Gao,
Jerome Ramos, Hao Chen, Xinglei Wang, and Junyuan Liu of UCL SpaceTimeLab for
their help facilitating access to the Gemini API.

## References

  * (1)
  * Baack (2024) Stefan Baack. 2024.  Training Data for the Price of a Sandwich: Common Crawl’s Impact on Generative AI.  (Feb. 2024).
  * Bartlett et al. (2001) James E Bartlett, Joe W Kotrlik, and Chadwick C Higgins. 2001.  Organizational Research: Determining Appropriate Sample Size in Survey Research.  (2001).
  * Bhandari et al. (2023) Prabin Bhandari, Antonios Anastasopoulos, and Dieter Pfoser. 2023.  Are Large Language Models Geospatially Knowledgeable?. In _Proceedings of the 31st ACM International Conference on Advances in Geographic Information Systems_ _(SIGSPATIAL ’23)_. Association for Computing Machinery, New York, NY, USA, 1–4.  https://doi.org/10.1145/3589132.3625625
  * Birhane et al. (2021) Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe. 2021.  Multimodal datasets: misogyny, pornography, and malignant stereotypes.  http://arxiv.org/abs/2110.01963 arXiv:2110.01963 [cs].
  * Bossler (2010) John Bossler. 2010.  _Manual of Geospatial Science and Technology_.  CRC Press.  Google-Books-ID: UdZ3uDekqwwC.
  * Chang and Li (2010) Chia-Hui Chang and Shu-Ying Li. 2010.  MapMarker: Extraction of Postal Addresses and Associated Information for General Web Pages. In _2010 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology_. IEEE, Toronto, AB, Canada, 105–111.  https://doi.org/10.1109/WI-IAT.2010.64
  * Crawl (2024) Common Crawl. 2024.  Common Crawl - Open Repository of Web Crawl Data.  https://commoncrawl.org/
  * Dodge et al. (2021) Jesse Dodge, Maarten Sap, Ana Marasović, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, and Matt Gardner. 2021.  Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus.  http://arxiv.org/abs/2104.08758 arXiv:2104.08758 [cs].
  * Efremova et al. (2018) Julia Efremova, Ian Endres, Isaac Vidas, and Ofer Melnik. 2018.  A Geo-Tagging Framework for Address Extraction from Web Pages. In _Advances in Data Mining. Applications and Theoretical Aspects_ , Petra Perner (Ed.). Springer International Publishing, Cham, 288–295.  https://doi.org/10.1007/978-3-319-95786-9_22
  * Fulman et al. (2024) Nir Fulman, Abdulkadir Memduhoğlu, and Alexander Zipf. 2024.  Evidence for Systematic Bias in the Spatial Memory of Large Language Models.  (2024).
  * Gurnee and Tegmark (2024) Wes Gurnee and Max Tegmark. 2024.  Language Models Represent Space and Time.  https://doi.org/10.48550/arXiv.2310.02207 arXiv:2310.02207 [cs].
  * Ilyankou et al. (2024) Ilya Ilyankou, James Haworth, and Stefano Cavazzi. 2024.  CC-GPX: Extracting High-Quality Annotated Geospatial Data from Common Crawl.  http://arxiv.org/abs/2405.11039 arXiv:2405.11039 [cs].
  * Juhasz and Mooney (2022) Levente Juhasz and Peter Mooney. 2022.  “I Think i Discovered a Military Base in the Middle of the Ocean”—Null Island, the Most Real of Fictional Places.  _IEEE Access_ 10 (2022), 84147–84165.  https://doi.org/10.1109/ACCESS.2022.3197222
  * Kreutzer et al. (2022) Julia Kreutzer, Isaac Caswell, Lisa Wang, Ahsan Wahab, Daan van Esch, Nasanbayar Ulzii-Orshikh, Allahsera Tapo, Nishant Subramani, Artem Sokolov, Claytone Sikasote, Monang Setyawan, Supheakmungkol Sarin, Sokhar Samb, Benoît Sagot, Clara Rivera, Annette Rios, Isabel Papadimitriou, Salomey Osei, Pedro Ortiz Suarez, Iroro Orife, Kelechi Ogueji, Andre Niyongabo Rubungo, Toan Q. Nguyen, Mathias Müller, André Müller, Shamsuddeen Hassan Muhammad, Nanda Muhammad, Ayanda Mnyakeni, Jamshidbek Mirzakhalov, Tapiwanashe Matangira, Colin Leong, Nze Lawson, Sneha Kudugunta, Yacine Jernite, Mathias Jenny, Orhan Firat, Bonaventure F. P. Dossou, Sakhile Dlamini, Nisansa de Silva, Sakine Çabuk Ballı, Stella Biderman, Alessia Battisti, Ahmed Baruwa, Ankur Bapna, Pallavi Baljekar, Israel Abebe Azime, Ayodele Awokoya, Duygu Ataman, Orevaoghene Ahia, Oghenefego Ahia, Sweta Agrawal, and Mofetoluwa Adeyemi. 2022.  Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets.  _Transactions of the Association for Computational Linguistics_ 10 (Jan. 2022), 50–72.  https://doi.org/10.1162/tacl_a_00447 arXiv:2103.12028 [cs].
  * Luccioni and Viviano (2021) Alexandra Sasha Luccioni and Joseph D. Viviano. 2021.  What’s in the Box? A Preliminary Analysis of Undesirable Content in the Common Crawl Corpus.  http://arxiv.org/abs/2105.02732 arXiv:2105.02732 [cs].
  * Mai et al. (2023) Gengchen Mai, Weiming Huang, Jin Sun, Suhang Song, Deepak Mishra, Ninghao Liu, Song Gao, Tianming Liu, Gao Cong, Yingjie Hu, Chris Cundy, Ziyuan Li, Rui Zhu, and Ni Lao. 2023.  On the Opportunities and Challenges of Foundation Models for Geospatial Artificial Intelligence.  http://arxiv.org/abs/2304.06798 arXiv:2304.06798 [cs].
  * Manvi et al. (2024) Rohin Manvi, Samar Khanna, Marshall Burke, David Lobell, and Stefano Ermon. 2024.  Large Language Models are Geographically Biased.  http://arxiv.org/abs/2402.02680 arXiv:2402.02680 [cs].
  * Moayeri and Feizi (2024) Mazda Moayeri and Soheil Feizi. 2024.  WorldBench: Quantifying Geographic Disparities in LLM Factual Recall.  https://openreview.net/forum?id=fubvUIBggI
  * Mühleisen and Bizer (2012) Hannes Mühleisen and Christian Bizer. 2012.  Web Data Commons – Extracting Structured Data from Two Large Web Corpora.  (2012).
  * Penedo et al. (2023) Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023.  The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only.  http://arxiv.org/abs/2306.01116 arXiv:2306.01116 [cs].
  * Radford et al. (2018) Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018.  Improving Language Understanding by Generative Pre-Training.  (2018).
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  http://arxiv.org/abs/1910.10683 arXiv:1910.10683 [cs, stat].
  * Ruth et al. (2013) Luke Ruth, David Wood, Marsha Zaidman, and Michael Hausenblas. 2013.  _Linked Data: Structured data on the Web_.  Simon and Schuster.  Google-Books-ID: hTozEAAAQBAJ.
  * Schmidt et al. (2013) Sebastian Schmidt, Simon Manschitz, Christoph Rensing, and Ralf Steinmetz. 2013.  Extraction of Address Data from Unstructured Text using Free Knowledge Resources. In _Proceedings of the 13th International Conference on Knowledge Management and Knowledge Technologies_. ACM, Graz Austria, 1–8.  https://doi.org/10.1145/2494188.2494193
  * Suárez et al. (2021) Pedro Javier Ortiz Suárez, Benoît Sagot, and Laurent Romary. 2021.  Asynchronous Pipeline for Processing Huge Corpora on Medium to Low Resource Infrastructures.  (2021).
  * Team (2024) Gemini Team. 2024.  Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context.  arXiv:2403.05530 [cs.CL]
  * Van Hage et al. (2014) Willem Robert Van Hage, Thomas Ploeger, and Jesper Hoeksema. 2014.  Number frequency on the web. In _Proceedings of the 23rd International Conference on World Wide Web_. ACM, Seoul Korea, 571–572.  https://doi.org/10.1145/2567948.2576962
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022.  Finetuned Language Models Are Zero-Shot Learners.  http://arxiv.org/abs/2109.01652 arXiv:2109.01652 [cs].
  * Xie et al. (2023) Tingyu Xie, Qi Li, Jian Zhang, Yan Zhang, Zuozhu Liu, and Hongwei Wang. 2023.  Empirical Study of Zero-Shot NER with ChatGPT.  (2023).
  * Zhou et al. (2024) Wenxuan Zhou, Sheng Zhang, Yu Gu, Muhao Chen, and Hoifung Poon. 2024.  UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition.  http://arxiv.org/abs/2308.03279 arXiv:2308.03279 [cs].