#  SpaBERT: A Pretrained Language Model from Geographic Data for Geo-Entity
Representation

Zekun Li1, Jina Kim1, Yao-Yi Chiang1, Muhao Chen2
1Department of Computer Science and Engineering, University of Minnesota, Twin
Cities
2Department of Computer Science, University of Southern California
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Named geographic entities (geo-entities for short) are the building blocks of
many geographic datasets. Characterizing geo-entities is integral to various
application domains, such as geo-intelligence and map comprehension, while a
key challenge is to capture the spatial-varying context of an entity. We
hypothesize that we shall know the characteristics of a geo-entity by its
surrounding entities, similar to knowing word meanings by their linguistic
context. Accordingly, we propose a novel spatial language model, SpaBERT (),
which provides a general-purpose geo-entity representation based on
neighboring entities in geospatial data. SpaBERT extends BERT to capture
linearized spatial context, while incorporating a spatial coordinate embedding
mechanism to preserve spatial relations of entities in the 2-dimensional
space. SpaBERT is pretrained with masked language modeling and masked entity
prediction tasks to learn spatial dependencies. We apply SpaBERT to two
downstream tasks: geo-entity typing and geo-entity linking. Compared with the
existing language models that do not use spatial context, SpaBERT shows
significant performance improvement on both tasks. We also analyze the entity
representation from SpaBERT in various settings and the effect of spatial
coordinate embedding.

##  1 Introduction

Figure 1: Overview for generating the pivot entity representation, $E^{p}$ ( :
pivot entity; : neighboring geo-entities). SpaBERT sorts and concatenate
neighboring geo-entities by their distance to pivot in ascending order to form
a pseudo-sentence. [CLS] is prepended at the beginning. [SEP] separates
entities. SpaBERT generates token representations and aggregates
representations of pivot tokens to produce $E^{p}$.

Interpreting human behaviors requires considering human activities and their
surrounding environment. Looking at a stopping location, [ Speedway,],111A
geographic entity name  Speedway and its location (e.g., latitude and
longitude). Best viewed in color. from a person’s trajectory, we might assume
that this person needs to use the location’s amenities if  Speedway implies a
gas station, and is near a highway exit. We might predict a meetup at [
Speedway,] if the trajectory travels through many other locations, , , …, of
the same name,  Speedway, to arrive at [ Speedway,] in the middle of
farmlands. As humans, we are able to make such inferences using the name of a
geographic entity (geo-entity) and other entities in a spatial neighborhood.
Specifically, we contextualize a geo-entity by a reasonable surrounding
neighborhood learned from experience and, from the neighborhood, relate other
relevant geo-entities based on their name and spatial relations (e.g.,
distance) to the geo-entity. This way, even if two gas stations have the same
name (e.g.,  Speedway) and entity type (e.g., ‘gas station’), we can still
reason about their spatially varying semantics and use the semantics for
prediction.

Capturing this spatially varying location semantics can help recognizing and
resolving geospatial concepts (e.g., toponym detection, typing and linking)
and the grounding of geo-entities in documents, scanned historical maps, and a
variety of knowledge bases, such as Wikidata, OpenStreetMap, and GeoNames.
Also, the location semantics can support effective use of spatial textual
information (geo-entities names) in many spatial computing task, including
moving behavior detection from visiting locations of trajectories Yue et al.
(2021, 2019), point of interest recommendations Yin et al. (2017); Zhao et al.
(2022), air quality Lin et al. (2017, 2018, 2020); Jiang et al. (2019) and
traffic prediction Yuan and Li (2021); Gao et al. (2019) using location
context.

Recently, the research community has seen a rapid advancement in pretrained
language models (PLMs) Devlin et al. (2019); Liu et al. (2019); Lewis et al.
(2020); Sanh et al. (2019), which supports strong contextualized language
representation abilities Lan et al. (2020) and serves as the backbones of
various NLP systems Rothe et al. (2020); Yang et al. (2019). The extensions of
these PLMs help NL tasks in different data domains (e.g., biomedicine Lee et
al. (2020); Phan et al. (2021), software engineering Tabassum et al. (2020),
finance Liu et al. (2021)) and modalities (e.g., tables Herzig et al. (2020);
Yin et al. (2020); Wang et al. (2022) and images Li et al. (2020a); Su et al.
(2019)). However, it is challenging to adopt existing PLMs or their extensions
to capture geo-entities’ spatially varying semantics. First, geo-entities
exist in the physical world. Their spatial relations (i.e., distance and
orientation) do not have a fixed structure (e.g., within a table or along a
row of a table) that can help contextualization. Second, existing language
models (LMs) pretrained on general domain corpora Devlin et al. (2019); Liu et
al. (2019) require fine-tuning for domain adaptation to handle names of geo-
entities.

To tackle these challenges, we present SpaBERT (), a LM that captures the
spatially varying semantics of geo-entity names using large geographic
datasets for entity representation. Built upon BERT Devlin et al. (2019),
SpaBERT generates the contextualized representation for a geo-entity of
interest (referred to as the pivot), based on its geographically nearby geo-
entities. Specifically, SpaBERT linearizes the 2-dimensional spatial context
by forming pseudo sentences that consist of names of the pivot and neighboring
geo-entities, ordered by their spatial distance to the pivot. SpaBERT also
encodes the spatial relations between the pivot and neighboring geo-entities
with a continuous _spatial coordinate embedding_. The spatial coordinate
embedding models horizontal and vertical distance relations separately and, in
turn, can capture the orientation relations. These techniques make the inputs
compatible with the BERT-family structures.

In addition, the backbone LM, BERT, in SpaBERT is pretrained with general-
purpose corpora and would not work well directly on geo-entity names because
of the domain shift. We thus train SpaBERT using pseudo sentences generated
from large geographic datasets derived from OpenStreetMap
(OSM).222OpenStreetMap: https://www.openstreetmap.org/ This pretraining
process conducts Masked Language Modeling (MLM) and Masked Entity Prediction
(MEP) that randomly masks subtokens and full geo-entity names in the pseudo
sentences, respectively. MLM and MEP enable SpaBERT to learn from pseudo
sentences for generating spatially varying contextualized representations for
geo-entities.

SpaBERT provides a general-purpose representation for geo-entities based on
their spatial context. Similar to linguistic context, spatial context refers
to the surrounding environment of a geo-entity. For example, [ Speedway,], [
Speedway,], and [ Speedway,] would have different representations since the
surrounding environment of , , and could vary. We evaluate SpaBERT on two
tasks: 1) geo-entity typing and 2) geo-entity linking to external knowledge
bases. Our analysis includes the performance comparison of SpaBERT in various
settings due to characteristics of geographic data sources (e.g., entity
omission).

To summarize, this work has the following contributions. We propose an
approach to linearize the 2-dimensional spatial context, encode the geo-entity
spatial relations, and use a LM to produce spatial varying feature
representations of geo-entities. We show that SpaBERT is a general-purpose
encoder by supporting geo-entity typing and geo-entity linking, which are keys
to effectively integrating large varieties of geographic data sources, the
grounding of geo-entities as well as supports for a broad range of spatial
computing applications. The experiments demonstrate that SpaBERT is effective
for both tasks and outperforms the SOTA LMs.

##  2 SpaBERT

SpaBERT is a LM built upon a pretrained BERT and further trained to produce
contextualized geo-entity representations given large geographic datasets.
Figure 1 shows the outline of SpaBERT, with the details below. This section
first presents the preliminary (Section 2.1) and then describes the overall
approach for learning geo-entities’ contextualized representations (Section
2.2), the pretraining strategies (Section 2.3), and inference procedures
(Section 2.4).

Figure 2: Embedding modules. Inputs to the token-embedding are the tokenized
geo-entity names in the pseudo-sentence. Inputs to the sequence position
embedding are the sequence indices of the tokens. Inputs to the spatial
coordinate embedding are the normalized distances from the pivot in each
spatial dimension (details in Section 2.2).

###  2.1 Preliminary

We assume that given a geographic dataset (e.g., OSM) with many geo-entities,
$S=\\{g_{1},g_{2},...,g_{l}\\}$, each geo-entity $g_{i}$ (e.g., [ Speedway,])
has two attributes: name $g^{name}$( Speedway) and location $g^{loc}$(). The
location attribute, $g^{loc}$, is a tuple $g^{loc}=(g^{locx},g^{locy},...)$
that identifies the location in a coordinate system (e.g., x and y image pixel
coordinates or latitude and longitude geo-coordinates with altitudes). WLOG,
here we assume a 2-dimensional space. SpaBERT aims to generate a
contextualized representation for each geo-entity $g_{i}$ in $S$ given its
spatial context. We denote a geo-entity that we seek to contextualize as the
_pivot entity_ , $g_{p}$, $p$ for short. The spatial context of $p$ is
$SC(p)=\\{g_{n_{1}},...,g_{n_{k}}\\}$ where $distance(p,g_{n_{k}})<T$. $T$ is
a spatial distance parameter defining a local area. We call
$g_{n_{1}},...,g_{n_{k}}$ as $p$’s neighboring geo-entities and denote them as
$n_{1},...,n_{k}$ when there is no confusion.

###  2.2 Contextualizing Geo-entities

Linearizing Neighboring Geo-entity Names For a pivot, $p$, SpaBERT first
linearizes its neighboring geo-entitie names to form a BERT-compatible input
sequence, called a _pseudo sentence_. The corresponding pseudo sentence for
the example in Figure 1 is constructed as:

[CLS] University of Minnesota [SEP] Minneapolis [SEP] St. Anthony Park [SEP]
Bloom Island Park [SEP] Bell Museum [SEP]

The pseudo sentence starts with the pivot name followed by the names of the
pivot’s neighboring geo-entities, ordered by their spatial distance to the
pivot in ascending order. The idea is that nearby geo-entities are more
related (for contextualization) than distant geo-entities. SpaBERT also
tokenizes the pseudo sentences using the original BERT tokenizer with the
special token [SEP] to separate entity names. The subtokens of a neighbor
$n_{k}$ is denoted as $T^{n_{k}}_{j}$ as in Figure 2.

Encoding Spatial Relations SpaBERT adopts two types of position embeddings in
addtion to the token embeddings in the pseudo sentences (Figure 2). The
sequence position embedding represents the token order, same as the original
position embedding in BERT and other Transformer LMs. Also, SpaBERT
incorporates a _spatial coordinate embedding_ mechanism, which seeks to
represent the spatial relations between the pivot and its neighboring geo-
entities. SpaBERT’s spatial coordinate embeddings encode each location
dimension separately to capture the relative distance and orientation between
geo-entities. Specifically, for the 2-dimensional space, SpaBERT generates
normalized distance $dist_{x}^{n_{k}}$ and $dist_{y}^{n_{k}}$ for each
neighboring geo-entity using the following equations:

$\displaystyle dist_{x}^{n_{k}}=(g_{n_{k}}^{locx}-g_{p}^{locx})/Z$
$\displaystyle dist_{y}^{n_{k}}=(g_{n_{k}}^{locy}-g_{p}^{locy})/Z$

where $Z$ is a normalization factor, and ($g_{p}^{locx}$, $g_{p}^{locy}$),
($g_{n_{k}}^{locx}$, $g_{n_{k}}^{locy}$) are the locations of pivot $p$ and
neighboring entity $n_{k}$. Note that the tokens belong to the same geo-entity
name have the same $dist_{x}^{n_{k}}$ and $dist_{y}^{n_{k}}$. Also, SpaBERT
uses DSEP, a constant numerical value larger than
$max(dist_{x}^{n_{k}},dist_{y}^{n_{k}})$ for all neighboring entities to
differentiate special tokens from entity name tokens.

SpaBERT encodes $dist_{x}^{n_{k}}$ and $dist_{y}^{n_{k}}$ using a continuous
spatial coordinate embedding layer with real-valued distances as input to
preserve the continuity of the output embeddings. Let $\mathcal{S}_{n_{k}}$ be
the spatial coordinate embedding of the neighbor entity $n_{k}$, $M$ be the
embedding’s dimension, and $\mathcal{S}_{n_{k}}\in\mathbf{R}^{M}$. We define
$\mathcal{S}_{n_{k}}$ as:

$\displaystyle\mathcal{S}_{n_{k}}^{(m)}=\begin{cases}sin(dist^{n_{k}}/10000^{2j/M}),m=2j\\\
cos(dist^{n_{k}}/10000^{2j/M}),m=2j+1\end{cases}$

where $\mathcal{S}_{n_{k}}^{(m)}$ is the $m$-th component of
$\mathcal{S}_{n_{k}}$. $dist^{n_{k}}$ represents $dist_{x}^{n_{k}}$ and
$dist_{y}^{n_{k}}$ for the spatial coordinate embeddings along the horizontal
and vertical directions, respectively.

The token embedding, sequence position embedding and spatial coordinate
embedding are summed up then fed into the encoder. Similar to BERT, SpaBERT
encoder calculates an output embedding for each token. Then SpaBERT averages
the pivot’s token-level embeddings to produce a fixed-length embedding for the
pivot’s contextualized representation.

###  2.3 Pretraining

We train SpaBERT with two tasks to adapt the pretrained BERT backbone to geo-
entity pseudo sentences. One task is the masked language modeling (MLM) Devlin
et al. (2019), for which SpaBERT needs to learn how to complete the full names
of geo-entities from pseudo sentences with randomly masked subtokens using the
remaining subtokens and their spatial coordinates (i.e., partial names and
spatial relations between subtokens). The block below shows an example of the
masked input for MLM, where ### are the masked subtokens.

[CLS] ### of Minnesota [SEP] Minneapolis [SEP] St. ### Park [SEP] ### Island
Park ### Bell Museum [SEP]

In addition, we hypothesize that given common spatial co-occurrence patterns
in the real world, one can use neighboring geo-entities to predict the name of
a geo-entity. Therefore, we propose and incorporate a masked entity prediction
(MEP) task, which randomly masks all subtokens of an entity name in a pseudo
sentence. For MEP, SpaBERT relies on the masked entity’s spatial relation to
neighboring entities to recover the masked name. The block below shows an
example of the masked input for MEP.

[CLS] University of Minnesota [SEP] Minneapolis [SEP] ### ### ### [SEP] Bloom
Island Park [SEP] Bell Museum [SEP]

Both MLM and MEP have a masking rate of 15% (without masking [CLS] and [SEP]).
The sequence position and spatial coordinates are not masked.

Figure 3: Example of the Geohash representations with a pivot entity named
‘Kenneth H Keller Hall’ and its neighboring entities from OpenStreetMap. ( :
pivot entity; : neighboring entities; : entities not within the nearest nine
grids). The upper hash grids have a string length of six, and the lower ones
have a length of seven.

Pretraining Data We construct a large training set from OSM covering the City
of London and the State of California. Since OSM is crowd-sourced, we clean
the raw data by removing non-alpha-numeric place names and geo-entities that
do not have a place name or geocoordinates. We randomly select geo-entities as
pivots and construct pseudo sentences as described in Section 2.2.

To efficiently find the neighboring entities from a pivot sorted by distances,
we leverage the Geohash algorithm introduced by Gustavo Niemeyer in 2008. We
first generate the Geohash string for geo-entities on OpenStreetMap, which
encodes a location into a string of a maximum length of 20 with base 32. For
example, a city ‘Madelia’ with the geolocation (44.0508° N, 94.4183° W) has a
Geohash string of ‘9zufe7nwjefpjksty0m8’. The length of the hash string is
associated with the size of the geographic area that the string represents.
The first character divides the entire Earth’s surface into 32 large grids,
and the second character further divides one large grid into 32 grids. Thus,
by comparing the leading characters of two hash strings, we can estimate the
spatial proximity of two locations without calculating the exact distance
between them. We use the hash strings to first filter out the geo-entities
guaranteed to be outside the spatial context radius and keep the neighbors
within the nearest nine hash grids for distance calculation. The use of
Geohash reduces the computation time when constructing the pseudo sentences.
Figure 3 shows an example of Geohash representations. Given the pivot ‘Kenneth
H Keller Hall’, we can calculate its hash string of different lengths to
quickly retrieve the neighboring entities within a desired range (i.e., the
green box).

This way, we generate 69,168 and 148,811 pseudo sentences in London and
California, respectively, leading to a pretraining corpus of around 150M
words. We use all of them for pretraining SpaBERT with MLM and MEP.

###  2.4 Inference

The pretrained SpaBERT can already generate spatially varying contextualized
representations for geo-entities. The representations support various
downstream applications, including contextualized geo-entity classification
and similarity-based inference tasks, such as geo-entity typing and linking.

Contextualized Geo-entity Classification Classification of a geo-entity is
crucial for recognizing and resolving geospatial concepts and the grounding of
geo-entities in various data sources. Here, we use geo-entity typing as an
example task to demonstrate the effectiveness of contextualized geo-entity
representations. In this task, we aim to predict the geo-entity’s semantic
type (e.g., transportation and healthcare). We stack a softmax prediction head
on top of the final-layer hidden states (i.e., geo-entity embeddings) to
perform the classification.

Similarity-based Inference Integrating multi-source geographic datasets often
involves finding the top $K$ nearest geo-entities in some representation
space. One example task, which we refer to as geo-entity linking, is to link
geo-entities from a geographic information system (GIS) oriented dataset to
graph-based knowledge bases (KBs). In such a task, we can directly use the
contextualized geo-entity embeddings from SpaBERT and calculate the embedding
similarity based on some metrics, such as the cosine distance, to match the
corresponding geo-entities from separate data sources.

##  3 Experiments

We evaluate SpaBERT on supervised geo-entity typing and unsupervised geo-
entity linking tasks (Section 3.1-Section 3.3). We also investigate how
SpaBERT performs under various common characteristics of geographic data
sources (e.g., entity omission from cartographic generalization) and how
critical technical components and their parameters affect the performance
(Section 3.4).

###  3.1 Experimental Setup

Datasets Open Street Map (OSM)2 is a large-scale geographic database, and it
is widely used in many popular map-related services. For _supervised geo-
entity typing_ , we randomly select 25,872 and 7,726 pivot geo-entities
together with their neighboring entities from point features of nine semantic
types on OSM in the State of California and the City of London, respectively
(Table 1). Each geo-entity is associated with a name and its geocoordinates.
We use 80% of the data in both regions for training and 20% for testing. The
task is to predict the OSM semantic type for each geo-entity. (Section 3.2).

For _unsupervised geo-entity linking_ , we randomly select 14 scanned
historical maps in California from the United States Geological Survey (USGS)
Historical Topographic Maps Collection. We manually transcribe text labels in
these maps to generate 154 geo-entities, each containing a name (from the text
label) and the image coordinates (text label’s pixel location in the scanned
map) The task is to find the corresponding Wikidata geo-entity333Entities with
the coordinate location attribute
(https://www.wikidata.org/wiki/Property:P625) for each USGS geo-entity with
only their spatial relations from pixel locations. Note that the
geocoordinates of geo-entities in the scanned maps are unknown. We select 15K
Wikidata geo-entities having an identical name to one of the USGS geo-
entities444Two geo-entities can have the same name (e.g., Los Angeles, CA vs.
Los Angeles, TX) and then randomly add another 30K Wikidata geo-entities to
construct the final Wikidata dataset of 45K geo-entities. For the ground
truth, we manually identify the matched Wikidata geo-entity for each USGS geo-
entity.

For geo-entity linking task, the annotation details are described below. We
hire three undergraduate students (not the co-authors) to transcribe text
labels on the USGS maps. They draw bounding boxes/polygons around text labels
and transcribe the text string to indicate geo-entity names and their
locations. The results are verified and corrected by another undergraduate
student and one Ph.D. student. Then, we run an automatic script to find the
potential corresponding geo-entity in Wikidata by matching their names. The
collected Wikidata URIs are noisy, and we have one Ph.D. student verify the
URIs by marking them as True or False linking. The verification results are
randomly further reviewed by a senior digital humanities scholar specialized
in the representation and reception of historical places.

Classes | California | London
---|---|---
Education | 6,222 | 618
Entertainment_Arts_Culture | 1,380 | 601
Facilities | 574 | 179
Financial | 2,590 | 769
Healthcare | 3,779 | 1,779
Public_Service | 2,658 | 393
Sustenance | 4,276 | 1,693
Transportation | 4,226 | 1,618
Waste_Management | 167 | 76
Total | 25,872 | 7,726
Table 1:  OSM Geo-entity typing dataset statistics. The first column shows the nine OSM semantic types, following by the numbers of samples for each class in California and London. Classes $\rightarrow$ | Edu. | Ent. | Fac. | Fin. | Hea. | Pub. | Sus. | Tra. | Was. | Micro Avg
---|---|---|---|---|---|---|---|---|---|---
BERTBase | .674 | .634 | .763 | .929 | .856 | .872 | .856 | .862 | .678 | .835
RoBERTaBase | .626 | .627 | .605 | .951 | .869 | .818 | .838 | .850 | .475 | .820
SpanBERTBase | .633 | .589 | .608 | .916 | .859 | .882 | .824 | .867 | .735 | .819
LUKEBase | .648 | .608 | .598 | .945 | .857 | .867 | .854 | .851 | .517 | .825
SimCSEBERT-Base | .623 | .590 | .504 | .925 | .867 | .852 | .857 | .810 | .470 | .810
SimCSERoBERTa-Base | .621 | .629 | .499 | .951 | .841 | .853 | .828 | .856 | .500 | .814
SpaBERTBase | .674 | .653 | .680 | .959 | .865 | .900 | .883 | .888 | .703 | .852
BERTLarge | .707 | .661 | .647 | .937 | .874 | .850 | .873 | .864 | .526 | .841
RoBERTaLarge | .657 | .626 | .682 | .907 | .855 | .805 | .831 | .859 | .587 | .817
SpanBERTLarge | .683 | .652 | .661 | .931 | .868 | .853 | .851 | .848 | .624 | .829
LUKELarge | .665 | .607 | .660 | .899 | .855 | .809 | .813 | .844 | .587 | .808
SimCSEBERT-Large | .693 | .661 | .713 | .940 | .880 | .871 | .864 | .867 | .564 | .844
SimCSERoBERTa-Large | .683 | .630 | .648 | .916 | .865 | .802 | .807 | .848 | .587 | .811
SpaBERTLarge | .731 | .690 | .710 | .956 | .901 | .892 | .893 | .903 | .677 | .871
Table 2:  Geo-entity typing with the state-of-the-art LMs and SpaBERT. Column
names are the OSM classes. Bold numbers are the highest scores in each column.
Underlined numbers are the highest scores among baselines.

Model Configuration We create two variants of SpaBERT, namely SpaBERTbase and
SpaBERTlarge with weights and tokenizers initialized from the uncased versions
of the BERTbase and BERTlarge, respectively. During pretraining, we mix the
masked instances for MLM and MEP tasks in the same ratio. We primarily use the
same optimization parameters as in Devlin et al. (2019) and train the model
with the AdamW optimizer. We save the model checkpoints every 2K iterations.
The learning rate is $5\times 10^{-5}$ for SpaBERTBase and $1\times 10^{-6}$
for SpaBERTLarge for stability. Distance normalization factor $Z$ is 0.0001,
and distance separator DSEP is 20. The batch size is 12 to fit in one NVIDIA
RTX A5000 GPU with 24GB memory.

###  3.2 Supervised Geo-Entity Typing

Figure 4: TSNE visualization for entity features produced by SpaBERTBase. The
color indicates the ground-truth labels. Financial and Public_Service (pointed
by the arrows) are well separated from other classes which conforms to the
high F1 scores in Table 2 (best viewed in color).

Task Description We tackle geo-entity typing as a supervised classification
task using the modified SpaBERT architecture in Section 2.4 and finetune the
model end-to-end by optimizing the cross-entropy loss. Specifically, the
training set of this task contains $\\{(g_{i},SC(g_{i}),y_{i})\\}_{i=1}^{N}$,
where $g_{i}$ is the pivot entity; $y_{i}$ is its OSM semantic type label;
$SC(g_{i})$ contains its neighboring entities in the nearest nine hash grids.
Note that during training, the model does not use the neighboring geo-
entities’ OSM semantic types. We report the F1 score for each class (i.e., an
OSM semantic type) and the micro-F1 for all samples in the test set.

Model Comparison We compare SpaBERT with several strong PLMs, including BERT
Devlin et al. (2019), RoBERTa Liu et al. (2019), SpanBERT Joshi et al. (2020),
LUKE Yamada et al. (2020), and SimCSE Gao et al. (2021). For all these models,
we use the pretrained weights with the uncased version. For SimCSE, we use
both the BERT and RoBERTa versions with the weights obtained from unsupervised
training. We further finetune these baseline LMs end-to-end on the same
pseudo-sentence dataset as SpaBERT (Table 1) with a softmax prediction head
appended after the baseline model. Note that these baseline models only use
the token and position embeddings but not the spatial coordinate embeddings.

Result Analysis Table 2 summarizes the geo-entity typing results on the OSM
test set with finetuned baselines and SpaBERT. The last column shows the
micro-F1 scores (per-instance average). The remaining columns are the F1
scores for individual classes. For all baseline models, BERTBase and
SimCSEBERT-Large (BERTLarge as a close second) have the highest F1 in the base
and large groups, respectively. In addition, SpaBERT shows the best
performance over the existing context-aware LMs for both base and large
groups. We hypothesize that SpaBERT’s advantage comes from 1) the pretraining
tasks, especially MEP, and 2) the spatial coordinate embeddings. Additionally,
compared to its backbone, BERT, SpaBERT shows performance improvement in
almost all classes for both base and large models. The results demonstrate
that SpaBERT can effectively contextualize geo-entities with their spatial
context.

We can also observe that the Financial class has high F1 scores for all
models. The reason is that when entity names are strongly associated with the
semantic type (e.g., U.S. Bank), pretrained LMs could produce meaningful
entity representation even without the spatial context. However, the typing
performance can still be further improved after encoding the spatial context
with SpaBERT. Figure 4 visualizes the geo-entity features from SpaBERTBase. It
shows that Financial and Public_Service have the most separable features from
other classes. The non-separable region in the middle is mainly due to the
similar spatial context and semantics of geo-entities (e.g. “nursing home" can
be close to both Facilities and Healthcare).

###  3.3 Unsupervised Geo-Entity Linking

Task Description We define the problem as follows. For a query set
$Q=\\{(q_{i},SC(q_{i}))\\}_{i=1}^{|Q|}$, the goal is to find the corresponding
geo-entity for each $q_{i}$ in the candidate set
$C=\\{(c_{i},SC(c_{i}))\\}_{i=1}^{|C|}$ (typically $|C|\gg|Q|$). Here, $Q$ and
$C$ are the USGS and Wikidata geo-entities (Section 3.1). $SC(q_{i})$ contains
$q_{i}$’s nearest 20 geo-entities, and $SC(c_{i})$ includes $c_{i}$’s
neighboring geo-entities within a 5km radius.555Because $Q$ is not
georeferenced, we use the top K nearest neighbors instead of pixel distances.
For this task, we use the evaluation metrics of Recall@K and Mean-Reciprocal-
Rank (MRR).

Model Comparison We use the same baseline models as in the typing task without
finetuning.

Model | MRR | R@1 | R@5 | R@10
---|---|---|---|---
BERTBase | .400 | .289 | .559 | .635
RoBERTaBase | .326 | .232 | .446 | .540
SpanBERTBase | .164 | .138 | .201 | .213
LUKEBase | .306 | .188 | .440 | .547
SimCSEBERT-Base | .453 | .371 | .547 | .628
SimCSERoBERTa-Base | .227 | .188 | .264 | .301
SpaBERTBase | .515 | .338 | .744 | .850
BERTLarge | .337 | .245 | .459 | .509
RoBERTaLarge | .379 | .220 | .603 | .704
SpanBERTLarge | .229 | .176 | .308 | .339
LUKELarge | .402 | .232 | .635 | .767
SimCSEBERT-Large | .475 | .402 | .559 | .616
SimCSERoBERTa-Large | .214 | .176 | .239 | .283
SpaBERTLarge | .537 | .383 | .744 | .864

Table 3:  Geo-entity linking result. Bold and underlined numbers are the
highest scores in each column and the highest scores among the baselines,
respectively.

Result Analysis Table 3 shows the geo-entity linking results. Large models
perform better than their base counterparts for both baseline models and
SpaBERT. In particular, among the baseline models, SimCSEBERT has the highest
score for MRR and R@1. SpaBERT has significantly higher R@5 and R@10 than
SimCSE. Also, SpaBERT outperforms all baselines on MRR, R@5, and R@10. For
both base and large versions, SpaBERT outperforms its backbone model BERT. The
difference between BERT and SpaBERT shows the effectiveness of the spatial
coordinate embeddings and domain adaptation using MLM and the proposed MEP.
Also, SpanBERT shows the worst performance among all models. This could be
that SpanBERT utilizes masked random spans in pretraining, which does not work
well for pseudo sentences of geo-entity names. In contrast, SpaBERT’s MEP
specifically masks tokens from the same geo-entity and could effectively
contextualize geo-entity names distributed in the 2D space with the spatial
coordinate embeddings.

###  3.4 Spatial Context and Ablation Study

Impact of Entity Omission for Linking We analyze how SpaBERT handles different
USGS map scales for geo-entity linking with entity omission from cartographic
generalization (e.g., some entities only exist in certain scales). Here
SpaBERT performs the best for the 1:125K maps (30-CA) (Table 4). Our
hypothesis is that the 1:125K maps contain denser geo-entities than the test
maps at other scales. When selecting the top K nearest neighbors for a pivot
in other map scales, some neighbors could be far away and would not provide
relevant context.

Map Set | Scale $\Downarrow$ | MRR | R@1 | R@5 | R@10
---|---|---|---|---|---
15-CA | 1:62500 | 0.503 | 0.422 | 0.680 | 0.914
30-CA | 1:125000 | 0.639 | 0.599 | 0.862 | 0.932
60-CA | 1:250000 | 0.404 | 0.133 | 0.678 | 0.800

Table 4:  Impact of entity omission for linking (USGS and Wikidata). Results
from SpaBERTLarge. 1:625000 means 1 centimeter on a map represents 625 meters
in the physical world.

Since the entity omission criteria of the USGS maps are unknown and Wikidata
mostly contain important landmark geo-entities, we also simulate omission
using the OSM dataset by gradually removing random neighbors of a pivot entity
within a fixed neighborhood. We test this scenario for geo-entity linking to
see if the same pivot entity would have similar representations from
decreasing neighbor densities (i.e., the original set of neighbors vs. after
random omission at varying rates). We use the same evaluation metric as in
section 3.3, as they reflect the similarity of the learned geo-entity
representation (Figure 5). The linking scores decrease with additional
entities removed, indicating the similarity between the same pivot entity’s
representations learned from the original data and the omitted data declines
when omission percentage becomes larger. The elbow point is at about 30%,
showing that SpaBERT is reasonably robust if the percentage of the removed
neighbors is within 30%.

Impact of Pseudo Sentence Length We analyze the impact of pseudo sentence
length on geo-entity typing by varying the number of neighboring entities.
Table 5 shows the results of SpaBERTBase and SpaBERTLarge with an increasing
number of neighbors. We observe that as #Neighbor increases, the micro-F1
score increases accordingly, as expected. The increment is more evident at the
beginning, illustrating that the spatial context helps contextualize the pivot
entity, especially in a local neighborhood. The benefit of adding additional
neighboring entities decreases as distance increases beyond a certain point.
This conforms to the well-studied spatial autocorrelation.

#Neighbor | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9
---|---|---|---|---|---|---|---|---|---
Base | .773 | .808 | .814 | .827 | .835 | .831 | .835 | .836 | .840
Large | .795 | .822 | .834 | .838 | .843 | .847 | .852 | .848 | .854
#Neighbor | 10 | 20 | 30 | 40 | 50 | 60 | 70 | 80 | 90
Base | .843 | .844 | .846 | .851 | .852 | .850 | .849 | .851 | .852
Large | .850 | .857 | .862 | .858 | .858 | .868 | .863 | .867 | .869

Table 5:  Impact of the pseudo sentence length. Numbers in the table are
micro-F1 scores on the OSM typing dataset. Base and Large are short for
SpaBERTBase and SpaBERTLarge.  Figure 5: Geo-entity feature similarity with a
decreasing neighbor density. X-axis indicates the percentage of the neighbors
removed for the pivot entities.

Effect of Spatial Coordinate Embedding We train two additional SpaBERT
variants that do not include the spatial coordinate embedding during MLM and
MEP pretraining and use them for geo-entity linking. For MRR, SpaBERTBase and
SpaBERTLarge drop from 0.515 to 0.458 and from 0.537 to 0.478 compared to
their original SpaBERT version. Also, the variants have lower recall scores
compared to their original SpaBERT version. The most significant drop in the
recall is on R@1 from 0.383 to 0.283 for SpaBERTLarge. The results demonstrate
the effectiveness of the spatial coordinate embedding.

##  4 Related Work

Pretrained Language Models PLMs have been the dominant paradigm for language
representation. Following the success of MLMs Devlin et al. (2019); Liu et al.
(2019) and autoregressive PLMs Peters et al. (2018); Radford et al. (2019),
more recent work has extended the pretraining process with more tasks or
learning objectives for span prediction Joshi et al. (2020), cross-encoders
Reimers and Gurevych (2019), contrastive learning Gao et al. (2021), and
massively multi-task learning Raffel et al. (2020). To support entity
representation, several approaches propose to perform mention detection Yamada
et al. (2020), incorporate mention memory cells de Jong et al. (2021), or
injecting structural knowledge representations Wang et al. (2021); Zhang et
al. (2019); Peters et al. (2019); Zhou et al. (2022). Due to the large body of
work, we refer readers to recent surveys summarizing this line of work Qiu et
al. (2020); Wei et al. (2021).

To extend the use of PLMs beyond language, much exploration has also been
conducted to represent other modalities. For example, a significant amount of
vision-language models Kim et al. (2021); Zhai et al. (2022); Li et al.
(2020a); Lu et al. (2019) have been developed by jointly pretraining on co-
occurring vision and language corpora, and are in the support of grounding
Zhang et al. (2020); Chen et al. (2020), generation Cho et al. (2021); Yu et
al. (2022) and retrieval tasks Zhang et al. (2021); Li et al. (2020b) on the
vision modality. Other PLMs capture semi-structure tabular data by linearizing
table cells Herzig et al. (2020); Yin et al. (2020); Iida et al. (2021) or
incorporating structural prior in the attention mechanism Wang et al. (2022);
Trabelsi et al. (2022); Eisenschlos et al. (2021). To the best of our
knowledge, none of the prior studies have extended pretrained LMs to geo-
entity representation, nor do they support a suitable mechanism to capture the
geo-entities’ spatial relations, which do not have structural prior. This is
exactly the focus of our work.

Domain-specific Language Modeling Another line of studies has been conducted
to adapt PLMs to specific domains typically by pretraining on domain-specific
corpora. For example, in the biomedicine domain, a series of models Lee et al.
(2020); Peng et al. (2019); Alsentzer et al. (2019); Phan et al. (2021) have
been developed by training PLMs on corpora derived from PubMed. Similarly,
PLMs have been trained on corpora specific to software engineering Tabassum et
al. (2020), finance Liu et al. (2021), and proteomics Zhou et al. (2020)
domains. In this context, SpaBERT represents a pilot study in the geographical
domain by allowing the PLM to learn from spatially distributed text.

##  5 Conclusion

This paper presented SpaBERT (), a language model trained on geographic
datasets for contextualizing geo-entities. SpaBERT utilizes a novel spatial
coordinate embedding mechanism to capture spatial relations between 2D geo-
entities and linearizes the geo-entities into 1D sequences, compatible with
the BERT-family structures. The experiments show that the general-purpose
representations learned from SpaBERT achieve better or competitive results on
the geo-entity typing and geo-entity linking tasks compared to SOTA pretrained
LMs. We plan to evaluate SpaBERT on additional related tasks, such as geo-
entity to natural language grounding. Also, we plan to extend SpaBERT to
support other geo-entity geometry types, including lines and polygons.

## Limitations

The current model design only considers points but not polygon and line
geometries, which could also help provide meaningful spatial relations for
contextualizing a geo-entity. Training of SpaBERT also requires considerable
GPU resources which might produce environmental impacts.

## Ethical Consideration

SpaBERT was evaluated on the English-spoken regions, so the model and results
could have a bias towards these regions and their commonly used languages.
Replacing the backbone of SpaBERT with a multi-lingual model and training
SpaBERT with diverse regions could mitigate the bias.

## Acknowledgement

We thank the reviewers for their insightful comments and suggestions. We thank
Dr. Valeria Vitale for her valuable input. This material is based upon work
supported in part by NVIDIA Corporation, the National Endowment for the
Humanities under Award No. HC-278125-21 and Council Reference AH/V009400/1,
the National Science Foundation of United States Grant IIS 2105329, a Cisco
Faculty Research Award (72953213), and the University of Minnesota, Computer
Science & Engineering Faculty startup funds.

## References

  * Alsentzer et al. (2019) Emily Alsentzer, John Murphy, William Boag, Wei-Hung Weng, Di Jindi, Tristan Naumann, and Matthew McDermott. 2019.  Publicly available clinical BERT embeddings.  In _Proceedings of the 2nd Clinical Natural Language Processing Workshop_ , pages 72–78, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
  * Chen et al. (2020) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020.  UNITER: UNiversal Image-TExt Representation Learning.  In _European Conference on Computer Vision_ , pages 104–120. Springer.
  * Cho et al. (2021) Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. 2021.  Unifying vision-and-language tasks via text generation.  In _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 1931–1942. PMLR.
  * de Jong et al. (2021) Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald, Fei Sha, and William W Cohen. 2021.  Mention Memory: incorporating textual knowledge into Transformers through entity mention attention.  In _International Conference on Learning Representations_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Eisenschlos et al. (2021) Julian Eisenschlos, Maharshi Gor, Thomas Müller, and William Cohen. 2021.  MATE: Multi-view attention for table transformer efficiency.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7606–7619, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple contrastive learning of sentence embeddings.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Gao et al. (2019) Y Gao, Z Duan, W Shi, J Feng, and Y-Y Chiang. 2019.  Personalized Recommendation Method of POI Based on Deep Neural Network.  In _Proceedings of the 2019 6th International Conference on Behavioral, Economic and Socio-Cultural Computing (BESC)_ , pages 1–6.
  * Herzig et al. (2020) Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno, and Julian Eisenschlos. 2020.  TaPas: Weakly supervised table parsing via pre-training.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4320–4333, Online. Association for Computational Linguistics.
  * Iida et al. (2021) Hiroshi Iida, Dung Thai, Varun Manjunatha, and Mohit Iyyer. 2021.  TABBIE: Pretrained representations of tabular data.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3446–3456, Online. Association for Computational Linguistics.
  * Jiang et al. (2019) Jyun-Yu Jiang, Xue Sun, Wei Wang, and Sean Young. 2019.  Enhancing air quality prediction with social media and natural language processing.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 2627–2632, Florence, Italy. Association for Computational Linguistics.
  * Joshi et al. (2020) Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020.  SpanBERT: Improving pre-training by representing and predicting spans.  _Transactions of the Association for Computational Linguistics_ , 8:64–77.
  * Kim et al. (2021) Wonjae Kim, Bokyung Son, and Ildoo Kim. 2021.  Vilt: Vision-and-language transformer without convolution or region supervision.  In _International Conference on Machine Learning_ , pages 5583–5594. PMLR.
  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020.  ALBERT: A lite BERT for self-supervised learning of language representations.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Lee et al. (2020) Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2020.  Biobert: a pre-trained biomedical language representation model for biomedical text mining.  _Bioinformatics_ , 36(4):1234–1240.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Li et al. (2020a) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2020a.  What does BERT with vision look at? In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 5265–5275, Online. Association for Computational Linguistics.
  * Li et al. (2020b) Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020b.  OSCAR: Object-Semantics Aligned Pre-training for Vision-Language Tasks.  In _European Conference on Computer Vision_ , pages 121–137. Springer.
  * Lin et al. (2020) Yijun Lin, Yao-Yi Chiang, Meredith Franklin, Sandrah P. Eckel, and José Luis Ambite. 2020.  Building Autocorrelation-Aware Representations for Fine-Scale Spatiotemporal Prediction.  In _2020 IEEE International Conference on Data Mining (ICDM)_ , pages 352–361.
  * Lin et al. (2017) Yijun Lin, Yao-Yi Chiang, Fan Pan, Dimitrios Stripelis, Jose Luis Ambite, Sandrah P Eckel, and Rima Habre. 2017.  Mining Public Datasets for Modeling Intra-City PM2.5 Concentrations at a Fine Spatial Resolution.  In _Proceedings of the 25th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems_ , volume 8, pages 1–10, New York, NY, USA. ACM.
  * Lin et al. (2018) Yijun Lin, Nikhit Mago, Yu Gao, Yaguang Li, Yao-Yi Chiang, Cyrus Shahabi, and José Luis Ambite. 2018.  Exploiting spatiotemporal patterns for accurate air quality forecasting using deep learning.  In _Proceedings of the 26th ACM SIGSPATIAL International Conference on Advances in Geographic Information Systems_ , SIGSPATIAL ’18, pages 359–368, New York, NY, USA. Association for Computing Machinery.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Liu et al. (2021) Zhuang Liu, Degen Huang, Kaiyu Huang, Zhuang Li, and Jun Zhao. 2021.  FinBERT: A Pre-trained Financial Language Representation Model for Financial Text Mining.  In _Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence_ , pages 4513–4519.
  * Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019.  Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks.  In _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc.
  * Peng et al. (2019) Yifan Peng, Shankai Yan, and Zhiyong Lu. 2019.  Transfer learning in biomedical natural language processing: An evaluation of BERT and ELMo on ten benchmarking datasets.  In _Proceedings of the 18th BioNLP Workshop and Shared Task_ , pages 58–65, Florence, Italy. Association for Computational Linguistics.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep contextualized word representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 2227–2237, New Orleans, Louisiana. Association for Computational Linguistics.
  * Peters et al. (2019) Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019.  Knowledge enhanced contextual word representations.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 43–54, Hong Kong, China. Association for Computational Linguistics.
  * Phan et al. (2021) Long N Phan, James T Anibal, Hieu Tran, Shaurya Chanana, Erol Bahadroglu, Alec Peltekian, and Grégoire Altan-Bonnet. 2021.  Scifive: a text-to-text transformer model for biomedical literature.  _arXiv preprint arXiv:2106.03598_.
  * Qiu et al. (2020) Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020\.  Pre-trained models for natural language processing: A survey.  _arXiv preprint arXiv:2003.08271_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21(140):1–67.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.
  * Rothe et al. (2020) Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020.  Leveraging pre-trained checkpoints for sequence generation tasks.  _Transactions of the Association for Computational Linguistics_ , 8:264–280.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_.
  * Su et al. (2019) Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019\.  VL-BERT: Pre-training of Generic Visual-Linguistic Representations.  In _International Conference on Learning Representations_.
  * Tabassum et al. (2020) Jeniya Tabassum, Mounica Maddela, Wei Xu, and Alan Ritter. 2020.  Code and named entity recognition in StackOverflow.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4913–4926, Online. Association for Computational Linguistics.
  * Trabelsi et al. (2022) Mohamed Trabelsi, Zhiyu Chen, Shuo Zhang, Brian D Davison, and Jeff Heflin. 2022\.  StruBERT: Structure-aware BERT for Table Search and Matching.  In _Proceedings of the ACM Web Conference 2022_ , pages 442–451.
  * Wang et al. (2022) Fei Wang, Zhewei Xu, Pedro Szekely, and Muhao Chen. 2022.  Robust (controlled) table-to-text generation with structure-aware equivariance learning.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5037–5048, Seattle, United States. Association for Computational Linguistics.
  * Wang et al. (2021) Ruize Wang, Duyu Tang, Nan Duan, Zhongyu Wei, Xuanjing Huang, Jianshu Ji, Guihong Cao, Daxin Jiang, and Ming Zhou. 2021.  K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_ , pages 1405–1418, Online. Association for Computational Linguistics.
  * Wei et al. (2021) Xiaokai Wei, Shen Wang, Dejiao Zhang, Parminder Bhatia, and Andrew Arnold. 2021\.  Knowledge enhanced pretrained language models: A compreshensive survey.  _arXiv preprint arXiv:2110.08455_.
  * Yamada et al. (2020) Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020\.  LUKE: Deep contextualized entity representations with entity-aware self-attention.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6442–6454, Online. Association for Computational Linguistics.
  * Yang et al. (2019) Sen Yang, Dawei Feng, Linbo Qiao, Zhigang Kan, and Dongsheng Li. 2019.  Exploring pre-trained language models for event extraction and generation.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5284–5294, Florence, Italy. Association for Computational Linguistics.
  * Yin et al. (2017) Hongzhi Yin, Weiqing Wang, Hao Wang, Ling Chen, and Xiaofang Zhou. 2017.  Spatial-aware hierarchical collaborative deep learning for poi recommendation.  _IEEE Transactions on Knowledge and Data Engineering_ , 29(11):2537–2551.
  * Yin et al. (2020) Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020.  TaBERT: Pretraining for joint understanding of textual and tabular data.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8413–8426, Online. Association for Computational Linguistics.
  * Yu et al. (2022) Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. 2022.  Scaling Autoregressive Models for Content-Rich Text-to-Image Generation.
  * Yuan and Li (2021) Haitao Yuan and Guoliang Li. 2021.  A survey of traffic prediction: from spatio-temporal data to intelligent transportation.  _Data Science and Engineering_ , 6(1):63–85.
  * Yue et al. (2019) M Yue, Y Li, H Yang, R Ahuja, Y Chiang, and C Shahabi. 2019.  DETECT: Deep Trajectory Clustering for Mobility-Behavior Analysis.  In _Proceedings of the 2019 IEEE International Conference on Big Data (Big Data)_ , pages 988–997.
  * Yue et al. (2021) Mingxuan Yue, Yao-Yi Chiang, and Cyrus Shahabi. 2021.  VAMBC: A Variational Approach for Mobility Behavior Clustering.  In _Machine Learning and Knowledge Discovery in Databases. Applied Data Science Track_ , pages 453–469. Springer International Publishing.
  * Zhai et al. (2022) Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. 2022.  LiT: Zero-Shot Transfer With Locked-Image Text Tuning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 18123–18133.
  * Zhang et al. (2020) Bowen Zhang, Hexiang Hu, Vihan Jain, Eugene Ie, and Fei Sha. 2020.  Learning to represent image and text with denotation graph.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 823–839, Online. Association for Computational Linguistics.
  * Zhang et al. (2021) Bowen Zhang, Hexiang Hu, Linlu Qiu, Peter Shaw, and Fei Sha. 2021.  Visually grounded concept composition.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 201–215, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zhang et al. (2019) Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019.  ERNIE: Enhanced language representation with informative entities.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 1441–1451, Florence, Italy. Association for Computational Linguistics.
  * Zhao et al. (2022) Pengpeng Zhao, Anjing Luo, Yanchi Liu, Jiajie Xu, Zhixu Li, Fuzhen Zhuang, Victor S. Sheng, and Xiaofang Zhou. 2022.  Where to Go Next: A Spatio-Temporal Gated Network for Next POI Recommendation.  _IEEE Transactions on Knowledge and Data Engineering_ , 34(5):2512–2524.
  * Zhou et al. (2020) Guangyu Zhou, Muhao Chen, Chelsea JT Ju, Zheng Wang, Jyun-Yu Jiang, and Wei Wang. 2020.  Mutation effect estimation on protein–protein interactions using deep contextualized representation learning.  _NAR genomics and bioinformatics_ , 2(2):lqaa015.
  * Zhou et al. (2022) Wenxuan Zhou, Fangyu Liu, Ivan Vulić, Nigel Collier, and Muhao Chen. 2022.  Prix-LM: Pretraining for multilingual knowledge base construction.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5412–5424, Dublin, Ireland. Association for Computational Linguistics.


# High fidelity state preparation and measurement of ion hyperfine qubits with
$I>\frac{1}{2}$

Fangzhao Alex An<EMAIL_ADDRESS>Quantinuum, Golden Valley, MN,
USA  Anthony Ransford<EMAIL_ADDRESS>Quantinuum, Broomfield,
CO, USA  Andrew Schaffer  Lucas R. Sletten  Quantinuum, Golden Valley, MN, USA
John Gaebler  Quantinuum, Broomfield, CO, USA  James Hostetter  Grahame
Vittorini  Quantinuum, Golden Valley, MN, USA

###### Abstract

We present a method for achieving high fidelity state preparation and
measurement (SPAM) using trapped ion hyperfine qubits with nuclear spins
higher than $I=1/2$. The ground states of these higher nuclear spin isotopes
do not afford a simple frequency-selective state preparation scheme. We
circumvent this limitation by stroboscopically driving strong and weak
transitions, blending fast optical pumping using E1 transitions and narrow
microwave or optical E2 transitions. We demonstrate this method with the
$I=3/2$ isotope ${}^{137}\mbox{Ba}^{+}$ to achieve a SPAM infidelity of
9.6(1.4)$\times 10^{-5}$ ($-40.2(0.6)$ dB), facilitating the use of a wider
range of ion isotopes with favorable wavelengths and masses for quantum
computation.

##  I Introduction

State preparation and measurement (SPAM) is fundamental to quantum computation
and covers two of the five necessary DiVincenzo criteria [1]. Modern quantum
systems are not error corrected, and the fidelity of an uncorrected $N$-qubit
register typically decreases exponentially with size as
$(\mathcal{F}_{\mathrm{SPAM}})^{N}$, where $\mathcal{F}_{\mathrm{SPAM}}$ is
the single-qubit SPAM fidelity. Currently implemented error correction codes
require mid-circuit measurement and reset (MCMR) and thus SPAM errors
contribute to the error correction budget as the ratio of gates to
measurements [2]. Given the historic difficulty of improving gate fidelity, it
will likely be desirable to make the SPAM contribution negligible and thus on
par with the gate fidelity. Moreover, the speed at which SPAM can be performed
can affect memory errors during MCMR, and the compatibility of the exact SPAM
technique with other qubit operations must also be considered. More broadly,
it is desirable to have fast, high-fidelity SPAM in ionic qubits with other
practical advantages to quantum computation such as light mass and visible
wavelengths for gates.

Although the ${}^{171}\mbox{Yb}^{+}$ ion boasts a hyperfine clock qubit and
its $I=1/2$ nuclear spin affords it a singlet state for simple, high fidelity
state preparation [3, 4], the limitations on measurement fidelity, necessity
for high power UV light, and associated photo-induced charging [5] present
significant engineering challenges. Other systems with similar structure
(nuclear spin $I=1/2$) have been explored, with the most promising being
${}^{133}\mbox{Ba}^{+}$ [6, 7] due to its visible wavelength transitions, but
these species either require lasers deep in the UV or are not naturally
occurring, presenting formidable challenges for scaling systems.

Figure 1:  (a) Microwave-assisted optical pumping (MAOP) using coherent
microwave pulses (black) and 493 nm flush pulses (aqua). (b) 1762 nm narrow-
band optical pumping (NBOP) using coherent 1762 nm pulses (gray), 614 nm
quenching (orange), and 493 nm flush (aqua). (c) Cabinet shelving with
repeated 1762 nm pulses addressing $|0\rangle$. Hyperfine quantum numbers $F$
and $m_{F}$ are omitted for clarity (see Supplementary Materials for detailed
labeling and timing diagram [8]).

Despite their complicated structure, ions with nuclear spin $I>1/2$ have
advantages [9] and are used in quantum information and quantum computing,
including ${}^{9}\mbox{Be}^{+}$ [10], ${}^{43}\mbox{Ca}^{+}$ [11],
${}^{25}\mbox{Mg}^{+}$ [12] and ${}^{137}\mbox{Ba}^{+}$ [13]. Currently, all
methods of state preparation in these species are limited in performance by
their requirement for either light with high polarization purity or a series
of coherent gate operations to map a prepared state to the clock qubit state,
where this transfer infidelity shows up as a preparation error [14, 10].

Species | $I$ | $\eta_{{}_{F^{\prime}+1,F+1}}$ | $\eta_{{}_{F^{\prime},F+1}}$ | $\frac{\Gamma}{2\pi}(\mathrm{MHz})$ | Hyperfine S (GHz) | Hyperfine P (GHz) | $\epsilon_{\mathrm{prep}}$
---|---|---|---|---|---|---|---
${}^{9}\text{Be}^{+}$ | 3/2 | 1/2 | 5/6 | 22.4 [15] | 1.25 [16] | 0.194 [17] | 5.4e-4
${}^{25}\text{Mg}^{+}$ | 5/2 | 4/9 | 7/9 | 42.4 [18] | 1.788 [18] | 0.307 [18] | 1.0e-3
${}^{43}\text{Ca}^{+}$ | 7/2 | 5/12 | 3/4 | 22.4 [19] | 3.226 [20] | 0.581 [20] | 8.9e-5
${}^{87}\text{Sr}^{+}$ | 9/2 | 2/5 | 11/15 | 21.5 [21] | 5.0 [22] | 0.89 [22] | 3.4e-5
${}^{135}\text{Ba}^{+}$ | 3/2 | 1/2 | 5/6 | 20.1 [23] | 7.18 [24] | 1.33 [24] | 1.4e-5
${}^{137}\text{Ba}^{+}$ | 3/2 | 1/2 | 5/6 | 20.1 [23] | 8.03 [13] | 1.49 [13] | 1.1e-5
${}^{173}\text{Yb}^{+}$ | 5/2 | 4/9 | 7/9 | 19.7 [25] | 10.5 [26] | 1.85 [27] | 6.3e-6
Table 1: Predicted state preparation infidelities in commonly used ion
species. The infidelity generally decreases with mass with the exception of
25Mg+, which has a transition linewidth that is two times broader.

In this work, we prepare a single ${}^{137}\mbox{Ba}^{+}$ ion into the qubit
state $|0\rangle\equiv\Ket{6\text{S}_{1/2},F=1,m_{F}=0}$ without the need of
highly polarized light or coherent mapping of the qubit to the clock state. By
alternating between standard optical pumping on a dipole transition and
driving narrow, state-selective microwave or optical E2 transitions, we are
able to address and minimize the state preparation error in the Zeeman
sublevels of the ground state $6\text{S}_{1/2}$. This technique, generalizable
to many $I>1/2$ species, allows us to achieve the highest SPAM fidelity
recorded with any qubit to the best of our knowledge (previously
${}^{171}\mbox{Yb}^{+}$ [28]).

We present two varieties of the state preparation scheme, both of which are
cyclic in nature and seek to reduce state preparation errors in all Zeeman
sublevels of $\text{S}_{1/2}$ except $\ket{0}$. The microwave scheme is shown
in Fig. 1(a), and is similar to the microwave-assisted optical pumping (MAOP)
recently performed on an atomic ensemble [29]. We first use dipole optical
pumping at $493$ nm to “flush” out errors from the $(\text{S}_{1/2},F=2)$
manifold, redistributing the errors in the Zeeman sublevels of
$(\text{S}_{1/2},F=1)$. Next, errors in the non-qubit Zeeman sublevels of the
$(\text{S}_{1/2},F=1)$ manifold are moved up to $F=2$ with microwave
$\pi$-pulses near the hyperfine splitting, and are subsequently flushed by the
$493$ nm light when the cycle repeats. Alternatively, the errors in
$(\text{S}_{1/2},F=1)$ can be addressed with narrow-band optical pumping
(NBOP) [28] to $\text{D}_{5/2}$ on the 1762 nm E2 transition, and then
redistributed back into the $(\text{S}_{1/2},F=1)$ manifold with a $614$ nm
pulse (Fig. 1(b)).

State preparation with either MAOP or NBOP can be generalized to different ion
species with high nuclear spin, and we model MAOP state preparation for a
general alkaline earth ion with a dipole transition from S (hyperfine states
$F,F+1$) to P ($F^{\prime},F^{\prime}+1$) [8]. In the limit of many cycles and
low flush beam power, we derive the state preparation error to be

$\displaystyle\epsilon_{\mathrm{prep}}=\frac{\Gamma^{2}}{2}\Bigg{[}\frac{1}{\delta_{{}_{\mathrm{HF},\mathrm{S}}}^{2}}+\frac{\eta_{{}_{F^{\prime},F+1}}}{\eta_{{}_{F^{\prime}+1,F+1}}}\frac{1}{\delta_{-}^{2}}\Bigg{]},$
(1)

where $\Gamma$ is the transition linewidth, $\delta_{\mathrm{HF},\mathrm{S}}$
is the ground state hyperfine splitting, and $\eta_{i,j}$ denotes the
branching from state $i$ to state $j$. The first term is identical to twice
the limit of state preparation in an $I=1/2$ ion and the second term is a
correction factor that depends on the branching ratio and hyperfine splitting
difference between the ground and excited states $\delta_{-}$. The parameters
for Eq. (1) and the associated fidelities for various $I>1/2$ hyperfine qubits
are shown in Table 1.

There are a variety of high fidelity measurement schemes in general from
quantum logic and quantum non-demolition [30] to coherent and incoherent
shelving. For ${}^{137}\mathrm{Ba}^{+}$ we use a “cabinet shelving” procedure
shown in Fig. 1(c) where $\ket{0}$ is shelved three times in succession to
multiple Zeeman sublevels in the metastable $\text{D}_{5/2}$ state ($30$ s
lifetime [31]), and then we apply resonant 493 nm light for $350~{}\mu$s to
detect fluorescence. The resulting bright and dark state histograms are well
separated and the SPAM infidelity for the 1762 nm scheme is measured to be
$9.6(1.4)\times 10^{-5}$ with a simple discriminator (Fig. 2(c)). The data is
taken without any pre- or post-processing, such as discarding trials based on
low fluorescence counts during Doppler cooling. We do not perform statistical
detection of D-state decays for our SPAM fidelity [15], but consider the
effect in the error budget.

In this experiment, we trap single ${}^{137}\mathrm{Ba}^{+}$ ions 70 $\mu$m
above a planar surface trap similar to that described in Ref. [32], and apply
a magnetic field of $4.96$ G to define the quantization axis. Single qubit
rotations between the states $\ket{0}$ and
$\ket{1}\equiv\ket{\text{S}_{1/2},2,0}$ are driven by a microwave horn tuned
near the transition frequency of $8.038$ GHz. The ion is Doppler cooled by
cycling through the $\text{S}_{1/2}\leftrightarrow\text{P}_{1/2}$ (493 nm
cycling) and $\text{D}_{3/2}\leftrightarrow\text{P}_{1/2}$ (650 nm repump)
transitions. To address the many hyperfine levels of 137Ba+, frequency
sidebands are applied to all of our lasers with electro-optic modulators
(EOMs) [8]. The 493 nm cycling light is wavemeter locked to the
$F=2\leftrightarrow F=2$ ($2\leftrightarrow 2$) hyperfine transition, and an
EOM applies sidebands to address the other possible hyperfine transitions
($1\leftrightarrow 1$, $1\leftrightarrow 2$, and $2\leftrightarrow 1$). For
state preparation and measurement, we address the narrow shelving transition
$\text{S}_{1/2}\leftrightarrow\text{D}_{5/2}$ with a $1762$ nm laser locked to
a high-finesse optical cavity. This laser is red-detuned by $88$ MHz from the
$\ket{\text{S}_{1/2},1,0}\rightarrow\ket{\text{D}_{5/2},3,2}$ transition to
avoid off-resonant shelving, and different shelving transitions are addressed
with sidebands generated by an EOM. A $614$ nm laser is used to deshelve ions
from $\text{D}_{5/2}$ to $\text{P}_{3/2}$, from which ions preferentially fall
into $\text{S}_{1/2}$.

Figure 2: (a) SPAM infidelity vs. increasing cycles of microwave-assisted
(purple, MAOP) and narrow-band (green, NBOP) optical pumping. Infidelities are
calculated from datasets with $50\times 10^{3}$ (for 0 to 9 cycles),
$100\times 10^{3}$ (10 to 18 cycles), and $200\times 10^{3}$ (20 to 30 cycles)
trials. Both techniques are initially close to 1/3 reduction in error per
cycle (gray line), but level off at high fidelity. Solid points and
fluorescence count histograms represent separate datasets for (b) MAOP (35
cycles) and (c) 1762 nm NBOP (30 cycles with 493 nm flush pulse and 5 cycles
without flush), preparing the $\ket{0}$ (orange) and $\ket{1}$ (blue) qubit
state over $10^{6}$ trials per state. The SPAM infidelities were measured to
be $(14.8\pm 1.7)\times 10^{-5}$ (microwaves) and $(9.6\pm 1.4)\times 10^{-5}$
(1762 nm) using state detection thresholds indicated by vertical lines. All
error bars denote one Wilson interval.

We begin our state preparation with a fast ($40~{}\mu$s) polarization-limited
step. We address the $(\text{S}_{1/2},F=2)\rightarrow(\text{P}_{1/2},F=2)$
transition with $\pi$-polarized $493$ nm light and stroboscopically apply
sidebands to address the $2\leftrightarrow 1$ and $1\leftrightarrow 1$
transitions (see Supplementary Materials for exact pulse sequences [8]).
Because of the forbidden selection rule
$\ket{\text{S}_{1/2},1,0}\nleftrightarrow\ket{\text{P}_{1/2},1,0}$, this
stochastically pumps the population into the qubit $\ket{0}$ state. The
stroboscopic pulsing prevents frequency mixing of the sidebands by the EOM
which drives the unwanted $1\leftrightarrow 2$ transition and could thus drive
population out of the $\ket{0}$ state. This process typically achieves state
preparation infidelities of $\sim$$7\times 10^{-3}$, where the fidelity is
limited by polarization impurities and the laser orientation with respect to
the magnetic field [8]. While one could attempt to improve these
imperfections, it is technically difficult given vacuum window birefringence,
finite quality retarders, and incomplete magnetic field control, all of which
will be further exacerbated in a larger system with many spatially-separated
qubits.

After polarization-limited state preparation, some unwanted population is left
as error in the remaining Zeeman sublevels of $\text{S}_{1/2}$. To flush out
the leakage population in the $F=2$ manifold, we apply a weak pulse ($1$
$\mu$s, $40$ mW/cm2) of 493 nm light carrying all polarizations and tuned to
$(\text{S}_{1/2},F=2)\rightarrow(\text{P}_{1/2},F=2)$ to pump population
equally into the three Zeeman sublevels of $(\text{S}_{1/2},F=1)$. During this
pulse, $650$ nm light is also applied to pump population out of the
$\text{D}_{3/2}$ manifold. Next, to address the leakage population in
$\ket{\text{S}_{1/2},1,\pm 1}$, we can use either microwave pulses to
$\Ket{\text{S}_{1/2},2,\pm 1}$ or shelving pulses to
$\ket{\text{D}_{5/2},1,\mp 1}$. Under MAOP, two successive $\pi$-pulses
$\Ket{\text{S}_{1/2},1,\pm 1}\rightarrow\Ket{\text{S}_{1/2},2,\pm 1}$
($\sim$$50~{}\mu$s) are applied to bring population into $F=2$ in preparation
for another 493 nm flush pulse. Each cycle of this procedure should ideally
transfer the errors out of $F=1,m_{F}=\pm 1$ with microwaves, and then
redistribute those errors equally back into the $F=1$ manifold with the $493$
nm flush pulse, leading to a $1/3$ reduction in error per cycle. Since the
microwave transitions are well separated from other transitions at our Rabi
rates ($2\pi\times 10$ kHz), off-resonant excitations of the qubit state by
the 493 nm enforce a more fundamental limit.

For NBOP, we apply two $\pi$-pulses of 1762 nm light
$\Ket{\text{S}_{1/2},1,\pm 1}\rightarrow\Ket{\text{D}_{5/2},1,\mp 1}$ ($45$,
$55~{}\mu$s) to pump error into the $\text{D}_{5/2}$ state. The orientation
and polarization of the 1762 nm laser allows for only $\Delta m_{F}=\pm 2$
transitions, meaning that any accidental shelving of the qubit state is
suppressed by frequency separation, polarization, and laser $k$-vector. Next,
we apply a $4~{}\mu$s pulse of 614 nm light to address the dipole transition
$(\text{D}_{5/2},F=1)\rightarrow(\text{P}_{3/2},F=0)$, from which population
decays to the $F=1$ of the $\text{S}_{1/2}$ state with branching 73.8% (zero
branching to $F=2$) [33]. To address any population that falls back into
$\text{D}_{5/2}$, we apply sidebands to the 614 nm light that march the
population to $(\text{P}_{3/2},F=0)$: $2\rightarrow 1$, $3\rightarrow 2$, and
$4\rightarrow 3$. Similarly, 650 nm light is applied to repump population that
falls into $\text{D}_{3/2}$. In contrast to MAOP, this scheme does not
explicitly pump population into the $(\text{S}_{1/2},F=2)$ manifold, so we
need not apply the 493 nm $F=2$ flush beam as frequently, lowering the error
from 493 nm off-resonant excitations.

To characterize SPAM fidelity, we must also prepare
$\ket{1}\equiv\ket{\text{S}_{1/2},2,0}$ by applying a composite microwave
$\pi$-pulse (CP Robust 180 sequence [6, 34]) at the qubit frequency after
preparing $\ket{0}$. To distinguish between the two qubit states, we cabinet
shelve population in $\ket{0}$ with three $\pi$-pulses to the states
$\ket{\text{D}_{5/2},3,2},\ket{\text{D}_{5/2},2,\pm 2}$ (165, 45, 55 $\mu$s).
Each of these pulses is limited to a fidelity of only $\sim$0.99 due to
magnetic field and laser intensity noise, but with three pulses we are able to
shelve with an implied infidelity of $1\times 10^{-6}$. After shelving, $493$
nm and $650$ nm light are applied with all sidebands to detect any population
in $\text{S}_{1/2}$, $\text{P}_{1/2}$, and $\text{D}_{3/2}$.

Figure 2(a) presents the SPAM performance of both state preparation procedures
over a variable number of cycles including the $493$ nm flush beam. After
$>$$25$ cycles of either MAOP or NBOP, we observe a drop in the SPAM
infidelity of nearly two orders of magnitude compared to using polarization
state preparation alone. Both state preparation schemes initially exhibit
close to the expected 1/3 reduction in error (gray line), but poor shelving
due to ion heating as well as off-resonant scattering errors from the $493$ nm
flush beam cause the performance to deviate at large cycle number.

Error Source | Predicted Error ($\times 10^{-5})$
---|---
$\ket{0}$ State |  $\ket{1}$ State
$\ket{0}$ state preparation |  $<$$9.1$ | —
$\ket{0}\rightarrow\ket{1}$ transfer | — | $5.4\pm 0.5$
Shelving infidelity |  $\sim$$0.1$ | —
$\text{D}_{5/2}$ decay |  $\sim$$1.5$ | —
Correlated errors | $4.0$ | —
Subtotal (measured) | $14.7\pm 2.4$ | $4.2\pm 1.3$
Total (measured) | $9.6\pm 1.4$
Table 2: Error budget for 1762 nm narrow-band optical pumping SPAM results of
Fig. 2(c).

The performance of MAOP and NBOP appear comparable up until $30$ cycles with
the flush pulse, but the 1762 nm scheme proves superior in a larger dataset
($10^{6}$ trials) at $35$ cycles, where we omit the $493$ flush pulse in the
final five cycles of NBOP. The fluorescence count histograms for this $35$
cycle data are shown in Fig. 2(b-c), and using the same discriminator
(vertical dashed line) we extract SPAM infidelities of $\left(1.48\pm
0.17\right)\times 10^{-4}$ (MAOP) and $(9.6\pm 1.4)\times 10^{-5}$ (NBOP). We
examine the SPAM errors of the higher fidelity NBOP result in more detail in
Table 2. Some errors are caused by shelved ions spontaneously decaying from
the $\text{D}_{5/2}$ state into states bright to the measurement light, and
these can be reduced by shortening our measurement time and increasing the
shelving Rabi rates. The majority of the errors that occur when preparing
$\ket{1}$ are due to an imperfect microwave transfer pulse between $\ket{0}$
and $\ket{1}$, and we find that they are within error of an independent
measurement of the microwave infidelity $(5.4\pm 0.5)\times 10^{-5}$ using
single-qubit randomized benchmarking [35]. Other errors include collision-
induced correlated errors which we detail further in the Supplementary
Materials [8].

Our present implementation of these state preparation techniques takes about
$3.5$ ms for 35 cycles using $\sim$50 $\mu$s microwave and 1762 nm
$\pi$-times. These long durations can result in hot ions, limiting achievable
fidelities mainly through failed $\text{D}_{5/2}$ state shelving. Moreover,
this timescale is significantly longer than typical ion qubit gate times of
$50~{}\mu$s, impacting overall circuit runtime. We can reduce the required
time for either scheme by performing both $\pi$-pulses in parallel.
Furthermore, if we allow an error of $10^{-5}$ due to off-resonant excitation
of $\ket{0}$, then the minimum allowed microwave and $1762$ nm $\pi$-times are
$12.5~{}\mu$s and $2.5~{}\mu$s, respectively. We project that these
improvements could cut down the state preparation time to $450~{}\mu$s
(microwave) and $250~{}\mu$s (1762 nm).

In this work, we have achieved the highest reported SPAM fidelity of any qubit
using a ${}^{137}\mbox{Ba}^{+}$ ion with nuclear spin $I=3/2$. Our two similar
state preparation techniques are generalizable to many ion species with higher
nuclear spin $I>1/2$, facilitating quantum computation with new ion species
with benefits like visible wavelengths and lighter masses.

###### Acknowledgements.

The authors would like to thank Stephen Erickson, Chris Gilbreth, Colin
Kennedy, Conrad Roman, and Jonathan Sedlacek for their suggestions and advice,
as well as the rest of the Quantinuum team for their contributions.

## References

  * DiVincenzo [2000] D. P. DiVincenzo, Fortschr. Phys. 48, 771 (2000).
  * Ryan-Anderson  _et al._ [2021] C. Ryan-Anderson, J. G. Bohnet, K. Lee, D. Gresh, A. Hankin, J. P. Gaebler, D. Francois, A. Chernoguzov, D. Lucchetti, N. C. Brown, T. M. Gatterman, S. K. Halit, K. Gilmore, J. A. Gerber, B. Neyenhuis, D. Hayes, and R. P. Stutz, Phys. Rev. X 11, 041058 (2021).
  * Olmschenk  _et al._ [2007] S. Olmschenk, K. C. Younge, D. L. Moehring, D. N. Matsukevich, P. Maunz, and C. Monroe, Phys. Rev. A 76, 052314 (2007).
  * Crain  _et al._ [2019] S. Crain, C. Cahall, G. Vrijsen, E. E. Wollman, M. D. Shaw, V. B. Verma, S. W. Nam, and J. Kim, Commun. Phys. 2, 97 (2019).
  * Wang  _et al._ [2011] S. X. Wang, G. Hao Low, N. S. Lachenmyer, Y. Ge, P. F. Herskind, and I. L. Chuang, J. Appl. Phys. 110, 104901 (2011).
  * Christensen  _et al._ [2020] J. E. Christensen, D. Hucul, W. C. Campbell, and E. R. Hudson, npj Quantum Inf. 6, 35 (2020).
  * Lee [2006] P. J. Lee,  _Quantum information processing with two trapped cadmium ions_ , Ph.D. thesis, University of Michigan (2006).
  * [8] See the Supplementary Materials for more detailed level diagram of ${}^{137}\mathrm{Ba}^{+}$, derivation of state preparation error, details on initial polarization state preparation, specifics on SPAM error budget, and timing diagrams for state preparation techniques.
  * Allcock  _et al._ [2021] D. T. C. Allcock, W. C. Campbell, J. Chiaverini, I. L. Chuang, E. R. Hudson, I. D. Moore, A. Ransford, C. Roman, J. M. Sage, and D. J. Wineland, Appl. Phys. Lett. 119, 214002 (2021).
  * Gaebler  _et al._ [2016] J. P. Gaebler, T. R. Tan, Y. Lin, Y. Wan, R. Bowler, A. C. Keith, S. Glancy, K. Coakley, E. Knill, D. Leibfried, and D. J. Wineland, Phys. Rev. Lett. 117, 060505 (2016).
  * Ballance  _et al._ [2016] C. J. Ballance, T. P. Harty, N. M. Linke, M. A. Sepiol, and D. M. Lucas, Phys. Rev. Lett. 117, 060504 (2016).
  * Tan  _et al._ [2015] T. R. Tan, J. P. Gaebler, Y. Lin, Y. Wan, R. Bowler, D. Leibfried, and D. J. Wineland, Nature 528, 380 (2015).
  * Dietrich  _et al._ [2010] M. R. Dietrich, N. Kurz, T. Noel, G. Shu, and B. B. Blinov, Phys. Rev. A 81, 052328 (2010).
  * Harty  _et al._ [2014] T. P. Harty, D. T. C. Allcock, C. J. Ballance, L. Guidoni, H. A. Janacek, N. M. Linke, D. N. Stacey, and D. M. Lucas, Phys. Rev. Lett. 113, 220501 (2014).
  * Langer [2006] C. E. Langer,  _High fidelity quantum information processing with trapped ions_ , Ph.D. thesis, University of Colorado at Boulder (2006).
  * Wineland  _et al._ [1983] D. J. Wineland, J. J. Bollinger, and W. M. Itano, Phys. Rev. Lett. 50, 628 (1983).
  * Bollinger  _et al._ [1985] J. J. Bollinger, J. S. Wells, D. J. Wineland, and W. M. Itano, Phys. Rev. A 31, 2711 (1985).
  * Nguyen [2009] J. Nguyen,  _The linewidth and hyperfine A constant of the ${}^{2}{P}_{1/2}$ state of a magnesium ion confined in a linear Paul trap_, Ph.D. thesis, McMaster University (2009).
  * James [1998] D. F. V. James, Appl. Phys. B 66, 181 (1998).
  * Lucas  _et al._ [2004] D. M. Lucas, A. Ramos, J. P. Home, M. J. McDonnell, S. Nakayama, J.-P. Stacey, S. C. Webster, D. N. Stacey, and A. M. Steane, Phys. Rev. A 69, 012711 (2004).
  * Pinnington  _et al._ [1995] E. H. Pinnington, R. W. Berends, and M. Lumsden, J. Phys. B: At. Mol. Opt. Phys. 28, 2095 (1995).
  * Yuan  _et al._ [1995] X. Yuan, S. N. Panigrahy, R. W. Dougherty, T. P. Das, and J. Andriessen, Phys. Rev. A 52, 197 (1995).
  * Christensen [2020] J. Christensen,  _High-fidelity operation of a radioactive trapped-ion qubit ${}^{133}\mathrm{Ba}^{+}$_, Ph.D. thesis, University of California Los Angeles (2020).
  * Villemoes  _et al._ [1993] P. Villemoes, A. Arnesen, F. Heijkenskjold, and A. Wannstrom, J. Phys. B: At. Mol. Opt. Phys. 26, 4289 (1993).
  * Olmschenk  _et al._ [2009] S. Olmschenk, D. Hayes, D. N. Matsukevich, P. Maunz, D. L. Moehring, K. C. Younge, and C. Monroe, Phys. Rev. A 80, 022502 (2009).
  * Münch  _et al._ [1987] A. Münch, M. Berkler, C. Gerz, D. Wilsdorf, and G. Werth, Phys. Rev. A 35, 4147 (1987).
  * Roman [2021] C. H. Roman,  _Expanding the ${}^{171}\mathrm{Yb}^{+}$ toolbox: the ${}^{2}F_{7/2}^{o}$ state as resource for quantum information science_, Ph.D. thesis, University of California Los Angeles (2021).
  * Ransford  _et al._ [2021] A. Ransford, C. Roman, T. Dellaert, P. McMillin, and W. C. Campbell, Phys. Rev. A 104, L060402 (2021).
  * Tretiakov  _et al._ [2021] A. Tretiakov, C. Potts, Y. Lu, J. Davis, and L. LeBlanc, arXiv preprint arXiv:2110.10673 (2021).
  * Erickson  _et al._ [2021] S. D. Erickson, J. J. Wu, P.-Y. Hou, D. C. Cole, S. Geller, A. Kwiatkowski, S. Glancy, E. Knill, D. H. Slichter, A. C. Wilson, _et al._ , arXiv preprint arXiv:2112.06341 (2021).
  * Zhang  _et al._ [2020] Z. Zhang, K. J. Arnold, S. R. Chanu, R. Kaewuam, M. S. Safronova, and M. D. Barrett, Phys. Rev. A 101, 062515 (2020).
  * Pino  _et al._ [2021] J. M. Pino, J. M. Dreiling, C. Figgatt, J. P. Gaebler, S. A. Moses, M. S. Allman, C. H. Baldwin, M. Foss-Feig, D. Hayes, K. Mayer, C. Ryan-Anderson, and B. Neyenhuis, Nature 592, 209 (2021).
  * Dutta  _et al._ [2016] T. Dutta, D. De Munshi, D. Yum, R. Rebhi, and M. Mukherjee, Sci. Rep. 6, 29772 (2016).
  * Ryan  _et al._ [2010] C. A. Ryan, J. S. Hodges, and D. G. Cory, Phys. Rev. Lett. 105, 200402 (2010).
  * Knill  _et al._ [2008] E. Knill, D. Leibfried, R. Reichle, J. Britton, R. B. Blakestad, J. D. Jost, C. Langer, R. Ozeri, S. Seidelin, and D. J. Wineland, Phys. Rev. A 77, 012307 (2008).