# SKILL: Structured Knowledge Infusion for Large Language Models

Fedor Moiseev 1,2 Zhe Dong 2 Enrique Alfonseca 2 Martin Jaggi 1
1EPFL, Switzerland 2Google, Switzerland
{femoiseev, zhedong<EMAIL_ADDRESS><EMAIL_ADDRESS>Work done
during internship at Google. Correspondence Author.

###### Abstract

Large language models (LLMs) have demonstrated human-level performance on a
vast spectrum of natural language tasks. However, it is largely unexplored
whether they can better internalize knowledge from a structured data, such as
a knowledge graph, or from text. In this work, we propose a method to infuse
structured knowledge into LLMs, by directly training T5 models on factual
triples of knowledge graphs (KGs). We show that models pre-trained on Wikidata
KG with our method outperform the T5 baselines on FreebaseQA and WikiHop, as
well as the Wikidata-answerable subset of TriviaQA and NaturalQuestions. The
models pre-trained on factual triples compare competitively with the ones on
natural language sentences that contain the same knowledge. Trained on a
smaller size KG, WikiMovies, we saw $3\times$ improvement of exact match score
on MetaQA task compared to T5 baseline. The proposed method has an advantage
that no alignment between the knowledge graph and text corpus is required in
curating training data. This makes our method particularly useful when working
with industry-scale knowledge graphs.

SKILL: Structured Knowledge Infusion for Large Language Models

Fedor Moiseev††thanks:  Work done during internship at Google. 1,2 Zhe
Dong††thanks:  Correspondence Author. 2 Enrique Alfonseca 2 Martin Jaggi 1
1EPFL, Switzerland 2Google, Switzerland {femoiseev, zhedong,
<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

Large pre-trained language models, such as BERT (Devlin et al., 2019), GPT-3
(Brown et al., 2020), T5 (Raffel et al., 2019), REALM (Guu et al., 2020) and
ERNIE (Sun et al., 2021) have become the state-of-the-art technology for many
tasks. They are commonly pre-trained using unstructured text corpora, on tasks
such as next word prediction, next sentence prediction (NSP) or masked
language modelling (MLM). Especially for T5, self-supervised learning on
unlabelled text corpus with MLM has been a common pre-training recipe Roberts
et al. (2020). This is normally followed by a fine-tuning step on the task of
interest Ruder et al. (2019), although large language models have also proved
useful without this task-specific finetuning Brown et al. (2020).

Beyond the capacity of contextual understanding, human-level language
understanding pivots on the knowledge about the world. The world knowledge is
often expressed as factual triples (c.f. Ji et al., 2020), in the form of
(subject entity, relation, object entity). A knowledge graph (KG) defined by a
set of factual triples consists of the subjects and objects as vertices/nodes,
and the relations forming the edges connecting them. Most of the large scale
KGs (e.g. Wikidata, Vrandečić and Krötzsch, 2014) are stored in triple format.

LLMs demonstrate some capacity of learning world knowledge from the natural
text corpus Roberts et al. (2020), but it is unclear to what degree they are
also able to learn and memorize new knowledge directly from structured KG
triples, or from text describing them explicitly.

In order to infuse knowledge into a LLM, one option is to generate a textual
version of the knowledge base, and apply the standard training objectives,
e.g. MLM. This is unfortunately highly nontrivial. One can either align
sentences with KG triples, as done in ERNIE Sun et al. (2021), or generate
sentences from triples, as done in KELM Agarwal et al. (2021). These
approaches are unfortunately hard to port to knowledge graphs with different
schemas. These processes are also lossy in that not every triple can be
aligned or produce a valid sentence, and there is not a good understanding
whether this can introduce unnecessary selection biases on top of biases
existing in the original KG.

In this work, we propose a method of Knowledge Infusion for Large Language
Models (SKILL), where LLMs directly learns from knowledge triples. Experiment
results shows the checkpoints trained with proposed method on Wikidata KG
outperform the T5 baselines on four standard closed-book question-answering
(QA) tasks. With a smaller KG, WikiMovies, the proposed method gain $3\times$
exact match score performance improvement on MetaQA task. The models learning
directly from knowledge triples performs competitively with the ones with the
aligned natural sentences that contain the same amount of knowledge. Being
able to learn directly from knowledge triples enables easy addition of
structured knowledge into language modeling pre-training.

##  2 Related work

Previous works that use knowledge graphs to enhance the quality of knowledge-
intensive downstream tasks can be divided into two groups: using knowledge
graphs at the inference time, and infusing knowledge into the model weights at
the pre-training time. The proposed method falls in the latter group.

#### Explicit usage of knowledge graphs.

A retrieval-augmented model is commonly used, in order to retrieve and apply
the knowledge from external memories or sources. FILM (Verga et al., 2021) and
EaE (Févry et al., 2020) extend Transformer (Vaswani et al., 2017) models with
external entity (both FILM and EaE) and fact (FILM) memories. REALM (Guu et
al., 2020) is pre-trained to perform reasoning over a large textual knowledge
corpus on-the-fly during inference. UniK-QA (Oguz et al., 2020) combines the
structured and unstructured information to improve the open-domain QA tasks
with a retriever-reader framework. The main difference between the proposed
method, SKILL, and retrieval-augmented models is that SKILL doesn’t introduce
retrieval system or external memories to the model, but it directly embeds
knowledge into the model parameters, which introduces no extra cost at
inference time.

#### Knowledge infusion.

A common way of parameterized knowledge infusion is to map or convert
structured knowledges into natural language text. ERNIE 3.0 (Sun et al., 2021)
trains a knowledge-enhanced model on a corpus combining triples and their
aligned sentences, by randomly masking relation in a triple or words in a
sentence. On the contrary, SKILL trains only on triples.

KnowBert (Peters et al., 2019) incorporates knowledge from Wikipedia and
WordNet (Miller, 1995) into a BERT model through entity embeddings with
knowledge-attention and re-contextualization mechanism. BERT-MK (He et al.,
2020) is a BERT-based model that integrates graph contextual knowledge of a
medical KG, which demonstrates the utility of graph-level knowledge. These
approaches requires entity linking and sentences contextualizing the knowledge
graph information.

KG-FiD (Yu et al., 2021) extends the Fusion-in-Decoder model (Izacard and
Grave, 2021), with a module that filters and re-ranks passages based on
structural connections in knowledge graph between entities described in those
passages. In contrast to the SKILL method that we propose, it requires the
existence of natural text passages describing each knowledge graph entity, so
Wikipedia corpus was used since it naturally provides articles that describe
entities.

Heinzerling and Inui (2021) explored the ability of language models to
memorize and understand information from knowledge graphs, but used natural
language representation of triples based on predefined templates instead of
structured representation. Usage of predefined templates significantly limits
scalability and therefore only relatively small knowledge graphs were used,
such as Google-RE111https://ai.googleblog.com/2013/04/50000-lessons-on-how-to-
read-relation.html.

In contrast to the new method presented in this paper, all of these approaches
require an explicit mapping between the knowledge graph entities or facts and
corresponding natural language sentences, which can limit applications to
industry-scale knowledge graphs that don’t have such a mapping.

#### Different goals of using knowledge graphs.

Besides that, some papers embed knowledge into model weights but pursue
different goals rather than improving performance on downstream tasks. COMET
(Bosselut et al., 2019) is most similar to our work and trains a commonsense-
aware Transformer Language Model by learning to generate loosely structured
commonsense descriptions in the natural language given the structured
knowledge. Similar to us, it also uses KG triples in surface form as a source
for training data, but in contrast to our research, the final goal of COMET is
to generate new knowledge instead of utilizing existing ones. Another
important difference is the scale: COMET uses Atomic (Sap et al., 2019) and
ConceptNet (Speer et al., 2017) Knowledge Graphs that are much smaller than
Wikidata (Vrandečić and Krötzsch, 2014).

KELM (Agarwal et al., 2021) fine-tunes a T5 model to convert KGs to synthetic
natural language sentences to augment existing pre-training corpora. We build
our research on top of it and use the KELM dataset to compare structured and
natural language representations of knowledge.

Wikidata triple KELM sentence Wikidata input KELM input Target ("Pulp
Fiction", "award received", "Palme d’Or") Quentin Tarantino won the Palme d’Or
in 1994 for Pulp Fiction. Pulp Fiction, award received, [MASK] Quentin
Tarantino won the [MASK] in 1994 for Pulp Fiction. Palme d’Or

Table 1:  Example inputs for SKILL pre-training with Wikidata and KELM
corpora.

##  3 Method

There are two components of knowledge infusion for LLMs (SKILL): the corpus
and the training method. We introduce the method based on Wikidata KG, but it
can be applied to any other KGs.

#### Training corpus.

We use two corpora with different knowledge representations: Wikidata KG
Vrandečić and Krötzsch (2014) in triple format, and KELM corpus222Data is
available at https://github.com/google-research-datasets/KELM-corpus (Agarwal
et al., 2021) as synthetic natural language sentences converted from Wikidata
KG. The KELM corpus contains $15,628,486$ synthetic sentences. To ensure two
corpora share the same knowledge, we take the snapshot of the Wikidata KG used
to created the KELM corpus, which contains $35,697,715$ triples.

To prevent the degradation of model performance on natural language
understanding, we mix the Wikidata corpus or KELM corpus with natural text
from C4 Raffel et al. (2019), $50:50$, for the knowledge infusion training
data.

#### Training method.

T5 Raffel et al. (2019) was trained through masked-language modelling with
random span corruption on the C4 corpus. Roberts et al. (2020) found that
masking salient terms Guu et al. (2020) in pre-training T5 models, instead of
masking random token spans, could significantly improve the performance on
downstream tasks, e.g. closed-book QA.

We apply salient span masking for unsupervised learning in our knowledge-
infusing training. To mask the same amount of information is for both corpora,
the following method is applied. For a knowledge triple, we mask either the
subject or object entity. For a KELM sentence, we identify the aligned triple,
with details in Appendix A, and mask the full spans corresponding to the
subject or object in the triple. The relation tokens are never masked, as
there is no robust way to map the abstract relation in knowledge triples to
natural language tokens in KELM sentences. Examples of the inputs for both
corpora are in Table 1.

##  4 Experiments

We assess SKILL by training and evaluating the knowledge infused models on
closed-book QA tasks, where questions are provided without supporting context
and external knowledge.

###  4.1 Experiment Setup

#### SKILL pre-training.

We apply SKILL on three T5.1.1 pre-trained
checkpoints333https://goo.gle/t5-checkpoints, base, large, and XXL, with sizes
of $\sim 250$M, $\sim 800$M and $\sim 11$B parameters, respectively. For
T5.1.1-base and -large, SKILL training is performed for $500$K steps with
batch size $1024$, which translates to $\sim 7.17$ epochs on Wikidata KG and
$\sim 16.38$ epochs in KELM sentences. For T5.1.1-XXL, the model is trained
for $100$K steps to finish training in a feasible time.

As baseline we use pre-trained T5 checkpoints of the same size. To make sure
that improvements come from knowledge infusion instead of from longer C4 pre-
training, we use a second baseline by further training the T5 checkpoints on
C4 for half of the aforementioned steps, to match the amount of C4 pre-
training used in SKILL.

All the model variations are optimized by AdaFactor Shazeer and Stern (2018)
with $10^{-3}$ learning rate and $0.1$ dropout rate, the same settings that
were used for T5.

#### Fine-tuning on closed-book QA tasks.

We evaluate the checkpoints by fine-tuning on the following QA benchmarks:
FreebaseQA (Jiang et al., 2019), WikiHop (Welbl et al., 2018), TriviaQA (Joshi
et al., 2017) and NaturalQuestions (Kwiatkowski et al., 2019), with the
aforementioned hyper-parameters for optimization and $128$ batch size. For the
benchmarks without a test split, we use the dev split for test, and the last
$10\%$ of train as dev split.

The Exact Match (EM) scores on the test sets are calculated after being fine-
tuned for $50$K steps for T5.1.1-base and -large models, and $10$K steps for
-XXL models. All models converged with no noticeable over-fitting according to
the EM scores on validation sets.

#### Wikidata-answerable QA.

Model FreebaseQA WikiHop TQA-matched TQA NQ-matched NQ dev test dev test dev
test dev test dev test dev test base $25.24$ $27.55$ $19.09$ $18.38$ $31.24$
$33.55$ $22.64$ $22.93$ $36.64$ $32.68$ $25.04$ $25.48$ base + C4 $26.19$
$28.33$ $19.57$ $19.36$ $32.9$ $34.4$ $24.54$ $25.39$ $36.98$ $32.03$
$\mathbf{25.88}$ $25.84$ base + WikiKG $\mathbf{26.92}$ $\mathbf{28.38}$
$20.28$ $\mathbf{20.22}$ $\mathbf{34.21}$ $35.08$ $24.73$ $\mathbf{25.77}$
$\mathbf{37.41}$ $\mathbf{33.33}$ $25.51$ $25.76$ base + KELM $26.64$ $28.15$
$\mathbf{20.62}$ $19.81$ $33.64$ $\mathbf{35.54}$ $\mathbf{25.22}$ $25.75$
$36.98$ $32.9$ $25.31$ $\mathbf{26.2}$ large $30.22$ $32.88$ $20.92$ $21.12$
$36.7$ $38.09$ $29.24$ $30.03$ $39.22$ $35.06$ $27.12$ $27.15$ large + C4
$32.55$ $34.01$ $22.5$ $21.51$ $38.78$ $40.6$ $30.32$ $30.83$ $39.74$ $35.5$
$27.46$ $28.17$ large + WikiKG $\mathbf{33.22}$ $\mathbf{35.29}$
$\mathbf{23.5}$ $\mathbf{23.4}$ $39.19$ $\mathbf{41.02}$ $29.74$ $30.47$
$\mathbf{41.12}$ $\mathbf{35.93}$ $27.38$ $27.89$ large + KELM $32.65$ $34.16$
$23.34$ $22.91$ $\mathbf{39.45}$ $40.76$ $\mathbf{30.51}$ $\mathbf{30.65}$
$40.95$ $35.5$ $\mathbf{27.67}$ $\mathbf{28.56}$ XXL $43.67$ $45.02$ $24.76$
$24.8$ $51.73$ $53.1$ $42.44$ $42.21$ $46.47$ $43.72$ $31$ $32.27$ XXL + C4
$42.01$ $44.14$ $23.34$ $22.23$ $50.59$ $52.19$ $40.66$ $40.99$ $45.43$
$40.26$ $30.35$ $31.08$ XXL + WikiKG $45.22$ $\mathbf{47.25}$ $\mathbf{27.57}$
$\mathbf{27.65}$ $\mathbf{54.17}$ $54.18$ $42.55$ $\mathbf{43.54}$
$\mathbf{49.14}$ $\mathbf{44.37}$ $31.11$ $\mathbf{32.74}$ XXL + KELM
$\mathbf{45.42}$ $45.9$ $26.11$ $26.26$ $53.65$ $\mathbf{54.21}$
$\mathbf{42.68}$ $42.95$ $48.53$ $44.16$ $\mathbf{31.79}$ $32.6$

Table 2:  Exact match scores achieved by fine-tuning the checkpoints on
closed-book QA tasks. base, large, XXL represent the corresponding T5.1.1-*
checkpoints. *-C4 are the checkpoints additionally trained on C4 corpus as
discussed in Section 3. *-WikiKG and *-KELM are the checkpoints trained on
Wikidata KG triple corpus and KELM sentence corpus. The best performed
checkpoints are in bold. Details about datasets are in Appendix B.

We found that the majority of the questions in FreebaseQA and WikiHop can be
answered directly from triples in Wikidata. This is because FreebaseQA was
created by matching question-answer pairs with triples in Freebase (Bollacker
et al., 2008), most of which was imported into Wikidata (Vrandečić and
Krötzsch, 2014). For WikiHop, the questions were generated from Wikidata
triples.

However, TriviaQA and NaturalQuestions were created independently of Wikidata,
and not every question can be answered using this knowledge base. We found
frequent freshness issues, e.g. the golden answer for question "Who is the
largest supermarket chain in the UK?" is "Aldi", while today it would be
"Tesco". Some other questions can not be answered by WikiData, e.g. "Who,
during a radio microphone test in 1984 said, ’I just signed legislation which
outlaws Russia forever. The bombing begins in five minutes?’", with the golden
answer "Ronald Reagan".

To mitigate this, we created subsets of TriviaQA (TQA) and NaturalQuestions
(NQ) that were somewhat more likely to have answers in Wikidata. We selected
all the items for which there exist a triple in Wikidata that has the answer
either as subject or object, and the other entity in the triple is mentioned
in the question. We match the entities by entity name, case-insensitive. We
name the Wikidata-aligned version of TQA and NQ as TQA-matched and NQ-matched,
respectively. The dataset sizes of all QA tasks are summarized in Appendix B.

###  4.2 Results

The results for closed-book QA tasks are summarized in Table 2. SKILL pre-
trained models show improvements on FreebaseQA, WikiHop, and Wikidata-
answerable versions of TriviaQA and NaturalQuestions, but no significant
improvement on original TriviaQA and NaturalQuestions. As discussed in
previous section, we believe this is due to the misalignment between the
datasets and Wikidata.

Models pre-trained on Wikidata KG gives competitive results with ones on KELM
sentences. It shows that the triple representation is as good as natural
language representation, while being much easier to scale up for larger KG.

For T5.1.1-base and -large, additional pre-training on C4 boosts performance
in comparison to the original baseline. For T5.1.1-XXL, this additional pre-
training leads to a performance regress. In (Raffel et al., 2019), it is
mentioned that training on C4 for multiple times may reduce the performance of
a T5 model.

#### Impact of model size.

As shown in Figure 1, SKILL pre-training introduces bigger improvements when
applied on larger models. With more than $35$M triples in Wikidata KG, it is
harder for smaller size models, e.g. T5.1.1.-base with $300$M parameters, to
memorize them efficiently. We view this as an encouraging result, suggesting
that as model size grows, gains from SKILL pre-training may increase further.

Figure 1:  Performance improvements on closed-book QA tasks for different
model sizes. The improvements are measured by the difference of exact match
score ($\Delta$EM) between knowledge-infused model trained with Wikidata
triples and the baseline trained with C4 corpus.

#### Performance on a smaller KG.

The WikiMovies KG Miller et al. (2016) contains $134,741$ triples.
T5.1.1-large should have enough parameters to memorize the KG. We train a
T5.1.1-large model on the KG for $100$K steps, $\sim 380$ epochs, with the
same hyperparameters as for Wikidata KG. We evaluate the checkpoints with
MetaQA (Zhang et al., 2018) benchmark that was constructed over WikiMovies KG.
The benchmark contains 3 different sub-tasks: 1-hop QA (e.g. "What films does
Paresh Rawal appear in?"), 2-hop QA (e.g. "Who are the directors of the films
written by Laura Kerr?"), 3-hop QA (e.g. "Who directed the movies written by
the writer of Millennium Actress?").

The results in Table 3 demonstrate the effectiveness of SKILL pre-training,
when it’s possible to memorize the whole knowledge graph.

Dataset Split Baseline \+ C4 \+ KG 1-hop dev $24.3$ $23.12$ $\mathbf{71.52}$
test $24.5$ $23.53$ $\mathbf{71.47}$ 2-hop dev $32.05$ $32.23$
$\mathbf{33.49}$ test $32.65$ $32.78$ $\mathbf{33.57}$ 3-hop dev $42.08$
$39.22$ $\mathbf{43.79}$ test $42.31$ $39.66$ $\mathbf{43.41}$

Table 3:  Exact match scores achieved by fine-tuning different T5.1.1-large
checkpoints on MetaQA task.

As 1-hop questions are supported by single triples in the WikiMovies KG, a
$3\times$ improvement on EM score is observed for the sub-task. In order to
answer 2/3-hop questions it is not enough to memorize the triples, the model
needs to be able to reason with them. This requires a better understanding of
the graph structure. Training with single triples may not be enough, and the
observed improvement is notably smaller. The performance could be further
improved by representing more explicitly the graph structure in the training
data, which we leave for future work.

##  5 Conclusion

We proposed a method to directly infuse knowledge from knowledge graphs into
T5 models through pre-training. Empirical results show that T5 can learn
directly from structured data and apply the learned knowledge to improve
closed-book QA results. We also demonstrated that the models pre-trained on
factual triples perform competitively with the ones on natural language
sentences that contain the same knowledge. By enabling knowledge infusion
directly from triples, this method can be very easily applied to industry-
scale KGs.

##  6 Ethical and Broader Impact

In this work, we are introducing a new method to pre-train a well known
natural language understanding model, T5, on the full corpora of public
knowledge graphs. To the best of our knowledge, the method will not introduce
extra bias to either the model or the dataset beyond the one potentially
inherited from Wikidata Vrandečić and Krötzsch (2014) and WikiMovies Miller et
al. (2016) knowledge graphs. On the other hand, through knowledge fusion pre-
training introduced in this work, a language model will be able to learn
factual information to improve the quality of parameterized knowledge embedded
in the model, which is demonstrated by improvements on various closed-book
question-answering tasks. The proposed method and recipe will provide positive
impact to the natural language processing community and help to improve the
factualness in pre-trained large language model checkpoints.

#### Limitations.

A factual triple is the basic ingredient of a knowledge graph. However, as a
semantic network, the graph structure of a knowledge graph describes how the
factual triples are connected. This information is not easy to directly
represent by random set of triples. We leave the exploration of how to infuse
the information implied by the graph structure for future work. We expect that
this will further improve the results, especially for multi-hop question-
answering tasks.

## References

  * Agarwal et al. (2021) Oshin Agarwal, Heming Ge, Siamak Shakeri, and Rami Al-Rfou. 2021.  Knowledge graph based synthetic corpus generation for knowledge-enhanced language model pre-training.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3554–3565, Online. Association for Computational Linguistics.
  * Bollacker et al. (2008) Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008\.  Freebase: A collaboratively created graph database for structuring human knowledge.  In _Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data_ , SIGMOD ’08, page 1247–1250, New York, NY, USA. Association for Computing Machinery.
  * Bosselut et al. (2019) Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. 2019.  COMET: Commonsense transformers for automatic knowledge graph construction.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4762–4779, Florence, Italy. Association for Computational Linguistics.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Févry et al. (2020) Thibault Févry, Livio Baldini Soares, Nicholas FitzGerald, Eunsol Choi, and Tom Kwiatkowski. 2020.  Entities as experts: Sparse memory access with entity supervision.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4937–4951, Online. Association for Computational Linguistics.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020.  Retrieval augmented language model pre-training.  In _International Conference on Machine Learning_ , pages 3929–3938. PMLR.
  * He et al. (2020) Bin He, Di Zhou, Jinghui Xiao, Xin Jiang, Qun Liu, Nicholas Jing Yuan, and Tong Xu. 2020.  BERT-MK: Integrating graph contextualized knowledge into pre-trained language models.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2281–2290, Online. Association for Computational Linguistics.
  * Heinzerling and Inui (2021) Benjamin Heinzerling and Kentaro Inui. 2021.  Language models as knowledge bases: On entity representations, storage capacity, and paraphrased queries.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1772–1791, Online. Association for Computational Linguistics.
  * Izacard and Grave (2021) Gautier Izacard and Edouard Grave. 2021.  Leveraging passage retrieval with generative models for open domain question answering.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 874–880, Online. Association for Computational Linguistics.
  * Ji et al. (2020) Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and Philip S. Yu. 2020\.  A survey on knowledge graphs: Representation, acquisition and applications.  _CoRR_ , abs/2002.00388.
  * Jiang et al. (2019) Kelvin Jiang, Dekun Wu, and Hui Jiang. 2019.  FreebaseQA: A new factoid QA data set matching trivia-style question-answer pairs with Freebase.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 318–323, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Miller et al. (2016) Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016.  Key-value memory networks for directly reading documents.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1400–1409, Austin, Texas. Association for Computational Linguistics.
  * Miller (1995) George A. Miller. 1995.  Wordnet: A lexical database for english.  _Commun. ACM_ , 38(11):39–41.
  * Oguz et al. (2020) Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Scott Yih. 2020.  UniK-QA: Unified representations of structured and unstructured knowledge for open-domain question answering.  _arXiv preprint arXiv:2012.14610_.
  * Peters et al. (2019) Matthew E. Peters, Mark Neumann, Robert Logan, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A. Smith. 2019.  Knowledge enhanced contextual word representations.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 43–54, Hong Kong, China. Association for Computational Linguistics.
  * Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _CoRR_ , abs/1910.10683.
  * Roberts et al. (2020) Adam Roberts, Colin Raffel, and Noam Shazeer. 2020.  How much knowledge can you pack into the parameters of a language model? In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 5418–5426, Online. Association for Computational Linguistics.
  * Ruder et al. (2019) Sebastian Ruder, Matthew E. Peters, Swabha Swayamdipta, and Thomas Wolf. 2019.  Transfer learning in natural language processing.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials_ , pages 15–18, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Sap et al. (2019) Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. 2019.  Atomic: An atlas of machine commonsense for if-then reasoning.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33.
  * Shazeer and Stern (2018) Noam Shazeer and Mitchell Stern. 2018.  Adafactor: Adaptive learning rates with sublinear memory cost.  _CoRR_ , abs/1804.04235.
  * Speer et al. (2017) Robyn Speer, Joshua Chin, and Catherine Havasi. 2017.  Conceptnet 5.5: An open multilingual graph of general knowledge.  In _Thirty-first AAAI conference on artificial intelligence_.
  * Sun et al. (2021) Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, et al. 2021.  Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation.  _arXiv preprint arXiv:2107.02137_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Verga et al. (2021) Pat Verga, Haitian Sun, Livio Baldini Soares, and William Cohen. 2021.  Adaptable and interpretable neural MemoryOver symbolic knowledge.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3678–3691, Online. Association for Computational Linguistics.
  * Vrandečić and Krötzsch (2014) Denny Vrandečić and Markus Krötzsch. 2014.  Wikidata: A free collaborative knowledgebase.  _Communications of the ACM_ , 57(10):78–85.
  * Welbl et al. (2018) Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018.  Constructing datasets for multi-hop reading comprehension across documents.  _Transactions of the Association for Computational Linguistics_ , 6:287–302.
  * Yu et al. (2021) Donghan Yu, Chenguang Zhu, Yuwei Fang, Wenhao Yu, Shuohang Wang, Yichong Xu, Xiang Ren, Yiming Yang, and Michael Zeng. 2021.  Kg-fid: Infusing knowledge graph in fusion-in-decoder for open-domain question answering.  _arXiv preprint arXiv:2110.04330_.
  * Zhang et al. (2018) Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexander J Smola, and Le Song. 2018\.  Variational reasoning for question answering with knowledge graph.  In _AAAI_.

##  Appendix A Matching of entities in KELM sentences

To find Wikidata KG entities in corresponding KELM sentences, we use Algorithm
1. Additional cycle on line 22 is needed because some entities have an
information in brackets that should not be in a sentence, for example John Doe
(born 1990). This algorithm matched at least one entity to $15,383,248$ out of
$15,628,486$ KELM sentences.

1:$KELM_{matched}\leftarrow\emptyset$

2:for each $k\in KELM$ sentences do

3: for each $t\in triples(k)$ do

4: for each $e\in entities(t)$ do

5: $e_{p}\leftarrow$ Preprocess($e$)

6: $k_{p}\leftarrow$ Preprocess($k$)

7: $spans\leftarrow$ MatchEntity($e_{p}$, $k_{p}$)

8: $KELM_{matched}.insert([k,spans])$

9: end for

10: end for

11:end for

12:

13:function MatchEntity($e$: entity, $k$: KELM sentence)

14: $spans\leftarrow\emptyset$

15: for each $s\subset k:date(e)=date(s)$ do

16: $spans.insert(s)$

17: end for

18: for each $\exists s\subset k:e=s$ do

19: $spans.insert(s)$

20: end for

21: if $spans=\emptyset$ then

22: for each $\exists s\subset k:e=s+$" (*)" do

23: $spans.insert(s)$

24: end for

25: end if

26: return $spans$

27:end function

28:

29:function Preprocess($str$: string)

30: $str\leftarrow Lowercase(str)$

31: $str\leftarrow RemovePunctuation(str)$

32: return $str$

33:end function

Algorithm 1 KELM-Wikidata matching algorithm that finds spans in KELM
sentences corresponding to Wikidata KG entities. $a\subset b$ means that $a$
is a substring of $b$. $*$ represents any string.

We don’t try to match relation part of triples, because it could be
represented in many different forms. For example, the triple (Pulp Fiction,
cast member, John Travolta) could be represented as "John Travolta was an
actor in Pulp Fiction", "John Travolta starred in Pulp Fiction", "John
Travolta played Vincent Vega in Pulp Fiction", etc., and there is no way to
robustly align a relation to all possible surface forms.

##  Appendix B Dataset

Wikidata Vrandečić and Krötzsch (2014) was released under the Creative Commons
CC0 License. KELM Agarwal et al. (2021) was released under the Creative
Commons CC BY-SA 2.0 License. NaturalQuestions Kwiatkowski et al. (2019) and
WikiHop (Welbl et al., 2018) were released under Creative Commons CC BY-SA 3.0
License. MetaQA (Zhang et al., 2018) was released under Creative Commons CC
BY-ND 3.0 License. C4 Raffel et al. (2019) and TriviaQA Joshi et al. (2017)
were released under Apache-2.0 License. WikiMovies Miller et al. (2016) was
released under MIT License. FreebaseQA (Jiang et al.,
2019)444https://github.com/kelvin-jiang/FreebaseQA was released without a
license.

train dev test FreebaseQA $20,358$ $3,994$ $3,996$ WikiHop $39,364$ $4,374$
$5,129$ TQA $78,785$ $8,837$ $11,313$ TQA-matched $20,948$ $2,289$ $3,064$ NQ
$79,168$ $8,757$ $3,610$ NQ-matched $10,487$ $1,160$ $462$ MetaQA-1hop
$96,106$ $9,992$ $9,947$ MetaQA-2hop $118,980$ $14,872$ $14,872$ MetaQA-3hop
$114,196$ $14,274$ $14,274$

Table 4:  Dataset sizes for the closed-book QA tasks. TQA and NQ stands for
TriviaQA and NaturalQuestions, respectively. *-matched are the selected
dataset with the Wikidata KG answerable questions, and the KG alignment
details can be found in Section 4.1.


Goldman-type Lie algebras from hyperbolic knots

Takefumi Nosaka111 E-mail address<EMAIL_ADDRESS>
###### Abstract

We define Lie algebras from a hyperbolic knot in the 3-sphere. Since the
definitions in terms of group homology are analogous to Goldman Lie algebra
[Gold], we discuss relations among these Lie algebras.

Keywords
hyperbolic geometry, Lie algebra, knots, string topology

##  1 Introduction

Given a group $G$, by $\hat{G}$ we mean the set of representatives of the
conjugacy classes of $G$. Let $\Sigma_{g,1}$ be a connected oriented compact
genus-$g$ surface with one circle boundary. When $G=\pi_{1}(\Sigma_{g,1})$,
Goldman [Gold] gives a Lie algebra structure on the vector space
${\mathbb{Q}}[\hat{G}]$ spanned by $\hat{G}$. As shown in [Gold, KK2, MT,
Tur1] and references therein, the Goldman Lie algebra is related to topics
including (quantizations of) flat moduli spaces, the Teichmüllar space, and
the Johnson homomorphism; see also [ACC, CS, KK1] for studies from string
topology. In studies on the Goldman Lie algebra, the hyperbolicity of
$\Sigma_{g,1}$ sometimes plays a key role. Some properties of hyperbolic
surfaces have been (virtually) extended to those of hyperbolic 3-manifolds.
Hence, it is reasonable to hope for similar Lie algebras from hyperbolic
3-manifolds.

In this paper, analogously to the Goldman Lie algebra, we introduce Lie
algebras from hyperbolic knots. The main result is as follows:

######  Theorem 1.1.

Let $K$ be a knot $K$ in an integral homology 3-sphere $\Sigma$ and $G$ be the
commutator subgroup of the knot group, that is, $G=[\pi_{1}(\Sigma\setminus
K),\pi_{1}(\Sigma\setminus K)]$. Assume that $K$ is either hyperbolic or
fibered.

Then, the ${\mathbb{Q}}$-vector space, ${\mathbb{Q}}[\hat{G}]$, spanned by
$\hat{G}$ has a Lie algebra structure and is equipped with a
${\mathbb{Q}}[t^{\pm 1}]$-module, where
$t:{\mathbb{Q}}[\hat{G}]\rightarrow{\mathbb{Q}}[\hat{G}]$ is a Lie algebra
isomorphism. Furthermore, if $K$ is fibered of genus $g$, the algebra
structure coincides with the Goldman Lie algebra of $\Sigma_{g,1}$.

As a corollary, we will introduce Lie algebra structures on
${\mathbb{Q}}[\Pi_{K}]$ and ${\mathbb{Q}}[G/[G,G]]$, where $\Pi_{K}$ is the
set $\\{[k]\in\hat{G}\ |\ \mathrm{abelianization\ of\ }k=0\\}$, and the latter
$G/[G,G]$ is the so-called Alexander module of $K$; see §3 for details.

As shown in [Gold], the Goldman Lie algebra is, in some sense, universal among
moduli spaces of flat $G$-bundles on $F$, where $G$ is a semi-simple Lie
group. Thus, the Lie algebras and the isomorphism $t$ in Theorem 1.1 would
include interesting information on representations varieties
$\mathop{\mathrm{Hom}}\nolimits(\pi_{1}(\Sigma\setminus K),G)/G$, and would
provide topological invariants of hyperbolic knots in terms of Lie algebras.

This paper is organized as follows. After a review of group homology and
string products (see Section 2), Section 3 introduces the Lie bracket in
Theorem 1.1. Sections 4 and 5 give the proofs of our results.

Conventional notation. We denote by $\hat{G}$ the set of representatives of
the conjugacy classes of a group $G$. For $x\in G$, we denote $Z_{G}(x)\subset
G$ by the centralizer subgroup. By $K$ we mean a knot in an integral homology
3-sphere $\Sigma.$

##  2 Preliminaries

We will describe this product structure directly in terms of the relative
homology of groups. For this, we begin by reviewing the (relative) group
(co)-homology. Denote the group ring of a group $G$ over ${\mathbb{Q}}$ by
${\mathbb{Q}}[G]$. Let $F_{n}(G)$ be the free ${\mathbb{Q}}[G]$-module with
basis $\\{[g_{1}|\cdots|g_{n}];g_{i}\in G\\}$. Let $\bar{F}_{n}(G)$ be the
quotient of $F_{n}(G)$ by the ${\mathbb{Q}}[G]$-submodule generated by
$\\{[g_{1}|\cdots|g_{n}];g_{i}=1\mathrm{\ for\ some\ }i\\}$. Given a left
${\mathbb{Q}}[G]$-module $A$, we define $C_{n}(G;A)$ to be
$A\otimes_{{\mathbb{Q}}[G]}\bar{F}_{n}(G)$ and the differential operator
$\partial$ by

$\\!\\!\\!\\!\\!\\!(g_{1}^{-1}a)\otimes[g_{2}|\cdots|g_{n}]+\\!\\!\\!\\!\\!\sum_{i:\
1\leq i\leq
n-1}\\!\\!\\!\\!\\!\\!(-1)^{i}a\otimes[g_{1}|\dots|g_{i-1}|g_{i}g_{i+1}|g_{i+2}|\cdots|g_{n}]+(-1)^{n}a\otimes[g_{1}|\dots|g_{n-1}].$

Then, the group homology is defined to be the homology. Furthermore, for a
subgroup $S\subset G$, the relative homology is defined to be the homology of
the quotient complex $C_{n}(G;A)/C_{n}(S;A)$. In this paper, we often consider
the case $A={\mathbb{Q}}[G/L]$ for some subgroup $L\subset G$, where $G/L$ is
acted on by left multiplication.

Dually, we can define the group cohomology, $H^{n}(G,S;A)$, the cup product
$\smile$, and the cap product  $\smile$ ; see, e.g., [BE] for details.

This paper focuses on rational duality groups : here, a pair of groups
$S\subset G$ (possibly $S=\emptyset$) is said to have a
${\mathbb{Q}}$-relative (Poincaré) duality of dimension $n$ if there is a
relative homology $n$-class $\sigma_{G,S}\in H_{n}(G,S;{\mathbb{Q}})$ such
that the cap product with this class provides an isomorphism

$D:=\bullet{\rotatebox[origin={c}]{180.0}{
$\smile$}}\sigma_{G,S}:H^{p}(G,S;{\mathbb{Q}}[G/Z_{G}(x)]])\stackrel{{\scriptstyle\sim}}{{\longrightarrow}}H_{n-p}(G;{\mathbb{Q}}[G/Z_{G}(x)]])$

for any $x\in G$. We give a topological example:

######  Example 2.1.

Let $M$ be an oriented compact $n$-manifold such that $M$ and the boundary
$\partial M$ are aspherical. Let $G:=\pi_{1}(M)$ and $S:=\pi_{1}(\partial M)$.
Then, there are isomorphisms $H^{p}(M,\partial
M;{\mathbb{Q}}[G/Z_{G}(x)]])\cong H^{p}(G,S;{\mathbb{Q}}[G/Z_{G}(x)])$ and the
relative fundamental class $\sigma_{G,S}:=[M,\partial M]\in H_{n}(M,\partial
M;{\mathbb{Z}})$, which ensures relative duality.

Using the map $D$, let us prepare the intersection product in the homology of
subgroups of a ${\mathbb{Q}}$-relative Poincaré duality, as a slight
modification of [ACC, §1]. For any subgroups $L,H\subset G$, it is known [ACC,
Lemma 1] that the correspondence $g_{1}L\otimes g_{2}H\mapsto
L(g_{1}^{-1}g_{2})H$ gives rise to a ${\mathbb{Q}}[G]$-module isomorphism,

$\psi:{\mathbb{Q}}[G/L]\otimes{\mathbb{Q}}[G/H]\cong\bigoplus_{[g]\in
I}{\mathbb{Q}}[G/(L\cap gHg^{-1})],$

where $I$ is a set of representatives for the double coset $L\backslash G/H$.
If $L=Z_{G}(x)$ and $H=Z_{G}(y)$ for some $x,y\in G$, then the intersection
pairing,

${\rotatebox[origin={c}]{180.0}{ $\smile$}}:H_{p}(G;{\mathbb{Q}}[G/L])\otimes
H_{q}(G;{\mathbb{Q}}[G/H])\longrightarrow\bigoplus_{[g]\in
I}H_{p+q-n}(G;{\mathbb{Q}}[G/L\cap gHg^{-1}]),$

defined in [ACC, Definition 2] is given by the composition,

$H_{p}(G;{\mathbb{Q}}[G/L])\otimes
H_{q}(G;{\mathbb{Q}}[G/H])\xrightarrow{D^{-1}\otimes
D^{-1}}H^{n-p}(G,S;{\mathbb{Q}}[G/L])\otimes H^{n-q}(G,S;{\mathbb{Q}}[G/H])$
$\stackrel{{\scriptstyle\smile}}{{\longrightarrow}}H^{2n-p-q}(G,S;{\mathbb{Q}}[G/L]\otimes{\mathbb{Q}}[G/H])\stackrel{{\scriptstyle
D}}{{\longrightarrow}}H_{p+q-n}(G;{\mathbb{Q}}[G/L]\otimes{\mathbb{Q}}[G/H])$
$\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \
\stackrel{{\scriptstyle\psi_{*}}}{{\longrightarrow}}\bigoplus_{[g]\in
I}H_{p+q-n}(G;{\mathbb{Q}}[G/(L\cap gHg^{-1})]).$

If $S$ is the empty set, this definition is exactly Definition 2 in [ACC].

Next, we review string products. Let $\mathrm{Ad}$ be ${\mathbb{Q}}[G]$ as a
coefficient module, where the action of $G$ is given by conjugation. Then,

$H_{*}(G,S;\mathrm{Ad})=\bigoplus_{[x]\in\hat{G}}H_{*}(G;{\mathbb{Q}}[G/Z_{G}(x)])\cong\bigoplus_{[x]\in\hat{G}}H_{*}(Z_{G}(x);{\mathbb{Q}}),$

where the second isomorphism is immediately obtained from the Shapiro Lemma.
As mentioned in the introduction of [ACC], $H_{*}(G;\mathrm{Ad})$ is
isomorphic to $H_{*}(L(BG);{\mathbb{Q}})$, where $L(BG)$ is the free loop
space of the Eilenberg-MacLane space of $G$. Then, as in Definition 4 in
[ACC], the string product,

$\mu:H_{p}(G;\mathrm{Ad})\otimes H_{q}(G;\mathrm{Ad})\longrightarrow
H_{p+q-n}(G;\mathrm{Ad}),$ (1)

is defined to be the composition,

$\mu:=j_{*}\circ{\rotatebox[origin={c}]{180.0}{
$\smile$}}:H_{p}(G;{\mathbb{Q}}[G/Z_{G}(x)])\otimes
H_{q}(G;{\mathbb{Q}}[G/Z_{G}(y)])\longrightarrow$
$\rightarrow\bigoplus_{[g]\in I}H_{p+q-n}(G;{\mathbb{Q}}[G/(Z(x)\cap
gZ_{G}(y)g^{-1})])\longrightarrow\bigoplus_{[g]\in
I}H_{p+q-n}(G;{\mathbb{Q}}[G/Z_{G}(xgyg^{-1})]).$

Here, $j$ is induced from the following inclusion of subgroups:

$j:Z_{G}(x)\cap g^{-1}Z_{G}(y)g=Z_{G}(x)\cap Z_{G}(g^{-1}yg)\subset
Z_{G}(xgyg^{-1}).$

Then, as in [ACC, Theorem 5], we can easily check that $\mu$ turns
$H_{*+n}(G;\mathrm{Ad})$ into a graded, associated, commutative algebra. For
instance, it is shown in [ACC] that, if $S=\emptyset$ and $N$ is an aspherical
closed $n$-manifold with $G=\pi_{1}(N)$, then the string product coincides
with the loop product in the string topology of $N$ [CS].

Finally, we review the homomorphism $\lambda$ and define a bracket (Definition
2.2). Let us consider the correspondence $\lambda:G\rightarrow
C_{1}(G;\mathrm{Ad});x\mapsto x\otimes[x]$. As can be seen in [KK1, Lemma
3.4.2], we can easily check that $\lambda(x)$ is a 1-cycle and
$\lambda(y^{-1}xy)=\lambda(x)$ holds in $H_{1}(G;\mathrm{Ad})$. Thus, we have
a homomorphism $\lambda:{\mathbb{Q}}[\hat{G}]\rightarrow
H_{1}(G;\mathrm{Ad})$.

######  Definition 2.2.

Suppose that $S\subset G$ are ${\mathbb{Q}}$-relative duality groups of
dimension $2$ and identify $H_{0}(G;\mathrm{Ad})$ with ${\mathbb{Q}}[\hat{G}]$
as a ${\mathbb{Q}}$-vector space. Then, we define a binary bracket,

$[,]:{\mathbb{Q}}[\hat{G}]^{\otimes 2}\longrightarrow
H_{0}(G;\mathrm{Ad})={\mathbb{Q}}[\hat{G}]$

by setting $[x,y]=\mu(\lambda(x),\lambda(y))$. Here, $\mu$ is the string
product with $p=q=1$.

######  Example 2.3.

Let $G$ be the surface group $\pi_{1}(\Sigma_{g,1})$ and $S$ be
$\pi_{1}(\partial\Sigma_{g,1})={\mathbb{Z}}$. It is shown in [KK1, Proposition
3.4.3] that the bracket is exactly equal to the Goldman Lie bracket [Gold] on
${\mathbb{Q}}[\hat{G}]$.

In so doing, we hope that this $[,]$ has the anti-commutativity property and
satisfies the Jacobi identity for many pairs $S\subset G$ of dimension two;
however, this paper focuses on the fundamental groups from hyperbolic knots.

##  3 Results

Here, we state Theorem 3.1; the proof is in §5. We denote the complement
$\Sigma\setminus K$ by $E_{K}$. With a choice of base point in $\partial
E_{K}$, we fix an inclusion ${\mathbb{Z}}^{2}=\pi_{1}(\partial
E_{K})\hookrightarrow\pi_{1}(E_{K})$. Since
$H_{1}(E_{K};{\mathbb{Z}})\cong{\mathbb{Z}}$, we focus on the infinite cyclic
cover $\tilde{E}_{K}$ of $E_{K}$. Let $\pi_{K}$ and $\partial\pi_{K}$ be
$\pi_{1}(\tilde{E}_{K})$ and $\pi_{1}(\partial\tilde{E}_{K})$, respectively.
In other words,
$\mathop{\pi_{K}}\nolimits=\pi_{1}(\tilde{E}_{K})=[\pi_{1}(E_{K}),\pi_{1}(E_{K})]$,
and $\partial\mathop{\pi_{K}}\nolimits\cong{\mathbb{Z}}$.

######  Theorem 3.1.

Suppose we have a hyperbolic knot $K$ in $\Sigma.$ In the above notation, the
pair $\partial\mathop{\pi_{K}}\nolimits\subset\mathop{\pi_{K}}\nolimits$ has a
${\mathbb{Q}}$-relative duality of dimension two. Furthermore, the bracket
$[,]$ in Definition 2.2 admits a Lie algebra structure on
$H_{0}(\pi;\mathrm{Ad})={\mathbb{Q}}[\widehat{\mathop{\pi_{K}}\nolimits}]$.

######  Remark 3.2.

As seen in Remarks 7.4 and 8.5 in [MT], some bracket on a completed module of
${\mathbb{Q}}[\widehat{\mathop{\pi_{K}}\nolimits}]$ is discussed in terms of
“Fox pairings” (see also [Tur2, Theorem E]); however, the point of Theorem 3.1
is that the bracket [,] on ${\mathbb{Q}}[\widehat{\mathop{\pi_{K}}\nolimits}]$
is homologically defined before any completion and is shown to have the anti-
commutativity property and satisfy the Jacobi identity.

Now let us discuss the relation to the Goldman Lie algebra. Let $F$ be a
Seifert surface in $\Sigma\setminus K$, where a Seifert surface is a smooth
embedding $i$ of $\Sigma_{g,1}$ for some $g$ to $\Sigma$ such that
$i(\partial\Sigma_{g,1})=K$ and the orientations are compatible with the
embedding. We have a lift of $F$ as a surface in $\tilde{E}_{K}$, where the
lift is unique up to covering transformations. Recall from Example 2.3 that we
have the Goldman Lie algebra on ${\mathbb{Q}}[\pi_{1}(F)]$.

######  Proposition 3.3.

The linear homomorphism induced by the inclusion
$F\hookrightarrow\tilde{E}_{K}$ preserves the brackets. In particular, if $K$
is fibered and $F$ is a fiber surface, the bracket on
${\mathbb{Q}}[\widehat{\mathop{\pi_{K}}\nolimits}]$ is equal to the Goldman
Lie bracket on $F$.

We defer to the proof in §4. Next, as an application, we introduce other Lie
algebras from Theorem 3.1. Consider the covering transformation
$t:\tilde{E}_{K}\rightarrow\tilde{E}_{K}$. This induces a group isomorphism
$t_{*}:\mathop{\pi_{K}}\nolimits\rightarrow\mathop{\pi_{K}}\nolimits$ and a
bijection
$t_{*}:\widehat{\mathop{\pi_{K}}\nolimits}\rightarrow\widehat{\mathop{\pi_{K}}\nolimits}$.
Thus, the Lie algebra on ${\mathbb{Q}}[\widehat{\mathop{\pi_{K}}\nolimits}]$
can be regarded as a ${\mathbb{Q}}[t^{\pm}]$-module. Furthermore, as is
mentioned in Remark 4.4 later,
$\sigma_{\mathop{\pi_{K}}\nolimits,\partial\mathop{\pi_{K}}\nolimits}=t_{*}\sigma_{\mathop{\pi_{K}}\nolimits,\partial\mathop{\pi_{K}}\nolimits}\in
H_{2}(\mathop{\pi_{K}}\nolimits,\partial\mathop{\pi_{K}}\nolimits;{\mathbb{Q}})$.
Thus, we have $[tx,ty]=t[x,y]$ by the definitions of the string product and
$\lambda$. Then, by the definition of the semi-direct product
$\pi_{1}(E_{K})\cong\mathop{\pi_{K}}\nolimits\rtimes{\mathbb{Z}}$, the
quotient set, $\widehat{\mathop{\pi_{K}}\nolimits}/\sim$, of
$\widehat{\mathop{\pi_{K}}\nolimits}$ determined by the relation $x\sim
t_{*}x$ can be regarded as the subset,

$\Pi_{K}:=\\{[x]\in\widehat{\pi_{1}(E_{K})}\ |\ \mathrm{The\ abelianization\
of\ }x=0\\}.$

Thus, the quotient module ${\mathbb{Q}}[\widehat{\mathop{\pi_{K}}\nolimits}]$
subject to the ideal ($t-1$) is isomorphic to ${\mathbb{Q}}[\Pi_{K}]$ and is
equipped with the induced Lie algebra.

Moreover, let us consider the abelianization, $H_{1}(\pi_{K};{\mathbb{Z}})$,
of $\pi_{K}$, which is called the Alexander module (of $K$) in knot theory;
see [Lic]. This can be identified with the set
$\widehat{\mathop{\pi_{K}}\nolimits}\setminus\widehat{[\mathop{\pi_{K}}\nolimits,\mathop{\pi_{K}}\nolimits]}\cup\\{1\\}$.
Therefore, the group ring ${\mathbb{Q}}[H_{1}(\pi_{K};{\mathbb{Z}})]$ has the
quotient Lie algebra from Theorem 3.1, where $[tx,ty]=t[x,y]$ holds. By
construction, if $K$ is fibered, the Lie algebra is isomorphic to the
homological Goldman Lie algebra in [Gold, p.295–p.297]. To summarize,

######  Corollary 3.4.

Let $\Pi_{K}$ be the above set and $H$ be the abelianization
$H_{1}(\pi_{K};{\mathbb{Z}})$. Then, the Lie algebra structure in Theorem 3.1
induces those on ${\mathbb{Q}}[\Pi_{K}]$ and ${\mathbb{Q}}[H]$. The covering
transformation induces Lie algebra isomorphisms
$t_{*}:{\mathbb{Q}}[\Pi_{K}]\stackrel{{\scriptstyle\sim}}{{\rightarrow}}{\mathbb{Q}}[\Pi_{K}]$
and
$t_{*}:{\mathbb{Q}}[H]\stackrel{{\scriptstyle\sim}}{{\rightarrow}}{\mathbb{Q}}[H].$

##  4 Preliminaries for the proofs

For the proof of the above theorems, we need to investigate properties of
hyperbolic knot groups. We will use the following terminology: $E_{K}$ is the
complement $\Sigma\setminus K$, and $\tilde{E}_{K}$ is the infinite cyclic
cover. As above, let $\mathop{\pi_{K}}\nolimits$ and
$\partial\mathop{\pi_{K}}\nolimits$ be $\pi_{1}(\tilde{E}_{K})$ and
$\pi_{1}(\partial\tilde{E}_{K})$, respectively. Throughout this section, $K$
is assumed to be hyperbolic. By hyperbolicity, we first notice an elementary
property:

######  Lemma 4.1.

For any $g\in\mathop{\pi_{K}}\nolimits\setminus\\{1\\}$, the centralizer,
$Z_{\mathop{\pi_{K}}\nolimits}(g)\subset\mathop{\pi_{K}}\nolimits$, is
infinite cyclic.

###### Proof.

The proof is almost the same as the proof of [AFW, Theorem 2.5.1]. By Jordan
decomposition, the centralizer of any non-trivial matrix in
$PSL_{2}(\mathbb{C})$ is abelian. Since $\pi_{1}(E_{K})$ is a discrete,
torsion-free subgroup of $PSL_{2}(\mathbb{C})$, so is the subgroup
$\mathop{\pi_{K}}\nolimits$; it is straightforward to see that
$Z_{\mathop{\pi_{K}}\nolimits}(g)$ is isomorphic to either ${\mathbb{Z}}$ or
${\mathbb{Z}}^{2}$. If
$Z_{\mathop{\pi_{K}}\nolimits}(g)\cong{\mathbb{Z}}^{2}$, Theorem 1.7.2 in
[AFW] as a basic fact directly admits $h\in\pi_{1}(E_{K})$ such that
$Z_{\mathop{\pi_{K}}\nolimits}(g)\subset h^{-1}\pi_{1}(\partial(E_{K}))h$.
However,
$h^{-1}\pi_{1}(\partial(E_{K}))h\cap\mathop{\pi_{K}}\nolimits=h^{-1}\partial\mathop{\pi_{K}}\nolimits
h$ is cyclic and generated by the longitude; this means
${\mathbb{Z}}^{2}\subset{\mathbb{Z}}$, which is a contradiction. Hence,
$Z_{\mathop{\pi_{K}}\nolimits}(g)$ must be isomorphic to ${\mathbb{Z}}$. ∎

Using this, we will show four propositions:

######  Proposition 4.2 (cf. [KK1, Proposition 3.4.3]).

If $i>1$, the homology $H_{i}(\mathop{\pi_{K}}\nolimits;\mathrm{Ad})$
vanishes.

There is an isomorphism
$H_{0}(\mathop{\pi_{K}}\nolimits;\mathrm{Ad})\cong{\mathbb{Q}}[\widehat{\mathop{\pi_{K}}\nolimits}]$
and an exact sequence,

$0\longrightarrow{\mathbb{Q}}[\widehat{\mathop{\pi_{K}}\nolimits}\setminus\\{1\\}]\stackrel{{\scriptstyle\lambda}}{{\longrightarrow}}H_{1}(\mathop{\pi_{K}}\nolimits;\mathrm{Ad})\stackrel{{\scriptstyle\sigma}}{{\longrightarrow}}H_{1}(\mathop{\pi_{K}}\nolimits;{\mathbb{Q}})\longrightarrow
0.$

Here, $\lambda$ is defined in §2, and $\sigma$ is the homomorphism induced
from the augmentation $\mathrm{Ad}\rightarrow{\mathbb{Q}}$. Furthermore, if
$x\in G$ with $x\neq 1$, then
$H_{*}(\mathop{\pi_{K}}\nolimits;{\mathbb{Q}}[\mathop{\pi_{K}}\nolimits/Z_{\mathop{\pi_{K}}\nolimits}(x)])\cong
H_{*}(S^{1};{\mathbb{Q}})$.

###### Proof.

Using the Shapiro lemma, we can compute the homology
$H_{*}(\mathop{\pi_{K}}\nolimits;\mathrm{Ad})$ as

$\bigoplus_{[x]\in\widehat{\mathop{\pi_{K}}\nolimits}}H_{*}(\mathop{\pi_{K}}\nolimits;{\mathbb{Q}}[\mathop{\pi_{K}}\nolimits/Z_{\mathop{\pi_{K}}\nolimits}(x)])\cong\bigoplus_{[x]\in\widehat{\mathop{\pi_{K}}\nolimits}}H_{*}(Z_{\mathop{\pi_{K}}\nolimits}(x);{\mathbb{Q}})\cong
H_{*}(\mathop{\pi_{K}}\nolimits;{\mathbb{Q}})\oplus(\bigoplus_{[x]\in\widehat{\mathop{\pi_{K}}\nolimits}\setminus\\{1\\}}H_{*}(S^{1};{\mathbb{Q}})).$

Here, the second isomorphism is obtained from Lemma 4.1. If $*>1$, the
homology $H_{*}(\pi;\mathrm{Ad})$ vanishes, since
$H_{*}(\mathop{\pi_{K}}\nolimits;{\mathbb{Q}})=0$, as is well known in knot
theory (see [Lic, Chapters 7–9]). Since
$H_{0}(\mathop{\pi_{K}}\nolimits;{\mathbb{Q}})={\mathbb{Q}}$, the homology
$H_{*}(\mathop{\pi_{K}}\nolimits;\mathrm{Ad})$ is canonically identified with
${\mathbb{Q}}[\widehat{\mathop{\pi_{K}}\nolimits}]$ by the above calculation.
Finally, if $*=1$ and $x\neq 1$, it is not hard to check that the cycle
$\lambda(x)$ corresponds to a generator of
$H_{1}(Z(x);{\mathbb{Q}})\cong{\mathbb{Q}}$; this implies the required exact
sequence. Moreover, the last claim is readily obtained. ∎

######  Proposition 4.3.

The pair $\partial\mathop{\pi_{K}}\nolimits\subset\mathop{\pi_{K}}\nolimits$
has a ${\mathbb{Q}}$-relative duality of dimension two.

###### Proof.

The proof is a priori due to the Milnor pairing [Mil]. Take a Seifert surface
$F\subset E_{K}$, and fix a lift of $F$ as a surface in $\widetilde{E}_{K}$ up
to covering transformations. Then, as a modification of the Milnor paring with
local coefficient $A$ over ${\mathbb{Q}}$, if $H_{*}(\widetilde{E}_{K};A)$ is
of finite dimension, there is an isomorphism,

$\delta:H^{i}(\widetilde{E}_{K},\partial\widetilde{E}_{K};A)\cong H^{i+1}_{\rm
compact\ support}(\widetilde{E}_{K},\partial\widetilde{E}_{K};A),$

such that the composite,

$\mathrm{Poincare\
dual}\circ\delta:H^{2-i}(\widetilde{E}_{K},\partial\widetilde{E}_{K};A)\stackrel{{\scriptstyle\sim}}{{\longrightarrow}}H^{3-i}_{\rm
compact\
support}(\widetilde{E}_{K},\partial\widetilde{E}_{K};A)\stackrel{{\scriptstyle\sim}}{{\longrightarrow}}H_{i}(\widetilde{E}_{K};A),$

is equal to the cap product with $F$; see [Mil, Assertion 9 and Remark 3] or
[Yana] for the proof.

Since $\widetilde{E}_{K}$ and $\partial\widetilde{E}_{K}$ are Eilenberg-
MacLane spaces by hyperbolicity, we have an isomorphism
$H_{*}(\widetilde{E}_{K};A)\cong H_{*}(\mathop{\pi_{K}}\nolimits;A)$, which
preserves the cup and cap products. In addition,
$H_{*}(\widetilde{E}_{K};{\mathbb{Q}}[\mathop{\pi_{K}}\nolimits/Z_{\mathop{\pi_{K}}\nolimits}(x)])$
is of finite dimension by Proposition 4.2 (here, we should notice that
$H_{1}(\mathop{\pi_{K}}\nolimits;{\mathbb{Q}})$ is also finite dimensional by
[Mil, Assertion 5]). Hence, the pair
$\partial\mathop{\pi_{K}}\nolimits\subset\mathop{\pi_{K}}\nolimits$ is a
${\mathbb{Q}}$-relative duality group pair of dimension 2. ∎

######  Remark 4.4.

Since $t(F)$ is bordant to $F$ in $\widetilde{E}_{K}$ from the construction of
$\widetilde{E}_{K}$ using $F$ (see [Lic, Chapters 6–7]) and the relative
homology 2-class
$\sigma_{\mathop{\pi_{K}}\nolimits,\partial\mathop{\pi_{K}}\nolimits}$ is
represented by $F$, we have
$t_{*}(\sigma_{\mathop{\pi_{K}}\nolimits,\partial\mathop{\pi_{K}}\nolimits})=\sigma_{\mathop{\pi_{K}}\nolimits,\partial\mathop{\pi_{K}}\nolimits}$.

######  Proposition 4.5.

Take duality group pairs $S\subset G$ and $S^{\prime}\subset G^{\prime}$ of
dimension two and a group homomorphism $f:G\rightarrow G^{\prime}$ satisfying
$f(S)\subset S^{\prime}$ and
$f_{*}(\sigma_{G,S})=N_{f}\sigma_{G^{\prime},S^{\prime}}$ for some
$N_{f}\in{\mathbb{Q}}.$ Assume that, for any $g\in G\setminus\\{1\\}$, the
centralizers $Z_{G}(g)$ is infinite cyclic.

Then, the string product has the naturality
$f_{*}(\mu(x,y))=N_{f}\mu^{\prime}(f_{*}(x),f_{*}(y))$.

###### Proof.

We claim that, for any $g\in G\setminus\\{1\\}$, the image $f(Z_{G}(g))$ is
contained in the centralizer $Z_{G^{\prime}}(f(g))$. By assumption, any $h\in
Z_{G}(g)$ admits $m\in{\mathbb{Z}}$ with $h^{m}=g$. In general, if
$g^{\prime},h^{\prime}\in G^{\prime}$ satisfy $(h^{\prime})^{m}=g^{\prime}$,
then the roof $h^{\prime}$ of $g^{\prime}$ necessarily lies in
$Z_{G^{\prime}}(g^{\prime})$; see [AFW, Page 37]. Therefore, $f(h)^{m}=f(g)$
implies $f(h)\in Z_{G^{\prime}}(f(g))$. Then, the required naturality follows
from the claim and the definition of the string product. ∎

###### Proof of Proposition 3.3.

Let $G$ be $\pi_{1}(\Sigma_{g,1})$ and $G^{\prime}$ be
$\mathop{\pi_{K}}\nolimits$, where the Seifert surface of genus $g$ is
regarded as $\Sigma_{g,1}$. Similarly, $Z(h)$ for any $h\in G\setminus\\{1\\}$
is widely known to be infinite cyclic (see, e.g., [KK1, Proposition 3.4.3]).
By Proposition 4.5 with $N_{f}=1$, the linear homomorphism preserves the Lie
bracket. ∎

##  5 Proofs of the theorems

For the proofs, we review the Heegaard decomposition of $E_{K}$ and group
presentation of $\pi$; see (3) below. It is known (see, e.g., [Sav, Lemma
17.2]) that we can choose a Seifert surface $F$ of genus $g$ such that the
complement $\Sigma\setminus F$ is homeomorphic to a handlebody of genus $g$.
Fix a meridian $\mathfrak{m}\in\pi_{1}(E_{K}).$ Take a bicollar
$F\times[-1,1]$ of $F$ such that $S\times\\{0\\}=F$. Let
$\iota_{\pm}:F\rightarrow\Sigma\setminus F$ be embeddings whose images are
$F\times\\{\pm 1\\}$. Take generating sets $W:=\\{u_{1},\dots,u_{2g}\\}$ of
$\pi_{1}S$ and $X:=\\{x_{1},\dots,x_{2g}\\}$ of $\pi_{1}(\Sigma\setminus F)$.
Set $y_{i}:=(\iota_{+})_{*}(u_{i})$ and $z_{i}=(\iota_{-})_{*}(u_{i})$; a von
Kampen argument yields a presentation of $\pi_{1}(\Sigma\setminus K)$:

$\langle\ \mathfrak{m},x_{1},\dots,x_{2g},\ |\
\mathfrak{m}^{-1}y_{i}\mathfrak{m}\ z_{i}^{-1}\ \ \ (1\leq i\leq 2g)\ \
\rangle.$ (2)

Then, by using the Reidemeister-Schreier method, $\pi_{1}(\widetilde{E}_{K})$
is presented by

$\langle\ x_{1}^{(n)},\dots,x_{2g}^{(n)}\ \ \ (n\in{\mathbb{Z}})\ |\
y_{i}^{(k)}(z_{i}^{(k+1)})^{-1}\ \ \ (1\leq i\leq 2g,\ n\in{\mathbb{Z}})\ \
\rangle.$ (3)

###### (Proof of Theorem 3.1).

As a preparation, for $k,\ell\in{\mathbb{Z}}$ with $k<\ell$, consider the
copies $t^{k}(F),t^{k-1}(F),\dots,t^{\ell}(F)$, and the connected sum
$F_{k,\ell}:=t^{k}(F)\sharp t^{k-1}(F)\sharp\dots\sharp t^{\ell}(F)$ in
$\widetilde{E}_{K}$, where the connected sum is obtained by adding half-tubes
around $\partial\widetilde{E}_{K}=\partial F\times(-\infty,\infty)$. Then, by
Remark 4.4, $[F_{k,\ell},\partial F_{k,\ell}]=(\ell-k+1)[F,\partial F]$ in
$H_{2}(\widetilde{E}_{K};{\mathbb{Q}})$. By the construction of (3), the image
of $\pi_{1}(F_{k,\ell})$ by inclusion $F_{k,\ell}\subset\widetilde{E}_{K}$ is
the subgroup of $\pi_{1}(\widetilde{E}_{K})$ generated by the set
$\\{x_{1}^{(n)},\dots,x_{2g}^{(n)}|\ k\leq n\leq\ell\\}$. Let us denote the
subgroup by $\mathcal{G}_{k,\ell}$.

Next, we show the Jacobi identity of $[,]$. For
$[a_{1}],[a_{2}],[a_{3}]\in\widehat{\mathop{\pi_{K}}\nolimits}$, take
respective representatives $a_{1},a_{2},a_{3}\in\mathop{\pi_{K}}\nolimits$. By
(3), there are $k,\ell\in{\mathbb{Z}}$ such that
$a_{1},a_{2},a_{3}\in\mathcal{G}_{k,\ell}.$ By Proposition 4.5 with
$N_{f}=\ell-k+1$, the inclusion $i:F_{k,\ell}\subset\widetilde{E}_{K}$ induces
$i_{*}:{\mathbb{Q}}[\widehat{\mathcal{G}_{k,\ell}}]\rightarrow{\mathbb{Q}}[\widehat{\mathop{\pi_{K}}\nolimits}]$,
which preserves the bracket up to constant multiples and admits
$b_{j}\in\pi_{1}(F_{k,\ell})$ with
$i_{*}([b_{j}])=[\tilde{a}_{j}]\in\widehat{\mathop{\pi_{K}}\nolimits}$. The
Goldman Lie algebra on ${\mathbb{Q}}[\widehat{\mathcal{G}_{k,\ell}}]$ in
Example 4.4 inherits the required Jacobi identity. Precisely,

$[[a_{1},a_{2}],a_{3}]+[[a_{2},a_{3}],a_{1}]+[[a_{3},a_{1}],a_{2}]=(\ell-k+1)i_{*}([[b_{1},b_{2}],b_{3}]+[[b_{2},b_{3}],b_{1}]+[[b_{3},b_{1}],b_{2}])=0.$

Finally, the anti-commutativity of $[,]$ directly comes from that of the
intersection pairing. ∎

###### (Proof of the main theorem 1.1).

Recall from §3 that the covering transformation $t$ induces an isomorphism
$t:{\mathbb{Q}}[\mathop{\pi_{K}}\nolimits]\rightarrow{\mathbb{Q}}[\mathop{\pi_{K}}\nolimits]$
preserving the bracket $[,]$. Hence, the former claim readily follows from
Theorem 3.1; the latter part follows from Proposition 3.3, since
$\mathop{\pi_{K}}\nolimits$ of a fibered knot $K$ of genus $g$ is isomorphic
to $\pi_{1}(\Sigma_{g,1})$. ∎

###  5.1 Acknowledgments

The author is indebted to Tetsuya Ito, Nariya Kawazumi, Yusuke Kuno, Gwénaël
Massuyeau, Takahito Naito, and Hidetoshi Masai for useful comments and
discussions.

## References

  * [1]
  * [AFW] Matthias Aschenbrenner, Stefan Friedl, and Henry Wilton. 3-manifold groups, EMS Series of Lectures in Mathematics. European Mathematical Society, Zürich, 2015.
  * [ACC] Abbaspour, Hossein; Cohen, Ralph; Gruher, Kate. String topology of Poincaré duality groups. Groups, 1–10, Geom. Topol. Monogr., 13, Geom. Topol. Publ., Coventry, 2008. MR2508199
  * [BE] Bieri, Robert; Eckmann, Beno. Relative homology and Poincaré duality for group pairs. J. Pure Appl. Algebra 13 (1978), no. 3, 277–319. MR0509165
  * [CS] Chas. M, Sullivan Dennis, String topology, Preprint, math.GT/9911159.
  * [Gold] Goldman, William M. Invariant functions on Lie groups and Hamiltonian flows of surface group representations. Invent. Math. 85 (1986), no. 2, 263–302. MR0846929
  * [Lic] Lickorish, W. B. Raymond. An introduction to knot theory. Graduate Texts in Mathematics, 175. Springer-Verlag, New York, 1997. x+201 pp. ISBN: 0-387-98254-X MR1472978
  * [KK1] Kawazumi, Nariya; Kuno, Yusuke. The logarithms of Dehn twists, Quantum Topol. 5 (2014), no. 3, 347–423. MR3283405
  * [KK2] Kawazumi, Nariya; Kuno, Yusuke. The Goldman-Turaev Lie bialgebra and the Johnson homomorphisms. Handbook of Teichmüller theory. Vol. V, 97–165, IRMA Lect. Math. Theor. Phys., 26, Eur. Math. Soc., Zürich, 2016. MR3497295
  * [Mil] Milnor, John W. Infinite cyclic coverings. 1968 Conference on the Topology of Manifolds (Michigan State Univ., E. Lansing, Mich., 1967) pp. 115–133 Prindle, Weber & Schmidt, Boston, Mass. MR0242163
  * [MT] Massuyeau, Gwénaël; Turaev, Vladimir. Fox pairings and generalized Dehn twists. Ann. Inst. Fourier (Grenoble) 63 (2013), no. 6, 2403–2456. MR3237452
  * [Sav] Saveliev, Nikolai. Lectures on the topology of 3-manifolds. An introduction to the Casson invariant. Second revised edition. De Gruyter Textbook. Walter de Gruyter & Co., Berlin, 2012. xii+207 pp. ISBN: 978-3-11-025035-0 MR2893651
  * [Tur1] Turaev, Vladimir. Skein quantization of Poisson algebras of loops on surfaces. Ann. Sci. École Norm. Sup. (4) 24 (1991), no. 6, 635–704. MR1142906
  * [Tur2] Turaev, Vladimir. Multiplace generalizations of the Seifert form of a classical knot. (Russian) Mat. Sb. (N.S.) 116(158) (1981), no. 3, 370–397, 463–464. MR0665689
  * [Yana] Yanagida Koki, in preparation

Department of Mathematics, Tokyo Institute of Technology 2-12-1 Ookayama,
Meguro-ku Tokyo 152-8551 Japan


# Understanding Reward Ambiguity Through Optimal Transport Theory in Inverse
Reinforcement Learning

Ali Baheri
Department of Mechanical Engineering
Rochester Institute of Technology
Rochester, NY 14623
<EMAIL_ADDRESS>

###### Abstract

In inverse reinforcement learning (IRL), the central objective is to infer
underlying reward functions from observed expert behaviors in a way that not
only explains the given data but also generalizes to unseen scenarios. This
ensures robustness against reward ambiguity—where multiple reward functions
can equally explain the same expert behaviors. While significant efforts have
been made in addressing this issue, current methods often face challenges with
high-dimensional problems and lack a geometric foundation. This paper
harnesses the optimal transport (OT) theory to provide a fresh perspective on
these challenges. By utilizing the Wasserstein distance from OT, we establish
a geometric framework that allows for quantifying reward ambiguity and
identifying a central representation or centroid of reward functions. These
insights pave the way for robust IRL methodologies anchored in geometric
interpretations, offering a structured approach to tackle reward ambiguity in
high-dimensional settings.

##  1 Introduction

In the reinforcement learning (RL), the derivation of reward functions has
always been a subject of intricate debates and investigations [1]. Inverse RL
(IRL) emerges as a pivotal methodology in this dialogue, where the aim is to
decipher the underlying reward structure from observed expert behavior [2].
However, one of the intrinsic challenges that plagues IRL is the issue of
reward ambiguity. A fundamental realization in IRL is that multiple distinct
reward functions can lead to the same or similar expert behaviors. This many-
to-one relationship between reward functions and policies complicates the task
of pinpointing a unique reward landscape from demonstrations [2, 3].

Attempts to navigate this conundrum have been diverse. The maximum entropy IRL
framework, for instance, integrates a probabilistic lens into IRL [4]. By
postulating that expert demonstrations align not solely with optimality but
also with an exponential reward distribution, this approach employs entropy
maximization to favor reward functions yielding more uniformly distributed
trajectories, serving as an inherent countermeasure to reward ambiguity.
Further advancements came from Bayesian IRL, where the uncertainty in reward
functions is addressed directly by employing a Bayesian framework [5, 6, 7].
Instead of seeking a single reward function, Bayesian IRL estimates a
posterior distribution over reward functions given the demonstrations.
InfoGAIL, an extension of the GAIL framework, is another noteworthy mention
[8, 9]. It incorporates mutual information into the objective, aiming to
capture multiple modes of the expert’s behavior. By doing so, InfoGAIL
effectively disentangles different underlying reasons (latent variables) for
the expert’s actions, which can be seen as a means to shed light on the
ambiguous regions of the reward landscape.

Optimal transport (OT) theory presents a fresh, promising avenue for reward
ambiguity problem. The fundamental tenet of OT lies in its ability to measure
the _distance_ between different probability distributions, or in our context,
between various reward functions. This is achieved using a concept called the
Wasserstein distance. By understanding how far apart or close together
different reward functions are, one can gain clearer insights into the
nebulous territory of reward ambiguity that IRL grapples with. The true
potential of OT for IRL is its _geometrically-grounded_ framework. Where
traditional approaches might only offer probabilistic insights, the marriage
of IRL and OT provides a more structured, spatial understanding of reward
landscapes. Imagine mapping reward functions within a geometric space, where
the proximity between points indicates similarity. OT provides the tools to
construct, measure, and interpret this space, and in doing so, can potentially
revolutionize our comprehension of reward ambiguity. In this paper, our
primary contribution is the fusion of the analytical power of OT with the
challenges of reward ambiguity inherent in IRL. We advocate for a synthesis,
where the discerning capabilities of IRL meet the geometric precision of OT,
aiming to deliver a more holistic solution to one of RL’s most enduring
challenges.

##  2 Preliminaries

OT finds its roots in the study of moving and redistributing masses in the
most efficient manner [10, 11]. Historically conceived by Gaspard Monge in the
late 18th century, its motivations were primarily combinatorial. However, with
the reformation of the theory by Kantorovich in the 20th century, it adopted a
more functional analysis perspective, making it more broadly applicable [12,
13]. In the context of OT, we deal with two probability measures, $\mu$ and
$\nu$, defined on a common metric space. The foundational goal of OT is to
transport $\mu$ to $\nu$ while minimizing a certain cost associated with
moving each infinitesimal unit of mass. This cost is typically governed by a
cost function $c$, which dictates the expense of moving a unit mass across a
given distance. Mathematically, the $p$-Wasserstein distance between two
probability measures $\mu$ and $\nu$, defined with respect to a ground metric
$d$ on the space, is given by:

$W_{p}(\mu,\nu)=\left(\inf_{\gamma\in\Gamma(\mu,\nu)}\int
d(x,y)^{p}d\gamma(x,y)\right)^{\frac{1}{p}}$ (1)

In this representation, $x$ and $y$ are points within the metric space over
which the probability measures $\mu$ and $\nu$ are defined. The function
$d(x,y)$ computes the distance between these two points, rooted in the
foundational ground metric $d$. Meanwhile, $\Gamma(\mu,\nu)$ signifies the set
of all conceivable joint distributions (or transport plans) having $\mu$ and
$\nu$ as marginals. The Wasserstein distance’s merit is its efficacy to
encapsulate the geometric discrepancies between probability measures. This
capability has rendered it invaluable across various domains, from image
processing to machine learning, economics, and computational biology [14, 15,
16, 17, 18, 19]. In the context of our study, the appeal of OT is derived from
its capacity to numerically elucidate disparities between reward function
distributions. This offers a unique perspective to examine the intricacies of
reward ambiguity inherent in IRL.

##  3 Problem Formulation

IRL revolves around the fundamental task of inferring the most probable reward
function that an expert is optimizing, given their behavior. A key challenge
in IRL is the ambiguity of reward functions: multiple reward functions might
give rise to the same, or nearly identical, expert policies. This set of
reward functions is denoted as $R\left(\pi^{*}\right)$, where $\pi^{*}$ is the
expert’s policy. A pertinent factor influencing the dynamics of this ambiguity
is the dimensionality of the problem. In our context, the state space is
represented as $\mathcal{S}$ and the action space as $\mathcal{A}$. The
dimensionality, $d$, encapsulates the total number of state-action pairs and
is given by:

$d=|\mathcal{S}|\times|\mathcal{A}|$ (2)

Higher the dimensionality, the more intricate and vast our reward space
becomes, potentially complicating our understanding of
$R\left(\pi^{*}\right)$. To navigate this complex space and to better
understand the distribution and relationships between these reward functions,
we turn to OT theory. This theory allows us to map each reward function in
$R\left(\pi^{*}\right)$ to a representation in a space where distances
(dissimilarities) between these functions can be meaningfully measured. We
term this mapping as $\Phi$, which translates a reward function, $R$, to its
representative point, $W$, in the Wasserstein space. With this transformation,
our aims become twofold:

1\. Quantification. Measure the Wasserstein distance between different
representations in the Wasserstein space,
$\mathcal{W}\left(W_{1},W_{2}\right)$. This measurement provides insight into
the degree of ambiguity among reward functions.

2\. Centroid Identification. Beyond just understanding the space, it is
crucial to pinpoint a central, representative point in this space, a centroid
$W^{*}$, that stands, on average, closest to all other reward function
representations. Mathematically, this can be articulated as:

$W^{*}=\operatorname{argmin}_{W\in\Phi\left(R\left(\pi^{*}\right)\right)}\sum_{W_{i}\in\Phi\left(R\left(\pi^{*}\right)\right)}\mathcal{W}\left(W,W_{i}\right)$
(3)

This centroid aids in identifying a central reward function representation,
which can serve as a reference or benchmark in the Wasserstein space.

##  4 Theoretical Results

Upon formalizing the inherent challenges of reward ambiguity, particularly in
the context of escalating dimensionality, the immediate imperative is to
elucidate these complexities with rigorous theoretical underpinnings. The
potential of OT, as postulated in the preceding sections, necessitates
concrete validation. In this section, a suite of theorems is presented. These
results not only affirm our foundational hypotheses but also offer a
systematic framework to comprehend and navigate the intricacies of reward
function ambiguity in IRL. 111Proofs for the presented theorems can be found
in the supplementary material.

Theorem 1: Convergence of Inferred Rewards in Wasserstein Space.  Let
$R_{\text{true }}$ be the true underlying reward function of an expert, and
$R_{n}$ be the inferred reward function based on $n$ expert trajectories.
Then, under certain regularity conditions on the environment and expert
behavior, as $n\rightarrow\infty$ :

$W_{p}\left(R_{n},R_{\text{true }}\right)\rightarrow 0$ (4)

where $W_{p}$ denotes the $p$-Wasserstein distance.

Implications. The theorem explaining the convergence of inferred rewards in
the Wasserstein space provides key guarantees for IRL. At its core, it
confirms that, given a broad range of expert demonstrations, our estimated
reward structures gradually approach the true underlying reward function
driving the expert’s decisions. This points to a consistent pattern in expert
behaviors, addressing the well-known issue in IRL where multiple reward
functions could seemingly explain the same expert actions. Furthermore, the
theorem’s foundation in OT enhances the reliability of IRL, suggesting that
even with potential inaccuracies and variability in real-world demonstrations,
the geometric and topological insight from the Wasserstein distance offers a
way to identify and get closer to the true reward function. Therefore, this
convergence theorem not only strengthens the theoretical basis of IRL but also
highlights the crucial role of OT in improving the precision of reward
predictions based on expert data.

Theorem 2: Existence and Uniqueness of Optimal Reward. Let $\mathcal{R}$ be a
compact metric space representing the space of all reward functions, $\pi^{*}$
be an expert’s policy, $\mathcal{S}=\left\\{R_{1},R_{2},\ldots,R_{N}\right\\}$
be a set of inferred reward functions from $\mathcal{R}$, where each $R_{i}$
induces a policy equivalent to $\pi^{*}$, and $W_{p}(\cdot,\cdot)$ be the
$p$-Wasserstein distance induced by a ground metric on $\mathcal{R}$. Then:
1\. Existence: There exists a reward function $R_{\text{centroid
}}\in\mathcal{R}$ such that

$R_{\text{centroid
}}=\underset{R\in\mathcal{R}}{\operatorname{argmin}}\sum_{i=1}^{N}W_{p}\left(R,R_{i}\right)$
(5)

This $R_{\text{centroid }}$ minimizes the total $p$-Wasserstein distance to
all reward functions in $\mathcal{S}$.

2\. Uniqueness: If the ground metric on $\mathcal{R}$ induces strict
convexity, then $R_{\text{centroid}}$ is unique.

Implications. The theorem elucidating the existence and uniqueness of the
optimal reward representation within the Wasserstein space offers profound
insights for the IRL. It posits that amidst the vast space of plausible reward
functions that can induce a given expert policy, there emerges a central or
centroid reward function that serves as the most representative embodiment
with respect to the Wasserstein distance. This centrality implies a kind of
mean or average behavior which, in the face of reward ambiguities, can provide
a benchmark for the modeling of underlying motivations or preferences.
Furthermore, the condition for uniqueness, contingent on strict convexity of
the ground metric, reassures that this centroid reward is not only
representative but singularly distinctive, thereby eliminating concerns of
multiple competing central representations.

Theorem 3: Robustness of OT-Integrated IRL to Noisy Demonstrations. Let
$\epsilon$ be a measure of the noise introduced in the demonstrations such
that $\epsilon\geq 0$, with $\epsilon=0$ representing noise-free
demonstrations. Then, for every $\epsilon>0$, there exists a constant
$C(\epsilon)$ such that:

$W_{p}\left(R^{*}\left(\pi^{*}\right),R^{*}\left(\pi^{*\prime}\right)\right)\leq
C(\epsilon)$ (6)

This implies that the distance between reward functions inferred from noisy
and noise-free demonstrations remains bounded in the Wasserstein space,
ensuring robustness of the OT-integrated IRL process to noisy expert
demonstrations.

Implications. The theorem underscoring the robustness of the OT integration in
IRL against noisy demonstrations bears significant ramifications for practical
implementations. It suggests that even when expert demonstrations are marred
by imperfections or inaccuracies – a common occurrence in real-world scenarios
– the inferred motivations or reward structures remain steadfastly close to
those derived from noise-free demonstrations. This bounded divergence in the
Wasserstein space acts as a guarantee of stability, ensuring that the model’s
interpretation of an expert’s intent is not drastically swayed by minor
fluctuations or disturbances in the input data.

Theorem 4: Impact of Dimensionality on Reward Ambiguity in Wasserstein Space.
Given the set $S_{d}$ of reward functions for dimension $d$, let $\Delta_{d}$
be the average pairwise $p$-Wasserstein distance between reward functions in
$S_{d}$ defined as

$\Delta_{d}=\frac{2}{\left|S_{d}\right|\left(\left|S_{d}\right|-1\right)}\sum_{R_{i},R_{j}\in
S_{d},i<j}W_{p}\left(R_{i},R_{j}\right)$ (7)

Then, as $d$ increases, $\Delta_{d}$ also increases. This indicates that as
the dimensionality of the state-action space grows, the average pairwise
Wasserstein distance between potential reward functions grows as well,
suggesting a more dispersed reward landscape and thus intensified reward
ambiguity.

Implications. The theorem indicates that higher dimensionality in the state-
action space corresponds to a greater average pairwise Wasserstein distance
between possible reward functions. This, in turn, points to a more scattered
distribution of reward functions in the geometric space. The spread of these
reward functions implies that as we consider problems with more dimensions,
the ambiguity in determining the true reward function (among many that can
explain the expert behavior) becomes more pronounced.

##  5 Discussion

OT is effective at analyzing and comparing probability distributions by
viewing them as comprehensive entities. However, this approach can pose
challenges when dealing with reward functions that exhibit multiple modes. A
unimodal representation in the Wasserstein space may not capture all the
distinct peaks of a multimodal function. As a result, the overall
representation might be oversimplified or inaccurate. To capture the diverse
peaks of multimodal reward functions, it could be beneficial to represent them
using multiple points or clusters in the Wasserstein space. Each cluster could
represent a different peak or mode, allowing for a more detailed depiction of
the reward function. Yet, this method introduces its own set of challenges,
particularly in computation. Using multiple clusters necessitates specialized
clustering algorithms suitable for the Wasserstein space, which could increase
computational demands. Nonetheless, recent improvements in OT algorithms
provide promising solutions to these challenges [20]. While employing multiple
clusters enhances the representation’s accuracy, it can make the decision-
making process more complex. How should an agent prioritize between multiple
modes of high reward? Incorporating these insights into actionable policies
might require additional layers of analysis or even a hierarchical approach to
decision-making.

##  6 Conclusion

IRL faces with issues such as reward ambiguity, where multiple reward
functions can justify identical expert behaviors. In this paper, we presented
an approach by incorporating optimal transport theory, specifically utilizing
the Wasserstein distance. This method brings forth a geometric insight into
reward ambiguity, paving the way for a more systematic understanding. While
the theoretical underpinnings presented in this work shed light on several
facets of the reward ambiguity challenge, a crucial avenue for future research
is empirical validation. Rigorous testing and application of our theoretical
insights to real-world IRL scenarios will be essential to fully gauge the
efficacy and broad applicability of our proposed framework.

## References

  * [1] Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané.  Concrete problems in AI safety.  arXiv preprint arXiv:1606.06565, 2016.
  * [2] Andrew Y Ng, Stuart Russell, et al.  Algorithms for inverse reinforcement learning.  In Icml, volume 1, page 2, 2000.
  * [3] Pieter Abbeel and Andrew Y Ng.  Apprenticeship learning via inverse reinforcement learning.  In Proceedings of the twenty-first international conference on Machine learning, page 1, 2004.
  * [4] Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al.  Maximum entropy inverse reinforcement learning.  In Aaai, volume 8, pages 1433–1438. Chicago, IL, USA, 2008.
  * [5] Deepak Ramachandran and Eyal Amir.  Bayesian inverse reinforcement learning.  In IJCAI, volume 7, pages 2586–2591, 2007.
  * [6] Jaedeug Choi and Kee-Eung Kim.  Nonparametric bayesian inverse reinforcement learning for multiple reward functions.  Advances in neural information processing systems, 25, 2012.
  * [7] Jaedeug Choi and Kee-Eung Kim.  Map inference for bayesian inverse reinforcement learning.  Advances in neural information processing systems, 24, 2011.
  * [8] Yunzhu Li, Jiaming Song, and Stefano Ermon.  Infogail: Interpretable imitation learning from visual demonstrations.  Advances in neural information processing systems, 30, 2017.
  * [9] Jonathan Ho and Stefano Ermon.  Generative adversarial imitation learning.  Advances in neural information processing systems, 29, 2016.
  * [10] Cédric Villani et al.  Optimal transport: old and new, volume 338.  Springer, 2009.
  * [11] Filippo Santambrogio.  Optimal transport for applied mathematicians.  Birkäuser, NY, 55(58-63):94, 2015.
  * [12] Leonid V Kantorovich.  On the translocation of masses.  Journal of mathematical sciences, 133(4):1381–1382, 2006.
  * [13] Leonid Vitalevich Kantorovich.  On a problem of monge.  Journal of Mathematical Sciences, 133(4):1383–1383, 2006.
  * [14] Sira Ferradans, Nicolas Papadakis, Gabriel Peyré, and Jean-François Aujol.  Regularized discrete optimal transport.  SIAM Journal on Imaging Sciences, 7(3):1853–1882, 2014.
  * [15] Marco Cuturi.  Sinkhorn distances: Lightspeed computation of optimal transport.  Advances in neural information processing systems, 26, 2013.
  * [16] Ali Baheri.  Risk-aware reinforcement learning through optimal transport theory.  arXiv preprint arXiv:2309.06239, 2023.
  * [17] Yicheng Luo, Zhengyao Jiang, Samuel Cohen, Edward Grefenstette, and Marc Peter Deisenroth.  Optimal transport for offline imitation learning.  arXiv preprint arXiv:2303.13971, 2023.
  * [18] Alfred Galichon and Bernard Salanié.  Cupid’s invisible hand: Social surplus and identification in matching models.  The Review of Economic Studies, 89(5):2600–2629, 2022.
  * [19] Geoffrey Schiebinger, Jian Shu, Marcin Tabaka, Brian Cleary, Vidya Subramanian, Aryeh Solomon, Joshua Gould, Siyan Liu, Stacie Lin, Peter Berube, et al.  Optimal-transport analysis of single-cell gene expression identifies developmental trajectories in reprogramming.  Cell, 176(4):928–943, 2019.
  * [20] Gabriel Peyré, Marco Cuturi, et al.  Computational optimal transport.  Center for Research in Economics and Statistics Working Papers, (2017-86), 2017.

##  7 Supplementary Material

Proof of Theorem 1.

Lemma 1. For every trajectory and any inferred reward function $R$, there
exists a constant $C$ such that: $|R(s,a)|\leq C$ for all state-action pairs
$(s,a)$.

Proof. The reward values should be bounded for practical tasks and
environments; otherwise, they could lead to unbounded utility values making
the problem ill-posed.

Lemma 2. Consistency of IRL Inference. Given the true reward function
$R_{\text{true, }}$, as the number of expert trajectories
$n\rightarrow\infty$, the expected reward under the inferred policy $\pi_{n}$
based on $R_{n}$ converges to the expected reward under the expert’s policy
$\pi^{*}$ based on $R_{\text{true }}$.

Proof. Let’s consider the set of all possible trajectories $\tau$ generated
using the expert’s policy $\pi^{*}$. Since the policy is fixed and
deterministic, the distribution of trajectories is also fixed, denoted by
$p\left(\tau\mid\pi^{*}\right)$. Given a sufficiently large number $n$, the
likelihood of observing a particular trajectory $\tau$ using the inferred
policy $\pi_{n}$ should approach the likelihood of observing $\tau$ using the
expert’s policy $\pi^{*}$. Therefore, as $n\rightarrow\infty$ :

$\mathbb{E}_{\pi_{n}}\left[R_{n}(s,a)\right]\rightarrow\mathbb{E}_{\pi^{*}}\left[R_{\text{true
}}(s,a)\right]$ (8)

Proposition. Compactness in Wasserstein Space. Given a set of inferred reward
functions $\left\\{R_{n}\right\\}$ based on increasing number of expert
trajectories, the sequence is relatively compact in the Wasserstein space.

Proof of Theorem:

From Lemma 1, we know the rewards are bounded. This implies that the set of
potential reward functions resides in a compact space. From Lemma 2, as
$n\rightarrow\infty$, the difference in expected rewards under the inferred
policy and the expert’s policy diminishes. Given the compactness of the space
of reward functions (from the proposition) and the convergence of expected
rewards, it implies that the sequence $\left\\{R_{n}\right\\}$ has a
convergent subsequence in the Wasserstein space. By the properties of
Wasserstein convergence, this convergent subsequence must converge to the true
reward function $R_{\text{true }}$ in the Wasserstein metric. Thus, for any
$\epsilon>0$, there exists an $N$ such that for all $n>N$ :

$W_{p}\left(R_{n},R_{\text{true }}\right)<\epsilon$ (9)

This implies that as $n\rightarrow\infty,W_{p}\left(R_{n},R_{\text{true
}}\right)\rightarrow 0$.

Proof of Theorem 2.

Lemma 1. The sum of $p$-Wasserstein distances, as a function on $\mathcal{R}$,
is lower bounded and continuous.

Proof. Lower Bounded: By the non-negativity of the Wasserstein distance, the
sum is clearly nonnegative and thus lower bounded by 0. Continuity: The
Wasserstein distance itself is continuous by its definition, and the sum of
continuous functions remains continuous. Given that $\mathcal{R}$ is compact
and the sum of $p$-Wasserstein distances is continuous on $\mathcal{R}$, by
the Extreme Value Theorem, the function attains its minimum. Hence, a
minimizer $R_{\text{centroid }}$ exists.

Lemma 2. If the ground metric space is strictly convex, then the
$p$-Wasserstein distance is also strictly convex.

Proof. Assume two distinct reward functions $R_{1}$ and $R_{2}$ in
$\mathcal{R}$. Consider a $t$ such that $0<t<1$ and look at the function:

$f(t)=W_{p}\left(tR_{1}+(1-t)R_{2},R_{i}\right)$ (10)

Given the properties of Wasserstein distances, $f(t)$ is a convex function.
However, with strict convexity in the ground metric, $f(t)$ is strictly
convex. This means:
$f(t)<tW_{p}\left(R_{1},R_{i}\right)+(1-t)W_{p}\left(R_{2},R_{i}\right)$ for
all $i$ and for all $t$ in $0<t<1$. When summing over all $i$, if $R_{1}$ and
$R_{2}$ were both minimizers, this would lead to a contradiction, as a convex
combination of them would yield a strictly smaller sum of Wasserstein
distances. Therefore, only one of them can be the minimizer, establishing
uniqueness.

Proof of Theorem:

Directly follows from Lemma 1, the sum of $p$-Wasserstein distances as a
function over the compact space $\mathcal{R}$ is lower bounded and continuous.
Hence, it attains a minimum. Thus, an $R_{\text{centroid }}$ that minimizes
this function exists. Directly follows from Lemma 2, if the ground metric is
strictly convex, then the $p$-Wasserstein distance is strictly convex. This
strict convexity ensures that the minimizer $R_{\text{centroid }}$ is unique.

Proof of Theorem 3.

Let $R^{*}\left(\pi^{*}\right)$ be the true reward function obtained from the
noise-free expert policy $\pi^{*}$. Given the noisy demonstration policy
$\pi^{*\prime}$, the reward function inferred is
$R^{*}\left(\pi^{*\prime}\right)$. Define the noise function as $\eta(s,a)$
for state-action pair $(s,a)$. This function produces a noise value that
perturbs the action taken by the expert.

$\pi^{*\prime}(a\mid s)=\pi^{*}(a\mid s)+\epsilon\eta(s,a)$ (11)

Using the definition of the Wasserstein distance, we express the distance
between these two reward functions:

$W_{p}\left(R^{*}\left(\pi^{*}\right),R^{*}\left(\pi^{*\prime}\right)\right)=\left(\int\left|R^{*}\left(\pi^{*}\right)(s,a)-R^{*}\left(\pi^{*\prime}\right)(s,a)\right|^{p}dsda\right)^{\frac{1}{p}}$
(12)

Because the only difference between $R^{*}\left(\pi^{*}\right)$ and
$R^{*}\left(\pi^{*\prime}\right)$ is the noise, express this difference as a
function of $\epsilon$ :

$R^{*}\left(\pi^{*\prime}\right)(s,a)=R^{*}\left(\pi^{*}\right)(s,a)+g(\epsilon,s,a)$
(13)

where $g(\epsilon,s,a)$ is the change in reward due to the noise. Using the
Triangle Inequality, we bound the difference:

$\left|R^{*}\left(\pi^{*}\right)(s,a)-R^{*}\left(\pi^{*\prime}\right)(s,a)\right|\leq|g(\epsilon,s,a)|$
(14)

Plugging this into our expression for $W_{p}$, we get:

$W_{p}\left(R^{*}\left(\pi^{*}\right),R^{*}\left(\pi^{*\prime}\right)\right)\leq\left(\int|g(\epsilon,s,a)|^{p}dsda\right)^{\frac{1}{p}}$
(15)

Assume $g(\epsilon,s,a)$ can be upper-bounded by some function $h(\epsilon)$
that only depends on $\epsilon$. This means that for each $\epsilon>0$, the
change in the reward due to the noise is bounded by $h(\epsilon)$.

$W_{p}\left(R^{*}\left(\pi^{*}\right),R^{*}\left(\pi^{*\prime}\right)\right)\leq
h(\epsilon)$ (16)

Let $C(\epsilon)=h(\epsilon)$. Therefore, the Wasserstein distance between the
reward functions obtained from the true and noisy demonstrations remains
bounded by $C(\epsilon)$, showing the robustness of the approach to noisy
demonstrations.

$W_{p}\left(R^{*}\left(\pi^{*}\right),R^{*}\left(\pi^{*\prime}\right)\right)\leq
C(\epsilon)$ (17)

Proof of Theorem 4.

We assume:

  1. 1.

The reward functions in $\mathcal{R}_{d}$ are Lipschitz continuous.

  2. 2.

The state-action space $\mathcal{S}\times\mathcal{A}$ is a compact subset of
$\mathbb{R}^{d}$.

  3. 3.

The set of reward functions $S_{d}$ that induce the same policy is non-empty
and has positive measure for each dimensionality $d$.

Lemma. For any fixed policy $\pi^{*}$, the variance of the reward functions in
$S_{d}$ that induce $\pi^{*}$ grows with the dimensionality $d$.

Proof. Consider two reward functions $R_{1},R_{2}\in S_{d}$ that both lead to
$\pi^{*}$ and are sampled randomly (with respect to some fixed distribution on
$S_{d}$ ). The variance for a specific state-action pair $(s,a)$ can be
considered as:

$\operatorname{Var}(R(s,a))=E\left[R^{2}(s,a)\right]-E^{2}[R(s,a)]$ (18)

For the entire space, the average variance is:

$V_{d}=\int_{\mathcal{S}\times\mathcal{A}}\operatorname{Var}(R(s,a))dsda$ (19)

Given our assumptions, as $d$ increases, the state-action space
$\mathcal{S}\times\mathcal{A}$ becomes more diverse. Consequently, $V_{d}$
should also increase due to the Lipschitz continuity of the reward functions
and the compactness of the state-action space.

Main Theorem:

The average pairwise $p$-Wasserstein distance, $\Delta_{d}$, grows with
dimensionality $d$.

Proof. Given the above lemma, we know that the variance of reward functions
grows as $d$ increases. By the properties of Wasserstein distances, when the
variance of a distribution increases, the Wasserstein distance also typically
grows. More formally, consider two reward functions $R_{1},R_{2}$ in $S_{d}$.
The $p$-Wasserstein distance between them is:

$W_{p}\left(R_{1},R_{2}\right)=\left(\int_{\mathcal{S}\times\mathcal{A}}\left|R_{1}(s,a)-R_{2}(s,a)\right|^{p}dsda\right)^{\frac{1}{p}}$
(20)

With the growth of $d$, the Lipschitz continuity and our lemma suggest that
the average difference between $R_{1}(s,a)$ and $R_{2}(s,a)$ for any random
$(s,a)$ pair will increase. Consequently, $W_{p}\left(R_{1},R_{2}\right)$ will
also grow, on average. Thus, integrating over all possible pairs $R_{1},R_{2}$
in $S_{d}$, it follows that: 222The term
$\frac{2}{\left|S_{d}\right|\left(\left|S_{d}\right|-1\right)}$ arises from
considering the number of unique pairwise combinations of reward functions in
$S_{d}$.

$\Delta_{d}=\frac{2}{\left|S_{d}\right|\left(\left|S_{d}\right|-1\right)}\sum_{R_{i},R_{j}\in
S_{d},i<j}W_{p}\left(R_{i},R_{j}\right)$ (21)

will increase as $d$ grows, which completes our proof.

Remarks. The proof provides a high-level understanding grounded in the
inherent properties of high-dimensional spaces and the complexity of policy
definition.