11institutetext: University of Information Technology, Ho Chi Minh City,
Vietnam 22institutetext: Vietnam National University Ho Chi Minh City, Vietnam
22email<EMAIL_ADDRESS>
22email<EMAIL_ADDRESS>
# SPBERTQA: A Two-Stage Question Answering System Based on Sentence
Transformers for Medical Texts

Nhung Thi-Hong Nguyen 1122 Phuong Phan-Dieu Ha 1122 Luan Thanh Nguyen
Corresponding Author1122 Kiet Van Nguyen 1122 Ngan Luu-Thuy Nguyen 1122

###### Abstract

Question answering (QA) systems have gained explosive attention in recent
years. However, QA tasks in Vietnamese do not have many datasets.
Significantly, there is mostly no dataset in the medical domain. Therefore, we
built a Vietnamese Healthcare Question Answering dataset (ViHealthQA),
including 10,015 question-answer passage pairs for this task, in which
questions from health-interested users were asked on prestigious health
websites and answers from highly qualified experts. This paper proposes a two-
stage QA system based on Sentence-BERT (SBERT) using multiple negatives
ranking (MNR) loss combined with BM25. Then, we conduct diverse experiments
with many bag-of-words models to assess our system’s performance. With the
obtained results, this system achieves better performance than traditional
methods.

###### Keywords:

Information Retrieval Sentence Transformer SBERT Question Answering

##  1 Introduction

Today, many websites have QA forums, where users can post their questions and
answer other users’ questions. However, they usually take time to wait for
responses. Moreover, data for question answering has become enormous, which
means new questions inevitably have duplicate meanings from the questions in
the database. In order to reduce latency and effort, QA systems based on
information retrieval (IR) retrieving a good answer from the answer collection
is essential. QA relies on open domain datasets such as texts on the web or
closed domain datasets such as collections of medical papers like PubMed [8]
to find relevant passages. Moreover, in the COVID-19 pandemic, people care
more about their health, and the number of questions posted on health forums
has increased rapidly. Therefore, QA in the medical domain plays an important
role. Lexical gaps between queries and relevant documents that occur when both
use different words to describe similar contents have been a significant
issue. Table 1 shows a typical example of this issue in our dataset. Previous
studies applied word embeddings to estimate semantic similarity between texts
to solve [26]. Various research studies approached deep neural networks and
BERT to extract semantically meaningful texts [11]. Primarily, SBERT has
recently achieved state-of-the-art performance on several tasks, including
retrieval tasks [7]. This paper focuses on exploring fine-tuned SBERT models
with MNR.

We contribute: (1) Introduce a ViHealthQA dataset containing 10,015 pairs in
the medical domain. (2) Propose two-stage QA system based on SBERT with MNR
loss. (3) Perform multiple experiments, including traditional models such as
BM25, TF-IDF cosine similarity, and Language Model to compare our system.

Table 1: A typical example of Lexical gaps in ViHealthQA dataset. ID |  392
---|---
Question |  Tôi bị dị ứng thuốc kháng sinh và dị ứng khi ăn thịt cua đồng. Trường hợp của tôi có được tiêm vaccine phòng Covid-19 không? (_I am allergic to antibiotics and eating crab meat. Can my case be vaccinated against Covid-19?_)
Answer passage |  Trường hợp của anh theo hướng dẫn của Bộ Y tế là thuộc đối tượng cần cẩn trọng khi tiêm vaccine Covid-19 và tiêm tại bệnh viện hoặc cơ sở y tế có đầy đủ năng lực cấp cứu ban đầu. (_According to the guidance of the Ministry of Health, your case is one of the subjects that need to be careful when injecting the Covid-19 vaccine and injecting it at a hospital or medical facility with total initial first aid capacity._)

##  2 Related work

In early-stage works of QA retrieval, several studies [3] presented sparse
vector models. Using unigram word counts, these models map queries and
documents to vectors having many 0 values and rank the similarity values to
extract potential documents. In 2008, Manning et al. [14] did many experiments
to gain a deeper understanding of the role of vectors, including how to
compare queries with documents. Moreover, many researchers [4, 19] pay
attention to BM25 methods in IR tasks.

IR methods with sparse vectors have a significant drawback: lexical gap
challenges. The solution to this problem is using dense embedding to represent
queries and documents. This idea was proposed early with the LSI approach [2].
However, the most well-known model is BERT. BERT applied encoders to compute
embeddings for the queries and the documents. Liu et al. [13] installed the
final mean pooling layer and then calculated similarity values between
outputs. Instead, Karpukhin et al. [9] used the initial CLS token. Many
studies [10, 12] applied BERT and reached significant results. Significantly,
SBERT [18] uses Siamese and triplet network structures to represent
semantically meaningful sentence embeddings. Multiple research approaches have
approached SBERT for Semantic Textual Similarity (STS) and Natural Language
Inference (NLI) benchmarks. In 2021, Ha et al. [5] utilized SBERT to find
similar questions in community question answering. They did several
experiments on SBERT with multiple losses, including MNR loss.

Because of our task in the medical domain, we reviewed some related corpus.
For example, CliCR [22] comprises around 100,000 gap-filling queries based on
clinical case reports, and MedQA [28] includes answers for real-world
multiple-choice questions. In Vietnam, Nguyen et al., 2021 [24] published
ViNewsQA, including 22,057 human-generated question-answer pairs. This dataset
supports machine reading comprehension tasks.

##  3 Task description

There are $n$ question-answer passage pairs in the database. We have a
collection of questions ${Q=\\{q_{1},q_{2},...,q_{n}\\}}$ and a collection of
answer passages ${A=\\{a_{1},a_{2},...,a_{n}\\}}$. Our task is creating models
with question $i$ ${(q_{i})}$ belongs to collection $Q$ ${(q_{i}\in Q)}$ can
retrieve precise answer passage ${a_{i}}$ ${(a_{i}\in A)}$.

##  4 Dataset

###  4.1 Dataset characteristics

We release ViHealthQA, a novel Vietnamese dataset for question answering and
information retrieval, including 10,015 question-answer passage pairs. We
collect data from Vinmec111https://www.vinmec.com/ and
VnExpress222https://vnexpress.net/ websites by using the
BeautifulSoup333https://pypi.org/project/beautifulsoup4/ library. These ones
are forums where users ask health-related questions answered by qualified
doctors. The dataset consists of 4 features: index, question, answer passage,
and link.

###  4.2 Overall statistics

After the collecting data phase, we divide our dataset into train, dev, and
test sets. In particular, there are 7,009 pairs in Train, 993 pairs in Dev,
and 2,013 pairs in Test (Table 3).

According to Table 3, most of the answer passages are in the range of 101 –
300 words (34.1%), the second ratio is the number of answer passages with 301
– 500 words (31.13%), followed by 501 - 700 words (15.88%), and 701 - 1000
words (9.98%). Longer answer passages (over 1000 words) comprise a small
proportion (above 7.58%).

Table 2: Statistics of ViHealthQA dataset. ViHealthQA | Value
---|---
Train | 7,009
Dev | 993
Test | 2,013
Average length answer | 495.33
Average length question | 103.87
Vocabulary (word) | 18,271
Average number of sentences | 3.95

Table 3: Distribution of the answer passage length (%). Length | Answer passage
---|---
Train | Val | Test | All
< 100 | 1.24 | 1.31 | 1.64 | 1.33
101 – 300 | 34.46 | 34.34 | 32.89 | 34.1
301 – 500 | 31.13 | 30.72 | 31.25 | 31.13
501 – 700 | 15.99 | 16.31 | 15.2 | 15.88
701 – 1000 | 9.8 | 8.66 | 11.23 | 9.98
> 1000 | 7.38 | 8.66 | 7.8 | 7.58

###  4.3 Vocabulary-based analysis

To understand the medical domain, we use the WordClouds
tool444https://www.wordclouds.com/ to display visual word frequency that
appears commonly in the dataset (Figure 1). Table 4 shows the top 10 words
with the most frequency. These words are related to the medical domain.
Besides, users ask many questions about Coronavirus (COVID-19), children,
inflammatory diseases, and allergies.

Table 4: Top 10 common words in the ViHealthQA dataset. No. Word Freq. English
1 bác sĩ 7790 doctor 2 bé 3409 baby 3 xét nghiệm 3316 test 4 trẻ 3012 children
5 triệu chứng 2858 symptom 6 dị ứng 2628 allergic 7 mũi 2479 nose 8 da 1979
skin 9 tiêm chủng 1912 vaccination 10 gan 1856 liver Figure 1: Word
distribution of ViHealthQA.

##  5 SPBERTQA: A Two-Stage Question Answering System Based on Sentence
Transformers

In this paper, we propose a two-stage question answering system called
SPBERTQA (Figure 2), including BM25-based sentence retriever and SBERT using
PhoBERT fine-tuning with MNR loss. After training, the inputs (the question
and the document collection) feed into BM25-SPhoBERT. Then, we rank the top K
cosine similarity scores between sentence-embedding outputs to extract top K
candidate documents.

Figure 2: Overview of our system.

###  5.1 BM25 Based Sentence Retriever

We aim to train the model by focusing on the meaningful knowledge of our
dataset. Thus, we propose the sentence retriever stage that extracts the $K$
sentences in every answer passage the most relevant to the corresponding
question. Moreover, this stage helps solve the obstacle of the maximum length
sequence of every pre-trained BERT model is 512 tokens ($max\\_seq\\_length$
of PhoBERT = 248 tokens), while the number of answer passages over 300 tokens
in Train accounts for above 65.47%.

We use BM25 for the first stage because BM25 mostly brings good results in IR
systems [20]. Besides, most answer passages have below four sentences (Average
number of sentences in every answer passage $=3.95$ in Table 3), so we choose
$K=5$.

###  5.2 SBERT using PhoBERT and fine-tuning with MNR loss

Multiple negatives ranking (MNR) loss: MNR loss works great for IR, and
semantic search [7]. The loss function is given by Equation (1).

$L=-\frac{1}{N}\cdot\frac{1}{K}\cdot\sum_{i=1}^{K}\left[S\left(x_{i},y_{i}\right)-\log\sum_{j=1}^{K}e^{S\left(x_{i},y_{j}\right)}\right](1)$

In every batch, there are $K$ positive pairs (${x_{i},y_{i}}$: question and
positive answer passage), and each positive pair has $K-1$ random negative
answer passages ${(y_{j},i\neq j)}$. The similarity between question and
answer passage $(S(x,y))$ is cosine similarity. Moreover, $N$ is the Train
size.

In the second stage, we use the pre-trained PhoBERT model. PhoBERT [15] is the
first public large-scale monolingual language model for Vietnamese. PhoBERT
pre-training approach is based on RoBERTa, which optimizes more robust
performance. Then, we fine-tune PhoBERT with MNR loss.

##  6 Experiments

###  6.1 Comparative methods

We compare our system with traditional methods such as BM25, TFIDF-Cos, and
LM; pre-trained PhoBERT; and fine-tuned SBERT such as BM25-SXMLR and
BM25-SmBERT.

####  6.1.1 BM25

BM25 is an optimized version of TF-IDF. Equation (2) portrays the BM25 score
of document ${D}$ given a query ${q}$. ${d_{avg}}$ is the length of the
average document. Moreover, BM25 adds two parameters: ${k}$ helps balance the
value between term frequency and ${IDF}$, and ${b}$ adjusts the importance of
document length normalization. In 2008, Manning et al. [14] suggested
reasonable values are $k=[1.2,2.0]$ and $b=0.75$.

$BM25(D,q)=\underbrace{\frac{f(q,D)*(k+1)}{f(t,D)+k*\left(1-b+b*\frac{D}{d_{avg}}\right)}}_{TF}*\underbrace{\log\left(\frac{N-N(q)+0.5}{N(q)+0.5}+1\right)}_{IDF}\quad\quad(2)$

####  6.1.2 TF-IDF Cosine Similarity (TFIDF-Cos)

Cosine similarity is one of the most popular similarity measures applied to
information retrieval applications and is superior to the other measures such
as the Jaccard measure and Euclidean measure [21]. Given ${a}$ and ${b}$ as
the respective TF-IDF bag-of-words of question and answer passage. The
similarity between ${a}$ and ${b}$ is calculated by Equation (3) [16].

$Cos(\vec{a},\vec{b})=\frac{\vec{a}\cdot\vec{b}}{\|\vec{a}\|\|\vec{b}\|}=\frac{\sum_{1}^{n}a_{i}b_{i}}{\sqrt{\sum_{1}^{n}a_{i}^{2}}\sqrt{\sum_{1}^{n}b_{i}^{2}}}\quad\quad(3)$

####  6.1.3 Language Model (LM)

LM is a probabilistic model of text [23]. Questions and answers are modeled
based on a probability distribution over sequences of words. The original and
basic method for using LM is unigram query likelihood (Equation (4)).

$P\left(q_{i}\mid D\right)=\left(1-\alpha_{D}\right)*P\left(q_{i}\mid
D\right)+\alpha_{D}*P\left(q_{i}\mid C\right)\quad\quad(4)$

${P(q|D)}$ is the probability of the query q under the language model derived
from ${D}$. ${P(q|C)}$ denotes a background corpus to compute unigram
probabilities to avoid 0 scores [27]. Besides, various smoothing based on how
to handle ${\alpha_{D}}$ and ${\alpha_{D}\in[0,1]}$.

####  6.1.4 PhoBERT

We directly use PhoBERT to encode question and answer passages. Then, we rank
the top K answer passages having the highest cosine similarity scores with the
corresponding question.

####  6.1.5 BM25-SXLMR

Similar to our model, but in the second stage, we use XLM-RoBERTa instead of
PhoBERT. XLM-RoBERTa [1] was pre-trained on 2.5TB of filtered CommonCrawl data
containing 100 languages (including Vietnamese).

####  6.1.6 BM25-SmBERT

Similar to our model, but in the second stage, we use BERT multilingual. BERT
multilingual was introduced by [17]. This model is a transformers model pre-
trained on the enormous Wikipedia corpus with 104 languages (including
Vietnamese) using a masked language modeling (MLM) objective.

###  6.2 Data preprocessing

We pre-process data such as lowercase, removing uninterpretable characters
(e.g., new-line and extra whitespace). In order to tokenize data, we employ
the RDRSegmenter of VnCoreNLP [25]. Moreover, stop-words can become noisy
factors for traditional methods working well on pairs with high word matching
between query and answer. Therefore, we conduct the removing stop-words phase.
Firstly, we use TF-IDF to extract stop-words, and then we remove these words
from the data.

###  6.3 Experimental settings

We choose xlm-roberta-base555https://huggingface.co/xlm-roberta-base, bert-
base-multilingual-cased666https://huggingface.co/bert-base-multilingual-cased,
and vinai/phobert-base777https://huggingface.co/vinai/phobert-base. Then, we
fine-tune SBERT with 15 epochs, batch size of 32, learning rate of $2e^{-5}$,
and maximum length of 256. Our experiments are performed on a single NVIDIA
Tesla P100 GPU on the Google Collaboratory
server888https://colab.research.google.com/.

###  6.4 Evaluation metric

$P@K$ (Equation (5)) is the percentage of questions for which the exact answer
passage appears in one of the $K$ retrieved passages [24].

$P@K=\frac{1}{|Q|}\sum_{1}^{n}\left\\{\begin{array}[]{c}1\quad a_{q}\in
A_{K}(q)\\\ 0\quad Otherwise\end{array}\right.(5)$

Where, ${Q={q_{1},q_{2},...,q_{n}}}$: collection of questions and ${q\in Q}$.
${A={a_{1},a_{2},...,a_{n}}}$: collection of answer passages. ${a_{q}}$ is
exact answer-passage of question $q$. ${A_{K}(q)\subseteq A}$ is the $K$ most
relevant passages extracted for question $q$.

Besides, mean average precision (mAP) is used to evaluate the performance of
models.

##  7 Results and Discussion

###  7.1 Results and discussion

With the results shown in Table 6 and 6, our system achieves the best
performance with 62.25% mAP score, 50,92% $P@1$ score, and 83.76% $P@10$ score
on the Test. BM25-SXLMR and BM25-SmBERT utilizing multilingual BERT do not
work better than our system using monolingual PhoBERT. Compared to the PhoBERT
model without fine-tuning with MNR loss, models fine-tuned with MNR
(BM25-SXLMR, BM25-SmBERT, and our system) have good results, which proves that
using MNR loss to fine-tune models for this task is suitable.

Table 5: Results on Dev and Test with $P@K$ score (%). Model | $P@1$ | $P@10$
---|---|---
Dev | Test | Dev | Test
BM25 | 51.86 | 44.96 | 75.93 | 70.09
LM | 52.27 | 47.19 | 78.15 | 72.38
TFIDF-Cos | 47.63 | 39.54 | 75.13 | 70.39
PhoBERT | 8.36 | 6.95 | 31.72 | 23.10
| BM25 - SXLMR
---
53.58 | 46.05 | 85.90 | 79.04
| BM25 - SmBERT
---
49.85 | 44.91 | 81.97 | 75.71
Our system | 69.52 | 50.92 | 89.12 | 83.76

Table 6: Results on Dev and Test with mAP score (%). Model | Dev | Test
---|---|---
BM25 | 64.62 | 56.93
LM | 56.01 | 56.00
TFIDF-Cos | 57.12 | 50.31
PhoBERT | 16.08 | 12.45
BM25-SXLMR | 59.96 | 53.85
BM25-SmBERT | 60.77 | 55.52
Our system | 69.52 | 62.25

###  7.2 Analysis

Figure 3: Results of lexical overlap experiments with P@1 (%).

To understand deeply about our system is more robust than traditional methods,
and traditional methods have disadvantages in lexical gap issues, we run
models on pairs having lexical overlap (the number of duplicate words between
question and answer passage – $X$) from 0 to 10. As results are shown in
Figure 3, with $X<4$, bag-of-words methods cannot extract the precise answer.
Especially with $X=0$, these models mostly do not work. While, with $X=0$,
fine-tuned models have results with an upper 50% $P@1$ score. From $K=3$,
these models have good scores with an upper 80% $P@1$ score. Moreover, we
provide typical examples of Dev predicted by BM25, LM, and our system (Table
7). ID 169 has word matching between question and answer passage. The models
that can retrieve precise answers are BM25, LM, and our system. In contrast,
in ID 776, no words of question appear in the answer passage. Hence, the
models must understand the semantic backgrounds instead of capturing high
lexical overlap information to retrieve the precise answer. BERT models
capture context and meaning better than bag-of-words methods [6]. In
particular, SBERT can derive semantically meaningful sentence embeddings [18].
Therefore, our system based on sentence transformers can find the exact answer
passage for the question with ID 776.

Table 7: Examples in Dev predicted by traditional methods and our system. ID |  Question |  Answer passage |  Models
---|---|---|---
776 |  Tai biến, chân tay tê bì điều trị như thế nào? (_How is the stroke and tingling in in hands and feet treated?_) |  Nếu vấn đề chính là rối loạn điện giải, nhiễm trùng huyết và gan thận, bạn nên đưa bố đến khám chuyên khoa Nội tiết hoặc Nội tổng quát. Về thần kinh, bác sĩ khám cần xem lại phim CT/MRI não để đánh giá lại tổn thương não mới có thể có được kế hoạch phòng ngừa đột quỵ tái phát, điều trị giảm đau thần kinh và phục hồi chức năng tối ưu. (_If the main problem is electrolyte disturbances, sepsis, and hepatobiliary disease, you should take him to see an Endocrinologist or General Internal Medicine. Neurologically, the examining doctor needs to review the brain CT/MRI film to re-evaluate the brain damage so that he can have a plan to prevent recurrent stroke, treat neuropathic pain, and restore optimal function._) |  Our system
169 |  Bệnh suy tủy xương vô căn có nguy hiểm không và điều trị thế nào? (_Is bone marrow failure syndromes dangerous and how is treatment?_) |  Suy tủy xương vô căn tùy thuộc vào từng giai đoạn thì cách điều trị khác nhau. Nếu số lượng máu quá thấp thì phải điều trị ức chế miễn dịch hoặc ghép tủy. Có những bệnh nhân không đáp ứng với điều trị, tuy nhiên cũng có nhiều bệnh nhân chữa khỏi. (_Bone marrow failure syndromes depends on the stage, the treatment is different. If the blood count is too low, then immunosuppressive therapy or bone marrow transplant is required. Some patients do not respond to treatment, but many patients are cured._) |  BM25, LM, and our system

##  8 Conclusion and Future Work

In this paper, we created the ViHealthQA dataset that comprises 10,015
question-answer passage pairs in the medical domain. Every answer passage is a
doctor’s reply to the corresponding user’s question, so the ViHealthQA dataset
is suitable for real search engines. Secondly, we propose the SPBERTQA, a two-
stage question answering system based on sentence transformers on our dataset.
Our proposed system performs best over bag-of-word-based models and fine-tuned
multilingual pre-trained language models. This system solves the problem of
linguistic gaps.

In future, we plan to employ the machine reading comprehension (MRC) module.
This module helps extract answer spans from answer passages so that users can
comprehend the meaning of the answer faster.

## Acknowledgement

Luan Thanh Nguyen was funded by Vingroup JSC and supported by the Master
Scholarship Programme of Vingroup Innovation Foundation (VINIF), Vingroup Big
Data Institute (VinBigdata), VINIF.2021.ThS.41.

## References

  * [1] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov.  Unsupervised cross-lingual representation learning at scale.  pages 8440–8451, 01 2020.
  * [2] Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman.  Indexing by latent semantic analysis.  Journal of the American society for information science, 41(6):391–407, 1990.
  * [3] S. F. Dierk.  The smart retrieval system: Experiments in automatic document processing — gerard salton, ed. (englewood cliffs, n.j.: Prentice-hall, 1971, 556 pp., $15.00).  IEEE Transactions on Professional Communication, PC-15(1):17–17, 1972.
  * [4] Mathias Géry and Christine Largeron.  Bm25t: A bm25 extension for focused information retrieval.  Knowledge and Information Systems - KAIS, 32:1–25, 07 2011.
  * [5] Thi-Thanh Ha, Van-Nha Nguyen, Kiem-Hieu Nguyen, Kim-Anh Nguyen, and Quang-Khoat Than.  Utilizing sbert for finding similar questions in community question answering.  In 2021 13th International Conference on Knowledge and Systems Engineering (KSE), pages 1–6, 2021.
  * [6] Shuguang Han, Xuanhui Wang, Michael Bendersky, and Marc Najork.  Learning-to-rank with bert in tf-ranking.  ArXiv, abs/2004.08476, 2020.
  * [7] Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-Hsuan Sung, László Lukács, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil.  Efficient natural language response suggestion for smart reply.  ArXiv, abs/1705.00652, 2017.
  * [8] Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu.  Pubmedqa: A dataset for biomedical research question answering.  In EMNLP, 2019.
  * [9] Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih.  Dense passage retrieval for open-domain question answering.  arXiv preprint arXiv:2004.04906, 2020.
  * [10] Md Tahmid Rahman Laskar, Jimmy Xiangji Huang, and Enamul Hoque.  Contextualized embeddings based transformer encoder for sentence similarity modeling in answer selection task.  In Proceedings of the 12th Language Resources and Evaluation Conference, pages 5505–5514, Marseille, France, May 2020. European Language Resources Association.
  * [11] Md Tahmid Rahman Laskar, Xiangji Huang, and Enamul Hoque.  Contextualized embeddings based transformer encoder for sentence similarity modeling in answer selection task.  In Proceedings of The 12th Language Resources and Evaluation Conference, pages 5505–5514, 2020.
  * [12] Kenton Lee, Ming-Wei Chang, and Kristina Toutanova.  Latent retrieval for weakly supervised open domain question answering.  pages 6086–6096, 01 2019.
  * [13] Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau.  How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation.  In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2122–2132, Austin, Texas, November 2016. Association for Computational Linguistics.
  * [14] Christopher Manning, Prabhakar Raghavan, and Hinrich Schütze.  An Introduction to Information Retrieval DRAFT, volume 1.  01 2008.
  * [15] Dat Quoc Nguyen and Anh Tuan Nguyen.  PhoBERT: Pre-trained language models for Vietnamese.  In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1037–1042, Online, November 2020. Association for Computational Linguistics.
  * [16] Bhagyashree Pathak and Niranjan Lal.  Information retrieval from heterogeneous data sets using moderated idf-cosine similarity in vector space model.  In 2017 International Conference on Energy, Communication, Data Analytics and Soft Computing (ICECDS), pages 3793–3799, 2017.
  * [17] Telmo Pires, Eva Schlinger, and Dan Garrette.  How multilingual is multilingual bert?  arXiv preprint arXiv:1906.01502, 2019.
  * [18] Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  pages 3973–3983, 01 2019.
  * [19] Stephen Robertson, Hugo Zaragoza, and Michael Taylor.  Simple bm25 extension to multiple weighted fields.  pages 42–49, 01 2004.
  * [20] Stephen Robertson, Hugo Zaragoza, and Michael Taylor.  Simple bm25 extension to multiple weighted fields.  In Proceedings of the thirteenth ACM international conference on Information and knowledge management, pages 42–49, 2004.
  * [21] R. Subhashini and V. Jawahar Senthil Kumar.  Evaluating the performance of similarity measures used in document clustering and information retrieval.  In 2010 First International Conference on Integrated Intelligent Computing, pages 27–31, 2010.
  * [22] Simon Šuster and Walter Daelemans.  CliCR: a dataset of clinical case reports for machine reading comprehension.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1551–1563, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
  * [23] Bin Tan, Xuehua Shen, and ChengXiang Zhai.  Mining long-term search history to improve search accuracy.  In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’06, page 718–723, New York, NY, USA, 2006. Association for Computing Machinery.
  * [24] Kiet Van Nguyen, Tin Van Huynh, Duc-Vu Nguyen, Anh Gia-Tuan Nguyen, and Ngan Luu-Thuy Nguyen.  New vietnamese corpus for machine reading comprehension of health news articles.  arXiv preprint arXiv:2006.11138, 2020.
  * [25] Thanh Vu, Dat Quoc Nguyen, Dai Quoc Nguyen, Mark Dras, and Mark Johnson.  VnCoreNLP: A Vietnamese natural language processing toolkit.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations, pages 56–60, New Orleans, Louisiana, June 2018. Association for Computational Linguistics.
  * [26] Xin Ye, Hui Shen, Xiao Ma, Razvan Bunescu, and Chang Liu.  From word embeddings to document similarities for improved information retrieval in software engineering.  In 2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE), pages 404–415, 2016.
  * [27] Chengxiang Zhai and John Lafferty.  A study of smoothing methods for language models applied to information retrieval.  ACM Trans. Inf. Syst., 22(2):179–214, apr 2004.
  * [28] Xiao Zhang, Ji Wu, Zhiyang He, Xien Liu, and Ying Su.  Medical exam question answering with large-scale reading comprehension.  In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence and Thirtieth Innovative Applications of Artificial Intelligence Conference and Eighth AAAI Symposium on Educational Advances in Artificial Intelligence, AAAI’18/IAAI’18/EAAI’18. AAAI Press, 2018.


# Tell my why: Training preferences-based RL with human preferences and step-
level explanations

Jakob Karalus
Ulm University
Institute of Artificial Intelligence
Ulm, Germany
<EMAIL_ADDRESS>

###### Abstract

Human-in-the-loop reinforcement learning (HRL) allows the training of agents
through various interfaces, even for non-expert humans. Recently, preference-
based methods (PBRL), where the human has to give his preference over two
trajectories, increased in popularity since they allow training in domains
where more direct feedback is hard to formulate. However, the current PBRL
methods have limitations and do not provide humans with an expressive
interface for giving feedback. With this work, we propose a new preference-
based learning method that provides humans with a more expressive interface to
provide their preference over trajectories and a factual explanation (or
annotation of why they have this preference). These explanations allow the
human to explain what parts of the trajectory are most relevant for the
preference. We allow the expression of the explanations over individual
trajectory steps. We evaluate our method in various simulations using a
simulated human oracle (with realistic restrictions), and our results show
that our extended feedback can improve the speed of learning. Code & data:
github.com/under-rewiev

##  1 Introduction

To allow for more flexible deployments of agents in all areas of human life,
it is highly desirable that these can be quickly and easily adapted to
different needs. While there are many different methods for training agents,
Human-in-the-Loop Reinforcement Learning puts the responsibility on the human
who gives feedback (or other methods like advice or demonstrations) from which
the robots can learn (Amershi et al. (2014)). This has the advantage that
there is no need to program hard-coded actions or extensive modeling for
planning or designing reward functions, which can be a cumbersome and brittle
process in itself (Booth et al. (2023)). Pushing the responsibility directly
to the human in the training process alleviates some of these problems while
opening the learning process up to more non-technical users (or trainers).
Naturally, there are multiple options for how a human can direct their
feedback toward the agent. In this work, we want to focus on one of the more
prominent methods of teaching agents by giving preferences over two
trajectories, often called preferences-based reinforcement learning (PBRL).
While Human-in-the-loop Reinforcement Learning (HRL) recently gained more
interest from the research community, most research is focused on purely
technical aspects with the same interface. We argue that extending that
interface and giving humans more options to control learning is important.

Our work is also motivated by the concept of "scaffolding" (van de Pol et al.
(2010), which human teachers often use when teaching tasks to students. While
scaffolding contains many techniques, like breaking the task into sub-parts,
using verbal cues, or relating the content to other knowledge, we focus on a
single technique in this work. Teachers often highlight essential features of
the tasks (or solutions) to the student. This highlighting is often called
factual explanations.

Inspired by human scaffolding behavior, our main contribution is extending the
common preference-based interface to allow the human to explain his decision
more. More specifically, we allow the human to select timesteps of both
trajectories (of the pair) to annotate which steps they deem important for his
decision. This can be seen as a factual explanation. Our method then uses
these explanations as additional training input. We archive this by generating
an explanation of the reward model through gradients (e.g., saliency-based
explanations) and comparing the generated explanations with the human
explanations. This additional loss can be transparently integrated into the
reward model learning of any preference-based reinforcement learning
framework.

##  2 Related Work

Multiple works have attempted to extend the pipeline of Human-in-the-Loop RL
to include more than (evaluative) feedback. Guan et al. (2020) implemented an
HRL method that allows the user to highlight important areas of an image in
addition to evaluative feedback. Our implementation can be used with any
preference-based method (which can be used with any RL algorithm), while they
designed a custom HRL method. Additionally, our proposed solution has the
advantage that it is not restricted to an image-based state space but can deal
with any state space.
Mahmud et al. (2023) showed that human explanation on individual feature space
level can help to create better rewards models from an offline dataset of
demonstrations. Again, their method is limited to cases where humans can
understand the feature space. And secondly, their work is on offline learning,
while we learn online.
Basu et al. (2018) allow the human to select which features influence their
decision between two trajectories. While they show that their method can lead
to increased learning performance, it is limited to a linear reward model,
which limits its application quite drastically. Additionally, their work is
limited in that they rely on complete trajectories (instead of segments in the
case of PBRL). From the human viewpoint, the work of Basu et al. (2018) is the
closest comparison to ours.
Saran et al. (2021) improved the efficiency of Imitation Learning by
collecting human gaze data, which is then used in an auxiliary loss
constructed from the network attention and the human. Their idea of including
additional information through an auxiliary loss, which is constructed through
gradients of the network, is similar to the approach. But their focus is more
on state regions, not timesteps, and the different settings of imitation
learning.
Karalus and Lindner (2022) used counterfactuals (instead of factuals like our
method) to increase the learning speed of TAMER (Knox and Stone (2008)
alternative to preference-based RL for evaluative feedback). They reported
increased learning performance, but their methods are for different types of
human explanations and formulated in a different framework.
Gajcin et al. (2023) lets users select part of a trajectory that they do not
like. With this information, they perform reward shaping. While they allow
humans to select parts of trajectories that they don’t like, their annotations
focus on either states or actions, not timesteps. Additionally, their setting
of reward shaping still assumes the existence of an environmental reward and
does not allow the training of agents purely from human feedback.
Wu et al. (2024) let the user select text parts (i.e., parts of their
trajectory) to specify if that part is relevant for one of their pre-defined
classes (Relevance, Factuality, etc..) While their work is solely focused on
LLM/RLHF, the concept of selecting parts of the input trajectory is similar to
our work. But this additional information is then used to train a pre-selected
hard-coded set of reward models, which makes the transfer to other domains
different. While our method is much more flexible and requires less oversight.

Therefore, the opportunity to extend the setting to allow humans to give
additional explanations has been investigated in different settings but not in
the preference-based framework. Additionally, most of the work requires hand-
crafted algorithms, while our contribution is much more independent of the
concrete PBRL method. However, most methods reported increased learning speeds
when including additional human annotations (in any form); therefore, it’s
realistic to expect changes in learning speed from our work.

Metz et al. (2023) proposed an interface with multitudes of different
annotation options for collecting human preferences over trajectories. Their
focus was purely on the interface design, which is not connected to any real
algorithm. To our knowledge, an explanatory annotation of timesteps inside
trajectories is impossible in their theoretical interface because they drew
heavy inspiration from existing research. This shows the importance of
showcasing the technical feasibility of possible feedback interface avenues so
they can be included in future human-computer-interaction research. Casper et
al. (2023) specified "fine-grained feedback" as a possibility for problems
where precise information is necessary to solve the task.

In settings other than PBRL, the concept of using gradient-based explanations
to enhance training has been applied successfully. Saliency-based methods have
been used in other contexts to speed up the learning process or force the
learner to learn the right reasons. Common examples where gradient-based
explanations have been used (in supervised learning) to accelerate learning
are ExpectedGradients (Erion et al. (2021)), Right-for-the-right-reasons (Ross
et al. (2017) or Ismail et al. (2021). These ideas are not yet been explored
in PBRL.

##  3 Background

We consider a setting wherein the sequential interactions between an agent and
its environment are formally characterized by a Markov Decision Process (MDP),
a mathematical framework comprising states, actions, and transitions. Within
this formalism, the system evolves discretely at time intervals $t.$ Within
the episodic framework, the agent’s involvement persists until reaching a
terminal time step $T$, signifying the conclusion of an episode. This temporal
progression is succinctly represented by the trajectory, expressed as the
ordered sequence $(s_{1},a_{1}),\ldots,(s_{T},a_{T}),$ encapsulating the
observation-action pairs throughout an episode.

In RL the agent experiences rewards at each time step. In our Human-in-the-
Loop context, we abstain from presuming access to such rewards. Instead, we
introduce a human overseer controlling the agent’s task intention through two
distinct feedback channels. First, their preferences (as generally in PBRL)
and second, their explanations (our contribution).

###  3.1 Preference-based Reinforcement Learning

The main goal of preference-based reinforcement learning (PBRL, Christiano et
al. (2017)) is to learn a reward function, denoted as $\hat{r}_{\psi}$, from a
set of expressed segment pairs. These pairs of segments are collected from the
human throughout the training process. Within this framework, a segment,
$\sigma$, is defined as a sequence of states and actions,
$\\{s_{k},a_{k},...,s_{k+H},a_{k+H}\\}$. Preferences, denoted as $y$, are
elicited for segments $\sigma_{0}$ and $\sigma_{1}$, with $y$ representing a
distribution indicating the preferred segment, i.e.,
$y\in\\{(0,1),(1,0),(0.5,0.5)\\}$. This evaluative judgment is recorded in a
dataset $D$ as a triple $(\sigma_{0},\sigma_{1},y)$. This dataset of binary
preference can be then used to train a reward model with the Bradley-Terry
model (Bradley and Terry (1952)):

$P_{\psi}[\sigma_{1}\succ\sigma_{0}]=\frac{\exp\sum_{t}\hat{r}_{\psi}(s_{1t},a_{1t})}{\sum_{i\in\\{0,1\\}}\exp\sum_{t}\hat{r}_{\psi}(s_{it},a_{it})}$
(1)

Here, $\sigma_{i}\succ\sigma_{j}$ denotes that segment $i$ is preferred to
segment $j$. This formulation states that the probability of preferring a
segment exponentially depends on the reward function’s sum over the segment.
While $\hat{r}_{\psi}$ itself is not inherently a binary classifier, the
learning procedure is akin to binary classification, where a supervisor
supplies labels $y$. To learn the reward function, instantiated as a neural
network with parameters $\psi$, the following loss is minimized:

$\mathcal{L}_{\text{preference}}=-\mathbb{E}_{(\sigma_{0},\sigma_{1},y)\sim\mathcal{D}}\left[y_{0}\log
P_{\psi}[\sigma_{0}\succ\sigma_{1}]+y_{1}\log
P_{\psi}[\sigma_{1}\succ\sigma_{0}]\right]$ (2)

Even in PBRL the final goal is to learn a policy $\pi$ that maximizes the
expected cumulative reward. The reward function $\hat{r}_{\psi}$ obtained from
preferences is a surrogate to the true, unknown reward function. Therefore
$\hat{r}_{\psi}$ is utilized instead of the actual reward signal while
optimizing the policy. The policy can be learned through standard RL
algorithms like PPO (Schulman et al. (2017)) or SAC (Haarnoja et al. (2018)).

###  3.2 Training Guided by Saliency-Based Methods

Saliency-based explanation methods allow an understanding of neural network
decision-making processes, offering a quantitative means to assess the impact
of input features on model predictions. Consider a neural network $f_{\theta}$
with parameters $\theta$, trained on a dataset
$\mathcal{D}=\\{(x_{i},y_{i})\\}_{i=1}^{N}$, where $x_{i}$ denotes input
samples and $y_{i}$ represents corresponding labels. To calculate a saliency-
based explanation (or saliency map) for an input $x_{i}$, a normal loss
between the prediction of the network $f_{\theta}$ and the label $y_{i}$ is
calculated. That loss then allows us to calculate the gradients of the input
$x_{i}$ (with respect to the before-mentioned loss). These gradients are then
used as an explanation of the prediction.

One notable saliency-based method is SmoothGrad (Smilkov et al. (2017)),
designed to stabilize saliency maps. The smoothed saliency map
$S_{\text{SmoothGrad}}$ is computed as the average gradient over
$N_{\text{smooth}}$ perturbed samples:

$S_{\text{SmoothGrad}}=\frac{1}{N_{\text{smooth}}}\sum_{i=1}^{N_{\text{smooth}}}\nabla_{x}f_{\theta}(x+\epsilon_{i})$
(3)

Here, $\epsilon_{i}$ represents the perturbation, and
$\nabla_{x}f_{\theta}(x+\epsilon_{i})$ is the gradient of the model’s
prediction with respect to the perturbed input. This approach introduces
controlled noise to the input, providing a more stable saliency map. The
resulting map highlights the regions crucial for the network’s predictions,
contributing to a nuanced understanding of feature importance in neural
network decision-making. Similar methods like IntegratedGradients Sundararajan
et al. (2017) calculate multiple explanations with respect to a baseline and
average the results to increase stability.

Saliency-based training methods (Ismail et al. (2021)), such as
ExpectedGradients (Erion et al. (2021)) and Right-for-the-Right-Reasons (Ross
et al. (2017)), allow the shaping of the training dynamics of neural networks.
These approaches utilize saliency information to refine the training process,
enhancing model interpretability and predictive performance. Right-for-the-
Right-Reasons emphasize salient features contributing to correct predictions
during training, steering the model towards more meaningful representations.
This is done by adding additional terms to the loss term. Instead of a single
loss term between the prediction’s distance from the target (i.e., being
right), a second (weighted) term is added to constrain the distance between a
saliency-based explanation and a given explanation (i.e., the right reason).

ExpectedGradients Erion et al. (2021) allows to guide the training even in
cases where no ground-truth explanations are available. This is done by
imposing priors on the structure of the explanation. For example, explanations
should be either locally-smooth or sparse regarding features. These saliency-
guided training strategies embody a synergy between interpretability and
maintaining predictive accuracy.

##  4 Implementation

Our main contribution is the extension of the feedback possibilities in PBRL
for humans. In addition to their preferences between the two trajectories, the
human can now explain which timesteps they deems essential for their decision.
This evaluative judgment, accompanied by detailed binary explanations provided
by the human over timesteps, is recorded the dataset $\mathcal{D}$ as a
quintuple $(\sigma_{0},\sigma_{1},y,\text{e}_{1},\text{e}_{2})$. Here,
$\text{e}_{1}$ and $\text{e}_{2}$ directly represent binary vectors of human-
provided explanations, each having the same length as the corresponding
segments $\sigma_{0}$ and $\sigma_{1}$. These binary vectors, $\text{e}_{1}$
and $\text{e}_{2}$, serve to elucidate the significance of individual
timesteps within their respective segments, providing binary indications (0 or
1) of the human’s perceived importance at each step. Figure 1 shows an
overview of our approach.

Figure 1: Overview of our approach. Left: The agent optimizes his policy with
respect to the trained proxy reward model. Right: The proxy reward model is
trained from human preferences over trajectories and the importance of each
timestep in the trajectory. Humans provide both.

###  4.1 Reward Learning with annotated preferences

Incorporating preference-based learning alongside annotations necessitates
introducing two additional steps within the learning process. First, we must
predict an explanation $\hat{e}$ and then compare this generated explanation
with the human explanation $e$.

Initially, an explanation of the existing prediction (of the reward model)
concerning the current trajectory pair is formulated. The generation of this
explanation leverages established saliency-based methods, and we employ the
SmoothGrad technique (see Equation 3) in our implementation. Here, a set of
$n_{\text{smooth}}$ perturbations is generated for a given input trajectory by
sampling from a normal distribution with mean and standard deviation
parameters derived from the input trajectory. On these perturbations, we then
calculate the saliency-based explanations.

$\xi(\sigma)=\left[\sum\left|\frac{1}{N_{\text{smooth}}}\sum_{i}^{N_{\text{smooth}}}\nabla_{s,a}\hat{r}_{\psi}(s+\epsilon_{i},a+\epsilon_{i})\right|\>\right]\>\forall
s,a\in\sigma$ (4)

$\epsilon$ represents the perturbation of the input. We then predict the
outcome (reward) with the reward model $\hat{r}_{\psi}$ and calculate the
gradient with respect to the input, i.e., the state and action. This done for
all timesteps $t$ in the segment $\sigma$.

Since the human explanations ($\text{e}_{1}$ and $\text{e}_{2}$) are binary
vectors with the length of the segments $(\sigma_{0},\sigma_{1})$, but the
saliency-based explanations $\hat{e}$ are of shape $(s,a)_{t}$ (a
concatenation of state and actions space for each timestep in the segment).
Therefore, the generated explanations must be transformed from the state &
action frame space to a step-level explanation. To achieve this, we sum the
absolute values over each timestep to transform the frame-based values into a
step-level explanation. These unnormalized values then serve as logits for the
binary-cross entropy loss, aligning the calculated explanation with the
provided annotation, the latter specified by the user’s annotations.

The annotation loss $L_{\text{annotation}}$ is the standard binary multi-class
cross-entropy between our explanation-logits and the true explanation:

$L_{\text{annotation}}=\frac{1}{2N}\sum_{e,\sigma\in\\{(e_{0},\sigma_{0})(e_{1},\sigma_{1})\\}\in\mathcal{D}}[-(e\cdot\log(f(\xi(\sigma)))+(1-e\cdot\log(1-f(\xi(\sigma)))]$
(5)

where $e$ is the explanation of the user (for the human preference $y$) and
$\xi(\sigma)$ generates the explanation on a trajectory step level. $N$ is the
total number of preferences (i.e., a pair of segments) in the database $D$.
The user’s preferences are $e$, and our generated preferences are transformed
with $\xi(\sigma)$. Since $\xi$ produces normalized logits, we apply the
sigmoid function for $f$. To relax the learning constraints, we employ label
smoothing for the human explanations $e$ to relax the constraints of having
the correct explanations.

Theoretically, we could allow the user to give specific annotations, i.e.,
highlighting single entries in the state. Still, there are reasons to limit
the annotations’ granularity to the trajectory’s timestep level. First, the
timestep level of the trajectory is quite application-independent and does not
require the human to fully understand lower levels (like the state and action
space). For example, in many robots, the state space includes a mix of
different sensors (LIDAR, Camera, etc.), which can be hard to understand fully
for humans. Lower levels also drastically increase the mental load on humans
due to the massive number of options. Therefore, we want to show in this work
that even explanations on the timestep level contain enough information to
increase learning speed.

###  4.2 Structural Loss

Like ExpectedGradients, we want to constrain the structure of our generated
explanation and impose a structural loss on the derived explanation. Since
most human explanations are sparse, we apply this prior to our generated
explanation and force the generated explanations to reflect this sparse
characteristic. This structural loss is realized by applying an L1 norm to the
calculated explanation.

Mathematically, the structural loss $L_{\text{structural}}$ is formulated as:

$L_{\text{structural}}=\frac{1}{2N}\sum_{\sigma\in\\{\sigma_{0},\sigma_{1}\\}\in\mathcal{D}}\|\xi(\sigma)\|_{1}$
(6)

where $N$ denotes the total number of trajectories in $D$, and $\xi$ generates
a step level explanation for the segment $\sigma$. Incorporating this
structural loss contributes to the regularization of the explanation,
fostering a more coherent and interpretable representation.

To derive our reward model $\hat{r}_{\phi}$, we integrate all three
distinctive loss terms: the primary preference loss (Equation 2), the
annotation loss (Equation 5), and the structural loss (Equation 6), merging
them into a unified loss term. The incorporation of the additional losses is
accompanied by weightings ($\alpha_{1}$ and $\alpha_{2}$):

$L_{\text{total}}=L_{\text{preference}}+\alpha_{1}\cdot
L_{\text{annotation}}+\alpha_{2}\cdot L_{\text{structural}}$ (7)

This loss formulation ensures the simultaneous consideration of preference-
based learning, explanation fidelity, and structural regularization in the
training of the final reward model $\hat{r_{\phi}}$.

###  4.3 Policy Learning

To learn an actual policy from the learned reward function, we use the same
setup as PEBBLE Lee et al. (2021a), in which the offline learning algorithm
SAC is leveraged. Like in PEBBLE, we update the replay buffer every time the
reward function is updated. This allows the agent to always learn from the
latest experience. Since we extend from PEBBLE, it’s also the choice of our
baseline for the evaluation.

##  5 Evaluation

We evaluate our changes in three common robotics environments, Walker,
HalfCheetah, and Hopper of the mujoco suite. Most of the hyperparameters are
taken from our baseline PEBBLE (Lee et al. (2021a)), which we then briefly
tuned only on the baseline (since we argue that our work should be viewed as
an extension to the existing method, we believe it should not deserve a full
hyperparameter search). The ballpark for the hyperparameter of our extension
(i.e., the weight of additional loss terms) was also taken from similar
methods (Ross et al. (2017)) and briefly sanity-checked. The agent trains for
2 million steps in each environment, with a total feedback budget of 700
comparisons. A full overview of all hyperparameters can be found in Appendix A

While we are learning in an HRL setting, where we don’t have access to the
true reward function, we still use the true reward function for the
evaluation. To provide a solid measurement of the learning progress, we
perform every 10k environment steps performance measurement of the agent. This
measurement consists of 5 episodes in a newly seeded (with a different random
seed than the current iteration) environment. At each measurement, we collect
total true environmental rewards and calculate the mean between the 5
measurement episodes. With this setting, we collect 200 measurements
throughout the whole training process (of a single evaluation run). This
extensive collection of measurements gives us a more stable estimation of the
true performance (of the agent) throughout training than the normally used
rollout mean rewards. These collected measurements are then aggregated and
normalized between zero and one. We use the same procedure proposed by Agarwal
et al. (2021) to calculate mean scores and confidence intervals via
bootstrapping over 10 runs in each condition.

We opted to create a synthetic oracle instead of actual humans to allow for an
evaluation with enough statistical significance. For the comparison, we used
the environmental ground truth rewards (since in DM-control, it exists here);
for the trajectory step-level explanation, we resorted to an XAI technique
called IntegratedGradients (Sundararajan et al. (2017)) to extract the
explanations from a pre-trained policy.

The random seed is separated between the environment dynamics and the
network/agent dynamics. We select the same starting seeds for the comparisons
between our work and the baseline (to reduce the impact of "favorable" seeds).

Since the training with a perfect oracle is not a realistic setting, we test
the stability of our extension properly and use the five irrationalities of
human teachers as defined by the standard B-Pref benchmark (see Lee et al.
(2021b)). These different common failure patterns each represent different
"sub-optimality" that can occur in human feedback. We also train with the
perfect case (the oracle case).

Figure 2: The mean throughout the training process in each environment. Shaded
areas represent the confidence interval. Higher is better.

###  5.1 Results

As visible in Figure 2, we can achieve better mean rewards (the mean of 10
runs, each point is measured in 5 evaluation episodes, as described in section
5) results than the baseline with our extension in all environments with the
perfect oracle. While in the Walker and HalfCheetah environments, the mean
(bold line) is higher, the confidence interval (shaded areas) still overlaps;
therefore, we can’t extensively say that our methods are conclusively better.
Only in the Hopper case can we conclusively answer that the human explanation
helps the reward model learn faster. While not pictured, both methods converge
to the same reward in the long run. However, our method’s major motivation and
driving force was increasing learning speed, especially in the earlier phases
when the human is still in the loop.

Figure 3: Optimality Gap (how fast a run converges to the ideal score, lower
is better) under different human irrationality. The vertical line visualizes
the mean optimality gap, and the bars represent the confidence interval.

Since we want to highlight the increase in learning speed, we use the
optimality gap as our primary metric for further evaluation since it condenses
the learning speed into a single, more understandable metric. In Figure 3, we
use the optimality gap to compare our modifications in the different
irrationalities defined by BPref. Notable is the significance (indicated by no
overlapping bars, as defined in Agarwal et al. (2021)) between our work and
the baseline in quite a few cases. These results confirm that our extension
leads to better learning performance, even in settings where the human
feedback follows a more realistic pattern.

##  6 Discussion

Additional mental load Labelling trajectories not only with the binary
preference but also with the annotation of which parts are better/worse does,
of course, require more mental effort than pure preference-based (or other
evaluative approaches). In contrast, we showed that these annotations speed up
the learning process, and therefore, less total human interaction is needed.
We believe this trade-off has to be evaluative on a case-by-case basis since
it’s dependent on multiple factors like the environment, the learning
algorithm, the feedback interface, and the expertise and type of trainers.
Additionally, we want to highlight that our annotations might relieve some of
the frustrations human trainers often have when using a very restricted
feedback interface (binary in the case of preference-based learning). Our
methods give the human trainer more avenues to express his feedback, which
should increase user experience.

Granularity of the annotations While it’s technically feasible to annotate
whole states/actions in a trajectory and annotate specific parts of the state
space directly, we refrain from evaluating this scenario. We believe
preference-based learning shines in settings where the human can’t highlight
single points in the state-space because of the complexity of the state space.
We believe in areas where its possible for human to understand single points
in the state-space, methods like TAMER Knox and Stone (2008) are probably a
better alternative.

Evaluation with a simulated oracle Throughout this work, we evaluated our
modification with a synthetic oracle instead of an actual human. While this is
a weakness of the evaluation, it also allowed us to collect more runs (with
different random seeds) and perform different ablations. This increase makes
the statistical analysis more sound. We believe this benefits the research
community more than a human study with a few participants, which would not
allow us to show our gains with (statistical) confidence.

## Conclusion

With this work, we showed how preference-based reinforcement learning could be
extended so humans can give not only preferences but also explanations for
their decisions. In experiments, we showed that with these explanations, we
can increase the performance of current state-of-the-art PBRL even further.
Not only in the ideal case but also under a variety of different human
irrationalities, our extension either increases the performance or does not
reduce it.

Our method, as presented here, is meant as a first demonstration, favoring
simplicity over sophistication. A significant positive attribute of PBRL is
that its RL part is mainly decoupled from the human-in-the-loop, which means
that future advantages in RL, which bring new algorithms, can be easily
included. Our extensions continue that attribute since we only require changes
to the reward model, not the agent.

Overall, we showed potential in opening up the interface in human-in-the-
loop/preference-based RL to include more natural and human-like feedback
mechanisms instead of relying only on binary feedback.

## References

  * Agarwal et al. (2021) Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare.  Deep reinforcement learning at the edge of the statistical precipice.  _Advances in Neural Information Processing Systems_ , 2021.
  * Amershi et al. (2014) Saleema Amershi, Maya Cakmak, William Bradley Knox, and Todd Kulesza.  Power to the people: The role of humans in interactive machine learning.  _AI Magazine_ , 35(4):105–120, Dec. 2014.
  * Basu et al. (2018) Chandrayee Basu, Mukesh Singhal, and Anca D. Dragan.  Learning from richer human guidance: Augmenting comparison-based learning with feature queries.  _2018 13th ACM/IEEE International Conference on Human-Robot Interaction (HRI)_ , 2018.
  * Booth et al. (2023) Serena Booth, W. Bradley Knox, Julie Shah, Scott Niekum, Peter Stone, and Alessandro Allievi.  The perils of trial-and-error reward design: Misdesign through overfitting and invalid task specifications.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 2023.
  * Bradley and Terry (1952) Ralph Allan Bradley and Milton E. Terry.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  _Biometrika_ , 1952.  ISSN 00063444.
  * Casper et al. (2023) Stephen Casper, Xander Davies, and et al.  Open problems and fundamental limitations of reinforcement learning from human feedback.  _Transactions on Machine Learning Research_ , 2023.
  * Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ , 30, 2017.
  * Erion et al. (2021) Gabriel G. Erion, Joseph D. Janizek, Pascal Sturmfels, Scott M. Lundberg, and Su-In Lee.  Learning explainable models using attribution priors.  _Nature machine intelligence_ , 3, 2021.
  * Gajcin et al. (2023) Jasmina Gajcin, James McCarthy, Rahul Nair, Radu Marinescu, Elizabeth M. Daly, and Ivana Dusparic.  Iterative reward shaping using human feedback for correcting reward misspecification.  _Proceedings of the 2023 European Conference on Artificial Intelligence_ , abs/2308.15969, 2023.
  * Guan et al. (2020) L. Guan, Mudit Verma, Sihang Guo, Ruohan Zhang, and Subbarao Kambhampati.  Widening the pipeline in human-guided reinforcement learning with explanation and context-aware data augmentation.  In _Neural Information Processing Systems_ , 2020.
  * Haarnoja et al. (2018) Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine.  Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.  _International conference on machine learning_ , pages 1861–1870, 2018.
  * Ismail et al. (2021) Aya Abdelsalam Ismail, Hector Corrada Bravo, and Soheil Feizi.  Improving deep learning interpretability by saliency guided training.  _Advances in Neural Information Processing Systems_ , 34, 2021.
  * Karalus and Lindner (2022) Jakob Karalus and Felix Lindner.  Accelerating the learning of TAMER with counterfactual explanations.  In _2022 IEEE International Conference on Development and Learning (ICDL)_ , pages 362–368. IEEE, 2022.
  * Knox and Stone (2008) W. Bradley Knox and Peter Stone.  TAMER: Training an Agent Manually via Evaluative Reinforcement.  In _IEEE 7th International Conference on Development and Learning_ , August 2008.
  * Lee et al. (2021a) Kimin Lee, Laura Smith, and Pieter Abbeel.  PEBBLE: Feedback-efficient interactive reinforcement learning via relabeling experience and unsupervised pre-training.  In _International Conference on Machine Learning_ , 2021a.
  * Lee et al. (2021b) Kimin Lee, Laura Smith, Anca Dragan, and Pieter Abbeel.  B-Pref: Benchmarking preference-based reinforcement learning.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)_ , 2021b.
  * Mahmud et al. (2023) Saaduddin Mahmud, Sandhya Saisubramanian, and Shlomo Zilberstein.  Explanation-guided reward alignment.  In _Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23_. International Joint Conferences on Artificial Intelligence Organization, 2023.
  * Metz et al. (2023) Yannick Metz, David Lindner, Raphael Baur, Daniel A. Keim, and Mennatallah El-Assady.  RLHF-Blender: A configurable interactive interface for learning from diverse human feedback.  _Interactive Learning with Implicit Human Feedback Workshop at ICML 2023_ , abs/2308.04332, 2023.
  * Ross et al. (2017) Andrew Slavin Ross, Michael C. Hughes, and Finale Doshi-Velez.  Right for the right reasons: Training differentiable models by constraining their explanations.  _Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17_ , 2017.
  * Saran et al. (2021) Akanksha Saran, Ruohan Zhang, Elaine Schaertl Short, and Scott Niekum.  Efficiently guiding imitation learning algorithms with human gaze.  _International Conference on Autonomous Agents and Multiagent Systems (AAMAS)_ , abs/2002.12500, 2021.
  * Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov.  Proximal policy optimization algorithms.  _ArXiv_ , abs/1707.06347, 2017.
  * Smilkov et al. (2017) Daniel Smilkov, Nikhil Thorat, Been Kim, Fernanda Viégas, and Martin Wattenberg.  Smoothgrad: removing noise by adding noise.  _Workshop on Visualization for Deep Learning, ICML_ , 2017.
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan.  Axiomatic attribution for deep networks.  In _International conference on machine learning_. PMLR, 2017.
  * van de Pol et al. (2010) Janneke van de Pol, Monique Volman, and Jos Beishuizen.  Scaffolding in teacher–student interaction: A decade of research.  _Educational Psychology Review_ , 22, 09 2010.
  * Wu et al. (2024) Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi.  Fine-grained human feedback gives better rewards for language model training.  _Advances in Neural Information Processing Systems_ , 36, 2024.

##  Appendix A Hyperparameters

We use the following hyperparameters for our experiments. Most of the common
hyperparameters are from Lee et al. [2021a] or Christiano et al. [2017].

Hyperparameter | Value
---|---
SAC
Learning rate | 0.0003
Batch size | 512
Buffer size | 1000000
$\tau$ | 0.005
$\gamma$ | 0.99
Reward Model
Ensemble size | 3
Number of hidden units | 300
Number of hidden layers | 3
Learning rate | 5e-4
Batch size | 32
Adam $\beta^{\prime}s$ | (0.9, 0.9)
AdamW weight decay | 0.05
Our
$\alpha_{1}$ | 0.25
$\alpha_{2}$ | 0.1
Smoothgrad number of pertubations | 16
Smoothgrad noise | 0.01
Feedback
Maximum number of feedback | 700
Feedback per interval | 70
Feedback is given every X steps | 20000
Maximum segment size | 50
Segment sampling scheme | Disagrement
Table 1: Hyperparameters

###  A.1 Compute Power

Most of the experiments were performed on a single NVIDIA A100 with a single
seed run in environments that took roughly 4 hours. Many runs were calculated
in parallel on the same machine to increase speed.