###### Abstract

Recent advances in deep neural networks, language modeling and language
generation have introduced new ideas to the field of conversational agents. As
a result, deep neural models such as sequence-to-sequence, memory networks,
and the Transformer have become key ingredients of state-of-the-art dialog
systems. While those models are able to generate meaningful responses even in
unseen situations, they need a lot of training data to build a reliable model.
Thus, most real-world systems have used traditional approaches based on
information retrieval (IR) and even hand-crafted rules, due to their
robustness and effectiveness, especially for narrow-focused conversations.
Here, we present a method that adapts a deep neural architecture from the
domain of machine reading comprehension to re-rank the suggested answers from
different models using the question as a context. We train our model using
negative sampling based on question–answer pairs from the Twitter Customer
Support Dataset. The experimental results show that our re-ranking framework
can improve the performance in terms of word overlap and semantics both for
individual models as well as for model combinations.

###### keywords:

conversational agents; chatbots; machine reading comprehension; question
answering; information retrieval; answer re-ranking.

10 1 82 Received: date; Accepted: date; Published: date Machine Reading
Comprehension for Answer Re-Ranking in Customer Support Chatbots Momchil
Hardalov 1,∗, Ivan Koychev 1 and Preslav Nakov 2 Momchil Hardalov, Ivan
Koychev and Preslav Nakov Correspondence<EMAIL_ADDRESS>
##  1 Introduction

The growing popularity of smart devices, personal assistants, and online
customer support systems has driven the research community to develop various
new methodologies for automatic question answering and chatbots. In the domain
of conversational agents, two general types of systems have become dominant:
(i) retrieval-based, and (ii) generative. While the former produce clear and
smooth output, the latter bring flexibility and the ability to generate new
unseen answers.

In this work, we focus on finding the most suitable answer for a question,
where each candidate can be produced by a different system, e.g., knowledge-
based, rule-based, deep neural network, retrieval, etc. In particular, we
propose a re-ranking framework based on machine reading comprehension Yu et
al. (2018); Seo et al. (2017); Chen et al. (2017) for question–answer pairs.
Moreover, instead of selecting the top candidate from the re-ranker’s output,
we use probabilistic sampling that aims to diversify the agent’s language and
to up-vote popular answers from different input models. We train our model
using negative sampling based on question–answer pairs from the Twitter
Customer Support Dataset.

In our experimental setup, we adopt a real-world application scenario, where
we train on historical logs for some period of time, and then we test on logs
for subsequent days. We evaluate the model using both semantic similarity
measures, as well as word-overlap ones such as BLEU Papineni et al. (2002) and
ROUGE Lin (2004), which come from machine translation and text summarization.

The remainder of this paper is organized as follows: Section 2 presents some
related work in the domain of conversational agents and answer combination.
Section 3 describes our framework and the general workflow for answer re-
ranking. Section 4 introduces the original dataset and explains how we used in
to build a new, task-specific one with negative sampling; it also offers
insights about the dialogs and the pre-processing. Section 5 describes our
experiments, and gives details about the training parameters. Section 6
presents the performance of each model and discusses the results. Finally,
Section 7 concludes and suggests possible directions for future work.

##  2 Related Work

###  2.1 Conversational Agents

The emergence of large conversational corpora such as the Ubuntu Dialog corpus
(Lowe et al., 2015), OpenSubtitles (Lison and Tiedemann, 2016), CoQA (Reddy et
al., 2018) and the Microsoft Research Social Media Conversation Corpus
(Microsoft Research, 2014) has enabled the use of generative models and end-
to-end neural networks in the domain of conversational agents. In particular,
sequence-to-sequence (seq2seq) models, which were initially proposed for
machine translation (Luong et al., 2015; Sutskever et al., 2014; Bahdanau et
al., 2014), got adapted to become a standard tool for training end-to-end
dialogue systems. Early vanilla seq2seq models (Vinyals and Le, 2015) got
quickly extended to model hierarchical structure (Serban et al., 2015),
context (Sordoni et al., 2015a), and combination thereof (Sordoni et al.,
2015b). While models were typically trained on corpora such as Ubuntu, some
work (Boyanov et al., 2017) has also used data from Community Question
Answering forums (Nakov et al., 2017); this means forming a training pair
involving a question and each good answer in the corresponding question-answer
thread.

More recently, the Transformer, a model without recurrent connections, was
proposed (Vaswani et al., 2017), demonstrating state-of-the-art results for
Machine Translation in various experimental scenarios for several language
pairs and translation directions, thus, emerging as a strong alternative to
seq2seq methods. The fact that it only uses self-attention makes it a lot
faster both at training and at inference time, even though its deep
architecture requires more calculations than a seq2seq model, it enables high
degree of parallelism, while maintaining the ability to model word sequences
through the mechanism of attention and positional embeddings.

In the domain of customer support, it has been shown that generative models
such as seq2seq and the Transformer perform better then retrieval-based
models, but they fail in the case of insufficient training data (Hardalov et
al., 2018). Other works have incorporated intent categories and semantic
matching into an answer selection model, which uses a knowledge base as its
source (Li et al., 2018). In the insurance domain, Feng et al. (2015) proposed
a generic deep learning approach for answer selection, based on convolutional
neural networks (CNN) LeCun et al. (1989). In Li et al. (2015) combined
recurrent neural networks based on long short-term memory (LSTM) cells
Hochreiter and Schmidhuber (1997) and reinforcement learning (RL) to learn
without the need of prior domain knowledge.

###  2.2 Answer Combination

Answer combination has been recognized as an important research direction in
the domain of customer support chatbots. For example, Qiu et al. (2017) used
an attentive seq2seq re-ranker to choose dynamically between the outputs of a
retrieval-based and a seq2seq model. Similarly, Cui et al. (2017) combined a
fact database, FAQs, opinion-oriented answers, and a neural-based chit-chat
generator, by training a meta-engine that chooses between them.

Answer combination is also a key research topic in the related field of
information retrieval (IR). For example, Pang et al. (2017) proposed a generic
relevance ranker based on deep learning and CNNs LeCun et al. (1989), which
tries to maintain standard IR search engine characteristics, such as exact
matching and query term importance, while enriching the results based on
semantics, proximity heuristics, and diversification.

##  3 Re-Ranking Model

Our re-ranking framework uses a classifier based on QANet Yu et al. (2018), a
state-of-the-art architecture for machine reading comprehension, to evaluate
whether a given answer is a good fit for the target question. It then uses the
posterior probabilities of the classifier to re-rank the candidate answers, as
shown in Figure 1.

Figure 1: Our answer re-ranking framework, based on the QANet architecture.

###  3.1 Negative Sampling

Our goal is to distinguish “good” vs. “bad” answers, but the original dataset
only contains valid, i.e., “good” question–answer pairs. Thus, we use
_negative sampling_ Mikolov et al. (2013), where we replace the original
answer to the target question with a random answer from the training dataset.
We further compare the word-based cosine similarity between the original and
the sampled answer, and, in some rare cases, we turn a “bad” answer into
“good” one if it is too similar to the original “good” answer.

###  3.2 QANet Architecture

Machine reading comprehension aims to answer a question by looking to extract
a string from a given text context. Here, we use that model to measure the
goodness of a given question–answer pair.

The first layer of the network is a standard an embedding layer, which
transforms words into low-dimensional dense vectors. Afterwards, a two-layer
highway network (Srivastava et al., 2015) is added on top of the embedding
representations. This allows the network to regulate the information flow
using a gated mechanism. The output of this layer is of dimensionality
$\\#words\times d$, where $\\#words$ is the number of words in the encoded
sentence (Note that it differs for the question vs. the answer. See Section
5.1 for more detail.) and $d$ is the input/output dimensionality of the model
for all Transformer layers, which is required by the architecture.

We experiment with two types of input embeddings. First, we use
200-dimensional GloVe (Pennington et al., 2014) vectors trained on 27 billion
Twitter posts. We compare their performance to ELMo Peters et al. (2018), a
recently proposed way to train contextualized word representations. In ELMo,
these word vectors are learned activation functions of the internal states of
a deep bi-directional language model. The latter is built upon a single
(embedding) layer, followed by two LSTM (Hochreiter and Schmidhuber, 1997)
layers, which are fed the words from a target sentence in a forward and a
backward direction, accordingly. We obtain the final embedding by taking a
weighted average over all three layers as suggested in (Peters et al., 2018).

The embedding encoder layer is based on a convolution, followed by self-
attention Vaswani et al. (2017) and a feed-forward network. We use a kernel
size of seven, $d$ filters, and four convolutional layers within a block. The
output of the layer is $f(layernorm(x))+x$, where $layernorm$ is the layer
normalization operation (Ba et al., 2016). The output again is mapped to
$\\#words\times d$ by a 1D convolution. The input and the embedding layers are
learned separately for the question and the answer.

The attention layer is a standard module for machine reading comprehension
models. We call it _answer-to-question_ (_A2Q_) and _question-to-answer_
(_Q2A_) attention, which are also known as _context-query_ and _query-context_
, respectively. Let us denote the output of the encoder for the question as
$Q$ and for the answer as $A$. In order to obtain the attention, the model
first computes a matrix $S$ with similarities between each two words for the
question and the answer, then the values are normalized using softmax. The
similarity function is defined as follows: $f(a,q)=W_{0}[a;q;a\odot q]$.

We adopt the notation $\overline{S}=softmax(S)$, which is a softmax
normalization over the rows of $S$, and
$\overline{\overline{S}}=softmax(S^{\intercal})$ is a normalization over the
columns. Then, the two attention matrices are computed as
$A2Q=\overline{S}\cdot Q^{\intercal}$, and
$Q2A=\overline{S}\cdot\overline{\overline{S}}^{\intercal}\cdot C^{\intercal}$.

The attention layer is followed by a model layer, which takes as input the
concatenation of $[a;a2q;a\odot a2q;a\odot q2a]$, where we use small letters
to denote rows from the original matrices. For the output layer, we learn two
different representations by passing the output of the model layer to two
residual blocks, applying dropout (Srivastava et al., 2014) only to the inputs
of the first one. We predict the output as
$P(a|q)=\sigma(W_{o}[M_{0};M_{1}])$. The weights are learned by minimizing a
binary cross-entropy loss.

###  3.3 Answer Selection

We experimented with two answer selection strategies: (i) max, and (ii)
proportional sampling after softmax normalization. The former strategy is
standard and it selects the answer with the highest score, while the latter
one returns a random answer with probability proportional to the score
returned by the softmax, aiming at increasing the variability of the answers.

For both strategies, we use a linear projection applied on the output of the
last residual model block, which is shows as “linear block” in Figure 1. We
can generalize the latter as follows: $o(q,a_{k})=W_{o}[M]$, where $M$ is the
concatenation of the outputs of one or more residual model blocks.

We present the formulation of the two strategies, as we introduce the
following notation: $Ans$ is the selected utterance by the agent; $o(q,a_{k})$
is the output of the model before applying the sigmoid function; $q$ is the
original question by the user; $A$ is the set of possible answers that we want
to re-rank. Equation (1) shows the selection process in the max case.

$Ans=\operatorname*{arg\,max}_{a\in A}(o(q,a))$ (1)

We empirically found that the answer selection based on the _max_ strategy
does not always perform well. As our experimental results in Tables 3 and 4
show, we can gain notable improvement by using proportional sampling after
softmax normalization, instead of always selecting the answer with the highest
probability. In our experiments, we model $Ans$ as a random variable that
follows a categorical distribution over $K=|A|$ events (candidate answers).
For each of the question–answer pairs ($q$, $a$), we define the probability
$p$ that $a$ is a good answer to $q$ using softmax as shown in Equations (2)
and (3). Finally, we draw a random sample from Equation (3) to obtain the best
matching answer.

$p|q,A\sim\textup{softmax}({o(q,a_{1}),\cdots,o(q,a_{K})})$ (2)

$Ans|p\sim\textup{Cat}(K,p)$ (3)

##  4 Data

The data and the resources that could be used to train customer support
conversational agents are generally very scarce, as companies keep
conversations locked on their own proprietary support systems. This is due to
customer privacy concerns and to companies not wanting to make public their
know-how and the common issues about their products and services. An extensive
2015 survey on available dialog corpora by Serban et al. (2018) found no good
publicly available dataset for real-world customer support.

In early 2018, this situation changed as a new open dataset for Customer
Support on Twitter (Twitter, 2019) was made available on Kaggle. It contains
3M tweets and replies for twenty big companies such as Amazon, Apple, Uber,
Delta, and Spotify, among others. As customer support topics from different
organizations are generally unrelated to each other, we focus only on tweets
related to Apple support, which represents the largest number of tweets in the
corpus.

We filtered all utterances that redirect the user to another communication
channel, e.g., direct messages, which are not informative for the model and
only bring noise. Moreover, since answers evolve over time, we divided our
dataset into a training and a testing part, keeping earlier posts for training
and the latest ones for testing. We further excluded from the training set all
conversations that are older then sixty days. For evaluation, we used dialogs
from the last five days in the dataset, to simulate a real-world scenario for
customer support. We ended up with a dataset of 49,626 question–answer pairs
divided into 45,582 for training and 4,044 for testing. Finally, we open-
sourced our code for pre-processing and filtering the data, making it
available to the research community (customer-support-chatbot, 2018).

Table 1 shows some statistics about our dataset. On the top of the table, we
can see that the average number of turns per dialog is under three, which
means that most of the dialogues finish after one answer from the customer
support. The bottom of the table shows the distribution of the words in the
user questions vs. the customer support answers. We can see that answers tend
to be slightly longer, which is natural as replies by customer support must be
extensive and helpful.

Table 1: Statistics about our dataset. (Reprinted by permission from Springer Nature: Springer Lecture Notes in Computer Science (Hardalov, M.; Koychev, I.; Nakov, P. Towards Automated Customer Support Hardalov et al. (2018)), 2018) |  Questions |  Answers
---|---|---
Avg. # words |  $21.314\,814$ |  $25.884\,436$
Min # words |  $1.000\,000$ |  $3.000\,000$
1st quantile (#words) |  $13.000\,000$ |  $20.000\,000$
Mode (# words) |  $20.000\,000$ |  $23.000\,000$
3rd quantile (#words) |  $27.000\,000$ |  $29.000\,000$
Max # words |  $136.000\,000$ |  $70.000\,000$
Overall
# question–answer pairs |  49,626
# words (in total) |  26,140
Min # turns per dialog |  $2$
Max # turns per dialog |  $106$
Avg. # turns per dialog |  2.6
Training set: # of dialogs |  45,582
Testing set: # of dialogs |  4,044

##  5 Experiments

###  5.1 Preprocessing

Since Twitter has its own specifics of writing in terms of both length (by
design, tweets have been strictly limited to 140 characters; this constraint
has been relaxed to 280 characters in 2017) and style, standard text
tokenization is generally not suitable for tweets. Therefore, we used a
specialized Twitter tokenizer Manning et al. (2014) to preprocess the data.
Then, we further replaced shorthand entries such as _’ll_ , _’d_ , _’re_ ,
_’ve_ , with the most corresponding literary form, e.g., _will_ , _would_ ,
_are_ , _have_. We also replaced shortened slang words, e.g., _’bout_ and
_’til_ , with the standard words, e.g., _about_ and _until_. Similarly, we
replaced URLs with the special word <url>, all user mentions with <user>, and
all hashtags with <hashtag>.

Due to the nature of writing in Twitter and the free form of the conversation,
some of the utterances contain emoticons and emojis. They are handled
automatically by the Twitter tokenizer and treated as a single token. We keep
them in their original form, as they can be very useful for detecting emotions
and sarcasm, which pose serious challenges for natural language understanding.

Based on the statistics presented in Section 4, we chose to trim the length of
the questions and of the answers to 60 and 70 words, respectively.

###  5.2 Training Setup

For training, we use the Adam Kingma and Ba (2015) optimizer with decaying
learning rate, as implemented in TensorFlow Abadi et al. (2016). We start with
the following values: learning rate $\eta=$5\text{\times}{10}^{-04}$$,
exponential decay rate for the 1st and the 2nd momentum $\beta_{1}=$0.9$$ and
$\beta_{2}=$0.999$$, and constant for prevention of division by zero
$\epsilon=$1\text{\times}{10}^{-7}$$. Then, we decay the learning after each
epoch by a factor of 0.99. We also apply dropout with a probability of 0.1,
and L2 weight decay on all trainable variables with
$\lambda=$3\text{\times}{10}^{-7}$$. We train each model for 42K steps with a
batch size of 64. We found these values by running a grid search on a dev set
(extracted as a fraction of the training data) and using the values suggested
in (Yu et al., 2018), where applicable.

###  5.3 Individual Models

Following (Hardalov et al., 2018), we experiment with three individual models:
(i) Information Retrieval-based (IR), (ii) Sequence-to-sequence (seq2seq) and
(iii) the Transformer.

For IR, we use ElasticSearch (ElasticSearch, 2019) with English analyzer
enabled, whitespace- and punctuation-based tokenization, and word 3-grams. We
further use the default BM25 algorithm Robertson and Zaragoza (2009), which is
an improved version of TF.IDF. For all training questions and for all testing
queries, we append the previous turns in the dialog as context.

For seq2seq, we use a bi-directional LSTM network with 512 hidden units per
direction. The decoder has two uni-directional layers connected directly to
the bi-directional layer in the encoder. The network takes as input words
encoded as 200-dimensional embeddings. It is a combination of pre-trained
GloVe Pennington et al. (2014) vectors for the known words, and a positional
embedding layer, learned as model parameters, for the unknown words. The
embedding layers for the encoder and for the decoder are not shared, and are
learned separately. This separation is due to the words used in utterances by
the customers being very different from the posts by the customer support.

For the Transformer, we use two identical layers for the encoder and for the
decoder, with four heads for the self-attention. The dimensionality of the
input and of the output is $d_{model}=256$, and the inner dimensionality is
$d_{inner}=512$. The input consists of queries with keys of dimensionality
$d_{k}=64$ and values of the same dimensionality $d_{v}=64$. The input and the
output embedding are learned separately with sinusoidal positional encoding.

###  5.4 Evaluation Measures

How to evaluate a chatbot is an open research question. As the problem is
related to machine translation (MT) and text summarization (TS), which are
nowadays also addressed using seq2seq models, researchers have been using MT
and TS evaluation measures such as BLEU Papineni et al. (2002) and ROUGE Lin
(2004), which focus primarily on word overlap and measure the similarity
between the chatbot’s response and the gold answer to the user question (here,
the answer by the customer support). However, it has been argued Liu et al.
(2016); Lowe et al. (2017) that such word-overlap measures are not very
suitable for evaluating chatbots. Thus, we adopt three additional measures,
which are more semantic in nature.

The embedding average (Lowe et al., 2015) constructs a vector for a piece of
text by taking the average of the word embeddings of its constituent words.
Then, the vectors for the chatbot response and for the gold human answer are
compared using the cosine similarity.

The greedy matching was introduced in the context of intelligent tutoring
systems Rus and Lintean (2012). It matches each word in the chatbot’s output
to the most similar word in the gold human response, where the similarity is
measured as the cosine between the corresponding word embeddings, multiplied
by a weighting term (which we set to 1), as shown in Equation (4). Since this
measure is asymmetric, we also calculate it with the arguments swapped, and
then we take the average as shown in Equation (5).

$greedy(u_{1},u_{2})=\frac{\sum_{v\in u_{1}}{}weight(v)*\max_{w\in
u_{2}}cos(v,w)}{\sum_{v\in u_{1}}{}weight(v)}$ (4)

$simGreedy(u_{1},u_{2})=\frac{greedy(u_{1},u_{2})+greedy(u_{2},u_{1})}{2}$ (5)

The _vector extrema_ (Forgues et al., 2014) was proposed for dialogue systems.
Instead of averaging the word embeddings of the words in a piece of text, it
takes the coordinate-wise maximum (or minimum), as shown in Equation (6).
Finally, the resulting vectors for the chatbot output and for the gold human
answer are compared using the cosine similarity.

$extrema(u_{i})=\begin{cases}\max u_{i},&if\max u_{i}\geq|\min u_{i}|\\\ \min
u_{i},&\text{otherwise}\end{cases}$ (6)

##  6 Evaluation Results

Below, we first discuss our auxiliary classification task, where the objective
is to predict which question–answer pair is “good”, and then we move to the
main task of answer re-ranking.

###  6.1 Auxiliary Task: Question–Answer Goodness Classification

Table 2 shows the results for the auxiliary task of question–answer goodness
classification. The first column is the name of the model. It is followed by
three columns showing the type of embedding used, the size of the hidden
layer, and the number of heads (see Section 3.2). The last column reports the
accuracy. Since our dataset is balanced (we generate about 50% positive, and
about 50% negative examples), accuracy is a suitable evaluation measure for
this task. The top row of the table shows the performance for a majority class
baseline. The following lines show the results for our full QANet-based model
when using different kinds of embeddings. We can see that contextualized
sentence-level embeddings are preferable to using simple word embeddings as in
GloVe or token-level ELMo embeddings. Moreover, while token-level ELMo
outperforms GloVe when the size of the network is small, there is no much
difference when the number of parameters grows ($d_{model}=128$,
$\\#Heads=8$).

Table 2: Auxiliary task: question–answer goodness classification results. Model | Embedding Type | d_model | Heads | Accuracy
---|---|---|---|---
Majority class | – | – | – | $50.515\,625$
QANet | GloVe | 64 | 4 | $80.578\,125$
64 | 8 | $82.828\,12$
128 | 8 | $83.421\,874$
QANet | ELMo (token level) | 64 | 4 | $82.921\,875$
64 | 8 | $83.875$
128 | 8 | $83.484\,375$
QANet | ELMo (sentence level) | 64 | 8 | $84.093\,75$
128 | 8 | $85.453\,12$

###  6.2 Answer Selection/Generation: Individual Models

Table 3 reports the performance of the individual models: information
retrieval (IR), Sequence-to-sequence (seq2seq), and the Transformer (see
Section 5.3 for more details about these models). In our earlier work
(Hardalov et al., 2018), we performed these experiments using exactly the same
experimental setup. The table is organized as follows: The first column
contains the name of the model used to obtain the best answer. The second and
the third columns report the word overlap measures: (i) BLEU@2, which uses
uni-gram and bi-gram matches between the hypothesis and the reference
sentence, and (ii) ROUGE-L Lin and Och (2004), which uses Longest Common
Subsequence (LCS). The last three columns are for the semantic similarity
measures: (i) Embedding Average (Emb Avg) with cosine similarity, (ii) Greedy
Matching (Greedy Match), and (iii) Vector Extrema (Vec Extr) with cosine
similarity. In the three latter measures, we used the standard pre-trained
word2vec embeddings because they are not learned during training, which helps
avoid bias, as has been suggested in Liu et al. (2016); Lowe et al. (2017).

We can see in Table 3 that the seq2seq model outperforms IR by a margin on all
five evaluation measures, which is consistent with previous results in the
literature. What is surprising, however, is the relatively poor performance
for the Transformer, which trails behind the seq2seq model on all evaluation
measures. We hypothesize that this is due to the Transformer having to learn
more parameters as it operates with higher-dimensional word embeddings.
Overall, the Transformer is arguably slightly better than the IR model,
outperforming it on three of the five evaluation measures.

The last row of Table 3 is not an individual model; it is our re-ranker
applied to the top answers returned by the IR model. In particular, we use
_QANet with Sentence level ELMo_ ($d_{model}=128$, $\\#Heads=8$). We took the
top-5 answer candidates (the value of 5 was found using cross-validation on
the training dataset) from the IR model, and we selected the best answer based
on our re-ranker’s scores. We can see that re-ranking yields improvements for
all evaluation measures: $+1.18$ on BLEU@2, $+0.93$ on ROUGE_L, $+1.12$ on
Embedding Average, $+0.67$ on Greedy Matching, and +1.64 in Vector Extrema.
These results show that we can get sizable performance gains when re-ranking
the top-$K$ predictions of a single model; below we will combine multiple
models.

Table 3: Main task: performance of the individual models. | Word Overlap | Semantic Similarity
---|---|---
Model | BLEU@2 | ROUGE_L | Emb Avg | Greedy Match | Vec Extr
Transformer Hardalov et al. (2018) | $12.427\,277\,110\,577\,412$ | $25.328\,589\,592\,296\,84$ | $75.350\,357\,724\,641\,75$ | $30.077\,073\,951\,984\,16$ | $39.398\,955\,533\,090\,98$
IR-BM25 Hardalov et al. (2018) | $13.732\,301\,176\,466\,532$ | $22.347\,860\,605\,005\,877$ | $76.526\,586\,397\,381\,92$ | $29.719\,912\,622\,416\,917$ | $37.986\,670\,995\,115\,75$
seq2seq Hardalov et al. (2018) | $15.104\,511\,595\,780\,986$ | $26.597\,981\,802\,417\,063$ | $77.109\,255\,163\,930\,53$ | $30.809\,532\,318\,667\,042$ | $40.231\,592\,168\,970\,955$
QANet on IR (Individual) | $$14.918\,819\,893$\pm$0.128\,209\,477$$ | $$23.299\,038\,571$\pm$0.121\,935\,742$$ | $$77.470\,088\,663$\pm$0.059\,103\,246$$ | $$30.396\,535\,191$\pm$0.059\,016\,826$$ | $$39.627\,030\,115$\pm$0.056\,555\,891$$

###  6.3 Main Task: Multi-Source Answer Re-Ranking

Next, we combine the top-$K$ answers from different models: IR and seq2seq. We
did not include the Transformer in the mix as its output is generative and
similar to that of the seq2seq model; moreover, as we have seen in Table 3
above, it performs worse than seq2seq on our dataset. We set $K=2$ for the
baseline, _Random Top Answer_ , which selects a random answer from the union
of the top $K$ answers by the models involved in the re-ranking. For the
remaining re-ranking experiments, we use $K=5$. We found these values using
cross-validation on the training dataset, trying 1–5.

The results are shown in Table 4, where different representations are
separated by a horizontal line. The first row of each group contains the name
of the model. Then, on the even rows (second, forth, etc.), we show the
results from a greedy answer selection strategy, while on the odd rows are the
results from an exploration strategy (softmax sampling). Since softmax
sampling and random selection are stochastic in nature, we include a $95\%$
confidence interval for them.

Table 4: Main task: re-ranking the top $K=5$ answers returned by the IR and the seq2seq models. Model | Word Overlap | Semantic Similarity
---|---|---
BLEU@2 | ROUGE_L | Emb Avg | Greedy Match | Vec Extr
Random Top Answer | $$14.516\,328\,934$\pm$0.123\,082\,790$$ | $$23.413\,930\,960$\pm$0.118\,796\,211$$ | $$77.214\,657\,500$\pm$0.064\,671\,076$$ | $$30.244\,576\,358$\pm$0.065\,005\,588$$ | $$38.247\,697\,434$\pm$0.199\,359\,940$$
QANet+GloVe |
d=64, h=4 | $15.178\,345\,469\,681\,423$ | $24.127\,489\,239\,551\,984$ | $78.383\,814\,994\,509\,07$ | $31.142\,418\,111\,378\,483$ | $40.854\,223\,779\,457\,804$
Softmax | $$15.812\,672\,681$\pm$0.089\,978\,745$$ | $$24.527\,707\,352$\pm$0.053\,996\,400$$ | $$78.315\,529\,844$\pm$0.079\,793\,894$$ | $$31.097\,286\,872$\pm$0.028\,871\,124$$ | $$40.509\,609\,850$\pm$0.122\,710\,191$$
d=64, h=8 | $15.413\,995\,490\,859\,303$ | $23.615\,145\,621\,599\,66$ | $78.483\,518\,493\,617\,24$ | $30.969\,496\,032\,273\,93$ | $40.805\,487\,519\,474\,156$
Softmax | $$15.897\,469\,640$\pm$0.060\,322\,919$$ | $$24.392\,476\,297$\pm$0.028\,498\,873$$ | $$78.381\,206\,911$\pm$0.038\,577\,508$$ | $$31.106\,734\,601$\pm$0.017\,817\,047$$ | $$40.662\,627\,653$\pm$0.056\,110\,335$$
d = 128, h = 8 | $15.940\,853\,825\,830\,132$ | $24.589\,339\,773\,822\,14$ | $78.291\,357\,307\,508\,48$ | $31.191\,942\,261\,508\,515$ | $40.626\,509\,935\,461\,46$
Softmax | $$16.035\,070\,659$\pm$0.080\,839\,093$$ | $$24.714\,711\,934$\pm$0.063\,554\,276$$ | $$78.362\,449\,621$\pm$0.074\,696\,344$$ | $$31.196\,202\,485$\pm$0.067\,497\,757$$ | $$40.704\,445\,016$\pm$0.053\,693\,214$$
QANet+ELMo (Token) |
d = 64, h = 4 | $15.230\,794\,657\,823\,363$ | $23.482\,390\,989\,589\,16$ | $78.246\,333\,058\,784$ | $30.769\,323\,099\,246\,908$ | $40.217\,083\,745\,604\,9$
Softmax | $$15.774\,599\,918$\pm$0.145\,520\,634$$ | $$24.441\,624\,727$\pm$0.092\,950\,497$$ | $$78.272\,501\,118$\pm$0.032\,568\,643$$ | $$31.062\,447\,538$\pm$0.048\,212\,825$$ | $$40.459\,904\,884$\pm$0.105\,238\,931$$
d = 64, h = 8 | $15.302\,739\,194\,160\,967$ | $23.407\,297\,463\,506\,286$ | $78.542\,117\,611\,342\,08$ | $30.966\,868\,253\,418\,33$ | $40.189\,469\,793\,465\,18$
Softmax | $$15.858\,178\,111$\pm$0.068\,446\,490$$ | $$24.397\,360\,279$\pm$0.064\,433\,766$$ | $$78.360\,976\,244$\pm$0.075\,414\,933$$ | $$31.111\,994\,054$\pm$0.037\,515\,481$$ | $$40.491\,637\,800$\pm$0.050\,358\,916$$
d = 128, h = 8 | $15.238\,346\,336\,693\,425$ | $23.586\,940\,133\,301\,03$ | $78.341\,047\,379\,888\,61$ | $30.895\,847\,682\,105\,25$ | $40.187\,819\,524\,983\,74$
Softmax | $$15.890\,561\,845$\pm$0.075\,167\,358$$ | $$24.552\,827\,022$\pm$0.100\,807\,807$$ | $$78.328\,864\,540$\pm$0.058\,647\,350$$ | $$31.110\,185\,304$\pm$0.047\,704\,418$$ | $$40.400\,958\,397$\pm$0.053\,828\,640$$
QANet+ELMo (Sentence) |
d = 64, h = 8 | $15.476\,033\,595\,794\,37$ | $23.881\,348\,518\,540\,29$ | $78.435\,553\,876\,916\,27$ | $30.961\,283\,999\,884\,152$ | $40.332\,224\,328\,956\,094$
Softmax | $$15.995\,253\,684$\pm$0.142\,636\,417$$ | $$24.499\,412\,456$\pm$0.330\,357\,113$$ | $$78.341\,333\,969$\pm$0.097\,839\,071$$ | $$31.129\,990\,155$\pm$0.083\,475\,562$$ | $$40.562\,566\,876$\pm$0.087\,607\,130$$
d = 128, h = 8 | $15.643\,673\,497\,249\,111$ | $24.132\,396\,421\,511\,17$ | $78.516\,467\,311\,058\,34$ | $31.139\,380\,135\,359\,822$ | $40.632\,245\,181\,311\,14$
Softmax | $$16.054\,222\,562$\pm$0.056\,937\,106$$ | $$24.807\,205\,017$\pm$0.084\,232\,102$$ | $$78.398\,183\,357$\pm$0.070\,499\,616$$ | $$31.201\,663\,932$\pm$0.056\,794\,468$$ | $$40.579\,787\,762$\pm$0.029\,219\,118$$

We can see in Table 4 that _QANet with sentence-level ELMo_ ($d_{model}=128$,
$\\#Heads=8$) performs best in terms of BLEU@2, ROUGE_L, and Greedy Matching.
Note also the correlation between higher results on the auxiliary task (see
Table 2) and improvement in terms of word-overlap measures, where we find the
largest difference between individual and re-ranked models ($+1.5$ points
absolute over the baseline, and $+0.95$ over seq2seq in terms of BLEU@2). In
terms of semantic similarity, we note the highest increase for Embedding
Average ($+1.3$ over the baseline, and $+1.4$ over seq2seq), and a smaller one
for Greedy Matching ($+1.0$ over the baseline, and $+0.4$ over seq2seq), and
Vector Extrema ($+2.6$ over the baseline, and $+0.6$ over seq2seq).

Overall, the re-ranked models are superior as evaluated on word-matching
measures, which is supported by the improvement of BLEU@2 and Embedding
Average. The smaller improvement for Greedy Matching and Vector Extrema can be
explained by the training procedure for the re-ranking model, which is based
on word comparison. However, these two measures focus on keyword similarity
between the target and the proposed answers, and generative models are better
at this. This is supported by comparing the combined model to IR-BM25, where
we see sizable improvements of $+1.5$ and $+2.0$ in terms of Greedy Matching
and Vector Extrema, respectively.

We can further see in Table 4 that using a stochastic approach to select the
best answer yields additional improvements. This strategy accounts for the
predicted goodness score for each candidate, thus, enriching the model in two
ways. First, implicit voting is used, as duplicate answer candidates are not
removed, resulting in higher selection probability of popular answers from
different input modules. Second, albeit two answers may have a very different
structure, they still can be similar in meaning, leading to very similar
scores and promoting only the first one. This behavior can be mitigated by
choosing the winner proportionally to its ranking, thus, also introducing
diversity in the chatbot’s language. This hypothesis is supported by the
results in Table 4: compare each model to the corresponding one with _softmax_
selection.

##  7 Conclusions and Future Work

We have presented a novel framework for re-ranking answer candidates for
conversational agents. In particular, we adopted techniques from the domain of
machine reading comprehension Chen et al. (2017); Seo et al. (2017); Yu et al.
(2018) to evaluate the quality of a question–answer pair. Our framework
consists of two tasks: (i) an auxiliary one, aiming to fit a goodness
classifier using QANet and negative sampling, and (ii) a main task that re-
ranks answer candidates using the learned model. We further experimented with
different model sizes and two types of embedding models: GloVe Pennington et
al. (2014) and ELMo Peters et al. (2018). Our experiments showed improvements
in answer quality in terms of word-overlap and semantics when re-ranking using
the auxiliary model. Last but not least, we argued that choosing the top-
ranked answer is not always the best option. Thus, we introduced probabilistic
sampling that aims to diversify the agent’s language and to up-vote the
popular answers, while taking their ranking scores into consideration.

In future work, we plan to experiment with different exploration strategies
such as Boltzmann exploration (softmax is a degraded version of Boltzmann
exploration with $\tau=1$). $\epsilon$-Confident, Upper confidence bound, and
other bandit methods Sutton and Barto (1998) to widen the possible context for
each answer over time. We see an interesting research direction in applying
deep reinforcement learning (i) to improve the answer selection models when
applied to unseen questions, and (ii) to account for user feedback and
customer support task success.

Conceptualization, M.H., P.N. and I.K.; Investigation, M.H; Writing–original
draft, M.H.; Writing–review & editing, P.N. and I.K.

This research is partially supported by Project UNITe BG05M2OP001-1.001-0004
funded by the OP “Science and Education for Smart Growth”, co-funded by the EU
through the ESI Funds.

The authors declare no conflict of interest.

References

## References

  * Yu et al. (2018) Yu, A.W.; Dohan, D.; Luong, M.T.; Zhao, R.; Chen, K.; Norouzi, M.; Le, Q.V.  QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension.  In Proceedings of the 2018 International Conference on Learning Representations; Vancouver, BC, Canada, 30 April–3 May 2018.
  * Seo et al. (2017) Seo, M.; Kembhavi, A.; Farhadi, A.; Hajishirzi, H.  Bi-directional attention flow for machine comprehension.  In Proceedings of the 2017 International Conference on Learning Representations; Toulon, France, 24–26 April 2017.
  * Chen et al. (2017) Chen, D.; Fisch, A.; Weston, J.; Bordes, A.  Reading Wikipedia to Answer Open-Domain Questions.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, BC, Canada, 30 July–4 August 2017; pp. 1870–1879.
  * Papineni et al. (2002) Papineni, K.; Roukos, S.; Ward, T.; Zhu, W.J.  BLEU: A Method for Automatic Evaluation of Machine Translation.  In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Philadelphia, PA, USA, 7–12 July 2002; pp. 311–318.
  * Lin (2004) Lin, C.Y.  ROUGE: A Package for Automatic Evaluation of Summaries.  In Proceedings of the ACL Workshop on Text Summarization Branches Out, Barcelona, Spain, 25–26 July, 2004; pp. 74–81.
  * Lowe et al. (2015) Lowe, R.; Pow, N.; Serban, I.; Pineau, J.  The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems.  In Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Prague, Czech Republic, 2–4 September 2015; pp. 285–294.
  * Lison and Tiedemann (2016) Lison, P.; Tiedemann, J.  OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles.  In Proceedings of the Tenth International Conference on Language Resources and Evaluation, Portorož, Slovenia, 23–28 May 2016.
  * Reddy et al. (2018) Reddy, S.; Chen, D.; Manning, C.D.  CoQA: A conversational question answering challenge.  arXiv 2018, arXiv:1808.07042.
  * Microsoft Research (2014) Microsoft Research Social Media Conversation Corpus. Available online: http://research.microsoft.com/convo/ (accessed on 20 February 2019).
  * Luong et al. (2015) Luong, T.; Pham, H.; Manning, C.D.  Effective Approaches to Attention-based Neural Machine Translation.  In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal, 17–21 September 2015; pp. 1412–1421.
  * Sutskever et al. (2014) Sutskever, I.; Vinyals, O.; Le, Q.V.  Sequence to Sequence Learning with Neural Networks.  In Proceedings of the 27th Annual Conference on Neural Information Processing Systems, Montreal, QC, Canada, 8–13 December 2014; pp. 3104–3112.
  * Bahdanau et al. (2014) Bahdanau, D.; Cho, K.; Bengio, Y.  Neural machine translation by jointly learning to align and translate.  arXiv 2014, arXiv:1409.0473.
  * Vinyals and Le (2015) Vinyals, O.; Le, Q.V.  A Neural Conversational Model.  arXiv 2015, arXiv:1506.05869.
  * Serban et al. (2015) Serban, I.V.; Sordoni, A.; Bengio, Y.; Courville, A.C.; Pineau, J.  Hierarchical neural network generative models for movie dialogues.  arXiv 2015, arXiv:1507.04808.
  * Sordoni et al. (2015a) Sordoni, A.; Galley, M.; Auli, M.; Brockett, C.; Ji, Y.; Mitchell, M.; Nie, J.Y.; Gao, J.; Dolan, B.  A Neural Network Approach to Context-Sensitive Generation of Conversational Responses.  In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Denver, CO, USA, 31 May–5 June 2015; pp. 196–205.
  * Sordoni et al. (2015b) Sordoni, A.; Bengio, Y.; Vahabi, H.; Lioma, C.; Grue Simonsen, J.; Nie, J.Y.  A Hierarchical Recurrent Encoder-Decoder for Generative Context-Aware Query Suggestion.  In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, Melbourne, Australia, 18–23 October 2015; pp. 553–562.
  * Boyanov et al. (2017) Boyanov, M.; Nakov, P.; Moschitti, A.; Da San Martino, G.; Koychev, I.  Building Chatbots from Forum Data: Model Selection Using Question Answering Metrics.  In Proceedings of the International Conference Recent Advances in Natural Language Processing, Varna, Bulgaria, 2–8 September 2017; pp. 121–129.
  * Nakov et al. (2017) Nakov, P.; Hoogeveen, D.; Màrquez, L.; Moschitti, A.; Mubarak, H.; Baldwin, T.; Verspoor, K.  SemEval-2017 Task 3: Community Question Answering.  In Proceedings of the 11th International Workshop on Semantic Evaluation, Vancouver, BC, Canada, 3–4 August 2017; pp. 27–48.
  * Vaswani et al. (2017) Vaswani, A.; Shazeer, N.; Parmar, N.; Uszkoreit, J.; Jones, L.; Gomez, A.N.; Kaiser, L.u.; Polosukhin, I.  Attention is All you Need.  In Proceedings of the 30th Annual Conference on Neural Information Processing Systems, Long Beach, CA, USA, 4–9 December 2017; pp. 5998–6008.
  * Hardalov et al. (2018) Hardalov, M.; Koychev, I.; Nakov, P.  Towards Automated Customer Support.  In Proceedings of the 18th International Conference on Artificial Intelligence: Methodology, Systems, and Applications, Varna, Bulgaria, 12–14 September 2018; pp. 48–59.
  * Li et al. (2018) Li, Y.; Miao, Q.; Geng, J.; Alt, C.; Schwarzenberg, R.; Hennig, L.; Hu, C.; Xu, F.  Question Answering for Technical Customer Support.  In Proceedings of the 7th CCF International Conference, NLPCC 2018, Hohhot, China, 26–30 August 2018; pp. 3–15.
  * Feng et al. (2015) Feng, M.; Xiang, B.; Glass, M.R.; Wang, L.; Zhou, B.  Applying deep learning to answer selection: A study and an open task.  In Proceedings of IEEE Workshop on Automatic Speech Recognition and Understanding, Scottsdale, AZ, USA, 13–17 December 2015; pp. 813–820.
  * LeCun et al. (1989) LeCun, Y.; Boser, B.; Denker, J.S.; Henderson, D.; Howard, R.E.; Hubbard, W.; Jackel, L.D.  Back-propagation Applied to Handwritten Zip Code Recognition.  Neural Comput. 1989, 1, 541–551.
  * Li et al. (2015) Li, X.; Li, L.; Gao, J.; He, X.; Chen, J.; Deng, L.; He, J.  Recurrent reinforcement learning: A hybrid approach.  In Proceedings of the 2016 International Conference on Learning Representations, San Juan, Puerto Rico, 2–4 May 2016.
  * Hochreiter and Schmidhuber (1997) Hochreiter, S.; Schmidhuber, J.  Long Short-Term Memory.  Neural Comput. 1997, 9, 1735–1780.
  * Qiu et al. (2017) Qiu, M.; Li, F.L.; Wang, S.; Gao, X.; Chen, Y.; Zhao, W.; Chen, H.; Huang, J.; Chu, W.  AliMe Chat: A Sequence to Sequence and Rerank based Chatbot Engine.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, BC, Canada, 30 July–4 August 2017; pp. 498–503.
  * Cui et al. (2017) Cui, L.; Huang, S.; Wei, F.; Tan, C.; Duan, C.; Zhou, M.  SuperAgent: A Customer Service Chatbot for E-commerce Websites.  In Proceedings of the Association for Computational Linguistics 2017, System Demonstrations, Vancouver, BC, Cananda, 30 July–4 August 2017; pp. 97–102.
  * Pang et al. (2017) Pang, L.; Lan, Y.; Guo, J.; Xu, J.; Xu, J.; Cheng, X.  DeepRank: A New Deep Architecture for Relevance Ranking in Information Retrieval.  In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management; Singapore, Singapore, 6–10 November 2017; pp. 257–266.
  * Mikolov et al. (2013) Mikolov, T.; Sutskever, I.; Chen, K.; Corrado, G.S.; Dean, J.  Distributed Representations of Words and Phrases and Their Compositionality.  In Proceedings of the 27th Annual Conference on Neural Information Processing Systems, Lake Tahoe, NV, USA, 5–8 December 2013; pp. 3111–3119.
  * Srivastava et al. (2015) Srivastava, R.K.; Greff, K.; Schmidhuber, J.  Highway networks.  arXiv 2015, arXiv:1505.00387.
  * Pennington et al. (2014) Pennington, J.; Socher, R.; Manning, C.  GloVe: Global Vectors for Word Representation.  In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, Doha, Qatar, 25–29 October 2014; pp. 1532–1543.
  * Peters et al. (2018) Peters, M.; Neumann, M.; Iyyer, M.; Gardner, M.; Clark, C.; Lee, K.; Zettlemoyer, L.  Deep Contextualized Word Representations.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies; Association for Computational Linguistics, New Orleans, LA, USA, 1–6 June 2018; pp. 2227–2237.
  * Ba et al. (2016) Ba, J.L.; Kiros, J.R.; Hinton, G.E.  Layer normalization.  arXiv 2016, arXiv:1607.06450.
  * Srivastava et al. (2014) Srivastava, N.; Hinton, G.; Krizhevsky, A.; Sutskever, I.; Salakhutdinov, R.  Dropout: A simple way to prevent neural networks from overfitting.  J. Mach. Learn. Res.  2014, 15, 1929–1958.
  * Serban et al. (2018) Serban, I.V.; Lowe, R.; Henderson, P.; Charlin, L.; Pineau, J.  A Survey of Available Corpora For Building Data-Driven Dialogue Systems: The Journal Version.  Dialog. Discourse 2018, 9, 1–49.
  * Twitter (2019) Customer Support on Twitter. Available online: http://www.kaggle.com/thoughtvector/customer-support-on-twitter (accessed on 18 February 2019).
  * customer-support-chatbot (2018) Codebase of Towards Automated Customer Support. Available online: https://github.com/mhardalov/customer-support-chatbot (accessed on 12 February 2019).
  * Manning et al. (2014) Manning, C.; Surdeanu, M.; Bauer, J.; Finkel, J.; Bethard, S.; McClosky, D.  The Stanford CoreNLP Natural Language Processing Toolkit.  In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, Baltimore, MA, USA, 22–27 June 2014; pp. 55–60.
  * Kingma and Ba (2015) Kingma, D.P.; Ba, J.  Adam: A method for stochastic optimization.  In Proceedings of the 2015 International Conference on Learning Representations, San Diego, CA, USA, 7–9 May 2015.
  * Abadi et al. (2016) Abadi, M.; Barham, P.; Chen, J.; Chen, Z.; Davis, A.; Dean, J.; Devin, M.; Ghemawat, S.; Irving, G.; Isard, M.; others.  Tensorflow: A system for large-scale machine learning.  OSDI 2016, 16, 265–283.
  * ElasticSearch (2019) ElasticSearch. Available online: http://www.elastic.co/products/elasticsearch (accessed on 20 February 2019).
  * Robertson and Zaragoza (2009) Robertson, S.; Zaragoza, H.  The Probabilistic Relevance Framework: BM25 and Beyond.  Found. Trends Inf. Retr. 2009, 3, 333–389.
  * Liu et al. (2016) Liu, C.W.; Lowe, R.; Serban, I.; Noseworthy, M.; Charlin, L.; Pineau, J.  How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation.  In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, TX, USA, 1–5 November 2016; pp. 2122–2132.
  * Lowe et al. (2017) Lowe, R.; Noseworthy, M.; Serban, I.V.; Angelard-Gontier, N.; Bengio, Y.; Pineau, J.  Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, Vancouver, BC, Canada, 30 July–4 August 2017; pp. 1116–1126.
  * Rus and Lintean (2012) Rus, V.; Lintean, M.  A Comparison of Greedy and Optimal Assessment of Natural Language Student Input Using Word-to-Word Similarity Metrics.  In Proceedings of the Seventh Workshop on Building Educational Applications Using NLP, Montreal, QC, Canada, 7 June 2012; pp. 157–162.
  * Forgues et al. (2014) Forgues, G.; Pineau, J.; Larchevêque, J.M.; Tremblay, R.  Bootstrapping dialog systems with word embeddings.  In Proceedings of the NIPS Workshop on Modern Machine Learning and Natural Language Processing, Montreal, QC, Canada, 12 December 2014.
  * Lin and Och (2004) Lin, C.Y.; Och, F.J.  Automatic Evaluation of Machine Translation Quality Using Longest Common Subsequence and Skip-Bigram Statistics.  In Proceedings of the 42nd Annual Conference of the Association for Computational Linguistics, Barcelona, Spain, 21–26 July 2004; pp. 605–612.
  * Sutton and Barto (1998) Sutton, R. S.; Barto, A.G.  Introduction to Reinforcement Learning; MIT Press: Cambridge, MA, USA, 1998.


The Open Journal of Astrophysics submitted XXX; accepted YYY

††thanks: ⋆ Corresponding author<EMAIL_ADDRESS>
# SDSS J125417.98+274004.6: An X-ray Detected Minor Merger Dual AGN

Marko Mićić 1⋆ Brenna N. Wells 1 Olivia J. Holmes 1 Jimmy A. Irwin 1 1
Department of Physics and Astronomy, The University of Alabama, Tuscaloosa, AL
35401, USA

###### Abstract

In this paper, we present the discovery of a dual AGN in a $\sim$14:1 minor
merger between the galaxy SDSS J125417.98+274004.6 and its unnamed dwarf
satellite. We calculated stellar masses of the primary and secondary galaxy to
be 3.8$\times$1010M⊙ and 2.7$\times$109M⊙, respectively. We used archival
Chandra X-ray observations to assess AGN presence. We found that both AGN have
comparable luminosities of $\sim$2$\times$1042 erg s-1, with the secondary AGN
being more likely to be the dominant one. The galaxies are in the early stages
of the merger and are connected by a tidal bridge. Computational works suggest
that the secondary AGN should experience a brief but intensive period of
Eddington-limit approaching accretion during the early stages of the merger.
During the merger, the secondary black hole can increase its mass by a factor
of ten. SDSS J125417.98+274004.6 is the first known dual AGN in an early-stage
minor merger with a comparably or more luminous secondary AGN. As such, it
will be of great value for future studies of merger-triggered accretion and
black hole growth.

###### keywords:

dual AGN – dwarf galaxies – galaxy mergers

##  1 Introduction

Galaxy mergers are major drivers of galaxy evolution. They lead to changes in
morphology (e.g., Toomre & Toomre (1972); Lotz et al. (2008)), color, and
chemical composition (e.g., Lin et al. (2008); Buck et al. (2023)), can affect
star formation (e.g., Ellison et al. (2008); Scudder et al. (2012); Kaviraj
(2013)), increase the stellar mass of the progenitor galaxies, and trigger the
supermassive black hole accretion (e.g., Gao, F. et al. (2020)). Since almost
every galaxy is expected to harbor a supermassive black hole in its nucleus,
galaxy mergers are a logical precursor to dual supermassive black hole
systems. If during the merger both black holes turn active, a dual AGN will be
formed. Dual AGN are of great interest for various astrophysical topics
because they represent unique laboratories for studying the physics at the
intersection of galaxy interactions, star formation, AGN activity, and
gravitational waves. Major merger dual AGN, involving two large galaxies with
comparable masses, have been studied extensively. Koss et al. (2012) found
that the dual AGN are more likely to occur in major mergers and that the X-ray
luminosity of both AGN tends to increase with the decrease of galaxy
separation, suggesting the merger-related accretion mechanism. Similar
conclusions were drawn by Comerford et al. (2015) who found that dual AGN in
major mergers are significantly more luminous than single AGN, indicating that
mergers and higher AGN luminosities and causally connected. On the other hand,
dual AGN in mergers involving one or two dwarf galaxies are usually overlooked
and understudied.
Since every large galaxy is associated with multiple dwarf satellites, minor
mergers, involving galaxies with disproportional masses (mass ratio at least
3:1), are much more common than major mergers. However, dual AGN in minor
mergers are exceptionally rare, with only a handful of known objects (Koss et
al., 2012; Comerford et al., 2015; Secrest et al., 2017; Liu et al., 2018).
Various hydrodynamical simulations of dual AGN in minor mergers have been
carried out (Callegari et al., 2011; Wassenhove et al., 2012; Capelo et al.,
2015). They all found that during minor mergers, the primary AGN is mostly
unaffected, but the secondary galaxy always responds strongly to the
interaction. The secondary AGN grows significantly faster than the primary
AGN, and the growth happens in two main phases. During the early stages of the
merger, the secondary AGN is expected to accrete at higher Eddington ratios.
However, ram pressure striping eventually removes all the gas from the
secondary galaxy and the accretion ceases. The secondary AGN will accrete
again once it enters the disk of the primary galaxy and starts consuming the
gas reservoir. During this process, the secondary black hole can increase its
mass by a factor of ten, but most of the time it is unable to accrete
efficiently due to tidal forces. Furthermore, the activity of the primary and
secondary black holes for most of the merger is not simultaneous, providing an
explanation for the observational scarcity of minor merger dual AGN. However,
the accretion physics and triggering mechanisms of minor merger dual AGN are
not fully understood.
Minor mergers in which the secondary AGN has comparable or higher luminosity
than the primary AGN are rare. Was49 is an example of a late-stage minor
merger, with a mass ratio between 1:7 and 1:15, where the dominant AGN resides
in the smaller galaxy, the only known object of this nature (Secrest et al.,
2017), and as such represents a unique system for studies of interaction-
triggered accretion onto secondary black holes. Some other notable works
important for understanding the activity of AGN in minor mergers include
discoveries of single AGN residing in dwarf galaxies undergoing mergers with
their more massive companion. One such example is NGC 3341 (Bianchi et al.,
2013), a triple merger between a large disk galaxy, and two of its dwarf
satellites, where a single AGN resides in one of the dwarfs. Some other works
discovered intermediate-mass black hole candidates (IMBHs) undergoing tidal
disruption events, residing in dwarf galaxies that are being actively
cannibalized by their larger neighbors (e.g., Lin et al. (2017)).
In this letter, we report the discovery of a new dual AGN detected using
Chandra, Dark Energy Survey (DES), and Canadian-French-Hawaiian Telescope
(CFHT). The dual AGN resides in an apparent early-stage minor merger of the
galaxy SDSS J125417.98+274004.6 (SDSS J12, hereafter) with its smaller,
unnamed companion (SDSS J12b, hereafter). The secondary AGN has a comparable
X-ray luminosity to the primary AGN, making this the second such discovery,
and the first in an early stage of the merger.
In Section 2 we describe the methodology and data reduction; in Section 3 we
present our results; in Section 4 we discuss the implications of our findings.
Throughout the paper, we assume the $\Lambda$CDM cosmology with parameters
H0=69.6 km s-1 Mpc-1, $\Omega_{\Lambda}$=0.71, $\Omega_{m}$=0.29. All errors
are reported at the 1$\sigma$ level unless specified otherwise.

Table 1: Summary of the Chandra ACIS-I observations used in this paper. Obs. ID | t${}_{\textup{exp}}$ | Date | Off-axis angle
---|---|---|---
| (ksec) |  | (arcmin)
15023 | 43.5 | 2014-03-15 | 2.5
15024 | 19.8 | 2014-03-16 | 2.6
16599 | 28.7 | 2014-03-13 | 2.6
16600 | 96.8 | 2014-03-11 | 2.6

##  2 Methodology, Data, and Analysis

###  2.1 Target selection

This discovery is part of a larger effort aimed at detecting dual AGN in
previously unknown galaxy mergers. We first surveyed deep Chandra archival
observations, searching for potential dual X-ray sources, separated by no more
than 5$\arcsec$. Then, the sample of potential dual X-ray sources was further
examined using archival optical observations, mostly by using the CFHT and
DESI Legacy Survey data. We study the galaxies hosting potential X-ray pairs
and with the data in hand, we try to establish the status of the interaction.
The majority of potential dual X-ray pairs reside in galaxies with no
available spectroscopic redshift or poorly constrained photometric redshift,
making the calculations of physical distance between the two galaxies
impossible. We resort to detecting the existence of tidal debris around the
galaxies hosting the potential X-ray sources as an indicator of an ongoing
interaction. A similar approach has been used in literature (e.g., Kado-Fong
et al. (2020)). Interacting galaxies often develop a variety of tidal
features: tails, bridges, and shells. We find that the overwhelming majority
of our potential dual X-ray sources reside in galaxies that are chance-aligned
background/foreground objects and not bona fide pairs, or in galaxies in which
tidal debris is undetectable. However, for a subset of potential pairs, the
host galaxies exhibit clear tidal features and provide conclusive evidence for
the galaxy association. Following the steps described above and utilizing
multiwavelength archival data, we have discovered a dual AGN residing in a
minor merger.

###  2.2 X-ray and optical data

SDSSJ12 was serendipitously observed with Chandra ACIS-I detector on four
occasions in March 2014, with a total exposure time of 188.8 kiloseconds (PI:
Young). For data analysis, we used CIAO 4.12 and the calibration database
CALDB 4.9.3. Observations were reprocessed using the CIAO script
chandra_repro. We merged all observations and created the PSF maps using the
merge_obs script. We calculated that the 90$\%$ time-exposure weighted point-
spread function is $\sim$16\. X-ray spectral fitting was performed using
SHERPA. The source extraction regions were circular apertures with a radius of
16, while the background extraction region was an annulus with inner and outer
radii of 5$\arcsec$ and 8$\arcsec$, respectively. The extraction regions are
shown in Figure 1. Astrometric corrections were performed by running the CIAO
tool wavdetect across the full field of view of the X-ray image in order to
detect X-ray sources. We then matched Chandra sources with the USNO-A2.0
catalog. The CIAO script wcs_match was then used to take in previously
selected X-ray and USNO-A2.0 sources, crossmatch them, and determine
transformation parameters to minimize positional discrepancies. Finally, the
CIAO script wcs_update was used to apply the correction parameters and update
the event and aspect solution files. The summary of Chandra X-ray observations
used in this work is given in Table 1.

##  3 Results

###  3.1 Galaxy properties

Figure 1: Stacked Chandra ACIS-I image. The black circles are source
extraction regions with radii of 16. The dotted white annulus is the
background extraction region.

The optical spectrum of SDSSJ12 is not available, thus, no information on
spectroscopic redshift exists. SDSS Data Release 12 (Alam et al., 2015)
provides the photometric redshift of $z_{\textup{phot}}$=0.453$\pm$0.086. The
DESI Legacy Survey (Duncan, 2022) provides a similar photometric redshift of
$z_{\textup{phot}}$=0.435$\pm$0.047. Having two independent surveys provide
the same (within the uncertainty) well-constrained photometric redshift is a
strong indicator of SDSSJ12’s actual distance. We adopt the DESI
$z_{\textup{phot}}$ value as a working value. At this redshift, 1$\arcsec$
corresponds to $\sim$5.5 kpc.
We estimated the stellar mass of SDSSJ12 by using the K-corrected SDSS g-band
absolute magnitude and g-r color. This approach was adopted from Bell et al.
(2003) and was commonly used in the field of dwarf-related dual AGN (e.g., Liu
et al. (2018); Reines et al. (2014)). The relation between the galaxy’s
stellar mass, absolute magnitudes, and colors was derived from the empirical
relation for stellar mass-to-light ratio and is given as:

$\textup{log}(\frac{M_{*}}{L_{g}})=a_{\lambda}+b_{\lambda}\times color$ (1)

where mass and luminosity are expressed in solar units, aλ=-0.499, bλ=1.519,
color is g-r=0.46 after applying the K-correction, and the Sun’s absolute
g-band magnitude, Mg,⊙=5.12 (Blanton & Roweis, 2007), was used to calculate
the mass-to-light ratio. We calculate the stellar mass of SDSSJ12 to be
3.8$\times$1010 M⊙. This result is estimated to have an uncertainty of 20$\%$
to 50$\%$.
The smaller companion, SDSSJ12b, assuming the same distance, is many times
smaller and fainter than SDSSJ12, and makes this a likely minor merger
candidate. However, due to its faintness, no information on its magnitudes
exists in the literature. We analyzed the DES g- and r-band images with the
aim of deriving tentative magnitudes. We calculated SDSSJ12b’s approximate g-
and r-band magnitudes by converting the linear fluxes to apparent magnitudes
and we found that they are 24.7 and 23.8, respectively. These magnitudes are
then K-corrected, absolute magnitudes are derived by adopting the same
distance, and a stellar mass of 2.7$\times$109 M⊙ is calculated using the
procedure described above. This implies that the merger in question is a 14:1
mass ratio minor merger. Also, the usual upper limit for a dwarf galaxy is
3$\times$109M⊙ (e.g., Mezcua et al. (2018)) suggesting that SDSSJ12b is an
intriguing dwarf-galaxy candidate. However, we note that the stellar mass
derivation for SDSSJ12b is of an approximative nature, associated
uncertainties are large, it should be taken with caution, and is intended to
serve only as an indicator of a minor merger. Follow-up observations will be
required to better constrain SDSSJ12b’s properties. Nevertheless, solely based
on the physical appearance of the two galaxies, their discrepancy in size and
luminosity strongly hints at the minor merger scenario.

Table 2: Summary of X-ray spectral fitting procedure.

IDa | Cstat/d.o.f.b | N${}_{\textup{H}}$c | FXd | LXe
---|---|---|---|---
SDSSJ12 | 16.5/13 | 2.2$\pm$1.6 | 2.8${}^{+0.4}_{\\--0.6}$ | 2.0${}^{+0.2}_{\\--0.5}$
SDSSJ12b | 10.6/14 |  $<$1.4 | 3.0${}^{+0.3}_{\\--0.7}$ | 2.1${}^{+0.3}_{\\--0.5}$

(a) is the name of the galaxy; (b) is the Cstatistics of the best fit and the
number of degrees of freedom; (c) is the intrinsic absorption in the units of
1022 cm-2; (d) is the unabsorbed 0.5-10 keV flux in the units of 10-15 erg s-1
cm-2; (e) is the unabsorbed 0.5-10 keV luminosity in the units of 1042 erg
s-1. The uncertainties are estimated at the 68$\%$ confidence level. The
spectral index $\Gamma$ was kept fixed at 1.9.

###  3.2 X-ray spectral fitting

SDSSJ12 and SDSSJ12b were detected with Chandra with 26$\pm$5.5 and 33$\pm$6
net counts, respectively. We used Sherpa to perform the X-ray spectral
fitting. The CIAO tool specextract was used to prepare the auxiliary response
file and the redistribution matrix file for each observation. Then, the
combine_spectra tool was used to create summed spectral files. Due to the low-
count nature of both of our sources, we used Cstat statistics (Cash, 1979) to
perform our analysis since it is usually adopted in the literature for low-
count spectral fitting. We specified a model composed of two components: a
photoelectric absorption model and a power-law emission model,
xszphabs*powlaw1d. The fit is performed by fixing the photon index to a
typical AGN value of $\Gamma$=1.9, and the intrinsic absorption
N${}_{\textup{H}}$ was left free to vary. Then, the fit is repeated leaving
both parameters free to vary, and we check if the fit is significantly
improved, (i.e., if Cstatold-Cstatnew ¿2.71, where Cstatold and Cstatnew are
the Cstat values for the fit without and with $\Gamma$ free to vary,
respectively). This procedure is adopted from Mezcua et al. (2018) and
references therein. Both sources showed better statistics for fits performed
with fixed $\Gamma$ value. The resulting unabsorbed 0.3-10 keV luminosities
for SDSSJ12 and SDSSJ12b are 2.0${}^{+0.2}_{\\--0.5}$$\times$1042 erg s-1 and
2.1${}^{+0.3}_{\\--0.5}$$\times$1042 erg s-1, respectively. The summary of the
spectral fitting results is given in Table 2.

##  4 Discussion

Figure 2: a) Subaru Hyper-Suprime Cam r-band image of SDSSJ12 and SDSSJ12b; b)
DESI Legacy Survey g-band; c) DESI Legacy Survey i-band; d) DESI Legacy Survey
r-band; e) DESI Legacy Survey color image (g, r, i, z-bands); f) Chandra HiPS
image (red-soft band photons, green-medium band photons, blue-hard band
photons). The length scale is shown in panel a) and is the same in all other
panels.

###  4.1 The stage of the merger

Based on this analysis, the two AGN have comparable X-ray luminosities, with
the secondary AGN being more likely to be more luminous. The projected
separation between the two AGN is 3.2$\arcsec$ or $\sim$17 kpc. The available
archival optical imaging suggests that the two galaxies are in the early
stages of a merger. The Subaru Suprime Hyper Cam shows two separate nuclei and
a faint diffuse emission between them. The DESI Legacy Survey reveals the link
between the galaxies more conclusively; images in all available bands indicate
the existence of a physical bridge between SDSJ12 and SDSSJ12b. Archival CFHT
imaging further solidifies the early-merging hypothesis since it also reveals
two separate nuclei and a faint bridge between the two. Various optical images
are shown in Figure 2. Figure 3 shows the adaptively smoothed CFHT image with
superposed locations of X-ray sources with positional uncertainties. This
physical link between the two galaxies is likely a tidal bridge made out of
the material stripped from the smaller galaxy by the more massive companion.
The galaxies are likely in the early stages of the merger and similar tidal
bridges have been observed in many similar early-stage galaxy interactions
(e.g., Kimbro et al. (2021); Mićić et al. (2023)).

###  4.2 AGN activity

The tentative stellar mass analysis suggests a 14:1 mass ratio minor merger.
As described above, hydrodynamical simulations claim that the secondary black
hole is not expected to accrete efficiently during the majority of the minor
merger, but it is expected to undergo two phases of growth, one in the early
and the other in the late stages of the merger. During the early stage of
growth, the secondary black hole is expected to accrete at high Eddington
rates, and this would explain the existence of a comparably or more luminous
AGN in a dwarf satellite galaxy, powered by a many times smaller black hole.
However, this phase is expected to end abruptly once the gas is tidally
stripped from the dwarf galaxy.
We calculate the secondary black hole mass, assuming it accretes at the
Eddington limit. The Eddington luminosity, in the units of erg s-1 is given
as:

$L_{\textup{Edd}}=1.38\times 10^{38}\frac{M_{\textup{BH}}}{M_{\odot}}.$ (2)

Assuming the standard bolometric correction of
L${}_{\textup{bol}}$/L${}_{\textup{X}}$=10, and equating the Eddington
luminosity with the bolometric luminosity we find the black hole mass of
M${}_{\textup{BH}}$=150,000 M⊙.
The purported high accretion rates of SDSSJ12b should be reflected in its
X-ray spectral properties. Brightman et al. (2013, 2016) found a statistically
significant relation between the accretion rates and the X-ray spectral index
$\Gamma$. Higher accretion rates will produce a softer X-ray spectrum with a
higher value of $\Gamma$ dominated by the low-energy X-ray photons. On the
other hand, a lower accretion rate will typically result in lower $\Gamma$
values and a harder X-ray spectrum dominated by high-energy photons. They
concluded that the accretion rate is what drives the conditions of the X-ray
corona responsible for X-ray emission and that there is no relation between
$\Gamma$ and black hole mass. However, our X-ray sources have low count rates
and the spectral fitting was performed using the fixed value of $\Gamma$=1.9,
for both sources. The hardness ratio given as $\frac{H-S}{H+S}$, where H and S
are net counts in the hard and soft band, can be used as a proxy for the X-ray
spectral index $\Gamma$. For that purpose, we filtered the Chandra images by
energies in soft (0.3-1 keV) and hard (1-6 eV) bands. After examining the
energy-dependent count distribution we note that SDSSJ12 shows no emission
under 1 keV, with a 1-$\sigma$ upper limit of $<$1.8 counts (Gehrels, 1986).
On the other hand, $\sim$20$\%$ of SDSSJ12b photons have energies under 1 keV,
with 6$\pm$2.5 counts. The resulting hardness ratios for SDSSJ12 and SDSSJ12b
are HR$>$0.9 and HR=0.6$\pm$0.3, respectively. This hints that SDSSJ12b does
indeed have a softer X-ray spectrum, higher spectral index $\Gamma$, and
higher accretion rates. However, we note that the hardness ratio difference
could also be caused by the higher absorption around the primary AGN SDSSJ12.
Even though not conclusively, this tentatively corroborates the scenario of a
dwarf-related early-stage minor merger, with an AGN powered by a low-mass
black hole, accreting at high Eddington rates, until the activity abruptly
shuts down due to tidal stripping.

###  4.3 Comparison to Was49

SDSSJ12 is the second-ever minor merger with a comparably/more luminous AGN
residing in a secondary nucleus after Was49. However, unlike SDSSJ12, Was49 is
a late-stage merger and the secondary nucleus has already circularized within
the primary galaxy disk. Was49 was first identified by Wasilewski (1983) as a
galaxy with two knots of emission. Bothun et al. (1989) and Moran et al.
(1992) performed follow-up spectroscopic studies and found that the primary
galaxy, Was49a is a Seyfert 2 galaxy, and the secondary galaxy Was49b a
Seyfert 1.8 galaxy. They also found that the secondary Was49b has an unusually
high bolometric luminosity of L${}_{\textup{bol}}>$1045 erg s-1, significantly
stronger than that of the primary Was49a. Secrest et al. (2017) studied the
Was49 system using X-rays and confirmed that the secondary nucleus bolometric
luminosity exceeds 1045 erg s-1. However, the primary nucleus is detected with
only seven X-ray counts, all with energies under 2 keV, suggesting strong
contamination by non-AGN photons. With the data they had in hand, they were
unable to further examine the nature of the primary AGN in Was49a. Finally,
they found that the secondary AGN is powered by an overmassive black hole,
M${}_{\textup{BH}}$=1.9$\times$108 M⊙, accounting for 2.3% of the stellar mass
of the Was49b nucleus. It was suggested that this unusually high ratio is due
to a significant loss of stellar material due to interaction with Was49a. On
the other hand, the SDSSJ12 system is an early-stage merger, with two separate
nuclei, and an unambiguous presence of dual AGN. Even though the accretion
rates of the secondary AGN during an early-stage merger are expected to
approach Eddington limits, SDSSJ12b is $\sim$two orders of magnitude fainter
than Was49b. This suggests, unlike the overmassive black hole case of Was49b,
a scenario of a low-mass black hole powering the secondary AGN.

##  5 Conclusions

In this paper, we presented the evidence for an ongoing merger between the
galaxy SDSSJ12 and its smaller companion SDSSJ12b at $z$=0.435. The available
archival imaging reveals a physical link in the form of a tidal bridge between
the two galaxies, suggesting they are in the early stages of a merger. We
analyzed Chandra X-ray data and showed that two luminous point X-ray sources
reside in each nucleus. The primary and secondary AGN have comparable X-ray
luminosities, with the secondary being more likely to be more luminous, making
this the second discovery ever of this kind, and the first in an early-stage
merger. Further analysis suggests that the tentative mass ratio is 14:1 and
that the secondary AGN is powered by a low-mass black hole undergoing high
accretion rates. This is in accordance with computational predictions,
suggesting that the secondary AGN in minor mergers does undergo episodes of
high accretion during the early stages of the merger until it abruptly shuts
down due to the tidal stripping of its gas. The follow-up multiwavelength
observations will shed more light on this object, allowing us to study the,
mostly inaccessible, accretion physics of the secondary AGN in minor mergers.

Figure 3: Adaptivelly smoothed CFHT g-band image. Green crosses indicate the
positions of X-ray sources with coordinates provided by the CIAO tool
wavdetect. The dashed red circles indicate 0$\farcs$24 count contribution
weighted positional uncertainties, at the 95% confidence level (Kim et al.,
2007).

#### Acknowledgements

MM is supported by the University of Alabama Graduate Council Fellowship. BNW
and OJH are supported by the University of Alabama Department of Physics and
Astronomy Undergraduate Barr Scholarship. Authors acknowledge the use of the
Cosmology Calculator (Wright, 2006) and the K-correction calculator
(Chilingarian et al., 2010).

## References

  * Alam et al. (2015) Alam S., et al., 2015, ApJS, 219, 12
  * Bell et al. (2003) Bell E. F., McIntosh D. H., Katz N., Weinberg M. D., 2003, ApJS, 149, 289
  * Bianchi et al. (2013) Bianchi S., Piconcelli E., Pérez-Torres M. A., Fiore F., La Franca F., Mathur S., Matt G., 2013, Monthly Notices of the Royal Astronomical Society, 435, 2335
  * Blanton & Roweis (2007) Blanton M. R., Roweis S., 2007, AJ, 133, 734
  * Bothun et al. (1989) Bothun G. D., Halpern J. P., Lonsdale C. J., Impey C., Schmitz M., 1989, ApJS, 70, 271
  * Brightman et al. (2013) Brightman M., et al., 2013, MNRAS, 433, 2485
  * Brightman et al. (2016) Brightman M., et al., 2016, The Astrophysical Journal, 826, 93
  * Buck et al. (2023) Buck T., Obreja A., Ratcliffe B., Lu Y., Minchev I., Macciò A. V., 2023, MNRAS, 523, 1565
  * Callegari et al. (2011) Callegari S., Kazantzidis S., Mayer L., Colpi M., Bellovary J. M., Quinn T., Wadsley J., 2011, The Astrophysical Journal, 729, 85
  * Capelo et al. (2015) Capelo P. R., Volonteri M., Dotti M., Bellovary J. M., Mayer L., Governato F., 2015, Monthly Notices of the Royal Astronomical Society, 447, 2123
  * Cash (1979) Cash W., 1979, ApJ, 228, 939
  * Chilingarian et al. (2010) Chilingarian I. V., Melchior A.-L., Zolotukhin I. Y., 2010, MNRAS, 405, 1409
  * Comerford et al. (2015) Comerford J. M., Pooley D., Barrows R. S., Greene J. E., Zakamska N. L., Madejski G. M., Cooper M. C., 2015, The Astrophysical Journal, 806, 219
  * Duncan (2022) Duncan K. J., 2022, MNRAS, 512, 3662
  * Ellison et al. (2008) Ellison S. L., Patton D. R., Simard L., McConnachie A. W., 2008, AJ, 135, 1877
  * Gao, F. et al. (2020) Gao, F. et al., 2020, A&A, 637, A94
  * Gehrels (1986) Gehrels N., 1986, ApJ, 303, 336
  * Kado-Fong et al. (2020) Kado-Fong E., Greene J. E., Greco J. P., Beaton R., Goulding A. D., Johnson S. D., Komiyama Y., 2020, AJ, 159, 103
  * Kaviraj (2013) Kaviraj S., 2013, MNRASL, 437, L41
  * Kim et al. (2007) Kim M., et al., 2007, The Astrophysical Journal Supplement Series, 169, 401
  * Kimbro et al. (2021) Kimbro E., Reines A. E., Molina M., Deller A. T., Stern D., 2021, The Astrophysical Journal, 912, 89
  * Koss et al. (2012) Koss M., Mushotzky R., Treister E., Veilleux S., Vasudevan R., Trippe M., 2012, The Astrophysical Journal Letters, 746, L22
  * Lin et al. (2008) Lin L., et al., 2008, ApJ, 681, 232
  * Lin et al. (2017) Lin D., et al., 2017, Nature Astronomy, 1, 0033
  * Liu et al. (2018) Liu X., Guo H., Shen Y., Greene J. E., Strauss M. A., 2018, The Astrophysical Journal, 862, 29
  * Lotz et al. (2008) Lotz J. M., et al., 2008, ApJ, 672, 177
  * Mezcua et al. (2018) Mezcua M., Civano F., Marchesi S., Suh H., Fabbiano G., Volonteri M., 2018, MNRAS, 478, 2576
  * Mićić et al. (2023) Mićić M., Holmes O. J., Wells B. N., Irwin J. A., 2023, The Astrophysical Journal, 944, 160
  * Moran et al. (1992) Moran E. C., Halpern J. P., Bothun G. D., Becker R. H., 1992, AJ, 104, 990
  * Reines et al. (2014) Reines A. E., Plotkin R. M., Russell T. D., Mezcua M., Condon J. J., Sivakoff G. R., Johnson K. E., 2014, The Astrophysical Journal Letters, 787, L30
  * Scudder et al. (2012) Scudder J. M., Ellison S. L., Torrey P., Patton D. R., Mendel J. T., 2012, MNRAS, 426, 549
  * Secrest et al. (2017) Secrest N. J., Schmitt H. R., Blecha L., Rothberg B., Fischer J., 2017, The Astrophysical Journal, 836, 183
  * Toomre & Toomre (1972) Toomre A., Toomre J., 1972, ApJ, 178, 623
  * Wasilewski (1983) Wasilewski A. J., 1983, ApJ, 272, 68
  * Wassenhove et al. (2012) Wassenhove S. V., Volonteri M., Mayer L., Dotti M., Bellovary J., Callegari S., 2012, The Astrophysical Journal Letters, 748, L7
  * Wright (2006) Wright E. L., 2006, PASP, 118, 1711