#  SE-PQA: Personalized Community Question Answering

Pranav Kasela  0000-0003-0972-2424 University of Milano-BicoccaMilanItaly
ISTI-CNR, Pisa, ItalyPisaItaly<EMAIL_ADDRESS>,  Marco Braga
0009-0004-7619-8399 University of Milano-BicoccaMilanItaly Politecnico di
Torino, DAUINTurinItaly<EMAIL_ADDRESS>,  Gabriella Pasi
0000-0002-6080-8170 University of Milano-BicoccaMilanItaly
<EMAIL_ADDRESS>and  Raffaele Perego  0000-0001-7189-4724 ISTI-CNR,
Pisa, ItalyPisaItaly<EMAIL_ADDRESS>
###### Abstract.

Personalization in Information Retrieval is a topic studied for a long time.
Nevertheless, there is still a lack of high-quality, real-world datasets to
conduct large-scale experiments and evaluate models for personalized search.
This paper contributes to filling this gap by introducing SE-PQA
(StackExchange - Personalized Question Answering), a new curated resource to
design and evaluate personalized models related to the task of community
Question Answering (cQA). The contributed dataset includes more than 1 million
queries and 2 million answers, annotated with a rich set of features modeling
the social interactions among the users of a popular cQA platform. We describe
the characteristics of SE-PQA and detail the features associated with
questions and answers. We also provide reproducible baseline methods for the
cQA task based on the resource, including deep learning models and
personalization approaches. The results of the preliminary experiments
conducted show the appropriateness of SE-PQA to train effective cQA models;
they also show that personalization remarkably improves the effectiveness of
all the methods tested. Furthermore, we show the benefits in terms of
robustness and generalization of combining data from multiple communities for
personalization purposes.

##  1\. Introduction

Personalization is a problem studied for a long time in Information Retrieval
(IR) (Borisov et al., 2016; Calegari and Pasi, 2013; Speretta and Gauch, 2005;
Huang et al., 2021; Ma et al., 2020; Braga et al., 2023) and Natural Language
Processing (NLP) (Braga et al., 2023). Personalized search aims to tailor the
search outcome to a specific user (or group of users) based on the knowledge
of her/his interests and online behaviour. Given the ability of Deep Neural
Network (DNN) models to face many different tasks by extracting relevant
features from both texts and structured sources (Mitra and Craswell, 2018),
there is the expectation of a huge potential also for their application in
Personalized IR (PIR) and Recommender Systems (RS). However, the lack of
publicly-available, large-scale datasets that include user-related information
is one of the biggest obstacles to the training and evaluation of DNN-based
personalized models. Some real-world datasets are commonly used in the
literature to design and assess personalization models. These datasets include
the AOL query log (Pass et al., 2006), the Yandex query
log111https://www.kaggle.com/c/yandex-personalized-web-search-challenge, and
the CIKM Cup 2016
dataset222https://competitions.codalab.org/competitions/11161. Moreover, even
synthetically enriched datasets have been used such as: PERSON (Tabrizi et
al., 2018), the Amazon product search dataset (Ai et al., 2017), and a dataset
based on the Microsoft Academic Knowledge Graph (Bassani et al., 2022).
However, all of them have some issues. For example, ethical and privacy issues
are related to using the AOL query log (Barbaro et al., 2006). In contrast,
the anonymization performed on the Yandex query log prevents its use for
training or fine-tuning natural language models.

This paper aims to fill this gap by contributing SE-PQA (StackExchange -
Personalized Question Answering), a large dataset rich in user-level features
that can be exploited for training and evaluating personalized models
addressing the community Question Answering (cQA) task. SE-PQA is based on
StackExchange333https://stackexchange.com, a popular cQA platform with a
network of $178$ open forums. A dump of the StackExchange user-contributed
content is publicly available444https://archive.org/details/stackexchange
according to a cc-by-sa 4.0
license555https://archive.org/details/stackexchange according to a cc-by-sa
4.0 license666There are although some disputes concerning a unilateral change
of licensing policy by StackExchange:
https://meta.stackexchange.com/questions/333089/stack-exchange-and-stack-
overflow-have-moved-to-cc-by-sa-4-0. With great care, we have preprocessed the
original dump by building SE-PQA, a curated dataset with about one million
questions and two million associated answers annotated with a rich set of
features modeling the social interactions of the user community. The features
include, for example, the positive or negative votes received by a question or
an answer, the number of views, the number of users that selected a given
question as a favorite one, the tags from a controlled folksonomy describing
the topic dealt with, the comments that other users might have written under a
question or an answer. To favor the design and evaluation of personalized
models, the users in SE-PQA are associated with their past questions and
answers, their social autobiography, their reputation score, and the number of
views received by their profile. The cQA task can be addressed on SE-PQA with
different methodologies exploiting either the textual description of questions
and answers, the folksonomy, the features modeling the social interactions, or
a combination of the above information sources. In this paper, we focus on IR
approaches to cQA. Thus, we adapt the cQA task to an ad-hoc retrieval task
where the question is seen as a query, and the answers are retrieved from the
pool of past answers indexed for the purpose. In this particular setting, the
system aims to retrieve a (small) ranked set of documents that contain the
correct answers to the user question. There can be multiple correct answers
given a question, so, in this case, personalization can be used to understand
the user’s context and background and rank higher the answers that are more
relevant to the specific user.
In summary, the novel contribution of this paper is the following:

  * •

We contribute the SE-PQA dataset, a novel public resource consisting of a
comprehensive corpus including more than one million questions and two million
answers by about $600k$ users. The richness and variety of features provided
with the dataset enable its use for the design and evaluation of both
classical and personalized cQA.

  * •

We provide a detailed analysis of the resource made available in SE-PQA
compared to those previously available and used in the research community.

  * •

We report a preliminary comparison of the performance of different methods for
cQA applied to the questions, answers, and users in SE-PQA. The results show
that models based on deep learning outperform in effectiveness traditional
retrieval models, and that by exploiting personalization features we can
obtain a significant performance boost.

The paper is organized as follows. Section 2 introduces the SE-PQA dataset and
reports some statistics about its content. Furthermore, the section details
the personalized cQA tasks addressed in this paper by using SE-PQA. Moreover,
it provides a comparison of SE-PQA with respect to other publicly-available
resources in the field. Section 3 presents a preliminary comparison of
traditional and personalized models for cQA applied to SE-PQA. In Section 4 we
discuss the utility and the practical implications of the new resource.
Finally, Section 5 concludes the work and draws some future lines of
investigation.

##  2\. The SE-PQA Dataset

The textual posts in StackExchange forums are associated with rich social
metadata information. When users ask a question to the community, they assign
some tags specifying the topic to make the question searchable and visible to
the users interested in it. The questions are up-voted or down-voted by the
community based on their interest and adherence to the community
guidelines777https://meta.stackexchange.com/help/how-to-answer. In many cases,
the community suggests to the question author how to improve the question if
it is poorly expressed or formatted. Similar treatment is given to the
answers, which can be up-voted or down-voted by the community; moreover, the
user who asked the question can also choose the answer he/she deems the best,
which may differ from the one that received the most up-votes from others. We
note, however, that 87.6% questions and answers are assigned a score given by
the difference between the number of up and down-votes. A positive score thus
indicates that the post has more up-votes than down-votes, while a negative
score indicates that more users down-voted it.

Figure 1. Illustration of StackExchange data.

UI example of the StackExchange website. The figure highlights the position of
some of the most important features available, for example, post score,
comment location, date of publication, username.

StackExchange is quite well known in the IR community: for example, it has
been used for training a language model for sentence similarity (HuggingFace,
2021). To the best of our knowledge, the usage of StackExchange for Q&A tasks
has been, however, limited just to selecting similar sentence training pairs
without exploiting user-level/social features for personalized information
retrieval tasks. Another study uses StackExchange for duplicate question
retrieval (Hoogeveen et al., 2015). The dataset built for this task consists
of 12 separate communities, and the authors do not address the de-duplication
task across community boundaries.

With SE-PQA we overcome the previous limitations and provide a complete,
curated dataset of textual questions and answers belonging to different,
heterogeneous forums. In SE-PQA a user can belong to multiple communities; if
we take into consideration users that wrote at least 2 questions, about 50% of
them have asked questions in multiple communities. As we increase the minimum
number of questions, also the percentage of users using multiple communities
increases. For instance, if we take only the users that wrote at least 5
documents (either questions or answers) and consider both the questions and
the answers written by the user, we note that of the resulting 62k users only
23k (37%) wrote either a question or an answer in only a single community,
while 40k (63%) wrote documents in at least two different communities, 26k
(42%) in at least three and 18k (28%) in more than three communities.

We claim that personalization is particularly useful for multi-domain
collections, where we can exploit information about users’ interests in
multiple topics of different domains; when the data is instead derived from a
single domain or a specific topic of a domain, personalization may become less
important. We provide evidence of this assertion in Section 3.2 where we
report about our experiments applying the same personalization approach to
both the complete SE-PQA dataset and to the data sampled from separate
communities: the results show that personalization on the multi-domain dataset
yields better improvements than on only single communities.
To increase diversity, in SE-PQA we thus combine data from multiple networks
that can be categorized under the large umbrella of humanistic communities.
These communities focus on different to pics, but the language used is not too
diverse among them. In particular, we choose the following 50 communities:

writers, workplace, woodworking, vegetarianism, travel, sustainability,
sports, sound, skeptics, scifi, rpg, politics, philosophy, pets, parenting,
outdoors, opensource, musicfans, music, movies, money, martialarts,
literature, linguistics, lifehacks, law, judaism, islam, interpersonal, hsm,
history, hinduism, hermeneutics, health, genealogy, gardening, gaming,
freelancing, fitness, expatriates, english, diy, cooking, christianity,
buddhism, boardgames, bicycles, apple, anime, academia.

The training, validation, and test split are done temporally to avoid any kind
of data leakage. The training set includes all questions written from
2008-09-10 to 2019-12-31 (included), the validation set is formed by questions
asked between 2019-12-31 and 2020-12-31 (included), while the test set
contains the questions from 2019-12-31 till 2022-09-25 (included).

There are a total of $1,125,407$ questions in the dataset, $1,001,706$ of
which have at least one answer ($89\%$ of all questions) and $525,030$ of
which have a response that the questioner has selected as the best one ($47\%$
of all questions). We are left with 822 974 training questions, 78 854
validation questions, and 99 878 test questions after the temporal splits.
There are $2,173,139$ answers and $588,688$ users. Many users in the
communities register themselves just for asking a question and then never use
their accounts again. In fact, the dataset has a median of 1 user-generated
document (either a question or an answer), with about 80% of users having no
more than 2 documents. The text in the dataset is preprocessed by removing
HTML tags present in the original documents. In Table 1 we report the basic
statistics for the dataset. Specifically: document length, measured in the
number of words, document score, which is the difference between the number of
up- and down-votes assigned by the community; answers’ count, the number of
answers given to a question; comments’ count, the number of user comments to a
given question or answer; favorite count, that indicates the number of users
that flagged the question as their favorite, showing their interest in that
topic; tags count, the number of tags associated to the question by the asking
user. From Table 1 and Figure 2 we can notice that, as expected, most
documents are short, with answers generally longer than questions. The dataset
is available at Zenodo888https://doi.org/10.5281/zenodo.10679181. The code to
reproduce the dataset and the baseline is publicly
available999https://github.com/pkasela/SE-PQA.

Table 1. Basic feature statistics for  question and answers.

| Type | mean | std | median | 25% | 75% | 99%
---|---|---|---|---|---|---|---
Questions | Length | 125.69 | 112.90 | 94 | 60 | 153 | 553
Score | 5.13 | 10.73 | 2 | 1 | 6 | 45
#docs = | Answers Count | 1.93 | 1.92 | 1 | 1 | 2 | 9
1,125,407 | Comments Count | 2.78 | 3.37 | 2 | 0 | 4 | 15
| Favorite Count | 2.07 | 4.88 | 1 | 1 | 2 | 16
| Tags Count | 2.45 | 1.21 | 2 | 1 | 3 | 5
Answers | Length | 178.15 | 210.09 | 117 | 61 | 218 | 1000
#docs = | Score | 5.13 | 12.43 | 2 | 1 | 5 | 51
2,173,139 | Comments Count | 1.62 | 2.62 | 1 | 0 | 2 | 12

Figure 2. Word length distribution for questions and answers in SE-PQA.

The plot shows the distribution of questions and answer posts, in particular,
how they are skewed to the left, which indicates that there are many posts
with few words while in some cases the posts can be longer.

###  2.1. Task Definition

Even though SE-PQA can be used for many IR tasks (e.g., duplicate and related
question retrieval or expert finding), we address here the cQA task only, by
illustrating how it can be addressed by using the resources in SE-PQA. The
addressed cQA task focuses on satisfying the information needs expressed in
user questions by retrieving relevant documents from a collection of
historical answers posted by the community members. We infer the relevance of
an answer to a question from the number of up-votes given by community
members. Concerning the experiments involving personalized cQA models, we only
consider relevant the single answer that is explicitly labeled as the best
answer by the user who submitted the question. More formally we provide the
following definition. Let $\mathcal{A}$ be a set of answers
$\\{a_{1},\ldots,a_{n}\\}$ posted by the members of the community and let
$\mathbf{q}$ be a question asked by the user $\mathbf{u}$. The objective of
cQA is to retrieve a ranked list of $k$ answers $\\{a_{q,1},\ldots,a_{q,k}\\}$
from $\mathcal{A}$ based on their relevance to $\mathbf{q}$. In our
experiments with SE-PQA we infer the relevance of an answer to a question from
the up-votes given by community members. With regards to the experiments
involving personalized cQA models, we consider instead only the answer
$a_{q,u}\in\mathcal{A}$ that is the most relevant for the question
$\mathbf{q}$ and for the specific user $\mathbf{u}$. This can be assessed on
SE-PQA by considering the single answer that is explicitly labelled as the
best answer by the user who submitted the question.

In order to address the above-defined cQA task, we preprocess the collection
of answers of SE-PQA. Specifically, we discard answers with negative scores
since they are assumed to be of low quality and not relevant to the cQA task.
This cleaning step affects about 100k answers. As a result, $2,073,370$
answers are left in the dataset. Moreover, we discard all the questions that
have not received an answer.

To create the set of relevant answers for the questions, i.e., the golden
standard, we consider the answers given to each question.
A total of 525,030 questions out of 1,001,706 have an answer selected as the
best by the user who asked the question. We are sure that this answer received
a positive score from the community since we have removed all the answers with
negative scores. Thus, the answer given to a question $\mathbf{q}$ of user
$\mathbf{u}$ and selected as best can be considered as both relevant to the
question $\mathbf{q}$ (positive score from the community) and to the user
$\mathbf{u}$ (selection of the best answer).
By using this information, we define two versions of this dataset: the base
version, where we consider as relevant for a question all the answers having a
positive score, and the personalized (pers) version, which, instead, considers
relevant for both the user and the question only the single answer that the
user selected as the best answer. We note that both versions of the dataset,
base and pers, can be used for personalized cQA, with the following
difference: in the pers version each query is potentially personalizable,
while the base version also includes queries that cannot be always used to
train personalized models since the choices of the answers preferred by the
users are not always available.

A variety of user-generated information from the training set can be used in
the personalization phase. For each question, we include all the user posts
(questions and answers of the user asking the question) that were written
prior to the question being asked. This is done to avoid any data leakage for
query-wise training, but the user data is not limited to these documents; in
fact, one can also consider the social interaction between users, the tags
assigned by the users to the previous questions asked along with their
meaning, the badges earned by users. Furthermore, the dataset includes the
biographic text (about me) self-introducing each user, a rich set of numeric
features (e.g., user reputation score, number of up-votes and down-votes of
each post, number of views), a set of temporal information (e.g., user
creation date, last access date, post creation timestamp).

###  2.2. Comparison with available datasets

Table 2. Comparison between SE-PQA and other text-based datasets for
personalized IR. All the datasets are in English.

Dataset | Documents | Train Queries | Val Queries | Test Queries | Users | Avg. user docs | Avg. # relevant | Rel. Assessment
---|---|---|---|---|---|---|---|---
AOLIA(MacAvaney et al., 2022) | 1 291 695 | 212 386 | 31 064 | 36 052 | 30 166 | 136.62 $\pm$ 134.17 | 1.15 $\pm$ 0.46 | inferred from click
PERSON(Tabrizi et al., 2018) | 616 889 | - | - | - | 558 898 | - | 5.5 $\pm$ 5.3 | inferred from citation
MAG Computer Science(Bassani et al., 2022) | 4 809 684 | 552 798 | 5 583 | 6 497 | 5 260 279 | 61.94 $\pm$ 60.32 | 3.25 $\pm$ 3.27 | inferred from citations
MAG Physics | 4 926 753 | 728 171 | 7 355 | 6 366 | 5 835 016 | 60.98 $\pm$ 56.54 | 4.17 $\pm$ 4.15 | inferred from citations
MAG Political Science | 4 814 084 | 162 597 | 1 642 | 5 715 | 6 347 092 | 40.64 $\pm$ 29.32 | 3.88 $\pm$ 5.17 | inferred from citations
MAG Psychology | 4 215 384 | 544 882 | 5 503 | 12 625 | 4 825 578 | 61.66 $\pm$ 62.72 | 4.73 $\pm$ 4.4 | inferred from citations
Amazon Electorincs(Ai et al., 2017) | 1 689 188 | 904 | - | 85 | 192 403 | 8.78 $\pm$ 8.26 | 1.12 $\pm$ 0.48 | synthetic
Amazon Kindle Store | 982 618 | 3 313 | - | 1 290 | 68 223 | 35.65 $\pm$ 37.48 | 1.87 $\pm$ 3.3 | synthetic
Amazon CDs | 1 097 591 | 534 | - | 160 | 75 258 | 21.75 $\pm$ 16.53 | 2.57 $\pm$ 6.59 | synthetic
Amazon Cell Phones | 194 439 | 134 | - | 31 | 27 879 | 4.95 $\pm$ 2.6 | 1.52 $\pm$ 1.13 | synthetic
SE-PQA [Base] | 2 073 370 | 822 974 | 78 854 | 99 878 | 588 688 | 34.71 $\pm$ 107.33 | 2.07 $\pm$ 1.81 | inferred from answer scores
SE-PQA [Pers] | 2 073 370 | 224 366 | 18 086 | 19 811 | 17 963 | 75.67 $\pm$ 142.41 | 1 | manually selected by user

We survey other works contributing datasets for personalized IR and discuss
their limitations. These datasets cover a huge variety of tasks ranging from
web search to product search and academic search.

In Table 2 we summarize the basic statistics of the main datasets used in the
literature for personalized IR tasks.
The AOL query log was released in 2006 and even after the harsh criticism it
received due to privacy-related issues, it remains to this date a widely
employed resource for a variety of tasks, especially personalized ad-hoc
retrieval. This dataset includes about 20 million Web queries issued by more
than 657,000 users over three months (from 03/01/2006 to 05/31/2006). For each
query, the dataset details provide the userid, the URL, and the rank of the
web page clicked, if any. A huge limitation of this dataset is that the web
pages in the corpus are represented by their URLs and the text is not
provided. To cope with this issue, researchers use a version of the corpus
collected in 2017 (Ahmad et al., 2019) scraping the text content of the web
pages in the corpus. This additional dataset comes however with another
problem: the content of web pages can change over time, and many documents
might have changed from 2006 to 2017, thus making the dataset less reliable.
Recently, a new version of the dataset was proposed, which used Internet
Archive to retrieve documents as they were in 2006 (MacAvaney et al., 2022).
The AOLIA dataset (MacAvaney et al., 2022) is a derivative of the original
dataset (AOL Query Log (Pass et al., 2006)); it has been cleaned to generate a
higher-quality query set. First, queries with no clicks (and consequently no
relevant documents) are removed. Then, all the queries with domain references
(.com, .org, etc.) and queries pointing to adult or illegal websites are
eliminated. Furthermore, all queries with fewer than three characters are
discarded as well. Finally, queries from users with less than twenty
associated queries are removed in order to have enough user-related data to
perform personalization. As a result, the AOLIA dataset contains about 1,3M
documents and around 30k different users.
To tackle the issues that come with the real words datasets, some synthetic
datasets, and associated evaluation frameworks have been proposed in recent
years: PERSON (Tabrizi et al., 2018), Amazon product search (Ai et al., 2017)
and MAG, a dataset based on the Microsoft Academic Knowledge Graph (Bassani et
al., 2022).
Tabrizi et al. (Tabrizi et al., 2018) proposed PERSON as a synthetic
personalized evaluation framework for IR based on citation networks. The
authors base the dataset on the ArnetMiner citation network (Tang et al.,
2008). The idea is that, from the authors’ perspective, the papers referenced
in a document are somehow related to the document it is cited in. From an IR
point of view, the document content (title or abstract) are considered as the
user query, while the cited documents are assumed relevant to the query.
The dataset based on the Microsoft Academic Graph (MAG) (Bassani et al., 2022)
follows a procedure similar to the one used by Tabrizi et. al; it uses a much
larger citation graph to derive four different datasets, one for each of the
following subjects: Political Science, Psychology, Computer Science and
Physics. Similarly to the PERSON framework, queries come from paper titles,
and the previous papers of a user are used to build her personal profile.
Paper titles from users authoring less than twenty previous papers are removed
from the dataset, providing the user-related information necessary to define
appropriate user models. As explained in Tabrizi et al. (2018), such a dataset
can be employed to develop and compare various user models, but cannot be used
to assess personalized search effectiveness due to the strong assumptions made
to determine relevant documents in this framework.

The Amazon product search dataset (Ai et al., 2017) is based on the Amazon
Review dataset (He and McAuley, 2016). The dataset is created in a very
synthetic manner using item categories and properties to generate user
queries: the terms from each product’s category are concatenated following
their hierarchy order to create a topic string. Stopwords and duplicate terms
are removed from the topic string that is then used as a query for the
associated item. When removing the duplicated words, the terms from a lower
category are preserved, e.g., $Camera\to Photo\to Digital\ Camera\ Lenses$ is
converted to “photo digital camera lenses”. Given an item $i$ purchased by a
user $u$, the item is considered relevant for $u$, and the synthetic query is
generated as explained above. This process comes with some drawbacks: it
generates a low number of unique queries that do not resemble real-world
queries as rarely a user writes down all the categories of a product in a
hierarchical order to search for it.

Table 2 shows that the proposed dataset is, in terms of corpus volume, very
similar to the other datasets, and it has a comparable value also for the
other statistics. Actually, it is the largest one in terms of number of
queries provided. In terms of relevance assessment, it is the only one that
has been explicitly annotated by the users: by a single user for the best
answer and by various community users for relevance, by either up-voting it if
the answer was relevant according to them or down-voting it otherwise.

##  3\. Preliminary experiments with SE-PQA

In this section, we briefly describe the experimental setup and introduce the
methods employed to showcase SE-PQA on the cQA task defined in Section 2.1.
Finally, we report and discuss the results of the preliminary experiments
conducted.

###  3.1. Experimental settings

We adopt a two-stage ranking architecture aimed at trading-off effectiveness
and efficiency by applying two increasingly accurate and computationally
expensive ranking models. The first stage is inexpensive and recall-oriented.
It aims at selecting for each query a set of candidate documents that are
eventually re-ranked by the second, precision-oriented ranker. The first stage
is based on elasticsearch, and uses BM25 as a fast ranker. To increase the
recall in the set of candidate documents retrieved by the first stage, we
optimize BM25 parameters by performing a grid search driven by Recall at 100
on a subset of 5000 queries randomly sampled from the validation set. The
optimal values for b and k1 found are 1 and 1.75, respectively. For the
second, precision-oriented stage, we rely on a linear combination of the
scores computed by BM25, a neural re-ranker based on a pre-trained language
model, and, when used, a personalization model exploiting user history,
represented by the tags used by the users. In all the experiments the second
stage re-ranks the top 100 results retrieved with BM25.

#### Neural models

We use the following three neural models in the second stage:

  * •

The first model is MiniLM101010https://huggingface.co/sentence-
transformers/all-MiniLM-L6-v2. This model was trained and tuned using billions
of training pairs, given the presence of StackExchange pairs in the training
data, we use the model as it is, without any fine-tuning;

  * •

The second one is DistilBERT111111https://huggingface.co/distilbert-base-
uncased. In this case we fine-tune the model using all the training queries of
SE-PQA. For each query, one positive document and two negative documents are
randomly sampled, one from the list retrieved by BM25, and one in-batch random
negative (Henderson et al., 2017). We fine-tune DistilBERT for 10 epochs, with
a batch size of 16 and a learning rate of $10^{-6}$ by using Triplet Margin
Loss (Reimers and Gurevych, 2019), with a margin $\gamma=0.5$.

  * •

The third one is MonoT5 (Nogueira et al., 2020). It is based on a T5 (Raffel
et al., 2020) re-ranker, which is fine-tuned on the MS MARCO passage
dataset121212https://huggingface.co/castorini/monot5-small-
msmarco-10k,131313https://huggingface.co/castorini/monot5-base-msmarco-10k. We
train two different versions of MonoT5: small and base. To further fine-tune
MonoT5-small, we follow the same setting proposed in (Nogueira et al., 2020),
i.e. batch size equal to $128$ and learning rate of $10^{-3}$. For each query,
we sample one positive document and one negative document from the list
retrieved by BM25. To fine-tune MonoT5-base, we reduce the batch size to $64$
due to hardware constraints. Also in this case the models are trained for a
total of 10 epochs. Instead of fine-tuning the whole model, we rely on Adapter
modules (Poth et al., 2023; Pfeiffer et al., 2021), composed of two Feed-
Forward layers: the first one is a down projection of the input vector into an
intermediate dimension, which is followed by a non-linear activation function;
the second one is an up projection to the dimension of the input vector.
Following Karimi et al. (Karimi Mahabadi et al., 2021), the intermediate
dimension is set to 48.

For DistilBERT and the two T5 models, we rely on AdamW (Loshchilov and Hutter,
2019) as the optimizer. For reproducibility purposes, we set the random seed
to 42 for training DistilBERT and 0 for training the T5 models.

#### Personalized TAG model for cQA

For a given answer a produced in response to a query q formulated by a user u,
a personalization score is computed as explained below. As previously
explained this score is linearly combined with the BM25 score and with the
score produced by the neural re-ranker. Given a question q, asked by user u at
time t, let $T_{u,t}$ be the set of tags assigned by u to all her/his
questions posted before t (including q). $T_{u,t}$ thus represents the
interests of u as expressed in her/his previous interactions. The authors of
the answers to query q do not have the possibility of tagging explicitly their
answers, so for each answer, we consider the tags associated with the answered
questions. Specifically, given an answer a from a user u’, we represent a with
the set $T_{u^{\prime},t}$, i.e. the set of all the tags associated to the
questions to which $u^{\prime}$ answered before t (excluding q). It is worth
noting that in computing $T_{u^{\prime},t}$ we do not consider the tags
associated with the current question q to avoid data leakage. The TAG model
assigns to each answer a, which has been retrieved for question q in the first
stage, the following score:

$s_{a}=\frac{|T_{u^{\prime},t}\cap T_{u,t}|}{|T_{u,t}|+1},$

where we add $1$ in the denominator as a smoothing factor, needed for cases
where set $T_{u,t}$ is empty. The rationale behind the proposed formula is
that an answer is assigned a higher score if the question author shares
similar interests (represented by means of tags s/he assigned to her/his asked
questions) to the answerer.

#### Score combination

The final ranking is obtained by computing the weighted sum of the normalized
scores from the above single models, i.e., by using the weights
$\lambda_{BM25},\lambda_{Neural},$ and $\lambda_{TAG}$, for the BM25, Neural
(MiniLM, DistilBERT or T5), and TAG models, respectively, with
$\sum_{i}\lambda_{i}=1$. The $\lambda$ values are optimized on the validation
set by performing a grid search in the interval $[0,1]$ with step $0.1$.

#### Evaluation Metrics

We use P@1, NDCG@3, NDCG@10, Recall@100, and MAP@100 as our evaluation
metrics. The cutoffs considered are low as it is important to find the
relevant results at the top of the ranked lists. All the metrics are computed
by using the ranx library (Bassani, 2022; Bassani and Romelli, 2022). We make
our runs publicly available, so different metrics can be computed if
necessary.

###  3.2. Experimental Results

Table 3. Results for the cQA task on Base SE-PQA.

Model | P@1* | NDCG@3* | NDCG@10* | R@100 | MAP@100* | $\lambda$
---|---|---|---|---|---|---
BM25 | 0.330* | 0.325* | 0.359* | 0.615 | 0.320* | -
BM25 + TAG | 0.355* | 0.349* | 0.383* | 0.615 | 0.342 | (.7;.3)
BM25 + DistilBERT | 0.404* | 0.400* | 0.435* | 0.615 | 0.389* | (.3;.7)
BM25 + DistilBERT + TAG | 0.422* | 0.415* | 0.448* | 0.615 | 0.402* | (.3;.5;.2)
BM25 + MiniLM | 0.473* | 0.459* | 0.486* | 0.615 | 0.443* | (.1;.9)
BM25 + MiniLM + TAG | 0.493* | 0.475* | 0.500* | 0.615 | 0.457* | (.1;.8;.1)
BM25 + T5-small + TAG | 0.448* | 0.442* | 0.471* | 0.615 | 0.426* | (.1;.9;.0)
BM25 + T5-base + TAG | 0.497* | 0.491* | 0.514* | 0.615 | 0.470* | (.1;.9;.0)

Table 4. Results for the cQA task on Pers SE-PQA.

Model | P@1* | NDCG@3* | NDCG@10* | R@100 | MAP@100* | $\lambda$
---|---|---|---|---|---|---
BM25 | 0.279* | 0.353* | 0.394* | 0.707 | 0.362* | -
BM25 + TAG | 0.306* | 0.383* | 0.425* | 0.707 | 0.392* | (.7;.3)
BM25 + DistilBERT | 0.351* | 0.437* | 0.478* | 0.707 | 0.441* | (.3;.7)
BM25 + DistilBERT + TAG | 0.375* | 0.460* | 0.500* | 0.707 | 0.463* | (.3;.5;.2)
BM25 + MiniLM | 0.403* | 0.491* | 0.525* | 0.707 | 0.490* | (.1;.9)
BM25 + MiniLM + TAG | 0.426* | 0.512* | 0.543* | 0.707 | 0.509* | (.1;.8;.1)
BM25 + T5-small | 0.376* | 0.469* | 0.506* | 0.707 | 0.468* | (.1;.9)
BM25 + T5-small + TAG | 0.400* | 0.491* | 0.525* | 0.707 | 0.489* | (.1;.8;.1)
BM25 + T5-base | 0.417* | 0.517* | 0.548* | 0.707 | 0.510* | (.1;.9)
BM25 + T5-base + TAG |  0.440* |  0.535* |  0.563* | 0.707 |  0.528* | (.1;.8;.1)

The results of the experiments conducted are reported in Table 4 and 4 for the
base and pers versions of the dataset, respectively. In all tables, the symbol
* indicates a statistically significant improvement over the respective non-
personalized method not using any contribution from the TAG model. Statistical
significance is assessed with a Bonferroni-corrected two-sided paired
student’s t-test with 99% confidence. Moreover, in the column labeled
$\lambda$ we report the optimized weights used for combining the scores
computed by BM25, DistilBERT/MiniLM, and TAG models. From the two tables, we
notice first that neural re-rankers are effective and that the methods using
MiniLM outperform those based on DistilBERT and MonoT5-small. This was somehow
expected due to the huge training set used for training MiniLM (HuggingFace,
2021). After fine-tuning, MonoT5-base, with more parameters, is able to
achieve better results than MiniLM. Moreover, MonoT5-small and MonoT5-base,
fine-tuned for just 10 epochs on the proposed dataset, improve by 33% and 47%
in MAP@100, respectively, the BM25 performance. However, the most notable
result is that TAG improves, by a statistically significant margin, any cQA
method it is combined with and for all the metrics considered on the pers
version of the dataset (Table 4), where non-personalizable queries are
removed, thus showing the advantages of personalization. The improvement due
to the addition of this simple personalized model reaches up to 8% in terms of
MAP@100 compared to their non-personalized baseline.

Table 5. Results for the cQA task on single-community data extracted from Base
SE-PQA.

Community | Model (BM25 +) | P@1 | NDCG@3 | NDCG@10 | R@100* | MAP@100 | $\lambda$
---|---|---|---|---|---|---|---
Academia | MiniLM | 0.438* | 0.382* | 0.395* | 0.489* | 0.344* | (.1,.9)
MiniLM + TAG |  0.453* |  0.392* |  0.403* | 0.489* |  0.352* | (.1,.8,.1)
Apple | MiniLM | 0.327* | 0.351* | 0.381* | 0.514* | 0.349* | (.1,.9)
MiniLM + TAG |  0.335* |  0.361* |  0.389* | 0.514* |  0.357* | (.1,.8,.1)
Bicycles | MiniLM | 0.405* | 0.380* | 0.421* | 0.600* | 0.365* | (.1,.9)
MiniLM + TAG |  0.436* |  0.405* |  0.441* | 0.600* |  0.386* | (.1,.8,.1)
Christianity | MiniLM | 0.534* | 0.505* | 0.555* | 0.783* | 0.497* | (.2,.8)
MiniLM + TAG |  0.549* |  0.521* |  0.564* | 0.783* |  0.507* | (.1,.8,.1)
Cooking | MiniLM | 0.600* | 0.567* | 0.600* | 0.719* | 0.553* | (.1,.9)
MiniLM + TAG |  0.619* |  0.583* |  0.614* | 0.719* |  0.568* | (.1,.8,.1)
DIY | MiniLM | 0.323* | 0.313* | 0.346* | 0.501* | 0.302* | (.1,.9)
MiniLM + TAG |  0.335* |  0.324* |  0.356* | 0.501* |  0.312* | (.1,.8,.1)
Hermeneutics | MiniLM | 0.589* | 0.538* | 0.593* | 0.828* | 0.526* | (.2,.8)
MiniLM + TAG |  0.632* |  0.570* |  0.617* | 0.828* |  0.552* | (.1,.8,.1)
Law | MiniLM | 0.663* | 0.647* | 0.678* | 0.803* | 0.639* | (.2,.8)
MiniLM + TAG |  0.677* |  0.657* |  0.687* | 0.803* |  0.649* | (.1,.8,.1)
Money | MiniLM | 0.545* | 0.535* | 0.563* | 0.706* | 0.515* | (.2,.8)
MiniLM + TAG |  0.559* |  0.542* |  0.571* | 0.706* |  0.523* | (.1,.8,.1)
Music | MiniLM | 0.508* | 0.447* | 0.476* | 0.602* | 0.418* | (.2,.8)
MiniLM + TAG |  0.522* |  0.460* |  0.486* | 0.602* |  0.427* | (.1,.8,.1)
Rpg | MiniLM | 0.657* | 0.646* | 0.685* | 0.849* | 0.640* | (.2,.8)
MiniLM + TAG |  0.677* |  0.660* |  0.695* | 0.849* |  0.651* | (.1,.8,.1)
Scifi | MiniLM | 0.532* | 0.563* | 0.596* | 0.745* | 0.559* | (.2,.8)
MiniLM + TAG |  0.549* |  0.574* |  0.606* | 0.745* |  0.569* | (.1,.8,.1)
$\lambda_{TAG}=0$ |  english, health, history, travel, workplace, writers, woodworking, vegetarianism, skeptics, politics, philosophy, parenting, outdoors, musicfans, literature, linguistics, judaism, interpersonal, hsm, genealogy, freelancing, fitness, expatriates, buddhism, anime
No Statistical Improvement |  boardgames, gardening, gaming, hinduism, islam, lifehacks, martialarts, movies, opensource, pets, sound, sports, sustainability

Finally, in order to validate our hypothesis that personalization is more
useful on a multi-domain, heterogeneous collection than on a single-domain,
homogeneous one, we perform a series of experiments considering single-domain
data extracted from SE-PQA. Specifically, we consider 50 partitions of SE-PQA
(only base version) built by isolating the data from the 50 communities. We
apply to each one of these subsets the non-personalized and personalized
combinations of models using the best performing bi-encoder model MiniLM, and
measure the performance according to the same metrics used for the previous
cQA tests. We did not employ the T5 models due to their long computation time.
For a fair comparison, we performed for each community the optimization of the
$\lambda$ weights on single-domain validation data. Differently from the
multi-domain results shown in Table 4, we notice that the contribution of the
TAG model is lower, and in some cases missing. Specifically, for 25 out of 50
communities, personalization does not lead to any improvement, i.e.,
$\lambda_{TAG}=0$. On the other 13 communities, we do not observe
statistically significant improvements for P@1 over the non-personalized
methods. Since statistical significance is affected also by the size of the
sample, we computed also the performance metrics averaged on all the runs with
single-domain data. As expected, the absolute metrics are slightly higher for
single-domain tests due to the higher recall in the first-stage retrieval. In
fact, by considering single-community data at a time, we drastically reduce
the size of the collection indexed, allowing the first-stage ranker to perform
better. However, in terms of the absolute performance boost due to the TAG
model, we achieve a 2% improvement on P@1 when using all communities together,
while the boost decreases to 1.1% when considering the communities separately.
The results of these experiments are reported integrally in the SE-PQA Zenodo
and Github page. Here, in Table 5, we report the results for the 12
communities for which personalization achieves statistically significant
improvements.

##  4\. Utility and predicted impact

The SE-PQA resource we make available to the research community is a step
ahead toward a fair and robust evaluation of personalization approaches in
Information Retrieval. The features provided with the dataset include explicit
signals to create relevance judgments and a large amount of historical user-
level information allowing to design and test classical and novel
personalization methods. We expect the SE-PQA dataset to be useful for many
researchers and practitioners working in personalized IR and in the
application of machine/deep learning techniques for personalization. In recent
years, the IR community spent important effort in studying personalization.
However, a comprehensive dataset for evaluating and comparing different
approaches is still missing. Researchers mainly rely on synthetic datasets or
use non-public data, which makes the comparison between different methods less
reliable or, worse, not possible at all. The SE-PQA dataset advances this
research area by filling this gap with a large-scale dataset covering the
activity of StackExchange users in a period of 14 years. For this reason, we
expect that the dataset will impact the research community working on
personalized IR as it provides a single common ground of evaluation built on
questions & answers from real users socially interacting via a community-
oriented web platform.

##  5\. Conclusion and Future Works

This paper discussed the characteristics of SE-PQA (StackExchange -
Personalized Question Answering), a large real-world dataset including about 1
million questions and 2 million associated answers contributed by the users of
StackExchange communities. The data comes with a rich set of user-level
features modeling the interactions among the members of the online
communities, e.g., the positive or negative votes received by questions and
answers, the tags associated with questions, the comments that other users
might have written under a question or an answer, the users’ autobiographies,
reputation score, and the number of views received by their profile.

We detailed all the information available in the dataset and discussed how it
can be exploited for training and evaluating classical and personalized models
addressing cQA task. As exemplifying methodologies, we focused on IR
approaches for these tasks based on a two-stage architecture where the second
re-ranking stage exploits a combination of the scores computed by BM25,
DistilBERT/MiniLM/T5, and TAG models. The results of the preliminary
experiments conducted show that personalization works effectively on this
dataset, improving by a statistically significant margin, in most of the
cases, state-of-the-art methods based on pre-trained large language models.

The analysis conducted and the peculiarities of the SE-PQA resource suggest
several lines of future investigation. For example, in this work we employed a
relatively simple user model for personalization, we leave the development of
more complex personalized models for future works that could exploit user
features of SE-PQA that were not used in the proposed models.

## References

  * (1)
  * Ahmad et al. (2019) Wasi Uddin Ahmad, Kai-Wei Chang, and Hongning Wang. 2019\.  Context Attentive Document Ranking and Query Suggestion. In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Paris, France) _(SIGIR’19)_. Association for Computing Machinery, New York, NY, USA, 385–394.  https://doi.org/10.1145/3331184.3331246
  * Ai et al. (2017) Qingyao Ai, Yongfeng Zhang, Keping Bi, Xu Chen, and W. Bruce Croft. 2017.  Learning a Hierarchical Embedding Model for Personalized Product Search. In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Shinjuku, Tokyo, Japan) _(SIGIR ’17)_. Association for Computing Machinery, New York, NY, USA, 645–654.  https://doi.org/10.1145/3077136.3080813
  * Barbaro et al. (2006) Michael Barbaro, Tom Zeller, and Saul Hansell. 2006\.  A face is exposed for AOL searcher no. 4417749.  _New York Times_ 9, 2008 (2006), 8.
  * Bassani (2022) Elias Bassani. 2022\.  ranx: A Blazing-Fast Python Library for Ranking Evaluation and Comparison. In _ECIR (2)_ _(Lecture Notes in Computer Science, Vol. 13186)_. Springer, Cham, 259–264.
  * Bassani et al. (2022) Elias Bassani, Pranav Kasela, Alessandro Raganato, and Gabriella Pasi. 2022.  A Multi-Domain Benchmark for Personalized Search Evaluation. In _Proceedings of the 31st ACM International Conference on Information & Knowledge Management_ (Atlanta, GA, USA) _(CIKM ’22)_. Association for Computing Machinery, New York, NY, USA, 3822–3827.  https://doi.org/10.1145/3511808.3557536
  * Bassani and Romelli (2022) Elias Bassani and Luca Romelli. 2022.  Ranx.Fuse: A Python Library for Metasearch. In _Proceedings of the 31st ACM International Conference on Information & Knowledge Management_ (Atlanta, GA, USA) _(CIKM ’22)_. Association for Computing Machinery, New York, NY, USA, 4808–4812.  https://doi.org/10.1145/3511808.3557207
  * Borisov et al. (2016) Alexey Borisov, Ilya Markov, Maarten de Rijke, and Pavel Serdyukov. 2016.  A Context-Aware Time Model for Web Search. In _Proceedings of the 39th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Pisa, Italy) _(SIGIR ’16)_. Association for Computing Machinery, New York, NY, USA, 205–214.  https://doi.org/10.1145/2911451.2911504
  * Braga et al. (2023) Marco Braga, Alessandro Raganato, Gabriella Pasi, et al. 2023\.  Personalization in bert with adapter modules and topic modelling. In _Proceedings of the 13th Italian Information Retrieval Workshop (IIR 2023). Pisa, Italy_. 24–29.
  * Calegari and Pasi (2013) Silvia Calegari and Gabriella Pasi. 2013.  Personal ontologies: Generation of user profiles based on the YAGO ontology.  _Information Processing & Management_ 49, 3 (2013), 640–658.  https://doi.org/10.1016/j.ipm.2012.07.010 Personalization and Recommendation in Information Access.
  * He and McAuley (2016) Ruining He and Julian McAuley. 2016.  Ups and Downs: Modeling the Visual Evolution of Fashion Trends with One-Class Collaborative Filtering. In _Proceedings of the 25th International Conference on World Wide Web_ (Montréal, Québec, Canada) _(WWW ’16)_. International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, CHE, 507–517.  https://doi.org/10.1145/2872427.2883037
  * Henderson et al. (2017) Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-hsuan Sung, Laszlo Lukacs, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017.  Efficient Natural Language Response Suggestion for Smart Reply.  https://doi.org/10.48550/ARXIV.1705.00652
  * Hoogeveen et al. (2015) Doris Hoogeveen, Karin M. Verspoor, and Timothy Baldwin. 2015\.  CQADupStack: A Benchmark Data Set for Community Question-Answering Research. In _Proceedings of the 20th Australasian Document Computing Symposium (ADCS)_ (Parramatta, NSW, Australia) _(ADCS ’15)_. ACM, New York, NY, USA, Article 3, 8 pages.  https://doi.org/10.1145/2838931.2838934
  * Huang et al. (2021) Minghui Huang, Wei Peng, and Dong Wang. 2021.  TPRM: A Topic-based Personalized Ranking Model for Web Search.  https://doi.org/10.48550/ARXIV.2108.06014
  * HuggingFace (2021) HuggingFace. 2021\.  _Train a Sentence Embedding Model with 1B Training Pairs_.  HuggingFace.  https://huggingface.co/blog/1b-sentence-embeddings
  * Karimi Mahabadi et al. (2021) Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. 2021\.  Compacter: Efficient low-rank hypercomplex adapter layers.  _Advances in Neural Information Processing Systems_ 34 (2021), 1022–1035.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled Weight Decay Regularization. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.  https://openreview.net/forum?id=Bkg6RiCqY7
  * Ma et al. (2020) Zhengyi Ma, Zhicheng Dou, Guanyue Bian, and Ji-Rong Wen. 2020\.  PSTIE: Time Information Enhanced Personalized Search. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_ (Virtual Event, Ireland) _(CIKM ’20)_. Association for Computing Machinery, New York, NY, USA, 1075–1084.  https://doi.org/10.1145/3340531.3411877
  * MacAvaney et al. (2022) Sean MacAvaney, Craig Macdonald, and Iadh Ounis. 2022\.  Reproducing Personalised Session Search Over the AOL Query Log. In _Advances in Information Retrieval: 44th European Conference on IR Research, ECIR 2022, Stavanger, Norway, April 10–14, 2022, Proceedings, Part I_ (Stavanger, Norway). Springer-Verlag, Berlin, Heidelberg, 627–640.  https://doi.org/10.1007/978-3-030-99736-6_42
  * Mitra and Craswell (2018) Bhaskar Mitra and Nick Craswell. 2018.  An Introduction to Neural Information Retrieval.  _Foundations and Trends® in Information Retrieval_ 13, 1 (2018), 1–126.  https://doi.org/10.1561/1500000061
  * Nogueira et al. (2020) Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020\.  Document Ranking with a Pretrained Sequence-to-Sequence Model. In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, Online, 708–718.  https://doi.org/10.18653/v1/2020.findings-emnlp.63
  * Pass et al. (2006) Greg Pass, Abdur Chowdhury, and Cayley Torgeson. 2006\.  A Picture of Search. In _Proceedings of the 1st International Conference on Scalable Information Systems_ (Hong Kong) _(InfoScale ’06)_. Association for Computing Machinery, New York, NY, USA, 1–es.  https://doi.org/10.1145/1146847.1146848
  * Pfeiffer et al. (2021) Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2021\.  AdapterFusion: Non-Destructive Task Composition for Transfer Learning. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (Eds.). Association for Computational Linguistics, Online, 487–503.  https://doi.org/10.18653/v1/2021.eacl-main.39
  * Poth et al. (2023) Clifton Poth, Hannah Sterz, Indraneil Paul, Sukannya Purkayastha, Leon Engländer, Timo Imhof, Ivan Vulić, Sebastian Ruder, Iryna Gurevych, and Jonas Pfeiffer. 2023\.  Adapters: A Unified Library for Parameter-Efficient and Modular Transfer Learning. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_. Association for Computational Linguistics, Singapore, 149–160.  https://aclanthology.org/2023.emnlp-demo.13
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ 21, 1, Article 140 (jan 2020), 67 pages.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.  https://doi.org/10.48550/ARXIV.1908.10084
  * Speretta and Gauch (2005) Mirco Speretta and Susan Gauch. 2005.  Personalized Search Based on User Search Histories, In The 2005 IEEE/WIC/ACM International Conference on Web Intelligence (WI’05).  _Proceedings - 2005 IEEE/WIC/ACM InternationalConference on Web Intelligence, WI 2005_ 2005, 622– 628.  https://doi.org/10.1109/WI.2005.114
  * Tabrizi et al. (2018) Shayan A. Tabrizi, Azadeh Shakery, Hamed Zamani, and Mohammad Ali Tavallaei. 2018.  PERSON: Personalized information retrieval evaluation based on citation networks.  _Information Processing & Management_ 54, 4 (2018), 630–656.  https://doi.org/10.1016/j.ipm.2018.04.004
  * Tang et al. (2008) Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. 2008.  ArnetMiner: Extraction and Mining of Academic Social Networks. In _Proceedings of the 14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ (Las Vegas, Nevada, USA) _(KDD ’08)_. Association for Computing Machinery, New York, NY, USA, 990–998.  https://doi.org/10.1145/1401890.1402008