# Enhancing Travel Decision-Making: A Contrastive Learning Approach for
Personalized Review Rankings in Accommodations

Reda Igebaria<EMAIL_ADDRESS>Booking.com, Tel AvivIsrael ,
Eran Fainman<EMAIL_ADDRESS>Booking.com, Tel AvivIsrael ,  Sarai
Mizrachi<EMAIL_ADDRESS>Booking.com, Tel AvivIsrael ,  Moran
Beladev<EMAIL_ADDRESS>Booking.com, Tel AvivIsrael and  Fengjun
Wang<EMAIL_ADDRESS>Booking.com, AmsterdamNetherlands

(Date: January 2024)

###### Abstract.

User-generated reviews significantly influence consumer decisions,
particularly in the travel domain when selecting accommodations. This paper
contribution comprising two main elements. Firstly, we present a novel dataset
of authentic guest reviews sourced from a prominent online travel platform,
totaling over two million reviews from 50,000 distinct accommodations.
Secondly, we propose an innovative approach for personalized review ranking.
Our method employs contrastive learning to intricately capture the
relationship between a review and the contextual information of its respective
reviewer. Through a comprehensive experimental study, we demonstrate that our
approach surpasses several baselines across all reported metrics. Augmented by
a comparative analysis, we showcase the efficacy of our method in elevating
personalized review ranking. The implications of our research extend beyond
the travel domain, with potential applications in other sectors where
personalized review ranking is paramount, such as online e-commerce platforms.

##  1\. Introduction

Large online travel platforms allow millions of users to book stays worldwide.
The influence of user-generated reviews has become pivotal in the decision-
making process for those seeking to book accommodations (Gretzel and Yoo,
2008; Baka, 2016; Tuominen, 2011; Ricci and Wietsma, 2006). The growing volume
of reviews underscores the imperative for an effective review ranking
mechanism. The one-size-fits-all approach of traditional review ranking models
poses a significant challenge in catering to the diverse preferences and
priorities of individual users. Current algorithms often prioritize reviews
based on helpfulness votes (Tsamis et al., 2021; Ghose and Ipeirotis, 2007;
Korfiatis et al., 2012; Yang et al., 2015; Ghose and Ipeirotis, 2011),
introducing biases and neglecting the nuanced perspectives of users with
distinct needs (Yue et al., 2010; Agarwal et al., 2018; Wang et al., 2016).
One common issue is the sparsity of helpfulness votes, with most reviews
lacking any such votes (Kuan et al., 2015). In our dataset, this problem is
compounded by the fact that only 8.7% of reviews receive helpful votes, making
the signal notably sparse. Furthermore, these votes are anonymous which makes
it not feasible to use them for developing personalized review ranking models.

In response to these limitations, we present a novel approach for personalized
reviews ranking. Our approach acknowledges that different users have different
characteristics (like families, solo travelers, couples etc), different trip
types (e.g., beach, city, nature, etc) and therefore prioritize different
accommodation features (like accessibility, sustainability, inclusivity etc).
We consider these user and accommodation characteristics as context. We
leverage contrastive learning (Bengio et al., 2013) to capture the intricate
relationship between users’ context and their reviews. This approach relies on
that user-generated reviews reflect their personal experience with a focus on
what they most liked and disliked about their stay, taking into account their
personal preferences and the key elements that were crucial to their stay
(Baka, 2016).

Publishing a dataset of user-generated reviews holds paramount importance in
the travel domain due to its pivotal role in facilitating informed decision-
making for users who book accommodations. Unlike e-commerce, where review
datasets are relatively abundant, the availability of comprehensive
accommodation review datasets remains scarce. This scarcity poses a
significant challenge for researchers, developers, and industry stakeholders
seeking to enhance tourism products and services. Events like the Rectour
workshop, held annually within the RecSys conference, underscore the growing
recognition of the critical role that tourism review datasets play in
advancing recommendation systems tailored specifically to the complexities of
the tourism industry. Thus, while e-commerce benefits from an abundance of
review data, the tourism sector faces a pressing need for the availability of
comprehensive datasets to drive innovation and enhance tourism experiences.

As a part of this work, we publish a dataset that consists of over two million
reviews from 50,000 unique accommodations. It contains contextual details
about the guest who wrote the review (e.g., number of booked nights and guest
type), the accommodation (e.g., accommodation type and average review score),
and the review itself (including its textual fields, number of helpful votes
and the overall rating score). The process of curating the dataset leveraged
the Text2topic model (Fengjun Wang, Moran Beladev et al., 2023) to select
informative reviews, i.e., reviews that provide insights about topics related
to the stay (such as cleanliness, value for money, breakfast etc).

In the following sections, we describe the details of our dataset curation and
our modeling approach, present the findings of our experiments, and showcase
the effectiveness of our methodology throughout a comparative analysis.

This paper contributions are summarized as follows:

  * •

We present a new large-scale, real-world dataset of informative reviews
sourced from a major online travel platform. This dataset is publicly
accessible via GitHub111https://github.com/bookingcom/ml-dataset-reviews.

  * •

We introduce a novel formulation for the personalized ranking task, which is
designed to rank reviews based on their relevance to the reviewer’s context.

  * •

We propose a contrastive learning approach to tackle the task, utilizing a
unique in-accommodation batch sampling method.

  * •

Experimental results, conducted across various settings, consistently
demonstrate that our method significantly outperforms ranking based on helpful
votes. This is evident in terms of Mean Reciprocal Rank (MRR), precision@1,
and precision@10 metrics.

  * •

We provide interpretability and explainability of our model outputs by
performing a comparative analysis highlighting common topics between reviews
ranked by our model and reviews written by the users with the given context.

##  2\. Related Work

In this section, we delve into the existing literature related to review
ranking. We begin by providing an overview of both non-contextual and
contextual review ranking methodologies, highlighting their key features and
differences. Subsequently, we present an outline of the currently available
datasets in this field, comparing and contrasting them with our dataset.
Lastly, we examine contrastive learning, a technique that is leveraged by our
work.

###  2.1. Non-contextual Review Ranking

Early attempts on review helpfulness prediction were focused on predicting
helpfulness using simple ML models such as SVM and linear regression (Ghose
and Ipeirotis, 2007). These were done with hand-crafted features such as
review length, readability (Korfiatis et al., 2012), sentiment (Yang et al.,
2015) and subjectivity of the review (Ghose and Ipeirotis, 2011). Nayeem and
Rafiei (2023) used reviewer and temporal information in the decision making.
Specifically, they leveraged mean votes of the user’s past reviews and applied
a time decay over the review age. The motivation for time decay is that old
reviews might become irrelevant over time, e.g., a hotel might address
cleaning issues by improving inspection following negative reviews. Du et al.
(2019) empirically analyzed features that have been used in 149 previous
papers. They found that semantic features (i.e., TF-IDF vector and pre-trained
word embeddings) are the most predictive of helpfulness compared to other
features related to sentiment, readbility, structure and syntax. Bilal and
Almazroi (2023) showed that fine-tuned BERT model outperforms bag-of-words
approaches in identifying helpful reviews. Fan et al. (2019) implemented a Bi-
LSTM based model that predicts helpfulness given the review and product
description. Finally, Han et al. (2022); Ren et al. (2024) used a multi-modal
approach and incorporated images from the reviews into the model.

The studies cited above established ground truth labels based on the number of
helpful votes. They approached the task of predicting helpfulness as a
supervised binary classification problem, setting a threshold on the number of
votes to categorize reviews as helpful or not. Notably, these studies did not
incorporate user features or personalized mechanisms into their ranking
methodologies, which presents a gap we aim to address in our model. Moreover,
these methods exhibit a presentation bias, as users tend to vote on reviews
they read, influenced by the original ranking (Yue et al., 2010). Our approach
mitigates these biases by explicitly modeling the relationship between the
review content and the corresponding reviewer’s context, thereby removing the
dependency on helpful votes.

###  2.2. Context-aware Review Ranking

Another line of work addresses the task of personalized reviews ranking. This
task differs from the task discussed in the previous section in two key
aspects: (1) the user’s context is taken into account, and (2) the ground
truth is user subjective, i.e., the same review might be helpful to one user
but not to another. Moghaddam et al. (2011) employed traditional
recommendation methods like matrix factorization to suggest reviews based on a
latent representation of the user. Wang et al. (2013) integrated social
relations between users into the ranking, by constructing a social graph and
integrating the connections as features for a linear regression model.
Furthermore, Peddireddy (2020) utilized a weighted term-frequency vector
representing past user interactions, e.g. product purchases and reviews, and
used Okapi BM25 (Robertson et al., 1995) to rank the reviews. Lastly, Huang et
al. (2020) adopted a graph-based methodology to determine similarity between
the user and reviewers, and rank accordingly.

While these studies aim to deliver more personalized reviews, they may
encounter the cold-start problem when users lack sufficient interactions
(Jalan and Gawande, 2017). Additionally, they rely on non-anonymous helpful
votes, which may not always be available and could be subject to presentation
bias (Moghaddam et al., 2011; Wang et al., 2013). In contrast, our approach
utilizes a context-aware strategy that does not rely on user-attributed
helpful votes and is immune to the influence of review rank.

###  2.3. User-generated Review Datasets

There are several public user-generated review datasets in the travel domain,
mainly crawled from leading online travel platforms such as TripAdvisor and
Booking.com. For example, Fang et al. (2016) crawled 41k reviews for
attractions in New Orleans to analyze the perceived helpfulness of reviews
(judged by the number of votes). Tsamis et al. (2021) crawled 65k English
hotel reviews from TripAdvisor and trained a DNN on helpfulness prediction.
Finally, the largest
dataset222https://www.kaggle.com/datasets/jiashenliu/515k-hotel-reviews-data-
in-europe/discussion we could find contains 515k reviews crawled from
Booking.com published between August 2015 to August 2017.

In terms of scale, public review datasets in the travel domain contain
hundreds of thousands of reviews at most. However, larger datasets exist in
non-travel domains, e.g., Amazon Product Reviews dataset (McAuley et al.,
2015) contains 82.8 million reviews across 24 domains. Furthermore, the
available contextual information about the user varies between the datasets.
For example, the Booking.com crawled dataset2 contains the reviewer’s origin
country and review publication date but doesn’t include information about the
stay (such as number of nights and traveler type).

The dataset we provide is a large-scale dataset with millions of user-
generated accommodation reviews. It contains comprehensive information about
the review and the reviewer’s context. The dataset details and its curation
process are described in Section 3.

###  2.4. Contrastive Learning

Contrastive Learning (CL) is a powerful paradigm that effectively captures
intricate relationships within data. Introduced as a technique to learn
representations by contrasting positive pairs against negative pairs, CL aims
to maximize the similarity between instances that should be similar while
minimizing the similarity between those that should differ. This concept was
first introduced by Bengio et al. (2013), and gained significant popularity
and widespread attention with the introduction of the CLIP model (Radford et
al., 2021). In the CLIP framework, CL was shown to be effective when applied
to a joint pre-training of visual and textual representations.

CL has since found applications beyond computer vision, including natural
language processing and recommendation systems (Liu et al., 2021). The
versatility of CL lies in its ability to distill intricate patterns and
semantic relationships from unlabeled data, significantly reducing the
reliance on labeled datasets.

In the context of our study, we harness the potential of CL to model the
nuanced relationship between reviewers’ contexts and their corresponding
reviews. Our approach contrasts positive instances, where reviews are paired
with their reviewers’ contexts, against negative instances, where reviews are
paired with their non-reviewers’ contexts from the same accommodation.

##  3\. Dataset

The dataset we publish contains authentic user-generated reviews from 50,000
accommodations. This includes information on the user reservation, the review
and the accommodation. The dataset is from a leading online travel platform
allowing reviews only from guests who stayed at the property. Therefore, every
review is associated with the guest context, such as the number of nights,
month, and the traveller type (e.g., solo traveller, family etc). There are
several fields describing the review: the title, the positive (”liked”)
section, the negative (”disliked”) section, the overall review score and the
number of helpful votes. Figure 1 demonstrates an example of a guest review.
The dataset is published under a non-commercial
license333https://creativecommons.org/licenses/by-sa/4.0/deed.en, available
via GitHub444https://github.com/bookingcom/ml-dataset-reviews.

Figure 1. A guest review example. The respective names of the fields in our
dataset are mentioned in green rectangles.

###  3.1. Data Selection

The dataset consists of English reviews published in 2023. All reviews have
passed a moderation process ensuring they are genuine and do not violate the
platform guidelines555See more details in
https://www.booking.com/reviews_guidelines.html. In order to preserve user
privacy, no personally identifiable information was included in the data.
Similarly, to protect business-sensitive statistics, the dataset is limited to
only tens of thousands of accommodations.

To identify informative reviews, we utilize a topic detection model called
Text2topic, specifically designed for travel-related tasks by Fengjun Wang,
Moran Beladev et al. (2023). We select reviews that have at least three
topics.

Our observations 666Based on the browser versions of the Booking.com and
Expedia travel platforms indicate that leading online travel platforms show 10
reviews in a page. Therefore, we select accommodations with at least 10
reviews. Finally, we uniformly sample 50,000 accommodations.

Figure 2 shows the distribution of the number of reviews per accommodation.
36.8 is the average, 20 is the median, and 1,818 is the maximum.

Figure 2. Histogram of number of reviews per accommodation

###  3.2. Data Schema

The dataset consists of 2,031,914 reviews, along with guest and accommodation
context. Table 1 describes the dataset fields. Key statistics of the data are
described in table 2.

Table 1. Dataset description: review, user and accommodation fields Field name |  Description
---|---
review_title |  Review title
review_positive |  Positive (”liked”) section in review
review_negative |  Negative (”disliked”) section in review
review_score |  Overall review score for the stay
review_helpful_votes |  How many users marked the review as helpful
guest_type |  There are 4 traveller types: Solo traveller (1 adult) / Couple (2 adults) / Group (¿2 adults) / Family with children (adults & children)
guest_country |  Anonymized country from which the reservation was made
room_nights |  The length of the reservation, i.e. number of nights booked
month |  The month of the check-in date of the reservation
accommodation_id |  Anonymized accommodation ID
accommodation_type |  The type of the accommodation, e.g. hotel, apartment, hostel
accommodation_score |  The overall average guest review score of the accommodation
accommodation_country |  Country of the accommodation
accommodation_star_rating |  Accommodation star rating is provided by the property, and is usually determined by an official accommodation rating organisation or another third party
location_is_beach |  Is the accommodation located in a beach location
location_is_ski |  Is the accommodation located in a ski location
location_is_city_center |  Is the accommodation located in the city center
Table 2. Training dataset field statistics Field name | Unique Values | Mean Value | Mode Value | Min Value | Max Value
---|---|---|---|---|---
review_score | 22 | 8.87 | 9.0 | 1.0 | 10.0
review_helpful_votes | 50 | 0.12 | 0 | 0 | 91
guest_type | 4 | Couple | - | - | -
guest_country | 271 | - | Cobra Island | - | -
room_nights | 72 | 2.58 | 4 | 1 | 112
month | 12 | - | July | - | -
accommodation_type | 27 | - | Hotel | - | -
accommodation_score | 71 | 8.38 | 8.5 | 2.4 | 10.0
accommodation_country | 180 | - | Australia | - | -
accommodation_star_rating | 10 | 2.38 | 4.0 | 0.0 | 5.0
location_is_beach | 2 | 0.23 | 0 | 0 | 1
location_is_ski | 2 | 0.02 | 0 | 0 | 1
location_is_city_center | 2 | 0.29 | 0 | 0 | 1

###  3.3. Data Split

The data is split to train, validation and test sets using 80% / 10% / 10%
random splits based on accommodation id. This means that every accommodation
and its corresponding reviews will appear only in one of the sets. The dataset
consists of the following files:

  * •

train.csv \- Training dataset of 1,628,989 reviews from 40,000 accommodations.

  * •

validation.csv \- Validation dataset of 203,787 reviews from 5,000
accommodations.

  * •

test.csv \- Test dataset of 199,138 reviews from 5,000 accommodations.

Our repository currently consists only the training set (i.e., train.csv file)
since we plan publishing a challenge in which the validation and test sets
will not be exposed to its participants. These sets will be published right
after the challenge.

##  4\. Problem Formulation

Our objective is to create a model that predicts the helpfulness of every
review tailored to individual users. In essence, we aim to construct a
personalized helpfulness function, denoted as $f(r_{j}|c_{i})$, which
evaluates the relevance of review $r_{j}$ to user $i$ given their context
$c_{i}$. This function assigns a score indicating the degree to which review
$j$ is beneficial for user $i$. These scores enable us to rank reviews,
ensuring that those with the highest $f$ values are deemed most helpful within
context $c_{i}$.

Using the number of helpful votes as the target signal inherits multiple
issues. First, it introduces a presentation bias towards the previous review
ranking algorithm (usually sorted by votes). Additionally, the signal of votes
is sparse as most of the reviews are not presented and therefore not voted.
Moreover, there might be a cold-start problem where new reviews don’t have as
many votes as older reviews which might be less relevant over time. Finally,
in many cases, only the final number of votes is stored and therefore it’s not
feasible to use this signal for developing personalized review ranking models.

Thus, we introduce a more feasible and novel approach for modeling
personalized helpfulness measure. We propose to model the personalized
helpfulness of a review as the likelihood that it is written by its reviewer
given the reviewer’s context. Notably, we define $f$ such that given a user
context $c_{i}$, it estimates the likelihood that review $r_{j}$ was written
by the user. Formally, we optimize $f$ such that given that review $r_{i}$ was
written by a user with context $c_{i}$, it holds:

(1) $f(r_{j}|c_{i})=\left\\{\begin{array}[]{ll}1&\text{if }i=j\\\ 0&\text{if
}i\neq j\end{array}\right.$

In practice, we learn $f$ through a contrastive modeling approach, described
in Section 5.

##  5\. Method

We propose a CL approach inspired by Radford et al. (2021). Our suggested
approach entails several key steps. Initially, it consolidates the fields of
the review into a unified string. Subsequently, it combines the fields
representing the context into another string. Thirdly, both strings undergo
separate encoding processes: one generates a latent representation for the
context, while the other produces a latent representation for the review.
Ultimately, our training objective aims to maximize the similarity between the
latent representation of each review and its corresponding reviewer’s context,
while simultaneously minimizing the similarity between each review and
contexts unrelated to the reviewer. In a live setting, our intention is to
employ the resulting model to rank a list of accommodation reviews based on
the context of the browsing user. Figure 3 offers a visual depiction of our
approach. The subsequent paragraphs delve into a comprehensive explanation of
each of the aforementioned steps.

Figure 3. The proposed approach: (1) user-generated review and (2) user
context are transformed into text and passed to (3) encoding layers that are
fine-tuned to optimize $f_{i,j}$ diagonal to 1 and the rest to 0. (4) In
inference time our model generates $f_{i,j}$ scores for a user context and a
set of reviews, and ranks based on the descending order of scores.

###  5.1. Encoding

Both context and review can be described as a set of multiple fields of
different types (textual, numeric and boolean). Inspired by (Hegselmann et
al., 2023), we consolidate all the review related fields into a single string
and all the context related fields into another string.

We utilize two distinct encoders to process the context and review strings.
Each encoder generates latent representations for $c_{i}$ and $r_{j}$
respectively. The encoder architecture incorporates a pre-trained language
model designed to extract contextual embeddings from the input tokens. We
adopt the $[CLS]$ token embedding as the latent representation for both the
context and review strings.

###  5.2. Interaction Matrix

Given a batch of $N$ user context embeddings and their corresponding review
embeddings, denoted as $C={c_{1},...,c_{N}}$ and $R={r_{1},...,r_{N}}$
respectively, where each review $r_{i}$ was written by a user with context
$c_{i}$, we construct an interaction matrix $F$. Each element $f_{i,j}$ of $F$
signifies the similarity between context $c_{i}$ and review $r_{j}$. Aligning
with the problem formulation described in Section 4, we use a similarity
function that guarantees $f_{i,j}\in[0,1]$. Specifically, we use sigmoid
function, i.e., $\sigma(x)=\frac{1}{1+e^{-x}}$, over the dot product of the
context and review embedding vectors:

(2) $f_{i,j}=\sigma({\mathbf{c_{i}}\cdot\mathbf{r_{j}}})$

###  5.3. Loss Functions

To optimize $F$ with respect to formula 1, we experiment with the following
loss functions:

InfoNCE loss \- inspired by CLIP model loss (Radford et al. (2021)), we use
n-pair / InfoNCE (Noise-Contrastive Estimation) loss, first introduced by Sohn
(2016). This loss maximizes the mutual information between positive pairs and
minimizes it for negative pairs. We apply it on both row-wise and column-wise,
and then take the average, as shown in the below formula:

(3)
$\mathcal{L}_{\text{InfoNCE}}=-\frac{1}{2N}\left(\sum_{i=1}^{N}\log\frac{\exp(f_{i,i})}{\sum_{j=1}^{N}\exp(f_{i,j})}+\sum_{j=1}^{N}\log\frac{\exp(f_{j,j})}{\sum_{i=1}^{N}\exp(f_{i,j})}\right)$

Binary cross entropy (BCE) loss \- this loss computes binary cross entropy
between the interaction matrix values and the targets, i.e., the identity
matrix $I$:

(4)
$\mathcal{L}_{\text{BCE}}=-\frac{1}{N^{2}}\sum_{i=1}^{N}\sum_{j=1}^{N}[\mathbf{I_{i,j}}\log(f_{i,j})+(1-\mathbf{I_{i,j}})\log(1-f_{i,j})]$

###  5.4. In-accommodation Sampling

Usually, in contrastive learning, training batches are randomly sampled from
the training set (Radford et al., 2021). The assumption is that every sample
is sufficiently distant from the other samples in the batch in terms of its
concept / meaning. In our case, reviews from different accommodations can be
easily differentiated. For example, reviews of a beach resort versus reviews
of a hotel in an urban area. Therefore, randomly sampling the batch might
result in a model that distinguishes between accommodations and ignores the
user context. In addition, in a live setting reviews are always ranked against
reviews from the same accommodation. Thus, we suggest an in-group sampling on
the accommodation level where each batch consists of reviews and contexts from
the same accommodation. We evaluate the efficacy of this sampling approach by
comparing it to random sampling in our experiments.

##  6\. Experiments

###  6.1. Data Preprocessing

To produce the textual inputs for the encoder, we preprocess the review and
context fields. We create a textual representation for every field using the
template: `"<field_name>: <field_value>\n"`. Fields with empty values are
skipped. Then, we concatenate the textual representations of the fields
according to a predefined order. The review input string consists of the
following fields in the following order: review title, review positive, review
negative and guest score. The context input string consolidates user-related
fields and accommodation-related fields. The user related fields occur in the
following order: guest country, guest type, number of nights and check-in
month. The accommodation-related fields are organized in the following order:
accommodation type, accommodation star rating, accommodation score,
accommodation type, location is beach, location is ski and location is city
center.

###  6.2. Encoder Architecture

We fine-tuned `all-MiniLM-L6-v2` model from Hugging Face Sentence Transformers
repository777https://huggingface.co/sentence-transformers. This model is pre-
trained on a variety of semantic similarity prediction tasks using a
contrastive objective over above 1 billion sentence pairs. Considering other
models and architectures might be beneficial as well, however, finding an
optimal combination of such is not a central aim of this research.

###  6.3. Fine-tune Details

We applied AdamW optimizer (Loshchilov and Hutter, 2018) with a weight decay
of `0.01` and an initial learning rate of `3e-5`. We fine-tuned for 4 epochs
as observed to be enough for the fine-tune process to saturate. We employed a
batch size of 64 and a warm-up rate of 0.05. We experimented with batch sizes
of $[16,32,64,128]$ and didn’t observe significant differences in performance.
All of our experiments were performed on a computation instance equipped with
1 NVIDIA A10G Tensor Core GPU, 8 vCPU and 32GB RAM. The fine-tune process took
9 hours.

###  6.4. Baselines

We compare our approach with two baselines: (1) helpful votes ranking - in
which $f_{i,j}$ are assigned with the number of helpful votes $r_{j}$ has.
Ranking is performed based on the descending order of $f_{i,j}$ values. (2)
Pre-trained model - in which we used `all-MiniLM-L6-v2` pre-trained model
without fine-tuning it.

Table 3. Performance of our approach and baselines. The highest results for each of the metrics are highlighted in bold font. | MRR | Precision@1 | Precision@10
---|---|---|---
Model | Mean | Std | Mean | Std | Mean | Std
Helpful votes baseline | 0.096 | 0.039 | 0.025 | 0.034 | 0.248 | 0.097
Pre-trained baseline | 0.098 | 0.039 | 0.026 | 0.035 | 0.251 | 0.095
Random sampling, InfoNCE loss | 0.147 | 0.051 | 0.049 | 0.048 | 0.375 | 0.106
Random sampling, BCE loss | 0.191 | 0.063 | 0.089 | 0.064 | 0.419 | 0.111
In-accommodation sampling, InfoNCE loss | 0.237 | 0.067 | 0.111 | 0.069 | 0.519 | 0.111
In-accommodation sampling, BCE loss | 0.278 | 0.074 | 0.154 | 0.079 | 0.549 | 0.110

###  6.5. Metrics

We segment the test set into groups of context-review tuples from the same
accommodation. We denote the set of accommodations with $A$ and the $k^{th}$
accommodation with $a_{k}$. For every accommodation $a_{k}$ there is a set of
context-review tuples $C_{k}$ and $R_{k}$ such that $|C_{k}|=|R_{k}|$, and
every review $r_{i}\in R_{k}$ was written by the reviewer with context
$c_{i}\in C_{k}$. For every $c_{i}\in C_{k}$ we produce a ranked list of the
reviews within the same accommodation $R_{k}$. We denote the rank of $r_{j}$
(the review written by the user with context $c_{j}$) in the ranked list of
$c_{j}$ with $Rank(j)$. Note that our goal is that given a context $c_{j}$,
review $r_{j}$ should have the highest rank in the ranked list of reviews.

We use 3 common metrics from the world of recommendation: Mean Reciprocal Rank
(MRR) \- we formulate the MRR metric as follows:

(5)
$MRR=\frac{1}{|A|}\sum_{k=1}^{|A|}\frac{1}{|C_{k}|}\sum_{j=1}^{|C_{k}|}\frac{1}{Rank(j)}$

Precision@k \- motivated by that reviews are usually presented in pages of 10
reviews per page, we measure precision@k for $k\in[1,10]$:

(6)
$Precision@k=\frac{1}{|A|}\sum_{k=1}^{|A|}\frac{1}{|C_{k}|}\sum_{j=1}^{|C_{k}|}{\text{I}(Rank(j)\leq
k)}$

###  6.6. Results

Results are described in Table 3 and Figure 4. We performed a Friedman test
(Milton, 1939) that showed there was a statistically significant difference
between the methods (p-value¡0.001). A post-hoc analysis using the pairwise
Dunn test (Dinno, 2015) showed that the model combines in-accommodation
sampling with BCE loss significantly outperformed all other methods
(p-value¡0.001). Moreover, the in-accommodation sampling significantly
outperformed random sampling for both InfoNCE and BCE loss functions
(p-value¡0.001). Finally, the BCE loss achieved significantly better
performance compared to InfoNCE loss (p-value¡0.001). This is probably due to
the softmax applied in the InfoNCE loss which normalizes the similarity values
one time on the row level and another time on the column level. The BCE, by
not applying softmax, allows multiple $f_{i,j}$ to have high values (i.e.,
close to 1) on the same row / column without forcing any dependency between
them.

Figure 4. Precision@k over $k\in[1,10]$ Table 4. Comparative analysis. ’Original Review Text’ is the original text written by the reviewer, ’Best Performing Model’s Top Review’ is our model top ranked review (excluding the original review), ’Pre-trained Baseline Model’s Top Review’ is the baseline top ranked review (excluding the original review). It can be clearly noticed that our model shares much more common topics with the original review compared to the baseline, showing our model better captures the user segment needs. Guest Type |  Original Review Text |  Best Performing Model’s Top Review |  Pre-trained Baseline Model’s Top Review
---|---|---|---
couple |  Fantastic views across the Sassi from terrace. Fabulous location. Easy access to property. Helpful and communicative host. |  Fantastic location with excellent view. Comfortable and well laid out small apartment. Very responsive and helpful host. |  Fantastic place! Wonderful terrace and very comfortable and well equipped place.
couple |  The view and the pool are amazing. The owner try is best to make your stay good. Not to far from the shops and beaches. |  \- the location near the best beaches \- the view from the rooms and from the pool is insane \- the kindness of the family owner. |  Perfect stay. Great hammock and views. My room was quite a ways up the hill and I liked it and the view from there. It was peaceful.
family with children |  Very close to Retiro Park, about 1.5 kms from the normal touristy areas. Quiet surroundings. Huge room 305- immaculate bathroom. |  Superb location, next to Retiro and Prado Museum. Very convenient facilities, with family rooms, and easy check in process. Staff was attentive. |  Located in my favorite spot in Madrid, cool Lobby area and good breakfast to start off the day.
family with children |  The receptionist was welcoming and the room was spacious and it has amenities like shops with free delivery and also a pool. |  Like the pool & the playground kids really enjoyed also the place is accessible a lot of options were you can buy some stuff. |  Very good value large apartment. Large pool outside, good location.
solo traveller |  Very attentive and friendly staff! Everything was clean and the other guests were nice, huge rooftop to sit in the evening and enjoy the fresh breeze. |  I like the inner patio. The location is great! I liked the owner and the manager at the hotel. I like there is the washing machine and dryer. |  Best location, close to many main places, nice room.
solo traveller |  Beautiful setting and great value for money. Location is safe and easy access to the highway. Loved the shower with a view of the hill and the staff is very friendly. |  The place was beautiful and in a good neighbourhood that felt safe. The host and staff were very friendly. Stayed only one night and hopefully I will stay longer in the future. |  Personal touch. Gina was a lovely host. Beautiful views over Johannesburg.
group |  Great place. The beds were really comfortable and the place is quiet. They offer tours to the Aral Sea, which is why we booked here. Would recommend. |  Very helpful staff, great breakfast. Nice relaxed yard, calm and clean rooms. Thank you also for helpful organization of tour to Elliq Qala. |  The hostel is a very nice place. We had a large room and in the morning we took off for Muynak, so we didn’t have breakfast at the place. People at the reception desk are very kind.
group |  I like the breakfast. Location is good, very close to the old town center. Room is not very big, but comfortable. |  Location is premium, with easy access to all city areas (especially Old Town). The room is very comfortable, and the common areas are well decorated. |  Spotless modern hotel in an excellent area close to the main Sq.

###  6.7. Comparative Analysis

In the following section, we delve into a comparative analysis of the results
generated by our best performing model against the pre-trained baseline. Our
aim is to demonstrate the interpretability and explainability of our model,
thereby improving transparency and fostering trust.

Table 4 presents 8 comparisons of the best performing model (i.e., in-
accommodation sampling with BCE loss) vs the pre-trained baseline model. For
simplicity, we show only part of the guest’s context (i.e., the guest type)
and only the positive (”liked”) section of the reviews. However, the process
we perform can be applied over the full set of fields. The table describes the
original review (written by the guest), the model’s top review and the pre-
trained baseline’s top review. In case any of the models selected the original
review to be on top of the ranked list, we show the second result in the list.
We do this to simulate a production setting where the user hasn’t written the
review yet. We selected reviews with 100-200 characters for ease of
comparison. Examples were randomly selected, 2 for every guest type. We
identified the topics mentioned in the reviews using the Text2topic model
(Fengjun Wang, Moran Beladev et al., 2023) and colored the common topics
between the original reviews and the models’ top reviews. For example, the
first row in the table describes a review written by a couple. The review
highlights the views, the terrace, the location, and the host helpfulness.
While the pre-trained baseline selected a review that highlights the terrace,
our best performing model selected a review that highlights the location, the
views and the host helpfulness but without mentioning the terrace.

As can be seen, this analysis provides an insightful visual comparison between
models. It enables to identify which model has a higher topic intersection
with the guest original review which indicates how effectively it interprets
contexts into user preferences. Moreover, it enables to identify the topic
intersection between the models, and whether different models identifies
different topics. Such insight for example, might motivate model ensemble.
Finally, it enables us to learn and explore the preferences of the different
user segments.

##  7\. Conclusions and Future Work

In this paper, we introduce a comprehensive review dataset sourced from a
prominent online travel platform. Our work proposes a novel formulation for
personalized review ranking, aiming to mitigate common biases and challenges
observed in current methodologies. We employ a contrastive learning approach
to tackle this task and evaluate its efficacy across various experimental
setups. Through comparative analysis, we demonstrate how our end-to-end
solution effectively captures the interplay between a reviewer’s context and
their review, thus enabling a personalized review ranking experience.

Our future work includes deploying and conducting experiments with our best
performing model in production via an A/B test. We seek to quantify user
engagement metrics and assess the business impact of our approach.
Furthermore, we strive to publish a challenge encouraging research individuals
and groups to address the task based on our dataset and problem formulation.
Additionally, we plan to enrich both context and review data by integrating
new signals such as user-provided review images. Finally, we aim to expand our
dataset to include more languages, enabling the development of multilingual
models to better serve diverse user segments.

## References

  * (1)
  * Agarwal et al. (2018) Aman Agarwal, Ivan Zaitsev, and Thorsten Joachims. 2018.  Consistent position bias estimation without online interventions for learning-to-rank.  _arXiv preprint arXiv:1806.03555_ (2018).
  * Baka (2016) Vasiliki Baka. 2016.  The becoming of user-generated reviews: Looking at the past to understand the future of managing reputation in the travel sector.  _Tourism management_ 53 (2016), 148–162.
  * Bengio et al. (2013) Yoshua Bengio, Aaron Courville, and Pascal Vincent. 2013.  Representation learning: A review and new perspectives.  _IEEE transactions on pattern analysis and machine intelligence_ 35, 8 (2013), 1798–1828.
  * Bilal and Almazroi (2023) Muhammad Bilal and Abdulwahab Ali Almazroi. 2023.  Effectiveness of Fine-tuned BERT Model in Classification of Helpful and Unhelpful Online Customer Reviews.  _Electronic Commerce Research_ 23, 4 (December 2023), 2737–2757.  https://doi.org/10.1007/s10660-022-09560-
  * Dinno (2015) Alexis Dinno. 2015.  Nonparametric pairwise multiple comparisons in independent groups using Dunn’s test.  _The Stata Journal_ 15, 1 (2015), 292–300.
  * Du et al. (2019) Jiahua Du, Jia Rong, Sandra Michalska, Hua Wang, and Yanchun Zhang. 2019.  Feature selection for helpfulness prediction of online product reviews: An empirical study.  _PLOS ONE_ 14, 12 (12 2019), 1–26.  https://doi.org/10.1371/journal.pone.0226902
  * Fan et al. (2019) Miao Fan, Chao Feng, Lin Guo, Mingming Sun, and Ping Li. 2019.  Product-Aware Helpfulness Prediction of Online Reviews. In _The World Wide Web Conference_ (San Francisco, CA, USA) _(WWW ’19)_. Association for Computing Machinery, New York, NY, USA, 2715–2721.  https://doi.org/10.1145/3308558.3313523
  * Fang et al. (2016) Bin Fang, Qiang Ye, Deniz Kucukusta, and Rob Law. 2016.  Analysis of the perceived value of online tourism reviews: Influence of readability and reviewer characteristics.  _Tourism Management_ 52 (2016), 498–506.  https://doi.org/10.1016/j.tourman.2015.07.018
  * Fengjun Wang, Moran Beladev et al. (2023) Fengjun Wang, Moran Beladev, Ofri Kleinfeld, Elina Frayerman, Tal Shachar, Eran Fainman, Karen Lastmann Assaraf, Sarai Mizrachi, and Benjamin Wang. 2023.  Text2Topic: Multi-Label Text Classification System for Efficient Topic Detection in User Generated Content with Zero-Shot Capabilities.  arXiv:2310.14817 [cs.LG]
  * Ghose and Ipeirotis (2007) Anindya Ghose and Panagiotis G. Ipeirotis. 2007.  Designing novel review ranking systems: predicting the usefulness and impact of reviews. In _Proceedings of the Ninth International Conference on Electronic Commerce_ (Minneapolis, MN, USA) _(ICEC ’07)_. Association for Computing Machinery, New York, NY, USA, 303–310.  https://doi.org/10.1145/1282100.1282158
  * Ghose and Ipeirotis (2011) Anindya Ghose and Panagiotis G. Ipeirotis. 2011.  Estimating the Helpfulness and Economic Impact of Product Reviews: Mining Text and Reviewer Characteristics.  _IEEE Transactions on Knowledge and Data Engineering_ 23, 10 (2011), 1498–1512.  https://doi.org/10.1109/TKDE.2010.188
  * Gretzel and Yoo (2008) Ulrike Gretzel and Kyung Hyan Yoo. 2008.  Use and impact of online travel reviews.  In _Information and communication technologies in tourism 2008_. Springer, 35–46.
  * Han et al. (2022) Wei Han, Hui Chen, Zhen Hai, Soujanya Poria, and Lidong Bing. 2022.  SANCL: Multimodal Review Helpfulness Prediction with Selective Attention and Natural Contrastive Learning. In _Proceedings of the 29th International Conference on Computational Linguistics_ , Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung-Hoon Na (Eds.). International Committee on Computational Linguistics, Gyeongju, Republic of Korea, 5666–5677.  https://aclanthology.org/2022.coling-1.499
  * Hegselmann et al. (2023) Stefan Hegselmann, Alejandro Buendia, Hunter Lang, Monica Agrawal, Xiaoyi Jiang, and David Sontag. 2023.  Tabllm: Few-shot classification of tabular data with large language models. In _International Conference on Artificial Intelligence and Statistics_. PMLR, 5549–5581.
  * Huang et al. (2020) Chunli Huang, Wenjun Jiang, Jie Wu, and Guojun Wang. 2020.  Personalized Review Recommendation based on Users’ Aspect Sentiment.  _ACM Trans. Internet Technol._ 20, 4, Article 42 (oct 2020), 26 pages.  https://doi.org/10.1145/3414841
  * Jalan and Gawande (2017) Khushbu Jalan and Kiran Gawande. 2017.  Context-aware hotel recommendation system based on hybrid approach to mitigate cold-start-problem. In _2017 International Conference on Energy, Communication, Data Analytics and Soft Computing (ICECDS)_. IEEE, 2364–2370.
  * Korfiatis et al. (2012) Nikolaos Korfiatis, Elena García-Bariocanal, and Salvador Sánchez-Alonso. 2012.  Evaluating content quality and helpfulness of online product reviews: The interplay of review helpfulness vs. review content.  _Electronic Commerce Research and Applications_ 11, 3 (2012), 205–217.  https://doi.org/10.1016/j.elerap.2011.10.003
  * Kuan et al. (2015) Kevin KY Kuan, Kai-Lung Hui, Pattarawan Prasarnphanich, and Hok-Yin Lai. 2015.  What makes a review voted? An empirical investigation of review voting in online review systems.  _Journal of the Association for Information Systems_ 16, 1 (2015), 1.
  * Liu et al. (2021) Zhuang Liu, Yunpu Ma, Yuanxin Ouyang, and Zhang Xiong. 2021.  Contrastive learning for recommender system.  _arXiv preprint arXiv:2101.01317_ (2021).
  * Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter. 2018.  Fixing weight decay regularization in adam.  (2018).
  * McAuley et al. (2015) Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. 2015.  Image-Based Recommendations on Styles and Substitutes. In _Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Santiago, Chile) _(SIGIR ’15)_. Association for Computing Machinery, New York, NY, USA, 43–52.  https://doi.org/10.1145/2766462.2767755
  * Milton (1939) Friedman Milton. 1939.  A correction: The use of ranks to avoid the assumption of normality implicit in the analysis of variance.  _Journal of the American Statistical Association. American Statistical Association_ 34, 205 (1939), 109.
  * Moghaddam et al. (2011) Samaneh Moghaddam, Mohsen Jamali, and Martin Ester. 2011.  Review recommendation: personalized prediction of the quality of online reviews. In _Proceedings of the 20th ACM International Conference on Information and Knowledge Management_ (Glasgow, Scotland, UK) _(CIKM ’11)_. Association for Computing Machinery, New York, NY, USA, 2249–2252.  https://doi.org/10.1145/2063576.2063938
  * Nayeem and Rafiei (2023) Mir Tafseer Nayeem and Davood Rafiei. 2023.  On the Role of Reviewer Expertise in Temporal Review Helpfulness Prediction. In _Findings of the Association for Computational Linguistics: EACL 2023_ , Andreas Vlachos and Isabelle Augenstein (Eds.). Association for Computational Linguistics, Dubrovnik, Croatia, 1684–1692.  https://doi.org/10.18653/v1/2023.findings-eacl.125
  * Peddireddy (2020) Akhil Sai Peddireddy. 2020.  Personalized Review Ranking for Improving Shopper’s Decision Making: A Term Frequency based Approach.  arXiv:2009.03258 [cs.IR]
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021\.  Learning transferable visual models from natural language supervision. In _International conference on machine learning_. PMLR, 8748–8763.
  * Ren et al. (2024) Gang Ren, Lei Diao, Fanjia Guo, and Taeho Hong. 2024.  A co-attention based multi-modal fusion network for review helpfulness prediction.  _Information Processing & Management_ 61, 1 (2024), 103573.  https://doi.org/10.1016/j.ipm.2023.103573
  * Ricci and Wietsma (2006) Francesco Ricci and René TA Wietsma. 2006.  Product reviews in travel decision making.  In _Information and communication technologies in tourism 2006_. Springer, 296–307.
  * Robertson et al. (1995) Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995\.  Okapi at TREC-3.  _Nist Special Publication Sp_ 109 (1995), 109.
  * Sohn (2016) Kihyuk Sohn. 2016.  Improved deep metric learning with multi-class n-pair loss objective.  _Advances in neural information processing systems_ 29 (2016).
  * Tsamis et al. (2021) Konstantinos Tsamis, Andreas Komninos, Konstantinos Kovas, and Nikolaos Zotos. 2021.  Ranking Online User Reviews for Tourism Based on Usefulness. In _Proceedings of the 24th Pan-Hellenic Conference on Informatics_ (Athens, Greece) _(PCI ’20)_. Association for Computing Machinery, New York, NY, USA, 208–213.  https://doi.org/10.1145/3437120.3437308
  * Tuominen (2011) Pasi Tuominen. 2011.  The influence of TripAdvisor consumer-generated travel reviews on hotel performance.  (2011).
  * Wang et al. (2013) Bingkun Wang, Yulin Min, Yongfeng Huang, Xing Li, and Fangzhao Wu. 2013.  Review rating prediction based on the content and weighting strong social relation of reviewers. In _Proceedings of the 2013 International Workshop on Mining Unstructured Big Data Using Natural Language Processing_ (San Francisco, California, USA) _(UnstructureNLP ’13)_. Association for Computing Machinery, New York, NY, USA, 23–30.  https://doi.org/10.1145/2513549.2513554
  * Wang et al. (2016) Xuanhui Wang, Michael Bendersky, Donald Metzler, and Marc Najork. 2016.  Learning to rank with selection bias in personal search. In _Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval_. 115–124.
  * Yang et al. (2015) Yinfei Yang, Yaowei Yan, Minghui Qiu, and Forrest Bao. 2015.  Semantic Analysis and Helpfulness Prediction of Text for Online Product Reviews. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , Chengqing Zong and Michael Strube (Eds.). Association for Computational Linguistics, Beijing, China, 38–44.  https://doi.org/10.3115/v1/P15-2007
  * Yue et al. (2010) Yisong Yue, Rajan Patel, and Hein Roehrig. 2010.  Beyond position bias: Examining result attractiveness as a source of presentation bias in clickthrough data. In _Proceedings of the 19th international conference on World wide web_. 1011–1018.