[1]Bereket A. Yilma

1]Department of Computer Science, University of Luxembourg, 6, avenue de la
Fonte, Belval, 4364, Esch-sur-Alzette, Luxembourg

# MOSAIC: Multimodal Multistakeholder-aware Visual Art Recommendation

<EMAIL_ADDRESS>Luis A. Leiva<EMAIL_ADDRESS>[

###### Abstract

Visual art (VA) recommendation is complex, as it has to consider the interests
of users (e.g. museum visitors) and other stakeholders (e.g. museum curators).
We study how to effectively account for key stakeholders in VA recommendations
while also considering user-centred measures such as novelty, serendipity, and
diversity. We propose MOSAIC, a novel multimodal multistakeholder-aware
approach using state-of-the-art CLIP and BLIP backbone architectures and two
joint optimisation objectives: popularity and representative selection of
paintings across different categories. We conducted an offline evaluation
using preferences elicited from 213 users followed by a user study with 100
crowdworkers. We found a strong effect of popularity, which was positively
perceived by users, and a minimal effect of representativeness. MOSAIC’s
impact extends beyond visitors, benefiting various art stakeholders. Its user-
centric approach has broader applicability, offering advancements for content
recommendation across domains that require considering multiple stakeholders.

###### keywords:

Recommendation, Personalization, Artwork, User Experience, Machine Learning

##  1 Introduction

Appreciating visual art (VA) is a complex and subjective experience that is
influenced by a range of factors, including personal preferences, cultural
context, and historical significance. Paintings are important VA items that
bring together complex elements such as drawings, narration, composition, or
abstraction [1]. As VA becomes more diverse and accessible, there is an
increasing need for innovative approaches to facilitate art discovery, promote
engagement, and ultimately support user satisfaction. This is challenging,
since the emotional and cognitive responses triggered by VA can vary widely
among users [2]. In this context, VA recommender systems (VA RecSys) have
emerged as a promising solution to these challenges, as they can provide
personalised recommendations based on users’ preferences and interests [3].

Overall, current approaches in VA RecSys essentially try to leverage different
representations of paintings to identify similarities with user preferences
and derive personalised recommendations [4, 5]. Unlike other domains
considering e.g. our cultural heritage, the development of VA RecSys poses
additional unique challenges. This originates from the growing need to factor
in the diverse perspectives and expectations of stakeholders involved in the
art ecosystem (such as artists, curators, collectors, museums, art
enthusiasts, and educators) to generate suitable recommendations [6]. The
underlining assumption is that optimising only for previously elicited user
preferences overlooks the diverse and evolving nature of human taste [7].
However, the very essence of art is rooted in the belief that humans are
learning creatures on a journey of exploration and discovery, picking up
preferences and developing interests as they come across different content
[8].

To truly capture the essence of human experience towards art, VA RecSys must
be reimagined, such that it reflects the natural, evolving nature of human
preferences. As such, it is crucial for VA RecSys to expose users to new
content and perspectives, rather than merely relying on a narrow set of
assumptions regarding user preferences, which could be a disservice to the
complex and diverse nature of art [3]. Nonetheless, such a task for a VA
RecSys is far from trivial. It calls for algorithms and techniques that can
consider novel and diverse perspectives while striking a reasonable balance
between already established user preferences, while also allowing users to
control the level of exposition and exploration of new VA content.

Early work on VA RecSys has primarily relied on inferring similarities and
relationships among paintings based on high-level features derived from
traditional metadata such as artist names, styles, and materials [9, 8].
Nonetheless, these features are not sufficiently expressive to cater for the
subjective taste of users. In recent years, researchers have started to focus
on more advanced VA representation learning techniques to better capture
semantic relatedness of art works. For example, He et al. [10] and Messina et
al. [11] demonstrated that latent visual features extracted using Deep Neural
Networks outperform textual metadata. Other studies have also argued in favor
of visual features over textual metadata in VA RecSys [9, 3].

Recent research has shown that both text and visual modalities capture the
complex semantics embedded in VA, but it is their combination, using either
late or early fusion techniques, the approach that delivers better performance
[12]. A late fusion strategy combines rankings generated from independently
trained modes, whereas early fusion, using multimodal representation learning
techniques, consider a joint embedding from visual and textual data. Early
fusion approaches have shown remarkable success in various down stream tasks
[13], including in VA RecSys [14].

Contrastive Language-Image Pre-training (CLIP) [15] and Bootstrapping
Language-Image Pre-training (BLIP) [16] are the two most prominent early
fusion approaches that have been used recently for VA RecSys [14]. These
multimodal models have also been successfully applied in several downstream
tasks such as scene understanding [17], image retrieval [18], or question
answering [19]. In VA RecSys, the learned representations can be leveraged to
match elicited preferences from users and derive recommendations that are
presumably accurate. However, in addition to finding efficient
representations, seamlessly blending multiple stakeholder perspectives with
user preferences remains an open challenge in VA RecSys. Hence, in this work
we address the following research question: How to effectively account for
multiple stakeholders in user-centric VA recommendations?

To answer this research question, we propose MOSAIC (MultimOdal
multiStakeholder-Aware vIsual art reCommendation). We build multimodal
representations of paintings by jointly training on textual descriptions and
images of paintings and then propose different strategies to jointly optimise
for multistakeholder objectives that promote user-centric beyond-accuracy
measures [20] in VA recommendations. In sum, this paper makes the following
contributions:

  * •

We introduce MOSAIC, a novel VA RecSys approach that optimises for multiple
stakeholders in the art ecosystem.

  * •

We develop and study six variants of MOSAIC on top of two backbone
architectures, to progressively and seamlessly introduce joint optimisation
objectives.

  * •

We conduct an offline evaluation to assess the impact of such joint objectives
and then conduct a user study to assess MOSAIC’s performance from a user-
centric perspective.

  * •

We contextualise our findings and provide valuable insights on how to design
next-generation VA RecSys blending learning, discovery, and user satisfaction.

##  2 Related Work

The concept of multistakeholder recommendation has gained significant
attention recently, with the growing need for an inclusive approach that
expands outward from the user, in order to include the perspectives and
utilities of multiple stakeholders [21]. Historically, the RecSys community
has been focused on creating value for consumers. However, these systems in
practice serve some business or organisational objectives as well. Thus,
recently more research has been devoted to properly consider the needs of all
stakeholders involved by viewing recommendation as a problem that requires
balancing the interests and objectives of multiple parties, taking into
account different optimization time horizons [22].

In a multistakeholder environment, an ideal recommender system should aim to
balance the objectives of all stakeholders to ensure a satisfactory outcome
for everyone involved, including consumers, providers, and the recommendation
platform, as well as potentially relevant side stakeholders such as society
[23]. Many RecSys studies operate under the assumption that improving the
accuracy of recommendations will ultimately lead to more valuable systems for
users. It is a logical assumption that a better algorithm should present more
relevant items at the top of the list, resulting in a better user experience.
However, this assumption does not always hold, as just being accurate is not
enough for a recommender system to be successful [24]. Therefore, while
accuracy is undoubtedly an essential factor, it is not the only one. A good
recommender system must consider other user-related aspects that go beyond
accuracy [25].

Such observations have led to a strand of research on “beyond-accuracy”
measures like diversity, novelty, or serendipity [26]. The domain of VA RecSys
is among the fields where the notion of multistakeholder-awareness is vital
yet often overlooked [27]. This is primarily due to the fact that paintings
are both high dimensional and semantically complex, leading to a diverse and
subjective set of emotional and cognitive reflections on users [28, 2].
Indeed, art paintings reflect the complex and intricate interplay of concepts
ranging from individual ideas and beliefs to concepts with cultural and
historical significance of a society [7]. Furthermore, in the VA ecosystem
there are multiple stakeholders that are directly or indirectly affected by
the way recommendations are delivered to visitors such as artists, curators,
collectors, museums, art enthusiasts, educators as well as society at large
[6].

The recent VA RecSys literature has been mostly focused on finding efficient
data representation techniques that are capable of capturing the complex
semantics embedded in paintings [9, 29, 10]. A recently notable work by Yilma
et al. [12] demonstrated that the key explanatory factors for semantic
relatedness of art works can be captured from the fusion of textual and visual
modalities. That work leveraged a late rank fusion strategy that combined
rankings of recommendations generated from models independently trained on
image and textual data. More recently, it has been shown that, instead of late
rank fusion, jointly learning embedding from textual and visual data (early
fusion) outperforms the late rank fusion strategy [14]. This later work
underlines the fact that the elements of visual art recommendation lie at the
intersection of textual and visual modalities. Although efficient multimodal
representations to capture abstract semantics in VA are important steps
towards quality VA recommendations, it is still unclear how to factor in
multistakeholder perspectives and generate VA recommendations that strike an
optimal balance between accuracy, novelty, serendipity, and diversity. Thus,
research effort to seamlessly incorporate learning and discovery with user
preferences in VA RecSys is still considered a worthwhile endeavour,
especially with regards to evaluating the quality of the recommendations from
a user-centric perspective [30].

##  3 Multimodal Visual art representation learning

Traditionally, late fusion (i.e. at post-hoc) has been preferred over early
fusion (i.e. at the feature level) because the fused models can be independent
of each other, so each can use their own features, numbers of dimensions, etc.
[31]. However, recent advances in multimodal representation learning
techniques have shown promising results in several downstream tasks [32],
including VA RecSys [14, 33]. We now present two multimodal representation
learning techniques based on the state-of-the-art CLIP and BLIP architectures,
and show how the learned representations can be utilised to generate VA
recommendations.

###  3.1 Contrastive Language-Image Pre-training (CLIP)

CLIP is a technique for pre-training a language model on a large dataset of
images and their associated text [15]. Unlike traditional models that use an
image encoder (e.g., CNN) and a classifier (e.g., a fully-connected network),
CLIP jointly trains an image encoder and a text encoder to encourage a close
embedding space between the ones that form a pair via contrastive learning.
The model predicts the correct image given a text caption, and vice versa. It
learns to generate a shared embedding space for both image and text inputs,
where similar images and texts are close to each other in the embedding space.
The idea is that by pre-training on this task, the model will learn to
understand the relationship between language and visual concepts, which can
then be fine-tuned for a variety of natural language understanding and image
understanding tasks.

Particularly, given a batch of $M<$image,text$>$ pairs, CLIP is trained to
predict which of the $M$ $\times$ $M$ possible image-text pairings across a
batch actually occurred. Thus, the model learns a multimodal embedding space
by jointly training an image encoder, either a ResNet or a Vision Transformer
(ViT), and a text encoder (a Transformer model, such as BERT) to maximise the
cosine similarity of the image and text embeddings of the $M$ real pairs in
the batch, while minimizing the cosine similarity of the embeddings of the
$M^{2}$ \- $M$ incorrect pairings. Hence, a symmetric cross-entropy loss is
optimised over these similarity scores.

In this work, we leveraged CLIP as a backbone feature extractor to learn a
joint embedding space for images of paintings and their corresponding textual
descriptions in order to uncover the latent semantic relationships between
paintings embedded within the two modalities. We use ResNet-50 as our image
encoder and BERT as our text encoder in our CLIP architecture. Once the
features from both modalities are extracted, a matrix
$\mathbf{A}\in\mathbb{R}^{m\times m}$ is computed where each entry
$\mathbf{A}_{ij}$ is the cosine similarity measure between the joint painting
embeddings, which is then leveraged to compute semantic similarities of
paintings for VA RecSys tasks. Figure 1 illustrates our multimodal approach to
learn latent semantic representations of paintings with CLIP.

Figure 1: Overview of our multimodal approach with CLIP to learn latent
semantic representations of paintings.

###  3.2 Bootstrapping Language-Image Pre-training (BLIP)

BLIP is also a method for pre-training a neural network that combines language
and image modalities [16]. However, unlike CLIP, BLIP pre-trains a model by
learning to predict an image or a text given the other modality, with the goal
of bootstrapping the model’s understanding of multimodal relationships. BLIP
introduces a mixture of encoder-decoder, a unified model which can operate in
one of the following three functionalities. (1)  _Unimodal encoder_ , which
separately encode image and text. The image encoder is a ViT and the text
encoder is BERT. A [CLS] token is appended to the beginning of the text input
to summarise the sentence. (2)  _Image-grounded text encoder_ , which injects
visual information by inserting a cross-attention layer between the self-
attention layer and the feed forward network for each transformer block of the
text encoder. A special [Encode] token is appended to the text, and its output
embedding is used as the multimodal representation of the image-text pair. (3)
_Image-grounded text decoder_ , which replaces the bi-directional self-
attention layers in the text encoder with causal self-attention layers. A
special [Decode] token is used to signal the beginning of a sequence.

BLIP jointly optimises three objectives, namely Image-Text Contrastive Loss
(ITC), Image-Text Matching Loss (ITM), and Language Modeling Loss (LM). ITC
activates the unimodal encoder with the aim of aligning the feature space of
the visual and text transformers by encouraging positive image-text pairs to
have similar representations in contrast to the negative pairs. ITM activates
the image-grounded text encoder for a binary classification task, where the
model is asked to predict whether an image-text pair is positive (matched) or
negative (unmatched) given their multimodal feature. Finally, LM activates the
image-grounded text decoder, which aims to generate textual descriptions
conditioned on the images.

Figure 2: Overview of our multimodal approach with BLIP to learn latent
semantic representations of paintings.

For the task of VA representation learning, we leveraged the pre-trained BLIP
model as a multimodal feature extractor. We compute ITM scores for every
painting by first extracting multimodal features and then passing them to the
ITM head, which generates probability matching (i.e., similarity) scores for
each $<$image,text$>$ pairs. Afterwards, a matrix
$\mathbf{A}\in\mathbb{R}^{m\times m}$ is computed where each entry
$\mathbf{A}_{ij}$ is the probability matching score between the joint painting
embeddings which is then leveraged to compute semantic similarities of
paintings for VA RecSys tasks. Figure 2 illustrates our multimodal approach to
learn latent semantic representations of paintings with BLIP.

##  4 Multistakeholder-aware Visual Art Recommendation

In addition to the users of VA RecSys, the art ecosystem brings together
different stakeholders that have a direct or indirect influence on users. The
two most important stakeholders in VA recSys are the crowd, who contribute to
deciding the popularity of art pieces, and museum curators, who aim to promote
certain topics, themes, artists, and stories. Concurrently optimising for
these two objectives also reflects a subset of stakeholders, including
artists, collectors, and society at large [27, 6]. Hence, VA RecSys needs to
consider such relevant stakeholders to blend learning and discovery in
personalised recommendations. In this section, we propose strategies starting
from purely personalised recommendations (baseline) to progressively introduce
joint optimisation objectives. We work with a dataset described in Section 5.1
that comprises several painting collections from the National Gallery,
London.111https://www.nationalgallery.org.uk/

###  4.1 Personalised recommendations

Let $\mathcal{P}=\\{p_{1},p_{2},\dots,p_{m}\\}$ be a set of paintings with
their associated embeddings (latent feature vectors)
$\mathcal{P}^{*}=\\{\boldsymbol{p}_{1},\boldsymbol{p}_{2},\dots,\boldsymbol{p}_{m}\\}$
according to CLIP or BLIP, and let
$\mathcal{P}^{u}=\\{p^{u}_{1},p^{u}_{2},\dots,p^{u}_{n}\\}$ be the set of
paintings a user $u$ has rated, where $\mathcal{P}^{u}\subset\mathcal{P}$ and
$\omega^{u}=\\{\omega^{u}_{1},\omega^{u}_{2},\dots,\omega^{u}_{n}\\}$ are the
normalised ratings222In our work, users elicit their preferences in a 5-point
scale rating (higher is better), thus we transform those values into weights
$\omega^{u}_{i}\in[0,1]$ for every painting $p^{u}_{i}$ the user has rated.
that $u$ gave to a small set of paintings $\mathcal{P}^{u}$. Once the dataset
embeddings are learned using our backbone models (CLIP or BLIP), we compute
the similarity matrix $\mathbf{A}$ for all the paintings as discussed in
Section 3. Then, the predicted score $S^{u}(p_{i})$ the user would give to
each painting in the collection $P$ is calculated based on the weighted
average distance between the rated paintings and all other paintings:

$S^{u}(p_{i})=\frac{1}{n}\sum^{n}_{j=1}\left(\omega^{u}_{j}\,\mathbf{A}_{ij}\right)$
(1)

where $\mathbf{A}_{ij}=d(\boldsymbol{p}_{i},\boldsymbol{p}_{j})$ is the
similarity score between embeddings of paintings $p_{i}$ and $p_{j}$ in the
computed similarity matrix. The summation in Equation 1 is taken over all
user’s rated paintings $n=|\mathcal{P}^{u}|$. Once the scoring procedure is
complete, the paintings are sorted and the $r$ most similar paintings
constitute a ranked recommendation list (in this work we use $r=9$). In sum, a
VA RecSys task consists of recommending the most similar paintings to a user
based on a small set of paintings rated before, i.e., the elicited
preferences.

###  4.2 Popular recommendations

In addition to individual preferences, users also vary in their interest to
visit famous paintings. To account for this, we introduce a popularity score,
denoted by $S^{\text{pop}}(p_{i})$, for each painting in our dataset. This
score is based on a “must-see” list, generated according to public reviews
from the National Gallery website. To consider both the user’s individual
preferences and the additional influence of the crowd on the popularity of a
painting, we create an aggregated preference score $S^{u}_{+}(p_{i})$ for each
painting in the collection:

$S^{u}_{+}(p_{i})=S^{u}(p_{i})+\beta\;S^{\text{pop}}(p_{i})$ (2)

where $\beta$ is a user-provided hyperparameter, which determines their level
of interest in (or their tolerance to) seeing popular items. Note that $\beta$
is considered as a boosting scoring method, since popularity has a direct
influence on the user recommendations.

###  4.3 Representative recommendations

As mentioned before, other key stakeholders are museum curators, responsible
for preparing the art exhibitions, therefore VA recommendations should reflect
certain themes or topics that curators may want to promote. The National
Gallery dataset, for example, features nine curated stories defined by artists
(Women’s Lives, Contemporary Style & Fashion, Water, Women Artists & Famous
Women, Warfare, Monsters & Demons, Migration & Exile, Death, Battles &
Commanders). Each story constitutes a certain number of paintings from the
collection. The objective of the curators here is to increase the number of
curated stories in the recommendation set; i.e., the recommended set contains
paintings that are representative or fairly selected from the curated story
groups. Thus, we define a representative story selection strategy, adopted
from Mehrotra et al. [34]. A recommendation set $\mathcal{R}$ is fairly
representative if it contains paintings that belong to different story groups
in a balanced way, so a representative story selection function
$\psi(\mathcal{R})$ is given by:

$\psi(\mathcal{R})=\sum^{K}_{a=1}\sqrt{\sum_{p_{i}\in\mathcal{S}_{a}\cap\mathcal{R}}}\gamma_{p_{i}}$
(3)

where $K$ is the total number of story groups, $\mathcal{S}_{a}$ is the $a$th
story group and $\gamma_{p_{i}}$ is a count for every painting $p_{i}$
selected from a story group $i$. The function $\psi(\mathcal{R})$ rewards
recommendation sets that are diverse in terms of the different story groups
covered.

It can be noticed that the representativeness score for a set that contains
paintings belonging to only one or few of the story groups is lower than a set
that covers all or most of the story groups, owing to the square root
function. For example assuming three story groups ($K=3$) a recommendation set
$\mathcal{R}$ that chooses 2 paintings from $\mathcal{S}_{1}$ and 1 painting
from each of $\mathcal{S}_{2}$ and $\mathcal{S}_{3}$ gets a higher
$\psi(\mathcal{R})$ score as compared to a recommendation set that chooses 4
paintings from just $\mathcal{S}_{1}$, since
$\sqrt{2}+\sqrt{1}+\sqrt{1}>\sqrt{4}+\sqrt{0}+\sqrt{0})$.

###  4.4 MOSAIC policies

The discussion above yields three different policies worth of investigation,
related to our key VA RecSys stakeholders. The first policy relates to
matching user preferences. Given a set of paintings $\mathcal{P}$ and a user
$u$, we select the most relevant set $\mathcal{R}$ to recommend, which
maximises:

$\text{Policy}\;1:\left\\{\begin{matrix}\arg\max\sum_{i=1}^{\mathcal{R}}S^{u}(p_{i})\end{matrix}\right.$
(4)

The second policy comes from the crowd influence on the popularity of
paintings, which maximises:

$\text{Policy}\;2:\left\\{\begin{matrix}\arg\max\sum_{i=1}^{\mathcal{R}}S^{\text{pop}}(p_{i})\end{matrix}\right.$
(5)

The third policy investigates matching the curators’ goal of finding a
recommendation set that features representative paintings from each of the
story groups, which maximises:

$\text{Policy}\;3:\left\\{\begin{matrix}\arg\max\
\psi(\mathcal{R})=\arg\max\sum^{K}_{a=1}\sqrt{\sum_{p_{i}\in\mathcal{S}_{a}\cap\mathcal{R}}}\gamma_{p_{i}}\end{matrix}\right.$
(6)

Note that maximising over $\psi(\mathcal{R})$ ensures diversity of the
recommendations in terms of the story groups to be covered. However,
minimising over $\psi(\mathcal{R})$ ensures consistency of the recommendations
in terms of multimodal information. Also note that contrary to the influence
of the crowd,333Making items popular is not an objective of the crowd, but
rather a consequence of their behaviour, indirectly influencing user
preferences. curators and users might have competing objectives, therefore our
formulation accounts for that by introducing a complementary dependency
through a $\xi$ parameter, as described in the next section.

###  4.5 MOSAIC engines

With the goal of introducing multistakeholder awareness in VA RecSys, we
designed different versions of MOSAIC engines by gradually departing from
optimizing for user preferences and combining the above policies. We start by
introducing the impact of popularity on personalised recommendations. Thus, we
combine Policies 1 and 2 and create two VA RecSys engines: CLIP-P and BLIP-P.
These engines use CLIP and BLIP respectively as a backbone multimodal engines
to compute the user’s preference scores $S^{u}(p_{i})$ according to the
scoring function in Equation 2.

To study the impact of representativeness on personalised recommendations, we
combine Policies 1 and 3 and create two new VA RecSys engines: CLIP-F and
BLIP-F. These engines try jointly to satisfy individual user preferences and
curator goals. This translates to finding a recommendation set $\mathcal{R}$
of paintings by solving the following Mixed Integer Programming (MIP) problem:

$\text{MIP}_{1}=\arg\max\left((1-\xi)\sum_{i=1}^{R}p_{i}\,S^{u}(p_{i})+\xi\sum^{K}_{a=1}\sqrt{\sum_{p_{i}\in
S_{a}\cap\mathcal{R}}}\gamma_{p_{i}}\right)$ (7)

where $\xi$ is a user-provided hyperparameter, indicating their tolerance to
receiving diverse painting recommendations.

Finally, by combining all the three policies above, we introduce two full-
versions of MOSAIC, CLIP-M and BLIP-M, each using either one of the backbone
architectures as preference scoring models. Ultimately, MOSAIC aims for a
recommended set $\mathcal{R}$ that concurrently satisfies the above three
policies (i.e., individual user preferences, crowd influence in terms of
popularity bias, and curators’ objective in terms of maximising the number of
story groups represented in the recommendation set) achieved by solving the
following MIP problem:

$\text{MIP}_{2}=\arg\max\left((1-\xi)\sum_{i=1}^{\mathcal{R}}p_{i}\,S^{u}_{+}(p_{i})+\xi\sum^{K}_{a=1}\sqrt{\sum_{p_{i}\in\mathcal{S}_{a}\cap\mathcal{R}}}\gamma_{p_{i}}\right)$
(8)

where $S^{u}_{+}(p_{i})$ combines Policy 1 and 2 and $\xi$ indicates the
user’s tolerance to receiving diverse painting recommendations.

Overall, we developed eight VA RecSys engines. The first two engines, CLIP and
BLIP, are considered baselines, since they generate pure personalised
recommendations. Next, CLIP-P and BLIP-P factor in the influence of crowds, in
terms of popularity bias. Then, CLIP-F and BLIP-F take into account the
curator’s preferences, in terms of fair (i.e., representative) content
selection. Finally, the full-version of MOSAIC (CLIP-M and BLIP-M) combine the
three key stakeholders considered relevant in this context, informed by
Cultural Heritage studies [35, 36, 8]: individual users, crowd of visitors,
and museum curators. Our proposed VA RecSys engines are outlined in Algorithms
1 to 5, respectively.

Algorithm 1 Dataset preprocessing.

1:procedure Preprocess($\text{Model},P$) $\triangleright$ Model
$\in\\{\text{CLIP, BLIP}\\}$

2: $\mathcal{P}\leftarrow\textsc{FeaturizePaintings}(\text{Model},P)$
$\triangleright$ Compute painting embeddings

3: $\mathbf{A}\leftarrow\varnothing$ $\triangleright$ Initialize
$\mathbf{A}\in\mathbb{R}^{m\times m}$ where $m=|\mathcal{P}|$

4: for $\boldsymbol{p}_{i}$ and $\boldsymbol{p}_{j}\in\mathcal{P}$ do
$\triangleright$ Iterate over collection

5: $\mathbf{A}_{ij}\leftarrow$
Similarity($\boldsymbol{p}_{i},\boldsymbol{p}_{j}$) $\triangleright$ Compute
embeddings similarity

6: end for

7: return $\mathbf{A}$

8:end procedure

Algorithm 2 Personalised VA RecSys (baseline engine).

1:Similarity matrix $\mathbf{A}$ of featurised paintings

2:procedure RecommendPaintings($\text{Model},P^{u},\omega^{u},r$)
$\triangleright$ Model $\in\\{\text{CLIP, BLIP}\\}$

3: $\mathcal{P}^{u}\leftarrow\textsc{FeaturizePaintings}(\text{Model},P^{u})$
$\triangleright$ Load computed embeddings, as $P^{u}\subset P$

4: $S^{u}\leftarrow\varnothing$ $\triangleright$ Initialize recommendations
list

5: for $\boldsymbol{p}_{i}\in\mathcal{P}^{u}$ and
$\boldsymbol{p}_{j}\in\mathcal{P}$ do $\triangleright$ Iterate over collection

6:
$S^{u}(\boldsymbol{p}_{i})=\frac{1}{n}\sum^{n}_{j=1}\omega^{u}_{j}\mathbf{A}_{ij}$
$\triangleright$ Bias similarity matrix towards user’s preferences

7: end for

8: Sort($S^{u}$) $\triangleright$ Descending sort, as higher score means
higher similarity

9: return Slice($S^{u},r$) $\triangleright$ Pick the first $r$ elements in the
ranking

10:end procedure

Algorithm 3 VA RecSys with popular recommendations (CLIP-P & BLIP-P).

1:Similarity matrix $\mathbf{A}$ of featurised paintings; Popularity score
$S^{\text{pop}}\,\forall p_{i}\in P$

2:procedure RecommendPaintings($\text{Model},P^{u},\omega^{u},\beta,r$)
$\triangleright$ Model $\in\\{\text{CLIP, BLIP}\\}$

3: $\mathcal{P}^{u}\leftarrow\textsc{FeaturizePaintings}(\text{Model},P^{u})$
$\triangleright$ Load computed embeddings, as $P^{u}\subset P$

4: $S^{u}_{AG}\leftarrow\varnothing$ $\triangleright$ Initialize
recommendations list

5: for $\boldsymbol{p}_{i}\in\mathcal{P}^{u}$ and
$\boldsymbol{p}_{j}\in\mathcal{P}$ do $\triangleright$ Iterate over collection

6:
$S^{u}(\boldsymbol{p}_{i})=\frac{1}{n}\sum^{n}_{j=1}\omega^{u}_{j}\mathbf{A}_{ij}$
$\triangleright$ Bias similarity matrix towards user’s preferences

7:
$S^{u}_{+}(\boldsymbol{p}_{i})=S^{u}(\boldsymbol{p}_{i})+\beta\;S^{\text{pop}}(\boldsymbol{p}_{i})$
$\triangleright$ Compute aggregated preferences score

8: end for

9: Sort($S^{u}_{+}$) $\triangleright$ Descending sort, as higher score means
higher relevance

10: return Slice($S^{u}_{+},r$) $\triangleright$ Pick the first $r$ elements
in the ranking

11:end procedure

Algorithm 4 VA RecSys with representative recommendations (CLIP-F & BLIP-F).

1:Similarity matrix $\mathbf{A}$ of featurised paintings

2:procedure
RecommendPaintings($\text{Model},P^{u},\omega^{u},\xi,\gamma_{p_{i}},K,r$)
$\triangleright$ Model $\in\\{\text{CLIP, BLIP}\\}$; $\forall p_{i}\in P$

3: $\mathcal{P}^{u}\leftarrow\textsc{FeaturizePaintings}(\text{Model},P^{u})$
$\triangleright$ Load computed embeddings, as $P^{u}\subset P$

4: $S^{u}\leftarrow\varnothing$ $\triangleright$ Initialize recommendations
list

5: for $\boldsymbol{p}_{i}\in\mathcal{P}^{u}$ and
$\boldsymbol{p}_{j}\in\mathcal{P}$ do $\triangleright$ Iterate over collection

6:
$S^{u}(\boldsymbol{p}_{i})=\frac{1}{n}\sum^{n}_{j=1}\omega^{u}_{j}\mathbf{A}_{ij}$
$\triangleright$ Bias similarity matrix towards user’s preferences

7: end for

8: $\mathcal{R}\leftarrow$ Solve
$\text{MIP}_{1}$($\text{Model},S^{u},P^{u},\omega^{u},\xi,\gamma_{p_{i}},K,r$)
$\triangleright$ Generate optimal recommendation set
$\mathcal{R}=\\{p_{1},\dots,p_{r}\\}$

9: return $\mathcal{R}$

10:end procedure

Algorithm 5 VA RecSys with popular and representative recommendations (CLIP-M
& BLIP-M).

1:Similarity matrix $\mathbf{A}$ of featurised paintings; Popularity score
$S^{\text{pop}}\,\forall p_{i}\in P$

2:procedure
RecommendPaintings($\text{Model},P^{u},\omega^{u},\beta,\xi,\gamma_{p_{i}},K,r$)
$\triangleright$ Model $\in\\{\text{CLIP, BLIP}\\}$; $\forall p_{i}\in P$

3: $\mathcal{P}^{u}\leftarrow\textsc{FeaturizePaintings}(\text{Model},P^{u})$
$\triangleright$ Load computed embeddings, as $P^{u}\subset P$

4: $S^{u}_{+}\leftarrow\varnothing$ $\triangleright$ Initialize
recommendations list

5: for $\boldsymbol{p}_{i}\in\mathcal{P}^{u}$ and
$\boldsymbol{p}_{j}\in\mathcal{P}$ do $\triangleright$ Iterate over collection

6: $S^{u}(p_{i})=\frac{1}{n}\sum^{n}_{j=1}\omega^{u}_{j}\mathbf{A}_{ij}$
$\triangleright$ Bias similarity matrix towards user’s preferences

7: $S^{u}_{+}(p_{i})=S^{u}(p_{i})+\beta\;S^{\text{pop}}(p_{i})$
$\triangleright$ Compute aggregate preferences score

8: end for

9: $\mathcal{R}\leftarrow$ Solve
$\text{MIP}_{2}$($\text{Model},S^{u}_{+},P^{u},\omega^{u},\xi,\gamma_{p_{i}},K,r$)
$\triangleright$ Generate optimal recommendation set
$\mathcal{R}=\\{p_{1},\dots,p_{r}\\}$

10: return $\mathcal{R}$

11:end procedure

###  4.6 Modality Gap

Multimodal models map inputs from different data modalities (e.g. image and
text) into a shared latent representation space [16, 15]. Although multi-modal
approaches like CLIP and BLIP have demonstrated their effectiveness on various
tasks, a recent study by Liang et al. [37] has shown the presence of a
modality gap on their representation spaces, where embeddings from different
modalities are located in completely separate regions of the embedding space.
Liang et al. also demonstrated that a variation in this gap is caused by a
combination of model initialisation and contrastive learning optimisation,
significantly affecting performance in downstream tasks.

This modality gap effect has been shown to be more prevalent on models using
only contrasting learning like CLIP as compared to BLIP, which optimises not
only contrastive loss (ITC) but also other objectives jointly (ITM and LM)
[37]. Hence, this provides supporting evidence for the fact that BLIP
outperforms CLIP on various downstream tasks [38], including VA RecSys [14].

######  Postulate 1.

We know that a reduced modality gap has a positive impact on downstream
performance [37] and that BLIP outperforms CLIP in VA RecSys tasks [14].
Therefore we expect all BLIP-based engines to achieve higher performance than
the ones using CLIP.

######  Postulate 2.

We expect Postulate 1 to be true unless the introduction of popularity and
representativeness compensate for the modality gap effect on VA RecSys tasks.

##  5 Offline evaluation

We conducted a series of experiments to assess the effectiveness of the three
proposed MOSAIC variants when it comes to seamlessly blending learning and
discovery with personal user preferences. Concretely, we measured the impact
of introducing popularity and representativeness (fair story selection)
strategies in both baseline VA RecSys engines CLIP and BLIP.

All algorithms were implemented using Python 3.9 with Gurobi
10.0444https://www.gurobi.com/ to solve the core MIP problems. All experiments
were run on a 3.10 GHz Intel(R) Xeon(R) Gold 6254 CPU with 4 cores and 16 GB
of RAM.

###  5.1 Dataset

We used a public dataset containing 2,368 paintings from The National Gallery,
London to train our backbone models BLIP and CLIP. This curated set of
paintings belongs to the CrossCult Knowledge Base.555https://www.crosscult.lu/
Each painting image is accompanied by a set of text-based metadata, which
makes this dataset suitable for multimodal feature learning approaches. A
sample data point is shown in Figure 3. For learning textual information, we
considered all available painting attributes such as artist name, painting
title, technique used, etc. combined with the description provided by museum
curators. The dataset also provides nine story groups, as previously mentioned
in Section 4.3, and 213 elicited user preferences, collected in previous
studies [12, 14] making it the most suitable dataset for this study as other
publicly available art datasets such as
Smithsonian666https://library.si.edu/data lack such user preference
information.

Figure 3: Sample painting and associated metadata from the National Gallery
dataset.

###  5.2 Procedure

We started by processing the aforementioned elicited user preferences in the
dataset. Users provided 5-star ratings for up to nine paintings each (one
painting per story group), that we normalized in the [0,1] range. Next, since
the dataset has no user preferences related to popularity and
representativeness, we simulated them. Specifically, from the users’
perspective, popularity is defined as the tolerance (or willingness) to
receiving popular paintings, whereas representativeness is defined as the
tolerance to receiving diverse paintings, that are fairly selected from the
curated story groups. Then, tolerance to both popular and diverse
(representative selection) of paintings is simulated by sampling from a
Poisson distribution. We simulated three scenarios, using mean rates of $0$,
$0.5$, and $1$, where 0 denotes no tolerance and 1 denotes maximal tolerance.

These simulated tolerances are the values we use for $\beta$ and $\xi$ in the
joint optimisation objectives, as a user-provided hyperparameter discussed in
Section 4. This way, we can anticipate the performance of MOSAIC under
different use case scenarios. For example, $\beta=1$ means that the user is
highly interested in receiving popular painting recommendations. Thus, more
weight is given to the second half of the objective function in Equation 2 to
promote the recommendation of popular paintings. Similarly, $\xi=0$ means that
a user is not interested in receiving diverse recommendations, thus the first
half of the objective function in Equation 7 and Equation 8 will be optimised
to promote the consistency of recommendations with user preferences instead of
promoting a representative selection of paintings.

###  5.3 Results

We compute the Jaccard index (Intersection over Union, IoU) and Rank-Biased
Overlap (RBO) in a pairwise fashion between the generated recommendations from
each MOSAIC variant. Both IoU and RBO are widely used measures in information
retrieval [39] as they complement each other: IoU measures differences between
sets, whereas RBO measures differences between rankings.777The recommendations
are assessed holistically, therefore IoU is an informative measure to our
study.However, the recommendations are actually ranked, therefore RBO is also
an informative measure to report. This assessment of MOSAIC is particularly
relevant to see the influence of introducing popularity (policy 2) and
representative selection (policy 3) on personalised recommendations (policy
1). Tables 2 and 2 summarise the results reporting the mean values of IoU and
RBO along with their standard deviation (SD), which gives an overview about
the dispersion of the measures.

| CLIP |  CLIP-$\text{P}_{0}$ |  CLIP-$\text{P}_{0.5}$ |  CLIP-$\text{P}_{1}$ |  CLIP-$\text{F}_{0}$ |  CLIP-$\text{F}_{0.5}$ |  CLIP-$\text{F}_{1}$ |  CLIP-$\text{P}_{0}\,\text{F}_{0}$ |  CLIP-$\text{P}_{0.5}\,\text{F}_{0.5}$ |  CLIP-$\text{P}_{1}\,\text{F}_{1}$ |  CLIP-$\text{P}_{0}\,\text{F}_{1}$ |  CLIP-$\text{P}_{1}\,\text{F}_{0}$
---|---|---|---|---|---|---|---|---|---|---|---|---
CLIP |  | 0.01 $\pm$ 0.38 | 0.01 $\pm$ 0.08 | 0.01 $\pm$ 0.38 | 0.63 $\pm$ 0.28 | 0.85 $\pm$ 0.24 | 0.97 $\pm$ 0.12 | 0.65 $\pm$ 0.25 | 0.87 $\pm$ 0.20 | 0.95 $\pm$ 0.12 | 0.95 $\pm$ 0.12 | 0.64 $\pm$ 0.28
CLIP-$\text{P}_{0}$ | 0.01 $\pm$ 0.02 |  | 0.99 $\pm$ 0.07 | 1.00 $\pm$ 0.00 | 0.01 $\pm$ 0.03 | 0.01 $\pm$ 0.03 | 0.01 $\pm$ 0.03 | 0.01 $\pm$ 0.03 | 0.01 $\pm$ 0.04 | 0.01 $\pm$ 0.03 | 0.01 $\pm$ 0.03 | 0.01 $\pm$ 0.03
CLIP-$\text{P}_{0.5}$ | 0.01 $\pm$ 0.02 | 0.99 $\pm$ 0.07 |  | 0.99 $\pm$ 0.06 | 0.01 $\pm$ 0.06 | 0.01 $\pm$ 0.05 | 0.01 $\pm$ 0.08 | 0.01 $\pm$ 0.03 | 1.00 $\pm$ 0.06 | 0.01 $\pm$ 0.08 | 0.01 $\pm$ 0.06 | 0.01 $\pm$ 0.05
CLIP-$\text{P}_{1}$ | 0.01 $\pm$ 0.04 | 1.00 $\pm$ 0.00 | 0.99 $\pm$ 0.06 |  | 0.01 $\pm$ 0.03 | 0.01 $\pm$ 0.03 | 0.01 $\pm$ 0.04 | 0.01 $\pm$ 0.04 | 0.01 $\pm$ 0.04 | 0.01 $\pm$ 0.04 | 0.01 $\pm$ 0.04 | 0.01 $\pm$ 0.04
CLIP-$\text{F}_{0}$ | 0.51 $\pm$ 0.29 | 0.01 $\pm$ 0.02 | 0.01 $\pm$ 0.05 | 0.04 $\pm$ 0.02 |  | 0.67 $\pm$ 0.24 | 0.72 $\pm$ 0.25 | 0.82 $\pm$ 0.21 | 0.71 $\pm$ 0.26 | 0.66 $\pm$ 0.28 | 0.66 $\pm$ 0.27 | 0.82 $\pm$ 0.20
CLIP-$\text{F}_{0.5}$ | 0.79 $\pm$ 0.30 | 0.01 $\pm$ 0.02 | 0.01 $\pm$ 0.07 | 0.04 $\pm$ 0.02 | 0.52 $\pm$ 0.32 |  | 0.94 $\pm$ 0.12 | 0.54 $\pm$ 0.23 | 0.94 $\pm$ 0.12 | 1.00 $\pm$ 0.00 | 0.94 $\pm$ 0.12 | 0.54 $\pm$ 0.23
CLIP-$\text{F}_{1}$ | 0.95 $\pm$ 0.15 | 0.01 $\pm$ 0.02 | 0.01 $\pm$ 0.06 | 0.01 $\pm$ 0.02 | 0.60 $\pm$ 0.30 | 0.91 $\pm$ 0.17 |  | 0.85 $\pm$ 0.24 | 0.69 $\pm$ 0.26 | 0.80 $\pm$ 0.24 | 0.85 $\pm$ 0.23 | 0.69 $\pm$ 0.26
CLIP-$\text{P}_{0}\,\text{F}_{0}$ | 0.52 $\pm$ 0.30 | 0.01 $\pm$ 0.02 | 0.01 $\pm$ 0.03 | 0.01 $\pm$ 0.02 | 0.73 $\pm$ 0.28 | 0.79 $\pm$ 0.30 | 0.39 $\pm$ 0.29 |  | 0.72 $\pm$ 0.25 | 0.68 $\pm$ 0.28 | 0.68 $\pm$ 0.25 | 0.83 $\pm$ 0.20
CLIP-$\text{P}_{0.5}\,\text{F}_{0.5}$ | 0.80 $\pm$ 0.28 | 0.01 $\pm$ 0.03 | 0.01 $\pm$ 0.08 | 0.01 $\pm$ 0.02 | 0.59 $\pm$ 0.31 | 0.58 $\pm$ 0.29 | 0.60 $\pm$ 0.30 | 0.40 $\pm$ 0.29 |  | 0.86 $\pm$ 0.19 | 0.85 $\pm$ 0.20 | 0.71 $\pm$ 0.26
CLIP-$\text{P}_{1}\,\text{F}_{1}$ | 0.93 $\pm$ 0.17 | 0.01 $\pm$ 0.02 | 0.01 $\pm$ 0.04 | 0.01 $\pm$ 0.02 | 0.53 $\pm$ 0.31 | 0.72 $\pm$ 0.31 | 0.56 $\pm$ 0.31 | 0.39 $\pm$ 0.29 | 0.80 $\pm$ 0.27 |  | 0.92 $\pm$ 0.16 | 0.66 $\pm$ 0.27
CLIP-$\text{P}_{0}\,\text{F}_{1}$ | 0.95 $\pm$ 0.14 | 0.01 $\pm$ 0.02 | 0.01 $\pm$ 0.06 | 0.01 $\pm$ 0.02 | 0.54 $\pm$ 0.32 | 0.77 $\pm$ 0.29 | 0.56 $\pm$ 0.31 | 0.57 $\pm$ 0.31 | 0.89 $\pm$ 0.23 | 1.00 $\pm$ 0.00 |  | 0.66 $\pm$ 0.27
CLIP-$\text{P}_{1}\,\text{F}_{0}$ | 0.52 $\pm$ 0.31 | 0.01 $\pm$ 0.03 | 0.01 $\pm$ 0.08 | 0.01 $\pm$ 0.02 | 0.73 $\pm$ 0.27 | 0.58 $\pm$ 0.31 | 0.74 $\pm$ 0.26 | 0.74 $\pm$ 0.26 | 0.59 $\pm$ 0.31 | 0.54 $\pm$ 0.32 | 0.54 $\pm$ 0.31 |

Table 1: Offline evaluation of CLIP-based MOSAIC engines. We report Mean $\pm$
SD of the following pairwise ranking similarity measures: Jaccard index (IoU)
and Rank-biased overlap (RBO)  .

| BLIP |  BLIP-$\text{P}_{0}$ |  BLIP-$\text{P}_{0.5}$ |  BLIP-$\text{P}_{1}$ |  BLIP-$\text{F}_{0}$ |  BLIP-$\text{F}_{0.5}$ |  BLIP-$\text{F}_{1}$ |  BLIP-$\text{P}_{0}\,\text{F}_{0}$ |  BLIP-$\text{P}_{0.5}\,\text{F}_{0.5}$ |  BLIP-$\text{P}_{1}\,\text{F}_{1}$ |  BLIP-$\text{P}_{0}\,\text{F}_{1}$ |  BLIP-$\text{P}_{1}\,\text{F}_{0}$
---|---|---|---|---|---|---|---|---|---|---|---|---
BLIP |  | 0.16 $\pm$ 0.32 | 0.04 $\pm$ 0.16 | 0.03 $\pm$ 0.13 | 0.99 $\pm$ 0.03 | 0.99 $\pm$ 0.03 | 0.99 $\pm$ 0.03 | 0.99 $\pm$ 0.03 | 0.99 $\pm$ 0.03 | 0.99 $\pm$ 0.03 | 0.99 $\pm$ 0.05 | 0.99 $\pm$ 0.03
BLIP-$\text{P}_{0}$ | 0.16 $\pm$ 0.30 |  | 0.83 $\pm$ 0.32 | 0.84 $\pm$ 0.33 | 0.16 $\pm$ 0.32 | 0.16 $\pm$ 0.32 | 0.16 $\pm$ 0.32 | 0.16 $\pm$ 0.32 | 0.16 $\pm$ 0.32 | 0.16 $\pm$ 0.32 | 0.16 $\pm$ 0.32 | 0.16 $\pm$ 0.30
BLIP-$\text{P}_{0.5}$ | 0.04 $\pm$ 0.15 | 0.74 $\pm$ 0.37 |  | 0.94 $\pm$ 0.18 | 0.04 $\pm$ 0.14 | 0.04 $\pm$ 0.14 | 0.04 $\pm$ 0.14 | 0.04 $\pm$ 0.14 | 0.04 $\pm$ 0.14 | 0.04 $\pm$ 0.14 | 0.04 $\pm$ 0.14 | 0.04 $\pm$ 0.14
BLIP-$\text{P}_{1}$ | 0.02 $\pm$ 0.12 | 0.79 $\pm$ 0.36 | 0.91 $\pm$ 0.23 |  | 0.03 $\pm$ 0.13 | 0.03 $\pm$ 0.13 | 0.03 $\pm$ 0.13 | 0.03 $\pm$ 0.13 | 0.03 $\pm$ 0.12 | 0.03 $\pm$ 0.13 | 0.03 $\pm$ 0.13 | 0.03 $\pm$ 0.12
BLIP-$\text{F}_{0}$ | 0.99 $\pm$ 0.03 | 0.16 $\pm$ 0.30 | 0.04 $\pm$ 0.15 | 0.03 $\pm$ 0.12 |  | 0.99 $\pm$ 0.00 | 0.99 $\pm$ 0.01 | 1.00 $\pm$ 0.00 | 0.99 $\pm$ 0.07 | 0.99 $\pm$ 0.07 | 0.99 $\pm$ 0.07 | 1.00 $\pm$ 0.00
BLIP-$\text{F}_{0.5}$ | 0.99 $\pm$ 0.03 | 0.16 $\pm$ 0.30 | 0.04 $\pm$ 0.15 | 0.03 $\pm$ 0.12 | 0.99 $\pm$ 0.02 |  | 0.99 $\pm$ 0.01 | 0.99 $\pm$ 0.01 | 0.98 $\pm$ 0.00 | 0.99 $\pm$ 0.01 | 0.99 $\pm$ 0.05 | 0.99 $\pm$ 0.01
BLIP-$\text{F}_{1}$ | 0.99 $\pm$ 0.03 | 0.16 $\pm$ 0.30 | 0.04 $\pm$ 0.15 | 0.03 $\pm$ 0.12 | 0.99 $\pm$ 0.04 | 0.98 $\pm$ 0.05 |  | 0.99 $\pm$ 0.01 | 0.99 $\pm$ 0.01 | 0.99 $\pm$ 0.01 | 0.99 $\pm$ 0.03 | 0.99 $\pm$ 0.01
BLIP-$\text{P}_{0}\,\text{F}_{0}$ | 0.99 $\pm$ 0.03 | 0.16 $\pm$ 0.30 | 0.04 $\pm$ 0.15 | 0.03 $\pm$ 0.12 | 1.00 $\pm$ 0.00 | 0.99 $\pm$ 0.02 | 0.99 $\pm$ 0.04 |  | 0.99 $\pm$ 0.00 | 0.99 $\pm$ 0.00 | 0.99 $\pm$ 0.03 | 1.00 $\pm$ 0.00
BLIP-$\text{P}_{0.5}\,\text{F}_{0.5}$ | 0.99 $\pm$ 0.03 | 0.16 $\pm$ 0.30 | 0.04 $\pm$ 0.15 | 0.03 $\pm$ 0.12 | 0.99 $\pm$ 0.07 | 0.99 $\pm$ 0.04 | 0.99 $\pm$ 0.04 | 0.99 $\pm$ 0.01 |  | 0.99 $\pm$ 0.00 | 0.99 $\pm$ 0.03 | 0.99 $\pm$ 0.00
BLIP-$\text{P}_{1}\,\text{F}_{1}$ | 0.99 $\pm$ 0.03 | 0.16 $\pm$ 0.30 | 0.04 $\pm$ 0.15 | 0.03 $\pm$ 0.12 | 0.99 $\pm$ 0.07 | 0.99 $\pm$ 0.05 | 0.98 $\pm$ 0.03 | 0.99 $\pm$ 0.07 | 0.99 $\pm$ 0.03 |  | 0.99 $\pm$ 0.03 | 0.99 $\pm$ 0.00
BLIP-$\text{P}_{0}\,\text{F}_{1}$ | 0.98 $\pm$ 0.08 | 0.16 $\pm$ 0.30 | 0.04 $\pm$ 0.15 | 0.03 $\pm$ 0.12 | 0.99 $\pm$ 0.07 | 0.99 $\pm$ 0.05 | 0.98 $\pm$ 0.08 | 0.99 $\pm$ 0.07 | 0.99 $\pm$ 0.08 | 0.98 $\pm$ 0.08 |  | 0.99 $\pm$ 0.03
BLIP-$\text{P}_{1}\,\text{F}_{0}$ | 0.99 $\pm$ 0.02 | 0.16 $\pm$ 0.30 | 0.04 $\pm$ 0.15 | 0.03 $\pm$ 0.12 | 1.00 $\pm$ 0.00 | 0.99 $\pm$ 0.05 | 0.99 $\pm$ 0.07 | 1.00 $\pm$ 0.00 | 0.99 $\pm$ 0.01 | 0.99 $\pm$ 0.03 | 0.98 $\pm$ 0.08 |

Table 2: Offline evaluation of BLIP-based MOSAIC engines. We report Mean $\pm$
SD of the following pairwise ranking similarity measures: rank-biased overlap
(RBO)  and Jaccard index (IoU)  .

Our results indicate that the popularity policy has a strong impact on the
recommendations, as in all cases, the ranking similarity measures are close to
$0$ with regard to other engines, denoting no similarity and thus that all
recommendations are different. On the contrary, the representative selection
policy has a rather moderate impact, and it is only apparent when the value of
$\xi$ is very low, i.e., when the user tolerance to seeing diverse paintings
is minimal. For users with high tolerance to diversity (i.e., $\xi\approx 1$)
both CLIP-$\text{F}_{1}$ and BLIP-$\text{F}_{1}$ produced very similar
recommendations to their respective baseline engines. Then, as $\xi$
decreases, the ranking similarity values tend to decrease as well, but only
slightly and still, on average, more than 50% of the recommendations were
found to be similar; see e.g. CLIP-$\text{F}_{0.5}$ and CLIP-$\text{F}_{0}$. A
more extreme case was observed for BLIP-$\text{F}_{0.5}$ and
BLIP-$\text{F}_{0}$, where ranking similarity remained consistently high,
close to 1.

We can also observe similar results for the full-version of MOSAIC.
Particularly, while the effect of popularity was found to be strong when the
values of $\xi$ are lower, this engine version produces similar
recommendations than the baseline versions when the values of $\xi$ are
higher. Hence, from these observations we can assume that the embeddings
produced by the baseline multimodal engines (CLIP and BLIP) are already
promoting paintings fairly across the story groups in the analysed paintings
collection.

We performed an additional rank similarity analysis between CLIP and BLIP
variants. We observed no overlap in both IoU and RBO values, all being very
close to $0$. This suggests that the recommendations generated with each of
the multimodal backbone architectures are completely different, even when
considering the three different policies we have studied. From previous work,
we know that and BLIP performs significantly better than CLIP on various
downstream tasks, including VA RecSys, which is attributed to the modality gap
effect [37, 14]. Thus, as stipulated in Postulates 1 and 2, we expect that
BLIP-based recommendations would be perceived as being of higher quality for
users. We expect the contrary to happen with the introduction of the
popularity factor, which causes the recommendations to be completely different
from the baseline engines, thereby compensating for the aforementioned
modality gap effect.

Overall, from this offline evaluation we discovered that the effect of
representative selection (policy 3) is minimal but slightly higher on CLIP
than on BLIP engines, and that the effect of popularity (policy 2) is strong
for both baseline engines. Therefore, to assess these observed effects on the
quality of recommendations from a user-centric perspective, we conducted a
user study described in the following section.

##  6 User study

We conducted a subsequent user study to understand the users’ perception
towards the quality of our MOSAIC engines. The study was approved by the
Ethics Review Panel of the University of Luxembourg with ID 22-031.

Building upon our offline evaluation, we know that representativeness has
little impact on the baseline engines. We also know that the recommendations
provided by the full-version of MOSAIC and the popularity-based engines are
substantially different, but it is unclear how they would be perceived by end
users. Therefore, a user study is ultimately required. Hence, based on the
results from the offline evaluation, we selected the four engines that
generated substantially different recommendations for our user study: CLIP-M
and BLIP-M (reflecting the combined effect of popularity and
representativeness) as well as the popularity-only variants CLIP-P and BLIP-P.

###  6.1 Dataset

We used the same paintings dataset as in the offline evaluation. Please refer
to Section 5.1 for more details.

###  6.2 Apparatus

We created a web application that collected preference elicitation ratings
from users, together with their tolerance to receiving popular and diverse
paintings. The application then retrieved a set of VA recommendations that
users had to assess. As shown in Figure 4, participants were provided with one
set of recommendations from each VA RecSys engine at a time.

Figure 4:  Screenshot of our web application for evaluation.

###  6.3 Participants

We recruited a representative pool of $N=100$ crowdworkers via the Prolific
crowdsourcing platform.888https://www.prolific.co/, according to the following
eligibility criteria: being fluent in English, art is listed among their
interests/hobbies, minimum approval rate of 100% in previous crowdsourcing
studies in the platform, and being active for over one year in the platform.

Our recruited participants (66 female, 34 male) were aged 27.24 years
(SD=7.75) and could complete the study only once. Most of them had Portuguese
(22%), South African (20%), or Mexican (14%) nationality. The study took 4.9
min on average to complete (SD=3.8) and participants were paid an equivalent
hourly wage of $15/h.

###  6.4 Procedure

Participants accessed our web application and entered their demographic
information (age, gender) on a welcome screen. There, they were informed about
the purpose of the study and the data collection policy. Participants also
indicated their tolerance to receiving popular and diverse paintings, both in
a 1–5 scale that was later normalised to the [0,1] range according to MOSAIC’s
formulation. Lower values denote less user tolerance.

Then, participants advanced to the preference elicitation screen, where they
were shown one painting at random from each of the nine curated story groups.
They rated each painting in a 5-point numerical scale (5 is better, i.e. the
user likes the painting the most). Finally, users advanced to the assessment
screen, where they were shown a set of nine painting recommendations drawn
from each VA RecSys engine (the order of the engines was randomized for each
user) and were asked to rate each recommended set in a 5-point Likert scale
(1: Strongly disagree, …, 5: Strongly agree).

###  6.5 Design

Participants were exposed to all engines once (within-subjects design). Our
dependent variables are widely accepted proxies of recommendation quality
[40]:

Accuracy:

The paintings match my personal preferences and interests.

Diversity:

The paintings are diverse.

Novelty:

I discovered paintings I did not know before.

Serendipity:

I found surprisingly interesting paintings.

We duplicated one of the engines (chosen at random for each user) to serve as
an attention check and to ensure a high intra-rater agreement. That is, if a
user sees the same set of recommended paintings twice, they should rate it as
much similar as before. The attention check was shown at random for each user.
Based on this rationale, we excluded 5 participants who deviated for more than
2 points in at least one of the four dependent variables.

###  6.6 Results

We investigated whether there were differences between any of the four studied
MOSAIC engines, for which we used a linear mixed-effects (LME) model where
each dependent variable is explained by each engine. User’s tolerance to
popular and diverse paintings are considered interaction effects (model
covariates), and participants are considered random effects. An LME model is
appropriate here because the dependent variables are discrete and have a
natural order. In addition, LME models are quite robust to violations of
several distributional assumptions [41].

We fitted the LME models (one model per dependent variable) and computed the
estimated marginal means for specified factors. We then ran pairwise
comparisons (also known as _contrasts_) with Bonferroni-Holm correction to
guard against multiple comparisons.

Figure 5 shows the distributions of user ratings. We report below statistical
significance ($p$-values) and effect sises ($r$) to better gauge the
differences between our four MOSAIC engines.

(a) Accuracy (b) Diversity (c) Novelty (d) Serendipity

Figure 5: Distribution (box plots) of user ratings. Dots denote mean values.

#### Accuracy analysis

Differences were statistically significant between BLIP-P and BLIP-M
($p<.01,r=0.18$) and between CLIP-P and BLIP-M ($p<.01,r=0.18$). Differences
between CLIP-M and BLIP-M were also statistically significant
($p<.01,r=0.21$). All other comparisons were not found to be statistically
significant. Effect sizes suggest a small to moderate practical importance of
the results.

#### Diversity analysis

No statistically significant differences were found between engines ($p>.05$).
The largest effect size was $r=0.13$ when comparing CLIP-M and BLIP-M,
suggesting therefore a small practical importance of the results.

#### Novelty analysis

Differences between BLIP-P and all the other engines were found to be
statistically significant, i.e., BLIP-P vs. CLIP-P ($p<.01,r=0.17$), BLIP-P
vs. BLIP-M ($p<.001,r=0.21$), and BLIP-P vs. CLIP-M ($p<.001,r=0.22$). All
other comparisons were not found to be statistically significant. Effect sizes
suggest a small to moderate practical importance of the results.

#### Serendipity analysis

Differences were statistically significant between BLIP-P and BLIP-M
($p<.05,r=0.15$) and between CLIP-M and BLIP-M ($p<.05,r=0.16$). All other
comparisons were not found to be statistically significant. Effect sizes
suggest a small practical importance.

#### Ranking similarity analysis

We conducted an additional analysis that checked whether our participants did
receive truly personalised recommendations. For this, we computed RBO and IoU
in a pairwise manner among all users exposed to the same engine. Tables 3 and
4 present the results of this analysis. As shown in the tables, there is
little overlap in the rankings produced by each engine. This analysis thus
indicates that each participant indeed was shown a personalisd set of
recommendations and that each engine produced very different rankings.

Table 3: Within-ranking overlap results, showing Mean $\pm$ SD of IoU and RBO measures. | CLIP-P | BLIP-P | CLIP-M | BLIP-M
---|---|---|---|---
IoU |  0.12 $\pm$ 0.76 |  0.01 $\pm$ 0.07 |  0.25 $\pm$ 0.26 |  0.25 $\pm$ 0.18
RBO |  0.01 $\pm$ 0.07 |  0.01 $\pm$ 0.07 |  0.18 $\pm$ 0.17 |  0.28 $\pm$ 0.18
Table 4: Between-ranking overlap results, showing Mean $\pm$ SD of IoU  and RBO  measures.  | CLIP-P | BLIP-P | CLIP-M | BLIP-M
---|---|---|---|---
CLIP-P |  | 0.01 $\pm$ 0.16 | 0.01 $\pm$ 0.04 | 0.00 $\pm$ 0.01
BLIP-P | 0.01 $\pm$ 0.01 |  | 0.00 $\pm$ 0.01 | 0.00 $\pm$ 0.00
CLIP-M | 0.01 $\pm$ 0.01 | 0.00 $\pm$ 0.01 |  | 0.00 $\pm$ 0.01
BLIP-M | 0.00 $\pm$ 0.01 | 0.00 $\pm$ 0.01 | 0.00 $\pm$ 0.01 |

In our user study, participants indicated a high tolerance to receiving
diverse recommendations, with a mean $\xi=0.74$ (SD=0.26). Only 3 participants
indicated $\xi=0$. Consequently, the implementation of a representative
selection policy yielded minimal impact overall, as seen in Figure 5. This is
consistent with our offline evaluation (Section 5.3), where the impact of
representative selection was found to be moderate and only apparent when the
value of $\xi$ is close to zero, i.e., denoting minimal user tolerance to
seeing diverse paintings. This was also reflected on the actual diversity of
the recommendations, measured as the median number of different story groups
covered per set of 9 paintings (BLIP-M = 7, CLIP-M = 6, CLIP-P = 5, BLIP-P =
6). While BLIP-M produced the most diverse recommendations overall, the
differences between engines are not statistically significant ($p>.05$).
Overall, from this user study, we can conclude that users appreciated the
inclusion of the popularity policy in MOSAIC, indicating that other
stakeholders such as collectors or art enthusiasts play an important role in
the curation of VA content. According to the four user-centric measures
considered, BLIP-M was consistently perceived as the top-performer VA RecSys
engine, followed by BLIP-P. We can conclude therefore that BLIP is the most
adequate backbone engine for MOSAIC. In the following section we discuss the
implications of our findings.

##  7 Discussion

We reflect on the implications of our findings and derive valuable insights on
how to design next-generation VA RecSys engines blending learning, discovery,
and user satisfaction.

###  7.1 A Palette of Possibilities: From Personalisation to Learning and
Discovery

VA RecSys have traditionally focused on providing users with personalised
recommendations based on their past preferences and behaviour. However, as
these systems continue to evolve, there is a growing realisation that they
have the potential to offer much more. Considering the vast and dynamic status
quo in the art domain, by jointly optimising for other objectives, such as
learning and discovery, VA RecSys can create a palette of possibilities that
goes beyond simply catering to users’ existing tastes. This transition from
pure personalisation to learning and discovery is important for several
reasons. First, it enables users to expand their knowledge and appreciation of
VA beyond their existing preferences. Second, it also enables VA RecSys to
serve as a tool for lifelong learning and personal growth, rather than simply
a means of entertainment. Further, by discovering new art styles and genres,
users can gain a deeper understanding of the cultural and historical contexts
in which different art forms originated, and broaden their aesthetic horizons.
This can be especially important for users who may not have had much exposure
to the arts in the past, and who may not be aware of the rich and diverse
range of artistic traditions that exist around the world.

Furthermore, the transition from pure personalisation to learning and
discovery also has significant implications for the development of VA RecSys.
By optimising these systems to serve a broader range of objectives, developers
can create a more flexible and adaptable framework that can be applied to a
wide range of use cases. For example, VA RecSys could be used in educational
contexts to help students learn about different art styles and movements, or
in museum settings to provide visitors with a more personalised and engaging
tour experience. By expanding these possibilities beyond pure personalisation,
we can unlock a whole new range of applications and use cases. Our results
indicate that departing from solely optimising for user preferences to factor
in additional objectives such as paintings’ popularity have managed to strike
a balance across all beyond-accuracy measures. Particularly, BLIP-P was mostly
preferred while the other multistakeholder-aware engines also performed
similarly, thus opening a new avenue for designing innovative policies to be
jointly optimised.

###  7.2 Into the Canvas: Multimodality and Semantics

Recent research has focused on advanced VA representation learning techniques
to capture the semantic relatedness of artworks [12, 29, 10]. This is crucial
for VA RecSys to derive recommendations that preserve some notion of art
semantics. In this regard, multimodal data, which includes both visual and
textual information, has become increasingly important. By analysing both
image content and accompanying text, VA RecSys can better understand the
context and meaning of an artwork, which in turn allows for more relevant
recommendations. Multimodal representation learning techniques have shown
great potential in terms of capturing latent semantics that are not obvious to
the naked eye.

Particularly the state-of-the-art architectures CLIP and BLIP have undoubtedly
demonstrated notable success in VA RecSys. However, a pressing question looms:
to what extent and depth do they truly capture the rich semantics embedded
within visual art?

Especially since the semantics of artwork is a broader notion that
encapsulates various perspectives from an array of stakeholders in the art
ecosystem and concepts beyond mere aesthetic elements depicted in paintings or
written in their textual descriptions. Our user study shows that introducing
the elements of popularity or representativeness to either CLIP or BLIP
achieved results on par with their respective baselines. This signifies that,
while multimodal approaches present a promising foundation, it becomes evident
that comprehending the semantic depth of artwork transcends the modalities and
therefore requires looking “beyond the canvas”. This opens up exciting
research avenues for designing more comprehensive VA RecSys policies that not
only accommodate the diverse perspectives of relevant stakeholders but also
pave the way for a deeper understanding of the intricate world of visual arts.

Furthermore, in our user study, we observed that BLIP-P was preferred over
CLIP-P and BLIP-M was preferred over CLIP-M, confirming that recommendations
powered by BLIP are perceived to be better than CLIP. This indicates that the
two backbone architectures allow different levels of exposition to VA
semantics. This adheres to the findings about the modality gap effect reported
in previous works, which validates Postulate 1. Additionally, CLIP-P
outperformed CLIP-M which produced similar recommendations with the baseline
CLIP as informed by our offline evaluation, suggesting that the popularity
policy indeed compensated for the modality gap effect. However, it was not
enough to outperform BLIP therefore partially validating Postulate 2.
Nonetheless, our results indicate a potential for addressing the modality gap
effect in CLIP by introducing joint optimisation objectives.

###  7.3 Beyond the Canvas: Navigating the Art Landscape

While VA RecSys is primarily intended to personalise recommendations to
individual users, there has been a missed opportunity to navigate the larger
art ecosystem which includes a number of stakeholders. For example, art
collectors, who are often interested in acquiring unique or rare pieces of
art, can drive up the demand and value of certain works, thereby making
certain artists or artworks more popular. Popularity is therefore an important
aspect in VA RecSys, which has not been sufficiently explored in the research
literature. The results from our user study indicate that factoring in
popularity was perceived by users to improve the quality of recommendations,
not only in terms of accuracy but also in terms of serendipity and diversity.
This highlights the need for further investigation in this regard, especially
since there are many factors and stakeholders that contribute to the
popularity of a painting. Furthermore, their historical significance, artistic
merit, and cultural relevance also play a vital role. In this work, we
conceptualise popularity as an influential factor shaping user preferences,
driven by crowd behaviour, which represents a significant stakeholder
contributing substantially to the context of VA RecSys. Future work can also
explore integrating various crowd objectives and facets into the MOSAIC
framework, enhancing the framework’s ability to efficiently represent
stakeholders. Hence, we are confident that our findings will highlight the
relevance of multistakeholder-awareness in VA RecSys.

###  7.4 Looking Ahead: Limitations and Future work

One limitation of our experiments is that, despite the large number of
participants, we have considered a relatively small set of elicited
preferences (ratings of only 9 paintings per user). Preference elicitation is
a longstanding challenge in RecSys, as collecting more user preferences
usually improves recommendation quality, but this is not always feasible or
accessible. Although our results confirm the relevance of MOSAIC in tackling
the elicitation challenge, more studies on VA RecSys performance with varying
preference data are needed. Another limitation is that we only considered two
stakeholders to optimise for, in addition to the user preferences, and there
may be others to cater for learning and discovery. This is nonetheless an
exciting opportunity for future work.

Our experiments show that using multimodal data leads to significant
improvements in recommendation quality and that the effect of the modality gap
in CLIP can potentially be minimised by introducing joint optimisation
policies. However, further research is needed to determine the extent to which
these findings will be observed in other domains. Future work should also
consider more stakeholders, in order to provide a more nuanced understanding
of the art ecosystem.

##  8 Conclusion

We have explored the challenging task of visual art recommendation, taking
into account subjective user experiences as well as the input of relevant
stakeholders such as museum curators and crowds of visitors. We have proposed
MOSAIC, a novel multimodal multistakeholder-aware approach that goes beyond
traditional accuracy measures and promotes user-centric measures such as
novelty, serendipity, and diversity. Our results showed that departing from
solely optimising for user preferences to factor in multistakeholder awareness
has a positive impact in promoting learning and discovery. These findings
suggest that MOSAIC has the potential to add value not only for users, but
also for other stakeholders, including artists, curators, and art
organizations. As the field of VA RecSys continues to evolve, MOSAIC
represents the first cornerstone to navigate the complex landscape of the art
ecosystem.

Acknowledgements This work is supported by the Horizon 2020 FET program of the
European Union through the ERA-NET Cofund funding (project BANANA, grant
CHIST-ERA-20-BCI-001) and Horizon Europe’s European Innovation Council through
the Pathfinder program (SYMBIOTIK project, grant 101071147).

## References

  *   * Mayer and Smith [1973] Mayer, R., Smith, E.: The Artist’s Handbook of Materials and Techniques, 3rd ed. revised and expanded / edited by edwin smith edn. Faber, ??? (1973). https://cir.nii.ac.jp/crid/1130282273074769280
  * Naudet et al. [2015] Naudet, Y., Antoniou, A., Lykourentzou, I., Tobias, E., Rompa, J., Lepouras, G.: Museum personalization based on gaming and cognitive styles: the blue experiment. International Journal of Virtual Communities and Social Networking (IJVCSN) 7(2), 1–30 (2015)
  * Yilma et al. [2021] Yilma, B.A., Naudet, Y., Panetto, H.: Personalisation in cyber-physical-social systems: A multi-stakeholder aware recommendation and guidance. In: Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization. UMAP ’21, pp. 251–255. Association for Computing Machinery, New York, NY, USA (2021). https://doi.org/10.1145/3450613.3456847 . https://doi.org/10.1145/3450613.3456847
  * Li et al. [2023] Li, R., Moh, M., Moh, T.-S.: Siamese neural networks for content-based visual art recommendation. In: 2023 17th International Conference on Ubiquitous Information Management and Communication (IMCOM), pp. 1–8 (2023). IEEE
  * Wang and Gong [2023] Wang, S., Gong, S.: Research on oil painting recommendation algorithm based on collaborative filtering. In: 2023 IEEE 2nd International Conference on Electrical Engineering, Big Data and Algorithms (EEBDA), pp. 1369–1372 (2023). IEEE
  * Nobre and Sousa [2022] Nobre, H., Sousa, A.: Cultural heritage and nation branding–multi stakeholder perspectives from portugal. Journal of Tourism and Cultural Change 20(5), 699–717 (2022)
  * Palumbo et al. [2023] Palumbo, L., Harrison, N.R., Trawiński, T., Kass, J., Metelmann, A.C., Bari, R.S.G., Donnelly, N.: Visual exploration mediates the influence of personal traits on responses to artworks in an art gallery setting. Psychology of Aesthetics, Creativity, and the Arts (2023)
  * Zhitomirsky-Geffet et al. [2023] Zhitomirsky-Geffet, M., Kizhner, I., Minster, S.: What do they make us see: a comparative study of cultural bias in online databases of two large museums. Journal of Documentation 79(2), 320–340 (2023)
  * Messina et al. [2017] Messina, P., Dominguez, V., Parra, D., Trattner, C., Soto, A.: Exploring content-based artwork recommendation with metadata and visual features. arXiv preprint arXiv:1706.05786 (2017)
  * He et al. [2016] He, R., Fang, C., Wang, Z., McAuley, J.: Vista: a visually, socially, and temporally-aware model for artistic recommendation. In: Proceedings of the 10th ACM Conference on Recommender Systems, pp. 309–316 (2016)
  * Messina et al. [2019] Messina, P., Dominguez, V., Parra, D., Trattner, C., Soto, A.: Content-based artwork recommendation: integrating painting metadata with neural and manually-engineered visual features. User Modeling and User-Adapted Interaction 29(2), 251–290 (2019)
  * Yilma and Leiva [2023] Yilma, B.A., Leiva, L.A.: The elements of visual art recommendation: Learning latent semantic representations of paintings. In: Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23), April 23–28, 2023, Hamburg, Germany. ACM Press, ??? (2023). https://doi.org/10.1145/3544548.3581477
  * Lialin et al. [2023] Lialin, V., Rawls, S., Chan, D., Ghosh, S., Rumshisky, A., Hamza, W.: Scalable and accurate self-supervised multimodal representation learning without aligned video and text data. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 390–400 (2023)
  * Yilma and Leiva [2023] Yilma, B.A., Leiva, L.A.: Together yet apart: Multimodal representation learning for personalised visual art recommendation. In: Proceedings of the 31st ACM Conference on User Modeling, Adaptation and Personalization. UMAP ’23, pp. 204–214. Association for Computing Machinery, New York, NY, USA (2023). https://doi.org/10.1145/3565472.3592964 . https://doi.org/10.1145/3565472.3592964
  * Radford et al. [2021] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International Conference on Machine Learning, pp. 8748–8763 (2021). PMLR
  * Li et al. [2022] Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. arXiv preprint arXiv:2201.12086 (2022)
  * Chen et al. [2023] Chen, R., Liu, Y., Kong, L., Zhu, X., Ma, Y., Li, Y., Hou, Y., Qiao, Y., Wang, W.: Clip2scene: Towards label-efficient 3d scene understanding by clip. arXiv preprint arXiv:2301.04926 (2023)
  * Sain et al. [2023] Sain, A., Bhunia, A.K., Chowdhury, P.N., Koley, S., Xiang, T., Song, Y.-Z.: Clip for all things zero-shot sketch-based image retrieval, fine-grained or not. arXiv preprint arXiv:2303.13440 (2023)
  * Zhu et al. [2023] Zhu, D., Chen, J., Haydarov, K., Shen, X., Zhang, W., Elhoseiny, M.: Chatgpt asks, blip-2 answers: Automatic questioning towards enriched visual descriptions. arXiv preprint arXiv:2303.06594 (2023)
  * Kaminskas and Bridge [2016] Kaminskas, M., Bridge, D.: Diversity, serendipity, novelty, and coverage: A survey and empirical analysis of beyond-accuracy objectives in recommender systems. ACM Trans. Interact. Intell. Syst. 7(1) (2016)
  * Abdollahpouri et al. [2020] Abdollahpouri, H., Adomavicius, G., Burke, R., Guy, I., Jannach, D., Kamishima, T., Krasnodebski, J., Pizzato, L.: Multistakeholder recommendation: Survey and research directions. User Modeling and User-Adapted Interaction 30, 127–158 (2020)
  * Sürer et al. [2018] Sürer, Ö., Burke, R., Malthouse, E.C.: Multistakeholder recommendation with provider constraints. In: Proceedings of the 12th ACM Conference on Recommender Systems, pp. 54–62 (2018)
  * Abdollahpouri and Burke [2021] Abdollahpouri, H., Burke, R.: Multistakeholder recommender systems. In: Recommender Systems Handbook, pp. 647–677. Springer, ??? (2021)
  * McNee et al. [2006] McNee, S.M., Riedl, J., Konstan, J.A.: Being accurate is not enough: how accuracy metrics have hurt recommender systems. In: CHI’06 Extended Abstracts on Human Factors in Computing Systems, pp. 1097–1101 (2006)
  * Shi [2013] Shi, L.: Trading-off among accuracy, similarity, diversity, and long-tail: a graph-based recommendation approach. In: Proceedings of the 7th ACM Conference on Recommender Systems, pp. 57–64 (2013)
  * Isufi et al. [2021] Isufi, E., Pocchiari, M., Hanjalic, A.: Accuracy-diversity trade-off in recommender systems via graph convolutions. Information Processing & Management 58(2), 102459 (2021)
  * Vlachidis et al. [2017] Vlachidis, A., Bikakis, A., Kyriaki-Manessi, D., Triantafyllou, I., Antoniou, A.: The crosscult knowledge base: A co-inhabitant of cultural heritage ontology and vocabulary classification. In: Kirikova, M., Nørvåg, K., Papadopoulos, G.A., Gamper, J., Wrembel, R., Darmont, J., Rizzi, S. (eds.) New Trends in Databases and Information Systems, pp. 353–362. Springer, Cham (2017)
  * Wu et al. [2020] Wu, H., Liu, S.Y., Zheng, W., Yang, Y., Gao, H.: Paintkg: the painting knowledge graph using bilstm-crf. In: 2020 International Conference on Information Science and Education (ICISE-IE), pp. 412–417 (2020). IEEE
  * Messina et al. [2020] Messina, P., Cartagena, M., Cerda, P., Rio, F., Parra, D.: Curatornet: Visually-aware recommendation of art images (2020)
  * Han and Yamana [2017] Han, J., Yamana, H.: A survey on recommendation methods beyond accuracy. IEICE TRANSACTIONS on Information and Systems 100(12), 2931–2944 (2017)
  * Ramachandram and Taylor [2017] Ramachandram, D., Taylor, G.W.: Deep multimodal learning: A survey on recent advances and trends. IEEE Signal Process. Mag. 34(6), 96–108 (2017)
  * Li et al. [2021] Li, J., Selvaraju, R., Gotmare, A., Joty, S., Xiong, C., Hoi, S.C.H.: Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems 34, 9694–9705 (2021)
  * Yilma et al. [2024] Yilma, B.A., Kim, C.M., Cupchik, G.C., Leiva, L.A.: Artful path to healing: Using machine learning for visual art recommendation to prevent and reduce post-intensive care syndrome (pics). In: Proceedings of the CHI Conference on Human Factors in Computing Systems. CHI ’24. Association for Computing Machinery, New York, NY, USA (2024). https://doi.org/10.1145/3613904.3642636 . https://doi.org/10.1145/3613904.3642636
  * Mehrotra and Yilmaz [2015] Mehrotra, R., Yilmaz, E.: Representative & informative query selection for learning to rank using submodular functions. In: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 545–554 (2015)
  * Kontiza et al. [2018] Kontiza, K., Loboda, O., Deladiennee, L., Castagnos, S., Naudet, Y.: A museum app to trigger users’ reflection. In: International Workshop on Mobile Access to Cultural Heritage (MobileCH2018) (2018)
  * Vlachidis et al. [2018] Vlachidis, A., Bikakis, A., Terras, M., Padfield, J., Kontiza, K., Naudet, Y., Kyriaki-Manessi, D., Antoniou, A., Bassani, M.: Crosscult d2. 4 refined digital cultural resource data & data structure (2018)
  * Liang et al. [2022] Liang, V.W., Zhang, Y., Kwon, Y., Yeung, S., Zou, J.Y.: Mind the gap: Understanding the modality gap in multi-modal contrastive representation learning. In: Koyejo, S., Mohamed, S., Agarwal, A., Belgrave, D., Cho, K., Oh, A. (eds.) Advances in Neural Information Processing Systems, vol. 35, pp. 17612–17625. Curran Associates, Inc., ??? (2022)
  * Shi et al. [2023] Shi, D., Tao, C., Jin, Y., Yang, Z., Yuan, C., Wang, J.: Upop: Unified and progressive pruning for compressing vision-language transformers. arXiv preprint arXiv:2301.13741 (2023)
  * Webber et al. [2010] Webber, W., Moffat, A., Zobel, J.: A similarity measure for indefinite rankings 28(4) (2010) https://doi.org/10.1145/1852102.1852106
  * Pu et al. [2011] Pu, P., Chen, L., Hu, R.: A user-centric evaluation framework for recommender systems. In: Proceedings of the Fifth ACM Conference on Recommender Systems, pp. 157–164 (2011)
  * Schielzeth et al. [2020] Schielzeth, H., Dingemanse, N.J., Nakagawa, S., Westneat, D.F., Allegue, H., Teplitsky, C., Réale, D., Dochtermann, N.A., Garamszegi, L.Z., Araya‐Ajoy, Y.G.: Robustness of linear mixed‐effects models to violations of distributional assumptions. Methods Ecol. Evol. 11(9), 1141–1152 (2020)


* Wang et al. (2022) Wang, Lijie, Yaozong Shen, Shuyuan Peng, Shuai Zhang, Xinyan Xiao, Hao Liu, Hongxuan Tang, Ying Chen, Hua Wu, and Haifeng Wang. 2022.  A fine-grained interpretability evaluation benchmark for neural NLP.  In _Proceedings of the 26th Conference on Computational Natural Language Learning (CoNLL)_ , pages 70–84, Association for Computational Linguistics, Abu Dhabi, United Arab Emirates (Hybrid).
  * Wei et al. (2022) Wei, Jason, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  In _Advances in Neural Information Processing Systems_.
  * Wiegreffe and Marasovic (2021) Wiegreffe, Sarah and Ana Marasovic. 2021.  Teach me to explain: A review of datasets for explainable natural language processing.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1)_.
  * Wiegreffe and Pinter (2019) Wiegreffe, Sarah and Yuval Pinter. 2019.  Attention is not not explanation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 11–20, Association for Computational Linguistics, Hong Kong, China.
  * Wu et al. (2018) Wu, Lijun, Fei Tian, Tao Qin, Jianhuang Lai, and Tie-Yan Liu. 2018.  A study of reinforcement learning for neural machine translation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 3612–3621, Association for Computational Linguistics, Brussels, Belgium.
  * Wu et al. (2021) Wu, Tongshuang, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. 2021.  Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6707–6723, Association for Computational Linguistics, Online.
  * Xu et al. (2023) Xu, Wenda, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Yang Wang, and Lei Li. 2023.  Instructscore: Towards explainable text generation evaluation with automatic feedback.
  * Yao et al. (2023) Yao, Shunyu, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023.  Tree of thoughts: Deliberate problem solving with large language models.
  * Yuan et al. (2021) Yuan, Weizhe, Graham Neubig, and Pengfei Liu. 2021.  Bartscore: Evaluating generated text as text generation.  In _Advances in Neural Information Processing Systems_ , volume 34, pages 27263–27277, Curran Associates, Inc.
  * Zerva et al. (2022) Zerva, Chrysoula, Frédéric Blain, Ricardo Rei and Piyawat Lertvittayakumjorn, José G. C. de Souza, Steffen Eger, Diptesh Kanojia, Duarte Alves, Constantin Orăsan, Marina Fomicheva, André F. T. Martins, and Lucia Specia. 2022.  Findings of the wmt 2022 shared task on quality estimation.  In _Proceedings of the Seventh Conference on Machine Translation_ , Association for Computational Linguistics, Abu Dhabi.
  * Zhang et al. (2020a) Zhang, Tianyi, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020a.  Bertscore: Evaluating text generation with bert.  In _International Conference on Learning Representations_.
  * Zhang et al. (2020b) Zhang, Yunfeng, Q Vera Liao, and Rachel KE Bellamy. 2020b.  Effect of confidence and explanation on accuracy and trust calibration in ai-assisted decision making.  In _Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency_ , pages 295–305.
  * Zhao et al. (2020) Zhao, Wei, Goran Glavaš, Maxime Peyrard, Yang Gao, Robert West, and Steffen Eger. 2020.  On the limitations of cross-lingual encoders as exposed by reference-free machine translation evaluation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1656–1671, Association for Computational Linguistics, Online.
  * Zhao et al. (2019) Zhao, Wei, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019.  MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 563–578, Association for Computational Linguistics, Hong Kong, China.
  * Zhao et al. (2023) Zhao, Wei, Michael Strube, and Steffen Eger. 2023.  Discoscore: Evaluating text generation with bert and discourse coherence.  In _EACL_.
  * Zini and Awad (2022) Zini, Julia El and Mariette Awad. 2022.  On the explainability of natural language processing deep models.  _ACM Comput. Surv._ , 55(5).
  * Zouhar et al. (2023) Zouhar, Vilém, Shehzaad Dhuliawala, Wangchunshu Zhou, Nico Daheim, Tom Kocmi, Yuchen Eleanor Jiang, and Mrinmaya Sachan. 2023.  Poor man’s quality estimation: Predicting reference-based MT metrics without the reference.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 1311–1325, Association for Computational Linguistics, Dubrovnik, Croatia.