# A Unified Graph Transformer for Overcoming Isolations in Multi-modal
Recommendation

Zixuan Yi<EMAIL_ADDRESS>University of
GlasgowGlasgowScotlandUnited Kingdom and  Iadh Ounis<EMAIL_ADDRESS>University of GlasgowGlasgowScotlandUnited Kingdom

(2024)

###### Abstract.

With the rapid development of online multimedia services, especially in
e-commerce platforms, there is a pressing need for personalised recommender
systems that can effectively encode the diverse multi-modal content associated
with each item. However, we argue that existing multi-modal recommender
systems typically use isolated processes for both feature extraction and
modality encoding. Such isolated processes can harm the recommendation
performance. Firstly, an isolated extraction process underestimates the
importance of effective feature extraction in multi-modal recommendations,
potentially incorporating non-relevant information, which is harmful to item
representations. Second, an isolated modality encoding process produces
disjoint embeddings for item modalities due to the individual processing of
each modality, which leads to a suboptimal fusion of user/item representations
for an effective user preferences prediction. We hypothesise that the use of a
unified model for addressing both aforementioned isolated processes will
enable the consistent extraction and cohesive fusion of joint multi-modal
features, thereby enhancing the effectiveness of multi-modal recommender
systems. In this paper, we propose a novel model, called Unified multi-modal
Graph Transformer (UGT), which firstly leverages a multi-way transformer to
extract aligned multi-modal features from raw data for top-$k$ recommendation.
Subsequently, we build a unified graph neural network in our UGT model to
jointly fuse the multi-modal user/item representations derived from the output
of the multi-way transformer. Using the graph transformer architecture of our
UGT model, we show that the UGT model achieves significant effectiveness
gains, especially when jointly optimised with the commonly used recommendation
losses. Our extensive experiments conducted on three benchmark datasets
demonstrate the superiority of our proposed UGT model over nine existing
state-of-the-art recommendation approaches.

††journalyear: 2024††copyright: acmcopyright††ccs: Information systems
Recommender systems

##  1\. Introduction

In specific domains such as fashion (Wu et al., 2022), music (Park and Lee,
2022), and micro-video recommendation (Yi et al., 2022), the effectiveness of
recommender systems can be improved by using multi-modal data sources that the
users usually interact with (e.g., product images and descriptions, users’
reviews, audio tracks). Indeed, several previous studies (Wang et al., 2023b;
Liu et al., 2023; Zhou et al., 2023b) have proposed multi-modal recommender
systems that leverage multi-modal (i.e., audio, visual, textual) content data
to augment the representation of items, so as to tackle the problem of data
sparsity in the user-item matrix.

Figure 1. Fig.1: Recommendation working flow in exiting multi-modal
recommendation models.

Recent research (Liu et al., 2023; Deldjoo et al., 2020) has shown the
importance and benefits of coherently using multi-modal data in enhancing
multi-modal recommender systems. Multi-modal feature extraction, which aims to
derive meaningful patterns and information from multiple types of data, is
fundamental to the initial stage of any multi-modal recommendation pipeline.
Indeed, effectively performing such feature extraction is critical for
ensuring high-quality recommendations (Liu et al., 2023; Zhou et al., 2023b).
On the other hand, multi-modal fusion is also crucial for creating user/item
representations from various contextual data, including the items’ images and
descriptions. Such a fusion typically occurs after feature extraction. In
fact, most existing multi-modal recommendation models primarily function as a
fusion component, designed to merge the extracted features into a joint
representation of the items (Zhou et al., 2023b). Figure 1 illustrates how
multi-modal recommendation typically consists of a pipeline of workflows
ranging from data ingestion and feature extraction, to multi-modal fusion and
optimisation. Within this pipeline, two important problems emerge, namely, the
use of (i) an isolated feature extraction process (c.f. Isolation 1 in the
figure) and (ii) an isolated modality encoding process (c.f. Isolation 2).
These two problems impede the system from exploring the full potential of the
multi-modal data by creating isolated processes, as they ignore the
collaborative power of unifying various modalities and leveraging an
integrated architecture that effectively combines the extraction process with
the fusion process. The ‘Isolation 1’ problem refers to the challenges arising
from employing an isolated feature extraction process within each workflow of
the multi-modal recommendation system. Specifically, the pre-trained feature
extractors function independently alongside the subsequent fusion component.
This isolated extraction process potentially leads to the incorporation of
non-relevant information to the subsequent fusion component, thereby impeding
the extraction of more effective multi-modal features. On the other hand, as
indicated by the ‘Isolation 2’ problem in Figure 1, the existing recommender
systems process each modality separately before finally fusing them with
simple concatenations (Yi et al., 2022; Wei et al., 2019). Such an isolated
modality encoding process misses opportunities to jointly optimise the
user/item embeddings resulting from the different modalities.

To address these isolated processes for both feature extraction (Isolation 1)
and modality modelling (Isolation 2), we introduce a Unified Graph Transformer
(UGT) model for top-$k$ multi-modal recommendation. UGT combines a multi-way
transformer with a newly designed unified Graph Neural Network (GNN) in a
novel cascading graph transformer architecture. We use the multi-way
transformer as the extraction component and integrate its output in the
unified GNN that serves as the fusion component, so as to address the
‘Isolation 1’ problem. This new unified GNN within our proposed UGT
architecture allows to address the ‘Isolation 2’ problem by uniformly
aggregating the users/items’ neighbour information and their corresponding
multi-modal features, thereby enriching the multi-modal fusion for better
user/item representations. The integration of the multi-way transformer and
the unified GNN enables a simultaneous optimisation of the extraction of
effective multi-modal features and their subsequent fusion, thereby solving
the isolated feature extraction process. Our work is the first to use a multi-
way transformer as an extraction component in multi-modal recommendation,
thereby going beyond its current use in the literature for encoding various
modalities (e.g., images, text) within a shared transformer block (Bao et al.,
2022) in Image-Text retrieval or Visual Question Answering (VQA) (Wang et al.,
2023a; Ge et al., 2021, 2024). Indeed, we instead apply the multi-way
transformer in a recommendation setting to transform heterogeneous data into a
unified latent space, thereby producing aligned embeddings from each modality.
We then use our proposed unified GNN component to seamlessly fuse the
resulting aligned multi-modal embeddings, a significant departure from the
existing models that process modalities separately.

In summary, our contributions in this paper are three-folds: $(1)$ We propose
a novel Unified Graph Transformer (UGT) model to resolve the isolated feature
extraction problem in multi-modal recommendation. To the best of our
knowledge, UGT is the first end-to-end multi-modal recommendation model that
directly facilitates multi-modal recommendation from raw data. Our UGT model
not only extracts the items’ features and integrates them with the user-item
interactions but also simultaneously optimises extraction and fusion under a
unified architecture; $(2)$ We introduce a new unified GNN as the fusion
component to uniformly fuse the user-item interactions and their multi-modal
features in a unified manner. This unified approach addresses the problem of
isolated modality encoding in the existing multi-modal recommender systems, so
as to enhance the recommendation performance by ensuring a more effective
modelling of the target users from the items’ images and descriptions; $(3)$
We conduct extensive experiments on three public datasets to demonstrate the
effectiveness of our UGT model in comparison to nine existing state-of-the-art
multi-modal recommender systems. Moreover, we provide visualisations and
measure the average Mean Square Error between the visual and textual
embeddings to show that UGT successfully aligns these embeddings to a unified
and tighter semantic space.

##  2\. Related Work

In this section, we position our work in relation to related methods and
techniques in the literature, namely multi-modal recommendation (Section 2.1)
and graph transformers (Section 2.2).

###  2.1. Multi-modal Recommendation

Existing multi-modal recommendation models incorporate multi-modal data
sources for items into the traditional Collaborative Filtering (CF) paradigm
through the application of deep or graph learning techniques alongside
different feature extraction methods (Liu et al., 2023; Zhou et al., 2023b).
All such models capitalise on the items’ multi-modalities to enhance the
recommendation performance. For instance, VBPR (He and McAuley, 2016) enhanced
the BPR method (Rendle et al., 2009a) by modelling the users’ preferences in
relation to visual information and extracted visual features from the items’
images through a pre-trained Deep Convolutional Neural Network (CNN) (Donahue
et al., 2014). Moreover, MMGCN (Wei et al., 2019) used Graph Convolutional
Networks (GCN) to propagate modality-specific embeddings and to capture the
users’ preferences related to these modalities. For feature extraction, MMGCN
employed a variant of the Residual Network model (ResNet50 (He et al., 2016))
for visual features, Sentence2Vector (Le and Mikolov, 2014) for textual
features, and VGGish (Gemmeke et al., 2017) for acoustic features. MMGCL (Yi
et al., 2022), SLMRec (Tao et al., 2022) and BM3 (Zhou et al., 2023a) followed
the same feature extraction methods as MMGCN. However, MMGCL proposed a self-
supervised graph learning model that leverages modality-based edge dropout and
modality masking to capture the complex user preferences. On the other hand,
SLMRec constructed fine and coarse spaces to align features across modalities,
then merely applied a contrastive learning loss for multi-modal
recommendation. BM3 bootstrapped the user/item node embeddings through a
dropout augmentation and reconstructed the user-item graph for multi-modal
recommendation. In a different line of work, LATTICE (Zhang Jinghao et al.,
2021) built item-item graphs from multi-modal features and uncovered latent
semantic structures between items. Later, FREEDOM (Zhou and Shen, 2023)
further improved LATTICE by pruning popular edges on the item-item graphs.
Both LATTICE and FREEDOM used Deep CNN (Donahue et al., 2014) and Sentence-
Transformer (Reimers and Gurevych, 2019) for visual and textual feature
extraction, respectively. While each multi-modal recommendation model
leverages features extracted from pre-trained extraction models and employs
specific strategies like Graph Neural Network (GNN) for multi-modal fusion,
these recommendation models primarily rely on pre-trained extraction models
for feature extraction. As a result, the use of pre-trained models for
extraction isolates the extraction step from the entire process of multi-modal
recommendation. This isolation occurs because the extraction models are not
tailored or fine-tuned for the multi-modal recommendation task, thus
potentially incorporating non-relevant information that could harm the item
representations in a multi-modal recommendation context. In contrast to
previous multi-modal recommendation models that use separate extraction
models, in this paper, we propose instead a novel unified graph transformer
model that seamlessly integrates the extraction and fusion processes by
combining a multi-way transformer with a unified GNN in a cascading
architecture, thereby overcoming the isolated extraction process (Isolation 1)
in multi-modal recommendation.

###  2.2. Graph Transformers

The integration of transformer architectures into graph neural networks has
been proposed as a promising solution to enhance existing Graph Neural
Networks (GNNs) (Ying et al., 2021; Yi et al., 2023c). In the recommendation
domain, taking advantage of because of its contextualised information
modelling, the transformer network recently drew much research attention from
the recommendation research community to explore the relationships among
items. For instance, prior works (Li et al., 2023a; Cao et al., 2022)
leveraged GNNs to model the item embeddings within a user’s sequence of items,
subsequently integrating them into transformer-based recommendation models
(e.g. SASRec (Kang and McAuley, 2018), Bert4Rec (Sun et al., 2019)) in order
to better model user behaviour sequences. Moreover, existing approaches (Li et
al., 2023b; Zhu et al., 2022; Chen et al., 2024; Yi et al., 2023a, b, 2024)
also integrated the vanilla transformer with a GNN to exploit the global user-
item relationships. Such approaches enhance the recommendation performance by
going beyond merely aggregating local neighbours to encompassing a global
context. Different from these sequential and general recommendation models,
the recent graph transformers in multi-modal recommendation (Liu et al., 2021;
Wei et al., 2023) explicitly modelled the correlation between the item
representations by jointly extracting the items’ images and descriptions as
node features. Specifically, PMGT (Liu et al., 2021) constructed an item-item
graph from pre-extracted multi-modal features and used a transformer network
to model the item embeddings with their corresponding neighbours. LightGT (Wei
et al., 2023) proposed a simplified graph transformer architecture, featuring
a single-head attention to distil each type of pre-extracted multi-modal
features. Different from the aforementioned multi-modal graph transformers
that process each modality individually, we introduce a new unified GNN as a
fusion component. This fusion component is closely paired with a multi-way
transformer in our UGT architecture so as to effectively fuse the extracted
multi-modal features from the multi-way transformer. In particular, we
incorporate a novel attentive-fusion component within the unified GNN to
uniformly fuse the resulting multi-modal features from the multi-way
transformer, thereby addressing the problem of isolated modality encoding
(Isolation 2) in multi-modal recommendation.

##  3\. Model Architecture

Figure 2. Fig.2: Our Unified multi-modal Graph Transformer (UGT).

In this section, we describe our UGT model for top-$k$ multi-modal
recommendation. We first introduce the architecture of our UGT model in
Section 3.2, which is illustrated in Figure 2. Section 3.3 presents the
unified multi-modal encoding process of our proposed multi-way transformer.
Then, in Section 3.4, we introduce the unified GNN, which explicitly fuses
multi-modal features and user-item interactions into final user/item
representations. In Section 3.5, we describe the optimisation of UGT with its
corresponding losses.

###  3.1. Preliminaries

Let $\mathcal{U}=\\{u\\}$ denote the user set and $I=\\{i\\}$ denote the item
set. The input ID embeddings of user $u$ and item $i$ are
$\mathrm{E}_{id}\in\mathbb{R}^{d\times(|U|+|I|)}$. $d$ is the dimension of the
user/item embedding. Then, we denote each item modality feature as
$\mathbf{E}_{i,m}\in\mathbb{R}^{d_{m}\times|I|}$, where $d_{m}$ is the
dimension of that modality’s features, $m\in\mathcal{M}$ is a modality, and
$\mathcal{M}$ is the set of modalities. In this paper, we mainly consider
visual and textual modalities because the datasets used only contain these two
types of raw data, hence $\mathcal{M}=\\{v,t\\}$. The user’s historical
behaviour data is denoted by $\mathcal{R}\in\\{0,1\\}^{|U|\times|I|}$, where
each entry $\mathcal{R}_{u,i}=1$ if user $u$ clicked item $i$, otherwise
$R_{u,i}=0$. The historical interaction data $\mathcal{R}$ can be seen as a
sparse behaviour graph $\mathcal{G}=\\{\mathcal{V},\mathcal{E}\\}$, where
$\mathcal{V}=\\{\mathcal{U}\cup I\\}$ denotes the set of nodes and
$\mathcal{E}=\left\\{(u,i)\mid u\in\mathcal{U},i\in I,R_{ui}=1\right\\}$
denotes the set of edges. The purpose of the multi-modal recommendation model
is to accurately predict the users’ preferences by ranking items for each user
according to the predicted preference scores $\hat{y}_{ui}$.

###  3.2. Model Overview

Figure 2 shows the model architecture of UGT, our proposed graph transformer
model, which is composed of two main components: an extraction component
(i.e., a multi-way transformer) and a fusion component (i.e., a unified GNN).
Given an item image, an item text and an item ID, we follow the common
practice in the computer vision domain (Kolesnikov et al., 2022) by splitting
the item image into patches, flattening these patches, and linearly projecting
them to form patch embeddings. Similarly, we obtain the word embeddings from
an item’s description (Radford et al., 2019). Our UGT model then uses the
patch embeddings from the image and the word embeddings as inputs.
Specifically, we use a shared multi-head self-attention component to extract
the items’ respective modality embeddings. After the shared self-attention
component, we employ two specialised feed-forward networks, known as modality
experts, corresponding to the two modalities available in our datasets –
visual and textual111Although our UGT model primarily focuses on visual and
textual modalities, we note that it can generalise to other modalities when
they are present in the datasets.. These feed-forward networks produce the
corresponding multi-modal item embeddings. In our UGT model, the shared self-
attention component in the multi-way transformer is designed to align
different modalities, while the modality experts are used to capture
information unique to each modality. Consequently, the multi-way transformer
acts as a unified feature extractor, returning the extracted item’s visual and
textual embeddings before their combination with the user/item ID embeddings
in a later new unified GNN component. We design this integration between the
extraction and fusion components to address the isolated extraction problem in
exiting multi-modal recommender systems. As illustrated in Figure 2, once the
extracted multi-modal item features have been obtained, our model’s unified
GNN component jointly combines these features with their ID embeddings to
produce user/item embeddings based on the user-item interaction graph. The
unified GNN is needed in order to obtain a smoother user/item embedding by
aggregating the multi-modal features from the neighbours. This aggregation
includes both the user-item interactions and the semantic information, which
is derived from the multi-modal features processed by the multi-way
transformer, into the final user/item representations. As shown in Figure 2,
within the unified GNN, we incorporate an attentive-fusion component to fuse
the jointly-learned and aggregated embeddings from different modalities, in
order to enhance the multi-modal user/item representations. The improved joint
multi-modal features in our UGT model are designed to overcome the limitations
of modelling each modality separately (Isolation 2) in multi-modal
recommendation.

###  3.3. Unified Multi-modal Encoding

As discussed in Section 2.1, we propose a multi-way transformer paired with a
fusion component to alleviate the isolated extraction problem (Isolation 1) in
existing multi-modal recommender systems. Specifically, we use this multi-way
transformer to encode the raw images and textual descriptions of items,
replacing the feed-forward network with modality experts in a standard
Transformer (Vaswani et al., 2017). Using the modality experts allows to
capture modality-specific information, with each modality expert specifically
handling different item modalities. As described in Section 3.2, the use of
Multi-Head Self-Attention (MHSA), shared across modalities, allows the
alignment of the visual and textual features. Hence, for a given item, the
embedding is defined as follows:

(1)
$h_{i}^{(l_{t})}=\text{LN}\left(h_{i}^{(l_{t}-1)}+\text{MHSA}(h_{i}^{(l_{t}-1)},h_{i}^{(l_{t}-1)},m_{i})+\text{FFN}(h_{i}^{(l_{t}-1)},m_{i})\right)$

where $h_{i}^{(l_{t})}$ is the hidden state of the $i$-th item in the
$l_{t}$-th layer, LN and MHSA are the layer normalisation operation and the
multi-head self-attention mechanism (Vaswani et al., 2017), respectively, and
$m_{i}$ is the corresponding multi-modal embedding of the item. FFN is the
feed-forward network. Hence, the multi-way transformer chooses the right
expert among multiple modality experts to handle the input based on both the
modality’s type and the transformer layer’s position. Note that our UGT model
can extract features from any number of modalities, not just two as per our
used datasets. The multi-way transformer, combined with the fusion component
shown in Figure 2, is designed to unify the extraction and fusion processes,
addressing the problem of isolated extraction (Isolation 1).

###  3.4. Unified Multi-modal Fusion

In Section 2.2, we argued for the need for a unified approach to process the
extracted multi-modal features for an effective fusion, instead of handling
each modality individually. Hence, we introduce our new unified Graph Neural
Network (GNN), which seamlessly fuses the image and text embeddings in a joint
manner, closely integrated with the multi-way transformer. The propagation
function of our unified GNN is defined as follows:

(2) $x_{i}^{(l_{g})}=(1+\epsilon)\cdot
x_{i-vt}^{(l_{g}-1)}+\left(x_{u-vt}^{(l_{g})}+x_{u-id}^{(l_{g})}\right)$

where $x_{i}$ denotes the item representations in the $l_{g}$-th graph
convolution layer; $x_{i-vt}$ and $x_{u-vt}$ denote the joint multi-modal
embeddings of the item and its neighbours; $x_{u-id}$ denotes the ID
embeddings of the item’s neighbours; $\epsilon$ is a scaling factor that
controls the contribution of an item’s self-connection with its associated
joint multi-modal features. The user embeddings are similarly calculated. To
obtain the user/item ID embeddings from the neighbours, we use LightGCN (He et
al., 2020) to propagate the ID embeddings of the users and items over the
user-item interaction graph. The propagation function of the ID embeddings is
defined as follows:

(3) $\displaystyle
x_{i-id}^{(l_{g})}=\sum_{i\in\mathcal{N}_{u}}\frac{x_{i-id}^{(l_{g}-1)}}{\sqrt{\left|\mathcal{N}_{i}\right|\left|\mathcal{N}_{u}\right|}},$

where $\mathcal{N}_{i}$ and $\mathcal{N}_{u}$ denote the set of neighbours for
user $u$ and item $i$, respectively, while $\left|\mathcal{N}_{u}\right|$ and
$\left|\mathcal{N}_{i}\right|$ represent the size of $\mathcal{N}_{u}$ and
$\mathcal{N}_{i}$.

A key addition to our UGT model is to propose an attentive-fusion method to
enhance the joint embeddings based on the extracted multi-modal features, as
illustrated in Figure 2. Given the obtained visual and textual embeddings,
$h_{i-v}$ and $h_{i-v}$, derived from Equation (1), we combine the multi-modal
embeddings with an attentive-fusion approach to obtain a unified joint
embedding, as follows:

(4) $x_{i-vt}^{(l_{g})}=\alpha
h_{i-v}^{(l_{g})}\|(1-\alpha)h_{i-t}^{(l_{g})},$

where $\alpha$ is a learned parameter for the attentive-fusion and $\|$
represents the concatenation operation. As a consequence, this fusion approach
is designed to uniformly enhance the resulting user/item embeddings by
weighting the importance of each modality, in contrast to the simple
concatenation operation used in previous works (Yi et al., 2022; Wei et al.,
2023). Our UGT model uses the enhanced joint multi-modal features to address
the problem of isolated modality encoding (Isolation 2) in multi-modal
recommendation.

###  3.5. Model Optimisation

To ensure the effective extraction and multi-modal fusion of the visual and
textual modalities, our UGT model employs the Image-Text Contrastive (ITC)
loss to train the model in a joint embedding space where the similarity
between an item’s image and its corresponding text description is maximised,
while minimising the similarity between mismatched image-text pairs (Li et
al., 2022). The ITC loss is defined as follows:

(5)
$\mathcal{L_{\text{ITC}}}=-\frac{1}{N}\sum_{z=1}^{N}\log\frac{p_{v2t}(z)}{\sum_{z^{\prime}\neq
z}^{N}p_{v2t}(z^{\prime})}-\frac{1}{N}\sum_{z=1}^{N}\log\frac{p_{t2v}(z)}{\sum_{z^{\prime}\neq
z}^{N}p_{t2v}(z^{\prime})}$

where $N$ is the batch size, while $p_{v2t}(z)$ and $p_{t2v}(z)$ are the
softmax-normalised image-to-text and text-to-image similarities of the $z$-th
pair, respectively. As a result, this ITC loss simultaneously optimises our
UGT model during training so as to extract and fuse more effective multi-modal
features from raw data.

Following (Yi et al., 2022; Zhang Jinghao et al., 2021), we also implement a
multi-task training strategy in our UGT model that simultaneously optimises
the commonly used pairwise ranking loss in recommendation, specifically the
Bayesian Personalised Ranking (BPR) (Rendle et al., 2009b) loss, denoted as
$\mathcal{L}_{BPR}$, along with the ITC loss $\mathcal{L}_{ITC}$:

(6) $\displaystyle\mathcal{L}$
$\displaystyle=\mathcal{L}_{BPR}+\lambda_{c}\mathcal{L}_{ITC}+\lambda\|\Theta\|_{2},$
$\displaystyle\text{where}\quad\mathcal{L}_{BPR}$
$\displaystyle=\sum_{(u,i,j)\in
D_{s}}\text{log}\sigma({y}_{ui}-{x_{u}}^{\top}{x_{i}})$

where $\lambda_{c}$ and $\lambda$ are hyper-parameters that control the
strengths of the ITC loss and the $L_{2}$ regularisation, respectively,
$\Theta$ is the set of model parameters; $\boldsymbol{x_{u}}$ is the user
embedding, $\boldsymbol{x_{i}}$ denotes the positive item embedding and
${y}_{ui}$ is the ground truth value, $D_{s}\ =\\{(u,i,j)|(u,i)\in
R^{+},(u,j)\in R^{-}\\}$ is the set of the training data, $R^{+}$ indicates
the observed interactions and $R^{-}$ indicates the unobserved interactions,
while $\sigma(\cdot)$ is the sigmoid function. Hence, we optimise both the
multi-way transformer and the unified GNN using the loss functions above.
These loss functions prevent the incorporation of non-relevant information
that could be harmful to the learned user/item embeddings, further supporting
our UGT model in addressing the isolated extraction problem (Isolation 1) in a
unified manner. By applying Equation (1), we extract multi-modal item
embeddings from each modality using the multi-way transformer. Next, we fuse
the extracted visual and textual item embeddings using the attentive fusion as
detailed in Equation (4). We then integrate the resulting joint item
embeddings with the user/item ID embeddings and the user-item interactions
within our unified GNN, as detailed in Equation (2), so as to obtain the final
user/item embeddings. We use Equation (6) to optimise both the multi-way
transformer and the unified GNN within our UGT model, facilitating a unified
approach to address the isolated extraction problem (Isolation 1). In
addition, by incorporating an attentive-fusion component, the unified GNN
uniformly fuses the multi-modal features, thereby tackling the problem of
isolated modality encoding (Isolation 2) in multi-modal recommendation.

##  4\. Experiments

We conduct experiments to validate the effectiveness of our UGT model on three
public datasets, in comparison to nine existing state-of-the-art multi-modal
recommendation models. To examine and analyse the effectiveness of our UGT
model, we conduct experiments to answer the following four research questions:
RQ1: How does our proposed UGT model perform compared with existing multi-
modal recommendation models?
RQ2: How do the two components of UGT, namely the unified GNN and the multi-
way transformer as well as the UGT’s loss function affect the performance of
the model?
RQ3: How do different parameters (i.e. $\epsilon$, $\lambda_{c}$) in our UGT
model affect its performance?
RQ4: Does the UGT model exihibit a better alignment of modalities compared to
the strongest baseline FREEDOM?

###  4.1. Experimental Settings

Table 1. Statistics of the used Amazon datasets.

| Sports | Clothing | Baby
---|---|---|---
#Users | 35,598 | 39,387 | 19,445
#Items | 18,287 | 22,499 | 7,037
#Interactions | 295,366 | 271,001 | 160,522
Density | 0.00045 | 0.00030 | 0.00012
CNN/UGT Visual Dimension | 4096/768 | 4096/768 | 4096/768
Sentence-Transformer/UGT Textual Dimension | 384/768 | 384/768 | 384/768

####  4.1.1. Datasets

In order to evaluate the effectiveness of our UGT model in the top-$k$ multi-
modal recommendation task, we conduct experiments on three commonly used
Amazon Review datasets222https://jmcauley.ucsd.edu/data/amazon/, namely Sports
and Outdoors (Sports for short), Clothing, Shoes and Jewelry (Clothing for
short), and Baby. Table 1 shows the statistics of the used datasets. We choose
these datasets due to their comprehensive coverage of user-item interactions
and their abundant multi-modal data, including the items’ image URLs and
textual descriptions. These datasets are also unique in providing extensive
raw data, unlike other multi-modal recommendation datasets such as
TikTok333http://ai-lab-challenge.bytedance.com/ and
Kwai444https://www.kuaishou.com/activity/uimc, which do not provide raw data
in relation to the various modalities. In addition, these datasets have been
widely evaluated by several existing state-of-the-art baselines, such as
(Zhang Jinghao et al., 2021; Zhou et al., 2023a; Wei et al., 2023), further
confirming their suitability for our analysis.

####  4.1.2. Evaluation Protocols

Following the evaluation setting in (Zhang Jinghao et al., 2021; Zhou et al.,
2023b), we randomly split the datasets into training, validation, and testing
sets using an 8:1:1 ratio. Both our UGT model and the used baselines have
their hyper-parameters optimised using a grid search on the validation set,
and refined by the Adam (Kingma and Ba, 2014) optimiser. For the other hyper-
parameters unique to our UGT model ($\epsilon$ and $\lambda_{c}$), we tune the
parameters as follows: $\epsilon\in\left\\{0,0.1,0.2,...,1.0\right\\}$ and
$\lambda_{c}\in\left\\{0,0.1,0.2,...,1.0\right\\}$. We use two commonly used
evaluation metrics, namely Recall@K and NDCG@K, to examine the top-$K$
recommendation performance. We set K to 10 and 20 (Zhang Jinghao et al.,
2021), and report the average performance achieved for all users in the
testing set. We also examine the statistical significance of UGT’s improved
performance over the used baselines using the paired t-test with the Holm-
Bonferroni correction for $p<0.05$. We apply an early-stopping strategy that
terminates the training of both UGT and the baseline models if no decrease in
validation loss is observed over 50 epochs. All used baselines and our UGT
model are implemented with PyTorch and were ran on a GPU A6000 with 48GB
memory.

####  4.1.3. Baselines

To examine the effectiveness of our UGT model, we perform a comparative
analysis between the baselines, which use isolated extraction methods and
isolated modality fusion methods, and our proposed unified UGT model. To
ensure a fair comparison, we employ the widely-used CNN and Sentence-
Transformer as modality-specific extractors in our baselines (Zhou et al.,
2023b). These extractors are used to derive item features from the raw images
and descriptions within the used datasets. In doing so, we ensure that both
the CNN and Sentence-Transformer, as well as our multi-way transformer, have a
similar number of parameters (170 million vs. 167 million), ensuring a fair
comparison. For our experiments, we select nine existing state-of-the-art
multi-modal recommendation models, namely: GNN-based, Non-GNN-based and Graph
Transformer models, as detailed below: (1) GNN-based & Non-GNN-based Models:
VBPR (He and McAuley, 2016), a Non-GNN-based recommendation model, uses VGG
net (Gemmeke et al., 2017) to extract visual features and exclusively
integrates these features with user/item IDs into matrix factorisation to
allows multi-modal recommendation; MMGCN (Wei et al., 2019) uses graph
convolutional networks (GCN) to learn modality-specific features from
individual graphs using pre-extracted features. Then, it combines these
features through simple concatenation to generate final user and item
representations; MMGCL (Yi et al., 2022) also uses pre-extracted features and
introduces modality edge dropout and modality masking augmentations to the
concatenated multi-modal user/item embeddings, enhancing multi-modal
representation learning through a self-supervised learning paradigm; SLMRec
(Tao et al., 2022), similar to MMGCL, is a GNN-based model that leverages
self-supervised learning with pre-extracted features to generate supervised
signals by contrasting fine and coarse-item embeddings across each modality;
LATTICE (Zhang Jinghao et al., 2021), unlike the aforementioned GNN-based
models, builds item-item graphs from pre-extracted multi-modal features and
performs individual graph convolutional operations on both these graphs and
the user-item interaction graphs to produce user and item representations; BM3
(Zhou et al., 2023a) bootstraps latent user/item representations in
contrastive learning by reconstructing the user-item interaction graph to
enhance the recommendation performance; FREEDOM (Zhou and Shen, 2023) freezes
the structure of LATTICE’s item-item graph while denoising the user-item
interaction graph by pruning the popular edges. (2) Graph Transformer Models:
PGMT (Liu et al., 2021) constructs an item-item graph to aggregate the multi-
modal features of items and leverages a transformer network to model the item
embeddings along with their contextual neighbours within the obtained graph;
LightGT (Wei et al., 2023) uses a vanilla transformer network to model the
users’ preferences by considering their interacted items along with
individually pre-extracted multi-modal features and integrates these
preferences with the aggregated user/item ID embeddings to predict the user-
item interactions.

###  4.2. Performance Comparison (RQ1)

Table 2. Experimental results comparing our UGT model with the used baselines
in multi-modal recommendation. The best performance of each model is
highlighted in bold. ∗ denotes a significant difference compared to the
indicated baseline performance using the paired t-test with the Holm-
Bonferroni correction for $p<0.05$.

Dataset Sports Clothing Baby Methods Recall@10 Recall@20 NDCG@10 NDCG@20
Recall@10 Recall@20 NDCG@10 NDCG@20 Recall@10 Recall@20 NDCG@10 NDCG@20 VBPR
${0.0509}$∗ ${0.0771}$∗ ${0.0280}$∗ ${0.0349}$∗ ${0.0409}$∗ ${0.0611}$∗
${0.0226}$∗ ${0.0277}$∗ ${0.0479}$∗ ${0.0740}$∗ ${0.0262}$∗ ${0.0329}$∗ MMGCN
${0.0290}$∗ ${0.0475}$∗ ${0.0154}$∗ ${0.0201}$∗ ${0.0151}$∗ ${0.0246}$∗
0.0077∗ 0.0100∗ 0.0391∗ 0.0642∗ 0.0201∗ 0.0266∗ MMGCL ${0.0617}$∗ ${0.0913}$∗
0.0351∗ 0.0428∗ 0.0410∗ 0.0607∗ 0.0227∗ 0.0277∗ 0.0521∗ 0.0790∗ 0.0283∗
0.0352∗ SLMRec 0.0605∗ 0.0901∗ ${0.0341}$∗ 0.0417∗ 0.0430∗ 0.0623∗ 0.0238∗
0.0287∗ 0.0527∗ 0.0810∗ 0.0288∗ 0.0361∗ LATTICE 0.0633∗ 0.0944∗ 0.0334∗
0.0424∗ 0.0484∗ 0.0704∗ 0.0280∗ 0.0336∗ 0.0539∗ 0.0860∗ 0.0291∗ 0.0374∗ BM3
0.0661∗ 0.0970∗ 0.0350∗ 0.0438∗ 0.0535∗ 0.0797∗ 0.0305∗ 0.0358∗ 0.0542∗
0.0863∗ 0.0296∗ 0.0380∗ FREEDOM 0.0605∗ 0.0918∗ 0.0352∗ 0.0449∗ 0.0538∗
0.0809∗ 0.0290∗ 0.0356∗ 0.0551∗ 0.0874∗ 0.0303∗ 0.0385∗ PMGT ${0.0524}$∗
${0.0796}$∗ ${0.0303}$∗ ${0.0371}$∗ ${0.0403}$∗ ${0.0608}$∗ ${0.0219}$∗
${0.0264}$∗ ${0.0500}$∗ ${0.0764}$∗ ${0.0271}$∗ ${0.0338}$∗ LightGT 0.0652∗
0.0953∗ 0.0347∗ 0.0440∗ 0.0514∗ 0.0755∗ 0.0299∗ 0.0353∗ 0.0544∗ 0.0867∗
0.0294∗ 0.0381∗ UGT 0.0705 0.1034 0.0391 0.0477 0.0603 0.0922 0.0330 0.0402
0.0602 0.0930 0.0325 0.0406 %Improv. 6.66% 6.60% 11.10% 6.24% 12.08% 13.97%
8.20% 12.29% 9.26% 6.41% 7.26% 5.45%

Table 2 compares the performance of our UGT model to that of nine multi-modal
recommendation baselines. In the table, the top and second-best results are
highlighted in bold and underlined, respectively. We also evaluate the
statistical significance of the difference in performance between our UGT
model and that of the used baselines according to the paired t-test with the
Holm-Bonferroni correction. From the results in Table 2, we observe the
following:

  * •

For all three datasets, UGT outperforms all the baseline models on all metrics
by a large margin, and the differences are statistically significant in all
cases. In particular, our UGT model significantly outperforms the strongest
baseline, LightGT, in terms of Recall@20, showing improvements of 6.60%,
13.97% and 6.41% over the three used datasets, respectively. These significant
improvements demonstrate the effectiveness of our graph transformer
architecture in comparison to using an independent multi-modal recommendation
model alongside the corresponding pre-trained feature extractors. We attribute
this improved performance to the combined effect of the multi-way transformer
and the unified GNN, along with the used losses (i.e., BPR loss, ITC loss).
These components enable a simultaneous optimisation for extracting effective
multi-modal features, which are then fused to enhance the final user/item
embeddings.

  * •

In Table 2, the comparison of the graph transformer models (PGMT, LightGT,
UGT) with the GNN-based models (MMGCN, MMGCL, SLMRec, LATTICE, BM3, FREEDOM),
shows that, with the exception of PGMT, the graph transformer models generally
exhibit a performance on par with the GNN-based models. This result indicates
that the application of the transformer network in the GNN-based models is
more suitable for the extraction or distillation of multi-modal features, in
comparison to using the transformer for aggregation. Similar observations have
been made in other tasks such as VQA, as reported in (He and Wang, 2023).

Hence, in answer to RQ1, we conclude that our UGT model effectively leverages
the graph transformer architecture, consisting of multi-way transformer paired
with a unified GNN, to significantly enhance the multi-modal recommendation
performance in comparison to 9 strong baselines from the literature. UGT
indeed offers a comprehensive end-to-end solution that effectively and
seamlessly combines the extraction and fusion components, thereby addressing
the issue of isolated feature extraction (Isolation 1) in multi-modal
recommender systems.

###  4.3. Ablation Study (RQ2)

Table 3. Ablation study on the key components of UGT. ∗ indicates the
significant differences according to the paired-t test.

Dataset Sports Clothing Baby Methods Recall@10 Recall@20 NDCG@10 NDCG@20
Recall@10 Recall@20 NDCG@10 NDCG@20 Recall@10 Recall@20 NDCG@10 NDCG@20 UGT
w/o Attn-Fuse 0.0675∗ 0.0986∗ 0.0375∗ 0.0464 0.0564∗ 0.0823∗ 0.0310∗ 0.0373∗
0.0568∗ 0.0894∗ 0.0308∗ 0.0404 UGT w/o UGNN ${0.0667}^{*}$ ${0.0980}^{*}$
0.0371∗ 0.0452∗ ${0.0540}^{*}$ ${0.0811}^{*}$ ${0.0302}^{*}$ ${0.0356}^{*}$
${0.0551}^{*}$ ${0.0865}^{*}$ ${0.0298}^{*}$ ${0.0383}^{*}$ UGT w/o Trans
0.0628∗ 0.0929∗ 0.0357∗ 0.0440∗ 0.0483∗ 0.0784∗ 0.0269∗ 0.0323∗ 0.0535∗
0.0834∗ 0.0290∗ 0.0364∗ UGT w/o CL 0.0680∗ 0.0953∗ 0.0381∗ 0.0458∗ 0.0578∗
0.0896∗ 0.0309∗ 0.0360∗ 0.0543∗ 0.0855∗ 0.0303∗ 0.0377∗ UGT 0.0705 0.1034
0.0391 0.0477 0.0603 0.0922 0.0330 0.0402 0.0602 0.0930 0.0325 0.0406

In this section, we evaluate how each component of the UGT model, the multi-
way transformer (referred to as Trans) and the unified GNN (UGNN), influences
the recommendation performance. Recall that the unified GNN component
incorporates an attentive-fusion (Attn-Fuse) as an internal component (see
Fig. 2). Specifically, to evaluate the impact on effectiveness, we replace the
multi-way transformer (Trans) and the unified GNN (UGNN) components with a CNN
+ Sentence Transformer (Zhou et al., 2023b) extractor and LightGCNs (He et
al., 2020) respectively, which are commonly used in the baseline models. In
addition, we simplify the attentive-fusion (Attn-Fuse) component by changing
its attentive concatenation to a straightforward concatenation of the
extracted multi-modal features so as to assess the effectiveness of the
attentive-fusion component. Finally, we evaluate the effect of contrastive
learning (CL) within the UGT’s loss function by removing the contrastive loss
(c.f. Equation (6)). Table 3 presents the performance outcomes of UGT’s
variants in multi-modal recommendation across the three datasets we used. We
observe the following:

  * •

Comparing UGT to its variant without the attentive concatenation (UGT w/o
Attn-Fuse), we note a marked decrease in effectiveness of UGT across all three
datasets. This result indicates the importance of our attentive-fusion
component in effectively fusing the multi-modal features into the final
user/item embeddings.

  * •

The ablation of the unified GNN (UGNN) and its replacement with a LightGCN
multi-stream processing on each modality 555Indeed, most GNN-based models
(MMGCL, SLMRec, LATTICE) apply LightGCNs in multi-stream processing, leads to
a significant decrease of UGT’s performance.. This observation emphasises the
essential role of applying a unified processing on the multi-modal inputs, in
contrast to the traditional method that processes each modality input
separately.

  * •

Table 3 also shows that replacing our multi-way transformer (Trans) component
with a CNN + Sentence Transformer extractor markedly degrades the UGT’s
recommendation performance. This result highlights the advantage of replacing
the pre-trained extractors, typically used in the existing methods, with our
multi-way transformer, thereby enabling the simultaneous optimisation of the
extraction process and the generation of the user/item embeddings for
effective multi-modal recommendation.

  * •

The UGT model, which uses both the ITC and BPR losses, significantly
outperforms its variant without contrastive learning (i.e., UGT$\,w/o\,CL$).
This suggests that the contrastive loss, as introduced in Sec 3.5, is
beneficial for multi-modal recommendation. Indeed, CL, simultaneously provides
large gradients to optimise both the extraction (multi-way transformer) and
fusion (UGNN) components in our UGT model, thereby improving the
recommendation performance.

Hence, in answer to RQ2, we conclude that UGT successfully uses each of its
key components as well as its loss function to provide an effective unified
approach for learning effective user/item representations in multi-modal
recommendation. Specifically, our unified GNN (UGNN) component effectively
fuses the multi-modal features using an attentive-fusion (Attn-Fuse) method,
thereby addressing the problem of isolated modality encoding (Isolation 2) in
the existing multi-modal recommenders. Furthermore, when we replace the CNN +
Sentence Transformer extractor with our multi-way transformer (Trans)
component and pair it with the unified GNN, we observe an improvement in
recommendation performance, thereby addressing the isolated feature extraction
problem (Isolation 1). Finally, the contrastive loss (CL) effectively
optimises both the unified GNN (UGNN) and the multi-way transformer (Trans)
components, thereby improving the recommendation performance.

###  4.4. Hyper-parameter Study (RQ3)

Figure 3. Performance of our UGT model with respect to different
$\lambda_{{\color[rgb]{0,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{0,0,0}\pgfsys@color@gray@stroke{0}\pgfsys@color@gray@fill{0}c}}$
on the Sports and Clothing datasets.

Figure 4. Performance of our UGT model using different $\epsilon$ values on
the Sports and Clothing datasets.

We now study the sensitivity of our UGT model to the hyper-parameters. We
focus on Recall@10 for reporting the recommendation performance, since we
observe the same trends and conclusions with NDCG@10 across the used datasets.
For brevity, Figure 3 presents the effects of the hyper-parameters for the
Sports and Clothing datasets, since the results on the Baby dataset lead to
similar conclusions. We primarily analyse two important parameters in our UGT
model, namely: (i) the scaling factor $\epsilon$, which controls the influence
of an item’s intrinsic multi-modal features in the graph convolution
operations; and (ii) the contrastive factor $\lambda_{c}$, regulating the
strength of the Image-Text Contrastive (ITC) loss during training. Figure 3
and Figure 4 show the performance of UGT with different values of $\epsilon$
and $\lambda_{c}$, respectively.

####  4.4.1. Impact of the contrastive factor $\lambda_{c}$

The contrastive factor $\lambda_{c}$ indicates the importance of the
contrastively learned loss, balancing the contribution of the ITC loss within
the joint training loss in Equation (6). As mentioned in Section 4.1, we vary
$\lambda_{c}$ within the range {0.0, 0.1, …, 0.9, 1.0} with a step size of
0.1, From Figure 3, we observe that the best performance of our UGT model
occurs at $\lambda_{c}$ = 0.6, and at $\lambda_{c}$ = 0.4 on the Sports and
Clothing datasets, respectively, with a marked performance degradation at
higher $\lambda_{c}$ values. A high $\lambda_{c}$ value indicates that UGT
emphasises the item’s image-text similarity as measured by the ITC loss, over
the BPR pairwise ranking loss in the recommendation task. These results
highlight the importance of carefully choosing $\lambda_{c}$ to tailor the
loss function to the task, thereby improving the model’s performance.

####  4.4.2. Impact of the multi-modal scaling factor $\epsilon$

To improve the multi-modal fusion within our unified GNN, we introduced a
scaling factor, $\epsilon$, which incorporates each item’s own multi-modal
features into the graph convolution operations, as detailed in Equation (2).
To analyse the effect of incorporating an item’s own multi-modal features in
the graph convolution operations, we vary $\epsilon$ in the range {0.0, 0.1,
…, 0.9, 1.0} with a step size of 0.1, as previously described in Section 4.1.
From Figure 4, we observe that our UGT model reaches its peak performance on
the Sports and Clothing datasets when $\epsilon$ = 0.5 and $\epsilon$ = 0.4,
respectively. This suggests that the best value $\epsilon$ differs slightly
depending on the recommendation scenario. However, the performance drops
markedly for a higher value of $\epsilon$. These results indicate that the
inclusion of an item’s individual multi-modal features effectively enhances
the user/item embeddings within a multi-modal recommendation system. However,
excessively weighting the item’s own features (indicated by a high value of
$\epsilon$) will decrease the recommendation performance.

Note that for the results in Table 2, UGT learned effective values for
$\epsilon$ and $\lambda_{c}$ ($\epsilon$ = 0.5, $\lambda_{c}$ = 0.6 on the
Sports dataset and $\epsilon$ = 0.4, $\lambda_{c}$ = 0.4 on the Clothing
dataset). This shows that our UGT model is capable of automatically
determining suitable values for $\epsilon$ and $\lambda_{c}$ through a grid
search on the validation data (as per Figures 3 and 4). Hence, in answer to
RQ3, we find that the performance of UGT is overall fairly sensitive to the
changes in the $\epsilon$ and $\lambda_{c}$ parameters. Indeed, while our UGT
model is able to automatically learn adequate values for different datasets,
our parameter analysis shows that a careful selection of these parameters is
in general needed for optimal results.

Figure 5. The t-SNE visualisation of the item embeddings on the Sports and
Clothing datasets. The star refers to the visual embeddings while the pentagon
represents the text embeddings. The average Mean Squared Error (MSE) value
indicates the average distance between the visual and textual embeddings.

###  4.5. Modality Alignment of UGT (RQ4)

We have already shown in Table 2 that our UGT model, which addresses both the
Isolation 1 and Isolation 2 problems outperforms the existing baselines. In
this section, we analyse whether the improved performance is reflected in the
quality of features produced by our UGT model in comparison to those generated
by the baselines. In the following, we examine the most competitive baseline,
namely FREEDOM. However, we note that we observed similar trends and
conclusions with other baselines. To assess the quality of the generated
features, we measure the average Mean Squared Error (MSE) between the visual
and textual embeddings produced by UGT and FREEDOM. This metric allows to
assess how well the multi-modal embeddings align. We also use t-SNE
visualisations (Van der Maaten and Hinton, 2008) to show the visual and
textual embeddings from UGT and FREEDOM across the used datasets. Figure 5
shows the visualisations of the obtained visual and textual embeddings on the
Sports and Clothing datasets, with stars and pentagons representing the visual
and textual embeddings, respectively. Due to space limitations, we omit the
results from the Baby dataset since it exhibits the same trends and
conclusions. In Figure 5, we link the two types of embeddings with a light
grey line to show their pairwise relationship. We anticipate that higher-
quality multi-modal features will exhibit cohesive distributions and reduced
Mean Squared Error (MSE) values, indicating a successful integration of
various modalities into a unified representation. Conversely, poorer-quality
features will likely appear more dispersed and have higher MSE values,
suggesting an insufficient alignment between the modalities. From Figure 5, we
observe that the visual embeddings of items in FREEDOM are densely clustered
in the central space of the Sports dataset while their corresponding textual
embeddings are more dispersed around this central cluster. In contrast, the
UGT model produces visual and textual embeddings that are cohesively
distributed, indicating a tighter semantic space on the Sports dataset. The
markedly lower average MSE value (0.1483) for the UGT model compared to that
of FREEDOM 10.3732 on the Sports dataset further demonstrates that UGT better
aligns the visual and textual embeddings than FREEDOM. We observe similar
trends in the Clothing dataset as shown in Figure 5, where the embeddings of
UGT are cohesively distributed, in contrast to FREEDOM’s embeddings, which are
more widely scattered with their corresponding pairs. Indeed, UGT exhibits a
lower average MSE value (0.1615) compared to FREEDOM (8.4460), confirming that
UGT more effectively unifies the multi-modal embeddings into a tighter
semantic space. Overall, these observations indicate that UGT, by addressing
the issues of isolated feature extraction and modality encoding through a
unified approach, produces higher-quality multi-modal features than FREEDOM.
The tighter alignment of the visual and textual embeddings within UGT is
reflected in its improved performance, highlighting the importance of better
aligning the multi-modal data for an effective multi-modal recommendation
system.

Hence, in response to RQ4, our UGT model aligns the multi-modal embeddings
more cohesively than the strongest baseline FREEDOM, resulting in higher-
quality features and a more enhanced recommendation performance.

Supplementary Experiments: In the supplementary material of this paper, we
also conduct a user case study using our UGT model. We also analyse the
contribution of each modality to the performance of the system for both the
UGT and baseline models. We show results in the user case study demonstrating
that our UGT model identifies the visual and textual semantics of potential
items more effectively than the strongest baseline FREEDOM. Our study of the
contribution of each modality to the performance of the system shows that our
UGT model more effectively aggregates the visual and textual features in
comparison to all used baselines, thereby obtaining more effective user/item
embeddings. Due to page constraints, we include these details in the provided
supplementary material.

##  5\. Conclusions

We proposed the Unified multi-modal Graph Transformer (UGT) model to address
the problems of the isolated extraction process (Isolation 1) and the isolated
modality encoding (Isolation 2) in existing multi-modal recommendation models.
UGT uses a novel graph transformer architecture including two new key
components. Specifically, the UGT architecture addressed the isolated
extraction problem by integrating a multi-way transformer as an extraction
component before later integrating it to a new unified GNN, which serves as a
fusion component. The new unified GNN allows also to address the problem of
isolated modality encoding. This component uniformly fuses the extracted
multi-modal features. The obtained multi-modal features are then combined
using an attentive-fusion method to further enhance the final user/item
embeddings for effective multi-modal recommendation. Our results on three
benchmark datasets showed that our UGT model effectively leverages the graph
transformer architecture in a unified approach, resulting in a significant
performance improvement compared to nine strong baseline models. The
performance improvement reaches up to 13.97% in comparison to the strongest
baseline model (i.e., FREEDOM). In addition, we conducted an ablation study to
demonstrate the positive impact of each component of our UGT model on the
recommendation performance. Finally, we measured and visualised the average
MSE distances between the visual and textual embeddings of items from UGT and
FREEDOM. We found that UGT produces higher-quality features than FREEDOM,
indicating a better alignment between the modalities, and further confirming
UGT’s recommendation effectiveness.

## References

  * (1)
  * Bao et al. (2022) Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. 2022.  Vlmo: Unified vision-language pre-training with mixture-of-modality-experts. In _Proc. of NeurIPS_.
  * Cao et al. (2022) Jiangxia Cao, Xin Cong, Jiawei Sheng, Tingwen Liu, and Bin Wang. 2022.  Contrastive cross-domain sequential recommendation. In _Proc. of CIKM_. 138–147.
  * Chen et al. (2024) Sirui Chen, Jiawei Chen, Sheng Zhou, Bohao Wang, Shen Han, Chanfei Su, Yuqing Yuan, and Can Wang. 2024.  SIGformer: Sign-aware graph transformer for recommendation.  _Proc. of SIGIR_ (2024).
  * Deldjoo et al. (2020) Yashar Deldjoo, Markus Schedl, Paolo Cremonesi, and Gabriella Pasi. 2020.  Recommender systems leveraging multimedia content.  _Computing Surveys_ (2020).
  * Donahue et al. (2014) Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. 2014.  Decaf: A deep convolutional activation feature for generic visual recognition. In _Proc. of ICML_.
  * Ge et al. (2021) Xuri Ge, Fuhai Chen, Joemon M Jose, Zhilong Ji, Zhongqin Wu, and Xiao Liu. 2021.  Structured multi-modal feature embedding and alignment for image-sentence retrieval. In _Proceedings of the 29th ACM international conference on multimedia_. 5185–5193.
  * Ge et al. (2024) Xuri Ge, Songpei Xu, Fuhai Chen, Jie Wang, Guoxin Wang, Shan An, and Joemon M Jose. 2024.  3SHNet: Boosting image–sentence retrieval via visual semantic–spatial self-highlighting.  _Information Processing & Management_ 61, 4 (2024), 103716.
  * Gemmeke et al. (2017) Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. 2017.  Audio set: An ontology and human-labeled dataset for audio events. In _Proc. of ICASSP_.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep residual learning for image recognition. In _Proc. of CVPR_.
  * He and McAuley (2016) Ruining He and Julian McAuley. 2016.  VBPR: Visual bayesian personalized ranking from implicit feedback. In _Proc. of AAAI_.
  * He et al. (2020) Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. 2020.  LightGCN: Simplifying and powering graph convolution network for recommendation. In _Proc. of SIGIR_.
  * He and Wang (2023) Xuehai He and Xin Eric Wang. 2023.  Multi-modal graph transformer for multimodal question answering. In _Proc. of ACL_.
  * Kang and McAuley (2018) Wang-Cheng Kang and Julian McAuley. 2018.  Self-attentive sequential recommendation. In _Proc. of ICDM_.
  * Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014.  Adam: A method for stochastic optimization. In _Proc. of ICLR_.
  * Kolesnikov et al. (2022) Alexander Kolesnikov, Alexey Dosovitskiy, Dirk Weissenborn, Georg Heigold, Jakob Uszkoreit, Lucas Beyer, Matthias Minderer, Mostafa Dehghani, Neil Houlsby, Sylvain Gelly, Thomas Unterthiner, and Xiaohua Zhai. 2022.  An image is worth 16x16 words: Transformers for image recognition at scale. In _Proc. of ICLR_.
  * Le and Mikolov (2014) Quoc Le and Tomas Mikolov. 2014.  Distributed representations of sentences and documents. In _Proc. of ICML_.
  * Li et al. (2023b) Chaoliu Li, Lianghao Xia, Xubin Ren, Yaowen Ye, Yong Xu, and Chao Huang. 2023b.  Graph transformer for recommendation. In _Proc. of SIGIR_.
  * Li et al. (2022) Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022.  BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In _Proc. of ICML_.
  * Li et al. (2023a) Yunyi Li, Yongjing Hao, Pengpeng Zhao, Guanfeng Liu, Yanchi Liu, Victor S Sheng, and Xiaofang Zhou. 2023a.  Edge-enhanced global disentangled graph neural network for sequential recommendation.  _Transactions on Knowledge Discovery from Data_ (2023).
  * Liu et al. (2023) Qidong Liu, Jiaxi Hu, Yutian Xiao, Jingtong Gao, and Xiangyu Zhao. 2023.  Multi-modal recommender systems: A Survey.  _arXiv preprint arXiv:2302.03883_ (2023).
  * Liu et al. (2021) Yong Liu, Susen Yang, Chenyi Lei, Guoxin Wang, Haihong Tang, Juyong Zhang, Aixin Sun, and Chunyan Miao. 2021.  Pre-training graph transformer with multimodal side information for recommendation. In _Proc. of MM_.
  * Park and Lee (2022) Minju Park and Kyogu Lee. 2022.  Exploiting negative preference in content-based music recommendation with contrastive learning. In _Proc. of RecSys_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019\.  Language models are unsupervised multitask learners.  _OpenAI Blog_ (2019).
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using siamese bert-networks. In _Proc. of EMNLP-IJCNLP_.
  * Rendle et al. (2009a) Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009a.  BPR: Bayesian personalised ranking from implicit feedback. In _Proc. of UAI_.
  * Rendle et al. (2009b) Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009b.  BPR: Bayesian personalized ranking from implicit feedback. In _Proc. of UAI_.
  * Sun et al. (2019) Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019.  BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In _Proc. of CIKM_.
  * Tao et al. (2022) Zhulin Tao, Xiaohao Liu, Yewei Xia, Xiang Wang, Lifang Yang, Xianglin Huang, and Tat-Seng Chua. 2022.  Self-supervised learning for multimedia recommendation.  _Transactions on Multimedia_ (2022).
  * Van der Maaten and Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-SNE.  _Journal of machine learning research_ 9, 11 (2008).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need. In _Proc. of NeurIPS_.
  * Wang et al. (2023a) Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. 2023a.  Image as a foreign language: BEiT pretraining for vision and vision-language tasks. In _Proc. of CVPR_.
  * Wang et al. (2023b) Xiao Wang, Guangyao Chen, Guangwu Qian, Pengcheng Gao, Xiao-Yong Wei, Yaowei Wang, Yonghong Tian, and Wen Gao. 2023b.  Large-scale multi-modal pre-trained models: A comprehensive survey.  _Machine Intelligence Research_ (2023).
  * Wei et al. (2023) Yinwei Wei, Wenqi Liu, Fan Liu, Xiang Wang, Liqiang Nie, and Tat-Seng Chua. 2023.  LightGT: A light graph transformer for multimedia recommendation. In _Proc. of SIGIR_.
  * Wei et al. (2019) Yinwei Wei, Xiang Wang, Liqiang Nie, Xiangnan He, Richang Hong, and Tat-Seng Chua. 2019.  MMGCN: Multi-modal graph convolution network for personalised recommendation of micro-video. In _Proc. of MM_.
  * Wu et al. (2022) Yaxiong Wu, Craig Macdonald, and Iadh Ounis. 2022.  Multimodal conversational fashion recommendation with positive and negative natural-language feedback. In _Proc. of CCUI_.
  * Yi et al. (2023a) Zixuan Yi, Zijun Long, Iadh Ounis, Craig Macdonald, and Richard Mccreadie. 2023a.  Large multi-modal encoders for recommendation.  _arXiv preprint arXiv:2310.20343_ (2023).
  * Yi et al. (2023b) Zixuan Yi, Iadh Ounis, and Craig Macdonald. 2023b.  Contrastive graph prompt-tuning for cross-domain recommendation.  _ACM Transactions on Information Systems_ 42, 2 (2023), 1–28.
  * Yi et al. (2023c) Zixuan Yi, Iadh Ounis, and Craig Macdonald. 2023c.  Graph contrastive learning with positional representation for recommendation. In _European Conference on Information Retrieval_. Springer, 288–303.
  * Yi et al. (2024) Zixuan Yi, Xi Wang, and Iadh Ounis. 2024.  A Directional Diffusion Graph Transformer for Recommendation.  _arXiv preprint arXiv:2404.03326_ (2024).
  * Yi et al. (2022) Zixuan Yi, Xi Wang, Iadh Ounis, and Craig Macdonald. 2022.  Multi-modal graph contrastive learning for micro-video recommendation. In _Proc. of SIGIR_.
  * Ying et al. (2021) Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, and Tie-Yan Liu. 2021.  Do transformers really perform badly for graph representation. In _Proc. of NeuIPS_.
  * Zhang Jinghao et al. (2021) Liu Qiang Zhang Jinghao, Zhu Yanqiao, Wu Shu, Wang Shuhui, and Wang Liang. 2021.  Mining latent structures for multimedia recommendation. In _Proc. of MM_.
  * Zhou et al. (2023b) Hongyu Zhou, Xin Zhou, Zhiwei Zeng, Lingzi Zhang, and Zhiqi Shen. 2023b.  A comprehensive survey on multi-modal recommender systems: Taxonomy, evaluation, and future directions.  _arXiv preprint arXiv:2302.04473_ (2023).
  * Zhou and Shen (2023) Xin Zhou and Zhiqi Shen. 2023.  A tale of two graphs: Freezing and denoising graph structures for multimodal recommendation. In _Proc. of MM_.
  * Zhou et al. (2023a) Xin Zhou, Hongyu Zhou, Yong Liu, Zhiwei Zeng, Chunyan Miao, Pengwei Wang, Yuan You, and Feijun Jiang. 2023a.  Bootstrap latent representations for multi-modal recommendation. In _Proc. of WWW_.
  * Zhu et al. (2022) Wenhao Zhu, Yujun Xie, Qun Huang, Zehua Zheng, Xiaozhao Fang, Yonghui Huang, and Weijun Sun. 2022.  Graph transformer collaborative filtering method for multi-behavior recommendations.  _Mathematics_ (2022).