# On Softmax Direct Preference Optimization for Recommendation

Yuxin Chen1 Junfei Tan211footnotemark: 1 An Zhang1
Zhengyi Yang2 Leheng Sheng1 Enzhi Zhang3
Xiang Wang2 Tat-Seng Chua1
1National University of Singapore
2University of Science and Technology of China
3Hokkaido University
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>These
authors contributed equally to this work.An Zhang is the corresponding author.

###### Abstract

Recommender systems aim to predict personalized rankings based on user
preference data. With the rise of Language Models (LMs), LM-based recommenders
have been widely explored due to their extensive world knowledge and powerful
reasoning abilities. Most of the LM-based recommenders convert historical
interactions into language prompts, pairing with a positive item as the target
response and fine-tuning LM with a language modeling loss. However, the
current objective fails to fully leverage preference data and is not optimized
for personalized ranking tasks, which hinders the performance of LM-based
recommenders. Inspired by the current advancement of Direct Preference
Optimization (DPO) in human preference alignment and the success of softmax
loss in recommendations, we propose Softmax-DPO (S-DPO) to instill ranking
information into the LM to help LM-based recommenders distinguish preferred
items from negatives, rather than solely focusing on positives. Specifically,
we incorporate multiple negatives in user preference data and devise an
alternative version of DPO loss tailored for LM-based recommenders, connected
to softmax sampling strategies. Theoretically, we bridge S-DPO with the
softmax loss over negative sampling and find that it has a side effect of
mining hard negatives, which assures its exceptional capabilities in
recommendation tasks. Empirically, extensive experiments conducted on three
real-world datasets demonstrate the superiority of S-DPO to effectively model
user preference and further boost recommendation performance while mitigating
the data likelihood decline issue of DPO. Our codes are available at
https://github.com/chenyuxin1999/S-DPO.

##  1 Introduction

Recommender systems aim to predict personalized rankings based on user
preference data, _i.e.,_ historical interactions such as purchases, clicks,
and ratings [1, 2]. Recently, leveraging the extensive world knowledge and
powerful reasoning abilities of language models (LMs) [3, 4, 5], LM-based
recommenders have been broadly explored [6, 7, 8]. These recommenders convert
historical interaction data into language prompts and either perform in-
context learning or fine-tune LMs, demonstrating notable advantages, including
zero-shot and few-shot reasoning [9, 10, 11, 12], enhanced generalization
abilities [13, 14], and rich semantic understanding [15, 16, 17, 18]. However,
current LM-based recommenders typically utilize language modeling loss for
personalized ranking objectives—predicting the next token—which significantly
differs from the objective of modeling user preferences in recommendation
tasks [19, 20].

We argue that the current objective of LM-based recommenders does not fully
utilize preference data and is not optimized for personalized ranking tasks,
thereby hindering recommendation performance. Most LM-based recommenders
address recommendation tasks by leveraging specialized language prompts [13,
16, 21, 22, 23], incorporating collaborative signals as a new modality [17,
18, 24], or extending the vocabulary of LMs with item tokens [15, 25, 26, 27,
28, 29]. Typically, these recommenders pair each language prompt, including
the user’s historical interaction item lists, with a single positive item and
then update LM parameters using language modeling loss [13, 18]. Despite being
designed for recommendation tasks, these LM-based recommenders do not consider
negative items and are not directly optimized for personalized rankings. Such
a training paradigm fails to fully leverage user preference data and overlooks
the role of negative items in recommendations, thereby impeding the alignment
of LMs with user preferences.

Inspired by the success of using human-labeled data to align LMs with human
preferences [30, 31, 32] and advancements in direct preference optimization
(DPO) [33, 34, 35], we make progress on aligning LMs with recommendations by
fine-tuning them to predict the next item in accordance with the user’s
preference. This preference alignment stage aims to instill ranking
information into the LMs and help recommenders distinguish preferred items
from negatives, rather than solely focusing on positives.

Towards this end, we incorporate multiple negatives in user preference data
and devise an alternative version of DPO loss tailored for recommendation,
connected to softmax sampling strategies [19, 36, 37, 38], which we call
S-DPO. Specifically, we first devise supervised fine-tuning to inject domain
knowledge and improve LM’s ability to follow the instructions before
preference alignment phase, following [13, 32]. In the preference alignment
stage, instead of constructing solely positive pairs, we initially pair each
language prompt with both positive and randomly sampled multiple negatives to
build text-based preference data. Building upon these preference data, we
extend conventional DPO with the Bradley-Terry preference model [33, 39] on
pairwise data to the Plackett-Luce preference model [40, 41], which handles
relative rankings in recommendation tasks.

Benefiting from the use of multiple negatives in preference data, our S-DPO
offers three appealing properties. On the one hand, S-DPO serves as the first
specialized personalized ranking loss for LM-based recommenders, effectively
utilizing multiple negatives and acknowledging the importance of preference
data. Empirically, we demonstrate that it provides more effective ranking
gradients and mitigates the instability associated with DPO training (_cf._
Section 4.2). On the other hand, we theoretically bridge the DPO loss with the
traditional BPR loss [42] over pairwise data and connect S-DPO with the
softmax loss over negative sampling (also known as contrastive loss in self-
supervised recommendations, which achieves state-of-the-art performance [36,
43, 44]). This connection naturally underscores the ranking performance of
S-DPO and highlights the critical role of multiple negatives. Furthermore, we
find that S-DPO has a side effect of mining hard negative examples similar to
contrastive learning paradigm [37], which not only boosts the performance but
also accelerates the training process (_cf._ Section 3.1), assuring its
exceptional capabilities in recommendation tasks.

Overall, our contributions can be concluded as follows:

  * •

We are among the first to point out that the widely used language modeling
loss in LM-based recommendation is not designed for ranking tasks and fails to
fully utilize user preference data, thereby hindering recommendation
performance.

  * •

We propose S-DPO, an alternative version of DPO loss tailored for LM-based
recommenders, incorporating multiple negatives to instill ranking information
into LM.

  * •

We theoretically bridge S-DPO with the softmax loss over negative sampling to
highlight the critical role of multiple negatives and find its side effect of
mining hard negatives, assuring its capabilities.

Figure 1: Framework of S-DPO. Different from existing methods which fine-tune
LMs with a language modeling loss without tailoring for recommendations, S-DPO
proposes to explicitly instill ranking information into LMs. To take one step
further, S-DPO incorporates multiple negatives in user preference data and
generalizes pairwise DPO loss to softmax ranking loss.

##  2 Preliminary

In this section, we first formalize sequential recommendation as the task of
aligning language models (LMs) with user preferences. Then, we discuss the
general framework of current LM-based recommenders that utilizes language
modeling loss to fine-tune LMs. Finally, we outline the training process
widely used to align LMs with human preferences, including reinforcement
learning from human feedback (RLHF) and direct preference optimization (DPO).

#### Task Formulation.

Given the historical interactions $\mathcal{H}_{u}$ of one user $u$ in
chronological order, the goal of LM-based sequential recommender
$\mathcal{M}_{\theta}$, where $\theta$ represents trainable parameters, is to
select the item $i_{p}$ preferred by user $u$ from candidate set
$C=\\{i_{j}\\}_{j=1}^{N}$, where N is the number of candidates. This task
requires that item $i_{p}$ be preferred over the other candidate items,
denoted by $\mathcal{I}_{d}=C\backslash\\{i_{p}\\}$. This requirement
explicitly defines a multi-negative preference understanding for LM-based
recommenders, which can be formulated as follows:

$\displaystyle\forall i_{d}\in\mathcal{I}_{d},\quad i_{p}>_{u}i_{d},$ (1)

wherein $>_{u}$ stands for the preference of user $u$.

#### Fine-tuning LM-based recommenders.

Current LM-based recommenders widely adopt supervised fine-tuning (SFT) [32]
on recommendation-specific data to enhance their performance [6, 8].
Generally, this involves two steps: structuring recommendation data as text-
based pairs and then fine-tuning LMs based on these pairs. In the first step,
for user $u$, a recommendation task prompt $x_{u}$ encompasses the user’s
historical interactions $\mathcal{H}_{u}$, the candidate item set $C$, and a
description of the sequential recommendation task. This prompt $x_{u}$ is
paired with the title of the preferred item $i_{p}$ in the candidate set $C$,
denoted as $e_{p}$, to form the pair data $(x_{u},e_{p})$. In the second step,
the $(x_{u},e_{p})$ pairs are utilized to fine-tune the LM-based recommender
$\mathcal{M}_{\theta}$ through language modeling loss. This loss, commonly
used in SFT in language modeling tasks, implicitly treats the recommendation
task as predicting the next token based on preceding tokens. Formally, the
objective of optimizing the LM-based recommender $\mathcal{M}_{\theta}$ with
pair data $(x_{u},e_{p})$ can be formulated as:

$\displaystyle\underset{\theta}{\rm
max}\sum_{(x_{u},e_{p})}\sum_{t=1}^{|e_{p}|}{\rm
log}(P_{\theta}({(e_{p})}_{t}|x_{u},{(e_{p})}_{<t}),$ (2)

where $|e_{p}|$ is the number of tokens in $e_{p}$, $(e_{p})_{t}$ is the
$t$-th token of $e_{p}$ and $(e_{p})_{<t}$ is the tokens preceding
$(e_{p})_{t}$.

However, recommendation tasks are essentially user preference alignment tasks,
as formalized in the above task formulation, and differ from user modeling
tasks that consider only positive responses. Such a gap necessitates further
exploration into aligning LM-based recommenders with user preference, an area
that has been underexplored.

#### RLHF pipeline and DPO.

Recent studies in natural language processing (NLP) explore the use of human-
labeled pairwise data as a reward signal to align LMs with human preferences,
such as RLHF [32] and DPO [33]. Specifically, the RLHF [32] pipeline adds two
additional phases after the SFT phase: reward model training and reinforcement
learning (RL) optimization. After obtaining the SFT model $\pi^{\rm SFT}$,
RLHF further optimizes it with pairwise preference data.

Inspired by the success of RLHF in NLP, we leverage RLHF to inject
recommendation-specific user pairwise preference into LM-based recommenders.
Let $\mathcal{E}=\\{e_{j}\\}_{j=1}^{N}$ denote the title set of candidate
items, where $e_{j}$ denotes the title of item $i_{j}$. Given two items
$i_{j},i_{k}\in\mathcal{C}$, the user preference $i_{j}>_{u}i_{k}$ can be
seamlessly transformed into a response preference, stipulating that $e_{j}$ is
preferred over $e_{k}$ when presented with prompt $x_{u}$, denoted as
$e_{j}\succ e_{k}|x_{u}$. By sampling one dispreferred item $i_{d}$ from
dispreferred candidate set $\mathcal{I}_{d}$, we can curate a preference
dataset $\\{(e_{p},e_{d},x_{u})\\}$.

After that, RLHF utilizes a preference model for preference distribution
modeling, such as Bradley-Terry (BT) model [39]. This preference model assumes
there is a latent function $r(x_{u},e_{j})$ representing the reward of prompt-
response pair $(x_{u},e_{j})$. The bigger reward $r(x_{u},e_{j})$ means the
more user $u$ prefers item $i$. From this perspective, reward function
$r(x_{u},e_{j})$ serves as a scoring function that quantifies the preference
of user $u$ to item $i$. Besides, the preference model defines a mapping from
the reward function $r(x_{u},e_{j})$ to a preference distribution
$p_{r}(e_{j}\succ e_{k}|x_{u})$. Based on preference distribution, an optimal
reward function is trained by maximizing the likelihood of preference data.
The training objective of this phase is as follows:

$\displaystyle\mathcal{L}_{\rm RM}=-\mathbb{E}_{(x_{u},e_{p},e_{d})}[{\rm
log}\,p_{r}(e_{p}\succ e_{d}|x_{u})].$ (3)

Let $\pi_{\theta}(e|x_{u})$ be the probability that LM-based recommender
$\mathcal{M}_{\theta}$ output title $e$ given prompt $x_{u}$. The final
reinforcement learning phase aims to maximize the expected reward of policy
while not deviate too far from the reference model, formulating the following
objective for optimal policy:

$\displaystyle\underset{\pi_{\theta}}{\rm
max}\,\mathbb{E}_{x_{u}\sim\mathcal{D},e\sim\pi_{\theta}(e|x_{u})}[r(x_{u},e)]-\beta\mathbb{D}_{\rm
KL}[\pi_{\theta}(e|x_{u})||\pi_{\rm ref}(e|x_{u})],$ (4)

where $\mathcal{D}$ denotes the distribution of $x_{u}$ and $\pi_{\rm
ref}=\pi^{\rm SFT}$.

A recent study, DPO [33], theoretically proves the optimal policy in a closed
form to Eq.(4) is

$\displaystyle\pi^{*}(e|x_{u})=\frac{1}{Z(x_{u})}\pi_{\rm ref}(e|x_{u}){\rm
exp}\left(\frac{1}{\beta}r(x_{u},e)\right),$ (5)

which is equivalent to

$\displaystyle r(x_{u},e)=\beta\,{\rm log}\,\frac{\pi(e|x_{u})}{\pi_{\rm
ref}(e|x_{u})}+\beta\,{\rm log}\,Z(x_{u}),$ (6)

where $Z(x_{u})=\sum_{e}\pi_{\rm ref}(e|x_{u}){\rm
exp}\left(\frac{1}{\beta}r(x_{u},e)\right)$ is the partition function.

By defining $p_{r}(e_{p}\succ e_{d}|x_{u})$ as
$\sigma(r(x_{u},e_{p})-r(x_{u},e_{d}))$ in Eq.(3) according to the BT model
used in RLHF and substituting term $r(x_{u},e)$ in Eq.(3) with Eq.(6), the
last two phases of RLHF pipeline can be equivalently transformed into
optimizing DPO loss below:

$\displaystyle\mathcal{L}_{\rm
DPO}=-\mathbb{E}_{(x_{u},e_{p},e_{d})}\left[{\rm log}\,\sigma\left(\beta\,{\rm
log}\,\frac{\pi_{\theta}(e_{p}|x_{u})}{\pi_{\rm ref}(e_{p}|x_{u})}-\beta\,{\rm
log}\,\frac{\pi_{\theta}(e_{d}|x_{u})}{\pi_{\rm
ref}(e_{d}|x_{u})}\right)\right],$ (7)

wherein $\sigma(x)$ is the sigmoid function.

DPO is able to directly extract the optimal policy from pairwise preference
data, making it more practical for preference alignment than RLHF.
Nevertheless, DPO and RLHF are usually designed for pairwise preference. The
oversight of other negative items impedes the performance of the LM-based
recommenders. To bridge the gap, we expand DPO to S-DPO in recommendation
tasks, in consideration of multiple negative items.

##  3 Methodology

###  3.1 Derivation of S-DPO loss

To align LM-based recommender $\mathcal{M}_{\theta}$ with multi-negative
preference, we first derive the preference distribution and then propose a new
loss function called S-DPO (depicted in Figure 1).

####  Multi-negative Preference Distribution.

As mentioned in Section 2, for user $u$, there is a partial ranking
stipulating $i_{p}>_{u}i_{d},\forall\,i_{d}\in\mathcal{I}_{d}$ in sequential
recommendation tasks. Let $\mathcal{E}_{d}$ be the titles of dispreferred
items $\mathcal{I}_{d}$. The aforementioned partial ranking is equivalent to
$e_{p}\succ e_{d}|x_{u},\forall e_{d}\in\mathcal{E}_{d}$, from which a multi-
negative preference dataset $\\{x_{u},e_{p},\mathcal{E}_{d}\\}$ can be curated
in an analogous way to RLHF.

For the dataset pairing one preferred item with multiple dispreferred items,
we leverage the Plackett-Luce (PL) model [40, 41] to build preference
distribution. Given prompt $x_{u}$, $K$ titles $e_{1},e_{2},\cdots,e_{K}$ and
a permutation $\tau:[K]\rightarrow[K]$ reflecting user preference, with
$\tau(j)$ denoting the $j$-th element of permutation $\tau$, the PL model
estimates that the ranking $e_{\tau(1)},e_{\tau(2)},\cdots,e_{\tau(K)}$ turns
out true, as:

$\displaystyle
p(\tau|e_{1},e_{2},\cdots,e_{K},x_{u})=\prod_{j=1}^{K}\frac{{\rm
exp}\left(r(x_{u},e_{\tau(j)})\right)}{\Sigma_{l=j}^{K}{\rm
exp}(r\left(x_{u},e_{\tau(l)})\right)}.$ (8)

By enumerating all the permutations starting with $p$ and calculating sum of
their probability given by the PL model, the final multi-negative preference
distribution $p^{*}$ can be derived as:

$\displaystyle p^{*}(e_{p}\succ e_{d},\forall
e_{d}\in\mathcal{E}_{d}|x_{u})=\frac{{\rm
exp}(r(x_{u},e_{p}))}{\sum_{j=1}^{K}{\rm exp}(r(x_{u},e_{j}))}.$ (9)

For brevity, the complete derivation is delegated to Appendix A.1.

#### Deriving S-DPO.

By substituting reward function $r(x_{u},e)$ in Eq.(9) with Eq.(6), the mutli-
negative preference distribution can be rewritten as:

$\displaystyle p^{*}(e_{p}\succ
e_{d},\forall\,e_{d}\in\mathcal{E}_{d}|x_{u})=\frac{1}{1+\sum_{e_{d}\in\mathcal{E}_{d}}{\rm
exp}\left(\beta\,{\rm log}\,\frac{\pi(e_{d}|x_{u})}{\pi_{\rm
ref}(e_{d}|x_{u})}-\beta\,{\rm log}\,\frac{\pi(e_{p}|x_{u})}{\pi_{\rm
ref}(e_{p}|x_{u})}\right)}.$ (10)

Through plugging distribution given by Eq.(10) in the reward learning
objective in Eq.(3), our S-DPO loss can be formulated for policy
$\pi_{\theta}$ as:

$\displaystyle\mathcal{L}_{\rm S-DPO}(\pi_{\theta};\pi_{\rm
ref})=-\mathbb{E}_{(x_{u},e_{p},\mathcal{E}_{d})\sim\mathcal{D}}\left[{\rm
log}\,\sigma\left(-{\rm log}\,\sum_{e_{d}\in\mathcal{E}_{d}}{\rm
exp}\left(\beta\,{\rm log}\,\frac{\pi_{\theta}(e_{d}|x_{u})}{\pi_{\rm
ref}(e_{d}|x_{u})}-\beta\,{\rm log}\,\frac{\pi_{\theta}(e_{p}|x_{u})}{\pi_{\rm
ref}(e_{p}|x_{u})}\right)\right)\right].$ (11)

Notably, when the number of candidates $N$ is 2, which means there is only one
dispreferred item, S-DPO reduces to DPO. The proof is provided in Appendix
A.2.

#### Gradient Analysis.

We conduct gradient analysis on S-DPO. The gradient of $\mathcal{L}_{\rm
S-DPO}$ with respect to parameters $\theta$ takes the following formulation:

$\displaystyle\nabla_{\theta}\mathcal{L}_{\rm S-DPO}(\pi_{\theta};\pi_{\rm
ref})=$
$\displaystyle-\beta\mathbb{E}_{(x_{u},e_{p},\mathcal{E}_{d})}\bigg{[}\underbrace{\sigma\left({\rm
log}\sum_{e_{d}\in\mathcal{E}_{d}}{\rm
exp}(g(e_{d},e_{p},x_{u}))\right)}_{\text{higher weight when reward deviates
from preference}}\cdot\bigg{[}\nabla_{\theta}\,{\rm
log}\,\pi_{\theta}(e_{p}|x_{u})-\sum_{e_{d}\in\mathcal{E}_{d}}\frac{\nabla_{\theta}\,{\rm
log}\,\pi_{\theta}(e_{d}|x_{u})}{\underbrace{\sum_{e^{\prime}_{d}\in\mathcal{E}_{d}}{\rm
exp}(g(e^{\prime}_{d},e_{d},x_{u}))}_{\text{higher weight when reward is
larger}}}\bigg{]}\bigg{]},$

wherein $g(e_{j},e_{k},x_{u})=r_{\theta}(x_{u},e_{j})-r_{\theta}(x_{u},e_{k})$
and similar to DPO, $r_{\theta}(x_{u},e)=\beta{\rm
log}\,\frac{\pi_{\theta}(e|x_{u})}{\pi_{\rm ref}(e|x_{u})}$ is the implicit
reward function defined by $\pi_{\theta}$. See Appendix A.3 for a complete
derivation.

Recap the DPO gradient below:

$\displaystyle\nabla_{\theta}\mathcal{L}_{\rm DPO}(\pi_{\theta};\pi_{\rm
ref})=-\beta\mathbb{E}_{(x_{u},e_{p},e_{d})}\bigg{[}\underbrace{\sigma(g(e_{d},e_{p},x_{u}))}_{\text{higher
weight when reward is wrong}}\cdot\left[\nabla_{\theta}\,{\rm
log}\pi_{\theta}(e_{p}|x_{u})-\nabla_{\theta}\,{\rm
log}\pi_{\theta}(e_{d}|x_{u})\right]\bigg{]}.$

Similar to DPO, the gradient of S-DPO loss increases the likelihood of the
preferred item and decreases the likelihood of all the dispreferred items.
Each example is also weighed by how much the implicit reward $r(x_{u},e)$
deviates from the preference data. However, compared with DPO, S-DPO harnesses
information of multiple dispreferred items in this weight.

Moreover, S-DPO contains an extra weight term only assigned to the gradient of
dispreferred items. The term
$\frac{1}{\sum_{e^{\prime}_{d}\in\mathcal{E}_{d}}{\rm
exp}(g(e^{\prime}_{d},e_{d},x_{u}))}=\frac{{\rm
exp}(r_{\theta}(x_{u},e_{d}))}{\sum_{e^{\prime}_{d}\in\mathcal{E}_{d}}{\rm
exp}(r_{\theta}(x_{u},e^{\prime}_{d}))}$ mirrors the relative reward of item
$i_{d}$ in dispreferred items. The larger the reward of item $i_{d}$ is
compared with other dispreferred items, the higher the weight will be.
Therefore, for dispreferred items with larger reward, which can be considered
as hard negative items, their likelihood will decline more in the next update.
This mechanism endows S-DPO with more effectiveness and stability than DPO.
The gradient of S-DPO also tends to be larger, enabling S-DPO to converge
faster .

###  3.2 Properties of S-DPO

In this section, we will discuss the structural correlation between DPO and
BPR [42], together with S-DPO and softmax loss [38], which demonstrates the
advantage of S-DPO over DPO and language modeling objective.

For user $u$, preferred item $i_{p}$ and one dispreferred
$i_{d}\in\mathcal{I}_{d}$, BPR loss takes the form:

$\displaystyle\mathcal{L}_{\rm BPR}=-\mathbb{E}_{(u,i_{p},i_{d})}\left[{\rm
log}\,\sigma\left(f(u,i_{p})-f(u,i_{d})\right)\right],$ (12)

wherein $f(u,i)$ represents preference score of user $u$ for item $i$.

Similarly, given dispreferred item set $\mathcal{I}_{d}$, the softmax loss
takes the form:

$\displaystyle\mathcal{L}_{\rm
softmax}=-\mathbb{E}_{(u,i_{p},\mathcal{I}_{d})}\left[{\rm
log}\,\sigma\left(-{\rm log}\sum_{i_{d}\in\mathcal{I}_{d}}{\rm
exp}\left(f(u,i_{d})-f(u,i_{p})\right)\right)\right].$ (13)

Review the DPO loss in Eq.(7) and S-DPO loss in Eq.(11). Notably, term
$\beta\,{\rm log}\,\frac{\pi_{\theta}(e|x_{u})}{\pi_{\rm ref}(e|x_{u})}$ is
the implicit reward function, denoted by $r_{\theta}(x_{u},e)$ in Section 3.1.
According to Section 2, $r_{\theta}(e,x_{u})$ reflects the preference of user
$u$ to item $i$ corresponding to title $e$. When the reference model has no
knowledge about recommendation, _i.e.,_ $\pi_{\rm ref}(e|x_{u})$ is
approximately a uniform distribution, term $r_{\theta}(x_{u},e)=\beta\,{\rm
log}\,\frac{\pi_{\theta}(e|x_{u})}{\pi_{\rm ref}(e|x_{u})}$ exactly reveals
absolute preference. Hence, $r_{\theta}(x_{u},e)$ possesses a similar function
to $f(u,i)$ .

From this perspective, DPO and S-DPO can be seen as special patterns of BPR
and softmax loss, respectively. The structural correlation indicates that DPO
and S-DPO are more suitable for recommendation than the language-modeling
loss. Moreover, as softmax loss works better than BPR loss in multi-negative
scenarios [19], it can be inferred that S-DPO will be more tailored for multi-
negative user preference alignment than DPO.

##  4 Experiments

In this section, we aim to answer the following research questions:

  * •

RQ1: How does S-DPO compare with traditional and LM-based sequential
recommendation models on performance?

  * •

RQ2: How does the LM-based recommender benefit from the multiple negatives?

#### Baselines.

We thoroughly compare S-DPO with three categories of recommenders in
sequential recommendations: traditional recommenders (GRU4Rec [45], Caser
[46], SASRec [47]), LM-enhanced recommenders (MoRec [48]) and LM-based
recommenders (LLaMA2 [31], Chat-REC [14], TALLRec [13], LLaRA [18]). See
detailed introduction and comparison of baselines in Appendix B.

#### Datasets.

We conduct extensive experiments on three real-world benchmark datasets which
differ in size and domain (Movielens [49],
Goodreads111https://www.goodreads.com, and LastFM [50]). Following standard
settings of [14, 18], we employ a commonly used metric Hit Ratio@1 (HR@1) for
performance evaluation and an additional metric Valid Ratio to evaluate the
LM-based methods’ ability to generate appropriate responses. See detailed
introductions of datasets and evaluation metrics in Appendix B.

#### Implementation.

We implement all LM-based recommenders on 4 NVIDIA A100 GPUs. For all LM-based
recommenders, we conduct a supervised fine-tuning stage for a maximum of 5
epochs. For S-DPO and its variants, we conduct a preference alignment stage
for further 3 epochs. Different from existing methods, we only optimize loss
on item titles and find it effective in recommendation tasks. Refer to
Appendix B for more implementation details.

###  4.1 Overall Performance Comparison (RQ1)

Table 1 presents a comparative analysis of the performance of our proposed
S-DPO and baselines. “Rel.Ipv” denotes the relative improvement of S-DPO
compared with baselines. Bold and underlined indicate the best and the second-
best performance, respectively. We observe that:

  * •

LM-based recommenders have driven impressive performance breakthroughs
compared with traditional recommenders. Our results reveal that traditional
recommenders outperform untuned LM-based recommenders (LLaMA, ChatRec) but
fall short compared to LM-based recommenders fine-tuned on historical
interactions (TALLRec and LLaRA). It is noted that untuned LM-based
recommenders are limited by inadequate instruction-following capabilities or a
lack of domain-specific knowledge indicated by the low valid ratio and
suboptimal performance, which highlights the necessity of the supervised fine-
tuning stage to further ground the inherent ability of language models down to
sequential recommendation tasks. Moreover, MoRec also exhibits suboptimal
performance compared to its traditional variant because it leaves the
reasoning ability of LM untouched. The superior performance of recent LM-based
recommenders indicates the significant roles of knowledge and reasoning
ability in language models for recommendation tasks, which highlights the
potential of LM-based recommenders.

  * •

Tailoring language model for recommendation task further boosts the
performance of LM-based recommenders. For LM-based recommenders, the
substantial performance gap between fine-tuned and untuned approaches
emphasizes the importance of tailoring models for recommendations. TALLRec
adapts LM for recommendation by supervised fine-tuning LM on historical
interactions, surpassing traditional recommenders. Additionally, LLaRA
consistently outperformed TALLRec across all datasets, suggesting that
introducing collaborative signals through appropriate item representations is
a viable direction for further adapting LM. However, existing LM-based methods
adapt LM from either item representation methods or corpus construction,
leaving the adaptation of optimization objectives unexplored. Instead, S-DPO
aligns the language model with multi-negative user preference data by
extending DPO to include a softmax ranking loss, making it a more appropriate
loss function for recommendation tasks.

  * •

S-DPO consistently outperforms all traditional recommenders and the state-of-
the-art LM-based recommenders on all datasets. S-DPO shows an improvement
ranging from 11.10% to 47.03% on Hit Ratio@1 compared to the second-best
baseline. Building on a supervised fine-tuning stage, we attribute this
further improvement to the preference alignment stage, which explicitly
instills ranking information into LM and utilizes preference data with
multiple negative samples. Such superior performance suggests that explicitly
tailoring LM for recommendation using user preference data at the training
objective level is more effective than other LM-based recommenders. By
leveraging the inherent abilities of the LM and incorporating ranking
information from user preference data, S-DPO effectively differentiates
between preferred and less preferred items. Notably, the preference alignment
stage hardly harms the inherent ability of LM, illustrated by a high valid
ratio.

Table 1: The performance comparison on three real-world datasets. The
improvement achieved by S-DPO is significant ($p$-value $<<$ 0.05).

|  | LastFM | Goodreads | MovieLens
---|---|---|---|---
|  | HR@1 | ValidRatio | Rel.Ipv | HR@1 | ValidRatio | Rel.Ipv | HR@1 | ValidRatio | Rel.Ipv
Traditional | GRU4Rec [45] | 0.3867 | 1.0000 | 70.91% | 0.2616 | 1.0000 | 153.36% | 0.3750 | 1.0000 | 40.35%
| Caser [46] | 0.4174 | 1.0000 | 58.34% | 0.2233 | 1.0000 | 196.82% | 0.3861 | 1.0000 | 36.31%
| SASRec [47] | 0.3581 | 1.0000 | 84.56% | 0.2233 | 1.0000 | 196.82% | 0.3444 | 1.0000 | 52.82%
LM-based | LLaMA2 [31] | 0.0233 | 0.3845 | 2736.48% | 0.0246 | 0.3443 | 2594.31% | 0.0421 | 0.4421 | 1150.12%
| ChatRec [51] | 0.3306 | 1.0000 | 99.91% | 0.3770 | 1.0000 | 75.81% | 0.2000 | 0.9895 | 163.15%
| MoRec [48] | 0.2877 | 1.0000 | 129.72% | 0.1652 | 1.0000 | 301.21% | 0.2822 | 1.0000 | 86.50%
| TALLRec [13] | 0.4983 | 0.9573 | 32.63% | 0.4180 | 0.9836 | 58.56% | 0.3895 | 0.9263 | 35.12%
| LLaRA [18] | 0.5292 | 0.9950 | 24.89% | 0.4508 | 0.9918 | 47.03% | 0.4737 | 0.9684 | 11.10%
Ours | S-DPO | 0.6609 | 0.9900 | - | 0.6628 | 0.9992 | - | 0.5263 | 0.9895 | -

###  4.2 Study on S-DPO (RQ2)

#### Ablation Study.

(a) Ablation study.

(b) Study of validation loss.

(c) Study of preferred item reward.

Figure 2: Study on S-DPO. (2(a)) Ablation study of S-DPO compared with SFT and
DPO on three datasets. (2(b)) Comparison of the trend of validation loss
between DPO and S-DPO on LastFM. (2(c)) Comparison of the reward of preferred
items between DPO and S-DPO on LastFM.

To investigate the effect of explicit ranking optimization and multiple
negative samples of S-DPO, we compare it with Supervised Fine-Tuning(SFT), and
a variant of S-DPO with only a single negative sample, downgrading to pairwise
DPO loss. The experimental results are reported in Figure 2(a). We can observe
that DPO can achieve an overall better performance compared to SFT, which
underscores the effectiveness of instilling ranking relationships into
existing LM-based recommenders. With a more effective ranking gradient
provided by multiple negative samples, S-DPO can further boost performance and
achieve the best among all baseline methods and variants.

(a) Study of negative samples.

(b) Study of $\beta$ on Hit Ratio@1.

(c) Study of $\beta$ on Valid Ratio.

Figure 3: Studies on values of $\beta$ and negative samples numbers of S-DPO
on LastFM. (3(a)) Performance comparisons with varying numbers of negative
samples ($\beta$ = 1). (3(b)) Performance comparisons with varying values of
$\beta$ setting negative samples number as 3. (3(c)) Validity comparisons with
varying values of $\beta$ setting negative samples number as 3.

#### Study on number of negative samples.

Benefiting from the utilization of multiple negative pairs in preference data,
our S-DPO offers two empirically appealing properties compared to DPO: 1)
S-DPO has more effective gradients facilitating the optimization; 2) S-DPO
mitigates the data likelihood decline issue of DPO. Figure 2(b) provides the
comparison of validation loss between S-DPO and DPO, illustrating that the
loss of S-DPO decreases faster and more significantly. This observation
demonstrates that multiple negative pairs provide larger and more meaningful
gradients for model optimization, which is attributed to the inherent ability
of S-DPO to mine negative samples [37] (_cf._ Section 3.1). More results of
loss analysis can be found in Appendix C.

On the other hand, it is widely accepted that the optimization of DPO suffers
from data likelihood decline issues [52, 53], which implies that the log-
likelihood of the preferred completions is reduced below the original log-
likelihood from the reference model, hindering the performance of DPO. We,
therefore, study the behavior of S-DPO and surprisingly find that it has the
potential property of mitigating data likelihood decline issues. As
illustrated in Figure 2(c), S-DPO exhibits continually increasing rewards of
preferred items, while DPO struggles to increase the reward of preferred
items.

To further verify the superiority of the multiple negative samples of S-DPO
compared with DPO, we conduct experiments to explore the potential of the
number of negative samples, with the results depicted in Figure 3(a). It can
be observed that utilizing multiple negative samples allows the model to
achieve better performance than with a single one. Furthermore, as the number
of negative samples increases, the model’s performance exhibits continual
improvements. We attribute this success of S-DPO to more effective ranking
gradients provided by multiple negatives which can be connected to the
superior performance of contrastive loss in self-supervised recommendations
[37, 38, 54]. More results of preferred item rewards can be found in Appendix
C.

#### Study on values of $\beta$.

In S-DPO, $\beta$ is a hyperparameter controlling the deviation of LM from the
base reference policy [33]. Typically, a smaller value of $\beta$ implies that
the language model is more heavily influenced by the preference signals and
vice versa. In this section, we explore the effect of $\beta$ on S-DPO. As
indicated in Figure 3(b) through 3(c), a higher $\beta$ can achieve overall
better performance in our task, while a lower $\beta$ may overwhelm the
model’s learned knowledge from the supervised fine-tuning stage, as evidenced
by both low valid ratio and hit ratio. On the other hand, an excessively large
$\beta$ prevents the model from effectively learning ranking relationships,
leading to suboptimal performance. In all our main experiments and studies, we
set $\beta$ as 1 to achieve a balance between ranking signals and inherent
knowledge of language models.

##  5 Related Work

###  5.1 LM for Recommendation

Recent advancements in recommendation systems have increasingly incorporated
Language Models (LMs) due to their extensive knowledge and robust reasoning
abilities. This integration occurs primarily in two forms: LM-enhanced
recommenders and LM-based recommenders. LM-enhanced recommenders utilize LM
embedding as semantic representations to provide contrastive signals [55, 56,
57] or utilize LM as advanced feature extractors improving the representation
of user and item features [58, 59, 60]. However, these systems still rely on
traditional recommenders for the final recommendation task, which leaves the
reasoning ability of LM largely untouched.

On the other hand, LM-based recommenders directly employ LMs for making
recommendations. Early works leverage LMs’ in-context learning capabilities
for zero-shot or few-shot recommendations, demonstrating significant potential
[9, 10, 11, 14]. However, untuned LM-based recommenders are limited by
inadequate instruction-following capabilities and a lack of domain-specific
knowledge. To bridge this gap, recent efforts in this category include
supervised fine-tuning of LMs on the historical interactions to enhance their
performance in recommendation tasks [13, 16, 22, 23, 26]. More recently,
researchers have discovered that exploring item representation methods in the
finetuning phase may further boost LM’s ability for recommendation [29]. This
branch of works includes integrating collaborative signals [17, 18, 24, 61],
adjusting numeric representations [21, 62, 63] or introducing additional item
tokens [15, 25, 27, 28].

However, existing finetuned methods follow the training objective of language
generation without any specific adjustments for personalized ranking.
Different from them, S-DPO proposes to explicitly optimize item ranking
information on preference data.

###  5.2 Preference Alignment of Language Models

Reinforcement Learning from Human Feedback (RLHF) [30, 31, 32] is a prevalent
method of LMs to learn from human preferences. The RLHF pipeline comprises
reward model learning and reinforcement learning (RL) optimization, the latter
of which suffers instability and inefficiency. Direct Preference Optimization
(DPO) [33] bypasses the brittle RL phase via a particular reward model
parameterization and is thus simpler to implement while still keeping the
performance of RLHF.

DPO proves to be effective in many scopes, like NLP [33, 64] and multimodal
LMs [34, 65, 66, 67]. Besides, several variants have been proposed for further
improvement of DPO. $\Psi$PO [68] is a generalization of DPO loss and its
representative IPO can better overcome the problem of overfitting. ODPO [35]
treats preference pairs differently by stipulating that the likelihood gap of
two responses should be greater than a corresponding offset value. Other
variants like KTO [69], $f$-DPO [70], RSO [71], etc. also enhance DPO in
various aspects. Despite these contributions, the possibilities for leveraging
and further adapting DPO for recommendation are still largely unexplored.
Moreover, there are few studies that discuss extending DPO to handle multi-
negative scenarios.

##  6 Limitation

Despite effectiveness, there are several limitations not addressed in this
paper. On the one hand, the number of negative samples is capped at 15 in our
experiments. The potential of multiple negative samples hasn’t been fully
explored due to the limited time and computation resources. On the other hand,
increasing the number of negative examples inevitably results in higher
training costs, a phenomenon that becomes more pronounced as the number of
negative examples grows in the context of language models. Finally, despite
the empirical success and theoretical connection to softmax loss in
recommendation tasks, the profound understanding of softmax ranking loss on LM
needs to be further explored.

##  7 Conclusion

In this work, we devised a principled Softmax-DPO (S-DPO) loss specially
tailored for LM-based recommenders, utilizing multiple negatives in preference
data to explicitly instill ranking information into LM. Empirically, S-DPO
surpasses all baseline models including traditional and LM-based methods on
three datasets in sequential recommendation tasks while successfully
mitigating the data likelihood decline issue of DPO. Grounded by theoretical
proof, we bridge S-DPO with the softmax loss in self-supervised
recommendations, underscoring the ranking performance of S-DPO and
highlighting the critical roles of multiple negatives. Also, we theoretically
find that S-DPO has an inherent ability to mine hard negatives which provide
larger and more effective gradients to model optimization, assuring its
exceptional capabilities in recommendation tasks. We believe that our S-DPO,
as a generalization of DPO, provides valuable insights for future LM-based
recommenders and has the potential to benefit research fields other than
recommender systems222Thc extension to broader impact will be detailed in
Appendix D.

## References

  * Fang et al. [2020] Hui Fang, Danning Zhang, Yiheng Shu, and Guibing Guo.  Deep learning for sequential recommendation: Algorithms, influential factors, and evaluations.  _ACM Trans. Inf. Syst._ , 2020.
  * Wang et al. [2019] Shoujin Wang, Liang Hu, Yan Wang, Longbing Cao, Quan Z. Sheng, and Mehmet A. Orgun.  Sequential recommender systems: Challenges, progress and prospects.  In _IJCAI_ , 2019.
  * Zhao et al. [2023a] Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen.  A survey of large language models.  _CoRR_ , abs/2303.18223, 2023a.
  * Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners.  In _NeurIPS_ , 2020.
  * Alayrac et al. [2022] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan.  Flamingo: a visual language model for few-shot learning.  In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh, editors, _NeurIPS_ , 2022.
  * Fan et al. [2023] Wenqi Fan, Zihuai Zhao, Jiatong Li, Yunqing Liu, Xiaowei Mei, Yiqi Wang, Jiliang Tang, and Qing Li.  Recommender systems in the era of large language models (llms).  _CoRR_ , abs/2307.02046, 2023.
  * Lin et al. [2023a] Jianghao Lin, Xinyi Dai, Yunjia Xi, Weiwen Liu, Bo Chen, Xiangyang Li, Chenxu Zhu, Huifeng Guo, Yong Yu, Ruiming Tang, and Weinan Zhang.  How can recommender systems benefit from large language models: A survey.  _CoRR_ , abs/2306.05817, 2023a.
  * Wu et al. [2023] Likang Wu, Zhi Zheng, Zhaopeng Qiu, Hao Wang, Hongchao Gu, Tingjia Shen, Chuan Qin, Chen Zhu, Hengshu Zhu, Qi Liu, Hui Xiong, and Enhong Chen.  A survey on large language models for recommendation.  _CoRR_ , abs/2305.19860, 2023.
  * Liu et al. [2023a] Junling Liu, Chao Liu, Renjie Lv, Kang Zhou, and Yan Zhang.  Is chatgpt a good recommender? A preliminary study.  _CoRR_ , abs/2304.10149, 2023a.
  * Dai et al. [2023] Sunhao Dai, Ninglu Shao, Haiyuan Zhao, Weijie Yu, Zihua Si, Chen Xu, Zhongxiang Sun, Xiao Zhang, and Jun Xu.  Uncovering chatgpt’s capabilities in recommender systems.  In _RecSys_ , 2023.
  * Hou et al. [2024] Yupeng Hou, Junjie Zhang, Zihan Lin, Hongyu Lu, Ruobing Xie, Julian J. McAuley, and Wayne Xin Zhao.  Large language models are zero-shot rankers for recommender systems.  In _ECIR_ , 2024.
  * Zhang et al. [2023a] An Zhang, Leheng Sheng, Yuxin Chen, Hao Li, Yang Deng, Xiang Wang, and Tat-Seng Chua.  On generative agents in recommendation.  _CoRR_ , abs/2310.10108, 2023a.
  * Bao et al. [2023] Keqin Bao, Jizhi Zhang, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He.  Tallrec: An effective and efficient tuning framework to align large language model with recommendation.  In _RecSys_ , 2023.
  * Gao et al. [2023] Yunfan Gao, Tao Sheng, Youlin Xiang, Yun Xiong, Haofen Wang, and Jiawei Zhang.  Chat-rec: Towards interactive and explainable llms-augmented recommender system.  _CoRR_ , abs/2303.14524, 2023.
  * Zhu et al. [2023] Yaochen Zhu, Liang Wu, Qi Guo, Liangjie Hong, and Jundong Li.  Collaborative large language model for recommender systems.  _CoRR_ , abs/2311.01343, 2023.
  * Zhang et al. [2023b] Junjie Zhang, Ruobing Xie, Yupeng Hou, Wayne Xin Zhao, Leyu Lin, and Ji-Rong Wen.  Recommendation as instruction following: A large language model empowered recommendation approach.  _CoRR_ , abs/2305.07001, 2023b.
  * Yang et al. [2023] Zhengyi Yang, Jiancan Wu, Yanchen Luo, Jizhi Zhang, Yancheng Yuan, An Zhang, Xiang Wang, and Xiangnan He.  Large language model can interpret latent space of sequential recommender.  _CoRR_ , abs/2310.20487, 2023.
  * Liao et al. [2023] Jiayi Liao, Sihang Li, Zhengyi Yang, Jiancan Wu, Yancheng Yuan, and Xiang Wang.  Llara: Aligning large language models with sequential recommenders.  _CoRR_ , abs/2312.02445, 2023.
  * Rendle [2022] Steffen Rendle.  Item recommendation from implicit feedback.  In _Recommender Systems Handbook_ , pages 143–171. Springer US, 2022.
  * Xu et al. [2024] Lanling Xu, Junjie Zhang, Bingqian Li, Jinpeng Wang, Mingchen Cai, Wayne Xin Zhao, and Ji-Rong Wen.  Prompting large language models for recommender systems: A comprehensive framework and empirical analysis.  _CoRR_ , abs/2401.04997, 2024.
  * Geng et al. [2022] Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang.  Recommendation as language processing (RLP): A unified pretrain, personalized prompt & predict paradigm (P5).  In _RecSys_ , 2022.
  * Cui et al. [2022] Zeyu Cui, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang.  M6-rec: Generative pretrained language models are open-ended recommender systems.  _CoRR_ , abs/2205.08084, 2022.
  * Lin et al. [2023b] Jianghao Lin, Rong Shan, Chenxu Zhu, Kounianhua Du, Bo Chen, Shigang Quan, Ruiming Tang, Yong Yu, and Weinan Zhang.  Rella: Retrieval-enhanced large language models for lifelong sequential behavior comprehension in recommendation.  _CoRR_ , abs/2308.11131, 2023b.
  * Zhang et al. [2023c] Yang Zhang, Fuli Feng, Jizhi Zhang, Keqin Bao, Qifan Wang, and Xiangnan He.  Collm: Integrating collaborative embeddings into large language models for recommendation.  _CoRR_ , abs/2310.19488, 2023c.
  * Rajput et al. [2023] Shashank Rajput, Nikhil Mehta, Anima Singh, Raghunandan Hulikal Keshavan, Trung Vu, Lukasz Heldt, Lichan Hong, Yi Tay, Vinh Q. Tran, Jonah Samost, Maciej Kula, Ed H. Chi, and Mahesh Sathiamoorthy.  Recommender systems with generative retrieval.  In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine, editors, _NeurIPS_ , 2023.
  * Liu et al. [2024] Qijiong Liu, Nuo Chen, Tetsuya Sakai, and Xiao-Ming Wu.  ONCE: boosting content-based recommendation with both open- and closed-source large language models.  In Luz Angelica Caudillo-Mata, Silvio Lattanzi, Andrés Muñoz Medina, Leman Akoglu, Aristides Gionis, and Sergei Vassilvitskii, editors, _WSDM_ , 2024.
  * Zhai et al. [2024] Jiaqi Zhai, Lucy Liao, Xing Liu, Yueming Wang, Rui Li, Xuan Cao, Leon Gao, Zhaojie Gong, Fangda Gu, Michael He, Yinghai Lu, and Yu Shi.  Actions speak louder than words: Trillion-parameter sequential transducers for generative recommendations.  _CoRR_ , abs/2402.17152, 2024.
  * Zheng et al. [2023] Bowen Zheng, Yupeng Hou, Hongyu Lu, Yu Chen, Wayne Xin Zhao, Ming Chen, and Ji-Rong Wen.  Adapting large language models by integrating collaborative semantics for recommendation.  _CoRR_ , abs/2311.09049, 2023.
  * Hua et al. [2023] Wenyue Hua, Shuyuan Xu, Yingqiang Ge, and Yongfeng Zhang.  How to index item ids for recommendation foundation models.  In Qingyao Ai, Yiqin Liu, Alistair Moffat, Xuanjing Huang, Tetsuya Sakai, and Justin Zobel, editors, _SIGIR-AP_ , 2023.
  * Stiennon et al. [2020] Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano.  Learning to summarize with human feedback.  In _NeurIPS_ , 2020.
  * Touvron et al. [2023] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models.  _CoRR_ , abs/2307.09288, 2023.
  * Ouyang et al. [2022] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback.  In _NeurIPS_ , 2022.
  * Rafailov et al. [2023] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning, Stefano Ermon, and Chelsea Finn.  Direct preference optimization: Your language model is secretly a reward model.  In _NeurIPS_ , 2023.
  * Zhang et al. [2024] Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, and Yiming Yang.  Direct preference optimization of video large multimodal models from language model reward.  _CoRR_ , abs/2404.01258, 2024.
  * Amini et al. [2024] Afra Amini, Tim Vieira, and Ryan Cotterell.  Direct preference optimization with an offset.  _CoRR_ , abs/2402.10571, 2024.
  * Zhang et al. [2023d] An Zhang, Leheng Sheng, Zhibo Cai, Xiang Wang, and Tat-Seng Chua.  Empowering collaborative filtering with principled adversarial contrastive loss.  In _NeurIPS_ , 2023d.
  * Wu et al. [2021] Jiancan Wu, Xiang Wang, Fuli Feng, Xiangnan He, Liang Chen, Jianxun Lian, and Xing Xie.  Self-supervised graph learning for recommendation.  In _SIGIR_ , 2021.
  * van den Oord et al. [2018] Aäron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  _CoRR_ , abs/1807.03748, 2018.
  * Bradley and Terry [1952] Ralph Allan Bradley and Milton E. Terry.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  _Biometrika_ , 1952.
  * Plackett [1975] R. L. Plackett.  The analysis of permutations.  _Journal of the Royal Statistical Society. Series C (Applied Statistics)_ , 1975.
  * Luce [1959] R. Duncan Luce.  _Individual Choice Behavior: A Theoretical analysis_.  1959\.
  * Rendle et al. [2012] Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme.  BPR: bayesian personalized ranking from implicit feedback.  _CoRR_ , abs/1205.2618, 2012.
  * Yu et al. [2022a] Junliang Yu, Xin Xia, Tong Chen, Lizhen Cui, Nguyen Quoc Viet Hung, and Hongzhi Yin.  Xsimgcl: Towards extremely simple graph contrastive learning for recommendation.  _CoRR_ , abs/2209.02544, 2022a.
  * Zhang et al. [2022] An Zhang, Wenchang Ma, Xiang Wang, and Tat-Seng Chua.  Incorporating bias-aware margins into contrastive loss for collaborative filtering.  In _NeurIPS_ , 2022.
  * Hidasi et al. [2016] Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk.  Session-based recommendations with recurrent neural networks.  In _ICLR_ , 2016.
  * Tang and Wang [2018] Jiaxi Tang and Ke Wang.  Personalized top-n sequential recommendation via convolutional sequence embedding.  In _WSDM_ , 2018.
  * Kang and McAuley [2018] Wang-Cheng Kang and Julian J. McAuley.  Self-attentive sequential recommendation.  In _ICDM_ , 2018.
  * Yuan et al. [2023] Zheng Yuan, Fajie Yuan, Yu Song, Youhua Li, Junchen Fu, Fei Yang, Yunzhu Pan, and Yongxin Ni.  Where to go next for recommender systems? ID- vs. modality-based recommender models revisited.  In _SIGIR_ , 2023.
  * Harper and Konstan [2016] F. Maxwell Harper and Joseph A. Konstan.  The movielens datasets: History and context.  _ACM Trans. Interact. Intell. Syst._ , 2016.
  * Cantador et al. [2011] Iván Cantador, Peter Brusilovsky, and Tsvi Kuflik, editors.  _Proceedings of the 2nd International Workshop on Information Heterogeneity and Fusion in Recommender Systems, HetRec ’11, Chicago, Illinois, USA, October 27, 2011_ , 2011. ACM.
  * OpenAI [2023] OpenAI.  GPT-4 technical report.  _CoRR_ , abs/2303.08774, 2023.
  * Pal et al. [2024] Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, and Colin White.  Smaug: Fixing failure modes of preference optimisation with dpo-positive.  _CoRR_ , abs/2402.13228, 2024.
  * Chen et al. [2024] Huayu Chen, Guande He, Hang Su, and Jun Zhu.  Noise contrastive alignment of language models with explicit rewards.  _CoRR_ , abs/2402.05369, 2024.
  * Yu et al. [2022b] Junliang Yu, Hongzhi Yin, Xin Xia, Tong Chen, Lizhen Cui, and Quoc Viet Hung Nguyen.  Are graph augmentations necessary?: Simple graph contrastive learning for recommendation.  In _SIGIR_ , 2022b.
  * Wei et al. [2024] Wei Wei, Xubin Ren, Jiabin Tang, Qinyong Wang, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, and Chao Huang.  Llmrec: Large language models with graph augmentation for recommendation.  In Luz Angelica Caudillo-Mata, Silvio Lattanzi, Andrés Muñoz Medina, Leman Akoglu, Aristides Gionis, and Sergei Vassilvitskii, editors, _WSDM_ , 2024.
  * Ren et al. [2023] Xubin Ren, Wei Wei, Lianghao Xia, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, and Chao Huang.  Representation learning with large language models for recommendation.  _CoRR_ , abs/2310.15950, 2023.
  * Hou et al. [2023] Yupeng Hou, Zhankui He, Julian J. McAuley, and Wayne Xin Zhao.  Learning vector-quantized item representation for transferable sequential recommenders.  In _WWW_ , pages 1162–1171. ACM, 2023.
  * Xi et al. [2023] Yunjia Xi, Weiwen Liu, Jianghao Lin, Jieming Zhu, Bo Chen, Ruiming Tang, Weinan Zhang, Rui Zhang, and Yong Yu.  Towards open-world recommendation with knowledge augmentation from large language models.  _CoRR_ , abs/2306.10933, 2023.
  * Geng et al. [2024] Binzong Geng, Zhaoxin Huan, Xiaolu Zhang, Yong He, Liang Zhang, Fajie Yuan, Jun Zhou, and Linjian Mo.  Breaking the length barrier: Llm-enhanced CTR prediction in long textual user behaviors.  _CoRR_ , abs/2403.19347, 2024.
  * Chen [2023] Zheng Chen.  PALR: personalization aware llms for recommendation.  _CoRR_ , abs/2305.07622, 2023.
  * Li et al. [2023a] Xiangyang Li, Bo Chen, Lu Hou, and Ruiming Tang.  Ctrl: Connect tabular and language model for ctr prediction.  _arXiv preprint arXiv:2306.02841_ , 2023a.
  * Geng et al. [2023] Shijie Geng, Juntao Tan, Shuchang Liu, Zuohui Fu, and Yongfeng Zhang.  VIP5: towards multimodal foundation models for recommendation.  In _EMNLP (Findings)_ , 2023.
  * Lin et al. [2023c] Xinyu Lin, Wenjie Wang, Yongqi Li, Fuli Feng, See-Kiong Ng, and Tat-Seng Chua.  A multi-facet paradigm to bridge large language model and recommendation.  _CoRR_ , abs/2310.06491, 2023c.
  * Yuan et al. [2024] Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston.  Self-rewarding language models.  _CoRR_ , abs/2401.10020, 2024.
  * Zhao et al. [2023b] Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He.  Beyond hallucinations: Enhancing lvlms through hallucination-aware direct preference optimization.  _CoRR_ , abs/2311.16839, 2023b.
  * Gunjal et al. [2024] Anisha Gunjal, Jihan Yin, and Erhan Bas.  Detecting and preventing hallucinations in large vision language models.  In _AAAI_ , 2024.
  * Li et al. [2023b] Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong.  Silkie: Preference distillation for large visual language models.  _CoRR_ , abs/2312.10665, 2023b.
  * Azar et al. [2024] Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Rémi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello.  A general theoretical paradigm to understand learning from human preferences.  In _AISTATS_ , 2024.
  * Ethayarajh et al. [2024] Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela.  KTO: model alignment as prospect theoretic optimization.  _CoRR_ , abs/2402.01306, 2024.
  * Wang et al. [2023] Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, and Yuxin Chen.  Beyond reverse KL: generalizing direct preference optimization with diverse divergence constraints.  _CoRR_ , abs/2309.16240, 2023.
  * Liu et al. [2023b] Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu, and Jialu Liu.  Statistical rejection sampling improves preference optimization.  _CoRR_ , abs/2309.06657, 2023b.

##  Appendix A Mathematical Derivations

###  A.1 Deriving Preference Distribution

The PL model takes the form:

$\displaystyle
p^{*}(\tau|e_{e},i_{2},\cdots,e_{K},x_{u})=\prod_{j=1}^{K}\frac{{\rm
exp}\left(r(x_{u},e_{\tau(j)})\right)}{\Sigma_{l=j}^{K}{\rm
exp}(r\left(x_{u},e_{\tau(l)})\right)}.$ (14)

.

The ranking in multi-negative preference data is $e_{p}\succ
e_{d}|x_{u},\forall e_{d}\in\mathcal{E}_{d}$. Our new preference distribution
that estimates the probability of the ranking can be derived:

$\displaystyle p^{*}(e_{p}\succ
e_{d},\forall\,e_{d}\in\mathcal{I}_{d}|x_{u},e_{p},\mathcal{E}_{d})$
$\displaystyle=\sum_{\tau\in\\{\tau^{\prime}|\tau^{\prime}(1)=p\\}}p^{*}(\tau|x_{u},e_{p},\mathcal{E}_{d})$
$\displaystyle=\sum_{\tau\in\\{\tau^{\prime}|\tau^{\prime}(1)=p\\}}\prod_{j=1}^{K}\frac{{\rm
exp}\left(r(x_{u},e_{\tau(j)})\right)}{\sum_{l=j}^{K}{\rm
exp}\left(r(x_{u},e_{\tau(l)})\right)}$ $\displaystyle=\frac{{\rm
exp}(r(x_{u},e_{p}))}{\sum_{j=1}^{K}{\rm
exp}(r(x_{u},e_{j}))}\times\sum_{\tau^{\prime}\in
Per(ind(\mathcal{I}_{d}))}\prod_{j=1}^{K-1}\frac{{\rm
exp}\left(r(x_{u},e_{\tau^{\prime}(j)})\right)}{\sum_{l=j}^{K-1}{\rm
exp}(r(x_{u},e_{\tau^{\prime}(l)}))}$ $\displaystyle=\frac{{\rm
exp}(r(x_{u},e_{p}))}{\sum_{j=1}^{K}{\rm
exp}(r(x_{u},e_{j}))}\times\sum_{\tau^{\prime}\in
Per(ind(\mathcal{I}_{d}))}p^{*}(\tau^{\prime}|x_{u},\mathcal{E}_{d})$
$\displaystyle=\frac{{\rm exp}(r(x_{u},e_{p}))}{\sum_{j=1}^{K}{\rm
exp}\left(r(x_{u},e_{j})\right)},$

wherein $ind(\mathcal{E}_{d})$ denotes the indices of titles in
$\mathcal{E}_{d}$ and $Per(ind(\mathcal{E}_{d}))$ denotes the set of
permutations of index set $ind(\mathcal{E}_{d})$. The third equation is
because a permutation of $\\{1,2\cdots,K\\}$ starting with $p$ can be divided
into the prefix $p$ and a subsequent permutation of the rest indices, which is
exactly $Per(ind(\mathcal{E}_{d}))$.

###  A.2 Connection Between DPO and S-DPO

When $N=2$, the following equations hold:

$\displaystyle\mathcal{L}_{\rm S-DPO}(\pi_{\theta};\pi_{\rm ref})$ (Eq.(11))
$\displaystyle=-\mathbb{E}_{(x_{u},e_{p},\mathcal{E}_{d})}\left[{\rm
log}\,\sigma\left(-{\rm log}\,\sum_{e_{d}\in\mathcal{E}_{d}}{\rm
exp}\left(\beta\,{\rm log}\,\frac{\pi_{\theta}(e_{d}|x_{u})}{\pi_{\rm
ref}(e_{d}|x_{u})}-\beta\,{\rm log}\,\frac{\pi_{\theta}(e_{p}|x_{u})}{\pi_{\rm
ref}(e_{p}|x_{u})}\right)\right)\right]$
$\displaystyle=-\mathbb{E}_{(x_{u},e_{p},e_{d})}\left[{\rm
log}\,\sigma\left(-{\rm log}\,{\rm exp}\left(\beta\,{\rm
log}\,\frac{\pi_{\theta}(e_{d}|x_{u})}{\pi_{\rm ref}(e_{d}|x_{u})}-\beta\,{\rm
log}\,\frac{\pi_{\theta}(e_{p}|x_{u})}{\pi_{\rm
ref}(e_{p}|x_{u})}\right)\right)\right]$ ($N=2$)
$\displaystyle=-\mathbb{E}_{(x_{u},e_{p},e_{d})}\left[{\rm
log}\,\sigma\left(\beta\,{\rm log}\,\frac{\pi_{\theta}(e_{p}|x_{u})}{\pi_{\rm
ref}(e_{p}|x_{u})}-\beta\,{\rm log}\,\frac{\pi_{\theta}(e_{d}|x_{u})}{\pi_{\rm
ref}(e_{d}|x_{u})}\right)\right]$ $\displaystyle=\mathcal{L}_{\rm
DPO}(\pi_{\theta};\pi_{\rm ref}).$

Therefore, DPO is a special case of S-DPO.

###  A.3 Deriving the Gradient of S-DPO Loss

Let $V(\theta;e_{d})=g(e_{d},e_{p},x_{u})=\beta\,{\rm
log}\,\frac{\pi_{\theta}(e_{d}|x_{u})}{\pi_{\rm ref}(e_{d}|x_{u})}-\beta\,{\rm
log}\,\frac{\pi_{\theta}(e_{p}|x_{u})}{\pi_{\rm ref}(e_{p}|x_{u})}$ and the
S-DPO loss takes the following form:

$\displaystyle\mathcal{L}_{\rm S-DPO}(\pi_{\theta};\pi_{\rm ref})$
$\displaystyle=-\mathbb{E}_{(x_{u},e_{p},\mathcal{E}_{d})}\left[{\rm
log}\,\sigma\left(-{\rm log}\,\sum_{e_{d}\in\mathcal{E}_{d}}{\rm
exp}(V(\theta;e_{d}))\right)\right]$ (15)

The gradient of $V(\theta;e_{d})$ can be formulated as:

$\displaystyle\nabla_{\theta}\,V(\theta;e_{d})=\beta(\nabla_{\theta}\,{\rm
log}\,\pi_{\theta}(e_{d}|x_{u})-\nabla_{\theta}\,{\rm
log}\,\pi_{\theta}(e_{p}|x_{u}))$ (16)

Using properties of sigmoid function that
$\sigma^{\prime}(x)=\sigma(x)(1-\sigma(x))=\sigma(x)\sigma(-x)$ and thus
$\left(({\rm
log}\,\sigma(x)\right)^{\prime}=\frac{1}{\sigma(x)}\times\sigma(x)\sigma(-x)=\sigma(-x)$,
we have:

$\displaystyle\nabla_{\theta}\,\mathcal{L}_{\rm S-DPO}(\pi_{\theta};\pi_{\rm
ref})$
$\displaystyle=-\mathbb{E}_{(x_{u},e_{p},\mathcal{E}_{d})}\left[\nabla_{\theta}\,{\rm
log}\,\sigma\left(-{\rm log}\,\sum_{e_{d}\in\mathcal{E}_{d}}{\rm
exp}(V(\theta;e_{d}))\right)\right]$
$\displaystyle=\mathbb{E}_{(x_{u},e_{p},\mathcal{E}_{d})}\left[\sigma\left({\rm
log}\,\sum_{e_{d}\in\mathcal{E}_{d}}{\rm
exp}(V(\theta;e_{d}))\right)\cdot\nabla_{\theta}\,{\rm
log}\,\sum_{e_{d}\in\mathcal{E}_{d}}{\rm exp}(V(\theta;e_{d}))\right]$
($\left({\rm log}\sigma(x)\right)^{\prime}=\sigma(-x)$)
$\displaystyle=\mathbb{E}_{(x_{u},e_{p},\mathcal{E}_{d})}\left[\sigma\left({\rm
log}\,\sum_{e_{d}\in\mathcal{E}_{d}}{\rm
exp}(V(\theta;e_{d}))\right)\cdot\frac{\sum_{e_{d}\in\mathcal{E}_{d}}{\rm
exp}(V(\theta;e_{d}))\cdot\nabla_{\theta}\,V(\theta;e_{d})}{\sum_{e^{\prime}_{d}\in\mathcal{E}_{d}}{\rm
exp}(V(\theta;e^{\prime}_{d}))}\right]$
$\displaystyle=-\beta\mathbb{E}_{(x_{u},e_{p},\mathcal{E}_{d})}\left[\sigma\left({\rm
log}\,\sum_{e_{d}\in\mathcal{E}_{d}}{\rm
exp}(V(\theta;e_{d}))\right)\cdot\sum_{e_{d}\in\mathcal{E}_{d}}\frac{\nabla_{\theta}\,{\rm
log}\,\pi_{\theta}(e_{p}|x_{u})-\nabla_{\theta}\,{\rm
log}\,\pi_{\theta}(e_{d}|x_{u})}{\sum_{e^{\prime}_{d}\in\mathcal{E}_{d}}{\rm
exp}(V(\theta;e^{\prime}_{d})-V(\theta;e_{d}))}\right]$ (Eq. (16))
$\displaystyle=-\beta\mathbb{E}_{(x_{u},e_{p},\mathcal{E}_{d})}\left[\sigma\left({\rm
log}\,\sum_{e_{d}\in\mathcal{E}_{d}}{\rm
exp}(g(e_{d},e_{p},x_{u}))\right)\cdot\sum_{e_{d}\in\mathcal{E}_{d}}\frac{\nabla_{\theta}\,{\rm
log}\,\pi_{\theta}(e_{p}|x_{u})-\nabla_{\theta}\,{\rm
log}\,\pi_{\theta}(e_{d}|x_{u})}{\sum_{e^{\prime}_{d}\in\mathcal{E}_{d}}{\rm
exp}(g(e^{\prime}_{d},e_{d},x_{u}))}\right]$ (Definition of $V(\theta;e_{d})$)
$\displaystyle=-\beta\mathbb{E}_{(x_{u},e_{p},\mathcal{E}_{d})}\left[\sigma\left({\rm
log}\,\sum_{e_{d}\in\mathcal{E}_{d}}{\rm
exp}(g(e_{d},e_{p},x_{u}))\right)\cdot\left[\nabla_{\theta}\,{\rm
log}\,\pi_{\theta}(e_{p}|x_{u})-\sum_{e_{d}\in\mathcal{E}_{d}}\frac{\nabla_{\theta}\,{\rm
log}\,\pi_{\theta}(e_{d}|x_{u})}{\sum_{e^{\prime}_{d}\in\mathcal{E}_{d}}{\rm
exp}(g(e^{\prime}_{d},e_{d},x_{u}))}\right]\right]$

The last equation is because:

$\displaystyle\sum_{e_{d}\in\mathcal{E}_{d}}\frac{1}{\sum_{e^{\prime}_{d}\in\mathcal{E}_{d}}{\rm
exp}(g(e^{\prime}_{d},e_{d},x_{u}))}=\frac{\sum_{e_{d}\in\mathcal{E}_{d}}{\rm
exp}(V(\theta;e_{d}))}{\sum_{e^{\prime}_{d}\in\mathcal{E}_{d}}{\rm
exp}(V(\theta;e^{\prime}_{d}))}=1$

##  Appendix B Experimental Settings

###  B.1 Baselines

We compare the performance of S-DPO, against both traditional and LM-based
baselines to showcase the effectiveness of our method. Specifically, for
traditional methods, we have:

  * •

GRU4Rec [45] utilizes the GRU (Gated Recurrent Unit) architecture to model
sequences, enabling effective prediction in recommendation tasks.

  * •

Caser [46] employs both horizontal and vertical convolutional operations to
enhance the capture of high-order interactions within item sequences,
improving recommendation accuracy.

  * •

SASRec [47] incorporates a multi-head self-attention mechanism in its self-
attentive sequential recommendation model, facilitating the modeling of
intricate sequential data patterns.

For LM-enhanced method, we have:

  * •

MoRec [48] advances traditional recommendation systems by incorporating the
modality features of items instead of the id feature. we employ BERT for the
text encoder and SASRec for the recommendation architecture.

For LM-based methods, we have:

  * •

LLaMA2 [31] utilized vanilla LLaMA2-7B to directly generate recommendation
results through direct prompting.

  * •

Chat-REC [14] is implemented based on the framework discussed in [14], we
retain user interaction sequences consisting of item titles as use profiles
for a fair comparison. We use GPT4 [51] as its primary large language model.

  * •

TallRec [13] first propose to transform interaction sequences into textual
prompts and then fine-tunes large language models using domain-specific
corpus.

  * •

LLaRA [18] combines collaborative signals from traditional recommendation
systems into the fine-tuning of large language models for improved
recommendation performance.

###  B.2 Datasets

Table 2: Statistics of datasets.

Dataset | MovieLens | Goodreads | LastFM
---|---|---|---
#Sequence | 943 | 6,031 | 1,220
#Items | 1,682 | 4,500 | 4,606
#Interactions | 100,000 | 220,100 | 73,510

To evaluate the effectiveness of S-DPO, we conduct experiments on three widely
used real-world datasets: Movielens [49],
Goodreads333https://www.goodreads.com, and LastFM [50]. The statistics of
datasets are illustrated in Table 2. The MovieLens dataset is widely used for
movie recommendation tasks and includes user ratings and movie titles, we
select the MovieLens100K dataset in our experiment. Similarly, Goodreads is
sourced from a social book cataloging website, where users can explore, rate,
and review a variety of books. LastFM dataset comprises users’ listening
history and artists’ names from the Last.fm online music platform. Following
[18], we maintain their titles as textual descriptions for each dataset. For
Goodreads, we remove users and books with less than 20 interactions, which
keeps the same as the processing of MovieLens. For all datasets, we organize
sequences chronologically before dividing the data into training, validation,
and testing sets in an 8:1:1 ratio to prevent any potential information
leakage.

###  B.3 Implementation Details

We implement all approaches with Python 3.9.7, PyTorch 2.2.2, and transformers
4.39.0 on 4 NVIDIA A100s. We select Llama2-7B [31] as the LM backbone for
S-DPO. Following [18], we randomly select prompts from several prompt formats
during training and evaluation to ensure flexibility and generality. For
optimization of all the traditional methods, the Adam optimizer is employed
with a learning rate adjusted to 0.001, and a batch size configured at 256.
All models undergo L2 regularization, with coefficients experimentally
determined from [1e-3, 1e-4, 1e-5, 1e-6, 1e-7]. In all experiments involving
large language models, we train each method for a maximum of 5 epochs using a
batch size of 128 and select the checkpoint with the lowest loss on the
validation set as the final checkpoint. A warm-up strategy is applied to the
learning rate, starting at 5% of its maximum value, and gradually adjusting it
through a cosine scheduler throughout the training process. For S-DPO and all
of its ablation studies, we further conduct preference training for further 3
epochs with a batch size of 128 and a learning rate of 1e-5. Setting the value
of $\beta$ as 1, we search the number of negative samples in [3,5] for the
main results. The effects of both factors are further explored in 4.2.

###  B.4 Evaluation Metrics

Given that LMs primarily produce textual responses rather than comprehensive
item rankings, we utilize a re-ranking metric in line with previous research
[18] to assess recommendation performance. For each sequence, a candidate set
is constructed by randomly selecting 20 non-interacted items and always
includes the correct item. We assess all models based on their ability to
pinpoint the correct item within this candidate set, employing the HitRatio@1
(HR@1) metric for performance evaluation. Following [18], we also introduce an
additional metric called the Valid Ratio to evaluate the LM-based methods’
adherence to instructions and their ability to generate appropriate responses.
Due to the difficulty LMs face in producing ranked results for candidate
items, position-aware metrics like NDCG are deemed unsuitable for this
evaluation.

##  Appendix C Study of Negative Samples Number

(a) Study of valid loss on MovieLens.

(b) Study of valid loss on Goodreads.

Figure 4: Study on validation loss between S-DPO and DPO on MovieLens and
Goodreads.

Figure 4(a) through figure 4(b) provide the comparison of validation loss
between S-DPO and DPO on MovieLens and Goodreads, illustrating that the loss
of S-DPO decreases faster and more significantly. This observation is
consistent with Figure 2(b).

On the other hand, figure 5(a) through figure 5(b) showcase that S-DPO
exhibits continually increasing rewards of preferred items, while DPO
struggles to increase the reward of preferred items on both MovieLens and
Goodreads, which is aligned with Figure 2(c).

(a) Study of preferred item rewards on MovieLens.

(b) Study of preferred item rewards on Goodreads.

Figure 5: Study on validation loss between S-DPO and DPO on MovieLens and
Goodreads.

##  Appendix D Broader Impact

We left further exploration of softmax ranking loss in LM including more
negative samples and validation on various settings as future works. We
believe that S-DPO, a generalization of DPO loss has the potential to benefit
other research areas other than recommender systems. This paper presents work
whose goal is to advance the field of Machine Learning. There are many
potential societal consequences of our work, none of which we feel must be
specifically highlighted here.


# Number of ordered factorizations and recursive divisors

T. M. A. Fink  London Institute for Mathematical Sciences, Royal Institution,
21 Albermarle St, London W1S 4BS, UK

###### Abstract.

The number of ordered factorizations and the number of recursive divisors are
two related arithmetic functions that are recursively defined. But it is hard
to construct explicit representations of these functions. Taking advantage of
their recursive definition and a geometric interpretation, we derive three
closed-form expressions for them both. These expressions shed light on the
structure of these functions and their number-theoretic properties.
Surprisingly, both functions can be expressed as simple generalized
hypergeometric functions.

##  1\. Introduction

This paper is devoted to two related arithmetic functions that are recursively
defined. The first is the number of ordered factorizations into integers
greater than one. The second is the number of recursive divisors, which
measures the extent to which a number $n$ is highly divisible, whose quotients
are highly divisible, and so on.
The first function was introduced 90 years ago by Kalmar [1], and for this
reason is called $K(n)$. For example, $K(8)=4$, since 8 can be factorized in 4
ways: $8=2\cdot 4=4\cdot 2=2\cdot 2\cdot 2$. Other values of $K(n)$ are given
in Table 1. Hille [2] extended Kalmar’s results and gave them prominence.
Canfield et al. [3] and Deléglise et al. [4] studied the indices of sequence
records of $K$, that is, values of $n$ for which $K(n)>K(m)$ for all $m<n$.
Newburg and Naor [5] showed that $K$ arises in computational biology, in the
so-called probed partial digest problem, which prompted Chor et al. [6] to
study the upper bound of $K$. This bound was improved by Klazar and Luca [7],
who also considered arithmetic properties of the function.
$K(n)$ can be defined recursively as follows. Let $n=m\,d$, where $m>1$. All
of the factorizations of $n$ that begin with $m$ can be had by counting the
ordered factorizations of $d$, of which there are $K(d)$. Thus we obtain the
recursion relation $K(n)=\sum_{d\lfloor n}K(d)$, where $m\lfloor n$ means
$m|n$ and $m<n$. Along with the initial condition $K(1)=1$, this completely
determines $K(n)$. This, at least, is how $K$ was originally defined [2]. But
it is much more fruitful to embed the initial condition into the recursion
relation itself:

(1) $K(n)=\varepsilon+\sum_{d\lfloor n}K(d),$

where $\varepsilon=\lfloor 1/n\rfloor=1,0,0,0,\ldots$. As we shall see, this
lets us manipulate the defining equation without having to keep track of the
corresponding initial condition.
In contrast with $K(n)$, the second function $\kappa_{0}(n)$ is much more
recent [8, 9]. It counts the number of recursive divisors:

(2) $\kappa_{0}(n)=1+\sum_{d\lfloor n}\kappa_{0}(d).$

For example, $\kappa_{0}(8)=1+\kappa_{0}(1)+\kappa_{0}(2)+\kappa_{0}(4)=8$,
and other values of $\kappa_{0}$ are given in Table 1. $\kappa_{0}(n)$ is the
simplest case of the more general

$\kappa_{x}(n)=n^{x}+\sum_{d\lfloor n}\kappa_{x}(d),$

which was introduced as a recursive analogue of the usual divisor function
$\sigma_{x}(n)$ [8]. Like $K(n)$, $\kappa_{0}(n)$ depends only on the prime
signature of $n$—though this is not the case for other values of $x$. The
analogy with $\sigma_{x}(n)$ motivated the study of recursively perfect
numbers ($\kappa_{0}(n)=n$) and recursively abundant numbers
($\kappa_{0}(n)>n$) [9].
The two sequences $K$ and $\kappa_{0}$ are intimately related. In particular,
as we showed in [8], for $n\geq 2$, $\kappa_{0}(n)=2K(n)$. Furthermore,

$\kappa_{0}(n)=\sum_{d|n}K(d).$

Both $\kappa_{0}$ and $K$ have a geometric interpretation: $\kappa_{0}$ is the
number of squares in the divisor tree of $n$, whereas $K$ is the number of
squares of size 1 in the divisor tree of $n$ (see Fig. 1E). The divisor tree
is constructed as follows. Starting with a square of side length $n$ (Fig.
1A), the main arm of the divisor tree is made up of smaller squares with side
lengths equal to the proper divisors of $n$ (Fig. 1B). For each square in the
main arm, a secondary arm is made up of squares with side lengths equal to
that square’s proper divisors (Fig. 1C). The process is repeated, creating
sub-arms off of sub-arms, until the last sub-arms are of size 1 (Fig. 1E).
A few words on notation. As we mentioned, $m\lfloor n$ means $m|n$ and $m<n$,
that is, $m$ is a proper divisor of $n$. We denote the Dirichlet series of an
arithmetic function $f$ by $\widetilde{f}$ and the Dirichlet convolution of
two arithmetic functions $f$ and $g$ by

$\displaystyle f\star g=\sum_{d|n}f(d)\,g\left(\frac{n}{d}\right).$

Figure 1. Divisor trees. Both $\kappa_{0}(n)$ and $K(n)$ have a geometric
interpretation. The number of recursive divisors $\kappa_{0}(n)$ is the number
of squares in the divisor tree of $n$ (E), whereas the number of recursive
divisors $K(n)$ is the number of squares of size 1 (orange squares in E). As
the divisor tree is built up over successive generations, from generation
$i=0$ (A) to $i=\Omega(36)=4$ (E), the number of squares increases by
$\upsilon_{i+1}$. For $n=36$, the values of $\upsilon_{i}$ are
$\upsilon_{1}=1$, $\upsilon_{2}=8$, $\upsilon_{3}=19$, $\upsilon_{4}=18$ and
$\upsilon_{5}=6$. Then $\kappa_{0}(36)=\upsilon_{1}+\ldots+\upsilon_{5}=52$.
As expected, $K(36)=\kappa_{0}(36)/2=26$.

##  2\. Statement of results

In this paper, we give three closed form representations of $\kappa_{0}=2K$
for arbitrary $n$. As well as making it easier to compute both functions,
these representations also shed light on their structure and their number-
theoretic properties.
Let

$n=p_{1}^{\alpha_{1}}p_{2}^{\alpha_{2}}\ldots p_{\omega}^{\alpha_{\omega}}$

be the prime factorization of $n$ and let

$\Omega=\alpha_{1}+\alpha_{2}+\ldots+\alpha_{\omega}.$

######  Theorem 1.

$\displaystyle\kappa_{0}(n)=2K(n)$
$\displaystyle=\frac{1}{2}\sum_{i=0}^{\infty}\frac{1}{2^{i}}\prod_{k=1}^{\omega}\binom{\alpha_{k}+i}{\alpha_{k}}$
$\displaystyle=\textstyle{\frac{1}{2}}\,{}_{\omega}F_{\omega-1}\\!\left[\begin{matrix}\alpha_{1}+1,\ldots,\alpha_{\omega}+1\\\
1,\ldots,1\end{matrix};\frac{1}{2}\right],$

where ${}_{\omega}F_{\omega-1}$ is the generalized hypergeometric function.

######  Theorem 2.

$\displaystyle\kappa_{0}(n)=2K(n)$
$\displaystyle=\sum_{i=0}^{\Omega}\sum_{j=0}^{i}(-1)^{i-j}\binom{i}{j}\prod_{k=1}^{\omega}\binom{\alpha_{k}+j}{\alpha_{k}}.$

######  Conjecture 1.

$\displaystyle\kappa_{0}(n)=2K(n)$
$\displaystyle=2^{\alpha_{\omega}}\sum_{i_{1}=0}^{\alpha_{1}}\ldots\sum_{i_{\omega-1}=0}^{\alpha_{\omega-1}}\prod_{k=1}^{\omega-1}\binom{\alpha_{k}}{i_{k}}\binom{\alpha_{k+1}+i_{1}+\ldots+i_{k}}{\alpha_{k+1}}.$

We tested the conjecture for $n\in[1,10^{5}]$ and it is correct in all cases.
Example. The easiest way to gain some intuition for these three different
expressions is to consider an example:
$n=p_{1}^{\alpha_{1}}p_{2}^{\alpha_{2}}p_{3}^{\alpha_{3}}p_{4}^{\alpha_{4}}$.
Then $\omega=4$ and $\Omega=\alpha_{1}+\alpha_{2}+\alpha_{3}+\alpha_{4}$. The
two theorems and the conjecture give

$\displaystyle\kappa_{0}(n)$
$\displaystyle=\frac{1}{2}\sum_{i=0}^{\infty}\frac{1}{2^{i}}\binom{\alpha_{1}+i}{\alpha_{1}}\binom{\alpha_{2}+i}{\alpha_{2}}\binom{\alpha_{3}+i}{\alpha_{3}}\binom{\alpha_{4}+i}{\alpha_{4}}$
$\displaystyle\kappa_{0}(n)$
$\displaystyle=\sum_{i=0}^{\Omega}\sum_{j=0}^{i}(-1)^{i-j}\binom{i}{j}\binom{\alpha_{1}+j}{\alpha_{1}}\binom{\alpha_{2}+j}{\alpha_{2}}\binom{\alpha_{3}+j}{\alpha_{3}}\binom{\alpha_{4}+j}{\alpha_{4}}$
$\displaystyle\kappa_{0}(n)$
$\displaystyle=2^{\alpha_{4}}\\!\sum_{i_{1}=0}^{\alpha_{1}}\sum_{i_{2}=0}^{\alpha_{2}}\sum_{i_{3}=0}^{\alpha_{3}}\\!\binom{\\!\alpha_{1}\\!}{i_{1}}\binom{\\!\alpha_{2}\\!}{i_{2}}\binom{\\!\alpha_{3}\\!}{i_{3}}\binom{\\!\alpha_{2}+i_{1}\\!}{\alpha_{2}}\binom{\\!\alpha_{3}+i_{1}+i_{2}\\!}{\alpha_{3}}\binom{\\!\alpha_{4}+i_{1}+i_{2}+i_{3}\\!}{\alpha_{4}}.$

|  | _Dirichlet_ |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
|  | _series_ |  |  _Values from_ $n=1$ _to_ $n=12$ |
$K$ |  | $1/(2-\zeta(s))$ |  | 1 | 1 | 1 | 2 | 1 | 3 | 1 | 4 | 2 | 3 | 1 | 8 |  | A074206
$\kappa_{0}$ |  | $\zeta(s)/(2-\zeta(s))$ |  | 1 | 2 | 2 | 4 | 2 | 6 | 2 | 8 | 4 | 6 | 2 | 16 |  | A067824
$\tau_{1}\equiv{\bf 1}$ |  | $\zeta(s)$ |  | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |  |
$\tau_{2}$ |  | $\zeta^{2}(s)$ |  | 1 | 2 | 2 | 3 | 2 | 4 | 2 | 4 | 3 | 4 | 2 | 6 |  | A000005
$\tau_{3}$ |  | $\zeta^{3}(s)$ |  | 1 | 3 | 3 | 6 | 3 | 9 | 3 | 10 | 6 | 9 | 3 | 18 |  | A007425
$\tau_{4}$ |  | $\zeta^{4}(s)$ |  | 1 | 4 | 4 | 10 | 4 | 16 | 4 | 20 | 10 | 16 | 4 | 40 |  | A007426
$\upsilon_{1}\equiv{\bf 1}$ |  | $\zeta(s)$ |  | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |  |
$\upsilon_{2}$ |  | $\zeta(s)(\zeta(s)-1)$ |  | 0 | 1 | 1 | 2 | 1 | 3 | 1 | 3 | 2 | 3 | 1 | 5 |  | A032741
$\upsilon_{3}$ |  | $\zeta(s)(\zeta(s)-1)^{2}$ |  | 0 | 0 | 0 | 1 | 0 | 2 | 0 | 3 | 1 | 2 | 0 | 7 |  | A343879
$\upsilon_{4}$ |  | $\zeta(s)(\zeta(s)-1)^{3}$ |  | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | 0 | 0 | 3 |  |
Table 1. Arithmetic functions. For each of the arithmetic functions used in
this paper, we give its Dirichlet series, the first 12 terms of its sequence,
and its OEIS reference [10]. Notice that $\kappa_{0}(n)$ is the sum of the of
the $\upsilon_{i}$ and $\kappa_{0}(n)=2K(n)$ for $n\geq 2$.

##  3\. Proof of Theorem 1

It is convenient to rewrite (2) as

$2\kappa_{0}(n)=1+\sum_{d|n}\kappa_{0}(d),$

where now the sum is over all the divisors of $n$ rather than just the proper
divisors. We can express this in the language of Dirichlet convolutions:

$\kappa_{0}=({\bf 1}+{\bf 1}\star\kappa_{0})/2,$

where ${\bf 1}$ is the all 1s sequence, $1,1,1,\ldots$. Iterating this
recursive identity leads to the infinite series

(3) $\displaystyle\kappa_{0}$ $\displaystyle=\frac{{\bf 1}}{2}+\frac{{\bf
1}\star{\bf 1}}{2^{2}}+\frac{{\bf 1}\star{\bf 1}\star{\bf 1}}{2^{3}}+\ldots.$

We can rewrite this as

(4) $\displaystyle\kappa_{0}(n)$
$\displaystyle=\frac{\tau_{1}(n)}{2}+\frac{\tau_{2}(n)}{2^{2}}+\frac{\tau_{3}(n)}{2^{3}}+\ldots,$

where $\tau_{1}=1,1,1,\ldots$ and

$\displaystyle\tau_{i}(n)$ $\displaystyle=\sum_{d|n}\tau_{i-1}(d).$

Values of $\tau_{1}$ to $\tau_{4}$ are shown in Table 1. These quantities have
a natural interpretation: $\tau_{2}\equiv d$ is the number of divisors of $n$;
$\tau_{3}$ is the number of divisors of the divisors of $n$; and so on. It is
well known that

$\displaystyle\tau_{2}\equiv d=\prod_{k=1}^{\omega}(\alpha_{k}+1),$

and in general

(5) $\displaystyle\tau_{i}$
$\displaystyle=\prod_{k=1}^{\omega}\binom{\alpha_{k}+i-1}{i-1}.$

Substituting this into (4), we obtain the desired result, namely,

$\displaystyle\kappa_{0}(n)$
$\displaystyle=\sum_{i=1}^{\infty}\frac{1}{2^{i}}\prod_{k=1}^{\omega}\binom{\alpha_{k}+i-1}{i-1}$
$\displaystyle=\frac{1}{2}\sum_{i=0}^{\infty}\frac{1}{2^{i}}\prod_{k=1}^{\omega}\binom{\alpha_{k}+i}{i}.$

This can be expressed as a generalized hypergeometric function:

$\kappa_{0}(n)=\textstyle{\frac{1}{2}}\,{}_{\omega}F_{\omega-1}\\!\left[\begin{matrix}\alpha_{1}+1,\ldots,\alpha_{\omega}+1\\\
1,\ldots,1\end{matrix};\frac{1}{2}\right].\qed$

##  4\. Proof of Theorem 2

We know that $\kappa_{0}(n)$ is the total number of squares in the divisor
tree of $n$ (Fig. 1E). Let’s consider the construction of a divisor tree,
generation by generation, as shown in Fig. 1. Let $\upsilon_{1}(n)=1$ be the
number of squares in the root of the tree, namely, the single largest square
(Fig 1A). Let $\upsilon_{2}(n)$ be the number of squares in the main arm of
the tree, not including the root (Fig 1B). Let $\upsilon_{3}(n)$ be the number
of squares in the secondary arms, not including their roots (Fig. 1C), and so
on. The quantity $\upsilon_{i}(n)$ has a natural interpretation:
$\upsilon_{2}(n)$ is the number of proper divisors of $n$, $\upsilon_{3}(n)$
is the number of proper divisors of the proper divisors of $n$, and so on. As
with $\tau_{i}(n)$, we have $\upsilon_{1}=1,1,1,\ldots$ and

(6) $\displaystyle\upsilon_{i}(n)$ $\displaystyle=\sum_{d\lfloor
n}\upsilon_{i-1}(d).$

Values of $\upsilon_{1}$ to $\upsilon_{4}$ are shown in Table 1. We can then
express $\kappa_{0}(n)$ as the sum of the $\upsilon_{i}$ over the root and the
$\Omega(n)$ generations of arms:

(7) $\displaystyle\kappa_{0}(n)=\sum_{i=0}^{\Omega}\upsilon_{i+1}.$

The $\upsilon_{k}$ are related to the $\tau_{k}$ as follows:

$\displaystyle\upsilon_{1}$ $\displaystyle=\tau_{1}$
$\displaystyle\upsilon_{2}$ $\displaystyle=\tau_{2}-\tau_{1}$
$\displaystyle\upsilon_{3}$ $\displaystyle=\tau_{3}-2\tau_{2}+\tau_{1}$
$\displaystyle\upsilon_{4}$
$\displaystyle=\tau_{4}-3\tau_{3}+3\tau_{2}-\tau_{1},$

and in general, by the principle of inclusion and exclusion,

$\displaystyle\upsilon_{i}$
$\displaystyle=\sum_{j=0}^{i-1}(-1)^{i-1-j}\binom{i-1}{j}\tau_{j+1}.$

Substituting this into (7),

$\displaystyle\kappa_{0}(n)$
$\displaystyle=\sum_{i=0}^{\Omega}\sum_{j=0}^{i}(-1)^{i-j}\binom{i}{j}\tau_{j+1}.$

Substituting $\tau_{i}$ from (5) into this, we obtain the desired result:

$\displaystyle\kappa_{0}(n)$
$\displaystyle=\sum_{i=0}^{\Omega}\sum_{j=0}^{i}(-1)^{i-j}\binom{i}{j}\prod_{k=1}^{\omega}\binom{\alpha_{k}+j}{j}.\qed$

##  5\. Discussion

Theorem 1 tells us that, to our surprise, $\kappa_{0}(n)=2K(n)$ can be
expressed as a generalized hypergeometric function. It is in some sense the
simplest such function that can be naturally tied to the prime signature of a
number. This connection opens the door to the considerable machinery that is
known for the generalized hypergeometric function.
Theorem 1 offers some insight into the properties of $\kappa_{0}(n)$ and
$K(n)$. Our proof of it suggests a simple demonstration that, for $n\geq 2$,
$\kappa_{0}(n)=2K(n)$. As with $\kappa_{0}(n)$, it is convenient to express
(2) as

$2K(n)=\varepsilon+\sum_{d|n}K(d).$

In the language of Dirichlet convolutions, this is

(8) $K=(\varepsilon+{\bf 1}\star K)/2,$

where recall $\varepsilon=1,0,0,0,\ldots$. Iterating this leads to the
infinite series

$\displaystyle K$ $\displaystyle=\frac{\varepsilon}{2}+\frac{{\bf
1}}{2^{2}}+\frac{{\bf 1}\star{\bf 1}}{2^{3}}+\frac{{\bf 1}\star{\bf
1}\star{\bf 1}}{2^{4}}+\ldots$
$\displaystyle=\frac{1}{2}\left(\varepsilon+\frac{{\bf 1}}{2}+\frac{{\bf
1}\star{\bf 1}}{2^{2}}+\frac{{\bf 1}\star{\bf 1}\star{\bf
1}}{2^{3}}+\ldots\right)$
$\displaystyle=\left(\varepsilon+\kappa_{0}\right)/2,$

where the last step makes use of (3).
We can also readily calculate the Dirichlet series for $\kappa_{0}(n)$ and
$K(n)$. Our starting point is (4). Denoting the Dirichlet series of
$\kappa_{0}$ and $K$ by $\widetilde{\kappa}_{0}$ and $\widetilde{K}$, we can
write

$\displaystyle\widetilde{\kappa}_{0}$
$\displaystyle=\frac{\widetilde{\tau}_{1}}{2}+\frac{\widetilde{\tau}_{2}}{2^{2}}+\frac{\widetilde{\tau}_{3}}{2^{3}}+\ldots.$

Since $\tau_{1}={\bf 1}$, $\tau_{2}={\bf 1}\star{\bf 1}$, and so on, and the
Dirichlet series for ${\bf 1}$ is $\zeta(s)$, we have
$\widetilde{\tau}_{i}=\zeta^{i}(s)$. Then

$\displaystyle\widetilde{\kappa}_{0}$
$\displaystyle=\frac{\zeta(s)}{2}+\frac{\zeta(s)^{2}}{2^{2}}+\frac{\zeta(s)^{3}}{2^{3}}+\ldots$
$\displaystyle=\frac{\zeta(s)}{2-\zeta(s)}.$

As for $\widetilde{K}$, from (8), $2\widetilde{K}=1+\widetilde{\kappa}_{0}$,
so $\widetilde{K}=1/(2-\zeta(s))$.
When $n=p_{1}p_{2}\ldots p_{\omega}$ is the product of $\omega$ distinct
primes, all of the $\alpha_{i}$ equal one. Then Theorem 1 reduces to

$\displaystyle\kappa_{0}(p_{1}\ldots p_{\omega})$
$\displaystyle=\frac{1^{\omega}}{2}+\frac{2^{\omega}}{2^{2}}+\frac{3^{\omega}}{2^{3}}+\ldots$
$\displaystyle={\rm Li}_{-\omega}(1/2),$

where Li is the polylogarithm. For $\omega=1,2,3,\ldots$, this has values 2,
6, 26, 150, 1082, …. Its exponential generating function is

$\displaystyle{\rm EG}(\kappa_{0}(p_{1}\ldots p_{\omega}),x)$
$\displaystyle=\sum_{\omega=0}^{\infty}\frac{x^{\omega}}{\omega!}\sum_{i=1}^{\infty}\frac{i^{\omega}}{2^{i}}$
$\displaystyle=\sum_{i=1}^{\infty}\frac{1}{2^{i}}\sum_{\omega=0}^{\infty}\frac{(ix)^{\omega}}{\omega!}$
$\displaystyle=\sum_{i=1}^{\infty}\frac{e^{ix}}{2^{i}}$
$\displaystyle=\frac{e^{x}}{2-e^{x}}.$

MacMahon [11] derived a somewhat more complex version of Theorem 2, using a
more laborious approach. It is

$\displaystyle K(n)$
$\displaystyle=\sum_{i=1}^{\Omega}\sum_{j=0}^{i-1}(-1)^{j}\binom{i}{j}\prod_{k=1}^{\omega}\binom{\alpha_{k}+i-j-1}{\alpha_{k}}.$

In proving Theorem 2, we made use of the sequences $\upsilon_{i}$, shown in
Table 1. We can also calculate their Dirichlet series. Adding $\upsilon_{i-1}$
to (6), and turning to the language of Dirichlet convolutions,

$\displaystyle\upsilon_{i}+\upsilon_{i-1}$ $\displaystyle={\bf
1}\star\upsilon_{i-1}.$

Denoting the Dirichlet series of $\upsilon_{i}$ by $\widetilde{\upsilon}_{i}$,
this implies
$\widetilde{\upsilon}_{i}+\widetilde{\upsilon}_{i-1}=\zeta(s)\widetilde{\upsilon}_{i-1}$,
that is,

$\displaystyle\widetilde{\upsilon}_{i}$
$\displaystyle=(\zeta(s)-1)\widetilde{\upsilon}_{i-1}.$

Since $\widetilde{\upsilon}_{1}=\zeta(s)$, it follows that

$\widetilde{\upsilon}_{i}=\zeta(s)(\zeta(s)-1)^{i-1}.$

Conjecture 1, which is correct for $n\leq 10^{5}$, can be written more
symmetrically:

$\displaystyle\kappa_{0}(n)$
$\displaystyle=\sum_{i_{1}=0}^{\alpha_{1}}\ldots\sum_{i_{\omega}=0}^{\alpha_{\omega}}\prod_{k=1}^{\omega}\binom{\alpha_{k}}{i_{k}}\binom{\alpha_{k}+i_{1}+\ldots+i_{k-1}}{\alpha_{k}}.$

But since $i_{\omega}$ only appears in $\binom{\alpha_{\omega}}{i_{\omega}}$,
it sums to $2^{\alpha_{\omega}}$, giving the original form of Conjecture 1.
Since the $\alpha_{k}$ can be permuted at will, a corollary of this is that
$\kappa_{0}(n)$ is divisible by $2^{\alpha^{*}}$, where $\alpha^{*}$ is the
largest of the $\alpha_{k}$s, which we proved in [8]. This makes Conjecture 1
the most efficient of our three expressions for calculating values of $K$ and
$\kappa_{0}$ for very large values of $n$. In particular, it is useful for
calculating the indices of the sequence records of $K$ and $\kappa_{0}$—the
K-champion numbers [4] (A307866 [10]) and the recursively highly composite
numbers [8] (A333952 [10]).
There are a number of open questions about $K(n)$ and $\kappa_{0}(n)$. Here
are four.
1\. Can Conjecture 1 be proved?
2\. Can Theorem 1 be be generalized from $\kappa_{0}(n)$ to $\kappa_{x}(n)$?
3\. When $\alpha_{i}=1$ for all $i$, Theorem 1 reduces to the polylogarithm
and has a simple exponential generating function. What about when
$\alpha_{i}=j$ for all $i$?
4\. What is the significance of the sequence
$\kappa_{0}(n)/2^{\alpha^{*}(n)}$, namely, 1, 1, 1, 1, 1, 3, 1, 1, 1, 3, 1, 4,
1, 3, 3, 1, 1, 4, 1, 4, 3, 3, 1, 5, 1, 3, 1, 4, 1, 13? (The last and 30th term
distinguishes this sequence from others.)

## References

  * [1] L. Kalmar, A factorisatio numerorum probelmajarol, _Mat Fiz Lapok_ 38, 1 (1931).
  * [2] E. Hille, A problem in factorisatio numerorum, _Acta Arith_ 2, 134 (1936).
  * [3] E. Canfield, P. Erdös, C, Pomerance, On a problem of Oppenheim concerning “factorisatio numerorum”, _J Number Theory_ 17, 1 (1983).
  * [4] M. Deléglise, M. Hernane, J.-L. Nicolas, Grandes valeurs et nombres champions de la fonction arithmétique de Kalmár, _J Number Theory_ 128, 1676 (2008).
  * [5] L. Newberg, D. Naor, A lower bound on the number of solutions to the probed partial digest problem, _Adv Appl Math_ 14, 172 (1993).
  * [6] B. Chor, P. Lemke, Z. Mador, On the number of ordered factorizations of natural numbers, _Disc Math_ 214, 123 (2000).
  * [7] M. Klazar, F. Luca, On the maximal order of numbers in the factorisatio numerorum problem, _J Number Theory_ 124, 470 (2007).
  * [8] T. Fink, Recursively divisible numbers, arxiv.org/abs/1912.07979.
  * [9] T. Fink, Recursively abundant and recursively perfect numbers, arxiv.org/abs/2008.10398.
  * [10] N. J. A. Sloane, editor, The On-Line Encyclopedia of Integer Sequences, published electronically at https://oeis.org, 2018.
  * [11] P. A. MacMahon, Memoir on the theory of the compositions of numbers, _Philos T R Soc Lond A_ 184, 835 (1893).