# Accelerating Relational Database Analytical Processing with Bulk-Bitwise
Processing-in-Memory ††thanks: This work was supported by the European
Research Council through the European Union’s Horizon 2020 Research and
Innovation Programme under Grant 757259 and through the European Union’s
Horizon Europe Research and Innovation Programme under Grant 101069336.

Ben Perach Ronny Ronen Shahar Kvatinsky  Viterbi Faculty of Electrical &
Computer Engineering
Technion – Israel Institute of Technology
Haifa, Israel
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Online Analytical Processing (OLAP) for relational databases is a business
decision support application. The application receives queries about the
business database, usually requesting to summarize many database records, and
produces few results. Existing OLAP requires transferring a large amount of
data between the memory and the CPU, having a few operations per datum, and
producing a small output. Hence, OLAP is a good candidate for processing-in-
memory (PIM), where computation is performed where the data is stored, thus
accelerating applications by reducing data movement between the memory and
CPU. In particular, bulk-bitwise PIM, where the memory array is a bit-vector
processing unit, seems a good match for OLAP. With the extensive inherent
parallelism and minimal data movement of bulk-bitwise PIM, OLAP applications
can process the entire database in parallel in memory, transferring only the
results to the CPU. This paper shows a full stack adaptation of a bulk-bitwise
PIM, from compiling SQL to hardware implementation, for supporting OLAP
applications. Evaluating the Star Schema Benchmark (SSB), bulk-bitwise PIM
achieves a $4.65\times$ speedup over Monet-DB, a standard database system.

###### Index Terms:

Processing-in-memory, Database, OLAP, Memristors

##  I Introduction

Analytical processing of relational databases is a business decision support
application, allowing decision-makers to analyze their business data [1].
Often, these analyses require summarizing large sections of the database,
taking a long execution time [2], mostly on transferring data from the memory
to the CPU. Summarizing large sets of data to few results hints that this
application can benefit from processing-in-memory (PIM) techniques.

PIM techniques come in various types and technologies [3, 4, 5, 6, 7, 8, 9,
10], all operate on data at (or close to) where it is stored, i.e., the
memory. In this paper, we are interested in accelerating analytical processing
of relational databases with a specific PIM technique called bulk-bitwise PIM
[4, 5, 7, 3, 6]. Bulk-bitwise PIM is characterized by utilizing the memory
cell arrays to both process the data and directly store the result. Thus,
computing with bulk-bitwise PIM can minimize data movement. As data-intensive
applications invest most of their time and energy in memory access [11, 12,
13], our approach for accelerating analytical processing consists of reducing
the required memory accesses by the host. We do so by leveraging the in-place
processing of bulk-bitwise PIM to distill the information stored in memory,
transferring as little information as possible to the host. Data transfer
reduction is possible because databases can be transferred into PIM memory
once and used many times.

Note that using bulk-bitwise PIM in this way is orthogonal to the host
processor type (e.g., CPU, GPU, FPGA) since any host requires to get the
information from the memory. Furthermore, using bulk-bitwise PIM to reduce the
data movement out of the memory cell arrays is complementary to other PIM
techniques where dedicated processing units are placed close to the memory
arrays. These dedicated processing units can act as the hosts for the bulk-
bitwise PIM technique, profiting from the same reduced data transfer.

In this paper, we combine the thread from several of our previous works [3,
14, 15] to show how analytical processing for relational databases can be
supported with bulk-bitwise PIM. As the processing capabilities of bulk-
bitwise PIM depend on the data arrangement within the memory arrays, we show
how to arrange the database relations within the memory arrays, creating a
dedicated data structure for bulk-bitwise PIM. We then identify the basic
primitive of bulk-bitwise PIM that can reduce data transfer with this new data
structure (i.e., filter and aggregate). Afterward, we show how more complex
database operations are supported for the star schema database [16, 1] (i.e.,
GROUP-BY and JOIN), allowing execution of full queries. Using this support, we
can execute the full Star Schema Benchmark (SSB) [16]. We evaluate these
techniques with a memristive bulk-bitwise PIM design [3, 14] using the gem5
[17] simulation environment.

##  II Background

###  II-A Relational Databases and Analytical Processing

The relational database is a data model organized into one or more relations
(tables). Each relation is constructed as multiple records and attributes
(shown in Fig. 1(a)), represented by the relation’s rows and columns,
respectively. Records are independent, holding information belonging to a
single item with a single value for each attribute. Each relation has an
attribute (or a set of attributes) that uniquely identifies the records in the
relation and is called the key of that relation. When a relation has an
attribute that has values from a key of another relation, this attribute is
called a foreign key.

Queries on the database are questions about the data held in the database. In
analytical processing, queries require finding all records fulfilling certain
conditions on their attributes and summarizing (e.g., sum, average, max) one
or more attributes across the found records [2, 1]. This summarizing can also
be requested per subgroups of the found records, where subgroups are defined
according to unique values of some attributes. This division for subgroups is
called a GROUP-BY operation.

When a query includes conditions involving attributes from multiple relations,
records from the different relations are matched according to these
conditions. The operation of matching records is called a JOIN. When the
condition between the relations’ attributes for JOIN is equality, the JOIN is
named equi-JOIN. For analytical processing queries, JOIN operations are
usually the most time-consuming part of query execution [18].

A common database structure for analytical processing is the star schema [1,
16]. An illustration of the star schema structure is shown in Fig. 2. In this
schema, there is a single large relation, called the fact relation, and
several small relations called the dimension relations. The fact relation has
a foreign key to each of the dimension relations. In the star schema, JOIN
operations required by queries are, by and large, only equi-JOIN between a
dimension relation’s key and its respective fact relation foreign key [1]. All
the SSB benchmark [16] queries use only this kind of JOIN.

(a)

(b)

Figure 1: (a) A database relation. (b) The embedding data structure within a
memory cell array. Figure 2: A star schema illustration.

###  II-B Bulk-Bitwise PIM

Bulk-bitwise PIM uses the memory cell arrays as processing units, which can be
implemented with DRAM or emerging nonvolatile memory technologies [4, 5, 7, 3,
6]. Because of the regular structure of these arrays, the supported operations
are bitwise logic operations (e.g., AND, NOT, NOR) between array rows or
columns (shown in Fig. 3). When multiple memory cell arrays operate
concurrently, the effective operation is a very wide bitwise operation, i.e.,
bulk-bitwise operations. As such, bulk-bitwise PIM can process data where it
is stored and exhibit high computational bandwidth.

To support virtual memory, bulk-bitwise PIM operations are restricted to use
and rewrite data within a single virtual page [3], usually a huge page. This
way, when a program sends a PIM operation to memory with a virtual address,
the virtual address can be translated in the standard fashion and the PIM
operation is routed by the hardware to its designated place. To perform the
same operation on several pages, however, the same operation has to be sent to
each page separately. In addition, since the PIM computation is tightly
coupled with the layout of data in the memory cell arrays, data for PIM has to
be structured in a specific, dedicated way. To allow software in virtual
memory fine-grain control over the layout of data structures within the cell
arrays, the mapping of addresses to cell array location is part of the bulk-
bitwise programming model [3]. The specified mapping is on the page offset
bits of the virtual address since they do not change on virtual-physical
address translation, giving the virtual space software control over these
bits.

To guarantee program correctness, the ordering rules of PIM operations with
the standard memory operations (e.g., loads, stores) have to be defined [14].
Having well-defined ordering rules for bulk-bitwise PIM requires the PIM
memory and host caches to be coherent, supported by the host hardware [14].

Figure 3: A $3\times 4$ memory array with a column-wise logic operation.

##  III The Relation Data Structure

A database relation is stored in a memory cell array, as shown in Fig. 1. It
is necessary to store data in a certain way so it can be directly processed
without further movement since the processing capabilities of bulk-bitwise PIM
are directly connected to the layout of data within the memory cell array. To
this end, each record is set across a cell array row, and each attribute spans
several columns. In this data structure, column-wise operations can be used to
process all records in parallel. Some cell array columns, however, have to
remain empty to store the PIM results before they are read from the memory.
Note that a record’s bytes are not consecutive in virtual address space due to
the address mapping.

If a single array row is not enough to hold all the attributes of a relation,
the relation’s attributes must be split on more than a single cell array.
These additional cell arrays are put on different pages and might require
moving data between the pages using standard loads and stores.

##  IV Bulk-Bitwise PIM Primitives

This section presents the basic database operations using the relation data
structure and bulk-bitwise PIM capabilities. These database primitives reduce
the required data transfer between the host and memory.

###  IV-A Filter

A filter operation, shown in Fig. 4(a), checks a condition across all relation
records. This operation filters all records for a query or a single subgroup
in a GROUP-BY operation (Section V-B). The condition checking is done with
PIM, where the result is a single bit per record. The resulting bit is stored
in the same cell array column across all cell arrays of the relation. Hence,
to assert which record passed the condition, the host only needs to read a
single bit per record instead of the condition’s attributes per record. The
data transfer reduction depends on the number of attributes in the condition,
the attributes’ lengths, and the data itself (as non-PIM techniques are data
depended [2]).

To support the filter primitive, the PIM module instruction set includes
comparison operations (e.g., equality, less-than), logic operations (e.g.,
AND, OR, NOT), and arithmetic operations (e.g., addition, multiplication).
These operations are supported for both an attribute with an attribute and an
attribute with an immediate. Additionally, these operations are supported for
multiple attribute lengths.

(a)

(b)

Figure 4: Illustration of PIM primitives: (a) filter and (b) aggregation. The
results are marked with red cells.

###  IV-B Aggregation

The aggregation operation, shown in Fig. 4(b), reduces a specific relation
attribute, across all of the relations, to a single value (e.g., sum, average,
max). This reduction, however, includes values only from selected records.
This selection is done by first filtering the records according to the
condition (as in the filter primitive in Section IV-A). Instead of reading the
filter result, it is used to generate a masked version of the attribute to be
aggregated (without overwriting the original attribute). The mask operation
nullifies the non-selected values so they will not affect the aggregation
operation. The masked attribute is then aggregated, according to the desired
operation, within each cell array, resulting in a single value per cell array.
Afterward, the host reads the single value from each cell array and aggregates
them to complete the operation. Hence, the host only has to read a single
value per cell array per aggregation.

For example, when summing an attribute, the attribute’s bits are ANDed with
the filter result bit to create the masked attribute. Hence, the unselected
records have a zero value in the masked attribute, while the selected records’
values remain the same. Summing the masked attribute will have the same result
as summing only the attribute for the selected records. The host reads the
single values in each cell array and sums them together, producing the final
result.

Since the aggregation operations are performed value by value, they must be
commutative and associative. Hence, the PIM module supports the aggregation
operation for sum, min, and max within each cell array. Other aggregation
operations can be supported as a combination of commutative and associative
aggregation. For example, to perform an average (a non-associative operation)
on an attribute, the attribute is first summed. Then, the filter result is
also aggregated using sum, resulting in the count of the selected records. The
host then divides the total sum of the records by their count to produce the
average.

###  IV-C Evaluating PIM Primitives

To evaluate the performance of the PIM primitives, we used the TPC-H benchmark
[19], a relation database analytical processing benchmark. The baseline
compared to is an in-house implementation of an in-memory database [2], where
the entire database is stored in DRAM main memory. The baseline ran on the
same system as with PIM and executed the same query section as the PIM system.
A full evaluation description and more analysis are presented in [3].

(a)

(b)

Figure 5: Primitive operation speedup and data transfer reduction (last-level
cache misses) on the TPC-H benchmark. (a) Filter-only and (b) full queries.

The speedup and memory accesses reduction (LLC misses) for the PIM over the
baseline are shown in Fig. 5. The queries are divided into two groups, full
queries and filter-only queries. Full queries can perform the entire query,
including the required aggregations, while filter-only queries can perform
only the filter part of the query. Since the aggregation performs a more
substantial data transfer reduction, it also achieves a significantly higher
speedup. We also see that the data transfer reduction is similar in magnitude
to the speedup, supporting our approach that bulk-bitwise PIM speedup comes
from the data transfer reduction.

##  V Supporting the Star Schema

Using the PIM database primitives and understanding their strengths and
weaknesses, more complex operations can be designed and supported. This
section presents the support for JOIN and GROUP-BY for the star schema. Full
details and full evaluation are presented in [15].

###  V-A Supporting JOIN

JOIN requires matching records from different relations, where the matching
itself is data-dependent. Hence, performing JOIN requires many data movements,
which conflicts with the goal of data movement reduction. To avoid this, JOIN
is accomplished by pre-computing the required JOIN output and storing the
relations as a single JOINed relation. Pre-computing a JOIN operation is a
known technique to accelerate query execution, appearing as denormalization
[20] or materialized views [21]. Storing a pre-computed JOIN, however, has
drawbacks. Characteristics of the star schema and bulk-bitwise PIM can
mitigate these drawbacks.

Figure 6: Execution time relative to one_xb for the Star Schema Benchmark
(SSB) benchmark queries.

A significant drawback of pre-computed JOINs is their flexibility. If a query
requires a different JOIN than the one pre-computed, then the pre-computation
is not helpful, and the required JOIN has to be performed. For the star
schema, most JOIN operations are an equi-JOIN between a dimension relation and
the fact relation on the dimension key [1, 16]. All of the JOIN operations of
the SSB [16] benchmark are of this type. Hence, we cover and accelerate the
vast majority of cases by storing the fact relation equi-JOINed to all
dimension relations on their key.

Using pre-computed JOINs also complicates maintenance [20, 21]. Since JOIN
operations duplicate data, performing an UPDATE on a pre-computed JOIN
requires changing many entries. Using bulk-bitwise PIM, however, alleviates
this drawback. The filter primitive can efficiently find all records to
update. Furthermore, PIM operations can perform the update itself, not
requiring reading any value from PIM memory (i.e., implementing a PIM MUX with
the filter result as select [15]).

###  V-B Supporting GROUP-BY

Performing GROUP-BY can be simply done by using PIM aggregation for every
subgroup, aggregating subgroups one by one. The number of subgroups, however,
can be large. Performing many aggregations may result in long latency, high
energy, and high power, and reduce the lifetime of the PIM module due to the
limited endurance of memory cells [3].

To mitigate these deficiencies of PIM aggregation, a standard CMOS logic
circuit is added to the memory cell array peripherals. The circuit performs
the aggregation by reading the masked attribute to be aggregate value by value
from its cell array. The circuit aggregates the read values and writes the
final result back to the cell array. Thus, the latency, energy, and power of
the aggregation primitive are reduced compared to a pure bulk-bitwise PIM
aggregation. Furthermore, the PIM module lifetime is increased, as cells are
written only at the end of the aggregation operation.

Nevertheless, the latency of the entire GROUP-BY operation can still be high
if there are many subgroups. We note that aggregation can be performed in
another way. The records for the query (for all subgroups) can be filtered,
then read record-by-record by the host. The host then classifies and
aggregates the records in their subgroups. This host aggregation technique is
dependent on the number of records in the query and is independent of the
number of subgroups. PIM aggregation, however, depends on the number of
subgroups and not the number of records in the query. Hence, it is beneficial
to have PIM aggregating a few large subgroups, leaving many small subgroups to
host aggregation, exploiting the strength of each method. To divide the
subgroups between PIM and host aggregation, a performance model is used along
with an estimate for subgroups’ sizes [15]. This estimate is done by taking a
small sample from the database. This GROUP-BY technique is adapted from an in-
cloud processing work [22].

###  V-C Star Schema Evaluation

We evaluate our solutions on the SSB benchmark [16]. The execution times are
shown in Fig. 6. We compared three versions of PIM. one-xb stores records of
the JOINed relation in a single memory cell array. two-xb is where the JOINed
relation’s records are vertically split into two memory cell arrays, having
the fact relation’s attributes in one array and all of the dimension
relations’ attributes in another array. pimdb is the same as one-xb, only the
PIM aggregation (PIM-gb) is performed with pure bulk-bitwise operations
(without the added aggregation circuit). Two baselines are compared, mnt-reg
and mnt-join. These baselines run Monet-DB [23], a real in-memory database
system running on real hardware. The mnt-reg and mnt-join hold the SSB
database in its standard and pre-computed JOIN, respectively.

The best execution time is for one-xb, having a geo-mean speedup of
$7.46\times$ and $4.65\times$ over mnt-reg and mnt-join, respectively. two-xb
is $3.39\times$ slower (in geo-mean) than one-xb since many data transfers are
required between the memory cell arrays of the relation. two-xb, however,
still has $1.37\times$ speedup on mnt-join. The cases where a non-PIM
execution is faster than a PIM execution (Q2.1, Q3.1, and Q4.1) are where the
PIM execution does not achieve a significant data transfer reduction. In these
cases, the number of records required by the query is high. Due to the data
structure of the PIM relation, most of the relation’s records are transferred
to the host, resulting in little to no data reduction and removing the
advantage of PIM. See [15] for more details and discussions.

##  VI Conclusions

This paper shows how to support analytical processing of relational databases
using bulk-bitwise PIM. Our bulk-bitwise PIM technique aimed to reduce the
required data transfers. By substituting serial data accesses to memory with
very wide and short operations within the memory, we achieve a significant
speedup over von Neumann machines.

We first designed a data structure suited for bulk-bitwise PIM. Then, we
identified and supported primitive operations (filter and aggregate),
performing relevant functionality and reducing data transfer. These primitives
were evaluated and studied. Based on these primitives, we support more complex
operations (JOIN and GROUP-BY) and evaluated a full database benchmark (SSB)
for a system based on memristive bulk-bitwise PIM. We believe this work will
inspire other research for further adaptation of applications for bulk-bitwise
PIM.

## References

  * [1] R. Kimball and M. Ross, _The Data Warehouse Toolkit: The Definitive Guide to Dimensional Modeling_ , 3rd ed. Wiley Publishing, 2013.
  * [2] H. Plattner, _A Course in In-Memory Data Management: The Inner Mechanics of In-Memory Databases_. Springer Publishing Company, Incorporated, 2013.
  * [3] B. Perach, R. Ronen, B. Kimelfeld, and S. Kvatinsky, “PIMDB: Understanding Bulk-Bitwise Processing In-Memory Through Database Analytics,” 2022. [Online]. Available: https://arxiv.org/abs/2203.10486
  * [4] V. Seshadri, D. Lee, T. Mullins, H. Hassan, A. Boroumand, J. Kim, M. A. Kozuch, O. Mutlu, P. B. Gibbons, and T. C. Mowry, “Ambit: In-memory accelerator for bulk bitwise operations using commodity dram technology,” in _2017 50th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO)_ , 2017, pp. 273–287.
  * [5] N. Hajinazar, G. F. Oliveira, S. Gregorio, J. a. D. Ferreira, N. M. Ghiasi, M. Patel, M. Alser, S. Ghose, J. Gómez-Luna, and O. Mutlu, “Simdram: A framework for bit-serial simd processing using dram,” in _Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems_ , ser. ASPLOS ’21. New York, NY, USA: Association for Computing Machinery, 2021, p. 329–345. [Online]. Available: https://doi.org/10.1145/3445814.3446749
  * [6] S. Resch, S. K. Khatamifard, Z. I. Chowdhury, M. Zabihi, Z. Zhao, H. Cilasun, J. P. Wang, S. S. Sapatnekar, and U. R. Karpuzcu, “MOUSE: Inference In Non-volatile Memory for Energy Harvesting Applications,” in _2020 53rd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO)_ , 2020, pp. 400–414.
  * [7] S. Li, C. Xu, Q. Zou, J. Zhao, Y. Lu, and Y. Xie, “Pinatubo: A processing-in-memory architecture for bulk bitwise operations in emerging non-volatile memories,” in _2016 53nd ACM/EDAC/IEEE Design Automation Conference (DAC)_ , 2016, pp. 1–6.
  * [8] V. Seshadri, Y. Kim, C. Fallin, D. Lee, R. Ausavarungnirun, G. Pekhimenko, Y. Luo, O. Mutlu, P. B. Gibbons, M. A. Kozuch, and T. C. Mowry, “RowClone: Fast and energy-efficient in-DRAM bulk data copy and initialization,” in _2013 46th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO)_ , 2013, pp. 185–197.
  * [9] J. H. Kim, S.-h. Kang, S. Lee, H. Kim, W. Song, Y. Ro, S. Lee, D. Wang, H. Shin, B. Phuah, J. Choi, J. So, Y. Cho, J. Song, J. Choi, J. Cho, K. Sohn, Y. Sohn, K. Park, and N. S. Kim, “Aquabolt-xl: Samsung hbm2-pim with in-memory processing for ml accelerators and beyond,” in _2021 IEEE Hot Chips 33 Symposium (HCS)_ , 2021, pp. 1–26.
  * [10] J. Reuben, R. Ben-Hur, N. Wald, N. Talati, A. H. Ali, P.-E. Gaillardon, and S. Kvatinsky, “Memristive logic: A framework for evaluation and comparison,” in _2017 27th International Symposium on Power and Timing Modeling, Optimization and Simulation (PATMOS)_ , 2017, pp. 1–8.
  * [11] W. A. Wulf and S. A. McKee, “Hitting the memory wall: Implications of the obvious,” _ACM SIGARCH Computer Architecture News_ , vol. 23, no. 1, p. 20–24, 1995. [Online]. Available: https://doi.org/10.1145/216585.216588
  * [12] J. Langguth, X. Cai, and M. Sourouri, “Memory bandwidth contention: Communication vs computation tradeoffs in supercomputers with multicore architectures,” in _2018 IEEE 24th International Conference on Parallel and Distributed Systems (ICPADS)_ , 2018, pp. 497–506.
  * [13] A. Pedram, S. Richardson, M. Horowitz, S. Galal, and S. Kvatinsky, “Dark memory and accelerator-rich system optimization in the dark silicon era,” _IEEE Design & Test_, vol. 34, no. 2, pp. 39–50, 2017.
  * [14] B. Perach, R. Ronen, and S. Kvatinsky, “On consistency for bulk-bitwise processing-in-memory,” in _2023 IEEE International Symposium on High-Performance Computer Architecture (HPCA)_ , 2023, pp. 705–717.
  * [15] ——, “Enabling relational database analytical processing in bulk-bitwise processing-in-memory,” 2023. [Online]. Available: https://arxiv.org/abs/2302.01675
  * [16] P. O’Neil _et al._ , _The Star Schema Benchmark and Augmented Fact Table Indexing_. Berlin, Heidelberg: Springer-Verlag, 2009, p. 237–252. [Online]. Available: https://doi.org/10.1007/978-3-642-10424-4_17
  * [17] N. Binkert _et al._ , “The gem5 simulator,” _SIGARCH Comput. Archit. News_ , vol. 39, no. 2, p. 1–7, August 2011.
  * [18] M. Dreseler, M. Boissier, T. Rabl, and M. Uflacker, “Quantifying tpc-h choke points and their optimizations,” _Proc. VLDB Endow._ , vol. 13, no. 8, p. 1206–1220, 2020. [Online]. Available: https://doi.org/10.14778/3389133.3389138
  * [19] “TPC benchmark H standard specification revision 3.0.0,” http://tpc.org/tpch/, Transaction Processing Performance Council, 2021.
  * [20] S. K. Shin and G. L. Sanders, “Denormalization Strategies for Data Retrieval from Data Warehouses,” _Decis. Support Syst._ , vol. 42, no. 1, p. 267–282, 2006.
  * [21] R. Chirkova and J. Yang, “Materialized Views,” _Foundations and Trends® in Databases_ , vol. 4, no. 4, pp. 295–405, 2012.
  * [22] X. Yu _et al._ , “PushdownDB: Accelerating a DBMS Using S3 Computation,” in _ICDE-36_ , 2020.
  * [23] S. Idreos _et al._ , “MonetDB: Two Decades of Research in Column-oriented Database Architectures,” _IEEE Data Eng. Bull._ , vol. 35, no. 1, pp. 40–45, 2012.


# UniTRec: A Unified Text-to-Text Transformer and Joint Contrastive Learning
Framework for Text-based Recommendation

Zhiming Mao1,2, Huimin Wang1,3, Yiming Du1,2, Kam-Fai Wong1,2
1The Chinese University of Hong Kong, Hong Kong, China
2MoE Key Laboratory of High Confidence Software Technologies, China
3Jarvis Lab, Tencent, Shenzhen, China
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Prior study has shown that pretrained language models (PLM) can boost the
performance of text-based recommendation. In contrast to previous works that
either use PLM to encode user history as a whole input text, or impose an
additional aggregation network to fuse multi-turn history representations, we
propose a unified local- and global-attention Transformer encoder to better
model two-level contexts of user history. Moreover, conditioned on user
history encoded by Transformer encoders, our framework leverages Transformer
decoders to estimate the language perplexity of candidate text items, which
can serve as a straightforward yet significant contrastive signal for user-
item text matching. Based on this, our framework, UniTRec, unifies the
contrastive objectives of discriminative matching scores and candidate text
perplexity to jointly enhance text-based recommendation. Extensive evaluation
shows that UniTRec delivers SOTA performance on three text-based
recommendation tasks.111Our code is available at https://github.com/Veason-
silverbullet/UniTRec.

##  1 Introduction

Text-based recommendation (Li et al., 2010; Gu et al., 2016; Okura et al.,
2017; Malkiel et al., 2020) aims to recommend relevant textual content (e.g.,
news articles, Twitter posts) to people based on their behaviors as
represented in historical log texts. For instance, engagement recommendation
(Cheng et al., 2022) on social media (e.g., Twitter and Reddit) helps users
discover and engage with interested threads by modeling their browsing
history.

Pretrained language models (Devlin et al., 2019; Brown et al., 2020) have made
waves in recent text-based recommendation research (Zhang et al., 2021; Qi et
al., 2022; Geng et al., 2022). The most common practice is using PLM encoders
(BERT family) to learn representations of user history and candidate item
texts. Recommendation matching scores are computed over the user and item
representations and finally optimized by noise contrastive estimation (NCE)
loss (Gutmann and Hyvärinen, 2010) for ranking multiple candidates.

Unlike encoding single text, using PLM to encode multi-turn texts of user
history is nontrivial. Existing works (Malkiel et al., 2020; Qi et al., 2022;
Geng et al., 2022) concatenate multi-turn history texts as a whole input text,
then use one PLM encoder to learn the holistic user representation. This is a
standard PLM encoding manner but ignores the relation among history turns, as
all word tokens from different history turns are equally attended222There is
no inductive bias of turn-level and history-level relations introduced to
Transformer self-attention computation, where each token plays an equal role..
In contrast, previous studies point out that learning the relation among user
history turns is also beneficial (Zeng et al., 2020; Qi et al., 2021). Another
approach is using PLM encoders to learn representations from multi-turn
history texts, followed by an additional aggregation network to fuse the
multi-turn representations (Wu et al., 2021; Li et al., 2022). However, the
imposed aggregation networks (with newly initialized parameters) weaken the
representation power of PLM encoders which are already pretrained on large-
scale corpora.

This work introduces UniTRec, a Unified text-to-text Transformer framework for
text-based Recom-mendation. In the encoder component of UniTRec, we design
local- and global-attention to learn user history representations through
tailored attention masking, which aims to jointly model word-level and turn-
level relations of user history. UniTRec can utilize the full power of PLM
encoders because it preserves the intact structure of PLM encoders without
newly imposed parameters.

Figure 1: An example of perplexity-based ranking for candidate item texts,
conditioned on user history. The illustrated task is text-based news
recommendation.

Different from most previous works that predict user-candidate matching scores
solely based on the representations learned by Transformer encoders, we argue
that conditioned on user representations learned by Transformer encoders,
candidate text perplexity (PPL) estimated by pretrained Transformer decoders
is also a straightforward yet significant signal for text-based
recommendation. As shown in Figure 1, we hypothesize that the candidate text
perplexity estimated by pretrained LM decoders can directly measure the text
matching degree between user history and candidate texts. It is because the
perplexity estimates the likelihood of candidate texts based on encoder
outputs, which naturally indicates the probabilities of candidate texts given
the user history. Besides, UniTRec can use the last hidden states of
Transformer decoders to directly predict matching scores. Hence, this work
unifies the contrastive objectives of discriminative matching scores and
candidate text perplexity to jointly enhance text-based recommendation.

The contributions of this work are: ($1$) We propose local- and global-
attention to model two-level relation of user history without additional
parameters, which enjoys the full power of PLM encoders. ($2$) We introduce
PLM perplexity to measure user-candidate text matching and unify the
objectives of discriminative matching scores and candidate text perplexity to
enhance text-based recommendation. ($3$) Experiments on three text-based
recommendation datasets validate the effectiveness of UniTRec.

##  2 Approach

###  2.1 Unified User-history Modeling

Formally, multi-turn history of a user is represented as
$H=[t_{1},t_{2},...,t_{N}]$, and each turn text $t_{i}$ contains $|t_{i}|$
words as $t_{i}=[x_{i}^{1},x_{i}^{2},...,x_{i}^{|t_{i}|}]$. UniTRec aims to
unify learning word- and turn-level context representations in one Transformer
encoder.

Local attention on word-level context. We first concatenate the multi-turn
history texts as the input tokens
$X=[x_{1}^{1},x_{1}^{2},...,x_{1}^{|t_{1}|},...,x_{N}^{1},x_{N}^{2},...,x_{N}^{|t_{N}|}]$.
Inspired by Dong et al. (2019), we tailor the attention masking in Transformer
self-attention to learn the word-level context of each turn. Specifically, we
allow word tokens from the same turn to attend to each other, while tokens
from different turns are excluded from self-attention computation:

$\begin{split}\mathbf{M}_{i,j}&=\begin{cases}0,\;\quad\;\mbox{token $x_{i}$
and $x_{j}$ in the same turn}\\\ -\infty,\;\mbox{otherwise}\end{cases}\\\
\mathrm{Atte}&\mathrm{ntion}(Q,K,V)=\mathrm{softmax}(\frac{QK^{T}}{\sqrt{d_{k}}}\mathbf{+M})V\end{split}$
(1)

, where $Q,K,V$ are self-attention query, key, and value in Vaswani et al.
(2017), $\mathbf{M}$ is the mask matrix to achieve local-attention inside each
turn text. The local self-attention blocks consist of $L_{1}$ layers, by which
original PLM encoders can be adapted to learn word-level context
representations of turns.

Global attention on turn-level context. Over the local self-attention layers,
we leverage global self-attention to model the relation among history turns.
Specifically, tokens from all turns attend to each other in self-attention
computation (by setting the mask matrix $\mathbf{M}=\mathbf{0}$). In this way,
Transformer encoders can perform global interaction among each token (and
turn) to learn turn-level context representations of user history. There are
$L_{2}$ layers in the global self-attention blocks, which can also be
inherited from PLM encoders directly.

###  2.2 Joint Contrastive Ranking Objectives

Conditioned on the history representation, we input the candidate text to
Transformer decoders to predict how likely it should be recommended. It is
worth noting that Transformer decoders can naturally perform effective cross-
attention interaction between history and candidate hidden states.

####  2.2.1 Objective on Discriminative Scores

Motivated by Lewis et al. (2020), we feed the last hidden state of decoder
output $h_{T}$ to an MLP score-head which predicts the user-candidate matching
score $S^{d}=\mathrm{ScoreHead}(h_{T})$. The matching score is discriminative,
as higher scores indicate higher user-candidate matching probabilities.

Figure 2: Overview of UniTRec. In training, matching scores $S^{d}$ and
$S^{p}$ are optimized by the NCE loss, respectively. In inference, $S^{d}$ and
$S^{p}$ are normalized and combined to derive the final output ranking.

Following previous works (Li et al., 2022; Qi et al., 2022), we adopt negative
sampling with NCE loss to optimize matching score prediction. Given the user
history and its ground truth matched candidate $C_{i}$, UniTRec predicts the
matching score as $S^{d+}_{i}$. In addition, $K$ unmatched negative candidates
$\\{C_{j}\\}_{j=1}^{K}$ are sampled from the candidate set, and their matching
scores are $\\{S^{d-}_{j}\\}_{j=1}^{K}$. The NCE loss is represented in a
contrastive form:

$\mathcal{L}_{i}^{d}=-\log\frac{\mathrm{exp}(S^{d+}_{i})}{\mathrm{exp}(S^{d+}_{i})+\sum\nolimits_{j=1}^{K}{\mathrm{exp}(S^{d-}_{j})}}$
(2)

####  2.2.2 Objective on Candidate Text Perplexity

As aforementioned, UniTRec leverages perplexity to rank candidate texts. Since
lower perplexity indicates higher user-candidate matching probability,
regarding the candidate text $Y=[y_{1},y_{2},...,y_{T}]$, we define the
perplexity-based matching score $S^{p}$ as its negative perplexity333Note
https://huggingface.co/docs/transformers/perplexity for LM perplexity
calculation. We empirically discard the outer exponential term in the PPL
formula, because it already exists in NCE loss Eq. (4) and does not affect the
final ranking.:

$S^{p}=-\mathrm{PPL}(Y)=\frac{1}{T}\sum\nolimits_{i=1}^{T}\log
p_{\theta}(y_{i}|y_{<i})$ (3)

, where $p_{\theta}(\cdot)$ denotes the target probability output from the
UniTRec Transformer decoder. Similar to Eq. (2), we optimize the perplexity-
based matching score $S^{p}$ in the NCE loss form. As perplexity empirically
varies in a wide range, we introduce a temperature parameter $\tau$ to balance
the joint NCE loss gradients following Radford et al. (2021).

$\mathcal{L}_{i}^{p}=-\log\frac{\mathrm{exp}(\tau\cdot
S^{p+}_{i})}{\mathrm{exp}(\tau\cdot
S^{p+}_{i})+\sum\nolimits_{j=1}^{K}{\mathrm{exp}(\tau\cdot S^{p-}_{j})}}$ (4)

, where $\tau$ is learnable and initialized to $1$. On the training dataset
$\mathcal{D}$, the joint contrastive learning objective is formulated as:

$\mathcal{L}=\sum\nolimits_{i=1}^{|\mathcal{D}|}\big{(}\mathcal{L}_{i}^{d}+\mathcal{L}_{i}^{p}\big{)}$
(5)

###  2.3 Model Initialization and Inference

As UniTRec is a standard text-to-text Transformer, we initialize the
parameters from pretrained BART (Lewis et al., 2020). In inference, UniTRec
predicts the discriminative and perplexity-based scores for each candidate
item, respectively. The two separate scores $S^{d}$ and $S^{p}$ are
normalized, averaged, and finally ranked as the output. Detailed ranking
process is provided in Appendix B.

##  3 Experiments

We evaluate UniTRec on three text-based recommendation tasks: $1$) NewsRec, to
recommend news articles to users based on their browsing history. We use the
MIND-small dataset (Wu et al., 2020) for experiments. 2) QuoteRec, to
recommend quotations to users based on their conversation history. We use the
Reddit-quotation dataset Wang et al. (2021) for experiments. 3) EngageRec, to
recommend social media posts for users to engage with based on their comment
history. We use the dataset released by Zeng et al. (2020) for experiments.
Detailed dataset statistics is provided in Appendix A.

Implementation Details. The UniTRec encoder and decoder both consist of $6$
Transformer layers with $768$-dimensional hidden states and $12$ attention
heads. We set $L_{1}=3$ and $L_{2}=3$. We use AdamW optimizer (Loshchilov and
Hutter, 2019) to train UniTRec with cosine learning rate decay.

| NewsRec | QuoteRec | EngageRec
---|---|---|---
Model | MRR | NDCG@5/10 | HR@5/10 | MRR | NDCG@5/10 | HR@5/10 | MRR | NDCG@5/10 | HR@5/10
GRU4Rec | 32.91 | 36.20/42.53 | 50.33/68.35 | 34.08 | 34.65/37.93 | 44.45/54.63 | 2.12 | 1.04/1.51 | 1.27/2.65
SASRec | 32.60 | 36.03/42.37 | 50.63/68.64 | 33.63 | 34.30/37.49 | 44.32/54.20 | 2.40 | 1.49/1.95 | 2.16/3.47
BERT4Rec | 32.87 | 36.18/42.40 | 50.21/67.97 | 33.59 | 34.26/37.27 | 43.76/53.05 | 3.04 | 1.98/3.23 | 2.81/6.67
RoBERTa-Sim | 32.96 | 36.47/42.81 | 51.06/69.08 | 37.13 | 37.96/41.18 | 48.14/58.06 | 3.74 | 2.66/3.75 | 4.42/7.70
UNBERT | 33.09 | 36.53/42.84 | 50.87/68.82 | 39.75 | 40.74/43.69 | 50.90/60.04 | 2.83 | 1.96/2.67 | 3.11/5.24
UniTRec | 33.76 |  37.63/43.74 |  52.61/69.89 | 41.24 |  42.38/45.31 |  52.87/61.88 | 4.06 |  3.23/4.29 |  4.58/7.68

Table 1: Experiment results on three text-based recommendation tasks. MRR
denotes mean reciprocal rank, NDCG denotes normalized discounted cumulative
gain, and HR denotes hit ratio (presented in percentage). The overall
performance of UniTRec is better than other baseline models with $p$-value
$<0.05$, validated by unpaired t-test.

| NewsRec | QuoteRec | EngageRec
---|---|---|---
Model | MRR | NDCG@5/10 | HR@5/10 | MRR | NDCG@5/10 | HR@5/10 | MRR | NDCG@5/10 | HR@5/10
UniTRec | 33.76 | 37.63/43.74 | 52.61/69.89 | 41.24 | 42.38/45.31 | 52.87/61.88 | 4.06 | 3.23/4.29 | 4.58/7.68
w/o BART Init | 30.31 | 33.32/39.69 | 47.55/65.78 | 19.02 | 17.66/20.80 | 22.45/32.16 | 2.24 | 0.86/1.61 | 1.27/3.62
w/o Local-Att | 33.34 | 37.22/43.32 | 52.28/69.54 | 40.44 | 41.63/44.56 | 52.09/61.15 | 3.92 | 3.19/4.15 | 4.38/7.36
w/o Global-Att | 33.22 | 37.06/43.17 | 52.14/69.47 | 40.25 | 41.47/44.26 | 52.07/60.76 | 3.64 | 2.78/3.59 | 3.89/6.35
Disc-Score only | 33.07 | 36.76/43.03 | 51.68/69.46 | 40.59 | 41.81/44.65 | 52.39/61.14 | 3.82 | 2.99/3.60 | 4.49/6.85
PPL-Score only | 32.83 | 36.39/42.59 | 51.05/68.67 | 40.31 | 41.43/44.47 | 52.13/61.20 | 3.29 | 2.39/3.03 | 3.86/5.66

Table 2: Recommendation performance of ablation model variants.

Baselines. We compare UniTRec with competitive baselines: $1$) GRU4Rec (Balázs
et al., 2016) utilizes a GRU network to learn multi-turn history. $2$) SASRec
(Kang and McAuley, 2018) encodes user history with a self-attention based
sequential model. $3$) BERT4Rec (Sun et al., 2019) employs bidirectional self-
attention to model user history. $4$) RoBERTa-Sim, a simple yet strong
baseline mentioned in Qi et al. (2022), uses the hidden states of [CLS] tokens
to measure user-candidate similarity. $5$) UNBERT, implemented as Zhang et al.
(2021), concatenates history and candidate texts as the input to BERT and
predicts matching scores from the final hidden states of [CLS] tokens.

Note that we do not consider other methods that use non-text inputs (e.g.,
user profile, text topic labels). For fair comparison, all baseline models use
pretrained $12$-layer RoBERTa-base (Liu et al., 2019) as text encoders to
learn embeddings of texts.

###  3.1 Main Results

Table 1 shows the performance of experiment models. From the results of
NewsRec and QuoteRec, we can see that UniTRec outperforms all baseline models
by a clear margin. Also, RoBERTa-Sim and UNBERT that directly use the [CLS]
hidden states to represent user history, surpass other baselines that build
additional aggregation networks upon the whole RoBERTa outputs. As displayed
in the results, EngageRec is the most difficult task. We inspect the dataset
and find that the texts on social media contain too much noise (e.g., URL and
emoji), and the user history contains less number of turns. Nevertheless,
UniTRec achieves better overall performance than other baseline models,
validating its robustness on noisy text inputs and limited user history.

###  3.2 Ablation Studies and Analyses

We further conduct ablation studies on UniTRec. The experiment results are
reported in Table 2.

##### Initialization of UniTRec.

We train UniTRec from scratch without initialization from pretrained BART
(refer to w/o BART Init). The recommendation performance significantly drops
in all three tasks, which indicates that acquiring effective text
understanding ability from PLM is a necessary key to UniTRec performance.

##### Local and global attention.

We investigate the function of two-level attention modules of the UniTRec
history encoder. Concretely, we set $L_{1}=0$ in w/o Local-Att and $L_{2}=0$
in w/o Global-Att, where $L_{1}+L_{2}=6$. We can observe that removing local
and global attention from the original UniTRec history encoder both lead to
suboptimal performance, while the performance drop is more significant in w/o
Global-Att. The results justify the effectiveness of jointly modeling two-
level history contexts through adapted Transformer attention masking without
additional parameters.

##### Discriminative and perplexity-based objectives.

We probe into training UniTRec with standalone discriminative (Disc-Score
only) and perplexity-based (PPL-Score only) contrastive objectives,
respectively. We can see that the discriminative objective yields better
performance than the perplexity-based objective. Besides, the model
performance on both standalone objectives declines compared to the original
joint objective. The results indicate that the discriminative and perplexity-
based matching scores are complementary and can jointly provide more accurate
signals of user history and candidate text matching for text-based
recommendation.

##  4 Conclusion

We present a unified Transformer UniTRec for text-based recommendation.
UniTRec learns two-level contexts of multi-turn user history and jointly
exploits discriminative matching scores and candidate text perplexity as
matching objectives. Empirical experiments on three text-based recommendation
datasets corroborate the effectiveness of UniTRec.

##  5 Limitations

Our model only focuses on utilizing text information for recommendation, which
is a key limitation of this work. In real-world settings, recommender systems
are usually required to handle heterogeneous information inputs. UniTRec is a
pure text-based recommender modeling user history and candidate texts as
inputs. However, incorporating additional side information (e.g., user
profile, text topic labels, and dwell time of user behaviors) could further
improve the recommendation performance and alleviate the cold start problem.
Furthermore, UniTRec only models two-level relations of user behavior history.
Nonetheless, incorporating more user behavior information, such as implicit
and negative feedback, could further enhance the recommendation performance.

## Acknowledgements

We appreciate constructive comments from anonymous reviewers. The research
described in this paper is partially supported by CUHK under Project No.
3230366.

## References

  * Balázs et al. (2016) Hidasi Balázs, Karatzoglou Alexandros, Baltrunas Linas, and Tikk Domonkos. 2016\.  Session-based recommendations with recurrent neural networks.  In _4th International Conference on Learning Representations ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Cheng et al. (2022) Daniel Cheng, Kyle Yan, Phillip Keung, and Noah A. Smith. 2022.  The engage corpus: A social media dataset for text-based recommender systems.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 1885–1889, Marseille, France. European Language Resources Association.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 4171–4186. Association for Computational Linguistics.
  * Dong et al. (2019) Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019.  Unified language model pre-training for natural language understanding and generation.  In _33rd Conference on Neural Information Processing Systems (NeurIPS 2019)_.
  * Geng et al. (2022) Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. 2022.  Recommendation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5).  In _Proceedings of the 16th ACM Conference on Recommender Systems_ , RecSys ’22, page 299–315, New York, NY, USA. Association for Computing Machinery.
  * Gu et al. (2016) Youyang Gu, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2016.  Learning to refine text based recommendations.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2103–2108, Austin, Texas. Association for Computational Linguistics.
  * Gutmann and Hyvärinen (2010) Michael Gutmann and Aapo Hyvärinen. 2010.  Noise-contrastive estimation: A new estimation principle for unnormalized statistical models.  In _Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics_ , volume 9 of _Proceedings of Machine Learning Research_ , pages 297–304, Chia Laguna Resort, Sardinia, Italy. PMLR.
  * Kang and McAuley (2018) Wang-Cheng Kang and Julian McAuley. 2018.  Self-attentive sequential recommendation.  In _2018 IEEE International Conference on Data Mining (ICDM)_ , pages 197–206.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Li et al. (2022) Jian Li, Jieming Zhu, Qiwei Bi, Guohao Cai, Lifeng Shang, Zhenhua Dong, Xin Jiang, and Qun Liu. 2022.  MINER: Multi-interest matching network for news recommendation.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 343–352, Dublin, Ireland. Association for Computational Linguistics.
  * Li et al. (2010) Yize Li, Jiazhong Nie, Yi Zhang, Bingqing Wang, Baoshi Yan, and Fuliang Weng. 2010\.  Contextual recommendation based on text mining.  In _Coling 2010: Posters_ , pages 692–700, Beijing, China. Coling 2010 Organizing Committee.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  In _arXiv preprint arXiv: 1907.11692_. arXiv.
  * Loshchilov and Hutter (2019) Ilya Loshchilov and Frank Hutter. 2019.  Decoupled weight decay regularization.  In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
  * Malkiel et al. (2020) Itzik Malkiel, Oren Barkan, Avi Caciularu, Noam Razin, Ori Katz, and Noam Koenigstein. 2020.  RecoBERT: A catalog language model for text-based recommendations.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , Online. Association for Computational Linguistics.
  * Okura et al. (2017) Shumpei Okura, Yukihiro Tagami, Shingo Ono, and Akira Tajima. 2017.  Embedding-based news recommendation for millions of users.  In _Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , page 1933–1942, New York, NY, USA. Association for Computing Machinery.
  * Qi et al. (2022) Fanchao Qi, Yanhui Yang, Jing Yi, Zhili Cheng, Zhiyuan Liu, and Maosong Sun. 2022\.  QuoteR: A benchmark of quote recommendation for writing.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 336–348, Dublin, Ireland. Association for Computational Linguistics.
  * Qi et al. (2021) Tao Qi, Fangzhao Wu, Chuhan Wu, Peiru Yang, Yang Yu, Xing Xie, and Yongfeng Huang. 2021.  HieRec: Hierarchical user interest modeling for personalized news recommendation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5446–5456, Online. Association for Computational Linguistics.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021.  Learning transferable visual models from natural language supervision.  In _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 8748–8763. PMLR.
  * Sun et al. (2019) Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019\.  Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer.  In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_ , CIKM ’19, page 1441–1450, New York, NY, USA. Association for Computing Machinery.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_ , volume 30, pages 5998–6008. Curran Associates, Inc.
  * Wang et al. (2021) Lingzhi Wang, Xingshan Zeng, and Kam-Fai Wong. 2021.  Quotation recommendation and interpretation based on transformation from queries to quotations.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 754–758, Online. Association for Computational Linguistics.
  * Wu et al. (2021) Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2021.  Empowering news recommendation with pre-trained language models.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’21, page 1652–1656, New York, NY, USA. Association for Computing Machinery.
  * Wu et al. (2020) Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, and Ming Zhou. 2020.  MIND: A large-scale dataset for news recommendation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3597–3606, Online. Association for Computational Linguistics.
  * Zeng et al. (2020) Xingshan Zeng, Jing Li, Lu Wang, Zhiming Mao, and Kam-Fai Wong. 2020.  Dynamic online conversation recommendation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3331–3341, Online. Association for Computational Linguistics.
  * Zhang et al. (2021) Qi Zhang, Jingjie Li, Qinglin Jia, Chuyuan Wang, Jieming Zhu, Zhaowei Wang, and Xiuqiang He. 2021.  Unbert: User-news matching bert for news recommendation.  In _Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, IJCAI-21_ , pages 3356–3362. International Joint Conferences on Artificial Intelligence Organization.  Main Track.

Dataset | NewsRec | QuoteRec | EngageRec
---|---|---|---
Avg. history turns | 26.09 | 4.24 | 3.29
Avg. history tokens | 414.40 | 279.82 | 286.82
Avg. candidates | 37.23 | 1111 | 7163
Avg. candidate tokens | 16.15 | 19.11 | 102.42

Table 3: Statistics of three text-based recommendation training datasets.
History and candidate tokens denote the number of BPE-tokenized tokens. The
test set distribution is closed to the training sets (except candidates of
EngageRec) and hence omitted. Note that the max length of each history log is
truncated to $1024$ tokens.

##  Appendix A Dataset Statistics

The detailed statistics of the three text-based recommendation datasets are
displayed in Table 3. Note that we use news titles as the text inputs for
NewsRec following Qi et al. (2021). NewsRec regards the user clicked and non-
clicked news as candidate texts, while QuoteRec and EngageRec regard all
potential quotation texts and post texts as candidates. Different from Zeng et
al. (2020) that formulates the task as recommending candidate users to given
posts based on post content, we formulate the task as recommending candidate
posts to given users based on user history.

Algorithm 1 Candidate Ranking Processs

0: discriminative scores $S^{d}=\\{S^{d}_{1},S^{d}_{2},...,S^{d}_{M}\\}$,
perplexity-based scores $S^{p}=\\{S^{p}_{1},S^{p}_{2},...,S^{p}_{M}\\}$.

0: final averaged ranking $\bar{R}$.

1: Derive the normalized discriminative scores
$S^{d}_{norm}=\mathrm{softmax}(S^{d})$.

2: Derive the normalized perplexity-based scores
$S^{p}_{norm}=\mathrm{softmax}(S^{p})$.

3: Derive the geometric average scores
$\bar{S}=\log{(S^{d}_{norm})}+\log{(S^{p}_{norm})}$.

4: Sort the averaged scores $\bar{S}$ by descending order to derive the final
ranking: $\bar{R}\leftarrow\mathrm{Rank_{des}}(\bar{S})$.

5: return $\bar{R}$

##  Appendix B Inference Ranking

Given the user history and $M$ candidate texts, UniTRec first predicts the
discriminative ranking scores $S^{d}=\\{S^{d}_{1},S^{d}_{2},...,S^{d}_{M}\\}$
and perplexity-based ranking scores
$S^{p}=\\{S^{p}_{1},S^{p}_{2},...,S^{p}_{M}\\}$ of the candidates. Algorithm 1
outlines an approach to aggregate the final ranking based on $S^{d}$ and
$S^{p}$. Note that the function $\mathrm{Rank}(S)$444$\mathrm{Rank}(S)$ works
similarly to scipy.stats.rankdata(). For example in ascending order,
$\mathrm{Rank_{asc}}(\\{0.2,0.6,0.7,0.4\\})=\mathrm{scipy.stats.rankdata}([0.2,0.6,0.7,0.4])=[1,3,4,2]$
denotes outputting the sorted order of elements in a score list $S$. There
exist other ways to average the ranking of $S^{d}$ and $S^{p}$, which we leave
for future work to explore.

##  Appendix C Qualitative Analysis

We show randomly sampled outputs of UniTRec, for instance, demonstrated on the
news recommendation and quote recommendation tasks. Table 4 and 5 showcase the
qualitative samples.

Turn |  History News Texts
---|---
$\\#1$ |  Mac Engel: As long as these results are acceptable, Dallas Cowboys will continue to be losers
$\\#2$ |  NFL world reacts to officials handing Packers win over Lions
$\\#3$ |  Maryland Congressman Elijah Cummings, a Democrat and Chair of House Oversight and Reform Committee, has died: CNN
$\\#4$ |  Unprecedented movement detected on California earthquake fault capable of 8.0 temblor
$\\#5$ |  Bag Explodes While Being Loaded On Volaris Flight At Midway Airport
$\\#6$ |  Orlando Scandrick rips Eagles: They have "accountability issues"
$\\#7$ |  Meghan King Edmonds, Jim Edmonds’ Nanny Denies Cheating Allegations
$\\#8$ |  Nearly $400M worth of cocaine and marijuana intercepted by US Coast Guard
$\\#9$ |  Former NBA first-round pick arrested in sex sting operation
$\\#10$ |  China’s trade with US shrinks in October despite optimism
|

Candidate News Texts | $S^{d}$ | $S^{p}$ | $\bar{R}$ | Clicked
---|---|---|---|---
Taylor Swift Rep Hits Back at Big Machine, Claims She’s Actually Owed $7.9 Million in Unpaid Royalties | $0.095$ | $0.069$ | $4$ | ✗
Former North Carolina State, NBA player Anthony Grundy dies in stabbing, police say | $0.172$ | $0.155$ | $3$ | ✗
13 Reasons Why’s Christian Navarro Slams Disney for Casting "the White Guy" in The Little Mermaid | $0.048$ | $0.065$ | $7$ | ✗
Opinion: Colin Kaepernick is about to get what he deserves: a chance | $0.303$ | $0.250$ | $1$ | ✓
3 Indiana judges suspended after a night of drinking turned into a White Castle brawl | $0.076$ | $0.059$ | $5$ | ✗
66 Cool Tech Gifts Anyone Would Be Thrilled to Receive | $0.009$ | $0.005$ | $9$ | ✗
Police find 26 children behind false wall at Colorado day care | $0.034$ | $0.116$ | $6$ | ✗
I’ve been writing about tiny homes for a year and spent 2 nights in a 300-foot home to see what it is all about | $0.029$ | $0.019$ | $8$ | ✗
Report: Police investigating woman’s death after Redskins’ player Montae Nicholson took her to hospital | $0.235$ | $0.261$ | $2$ | ✓
|  |  |  |

(i) Qualitative Example-A from news recommendation.

Turn |  History News Texts
---|---
$\\#1$ |  Toddler dancing to celebrate 11 months cancer-free goes viral
$\\#2$ |  NFL Week 8 Power Rankings: Old-school football rules the day
$\\#3$ |  The 25 US cities where it’s easiest to get a mortgage
$\\#4$ |  Burning questions for Cowboys vs Giants on "Monday Night Football"
$\\#5$ |  Who’s the favorite to win 2019 NFL rushing title?
$\\#6$ |  Grading all 32 NFL teams heading into the last eight weeks of the 2019 season
$\\#7$ |  Jennifer Aniston looks amazing in a makeup-free selfie, plus more news
$\\#8$ |  This $12 million "mansion yacht" is made entirely of stainless steel and it’s a first for the industry. Take a peek inside
|

Candidate News Texts | $S^{d}$ | $S^{p}$ | $\bar{R}$ | Clicked
---|---|---|---|---
Opinion: Colin Kaepernick is about to get what he deserves: a chance | $0.330$ | $0.400$ | $1$ | ✓
U.S. Troops Will Die If They Remain in Syria, Bashar Al-Assad Warns | $0.024$ | $0.011$ | $10$ | ✗
Pete Davidson, Kaia Gerber Are Dating, Trying to Stay "Low Profile" | $0.064$ | $0.033$ | $6$ | ✗
The Hottest Tech Gifts This Holiday Season | $0.050$ | $0.027$ | $8$ | ✗
Taylor Swift Rep Hits Back at Big Machine, Claims She’s Actually Owed $7.9 Million in Unpaid Royalties | $0.046$ | $0.038$ | $7$ | ✗
13 Reasons Why’s Christian Navarro Slams Disney for Casting "the White Guy" in The Little Mermaid | $0.060$ | $0.096$ | $4$ | ✓
Some believe Mason Rudolph, hit in head with his own helmet, isn’t getting enough blame | $0.154$ | $0.179$ | $2$ | ✓
South Carolina teen gets life in prison for deadly elementary school shooting | $0.066$ | $0.046$ | $5$ | ✗
The Unlikely Star of My Family’s Thanksgiving Table | $0.047$ | $0.021$ | $9$ | ✗
Police investigating woman’s death after Redskins’ player Montae Nicholson took her to hospital | $0.158$ | $0.149$ | $3$ | ✗
|  |  |  |

(ii) Qualitative Example-B from news recommendation. Table 4: Case analyses of
news recommendation. History News Texts are sorted by user-clicked timestamps.
$S^{d}$, $S^{p}$, and $\bar{R}$ are normalized discriminative, perplexity-
based scores, and average ranking as described in Appendix B. Clicked denotes
the ground truth user-click labels. Note that the experiment history logs are
anonymized and delinked, which is always the first priority of the
recommendation study.

Turn |  Conversation Threading History
---|---
$\\#1$ |  I own an FJ. It’s a great car and even on stockies. It s great offroad.
$\\#2$ |  I feel bad for you that you run the risk of being associated with the typical FJ owner.
$\\#3$ |  What is a typical FJ owner? I’ve not heard anything bad about FJ owners.
$\\#4$ |  It’s like someone who drives a jeep wrangler in NYC. There’s no need. Tons of FJ owners do that have it and not use it for what it’s made for.
$\\#5$ |  God forbid someone likes the design of a car and doesn’t use it offroad.
$\\#6$ |  Then buy a much more economic environmentalist friendly version. If you buy something and always use it for much less than it’s purpose, why buy it?
$\\#7$ |  Or people can buy whatever the hell they want because it’s their money and not yours.
$\\#8$ |  You’re entirely right. Just like people can be rude just because you can do it, because you have the ability but why should you ass.
$\\#9$ |  I wasn’t aware that somebody buying a vehicle that they like and you don’t was morally wrong.
$\\#10$ |  I love FJs. It’s perfectly fine to buy whatever you think looks nice.
|

Candidate Quote Texts | $S^{d}$ | $S^{p}$ | $\bar{R}$ | Ground truth
---|---|---|---|---
Beauty is in the eye of the beholder. | $0.480$ | $0.471$ | $1$ | ✓
A fool and his money are soon parted. | $0.176$ | $0.140$ | $2$ |
Form follows function. | $0.051$ | $0.046$ | $3$ |
Everything is worth what its purchaser will pay for it. | $0.040$ | $0.058$ | $4$ |
Because it’s there. | $0.038$ | $0.029$ | $5$ |
You can’t fix stupid. | $0.021$ | $0.034$ | $6$ |
The lady doth protest too much, methinks. | $0.022$ | $0.013$ | $7$ |
It’s all about the money. | $0.020$ | $0.013$ | $8$ |
Anybody driving slower than you is an idiot, and anyone going faster than you is a maniac? | $0.012$ | $0.018$ | $9$ |
Opportunity is missed by most people. | $0.018$ | $0.008$ | $10$ |
|  |  |  |

(iii) Qualitative Example-C from quote recommendation.

Turn |  Conversation Threading History
---|---
$\\#1$ |  Society is becoming more efficient, which is a good thing. People should realize there’s no point in holding back this technology just for the sake of keeping people employed. If this were beneficial, then calculators and computers shouldn’t exist either.
$\\#2$ |  One small problem is that people need to pay rent and eat.
$\\#3$ |  So we should ditch computers and go back to the typing pool? Should we get rid of heavy earth moving equipment and just use hundreds of guys with hand tools to build everything? It would employ a hell of a lot more people.
$\\#4$ |  No one’s saying that. I don’t think anyone is really against automation, but as it increases, there are soon going to be more people that there are jobs that actually need doing. I actually believe we’ve already passed this point. So what do we do with the people, who can’t get jobs simply because there are none? It’s an issue that need assessed immediately.
$\\#5$ |  Tons and tons and tons of American jobs have been replaced by new jobs created by technology or in support of technology years ago. An office might have needed people to handle filing paperwork, keeping it in order, and retrieving, where now a document management system has made them completely redundant. The upshot is that to access that DMS, people are out there selling computers, installing computers, servicing computers, and supporting end users building the servers installing, supporting monitoring backing them up, and all that jobs that come in support of those progress is progress. And it advances human efficiency and knowledge. These are just one or two examples, but the answer is not to kill progress. Other countries simply won’t. The answer is to push education to the forefront, so people are prepared for these jobs and whatever other challenges the future may bring.
$\\#6$ |  This is true. But it s unfortunate technological advances tend to reduce low skill jobs and replace them with high skill jobs. It would feel more fair if the low skilled workers could all do training programs and become high skilled workers. But this isn’t really the case. Those jobs end up being taken by someone who had better educational opportunities or someone younger who still has time to take advantage of education.
$\\#7$ |  The reality is the reality. Unfortunate or not educating people will create more educated people to handle high skill jobs, and I’ll tell you being a desktop support technician isn’t high skill. As that’s where we push in the future, any amount of hand wringing won’t change the facts. We must educate our people if we want to be a global leader in more than homelessness poverty.
$\\#8$ |  Education won’t matter. We are at the end of the job age at some point in the near future. We are going to have to deal with the fact that getting a job isn’t a reality for a significant percentage of the population. Society will have to radically change as it did during the industrial revolution.
$\\#9$ |  Much cheaper to heavily discourage having more children free abortions. Then in years there won’t be so many useless people who can apparently be replaced by a simple robot.
$\\#10$ |  Virtually every job will be replaced by automation name skilled trades that can’t be automated. I imagine you’d be surprised at how hard this is. Are pharmacists useless, surgeons, accountants? I’d bet that your job is just as replaceable as these.
|

Candidate Quote Texts | $S^{d}$ | $S^{p}$ | $\bar{R}$ | Ground truth
---|---|---|---|---
There’s no such thing as a free lunch. | $0.365$ | $0.417$ | $1$ |
I can’t predict the future. | $0.185$ | $0.210$ | $2$ | ✓
I have never let my schooling interfere with my education. | $0.104$ | $0.059$ | $3$ |
Prevention is better than cure. | $0.044$ | $0.083$ | $4$ |
Knowledge is power. | $0.059$ | $0.052$ | $5$ |
Don’t let schooling interfere with your education. | $0.044$ | $0.043$ | $6$ |
Nature abhors a vacuum. | $0.036$ | $0.024$ | $7$ |
There is no substitute for hard work. | $0.024$ | $0.017$ | $8$ |
There are three kinds of lies: lies, damned lies, and statistics. | $0.022$ | $0.013$ | $9$ |
You can’t fix stupid. | $0.019$ | $0.010$ | $10$ |
|  |  |  |

(iv) Qualitative Example-D from quote recommendation. Table 5: Case analyses
of quote recommendation. We demonstrate the candidate quotes of the top $10$
rankings out of all candidates. Note that there is only one ground truth quote
for each conversation history.