# Denoising Time Cycle Modeling for Recommendation

Sicong Xie  Ant GroupHangzhouZhejiangChina<EMAIL_ADDRESS>,  Qunwei
Li  Ant GroupHangzhouZhejiangChina<EMAIL_ADDRESS>,  Weidi Xu  Ant
GroupShanghaiChina<EMAIL_ADDRESS>,  Kaiming Shen  Ant
GroupBeijingChina<EMAIL_ADDRESS>,  Shaohu Chen  Ant
GroupBeijingChina<EMAIL_ADDRESS>and  Wenliang Zhong  Ant
GroupHangzhouZhejiangChina<EMAIL_ADDRESS>
(2022)

###### Abstract.

Recently, modeling temporal patterns of user-item interactions have attracted
much attention in recommender systems. We argue that existing methods ignore
the variety of temporal patterns of user behaviors. We define the subset of
user behaviors that are irrelevant to the target item as noises, which limits
the performance of target-related time cycle modeling and affect the
recommendation performance. In this paper, we propose Denoising Time Cycle
Modeling (DiCycle), a novel approach to denoise user behaviors and select the
subset of user behaviors that are highly related to the target item. DiCycle
is able to explicitly model diverse time cycle patterns for recommendation.
Extensive experiments are conducted on both public benchmarks and a real-world
dataset, demonstrating the superior performance of DiCycle over the state-of-
the-art recommendation methods.

Time cycle, denoise, user behaviors, recommendation

††journalyear: 2022††copyright: acmcopyright††conference: Proceedings of the
45th International ACM SIGIR Conference on Research and Development in
Information Retrieval; July 11–15, 2022; Madrid, Spain††booktitle: Proceedings
of the 45th International ACM SIGIR Conference on Research and Development in
Information Retrieval (SIGIR ’22), July 11–15, 2022, Madrid, Spain††price:
15.00††doi: 10.1145/3477495.3531785††isbn: 978-1-4503-8732-3/22/07††ccs:
Information systems Recommender systems††ccs: Information systems
Personalization††ccs: Information systems Retrieval models and ranking

##  1\. Introduction

Personalized recommender systems can capture dynamic user interests and
potential demands through analyzing a large amount of features, such as user
profile, user behaviors, item category, etc (Guo et al., 2017; Lian et al.,
2018; Xiao et al., 2017). Among them, temporal information in user behaviors
plays a vital role, which is extracted from user-item interaction records and
can present different time-related patterns of user behaviors.

We analyze user’s click actions on two items in our real-world dataset to
illustrate two typical temporal patterns. Figure 1(a) shows how the number of
clicks on an item of Food-Delivery service changes every hour in a day. The
click intensity reaches its peaks at 11, 18, and 21 o’clock, corresponding to
probable lunch, dinner, and midnight meal time. We call such a time pattern as
Absolute Time Cycle Pattern (ATC), which characterizes the impact of semantic
time (e.g., hour, weekday, day of month, etc) on user-item interactions and
reveals the specific time points that a user may be interested in an item.
Besides, the time span $|t_{2}-t_{1}|$ of timestamp $t_{1}$ and $t_{2}$ of
user behaviors reveals critical temporal information as well. Figure 1(b)
shows that user’s click intensity on an item of Taxi-Hailing service varies
periodically after user’s last click, which we refer to as the Relative Time
Cycle Pattern (RTC). Such an observation hints that people tend to hail a taxi
again 24 hours later after the last click and the intensity declines as time
goes on, but still reaches local peaks every 24 hours. RTC directly
illustrates how user’s past behavior influences their current intentions after
a certain time interval.

Figure 1.  Analysis of two typical temporal patterns of user behaviors.

Existing works have studied the dynamic temporal patterns of user behaviors
and proposed elaborate model structures (Li et al., 2020; Cho et al., 2021; Ye
et al., 2020). However, these studies model time information of user behaviors
of all the items without carefully differentiation and some items might be
irrelevant to the target item. For example, when the target is an item of
Taxi-Hailing service, the user’s past behaviors on taxi-related items would
benefit the modeling of time cycle patterns and improve recommendation
performance. On the contrary, past behaviors on completely irrelevant items
(e.g., items of E-Commerce service) may bring adverse impact on time cycle
modeling, which are defined as the noises of user behaviors given a specific
target item in our paper. Therefore, denoising user behaviors is important for
capturing target-related time cycle patterns and improving the recommendation
performance.

In this paper, we propose Denoising Time Cycle Recommendation (DiCycle), a
framework that disentangles noises in user behaviors and jointly learns the
two time cycle patterns, i.e., ATC and RTC. Representing the time has always
been a challenge in machine learning. As for ATC, the semantic time such as
hour, day, and week, is a discrete variable, and representation of discretized
time values can lose some information. In DiCycle, a convolutional module is
applied to aggregate the adjacent time representation to mitigate the
information loss caused by time discreteness. As for RTC, we utilize a
continuous translation-invariant kernel to convert timestamp into embedding
representations motivated by Bochner’s Theorem (Xu et al., 2019a, 2020). Based
on this kernel, the inner product of the embeddings of timestamp $t_{1}$ and
$t_{2}$ can characterize the relevance of user’s behaviors on $t_{1}$ and
$t_{2}$, depending on the time interval $|t_{2}-t_{1}|$. Moreover, a gated
filter unit softly selects a subset from user behaviors based on its relevance
to the target item, filtering out noises for time cycle modeling. Combined
with two temporal representations of ATC and RTC, a time cycle attention is
applied to generate the final representation, indicating whether the user will
click the target item from the perspective of time cycle pattern.

We verify DiCycle on three public benchmarks and a real-world dataset from our
company. The results show the proposed method can effectively capture diverse
time cycle patterns of user behaviors and prominently improve the
recommendation performance.

##  2\. Related Work

Recently, there have been great advances in modeling dynamic patterns of user
behaviors in deep learning, including recursive neural network (RNN) (Tan et
al., 2016), convolutional neural network (Tang and Wang, 2018; Xu et al.,
2019b; Yan et al., 2019) and attention mechanism based structures (Kang and
McAuley, 2018; Zhou et al., 2018a, b, 2019). These methods consider the
sequential nature of user behaviors, processing user behavior sequences with
natural language processing techniques. Session-based approaches separate user
behaviors according to their occurring time and model user short-term
preferences, indirectly incorporating temporal information (Chen et al., 2018;
Feng et al., 2019). More recent researches directly focus on modeling temporal
information, demonstrating its importance on many tasks, such as sequence
prediction, and click-through rate (CTR) prediction (Zhu et al., 2017; Xu et
al., 2019a, 2020; Ye et al., 2020; Cho et al., 2021). TimeLSTM equips LSTM
with time gates to model time intervals, which can better capture both user’s
short-term and long-term interests (Zhu et al., 2017). Several time-aware
frameworks are specially designed for recommender systems, which learns
heterogeneous temporal patterns of user preference (Ye et al., 2020; Cho et
al., 2021; Costa and Dolog, 2019; Li et al., 2020). However, how to extract
denoised temporal information from user behavior for end-to-end applications
as described above remains a challenge. The proposed DiCycle attempts to solve
this problem, which is introduced in detail in the next section.

##  3\. Denoising Time Cycle Modeling

In this section, we first define the problem formally, and then introduce the
approach of time encoding for both ATC and RTC. Together with a gated filter
unit to denoise user behaviors, we finally propose DiCycle for both ATC and
RTC, the overall structure of which is shown in Figure 2.

Figure 2. The overall architecture of the proposed model.

###  3.1. Problem Formulation

Given a user $u\in\mathcal{U}$, a target item $i$ $\in\mathcal{I}$ and target
timestamp $t$, our CTR task is to predict the probability that user $u$ will
click item $i$ at timestamp $t$ as $P(y=1|u,i,t)$, where $y=1$ refers to click
and $y=0$ refers to non-click. The target behavior to be predicted is defined
as $s_{a}^{u}=(i,t)$. Behaviors of user $u$ before time $t$ are organized as
$S^{u}=\\{s^{u}_{k}|1\leq{k}\leq{l^{u}}\\}=\\{(i^{u}_{k},t^{u}_{k})|1\leq{k}\leq{l^{u}}\\}$,
where $(i_{k}^{u},t_{k}^{u})$ means that user $u$ clicked on item $i_{k}^{u}$
at time $t_{k}^{u}$, and $t_{1}^{u}\leq t_{2}^{u}\leq\cdots\leq
t_{l_{u}}^{u}\leq t$ holds. Thus the CTR probability in our task turns to
$P(y=1|S^{u},i,t)$. In the following, we omit $u$ in the notation for
convenience.

###  3.2. Time Encoding Unit

In this section, we introduce the time encoding unit, including the
representation learning of absolute time and of relative time

####  3.2.1. Learning Absolute Time Representation for ATC

Normally, for timestamp $t$, an embedding layer is applied to encode it into
semantic time embeddings with different granularities, containing hour $h$,
weekday $w$, month $m$ and etc, which can be formulated as
$\Lambda_{g}(t)\in{\mathbb{R}^{d}}$ for granularity $(g\in
G={\\{h,w,m,\cdots\\}})$. However, time continuity is broken in such a
modeling. For example, the time point of 17:50 and 18:10 is quite close, but
the hour time is 17 and 18, respectively, whose semantic embedding could be
entirely different. To combat such a loss of continuity of time, we propose an
absolute time convolution module, which symmetrically incorporates surrounding
time slots. Take time 17:50 and the granularity of hour for an example.
Instead of just using the information of hour 17, its embedding by the
granularity of hour is combined with the embedding of its surrounding hours,
i.e., hour 15, 16, 17, 18, 19 and etc. Formally, the surrounding time
embeddings of time $t$ by granularity $g$ is denoted as
$\Lambda_{g,j}^{S}(t)=[\Lambda_{g}(t^{-j}),\cdots,\Lambda_{g}t^{0}),\cdots,\Lambda_{g}(t^{+j})]^{\mathrm{T}}\in{\mathbb{R}^{(2j+1)\times
d}}$, where $j$ denotes the surrounding range of the target time and
$\Lambda_{g}(t^{0})=\Lambda_{g}(t)$ holds.

With the convolution kernel $K\in{\mathbb{R}^{n\times{d}}}$ and the activation
function ReLU, where $n$ is the kernel size, we conduct one-dimensional
convolution on $\Lambda_{g,j}^{S}(t)$ to get
$\Lambda_{g,j}^{C}(t)\in{\mathbb{R}^{(2j+1)\times d}}$. Subsequently, max-
pooling is conducted on the first dimension of $\Lambda_{g,j}^{C}(t)$ to get
$\Lambda_{g,j}^{P}(t)\in{\mathbb{R}^{d}}$. CNN has the capability of capturing
local features, which enables us to capture the similarities in surrounding
time and maintain some time continuity.

As is shown in Figure 2, we set the surrounding range $j$ from $0$ to $J$
($J=2$ in default), based on which the impact of the surrounding time
gradually diminishes as it goes larger. The final representation of absolute
time $\Lambda(t)$ is composed of all the granularities as

(1) $\displaystyle\Lambda(t)=\sum_{g\in{G}}\sum_{j=1}^{J}{\rm
Pooling}\left({\rm ReLU}\left({\rm
Conv}\left(\Lambda_{g,j}^{S}(t),K\right)\right)\right)\,.$

####  3.2.2. Learning Relative Time Representation for RTC

Relative time is defined as the time interval between the target timestamp $t$
and the timestamp $t_{k}$ in user’s past behaviors, which reflects how much
influence her past behaviors are on current intention. We aim to find a
mapping function $\Phi$ that transforms time interval from time domain
$\mathcal{T}$ to $d-$dimensional vector space, preserving the evolving nature
of time information. In this way, the information of time interval
$|t_{2}-t_{1}|$ of timestamps $t_{1}$ and $t_{2}$ can be extracted by the
inner product $<\Phi(t_{1}),\Phi(t_{2})>$. Therefore, the objective turns to
learning a translation-invariant temporal kernel
$\mathcal{K}(t_{1},t_{2})=\psi(t_{2}-t_{1})=<{\Phi}(t_{1}),{\Phi}(t_{2})>$,
where $\psi:[-t_{max},t_{max}]\rightarrow\mathbb{R}$. Inspired by Bochner’s
theorem (Xu et al., 2019a, 2020), the mapping function is defined as

(2)
$\displaystyle{\Phi}(t)=\sqrt{\frac{1}{d/2}}[cos(\omega_{1}t),sin(\omega_{1}t),\cdots,cos(\omega_{d/2}t),sin(\omega_{d/2}t)]\,,$

where $\omega_{1},\omega_{2},\cdots,\omega_{d/2}$ are trainable parameters.

###  3.3. Gated Filter Unit

As mentioned in the previous section, various temporal patterns are supposed
to be extracted from user behaviors that are highly related to the target
item. For such a purpose, we propose a gated filter unit that weighs each
element of user behavior $S$ dynamically.

Suppose that the target item embedding is ${e_{a}}$ and the item embedding in
the user behavior ${S}$ is $\\{e_{k}|1\leq{k}\leq{l}\\}$. Note that item
embedding dose not contain time information. The gated weight between the
target item and $k$-th behavior $s_{k}$ is expressed as

(3) ${\rm f}_{k}=\left\\{\begin{array}[]{ccl}{\rm sim}({e_{a}},{e_{k}})&&{\rm
sim}({e_{a}},{e_{k}})\geq{\delta_{thred}}\\\ 0&&{\rm
sim}({e_{a}},{e_{k}})<\delta_{thred}\,,\end{array}\right.$

where

(4) ${\rm sim}({e_{a}},{e_{k}})=\frac{1+\cos({e_{a}},{e_{k}})}{2},\
\cos({e_{a}},{e_{k}})=\frac{{e_{a}}{e_{k}}^{\mathrm{T}}}{|{e_{a}}|{|{e_{k}}|}}\,.$

where $\delta_{thred}$ is a hyper-parameter and can be cross validated using
the test dataset for best performance. Here we use $\cos(\cdot)$ as the
activation, which performs well in practice. Additionally, ${\rm f}_{k}$ is
generated by a piece-wise function with threshold $\delta_{thred}$. Only when
the gated weight reaches this threshold, its value remains; otherwise, it is
set to $0$. Our purpose is to completely filter out the most irrelevant
behaviors, which are defined as noises in this paper, and reserve the relevant
behaviors with the weights ${\rm f}_{k}$.

###  3.4. Time Cycle Modeling

Together with time encoding unit and gated filter unit, we finally propose the
time cycle modeling. Suppose the time embedding of $k$-th behavior $s_{k}$ is
${q_{k}}$ and the target time embedding of the target behavior $s_{a}$ is
${q_{a}}$. We aggregate the absolute time representation and relative time
representation to obtain the time embeddings as

(5) $\displaystyle{q_{k}}={\Phi}(t-t_{k})+\Lambda(t_{k}),\
{q_{a}}={\Phi}(0)+\Lambda(t_{a})\,,$

where $t$ is the target timestamp and $\Phi(0)$ for $q(a)$ denotes that time
interval is $0$. As is shown in Figure 2, the model is divided into two
modules: user interest module and time cycle module. We introduce these two
modules in detail in the next subsections.

####  3.4.1. User Interest Module

The general structure of user interest module is quite similar to existing
models like DIN (Zhou et al., 2018b). The only difference is that we add time
embedding to the item of user behaviors and to the target item as side
information. Following the notation above, the embedding of user behavior $S$
is denoted as $\\{{r_{k}}|1\leq{k}\leq{l}\\}$, where
${r_{k}}={e_{k}}+{q_{k}}$, and the target embedding of the target behavior
$s_{a}$ is computed by ${r_{a}}={e_{a}}+{q_{a}}$. Subsequently, the target
attention is conducted between $r_{a}$ and $\\{r_{k}|1\leq{k}\leq{l}\\}$,
which is then in the weighted sum over $r_{k}$ to yield the output embedding
${r}$ of user interest module.

Table 1. Statistics of the datasets, including the numbers of the users, items, interactions, average interactions per user (Inter. Avg. for short in the table) and train samples.  Dataset | Users | Items | Inter. | Avg. | Samples
---|---|---|---|---|---
LastFM | 627 | 341,169 | 14,612,388 | 23,305 | 1,461,464
ML-1M | 6,040 | 3,416 | 999,611 | 165 | 390,164
Books | 264,522 | 954,865 | 13,616,232 | 51 | 2,577,098
IndRec | 0.1 billion | 1 million | 20 billion | 200 | 0.5 billion
Table 2.  Performance comparison on four datasets. The methods of the best and the second best performance are in bold and underlined, respectively.  Dataset |  Metric |  LR |  DNN |  DIN |  SASRec |  GRU4Rec |  TimeLSTM |  TiSASRec |  TimelyRec |  DiCycle |  RelaImpr
---|---|---|---|---|---|---|---|---|---|---|---
LastFM |  AUC |  0.7353 |  0.7376 |  0.7368 |  0.7475 |  0.7389 |  0.7479 |  0.7480 |  0.7460 |  0.7801 |  12.94%
GAUC |  0.7418 |  0.7416 |  0.7432 |  0.7480 |  0.7423 |  0.7524 |  0.7464 |  0.7416 |  0.7961 |  17.31%
ML-1M |  AUC |  0.7883 |  0.8532 |  0.8561 |  0.8687 |  0.8598 |  0.8622 |  0.8648 |  0.8754 |  0.8955 |  5.35%
GAUC |  0.7962 |  0.8535 |  0.8498 |  0.8651 |  0.8601 |  0.8619 |  0.8548 |  0.8721 |  0.8912 |  5.13%
Books |  AUC |  0.7647 |  0.7802 |  0.7804 |  0.7820 |  0.7789 |  0.7833 |  0.7912 |  0.7875 |  0.7988 |  2.61%
GAUC |  0.7594 |  0.7597 |  0.7609 |  0.7546 |  0.7521 |  0.7611 |  0.7673 |  0.7691 |  0.7767 |  2.82%
IndRec |  AUC |  0.7736 |  0.7869 |  0.7930 |  0.7944 |  0.7943 |  0.7968 |  0.7979 |  0.7984 |  0.8046 |  2.08%
GAUC |  0.7198 |  0.7283 |  0.7331 |  0.7369 |  0.7352 |  0.7375 |  0.7379 |  0.7386 |  0.7461 |  3.14%
Table 3. Ablation study. Metric AUC is adopted for evaluation.  Dataset | LastFM | ML-1M | Books | IndRec
---|---|---|---|---
DiCycle | 0.7801 | 0.8955 | 0.7988 | 0.8046
Remove Absolute Time | 0.7761 | 0.8906 | 0.7941 | 0.8036
Remove Relative Time | 0.7764 | 0.8928 | 0.7932 | 0.8016
Remove Time Cycle Module | 0.7539 | 0.8797 | 0.7862 | 0.7974

####  3.4.2. Time Cycle Module

This module aims to learn the diverse time cycle patterns of user behaviors,
which are denoised by the gated weight ${\rm f}_{k}$. Time embedding $q_{k}$
of k-th behavior $s_{k}$ turns to ${\tilde{q_{k}}}={\rm f}_{k}\cdot{{q_{k}}}$.
Some elements of $\tilde{q}=\\{{\tilde{q}_{k}}|1\leq{k}\leq{l}\\}$ are
directly set to zero vectors by Equation 3, which are regarded as noises and
can be harmful for time cycle modeling.

Next, the time cycle attention is applied to capture time cycle patterns based
on denoised time embedding sequence ${\tilde{q}}$, where the time cycle
attention is formulated as:

(6) $\displaystyle
c_{aj}=\frac{{q_{a}}{W_{Q}}({\tilde{q_{j}}}{W_{K}})^{\mathrm{T}}}{\sqrt{d}},\
\alpha_{aj}=\frac{\exp c_{aj}}{\sum_{k=1}^{l}\exp c_{ak}},\
{h}=\sum_{k=1}^{l}\alpha_{aj}({\tilde{q_{k}}}{W_{V}})\,.$

where $h$ is the output embedding of time cycle module and
${W_{Q}}\in\mathbb{R}^{d\times d}$, ${W_{K}}\in\mathbb{R}^{d\times d}$,
${W_{V}}\in\mathbb{R}^{d\times d}$ are parameter matrices for linear
projections. Equipped with absolute time representation $\Lambda(\cdot)$ and
relative time representation $\Phi(\cdot)$, the time cycle attention can find
time cycle patterns that are relevant to the target item in the user
behaviors. We concatenate the output embedding of user interest module ${r}$
and the output embedding of time cycle module ${h}$, then pass it into a
3-layer fully connected neural network. With activation function ${\rm
Sigmoid}(\cdot)$, we obtain the probability $P(y_{j})$ that user $u$ will
click on item $i$. Finally, the commonly used cross-entropy loss is chosen to
train the model, which is formulated as

(7) $\displaystyle\mathcal{L}=-\sum_{j=1}^{N}y_{j}{\rm
log}(P(y_{j}))+(1-y_{j}){\rm log}(1-P(y_{j}))\,.$

##  4\. EXPERIMENTS

###  4.1. Experimental Settings

Datasets. We conduct experiments on three public benchmarks, including LastFM
111https://www.last.fm/ (Celma Herrada et al., 2009), ML-1M
222https://grouplens.org/datasets/movielens/1m/ (Harper and Konstan, 2015),
Amazon Books 333http://deepyeti.ucsd.edu/jianmo/amazon/index.html (Ni et al.,
2019), and a real-world dataset extracted from a recommendation scenario of
our company, which is named IndRec. The interactions of user and item in these
datasets include click, review, and listening to music, which are uniformly
termed as interaction in the following. Statistical details of these datasets
are presented in Table 1. For all the public datasets, the user-item
interactions are regarded as positive samples, and negative samples are
randomly sampled from the items that users have not interacted with, where the
sampling rate of positive and negative samples is 1:1. We treat the last item
of the behavior as test data for each user, and use the remaining for
training, following (Kang and McAuley, 2018; Sun et al., 2019). IndRec is
extracted from two-week online logs, where samples clicked by the users are
labeled as positive and samples exposed but not clicked are labeled as
negative. For each sample of these datasets, the length of user behaviors is
set to 200, 200, 100, 100 for LastFM, ML-1M, Books, and IndRec, respectively.

Baselines. Several the state-of-the-art methods serve as baselines in the
experiments. We use simple LR (Logistic Regression) and fully connected DNN
(Deep Neural Network) for comparison. As for other baselines, DIN (Zhou et
al., 2018b) uses target attention to model user behaviors. SASRec (Kang and
McAuley, 2018) is a self-attention recommendation framework to model user
behaviors. GRU4Rec (Hidasi et al., 2016) is a session-based method with
structured GRU. Time-LSTM (Zhu et al., 2017) is a variant of LSTM equipped
with time gates to model time intervals. TiSASRec (Li et al., 2020) explicitly
models the timestamps of interactions with self attention to explore the
influence of different time intervals. TimelyRec (Cho et al., 2021) jointly
learns heterogeneous temporal patterns of user preference.

Parameters and Evaluations. The dimension $d$ is set to 64, the batch size is
128 and the learning rate is $1e^{-4}$ for three public datasets, and we set
such numbers to 16, 1024, and $1e^{-4}$ for IndRec, respectively. We adopt two
commonly used metrics to evaluate the performance, including the area under
the ROC curve (AUC) and the group weighted area under the ROC curve (GAUC).
Moreover, we also report the relative improvement of DiCycle over the best
baselines to show its superiority, which is measured following (Shen et al.,
2021; Yan et al., 2014) by

(8) $RelaImpr=\left(\frac{metric(target\ model)-0.5}{metric(base\
model)-0.5}-1\right)\times{100\%},$

where the $metric$ is AUC or GAUC.

###  4.2. Results and Discussions

Performance. The overall comparison results are listed in Table 2. The
RelaImpr in AUC of DiCycle over the best baseline is 12.94%, 5.35%, 2.61%,
2.08% for LastFM, ML-1M, Books, and IndRec, respectively; the RelaImpr in GAUC
of DiCycle over the best baseline is 17.31%, 5.13%, 2.82%, 3.14% for LastFM,
ML-1M, Books, and IndRec, respectively. The results show that DiCycle performs
consistently better than all of the state-of-the-art baselines in four
datasets, demonstrating its effectiveness in learning denoised temporal
patterns in user behaviors. Moreover, an extensive ablation study is conducted
with DiCycle to analyze the effect of each module, where the absolute time
encoding, the relative time encoding, and the time cycle module are removed,
respectively. The results in Table 3 show that all of these modules are
critical to achieving final performance in the four datasets. The relative
encoding has a higher effect than the absolute time encoding in Books and
IndRec, whereas the opposite holds in LastFM and ML-1M. It is illustrated that
the time cycle module plays a decisive role in DiCycle, without which there is
a performance degradation of -9.35%, -3.99%, -4.22%, and -2.36% for LastFM,
ML-1M, Books, and IndRec, respectively.

Figure 3. (a), (b): Changing next interaction time for users who had many
interactions and a few interactions with target item, respectively.

To provide intuitive evidence for recommendation by DiCycle, we gradually
change the target timestamp given a certain user and item pair to see the
variance of the prediction. As is shown in Figure 3, we select two interaction
records from IndRec. For the first record, the user had many interactions with
the target item in her past behaviors whereas very few is observed for the
second one. We gradually add additional time to the target timestamp, 3600
seconds for each time. Corresponding to the first record, predicted score in
terms of the CTR probability in Figure 3(a) show strong temporal patterns. It
has a local peak every 24 hours, and the overall score range declines at first
and then gets larger with the change of day. The score in Figure 3(b)
gradually declines over time, where time cycle patterns are comparatively
weaker. DiCycle filters out noises in user behaviors and can select behaviors
highly related to the target time, which explains the difference between
Figure 3(a) and Figure 3(b). We conclude from this case study that DiCycle has
a strong capability to capture time cycle patterns for users with many
interactions on items that are highly relevant to the target item.

##  5\. CONCLUSION

In this paper, we proposed DiCycle to capture dynamic time cycle patterns.
Combined with the gated filter unit, DiCycle can filter out the noises in user
behaviors, which is defined as the subset of user behaviors that are
irrelevant to the target item. DiCycle is able to model both absolute time
pattern and relative time pattern in user behaviors which have high relevance
to the target item. Extensive experiments on public benchmarks and a real-
world dataset verified the effectiveness of DiCycle.

## References

  * (1)
  * Celma Herrada et al. (2009) Òscar Celma Herrada et al. 2009\.  _Music recommendation and discovery in the long tail_.  Universitat Pompeu Fabra.
  * Chen et al. (2018) Xu Chen, Hongteng Xu, Yongfeng Zhang, Jiaxi Tang, Yixin Cao, Zheng Qin, and Hongyuan Zha. 2018.  Sequential recommendation with user memory networks. In _WSDM_. 108–116.
  * Cho et al. (2021) Junsu Cho, Dongmin Hyun, SeongKu Kang, and Hwanjo Yu. 2021\.  Learning Heterogeneous Temporal Patterns of User Preference for Timely Recommendation. In _Proceedings of the Web Conference 2021_. 1274–1283.
  * Costa and Dolog (2019) Felipe Soares da Costa and Peter Dolog. 2019.  Collective embedding for neural context-aware recommender systems. In _Proceedings of the 13th ACM Conference on Recommender Systems_. 201–209.
  * Feng et al. (2019) Yufei Feng, Fuyu Lv, Weichen Shen, Menghan Wang, Fei Sun, Yu Zhu, and Keping Yang. 2019.  Deep session interest network for click-through rate prediction.  _arXiv preprint arXiv:1905.06482_ (2019).
  * Guo et al. (2017) Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017.  DeepFM: a factorization-machine based neural network for CTR prediction. In _IJCAI_. 1725–1731.
  * Harper and Konstan (2015) F Maxwell Harper and Joseph A Konstan. 2015.  The movielens datasets: History and context.  _Acm transactions on interactive intelligent systems (tiis)_ 5, 4 (2015), 1–19.
  * Hidasi et al. (2016) Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2016.  Session-based recommendations with recurrent neural networks. In _ICLR_.
  * Kang and McAuley (2018) Wang-Cheng Kang and Julian McAuley. 2018.  Self-attentive sequential recommendation. In _2018 IEEE International Conference on Data Mining (ICDM)_. IEEE, 197–206.
  * Li et al. (2020) Jiacheng Li, Yujie Wang, and Julian McAuley. 2020.  Time interval aware self-attention for sequential recommendation. In _Proceedings of the 13th international conference on web search and data mining_. 322–330.
  * Lian et al. (2018) Jianxun Lian, Xiaohuan Zhou, Fuzheng Zhang, Zhongxia Chen, Xing Xie, and Guangzhong Sun. 2018\.  xdeepfm: Combining explicit and implicit feature interactions for recommender systems. In _Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining_. 1754–1763.
  * Ni et al. (2019) Jianmo Ni, Jiacheng Li, and Julian McAuley. 2019.  Justifying recommendations using distantly-labeled reviews and fine-grained aspects. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. 188–197.
  * Shen et al. (2021) Qijie Shen, Wanjie Tao, Jing Zhang, Hong Wen, Zulong Chen, and Quan Lu. 2021\.  SAR-Net: A Scenario-Aware Ranking Network for Personalized Fair Recommendation in Hundreds of Travel Scenarios. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_. 4094–4103.
  * Sun et al. (2019) Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang. 2019.  BERT4Rec: Sequential recommendation with bidirectional encoder representations from transformer. In _Proceedings of the 28th ACM international conference on information and knowledge management_. 1441–1450.
  * Tan et al. (2016) Yong Kiam Tan, Xinxing Xu, and Yong Liu. 2016.  Improved recurrent neural networks for session-based recommendations. In _RecSys Workshop_. 17–22.
  * Tang and Wang (2018) Jiaxi Tang and Ke Wang. 2018.  Personalized top-n sequential recommendation via convolutional sequence embedding. In _WSDM_. 565–573.
  * Xiao et al. (2017) Jun Xiao, Hao Ye, Xiangnan He, Hanwang Zhang, Fei Wu, and Tat-Seng Chua. 2017\.  Attentional factorization machines: Learning the weight of feature interactions via attention networks. In _IJCAI_. 3119–3125.
  * Xu et al. (2019b) Chengfeng Xu, Pengpeng Zhao, Yanchi Liu, Jiajie Xu, Victor S Sheng S. Sheng, Zhiming Cui, Xiaofang Zhou, and Hui Xiong. 2019b.  Recurrent convolutional neural network for sequential recommendation. In _WWW_. 3398–3404.
  * Xu et al. (2019a) Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. 2019a.  Self-attention with functional time representation learning.  _Advances in neural information processing systems_ 32 (2019).
  * Xu et al. (2020) Da Xu, Chuanwei Ruan, Evren Korpeoglu, Sushant Kumar, and Kannan Achan. 2020.  Inductive representation learning on temporal graphs.  _arXiv preprint arXiv:2002.07962_ (2020).
  * Yan et al. (2019) An Yan, Shuo Cheng, Wang-Cheng Kang, Mengting Wan, and Julian McAuley. 2019.  CosRec: 2D convolutional neural networks for sequential recommendation. In _CIKM_. 2173–2176.
  * Yan et al. (2014) Ling Yan, Wu-Jun Li, Gui-Rong Xue, and Dingyi Han. 2014\.  Coupled group lasso for web-scale ctr prediction in display advertising. In _International Conference on Machine Learning_. PMLR, 802–810.
  * Ye et al. (2020) Wenwen Ye, Shuaiqiang Wang, Xu Chen, Xuepeng Wang, Zheng Qin, and Dawei Yin. 2020\.  Time Matters: Sequential Recommendation with Complex Temporal Information. In _SIGIR_. 1459–1468.
  * Zhou et al. (2018a) Chang Zhou, Jinze Bai, Junshuai Song, Xiaofei Liu, Zhengchao Zhao, Xiusi Chen, and Jun Gao. 2018a.  Atrank: An attention-based user behavior modeling framework for recommendation. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 32.
  * Zhou et al. (2019) Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. 2019.  Deep interest evolution network for click-through rate prediction. In _Proceedings of the AAAI conference on artificial intelligence_ , Vol. 33. 5941–5948.
  * Zhou et al. (2018b) Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018b.  Deep interest network for click-through rate prediction. In _SIGKDD_. 1059–1068.
  * Zhu et al. (2017) Yu Zhu, Hao Li, Yikang Liao, Beidou Wang, Ziyu Guan, Haifeng Liu, and Deng Cai. 2017.  What to Do Next: Modeling User Behaviors by Time-LSTM.. In _IJCAI_. 3602–3608.


# Is this sentence valid? An Arabic Dataset for Commonsense Validation

Saja Tawalbeh, Mohammad AL-Smadi
Computer Science Department
Jordan University of Science and Technology
P.O.Box: 3030 Irbid 22110, Jordan.
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The commonsense understanding and validation remains a challenging task in the
field of natural language understanding. Therefore, several research papers
have been published that studied the capability of proposed systems to
evaluate the models ability to validate commonsense in text. In this paper, we
present a benchmark Arabic dataset for commonsense understanding and
validation as well as a baseline research and models trained using the same
dataset. To the best of our knowledge, this dataset is considered as the first
in the field of Arabic text commonsense validation. The dataset is distributed
under the Creative Commons BY-SA 4.0 license and can be found on
GitHub111Arabic dataset for commonsense validation,
https://github.com/msmadi/Arabic-Dataset-for-Commonsense-Validationion.

Index terms— Natural Language Processing, Commonsense Validation, Language
Models, Sense-making, Arabic Dataset

##  1 Introduction

Text understanding and validation demands the ability to understand between
lines instead of the general idea. The essential difference between the human
being and a machine regarding text understanding is that humans have the
ability to process and validate textual data and make sense out of it.
Consequently, the essential question is how to increase the ability of the
machine to have the ability of commonsense understanding. Therefore,
introducing commonsense to natural language understanding is considered as a
qualitative change of research to investigate the machine ability in solving
commonsense problems. Commonsense, in general, represents the aspect of the
relation semantic properties and a piece of text [Asher and Vieu, 1995].

In the past fifty years of AI research, automated commonsense reasoning has
barely progressed in the field of NLP research [Davis and Morgenstern, 2004].
In NLP, using transfer learning models [Yosinski et al., 2014] as well as
introducing transformers [Vaswani et al., 2017] has revolutionized the
research field and increased the ability of machines in text commonsense
validation and understanding. Furthermore, modern language models such as
BERT, ELMo has enhanced machines’ performance make-sense, validate and
understand text [Davis, 2017].

The existing dataset, covers commonsense knowledge indirectly. However,
recently, [Wang et al., 2019, Wang et al., 2020a] presented a dataset in
English that consists of two sentences, the machine should determine which one
is against commonsense, then should explain why the sentence against
commonsense, and finally, should generate the reason automatically.

To the best of our knowledge, there is no publicly available dataset in Arabic
for commonsense validation and understanding of text. Therefore, in this paper
present an Arabic dataset for commonsense validation. Moreover, we trained
popular transformers i.e. BERT, USE, and ULMFit to provide a baseline model
for the presented dataset. The models’ task is then to validate two natural
language sentences of similar wording and to identify the one that does not
make-sense.

##  2 Related Work

The natural language understanding frameworks are getting more attention after
commonsense studies were conducted [Wang et al., 2018]. Several datasets were
prepared such as the Choice of Plausible Alternatives (COPA) [Roemmele et al.,
2011] which focuses on events and consequences, in which each question given
two alternatives to recognize the appropriate cause or result of the premise.
Other datasets inspired by COPA was released, for instance, JHU Ordinal
Commonsense Inference (JOCI) dataset [Zhang et al., 2017] which has five
labels starting from 5 (very likely) to 1 (impossible), tends to identify the
plausibility of human response after a certain situation. Moreover, Situations
with Adversarial Generations (SWAG) [Zellers et al., 2018] is a large-scale
adversarial dataset for a grounded commonsense inference that is used to
determine the action that probably happens after a specific situation. The
main intention of SWAG dataset is to focus on the pre-situations and/or the
after-situations of certain situations without looking to the reasons why they
occur. The Story Cloze Test and ROCStories Corpora provided by [Mostafazadeh
et al., 2016, Sharma et al., 2018] inspired by grasp reading and text material
questions are used to identify the appropriate answer from the provided
materials. Ostermann, 2018 proposed a dataset for narrative text, which tells
about several type of questions with two candidate answers [Ostermann et al.,
2018].

Other Question Answering (QA) datasets for factual commonsense knowledge
reasoning were released. For instance, SQUABU [Davis, 2016] presented a simple
test of commonsense and scientific questions. Moreover, CommonsenseQA [Talmor
et al., 2018] provided a ConceptNet that has been used by crowd workers to
create questions for the presented dataset. Similarly, BookQA [Mihaylov et
al., 2018] presented questions and their candidate answers. COSMOS QA [Huang
et al., 2019] is large-scale dataset contains multiple-choice questions with
four answers.

Recently, Wang, 2020 proposed a multi-task [Wang et al., 2019, Wang et al.,
2020a] aiming to recognize the hidden inference of the commonsense facts using
SemEval commonsense validation and explanation dataset as well as performing
RoBERTa to solve the explanation task [Wang et al., 2020b]. Similarly, [Saeedi
et al., 2020] proposed a model using RoBERTa to solve the SemEval commonsense
validation and explanation task, they conducted this research by reframing the
classification task to multiple-choice task aims to increase the system’s
performance.

##  3 Dataset Collection and Translation

The Arabic dataset for commonsense validation is based on the Commonsense
Validation and Explanation (ComVE) task which is inspired by the work of [Wang
et al., 2019]. The original dataset 222
https://github.com/wangcunxiang/SemEval2020-Task4-Commonsense-Validation-and-
Explanation was prepared using Amazon Mechanical Turk where crowds were asked
to write a sensible sentence and non-sensible sentences as (s1 and s2). For
the sake of this research, we translated the original English dataset for
commonsense validation. Each example in the provided dataset is composed of 2
sentences: {s1, s2} and a label indicating which one is invalid. The pair of
sentences are too similar with a slight difference between there words to
identify which sentence does not make sense.

To the best of our knowledge, there is no Arabic dataset publicly available
for commonsense validation. The provided dataset has 12k rows and consists of
three files: train, validation, and test file. Each row consists of two
sentences and the label of the non-sensible sentence. Table 1 shows the
distribution ans size of the dataset. Whereas, Table 2 presents examples from
the Arabic dataset for the sense-making task.

File | Train | Validation | Test | Total
---|---|---|---|---
# of rows | 10000 | 1000 | 1000 | 12000
Table 1: The Dataset distribution id | Sent0 | Sent1 | label
---|---|---|---
26 |  أنا ألبس فيل I dress an elephant |  أنا ألبس بنفسي I dress myself | 0
34 |  قرأت النجوم I read stars |  قرأتني النجوم Stars read me | 1
4271 |  كرة القدم مربعة Football is square |  كرة القدم مستديرة Football is round | 0
5805 |  الأرض أصغر من الشمس the earth is smaller than the sun |  الأرض أصغر من القمر the earth is smaller than the moon | 1
9231 |  تطير الطيور في السماء birds fly in the sky |  يطير البشر في السماء humans fly in the sky | 1
Table 2:  Examples from the provided Arabic dataset with the English original
ones. The label indicates the id of the invalid sentence (i.e. 0: for the
first sentence or 1: for the second sentence)

##  4 Experimentation & Results

As baseline research, several state-of-the-art models i.e. BERT [Devlin et
al., 2018], USE [Cer et al., 2018], and ULMFit [Howard and Ruder, 2018] have
been trained using the proposed Arabic dataset to make-sense of the provided
couple of sentences. Trained models were evaluated based on their performance
accuracy.

As shown in Table 3, BERT has a significant advantage compared to ULMFiT and
USE. Results go in line with previous research where BERT has achieved 70.1%
accuracy for the original English dataset [Wang et al., 2019] overcoming other
models such as ULMFit.

Model | Accuracy
---|---
USE | 48.3
ULMFiT | 50.5
BERT | 62.7%
Table 3:  The experimentation results for the trained baseline models

##  5 Conclusion

This paper presents a benchmark dataset for Arabic text commonsense and
validation. Presented dataset were used to train baseline models such as BERT
[Devlin et al., 2018], USE [Cer et al., 2018], and ULMFit [Howard and Ruder,
2018]. Evaluation results show that BERT outperformed the other two models and
achieved a 62.7% in making-sense and validation input sentences.

## Acknowledgments

This research is partially funded by Jordan University of Science and
Technology, Research Grant Number: 20170107.

## References

  * [Asher and Vieu, 1995]  Nicholas Asher and Laure Vieu.  1995\.  Toward a geometry of common sense: A semantics and a complete axiomatization of mereotopology.  In IJCAI (1), pages 846–852.
  * [Cer et al., 2018]  Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al.  2018\.  Universal sentence encoder.  arXiv preprint arXiv:1803.11175.
  * [Davis and Morgenstern, 2004]  Ernest Davis and Leora Morgenstern.  2004\.  Introduction: Progress in formal commonsense reasoning.  Artificial Intelligence, 153(1-2):1–12.
  * [Davis, 2016]  Ernest Davis.  2016\.  How to write science questions that are easy for people and hard for computers.  AI magazine, 37(1):13–22.
  * [Davis, 2017]  Ernest Davis.  2017\.  Logical formalizations of commonsense reasoning: a survey.  Journal of Artificial Intelligence Research, 59:651–723.
  * [Devlin et al., 2018]  Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  2018\.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805.
  * [Howard and Ruder, 2018]  Jeremy Howard and Sebastian Ruder.  2018\.  Universal language model fine-tuning for text classification.  arXiv preprint arXiv:1801.06146.
  * [Huang et al., 2019]  Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.  2019\.  Cosmos qa: Machine reading comprehension with contextual commonsense reasoning.  arXiv preprint arXiv:1909.00277.
  * [Mihaylov et al., 2018]  Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.  2018\.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  arXiv preprint arXiv:1809.02789.
  * [Mostafazadeh et al., 2016]  Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen.  2016\.  A corpus and cloze evaluation for deeper understanding of commonsense stories.  In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 839–849.
  * [Ostermann et al., 2018]  Simon Ostermann, Ashutosh Modi, Michael Roth, Stefan Thater, and Manfred Pinkal.  2018\.  Mcscript: A novel dataset for assessing machine comprehension using script knowledge.  arXiv preprint arXiv:1803.05223.
  * [Roemmele et al., 2011]  Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon.  2011\.  Choice of plausible alternatives: An evaluation of commonsense causal reasoning.  In 2011 AAAI Spring Symposium Series.
  * [Saeedi et al., 2020]  Sirwe Saeedi, Aliakbar Panahi, Seyran Saeedi, and Alvis C Fong.  2020\.  Cs-nlp team at semeval-2020 task 4: Evaluation of state-of-the-artnlp deep learning architectures on commonsense reasoning task.  arXiv preprint arXiv:2006.01205.
  * [Sharma et al., 2018]  Rishi Sharma, James Allen, Omid Bakhshandeh, and Nasrin Mostafazadeh.  2018\.  Tackling the story ending biases in the story cloze test.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 752–757.
  * [Talmor et al., 2018]  Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant.  2018\.  Commonsenseqa: A question answering challenge targeting commonsense knowledge.  arXiv preprint arXiv:1811.00937.
  * [Vaswani et al., 2017]  Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  2017\.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008.
  * [Wang et al., 2018]  Su Wang, Greg Durrett, and Katrin Erk.  2018\.  Modeling semantic plausibility by injecting world knowledge.  arXiv preprint arXiv:1804.00619.
  * [Wang et al., 2019]  Cunxiang Wang, Shuailong Liang, Yue Zhang, Xiaonan Li, and Tian Gao.  2019\.  Does it make sense? and why? a pilot study for sense making and explanation.  arXiv preprint arXiv:1906.00363.
  * [Wang et al., 2020a]  Cunxiang Wang, Shuailong Liang, Yili Jin, Yilong Wang, Xiaodan Zhu, and Yue Zhang.  2020a.  Semeval-2020 task 4: Commonsense validation and explanation.  arXiv preprint arXiv:2007.00236.
  * [Wang et al., 2020b]  Hongru Wang, Xiangru Tang, Sunny Lai, and Kwong Sak Leung.  2020b.  Cuhk at semeval-2020 task 4: Commonsense explanation, reasoning and prediction with multi-task learning.  arXiv preprint arXiv:2006.09161.
  * [Yosinski et al., 2014]  Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson.  2014\.  How transferable are features in deep neural networks?  In Advances in neural information processing systems, pages 3320–3328.
  * [Zellers et al., 2018]  Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi.  2018\.  Swag: A large-scale adversarial dataset for grounded commonsense inference.  arXiv preprint arXiv:1808.05326.
  * [Zhang et al., 2017]  Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme.  2017\.  Ordinal common-sense inference.  Transactions of the Association for Computational Linguistics, 5:379–395.