#  Hierarchically Fusing Long and Short-Term User Interests for Click-Through
Rate Prediction in Product Search

Qijie Shen  Alibaba GroupHangzhouChina<EMAIL_ADDRESS>,  Hong Wen
Alibaba GroupHangzhouChina<EMAIL_ADDRESS>,  Jing Zhang  The
University of SydneyDarlington NSW 2008Australia<EMAIL_ADDRESS>and
Qi Rao  Alibaba GroupHangzhouChina<EMAIL_ADDRESS>
(2022)

###### Abstract.

Estimating Click-Through Rate (CTR) is a vital yet challenging task in
personalized product search. However, existing CTR methods still struggle in
the product search settings due to the following three challenges including
how to more effectively extract users’ short-term interests with respect to
multiple aspects, how to extract and fuse users’ long-term interest with
short-term interests, how to address the entangling characteristic of long and
short-term interests. To resolve these challenges, in this paper, we propose a
new approach named Hierarchical Interests Fusing Network (HIFN), which
consists of four basic modules namely Short-term Interests Extractor (SIE),
Long-term Interests Extractor (LIE), Interests Fusion Module (IFM) and
Interests Disentanglement Module (IDM). Specifically, SIE is proposed to
extract user’s short-term interests by integrating three fundamental interests
encoders within it namely query-dependent, target-dependent and causal-
dependent interest encoder, respectively, followed by delivering the resultant
representation to the module LIE, where it can effectively capture user long-
term interests by devising an attention mechanism with respect to the short-
term interests from SIE module. In IFM, the achieved long and short-term
interests are further fused in an adaptive manner, followed by concatenating
it with original raw context features for the final prediction result. Last
but not least, considering the entangling characteristic of long and short-
term interests, IDM further devises a self-supervised framework to disentangle
long and short-term interests. Extensive offline and online evaluations on a
real-world e-commerce platform demonstrate the superiority of HIFN over state-
of-the-art methods.

Personalized Product Search, Click-Through Rate Prediction

††copyright: acmcopyright††journalyear: 2022††copyright:
acmcopyright††conference: Proceedings of the 31st ACM International Conference
on Information and Knowledge Management; October 17–21, 2022; Atlanta, GA,
USA††booktitle: Proceedings of the 31st ACM International Conference on
Information and Knowledge Management (CIKM ’22), October 17–21, 2022, Atlanta,
GA, USA††price: 15.00††doi: 10.1145/3511808.3557351††isbn:
978-1-4503-9236-5/22/10††ccs: Information system Information retrieval

##  1\. Introduction

With the explosive emergence of products in e-commerce platforms, product
search serves an indispensable role in discovering desirable items that
satisfy users. When a user submits a query, the product search will deliver a
small yet well ranked product set from billions of candidates to end-users
through a typical multi-stage pipeline of
“match$\to$prerank$\to$rank$\to$rerank” (Li et al., 2021, 2020; Zhang and Tao,
2020). In this paper, we only focus on the Click Through-Rate (CTR) prediction
task in the _rank_ stage, which aims to predict the probability of users
clicking items and has a great impact on improving user experience and
boosting the revenue of e-commerce platforms.

The prerequisite for a CTR prediction task in personalized product search is
to accurately extract the user preferences from historical behaviors and
effectively integrate them with the current query (Guo et al., 2019). Besides,
it has been well recognized that there are two types of user preferences
(Bennett et al., 2012), $i.e.$, long and short-term ones, where the former
exhibits users’ inherent and relatively stable (evolving slowly) preferences,
such as preferred color, fitting size, price preferences, imperceptibly
influenced by the user’s family background, age, marital status, education,
$etc$. While the short-term interests convey user preference intention in a
relatively short period, which can be inferred from their recent behaviors and
also affected by incidentally transient events, such as new product release,
season change and special personal occasions like birthday (Xiang et al.,
2010). In a nutshell, users’ short-term interests evolve in a more high-
frequently and drastically manner compared with the long-term ones.

A typical paradigm of existing personalized product search models is to
represent the user intents and items with embedding vectors explicitly,
followed by matching them in the latent space with dot product or feeding them
to neural layers to yield the prediction score (Bi et al., 2020). For example,
HEM (Ai et al., 2017) fuses user vector and query vector to represent user
intent with a convex combination. AEM (Ai et al., 2019) is the first
attention-based embedding model which constructs query-dependent user
embeddings by employing attention mechanism to users’ historical purchased
items with respect to the query. ZAM (Ai et al., 2019) devises a zero
attention mechanism to determine when and how to conduct personalization under
various scenarios. TEM (Bi et al., 2020) dynamically control the influence of
personalization by encoding the sequence of query and user’s historical
behaviors with a transformer architecture. Guo (Guo et al., 2019) proposes a
novel Attentive Long Short-Term Preference model (abbreviated as ALSTP) to
learn and integrate the long and short-term user preferences with respect to
the current query in a typical personalized product search scenario. Despite
its effectiveness, we argue that the existing product search paradigm has
several obvious limitations.

First, existing methods always extract users’ interest representation by
aggregating user historical behaviors with respect to the query submitted by
users. We argue that the strategy can work efficiently only when users exhibit
their demands with submitted queries explicitly. For example, when a user
submits a query ”red canvas shoes”, it can express the user’s requirements on
color, type, and material clearly, resulting in the effectiveness of
extracting query-dependent interest by straightforwardly employing attention
mechanism to user behaviors with respect to the submitted query. However,
users sometimes could express their demands with obscure intention. For
example, when a user issues a query ”gift”, which indicates the user does not
specify the brand, category, style, price, $etc.$ he (she) is desiring at this
moment, resulting in the various types of candidate items delivered by
previous stage _match_. Therefore, we need to extract user’s current precious
interest delivered from historical behaviors with respect to individual target
item. In other words, an alternative solution is to employ attention mechanism
with respect to target product instead of query in such situation to extract
target-dependent user interest.

Second, there exists various types of historical behaviors, such as _click_ ,
_favorite_ , or _purchase_ , which are inadvertently neglected by previous
works or just regarded as certain a kind of behavior features integrated into
original framework. We argue multiple kinds of behaviors contain abundant
information that is worth investigating deeply for superior user interests
representation. However, it is not trivial to integrate them into original
framework. For example, the purchase behaviors delivered by some users denotes
their recent desired demands have been satisfied eventually, which implies a
smaller weight on the purchase behavior contributing to the final
representation of user interests. On the contrary, some users probably click,
or favorite after a certain click behavior on a special category, which
denotes the intense of users’ interests are continuously increasing, resulting
in a higher weight should be given to this kind of behavior for final
representation of user interest. In this paper, we regard the phenomenon,
$i.e.$, the causality of multiple types of user behaviors, as causal-dependent
user interest.

Third, although the state-of-the-art long and short- term interests method,
$i.e.$, ALSTP, has achieved a significant performance, it still struggle in
achieving better performance since it only considers the query-dependent user
interest, neglecting the target-dependent interest and the causal-dependent
interest. Therefore, it will lead to inferior performance in some cases.
Besides, ALSTP just concatenates the long and short-term interests and
entangles both aspects together for final prediction without explicitly
distinguishing the importance of long and short-term interests to final
prediction, which will lead to inferior accuracy and interpretability.

To address aforementioned challenges, we proposed a novel method named
Hierarchical Interests Fusing Network (HIFN) for CTR prediction in product
search, which consists of four basic modules namely Short-term Interests
Extractor (SIE), Long-term Interests Extractor (LIE), Interests Fusion Module
(IFM) and Interests Disentanglement Module (IDM). Specifically, SIE is
proposed to extract user’s short-term interests by employing three interest
encoders, $i.e.$, query-dependent interest encoder (QDIE), target-dependent
interest encoder (TDIE) and causal-dependent interest encoder (CDIE) to
extract query-dependent, target-dependent and causal-dependent interests,
respectively, followed by fusing these three interests in an adaptive manner.
In LIE, we capture user long-term interest by devising an attention mechanism
with respect to the output representation of SIE module, followed by fusing
the long and short-term interest in module IFM. Next, all the representation
features are concatenated with original raw context features and fed into
fully connected layers to generate the final prediction results. Last but not
least, motivated by DLSR (Zheng et al., 2022), IDM injects pseudo labels for
both long and short-term interest and employs an self-supervised framework to
disentangle long and short-term interests. But different from DLSR, we devise
an updating strategy to set long-term interest proxy, which is more suitable
and achieves better performance than the settings in DLSR.The contribution of
this paper is three-fold:

  * •

We propose a novel method named HIFN for CTR prediction in product search,
which facilitates learning long and short-term user interests representations
in a hierarchical manner.

  * •

We devise four key modules including SIE, LIE, IFM and IDM to jointly address
the aforementioned challenges in the context of product search, which can be
implemented efficiently in a multi-task end-to-end learning framework.

  * •

We conduct extensive experiments on real-world offline datasets and online A/B
test. Both results demonstrate the effectiveness of HIFN over state-of-the-art
methods.

Figure 1. The overview architecture of the proposed HIFN. It consists of
several modules, i.e., SIE, LIE, IFM and IDM.

##  2\. Proposed Method

In this paper, we propose a novel method named HIFN, as depicted in Figure 1.
It consists of four basic modules namely Short-term Interest Extractor (SIE),
Long-term Interest Extractor (LIE), Interest Fusion Module (IFM) and Interest
Disentanglement Module (IDM). We will detail them as follows.

###  2.1. Problem Definition

Given user profiles $u_{p}$, long-term user behaviors $u_{b_{l}}$, short-term
user behaviors $u_{b_{s}}$, target item $i$, search query $q$, context feature
$c$, where $u_{b_{l}}=\left\\{i_{1},i_{2},...i_{T_{l}}\right\\}$,
$u_{b_{s}}=\left\\{i_{1},i_{2},...i_{T_{s}}\right\\}$, and $i_{t}$ from set
$u_{b_{l}}$ (resp. $u_{b_{s}}$) represents user’s $t$-th interacted item from
long-term (resp. short-term) behaviors. Besides, $T_{l}$ (resp. $T_{s}$)
denotes the length of user’s long-term (resp. short-term) behaviors. Next, we
firstly encode them into one-hot high-dimensional vectors, followed by
transforming these vectors into low-dimensional dense vectors by ways of
multiplying corresponding embedding matrices (Tao et al., 2022b, a; Xu et al.,
2022). After concatenating various types of dense vectors, we obtain the
embedding vectors $e_{u_{p}}$, $x_{u_{b_{l}}}$, $x_{u_{b_{s}}}$, $e_{i}$,
$e_{q}$, $e_{c}$ for $u_{p}$, $u_{b_{l}}$, $u_{b_{s}}$, $i$, $q$, $c$,
respectively, where
$x_{u_{b_{s}}}=\left\\{e_{i_{1}},e_{i_{2}},...e_{i_{T_{s}}}\right\\}$,
$x_{u_{b_{l}}}=\left\\{e_{i_{1}},e_{i_{2}},...e_{i_{T_{l}}}\right\\}$, and
$e_{i_{t}}$ is the embedding vector of user’s $t$-th interacted item. Our goal
is to devise a model to predict the probability of user $u$ clicking target
item $i$ given search query $c$ and corresponding relevant features, formally
defined as:
$\hat{y}_{u,i}=\mathcal{F}(e_{u_{p}},x_{u_{b_{l}}},x_{u_{b_{s}}},e_{i},e_{q},e_{c};\theta)$,
where $\mathcal{F}$ is the learning objective with model parameters $\theta$.

###  2.2. Short-Term Interest Extractor

In general, users’ short-term behaviors play a vital role to infer their
recent interests. Although existing works towards extracting query-dependent
interest in product search have showed significant performance, they still
struggle in explicitly delivering users’ interests due to the confusion of
users’ multiple interests in some conditions. Therefore, besides query-
dependent interest extraction, an additional supplement is to extract
corresponding interest representation from users’ behaviors with respect to
individual candidate product, called _target-dependent_ interest extraction.
In addition, users always exhibit various types of behaviors, $i.e.$, _click_
or _purchase_ , while neglected by previous works when extracting users’
interests from their behaviors or just regarded as certain a kind of behavior
features. And we argue users’ multiple types of behaviors are very essential
for users’ final interest extraction, called _causal-dependent_ interest
extraction.

Based on the above considerations, we first adopt a GRU (Cho et al., 2014) to
model the short-term interests, as it has been successfully applied in the
session-based recommendation (Hidasi et al., 2015). The RNN module takes the
recent $s$ product representations
$x_{u_{b_{s}}}=\left\\{e_{i_{1}},e_{i_{2}},...e_{i_{T_{s}}}\right\\}$ as
inputs. We thus can obtain a set of high dimensional hidden representations
$x_{u_{b_{s}}}^{,}=\left\\{e_{i_{1}}^{,},e_{i_{2}}^{,},...e_{i_{T_{s}}^{,}}\right\\}$,
denoting the short-term user preference.

Then, We feed $x_{u_{b_{s}}}^{,}$ into three well-designed interest encoder
named Query-dependent Interest Encoder, Target-dependent Interest Encoder,
Causal-dependent Interest Encoder, respectively.

####  2.2.1. Query-dependent Interest Encoder

Given users express their intention with submitted queries explicitly, we can
straightforwardly devise a module named _Query-Dependent Interest Encoder_
(QDIE) to accurately extract users’ current interests from their behaviors
with respect to the submitted query. After considering the varied relevance of
historical behaviors with respect to the current submitted queries, we resort
to the attention mechanism in QDIE module to extract query-dependent interest,
which can be formulated as:

(1) $\begin{split}u_{q}=\sum_{i\in
x_{u_{b_{s}}}^{\prime}}{\frac{exp(f(e_{q},e_{i}))}{\sum_{i^{\prime}\in
x_{u_{b_{s}}}^{\prime}}exp(f(e_{q},e_{i^{\prime}}))}{}}e_{i}\end{split}$

where $f(q,i)$ is an attention function that defines the weight of each item
$i$ with respect to the current query $q$, defined as:

(2) $\begin{split}f(q,i)=(i\cdot tanh(W_{f}\cdot q+b_{f}))\cdot
W_{h}\end{split}$

where $W_{h}\in\mathbb{R}^{\beta}$,
$W_{f}\in\mathbb{R}^{\alpha\times\beta\times\alpha}$,
$b_{f}\in\mathbb{R}^{\alpha\times\beta}$.

####  2.2.2. Target-dependent Interest Encoder

Practically, users sometimes could not clearly express their search intention
with submitted queries. Under such circumstances, we employ a module named
_Target-Dependent Interest Encoder_ (TDIE) to extract users’ current interest
from their behaviors with respect to individual target item. Take the travel
scenario as example, when a user has many online clicking behaviors on the
scenic spots from city _Hangzhou_ and after he(she) submitted a query named
_Hangzhou_ , we can probably infer the user prefers the well-known spot _West
Lake_ more compared with _snack_ in _Hangzhou_. Similar with Query-dependent
Interest Encoder, we directly employ an attention mechanism on
$x_{u_{b_{s}}}^{{}^{\prime}}$ with respect to target item $e_{i}$ to achieve
target-dependent interest $u_{t}$.

####  2.2.3. Causal-dependent Interest Encoder

Users usually exhibit diverse and continuous interests by ways of various
types of behaviors, $i.e.$, _click_ or _purchase_. For example, if users are
impressed by _clothes_ , they will naturally deliver continuous behaviors on
_clothes_ related products for a period of time until their needs are
satisfied, $i.e.$, desired clothes have been bought eventually.
Mathematically, let $e_{i}^{t}$ in user behaviors $x_{u_{b_{s}}}$ denote the
category of the $i$-th behavior item at time $t$. In fact, we can extract four
auxiliary sub-sequences from $x_{u_{b_{l}}}$ for the behavior item at time $t$
according to whether click (or purchase) behaviors before(or after) time $t$
have the same category with $e_{i}^{t}$, $i.e.$, click sub-sequence before
time $t$, $x_{bc}^{i}$, purchase sub-sequence before time $t$, $x_{bp}^{i}$,
click sub-sequence after time $t$, $x_{ac}^{i}$, purchase sub-sequence after
time $t$, $x_{ap}^{i}$. We argue that some users would express intense
interests since they continuously exhibit click behaviors with the same
category to the one at $t-$th behavior, whether before or after $t$,
especially when users have finished purchase behavior at time $t$. Meanwhile,
some users may have continuous click behaviors before time $t$ with the same
category to the one at time $t$, while they show the decrement of interests
after they finished purchase behaviors. Therefore, we employ the four
auxiliary sub-sequences to portray users’ precious intention at time $t$,
formulated as:

(3)
$\begin{split}\alpha_{i}=Sigmoid(FFN(Sum(x_{bc}^{i})||Sum(x_{bp}^{i})||Sum(x_{ac}^{i})||Sum(x_{ap}^{i})))\end{split}$

where $Sum(\cdot)$ indicates sum pooling operator, the reason for using sum
pooling is that it can express the density of behavior and achieving great
performance, however the computational complexity is lower than others (e.g.
GRU, LSTM). Then, causal-dependent Interest is calculated as follows:

(4) $\textit{u}_{c}=\sum_{i}\alpha_{i}e_{i},$

####  2.2.4. Short-term Interest Fusing

As mentioned earlier, the three aspects both play an important role under
different user intentions, we employ a gate network to fusing three aspects of
short-term interest. The structure of the gate network is based on a single-
layer feed-forward network with a SoftMax activation function. It acts as a
selector to calculate the weighted sum of the selected vector.

(5) $u_{short}=SoftMax(FFN(u_{t},u_{q},u_{c}))^{T}[u_{t},u_{q},u_{c}],$

###  2.3. Long-Term Interests Extractor

Generally, users’ behaviors are categorized into two disjoint groups, $i.e.$,
long-term behaviors and short-term ones. When users’ short-term behaviors are
sparse or even absent, the long-term behaviors will be as a kind of supplement
cues to discover their interests. Moreover, users’ long-term behaviors are
regarded as the reflection of their intrinsic tastes while evolving slowly
over time, which further affects their short-term behaviors in a relatively
recent period. Motivated by this, we propose a Long-Term Interest Extractor
(LTIE) module to extract users’ long interests. Specially, we employ an
attention mechanism to long-term behaviors with respect to the generated
short-term interest $u_{short}$ like Eq. ( 1 ) and Eq. ( 2 ) to deliver users’
long-term interests $u_{long}$.

###  2.4. Interests Fusing Module

Given users’ long- and short-term interests, how to fuse them is not trivial
due to the different contributions with respect to individual users. One
straightforward solution is employing concatenation operator to integrate
them, while it still struggles in how to effectively distinguish the
contribution of each interest. For example, basketball fans may continuously
click basketball shoes when a query ”shoes” submitted due to their long-term
interest even they have delivered a lots of canvas shoes related behaviors in
a recent period. Alternatively, we devise a module named Interest Fusing
Module to seamlessly integrate both interest representation together with
submitted query, historical behaviors, target item. Formally, the process can
be formulated as :

(6) $\alpha=sigmoid(FFN(e_{i}||e_{q}||u_{short}||u_{long}||\tau_{u_{b_{l}}}))$

(7) $\tau_{u_{b_{l}}}=GRU(x_{u_{b_{l}}})$

where, $\alpha$ denotes the estimated fusion weight based on aforementioned
information. Then, the final user interest representation can be formulated as
follows.

(8) $u=\alpha\cdot u_{short}+(1-\alpha)\cdot u_{long}$

Finally, we concatenate $u$, $e_{i}$, $e_{q}$ and $e_{c}$ to form a
comprehensive vector, followed by feeding it to a MLP layer with _Relu_ as
activation function to get the final predicted CTR score $\hat{y}$. Finally,
we define the objective function as follows:

(9) $Loss_{t}=-\frac{1}{N}\sum_{(x,y)\in S}(ylog\hat{y}+(1-y)log(1-\hat{y})),$

where $S$ denotes the training set with total size $N$, and
$y\in\left\\{0,1\right\\}$ is the ground truth label representing whether
click or not.

###  2.5. Interests Disentanglement Module

Desired by Zheng (Zheng et al., 2022) who introduces a self-supervised
framework for disentanglement, we adapt it for product search. We take the
mean representation of short-term behaviors as the proxy of short-term
interests. As for long-term interests, we argue that directly mean
representation of entire long-term behaviors applied by Zheng (Zheng et al.,
2022) will lead to sub-optimal performance. Long-term user preference is
relatively stable and updates slowly, if we mean representations simply, the
temporal variability of long-term interests would be ignored. Since short-term
interests are based on the most recent $s$ interacted items, the impact of
this drawback is minimal. We apply an updating mechanism to the extraction of
long-term interests proxy to capture the slowly evolving characteristic of
long-term interests. For simplicity, we use the first $Ts$ products to set it
up, and then update it with every $Ts$ product interacted (evolving
gradually). In this setting, $p$ is updated based on a session of $Ts$
products, which is also the window size of modeling the short-term user
preference. It is illustrated as follows:

(10) $p_{long}=(1-\beta)p_{long}+\beta FFN(h_{t-1})$

where, FFN is multiple layers of feed forward network, $h_{t-1}$ is the mean
representation of previous $Ts$ interacted items. As for short-term proxy, it
is calculated as follows:

(11)
$p_{short}=MeanPooling\left\\{e_{i_{1}},e_{i_{2}},...e_{i_{T_{s}}}\right\\}$

With proxies serving as labels, we can utilize them to supervise the
disentanglement of long and short-term interests. Specifically, we perform
contrastive learning (Liu et al., 2021) between the outputs of interest
extractors and proxies,which requires the learned representations of long and
short-term interests to be more similar to their corresponding proxies than
the opposite proxies. Formally, there are four contrastive tasks as follows,

(12) $sim(u_{long},p_{long})>sim(u_{long},p_{short})$ (13)
$sim(p_{long},u_{long})>sim(p_{long},u_{short})$ (14)
$sim(u_{short},p_{short})>sim(u_{short},p_{long})$ (15)
$sim(p_{short},u_{short})>sim(p_{short},u_{long})$

where Eqn (12)-(13) supervise long-term interests, and Eqn (14)-(15) supervise
short-term interests, and $sim(\cdot,\cdot)$ measures embedding similarity. we
add self-supervision on long and short-term interests modeling which can
achieve stronger disentanglement compared with existing unsupervised
approaches. We implement a pairwise loss functions based on Bayesian
Personalized Ranking (BPR) to accomplish contrastive learning, are computed as
follows:

(16) $\text{\pounds}_{bpr}(a,p,q)=softplus(<a,q>-<a,p>)$

(17)
$\begin{split}\text{\pounds}_{con}=\text{\pounds}_{bpr}(u_{long},p_{long},p_{short})+\text{\pounds}_{bpr}(p_{long},u_{long},u_{short})+\\\
\text{\pounds}_{bpr}(u_{short},p_{short},p_{long})+\text{\pounds}_{bpr}(p_{short},u_{short},u_{long}).\end{split}$

We train the model in an end-to-end manner with multi-task learning on two
objectives. Specifically, the joint loss function with a hyper-parameter
$\lambda$ to balance objectives, formulated as follows:

(18) $\text{\pounds}=Loss_{t}+\lambda\text{\pounds}_{con}$

##  3\. Experiment

To verify the effectiveness of HIFN, we conduct extensive experiments to
compare it with representative SOTA methods on both offline datasets and
online deployment.

###  3.1. Dataset

We use two datasets to conduct experiments, including a public dataset and an
industrial dataset namely Amazon and Fliggy, respectively, which are both with
million users scale and collected from real-world applications. The details of
the adopted datasets are introduced as follows. Statistics of datasets are
listed in Table 1.

Fliggy111One of the most popular online travel platforms in china,
www.fliggy.com The Fliggy dataset which contains properties of users, queries,
user historically behaviors ( including click, favorite, and purchase ), and
target product, is generated based on user logs collected from October 1th to
October 31th, 2021. Note that users with fewer than 20 historical interactions
are removed. User logs of the first 30 days in October 2021 is used as the
training data, while reserve the last day in October for validation (before 12
pm) and test (after 12 pm). Negative samples in the training dataset are set
to those impressed products but not clicked by users, and posivtie samples are
clicked.

Table 1. Statistics of the offline dataset. ( QL indicates Query Length, PL indicated Product Length. )  Dataset |  Clothing, shoes&jewelry | Electronics | Toys&Gift | Fliggy
---|---|---|---|---
Users | 36421 | 34489 | 76234 | 2029216
Products | 22521 | 18397 | 63323 | 920487
Queries | 1998 | 1499 | 645 | 82409
Avg.QL | 8.23 | 8.74 | 6.21 | 13.41
Avg.PL | 75.36 | 95.36 | 196.3 | 53.67
#Query-User Pairs |  |  |
Train | 456521 | 273890 | 1493132 | 26379808
Valid/Test | 4006/4151 | 753/890 | 10210/10131 | 83129/89936

Amazon222http://jmcauley.ucsd.edu/data/amazon As there are no large-scale
public datasets in the personalized product search area, we use Amazon dataset
as our experiment corpus, consistent with existing approaches (Guo et al.,
2019; Ai et al., 2019). In our experiments, we adopted the five-core version
provided, whereby the remaining users and products have at least five reviews,
respectively. Besides, we selected three categories with different sizes:
Clothing, Shoes & Jewelry, Toys & Games, Electronics. These datasets both
contain several categories so that users may have different interests.
Following the strategy in References (He and McAuley, 2016), we extracted the
users’ product purchasing behaviors based on their reviews, i.e., the products
they reviewed are the ones they purchased. Our model uses the previously
purchased products in a neighboring window size to model the short-term user
interests. We further filtered the dataset to make sure each user has at least
20 purchased products (i.e., 20 reviews).

Following the same paradigm used in (Guo et al., 2019), we construct queries
for each purchased item with category information. This strategy is based on
the finding that directed product search is users’ search for a producer’s
name, a brand, or a set of terms describing product category. We partitioned
each of the four datasets into three sets: training, validation and testing
sets. We first extracted the user-product pairs from users’ reviews, and then
extracted the queries for these products, getting triplets. For each dataset,
the last purchasing transaction of each user is held for the testing set, the
second last for the validation set, and the rest for the training set.
Moreover, we hid the reviews of the validation and testing sets in the
training phase to simulate the real-world scenarios.

Since Amazon datasets do not provide negative samples, we adopt a negative
sampling method to evaluate it. In detail, for each positive user-query-item
triplet, we generate negative products with the same category as query
randomly to simulate a real search scenario. Then, the generated products,
query and user form negative triplet samples. We generate ten negative samples
for each positive triplet.

###  3.2. Competitors

To verify the effectiveness of the proposed method, we compare it with
following methods:

  * •

AEM (Ai et al., 2019): The Attention-based Embedding Model (AEM) constructs
query-dependent user embeddings by attending to users’ historical purchased
items with the query.

  * •

ZAM (Ai et al., 2019): The Zero Attention Model (ZAM) extends AEM with a zero
vector and conducts differentiated personalization by allowing the query to
attend to the zero vector.

  * •

TEM (Bi et al., 2020): The Transformer-based Embedding Model (TEM) [6] is a
state-of-the-art model that encodes query and historically purchased items
with a transformer and does item generation based on the encoded query-user
information.

  * •

DeepFM (Guo et al., 2017): It combines the explicit high-order interaction
module with deep MLP module and traditional FM module, and requires no manual
feature engineering.

  * •

DIN (Zhou et al., 2018): It utilizes attention mechanism to activate relevant
users’ behaviors with respect to corresponding targets and learns an adaptive
representation vector for users’ interests.

  * •

DIEN (Zhou et al., 2019): It adopts an interest extractor layer to capture
temporal interests from users’ historical behaviors and integrates GRUs with
attention mechanism for further capturing the involved interests with respect
to the target item.

  * •

ZAM+TDIE : Compared with ZAM, we add a Target-dependent Interest Encoder
(TDIE) to model the target-dependent user interest. The target-dependent user
interest is concatenated with user representation to form a new user interest
vector.

  * •

TEM+TDIE : Do the same operation to TEM as ZAM+TDIE.

  * •

ALSTP (Guo et al., 2019): It represent long and short-term user preferences
with an attention mechanism applied to users’ recent interacted products and
their global vectors. It is the sota long and short-term modeling method for
personalized product search.

###  3.3. Metrics

In the experiments, we use AUC and logloss (or cross-entropy) (Shen et al.,
2021, 2022) to evaluate the ranking ability of all comparison models on the
Fliggy dataset. The larger AUC and smaller Logloss mean better ranking
performance. On public datasets, namely Amazon, the NDCG@10, MRR and MAP are
used (Guo et al., 2019), since these datasets do not provide negative samples
and the computation of AUC requires both positive and negative samples in a
dataset.

###  3.4. Implement details

We use the Adam optimizer. Embedding size d is set as 16. Batch normalization
is enabled for the MLP, and the activation function is ReLU. We use grid-
search to find the best hyper-parameters. The optimal settings for our
proposed implementation are: Batchsize is 512\. Learning rate is 0.001. Feed
forward networks (FFN) in IDM is set [128, 64, 32, 16], and FFN in short term
interest fusing is set [128, 64, 32, 3]. $\lambda$ is 0.1 in both datasets.
$Ts$ is 10 for Fliggy, 8 for Clothing,shoes&jewelry, 14 for Electronics, 12
for Toys&gift. $\beta$ is 0.55 for Fliggy, 0.35 for Clothing,shoes&jewelry,
0.45 for Electronics, 0.6 for Toys&gift. Value of each experimental result is
the average of 10 repeated tests.

###  3.5. Main Comparison Results

We illustrate the overall performance on two adopted datasets in Table 2. From
the results, we have the following observations:

  * •

It is important to extract the target-dependent user interest in product
search. AEM, ZAM, and TEM all model the correlation between query and user
behaviors to represent user interests. However, the user interests are not
always focused. For example, when a user issues a query ”clothing”, he does
not express preferences for colors, brands, and styles. Users will click navy
blue clothings instead of other colors in the result page, because the navy
blue color is the most common among the products purchased by users in the
history. If we only model the relationship between query and historical
behaviors to extract user interest vectors, we will ignore this important
point. It can be seen from the table that the methods of modeling query and
user historical behavior such as AEM, ZAM, TEM and the methods of modeling
target item and user historical behavior such as DIN and DIEN have different
performances in different datasets. However, after adding the module for
modeling the relationship between target item and user behavior sequence
(TDIE), the performance is significantly improved, which shows the importance
of modeling user interests from two perspectives at the same time.

  * •

Joint modeling of long and short-term interests does not always bring
performance gains for product search. Although ALSTP is the SOTA long and
short-term interests method in product search. However, there are many issues
have not been addressed carefully. First, short-term interests play an
important role for final prediction, but ALSTP only considers query-dependent
interest, neglecting the target-dependent and causal-dependent interests.
Second, simply concatenating long and short-term interests neglect the fact
that long and short-term interests have varying degrees of importance in
different situations. Third, modeling long and short-term interests entangled
with each other will increase model redundancy and leads to inferior accuracy.
Results demonstrate that ALSTP is not consistently effective across different
metrics and datasets. For example, ALSTP is the best baseline on Clothing
dataset, but its NDCG performance is poorer than ZAM by about 5.35% on
Electronics. It indicates that without carefully address the above issues,
modeling long and short-term interests can not get consistent improvements.

  * •

HIFN can achieve significant improvements over all datasets and metrics. HIFN
outperforms all competitors with significant progress. Specifically, HIFN
improves NDCG by about 0.07 on Clothing dataset and improves AUC by about
0.017 on Fliggy dataset, agaisnt SOTA methods. The consistent and significant
progress indicate that HIFN models long and short-term user interests in a
more proper manner than previous works.

Table 2. Overall performance on Fliggy and Amazon datasets. Dataset | Clothing,shoes&jewelry | Electronics | Toys&Games | Fliggy
---|---|---|---|---
Model | MRR | NDCG | MAP | MRR | NDCG | MAP | MRR | NDCG | MAP | AUC | Logloss
AEM | 0.024 | 0.025 | 0.019 | 0.269 | 0.291 | 0.265 | 0.068 | 0.072 | 0.062 | 0.735 | 0.598
ZAM | 0.025 | 0.027 | 0.021 | 0.289 | 0.315 | 0.288 | 0.079 | 0.112 | 0.088 | 0.734 | 0.601
TEM | 0.027 | 0.026 | 0.023 | 0.201 | 0.261 | 0.194 | 0.138 | 0.158 | 0.129 | 0.731 | 0.621
DeepFM | 0.020 | 0.028 | 0.016 | 0.222 | 0.296 | 0.286 | 0.073 | 0.104 | 0.073 | 0.737 | 0.588
DIN | 0.022 | 0.029 | 0.018 | 0.223 | 0.298 | 0.289 | 0.075 | 0.105 | 0.076 | 0.739 | 0.581
DIEN | 0.023 | 0.034 | 0.022 | 0.288 | 0.319 | 0.301 | 0.099 | 0.127 | 0.118 | 0.741 | 0.579
ZAM+TDIE | 0.031 | 0.036 | 0.026 | 0.311 | 0.318 | 0.322 | 0.106 | 0.144 | 0.123 | 0.746 | 0.552
TEM+TDIE | 0.032 | 0.034 | 0.028 | 0.315 | 0.289 | 0.291 | 0.141 | 0.162 | 0.134 | 0.749 | 0.541
ALSTP | 0.034 | 0.037 | 0.029 | 0.302 | 0.299 | 0.294 | 0.101 | 0.134 | 0.120 | 0.742 | 0.576
HIFN | 0.041 | 0.044 | 0.035 | 0.331 | 0.342 | 0.351 | 0.162 | 0.171 | 0.143 | 0.759 | 0.533
Table 3. Study on Short-term Interests Extractor. Dataset | Clothing,shoes&jewelry | Fliggy
---|---|---
Model | MRR | NDCG | MAP | AUC | Logloss
HIFN∗ | 0.033 | 0.032 | 0.023 | 0.739 | 0.581
HIFN∗+QDIE | 0.037 | 0.038 | 0.028 | 0.746 | 0.553
HIFN∗+TDIE | 0.036 | 0.037 | 0.026 | 0.745 | 0.558
HIFN∗+CDIE | 0.033 | 0.032 | 0.024 | 0.744 | 0.561
HIFN w/o CDIE | 0.039 | 0.040 | 0.032 | 0.753 | 0.541
HIFN w/o TDIE | 0.036 | 0.039 | 0.031 | 0.751 | 0.549
HIFN w/o QDIE | 0.037 | 0.038 | 0.028 | 0.749 | 0.552
HIFN w/o gate | 0.034 | 0.035 | 0.024 | 0.743 | 0.563
HIFN | 0.041 | 0.044 | 0.035 | 0.759 | 0.533

###  3.6. Study on Short-term Interests Extractor

In this part, we will discuss how each module of short-term interests
extractor impact the effectiveness of HIFN. The base model is HIFN∗, which
using a mean pooling operator (No Attention) to aggregate the output of GRU
layer. In particular, we consider the following settings: 1) add QDIE, 2) add
TIDE, 3) add CDIE, 4) add QDIE and TIDE, i.e.,without CDIE, 5) add QDIE and
CDIE, i.e., without TIDE, 6) add TIDE and CDIE, i.e, without QDIE. 7)
replacing the fusing gate with mean pooling operator.

####  3.6.1. Ablation Study of each module

In this section, we investigate the impact of Short-term Interests Encoder. As
illustrated in Table 3 , base model performs the worst, showing that useful
signals could be easily buried in noise without distilling. In addition,
either QDIE, TDIE or CDIE can improve the AUC compared with the base model and
the former has the most gain, while the latter has the least gain,
demonstrating that capturing query-, target- and causal-dependent user
interest all can bring gains and query-dependent user interest helps the most
in prediction accuracy, which is also consistent with the characteristics of
the product search. (Note that, adding CDIE module just have a weak
improvement in Clothing but strong relatively in Fliggy. This is because
Amazon only contains user purchase bahavior, while FLiggy contaions purchase
and click, and CDIE is proposed to extract information from user’s multiple
historical hehavior types.) 4), 5) and 6) all achieve better performance than
previous method, implying that the information extracted by these three
interest extraction modules is complementary. In contrast, HIFN learns user
short-term interest from three different perspective and fusing the output
vector of them with a well-designed gate network, obtaining the highest gains.
It shows that capturing query-, target- and causal-dependent user interest can
improve the accuracy of prediction.

Table 4. Study on SIE with different query types. Query Type | Abstract | Precise
---|---|---
Model | AUC | AUC
HIFN | 0.748 | 0.776
HIFN w/o QDIE | 0.745 | 0.759
HIFN w/o TDIE | 0.739 | 0.772
HIFN w/o CDIE | 0.742 | 0.768

####  3.6.2. Study on datasets of different query types

In this section, we investigate the impact of different interest extractors in
datasets of different query types. We first divide the Fliggy test set into
two sets according to the type of query, one is the abstract search query, and
the other is the precise (i.e. non-abstract) search query. We define the
abstract search query as 1) destination,e.g., Hangzhou, Shanghai, Miami; 2)
tag, e.g., playing, surfing. Different from precise search, users do not
express obvious demand mentality when issuing the abstract query. As
illustrated in Table 4, there are several points worth noting. On the one
hand, HIFN w/o QDIE performs worst than the other two in precise query
dataset, it implies that when user has clear requirements , extracting user
interest with respect to query can obtain more gains. On the other hand, HIFN
w/o TDIE performs worst than the other two in abstract query dataset, implying
that when user has no clear intent, taget-dependent interest plays the most
important role, and extracting user interest with respect to target item is an
effective way to mining the intent from the multi-interest of users. It
further validates that jointly modeling query-dependent, target-dependent,
causal-dependent user interest can achieve significant performance.

Table 5. Study on Long-term Interests Extractor. Dataset | Clothing,shoes&jewelry | Fliggy
---|---|---
Model | MRR | NDCG | MAP | AUC | Logloss
HIFN† | 0.035 | 0.036 | 0.029 | 0.745 | 0.554
HIFN†+Target Attention | 0.036 | 0.037 | 0.03 | 0.746 | 0.553
HIFN†+Query Attention | 0.039 | 0.042 | 0.032 | 0.754 | 0.558
HIFN†+User Attention | / | / | / | 0.752 | 0.564
HIFN | 0.041 | 0.044 | 0.035 | 0.759 | 0.533

###  3.7. Study on Long-term Interests Extractor

In this section, we investigate the impact of the Long-term Interests
Extractor in HIFN, the base model is HIFN†, which extracts user behaviors
using mean pooling operator (No Attention). In particular, we consider the
following settings: 1) Target Attention: taking the target item as the query
and the behavior item as the key in the attention mechanism; 2) Query
Attention: the same long-term attention mechanism with the SOTA long and
short-term methods in product search, ALSTP; 3) User Attention: take the user
field feature as the query and the behavior item as the key in the attention
mechanism.

As can be seen from table 5, each attention mechanism can beat base model,
implying that directly meaning pooling user long-term behaviors could not
fully attend and exploit the useful information. The query attention mechanism
used in ALSTP and user attention mechanism both get better performance than
target attention, we suppose that Long-term interests of users are stable and
change slowly, while user field and query field features represent user’s
intrinsic characteristic, which is consistent with the user’s long-term
interest to some extent. (Since there is no user profile feature in Amazon
dataset, we do experiments about user attention only on Fliggy dataset). HIFN,
leveraging short-term interests to activate long-term interests, achieves the
best performance, we suspect that this is because short-term interests are a
unified vector fusing user profile, query and recent behaviors, which can
extract users’ long-term interests more properly.

Table 6. Study on Interests Fusing Module. Dataset | Clothing,shoes&jewelry | Fliggy
---|---|---
Model | MRR | NDCG | MAP | AUC | Logloss
concatenate | 0.035 | 0.036 | 0.029 | 0.750 | 0.539
$\alpha$=0.9 | 0.037 | 0.036 | 0.028 | 0.747 | 0.544
$\alpha$=0.75 | 0.036 | 0.034 | 0.025 | 0.749 | 0.541
$\alpha$=0.6 | 0.034 | 0.033 | 0.022 | 0.751 | 0.537
$\alpha$=0.45 | 0.035 | 0.034 | 0.024 | 0.753 | 0.535
$\alpha$=0.3 | 0.036 | 0.035 | 0.026 | 0.747 | 0.543
$\alpha$=0.15 | 0.032 | 0.031 | 0.021 | 0.748 | 0.545
HIFN | 0.041 | 0.044 | 0.035 | 0.759 | 0.533

###  3.8. Study on Disentanglement of Long and Short-Term Interests

In order to prove the effectiveness of the disentanglement framework, we
designed two kinds of experiments. Firstly, we will verify whether the
framework can disentangle short-term and long-term interests well. Secondly,
we will verify the impact of different proxy settings on the results.

####  3.8.1. Counterfactual Evaluation

If the importance of different factors change over time, learning
disentangling representations of underlying factors is beneficial to final
accuracy (Schölkopf et al., 2021). In e-commerce product search, users will
have various types of behaviors, such as purchase, favorite, click, the user
preferences revealed by each type of behavior range from strong to weak, i.e
purchase expresses strong preferences, while click expresses relatively weak
preferences. Grbovic (Grbovic and Cheng, 2018) acknowledged that behaviors of
higher costs tend to be more driven by users’ long-term interests, and
behaviors of lower costs such as click indicate more about short-term
interests. As we known, purchase (cost of money) have higher costs than
favorite (cost of double clicks), while favorite have higher costs than click
(only one click). Therefore, a set of experiments were designed to verify the
effectiveness of the disentanglement. Because Amazon dataset only contains
user purchase behavior, we verify the effect on Fliggy dataset. We firstly
train a model with exposure-click dataset, and predicting on both clicks,
favorites and purchases test dataset of Fliggy. Intuitively, it is expected
that the weight for short-term interests is smaller in predicting purchase
than predicting click when fusing the two aspects. We compare our proposed
model HIFN with HIFN without self-supervised framework, and the SOTA long and
short-term methods for product search, ALSTP.

Table 7 illustrates the AUC, Logloss and the average $\alpha$ for clicked
items, favorited items and purchased items. We have the following findings.
First, HIFN w/o IDM achieves a better performance than ALSTP, demonstrating
that HIFN extracts and fuses long and short-term interests in a more proper
and effective way than ALSTP. Second, although predicting purchase/favorite
with model trained on click data is challenging, HIFN also gets a good results
and beats other competitors in favorite test set and purchase test set. Third,
the average $\alpha$ of HIFN in click is larger than favorite, and favorite is
larger than purchase, which is consistent with the previous argument, that
behaviors of higher costs tend to be more driven by users’ long-term interest,
and behaviors of lower costs such as click indicate more about short-term
interest. However the average $\alpha$ of HIFN w/o IDM in favorite is samller
than purchase and the average $\alpha$ of ALSTP in click is samller than
purchase/favorite. It demonstrates that HIFN successfully disentangle the long
and short-term interest to a certain degree, while HIFN w/o IDM and ALSTP
entangles two aspects with each other.

Table 7. Study on Interest Disentanglement Module. Dataset | Click | Favorite | Purchase
---|---|---|---
Model | AUC | Logloss |  AVG ( $\alpha$ ) | AUC | Logloss |  AVG ( $\alpha$ ) | AUC | Logloss |  AVG ( $\alpha$ )
HIFN | 0.759 | 0.533 | 0.783 | 0.746 | 0.569 | 0.721 | 0.741 | 0.573 | 0.699
HIFN w/o IDM | 0.753 | 0.549 | 0.637 | 0.741 | 0.578 | 0.526 | 0.737 | 0.586 | 0.672
ALSTP | 0.742 | 0.576 | 0.433 | 0.733 | 0.591 | 0.621 | 0.729 | 0.598 | 0.596

####  3.8.2. The effectiveness of long-term interests proxy updating

Although Zheng (Zheng et al., 2022) proposes a simple strategy for proxy
setting to disentangling user interests, we argue that this strategy ignores
the nature that long-term interests will evolve slowly, so we propose an
updating strategy for long-term interests proxy to address the issue. In this
section, we study the effectiveness of the long-term interests proxy updating
strategy on the four datasets with respect to different metrics. As shown in
table 8, the performance with the long-term user preference updating strategy
surpasses the one without updating. In detail, updating strategy improves NDCG
by about 0.003, 0.002, and 0.003 in three Amazon datasets respectively, and
improves AUC by about 0.007 in Fliggy dataset. This is in accordance with our
assumption that the long-term user preference should update gradually.

Table 8. Study on Different Proxy Settings. Dataset | Clothing | Electronics | Toys&Games | Fliggy
---|---|---|---|---
Strategy | MRR | NDCG | MAP | MRR | NDCG | MAP | MRR | NDCG | MAP | AUC | Logloss
without updating | 0.038 | 0.041 | 0.032 | 0.329 | 0.340 | 0.347 | 0.159 | 0.168 | 0.137 | 0.752 | 0.542
with updating | 0.041 | 0.044 | 0.035 | 0.331 | 0.342 | 0.351 | 0.162 | 0.171 | 0.143 | 0.759 | 0.533

###  3.9. Study on Interests Fusing Module

Althogh ALSTP achieves the significant performance, we argue that
concatenating long and short-term interests will lead to inferior performance.
In HIFN, we propose to aggregate them adaptively. We compare HIFN with two
different settings: 1) concatenation, 2) using a fixed $\alpha$ when combining
the two aspects. The result is illustrated in Table 6 , we can discover that
HIFN with IFM outperforms all different settings. These results verify the
necessity of adaptive fusion of long and short-term interests.

###  3.10. Online A/B Test

We also deploy HIFN on our platform for A/B test, where baseline model is
TEM+TDIE (Previously, TEM+TDIE had demonstrated online that its results were
superior to other competitors in Section 3.5). The proposed method achieves
average $\boldsymbol{5.13\%}$ overall CTR gains over the baseline model in
successive fifteen days, which is consistent with the offline evaluation
results. Now, HIFN has been deployed online and serves the main traffic.

##  4\. Related Work

###  4.1. Product Search

Aware of the importance of personalization in product search, Ai et al. (Ai et
al., 2017) proposed a hierarchical embedding model where they use a convex
combination of the query and user vector to predict purchased items. Guo et
al. (Guo et al., 2019) represent long and short-term user preferences with an
attention mechanism applied to users’ recent purchases and their global
vectors. Recently, from the analysis of commercial search logs, Ai et al. (Ai
et al., 2019) observed that personalization does not always have a positive
effect. They further proposed a zero-attention model (ZAM) that can control
the influence of personalization. However, the maximal effect personalization
can have is equal to the query. Bi et al. (Bi et al., 2020) found this
limitation and proposed a transformer model to encode the query and
historically purchased items where personalization can have none to full
effect. These models construct user profiles by capturing dynamic user
interests with respect to query. However, they ignore the target-dependent
interest and causal-dependent interest, which will bring additional
information for more accurate prediction.

###  4.2. CTR Prediction

Existing CTR prediction works mainly focus on single scenario modeling from
the following aspects: 1) feature interaction (e.g., FM (Rendle, 2010), DeepFM
(Guo et al., 2017)); 2) user historical behavior (e.g., DIN(Zhou et al.,
2018), DIEN (Zhou et al., 2019)); Factorization Machine (FM) is proposed to
model feature interactions explicitly, while previous generalized linear
models such as Logistic Regression (LR) (Richardson et al., 2007) and Follow-
The-Regularized-Leader (FTRL) (McMahan et al., 2013) lack the ability to solve
interaction issue. Wide&Deep (Cheng et al., 2016) and DeepFM (Guo et al.,
2017) combine low-order and high-order features to improve the performance.
FmFM (Sun et al., 2021) makes each field feature have different embedding
dimensions, so as to reduce the amount of model parameters and avoid over
fitting problem. DIN (Zhou et al., 2018) utilizes the attention mechanism to
capture relative interests from the user behavior sequence with regard to the
candidate item. DIEN (Zhou et al., 2019) further uses a GRU structure to
capture the evolution of user interest. Considering a single vector might be
insufficient to capture complicated user patterns, DMIN (Xiao et al., 2020)
models user’s multiple interests by a special designed extractor layer.

###  4.3. Long and Short-term Interests Modeling

Long and short-term interests modeling has been widely studied in
recommendation, several methods (An et al., 2019; Zhao et al., 2018; Hu et
al., 2020) were proposed to explicitly differentiate between long and short-
term interests. For example, Zhao (Zhao et al., 2018) use matrix factorization
for long-term interests and use RNN for short-term interests. Yu (Yu et al.,
2019) develop a variant of LSTM for short-term interests and adopt asymmetric
SVD for long-term interests. However, in product search, modeling long and
short-interests become difficult because of the query user issued. ALSPT,
which is the sota long and short-term interests modeling method in product
search, represents long and short-term preferences with attention mechanisms
applied to users’ recent behaviors and their global vectors, achieves a
significant performance, but it still contains weakness about interests
modeling and fusing.

##  5\. Conclusion

In this paper, we introduce a novel CTR model named HIFN for product search,
which consists of four elaborately designed modules including SIE, LIE, IFM
and IDM . SIE aims to extract short-term interests from different aspects. LIE
is proposed to extract long-term interests with respect to the output of SIE.
Then IFM fuses the long and short-term interests with an adaptive manner. IDM
is used to disentangle the two aspects of interests. These modules are
integrated together in an end-to-end multi-task learning framework. Extensive
offline and online experiments on real-world e-commerce platforms demonstrate
the superiority of HIFN over SOTA methods.

## References

  * (1)
  * Ai et al. (2019) Qingyao Ai, Daniel N Hill, SVN Vishwanathan, and W Bruce Croft. 2019.  A zero attention model for personalized product search. In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_. 379–388.
  * Ai et al. (2017) Qingyao Ai, Yongfeng Zhang, Keping Bi, Xu Chen, and W Bruce Croft. 2017.  Learning a hierarchical embedding model for personalized product search. In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 645–654.
  * An et al. (2019) Mingxiao An, Fangzhao Wu, Chuhan Wu, Kun Zhang, Zheng Liu, and Xing Xie. 2019\.  Neural news recommendation with long-and short-term user representations. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_. 336–345.
  * Bennett et al. (2012) Paul N Bennett, Ryen W White, Wei Chu, Susan T Dumais, Peter Bailey, Fedor Borisyuk, and Xiaoyuan Cui. 2012.  Modeling the impact of short-and long-term behavior on search personalization. In _Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval_. 185–194.
  * Bi et al. (2020) Keping Bi, Qingyao Ai, and W Bruce Croft. 2020.  A transformer-based embedding model for personalized product search. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1521–1524.
  * Cheng et al. (2016) Heng-Tze Cheng, Levent Koc, Jeremiah Harmsen, Tal Shaked, Tushar Chandra, Hrishi Aradhye, Glen Anderson, Greg Corrado, Wei Chai, Mustafa Ispir, et al. 2016\.  Wide & deep learning for recommender systems. In _Proceedings of the 1st workshop on deep learning for recommender systems_. 7–10.
  * Cho et al. (2014) Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014.  On the properties of neural machine translation: Encoder-decoder approaches.  _arXiv preprint arXiv:1409.1259_ (2014).
  * Grbovic and Cheng (2018) Mihajlo Grbovic and Haibin Cheng. 2018.  Real-time personalization using embeddings for search ranking at airbnb. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 311–320.
  * Guo et al. (2017) Huifeng Guo, Ruiming Tang, Yunming Ye, Zhenguo Li, and Xiuqiang He. 2017.  DeepFM: a factorization-machine based neural network for CTR prediction. In _Proceedings of the 26th International Joint Conference on Artificial Intelligence_. 1725–1731.
  * Guo et al. (2019) Yangyang Guo, Zhiyong Cheng, Liqiang Nie, Yinglong Wang, Jun Ma, and Mohan Kankanhalli. 2019\.  Attentive long short-term preference modeling for personalized product search.  _ACM Transactions on Information Systems (TOIS)_ 37, 2 (2019), 1–27.
  * He and McAuley (2016) Ruining He and Julian McAuley. 2016.  Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In _proceedings of the 25th international conference on world wide web_. 507–517.
  * Hidasi et al. (2015) Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk. 2015.  Session-based recommendations with recurrent neural networks.  _arXiv preprint arXiv:1511.06939_ (2015).
  * Hu et al. (2020) Linmei Hu, Chen Li, Chuan Shi, Cheng Yang, and Chao Shao. 2020.  Graph neural news recommendation with long-term and short-term interest modeling.  _Information Processing & Management_ 57, 2 (2020), 102142\.
  * Li et al. (2021) Sen Li, Fuyu Lv, Taiwei Jin, Guli Lin, Keping Yang, Xiaoyi Zeng, Xiao-Ming Wu, and Qianli Ma. 2021\.  Embedding-based Product Retrieval in Taobao Search. In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_. 3181–3189.
  * Li et al. (2020) Zhao Li, Xin Shen, Yuhang Jiao, Xuming Pan, Pengcheng Zou, Xianling Meng, Chengwei Yao, and Jiajun Bu. 2020\.  Hierarchical bipartite graph neural networks: Towards large-scale e-commerce applications. In _2020 IEEE 36th International Conference on Data Engineering (ICDE)_. IEEE, 1677–1688.
  * Liu et al. (2021) Yixin Liu, Zhao Li, Shirui Pan, Chen Gong, Chuan Zhou, and George Karypis. 2021\.  Anomaly detection on attributed networks via contrastive self-supervised learning.  _IEEE transactions on neural networks and learning systems_ 33, 6 (2021), 2378–2392.
  * McMahan et al. (2013) H Brendan McMahan, Gary Holt, David Sculley, Michael Young, Dietmar Ebner, Julian Grady, Lan Nie, Todd Phillips, Eugene Davydov, Daniel Golovin, et al. 2013\.  Ad click prediction: a view from the trenches. In _Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining_. 1222–1230.
  * Rendle (2010) Steffen Rendle. 2010\.  Factorization machines. In _2010 IEEE International Conference on Data Mining_. IEEE, 995–1000.
  * Richardson et al. (2007) Matthew Richardson, Ewa Dominowska, and Robert Ragno. 2007\.  Predicting clicks: estimating the click-through rate for new ads. In _Proceedings of the 16th international conference on World Wide Web_. 521–530.
  * Schölkopf et al. (2021) Bernhard Schölkopf, Francesco Locatello, Stefan Bauer, Nan Rosemary Ke, Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. 2021\.  Toward causal representation learning.  _Proc. IEEE_ 109, 5 (2021), 612–634.
  * Shen et al. (2021) Qijie Shen, Wanjie Tao, Jing Zhang, Hong Wen, Zulong Chen, and Quan Lu. 2021\.  SAR-Net: A Scenario-Aware Ranking Network for Personalized Fair Recommendation in Hundreds of Travel Scenarios. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_. 4094–4103.
  * Shen et al. (2022) Qijie Shen, Hong Wen, Wanjie Tao, Jing Zhang, Fuyu Lv, Zulong Chen, and Zhao Li. 2022.  Deep Interest Highlight Network for Click-Through Rate Prediction in Trigger-Induced Recommendation. In _Proceedings of the ACM Web Conference 2022_. 422–430.
  * Sun et al. (2021) Yang Sun, Junwei Pan, Alex Zhang, and Aaron Flores. 2021\.  FM^2: Field-matrixed Factorization Machines for Recommender Systems.  _arXiv preprint arXiv:2102.12994_ (2021).
  * Tao et al. (2022a) Wanjie Tao, Liangyue Li, Chen Chen, Zulong Chen, and Hong Wen. 2022a.  When Online Meets Offline: Exploring Periodicity for Travel Destination Prediction. In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 2366–2370.
  * Tao et al. (2022b) Wanjie Tao, Yu Li, Liangyue Li, Zulong Chen, Hong Wen, Peilin Chen, Tingting Liang, and Quan Lu. 2022b.  SMINet: State-Aware Multi-Aspect Interests Representation Network for Cold-Start Users Recommendation. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 36. 8476–8484.
  * Xiang et al. (2010) Liang Xiang, Quan Yuan, Shiwan Zhao, Li Chen, Xiatian Zhang, Qing Yang, and Jimeng Sun. 2010.  Temporal recommendation on graphs via long-and short-term preference fusion. In _Proceedings of the 16th ACM SIGKDD international conference on Knowledge discovery and data mining_. 723–732.
  * Xiao et al. (2020) Zhibo Xiao, Luwei Yang, Wen Jiang, Yi Wei, Yi Hu, and Hao Wang. 2020.  Deep Multi-Interest Network for Click-through Rate Prediction. In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_. 2265–2268.
  * Xu et al. (2022) Jia Xu, Jin Huang, Zulong Chen, Yang Li, Wanjie Tao, and Chuanfei Xu. 2022\.  ODNET: A Novel Personalized Origin-Destination Ranking Network for Flight Recommendation. In _2022 IEEE 38th International Conference on Data Engineering (ICDE)_. IEEE, 3481–3493.
  * Yu et al. (2019) Zeping Yu, Jianxun Lian, Ahmad Mahmoody, Gongshen Liu, and Xing Xie. 2019.  Adaptive User Modeling with Long and Short-Term Preferences for Personalized Recommendation.. In _IJCAI_. 4213–4219.
  * Zhang and Tao (2020) Jing Zhang and Dacheng Tao. 2020.  Empowering things with intelligence: a survey of the progress, challenges, and opportunities in artificial intelligence of things.  _IEEE Internet of Things Journal_ 8, 10 (2020), 7789–7817.
  * Zhao et al. (2018) Wei Zhao, Benyou Wang, Jianbo Ye, Yongqiang Gao, Min Yang, and Xiaojun Chen. 2018\.  PLASTIC: Prioritize Long and Short-term Information in Top-n Recommendation using Adversarial Training.. In _Ijcai_. 3676–3682.
  * Zheng et al. (2022) Yu Zheng, Chen Gao, Jianxin Chang, Yanan Niu, Yang Song, Depeng Jin, and Yong Li. 2022.  Disentangling Long and Short-Term Interests for Recommendation. In _Proceedings of the ACM Web Conference 2022_. 2256–2267.
  * Zhou et al. (2019) Guorui Zhou, Na Mou, Ying Fan, Qi Pi, Weijie Bian, Chang Zhou, Xiaoqiang Zhu, and Kun Gai. 2019.  Deep interest evolution network for click-through rate prediction. In _Proceedings of the AAAI Conference on Artificial Intelligence_.
  * Zhou et al. (2018) Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai. 2018\.  Deep interest network for click-through rate prediction. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. ACM, 1059–1068.


# Using Cyclic Noise as the Source Signal for Neural Source-Filter-based
Speech Waveform Model

###### Abstract

Neural source-filter (NSF) waveform models generate speech waveforms by
morphing sine-based source signals through dilated convolution in the time
domain. Although the sine-based source signals help the NSF models to produce
voiced sounds with specified pitch, the sine shape may constrain the generated
waveform when the target voiced sounds are less periodic. In this paper, we
propose a more flexible source signal called cyclic noise, a quasi-periodic
noise sequence given by the convolution of a pulse train and a static random
noise with a trainable decaying rate that controls the signal shape. We
further propose a masked spectral loss to guide the NSF models to produce
periodic voiced sounds from the cyclic noise-based source signal. Results from
a large-scale listening test demonstrated the effectiveness of the cyclic
noise and the masked spectral loss on speaker-independent NSF models in copy-
synthesis experiments on the CMU ARCTIC database.

Index Terms: speech synthesis, source-filter model, harmonic-plus-noise
waveform model, neural network

##  1 Introduction

Neural waveform models are neural network (NN)-based generative models that
produce waveforms given acoustic features, linguistic features, other types of
input data. In addition to the pioneering work using feedforward networks for
speech enhancement [1, 2], many new types of neural waveform models have been
proposed recently for text-to-speech (TTS) synthesis [3, 4, 5], band extension
[6], music signal generation [7, 8], and other tasks. While new models such as
WaveNet [4] and WaveGlow [5] are based on the latest deep learning methods,
many other models combine NNs with classical signal processing techniques such
as adaptive cepstral analysis [3], linear prediction coding (LPC) [9], and
glottal waveform modeling [10]. These models can efficiently produce high-
quality waveforms by leveraging signal processing algorithms that cannot be
easily learned by neural networks [11].

For TTS, we have proposed a family of neural source-filter (NSF) waveform
models [12, 13, 14] that combines the dilated convolution (CONV) NN with the
idea of the source-filter speech production model. The NSF models share three
common components: a condition module that pre-processes the input spectral
features, a source module that produces sine-based source signals from input
F0, and a dilated-CONV-based filter module that converts the source signals
into output waveforms. Compared with other neural waveform models, NSF models
are easy to train and work fast for waveform generation, and our previous
study showed their good performance in a Japanese TTS system with a female
voice [13].

The source signal is essential to NSF models as these NSF models produce
output waveforms by gradually morphing the source signal in the time domain.
In our previous works on TTS, we used sine-based source signals because their
periodicity can be accurately maintained in the generated voiced sounds [13].
In addition to the speech waveforms, the sine-based waveform also helps the
NSF models to produce high-quality music signals for woodwind, string, and
brass instruments [15].

However, for speech waveform generation, the sine-based source signal may
cause artifacts when the target sounds are less periodic, e.g., when they are
creaky or breathy [16]. To relax the constraint from using a specific waveform
shape while maintaining the periodicity, we propose using a cyclic noise as
the source signal. We compute this cyclic noise by convolving a pulse train of
the specified F0 and a random noise with a tunable decaying rate, which keeps
the quasi periodicity of the source signal and allows cyclostationary
randomness across periods. We also propose a masked spectral loss that
encourages the NSF models to produce periodic voiced sounds from the quasi-
periodic cyclic noise. Experimental results demonstrated that the proposed
cyclic noise outperformed the sine-based source signal on male speaker data
while performing equally well on female speaker data.

After we briefly review the latest NSF model in Section 2, we describe the
proposed source signal and loss in Section 3 and explain the experiment
results in Section 4. We conclude in Section 5 with a brief summary. Although
we focus on NSF models in this paper, we hope that our findings can be useful
to related works that use random noises [5, 17], LPC residuals [9], or other
periodic signals [18, 19] as the source signals .

##  2 Review of hn-sinc-NSF model

Our focus in this paper is the latest harmonic-plus-noise NSF model with sinc
filters (hn-sinc-NSF) [14]. In the hn-sinc-NSF model, which is shown in Figure
1, the input frame-level F0s and Mel-spectrogram are transformed and up-
sampled into feature sequences that have the same length $T$ as the target
waveform $\bm{o}_{1:T}=\\{o_{1},\cdots,o_{T}\\}$. The up-sampled F0
$\bm{f}_{1:T}=\\{f_{1},\cdots,f_{T}\\}$ is fed to the source module to produce
a source signal $\bm{e}_{1:T}$, and the processed acoustic feature
$\tilde{\bm{c}}_{1:T}$ is fed to neural filter blocks that consist of dilated
CONV layers and skip connections. The neural filter blocks form two branches:
a harmonic branch that transforms $\bm{e}_{1:T}$ into
$\widehat{\bm{o}}_{1:T}^{(p)}$ and a noise branch that converts a noise
sequence $\bm{a}_{1:T}$ into $\widehat{\bm{o}}_{1:T}^{(a)}$. The two signals
$\widehat{\bm{o}}_{1:T}^{(p)}$ and $\widehat{\bm{o}}_{1:T}^{(a)}$ are then
filtered by low-pass and high-pass sinc filters, respectively, and summed as
the output waveform $\widehat{\bm{o}}_{1:T}$. The time-variant cut-off
frequency of the sinc filters, or maximum voiced frequency (MVF) [20], is
predicted from the input acoustic features. The model is trained by minimizing
spectral amplitude losses between $\widehat{\bm{o}}_{1:T}$ and
${\bm{o}}_{1:T}$.

Figure 1: Hn-sinc-NSF during training. LP and HP denote low- and high-pass
digital filters, respectively. Up-sampled feature sequence is smoothed except
$\bm{f}_{1:T}$. Compared with [14], $\tilde{\bm{c}}_{1:T}$ and MVF are
produced from the same Bi-LSTM and CONV layers to simplify condition module.
Shaded components are those proposed in this paper.

The source signal $\bm{e}_{1:T}=\\{e_{1},\cdots,e_{T}\\}$ in the harmonic
branch is based on sine waveforms and can be generated in two steps. First,
with the up-sampled F0 $\bm{f}_{1:T}$, a sine waveform $\bm{e}_{1:T}^{<h>}$
that carries the F0 or the $h$-th harmonic is generated by

$\displaystyle{e}_{t}^{<h>}=\begin{dcases}\alpha\sin(\sum_{k=1}^{t}2\pi\frac{h{f}_{k}}{N_{s}}+{\phi})+{n}_{t},&\text{if
}{f_{t}}>0\\\ \frac{\alpha}{3\sigma}{n}_{t},&\text{if }f_{t}=0\\\
\end{dcases},$ (1)

where $f_{t}=0$ denotes being unvoiced at the $t$-th time step,
$\phi\in[-\pi,\pi]$ is a random initial phase, $N_{s}$ is the waveform
sampling rate, and ${n}_{t}\sim\mathcal{N}(0,\sigma^{2})$ is a Gaussian noise.
Then, for each $t\in\\{1,\cdots,T\\}$, ${e}_{t}$ is computed as

${{e}_{t}}=\text{tanh}(\sum_{h=1}^{H}w_{h}{{e}}_{t}^{<h>}+w_{b}),$ (2)

where $H$ is the number of harmonics and $\\{w_{1},\cdots w_{H},w_{b}\\}$ are
the weights of a trainable feedforward layer with a tanh activation function.
In our implementation, we used $H=8$, $\sigma=0.003$, and $\alpha=0.1$ as the
hyper-parameters [13]. Note that $n_{t}$ in Eq.(1) is independent of the noise
source signal ${a}_{t}$ for the noise branch of the hn-sinc-NSF, even though
both of them can be sampled from the same Gaussian distribution.

##  3 Proposed methods

Figure 2: Example of different source signals. Maximum signal amplitude is
scaled to 1 for visualization.

###  3.1 Cyclic noise-based source signal

Although a sine-based source is useful for generating periodic waveforms [13],
the sine shape may cause perceptual artifacts. For example, it may be
inappropriate for creaky voices with irregular glottal pulses [16, 21] and
low-pitch voices where the shape of the source signal has a greater perceptual
effect [22, 23]. We hypothesize that a better NSF source signal for voiced
sounds may contain a certain degree of randomness in the short term while
preserving long-term periodicity. Although source signals for classical speech
vocoders [24, 25, 26, 27] may be used, we focus on source signals that have a
simple parametric form and require no additional analysis loop.

Inspired by an experimental study on speech perception [23], we propose a
source signal called cyclic noise, which has a simple parametric form and can
preserve both periodicity and randomness. Let
$\bm{p}_{1:T}=[p_{1},\cdots,p_{T}]$ denote a unit impulse train corresponding
to the up-sampled F0 $\bm{f}_{1:T}$, and let $\bm{n}_{1:T}$ denote a Gaussian
noise sequence. The proposed source signal $e_{t}$ at the $t$-th time step is
computed in two steps:

$\displaystyle\tilde{e}_{t}=\begin{dcases}\sum_{k=0}^{t-1}n_{k+1}\exp(-\frac{kf_{t}}{\beta_{t}N_{s}})p_{t-k},&\text{if
}{f_{t}}>0\\\ {n}_{t},&\text{if }f_{t}=0\\\ \end{dcases},$ (3)

and

${{e}_{t}}=\text{tanh}(w_{1}{\tilde{e}}_{t}+w_{b}).$ (4)

Equation (4) is similar to Eq. (2) but uses only $\tilde{e}_{t}$ rather than
multiple signals with higher harmonics. While $\tilde{e}_{t}$ in the unvoiced
regions is still random noise, $\tilde{e}_{t}$ in the voiced regions is the
convolution of $\bm{p}_{1:T}$ and an exponentially decaying noise whose value
is $n_{k+1}\exp(-\frac{kf_{t}}{\beta_{t}N_{s}}),\forall{k}\in[0,t-1]$. The
factor $\frac{f_{t}}{N_{s}}$ makes the decaying rate independent of the period
length or F0 — no matter how long the interval between two pulses is, the
noise value is scaled by $\exp(-\frac{1}{\beta_{t}})$ when $k$ is equal to the
period length $\frac{N_{s}}{f_{t}}$. In such a manner, $\beta_{t}$ becomes the
main parameter that controls the decaying rate.

By adjusting $\beta_{t}$, we can change the randomness and periodicity of the
cyclic noise-based source signal for voiced sounds. Figure 2 illustrates a few
source signals with different values of $\beta_{t}$ for $t\in[1,T]$. Note that
when $\beta_{t}=0.435$, the noise amplitude at $t=\frac{N_{s}}{f_{t}}$ is
scaled by $\exp(-\frac{1}{0.435})\approx{0.1}$. When $\beta_{t}$ is smaller,
the random noise decays more quickly, and the source signal is more similar to
the pulse train. When $\beta_{t}$ is increased, the source signal becomes
noisy and less periodic.

We produce $\bm{p}_{1:T}$ by setting $p_{t}=1$, where $t$ corresponds to the
local maxima of the sine waveform that carries the F0. Otherwise, $p_{t}=0$.
The noise sequence $\bm{n}_{1:T}$ is drawn from the same Gaussian distribution
as that in Eq. (1), and it changes for every input training sample.

###  3.2 Predicting $\beta_{t}$ from acoustic features

We may treat the decaying rate $\beta$ as a hyper-parameter and decide its
value through trial and error. Alternatively, we can use a small neural
network to predict the value of $\beta_{t}$ from the input acoustic features
for each time step $t$.

Our implementation is illustrated in Figure 1. We use the Bi-LSTM and CONV
layers in the condition module to predict a one-dimensional signal at the
frame level. This signal is then up-sampled and smoothed as
$\bm{\beta}_{1:T}=\\{\beta_{1},\cdots,\beta_{T}\\}$. The smoothing is
conducted twice by moving averaging over a window of size 320. During
training, an $L_{1}$ norm $|\beta_{t}-0.870|_{1}$ with weight 0.01 is added to
prevent $\beta_{t}$ from going wild. The value 0.870 is used because
$\beta_{t}=0.870$ keeps periodicity and randomness, as shown in Figure 2.

###  3.3 Additional masked spectral loss to stabilize pitch

In our experiments, we observed that the randomness in the cyclic noise made
it more difficult to produce a waveform with stable pitch. Since the source
signal is gradually morphed by the NSF neural filter blocks, it may be helpful
if additional loss can be defined to penalize the mismatch of the harmonic
structure between the natural waveform and the output of the filter blocks.

We propose a masked spectral loss for this purpose. Let
${\bm{y}}^{(n)}=[{y}^{(n)}_{1},\cdots,{y}^{(n)}_{K}]^{\top}\in\mathbb{C}^{K}$
denote the complex-valued spectrum of the $n$-th frame of the natural waveform
calculated using $K$-points FFT. Similarly, let $\widehat{\bm{p}}^{(n)}$ and
$\bm{m}^{(n)}$ denote the spectrum calculated on a neural filter block’s
output signal and a mask signal, respectively. The proposed loss is defined as

$\mathcal{L}_{\text{mask}}=\frac{1}{2NK}\sum_{n=1}^{N}\sum_{k=1}^{K}\Big{[}\log\frac{|y_{k}^{(n)}|^{2}|m_{k}^{(n)}|^{2}+\eta}{|\widehat{p}_{k}^{(n)}|^{2}|m_{k}^{(n)}|^{2}+\eta}\Big{]}^{2},$
(5)

where $N$ is the number of frames and $\eta=1\mathrm{e}{-5}$ is a constant to
ensure numerical stability. The mask signal is defined as
$\frac{1}{H}\sum_{h=1}^{H}{\bm{e}}_{1:T}^{<h>}$, where ${\bm{e}}_{1:T}^{<h>}$
is the sine waveform that carries the $i$-th harmonics of the input F0 (see
Eq. (1)). The spectrum amplitude of such a mask signal, i.e.,
$|\bm{m}^{(n)}|^{2}$, has peaks of roughly equal height at the harmonic
frequencies, and $\mathcal{L}_{\text{mask}}$ thus penalizes the mismatch of
the harmonic structure rather than the whole spectral envelope, as Figure 3
illustrates.

The masked spectral loss $\mathcal{L}_{\text{mask}}$ can be computed using
multiple STFT configurations in the same manner as the unmasked spectral
amplitude loss $\mathcal{L}$ between $\bm{o}_{1:T}$ and $\widehat{o}_{1:T}$.
We used the STFT configurations in our previous work (Table III in [13]) for
both $\mathcal{L}$ and $\mathcal{L}_{\text{mask}}$. Further, as Figure 1
plots, $\mathcal{L}_{\text{mask}}$ is measured between the natural waveform
and the outputs of the five neural filter blocks (i.e., block 1 to block 5 in
Figure 1).

##  4 Experiments

###  4.1 Data and feature configuration

We used the data of four US English speakers from the CMU ARCTIC database [28]
for our experiments: SLT (female), CLB (female), BDL (male), and RMS (male).
Each speaker provided 1000, 64, and 64 utterances for the training,
validation, and test sets, respectively, and the data were pooled together to
train the experimental models in a speaker-independent manner.

The acoustic features input to the experimental models included the Mel-
spectrogram of 80 dimensions and the F0 sequence, both of which were extracted
with a frame shift of 5 ms. The target speech waveforms had a sampling rate of
16 kHz and were equalized to -26 dBov using SV56 [29]. During the test stage,
the natural acoustic features were used to produce the waveforms. Speaker
vectors were not used.

Figure 3: Sine-masked spectral loss. $\bm{y}^{(n)}$, $\widehat{\bm{p}}^{(n)}$,
and $\bm{m}^{(n)}$ denote the spectrum of natural waveform, output signal from
a neural filter block, and sine mask, respectively, at $n$-th frame.

###  4.2 Experimental models

We utilized the hn-sinc-NSF models listed in Table 1. Except the source signal
and $\mathcal{L}_{\text{mask}}$, all these models used the network
architecture detailed in our previous work (i.e., sinc1-h-NSF in [14]): five
and one neural filter blocks in the harmonic and noise branches, respectively,
ten dilated-CONV layers in each block, and sinc filters of order $31$. The
$k$-th dilated-CONV layer in each neural filter block had a dilation size of
$2^{k-1}$. The experiments included the natural waveforms (Natural) and
WaveNet (WaveNet) as a reference. WaveNet used 30 dilated CONV layers, where
the $k$-layer had a dilation size of $2^{\text{mod}(k-1,10)}$. It used the
Mel-spectrogram as the input features and generated 10-bit $\mu$-law quantized
waveform values.

All the experimental models were trained for 100 epochs using an ADAM
optimizer with a learning rate $0.0003$, $\beta_{1}=0.9$, $\beta_{2}=0.999$,
and $\epsilon=10^{-8}$ [30]. The model parameters from the epoch with the
lowest validation error were saved.

###  4.3 Results and analysis

Each trained experimental model generated 264 test set waveforms (64
utterances per speaker). The quality of the generated waveforms was evaluated
by means of a large-scale crowd-sourced listening test. All the waveforms were
converted to a 16-bit PCM format for the test111Code, models, and samples:
https://nii-yamagishilab.github.io/samples-nsf/nsf-v4.html..

Each test round, an evaluator was asked to complete 22 test pages. The
evaluator listened to one waveform on each page and rated its quality on a
1-to-5 mean opinion score (MOS) scale. The 22 test pages contained two
randomly selected natural waveforms and two generated waveforms from each of
the experimental models from the same speaker. The page order was randomly
shuffled, and the waveform on each page could be played multiple times. Around
710 evaluators participated in the listening test and 792 test rounds were
conducted. Accordingly, each experimental model received 1584 MOS scores, the
boxplots of which are shown in Figure 5.

The results reveal interesting cross-gender differences. As the scores of Sin,
Pul, and Rno demonstrate, the sine and pulse were better than the random noise
for female speakers while the random noise outperformed the sine and pulse for
male speakers. The differences in both cases were statistically significant
(Mann-Whitney two-sided test, $p<0.01$)222Results of significant tests are
uploaded to the website above.. While cross-gender differences have also been
observed on the source signals for classical speech vocoders [23, 31, 32], new
findings can be observed here. For male voices, for example, Rno produced
periodic voiced sounds, and they were perceptually better than those using
sine or pulse. The performance of Rno suggests that the neural filter blocks
can transform random noise into periodic waveforms with long periodic cycles,
which is difficult for linear filters in classical vocoders. For the female
voice, we found that it was still difficult for neural filter blocks to
produce periodic waveforms with a short period cycle, so it is better to use
periodic source signals.

The cyclic noise worked well for both the female and male voices when a good
$\beta$ was selected. Both Cno${}_{\beta_{2}}$ and Cno${}_{\beta_{3}}$
performed quite well: they outperformed Sin on the male voices and were as
good as Sin on the female voices. For female speakers, the cyclic noises in
Cno${}_{\beta_{2}}$ and Cno${}_{\beta_{3}}$ were quasi-periodic (as Figure 2
illustrates) and may be sufficient for generating periodic waveforms. For male
speakers, Cno${}_{\beta_{2}}$ and Cno${}_{\beta_{3}}$ were perceptually better
than Sin because the generated waveforms in each periodic cycle had one
obvious spike in the time domain, which is similar to how natural waveforms
behave (as Figure 4 shows). In contrast, the waveform from Sin scattered
within each periodic cycle in the time domain. Cno${}_{\beta_{1}}$ performed
worse on male voices and its generated speech was perceptually similar to
those from Sin.

Interestingly, compared with the LPC analysis on low pitch voices (e.g., M4 in
[23]), the waveforms from Cno${}_{\beta_{2}}$ and Cno${}_{\beta_{3}}$ were
perceptually similar to those of using a zero-phase source signal in LPC,
while the waveforms from Sin sounded similar to the LPC-produced waveforms
using cyclostationary random phase source signals, even though the source
signals in Cno${}_{\beta_{2}}$ and Cno${}_{\beta_{3}}$ were cyclostationary
rather than zero-phased. The discrepancy from the LPC analysis again may be
due to the difference between NNs and linear filters in LPC.

Although we used Cno${}_{\beta_{\text{tr}}}$ to infer time-variant and
speaker-dependent $\beta_{t}$, the predicted $\beta_{t}$ was roughly equal to
$\beta=0.86$ for all $t$ and all speakers. This may be due to the difficulty
of optimization. Another reason could be that a better spectral loss is
necessary to evaluate the difference caused by the varying $\beta$.

Table 1: Experimental models based on hn-sinc-NSF. Model | Type of source signal | $\mathcal{L}_{mask}$
---|---|---
Sin |  Sine-based waveform (Eq. (1-2)) | –
Pul | Pulse with noise (Gaussian) | –
Rno | Random noise (Gaussian) | +
Cno${}_{\beta_{1}}$ |  Cyclic noise, $\beta_{t}=0.435$ | +
Cno${}_{\beta_{2}}$ |  Cyclic noise, $\beta_{t}=0.870$ | +
Cno${}_{\beta_{3}}$ |  Cyclic noise, $\beta_{t}=1.739$ | +
Cno${}_{\beta_{\text{tr}}}$ |  Cyclic noise, trainable $\beta_{t}$ | +
Rno${}_{\text{no}\mathcal{L}_{\text{mask}}}$ | Random Gaussian noise | –
Cno${}_{\beta_{2}\text{no}\mathcal{L}_{\text{mask}}}$ |  Cyclic noise, $\beta_{t}=0.870$ | –
Figure 4: Waveform sample from BDL (arctic_b0477).

As a final remark, Cno${}_{\beta_{2}}$ outperformed
Cno${}_{\beta_{2}\text{no}\mathcal{L}_{\text{mask}}}$ for all four speakers,
and the difference was statistically significant. This result indicates the
usefulness of the masked spectral loss $\mathcal{L}_{\text{mask}}$ for
Cno${}_{\beta_{\text{2}}}$. However, $\mathcal{L}_{\text{mask}}$ was less
effective for Rno, especially on female speakers. We conclude that
$\mathcal{L}_{\text{mask}}$ is insufficient to guide the model to reproduce
stable pitch from noise source signals for female speakers.

(a) Results of all speakers pooled together

(b) Results on SLT (female speaker)

(c) Results on CLB (female speaker)

(d) Results on RMS (male speaker)

(e) Results on BDL (male speaker)

Figure 5: Boxplots of MOS. Dot denotes mean value of MOS.

##  5 Conclusion

We proposed a new type of source signal called cyclic noise for NSF models.
Cyclic noise has a simple parametric form based on the convolution of a pulse
train and an exponentially decaying noise sequence. We demonstrated through
experiments that an appropriate $\beta$ can be specified to maintain the
periodicity and introduce randomness to the cyclic noise signal. Compared with
the sine-based source signal, which worked well for female speakers but not
for male speakers, the proposed cyclic noise worked well for both female and
male voices.

Although the findings in this paper were derived from NSF models, they showed
interesting differences from those on the classical vocoders. Future work will
examine the link between the perceptual difference and the phase [23, 32, 33].

Acknowledgement: This work was partially supported by a JST CREST Grant
(JPMJCR18A6, VoicePersonae project), Japan, MEXT KAKENHI Grants (19K24371,
16H06302, 17H04687, 18H04120, 18H04112, 18KT0051), Japan, and Google AI for
Japan program. The experiments partially were conducted on TSUBAME 3.0 of
Tokyo Institute of Technology. We thank Mr. Yusuke Yasuda for setting up the
server for listening test. We also thank Dr. Heiga Zen and Dr. Yotaro Kubo for
comments and discussions.

## References

  * [1] S. Tamura and A. Waibel, “Noise reduction using connectionist models,” in _Proc. ICASSP_ , 1988, pp. 553–554.
  * [2] X.-M. Gao, S. J. Ovaska, and Z. O. Hartimo, “Speech signal restoration using an optimal neural network structure,” in _Proceedings of International Conference on Neural Networks (ICNN’96)_ , vol. 4. IEEE, 1996, pp. 1841–1846.
  * [3] K. Tokuda and H. Zen, “Directly modeling speech waveforms by neural networks for statistical parametric speech synthesis,” in _Proc. ICASSP_. IEEE, 2015, pp. 4215–4219.
  * [4] A. van den Oord, S. Dieleman, H. Zen, K. Simonyan, O. Vinyals, A. Graves, N. Kalchbrenner, A. Senior, and K. Kavukcuoglu, “WaveNet: A generative model for raw audio,” _arXiv preprint arXiv:1609.03499_ , 2016.
  * [5] R. Prenger, R. Valle, and B. Catanzaro, “WaveGlow: A Flow-based Generative Network for Speech Synthesis,” in _Proc. ICASSP_ , 2019, pp. 3617–3621.
  * [6] V. Kuleshov, S. Z. Enam, and S. Ermon, “Audio super-resolution using neural nets,” in _ICLR (Workshop Track)_ , 2017.
  * [7] J. Engel, C. Resnick, A. Roberts, S. Dieleman, M. Norouzi, D. Eck, and K. Simonyan, “Neural audio synthesis of musical notes with WaveNet autoencoders,” in _Proc. ICML_ , 2017, pp. 1068–1077. [Online]. Available: http://proceedings.mlr.press/v70/engel17a.html
  * [8] J. Engel, K. K. Agrawal, S. Chen, I. Gulrajani, C. Donahue, and A. Roberts, “GANSynth: Adversarial neural audio synthesis,” _arXiv preprint arXiv:1902.08710_ , 2019.
  * [9] J.-M. Valin and J. Skoglund, “LPCNet: Improving Neural Speech Synthesis Through Linear Prediction,” in _Proc. ICASSP_ , 2019, pp. 5891–5895.
  * [10] L. Juvela, B. Bollepalli, V. Tsiaras, and P. Alku, “Glotnet—a raw waveform model for the glottal excitation in statistical parametric speech synthesis,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 27, no. 6, pp. 1019–1030, 2019.
  * [11] J. Engel, L. Hantrakul, C. Gu, and A. Roberts, “DDSP: Differentiable Digital Signal Processing,” _Proc. ICLR_ , 2020.
  * [12] X. Wang, S. Takaki, and J. Yamagishi, “Neural source-filter-based waveform model for statistical parametric speech synthesis,” in _Proc. ICASSP_ , 2019, pp. 5916–5920.
  * [13] ——, “Neural Source-Filter Waveform Models for Statistical Parametric Speech Synthesis,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 28, pp. 402–415, 2020.
  * [14] X. Wang and J. Yamagishi, “Neural Harmonic-plus-Noise Waveform Model with Trainable Maximum Voice Frequency for Text-to-Speech Synthesis,” in _Proc. SSW_ , 2019, pp. 1–6.
  * [15] Y. Zhao, X. Wang, L. Juvela, and J. Yamagishi, “Transferring neural speech waveform synthesizers to musical instrument sounds generation,” _Proc. ICASSP_ , 2020.
  * [16] M. Gordon and P. Ladefoged, “Phonation types: a cross-linguistic overview,” _Journal of phonetics_ , vol. 29, no. 4, pp. 383–406, 2001.
  * [17] L. Juvela, B. Bollepalli, J. Yamagishi, and P. Alku, “GELP: GAN-excited linear prediction for speech synthesis from Mel-spectrogram,” _Proc. Interspeech_ , pp. 694–699, 2019.
  * [18] O. Watts, C. Valentini-Botinhao, and S. King, “Speech waveform reconstruction using convolutional neural networks with noise and periodic inputs,” in _Proc. ICASSP_. IEEE, 2019, pp. 7045–7049.
  * [19] Y. Ai and Z.-H. Ling, “A neural vocoder with hierarchical generation of amplitude and phase spectra for statistical parametric speech synthesis,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 28, pp. 839–851, 2020.
  * [20] Y. Stylianou, “Applying the harmonic plus noise model in concatenative speech synthesis,” _IEEE Transactions on speech and audio processing_ , vol. 9, no. 1, pp. 21–29, 2001.
  * [21] T. Drugman, J. Kane, and C. Gobl, “Data-driven detection and analysis of the patterns of creaky voice,” _Computer Speech & Language_, vol. 28, no. 5, pp. 1233–1253, 2014.
  * [22] A. E. Rosenberg, “Effect of glottal pulse shape on the quality of natural vowels,” _The Journal of the Acoustical Society of America_ , vol. 49, no. 2B, pp. 583–590, 1971.
  * [23] T. Raitio, L. Juvela, A. Suni, M. Vainio, and P. Alku, “Phase perception of the glottal excitation and its relevance in statistical parametric speech synthesis,” _Speech Communication_ , vol. 81, pp. 104–119, 2016.
  * [24] J. P. Cabral, S. Renals, K. Richmond, and J. Yamagishi, “Towards an improved modeling of the glottal source in statistical parametric speech synthesis,” _Proc. SSW6_ , pp. 113–118, 2007.
  * [25] R. Maia, T. Toda, H. Zen, Y. Nankaku, and K. Tokuda, “An excitation model for HMM-based speech synthesis based on residual modeling,” in _Proc. SSW6_ , 2007, pp. 131–136.
  * [26] T. Drugman and T. Dutoit, “The deterministic plus stochastic model of the residual signal and its applications,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 20, no. 3, pp. 968–981, 2011.
  * [27] T. Raitio, A. Suni, J. Yamagishi, H. Pulakka, J. Nurminen, M. Vainio, and P. Alku, “HMM-based speech synthesis utilizing glottal inverse filtering,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 19, no. 1, pp. 153–165, 2011.
  * [28] J. Kominek and A. W. Black, “The CMU Arctic speech databases,” in _Fifth ISCA workshop on speech synthesis_ , 2004.
  * [29] R. ITU-T, “P. 56: Objective Measurement of Active Speech Level,” _International Telecommunication Union_ , 2011. [Online]. Available: https://www.itu.int/rec/T-REC-P.56-201112-I/en
  * [30] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in _Proc. ICLR_ , 2014.
  * [31] T. Drugman and T. Raitio, “Excitation modeling for HMM-based speech synthesis: breaking down the impact of periodic and aperiodic components,” in _Proc. ICASSP_ , 2014, pp. 260–264.
  * [32] J. Skoglund, W. B. Kleijn, and P. Hedelin, “Audibility of pitch-synchronously modulated noise,” in _Proc. IEEE Workshop on Speech Coding for Telecommunications Proceedings. Back to Basics: Attacking Fundamental Problems in Speech Coding_. IEEE, 1997, pp. 51–52.
  * [33] H. Pobloth and W. B. Kleijn, “On phase perception in speech,” in _Proc. ICASSP_ , vol. 1. IEEE, 1999, pp. 29–32.

##  Appendix A Results from Mann-Whitney U test

In the tables below, $\square$ denotes significant difference ($p<0.01$), and
$\bullet$ denotes insignificant difference.

Table 2: Results over all speakers | Nat | WaN | Sin | Pul | Rno |  Cno${}_{\beta_{1}}$ |  Cno${}_{\beta_{2}}$ |  Cno${}_{\beta_{3}}$ |  Cno${}_{\beta_{\text{tr}}}$ |  RNo${}_{\text{no}\mathcal{L}_{\text{mask}}}$ |  Cno${}_{\beta_{\text{tr}},\text{no}\mathcal{L}_{\text{mask}}}$
---|---|---|---|---|---|---|---|---|---|---|---
Nat | - |  |  |  |  |  |  |  |  |  |
WaN | $\bullet$ | - |  |  |  |  |  |  |  |  |
Sin | $\bullet$ | $\square$ | - |  |  |  |  |  |  |  |
Pul | $\bullet$ | $\bullet$ | $\bullet$ | - |  |  |  |  |  |  |
Rno | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | - |  |  |  |  |  |
Cno${}_{\beta_{1}}$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | - |  |  |  |  |
Cno${}_{\beta_{2}}$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | - |  |  |  |
Cno${}_{\beta_{3}}$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\square$ | - |  |  |
Cno${}_{\beta_{\text{tr}}}$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\square$ | $\square$ | - |  |
RNo${}_{\text{no}\mathcal{L}_{\text{mask}}}$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\square$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | - |
Cno${}_{\beta_{\text{tr}},\text{no}\mathcal{L}_{\text{mask}}}$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | -
Table 3: Results on SLT | Nat | WaN | Sin | Pul | Rno |  Cno${}_{\beta_{1}}$ |  Cno${}_{\beta_{2}}$ |  Cno${}_{\beta_{3}}$ |  Cno${}_{\beta_{\text{tr}}}$ |  RNo${}_{\text{no}\mathcal{L}_{\text{mask}}}$ |  Cno${}_{\beta_{\text{tr}},\text{no}\mathcal{L}_{\text{mask}}}$
---|---|---|---|---|---|---|---|---|---|---|---
Nat | - |  |  |  |  |  |  |  |  |  |
WaN | $\bullet$ | - |  |  |  |  |  |  |  |  |
Sin | $\bullet$ | $\bullet$ | - |  |  |  |  |  |  |  |
Pul | $\bullet$ | $\square$ | $\bullet$ | - |  |  |  |  |  |  |
Rno | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | - |  |  |  |  |  |
Cno${}_{\beta_{1}}$ | $\bullet$ | $\square$ | $\square$ | $\bullet$ | $\bullet$ | - |  |  |  |  |
Cno${}_{\beta_{2}}$ | $\bullet$ | $\square$ | $\square$ | $\square$ | $\bullet$ | $\square$ | - |  |  |  |
Cno${}_{\beta_{3}}$ | $\bullet$ | $\square$ | $\bullet$ | $\square$ | $\bullet$ | $\bullet$ | $\bullet$ | - |  |  |
Cno${}_{\beta_{\text{tr}}}$ | $\bullet$ | $\square$ | $\square$ | $\square$ | $\bullet$ | $\square$ | $\square$ | $\bullet$ | - |  |
RNo${}_{\text{no}\mathcal{L}_{\text{mask}}}$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\square$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | - |
Cno${}_{\beta_{\text{tr}},\text{no}\mathcal{L}_{\text{mask}}}$ | $\bullet$ | $\bullet$ | $\bullet$ | $\square$ | $\bullet$ | $\bullet$ | $\bullet$ | $\square$ | $\bullet$ | $\bullet$ | -
Table 4: Results on CLB | Nat | WaN | Sin | Pul | Rno |  Cno${}_{\beta_{1}}$ |  Cno${}_{\beta_{2}}$ |  Cno${}_{\beta_{3}}$ |  Cno${}_{\beta_{\text{tr}}}$ |  RNo${}_{\text{no}\mathcal{L}_{\text{mask}}}$ |  Cno${}_{\beta_{\text{tr}},\text{no}\mathcal{L}_{\text{mask}}}$
---|---|---|---|---|---|---|---|---|---|---|---
Nat | - |  |  |  |  |  |  |  |  |  |
WaN | $\bullet$ | - |  |  |  |  |  |  |  |  |
Sin | $\bullet$ | $\bullet$ | - |  |  |  |  |  |  |  |
Pul | $\bullet$ | $\square$ | $\bullet$ | - |  |  |  |  |  |  |
Rno | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | - |  |  |  |  |  |
Cno${}_{\beta_{1}}$ | $\bullet$ | $\bullet$ | $\square$ | $\bullet$ | $\bullet$ | - |  |  |  |  |
Cno${}_{\beta_{2}}$ | $\bullet$ | $\bullet$ | $\square$ | $\bullet$ | $\bullet$ | $\square$ | - |  |  |  |
Cno${}_{\beta_{3}}$ | $\bullet$ | $\bullet$ | $\square$ | $\bullet$ | $\bullet$ | $\square$ | $\square$ | - |  |  |
Cno${}_{\beta_{\text{tr}}}$ | $\bullet$ | $\bullet$ | $\square$ | $\bullet$ | $\bullet$ | $\square$ | $\square$ | $\square$ | - |  |
RNo${}_{\text{no}\mathcal{L}_{\text{mask}}}$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\square$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | - |
Cno${}_{\beta_{\text{tr}},\text{no}\mathcal{L}_{\text{mask}}}$ | $\bullet$ | $\square$ | $\bullet$ | $\square$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | -
Table 5: Results on RMS | Nat | WaN | Sin | Pul | Rno |  Cno${}_{\beta_{1}}$ |  Cno${}_{\beta_{2}}$ |  Cno${}_{\beta_{3}}$ |  Cno${}_{\beta_{\text{tr}}}$ |  RNo${}_{\text{no}\mathcal{L}_{\text{mask}}}$ |  Cno${}_{\beta_{\text{tr}},\text{no}\mathcal{L}_{\text{mask}}}$
---|---|---|---|---|---|---|---|---|---|---|---
Nat | - |  |  |  |  |  |  |  |  |  |
WaN | $\bullet$ | - |  |  |  |  |  |  |  |  |
Sin | $\bullet$ | $\bullet$ | - |  |  |  |  |  |  |  |
Pul | $\bullet$ | $\bullet$ | $\bullet$ | - |  |  |  |  |  |  |
Rno | $\bullet$ | $\square$ | $\bullet$ | $\bullet$ | - |  |  |  |  |  |
Cno${}_{\beta_{1}}$ | $\bullet$ | $\bullet$ | $\bullet$ | $\square$ | $\bullet$ | - |  |  |  |  |
Cno${}_{\beta_{2}}$ | $\bullet$ | $\square$ | $\bullet$ | $\bullet$ | $\square$ | $\bullet$ | - |  |  |  |
Cno${}_{\beta_{3}}$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\square$ | $\bullet$ | $\square$ | - |  |  |
Cno${}_{\beta_{\text{tr}}}$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\square$ | $\bullet$ | $\square$ | $\square$ | - |  |
RNo${}_{\text{no}\mathcal{L}_{\text{mask}}}$ | $\bullet$ | $\square$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | - |
Cno${}_{\beta_{\text{tr}},\text{no}\mathcal{L}_{\text{mask}}}$ | $\bullet$ | $\bullet$ | $\bullet$ | $\square$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | -
Table 6: Results on BDL | Nat | WaN | Sin | Pul | Rno |  Cno${}_{\beta_{1}}$ |  Cno${}_{\beta_{2}}$ |  Cno${}_{\beta_{3}}$ |  Cno${}_{\beta_{\text{tr}}}$ |  RNo${}_{\text{no}\mathcal{L}_{\text{mask}}}$ |  Cno${}_{\beta_{\text{tr}},\text{no}\mathcal{L}_{\text{mask}}}$
---|---|---|---|---|---|---|---|---|---|---|---
Nat | - |  |  |  |  |  |  |  |  |  |
WaN | $\bullet$ | - |  |  |  |  |  |  |  |  |
Sin | $\bullet$ | $\bullet$ | - |  |  |  |  |  |  |  |
Pul | $\bullet$ | $\bullet$ | $\bullet$ | - |  |  |  |  |  |  |
Rno | $\bullet$ | $\square$ | $\bullet$ | $\bullet$ | - |  |  |  |  |  |
Cno${}_{\beta_{1}}$ | $\bullet$ | $\bullet$ | $\bullet$ | $\square$ | $\bullet$ | - |  |  |  |  |
Cno${}_{\beta_{2}}$ | $\bullet$ | $\square$ | $\square$ | $\bullet$ | $\square$ | $\bullet$ | - |  |  |  |
Cno${}_{\beta_{3}}$ | $\bullet$ | $\square$ | $\bullet$ | $\bullet$ | $\square$ | $\bullet$ | $\square$ | - |  |  |
Cno${}_{\beta_{\text{tr}}}$ | $\bullet$ | $\square$ | $\square$ | $\bullet$ | $\square$ | $\bullet$ | $\square$ | $\square$ | - |  |
RNo${}_{\text{no}\mathcal{L}_{\text{mask}}}$ | $\bullet$ | $\bullet$ | $\square$ | $\bullet$ | $\bullet$ | $\bullet$ | $\square$ | $\bullet$ | $\bullet$ | - |
Cno${}_{\beta_{\text{tr}},\text{no}\mathcal{L}_{\text{mask}}}$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | $\bullet$ | -