# Knowledge Distillation based Contextual Relevance Matching
for E-commerce Product Search

Ziyang Liu§,Chaokun Wang§*, Hao Feng§, Lingfei Wu†, Liqun Yang‡
§Tsinghua University,†JD.com,‡CNAEIT
§liu-zy21<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Online relevance matching is an essential task of e-commerce product search to
boost the utility of search engines and ensure a smooth user experience.
Previous work adopts either classical relevance matching models or
Transformer-style models to address it. However, they ignore the inherent
bipartite graph structures that are ubiquitous in e-commerce product search
logs and are too inefficient to deploy online. In this paper, we design an
efficient knowledge distillation framework for e-commerce relevance matching
to integrate the respective advantages of Transformer-style models and
classical relevance matching models. Especially for the core student model of
the framework, we propose a novel method using $k$-order relevance modeling.
The experimental results on large-scale real-world data (the size is
6$\sim$174 million) show that the proposed method significantly improves the
prediction accuracy in terms of human relevance judgment. We deploy our method
to the anonymous online search platform. The A/B testing results show that our
method significantly improves 5.7% of UV-value under price sort mode.

**footnotetext: Chaokun Wang is the corresponding author.

##  1 Introduction

Relevance matching Guo et al. (2016); Rao et al. (2019); Wang et al. (2020) is
an important task in the field of ad-hoc information retrieval Zhai and
Lafferty (2017), which aims to return a sequence of information resources
related to a user query Huang et al. (2020); Chang et al. (2021a); Sun and Duh
(2020). Generally, texts are the dominant form of the user query and returned
information resources. Given two sentences, the target of relevance matching
is to estimate their relevance score and then judge whether they are relevant
or not. However, text similarity does not mean semantic similarity. For
example, while “mac pro 1.7GHz” and “mac lipstick 1.7ml” look alike, they
describe two different and irrelevant products. Therefore, relevance matching
is important, especially for long-term user satisfaction of e-commerce search
Cheng et al. (2022); Niu et al. (2020); Xu et al. (2021); Zhu et al. (2020).

Related work. With the rapid development of deep learning, the current
research on relevance matching can be grouped into two camps (see Appendix A
for further details): 1.Classical Relevance Matching Models. For the given
query and item, the classical relevance matching models either learn their
individual embeddings or learn an overall embedding based on the calculation
from word-level interaction to sentence-level interaction. The representative
methods include ESIM Chen et al. (2017) and BERT2DNN Jiang et al. (2020).
2.Transformer-style Models. These models adopt the multi-layer Transformer
network structure Vaswani et al. (2017a). They have achieved breakthroughs on
many NLP tasks and even reached human-level accuracy. The representative
methods include BERT Devlin et al. (2019) and ERNIE Sun et al. (2019b).

Although Transformer-style models show satisfactory performance on relevance
matching, they are hard to deploy to the online environment due to their high
time complexity. Moreover, the above methods cannot deal with the abundant
context information (i.e., the neighbor features in a query-item bipartite
graph) in e-commerce product search. Last but not least, when applied to real-
world scenarios, existing classical relevance matching models directly use
user behaviors as labeling information (Fig. 1). However, this solution is not
directly suitable for relevance matching because user behaviors are often
noisy and deviate from relevance signals Mao et al. (2019); Liu and Mao
(2020).

Figure 1: Shortcoming of the existing relevance matching model. Here we take
the ARC-I model as an example. The right part shows the ground truth of
queries and item titles. The left part shows two problematic examples in
ARC-I, which deviate from the ground truth.

In this paper, we propose to incorporate bipartite graph embedding into the
knowledge distillation framework Li et al. (2021a); Dong et al. (2021); Rashid
et al. (2021); Wu et al. (2021b); Zhang et al. (2020) to solve the relevance
matching problem in the scene of e-commerce product search. We adopt BERT
Devlin et al. (2019) as the teacher model in this framework. We design a novel
model called BERM, Bipartite graph Embedding for Relevance Matching (BERM),
which acts as the student model in our knowledge distillation framework. This
model captures the 0-order relevance using a word interaction matrix attached
with positional encoding and captures the higher-order relevance using the
metapath embedding with graph attention scores. For online deployment, it is
further distilled into a tiny model BERM-O.

Our main contributions are as follows:

  * •

We formalize the $k$-order relevance problem in a bipartite graph (Section
2.1) and address it by a knowledge distillation framework with a novel student
model called BERM.

  * •

We apply BERM to the e-commerce product search scene with abundant context
information (Section 2.3) and evaluate its performance (Section 3). The
results indicate that BERM outperforms the state-of-the-art methods.

  * •

To facilitate online applications, we further distill BERM into a faster
model, i.e., BERM-O. The results of online A/B testing indicate that BERM-O
significantly improves 5.7% (relative value) of UV-value under price sort
mode.

##  2 Method

###  2.1 Problem Definition

Figure 2: The e-commerce knowledge distillation framework proposed in our
work. Three models are used in this framework: teacher model BERT, student
model BERM, and online model BERM-O.

We first give the definition of the bipartite graph:

###### Definition 1

Bipartite Graph. Given a graph
$\mathcal{G}=(\mathcal{U},\mathcal{V},\mathcal{E},$
$\mathcal{A},\mathcal{R})$, it contains two disjoint node sets
$\mathcal{U}:\left\\{u_{1},u_{2},\cdots,u_{n}\right\\}$ and
$\mathcal{V}:\left\\{v_{1},v_{2},\cdots,v_{n^{\prime}}\right\\}$. For edge set
$\mathcal{E}:\left\\{e_{1},\cdots,e_{m}\right\\}$, each edge $e_{i}$ connects
$u_{j}$ in $\mathcal{U}$ and $v_{k}$ in $\mathcal{V}$. In addition, there is a
node type mapping function
$f_{1}:\mathcal{U}\cup\mathcal{V}\rightarrow\mathcal{A}$ and an edge type
mapping function $f_{2}:\mathcal{E}\rightarrow\mathcal{R}$. Such a graph
$\mathcal{G}$ is called a bipartite graph.

###### Example 1

Given an e-commerce search log, we can build a query-item bipartite graph as
shown in the left part of Fig. 1, where
$\mathcal{A}=\left\\{Query,Item\right\\}$ and
$\mathcal{R}=\left\\{Click\right\\}$.

In a bipartite graph, we use the metapath and metapath instance to incorporate
the neighboring node information into relevance matching. They are defined as
follows:

###### Definition 2

Metapath and Metapath Instance in Bipartite Graph. Given a bipartite graph
$\mathcal{G}=(\mathcal{U},\mathcal{V},\mathcal{E},\mathcal{A},\mathcal{R})$, a
metapath
$P_{i}=a_{1}\overset{r_{1}}{\rightarrow}a_{2}\overset{r_{2}}{\rightarrow}\cdots\overset{r_{l}}{\rightarrow}a_{l+1}$
($a_{j}\neq a_{j+1},1\leqslant j\leqslant l$) represents the path from $a_{1}$
to $a_{l+1}$ successively through $r_{1},r_{2},\cdots,r_{l}$
($a_{j}\in\mathcal{A}$, $r_{j}\in\mathcal{R}$). The length of $P_{i}$ is
denoted as $\left|P_{i}\right|$ and $\left|P_{i}\right|=l$. For brevity, the
set of all metapaths on $\mathcal{G}$ can be represented in regular expression
as
$P^{\mathcal{G}}=\left(a|\varepsilon\right)\left(a^{\prime}a\right)^{+}\left(a^{\prime}|\varepsilon\right)$
where $a,a^{\prime}\in\mathcal{A}$, $a\neq a^{\prime}$. The metapath instance
$p$ is a definite node sequence instantiated from metapath $P_{i}$. All
instances of $P_{i}$ is denoted as $I(P_{i})$, then $p\in I(P_{i})$.

###### Example 2

As shown in Fig. 1, an instance of metapath “$Query$-$Item$-$Query$” is
“$q_{2}$-$i_{3}$-$q_{3}$”.

###### Definition 3

$k$-order Relevance. Given a bipartite graph
$\mathcal{G}=(\mathcal{U},\mathcal{V},\mathcal{E},\mathcal{A},\mathcal{R})$, a
function $F^{k}_{rel}:\mathcal{U}\times\mathcal{V}\rightarrow\left[0,1\right]$
is called a $k$-order relevance function on $\mathcal{G}$ if
$F^{k}_{rel}(u_{i},v_{j})=G(\Phi(u_{i}),\Phi(v_{j})|C_{k})$, where
$\Phi(\cdot)$ is a function to map each node to a representation vector,
$G(.)$ is the score function, $u_{i}\in\mathcal{U}$, $v_{j}\in\mathcal{V}$,
and context information $C_{k}=\bigcup_{I_{P_{i}}\subseteq I(P_{i}),P_{i}\in
P^{\mathcal{G}},\left|P_{i}\right|=k}I_{P_{i}}$.

Many existing relevance matching models Huang et al. (2013); Shen et al.
(2014); Hu et al. (2014b) ignore context information $C_{k}$ and only consider
the sentences w.r.t. the query and item to be matched, which corresponds to
0-order relevance (for more details, please see the “Related Work” part in
Appendix A). We call it _context-free relevance matching_ in this paper.
Considering that both the 0-order neighbor (i.e., the node itself) and
$k$-order neighbor ($k\geqslant 1$) are necessary for relevance matching, we
argue that a reasonable mechanism should ensure that they can cooperate with
each other. Then the research objective of our work is defined as follows:

###### Definition 4

Contextual Relevance Matching. Given a bipartite graph
$\mathcal{G}=(\mathcal{U},\mathcal{V},\mathcal{E},\mathcal{A},\mathcal{R})$,
the task of contextual relevance matching is to determine the context
information $C_{k}$ on $\mathcal{G}$ and learn the score function
$G(\cdot)\;$.

###  2.2 Overview

We propose a complete knowledge distillation framework (Fig. 2), whose student
model incorporates the context information, for contextual relevance matching
in e-commerce product search. The main components of this framework are
described as follows (see Appendix B for further details):

  * •

Graph construction. We firstly construct a raw bipartite graph $\mathcal{G}$
based on the search data collected from the anonymous platform. Then we
construct a knowledge-enhanced bipartite graph $\mathcal{G}^{\prime}$ with the
help of BERT, which is fine-tuned by the human-labeled relevance data.

  * •

Student model design. We design a novel student model BERM corresponding to
the score function $G(\cdot)$ in Def. 4. Specifically, macro and micro
matching embeddings are derived in BERM to capture the sentence-level and
word-level relevance matching signal, respectively. Also, based on the
metapaths ‘$Q$-$I$-$Q$” and ‘$I$-$Q$-$I$”, we design a node-level encoder and
a metapath-instance-level aggregator to derive metapath embeddings.

  * •

Online application. To serve online search, we conduct further distillation to
BERM and obtain BERM-O, which can be easily deployed to the online search
platform.

###  2.3 BERM Model

Next we describe BERM in detail, including 0-order relevance modeling (Section
2.3.1), $k$-order relevance modeling (Section 2.3.2), and overall learning
objective (Section 2.3.3).

####  2.3.1 0-order Relevance Modeling

The whole structure of BERM includes both the 0-order relevance modeling and
$k$-order relevance modeling. This subsection introduces the 0-order relevance
modeling which captures sentence-level and word-level matching signals by
incorporating the macro matching embedding and micro matching embedding,
respectively.

Word embedding in e-commerce scene. In e-commerce scene, the basic
representations of query or item is an intractable problem. On one hand, it is
infeasible to represent queries and items as individual embeddings due to the
unbounded entity space. On the other hand, product type names (like
“iphone11”) or attribute names (like “256GB”) have special background
information and could contain complex lexicons such as different languages and
numerals. To address these problems, we adopt word embedding in BERM, which
dramatically reduces the representation space. Also, we treat contiguous
numerals, contiguous English letters, or single Chinese characters as one word
and only retain the high-frequency words (such as the words occurring more
than fifty times in a six-month search log) in the vocabulary. The final
vocabulary is only in the tens of thousands, which saves memory consumption
and lookup time of indexes by a large margin. Each word is represented by a
$d$-dimensional embedding vector, which is trained by Word2Vec Mikolov et al.
(2013). The $i$-th word’s embedding of query $Q$ (or item title $I$) is
denoted as  $\bm{E}_{Q}^{i}\in\mathbb{R}^{d}$ (or
$\bm{E}_{I}^{i}\in\mathbb{R}^{d}$ ).

Macro and micro matching embeddings. To capture sentence-level and word-level
matching signals, we employ macro matching embedding and micro matching
embedding, respectively. For the macro matching embedding, taking query $Q$
with $l_{Q}$ words and item $I$ with $l_{I}$ words as examples, their macro
embeddings
${\boldsymbol{E}}_{\mathrm{seq}}^{Q},{\boldsymbol{E}}_{\mathrm{seq}}^{I}\in\mathbb{R}^{d}$
are calculated by the column-wise mean value of
${\boldsymbol{E}}_{Q}\in\mathbb{R}^{l_{Q}\times
d},{\boldsymbol{E}_{I}}\in\mathbb{R}^{l_{I}\times d}$ :

$\bm{E}_{\mathrm{seq}}^{Q}=\frac{1}{l_{Q}}\sum_{i=1}^{l_{Q}}\bm{E}_{Q}^{i},\quad\bm{E}_{\mathrm{seq}}^{I}=\frac{1}{l_{I}}\sum_{i=1}^{l_{I}}\bm{E}_{I}^{i}\;.$
(1)

For the micro matching embedding, we first build an interaction matrix
$\bm{M}_{\mathrm{int}}\in\mathbb{R}^{l_{Q}\times l_{I}}$ whose  $(i,j)$ -th
entry is the dot product of  $\bm{E}_{Q}^{i}$ and $\bm{E}_{I}^{j}$ :

$\bm{M}_{\mathrm{int}}=\left\\{m_{\mathrm{int}}^{i,j}\right\\}_{l_{Q}\times
l_{I}},\quad
m_{\mathrm{int}}^{i,j}=\langle\bm{E}_{Q}^{i},\bm{E}_{I}^{j}\rangle\;.$ (2)

Then the micro matching embedding
$\bm{E}_{\mathrm{int}}\in\mathbb{R}^{l_{Q}l_{I}}$ is the vectorization of
$\bm{M}_{\mathrm{int}}$ , i.e.,
$\bm{E}_{\mathrm{int}}=\mathrm{vec}(\bm{M}_{\mathrm{int}})$ .

####  2.3.2 k-order Relevance Modeling

The $k$-order relevance model contains a node-level encoder and a metapath-
instance-level aggregator.

Node-level encoder. The input of the node-level encoder is node embeddings and
its output is an instance embedding (i.e., the embedding of a metapath
instance). Specifically, to obtain the instance embedding, we integrate the
embeddings of neighboring nodes into the anchor node embedding with a mean
encoder. Taking “$Q$-$I_{\mathrm{{top1}}}$-$Q_{\mathrm{{top1}}}$” (note that
“$I_{\mathrm{{top1}}}$” is the top-1 node in the 1-hop neighbor list of node
$Q$, and see Appendix B.1 for further details) as an example, its embedding
$\bm{E}_{Q-I_{\mathrm{{top1}}}-Q_{\mathrm{{top1}}}}\in\mathbb{R}^{d}$ is
calculated by:

$\bm{E}_{Q-I_{\mathrm{{top1}}}-Q_{\mathrm{{top1}}}}=\mathrm{MEAN}(\bm{E}_{\mathrm{seq}}^{Q},\bm{E}_{\mathrm{seq}}^{I_{\mathrm{top1}}},\bm{E}_{\mathrm{seq}}^{Q_{\mathrm{top1}}}).$
(3)

The metapath instance bridges the communication gap between different types of
nodes and can be used to update the anchor node embedding from structure
information.

Figure 3: Calculation process of $\bm{E}_{Q-I-Q}$ and $\bm{E}_{I-Q-I}$ in
BERM.

Metapath-instance-level aggregator. The inputs of the metapath-instance-level
aggregator are instance embeddings and its output is a metapath embedding.
Different metapath instances convey different information, so they have
various effects on the final metapath embedding. However, the mapping
relationship between the instance embedding and metapath embedding is unknown.
To learn their relationship automatically, we introduce the “graph attention”
mechanism to generate metapath embeddings Wu et al. (2021a); Liu et al.
(2022b). Taking metapath “$Q$-$I$-$Q$” as an example, we use graph attention
to represent the mapping relationship between “$Q$-$I$-$Q$” and its instances.
The final metapath embedding  $\bm{E}_{Q-I-Q}\in\mathbb{R}^{d}$ is obtained (
$\bm{E}_{I-Q-I}\in\mathbb{R}^{d}$ is calculated similarly) by accumulating all
instance embeddings with attention scores
$\mathrm{Att}_{1},\mathrm{Att}_{2},\mathrm{Att}_{3},\mathrm{Att}_{4}\in\mathbb{R}$:

$\displaystyle\bm{E}_{Q-I-Q}$
$\displaystyle=\mathrm{LeakyReLU}(\mathrm{Att}_{1}\cdot\bm{E}_{Q-I_{\mathrm{{top1}}}-Q_{\mathrm{{top1}}}}$
(4)
$\displaystyle+\mathrm{Att}_{2}\cdot\bm{E}_{Q-I_{\mathrm{{top1}}}-Q_{\mathrm{{top2}}}}$
$\displaystyle+\mathrm{Att}_{3}\cdot\bm{E}_{Q-I_{\mathrm{{top2}}}-Q_{\mathrm{{top1}}}}$
$\displaystyle+\mathrm{Att}_{4}\cdot\bm{E}_{Q-I_{\mathrm{{top2}}}-Q_{\mathrm{{top2}}}}).$

Though $\mathrm{Att}_{i}$ can be set as a fixed value, we adopt a more
flexible way, i.e., using the neural network to learn $\mathrm{Att}_{i}$
automatically. Specifically, we feed the concatenation of the anchor node
embedding and metapath instance embedding into a one-layer neural network (its
weight is  $\bm{W}_{\mathrm{att}}\in\mathbb{R}^{6d\times 4}$ and its bias is
$\bm{b}_{\mathrm{att}}\in\mathbb{R}^{1\times 4}$ ) with a softmax layer, which
outputs an attention distribution:

$(\mathrm{Att}_{i})_{1\leq i\leq
4}=\mathrm{softmax}(\bm{E}_{\mathrm{concat}}*\bm{W}_{\mathrm{att}}+\bm{b}_{\mathrm{att}}),$
(5)

$\displaystyle\bm{E}_{\mathrm{concat}}$
$\displaystyle=[\bm{E}_{\mathrm{seq}}^{Q}|\bm{E}_{\mathrm{seq}}^{I}|\bm{E}_{Q-I_{\mathrm{{top1}}}-Q_{\mathrm{{top1}}}}|\bm{E}_{Q-I_{\mathrm{{top1}}}-Q_{\mathrm{{top2}}}}$
(6)
$\displaystyle|\bm{E}_{Q-I_{\mathrm{{top2}}}-Q_{\mathrm{{top1}}}}|\bm{E}_{Q-I_{\mathrm{{top2}}}-Q_{\mathrm{{top2}}}}].$

The above process is shown in Fig. 3.

Embedding fusion. By the 0-order and $k$-order relevance modeling, three types
of embeddings are generated, including macro matching embedding (
$\bm{E}^{Q}_{\mathrm{seq}},\bm{E}^{I}_{\mathrm{seq}}\in\mathbb{R}^{d}$ ),
micro matching embedding (  $\bm{E}_{\mathrm{int}}\in\mathbb{R}^{l_{Q}l_{I}}$
), and metapath embedding (  $\bm{E}_{Q-I-Q},\bm{E}_{I-Q-I}\in\mathbb{R}^{d}$
). We concatenate them together and feed the result to a three-layer neural
network (its weights are  $\bm{W}_{0}\in\mathbb{R}^{(4d+l_{Q}l_{I})\times
d},\bm{W}_{1},\bm{W}_{2}\in\mathbb{R}^{d\times
d},\bm{W}_{3}\in\mathbb{R}^{d\times 1}$ and biases are
$\bm{b}_{\mathrm{0}},\bm{b}_{\mathrm{1}},\bm{b}_{\mathrm{2}}\in\mathbb{R}^{1\times
d},\bm{b}_{\mathrm{3}}\in\mathbb{R}^{1\times 1}$ ), which outputs the final
relevance estimation score $\hat{y}_{i}$:

$\hat{y}_{i}=\mathrm{Sigmoid}(\bm{E}_{3}*\bm{W}_{3}+\bm{b}_{3}),$ (7)
$\bm{E}_{j+1}=\mathrm{ReLU}(\bm{E}_{j}*\bm{W}_{j}+\bm{b}_{j}),\bm{E}_{0}=\bm{E}_{all},j=0,1,2,$
(8)
$\bm{E}_{all}=[\bm{E}^{Q}_{\mathrm{seq}}|\bm{E}^{I}_{\mathrm{seq}}|\bm{E}_{\mathrm{int}}|\bm{E}_{Q-I-Q}|\bm{E}_{I-Q-I}].$
(9)

####  2.3.3 Overall Learning Objective

We evaluate the cross-entropy error on the estimation score $\hat{y}_{i}$ and
label $y_{i}$ (note that $y_{i}\in[0,1]$ is the score of the teacher model
BERT, see Appendix B.1 for further details), and then minimize the following
loss function:

$L=-\sum_{i=1}^{\tilde{n}}y_{i}\log(\hat{y}_{i})+(1-y_{i})\log(1-\hat{y}_{i}).$
(10)

In Appendix C, we analyze the complexities of BERT, BERM, and BERM-O.

##  3 Experiments

In this section, we present the offline and online experimental results of
BERM ***We provide the description of baselines, implementation details, and
additional experiments in Appendix D.1, D.2, E (Code URL:
https://github.com/Young0222/EMNLP-BERM)..

###  3.1 Experimental Setting

Datasets. We collect three datasets from the anonymous platform, including the
“Electronics” category (Data-E), all-category (Data-A), and sampled all-
category (Data-S). In the platform, there are mainly three different levels of
item categories: $Cid_{1}$ (highest level, e.g., “Electronics”), $Cid_{2}$
(e.g., “Mobile phone”), and $Cid_{3}$ (lowest level, e.g., “5G phone”).
Data-A, Data-S, and Data-E have different data distributions. Specifically,
Data-A covers all first-level categories $Cid_{1}$ in the platform, Data-S is
generated by uniformly sampling 5,000 items from $Cid_{1}$, and Data-E only
focuses on “Electronics” in $Cid_{1}$.

For the training data $S_{\mathrm{train}}$ (also called
$S_{\mathrm{transfer}}$), the collected user behaviors include click and
purchase. For the testing data $S_{\mathrm{test}}$, whose queries are
disjointed with those of $S_{\mathrm{train}}$, we use human labeling to
distinguish between relevant and irrelevant items. Specifically, editors are
asked to assess the relevance scores between queries and items. In the
anonymous platform, the candidate set of relevance scores is
$\left\\{1,2,3,4,5\right\\}$, where $5$ means most relevant and $1$ means
least relevant. To simplify it, we use binary labeling including the positive
label (i.e., 4 or 5) and negative label (i.e., 1, 2, or 3). We report the
detailed statistics of Data-E, Data-A, and Data-S in Tab. 1.

Table 1: Statistics of the used datasets. Sets | Names | Data-E | Data-A | Data-S
---|---|---|---|---
$S_{\mathrm{train}}$ | # Example | 6,369,396 | 174,863,375 | 11,397,439
# $\mathrm{Node_{query}}$ | 398,824 | 5,952,020 | 3,284,480
# $\mathrm{Node_{item}}$ | 728,405 | 49,517,217 | 1,307,557
# Edge | 5,070,460 | 159,205,320 | 7,525,355
# Click | 1,471,079,596 | 5,109,731,591 | 1,431,899,847
# Purchase | 33,285,887 | 322,151,488 | 118,495,170
$S_{\mathrm{test}}$ | # Example | 30,563 | 39,743 | 39,743
# $\mathrm{Node_{query}}$ | 3,374 | 3,108 | 3,108
# $\mathrm{Node_{item}}$ | 16,137 | 30,097 | 30,097
# Edge | 18,988 | 30,661 | 30,661

Evaluation Metrics. To measure the performance of baseline methods and our
BERM, we use three kinds of evaluation metrics, including Area Under the
receiver operating characteristic Curve (AUC), F1-score, and False Negative
Rate (FNR). The low value of FNR indicates the low probability of fetching
irrelevant items, which is closely related to the user’s search experience.
Therefore, we include it in the evaluation metrics.

Table 2: Comparisons on Data-E and Data-S. In each column, the best result is bolded and the runner-up is underlined. The symbol of “$\downarrow$” represents that the lower value corresponds to better performance. “I, II, III” represent the representation-focused, interaction-focused, and both-focused relevance matching models, respectively. “IV” represents the graph neural network models. | Models | Data-E | Data-S
---|---|---|---
| AUC | F1-score | FNR($\downarrow$) | AUC | F1-score | FNR($\downarrow$)
I | DSSM | 0.6246 | 0.6923 | 0.9953 | 0.8219 | 0.8691 | 1.0000
MVLSTM | 0.8602 | 0.8055 | 0.3416 | 0.7877 | 0.8857 | 0.7802
ARC-I | 0.8343 | 0.7949 | 0.3857 | 0.6919 | 0.8750 | 0.9388
II | DRMM | 0.6720 | 0.6891 | 0.7692 | 0.6781 | 0.8722 | 0.9401
MatchPyramid | 0.7826 | 0.7481 | 0.5615 | 0.7859 | 0.8786 | 0.8475
ARC-II | 0.8128 | 0.7864 | 0.4377 | 0.7606 | 0.8784 | 0.9076
K-NRM | 0.7462 | 0.7291 | 0.6510 | 0.7314 | 0.8733 | 0.9081
DRMM-TKS | 0.7678 | 0.7383 | 0.5462 | 0.7793 | 0.8789 | 0.7893
Conv-KNRM | 0.8369 | 0.7879 | 0.3469 | 0.8029 | 0.8789 | 0.8913
ESIM | 0.8056 | 0.7769 | 0.3373 | 0.7987 | 0.8623 | 1.0000
III | Duet | 0.7693 | 0.7219 | 0.8173 | 0.7968 | 0.8754 | 0.9458
BERT2DNN | 0.8595 | 0.8037 | 0.3464 | 0.8313 | 0.9061 | 0.4450
IV | GAT | 0.7526 | 0.7361 | 0.7529 | 0.7411 | 0.8746 | 0.9234
GraphSAGE-Mean | 0.7493 | 0.7330 | 0.7422 | 0.7406 | 0.8719 | 0.9119
GraphSAGE-LSTM | 0.7588 | 0.7509 | 0.6536 | 0.7529 | 0.8743 | 0.8652
TextGNN | 0.8310 | 0.8029 | 0.4525 | 0.8277 | 0.8779 | 0.7549
GEPS | 0.8405 | 0.8037 | 0.4892 | 0.8254 | 0.8794 | 0.6340
| BERM | 0.8785 | 0.8256 | 0.2966 | 0.8758 | 0.9079 | 0.3625
Table 3: Cases of e-commerce product search. “$y_{i}$” is the prediction score of the teacher model BERT and “$\hat{y}_{i}$” is the relevance estimation score of the student model BERM. Query | Item title | Human labeling | $y_{i}$ | $\hat{y}_{i}$
---|---|---|---|---
whistle | Li Ning whistle for basketball or volleyball game | Positive | 0.9961 | 0.9681
women’s dance shoe | Sansha modern dance shoe P22LS (black, women) | Positive | 0.9973 | 0.9908
violin adult | FineLegend 1/8 violin FLV1114 | Positive | 0.9950 | 0.9769
skating knee panels | RMT sports knee panels (black, L size) | Positive | 0.9652 | 0.9841
DJI g1200a | DJI Mavic Mini unmanned aerial vehicle | Negative | 0.0049 | 0.0514
Berkshire Hathaway Letters to Shareholders | The Snowball: Warren Buffett and the Business of Life | Negative | 0.0258 | 0.0874
My brother called Shun Liu, ZHU SU JIN | Brothers: A Novel; Author: Yu Hua | Negative | 0.1624 | 0.0263
nissan thermos cup | Disney thermos cup 500ML | Negative | 0.4938 | 0.2513
java web exercises | JSP project development Case Full Record | Negative | 0.5106 | 0.3111

###  3.2 Offline Performance

We compare BERM with 12 state-of-the-art relevance matching methods and 5
graph neural network models on our in-house product search data. The results
are shown in Tab. 2. Because some baseline methods (e.g., DRMM and ESIM) have
high time complexities, we use Data-E and Data-S for training and testing
models.

As shown in Tab. 2, BERM outperforms all the baselines according to the
metrics of AUC, F1-score, and FNR. More specifically, we have the following
findings: 1) Compared to the second-best method MVLSTM (BERT2DNN), BERM
surpasses it 1.83% (4.45%) according to AUC on Data-E (Data-S). Furthermore,
BERM achieves the lowest value of FNR on both Data-E and Data-S. This implies
that BERM can easily identify irrelevant items so that it can return a list of
satisfactory items in the real-world scene. 2) The collected training data
have imbalanced classes (i.e., the positive examples are far more than the
negative examples), which poses a challenge to model learning. Most baselines
are sensitive to class imbalance. Since BERM learns explicit node semantics by
integrating the neighboring node information, our method is robust when the
data are imbalanced.

###  3.3 Case Study

Apart from the above quantitative analysis, we conduct qualitative analysis
based on some cases of e-commerce product search. For these cases, we list the
query phrase, item title, human labeling, score of BERT, and score of BERM in
Tab. 3. We have the following empirical conclusions: 1) Most of the student’s
scores are close to the teacher’s, which indicates the success of the proposed
knowledge distillation framework. 2) Some cases imply that context information
is necessary for relevance matching. For example, for the query “nissan
thermos cup”, the teacher model cannot explicitly judge whether or not the
item entitled “Disney thermos cup 500ML” is relevant to it. With the help of
context information in the query-item bipartite graph, BERM can recognize that
this query is related to “nissan”, rather than “Disney”.

Table 4: Online performance of BERM-O under price sort mode and default sort mode. Metrics | Price Sort Mode | Default Sort Mode
---|---|---
Improvement | P-value | Improvement | P-value
UV-value | 5.713% | 3.20e-2 | 0.5013% | 1.10e-1
UCVR | 1.540% | 7.81e-2 | 0.3058% | 1.75e-2
CVR | 1.829% | 1.01e-2 | 0.1218% | 1.60e-1
RPM | 5.587% | 3.03e-2 | 0.6886% | 2.32e-2

###  3.4 Deployment & Online A/B Testing

We conduct further distillation to BERM and obtain a lighter model BERM-O
whose basic structure is a two-layer neural network. The process of further
distillation is almost the same as the first knowledge distillation. The
transfer set generated by further distillation has graph-context labels. To
further evaluate BERM-O’s performance in the real search scene, we deploy it
to the anonymous online search platform. On this platform, there are about one
hundred million daily active users (DAU) and two billion items. It processes
over 150 million search queries per day. The online baseline group BERT2DNN
Jiang et al. (2020) and control group BERM-O are deployed in a cluster, where
each node is with 64 core Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz, 256GB RAM
as well as 4 NVIDIA TESLA P40 GPU cards. For both groups, the only needed
input data are queries and item titles, which can be easily caught from the
online environment. Since BERM-O is lighter than BERT or BERM, deploying it to
the online search chain requires less engineering work in the system.

Online results. We compare BERM-O with BERT2DNN Jiang et al. (2020) which is
our online baseline model using knowledge distillation without context
information. The results of A/B testing are reported in Tab. 4. These results
are from one observation lasting more than ten days. Four widely-used online
business metrics are adopted 1) conversion rate (CVR): the average order
number of each click behavior, 2) user conversion rate (UCVR): the average
order number of each user, 3) unique visitor value (UV-value): the average
gross merchandise volume of each user, and 4) revenue per mile (RPM): the
average gross merchandise volume of each retrieval behavior. The results show
that BERM-O outperforms BERT2DNN in the platform according to all of the
business metrics. For example, BERM-O significantly improves 5.7% (relative
value) of UV-value under price sort mode.

##  4 Conclusions and Future Work

In this paper, we propose the new problem of contextual relevance matching in
e-commerce product search. Different from the previous work only using the
0-order relevance modeling, we propose a novel method of the $k$-order
relevance modeling, i.e., employing bipartite graph embedding to exploit the
potential context information in the query-item bipartite graph. Compared to
the state-of-the-art relevance matching methods, the new method BERM performs
robustly in the experiments. We deploy BERM-O (distilled from BERM) to the
anonymous online e-commerce product search platform. The results of A/B
testing indicate that BERM-O improves the user’s search experience
significantly. In the future, we plan to apply our method to other e-commerce
applications such as recommendation systems and advertisements.

## References

  * Avron et al. (2021) Uri Avron, Shay Gershtein, Ido Guy, Tova Milo, and Slava Novgorodov. 2021.  Concat: Construction of category trees from search queries in e-commerce.  In _2021 IEEE 37th International Conference on Data Engineering (ICDE)_ , pages 2701–2704. IEEE.
  * Berger et al. (2000) Adam Berger, Rich Caruana, David Cohn, Dayne Freitag, and Vibhu Mittal. 2000.  Bridging the lexical chasm: statistical approaches to answer-finding.  In _Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval_ , pages 192–199.
  * Cao et al. (2021) Jiangxia Cao, Xixun Lin, Shu Guo, Luchen Liu, Tingwen Liu, and Bin Wang. 2021.  Bipartite graph embedding via mutual information maximization.  In _Proceedings of the 14th ACM International Conference on Web Search and Data Mining_ , pages 635–643.
  * Chang et al. (2021a) Wei-Cheng Chang, Daniel Jiang, Hsiang-Fu Yu, Choon Hui Teo, Jiong Zhang, Kai Zhong, Kedarnath Kolluri, Qie Hu, Nikhil Shandilya, Vyacheslav Ievgrafov, et al. 2021a.  Extreme multi-label learning for semantic matching in product search.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, pages 2643–2651.
  * Chang et al. (2021b) Wei-Cheng Chang, Daniel Jiang, Hsiang-Fu Yu, Choon Hui Teo, Jiong Zhang, Kai Zhong, Kedarnath Kolluri, Qie Hu, Nikhil Shandilya, Vyacheslav Ievgrafov, et al. 2021b.  Extreme multi-label learning for semantic matching in product search.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, pages 2643–2651.
  * Chen et al. (2021a) Hsi-Wen Chen, Hong-Han Shuai, De-Nian Yang, Wang-Chien Lee, Chuan Shi, S Yu Philip, and Ming-Syan Chen. 2021a.  Structure-aware parameter-free group query via heterogeneous information network transformer.  In _2021 IEEE 37th International Conference on Data Engineering (ICDE)_ , pages 2075–2080. IEEE.
  * Chen et al. (2021b) Lu Chen, Chengfei Liu, Rui Zhou, Jiajie Xu, and Jianxin Li. 2021b.  Efficient exact algorithms for maximum balanced biclique search in bipartite graphs.  In _Proceedings of the 2021 International Conference on Management of Data_ , pages 248–260.
  * Chen et al. (2021c) Pu-Chin Chen, Henry Tsai, Srinadh Bhojanapalli, Hyung Won Chung, Yin-Wen Chang, and Chun-Sung Ferng. 2021c.  A simple and effective positional encoding for transformers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 2974–2988, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Chen et al. (2017) Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017\.  Enhanced lstm for natural language inference.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1657–1668.
  * Cheng et al. (2022) Dian Cheng, Jiawei Chen, Wenjun Peng, Wenqin Ye, Fuyu Lv, Tao Zhuang, Xiaoyi Zeng, and Xiangnan He. 2022.  Ihgnn: Interactive hypergraph neural network for personalized product search.  In _Proceedings of the ACM Web Conference 2022_ , pages 256–265.
  * Chi et al. (2021) Zewen Chi, Li Dong, Shuming Ma, Shaohan Huang, Saksham Singhal, Xian-Ling Mao, Heyan Huang, Xia Song, and Furu Wei. 2021.  mT6: Multilingual pretrained text-to-text transformer with translation pairs.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 1671–1683, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Chuklin et al. (2015a) Aleksandr Chuklin, Ilya Markov, and Maarten de Rijke. 2015a.  Click models for web search.  _Synthesis lectures on information concepts, retrieval, and services_ , 7(3):1–115.
  * Chuklin et al. (2015b) Aleksandr Chuklin, Ilya Markov, and Maarten de Rijke. 2015b.  Click models for web search.  _Synthesis lectures on information concepts, retrieval, and services_ , 7(3):1–115.
  * Dagan et al. (2005) Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005.  The pascal recognising textual entailment challenge.  In _Machine Learning Challenges Workshop_ , pages 177–190. Springer.
  * Devlin (2020) Jacob Devlin. 2020.  Bert.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186.
  * Dolan et al. (2004) William Dolan, Chris Quirk, Chris Brockett, and Bill Dolan. 2004.  Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources.  In _Proceedings of the 20th International Conference on Computational Linguistics_.
  * Dong et al. (2021) Chenhe Dong, Yaliang Li, Ying Shen, and Minghui Qiu. 2021.  HRKD: Hierarchical relational knowledge distillation for cross-domain language model compression.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3126–3136, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Dou et al. (2022) Wenzhou Dou, Derong Shen, Tiezheng Nie, Yue Kou, Chenchen Sun, Hang Cui, and Ge Yu. 2022.  Empowering transformer with hybrid matching knowledge for entity matching.  In _International Conference on Database Systems for Advanced Applications_ , pages 52–67. Springer.
  * Esteva et al. (2021) Andre Esteva, Anuprit Kale, Romain Paulus, Kazuma Hashimoto, Wenpeng Yin, Dragomir Radev, and Richard Socher. 2021.  Covid-19 information retrieval with deep-learning based semantic search, question answering, and abstractive summarization.  _NPJ digital medicine_ , 4(1):1–9.
  * Fan et al. (2019) Shaohua Fan, Junxiong Zhu, Xiaotian Han, Chuan Shi, Linmei Hu, Biyu Ma, and Yongliang Li. 2019.  Metapath-guided heterogeneous graph neural network for intent recommendation.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 2478–2486.
  * Fan (2015) Yixing Fan. 2015.  Matchzoo.
  * Fan et al. (2017) Yixing Fan, Liang Pang, JianPeng Hou, Jiafeng Guo, Yanyan Lan, and Xueqi Cheng. 2017\.  Matchzoo: A toolkit for deep text matching.  In _Neu-IR: The SIGIR 2017 Workshop on Neural Information Retrieval_.
  * Fei et al. (2021) Hongliang Fei, Jingyuan Zhang, Xingxuan Zhou, Junhao Zhao, Xinyang Qi, and Ping Li. 2021.  Gemnn: gating-enhanced multi-task neural networks with feature interaction learning for ctr prediction.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 2166–2171.
  * Gao et al. (2021) Chen Gao, Tzu-Heng Lin, Nian Li, Depeng Jin, and Yong Li. 2021.  Cross-platform item recommendation for online social e-commerce.  _IEEE Transactions on Knowledge and Data Engineering_.
  * Grover and Leskovec (2016) Aditya Grover and Jure Leskovec. 2016.  node2vec: Scalable feature learning for networks.  In _Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 855–864.
  * Guo et al. (2016) Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft. 2016.  A deep relevance matching model for ad-hoc retrieval.  In _Proceedings of the 25th ACM International on Conference on Information and Knowledge Management_ , pages 55–64.
  * Guo et al. (2020) Ruocheng Guo, Xiaoting Zhao, Adam Henderson, Liangjie Hong, and Huan Liu. 2020.  Debiasing grid-based product search in e-commerce.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 2852–2860.
  * Hamilton et al. (2017a) Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017a.  Inductive representation learning on large graphs.  In _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc.
  * Hamilton et al. (2017b) Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017b.  Inductive representation learning on large graphs.  _Advances in neural information processing systems_ , 30.
  * Hinton et al. (2015) Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015.  Distilling the knowledge in a neural network.  _arXiv preprint arXiv:1503.02531_.
  * Hu et al. (2014a) Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014a.  Convolutional neural network architectures for matching natural language sentences.  In _Advances in neural information processing systems_ , pages 2042–2050.
  * Hu et al. (2014b) Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014b.  Convolutional neural network architectures for matching natural language sentences.  In _Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada_ , pages 2042–2050.
  * Huang et al. (2021) Chao Huang, Jiashu Zhao, and Dawei Yin. 2021.  Purchase intent forecasting with convolutional hierarchical transformer networks.  In _2021 IEEE 37th International Conference on Data Engineering (ICDE)_ , pages 2488–2498. IEEE.
  * Huang et al. (2020) Jui-Ting Huang, Ashish Sharma, Shuying Sun, Li Xia, David Zhang, Philip Pronin, Janani Padmanabhan, Giuseppe Ottaviano, and Linjun Yang. 2020.  Embedding-based retrieval in facebook search.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 2553–2561.
  * Huang et al. (2013) Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013\.  Learning deep structured semantic models for web search using clickthrough data.  In _Proceedings of the 22nd ACM international conference on Information & Knowledge Management_, pages 2333–2338.
  * Jiang et al. (2019) Yunjiang Jiang, Yue Shang, Rui Li, Wen-Yun Yang, Guoyu Tang, Chaoyi Ma, Yun Xiao, and Eric Zhao. 2019.  A unified neural network approach to e-commerce relevance learning.  In _Proceedings of the 1st International Workshop on Deep Learning Practice for High-Dimensional Sparse Data_ , page 10. ACM.
  * Jiang et al. (2020) Yunjiang Jiang, Yue Shang, Ziyang Liu, Hongwei Shen, Yun Xiao, Wei Xiong, Sulong Xu, Weipeng Yan, and Di Jin. 2020.  Bert2dnn: Bert distillation with massive unlabeled data for online e-commerce search.  In _2020 IEEE International Conference on Data Mining (ICDM)_ , pages 212–221. IEEE.
  * Jiao et al. (2020) Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020.  Tinybert: Distilling bert for natural language understanding.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings_ , pages 4163–4174.
  * Kapanipathi et al. (2020) Pavan Kapanipathi, Veronika Thost, Siva Sankalp Patel, Spencer Whitehead, Ibrahim Abdelaziz, Avinash Balakrishnan, Maria Chang, Kshitij Fadnis, Chulaka Gunasekara, Bassem Makni, et al. 2020.  Infusing knowledge into the textual entailment task using graph convolutional networks.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 8074–8081.
  * Kim et al. (2019) Seonhoon Kim, Inho Kang, and Nojun Kwak. 2019.  Semantic sentence matching with densely-connected recurrent and co-attentive information.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 33, pages 6586–6593.
  * Kipf and Welling (2016) Thomas N Kipf and Max Welling. 2016.  Semi-supervised classification with graph convolutional networks.  _arXiv preprint arXiv:1609.02907_.
  * Li (2011) Hang Li. 2011.  Learning to rank for information retrieval and natural language processing.  _Synthesis Lectures on Human Language Technologies_ , 4(1):1–113.
  * Li et al. (2021a) Lei Li, Yankai Lin, Shuhuai Ren, Peng Li, Jie Zhou, and Xu Sun. 2021a.  Dynamic knowledge distillation for pre-trained language models.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 379–389, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Li et al. (2021b) Rundong Li, Pinghui Wang, Peng Jia, Xiangliang Zhang, Junzhou Zhao, Jing Tao, Ye Yuan, and Xiaohong Guan. 2021b.  Approximately counting butterflies in large bipartite graph streams.  _IEEE Transactions on Knowledge and Data Engineering_.
  * Li et al. (2021c) Sen Li, Fuyu Lv, Taiwei Jin, Guli Lin, Keping Yang, Xiaoyi Zeng, Xiao-Ming Wu, and Qianli Ma. 2021c.  Embedding-based product retrieval in taobao search.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, pages 3181–3189.
  * Lin et al. (2021a) Rongmei Lin, Xiang He, Jie Feng, Nasser Zalmout, Yan Liang, Li Xiong, and Xin Luna Dong. 2021a.  Pam: understanding product images in cross product category attribute extraction.  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, pages 3262–3270.
  * Lin et al. (2021b) Ye Lin, Yanyang Li, Tong Xiao, and Jingbo Zhu. 2021b.  Bag of tricks for optimizing transformer efficiency.  In _EMNLP_.
  * Liu et al. (2020) Bin Liu, Niannan Xue, Huifeng Guo, Ruiming Tang, Stefanos Zafeiriou, Xiuqiang He, and Zhenguo Li. 2020.  Autogroup: Automatic feature grouping for modelling explicit high-order feature interactions in ctr prediction.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 199–208.
  * Liu et al. (2022a) Fanzhen Liu, Zhao Li, Baokun Wang, Jia Wu, Jian Yang, Jiaming Huang, Yiqing Zhang, Weiqiang Wang, Shan Xue, Surya Nepal, et al. 2022a.  eriskcom: an e-commerce risky community detection platform.  _The VLDB Journal_ , pages 1–17.
  * Liu et al. (2022b) Hao Liu, Jindong Han, Yanjie Fu, Yanyan Li, Kai Chen, and Hui Xiong. 2022b.  Unified route representation learning for multi-modal transportation recommendation with spatiotemporal pre-training.  _The VLDB Journal_ , pages 1–18.
  * Liu et al. (2019a) Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019a.  Improving multi-task deep neural networks via knowledge distillation for natural language understanding.  _arXiv preprint arXiv:1904.09482_.
  * Liu et al. (2021) Xuye Liu, Dakuo Wang, April Wang, Yufang Hou, and Lingfei Wu. 2021.  HAConvGNN: Hierarchical attention based convolutional graph neural network for code documentation generation in Jupyter notebooks.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 4473–4485, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Liu et al. (2019b) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Liu and Mao (2020) Yiqun Liu and Jiaxin Mao. 2020.  " revisiting information retrieval tasks with user behavior models" by yiqun liu and jiaxin mao with martin vesely as coordinator.  _ACM SIGWEB Newsletter_ , pages 1–8.
  * Luo et al. (2018) Kangqi Luo, Fengli Lin, Xusheng Luo, and Kenny Zhu. 2018.  Knowledge base question answering via encoding of complex query graphs.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2185–2194.
  * Lyu et al. (2022) Bingqing Lyu, Lu Qin, Xuemin Lin, Ying Zhang, Zhengping Qian, and Jingren Zhou. 2022\.  Maximum and top-k diversified biclique search at scale.  _The VLDB Journal_ , pages 1–25.
  * Mao et al. (2019) Jiaxin Mao, Zhumin Chu, Yiqun Liu, Min Zhang, and Shaoping Ma. 2019.  Investigating the reliability of click models.  In _Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval_ , pages 125–128.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient estimation of word representations in vector space.  In _1st International Conference on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings_.
  * Mitra et al. (2017) Bhaskar Mitra, Fernando Diaz, and Nick Craswell. 2017.  Learning to match using local and distributed representations of text for web search.  In _Proceedings of the 26th International Conference on World Wide Web, WWW 2017, Perth, Australia, April 3-7, 2017_ , pages 1291–1299. ACM.
  * Nigam et al. (2019) Priyanka Nigam, Yiwei Song, Vijai Mohan, Vihan Lakshman, Weitian Ding, Ankit Shingavi, Choon Hui Teo, Hao Gu, and Bing Yin. 2019.  Semantic product search.  In _Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 2876–2885.
  * Niu et al. (2020) Xichuan Niu, Bofang Li, Chenliang Li, Rong Xiao, Haochuan Sun, Hongbo Deng, and Zhenzhong Chen. 2020.  A dual heterogeneous graph attention network to improve long-tail performance for shop search in e-commerce.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 3405–3415.
  * Pang et al. (2016a) Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2016a.  Text matching as image recognition.  In _Proceedings of the 30th AAAI Conference on Artificial Intelligence_.
  * Pang et al. (2016b) Liang Pang, Yanyan Lan, Jiafeng Guo, Jun Xu, Shengxian Wan, and Xueqi Cheng. 2016b.  Text matching as image recognition.  In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA_ , pages 2793–2799.
  * Qin et al. (2022) Chuan Qin, Kaichun Yao, Hengshu Zhu, Tong Xu, Dazhong Shen, Enhong Chen, and Hui Xiong. 2022.  Towards automatic job description generation with capability-aware neural networks.  _IEEE Transactions on Knowledge and Data Engineering_.
  * Qiu et al. (2021) Yiming Qiu, Kang Zhang, Han Zhang, Songlin Wang, Sulong Xu, Yun Xiao, Bo Long, and Wen-Yun Yang. 2021.  Query rewriting via cycle-consistent translation for e-commerce search.  In _2021 IEEE 37th International Conference on Data Engineering (ICDE)_ , pages 2435–2446. IEEE.
  * Qu et al. (2021) Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021.  Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 5835–5847.
  * Rao et al. (2019) Jinfeng Rao, Linqing Liu, Yi Tay, Wei Yang, Peng Shi, and Jimmy Lin. 2019.  Bridging the gap between relevance matching and semantic matching for short text similarity modeling.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5370–5381.
  * Rashid et al. (2021) Ahmad Rashid, Vasileios Lioutas, Abbas Ghaddar, and Mehdi Rezagholizadeh. 2021.  Towards zero-shot knowledge distillation for natural language processing.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 6551–6561, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Reid et al. (2021) Machel Reid, Edison Marrese-Taylor, and Yutaka Matsuo. 2021.  Subformer: Exploring weight sharing for parameter efficiency in generative transformers.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 4081–4090, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Roßrucker (2022) Georg P Roßrucker. 2022.  State-of-the-art survey on web search.  In _The Autonomous Web_ , pages 1–24. Springer.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  _arXiv preprint arXiv:1910.01108_.
  * Shen et al. (2014) Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. 2014.  Learning semantic representations using convolutional neural networks for web search.  In _Proceedings of the 23rd international conference on world wide web_ , pages 373–374.
  * Su et al. (2020) Yumin Su, Liang Zhang, Quanyu Dai, Bo Zhang, Jinyao Yan, Dan Wang, Yongjun Bao, Sulong Xu, Yang He, and Weipeng Yan. 2020.  An attention-based model for conversion rate prediction with delayed feedback via post-click calibration.  In _IJCAI_ , pages 3522–3528.
  * Sun and Duh (2020) Shuo Sun and Kevin Duh. 2020.  CLIRMatrix: A massively large collection of bilingual and multilingual datasets for cross-lingual information retrieval.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4160–4170, Online. Association for Computational Linguistics.
  * Sun et al. (2019a) Siqi Sun, Yu Cheng, Zhe Gan, and Jingjing Liu. 2019a.  Patient knowledge distillation for bert model compression.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4323–4332.
  * Sun et al. (2019b) Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, and Hua Wu. 2019b.  Ernie: Enhanced representation through knowledge integration.  _arXiv preprint arXiv:1904.09223_.
  * Sybrandt and Safro (2019) Justin Sybrandt and Ilya Safro. 2019.  Fobe and hobe: First-and high-order bipartite embeddings.  _arXiv preprint arXiv:1905.10953_.
  * Tang et al. (2015) Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. 2015.  Line: Large-scale information network embedding.  In _Proceedings of the 24th international conference on world wide web_ , pages 1067–1077.
  * Tang et al. (2019) Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. 2019\.  Distilling task-specific knowledge from bert into simple neural networks.  _arXiv preprint arXiv:1903.12136_.
  * Thorne et al. (2021) James Thorne, Majid Yazdani, Marzieh Saeidi, Fabrizio Silvestri, Sebastian Riedel, and Alon Halevy. 2021.  From natural language processing to neural databases.  In _Proceedings of the VLDB Endowment_ , volume 14, pages 1033–1039. VLDB Endowment.
  * Vaswani et al. (2017a) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017a.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Vaswani et al. (2017b) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017b.  Attention is all you need.  In _Advances in neural information processing systems_ , pages 5998–6008.
  * Veličković et al. (2018) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018.  Graph attention networks.  In _International Conference on Learning Representations_.
  * Veličković et al. (2018) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. 2018.  Graph attention networks.  In _International Conference on Learning Representations_.
  * Wan et al. (2016a) Shengxian Wan, Yanyan Lan, Jiafeng Guo, Jun Xu, Liang Pang, and Xueqi Cheng. 2016a.  A deep architecture for semantic matching with multiple positional sentence representations.  In _Proceedings of the 30th AAAI Conference on Artificial Intelligence_.
  * Wan et al. (2016b) Shengxian Wan, Yanyan Lan, Jiafeng Guo, Jun Xu, Liang Pang, and Xueqi Cheng. 2016b.  A deep architecture for semantic matching with multiple positional sentence representations.  In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, February 12-17, 2016, Phoenix, Arizona, USA_ , pages 2835–2841.
  * Wang et al. (2016) Daixin Wang, Peng Cui, and Wenwu Zhu. 2016.  Structural deep network embedding.  In _Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining_ , pages 1225–1234.
  * Wang et al. (2020) Junmei Wang, Min Pan, Tingting He, Xiang Huang, Xueyan Wang, and Xinhui Tu. 2020\.  A pseudo-relevance feedback framework combining relevance matching and semantic matching for information retrieval.  _Information Processing & Management_, 57(6):102342.
  * Wang et al. (2022a) Kai Wang, Xuemin Lin, Lu Qin, Wenjie Zhang, and Ying Zhang. 2022a.  Accelerated butterfly counting with vertex priority on bipartite graphs.  _The VLDB Journal_ , pages 1–25.
  * Wang et al. (2021) Kai Wang, Wenjie Zhang, Xuemin Lin, Ying Zhang, Lu Qin, and Yuting Zhang. 2021.  Efficient and effective community search on large-scale bipartite graphs.  In _2021 IEEE 37th International Conference on Data Engineering (ICDE)_ , pages 85–96. IEEE.
  * Wang et al. (2019a) Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019a.  Neural graph collaborative filtering.  In _Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR’19, page 165–174, New York, NY, USA. Association for Computing Machinery.
  * Wang et al. (2019b) Zekai Wang, Hongzhi Liu, Yingpeng Du, Zhonghai Wu, and Xing Zhang. 2019b.  Unified embedding model over heterogeneous information network for personalized recommendation.  In _Proceedings of the 28th International Joint Conference on Artificial Intelligence_ , pages 3813–3819. AAAI Press.
  * Wang et al. (2022b) Zhen Wang, Long Zhang, Rong Wang, Feiping Nie, and Xuelong Li. 2022b.  Semi-supervised learning via bipartite graph construction with adaptive neighbors.  _IEEE Transactions on Knowledge and Data Engineering_.
  * Wu et al. (2021a) Xinle Wu, Dalin Zhang, Chenjuan Guo, Chaoyang He, Bin Yang, and Christian S Jensen. 2021a.  Autocts: Automated correlated time series forecasting.  _Proceedings of the VLDB Endowment_ , 15(4):971–983.
  * Wu et al. (2021b) Yimeng Wu, Mehdi Rezagholizadeh, Abbas Ghaddar, Md Akmal Haidar, and Ali Ghodsi. 2021b.  Universal-KD: Attention-based output-grounded intermediate layer knowledge distillation.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7649–7661, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Xia et al. (2022) Lianghao Xia, Chao Huang, Yong Xu, and Jian Pei. 2022.  Multi-behavior sequential recommendation with temporal graph transformer.  _IEEE Transactions on Knowledge and Data Engineering_.
  * Xiong et al. (2017) Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. 2017.  End-to-end neural ad-hoc ranking with kernel pooling.  In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, Shinjuku, Tokyo, Japan, August 7-11, 2017_ , pages 55–64. ACM.
  * Xiong et al. (2020) Xi Xiong, Yuanyuan Li, Rui Zhang, Zhan Bu, Guiqing Li, and Shenggen Ju. 2020.  Dgi: recognition of textual entailment via dynamic gate matching.  _Knowledge-Based Systems_ , 194:105544.
  * Xu et al. (2021) Song Xu, Haoran Li, Peng Yuan, Yujia Wang, Youzheng Wu, Xiaodong He, Ying Liu, and Bowen Zhou. 2021.  K-PLUG: Knowledge-injected pre-trained language model for natural language understanding and generation in E-commerce.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 1–17, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Yang et al. (2020) Carl Yang, Yuxin Xiao, Yu Zhang, Yizhou Sun, and Jiawei Han. 2020.  Heterogeneous network representation learning: Survey, benchmark, evaluation, and beyond.  _arXiv preprint arXiv:2004.00216_.
  * Yang et al. (2021) Jia-Qi Yang, Xiang Li, Shuguang Han, Tao Zhuang, De-Chuan Zhan, Xiaoyi Zeng, and Bin Tong. 2021.  Capturing delayed feedback in conversion rate prediction via elapsed-time sampling.  pages 4582–4589.
  * Ying et al. (2018) Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. 2018.  Graph convolutional neural networks for web-scale recommender systems.  In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 974–983.
  * Yu et al. (2018) Junliang Yu, Min Gao, Jundong Li, Hongzhi Yin, and Huan Liu. 2018.  Adaptive implicit friends identification over heterogeneous network for social recommendation.  In _Proceedings of the 27th ACM International Conference on Information and Knowledge Management_ , pages 357–366.
  * Yu et al. (2021) Xueli Yu, Weizhi Xu, Zeyu Cui, Shu Wu, and Liang Wang. 2021.  Graph-based hierarchical relevance matching signals for ad-hoc retrieval.  _CoRR_ , abs/2102.11127.
  * Zhai and Lafferty (2017) Chengxiang Zhai and John Lafferty. 2017.  A study of smoothing methods for language models applied to ad hoc information retrieval.  In _ACM SIGIR Forum_ , volume 51, pages 268–276. ACM New York, NY, USA.
  * Zhang and Chen (2019) Muhan Zhang and Yixin Chen. 2019.  Inductive matrix completion based on graph neural networks.  _arXiv preprint arXiv:1904.12058_.
  * Zhang et al. (2019a) Yuan Zhang, Dong Wang, and Yan Zhang. 2019a.  Neural ir meets graph embedding: A ranking model for product search.  _The World Wide Web Conference_.
  * Zhang et al. (2019b) Yuan Zhang, Dong Wang, and Yan Zhang. 2019b.  Neural ir meets graph embedding: a ranking model for product search.  In _The World Wide Web Conference_ , pages 2390–2400.
  * Zhang et al. (2020) Yuan Zhang, Xiaoran Xu, Hanning Zhou, and Yan Zhang. 2020.  Distilling structured knowledge into embeddings for explainable and accurate recommendation.  In _Proceedings of the 13th International Conference on Web Search and Data Mining_ , pages 735–743.
  * Zhang et al. (2021) Yufeng Zhang, Jinghao Zhang, Zeyu Cui, Shu Wu, and Liang Wang. 2021.  A graph-based relevance matching model for ad-hoc retrieval.  _CoRR_ , abs/2101.11873.
  * Zhao et al. (2019) Huan Zhao, Yingqi Zhou, Yangqiu Song, and Dik Lun Lee. 2019.  Motif enhanced recommendation over heterogeneous information network.  In _Proceedings of the 28th ACM International Conference on Information and Knowledge Management_ , pages 2189–2192.
  * Zhao et al. (2020) Jianan Zhao, Xiao Wang, Chuan Shi, Zekuan Liu, and Yanfang Ye. 2020.  Network schema preserving heterogeneous information network embedding.  In _International Joint Conference on Artificial Intelligence (IJCAI)_.
  * Zhou et al. (2021) Alexander Zhou, Yue Wang, and Lei Chen. 2021.  Butterfly counting on uncertain bipartite graphs.  _Proceedings of the VLDB Endowment_ , 15(2):211–223.
  * Zhu et al. (2021) Jason Zhu, Yanling Cui, Yuming Liu, Hao Sun, Xue Li, Markus Pelger, Tianqi Yang, Liangjie Zhang, Ruofei Zhang, and Huasha Zhao. 2021.  Textgnn: Improving text encoder via graph neural network in sponsored search.  In _WWW ’21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021_ , pages 2848–2857. ACM / IW3C2.
  * Zhu et al. (2020) Tiangang Zhu, Yue Wang, Haoran Li, Youzheng Wu, Xiaodong He, and Bowen Zhou. 2020\.  Multimodal joint attribute prediction and value extraction for E-commerce product.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2129–2139, Online. Association for Computational Linguistics.

##  Appendix A Related Work

In this section, we review classical relevance matching models (Section A.1),
Transformer-style models (Section A.2), and online knowledge distillation
methods (Section A.3). In each subsection, we conclude the difference between
the previous work and ours.

###  A.1 Classical Relevance Matching Models

Traditional relevance matching uses TF-IDF or BM25 to calculate the text
similarity between the query and document. This type of method is mainly based
on the word frequency in the text and cannot measure the similarity
relationship between different words or phrases. With the development of deep
learning, relevance matching combined with the characterization’s advantage of
vector representations on the semantic relationship is booming. The prevailing
methods are either representation-focused (e.g., DSSM Huang et al. (2013),
CDSSM Shen et al. (2014) and ARC-I Hu et al. (2014a)) or interaction-focused
(e.g., MatchPyramid Pang et al. (2016a), ARC-II Hu et al. (2014a) and ESIM
Chen et al. (2017)). The representation-focused methods learn the low-
dimension representations of both sentences and then predict their
relationship by calculating the similarity (such as cosine similarity) of
representations. The interaction-focused methods directly learn an interaction
representation of both sentences based on the interaction calculation from
word-level to sentence-level. These methods are typically effective for
context-free relevance matching.

However, contextual relevance matching is necessary to real applications such
as e-commerce product search. The above methods ignore the inherent context
information contained in search logs Qin et al. (2022); Roßrucker (2022). In
this work, we incorporate the advantages of representation-based and
interaction-based embeddings into BERM, which is focused on contextual
relevance matching.

###  A.2 Transformer-style Models

More recently, Transformer-based models Chen et al. (2021c); Chi et al.
(2021); Lin et al. (2021b); Reid et al. (2021) have achieved breakthroughs on
many NLP tasks and reached human-level accuracy. The representative models
include BERT Devlin et al. (2019), ERNIE Sun et al. (2019b), and RoBERTa Liu
et al. (2019b). However, the multi-layer stacked Transformer structure in
these models leads to high time complexity, so they are hard to deploy online.
To effectively apply Transformer-based models, in this work, we use BERT to
generate the labels of transfer set Hinton et al. (2015) as the supervised
information for BERM; 2) refine the noisy behavior data to ensure that the
constructed bipartite graph is credible. Additionally, GRMM Zhang et al.
(2021) and GHRM Yu et al. (2021) use graph information to enforce the
relevance matching model for information retrieval.

However, the above work is essentially different from ours on the definition
of graphs. In their constructed graph, nodes are unique words and edges are
the co-occurrent relationships. In this work, we leverage query phrases (or
item titles) as nodes and user behaviors as edges, which is more suitable for
the product search problem.

###  A.3 Online Knowledge Distillation Methods

Knowledge distillation is firstly proposed in Hinton et al. (2015). Its main
idea is to transfer the knowledge generated by a massive teacher model into a
light student model. Because of the low complexity of the student model, it is
easy to deploy the student model to the online platform. Considering the
strong semantic understanding ability of BERT, some studies exploit the
potential of BERT as the teacher model of knowledge distillation. Two types of
design principles are general: isomorphic principle and isomeric principle.
Specifically, the distillation methods that follow the isomorphic principle
use the same model architecture for teacher and student models, such as
TinyBERT Jiao et al. (2020), BERT-PKD Sun et al. (2019a), MTDNN Liu et al.
(2019a), and DistilBERT Sanh et al. (2019). As a more advanced design
principle, the isomeric principle uses different model architectures for
teacher and student models, such as Distilled BiLSTM Tang et al. (2019) and
BERT2DNN Jiang et al. (2020).

Although the above methods reduce the total time costs by learning a light
student model, they ignore the context information in either the teacher model
or student model. Our proposed knowledge distillation framework follows the
isomeric principle and further integrates the context information into the
student model by bipartite graph embedding.

##  Appendix B Details of our Framework

In this section, we introduce more details about the proposed knowledge
distillation framework, including the overview (Section 2.2), bipartite graph
construction (Section B.1), and bipartite graph analysis (Section B.2).

###  B.1 Bipartite Graph Construction

As we referred to in the body of the paper, queries and items in the product
search can be naturally abstracted as a bipartite graph where edges are
generated by multi-typed user behaviors like view, click, and purchase.
However, user behaviors are biased and noisy. Directly learning the embedding
on the raw user behavior graph $\mathcal{G}$ makes it difficult to estimate
the relevance relationship between nodes. Many click models (see the survey in
Chuklin et al. (2015a)) have been proposed to absorb all the potential biases
into their designed model. However, these click models are sophisticated,
making them consume lots of memory resources in the online environment. To
address this problem, we adopt a simple but effective strategy: we introduce
the external knowledge from BERT to refine the raw user behavior graph
$\mathcal{G}$ and then construct a knowledge-enhanced bipartite graph
$\mathcal{G}^{\prime}$. The whole graph construction includes the following
phases.

Fine-tuning BERT. Here the external knowledge is provided by the BERT model.
Transformer-based models such as BERT Devlin et al. (2019) and ERNIE Sun et
al. (2019b) have been preferably used as NLP benchmarks in recent years. Here
we use the BERT model as the teacher model, which is pre-trained on a large
text corpus and then fine-tuned on our in-house data. The positive and
negative examples in the data are human-labeled and cover various categories
of items. The fine-tuned BERT is equipped with a great ability of relevance
discrimination and thus can act as an expert on filtering noisy data. For each
example pair $p_{i}$ in the transfer set $S_{\mathrm{transfer}}$, we use BERT
to predict its score $y_{i}$ as the training label of the student model BERM.

Behavior graph construction. The user behavior graph $\mathcal{G}$ is built on
the user search log over six months which records click and purchase behaviors
as well as their frequencies. In $\mathcal{G}$, an edge represents an existing
click behavior or purchase behavior between a query and an item. The click
behavior edges are dense and highly noisy, so we introduce the fine-tuned BERT
model to refine $\mathcal{G}$.

Knowledge-enhanced graph refinement. We retain all the raw purchase behavior
edges, and meanwhile use the knowledge generated by the fine-tuned BERT to
refine the click behavior edges. Specifically, we set two different thresholds
$\alpha$ and $\beta$ to decide which click edges are removed and which
important edges are added. This strategy can help remove the noise in user
behaviors, and at the same time retrieve the missing but relevant neighbors
which are not captured by user behaviors. To preserve the high-quality
neighbor set, for each anchor node, we rank its 1-hop neighbors with the
priority of “purchase$>$high click$>$low click” and sample the top two of them
as the final neighbor list, e.g., the neighbor list of a query node $Q$ is
represented as $[I_{\mathrm{top1}},I_{\mathrm{top2}}]$. The implementation of
graph construction is shown in Algorithm 1.

Algorithm 1 Bipartite Graph Construction

0: Thresholds $\alpha$ and $\beta$; Collected dataset
$S_{\mathrm{input}}=\left\\{p_{i}\right\\}_{\tilde{n}}$ (note that $\tilde{n}$
is the number of examples); Raw user behavior graph
$\mathcal{G}=(\mathcal{U},\mathcal{V},\mathcal{E},\mathcal{A},\mathcal{R})$
where $\mathcal{E}=\left\\{e_{1},\cdots,e_{m}\right\\}$,
$\mathcal{A}=\left\\{Query,Item\right\\}$,
$\mathcal{R}=\left\\{Click,Purchase\right\\}$.

0: Transfer set
$S_{\mathrm{transfer}}=\left\\{p_{i};y_{i}\right\\}_{\tilde{n}}$ where $y_{i}$
is the training label of query-item pair $p_{i}$ ($y_{i}\in\left[0,1\right]$);
Refined bipartite graph
${\mathcal{G}}^{\prime}=(\mathcal{U},\mathcal{V},{\mathcal{E}}^{\prime},\mathcal{A},\mathcal{R})$.

1: Initialize ${\mathcal{E}}^{\prime}$:
${\mathcal{E}}^{\prime}={\mathcal{E}}$.

2: Fine-tune BERT on the human-labeled data.

3: Use the fine-tuned BERT to predict on $S_{\mathrm{input}}$ and then obtain
$S_{\mathrm{transfer}}=\left\\{p_{i};y_{i}\right\\}_{\tilde{n}}$.

4: for $p_{i},y_{i}$ in $S_{\mathrm{transfer}}$ do

5: Build an edge between the pair $p_{i}$ and denote it as
$e_{i}=edge(p_{i})$.

6: if $e_{i}\in\mathcal{E}$ then

7: if $f_{2}(e_{i})=Purchase$ then

8: continue;

9: else if $y_{i}<\alpha$ then

10:
${\mathcal{E}}^{\prime}={\mathcal{E}}^{\prime}\setminus\left\\{{e_{i}}\right\\}$

11: end if

12: else if $y_{i}>\beta$ then

13:
${\mathcal{E}}^{\prime}={\mathcal{E}}^{\prime}\cup\left\\{{e_{i}}\right\\}$

14: end if

15: end for

###  B.2 Bipartite Graph Analysis

Network schema is an abstraction of the basic units in the graph Zhao et al.
(2020). In BERM, the basic network schema includes query node, item node, and
the refined user-behavior-oriented edge between them. Based on this schema, we
employ two metapaths: “$Q$-$I$-$Q$” and “$I$-$Q$-$I$” where “$Q$” and “$I$”
represent “$Query$” and “$Item$”, respectively. These two metapaths decide the
used context information $C_{k}$ (Def. 4) in our model. Compared to some more
complex metapaths (such as “$Q$-$I$-$Q$-$I$-$Q$” and “$I$-$Q$-$I$-$Q$-$I$”),
the adopted metapaths in our model are both effective and computationally
efficient. We further choose four instances for each metapath if they are
available and pad them with zero embeddings if not. Taking “$Q$-$I$-$Q$” as an
example, its instances include
“$Q$-$I_{\mathrm{top1}}$-$Q_{\mathrm{{top1}}}$”,
“$Q$-$I_{\mathrm{{top1}}}$-$Q_{\mathrm{{top2}}}$”,
“$Q$-$I_{\mathrm{top2}}$-$Q_{\mathrm{{top1}}}$”, and
“$Q$-$I_{\mathrm{{top2}}}$-$Q_{\mathrm{{top2}}}$”. For the incomplete
metapaths, we pad them with null nodes which are not equipped with real
semantic information.

##  Appendix C Complexity Analysis

In this section, we analyze the time and space complexities of BERT (teacher
model), BERM (student model), and BERM-O (online model).

###  C.1 Time Complexity

For the lookup operation on the static vocabulary table (i.e., a word
embedding table whose size is $n_{w}$), the time complexities of BERT, BERM,
and BERM-O are the same, i.e., O($n_{w}$). For the model calculation part,
BERT uses Transformer networks. We denote the word embedding size, head
number, network number, query length, and item length as
$d_{1},h_{1},k_{1},l_{Q},l_{I}$, respectively. For the one-layer multi-head
attention mechanism, the complexity of linear mapping (input part) is
O($\frac{1}{h_{1}}(l_{Q}+l_{I})d_{1}^{2}$), the complexity of attention
operation is O($h_{1}l_{Q}^{2}d+hl_{I}^{2}d_{1}$), and the complexity of
linear mapping (output part) is O($(l_{Q}+l_{I})d_{1}^{2}$). Therefore, the
total model calculation complexity of $k_{1}$-layer BERT is
O($\frac{h_{1}+1}{h_{1}}(l_{Q}+l_{I})d_{1}^{2}k_{1}+(l_{Q}^{2}+l_{I}^{2})h_{1}d_{1}k_{1}$)
. For the student model BERM, we denote the word embedding size, hidden size,
and network number as $d,h_{2},k_{2}$, respectively. The complexity of
calculating micro matching embedding is O($l_{Q}l_{I}d$), which is far more
than that of calculating micro matching embedding. The complexity of
$k_{2}$-layer DNN is O($h_{2}dk_{2}$). Therefore, the total model calculation
complexity of $k_{2}$-layer BERM is O($l_{Q}l_{I}d+h_{2}dk_{2}$). For the
online model BERM-O, we denote the word embedding size, hidden size, and
network number as $d_{3},h_{3},k_{3}$, respectively. The model calculation
complexity of $k_{3}$-layer BERM-O is O($d_{3}h_{3}k_{3}$). Note that the
complexity of BERM-O is independent of $l_{Q}$ and $l_{I}$ because BERM-O only
receives sentence embeddings and does not calculate word-level matching
signals. Based on the above analysis, we can conclude that BERM is more
efficient than BERT and meanwhile BERM-O has more advantages than BERM on time
complexity.

###  C.2 Space Complexity

The storage of the static vocabulary table takes up the majority of the total
space storage. Therefore, the space complexities of BERT, BERM, and BERM-O are
the same, i.e., O($n_{w}d$).

##  Appendix D Details of Experimental setups

###  D.1 Baselines

The model BERM is compared with some state-of-the-art models. Like BERM, these
models are used as the student model of the proposed knowledge distillation
framework. We adopt the hyper-parameter settings recommended by the original
papers for all the methods. According to the formulation process of embedding,
these methods can be divided into the following three types:

  * •

Three representation-focused relevance matching methods (denoted by “§” in
Tab. 2): DSSM Huang et al. (2013), MVLSTM Wan et al. (2016b), and ARC-I Hu et
al. (2014a). They learn the low-dimension representations of both sentences
w.r.t. a query and an item, and then predict their relationship by calculating
the similarity (such as cosine similarity) of representations.

  * •

Seven interaction-focused relevance matching methods (denoted by “†” in Tab.
2): DRMM Guo et al. (2016), MatchPyramid Pang et al. (2016b), ARC-II Hu et al.
(2014b), K-NRM Xiong et al. (2017), DRMM-TKS Guo et al. (2016), Conv-KNRM
Xiong et al. (2017), and ESIM Chen et al. (2017). They learn an interaction
representation of both sentences based on the interaction calculation from
word-level to sentence-level.

  * •

Two integrated relevance matching methods (denoted by “‡” in Tab. 2): Duet
Mitra et al. (2017) and BERT2DNN Jiang et al. (2020). They combine the
features of the above two types of methods into themselves.

  * •

Five graph neural network models (denoted by “$\star$” in Tab. 2): GAT
Veličković et al. (2018), GraphSAGE-Mean Hamilton et al. (2017a), GraphSAGE-
LSTM Hamilton et al. (2017a), TextGNN Zhu et al. (2021), and GEPS Zhang et al.
(2019a). They aggregate the neighbor information from the query graph or item
graph to update the embedding of the anchor node.

###  D.2 Implementation Details

Here we introduce the implementation details of the whole knowledge
distillation framework as follows:

  * •

Teacher model. For the teacher model, we adopt BERT-
Base***https://github.com/google-research/bert with a 12-layer ($k_{1}=12$)
Transformer encoder where the word embedding size $d_{1}$ is 768 and head
number $h_{1}$ is 12. We pre-train BERT-Base on a human-labeled dataset with
380,000 query-item pairs. The fine-tuned BERT-Base is then used as an expert
to refine the noisy click behavior data from $S_{\mathrm{train}}$. The
refinement rule is: if the prediction score $y_{i}$ of BERT-Base is less than
$\alpha$ (the default value of $\alpha$ is 0.3), then the raw edge is deleted;
if the score is larger than $\beta$ (the default value of $\beta$ is 0.7),
then a new edge is added.

  * •

Student model. For the student model, we adopt the proposed BERM model. We
implement BERM in TensorFlow 2.0 with the high-level Estimator API. For each
input query phrase $Q$ or item title $I$, we split it into several words and
then truncate or pad its length to 10 or 65words (i.e., $l_{Q}=10,l_{I}=65$).
Each word embedding is acquired by the lookup operation on a static vocabulary
table whose total size $n_{w}$ is 39,846. This table is generated by pre-
training two billion search data with the tool of Word2Vec. The size $d$ of
pre-trained embeddings or trained embeddings is 128.

  * •

Training details. We use Lazy-Adam as the optimizer and its learning rate is
0.001. To reduce the overfitting of the training data, we use L2
regularization on each layer of neural networks. For Data-E, we set the
training epoch as 20. For Data-A and Data-S, we set the training epoch as 3.
All results reported are selected according to the best AUC values on the
testing set.

##  Appendix E Additional Experiments

We conduct some additional experiments, including an ablation study (Section
E.1) and sensitivity analysis (Section E.2). In these experiments, we adopt
Data-E and Data-A consistently.

###  E.1 Ablation Study

####  E.1.1 Integration of Embeddings

There are three types of components in the complete BERM: the representation-
based embeddings $\bm{E}^{Q}_{\mathrm{seq}},\bm{E}^{I}_{\mathrm{seq}}$,
interaction-based embedding $\bm{E}_{\mathrm{int}}$, and metapath embeddings
$\bm{E}_{Q-I-Q},\bm{E}_{I-Q-I}$. To further examine the importance of each
component in the final embedding of BERM, we remove one or two components from
it (Eq. 9) at a time and examine how the change affects its overall
performance.

Table 5: Ablation study on Data-E. In each column, the best result is bolded. Models | Data-E
---|---
AUC | F1-score | FNR($\downarrow$)
$\bm{E}^{Q}_{\mathrm{seq}},\bm{E}^{I}_{\mathrm{seq}}$ | 0.8537 | 0.8044 | 0.3560
$\bm{E}_{\mathrm{int}}$ | 0.8595 | 0.8037 | 0.3464
$\bm{E}_{Q-I-Q},\bm{E}_{I-Q-I}$ | 0.8430 | 0.8173 | 0.2995
$\bm{E}^{Q}_{\mathrm{seq}},\bm{E}^{I}_{\mathrm{seq}},\bm{E}_{\mathrm{int}}$ | 0.8638 | 0.8086 | 0.3331
$\bm{E}_{\mathrm{int}},\bm{E}_{Q-I-Q},\bm{E}_{I-Q-I}$ | 0.8761 | 0.8221 | 0.2758
$\bm{E}^{Q}_{\mathrm{seq}},\bm{E}^{I}_{\mathrm{seq}},\bm{E}_{Q-I-Q},\bm{E}_{I-Q-I}$ | 0.8656 | 0.8190 | 0.2922
BERM | 0.8785 | 0.8256 | 0.2966
Table 6: Ablation study on Data-A. In each column, the best result is bolded. Models | Data-A
---|---
AUC | F1-score | FNR($\downarrow$)
$\bm{E}^{Q}_{\mathrm{seq}},\bm{E}^{I}_{\mathrm{seq}}$ | 0.8067 | 0.8660 | 0.3697
$\bm{E}_{\mathrm{int}}$ | 0.8289 | 0.9084 | 0.4459
$\bm{E}_{Q-I-Q},\bm{E}_{I-Q-I}$ | 0.8500 | 0.9114 | 0.4110
$\bm{E}^{Q}_{\mathrm{seq}},\bm{E}^{I}_{\mathrm{seq}},\bm{E}_{\mathrm{int}}$ | 0.8776 | 0.9070 | 0.4743
$\bm{E}_{\mathrm{int}},\bm{E}_{Q-I-Q},\bm{E}_{I-Q-I}$ | 0.8824 | 0.9099 | 0.3705
$\bm{E}^{Q}_{\mathrm{seq}},\bm{E}^{I}_{\mathrm{seq}},\bm{E}_{Q-I-Q},\bm{E}_{I-Q-I}$ | 0.8750 | 0.9094 | 0.3753
BERM | 0.8862 | 0.9107 | 0.3673

The corresponding results on Data-E and Data-A are reported in Tab. 5 and 6.
We have the following empirical observation and analysis:

  * •

In general, the both-component setting outperforms the single-component
setting but is worse than the triple-component setting (i.e., BERM). It
demonstrates that different components in BERM have different positive effects
on the overall performance and they cannot replace each other.

  * •

The introduction of $k$-order relevance modeling can bring stable advancement
to each 0-order relevance model. For example, the combination of “
$\bm{E}^{Q}_{\mathrm{seq}},\bm{E}^{I}_{\mathrm{seq}},\bm{E}_{Q-I-Q},\bm{E}_{I-Q-I}$
” surpasses the combination of “
$\bm{E}^{Q}_{\mathrm{seq}},\bm{E}^{I}_{\mathrm{seq}}$ ” 6.83% according to the
metric of AUC on Data-A. This demonstrates that applying metapath embedding to
relevance matching can make effective use of the neighboring nodes’
information in the user behavior graph.

####  E.1.2 Effect of the Intermediate Node

The metapath defined in BERM includes the intermediate node. To further
investigate the effect of the intermediate node, we compare the performances
of BERM with the intermediate node (i.e., “$Q$-$I$-$Q$” and “$I$-$Q$-$I$”) and
BERM without the intermediate node (i.e., “$Q$-$Q$” and “$I$-$I$”) on Data-E
and Data-A in Fig. 4. We observe that BERM with the intermediate node performs
better than the other one. We infer that the intermediate node has strong
semantic closeness to the anchor node and thus it is helpful to accurate
semantic recognition.

(a) Data-E

(b) Data-A

Figure 4: Effect of the intermediate node. The red (blue) bar represents BERM
with (without) the intermediate node.

###  E.2 Sensitivity Analysis

####  E.2.1 Thresholds $\alpha$ and $\beta$

In BERM, $\alpha$ decides how many edges of the noisy click behavior should be
deleted and $\beta$ decides how many hidden useful edges should be retrieved.
To investigate the sensitivity of $\alpha$ and $\beta$, we conduct experiments
with 16 different hyper-parameter settings where $\alpha$ ranges from 0.2 to
0.5 and $\beta$ ranges from 0.5 to 0.8. We apply the three-order curve
interpolation method to show the final results in Fig. 5. In general, the
results of BERM are robust to the change of hyper-parameter $\alpha$ and
$\beta$ on either Data-E or Data-A. For example, the maximum error of AUC is
no more than 1%. So we conclude that user behaviors play a major role in the
performance of BERM and the knowledge from BERT provides auxiliary effects for
it.

(a) AUC of Data-E

(b) F1-score of Data-E

(c) FNR of Data-E

(d) AUC of Data-A

(e) F1-score of Data-A

(f) FNR of Data-A

Figure 5: Effect of different values of $\alpha$ and $\beta$. (a), (b), and (c) are the results of Data-E; (d), (e), and (f) are the results of Data-A. The red (blue) color corresponds to the high (low) value. Table 7: Effect of different neighbor selection strategies on Data-E. Rate | Click+BERT’s score | Purchase+BERT’s score
---|---|---
AUC | F1-score | FNR($\downarrow)$ | AUC | F1-score | FNR($\downarrow$)
$\lambda=0.0$ | 0.8785 | 0.8256 | 0.2966 | 0.8785 | 0.8256 | 0.2966
$\lambda=0.2$ | 0.8779 | 0.8237 | 0.2834 | 0.8777 | 0.8236 | 0.2810
$\lambda=0.4$ | 0.8770 | 0.8200 | 0.3198 | 0.8756 | 0.8214 | 0.3068
$\lambda=0.6$ | 0.8670 | 0.8085 | 0.4000 | 0.8696 | 0.8045 | 0.3694
$\lambda=0.8$ | 0.8679 | 0.8101 | 0.3901 | 0.8660 | 0.8105 | 0.3262
$\lambda=1.0$ | 0.8656 | 0.8091 | 0.3850 | 0.8671 | 0.8109 | 0.3727
Table 8: Effect of different neighbor selection strategies on Data-A. Rate | Click+BERT’s score | Purchase+BERT’s score
---|---|---
AUC | F1-score | FNR($\downarrow$) | AUC | F1-score | FNR($\downarrow$)
$\lambda=0.0$ | 0.8862 | 0.9107 | 0.3673 | 0.8862 | 0.9107 | 0.3673
$\lambda=0.2$ | 0.8849 | 0.9113 | 0.3794 | 0.8830 | 0.9117 | 0.3831
$\lambda=0.4$ | 0.8821 | 0.9112 | 0.4016 | 0.8818 | 0.9100 | 0.4131
$\lambda=0.6$ | 0.8779 | 0.9068 | 0.4793 | 0.8783 | 0.9064 | 0.4844
$\lambda=0.8$ | 0.8802 | 0.9076 | 0.4676 | 0.8790 | 0.9084 | 0.4683
$\lambda=1.0$ | 0.8792 | 0.9077 | 0.4671 | 0.8787 | 0.9079 | 0.4716

####  E.2.2 Thresholds $k$

Here we evaluate the effects of $k$ on the performance of BERM by sampling
neighboring nodes with different hops from the bipartite graph. The comparison
results on Data-E and Data-S are shown in Fig. 6. We can see that the BERM
with $k$=2 achieves the best performance among them. When $k$ is too large
such as $k=5$, many distant neighbors are aggregated into the anchor nodes,
then it leads to the performance degradation of BERM due to lots of noise
gathering in these distant neighbors. Therefore, we conclude that $2$-order
relevance matching modeling is the optimal choice for our e-commerce scene.

Figure 6: The effects of different $k$.

####  E.2.3 Selection of Neighbor Structure

In BERM, the selection of neighbor structure directly affects which context
information is transmitted to the anchor node. A good selection strategy can
aggregate valuable neighboring node information to enrich the anchor node’s
representation. To investigate the effect of different neighbor structure
selection strategies on BERM and seek a relatively optimal solution, we use
different values of hyper-parameter $\lambda$ to control the ratio between
user behavior and BERT’s score. Specifically, we calculate a new score
$Score_{new}(Q,I)=\lambda*User(Q,I)+(1-\lambda)*Score(Q,I)$ where $User(Q,I)$
is the user behavior feature (e.g., for click behavior, $User(Q,I)=1$ if click
behavior happens between query $Q$ and item $I$). The addition and deletion of
edges refer to $Score_{new}(Q,I)$, rather than $Score_{(}Q,I)$. We report the
results with different $\lambda$ in Tab. 7 and 8. From them, we can conclude
that:

  * •

Using BERT’s score is better than using user behaviors for the selection of
neighbors. Therefore, the value of AUC or F1-score gradually decreases with
the increase of $\lambda$; the value of FNR increases with the increase of
$\lambda$. The optimal $\lambda\in\left[0.0,0.2\right]$.

  * •

According to the metric of FNR, the purchase behavior is better than the click
behavior on Data-E. The reason for it is that purchase behaviors reveal more
accurate semantic relevance information than click behaviors. However, the
purchase behavior is worse than the click behavior on Data-A. We think that it
is caused by the sparsity of purchase behaviors in the dataset of all
categories.