# TPDR: A Novel Two-Step Transformer-based Product and Class Description Match
and Retrieval Method

Washington Cunha  Federal University of Minas GeraisBrazil
<EMAIL_ADDRESS>,  Celso França  Federal University of Minas
GeraisBrazil<EMAIL_ADDRESS>,  Leonardo Rocha  Federal University of
São João del ReiBrazil<EMAIL_ADDRESS>and  Marcos André Gonçalves
Federal University of Minas GeraisBrazil<EMAIL_ADDRESS>
###### Abstract.

There is a niche of companies responsible for intermediating the purchase of
large batches of varied products for other companies, for which the main
challenge is to perform product description standardization, i.e., matching an
item described by a client with a product described in a catalog. The problem
is complex since the client’s product description may be: (1) potentially
noisy; (2) short and uninformative; (e.g., missing information about model and
size); and (3) cross-language. In this paper, we formalize this problem as a
ranking task: given an initial client product specification (query), return
the most appropriate standardized descriptions (response). Traditional ranking
strategies are difficult to be applied to this problem since there is a single
relevant catalog item for every client description. Traditional IR techniques
based only on syntactic matching are not enough since the initial client
specification (IS) may be very different from the standardized description
(SD). Furthermore, syntactic matchings usually cannot focus on the essential
parts of the product descriptions and cannot exploit the correlations among
these parts. Thus, we propose TPDR, a two-step Transformer-based Product and
Class Description Retrieval method that is able to explore the semantic
correspondence between IS and SD, by exploiting attention mechanisms and
contrastive learning. First, TPDR employs the transformers as two encoders
sharing the embedding vector space: one for encoding the IS and another for
the SD, in which corresponding pairs (IS, SD) must be close in the vector
space. Closeness is further enforced by a contrastive learning mechanism
leveraging a specialized loss function. TPDR also exploits a (second) re-
ranking step based on syntactic features that are very important for the exact
matching (model, dimension) of certain products that may have been neglected
by the transformers. To evaluate our proposal, we consider 11 datasets from a
real company, covering different application contexts. Our solution was able
to retrieve the correct standardized product before the $5^{th}$ ranking
position in 71% of the cases and its correct category in the first position in
80% of the situations. Moreover, the effectiveness gains over purely syntactic
or semantic baselines reach up to 3.7 times, solving cases that none of the
approaches in isolation can do by themselves.

Product Matching, Ranking, Transformer Neural Networks

††conference: 29th ACM SIGKDD Conference on Knowledge Discovery and Data
Mining; August 06–15, 2023; Long Beach, CA††price: 15.00††isbn:
978-1-4503-XXXX-X/18/06

##  1\. Introduction

Large companies that need to buy large batches of varied products commonly
hire intermediary companies, such as Astrein111https://www.astrein.com.br/ – a
highlight of the branch in Brazil, to perform a product description
standardization process in order to avoid delays and financial losses in case
of purchase of wrong products. This standardization process consists of
matching, potentially noisy and imprecise client product descriptions (initial
specification - IS) lists with fine-grained and standardized detailed product
descriptions (standardized description - SD) that precisely describe the
desired list of purchases, usually compiled in product catalogs.

To better illustrate the problem, let us consider the following example
scenario to understand how these “matching” companies operate. A construction
company X will build a research complex. Company X will be responsible for
constructing the complex’s buildings and assembling all its laboratories. This
company will need an extensive diversity of products, from construction
materials such as cement, bricks, structural metals, and screws to office
supplies and, potentially, specific research laboratory materials, such as
computers with specialized hardware, for example. To make the execution of the
project feasible, company X obtains from its employees’ several purchase
lists. These lists are compiled by company X´s managers and employees, who
will ultimately perform their research activities in the new facilities. The
company X hires a company Y (specialized in intermediating purchases), which,
in turn, has the product catalogs of several potential suppliers of the
necessary materials. Before placing the orders themselves, company Y needs to
standardize all the specified items, and only after this process is finalized,
it consults the materials with the suppliers. Finally, company Y chooses the
best suppliers in order to reduce the total cost and delivery time of
purchases. This intermediation is advantageous since other companies can also
present similar orders, allowing the purchase in large batches and obtaining
product discounts (cost reduction) and special delivery conditions (time
reduction). On the other hand, a large purchase of wrong products due to
inconsistencies and ambiguities in the initial (client) product list may cause
all types of delays and financial problems.

The most considerable technical challenge in this practical problem is the
vocabulary mismatch between the usually noisy, imprecise, incomplete, and/or
ambiguous initial (client) specification (IS) and the standardized description
(SD) detailed in a product’s catalog, meaning that the matching task is not an
easy one. This challenge refers to a common phenomenon of human communication
in which clients who write the description of the product to be purchased and
people who built the standardized catalogs of product descriptions likely used
different or inconsistent vocabularies with different levels of details. We
further elaborate on this problem next.

##### Industrial Problem Definition:

The main challenge for these purchasing intermediation companies is the
product description standardization stage. Let us consider the description
$IS-X_{i}$ made by a customer, for example, company $X$. Company $Y$ must be
able to retrieve the standardized description $SD-X_{i}$ from its product
catalog, consisting of thousands or even millions of descriptions. In general,
customer purchase product specifications are built by putting together several
sub-lists that, in turn, are written by people with different backgrounds,
with varying levels (or lack thereof) of detail, creating several challenges
to be overcome, among them: (1) potentially noisy data; (2) short and
uninformative descriptions (missing information about model and size, for
example); and (3) multi-language descriptions. We illustrate this problem in
Table 1 with real examples experienced by a purchases intermediation company.

Table 1. Examples of the product description standardization.

##### Failure Implications:

Minor errors in retrieving standardized items can lead to significant
financial losses and severe accounting errors. Errors in the purchasing
process lead to a decrease in the agility of the process as a whole. When a
product is delivered and, during the inspection, it turns out to be a wrong
product, the product exchange is estimated to be at least three times more
time-consuming than the standard delivery process (two deliveries and one
return process), according to Astrein. In addition to being a total waste of
resources, it can take a long time to resolve it appropriately.

##### Main Goal:

The matching process we described above can be done manually searching for
keywords, in a very time-consuming and still prone-to-error process. Our goal
here is to help to automate the process as much as possible by addressing two
distinct issues in an end-to-end approach: (i) association of customer-
supplied material description –IS– to a previously annotated Description
Pattern (DP)222A Description Pattern is equivalent to a category or class of
similar products, e.g., a metal tube or valve, shown in the third column of
Table 1; and (ii) Matching the client description with a previously
standardized item –SD– in a product catalog.

##### Summary of our proposal:

Our proposed solution to the product standardization problem faces the problem
as an Information Retrieval task, more explicitly, a ranking task: given an
initial specification (query), return, from the product catalog, the most
appropriate standardized descriptions (response), as well as their associated
DPs. Accordingly, in this paper, we propose TPDR – a novel Transformer-based
Product and Class Description Retrieval method. Traditional ranking strategies
are difficult to be applied to this problem since there is a single relevant
SD for every single IS, if any exists (the product may not have been
previously cataloged). Applying traditional IR techniques based on the
syntactic descriptions of the products is also not enough, as a customer can
present a product description very different from the SD. For example, in
Table 1, we have the example IS as “brass ring 001 3/16” and the corresponding
SD is “penetration ring - brass - 3/16” ”. Moreover, syntactic matchings are
not able to focus on the most important parts of the product descriptions
(which may be different depending on the product). They cannot exploit
important correlations of these parts (e.g., product type, material, and
size). In this sense, it is necessary to adopt ranking strategies that also
considerate the semantic correspondence between ISs and SDs, as our proposal.

Accordingly, TPDR’s first step is to employ transformers as two encoders
sharing the embedding vector space: one for encoding the initial specification
(query) and another for the standardized descriptions (response). The
transformers exploit attention as well as constrastive learning mechanisms
that allow focusing on the most important parts of both product descriptions
(IS and SD) for the sake of the matching as well as relevant correlations
among the products´ parts. Figure 1 represents this stage of the proposed
model. Given a pair (query, response), we aim for both to be close in the
shared vector space. In the example of Figure 1, given the initial
specification “stopper national 455165” and the standardized description
“retainer 45 nbr 70sha bid 85,73x114x3x11,9mm”, our model needs to be able to
encode them so that both representations are close in the vector space of
their representations (Figure 1). To further enforce closeness between both
the client and the corresponding standardized representation, we leverage a
contrastive learning mechanism (Hadsell et al., 2006) that uses an adaptation
of the N-Pair loss function(Sohn, 2016). In the retrieval/matching phase, our
approach helps, given an initial client description, to rank the respective SD
and associated DP among the first positions.

Figure 1. Proposed Model

For doing so, we propose a novel training strategy at training time in which
the transformer encoders train cooperatively, optimizing their parameters at
alternating training steps mirroring a two-player turn-based game. We call
this mechanism TaG-Training (Turn-based Bargaining Game Training), as detailed
in Section 7. Our results revealed that tag training ensures a more accurate
and fine-grained optimization. Efficiency is also improved from the
optimization perspective by leveraging a learning-based loss function (Bellet
et al., 2015; Huai et al., 2019; KAYA and BİLGE, 2019): our encoders aim to
learn nonlinear embedding representations of the input languages that place
similar data points close while positioning dissimilar ones far on the
embedding space (Zhu et al., 2020; Sohn, 2016). Previous work has exploited
either contrastive loss (Hadsell et al., 2006) or triplet loss (Hoffer and
Ailon, 2015) that suffer from slow convergence, partially because they exploit
only one negative example at each update step. Differently, within TPDR, we
propose to take advantage of an adapted N-pair loss function, which
generalizes triplet loss by allowing joint comparisons among more than one
negative example. For ranking tasks such as product retrieval, $N$-pair loss
is more effective than other ranking losses, as demonstrated in our
experiments because a single update step pulls up relevant products for its
query while pushing down $(N-1)$ non-relevant products. The number of non-
relevant products for a given query usually greatly surpasses the relevant
ones, making a significant difference.

After generating this initial ranking, we proposed an additional step: re-
ranking. Unlike traditional two-phase (ranking and re-ranking) IR approaches,
where an initial ranking is first generated from a ”weak” approach and, then,
is applied a re-ranking based on a neural approach (Petrov and Makarov, 2021),
our approach performs the inverse process. Our approach initially generates
the initial ranking based on a strong neural network process (described
above). This first step reduces the number of candidates considerably. After
that, we apply a re-ranking step based on syntactic features that are very
important for the exact matching (model, dimension, size) of certain products
but may have been neglected by the transformers. In this way, our solution
innovates by exploiting both aspects: semantics and syntax. We propose to
consider as the first metric the similarity value of the contextual model
(without re-ranking step), that is, the weighting of the output of the
previous prediction phase. Note that, given the various advantages arising
from the use of the proposed contextual model, in order to enable improvements
in terms of effectiveness, it is essential to reuse the weighting of the pre-
stage model of re-ranking. Furthermore, we consider three additional syntactic
metrics: (1) TFIDF Cosine Similarity; (2) Jaccard Textual bigrams Comparison;
and (3) to further improve the results, the BM25 score.

To evaluate our proposal, we consider three real domains – consisting of 11
datasets – from the Astrein company, covering different application contexts
and dataset sizes. As a result, considering a general application context
covered by the first nine distinct datasets (Section 4.5.1), with a product
catalog (search index) consisting of almost 500 thousand products, the
developed solution was able to retrieve the actual final product description
before the $5^{th}$ ranking position in 71% of the cases (on average).
Furthermore, we observe that in 85% of the cases, the respective standardized
product is retrieved before the $100^{th}$ position of the rank. In cases
where we could not retrieve the standardized product in the first positions of
the ranking, the correct association of DPs is of paramount importance since
this process facilitates the manual search of the real item constituted only
by products of the same DP. In fact, the proposed solution can retrieve the
correct DP in the first ranking position for 80% of queries (in a search
universe consisting of 3972 distinct DPs). Furthermore, the correct associated
DP is contained up to the $5^{th}$ ranking position in 88% of queries,
representing an excellent practical result in the presented domain.

Finally, in the other domains (D10 and D11 datasets), both consisting of
products associated with a single DP, namely Orthoses, Prostheses and Medical
Materials - OPMM and Medicines, the results confirm the effectiveness of the
proposed model and demonstrate its flexibility. More than 70% of standardized
products are present up to the $5^{th}$ ranking position for both scenarios.
We also evaluate each step of our proposal individually, demonstrating the
complementary between syntactic and semantic information to properly deal with
the product standardization problem.

##  2\. Background

###  2.1. Self-Attention

Self-attention is a mechanism that allows finding meaningful words in a
sentence given a query word in the same sentence (hence the ”self”). The
objective is to capture those words that contribute the most to the semantics
of the query word. Mathematically, self-attention is simply a function that
receives a sequence $X$ of $N$ vectors $x_{i}\in R^{d}$ as input and returns
as output another sequence $Y$ of $N$ vectors $y_{i}\in R^{d}$, where each
vector in $Y$ is simply a weighted average (the weights are called attention
scores) of the vectors in $X$.

Figure 2. Architecture of the self-attention deriving key, query, and value
representations from input X, calculating attention scores and outputting the
final representation Y.

As shown in Figure 2, from the input sequence $X$, distinct feed-forward
neural networks output Q (Query), K (Key), and $V$ (Value) matrices, and the
attention scores are calculated as defined in Equation 1. Finally, the output
$Y$ is obtained by multiplying the attention scores by matrix $V$

(1) $Attention(Q,K,V)=softmax\left(\frac{QK^{T}}{\sqrt{d_{k}}}\right)V$

where the scaling factor $d_{k}$ is the dimension of vectors in $K$.

###  2.2. Multi-Head Self-Attention

Multi-head self-attention involves applying different attention mechanisms to
the same input sequence $X$. Each head can extract different characteristics
from the input sequence. In Figure 1, for instance, when using multi-head
self-attention to represent the fine-grained retainer specification, while one
head focuses on the retainer word, another head can focus on the words defined
by product specification, such as length (83.73x114.3x11.9mm) and material
(70sha). The model can capture multiple patterns over both product
specifications by leveraging multi-head self-attention. The independent self-
attention outputs are then concatenated and linearly transformed to generate
the $Y$ output.

###  2.3. Transformer’s Encoder

Proposed by Vaswani et al. (Vaswani et al., 2017), the transformer emerges as
an answer to the CNNs and RNNs drawbacks. Specifically, while RNNs do not
allow parallelization during training because of their sequential nature and
suffer from vanishing gradients, when using CNNs it is challenging to find
relevant patterns in distant positions within the input sequence (Vaswani et
al., 2017).

Transformer dispenses recurrence and convolutions entirely and improves the
sequence representation by adopting only the self-attention mechanism. Bert
(Devlin et al., 2019), GPT (Brown et al., 2020), RoBERTa (Liu et al., 2019),
and other transformer-based models have been the state-of-the-art for a myriad
of Natural Language Understanding tasks such as question-answering, named
entity recognition, text classification, and other (Cunha et al., 2021; Garg
et al., 2020; Cunha et al., 2023b).

The transformer follows the architecture of $N$ identical blocks composed of
two layers: a multi-head self-attention mechanism and a simple position-wise
fully connected feed-forward network, as shown in Figure 3. There is also a
residual connection (He et al., 2016) around each layer, followed by a
normalization layer.

Figure 3. The transformer’s encoder architecture.

Before forwarding through the $N$ blocks, each sequence token is passed
through a learned embedding layer. Next, as the meaning of a word depends on
its place in the sentence, its embedding should encode its position. To this
end, the positional encoding layer adds numbers between $\left[-1,1\right]$
using non-learned sine and cosine functions to the embedding tokens.

##  3\. The Proposed Approach

Figure 4 shows the TPDR architecture in which, given a query (i.e., initial
client description), we first use an embedding-based retrieval system to
retrieve a reduced list (when compared to the whole catalog) of potentially
relevant standardized products based on semantic matching. As we observed that
in practice, most relevant products could not be retrieved at the top of the
ranking due to syntactic mismatches, we use a term-based retrieval to re-rank
the (query product) candidate pairs in the second stage to output the final
product ranking that better matches specific characteristics of the desired
product (e.g., model, dimensions).

Figure 4. The overall architecture of TPDR.

###  3.1. Embedding-based Retrieval

We employ dual transformer encoders (Section 2.3) for our Embedding-based
semantic Retrieval. These encoders work as a logical framework to represent
the queries, and product descriptions in the same embedding space as Figure 5
illustrate. At indexing time, our search engine indexes the product catalog
description collection encoded by the Product Encoder. At the searching time,
the Query Encoder should only represent a single query. Then, the closest $k$
standardized products concerning the query are selected from the collection
using a similarity search method.

We also optimize both encoders alternately, Query Encoder’s parameters in one
training step and the Product Encoder’s parameters in the next step. We call
this mechanism TaG-Training (as explained in Section 3.4). It resembles a two-
player turn-based game where the combined action of the players contributes to
optimizing the payoff (or minimizing the loss function) (Maschler et al.,
2020). Conveniently, the training exploits the $N$-pair loss function that is
suitable for the product retrieval task, as explained in Section 3.3.

Figure 5. The overall architecture of TPDR.

Each TPDR encoder is a transformer with $12$-layers, generates a
$768$-dimensional vector representation for each token from the input
language, uses a $12$-head self-attention, and contains $110$M parameters. As
input, each encoder receives a token sequence resulting from the byte-pair
encoding (BPE) (Gage, 1994; Sennrich et al., 2016) of the respective input
languages. The transformer returns a sequence of hidden states from the
model’s last layer as output. Finally, an average pooling over the last hidden
states generates a single $768$-dimensional dense vector representing the
input language.

To complete our end-to-end product search engine, the fine-tuned Product
Encoder is employed to generate product embeddings from the collection. These
embeddings are indexed by leveraging Hierarchical Navigable Small World graphs
(Malkov and Yashunin, 2020) that offer a better logarithmic complexity scaling
and allow an efficient nearest neighbor search. Finally, a query is answered
by generating a query embedding (using Query Encoder) and returning its
nearest $k$ products.

###  3.2. Term-Based Retrieval

Our main motivation for proposing a re-ranking strategy was that in our
preliminary experiments, although $85\%$ of the final product descriptions
were present up to the $100^{th}$ ranking position, in several cases, relevant
products were not retrieved in the top positions of the ranking. This is due
to the existence of several similar products in the catalog and a ´´syntactic
mismatch´´ between the client description and relevant standardized
description regarding specific characteristics such as the model or dimensions
of the product that is better captured by a syntactic match. Thus, from the
list of candidate products received from the Embedding-based retrieval, we use
a Term-based retrieval to re-rank the product query pairs and generate the
final product ranking in the second step, aiming to solve the aforementioned
issues.

In more detail, our Term-based retrieval consists of a linear combination of
pre-fixed scores to improve the retrieval results. Specifically, we considered
four ranking functions: the Embedding-base Retrieval ($s_{1}$) score (from the
previous step), cosine similarity ($s_{2}$), Jaccard similarity ($s_{3}$), and
BM25 ($s_{4}$) score.

#####  Cosine Similarity (Tan et al., 2005):

we first leverage bag-of-words representation based on TF-IDF weighting-scheme
for representing query and product as a sparse vector, and then we measure the
cosine similarity of them using the Equation 2.

(2)
$s_{2}(\vv{q}{,}\vv{p})=\frac{\vv{q}.\vv{p}}{\mid\mid\vv{q}\mid\mid.\mid\mid\vv{p}\mid\mid}$

#####  Jaccard Similarity:

we first convert the query and product description into two sets of bigrams,
then measure the similarity between two sets: capturing which members are
shared and which are distinct. Finally, we calculate the Jaccard similarity by
dividing the size of the intersection divided by the size of the union of two
sets, as shown in Equation 3.

(3) $~{}s_{3}(Q,P)=\frac{Q\cap P}{Q\cup P}$

#####  BM25 (Jonesa et al., 2000)($s_{4}$):

similar to the cosine similarity, we leverage the TF-IDF to measure BM25 (Eq.
8) to score each pair query product.

##### Final Weighting

The final ranking score $S$, (Equation 4), corresponds to the linear
combination of the previous ranking functions.

(4) $S(Q,D)=\frac{[3*s_{1}(Q,P)+s_{2}(Q,D)+s_{3}(Q,D)+s_{4}(Q,D)]}{6}$

Note that, as the results of the contextual Embedding model were already
considerably effective, we weighted them with larger importance in the final
model (3 times higher than the syntactic ones). We leave for future work to
adjust the weights of each component for each dataset.

###  3.3. Learning Objective

A contrastive learning method is particularly suitable for IR tasks because it
pulls closer entities with similar semantics, such as a developer’s query and
its relevant product, and pushes away others with distinct semantics. To
optimize the encoders’ parameters, we leverage an adaptation of $N$-pair
(Sohn, 2016), a loss function that demonstrates superiority for a variety of
IR tasks, including fine-grained object recognition (Xie et al., 2015), image
clustering (Song et al., 2016), and face identification (Schroff et al.,
2015).

More formally, consider $N$ training samples
$\\{(q_{1},c_{1}),...(q_{N},c_{N})\\}$, where $q_{i}$ is a query in natural
language perfectly answered by the product $c_{i}$. The core contribution of
the $N$-pair loss described in Equation 5, is to optimize the embedding
functions $f_{i}=f(q_{i};\theta)$ and $g_{i}=g(c_{i};\sigma)$ to be able to
map $q_{i}$ and the corresponding $c_{i}$ into dense vectors where the
similarity $sim(f_{i},g_{i})$ is greater than the similarity
$sim(f_{i},g_{j})$; $g_{j}$ being the embedding of any other product not
answering $q_{i}$. That is, at the same step that a relevant product is placed
closer to a query, it is also pushed further away from (N-1) non-relevant
products.

(5)
$\mathcal{L}(q_{i},c_{i},\\{c_{j}\\})=-log\frac{exp(f_{i}^{T}g_{i})}{exp(f_{i}^{T}g_{i})+\sum_{j=1}^{N-1}exp(f_{i}^{T}g_{j})}$

where $c_{i}$ is a relevant product with respect to $q_{i}$ and all other
products $c_{j}$ (with $i\neq j$) are non-relevant ones.

Other ranking losses such as contrastive loss (Hadsell et al., 2006) and
triplet loss (Hoffer and Ailon, 2015), widely used in product retrieval,
employ only one negative example, not interacting with the other negative
samples in each update step, as shown in Figure 6. Vertices are
representations of queries and products, and the edges express the distance.
The triplet loss pulls the positive example while pushing one negative example
at a time. On the other hand, the $N$-pair loss pushes $(N-1)$ negative
examples all at once. Reaching the same effect would require an inefficient
batch construction strategy resulting in $N(N-1)$ pairs when using triplet
loss.

Figure 6. Embeddings of a model trained to satisfy the constraints of triplet
loss (bottom) and $N$-pair loss (top).

$N$-pair loss is very suitable for the product retrieval task, characterized
by a few (or a single) relevant items in a collection of numerous irrelevant
ones. It also converges faster, uses an efficient batch construction strategy,
and the use of multiple negative examples enables the encoders to generate
discriminative representations.

###  3.4. TaG-Training

Unlike in Generative Adversarial Networks (GANs) (Goodfellow et al., 2014),
where the training corresponds to a minimax two-player game, the training of
the Product and Query Encoders mirrors a two-player turn-based bargaining game
in which the joint actions players take result in collective payoffs.

Specifically, the optimization of the Query Encoder causes it to move the
noisier natural language’s representation closer to the standardized natural
language representation. In the same optimization step, the Product Encoder
moves the product representation closer to the region in the space where the
query representation lies before optimization. Then, updating both encoders at
the same training step may not be the best solution as the model’s parameters
may oscillate and have sub-optimal convergence.

Employing an optimizer for each encoder promotes a more refined parameter
update. We propose that updates occur in turns, with one encoder per
optimization step. Therefore, each encoder behaves like a player, playing
alternate turns to minimize the loss function. We call these mechanisms TaG-
Training (Turn-based Bargaining Game Training).

Figure 7 shows an illustrative example of three successive optimizing steps of
the two encoders’ parameters without and with tag-training. Both approaches
have the same initial configuration in which we have a product embedding $c$
and a query embedding $q$ in a two-dimensional space. The loss emphasizes the
distance between these two embeddings, and the arrow represents the gradient.

Figure 7. Three successive optimizing steps of the encoder’s parameters
without and with tag-training.

Without tag-training, the optimizer updates both encoders’ parameters at all
training steps. In step $1$, the Product Encoder’s optimization (opt1) drives
$c$ to the surroundings of $q$. A position miss-match occurs because, in the
same step, the Query Encoder’s optimization (opt2) moves $q$ to be closer to a
position where $c$ was. The same problem occurs in step $2$ and in the
following steps, where $c$ and $q$ orbit each other before converging.

With tag-training, optimizing the encoder’s parameters occurs in alternating
steps. The initial position of $c$ and $q$ in step $0$ is the same as in the
previous scenario. The difference occurs in step $1$ where only the Query
Encoder is optimized (opt1), moving $q$ to a position closer to $c$, which
remains unchanged. In the last step, the Product Encoder’s parameters are
optimized (opt2), placing $c$ nearby to $q$.

TaG-training can potentially improve effectiveness, yet its direct impact is
on efficiency. Leveraging tag-training not only allows a fine-grained joint
optimization of each encoder but is also responsible for reducing the amount
of the encoder’s parameter updates and, ultimately, the overall training time.
Alternating between both encoders in the optimization step, TPDR has an equal
training time if it had used a single transformer. Regarding effectiveness,
the joint optimization with fewer parameter updates may help avoid
overfitting, contributing to the model’s generalization.

###  3.5. TaG-training as a Two-player Bargaining Game

Let $G=(S,d)$ be a two-player bargaining game where (Maschler et al., 2020):

  * •

$S\subseteq{\rm I\\!R}^{2}$ is a nonempty, compact, and convex set called the
set of alternatives.

  * •

$d=(d_{1},d_{2})\in S$ is called the disagreement point (or conflict point).

  * •

There exists an alternative $x=(x_{1},x_{2})\in S$ satisfying $x>d$ (e.g.,
$x_{1}>d_{1}$ and $x_{2}>d_{2}$)

We interpret the tag-training as a bargaining game where players Query Encoder
and Product Encoder need to agree on an alternative $x=(x_{1},x_{2})\in S$. If
they come to such an agreement, Query Encoder’s payoff is $x_{1}$, and Product
Encoder’s payoff is $x_{2}$. If the players cannot agree, the game’s outcome
is $d=(0,0)$ (the encoders’ payoff is $d_{1}$ and $d_{2}$, respectively).

The utility function for the Query Encoder (and the Product Encoder, by
symmetry) defined in Equation 6 measures the proportion of query $q_{i}$ in
which the relevant product $c_{i}$ could be found up to the $k$ ranking
position. By taking $k=1$ we are interested in the proportion of queries where
$c_{i}$ is closest to $q_{i}$ for all $i\in\\{1,2,...N\\}$. Therefore, the set
of possible agreements is $S=\\{(x,x):0\leq x\leq 1\\}$, and the vector of
disagreement is $d=(0,0)$.

(6) $Recall@k=\frac{1}{|N|}\sum_{i=1}^{|N|}\sigma(p_{i})$

where $p_{i}$ is the position $c_{i}$ in the ranking of the closest points
relative to a query $q_{i}$ and $\sigma$ function returns $1$ if $p_{i}\leq
k$, otherwise it returns $0$.

Equation 7 defines the point $N(S,d)$ called the Nash agreement point (or the
Nash solution, first proposed by John Nash in 1953) (Nash, 1953). For $G$, the
alternative $x=(1,1)\in S$ is the Nash solution.

(7) $argmax_{x\in S,x\geq d}(x_{1}-d_{1})(x_{2}-d_{2})$

From an optimization perspective, the global minimum $M^{*}$ of the $N$-pair
loss occurs when $f_{i}^{T}g_{j}$ becomes infinitely small
($f_{i}^{T}g_{j}\to-\infty$): the distance between a query and any non-
relevant product tends to infinity. Although this theoretical result cannot be
achieved in practice, the purpose of $N$-pair loss is to approximate a query
$q_{i}$ to its relevant product $c_{i}$ while pushing away the non-relevant
products $c_{j}$ for all $i,j\in\\{1,2,...,N\\}$ and $i\neq j$.

Therefore, any configuration where the query $q_{i}$ is closer to $c_{i}$ than
any other points $c_{j}$ constitutes a solution $M$ to the product retrieval
task that can be achieved via gradient descent. Since $M$ causes $x=(1,1)\in
S$, it is a Nash agreement point for $G$.

##  4\. Experiments

In this section, we will present the results obtained from the instantiation
of our proposal. All experiments were performed using a machine with the
following hardware configuration: 64GB of RAM, 16 vCPUs, and 1 NVIDIA 3090 GPU
with 24GB.

###  4.1. Data

As previously mentioned, the proposed model receives a set of pairs as input
(query, response). In our domain, we represent pairs (Initial Customer
Specification – Query, Standardized Description - Product). In addition, each
product is associated with a class named Description Pattern - DP, which is
equivalent to a class or category of products. As mentioned, the correct
association of this DP is important because, in cases where we cannot retrieve
the real products in the first positions of the ranking, this process
facilitates the manual search of the real item consisting only of products
from the same DP. Next, we describe the data inputs generated for the training
of the proposed algorithm.

The first set of datasets (D1–D9) – general purposes – consists of matching
items (query and product) of different dimensions333The amount of data in each
dataset is described in parentheses. For instance, D1 is composed of 20,707
(query, product) pairs. referring to diverse companies. This dataset is
associated with 3,972 distinct DPs. This set of datasets is associated with a
shared search index (catalog) composed of 441,223 different products.

The D10 dataset consists of 21,225 pairs referring to a single DP - OPME -
Ortheses, Prostheses, and Medical Materials and has a search index of 1
million different products. Thus, even though being associated with only one
DP domain, this dataset is constituted of a few training pairs compared with
the product catalog size. Indeed, the search space is considerably larger than
the previous case, making it a more challenging scenario. Finally, the D11
dataset consists of 2 thousand pairs (query and product) referring to a single
DP - Medicines. As a search index, we adopted a product catalog containing
approximately 29,000 different products.

Following the scientific methodology present in (Lu et al., 2021; LeClair and
McMillan, 2019), we randomly divided the dataset pairs into 80% for the
training phase; 10% for the validation phase (used for the calibration process
and parameter refinement); and 10% for the test phase.

###  4.2. Evaluation Metrics

In our results analysis, we considered three traditional retrieve metrics: (i)
Mean Reciprocal Rank (MRR) (Aslanyan et al., 2020) required to assess in what
ranking position the relevant products tend to be placed and want that item to
be at a higher position (MRR is very informative since we have one relevant
product per query); w(ii) Normalized Discounted Cumulative Gain (nDCG)
(Karmaker Santu et al., 2017) which penalize highly relevant products when
appearing lower in ranking; and(ii) Recall (Li et al., 2022) to measures the
proportion of relevant products retrieved.

Therefore, given a pair (Initial Customer Description, Standardized
Description), Initial Customer Description corresponds to a query in which we
consider the Standardized Description as the only relevant result – a
challenging scenario compared to traditional search situations, as we only
have a single relevant result per query.

###  4.3. Proposed Model Instantiation

As mentioned, the proposed model is based on a BERT architecture for encoding
descriptions via both encoders (query and product). Considering the validation
set, we perform a preliminary set of model refinement experiments by varying
the set of the model´s hyperparameters. The best variation of hyperparameters
according to these preliminary experimentations of the proposed model is
described below:

  * •

Max number of training epochs: 10

  * •

Client Initial Description max length: 64 terms

  * •

Standardized Product Description Max length: 64 terms

  * •

Batch size: 32

  * •

Pre-trained Embedding: sentence-transformers/LaBSE

  * •

Re-Ranking: i. Prediction Phase Score; ii. Cosine similarity; iii. Jaccard
Comparison of bigrams; iv. BM25.

Given that both descriptions (initial and standardized) are mostly written in
Portuguese and English, we propose the use of LaBSE (Feng et al., 2020)
embeddings for two main reasons: i. the results of pre-training with this
embedding model were considerably higher than those obtained with alternatives
such as BERTimbau (Souza et al., 2020) and MUSE (Yang et al., 2020); and ii.
this particular type of embedding can capture term relationships at a multi-
lingual level. As we will show below, the use of this model can provide higher
effectiveness and deal with cases of the same word written in different
languages (e.g., paper and papel in Portuguese).

Finally, at the description indexing phase, we opted for using the exact
solution (brute_force method) in order to optimize the generation of indexes
generated by the HNSW - nmslib (Malkov and Yashunin, 2020) library. The use of
this method provides considerable improvements in terms of effectiveness
(MRR). Preliminary results varying the set of hyperparameters showed that
brute_force is up to 89% more effective than the approximate method hnsw in
the considered product retrieval context. However, when effectiveness is
prioritized, efficiency may be penalized. Despite that, considering that the
validation set had 10K queries, the brute_force method represented an increase
of only 15 minutes in the description retrieval step. Therefore, we believe
using the exact solution provides a better tradeoff between effectiveness and
efficiency.

###  4.4. Baselines

####  4.4.1. Classical Syntactic Search Engine

As a syntactic baseline, we rely on a traditional probabilistic model, which
estimates the probability that a product will be relevant to a query. We apply
$TF\text{-}IDF$ as a weighting scheme, a logical framework for representing
documents and queries, and BM25 as a ranking function. BM25 is a family of
ranking functions that scores a set of products based on relative proximity
between the query within product description terms. Eq. 8 shows one of the
most prominent instantiations of the BM25 given a query q containing keywords
$q_{1},...,q_{n}$ and a product $p$.

(8)
$sim_{BM25}(q,p)=\sum_{i=1}^{n}TF\text{-}IDF(q_{i}).\frac{f(q_{i},p)(k_{1}+1)}{f(q_{i},p)+k_{1}(1-b+b\frac{|D|}{\overline{d}})}$

where $f(q_{i},p)$ is the $q_{i}$ frequency in d, $\frac{|D|}{\overline{d}}$
is the relative product description length over the average product
description length in the collection. $k_{1}$ and $b$ are free parameters,
usually chosen, in the absence of optimization, $1$ and $0.75$, respectively.
Finally, $TF\text{-}IDF(q_{i})$ is the $TF\text{-}IDF$ weight of the query
term $q_{i}$.

####  4.4.2. Pure Semantic Search Engine

Classical search engines treat product descriptions in the same way
traditional search engines deal with natural language documents. In the face
of the vocabulary mismatch (Gysel et al., 2018), they are unable to match the
user’s needs with the relevant products. Recent studies have exploited models
that learn to encode both inputs into a shared vector space. Ideally, relevant
product vectors will be closer to the respective query vectors (while the
irrelevant ones will be farther away). Therefore we defined the TPDR (noRR) –
our proposal without the re-ranking step –, fine-tuning a LaBSE transformer
encoder to represent queries and products with the same vector space, and we
adopted the same parameters used in our approach. TPDR (noRR) also generates
product embeddings from the product collection and indexes them using HNSW
library. Finally, a query is answered by generating by encoding a query and
returning its nearest $k$ products.

###  4.5. Experimental Results

####  4.5.1. Effectiveness Analysis of the client description match with a
previously standardized item in a product catalog

In Table 2, we present the effectiveness results of our approach using the
MRR, NDCG, and Recall metrics. A first important observation is that the
purely semantic approach, which corresponds to the first step of our method –
TPDR (noRR)) – in almost all cases, considering all datasets and metrics, with
a few exceptions (e.g., D7 and D9) is better, sometimes by large margins (up
to 6x as in D2), than the purely syntactic approach (BM25), demonstrating the
importance of our innovations regarding the semantic aspects of the problem.

Table 2. Effectiveness – MRR, NDCG and Recall metrics.

Regarding our complete proposal TPDR, MRR@1, for instance, shows that
considering the mean of the queries of all datasets, 64% of the standardized
descriptions present in the test datasets are retrieved, on average, in the
first position – varying between 11% in (D9), which has the lowest number of
samples and 74% in (D1), the largest dataset of the first scenario (several
DPs). This behavior is evidence that the number of pairs for training has a
direct impact on the effectiveness of the proposed model. Furthermore, MRR@5
and MRR@10 results show that the actual standardized descriptions retrieved by
our approach are present, on average, between the first and second positions
for all queries444An average MRR value of 0.5 is produced as on average the
actual result appears in the second position, i.e., $\frac{1}{rank=2}$.
Finally, still considering the MRR metric (in variations $@1$, $@5$ and
$@10$), our proposed model was able to produce average improvements of 3.76
times – ranging from 1.01 times to 9.8 times – when compared to the pure
syntactic approach (BM25). And when compared to the pure semantic approach
(TPDR (noRR)), the complete TPDR instantiation produced average improvements
of 2.3 times, demonstrating the importance of the syntactic-based re-ranking.

TPDR is also very effective when considering nDCG and Recall. In the most
challenging scenario (the first ranking position), nDCG@1 ranges from $0.16$
in the $D8$ dataset to $0.74$ in $D1$ (and it is above $0.41$ in half of the
datasets). Besides, TPDR presents an increasing $nDCG$ when inspecting higher
ranking positions, demonstrating that the relevant items, when not retrieved
immediately in the 1st or 2nd position of the rank, are usually placed up to
10th position. The $recall@10$ of $0.88$ on the largest dataset demonstrates
TPDR’s ability to represent and retrieve products even when the number of
queries is very large. In the worst scenarios, the $recall@10$ is $0.32$
($D8$), where there exists a large discrepancy between the query and product
description terms, which is emphasized by the very low BM25 effectiveness in
this dataset, that only reaches a $recall@10$ of $0.13$.

Figure 8. Ranking Distribution of Product Description Position

Figure 8 presents the number of standardized descriptions in each retrieved
position for all datasets. Note that most SDs in the ranking distribution are
present up to the 10th ranking position. Indeed, the largest portion of
relevant SDs is found in the first positions of the rank (1st or 2nd). With a
few exceptions, all relevant standardized products can be found by examining
the descriptions up to the 100th position. Considering our proposed model
TPDR, we noticed that 71% of the standardized descriptions are retrieved up to
the 5th position of the ranking and 85% of actual descriptions are up to the
100th position.

When compared to the purely semantic model TPDR (noRR), 44% of the real
products were retrieved up to the 5th position of the ranking with this model,
while 72% of actual descriptions were presented up to the 100th position.
Finally, the purely syntactic model – BM25 – retrieved 22% of the real
products up to the 5th position of the ranking, with only 37% of actual
descriptions present up to the 100th position. In sum, there is a significant
improvement in the results achieved by TPDR that combines both syntactic and
semantic approaches by means of a strong semantic ranker for producing a
reduced and effective candidate list and a syntactic-based re-ranking strategy
to deal with product specificities.

####  4.5.2. Analysis of DP’s Association

In this section, we demonstrate the results regarding the association of DP’s
obtained by the proposed approach and the considered baselines. It is
important to point out that these analyzes were performed considering only
datasets D1 – D9 (whose DP search space is composed of 3972 different DPs)
since datasets D10 and D11 are associated with only one DP, namely Orthoses,
Prostheses, and Medical Materials - OPMM and Medicines.

In the previous section, we noticed that 71% of the real standardized
descriptions are present up to the 5th ranking position. Despite being very
effective, this result still leaves room for improvement as 29% of
standardized descriptions are not retrieved in the top five positions. For
this reason, the Correct Association of DP’s is critical in cases where the
standardized description is not retrieved or is retrieved very far in the
rank, since it facilitates the search (e.g., filtered by DP) for the actual
associated SD.

Considering TPDR, in our evaluation, we observed that, even when the
standardized description is not retrieved at the top of the rank, the correct
DP is present in the $1^{st}$ position in 80.2% of the queries. Furthermore,
the associated correct DP is retrieved up to the 5th ranking position in 88.2%
of queries.

With the semantic model without re-ranking – TPDR (noRR) – the correct DP was
present in the first ranking position in only 61.5% of queries and among the
top five positions in just 74.5% of cases. Finally, the BM25 syntactic model
returned the correct DP in the first position of the ranking in only 38.1% of
queries and among the top five positions in 48.5% of cases. These results
reinforce the need for combining syntactic and semantic information to
adequately deal with the posed problem.

####  4.5.3. Manual Inspection of the Obtained Results

We discuss next some qualitative results of our proposed approach TPDR. We
present in Table 3 examples of descriptions retrieved in the $1^{st}$ ranking
position. In none of the cases, purely syntactic approaches would be able to
retrieve the description correctly since, considering the client and
standardized descriptions, both are composed of different words with similar
meanings. There are also abbreviations, syntactically different names, or even
typographical errors in all descriptions. Therefore, our current proposal
manages to overcome the syntactic limitations, achieving semantically superior
results.

Table 3. Descriptions retrieved in the $1^{st}$ ranking position.

From a pure semantic perspective, TPDR is similar to TPDR(noRR). However, it
is possible to notice that the ranking results improve in almost all cases in
Table 2, when the syntactic re-ranker is applied, as is the case of the
description ”font semikron msk b 500/445-1.5 (9727)”, seen in Table 4,
retrieved in the $5^{th}$ position by TPDR(noRR) and in $3^{rd}$ by the full
TPDR solution.

Table 4 presents descriptions retrieved erroneously in the $1^{st}$ position,
but with the standardized product ranked up to the 3rd position. We can see
that there is high similarity between them, often differing in aspects related
to model, size, and other characteristics.

Table 4. Descriptions retrieved erroneously in the $1^{st}$ position, with the
correct SD present in the top 3.

Finally, in Table 5, we present descriptions retrieved erroneously but with
the correct associated DP in the $1^{st}$ position of the rank. Note that, as
shown in the result of the previous table, most of the items retrieved
erroneously are due to product specificities. However, the correct association
of the PD helps to filter out the correct description, as it limits the search
for similar products.

Table 5. Description retrieved erroneously in the top, but with PD correctly
associated.

The presented set of experiments so far demonstrated the flexibility and the
proposal’s effectiveness. All these results were properly reported and
discussed with the contracting company – Astrein – which was very satisfied
with the proposed solution.

##  5\. Conclusion

In this paper, we formalized the Product and Class Description Match problem –
matching an item described by a client with a product described in a catalog –
as a ranking task: given an initial client product specification (query),
return the most appropriate standardized description (response). We
demonstrated the complexity of this problem since the client’s product
descriptions are often noisy, short/uninformative, and may be formed of multi-
language specifications. Also, the initial client specification (IS) may
differ significantly from the standardized description (SD). To deal with this
complex problem, we proposed TPDR, a two-step method that is able to explore
the semantic correspondence between IS and SD by exploiting attention
mechanisms, contrastive learning, and a syntactic-based re-ranking approach.
In our evaluation, considering 11 datasets from a real company, covering
different application contexts and dataset sizes, TPDR was able to retrieve
the correct standardized product before the $5^{th}$ ranking position in 71%
of the cases. Our proposed approach also managed to retrieve the correct
Description Pattern in the first position of the ranking in 80.2% of the
queries. Moreover, the effectiveness gains over the baselines (purely
syntactic or semantic) reach up to 3.7 times, solving cases that none of the
approaches in isolation could do.

In future work, we plan to advance mainly in five directions: (i) evaluating
other varieties of transformers as encoders (Cunha et al., 2023b, a); (ii)
further investing in fine-tuning the models due to the specificity of the
vocabularies (de Andrade et al., 2023; Cunha et al., 2020); (iii) evaluating
alternative re-ranking score functions with dataset-specific coefficients;
and, finally (iv) investigating new strategies to increase the recall of the
first retrieval step (Embedding-based Retrieval) (Bianco et al., 2023), for
instance by a semantic expansion of specifications (Belém et al., 2011; Viegas
et al., 2019, 2020).

## References

  * (1)
  * Aslanyan et al. (2020) Grigor Aslanyan, Aritra Mandal, Prathyusha Senthil Kumar, Amit Jaiswal, and Manojkumar Rangasamy Kannadasan. 2020.  Personalized Ranking in ECommerce Search. In _Companion Proceedings of the Web Conference 2020_ (Taipei, Taiwan) _(WWW ’20)_. Association for Computing Machinery, New York, NY, USA, 96–97.  https://doi.org/10.1145/3366424.3382715
  * Belém et al. (2011) Fabiano Belém, Eder Martins, Tatiana Pontes, Jussara Almeida, and Marcos Gonçalves. 2011.  Associative Tag Recommendation Exploiting Multiple Textual Features. In _Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Beijing, China) _(SIGIR ’11)_. Association for Computing Machinery, New York, NY, USA, 1033–1042.  https://doi.org/10.1145/2009916.2010053
  * Bellet et al. (2015) A. Bellet, A. Habrard, and M. Sebban. 2015.  _Metric Learning_.  Morgan & Claypool Publishers.
  * Bianco et al. (2023) Guilherme Dal Bianco, Denio Duarte, and Marcos André Gonçalves. 2023.  Reducing the user labeling effort in effective high recall tasks by fine-tuning active learning.  _Journal of Intelligent Information Systems_ (2023).  https://doi.org/10.1007/s10844-022-00772-y
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language Models are Few-Shot Learners. In _NIPS_.
  * Cunha et al. (2020) Washington Cunha, Sérgio Canuto, Felipe Viegas, Thiago Salles, Christian Gomes, Vitor Mangaravite, Elaine Resende, Thierson Rosa, Marcos André Gonçalves, and Leonardo Rocha. 2020.  Extended pre-processing pipeline for text classification: On the role of meta-feature representations, sparsification and selective sampling.  _Information Processing & Management_ 57, 4 (2020), 102263.
  * Cunha et al. (2023a) Washington Cunha, Celso França, Guilherme Fonseca, Leonardo Rocha, and Marcos André Gonçalves. 2023a.  An Effective, Efficient, and Scalable Confidence-Based Instance Selection Framework for Transformer-Based Text Classification. In _Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 665–674.
  * Cunha et al. (2021) Washington Cunha, Vítor Mangaravite, Christian Gomes, Sérgio Canuto, Elaine Resende, Cecilia Nascimento, Felipe Viegas, Celso França, Wellington Santos Martins, Jussara M. Almeida, Thierson Rosa, Leonardo Rocha, and Marcos André Gonçalves. 2021.  On the cost-effectiveness of neural and non-neural approaches and representations for text classification: A comprehensive comparative study.  _IP &M_ (2021).
  * Cunha et al. (2023b) Washington Cunha, Felipe Viegas, Celso França, Thierson Rosa, Leonardo Rocha, and Marcos André Gonçalves. 2023b.  A Comparative Survey of Instance Selection Methods applied to NonNeural and Transformer-Based Text Classification.  _Comput. Surveys_ (2023).
  * de Andrade et al. (2023) Cláudio Moisés Valiense de Andrade, Fabiano Belém, Washington Cunha, Celso França, Felipe Viegas, Leonardo Rocha, and Marcos André Gonçalves. 2023.  On the class separability of contextual embeddings representations - or ”The classifier does not matter when the (text) representation is so good!”.  _Inf. Process. Manag._ 60, 4 (2023), 103336.  https://doi.org/10.1016/j.ipm.2023.103336
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _NAACL_.
  * Feng et al. (2020) Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020.  Language-Agnostic Bert Sentence Embedding.  _arXiv preprint arXiv:2007.01852_ (2020).
  * Gage (1994) Philip Gage. 1994.  A New Algorithm for Data Compression.  _C Users J._ (1994).
  * Garg et al. (2020) Siddhant Garg, Thuy Vu, and Alessandro Moschitti. 2020.  TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection.  _AAAI_ (2020).
  * Goodfellow et al. (2014) Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014.  Generative Adversarial Nets. In _NIPS_.
  * Gysel et al. (2018) Christophe Van Gysel et al. 2018\.  Neural Vector Spaces for Unsupervised Information Retrieval.  _ACM TOIS_ (2018).
  * Hadsell et al. (2006) R. Hadsell, S. Chopra, and Y. LeCun. 2006.  Dimensionality Reduction by Learning an Invariant Mapping. In _CVPR_.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016.  Deep Residual Learning for Image Recognition. In _CVPR_.
  * Hoffer and Ailon (2015) Elad Hoffer and Nir Ailon. 2015.  Deep Metric Learning Using Triplet Network. In _SIMBAD_.
  * Huai et al. (2019) Mengdi Huai, Hongfei Xue, Chenglin Miao, Liuyi Yao, Lu Su, Changyou Chen, and Aidong Zhang. 2019.  Deep Metric Learning: The Generalization Analysis and an Adaptive Algorithm. In _IJCAI_.
  * Jonesa et al. (2000) K Sparck Jonesa, S Walkerb, and SE Robertsonb. 2000.  A probabilistic model of information retrieval: development and comparative experiments.  _Information Processing and Management_ 36, 809 (2000), 840.
  * Karmaker Santu et al. (2017) Shubhra Kanti Karmaker Santu, Parikshit Sondhi, and ChengXiang Zhai. 2017.  On Application of Learning to Rank for E-Commerce Search. In _Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Shinjuku, Tokyo, Japan) _(SIGIR ’17)_. Association for Computing Machinery, New York, NY, USA, 475–484.  https://doi.org/10.1145/3077136.3080838
  * KAYA and BİLGE (2019) Mahmut KAYA and Hasan Şakir BİLGE. 2019.  Deep Metric Learning: A Survey.  _Symmetry_ (2019).
  * LeClair and McMillan (2019) Alexander LeClair and Collin McMillan. 2019.  Recommendations for Datasets for Source Code Summarization. In _NAACL_.
  * Li et al. (2022) Sen Li, Fuyu Lv, Taiwei Jin, Guiyang Li, Yukun Zheng, Tao Zhuang, Qingwen Liu, Xiaoyi Zeng, James Kwok, and Qianli Ma. 2022.  Query Rewriting in TaoBao Search. In _Proceedings of the 31st ACM International Conference on Information & Knowledge Management_ (Atlanta, GA, USA) _(CIKM ’22)_. Association for Computing Machinery, New York, NY, USA, 3262–3271.  https://doi.org/10.1145/3511808.3557068
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  RoBERTa: A Robustly Optimized BERT Pretraining Approach.  _CoRR_ (2019).
  * Lu et al. (2021) Shuai Lu, Daya Guo, Shuo Ren, Junjie Huang, Alexey Svyatkovskiy, Ambrosio Blanco, Colin Clement, Dawn Drain, Daxin Jiang, Duyu Tang, et al. 2021\.  Codexglue: A machine learning benchmark dataset for code understanding and generation. In _NeurIPS_.
  * Malkov and Yashunin (2020) Yu A. Malkov and D. A. Yashunin. 2020.  Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs.  _IEEE TPAMI_ (2020).
  * Maschler et al. (2020) Michael Maschler, Eilon Solan, and Shmuel Zamir. 2020.  Strategic-form Games.  In _Game Theory_. Cambridge Press.
  * Nash (1953) John Nash. 1953.  Two-Person Cooperative Games.  _Econometrica_ (1953).
  * Petrov and Makarov (2021) Aleksandr Petrov and Yuriy Makarov. 2021.  Attention-based neural re-ranking approach for next city in trip recommendations.  (2021).  https://doi.org/10.48550/ARXIV.2103.12475
  * Schroff et al. (2015) Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015.  Facenet: A unified embedding for face recognition and clustering. In _CVPR_.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural Machine Translation of Rare Words with Subword Units. In _ACL_.
  * Sohn (2016) Kihyuk Sohn. 2016.  Improved Deep Metric Learning with Multi-class N-pair Loss Objective. In _NIPS_.
  * Song et al. (2016) Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese. 2016.  Deep Metric Learning via Lifted Structured Feature Embedding. In _CVPR_.
  * Souza et al. (2020) Fábio Souza, Rodrigo Nogueira, and Roberto Lotufo. 2020.  BERTimbau: pretrained BERT models for Brazilian Portuguese. In _Intelligent Systems: 9th Brazilian Conference, BRACIS 2020, Rio Grande, Brazil, October 20–23, 2020, Proceedings, Part I 9_. Springer, 403–417.
  * Tan et al. (2005) Pang-Ning Tan, Michael Steinbach, and Vipin Kumar. 2005.  Introduction to data mining. ed.  _Addison-Wesley Longman Publishing Co., Inc._ (2005).
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is All you Need. In _NIPS_.
  * Viegas et al. (2019) Felipe Viegas, Sérgio D. Canuto, Christian Gomes, Washington Luiz, Thierson Rosa, Sabir Ribas, Leonardo Rocha, and Marcos André Gonçalves. 2019.  CluWords: Exploiting Semantic Word Clustering Representation for Enhanced Topic Modeling. In _Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, WSDM 2019, Melbourne, VIC, Australia, February 11-15, 2019_ , J. Shane Culpepper, Alistair Moffat, Paul N. Bennett, and Kristina Lerman (Eds.). ACM, 753–761.  https://doi.org/10.1145/3289600.3291032
  * Viegas et al. (2020) Felipe Viegas, Washington Cunha, Christian Gomes, Antônio Pereira De Souza Júnior, Leonardo Rocha, and Marcos André Gonçalves. 2020.  CluHTM - Semantic Hierarchical Topic Modeling based on CluWords. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault (Eds.). Association for Computational Linguistics, 8138–8150.  https://doi.org/10.18653/v1/2020.acl-main.724
  * Xie et al. (2015) Saining Xie, Tianbao Yang, Xiaoyu Wang, and Yuanqing Lin. 2015.  Hyper-class augmented and regularized deep learning for fine-grained image classification. In _CVPR_.
  * Yang et al. (2020) Yinfei Yang, Daniel Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, Gustavo Hernandez Abrego, Steve Yuan, Chris Tar, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2020.  Multilingual Universal Sentence Encoder for Semantic Retrieval. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_. Association for Computational Linguistics, Online, 87–94.  https://doi.org/10.18653/v1/2020.acl-demos.12
  * Zhu et al. (2020) Runjie Zhu, Xinhui Tu, and Jimmy Xiangji Huang. 2020.  Deep learning on information retrieval and its applications.  In _Deep Learning for Data Analytics_. Academic Press.