# CERES: Pretraining of Graph-Conditioned Transformer for Semi-Structured
Session Data

Rui Feng  Georgia Institute of Technology  Chen Luo  Amazon Inc.  Qingyu Yin
Amazon Inc.  Bing Yin  Amazon Inc.  Tuo Zhao  Georgia Institute of Technology
Chao Zhang  Georgia Institute of Technology

###### Abstract

User sessions empower many search and recommendation tasks on a daily basis.
Such session data are semi-structured, which encode heterogeneous relations
between queries and products, and each item is described by the unstructured
text. Despite recent advances in self-supervised learning for text or graphs,
there lack of self-supervised learning models that can effectively capture
both intra-item semantics and inter-item interactions for semi-structured
sessions. To fill this gap, we propose CERES, a graph-based transformer model
for semi-structured session data. CERES learns representations that capture
both inter- and intra-item semantics with (1) a graph-conditioned masked
language pretraining task that jointly learns from item text and item-item
relations; and (2) a graph-conditioned transformer architecture that
propagates inter-item contexts to item-level representations. We pretrained
CERES using $\sim$468 million Amazon sessions and find that CERES outperforms
strong pretraining baselines by up to 9% in three session search and entity
linking tasks.

CERES: Pretraining of Graph-Conditioned Transformer for Semi-Structured
Session Data

##  1 Introduction

User sessions are ubiquitous in online e-commerce stores. An e-commerce
session contains customer interactions with the platform in a continuous
period. Within one session, the customer can issue multiple queries and take
various actions on the retrieved products for these queries, such as clicking,
adding to cart, and purchasing. Sessions are important in many e-commerce
applications, e.g., product recommendation Wu et al. (2019a), query
recommendation Cucerzan and White (2007), and query understanding Zhang et al.
(2020).

Figure 1: Illustration of a customer session. A session consists of two types
of items: queries and products. The customer searched for 3 keywords
sequentially and interacted with the products returned by the search engine.

This paper considers sessions as _semi-structured_ data, as illustrated in
Figure 1. At the higher level, sessions are heterogeneous graphs that contain
interactions between items. At the lower level, each graph node has
unstructured text descriptions: we can describe queries by search keywords and
products by titles, attributes, customer reviews, and other descriptors. Our
goal is to simultaneously encode both the graph and text aspects of the
session data to understand customer preferences and intents in a session
context.

Pretraining on semi-structured session data remains an open problem. First,
existing works on learning from session data usually treat a session as a
sequence or a graph Xu et al. (2019); You et al. (2019); Qiu et al. (2020b).
While they can model inter-item relations, they do not capture the rich intra-
item semantics when text descriptions are available. Furthermore, these models
are usually large neural networks that require massive labeled data to train
from scratch. Another line of research utilizes large-scale pretrained
language models Lan et al. (2019); Liu et al. (2019); Clark et al. (2020) as
text encoders for session items. However, they fail to model the relational
graph structure. Several works attempt to improve language models with a
graph-structured knowledge base, such as in Liu et al. (2020); Yao et al.
(2019); Shen et al. (2020). While adjusting the semantics of entities
according to the knowledge graph, they fail to encode general graph structures
in sessions.

We propose CERES (Graph Conditioned Encoder Representations for Session Data),
a pretraining model for semi-structured e-commerce session data, which can
serve as a generic session encoder that simultaneously captures both intra-
item semantics and inter-item relations. Beyond training a potent language
model for intra-item semantics, our model also conditions the language
modeling task on graph-level session information, thus encouraging the
pretrained model to learn how to utilize inter-item signals. Our model
architecture tightly integrates two key components: (1) an _item Transformer
encoder_ , which captures text semantics of session items; and (2) a _graph
conditioned Transformer_ , which aggregates and propagates inter-item
relations for cross-item prediction. As a result, CERES models the higher-
level interactions between items.

We have pretrained CERES using 468,199,822 sessions and performed experiments
on three session-based tasks: product search, query search, and entity
linking. By comparing with publicly available state-of-the-art language models
and domain-specific language models trained on alternative representations of
session data, we show that CERES outperforms strong baselines on various
session-based tasks by large margins. Experiments show that CERES can
effectively utilize session-level information for downstream tasks, better
capture text semantics for session items, and perform well even with very
scarce training examples.

We summarize our contributions as follows: 1) We propose CERES , a pretrained
model for semi-structured e-commerce session data. CERES can effectively
encode both e-commerce items and sessions and generically support various
session-based downstream tasks. 2) We propose a new graph-conditioned
transformer model for pretraining on general relational structures on text
data. 3) We conducted extensive experiments on a large-scale e-commerce
benchmark for three session-related tasks. The results show the superiority of
CERES over strong baselines, including mainstream pretrained language models
and state-of-the-art deep session recommendation models.

##  2 Customer Sessions

A _customer session_ is the search log before a final purchase action. It
consists of customer-query-product interactions: a customer submits search
queries obtains a list of products. The customer may take specific actions,
including _view_ and _purchase_ on the retrieved products. Hence, a session
contains two types of items: _queries_ and _products_ , and various relations
between them established by customer actions.

We define each session as a _relational graph_ $G=(\mathcal{V},\mathcal{E})$
that contains all queries and products in a session and their relations. The
vertex set $\mathcal{V}=(\mathcal{Q},\mathcal{P})$ is partitioned into ordered
query set $\mathcal{Q}$ and unordered product set $\mathcal{P}$. The queries
$\mathcal{Q}=(q_{1},\ldots,q_{n})$ are indexed by order of the customer’s
searches. The edge set $\mathcal{E}$ contains two types of edges:
$\\{(q_{i},q_{j}),i<j\\}$ are one-directional edges that connect each query to
its previous queries; and $\\{q_{i},p_{j},a_{ij}\\}$ are bidirectional edges
that connects the $i$th query and $j$th product, if the customer took action
$a_{ij}$ on product $p_{j}$ retrieved by query $q_{j}$.

The queries and products are represented by textual descriptions.
Specifically, each query is represented by customer-generated search keywords.
Each product is represented with a table of textual attributes. Each product
is guaranteed to have a product title and description. In this paper, we call
“product sequence” as the concatenation of title and description. A product
may have additional attributes, such as product type, color, brand, and
manufacturer, depending on their specific categories.

##  3 Our Method

In this section we present the details of CERES. We first describe our
designed session pretraining task in Section 3.1, and then describe the model
architecture of CERES in Section 3.2.

Figure 2: Model illustration. CERES first produces intra-item embeddings in
the Item Transformer Encoder. Then, the Graph-Conditioned Transformer
aggregates and propagates session-level information to obtain inter-item
embeddings.

###  3.1 Graph-Conditioned Masked Language Modeling Task

Suppose $\mathcal{G}=(\mathcal{V},\mathcal{E})$ is a graph on $T$ text items
as vertices, $v_{1},\ldots,v_{T}$, each of which is a sequence of text tokens:
$v_{i}=[v_{i1},\ldots,v_{iT_{i}}]$, $i=1,\ldots,T$. We propose _graph-
conditioned masked language modeling_ (GMLM), where masked tokens are
predicted with both intra-item context and inter-item context:

$p_{\textrm{GMLM}}(v_{\textrm{masked}})=\prod_{j\textrm{th\
masked}}\mathbb{P}(v_{ij}|\mathcal{G},\\{v_{ik}\\}_{k\textrm{th\ unmasked}}),$
(1)

which encourages the model to leverage information graph-level inter-item
semantics efficiently in order to predict masked tokens. To optimize (1), we
need to learn token-level embeddings that are infused with session-level
information, which we introduce in Section 3.2.2. Suppose certain tokens in
the input sequence of items as masked (detailed below), we optimize the
predictions of the masked tokens with cross entropy loss. The pretraining
framework is illustrated in Figure 3.

Figure 3: Pretraining framework illustration. CERES learns both inter-item and
intra-item embeddings for item tokens for Masked LM and Graph-Conditioned
Masked LM. In practice, we find it beneficial to optimize both.

Token Masking Strategy. To mask tokens in long sequences, including product
titles and descriptions, we follow Devlin et al. (2018) and choose 15% of the
tokens for masking. For short sequences, including queries and product
attributes, there is a 50% probability that a short sequence will be masked,
and for those sequences 50% of their tokens are randomly selected for masking.

###  3.2 Model Architecture

To model the probability in (1), we design two key components in the CERES
model: 1) a _Transformer-based item encoder_ , which produces token-level
intra-item embeddings that contain context information within a single item;
and 2) a _graph-conditioned Transformer for session encoding_ , which produces
session-level embeddings that encodes inter-item relations, and propagates the
session information back to the token-level. We illustrate our model
architecture in Figure 2.

####  3.2.1 Item Transformer Encoder

The session item encoder aims to encode intra-item textual information for
each item in a session. We design the item encoder based on Transformers,
which allows CERES to leverage the expressive power of the self-attention
mechanism for modeling domain-specific language in e-commerce sessions. Given
an item $i$, the transformer-based item encoder compute its token embeddings
as follows:

$\displaystyle\ [\mathbf{v}_{i1},\ldots,\mathbf{v}_{iT_{i}}]$
$\displaystyle=\mathrm{Transformer}_{\mathrm{item}}([v_{i1},\ldots,v_{iT_{i}}])$
(2) $\displaystyle\mathbf{v}_{i}$
$\displaystyle=\mathrm{Pool}([v_{i1},\ldots,v_{iT_{i}}]),$

where $\mathbf{v}_{ij}$ is the embedding of the $j$th token in the $i$th item,
and $\mathbf{v}_{i}$ is the pooled embedding of the $i$th item. At this stage,
$\\{\mathbf{v}_{ij}\\},\\{\mathbf{v}_{i}\\}$ are embeddings that only encode
the intra-item information.

Details of Item Encoding. We detail the encoding method for the two types of
items, queries and products, in the following paragraphs.

Each query $q_{i}=[q_{i1},\ldots,q_{iT_{i}}]$ is a sequence of tokens
generated by customers as search keywords. We add a special token at the
beginning of the queries, [SEARCH], to indicate that the sequence represents a
customer’s search keywords. Then, to obtain the token-level embedding of the
queries and the pooled query embedding by taking the embedding of the special
token [SEARCH].

Each product $p_{i}$ is a table of $K$ attributes: $p^{1},\ldots,p^{K}$, where
$p^{1}$ is always the _product sequence_ , which is the concatenation of
product title and bullet description. Each attribute
$p_{i}^{k}=[p_{i1}^{k},p_{i2}^{k},\ldots]$ starts with a special token
$[\texttt{ATTRTYPE}]$, where ATTRTYPE is replaced with the language descriptor
of the attribtue. Then, the Transformer is used to compute token and sentence
embeddings for all attributes. The product embedding is obtained by average
pooling of all attribute’s sentence embeddings.

####  3.2.2 Graph-Conditioned Session Transformer

The Graph-Conditioned Session Transformer aims to infuse intra-item and inter-
item information to produce item and token embeddings. For this purpose, we
first design a position-aware graph neural network (PGNN) to capture the
inter-item dependencies in a session graph to produce item embeddings. Then
conditioned on the PGNN-learned item embedding, we propose a cross-attention
Transformer, which produces infused item and token embeddings for the Graph-
Conditioned Masked Language Modeling task.

Position-Aware Graph Neural Network. We use a GNN to capture inter-item
relations. This will allow CERES to obtain item embeddings that encode the
information from other locally correlated items in the session. Let
$[\mathbf{v}_{1},\ldots,\mathbf{v}_{N}]$ denote the item embeddings produced
by the intra-item transformer encoder. We treat them as hidden states of nodes
in the session graph $\mathcal{G}$ and feed them to the GNN model, obtaining
session-level item embeddings
$[{\mathbf{v}}^{h}_{1},\ldots,{\mathbf{v}}^{h}_{N}]$.

The items in a session graph are sequential according to the order the
customers generated them. To let the GNN model learn of the positional
information of items, we train an _item positional embedding_ in analogous to
positional embedding of tokens. Before feeding the item embeddings to GNN, the
pooled item embeddings are added item positional embeddings according to their
positions in the session’s item sequence. In this way, the item embeddings
$\\{\mathbf{v}^{i}\\}_{i\in\mathcal{V}}$ are encoded their positional
information as well.

Figure 4: Illustration of cross-attention over latent conditioning tokens. The
item token embeddings perform self-attention as well as cross-attention over
latent conditioning tokens, thus incorporating session-level information.
Latent conditioning tokens perform self-attention to update their embeddings,
but do not attend to item tokens to preserve session-level information.

Cross-Attention Transformer. Conditioned on PGNN, we design a cross-attention
transformer which propagates session-level information in PGNN-produced item
embeddings to all tokens to produce token embeddings that are infused with
both intra-item and inter-item information.

In order to propagate item embeddings to tokens, we treat item embeddings as
latent tokens that can be treated as a “part” of item texts. for each item
$i$, we first expand $\mathbf{v}_{i}^{h}$ to $K$ latent conditioning tokens by
using a multilayer perceptron module to map $\mathbf{v}_{i}^{h}$ to $K$
embedding vectors $[\mathbf{v}_{i1}^{h},\ldots,\mathbf{v}_{iK}^{h}]$ of the
same size. For each item $i$, we compute its latent conditioning tokens by
averaging all latent tokens in its neighborhood. Suppose $N(i)$ is the set of
all neighboring items in the session graph, itself included. In each position,
we take the average of the latent token embeddings in $N(i)$ as the $k$th
latent conditioning token, $\overline{\mathbf{v}}_{ik}^{h}$, for the $i$th
item. Then, we concatenate the latent conditioning token embeddings and the
item token embeddings obtained by the session item encoder:

$[\overline{\mathbf{v}}_{i1}^{h},\ldots,\overline{\mathbf{v}}_{iK}^{h},\mathbf{v}_{i1},\ldots,\mathbf{v}_{iN_{i}}].$
(3)

Finally, we compute the token-level embeddings with session information by
feeding the concatenated sequence to a shallow cross-attention Transformer.
The cross-attention Transformer is of the same structure as normal
Transformers. The difference is that we prohibit the latent conditioning
tokens from attending over original item tokens to prevent the influx of
intra-item information potentially diluating session-level information stored
in latent conditioning tokens. Illustration of cross-attention Transformer is
provided in Figrue 4.

We use the embeddings produced by this cross-attention Transformer as the
final embeddings for modeling the token probabilities in Equation (1) and
learning the masked language modeling tasks. During training, the model is
encouraged to learn good token embeddings with the Item Transformer Encoder,
as better embeddings $\\{{\mathbf{v}_{ij}}\\}_{j=1}^{N_{i}}$ is necessary to
improve the quality of $\\{{\mathbf{v}^{c}_{ij}}\\}_{j=1}^{N_{i}}$. The Graph-
Conditioned Transformer will be encouraged to produce high-quality session-
level embeddings for the GMLM task. Hence, CERES is encouraged to produce
high-quality embeddings that unify both intra-item and inter-item information.

###  3.3 Finetuning

When finetuning CERES for downstream tasks, we first obtain session-level item
embeddings. The session embedding is computed as the average of all item
embeddings. To obtain embedding for a single item without session context,
such as for retrieved items in recommendation tasks, only the Item Transformer
Encoder is used.

To measure the relevance of an item to a given session, we first transform the
obtained embeddings by separate linear maps. Denote the transformed session
embeddings as $\mathbf{s}$ and item embeddings as $\mathbf{y}$. The similarity
between them is computed by cosine similarity
$\mathrm{d}_{\mathbf{cos}}(\mathbf{s},\mathbf{y})$. To finetune the model, we
optimize a hinge loss on the cosine similarity between sessions and items.

##  4 Experiments

###  4.1 Experiment Setup

Dataset. We collected customer sessions from Amazon for pretraining and
finetuning on downstream tasks. 468,199,822 customer sessions are collected
from August 1 2020 to August 31 2020 for pretraining. 30,000 sessions are
collected from September 2020 to September 7 2020 for downstream tasks. The
pretraining and downstreaming datasets are from disjoint time spans to prevent
data leakage. All data are cleaned and anonymized so that no personal
information about customers was used. Each session is collected as follows:
when a customer perform a purchase action, we backtrace all actions by the
customer in 600 seconds before the purchase until a previous purchase is
encountered. The actions of customers include: 1) search, 2) view, 3), add-to-
cart, and 4) purchase. Search action is associated with customer generated
query keywords. View, add-to-cart, and purchase are associated with the target
products. All the products in the these sessions are gathered with their
product title, bullet description, and various other attributes, including
color, manufacturer, product type, size, etc. In total, we have 37,580,637
products. The sessions have an average of 3.24 queries and 4.36 products.
Queries have on average 5.63 tokens, while product titles and bullet
descriptions have averagely 17.42 and 96.01 tokens.

Evaluation Tasks and Metrics. We evaluate all the compared models on the
following tasks: 1) Product Search. In this task, given observed customer
behaviors in a session, the model is asked to predict which product will be
purchased from a pool of candidate products. The purchased products are
removed from sessions to avoid trivial inference. The candidate product pool
is the union of all purchased products in the test set and the first 10
products returned by the search engine of all sessions in the test set.

2) Query Search. Query Search is a recommendation task where the model
retrieves next queries for customers which will lead to a purchase. Given a
session, we hide the last query along with products associated with it, i.e.
viewed or purchased with the removed query. Then, we ask the model to predict
the last query from a pool of candidate queries. The candidate query pool
consists of all last queries in the test set.

3) Entity Linking. In this task we try to understand the deeper semantics of
customer sessions. Specifically, if customer purchases a product in a session,
the task is to predict the attributes of the purchased product from the rest
contexts in the session. In total, we have 60K possible product attributes.

Baselines. The compared baselines can be categorized into three groups:

1) General-domain pretrained language models which include BERT Devlin et al.
(2018), RoBERTa Liu et al. (2019), and ELECTRA Clark et al. (2020). These
models are state-of-the-art pretrained language models, which can serve as
general-purpose language encoders for items and enable downstream session-
related tasks. Specifically, the language encoders produce item embeddings
first, and compose session embeddings by pooling the items in sessions. To
retrieve items for sessions, one can compare the cosine similarity between
sessions and retrieved items.

2) Pretrained session models which are pretrained models on e-commerce session
data. Specifically, we pretrain the following language models using our
session data: a) _Product-BERT_ , which is a domain-specific BERT model
pretrained with product information; b) _SQSP-BERT_ , where _SQSP_ is short
for _Single-query Single-Product_. SQSP-BERT is pretrained on query-product
interaction pairs with language modeling and contrastive learning objectives.
They are used in the same manner in downstream tasks as general-domain
pretrained language models. The detailed configurations are provided in the
Appendix.

3) Session-based recommendation methods including SR-GNN Wu et al. (2019b) and
NISER+ Gupta et al. (2019), which are state-of-the-art models for session-
based product recommendation on traditional benchmarks, including YOOCHOOSE
and DIGINETICA; and Nvidia’s MERLIN Mobasher et al. (2001), which is the best-
performing model in the recent SIGIR Next Items Prediction challenge Kallumadi
et al. (2021)

To evaluate the performance on these tasks, we employ standard metrics for
recommendation systems, including MAP@K, and Recall@K.

###  4.2 Implementation Details

The implementation details for pretraining and finetuning stages are described
as follows.

Pretraining details. We developed our model based on Megatron-LM Shoeybi et
al. (2019). We used 768 as the hidden size, a 12-layer transformer blocks as
the backbone language model, a two-layer Graph Attention Network and three-
layer Transformer as the conditioned language model layers. In total, our
model has 141M parameters. The model is trained for 300,000 steps with a batch
size of 512 sessions. The parameters are updated with Adam, with peak learning
rate as $3e-5$, $1\%$ steps for linear warm-up, and linear learning rate decay
after warm-up until the learning rate reaches the minimum $1e-5$. We trained
our model on 16 A400 GPUs on Amazon AWS for one week.

Finetuning details. For each downstream task, we collected 30,000 sessions for
training, 3000 for validation and 5000 for testing. For each of the pretrained
model, we finetune them for 10 epochs with a maximal learning rate chosen from
[1e-4, 1e-5, 5e-5, 5e-6] to maximize MAP@1 on the validation set. The rest of
the configuration of optimizers is the same as in pretraining.

###  4.3 Main Results

Method | map@1 | recall@1 | map@32 | recall@32 | map@64 | recall@64
---|---|---|---|---|---|---
SR-GNN | 36.313 | 37.284 | 50.683 | 99.592 | 60.413 | 99.689
NISER+ | 37.193 | 38.144 | 52.855 | 98.293 | 62.371 | 99.111
MERLIN | 89.744 | 90.166 | 93.067 | 98.98 | 93.075 | 99.33
BERT | 85.096 | 84.688 | 89.172 | 99.082 | 89.18 | 99.301
RoBERTa | 79.647 | 78.963 | 83.207 | 95.396 | 83.25 | 97.494
Electra | 85.897 | 86.32 | 89.841 | 99.344 | 89.845 | 99.519
Product-Bert | 91.026 | 91.71 | 93.856 | 99.563 | 93.856 | 99.563
SQSP-Bert | 85.577 | 85.795 | 90.049 | 99.038 | 90.057 | 99.301
CERES | 92.628 | 93.094 | 94.848 | 99.551 | 94.853 | 99.65
Table 1:  The performance of different methods for Product Search, after fine-tuning with 30,000 training sessions. Method | map@1 | recall@1 | map@32 | recall@32 | map@64 | recall@64
---|---|---|---|---|---|---
BERT | 47.276 | 47.627 | 60.143 | 92.553 | 60.214 | 95.417
RoBERTa | 26.603 | 26.323 | 37.722 | 74.468 | 37.839 | 80.196
Electra | 32.853 | 32.788 | 47.512 | 90.426 | 47.632 | 95.663
Product-BERT | 52.724 | 52.973 | 66.035 | 95.99 | 66.065 | 97.463
SQSP-BERT | 45.833 | 46.29 | 60.195 | 92.881 | 60.26 | 95.499
CERES | 59.936 | 60.284 | 72.329 | 97.463 | 72.331 | 97.627
Table 2:  The performance of different methods for Query Search, after fine-tuning with 30,000 training sessions.  Method | map@1 | recall@1 | map@32 | recall@32 | map@64 | recall@64
---|---|---|---|---|---|---
BERT | 55.609 | 55.353 | 66.386 | 90.511 | 66.481 | 95.073
RoBERTa | 66.506 | 65.754 | 74.516 | 93.248 | 74.561 | 95.438
Electra | 62.321 | 62.365 | 62.985 | 68.296 | 63.122 | 74.318
Product-Bert | 66.827 | 66.393 | 74.611 | 94.404 | 74.641 | 96.046
SQSP-Bert | 63.942 | 64.872 | 72.232 | 91.241 | 72.307 | 94.891
CERES | 75.481 | 75.456 | 81.121 | 95.255 | 81.16 | 96.898
Table 3:  The performance of different methods for Entity Linking, after fine-
tuning with 30,000 training sessions.

####  4.3.1 Product Search

Table 1 shows the performance of different methods for the product search
task. We observe that CERES outperforms domain-specific methods by more than
1% and general-domain methods by over 6% in MAP@1. The second best performing
model is Product-BERT, which is pretrained on product information alone.

We also compared with session-based recommendation systems. SR-GNN and NISER+
model only session graph structure but not text semantics; hence they have
limited performance because of the suboptimal representation of session items.
While MERLIN can capture better text semantics, its text encoder is not
trained on domain-specific e-commerce data. While it can outperform general-
domain methods, its performance is lower than Product-BERT and CERES. The
benefits of joint modeling of text and graph data and the Graph-Conditioned
MLM allow CERES to outperform existing session recommendation models.

####  4.3.2 Query Search

Table 2 shows the performance of different methods on Query Search. Query
Search is a more difficult task than Product Search because customer-generated
next queries are of higher variance. In this challenging task, CERES
outperforms the best domain-specific model by over 7% and general-domain model
by 12% in all metrics.

####  4.3.3 Entity Linking

Table 3 shows the results on Entity Linking. Similar to Query Search, this
task also requires the models to tie text semantics (queries/product
attributes) to a customer session, which requires a deeper understanding of
customer preferences. It is easier than Query Search as product attributes are
of lower variance. However, the product attributes that the customer prefer
rely more on session information, as they may have been reflected in the past
search queries and viewed products. In this task, CERES outperforms domain-
specific models and general-domain models by averagely 9% in MAP@1 and 6% in
MAP@32 and MAP@64.

###  4.4 Further Analysis and Ablation Studies

In this section we present further studies to understand: 1) the effect of
training data sizes in the downstream task; 2) the effects of different
components in CERES for both the pretraining and finetuning stages. following
observations:

CERES is highly effective when training data are scarce. We compare CERES with
two strongest baselines (BERT, and Product-BERT) when the training sample size
varies. Figure 5 shows the MAP@64 scores of these methods on Product Search
and Query Search when training size varies. Clearly, the advantage of CERES is
greater when training data is extremely small. With a training size of 300,
CERES can achieve a decent performance of about 37.55% in Product Search and
36.37% in Query Search, while the baseline models cannot be trained
sufficiently with such small-sized data. This shows that the efficient
utilization of session-level information in pretraining and fine-tuning stages
make the model more data efficient than other pretrained models.

(a) Product Search (b) Query Search

Figure 5: Effect of sample size on Product Search and Query Search. x-axis
represents the training data size and y-axis represents MAP@64.

Graph-Conditioned Transformer is Vital to Pretraining. Without the Graph-
Conditioned Transformer in pretraining, our model is essentially the same as
domain-specific baselines, such as Product-BERT, which are trained on session
data but only with intra-item text signals. While SQSP-BERT has access to
session-level information when maximizing the masked language modeling
objective, the lack of a dedicated module for GMLM results in worse
performance, as shown in the main experiment results.

We could train the Graph-Conditioned Transformer from scratch in the
finetuning stage. We present a model called _CERES w/o Pretrain_ , which
attaches the Graph-Conditioned Session Transformer to Product-BERT as the Item
Transformer Encoder. As shown in Figure 6, this ablation method achieves
MAP@64 scores of 89.341% in Product Search, 64.890% in Query Search, and
74.031% in Entity Linking, which are below Product-BERT. This shows that the
pretraining stage of the Graph-Conditioned Transformer is necessary to
facilitate its ability to aggregate and propagate session-level information
for downstream tasks.

Figure 6: Results on three tasks on ablation models. y-axis represents MAP@64.
_CERES w/o Cond_ is CERES without the Graph-Conditioned Transformer in the
finetuning stage. _CERES w/o Pretrain_ is CERES without pretraining the Graph-
Conditioned Transformer, but instead training it from scratch in the
finetuning stage. _CERES w/o GNN_ is CERES pretrained without the GNN module.

Graph-Conditioned Transformer Improves Item-level Embeddings. We also present
_CERES w/o Cond_ , which has the same pretrained model as CERES, but only uses
the Item Transformer Encoder in the finetuning stage. The Item Transformer
Encoder is used to compute session item embeddings that contain only item-
level information, and then takes the average of these embeddings as session
embedding. As shown in Figure 6, _CERES w/o Cond_ acheives 94.741%, 72.175%,
and 81.03% respectively in Product Search, Query Search, and Entity Linking,
observing a drop of 0.1% to 0.2% in performance compared with _CERES_. The
performance drop is minor and _CERES w/o Cond_ still outperforms baseline
pretrained language models. Hence, the Graph-Conditioned Transformer in the
pretraining stage helps the Item Transformer Encoder to learn better item-
level embeddings that can be used for more effective leveraging of session
information in the downstream tasks.

Graph Neural Networks Improve Representation of Sessions.  In _CERES w/o GNN_
, we pretrain a _CERES_ model without a Graph Neural Network. Specifically,
_CERES w/o GNN_ skips the neighborhood information aggregation for items, and
uses item-level embeddings obtained by the Item Transformer Encoder directly
as latent conditioning tokens. We train and finetune this model with the same
setup as CERES. Without GNN, the model’s performance is consistently lower
than CERES, achieving 93.453%, 71.231%, 80.26% MAP@64 in three downstream
tasks, observing a 1.13% performance drop. This shows that GNN’s aggregation
of information can help item-level embeddings encode more session-level
information, improving performance in downstream tasks.

Model Efficiency. CERES has additional few GNN and Transformer layers attached
to the end of the model. The additional layers bring $\sim$20% additional
inference time compared to standard BERT with 12 layers and 768 hidden size.

##  5 Related work

Pretrained language models such as BERT Devlin et al. (2018), BART Lewis et
al. (2019), ELECTRA Clark et al. (2020), RoBERTa Liu et al. (2019) have pushed
the frontiers of many NLP tasks by large margins. Their effectiveness and
efficiency in parallelism have made them popular and general-purpose language
encoders for many text-rich applications. However, they are not designed to
model relational and graph data, and hence are not the best fit for e-commerce
session data.

Researchers have also sought to enhance text representations in pretrained
models with knowledge graphs Shen et al. (2020); Liu et al. (2020); Yao et al.
(2019); Sun et al. (2020, 2021). While these models consider a knowledge graph
structure on top of text data, they generally use entities or relations in
knowledge graphs to enhance text representations, but cannot encode arbitrary
graph structures. This is not sufficient in session-related applications as
session structures are ignored.

Many works have been proposed to learn pretrained graph neural networks.
Initially, methods were proposed for domain-specific graph pretraining Hu et
al. (2019a, b); Shang et al. (2019). However, they rely on pre-extracted
domain-specific node-level features, and cannot be extended to either session
data or text data as nodes. Recently, many works have been proposed to
pretrain on general graph structure Hu et al. (2020); You et al. (2020); Qiu
et al. (2020a). However, they cannot encode the semantics of text data as
nodes.

Contextual information in sessions have been shown beneficial to various
related recommendation tasks, such as product recommendation Wu et al.
(2019b); Dehghani et al. (2017); Jannach and Ludewig (2017); Gupta et al.
(2019) and query rewriting Li et al. (2017); Cucerzan and White (2007). Many
existing session-based recommendation methods seek to exploit the transitions
between items Yap et al. (2012); Rendle et al. (2010); Wang et al. (2018); Li
et al. (2017) and considering sessions as graphs Xu et al. (2019); Ruihong et
al. (2021); Wang et al. (2020).

##  6 Limitations and Risks

This paper limits the application of CERES to session data with text
descriptions. CERES has the potential of being a universal pretraining
framework for arbitrary heterogeneous data. For example, sessions can include
product images and customer reviews for more informative multimodal graphs. We
leave this extension for future work.

Session data are personalized experience for customers and could cause privacy
issues if data are not properly anonymized. In application, the model should
be used to avoid exploitation or leakage of customers personal profiles and
preferences.

##  7 Conclusion

We proposed a pretraining framework, CERES, for learning representations for
semi-structured e-commerce sessions. We are the first to jointly model intra-
item text and inter-item relations in session graphs with an end-to-end
pretraining framework. By modeling Graph-Conditioned Masked Language Modeling,
our model is encouraged to learn high-quality representations for both intra-
item and inter-item information during its pretraining on massive unlabeled
session graphs. Furthermore, as a generic session encoder, our model enabled
effective leverage of session information in downstream tasks. We conducted
extensive experiments and ablation studies on CERES in comparison to state-of-
the-art pretrained models and recommendation systems. Experiments show that
CERES can produce higher quality text representations as well as better
leverage of session graph structure, which are important to many e-commerce
related tasks, including product search, query search, and query
understanding.

## References

  * Clark et al. (2020) Kevin Clark, Minh-Thang Luong, Quoc V Le, and Christopher D Manning. 2020.  Electra: Pre-training text encoders as discriminators rather than generators.  _arXiv preprint arXiv:2003.10555_.
  * Cucerzan and White (2007) Silviu Cucerzan and Ryen W White. 2007.  Query suggestion based on user landing pages.  In _Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval_ , pages 875–876.
  * Dehghani et al. (2017) Mostafa Dehghani, Sascha Rothe, Enrique Alfonseca, and Pascal Fleury. 2017.  Learning to attend, copy, and generate for session-based query suggestion.  In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_ , pages 1747–1756.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Gupta et al. (2019) Priyanka Gupta, Diksha Garg, Pankaj Malhotra, Lovekesh Vig, and Gautam M Shroff. 2019.  Niser: Normalized item and session representations with graph neural networks.  _arXiv preprint arXiv:1909.04276_.
  * Hu et al. (2019a) Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. 2019a.  Pre-training graph neural networks.  _arXiv preprint arXiv:1905.12265_.
  * Hu et al. (2019b) Weihua Hu, Bowen Liu, Joseph Gomes, Marinka Zitnik, Percy Liang, Vijay Pande, and Jure Leskovec. 2019b.  Strategies for pre-training graph neural networks.  _arXiv preprint arXiv:1905.12265_.
  * Hu et al. (2020) Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun. 2020.  Gpt-gnn: Generative pre-training of graph neural networks.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 1857–1867.
  * Jannach and Ludewig (2017) Dietmar Jannach and Malte Ludewig. 2017.  When recurrent neural networks meet the neighborhood for session-based recommendation.  In _Proceedings of the Eleventh ACM Conference on Recommender Systems_ , pages 306–310.
  * Kallumadi et al. (2021) Surya Kallumadi, Tracy Holloway King, Shervin Malmasi, and Maarten de Rijke. 2021\.  Ecom’21: The sigir 2021 workshop on ecommerce.  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 2685–2688.
  * Lan et al. (2019) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019.  Albert: A lite bert for self-supervised learning of language representations.  _arXiv preprint arXiv:1909.11942_.
  * Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019.  Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  _arXiv preprint arXiv:1910.13461_.
  * Li et al. (2017) Jing Li, Pengjie Ren, Zhumin Chen, Zhaochun Ren, Tao Lian, and Jun Ma. 2017.  Neural attentive session-based recommendation.  In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_ , pages 1419–1428.
  * Liu et al. (2020) Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. 2020.  K-bert: Enabling language representation with knowledge graph.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 2901–2908.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Mobasher et al. (2001) Bamshad Mobasher, Honghua Dai, Tao Luo, and Miki Nakagawa. 2001.  Effective personalization based on association rule discovery from web usage data.  In _Proceedings of the 3rd international workshop on Web information and data management_ , pages 9–15.
  * Qiu et al. (2020a) Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang. 2020a.  Gcc: Graph contrastive coding for graph neural network pre-training.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 1150–1160.
  * Qiu et al. (2020b) Ruihong Qiu, Zi Huang, Jingjing Li, and Hongzhi Yin. 2020b.  Exploiting cross-session information for session-based recommendation with graph neural networks.  _ACM Transactions on Information Systems (TOIS)_ , 38(3):1–23.
  * Rendle et al. (2010) Steffen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thieme. 2010.  Factorizing personalized markov chains for next-basket recommendation.  In _Proceedings of the 19th international conference on World wide web_ , pages 811–820.
  * Ruihong et al. (2021) Qiu Ruihong, Huang Zi, Chen Tong, and Yin Hongzhi. 2021.  Exploiting positional information for session-based recommendation.  _arXiv preprint arXiv:2107.00846_.
  * Shang et al. (2019) Junyuan Shang, Tengfei Ma, Cao Xiao, and Jimeng Sun. 2019.  Pre-training of graph augmented transformers for medication recommendation.  _arXiv preprint arXiv:1906.00346_.
  * Shen et al. (2020) Tao Shen, Yi Mao, Pengcheng He, Guodong Long, Adam Trischler, and Weizhu Chen. 2020\.  Exploiting structured knowledge in text via graph-guided representation learning.  _arXiv preprint arXiv:2004.14224_.
  * Shoeybi et al. (2019) Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019.  Megatron-lm: Training multi-billion parameter language models using model parallelism.  _arXiv preprint arXiv:1909.08053_.
  * Sun et al. (2021) Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, et al. 2021.  Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation.  _arXiv preprint arXiv:2107.02137_.
  * Sun et al. (2020) Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2020.  Ernie 2.0: A continual pre-training framework for language understanding.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 8968–8975.
  * Wang et al. (2018) Shoujin Wang, Liang Hu, Longbing Cao, Xiaoshui Huang, Defu Lian, and Wei Liu. 2018\.  Attention-based transactional context embedding for next-item recommendation.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 32.
  * Wang et al. (2020) Ziyang Wang, Wei Wei, Gao Cong, Xiao-Li Li, Xian-Ling Mao, and Minghui Qiu. 2020\.  Global context enhanced graph neural networks for session-based recommendation.  In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_ , pages 169–178.
  * Wu et al. (2019a) Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. 2019a.  Session-based recommendation with graph neural networks.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 346–353.
  * Wu et al. (2019b) Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. 2019b.  Session-based recommendation with graph neural networks.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 33, pages 346–353.
  * Xu et al. (2019) Chengfeng Xu, Pengpeng Zhao, Yanchi Liu, Victor S Sheng, Jiajie Xu, Fuzhen Zhuang, Junhua Fang, and Xiaofang Zhou. 2019.  Graph contextualized self-attention network for session-based recommendation.  In _IJCAI_ , volume 19, pages 3940–3946.
  * Yao et al. (2019) Liang Yao, Chengsheng Mao, and Yuan Luo. 2019.  Kg-bert: Bert for knowledge graph completion.  _arXiv preprint arXiv:1909.03193_.
  * Yap et al. (2012) Ghim-Eng Yap, Xiao-Li Li, and S Yu Philip. 2012.  Effective next-items recommendation via personalized sequential pattern mining.  In _International conference on database systems for advanced applications_ , pages 48–64. Springer.
  * You et al. (2019) Jiaxuan You, Yichen Wang, Aditya Pal, Pong Eksombatchai, Chuck Rosenburg, and Jure Leskovec. 2019.  Hierarchical temporal convolutional networks for dynamic recommender systems.  In _The world wide web conference_ , pages 2236–2246.
  * You et al. (2020) Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. 2020.  Graph contrastive learning with augmentations.  _Advances in Neural Information Processing Systems_ , 33:5812–5823.
  * Zhang et al. (2020) Hanchu Zhang, Leonhard Hennig, Christoph Alt, Changjian Hu, Yao Meng, and Chao Wang. 2020.  Bootstrapping named entity recognition in e-commerce with positive unlabeled learning.  _arXiv preprint arXiv:2005.11075_.

##  Appendix A Details on Session Data

###  A.1 Product Attributes.

A product is represented with a table of attributes.

Attribute | Value
---|---
Title | Chemex Pour-over Coffee Maker
Bullet Description | Just coffee maker.
Color | N/A
Brand | Chemex
Manufacturer | Chemex
Product Type | Coffee Maker
Table 4: Example Product Table. Each product is guaranteed to have a title.
Most products have bullet descriptions, which can be split into multiple
entries. Products could have other attributes, such as color, brand, product
type, etc. as well.

Each product is guaranteed to have a product title and bullet description. In
this paper, we regard the product title as the representative sequence of the
product, called “product sequence”. A product may have additional attributes,
such as product type, color, brand, and manufacturer, depending on specific
products.

###  A.2 Alternative Pretraining Corpora

In this section we introduce alternative pretraining corpora that encode
information in a session, including products and queries, but not treating
sessions as a whole.

####  A.2.1 Product Corpus

In this corpus, we gathered all product information that appeared in the
sessions from August 2020 to September 2020. Each product will have
descriptions such as _product title_ and _bullet description_ , and other
attributes like _entity type_ , _product type_ , _manufacturer_ , etc.
Particularly, bullet description is composed of several lines of descriptive
facts about the product. All products without titles are removed. Each of the
remaining product forms a paragraph, where the product title comes as the
first sentence, followed by the entries of bullet descriptions each as a
sentence, and product attributes.

An example document in this corpora is as follows:

        [Title] Product Title
        [Bullet Description] Description bullet 1
        [Bullet Description] Description bullet 2
        [Product Type] Product Type
        [Color] Color

####  A.2.2 Single-Query Single-Product (SQSP) Corpus

In this corpus, we treat each session as a document and each query-product
pair as a sentence. A query-product pair in the document are the pairs of
queries and products that are either viewed or clicked with the given queries.
A query-product pair looks like the follows:

        [SEARCH] search keywords [TITLE] product title
        [BULLET_DESCRIPTION] description
        [ENTITY_TYPE] entity type

where the first `[SEARCH]` special token indicates a field of query keywords,
and `[TITLE]` indicates fields of product information starting with product
tittles. In this corpus, we model the one-to-one relation between queries and
products.

####  A.2.3 Session Corpus

In this corpus, we treat each session as a document and sequentially put text
representations of items in a session to the document with special tokens
indicating the fields of items. An example document looks like the follows:

        [SEARCH] keywords 1 [SEARCH] keywords 2 [CLICK]
        [TITLE] product 1 [SEARCH] keywords 3 [PURCHASE]
        [TITLE] product 2

In this example, the customer first attempted to search with _keywords 1_ and
then modified the keywords to _keywords 2_. The customer then clicked on
_product 1_. At last, the customer modified his search to _keywords 3_ and
purchased _product 2_. In this corpus, session information is present in a
document, but the specific relations between elements are not specified. The
comparison of different datasets are in Table 5.

Corpus | Product Info | Query Info | Relational | Session Context
---|---|---|---|---
Product | ✓ | ✗ | ✗ | ✗
SQSP | ✓ | ✓ | ✓ | ✗
Session-Corpus | ✓ | ✓ | ✗ | ✓
Session-Graph | ✓ | ✓ | ✓ | ✓
Table 5: Comparision of different pretraining dataset. Product Corpus has
access only to product information. SQSP models on the queries and query-
product relations, without access to session context. Session Corpus has
access to contextual information in a session, but does not model on relations
between objects. Session-Graph has access to all information and models on the
relational nature of nodes in the session graph.

###  A.3 Alternative Pretraining Methods

We introduce the alternative pretraining models.

  * •

Product-Bert. It is pretrained on the Product Corpus. Specifically, we treat
each product in the Product Corpus as an article. Product titles is always the
first sentence, followed by paragraphs of bullet descriptions, which can
contain multiple sentences. Then, each additional product attribute is a
sentence added after the bullet descriptions.

Product Bert is trained for 300,000 steps, with a 12-layer transformer with a
batch size of 6144 and peak learning rate of 1e-3, 1% linear warm-up steps,
and $1e-2$ linear weight decay to a minimum learning rate of 1e-5.

  * •

SQSP-Bert. It is pretrained on SQSP Corpus. The SQSP Bert uses the same
Transformer backbone as Product Bert. Given each query-product pair, SQSP
feeds the text pair sequence to the Transformer for token embeddings for
masked language modeling loss. In addition to language modeling, for each
query-product pair, we sample a random product for the query as a negative
query-product pair. The text pair sequence of the negative sample is also fed
to the Transformer. Then, a discriminator is trained in the pretraining stage
to distinguish the ground-truth query-product pairs and randomly sampled
pairs. The discriminator’s classification loss should serve as a contrastive
loss.

SQSP Bert is trained with the same configuration of Product Bert.

##  Appendix B Details on Evaluation Metrics

##### Mean Average Precision.

Suppose that for a session, _m_ items are relevant and _N_ items are retrieved
by the model, the _Average Precision_ (AP) of a session is defined as

$\mathrm{AP}@\mathrm{N}=\frac{1}{\min(m,N)}\sum_{k=1}^{N}P(k)\mathrm{rel}(k),$
(4)

where $P(k)$ is the precision of the top _k_ retrieved items, and $rel(k)$ is
an indicator function of whether the $k$th item is relevant. As we have at
most one relevant item for each session, the above metric reduces to
$\frac{1}{r}$, where $r$ is the rank of the relevant item in the retrieved
list, and $k=\infty$ when the relevant item is not retrieved. MAP@N averages
AP@N over all sessions,

$\mathrm{MAP}@\mathrm{N}=\frac{1}{|\mathcal{S}|}\sum_{s\in\mathcal{S}}\frac{1}{r_{s}}$
(5)

where $r_{s}$ is the rank of the relevant item for a specific session $s$. MAP
in this case is equivalent to MRR.

##### Mean Average Precision by Queries (MAPQ).

Different from MAP, MAPQ averages AP over last queries instead of sessions.
Suppose $\mathcal{Q}$ is the set of unique last queries, and
$S(q),q\in\mathcal{Q}$ is the set of sessions whose last queries are $q$, then
the average precision for one query $q$ is

$\mathrm{APQ}@\mathrm{N}=\frac{1}{\sum_{i=1}^{k}\mathrm{rel}(k)}\sum_{k=1}^{N}\min(1,\frac{\sum_{r_{s}\leq
k}\mathrm{rel}(k)}{k})$ (6)

then we sum over all queries to obtain MAPQ@N.

##### Mean Reciprocal Rank by Queries (MRRQ).

MRRQ averages MRR over session last queries instead of sessions.

$MRRQ@N=\frac{1}{|\mathcal{Q}|}\sum_{q\in\mathcal{Q}}\max_{s\in S(q)}(r_{s})$
(7)

##### Recall.

Recall@N calculates the percentage of sessions whose relevant items were
retrieved among the top N predictions.


# Automatic Speech Recognition for Biomedical Data in Bengali Language

Shariar Kabir
MedAI Limited
<EMAIL_ADDRESS>
Nazmun Nahar
MedAI Limited
<EMAIL_ADDRESS>
Shyamasree Saha
MedAI Limited
<EMAIL_ADDRESS>Mamunur Rashid
University of Birmingham
<EMAIL_ADDRESS>
Corresponding author.Corresponding author.

###### Abstract

This paper presents the development of a prototype Automatic Speech
Recognition (ASR) system specifically designed for Bengali biomedical data.
Recent advancements in Bengali ASR are encouraging, but a lack of domain-
specific data limits the creation of practical healthcare ASR models. This
project bridges this gap by developing an ASR system tailored for Bengali
medical terms like symptoms, severity levels, and diseases, encompassing two
major dialects: Bengali and Sylheti. We train and evaluate two popular ASR
frameworks on a comprehensive 46-hour Bengali medical corpus. Our core
objective is to create deployable health-domain ASR systems for digital health
applications, ultimately increasing accessibility for non-technical users in
the healthcare sector.

_K_ eywords Automatic Speech Recognition $\cdot$ Biomedical Data $\cdot$
Bengali Language

##  1 Introduction

Recent advancements in domain specific Automated Speech Recognition (ASR) and
Large Language Models (LLM), have significantly boosted the adoption of AI in
digital services across many different industries such as financial service,
healthcare. In the healthcare industry in particular, integration of AI-driven
solutions such as conversational chatbots, voice interactive guidance is
opening new avenues to engage patients and healthcare providers ([1], [2]).
Many healthcare systems in the developed world have been adopting these
systems to increase patient satisfaction. One key shortcomings in this is that
the majority of the developments in this domain are focused towards patients
of European descent, their medical vocabularies. Many non-European languages,
though spoken by millions, have seen very limited advancements. Bengali,
despite being the seventh most popular language with 270 million speakers
worldwide, has seen very limited progress in Bengali NLP and ASR research.
This has hindered the integration of these technologies into digital health
services for Bengali speakers which in turn slowed down the adoption of
digital health solutions. While many European language speakers are benefiting
from AI-driven services (conversational chatbot assisted) like digital
appointment booking, symptom reporting before appointment and mental health
support, Bengalis speakers are not able to benefit from these advancements.

Bengali ASR research has seen a significant surge in recent years, fueled by
the release of large public speech corpora like Google’s "Large Bengali ASR
training data" (LB-ASRTD). This rich dataset has empowered researchers to
explore advanced deep learning techniques like Baidu’s Deep Speech 2 (DS2) and
transfer learning approaches, yielding promising results with improved word
error rates [3]. The focus extends beyond deep learning architectures, with
studies investigating feature engineering advancements to enhance Bengali ASR
performance. Hasan M. et al. explored the effectiveness of improved Mel-
frequency cepstral coefficients (MFCCs) combined with deep Long Short-Term
Memory (LSTM) networks for acoustic modeling. They further compare decoding
strategies using a combination of Connectionist Temporal Classification (CTC)
and a statistical language model (LM) with a CTC-based greedy decoder [4].
Additionally, Himadri Mukherjee et al. propose a Bangla phoneme recognition
system using a novel feature extraction method called LPCC-2, achieving high
accuracy on vowel phoneme recognition. This demonstrates the potential of
improved feature representations for Bengali ASR tasks [5].

Domain-specific automated speech recognition (ASR) models for medical
applications in high-resource languages like English have shown significant
promise in boosting healthcare delivery efficiency. These models achieve
greater accuracy compared to generic ASR models due to their tailored focus on
medical terminology and vocabulary, and have demonstrably enhanced various
healthcare functionalities [6], [7].

While recent advancements are in Bengali ASR are promising, significant
hurdles remain before it can be effectively deployed in critical real-world
domains such as healthcare. One major challenge is the presence of numerous
dialects and the influence of Sanskrit vocabulary create ambiguity for ASR
systems, as words may be pronounced differently or share similar
pronunciations with distinct meanings. This can lead to errors in interpreting
spoken language, particularly in a sensitive domains such as healthcare, where
misinterpretations can have serious consequences. This is further compounded
by lack of domain specific quality labeled data. Unlike English and many
European languages where vast amounts of conversation data for exist from
sources like health insurance and healthcare providers, Bengali currently
lacks such a comprehensive resource. This data scarcity hinders the
development of robust ASR models that can accurately recognize medical
terminology and nuances of healthcare conversations.

Medical domain-specific ASR systems hold immense potential for expanding
digital health systems in LMIC countries like Bangladesh, which suffers from a
low patient-doctor ratio.

Voice-interactive remote patient triage: AI-driven remote patient triage via
digital health applications has gained significant popularity in recent years.
These applications collect patients’ symptoms before appointments to reduce
consultation time [8] A Bengali medical ASR system can significantly improve
accessibility for people with low digital literacy, removing a current barrier
to these advancements.

Transcription of doctor-patient consultations: With a shortage of trained
doctors, Bangladesh has one of the shortest average medical consultation
durations globally [9]. ASR systems offer a solution by transcribing and
summarizing doctor-patient conversations, automatically increasing efficiency.

Development of voice-based diagnostic tools: ASR has the potential to play a
crucial role in developing tools that analyze speech patterns for early
detection of certain health conditions.

Accessibility for Low Digital Literacy Users: The healthcare sector shouldn’t
be limited to tech-savvy individuals. ASR can empower people with disabilities
or limited mobility by enabling interaction with medical equipment through
voice commands.

This paper presents a proof-of-concept automated speech recognition (ASR)
system specifically designed for Bengali biomedical data. We evaluate the
performance of two popular ASR frameworks: DeepSpeech2 [10] and a fine-tuned
Whisper BanglaASR. Both models are trained on a comprehensive Bengali medical
corpus encompassing disease names, symptoms, and symptom severity. This corpus
comprises 57.59 hours of data. Notably, both models achieve significant
improvements in word error rate (WER) when transcribing medical conversations,
with WER of 17.25% and 9.05% on a 5.8 hours of held-out test dataset. By
addressing these challenges, we aim to bridge the gap in AI accessibility for
the Bengali language and pave the way for the development of inclusive AI-
powered healthcare solutions.

Model availability: The fine-tuned Whisper Bangla medical ASR model is
currently deployed in AmarDoctor platform. Details of model testing data is
available at: http://amardoctor.health/banglamedasr.html

##  2 Methodology

###  2.1 Data Collection

Our data collection process involved gathering various types of audio data
related to medical symptoms, specifically designed to create a diverse and
comprehensive dataset for Bengali medical ASR. This section provides a
detailed overview of the data collection methodology:

We collected data from three distinct sources:

Mapped Medical Symptoms: We mapped 1,264 unique English medical symptoms to
Bengali and Sylheti dialects 1. These symptoms were then incorporated into
short, colloquial sentences. Audio recordings of human participants uttering
these sentences were collected. Participants were recruited from a diverse
range of demographics, encompassing gender (male and female) and age groups
(young, middle-aged, and older adults).

Synthetic Speech Data: In addition to human recordings, we also collected
synthetic data for the same sentences using Google Text-to-Speech. This
approach helps to expand the dataset and introduces additional variation in
pronunciation.

Simulated Medical Conversations: We further enriched the dataset by creating
artificial medical scenarios. With informed consent, de-identified audio data
was collected from real patient-doctor conversations based on these scenarios.
This approach helps the model better understand the nuances of natural medical
speech.

By combining these three data sources, we aimed to create a robust and
versatile dataset that can effectively train our Bengali medical ASR system.
The overall composition of our dataset is illustrated in Figure 1. This figure
highlights the distribution of the data across different categories and
languages, showcasing the diverse nature of the collected audio recordings.

####  2.1.1 Data Sources

We collected audio data from three primary sources:

  * •

Human Recorded Symptom Data: This dataset consists of recordings of medical
symptoms spoken by native Bengali speakers. The recordings were further
divided into two subsets based on the accent:

    * –

Standard Bengali: Comprising 34.73 hours of audio data across 36,982 files.

    * –

Sylheti Bengali: Consisting of 15.72 hours of audio data across 20,076 files,
capturing the regional accent variations.

  * •

Synthetic Data: This dataset was generated using the Google Text-to-Speech API
to synthesize medical symptom audio recordings. It includes 6.45 hours of
audio data across 8,982 files.

  * •

Doctor-Patient Conversation Data: We also collected 177 audio recordings of
actual doctor-patient conversations, amounting to 0.68 hours. These recordings
provide real-world interaction scenarios and enrich the dataset with natural
dialogue patterns.

####  2.1.2 Data Collection Summary

Table 1 summarizes the collected datasets, including the number of files and
the total duration of each dataset.

Dataset | Number of Files | Duration (hours)
---|---|---
Standard Bengali | 36,982 | 34.73
Sylheti Bengali | 20,076 | 15.72
Synthetic Data | 8,982 | 6.45
Doctor-Patient Conversations | 177 | 0.68
Total | 66217 | 57.59
Table 1: Summary of collected datasets Figure 1: Distribution of the collected
audio datasets.

####  2.1.3 Data Statistics

Table 2 shows an analysis of the statistics of the audio files within our
datasets:

Mean audio length | 3.130898 seconds
---|---
Standard deviation of audio length | 1.725558 seconds
Shortest audio length | 0.600000 seconds
25% | 2.160000 seconds
50% | 2.820000 seconds
75% | 3.744000 seconds
Longest audio length | 150.048438 seconds
Table 2: Basic statistics of collected data

This comprehensive data collection effort ensures that our dataset is well-
suited for training and evaluating automated speech recognition systems in
Bengali, with robust representation across different accents, recording
methods, and interaction types.

###  2.2 Data Preprocessing and Training

Our data preprocessing and training pipeline involved several critical steps
to ensure the quality and effectiveness of the automatic speech recognition
(ASR) models. This section details the preprocessing methods and training
procedures for both the DeepSpeech and Whisper models.

####  2.2.1 Data Preprocessing

Prior to model training (DeepSpeech) or fine-tuning (Whisper), the collected
audio data underwent a multi-step processing pipeline. Figure 2 depicts the
multi-step preprocessing pipeline employed to process the different datasets
and our training architecture.

Figure 2: Data preprocessing and training pipeline for DeepSpeech and Whisper
models.

##### Audio Conversion

All audio files were converted to 16 kHz single-channel WAV format. This
standardization ensures uniformity across the dataset, reduces computational
load, and maintains the audio quality necessary for effective speech
recognition. The conversion to mono channel simplifies the audio input for the
models, focusing on a single source of sound.

##### Transcription Normalization

Transcriptions were normalized by removing any punctuation marks. This step
helps in creating clean and consistent text data for training, which is
crucial for the ASR models to learn accurate mappings from audio to text. Each
audio file contains a single symptom spoken by one speaker/patient, ensuring
that the dataset remains focused and consistent.

##### Alphabet Configuration

An ‘alphabets.csv‘ file was used to define the set of characters used by the
DeepSpeech model. This file includes all possible characters present in the
transcriptions, ensuring that the model can handle the full range of phonetic
and linguistic variations in Bengali.

####  2.2.2 Noise Augmentation

To enhance the robustness of DeepSpeech, various noise augmentation techniques
were employed. These are explained in Table 3.

Name | Description | Probability
---|---|---
Overlay augmentation | Layers another audio source onto the samples | 50%
Warp augmentation | Applies a non-linear image warp to the spectrogram | 10%
Reverb augmentation | Adds simplified Schroeder reverberation to the samples | 10%
Frequency mask augmentation [11] | Sets random frequency intervals to zero | 10%
Resample augmentation | Resamples the samples to another rate and back | 10%
Time mask augmentation | Sets random time intervals to zero | 10%
Codec augmentation | Compresses and decompresses using Opus codec | 10%
Dropout augmentation | Randomly drops segments | 10%
Volume augmentation | Levels samples to a target dBFS value | 10%
Pitch augmentation | Changes pitch by scaling spectrogram on the frequency axis | 10%
Tempo augmentation | Changes playback tempo by scaling spectrogram on the time axis | 10%
Table 3: Applied Noise Augmentations

####  2.2.3 Model Training

##### DeepSpeech:

DeepSpeech, developed by Baidu lab [10], was employed for training the ASR
model. The architecture includes a Deep Neural Network (DNN) that converts
audio into a sequence of probabilities over characters, followed by an N-gram
language model (KenLM) [12] that converts these probabilities into a sequence
of characters.

  * •

Training Data: Our dataset was divided into training and validation sets, with
specific focus on Symptom Data, Sylhet Data, and Synthetic Data.

  * •

KenLM Scorer: The language model was built using transcriptions of our symptom
data to improve accuracy. We evaluated the model’s performance with and
without the KenLM scorer.

##### Whisper:

Whisper [13] is an advanced ASR model trained on 680,000 hours of multilingual
and multitask supervised data collected from the web. We fine-tuned the
Whisper model using the Mozilla Common Voice Bengali Dataset (MCVB) [14],
leveraging its robustness to accents, background noise, and technical
language.

  * •

Model Variant: The Whisper small variant, with 244 million parameters, was
selected for its balance between performance and computational efficiency.

  * •

Training Process: The model was fine-tuned over 4K steps using 53K training
samples and 6.6K validation samples, resulting in a word error rate (WER) of
3.22% on the validation set.

##  3 Experimental Results

This section presents the Word Error Rate (WER) achieved by each model on
various test datasets, including Symptom Data, Synthetic Data, Sylhet Data,
Noisy Audio Data, and a Mixed Dataset (Combined Common Voice & Symptom Data).
WER is a common metric used in ASR tasks, reflecting the percentage of words
incorrectly recognized by the model. Lower WER indicates better performance.

Performance on Domain-Specific Data:

Both DeepSpeech2 and fine-tuned Whisper BanglaASR significantly outperformed
generic Bengali ASR models on the Symptom Data, as expected. Wav2vec2
displayed the highest WER (most errors) in Sylheti Data (93.4%), likely due to
its dialectal limitations. Conversely, it achieved the lowest WER (fewest
errors) in Synthetic Data (61.36%), which may be attributed to the cleaner and
controlled nature of synthetic speech.

DeepSpeech2 performed exceptionally well in Synthetic Data with a WER of only
2.07%, demonstrating its proficiency in transcribing clear speech. However, it
faced challenges with Sylheti Data, resulting in a higher WER of 22.30%. This
highlights the model’s sensitivity to dialectal variations.

Whisper, when trained solely on generic Bengali data from Common Voice,
struggled significantly with Sylheti Data, registering a WER of 110.01%. This
suggests that a generic model may not generalize well to specific domains or
dialects without further adaptation. However, Whisper showed improved
performance in Synthetic Data (46.46%), indicating its potential with cleaner
audio.

Performance Across Different Data Types:

When evaluating the models across the Mixed Dataset, Whisper achieved its best
result in Synthetic Data (1.05%), further confirming its strength with clean
speech. Conversely, its performance dropped considerably in Noisy Audio Data
(20.21%), suggesting a need for further noise reduction techniques.

These results highlight the importance of using domain-specific data for
training ASR models. While DeepSpeech2 and fine-tuned Whisper achieved better
overall performance with medical data, both models still face challenges with
dialectal variations and noisy audio.

Table 4 presents the Word Error Rate (WER) for each model on the Symptom Data,
Synthetic Data, and Sylhet Data, as well as the overall WER.

Dataset |  wav2vec2 on MCVB data |  DeepSpeech2 on MCVB data & Symptom |  Whisper on MCVB data |  Whisper on MCVB & Symptom Data
---|---|---|---|---
Symptom Data |  71.35% |  18.20% |  58.74% |  7.83%
Synthetic Data |  61.36% |  2.07% |  46.46% |  1.05%
Sylhet Data |  93.45% |  22.30% |  110.01% |  14.89%
Overall |  76.71% |  17.25% |  72.68% |  9.05%
Noisy Audio |  89.91% |  41.7 % |  73.69% |  20.21 %
Table 4: WER performance of DeepSpeech and Whisper models on different
datasets

##  4 Discussion and Future Work

The development of robust Automated Speech Recognition (ASR) systems for
Bengali is crucial for improving accessibility and enabling AI-powered
healthcare solutions in this region. Our work involves training and fine-
tuning DeepSpeech and Whisper models on diverse Bengali speech datasets,
including those with medical symptoms and different dialects. In this section,
we discuss the performance of these models, particularly their handling of
noisy audio, and outline future work to enhance their capabilities.

###  4.1 Noisy Audio Issue

One of the key hurdles hindering real-world deployment is the limited
capability of both models to handle noisy audio data. While during DeepSpeech
model training we have employed an extensive noise augmentation on training
data, noise patterns available in these augmentation libraries were primarily
reflecting noise patterns prevalent in standard developed countries, which
often differ significantly from those encountered in South Asian environments.
Consequently, neither model performed well on real conversation data from
Bangladesh due to the mismatch between training noise and real-world noise
profiles. Whisper, however, exhibited a slight advantage over DeepSpeech in
noisy scenarios. This suggests potential for improvement by incorporating more
real-world noisy data from Bangladesh during the fine-tuning process.

###  4.2 Whisper Offers Superior Performance

Among the evaluated models, Whisper fine-tuned on medical symptom data emerged
as the clear leader. This superiority stems from its robust training on a vast
multilingual and multi-task dataset, allowing it to handle diverse accents,
noises, and speech patterns. Additionally, its training on a massive dataset
(680,000 hours) encompassing various background noises contributes to its
resilience in noisy environments. Finally, the significant performance
improvement of the Whisper model fine-tuned on both MCVB (presumably Bengali
speech data) and medical data compared to the generic data model underscores
the critical role of domain-specific data for optimal ASR performance.

However, we still consider carrying forward the DeepSpeech model for several
reasons:

  * •

Customizability: DeepSpeech is highly customizable. We have full access to its
architecture, allowing us to tailor the model to the specific requirements of
our application.

  * •

Licensing: There are no licensing concerns with DeepSpeech, making it a viable
option for commercialization and integration into proprietary systems.

  * •

Control Over Training: Although DeepSpeech requires training from scratch,
which demands more preprocessing and larger amounts of data, this process
allows for fine-grained control over the training parameters and dataset
composition.

Despite these advantages, it is clear that Whisper’s pretraining on a massive
dataset gives it a performance edge. As such, Whisper, being pretrained on a
huge amount of data, will likely always outperform DeepSpeech in general
scenarios.

###  4.3 Key Limitations

This work emphasizes the critical role of domain-specific data in enhancing
Bengali medical ASR performance. However, several limitations warrant further
exploration. Firstly, noise resilience necessitates improvement, as real-world
healthcare environments in developing countries frequently present
acoustically challenging conditions. Secondly, the model exhibits limitations
in handling extended conversational speech. Because current models are trained
with short medical symptom descriptions, they are not adequately equipped to
deal with longer conversations (Figure 3, example 4). As a result, their
application is limited in healthcare applications such as medical conversation
chatbots. To bridge this gap, future endeavors should focus on incorporating
robust noise reduction techniques and expanding the training data with
extensive real-world conversational data collected from healthcare settings.

Figure 3: Example speech to text output of the compared models on held out
test set.

###  4.4 Future Work

Our future work aims to enhance the performance of both DeepSpeech and Whisper
models, particularly in noisy environments and with diverse Bengali dialects:

  * •

Improved Noise Modeling: We will train DeepSpeech with better noise modeling
techniques specifically designed for Bangladeshi environments. This involves
collecting and profiling noise samples typical to Bangladeshi households and
outdoor settings and incorporating these into the training process.

  * •

Fine-Tuning Whisper with Longer Conversation Data: We plan to fine-tune
Whisper with additional lengthy conversation data and noisy audio data from
real-world Bangladeshi settings to further improve its robustness and accuracy
in noisy conditions.

##  5 Conclusion

This work lays the foundation for a Bengali biomedical speech recognition
system. By addressing dialectal variations and accessibility concerns, we aim
to empower users with low digital literacy and pave the way for the
development of inclusive AI-powered healthcare solutions in the Bengali
language.

## References

  * [1] Shuroug A. Alowais, Sahar S. Alghamdi, Nada Alsuhebany, Tariq Alqahtani, Abdulrahman I. Alshaya, Sumaya N. Almohareb, Atheer Aldairem, Mohammed Alrashed, Khalid Bin Saleh, Hisham A. Badreldin, Majed S. Al Yami, Shmeylan Al Harbi, and Abdulkareem M. Albekairy.  Revolutionizing healthcare: the role of artificial intelligence in clinical practice.  BMC Medical Education 2023 23:1, 23:1–15, 9 2023.
  * [2] Silvana Secinaro, Davide Calandra, Aurelio Secinaro, Vivek Muthurangu, and Paolo Biancone.  The role of artificial intelligence in healthcare: a structured literature review.  BMC medical informatics and decision making, 21, 12 2021.
  * [3] Tushar Talukder Showrav.  An automatic speech recognition system for bengali language based on wav2vec2 and transfer learning.
  * [4] Towards lexicon-free bangla automatic speech recognition system.  2019 International Conference on Bangla Speech and Language Processing, ICBSLP 2019, 9 2019.
  * [5] Himadri Mukherjee, Santanu Phadikar, and Kaushik Roy.  An ensemble learning-based bangla phoneme recognition system using lpcc-2 features.  Advances in Intelligent Systems and Computing, 695:61–69, 2018.
  * [6] The impact of implementing speech recognition technology on the accuracy and efficiency (time to complete) clinical documentation by nurses: A systematic review.  Journal of Clinical Nursing, 29:2125–2137, 7 2020.
  * [7] Suzanne V. Blackley, Valerie D. Schubert, Foster R. Goss, Wasim Al Assad, Pamela M. Garabedian, and Li Zhou.  Physician use of speech recognition versus typing in clinical documentation: A controlled observational study.  International Journal of Medical Informatics, 141:104178, 9 2020.
  * [8] Stephen Gilbert, Alicia Mehl, Adel Baluch, Caoimhe Cawley, Jean Challiner, Hamish Fraser, Elizabeth Millen, Maryam Montazeri, Jan Multmeier, Fiona Pick, Claudia Richter, Ewelina Türk, Shubhanan Upadhyay, Vishaal Virani, Nicola Vona, Paul Wicks, and Claire Novorol.  How accurate are digital symptom assessment apps for suggesting conditions and urgency advice? a clinical vignettes comparison to gps.  BMJ Open, 10:e040269, 12 2020.
  * [9] Greg Irving, Ana Luisa Neves, Hajira Dambha-Miller, Ai Oishi, Hiroko Tagashira, Anistasiya Verho, and John Holden.  International variations in primary care physician consultation time: a systematic review of 67 countries.  BMJ Open, 7:e017902, 10 2017.
  * [10] Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, Jie Chen, Jingdong Chen, Zhijie Chen, Mike Chrzanowski, Adam Coates, Greg Diamos, Ke Ding, Niandong Du, Erich Elsen, Jesse Engel, Weiwei Fang, Linxi Fan, Christopher Fougner, Liang Gao, Caixia Gong, Aw Ni Hannun, Tony Han, Lappi Vaino Johannes, Bing Jiang, Cai Ju, Billy Jun, Patrick Legresley, Libby Lin, Junjie Liu, Yang Liu, Weigao Li, Xiangang Li, Dongpeng Ma, Sharan Narang, Andrew Ng, Sherjil Ozair, Yiping Peng, Ryan Prenger, Sheng Qian, Zongfeng Quan, Jonathan Raiman, Vinay Rao, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Kavya Srinet, Anuroop Sriram, Haiyuan Tang, Liliang Tang, Chong Wang, Jidong Wang, Kaifu Wang, Yi Wang, Zhijian Wang, Zhiqian Wang, Shuang Wu, Likai Wei, Bo Xiao, Wen Xie, Yan Xie, Dani Yogatama, Bin Yuan, Jun Zhan, and Zhenyao Zhu.  Deep speech 2: End-to-end speech recognition in english and mandarin.  33rd International Conference on Machine Learning, ICML 2016, 1:312–321, 12 2015.
  * [11] Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D Cubuk, and Quoc V Le.  Specaugment: A simple data augmentation method for automatic speech recognition.  arXiv preprint arXiv:1904.08779, 2019.
  * [12] Kenneth Heafield.  Kenlm: Faster and smaller language model queries.  In Proceedings of the sixth workshop on statistical machine translation, pages 187–197, 2011.
  * [13] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever.  Robust speech recognition via large-scale weak supervision.  In International Conference on Machine Learning, pages 28492–28518. PMLR, 2023.
  * [14] Md Saiful Islam.  Transformer based whisper bangla asr model, 2023.  Available online at: https://huggingface.co/bangla-speech-processing/BanglaASR, last accessed on 11.05.2024.