# Improving Contrastive Learning with Model Augmentation

Zhiwei Liu1 Yongjun Chen1 Jia Li1 Man Luo2 Philip S. Yu3 and Caiming Xiong1
1Salesforce Research, CA, USA
{zhiweiliu, yongjun.chen, jia.li<EMAIL_ADDRESS>
2Arizona State University, AZ, USA<EMAIL_ADDRESS>
3University of Illinois at Chicago, IL, USA<EMAIL_ADDRESS>
###### Abstract

The sequential recommendation aims at predicting the next items in user
behaviors, which can be solved by characterizing item relationships in
sequences. Due to the data sparsity and noise issues in sequences, a new self-
supervised learning (SSL) paradigm is proposed to improve the performance,
which employs contrastive learning between positive and negative views of
sequences. However, existing methods all construct views by adopting
augmentation from data perspectives, while we argue that 1) optimal data
augmentation methods are hard to devise, 2) data augmentation methods destroy
sequential correlations, and 3) data augmentation fails to incorporate
comprehensive self-supervised signals. Therefore, we investigate the
possibility of model augmentation to construct view pairs. We propose three
levels of model augmentation methods: neuron masking, layer dropping, and
encoder complementing. This work opens up a novel direction in constructing
views for contrastive SSL. Experiments verify the efficacy of model
augmentation for the SSL in the sequential recommendation. Code is
available111https://github.com/salesforce/SRMA.

##  1 Introduction

The sequential recommendation (Fan et al., 2021; Liu et al., 2021c; Chen et
al., 2018; Tang & Wang, 2018; Zheng et al., 2019) aims at predicting future
items in sequences, where the crucial part is to characterize item
relationships in sequences. Recent developments in sequence modeling (Fan et
al., 2021; Liu et al., 2021c) verify the superiority of Transform (Vaswani et
al., 2017), i.e. the self-attention mechanism, in revealing item correlations
in sequences. A Transformer (Kang & McAuley, 2018) is able to infer the
sequence embedding at specified positions by weighted aggregation of item
embeddings, where the weights are learned via self-attention. Existing works
(Fan et al., 2021; Wu et al., 2020) further improve Transformer by
incorporating additional complex signals.

However, the data sparsity issue (Liu et al., 2021c) and noise in sequences
undermine the performance of a model in sequential recommendation. The former
hinders performance due to insufficient training since the complex structure
of a sequential model requires a dense corpus to be adequately trained. The
latter also impedes the recommendation ability of a model because noisy item
sequences are unable to reveal actual item correlations. To overcome both, a
new contrastive self-supervised learning (SSL) paradigm (Liu et al., 2021b;
Xie et al., 2020; Zhou et al., 2020) is proposed recently. This paradigm
enhances the capacity of encoders by leveraging additional self-supervised
signals. Specifically, the SSL paradigm constructs positive view pairs as two
data augmentations from the same sequences (Xie et al., 2020), while negative
pairs are augmentations from distinct sequences. Incorporating augmentations
during training increases the amount of training data, thus alleviating the
sparsity issue. And the contrastive loss (Chen et al., 2020) improves the
robustness of the model, which endows a model with the ability to against
noise.

Though being effective in enhancing sequential modeling, the data augmentation
methods adopted in the existing SSL paradigm suffer from the following
weaknesses:

  * •

Optimal data augmentation methods are hard to devise. Current sequence
augmentation methods adopts random sequence perturbations (Liu et al., 2021b;
Xie et al., 2020), which includes crop, mask, reorder, substitute and insert
operations. Though a random combination of those augmenting operations
improves the performance, it is rather time-consuming to search the optimal
augmentation methods from a large number of potential combinations for
different datasets (Liu et al., 2021b).

  * •

Data augmentation methods destroy sequential correlations, leading to less
confident positive pairs. The existing SSL paradigm requires injecting
perturbations into the augmented views of sequences for contrastive learning.
However, because the view construction process is not optimized to
characterize sequential correlations, two views of one sequence may reveal
distinct item relationships, which should not be recognized as positive pairs.

  * •

Data augmentation fails to incorporate comprehensive self-supervised signals.
Current data augmentation methods are designed based on heuristics, which
already requires additional prior knowledge. Moreover, since the view
construction process is not optimized with the encoder, data augmentation may
only reveal partial self-supervised signals from data perspectives. Hence, we
should consider other types of views besides data augmentation.

Therefore, we investigate the possibility of model augmentation to construct
view pairs for contrastive learning, which functions as a complement to the
data augmentation methods. We hypothesis that injecting perturbations into the
encoder should enhance the self-supervised learning ability to existing
paradigms. The reasons are threefold: Firstly, model augmentation is jointly
trained with the optimization process, thus endows the end-to-end training
fashion. As such, it is easy to discover the optimal view pairs for
contrastive learning. Secondly, model augmentation constructs views without
manipulation to the original data, which leads to high confidence of positive
pairs. Last but not least, injecting perturbation into the encoder has
distinct characteristics to data augmentation, which should be an important
complement in constructing view pairs for existing self-supervised learning
scheme (Liu et al., 2021b; Zhou et al., 2020).

This work studies the model augmentation for a self-supervised sequential
recommendation from three levels: 1) neuron masking (dropout), which adopts
the dropout layer to randomly mask partial neurons in a layer. By operating
the dropout twice to one sequence, we can perturb the output of the embedding
from this layer, which thus constructs two views from model augmentation
perspective (Gao et al., 2021). 2) layer dropping. Compared with neuron masks,
we randomly drop a complete layer in the encoder to inject more perturbations.
By randomly dropping layers in an encoder twice, we construct two distinct
views. Intuitively, layer-drop augmentation enforces the contrast between deep
features and shallows features of the encoder. 3) encoder complementing, which
leverages other encoders to generate sequence embeddings. Encoder
complementing augmentation is able to fuse distinct sequential correlations
revealed by different types of encoders. For example, RNN-based sequence
encoder (Hidasi et al., 2015) can better characterize direct item transition
relationships, while Transformer-based sequence encoder models position-wise
sequential correlations. Though only investigating SSL for a sequential
recommendation, we remark that model augmentation methods can also be applied
in other SSL scenarios. The contributions are as follows:

  * •

We propose a new contrastive SSL paradigm for sequential recommendation by
constructing views from model augmentation, which is named as SRMA.

  * •

We introduce three levels of model augmentation methods for constructing view
pairs.

  * •

We discuss the effectiveness and conduct a comprehensive study of model
augmentations for the sequential recommendation.

  * •

We investigate the efficacy of different variants of model augmentation.

##  2 Related Work

###  2.1 Sequential Recommendation

Sequential recommendation predicts future items in user sequences by encoding
sequences while modeling item transition correlations (Rendle et al., 2010;
Hidasi et al., 2015). Previously, Recurrent Neural Network (RNN) have been
adapted to sequential recommendation (Hidasi et al., 2015; Wu et al., 2017),
ostensibly modeling sequence-level item transitions. Hierarchical RNNs
(Quadrana et al., 2017) incorporate personalization information. Moreover,
both long-term and short-term item transition correlations are modelled in
LSTM (Wu et al., 2017) . Recently, the success of self-attention models
(Vaswani et al., 2017; Devlin et al., 2018) promotes the prosperity of
Transformer-based sequential recommendation models. SASRec (Kang & McAuley,
2018) is a pioneering work adapting Transformer to characterize complex item
transition correlations. BERT4Rec (Sun et al., 2019) adopts the bidirectional
Transformer layer to encode sequence. ASReP (Liu et al., 2021c) reversely pre-
training a Transformer to augment short sequences and fine-tune it to predict
the next-item in sequences. TGSRec (Fan et al., 2021) models temporal
collaborative signals in sequences to recognize item relationships.

###  2.2 Self-supervised Learning

Self-supervised learning (SSL) is proposed recently to describe “the machine
predicts any parts of its input for any observed part”(Bengio et al., 2021),
which stays within the narrow scope of unsupervised learning. To achieve the
self-prediction, endeavors from various domains have developed different SSL
schemes from either generative or contrastive perspectives (Liu et al.,
2021a). For generative SSL, the masked language model is adopted in BERT
(Devlin et al., 2018) to generate masked words in sentences. GPT-GNN (Hu et
al., 2020) also generates masked edges to realize SSL. Other generative SSL
paradigms in computer vision (Oord et al., 2016) are proposed. Compared with
generative SSL, contrastive SSL schemes have demonstrated more promising
performance. SimCLR (Chen et al., 2020) proposes simple contrastive learning
between augmented views for images, which is rather effective in achieving
SSL. GCC (Qiu et al., 2020) and GraphCL (You et al., 2020) adopts contrastive
learning between views from corrupted graph structures. CL4SRec (Xie et al.,
2020) and CoSeRec (Liu et al., 2021b) devise the sequence augmentation methods
for SSL on sequential recommendation. This paper also investigates the
contrastive SSL for a sequential recommendation. Instead of adopting the data
augmentation for constructing views to contrast, we propose the model
augmentation to generate contrastive views.

##  3 Preliminary

###  3.1 Problem Formulation

We denote user and item sets as $\mathcal{U}$ and $\mathcal{V}$ respectively.
Each user $u\in\mathcal{U}$ is associated with a sequence of items in
chronological order $s_{u}=[v_{1},\dots,v_{t},\dots,v_{|s_{u}|}]$, where
$v_{t}\in\mathcal{V}$ denotes the item that $u$ has interacted with at time
$t$ and $|s_{u}|$ is the total number of items. Sequential recommendation is
formulated as follows:

$\underset{v_{i}\in\mathcal{V}}{\mathrm{arg\,max}}~{}P(v_{|s_{u}|+1}=v_{i}\left|s_{u}\right.),$
(1)

where $v_{|s_{u}|+1}$ denotes the next item in sequence. Intuitively, we
calculate the probability of all candidate items and recommend items with high
probability scores.

###  3.2 Sequential Recommendation Framework

The core of a generic sequential recommendation framework is a sequence
encoder $\mathsf{SeqEnc}(\cdot)$, which transforms item sequences to
embeddings for scoring. We formulate the encoding step as:

$\mathbf{h}_{u}=\mathsf{SeqEnc}(s_{u}),$ (2)

where $\mathbf{h}_{u}$ denotes the sequence embedding of $s_{u}$. To be
specific, if we adopt a Transformer (Kang & McAuley, 2018; Vaswani et al.,
2017) as the encoder, $\mathbf{h}_{u}$ is a bag of embeddings, where at each
position $t$, $\mathbf{h}_{u}^{t}$, represents a predicted next-item. We adopt
the log-likelihood loss function to optimize the encoder for next-item
prediction as follows:

$\mathcal{L}_{\text{rec}}(u,t)=-\log(\sigma(\mathbf{h}_{u}^{t}\cdot\mathbf{e}_{v_{t+1}}))-\sum_{v_{j}\not\in
s_{u}}\log(1-\sigma(\mathbf{h}_{u}^{t}\cdot\mathbf{e}_{v_{j}})),$ (3)

where $\mathcal{L}_{\text{rec}}(u,t)$ denotes the loss score for the
prediction at position $t$ in sequence $s_{u}$, $\sigma$ is the non-linear
activation function, $\mathbf{e}_{v_{t+1}}$ denotes the embedding for item
$v_{t+1}$, and $v_{j}$ is the sampled negative item for $s_{u}$. The
embeddings of items are retrieved from the embedding layer in
$\mathsf{SeqEnc}(\cdot)$, which is jointly optimized with other layers.

###  3.3 Contrastive Self-supervised Learning Paradigm

Other than the next-item prediction, we can leverage other pretext tasks (Sun
et al., 2019; Liu et al., 2021a; b) over the sequence to optimize the encoder,
which harnesses the self-supervised signals within the sequence. This paper
investigate the widely adopted contrastive SSL scheme (Liu et al., 2021b; Xie
et al., 2020). This scheme constructs positive and negative view pairs from
sequences, and employs the contrastive loss (Oord et al., 2018) to optimize
the encoder. We formulate the SSL step as follows:

$\mathcal{L}_{\mathrm{ssl}}(\mathbf{\tilde{h}}_{2u-1},\mathbf{\tilde{h}}_{2u})=-\log\frac{\exp(\text{sim}(\mathbf{\tilde{h}}_{2u-1},\mathbf{\tilde{h}}_{2u}))}{\sum_{m=1}^{2N}\mathbbm{1}_{m\neq
2u-1}\exp(\text{sim}(\mathbf{\tilde{h}}_{2u-1},\mathbf{\tilde{h}}_{m}))},$ (4)

where $\mathbf{\tilde{h}}_{2u}$ and $\mathbf{\tilde{h}}_{2u-1}$ denotes two
views constructed for sequence $s_{u}$. $\mathbbm{1}$ is an indication
function. $\text{sim}(\cdot,\cdot)$ is the similarity function, e.g. dot-
product. Since each sequence has two view, we have $2N$ samples in a batch
with $N$ sequences for training. The nominator indicates the agreement
maximization between a positive pair, while the denominator is interpreted as
push away those negative pairs. Existing works apply data augmentation for
sequences to construct views, e.g. Xie et al. (2020) propose crop, mask, and
reorder a sequence and (Liu et al., 2021b) devises insert and substitute
sequence augmentations. For sequential recommendation, since both SSL and the
next-item prediction charaterize the item relationships in sequences, we add
them together to optimize the encoder. Therefore, the final loss
$\mathcal{L}=\mathcal{L}_{\mathrm{rec}}+\lambda\mathcal{L}_{\mathrm{ssl}}.$
Compared with them, we adopt both the data augmentation and model augmentation
to generate views for contrast. We demonstrate the contrastive SSL step in
Figure 1(a).

Figure 1: (a) The contrastive SSL framework with model augmentation. We apply
the model augmentation to the encoder, which constructs two views for
contrastive learning. (b) the neuron masking augmentation. We demonstrate the
neuron masking for the Feed-Forward network. (c) the layer dropping
augmentation. We add $K$ FFN layers after the encoder and randomly drop $M$
layers (dash blocks) during each batch of training. And (d) the encoder
complementing augmentation. We pre-train another encoder for generating the
embedding of sequences. The embedding from the pre-trained encoder is combined
with the model encoder for contrastive learning.

##  4 Model Augmentation

In this section, we introduce the model augmentation to construct views for
sequences. We discuss three type of model augmentation methods, which are
neuron mask, layer drop and encoder complement. We illustrate these
augmentation methods in Figure 1(b), 1(c) and 1(d), respectively.

###  4.1 Neuron Masking

This work adopts the Transformer as the sequence encoder, which passes the
hidden embeddings to the next layer through a feed-forward network (FFN).
During training, we randomly mask partial neurons in each FFN layer, which
involves a masking probability $p$. The large value of $p$ leads to intensive
embedding perturbations. As such, we generate a pair of views from one
sequence from model perspectives. Besides, during each batch of training, the
masked neurons are randomly selected, which results in comprehensive
contrastive learning on model augmentation. Note that, though we can utilize
different probability values for distinct FFN layers, we enforce their neuron
masking probability to be the same for simplicity. The neuron masking
augmentation on FFN is shown in Figure 1(b). Additionally, we remark that the
neuron mask can be applied to any neural layers in a model to inject more
perturbations.

###  4.2 Layer Dropping

Dropping partial layers of a model decreases the depth and reduces complexity.
Previous research argues that most recommender systems require only shallow
embeddings for users and items (Dacrema et al., 2019). Therefore, it is
reasonable to randomly drop a fraction of layers during training, which
functions as a way of regularization. Additionally, existing works (Liu et
al., 2020; He et al., 2016) claim that embeddings at shallow layers and deep
layers are both important to reflect the comprehensive information of the
data. Dropping layers enable contrastive learning between shallow embeddings
and deep embeddings, thus being an enhancement of existing works that only
contrasting between deep features.

On the other hand, dropping layers, especially those necessary layers in a
model, may destroy original sequential correlations. Thus, views generating by
dropping layers may not be a positive pair. To this end, instead of
manipulating the original encoder, we stack $K$ FFN layers after the encoder
and randomly drop $M$ of them during each batch of training, where $M<K$. We
illustrate the layer dropping as in Figure 1(c), where we append $K$
additional FFN layers after the encoder and use dash blocks to denote the
dropped layers.

###  4.3 Encoder Complementing

During self-supervised learning, we employ one encoder to generate embeddings
of two views of one sequence. Though this encoder can be effective in
revealing complex sequential correlations, contrasting on one single encoder
may result in embedding collapse problems for self-supervised learning (Hua et
al., 2021). Moreover, one single encoder is only able to reflect the item
relationships from a unitary perspective. For example, the Transformer encoder
adopts the attentive aggregation of item embeddings to infer sequence
embedding, while an RNN structure (Hidasi et al., 2015) is more suitable in
encoding direct item transitions. Therefore, contrasting between views from
distinct encoders enables the model to learn comprehensive sequential
relationships of items.

However, embeddings from two views of a sequence with distinct encoders lead
to a non-Siamese paradigm for self-supervised learning, which is hard to train
and suffers the embedding collapse problem (Koch et al., 2015; Chen & He,
2021). Additionally, if two distinct encoders reveal significantly diverse
sequential correlations, the embeddings are far away from each other, thus
being bad views for contrastive learning (Tian et al., 2020). Moreover, though
we can optimize two encoders during a training phase, it is still problematic
to combine them for the inference of sequence embeddings to conduct
recommendations.

As a result, instead of contrastive learning with distinct encoders, we
harness another pre-trained encoder as an encoder complementing model
augmentation for the original encoder. To be more specific, we first pre-train
another encoder with the next-item prediction target. Then, in the self-
supervised training stage, we utilize this pre-trained encoder to generate
another embedding for a view. After that, we add the view embeddings from a
model encoder and the pre-trained encoder. We illustrate the encoder
complementing augmentation in Figure 1(d). Note that we only apply this model
augmentation in one branch of the SSL paradigm. And the embedding from the
pre-trained encoder is re-scaled by a hyper-parameter $\gamma$ before adding
to the embedding from the framework’s encoder. The smaller value of $\gamma$
implies injecting fewer perturbations from a distinct encoder. The pre-trained
encoder are not trainable during training. Hence, there is no optimization for
this pre-trained encoder and it is no longer required to take account of both
encoders during the inference stage.

##  5 Experiments

###  5.1 Experimental Settings

Dataset We conduct experiments on three public datasets. Amazon Sports, Amazon
Toys and Games (McAuley et al., 2015) and Yelp222https://www.yelp.com/dataset,
which are Amazon review data in Sport and Toys categories, and a dataset for
the business recommendation, respectively. We follow common practice in (Liu
et al., 2021c; Xie et al., 2020) to only keep the ‘5-core’ sequences. In
total, Sports dataset has 35,598 users, 18,357 items and 296,337 interactions.
Toys dataset contains 19,412 users, 11,924 items, and 167,597interactions.
Yelp dataset consists 30,431 users, 20,033 items and 316,354 interactions.

Evaluation Metrics We follow existing works (Wang et al., 2019; Krichene &
Rendle, 2020; Liu et al., 2021c) to evaluate models’ performances based on the
whole item set without negative sampling and report standard Hit Ratio$@k$
($\mathrm{HR}@k$) and Normalized Discounted Cumulative Gain$@k$
($\mathrm{NDCG}@k$) on all datasets, where $k\in\\{5,10,20\\}$.

Baselines We include two groups of sequential models as baselines for
comprehensive comparisons. The first group baselines are sequential models
that use different deep neural architectures to encode sequences with a
supervised objective. These include GRU4Rec (Hidasi et al., 2015) as an RNN-
based method, Caser (Tang & Wang, 2018) as a CNN-based approach, and SASRec
(Kang & McAuley, 2018) as one of the state-of-the-art Transformer based
solution. The second group baselines additionally leverage SSL objective.
BERT4Rec (Sun et al., 2019) employs a _Cloze_ task (Taylor, 1953) as a
generative self-supervised learning sigal. S${}^{3}\text{Rec}$ (Zhou et al.,
2020) uses contrastive SSL with ‘mask’ data augmentation to fuse correlation-
ships among item, sub-sequence, and correspondinng attributes into the
networks. We remove the components for fusing attributes for fair comparison.
CL4SRec (Xie et al., 2020) maximize the agreements between two sequences
augmentation, where the data augmentation are randomly selected from ’crop’,
‘reorder’, and ‘mask’ data augmentations. CoSeRec (Liu et al., 2021b) improves
the robustness of data augmentation under contrastive SSL framework by
leveraging item-correlations.

Implementation Details The model encoder in SRMA is the basic Transformer-
based encoder. We adopt the widely used SASRec encoder. The neuron masking
probability is searched from $\\{0.0,0.1,0.2,\dots,0.9\\}$. For layer
dropping, the $K$ is searched from $\\{1,2,3,4\\}$, and $M$ is searched
accordingly. As for encoder complementing, we search the re-scale hyper-
parameter $\gamma$ from $\\{0.005,0.01,0.05,0.1,0.5,1.0\\}$ and the pre-
trained encoder is selected from a $1$-layer Transformer and a GRU encoder.

###  5.2 Overall Performance

Table 1: Performance comparisons of different methods. The best score is in bold in each row, and the second best is underlined. Dataset | Metric | GRU4Rec | Caser | SASRec | BERT4Rec | S3-Rec | CL4SRec | CoSeRec | SRMA
---|---|---|---|---|---|---|---|---|---
Sports | HR@5 | 0.0162 | 0.0154 | 0.0206 | 0.0217 | 0.0121 | 0.0231 |  0.0287 | 0.0299
HR@10 | 0.0258 | 0.0261 | 0.0320 | 0.0359 | 0.0205 | 0.0369 |  0.0437 | 0.0447
HR@20 | 0.0421 | 0.0399 | 0.0497 | 0.0604 | 0.0344 | 0.0557 |  0.0635 | 0.0649
NDCG@5 | 0.0103 | 0.0114 | 0.0135 | 0.0143 | 0.0084 | 0.0146 |  0.0196 | 0.0199
NDCG@10 | 0.0142 | 0.0135 | 0.0172 | 0.019 | 0.0111 | 0.0191 |  0.0242 | 0.0246
NDCG@20 | 0.0186 | 0.0178 | 0.0216 | 0.0251 | 0.0146 | 0.0238 |  0.0292 | 0.0297
Yelp | HR@5 | 0.0152 | 0.0142 | 0.0160 | 0.0196 | 0.0101 | 0.0227 |  0.0241 | 0.0243
HR@10 | 0.0248 | 0.0254 | 0.0260 | 0.0339 | 0.0176 | 0.0384 |  0.0395 | 0.0395
HR@20 | 0.0371 | 0.0406 | 0.0443 | 0.0564 | 0.0314 | 0.0623 | 0.0649 |  0.0646
NDCG@5 | 0.0091 | 0.008 | 0.0101 | 0.0121 | 0.0068 | 0.0143 |  0.0151 | 0.0154
NDCG@10 | 0.0124 | 0.0113 | 0.0133 | 0.0167 | 0.0092 | 0.0194 |  0.0205 | 0.0207
NDCG@20 | 0.0145 | 0.0156 | 0.0179 | 0.0223 | 0.0127 | 0.0254 |  0.0263 | 0.0266
Toys | HR@5 | 0.0097 | 0.0166 | 0.0463 | 0.0274 | 0.0143 | 0.0525 |  0.0583 | 0.0598
HR@10 | 0.0176 | 0.0270 | 0.0675 | 0.0450 | 0.0094 | 0.0776 |  0.0812 | 0.0834
HR@20 | 0.0301 | 0.0420 | 0.0941 | 0.0688 | 0.0235 | 0.1084 |  0.1103 | 0.1132
NDCG@5 | 0.0059 | 0.0107 | 0.0306 | 0.0174 | 0.0123 | 0.0346 |  0.0399 | 0.0407
NDCG@10 | 0.0084 | 0.0141 | 0.0374 | 0.0231 | 0.0391 | 0.0428 |  0.0473 | 0.0484
NDCG@20 | 0.0116 | 0.0179 | 0.0441 | 0.0291 | 0.0162 | 0.0505 |  0.0547 | 0.0559

We compare the proposed paradigm SRMA to existing methods with respect to the
performance on the sequential recommendation. Results are demonstrated in
Table 1. We can observe that Transformer-based sequence encoders, such SASRec
and BERT4Rec are better than GRU4Rec or Caser sequence encoders. Because of
this, our proposed model SRMA also adopts the Transformer as sequence encoder.
Moreover, the SSL paradigm can significantly improve performance. For example,
the CL4SRec model, which adopts the random data augmentation, improves the
performance of SASRec on HR and NDCG by $13.2\%$ and $9.8\%$ on average
regarding the Sports dataset, respectively. Also, since SRMA enhances the SSL
with both data augmentation and model augmentation, SRMA thus outperforms all
other SSL sequential recommendation models. SRMA adopts the same data
augmentation methods as CL4SRec. Nevertheless, SRMA significantly outperforms
CL4SRec. On the sports dataset, we achieve $18.9\%$ and $27.9\%$ relative
improvements on HR and NDCG, respectively. On the Yelp dataset, we achieve
$4.5\%$ and $6.4\%$ relative improvements on HR and NDCG, respectively. And on
Toys data, we achieve $8.6\%$ and $13.8\%$ relative improvements on HR and
NDCG, respectively. In addition, SRMA also performs better than CoSeRec which
leverages item correlations for data augmentation. Those results all verify
the effectiveness of model augmentation in improving the SSL paradigm.

###  5.3 Comparison between Model and Data Augmentation

Because SRMA adopts the random sequence augmentation, we mainly focus on
comparing with CL4SRec to justify the impacts of model augmentation and data
augmentation. In fact, CL4SRec also implicitly uses the neuron masking model
augmentation, where dropout layers are stacked within its original sequence
encoder. To separate the joint effects of model and data augmentation, we
create its variants ‘CL4S. $p=0$’, which sets all the dropout ratios to be
$0$, thus disables the neuron masking augmentation. Also, another variant
‘CL4S. w/o D’, which has no data augmentation are also compared. Additionally,
we create two other variants of SRMA as ‘SRMA w/o M’ and ‘SRMA w/o D’ by
disabling the model augmentation and data augmentation respectively. ‘SRMA w/o
M’ has additional FFN layers compared with ‘CL4S. $p=0$’. The recommendation
performance on the Sports and Toys dataset is presented in Table 2. We have
the following observations. Firstly, we notice a significant performance drop
of the variant ‘CL4S. $p=0$’, which suggests that the neuron masking
augmentation is rather crucial. It benefits both the regularization of the
training encoder and model augmentation of SSL. Secondly, ‘SRMA w/o D’
outperforms other baselines on the Sports dataset and has comparable
performance to ‘CL4S.’, which indicates the model augmentation is of more
impact in the SSL paradigm compared with data augmentation. Thirdly, SRMA
performs the best against all the variants. This result suggests that we
should jointly employ the data augmentation and model augmentation in an SSL
paradigm, which contributes to comprehensive contrastive self-supervised
signals.

Table 2: Performance comparison w.r.t. the variants of CL4SRec (CL4S.) and
SRMA. M and D denote the model augmentation and data augmentation,
respectively. $p=0$ indicates no neuron masking. The best score in each column
are in bold, where the second-best are underlined.

Model | Sports | Toys
---|---|---
HR | NDCG | HR | NDCG
@5 | @10 | @5 | @10 | @5 | @10 | @5 | @10
CL4S. w/o D | 0.0162 | 0.0268 | 0.0108 | 0.0142 | 0.0444 | 0.0619 | 0.0306 | 0.0363
CL4S. $p=0$ | 0.0177 | 0.0292 | 0.0119 | 0.0156 | 0.0451 | 0.0654 | 0.0305 | 0.037
CL4S. | 0.0231 | 0.0369 | 0.0146 | 0.0191 |  0.0525 |  0.0776 |  0.0346 |  0.0428
SRMA w/o D |  0.0285 |  0.0432 |  0.0187 |  0.0234 | 0.0504 | 0.0724 | 0.0331 | 0.0402
SRMA w/o M | 0.0165 | 0.0272 | 0.0104 | 0.0138 | 0.0412 | 0.0590 | 0.0279 | 0.0336
SRMA | 0.0299 | 0.0447 | 0.0199 | 0.0246 | 0.0598 | 0.0834 | 0.0407 | 0.0484

###  5.4 Hyper-parameter Sensitivity

In this section, we vary the hyper-parameters in neuron masking and layer
dropping to draw a detailed investigation of model augmentation.

Effect of Neuron Masking. Though all neural layers can apply the neuron
masking augmentation, for simplicity, we only endow the FFN layer with the
neuron masking augmentation and set the masking probability as $p$ for all FFN
layers in the framework. We fix the settings of layer dropping and the encoder
complementing model augmentation and select $p$ from
$\\{0.0,0.1,0.2,\dots,0.9\\}$, where $0.0$ is equivalent to no neuron masking.
Also, we compare SRMA with SASRec to justify the effectiveness of the SSL
paradigm. The performance curves of HR$@5$ and NDCG$@5$ on the Sports and Toys
dataset are demonstrated in Figure 2. We can observe that the performance
improves first and then drops when increasing $p$ from $0$ to $0.9$. The
rising of the performance score implies that the neuron masking augmentation
is effective in improving the ability of the sequence encoder for a
recommendation. And the dropping of the performance indicates the intensity of
model augmentation should not be overly aggressive, which may lead to less
informative contrastive learning. As to SASRec, we recognize a higher score of
SASRec when $p$ is large, which indicates the optimal model augmentation
should be a slightly perturbation rather than a intensive distortion.
Moreover, SRMA consistently outperforms SASRec when $0.1<p<0.6$. Since the
only difference is that SASRec has no SSL module, we can thus concludes that
the performance gains result from the contrastive SSL step by using the neuron
masking.

Figure 2: Performance comparison betweeen SASRec and SRMA in HR@5 and NDCG@5
w.r.t different values of neuron masking probability $p$ on Sports and Toys
dataset.

Effect of Layer Dropping. The layer dropping model augmentation is controlled
by two hyper-parameters, the number of additional FFN layers and the number of
layers to drop during training, which is denoted as $K$ and $M$, respectively.
Since we can only drop those additional layers, we have $M<K$. We select $K$
from $\\{1,2,3,4\\}$ while $M$ are searched accordingly. Due to space
limitation, we only report the NDCG$@5$ on the Sports and Toys dataset in
Figure 3. We observe that $K=2,M=1$ achieves the best performance on both
datasets, which implies the efficacy of layer dropping. Additionally, we also
find that the performance on $K=4$ is consistently worse than $K=2$ on both
datasets, which suggests that adding too many layers increases the complexity
of the model, which is thus unable to enhance the SSL paradigm.

Figure 3: The NDCG@5 performance w.r.t. different $K$ and $M$ for layer
dropping augmentation on Sports and Toys dataset.

###  5.5 Analyses on Encoder Complementing

Table 3: Performance comparison among SRMA without encoder complementing (w/o
Enc.), with Transformer-based (-Trans) and with GRU-based (-GRU) complementary
pre-trained encoder. The best score in each column is in bold.

Encoders | Sports | Toys
---|---|---
HR | NDCG | HR | NDCG
@5 | @10 | @5 | @10 | @5 | @10 | @5 | @10
w/o Enc. | 0.0269 | 0.0401 | 0.0181 | 0.0224 | 0.0567 | 0.0806 | 0.0389 | 0.0466
-Trans | 0.0268 | 0.0408 | 0.0181 | 0.0226 | 0.0588 | 0.0811 | 0.0402 | 0.0474
-GRU | 0.0281 | 0.0411 | 0.0186 | 0.0228 | 0.0577 | 0.0811 | 0.0395 | 0.047

In this section, we investigate the effects of encoder complementing
augmentation for constructing views. Recall that we combine the embedding from
the model’s encoder and a distinct pre-trained encoder. For this complementary
encoder, we select from a Transformer-based and a GRU-based encoder. Since the
model’s encoder is a 2-layer Transformer, this pre-trained encoder is a
1-layer Transformer to maintain diversity. We first pre-train the
complementary encoder based on the next-item prediction task. As such, we
empower the pre-trained encoder to characterize the sequential correlations of
items. The comparison is conducted on both Sports and Toys datasets, which are
shown in Table 3. The observations are as follows: Firstly, on the Sports
dataset, pre-training a GRU encoder as a complement performs the best against
the other two, which indicates that injecting distinct encoders for
contrastive learning can enhance the SSL signals. Secondly, on the Toys
dataset, adopting a 1-layer pre-trained Transformer as the complementary
encoder yields the best scores on all metrics. Besides the effectiveness of
encoder complementing, this result also suggests that the complementary
encoder may not be overly different from the model’s encoder on some datasets,
which otherwise cannot enhance the comprehensive contrastive learning between
views. Lastly, both Transformer-based and GRU-based pre-trained complementary
encoders consistently outperform SRMA without encoder complementing, which
directly indicates the necessity of encoder complementing as a way of model
augmentation.

##  6 Conclusion

This work proposes a novel contrastive self-supervised learning paradigm,
which simultaneously employs model augmentation and data augmentation to
construct views for contrasting. We propose three-level model augmentation
methods for this paradigm, which are neuron masking, layer dropping, and
encoder complementing. We adopt this paradigm to the sequential recommendation
problem and propose a new model SRMA. This model adopts both the random data
augmentation of sequences and the corresponding three-level model augmentation
for the sequence encoder. We conduct comprehensive experiments to verify the
effectiveness of SRMA. The overall performance comparison justifies the
advantage of contrastive SSL with model augmentation. Additionally, detailed
investigation regarding the impacts of model augmentation and data
augmentation in improving the performance are discussed. Moreover, ablation
studies with respect to three-level model augmentation methods are
implemented, which also demonstrate the superiority of the proposed model.
Overall, this work opens up a new direction in constructing views from model
augmentation. We believe our proposed model augmentation can enhance existing
contrastive SSL paradigms which only have data augmentation.

## References

  * Bengio et al. (2021) Yoshua Bengio, Yann Lecun, and Geoffrey Hinton.  Deep learning for ai.  64(7), 2021.  ISSN 0001-0782.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In _International conference on machine learning_ , pp. 1597–1607. PMLR, 2020.
  * Chen & He (2021) Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 15750–15758, 2021.
  * Chen et al. (2018) Xu Chen, Hongteng Xu, Yongfeng Zhang, Jiaxi Tang, Yixin Cao, Zheng Qin, and Hongyuan Zha.  Sequential recommendation with user memory networks.  In _WSDM_ , pp. 108–116, 2018.
  * Dacrema et al. (2019) Maurizio Ferrari Dacrema, Paolo Cremonesi, and Dietmar Jannach.  Are we really making much progress? a worrying analysis of recent neural recommendation approaches.  In _Proceedings of the 13th ACM Conference on Recommender Systems_ , pp. 101–109, 2019.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ , 2018.
  * Fan et al. (2021) Ziwei Fan, Zhiwei Liu, Jiawei Zhang, Yun Xiong, Lei Zheng, and Philip S. Yu.  Continuous-time sequential recommendation with temporal graph collaborative transformer.  In _Proceedings of the 30th ACM International Conference on Information and Knowledge Management_. ACM, 2021.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen.  Simcse: Simple contrastive learning of sentence embeddings.  _arXiv preprint arXiv:2104.08821_ , 2021.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 770–778, 2016.
  * Hidasi et al. (2015) Balázs Hidasi, Alexandros Karatzoglou, Linas Baltrunas, and Domonkos Tikk.  Session-based recommendations with recurrent neural networks.  _arXiv preprint arXiv:1511.06939_ , 2015.
  * Hu et al. (2020) Ziniu Hu, Yuxiao Dong, Kuansan Wang, Kai-Wei Chang, and Yizhou Sun.  Gpt-gnn: Generative pre-training of graph neural networks.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pp. 1857–1867, 2020.
  * Hua et al. (2021) Tianyu Hua, Wenxiao Wang, Zihui Xue, Yue Wang, Sucheng Ren, and Hang Zhao.  On feature decorrelation in self-supervised learning.  _arXiv preprint arXiv:2105.00470_ , 2021.
  * Kang & McAuley (2018) Wang-Cheng Kang and Julian McAuley.  Self-attentive sequential recommendation.  In _ICDM_ , pp. 197–206. IEEE, 2018.
  * Koch et al. (2015) Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al.  Siamese neural networks for one-shot image recognition.  In _ICML deep learning workshop_ , volume 2. Lille, 2015.
  * Krichene & Rendle (2020) Walid Krichene and Steffen Rendle.  On sampled metrics for item recommendation.  In _SIGKDD_ , pp. 1748–1757, 2020.
  * Liu et al. (2020) Meng Liu, Hongyang Gao, and Shuiwang Ji.  Towards deeper graph neural networks.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pp. 338–348, 2020.
  * Liu et al. (2021a) Xiao Liu, Fanjin Zhang, Zhenyu Hou, Li Mian, Zhaoyu Wang, Jing Zhang, and Jie Tang.  Self-supervised learning: Generative or contrastive.  _IEEE Transactions on Knowledge and Data Engineering_ , 2021a.
  * Liu et al. (2021b) Zhiwei Liu, Yongjun Chen, Jia Li, Philip S Yu, Julian McAuley, and Caiming Xiong.  Contrastive self-supervised sequential recommendation with robust augmentation.  _arXiv preprint arXiv:2108.06479_ , 2021b.
  * Liu et al. (2021c) Zhiwei Liu, Ziwei Fan, Yu Wang, and Philip S. Yu.  Augmenting sequential recommendation with pseudo-prior items via reversely pre-training transformer.  ACM, 2021c.
  * McAuley et al. (2015) Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel.  Image-based recommendations on styles and substitutes.  In _SIGIR_ , pp. 43–52, 2015.
  * Oord et al. (2016) Aaron van den Oord, Nal Kalchbrenner, Oriol Vinyals, Lasse Espeholt, Alex Graves, and Koray Kavukcuoglu.  Conditional image generation with pixelcnn decoders.  _arXiv preprint arXiv:1606.05328_ , 2016.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ , 2018.
  * Qiu et al. (2020) Jiezhong Qiu, Qibin Chen, Yuxiao Dong, Jing Zhang, Hongxia Yang, Ming Ding, Kuansan Wang, and Jie Tang.  Gcc: Graph contrastive coding for graph neural network pre-training.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pp. 1150–1160, 2020.
  * Quadrana et al. (2017) Massimo Quadrana, Alexandros Karatzoglou, Balázs Hidasi, and Paolo Cremonesi.  Personalizing session-based recommendations with hierarchical recurrent neural networks.  In _RecSys_ , pp. 130–137, 2017.
  * Rendle et al. (2010) Steffen Rendle, Christoph Freudenthaler, and Lars Schmidt-Thieme.  Factorizing personalized markov chains for next-basket recommendation.  In _WWW_ , pp. 811–820, 2010.
  * Sun et al. (2019) Fei Sun, Jun Liu, Jian Wu, Changhua Pei, Xiao Lin, Wenwu Ou, and Peng Jiang.  Bert4rec: Sequential recommendation with bidirectional encoder representations from transformer.  In _CIKM_ , pp. 1441–1450, 2019.
  * Tang & Wang (2018) Jiaxi Tang and Ke Wang.  Personalized top-n sequential recommendation via convolutional sequence embedding.  In _WSDM_ , pp. 565–573, 2018.
  * Taylor (1953) Wilson L Taylor.  “cloze procedure”: A new tool for measuring readability.  _Journalism quarterly_ , 30(4):415–433, 1953\.
  * Tian et al. (2020) Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola.  What makes for good views for contrastive learning?  _arXiv preprint arXiv:2005.10243_ , 2020.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In _NIPS_ , pp. 5998–6008, 2017.
  * Wang et al. (2019) Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua.  Neural graph collaborative filtering.  In _Proceedings of the 42nd international ACM SIGIR conference on Research and development in Information Retrieval_ , pp. 165–174, 2019.
  * Wu et al. (2017) Chao-Yuan Wu, Amr Ahmed, Alex Beutel, Alexander J Smola, and How Jing.  Recurrent recommender networks.  In _WSDM_ , pp. 495–503, 2017.
  * Wu et al. (2020) Liwei Wu, Shuqing Li, Cho-Jui Hsieh, and James Sharpnack.  Sse-pt: Sequential recommendation via personalized transformer.  In _RecSys_ , pp. 328–337. ACM, 2020.
  * Xie et al. (2020) Xu Xie, Fei Sun, Zhaoyang Liu, Jinyang Gao, Bolin Ding, and Bin Cui.  Contrastive pre-training for sequential recommendation.  _arXiv preprint arXiv:2010.14395_ , 2020.
  * You et al. (2020) Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen.  Graph contrastive learning with augmentations.  _Advances in Neural Information Processing Systems_ , 33:5812–5823, 2020.
  * Zheng et al. (2019) Lei Zheng, Ziwei Fan, Chun-Ta Lu, Jiawei Zhang, and Philip S Yu.  Gated spectral units: Modeling co-evolving patterns for sequential recommendation.  In _SIGIR_ , pp. 1077–1080, 2019.
  * Zhou et al. (2020) Kun Zhou, Hui Wang, Wayne Xin Zhao, Yutao Zhu, Sirui Wang, Fuzheng Zhang, Zhongyuan Wang, and Ji-Rong Wen.  S3-rec: Self-supervised learning for sequential recommendation with mutual information maximization.  In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_, pp. 1893–1902, 2020.


# Automatic Identification and Extraction of Assumptions on GitHub

Chen Yang23, Zinan Ma2, Peng Liang41, Xiaohua Liu2 DOI reference number:
10.18293/DMSVIVA2023-071  2School of Artificial Intelligence, Shenzhen
Polytechnic, Shenzhen, China
3State Key Laboratory for Novel Software Technology, Nanjing University,
Nanjing, China
4School of Computer Science, Wuhan University, Wuhan, China
{yangchen, 21680260, lxh}szpt.edu.cn<EMAIL_ADDRESS>
###### Abstract

In software development, due to the lack of knowledge or information, time
pressure, complex context, and many other factors, various uncertainties
emerge during the development process, leading to assumptions scattered in
projects. Being unaware of certain assumptions can result in critical problems
(e.g., system vulnerability and failures). The prerequisite of analyzing and
understanding assumptions in software development is to identify and extract
those assumptions with acceptable effort. In this paper, we proposed a tool
(i.e., Assumption Miner) to automatically identify and extract assumptions on
GitHub projects. To evaluate the applicability of Assumption Miner, we first
presented an example of using the tool to mine assumptions from one large and
popular deep learning framework project: the TensorFlow project on GitHub. We
then conducted an evaluation of the tool. The results show that Assumption
Miner can effectively identify and extract assumptions from the repositories
on GitHub.

###### Index Terms:

Assumption, GitHub, Mining Software Repositories

##  I Introduction

Assumptions in the field of software development is a broad topic: different
types of assumptions (e.g., requirement assumptions [1], design assumptions
[2], and construction assumptions [3]) have been extensively discussed. For
instance, in the early phases of software development, there could be many
uncertain things. However, in order to meet the project business goals (e.g.,
schedule and deadlines), stakeholders have to work in the presence of such
uncertainties; these uncertainties can lead to assumptions. In this paper, we
advocate treating uncertainty and assumption as two different but related
concepts: one way to deal with uncertainties is to make implicit or explicit
assumptions (i.e., a thing that is uncertain, but accepted as true) [4].

The importance of assumptions and their management in software development has
been highlighted in many studies and industrial cases. For example, Corbató
[5] mentioned in his ACM Turing Award lecture that “design bugs are often
subtle and occur by evolution with early assumptions being forgotten as new
features or uses are added to systems.” Garlan et al. pointed out that
incompatible assumptions in software architecture can cause architectural
mismatch [6]. Lewis et al. also mentioned similar results in machine learning
systems: since there are different types of stakeholders (e.g., data
scientist, software engineer, and system user) of a machine learning system,
they could make different but incompatible or invalid assumptions, leading to
system misunderstanding, mismatch, etc [7]. In October 2018, Lion Air Flight
610 crashed 13 minutes after takeoff and killed all 189 people on board; In
March 2019, Ethiopian Airlines Flight 302 crashed and ended another 157 lives.
According to the reports from the government, one critical reason of the 737
MAX crashes is regarding not-well managed assumptions [8][9]. In the report,
they mentioned that the aircraft company made invalid assumptions about the
critical system components. Specifically, the invalid assumptions regarding
MCAS (Maneuvering Characteristics Augmentation System) are the root cause of
the crashes. The report also pointed out the need of identifying and re-
evaluating important assumptions in the system.

As evidenced from researchers and practitioners, stakeholders constantly make
assumptions in their work [4][10]. The assumptions can be further classified
as two types: self-claimed assumptions (SCAs) and potential assumptions (PAs).
Considering a sentence in a commit, pull request (PR), or issue of a GitHub
repository, an SCA is that the sentence includes an assumption and the
assumption is explicitly claimed using at least one of the assumption-related
terms (i.e., “assumption”, “assumptions”, “assume”, “assumes”, “assumed”,
“assuming”, “assumable”, and “assumably”). For example, in the sentence:
“[tf/xla] fixup numbering of xla parameters used for aliasing previously, the
xla argument parameter was incorrectly assumed to be corresponding to the
index in the vector of ‘xlacompiler::argument’”, it includes an SCA of
assuming the xla argument parameter is corresponding to the index in the
vector of xlacompiler::argument and the developer claimed that it was an
invalid assumption. A PA is that the sentence may include an assumption, i.e.,
it is a potential assumption that needs further confirmation from human
experts. This definition covers various aspects, such as expectations, future
events, possibilities, guesses, opinions, feelings, and suspicions, which can
indicate PAs. We provide three examples to further explain the PA concept. For
example, in the sentence: “i think the right way to create demo tensorboard
instances is to simply run a tensorboard in the cloud, rather than keep
maintaining this mocked-out backend.”, it includes a PA regarding thoughts of
the right way to create demo tensorboard instances. In another example: “The
system will not crash under heavy load”, the sentence describes a future state
of the system, which is uncertain, and includes a PA. The third example is:
“both false and true outputs should be considered independently”. This
sentence does not include assumption-related terms, but the sentence describes
an expectation (i.e., “something should be”), which is a PA. After further
confirmation by human experts, this PA can be transformed to an SCA or other
types of software artifacts. Besides SCAs and PAs (which belong to explicit
assumptions), there are also many implicit assumptions in projects (e.g., in
stakeholders’ heads or requiring reasoning). Identifying implicit assumptions
at the sentence level is much more tricky than identifying SCAs and PAs, since
there are no explicit clues in the sentences and stakeholders need to infer
the sentences based on the context, which involves assumption reasoning.
Identifying implicit assumptions is out of the scope of this study, and we
treat it as future work.

Assumptions are related to many types of software artifacts, such as
decisions, technical debt, and source code [11]. For example, in TensorFlow,
there is an SCA: “TODO: Looks like there is an assumption that weight has only
one user. We should add a check here”, which induces a technical debt.
Existing research on assumptions and their management in software development
usually use experiments, surveys, and case studies to manually identify and
extract assumptions through observation, questionnaires, interviews, focus
groups, and documentation analysis [4]. Since such approaches have high costs
(e.g., time and resources), the number of identified and extracted assumptions
in those studies is often limited, leading to various problems of developing
new theories, approaches, and methods of assumptions and their management in
software development.

In this work, to overcome the issues of manually identifying and extracting
assumptions in software development, we proposed a tool: Assumption Miner,
which can be used to automatically identify and extract assumptions (i.e.,
SCAs and PAs) on GitHub repositories. With over 100 million developers, 4
million organizations, and 330 million
repositories111https://github.com/about/, accessed on 2023-04-21, GitHub is
one of the most important sources for open source software development.
Besides assumptions, the tool can also be easily extended to other research
fields (e.g., identifying and extracting technical debt [12]).

How to access Assumption Miner. Assumption Miner is available at
222http://39.108.224.140. Users can register or use a guest account to login
the tool. We also provided a deployment package for users who want to try the
tool on their local environment [13]. Users can read and follow the
instructions in the description for the deployment of the tool.

The remainder of the paper is organized as follows. Section II provides
related work, Section III describes the details of Assumption Miner, Section
IV presents an example of using the tool, Section V describes an evaluation of
Assumption Miner, and Section VI concludes the paper with future directions.

##  II Related Work

In the field of assumptions and their management in software development, most
assumptions are manually identified and extracted by researchers and
practitioners. Landuyt and Joosen focused on assumptions made during the
application of a threat modeling framework (i.e., LINDDUN), which allows the
identification of privacy-related design flaws in the architecting phase [14].
They conducted a descriptive study with 122 master students, and the students
identified and extracted 845 assumptions from the models created by the
students. Yang et al. conducted an exploratory study of assumptions made in
the development of nine popular deep learning frameworks (e.g., TensorFlow,
Keras, and PyTorch) on GitHub [11]. They identified and extracted 3,084
assumptions from the code comments in over 50,000 files of the deep learning
frameworks. Xiong et al. studied assumptions in the Hibernate developer
mailing list, including their expression, classification, trend over time, and
related software artifacts [15]. In their study, they identified and extracted
832 assumptions. Li et al. developed a machine learning approach [16] to
identify and classify assumptions based on the dataset constructed by Xiong et
al. [15], which can read the data (i.e., sentences) from the dataset (i.e., a
.csv file), preprocess the data (e.g., using NLTK and Word2Vec), train
classifiers (e.g., Perception, Logistic Regression, and Support Vector
Machines), and evaluate the trained classifiers (e.g., precision, recall, and
F1-score). However, their approach is not specifically developed for PAs and
SCAs and cannot mine assumptions from other sources (e.g., GitHub
repositories).

Compared to the related work above, the tool (i.e., Assumption Miner) proposed
in this work focuses on GitHub repositories, and can automatically collect
data (e.g., issues, PRs, and commits) and identify and extract SCAs and PAs.
Assumption Miner can also be easily extended to work with other repositories
(e.g., Stack Overflow) or other types of software artifacts (e.g., technical
debt).

##  III Assumption Miner

Assumption Miner is composed of four modules: Repository Management, Data
Collection, Data Extraction, and System Management, as shown in Fig. 1.

Repository Management includes (1) getting information of the repositories and
their releases from the GitHub server, (2) searching and showing details of
the repositories, (3) downloading source code of each release of the
repositories, and (4) deleting all the data of specific repositories. Adding a
repository using Assumption Miner requires users to enter the owner (e.g.,
tensorflow) and name (e.g., tensorflow) of the repository. When adding a
repository, Assumption Miner first checks whether the repository exists in the
MySQL database and the GitHub server. If all the checks pass, Assumption Miner
gets information (e.g., URL, releases, and tags) of the repository from the
GitHub server and insert the data into the MySQL database.

Data Collection aims to (1) show the data models of Repository, Release, Tag,
PR, Commit, and Issue, (2) search and show data collection information of
repositories and collect issues, PRs, and commits based on the data models,
(3) monitor data collection processes, and (4) show data collection history.
The data models are predefined and currently cannot be changed by users. The
basic information of each repository (e.g., its releases and tags) and the
information of the data collection processes are stored in a MySQL database,
while the data of issues, PRs, and commits are stored in a MongoDB database.
When collecting data from the GitHub server using Assumption Miner, users can
also set a time (default: 10 seconds) for automatically refreshing the data
collection status (i.e., collecting, finished, and error). Assumption Miner
uses cursors (i.e., issue cursor, PR cursor, and commit cursor) to record each
batch of the data downloaded from the GitHub server, and therefore Assumption
Miner supports continuing data collection after it has been stopped due to
errors and exceptions (e.g., over the limits by GitHub).

Data Identification and Extraction is composed of three submodules: Assumption
Extraction, Data Search, and Knowledge Graph. Assumption Extraction contains
four functions: SCA Identification, SCA Extraction, PA Identification, and PA
Extraction. In the SCA Identification function, we used a keyword-based search
approach for identifying SCAs (i.e., word level), based on the assumption
related search terms (i.e., assumption, assumptions, assume, assumes, assumed,
assuming, assumable, and assumably) and the following search scope: (1) title,
body, body of comments of issues, (2) title, body, body of comments of PRs,
and (3) message of commits. Since the results from using the SCA
Identification function are at the word level (i.e., highlighting the matched
terms), in the SCA Extraction function, Assumption Miner provides support for
locating, matching, and extracting the related sentences that include the
matched terms (i.e., at the sentence level). As most of the description (e.g.,
description of an issue) is created and edited by GitHub users, there could be
punctuation problems existing in the description (e.g., a sentence does not
have a “.” or “.” is used in the source code, such as “a.b”), which may lead
to errors in the separation of the sentences. Therefore, we manually
identified the patterns of SCA description from the projects on GitHub, and
implemented the patterns in the SCA Extraction function to extract SCA
sentences. The outputs of SCA Extraction contain a set of data items as shown
in Table I and Table II. Moreover, there are sentences without using
assumption related keywords, but could act as assumptions (we call them
potential assumptions, PAs). Though PAs are not SCAs, they can be further
reviewed by stakeholders and transformed to SCAs (as the inputs for SCA
extraction). Therefore, Assumption Miner also provides support for identifying
and extracting such assumptions through the PA Identification and PA
Extraction function, which complements SCA identification and extraction. The
first author manually collected and labeled 35,855 sentences from the issues,
PRs, and commits of multiple repositories (e.g., Keras and Theano), following
the guidelines of assumption identification proposed in [4]. Then the other
authors reviewed the results and reached a consensus with the first author.
After assumption collection and labeling, we constructed a dataset for PAs,
fine-tuned a deep learning model based on ALBERT (a lite BERT, which
architecture is based on BERT), and trained and adjusted a classification
model for PA identification [17]. The reason of choosing ALBERT is because
ALBERT is one of the most powerful language models, which can achieve good
performance with fewer parameters (compared to BERT) in many tasks, such as
the binary single-sentence classification task [17]. Since we are using deep
learning models and identifying PAs at the sentence level, the identification
process could be rather slow on CPUs (e.g., it may take hours/days on our
server to identify PAs, depending on the amount of data to be processed).
Therefore, we used a queue (i.e., a waiting list with a first-in first-out
strategy) on the server to manage the tasks of identifying PAs. In the PA
Extraction function, Assumption Miner organizes the PAs (sentences) identified
from the PA Identification function into a file (similar to SCA Extraction),
and users can download the file for further review.

TABLE I: Data Items of Issues and PRs in SCA Extraction Data Item | Description
---|---
owner | The owner of the repository
repo_name | The name of the repository
author | The author of the issue or PR
title | The title of the issue or PR
state | The state of the issue or PR
url | The URL of the issue or PR
SCA | The SCA sentence in the issue or PR
TABLE II: Data Items of Commits in SCA Extraction Data Item | Description
---|---
owner | The owner of the repository
repo_name | The name of the repository
author_name | The author of the issue or PR
message | The message of the issue or PR
url | The URL of the issue or PR
SCA | The SCA sentence in the issue or PR

Data Search aims to search and show specific issues, PRs, and commits based on
keywords. In data search, Assumption Miner requires users to specify which
repository (e.g., TensorFlow), data type (i.e., issue, PR, and commit), search
scope (e.g., title), and keywords (e.g., assume) to search. For example, for
issues of the TensorFlow project, a search scope can be title body
comments.body, which means that Assumption Miner will search data within the
scope of the title of the issues, the body of the issues, and the body of the
comments of the issues in the TensorFlow project. For keywords, Assumption
Miner supports AND (i.e., using double quotation marks, e.g., “assume”
“software”) and OR (i.e., without quotation marks, e.g., assume software). The
search terms are highlighted in the search results. If the description of a
data item is too long, users can click on the “detail” button to see the full
information of the item.

Knowledge Graph supports both traditional knowledge graph and dynamic
knowledge graph. Assumption Miner provides three dimensions (i.e., release,
month, and day) to construct the timeline of the data. For each repository,
Assumption Miner creates and connects entities according to the timeline and
their states. For example, a PR can be published, merged, and closed, and
therefore Assumption Miner creates three connected entities if they are within
a timeline.

System Management supports user registration, login, and logoff, and provides
access control and system logs.

Figure 1: Modules of Assumption Miner

The architecture of Assumption Miner is shown in Fig. 2. When a user clicks on
a menu or button, the Web component organizes the data, generates a request,
and sends it to the Controller component through the Python Interface
component. The Controller component analyzes the request: (1) If the request
is regarding getting data from the GitHub server, the Controller component
organizes the data and calls the functions in the GitHub Service component.
The GitHub Service component reads the GitHub configuration (stored in the
system) and organizes queries based on the predefined data models. Then the
GitHub Service component sends requests to the GitHub server, gets responses
from the GitHub server, analyzes the responses, and sends back the data to the
Controller component. (2) If the request is regarding interacting with the
MySQL or the MongoDB database, the Controller component organizes the data and
calls the functions in the Data Service component. The Data Service component
further organizes the data and calls the functions in the DAO (Data Access
Object) component, which implements the interaction with the MySQL or the
MongoDB database. The DAO component reads the database configuration (stored
in the system), communicates with the databases, gets the results from the
databases, and sends them back to the Data Service component. The Data Service
component sends the data getting from the databases back to the Controller
component. (3) If the request is regarding using the trained model (based on
ALBERT) to identify PAs, the Controller component organizes the data and calls
the functions in the Data Service component. The Data Service component
preprocesses the data, loads the trained model, and uses the model to identify
PAs. The results are then sent back to the Controller component. Finally, the
Controller component organizes the data getting from GitHub, the MySQL
database, the MongoDB database or the deep learning model, sends the data back
to the Web component through the Python Interface component, and then the Web
component shows the results to the user.

Figure 2: Architecture of Assumption Miner

##  IV Using Assumption Miner

In this section, we walk the usage of Assumption Miner through an example: the
TensorFlow project on GitHub 333https://github.com/tensorflow/tensorflow.
TensorFlow is one of the most popular deep learning frameworks, which is
widely used in many deep learning systems and application domains. The
TensorFlow project on GitHub started in 2015, having 188 releases
444https://github.com/tensorflow/tensorflow/releases (accessed on 2023-04-21),
146,893 commits555https://github.com/tensorflow/tensorflow (accessed on
2023-04-21), 22,887 PRs
666https://github.com/tensorflow/tensorflow/pulls?q=is%3Apr (accessed on
2023-04-21), and 37,074 issues
777https://github.com/tensorflow/tensorflow/issues?q=is%3Aissue (accessed on
2023-04-21) till April 2023. Users need to register an account and set a
personal access token of GitHub 888http://www.m58.link/cxaNX when using
Assumption Miner. The token is used to access the GitHub Application
Programming Interface (API), since Assumption Miner needs to communicate with
the GitHub API to get data (e.g., issues, PRs, and commits). We also provide a
default token for Assumption Miner users. However, since GitHub has
limitations in place to protect against excessive or abusive calls to GitHub
servers (e.g., the rate limit is 5,000 points per hour and individual calls
cannot request more than 500,000 total
nodes)999https://docs.github.com/en/graphql/overview/resource-limitations,
using the default token may lead to errors in data collection because of these
limitations. After registration of the Assumption Miner account, users can
login Assumption Miner with the account. Below is the process of using
Assumption Miner to identify and extract assumptions from the TensorFlow
project on GitHub.

Create the TensorFlow repository. Users need to click on the Repository
Management module, then click on the “Add” button, enter the owner as
“tensorflow” and the name as “tensorflow”, and click on the “Save” button to
create the TensorFlow repository on Assumption Miner. For each release of a
repository, Assumption Miner provides users a link to download the source code
in the Repository Management module (this is an optional step). Then users can
use tools such as Visual Studio Code and PyCharm to further browse the code
and search assumptions in the code.

Collect issues, PRs, and commits on TensorFlow. After the TensorFlow
repository is created on Assumption Miner, users can further use the Data
Collection module to collect issues, PRs, and commits of the TensorFlow
repository. Users can start multiple tasks simultaneously, but this could
cause errors because of the limitation by GitHub.

Identify and Extract SCAs on TensorFlow. In the Assumption Extraction
submodule of the Data Extraction module, users need to select the TensorFlow
repository and a data type, and click on the “SCA Identification” button.
Assumption Miner will show the results and highlight all the SCAs. Users can
further extract the data to a csv file to construct a dataset by clicking on
the “SCA Extraction” button. The first line in the csv file is the title,
indicating the repository and type (i.e., issue, PR, and commit) of the
extracted data. If users want to search data on the TensorFlow repository
(optional step), they can use the Data Search submodule of the Data Extraction
module, by selecting the repository (e.g., “TensorFlow”) and the data type
(e.g., “issue”) and specifying the search scope (e.g., “title”) and the search
term (e.g., “assumption”).

Identify and Extract PAs on TensorFlow. In the Assumption Extraction submodule
of the Data Extraction module, users need to select the TensorFlow repository
and a data type, and click on the “PA Identification” button. Assumption Miner
will show the results and highlight all the sentences that may include a PA.
After PA identification, users can click on the “PA Extraction” button to
download the results of PA identification for further review.

Generate a knowledge graph of the assumptions on TensorFlow. This step is
optional. Users need to select the TensorFlow repository and a dimension
(i.e., release, month, or day) to construct a knowledge graph of assumptions
based on the chosen dimension.

The aforementioned results can be further used in various context. For
example, through identifying assumptions, users may better understand what was
assuming in a certain project and deal with such uncertainty in their future
work.

##  V Evaluation of Assumption Miner

We conducted an evaluation on data collection, SCA identification, SCA
extraction, PA identification, and PA extraction, as shown in Fig. 3. The
output of data collection is the input of SCA identification and PA
identification, the output of SCA identification is the input of SCA
extraction, and the output of PA identification is the input of PA extraction.

Figure 3: Evaluation of Assumption Miner

###  V-A Evaluation of Data Collection

We conducted an evaluation of using Assumption Miner to collect issues, PRs,
and commits on seven GitHub repositories: Caffe, CNTK, Theano, DeepLearning4J
(DL4J), MXNet, Keras, and TensorFlow, regarding the completeness and
performance of Assumption Miner on data collection. We first created an issue
data model (including repository name, title, ID, author, URL, labels, state,
body, and comments), a PR data model (including repository name, owner, title,
ID, author, URL, labels, state, body, comments, and reviews), and a commit
data model (including repository name, owner, OID, author name, author email,
committed date, URL, and message). The data items of each data model were
selected from the GraphQL API on
GitHub101010https://docs.github.com/en/graphql/reference/objects.

The configuration of the server we used for the evaluation of the data
collection is: (1) CPU: Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50GHz, 2
cores, (2) Memory: 2GB, (3) Hard Disk: 40GB SSD, (4) Operation System: Linux
VM-20-4-centos 3.10.0-1160.45.1.el7.x86_64 #1 SMP. The results of using
Assumption Miner to collect data are shown in Table III.

TABLE III: Completeness and Performance of Data Collection of using Assumption Miner Repository | Data Items on GitHub | Collected Data Items | Data Type | Time (s) | Access Date
---|---|---|---|---|---
Caffe | 4,786 | 4,786 | Issue | 400 | 2022/11/4
Caffe | 2,238 | 2,238 | PR | 152 | 2022/11/4
Caffe | 4,156 | 4,156 | Commit | 51 | 2022/11/4
CNTK | 3,288 | 3,288 | Issue | 228 | 2022/11/4
CNTK | 557 | 557 | PR | 27 | 2022/11/4
CNTK | 16,117 | 16,117 | Commit | 181 | 2022/11/4
Theano | 2,671 | 2,671 | Issue | 246 | 2022/11/4
Theano | 4,114 | 4,114 | PR | 198 | 2022/11/4
Theano | 28,127 | 28,127 | Commit | 335 | 2022/11/4
DL4J | 5,652 | 5,652 | Issue | 290 | 2022/11/7
DL4J | 4,185 | 4,185 | PR | 140 | 2022/11/7
DL4J | 2,606 | 2,606 | Commit | 25 | 2022/11/7
MXNet | 9,532 | 9,532 | Issue | 465 | 2022/11/8
MXNet | 11,090 | 11,090 | PR | 551 | 2022/11/8
MXNet | 11,893 | 11,893 | Commit | 145 | 2022/11/7
Keras | 11,518 | 11,518 | Issue | 519 | 2022/11/8
Keras | 5,670 | 5,670 | PR | 208 | 2022/11/8
Keras | 7,493 | 7,493 | Commit | 65 | 2022/11/8
TensorFlow | 35,966 | 35,966 | Issue | 2,402 | 2022/11/9
TensorFlow | 22,119 | 22,119 | PR | 1,105 | 2022/11/9
TensorFlow | 138,366 | 138,366 | Commit | 1,295 | 2022/11/9

The results show that Assumption Miner can effectively collect issues, PRs,
and commits of the repositories on GitHub. Certain repositories (e.g.,
TensorFlow) may be frequently updated, and there is a need to construct a
mechanism for continuously collecting data.

###  V-B Evaluation of SCA Identification

After data collection (as mentioned in Section V-A), we used the collected
data (i.e., issues, PRs, and commits) of the Keras and TensorFlow repository
to conduct an evaluation on identifying SCAs using Assumption Miner (i.e., the
SCA Identification function). The first author manually checked the identified
results to classify them as SCAs or non-SCAs. The evaluation results of SCA
identification are shown in Table IV.

The count of the identified SCAs could be larger than the search results
(e.g., count of messages in the commits of the Keras repository), since each
issue, PR, or commit may include multiple SCAs. For example, an issue
111111https://github.com/keras-team/keras/issues/395 of Keras mentions:
“Assume we are trying to learn a sequence to sequence map. For this we can use
Recurrent and TimeDistributedDense layers. Now assume that the sequences have
different lengths. We should pad both input and desired sequences with zeros,
right? But how will the objective function handle the padded values? There is
no choice to pass a mask to the objective function. Won’t this bias the cost
function?”, which includes two SCAs: “assume we are trying to learn a sequence
to sequence map” and “assume that the sequences have different lengths”.

Since Assumption Miner used a keyword-based (i.e., the assumption related
terms) search approach for SCA identification, it could go wrong in certain
context (e.g., a variable in a code snippet named assume). Moreover, we also
found that certain SCAs lack details. For example, an issue
121212https://github.com/keras-team/keras/issues/1174 of Keras mentioned:
“strict enforcement of user input assumptions, and raising of helpful error
messages.” However, we cannot understand what exactly the user input
assumptions are. These SCAs need to be further processed by Assumption Miner
(e.g., add warnings in the results).

TABLE IV: Results of Identifying SCAs using Assumption Miner Repository | Data Type | Search Field | Search Results | Identified SCAs | Misidentification
---|---|---|---|---|---
Keras | Issue | title | 3 | 3 | 0
Keras | Issue | body | 253 | 298 | 13
Keras | PR | title | 3 | 3 | 0
Keras | PR | body | 59 | 65 | 0
Keras | Commit | message | 10 | 11 | 0
TensorFlow | Issue | title | 13 | 13 | 0
TensorFlow | Issue | body | 662 | 797 | 47
TensorFlow | PR | title | 5 | 5 | 0
TensorFlow | PR | body | 136 | 150 | 6
TensorFlow | Commit | message | 567 | 616 | 39
Total | 1,711 | 1,961 | 105

Regarding the performance of the SCA Identification process, we used an Aliyun
server 131313https://www.alibabacloud.com/en with the following configuration:
(1) Server type: ecs.s6-c1m2.xlarge, (2) CPU: 4 cores (vCPU), (3) Memory: 8GB,
(4) Hard Disk: 40GB SSD, (5) Operation System: Windows Server 2022, and
conducted an evaluation on the issues, PRs, and commits of Keras and
Tensorflow. The results are shown in Table V.

TABLE V: Performance of the SCA Identification Process Repository | Data Type | Identification Field | Time (s)
---|---|---|---
Keras | Issue | title, body, comments.body | 7.287
Keras | PR | title, body, comments.body | 0.650
Keras | Commit | message | 0.378
TensorFlow | Issue | title, body, comments.body | 7.419
TensorFlow | PR | title, body, comments.body | 0.632
TensorFlow | Commit | message | 0.681

The results show that Assumption Miner can correctly identify 94.92% SCAs
(1,961 SCAs out of 2,066 SCAs) from the issues, PRs, and commits of the Keras
and TensorFlow repository. Certain variables and functions named as for
example assume exist in issues, PRs, and commits, which needs to be further
processed by Assumption Miner.

###  V-C Evaluation of SCA Extraction

We further evaluated whether Assumption Miner (i.e., the SCA Extraction
function) can correctly extract SCAs (i.e., at the sentence level) using the
identification results (i.e., 1961 identified SCAs) of the Keras and
TensorFlow repository from V-B. The first author manually checked the
extracted results to classify them as correct extraction and missed
extraction. The results are shown in Table VI. As an example, there are 298
SCAs in the body of Keras issues. Assumption Miner correctly extracted 290 of
the SCAs, but missed 8 SCAs.

TABLE VI: Results of Extracting SCAs using Assumption Miner Repository |  Data Type |  Extraction Field |  SCAs |  Correct |  Missed
---|---|---|---|---|---
Keras |  Issue |  title |  3 |  3 |  0
Keras |  Issue |  body |  298 |  290 |  8
Keras |  PR |  title |  3 |  3 |  0
Keras |  PR |  body |  65 |  65 |  0
Keras |  Commit |  message |  11 |  11 |  0
TensorFlow |  Issue |  title |  13 |  13 |  0
TensorFlow |  Issue |  body |  801 |  772 |  29
TensorFlow |  PR |  title |  5 |  5 |  0
TensorFlow |  PR |  body |  150 |  146 |  4
TensorFlow |  Commit |  message |  616 |  609 |  7
Total |  1,961 |  1,913 |  48

The results show that Assumption Miner can correctly extract 97.55% SCAs
(1,913 SCAs out of 1,961 SCAs) from the issues, PRs, and commits of the Keras
and TensorFlow repository. Certain structures of the issues, PRs, and commits
(e.g., “assume” and “assumption” exist in one sentence) may lead to errors in
SCA extraction, which needs further investigation and improvements.

###  V-D Evaluation of PA Identification

For PAs, we manually labeled 35,855 sentences from the issues, PRs, and
commits of multiple repositories (e.g., Keras and Theano), and constructed a
dataset, containing a training set and a test set with a data proportion of
8:2 from the labeled sentences. We created a vocabulary and tokenized the data
from the dataset based on the vocabulary. Then we constructed the deep
learning model (based on ALBERT), trained the model for 50,000 epochs with a
batch size of 32 and a learning rate of 2e-5.

Regarding the performance of the PA identification process, we used the same
configuration used in the evaluation of SCA identification (see Section V-B),
and conducted an evaluation on the issues, PRs, and commits of Keras and
Tensorflow. The results are shown in Table VII.

TABLE VII: Performance of the PA Identification Process Repository | Data Type | Identification Field | Time
---|---|---|---
Keras | Issue | title, body, comments.body | 11h 12m 58s
Keras | PR | title, body, comments.body | 2h 28m 6s
Keras | Commit | message | 24m 4s
TensorFlow | Issue | title, body, comments.body | 47h 2m 34s
TensorFlow | PR | title, body, comments.body | 12h 16m 59s
TensorFlow | Commit | message | 10h 19m 44s

Using GPU would significantly improve the performance of the PA identification
process. As an example, we used NVIDIA GeForce RTX 3060 Ti to run the PA
identification process on Keras. The results are 16 minutes 52 seconds
(compared to 11 hours 12 minutes 58 seconds using CPU) on issues, 3 minutes 51
seconds (compared to 2 hours 28 minutes 6 seconds using CPU) on PRs, and 43
seconds (compared to 24 minutes 4 seconds using CPU) on commits of Keras.

The results show that the best accuracy of identifying PAs using Assumption
Miner on the test set is 0.9451 on Epoch 22,000. Considering repositories may
have various context (e.g., different development policies), there is a need
to further extend the dataset to include more data from the repositories,
which will help to improve the generalization ability of Assumption Miner in
identifying PAs.

##  VI Conclusions

Assumptions and their management are important in software development. The
prerequisite of analyzing and understanding assumptions in software
development is to identify and extract those assumptions with acceptable
effort. To this end, we proposed Assumption Miner to automatically identify
and extract assumptions on GitHub projects. Besides providing a running
example of using Assumption Miner on the TensorFlow project, we also evaluated
the performance of Assumption Miner, and the results show that Assumption
Miner can effectively identify and extract assumptions from the repositories
on GitHub. Assumption Miner can be potentially used for the research topics
regarding assumptions and their management in software development, such as
assumption making, evolution, evaluation, and reasoning.

For future work, the following aspects of Assumption Miner can be further
optimized: (1) There is a need to construct a mechanism for continuously
collecting data using Assumption Miner. (2) The identification of SCAs and PAs
can be further optimized (e.g., develop new deep learning models, construct a
larger dataset and train deep learning models based on the dataset). (3)
Certain patterns of the issues, PRs, and commits (e.g., a variable named
“assume”) may lead to incorrect SCA identification and extraction, which can
be further addressed in Assumption Miner. (4) Assumptions are related to
various types of software artifacts (e.g., requirements, design decisions, and
technical debt). Automatically recovering the relationships between
assumptions and such artifacts in Assumption Miner is a promising future
direction. (5) Besides SCAs and PAs, there are many implicit assumptions in
projects, which should also be identified and extracted in the future.

## Acknowledgments

This work is funded by Shenzhen Polytechnic with Grant No. 6022312043K, State
Key Laboratory for Novel Software Technology at Nanjing University with Grant
No. KFKT2022B37, and the National Natural Science Foundation of China (NSFC)
with Grant No. 62172311.

## References

  * [1] C. B. Haley, R. C. Laney, J. D. Moffett, and B. Nuseibeh, “Using trust assumptions with security requirements,” _Requirements Engineering_ , vol. 11, no. 2, pp. 138––151, 2006.
  * [2] R. Roeller, P. Lago, and H. van Vliet, “Recovering architectural assumptions,” _Journal of Systems and Software_ , vol. 79, no. 4, pp. 552–573, 2006.
  * [3] M. M. Lehman and J. F. Ramil, “Rules and tools for software evolution planning and management,” _Annals of SE_ , vol. 11, no. 1, p. 15–44, 2001.
  * [4] C. Yang, P. Liang, and P. Avgeriou, “Assumptions and their management in software development: A systematic mapping study,” _Information and Software Technology_ , vol. 94, no. 2, pp. 82–110, 2018.
  * [5] F. J. Corbató, “On building systems that will fail,” _Communications of the ACM_ , vol. 34, no. 9, p. 72–81, 1991.
  * [6] D. Garlan, R. Allen, and J. Ockerbloom, “Architectural mismatch: Why reuse is still so hard,” _IEEE Software_ , vol. 26, no. 4, p. 66–69, 2009.
  * [7] G. A. Lewis, S. Bellomo, and I. Ozkaya, “Characterizing and detecting mismatch in machine-learning-enabled systems,” in _Proceedings of the 1st Workshop on AI Engineering: Software Engineering for AI (WAIN)_ , Madrid, Spain, 2021, pp. 133–140.
  * [8] “Status of the Boeing 737 MAX - hearing before the subcommittee on aviation of the committee on transportation and infrastructure house of representatives,” U.S. Government Publishing Office, 2019. [Online]. Available: https://www.govinfo.gov/content/pkg/CHRG-116hhrg37277/pdf/CHRG-116hhrg37277.pdf
  * [9] “Final committee report - the design, development, and certification of the Boeing 737 MAX,” The Committee on Transportation and Infrastructure, 2020. [Online]. Available: https://www.edpierson.com/final-737-max-report-for-public-release
  * [10] C. Yang, P. Liang, and P. Avgeriou, “A survey on software architectural assumptions,” _Journal of Systems and Software_ , vol. 113, no. 3, pp. 362–380, 2016.
  * [11] C. Yang, P. Liang, L. Fu, and Z. Li, “Self-claimed assumptions in deep learning frameworks: An exploratory study,” in _Proceedings of the 25th International Conference on Evaluation and Assessment in Software Engineering (EASE)_ , Trondheim, Norway, 2021, pp. 139–148.
  * [12] Z. Li, P. Avgeriou, and P. Liang, “A systematic mapping study on technical debt and its management,” _Journal of Systems and Software_ , vol. 101, pp. 193–220, 2015.
  * [13] C. Yang, Z. Ma, P. Liang, and X. Liu, “Deployment package of Assumption Miner,” 2023. [Online]. Available: https://doi.org/10.5281/zenodo.7725115
  * [14] D. V. Landuyt and W. Joosen, “A descriptive study of assumptions made in linddun privacy threat elicitation,” in _Proceedings of the 35th Annual ACM Symposium on Applied Computing (SAC)_ , Brno, Czech Republic, 2020, pp. 1280–1287.
  * [15] Z. Xiong, P. Liang, C. Yang, and T. Liu, “Assumptions in OSS development: An exploratory study through the hibernate developer mailing list,” in _Proceedings fo the 25th Asia-Pacific Software Engineering Conference (APSEC)_ , Nara, Japan, 2018, pp. 455–464.
  * [16] R. Li, P. Liang, C. Yang, G. Digkas, A. Chatzigeorgiou, and Z. Xiong, “Automatic identification of assumptions from the hibernate developer mailing list,” in _Proceedings fo the 26th Asia-Pacific Software Engineering Conference (APSEC)_ , Putrajaya, Malaysia, 2019, pp. 394–401.
  * [17] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “Albert: A lite bertt for self-supervised learning of language representations,” in _Proceedings of the 8th International Conference on Learning Representations (ICLR)_ , Virtual Conference, 2020.