11institutetext: School of Computer Science and Artificial Intelligence, Wuhan
University of Technology, Wuhan 430070, China
22institutetext: School of Computer Science, Wuhan University, Wuhan 430072,
China
22email: {268595, cathylilin<EMAIL_ADDRESS><EMAIL_ADDRESS>
# On Unified Prompt Tuning for Request Quality Assurance in Public Code Review

Xinyu Chen 11 Lin Li 1*1* Rui Zhang 11 Peng Liang 22

###### Abstract

Public Code Review (PCR) can be implemented through a Software Question
Answering (SQA) community, which facilitates high knowledge dissemination.
Current methods mainly focus on the reviewer’s perspective, including finding
a capable reviewer, predicting comment quality, and recommending/generating
review comments. Our intuition is that satisfying review necessity requests
can increase their visibility, which in turn is a prerequisite for better
review responses. To this end, we propose a unified framework called UniPCR to
complete developer-based request quality assurance (i.e., predicting request
necessity and recommending tags subtask) under a Masked Language Model (MLM).
Specifically, we reformulate both subtasks via 1) text prompt tuning, which
converts two subtasks into MLM by constructing prompt templates using hard
prompt; 2) code prefix tuning, which optimizes a small segment of generated
continuous vectors as the prefix of the code representation using soft prompt.
Experimental results on the Public Code Review dataset for the time span
2011-2023 demonstrate that our UniPCR framework adapts to the two subtasks and
outperforms comparable accuracy-based results with state-of-the-art methods
for request quality assurance. These conclusions highlight the effectiveness
of our unified framework from the developer’s perspective in public code
review.

###### Keywords:

Request quality assurance, Public code review, Prompt

##  1 Introduction

Code review can be conducted either within the team or by public developers
[24]. Team code review primarily focuses on identifying defects, ensuring the
need for code adjustments, and aiming to enhance the efficiency of the review
process [19, 31]. Public Code Review (PCR) serves as a complementary approach,
intended to improve knowledge sharing and expertise among relevant
professionals during the review process [2].

Different from code review within the team, PCR is developed in the Software
Question Answering (SQA) community, thus researchers are increasingly
exploring ways to achieve high-quality and efficient review services [41].
Recent studies have focused on code review within teams such as reducing
review time [30] and recommending suitable reviewers [28, 15]. Other studies
aim to normalize reviewer responses by predicting the quality of review
comments [29, 27] to obtain better review responses. The common goal of these
works is to optimize the review process for better team code review service.

In the PCR process implemented by the SQA community, request quality assurance
mainly depends on developers. As developers are required and encouraged to
submit necessary PCR requests in the community, the necessity of these
requests is evaluated by public via request necessity prediction subtask after
they are posted. Meanwhile, developers always need to choose a few technical
terms (e.g., c, pthreads) as review tags. These tags can be used to match
requests with suitable reviewers via tag recommendation subtask (see the
detailed PCR process in Section 2.1).

However, when the above subtasks need to be processed, prior researches
usually fine-tuned [37] modelling framework for each subtask. This means that
task-specific learning frameworks require additional costs to adapt the model
to different subtasks, which is contrary to the demand for simple and fast
tools in the field of software engineering [36]. Inspired by prompt tuning
[25], we aim to bridge this gap and improve request quality assurance
performance by reformulating the traditional discriminative learning widely
used in the request necessity prediction subtask and the tag recommendation
subtask as generative learning. To this end, we propose a unified prompt
tuning that allows us to learn and train both subtasks for request quality
assurance. Specifically, we reformulate these discriminative subtasks via 1)
text prompt tuning, which converts them into generative learning by
constructing prompt templates using hard prompt (see Section 3.1.1). The
description of the subtasks prompts the model to learn the subtasks; and 2)
code prefix tuning, which optimizes a small segment of continuous vectors in
the entire code feature space (see Section 3.1.2). Using this short prefix to
further optimize the representation learning of the code.

The contributions of this paper can be summarized as follows: 1) We propose a
Unified framework for Public Code Review (UniPCR) to consider the different
subtasks of request quality assurance in the public code review service. The
UniPCR framework completes the unified training of both subtasks through the
designed prompt template.

2) Experimental results on the Public Code Review dataset for the time span
2011-2023 demonstrate that UniPCR outperforms comparable accuracy-based
results with state-of-the-art methods for the request quality assurance.

##  2 Background

###  2.1 Public Code Review Process

In this section, we outline how these request quality assurance subtasks
(request necessity prediction and tag recommendation subtask) serve the public
code review process [24]. The process is comprised of five core steps (as
depicted in Figure 1), starting with the code writer submitting a request for
change or optimization in Step ①. In Step ②, practitioners in the community
browse and evaluate the request to assign a necessity feedback. Compliance
with community norms is essential for requests to receive a response, and
unnecessary requests may hinder visibility. Subsequently, in Step ③,
developers are asked to select several relevant technical terms to accompany
their request. These tags are then used in Step ④ to match the request with
appropriate reviewers, who provide their feedback on the code in Step ⑤.

Our task logic involves integrating the necessity prediction with Steps ② to
provide practitioners with necessity feedback in advance. This can guide them
in optimizing their request expression and improving request visibility while
reducing the time spent by reviewers in Step ⑤. Additionally, the tag
recommendation is associated with Steps ④, allowing requests to be correctly
matched with appropriate reviewers based on the use of tags within the PCR
community.

Figure 1: An Example of Public Code Review Process (with a data sample in the
middle

###  2.2 Request Quality Assurance Task Definition

The effectiveness of the PCR process hinges on the interaction between
developers and reviewers. Unfortunately, this interaction can be time-
consuming due to the need to identify issues, find appropriate reviewers, etc.
While current approaches focus on reducing time costs for reviewers, they
overlook the impact of developers on the review process. Our intuition is that
well-crafted code review requests are essential to eliciting quality
responses. To address this, we propose a PCR service that leverages the
request necessity prediction subtask and the tag recommendation subtask.

When considering a request in the PCR community, we assume the existence of a
corpus of PCR requests (denoted by $R$), a set of tag labels (denoted by $L$),
and a set of quality labels for all PCR requests (denoted by $Q$). Formally
speaking, given a PCR request $r\in R$, the request contains a title $t\in T$,
a text description $d\in D$, and a code segment $c\in C$. Code segments are
extracted from descriptions.

Request necessity prediction subtask. The request necessity prediction subtask
aims to acquire a function $f_{quality}$ that maps $r$ to a quality label
$q\in Q$. A review request $r$ corresponds to the truth label $q_{r}$. Thus,
the request necessity prediction subtask can be viewed as a classification
problem. The target is to identify an optimal function $f_{quality}$ that maps
the quality label $q$ as similar to the true quality label $q_{r}$.

Tag recommendation subtask. The tag recommendation subtask aims to obtain a
function $f_{tagrec}$ that maps $r$ to a subset of tags
$l=\\{l_{1},l_{2},\dots,l_{m}\\}\subset L$, which are most relevant to the
request $r$. A review request $r$ corresponds to a set of tags
$l_{r}=\\{l_{r1},l_{r2},\dots,l_{rk}\\}\subset l$, where $k$ denotes the
number of truth tags for the review request. Thus, the tag recommendation
subtask can be viewed as a multi-label classification problem. The target is
to identify an optimal function $f_{tagrec}$ that maps the tag subset $l$ as
similarly as possible to the truth tag set $l_{r}$.

According to the above definitions, in order to better use the UniPCR to solve
these two tasks, we use prompt engineering [1] to reformulate them as
generation tasks (details in Section 3). Here, we formally introduce the
problem of the PCR service under UniPCR as follows.

Our Unified Task. First, we design a descriptive prompt template for both two
subtasks, which is summarized as $T(\cdot)=$“x $[MASK]$” (where x denotes a
textual string). A request input sequence $r$ is passed through this
descriptive prompt and refactored into $r_{prompt}=T(r)$. The request
necessity prediction and tag recommendation subtasks is made by filling in the
position of $[MASK]$. Our problem is how to improve the proximity of the
predicted filler words against the truth label. We denote the total number of
training examples as $N$. For the tag recommendation subtask, the truth tags
for a request are $l_{r}$. Note that a PCR request can be labeled with up to
five tags, so $\left|l_{r}\right|\leq 5$.

##  3 Framework

Figure 2 plots the architecture of UniPCR for prompt tuning with three
modules: the text prompt tuning, the code prefix tuning, and the unified
prompt tuning.

In the field of natural language processing (NLP), language prompts have been
proven to be effective in leveraging the knowledge learned by pre-trained
language models (PLMs) [10, 40]. Through a series of research works on
knowledge exploration, hard prompt has been used to reformulate downstream
tasks with the help of text prompt templates. By selecting appropriate
prompts, the model’s behavior can be manipulated so that the PLM can predict
the desired output, even without any additional task-specific training [33,
32, 7].

Figure 2: The overview of unified framework for public code review (UniPCR).

For a request input $r$, we divide the input PCR request into: 1) natural
language text $r_{text}$, which is composed of the title and description (text
part) in the request, that is, $r_{text}=\\{t,d\\}$, and 2) code segment
$r_{code}$, which indicates the code snippet extracted from the description,
that is, $r_{code}=\\{c\\}$. This module uses hard prompt [1] to deal with the
input of natural language text by templates and soft prompt for code segment
by prefix.

###  3.1 Prompt Template Design

####  3.1.1 Text Prompt

First, we apply a descriptive prompt template $T(\cdot)$ to map the input
natural language text $r_{text}=\\{t,d\\}$ to $T(r_{text})$. The prompt
template preserves the original markup in $r_{text}$ and includes at least one
$[MASK]$ for the PLM to populate with label words as in the two subtasks.

Specifically, we modify the descriptive prompt text string x according to the
subtask goal to reconstruct the data for different tasks. Expressing
$r_{prompt}$ as $T(r_{text})$ = {“The topic of the request is $[MASK]$
$[MASK]$ $[MASK]$. This request necessity is $[MASK]$.”, $r$}.

Figure 3: An Example of Refactoring Request Prediction and Tag Recommendation
Subtasks under a UniPCR.

####  3.1.2 Code Prompt

The code segment possesses an inherent structure that provides valuable
semantic information for code understanding. To better capture this
information, Guo et al. [8] proposed using data flow graphs to analyze code
and represent its semantic information. Unlike the Abstract Syntax Tree (AST)
used in previous research for code parsing [35, 22], this graph-based data
flow structure effectively avoids the unnecessary deep hierarchical structure
that AST may introduce. Following this approach, we parse code segments into
data flow graphs to represent their semantics and variable dependent
relationships.

To optimize the code representation and better capture the knowledge embedded
in the code segment, soft prompt [25] is employed. Soft prompt is a prompt
paradigm that performs directly in the model’s embedding space and tunes the
prompt parameters to the downstream task. This approach has demonstrated
superior performance on various non-natural language data, such as image
coding and visual question answering [21, 20, 18]. It is well-suited for
learning and optimizing code segments with such a graph structure, as it can
better adapt data forms to a data flow graph.

###  3.2 Unified Prompt Tuning in PCR

####  3.2.1 Prompt Engineering.

In Step 1, we first parse a code segment $c$ into a data flow graph to
represent the dependency relationship $Graph_{c}$ between code variables
(shown on the red part one the right side of Figure 2).

Then the prompt project of prefix tuning is used to refactor the code segment
which creates the code input $c_{prefix}$ by obtaining a prefix vector
$P_{REFIXcode}$ defined in step 3. To address the issues of low performance
and high variance resulting from random initialization of $P_{REFIX}$, it has
been suggested that using real words to activate prefix vectors can achieve
better results [37]. Specifically, we initialize the prefix vector
$P_{REFIXcode}$ for code segment prefix tuning using a small piece of code
segment as $c^{\prime}$. The small segment vector $P_{REFIXcode}$ is used to
optimize its representation based on full-text understanding of code segments,
and then a non-updatable data flow graph is adopted based on semantic
relations to represent the meaning of individual code segments. This enables a
global and local understanding of the meaning of each code segment.

In Step 4, the code segment $c_{prefix}$ replaces the original code segment
vectors in the request $r$. Through prompt engineering, the review request is
refactored into $r_{prompt}$, and $\tilde{r_{prompt}}$ represents the hidden
states of this input. That is,
$\tilde{T(\cdot)},\,\tilde{t},\,\tilde{d},\,\tilde{c_{prefix}}$ respectively
represent the vectors obtained by $T(\cdot),\,t,\,d,\,c_{prefix}$.

Input: $Request=(R_{1},R_{2}\cdots R_{n})$, each data
$R_{i}=r=(t,d,c,l,q)$(title, description, code, tag labels, necessary labels)

Output: $\tilde{r_{prompt}}$

1 $Graph_{c}\leftarrow c$ , $Graph_{c}$ represents code data flow graph

2 $P_{REFIXcode}\leftarrow PLM(c^{\prime})$, $c^{\prime}$represents a part of
the whole code $c$, $PLM(c^{\prime})$ represents code embedding by the
encoding model

$c_{prefix}=[P_{REFIXcode},Graph_{c}]$
$\tilde{r_{prompt}}=\\{\tilde{T(\cdot)},\tilde{t},\tilde{d},\tilde{c_{prefix}}\\}$

Algorithm 1 A Request Prompt Pre-processing

####  3.2.2 Prompt Tuning.

Next, we concatenate the obtained text segment vectors and code segment
vectors into the input $\tilde{r_{prompt}}$ of the MLM model. In the entire
input, $\tilde{Graph_{c}}$ is a non-updatable vector, while the remaining
vectors update the representation of the target optimization vector according
to the gradient.

As shown in Figure 3, the vector ${\tilde{Graph_{c}}}$ encoded by the hidden
layer of code PLM will be frozen and not used for updating, while the prefix
is obtained by using the code segment text initialization. Finally, the vector
$\tilde{P_{REFIXcode}}$ can be trained. Code prefix tuning initializes a
trainable matrix $P_{\sigma}$ (parameterized by $\sigma$) of dimension
$|P_{idx}|\times dim(\tilde{P_{REFIXcode}})$ to store the prefix parameters.

As multiple $[MASK]$ tokens may be present in a descriptive prompt template,
considering all masked positions for prediction. The final learning objective
of our UniPCR framework is to maximize the result of Equation (1).

$\log{p_{\phi}(l|r)}=\prod_{j=1}^{n}p_{\phi}([MASK]_{j}=\varphi_{j}(l_{r})|T(r)).$
(1)

where $n$ is the number of masked positions in a descriptive prompt template
$T(r)$, and $\varphi_{j}(l_{r})$ is the mapping of the $j$th word in class $l$
and the predicted word to $j$th mask position $[MASK]$. The set of updatable
parameters has changed, and the model parameter $\sigma$ is a fixed parameter.
The prefix parameter $\sigma$ is the only trainable parameter.
$\tilde{P_{REFIXcode}}$ is a vector that can be used to update $P_{\sigma}$.

####  3.2.3 Answering Engineering.

The probability $p([MASK]=y)$ is calculated for each word at position $[MASK]$
by feeding $r_{prompt}$ into an MLM, where $y\in Y$ are tokens in the PLM
vocabulary. The probabilities are sorted to obtain a TOP K token list
$y_{list}$, which is mapped to the prediction label $P=\varphi(y_{list})$
through the answer project $\varphi(\cdot)$.

To illustrate how we design an answer space, a tag recommendation subtask is
used as an example. The answer engineering $\varphi(\cdot)$ will map the
predicted word to the final label set $L$ and $Q$. For a given request $r$, a
hidden vector $h_{[MASK]}$ can be obtained after passing through the MLM.
Subsequently, by passing through the MLM header, a set of probabilities
$p_{j}(l|r),j\in(0,\max{|v_{t}|})$ can be retrieved. By sorting the
probabilities, the score for each vocab in the PLM vocabulary $Y$ can be
obtained. Since the predicted answer word $v$ may not be exactly the same as
the words in the label set, we design a mapping function $\varphi$ to measure
the truth of the answer using a matching function and the minimum edit
distance. Finally, the mapped answer $\varphi(y)$ is the final top-k output
$v_{klist}$.

##  4 Experiments

###  4.1 Dataset and Metrics

#### Dataset.

To evaluate the effectiveness of UniPCR utilizing prompt learning for the PCR
(public Code Review) task, we conducted an empirical study using the dataset
available which is sourced from
StackExchange111https://archive.org/details/stackexchange, the largest public
code review website, and it covers the raw data version from 2011 to 2023 [6].
In an entire PCR dataset a tag is treated as rare if its occurrence is below a
predefined threshold $\theta$. Intuitively, if a tag rarely appears in such a
large corpus of data, it is likely to have been a miscreated tag that is not
widely recognized by developers. Therefore, these rare tags are removed, as
they may reduce the quality of the training and testing data. In line with
previous research [5, 4, 11], we set the threshold $\theta$ for rare tags to
50, while all requests and tags that are rare or contain only rare tags are
deleted. In the end, the whole dataset contains a total of 424 tags. After
cleaning, we obtained a dataset consisting of 76,161 requests. Following
previous work [4, 11], the dataset is split into training, validation, and
test sets using an 8:1:1 ratio.

Table 1: Statistics for Public Code Review Dataset (Span 2011-2023) | Train | Test | Validation |  | Train | Test | Validation
---|---|---|---|---|---|---|---
Total Data | 61688 | 7616 | 6855 |  |  |  |
Request Necessity |  |  |  | Tag Recommendation |  |  |
Necessary | 26632 | 3288 | 2959 | Avg. tags/request | 2.95 | 2.94 | 2.89
Unnecessary | 35056 | 4328 | 3896 | Avg. words/tag | 2.89 | 3.0 | 2.89

#### Metrics

For the tag recommendation subtask, previous recommendation tasks [11, 5, 4,
26] in the SQA community use $Presision@K$, $Recall@K$, $F1@K$ to evaluate the
performance of the approaches. To be consistent, we use the same evaluation
metrics in this work. The request necessity prediction subtask is consistent
with the review comment request necessity prediction subtask [27, 29] in the
PCR field, and $Accuracy$ and the label (necessary for positive and necessary
for negative) $F1$ are selected as prediction indicators [14].

###  4.2 Implementation Details

#### Implementation

Even though our UniPCR framework can be applied to various pre-trained
language models such as BERT [9], RoBERTa [43], ALBERT [44], GPT [34], etc.
Due to the limitations of hardware devices, BERT was selected as our encoding
model (BERT-base-uncased version). In the UniPCR, an MLM based on BERT is used
as the pre-trained model for the experiments below.

Building on the success of PLM in NLP, pre-trained models for programming
languages have also been developed. To better capture the inherent structure
of code, Guo et al. [8] proposed GraphCodeBert, a pre-trained model based on
BERT that uses the data flow graph structure of code and natural language.
This model shows significant improvements in downstream coding-related tasks.
In this work, we leverage GraphCodeBert [8] for code model to construct code
prefix tuning.

In our UniPCR, the best performance is obtained when the learning rate runs to
1e-5, the initial value of the trainable curvature to 3.0, then use two
hyperbolic graph convolution layers for two datasets. For margin ranking loss,
we run the margin as 1.5 and negative number as 256. We still use Adam
Optimizer [12] to optimize our model parameters by computing different
adaptive learning rates, running the epoch to 6 for the best results, and the
model is saved every 10,000 steps. All experiments were performed using
version 1.3.1 of Pytorch on a commodity machine equipped with 2 GTX 3090 and a
total of 48GB memory.

#### Baselines

For overall comparisons, a series of state-of-the-art approaches are chosen as
baselines for the tag recommendation subtask and the request necessity
prediction subtask, respectively.

1) For the request necessity prediction subtask, since existing research pays
less attention to request necessity-related tasks, comment-based review work
is selected as our baseline experiment [14, 41, 23].

2) For the tag recommendation subtask, recent methods are selected since they
work on recommending tags for the SQA communities [11, 5, 17, 4].

We reproduce the open source code provided by the authors, re-optimize the
parameters for each benchmark on our dataset, and select the best result as
the benchmark performance. Since comment services in PCR are often implemented
as tools, our replication is based on their published papers.

##  5 Experimental Results

###  5.1 Request Necessity Prediction Subtask

Table 2 shows the UniPCR experimental results on the request necessity
prediction compared with similarity-based and fine-tuning-based methods.

Table 2: Experiment Results of Request Necessity Prediction Subtask Model | Accuracy | F1 Necessary | F1 Unnecessary
---|---|---|---
CommentBERT [23] | 0.608 | 0.606 | 0.610
MetaTransformer [14] | 0.623 | 0.687 | 0.525
Commentfinder [41] | 0.616 | 0.711 | 0.426
Our UniPCR | 0.798 | 0.817 | 0.775
% Improv. | 28.0% | 14.9% | 27.0%

1) Our unified framework is effective on the request necessity prediction
subtask. PCR requests and comments typically consist of both text and code,
three baselines are used to compare the performance of our UniPCR. Those
baselines apply traditional finetuning training and similarity paradigm for
request necessity prediction downstream subtask. Our unified prompt tuning
framework and achieve an improvement of 18.9% - 28% over the best performance
of baselines. In general, UniPCR demonstrates its effectiveness on this
subtask by refactoring it into a framework approach such as a generation task.
Through this framework, the model learns label semantics and associates it
with the overall information of the request, resulting in high-quality
performance in request necessity prediction.

2) UniPCR performs better than traditional architecture. Specifically, the
baseline [14] shows sub-optimal performance, which uses fine-tuning paradigm
with a encoding layer for text embedding and a mapping layer for predicting.
Our UniPCR framework based on prompt learning has an improvement of 18.9% over
it. This result highlights that while fine-tuning paradigms can be designed
for specific tasks, a more unified framework can achieve competitive results
without modifying the internal architecture for specific tasks. The UniPCR
outperforms these different paradigms by 34%, 18.9%, and 14.9% on the
“unnecessary” F1-score, respectively. Similarly, the UniPCR also shows good
performance on the “necessary” F1 score 34.8%, 18.9% and 14.9%. Overall, our
proposed framework is able to achieve good performance predictions on request
necessary by using a unified training of text prompt tuning and code prefix
tuning.

###  5.2 Tag Recommendation Subtask

Tables 3 shows our experimental results on tag recommendation subtask.
Comparing the baselines of fine-tuning training methods based on CNN and
GraphCodeBERT as feature extractors, our UniPCR framework achieved better
performance.

Table 3: Experiment Results of Tag Recommendation Task

Model | @3 | @5 | @10
---|---|---|---
Precision | Recall | F1 | Precision | Recall | F1 | Precision | Recall | F1
TagDC [5] | 0.429 | 0.483 | 0.434 | 0.311 | 0.570 | 0.387 | 0.189 | 0.678 | 0.288
PORFIT [17] | 0.511 | 0.571 | 0.516 | 0.368 | 0.668 | 0.456 | 0.217 | 0.774 | 0.330
Post2Vec [4] | 0.550 | 0.624 | 0.558 | 0.384 | 0.707 | 0.478 | 0.223 | 0.799 | 0.339
PTM4Tag [11] | 0.488 | 0.557 | 0.496 | 0.339 | 0.628 | 0.423 | 0.199 | 0.720 | 0.303
Dual [16] | 0.564 | 0.635 | 0.572 | 0.396 | 0.724 | 0.491 | 0.227 | 0.806 | 0.345
Our UniPCR | 0.610 | 0.688 | 0.619 | 0.436 | 0.792 | 0.541 | 0.245 | 0.870 | 0.372
%Improv. | 8.2% | 8.3% | 8.3% | 10.1% | 9.3% | 10.2% | 7.9% | 8.0% | 7.8%

1) Experimental results show the effectiveness of the unified framework on the
tag recommendation subtask. In Table 3, our UniPCR framework is compared with
the state-of-the-art methods [5, 4, 11, 26] applied in the SQA community.
Experimental results on the public Code Review dataset demonstrate that our
unified prompt tuning method outperforms the best performance of baselines by
at least an 11% - 13.5% improvement in terms of these metrics.

2) Comparison of UniPCR and traditional network architecture on this task. Our
UniPCR framework based on prompt learning outperformed the sub-optimal results
in @3, @5, and @10, with improvements of 8.2%, 10.1%, and 7.8% in F1, 8.2%,
10.1%, and 7.9% in terms of Precision rate, and 8.3%, 9.3%, and 7.9% in terms
of Recall, respectively. Comparing different frameworks, the sub-optimal model
Dual add an similarity part with encoding part for tag recommendation.
Post2Vec and PTM4Tag adopt a framework that encodes text, code and tags
separately. It can be observed from the experimental results that the UniPCR
outperforms the existing two types of frameworks based on the fine-tuning
paradigm in terms of task performance. The improvements can attribute to the
UniPCR can better learn tag semantics and code representation through soft and
hard prompts used in the unified framework. Overall, our UniPCR framework has
better performance on the tag recommendation subtasks than other frameworks.

###  5.3 Ablation Study

To investigate the effectiveness of text prompt tuning and code prefix tuning,
an ablation study is conducted to evaluate the contribution of each component.
Specifically, we removed the code prefix tuning and the text prompt tuning
modules individually. Since our experiment uses continuous prompt to enhance
representation learning of code segments based on the reconstruction task,
using continuous prompt to make predictions while abandoning discrete prompt
in the experiment would go against the basis of our experiments.

#### W/O Code Prefix.

The results of code prefix ablation experiments show that this part makes a
certain contribution to both subtasks. For the code segment prefix tuning
ablation study, we modified the input sequence vector by splicing the code
segment after the text input and generating the representation of the loss
optimization code segment generated at $[MASK]$ positions. This resulted in a
new input sequence vector,

$\tilde{r^{\prime}_{prompt}}=\\{\tilde{T(\cdot)},\tilde{t},\tilde{d},h_{Graph_{c}}\\},T(\cdot)=\\{T_{t}(\cdot),T_{q}(\cdot)\\}$
(2)

Then the vector is used as input to the MLM to obtain the final predicted
label, using the same inference module as shown in Figure 2.

We investigated the impact of code prefix tuning on tag recommendation and
request necessity prediction tasks. Our results indicate that removing this
component leads to a significant degradation in performance. Specifically: 1)
In the tag recommendation task, $Precision@k$, $Recall@k$, and $F1@k$ are
dropped by 6.9%-7.4%, 2%-2.5%, and 5.6%-5.8%, respectively. 2) In the request
necessity prediction task, $Accuracy@k$, $High\,F1@k$, and $Low\,F1@k$ also
dropped by at least 6.6%, 5.6%, and 8.1%, respectively.

Our results indicate that optimizing code representations, while preserving
code segments but not using prefixes, leads to a drop in metrics. This
suggests that learning only the representation of code segments in the overall
content weakens the understanding of the differences between code segments in
different requests. By removing prefixes, the semantic angle of code segments
is reduced, making it more difficult to distinguish content representations
between similar tags. Consequently, code prefix tuning is essential for
capturing the request content and enhancing the ability to differentiate
between similar tags.

In the end, the results of the ablation analysis above suggests that code
prefixing is beneficial for understanding the representation of code segments.

#### W/O Text & Code Prompt.

The ablation experiment results show that text prompt tuning in UniPCR has a
greater contribution on the two subtasks. Based on the findings discussed so
far, the text prompt tuning is removed and we evaluate the two tasks using the
fine-tuning paradigm. In order to be consistent with our method, we use BERT
as the pre-training model. Tables 5 and 4 show the results of ablation studies
on the tag recommendation and request necessity prediction tasks.

Table 4: Ablation Experiments of Request Necessity Prediction Models | Accuracy | F1 Unnecessary | F1 Necessary | $\downarrow$
---|---|---|---|---
Our UniPCR | 0.798 | 0.817 | 0.775 | -
W/O Code Prefix | 0.745 | 0.771 | 0.712 | 5.6-8.1%
W/O Code & Text Prompt | 0.622 | 0.643 | 0.595 | 21.2-23.2%

Specifically: 1) The results on the tag recommendation subtask indicate that
the fine-tuning is not able to fully utilize all the knowledge learned in pre-
training, resulting in a significant drop in performance of at least 16% on
the overall index. 2) The request necessary subtask uses fine-tuning to
predict necessary labels by learning sentence-level representations of
requests, and the results show that using fine-tuning reduces the overall
metric by at least 14.9%.

This result highlights the importance of prompt tuning in reducing the gap
between downstream tasks and the pre-trained model. By reconstructing subtasks
to use the vector of the mask position to learn the representation of tags,
our framework strengthens the understanding of tag semantics compared to fine-
tuning. Additionally, the framework considers the association between tag
semantics and request content, further improving the representation learning.

Table 5: Ablation Experiments of Tag Recommendation Models | Precision@k
---|---
@3 | @5 | @10 | $\downarrow$
Our UniPCR |  0.610 |  0.436 | 0.245 | -
w/o Code Prefix | 0.565 | 0.404 | 0.228 | 6.9-7.3%
w/o Code & Text Prompt | 0.498 | 0.354 | 0.208 | 15.1-18.8%
Models | Recall@k
@3 | @5 | @10 | $\downarrow$
Our UniPCR |  0.688 |  0.792 | 0.870 | -
w/o Code Prefix | 0.671 | 0.772 | 0.853 | 2.0-2.5%
w/o Code & Text Prompt | 0.586 | 0.673 | 0.773 | 11.1-15.0%
Models | F1@k
@3 | @5 | @10 | $\downarrow$
Our UniPCR |  0.619 |  0.541 | 0.372 | -
w/o Code Prefix | 0.583 | 0.510 | 0.351 | 5.6-5.8%
w/o Code & Text Prompt | 0.513 | 0.446 | 0.319 | 14.2-17.6%

This finding highlights the importance of reconstruction tasks through text
prompt tuning to achieve better performance and narrow the gap between
downstream tasks and pre-trained models. The UniPCR has consistently delivered
effective performance across both subtasks (tag recommendation and request
necessary), indicating its potential to be applied to other PCR subtasks.
These results demonstrate the effectiveness of our approach and its potential
for wider applications in the field of public code review.

##  6 Related Work

Public Code Review (PCR) is a widely used software engineering practice in
which developers manually examine code written by their publics [1, 13]. Many
mature and successful Open Source Software (OSS) projects have recently
integrated public code review as a crucial quality assurance mechanism [3].

1) Request Necessity Prediction Subtask. Hong et al. [41] propose that even
though deep learning models such as T5 [7] achieve impressive performance in
the PCR domain, they require expensive computing time and resources, which
most software organizations may have limited access to [36]. To this end, a
simple and fast similarity-based retrieval model is proposed to recommend
reviews that may be written by reviewers to reduce manual costs. Chen et al.
[14] argue that promoting better CRs requires understanding requests and their
quality.

2) Tag Recommendation Subtask. Tag recommendation technology has become a
research hotspot for better serving the learning and communication of software
information stations, leading to numerous research works focusing on software
information stations [5, 4, 11, 26]. However, some existing research would
remove the code segment, assuming that it was noise that affected the final
model. For instance, Zhou et al. [26] proposed TagMulRec, a tool that
recommends post tags for developers based on index calculation similarity. Li
et al. [5] proposed TagDC, a composite tag recommendation method with deep
learning and collaborative filtering. In recent years, Xu et al. proposed
Post2Vec [4], a tag recommendation method based on deep learning that
incorporated an understanding of code segments into the model for the first
time. He et al. [11] incorporated PLMs into the software community domain and
used a feature extractor for each piece of content (title, description, and
code).

Prompt tuning can be roughly categorized into two groups: hard prompt is
constructed from natural language, and soft prompt is from continuous vectors.
Soft prompt, such as the method proposed by Liu et al. [38] called P-Tuning
v2, use trainable continuous vectors at each layer to increase the deep
capability of soft prompt. Similarly, Li et al. [37] proposed Prefix-tuning, a
lightweight alternative that optimizes a small vector of successive task-
specific parameters, achieving similar performance by learning only 0.1% of
the parameters compared to fine-tuning. Hard prompts are more interpretable
because they use natural language as a prompt template. Han et al. [39]
generated a set of rules called PTR, which decomposes a difficult task into
simpler subtasks. Sun et al. [42] proposed NSP-BERT, a method that refactors
multiple tasks into NSP form.

Existing research focuses on tasks from the reviewer’s perspective in team
code review, and needs to design an architecture for each task separately. In
the field of public code review, request quality depends more on developers.
To complete request quality assurance more simply and quickly, we propose a
framework to unify the training objectives of the request necessity prediction
subtask and tag recommendation subtask.

##  7 Conclusions and Future Work

This work proposes the UniPCR framework to jointly consider both developers
and reviewers in the PCR service. We consider the request necessity prediction
and tag recommendation subtasks in public code review process, and demonstrate
how these two tasks serve the process. Experimental results show that our
unified prompt tuning significantly outperforms state-of-the-art methods in
both subtasks. Additionally, we conducted a series of ablation studies to
investigate the effectiveness of text and code prompts. Finally, the UniPCR
framework is promising to be transferable to other public code review tasks
such as comment request necessity prediction task. In the next step, since the
UniPCR framework can also be adapted to Large Language Models (LLM), its
performance on LLMs will be a promising direction for further exploration.

## Acknowledgments

The research is supported by National Natural Science Foundation of China: No.
62276196, 52031009 and 62172311, and the Guangxi Science and Technology Major
Program (Guangxi New Energy Vehicle Laboratory Special Project: AA23062066).

## References

  * [1] Bacchelli, A., Bird, C.: Expectations, outcomes, and challenges of modern code review. In: ICSE. pp. 712–721 (2013)
  * [2] Baum, T., Schneider, K., Bacchelli, A.: On the optimal order of reading source code changes for review. In: ICSME. pp. 329–340 (2017)
  * [3] Bosu, A., Carver, J.C.: Impact of peer code review on peer impression formation: A survey. In: ESEM. pp. 133–142 (2013)
  * [4] Bowen Xu, Thong Hoang, A.S., et al.: Post2vec: Learning distributed representations of stack overflow posts. IEEE Trans. Softw. Eng. 48(9), 3423–3441 (2022)
  * [5] Can Li, Ling Xu, M.Y., et al.: Tagdc: A tag recommendation method for software information sites with a combination of deep learning and collaborative filtering. Journal of Systems and Software 170, 110783 (2020)
  * [6] Chen, X., Li, L., Zhang, R., Liang, P.: Dataset of the paper: On unified prompt tuning for request quality assurance in public code review (2024), https://archive.org/download/stackexchange/codereview.stackexchange.com.7z
  * [7] Colin Raffel, Noam Shazeer, A.R., et al.: Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. 21(140), 1–67 (2020)
  * [8] Daya Guo, Shuo Ren, S.L., et al.: Graphcodebert: Pre-training code representations with data flow. In: ICLR. pp. 1–18 (2021)
  * [9] Jacob Devlin, Ming-Wei Chang, K.L., et al.: BERT: pre-training of deep bidirectional transformers for language understanding. In: NAACL-HLT. pp. 4171–4186 (2019)
  * [10] Joe Davison, Joshua Feldman, A.M.R.: Commonsense knowledge mining from pretrained models. In: EMNLP-IJCNLP. pp. 1173–1178 (2019)
  * [11] Junda He, Bowen Xu, Z.Y., et al.: Ptm4tag: sharpening tag recommendation of stack overflow posts with pre-trained models. In: ICPC. pp. 1–11 (2022)
  * [12] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: ICLR. pp. 1–15 (2015)
  * [13] Laura MacLeod, Michaela Greiler, M.D.S., et al.: Code reviewing in the trenches: Challenges and best practices. IEEE Softw. 35(4), 34–42 (2018)
  * [14] Lawrence Chen, Rui Abreu, T.A., et al.: Leveraging test plan quality to improve code review efficacy. In: ESEC/FSE. pp. 1320–1330 (2022)
  * [15] Li, R., Liang, P., Avgeriou, P.: Code reviewer recommendation for architecture violations: An exploratory study. In: EASE. pp. 42–51 (2023)
  * [16] Lin Li, Peipei Wang, X.Z., et al.: Dual-interactive fusion for code-mixed deep representation learning in tag recommendation. Inf. Fusion 99, 101862 (2023)
  * [17] Liqiang Nie, Yongqi Li, F.F., et al.: Large-scale question tagging via joint question-topic embedding learning. ACM Trans. Inf. Syst. 38(2), 20:1–20:23 (2020)
  * [18] Liu, D., Li, L., Tao, X., et al.: Descriptive prompt paraphrasing for target-oriented multimodal sentiment classification. In: EMNLP. pp. 4174–4186 (2023)
  * [19] Lu, J., Yu, L., Li, X., Yang, L., Zuo, C.: Llama-reviewer: Advancing code review automation with large language models through parameter-efficient fine-tuning. In: ISSRE. pp. 647–658 (2023)
  * [20] Lüddecke, T., Ecker, A.S.: Image segmentation using text and image prompts. In: CVPR. pp. 7076–7086 (2022)
  * [21] Maria Tsimpoukelli, Jacob Menick, S.C., et al.: Multimodal few-shot learning with frozen language models. In: NeurIPS. pp. 200–212 (2021)
  * [22] Miltiadis Allamanis, Marc Brockschmidt, M.K.: Learning to represent programs with graphs. In: ICLR. pp. 1–17 (2018)
  * [23] Miroslaw Ochodek, Miroslaw Staron, W.M., et al.: Automated code review comment classification to improve modern code reviews. In: SWQD. pp. 23–40 (2022)
  * [24] Nasir U. Eisty, J.C.C.: Developers perception of peer code review in research software development. Empir. Softw. Eng. 27(1), Article No. 13 (2022)
  * [25] Pengfei Liu, Weizhe Yuan, J.F., et al.: Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv. 55(9), 195:1–195:35 (2023)
  * [26] Pingyi Zhou, Jin Liu, Z.Y., et al.: Scalable tag recommendation for software information sites. In: SANER. pp. 272–282 (2017)
  * [27] Prahar Pandya, S.T.: CORMS: a github and gerrit based hybrid code reviewer recommendation approach for modern code review. In: ESEC/FSE. pp. 546–557 (2022)
  * [28] Qiuyuan Chen, Dezhen Kong, L.B., et al.: Code reviewer recommendation in tencent: Practice, challenge, and direction. In: ICSE (SEIP). pp. 115–124 (2022)
  * [29] Rahman, M.M., Roy, C.K., Kula, R.G.: Predicting usefulness of code review comments using textual features and developer experience. In: MSR. pp. 215–226 (2017)
  * [30] Shan, Q., Sukhdeo, D., Huang, Q., Rogers, S., Chen, L., Paradis, E., Rigby, P.C., Nagappan, N.: Using nudges to accelerate code reviews at scale. In: ESEC/FSE. pp. 472–482 (2022)
  * [31] Tianyi Zhang, Tao Yu, T.H., et al.: Coder reviewer reranking for code generation. In: ICML. pp. 41832–41846 (2023)
  * [32] Tianyu Gao, Adam Fisch, D.C.: Making pre-trained language models better few-shot learners. In: ACL/IJCNLP. pp. 3816–3830 (2021)
  * [33] Timo Schick, H.S.: It’s not just size that matters: Small language models are also few-shot learners. In: NAACL-HLT. pp. 2339–2352 (2021)
  * [34] Tom B. Brown, Benjamin Mann, N.R., et al.: Language models are few-shot learners. In: NeurIPS. pp. 1–25 (2020)
  * [35] Vincent J. Hellendoorn, Charles Sutton, R.S., et al.: Global relational models of source code. In: ICLR. pp. 1–12 (2020)
  * [36] Wei Fu, T.M.: Easy over hard: a case study on deep learning. In: ESEC/FSE. pp. 49–60 (2017)
  * [37] Xiang Lisa Li, P.L.: Prefix-tuning: Optimizing continuous prompts for generation. In: ACL/IJCNLP. pp. 4582–4597 (2021)
  * [38] Xiao Liu, Kaixuan Ji, Y.F., et al.: P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. CoRR abs/2110.07602 (2022)
  * [39] Xu Han, Weilin Zhao, N.D., et al.: PTR: prompt tuning with rules for text classification. AI Open 3, 182–192 (2022)
  * [40] Xu Han, Tianyu Gao, Y.Y., et al.: Opennre: An open and extensible toolkit for neural relation extraction. In: EMNLP-IJCNLP. pp. 169–174 (2019)
  * [41] Yang Hong, Chakkrit Tantithamthavorn, P.T., et al.: Commentfinder: a simpler, faster, more accurate code review comments recommendation. In: ESEC/FSE. pp. 507–519 (2022)
  * [42] Yi Sun, Yu Zheng, C.H., et al.: NSP-BERT: A prompt-based few-shot learner through an original pre-training task - - next sentence prediction. In: COLING. pp. 3233–3250 (2022)
  * [43] Yinhan Liu, Myle Ott, N.G., et al.: Roberta: A robustly optimized BERT pretraining approach. CoRR abs/1907.11692 (2019)
  * [44] Zhenzhong Lan, Mingda Chen, S.G., et al.: ALBERT: A lite BERT for self-supervised learning of language representations. In: ICLR. pp. 1–17 (2020)