11institutetext: 1 Department of Physics and Astrophysics, University of
Delhi, Delhi 110007, India
11email<EMAIL_ADDRESS>
# Flavor violation at LHC in events with two opposite sign leptons and a b-jet

Nilanjana Kumar 11

###### Abstract

Hints of flavor violation at both charged current and neutral current decays
have been observed in experiments such as LHCb, Belle, and BABAR. The
anomalies in the result can be addressed in the effective field theory (EFT)
framework. The effective operators predict different beyond standard model
(BSM) signatures and the four point interaction vertices can be probed at
Large Hadron Collider (LHC). In this context, the discovery projection of two
opposite sign leptons and a b-jet signature is studied in this paper at 13 TeV
LHC.

###### Keywords:

Flavor violation, Opposite-sign leptons

##  1 Motivation

Recent experimental measurements at LHCb, Belle, and BABAR present deviations
in the SM prediction of B-meson decays and hints towards Lepton Flavor
Violation(LFV). LFV has been observed in charged current decay at tree level,
$b\rightarrow c\ell\nu$. Taking into account $R(D)$ and $R(D^{*})$
measurements by BABAR [1], Belle [2] and LHCb [3] and their correlations, the
difference between SM and the data is nearly $3.8\sigma$ [4, 5]. The anomaly
in $B_{c}\to J/\psi\ell\nu$ measurement is at the $2\sigma$ level. Whereas,
the neutral current transitions, namely $b\to s\ell^{+}\ell^{-}$ shows an
opposite effect in the measurements of $R_{K}$ and $R_{K^{*}}$. The recent
results by LHCb collaboration [6] and Belle [7], reflect that the data is more
consistent with the SM. A deviation is also seen in $B_{s}\to\phi\mu\mu$ [8],
that suggests that the discrepancies in $R_{K}$ and $R_{K^{*}}$ have been
caused by a diminution of the $b\to s\mu^{+}\mu^{-}$ channel, rather than an
enhancement in $b\to se^{+}e^{-}$.

To address these anomalies one can choose the models with leptoquarks [9] or
$Z^{\prime}$ [10]. Another way of addressing these anomalies would be to
consider an effective field theory (EFT) description with a set of Wilson
coefficients. It is possible to construct such a theory with a few unknown
parameters, if symmetry relations exist between the Wilson Coefficients. As
shown recently in [13, 11, 12], with a minimal set of new physics (NP)
operators, accompanied by a single lepton mixing angle, it is possible to
address the flavor observables. The parameters of these models can be
determined phenomenologically and if the scale of the new physics is a few
TeVs, this leads to interesting collider signatures at LHC.

##  2 Theoretical Framework

The Hamiltonian for the new physics can be expressed in terms of two operators
involving left handed 2nd and 3rd generation quark doublets $Q_{2L}$,
$Q_{3L}$, 3rd generation lepton doublet $L_{3L}$ and right handed singlet
$\tau_{R}$ as defined in Ref. [11, 12], where The terms that we are interested
in are
$(\bar{Q}_{2L}\gamma^{\mu}Q_{3L})_{3}\,(\bar{L}_{3L}\gamma^{\mu}L_{3L})_{3}$
and
$(\bar{Q}_{2L}\gamma^{\mu}Q_{3L})_{1}\,(\bar{\tau}_{R}\gamma^{\mu}\tau_{R})$,
with coefficients $3A_{1}/4$ and $A_{5}$, following the literature Ref. [11,
12]. $A_{i}$ are real unknown coefficients with dimension TeV-2. The
subscripts ‘3’ and ‘1’ represent the $SU(2)_{L}$ triplet and singlet currents
respectively. The flavor eigenstates can be expressed in terms of mass
eigenstates by a field rotation,

$\tau=cos\theta(\tau^{\prime})+sin\theta(\mu^{\prime})$

$\nu_{\tau}=cos\theta(\nu_{\tau}^{\prime})+sin\theta(\nu^{\prime}_{\mu})$

As a result of the mixing, the coupling with the second generation of leptons
are induced. The magnitude of this mixing is found to be small ($\sim 0.02)$
[11, 12]. Also, for all class of models the best fit values obtained can be
approximated as $A_{1}\sim$3.8, $A_{5}\sim$2.3.

The flavor violating process, generated by these operators are listed in Table
1. For a process (a,b) $\rightarrow$(c,d) with coefficient X in the operator,
we can write the four point coupling ($\lambda^{2}$) in the mass basis as,

$\frac{\lambda^{*}_{a,b}\lambda_{c,d}}{2M^{2}}(a,b)(c,d)=\epsilon^{abcd}\frac{4G_{F}}{\sqrt{2}}(a,b)(c,d)$,

$\lambda^{2}\sim\lambda^{*}_{a,b}\lambda_{c,d}\sim 2M^{2}X$,

where, M is the mass of the integrated-out field, and $\lambda$’s are the
dimensionless coupling. From perturbativity the bound on $\lambda$ is
$\lambda^{2}/(4\pi)^{2}\leq 1$.

Table 1: Operators and their effective couplings. For the notation, see Ref. [11]. Flavor basis | Mass basis | $\lambda^{2}$
---|---|---
$(3A_{1}/4)(s,b)(\tau\tau)$ | $(3A_{1}/4)cos^{2}\theta(s,b)(\tau^{\prime}\tau^{\prime})_{L}$ | $2M^{2}(3A_{1}/4)cos^{2}\theta$
- | $(3A_{1}/4)sin^{2}\theta(s,b)(\mu^{\prime}\mu^{\prime})_{L}$ | $2M^{2}(3A_{1}/4)sin^{2}\theta$
- | $(3A_{1}/4)sin2\theta(s,b)(\mu^{\prime}\tau^{\prime})_{L}$ | $2M^{2}(3A_{1}/4)sin2\theta$
$A_{5}(s,b)(\tau\tau)$ | $A_{5}cos^{2}\theta(s,b)(\tau^{\prime}\tau^{\prime})_{R}$ | $2M^{2}A_{5}cos^{2}\theta$
- | $A_{5}sin^{2}\theta(s,b)(\mu^{\prime}\mu^{\prime})_{R}$ | $2M^{2}A_{5}sin^{2}\theta$
- | $A_{5}sin2\theta(s,b)(\mu^{\prime}\tau^{\prime})_{R}$ | $2M^{2}A_{5}sin2\theta$

##  3 Results

As can be seen from Table 1, there are three possible signatures:

  * •

($\mu^{\pm}\mu^{\mp}$) + $b$-jet,

  * •

($\mu^{\pm}\tau^{\mp}$) + $b$-jet and

  * •

($\tau^{\pm}\tau^{\mp}$) + $b$-jet.

These processes can be generated at 13 TeV LHC via $g$-$g$ and $g$-$s$ fusion
in $p$-$p$ collision, with the major contribution coming from $g$-$s$ fusion.
Now as the coupling ($\lambda^{2}$) is a function of $A_{1}$ and $A_{5}$, the
cross section also varies with these parameter. We kept the value of $A_{1}$
fixed at the best fit 3.8 and varied $A_{5}$. The range of values of the
parameters are chosen such that the 95% C.L. upper bound of
$Br(B_{s}\to\tau^{\pm}\mu^{\mp})<4.2\times 10^{-5}$ [14] is satisfied.

The cross section of ($\mu^{\pm}\mu^{\mp}$) and a $b$-jet is very small,
because their coupling is suppressed by ($\sin^{2}\theta$) and hence we
neglect it in this study. The cross section of ($\mu^{\pm}\tau^{\mp}$) and a
$b$-jet will be suppressed by $\sin 2\theta$ and hence will be comparatively
larger than the previous one, shown in Fig 1 (left). The cross section of
($\tau^{\pm}\tau^{\mp}$) and a $b$-jet is relatively very high as shown in Fig
1(right) with red line. The tau can also decay leptonically to a muon with
branching ratio 0.174, enabling the final states with ($\mu^{\pm}\tau^{\mp}$)
and a $b$-jet and ($\mu^{\pm}\mu^{\mp}$) and a $b$-jet, as shown in Fig 1
(right) by green and blue lines respectively.

Figure 1: (L) The total production of one $\mu$ and one $\tau$ in association
with one $b$-jet in $g$-$g$ fusion and $g$\- $s$ fusion. (R) The total
production of two $\tau$,s in association with one $b$-jet in $g$-$g$ fusion
and $g$\- $s$ fusion (red), also one $\mu$ and one $\tau$ \+ $b$-jet (green)
and two $\mu$’s+$b$-jet (blue) when tau (one or both respectively) decays
leptonically.

The major SM backgrounds for these channels are $t\overline{t}$, single top
($Wt$), diboson ($W^{+}W^{-}$, $WZ$ and $ZZ$), $W$+jets, $WW$+jets,
$Z/\gamma$+jets. We found that the background for the signal with opposite
sign same flavor states ($\mu^{\pm}\mu^{\mp}$)or ($\tau^{\pm}\tau^{\mp}$) and
a $b$-jet are larger than the opposite sign opposite flavor state
($\mu^{\pm}\tau^{\mp}$). Also, the signal ($\tau^{\pm}\tau^{\mp}$)and a
$b$-jet will suffer from tau tagging efficiency at LHC as the both the tau
decay hadronically. Hence we study two channels,
($\mu^{\pm}\mu^{\mp}$)+$b$-jet and ($\mu^{\pm}\tau^{\mp}$)+$b$-jet. In Fig 2
we have shown the discovery projection of these two channels as a function of
the integrated luminosity at LHC. We have followed the search strategy as
mentioned in [15, 16]. Fig 2 shows that the ($\mu^{\pm}\mu^{\mp}$)+$b$-jet
channel requires much larger luminosity than ($\mu^{\pm}\tau^{\mp}$)+$b$-jet
channel for 5$\sigma$ discovery significance.

Figure 2: The 5$\sigma$ discovery projection at 13 TeV LHC in
($\mu^{\pm}\mu^{\mp}$) + $b$-jet and ($\mu^{\pm}\tau^{\mp}$) + $b$-jet channel
as a function of the integrated luminosity and model parameter $A_{5}$ with
the assumption of 25% uncertainty in the background events.

##  4 Conclusion

Recently observed anomalies in the decays of B-mesons hints towards new
physics interaction which involves a b-quark, a s-quark and a pair of opposite
sign leptons. The four point interactions among them can be probed at LHC p-p
collision via the direct production of b-quark and two opposite sign leptons.
The opposite sign lepton pair have either same or opposite flavor. In this
study, 5$\sigma$ discovery potential of ($\mu^{\pm}\mu^{\mp}$) + $b$-jet and
($\mu^{\pm}\tau^{\mp}$) + $b$-jet channels are discussed as a function of the
effective field theory model parameters. Overall, these channels have a very
good detection prospect even with the currently collected data at LHC and a
limit on the model parameter space can be set with the 13 TeV LHC data.

## References

  * [1] BaBar collaboration, J. P. Lees et al., _Phys. Rev._ D88 (2013) 072012,
  * [2] Belle collaboration, M. Huschle et al., _Phys.Rev._ D92 (2015) 072014,
  * [3] LHCb collaboration, R. Aaij et al., _Phys. Rev. Lett._ 115 (2015) 111803,
  * [4] Heavy Flavor Averaging Group (HFAG) collaboration, Y. Amhis et al., arXiv: 1412.7515 [hep-ph].
  * [5] “https://hflav-eos.web.cern.ch/hflav-eos/semi/summer18/RDRDs.html.”.
  * [6] LHCb Collaboration collaboration, Tech. Rep. CERN-EP-2019-043. LHCB-PAPER-2019-009, CERN, Geneva, Mar, 2019.
  * [7] Belle collaboration, A. Abdesselam et al.,
  * [8] LHCb collaboration, R. Aaij et al., _JHEP_ 09 (2015) 179,
  * [9] D. Bečirević, S. Fajfer, N. Košnik and O. Sumensari, _Phys. Rev._ D94 (2016) 115021,
  * [10] P. Langacker, _Rev. Mod. Phys._ 81 (2009) 1199–1228
  * [11] D. Choudhury, A. Kundu, R. Mandal and R. Sinha, _Phys. Rev. Lett._ 119 (2017) 151801,
  * [12] D. Choudhury, A. Kundu, R. Mandal and R. Sinha, _Nucl. Phys._ B933 (2018) 433–453,
  * [13] S. Bhattacharya, A. Biswas, Z. Calcuttawala and S. K. Patra, arXiv:1902.02796 [hep-ph].
  * [14] LHCb collaboration, R. Aaij et al., _Search for the lepton-flavour-violating decays $B^{0}_{s}\to\tau^{\pm}\mu^{\mp}$ and $B^{0}\to\tau^{\pm}\mu^{\mp}$_, arxXiv:1905.0661 [hep-ph].
  * [15] D. Choudhury, N. Kumar and A. Kundu, arXiv:1905.07982 [hep-ph].
  * [16] Y. Afik, J. Cohen, E. Gozani, E. Kajomovitz and Y. Rozen, _Establishing a Search for $b\rightarrow s\ell^{+}\ell^{-}$ Anomalies at the LHC_, _JHEP_ 08 (2018) 056.


# Are Prompt-based Models Clueless?

Pride Kavumba1,2
Ryo Takahashi3
1Tohoku University 2RIKEN AIP 3LegalForce Research
<EMAIL_ADDRESS>
{ryo.takahashi<EMAIL_ADDRESS>
Yusuke Oda3,1

###### Abstract

Finetuning large pre-trained language models with a task-specific head has
advanced the state-of-the-art on many natural language understanding
benchmarks. However, models with a task-specific head require a lot of
training data, making them susceptible to learning and exploiting dataset-
specific superficial cues that do not generalize to other datasets. Prompting
has reduced the data requirement by reusing the language model head and
formatting the task input to match the pre-training objective. Therefore, it
is expected that few-shot prompt-based models do not exploit superficial cues.
This paper presents an empirical examination of whether few-shot prompt-based
models also exploit superficial cues. Analyzing few-shot prompt-based models
on MNLI, SNLI, HANS, and COPA has revealed that prompt-based models also
exploit superficial cues. While the models perform well on instances with
superficial cues, they often underperform or only marginally outperform random
accuracy on instances without superficial cues.

##  1 Introduction

Finetuning large pre-trained language models with a task-specific head has
achieved remarkable performance on many natural language benchmarks Wang et
al. (2018, 2019). However, the task-specific head introduces a lot of random
task-specific parameters that require enormous finetuning data to attain
optimal performance. The exposure to enormous data increases the potential for
models to learn and exploit dataset-specific superficial cues that do not
generalize to other datasets without superficial cues Gururangan et al.
(2018); Poliak et al. (2018); Sugawara et al. (2018); Niven and Kao (2019);
Schuster et al. (2019); Kavumba et al. (2019). For example, Niven and Kao
(2019) found that task-specific head models exploit the presence of “not” in
the input of argument reasoning comprehension dataset Habernal et al. (2018)
to achieve state-of-the-art accuracy, but drop to random accuracy when the
superficial cue is neutralized. On the other hand, prompting reuses the pre-
training language model head, introducing no random task-specific parameters.
Thus, prompt-based models can achieve remarkable performance with only a few
training examples Brown et al. (2020); Schick and Schütze (2021a, b); Gao et
al. (2021); Le Scao and Rush (2021). Hence, few-shot prompting lowers the
potential for models to learn and exploit dataset-specific superficial cues.

Figure 1: (A) shows a prompt-based model receiving natural language inference
(NLI) prompts generated through a template. (B) Our analysis reveals that
prompt-based models exploit superficial cues (highlighted lexical overlap)
that are predictive of entailment relation between premise and hypothesis. (C)
The model fails to generalize to instances were the superficial cues no longer
predict entailment relation.

This work empirically investigates whether few-shot prompt-based models
exploit superficial cues. Specifically, we ask: _Do few-shot prompt-based
models exploit superficial cues?_ To answer this question, we examine
prompted-based models on two fundamental tasks of natural language
understanding: natural language inference (NLI) and commonsense reasoning;
comprehending natural language inference and commonsense are essential to make
progress in natural language understanding Bowman et al. (2015); Williams et
al. (2018); Roemmele et al. (2011). We analyze the performance of prompt-based
models trained on the Stanford Natural Language Inference dataset (Bowman et
al., 2015, SNLI), the Multi-Genre Natural Language Inference data (Williams et
al., 2018, MNLI), and the Choice of Plausible Alternatives dataset (Roemmele
et al., 2011, COPA) on instances with and without superficial cues.

To facilitate the analysis, we define two types of superficial cues that
abstract away from the underlying tasks: _context_ and _contextless_
superficial cues, where the definition of the _context_ depends on the task.
For example, in natural language inference tasks, we define the premise as the
_context_ , while in multiple-choice tasks, we define the question as the
_context_. Context superficial cues such as lexical overlap (Figure 1) coexist
in the context (premise) and the hypothesis. In contrast, contextless
superficial cues exist only in the hypothesis (in NLI) or in answer choices
(in multiple-choice tasks). A dataset can contain either one or both types of
superficial cues. Therefore, both types must be investigated to sufficiently
answer whether a model can exploit superficial cues.

As a prerequisite, we reanalyze superficial cues in MNLI, SNLI, and COPA
datasets to created evaluation sets that have and do not have superficial
cues. We find that these datasets contain more superficial cues than
previously known. Specifically, we find that 90.1% of MNLI matched instances
contain _contextless superficial cues_ in the hypothesis, while 71.9% SNLI
contains _contextless superficial cues_. Additionally, we find that COPA
contains not only contextless superficial cues Kavumba et al. (2019), but
78.0% of the instances also contain _context superficial cues_.

Finally, we examine whether few-shot prompt-based models also rely on
superficial cues to achieve remarkable performance on MNLI, SNLI, COPA, and
the HANS dataset McCoy et al. (2019). COPA experiments reveal that prompt-
based models do not rely on contextless superficial cues for typical few-shot
training sizes. However, the other empirical results show that prompt-based
models heavily rely on superficial cues—failing to generalize to data without
superficial cues (Figure 1).

In summary, our contributions are:

  1. 1.

We propose to divide superficial cues into _context_ and _contextless_
superficial cues, which abstracts away from the underlying tasks (§ 3).

  2. 2.

We established that the datasets of MNLI ( § 3.2), SNLI (§ 3.2) and COPA (§
3.1) contain more superficial cues than previously known. We release analyzed
datasets at https://github.com/legalforce-research/prompt-models-clueless.

  3. 3.

We present the first investigation of the exploitation of superficial cues by
prompt-based models, finding that prompt-based models also exploit superficial
cues (§ 5).

##  2 Background

We will begin with a review of the necessary concepts required to understand
the rest of the paper.

###  2.1 Prompting

Prompt-based finetuning has been demonstrated to be effective in few-shot
setup Brown et al. (2020); Schick and Schütze (2021a, b); Gao et al. (2021);
Le Scao and Rush (2021). By reusing the pretraining language model head,
prompting introduces no or only a few randomly initialized parameters.
Prompting reformulates any task to match the pretraining objective. For
example, consider the task of classifying the sentiment polarity of movie
reviews using a masked language model such as BERT Devlin et al. (2019). A
review such as “ _I liked the movie_ ” is converted to “ _I liked the movie.
It was [MASK]_ ”. The model, then, fills [MASK] with words such as _{ good,
nice, bad, terrible}_, which are mapped to the task labels—positive or
negative—through a verbalizer Schick and Schütze (2021a, b). In contrast, a
task-specific classification head model directly predicts positive or negative
sentiment. For an in-depth review, we will refer the interested reader to a
survey by Liu et al. (2021).

###  2.2 Superficial Cues

Superficial cues can be described as linguistic or non-linguistic
characteristics of instances that have nothing to do with the task itself but
are tied to a specific task label. These characteristics include lexical
overlap McCoy et al. (2019), distinct words frequently appearing in the
correct choices Niven and Kao (2019); Kavumba et al. (2019), and distinctive
style of the correct choices Trichelair et al. (2019). As a concrete example,
consider a sentiment classification dataset whose negative sentiment instances
contain “not”; for example, “I did not like the movie”. Here, “not” is a
superficial cue because it is predictive of the correct label.

###  2.3 Training Datasets

#### MNLI

The Multi-Genre Natural Language Inference (Williams et al., 2018, MNLI)
dataset is an important dataset of natural language inference which is also
part of the SuperGLUE benchmark Wang et al. (2019). Given a premise and a
hypothesis, the task asks to pick one label from among three, {contradiction,
neutral, entailment}. The test set of MNLI is divided into matched (in-domain
instances) and mismatched (out-of-domain instances) subsets based on whether
the domain of each test instance matches the training set domain.

#### SNLI

The Stanford Natural Language Inference (Bowman et al., 2015, SNLI) is a
popular natural language inference dataset with the same format as MNLI.

#### COPA

The Choice of Plausible Alternatives (Roemmele et al., 2011, COPA) dataset is
a popular multiple-choice commonsense dataset, which is also a part of the
SuperGLUE benchmark. Given a premise and a question, the task is to select the
most plausible cause or effect from the set of two candidates.

##  3 Superficial Cues in NLI and COPA

We investigate prompted-based models on two fundamental tasks of natural
language understanding: natural language inference (NLI) and commonsense
reasoning. As a prerequisite, we begin by creating test sets with and without
superficial cues that we will subsequently use to investigate whether prompt-
based models exploit superficial cues. We analyze and split test sets of
English language datasets into subsets with and without superficial cues in
the following subsections.

To facilitate easy analysis, we divide superficial cues into two categories:
_context_ superficial cues and _contextless_ superficial cues, where the
definition of _context_ is task dependent. For example, in natural language
inference tasks such as COPA, the _context_ can be defined as the premise,
while in multiple-choice tasks, the _context_ can be defined as the question.
Context superficial cues, such as lexical overlap found by McCoy et al. (2019)
can only be exploited when the context is available in the input. On the other
hand, contextless superficial cues, such as the occurrence of “not” in the
correct answer choices found by Niven and Kao (2019), are those that are
exploitable even in the absence of the context required to perform a task.

###  3.1 Context Superficial Cues

#### Natural Language Inference (NLI):

NLI has a good dataset designed to test for contextless superficial cues.
Specifically, the HANS dataset tests the models’ ability to exploit three
types of context superficial cues in NLI: _lexical overlap_ , _subsequence_ ,
and _constituent_ McCoy et al. (2019). Therefore, we evaluate prompt-based
models on the HANS dataset instead of splitting tests of MNLI and SNLI into
instances with and without superficial cues.

#### COPA

Eyeballing all instances to find common patterns that identify the correct
answer choice, but are unrelated to the task, can be challenging and error-
prone. To circumvent the need for manual examination, we propose to solve the
task in a setup that encourages the model to solve the task using superficial
cues. This setup is similar to providing only partial input Gururangan et al.
(2018); Poliak et al. (2018). Specifically, we randomly shuffle the words in
the answer choices such that identifying the correct choice is mainly based on
superficial cues in the question and the answer choice. For example, given the
original instance;

Premise: The host cancelled the party. What was the CAUSE of this?

a) She worried she would catch the flu.

b) She was certain she had the flu. (correct)

The new answer choices for the new instance becomes:

a) She would she catch the worried flu.

b) She had was she the certain flu. (correct)

In this setting, we find that RoBERTa achieves an average accuracy of 78%,
indicating the existence of context superficial cues. Following this result,
we split the test set into a subset with superficial cues containing instances
solved by the majority of models, and a subset without superficial cues,
containing all the remaining instances.

###  3.2 Contextless Superficial Cues

#### Natural Language Inference

Dataset | accuracy
---|---
Random | 33.3
MNLI | 90.1 $\pm$0.1
MNLI-mm | 90.0 $\pm$0.2
SNLI | 71.9 $\pm$0.1
Table 1: Average accuracy on matched MNLI (MNLI) and mismatched MNLI (MNLI-
mm), and SNLI for a head RoBERTa model trained on the hypothesis. MNLI and
MNLI-mm results are for a model trained on MNLI and SNLI results are for a
model trained on SNLI.

To investigate contextless superficial cues in NLI, we train RoBERTa Liu et
al. (2019) with a classification head on only the hypothesis of MNLI and SNLI.
This analysis is similar to the one done by Gururangan et al. (2018) using
fastText Joulin et al. (2017). If the model can not find contextless
superficial cues in the hypothesis, it is expected to achieve random
performance (33.3%). But, RoBERTa trained on MNLI achieve an average
performance of 90.1% and 90.0% on matched and mismatched instances,
respectively (Table 1), which is worse than previously known (53.9% matched
and 52.3% mismatched Gururangan et al. (2018)). On the test set of SNLI,
RoBERTa trained on SNLI achieves an average accuracy of 71.9% (Table 1), which
is 4.9 percentage points higher than previously known Gururangan et al.
(2018). Following this result, we split the testing sets of MNLI and SNLI such
that each test set has two subsets: instances with contextless superficial
cues, containing all instances that the majority of models solved correctly,
and instances without contextless superficial cues contain all the remaining
instances.

(a) w/ & w/o Context Cues

(b) w/ & w/o Contextless Cues

Figure 2: Average accuracy on instances with superficial cues (w/ cues) and
without superficial cues (w/o cues) for prompt-based RoBERTa models trained on
MNLI and SNLI. (a) shows results on the HANS dataset on three kinds of context
superficial cues: lexical overlap (lexical), subsequence (subs), and
constituent (consti). (b) shows results on instances with and without
contextless superficial cues of matched MNLI (MNLI) and mismatched MNLI (MNLI-
mm) for a model trained on MNLI; and evaluations results on SNLI for a model
trained on SNLI.

#### COPA

The test set of COPA has already been split into two subsets that have
instances with contextless superficial cues and instances that do not have
contextless superficial cues Kavumba et al. (2019). The subsets were
constructed based on the performance of RoBERTa trained on answers only.
Therefore we do not reanalyze COPA; instead, we will use the same publicly
available subsets in our evaluation.

##  4 Experimental Setup

The goal of our experimental setup is to answer the following research
question: Do prompt-based models exploit superficial cues? We decompose this
question into two sub-questions: 1) Do prompt-based models exploit context
superficial cues? 2) Do prompt-based models exploit contextless superficial
cues?

#### Training Details

For all our experiments, we use RoBERTa-large (355M parameters) because it is
the widely used model in prompt-based finetuning Schick and Schütze (2021a);
Gao et al. (2021); Le Scao and Rush (2021). We build on the source code by Gao
et al. (2021) 111https://github.com/princeton-nlp/LM-BFF and Le Scao and Rush
(2021) 222https://github.com/TevenLeScao/pet, and we load the pre-trained
weights from HuggingFace Wolf et al. (2019). We use the best-reported
hyperparameters and templates (Appendix B) from Gao et al. (2021) on NLI. All
NLI models are trained with 16 instances per label. We use the same partitions
used by Gao et al. (2021). On COPA we use the best hyperparameters and
templates (Appendix B) from Schick and Schütze (2021b); Le Scao and Rush
(2021). We ran all experiments three times with different random seeds and
report the average and standard deviation.

#### Evaluation

The goal of our experimental evaluation is to answer the following question:
Do prompt-based models exploit superficial cues? We answer this question by
investigating whether the model exploits or relies on either context or
contextless superficial cues. We train and evaluate our models on English
datasets.

#### Context Superficial Cues

To investigate whether models exploit context superficial cues, we train
prompt-based models on MNLI, SNLI, and COPA. We evaluate NLI models on the
HANS dataset that tests the models’ ability to exploit three types of context
superficial cues in NLI: _lexical overlap_ , _subsequence_ , and _constituent_
McCoy et al. (2019). We report the average accuracy and standard deviation on
the two subsets of the dataset: a subset where the superficial is informative
(Entailment) and a subset where the superficial cues are uninformative (Non-
entailment). A model that does not rely on context superficial cues is
expected to perform comparably on both subsets.

#### Contextless Superficial Cues

To investigate whether prompt-based models exploit contextless superficial
cues, we train a prompt-based model on MNLI, SNLI, and COPA; and evaluate them
on the corresponding test set of each dataset. Each test set consists of two
subsets obtained and described in sections 3: a subset of instances with
contextless superficial cues and a subset without contextless superficial
cues. A model that does not rely on contextless superficial cues is expected
to perform comparably on both subsets.

##  5 Results

###  5.1 Exploiting Context Superficial Cues

#### Natural Language Inference (NLI)

Figure 2(a) shows the results on the HANS dataset of the prompt-based model
trained on MNLI (left) and SNLI (right), respectively. The results show that
prompt-based RoBERTa trained on MNLI performs considerably well on instances
with superficial cues, an overall average of 98.7%. However, the model only
achieves an overall average accuracy of 7.4% on instances without context
superficial cues, failing to reach random accuracy of 50%. This indicates that
the prompt-based models trained on MNLI exploit superficial cues.

Similarly, figure 2(a) (right) shows that while RoBERTa performs considerably
well on instances with superficial cues (overall average 91.3%), it fails to
achieve the same performance on instances without superficial cues (overall
average of 31.7%). 333The high variance is similar to that reported by
previous work studying head models Bras et al. (2020). This result also leads
to the same conclusion that the model exploits context superficial cues.

# Examples | _w/ cues_ | _w/o cues_ | Overall |
---|---|---|---|---
8 | 83.0$\pm$0.5 | 55.7$\pm$1.4 | 78.1$\pm$0.3
16 | 81.6$\pm$1.9 | 57.9$\pm$1.0 | 77.3$\pm$1.4
32 | 82.4$\pm$2.0 | 53.5$\pm$0.5 | 77.1$\pm$1.5
64 | 84.4$\pm$1.4 | 53.8$\pm$4.1 | 78.9$\pm$1.2
96 | 87.0$\pm$1.7 | 57.9$\pm$4.1 | 81.7$\pm$0.8
100 | 87.9$\pm$1.7 | 54.2$\pm$1.9 | 81.7$\pm$1.2
Table 2: Average accuracy of few-shot prompt-based RoBERTa models on COPA
instances with context superficial cues between the context and the answer
choices; and instances without exploitable context superficial cues. Column “#
Examples” shows the number of examples used for finetuning.

#### COPA

Table 2 shows the results of the prompt-based RoBERTa trained on COPA and
evaluated on the two subsets of COPA: with and without superficial cues. The
results show RoBERTa performs well on instances with superficial cues but
barely exceeds random accuracy (50%) on instances without superficial cues.
This, too, indicates that the model exploit contextless superficial cues.

###  5.2 Exploiting Contextless Superficial Cues

#### NLI

Figure 2(b) shows the results of prompt-based RoBERTa train on MNLI and SNLI
and evaluated on the corresponding test set. The results show that the prompt-
based model trained on MNLI performs considerably better on instances with
superficial cues on both matched (69.5%) and mismatched (72.0%) instances. On
instances without superficial cues we observe a gap of 30.9% and 30.5% on
matched and mismatched instances, respectively. The high difference in
performance indicates that the models exploit contextless superficial cues.
For the model trained on SNLI and evaluated on SNLI subsets, we observe a gap
of 15.1% between performance on instances with and without superficial cues
(82.2% vs 67.1%). This also indicates that the model does exploit contextless
superficial cues.

#### COPA

# Examples | _w/ cues_ | _w/o cues_ | Overall
---|---|---|---
8 | 79.1$\pm$0.9 | 77.4$\pm$0.0 | 78.1$\pm$0.3
16 | 77.0$\pm$1.3 | 77.4$\pm$1.6 | 77.3$\pm$1.4
32 | 77.7$\pm$3.1 | 76.8$\pm$0.9 | 77.1$\pm$1.5
64 | 80.2$\pm$0.5 | 78.1$\pm$1.6 | 78.9$\pm$1.2
96 | 83.7$\pm$1.1 | 80.4$\pm$0.7 | 81.7$\pm$0.8
100 | 84.6$\pm$1.1 | 80.0$\pm$1.6 | 81.7$\pm$1.2
Table 3: Average accuracy of few-shot prompt-based RoBERTa models on COPA
instances with contextless superficial cues (w/ cues) in the answer choices
and instances without contextless superficial cues (w/o cues). Column “#
Examples” shows the number of examples used for finetuning.

Table 3 shows of the prompt-based RoBERTa trained on COPA and evaluated on the
COPA subsets with and without superficial cues. The results show that the
prompt-based model does not exploit superficial cues at a small enough
training set (less or equal to 32 instances). However, increasing the size
further increases the gap in performance between instances with and without
contextless superficial cues. It is encouraging to note that the model does
not exploit contextless superficial cues at sizes commonly used in few-shot
settings.

##  6 Discussion

###  6.1 Predictions Errors

The results on natural language inference instances without superficial cues
are worse than random performance. One wonders whether it is because the
instances are hard. We look at some instances that the model fails to solve
correctly. We show some of the instances in Table 4. The instances are simple
enough for anyone that understands English. One question that immediately
arises is; are prompt-based models sensitive to the meaning of the question?

Premise | Hypothesis | Label | Prediction
---|---|---|---
The president advised the doctor | The doctor advised the president | N | E
The student saw the managers | The managers saw the student | N | E
The presidents encouraged the banker | The bank encouraged the president | N | E
The actors avoided the bankers | The bankers avoided the bankers | N | E
The managers saw the secretaries | The secretary saw the managers | N | E
The lawyers helped the judge | The judge helped the lawyers | N | E
The banker thanked the tourist | The tourist thanked the banker | N | E
Table 4: Some examples of instances without superficial cues from the HANS
dataset that a prompt-based model trained on MNLI wrongly classify as
entailment (E) because of superficial cues: high lexical overlap.

###  6.2 Attention Visualization

To investigate whether prompt-based models are sensitive to meaning, we
compare the attention weight across all twenty-four layers of RoBERTa-large
for closely related instances that differ only in meaning and hence the
labels. While there have been many questions that have been raised over the
reliability of singly using attention weights for explanation Wiegreffe and
Pinter (2019); Vig and Belinkov (2019), here we use attention weights coupled
with other results to gain more insights into the models’ inner working. We
are interested in knowing whether there is a huge change in attention weights
responding to the change in meaning. For example, we take an instance with
superficial cues, which lead to the correct prediction of the Entailment
label:

Premise: The president was advised by the doctor.

Hypothesis: The doctor advised the president.

label: Entailment

And an instance without superficial cues:

Premise: The president advised the doctor.

Hypothesis: The doctor advised the president.

label: Non-Entailment

While the instances are completely different in meaning, the model predicts
Entailment in both cases because of the superficial cue of high overlap. When
we compare the attention maps for all the layers, we notice that there is
barely any change in response to the change in the meaning of the sentences.
Because of space limitation, we show attention maps only for the first two
layers and the last layer (Figure 3). The attention maps for all the 24 layers
are shown in Appendix E. The visualizations highlight the inability of the
model to respond to the change in meaning. We investigate this further in the
next section.

Figure 3: A comparison of prompt-based model attention maps on two natural
language inference instances that have some words in common but have entirely
different meanings. The first row attention maps are for an instance with
superficial cues, and the second row attention maps are for an instance
without superficial cues (Appendix E shows all layers).

###  6.3 Sensitivity to Word Order in NLI

The visual attention analysis revealed that the model does not respond well to
change in meaning. To investigate this at scale, we evaluate a model trained
on input with correct word order and input with randomly shuffled word order.
We will refer to input with correct word order as _meaningful_ input and input
with shuffled word order as _meaningless_ input. Specifically, given an
original test instance, we make it meaningless by randomly shuffling all the
words in the instance while maintaining the English end of sentence
punctuation mark if it exists in the original instance. We do this so we can
preserve the same number of English sentences as the original instance while
making them meaningless.

For example, given the original NLI instance:

Premise: The president was advised by the doctor.

Hypothesis: The doctor advised the president.

label: Entailment

The new instance becomes:

Premise: The doctor by the president advised was.

Hypothesis: The doctor the president advised.

label: Entailment

If the model is sensitive to meaning, we expect the performance on this
meaningless input to drop to random performance because the model was trained
on meaningful input.

(a) MNLI

(b) SNLI

Figure 4: The average accuracy of prompt-based RoBERTa trained on MNLI (a) and
SNLI (b) instances with the correct word order (“Meaningful”) when evaluated
on HANS dataset when the input has the correct word order (Yes in legend) and
when the word order is shuffled (No in legend).

Figure 4(a) and Figure 4(b) shows the results of prompt-based RoBERTa trained
on MNLI and SNLI, respectively. The figures show the results of a prompt-based
model trained on meaning-containing instances evaluated on the test set of
instances whose meaning is preserved (Yes) and when the instances are made
meaningless (No). The results show that when meaning is removed from the
instances, the model barely changes its predictions, indicating that the model
hardly relies on the meaning of the instances.

#### Future Work

At this point, some questions still remain unanswered: (1) What are the
specific superficial cues that the models exploit? This still remains a hard
interpretability question. (2) Are there any prompts that discourage models
from exploiting superficial cues? (3) Can incorporating task demonstrations
without superficial cues mitigate against the reliance on superficial cues?

##  7 Related Works

#### Few-shot Prompting

Language model prompting was popularized by recent work of GPT-3 Brown et al.
(2020). Brown et al. (2020) showed that by using prompts and some task
demonstrations, GPT-3 could perform a number of tasks in the few shot setup.
Following this work, Schick and Schütze (2021a) showed that even much smaller
language models such as RoBERTa-large could perform well when finetuned with
prompts. The subsequent work Schick and Schütze (2021b) demonstrated that a
smaller model could achieve similar performance to GPT-3 in a few shot setup
once finetuned with prompts. Many works proposed better ways of generating
prompts and answer keys Gao et al. (2021); Le Scao and Rush (2021). Our work
develops on these works to gain more insights into model predictions.
Specifically, we undertake the first investigation of whether prompt-based
models exploit superficial cues.

#### Superficial Cues in Datasets

Superficial cues have been analyzed across several natural language
understanding datasets. Gururangan et al. (2018) analyzed contextless
superficial cues—hypothesis only superficial cues—in the MNLI dataset and SNLI
dataset using fastText Joulin et al. (2017), finding that a little over half
of MNLI contain superficial cues and 67% of SNLI contain superficial cues. We
argue that these figures could be outdated. Hence, we reanalyze contextless
superficial cues in MNLI and SNLI. McCoy et al. (2019) analyzed context
superficial cues in MNLI, finding that lexical overlap is one superficial cue
that can be exploited to correctly predict entailment labels. Following their
analysis, they released the HANS dataset that has an equal number of instances
with superficial cues and those without superficial cues. In this work, we use
the HANS dataset to evaluate the models’ ability to exploit superficial cues.
Kavumba et al. (2019) analyzed contextless superficial cues in COPA using
RoBERTa and productivity measures introduced by Niven and Kao (2019). However,
they did not analyze context superficial cues. In this work, we analyze
context superficial cues in COPA, and we use the contextless superficial cues
from Kavumba et al. (2019).

#### Exploiting Superficial Cues in Head Language Models

Head language models that use a task-specific head for a downstream task have
been analyzed on their ability to exploit superficial cues in datasets. McCoy
et al. (2019) found that head BERT Devlin et al. (2019) exploits superficial
cues on MNLI dataset. Similarly, Niven and Kao (2019) found that head BERT
exploits superficial cues on the argument reasoning comprehension task and
Kavumba et al. (2019) analyzed BERT and RoBERTa’s ability to exploit
superficial cues on the COPA dataset. While head models have been analyzed
already, prompt-based models have not been analyzed yet. In this paper, we
investigated whether prompt-based models also exploit superficial cues.

##  8 Conclusions

We presented the first analysis of whether prompt-based models exploit
superficial cues. We found that prompt-based models exploit superficial cues
and fail to generalize well to instances without superficial cues on MNLI,
SNLI, COPA, and HANS. We, further, proposed to divide superficial cues into
two: context and contextless superficial cues. Our analysis of MNLI, SNLI, and
COPA has revealed more superficial cues than was previously known.

## References

  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 632–642, Lisbon, Portugal. Association for Computational Linguistics.
  * Bras et al. (2020) Ronan Le Bras, Swabha Swayamdipta, Chandra Bhagavatula, Rowan Zellers, Matthew Peters, Ashish Sabharwal, and Yejin Choi. 2020.  Adversarial filters of dataset biases.  In _Proceedings of the 37th International Conference on Machine Learning_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 1078–1088. PMLR.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Gao et al. (2021) Tianyu Gao, Adam Fisch, and Danqi Chen. 2021.  Making pre-trained language models better few-shot learners.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 3816–3830, Online. Association for Computational Linguistics.
  * Gururangan et al. (2018) Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith. 2018.  Annotation artifacts in natural language inference data.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 107–112, New Orleans, Louisiana. Association for Computational Linguistics.
  * Habernal et al. (2018) Ivan Habernal, Henning Wachsmuth, Iryna Gurevych, and Benno Stein. 2018.  SemEval-2018 task 12: The argument reasoning comprehension task.  In _Proceedings of The 12th International Workshop on Semantic Evaluation_ , pages 763–772, New Orleans, Louisiana. Association for Computational Linguistics.
  * Joulin et al. (2017) Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. 2017.  Bag of tricks for efficient text classification.  In _Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Short Papers_ , pages 427–431. Association for Computational Linguistics.
  * Kavumba et al. (2019) Pride Kavumba, Naoya Inoue, Benjamin Heinzerling, Keshav Singh, Paul Reisert, and Kentaro Inui. 2019.  When choosing plausible alternatives, clever hans can be clever.  In _Proceedings of the First Workshop on Commonsense Inference in Natural Language Processing_ , pages 33–42, Hong Kong, China. Association for Computational Linguistics.
  * Le Scao and Rush (2021) Teven Le Scao and Alexander Rush. 2021.  How many data points is a prompt worth? In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2627–2636, Online. Association for Computational Linguistics.
  * Liu et al. (2021) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * McCoy et al. (2019) Tom McCoy, Ellie Pavlick, and Tal Linzen. 2019.  Right for the wrong reasons: Diagnosing syntactic heuristics in natural language inference.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3428–3448, Florence, Italy. Association for Computational Linguistics.
  * Niven and Kao (2019) Timothy Niven and Hung-Yu Kao. 2019.  Probing neural network comprehension of natural language arguments.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 4658–4664, Florence, Italy. Association for Computational Linguistics.
  * Poliak et al. (2018) Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, and Benjamin Van Durme. 2018.  Hypothesis only baselines in natural language inference.  In _Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics_ , pages 180–191, New Orleans, Louisiana. Association for Computational Linguistics.
  * Roemmele et al. (2011) Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S Gordon. 2011.  Choice of plausible alternatives: An evaluation of commonsense causal reasoning.  In _AAAI Spring Symposium on Logical Formalizations of Commonsense Reasoning_ , Stanford University.
  * Schick and Schütze (2021a) Timo Schick and Hinrich Schütze. 2021a.  Exploiting cloze-questions for few-shot text classification and natural language inference.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 255–269, Online. Association for Computational Linguistics.
  * Schick and Schütze (2021b) Timo Schick and Hinrich Schütze. 2021b.  It’s not just size that matters: Small language models are also few-shot learners.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2339–2352, Online. Association for Computational Linguistics.
  * Schuster et al. (2019) Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. 2019.  Towards debiasing fact verification models.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3410–3416, Hong Kong, China. Association for Computational Linguistics.
  * Sugawara et al. (2018) Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. 2018.  What makes reading comprehension questions easier? In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 4208–4219, Brussels, Belgium. Association for Computational Linguistics.
  * Trichelair et al. (2019) Paul Trichelair, Ali Emami, Adam Trischler, Kaheer Suleman, and Jackie Chi Kit Cheung. 2019.  How reasonable are common-sense reasoning tasks: A case-study on the Winograd schema challenge and SWAG.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 3373–3378, Hong Kong, China. Association for Computational Linguistics.
  * Vig and Belinkov (2019) Jesse Vig and Yonatan Belinkov. 2019.  Analyzing the structure of attention in a transformer language model.  In _Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 63–76, Florence, Italy. Association for Computational Linguistics.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019.  SuperGLUE: A stickier benchmark for general-purpose language understanding systems.  _arXiv preprint 1905.00537_.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_ , pages 353–355, Brussels, Belgium. Association for Computational Linguistics.
  * Wiegreffe and Pinter (2019) Sarah Wiegreffe and Yuval Pinter. 2019.  Attention is not not explanation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 11–20, Hong Kong, China. Association for Computational Linguistics.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 1112–1122, New Orleans, Louisiana. Association for Computational Linguistics.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R’emi Louf, Morgan Funtowicz, and Jamie Brew. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _ArXiv_ , abs/1910.03771.

##  Appendix A Datasets

###  A.1 MNLI

The Multi-Genre Natural Language Inference (Williams et al., 2018, MNLI)
dataset is an important dataset of natural language inference which is also
part of the SuperGLUE benchmark Wang et al. (2019). Given a premise, such as
_“The Old One always comforted Ca’daan, except today.”_ , and a hypothesis, in
this case, _“Ca’daan knew the Old One very well.”_ (neutral), a model is asked
to pick one label from among three, {contradiction, neutral, entailment}. The
test set of MNLI is divided into two subsets according to whether the domain
of each test instance matches the domain of the training set: matched (in-
domain instances) and mismatched (out-of-domain instances).

###  A.2 SNLI

The Stanford Natural Language Inference (Bowman et al., 2015, SNLI) is a
popular natural language inference dataset with the same format as MNLI.

###  A.3 HANS Datasets

McCoy et al. (2019) identified three context superficial cues in MNLI:
_lexical overlap_ , _subsequence_ , and _constituent_. These superficial cues
are predictive of entailment label. Both _subsequence_ and _constituent_
superficial cues are special cases of lexical _lexical overlap_. Following
this analysis, McCoy et al. (2019) created the HANS (Heuristic Analysis for
NLI Systems) dataset, which includes instances with superficial cues—lexical
overlap entails an entailment label and instances without superficial
cues—lexical overlap does not necessarily mean it is an entailment label.

###  A.4 COPA

The Choice of Plausible Alternatives (Roemmele et al., 2011, COPA) dataset is
a popular commonsense dataset that asks the model to select the most plausible
answer choice from the set of two candidates, which is also a part of the
SuperGLUE benchmark. For example, given a premise such as _“I tipped the
bottle.”_ and a question that can either be the _cause_ or _effect_ , the
model is asked to select the most plausible alternative from either _“The
liquid in the bottle froze.”_ or _“The liquid in the bottle poured out.”_
(correct).

##  Appendix B Prompt Templates and Answer Keys

In the templates we show the part that is replaced with actual question input
between two braces ({}), and we represent the mask token for a language model
with <MASK>.

###  B.1 MNLI

We use the template and answer keys from Gao et al. (2021).

Template:

{premise}? <MASK>, {hypothesis}

For the answer key that maps from model output to task labels we used:

{yes: entailment, maybe: netural, No: contradiction}

###  B.2 SNLI

We use the template and answer keys from Gao et al. (2021).

Template:

{premise}? <MASK>, {hypothesis}

For the _answer key_ that maps from model output to task labels we used:

{yes: entailment, maybe: netural, No: contradiction}

###  B.3 COPA

For COPA we use different templates from Schick and Schütze (2021b); Le Scao
and Rush (2021) depending on the question type. COPA instances can have a
_cause_ or _effect_ question.

For a _cause_ question with a given _premise_ and two possible causes;
_choice1_ and _choice2_. We use the template;

Train set | Cue Type | W/ Superficial Cues | W/O Superficial Cues
---|---|---|---
Meaningful | Meaningless | Meaningful | Meaningless
MNLI | lexical_overlap | 99.0 $\pm$0.4 | 92.3$\pm$3.7 | 9.3 $\pm$2.9 | 6.8 $\pm$3.7
subsequence | 99.6 $\pm$0.1 | 89.5 $\pm$5.6 | 3.9 $\pm$0.3 | 7.6 $\pm$3.6
constituent | 97.4 $\pm$1.0 | 89.9 $\pm$3.8 | 8.9 $\pm$2.2 | 12.4 $\pm$4.7
SNLI | lexical_overlap | 87.8 $\pm$8.4 | 80.7 $\pm$8.3 | 57.0 $\pm$17.8 | 16.7 $\pm$7.1
subsequence | 90.9 $\pm$7.1 | 85.2 $\pm$6.7 | 20.6 $\pm$10.2 | 18.6 $\pm$8.1
constituent | 95.3 $\pm$3.9 | 75.8 $\pm$8.6 | 17.6 $\pm$15.9 | 25.8 $\pm$11.3
Table 5: Average accuracy on HANS Datasets for prompt-based RoBERTa models trained on meaningful MNLI and SNLI; and evaluated on instances whose meaning has been preserved (Meaningful) and instances whose meaning has been removed (Meaningless). The instances are further divided into instances with (W/) and without (W/O) context superficial cues Dataset | w/ s. cues | w/o s. cues | Overall
---|---|---|---
MNLI | 69.5$\pm$0.1 | 38.6$\pm$0.6 | 66.7$\pm$0.1
MNLI-mm | 72.0$\pm$0.2 | 41.5$\pm$0.7 | 69.1$\pm$0.2
SNLI | 82.2$\pm$0.9 | 67.1$\pm$1.0 | 78.0$\pm$0.9
Table 6: Results of of prompt-based RoBERTa on MNLI and SNLI instances with
contextless superficial cues in the hypothesis and instances without
contextless superficial cues.

‘‘{choice1}’’ or ‘‘{choice2}’’? {premise}, so <MASK>.

and

{choice1} or {choice2}? {premise}, so <MASK>.

For an _effect_ question with a given _premise_ and two possible causes;
_choice1_ and _choice2_. We use the template;

‘‘{choice1}’’ or ‘‘{choice2}’’? <MASK>, because {premise}.

and

{choice1} or {choice2}? <MASK>, because {premise}.

For the answer key we use the identity function.

##  Appendix C Training Details

All our experiments are run on a single NVIDIA Tesla T4 GPU with 16GB memory.
All the models are trained in a few-shot setup. Thus they training only takes
a few minutes to complete. We use a learning rate of 1e-5 for all our
experiments.

##  Appendix D Numeric Results

For all the figures containing results—figure 2 and figure 4—we also include
numeric results in table 5 and table 6—to show the exact values and standard
deviations to make it easier to compare with future work.

##  Appendix E Attention Maps

This appendix shows the complete attention maps partially visualized in
section 6.2.

The attention maps are for a instance with superficial cues below.

Premise: The president was advised by the doctor.

Hypothesis: The doctor advised the president.

label: Entailment

And an instance without superficial cues:

Premise: The president advised the doctor.

Hypothesis: The doctor advised the president.

label: Non-Entailment

To make it easier to compare the attention maps, we alternate the pages
between the attention maps for an instance with and without superficial cues.
Each page has six attention maps to make it easier to see all the points.


# A Review of Repository Level Prompting for LLMs

Douglas Schonholtz
Northeastern University, Boston, Masssachusetts, 02115
<EMAIL_ADDRESS>

###### Abstract

As coding challenges become more complex, recent advancements in Large
Language Models (LLMs) have led to notable successes, such as achieving a
94.6% solve rate on the HumanEval benchmark. Concurrently, there is an
increasing commercial push for repository-level inline code completion tools,
such as GitHub Copilot and Tab Nine, aimed at enhancing developer
productivity. This paper delves into the transition from individual coding
problems to repository-scale solutions, presenting a thorough review of the
current literature on effective LLM prompting for code generation at the
repository level. We examine approaches that will work with black-box LLMs
such that they will be useful and applicable to commercial use cases, and
their applicability in interpreting code at a repository scale. We juxtapose
the Repository-Level Prompt Generation technique with RepoCoder, an iterative
retrieval and generation method, to highlight the trade-offs inherent in each
approach and to establish best practices for their application in cutting-edge
coding benchmarks. The interplay between iterative refinement of prompts and
the development of advanced retrieval systems forms the core of our
discussion, offering a pathway to significantly improve LLM performance in
code generation tasks. Insights from this study not only guide the application
of these methods but also chart a course for future research to integrate such
techniques into broader software engineering contexts.

##  1 Introduction

The advent of Large Language Models (LLMs) has revolutionized the field of
automated code generation, offering solutions that range from simple code
snippets to complex blocks that fit into larger systems. The HumanEval
benchmark, which emerged as a standard for evaluating the capabilities of
these models, has shown that LLMs can achieve high success rates, often
outperforming traditional programming approaches (?). However, as we shift
from isolated challenges to the nuanced realm of repository-level development,
new complexities arise. Developers don’t just need code that works; they need
code that integrates seamlessly with existing codebases—a challenge that
requires a deep understanding of context, dependencies, and the idiosyncrasies
of each repository.

The concept of Retrieval Augmented Generation (RAG) presents a paradigm shift
in this landscape. By leveraging existing code in a repository as a knowledge
base, RAG enables LLMs to generate more accurate and contextually relevant
code completions. This process is akin to a seasoned developer recalling
similar problems and solutions from past experiences to tackle current coding
challenges.

Furthering this idea, the integration of another LLM for the purpose of prompt
generation opens up new avenues for contextual understanding. It allows for a
more dynamic interaction with the codebase, where the LLM not only generates
code but also assists in identifying the most relevant pieces of information
to guide its generation process. This meta-level operation stands to
significantly enhance the model’s performance by refining its focus and
adapting its output to the specific context of the repository in question.

This paper provides an in-depth analysis of these innovative approaches,
particularly focusing on their effectiveness at the repository level—a scale
that presents a more realistic and challenging scenario for LLMs. We explore
two pioneering methods: the Repository-Level Prompt Generation technique and
RepoCoder, an iterative retrieval and generation method. By juxtaposing these
methodologies, we aim to uncover the trade-offs they present and establish a
set of best practices for leveraging LLMs in repository-level code generation.
Our review does not merely highlight their potential but also underscores the
necessity for sophisticated retrieval systems that can keep pace with the
evolving complexity of software development tasks.

As we navigate through these methodologies, our narrative will unravel the
intricate tapestry of LLM capabilities, their limitations, and the prospects
of combining various strategies to create a more potent tool for developers.
The synthesis of iterative refinement of prompts with advanced retrieval
systems promises a future where LLMs could become indispensable assistants to
programmers, offering intelligent suggestions that streamline the development
process and elevate the quality of software engineering.

(?), (?), (?)

##  2 Repository Level Retrieval Techniques

In order to be able to make good code completions at the repository level, an
LLM must be prompted with relevant context so that it can infer what code it
should generate.

RepoCoder(?) uses BM25(?), a sparse retrieval technique, across sliding
windows of code to retrieve code relevant to the the generated code, and the
code around the hole the generated code fills. Then that generation is used as
a query to find other relevant code with the same BM25/bag of words retrieval
system. This can then be done iteratively potentially improving every time
this process is run. This process is illustrated in figure 1

Figure 1: The iterative process of the RepoCoder framework. As seen in
RepoCoder (?)

The generated code can be modeled with this equation:
$\hat{Y}^{i}=M(C_{\text{ret}}^{i},X).$. Where Y is the output generated code
to fill the missing gap, X is the surrounding code, and $C_{repo}$ is the
repository context which is initially empty.

The context retrieval process can be modeled with this equation.
$C_{\text{ret}}^{i}=R(C_{\text{repo}},X,\hat{Y}^{i-1}).$ Where
$C_{\text{ret}}^{i}$ is iteration i of context retrieval.

In contrast, Repo Level Prompting presents a significantly more complicated
approach as it’s paper focuses on optimizing retrieval instead of optimizing
the iteration process with an existing retrieval method.

The repository-level Prompt Generation paper compiles code prompts from
various relevant code segments, which are then processed by the OpenAI Codex
model. The effectiveness of each prompt is assessed based on whether it
successfully completes the code gaps, marked by a binary score indicating
success or failure. All of the prompting options are indexed into a vector,
for a given repository completion the ground truth vector is defined as a one
in each index of that vector where the associated prompt resulted in a
successful generation. (?).

This can then be used to build a loss function which will allow classification
of these prompts as useful or not in arbitrary code contexts. The loss
function for the model is an average of the individual losses across code
holes, formalized in equation 1.

$\mathcal{L}=\frac{1}{N}\sum_{h=1}^{N}\mathcal{L}^{h}=\frac{1}{N}\sum_{h=1}^{N}\frac{1}{M^{h}}\sum_{p=1}^{M^{h}}BCE(\hat{y}_{p}^{h},y_{p})\times
T_{p}^{h}$ (1)

where $M^{h}=\sum_{p}r_{p}^{h}$ represents the total number of pertinent
prompt proposals for each code gap $h$, $N$ denotes the aggregate number of
code gaps during training, and $BCE$ signifies the binary cross-entropy loss
(?).

To predict optimal prompts for code generation by LLMs, two models are
constructed: RLPG-H, a two-layer dense network featuring a ReLU non-linearity,
and RLPG-R, which utilizes a single multi-head attention layer. Both of these
models are placed on top of frozen embedding models and then trained to
attempt to classify the optimal prompts to choose given the missing code
context. Despite RLPG-R using multi-head attention, it is less performant than
the dense model and it underperforms compared to RLPG-H.

The prompts generated are based off of the current, parent, children,
imported, fuzzy matched, and sibling classes, methods, and files. Then these
chunks of context are added above the default completion in no particular
order. They also are truncated as necessary from the front or back to fit into
the context window.

##  3 Datasets

The datasets used were different. RepoCoder uses a collection of repositories
they extracted via the GitHub API as seen in Table 1.

Table 1: Function Body Completion Dataset and Line and API Invocation Completion Datasets. F is the total number of python files, while N is the number of extracted Samples. Name | License | Created | F. | N.
---|---|---|---|---
imagen | MIT License | 2022-05-23 | 14 | 67
tracr | Apache V2.0 | 2022-12-01 | 56 | 146
lightmmmm | Apache V2.0 | 2022-10-10 | 36 | 64
inspection | Apache V2.0 | 2022-05-05 | 16 | 32
omnivore | CC BY-NC 4.0 | 2022-01-20 | 66 | 22
redframes | BSD-2-Clause | 2022-08-21 | 49 | 42
Line and API Invocation Completion Datasets
rl | MIT License | 2022-02-01 | 165 | 400
ACE | Apache V2.0 | 2022-11-23 | 425 | 400
vizier | Apache V2.0 | 2022-02-16 | 188 | 400
fortuna | Apache V2.0 | 2022-11-17 | 168 | 400
evaluate | Apache V2.0 | 2022-03-30 | 180 | 400
diffusers | Apache V2.0 | 2022-05-30 | 305 | 400
nerfstudio | Apache V2.0 | 2022-05-31 | 357 | 400
FedScope | Apache V2.0 | 2022-03-24 | 443 | 400

Meanwhile, Repo Level Prompting used repositories from Google as they feared
that GitHub’s data was used for training OpenAI’s Codex model that they were
testing. These repositories can be found in Figure 2. This is a collection of
Java repositories. It is worth noting, that the dataset for repo level
prompting is far larger and more comprehensive, as the number of total holes
there is much larger than the number of samples found in the number of samples
extracted from RepoCoder.

Figure 2: Repositories used in RepoLevel Prompting Table 2: Repo Level Prompting Dataset Feature | Train | Val | Test | Total
---|---|---|---|---
# Repositories | 19 | 14 | 14 | 47
# Files | 2655 | 1060 | 1308 | 4757
# Holes | 92721 | 48548 | 48288 | 189557

For Repo Level Prompting you can also see the total number of holes and
breakdown by train, validation and test splits for all repositories in Table 2

##  4 Comparing Oracle Retrieval Systems

Both RepoCoder, and RepoLevel Prompting compare themselves to optimal oracle
retrieval systems. In RepoCoder, this means doing the retrieval based on the
ground truth code to fill the hole that the LLM normally would first attempt
to hallucinate before doing it’s first round of retrieval. In RepoLevel
prompting, this means for their vector Y of possible successful prompts, using
that set of optimal working prompts to search for code. In both cases, this
creates an upper bound showing the limit of capability of these methods.

Table 3: Performance of the oracle relative to Codex for repo level prompting. Split | SR Codex(%) | SR Oracle(%) | Rel. ↑Codex(%)
---|---|---|---
Train | 59.78 | 80.29 | 34.31
Val | 62.10 | 79.05 | 27.28
Test | 58.73 | 79.63 | 35.58

You can see the effectiveness of the oracle and the relative improvement for
it in Repo Level Prompting in Table 3. You can find a breakdown of how the
oracle does for RepoCoder in a later set of results tables. The most important
take away from these though is that the oracle shows a potential upper bound
with the codex model for Repo Level Prompting of 35.48% on the test set. While
RepoCoder shows up to a 90% relative performance improvement on some function
completion tests and a 45% improvement on the exact match benchmark with
GPT-3.5 when compared to in-file prompting. These methods will be expanded on
more in the Validation of Code Generation section where we cover results.

##  5 Locations of Text in Repositories for Optimal Retrieval

One of the crucial findings of both RepoCoder and Repo Level Prompting is
where the critical code is stored. This information can be used to build
future potentially more effective retrieval systems to power these code
generating LLMs.

RepoCoder’s breakdown in Figure 4 shows that fuzzy import and names are
extremely valuable with usage marking in the 50 and 80 percentiles for cases
where the repocoder method beat the naive in file completion method.

Figure 3: Locations of retrieved code snippets when the Oracle/RepoCoder
method outperforms the In-File completion method using GPT-3.5-Turbo on the
line and API completion datasets

We can contrast the findings of RepoCoder with those of Repo Level Prompting
in Figure 4 and in Figure 5.

Figure 4: Mean success rates of different prompt sources when they are
applicable.  Figure 5: (Left) Normalized success rate of prompt sources when
applicable, (Right) Normalized success rate of prompt context types when
applicable

###  5.1 Short Hand File Source References

To understand these references we also have to understand these short hand
labels. They are broken out for convenience below.

  * •

Current: take code from the current file excluding the contents of the target
hole. The current file is the file that contains the target hole. The code in
the current file (e.g. the lines after the hole position) can be very useful
in predicting the target hole.

  * •

Parent Class: take code from the file that contains the parent of the class to
which the target hole belongs. The intuition behind this is to account for
cases where a method present in the parent class is invoked in the current
file (i.e. the child class).

  * •

Import: take code from the import files used in the current file. The
dependencies specified via imports can provide useful cues to predict the
target hole.

  * •

Sibling: take code from the files that are in the same directory as the
current file. Files in the same directory tend to share code variables (e.g.
identifiers).

  * •

Similar Name: take code from files that have a similar name as the current
file. Similar names are determined by splitting the file name based on
underscore or camel-case formatting and then matching parts of the filename.
If one or more parts matches, two files are considered to have similar names.
The intuition behind this is that software developers tend to name files based
on the functionality of the code written in that file. Therefore, a similar
name file might contain some portion of the code that is common with the
current file and hence might be useful for predicting the target hole.

  * •

Child Class: take code from files that have the current file as their parent
class file.

  * •

Import of Parent Class: take code from the import files used in the parent
class files.

  * •

Import of Sibling: take code from the import files used in the sibling files.

  * •

Import of Similar Name: take code from the import files used in the similar
name files.

  * •

Import of Child Class: take code from the import files used in the child class
files.

###  5.2 Short Hand Data to be Extracted from Files

To understand, the figure to the right in Figure 5, we have to understand this
corresponding list as well.

  * •

Post Lines (PL): Take all the lines after the target hole line till the end of
the current file.

  * •

Identifiers (I): Take all the identifiers used in the prompt source.

  * •

Type Identifiers (TI): Take all the type identifiers used in the prompt
source.

  * •

Field Declarations (FD): Take all the field declarations used in the prompt
source.

  * •

String Literals (SL): Take all the string literals used in the prompt source.

  * •

Method Names (MN): Take all the method names along with their signatures used
in the prompt source.

  * •

Method Names and Bodies (MNB): Take all the method names along with their
signatures and corresponding bodies used in the prompt source.

##  6 Validation of Code Generation

RepoCoder evaluates their system on three separate benchmarks they build. The
first benchmark was line completion, completing a single line in a repository.
The second benchmark was function completion, where the model has to complete
a function which has had some subset of its lines removed. The results for
this benchmark can be seen in Table 4.

Table 4: Line Completion performance evaluation using RepoCoder for exact match (EM) and Normalized Edit Distance (NED) Metric | Oracle | In-File | RepoCoder Iterations
---|---|---|---
|  |  | 1 | 2 | 3 | 4
GPT-3.5-Turbo
EM | 57.75 | 40.56 | 55.31 | 56.81 | 57.00 | 56.63
NED | 75.43 | 65.06 | 74.38 | 75.11 | 75.30 | 75.10
CodeGen-Mono-6B
EM | 48.81 | 34.56 | 45.81 | 47.06 | 47.75 | 47.44
NED | 71.02 | 60.67 | 69.21 | 70.10 | 70.73 | 70.19
CodeGen-Mono-2B
EM | 47.31 | 33.63 | 44.56 | 46.94 | 46.69 | 47.13
NED | 69.80 | 58.99 | 67.68 | 68.82 | 68.62 | 68.92
CodeGen-Mono-350M
EM | 45.19 | 29.56 | 41.88 | 43.06 | 43.94 | 43.06
NED | 67.20 | 55.39 | 65.05 | 65.66 | 65.97 | 65.62

The third benchmark was API invocation completion. This included 1046 and 1086
samples that were run on GPT-3.5 Turbo and other smaller CodeGen models
respectively. Where an api invocation was just an api that was invoked
somewhere in the codebase and that was the code that had to be completed. All
of these methods are then evaluated against an exact match metric, a metric to
determine if the generated code matches character for character with the true
expected code. They are then also tested with a normalized edit distance
metric (NED), which is the Levenshtein distance(?) divided by the maximum
character count between the generated code and the ground truth. You can see
the results for this measurement in Table 5.

Table 5: API Invocation Completion performance evaluation using RepoCoder. Metric | Oracle | In-File | RepoCoder Iterations
---|---|---|---
|  |  | 1 | 2 | 3 | 4
GPT-3.5-Turbo
EM | 50.13 | 34.06 | 47.69 | 49.19 | 49.44 | 49.56
NED | 74.50 | 63.22 | 73.63 | 74.43 | 74.59 | 74.48
CodeGen-Mono-6B
EM | 40.25 | 26.19 | 36.69 | 38.88 | 39.13 | 39.31
NED | 67.94 | 56.45 | 64.20 | 65.52 | 65.53 | 65.90
CodeGen-Mono-2B
EM | 39.44 | 25.44 | 35.44 | 37.56 | 38.44 | 38.25
NED | 66.78 | 56.88 | 63.47 | 64.15 | 64.53 | 64.60
CodeGen-Mono-350M
EM | 34.88 | 22.19 | 31.75 | 33.88 | 33.75 | 33.81
NED | 63.06 | 52.24 | 59.82 | 61.03 | 60.96 | 61.06

Both of these results rely heavily on the exact match and NED metric. Such
evaluations, however, are limited by their syntactic focus, neglecting the
semantic similarity vital for grasping the intended code functionality.
Moreover, the approach does not consider anonymized variables common in
arbitrary solutions, nor does it acknowledge various valid problem-solving
methods.

To mitigate these limitations, test case validation is also employed, though
the methodology lacks a pre-and post-modification test state analysis, raising
concerns about the integrity of the test cases and their coverage of the
modified code. Meaning that verification that the tests pass before and after
the code is implemented in the ground truth state, and that the unchanged code
causes the modified tests to fail, as is described in other papers.(?).
Despite these concerns, the pass rate of 44.6% in repository test cases
reported by RepoCoder is promising, especially when juxtaposed with a mere
23.32% achievement through in-file retrieval. The improvements of 8% and 10%
in exact and Levenshtein match scores over the baseline model further
underscore the potential of the repo coder methodology. These results are
shown in Table 6

Table 6: RepoCoder Pass Rate against unit tests. N. | Oracle | In-File | RepoCoder Iterations
---|---|---|---
|  |  | 1 | 2 | 3 | 4
67 | 56.72 | 29.85 | 53.73 | 55.22 | 55.22 | 55.22
146 | 43.84 | 27.40 | 41.78 | 43.84 | 44.52 | 44.52
64 | 32.81 | 10.94 | 25.00 | 34.38 | 31.25 | 32.81
32 | 34.38 | 28.13 | 34.38 | 37.50 | 34.38 | 34.38
22 | 40.91 | 31.82 | 31.82 | 36.36 | 31.82 | 36.36
42 | 38.10 | 9.52 | 28.57 | 38.10 | 38.10 | 38.10
373 | 42.63 | 23.32 | 38.34 | 42.63 | 41.82 | 42.36

When we compare this to Repo Level Prompting, we can see that the NED relative
comparison is comprable with a large 25% relative improvement to the baseline.
You can see this in Table 7.

Table 7: Normalized Edit Distance (NED) based performance evaluation for repo level prompting. Method | NED(%) | Rel. ↑(%)
---|---|---
Codex | 30.73 | -
RLPG-H | 22.55 | 26.62
RLPG-R | 23.00 | 25.14

You can also see the impact of each of their different retrieval models. It is
worth noting that the file level BM25 is fairly good, but their RLPG-H is the
highest performing strategy. Considering how performant the RLPG-BM25 model
is, it is possible that taking the prompt generation strategy and doing a BM25
classification may be the optimal option in terms of latency, compute and
developer time for many developers who rapidly want something that is highly
performant. These results can be seen in 8.

Table 8: Comparison of methods for repo level prompting. Method | Success Rate(%) | Rel. ↑(%)
---|---|---
Codex (Chen et al., 2021) | 58.73 | -
Oracle | 79.63 | 35.58
Random | 58.13 | -1.02
Random NN | 58.98 | 0.43
File-level BM25 | 63.14 | 7.51
Identifier Usage (Random) | 64.93 | 10.55
Identifier Usage (NN) | 64.91 | 10.52
Fixed Prompt Proposal | 65.78 | 12.00
RLPG-BM25 | 66.41 | 13.07
RLPG-H | 68.51 | 16.65
RLPG-R | 67.80 | 15.44
Figure 6: Variation of RLPG and Fixed Prompt Proposal with Num. attempts (k),
when averaged over individual repositories (repo-wise) and all holes (hole-
wise)

Another experiment the authors of repo level prompting can be seen in Figure
6. There you can see the impact of choosing the top k prompt proposals the
classifiers chose. This shows the steadily increasing performance with further
prompts. This is somewhat surprising as you would expect truncation to fit
everything into the context window or filling the context window to rapidly be
a problem, but performance seems to only rise.

##  7 How to build an optimal prompting system

Considering both of these papers, RepoCoder and Repository Level Prompt
Generation can dramatically improve LLM performance when filling gaps in
missing chunks of code. On top of that, while one method focuses on
iteratively improving generation with retrieval results that are progressively
more relevant with an off the shelf retriever, the other focused on building a
better retrieval system by using a neural network to figure out exactly what
code should be retrieved. It therefore, would make sense to attempt to combine
these methods, especially since the sparse retrieval method performed so
poorly in Repo Level Prompting. To accomplish this, one would have to train
the neural network described in (?) but with the hole filled in and the
generated code present. Then to iteratively rerun the generation with the
context found.

##  8 Future Work

On top of naively combining these two methods, another next step would be to
integrate them into more complex use cases then doing auto-complete for a
single line, function or API request in a repository. One such example of this
is described in Software Engineering Bench(?). There they build an extremely
robust benchmark where they collect 2,294 github issues that have associated
pull requests and test cases. They verify that all of the test cases succeed
after the pull request, and if you run the code before the pull request
against the test, after the ground truth code is applied to the tests, that
some of them fail. This handles several of the edge cases and problems seen in
the RepoCoder Test Benchmark.

In SWE-Bench they provide several additional retrieval systems and tools, but
their retrieval system only retrieves entire files which will mostly contain
noise and not be useful. Also, many of the issues are very large and must be
condensed in some meaningful way. Even their oracle retrieval system pulls in
all of the full files that were modified instead of just subsets of them.
Because of this the performance of the language models tested is extremely
poor, Claude-2 solves 4.8% and GPT-4 solves 1.7%. Considering Claude is
significantly worse in other benchmarks like HumanEval, 71.2 compared to
GPT-4’s 86%(?) (?). This is likely because of Claude’s long context length and
the fact that their retrieval systems only support searching for entire files.
Therefore, it likely would be possible to top their online leaderboard at SWE-
Bench.com by combining these techniques of iterative generation and custom
retrieval proposed in RepoCoder (?) and Repo Level Prompting (?) with their
benchmark.

The next place to look after all of this may be to agent based LLM
usage.(?)(?)(?) Building on Reflexion(?), or Language Agent Tree Search (?),
the two top performing HumanEval papers which use a mix of reinforcement
learning where the LLM is performing the actor and critic work and other
methods like monte-carlo tree search to build optimal paths in traversing the
problem space. Although these methods are currently computationally expensive
and induce high latency, it is likely that as LLM inference time continues to
fall that some combination of these methods will create the best performance
and will still not force the user to wait for intractably long periods of
time.

##  9 Conclusion

This paper has provided a comprehensive review of two innovative approaches
for repository-level prompting in Large Language Models: Repository-Level
Prompt Generation and RepoCoder. Our analysis revealed that each method has
its strengths and trade-offs. RepoCoder’s iterative retrieval and generation
method excels in refining the prompt process, while Repository-Level Prompt
Generation offers a robust framework for evaluating the effectiveness of
various prompts against code completion tasks.

We have demonstrated that by leveraging these methods, LLMs can be prompted to
produce contextually relevant and syntactically coherent code completions that
align with the complex requirements of real-world software repositories. The
iterative nature of RepoCoder aligns well with the dynamic development
environments, offering incremental improvements in code generation.
Conversely, the classification-based approach of Repository-Level Prompt
Generation provides a more direct path to identifying effective prompts,
although it may require more extensive training data.

Our comparative analysis underscores the potential for integrating these
strategies to harness their combined strengths. By doing so, we can address
the current limitations of sparse retrieval methods and pave the way for more
sophisticated code generation models. The promising results from applying
these methods suggest that we are on the cusp of a new era in automated
coding, where LLMs can not only assist in code generation but also evolve with
the changing landscapes of software development.

As we look to the future, the integration of these methods into more complex
systems, such as those described in Software Engineering Bench, represents the
next frontier for LLMs. The potential for topping performance leaderboards by
combining iterative generation with custom retrieval systems is substantial
and warrants further exploration.

Ultimately, the journey towards fully autonomous and intelligent coding
assistants is just beginning. We anticipate that as LLM inference times
decrease, the amalgamation of agent-based LLM usage, reinforcement learning,
and other advanced techniques will set new benchmarks in the field. The
research community is encouraged to build upon the insights presented in this
study to create LLMs that not only augment the coding process but also
contribute to the art of programming itself.

## References

  * [Amati 2009] Amati, G.  2009\.  BM25.  Boston, MA: Springer US.  257–260.

  * [Anthropic 2023] Anthropic.  2023\.  Claude-2 technical report.
  * [Chen et al. 2021] Chen, M.; Tworek, J.; Jun, H.; Yuan, Q.; de Oliveira Pinto, H. P.; Kaplan, J.; Edwards, H.; Burda, Y.; Joseph, N.; Brockman, G.; Ray, A.; Puri, R.; Krueger, G.; Petrov, M.; Khlaaf, H.; Sastry, G.; Mishkin, P.; Chan, B.; Gray, S.; Ryder, N.; Pavlov, M.; Power, A.; Kaiser, L.; Bavarian, M.; Winter, C.; Tillet, P.; Such, F. P.; Cummings, D.; Plappert, M.; Chantzis, F.; Barnes, E.; Herbert-Voss, A.; Guss, W. H.; Nichol, A.; Paino, A.; Tezak, N.; Tang, J.; Babuschkin, I.; Balaji, S.; Jain, S.; Saunders, W.; Hesse, C.; Carr, A. N.; Leike, J.; Achiam, J.; Misra, V.; Morikawa, E.; Radford, A.; Knight, M.; Brundage, M.; Murati, M.; Mayer, K.; Welinder, P.; McGrew, B.; Amodei, D.; McCandlish, S.; Sutskever, I.; and Zaremba, W.  2021\.  Evaluating large language models trained on code.
  * [Jimenez et al. 2023] Jimenez, C. E.; Yang, J.; Wettig, A.; Yao, S.; Pei, K.; Press, O.; and Narasimhan, K.  2023\.  Swe-bench: Can language models resolve real-world github issues?
  * [Levenshtein 1965] Levenshtein, V. I.  1965\.  Binary codes capable of correcting deletions, insertions, and reversals.  Soviet physics. Doklady 10:707–710.
  * [OpenAI 2023] OpenAI.  2023\.  Gpt-4 technical report.
  * [Shinn et al. 2023] Shinn, N.; Cassano, F.; Berman, E.; Gopinath, A.; Narasimhan, K.; and Yao, S.  2023\.  Reflexion: Language agents with verbal reinforcement learning.
  * [Shrivastava, Larochelle, and Tarlow 2023] Shrivastava, D.; Larochelle, H.; and Tarlow, D.  2023\.  Repository-level prompt generation for large language models of code.
  * [Yao et al. 2023] Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.; and Cao, Y.  2023\.  React: Synergizing reasoning and acting in language models.
  * [Zhang et al. 2023] Zhang, F.; Chen, B.; Zhang, Y.; Keung, J.; Liu, J.; Zan, D.; Mao, Y.; Lou, J.-G.; and Chen, W.  2023\.  Repocoder: Repository-level code completion through iterative retrieval and generation.
  * [Zhou et al. 2023] Zhou, A.; Yan, K.; Shlapentokh-Rothman, M.; Wang, H.; and Wang, Y.-X.  2023\.  Language agent tree search unifies reasoning acting and planning in language models.