#  CodeBERT-nt: code naturalness via CodeBERT

Ahmed Khanfir, Matthieu Jimenez, Mike Papadakis and Yves Le Traon
University of Luxembourg, Luxembourg, Luxembourg
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

###### Abstract

Much of software-engineering research relies on the naturalness of code, the
fact that code, in small code snippets, is repetitive and can be predicted
using statistical language models like n-gram. Although powerful, training
such models on large code corpus is tedious, time consuming and sensitive to
code patterns (and practices) encountered during training. Consequently, these
models are often trained on a small corpora and estimate the language
naturalness that is relative to a specific style of programming or type of
project. To overcome these issues, we propose using pre-trained language
models to infer code naturalness. Pre-trained models are often built on big
data, are easy to use in an out-of-the-box way and include powerful learning
associations mechanisms. Our key idea is to quantify code naturalness through
its predictability, by using state-of-the-art generative pre-trained language
models. To this end, we infer naturalness by masking (omitting) code tokens,
one at a time, of code-sequences, and checking the models’ ability to predict
them. To this end, we evaluate three different predictability metrics; a)
measuring the number of exact matches of the predictions, b) computing the
embedding similarity between the original and predicted code, i.e., similarity
at the vector space, and c) computing the confidence of the model when doing
the token completion task irrespective of the outcome. We implement this
workflow, named CodeBERT-nt, and evaluate its capability to prioritize buggy
lines over non-buggy ones when ranking code based on its naturalness. Our
results, on 2,510 buggy versions of 40 projects from the SmartShark dataset,
show that CodeBERT-nt outperforms both, random-uniform and complexity-based
ranking techniques, and yields comparable results (slightly better) than the
n-gram models.

##  I Introduction

There is a large body of research demonstrating that code, in small snippets,
is repetitive and thus predictable [1]. This repetitiveness of code is
typically captured by statistical language models, like n-grams [2], that
determine the appearance likelihood of a token given its $n$ preceding (or
subsequent) ones. By computing the likelihood of tokens, in a sequence, and by
using cross-entropy [3, 4], we can estimate the appearance likelihood of a
given token sequence.

In essence, n-gram-based cross-entropy quantifies the conformance of a given
code sequence to the sequences appearing within the code corpus that was used
during the n-gram training process. This naturalness metric of code can be
quite powerful since it can identify unusual code sequences, that may reflect
code that is smelly [5, 6], of low readability [7, 8], or simply a rare
specific implementation instance [9].

Although powerful, training such models on large code corpus is tedious, time
consuming and sensitive to code patterns (and specificities) of the used
projects. Additionally, the models are sensitive to a number of meta-
parameters such as tokenizers, smoothing techniques, unknown threshold and $n$
values [10]. Consequently, these models are often trained on a small corpus
and estimate the language naturalness that is relative to a specific style of
programming or type of project [1]. Moreover, their generalization ability, in
a cross-project fashion, when trained on large corpus remains unclear since no
such evidence appear in the related literature.

To overcome these issues, we propose using generative pre-trained language
models, such as CodeBERT, to infer code naturalness. Pre-trained models have
been shown to provide strong results on several code-related tasks such as
code generation and translation. They are often built on big data, are easy to
use in an out-of-the-box way and include powerful learning associations
mechanisms that allow them to generalize well to unseen code and projects.

Therefore, our key objective is to form a powerful and flexible – easy to use
– tool for computing code naturalness that is applicable in a cross-project
fashion. However, due to the generative nature of these models, they do not
allow the computation of cross-entropy like measures, i.e., the probability of
the appearance of given tokens of code.

To deal with this issue we view code naturalness as a predictability metric.
Thus, we measure how well code tokens can be generated by the models, and use
it to infer naturalness. In particular, we mask (omit) code tokens, one at a
time, of code sequences, and check the models’ ability to predict them. We
then define and investigate the use of predictability metrics, that can be
computed based on CodeBERT, to infer code naturalness. To this end, we
evaluate three different predictability metrics:

  * •

counting the number of exact matches of the predictions.

  * •

computing the embedding similarity between the original and predicted code,
i.e., similarity at the vector space.

  * •

computing the confidence of the model when doing the token completion task,
regardless to the outcome.

We implement this workflow, which we call CodeBERT-nt, and evaluate its
suitability in inferring code naturalness. We also investigated the use of
different aggregation methods on top of the above-defined token predictability
metrics, such as minimum, maximum, mean, median and cross-entropy to infer the
naturalness of code statements/lines as a whole thereby computing (n-token)
naturalness and going beyond single tokens.

Since naturalness of code is a relative measure, i.e., it strongly depends on
the models and training data used, and every metric involves its own specific
building blocks, we cannot directly contrast the values of one metric over the
others. To bypass this problem we compare the metrics according to the
relative performance differences on an end task, for which we have a solid
ground truth. In particular, we evaluate the performance of our metrics w.r.t.
their ability to rank buggy lines of code.

Previous work has shown that unusual code is often linked with bug-proneness
and bugginess, thereby making n-gram-based cross entropy a tool capable to
identify likely buggy code [9, 11]. This means that buggy lines tend to be
unnatural (entropy is high), or at least more unnatural compared to the clean
ones that tend to be natural (entropy is low). Moreover, empirical studies
[11], have shown that code naturalness has a comparable performance to other
static bug finding approaches such as PMD [12] and FindBugs [13].

We, therefore, investigate the ability of CodeBERT-nt to distinguish natural
(clean) from unnatural (buggy) code. To do so, we need a solid ground truth of
buggy and clean code. Hence we select the SmartShark dataset [14], which
contains manually untangled buggy and fixed code versions. This allows us to
know the buggy code lines for which we can perform our experiment. We thus,
used 2,510 buggy versions from 40 projects and investigated the performance of
CodeBERT-nt and contrast it with that of typical baselines such as uniform-
random and complexity-based rankings [15].

To further strengthen our study, we also compared our results with that of
n-gram models when trained on an intra-project fashion (as typically performed
in literature [10, 11]). Precisely, for every subject bug, we split its
corresponding project-version code into two sets of lines: 1) the training set
which counts all neutral file lines (files not impacted by the fix-commit) and
2) the evaluation lines set which counts all lines from the remaining files.
We used recommended settings by Jimenez et al. [10] – unknown threshold$=1$,
$n=4$ with KN as smoother – and two fundamentally different tokenizers; 1)
UTF8 which tokenize the code as a natural-language text including the comments
and javadoc and 2) Java Parser [16] which tokenize the code according to the
java language grammar, excluding comments and documentation. Finally, we
compared CodeBERT-nt metrics with the cross-entropy measured by these models
in terms of ranking the buggy lines. Our results suggest that our (inter-
project) CodeBERT-nt yields comparable results (slightly better) than the
(intra-project) naturalness predictions of the n-gram models.

Overall, our primary contributions are:

  * •

We demonstrate that pre-trained generative models like CodeBERT capture the
language naturalness notion. We can infer the naturalness aspect of a source-
code from the CodeBERT prediction results.

  * •

We introduce a novel approach to detect source-code naturalness that works in
cross-project context and does not rely on the aspects of naturalness that are
tight to a specific project. For instance, It does not require any further
training or specific knowledge of the target project, to rank its lines by
bugginess likelihood, thus can be easily used in the future as a baseline.

  * •

We provide empirical evidence demonstrating that the proposed approach is
comparable to the current state-of-the-art.

##  II Background and related work

###  II-A Code-naturalness

Estimating the source-code naturalness and predictability has been widely
investigated. Based on the fact that code is repetitive, the measure of
naturalness was adopted on programming languages [1], using fundamentally the
same techniques as for natural languages. Notably, Language models and
particularly n-grams are the most popular medium of computing this measure.
N-gram models approximate the naturalness of a sequence of tokens based on its
occurrence likely-hood, estimated relatively to the sequences observed in the
training set. This probability follows a Markov chain conditional probability
series, where the probability $P(t)$ of a token $t$ to occur depends on the
$n-1$ preceding tokens. To highlight irregular sequences (low probabilities of
occurring), the naturalness of a sequence is expressed as a cross-entropy [3,
4] which is computed by aggregating the logarithm of its token probabilities
as follows:

$\text{H}(S)=-\frac{\sum_{i=1}^{m}\log(P(t_{i}|t_{i-n+1}...t_{i-1}))}{n},$ (1)

where $n$ denotes the order of the n-gram model, $\\{t_{1},...,t_{m}\\}$ the
set of $m$ tokens forming the sequence $S$ and $P(t_{i}|t_{i-n+1}...t_{i-1})$
the probability $P(t_{i})$ knowing $t_{i-n+1}...t_{i-1}$. Consequently, n-gram
models attribute high entropy values to unusual (not natural) code relatively
to regular (natural) code. To avoid assigning a zero probability to tokens
that do not appear in the model vocabulary (which is often the case of
variable names), n-gram models replace every token occurring less than $k$
times by a placeholder <UNK> and attributes a not zero probability to it,
where $k$ and <UNK> are usually called the unknown threshold and unknown word
[2, 17, 10]. Similarly, to deal with unseen sequences of tokens in the
training set, several smoothing techniques have been proposed and evaluated,
among which Kneser Ney (KN) [18] and Modified Kneser Ney (NKM) [17] are the
best performing smoothers [17, 1], with an eventual advantage for the latter
one [10]. Among the potential applications of the naturalness of code
property, Baishakhi et al.[11] have shown empirical evidence that buggy lines
are on average less natural than not-buggy ones and that n-gram entropy can be
useful in guiding bug-finding tasks at both file- and line-level. Jimenez et
al. [10] evaluated the sensitivity of n-gram w.r.t. its parameters and code
tokenization techniques, via a file level naturalness-study. Their results
confirmed Baishakhi et al. findings and provided recommendations on the best
n-gram configurations for naturalness based applications, including the
differentiation between buggy and fixed code.

###  II-B CodeBERT

Microsoft has recently introduced CodeBERT [19], a bimodal pre-trained
language model that supports multiple NL-PL (Natural Language and Programming
Language) applications such as code search, code documentation generation,
etc. The model was developed with a Multilayer Transformer [20] architecture,
which is adopted by the majority of large pre-trained models like BERT [21].
It has been trained on a large-scale dataset counting over 6 million projects
from GitHub in 6 different programming languages, including Java. To ensure
its bimodal functionalities, CodeBERT was trained towards a hybrid objective
function (based on replaced token detection) in a cross-modal style, based on
bimodal NL-PL data (precisely, source-code paired with its documentation) and
unimodal data (including NL and PL sequences). In this work, we incorporate
its Masked Language Modeling (MLM) functionality [22] in our experiments
pipeline, in order to study the possibility of inferring code naturalness from
the CodeBERT prediction results.

##  III Research Questions

Our main objective is to form a flexible and easy-to-use tool for computing
naturalness of code in a cross-project fashion. We thus, aim at using pre-
trained generative language models, such as CodeBERT, that are powerful,
readily available and trained over very large corpora. Since naturalness is a
metric of how predictable a piece of code is and such pre-trained models are
trained to predict code tokens, given their surrounding code, there should be
a way to exploit them and compute code naturalness. However, since these
models are generative, they do not compute the probability of the appearance
of given tokens of code thereby obviating the naturalness calculation. To this
end, we investigate the use of potential predictability metrics, that can be
computed based on CodeBERT, and allow quantifying the predictability of code,
which in fact represents code naturalness.

Since naturalness of code is a relative measure, i.e., it strongly depends on
the models and training data used, we cannot directly contrast the values of
one metric over the others. We therefore, contrast them based on their
performance on an end task, for which we have a solid ground truth. In
particular, we evaluate the performance of our metrics w.r.t. the naturalness
of bugs hypothesis that states: ”unnatural code is more likely to be buggy
than natural code” [11]. This means that a good naturalness metric should be
capable of distinguishing natural (clean) from unnatural (buggy) code. Hence,
we ask:

RQ1

_(Metrics and aggregation methods):_ Which metrics and line-level aggregation
methods lead to the best segregation between buggy and not buggy lines?

To answer this question, we check if any pairs of metric-aggregation
discriminate better the buggy from the clean lines. To do so, we rank and
compare the results of both - ascendant and descendant - sorting orders on all
possible pairs to determine the one that discriminates best. Using these
matching sorting orders for every pair, we select the most effective
aggregation method for every metric in terms of ranking buggy lines.

However, getting the best ranking does not necessarily mean that our metrics
perform well. We thus, turn our attention to the significance of these results
and contrast them with some obvious baselines, such as the random order and
source-code complexity order (ranking the most complex lines first). Random
order offers a sanity check for coincidental results, and complexity offers an
unsupervised baseline [23], i.e., shows that our metrics do not simply measure
complexity instead of naturalness. Therefore, we ask:

RQ2

_(Comparison with random and complexity based rankings):_ How does CodeBERT
naturalness compares with random and source-code complexity, in terms of buggy
lines ranking?

To check whether the naturalness captured by CodeBERT is useful, we compare
with 1) a uniform random selection based technique and 2) a code-complexity
based one, where the complexity is measured in terms of number of tokens and
the most complex lines are considered more likely to be buggy. To make a fair
comparison, we compare the techniques on ranking the same subject lines and
the same buggy versions, which have each at least one buggy line.

As the above-mentioned baselines are relatively weak, we also compare our
metrics with n-gram models that were originally used to show the naturalness
of buggy code hypothesis. This leads us to the following question:

RQ3

_(Comparison with n-gram):_ How does CodeBERT compare with n-gram language
models, in terms of lines ranking by bugginess?

We answer this question by ranking the subject lines based on their cross-
entropies measured by n-gram models trained on the source-code of the same
version of the considered project. In fact, for every considered bug we create
two n-gram models (using each of UTF8 and Java Parser tokenizers) trained on
the source code of the files that has not been changed by the fix patch. Then,
we use these models to calculate the cross-entropies of the subject lines.
Finally, we compare the ranking results by the n-gram models with our proposed
approach, the same way as we did in RQ2.

Obviously, the RQ3 comparison does not aim at answering the question of which
model is best, but to show a relative performance of the models. This is
because the training corpora of the models are significantly different with
each one having significant associated advantages. In particular, CodeBERT has
an advantage of being trained on a much larger corpus than the n-grams.
However, n-grams have the advantage of operating on an intra-project fashion,
which is significantly more powerful [24] than the inter-project cases, as the
one of CodeBERT.

##  IV Experimental Setup

###  IV-A Dataset & Benchmark

To perform our study, we use a recently crafted dataset of commits; SmartSHARK
[14]. This dataset is distributed as a MongoDB database describing commit-
details extracted from multiple open-source python and java projects. A
specificity of Smartshark is that its data are manually refined, i.e., the
commits are untangled and the fix validated, hence reducing the threat of
noisy data.

In our study, we use the issue-fixing commits of the 40 available java
projects the dataset includes.

####  IV-A1 Buggy versions selection

For most of the issues, SmartSHARK provides one or more related commits, among
which the fixing-commits are labeled as validated fix-commits. Thus, to build
our bugs dataset, we exclude all issues having no corresponding fix-commit, or
having multiple fix-commits. While the first case is straight forward, the
latter is applied to further reduce noise in the data as pinpointing bugs
origin from multiple fixing commits is harder and more error prone. This way,
we obtain one fix-commit per issue where the changed lines form the complete
set of buggy lines. Then, as we focus on Java source-code naturalness, we
exclude the issues whose fixes are not involving java files, e.g.,
configuration files. Finally, we map the resulting issues to their related
buggy version, i.e., the project version preceding the fix commit.

####  IV-A2 Lines subject to study

For every considered issue, we retrieve the files changed by the patching
commit, among which we mark as buggy any line changed by the fixed commit and
define as neutral ones the remaining lines in these files.

Then to ensure that the observed ranking performances are related to code
naturalness extraction and not to any non-uniform distribution of the buggy-
lines over the code, we exclude all lines that are not part of the business-
logic of the program like the imports, the fields declaration, the brackets,
etc. We do so to focus on logic types of faults, which are unlikely to be
caused by non-business-logic statements as also suggested by the work of
Rahman et al. [25]. Moreover, the naturalness of these lines is unlikely to be
insightful.

Additionally, this allows us to exclude any interference between comparing the
ranking-performance by naturalness and other forms of rankings related to the
buggy lines unequal distribution between business-logic and not business-logic
related source-code portions. In fact, close to half of the considered bugs
see all of their buggy lines located within the business-logic code, while
only 10% of the bugs see all of their buggy-lines outside of it. Unaccounting
for those 10%, we observe a median of 90% of buggy lines located within the
business-logic code per bug, while this type of line only amounts to 60% of
the lines overall.

To sum-up, by excluding these lines we exclude 257 bugs out of 2510 to end up
with a final dataset counting 2253 buggy versions, represented by a set of
business-logic buggy and not buggy lines.

###  IV-B Experimental Procedure

####  IV-B1 CodeBERT-based naturalness metrics

Figure 1: The CodeBERT-nt source-code metrics calculation workflow.

To infer the naturalness aspect of source-code from CodeBERT predictions we
propose a 3-steps process that we name CodeBERT-nt. CodeBERT-nt operates in 3
steps as illustrated in the Figure 1:

For every selected buggy file, the tool starts by parsing the Abstract Syntax
Tree (AST) and extracting the interesting nodes to mask, excluding the
language-specific tokens such as control flow and import-related tokens, e.g.,
if, else, for, while, import, etc. Then, for every line, the tool iterates
over the selected set of nodes and replaces each node’s content by the
placeholder <mask>, thus, generating one masked version of the input code per
selected node.

Next, every masked version is tokenized into a vector of tokens using the
CodeBERT tokenizer, and shrink to only encompass the masked token and its
surroundings, as the model encoder can only take up to 512 tokens. Once the
shrinking is done, the sequences of masked codes are fed to CodeBERT to
determine the best fitting substitute for the mask placeholder. By default,
the model provides 5 propositions ranked by likelihood (also called
confidence) to match the masked node original value. In our setting, we only
consider the first proposition as we believe it to be the most naturalness-
revealing one (we discuss this choice further in Section VI-C) and compute the
following metrics from it:

  * •

_CBnt_conf:_ the prediction confidence score of CodeBERT. This metric
represents a probability, thus is a floating number varying between 0 and 1,
where the closer to 1 its value is, the more CodeBERT is confident about the
prediction. We believe that this metric may mirror directly the naturalness of
code as it reflects how predictable and usual is the code, relatively to the
code knowledge learned by the model through its training phase on a large
scale code dataset. Thus, low confidence score may imply low naturalness.

  * •

_CBnt_cos:_ the cosine similarity between the CodeBERT embeddings of the
predicted and the original source-code. This metric has a float value varying
between 0 and 1, where 1 implies an exact similarity between the two
embeddings and 0 the absence of similarity. This metric is often used in NLP
and has shown some interesting results in filtering unnatural sentences [26].
CodeBERT embedding is the encoded representation of the code in the latin-
space, where every token is represented by 1 vector. To calculate the cosine
similarity between the embeddings, we start by concatenating their token-
vectors into two vectors, one for each embedding, then we compute the cosine
as follows:

$\text{Cosine}(\mathbf{V}_{o},\mathbf{V}_{p})=\frac{\mathbf{V}_{o}.\mathbf{V}_{p}}{\|\mathbf{V}_{o}\|.\|\mathbf{V}_{p}\|},$
(2)

where $\mathbf{V}_{o}$ and $\mathbf{V}_{p}$ are the concatenated embedding
vectors of respectively the original and predicted code.

Our intuition is that, the less natural the code is, the more CodeBERT will
have issues to notice small changes in it, i.e., a difference in a single
node, and thus, the higher the cosine will be. Consequently, the high
similarity between both embeddings – of the original and predicted code – may
be a symptom of unnaturalness of the code.

  * •

_CBnt_acc:_ the accuracy of prediction (whether the predicted code matches the
original one or not). This is a boolean metric where 1 is attributed to a
matching prediction and 0 otherwise. Intuitively, we believe that the more the
code is natural, the more CodeBERT predictions are accurate.

Finally, each line is mapped with its prediction-scores forming a matrix
$\mathbf{M}\in\mathbb{R}^{3\times n}$ where $n$ is the number of collected
propositions in that source-code line.

The AST parsing and node-location extraction part has been implemented in Java
and uses Spoon [27], a Java code-source analysis library. Whereas, the rest of
the process has been implemented in python and uses the CodeBERT Masked
Language Modeling (MLM) task to predict the masked codes.

####  IV-B2 n-gram ranking

To compare CodeBERT with n-gram models in terms of code naturalness aspect
capturing, we proceed as follows:

For every considered issue, we train two n-gram models specific to that
version of the project using two distinct tokenization techniques111We use
UTFLineTokenizer and JavaLemmeLineTokenizer available in Tuna [28] GitHub
repository under tokenizer/line/
(branch=master,repo=https://github.com/electricalwind/tuna, rev-id=44188e1) :

  * •

Java Paraser tokenizer (noted JP) which tokenizes the code according to the
Java grammar, and thus, discarding empty lines as well as java-doc and code-
comment lines.

  * •

UTF8 tokenizer (noted UTF8), which operates on the full raw representation of
the source-code.

We name the created models based on their underlying tokenization technique,
JP and UTF8 n-gram. In the training phase, we use all the lines from the files
that have not been changed by the fix commit, then we use each of these models
to attribute a cross-entropy score to the subject lines of the buggy files,
counting buggy and neutral lines as detailed in Sub-section IV-A2. Finally, we
rank the lines according to their cross-entropy score in a descendant order
such-as high values of cross-entropy are associated with less code naturalness
and higher likelihood of bugginess.

To run this experiment, we use the current version of the n-gram utilities
library Tuna [28] with one of the recommended configurations by Jimenez et al.
[10] for distinguishing buggy and fixed lines: 4 as n-order, 1 as unknown
threshold and Kneser Ney smoothing (KN). Note that we use KN instead of the
Modified Kneser Ney smoothing (MKN) because it is not suited for short
sequences.

###  IV-C Lines ranking

To assess the relevance of the information inferred from the CodeBERT
predictions, we rely on its ability to rank the buggy lines before the
supposed neutral ones. As we do not have any labelling of ”natural” and
”unnatural” source-code dataset and based on the naturalness hypothesis which
assumes that ”not natural lines are more likely to be buggy”, we believe that
the prediction variation of CodeBERT under naturalness variation of the input
source-code can be observed through its ranking of buggy and not buggy lines.

For every considered approach, metric and aggregation method considered in our
experiments, we rank all the lines by bug first, then normalise the ranks by
the total number of studied lines for that bug. We report two ranking results
per bug per approach:

  * •

the first hit: corresponds to the rank of the first-ranked buggy line.

  * •

mean: corresponds to the mean ranks of all buggy lines.

To cut ties when multiple lines share the same score, we attribute the
estimated rank by a uniform random selection. For instance, if we have 100
lines sharing the same rank, among which 3 lines are buggy, the random first
hit rank will be equal to 25, while the mean rank of the 3 buggy lines will be
50.

To check whether any of the 3 aforementioned metrics are impacted by the
naturalness variance of the source-code (answer to RQ1), we generate one
CodeBERT prediction only by masked token. Then we aggregate the scores of each
line’ predictions by applying one of the following aggregation metrics:
minimum, maximum, mean, median and entropy. Where the entropy is calculated as
the following:

$\text{H}(l,m)=-\frac{\sum_{i=1}^{n}\log(s_{i})}{n}$ (3)

where $\\{s_{1},s_{2},...,s_{n}\\}$ denotes the set of $n$ scores attributed
to the line $l$ for the metric $m$.

We then rank the subject lines by each of the metrics using the different
aggregation methods and following both sorting directions: ascending and
descending. As most of the results are close to each other, we calculate the
paired Vargha and Delaney $\hat{\text{A}}_{12}$ ratios, to conclude which
combinations are the best in terms of buggy lines ranking.

To check whether the extracted naturalness information from CodeBERT
predictions (answer RQ2), we compare the ranking results of the 3 metrics
using their best performing aggregation method and sorting order, with the
rankings results of random and complexity-based rankings. Where, the
complexity of a line corresponds to its number of Java Parser tokens. This is
inspired from the study of Leszak et al. [29], where complex source code has
been proven to be more likely to be buggy. For the random ranking, instead of
rerunning the ranking multiple times, we simply used a basic probability
calculation of the rankings.

Finally, to compare the effectiveness of CodeBERT-nt with similar techniques
in capturing code naturalness we compare its buggy-line rankings with the
n-gram cross-entropy, measured as described in Sub-section IV-B2 (answer to
RQ3). For this comparison, we consider the CodeBERT-nt rankings effectuated by
its best performing metric, selected from the previous research questions.

To have a better understanding on the differences significance we run
statistical tests of Wilkixon and Vargha and Delaney $\hat{\text{A}}_{12}$ on
all of our comparison results.

###  IV-D Threats to Validity

The question of whether our findings generalise, forms a typical threat to
validity of empirical studies. To reduce this threat, we used real-world
projects, real faults and their associated commits, from an established and
independently built benchmark. Still, we have to acknowledge that these may
not be representative of projects from other programming languages, domains or
industrial systems.

Other threats may also arise from the assumption that all changed lines by the
fix commits are buggy lines. While our heuristics are the standard in the
literature, we believe that this selection process is sufficient given that we
have used a dataset where the fix commits have been manually untangled.
Additionally, we focus our study on the sole business logic lines, thus
reducing further the risk of considering as buggy, lines irrelevant to the bug
at hand.

Finally, our evaluation metrics may induce some additional threats. Our
comparison basis measurement, i.e., comparing the ranking of source-code lines
that has been trained on the same source-code’s project with approaches that
are agnostic and has been trained on multiple projects. It is hard to compare
the advantage gained by training CodeBERT on a large number of projects,
including eventually the projects in our dataset, against the advantage gained
by the n-gram models when trained on a specific project source-code and
predicting the naturalness of lines of that same project.

##  V Results

###  V-A RQ1: Metrics and aggregation methods

TABLE I: Paired Vargha and Delaney $\hat{\text{A}}_{12}$ effect size values of
the buggy lines ranking by different pairs of metrics-aggregation methods in
ascendant and descendant order. cos refers to CBnt_cos (the Cosine similarity
between the embeddings of the original code and the predicted one), conf
refers to CBnt_conf (the CodeBERT-confidence that the predicted code is
probably correct) and the acc metric refers to CBnt_acc (whether the predicted
code matches the original one).

Metric | conf min | conf max | conf mean | conf median | conf entropy | cos min | cos max | cos mean | cos median | cos entropy | acc min | acc max | acc mean | acc median
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
1st hit | 0.6196 | 0.4993 | 0.5528 | 0.5542 | 0.4387 | 0.5084 | 0.4763 | 0.5016 | 0.4989 | 0.4980 | 0.5004 | 0.4996 | 0.5118 | 0.4996
Mean | 0.6325 | 0.5009 | 0.5613 | 0.5530 | 0.4350 | 0.5138 | 0.4818 | 0.5027 | 0.5058 | 0.4980 | 0.5004 | 0.4996 | 0.5122 | 0.4996

Figure 2: Buggy lines ranking using 1 prediction per token and the three
available metrics with different aggregation metrics. The less CodeBERT is
confident when predicting, the more likely the line is to be buggy.

To evaluate the CodeBERT-nt metrics, we ranked the subject lines according to
the aforementioned metrics. We start by calculating one score per line for
every metric, by aggregating the scores of the predictions in that line. Then,
we sort the lines according to their score in both orders - ascendant and
descendant - and calculate the paired (by bug) Vargha and Delaney
$\hat{\text{A}}_{12}$ difference between both orders effectiveness in terms of
attributing the lowest ranks to the buggy lines; more precisely the average
ranking of the buggy lines and the smallest rank attributed to a buggy line,
per bug. (Please refer to section IV for details)

These results are depicted in Table I where values around 0,5 depict that the
considered pair of metric-aggregation method does not bring any advantage in
ranking buggy lines first, while values above 0,5 confirm an advantage for the
ascendant sorting order and below 0,5 an advantage for the descendant one.

Figure 2 depicts the box-plots of the normalised rankings by number of lines
of each bug. Interestingly, the prediction confidence score seems to be a good
indicator of naturalness and thus of bugginess likelihood. Noticeably, low
confidence scores are more often attributed to buggy lines than neutral ones.
This explains why aggregating this metric by using the maximum value by line
gives the worst guidance to our target, while sorting by the minimum score
gives the best ranking over all considered pairs metric-aggregation.

At the same time, we do not observe any relevant differences when ranking the
lines by either the cosine of the embedding or the correctness of predictions.
This is also confirmed by the Table I where most of the $\hat{\text{A}}_{12}$
values are around 0,5 when using each of the aggregation methods on these two
metrics. Nevertheless, we notice that the best method to rank the buggy lines
by cosine similarity is via ranking the lines by their highest scores of
similarity, where the high (max) values are considered as symptoms of
unnaturalness. We also notice a trend that confirms the correlation between
naturalness and code predictability, such as the lower the mean of correct
predictions in a code is, the less natural is the code.

By checking Table I we also see that the difference of ranking the lines by
low confidence, from the other metric-aggregation methods is significant,
Vargha and Delaney $\hat{\text{A}}_{12}$ effect size values are more than 0.6,
which are significantly higher to the rest of confidence aggregation rankings.
Less noticeable, the ranking by increase of the line’s maximum embeddings
cosine-simalirity and mean prediction accuracy yield respectively
$\hat{\text{A}}_{12}$ values of around 0,52 and 0,51 when compared to the
other studied aggregation methods when applied on the same metrics.

CodeBERT-nt can infer the naturalness of code through masked token
predictions. The unnatural information can be inferred the best from the
decrease of prediction confidence (considering the minimum value per line),
the increase of cosine similarity between the embedded original and predicted
code (considering the maximum value per line) and the decrease of average
prediction accuracy per line.

###  V-B RQ2: Comparison with random and complexity based rankings

Figure 3: Comparison of the buggy lines rankings by CodeBERT, Random and
Complexity (number of tokens by line). CodeBERT outperforms Random and
Complexity in ranking buggy lines.

Figure 3 shows the distribution of the normalised rankings of the first ranked
buggy line and the average rank of buggy lines by bug when using CodeBERT-nt
metric-aggregation pairs selected from the results of RQ1 (ascendant minimum
CBnt_conf, descendent maximum CBnt_cos and ascendant mean CBnt_acc), uniform
random ranking and token-number-complexity descendant ranking.

Surprisingly, random and complexity lead to similar rankings with a small
advantage for random. This observation implies that tokens-number-complexity
does not capture well the code naturalness in the studied setup: naturalness
on the line-level-granularity of the business-logic code.

As can be seen from the boxplots, CodeBERT-nt outperforms the baseline
techniques in ranking the buggy lines first, using any of its three metrics.
In fact, except for a small portion of our dataset bugs, CodeBERT-nt low-
confidence performs the best in estimating the bugginess of the target lines,
with respectively a 1st hit and mean buggy-line ranks lower in average by
around 6% and 4.7% than the following ranks – attributed by the cosine
similarity – and around 11% and 5% lower than the least performing ranking
effectuated by uniform random. This noticeable difference between the
CodeBERT-nt results with the two considered techniques, more specifically the
low confidence CBnt_conf ranking, endorses the fact that CodeBERT-nt can be
considered as a comparison baseline and a new method for naturalness based
tasks.

Although, the two remaining CodeBERT-nt metrics, average accuracy CBnt_acc and
high embeddings similarity CBnt_cos, outperform both baselines and yield lower
rankings than uniform random by respectively 3% and 5% in ranking the first
hit buggy line, they perform however similarly to the baselines when comparing
their mean ranking of all the buggy lines, with a small advantage of only 0,4%
for the ranking by embeddings similarity.

TABLE II: Vargha and Delaney $\hat{\text{A}}_{12}$ of (CodeBERT-nt low-confidence-metric rankings compared to the other ones.)  CBnt_conf Vs | CBnt_cos | CBnt_acc | Complexity | Random
---|---|---|---|---
1st hit | 0.578 | 0.609 | 0.605 | 0.607
mean | 0.565 | 0.619 | 0.620 | 0.622

To validate this finding, we perform a statistical test (Wilcoxon paired test)
on the data of Figure 3 to check for significant differences. Our results
showed that the differences are significant, indicating the low probability of
this effect to be happening by chance. As illustrated in Table II, the size of
the difference is also big, with CodeBERT-nt low-confidence yielding Vargha
and Delaney $\hat{\text{A}}_{12}$ values between 0.58 and 0.6 indicating that
CBnt_conf ranks the buggy lines the best in the great majority of the cases.
However, the CodeBERT-nt accuracy and embeddings-similarity metrics outperform
random respectively, in only 51% and 48% of the cases.

CodeBERT-nt metrics describe source-code naturalness more accurately than the
baselines uniform-random-selection and tokens-count-complexity based rankings.
CodeBERT-nt low confidence – CBnt_conf – is the most effective metric and
outperforms the uniform-random-ranking by 11% in ranking the first hit buggy
line and 5% in ranking all the buggy lines, in average.

###  V-C RQ3: Comparison with n-gram

Figure 4: Comparison of the buggy lines rankings by CodeBERT, UTF8 n-gram and
JP n-gram models (created respectively using UTF8 and Java Parser tokenizers).
CodeBERT ranking is comparable to the n-gram models one.

To answer this question, we train two n-gram models per buggy version of our
dataset that we then use to compute the cross-entropies of the subject lines
corresponding to the corresponding bug, then we rank these lines according to
the resulting values and reproduce the same analysis as in RQ2. We illustrate
in Figure 4 the distribution of the normalised rankings of the first ranked
buggy line and the average rank of buggy lines by bug when using CodeBERT-nt
low confidences ranking – CBnt_conf –, the descendant cross-entropy ranking
from a UTF8-tokenizer-based n-gram model and a JavaParser-tokenizer-based one.
Additionally, we illustrate the random ranking in the boxplots as the simplest
baseline for this task.

As expected, the three approaches outperform the uniform-random-ranking in
most of the cases and yield very comparable results with a small advantage to
CodeBERT-nt on ranking the first buggy line over the n-gram techniques and a
small advantage to JP n-gram in regards of the average rank of buggy lines. In
both comparisons, UTF8 n-gram falls slightly behind these two latter
techniques.

TABLE III: Vargha and Delaney $\hat{\text{A}}_{12}$ of CodeBERT-nt low-confidence-metric rankings compared to n-gram and Random ones. Approaches $\hat{\text{A}}_{12}$ | Random | UTF8 | JP
---|---|---|---
1st hit | mean | 1st hit | mean | 1st hit | mean
CBnt_conf | 0.606 | 0.621 | 0.536 | 0.502 | 0.518 | 0.468
JP | 0.575 | 0.630 | 0.559 | 0.588 |  |
UTF8 | 0.560 | 0.602 |  |  |  |

To validate this finding, we performed a similar statistical test as in RQ2 on
the data of Figure 4 and found that the differences with random are
significant, while the differences between n-gram cross-entropy and CodeBERT-
nt low-confidence rankings are negligible. As illustrated in Table III, the
size of the $\hat{\text{A}}_{12}$ differences are equal to 0.518 and 0.468
between CodeBERT-nt and JP n-gram models and 0.536 and 0.502 for both reported
rankings, meaning that both approaches yield comparable results, which
confirms that CodeBERT-nt can carry out code naturalness related applications
with similar effectiveness as the statistical language n-gram models.

CodeBERT-nt masked token prediction confidence indicates naturalness of bugs
as accurately (slightly better) as program-specific n-gram models.

##  VI Discussion

###  VI-A Impact of interesting lines selection

Figure 5: Comparison of the buggy lines rankings by CodeBERT, UTF8 n-gram and
JP n-gram models (created respectively using UTF8 and Java Parser tokenizers)
when ranking all lines. CodeBERT ranking is comparable to the n-gram models
one and ranking the business-logic source-code lines first give it an
advantage over the n-gram ranking.

Our empirical results show evidence that CodeBERT-nt can infer source-code
naturalness yielding the same results as n-gram models and outperforming the
uniform-random and code-complexity based techniques in attributing higher
ranks to buggy lines when sorting source-code by naturalness. These
experiments have been driven on the same buggy versions source-code whose
lines count at least one buggy line. Precisely, we have excluded all the lines
outside the business-logic source-code and kept only the bugs that counted at
least one buggy line within their remaining lines.

To better understand the impact of this line selection step, we reintroduce
all the bugs with their full source-code in our dataset and reproduce the same
study as in RQ3. We have then attributed the worst rank to all unranked lines
by CodeBERT-nt, i.e., outside of the business-logic. This implies that for
CodeBERT-nt, the business-logic lines are ranked first by their min-confidence
and the remaining lines are ranked after with a random uniform selection
logic. The n-gram approaches ranking is applied as described previously, the
same way on all lines – business- and non-business-logic related ones – having
each two cross-entropy values from every corresponding n-gram model. The
ranking distributions are illustrated in Figure 5.

Although the results of the three approaches remain comparable, the trend is
that for a noticeable portion of studied bugs, CodeBERT-nt remains able to
rank buggy lines better than the n-gram models. The difference is wider and
more visible between the ranks of the first buggy line which can be seen in
the left boxplot of Figure 5. Interestingly, we observe that ranking the
business-logic lines with CodeBERT-nt and the remaining lines with a uniform
random ranking outperforms ranking all the lines (business- and not business-
logic related ones) by their n-gram calculated cross-entropies. These results
lead to the conclusion that the naturalness analysis of the non-business-logic
lines do not contribute with useful information to the considered ranking
tasks, but instead alters its results when attributing a higher rank to the
targeted lines.

To check whether this observed decrease of performance for n-gram is indeed
caused by the additional lines or because they performed worst on the
previously excluded bugs from our dataset, we reproduce the same comparison on
the subset of our dataset where all bugs are located outside of the business-
logic code, implying that CodeBERT-nt will attribute the worst score to every
buggy line. We illustrate the rankings distribution of the ”worst-score”
strategy (ranking all buggy-lines last), JP and UTF8 n-gram models and
uniform-random in Figure 6.

Although small, the n-gram models kept some advantage over random ranking as
in the Figures 4 and 5, in contrast to CodeBERT-nt’s ”worst-score” results.The
contrasting results between the Figures 6 and 5 highlight the negative impact
of ranking the not business-logic lines by naturalness as they compensated
CodeBERT-nt’s disadvantage of attributing the worst ranks to buggy lines, in
10% of the studied cases. Consequently, these results reinforce our conclusion
that including the non-business-logic lines in the analysis adds noise to the
search-space [25], and consequently hinders the ranking accuracy.

Figure 6: Comparison of the buggy lines rankings by worst possible scores,
UTF8 n-gram and JP n-gram models (created respectively using UTF8 and Java
Parser tokenizers) when targeting the buggy versions not exposing any
business-logic-related buggy line. n-gram techniques perform similarly to
random on these subject buggy versions.

###  VI-B Which metric to use for which bug?

Figure 7: Which metric ranks the best the buggy lines, in most of the cases?
CodeBERT confidence CBnt_conf performs the best for around 50% of the cases,
followed by CBnt_cos then CBnt_acc which perms almost similarly to random.
Therefore, There’s no big benefit in using CBnt_acc while it could be
interesting to complement CBnt_conf capabilities, using CBnt_cos.

(a) Scatterplot of bugs best-ranking-metric by standard deviation.

(b) Histogram of bugs best-ranking-metric by standard deviation.

Figure 8: Distribution of Bugs best-ranking-metric by standard deviation of
the metrics measured on their corresponding subject lines. Except for few
scores, CBnt_conf ranks the majority of the bugs the best independently from
the measured SDs.

.

The empirical results driven on our large set of buggy versions highlight the
effectiveness of CodeBERT-nt in capturing the naturalness of source-code,
especially via its low-confidence metric. However, from the outliers in the
Figures 2, 3, 4 and 5, we notice that CodeBERT-nt does not perform equally on
all considered bugs. Which means for instance that it outperforms uniform-
random ranking in the majority of the cases, but yields worst rankings for a
small portion of the dataset. Therefore, we turn our attention towards
investigating the possibility of better handling those bugs, leveraging one of
the other CodeBERT-nt metrics.

We start by mapping every metric with the bugs on which it attributed the best
mean ranking to the buggy lines. In figure 7, we illustrate a Venn diagram of
this distribution, including uniform-random-ranking as baseline. As shown in
our results, the CodeBERT-nt confidence is the best naturalness indicator for
the majority of bugs, followed by the cosine similarity and the prediction
accuracy. We also notice that except for a minority of 20 bugs, at least one
of the CodeBERT-nt metrics achieves better scores or similar to random-uniform
ranking. Additionally, except the large intersection set of bugs that are
best-ranked by the prediction accuracy and uniform-random-ranking, the metrics
rarely achieve their best rankings of the buggy lines for the same bugs. This
observation introduces the hypothesis that the metrics are complementary and
eventually, one could rely on different metrics for different bugs.

Aiming at distinguishing between our dataset bugs, we measure the metrics
standard deviation (SD) per studied lines. In this setup, we exclude from our
clusters the bugs that are intersecting with random and plot the SD
distributions of the remaining ones, in Figure 8(b). The plots illustrate that
the bugs from different clusters share similar ranges of SD with mean values
around 0.27, 0.09 and 0.31 for respectively CBnt_conf, CBnt_cos and CBnt_acc.
Also in the majority of the cases, the SD of the bugs best treated by other
metrics than CBnt_conf fall in the same range of this latter, thus, cannot be
distinguished from each other. However, we notice that for some SD values,
CBnt_cos ranks better more bugs than CBnt_conf. This difference is noticeable
for roughly: $SD(CBnt\\_conf{})$ values between 0.2 and 0.24 or above 0.35,
$SD(CBnt\\_cos{})$ values above 0.14 and $SD(CBnt\\_acc{})$ values below 0.05
or between 0.19 and 0.28 or between 0.395 and 0.42. These SD ranges represent
a small fraction of our dataset, exactly 357 bugs, among which 193 where
CBnt_cos performed the best and 200 where it outperformed CBnt_conf, which
correspond respectively to 15.8%, 8.5% and 8.8% of the studied bugs.
Nevertheless, these results may motivate future investigation on the use of
CBnt_cos over CBnt_conf in similar cases, on different setups.

###  VI-C Impact of generating more predictions per token?

Figure 9: Buggy lines ranking using 1, 2, 3, 4 and 5 predictions per token.
The more predictions we use, the more the information about the confidence
gets dissipated, thus the more the ranking performance decreases, except when
considering the accuracy metric.

To have a better understanding on whether generating more predictions from the
model could improve the bugginess information retrieved by CodeBERT-nt, we
extend our experiment of RQ1 by comparing the ranking results using the best
proven pairs of metric-aggregation from our results, when generating 1, 2, 3,
4 and 5 predictions per token. We illustrate in Figure 9 the box-plots of the
normalised rankings by number of lines of each bug. Although comparable, the
results depict a clear dissipation of the bugginess indicators retrieved from
the prediction confidence and the cosine similarity metric, when we aggregate
the values of more than one prediction by token. In the other hand, we can see
that the ranking performance effectuated by the prediction accuracy raises
when we consider 2 and 3 predictions then converges to a stable value. Besides
the fact that this increase confirms further the correlation between code-
naturalness and predictability, it remains negligible and keeps this metric-
ranking far below the low-confidence one. Therefore, we believe that it would
be more cost-efficient and appealing for similar studies, to generate only 1 –
eventually up to 3 – predictions instead of 5, as the default setting of
CodeBERT.

##  VII Conclusion

Naturalness of code forms an important attribute often needed by researchers
when building automated code analysis techniques. However, computing the
naturalness of code using n-gram requires significant amount of work and a
salable infrastructure that is not often available. An alternative solution is
to use other readily available language models, perhaps more powerful than
n-grams, such as transformer-based generative models (CodeBERT-like).
Unfortunately, these models do not offer any token-based appearance
estimations since they aim at generating tokens rather than computing their
likelihood. To this end, we investigate the use of predictability metrics, of
code tokens using the CodeBERT model and check their appropriateness in bug
detection. Our results show that computing the confidence of the model when
masking and generating a token, irrespective of whether the predicted token is
the one that was actually predicted by the model, offers the best results,
which are comparable (slightly better) to that of n-gram models trained on the
code of the same project (intra-project predictions).

## References

  * [1] A. Hindle, E. T. Barr, Z. Su, M. Gabel, and P. Devanbu, “On the naturalness of software,” in _Proceedings of the 34th International Conference on Software Engineering_ , ser. ICSE ’12. IEEE Press, 2012, p. 837–847.
  * [2] M. Allamanis and C. Sutton, “Mining source code repositories at massive scale using language modeling,” in _2013 10th working conference on mining software repositories (MSR)_. IEEE, 2013, pp. 207–216.
  * [3] C. E. Shannon, “Prediction and entropy of printed english,” _The Bell System Technical Journal_ , vol. 30, no. 1, pp. 50–64, 1951.
  * [4] ——, “A mathematical theory of communication,” _The Bell System Technical Journal_ , vol. 27, no. 3, pp. 379–423, 1948.
  * [5] T. Sharma, V. Efstathiou, P. Louridas, and D. Spinellis, “Code smell detection by deep direct-learning and transfer-learning,” _Journal of Systems and Software_ , vol. 176, p. 110936, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0164121221000339
  * [6] B. Lin, C. Nagy, G. Bavota, and M. Lanza, “On the impact of refactoring operations on code naturalness,” in _2019 IEEE 26th International Conference on Software Analysis, Evolution and Reengineering (SANER)_ , 2019, pp. 594–598.
  * [7] D. Posnett, A. Hindle, and P. Devanbu, “Reflections on: A simpler model of software readability,” _SIGSOFT Softw. Eng. Notes_ , vol. 46, no. 3, p. 30–32, jul 2021. [Online]. Available: https://doi.org/10.1145/3468744.3468754
  * [8] V. J. Hellendoorn, P. T. Devanbu, and A. Bacchelli, “Will they like this? evaluating code contributions with language models,” in _2015 IEEE/ACM 12th Working Conference on Mining Software Repositories_ , 2015, pp. 157–167.
  * [9] S. Wang, D. Chollak, D. Movshovitz-Attias, and L. Tan, “Bugram: Bug detection with n-gram language models,” in _Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering_ , ser. ASE 2016\. New York, NY, USA: Association for Computing Machinery, 2016, p. 708–719. [Online]. Available: https://doi.org/10.1145/2970276.2970341
  * [10] M. Jimenez, C. Maxime, Y. Le Traon, and M. Papadakis, “On the impact of tokenizer and parameters on n-gram based code analysis,” in _2018 IEEE International Conference on Software Maintenance and Evolution (ICSME)_. IEEE, 2018, pp. 437–448.
  * [11] B. Ray, V. Hellendoorn, S. Godhane, Z. Tu, A. Bacchelli, and P. Devanbu, “On the ”naturalness” of buggy code,” in _2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE)_ , 2016, pp. 428–439.
  * [12] T. Copeland, _PMD applied_. Centennial Books San Francisco, 2005, vol. 10.
  * [13] D. Hovemeyer and W. Pugh, “Finding more null pointer bugs, but not too many,” in _Proceedings of the 7th ACM SIGPLAN-SIGSOFT workshop on Program analysis for software tools and engineering_ , 2007, pp. 9–14.
  * [14] S. H. Alexander Trautsch, Fabian Trautsch, “The smartshark repository mining data,” 2021.
  * [15] T. T. Chekam, M. Papadakis, T. F. Bissyandé, Y. L. Traon, and K. Sen, “Selecting fault revealing mutants,” _Empir. Softw. Eng._ , vol. 25, no. 1, pp. 434–487, 2020. [Online]. Available: https://doi.org/10.1007/s10664-019-09778-7
  * [16] “Javaparser,” Available on https://github.com/javaparser/javaparser, https://javaparser.org/.
  * [17] S. F. Chen and J. Goodman, “An empirical study of smoothing techniques for language modeling,” _Computer Speech & Language_, vol. 13, no. 4, pp. 359–394, 1999. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0885230899901286
  * [18] R. Kneser and H. Ney, “Improved backing-off for m-gram language modeling,” in _1995 International Conference on Acoustics, Speech, and Signal Processing_ , vol. 1, 1995, pp. 181–184 vol.1.
  * [19] Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang, and M. Zhou, “Codebert: A pre-trained model for programming and natural languages,” in _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, EMNLP 2020, Online Event, 16-20 November 2020_ , ser. Findings of ACL, T. Cohn, Y. He, and Y. Liu, Eds., vol. EMNLP 2020. Association for Computational Linguistics, 2020, pp. 1536–1547. [Online]. Available: https://doi.org/10.18653/v1/2020.findings-emnlp.139
  * [20] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” _Advances in neural information processing systems_ , vol. 30, 2017.
  * [21] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _arXiv preprint arXiv:1810.04805_ , 2018.
  * [22] “Codebert,” https://github.com/microsoft/CodeBERT.
  * [23] M. K. Thota, F. H. Shajin, and P. Rajesh, “Survey on software defect prediction techniques,” _International Journal of Applied Science and Engineering_ , vol. 17, pp. 331–344, December 2020.
  * [24] M. Allamanis, E. T. Barr, P. T. Devanbu, and C. Sutton, “A survey of machine learning for big code and naturalness,” _CoRR_ , vol. abs/1709.06182, 2017\. [Online]. Available: http://arxiv.org/abs/1709.06182
  * [25] M. Rahman, D. Palani, and P. C. Rigby, “Natural software revisited,” in _2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE)_ , 2019, pp. 37–48.
  * [26] Z. Sun, J. M. Zhang, Y. Xiong, M. Harman, M. Papadakis, and L. Zhang, “Improving machine translation systems via isotopic replacement,” in _2022 IEEE/ACM 44th International Conference on Software Engineering (ICSE)_ , 2022, pp. 1181–1192.
  * [27] R. Pawlak, M. Monperrus, N. Petitprez, C. Noguera, and L. Seinturier, “Spoon: A Library for Implementing Analyses and Transformations of Java Source Code,” _Software: Practice and Experience_ , vol. 46, pp. 1155–1179, 2015\. [Online]. Available: https://hal.archives-ouvertes.fr/hal-01078532/document
  * [28] M. Jimenez, C. Maxime, Y. Le Traon, and M. Papadakis, “Tuna: Tuning naturalness-based analysis,” in _2018 IEEE International Conference on Software Maintenance and Evolution (ICSME)_ , 2018, pp. 715–715.
  * [29] M. Leszak, D. E. Perry, and D. Stoll, “Classification and evaluation of defects in a project retrospective,” _Journal of Systems and Software_ , vol. 61, no. 3, pp. 173–187, 2002. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0164121201001467


11institutetext: 1Department of Computer Science and Engineering,
The Chinese University of Hong Kong, Hong Kong, China
11email<EMAIL_ADDRESS>
2Department of Computer Science and Engineering,
The Hong Kong University of Science and Technology, Hong Kong, China

# Pseudo Bias-Balanced Learning for Debiased Chest X-ray Classification

Luyang Luo 11 Dunyuan Xu 11 Hao Chen 22
Tien-Tsin Wong 11 Pheng-Ann Heng 11

###### Abstract

Deep learning models were frequently reported to learn from shortcuts like
dataset biases. As deep learning is playing an increasingly important role in
the modern healthcare system, it is of great need to combat shortcut learning
in medical data as well as develop unbiased and trustworthy models. In this
paper, we study the problem of developing debiased chest X-ray diagnosis
models from the biased training data without knowing exactly the bias labels.
We start with the observations that the imbalance of bias distribution is one
of the key reasons causing shortcut learning, and the dataset biases are
preferred by the model if they were easier to be learned than the intended
features. Based on these observations, we proposed a novel algorithm, pseudo
bias-balanced learning, which first captures and predicts per-sample bias
labels via generalized cross entropy loss and then trains a debiased model
using pseudo bias labels and bias-balanced softmax function. We constructed
several chest X-ray datasets with various dataset bias situations and
demonstrated with extensive experiments that our proposed method achieved
consistent improvements over other state-of-the-art approaches.111Code
available at https://github.com/LLYXC/PBBL.

###### keywords:

Debias; Shortcut Learning; Chest X-ray

##  1 Introduction

To date, deep learning (DL) has achieved comparable or even superior
performance to experts on many medical image analysis tasks [17]. Robust and
trustworthy DL models are hence of greater need than ever to unleash their
huge potential in solving real-world healthcare problems. However, a common
trust failure of DL was frequently found where the models reach a high
accuracy without learning from the intended features. For example, using
backgrounds to distinguish foreground objects [19], using the gender to
classify hair colors [20], or worse yet, using patients’ position to determine
COVID-19 pneumonia from chest X-rays [4]. Such a phenomenon is called
_shortcut learning_ [5], where the DL models choose unintended features, or
_dataset bias_ , for making decisions.

More or less, biases could be generated during the creation of the datasets
[22]. As the dataset biases frequently co-occurred with the primary targets,
the model might take shortcuts by learning from such spurious correlation to
minimize the empirical risk over the training data. As a result, dramatic
performance drops could be observed when applying the models onto other data
which do not obtain the same covariate shift [13]. In the field of medical
image analysis, shortcut learning has also been frequently reported, such as
using hospital tokens to recognize pneumonia cases [25]; learning confounding
patient and healthcare variables to identify fracture cases; relying on chest
drains to classify pneumothorax case [16]; or leveraging shortcuts to
determine COVID-19 patients [4]. These findings reveal that shortcut learning
makes deep models less explainable and less trustworthy to doctors as well as
patients, and addressing shortcut learning is a far-reaching topic for modern
medical image analysis.

To combat shortcut learning and develop debiased models, a branch of previous
works use data re-weighting to learn from less biased data. For instance,
REPAIR [12] proposed to solve a minimax problem between the classifier
parameters and dataset re-sampling weights. Group distributional robust
optimization [20] prioritized worst group learning, which was also mianly
implemented by data re-weighting. Yoon et al. [24] proposed to address dataset
bias with a weighted loss and a dynamic data sampler. Another direction of
works emphasizes learning invariance across different environments, such as
invariant risk minimization [1], contrastive learning [21], and mutual
information minimization [29]. However, these methods all required dataset
biases to be explicitly annotated, which might be infeasible for realistic
situations, especially for medical images. Recently, some approaches have made
efforts to relax the dependency on explicit bias labels. Nam et al. [15]
proposed to learn a debiased model by mining the high-loss samples with a
highly-biased model. Lee et al. [11] further incorporated feature swapping
between the biased and debiased models to augment the training samples. Yet,
very few methods attempted to efficiently address shortcut learning in medical
data without explicitly labeling the biases.

In this paper, we are pioneered in tackling the challenging problem of
developing debiased medical image analysis models without explicit labels on
the bias attributes. We first observed that the imbalance of bias distribution
is one of the key causes to shortcut learning, and dataset biases would be
preferred when they were easier to be learned than the intended features. We
thereby proposed a novel algorithm, namely pseudo bias-balanced learning
(PBBL). PBBL first develops a highly-biased model by emphasizing learning from
the easier features. The biased model is then used to generate pseudo bias
labels that are later utilized to train a debiased model with a bias-balanced
softmax function. We constructed several chest X-ray datasets with various
bias situations to evaluate the efficacy of the debiased model. We
demonstrated that our method was effective and robust under all scenarios and
achieved consistent improvements over other state-of-the-art approaches.

##  2 Methodology

###  2.1 Problem Statement and Study Materials

Let $X$ be the set of input data, $Y$ the set of target attributes that we
want the model to learn, and $B$ the set of bias attributes that are
irrelevant to the targets. Our goal is to learn a function $f:X\rightarrow Y$
that would not be affected by the dataset bias. We here built the following
chest X-ray datasets for our study.

Source-biased Pneumonia (SbP):  For the training set, we first randomly
sampled 5,000 pneumonia cases from MIMIC-CXR [8] and 5,000 healthy cases (no
findings) from NIH [23]. We then sampled $5,000\times r\%$ pneumonia cases
from NIH and the same amount of healthy cases from MIMIC-CXR. Here, the data
source became the dataset bias, and health condition was the target to be
learned. We varied $r$ to be 1, 5, and 10, which led to biased sample ratios
of 99%, 95%, and 90%, respectively. We created the validation and the testing
sets by equally sampling 200 and 400 images from each group (w/ or w/o
pneumonia; from NIH or MIMIC-CXR), respectively. Moreover, as overcoming
dataset bias could lead to better external validation performance [5], we
included 400 pneumonia cases and 400 healthy cases from Padchest [2] to
evaluate the generalization capability of the proposed method. Note that we
converted all images to JPEG format to prevent the data format from being
another dataset bias.

Gender-biased Pneumothorax (GbP):  Previous study [10] pointed out that gender
imbalance in medical datasets could lead to a biased and unfair classifier.
Based on this finding, we constructed two training sets from the NIH dataset
[23]: 1) GbP-Tr1: 800 male samples with pneumothorax, 100 male samples with no
findings, 800 female samples with no findings, and 100 female samples with
pneumothorax; 2)GbP-Tr2: 800 female samples with pneumothorax, 100 female
samples with no findings, 800 male samples with no findings, and 100 male
samples with pneumothorax. For validation and testing sets, we equally
collected 150 and 250 samples from each group (w/ or w/o pneumothorax; male or
female), respectively. Here, gender became a dataset bias and health condition
was the target that the model was aimed to learn.

Following previous studies [15, 11], we call a sample bias-aligned if its
target and bias attributes are highly-correlated in the training set (e.g.,
(pneumonia, MIMIC-CXR) or (healthy, NIH) in the SbP dataset). On the contrary,
a sample is said to be bias-conflicting if the target and bias attributes are
dissimilar to the previous situation (e.g., (pneumonia, NIH) or (healthy,
MIMIC-CXR)).

(a)

(b)

(c)

Figure 1: We show (a) the testing results in AUC curves of a model trained on
Source-biased Pneumonia dataset; (b) the testing results in AUC curves of a
model trained with bias-balanced pneumonia dataset. We further show our causal
assumption of data generation process in (c). Blue curves: results on bias-
aligned samples; Red curves: results on bias-conflicting samples; Yellow
curves: averaged results of bias-aligned AUC and bias-conflicting AUC.

###  2.2 Bias-balanced Softmax

Our first observation is that _bias-imbalanced training data leads to a biased
classifier_. Based on the SbP dataset, we trained two different settings: i)
SbP with $r=10$; ii) Bias balancing by equally sampling 500 cases from each
group. The results are shown in Fig. 1(a) and Fig. 1(b), respectively.
Clearly, when the dataset is bias-imbalanced, learning bias-aligned samples
were favored. On the contrary, balancing the biases mitigates shortcut
learning even with less training data.

For a better interpretation, we adopt the causal assumption [14] that the data
$X$ is generated from both the target attributes $Y$ and the bias attributes
$B$, which are independent to each other, as shown in Fig. 1(c). The
conditional probability $p(y=j|x)$ hence can be formalized as follows:

$p(y=j|x,b)=\frac{p(x|y=j,b)p(y=j|b)}{p(x|b)},$ (1)

where $p(y=j|b)$ raises a distributional discrepancy between the biased
training data and the ideal bias-balanced data (e.g., the testing data).
Moreover, according to our experimental analysis before, the imbalance also
made the model favor learning from bias-aligned samples, which finally
resulted in a biased classifier. To tackle the bias-imbalance situation, let
$k$ be the number of classes and $n_{j,b}$ the number of training data of
target class $j$ with bias class $b$, we could derive a bias-balanced softmax
[6, 18] as follows:

######  Theorem 2.1.

(Bias-balanced softmax [6]) Assume
$\phi_{j}=p(y=j|x,b)=\frac{p(x|y=j,b)}{p(x|b)}\cdot\frac{1}{k}$ to be the
desired conditional probability of the bias-balanced dataset, and
$\hat{\phi}_{j}=\frac{p(x|y=j,b)}{\hat{p}(x|b)}\cdot\frac{n_{j,b}}{\sum_{i=1}^{k}n_{i,b}}$
to be the conditional probability of the biased dataset. If $\phi$ can be
expressed by the standard Softmax function of the logits $\eta$ generated by
the model, i.e., $\phi_{j}=\frac{exp(\eta_{j})}{\sum_{i=1}^{k}exp(\eta_{i})}$,
then $\hat{\phi}$ can be expressed as

$\hat{\phi}_{j}=\frac{p(y=j|b)\cdot{\rm
exp}(\eta_{j})}{\sum_{i=1}^{k}p(y=i|b)\cdot{\rm exp}({\eta_{i}})}.$ (2)

Theorem 2 (proof provided in the supplementary) shows that bias-balanced
softmax could well solve the distributional discrepancy between the bias-
imbalanced training set and the bias-balanced testing set. Denoting $M$ the
number of training data, we obtain the bias-balanced loss for training a
debiased model:

$\mathcal{L}_{\rm
BS}(f(x),y,b)=-\frac{1}{M}\sum_{i=1}^{M}log\Bigg{(}\frac{p(y=j|b)\cdot{\rm
exp}(\eta_{j})}{\sum_{i=1}^{k}p(y=i|b)\cdot{\rm exp}({\eta_{i}})}\Bigg{)}$ (3)

However, this loss requires estimation of the bias distribution on the
training set, while comprehensively labeling all kinds of attributes would be
unpractical, especially for medical data. In the next section, we elaborate on
how to obtain the estimation of the bias distribution without knowing the bias
labels.

###  2.3 Bias Capturing with Generalized Cross Entropy Loss

Inspired by [15], we conducted two experiments based on the Source-biased
Pneumonia dataset with $r=10$, where we set the models to classify data source
(Fig. 2(a)) or health condition (Fig. 2(b)), respectively. Apparently, the
model has almost no signs of fitting on the bias attribute (health condition)
when it’s required to distinguish data source. On the other hand, the model
quickly learns the biases (data source) when set to classify pneumonia from
healthy cases. From these findings, one could conclude that _dataset biases
would be preferred when they were easier to be learned than the intended
features_.

(a)

(b)

(c)

Figure 2:  Based on the SbP dataset, we show the learning curve of the vanilla
model by setting the {target, bias} pair to be (a) {data source, health
condition} and (b) {health condition, data source}. We also show in (c) the
learning curve of a highly-biased model trained with GCE loss with the
{target, bias} pair being {health condition, data source}. Blue curves: loss
of bias-aligned samples; Red curves: loss of bias-conflicting samples.

Based on this observation, we could develop a model to capture the dataset
bias by making it quickly fit on the easier features from the training data.
Therefore, we adopt the generalized cross entropy (GCE) loss [27], which was
originally proposed to address noisy labels by fitting on the easier clean
data and slowly memorizing the hard noisy samples. Inheriting this idea, the
GCE loss could also quickly capture easy and biased samples than the
categorical cross entropy (CE) loss. Giving $f(x;\theta)$ the softmax output
of the model, denoting $f_{y=j}(x;\theta)$ the probability of $x$ being
classified to class $y=j$ and $\theta$ the parameters of model $f$, the GCE
loss is formulated as follows:

$\mathcal{L}_{\rm GCE}(f(x;\theta),y=j)=\frac{1-f_{y=j}(x;\theta)^{q}}{q},$
(4)

where $q$ is a hyper-parameter. The gradient of GCE is
$\frac{\partial\mathcal{L}_{\rm
GCE}(f(x;\theta),y=j)}{\partial\theta}=f_{y=j}(x;\theta)^{q}\frac{\partial\mathcal{L}_{\rm
CE}(f(x;\theta),y=j)}{\partial\theta}$ (proof provided in the supplementary),
which explicitly assigns weights on the CE loss based on the agreement between
model’s predictions and the labels. As shown in Fig. 2(c), GCE loss fits the
bias-aligned samples quickly while yields much higher loss on the bias-
conflicting samples.

###  2.4 Bias-balanced Learning with Pseudo Bias

With the afore discussed observations and analysis, we propose a debiasing
algorithm, namely Pseudo Bias Balanced Learning. We first train a biased model
$f_{B}(x;\theta_{B})$ with the GCE loss and calculate the corresponding
receiver operating characteristics (ROC) over the training set. Based on the
ROC curve, we compute the sensitivity $u(\tau)$ and specificity $v(\tau)$
under each threshold $\tau$ and then assign pseudo bias labels to each sample
with the following:

$\tilde{b}(f_{B}(x;\theta_{B}))=\begin{cases}1,&\text{if
}f_{B}(x;\theta_{B})\geq{\rm argmax}_{\tau}(u(\tau)+v(\tau));\\\
0,&\text{otherwise}.\end{cases}$ (5)

Algorithm 1 Pseudo Bias Balanced Learning

Input: $\theta_{B}$, $\theta_{D}$, image $x$, target label $y$, numbers of
iterations $T_{B}$, $T_{D}$, $N$.
Output: Debiased model $f_{D}(x;\theta_{D})$.

1:Initialize $\tilde{b}=y$.

2:for n=1, $\cdots$, $N$ do

3: Initialize network $f_{B}(x;\theta_{B})$.

4: for t=1, $\cdots$, $T_{B}$ do

5: Update $f_{B}(x;\theta_{B})$ with $\mathcal{L}_{\rm
GCE}(f_{B}(x;\theta_{B}),\tilde{b})$

6: end for

7: Calculate $u$, $v$, and $\tau$ over training set.

8: Update pseudo bias labels $\tilde{b}$ with Eq. 5.

9:end for

10:Initialize network $f_{D}(x;\theta_{D})$.

11:for t=1, $\cdots$, $T_{D}$ do

12: Update $f_{D}(x;\theta_{D})$ with $\mathcal{L}_{\rm
BS}(f_{D}(x;\theta_{D}),y,\tilde{b})$

13:end for

Moreover, as the biased model could also memorize the correct prediction for
the hard bias-conflicting cases [26], we propose to capture and enhance the
bias via iterative model training. Finally, we train our debiased model
$f_{D}(x;\theta_{D})$ based on the pseudo bias labels and the bias-balance
softmax function, with different weights from $\theta_{B}$. The holistic
approach is summarized in Algorithm 1.

##  3 Experiments

Evaluation metrics are the area under the ROC curve (AUC) with four criteria:
i) AUC on bias-aligned samples; ii) AUC on bias-conflicting samples; iii)
Average of bias-aligned AUC and bias-conflicting AUC, which we call balanced-
AUC; iv) AUC on all samples. The difference between the first two metrics
could reflect whether the model is biased, while the latter two metrics
provide unbiased evaluations on the testing data.

Compared methods included four other approaches: i) Vanilla model, which did
not use any debiasing strategy and could be broadly regarded as a lower bound.
ii) Group Distribution Robust Optimization (G-DRO) [20], which used the bias
ground truth and could be regarded as the upper bound. G-DRO divides training
data into different groups according to their targets and bias labels. It then
optimized the model with priority on the worst-performing group and finally
achieved robustness on every single group. As in practical scenarios, the
labels for the dataset biases may not be known, we also implemented iii)
Learning from Failure (LfF) [15], which developed a debiased model by weighted
losses from a biased model; and iv) Disentangled Feature Augmentation (DFA)
[11], which was based on LfF and further adds feature swapping and
augmentation between the debiased and biased models.

Model training protocol is as follows: We used the same backbone, DenseNet-121
[7] with pre-trained weights from [3], for every method. Particularly, we
fixed the weights of DenseNet, replaced the final output layer with three
linear layers, and used the rectified linear units as the intermediate
activation function. We ran each model with three different random seeds, and
reported the test results corresponding to the best validation AUC. Each model
is optimized with Adam [9] for around 1,000 steps with batch size of 256 and
learning rate of 1e-4. $N$ in Algorithm 1 is empirically set to 1 for SbP
dataset and 2 for GbP dataset, respectively. $q$ in GCE loss is set to 0.7 as
recommended in [27].

Table 1: AUC results on SbP dataset. Best results without ground truth bias
labels are emphasized in bold. $\dagger$ means the method uses ground truth
bias labels.

Bias Ratio | Method | Aligned | Conflicting | Balanced | Overall | External
---|---|---|---|---|---|---
90% | G-DRO† [20] | 70.02±2.20 | 89.80±0.87 | 79.94±0.68 | 80.23±0.37 | 90.06±0.32
Vanilla | 96.51±0.26 | 31.21±3.04 | 63.86±1.39 | 69.84±1.32 | 71.57±0.90
LfF [15] | 68.57±2.16 | 87.46±2.17 | 78.02±0.18 | 78.26±0.18 | 87.71±2.66
DFA [11] | 74.63±4.61 | 83.30±3.96 | 78.96±0.33 | 78.76±0.15 | 74.58±7.56
Ours | 76.82±2.80 | 85.75±0.32 | 80.49±0.20 | 78.78±3.02 | 89.96±0.69
95% | G-DRO† [20] | 68.65±1.21 | 89.86±0.67 | 79.26±0.47 | 79.8±0.36 | 90.16±0.73
Vanilla | 97.91±0.75 | 20.45±5.96 | 59.18±2.61 | 67.11±1.85 | 68.61±3.50
LfF [15] | 69.56±2.01 | 86.43±1.67 | 77.99±0.18 | 78.28±0.22 | 88.56±3.37
DFA [11] | 69.04±4.21 | 84.94±2.56 | 76.99±0.85 | 77.26±0.49 | 76.37±3.26
Ours | 71.72±6.65 | 84.68±3.49 | 78.20±0.20 | 78.04±3.46 | 82.65±0.40
99% | G-DRO† [20] | 74.30±2.28 | 85.18±1.26 | 79.74±0.55 | 79.71±0.40 | 89.87±0.64
Vanilla | 99.03±0.95 | 4.93±3.68 | 51.98±1.60 | 59.21±3.76 | 60.79±0.98
LfF [15] | 77.50±11.08 | 64.38±8.75 | 70.94±1.30 | 71.86±1.72 | 73.90±4.42
DFA [11] | 69.33±1.74 | 75.48±2.61 | 72.40±0.48 | 72.49±0.45 | 61.67±6.86
Ours | 72.40±0.71 | 77.61±0.45 | 75.00±0.18 | 74.70±0.14 | 78.87±0.44

Quantitative results on Source-biased Pneumonia dataset are reported in Table
1. With the increasing of bias ratio, the vanilla model became more and more
biased and severe decreases in balanced-AUC and overall-AUC was observed. All
other methods also showed decreases on the two metrics, while G-DRO shows
quite robust performance under all situations. Meanwhile, our method achieved
consistent improvement over the compared approaches under most of the
situations, demonstrating its effectiveness in debiasing. Interestingly, the
change of external testing performance appeared to be in line with the change
of the balanced-AUC and overall AUC, which further revealed that overcoming
shortcut learning improves the model’s generalization capability. These
findings demonstrated our method’s effectiveness in solving shortcut learning,
with potential in robustness and trustworthiness for real-world clinic usage.

Table 2: AUC results on GbP dataset. Best results without ground truth bias
labels are emphasized in bold. $\dagger$ means the method uses ground truth
bias labels.

Training | Method | Aligned | Conflicting | Balanced | Overall
---|---|---|---|---|---
GbP-Tr1 | G-DRO† [20] | 85.81±0.16 | 83.96±0.17 | 84.86±0.05 | 84.93±0.01
Vanilla | 89.42±0.25 | 77.21±0.33 | 83.31±0.05 | 83.75±0.05
LfF [15] | 88.73±1.34 | 77.47±0.09 | 83.10±0.64 | 83.46±0.71
DFA [11] | 86.12±0.46 | 77.92±0.23 | 82.02±0.31 | 82.23±0.30
Ours | 90.17±0.42 | 77.07±1.73 | 83.62±0.68 | 84.13±0.56
GbP-Tr2 | G-DRO† [20] | 83.76±1.59 | 85.14±0.31 | 84.45±0.65 | 84.42±0.61
Vanilla | 89.39±0.85 | 76.13±0.93 | 82.76±0.78 | 82.93±0.78
LfF [15] | 87.25±0.62 | 79.07±0.96 | 83.16±0.45 | 83.19±0.44
DFA [11] | 80.44±0.58 | 85.51±0.57 | 82.98±0.19 | 83.09±0.21
Ours | 86.34±0.64 | 81.69±2.67 | 84.02±1.01 | 84.03±0.97

Figure 3: Class activation map [28] generated from vanilla model and our
method. Samples are from the SbP dataset (in blue box) and the GbP dataset (in
red box), respectively.

Quantitative results on Gender-biased Pneumothorax dataset are reported in
Table 2. By the performance of the vanilla model, gender bias may not affect
the performance as severely as data source bias, but it could lead to serious
fairness issues. We observed that G-DRO showed robust performance on the two
different training sets. Among approaches that do not use ground truth bias
labels, our proposed method achieved consistent improvement over others with
the two different training sets. The results also showed the potential of our
method in developing fair and trustworthy diagnosis models.

Qualitative results were visualized by class activation map [28], as shown in
Fig. 3. It can be observed that vanilla model might look for evidence outside
the lung regions, while our method could more correctly focus on the lung
regions.

##  4 Conclusion

In this paper, we studied the causes and solutions for shortcut learning in
medical image analysis, with chest X-ray as an example. We showed that
shortcut learning occurs when the bias distribution is imbalanced, and the
dataset bias is preferred when it is easier to be learned than the intended
features. Based on these findings, we proposed a novel pseudo bias balanced
learning algorithm to develop a debiased model without explicit labeling on
the bias attribute. We also constructed several challenging debiasing datasets
from public-available data. Extensive experiments demonstrated that our method
overcame shortcut learning and achieved consistent improvements over other
state-of-the-art methods under different scenarios, showing promising
potential in developing robust, fair, and trustworthy diagnosis models.

####  4.0.1 Acknowledgement.

This work was supported by Hong Kong Innovation and Technology Fund Project
No. GHP/110/19SZ and ITS/170/20.

## References

  * [1] Arjovsky, M., Bottou, L., Gulrajani, I., Lopez-Paz, D.: Invariant risk minimization. arXiv preprint arXiv:1907.02893 (2019)
  * [2] Bustos, A., Pertusa, A., Salinas, J.M., de la Iglesia-Vayá, M.: Padchest: A large chest x-ray image dataset with multi-label annotated reports. Medical image analysis 66, 101797 (2020)
  * [3] Cohen, J.P., Viviano, J.D., Bertin, P., Morrison, P., Torabian, P., Guarrera, M., Lungren, M.P., Chaudhari, A., Brooks, R., Hashir, M., et al.: Torchxrayvision: A library of chest x-ray datasets and models. arXiv preprint arXiv:2111.00595 (2021)
  * [4] DeGrave, A.J., Janizek, J.D., Lee, S.I.: Ai for radiographic covid-19 detection selects shortcuts over signal. Nature Machine Intelligence 3(7), 610–619 (2021)
  * [5] Geirhos, R., Jacobsen, J.H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., Wichmann, F.A.: Shortcut learning in deep neural networks. Nature Machine Intelligence 2(11), 665–673 (2020)
  * [6] Hong, Y., Yang, E.: Unbiased classification through bias-contrastive and bias-balanced learning. Advances in Neural Information Processing Systems 34 (2021)
  * [7] Huang, G., Liu, Z., Van Der Maaten, L., Weinberger, K.Q.: Densely connected convolutional networks. In: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. pp. 4700–4708 (2017)
  * [8] Johnson, A.E., Pollard, T.J., Berkowitz, S.J., Greenbaum, N.R., Lungren, M.P., Deng, C.y., Mark, R.G., Horng, S.: Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data 6(1), 1–8 (2019)
  * [9] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Proc. Int. Conf. Learn. Representations (2015)
  * [10] Larrazabal, A.J., Nieto, N., Peterson, V., Milone, D.H., Ferrante, E.: Gender imbalance in medical imaging datasets produces biased classifiers for computer-aided diagnosis. Proceedings of the National Academy of Sciences 117(23), 12592–12594 (2020)
  * [11] Lee, J., Kim, E., Lee, J., Lee, J., Choo, J.: Learning debiased representation via disentangled feature augmentation. Advances in Neural Information Processing Systems 34 (2021)
  * [12] Li, Y., Vasconcelos, N.: Repair: Removing representation bias by dataset resampling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 9572–9581 (2019)
  * [13] Luo, L., Chen, H., Xiao, Y., Zhou, Y., Wang, X., Vardhanabhuti, V., Wu, M., Heng, P.A.: Rethinking annotation granularity for overcoming deep shortcut learning: A retrospective study on chest radiographs. arXiv preprint arXiv:2104.10553 (2021)
  * [14] Mitrovic, J., McWilliams, B., Walker, J.C., Buesing, L.H., Blundell, C.: Representation learning via invariant causal mechanisms. In: International Conference on Learning Representations (2020)
  * [15] Nam, J., Cha, H., Ahn, S.S., Lee, J., Shin, J.: Learning from failure: De-biasing classifier from biased classifier. Advances in Neural Information Processing Systems 33 (2020)
  * [16] Oakden-Rayner, L., Dunnmon, J., Carneiro, G., Ré, C.: Hidden stratification causes clinically meaningful failures in machine learning for medical imaging. In: Proceedings of the ACM conference on health, inference, and learning. pp. 151–159 (2020)
  * [17] Rajpurkar, P., Chen, E., Banerjee, O., Topol, E.J.: Ai in health and medicine. Nature Medicine pp. 1–8 (2022)
  * [18] Ren, J., Yu, C., Ma, X., Zhao, H., Yi, S., et al.: Balanced meta-softmax for long-tailed visual recognition. Advances in neural information processing systems 33, 4175–4186 (2020)
  * [19] Ribeiro, M.T., Singh, S., Guestrin, C.: ” why should i trust you?” explaining the predictions of any classifier. In: Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining. pp. 1135–1144 (2016)
  * [20] Sagawa, S., Koh, P.W., Hashimoto, T.B., Liang, P.: Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In: International Conference on Learning Representations (2020)
  * [21] Tartaglione, E., Barbano, C.A., Grangetto, M.: End: Entangling and disentangling deep representations for bias correction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 13508–13517 (2021)
  * [22] Torralba, A., Efros, A.A.: Unbiased look at dataset bias. In: CVPR 2011. pp. 1521–1528. IEEE (2011)
  * [23] Wang, X., Peng, Y., Lu, L., Lu, Z., Bagheri, M., Summers, R.M.: Chestx-ray8: Hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2097–2106 (2017)
  * [24] Yoon, C., Hamarneh, G., Garbi, R.: Generalizable feature learning in the presence of data bias and domain class imbalance with application to skin lesion classification. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 365–373. Springer (2019)
  * [25] Zech, J.R., Badgeley, M.A., Liu, M., Costa, A.B., Titano, J.J., Oermann, E.K.: Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study. PLoS medicine 15(11), e1002683 (2018)
  * [26] Zhang, C., Bengio, S., Hardt, M., Recht, B., Vinyals, O.: Understanding deep learning requires rethinking generalization. In: ICLR (2017)
  * [27] Zhang, Z., Sabuncu, M.: Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems 31 (2018)
  * [28] Zhou, B., Khosla, A., Lapedriza, A., Oliva, A., Torralba, A.: Learning deep features for discriminative localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2921–2929 (2016)
  * [29] Zhu, W., Zheng, H., Liao, H., Li, W., Luo, J.: Learning bias-invariant representation by cross-sample mutual information minimization. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 15002–15012 (2021)

##  5 Supplementary

####  5.0.1 Proof of Theorem 1

is provided following [6, 18] for better reference. The exponential family
parameterization of the multinomial distribution provides the standard Softmax
function as the _canonical response function_ as follows:

$\phi_{j}=\frac{{\rm exp}(\eta_{j})}{\sum_{i=1}^{k}{\rm exp}(\eta_{i})}$ (6)

and the _canonical link function_ as:

$\eta_{j}={\rm log}(\frac{\phi_{j}}{\phi_{k}})$ (7)

By adding $-{\rm log}(\phi_{j}/\hat{\phi}_{j})$ to both sides of Eq. 7, we
have:

$\eta_{j}-{\rm log}(\frac{\phi_{j}}{\hat{\phi}_{j}})={\rm
log}(\frac{\phi_{j}}{\phi_{k}})-{\rm
log}(\frac{\phi_{j}}{\hat{\phi}_{j}})={\rm
log}(\frac{\hat{\phi}_{j}}{\phi_{k}}),$ (8)

from which we further have:

$\phi_{k}{\rm exp}(\eta_{j}-{\rm
log}(\frac{\phi_{j}}{\hat{\phi}_{j}}))=\hat{\phi}_{j}$ (9)
$\phi_{k}\sum_{i=1}^{k}{\rm exp}(\eta_{i}-{\rm
log}(\frac{\phi_{i}}{\hat{\phi}_{i}}))=\sum_{i=1}^{k}\hat{\phi}_{i}=1$ (10)
$\phi_{k}=1/\sum_{i=1}^{k}{\rm exp}(\eta_{i}-{\rm
log}(\frac{\phi_{i}}{\hat{\phi}_{i}}))$ (11)

Substitute Eq. 11 back to Eq. 9, we could have:

$\hat{\phi}_{j}=\phi_{k}{\rm exp}(\eta_{j}-{\rm
log}(\frac{\phi_{j}}{\hat{\phi}_{j}}))=\frac{{\rm exp}(\eta_{j}-{\rm
log}(\frac{\phi_{j}}{\hat{\phi}_{j}}))}{\sum_{i=1}^{k}{\rm exp}(\eta_{i}-{\rm
log}(\frac{\phi_{i}}{\hat{\phi}_{i}}))}$ (12)

We recall that

$\phi_{j}=p(y=j|x,b)=\frac{p(x|y=j,b)}{p(x|b)}\cdot\frac{1}{k}\text{;
}\hat{\phi}_{j}=\frac{p(x|y=j,b)}{\hat{p}(x|b)}\cdot\frac{n_{j,b}}{\sum_{i=1}^{k}n_{i,b}}$
(13)

Hence,

${\rm log}(\frac{\phi_{j}}{\hat{\phi}_{j}})={\rm
log}(\frac{\sum_{i=1}^{k}n_{i,b}}{kn_{j,b}})+{\rm
log}(\frac{\hat{p}(x|b)}{p(x|b)})$ (14)

For simplicity, we let $n_{b}=\sum_{i=1}^{k}n_{j,b}$ to be the number of
samples obtaining the bias label as $b$. Finally, by substituting Eq. 14 back
to Eq. 12, we have

$\begin{split}\hat{\phi_{j}}&=\frac{{\rm exp}(\eta_{j}-{\rm
log}\frac{n_{b}}{kn_{j,b}}-{\rm
log}\frac{\hat{p}(x|b)}{p(x|b)})}{\sum_{i=1}^{k}{\rm exp}(\eta_{i}-{\rm
log}\frac{n_{b}}{kn_{i,b}}-{\rm log}\frac{\hat{p}(x|b)}{p(x|b)})}\\\
&=\frac{\frac{n_{j,b}}{n_{b}}\cdot{\rm
exp}(\eta_{j})}{\sum_{i=1}^{k}\frac{n_{i,b}}{n_{b}}\cdot{\rm
exp}(\eta_{i})}=\frac{p(y=j|b)\cdot{\rm
exp}(\eta_{j})}{\sum_{i=1}^{k}p(y=i|b)\cdot{\rm exp}({\eta_{i}})}.\end{split}$
(15)

####  5.0.2 Gradient of Generalized Cross Entropy Loss [27]:

The form of the GCE loss is as follows:

$\mathcal{L}_{\rm GCE}(f(x;\theta),y=j)=\frac{1-f_{y=j}(x;\theta)^{q}}{q},$
(16)

Hence, the gradient is:

$\frac{\partial\mathcal{L}_{\rm
GCE}(f(x;\theta),y=j)}{\partial\theta}=-{f_{y=j}(x;\theta)}^{q-1}\frac{\partial
f_{y=j}(x;\theta)}{\partial\theta}$ (17)

Recall that the form of conventional cross entropy loss is $\mathcal{L}_{\rm
CE}(f(x;\theta),y=j)=-{\rm log}(f_{y=j}(x;\theta))$, hence

$\frac{\partial\mathcal{L}_{\rm
CE}(f(x;\theta),y=j)}{\partial\theta}=-{f_{y=j}(x;\theta)}^{-1}\frac{\partial
f_{y=j}(x;\theta)}{\partial\theta}$ (18)

Therefore,

$\frac{\partial\mathcal{L}_{\rm
GCE}(f(x;\theta),y=j)}{\partial\theta}=f_{y=j}(x;\theta)^{q}\frac{\partial\mathcal{L}_{\rm
CE}(f(x;\theta),y=j)}{\partial\theta}$ (19)