# Data-Driven Vulnerability Detection and Repair in Java Code

Ying Zhang, Mahir Kabir, Ya Xiao, Danfeng (Daphne) Yao, Na Meng  Computer
Science, Virginia Tech yingzhang, mdmahirasefk, yax99, danfeng<EMAIL_ADDRESS>
###### Abstract.

Java platform provides various APIs to facilitate secure coding. However,
correctly using security APIs is usually challenging for developers who lack
cyber security training. Prior work shows that many developers misuse security
APIs; such misuses can introduce vulnerabilities into software, void security
protections, and present security exploits to hackers. To eliminate such API-
related vulnerabilities, this paper presents Seader—our new approach that
detects and repairs security API misuses. Given an exemplar insecure code
snippet and its secure counterpart, Seader compares the snippets and conducts
data dependence analysis to infer the security API misuse templates and
corresponding fixing operations. Based on the inferred information, given a
program, Seader performs inter-procedural static analysis to search for any
security API misuse and to propose customized fixing suggestions for those
vulnerabilities.

To evaluate Seader, we applied it to 25 ¡insecure, secure¿ code pairs, and
Seader successfully inferred 18 unique API misuse templates and related fixes.
With these vulnerability repair patterns, we further applied Seader to 10
open-source projects that contain in total 32 known vulnerabilities. Our
experiment shows that Seader detected vulnerabilities with 100% precision, 84%
recall, and 91% accuracy. Additionally, we applied Seader to 100 Apache open-
source projects and detected 988 vulnerabilities; Seader always customized
repair suggestions correctly. Based on Seader’s outputs, we filed 60 pull
requests. Up till now, developers of 18 projects have offered positive
feedbacks on Seader’s suggestions. Our results indicate that Seader can
effectively help developers detect and fix security API misuses. Whereas prior
work either detects API misuses _or_ suggests simple fixes, Seader is the
first tool to do both for nontrivial vulnerability repairs.

vulnerability detection, vulnerability repair, inter-procedural

††copyright: none

##  1\. Introduction

Java platform provides libraries (e.g., Java Cryptography Architecture (JCA)
(Java Cryptography Architecture, [n.d.]) and Java Secure Socket Extension
(jss, 2020)) to ease developers’ secure software development (e.g.,
implementing key generation and secure communication). However, these
libraries are not easy to use for two reasons. First, some APIs have overly
complicated usage protocols that are poorly documented (Green and Smith, 2016;
Nadi et al., 2016). Second, developers lack the necessary cyber security
training for secure coding (dev, 2017; Meng et al., 2018; too, 2019).
Meanwhile, prior work shows that developers misused security APIs (Fischer et
al., 2017; Rahaman et al., 2019), and introduced vulnerabilities when building
security functionalities (Fahl et al., 2012; Georgiev et al., 2012). For
instance, Fischer et al. found that the security API misuses posted on
StackOverflow (so, 2020) were copied and pasted into 196,403 Android
applications available on Google Play (Fischer et al., 2017). Rahaman et al.
revealed similar API misuses in 39 high-quality Apache projects (Rahaman et
al., 2019). Fahl et al. (Fahl et al., 2012) and Georgiev et al. (Georgiev et
al., 2012) separately showed that such API-related vulnerabilities could be
exploited by hackers to steal data (e.g., user credentials).

Existing tools provide insufficient support to help developers eliminate
security API misuses. Specifically, some tools identify API misuses based on
either hardcoded rules, rules specified with a domain-specific language, or
machine learning (Egele et al., 2013; Fischer et al., 2017; Li et al., 2018;
Krüger et al., 2018; Rahaman et al., 2019; Fischer et al., 2019). However,
these tools cannot be easily extended to reveal new vulnerabilities, neither
can they suggest any security repair. General-purpose program repair tools
rely on automatic testing to reveal bugs, and only suggest trivial fixes
(e.g., single-line edits) in very limited cases (Long and Rinard, 2016; Qi et
al., 2014; DeMarco et al., 2014; Kim et al., 2013). These tools cannot repair
security API misuses because (1) vulnerabilities seldom fail tests, and (2)
some security patches involve nontrivial edits like overriding an interface
method. To design a better approach that overcomes the above-mentioned
limitations, we need to solve two technical challenges. First, the new
approach should accurately detect API misuses and suggest fixes. Second, the
approach should be flexibly extensible to handle new vulnerabilities.

Figure 1. The overview of Seader

In this paper, we present our new approach Seader, which detects the
vulnerabilities caused by security API misuses and suggests repairs. As shown
in Figure 1, there are two phases in Seader: pattern inference and pattern
application. In the first phase, suppose that a domain expert (e.g., security
researcher) provides

  * •

_I_ —insecure code with certain security API misuse, and

  * •

_S_ —the secure counterpart showing the correct API usage.

Seader compares the two code snippets and detects program changes that can
transform _I_ to _S_. Next, based on those changes, Seader derives a
vulnerability repair pattern from both snippets. Each pattern consists of two
parts: (i) a vulnerable code template together with matching-related
information, and (ii) the abstract fix. Seader stores all inferred patterns
into a JSON file. In Phase II, given a program _P_ , Seader loads all patterns
from the JSON file, and searches for code matching any template. For each code
match, Seader concretizes the corresponding abstract fix, and suggests code
replacements to developers.

Additionally, security researchers have summarized a number of typical API
misuse patterns related to Java security libraries (Fischer et al., 2017;
Rahaman et al., 2019). Based on our experience with example definition for
these patterns, we realized that some patterns (e.g., multiple secure options
for a parameter) cannot be easily described with plain Java code examples.
Therefore, we also defined an example definition library (EDL), which provides
APIs for security experts to specify examples, and for Seader to specialize
pattern inference in certain scenarios.

For evaluation, we crafted 25 ¡insecure, secure¿ code pairs based on some
known API misuse patterns summarized by prior research. After Seader inferred
patterns from those pairs, we further applied Seader to two program data sets
to evaluate its effectiveness in terms of vulnerability detection and repair
suggestion.

Specifically, the first data set contains 10 Apache open-source projects, with
32 real vulnerabilities manually identified in the JAR files. With this data
set, we observed Seader to detect vulnerabilities with 100% precision, 84%
recall, and 91% accuracy. The second data set contains 100 Apache open-source
projects. Seader found 988 vulnerabilities in them and proposed corresponding
repair suggestions. We manually checked 100 repair suggestions and found them
to be correctly customized. Finally, we filed 59 pull requests based on
Seader’s reports for 59 vulnerable Java classes and sought for developers’
opinions. Up till now, the maintainers of 14 vulnerable classes have confirmed
the reported vulnerabilities, and 6 of them have taken actions to fix or
document the vulnerabilities.

To sum up, we made the following contributions in this paper:

  * •

We designed and implemented Seader—a data-driven approach that infers
vulnerability repair patterns from ¡insecure, secure¿ code examples, and
applies those patterns to detect and fix security API misuses.

  * •

For pattern inference, Seader provides a software library—EDL—for users to
adopt so that they can define examples for some tricky scenarios. Our
experiment with 25 code pairs shows that Seader always infers patterns
correctly from ¡insecure, secure¿ code examples.

  * •

In terms of vulnerability detection, our evaluation with real vulnerabilities
from open-source projects shows that Seader worked as effectively as or even
better than the state-of-the-art tool—CryptoGuard—in 8 of the 10 projects.

  * •

In terms of vulnerability repair, our manual inspection based on another data
set shows that the repairing suggestions generated by Seader are correct.
Interestingly, developers have mixed opinions on Seader’s repair suggestions.

Seader has the data-driven nature by inferring vulnerability repair patterns
from code examples, which functionality ensures the approach extensibility
when new API misuses are revealed. Seader can accurately detect
vulnerabilities and suggest meaningful fixes, showing great potential of
helping eliminate API-related vulnerabilities.

##  2\. A Motivating Example

This section overviews our approach with a set of code examples. Prior
research shows that the security of symmetric encryption schemes depends on
the secrecy of any shared key (Egele et al., 2013). Thus, developers should
not generate secret keys from constant values hardcoded in software
applications (Fischer et al., 2017). Suppose that a security expert Alex wants
to detect and fix the vulnerable code that creates a secret key from a
constant value. To achieve the goal with Seader, Alex needs to craft (1) an
insecure code example showing the API misuse, and (2) a secure example to
demonstrate the correct API usage. As shown in Figure 2, the insecure code _I_
invokes the constructor of SecretKeySpec by passing in a constant byte array
(i.e., StringLiterals.CONSTANTS.getBytes()) as the first parameter. On the
other hand, the secure code _S_ invokes the same constructor by sending in
keyBytes instead, which parameter is created based on a randomly generated
number (see lines 1-4).

Insecure code (_I_)
---
⬇ 1SecretKey key = new SecretKeySpec(StringLiterals.CONSTANT.getBytes(),
”AES”);
Secure code (_S_)
⬇ 1SecureRandom random = new SecureRandom(); 2String defaultKey =
String.valueOf(random.nextInt()); 3byte[] keyBytes = defaultKey.getBytes();
4keyBytes = Arrays.copyOf(keyBytes,24); 5SecretKey key = new
SecretKeySpec(keyBytes, ”AES”);
Figure 2. A pair of examples to show the vulnerability and repair relevant to
secret key creation Vulnerable code template (T)
---
⬇ 1SecretKey $v_0 = new SecretKeySpec(StringLiterals.CONSTANT.getBytes(),
”AES”);
Matching-related data:
critical API: javax.crypto.spec.SecretKeySpec.SecretKeySpec(byte[], String)
other security APIs: {}
Abstract fix (F)
⬇ 1SecureRandom $v_1 = new SecureRandom(); 2String $v_2 =
String.valueOf($v_1.nextInt()); 3byte[] $v_3 = $v_2.getBytes(); 4$v_3 =
Arrays.copyOf($v_3, 24); 5SecretKey $v_0 = new SecretKeySpec($v_3, ”AES”);
Figure 3. The pattern inferred from the code pair in Figure 2

Taking the two examples as input, Seader generates abstract syntax trees
(ASTs) and compares them for any code change. Specifically, five operations
are detected: one expression update (StringLiterals. CONSTANT.getBytes()
replaced by keyBytes), and four statement insertions. Next, based on the
updated expression in _I_ , Seader conducts data dependency analysis to reveal
any security API that uses the expression, and treats it as a critical API.
Such critical APIs are important for Seader to later reveal similar
vulnerabilities in other codebases. Afterwards, Seader generalizes a
vulnerability repair pattern from the examples by abstracting concrete
variable names. As shown in Figure 3, the generalized pattern has two parts:
(i) the vulnerability template (T) together with matching-related data and
(ii) an abstract fix (F). Such pattern generalization ensures the inferred
program transformation to be applicable to codebases that use different
variable identifiers from the given examples.

Listing 1: A simplified version of _P_

⬇

1public class CEncryptor {

2 private char[] passPhrase;

3 private String alg = ”AES”;

4 public CEncryptor(String passPhrase) {

5 this.passPhrase = passPhrase.toCharArray();

6 }

7 public Result encrypt(byte[] plain) throws Exception {

8 SecretKey secret = new SecretKeySpec(new String(passPhrase).getBytes(),
alg);

9 …

10}

11public class Main {

12 public static void main(String[] args)

13 CEncryptor aes0 = new CEncryptor(”password”);

14 aes0.encrypt((byte[])args[0]);

15 …

16}

With a pattern inferred from the provided code pair, Alex can further apply
Seader to an arbitrary program _P_ to detect and resolve any occurrence of the
described vulnerability. In particular, given a program whose simplified
version is shown in Listing 1, Seader first scans for any invocation of the
critical API SecretKeySpec(...). If no such invocation exists, Seader
concludes that _P_ does not have the above-mentioned vulnerability; otherwise,
if the API is invoked (see line 8 in Listing 1), Seader further searches for
any code match for the template in Figure 3. Specifically, the template
matching process conducts inter-procedural analysis and checks for two
conditions:

  1. C1:

Does the API call have the first parameter derive from a constant?

  2. C2:

Does the API call have the second parameter match "AES"?

If any invocation of SecretKeySpec(...) satisfies both conditions, Seader
concludes that the code has the above-mentioned vulnerability. Notice that if
we simply check line 8 of Listing 1, it seems that neither new
String(passPhrase).getBytes() nor alg satisfies any condition. Thanks to the
usage of inter-procedural analysis, Seader is able to conduct backward slicing
to trace how both parameters are initialized before their current usage. With
more details, because alg is a private field of CEncryptor, whose value is
initialized on line 3 with "AES", Seader decides that C2 is satisfied.
Similarly, passPhrase is another field whose value is initialized with a
parameter of the constructor CEncryptor(...) (lines 4-6). When CEncryptor(...)
is called with parameter "password" before the invocation of
SecretKeySpec(...) (lines 7-14), C1 is satisfied. Therefore, Seader concludes
that line 8 matches the template; it thus matches concrete variable secret
with the template variable $v_0.

Listing 2: A customized fix for _P_ suggested by Seader

⬇

1SecureRandom $v_1 = new SecureRandom();

2String $v_2 = String.valueOf($v_1.nextInt());

3byte[] $v_3 = $v_2.getBytes();

4$v_3 = Arrays.copyOf($v_3,24);

5SecretKey secret = new SecretKeySpec($v_3, ”AES”);

Based on the found code match, Seader customizes the abstract fix shown in
Figure 3 by replacing the abstract variable $v_0 with concrete variable
secret. Seader then suggests the code snippet shown in Listing 2 to replace
line 8 of Listing 1. In the suggested fix, a SecureRandom instance is first
initialized to randomly generate a string value (lines 1-2 in Listing 2).
Next, the string is converted to a byte array (line 3), in order to match the
data type of the first parameter of SecretKeySpec(byte[], String). Afterwards,
the byte array is further converted to a 24-byte array (line 4), because the
AES algorithm is capable of using cryptographic keys of 16, 24, or 32 bytes
(i.e., 128, 192, or 256 bits) (of Standards and (NIST), 2001). Here, for
simplicity, we randomly pick 24 bytes to create a well-formatted variable
keyBytes. Finally, keyBytes and "AES" are passed to invoke the critical API.

##  3\. Approach

There are two phases in Seader (see Figure 1). In this section, we first
summarize the steps in each phase and then describe each step in detail
(Section 3.1-Section 3.4). Next, we introduce EDL—the software library we
created—to facilitate example definition by security experts (Section 3.5).

Phase I: Pattern Inference:

  * •

Given an ¡$I$, $S$¿ example pair, Seader builds an AST for each example; it
then compares the ASTs to infer an edit script that can transform I to S. We
denote the edit script with $E=\\{op_{1},op_{2},\ldots,op_{n}\\}$.

  * •

For any update $u\in E$ that changes an expression $e$ to $e^{\prime}$, Seader
analyzes data dependencies to locate the security API invoked with $e$ or
$e^{\prime}$, and treats the API _critical_. If there is no update operation
in $E$, Seader searches for any overridden security API or deleted API call.
Next, Seader infers a vulnerability repair pattern and stores it in a JSON
file.

Phase II: Pattern Application:

  * •

Given a program $P$, for each pattern $Pat=<T,F>$ in the JSON file, Seader
searches for any code invoking the critical API. If the API is invoked, Seader
further performs slicing based on the API invocation to locate other
statements matching the remaining part of $T$.

  * •

If $T$ is fully matched by a code snippet, Seader extracts the mappings
between abstract variables and concrete variables, and then customizes the
abstract fix $F$ by replacing abstract variables with concrete ones. In this
way, Seader reports a vulnerability and suggests a customized fix.

###  3.1. Change Recognition

Given an ¡$I$, $S$¿ example pair, Seader performs syntactic program
differencing to identify the edit operation(s) that can change $I$ to $S$.
This step consists of two parts: statement-level change recognition and
expression-level change recognition.

Figure 4. The simplified ASTs of the two statements related to a statement-
level update operation

####  3.1.1. Statement-level change recognition

Seader first adopts JavaParser (Hosseini and Brusilovsky, 2013) to generate
ASTs for $I$ and $S$. Next, Seader compares statement-level AST nodes between
the trees to generate an AST edit script that may contain three types of edit
operations:

  * •

delete (Node $a$): Delete node $a$.

  * •

insert (Node $a$, Node $b$, int $k$): Insert node $a$ and position it as the
$(k+1)^{th}$ child of node $b$.

  * •

update (Node $a$, Node $b$): Replace $a$ with $b$. This operation changes
$a$’s content.

Specifically, when comparing any two statements $s_{i}\in I$ and $s_{s}\in S$,
Seader checks whether the code string of $s_{i}$ exactly matches that of
$s_{j}$; if so, Seader considers $s_{i}$ unchanged while $I$ is transformed to
$S$. Otherwise, if the code strings of $s_{i}$ and $s_{j}$ are different,
Seader normalizes both statements by replacing concrete variables (e.g., key)
with abstract variables (e.g., $v_0), and replacing constant values (e.g.,
"AES") with abstract constants (e.g., $c_0). We denote the normalized
representations as $n_{i}$ and $n_{s}$. Next, Seader computes the Levenshtein
edit distance (Levenshtein, 1966) between $n_{i}$ and $n_{s}$, and computes
the similarity score (Fluri et al., 2007) with:

(1) $sim=1-\frac{edit\\_distance}{max\\_length(n_{i},n_{s})}$

The similarity score $sim$ is within [0, 1]. When $sim=1$, $n_{i}$ and $n_{j}$
are identical. We set a threshold $th=0.8$ such that if $sim>=th$, $n_{i}$ and
$n_{j}$ are considered to be similar enough to match each other. In this way,
Seader can detect updated statements and infer update operation(s). Compared
with string-based match, the normalization-based match is more flexible,
because it can match any two statements that have similar syntactic structures
but distinct variables or constants. Finally, if a statement $s_{i}\in I$ does
not find a match in $S$, Seader infers a delete operation; if $s_{s}\in S$ is
unmatched, Seader infers an insert operation.

####  3.1.2. Expression-level change recognition

When $s_{i}$ is updated to $s_{s}$, there is usually a portion of $s_{i}$
changed. The finer-granularity edit in the statement (e.g., expression
replacement) can help us better interpret changes and conduct pattern
generalization (see Section 3.2). Therefore, for each statement-level update,
Seader further identifies any finer-granularity edit by conducting top-down
matching between the ASTs of $s_{i}$ and $s_{s}$. Specifically, while
traversing both trees in a preorder manner, Seader compares roots and inner
nodes based on the AST node types and compares leaf nodes based on the code
content. Such node traversal and comparison continue until all unmatched
subtrees or leaf nodes are found.

In particular, for the example shown in Section 2, after Seader performs
statement-level change recognition, it reveals one statement update together
with four statement insertions. Figure 4 illustrates the simplified ASTs of
the statements involved in the update. By comparing the nodes of both ASTs in
a top-down manner, Seader detects that the first arguments sent to the
constructor function differ (e.g., MethodCallExpr vs. NameExpr). Therefore,
Seader creates a finer-granularity operation to replace the statement-level
update: update (StringLiterals.CONSTANT.getBytes(), keyBytes).

###  3.2. Pattern Generalization

When security experts present an ¡$I$, $S$¿ example pair to demonstrate any
API misuse, we expect that they provide the minimum code snippets to show only
one vulnerability and its repair. Additionally, based on our experience with
security API misuses, each vulnerability is usually caused by the misuse of
one security API. Therefore, to infer a general vulnerability-repair pattern,
we always analyze the given examples to answer two questions:

  1. 1.

What is the security API whose misuse is responsible for the vulnerability
(i.e., critical API)?

  2. 2.

What is the relationship between the critical API and its surrounding code?

####  3.2.1. Task 1: Identifying the critical API

Starting with the edit script $E$ created in Section 3.1, Seader looks for any
update operation $update(e,e^{\prime})$. If there is such an operation, Seader
searches for the security API whose invocation is data-dependent on $e$ or
$e^{\prime}$, and considers the API to be _critical_. For the example shown in
Figure 4, the critical API is SecretKeySpec(byte[], String) because it is
invoked with the updated expression as the first argument. Similarly, Figure 5
presents another example where a numeric literal is updated from 4 to 8. With
data dependency analysis, Seader reveals that the constants are used to define
variable salt, while salt is used as an argument when PBEParameterSpec(...) is
invoked. Therefore, the method invocation depends on the updated expression,
and the security API PBEParameterSpec(byte[], int) is considered _critical_.

Insecure code (_I_)
---
⬇ 1void test(int iterations) { 2 byte[] salt = new byte[4]; 3
AlgorithmParameterSpec paramSpec = new PBEParameterSpec(salt, iterations); 4}
Secure code (_S_)
⬇ 1void test(int iterations) { 2 byte[] salt = new byte[8]; 3
AlgorithmParameterSpec paramSpec = new PBEParameterSpec(salt, iterations); 4}
Figure 5. A pair of examples where the updated constant is indirectly depended
on by the invocation of critical API PBEParameterSpec(byte[], int)

If there is no update operation in $E$, Seader searches for any overridden
security API that encloses all edit operations, and considers the overridden
API to be _critical_. Take the code pair shown in Figure 6 as an example. By
comparing $I$ with $S$, Seader can identify one statement deletion and
multiple statement insertions. As there is no update operation and all edit
operations are enclosed by an overridden method verify(String, SSLSession)
(indicated by @Override), Seader further locates the interface or super class
declaring the method (e.g., HostnameVerifier). If the overridden method
together with the interface/super class matches any known security API, Seader
concludes the overridden method to be a _critical API_.

Lastly, if no update operation or overridden security API is identified,
Seader checks whether there is any deletion of security API call in $E$; if
so, the API is _critical_. To facilitate later template matching (Section
3.3), for each identified critical API, Seader records the method binding
information (e.g., javax.crypto.spec.SecretKeySpec. SecretKeySpec(byte[],
String)).

Insecure code (_I_)
---
⬇ 1public class HostVerifier implements HostnameVerifier { 2 @Override 3
public boolean verify(String hostname, SSLSession sslSession){ 4 return true;
5 }}
Secure code (_S_)
⬇ 1public class HostVerifier implements HostnameVerifier { 2 @Override 3
public boolean verify(String hostname, SSLSession sslSession){ 4 //Please
change ”example.com” as needed 5 if (”example.com”.equals(hostname)) { 6
return true; 7 } 8 HostnameVerifier hv =
HttpsURLConnection.getDefaultHostnameVerifier(); 9 return hv.verify(hostname,
sslSession); 10 }}
Figure 6. A pair of examples from which Seader infers the critical API to be
an overridden method

####  3.2.2. Task 2: Extracting relationship between the critical API and its
surrounding code

When a vulnerable code example contains multiple statements (e.g., Figure 5
and Figure 6), we were curious how the critical API invocation is related to
other statements. On one extreme, if the invocation is irrelevant to all
surrounding statements, we should not include any surrounding code into the
generalized pattern. On the other extreme, if the invocation is related to all
surrounding code, we should take all code into account when inferring a
vulnerability-repair pattern. Thus, this task intends to decide (1) which
statements of $I$ to include into the vulnerable code template, (2) what other
security API(s) whose invocations should be analyzed for template matching
(see Section 3.3), and (3) which statements of $S$ to include into the
abstract fix.

Specifically, Seader performs intra-procedural data dependency analysis. If
any statement defines a variable value that is (in)directly used by the
critical API invocation, the statement is extracted as _edit-relevant
context_. Seader relies on such context to characterize the demonstrated
vulnerability. For the insecure code $I$ shown in Figure 5, because the API
call (line 3) data-depends on variable salt, lines 2-3 are extracted as the
context. Additionally, when the critical API is an overridden method, its code
implementation in $I$ is considered as edit-relevant context (see lines 3-5 in
Figure 6). Based on the extracted edit-relevant context, Seader abstracts all
variables to derive a vulnerable code template $T$, and records mappings $M$
between abstract and concrete variables.

In addition to the critical API, Seader also extracts binding information for
any other security API invoked by the context code. Compared with edit-
relevant context, these APIs provide more succinct hints of code
vulnerabilities. In our later template matching process, these APIs can serve
as “ _anchors_ ” for Seader to efficiently decide whether a program slice
invokes all relevant APIs or is worth further comparison with the template
$T$.

To determine the fix-relevant code in secure version $S$, Seader identifies
any unchanged code in the edit-relevant context, the inserted statements, and
the new version of any updated statement. For the secure code $S$ shown in
Figure 5, lines 2-3 are fix-relevant, because line 2 is the new version of an
updated statement and line 3 is unchanged context code. Similarly, for the
secure code $S$ shown in Figure 6, lines 3-10 are fix-relevant, because lines
3 and 10 present the critical API while lines 4-9 are inserted statements.
Based on the above-mentioned variable mappings $M$ and fix-related code,
Seader further abstracts variables used in the fix-related code to derive an
abstract fix $F$. Seader ensures that the same concrete variables used in $I$
and $S$ are always consistently mapped to the same abstract variables recorded
in $M$.

To sum up, for each given ¡$I$, $S$¿ pair, Seader produces a pattern
$Pat=<T,F>$, which consists of a vulnerable code template $T$, an abstract fix
$F$, and metadata to describe $T$ (i.e., bindings of the critical API and
other invoked security APIs).

Input: P, T, D /* program, template, and related metadata */

Output: Matched /* a set of code matches from P to T */

Candi $:=$ $\emptyset$, Matched $:=\emptyset$;

/* 1\. search for matches of the critical API */

foreach  _code line x $\in$ P_ do

if  _x invokes D(critical) —— x declares D(critical)_ then

Candi := Candi $\cup$ x;

end if

end foreach

foreach  _x $\in$Candi_ do

if  _x invokes D(critical)_ then

/* 2(a). For API call, do program slicing and look for matches of other
security APIs */

Sli = getBackwardSlice(x);

if  _(Sli has all matches for D(other)) == false_ then

continue;

end if

/* 3\. check whether the data dependencies between security APIs in T match
those in Sli */

if  _dataDependConsist(T, Sli)_ then

Matched:=Matched $\cup$ {Sli, mappings};

end if

end if

else

/* 2(b). For API overriding, check the code */

if  _contentMatch(code(P, x), T)_ then

Matched := Matched $\cup$ {code(P, x), mappings};

end if

end if

end foreach

Algorithm 1 Matching Program P to template T

###  3.3. Template Matching

Given a program $P$, Seader uses a static analysis framework—WALA (wal,
2020)—to analyze the JAR file (i.e., byte code) of $P$. As shown in Algorithm
1, to find any code in $P$ that matches the template $T$, Seader first
searches for the critical API (i.e., invocation or method reimplementation).
If the critical API does not exist, Seader concludes that there is no match
for $T$. Next, if the critical API is invoked at least once, for each
invocation, Seader conducts inter-procedural backward slicing to retrieve all
code $Sli$ on which the API call is either control- or data- dependent (i.e.,
getBackwardSlice(x)). When $T$ invokes one or more security APIs in addition
to the critical API, Seader further examines whether $Sli$ contains matches
for those extra APIs; if not, the matching trial fails. Next, Seader checks
whether the matched code in $Sli$ preserves the data dependencies manifested
by $T$ (i.e., dataDependConsist(T, Sli). If those data dependencies also
match, Seader reveals a vulnerability.

Alternatively, if the critical API is reimplemented, for each
reimplementation, Seader compares the code content against $T$, and reports a
vulnerability if they match. At the end of this step, if any vulnerability is
detected, Seader presents the line number where the critical API is invoked or
is declared as an overridden method, and shows related matching details. The
matching details include both code matches and abstract-concrete variable
mappings.

###  3.4. Fix Customization

This step involves two types of customization: variable customization and edit
customization. To customize variables, based on the matching details mentioned
in Section 3.3, Seader replaces abstract variables in $F$ with the
corresponding concrete ones. We denote this customized version as $F_{c}$. For
edit customization, Seader suggests code replacements in two distinct ways
depeding on the inferred edit operations mentioned in Section 3.1.
Specifically, if there is only one update operation inferred, Seader simply
recommends an alternative expression to replace the original expression.
Otherwise, Seader presents $F_{c}$ for developers to consider.

Notice that Seader does not directly modify $P$ to automatically repair any
vulnerability for two reasons. First, when template $T$ contains multiple
statements, it is possible that the corresponding code match involves
statements from multiple method bodies. Automatically editing those statements
can be risky and can unpredictably impact program semantics. Second, some
fixes require for developers’ further customization based on their software
environments (e.g., network configurations, file systems, and security
infrastructures). As implied by Figure 6, the abstract fix derived from $S$
will contain a comment "//Please change ’example.com’ as needed", so will the
customized fix by Seader. This comment instructs developers to replace the
standard hostname based on their needs.

###  3.5. Specialized Handling for Certain Patterns

We believe that by crafting ¡$I$, $S$¿ code pairs, security researchers can
intuitively demonstrate the misuse and correct usage of security APIs.
However, we also noticed some scenarios where plain Java examples cannot
effectively indicate the vulnerability-repair patterns. To solve this problem,
we defined a Java library named EDL for user adoption and invented special
ways of example definition. Currently, EDL has two classes: StringLiterals and
IntLiterals, which provide APIs to help specify constant-related examples.
With the library support, Seader allows users to specially define examples for
three typical scenarios:

##### Scenario 1: A pattern involves the usage of any constant value instead
of a particular constant.

Plain examples only show the usage of particular constant values, but cannot
generally represent the constant concept. Consider the vulnerability
introduced in Section 2. Without using StringLiterals.CONSTANT, a domain
expert has to define a plain example to show the API misuse, such as

SecretKey key = new SecretKeySpec("ABCDE".getBytes(), "AES");

Seader is designed to preserve all string literals from $I$ when generalizing
template $T$, and to look for those values when matching code with $T$.
Consequently, given the above-mentioned example, Seader will inevitably embed
"ABCDE" into the inferred template. To help users avoid such unwanted literal
values in $T$, EDL provides StringLiterals.CONSTANT and IntLiterals.CONSTANT.
These APIs can be used as placeholders for string and integer constants to
represent any arbitrary value in examples. When Seader identifies such special
EDL APIs in examples, it keeps the APIs in $T$ and later specially matches
them with constants in $P$.

##### Scenario 2: A pattern has multiple alternative insecure (or secure)
options

Given a parameter of certain security API, suppose that there are (1) $m$
distinct values to cause API misuse and (2) $n$ other alternatives to ensure
security, where $m\geq 1$, $n\geq 1$. To enumerate all possible combinations
between the vulnerable code templates and repair options, users have to
provide $m\times n$ pairs of plain examples, which practice is inefficient. To
solve this issue, we defined another two method APIs in StringLiterals. As
shown in Figure 7, one API is a constructor of StringLiterals, which can take
in any number of string literals as arguments (see line 1 in $I$) and store
those values into an internal list structure. The other API is getAString(),
which randomly picks and returns a value from that list (see line 2 in $I$).
With these two APIs, a domain expert can efficiently illustrate multiple
secure/insecure options in just one code pair.

Insecure code (_I_)
---
⬇ 1StringLiterals literals=new StringLiterals(”AES”, ”RC2”, ”RC4”, ”RC5”,
”DES”, ”blowfish”, ”DESede”); 2Cipher.getInstance(literals.getAString());
Secure code (_S_)
⬇ 1StringLiterals literals = new StringLiterals(”AES/GCM/PKCS5Padding”,”RSA”,
”ECIES); 2Cipher.getInstance(literals.getAString());”
Figure 7. A code pair where multiple alternative secure and insecure options
are specified simultaneously

The examples in Figure 7 show that when security API Cipher. getInstance(...)
is called, the insecure parameter values include "AES", "RC2", "RC4", "RC5",
"DES", "blowfish", and "DESede". Each insecure value should be replaced by
either "AES/GCM/PKCS5Padding", "RSA", or "ECIES". Given such examples, Seader
extracts insecure/secure options from StringLiterals-related statements,
detects vulnerabilities in $P$ if the security API is invoked with any
insecure option, and suggests fixes by randomly picking one of the secure
options.

##### Scenario 3: A pattern requires for certain value range of a variable.

Given an integer parameter $p$ of certain security API, suppose that there is
a threshold value $th$ such that the API invocation is secure only when $p\geq
th$. To enumerate all possible vulnerable cases and related repairs via plain
examples, theoretically, a user should provide $(th-
Integer.MIN\\_VALUE)\times(Integer.MAX\\_VALUE-th+1)$ code pairs, which
practice is cumbersome and unrealistic. Therefore, we invented a special way
of example definition for such scenarios, which requires users to provide only
(1) one insecure example by setting $p$ to a concrete value less than $th$ and
(2) one secure example by setting $p=th$. As shown in Figure 5, if a security
expert wants to describe the pattern that _the array size of the first
parameter used in PBEparameterSpec(byte[], int) should be no less than 8_,
then he/she can define $I$ by creating an array with a smaller size (i.e., 4)
and define $S$ by setting the array size to 8. Seader was developed to
identify the usage of distinct integer literals between $I$ and $S$, and to
infer a secure value range accordingly.

##  4\. Evaluation

This section first introduces the data sets and metrics used for evaluation
(Sections 4.1-4.2). It then presents Seader’s effectiveness of pattern
inference (Section 4.3). Next, it describes Seader’s capabilities of
vulnerability detection (Section 4.4) and repair suggestion (Section 4.5).
Essentially, Sections 4.4 and 4.5 reflect Seader’s effectiveness of pattern
application.

###  4.1. Data Sets

There are two types of data used: data to evaluate pattern inference, and data
to evaluate pattern application.

Table 1. The API misuses and related fixes summarized by prior work (Mendel et al., 2013; tls, 2016; Fischer et al., 2017; Rahaman et al., 2019) Idx |  Security Class API |  Insecure |  Secure
---|---|---|---
1 |  Cipher |  The algorithm and/or mode is set as RC2, RC4, RC5, DES, DESede, AES/ECB, or Blowfish. |  The algorithm and/or mode is set as RSA, GCM, AES, or ECIES
2 |  HostnameVerifier |  Allow all hostnames. |  Implement logic to actually verify hostnames.
3 |  IvParameterSpec |  Create an initialization vector (IV) with a constant. |  Create an IV with a random value.
4 |  KeyPairGenerator |  Create an RSA key pair where key size ¡ 2048 bits or create an ECC key pair where key size ¡ 224 bits |  RSA key size ¿= 2048 bits, ECC key size ¿= 224 bits
5 |  KeyStore |  When loading a keystore from a given input stream, the provided password is a hardcoded constant non-null value. |  The password is retrieved from some external source (e.g., database or file)
6 |  MessageDigest |  The algorithm is MD2, MD5, SHA-1, or SHA-256. |  The algorithm is SHA-512.
7 |  PBEKeySpec |  Create a PBEKey based on a constant salt. |  Use a randomly generated salt value to create the key.
8 |  PBEParameterSpec |  Create a parameter set for password-based encryption (PBE) by setting salt size ¡ 64 bits or iteration count ¡ 1000 |  Salt size ¿= 64 bits, iteration count ¿=1000.
9 |  SecretKeyFactory |  Create secret keys with algorithm DES, DESede, ARCFOUR, PBEWithMD5AndDES, or PBKDF2WithHmacSHA1. |  Create secret keys with AES or PBEWithHmacSHA256AndAES_128.
10 |  SecretKeySpec |  Create a secret key with a constant value. |  Create a secret key with a randomly generated value.
11 |  SecureRandom |  Use Random to generate random values, or set SecureRandom to use a constant seed. |  Use SecureRandom instead of Random, and ensure the seed to be a random value.
12 |  SSLContext |  Protocol version ¡ TLSv1.3 |  Protocol version ¿= TLSv1.3
13 |  TrustManager |  Trust all clients or servers |  Check clients and/or check servers.

####  4.1.1. A data set to evaluate pattern inference

Prior research summarized a number of security API misuses and related correct
usage in Java (Sheffer et al., 2015; Kaliski, 2000; Manger, 2001; Fahl et al.,
2012; Mendel et al., 2013; tls, 2016; Fischer et al., 2017; Meng et al., 2018;
Chen et al., 2019; Rahaman et al., 2019). To evaluate Seader’s effectiveness
of pattern inference, we referred to some well-described API misuses and fixes
to craft code examples as inputs for Seader. With more details, Table 1 lists
the 13 security class APIs we focused on, the insecure usage of certain method
API(s) defined by these classes, and the related secure usage. With this
domain knowledge, we manually created a set of 25 ¡insecure, secure¿ code
pairs.

####  4.1.2. Two data sets to evaluate pattern application

The first data set contains 32 real vulnerabilities from 10 Apache open-source
projects. To create this set, the second author randomly picked 10 Apache
projects and searched for any security API usage based on keywords. For each
invoked or overridden security API, the second author leveraged his domain
knowledge to manually analyze the program context and to decide whether the
API is misused. Once the data set was created, another two authors checked the
data to decide whether all included vulnerabilities were true positives; if
any false positive was revealed, all three authors discussed until reaching an
agreement. Notice that none of these three authors has experience with Seader.
Because the data set was created in a tool-agnostic way, we can use it to
objectively evaluate Seader’s capability of vulnerability detection. We will
also use this data to evaluate the effectiveness of the state-of-the-art
vulnerability detection tool—CryptoGuard (Rahaman et al., 2019), and compare
Seader with CryptoGuard.

The other data set contains 100 widely used Apache open-source projects. We
decided to download Apache projects for three reasons. First, they are well-
maintained and have good code quality; it can be hard for us to identify
vulnerabilities in these projects. Second, the project developers are usually
responsive to the pull requests or code changes suggested by other developers.
Third, Apache projects are usually widely used, so any API misuses found in
those projects are important and may negatively impact many other projects
that depend on them. Therefore, to create this data set, we first ranked the
Apache projects available on GitHub (gh, 2020) in the descending order of
their stars. Namely, the more stars a project $Prj$ has, the more likely that
$Prj$ is popular, and the higher $Prj$ is ranked. Next, we located the top 100
projects that meet the following two criteria:

  * •

Each project uses at least one security API that Seader examines.

  * •

The project is compilable because Seader analyzes the compiled JAR files to
reveal vulnerabilities.

This data set is used to mainly evaluate Seader’s effectiveness of repair
suggestion.

###  4.2. Metrics

As with prior work (Rahaman et al., 2019), we leveraged the following three
metrics to measure tools’ capability of vulnerability detection:

_Precision (P)_ measures among all reported vulnerabilities, how many of them
are true vulnerabilities.

(2) $P=\frac{\text{\\# of correct reports}}{\text{Total \\# of reports}}\times
100\%$

_Recall (R)_ measures among all known vulnerabilities, how many of them are
detected by a tool.

(3) $R=\frac{\text{\\# of correct reports}}{\text{Total \\# of known
vulnerabilities}}\times 100\%$

_F-score (F)_ is the harmonic mean of precision and recall, and is used to
measure the overall detection accuracy of an approach:

(4) $F=\frac{2\times P\times R}{P+R}\times 100\%$

###  4.3. Effectiveness of Pattern Inference

Table 2. Evaluation results on the 32-vulnerability set Apache Project |  Manual | Seader | CryptoGuard
---|---|---|---
|  Ground Truth (GT) |  # of Reported Vulnerabilities |  # of Correct Reports |  P(%) |  R% |  F(%) |  # of Reported Vulnerabilities |  # of Correct Reports |  P(%) |  R% |  F(%)
apacheds-kerberos-codec-2.0.0.AM25.jar |  10 |  12 |  12 |  100 |  100 |  100 |  12 |  12 |  100 |  100 |  100
artemis-commons-2.11.0.jar |  3 |  4 |  4 |  100 |  100 |  100 |  4 |  4 |  100 |  100 |  100
deltaspike-core-impl-1.9.2.jar |  1 |  2 |  2 |  100 |  100 |  100 |  2 |  2 |  100 |  100 |  100
flume-file-channel-1.9.0.jar |  1 |  3 |  3 |  100 |  100 |  100 |  3 |  3 |  100 |  100 |  100
hadoop-hdfs-3.2.1.jar |  5 |  5 |  5 |  100 |  100 |  100 |  5 |  5 |  100 |  100 |  100
jclouds-core-2.2.0.jar |  2 |  7 |  7 |  100 |  100 |  100 |  5 |  5 |  100 |  100 |  100
shiro-crypto-cipher-1.5.0.jar |  0 |  0 |  0 |  100 |  - |  - |  0 |  0 |  100 |  - |  -
wagon-provider-api-3.3.4.jar |  0 |  1 |  1 |  100 |  - |  - |  1 |  1 |  100 |  - |  -
wss4j-ws-security-common-2.2.4.jar |  5 |  5 |  5 |  100 |  100 |  100 |  5 |  5 |  100 |  100 |  100
wss4j-ws-security-stax-2.2.4.jar |  5 |  0 |  0 |  - |  - |  - |  5 |  5 |  100 |  100 |  100
Overall |  32 |  39 |  39 |  100 |  84 |  91 |  42 |  42 |  100 |  100 |  100
When # of Reported Vulnerabilities ¿ Manual Ground Truth, we manually checked
each additional report to decide the correctness and calculated precision
accordingly.

As mentioned in Section 4.1, we crafted 25 code pairs to evaluate Seader’s
effectiveness of pattern inference. To facilitate explanation, we categorized
these pairs by checking two conditions:

  1. C1.

Do $I$ and $S$ contain single or multiple statements?

  2. C2.

Does the pattern generalization involve variable abstraction?

The two conditions actually reflect the difficulty levels or challenges of
these pattern inference tasks. For instance, if $I$ or $S$ has multiple
statements, Seader conducts data-dependency analysis to locate the edit-
relevant context in $I$ or to reveal the fix-relevant code in $S$. If $I$ or
$S$ uses variables, Seader abstracts all variable names to ensure the general
applicability of inferred patterns. As shown in Table 3, there are five
simplest pairs that can be handled by Seader without conducting any data-
dependency analysis or identifier generalization. Meanwhile, there are 14 most
complicated cases that require for both data-dependency analysis and
identifier generalization.

Table 3. The 25 code pairs for pattern inference |  Single statement |  Multiple statements
---|---|---
Identical |  5 |  5
Abstract |  1 |  14

In our evaluation, Seader correctly inferred patterns from all pairs. When
some pairs illustrate distinct secure/insecure options (e.g., distinct string
literals) for the same critical API, Seader merged the inferred patterns
because they demonstrate misuses of the same security API. In this way, Seader
derived 18 unique patterns.

Finding 1: _Our experiment with 25 code pairs shows Seader’s great capability
of pattern inference; it also indicates the great potential of example-based
vulnerability detection and repair._
---

###  4.4. Effectiveness of Vulnerability Detection

As described in Section 4.1.1, the 32 vulnerabilities were manually identified
in 10 randomly selected Apache projects. In particular, as shown in Table 2,
the manual ground truth set (GT) actually includes vulnerable code from eight
projects, but covers no vulnerability in two projects: shiro-crypto-
cipher-1.5.0.jar and wagon-provider-api-3.3.4.jar; thus, we were unable to
calculate tools’ recall rates based on GT for the two projects. Seader and
CryptoGuard separately revealed vulnerabilities in eight and nine projects.

For six projects (i.e., apacheds-kerberos-codec-2.0.0.AM25.jar, artemis-
commons-2.11.0.jar, deltaspike-core-impl-1.9.2.jar, flume-file-
channel-1.9.0.jar, jclouds-core-2.2.0.jar, and wagon-provider-api-3.3.4.jar),
both tools revealed more vulnerabilities than GT. Because GT can be
incomplete, it may miss some true vulnerabilities that either tool can
identify. To properly evaluate the precision rates in such scenarios, we first
intersected the tool-generated reports with GT, and then manually examined any
unmatched report for its correctness. Take artemis-commons-2.11.0.jar as an
example. Among the four vulnerabilities reported by Seader, three of them
match the known ones in GT; thus, we further checked the remaining one and
confirmed the vulnerability. In this way, the precision of Seader for the
project is 4/4$\times$100%=100%. We observed that all reports generated by
Seader and CryptoGuard are true positives, so both tools achieved 100%
precision.

For project jclouds-core-2.2.0.jar, Seader identified more vulnerabilities
than CryptoGuard (7 vs. 5), mainly because Seader checks for more insecure
options of the security API SSLContext.getInstance( String protocol).
Meanwhile, CryptoGuard identified more vulnerabilities in the project
wss4j-ws-security-stax-2.2.4.jar than Seader (5 vs. 0). This is mainly due to
the limitation of WALA—the static analysis framework used by Seader. When WALA
builds a call graph for the whole program $P$, it requires that all library
dependencies of $P$ should exist as JAR files in the class path. If some
dependencies are missing, the call graph built by WALA can be incomplete.
Consequently, when Seader traversed all methods in an incomplete graph $CG$,
it missed the API misuses of the methods that are excluded from $CG$.

Overall, given 10 open-source projects, Seader reported 39 vulnerabilities and
achieved 100% precision, 84% recall, and 91% accuracy. CryptoGuard detected 42
vulnerabilities and achieved 100% precision, 100% recall, and 100% accuracy.
The comparison shows that Seader effectively detected API-related
vulnerabilities, and has comparable performance with CryptoGuard in most
projects. In the future, we will further improve Seader by expanding its
pattern set and overcoming the limitation due to WALA.

Finding 2: _Given 10 open-source projects, Seader worked as effectively as the
state-of-the-art tool CryptoGuard in 8 projects. This indicates Seader’s great
capability of vulnerability detection. _
---

###  4.5. Effectiveness of Repair Suggestion

Table 4. The 988 vulnerabilities found in 100 Apache projects Security Class API | # of Detected Vulnerabilities | # of Suggested Fixes
---|---|---
Cipher | 23 | 23
HostnameVerifier | 68 | 68
IvParameterSpec | 6 | 6
KeyPairGenerator | 1 | 1
KeyStore | 10 | 10
MessageDigest | 147 | 147
PBEKeySpec | 8 | 8
PBEParameterSpec | 6 | 6
SecretKeyFactory | 12 | 12
SecretKeySpec | 11 | 11
SecureRandom | 422 | 422
SSLContext | 26 | 26
TrustManager | 248 | 248
Total | 988 | 988

By applying Seader to the second data set mentioned in Section 4.1.2, Seader
reported 988 real vulnerabilities in 100 open-source projects. About 43% of
these vulnerabilities (i.e., 422) are related to SecureRandom APIs; 25% of
vulnerabilities are about TrustManager API misuses; and 15% of vulnerabilities
are relevant to MessageDigest. To assess the quality of Seader’s fix
suggestions, we manually checked 100 fixes suggested by Seader. To ensure the
representativeness of our manual inspection results, we selected the 100 fixes
to cover at least one vulnerability for each security class API. Based on the
our inspection, the 100 fixes are all correctly customized for program
contexts. It means that Seader can properly suggest fixes.

To further investigate how developers assess the quality of Seader’s outputs,
we picked 59 vulnerable Java classes reported by Seader and filed 59 pull
requests (PRs) in the project repositories. In each PR, we (1) described the
vulnerable code, (2) suggested the fix(es), (3) explained the security
implication of each vulnerability, and (4) asked for developers’ opinions on
the program issues and solutions. Up till now, we have received developers’
feedback for 25 PRs. As shown in Table 5, developers confessed the revealed
vulnerabilities in 14 PRs, needed us to provide more clarification information
for 3 PRs, and believed the reports to be security-irrelevant for 8 PRs. More
importantly, for 8 of the 14 PRs, developers took actions to either fix the
vulnerabilities, disable the insecure implementation, or at least leave
comments to warn people of the insecure code.

Table 5. Developers’ opinions on Seader’s outputs Feedback | # of Pull Requests
---|---
Confess vulnerabilities | 14
Need more clarification | 3
Believe to be irrelevant | 8
Try to resolve the security issues | 6

Interestingly, some developers rejected our repair suggestions for three major
reasons. First, several developers would like to see real attacks to exploit
those vulnerabilities (e.g., TrustManager trusts all). Particularly, two
developers emphasized that the vulnerabilities (e.g.,
MessageDigest.getInstance("MD5") exist in test cases, which implies that these
vulnerabilities cannot lead to real attacks. Second, some developers are
reluctant to introduce API breaking changes by removing the insecure
functionalities from their software ($P$), because they are afraid that the
client projects based on their software ($CP$) can be negatively influenced
(e.g., getting compilation errors). Therefore, these developers rely on the
wisdom of $CP$ programmers to (1) identify the insecure options provided by
$P$ and (2) extend $P$ to implement stronger security protection as needed.

Third, certain developers believe that some reported API misuses are not
necessarily insecure. For instance, one developer explained that his Java
method computeSha256(File) invokes MessageDigest.getInstance("SHA-256") just
to generate a message digest for any given file. This functionality has
nothing to do with security.

Finding 3: _Seader revealed 988 vulnerabilities in 100 open-source projects.
By filing 59 PRs based on some of Seader’s outputs, we identified a mixture of
developers’ opinions on the revealed vulnerabilities and suggested fixes. _
---

##  5\. Related Work

The related work includes automatic program repair, detection of security API
misuses, and example-based program transformation.

###  5.1. Automatic Program Repair (APR)

Tools were proposed to generate candidate patches for certain bugs, and to
automatically check patch correctness using compilation and testing (Mechtaev
et al., 2016; Long and Rinard, 2016; Qi et al., 2014; DeMarco et al., 2014;
Kim et al., 2013; Le Goues et al., 2012). Such approaches usually make two
assumptions. First, software bugs can trigger some test failure(s). Second,
automatic fault localization techniques (Jones and Harrold, 2005; Lucia et
al., 2010; Dao et al., 2017) can be adopted to locate bugs in source code
based on the execution coverage of passed and failed tests. With such
assumptions, GenProg (Le Goues et al., 2012) and RSRepair (Qi et al., 2014)
create candidate patches by replicating, mutating, or deleting code randomly
from the existing buggy program. To further improve the quality of created
patches, PAR (Kim et al., 2013) and Prophet (Long and Rinard, 2016) prioritize
patch generation based on the frequently applied bug fixes by developers. Each
patch generated by current APR tools can only edit one or two lines of code
(e.g., changing one or two numeric values or adding an if-condition check).

Existing APR tools are not applicable to vulnerability detection and repair
for three reasons. First, their assumptions do not hold. Vulnerable code is
functionally correct and rarely fails any test case, so APR tools cannot rely
on test failures to reveal vulnerabilities or assess applied fixes. Second,
given a security API (e.g., MessageDigest.getInstance(...)), the secure
parameter options can be totally different from insecure ones (e.g., SHA-512
vs. MD1). Therefore, it is almost impossible for APR tools to suggest secure
options based on the existing codebases with solely insecure options. Third,
some security patches require for significant code modification (e.g.,
inserting more than 10 lines of code), but APR tools are unable to generate
such complex edits.

###  5.2. Detecting of Security API Misuses

Researchers built various tools to detect security vulnerabilities related to
API misuses (Fahl et al., 2012; Egele et al., 2013; He et al., 2015; Fischer
et al., 2017; Krüger et al., 2018; Fischer et al., 2019; Rahaman et al.,
2019). Specifically, most tools conduct static program analysis to check
programs against hardcoded templates of vulerabilities (Fahl et al., 2012;
Egele et al., 2013; He et al., 2015; Rahaman et al., 2019). For instance,
MalloDroid scans the decompiled code of Android apps to a) extract the
networking API calls and valid HTTP(S) URLs, b) check the validity of the SSL
certificates of all extracted HTTPS hosts, and c) identify apps that invoke
APIs to customize SSL usage (Fahl et al., 2012). Some other tools adopt
machine learning (ML)-based approaches to identify vulnerabilities (Fischer et
al., 2017; Fischer et al., 2019). For instance, Fischer et al. (Fischer et
al., 2017) transformed secure/insecure code corpus into numeric vectors, and
used support vector machine (SVN) to train a binary-class classifier based on
those vectors. Next, given arbitrary code $C$, the classifier predicts whether
C is insecure. Additionally, CrySL is a domain-specific language (DSL) that
developers can use to describe vulnerable usage of security APIs (Krüger et
al., 2018).

However, it is not easy to extend the above-mentioned tools to detect new
vulnerabilities. For instance, the tools based on hardcoded rules often
require users to modify the tool implementation; ML-based tools may require
for a large amount of labeled secure and insecure code samples for classifier
retraining; DSL-based tools require users to learn new languages for pattern
prescription, while the learning curve may be high for some security
researchers or developers. In comparison, Seader has great extensibility
because it only requires users to demonstrate vulnerabilities and related
repairs with a small number of Java code examples. By inferring patterns from
those examples, Seader gains the capability of detecting new vulnerabilities
and proposing related fixing suggestions.

###  5.3. Example-Based Program Transformation

Based on the insight that developers modify similar code in similar ways,
researchers developed tools to infer program transformations from exemplar
edits and to manipulate code or suggest changes accordingly (Miller and Myers,
2001; Toomim et al., 2004; Duala-Ekoko and Robillard, 2007; Meng et al., 2013;
Ma et al., 2017; Rolim et al., 2017; An et al., 2018; Xu et al., 2019). For
instance, while users interactively edit code in one program context,
simultaneous text editing approaches apply identical textual edits in other
preselected program contexts at the same time (Miller and Myers, 2001; Toomim
et al., 2004; Duala-Ekoko and Robillard, 2007). Given one or multiple code
change examples, LASE (Meng et al., 2013) and REFAZER (Rolim et al., 2017)
infer a program transformation from the examples; they then use the
transformation to locate similar code to edit, and apply customized
transformations to those locations. TINFERER generalizes language translation
rules from exemplar Java code and the equivalent implementation by Swift; it
then applies the Java-to-Swift rules to translate any given Java program into
Swift (An et al., 2018). The most closely related work to Seader is VuRLE (Ma
et al., 2017), which also detects and repairs vulnerabilities based on
¡insecure, secure¿ code examples.

However, simultaneous editing cannot identify edit locations or apply
customized edits. The other approaches only adopt (1) token-based or AST-based
matching and/or (2) simple intra-procedural analysis to identify edit
locations; they are insufficient for two reasons. First, token-based or AST-
based statement matching is inflexible. When an edit location has statements
formatted differently from the exemplar code (compare Figure 2($I$) with
Listing 1), current tools can miss the location. Second, intra-procedural
analysis only finds code matches in single methods; it does not go beyond
method boundaries or reveal any edit location that involves code from multiple
methods. In comparison, Seader conducts inter-procedural analysis and program
slicing to overcome the limitations. Actually, we also contacted the authors
of VuRLE to ask for their programs and data, but did not get the materials to
empirically compare Seader with VuRLE.

##  6\. Threats to validity

All inferred patterns and detected vulnerabilities are limited to our
experiment data sets. The observations may not generalize well to close-source
projects. In the future, we plan to include more ¡insecure, secure¿ examples
or even close-source projects into our evaluation, so that our findings are
more representative.

We evaluated the recall rates of Seader and CryptoGuard based on a set of 32
vulnerabilities manually found in 10 open-source projects. This ground truth
set is subject to human bias, and our data set constructor accidentally
overlooked some actual vulnerabilities in those projects. Such incomplete
ground truth can influence recall evaluation. In the future, we plan to
recruit more security experts to simultaneously inspect the same codebases, so
that the built ground truth set is more complete.

In some repair suggestions provided by Seader, there are placeholders that we
need developers to further customize (see “//Please change ‘example.com’ as
needed” in Figure 6). Such placeholders should be filled based on developers’
unique software environments (e.g., host name settings), or sometimes even
require developers to do extra configurations outside the codebase (e.g.,
installing SSL certificates). Therefore, the current repair suggestions by
Seader may sometimes seem vague and abstract. In the future, we plan to
provide more detailed suggestions and explore interactive approaches that
guide developers to apply complete repairs step-by-step.

##  7\. Conclusion

Security is important for software quality assurance. To help developers
better protect their software based on the state-of-the-art security research,
in this paper, we present Seader—a new approach that takes in ¡insecure,
secure¿ code examples, generalizes vulnerability-repair patterns from
examples, and applies those patterns for vulnerability detection and repair
suggestion. Compared with prior work, Seader lowers the technical barriers for
security experts to illustrate their domain knowledge, and concretizes
security expertise as customized code edits for software practitioners. In
this way, Seader intends to bridge the gap between security research and
software development.

Based on our evaluation results, Seader is capable of inferring various
patterns from diverse code examples; it detects real vulnerabilities and
suggests fixes in open-source projects with high accuracy. More importantly,
we filed 59 pull requests (PRs) to seek for developers’ opinions on the
revealed vulnerabilities and fix suggestions by Seader. Interestingly,
responders of 14 PRs agreed upon the revealed vulnerabilities; for 8 of these
PRs, responders took actions to mitigate or eliminate those security issues.
Meanwhile, responders of another eight PRs disliked Seader’s outputs due to
(1) the fear of breaking their software specifications when applying security
patches, (2) expectation for real exploits of those vulnerabilities, (3)
belief in software users to consciously adopt vulnerable code for good reasons
(e.g., quick prototyping), and (4) disagreement on certain security API
misuses. Our findings show interesting opinion contrasts between security
research and software practices. In the future, we will investigate approaches
to automate security attacks based on revealed vulnerabilities.

## References

  * (1)
  * tls (2016) 2016\.  SLOTH: TLS 1.2 vulnerability (CVE-2015-7575).  https://access.redhat.com/articles/2112261.
  * dev (2017) 2017\.  Developers lack skills needed for secure DevOps, survey shows.  https://www.computerweekly.com/news/450424614/Developers-lack-skills-needed-for-secure-DevOps-survey-shows.
  * too (2019) 2019\.  Too few cybersecurity professionals is a gigantic problem for 2019.  https://techcrunch.com/2019/01/27/too-few-cybersecurity-professionals-is-a-gigantic-problem-for-2019/.
  * gh (2020) 2020\.  GitHub.  https://github.com.
  * jss (2020) 2020\.  Java Secure Socket Extension (JSSE) Reference Guide.  https://docs.oracle.com/javase/9/security/java-secure-socket-extension-jsse-reference-guide.htm.
  * so (2020) 2020\.  StackOverflow.  https://stackoverflow.com.
  * wal (2020) 2020\.  WALA IR.  https://github.com/wala/WALA/wiki/Intermediate-Representation-(IR).
  * An et al. (2018) Kijin An, Na Meng, and Eli Tilevich. 2018.  Automatic Inference of Java-to-Swift Translation Rules for Porting Mobile Applications. In _Proceedings of the 5th International Conference on Mobile Software Engineering and Systems_ (Gothenburg, Sweden) _(MOBILESoft ?18)_. Association for Computing Machinery, New York, NY, USA, 180?190.  https://doi.org/10.1145/3197231.3197240
  * Chen et al. (2019) Mengsu Chen, Felix Fischer, Na Meng, Xiaoyin Wang, and Jens Grossklags. 2019.  How Reliable is the Crowdsourced Knowledge of Security Implementation?  _arXiv preprint arXiv:1901.01327_ (2019).
  * Dao et al. (2017) Tung Dao, Lingming Zhang, and Na Meng. 2017.  How Does Execution Information Help with Information-retrieval Based Bug Localization?. In _Proceedings of the 25th International Conference on Program Comprehension_ (Buenos Aires, Argentina) _(ICPC ’17)_. IEEE Press, Piscataway, NJ, USA, 241–250.  https://doi.org/10.1109/ICPC.2017.29
  * DeMarco et al. (2014) Favio DeMarco, Jifeng Xuan, Daniel Le Berre, and Martin Monperrus. 2014.  Automatic Repair of Buggy if Conditions and Missing Preconditions with SMT. In _Proceedings of the 6th International Workshop on Constraints in Software Testing, Verification, and Analysis_.
  * Duala-Ekoko and Robillard (2007) Ekwa Duala-Ekoko and Martin P. Robillard. 2007.  Tracking Code Clones in Evolving Software. In _ICSE ’07: Proceedings of the 29th International Conference on Software Engineering_. IEEE Computer Society, Washington, DC, USA, 158–167.  https://doi.org/10.1109/ICSE.2007.90
  * Egele et al. (2013) Manuel Egele, David Brumley, Yanick Fratantonio, and Christopher Kruegel. 2013.  An empirical study of cryptographic misuse in android applications. In _Proceedings of the 2013 ACM SIGSAC conference on Computer & communications security_. ACM, 73–84.
  * Fahl et al. (2012) Sascha Fahl, Marian Harbach, Thomas Muders, Lars Baumgärtner, Bernd Freisleben, and Matthew Smith. 2012.  Why Eve and Mallory love Android: An analysis of Android SSL (in) security. In _Proceedings of the 2012 ACM conference on Computer and communications security_. ACM, 50–61.
  * Fischer et al. (2017) Felix Fischer, Konstantin Böttinger, Huang Xiao, Christian Stransky, Yasemin Acar, Michael Backes, and Sascha Fahl. 2017.  Stack overflow considered harmful? the impact of copy&paste on android application security. In _2017 IEEE Symposium on Security and Privacy (SP)_. IEEE, 121–136.
  * Fischer et al. (2019) Felix Fischer, Huang Xiao, Ching-Yu Kao, Yannick Stachelscheid, Benjamin Johnson, Danial Razar, Paul Fawkesley, Nat Buckley, Konstantin Böttinger, Paul Muntean, and Jens Grossklags. 2019.  Stack Overflow Considered Helpful! Deep Learning Security Nudges Towards Stronger Cryptography. In _28th USENIX Security Symposium (USENIX Security 19)_. USENIX Association, Santa Clara, CA, 339–356.  https://www.usenix.org/conference/usenixsecurity19/presentation/fischer
  * Fluri et al. (2007) Beat Fluri, Michael Wuersch, Martin PInzger, and Harald Gall. 2007.  Change distilling: Tree differencing for fine-grained source code change extraction.  _IEEE Transactions on software engineering_ 33, 11 (2007), 725–743.
  * Georgiev et al. (2012) Martin Georgiev, Subodh Iyengar, Suman Jana, Rishita Anubhai, Dan Boneh, and Vitaly Shmatikov. 2012\.  The most dangerous code in the world: validating SSL certificates in non-browser software. In _Proceedings of the 2012 ACM conference on Computer and communications security_. ACM, 38–49.
  * Green and Smith (2016) Matthew Green and Matthew Smith. 2016.  Developers are not the enemy!: The need for usable security apis.  _IEEE Security & Privacy_ 14, 5 (2016), 40–46.
  * He et al. (2015) Boyuan He, Vaibhav Rastogi, Yinzhi Cao, Yan Chen, VN Venkatakrishnan, Runqing Yang, and Zhenrui Zhang. 2015.  Vetting SSL usage in applications with SSLint. In _2015 IEEE Symposium on Security and Privacy_. IEEE, 519–534.
  * Hosseini and Brusilovsky (2013) Roya Hosseini and Peter Brusilovsky. 2013.  Javaparser: A fine-grain concept indexing tool for java problems. In _CEUR Workshop Proceedings_ , Vol. 1009. University of Pittsburgh, 60–63.
  * Java Cryptography Architecture ([n.d.]) Java Cryptography Architecture [n.d.].  Java Cryptography Architecture.  https://docs.oracle.com/javase/9/security/java-cryptography-architecture-jca-reference-guide.htm.
  * Jones and Harrold (2005) James A. Jones and Mary Jean Harrold. 2005.  Empirical Evaluation of the Tarantula Automatic Fault-localization Technique. In _Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering_ (Long Beach, CA, USA) _(ASE ’05)_. ACM, New York, NY, USA, 273–282.  https://doi.org/10.1145/1101908.1101949
  * Kaliski (2000) B. Kaliski. 2000\.  PKCS #5: Password-Based Cryptography Specification Version 2.0.  RFC 2898 (Informational).  http://www.ietf.org/rfc/rfc2898.txt
  * Kim et al. (2013) Dongsun Kim, Jaechange Nam, Jaewoo Song, and Sunghun Kim. 2013\.  Automatic Patch Generation Learned from Human-written Patches. In _IEEE/ACM International Conference on Software Engineering (to appear)_.
  * Krüger et al. (2018) Stefan Krüger, Johannes Späth, Karim Ali, Eric Bodden, and Mira Mezini. 2018\.  CrySL: An Extensible Approach to Validating the Correct Usage of Cryptographic APIs. In _32nd European Conference on Object-Oriented Programming (ECOOP 2018)_. Schloss Dagstuhl-Leibniz-Zentrum fuer Informatik.
  * Le Goues et al. (2012) Claire Le Goues, ThanhVu Nguyen, Stephanie Forrest, and Westley Weimer. January 2012.  GenProg: A Generic Method for Automatic Software Repair.  _IEEE Trans. Softw. Eng._ 38, 1 (January 2012).
  * Levenshtein (1966) VI Levenshtein. 1966\.  Binary Codes Capable of Correcting Deletions, Insertions and Reversals.  _Soviet Physics Doklady_ 10 (1966), 707.
  * Li et al. (2018) Zhen Li, Deqing Zou, Shouhuai Xu, Xinyu Ou, Hai Jin, Sujuan Wang, Zhijun Deng, and Yuyi Zhong. 2018\.  VulDeePecker: A Deep Learning-Based System for Vulnerability Detection.  _CoRR_ abs/1801.01681 (2018).  arXiv:1801.01681 http://arxiv.org/abs/1801.01681
  * Long and Rinard (2016) Fan Long and Martin Rinard. 2016.  Automatic Patch Generation by Learning Correct Code.  _SIGPLAN Not._ (2016).
  * Lucia et al. (2010) Lucia, David Lo, Lingxiao Jiang, and Aditya Budi. 2010\.  Comprehensive evaluation of association measures for fault localization. In _Software Maintenance (ICSM), 2010 IEEE International Conference on_. 1–10.  https://doi.org/10.1109/ICSM.2010.5609542
  * Ma et al. (2017) Siqi Ma, Ferdian Thung, David Lo, Cong Sun, and Robert H Deng. 2017.  Vurle: Automatic vulnerability detection and repair by learning from examples. In _European Symposium on Research in Computer Security_. Springer, 229–246.
  * Manger (2001) James Manger. 2001\.  A Chosen Ciphertext Attack on RSA Optimal Asymmetric Encryption Padding (OAEP) as Standardized in PKCS #1 v2.0. In _Advances in Cryptology — CRYPTO 2001_ , Joe Kilian (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 230–238.
  * Mechtaev et al. (2016) S. Mechtaev, J. Yi, and A. Roychoudhury. 2016.  Angelix: Scalable Multiline Program Patch Synthesis via Symbolic Analysis. In _2016 IEEE/ACM 38th International Conference on Software Engineering (ICSE)_. 691–701.
  * Mendel et al. (2013) Florian Mendel, Tomislav Nad, and Martin Schläffer. 2013\.  Improving Local Collisions: New Attacks on Reduced SHA-256. In _Advances in Cryptology – EUROCRYPT 2013_ , Thomas Johansson and Phong Q. Nguyen (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 262–278.
  * Meng et al. (2013) Na Meng, Miryung Kim, and Kathryn McKinley. 2013.  LASE: Locating and Applying Systematic Edits. In _ICSE_. 10.
  * Meng et al. (2018) Na Meng, Stefan Nagy, Danfeng Yao, Wenjie Zhuang, and Gustavo Arango-Argoty. 2018.  Secure coding practices in java: Challenges and vulnerabilities. In _2018 IEEE/ACM 40th International Conference on Software Engineering (ICSE)_. IEEE, 372–383.
  * Miller and Myers (2001) Robert C. Miller and Brad A. Myers. 2001.  Interactive Simultaneous Editing of Multiple Text Regions. In _Proceedings of the General Track: 2002 USENIX Annual Technical Conference_. USENIX Association, Berkeley, CA, USA, 161–174.
  * Nadi et al. (2016) Sarah Nadi, Stefan Krüger, Mira Mezini, and Eric Bodden. 2016.  Jumping Through Hoops: Why Do Java Developers Struggle with Cryptography APIs?. In _Proceedings of the 38th International Conference on Software Engineering_ (Austin, Texas) _(ICSE)_. ACM, New York, NY, USA, 935–946.  https://doi.org/10.1145/2884781.2884790
  * of Standards and (NIST) (2001) United States National Institute of Standards and Technology (NIST). 2001\.  Announcing the ADVANCED ENCRYPTION STANDARD (AES).  _Federal Information Processing Standards Publication_ (2001).
  * Qi et al. (2014) Yuhua Qi, Xiaoguang Mao, Yan Lei, Ziying Dai, and Chengsong Wang. 2014.  The Strength of Random Search on Automated Program Repair. In _ICSE_.
  * Rahaman et al. (2019) Sazzadur Rahaman, Ya Xiao, Sharmin Afrose, Fahad Shaon, Ke Tian, Miles Frantz, Murat Kantarcioglu, and Danfeng (Daphne) Yao. 2019.  CryptoGuard: High Precision Detection of Cryptographic Vulnerabilities in Massive-Sized Java Projects. In _Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security_ (London, United Kingdom) _(CCS ’19)_. Association for Computing Machinery, New York, NY, USA, 2455–2472.  https://doi.org/10.1145/3319535.3345659
  * Rolim et al. (2017) Reudismam Rolim, Gustavo Soares, Loris D’Antoni, Oleksandr Polozov, Sumit Gulwani, Rohit Gheyi, Ryo Suzuki, and Björn Hartmann. 2017\.  Learning syntactic program transformations from examples. In _Proceedings of the 39th International Conference on Software Engineering_. IEEE Press, 404–415.
  * Sheffer et al. (2015) Yaron Sheffer, Ralph Holz, and Peter Saint-Andre. 2015\.  Recommendations for Secure Use of Transport Layer Security (TLS) and Datagram Transport Layer Security (DTLS).  RFC 7525.  https://doi.org/10.17487/RFC7525
  * Toomim et al. (2004) Michael Toomim, Andrew Begel, and Susan L. Graham. 2004\.  Managing Duplicated Code with Linked Editing. In _VLHCC ’04: Proceedings of the 2004 IEEE Symposium on Visual Languages - Human Centric Computing_. IEEE Computer Society, Washington, DC, USA, 173–180.  https://doi.org/10.1109/VLHCC.2004.35
  * Xu et al. (2019) Shengzhe Xu, Ziqi Dong, and Na Meng. 2019.  Meditor: Inference and Application of API Migration Edits. In _Proceedings of the 27th International Conference on Program Comprehension_ (Montreal, Quebec, Canada) _(ICPC ?19)_. IEEE Press, 335?346.  https://doi.org/10.1109/ICPC.2019.00052