#  : A Dataset of Real World Ethereum Smart Contracts Labelled with
Vulnerabilities

Chavhan Sujeet Yashavant  Indian Institute of Technology KanpurKanpurIndia
<EMAIL_ADDRESS>,  Saurabh Kumar  Indian Institute of Technology
KanpurKanpurIndia<EMAIL_ADDRESS>and  Amey Karkare  Indian Institute of
Technology KanpurKanpurIndia<EMAIL_ADDRESS>
(2018)

###### Abstract.

Smart contracts on Ethereum handle millions of U.S. Dollars and other
financial assets. In the past, attackers have exploited smart contracts to
steal these assets. The Ethereum community has developed plenty of tools to
detect vulnerable smart contracts. However, there is no standardized data set
to evaluate these existing tools, or any new tools developed. There is a need
for an unbiased standard benchmark of real-world Ethereum smart contracts. We
have created : an annotated data set of real-world smart contracts taken from
the Ethereum network. The data set is labelled using 5 tools that detect
various vulnerabilities in smart contracts, using majority voting.

Ethereum, Smart contracts, Annotated dataset, Vulnerabilities

††copyright: acmcopyright††journalyear: 2018††doi:
10.1145/1122445.1122456††journal: TOG††journalvolume: 37††journalnumber:
4††article: 111††publicationmonth: 8††conference: ; ;

##  1\. Introduction

Permissionless blockchains like Ethereum have tens of millions of smart
contracts (Miller et al., 2018). A smart contract is a code that resides on
the Ethereum network and executes when predetermined conditions get satisfied.
Smart contracts on Ethereum handle financial assets worth millions of U.S.
Dollars (USD). Attacking smart contracts can provide monetary benefits to the
attackers or can cause other damages (Atzei et al., 2017). Therefore, the
attacks on Ethereum smart contracts have increased rapidly over the last few
years (Chen et al., 2020).

To detect the attacks on Ethereum smart contracts, the Ethereum community has
developed plenty of tools (Feist et al., 2019; Luu et al., 2016; Tsankov et
al., 2018; Tikhomirov et al., 2018; ConsenSys, 2022; Nikolić et al., 2018).
These tools analyze smart contracts and produce vulnerability reports. The
authors of these tools use datasets of smart contracts to evaluate the tools’
efficiency, correctness, and other parameters. However, datasets used by the
authors to evaluate their tools vary significantly (Ren et al., 2021b). The
type of dataset has an impact on the tool’s performance. Therefore it is
necessary to evaluate tools using multi-type integrated benchmark suite (Ren
et al., 2021b).

Ren et al. (Ren et al., 2021b) have created a dataset having has two types of
smart contracts: annotated and non-annotated. However, the number of annotated
smart contracts in the database is very small (less than a thousand). Further,
most of these annotated smart contracts not real-world contracts. The dataset
created by Ferreira et al. (Ferreira et al., 2020) the same issues. Hence,
these datasets cannot be considered representatives for the real-world
Ethereum smart contracts. There is a need for a benchmark suite containing
real world smart contracts labelled with vulnerabilities.

We present : a dataset of real-world Ethereum smart contracts labelled with
vulnerabilities. The Ethereum community can use for an unbiased evaluation of
new and existing tools to analyse vulnerabilities. contains 6.7K labelled
real-world Ethereum smart contracts. Our methodology to label the smart
contracts is based on the work by Ren et. al. (Ren et al., 2021a). This paper
describes our solution that uses various tools (such as (Feist et al., 2019;
Tikhomirov et al., 2018; ConsenSys, 2022; Luu et al., 2016; Torres et al.,
2018)) to find and catalog the vulnerabilities in the smart contracts. The
resulting data set, , is released at https://github.com/sujeetc/ScrawlD.

##  2\. Related Work

Ren et al. (Ren et al., 2021b) crafted a benchmark suite that integrates
labeled and unlabeled Smart Contracts. The unlabeled dataset has 45,622 Real-
World Ethereum Smart Contracts having more than one transaction on the
Ethereum blockchain. The labeled dataset has artificially constructed
contracts (350 contracts) and confirmed vulnerable contracts (214 contracts).

Durieux et al. (Durieux et al., 2020) crafted a dataset of annotated and non-
annotated smart contracts. The annotated part contains 69 contracts tagged
with 115 vulnerabilities. The annotated part has ten categories of
vulnerabilities. In contrast, the non annotated part contains 47,518 unique
contracts, each with at least one transaction on the Ethereum network.

Researchers can’t use the unlabeled dataset to evaluate the tool as there is
no ground truth associated with it. Moreover, the labeled dataset doesn’t have
more than a thousand smart contracts.

Many vulnerability analysis tools used unlabelled datasets to evaluate the
tool. Due to huge number of contracts in these datasets, either the dataset is
not annotated or a very small subset is annotated with the vulnerabilities
present. To the best of our knowledge, there is no fully annotated dataset
released publicly. We describe existing efforts in this area. Smartcheck
(Tikhomirov et al., 2018) authors evaluate the tool’s accuracy by using three
annotated smart contracts. These annotated smart contracts do not represent
the diverse smart contracts from Ethereum. Moreover, the authors used 4,600
verified contracts from Ethereum to evaluate the tool’s efficiency. However,
the issues found in these 4,600 smart contracts are not manually verified.

Slither (Feist et al., 2019) does two types of experiments to evaluate the
tool. The first experiment contains DAO (Daian, 2022) and SpankChain
(SpankChain, 2022) smart contracts. DAO and SpankChain are two famous
reentrancy attacks on Ethereum. The second experiment contains 1000 most used
smart contracts from the Ethereum network. This dataset is unlabelled. The
authors manually reviewed the results of detecting reentrancy vulnerability
for 50 random smart contracts to evaluate the accuracy of Slither. Thus, their
validation is not representative of the possible vulnerabilities in a dataset
having large number of smart contracts.

##  3\. Methodology

Table 1. Vulnerabilities Description and SWC-ID as per SWC Registry  Vulnerability Type | SWC-ID | Description
---|---|---
ARTHM | SWC-101 | Arithmetic (Integer Overflow and Underflow)
DOS | SWC-113, SWC-128 | Denial of Service
LE | – | Locked Ether
RENT | SWC-107 | Reentrancy
TimeM | SWC-116 | Block values as a proxy for time
TimeO | SWC-114 | Transaction Order Dependence
UE | SWC-104 | Unchecked Call Return Value
TX-Origin | SWC-115 | Authorization through tx.origin
Table 2. Vulnerabilities Supported by Selected Tools. ✓ indicates that the
tool supports the vulnerability.
---
$\times$ indicates that the tool doesn’t support the vulnerability.
_ indicates Unknown.
Tool Name | Vulnerability Type
ARTHM | DOS | LE | RENT | TimeM | TimeO | TX-Origin | UE
Slither | $\times$ | ✓ | ✓ | ✓ | ✓ | $\times$ | ✓ | ✓
Smartcheck | ✓ | ✓ | ✓ | $\times$ | ✓ | $\times$ | ✓ | ✓
Mythril | ✓ | ✓ | – | ✓ | ✓ | ✓ | ✓ | ✓
Oyente | ✓ | – | $\times$ | ✓ | ✓ | ✓ | – | $\times$
Osiris | ✓ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$
Table 3. Threshold for each Vulnerability Type Vulnerability Type |  #Tools that can detect the Vulnerability |  Threshold
---|---|---
ARTHM |  4 |  2
DOS |  3 |  2
LE |  2 |  1
RENT |  3 |  2
TimeM |  4 |  2
TimeO |  2 |  1
UE |  3 |  2
TX-Origin |  3 |  2

This section explains the data source, tools used, and process to label the
dataset with an example.

###  3.1. Data Source

We took 45,622 addresses belonging to smart contracts from the dataset of the
paper by Ren et. al. (Ren et al., 2021b) The dataset contains all unique smart
contracts with more than one transaction on the Ethereum network. Moreover,
these smart contracts’ source code is publicly available. We crawled the
source codes of these 45,622 smart contracts through Etherscan’s API
(Etherscan, 2022). However, we released 6780 smart contracts in due to
challenges given in Section 6.

###  3.2. Tools Used

The following points explain the criteria to select the tools as used in the
paper by Durieux et. al. (Durieux et al., 2020)

  * •

Available and CLI: The tool is released publicly and has a command-line
interface (CLI).

  * •

Compatible Input: The tool requires input as Solidity smart contract. We do
not consider the tools that require only EVM bytecode.

  * •

Only Source: The tool runs on only Solidity source code. We do not consider
the tools that require a test suite or smart contract labelled with
assertions.

  * •

Vulnerability Finding: The tool finds vulnerabilities in smart contracts.

The following tools are selected based on the above criteria.

  * •

Slither: Slither (Feist et al., 2019) is a static analysis tool designed to
analyze Ethereum smart contracts. It has four prominent use cases: automated
detection of vulnerabilities, automated detection of code optimization
opportunities, improvement of users’ understanding of the contracts, and
assistant with code review.

  * •

Mythril: Mythril is a tool that does security analysis of Ethereum smart
contracts. It detects various security issues (ConsenSys, 2022).

  * •

Smartcheck: Smartcheck (Tikhomirov et al., 2018) is an extensible static
analysis tool that detects vulnerabilities in smart contracts. It converts
Solidity code into XML-based intermediate representation and checks it with
XPATH patterns.

  * •

Oyente: Oyente (Luu et al., 2016) is a symbolic execution tool to find
security bugs in smart contracts.

  * •

Osiris: Osiris (Torres et al., 2018) is a framework comprising symbolic
execution and taint analysis. It detects integer bugs in Ethereum smart
contracts.

These tools are neither sound nor complete. They can miss vulnerabilities
(False Negatives) or detect them when they are not present (False Positives).

###  3.3. Vulnerabilities

Table 1 shows vulnerabilities selected for evaluation. The column
“Vulnerability type” indicates the acronym used in the paper for a particular
vulnerability, while the column “SWC-ID” shows the ID provided by the SWC
Registry 111SWC Registry: https://swcregistry.io for the vulnerability where
available.

Table 2 shows vulnerabilities supported by selected tools. Symbol ✓indicates
that the tool can detect a particular vulnerability, whereas symbol
$\times$indicates that the tool can’t detect the vulnerability. Symbol –
indicates that we don’t know whether the tool has detection capability of the
particular vulnerability or not. Due to improper documentation of the tool, it
is hard to find whether it supports this vulnerability or not.

###  3.4. Process to Label the Dataset

We use the methodology given in the paper by Ren et al. (Ren et al., 2021a) to
find vulnerabilities in smart contracts. To understand the process of labeling
the dataset, consider the example shown in Listing 1. It shows a code snippet
of a vulnerable contract from the SWC registry. Here, the constructor name
should be “Missing”. However, accidentally, the smart contract developer
writes it as “IamMissing”. Now, any node on the Ethereum blockchain can create
a transaction that calls this function named “IamMissing” and can become the
owner of this contract. This type of bug is known as incorrect constructor
name.

⬇

1pragma solidity ^0.4.24;

2contract Missing{

3 address private owner;

4 function IamMissing() public {

5 owner = msg.sender;

6 }

7}

Listing 1: Solidity Bug: Incorrect Constructor name at line 4

Due to the presence of false positives in the reports generated by the tools,
we cannot depend on just one tool. We use majority voting, that is, at least
50% of the tools must report the same vulnerability at the same location. For
example, assume that tools T1, T2, and T3 can detect incorrect constructor
name vulnerability. Suppose, at least two tools report that incorrect
constructor name vulnerability is present at Line 4. Then we say that the
contract is vulnerable for incorrect constructor name; otherwise, we say that
the contract is not vulnerable.

Algorithm 1 shows the methodology to label the dataset. The following steps
explain the methodology given in the algorithm 1:

  1. (1)

First, collect the output of the selected tools for the smart contract to be
analyzed (line 2).

  2. (2)

Note the vulnerability name and line number for the vulnerability per tool
(line 4).

  3. (3)

For the same vulnerability, if different tools show different locations,
consider them as different warnings. We consider two warnings as the same only
if the vulnerability name and location match for different tools (line 5).

  4. (4)

The methodology says that the vulnerability exists at a location if more than
50% tools confirm the same vulnerability at the exact location. In that case,
label the location with the vulnerability (line 8 to 12).

Algorithm 1 Process to Label the Dataset

1:for $tool\leftarrow tool\\_1$ to $tool\\_n$ do

2: Collect the output of $tool$

3: for $vuln\leftarrow vuln\\_1$ to $vuln\\_n$ do

4: 1\. Note the $line\\_number$ for $vuln$ detected by $tool$

5: 2\. Increase the $tool\\_count$ for $vuln$ for $line\\_number$ detected

6: end for

7:end for

8:for $vuln\leftarrow vuln\\_1$ to $vuln\\_n$ do

9: if $tool\\_count$ ¿= n/2 for $line\\_number$ then

10: Vulnerability : $vuln$ is present at $line\\_number$

11: end if

12:end for

##  4\. Results

Figure 1. Number of unique Contracts having each Vulnerability (log-scale), Vulnerability Type DOS is not detected by majority voting Figure 2. Count of Vulnerability Warnings according to the Majority of Tools (log-scale), Vulnerability Type DOS is not detected by majority voting Table 4. Total Contracts with Unique Vulnerabilities #Unique Vulnerabilities |  #Contracts
---|---
NONE |  2688
1 |  2912
2 |  1082
3 |  86
4 |  12

Table 3 shows the number of tools that support each vulnerability. The
threshold column in the table shows the majority rule. For example, consider
Unhandled Exception (UE) vulnerability. Three tools support it. Its threshold
is two. Hence, if two or more tools say that the contract is vulnerable for UE
at the same location, we report it as a real vulnerability. We show the
vulnerability only if greater than or equal to 50% of the tools report that it
is present. The idea of marking the vulnerability as real or not is taken from
the paper by Ren et al. (Ren et al., 2021a).

Figure 1 shows the number of smart contracts with at least one occurrence of
particular vulnerability according to the majority vote. For example, consider
Reentrancy vulnerability. Table 3 shows that Slither, Mythril, and Oyente
support the detection reentrancy vulnerability. We say that the contract
contains reentrancy only if two of these three tools report that it is present
at the same location.

Figure 2 shows the total warnings reported for each vulnerability according to
the majority of tools. Please note that there can be multiple warnings in a
single file for each vulnerability.

Table 4 shows contracts having exclusive number of vulnerabilities. According
to our evaluation methodology, 2688 contracts have no vulnerabilities. 2912
contracts have only one type of vulnerability. Similarly, 1082 contracts have
two vulnerabilities each.

##  5\. Applications of The Dataset

###  5.1. Evaluation of New Tools

The Ethereum community can use our dataset for evaluating correctness and
other parameters of newly designed tools. comprises a diverse set of real-
world annotated smart contracts. Hence, researchers can use for evaluating new
tools.

###  5.2. Machine Learning based Tools

Vulnerability analysis tools based on Machine Learning needs dataset for
training. has 6.7K labelled smart contracts. Hence, researchers can use as a
basis for learning or training of ML-based tools.

###  5.3. Empirical Analysis of existing Tools

There are plenty of tools that analyze Ethereum smart contracts and detect
different vulnerabilities. Smart contract developers face difficulty choosing
the tool to analyze their smart contracts. There should be an empirical study
that compares various tools and suggest which tools are better. Previous works
(Ren et al., 2021b; Durieux et al., 2020) use dataset where the annotation is
limited to a few thousand smart contracts. The effectiveness of these studies
can be improved with .

##  6\. Challenges in creating the Annotated Dataset

Each tool depends on different packages, specific versions of these packages.
Most of the time, different tools are not compatible with each other. That’s
why we create docker (Docker, 2013) containers for these tools and run our
experiments.

Mythril tool is memory-intensive. It consumes more than 10 GB RAM, and hence
we cannot run it in parallel. It is cumbersome to get results from Mythril.

##  7\. Threats to Validity

Each tool selected for evaluation is neither sound nor complete. There can be
a lot of false positives or false negatives generated by a specific tool.
However, we used the majority approach to say whether a vulnerability is
present or not. But, this approach may fail if majority of the tools generate
false positives or false negatives.

We can solve these issues by integrating more tools or manually reviewing
detected vulnerabilities through crowdsourcing or other means. However, these
approaches are time and resource-intensive, and we can do it incrementally.

##  8\. Conclusion and Future Work

We collected 46K Ethereum smart contracts from Etherscan. However, due to
challenges in section 6, we could not label them all. We are working on
labeling more contracts, and will be updated regularly.

We will add more tools to and do a manual inspection of detected
vulnerabilities in smart contracts through crowdsourcing or some other means
in the future.

## References

  * (1)
  * Atzei et al. (2017) Nicola Atzei, Massimo Bartoletti, and Tiziana Cimoli. 2017\.  A Survey of Attacks on Ethereum Smart Contracts (SoK). In _Principles of Security and Trust_ , Matteo Maffei and Mark Ryan (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 164–186.
  * Chen et al. (2020) Huashan Chen, Marcus Pendleton, Laurent Njilla, and Shouhuai Xu. 2020.  A Survey on Ethereum Systems Security: Vulnerabilities, Attacks, and Defenses.  _ACM Comput. Surv._ 53, 3, Article 67 (jun 2020), 43 pages.  https://doi.org/10.1145/3391195
  * ConsenSys (2022) ConsenSys. 2022\.  Mythril.  https://github.com/ConsenSys/mythril-classic
  * Daian (2022) Phil Daian. 2022\.  Analysis of the dao exploit.  http://hackingdistributed.com/2016/06/18/analysis-of-the-dao-exploit
  * Docker (2013) Docker. Initial Release: 2013\.  Empowering App Development for Developers.  https://www.docker.com
  * Durieux et al. (2020) Thomas Durieux, João F. Ferreira, Rui Abreu, and Pedro Cruz. 2020.  Empirical Review of Automated Analysis Tools on 47,587 Ethereum Smart Contracts. In _Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering_ (Seoul, South Korea) _(ICSE ’20)_. Association for Computing Machinery, New York, NY, USA, 530–541.  https://doi.org/10.1145/3377811.3380364
  * Etherscan (2022) Etherscan. 2022\.  The Ethereum Blockchain Explorer.  https://etherscan.io
  * Feist et al. (2019) Josselin Feist, Gustavo Greico, and Alex Groce. 2019\.  Slither: A Static Analysis Framework for Smart Contracts. In _Proceedings of the 2nd International Workshop on Emerging Trends in Software Engineering for Blockchain_ (Montreal, Quebec, Canada) _(WETSEB ’19)_. IEEE Press, 8–15.  https://doi.org/10.1109/WETSEB.2019.00008
  * Ferreira et al. (2020) João F. Ferreira, Pedro Cruz, Thomas Durieux, and Rui Abreu. 2020.  SmartBugs: A Framework to Analyze Solidity Smart Contracts. In _Proceedings of the 35th IEEE/ACM International Conference on Automated Software Engineering_ (Virtual Event, Australia) _(ASE ’20)_. Association for Computing Machinery, New York, NY, USA, 1349–1352.  https://doi.org/10.1145/3324884.3415298
  * Luu et al. (2016) Loi Luu, Duc-Hiep Chu, Hrishi Olickel, Prateek Saxena, and Aquinas Hobor. 2016.  Making Smart Contracts Smarter. In _Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security_ (Vienna, Austria) _(CCS ’16)_. Association for Computing Machinery, New York, NY, USA, 254–269.  https://doi.org/10.1145/2976749.2978309
  * Miller et al. (2018) Andrew Miller, Zhicheng Cai, and Somesh Jha. 2018.  Smart Contracts and Opportunities for Formal Methods. In _Leveraging Applications of Formal Methods, Verification and Validation. Industrial Practice_ , Tiziana Margaria and Bernhard Steffen (Eds.). Springer International Publishing, Cham, 280–299.
  * Nikolić et al. (2018) Ivica Nikolić, Aashish Kolluri, Ilya Sergey, Prateek Saxena, and Aquinas Hobor. 2018.  Finding The Greedy, Prodigal, and Suicidal Contracts at Scale. In _Proceedings of the 34th Annual Computer Security Applications Conference_ (San Juan, PR, USA) _(ACSAC ’18)_. Association for Computing Machinery, New York, NY, USA, 653–663.  https://doi.org/10.1145/3274694.3274743
  * Ren et al. (2021a) Meng Ren, Fuchen Ma, Zijing Yin, Ying Fu, Huizhong Li, Wanli Chang, and Yu Jiang. 2021a.  Making Smart Contract Development More Secure and Easier. In _Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering_ (Athens, Greece) _(ESEC/FSE 2021)_. Association for Computing Machinery, New York, NY, USA, 1360–1370.  https://doi.org/10.1145/3468264.3473929
  * Ren et al. (2021b) Meng Ren, Zijing Yin, Fuchen Ma, Zhenyang Xu, Yu Jiang, Chengnian Sun, Huizhong Li, and Yan Cai. 2021b.  Empirical Evaluation of Smart Contract Testing: What is the Best Choice?. In _Proceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis_ (Virtual, Denmark) _(ISSTA 2021)_. Association for Computing Machinery, New York, NY, USA, 566–579.  https://doi.org/10.1145/3460319.3464837
  * SpankChain (2022) SpankChain. 2022\.  We got spanked: What we know so far.  https://medium.com/spankchain/we-got-spanked-what-we-know-so-far-d5ed3a0f38fe
  * Tikhomirov et al. (2018) Sergei Tikhomirov, Ekaterina Voskresenskaya, Ivan Ivanitskiy, Ramil Takhaviev, Evgeny Marchenko, and Yaroslav Alexandrov. 2018\.  SmartCheck: Static Analysis of Ethereum Smart Contracts. In _Proceedings of the 1st International Workshop on Emerging Trends in Software Engineering for Blockchain_ (Gothenburg, Sweden) _(WETSEB ’18)_. Association for Computing Machinery, New York, NY, USA, 9–16.  https://doi.org/10.1145/3194113.3194115
  * Torres et al. (2018) Christof Ferreira Torres, Julian Schütte, and Radu State. 2018\.  Osiris: Hunting for Integer Bugs in Ethereum Smart Contracts. In _Proceedings of the 34th Annual Computer Security Applications Conference_ (San Juan, PR, USA) _(ACSAC ’18)_. Association for Computing Machinery, New York, NY, USA, 664–676.  https://doi.org/10.1145/3274694.3274737
  * Tsankov et al. (2018) Petar Tsankov, Andrei Dan, Dana Drachsler-Cohen, Arthur Gervais, Florian Bünzli, and Martin Vechev. 2018.  Securify: Practical Security Analysis of Smart Contracts. In _Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security_ (Toronto, Canada) _(CCS ’18)_. Association for Computing Machinery, New York, NY, USA, 67–82.  https://doi.org/10.1145/3243734.3243780