README.md

---
# For reference on model card metadata, see the spec: https://github.com/huggingface/hub-docs/blob/main/modelcard.md?plain=1
# Doc / guide: https://huggingface.co/docs/hub/model-cards
#{{ card_data }}

license: mit
language:
- en
base_model: Salesforce/codegen-2B-multi
pipeline_tag: text-generation
library_name: transformers
inference:
  parameters:
    provider: "spaces"
    space_url: "https://huggingface.co/Chain-GPT/Solidity-LLM/"
tags:
    - Solidity
    - BlockChain
    - Smart Contracts
    - Code Generation
---

# **Solidity-Code-LLM**
Solidity-Code-LLM is a fine tuned large language model designed to understand, generate, and analyze smart contracts written in Solidity. Developed by ChainGPT—a leader in AI infrastructure for the Web3 and blockchain space—this model is purpose-built for the decentralized development ecosystem. 

- **Developed by:** ChainGPT
- **License:** MIT License
- **Finetuned from model:** Salesforce/codegen-2B-multi

![ChainGPT Logo](logo.png)


# Model Details

### Model Description
Solidity-Code-LLM is a specialized language model trained in two stages: pre-training on a large, unstructured Solidity dataset, followed by instruction-based fine-tuning on a cleaned, curated dataset. Unlike general-purpose code models, it is exclusively focused on Solidity—the dominant language for Ethereum-compatible blockchains—making it an efficient and accurate assistant for writing and debugging smart contracts across a wide range of use cases, including tokens, DApps, DAOs, and governance protocols.


### Model Features
- **Type**: Code Gen For Causal LLM
- **Tokenizer**: GPT2Tokenizer
- **Number of Parameters**: 2B
- **Number of Layers**: 32 Transformer blocks
- **Context Length**: Full 2048 tokens
- **Dtype**: bfloat16

### Model Sources
Model is deployed on huggingface space for inference
- **Demo:** [Demo On Hugging Face Space](https://huggingface.co/spaces/Chain-GPT/ChainGPT-Solidity-LLM)

# Model Comparison
We have compared our model with the following models
- GPT 4.5 Preview
- GPT 4o mini
- [Qwen 2.5-Coder-7B](https://huggingface.co/Qwen/Qwen2.5-Coder-7B)
- [DeepSeek-Coder-7B-Instruct-v1.5](https://huggingface.co/deepseek-ai/deepseek-coder-7b-instruct-v1.5)

On the following parameters
- **Compilation(%)** - Percentage of generated contracts that compile successfully without modification.
- **OpenZeppelin Compliance(%)** - Adherence to OpenZeppelin library usage and standards.
- **Gas Efficiency(%)** - Degree of gas optimization based on Slither’s suggestions.
- **Security(%)** - Percentage of code free from common vulnerabilities detected by Slither.
- **Average Lines of Code** - Average number of non-empty, commented-included lines in generated contracts, indicating verbosity or conciseness

## Benchmark
Below is a figure summarizing the performance of each model across the four evaluation metrics.
![Benchmark](Benchmark.png)

Following obvservation were made regarding Solidity LLM.
- Highest Compilation Success Rate (~83%), demonstrating strong Solidity syntax and structure generation.
- Good OpenZeppelin Compliance (~65%), indicating frequent use of standard libraries and contract patterns. While GPT-4.5, being a much larger model, naturally exhibits stronger adherence to OpenZeppelin standards due to its broader training data, Solidity LLM achieves commendable compliance given its smaller size.
- Top Gas Efficiency (~72%), producing optimized code as evaluated by tools like Slither.
- Moderate Security Score (~58%), showing acceptable security posture but room for improvement. GPT-4.5 benefits from its scale in handling more security cases.
- Concise Code (~70% LOC score), generating relatively compact and efficient smart contracts.


# Uses
### Direct Use
- Assisting developers in writing Solidity smart contracts.
- Educational tool for learning Solidity.
- Auto-generating documentation or contract templates.

### Downstream Use
- Integrated into IDEs or smart contract development platforms.
- Supporting autonomous agents that interact with blockchains.

### Out-of-Scope Use
- Not suitable for general-purpose code generation in languages other than Solidity.
- Not intended for legal auditing or formal verification without human oversight.
- Should not be used to deploy contracts to production without expert review.

### Bias, Risks, and Limitations
- May reflect biases from web-scraped content (e.g., outdated or insecure coding practices).
- Model might hallucinate code or provide syntactically valid but logically incorrect suggestions.
- Risks associated with using AI-generated code in high-stakes or financial environments without thorough vetting.

### Recommendations
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. Manual code review and testing are strongly recommended before deployment.

# How to Get Started with the Model
The model follows a two-step generation process: it first produces a natural language description of the code, and subsequently generates the corresponding source code based on the given prompt. The complete output is generated internally before being displayed to the user. For scenarios requiring direct code generation without intermediate descriptions, a streaming mode can be utilized to produce code in real time.

Requirements for model.
```Python
pip install transformers==4.51.3 torch==2.7.0 accelerate==1.6.0
```

Use the code below to get started with the model.

```Python
from transformers import AutoModelForCausalLM, AutoTokenizer

modelpath = "Chain-GPT/Solidity-LLM"

tokenizer = AutoTokenizer.from_pretrained(modelpath)
model = AutoModelForCausalLM.from_pretrained(modelpath).to("cuda")

prompt = "Write a Solidity function to transfer tokens."
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

outputs = model.generate(**inputs, max_new_tokens=1400, pad_token_id=tokenizer.eos_token_id)
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)

print(generated_text)
```

## Streaming
To stream the generated code. During streaming, description is not generated.

```Python
import time
import torch
from queue import Empty
from threading import Thread
from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer

model = AutoModelForCausalLM.from_pretrained(
    "Chain-GPT/Solidity-LLM",
    torch_dtype=torch.bfloat16,
    device_map="cuda"
)
tokenizer = AutoTokenizer.from_pretrained("Chain-GPT/Solidity-LLM")

CodeInstruction = "Develop a Solidity Contract for lottery which requires 1 eth for registration fee and the winner gets a reward of 10 eth."
prompt = (
    f'You are given a coding instruction. Generate only the code that completes the task. Do not include any explanation or description.\n'
    f'Instruction: {CodeInstruction}\nCode:'
)

inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
streamer = TextIteratorStreamer(tokenizer, skip_prompt=True, skip_special_tokens=True)
generation_thread = Thread(target=model.generate, kwargs={
    "input_ids": inputs["input_ids"],
    "max_new_tokens": 1800,
    "temperature": 0.7,
    "do_sample": True,
    "streamer": streamer
})
generation_thread.start()

def consume_streamer(streamer: TextIteratorStreamer):
    inside_code_block = False
    buffer = ""
    while True:
        try:
            for token in streamer:
                buffer += token

                while "```" in buffer:
                    parts = buffer.split("```", 1)
                    if not inside_code_block:
                        inside_code_block = True
                        buffer = parts[1]
                    else:
                        for char in parts[0]:
                            yield char
                        return
                if inside_code_block:
                    while buffer:
                        yield buffer[0]
                        buffer = buffer[1:]
            break
        except Empty:
            time.sleep(0.1)

for chunk in consume_streamer(streamer):
    print(chunk, end="", flush=True)
```


# Training Details

### Training Procedure
We uses the following compute for finetuning the model
- 80 GB 4 GPUs Cluster
- Training Duration: 1095 Hours which is approximately 1 and a half month

### Pre-training:
Trained on raw, unstructured data.
-1B tokens, raw corpus

### Fine-tuning:
- Contracts written in Solidity version >= 0.5.
- Token length between 200 and 4000.
- Deduplicated and filtered out interfaces, libraries, contracts with irrelevant comments, or relative imports.
- Only compilable and executable contracts retained.
- 650K instructions finetune


# Results
Prompt
```
Develop a Solidity program for a decentralized library where users can store and retrieve data after paying a small token fee.
```

Output Code
```Solidity
// SPDX-License-Identifier: MIT
pragma solidity ^0.8.0;

import "@openzeppelin/contracts/access/Ownable.sol";
import "@openzeppelin/contracts/token/ERC20/IERC20.sol";

contract DecentralizedLibrary is Ownable(msg.sender) {
    IERC20 public token;
    uint256 public tokenFee;
    
    struct DataEntry {
        string data;
        address owner;
    }
    
    mapping(uint256 => DataEntry) private dataStore;
    mapping(address => uint256) private userBalances;
    uint256 private dataCount;

    event DataStored(uint256 indexed dataId, address indexed owner, string data);
    event DataRetrieved(uint256 indexed dataId, address indexed owner);

    constructor(IERC20 _token, uint256 _initialTokenFee) {
        token = _token;
        tokenFee = _initialTokenFee;
    }

    function storeData(string memory _data) public {
        payTokenFee();
        dataStore[dataCount] = DataEntry(_data, msg.sender);
        emit DataStored(dataCount, msg.sender, _data);
        dataCount++;
    }

    function retrieveData(uint256 _dataId) public {
        payTokenFee();
        require(_dataId < dataCount, "Data does not exist");
        emit DataRetrieved(_dataId, msg.sender);
    }

    function payTokenFee() internal {
        require(token.balanceOf(msg.sender) >= tokenFee, "Insufficient token balance");
        token.transferFrom(msg.sender, address(this), tokenFee);
    }

    function getTokenBalance() public view returns (uint256) {
        return token.balanceOf(msg.sender);
    }

    function setTokenFee(uint256 _newTokenFee) public onlyOwner {
        tokenFee = _newTokenFee;
    }

    function withdrawTokens(uint256 _amount) public onlyOwner {
        require(token.balanceOf(address(this)) >= _amount, "Insufficient contract balance");
        token.transfer(msg.sender, _amount);
    }
}
```

# Evaluation Matrics
To evaluate the performance of our fine-tuned LLM specialized in Solidity smart contract generation, we used **[Slither](https://github.com/crytic/slither)**, a static analysis framework widely used for analyzing Solidity code.

We focused on following key evaluation criteria:

- **Compilation Success Rate**
We measured the percentage of generated smart contracts that compile successfully without modification. This helps assess the syntactic and structural correctness of the model outputs.

- **OpenZeppelin Standards Compliance**
We verified whether the generated contracts adhere to best practices by checking for proper usage of OpenZeppelin libraries. This includes ensuring the latest or stable versions of libraries are used and the overall contract structure aligns with established OpenZeppelin patterns.

- **Gas Optimization Opportunities**
Using Slither’s gas optimization analysis, we identified areas in the generated contracts where gas usage could be reduced. We measured the number and types of optimization suggestions as an indicator of how efficient the generated code is.

- **Security Vulnerabilities**
We analyzed each contract for known security vulnerabilities using Slither’s built-in detectors. We recorded the number and severity of the vulnerabilities detected, providing a measure of the security quality of the model’s outputs.

- **Average Lines of Code (LOC)**
Captures the average number of lines per generated contract, excluding blank lines but including comments. This metric reflects code verbosity or conciseness, and helps gauge implementation completeness versus potential redundancy.

These metrics collectively provide a multi-dimensional view of the model’s effectiveness, spanning correctness, efficiency, security, and usability. They are designed to reflect both automated benchmarks and real-world developer expectations.


# Summary
Solidity LLM, despite its compact 2B parameter size, delivers standout performance in generating Solidity smart contracts. It achieved the highest compilation success rate (83%), showcasing robust syntactic and structural understanding. Its strong OpenZeppelin compliance (65%), though slightly behind very large models like GPT-4.5, is impressive given the scale difference, reflecting reliable use of industry-standard patterns and libraries.

Further, Solidity LLM ranked highest in gas efficiency (72%), producing optimized code suitable for cost-sensitive deployments. While the security score (58%) indicates room for improvement, the model consistently generated secure-enough contracts for practical use. Its concise output (70% LOC score) also suggests an efficient coding style, balancing brevity with completeness.

Overall, Solidity LLM proves to be a resource-efficient, reliable, and well-balanced model for Solidity code generation.

Looking ahead, future releases will focus on improving support for newer versions of the Solidity language and OpenZeppelin libraries, enhancing user interaction by enabling contract modifications, expanding compatibility to other languages like Rust, and developing larger models capable of handling longer context windows.