rowsep=0.5pt,colsep=2pt

#  OriGen: Enhancing RTL Code Generation with Code-to-Code Augmentation and
Self-Reflection

Fan Cui  Peking University<EMAIL_ADDRESS>,  Chenyang Yin  Peking
University<EMAIL_ADDRESS>,  Kexing Zhou  Peking University
<EMAIL_ADDRESS>,  Youwei Xiao  School of Integrated Circuits, Peking
University<EMAIL_ADDRESS>,  Guangyu Sun  School of Integrated Circuits,
Peking University<EMAIL_ADDRESS>,  Qiang Xu  The Chinese University of Hong
Kong<EMAIL_ADDRESS>,  Qipeng Guo  Shanghai AI Laboratory
<EMAIL_ADDRESS>,  Demin Song  Shanghai AI Laboratory
<EMAIL_ADDRESS>,  Dahua Lin  Shanghai AI Laboratory
<EMAIL_ADDRESS>,  Xingcheng Zhang  Shanghai AI Laboratory
<EMAIL_ADDRESS>and  Yun (Eric) Liang  Peking UniversityBeijing
Advanced Innovation Center for Integrated Circuits<EMAIL_ADDRESS>
###### Abstract.

Recent studies have illuminated that Large Language Models (LLMs) exhibit
substantial potential in the realm of RTL (Register Transfer Level) code
generation, with notable advancements evidenced by commercial models such as
GPT-4 and Claude3-Opus. Despite their proficiency, these commercial LLMs often
raise concerns regarding privacy and security. Conversely, open-source LLMs,
which offer solutions to these concerns, have inferior performance in RTL code
generation tasks to commercial models due to the lack of high-quality open-
source RTL datasets.

To address this issue, we introduce OriGen, a fully open-source framework
featuring self-reflection capabilities and a dataset augmentation methodology
for generating high-quality, large-scale RTL code. We propose a novel code-to-
code augmentation methodology that leverages knowledge distillation to enhance
the quality of the open-source RTL code datasets. Additionally, OriGen is
capable of correcting syntactic errors by leveraging a self-reflection process
based on feedback from the compiler. The self-reflection ability of the model
is facilitated by a carefully constructed dataset, which comprises a
comprehensive collection of samples. Experimental results demonstrate that
OriGen remarkably outperforms other open-source alternatives in RTL code
generation, surpassing the previous best-performing LLM by 9.8% on the
VerilogEval-Human benchmark. Furthermore, OriGen exhibits superior
capabilities in self-reflection and error rectification, surpassing GPT-4 by
18.1% on the benchmark designed to evaluate the capability of self-reflection.

††copyright: none

##  1\. Introduction

Recently, large language models (LLMs) have demonstrated outstanding
performance in natural language comprehension and generation (Kalyan et al.,
2021). Studies have shown that LLMs exhibit considerable ability in code
generation tasks (Sun et al., 2024). Commercial LLMs, such as GPT-4 (Achiam et
al., 2023) and Claude3-Opus (Anthropic, 2024), generate high-quality software
code for common programming languages like C++ and Python, which enhance
coding productivity significantly. Moreover, LLMs have also illustrated
impressive capabilities in hardware code generation, especially for the
Register Transfer Level (RTL) code. Specifically, RTL code generation from
natural language instructions presents an innovative approach to enhance
hardware development productivity, offering the potential to revolutionize
existing HDL coding workflows by alleviating the burdensome task of HDL coding
for designers.

While commercial LLMs have demonstrated proficiency in generating RTL code,
they often raise concerns regarding privacy and security. These issues are
especially critical in the domain of hardware design, as RTL code frequently
contains valuable intellectual property. Furthermore, the closed-source nature
of these LLMs restricts researchers from conducting in-depth analyses and
customizations, impeding further fine-tuning of the models in specific
domains, such as hardware design. In contrast, open-source models provide
better privacy and security while facilitating further improvement and
customization. Open-source models, such as StarCoder2 (Lozhkov et al., 2024),
DeepSeek-Coder (Guo et al., 2024), and CodeQwen-V1.5 (Bai et al., 2023), have
demonstrated promising results in code generation for prevalent programming
languages like Python and C/C++, while their performance in RTL code
generation still lags behind GPT-3.5. This underscores the pressing need to
develop a specialized open-source LLM tailored for RTL code generation.

The scarcity of high-quality RTL code datasets is a significant factor
contributing to the poor quality of RTL code generation. In LLM training, both
the scale and quality of the dataset are essential for effective code
generation performance. However, compared to other popular programming
languages, there is a notable shortage of high-quality, large-scale RTL code
datasets, which poses a significant challenge for achieving satisfactory
results in RTL generation tasks. To address this issue, several studies have
focused on collecting open-source code snippets (Thakur et al., 2023a, b; Pei
et al., 2024; Dehaerne et al., 2023). Nevertheless, these open-source datasets
may include low-quality code that can impair the model’s performance (Liu et
al., 2023c). Other researchers have developed methodologies for synthesizing
RTL code using LLM (Liu et al., 2023c; Chang et al., 2024; Liu et al., 2023b).
VerilogEval (Liu et al., 2023c) generates datasets by adding descriptions to
open-source Verilog code but suffers from its average low quality. RTLCoder
(Liu et al., 2023b) selects hardware-related keywords as seeds to generate
natural language instructions, which are then converted into corresponding RTL
code by GPT-3.5. Although this dataset meets quality standards, it is limited
in scale because the generated data is restricted to the selected keywords.

Self-reflection is also a critical factor to be considered in the field of RTL
code generation. In the context of RTL code generation, self-reflection refers
to the model’s assessment of its generated code based on feedback from the
compiler and simulator. This process involves identifying the causes of errors
and refining the results accordingly, just as in the hardware design process,
where RTL code undergoes rigorous evaluation by the compiler and simulator to
ensure its correctness and adherence to design specifications.

However, existing open-source LLMs mainly focus on code generation, neglecting
the interactive process of compilation and simulation in hardware design
methodologies. In consideration of the above limitations, several studies have
integrated self-reflection methodologies into their frameworks (Tsai et al.,
2023; Thakur et al., 2023c; Wu et al., 2024). Nonetheless, both RTLFixer (Tsai
et al., 2023) and AutoChip (Thakur et al., 2023c) rely on commercial LLMs for
self-reflection, which raises potential concerns regarding privacy and
security, as previously discussed. Open-source LLMs generally exhibit weaker
self-reflection capabilities compared to commercial LLMs, primarily due to
their limited training data. Consequently, there is a need to develop a
methodology for constructing datasets specifically designed to enhance the
self-reflection abilities of open-source LLMs.

In this paper, we introduce OriGen, a fully open-source framework featuring
self-reflection capabilities and a dataset augmentation methodology. We
propose a novel code-to-code augmentation that leverages knowledge
distillation to improve the quality of open-source RTL code datasets, which
are of a large scale but have limited quality due to the lack of rigorous
review and validation processes.

Our approach involves extracting high-level code descriptions from the open-
source RTL code and refining the code based on these descriptions using the
Claude3-Haiku model, which acts as the teacher model. Commercial LLMs have
been trained on vast amounts of high-quality data and possess strong language
understanding and generation capabilities. By leveraging these models as
teachers, we can distill their knowledge and refine the open-source RTL code.
This method enables the generated code to potentially achieve even superior
performance of the teacher model.

To evaluate the LLM’s capability of self-reflection in RTL code, we construct
a benchmark named VerilogFixEval that comprises 221 cases of failed
compilation from VerilogEval (Liu et al., 2023c). OriGen is able to correct
syntactic errors by leveraging a self-reflection process based on feedback
from the compiler. When the generated code fails to pass the compiler’s
verification, the model initiates the self-reflection process. During this
phase, the model accepts the erroneous code and compiler error messages to fix
the RTL code, enhancing its correctness and reliability. The self-reflection
ability of the model is facilitated by a carefully constructed error-
correction dataset, which comprises a comprehensive collection of samples.
This dataset includes natural language instructions, erroneous code, compiler
error messages, and the corrected code generated by Claude3-Haiku. It serves
as a valuable resource to bridge the gap in self-reflection capabilities
between open-source and commercial closed-source LLMs. By training on this
dataset, OriGen can acquire self-reflection abilities comparable to those of
commercial LLMs, enabling it to generate more reliable and accurate RTL code.

Our contributions are summarized as follows:

  * •

We introduce OriGen, which significantly outperforms other alternatives
designed for RTL code generation and achieves performance comparable to the
latest version of GPT-4 Turbo.

  * •

We propose a novel code-to-code augmentation methodology for generating high-
quality, large-scale RTL code datasets, enhancing the model’s training data.

  * •

we introduce a self-reflection mechanism that enables OriGen to autonomously
fix syntactic errors by leveraging feedback from the compiler, improving its
code generation accuracy. Furthermore, we construct a dataset to improve the
model’s capability of self-reflection based on compiler error messages and
develop a benchmark to evaluate this capability.

Experimental results demonstrate that OriGen significantly outperforms other
alternatives designed for RTL code generation (Pei et al., 2024; Liu et al.,
2023b; Nadimi and Zheng, 2024; Liu et al., 2023c, a) surpassing the previous
best-performing LLM by 9.8% on the VerilogEval-Human (Liu et al., 2023c)
benchmark. Furthermore, experimental results on VerilogFixEval illustrate that
OriGen’s self-reflection capability in RTL code generation not only surpasses
other alternatives but also outperforms GPT-4 Turbo by 18.1%.

##  2\. Background and Related Work

###  2.1. LLMs for Verilog Generation

Large Language Models (LLMs) have emerged as powerful tools for code
generation across various programming languages. Trained on vast amounts of
code and natural language data, these models can learn the statistical
patterns and relationships within the training data, enabling them to generate
code that adheres to the syntax and style of the target programming language.
Recently, researchers in the field of hardware design have shown a growing
interest in leveraging LLMs for generating RTL code. RTL is a crucial
abstraction level in hardware design, describing the flow of data and control
signals between registers and combinational logic. By fine-tuning LLMs on RTL
code datasets, models are capable of generating RTL code, potentially
accelerating the hardware design process.

Among the pioneering endeavors, DAVE (Pearce et al., 2020) represents an
initial exploration into the generation of Verilog code from natural language
instructions through the fine-tuning of the GPT-2 model. Subsequently,
recognizing the notable scarcity of large-scale Verilog datasets relative to
more common programming languages, Verigen (Thakur et al., 2023b) collects a
comprehensive Verilog dataset from GitHub’s open-source repositories and
various textbooks. However, the performance of its optimally fine-tuned model
based on the collected dataset remains inferior to that of GPT-3.5 (Lu et al.,
2024). This can be attributed to the varying quality of the collected open-
source code, which was not filtered and processed.

Both VerilogEval (Liu et al., 2023c) and RTLCoder (Liu et al., 2023b)
acknowledge the importance of dataset quality and adopt data synthesis methods
using commercial LLMs to improve the quality of their datasets. VerilogEval
proposes a synthesis approach that utilizes GPT-3.5 to add descriptions to
each code snippet in the filtered VeriGen (Thakur et al., 2023b) dataset. This
approach allows the LLM trained on the dataset to learn not only the syntax of
Verilog but also to understand the correspondence between Verilog code and
natural language. However, due to the average low quality of code in the
dataset, VerilogEval still falls short of GPT-3.5. RTLCoder (Liu et al.,
2023b) adopts a synthesis approach to generate RTL code from natural language
specifications. It selects one or two keywords from a prepared list of
hundreds of digital design keywords and utilizes GPT-3.5 to generate RTL
design instructions based on these keywords. Subsequently, GPT-3.5 is employed
again to generate corresponding RTL code from these instructions. However, the
reliance on a limited set of keywords poses challenges for further expanding
this dataset, as it may restrict the scale and complexity of the generated RTL
code. In contrast, the dataset synthesized through the code-to-code approach
is not only of high quality, as demonstrated by the experimental results, but
also extensive in scale, encompassing a diverse range of RTL code samples.
This approach overcomes the limitations of above synthesis and enables the
generation of a comprehensive and high-quality dataset that can effectively
support the training of LLMs for RTL code generation.

Apart from fine-tuning open-source models, several studies (Blocklove et al.,
2023; Chang et al., 2023; Fu et al., 2023) investigate the direct application
of commercial LLMs for generating RTL code. Chip-Chat (Blocklove et al., 2023)
employs a conversational interface to design and verify an 8-bit accumulator-
based microprocessor using GPT-4. ChipGPT (Chang et al., 2023) introduces a
four-stage, zero-code logic design framework that leverages GPT model.

###  2.2. Reflection for Verilog Generation

Existing open-source LLMs designed for RTL code generation have mainly focused
on the generative aspects, neglecting the crucial stages of verification and
reflection that are integral to hardware design methodologies. The
effectiveness of self-reflection and receptivity to feedback in addressing
real-world issues has been demonstrated by SWE-agent (Yang et al., 2024),
which successfully resolved problems in GitHub’s repositories. This highlights
the potential benefits of incorporating self-reflection mechanisms into LLMs
for RTL code generation, as it could enable the models to learn from feedback
and iteratively improve the generated code, aligning with the rigorous
verification processes inherent to hardware design methodologies.

Recognizing the limitations of previous research on LLMs for RTL generation,
AutoChip (Thakur et al., 2023c) and RTLFixer (Tsai et al., 2023) decide to
introduce self-reflection into the their framework. Specifically, RTLFixer
employs GPT to generate and reflect on code. It includes approximately 80
erroneous code snippets that fail compilation, each paired with guidance.
During the RTL code generation process, if the generated code fails to
compile, RTLFixer (Tsai et al., 2023) queries the database for the guidance
most closely related to the error. However, the generalizability of this
approach is limited due to the constraint of the guidance database. In
contrast, AutoChip (Thakur et al., 2023c) directly provides the compiler error
messages to the LLM rather than querying a database for guidance.

The error-correction dataset we construct is diverse and not limited to a
specific set of errors, enabling the models to handle a broader range of
issues encountered during RTL code generation. By leveraging this dataset, the
models can improve their capabilities of self-reflection.

###  2.3. Knowledge Distillation

To enable open-source models to acquire capabilities comparable to commercial
closed-source models, knowledge distillation has become a focus for
researchers. (Xu et al., 2024). A common approach involves leveraging the
closed-source LLM to provide reference answers, which are then used to fine-
tune the open-source LLM through Supervised Fine-Tuning (SFT), enabling the
latter to learn from the knowledge embodied in the closed-source model.
Specifically, SFT is a effective knowledge distillation method. In this
approach, the teacher model, typically an advanced LLM, is first used to
generate a large number of input-output data pairs. Subsequently, these data
pairs generated by the teacher model are utilized as training data to fine-
tune the student model, typically a smaller open-source language model.
Through this process, the student model can mimic the behavior of the teacher
model, thereby acquiring the knowledge within the teacher model.

The code-to-code augmentation methodology employed in this work falls under
the realm of knowledge distillation techniques. By leveraging the knowledge
within these advanced models, the proposed method can effectively distill and
transfer their capabilities to the target model, enabling it to acquire the
desired capability of RTL code generation in a scalable and automated manner.

##  3\. Methodology

In this section, we provide comprehensive details of OriGen, a powerful
framework designed to enhance Verilog code generation.

###  3.1. Overview

Figure 1. Code-to-Code Augmentation

.

Figure 2. Generation and Self-Reflection

.

The overview of the code-to-code augmentation methodology and OriGen’s
generation and self-reflection framework are illustrated in Figure 1 and
Figure 2 respectively. As shown in Figure 1, in the code-to-code augmentation
process, two datasets are generated: an enhanced code dataset and an error-
correction dataset. The enhanced code dataset contains code-description paired
data samples, enabling the LLM to learn the correspondence between Verilog
code and natural language. The error-correction dataset comprises erroneous
code-fixed code paired data samples, allowing the LLM to enhance its self-
reflection capability by learning from examples of code errors and their
corrections.

As shown in Figure 2, in the code generation and error correction stage,
OriGen comprises a base LLM and two trained LoRA models: Gen LoRA and Fix
LoRA. Gen LoRA is trained on the enhanced code dataset and is responsible for
the initial generation of RTL code upon receiving natural language
instructions. Fix LoRA, which is trained on the error-correction dataset based
on Gen LoRA, is tasked with fixing erroneous code that fails to compile. The
fix process is implemented as a self-reflection loop, iteratively refining the
code until it passes verification or reaches a predetermined maximum number of
iterations.

###  3.2. Code-to-Code Augmentation

The code-to-code augmentation process aims to transfer advanced RTL code
generation and correction capabilities from commercial LLMs to our model. To
achieve this, a carefully filtered collection of comprehensive open-source RTL
code samples is utilized as a foundation.

To extract a valuable dataset from open-source RTL code samples (Lozhkov et
al., 2024; Thakur et al., 2023b), a rigorous filtration process is applied.
Initially, due to the constraints imposed by the model’s context window and
the challenges associated with incomplete descriptions for longer code
snippets, samples exceeding 300 lines or 1536 tokens are excluded.

Additionally, samples with an average of more than 30 tokens (approximately 90
characters) per line are considered non-standard and are consequently
eliminated, ensuring that only concise and standard code samples are retained.
Subsequently, to ensure the meaningfulness and substantive content of the code
snippets, a keyword-based filtration approach is employed. Each sample must
contain both the module and endmodule keywords, alongside at least one
occurrence of keywords related to procedural blocks, always (inclusive of
variants like always_comb, always_ff, always_latch, _etc_.) or assign. This
criterion guarantees that the selected snippets are representative of
functional and logical hardware designs. Lastly, all comments within the code
samples are removed. This step is crucial to prevent extraneous information
from influencing the generation of accurate and relevant specifications.

Following the implementation of the aforementioned filtering procedures, the
closed-source LLM Claude3-Haiku is utilized to generate detailed descriptions
corresponding to the filtered code samples as shown in Figure 1. The prompt we
use is illustrated in Figure 3. These descriptions are then used to regenerate
RTL code, which replaces the original code in the dataset. During the
regeneration process, the generated code undergoes verification using the
open-source compiler Icarus Verilog (Iverilog) (Williams and Baxter, 2002). If
the code fails to compile, the compiler’s error messages and the erroneous
code are fed back into the LLM for regeneration to fix error.

Description Prompt Explain the high-level functionality of the Verilog module.
Task:
Please analyze it and provide a detailed description of its signals and
functionality. Use as many high-level concepts that are directly applicable to
describe the code, but do not include extraneous details that aren’t
immediately applicable. Speak concisely as if this was a specification for a
circuit designer to implement. You should only reply with descriptive natural
language and not use any code. Code:
{code} Response: Figure 3. Prompt for Generating Code Descriptions

Simultaneously, code samples that fail compilation are utilized as
foundational elements for the error-correction dataset, as detailed in Section
3.3. This iterative process is repeated until the code successfully compiles
or until the maximum number of iterations is reached, further enhancing the
dataset’s quality. This method facilitates the transfer of knowledge from the
commercial LLM to our model, thereby improving its capability to generate and
correct RTL code efficiently.

A full example of dataset augmentation is shown in Figure 4.

Figure 4. Example of Dataset Augmentation Figure 5. Example of Self-Reflection
Debug Instruction As a professional Verilog designer, you are tasked with
debugging a Verilog module that has some errors. Below are the details of the
assignment, the original Verilog code with some syntax and functional errors,
and the error messages produced by the compiler. Please review this
information and provide a corrected version of the code. Task:
{task} Original Code:
{original_code} Compiler Error Message:
{error} Figure 6. Debug Instruction Template

###  3.3. Error-Correction Dataset

Although OriGen, after being trained on the enhanced code dataset,
demonstrates performance comparable to that of advanced closed-source LLMs in
RTL code generation, it exhibits weaker self-reflection capabilities compared
to commercial LLMs like other open-source LLMs.

Given the observed performance gap, we decide to adopt the knowledge
distillation method to acquire stronger error understanding and self-
reflection capabilities from the closed-source LLM. During the code-to-code
augmentation process discussed in Section 3.2, the closed-source LLM generates
a diverse set of RTL code samples that fail to compile, which are then
corrected in the subsequent self-reflection process. We select the code
samples generated in the code-to-code augmentation process that pass
compilation after correction, along with the corresponding samples that failed
to compile. The natural language instruction descriptions and compiler error
messages associated with these samples serve as data sources for constructing
the error-correction dataset. This approach enables the model to be exposed to
a broad spectrum of error types and their respective corrections, facilitating
the development of robust error understanding and self-reflection
capabilities.

Subsequently, these data samples are further filtered to ensure that the model
learns to make modifications within the module’s body rather than altering the
module’s declaration. Specifically, during the self-reflection and
rectification process, the model may attempt to correct syntactic errors by
modifying declarations to pass the compilation check though it is explicitly
prohibited in the prompt. To prevent this, we remove samples with such
modifications from the collected code. This approach is crucial for preserving
the overall structure and interface of the module, which is essential for
maintaining compatibility with other modules in the design. Meanwhile, by
restricting modifications to the module’s body, we encourage the model to
focus on identifying and correcting errors within the actual implementation
logic. Through the above methods, we have successfully constructed a high-
quality error-correction dataset. This dataset comprises a large number of
effectively repaired code samples generated by the Claude3-Haiku model, along
with the corresponding erroneous code, compiler error messages, and natural
language instructions.

To verify the effectiveness of the error-correction dataset, we introduce a
benchmark named VerilogFixEval detailed in Section 3.5.

###  3.4. Code Generation and Fix

As shown in Figure 2, in the code generation and error correction stage,
OriGen comprises a base LLM and two trained LoRA models: Gen LoRA and Fix
LoRA. Following training on the enhanced code dataset, Gen LoRA has developed
robust RTL code generation capabilities but exhibits limitations in self-
reflection. To further enhance its capabilities of self-reflection, Gen LoRA
is trained on the error-correction dataset, resulting in Fix LoRA.

The reason for employing two LoRA models, Gen LoRA and Fix LoRA, instead of a
single LoRA is based on experimental results, which indicate that Fix LoRA,
trained on the error-correction dataset, exhibits inferior performance
compared to Gen LoRA in the RTL code generation task. This performance
degradation may be attributed to the fact that training on a dataset
containing syntactic errors could potentially weaken the model’s overall
performance. Therefore, OriGen adopts a two-LoRA approach, where Gen LoRA is
responsible for generating the initial code, and Fix LoRA is tasked with
rectifying syntactic errors, utilizing its specialized training on the error-
correction dataset.

The generative and iterative self-reflection process is illustrated in Figure
2. Initially, OriGen accepts the user’s natural language specification and
proceeds to generate the corresponding RTL code using Gen LoRA. The generated
code is then subjected to the compiler’s verification process using Iverilog.
In cases where the code fails to compile, OriGen initiates a self-reflection
process to fix the code. This process involves utilizing the erroneous code,
compiler error messages, and natural language instructions to guide the model
in identifying and rectifying the issues. The self-reflection loop continues
until the generated code successfully passes compilation or reaches a
predefined maximum number of iterations.

An example of self-reflection is illustrated in Figure 5. The generated code
initially contains a syntactic error where the wires ply_temp and p2y_temp are
assigned values within always blocks, which is not permitted in Verilog as
wires cannot serve as left-hand side values for procedural assignments. This
error is subsequently rectified by leveraging the Fix LoRA model during the
self-reflection process, using the instruction template shown in Figure 6. The
rectified code employs the assign statement to perform the assignments, rather
than attempting to assign values within the always blocks, which was the
source of the initial error.

###  3.5. VerilogFixEval

To evaluate the capability of different models in reflecting on and improving
from Verilog compiler error messages, we constructed the VerilogFixEval
benchmark. The benchmark comprises code samples that failed to pass the
compilation verification, along with the corresponding natural language
instructions and compiler error messages. The erroneous RTL code samples were
selected from code generated by LLMs that performed comparable with GPT-3.5 on
the VerilogEval benchmark. This selection approach was adopted since OriGen
achieves a relatively high compilation pass rate. By including code samples
from poorly performing LLMs, the benchmark ensures the diversity of errors
while avoiding potential bias in the test results that may favor OriGen due to
errors generated by OriGen itself.

During the evaluation process, the model will receive natural language
instructions, erroneous RTL code, and compiler error messages, and will be
required to correct the RTL code. The final evaluation metrics include two
parts, syntactic correctness and functional correctness.

##  4\. Evaluation

###  4.1. Experimental Setting

For our pre-trained model, we selected the DeepSeek-Coder-7B-Instruct model,
as it exhibits the best performance in Verilog code generation among all 7B
models, to the best of our knowledge. It is to ensure that our pre-trained
model possesses strong capabilities in the domain of Verilog code generation,
providing a solid foundation for further fine-tuning and evaluation.

To evaluate the performance of Verilog code generation, we selected two
representative benchmarks: VerilogEval (Liu et al., 2023c) and RTLLM (Lu et
al., 2024). The former, VerilogEval, originates from approximately 150 Verilog
tasks on the HDLBits website, which were manually converted to create
VerilogEval-Human and generated by GPT-3.5 to produce VerilogEval-Machine. The
latter benchmark, RTLLM, consists of 29 Verilog tasks with more diverse levels
of difficulty, closely aligned with real-world design tasks. Both benchmarks
employ the widely adopted pass@k evaluation metric to assess the correctness
of the generated code’s functionality. In this metric, if any one of the k
samples passes the unit test, the problem is considered solved.

(1)
$pass@k:=\underset{Problems}{\operatorname*{\mathbb{E}}}\left[1-\frac{\binom{n-c}{k}}{\binom{n}{k}}\right]$

where we generate $n\geq k$ samples for each instruction in which $c\leq n$
samples pass testing. We choose $n=10$ in experiments.

To assess the models’ capability for self-reflection, we utilized the
VerilogFixEval benchmark, as discussed in Section 3.5. This benchmark employs
the pass@1 metric to evaluate the syntactic and functional accuracy of the
rectified RTL code.

###  4.2. Model Training

Table 1. Comparison of functional correctness on VerilogEval (Liu et al., 2023c) and RTLLM (Lu et al., 2024) [1.2pt] font= Source |  Name |  VerilogEval-human(%) |  |  |  VerilogEval-machine(%) |  |  | RTLLM(%)
---|---|---|---|---|---|---|---|---
font= |  | pass@1 | pass@5 | pass@10 | pass@1 | pass@5 | pass@10 | pass@5
Commercial LLM | GPT-3.5 (Achiam et al., 2023) | 35.6 | 48.8 | 52.6 | 49.4 | 72.7 | 77.6 | 44.8
| GPT-4 2023-06-13 (Achiam et al., 2023) | 43.5 | 55.8 | 58.9 | 60.0 | 70.6 | 73.5 |  65.5
| GPT-4 Turbo 2024-04-09 (Achiam et al., 2023) |  54.2 |  68.5 |  72.4 | 58.6 | 71.9 | 76.2 |  65.5
| Claude3-Haiku (Anthropic, 2024) | 47.5 | 57.7 | 60.9 |  61.5 | 75.6 | 79.7 | 62.1
| Claude3-Sonnet (Anthropic, 2024) | 46.1 | 56.0 | 60.3 | 58.4 | 71.8 | 74.8 | 58.6
| Claude3-Opus (Anthropic, 2024) |  54.7 |  63.9 |  67.3 | 60.2 | 75.5 | 79.7 |  69.0
Open Source Models | CodeLlama-7B-Instruct (Roziere et al., 2023) | 18.2 | 22.7 | 24.3 | 43.1 | 47.1 | 47.7 | 34.5
| CodeQwen1.5-7B-Chat (Bai et al., 2023) | 22.4 | 41.1 | 46.2 | 45.1 | 70.2 | 77.6 | 37.9
| DeepSeek-Coder-7B-Instruct-v1.5 (Guo et al., 2024) | 31.7 | 42.8 | 46.8 | 55.7 | 73.9 | 77.6 | 37.9
Verilog-Specific Models | ChipNeMo (Liu et al., 2023a) | 22.4 | - | - | 43.4 | - | - | -
| VerilogEval (Liu et al., 2023c) | 28.8 | 45.9 | 52.3 | 46.2 | 67.3 | 73.7 | -
| RTLCoder-DeepSeek (Liu et al., 2023b) | 41.6 | 50.1 | 53.4 | 61.2 |  76.5 |  81.8 | 48.3
| CodeGen-6B MEV-LLM (Nadimi and Zheng, 2024) | 42.9 | 48.0 | 54.4 | 57.3 | 61.5 | 66.4 | -
| BetterV-CodeQwen (Pei et al., 2024) | 46.1 | 53.7 | 58.2 |  68.1 |  79.4 |  84.5 | -
OriGen (ours) |  |  51.4 |  58.6 |  62.2 |  76.2 |  84.0 |  86.7 |  65.5
[1.2pt] |  |  |  |  |  |  |  |

We employ the LoRA (Low-Rank Adaptation) (Hu et al., 2021) method to train the
model’s capability in generating RTL code. This approach allows for enhancing
the model’s specific abilities in RTL code generation while minimizing the
impact on its other capabilities. For all the training processes, we employ
the float16 mixed precision method, although the model is trained in bfloat16
precision. We utilize the Adam optimizer with $\beta_{1}=0.9$,
$\beta_{2}=0.999$, and the cosine learning rate decay to schedule our learning
rate. The warm-up ratio is set to 0.03.

During the training process, we initially segment the dataset into smaller
subsets, each containing 10k samples, resulting in a total of 18 groups. We
then perform fine-tuning on these smaller datasets to evaluate their quality
and assess the model’s performance training on them. Throughout the
experiments, it is evident that the dataset’s performance is relatively
sensitive to the learning rate.

As illustrated in Figure 7, we choose several subsets of the dataset,
including 6 smaller 10k-samples datasets and 2 larger 30k-samples datasets.
Employing varying learning rates during the training phase lead to noticeable
differences in the pass@1 metric on the VerilogEval benchmark (Liu et al.,
2023c). Simultaneously, the results obtained from training on different
datasets exhibit significant variations. These observations highlight the
sensitivity of the model’s performance to the choice of learning rate and the
characteristics of the training dataset. In response, we implement a strategy
of training each small dataset with its optimal learning rate. Furthermore, we
conduct training exclusively on those datasets where the pass@1 metric surpass
42%. This approach mitigate the sensitivity of the model’s performance to the
learning rate, ensuring that each dataset subset is trained with the most
suitable learning rate to achieve optimal results. Additionally, it avoids the
detrimental influence of underperforming datasets.

Figure 7. Impact of Learning Rate for Different Datasets

###  4.3. Functional Correctness

Table 1 presents the results on the VerilogEval benchmark. To ensure fairness,
we did not utilize the self-reflection feature for this comparison and
generated code in a single attempt, like other models. The models compared
include closed-source commercial LLMs such as GPT-3.5/GPT-4,
Claude3-Haiku/Sonnet/Opus, general open-source code models (Guo et al., 2024;
Bai et al., 2023), and models customized for RTL code generation (Liu et al.,
2023a, c, b; Pei et al., 2024; Nadimi and Zheng, 2024).

In the VerilogEval benchmark (Liu et al., 2023c), for pass@1 in Human and
Machine categories, OriGen achieves 76.1% and 51.4% respectively. It
outperforms remarkably all other alternatives designed for RTL code
generation, for example, compared to the best-performed fully open-source
RTLCoder (Liu et al., 2023b), OriGen surpasses it by 9.8% in Human benchmark.
When compared with commercial closed-source models, OriGen also achieves
excellent performance. It significantly surpasses GPT-4 of 2023 version, which
was previously the widely compared baseline. Even when compared with the
state-of-the-art models, OriGen outperforms Claude3-Haiku/Sonnet, only
slightly inferior to the current best models GPT-4 Turbo and Claude3-Opus by
3% in Human benchmark. Moreover, as evident from Table 1, it achieves the best
performance on the Machine benchmark, remarkably outperforming other models
including GPT-4 Turbo.

To prevent our model from over-fitting on the VerilogEval benchmark, we also
conduct experiments on another benchmark designed for RTL code generation,
RTLLM (Lu et al., 2024). Similar results are observed on RTLLM as shown in
Table 1, where OriGen significantly outperforms other models, achieving
performance comparable to GPT-4 Turbo and Claude3-Opus.

In summary, OriGen outperforms all non-commercial models remarkably in all
metrics on both benchmarks.

Moreover, as illustrated in Figure 8, we observe that the strategy of training
each small dataset with its optimal learning rate and excluding the datasets
with relatively poor performance allows the model’s performance to gradually
improve as the dataset grows larger. This demonstrates the effectiveness of
the training methodology.

Figure 8. Pass@1 Variation with Dataset Size

###  4.4. Capability of Self-Reflection

The evaluation metrics of VerilogFixEval consist of two components: syntactic
correctness and functional correctness. Syntactic correctness assesses whether
the generated rectified RTL code successfully compiles. Functional correctness
further evaluates whether the generated rectified RTL code passes simulation
tests.

As illustrated in Table 2, experimental results on VerilogFixEval demonstrate
that OriGen achieves the best performance in syntactic error correction,
surpassing GPT-4 Turbo by 18.1%. This indicates that after training on the
code-correction dataset, the model acquires powerful self-reflection and code
rectification capabilities. For functional correctness, OriGen significantly
outperforms RTLCoder by 24.5% and falls short of the best-performing model,
Claude3-Opus, by only 6.8%. This showcases OriGen’s ability to consider the
functional correctness of the generated code while rectifying syntactic
errors.

Table 2. Comparison of Models on VerilogFixEval [1.2pt] font= Model | Syntactic correctness(%) | Functional correctness(%)
---|---|---
CodeQwen1.5-7B-Chat (Bai et al., 2023) | 27.6 | 10.4
RTLCoder (Liu et al., 2023b) | 46.6 | 15.3
DeepSeek-Coder-7B (Guo et al., 2024) | 50.7 | 19.0
GPT-3.5 (Achiam et al., 2023) | 40.7 | 13.1
GPT-4 Turbo 2024-04-09 (Achiam et al., 2023) | 69.2 | 37.1
Claude3-Haiku (Anthropic, 2024) | 48.9 | 23.5
Claude3-Opus (Anthropic, 2024) | 71.9 |  39.8
OriGen (Ours) |  87.3 | 33.0
[1.2pt] |  |

###  4.5. Ablation Studies

We perform two ablation experiments to investigate the efficacy of the code-
to-code augmentation method and to examine the model’s self-reflection
capability before and after training on the error-correction dataset.

For the ablation study of the code-to-code data augmentation method, we
compared the performance of models trained on RTL code dataset before and
after code-to-code augmentation method.

Table 3. Ablation Study of Code-to-Code Augmentation

rowsep=1pt,colsep=2pt  [1.2pt] font= Benchmark Metric Baseline(%) Augment(%)
VerilogEval-Human (Liu et al., 2023c) Pass@1 38.7 51.4 VerilogEval-Machine
(Liu et al., 2023c) Pass@1 62.5 76.2 RTLLM (Lu et al., 2024) Pass@5 41.4 65.5
[1.2pt]

Table 4. Ablation Study of Error-Correction Dataset on VerilogFixEval

rowsep=1pt,colsep=2pt  [1.2pt] font= Model Syntactic Correctness(%) Functional
Correctness(%) baseline 53.4 24.4 baseline + error message 62.9 25.8 finetune
82.3 31.6 finetune + error message 87.3 33.0 [1.2pt]

Table 3 demonstrates that across multiple evaluation metrics, the model
trained on the dataset enhanced by code-to-code augmentation significantly
outperforms the model trained on the unaugmented dataset. This indicates that
our model’s outstanding performance is primarily attributed to the code-to-
code augmentation methodology, as training on the unaugmented dataset does not
yield substantial improvements.

For the ablation study of the error-correction dataset, the results are
presented in Table 4. We evaluate four scenarios, considering the model’s
performance before and after training on the error-correction dataset, as well
as with and without the use of error messages from the compiler. The former is
denoted as ”baseline” and ”finetune,” indicating whether the model is trained
on the error-correction dataset, while the latter is represented by ”error
message.”

The results demonstrate that after training on the error-correction dataset,
the model significantly outperforms its pre-training performance in both
syntactic correctness and functional Correctness. Additionally, the trained
model exhibits a reduced reliance on error messages from the compiler.

##  5\. conclusion

This paper introduces OriGen, an open-source framework for RTL code
generation. It proposes a novel code-to-code augmentation methodology to
generate high-quality, large-scale RTL code datasets, which enhances the
model’s training data. The framework also introduces a self-reflection
mechanism that allows OriGen to autonomously fix syntactic errors by
leveraging compiler feedback, thereby improving its code generation accuracy.
Furthermore, we construct a dataset to improve the model’s capability of self-
reflection based on compiler error messages and erroneous code and develop a
benchmark to evaluate this capability. Experimental results demonstrate that
OriGen remarkably outperforms other open-source alternatives in RTL code
generation, surpassing the previous best-performing LLM by 9.8% on the
VerilogEval-Human benchmark and is comparable with GPT-4. Moreover, OriGen
exhibits superior capabilities in self-reflection and error rectification,
surpassing GPT-4 by 18.1% in syntactic correctness on the VerilogFixEval
benchmark.

## References

  * (1)
  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023\.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_ (2023).
  * Anthropic (2024) Anthropic. 2024\.  Introducing the next generation of Claude.  https://www.anthropic.com/news/claude-3-family
  * Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023.  Qwen Technical Report.  _arXiv preprint arXiv:2309.16609_ (2023).
  * Blocklove et al. (2023) Jason Blocklove, Siddharth Garg, Ramesh Karri, and Hammond Pearce. 2023.  Chip-chat: Challenges and opportunities in conversational hardware design. In _2023 ACM/IEEE 5th Workshop on Machine Learning for CAD (MLCAD)_. IEEE, 1–6.
  * Chang et al. (2024) Kaiyan Chang, Kun Wang, Nan Yang, Ying Wang, Dantong Jin, Wenlong Zhu, Zhirong Chen, Cangyuan Li, Hao Yan, Yunhao Zhou, et al. 2024\.  Data is all you need: Finetuning LLMs for Chip Design via an Automated design-data augmentation framework.  _arXiv preprint arXiv:2403.11202_ (2024).
  * Chang et al. (2023) Kaiyan Chang, Ying Wang, Haimeng Ren, Mengdi Wang, Shengwen Liang, Yinhe Han, Huawei Li, and Xiaowei Li. 2023\.  Chipgpt: How far are we from natural language hardware design.  _arXiv preprint arXiv:2305.14019_ (2023).
  * Dehaerne et al. (2023) Enrique Dehaerne, Bappaditya Dey, Sandip Halder, and Stefan De Gendt. 2023.  A deep learning framework for verilog autocompletion towards design and verification automation.  _arXiv preprint arXiv:2304.13840_ (2023).
  * Fu et al. (2023) Yonggan Fu, Yongan Zhang, Zhongzhi Yu, Sixu Li, Zhifan Ye, Chaojian Li, Cheng Wan, and Yingyan Celine Lin. 2023\.  Gpt4aigchip: Towards next-generation ai accelerator design automation via large language models. In _2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD)_. IEEE, 1–9.
  * Guo et al. (2024) Daya Guo, Qihao Zhu, Dejian Yang, Zhenda Xie, Kai Dong, Wentao Zhang, Guanting Chen, Xiao Bi, Y Wu, YK Li, et al. 2024\.  DeepSeek-Coder: When the Large Language Model Meets Programming–The Rise of Code Intelligence.  _arXiv preprint arXiv:2401.14196_ (2024).
  * Hu et al. (2021) Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021\.  Lora: Low-rank adaptation of large language models.  _arXiv preprint arXiv:2106.09685_ (2021).
  * Kalyan et al. (2021) Katikapalli Subramanyam Kalyan, Ajit Rajasekharan, and Sivanesan Sangeetha. 2021.  Ammus: A survey of transformer-based pretrained models in natural language processing.  _arXiv preprint arXiv:2108.05542_ (2021).
  * Liu et al. (2023a) Mingjie Liu, Teodor-Dumitru Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjian Liang, Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktaroglu, et al. 2023a.  Chipnemo: Domain-adapted llms for chip design.  _arXiv preprint arXiv:2311.00176_ (2023).
  * Liu et al. (2023c) Mingjie Liu, Nathaniel Pinckney, Brucek Khailany, and Haoxing Ren. 2023c.  Verilogeval: Evaluating large language models for verilog code generation. In _2023 IEEE/ACM International Conference on Computer Aided Design (ICCAD)_. IEEE, 1–8.
  * Liu et al. (2023b) Shang Liu, Wenji Fang, Yao Lu, Qijun Zhang, Hongce Zhang, and Zhiyao Xie. 2023b.  Rtlcoder: Outperforming gpt-3.5 in design rtl generation with our open-source dataset and lightweight solution.  _arXiv preprint arXiv:2312.08617_ (2023).
  * Lozhkov et al. (2024) Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. 2024\.  StarCoder 2 and The Stack v2: The Next Generation.  _arXiv preprint arXiv:2402.19173_ (2024).
  * Lu et al. (2024) Yao Lu, Shang Liu, Qijun Zhang, and Zhiyao Xie. 2024\.  RTLLM: An open-source benchmark for design rtl generation with large language model. In _2024 29th Asia and South Pacific Design Automation Conference (ASP-DAC)_. IEEE, 722–727.
  * Nadimi and Zheng (2024) Bardia Nadimi and Hao Zheng. 2024.  A Multi-Expert Large Language Model Architecture for Verilog Code Generation.  _arXiv preprint arXiv:2404.08029_ (2024).
  * Pearce et al. (2020) Hammond Pearce, Benjamin Tan, and Ramesh Karri. 2020.  Dave: Deriving automatically verilog from english. In _Proceedings of the 2020 ACM/IEEE Workshop on Machine Learning for CAD_. 27–32.
  * Pei et al. (2024) Zehua Pei, Hui-Ling Zhen, Mingxuan Yuan, Yu Huang, and Bei Yu. 2024.  BetterV: Controlled Verilog Generation with Discriminative Guidance.  _arXiv preprint arXiv:2402.03375_ (2024).
  * Roziere et al. (2023) Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. 2023\.  Code llama: Open foundation models for code.  _arXiv preprint arXiv:2308.12950_ (2023).
  * Sun et al. (2024) Qiushi Sun, Zhirui Chen, Fangzhi Xu, Kanzhi Cheng, Chang Ma, Zhangyue Yin, Jianing Wang, Chengcheng Han, Renyu Zhu, Shuai Yuan, Qipeng Guo, Xipeng Qiu, Pengcheng Yin, Xiaoli Li, Fei Yuan, Lingpeng Kong, Xiang Li, and Zhiyong Wu. 2024\.  A Survey of Neural Code Intelligence: Paradigms, Advances and Beyond.  arXiv:2403.14734
  * Thakur et al. (2023a) Shailja Thakur, Baleegh Ahmad, Zhenxing Fan, Hammond Pearce, Benjamin Tan, Ramesh Karri, Brendan Dolan-Gavitt, and Siddharth Garg. 2023a.  Benchmarking large language models for automated verilog rtl code generation. In _2023 Design, Automation & Test in Europe Conference & Exhibition (DATE)_. IEEE, 1–6.
  * Thakur et al. (2023b) Shailja Thakur, Baleegh Ahmad, Hammond Pearce, Benjamin Tan, Brendan Dolan-Gavitt, Ramesh Karri, and Siddharth Garg. 2023b.  Verigen: A large language model for verilog code generation.  _ACM Transactions on Design Automation of Electronic Systems_ (2023).
  * Thakur et al. (2023c) Shailja Thakur, Jason Blocklove, Hammond Pearce, Benjamin Tan, Siddharth Garg, and Ramesh Karri. 2023c.  Autochip: Automating hdl generation using llm feedback.  _arXiv preprint arXiv:2311.04887_ (2023).
  * Tsai et al. (2023) YunDa Tsai, Mingjie Liu, and Haoxing Ren. 2023.  Rtlfixer: Automatically fixing rtl syntax errors with large language models.  _arXiv preprint arXiv:2311.16543_ (2023).
  * Williams and Baxter (2002) Stephen Williams and Michael Baxter. 2002.  Icarus verilog: open-source verilog more than a year later.  _Linux Journal_ 2002, 99 (2002), 3.
  * Wu et al. (2024) Haoyuan Wu, Zhuolun He, Xinyun Zhang, Xufeng Yao, Su Zheng, Haisheng Zheng, and Bei Yu. 2024.  Chateda: A large language model powered autonomous agent for eda.  _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ (2024).
  * Xu et al. (2024) Xiaohan Xu, Ming Li, Chongyang Tao, Tao Shen, Reynold Cheng, Jinyang Li, Can Xu, Dacheng Tao, and Tianyi Zhou. 2024.  A survey on knowledge distillation of large language models.  _arXiv preprint arXiv:2402.13116_ (2024).
  * Yang et al. (2024) John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. 2024.  SWE-agent: Agent Computer Interfaces Enable Software Engineering Language Models.