#  Parendi: Thousand-Way Parallel RTL Simulation

Mahyar Emami<EMAIL_ADDRESS>EPFLLausanneSwitzerland ,  Thomas Bourgeat
<EMAIL_ADDRESS>EPFLLausanneSwitzerland and  James R. Larus
<EMAIL_ADDRESS>EPFLLausanneSwitzerland

###### Abstract.

Hardware development relies on simulations, particularly cycle-accurate RTL
(Register Transfer Level) simulations, which consume significant time. As
single-processor performance grows only slowly, conventional, single-threaded
RTL simulation is becoming less practical for increasingly complex chips and
systems. A solution is parallel RTL simulation, where ideally, simulators
could run on thousands of parallel cores. However, existing simulators can
only exploit tens of cores.

This paper studies the challenges inherent in running parallel RTL simulation
on a _multi-thousand-core_ machine (the Graphcore IPU, a 1472-core machine).
Simulation performance requires balancing three factors: synchronization,
communication, and computation. We experimentally evaluate each metric and
analyze how it affects parallel simulation speed, drawing on contrasts between
the large-scale IPU and smaller but faster x86 systems.

Using this analysis, we build Parendi, an RTL simulator for the IPU. It
distributes RTL simulation across 5888 cores on 4 IPU sockets. Parendi runs
large RTL designs up to 4$\times$ faster than a powerful, state-of-the-art x86
multicore system.

##  1\. Introduction

Hardware developers spend as much as 24% of development time _running_
simulations (SiemensVerifStudyPart8Asic, ; SiemensVerifStudyPart4Fpga, ).
Prominent is cycle-accurate RTL simulation, vital to debug and validate a
design. Unfortunately, its slow speed hampers the design process.

Fig. 1 shows the increasing gap between single-thread performance improvements
and transistor count in a package. An obvious consequence is that single-
thread RTL simulation of the next generations of chips on existing machines is
becoming less practical.

Figure 1. Chip growth and single-thread performance (processor_scaling_data,
). The dashed line predicts the number of cores necessary to simulate a new
generation at the same speed as in 2006.

One promising solution is to exploit the inherent parallelism of RTL designs
by running parallel RTL simulations on parallel computers (manticore_arxiv, ;
repcut, ; rtlflow, ). However, Fig. 1 shows that simulating today’s chips at
the same rate that we simulated chips in 2006111The release of Intel Core 2
Duo and the unofficial end of Dennard’s scaling. would require parallelizing
simulation across hundreds or thousands of cores.

This paper investigates the problem of parallelizing the RTL simulation of
large, 100-core SoCs across a few _thousand_ cores. We use an actual
_thousand-core_ machine since simulating an architecture with thousands of
cores would be too slow for a system-level study. The machine we use is the
Graphcore IPU (graphcore, ; DBLP:journals/corr/abs-1912-03413, ), a 1472-core
chip designed for machine learning applications. However, the IPU’s high core
count and fast synchronization and communication make it well-suited for
large-scale RTL simulation.

Developing an effective compilation strategy for parallel RTL simulation on a
thousand-core machine requires a keen understanding of the interplay between
three factors: synchronization, communication, and computation. We analyze the
three factors in depth to identify relations between them. These axes are not
independent, so a significant challenge is to use this understanding to craft
a practical partitioning algorithm that minimizes those costs.

We use these insights to build Parendi 222Parendi is the female Zoroastrian
angel of abundance.: the first multi-thousand-way parallel RTL simulator that
scales simulation speed across 4 IPU chips for a total of 5888 cores. Parendi
is open-source and useable with public cloud IPUs, facilitating further
research. For large designs, RTL simulation is up to 4$\times$ faster than
Verilator (the fastest multi-thread RTL simulator) running on large x86
systems.

The contributions of this work are:

  * •

A quantitative study of thousand-way parallel RTL simulation.

  * •

A new communication- and duplication-aware partitioning strategy for large-
scale RTL simulation.

  * •

Parendi, the first open-source333https://github.com/epfl-vlsc/parendi RTL
simulator to parallelize RTL code to thousands of cores.

  * •

An evaluation of Parendi on the IPU compared to Verilator on x86 showing a
$2.8\times$ geomean ($4.0\times$ max) speedup.

The paper is organized as follows: §2 describes the IPU architecture and its
programming model. §3 discusses our parallel simulation methodology. §4 is a
high-level study of parallel RTL simulation performance. §5 outlines Parendi’s
design. §6 evaluates and compares Parendi on the IPU and Verilator on x86. §7
surveys related work. And, §8 concludes.

##  2\. Graphcore IPU Background

A Graphcore IPU consists of 1472 tiles (physical cores) interconnected by a
high-bandwidth network on the chip (the IPU exchange) with an 11 TiB/s all-to-
all bandwidth (graphcore, ). The IPU is a multiple-instruction, multi-data
(MIMD) architecture where each tile runs an independent instruction stream.
This sets the IPU apart from SIMD or SIMT GPUs, where groups of threads
(warps) execute the same instruction on different data. Furthermore, the IPU
is a message-passing machine, where tiles can only access their own private
memories and must explicitly communicate through the exchange fabric. The
total on-chip memory in an IPU is about 900 MiB, with each tile having direct
access to 624 KiB.

Figure 2. The IPU processor and M2000 server blade.

Fig. 2 shows a Graphcore M2000 IPU server. The M2000 is a 1U server with 4
IPUs, connected via a dedicated 320 GiB/s on-board exchange fabric, totaling
5888 independent tiles. Multiple M2000 boards may also be connected to form
systems of 16 or 64 IPUs with dedicated networking. This work uses a single
M2000 board as the compilation target.

An IPU system (with any number of IPUs) is programmed using the bulk-
synchronous parallel (BSP) (valiant_bridging_1990, ) programming model in C++,
with the _poplar_ SDK and _popc_ , a clang-based C++ compiler. The next
section describes BSP and shows how to apply this execution model to RTL
simulation. The IPU _natively_ supports BSP communication and synchronization,
which enables efficient fine-grained parallelism.

##  3\. Parallel RTL Simulation

Hardware description languages (HDL) like SystemVerilog can express digital
sequential circuits. An HDL program describes stateful _clocked_ elements
called registers that are interconnected (by wires) via stateless
combinational logic (e.g., addition). An HDL register is associated with a
clock, and its value is updated at clock ticks. The collection of clocked
register updates and combinational logic defines the abstraction of the
register transfer level (RTL).

We focus on cycle-accurate RTL simulation, in which combinational logic has
zero delay. Furthermore, we perform a full-cycle simulation (activity-
oblivious) to evaluate the entire circuit at each RTL cycle. The alternative
method is to simulate the circuit in an event-driven fashion (activity-aware).
However, full-cycle simulators perform better—sometimes by orders of
magnitude—as the cost of tracking value changes in RTL is high
(DBLP:conf/dac/BeamerD20, ).

Parallel RTL simulation is better suited to _message-passing_ computation (and
architectures) than shared-memory. First, the fine-grained parallelism is
challenging to execute on a shared-memory multiprocessor due to costly
synchronization (manticore_arxiv, ; repcut, ). Second, the RTL tasks have a
high level of fine-grained point-to-point communication: Independent tasks
communicate only a few bytes of data to neighbors in the RTL design, which
must all go through the LLC. Last, the RTL can have a high reuse distance in
data and instructions, which makes caches perform poorly. Most data and
instructions are accessed once per simulated RTL cycle, which could span
millions of machine cycles. Large designs, which do not fit in the caches of a
shared-memory machine, must prefetch code and data at every RTL cycle.

###  3.1. BSP RTL Simulation

We apply Valiant’s bulk-synchronous parallel (BSP) (valiant_bridging_1990, )
model to coordinate parallel RTL simulation. The appeal of BSP is that it
vastly reduces synchronization overhead: to two global barriers per RTL clock
cycle.

BSP is a message-passing model that consists of two phases: (i) computation
and (ii) communication. In the computation phase, parallel _processes_ run,
reading shared variables but only modifying private data. Computation ends
with a barrier, followed by the communication phase, in which the new private
values propagate to other processes. The communication phase also ends with a
barrier, after which a new computation phase starts.

Fig. 3 shows an example _data dependence graph_ of an RTL circuit. In the
graph, each RTL register is split into two values: a read-only value
(_current_ , at the beginning of the clock) and a write-only value (_next_ ,
at the end). The read-only values (current) are at the top (e.g., read1) and
feed into stateless combinational logic (circles) that computes next register
values (e.g., write1).

The dashed lines in Fig. 3 _partition_ the data dependence graph into BSP
_processes_ p1 and p2. Each process reads a set of RTL registers and computes
new values for one or more (e.g., write1 in p2). Processes may have
overlapping computations; for instance, a3 is present in both p1 and p2. These
internal nodes are wire computations, which can be safely duplicated across
processes. As a result, we only communicate register values in bulk at the end
of the computations.

The right side of Fig. 3 shows the evaluation of the example. The two parallel
processes synchronize at a barrier (dashed vertical lines). The first barrier
marks the end of the computation, after which we exchange next values (write1,
write2, and write3) and update the current values accordingly (read1, read2,
and read3). We finish the communication with a barrier and conclude one RTL
simulation cycle.

A parallel RTL compiler needs to find a partition of the RTL data dependence
graph that minimizes time spent running each simulation cycle. p1 and p2 are
not the only possible partitions of Fig. 3. We call the atoms of BSP
simulation _fibers_. A fiber is the smallest BSP process that uniquely
computes the new value of a _single_ register. For instance, Fig. 3 consist of
three registers; hence there exists three fibers f1, f2, and f3, which can be
partitioned by assigning f1 to p1 and the other two fibers to p2.

Lastly, BSP is one of many possible parallel simulation methods. We could use
finer partitions that slice fibers in the data dependence graph to expose more
parallelism to avoid duplicating work. Verilator (snyder_mt_2018, ;
snyder_mt_2019, ; snyder_verilator_2020, ), the fastest RTL simulator and our
baseline, takes the latter approach (Sarkar86, ).

Figure 3. BSP Simulation of an RTL data dependence graph. The graph has three
fibers (f1, f2, f3), partitioned into two processes (p1, p2), running on two
threads (node a3 is duplicated on each). The run on the right shows the
computation and communication phases, separated by barriers.

##  4\. Analysis of BSP RTL Simulation

This section identifies and measures the principal factors affecting BSP RTL
simulation performance. This analysis uses both carefully crafted
microbenchmarks and illustrative integration benchmarks. We run the
experiments on a Graphcore M2000 quad-IPU system and an Intel Xeon Gold 6348
56-core dual-socket processor (see Table 2 for details).

Parallel RTL simulation performance depends on the synchronization,
communication, and computation cost. In the BSP model, the sum of the three is
the time to simulate one cycle, so the simulation rate (in thousand RTL cycles
per second or kHz) is

(1) $r_{cycle}=\frac{1}{t_{sync}+t_{comm}+t_{comp}},$

where $t_{sync}$, $t_{comm}$, and $t_{comp}$ are the per simulated RTL cycle
synchronization, communication, and computation latencies. We must reduce
their sum to increase the simulation rate.

Below, we explore how each component behaves as we increase parallelism
through measurements. Our analysis reveals salient architectural features of
both the IPU and x86, providing insight into compilation strategies.

###  4.1. Synchronization Latency

BSP requires two global synchronizations per simulated clock cycle, so
$t_{sync}$ costs two barriers. Therefore, $t_{sync}$ is independent of the
simulated design but depends on the number of hardware threads used for
simulation.

To show how much an increase in $t_{sync}$ affects simulation performance, we
simulate a set of xorshift32 pseudo-random number generators (PRNG), each
performing 3 XORs and 3 shifts (xorshift, ). The simulated PRNGs are
independent; hence, $t_{comm}=0$, but $t_{sync}>0$ as we still need to
synchronize all PRNGs with the RTL clock.

We simultaneously increase the number of PRNGs and the number of computation
units—tiles (IPU) or threads (x86). In doing so, we keep the amount of work
per tile (thread) constant. Therefore, if $t_{sync}$ is small compared to the
rest, we should observe a near-constant simulation rate. Note that each PRNG
consists of one fiber, and we sequentially execute multiple fibers on one tile
(thread).

We use the IPU’s built-in barrier and increase the tiles from 64 to 5888 by
64. On x86, we use a user-space barrier (atomic fetch-and-add), sweeping the
threads from 1 to 56.

Fig. 4 shows the measured rate on the IPU (normalized to the rate with 64
tiles) and on x86 (normalized to the rate with 1 thread). Each line shows the
performance with a fixed quantity of fibers per tile: 7, 56, and 448 on the
IPU and 736, 5888, and 47104 on the x86. The total work with 5888 tiles on IPU
is the same as 56 threads on x86. However, we normalized each experiment to
itself; as a result, we are not comparing absolute simulation rates between
the machines.

Figure 4. IPU and x86 PRNG rates

With 7 fibers per tile on the IPU, performance decreases by almost $50\%$ due
to increased synchronization overhead among an increasing number of tiles.
Synchronization latency becomes less noticeable as the work per tile increases
(e.g., 448 fibers). This agrees with our definition of $r_{cycle}$: the larger
$t_{comp}$ is, the less sensitive $r_{cycle}$ is to changes in $t_{sync}$. The
cost of synchronization on x86 is high, particularly with few fibers per
thread. With 736 fibers per thread, performance drops by more than $75\%$.

The IPU has native hardware support for global synchronization, taking only a
few hundred IPU cycles. By contrast, x86 barrier synchronization requires
expensive atomic memory accesses that could consume a few thousand cycles with
all 56 threads.

Fig. 4 implies a simple rule-of-thumb. To mask synchronization overhead on
x86, we need hundreds of thousands of instructions per thread (each fiber is
roughly 6 instructions), whereas on the IPU, a few thousand instructions
adequately hide synchronization overhead. The IPU potentially allows fine-
grain parallelism, whereas the x86 does not.

###  4.2. Communication Latency

Similar to $t_{sync}$, we expect $t_{comm}$ to increase as we increase
parallelism, as additional tiles (threads) mean more inter-tile (thread)
messages. Unlike synchronization, communication depends on the specifics of an
RTL design and the partitioning of fibers to tiles (threads). We summarize
these considerations into two parameters: bytes sent from each tile ($b$) and
number of tiles in the simulation ($m$).

To first order, we could expect $t_{comm}=\frac{m\times b}{bw}$, where $bw$ is
the bandwidth of communication and $m\times b$ corresponds to the total
communication volume. Additional parallelism can increase communication
latency as it also increases communication volume. Therefore, at some point,
the increases in $t_{comm}$ negate the gains from parallelism.

Alternatively, it could be that $t_{comm}$ is almost independent of $m$ and
only depends on $b$. Then, performance increases _monotonically_ with
parallelism, without increasing $t_{comm}$.

We found that communication within a single IPU mostly depends on $b$, and
communication between IPUs depends on $m\times b$. We demonstrate this using
two experiments.

First, we consider $2m$ tiles on one IPU. We _randomly_ partition the $2m$
tiles into two sets of $m$ tiles and send a fixed number of bytes in both
directions.

The left plot in Fig. 5 shows the measured IPU cycle counts (averaged over 10
random bi-partitions). Note that the cycle counts include $t_{sync}$ as
determining when all messages have been exchanged requires synchronization.
The horizontal axis is the number of bytes each tile sends and receives ($b$).
The vertical axis is the number of tiles ($m$). On-chip $t_{comm}$ increases
in only the direction of increasing $b$ as shown by the arrow in the left of
Fig. 5.

Figure 5. Measured communication cycles on the IPU, on- and off-chip.

In the second experiment, one tile in each pair resides on one IPU and the
other on another IPU, so all traffic goes off-chip. The right of Fig. 5
reports the results.

The off-chip exchange shows a vastly different behavior: $t_{comm}$ increases
with increasing parallelism _and_ bytes per tile as it depends on $m\times b$
(the diagonal arrow delineates the direction of change). Furthermore, the
increase is more pronounced compared to the on-chip latency.

At the plots’ darkest points, we consume $13\%$ and $82\%$ of the maximum
measured communication bandwidth, on- and off-chip, respectively (7.7 TiB/s
and 107 GiB/s). The on-chip experiment is far from saturating the bandwidth,
so the latency is insensitive to the tile count. By contrast, the off-chip
experiment runs near the fabrics’s maximum bandwidth, so additional tiles
increase contention and communication latency.

In conclusion, the difference between the communication fabrics implies that
minimizing off-chip communication volume should be a first-class concern.

Before analyzing $t_{comp}$, we discuss a subtle technicality of BSP
communication. A common programming practice to reduce $t_{comm}$ is to
overlap it with computation (i.e., software pipelining), which is particularly
valuable when a tile (thread) executes multiple fibers so final values are
produced throughout the computation. Cycle-accurate semantics of RTL
simulation requires double buffering. Overlap has some value on x86, though it
is difficult to implement on the IPU. The IPU’s exchange fabric is _statically
scheduled_ ; hence, the communication phase must begin with a synchronization
to ensure all tiles are at the same point in execution. As a result, we
ignored the potential effects of overlap.

###  4.3. Computation Latency

At first glance, optimizing $t_{comp}$ is similar to the classical
_multiprocessor independent task scheduling_ or _makespan minimization_
problem (DBLP:journals/ior/GareyGJ78, ). In this problem we consider a set of
tasks (fibers) $F=\\{f_{1},...,f_{n}\\}$ with corresponding execution times
$t_{i},...,t_{n}$. The goal is to schedule these tasks on $m$ tiles (threads)
to minimize the maximum execution time across all tiles. This problem is NP-
hard (mpsched_np, ), but polynomial-time approximation schemes exist
(DBLP:journals/jacm/Sahni76, ; DBLP:journals/ior/GareyGJ78, ).

In the classical problem, tasks (fibers) contain no duplicated work. However,
in BSP RTL simulation, two fibers might compute a common intermediate value
(for example the value produced by a3 in Fig. 3). This is an opportunity to
collocate the two fibers in the same process, however it complicates the
partitioning problem.

Each fiber is a set of computation nodes (the nodes in Fig. 3). If we denote
the execution time of a BSP process using $\tau(.)$, then a process made up of
fibers $f_{i}$ and $f_{j}$ would have $\tau(f_{i}\cup
f_{j})=t_{i}+t_{j}-\tau(f_{i}\cap f_{j})$ since we only need to execute the
shared code once in a process. Moreover, merging fibers reduces communication
($t_{comm}$) so $\tau(f_{i}\cup f_{j})\leq t_{i}+t_{j}-\tau(f_{i}\cap f_{j})$.

The scheduled execution time is called a submodular function, and this variant
of the scheduling problem is called submodular load balancing (SLB). SLB is
inherently more difficult than classical scheduling in the sense that it is
hard to get even modest approximation guarantees ($\sqrt{{n}/{log(n)}}$ (SLB,
)).

(a) Stragglers impose a lower bound on $t_{comp}$. Fibers sorted for
visualization.

(b) Fiber computation cycles in pico, bitcon, and rocket.

(c) Reducing $t_{comp}$ through parallel execution (base-2 log scale). Dashed
lines show a perfect scaling.

Figure 6. Straggler fibers and performance scaling regions.

Parendi |  Verilator on ix3
---|---
pico | bitcoin | rocket | pico | bitcoin | rocket
par. | kHz | par. | kHz | par. | kHz | par. | kHz | par. | kHz | par. | kHz
1 | 168.7 | 1 | 14.5 | 2 | 17.7 | 1 | 14141.7 | 1 | 537.4 | 1 | 220.3
111 | 629.4 | 270 | 935.2 | 1211 | 93.3 | 2 | 490.4 | 2 | 232 | 2 | 99.2

Table 1. Simulation rate in kHz. par is the tile- or thread-count used to
achieve the rate in kHz.

When fewer tasks exist than tiles (threads) ($n\leq m$), the optimal solution
is to assign one fiber to each tile (thread). It is impossible to improve
$t_{comp}$ beyond $\max_{i}t_{i}$ as the slowest fiber (the straggler) bounds
$t_{comp}$ from below. Encountering this bound on x86 hardware is unlikely:
even relatively small designs have a few hundred fibers, an order of magnitude
more than available cores. However, a single IPU chip has 1472 tiles,
sufficient for small to medium-sized designs so that a straggler fiber can be
the limit on the IPU.

Fig. 6a summarizes the SLB problem: mapping fibers to tiles results in a
linear regions were we could reduce $t_{comp}$ almost linearly by using more
tiles. The reductions gradually become less significant, and we plateau at
$\max_{i}t_{i}$ with sufficient tiles. In Fig. 6a $m_{crit}$ represents the
minimum number of tiles needed to reach $\max_{i}t_{i}$ (lowest value of
$t_{comp}$). To maximize simulation rate, we only need $m_{crit}$ tiles;
having more would not help as the straggler is the limit.

To put this into perspective, consider 3 small RTL designs: (i) a multi-cycle
RISC-V pico core (pico, ), (ii) a bitcoin miner (bitcoin, ), and (iii) a
_small_ rocket pipelined RISC-V core (rocket_chip, ). These small designs
contain more fibers than x86 systems core: 111, 270, and 1211 fibers,
respectively. We run each benchmark using Parendi, which we describe later in
§5.

Fig. 6b shows fiber computation latency ($t_{i}$ for each $f_{i}$) of the
three benchmarks (in IPU machine cycles). Fig. 6c illustrates the
corresponding scheduled execution times with the dashed diagonal representing
a perfect linear reduction. We normalize machine cycle counts to the minimum
parallel execution: 1 tile in pico and bitcoin, 2 tiles in rocket (a single
tile cannot hold sufficient code and state for rocket).

First, $t_{comp}$ in Fig. 6c follows the trend of Fig. 6a: imbalanced fibers
yield a small linear scaling region. pico is the most imbalanced and settles
to a final $t_{comp}$ extremely quickly. rocket is slightly more scalable but
also leaves the linear region early. bitcoin performs the best as its fibers
are quite balanced. Second, we see that $t_{sync}$ and $t_{comm}$ increase as
we use additional tiles. However, the $t_{comp}$ reduction is always larger,
and the sum of the three decreases overall.

Table 1 compares the wall-clock simulation rate of the IPU, using Parendi,
against an Intel Xeon 6348 (see Table 2 for details), using Verilator. We show
the simulation rate using a single tile (except for rocket which needs more
memory than available in one tile) and the maximum number of tiles, where we
assign one fiber per tile. For x86, we report the single-thread and best
multi-thread performance.

None of the three small benchmarks show any speedup on x86 from parallelism
since the synchronization cost is too high (see §4.1). The results in Table 1
do not mean that Verilator cannot speed up RTL simulation. In §6, we show that
Verilator does an excellent job of parallelizing code. However, our analysis
of $t_{sync}$ shows that a simulated design on the x86 must be large enough to
mask synchronization overhead, and these three benchmarks are small.

Verilator’s inability to scale these three benchmarks confirms our claim that
the straggler fiber is not a performance limit on x86 as synchronization
latency limits performance earlier. On the other hand, stragglers are a
fundamental concern for Parendi when compiling small designs. Table 1 shows
that Parendi does not manage to match Verilator’s single-thread performance
for pico and rocket, despite modest gains from parallelism. However, it does
outperform Verilator for bitcoin, where the balanced fibers scale well.

Last, single-tile execution of pico and bitcoin on the IPU are approximately
$84\times$ and $37\times$ slower than x86, respectively. Consequently, the IPU
has to _massively_ scale RTL simulation even to match Verilator’s single-
thread performance.

##  5\. Parendi Compiler

Parendi is a Verilog compiler for the IPU systems. It is based on Verilator to
take advantage of its optimizations and the maturity of its codebase. However,
Parendi contains significant changes that target the IPU (message-passing)
rather than the x86 (shared memory). Parendi also includes new scheduling and
partitioning strategies and IPU-specific optimizations—about 8K lines of code.

Parendi generates a C++ BSP program that uses Graphcore’s _poplar_ programming
framework. The code defines each tile’s computation and how the tiles
communicate.

At a high level, Parendi’s primary responsibility is to partition RTL across
the tiles of an IPU system. The user selects the number of tiles; for example,
a user could ask Parendi to produce code for 736 tiles (half an IPU) or 2944
tiles (2 IPUs). Parendi tries to maximize the simulation rate by finding an
appropriate partitioning of fibers to the tiles. We briefly describe our
partitioning strategy.

###  5.1. Partitioning

After generating a data dependence graph (see Fig. 3), we determine the fibers
by collecting the nodes that transitively feed into each sink node by crawling
the graph in reverse.

Once we form the set of fibers, we must solve the SLB problem introduced in
§4.3. It is crucial to recognize the interdependence between computation
($t_{comp}$) and communication latency ($t_{comm}$). Additionally, each tile
has a finite memory. Consequently, our partitioning algorithm must consider
duplication, communication, and memory limitations.

We solve this problem in multiple steps, each pursuing a different goal. At
each algorithm step, we _merge_ fibers into _processes_. On the IPU, a process
is a collection of fibers that will eventually run on a tile (e.g., see the
processes in Fig. 3). There are four stages in our algorithm:

  1. (1)

Reduce data memory footprint.

  2. (2)

Minimize off-chip communication.

  3. (3)

Reduce $t_{comm}$ while keeping $t_{comp}$ unchanged in each IPU.

  4. (4)

Match the number of fibers to available hardware parallelism.

In the first stage, we merge fibers referencing the same RTL array. We do this
to save memory, at the cost of increasing $t_{comp}$, but only for _very_
large arrays (e.g., $\geq$ 128 KiB, tunable) so as to not limit parallelism.

The second stage minimizes off-chip exchanges if Parendi is compiling for more
than one IPU. We do this by partitioning a hypergraph of fibers. In this
hypergraph, hypernodes represent fibers, and hyperedges represent RTL
registers. If two fibers access the same register (read or write), their
corresponding hypernodes share a hyperedge. The hyperedge weights are the
number of words in an RTL register, and hypernodes are unweighted.

For $k$ IPUs, we use the KaHyPar (DBLP:journals/jea/SchlagHGASS22, ) library
to find a $k$-way balanced partition of the hypergraph that minimizes the
_cut_ (hyperedges crossing the partitions). KaHyPar produces $k$ roughly
equally-sized sets of fibers. However, each set may contain more than 1472
fibers, in which case, we must further merge fibers to fit the available
tiles.

In the third stage, we _conservatively_ merge _tiny_ fibers to reduce
communication within each target IPU without increasing computation latency.
Intuitively, we crawl right to the left in the right subplot of Fig. 6a
towards $m_{crit}$. If we reduce the number of fibers to fit the tiles in an
IPU without crossing $m_{crit}$, then we can use the optimal
$t_{comp}=\max_{i}t_{i}$ and a pseudo-optimal $t_{comm}$.

We create one process for each fiber and estimate its execution cost. Recall
from §4.3 that the cost of a process is submodular with respect to its fibers.
We use a dense bitset data structure to represent duplication across fibers
and efficiently compute intersection and union in the submodular cost
function—$\tau(f_{i}\cup f_{j})=t_{i}+t_{j}-\tau(f_{i}\cap f_{j})$.

In addition, we use another bitset to calculate the memory usage after
merging, accounting for deduplication.

In each iteration, we select the process with the shortest execution time and
try to merge it with a process with which it communicates, so long as their
merged time does not surpass the worst existing execution time (straggler).

If we cannot perform the merge because of overflowing memory or exceeding the
straggler execution time, we consider merging the two smallest processes. If
that fails, we skip the candidate process and move to the next one. We merge
processes until we process all of them or reach the desired tile count.

The final stage only runs if the third stage fails to reach the desired tile
count. We follow the same strategy as in the third stage but allow an increase
in the worst-case execution time. However, we still disallow out-of-memory
merges.

At the end of this stage, the number of processes must fit the available
hardware. Otherwise, the compilation fails because the design is too large to
fit the hardware resources.

###  5.2. IPU-Specific Optimizations

Parendi extends Verilator’s optimizations with a few IPU-specific ones. We
briefly describe the most important ones.

Differential exchange. RTL arrays are common structures in hardware designs,
e.g., in a register file or a cache bank. If a process reads an array, it
needs a full copy on its tile. We avoid sending whole arrays by using static
analysis to determine the number of updates to an array, though not their
location or condition (e.g., a 2-port SRAM with byte-strobes). With this
analysis, we only send the changes instead of an entire array.

Aggressive block splitting. We extend Verilator’s V3Split pass, favoring
parallelism over code bloat, to maximally split all clocked code blocks.

Aggressive inline. Parendi ensures that the simulation program on the IPU is
free of function calls. Inlining can increase code size and produce excessive
instruction cache pressure on x86, especially in RTL simulation, where nearly
every instruction executes only once per RTL cycle, except for code in
functions invoked multiple times. An IPU tile has no instruction cache but a
624 KiB local memory, of which 200 KiB holds executable code. So, a single IPU
chip has $\approx$300 MiB of on-chip instruction memory space, which allows
Parendi to aggressively inline code.

###  5.3. Limitations

Currently, Parendi has some limitations as compared to Verilator. For
instance, Parendi can simulate an RTL design with one top-level clock and an
arbitrary number of driven ones (e.g., gated or divided). In principle,
Parendi _could_ support multiple top-level clocks, but we leave this feature
for future work. Verilator, on the other hand, has recently added support for
timing-accurate simulation with arbitrary clocking.

Parendi supports only a Verilog test driver, whereas Verilator allows both C++
and Verilog drivers. We do this for pragmatic reasons: to achieve good
performance, host interactions must be infrequent, and a C++ testbench
requires interactions at every simulated RTL cycle.

Parendi’s compilation fails if there is a combinational loops in the
design444In some situations, designs can have false combinational loops
created by dependencies within wide signals.

##  6\. Evaluation

We use the following set of benchmarks to evaluate Parendi:

  * •

mc (DBLP:conf/fpt/TianB08, ) is stock option price predictor.

  * •

vta (DBLP:journals/micro/MoreauCVRYZFJCG19, ) is an ML accelerator. We
configure vta with BlockIn/Out=64, larger than the default FPGA configuration
to expose more parallelism.

  * •

srN is a $\text{N}\times\text{N}$ Constellation (constellation, ) mesh NoC
that consists of $\text{N}\times\text{N}-3$ small Rocket cores (rocket_chip, )
(64-bit, no FPU, no VM), generated using the Chipyard
(DBLP:journals/micro/AmidBGGKLMMOPRS20, ) SoC generator (3 nodes in the NoC
connect to uncore components). We change N from 2 to 15.

  * •

lrN is similar to srN, but we use large cores with an FPU and VM. We change N
from 2 to 10.

Note that srN is different from rocket in §4.3. The latter is a bus-based
design, whereas srN uses a NoC.

These benchmarks reflect today’s chip designs, including accelerators and
multicore systems. By making the mesh bigger in srN and lrN, we can see how
Parendi performs with larger chips.

We wrap all benchmarks with simple drivers in Verilog, free of unnecessary DPI
calls555Some PLI calls like $readmemh, $display, $value$plusargs, and etc.
still exist.. This is different from Chipyard’s default simulation flow which
heavily relies on DPI calls that hook the simulation to various services
provided by the RISC-V frontend server in software. The advantage is that
without linking heavy-weight software to simulation, we ensure that our
evaluation does not include unwanted performance degradation (both in Parendi
and Verilator).

Compiler | Name/Short | Cores | GHz | MiB | $\times$ | Date
---|---|---|---|---|---|---
Verilator | EPYC 9554 / ae4 | 64 | 3.75 | 2/128/256 | 2 | Q4 2022
Xeon 6348 / ix3 | 28 | 3.5 | 2.2/35/42 | 2 | Q2 2021
Parendi | M2000 / ipu | 1472 | 1.35 | 897 | 4 | Q3 2020
Ubuntu 20.04 popc 3.3 (clang 16.0.0) Verilator v5.006 g++ 10.5.0
Parendi: tiles up to 5888 Verilator: threads up to 32
Compile on Intel Xeon 6132 1.5 TiB Memory

min / max | Compile time | Memory usage
---|---|---
Parendi | 26s / 40m | 335 MiB / 55 GiB
Verilator | 3s / 8h | 223MiB / 1043 GiB

Table 2. Evaluation setup: Cores is the physical core count per socket in each
machine. MiB is cache capacity (L1/L2/L3) for x86 and the on-chip memory for
the IPU. $\times$ is the number of sockets. We use the Short names for
brevity. We also report the min. and max. compilation time and memory used in
compilation for each simulator.

Baseline. We use Verilator as our baseline. Recent open-source work that
attempts to improve RTL simulation performance through parallelism has limited
usability. Verilog is a complex language, so academic works, including ours,
make concessions and focus on techniques, ideas, and feasibility of approach
rather than full language coverage and usability. We discuss a few technical
limitations of earlier work to explain why we could not use them as a baseline
in §7.

Evaluation Setup. Table 2 summarizes the hardware setup in our evaluation. For
our baseline, Verilator, we use two modern data center processors: ae4 is the
latest generation of AMD server lineup that offers very large caches and a
high core count. The 64 cores on one socket are arranged in _chiplets_ (zen4,
), each containing 8 cores. ix3 is a recent Intel lineup with less cache and
fewer cores. Unlike the AMD machine, ix3 is not a chiplet-based design. For
Parendi, we use an M2000 server with 4 IPUs666 The M2000 is not the fastest
IPU machine available. BOW-2000 has the same architecture and tile count but
clocked at 1.85 GHz (a $37\%$ increase). M2000 and BOW-2000 are available in
the public cloud (gcore_ipu, ). .

We use Verilator v5.006 (Parendi is forked from the same version) with all
optimizations enabled (-O3).

To find the best simulation performance for each design on Parendi, we
consider only 1472, 2944, 4416, and 5888 tiles (1, 2, 3, and 4 IPUs,
respectively). On Verilator, we parallelize each design from 2 to 32 threads
with a step of 2. We compile all the designs on an Intel Xeon 6132 with 1.5
TiB of memory.

Compile time and memory usage. Table 2 also shows the minimum and maximum
compilation time and memory usage during compilation. Verilator compiles small
designs very quickly but struggles with large ones, taking close to 8 hours to
generate multi-threaded code. Parendi, is relatively slower for small designs
but faster for bigger ones, only taking 40 minutes in the worst case. Besides,
Verilator uses 1043 GiB of memory to compile sr15. Because of this, we use a
dedicated machine with 1.5 TiB of memory and only consider parallelizing up to
32 threads. Overall, we spent more than 2 weeks compiling code with Verilator.

We traced this issue to the V3VariableOrder, which approximates the traveling
salesman problem to optimize shared-variable layout across threads (not needed
in Parendi). This pass runs irrespective of the optimization level. By
manually disabling it we noticed an improvement in compile time and memory
usage, but about a $30\%$ performance decrease. So, we keep V3VariableOrder
enabled.

Figure 7. IPU’s speedup versus multi-thread Verilator.

###  6.1. Parendi Vs. Verilator

Fig. 7 reports Parendi’s speedup against Verilator (on ix3 and ae4). Overall,
Parendi outperforms Verilator: geometric mean speedups are $2.81$ and $2.75$
compared to ix3 and ae4. Table 3 shows detailed performance numbers on each
platform. We also report a few size metrics for each benchmark: number of data
dependence graph nodes, fibers, x86 instructions needed to simulate one RTL
cycle on a single thread, and Verilator’s code footprint. Furthermore, for
multi-IPU points, we report the size of the variables exchanged in KiB (actual
exchange volume is higher due to fanout).

###  6.2. Verilator’s Performance

Table 3 shows Verilator’s best speedup relative to itself. Clearly, Verilator
speeds up RTL simulation quite well when the design is large, by more than
$20\times$. We expand on a few points from Table 3 to better understand
interesting trends.

High sync. cost. Fig. 8 shows that smaller designs only see limited speedups.
From §4.1, we expected this behavior: synchronization cost cancels the gains
from parallelism.

Figure 8. Verilator’s speedup diminish quickly for smaller designs as
synchronization is costly.

Comm. cost is non-uniform. From §4.1 and Fig. 4, we know that large designs
are not limited by synchronization cost and Fig. 9 shows Verilator’s
significant speedup for the largest designs. However, on ae4, speedups stagger
after 8 threads (chiplet boundary), whereas on ix3, we see a significant drop
after 28 threads (socket boundary). Therefore, the non-uniform communication
latency within and across these boundaries has a noticeable performance impact
and parallel simulators should be aware of them.

Figure 9.  Non-uniform communication (crossing chiplets or sockets) affects
speedups in larger designs.

Architecture matters. Fig. 10 shows no clear winner between the two machines.
In general, ae4 wins for the smaller designs, whereas ix3 performs better for
huge ones up to 28 threads. But in some cases, ae4 shows superlinear gains up
to the chiplet boundary. Such gains are exciting but not uncommon in RTL
simulation: using more cores relieves the private cache and boosts performance
(repcut, ; manticore_arxiv, ; DBLP:conf/dac/BeamerD20, ;
DBLP:journals/micro/Beamer20, ; DBLP:conf/date/LopezParadisLAWMB23, ). By
using more cores, there is more _aggregate_ private cache capacity (L1 and
L2). Therefore, we can essentially fully contain the working set within the
private caches and increase simulation speed superlinearly. However,
ultimately, superlinear gains disappear as we exhaust the cache capacity.

Figure 10. ae4 and ix3 have different scaling profiles: superlinear on one and
sublinear on another.

Overall, our analysis in §4 assumed a BSP execution model. Yet, we can still
qualitatively apply it to Verilator (non-BSP), as we showed when $t_{sync}$
and $t_{comm}$ surface as bottlenecks.

###  6.3. Parendi’s Performance

RTL simulation on IPU follows different trends than x86. We expand on a few
key differences. On x86, we identified two points of performance collapse:
synchronization and communication. We show that on IPU, only off-chip
communication becomes a bottleneck, which is in principle similar to off-
chiplet or -socket communication.

Single-IPU scaling. On x86, we cannot use _all_ the cores to increase
simulation speed reliably: synchronization or communication cost may undo all
the gains from parallelism. On the other hand, simulation performance on a
single IPU monotonically increases as we use more tiles. Fig. 11a shows the
simulation rate for three select designs as a function of the fraction of the
IPU used.

Unlike a general-purpose machine, we cannot compile a moderately large design
to a single tile on the IPU (not enough memory). Consequently, we cannot see
speedup numbers relative to a single tile, unlike Verilator. Therefore, we
consider using 184 tiles ($\frac{1}{8}$ of an IPU), 368, …, and 1472 tiles
only. Note that this is a fundamentally different starting point than a single
thread on x86, as at this point, most significant gains from parallelism have
already diminished; however, we still see improvements.

Fig. 11b shows the breakdown of simulation time for each of the designs in
Fig. 11a. The vertical axis is normalized to simulation time using
$\frac{1}{8}$ of the IPU in each benchmark. Communication ($t_{comm}$) and
synchronization ($t_{sync}$) remain roughly constant while computation time
($t_{comp}$) decreases with more tiles. However, in sr3, the reduction in
$t_{comp}$ slows to a halt which happens due to fiber imbalance (see §4.3).

Performance monotonically increases with more parallelism on the IPU due to
low-cost communication and synchronization, which is not the case on x86. On
x86, a hardware developer must tune the simulator’s parallelism for each
specific design _and_ the machine they use (e.g., ix3 or ae4). However, a
developer can confidently use all 1472 tiles on the IPU and operate at peak
speed.

(a) Simulation speed scales monotonically on one IPU. We start from 184 tiles
($1/8$ of an IPU) and scale performance to a full IPU (1472 tiles).

(b) Breakdown of simulation time.

Figure 11. Speedup of simulation within one IPU and the breakdown of
simulation time.

Multi-IPU scaling. Within one IPU, communication is relatively cheap, which
enables good scaling. However, crossing IPU boundaries is expensive, as
mentioned in §4.2. Therefore, preserving performance monotonicity across IPUs
is challenging: fundamentally, crossing IPUs is the same as crossing chiplets
or sockets since off-chip communication latency increases abruptly (see Fig.
5). However, non-uniform communication surfaces much later on the IPU: after
1472 tiles rather than 8 or 28 threads. Fig. 12 shows how the simulation speed
scales across multiple IPUs. Even for very large designs, maximum parallelism
may not yield the best result, and using fewer IPUs has marginal gains in
specific cases.

The speedups are also much smaller off-chip: going from 1472 tiles to 5888
tiles in lr9 improves performance by $60\%$, despite leveraging $4\times$ more
tiles. However, even a $60\%$ gain is quite interesting: on x86 it is
difficult to scale beyond 28 threads (ix3), but on the IPU, we keep increasing
performance all the way to 5888 tiles ($210\times$ more parallel).

Figure 12. Performance scaling across multiple IPUs.

Performance resilience. We showed that Parendi can scale the simulation rate
of a single design within and across IPUs (strong scaling). However, can we
maintain a constant simulation rate as we scale the design size (weak
scaling)?

Fig. 13 shows the maximum simulation rate of Parendi and Verilator as a
function the mesh size in srN and lrN. Neither Parendi nor Verilator can
perfectly keep the simulation rate constant, but Parendi shows more
resilience. For instance, in the right of Fig. 13, there is a long segment in
which Parendi simulates larger and larger designs at the same rate.
Verilator’s rate slowly drops in the same region; therefore, the speedup
(Parendi vs. Verilator) increases.

Figure 13. Parendi vs. Verilator in coping with increasing design size. The
left axis shows the best simulation rate in base-2 logarithmic scale. Right
axis shows the gmean speedup of Parendi against Verilator (dashed lines).
Figure 14. Fiber imbalance allows us to keep simulation rate constant in face
of increasing design size.

While fiber imbalance severely limits the performance of a small or medium RTL
design, thereby limiting strong scaling, it actually _enables_ better weak
scaling. We use Fig. 14 to show how this happens. Consider an SoC with $N$
cores and a sizable imbalance between its fibers, as shown on the left. If we
double the size of the SoC, we double the number of its fibers. Because a
small portion of fibers have significantly more execution time, we can
tolerate increasingly larger designs, e.g., simulate an SoC with twice the
number of cores but keep the simulation rate constant. Fig. 14 shows how we
absorb the extra fibers (different colors) needed to simulate a larger design
into the existing tiles without increasing the execution time. However, the
tiles start to balance out at some point, and increasing the design size will
drop the simulation rate (e.g., tripling the number of SoC cores).

| ix3 | ae4 | Parendi | Speedup |  |  |
---|---|---|---|---|---|---|---
Bench | st-kHz | mt-kHz | #T | gain | st-kHz | mt-kHz | #T | gain | kHz | #T | ix3 | ae4 | gmean | MiB | #I (M) | #N (K) | #F (K) | Int. | Ext.
vta | 30.91 | 113.75 | 4 | 3.7 | 44.79 | 164.73 | 4 | 3.7 | 454.10 | 1472 | 3.99 | 2.76 | 3.32 | 1.5 | 0.17 | 23.5 | 6.0 | 28.7 | —
mc | 28.68 | 88.96 | 8 | 3.1 | 37.55 | 143.88 | 8 | 3.8 | 592.83 | 1472 | 6.66 | 4.12 | 5.24 | 1.0 | 0.15 | 26.9 | 7.5 | 24.2 | —
sr2 | 123.40 | 76.22 | 2 | 0.6 | 176.49 | 145.75 | 4 | 0.8 | 91.20 | 1472 | 0.74 | 0.52 | 0.62 | 1.2 | 0.06 | 12.7 | 2.8 | 12.8 | —
sr3 | 20.95 | 40.95 | 8 | 2.0 | 28.71 | 77.66 | 8 | 2.7 | 83.95 | 1472 | 2.05 | 1.08 | 1.49 | 3.1 | 0.17 | 36.3 | 8.1 | 33.9 | —
sr4 | 8.79 | 30.93 | 22 | 3.5 | 7.23 | 54.79 | 8 | 7.6 | 85.09 | 1472 | 2.75 | 1.55 | 2.07 | 5.5 | 0.32 | 68.2 | 15.3 | 63.5 | —
sr5 | 5.23 | 24.34 | 26 | 4.6 | 4.26 | 40.09 | 8 | 9.4 | 84.28 | 1472 | 3.46 | 2.10 | 2.70 | 8.6 | 0.50 | 107.9 | 24.2 | 101.5 | —
sr6 | 3.53 | 23.11 | 20 | 6.5 | 2.90 | 30.51 | 8 | 10.5 | 76.63 | 1472 | 3.32 | 2.51 | 2.89 | 12.2 | 0.72 | 156.0 | 35.0 | 145.4 | —
sr7 | 2.47 | 18.83 | 28 | 7.6 | 2.10 | 22.72 | 8 | 10.8 | 71.33 | 2944 | 3.79 | 3.14 | 3.45 | 16.6 | 0.99 | 212.6 | 47.7 | 199.2 | 0.9
sr8 | 1.82 | 17.94 | 26 | 9.9 | 1.58 | 13.66 | 8 | 8.7 | 57.39 | 2944 | 3.20 | 4.20 | 3.66 | 21.6 | 1.29 | 277.3 | 62.3 | 259.0 | 1.1
sr9 | 1.37 | 15.56 | 28 | 11.4 | 1.22 | 11.72 | 32 | 9.6 | 58.79 | 4416 | 3.78 | 5.02 | 4.35 | 27.4 | 1.65 | 351.4 | 78.8 | 328.8 | 1.7
sr10 | 1.06 | 15.03 | 24 | 14.1 | 0.97 | 10.83 | 32 | 11.1 | 52.77 | 2944 | 3.51 | 4.87 | 4.14 | 33.8 | 2.03 | 433.5 | 97.2 | 396.3 | 1.3
sr11 | 0.85 | 13.59 | 26 | 16.0 | 0.79 | 10.21 | 32 | 12.9 | 47.71 | 5888 | 3.51 | 4.67 | 4.05 | 40.9 | 2.47 | 524.6 | 117.5 | 488.0 | 3.3
sr12 | 0.70 | 12.98 | 28 | 18.5 | 0.65 | 8.79 | 32 | 13.5 | 43.30 | 5888 | 3.34 | 4.93 | 4.05 | 48.6 | 2.93 | 623.7 | 139.7 | 579.5 | 3.1
sr13 | 0.58 | 11.40 | 28 | 19.5 | 0.54 | 8.18 | 32 | 15.2 | 37.83 | 4416 | 3.32 | 4.62 | 3.92 | 56.9 | 3.44 | 731.1 | 163.9 | 665.7 | 2.3
sr14 | 0.50 | 10.37 | 28 | 20.8 | 0.44 | 7.09 | 32 | 16.1 | 34.98 | 5888 | 3.37 | 4.93 | 4.08 | 65.9 | 3.99 | 847.0 | 189.9 | 775.2 | 3.4
sr15 | 0.43 | 9.22 | 28 | 21.6 | 0.33 | 6.51 | 32 | 20.0 | 31.69 | 5888 | 3.44 | 4.86 | 4.09 | 75.6 | 4.58 | 972.2 | 217.9 | 886.9 | 3.7
lr2 | 69.07 | 70.69 | 2 | 1.0 | 123.55 | 132.09 | 8 | 1.1 | 64.58 | 1472 | 0.91 | 0.49 | 0.67 | 1.6 | 0.09 | 16.5 | 3.7 | 16.2 | —
lr3 | 8.74 | 33.89 | 12 | 3.9 | 7.79 | 60.93 | 8 | 7.8 | 58.73 | 1472 | 1.73 | 0.96 | 1.29 | 5.7 | 0.36 | 59.4 | 13.3 | 55.5 | —
lr4 | 4.13 | 25.27 | 22 | 6.1 | 3.61 | 38.97 | 8 | 10.8 | 50.93 | 5888 | 2.02 | 1.31 | 1.62 | 11.1 | 0.73 | 118.2 | 26.7 | 109.9 | 1.8
lr5 | 2.36 | 23.56 | 26 | 10.0 | 2.15 | 21.87 | 8 | 10.2 | 50.09 | 5888 | 2.13 | 2.29 | 2.21 | 17.8 | 1.20 | 192.4 | 43.4 | 178.4 | 2.0
lr6 | 1.50 | 17.86 | 28 | 11.9 | 1.43 | 13.15 | 30 | 9.2 | 39.84 | 1472 | 2.23 | 3.03 | 2.60 | 26.0 | 1.77 | 282.8 | 63.7 | 256.2 | —
lr7 | 1.03 | 14.73 | 28 | 14.3 | 1.01 | 10.41 | 30 | 10.3 | 39.00 | 2944 | 2.65 | 3.74 | 3.15 | 35.8 | 2.45 | 389.4 | 87.7 | 354.8 | 1.3
lr8 | 0.74 | 12.52 | 28 | 16.9 | 0.74 | 8.60 | 32 | 11.6 | 39.02 | 2944 | 3.12 | 4.54 | 3.76 | 47.0 | 3.24 | 511.8 | 115.4 | 463.9 | 1.0
lr9 | 0.58 | 10.63 | 26 | 18.5 | 0.56 | 7.57 | 32 | 13.4 | 38.22 | 4416 | 3.60 | 5.05 | 4.26 | 59.8 | 4.14 | 651.3 | 146.7 | 595.8 | 1.6
lr10 | 0.45 | 9.27 | 28 | 20.6 | 0.37 | 6.27 | 32 | 17.0 | 38.24 | 5888 | 4.13 | 6.10 | 5.02 | 74.0 | 5.12 | 806.4 | 181.7 | 734.9 | 3.2
|  |  |  |  |  |  |  |  |  | gmean | 2.81 | 2.75 | 2.78 |  |  |  |  |  |

Table 3. st-, mt-kHz are single- and multi-thread Verilator performance (blue
is best of ix3–ae4). kHz is best Parendi rate. gain is Verilator’s self-
relative speedup (underscored superlinear). #T is threads or tile count of
parallel rate. Speedup is Parendi vs. Verilator (green $\geq 2$ and red $<1$).
gmean is reported across machines and benchmarks. MiB is Verilator’s binary
size. #I is the million number of x86 instructions per RTL cycle (Verilator).
#N is thousands of data dependence graph nodes. #F is thousands fibers. Int.
and Ext. are KiBs on- and off-chip cut size (less than actual communication
volume due to fanout).

###  6.4. Partitioning Strategies

So far, we have used the partitioning strategy outlined in §5.1. This section
considers alternative strategies for partitioning fibers within and across
IPUs.

####  6.4.1. Single-IPU Partitioning

We presented a bottom-up strategy in §5.1 that conservatively merges smallest
fibers to make processes. Recent work proposes casting this problem as a
hypergraph partitioning problem where hypergraph nodes represent clusters of
computation and hyperedges represent duplicated clusters across fibers
(repcut, ). The goal of this proxy problem is to find a balanced partitioning
of clusters while minimizing duplication. We implemented this strategy in
Parendi as an alternative for users. Fig. 15 compares our default bottom-up
(B) strategy with hypergraph partitioning (H) on a single IPU (1472-way
partitioning). The vertical axis shows the normalized number of IPU machine
cycles per RTL cycle (i.e., reciprocal of rate; lower is better).
Interestingly, neither strategy is uniformly better. Bottom-up performs best
with srN, whereas hypergraph partitioning performs better with lrN in some
cases.

Figure 15. Comparison of our method with the hypergraph partitioning from
repcut (repcut, ). We compare our bottom-up (B) fiber merge strategy with the
hypergraph (H) used in repcut. The vertical axis shows IPU machine cycles per
RTL cycle, normalized to B (lower is better).

####  6.4.2. Multi-IPU Partitioning

Fig. 16 compares three strategies for multi-device partitioning over 4 IPUs:

  * •

Pre: partition fibers across IPU before they are merged into processes
(default Parendi strategy)

  * •

Post: partition processes across IPU, i.e., after fibers are merged into
processes

  * •

None: do not partition fibers or process, i.e., multi-IPU oblivious

Not partitioning fibers or processes across IPUs yields objectively inferior
performance as expected. However, partitioning fibers works better than
partitioning processes, too. This is because, during the merge, we may
suboptimally absorb some _good cuts_ and land in a region of the design space
that is only locally optimal. Conversely, partitioning fibers before the merge
helps us better optimize globally.

Figure 16. Normalized simulation rate for 4-IPU partitioning strategies.
Partitioning fibers pre merge performs better than partitioning processes post
merge. Being oblivious to the muli-IPU setup (none), yields vastly inferior
results.

###  6.5. Fastest Simulation is Cheapest

A detailed cost analysis of our approach is out of the scope of this work.
However, it is worth noting that at the time of writing, GCore offers IPU-POD4
classic instances (M2000, used in our evaluation) for $2.13 per hour
(gcore_ipu, ). On the other hand, a 32-core Dav4 instance from Microsoft Azure
(AMD EPYC 7452) costs $1.536 per hour (azure_instance_prices, ). More powerful
machines, like the HB-series instances that are still one generation older
than the ae4 cost as much as $2.8-$3.36 per hour.

The IPU’s performance advantage for large designs makes it a more cost-
effective solution than any existing cloud VM. Let us consider simulating lr10
for one billion cycles. This takes 7.26 hours on the IPU and 44.30 hours on
ae4. Clearly, IPU’s faster simulation rate offsets its hourly cost ($17 on the
IPU vs. $\geq$ $69 on cloud VMs). To be competitive with the IPU, a cloud VM
(with many cores) should cost about 6$\times$ less than the IPU to offset the
performance disadvantage. Our analysis is quite forgiving to simulation on
ae4, because it does not consider the cost of compiling, which is again much
lower on the IPU (see Table 2). However, far more critical than the hourly
cost is the time savings the IPU offers to verification engineers, whose
hourly wage is at least an order of magnitude more than machine costs.

###  6.6. Discussion

Fast simulation requires a simulator architecture and implementation that
accommodates the fine-grain parallelism of RTL simulation and exploits the
features of the underlying hardware platform. Verilator fails to scale because
its frequent fine-grain synchronization does not match the costly
synchronization and communication on the x86. Parendi scales far better,
albeit from much lower single-core performance, because an IPU efficiently
supports the BSP model and offers high-bandwidth communication. However,
Graphcore’s high off-chip communication latency creates a constraint that
requires effective partitioning of the RTL design to minimize cross-chip
communication.

When we started this project, we expected no speed gains from multiple IPUs,
planning to use more IPUs when we ran out of memory. We were surprised to see
speedups with even 5888 cores and believe that additional improvements in
partitioning can increase performance further.

We explain why speedups exist even with thousands of cores by drawing and
analogy between VLSI design’s clock rate and simulation rate. First,
optimizing circuit performance for synthesis, placement, and routing, requires
a good _floorplan_ that is aware of physical constraints such as pin
placement: Therefore, a good design facilitating floorplanning, facilitates
optimizing off-chip communication in its simulation—there is a natural minimal
cut. Second, the critical path length in VLSI caps the clock rate, as does the
straggler fiber and its cone of logic limit simulation rate. Again, a good
circuit design minimizes the critical path (length), indirectly minimizing the
critical cone of logic (area); as such, faster circuits should simulate faster
at scale.

Our work on thousand-core simulation is valuable for x86 simulation as well:
it would benefit from a chiplet- and socket-aware partitioning methodology to
better exploit the x86’s memory hierarchy. Furthermore, by knowing the coarse
cost of synchronization and communication, we may be able to achieve
performance monotonicity on an x86, as we did for an IPU.

##  7\. Related Work

Parendi is the first RTL simulator to parallelize code across a few thousand
cores. Prior work parallelizes RTL for tens of cores on CPUs and GPUs and a
few hundred cores on specialized architectures. Other work simulates thousand-
core SoCs through software or FPGA emulation.

Tens of cores. RepCut (repcut, ) is a BSP RTL simulator (full-cycle) for the
firrtl (DBLP:conf/iccad/IzraelevitzKLLW17, ) intermediate representation on
x86. It shows that BSP can produce superlinear speedups within and across
sockets (we discovered a similar effect with chiplets). They sped simulation
up to 32 cores (48-core machine) but showed no gains beyond that limit.

We did no use RepCut as a baseline for a few reasons. First, RepCut is a
firrtl compiler and cannot parse arbitrary Verilog, and to our knowledge,
there is no reliable way of converting Verilog to firrtl, so a design not
written in Chisel (DBLP:conf/dac/BachrachVRLWAWA12, ) cannot be compiled using
RepCut, but can be compiled with Verilator (and Parendi) as firrtl can be
lowered to Verilog. Second, a comparison between RepCut and Verilator (or
Parendi) is biased by the quality of lowering firrtl to Verilog, which can
affect performance (verilator_firrtl_performance, ). Third, RepCut uses the
firrtl Scala compiler, which is now replaced by the CIRCT implementation.
Consequently, RepCut cannot ingest the designs srN and lrN generated by
Chipyard (repcut_chipyard, ). Last, RepCut is not engineered to handle large
designs; it generates a single C++ header file that takes days to compile with
gcc (essent_takes_forever, ).

There is considerable research on parallel event-driven methods with finer
granularity than full-cycle simulation. Past work employs intelligent
concurrency techniques on x86 to avoid computation/communication (ahmad_numa,
; DBLP:conf/date/KimCY11, ; DBLP:conf/date/YangHKKCPK14, ;
DBLP:conf/vlsid/LiGL04, ).

SAGA (DBLP:conf/dac/VincoCBF12, ) applies static scheduling to SystemC
(systemc, ) and achieves a $16\times$ parallel speedup on GPUs. GCS
(DBLP:conf/dac/ChatterjeeDB09, ; DBLP:conf/date/ChatterjeeDB09, ;
DBLP:journals/todaes/ChatterjeeDB11, ) employs _levelization_
(DBLP:conf/dac/WangHPZ87, ; DBLP:conf/dac/WangM90, ) to accelerate gate-level
simulation on GPUs. Levelization splits the combinational logic into
synchronous levels of computation, each separated by a barrier (similar to BSP
but at a lower level). Parendi could leverage levelization to break stragglers
into concurrent pieces at the cost of increased synchronization and
communication.

Hundreds of cores. Hundred-core general-purpose machines are recent, so work
on hundred-core simulation is for GPUs or specialized architectures.

Since RTL is irregular, GPU acceleration typically yields low thread
utilization. For instance, Qian et al. (DBLP:conf/iccad/QianD11, ) proposes a
GPU-accelerated event-driven simulator with one thread per GPU core (i.e.,
$\frac{1}{32}$ warp utilization). RTLFlow (rtlflow, ) more fully uses a GPU by
running independent simulations using multiple test vectors for a single
design. RTLFlow performs similarly to Verilator with 1K test vectors but runs
$40\times$ faster with 64K tests. RTLFlow is limited by available GPU memory:
sr15 touches about 42 MiB of data each cycle; which means $\geq$42 GiB of
memory with 1K test vectors.

Manticore (manticore_arxiv, ) is a 225-core architecture specifically for RTL
simulation and prototyped on an FPGA. Like Parendi, Manticore uses BSP for RTL
simulation, running on a dense and deeply pipelined architecture. Manticore’s
Yosys-based frontend lacks support for Verilog packed arrays that are used
abundantly in lrN and srN. Since FPGAs have limited memory, very large designs
do not fit on Manticore, these large designs would unlikely fit into
Manticore’s limited resources. Nevertheless, our evaluation shares three
benchmarks with Manticore: the bitcoin, mc, and vta. We provide raw
performance numbers in kHz (see Table 1 and Table 3) so that our results can
be directly compared to the reported numbers in Manticore (manticore_arxiv, )
and future work. On Parendi, bitcoin is 40% slower, but mc and vta are 40% and
63% faster, respectively. Owing to Manticore’s huge register file, it achieves
a higher single-core performance than the IPU despite being clocked at only
475 MHz. Therefore, a small design like bitcoin runs faster on Manticore,
whereas vta and mc that are larger and more parallel do better on Parendi
because Manticore has only 225 cores.

Nexus (nexus_emulation:22, ) is an FPGA-based parallel RTL simulator with 240
8-bit processors in a systolic array. Like Manticore, Nexus suffers from
limited SRAM resources on FPGAs. ASH (ash, ) extends the Swarm architecture
(DBLP:conf/micro/JeffreySYES15, ) with prioritized dataflow to accelerate RTL
simulation. It consists of 256 simple x86 cores with dedicated task queues to
enable efficient event-driven simulation, showing $32\times$ speedup over
Verilator. However, ASH is neither silicon proven nor prototyped on FPGAs.

Thousands of cores. Previous work on thousand-way parallel simulation differs
from Parendi since it does not start from RTL but rather C++ processor models
(i.e., an architectural simulator). An RTL simulator parallelizes a model (the
code), whereas an architectural simulator _implements_ parallel models. Hence,
the simulator developer must parallelize the code rather than a compiler.
Moreover, many architectural simulators compromise modeling accuracy to enable
efficient parallel execution (DBLP:conf/isca/SanchezK13, ;
DBLP:conf/hpca/MillerKKGBCEA10, ; DBLP:journals/sigarch/ChenAD09, ;
DBLP:conf/date/ZurstrassenCJYXL23, ), which is incompatible with RTL’s rigid
semantics.

Some architectural simulators use multiple machines to simulate large systems
(DBLP:conf/ispass/FuW14, ; DBLP:conf/hpca/MillerKKGBCEA10, ;
DBLP:conf/date/LopezParadisLAWMB23, ). The primary motivation for distributed
simulation is to leverage a cluster of machines’ computing and memory
resources, similar to Parendi. Metro-MPI (DBLP:conf/date/LopezParadisLAWMB23,
) is a hybrid example that generates C++ processor models from RTL using
Verilator and stitches them with a C++ driver. Their methodology requires
design expertise to manually and correctly partition designs. Parendi could
leverage a methodology similar to Metro-MPI to allow users to guide its
partitioning for improved performance or decreased compile time.

Emulating large systems on FPGAs is an alternative. Capacity and compile time
are significant challenges in FPGA-based emulation. Some systems, such as
FireSim (DBLP:journals/micro/KarandikarMKBAL19, ) and DIABLO
(DBLP:conf/asplos/TanQCAP15, ) simulate warehouse-scale computers, but the
individual FPGAs are limited in size (e.g., 8-core processor).

Other emulation platforms connect tens of FPGAs to make one giant logical FPGA
(DBLP:journals/tcad/BabbTDHHA97, ; fpga_emulation, ) that circumvents resource
limits. These systems face problems similar to software-based RTL simulation,
such as partitioning, but also suffer from extended compilation time (hours to
days) to map logic to FPGA primitives. However, they can emulate large systems
as fast as 1 MHz at a high cost.

##  8\. Conclusion

A thousand-way parallel RTL simulation is becoming necessary. Current
simulation techniques can adequately exploit only tens of cores in general-
purpose processors due to their high synchronization and communication costs.

We used a 1472-core computer, the Graphcore IPU, to study the feasibility and
challenges of a thousand-way parallel simulation. Our study analyzed three
dimensions of parallel simulation: synchronization, communication, and
computation. Using the results of this analysis, we implemented Parendi, an
RTL compiler that can compile an RTL simulation to use 5888 cores effectively.

Despite the IPU’s almost $84\times$ single-core performance disadvantage
compared to recent, powerful x86 machines, Parendi simulations run up to
$4\times$ faster for large designs. Our work demonstrated that a thousand-way
parallel RTL simulation is practical and beneficial. It opens new avenues for
future research to improve simulation performance beyond what is possible
today.

##  9\. Acknowledgements

We are grateful to Graphcore for lending us the M2000 hardware. We thank the
the Graphcore staff and engineers, especially Mark Pupilli, Dario Domizioli,
Peter Birch, Svetlomir Hristozkov, Marie-Ann Le Menn, and David Bozier. They
helped us throughout the project development, from getting us started to
detailed explanations of the inner workings of poplar and popc, and providing
feedback on our work.

At EPFL, Sahand Kashani and Rishabh Iyer’s feedback on the writing and paper’s
narrative helped us make tangible improvements. Furthermore, Sanidhya Kashyap
generously allowed us to use their machines to benchmark Verilator. Last but
not least, we thank Jiacheng Ma, who accidentally made us realize the
potential of using the IPUs for RTL simulation.

## References

  * [1] 4th gen AMD EPYC Processor Archiecture.  Technical report, AMD.
  * [2] AI IPU Cloud Infrastructure.  https://gcore.com/cloud/ai-platform.  Accessed: 22-11-2023.
  * [3] Azure pricing calculator.  https://azure.microsoft.com/en-us/pricing/calculator/.
  * [4] Introducing the Colussus MK2 GC200 IPU.  https://www.graphcore.ai/products/ipu.  Accessed: 2023-11-23.
  * [5] Long time to compile complicated processor.  https://github.com/ucsc-vama/essent/issues/15, sep 2022.
  * [6] Simulation performance differs with different Verilog styles.  https://github.com/verilator/verilator/issues/4547, oct 2023.
  * [7] Using essent with chipyard.  https://github.com/ucsc-vama/essent/issues/20, sep 2023.
  * [8] Tariq Bashir Ahmad, Namdo Kim, Byeong Min, Apurva Kalia, Maciej Ciesielski, and Seiyang Yang.  Scalable parallel event-driven HDL simulation for multi-cores.  In 2012 International Conference on Synthesis, Modeling, Analysis and Simulation Methods and Applications to Circuit Design (SMACD), pages 217–220, 2012.
  * [9] Alon Amid, David Biancolin, Abraham Gonzalez, Daniel Grubb, Sagar Karandikar, Harrison Liew, Albert Magyar, Howard Mao, Albert J. Ou, Nathan Pemberton, Paul Rigge, Colin Schmidt, John Charles Wright, Jerry Zhao, Yakun Sophia Shao, Krste Asanovic, and Borivoje Nikolic.  Chipyard: Integrated Design, Simulation, and Implementation Framework for Custom SoCs.  IEEE Micro, 40(4):10–21, 2020.
  * [10] Krste Asanović, Rimas Avižienis, Jonathan Bachrach, Scott Beamer, David Biancolin, Christopher Celio, Henry Cook, Palmer Dabbelt, John Hauser, Adam Izraelevitz, Sagar Karandikar, Benjamin Keller, Donggyu Kim, John Koenig, Yunsup Lee, Eric Love, Martin Maas, Albert Magyar, Howard Mao, Miquel Moreto, Albert Ou, David Patterson, Brian Richards, Colin Schmidt, Stephen Twigg, Huy Vo, and Andrew Waterman.  The Rocket Chip Generator.  Technical report, University of California, Berkeley, 2016.
  * [11] Jonathan Babb, Russell Tessier, Matthew Dahl, Silvina Hanono, David M. Hoki, and Anant Agarwal.  Logic emulation with virtual wires.  IEEE Trans. Comput. Aided Des. Integr. Circuits Syst., 16(6):609–626, 1997.
  * [12] Jonathan Bachrach, Huy Vo, Brian C. Richards, Yunsup Lee, Andrew Waterman, Rimas Avizienis, John Wawrzynek, and Krste Asanovic.  Chisel: constructing hardware in a Scala embedded language.  pages 1216–1225, 2012.
  * [13] Scott Beamer.  A Case for Accelerating Software RTL Simulation.  IEEE Micro, 40(4):112–119, 2020.
  * [14] Scott Beamer and David Donofrio.  Efficiently Exploiting Low Activity Factors to Accelerate RTL Simulation.  pages 1–6, 2020.
  * [15] Peter Birch.  Open source FPGA-based emulation with nexus.  In Workshop on Open-Source EDA Technology (WOSET), number 1, 2022\.
  * [16] Debapriya Chatterjee, Andrew DeOrio, and Valeria Bertacco.  Event-driven gate-level simulation with GP-GPUs.  pages 557–562, 2009.
  * [17] Debapriya Chatterjee, Andrew DeOrio, and Valeria Bertacco.  GCS: High-performance gate-level simulation with GPGPUs.  pages 1332–1337, 2009.
  * [18] Debapriya Chatterjee, Andrew DeOrio, and Valeria Bertacco.  Gate-Level Simulation with GPU Computing.  ACM Trans. Design Autom. Electr. Syst., 16(3):30:1–30:26, 2011\.
  * [19] Jianwei Chen, Murali Annavaram, and Michel Dubois.  SlackSim: a platform for parallel simulations of CMPs on CMPs.  SIGARCH Comput. Archit. News, 37(2):20–29, 2009.
  * [20] Fares Elsabbagh, Shabnam Sheikhha, Victor A. Ying, Quan M. Nguyen, Joel S. Emer, and Daniel Sanchez.  Accelerating RTL Simulation with Hardware Software Co-Design.  In MICRO-56: 56th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO ’23, New York, NY, USA, 2023. Association for Computing Machinery.
  * [21] Mahyar Emami, Sahand Kashani, Keisuke Kamahori, Mohammad Sepehr Pourghannad, Ritik Raj, and James R. Larus.  Manticore: Hardware-Accelerated RTL Simulation with Static Bulk-Synchronous Parallelism.  In ASPLOS (4), pages 219–237, 2023.
  * [22] Harry Foster.  Part 4: The 2020 Wilson Research Group Functional Verification Study, FPGA Verification Effort Trends, 12 2020.
  * [23] Harry Foster.  Part 8: The 2020 Wilson Research Group Functional Verification Study, IC/ASIC Resource Trends, 1 2021.
  * [24] Yaosheng Fu and David Wentzlaff.  PriME: A parallel and distributed simulator for thousand-core chips.  In Proceedings of the 2014 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS), pages 116–125, 2014.
  * [25] M. R. Garey, Ronald L. Graham, and David S. Johnson.  Performance Guarantees for Scheduling Algorithms.  Oper. Res., 26(1):3–21, 1978.
  * [26] Adam M. Izraelevitz, Jack Koenig, Patrick Li, Richard Lin, Angie Wang, Albert Magyar, Donggyu Kim, Colin Schmidt, Chick Markley, Jim Lawson, and Jonathan Bachrach.  Reusability is FIRRTL ground: Hardware construction languages, compiler frameworks, and transformations.  pages 209–216, 2017.
  * [27] Mark C. Jeffrey, Suvinay Subramanian, Cong Yan, Joel S. Emer, and Daniel Sánchez.  A scalable architecture for ordered parallelism.  In Proceedings of the 48th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), pages 228–241, 2015.
  * [28] Zhe Jia, Blake Tillman, Marco Maggioni, and Daniele Paolo Scarpazza.  Dissecting the Graphcore IPU Architecture via Microbenchmarking.  CoRR, abs/1912.03413, 2019.
  * [29] Sagar Karandikar, Howard Mao, Donggyu Kim, David Biancolin, Alon Amid, Dayeol Lee, Nathan Pemberton, Emmanuel Amaro, Colin Schmidt, Aditya Chopra, Qijing Huang, Kyle Kovacs, Borivoje Nikolic, Randy Howard Katz, Jonathan Bachrach, and Krste Asanovic.  FireSim: FPGA-Accelerated Cycle-Exact Scale-Out System Simulation in the Public Cloud.  IEEE Micro, 39(3):56–65, 2019.
  * [30] Dusung Kim, Maciej J. Ciesielski, and Seiyang Yang.  A new distributed event-driven gate-level HDL simulation by accurate prediction.  pages 547–550, 2011.
  * [31] Helena Krupnova and Gabriele Saucier.  FPGA-based emulation: Industrial and custom prototyping solutions.  In Proceedings of the The Roadmap to Reconfigurable Computing, 10th International Workshop on Field-Programmable Logic and Applications, FPL ’00, page 68–77, Berlin, Heidelberg, 2000. Springer-Verlag.
  * [32] Tun Li, Yang Guo, and Sikun Li.  Design and Implementation of a Parallel Verilog Simulator: PVSim.  In VLSI Design, pages 329–334, 2004.
  * [33] Dian-Lun Lin, Haoxing Ren, Yanqing Zhang, Brucek Khailany, and Tsung-Wei Huang.  From RTL to CUDA: A GPU Acceleration Flow for RTL Simulation with Batch Stimulus.  pages 88:1–88:12, 2022.
  * [34] Guillem López-Paradís, Brian Li, Adrià Armejach, Stefan Wallentowitz, Miquel Moretó, and Jonathan Balkind.  Fast Behavioural RTL Simulation of 10B Transistor SoC Designs with Metro-Mpi.  pages 1–6, 2023.
  * [35] George Marsaglia.  Xorshift RNGs.  Journal of Statistical Software, 8(14):1–6, 2003.
  * [36] Jason E. Miller, Harshad Kasture, George Kurian, Charles Gruenwald III, Nathan Beckmann, Christopher Celio, Jonathan Eastep, and Anant Agarwal.  Graphite: A distributed parallel simulator for multicores.  In Proceedings of the 16th IEEE Symposium on High-Performance Computer Architecture (HPCA), pages 1–12, 2010.
  * [37] Thierry Moreau, Tianqi Chen, Luis Vega, Jared Roesch, Eddie Q. Yan, Lianmin Zheng, Josh Fromm, Ziheng Jiang, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy.  A Hardware-Software Blueprint for Flexible Deep Learning Specialization.  IEEE Micro, 39(5):8–16, 2019.
  * [38] Open-Source FPGA Bitcoin Miner.  https://github.com/progranism/Open-Source-FPGA-Bitcoin-Miner.
  * [39] OSCI.  SystemC.  https://www.systemc.org.
  * [40] PicoRV32 - A Size-Optimized RISC-V CPU.  https://github.com/YosysHQ/picorv32.
  * [41] Hao Qian and Yangdong Deng.  Accelerating RTL simulation with GPUs.  pages 687–693, 2011.
  * [42] Karl Rupp.  Microprocessor trend data.  https://github.com/karlrupp/microprocessor-trend-data, 2022.  Accessed: 18-10-2023.
  * [43] Sartaj Sahni.  Algorithms for Scheduling Independent Tasks.  J. ACM, 23(1):116–127, 1976.
  * [44] Vivek Sarkar and John L. Hennessy.  Compile-time partitioning and scheduling of parallel programs.  In SIGPLAN Symposium on Compiler Construction, pages 17–26, 1986\.
  * [45] Sebastian Schlag, Tobias Heuer, Lars Gottesbüren, Yaroslav Akhremtsev, Christian Schulz, and Peter Sanders.  High-Quality Hypergraph Partitioning.  ACM J. Exp. Algorithmics, 27:1.9:1–1.9:39, 2022.
  * [46] Wilson Snyder.  Verilator, accelerated: Accelerating development, and case study of accelerating performance.  2nd Workshop on Open-Source Design Automation (OSDA).
  * [47] Wilson Snyder.  Verilator 4.0: Open simulation goes multithreaded.  The OPen Source Digital Design Conference (ORConf), 2018.
  * [48] Wilson Snyder.  Your Big 4th Simulator: 2019 intro and roadmap.  CHIPS Alliance, 2019.
  * [49] Zoya Svitkina and Lisa Fleischer.  Submodular Approximation: Sampling-based Algorithms and Lower Bounds.  SIAM J. Comput., 40(6):1715–1737, 2011.
  * [50] Daniel Sánchez and Christos Kozyrakis.  ZSim: fast and accurate microarchitectural simulation of thousand-core systems.  In Proceedings of the 40th International Symposium on Computer Architecture (ISCA), pages 475–486, 2013.
  * [51] Zhangxi Tan, Zhenghao Qian, Xi Chen, Krste Asanovic, and David A. Patterson.  DIABLO: A Warehouse-Scale Computer Network Simulator using FPGAs.  In Proceedings of the 20th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS-XX), pages 207–221, 2015.
  * [52] Xiang Tian and Khaled Benkrid.  Design and implementation of a high performance financial Monte-Carlo simulation engine on an FPGA supercomputer.  pages 81–88, 2008.
  * [53] Jeffrey D. Ullman.  NP-Complete Scheduling Problems.  J. Comput. Syst. Sci., 10(3):384–393, 1975.
  * [54] Leslie G. Valiant.  A Bridging Model for Parallel Computation.  Commun. ACM, 33(8):103–111, 1990.
  * [55] Sara Vinco, Debapriya Chatterjee, Valeria Bertacco, and Franco Fummi.  SAGA: SystemC acceleration on GPU architectures.  pages 115–120, 2012.
  * [56] Haoyuan Wang and Scott Beamer.  RepCut: Superlinear Parallel RTL Simulation with Replication-Aided Partitioning.  In ASPLOS (3), pages 572–585, 2023.
  * [57] L.-T. Wang, Nathan E. Hoover, Edwin H. Porter, and John J. Zasio.  SSIM: A Software Levelized Compiled-Code Simulator.  pages 2–8, 1987.
  * [58] Zhicheng Wang and Peter M. Maurer.  LECSIM: A Levelized Event Driven Compiled Logic Simulation.  pages 491–496, 1990.
  * [59] Seiyang Yang, Jaehoon Han, Doowhan Kwak, Namdo Kim, Daeseo Cha, Junhyuck Park, and Jay Kim.  Predictive parallel event-driven HDL simulation with a new powerful prediction strategy.  pages 1–3, 2014.
  * [60] Jerry Zhao, Animesh Agrawal, Borivoje Nikolic, and Krste Asanović.  Constellation: An open-source SoC-capable NoC generator.  In 2022 15th IEEE/ACM International Workshop on Network on Chip Architectures (NoCArc), pages 1–7, 2022.
  * [61] Niko Zurstraßen, José Cubero-Cascante, Jan Moritz Joseph, Li Yichao, Xinghua Xie, and Rainer Leupers.  par-gem5: Parallelizing gem5’s Atomic Mode.  pages 1–6, 2023.