# ACCL+: an FPGA-Based Collective Engine for Distributed Applications

Zhenhao He
Systems Group, ETH Zurich  Dario Korolija
Systems Group, ETH Zurich  Yu Zhu
Systems Group, ETH Zurich  Benjamin Ramhorst
Systems Group, ETH Zurich  Tristan Laan
University of Amsterdam Work done during internship at Research Labs AMD
Xilinx Lucian Petrica
Research Labs AMD Xilinx  Michaela Blott
Research Labs AMD Xilinx  Gustavo Alonso
Systems Group, ETH Zurich

###### Abstract

FPGAs are increasingly prevalent in cloud deployments, serving as Smart NICs
or network-attached accelerators. Despite their potential, developing
distributed FPGA-accelerated applications remains cumbersome due to the lack
of appropriate infrastructure and communication abstractions. To facilitate
the development of distributed applications with FPGAs, in this paper we
propose ACCL+, an open-source versatile FPGA-based collective communication
library. Portable across different platforms and supporting UDP, TCP, as well
as RDMA, ACCL+ empowers FPGA applications to initiate direct FPGA-to-FPGA
collective communication. Additionally, it can serve as a collective offload
engine for CPU applications, freeing the CPU from networking tasks. It is
user-extensible, allowing new collectives to be implemented and deployed
without having to re-synthesize the FPGA circuit. We evaluated ACCL+ on an
FPGA cluster with 100 Gb/s networking, comparing its performance against
software MPI over RDMA. The results demonstrate ACCL+’s significant advantages
for FPGA-based distributed applications and highly competitive performance for
CPU applications. We showcase ACCL+’s dual role with two use cases: seamlessly
integrating as a collective offload engine to distribute CPU-based vector-
matrix multiplication, and serving as a crucial and efficient component in
designing fully FPGA-based distributed deep-learning recommendation inference.

##  1 Introduction

FPGAs are increasingly being deployed in data centers [73, 14] as Smart NICs
[61, 30, 94, 58, 24], streaming processors [27, 26, 62, 48], and disaggregated
accelerators [84, 59, 55, 77, 36, 13, 106]. In scenarios where FPGAs are
directly connected to the network, an opportunity is created for distributed
computing with direct FPGA-to-FPGA communication. However, designing
distributed applications with FPGAs is difficult as it requires not only a
network stack on the FPGA compatible with the data center infrastructure, but
often also higher level abstraction, e.g., collective communication, for more
complex communication patterns. Unlike in the software ecosystem where many
open-source collective libraries exist [33, 70], there is a lack of such
resources for FPGA developers. While new development platforms [52, 46] are
improving FPGA programmability, and other recent FPGA virtualization platforms
[64, 54, 15, 66, 91, 8] focus on virtualizing FPGA resources for abstracting
data movement, they lack support for networking. This forces distributed
applications on FPGAs to rely on the CPU for networking [107, 83, 17],
significantly increasing the latency of data transfers between FPGAs. It has
not been until recently that hardware networking support [78, 12, 86, 40, 49]
has emerged in the FPGA ecosystem. However, they all fall short in providing
abstractions for collective communication, limiting their applicability in
larger distributed use cases.

Figure 1: CCL in different FPGA-accelerated systems.

Implementing high-performance and versatile collective abstractions for FPGAs
poses challenges specific to reconfigurable computing. Portability stands out
as a key challenge, as FPGAs are used in quite different configurations. The
first aspect of the portability challenge is the memory model. FPGAs support
various memory models, such as partitioned memory model [52, 54, 47] and
shared virtual memory model [56, 64, 97, 69], with each memory model being
suitable for different scenarios. Figure 1 illustrates how an FPGA-based
collective communication library (CCL) can be effectively utilized with
different memory models. In Figure 1(a), the FPGA application relies on the
CPU network stack for inter-FPGA data movement, while Figure 1(b) showcases
the FPGA CCL enabling direct networking between FPGAs. If the data originates
from CPU memory (dashed line), an explicit memory copy is required for
partitioned memory platform. Figure 1(c) illustrates CCL implementation for an
FPGA application with shared virtual memory, while Figure 1(d) exemplifies
CCL’s role as an offload engine for a CPU application with shared virtual
memory.

The second aspect is the portability across communication models as FPGAs
support two fundamentally different ones: message passing, i.e., MPI, where
communication occurs between buffers in memory, and streaming, where
communication occurs through continuous data streams. The choice of
communication model significantly influences how an application interacts with
the communication layer, determining whether data needs to be buffered in
memory before communication [41, 75, 76], or if it can be streamed directly to
the communication layer [21, 28, 53]. Existing FPGA communication frameworks
tend to address only one of them.

The third aspect of the portability involves supporting diverse communication
protocols due to application-specific needs and ensuring interoperability in
heterogeneous environments where FPGAs coexist with CPUs and accelerators.
This flexibility ensures seamless integration and communication between FPGA-
based offloaded components and other system elements. Existing works [31, 21,
92, 75, 76, 39, 20, 67, 103, 95] that provide collective abstractions on FPGAs
or provide multi-FPGA support are often tailored for scenarios where FPGAs are
directly connected to each other rather than integrated into a data center
packet-switched network. These approaches communicate through low-level link-
layer protocols, leading to scalability and integration challenges at a data-
center scale.

In summary, the key question to address is how to effectively design a
portable, flexible, high-level collective abstraction on FPGAs that can
support various memory models, communication models, and transport protocols,
while accommodating a broad spectrum of applications. Achieving this objective
is complex, given the significant impact of these configurations on runtime,
interfaces, and data movement. Moreover, given FPGAs’ extended compilation
times and lengthy hardware debugging cycles, we need a parameterized approach
that allows a FPGA-resident CCL to be modified without recompilation, in order
for the CCL to be practical in real-world use.

To address these challenges, we introduce ACCL+, a hardware collective
communication library (CCL) for FPGAs. ACCL+ provides MPI-like collective APIs
with explicit buffer allocation and streaming collective APIs with direct
channels to the communication layer. The modular system architecture decouples
platform-specific IO management and runtime from the collective
implementation, incorporating platform and network protocol-specific adapters
and drivers. Additionally, we have developed a platform and protocol-
independent collective offload engine that supports modifying the collective
implementation without hardware recompilation. ACCL+ can be used to enable
direct communication between FPGAs and can function as a collective offload
engine for the CPU. We test ACCL+ on two platforms, a commodity, partitioned
memory platform (AMD Vitis [52]) and a shared virtual memory platform (Coyote
[56]). We choose AMD Vitis for its recent integration of high-performance 100
Gb/s UDP [98] and TCP [40] hardware stacks, aligning with our goal of
leveraging cutting-edge networking capabilities for optimal communication
performance. Coyote is used due to its unique provisioning of unified and
virtualized memory across CPU-FPGA boundaries [56], coupled with comprehensive
network services, including RDMA. ACCL+ is also designed to minimize control
overheads, improve scalability, and facilitate simulation.

We first evaluate ACCL+ using micro benchmarks. ACCL+ achieves a peak
send/recv throughput of 95 Gbps, almost saturating the 100 Gb/s network
bandwidth. We evaluate collective operations under two scenarios: FPGA-to-FPGA
distributed applications with FPGA kernels directly interacting with ACCL+
(F2F), and CPU-to-CPU distributed applications with ACCL+ as a collective
offload engine (H2H). ACCL+ exhibits significantly lower latency in F2F
scenarios compared to software RDMA MPI for FPGA-generated data. In H2H
scenarios, ACCL+ has comparable performance to software RDMA MPI for CPU-
generated data while freeing up CPU cycles and reducing pressure on CPU
caches. Then, we examine ACCL+ with two use-case scenarios. First, distributed
vector-matrix multiplication with CPU computation and ACCL+-based reduction,
where ACCL+ improves performance compared to software MPI. Secondly, we show
that ACCL+ enables the distribution of an industrial recommendation model
across a cluster of 10 FPGAs, achieving more than two orders of magnitude
lower inference latency and more than an order of magnitude higher throughput
than CPU solutions. The use case study not only highlights ACCL+’s
effectiveness in different scenarios but also paves the way for future
research opportunities in investigating hybrid CPU-FPGA co-design for
distributed applications.

##  2 Background

FPGA Programming. In the past, hardware description language (HDL - Verilog,
VHDL) was the sole method to program FPGAs. With the emergence of High-Level-
Synthesis (HLS), the programmability of FPGAs is enhanced by allowing the
developers to program in C-like code with hints (pragmas) to infer parallel
hardware blocks. Despite the advancements, existing HLS-based libraries lack
networking and collective abstractions.

Communication Models. Message passing, represented by MPI, is a well-
established communication model in distributed programming on CPUs, whereby
communicating agents exchange messages, i.e., user buffers, typically
resulting from previous computation. This model can be applied to FPGAs, but a
more common communication model for FPGAs is the streaming model. FPGA kernels
support direct streaming interfaces, into which data can be pushed in a
pipelined fashion during processing. Kernels executing on the same FPGA can
stream data to each-other through low-level latency-insensitive channels, such
as AXI-Stream [7]. The streaming model can be applied for communication across
FPGAs, however, existing streaming communication framework [28, 29, 21] often
do not have transport protocols or collective abstractions.

FPGA Development Platforms. Modern FPGA platforms adopt various virtualization
methodologies [90, 74, 11] for FPGA resources. Most simplify development with
a static _shell_ for resource management and data movement, with some offering
additional services like transport layer networking, and the host-device
interaction relies on runtime libraries. This approach allows developers to
concentrate on designing the application kernel. Many commodity platforms [52,
47] implement a partitioned memory model, which permits data movement from
FPGA applications to FPGA memory while restricting direct access to host CPU
memory. In contrast, shared virtual memory platforms, such as Coyote [56] and
Optimus [64], offer an abstraction of virtualized and unified memory space
across CPU and FPGA that the FPGA kernel can directly access. This also
facilitates RDMA to CPU memory from FPGA network streams.

Network-Attached FPGAs. The data center FPGA landscape has evolved with FPGAs
featuring 100 Gb/s transceivers, enabling direct processing of network data
[73, 96]. FPGA-based Smart NICs [30, 61, 12, 94, 89, 110, 1, 60, 23] perform
programmable packet filtering but often leave the network stack to the CPU
software, limiting their applicability to FPGA applications. Distributed
machine learning [2, 16, 109, 10, 71], and data processing [59, 19, 55]
applications capitalize on network-attached FPGAs and thepir communication
pattern tends to be more complicated with larger designs, motivating the need
for high-level collective abstractions on FPGAs. Following this trend, there
is an increasing effort to develop hardware network stacks on FPGAs, such as
UDP [98, 42], TCP [22, 85, 49, 78], and RDMA [82, 86, 63, 59]. However, only
network stack offload is often not enough for complicated applications in
distributed settings.

##  3 Related Work

MPI implementation of collective communication has evolved to become more
accelerator-aware over time, e.g., GPU-aware MPI [93, 102, 72]. Some GPU-
direct MPI implementations enable direct data transfers between GPU memories
through RDMA NIC, bypassing host memory [68, 3]. However, FPGA-based
collective is different as FPGAs can connect directly to the network, removing
the need for an external NIC. Additionally, standard remote DMA with commodity
NICs lacks support for streaming interfaces to FPGA kernels, forcing streaming
kernels to buffer data before collective operations. Projects such as
Galapagos [88, 25] and EasyNet [40] provide in-FPGA communication stacks for
data exchange within a cluster, serving as a foundation for collectives
without an external NIC. TMD-MPI [80, 79] orchestrates in-FPGA collectives
using embedded processors, yet its bottleneck lies in control due to
sequential execution in low-frequency FPGA microprocessors. Collective offload
with NetFPGA [4, 5, 6] has been explored, but static collective offload
engines limit flexibility and often rely on software-defined network switches
for orchestration. SMI [21] proposes a streaming message-passing model,
exposing streaming collective interfaces to FPGA kernels. While SMI enables
kernels to initiate collectives directly, it employs dedicated FPGA logic for
collective control, limiting flexibility for post-synthesis reconfiguration.
ACCL [41] focuses primarily on message-passing collectives for FPGA
applications. The coordination of collectives requires CPU involvement,
lacking significant streaming support, and has not been tested at scale.
BluesMPI [9, 87] offloads collective operations to a BlueField Smart-NIC,
demonstrating comparable communication latency to host-based collectives, but
it does not target accelerator applications. The latency of ACCL+ targeting
host data matches BluesMPI, even with BluesMPI ARM cores working at ten times
the frequency.

Frameworks like ViTAL and its successors [103, 104, 105] propose FPGA resource
virtualization and compilation flows for mapping large designs onto multiple
FPGAs through latency-insensitive channels. OmpSs@cloudFPGA [20] introduces a
multi-FPGA programming framework that partitions large OpenMP programs with
domain-specific programs into smaller distributed parts for execution on FPGA
clusters, providing communication through static, compile-time-defined
send/recv and collective operations supporting only the unreliable UDP
protocol. Elastic-DF [2] and FCsN [35] present domain-specific frameworks for
automatically distributing large neural network model inference across FPGAs
with hardware UDP/TCP send/recv for FPGA-to-FPGA data movement. These projects
are complementary to our work, and integrating ACCL+ as their FPGA
communication backend can enhance their flexibility and performance.

##  4 ACCL+: An FPGA Collective Engine

Figure 2: System overview of the FPGA-based collective communication library.

The architecture of ACCL+ 111https://github.com/Xilinx/ACCL is modular,
featuring layers of abstraction in both software and hardware, as illustrated
in Figure 2, facilitating adaptability of a central ACCL+ CCL offload engine
(CCLO) to diverse platforms and communication protocols. In this section we
will describe each layer.

###  4.1 Application Interface

ACCL+ Drivers. FPGA-accelerated applications execute partially on the CPU and
partially on FPGA, whereby each of the CPU- and FPGA-resident components of
the application may require access to collective communication infrastructure.
For this reason ACCL+ implements two drivers which expose similar platform-
and protocol-agnostic collective APIs to each of these two components. The
host-side ACCL+ CCL driver allows initialization and runtime management of
platform and ACCL+ data structures and hardware, as well as POE
initialization, i.e., setting up sessions or queue-pairs. The CCL HLS driver
is not capable of such initialization, and therefore the application must
perform host-side initialization before any FPGA application kernels are
started.

MPI-like Collective API. ACCL+ drivers expose an MPI-like API, catering to the
message-passing paradigm and facilitating the porting of existing MPI-based
applications to ACCL+ collectives. This API require the application to store
data in memory before invoking collectives. Listing LABEL:lst:reduce shows the
MPI-like collective API, including arguments like datatype, buffer pointer,
and element count, along with flags indicating buffer location (host or FPGA
memory) and the option for synchronous calls. To facilitate portability,
message passing collectives operate on an ACCL+ specific buffer class which
can wrap normal C++ arrays with additional platform-specific information.
Common collectives, such as reduce, broadcast, and barrier, are supported.
Each MPI-like collective call in the host CCL driver has a corresponding HLS
API call with a similar syntax for direct invocation from FPGA kernels.

Listing 1: Reduce collective API in C++.

⬇

1CCLRequest *reduce(dataType src_data_type, BaseBuffer &buf, unsigned int
count, unsigned int root, reduceFunction func, communicatorId comm_id,
flagType flags);

Streaming Collective API. ACCL+ also provides a streaming collectives API
whereby data originates and terminates at the stream interfaces between the
FPGA application kernels and the ACCL+ hardware component, instead of memory
buffers. The HLS-based streaming APIs are tailored for FPGAs applications
running in streaming fashion.

Listing 2: Example kernel using streaming send in HLS.

⬇

1//set up command and data interfaces

2cclo_hls::Command cclo(cmd, sts, communicator);

3cclo_hls::Data data(data_to_cclo, data_from_cclo);

4//issue streaming send command without buffer argument

5cclo.send(type, count, dst_rank);

6// push data in streams to network without buffering

7for (int i = 0; i < N; i++) {

8 data.push(/* generate data */); }

9cclo.finalize(); // wait for send completion

Listing LABEL:lst:stream_send demonstrates an example FPGA kernel issuing a
streaming send command to the CCLO engine (line 5) and subsequent pushes to
the CCLO streaming data interface, 64B per cycle (line 8), followed by a wait
for CCLO completion. This code is synthesizable with HLS tools. HDL-based FPGA
kernels can interact with the collective engine directly, through the same
interfaces. Additionally, the host can also call streaming collectives via the
host-side CCL driver.

###  4.2 ACCL+ Platform Support

Figure 3: CCL driver for different memory managements.

A platform is defined by a software interface specification, defining how FPGA
memory is allocated and manipulated, and how FPGA kernels are called, and a
hardware interface specification, i.e., how FPGA kernels, including the CCLO,
plug into hardware services in the FPGA. To facilitate portability between
platforms, the ACCL+ host CCL driver layers the APIs on top of generic class
types, such as BaseBuffer for memory allocation and data movement between host
and FPGA, and BaseDevice for CCLO invocation. These are specialized to
individual platforms through class inheritance, as illustrated in Figure 3.
Each specific CCL class interfaces with platform-native drivers and employs
distinct processes for handling data movement. ACCL+ supports both the
commercial AMD Vitis platforms and the open-source Coyote platform [56], as
well as a virtual simulation platform. New platforms can be added easily.

Integration with Coyote. Coyote utilizes a shared-memory model with a central
memory management logic governed by a software-populated translation lookaside
buffer (TLB). This TLB records allocated pages and facilitates virtual-
physical address translation. The FPGA kernel issues memory requests through a
descriptor interface, using virtual addresses directed to either host or
device memory. The TLB interprets these requests, interacting with host DMA or
device DMA based on the physical location of the memory page, forwarding the
data to FPGA applications in a streaming manner. If a memory page is not
registered during TLB lookup, it triggers an interruption to the CPU,
resulting in a page fault and introducing a performance penalty. Therefore,
the CCL driver, specifically the CoyoteBuffer class, eagerly maps pages to the
Coyote TLBs when instantiating buffers. We modified Coyote, during
integration, to increase the associativity of the TLB cache and expand the
number of streaming interfaces Coyote provided to a single application region,
from a single interface to three interfaces which is required by the CCLO
engine. We also implemented a Coyote-specific adapter to convert from CCLO
(R)DMA request syntax to Coyote-specific syntax, as indicated in Figure 4.

Integration with Vitis. Vitis platforms implement a partitioned memory model
and the Xilinx Runtime (XRT) [52] is utilized by the ACCL+ CCL driver for low-
level interaction with the platform. A XRT-controlled XDMA IP core [101] moves
data between host and FPGA memory, while FPGA memory is accessed by FPGA
kernels through Data Movers [100]. We implemented dedicated adapters which
include Data Movers for compatibility of the CCLO Engine with Vitis platform
memory interfaces. As a result of the partitioned memory, the CCL driver
explicitly migrates buffers between host and FPGA memory prior to or after the
collective execution if the data originally resides in host memory - a process
denoted staging. Staging creates performance penalty when ACCL+ collectives
target host memory, as observed by related work on collective offload on DPUs
[87]. Therefore, Vitis platforms favor distributed FPGA applications where
data is streamed or resides in FPGA memory.

Simulation Platform We implemented an additional simulation platform for
debugging and performance optimization. This simulation platform roughly
models a Vitis platform, whereby FPGA chip interfaces (XDMA, Ethernet) are
replaced by ZMQ [43] interfaces. A stand-alone simulated FPGA node is compiled
to include memory and one ACCL+ CCLO Engine. The ACCL+ host driver includes
dedicated buffer and device abstractions capable of connecting to the
simulated node via ZMQ. ACCL+ provides convenient launch scripts to set up a
simulated cluster of such simulation nodes.

The simulated nodes connect to each other through ZMQ rather than real
Ethernet. While the simulated ZMQ network may lack realistic features like
packet loss and reordering, it serves as a valuable functional simulation.

ACCL+ provides two simulation levels of the CCLO engine: functional simulation
using compiled ACCL+ HLS source code and C firmware, and cycle-accurate (but
slow) simulation using Verilog HDL generated from compiling the CCLO HLS code
and firmware. For FPGA applications requiring streaming data exchange between
FPGA kernels and the CCLO, we provide a bus functional model of the CCLO that
connects via ZMQ to the simulated node.

###  4.3 Protocol Offload Engine

ACCL+ supports several 100 Gb/s protocol offload engines (POE) in hardware:
UDP [98] and TCP [40] on Vitis platforms, and all the network services
provided by Coyote. Notably, ACCL+ supports collectives with RDMA by
leveraging the unified and virtualized memory space across the FPGA and the
CPU provided by Coyote. All the POEs expose streaming control and data
interfaces to other modules and some POEs (e.g., TCP) require direct memory
access for packet buffering for re-transmission. For portability, the CCLO
Engine has a set of POE-independent internal interfaces - two pairs of meta
and data streaming interfaces (one for Tx and one for Rx). The meta interfaces
contains various sub fields to indicate the op code, data length,
communication session IDs, etc. The meta interfaces are then adapted to the
POE interfaces with dedicated FPGA components as exemplified in Figure 4. The
selection of the POE and its adapters is a compile time parameter of the CCLO
Engine.

Figure 4: ACCL+ with Coyote-RDMA data path with corresponding POE and memory
adapters.

Coyote RDMA POE. It supports standard RDMA verbs, including one-sided
operations like WRITE and two-sided operations like SEND. The RDMA POE
incorporates various streaming interfaces for RDMA commands, memory commands,
and data. Default Configuration: On the initiating side of a WRITE operation,
the RDMA POE issues memory requests directly to the Coyote memory management
logic, fetching data from either host or device memory and streaming it
through the network. On the passive side of WRITE, the data is directly
written to virtualized memory. ACCL+ Integration: In the ACCL+-enabled
configuration, the CCLO engine acts as a "bump-in-the-wire" engine between the
memory management unit and the RDMA POE, as shown in Figure 4. On the
initiating side of a WRITE, the CCLO engine issues RDMA commands and is
responsible for data preparation, either fetching from memory or obtaining it
from the application kernel in the form of streams. On the passive side, data
bypasses the CCLO and is directly forwarded to the memory management unit. For
single-sided WRITE, streaming into the application kernel is also possible by
configuring the datapath at compile time. The CCLO engine consistently manages
data and metadata streams from two-sided SEND. For CCL driver with RDMA, a
queue pair needs to be exchanged between each node and needs to be registered
to the RDMA POE.

TCP POE. The TCP POE supports up to 1,000 connections and can be configured to
support window scaling and out-of-order packet processing. As a reliable
transmission protocol, the TCP POE also needs to access protocol-internal
buffers for re-transmission. The CCLO engine prepares and accepts all the data
streams with the TCP POE. For CCL driver with TCP POE, a TCP session needs to
be established between each node to construct the communicator.

###  4.4 CCLO Engine

The CCLO Engine orchestrates the collective data movement through a set of
standardized CCLO interfaces. The CCLO accepts communication requests from the
host or application kernels, communicates with the protocol offload engine,
manages buffers in FPGA memory (HBM, DDR, BRAM), and manage data streams from
other kernels. To enable simultaneous flexibility, low latency, and high
throughput, the key design principle is to decouple the CCLO logic into the
flexible control plane and the parallel data processing plane, as indicated in
Figure 5.

Figure 5: Hardware architecture of CCLO engine, where 1 is collective
commands, 2 and 3 are commands for rendezvous protocol initialization and
done, 4 stands for microcode for data movement, 5 and 6 are commands to manage
Rx buffers for eager protocol.

####  4.4.1 Flexible Control Plane

The CCLO control plane is flexible, centered around an embedded micro-
controller (uC) [99], which responds to commands from other FPGA kernels or
the host and issues high-level data movement commands to a hardware-
accelerated data movement processor (DMP) and other components. The CCLO
control plane also contains a RxBuf Manager (RBM), which manages temporary Rx
buffers. The uC, DMP, and RBM store states in a small configuration memory
implemented as FPGA BRAM. The configuration memory is also accessible by the
CPU through MMIO and includes information about the communicator, e.g.,
session or queue pair IDs, pool of allocated Rx buffers.

Embedded Controller. The uC firmware implements different collective
algorithms and different synchronization protocols, e.g., eager and
rendezvous. Though all collectives could also be implemented in digital logic,
the uC provides the high flexibility to implement different collective
algorithms by updating the firmware without the need to refactorize the whole
design and re-synthesize. Considering the sequential execution nature of the
uC and a low clock frequency, to avoid too much overhead in the uC, the
firmware is carefully designed such that it only issues a set of coarse-
grained control commands to other hardware control blocks, which are highly
parallel and latency-optimized. Besides, FIFO queues are incorporated into all
command paths, allowing multiple in-flight instructions.

Data Movement Processor. The DMP accepts and executes microcode generated by
the uC 4 according to collective algorithms. Each microcode instruction has
three slots: two operand slots and one result slot. Each microcode slot
describes a data movement behavior to be executed by the data plane. Operand
slots describe data coming into the CCLO, while the result slot describes data
movement out of the CCLO. If the operand is expected to come over network and
buffered in temporary buffers, the DMP also sends out requests periodically to
the RBM to check if the message has arrived 6. For example, a send operation
uses an operand slot for data movement from FPGA memory into the CCLO and a
result slot for data leaving the CCLO into the network POE. Similarly,
reduction involves all three DMP instruction slots for orchestrating data
movement through two input buffers, arithmetic plugin, and finally to the
result buffer. The DMP operates in a pipelined fashion, and each operand slot
independently interprets its instruction fields, emitting commands for
corresponding datapath blocks. Upon receiving acknowledgements from datapath
blocks, the DMP signals instruction completion to the uC.

RxBuf Manager. The RBM is used when temporary buffering is needed, e.g.,
collective with eager protocol. Upon the notifications of incoming messages
from the network, as shown in path 5 in Figure 5, RBM retrieves a list of
available Rx buffers from the configuration memory and then it issues memory
requests to store the message in the selected Rx buffer. As messages are
typically divided into network packets, and packets from different sessions
may arrive in an interleaved manner, the RBM also contains data structures to
reassemble the packets into messages in corresponding Rx buffer. The RBM also
stores relevant metadata (source ID, tag, Rx buffer address) to be used by the
DMP.

####  4.4.2 Parallel Data Plane

Rx and Tx System. In ACCL+, we implement a lightweight communication protocol
above the transport layer protocol to carry metadata information for each
message. Each message consists of a signature and a payload consisting of user
data. The signature contains the rank IDs of the message, message type, source
and destination, message length, tag, a sequence number which is used to keep
track of the order of the messages and other meta information. The Tx and Rx
systems are responsible for packetizing and depacketizing the signature along
with user payload, and they issue commands to interact with the POEs. The
command issuing, signature insertion, and parsing processes can vary for
different synchronization protocols. Both the Rx and Tx systems incorporate a
finite state machine to respond appropriately to these variations.

Network On Chip. All the data streams internal to the CCLO can be routed in
packets based on the dest field that comes along with the data.

Streaming Plugins. The plug-ins are utilized for applying unary and binary
operations to in-flight data and can be enabled at compile time. Binary
operations are typically utilized to implement reductions - sum, max, etc.
Unary operators may implement compression or encryption. Each of the plug-ins
is a streaming kernel and may implement more than one function, in which case
the control plane will specify the desired function by setting its dest field
of the plugin input stream.

####  4.4.3 Message Synchronization Protocol

[Eager protocol] [Rendezvous protocol]

Figure 6: CCLO eager and rendezvous with send/recv.

Eager protocol allows sender CCLO to send as soon as the command arrives and
the data at the receiver is buffered in the CCLO Rx buffer before it is moved
to the destination (either in memory or in FPGA kernel streams depends on
runtime configuration), as shown in Figure 6(a). Eager protocol is sometimes
favored, especially with small messages, as the lack of a handshake phase
minimizes latency as copies incur little overhead at small message sizes. We
implement eager protocol with UDP/TCP or two-sided RDMA SEND verb. In
contrast, rendezvous protocol first requires the result buffer address
resolution before the actual message transmission, as shown in Figure 6(b).
Once the destination buffer is resolved, the data can be directly put into the
destination buffer, avoiding the cost of temporary buffering at Rx Buffer. We
use two-sided RDMA SEND verb for rendezvous handshake messages, such as
RNDZ_INIT and RDNZ_DONE, and we use RDMA WRITE for actual message
transmission.

Here we use examples to illustrate the control and data sequence for different
protocols. RDMA Rendezvous Send/Recv: Upon the receipt of invocation command
on the Recv rank, the uC initiates a Tx control command containing the result
buffer address to the Tx system 2. The Tx system generates an RDZV_INIT
message to the Send rank. The Rx system of the Send rank queues up a
notification with the result address for the uC 3 to pick up. The uC of Send
rank issues a microcode with RNDZ_MSG type to the DMP, which fetches data from
memory and issues a Tx control to the Tx system. The Tx System issues an RDMA
WRITE command to the POE, forwarding the data to the network. Once the RDMA
WRITE is complete, the Tx System issues an RDZV_DONE message with RDMA SEND.
On the receiving side, the RNDZ_MSG is directed straight to the memory,
bypassing the receiver CCLO intervention. The subsequent RDZV_DONE is
forwarded to the Rx system and then to the uC 3. TCP/RDMA Eager Send/Recv:
Upon the command’s arrival at the uC, it issues a microcode to the DMP 4,
which issues memory commands and a Tx control with type EAGER_MSG. The Tx
system waits for the data from the memory and appends a header with the
message signature. On the receiving side, the Rx System receives the eager
message, parses the header, and forwards it to the RBM 5. The RBM accesses the
exchange memory, which stores the RxBuf list. If there is an empty Rx buffer,
it issues memory commands to buffer the input message. When the receive
command arrives at the uC, it issues microcode to the DMP 4, which checks the
RBM for the expected message 6. If the message is already stored in an Rx
buffer, the DMP initiates a memory control operation to transfer the data to
the result buffer.

####  4.4.4 Collective Algorithms

We provide different implementations for various collectives, and users can
define their own. Collectives are realized by specifying a communication
pattern as a C function in uC firmware, and then executing this pattern
through instructions in DMP and Tx/Rx System on each FPGA in the communicator.
Table 1 summarizes the algorithms and communication patterns being used to
implement collectives. For eager protocols with unreliable transmission (e.g.,
UDP), we currently use simple algorithms like ring and one-to-all to minimize
the chances of packet loss. Future firmware improvements can enhance POE
awareness for finer-grained algorithm selection. In contrast, when using RDMA,
the rendezvous protocol employs more advanced algorithms like tree or
recursive doubling. The token-based flow control in RDMA makes it well-suited
for these sophisticated algorithms in the rendezvous protocol. For broadcast,
we implement a simple one-to-all algorithm with small rank size, while with
large rank size, we adopt more advanced recursive doubling such that the data
transmission is not bottlenecked at the root rank. For gather and reduce, we
apply a similar strategy. With small message size, we adopt an all-to-one
approach to reduce the number of intermediate hops needed. On the other hand,
with larger message sizes, to avoid a potential in-cast problem with the all-
to-one approach, we adopt a tree-based algorithm. The tuning of the algorithms
for specific collective can be done at runtime by setting configuration
parameters to the CCLO engine and we set these parameters according to our
empirical experiment results.

Table 1: Algorithms used for example collectives. Collective | Eager | Rendezvous
---|---|---
Bcast | One-to-all | One-to-all;Recursive doubling
Reduce | Ring | All-to-one;Binary tree
Gather | Ring | All-to-one;Binary tree
All-to-all | Linear | Linear

##  5 Microbenchmark Evaluation

We evaluate ACCL+ on a heterogeneous cluster with 10 AMD EPYC CPUs and 10
attached FPGA cards (Alveo-U55C). Each CPU is equipped with a 100 Gb/s
Mellanox NIC, while each FPGA features a 100 Gb/s Ethernet interface. All
devices are connected to Cisco Nexus 9336C-FX2 switches. Evaluation scenarios
consider data residing on the FPGA for distributed FPGA application (suffix
F2F) and on the CPU for distributed CPU applications (suffix H2H). For F2F,
the FPGA application data traverses the network directly through ACCL+. As a
baseline, the FPGA data initially is moved to the CPU memory and then is
transmitted via a commodity NIC. In H2H, the CPU application data is
transferred to the FPGA and then transmitted with ACCL+. This is compared to
transmitting the CPU data directly with a commodity NIC. The focus of these
experiments is evaluating RDMA running with Coyote (suffix cyt) due to space
limitations. We nevertheless present some results with ACCL+ running TCP on
top of the XRT platform to compare it to ACCL [16], which utilizes an embedded
micro-controller to orchestrate collective operations. Experiments configure
both MPI-like collectives with memory pointers and streaming collectives. For
the H2H experiments, MPI-like collectives are mandatory, while the F2F
experiments are configured to run with streaming collectives. ACCL+ operates
at 250 MHz in micro-benchmarks, with varying frequency in the use-case study
due to the design complexity. The comparisons involve MPICH 4.0.2 with TCP and
OpenMPI 4.1.3 compiled with RDMA using OpenUCX 1.13.1 on the cluster CPUs and
Mellanox 100 Gb/s NICs. MPI libraries self-configure for collective algorithms
and synchronization protocols. Each micro benchmark experiment is averaged
over 250 runs.

Figure 7: Send/Recv throughput comparison. Figure 8: CCLO invocation latency
from different parts. Figure 9: Latency breakdown of broadcasting FPGA
produced data using software MPI with eight ranks with Coyote.

(a) Broadcast

(b) Gather

(c) Reduce

(d) All-to-all

Figure 10: Collective latency comparison between ACCL+ RDMA and software MPI
RDMA with eight ranks and device data

(a) Broadcast

(b) Gather

(c) Reduce

(d) All-to-all

Figure 11: Collective latency comparison between ACCL+ RDMA and software MPI
RDMA with eight ranks and host data

Send/Recv Throughput. We first evaluate pure throughput using send/recv.
Figure 7 shows the throughput comparison of ACCL+ with Coyote RDMA and
software MPI with RoCE backend. Notably, ACCL+ with RDMA achieves a peak
throughput of 95 Gb/s, nearly saturating the network bandwidth. Compared to
software MPI variants, ACCL+ exhibits comparable and slightly higher peak
throughput. This is attributed to the FPGA network stack’s ability to process
network packets at line-rate in a pipelined fashion. Moreover, there is
minimal distinction between F2F and H2H for ACCL+, thanks to the unified
memory space provided by Coyote and both host memory access through PCIe and
FPGA memory access offer higher bandwidth than the network.

Invocation Latency. Figure 8 shows the invocation latency of the CCLO engine
to execute a dummy NOP operation, which includes the time from receiving
request untill the acknowledgement. For FPGA kernels that can directly
interact with the CCLO engine, the invocation latency is minimal compared to
software invocation from the host, showing a clear benefit of bypassing host
control with FPGA-based applications. Coyote software driver contains a thin
and optimized layer for invocation and scheduling and the resulting CCLO
invocation time mainly consists of a PCIe write and a PCIe read latency. In
contrast, the XRT invocation latency is significantly higher as it is not
intended for fine-grained data movement.

FPGA-to-FPGA with software MPI. To enable a more direct ACCL+ vs. software MPI
comparison for executing collectives between kernels on FPGA, we model the
execution time for MPICH- and OpenMPI-based device-to-device data movement,
which includes: (1) moving data from FPGA HBM/kernel to host DDR through the
PCIe, (2) executing the collective using software MPI, (3) moving data from
host DDR to FPGA HBM/kernel, and (4) invoking the next computation kernel. We
use the CCLO host invocation time as an approximation of the invocation time
of other computation kernels. We measure the duration of each of the above
steps and present a break-down of execution time of the collective with Coyote
platform in Figure 9. We could observe that the PCIe transfer time is dominant
for small messages while the collective time is dominant for large messages.
Such breakdown for XRT platform can be derived by changing the Coyote
invocation latency to XRT invocation latency.

F2F Collective Latency RDMA. Figure 10 illustrates the latency of ACCL+ RDMA
collectives with various message sizes on eight Alveo-U55C boards. This is
compared to FPGA-to-FPGA data movement with software MPI over RDMA. For
clarity, we present experiments showcasing better performance between eager
and rendezvous collectives. The algorithms for each collective in ACCL+ are
detailed in Table 1. Notably, ACCL+ exhibits significant performance benefits
compared to its software counterpart. This advantage stems from the hardware’s
efficient execution of collectives and the direct network access within the
FPGA device, eliminating the need for data copying to CPU memory.

H2H Collective Latency RDMA. Figure 11 illustrates a latency comparison
between ACCL+ and software MPI targeting host data. The performance gains with
ACCL+ vary across different collectives. Notably, for operations like
broadcast and gather, ACCL+ consistently outperforms software MPI across a
range of message sizes. However, for other collectives such as reduce and all-
to-all, ACCL+ shows only marginal benefits and, in some cases, falls short of
software MPI performance. One reason is that ACCL+ is clocked at much lower
frequency compared to commodity-NIC and another reason is that software MPI
adapts its algorithms more finely to different configurations, whereas ACCL+
currently supports only a limited set of options. However, by offloading
collective to hardware, CPU cycles could be freed for other computation tasks.
Besides, by comparing ACCL+ F2F and H2H performance, we could observe that the
ACCL+ collective latency has minimal difference because Coyote with unified
memory allows direct memory access to both host and FPGA memory.

Effect of Synchronization Protocol. Despite the simpler algorithms used by
most eager collectives, such as one-to-all or ring, we observe that eager
collectives can sometimes outperform rendezvous collectives with small message
sizes, as seen in broadcast. This is because eager collectives do not require
a handshake to resolve addresses.

[8KB message size.] [128KB message size.]

Figure 12:  Latency vs. rank sizes (Reduce).

Collective Algorithm and Scalability. Figure 12 illustrates the impact of
algorithm selection and scalability on both ACCL+ and software MPI during
collective executions. For an 8 KB message size, ACCL+’s reduce operation
adopts an all-to-one algorithm, resulting in minimal latency increase across
nodes. However, recognizing potential bottlenecks at the root node with this
approach, ACCL+ switches to a binary tree algorithm for larger message sizes,
such as 128 KB. In this case, an increase in latency is observed after four
nodes, stabilizing until eight nodes due to a consistent tree depth. On the
other hand, software MPI exhibits a more fine-grained approach to algorithm
selection based on the scale of the message size and the number of nodes. For
instance, it deploys three distinct algorithms within the 8 KB range: an all-
to-one algorithm for fewer than four nodes, a ring protocol for four to eight
nodes, and an optimized binomial algorithm for 8 nodes. Additionally, for
larger messages, software MPI switches between an all-to-one algorithm below
three nodes and a binomial tree algorithm between four and eight nodes. This
fine-grained algorithmic tuning contributes to its superior performance in
certain H2H scenarios. While software MPI’s approach involves detailed
algorithmic tuning, ACCL+’s flexible design allows for potential future
enhancements through additional fine-grained tuning to further optimize
performance.

XRT Platform and TCP. In Figure 13, we evaluate ACCL+ TCP with the XRT
platform and compare it against software MPI with TCP. We also include a
comparison with ACCL [41] collectives, which employs a similar embedded
processor to orchestrate collectives and supports TCP on the XRT platform.
Notably, ACCL+ TCP consistently outperforms its software counterpart across
all configurations, benefiting from the line-rate processing capabilities of a
hardware TCP POE. Furthermore, ACCL+ demonstrates superior performance
compared to ACCL. While both ACCL+ and ACCL utilize embedded microprocessors
for collective orchestration in hardware, ACCL+ distinguishes itself by
offloading more tasks to the hardware data plane, such as utilizing the Rx
Buffer Manager for packet assembling. In contrast, ACCL relies more on the
microprocessor, leading to lower performance. When comparing ACCL+ TCP for
serving host applications and device applications, a significant overhead is
observed for host applications. This is attributed to the limitation of XRT
platform, which prohibits direct data movement from the FPGA kernel to host
buffers, resulting in a memory-copy overhead. Additionally, the XRT software
invocation latency is notably higher, as indicated in Figure 8.

[Gather.] [Reduce.]

Figure 13: Comparison of collective performance between ACCL+ TCP with XRT,
software MPI TCP and ACCL TCP.

##  6 Case Study: Deep Learning Recommendation Model

Table 2: Parameters of the target recommendation model. Tables | Concat Vec Len | FC Layers | Embed Size
---|---|---|---
100 | 3200 | (2048, 512, 256) | 50GB

Deep Learning Recommendation Models (DLRM) are widely used in personalized
recommendation systems for various scenarios [32, 18, 108]. The structure of a
DLRM includes two major components: memory-bound embedding layers and
computation-bound fully-connected (FC) layers. These models handle both dense
and sparse features, with the latter stored as embedding vectors in tables. In
inference, these vectors are accessed via indexes, resulting in multiple
random memory accesses. The retrieved embedding vectors are then concatenated
with dense features and passed through several FC layers to predict the click-
through rate, incurring heavy computational loads due to vector-matrix
multiplication.

DLRM has been a focal point for acceleration on GPUs and FPGAs, given that CPU
solutions are generally constrained by both random memory access and
computation [57, 38, 44]. GPU-based solutions [44, 37, 45, 51] mostly
accelerate the computation-bound FC layers to gain high throughput. However,
the large batch sizes required for efficient GPU computation, coupled with
random memory access, often lead to increased latency (tens of milliseconds).
FPGA-based techniques [50, 65] overcome the embedding lookup bottleneck by
distributing tables across memory banks and enabling parallel accesses,
leveraging high-bandwidth-memory (HBM) and on-chip memory (BRAM/URAM).
However, this approach is constrained by the requirement for embedding tables
to fit within a single FPGA’s memory (e.g., 16 GB HBM on AMD Alveo-U55C),
limiting the size of embedding layer. Additionally, the finite computational
resources on a single node pose restrictions on overall throughput for all FC
layers.

###  6.1 Distributed DLRM Inference

We aim to demonstrate that ACCL+ can facilitate distributing DLRM inference
across FPGAs to accommodate larger embedding layers, as in many large-scale
industrial settings, while at the same time achieving low latency and high
throughput. Table 2 shows the detailed configuration of such an industrial-
level recommendation model [51]. In such a use case, the embedding table does
not fit into a single FPGA HBM and therefore both the embedding lookup and the
computation are distributed across the network. This poses significant
challenges for performance, scalability, and networking.

Vector-Matrix Multiplications Decomposition for DLRM. The computation pattern
in DLRM inference involves a chain of three vector-matrix multiplications,
with the inference output vector computed as a sequence of operations
involving three matrices of FC layers ($FC1$, $FC2$, $FC3$) and a concatenated
embedding vector. The concept of distributed vector-matrix multiplication has
been extensively studied in literature [81] across CPUs and the same principle
can be applied to an FPGA cluster. One common approach is checkerboard block
decomposition of matrix, as shown in Figure 14. This method involves
partitioning the matrix in terms of both rows and columns, while partitioning
the vector ensures that processes associated with the same matrix row
partition share the same sub-embedding vector. Each process can then perform
partial computations, and the results belonging to the same row partition are
concatenated and subsequently aggregated.

Figure 14: Checkerboard block decomposition. Figure 15: Conceptual design of
partitioned DLRM, with $FC1$ decomposed and $FC2$, $FC3$ pipelined across
nodes.

Decomposed and Pipelined Distributed DLRM. The partitioning strategy for the
DLRM considers the need for balanced resource utilization, ensuring that the
overall throughput is not limited by any process among all nodes. Typically,
the computation load of the $FC1$ is significantly larger than subsequent
layers like $FC2$ and $FC3$. To accommodate this, resource distribution should
reflect the varying computation requirements. Additionally, for modern FPGAs
with HBM, the capacity requires a minimum number of FPGAs to effectively store
the embedding layer. A conceptual partitioned DLRM is illustrated in Figure
15. In this scenario, $FC1$ is decomposed and distributed across multiple
FPGAs using the checkerboard block decomposition, and $FC2$ and $FC3$ are
assigned to one FPGA each. The embedding tables are evenly distributed across
nodes 1-4, with partial vectors transmitted to nodes 5-8, leveraging the
network’s low latency. Similarly, partial results computed on nodes 1-4 are
forwarded to corresponding nodes 5-8, where an overall reduction of all
partial $FC1$ results is conducted. The aggregated $FC1$ results are then
forwarded to node 9 for $FC2$ computation, followed by node 10 for $FC3$
computation and final processing. Scaling resources according to the
computation distribution requirements of each layer could lead to improved
performance. For example, increasing the allocation of FPGAs for different
layers based on their computational load. Such partitioning method requires
diverse communication patterns by each node, such as send-only, send/recv, and
reduction and ACCL+ provides a unified design supporting all the communication
requirements of the DLRM through a standard interface. Additionally, for nodes
that do not require reduction, the streaming reduction plugins of ACCL+ can be
removed with a compilation flag, reducing resource consumption and improving
routing and timing. Furthermore, the cross-node simulation provided by ACCL+
can facilitate the development process, reducing hardware debugging cycles.

###  6.2 Use Case Evaluation

Distributed FC layer execution on CPU. We use an illustrative example to
demonstrate how ACCL+ can be utilized to improve the efficiency of distributed
work executing on CPU. In this use case, we distribute an FC layer workload
(matrix-vector multiplication) by partitioning the weight matrix column-wise,
with each rank receiving part of the input vector and a subset of the weight
matrix columns. The matrix-vector product is obtained by summing the partial
rank products using the reduce collective. For the implementation, we use the
highly optimized Eigen library [34], distributing it with both ACCL+ RDMA and
MPI RDMA. In this experiment we do not overlap computation and communication.

The overall execution time of the distributed FC layer is compared to its
single-node execution, as depicted in Figure 16, where top-of-bar numbers
indicate the speed-up compared to single-node execution. We observe that
utilizing ACCL+ instead of MPI for the reduction generally results in lower
matrix-vector computation time. This performance increase is most likely due
to reduced pressure on the CPU cache, as ACCL+ utilizes FPGA memory for all
intermediate reduction data structures. The figure indicates two instances of
super-linear scaling, attributed to the weight matrix partitions fitting into
either L2 (8 MB) or L3 (128 MB) caches on the CPU after partitioning, whereas
the entire matrix did not fit in caches during single-node execution. The
reduction time itself is higher in most cases due to an additional copy
required to move data between Eigen result buffers and ACCL+ buffers, which
can be eliminated with further optimization. Overall, distributing work with
ACCL+ achieves lower latency, especially for specific configurations of FC
size and number of ranks.

Figure 16: Speedup comparison and latency breakdown of distributed vector-
matrix multiplication.

Distributed FPGA-based DLRM. We distribute an industrial DLRM model, as in
Table 2, with ACCL+ on 10 U55C FPGAs following the same design principle as
shown in Figure 15. The communication between the embedding node and the
reduce slave node during each inference requires the transmission of a 3.2 KB
partial embedding vector and a 4 KB partial result. Additionally, the
reduction process spanning nodes 5 to 9 operates with a message size of 8 KB
per inference. The achieved operating frequency is 115 MHz. We utilize 32-bit
fixed-point precision for computation. All the application kernels utilize
streaming collective APIs to interact with ACCL+. ACCL+ DLRM is configured
with the TCP backend from XRT. Though the communication latency could be
further optimized with ACCL+ RDMA, it is not on the critical path of overall
latency as it is overlapped with computation. We also compare with CPU
implementation [51], where the DLRM inference is run on an Intel Xeon Platinum
8259CL CPU @ 2.50 GHz (32 vCPU, Cascade Lake, SIMD supported) and 256 GB DRAM
with TensorFlow Serving enabled. Figure 17(a) shows the latency comparison
between ACCL+ and the CPU baseline. We evaluate various batch sizes on the
CPU. On the other hand, ACCL+ works with streaming data without batching. The
hardware implementation demonstrates two orders of magnitude lower latency
compared to the CPU. This substantial latency reduction in the hardware
implementation is attributed to the parallel arithmetic units in hardware and
the significant latency introduced by random memory accesses. Figure 17(b)
shows the throughput comparison. ACCL+ shows more than an order of magnitude
higher throughput compared to CPU baseline.

[DLRM latency.] [DLRM Throughput.]

Figure 17: ACCL+ DLRM performance comparison.

###  6.3 Resource Consumption

The resource utilization of ACCL+ components and the overall utilization of
DLRM across nodes are summarized in Table 3. In the ACCL+ subsystem, the
majority of resources are allocated to POEs, with the TCP POE being the most
resource-intensive, while the CCLO engine utilizes comparatively fewer LUT and
memory resources. DLRM utilization is categorized by different layers, and the
presented utilization values represent the sum across multiple FPGAs after
decomposition. Note that DLRM $FC1$ utilization exceeds 100%, reflecting the
decomposition across 8 FPGAs (max 800%). The primary resource bottlenecks for
DLRM are URAM, serving as fast on-chip memory for storing small embedding
tables, and DSP, essential for matrix computations.

Table 3: Resource utilization. Component | CLB kLUT | DSP | BRAM | URAM
---|---|---|---|---
U55C(100%) | 1303 | 9024 | 2016 | 960
CCLO | 12.1% | 1.6% | 5.7% | 0
TCP POE | 19.8% | 0 | 10.6% | 0
RDMA POE | 13.0% | 0 | 5.3% | 0
DLRM FC1 | 278.1% | 580.1% | 186.3% | 798.3%
DLRM FC2 | 29.6% | 85.1% | 34.2% | 97.9%
DLRM FC3 | 6.2% | 16.1% | 2.2% | 20.8%

##  7 Conclusion

In this paper, we introduce ACCL+, an open-source FPGA-based collective
library designed for portability across diverse platforms and communication
protocols. ACCL+ offers flexibility in implementing collectives without the
need for FPGA re-synthesis and demonstrates high performance as collective
abstractions for FPGA-distributed applications and as a collective offload
engine for CPU applications. With ACCL+, there is potential for exploring new
possibilities by extending collectives across CPU and FPGA boundaries and
orchestrating them for a unified application.

## References

  * [1] Mohammadreza Alimadadi, Hieu Mai, Shenghsun Cho, Michael Ferdman, Peter Milder, and Shuai Mu.  Waverunner: An elegant approach to hardware acceleration of state machine replication.  In 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), pages 357–374, Boston, MA, April 2023. USENIX Association.
  * [2] Tobias Alonso, Lucian Petrica, Mario Ruiz, Jakoba Petri-Koenig, Yaman Umuroglu, Ioannis Stamelos, Elias Koromilas, Michaela Blott, and Kees Vissers.  Elastic-df: Scaling performance of dnn inference in fpga clouds through automatic partitioning.  ACM Trans. Reconfigurable Technol. Syst., 15(2), dec 2021.
  * [3] AMD.  Rccl’s documentation.  https://rccl.readthedocs.io/en/rocm-4.3.0/, 2021.
  * [4] Omer Arap, Lucas R.B. Brasilino, Ezra Kissel, Alexander Shroyer, and Martin Swany.  Offloading collective operations to programmable logic.  IEEE Micro, 37(5):52–60, 2017.
  * [5] Omer Arap, Geoffrey M. Brown, Bryce Himebaugh, and D. Martin Swany.  Software defined multicasting for mpi collective operation offloading with the netfpga.  In European Conference on Parallel Processing, 2014.
  * [6] Omer Arap and Martin Swany.  Offloading collective operations to programmable logic on a zynq cluster.  In 2016 IEEE 24th Annual Symposium on High-Performance Interconnects (HOTI), pages 76–83, 2016.
  * [7] ARM.  AMBA 4 AXI4-Stream Protocol Specification.  https://developer.arm.com/documentation/ihi0051/a/, 2010.
  * [8] Mikhail Asiatici, Nithin George, Kizheppatt Vipin, Suhaib A. Fahmy, and Paolo Ienne.  Virtualized execution runtime for fpga accelerators in the cloud.  IEEE Access, 5:1900–1910, 2017.
  * [9] Mohammadreza Bayatpour, Nick Sarkauskas, Hari Subramoni, Jahanzeb Maqbool Hashmi, and Dhabaleswar K. Panda.  Bluesmpi: Efficient mpi non-blocking alltoall offloading designs on modern bluefield smart nics.  In Bradford L. Chamberlain, Ana-Lucia Varbanescu, Hatem Ltaief, and Piotr Luszczek, editors, High Performance Computing, pages 18–37, Cham, 2021. Springer International Publishing.
  * [10] Saman Biookaghazadeh, Pravin Kumar Ravi, and Ming Zhao.  Toward multi-fpga acceleration of the neural networks.  J. Emerg. Technol. Comput. Syst., 17(2), apr 2021.
  * [11] Christophe Bobda, Joel Mandebi Mbongue, Paul Chow, Mohammad Ewais, Naif Tarafdar, Juan Camilo Vega, Ken Eguro, Dirk Koch, Suranga Handagala, Miriam Leeser, Martin Herbordt, Hafsah Shahzad, Peter Hofste, Burkhard Ringlein, Jakub Szefer, Ahmed Sanaullah, and Russell Tessier.  The future of fpga acceleration in datacenters and the cloud.  ACM Trans. Reconfigurable Technol. Syst., 15(3), feb 2022.
  * [12] Junehyuk Boo, Yujin Chung, Eunjin Baek, Seongmin Na, Changsu Kim, and Jangwoo Kim.  F4t: A fast and flexible fpga-based full-stack tcp acceleration framework.  In Proceedings of the 50th Annual International Symposium on Computer Architecture, ISCA ’23, New York, NY, USA, 2023. Association for Computing Machinery.
  * [13] Irina Calciu, M. Talha Imran, Ivan Puddu, Sanidhya Kashyap, Hasan Al Maruf, Onur Mutlu, and Aasheesh Kolli.  Rethinking software runtimes for disaggregated memory.  In Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’21, page 79–92, New York, NY, USA, 2021. Association for Computing Machinery.
  * [14] Adrian M. Caulfield, Eric S. Chung, Andrew Putnam, Hari Angepat, Jeremy Fowers, Michael Haselman, Stephen Heil, Matt Humphrey, Puneet Kaur, Joo-Young Kim, Daniel Lo, Todd Massengill, Kalin Ovtcharov, Michael Papamichael, Lisa Woods, Sitaram Lanka, Derek Chiou, and Doug Burger.  A cloud-scale acceleration architecture.  In 2016 49th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), 2016.
  * [15] Fei Chen, Yi Shan, Yu Zhang, Yu Wang, Hubertus Franke, Xiaotao Chang, and Kun Wang.  Enabling fpgas in the cloud.  In Proceedings of the 11th ACM Conference on Computing Frontiers, CF ’14, New York, NY, USA, 2014. Association for Computing Machinery.
  * [16] Eric Chung, Jeremy Fowers, Kalin Ovtcharov, Michael Papamichael, Adrian Caulfield, Todd Massengill, Ming Liu, Daniel Lo, Shlomi Alkalay, Michael Haselman, Maleen Abeydeera, Logan Adams, Hari Angepat, Christian Boehn, Derek Chiou, Oren Firestein, Alessandro Forin, Kang Su Gatlin, Mahdi Ghandi, Stephen Heil, Kyle Holohan, Ahmad El Husseini, Tamas Juhasz, Kara Kagi, Ratna K. Kovvuri, Sitaram Lanka, Friedel van Megen, Dima Mukhortov, Prerak Patel, Brandon Perez, Amanda Rapsang, Steven Reinhardt, Bita Rouhani, Adam Sapek, Raja Seera, Sangeetha Shekar, Balaji Sridharan, Gabriel Weisz, Lisa Woods, Phillip Yi Xiao, Dan Zhang, Ritchie Zhao, and Doug Burger.  Serving DNNs in Real Time at Datacenter Scale with Project Brainwave.  IEEE Micro, 2018.
  * [17] Nicholas Contini, Bharath Ramesh, Kaushik Kandadi Suresh, Tu Tran, Ben Michalowicz, Mustafa Abduljabbar, Hari Subramoni, and Dhabaleswar Panda.  Enabling reconfigurable hpc through mpi-based inter-fpga communication.  In Proceedings of the 37th International Conference on Supercomputing, ICS ’23, page 477–487, New York, NY, USA, 2023. Association for Computing Machinery.
  * [18] Paul Covington, Jay Adams, and Emre Sargin.  Deep neural networks for youtube recommendations.  In Proceedings of the 10th ACM conference on recommender systems, 2016.
  * [19] Abhishek Das, David Nguyen, Joseph Zambreno, Gokhan Memik, and Alok Choudhary.  An fpga-based network intrusion detection architecture.  IEEE Transactions on Information Forensics and Security, 3(1):118–132, 2008.
  * [20] Juan Miguel de Haro, Rubén Cano, Carlos Álvarez, Daniel Jiménez-González, Xavier Martorell, Eduard Ayguadé, Jesús Labarta, Francois Abel, Burkhard Ringlein, and Beat Weiss.  Ompss@cloudfpga: An fpga task-based programming model with message passing.  In 2022 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 828–838, 2022.
  * [21] Tiziano De Matteis, Johannes de Fine Licht, Jakub Beránek, and Torsten Hoefler.  Streaming Message Interface: High-Performance Distributed Memory Programming on Reconfigurable Hardware.  In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’19. Association for Computing Machinery, 2019.
  * [22] Li Ding, Ping Kang, Wenbo Yin, and Linli Wang.  Hardware TCP Offload Engine based on 10-Gbps Ethernet for low-latency network communication.  In 2016 International Conference on Field-Programmable Technology (FPT), 2016.
  * [23] Haggai Eran, Maxim Fudim, Gabi Malka, Gal Shalom, Noam Cohen, Amit Hermony, Dotan Levi, Liran Liss, and Mark Silberstein.  Flexdriver: A network driver for your accelerator.  In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’22, page 1115–1129, New York, NY, USA, 2022. Association for Computing Machinery.
  * [24] Haggai Eran, Lior Zeno, Maroun Tork, Gabi Malka, and Mark Silberstein.  NICA: An infrastructure for inline acceleration of network applications.  In 2019 USENIX Annual Technical Conference (USENIX ATC 19), pages 345–362, Renton, WA, July 2019. USENIX Association.
  * [25] Nariman Eskandari, Naif Tarafdar, Daniel Ly-Ma, and Paul Chow.  A Modular Heterogeneous Stack for Deploying FPGAs and CPUs in the Data Center.  FPGA ’19. Association for Computing Machinery, 2019.
  * [26] Yuanwei Fang, Chen Zou, and Andrew A. Chien.  Accelerating Raw Data Analysis with the ACCORDA Software and Hardware Architecture.  Proc. VLDB Endow., page 1568–1582, jul 2019.
  * [27] Clément Farabet, Cyril Poulet, and Yann LeCun.  An fpga-based stream processor for embedded real-time vision with convolutional networks.  In 2009 IEEE 12th International Conference on Computer Vision Workshops, ICCV Workshops, pages 878–885, 2009.
  * [28] Kermin Fleming, Hsin-Jung Yang, Michael Adler, and Joel Emer.  The leap fpga operating system.  In 2014 24th International Conference on Field Programmable Logic and Applications (FPL), pages 1–8, 2014.
  * [29] Kermin Elliott Fleming, Michael Adler, Michael Pellauer, Angshuman Parashar, Arvind Mithal, and Joel Emer.  Leveraging latency-insensitivity to ease multiple fpga design.  In Proceedings of the ACM/SIGDA International Symposium on Field Programmable Gate Arrays, FPGA ’12, page 175–184, New York, NY, USA, 2012. Association for Computing Machinery.
  * [30] Alex Forencich, Alex C. Snoeren, George Porter, and George Papen.  Corundum: An Open-Source 100-Gbps Nic.  In 2020 IEEE 28th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), 2020.
  * [31] Shanyuan Gao, Andrew G. Schmidt, and Ron Sass.  Hardware implementation of MPI_Barrier on an FPGA cluster.  In 2009 International Conference on Field Programmable Logic and Applications, pages 12–17, 2009.
  * [32] Carlos A Gomez-Uribe and Neil Hunt.  The netflix recommender system: Algorithms, business value, and innovation.  ACM Transactions on Management Information Systems (TMIS), 2015\.
  * [33] Richard L. Graham, Timothy S. Woodall, and Jeffrey M. Squyres.  Open mpi: A flexible high performance mpi.  In Proceedings of the 6th International Conference on Parallel Processing and Applied Mathematics, PPAM’05, page 228–239, Berlin, Heidelberg, 2005. Springer-Verlag.
  * [34] Gaël Guennebaud, Benoit Jacob, et al.  Eigen.  URl: http://eigen. tuxfamily. org, 3(1), 2010.
  * [35] Anqi Guo, Tong Geng, Yongan Zhang, Pouya Haghi, Chunshu Wu, Cheng Tan, Yingyan Lin, Ang Li, and Martin Herbordt.  A framework for neural network inference on fpga-centric smartnics.  In 2022 32nd International Conference on Field-Programmable Logic and Applications (FPL), pages 01–08, 2022.
  * [36] Zhiyuan Guo, Yizhou Shan, Xuhao Luo, Yutong Huang, and Yiying Zhang.  Clio: A hardware-software co-designed disaggregated memory system.  In Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’22, page 417–433, New York, NY, USA, 2022. Association for Computing Machinery.
  * [37] Udit Gupta, Samuel Hsia, Vikram Saraph, Xiaodong Wang, Brandon Reagen, Gu-Yeon Wei, Hsien-Hsin S Lee, David Brooks, and Carole-Jean Wu.  DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inference.  In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), 2020.
  * [38] Udit Gupta, Carole-Jean Wu, Xiaodong Wang, Maxim Naumov, Brandon Reagen, David Brooks, Bradford Cottel, Kim Hazelwood, Mark Hempstead, Bill Jia, et al.  The architectural implications of Facebook’s DNN-based personalized recommendation.  In 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA), 2020.
  * [39] Pouya Haghi, Anqi Guo, Tong Geng, Justin Broaddus, Derek Schafer, Anthony Skjellum, and Martin Herbordt.  A reconfigurable compute-in-the-network fpga assistant for high-level collective support with distributed matrix multiply case study.  In 2020 International Conference on Field-Programmable Technology (ICFPT), pages 159–164, 2020.
  * [40] Zhenhao He, Dario Korolija, and Gustavo Alonso.  EasyNet: 100 Gbps Network for HLS.  In 2021 31st International Conference on Field-Programmable Logic and Applications (FPL), 2021.
  * [41] Zhenhao He, Daniele Parravicini, Lucian Petrica, Kenneth O’Brien, Gustavo Alonso, and Michaela Blott.  ACCL: FPGA-Accelerated Collectives over 100 Gbps TCP-IP.  In 2021 IEEE/ACM International Workshop on Heterogeneous High-performance Reconfigurable Computing (H2RC), pages 33–43, 2021.
  * [42] Fernando Luis Herrmann, Guilherme Perin, Josue Paulo Jose de Freitas, Rafael Bertagnolli, and Joao Baptista dos Santos Martins.  A gigabit udp/ip network stack in fpga.  In 2009 16th IEEE International Conference on Electronics, Circuits and Systems - (ICECS 2009), pages 836–839, 2009.
  * [43] Pieter Hintjens.  ZeroMQ: messaging for many applications.  " O’Reilly Media, Inc.", 2013.
  * [44] Samuel Hsia, Udit Gupta, Mark Wilkening, Carole-Jean Wu, Gu-Yeon Wei, and David Brooks.  Cross-stack workload characterization of deep recommendation systems.  In 2020 IEEE International Symposium on Workload Characterization (IISWC), 2020.
  * [45] Ranggi Hwang, Taehun Kim, Youngeun Kwon, and Minsoo Rhu.  Centaur: A chiplet-based, hybrid sparse-dense accelerator for personalized recommendations.  In 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA), 2020.
  * [46] Intel.  Intel FPGA Add-on for oneAPI Base Toolkit.  https://www.intel.com/content/www/us/en/developer/tools/oneapi/fpga.html.
  * [47] Intel.  Intel Quartus Prime Standard Edition User Guide: Getting Started.  https://www.intel.com/content/www/us/en/programmable/documentation/yoq1529444104707.html.
  * [48] Houxiang Ji, Mark Mansi, Yan Sun, Yifan Yuan, Jinghan Huang, Reese Kuper, Michael M. Swift, and Nam Sung Kim.  STYX: Exploiting SmartNIC capability to reduce datacenter memory tax.  In 2023 USENIX Annual Technical Conference (USENIX ATC 23), pages 619–633, Boston, MA, July 2023. USENIX Association.
  * [49] Yong Ji and Qing-Sheng Hu.  40Gbps multi-connection TCP/IP offload engine.  In 2011 International Conference on Wireless Communications and Signal Processing (WCSP), pages 1–5, 2011.
  * [50] Wenqi Jiang, Zhenhao He, Shuai Zhang, Thomas B Preußer, Kai Zeng, Liang Feng, Jiansong Zhang, Tongxuan Liu, Yong Li, Jingren Zhou, et al.  MicroRec: Efficient Recommendation Inference by Hardware and Data Structure Solutions.  In 2021 4th Conference on Machine Learning and Systems (MLSys), 2021\.
  * [51] Wenqi Jiang, Zhenhao He, Shuai Zhang, Kai Zeng, Liang Feng, Jiansong Zhang, Tongxuan Liu, Yong Li, Jingren Zhou, Ce Zhang, and Gustavo Alonso.  FleetRec: Large-Scale Recommendation Inference on Hybrid GPU-FPGA Clusters.  KDD ’21, New York, NY, USA, 2021. Association for Computing Machinery.
  * [52] Vinod Kathail.  Xilinx vitis unified software platform.  In The 2020 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, pages 173–174, 2020.
  * [53] Moazin Khatti, Xingyu Tian, Yuze Chi, Licheng Guo, Jason Cong, and Zhenman Fang.  Pasta: Programming and automation support for scalable task-parallel hls programs on modern multi-die fpgas.  In 2023 IEEE 31st Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM), pages 12–22, 2023.
  * [54] Ahmed Khawaja, Joshua Landgraf, Rohith Prakash, Michael Wei, Eric Schkufza, and Christopher J. Rossbach.  Sharing, protection, and compatibility for reconfigurable fabric with AmorphOS.  In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18), pages 107–127, Carlsbad, CA, October 2018. USENIX Association.
  * [55] Dario Korolija, Dimitrios Koutsoukos, Kimberly Keeton, Konstantin Taranov, Dejan Milojičić, and Gustavo Alonso.  Farview: Disaggregated memory with operator off-loading for database engines, 2021.
  * [56] Dario Korolija, Timothy Roscoe, and Gustavo Alonso.  Do OS abstractions make sense on FPGAs?  In 14th USENIX Symposium on Operating Systems Design and Implementation (OSDI 20), pages 991–1010. USENIX Association, November 2020\.
  * [57] Youngeun Kwon, Yunjae Lee, and Minsoo Rhu.  Tensordimm: A practical near-memory processing architecture for embeddings and tensor operations in deep learning.  In Proceedings of the 52nd Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), 2019.
  * [58] Joshua Lant, Emmanouil Skordalakis, Kyriakos Paraskevas, William B. Toms, Mikel Luján, and John Goodacre.  Diad – distributed acceleration for datacenter fpgas.  In 2023 33rd International Conference on Field-Programmable Logic and Applications (FPL), pages 166–173, 2023.
  * [59] Bojie Li, Zhenyuan Ruan, Wencong Xiao, Yuanwei Lu, Yongqiang Xiong, Andrew Putnam, Enhong Chen, and Lintao Zhang.  KV-Direct: High-Performance In-Memory Key-Value Store with Programmable NIC.  In Proceedings of the 26th Symposium on Operating Systems Principles, SOSP ’17, 2017.
  * [60] Junnan Li, Zhigang Sun, Jinli Yan, Xiangrui Yang, Yue Jiang, and Wei Quan.  Drawerpipe: A reconfigurable pipeline for network processing on fpga-based smartnic.  Electronics, 9(1), 2020.
  * [61] Jiaxin Lin, Kiran Patel, Brent E. Stephens, Anirudh Sivaraman, and Aditya Akella.  PANIC: A High-Performance Programmable NIC for Multi-Tenant Networks.  In Proceedings of the 14th USENIX Conference on Operating Systems Design and Implementation, OSDI’20. USENIX Association, 2020.
  * [62] Junyi Liu, Aleksandar Dragojevic, Shane Flemming, Antonios Katsarakis, Dario Korolija, Igor Zablotchi, Ho cheung Ng, Anuj Kalia, and Miguel Castro.  Honeycomb: ordered key-value store acceleration on an fpga-based smartnic, 2023.
  * [63] Yuanwei Lu, Guo Chen, Bojie Li, Kun Tan, Yongqiang Xiong, Peng Cheng, Jiansong Zhang, Enhong Chen, and Thomas Moscibroda.  Multi-Path transport for RDMA in datacenters.  In 15th USENIX Symposium on Networked Systems Design and Implementation (NSDI 18), pages 357–371, Renton, WA, April 2018. USENIX Association.
  * [64] Jiacheng Ma, Gefei Zuo, Kevin Loughlin, Xiaohe Cheng, Yanqiang Liu, Abel Mulugeta Eneyew, Zhengwei Qi, and Baris Kasikci.  A hypervisor for shared-memory fpga platforms.  In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’20, page 827–844, New York, NY, USA, 2020. Association for Computing Machinery.
  * [65] Chinmay Mahajan, Ashwin Krishnan, Manoj Nambiar, and Rekha Singhal.  Hetero-rec: Optimal deployment of embeddings for high-speed recommendations.  In Proceedings of the Second International Conference on AI-ML Systems, AIMLSystems ’22, New York, NY, USA, 2023. Association for Computing Machinery.
  * [66] Joel Mbongue, Festus Hategekimana, Danielle Tchuinkou Kwadjo, David Andrews, and Christophe Bobda.  Fpgavirt: A novel virtualization framework for fpgas in the cloud.  In 2018 IEEE 11th International Conference on Cloud Computing (CLOUD), pages 862–865, 2018.
  * [67] Kenji Mizutani, Hiroshi Yamaguchi, Yutaka Urino, and Michihiro Koibuchi.  Optweb: A lightweight fully connected inter-fpga network for efficient collectives.  IEEE Transactions on Computers, 70(6):849–862, 2021.
  * [68] NVDIA.  NVIDIA Collective Communications Library (NCCL).  https://docs.nvidia.com/deeplearning/nccl/index.html, 2021.
  * [69] Neal Oliver, Rahul R. Sharma, Stephen Chang, Bhushan Chitlur, Elkin Garcia, Joseph Grecco, Aaron Grier, Nelson Ijih, Yaping Liu, Pratik Marolia, Henry Mitchel, Suchit Subhaschandra, Arthur Sheiman, Tim Whisonant, and Prabhat Gupta.  A reconfigurable computing system based on a cache-coherent fabric.  In 2011 International Conference on Reconfigurable Computing and FPGAs, pages 80–85, 2011.
  * [70] Dhabaleswar Kumar Panda, Hari Subramoni, Ching-Hsiang Chu, and Mohammadreza Bayatpour.  The mvapich project: Transforming research into high-performance mpi library for hpc community.  Journal of Computational Science, 52:101208, 2021.  Case Studies in Translational Computer Science.
  * [71] Jongse Park, Hardik Sharma, Divya Mahajan, Joon Kyung Kim, Preston Olds, and Hadi Esmaeilzadeh.  Scale-out acceleration for machine learning.  In Proceedings of the 50th Annual IEEE/ACM International Symposium on Microarchitecture, MICRO-50 ’17, page 367–381, New York, NY, USA, 2017. Association for Computing Machinery.
  * [72] Sreeram Potluri, Hao Wang, Devendar Bureddy, Ashish Kumar Singh, Carlos Rosales, and Dhabaleswar K Panda.  Optimizing MPI communication on multi-GPU systems using CUDA inter-process communication.  In 2012 IEEE 26th International Parallel and Distributed Processing Symposium Workshops & PhD Forum, pages 1848–1857. IEEE, 2012.
  * [73] Andrew Putnam, Adrian Caulfield, Eric Chung, Derek Chiou, Kypros Constantinides, John Demme, Hadi Esmaeilzadeh, Jeremy Fowers, Jan Gray, Michael Haselman, Scott Hauck, Stephen Heil, Amir Hormati, Joo-Young Kim, Sitaram Lanka, Eric Peterson, Aaron Smith, Jason Thong, Phillip Yi Xiao, Doug Burger, Jim Larus, Gopi Prashanth Gopal, and Simon Pope.  A Reconfigurable Fabric for Accelerating Large-Scale Datacenter Services.  In Proceeding of the 41st Annual International Symposium on Computer Architecuture (ISCA). IEEE Press, June 2014.
  * [74] Masudul Hassan Quraishi, Erfan Bank Tavakoli, and Fengbo Ren.  A survey of system architectures and techniques for fpga virtualization.  IEEE Transactions on Parallel and Distributed Systems, 32(9):2216–2230, 2021.
  * [75] Burkhard Ringlein, Francois Abel, Alexander Ditter, Beat Weiss, Christoph Hagleitner, and Dietmar Fey.  ZRLMPI: A Unified Programming Model for Reconfigurable Heterogeneous Computing Clusters.  In 2020 IEEE 28th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM).
  * [76] Burkhard Ringlein, Francois Abel, Alexander Ditter, Beat Weiss, Christoph Hagleitner, and Dietmar Fey.  Programming Reconfigurable Heterogeneous Computing Clusters Using MPI With Transpilation.  In 2020 IEEE/ACM International Workshop on Heterogeneous High-performance Reconfigurable Computing (H2RC), 2020.
  * [77] Burkhard Ringlein, François Abel, Dionysios Diamantopoulos, Beat Weiss, Christoph Hagleitner, Marc Reichenbach, and Dietmar Fey.  A case for function-as-a-service with disaggregated fpgas.  In 2021 IEEE 14th International Conference on Cloud Computing (CLOUD), pages 333–344, 2021.
  * [78] Mario Ruiz, David Sidler, Gustavo Sutter, Gustavo Alonso, and Sergio López-Buedo.  Limago: An FPGA-Based Open-Source 100 GbE TCP/IP Stack.  In 2019 29th International Conference on Field Programmable Logic and Applications (FPL), pages 286–292, 2019.
  * [79] Manuel Saldaña, Arun Patel, Christopher Madill, Daniel Nunes, Danyao Wang, Paul Chow, Ralph Wittig, Henry Styles, and Andrew Putnam.  MPI as a Programming Model for High-Performance Reconfigurable Computers.  ACM Trans. Reconfigurable Technol. Syst., 3(4), 2010.
  * [80] Manuel Saldana and Paul Chow.  TMD-MPI: An MPI Implementation for Multiple Processors Across Multiple FPGAs.  In 2006 International Conference on Field Programmable Logic and Applications, pages 1–6, 2006.
  * [81] Martin D. Schatz, Robert A. van de Geijn, and Jack Poulson.  Parallel Matrix Multiplication: A Systematic Journey.  SIAM Journal on Scientific Computing, 2016.
  * [82] Niklas Schelten, Fritjof Steinert, Anton Schulte, and Benno Stabernack.  A high-throughput, resource-efficient implementation of the rocev2 remote dma protocol for network-attached hardware accelerators.  In 2020 International Conference on Field-Programmable Technology (ICFPT), pages 241–249, 2020.
  * [83] Junnan Shan, Mihai T. Lazarescu, Jordi Cortadella, Luciano Lavagno, and Mario R. Casu.  Cnn-on-aws: Efficient allocation of multikernel applications on multi-fpga platforms.  IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 40(2):301–314, 2021.
  * [84] Yizhou Shan, Will Lin, Zhiyuan Guo, and Yiying Zhang.  Towards a fully disaggregated and programmable data center.  In Proceedings of the 13th ACM SIGOPS Asia-Pacific Workshop on Systems, APSys ’22, page 18–28, New York, NY, USA, 2022. Association for Computing Machinery.
  * [85] David Sidler, Zsolt István, and Gustavo Alonso.  Low-latency TCP/IP stack for data center applications.  In 2016 26th International Conference on Field Programmable Logic and Applications (FPL), 2016.
  * [86] David Sidler, Zeke Wang, Monica Chiosa, Amit Kulkarni, and Gustavo Alonso.  StRoM: Smart Remote Memory.  In Proceedings of the Fifteenth European Conference on Computer Systems, EuroSys ’20. Association for Computing Machinery, 2020.
  * [87] Kaushik Kandadi Suresh, Benjamin Michalowicz, Bharath Ramesh, Nick Contini, Jinghan Yao, Shulei Xu, Aamir Shafi, Hari Subramoni, and Dhabaleswar Panda.  A novel framework for efficient offloading of communication operations to bluefield smartnics.  In 2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS), pages 123–133, 2023.
  * [88] Naif Tarafdar, Nariman Eskandari, Varun Sharma, Charles Lo, and Paul Chow.  Galapagos: A Full Stack Approach to FPGA Integration in the Cloud.  IEEE Micro, 2018.
  * [89] Maroun Tork, Lina Maudlej, and Mark Silberstein.  Lynx: A smartnic-driven accelerator-centric architecture for network servers.  In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’20, page 117–131, New York, NY, USA, 2020. Association for Computing Machinery.
  * [90] Anuj Vaishnav, Khoa Dang Pham, and Dirk Koch.  A survey on fpga virtualization.  In 2018 28th International Conference on Field Programmable Logic and Applications (FPL), pages 131–1317, 2018.
  * [91] Anuj Vaishnav, Khoa Dang Pham, Dirk Koch, and James Garside.  Resource elastic virtualization for fpgas using opencl.  In 2018 28th International Conference on Field Programmable Logic and Applications (FPL), pages 111–1117, 2018.
  * [92] John Paul Walters, Xiandong Meng, Vipin Chaudhary, Tim Oliver, Leow Yuan Yeow, Bertil Schmidt, Darran Nathan, and Joseph Landman.  MPI-HMMER-boost: distributed FPGA acceleration.  The Journal of VLSI Signal Processing Systems for Signal, Image, and Video Technology, 48(3):223–238, 2007.
  * [93] Hao Wang, Sreeram Potluri, Devendar Bureddy, Carlos Rosales, and Dhabaleswar K. Panda.  GPU-Aware MPI on RDMA-Enabled Clusters: Design, Implementation and Evaluation.  IEEE Transactions on Parallel and Distributed Systems, 25(10):2595–2605, 2014.
  * [94] Zeke Wang, Hongjing Huang, Jie Zhang, Fei Wu, and Gustavo Alonso.  FpgaNIC: An FPGA-based versatile 100gb SmartNIC for GPUs.  In 2022 USENIX Annual Technical Conference (USENIX ATC 22), pages 967–986, Carlsbad, CA, July 2022. USENIX Association.
  * [95] Jagath Weerasinghe, Francois Abel, Christoph Hagleitner, and Andreas Herkersdorf.  Enabling fpgas in hyperscale data centers.  In 2015 IEEE 12th Intl Conf on Ubiquitous Intelligence and Computing and 2015 IEEE 12th Intl Conf on Autonomic and Trusted Computing and 2015 IEEE 15th Intl Conf on Scalable Computing and Communications and Its Associated Workshops (UIC-ATC-ScalCom), pages 1078–1086, 2015.
  * [96] Jagath Weerasinghe, Raphael Polig, Francois Abel, and Christoph Hagleitner.  Network-attached FPGAs for data center applications.  In 2016 International Conference on Field-Programmable Technology (FPT), 2016.
  * [97] Gabriel Weisz, Joseph Melber, Yu Wang, Kermin Fleming, Eriko Nurvitadhi, and James C. Hoe.  A study of pointer-chasing performance on shared-memory processor-fpga systems.  In Proceedings of the 2016 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays, FPGA ’16, page 264–273, New York, NY, USA, 2016\. Association for Computing Machinery.
  * [98] AMD Xilinx.  XUP Vitis Network Example (VNx).  https://github.com/Xilinx/xup_vitis_network_example.
  * [99] AMD Xilinx.  Quick Start Guide:MicroBlaze Soft Processor for Vitis 2019.2.  https://www.xilinx.com/support/documentation/quick_start/microblaze-quick-start-guide-with-vitis.pdf, 2019\.
  * [100] AMD Xilinx.  Axi datamover v5.1 logicore ip product guide.  https://docs.xilinx.com/r/en-US/pg022_axi_datamover/AXI-DataMover-v5.1-LogiCORE-IP-Product-Guide, 2023\.
  * [101] AMD Xilinx.  Dma for pci express (pcie) subsystem.  https://www.xilinx.com/products/intellectual-property/pcie-dma.html, 2023\.
  * [102] Chao-Tung Yang, Chih-Lin Huang, and Cheng-Fang Lin.  Hybrid CUDA, OpenMP, and MPI parallel programming on multicore GPU clusters.  Computer Physics Communications, 182(1):266–269, 2011.
  * [103] Yue Zha and Jing Li.  Virtualizing fpgas in the cloud.  In Proceedings of the Twenty-Fifth International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’20, page 845–858, New York, NY, USA, 2020. Association for Computing Machinery.
  * [104] Yue Zha and Jing Li.  Hetero-vital: A virtualization stack for heterogeneous fpga clusters.  In 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA), pages 470–483, 2021.
  * [105] Yue Zha and Jing Li.  When application-specific isa meets fpgas: A multi-layer virtualization framework for heterogeneous cloud fpgas.  In Proceedings of the 26th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, ASPLOS ’21, page 123–134, New York, NY, USA, 2021. Association for Computing Machinery.
  * [106] Jie Zhang, Hongjing Huang, Lingjun Zhu, Shu Ma, Dazhong Rong, Yijun Hou, Mo Sun, Chaojie Gu, Peng Cheng, Chao Shi, and Zeke Wang.  Smartds: Middle-tier-centric smartnic enabling application-aware message split for disaggregated block storage.  In Proceedings of the 50th Annual International Symposium on Computer Architecture, ISCA ’23, New York, NY, USA, 2023. Association for Computing Machinery.
  * [107] Wentai Zhang, Jiaxi Zhang, Minghua Shen, Guojie Luo, and Nong Xiao.  An efficient mapping approach to large-scale dnns on multi-fpga architectures.  In 2019 Design, Automation & Test in Europe Conference & Exhibition (DATE), pages 1241–1244, 2019.
  * [108] Guorui Zhou, Xiaoqiang Zhu, Chenru Song, Ying Fan, Han Zhu, Xiao Ma, Yanghui Yan, Junqi Jin, Han Li, and Kun Gai.  Deep interest network for click-through rate prediction.  In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2018.
  * [109] Yu Zhu, Zhenhao He, Wenqi Jiang, Kai Zeng, Jingren Zhou, and Gustavo Alonso.  Distributed Recommendation Inference on FPGA Clusters.  In 2021 31th International Conference on Field-Programmable Logic and Applications (FPL). IEEE, 2021.
  * [110] Noa Zilberman, Yury Audzevich, G. Adam Covington, and Andrew W. Moore.  NetFPGA SUME: Toward 100 Gbps as Research Commodity.  IEEE Micro, 2014.