# MARS: Exploiting Multi-Level Parallelism for DNN Workloads on Adaptive
Multi-Accelerator Systems

Guan Shen1, Jieru Zhao1∗, Zeke Wang2, Zhe Lin3, Wenchao Ding4, Chentao Wu1,
Quan Chen1, Minyi Guo1∗ 1Shanghai Jiao Tong University, 2Zhejiang University,
3Sun Yat-sen University, 4Fudan University
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>{wuct, chen-quan<EMAIL_ADDRESS>
###### Abstract

Along with the fast evolution of deep neural networks, the hardware system is
also developing rapidly. As a promising solution achieving high scalability
and low manufacturing cost, multi-accelerator systems widely exist in data
centers, cloud platforms, and SoCs. Thus, a challenging problem arises in
multi-accelerator systems: selecting a proper combination of accelerators from
available designs and searching for efficient DNN mapping strategies. To this
end, we propose MARS, a novel mapping framework that can perform computation-
aware accelerator selection, and apply communication-aware sharding strategies
to maximize parallelism. Experimental results show that MARS can achieve 32.2%
latency reduction on average for typical DNN workloads compared to the
baseline, and 59.4% latency reduction on heterogeneous models compared to the
corresponding state-of-the-art method.

**footnotetext: Jieru Zhao and Minyi Guo are corresponding authors.

##  I Introduction

DNN models have achieved cutting-edge accuracy for a wide range of tasks in
various areas like computer vision[1], natural language processing[2], and
recommendation systems[3]. Simultaneously, the power of DNNs imposes a
significant computational and memory burden on traditional hardware systems.
For example, one of the largest language models, GPT-3, involves about $175$
billion parameters and $3.14\times 10^{8}$ PFLOPs, which require a large
amount of computational and memory resources. More critically, the completion
of model training does not mean that the work is done. The trained models
should be deployed to target platforms like cloud servers and edge devices for
inference. These scenarios tend to be more cost-sensitive, compared to
training systems.

Multi-accelerator systems have begun to attract the attention of researchers
and industry. As Moore’s Law fades out of effect, semiconductor manufacturers
can no longer exponentially increase the computational resources in chips. In
other words, manufacturing one large chip could be expensive. However, multi-
accelerator systems can achieve the same performance at a lower manufacturing
cost. For example, Microsoft[4] has applied customized accelerators based on
FPGA in the data center network. Amazon Web Services has also provided FPGA
clusters (F1 instances) equipped with up to 8 FPGAs. NVIDIA’s Jetson AGX
Xavier is an SoC design integrating an ARM CPU, a Volta GPU, 2 NVIDIA Deep
Learning Accelerators (NVDLA), and multimedia accelerators.

These systems can achieve high performance and scalability with the
interconnection between accelerators enabling collaborations. But this still
requires much engineering effort and expert knowledge because of a large
design space. First, for a certain layer in a given DNN workload, various
accelerator designs may demonstrate performance gaps due to different
computation patterns. In an adaptive multi-accelerator system, the design of
accelerators can be configured, which is natural for FPGA platforms and SoCs
at the design phase. Thus, one needs to choose a proper combination of
accelerator designs to accommodate these layers in the workload and achieve
the best overall performance. Second, to fully utilize computational resources
and relieve the memory burden, parallelism strategies, like data parallelism
and model parallelism, should be applied to partition the workload into
accelerators with the awareness of communication. Note that multiple
parallelism strategies can work together and form a multi-level decision
problem. Due to these factors, an effective mapping framework is urgently
required to explore the large design space.

There have been some previous works focusing on the mapping algorithm on
multi-accelerator systems. Zhang et al.[5] propose a mapping strategy based on
dynamic programming to partition the DNNs into different FPGAs. Herald[6]
focuses on mapping multi-DNN workloads onto the heterogeneous dataflow
accelerators. But it lacks communication awareness during the mapping. H2H[7]
provides mapping strategies with computation and communication awareness for
heterogeneous accelerator systems. All of them fail to perform intra-layer
parallelism to fully utilize computational resources.

Figure 1: The architecture of an F1.16xlarge instance on AWS

To this end, we propose MARS, a mapping framework that can exploit multi-level
parallelism on adaptive multi-accelerator systems with high scalability. Our
main contributions are summarized as follows:

  * •

We give a detailed system formulation together with the design space to cover
current multi-accelerator systems and DNN workloads.

  * •

We generalize parallelism strategies for multi-accelerator systems. We develop
a general representation to describe workload partitioning, generate multi-
level parallelism strategies, and integrate them for evaluation.

  * •

We develop a mapping algorithm based on a two-level genetic algorithm with
heuristics, which performs efficient design space exploration and finds a
proper choice of accelerators, workload allocation, and parallelism
strategies.

  * •

MARS achieves 32.2% latency reduction compared to the baseline for typical DNN
models and outperforms the existing SOTA with 59.4% latency reduction for
mapping heterogeneous models to heterogeneous accelerators.

##  II Background and Motivation

###  II-A Adaptive Multi-Accelerator Systems

As more and more multi-accelerator systems are developed, adaptive multi-
accelerator systems gain popularity for their flexibility. They can easily
adapt to the coming workload by configuring accelerators in the system to
maximize resource utilization. They can also work as a prototype validation
platform to guide the design of other multi-accelerator systems. Fig. 1 shows
the architecture of an EC2 F1.16xlarge instance from AWS, a representative
adaptive multi-accelerator system. Each instance has a host machine with eight
Xilinx UltraScale+ FPGAs. The host-side applications can access/transmit data
from/to FPGA’s local DRAM via the PCI-e bus. The programmable logic (PL) of
each FPGA can be configured independently through the FPGA image specified by
the user. Moreover, eight FPGAs in the system are separated into two groups,
as illustrated in Fig. 1. FPGAs from the same group can communicate with each
other without the interference of the host, which could reduce the
communication latency[8]. This asymmetrical communication pattern pose new
challenges. Users need to rearrange workloads to leverage the low-latency
communication feature. Note that though we take the F1 instance as an example,
this asymmetrical communication pattern widely exists in multi-accelerator
systems in industry. Microsoft[4] organizes accelerators through switches from
the data center network in a hierarchical manner. Accelerators in the same
rack can communicate much faster compared to the pairs from different racks.

###  II-B DNN Workloads on Multi-Accelerator Systems

DNN workloads can be represented as a computation graph, which is a directed
acyclic graph consisting of layers. For a CNN model, there are various layers
including convolution, batch normalization, pooling, activation and fully-
connected layers. Typically, convolution layers occupy most of the computation
resources. The convolution layer can be represented by a six-level nested
loop. Most CNN accelerators perform loop transformation, such as loop
interchange and loop tiling to map a given convolution layer to PEs
(processing elements) in hardware architectures. Accelerator designers can
surely find an optimal set of parameters to maximize performance for a
specific convolution layer. However, the heterogeneity of CNN models makes it
challenging to achieve the global optimum. When an image is fed into a CNN
model, it has a high resolution (e.g. $H\times W=224\times 224$) and low
channel width (e.g. $C=3$). As the network deepens, the feature map resolution
gradually decreases with increasing channel width. The heterogeneity of
feature map shapes reflects the variation of loop boundaries in the nested
loop. Existing accelerators can only achieve high resource utilization for
some layers.

Multi-accelerator systems make it possible to maintain a high utilization rate
through the whole network inference. We can deploy different layers onto
accelerators that are suitable for corresponding computation patterns. Besides
the computation heterogeneity, memory burden is another issue that needs to be
addressed. As the depth and width of the network enlarge rapidly, the off-chip
DRAM attached to each accelerator may not have sufficient space to buffer all
the parameters and intermediate results during the inference. Frequent access
to the host memory can lead to high latency. Thus, we need to 1) alleviate the
memory burden by reducing the size of parameters and intermediate results, and
2) manage the off-chip DRAM effectively to minimize the host memory accesses.

###  II-C Motivation

Putting it all together, we realize that mapping DNN workloads on adaptive
multi-accelerator systems are a challenging problem with extremely large
design space. There are many factors that hinder designers from finding a
high-quality mapping strategy. We analyze design choices in the design space.

Choices of accelerator designs & Workload allocation: Due to the heterogeneity
of DNN layers and various computation patterns that different accelerator
designs take, we may select different accelerator designs to configure
accelerators in the adaptive multi-accelerator system. Thus, we can allocate
the layers in the DNN to those accelerators with their preferred computation
patterns. In this way, there could be multiple layers mapped to a set of
accelerators with the same design.

Choices of parallelism strategies: For DNN layers mapped to accelerators with
the same design, partitioning each DNN layer with suitable parallelism
strategies can further distribute the computation load to these accelerators.
This naturally distributes the parameters of each layer and specific
strategies are necessary to reduce the memory cost.

Due to the importance and complexity of the above design choices, it is
necessary to give a clear formulation at first. We present system formulation
in Section III.

##  III System Formulation

Multi-accelerator system modeling: We formulate the topology of multi-
accelerators as a graph, $G(Acc,BW)$. As shown in Table I, the vertex
$Acc_{i}$ in the graph refers to the accelerator in the system of which the
design can be configured adaptively. The weight of the edge $BW_{i,j}$ refers
to the bandwidth of the communication between $Acc_{i}$ and $Acc_{j}$. This is
necessary because of the asymmetrical communication patterns mentioned in
Section II-A. Moreover, each accelerator $Acc_{i}$ in the system can access
the host memory with the bandwidth $BW_{i,host}$. Besides the communication,
each accelerator $Acc_{i}$ is equipped with off-chip DRAM with size $Mem_{i}$
to buffer the intermediate results and parameters.

Accelerator designs: For adaptive multi-accelerator systems, there are various
existing accelerator designs to choose from. We use a set
$Design=\\{d_{1},...,d_{M}\\}$ to represent the candidates. Following the
concepts in Section II-C, we define a set of accelerators with the same design
as an accelerator set, denoted by $AccSet$. Correspondingly, we use a map
$Config[AccSet_{i}]=d_{i}$ to express that the accelerators in $AccSet_{i}$
are configured to the design $d_{i}$. It should be satisfied that
$AccSet_{i}\cap AccSet_{j}=\varnothing$ for any $i\neq j$. For each
accelerator design, similar to H2H[7], we use their analytical performance
models to evaluate the number of cycles.

TABLE I: Explanations of Main Notations in System Formulation Notation |  Explanation
---|---
$Acc_{i}$ |  The accelerator that can be configured adaptively
$BW_{i,j}$ |  The communication bandwidth between $Acc_{i}$ and $Acc_{j}$
$Mem_{i}$ |  The size of off-chip DRAM attached to $Acc_{i}$
$d_{i}$ |  The available accelerator design in the adaptive system
$AccSet_{i}$ |  A set of accelerators with the same design
$Config$ |  A map from $AccSet_{i}$ to the chosen design $d_{i}$
$L_{i}$ |  The $i$-th layer of the DNN workload
$LayerSet_{i}$ |  A set of layers mapped to a certain accelerator set
$Map$ |  A map from $LayerSet_{i}$ to $AccSet_{i}$
$ES,SS$ |  The sets used to describe parallelism strategies

DNN workload allocation: the DNN workload can be represented as a computation
graph with a series of layers $\\{L_{1},...,L_{N}\\}$ (flattened in topology
order). Each layer holds its parameters like $(C_{out},C_{in},H,W,K)$ for
convolution. In our formulation, a subset of all the layers, $LayerSet_{i}$,
is mapped to a certain accelerator set, $AccSet_{i}$, which can be expressed
as $Map[LayerSet_{i}]=AccSet_{i}$.

Parallelism strategies: For each $LayerSet$ after DNN workload allocation, we
further perform parallel strategies to expedite its inference. Specifically,
we use two sets $ES$ and $SS$ which describe two different strategies used for
partitioning each layer in the layer set. Details will be discussed in Section
IV. Note that the chosen parallelism strategies are valid only if the tensor
sizes of these partitioned layers do not exceed the DRAM memory space of the
corresponding accelerator set.

Table I lists all the notations of our system formulation. MARS can evaluate
the whole network inference latency which consists of the latency of each
accelerator set and the communication latency between accelerator sets. We use
ASTRA-Sim[9] to simulate communication latency in the system. In addition, we
integrate analytical performance models of different accelerator designs into
ASTRA-Sim to evaluate the computation cycles. ASTRA-Sim is a simulator
featuring collective communication latency estimation for multi-accelerator
systems.

Figure 2: Using exclusive/shared shard to parallelize the layer computation

##  IV Parallelism Strategies in MARS

Figure 3: The Overview of MARS Mapping Algorithm

As introduced in our system formulation in Section III, we can perform
different parallelism strategies to layers in a subset $LayerSet$, further
exploiting the parallelism and boosting the performance. There have been some
previous works applying parallelism strategies on GPU clusters[10], and
chiplets[11]. And we generalize them and include the additional SS strategy
for accelerators. The computation-intensive layers like convolution in a DNN
workload can usually be represented as a nested loop, as shown in Fig. 2(a).
In this example, it takes the input feature map In with shape $(C_{in},H,W)$
and Weight with shape $(C_{out},C_{in},K,K)$ as input, and produces the output
feature map Out with shape $(C_{out},H,W)$. The essence of layer-level
parallelism is partitioning the nested loop along several dimensions. Then the
tensors involved in these dimensions will be partitioned into different
shards. We can distribute these shards to accelerators in the system and let
them perform the computation in parallel. Figure. 2 illustrates the idea of
our parallelism strategies. By default, As shown in Fig. 2(a), tensors are not
partitioned and we annotate each dimension with N to represent the nested loop
is not partitioned at the corresponding dimension. Figure 2(b) and 2(c)
presents two examples of our parallelism strategies, using different ways to
generate shards. Specifically, we classify tensor shards into two categories:
exclusive shards and shared shards. And we represent the parallelism strategy
by annotating corresponding dimensions with ES and SS.

  * •

Exclusive Shards (ES) of a tensor are held exclusively by accelerators
participating in the parallel computation of a layer. Note that a tensor can
be partitioned into exclusive shards along multiple specified dimensions.

  * •

Shared Shards (SS) of a tensor can be shared among accelerators. At the
beginning of the computation, each accelerator holds one shared shard of the
tensor. After the computation, the shared shards are transmitted to the
neighbor accelerator, which will form a ring communication logically. Such
computation and communication processes will alternate until accelerators
finish computation using all shared shards of the tensor.

In Fig.2(b), In and Out are split along the $W$-dimension (marked in blue)
into two exclusive shards. In and Weight are split along the
$C_{in}$-dimension (marked in yellow) into two exclusive shards. By performing
the ES strategy along these two dimensions, the computation workload is
partitioned into four equal parts, each of which can be executed on four
accelerators (Acc1-Acc4). In this way, each accelerator holds a quarter of the
input feature map and half of the weight. As shown in Fig.2(b), because Acc1
and Acc2 generate the same left half of the output feature map Out (i.e., left
side of the cuboid), the results need to be summed up using the collective
communication primitive, All-Reduce. The same process is performed on Acc3 and
Acc4 to compute the right half of Out. After that, Out is equally split into
two parts along the partitioned dimension, $W$, stored in Acc1&Acc2 and
Acc3&Acc4 respectively.

In addition to exclusive shards, parameters can further be partitioned into
shared shards. Fig. 2(c) shows how exclusive shards and shared shards coexist
in multi-accelerator systems. $In$ and $Out$ are split into exclusive shards
along the $W$-dimension. $Weight$ and $Out$ are split into shared shards along
the $C_{out}$-dimension. The workload can be executed on two accelerators
because of the sharing of parameters. In phase 1, Acc1 and Acc2 hold exclusive
shards of $In$ and shared shards of $Weight$, and generate the corresponding
quarters of $Out$. Then in phase 2, Acc1 and Acc2 transmit the shared shards
of $Weight$ they hold to each other. In phase 3, the two accelerators compute
the remaining part of $Out$ using the received shards of $Weight$. The SS
strategy can be useful due to the communication heterogeneity in adaptive
multi-accelerator systems, as introduced in Section II-A. It can leverage the
low-latency communication between accelerators within a group or a rack and
hence relieve the data bus congestion to/from the host.

We represent the parallelism strategies formally according to the dimensions
which are partitioned. For example, the strategies can be expressed as two
sets, i.e., $ES=\\{C_{in},W\\},SS=\varnothing$ in Fig. 2(b) and
$ES=\\{W\\},SS=\\{C_{out}\\}$ in Fig. 2(c), respectively. Because of the
coexistence of exclusive shards and shared shards in multi-level nested loops,
we need to select suitable parallelism strategies for all the dimensions from
a wide range of candidates. When applying exclusive shards on two dimensions
of the convolution layers, there are $C_{6}^{2}=15$ choices. In addition, when
applying shared shards on one certain dimension, the number of choices
increases to $C_{6}^{2}\cdot 6=90$. They will incur different communication
volumes and computation patterns. For the DNN workloads, the number of design
points in the design space increases exponentially with respect to the depth
of the network. To efficiently search the large design space to find a design
point with low system latency, we propose the mapping algorithm in MARS.

##  V MARS Mapping Algorithm

Figure 3 presents the overview of MARS mapping algorithm, using a two-level
genetic algorithm to explore the design space. Genetic algorithms are widely
used in deep learning kernel generation[12] and optimization on multi-core
accelerators[13]. We mainly focus on our improvements to adapt genetic
algorithms to the mapping problem.

As discussed in Section III, The accelerator configurations and workload
allocation may deeply influence the optimal parallelism strategy: the layer
with the same parallelism strategy can show a large performance gap when the
accelerator design changes. Simply tuning them in one pass of the search is
easy to fall into local optimums. To make the problem more solvable, we adopt
the idea of divide and conquer by dividing the problem into two levels. The
first-level genetic algorithm (the pink box in Fig. 3) aims at finding the
minimum overall latency for the multi-accelerator system. At the first level,
$AccSet$ with configured designs, and the layers mapped to them, $LayerSet$,
are decided. With these factors fixed, the mapping problem is further divided
into several sub-problems: For each layer set $LayerSet_{i}$ mapped to an
accelerator set $AccSet_{i}$, what parallelism strategies should be applied to
the layers in these layer sets? To solve these sub-problems, the second-level
genetic algorithm (green and blue boxes in Fig. 3) is responsible to find the
proper parallelism strategy for each layer to minimize the latency on the
given accelerator set, considering both computation and communication costs.
The population is iteratively updated in the mutation and crossover phase
according to the latency evaluated by the simulator. After several iterations,
the minimum latencies of accelerator sets will be aggregated to obtain the
overall latency with extra communication costs between the sets. The overall
latency will work as the fitness function to influence the succeeding mutation
and crossover of the first-level genetic algorithm.

We use several heuristics to prune the search space. The topology of multi-
accelerator systems is formulated as a graph, $G(Acc,BW)$, which has been
introduced in Section III. MARS iteratively removes the edge with the lowest
bandwidth in $G(Acc,BW)$. This will produce several connected sub-graphs,
which are regarded as candidates of $AccSet$. This strategy can help generate
$AccSet$ with minimal communication bottlenecks. In the decode step in Fig.3,
the candidate of $AccSet$ with the highest gene value will be chosen. The
accelerator design of each $AccSet$ is decided through the gene value of each
design. MARS profiles the performance of accelerator designs on the layers of
the DNN workload according to analytical models before the search. The gene
value of these designs at the first generation is initialized according to the
normalized performance. This means the design with higher computation ability
is most likely to be chosen at the beginning of the search. As for layer sets,
to avoid frequent communication between different accelerator sets, we limit
that each accelerator set is only mapped with a continuous series of layers in
topology order. Then MARS will call the second-level genetic algorithm to
optimize parallelism strategies over the accelerator set and the layer set. At
the second level, the parallelism strategies of each layer are decided. For
layer $L$, MARS uses individual genes to decide the $ES$ and $SS$ sets of $L$
respectively. It prioritizes parallelism at the dimensions with higher gene
values.

##  VI Evaluation

###  VI-A Experiment Setup

Hardware Platform Modeling: We model an adaptive multi-accelerator system in
modified ASTRA-Sim[9]. The system topology is modeled based on the
interconnection of F1 instances, as illustrated in Fig. 1. The system consists
of eight accelerators separated into two groups. We set the communication
bandwidth between accelerators in the same group to 8Gbps. The accelerator-to-
host bandwidth is set to 2Gbps to simulate the high latency of accessing host
memory. The size of off-chip DRAM on accelerators is set to 1GB.

Accelerator Designs: For the accelerator designs, we use three kinds of CNN
accelerators on FPGA with their performance models. To make their theoretical
performance comparable, we set the clock frequency to 200MHz uniformly and use
design parameters with similar numbers of PEs. The detailed setting can be
found in Table II.

TABLE II: Available Accelerator Designs |  Design |  Freq(MHz) |  #PEs |  Design Parameters
---|---|---|---|---
1 |  SuperLIP[14] |  200 |  438 |  $T_{m},T_{n},T_{r},T_{c}:64,7,7,14$
2 |  [15] |  200 |  572 |  $row,col,vec:11,13,8$
3 |  [16] |  200 |  576 |  $n,P_{n},P_{m}:6,2,8$

Models: We use several representative CNN models as benchmarks for evaluation,
including AlexNet, VGG, ResNet, and WideResNet. For some models, we use
multiple settings with different parameter sizes and FLOPs.

TABLE III: Latency comparison between baseline and MARS Model |  #Convs |  #Params |  FLOPs | Algorithm |  Latency/ms |  Mapping found by MARS
---|---|---|---|---|---|---
AlexNet |  5 |  61.1M |  727M | Baseline |  0.832 |  Conv1-2→4$\times$Design 1; Conv1:$ES=\\{H,W\\},SS=\varnothing$
Conv3-5→4$\times$Design 3; Conv5:$ES=\\{H,C_{Out}\\},SS=\varnothing$
|  |  |  | MARS |  0.748(-10.1%) |
VGG16 |  13 |  138M |  15.5G | Baseline |  20.6 |  Conv1-7→4$\times$Design 1; Conv1:$ES=\\{H,W\\},SS=\varnothing$
Conv8-10→2$\times$Design 2; Conv8:$ES=\\{H\\},SS=\\{C_{Out}\\}$
Conv11-13→2$\times$Design 3; Conv13:$ES=\\{C_{Out}\\},SS=\\{H\\}$
|  |  |  | MARS |  14.9(-27.7%) |
|  |  |  |  |  |
ResNet34 |  33 |  21.8M |  3.68G | Baseline |  4.43 |  Conv1-15→4$\times$Design 1; Conv2:$ES=\\{H,W\\},SS=\varnothing$
Conv16-33→4$\times$Design 3; Conv29:$ES=\\{C_{In},C_{Out}\\},SS=\varnothing$
|  |  |  | MARS |  2.76(-37.7%) |
ResNet101 |  100 |  44.55M |  7.85G | Baseline |  14.9 |  Conv1-22→2$\times$Design 1; Conv2:$ES=\\{H\\},SS=\varnothing$
Conv23-79→4$\times$Design 2; Conv24:$ES=\\{H,W\\},SS=\\{C_{Out}\\}$
Conv80-100→2$\times$Design 2; Conv99:$ES=\\{C_{Out}\\},SS=\\{W\\}$
|  |  |  | MARS |  7.95(-46.6%) |
|  |  |  |  |  |
WRN-50-2 |  49 |  68.8M |  11.4G | Baseline |  16.7 |  Conv1-17→2$\times$Design 1; Conv3:$ES=\\{H\\},SS=\varnothing$
Conv18-29→2$\times$Design 2; Conv18:$ES=\\{C_{Out}\\},SS=\\{H\\}$
Conv30-49→4$\times$Design 2; Conv48:$ES=\\{C_{In},C_{Out}\\},SS=\varnothing$
|  |  |  | MARS |  10.1(-39.5%) |
|  |  |  |  |  |

Baseline: To the best of our knowledge, there are no existing mapping
algorithms that take both multi-level parallelism and adaptive multi-
accelerator systems into consideration. To show the ability of MARS, we extend
the computation-prioritized mapping algorithm from [6] with parallelism
strategies integrated. The baseline uses fixed two accelerator sets which are
the same as two groups in the system topology. This is reasonable to avoid
high communication latency across groups. And it allocates half of the layers
to each accelerator set and chooses the accelerator design with the lowest
computation latency. About the parallelism strategies, each layer is
partitioned with ES along the longest two dimensions.

###  VI-B Performance Analysis

Following the experiment setup, we test the latency of the baseline mapping
algorithm and MARS mapping algorithm. We list the parameters of the models,
the overall latency, together with the accelerator sets, workload allocation,
and parallelism strategies of representative layers in each layer set found by
MARS in Table III.

As shown in the table, MARS outperforms the baseline for all models. The
latency reduction ranges from 10.1% to 46.6% (32.2% on average). The larger
design space enables MARS to find better solutions compared to the baseline.
Some patterns are shown in the mappings found by MARS. The first few layers of
these models are always mapped to accelerator sets configured with Design
1(SuperLIP). The reason is that the first few layers usually have larger
resolutions and fewer channels. Other designs suffer from low hardware
utilization because the shape of the layer cannot saturate the PEs in the
architecture, while the design parameter of SuperLIP($T_{n}=7$) can achieve
relatively high utilization. MARS tends to partition these layers along
$H/W$-dimension. We can also find that design 3 does not show up in ResNet101
and WRN-50-2. This is because design 3 is an accelerator based on Winograd
algorithm, which makes it impossible to effectively handle $1\times 1$
convolution in the bottleneck block of these models. Because $C_{In}$ and
$C_{Out}$ enlarge rapidly when the network goes deeper, MARS is more likely to
partition these layers along $C_{In}/C_{Out}$-dimension for parallelism. This
emphasizes the importance to select accelerator designs and parallelism
strategies based on the computation patterns of layers.

###  VI-C Comparison with H2H

H2H[7] focuses on mapping heterogeneous models to heterogeneous multi-
accelerators with fixed accelerator designs. Though MARS and H2H have
different problem formulations, we still compare their performance with
heterogeneous DNN models. We reuse performance models of convolution
accelerators used in H2H and model the cloud-scale multi-FPGA systems in
ASTRA-Sim following the 5-level bandwidth settings of H2H. For heterogeneous
accelerator designs, we assume that members in the accelerator set stall until
the slowest accelerator finishes computing. Then we evaluate the latencies in
milliseconds of MARS and H2H on two ResNet-based heterogeneous models. The
results are shown in Table IV. We can see that MARS achieves lower latency
than H2H on both models when limited to the same bandwidth (59.4% reduction on
average). When the bandwidth is extremely low, MARS tends to partition
convolution layers along $H/W$-dimension, which requires low communication
cost.

TABLE IV: Comparison of Latency ($\mathrm{ms}$) with H2H Bandwidth | CASIA-SURF[17] | FaceBag[18]
---|---|---
|  H2H |  MARS |  H2H |  MARS
Low-(1Gbps) |  360.0 |  124.6(-65.4%) |  520.0 |  237.4(-54.3%)
Low(1.2Gbps) |  340.0 |  120.3(-64.6%) |  450.0 |  224.6(-50.1%)
Mid-(2Gbps) |  260.0 |  100.9(-61.2%) |  320.0 |  159.4(-50.2%)
Mid(4Gbps) |  230.0 |  74.3(-67.7%) |  230.0 |  112.1(-51.3%)
High(10Gbps) |  180.0 |  46.8(-74.0%) |  170.0 |  76.5(-55.0%)

##  VII Conclusion

In this paper, we propose MARS, a mapping framework aiming at exploiting
multi-level parallelism on adaptive multi-accelerator systems. We formulate
design space including the choices of accelerator design, workload allocation,
and parallelism strategies. A two-level genetic algorithm with heuristics is
used to perform design space exploration. The mapping algorithm shows
significant latency reduction compared to the baseline mapping algorithm and
the state-of-the-art method.

##  VIII Acknowledgements

This work is partially sponsored by the National Natural Science Foundation of
China (62102249, 62232015) and Shanghai Pujiang Program (21PJ1408200).

## References

  * [1] Kaiming He et al.  Deep residual learning for image recognition.  In CVPR, pages 770–778, 2016.
  * [2] Ashish Vaswani et al.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [3] Heng-Tze Cheng et al.  Wide & deep learning for recommender systems.  In Proceedings of the 1st workshop on deep learning for recommender systems, pages 7–10, 2016.
  * [4] Jeremy Fowers et al.  A configurable cloud-scale dnn processor for real-time ai.  In ISCA, pages 1–14. IEEE, 2018.
  * [5] Wentai Zhang et al.  An efficient mapping approach to large-scale dnns on multi-fpga architectures.  In DATE, pages 1241–1244. IEEE, 2019.
  * [6] Hyoukjun Kwon et al.  Heterogeneous dataflow accelerators for multi-dnn workloads.  In HPCA, pages 71–83. IEEE, 2021.
  * [7] Xinyi Zhang et al.  H2h: Heterogeneous model to heterogeneous system mapping with computation and communication awareness.  DAC, 2022.
  * [8] deeppat kristopk.  How to use the pcie peer-2-peer version 1.0, 2021.
  * [9] Saeed Rashidi et al.  Astra-sim: Enabling sw/hw co-design exploration for distributed dl training platforms.  In ISPASS, pages 81–92, 2020.
  * [10] Lianmin Zheng et al.  Alpa: Automating inter-and intra-operator parallelism for distributed deep learning.  OSDI, 2022.
  * [11] Zhanhong Tan et al.  Nn-baton: Dnn workload orchestration and chiplet granularity exploration for multichip accelerators.  In ISCA, pages 1013–1026. IEEE, 2021.
  * [12] Nicolas Vasilache et al.  Tensor comprehensions: Framework-agnostic high-performance machine learning abstractions.  arXiv preprint arXiv:1802.04730, 2018.
  * [13] Sheng-Chun Kao et al.  Magma: An optimization framework for mapping multiple dnns on multiple accelerator cores.  In HPCA, pages 814–830. IEEE, 2022.
  * [14] Weiwen Jiang et al.  Achieving super-linear speedup across multi-fpga for real-time dnn inference.  ACM TECS, 18(5s):1–23, 2019.
  * [15] Xuechao Wei et al.  Automated systolic array architecture synthesis for high throughput cnn inference on fpgas.  In DAC, pages 1–6, 2017.
  * [16] Liqiang Lu et al.  Evaluating fast algorithms for convolutional neural networks on fpgas.  In FCCM, pages 101–108. IEEE, 2017.
  * [17] Shifeng Zhang et al.  Casia-surf: A large-scale multi-modal benchmark for face anti-spoofing.  IEEE TBIOM, 2(2):182–193, 2020.
  * [18] Tao Shen et al.  Facebagnet: Bag-of-local-features model for multi-modal face anti-spoofing.  In Proceedings of CVPR Workshops, 2019.