# Mixed-TD: Efficient Neural Network Accelerator with Layer-Specific Tensor
Decomposition

Zhewen Yu, Christos-Savvas Bouganis Imperial College London
London, UK
{zhewen.yu18<EMAIL_ADDRESS>
###### Abstract

Neural Network designs are quite diverse, from VGG-style to ResNet-style, and
from Convolutional Neural Networks to Transformers. Towards the design of
efficient accelerators, many works have adopted a dataflow-based, inter-layer
pipelined architecture, with a customized hardware towards each layer,
achieving ultra high throughput and low latency. The deployment of neural
networks to such dataflow architecture accelerators is usually hindered by the
available on-chip memory as it is desirable to preload the weights of neural
networks on-chip to maximise the system performance. To address this, networks
are usually compressed before the deployment through methods such as pruning,
quantization and tensor decomposition. In this paper, a framework for mapping
CNNs onto FPGAs based on a novel tensor decomposition method called Mixed-TD
is proposed. The proposed method applies layer-specific Singular Value
Decomposition (SVD) and Canonical Polyadic Decomposition (CPD) in a mixed
manner, achieving 1.73$\times$ to 10.29$\times$ throughput per DSP to state-
of-the-art CNNs. Our work is open-sourced: https://github.com/Yu-Zhewen/Mixed-
TD.

##  I Introduction

The recent advances in Machine Learning (ML) research have led to the design
of a large and diverse set of neural network designs. At a macroscopic level,
popular designs include Convolutional Neural Networks (CNNs) and Transformers,
where at a microscopic level the above structures contain layers with
different properties, including number of channels, kernel size, residual
connection, etc. Towards the design of neural network accelerators, many works
adopted a dataflow architecture [1], customizing the computational pipeline to
each layer to maximise efficiency and achieve high throughput.

A challenge for the dataflow-based accelerators is the storing of the
parameters of the neural network to on-chip memory. Let’s consider the AMD
Alveo U250, a data-center acceleration card. Despite its high performance,
this card has a limited internal SRAM capacity of only 54MB. A ResNet-50, a
popular neural network architecture with 23 million parameters, requires a
storage capacity of approximately 92MB in floating-point format, which exceeds
the SRAM capacity. Apart from the memory capacity, the on-chip memory
bandwidth also becomes a limitation when the architecture requires access to a
large number of the parameters concurrently to support parallel computation
for enhanced performance. Storing the parameters to off-chip memory addresses
the capacity problem but penalises the performance of the system.

To reduce the memory footprint of the model, existing approaches compress the
weights of a pre-trained neural network and fine-tune the compressed weights
before the network is mapped to a dataflow-based accelerator. In the case of a
pre-trained deep neural network, the data distribution and error tolerance
vary across different parts of the network [2]. As such, it is necessary to
make the compression method fine-grained and layer-specific to avoid
significant accuracy degradation. In previous work on weights pruning,
unstructured ways of pruning bring a larger compression ratio with negligible
accuracy degradation compared with the channel-wise structured pruning [3, 4].
Similarly, in weights quantization, mixed precision and block floating point
methods are favoured compared with uniform wordlength and range [5, 6, 7].

In this paper, we explore a different dimension for performing fine-grained,
layer-specific, and hardware-friendly weights compression, by using Tensor
Decomposition techniques. Furthermore, we propose the use of an ML-based proxy
for predicting the performance of the compressed model during its mapping to
an FPGA, making it possible to explore the large design space defined by the
introduced Tensor decomposition schemes. Our performance evaluation shows that
the proposed approach can achieve significant weight compression with
negligible accuracy penalty, as well as result in designs with competitive
latency and throughput to state-of-the-art approaches.

Figure 1: An example of a dataflow architecture, where each layer of the
network has its own customized hardware. Inter-layer and intra-layer pipelines
are usually applied.

The key contributions of this paper are as follows:

  * •

A novel weight compression method, Mixed-TD, that is based on tensor
decomposition techniques. It extends current approaches by introducing for the
first time a layer-specific mixture of Singular Value Decomposition (SVD) and
Canonical Polyadic Decomposition (CPD).

  * •

An efficient method for fast design space exploration through the evolutionary
search and a random-forest-based throughput predictor.

  * •

A dataflow-based accelerator design achieving low latency, high throughput,
and negligible accuracy loss at the same time. In terms of the Throughput per
DSP, we are achieving gains of 1.73$\times$ to 10.29$\times$ compared to
existing work.

##  II Background

###  II-A Tensor Decomposition

Tensor decomposition expresses one tensor as a set of elementary and simpler
tensors which act on each other [8, 9]. The decomposition of a tensor
$\mathcal{A}$ can be derived through the following optimization problem:

$\displaystyle\min_{a_{1},a_{2}\dots a_{M}}~{}r_{1},r_{2}\dots r_{M}$ (1)
$\displaystyle s.t.~{}||\mathcal{A}-f(a_{1},a_{2}\dots a_{M})||\leq\epsilon,$

where $a_{i},i\in[1,M]$ is the set of elementary tensors and $f$ is the
approximation function $f$ so that the given $M_{th}$-order tensor
$\mathcal{A}$ is approximated within the desired error boundary $\epsilon$ and
the ranks of these elementary tensors $r_{i}$ are also minimised. Depending on
the operations inside the function $f$, there are different formats of tensor
decomposition available. SVD and CPD are two of them that have been well
studied in compressing the weights of neural networks [10].

SVD targets the compression of a given $2_{nd}$-order tensor, which is $M=2$
and $\mathcal{A}\in\mathbb{R}^{d_{1}\times d_{2}}$. $\mathcal{A}$ is
decomposed into the form of

$\mathcal{A}\approx U_{r}\Sigma_{r}V^{T}_{r},$ (2)

where the rows of $U_{r}$ and the columns of $V_{r}$ are orthogonal bases, and
$\Sigma_{r}$ is a diagonal matrix containing the top-$r$ singular values in
descending order. After absorbing the diagonal matrix $\Sigma_{r}$ into
$U_{r}$ and $V_{r}$, the final format becomes the product of two tensors and
the number of parameters remaining is $(d_{1}+d_{2})r$. SVD can also be used
to compress the tensor with higher-order ($M>2$) which requires reducing the
order of the target tensor with slicing and reshaping operations beforehand
[11].

CPD can be applied directly to a high-order tensor without the need of
reducing its order to $M=2$ like SVD. Given the $M_{th}$-order tensor,
$\mathcal{A}\in\mathbb{R}^{d_{1}\times d_{2}\times\dots d_{M}}$, its CPD
format can be represented as the sum of the outer product of $1_{st}$-order
tensors.

$\mathcal{A}\approx\sum_{i=1}^{r}a_{1,i}\otimes a_{2,i}\otimes\dots a_{M,i}$
(3)

This set of $1_{st}$-order tensors can be computed via the Alternating Least
Squares (ALS) method [9], and the number of parameters remaining is
$(d_{1}+d_{2}+\dots d_{M})r$. The difference between SVD and CPD is visualised
in Fig. 2.

Figure 2: Represent SVD and CPD in the Tensor Diagram Notation[12]. Each solid
node denotes a $M_{th}$-order tensor with $M$ edges. Edges connected together
represent the tensor contraction operation between two tensors. In CPD, the
hollow node with a cross inside represents the sum of the outer product.

###  II-B Related Work

Many state-of-the-art approaches that target the acceleration of CNNs are
based on dataflow architectures, as they reduce the off-chip memory accesses,
and customize the hardware pipeline to the properties of the targeted CNN
load.

fpgaConvNet [13] generates a highly optimized architecture based on the target
CNN and FPGA board. They considered the splitting of the whole data flow into
multiple partitions, where each partition only contains a subgraph of the
neural network, allowing the tool to map large CNN models to small devices
through reconfiguration. The tool performs a faithful mapping of the CNN to
the FPGA, assuming an already optimized CNN model as its input.

FINN [14] implemented the dataflow architecture through extreme network
quantization. The authors fit a ResNet-50 network on a U250 device, after
quantizing the weights and the activations of the network to 1 bit and 2 bits
respectively, except the first and last layers which are quantized to 8 bits.
Their mixed precision implementation achieves an impressive 2703 Frames Per
Second (FPS) but they also reported 9.8 percentage points (pp) top-1 accuracy
degradation on the ImageNet dataset compared to the floating point version of
the network.

HPIPE [15] explored weights compression through unstructured sparsity. The
authors eliminated $85\%$ of the weights from ResNet-50 and encoded the
remaining parameters in a compressed format to save memory storage. The
authors reported a 5.2pp accuracy degradation and an achieved throughput of
4550 FPS on a Stratix 10 2800 FPGA.

Our approach takes a step to a different direction from prior work as it
explores a different dimension for performing fine-grained, layer-specific
weights compression, which achieves competitive performance compared with
existing methods including unstructured sparsity and mixed precision
quantization.

##  III Mixed Tensor Decomposition

This section introduces our proposed fine-grained compression method, termed
Mixed-TD, which opens the space for applying layer-specific SVD and CPD
decompositions in a mixed manner.

###  III-A Compute Decomposed Convolution

Let us consider an $N$-layer CNN whose weights are represented as
$4_{th}$-order tensors $\mathcal{W}_{j}\in\mathbb{R}^{c_{j+1}\times
c_{j}\times k_{j}\times k_{j}},j\in[1,N]$, where $c_{j+1}$, $c_{j}$ and
$k_{j}\times k_{j}$ are denoting the number of output channels, input channels
and the kernel size of the $j_{th}$ convolutional layer respectively. Its
input and output are denoted as $\mathcal{X}_{j}\in\mathbb{R}^{b\times
m_{j}\times n_{j}\times c_{j}}$ and $\mathcal{X}_{j+1}\in\mathbb{R}^{b\times
m_{j+1}\times n_{j+1}\times c_{j+1}}$, where $b$ denotes the batch size,
$m_{j}$ and $n_{j}$ denote the spatial size of the feature map.

The convolution operation can be represented as the tensor contraction along
$c_{j}$ and $k_{j}$ dimensions,

$\mathcal{X}_{j+1}=\sum_{c_{j},k_{j}}(\mathcal{W}_{j}\cdot
S(\mathcal{X}_{j})),$ (4)

where the sliding window function $S$ performs the padding and striding,
converting $\mathcal{X}_{j}$ into the shape of $b\times m_{j+1}\times
n_{j+1}\times c_{j}\times k_{j}\times k_{j}$.

Our proposed approach performs tensor decomposition on the weights tensor at
design time. After decomposition, the tensor $\mathcal{W}_{j}$ is substituted
by the form of (2) or (3), transforming as such the computation from one
tensor contraction operation into multiple consecutive ones, but with a
reduced total number of elements.

###  III-B Layer-Specific Design Choices

In the proposed Mixed-TD method, there are three types of layer-specific
design choices.

SVD or CPD: For each convolutional layer, its 4-d weight tensor can be
decomposed either in SVD or CPD format. For SVD, the 4-d tensor is reshaped
into 2-d and then decomposed into the product of two 2-d tensors, as (2)
shows. Otherwise, for CPD, the 4-d tensor is decomposed to the sum of the
outer product of four 1-d tensors, as (3) shows. The main difference between
these two formats is the number of tensors left after decomposition as well as
their representation abilities. In Section IV and VI, we will see the number
of tensors left affects the accelerator performance in a dataflow architecture
design. A rule of thumb is that the CPD format has more overhead than SVD in
hardware design but in terms of the impact on network accuracy, whether to use
SVD or CPD is really layer-specific. We use the enumeration variable
$t_{j}=\\{0:SVD,1:CPD\\}$ to represent this decision.

Channels grouping: Instead of directly applying the tensor decomposition to
the entire $\mathcal{W}_{j}$, we can choose to slice and split
$\mathcal{W}_{j}$ into multiple chunks where each chuck can then be decomposed
individually. As in CNN designs $c_{j+1}$ and $c_{j}$ are usually much larger
than $k_{j}$, our method provides the options of slicing the first ($c_{j+1}$)
and the second dimension ($c_{j}$) of $\mathcal{W}_{j}$ into $g_{1,j}$ and
$g_{2,j}$ groups respectively. As such, for each $\mathcal{W}_{j}$, it can be
split into $g_{1,j}g_{2,j}$ chunks in total, and each chuck has the shape of
$\mathbb{R}^{\frac{c_{j+1}}{g_{1,j}}\times\frac{c_{j}}{g_{2,j}}\times
k_{j}\times k_{j}}$. The intuition behind offering this type of design choice
is to leverage the diversity and similarity of feature maps [16].

Rank selection: In both SVD and CPD methods, a key design choice is the rank
selection, as it controls the compression ratio of the decomposition. Within a
network, the redundancy of each layer varies considerably and setting a
uniform compression ratio dramatically hurts the accuracy [17]. Therefore, it
is critical to have the layer-specific choice of the rank $r_{j}$ after
searching for the optimal combination of them.

| SVD Engine | CPD Engine
---|---|---
Rank | $r$
Inner Product | 2 | 2
Outer Product | 0 | 2
Weights |  $U_{r}$, $V_{r}$ | $a_{\\{1,2,3,4\\},r}$
Unroll factors |  $p_{in,U}$, $p_{in,V}$ | $p_{in,\\{1,2,3,4\\}}$
$p_{out,U}$, $p_{out,V}$ | $p_{out,\\{1,2,3,4\\}}$

Figure 3: Internal architecture of the proposed SVD Engine and CPD Engine.
Both architectures implement the convolution on the decomposed weight tensors,
but they differ in the number of stages of computation as well as the dataflow
between stages.

##  IV Accelerator Design

This section introduces the accelerator designs for deploying networks
compressed by our Mixed-TD method. A high level description of the dataflow
architecture is given in section IV-A, the internal structures of compute
engines are discussed in section IV-B, and the data connections between
engines are elaborated in section IV-C.

###  IV-A Dataflow Architecture

We extend an open-source dataflow architecture accelerator fpgaConvNet [13,
18]. As we are targeting both high throughput and low latency, we did not use
the device reconfiguration technique in our work. Each layer, including
convolution, relu, pooling, elementwise addition, fully connected, and etc.,
is mapped to a dedicated computation engine. These engines are connected in a
pipeline manner to maximise overall throughput. During inference, the input
data is sent from DDR to FPGA using the AXI-Stream interface, propagating
through all computation engines, and the final prediction results are sent
back to DDR using AXI-Stream. The system processes input data in batches and
the pipeline is emptied between different batches.

###  IV-B SVD v.s. CPD Engine

Each compressed convolutional layer of the network is mapped to either the SVD
Engine or the CPD Engine with all the weights stored at the on-chip memories
and preloaded before the inference starts. Both the SVD Engine and CPD Engine
adopt the structure of input buffer, MAC units and accumulation module to
compute the tensor contractions (Fig. 3).

  * •

Input Buffer: It applies the sliding window function to input feature maps
using a set of $p_{in}$ line buffers. At every clock cycle, each line buffer
fetches one word of data from the previous layer and dispatches a window with
the size of $k_{j}\times k_{j}$.

  * •

MAC Units: $p_{out}$ MAC units operate in parallel. Each MAC unit, which
contains a multiplier array followed by an adder tree, is responsible for a
vector-vector multiplication. The MAC units are fed by the input buffer as
well as the preloaded weights.

  * •

Accumulation Module: It gathers and accumulates the partial sums produced by
the MAC units before dispatching the data into the next stage of computation.

Now, we highlight the differences between the proposed SVD Engine and CPD
Engine.

  * •

In the case of SVD decomposition, the convolution kernel is decomposed into
two stages of tensor contraction, as (2) shows. In the case of CPD
decomposition, the computation is decomposed to four stages instead, as (3)
shows. In each stage, $p_{in}$ and $p_{out}$ can be tuned individually to
trade resources for throughput.

  * •

In the SVD Engine, the buffer broadcasts all the inputs to $p_{out}$ of MAC
units concurrently to compute the inner products. On the contrary, in the CPD
Engine, at the second and the third stages of computation (involving $a_{3,r}$
and $a_{4,r}$), the data coming out of the buffer is scattered to the
$p_{out}$ MAC units instead to compute the outer products.

###  IV-C Cross-layer Data Flow

In the case of decomposed convolutions without channel grouping (as explained
in Section III), each layer will require its own engine, and data will flow
between layers in the form of a flattened 4-dimensional tensor, in the form of
$b\times m_{j}\times n_{j}\times c_{j}$, where $j\in[1,N]$. In this notation,
$b$ represents the batch size, $m_{j}$ and $n_{j}$ represent the spatial size
of the feature map, and $c_{j}$ represents the number of feature maps in the
$j$-th layer.

In the case where channel grouping is applied, i.e. either $g_{1,j}$ or
$g_{2,j}$ is greater than 1, there will be $g_{1,j}g_{2,j}$ engines per layer.
In this case, data flows into the engines in a flattened form, as $b\times
m_{j}\times n_{j}\times\frac{c_{j}}{g_{2,j}}\times g_{2,j}$. After splitting
of the data into $g_{2,j}$ groups is performed, each engine fetches data in
the form of $b\times m_{j}\times n_{j}\times\frac{c_{j}}{g_{2,j}}$. Similarly,
at the layer’s output, data is in the form of $b\times m_{j+1}\times
n_{j+1}\times\frac{c_{j+1}}{g_{1,j}}\times g_{1,j}$.

In addition, if $g_{1,j}\neq g_{2,j+1}$, the data must be rearranged from
$\frac{c_{j+1}}{g_{1,j}}\times g_{1,j}$ to $\frac{c_{j+1}}{g_{2,j+1}}\times
g_{2,j+1}$ before being fed into the engines of the next layer. This is
achieved using an array of FIFOs with a width set to the Least Common Multiple
(LCM) of $g_{1,j}$ and $g_{2,j+1}$. The data streams are written and read from
this FIFO array in a round-robin fashion to effectively rearrange the data.

Figure 4: Design flow of our system. The flow includes the search and the
deployment stages. During the search stage, we query the accuracy and the
throughput of each design point and perform the design space exploration to
identify the optimal design. The optimal design is then fine-tuned to improve
its accuracy before being synthesised and deployed on the FPGA device.

##  V Design Space Exploration

Having introduced both the compression algorithm and accelerator architecture,
this section focuses on their integration, and how we can efficiently identify
the optimal design point.

###  V-A Problem Formulation

We define the optimal design point as the one that maximizes the prediction
accuracy of the network, while satisfying the given resource budget and the
target throughput in Frames Per Second (FPS). Several variables affect the
design point, including the applied decomposition parameters
$\tau=\\{g_{1,j},g_{2,j},t_{j},r_{j}\\},j\in[1,N]$ (defined in section III-B);
as well as the accelerator’s unrolling factors
$\psi=\\{p_{in,j},p_{out,j}\\},j\in[1,N]$ (defined in section IV-B).

$\max_{\tau,\psi}ACC~{}~{}~{}s.t.~{}FPS\geq FPS_{target},~{}~{}RSC\leq
RSC_{budget}$ (5)

The accuracy of the network depends on the choice of $\tau$ only. The FPS
depends on both $\tau$ and $\psi$, as $\tau$ controls the per-layer workload
and $\psi$ determines the initiation interval of the computation loop. The
total resource is represented by the sum of per-layer resources, depending on
$\psi$ only. To efficiently solve the constrained optimization problem, we
decouple the searches of decomposition parameters $\tau$ and unrolling factors
$\psi$, which are elaborated in the following two sections respectively.

###  V-B Evolutionary Search

Due to the fine-grained and layer-specific decisions made by the proposed
Mixed-TD method, the design space defined by $\tau$ is extremely large. To
illustrate, ResNet-18 alone consists of $5.7\times 10^{29}$ possible candidate
designs.

As such, for the efficient search of $\tau$, we adopt the evolutionary
searching algorithm proposed in [19]. Initially, we randomly sample from the
design space and keep only the valid design points that satisfy the throughput
and resource constraints until we obtain $|\textbf{P}|$ valid design points.
These $|\textbf{P}|$ design points form the first generation of the
“population”, referred to as “parents”. Using mutation and crossover, we
generate $|\textbf{C}|$ new valid samples, referred to as “children”. The
parents and children are then ranked together based on their prediction
accuracy, and only the top-$|\textbf{P}|$ samples are retained to become the
parents of the next generation. This ensures the continuation of high-
performing design points throughout the evolution process.

Algorithm 1 Constrained Evolutionary Search

1:procedure Validate(sample $i$)

2: query accuracy of $i$

3: query throughput of $i$ return $FPS\geq FPS_{target}$

4:procedure Searching

5: random sample $|\textbf{P}|$ valid designs

6: while step $\leq$ max_steps do

7: mutate P, obtain $|\textbf{C}|/2$ new valid samples.

8: crossover P, obtain $|\textbf{C}|/2$ new valid samples.

9: sort $\textbf{P}\cup\textbf{C}$ by accuracy

10: keep top-$|\textbf{P}|$ samples

###  V-C Performance Predictor

To further speed up the search of the evolutionary algorithm, it is crucial to
minimize the time spent on evaluating the accuracy and throughput of each
design point.

For accuracy estimation, we use the decomposed network without fine-tuning and
evaluate it on a single batch of data, rather than the entire validation
dataset. This significantly reduces the time required for accuracy queries,
reducing the duration from minutes to just seconds on a desktop GPU.

For throughput estimation, it is necessary to solve the following resource
allocation problem that identifies the optimal configuration of unrolling
factors $\psi$.

$\max_{\psi}FPS~{}~{}~{}s.t.~{}RSC\leq RSC_{budget}$ (6)

The unrolling factors impact both the resource utilization and system
throughput. The optimal configuration is which maximizes the system throughput
by balancing the delays of pipeline stages , under a given resource
constraint.

Existing optimizers are based on heuristics and take minutes or hours to run
for networks with about 10 to 50 layers [20]. Moreover, such a optimization
process has to be repeated whenever the compression decisions change, limiting
the search speed of Algorithm 1 and prohibiting the application to our case.

To address this challenge, the paper proposes building a proxy that predicts
the achievable throughput for a specific configuration based on network
characteristics and the optimizer used. For this purpose, a random forest was
selected as the proxy model.

Our approach is as follows: In the first few generations of the evolutionary
search, we use the solver from [20] to obtain the throughput of each design,
which we save and use to build a dataset. At a predetermined point in the
search, we use this dataset to train a random forest regressor, which is then
used to predict the throughput for subsequent designs during the evolutionary
algorithm. Training the regressor takes only a few minutes on a desktop CPU,
and its inference time is less than a second.

Our work differs from previous study [21], which aimed to build performance
predictors for general-purpose computing architectures, such as CPUs, GPUs, or
systolic array accelerators, where the architecture is fixed and only the
workload changes during the search. In contrast, our work targets dataflow
architectures, where the hardware is changed and customized for each design
point.

TABLE I: Performance and resource comparison with existing work on CNN-FPGA accelerators. In terms of the compression method, “Q” stands for quantization, “S” stands for sparsity (weights pruning) and “TD” stands for tensor decomposition. In terms of performance metrics, both FPS for batch size 1, and peak FPS for a large batch size (we and [22] use 256, [14] uses 1024, and others did not provide the details) are reported. As the pipeline is emptied between different batches, the pipeline depth impacts differently those two FPS metrics. The resources of Intel devices have already been converted to the equivalent resources on AMD devices, where 1 Intel ALM = 1.8 AMD LUT [23], and 1 Intel DSP = 2 AMD DSP [15]. | Dataflow | Network | Method | Acc. | Device |  Freq. (MHz) | URAM | BRAM | kLUT | DSP |  FPS batch 1 |  FPS peak
---|---|---|---|---|---|---|---|---|---|---|---|---
N3H-Core [24] | ✗ | ResNet-18 | Q | 70.4 | XC7Z045 | 100 | - | 541 | 153 | 900 | 31 | 123
FILM-QNN [25] | ✗ | ResNet-18 | Q | 70.5 | ZCU102 | 150 | - | 881 | 180 | 2092 | - | 215
MCBBS [3] | ✗ | VGG16 | Q, S | 64.8 | Arria GX1150 | 242 | - | 1785 | 605 | 2704 | 54 | -
StreamSVD [22] | ✓(partial) | ResNet-18 | Q, TD | 68.4 | XC7Z045 | 125 | - | 752 | - | 576 | - | 34
HPIPE [15] | ✓ | ResNet-50 | Q, S | 71.9 | S10 2800 | 580 | - | 11278 | 1064 | 10044 | 909 | 4550
FCMP [14] | ✓ | ResNet-50 | Q | 67.3 | Alveo U250 | 195 | 109 | 3870 | 1027 | 1611 | 526 | 2703
Ours | ✓ | ResNet-18 | Q, TD | 69.1 | Alveo U250 | 200 | 0 | 3564 | 1550 | 6394 | 1041 | 1138
Ours | ✓ | RepVGG-A0 | Q, TD | 71.5 | Alveo U250 | 200 | 0 | 3550 | 1555 | 5652 | 1162 | 1288

##  VI Experiments

Our experiment is carried out on a server using an Nvidia GTX 1080 Ti GPU for
accuracy queries and final model fine-tuning. The accelerator is evaluated
using Vivado 2020.1, targeting the AMD Alveo U250 device.

###  VI-A Benchmarks

As a case study, we focused on the ImageNet dataset and evaluated two state-
of-the-art models, ResNet-18 and RepVGG-A0 [26]. ResNet-18 features
representative residual block designs, while RepVGG-A0 is the latest model
from the VGG family. Both models were quantized to 8-bit Block Floating Point
(BFP) format [27]. Table II summarizes the results of our investigation. Our
proposed method successfully produced a model with a significantly reduced
number of parameters while maintaining an accuracy loss of less than 0.4pp
compared to a model that uses 8-bit BFP for both activations and weights.

TABLE II: Compression results on two CNN benchmarks trained on ImageNet dataset. BitOPs are counted as $2\times W\times A$ MACs. Models have been fine-tuned after the compression. Throughput is on Alveo U250 with a batch size of 1.  | Precision | Float32 | BFP-W8A8 | BFP-W8A8
---|---|---|---|---
| Decomposed | ✗ | ✗ | ✓
ResNet-18 | Top-1 Accuracy (%) | 69.7 | 69.3 | 69.1
Memory Size (Mb) | 374 | 94 | 35
BitOPs (G) | 3715 | 232 | 97
Throughput (FPS) | N/A | 702 | 1041
RepVGG-A0 | Top-1 Accuracy (%) | 72.4 | 71.9 | 71.5
Memory Size (Mb) | 266 | 67 | 35
BitOPs (G) | 2789 | 174 | 87
Throughput (FPS) | N/A | 864 | 1162

###  VI-B Performance Results

We used the proposed approach to generate designs and compared their
performance against state-of-the-art work that targets the same task i.e.
ImageNet classification using similar type models. Even though a direct
comparison with those approaches is not possible as each one utilises a
different model and device, it is useful to position the work against the
state-of-the-art on the task of ImageNet classification within the space of
achieved throughput and accuracy. The results are shown in Table I. The
produced designs by the proposed approach outperform significantly, non-
dataflow designs [24, 25, 3] with respect to both peak FPS, and latency
(inverse of FPS for batch size 1). In terms of FPS/DSP, our designs outperform
non-dataflow designs between 1.73$\times$ and 10.29$\times$.

With respect to dataflow architectures, StreamSVD [22] utilises tensor
decomposition based on SVD only to compress the weights of the model, which is
the closest approach to our work. Their work is a partial dataflow design as
device reconfiguration is required to overcome the resource constraint. When
compared with them, our work achieves 3.02$\times$ peak FPS/DSP. HPIPE [15]
utilises weight sparsity and we are outperforming them in terms of batch size
1 but not peak performance because their design is clocking at a much higher
frequency than ours, 580 MHz versus 200 MHz. FCMP is based on FINN [14] and
exploits binary quantization. Because of the binarization, their design
utilizes much fewer DSPs than ours but their classification accuracy is lower
than ours by 1.8pp on ResNet. Overall, the results show that our Mixed-TD
approach can lead to designs with competitive performance with similar task
accuracy on ImageNet.

Figure 5: Compare different tensor decomposition methods on ResNet-18. The
searching is without the involvement of the performance predictor. Curves do
not start from zero points as the initialisation is required to obtain the
first generation of valid design points. Figure 6: Compare the searching
results on ResNet-18, Mixed-TD, when using the predictor to estimate
throughput v.s. using MACs for the estimation. The first few steps of the
predictor run are slow as the solver [20] is launched to gather the training
data. Our predictor converges to a design point that better meets the target.

###  VI-C Ablation Studies

To better understand the individual contributions of the two main components
of the work, mixed tensor decomposition and random-forest-based predictor,
ablation studies were conducted. These studies allowed us to analyze the
impact of each component on the overall performance of the system.

The top-1 accuracy achieved over time when SVD-only, CPD-only, and the
proposed Mixed-TD approaches are used for the decomposition of the weights
tensors are illustrated in Fig. 5. As the Mixed-TD approach explores a larger
design space, the benefits in accuracy are only observed after the 90-hour
mark.

We investigated the performance benefits of other decomposition methods such
as Tensor Train [28] and Tensor Ring [29] into our Mixed-TD algorithm, but we
did not observe any further gains to the benchmarks that we investigated.

We have evaluated the performance of our proposed performance predictor for
accelerating the design space exploration and compared it to a baseline
approach that uses the number of Multiply-Accumulate (MAC) operations to guide
the exploration. As shown in Fig. 6, while the MAC-based approach is
$10\times$ faster, it leads to designs with significantly lower throughput
than the target. On the other hand, our performance predictor converges to a
design point that better meets the target throughput, resulting in significant
time savings compared to a full optimization process for mapping the model to
an FPGA. Please note that the full optimization process can only explore 22
designs/hour and it takes more than 150 hours to converge, where with the help
of the predictor, we are able to explore 4969 designs/hour and the searching
converges with less than 20 hours.

Furthermore, in our experiment, the target throughput is 1079 FPS, and our
predictor identifies the design point with 1041 FPS, which is only $3.5\%$
lower. To build the predictor, we chose a random forest due to its fast
training time, typically taking only a few minutes, as opposed to other
options such as Graph Convolutional Network [21] which can be relatively slow
to train.

##  VII Conclusion

The paper presents a novel method, called Mixed-TD, for fine-grained and
layer-specific model compression. This approach addresses the on-chip memory
limitations of dataflow architecture accelerators. Mixed-TD achieves
substantial weight compression while preserving high accuracy and considering
the target hardware. To efficiently navigate the extended design space, we
introduced an evolutionary search with a throughput predictor based on a
random forest. The paper demonstrates the benefits of tensor decomposition
methods in the space of mapping CNN models onto FPGAs, as well as the need for
proxies in order to navigate quickly the large design space.

## Acknowledgement

For the purpose of open access, the author(s) has applied a Creative Commons
Attribution (CC BY) license to any Accepted Manuscript version arising.

## References

  * [1] S. I. Venieris, A. Kouris, and C.-S. Bouganis, “Toolflows for mapping convolutional neural networks on fpgas: A survey and future directions,” _arXiv preprint arXiv:1803.05900_ , 2018.
  * [2] E. Wang, J. J. Davis, R. Zhao, H.-C. Ng, X. Niu, W. Luk, P. Y. Cheung, and G. A. Constantinides, “Deep neural network approximation for custom hardware: Where we’ve been, where we’re going,” _ACM Computing Surveys (CSUR)_ , vol. 52, no. 2, pp. 1–39, 2019.
  * [3] L. Liu and S. Brown, “Leveraging fine-grained structured sparsity for cnn inference on systolic array architectures,” in _2021 31st International Conference on Field-Programmable Logic and Applications (FPL)_. IEEE, 2021, pp. 301–305.
  * [4] J. Meng, S. K. Venkataramanaiah, C. Zhou, P. Hansen, P. Whatmough, and J.-s. Seo, “Fixyfpga: Efficient fpga accelerator for deep neural networks with high element-wise sparsity and without external memory access,” in _2021 31st International Conference on Field-Programmable Logic and Applications (FPL)_. IEEE, 2021, pp. 9–16.
  * [5] S. Rasoulinezhad, H. Zhou, L. Wang, and P. H. Leong, “Pir-dsp: An fpga dsp block architecture for multi-precision deep neural networks,” in _2019 IEEE 27th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM)_. IEEE, 2019, pp. 35–44.
  * [6] T. Alonso, L. Petrica, M. Ruiz, J. Petri-Koenig, Y. Umuroglu, I. Stamelos, E. Koromilas, M. Blott, and K. Vissers, “Elastic-df: Scaling performance of dnn inference in fpga clouds through automatic partitioning,” _ACM Transactions on Reconfigurable Technology and Systems (TRETS)_ , vol. 15, no. 2, pp. 1–34, 2021.
  * [7] C. Latotzke, T. Ciesielski, and T. Gemmeke, “Design of high-throughput mixed-precision cnn accelerators on fpga,” _arXiv preprint arXiv:2208.04854_ , 2022.
  * [8] M. Heath, A. Laub, C. Paige, and R. Ward, “Computing the singular value decomposition of a product of two matrices,” _SIAM Journal on Scientific and Statistical Computing_ , vol. 7, no. 4, pp. 1147–1159, 1986.
  * [9] T. G. Kolda and B. W. Bader, “Tensor decompositions and applications,” _SIAM review_ , vol. 51, no. 3, pp. 455–500, 2009.
  * [10] J. Kossaifi, Y. Panagakis, A. Anandkumar, and M. Pantic, “Tensorly: Tensor learning in python,” _arXiv preprint arXiv:1610.09555_ , 2016.
  * [11] Z. Yu and C.-S. Bouganis, “Svd-nas: Coupling low-rank approximation and neural architecture search,” in _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2023, pp. 1503–1512.
  * [12] “The tensor network.” [Online]. Available: https://tensornetwork.org/diagrams/#Penrose:19711
  * [13] S. I. Venieris and C.-S. Bouganis, “fpgaconvnet: A framework for mapping convolutional neural networks on fpgas,” in _2016 IEEE 24th Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM)_. IEEE, 2016, pp. 40–47.
  * [14] L. Petrica, T. Alonso, M. Kroes, N. Fraser, S. Cotofana, and M. Blott, “Memory-efficient dataflow inference for deep cnns on fpga,” in _2020 International Conference on Field-Programmable Technology (ICFPT)_. IEEE, 2020, pp. 48–55.
  * [15] M. Hall and V. Betz, “Hpipe: Heterogeneous layer-pipelined and sparse-aware cnn inference for fpgas,” _arXiv preprint arXiv:2007.10451_ , 2020.
  * [16] K. Goetschalckx, B. Moons, P. Wambacq, and M. Verhelst, “Efficiently combining svd, pruning, clustering and retraining for enhanced neural network compression,” in _Proceedings of the 2nd International Workshop on Embedded and Mobile Deep Learning_ , 2018, pp. 1–6.
  * [17] H. Yang, S. Gui, Y. Zhu, and J. Liu, “Automatic neural network compression by sparsity-quantization joint learning: A constrained optimization-based approach,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020, pp. 2178–2188.
  * [18] A. Montgomerie-Corcoran, S. I. Venieris, and C.-S. Bouganis, “Power-aware fpga mapping of convolutional neural networks,” in _2019 International Conference on Field-Programmable Technology (ICFPT)_. IEEE, 2019, pp. 327–330.
  * [19] X. Dai, P. Zhang, B. Wu, H. Yin, F. Sun, Y. Wang, M. Dukhan, Y. Hu, Y. Wu, Y. Jia _et al._ , “Chamnet: Towards efficient network design through platform-aware model adaptation,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 11 398–11 407.
  * [20] A. Montgomerie-Corcoran, Z. Yu, and C.-S. Bouganis, “Samo: Optimised mapping of convolutional neural networks to streaming architectures,” in _2022 32nd International Conference on Field-Programmable Logic and Applications (FPL)_. IEEE, 2022, pp. 418–424.
  * [21] L. Dudziak, T. Chau, M. Abdelfattah, R. Lee, H. Kim, and N. Lane, “Brp-nas: Prediction-based nas using gcns,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 10 480–10 490, 2020.
  * [22] Z. Yu and C.-S. Bouganis, “Streamsvd: Low-rank approximation and streaming accelerator co-design,” in _2021 International Conference on Field-Programmable Technology (ICFPT)_. IEEE, 2021, pp. 1–9.
  * [23] L. Shannon, V. Cojocaru, C. N. Dao, and P. H. Leong, “Technology scaling in fpgas: Trends in applications and architectures,” in _2015 IEEE 23rd Annual International Symposium on Field-Programmable Custom Computing Machines_. IEEE, 2015, pp. 1–8.
  * [24] Y. Gong, Z. Xu, Z. He, W. Zhang, X. Tu, X. Liang, and L. Jiang, “N3h-core: neuron-designed neural network accelerator via fpga-based heterogeneous computing cores,” in _Proceedings of the 2022 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays_ , 2022, pp. 112–122.
  * [25] M. Sun, Z. Li, A. Lu, Y. Li, S.-E. Chang, X. Ma, X. Lin, and Z. Fang, “Film-qnn: Efficient fpga acceleration of deep neural networks with intra-layer, mixed-precision quantization,” in _Proceedings of the 2022 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays_ , 2022, pp. 134–145.
  * [26] X. Ding, X. Zhang, N. Ma, J. Han, G. Ding, and J. Sun, “Repvgg: Making vgg-style convnets great again,” in _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2021, pp. 13 733–13 742.
  * [27] R. Banner, Y. Nahshan, and D. Soudry, “Post training 4-bit quantization of convolutional networks for rapid-deployment,” _Advances in Neural Information Processing Systems_ , vol. 32, 2019.
  * [28] I. V. Oseledets, “Tensor-train decomposition,” _SIAM Journal on Scientific Computing_ , vol. 33, no. 5, pp. 2295–2317, 2011.
  * [29] Q. Zhao, G. Zhou, S. Xie, L. Zhang, and A. Cichocki, “Tensor ring decomposition,” _arXiv preprint arXiv:1606.05535_ , 2016.