# Collaborative Pareto Set Learning in Multiple Multi-Objective Optimization
Problems
††thanks: This work was supported in part by the National Natural Science
Foundation of China (62006044) and in part by the Programme of Science and
Technology of Guangdong Province (202201010377).

Chikai Shang$1$, Rongguang Ye$2$, Jiaqi Jiang$1$, Fangqing Gu$1$∗ $1$School of
Mathematics and Statistics, Guangdong University of Technology, China
$2$Department of Computer Science and Engineering, Southern University of
Science and Technology, China<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Pareto Set Learning (PSL) is an emerging research area in multi-objective
optimization, focusing on training neural networks to learn the mapping from
preference vectors to Pareto optimal solutions. However, existing PSL methods
are limited to addressing a single Multi-objective Optimization Problem (MOP)
at a time. When faced with multiple MOPs, this limitation results in
significant inefficiencies and hinders the ability to exploit potential
synergies across varying MOPs. In this paper, we propose a Collaborative
Pareto Set Learning (CoPSL) framework, which learns the Pareto sets of
multiple MOPs simultaneously in a collaborative manner. CoPSL particularly
employs an architecture consisting of shared and MOP-specific layers. The
shared layers are designed to capture commonalities among MOPs
collaboratively, while the MOP-specific layers tailor these general insights
to generate solution sets for individual MOPs. This collaborative approach
enables CoPSL to efficiently learn the Pareto sets of multiple MOPs in a
single execution while leveraging the potential relationships among various
MOPs. To further understand these relationships, we experimentally demonstrate
that shareable representations exist among MOPs. Leveraging these shared
representations effectively improves the capability to approximate Pareto
sets. Extensive experiments underscore the superior efficiency and robustness
of CoPSL in approximating Pareto sets compared to state-of-the-art approaches
on a variety of synthetic and real-world MOPs. Code is available at
https://github.com/ckshang/CoPSL.

###### Index Terms:

Collaborative Pareto set learning, multi-objective optimization, multi-task
learning, deep learning.

##  I Introduction

Multi-objective Optimization Problems (MOPs) frequently occur in many real-
world applications, such as drug discovery [1], software engineering [2], etc.
In MOPs, the objectives are typically in conflict, rendering it impossible to
optimize all of them simultaneously with a single solution [3]. Instead, there
are Pareto optimal solutions, each representing a different trade-off among
the objectives. The primary goal in solving MOPs is to find a set of these
Pareto optimal solutions, known as the Pareto set. The image of the Pareto set
in the objective space is termed the Pareto front.

Over the past several decades, researchers have developed various population-
based Evolutionary Multi-objective Optimization (EMO) algorithms [4, 5, 6].
These algorithms are designed to optimize a population of solutions,
ultimately obtaining a well-distributed set of solutions along the Pareto
front. However, a key limitation of these population-based approaches is that
merely a finite set of approximate Pareto solutions can be obtained [7]. In
response to this limitation, a novel model-based approach known as Pareto Set
Learning (PSL) has recently been proposed [7, 8, 9]. PSL focuses on training a
neural network model to approximate the entire Pareto set of a given MOP. In
this approach, preference vectors serve as inputs to the model, which maps
them to Pareto solutions. These solutions are then used to calculate their
objective values, which are subsequently employed to refine the model via an
optimization process that utilizes a loss function, specifically the
aggregation functions used in decomposition-based EMO [5].

However, most current PSL approaches are designed to handle merely a single
MOP. When confronted with multiple MOPs, these approaches typically learn the
Pareto set of each MOP sequentially and independently, resulting in
significant inefficiencies in both time and memory. Furthermore, it is worth
noting that there exist latent synergies between different MOPs [10]. Such
sequential and independent learning approaches fail to capitalize on the
potential common relationships across these problems effectively. This neglect
is especially notable in model-based approaches, where there is a high
likelihood of shareable common representations across various MOPs.

In this paper, we propose a Collaborative Pareto Set Learning (CoPSL)
framework, which aims to solve multiple MOPs simultaneously in a collaborative
fashion. Drawing inspiration from hard parameter sharing in multi-task
learning [11], our CoPSL architecture is divided into layers that are either
shared across MOPs or dedicated to specific MOPs. This architecture enables
shared layers to collaboratively learn common relationships across MOPs, which
are then utilized by MOP-specific layers to generate the Pareto set for each
MOPs. It allows CoPSL to optimize multiple MOPs simultaneously with greater
efficiency in both time and memory.

To further explore the relationships among different MOPs, we delve into the
working mechanics of CoPSL through experimental investigations, demonstrating
the existence of common representations that can be shared among MOPs.
Leveraging these collaboratively learned common representations helps the
framework better approximate Pareto sets. To evaluate the efficacy of CoPSL,
we compare it with widely used EMO algorithms and state-of-the-art PSL
approaches. The experimental results demonstrate that the proposed CoPSL
operates with remarkable efficiency and outperforms the comparative algorithms
in terms of Pareto set approximation, albeit by a modest yet consistent
margin.

In summary, our key contributions are as follows:

  * •

We propose a collaborative Pareto set learning framework that collaboratively
learns common relationships across multiple MOPs through shared layers, which
are then utilized by MOP-specific layers to map them to solutions. This
collaborative learning enables more efficient handling of multiple MOPs in a
single run.

  * •

We study the working principle of CoPSL through experimental investigations.
It reveals that shareable representations exist within CoPSL, which helps the
framework better approximate the Pareto set of each MOP.

The remainder of this paper is organized as follows. In Section II, we
introduce the preliminary concepts. Section III and Section IV provide a
detailed description and further analysis of CoPSL. Section V presents
experimental studies comparing CoPSL with multiple state-of-the-art
algorithms. Finally, the conclusion and future work are discussed in Section
VI.

##  II Preliminaries

###  II-A Multi-objective Optimization

Without loss of generality, a multi-objective optimization problem with $m$
($m\geq 2$) objectives and $n$ decision variables can be formulated as
follows:

$\min_{\mathbf{x}\in\mathcal{X}}~{}\mathbf{F}(\mathbf{x})=\big{(}f_{1}(\mathbf{x}),\ldots,f_{m}(\mathbf{x})\big{)}^{T}$
(1)

where $\mathbf{x}$ is a solution in the decision space
$\mathcal{X}\subset\mathbb{R}^{n}$, and
$\mathbf{F}:\mathcal{X}\rightarrow\mathbb{R}^{m}$ is an objective function
vector. For an MOP, no single solution can optimize all objectives at the same
time, and we have to make a trade-off among them. We have the following
definitions of multi-objective optimization:

######  Definition 1 (Pareto Dominance)

Let $\mathbf{x},\mathbf{y}\in\mathcal{X}$, $\mathbf{x}$ is said to dominate
$\mathbf{y}$ (denoted as $\mathbf{x}\prec\mathbf{y}$) if and only if $\forall
i\in[m],f_{i}(\mathbf{x})\leq f_{i}(\mathbf{y})$ and $\exists
j\in[m],f_{j}(\mathbf{x})<f_{j}(\mathbf{y})$.

######  Definition 2 (Pareto Optimality)

A solution $\mathbf{x}^{\ast}\in\mathcal{X}$ is said to be Pareto optimal if
there is no other solution $\mathbf{x}\in\mathcal{X}$ such that
$\mathbf{x}\prec\mathbf{x}^{\ast}$.

######  Definition 3 (Pareto Set/Front)

The set consisting of all Pareto optimal solutions is called the Pareto set:
$PS=\\{\mathbf{x}\in\mathcal{X}\mid\forall\mathbf{y}\in\mathcal{X},\mathbf{y}\nprec\mathbf{x}\\}$
and the corresponding objective vector set of the PS is the Pareto front:
$PF=\\{\mathbf{F}(\mathbf{x})\mid\mathbf{x}\in PS\\}$.

In general, the goal of optimizing an MOP is to approximate both the Pareto
set and the Pareto front.

###  II-B Pareto Set Learning

Pareto set learning utilizes a neural network on user-defined preferences to
approximate the entire Pareto set [7, 8, 12]. Specifically, a preference
vector $\mathbf{p}=(p_{1},\ldots,p_{m})\in\mathbb{R}^{m}_{+}$ (with
$\sum_{i=1}^{m}p_{i}=1$) indicates the desired trade-off among objectives. The
network takes the preference vectors as inputs, with the network outputting
tailored solutions. Typically, an aggregation function is employed as the loss
function, enabling the optimization of MOPs through gradient-based training
[13]. Such an optimization process ensures that the generated solutions are
high quality and align closely with the defined preferences. This approach has
been successfully applied across various domains, including deep multi-task
learning [14, 15], neural multi-objective combinatorial optimization [12], and
so on.

###  II-C Multi-task Learning

Multi-task learning (MTL) leverages shared representations to facilitate
collaborative learning from multiple tasks. MTL architectures are generally
categorized into hard or soft parameter sharing techniques [11]. Hard
parameter sharing involves a shared encoder that branches out into task-
specific decoding heads [16, 17]. In contrast, soft parameter sharing assigns
separate parameter sets to each task and utilizes feature sharing mechanisms
to enable cross-task communication [18]. Due to the shared layer and
information sharing, hard parameter sharing methods often outperform their
single-task counterparts in both resource utilization efficiency and per-task
performance.

##  III Collaborative Pareto Set Learning

In the CoPSL framework, we simultaneously tackle $K$ MOPs, each characterized
by the same $m$ dimensions. This setup can be described as follows:

$\min_{\mathbf{x}_{i}\in\mathcal{X}_{i},i\in[K]}~{}\begin{cases}\mathbf{F}_{1}(\mathbf{x}_{1})=\big{(}f^{1}_{1}(\mathbf{x}_{1}),\ldots,f^{1}_{m}(\mathbf{x}_{1})\big{)}^{T}\\\
\mathbf{F}_{2}(\mathbf{x}_{2})=\big{(}f^{2}_{1}(\mathbf{x}_{2}),\ldots,f^{2}_{m}(\mathbf{x}_{2})\big{)}^{T}\\\
\cdots\\\
\mathbf{F}_{K}(\mathbf{x}_{K})=\big{(}f^{K}_{1}(\mathbf{x}_{K}),\ldots,f^{K}_{m}(\mathbf{x}_{K})\big{)}^{T}\end{cases}$
(2)

Figure 1: The fully connected neural network model of collaborative Pareto set
learning, which is structured with shared and MOP-specific parameters. This
allows it to concurrently learn from multiple MOPs with fewer parameters,
enabling more efficient training and inference.

We begin by outlining the basics of the CoPSL model (refer to Fig. 1). The
architecture of the CoPSL model is inspired by the concept of hard parameter
sharing in MTL. It aims to simultaneously learn the Pareto sets of multiple
MOPs with fewer parameters, enabling more efficient training and inference. In
the CoPSL model, the parameter set is divided into shared parameters and MOP-
specific parameters, as illustrated in Fig. 1. Let $h_{\phi}(\cdot)$ denote
the CoPSL model with parameters
$\phi=\\{\phi_{shr},\\{\phi^{i}_{spec}\\}_{i=1}^{K}\\}$, where $\phi_{shr}$
represents shared parameters and $\phi^{i}_{spec}$ represents specific
parameters for the $i$-th MOP ($i=1,\ldots,K$).

Input: number of MOPs $K$, number of iterations $T$, batch size of sampled
preferences $B$, learning rate $\eta$

1 Initialize parameters
$\phi=\\{\phi_{shr},\\{\phi^{i}_{spec}\\}_{i=1}^{K}\\}$

2 Initialize a weight vector $\mathbf{w}=(w_{1},\ldots,w_{K})$

3 for  _$t=1,\ldots,T$_ do

// Generate solution sets of multiple MOPs

4 Sample $B$ preference vectors $\\{\mathbf{p}^{b}\\}_{b=1}^{B}$ with a
Dirichlet distribution ${\rm Dir}(\bm{\alpha})$

5 Generate a set of solutions per MOP:
$\Big{\\{}\\{\mathbf{x}_{i}(\mathbf{p}^{b})\\}_{b=1}^{B}\Big{\\}}_{i=1}^{K}\leftarrow
h_{\phi}\Big{(}\\{\mathbf{p}^{b}\\}_{b=1}^{B}\Big{)}$

// Compute the losses of the solution sets

6 Compute loss per MOP:
$\mathcal{L}_{i}\leftarrow\frac{1}{B}\sum^{B}_{b=1}\ell_{i}(\mathbf{x}_{i}^{b}\mid\mathbf{p}^{b})~{}\forall
i\in[K]$

7 Compute total loss:
$\mathcal{L}_{tot}\leftarrow\sum_{i=1}^{K}w_{i}\mathcal{L}_{i}$

// Update the CoPSL model

8 Update the parameters:
$\phi\leftarrow\phi-\eta\nabla_{\phi}\mathcal{L}_{tot}$

9

10 end for

Output: CoPSL model
$\big{\\{}\mathbf{x}_{i}(\mathbf{p})\big{\\}}^{K}_{i=1}=h_{\phi}(\mathbf{p})$

Algorithm 1 Collaborative Pareto Set Learning

Algorithm 1 presents the pseudo-code of the proposed CoPSL framework. To
begin, the parameters $\phi$ is initialized along with a weight vector
$\mathbf{w}=(w_{1},\ldots,w_{K})$ (Line 1 and 2). In our implementation, we
set $w_{i}=1$ $\forall i$. Each iteration of CoPSL consists of the following
three steps:

Generate solution sets of multiple MOPs. Firstly, a batch of $B$ preference
vectors $\\{\mathbf{p}^{b}\\}_{b=1}^{B}$ is sampled from a Dirichlet
distribution with parameter vector $\bm{\alpha}$ (Line 5). The CoPSL model
takes these preferences as input and projects them into a higher-dimensional
space by shared parameters $\phi_{shr}$ to seek out the common relationships
among MOPs. Subsequently, each set of MOP-specific parameters
$\phi^{i}_{spec}$ maps these features to generate corresponding sets of
solutions $\\{\mathbf{x}_{i}(\mathbf{p}^{b})\\}_{b=1}^{B}$ for their
respective MOPs (Line 6).

Compute the losses of the solution sets. With the generated solution
$\mathbf{x}_{i}(\mathbf{p}^{b})$, we can calculate its corresponding objective
vector $\mathbf{F_{i}}\big{(}\mathbf{x}_{i}(\mathbf{p}^{b})\big{)}$. The
objective vectors are crucial for evaluating the solutions. By leveraging
these vectors, a loss function can be designed to accordingly update the
model, thereby refining its ability to approximate the Pareto set more
accurately. For that purpose, we explore four alternatives for CoPSL as
follows:

CoPSL-LS: CoPSL-LS employs the straightforward linear aggregation function
[14], which is particularly effective for MOPs with convex Pareto fronts [19].
The loss formula is expressed as:

$\ell_{i}(\mathbf{x}_{i}\mid\mathbf{p})=\sum^{m}_{j=1}p_{j}f^{i}_{j}(\mathbf{x}_{i})$
(3)

CoPSL-COSMOS: An extension of CoPSL-LS, CoPSL-COSMOS incorporates a cosine
similarity regularization to minimize the angle between the objective vector
of the generated solution and the preference vector [9]. The loss function for
CoPSL-COSMOS is:

$\ell_{i}(\mathbf{x}_{i}\mid\mathbf{p})=\sum^{m}_{j=1}p_{j}f^{i}_{j}(\mathbf{x})+\gamma\cos(\mathbf{p},\mathbf{F}_{i}(\mathbf{x}_{i}))$
(4)

where $\gamma>0$ is a penalty hyperparameter.

CoPSL-TCH: This method utilizes the Tchebycheff aggregation function, which is
designed to focus on the worst-case deviation from the ideal point, denoted as
$\mathbf{z}^{\ast}_{i}$, for the $i$-th objective vector [20]. Its loss
function is formulated as follows:

$\ell_{i}(\mathbf{x}_{i}\mid\mathbf{p})=\max_{1\leq j\leq
m}\\{p_{j}(f^{i}_{j}(\mathbf{x}_{i})-(z^{\ast}_{ij}-\epsilon))\\}$ (5)

where $\epsilon$ is a negligible positive value introduced to prevent the loss
from becoming zero. Notably, $\mathbf{z}^{\ast}_{i}$ is dynamically updated
based on the evaluated solutions during training.

CoPSL-MTCH: CoPSL-MTCH adopts the modified Tchebycheff aggregation function as
its loss function, as described in [7]. The formula is given by:

$\ell_{i}(\mathbf{x}_{i}\mid\mathbf{p})=\max_{1\leq j\leq
m}\\{\frac{1}{p_{j}}(f^{i}_{j}(\mathbf{x}_{i})-(z^{\ast}_{ij}-\epsilon))\\}$
(6)

Based on the aforementioned loss function $\ell_{i}(\cdot\mid\cdot)$, we can
compute the loss $\mathcal{L}_{i}$ for the $i$-th MOP (Line 8). In our
experiments detailed in Section V-C, we conduct a comparative analysis of the
four alternatives, both within and outside the CoPSL framework.

To evaluate the total loss across the solution sets of different MOPs, we
constructed a total loss function $\mathcal{L}_{tot}$ in a manner akin to MTL
(Line 9). It is defined as follows:

$\mathcal{L}_{tot}=\sum_{i=1}^{K}w_{i}\cdot\mathcal{L}_{i}=\sum_{i=1}^{K}w_{i}\cdot\frac{1}{B}\sum^{B}_{b=1}\ell_{i}(\mathbf{x}_{i}^{b}\mid\mathbf{p}^{b})$
(7)

Update the CoPSL model. The CoPSL model parameters $\phi$ enable to be updated
based on the total loss $\mathcal{L}_{tot}$ (Line 11). Particularly, the set
of MOP-specific parameters $\phi^{i}_{spec}$ for the $i$-th MOP is updated
according to the corresponding loss $\mathcal{L}_{i}$. This suggests that
$\phi^{i}_{spec}$ is fine-tuned based on its associated specific solution set.
The updated formula is given by:

$\phi^{i}_{spec}=\phi^{i}_{spec}-\eta\nabla_{\phi^{i}_{spec}}\mathcal{L}_{i}$
(8)

where $\eta$ denotes the learning rate. Regarding the shared parameters set
$\phi_{shr}$, it is updated by the aggregated loss
$\mathcal{L}_{tot}=\sum_{i=1}^{K}w_{i}\mathcal{L}_{i}$, indicating that
$\phi_{shr}$ collaboratively learns from all the Pareto sets across various
MOPs. The updated formula is as follows:

$\phi_{shr}=\phi_{shr}-\eta\sum^{K}_{i=1}w_{i}\nabla_{\phi_{shr}}\mathcal{L}_{i}$
(9)

Notably, this formula elucidates the essence of the collaborative learning
approach in CoPSL, specifically leveraging backpropagation informed by
insights from multiple MOPs and thereby harnessing the possible
interrelations.

Based on the above three steps, the CoPSL model achieves efficient training of
multiple MOPs simultaneously in a collaborative fashion.

CoPSL Scalability. Equation (9) reveals that when the gradients of MOPs are in
conflict, or when the magnitude of one MOP gradient overshadows others, the
resulting updates to the model may be suboptimal. In traditional MTL, various
techniques have been proposed to balance the gradient magnitudes by
dynamically adjusting the weight vector $\mathbf{w}$ [17, 21, 22]. Our
approach allows us to easily incorporate these balancing techniques following
the update of the CoPSL model (i.e., after Line 11 of Algorithm 1). Notably,
MOPs enable dynamic adjustments using indicators in addition to loss functions
(i.e., the aggregation functions) [23, 24]. These indicators, such as
hypervolume (HV) and inverted generational distance (IGD), provide a more
direct measure of the current state of the output solutions compared to loss
functions alone. They offer supplementary information beyond losses to better
guide the adjustment of the weight vector. We will discuss this as future work
in Section VI.

Differences with MTL. The primary distinctions between MTL and CoPSL lie in
the nature of their inputs and the setup of their shared layers. Specifically:

  * •

MTL typically processes image data as inputs. The image data for various tasks
may be similar or distinctly different, but it is rich in information. In
contrast, CoPSL utilizes preference vectors as inputs, which are essentially
low-dimensional vectors of weights. These preference vectors for different
MOPs are sampled in the same objective space, rendering them almost equivalent
and less informative compared to image data.

  * •

Hard parameter sharing models in MTL generally employ shared convolutional
layers that are adept at handling image inputs. This setup facilitates the
extraction of highly independent representations from the image data, which
can be shared across tasks to enhance performance. On the other hand, CoPSL
utilizes shared fully connected layers that receive the sampled preference
vectors as inputs, as depicted in Fig. 1. It remains to be seen whether these
fully connected layers are capable of learning the common relationships that
can be beneficially shared among MOPs.

To provide a deeper understanding of CoPSL, we will delve into a more thorough
analysis in the following section.

##  IV Understanding Shared Representations in CoPSL

Figure 2: A toy example to explore mechanisms of the CoPSL model. We train
several CoPSL model architectures, each sharing a different number of layers
for two MOPs. For each of these models, we plot their performance on each MOP
relative to the MOP-specific model. The results demonstrate that the presence
of shareable representations for preference vectors is more effectively
learned in a collaborative fashion.

Given the differences between MTL and CoPSL, we aim to investigate whether
CoPSL can learn shareable relationships across MOPs and deepen our
understanding of CoPSL’s mechanisms. To this end, we perform extensive
experimental analysis. Consider a simple experiment where we train a variety
of CoPSL model architectures with different numbers of shared layers, each
designed to tackle two arbitrary MOPs (e.g., synthetic two-dimensional
benchmarks F1 and F3 [7]). The upper part of Fig. 2 illustrates the spectrum
of model architectures that were constructed for this purpose. At one extreme,
we train two separate PSL models for each MOP, with no cross-talk between
them. At the other extreme, we train one generic model where all layers are
shared, and only the last layers are MOP-specific. We exhaustively enumerate
all possible architectures as depicted in the upper part of Fig. 2, and
evaluate their respective performance as shown in the lower part of Fig. 2.

The findings indicate that while there appears to be little correlation
between the two MOPs F1 and F3, employing CoPSL models with one or two shared
layers can still improve overall performance. It demonstrates that certain
representations of preferences (inputs) can be generalized and shared across
multiple MOPs, rather than being learned separately from scratch for each
individual MOP. Notably, this occurs even when the Pareto sets (outputs) are
irrelevant, revealing the existence of an effective shareable representation
space for preferences across diverse MOPs with limited inherent relation.
Leveraging shared knowledge across different MOPs is beneficial for better
approximating the Pareto sets.

Furthermore, we note that introducing only one shared layer improves the HV
metrics for both MOPs to a certain extent. However, increasing the number of
shared layers beyond one reveals a ‘seesaw’ pattern in the HV values for the
respective MOPs. This can be attributed to two primary reasons. Firstly, the
model’s input consists merely of two-dimensional preference vectors. In cases
where the dimensionality is low, the preference vectors present more
straightforward shared representations that a single shared layer can
effectively capture. Although additional shared layers have the potential to
learn more complex representations, they tend to diminish the model’s ability
to map these representations to Pareto solutions. Secondly, the disparity in
Pareto sets among various MOPs generates conflicting gradients. As delineated
in Equation (9), this conflict adversely affects the updates of shared
parameters $\phi_{shr}$. Consequently, as previously mentioned, the given
weight vectors face inherent limitations in optimizing for both MOPs
simultaneously.

Given that only two MOPs are considered in this section, we will further
explore the effectiveness of CoPSL by simultaneously tackling a greater number
of MOPs in Section V-C, including a suite of five to six MOPs.

##  V Experiments

TABLE I: Comparisons of EMO algorithms, PSL approaches, and our proposed algorithm on a suite of six synthetic benchmarks. The best results are highlighted in bold font. CoPSL not only runs significantly faster than the compared algorithms, but also slightly outperforms them in terms of approximation capability. | HV $\uparrow$ |  Runtime (s) $\downarrow$ |  FLOPs (K) $\downarrow$ |  Params (K) $\downarrow$
---|---|---|---|---
| F1 | F2 | F3 | F4 | F5 | F6
NSGA-II | 7.957e-01 | 7.922e-01 | 7.835e-01 | 7.843e-01 | 7.832e-01 | 8.098e-01 | 9.771e+00 | - | -
NSGA-III | 8.089e-01 | 8.106e-01 | 8.165e-01 | 8.014e-01 | 8.139e-01 | 8.295e-01 | 9.502e+00 | - | -
MOEA/D | 8.175e-01 | 8.192e-01 | 8.164e-01 | 8.023e-01 | 8.060e-01 | 8.276e-01 | 2.868e+01 | - | -
PSL-TCH | 8.578e-01 | 8.651e-01 | 8.002e-01 | 8.235e-01 | 8.299e-01 | 8.456e-01 | 1.003e+01 | 8.110e+02 | 4.086e+02
CoPSL-TCH | 8.600e-01 | 8.651e-01 | 8.019e-01 | 8.247e-01 | 8.306e-01 | 8.465e-01 | 8.364e+00 | 8.059e+02 | 4.048e+02

(a) F1

(b) F2

(c) F3

(d) F4

(e) F5

(f) F6

Figure 3: The log HV difference curves of four algorithms for six different
problems. The solid line is the mean value averaged over 10 independent runs
for each algorithm. The labels of the algorithms can be found in Subfig. (a).

###  V-A Experimental Setups

Baselines. We consider three popular model-free methods and four state-of-the-
art model-based approaches as baselines. 1) EMO algorithms: NSGA-II [4], NSGA-
III [6], MOEA/D [5], which are implemented using pymoo. 2) PSL algorithms:
PSL-LS [14], COSMOS [9], PSL-TCH [20], PSL-MTCH [7], which are executed in
Pytorch.

Benchmarks and Real-World Problems. The algorithms are first compared on a
suite of six synthetic two-dimensional test problems, including F1 to F6 in
[7]. Then experiments are also conducted on five different three-dimensional
real-world multi-objective engineering design problems [25], including 1) Two
bar truss design (RE31); 2) Welded beam design (RE32); 3) Disc brake design
(RE33); 4) Vehicle crashworthiness design (RE34); 5) Rocket injector design
(RE37). Comprehensive details regarding these problems are provided in the
Appendix.

Notably, in contrast to non-collaborative approaches that run separate
populations/models for each problem, CoPSL concurrently processes all problems
from a given problem suite in the same experimental run using a single model.
Additionally, due to limitations within the inputs of the CoPSL model, the
number of objective dimensions remained consistent across all problems in each
set of experiments.

Evaluation Metrics. The evaluation of algorithmic performance focuses on three
primary aspects. 1) Approximation quality of the Pareto set. We employ the HV
indicator, which comprehensively evaluates the diversity and convergence of
the solution set. For a nuanced analysis, we further examine the log HV
difference (i.e., the logarithm of the difference between the HV value of the
true/approximate PF and the learned PF) to evaluate the gap between the
true/approximate and the learned Pareto front. 2) Algorithmic efficiency. We
measure this in terms of the actual runtime required to process a problem
suite, which serves as a practical performance metric. For model-based
algorithms, we further delve into the computational time complexity by
analyzing Floating-point Operations (FLOPs), which offers a theoretical
perspective on the computational effort required. 3) Space complexity of
algorithms. Regarding model-based algorithms, we consider the total number of
parameters within the model, denoted as Params, as a measure of the
theoretical computational space complexity.

Parameter settings. For each experiment, all algorithms are executed 10 times
independently. Specifically, we set the number of iterations $T$ to 500 and
utilize the Adam optimizer [13] with its default parameters for model-based
algorithms. The learning rate $\eta$ is set to $10^{-3}$ and the
hyperparameter $\gamma$ in COSMOS is set to 100. Additionally, for our
proposed approach, we fix the weight vector $\mathbf{w}$ as an all-one vector.
The detailed exposition of the model architecture is provided in the Appendix.

TABLE II: Comparisons of PSL approaches and CoPSL on six synthetic problems. The best results are highlighted in bold font. CoPSL not only runs significantly faster than its counterparts, but also slightly outperforms them in approximation capability. | HV $\uparrow$ | Runtime (s) $\downarrow$
---|---|---
| F1 | F2 | F3 | F4 | F5 | F6
PSL-LS | 8.025e-01 | 8.579e-01 | 7.563e-01 | 7.934e-01 | 8.045e-01 | 8.347e-01 | 9.823e+00
CoPSL-LS | 8.305e-01 | 8.586e-01 | 7.522e-01 | 7.950e-01 | 8.072e-01 | 8.355e-01 | 8.346e+00
PSL-COSMOS | 3.758e-01 | 1.921e-01 | 2.004e-01 | 2.622e-01 | 4.362e-01 | 1.302e-01 | 1.078e+01
CoPSL-COSMOS | 3.971e-01 | 2.240e-01 | 2.080e0-01 | 3.149e-01 | 4.384e-01 | 1.256e-01 | 9.207e+00
PSL-TCH | 8.595e-01 | 8.660e-01 | 8.035e-01 | 8.266e-01 | 8.330e-01 | 8.467e-01 | 1.010e+01
CoPSL-TCH | 8.605e-01 | 8.654e-01 | 8.051e-01 | 8.287e-01 | 8.341e-01 | 8.472e-01 | 8.491e+00
PSL-MTCH | 8.249e-01 | 8.676e-01 | 7.881e-01 | 7.836e-01 | 7.947e-01 | 8.444e-01 | 1.028e+01
CoPSL-MTCH | 8.326e-01 | 8.681e-01 | 7.910e-01 | 7.850e-01 | 8.019e-01 | 8.456e-01 | 8.638e+00
TABLE III: Comparisons of PSL approaches and CoPSL on five real-world problems. The best results are highlighted in bold font. CoPSL not only runs significantly faster than its counterparts, but also marginally outperforms them in approximation capability. | HV $\uparrow$ | Runtime (s) $\downarrow$
---|---|---
| RE31 | RE32 | RE33 | RE34 | RE37
PSL-LS | 1.325e+00 | 1.200e+00 | 5.821e-01 | 4.085e-02 | 9.087e-01 | 8.448e+00
CoPSL-LS | 1.326e+00 | 1.223e+00 | 5.862e-01 | 4.085e-02 | 8.991e-01 | 6.747e+00
PSL-COSMOS | 1.327e+00 | 1.150e+00 | 5.183e-01 | 4.085e-02 | 7.130e-01 | 8.871e+00
CoPSL-COSMOS | 1.328e+00 | 1.208e+00 | 5.580e-01 | 4.085e-02 | 8.136e-01 | 7.535e+00
PSL-TCH | 1.325e+00 | 1.211e+00 | 6.925e-01 | 4.085e-02 | 1.030e+00 | 8.496e+00
CoPSL-TCH | 1.326e+00 | 1.235e+00 | 6.871e-01 | 4.085e-02 | 1.022e+00 | 6.949e+00
PSL-MTCH | 1.325e+00 | 1.134e+00 | 6.358e-01 | 4.085e-02 | 1.025e+00 | 8.566e+00
CoPSL-MTCH | 1.330e+00 | 1.222e+00 | 6.800e-01 | 4.085e-02 | 1.027e+00 | 7.052e+00

(a) F1

(b) Two Bar Truss Design (RE31)

(c) Welded Beam Design (RE32)

Figure 4: The log HV difference curves of four algorithms for three problems
over 10 independent runs. The labels of the algorithms can be found in Subfig.
(a).

###  V-B Comparisons with Base Methods

To demonstrate the efficacy of our proposed methodology, we first conducted
comparative analyses against traditional EMO algorithms and an advanced PSL
algorithm on a suite of synthetic benchmarks (i.e., F1 to F6). For these
experiments, we standardized the population size for EMOs and the batch size
$B$ for model-based algorithms to 15. The results, detailed in Table I, span
multiple performance metrics of these methods, while Fig. 3 illustrates the
log HV difference to the approximate Pareto front during the optimization
process.

CoPSL exhibits superior efficiency compared to all the baselines and
demonstrates slightly better performance overall in terms of approximation
capability. To be specific, CoPSL significantly outperforms the comparison
algorithms in runtime performance and boasts lower theoretical FLOPs and
Params than its model-based counterparts. This underscores the efficiency of
CoPSL in concurrently learning multiple MOPs. Furthermore, CoPSL shows
improvement over all the baselines in all but one MOP (i.e., F3), where it
still maintains a slight edge over PSL. As depicted in Fig. 3, CoPSL converges
slightly faster than the traditional single-MOP PSL approach. This indicates
that CoPSL can effectively enhance both the approximation of the Pareto set
and the robustness of the model. Additionally, it can be observed that EMOs
can outpace traditional PSL in runtime when running with small population
sizes. CoPSL addresses this limitation, extending the upper bounds of model-
based Pareto set learning.

###  V-C Comparisons with SOTA PSL Methods

(a) PSL-LS

(b) PSL-COSMOS

(c) PSL-TCH

(d) PSL-MTCH

(e) CoPSL-LS

(f) CoPSL-COSMOS

(g) CoPSL-TCH

(h) CoPSL-MTCH

Figure 5: Pareto front comparisons on synthetic benchmark F1. The top part
represents PSL, and the bottom part represents CoPSL. Each set of top and
bottom comparisons is derived from the same random seed.

To further explore the effectiveness of learning from multiple Pareto sets
concurrently in a collaborative manner, we executed a thorough comparative
analysis of PSL and CoPSL across multiple synthetic benchmarks and real-world
application problems. These comparisons can be viewed as ablation studies,
where each CoPSL alternative was benchmarked against a corresponding PSL setup
paired with the same loss function. For consistency, we standardized the batch
size $B=30$. The results, encapsulated in Tables II and III, showcase the HV
indicator and runtime for synthetic and real-world problem sets, respectively.
Furthermore, Fig. 4 presents the log HV difference to the approximate Pareto
front for a subset of the problems during the optimization process.

The results indicate that CoPSL significantly outperforms PSL in terms of
efficiency. Employing identical loss functions, CoPSL generally exhibits a
marginal but consistent advantage over PSL in both synthetic and real-world
problem suites with respect to their approximation capabilities. Specifically,
within each problem suite, CoPSL surpasses its PSL counterpart in all but one
instance or, in some cases, across the board. Moreover, as depicted in Fig. 4,
CoPSL demonstrates more robust updates compared to PSL. PSL, particularly PSL-
MTCH as shown in Fig. 4(b), are susceptible to fluctuations, whereas CoPSL
maintains a steadier progression. This suggests that CoPSL is capable of
learning more potent shared representations among MOPs, thus enhancing its
ability to approximate the Pareto set and improving its generalizability.

Additionally, Fig. 5 displays the non-dominated solutions generated by PSL and
CoPSL on benchmark F1. The lower section of the figure represents the CoPSL
alternatives, while the upper section represents the PSL counterparts. Each
top- and bottom-set comparison originates from the same random seed. The
results presented in Fig. 5 demonstrate that CoPSL is significantly better
than PSL, as the non-dominated solutions of CoPSL are much closer to the
Pareto set and exhibit a superior distribution. It highlights the advantages
of learning across multiple MOPs and the benefits of shared representations.
More experimental results are provided in the Appendix.

It should be highlighted that there appears to be little correlation between
the real-world problems in the nature of questions and the shape of the Pareto
sets. Nonetheless, CoPSL is still capable of concurrently learning an entire
suite of MOPs while also learning shared common representations that enhance
overall performance. This underscores the remarkable potential of CoPSL to
process numerous MOPs simultaneously and still derive valuable shared
representations.

##  VI Conclusion and Future Work

In this paper, we proposed the CoPSL framework, a collaborative approach
designed to tackle multiple MOPs simultaneously. Inspired by MTL, the
framework incorporates an architecture with both shared and MOP-specific
layers. The shared layers focus on uncovering commonalities among various
MOPs, while the MOP-specific layers are tailored to learn the Pareto set for
each MOP. We experimentally revealed that the shared layers excel at capturing
shared common representations of preference vectors across different MOPs.
This ability facilitates a more generalized representation space, ultimately
improving the approximation of the Pareto sets. We compared CoPSL with state-
of-the-art EMO and PSL algorithms on a variety of both synthetic and real-
world MOPs. Extensive experiments demonstrated CoPSL’s superior efficiency and
robustness in approximating Pareto sets.

Furthermore, we emphasized the dynamic adjustability of the weight vector in
the CoPSL framework during the optimization process. Drawing inspiration from
indicator-based EMO algorithms, we suggest leveraging additional indicators in
multi-objective optimization to accurately reflect the current state of the
solution sets. By dynamically adjusting the weight vector through the
introduction of evaluation indicators, we anticipate an enhanced capability to
handle multiple MOPs more effectively in CoPSL. This direction holds promise
for optimizing the CoPSL framework’s performance in solving multiple complex
MOPs concurrently, and we believe it merits further exploration in future
research.

## References

  * [1] S. Dara, S. Dhamercherla, S. S. Jadav, C. M. Babu, and M. J. Ahsan, “Machine learning in drug discovery: a review,” Artificial Intelligence Review, vol. 55, no. 3, pp. 1947–1999, 2022.
  * [2] M. Harman and X. Yao, “Software module clustering as a multi-objective search problem,” IEEE Transactions on Software Engineering, vol. 37, no. 2, pp. 264–282, 2010.
  * [3] C. Gong, Y. Nan, L. M. Pang, Q. Zhang, and H. Ishibuchi, “Effects of including optimal solutions into initial population on evolutionary multiobjective optimization,” in Proceedings of the Genetic and Evolutionary Computation Conference, pp. 661–669, 2023.
  * [4] K. Deb, A. Pratap, S. Agarwal, and T. Meyarivan, “A fast and elitist multiobjective genetic algorithm: Nsga-ii,” IEEE transactions on evolutionary computation, vol. 6, no. 2, pp. 182–197, 2002.
  * [5] Q. Zhang and H. Li, “Moea/d: A multiobjective evolutionary algorithm based on decomposition,” IEEE Transactions on evolutionary computation, vol. 11, no. 6, pp. 712–731, 2007.
  * [6] K. Deb and H. Jain, “An evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, part i: solving problems with box constraints,” IEEE transactions on evolutionary computation, vol. 18, no. 4, pp. 577–601, 2013.
  * [7] X. Lin, Z. Yang, X. Zhang, and Q. Zhang, “Pareto set learning for expensive multi-objective optimization,” in Advances in Neural Information Processing Systems, vol. 35, pp. 19231–19247, 2022.
  * [8] X. Lin, X. Zhang, Z. Yang, and Q. Zhang, “Evolutionary pareto set learning with structure constraints,” arXiv preprint arXiv:2310.20426, 2023.
  * [9] M. Ruchte and J. Grabocka, “Scalable pareto front approximation for deep multi-objective learning,” in 2021 IEEE international conference on data mining (ICDM), pp. 1306–1311, 2021.
  * [10] K. K. Bali, Y.-S. Ong, A. Gupta, and P. S. Tan, “Multifactorial evolutionary algorithm with online transfer parameter estimation: Mfea-ii,” IEEE Transactions on Evolutionary Computation, vol. 24, no. 1, pp. 69–83, 2019.
  * [11] S. Vandenhende, S. Georgoulis, W. Van Gansbeke, M. Proesmans, D. Dai, and L. Van Gool, “Multi-task learning for dense prediction tasks: A survey,” IEEE transactions on pattern analysis and machine intelligence, vol. 44, no. 7, pp. 3614–3633, 2021.
  * [12] Q. Z. Xi Lin, Zhiyuan Yang, “Pareto set learning for neural multi-objective combinatorial optimization,” in International Conference on Learning Representations, 2022.
  * [13] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980, 2014.
  * [14] A. Navon, A. Shamsian, E. Fetaya, and G. Chechik, “Learning the pareto front with hypernetworks,” in International Conference on Learning Representations, 2021.
  * [15] L. P. Hoang, D. D. Le, T. A. Tuan, and T. N. Thang, “Improving pareto front learning via multi-sample hypernetworks,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, pp. 7875–7883, 2023.
  * [16] I. Kokkinos, “Ubernet: Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6129–6138, 2017.
  * [17] Z. Chen, V. Badrinarayanan, C.-Y. Lee, and A. Rabinovich, “GradNorm: Gradient normalization for adaptive loss balancing in deep multitask networks,” in Proceedings of the 35th International Conference on Machine Learning, vol. 80, pp. 794–803, 2018.
  * [18] I. Misra, A. Shrivastava, A. Gupta, and M. Hebert, “Cross-stitch networks for multi-task learning,” in Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3994–4003, 2016.
  * [19] S. P. Boyd and L. Vandenberghe, Convex optimization.  Cambridge university press, 2004.
  * [20] T. A. Tuan, L. P. Hoang, D. D. Le, and T. N. Thang, “A framework for controllable pareto front learning with completed scalarization functions and its applications,” Neural Networks, vol. 169, pp. 257–273, 2024.
  * [21] S. Liu, E. Johns, and A. J. Davison, “End-to-end multi-task learning with attention,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1871–1880, 2019.
  * [22] M. Guo, A. Haque, D.-A. Huang, S. Yeung, and L. Fei-Fei, “Dynamic task prioritization for multitask learning,” in Proceedings of the European conference on computer vision (ECCV), pp. 270–287, 2018.
  * [23] T. Wagner, N. Beume, and B. Naujoks, “Pareto-, aggregation-, and indicator-based methods in many-objective optimization,” in International conference on evolutionary multi-criterion optimization, pp. 742–756, 2007.
  * [24] Y. Tian, R. Cheng, X. Zhang, F. Cheng, and Y. Jin, “An indicator-based multiobjective evolutionary algorithm with reference point adaptation for better versatility,” IEEE Transactions on Evolutionary Computation, vol. 22, no. 4, pp. 609–622, 2017.
  * [25] R. Tanabe and H. Ishibuchi, “An easy-to-use real-world multi-objective optimization problem suite,” Applied Soft Computing, vol. 89, p. 106078, 2020.

## Benchmarks and Real-World Problems

TABLE IV: Problem information and reference point for both synthetic benchmarks and real-world engineering design problems. Problem | $m$ | $n$ | Reference Point
---|---|---|---
F1 | 2 | 6 | (1.1,1.1)
F2 | 2 | 6 | (1.1,1.1)
F3 | 2 | 6 | (1.1,1.1)
F4 | 2 | 6 | (1.1,1.1)
F5 | 2 | 6 | (1.1,1.1)
F6 | 2 | 6 | (1.1,1.1)
Two Bar Truss Design | 3 | 3 | (550,9.9e+6,2.2e+7)
Welded Beam Design | 3 | 4 | (38.83,1.9e+4,4.6e+8)
Disc Brake Design | 3 | 4 | (5.83,3.43,27.5)
Vehicle Crashworthiness Design | 3 | 5 | (1865,12.98,0.32)
Rocket Injector Design | 3 | 4 | (1.08,1.05,1.08)

(a) PSL-LS

(b) PSL-COSMOS

(c) PSL-TCH

(d) PSL-MTCH

(e) CoPSL-LS

(f) CoPSL-COSMOS

(g) CoPSL-TCH

(h) CoPSL-MTCH

Figure 6: Pareto front comparisons on the disc brake design problem (i.e.,
RE33). The top part represents PSL, and the bottom part represents CoPSL. Each
set of top and bottom comparisons is derived from the same random seed.

To better evaluate our proposed CoPSL approach, we conducted experiments on
six synthetic benchmarks [7] and five real-world multi-objective engineering
design problems [25]. We utilized the approximated Pareto fronts as the actual
fronts to compute the HV values in both suites of problems. The problem
information and reference points can be found in Table IV, where the reference
points are used to compute the HV indicator.

## Model Structures

In this work, we employ fully connected networks with varying layers as our
experimental neural networks. Notably, we explored different architectural
layers in Sections IV and V.

For our primary experiments detailed in Section V, we utilized a model with 2
hidden layers, each comprising 256 hidden units, designated as the PSL model.
The architecture of this model is outlined as follows:

$\displaystyle\text{Input}~{}\mathbf{p}$
$\displaystyle\rightarrow\text{Linear}(m,256)\rightarrow\text{ReLU}$
$\displaystyle\rightarrow\text{Linear}(256,256)\rightarrow\text{ReLU}$
$\displaystyle\rightarrow\text{Linear}(256,n)\rightarrow\text{Sigmoid}\rightarrow\text{Output}~{}\mathbf{x}(\mathbf{p})$

To ensure a fair comparison, we employed a model in CoPSL that is almost
equivalent to this PSL model. The CoPSL model comprises one shared layer and
two MOP-specific layers, described as follows:

Shared layer:
$\displaystyle\text{Input}~{}\mathbf{p}\rightarrow\text{Linear}(m,256)\rightarrow\text{ReLU}$
MOP-specific layers:
$\displaystyle\text{Linear}(256,256)\rightarrow\text{ReLU}$
$\displaystyle\rightarrow$
$\displaystyle\text{Linear}(256,n)\rightarrow\text{Sigmoid}$
$\displaystyle\rightarrow$
$\displaystyle\text{Output}~{}\mathbf{x}(\mathbf{p})$

In the experimental investigations in Section IV, we utilized a model with 3
hidden layers, each containing 180 hidden units, as the PSL model. The
structure of this model is depicted as follows:

$\displaystyle\text{Input}~{}\mathbf{p}$
$\displaystyle\rightarrow\text{Linear}(m,180)\rightarrow\text{ReLU}$
$\displaystyle\rightarrow\text{Linear}(180,180)\rightarrow\text{ReLU}$
$\displaystyle\rightarrow\text{Linear}(180,180)\rightarrow\text{ReLU}$
$\displaystyle\rightarrow\text{Linear}(180,n)\rightarrow\text{Sigmoid}\rightarrow\text{Output}~{}\mathbf{x}(\mathbf{p})$

We utilized a CoPSL model nearly equivalent to this PSL model as well. Given
that all possible enumerations of the CoPSL architecture have been detailed in
Fig. 2, we will not repeat them here.

## Additional Experimental Results

For a more comprehensive comparison, we provide the non-dominated solutions
generated by PSL and CoPSL on the disc brake design problem (i.e., RE33) in
Fig. 6. The lower part of the figure represents the CoPSL alternatives, while
the upper part represents the PSL counterparts. Each top- and bottom-set
comparison originates from the same random seed. Compared with PSL-COSMOS and
CoPSL-COSMOS (i.e., Subfig. 6(b) and 6(f)), as well as PSL-MTCH and CoPSL-MTCH
(i.e., Subfig. 6(d) and 6(h)), the non-dominated solutions of CoPSL are
significantly closer to the Pareto set and have better diversity. These
experimental results emphasize the advantages of robust shared representations
that are learned in a collaborative manner.


# Recurrent Neural Networks Hardware Implementation on FPGA

Andre Xian Ming Chang, Berin Martini, Eugenio Culurciello  Department of
Electrical and Computer Engineering
Purdue University
West Lafayette, USA
<EMAIL_ADDRESS>
###### Abstract

Recurrent Neural Networks (RNNs) have the ability to retain memory and learn
data sequences. Due to the recurrent nature of RNNs, it is sometimes hard to
parallelize all its computations on conventional hardware. CPUs do not
currently offer large parallelism, while GPUs offer limited parallelism due to
sequential components of RNN models. In this paper we present a hardware
implementation of Long-Short Term Memory (LSTM) recurrent network on the
programmable logic Zynq 7020 FPGA from Xilinx. We implemented a RNN with $2$
layers and $128$ hidden units in hardware and it has been tested using a
character level language model. The implementation is more than $21\times$
faster than the ARM Cortex-A9 CPU embedded on the Zynq 7020 FPGA. This work
can potentially evolve to a RNN co-processor for future mobile devices.

###### Index Terms:

Recurrent Neural Network (RNN); Long Short Term Memory (LSTM); acceleration;
FPGA;

##  I Introduction

As humanity progresses into the digital era, more and more data is produced
and distributed across the world. Deep Neural Networks (DNN) provides a method
for computers to learn from this mass of data. This unlocks a new set of
possibilities in computer vision, speech recognition, natural language
processing and more. However, DNNs are computationally expensive, such that
general processors consume large amounts of power to deliver desired
performance. This limits the application of DNNs in the embedded world. Thus,
a custom architecture optimized for DNNs provides superior performance per
power and brings us a step closer to self-learning mobile devices.

Recurrent Neural Networks (RNNs) are becoming an increasingly popular way to
learn sequences of data [1, 2, 3, 4], and it has been shown to be successful
in various applications, such as speech recognition [4], machine translation
[1] and scene analysis [5]. A combination of a Convolutional Neural Network
(CNN) with a RNN can lead to fascinating results such as image caption
generation [6, 7, 8].

Due to the recurrent nature of RNNs, it is sometimes hard to parallelize all
its computations on conventional hardware. General purposes CPUs do not
currently offer large parallelism, while small RNN models do not get full
benefit from GPUs. Thus, an optimized hardware architecture is necessary for
executing RNNs models on embedded systems.

Long Short Term Memory, or LSTM [9], is a specific RNN architecture that
implements a learned memory controller for avoiding vanishing or exploding
gradients [10]. The purpose of this paper is to present a LSTM hardware module
implemented on the Zynq 7020 FPGA from Xilinx [11]. Figure 1 shows an overview
of the system. As proof of concept, the hardware was tested with a character
level language model made with $2$ LSTM layers and $128$ hidden units. The
next following sections present the background for LSTM, related work,
implementation details of the hardware and driver software, the experimental
setup and the obtained results.

Figure 1: The LSTM hardware was implemented using a Zedboard Zynq ZC7020.

##  II LSTM Background

One main feature of RNNs are that they can learn from previous information.
But the question is how far should a model remember, and what to remember.
Standard RNN can retain and use recent past information [12]. But it fails to
learn long-term dependencies. Vanilla RNNs are hard to train for long
sequences due to vanishing or exploding gradients [10]. This is where LSTM
comes into play. LSTM is an RNN architecture that explicitly adds memory
controllers to decide when to remember, forget and output. This makes the
training procedure much more stable and allows the model to learn long-term
dependencies [9].

There are some variations on the LSTM architecture. One variant is the LSTM
with peephole introduced by [13]. In this variation, the cell memory
influences the input, forget and output gates. Conceptually, the model peeps
into the memory cell before deciding whether to memorize or forget. In [2],
input and forget gate are merged together into one gate. There are many other
variations such as the ones presented in [14] and [15]. All those variations
have similar performance as shown in [16].

The LSTM hardware module that was implemented focuses on the LSTM version that
does not have peepholes, which is shown in figure 2. This is the vanilla LSTM
[17], which is characterized by the following equations:

Figure 2: The vanilla LSTM architecture that was implemented in hardware.
$\otimes$ represents matrix-vector multiplication and $\odot$ is element-wise
multiplication.

$\mathbf{i}_{t}=\sigma(W_{xi}\mathbf{x}_{t}+W_{hi}\mathbf{h}_{t-1}+\mathbf{b}_{i})$
(1)
$\mathbf{f}_{t}=\sigma(W_{xf}\mathbf{x}_{t}+W_{hf}\mathbf{h}_{t-1}+\mathbf{b}_{f})$
(2)
$\mathbf{o}_{t}=\sigma(W_{xo}\mathbf{x}_{t}+W_{ho}\mathbf{h}_{t-1}+\mathbf{b}_{o})$
(3)
$\mathbf{\tilde{c}}_{t}=\mathrm{tanh}(W_{xc}\mathbf{x}_{t}+W_{hc}\mathbf{h}_{t-1}+\mathbf{b}_{c})$
(4)
$\mathbf{c}_{t}=\mathbf{f}_{t}\odot\mathbf{c}_{t-1}+\mathbf{i}_{t}\odot\mathbf{\tilde{c}}_{t}$
(5) $\mathbf{h}_{t}=\mathbf{o}_{t}\odot\mathrm{tanh}(\mathbf{c}_{t})$ (6)

where $\sigma$ is the logistic sigmoid function, $\odot$ is element wise
multiplication, $\mathbf{x}$ is the input vector of the layer, $W$ is the
model parameters, $\mathbf{c}$ is memory cell activation,
$\mathbf{\tilde{c}}_{t}$ is the candidate memory cell gate, $\mathbf{h}$ is
the layer output vector. The subscript $t-1$ means results from the previous
time step. The $\mathbf{i}$, $\mathbf{f}$ and $\mathbf{o}$ are respectively
input, forget and output gate. Conceptually, these gates decide when to
remember or forget an input sequence, and when to respond with an output. The
combination of two matrix-vector multiplications and a non-linear function,
$f(W_{x}\mathbf{x}_{t}+W_{h}\mathbf{h}_{t-1}+\mathbf{b})$, extracts
information from the _input_ and _previous output_ vectors. This operation is
referred as gate.

One needs to train the model to get the parameters that will give the desired
output. In simple terms, training is an iterating process in which training
data is fed in and the output is compared with a target. Then the model needs
to backpropagate the error derivatives to update new parameters that minimize
the error. This cycle repeats until the error is small enough [18]. Models can
become fairly complex as more layers and more different functions are added.
For the LSTM case, each module has four gates and some element-wise
operations. A deep LSTM network would have multiple LSTM modules cascaded in a
way that the output of one layer is the input of the following layer.

##  III Related Work

Co-processors for accelerating computer vision algorithms and CNNs have been
implemented on FPGAs. A system that can perform recognition on mega-pixel
images in real-time is presented in [19]. A similar architecture for general
purpose vision algorithms called _neuFlow_ is described in [20]. _neuFlow_ is
a scalable architecture composed by a grid of operation modules connected with
an optimized data streaming network. This system can achieve speedups up to
$100\times$ in end-to-end applications.

An accelerator called _nn-X_ for deep neural networks is described in [21, 22,
23, 24]. _nn-X_ is a high performance co-processor implemented on FPGA. The
design is based on computational elements called collections that are capable
of performing convolution, non-linear functions and pooling. The accelerator
efficiently pipelines the collections achieving up to $240$ G-op/s.

RNNs are different from CNNs in the context that they require a different
arrangement of computation modules. This allows different hardware
optimization strategies that should be exploited. A LSTM learning algorithm
using Simultaneous Perturbation Stochastic Approximation (SPSA) for hardware
friendly implementation was described in [25]. The paper focuses on
transformation of the learning phase of LSTM for FPGA.

Another FPGA implementation that focus on standard RNN is described by [26].
Their approach was to unfold the RNN model into a fixed number of timesteps
$B$ and compute them in parallel. The hardware architecture is composed of a
hidden layer module and duplicated output layer modules. First, the hidden
layer serially processes the input $\mathbf{x}$ for $B$ timesteps. Then, with
the results of the hidden layer, the duplicated logic computes output
$\mathbf{h}$ for $B$ timesteps in parallel.

This work presents a different approach of implementing RNN in FPGA, focusing
the LSTM architecture. It is different than [26], in the sense that it uses a
single module that consumes input $\mathbf{x}$ and previous output
$\mathbf{h}_{t-1}$ simultaneously.

##  IV Implementation

###  IV-A Hardware

The main operations to be implemented in hardware are matrix-vector
multiplications and non-linear functions (hyperbolic tangent and logistic
sigmoid). Both are modifications of the modules presented in [24]. For this
design, the number format of choice is Q8.8 fixed point. The matrix-vector
multiplication is computed by a Multiply ACcumulate (MAC) unit, which takes
two streams: vector stream and weight matrix row stream. The same vector
stream is multiplied and accumulated with each weight matrix row to produce an
output vector with same size of the weight’s height. The MAC is reset after
computing each output element to avoid accumulating previous matrix rows
computations. The bias $\mathbf{b}$ can be added in the multiply accumulate by
adding the bias vector to the last column of the weight matrix and adding an
extra vector element set to unity. This way there is no need to add extra
input ports for the bias nor add extra pre-configuration step to the MAC unit.
The results from the MAC units are added together. The adder’s output goes to
an element wise non-linear function, which is implemented with linear mapping.

The non-linear function is segmented into lines $y=ax+b$, with $x$ limited to
a particular range. The values of $a$, $b$ and $x$ range are stored in
configuration registers during the configuration stage. Each line segment is
implemented with a MAC unit and a comparator. The MAC multiplies $a$ and $x$
and accumulates with $b$. The comparison between the input value with the line
range decides whether to process the input or pass it to the next line segment
module. The non-linear functions were segmented into 13 lines, thus the non-
linear module contains 13 pipelined line segment modules. The main building
block of the implemented design is the gate module as shown in figure 3.

Figure 3: The main hardware module that implements the LSTM gates. The non-
linear module can be configured to be a tanh or logistic sigmoid.

The implemented module uses Direct Memory Access (DMA) ports to stream data in
and out. The DMA ports use valid and ready handshake. Because the DMA ports
are independent, the input streams are not synchronized even when the module
activates the ports at same the time. Therefore, a stream synchronizing module
is needed. The sync block is a buffer that caches some streaming data until
all ports are streaming. When the last port starts streaming, the sync block
starts to output synchronized streams. This ensures that vector and matrix row
elements that goes to MAC units are aligned.

The gate module in figure 3 also contains a rescale block that converts $32$
bit values to $16$ bit values. The MAC units perform $16$ bit multiplication
that results into $32$ bit values. The addition is performed using $32$ bit
values to preserve accuracy.

All that is left are some element wise operations to calculate
$\mathbf{c}_{t}$ and $\mathbf{h}_{t}$ in equations 5 and 6. To do this, extra
multipliers and adders were added into a separate module shown in figure 4.

Figure 4: The module that computes the $\mathbf{c}_{t}$ and $\mathbf{h}_{t}$
from the results of the gates. $\odot$ is element-wise multiplication.

The LSTM module uses three blocks from figure 3 and one from figure 4. The
gates are pre-configured to have a non-linear function (tanh or sigmoid). The
LSTM module is shown in figure 5.

Figure 5: The LSTM module block diagram. It is mainly composed of three gates
and one ewise stage module.

The internal blocks are controlled by a state machine to perform a sequence of
operations. The implemented design uses four $32$ bit DMA ports. Since the
operations are done in $16$ bit, each DMA port can transmit two $16$ bit
streams. The weights $W_{x}$ and $W_{h}$ are concatenated in the main memory
to exploit this feature. The streams are then routed to different modules
depending on the operation to be performed. With this setup, the LSTM
computation was separated into three sequential stages:

  1. 1.

Compute $\mathbf{i}_{t}$ and $\mathbf{\tilde{c}}_{t}$.

  2. 2.

Compute $\mathbf{f}_{t}$ and $\mathbf{o}_{t}$.

  3. 3.

Compute $\mathbf{c}_{t}$ and $\mathbf{h}_{t}$.

In the first and second stage, two gate modules ($4$ MAC units) are running in
parallel to generate two internal vectors ($\mathbf{i}_{t}$,
$\mathbf{\tilde{c}}_{t}$, $\mathbf{f}_{t}$ and $\mathbf{o}_{t}$), which are
stored into a First In First Out (FIFO) for the next stages. The ewise module
consumes the FIFO vectors to output the $\mathbf{h}_{t}$ and $\mathbf{c}_{t}$
back to main memory. After that, the module waits for new weights and new
vectors, which can be for the next layer or next time step. The hardware also
implements an extra matrix-vector multiplication to generate the final output.
This is only used when the last LSTM layer has finished its computation.

This architecture was implemented on the Zedboard [27], which contains the
Zynq-7000 SOC XC7Z020. The chip contains Dual ARM Cortex-A9 MPCore, which is
used for running the LSTM driver C code and timing comparisons. The hardware
utilization is shown in table I. The module runs at $142$ MHz and the total
on-chip power is $1.942$ W.

TABLE I: FPGA hardware resource utilization for Zynq ZC7020. Components | Utilization [ / ] | Utilization [%]
---|---|---
FF | 12960 | 12.18
LUT | 7201 | 13.54
Memory LUT | 426 | 2.45
BRAM | 16 | 11.43
DSP48 | 50 | 22.73
BUFG | 1 | 3.12

###  IV-B Driving Software

The control and testing software was implemented with C code. The software
populates the main memory with weight values and input vectors, and it
controls the hardware module with a set of configuration registers.

The weight matrix have an extra element containing the bias value in the end
of each row. The input vector contains an extra unity value so that the
matrix-vector multiplication will only add the last element of the matrix row
(bias addition). Usually the input vector $\mathbf{x}$ size can be different
from the output vector $\mathbf{h}$ size. Zero padding was used to match both
the matrix row size and vector size, which makes stream synchronization
easier.

Due to the recurrent nature of LSTM, $\mathbf{c}_{t}$ and $\mathbf{h}_{t}$
becomes the $\mathbf{c}_{t-1}$ and $\mathbf{h}_{t-1}$ for the next time step.
Therefore, the input memory location for $\mathbf{c}_{t-1}$ and
$\mathbf{h}_{t-1}$ is the same for the output $\mathbf{c}_{t}$ and
$\mathbf{h}_{t}$. Each time step $\mathbf{c}$ and $\mathbf{h}$ are
overwritten. This is done to minimize the number of memory copies done by the
CPU. To implement a multi-layer LSTM, the output of the previous layer
$\mathbf{h}_{t}$ was copied to the $\mathbf{x}_{t}$ location of the next
layer, so that $\mathbf{h}_{t}$ is preserved in between layers for error
measurements. This feature was removed for profiling time. The control
software also needs to change the weights for different layers by setting
different memory locations in the control registers.

##  V Experiments

The training script by Andrej Karpathy of the character level language model
was written in Torch7. The code can be downloaded from
Github111https://github.com/karpathy/char-rnn. Additional functions were
written to transfer the trained parameters from the Torch7 code to the control
software.

The Torch7 code implements a character level language model, which predicts
the next character given a previous character. Character by character, the
model generates a text that looks like the training data set, which can be a
book or a large internet corpora with more than $2$ MB of words. For this
experiment, the model was trained on a subset of Shakespeare’s work. The batch
size was $50$, the training sequence was $50$ and learning rate was $0.002$.
The model is expected to output Shakespeare look like text.

The Torch7 code implements a $2$ layer LSTM with hidden layer size $128$
(weight matrix height). The character input and output is a $65$ sized vector
one-hot encoded. The character that the vector represents is the index of the
only unity element. The predicted character from last layer is fed back to
input $\mathbf{x}_{t}$ of first layer for following time step.

For profiling time, the Torch7 code was ran on other embedded platforms to
compare the execution time between them. One platform is the Tegra K1
development board, which contains quad-core ARM Cortex-A15 CPU and Kepler GPU
192 Cores. The Tegra’s CPU was clocked at maximum frequency of $2320.5$ MHz.
The GPU was clocked at maximum of $852$ MHz. The GPU memory was running at
$102$ MHz.

Another platform used is the Odroid XU4, which has the Exynos5422 with four
high performance Cortex-A15 cores and four low power Cortex-A7 cores (ARM
big.LITTLE technology). The low power Cortex-A7 cores was clocked at $1400$
MHz and the high performance Cortex-A15 cores was running at $2000$ MHz.

The C code LSTM implementation was ran on Zedboard’s dual ARM Cortex-A9
processor clocked at $667$ MHz. Finally, the hardware was ran on Zedboard’s
FPGA clocked at $142$ MHz.

##  VI Results

###  VI-A Accuracy

The number of weights of some models can be very large. Even our small model
used almost $530$ KB of weights. Thus it makes sense to compress those weights
into different number formats for a throughput versus accuracy trade off. The
use of fixed point Q8.8 data format certainly introduces rounding errors. Then
one may raise the question of how much these errors can propagate to the final
output. Comparing the results from the Torch7 code with the LSTM module’s
output for same $\mathbf{x}_{t}$ sequence, the average percentage error for
the $\mathbf{c}_{t}$ was $3.9\%$ and for $\mathbf{h}_{t}$ was $2.8\%$. Those
values are average error of all time steps. The best was $1.3\%$ and the worse
was $7.1\%$. The recurrent nature of LSTM did not accumulate the errors and on
average it stabilized at a low percentage.

The text generated by sampling $1000$ characters (timestep $t=1$ to $1000$) is
shown in figure 6. On the left is text output from FPGA and on the right is
text from the CPU implementation. The result shows that the LSTM model was
able to generate personage dialog, just like in one of Shakespeare’s book.
Both implementations displayed different texts, but same behavior.

Figure 6: On the left side is the output text from the LSTM hardware. On the
right side is text from CPU implementation. The model predict each next
character based on the previous characters. The model only gets the first
character (seed) and generates the entire text character by character.

###  VI-B Memory Bandwidth

The Zedboard Zynq ZC7020 platform has $4$ Advanced eXtensible Interface (AXI)
DMA ports available. Each is ran at $142$ MHz and send packages of $32$ bits.
This allows aggregate bandwidth up to $3.8$ GB/s full-duplex transfer between
FPGA and external DDR3.

At $142$ MHz, one LSTM module is capable of computing $388.8$ M-ops/s and uses
simultaneously $4$ AXI DMA ports for streaming weight and vector values.
During the peak memory usage, the module requires $1.236$ GB/s of memory
bandwidth. The high memory bandwidth requirements poses a limit to the number
of LSTM modules that can be ran on parallel. To replicated LSTM module, it is
required higher memory bandwidth or to introduce internal memory to lower
requirements of external DDR3 memory usage.

###  VI-C Performance

Figure 7 shows the timing results. One can observe that the implemented
hardware LSTM was significantly faster than other platforms, even running at
lower clock frequency of $142$ MHz (Zynq ZC7020 CPU uses $667$ MHz). Scaling
the implemented design by replicating the number of LSTM modules running in
parallel will provide faster speed up. Using 2 LSTM cells in parallel can be
$16\times$ faster than Exynos5422 on quad-core ARM Cortex-A7.

In one LSTM layer of size $128$, there are $132.1$ K-ops. Multiplying this by
number of samples and number of layers for the experimental application
($2000$) gives the total number of operations $264.4$ M-ops. The execution
time divided by number of operations gives performance (ops/s). The power
consumption of each platform was measured. Figure 8 shows the performance per
unit power.

Figure 7: Execution time of feedforward LSTM character level language model on
different embedded platforms (the lower the better). Figure 8: Performance per
unit power of different embedded platforms (the higher the better).

The GPU performance was slower because of the following reasons. The model is
too small for getting benefit from GPU, since the software needs to do memory
copies. This is confirmed by running the same Torch7 code on a MacBook PRO
2016. The CPU of the MacBook PRO 2016 executed the Torch7 code for character
level language model in $0.304$ s, whereas the MacBook PRO 2016’s GPU executed
the same test in $0.569$ s.

##  VII Conclusion

Recurrent Neural Networks have recently gained popularity due to the success
from the use of Long Short Term Memory architecture in many applications, such
as speech recognition, machine translation, scene analysis and image caption
generation.

This work presented a hardware implementation of LSTM module. The hardware
successfully produced Shakespeare-like text using a character level model.
Furthermore, the implemented hardware showed to be significantly faster than
other mobile platforms. This work can potentially evolve to a RNN co-processor
for future devices, although further work needs to be done. The main future
work is to optimize the design to allow parallel computation of the gates.
This involves designing a parallel MAC unit configuration to perform the
matrix-vector multiplication.

#### Acknowledgments

This work is supported by Office of Naval Research (ONR) grants 14PR02106-01
P00004 and MURI N000141010278 and National Council for the Improvement of
Higher Education (CAPES) through Brazil scientific Mobility Program (BSMP). We
would like to thank Vinayak Gokhale for the discussion on implementation and
hardware architecture and also thank Alfredo Canziani, Aysegul Dundar and
Jonghoon Jin for the support. We gratefully appreciate the support of NVIDIA
Corporation with the donation of GPUs used for this research.

## References

  * [1] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in _Advances in neural information processing systems_ , 2014, pp. 3104–3112.
  * [2] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using rnn encoder-decoder for statistical machine translation,” _EMNLP_ , 2014.
  * [3] W. Zaremba, I. Sutskever, and O. Vinyals, “Recurrent neural network regularization,” _arXiv preprint arXiv:1409.2329_ , 2014.
  * [4] A. Graves, A.-r. Mohamed, and G. Hinton, “Speech recognition with deep recurrent neural networks,” in _IEEE Int. Conf. on ICASSP_ , 2013, pp. 6645–6649.
  * [5] W. Byeon, M. Liwicki, and T. M. Breuel, “Scene analysis by mid-level attribute learning using 2d lstm networks and an application to web-image tagging,” _Pattern Recognition Letters_ , vol. 63, pp. 23–29, 2015.
  * [6] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, “Show and tell: A neural image caption generator,” _CVPR_ , 2015.
  * [7] J. Mao, W. Xu, Y. Yang, J. Wang, and A. L. Yuille, “Explain images with multimodal recurrent neural networks,” _arXiv preprint arXiv:1410.1090_ , 2014.
  * [8] H. Fang, S. Gupta, F. Iandola, R. Srivastava, L. Deng, P. Dollár, J. Gao, X. He, M. Mitchell, J. Platt _et al._ , “From captions to visual concepts and back,” _arXiv preprint arXiv:1411.4952_ , 2014.
  * [9] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” _Neural computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [10] Y. Bengio, P. Simard, and P. Frasconi, “Learning long-term dependencies with gradient descent is difficult,” _IEEE Trans. on Neural Networks_ , vol. 5, no. 2, pp. 157–166, 1994.
  * [11] _Zynq-7000 All Programmable SoC Overview_ , Xilinx, Jan. 2016, v1.9. [Online]. Available: http://www.xilinx.com/support/documentation/data_sheets/ds190-Zynq-7000-Overview.pdf
  * [12] J. Schmidhuber, “Deep learning in neural networks: An overview,” _Neural Networks_ , vol. 61, pp. 85–117, 2015.
  * [13] F. Gers, J. Schmidhuber _et al._ , “Recurrent nets that time and count,” in _Neural Networks, 2000. IJCNN 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on_ , vol. 3. IEEE, 2000, pp. 189–194.
  * [14] H. Sak, A. Senior, and F. Beaufays, “Long short-term memory recurrent neural network architectures for large scale acoustic modeling,” in _Proc. of the Annual Conf. of INTERSPEECH_ , 2014.
  * [15] S. Otte, M. Liwicki, and A. Zell, “Dynamic cortex memory: Enhancing recurrent neural networks for gradient-based sequence learning,” in _Artificial Neural Networks and Machine Learning–ICANN 2014_. Springer, 2014, pp. 1–8.
  * [16] K. Greff, R. K. Srivastava, J. Koutník, B. R. Steunebrink, and J. Schmidhuber, “Lstm: A search space odyssey,” _arXiv preprint arXiv:1503.04069_ , 2015.
  * [17] A. Graves and J. Schmidhuber, “Framewise phoneme classification with bidirectional lstm and other neural network architectures,” _Neural Networks_ , vol. 18, no. 5, pp. 602–610, 2005.
  * [18] C. M. Bishop, “Pattern recognition and machine learning,” pp. 225–284, 2006.
  * [19] C. Farabet, B. Martini, P. Akselrod, S. Talay, Y. LeCun, and E. Culurciello, “Hardware accelerated convolutional neural networks for synthetic vision systems,” in _Circuits and Systems (ISCAS), Proceedings of 2010 IEEE International Symposium on_. IEEE, 2010, pp. 257–260.
  * [20] C. Farabet, B. Martini, B. Corda, P. Akselrod, E. Culurciello, and Y. LeCun, “Neuflow: A runtime reconfigurable dataflow processor for vision,” in _IEEE Comp. Conf. on CVPRW_. IEEE, 2011, pp. 109–116.
  * [21] A. Dundar, J. Jin, V. Gokhale, B. Krishnamurthy, A. Canziani, B. Martini, and E. Culurciello, “Accelerating deep neural networks on mobile processor with embedded programmable logic,” in _Neural information processing systems conference (NIPS)_ , 2013.
  * [22] J. Jin, V. Gokhale, A. Dundar, B. Krishnamurthy, B. Martini, and E. Culurciello, “An efficient implementation of deep convolutional neural networks on a mobile coprocessor,” in _Circuits and Systems (MWSCAS), 2014 IEEE 57th International Midwest Symposium on_. IEEE, 2014, pp. 133–136.
  * [23] A. Dundar, J. Jin, V. Gokhale, B. Martini, and E. Culurciello, “Memory access optimized routing scheme for deep networks on a mobile coprocessor,” in _High Performance Extreme Computing Conference (HPEC), 2014 IEEE_. IEEE, 2014, pp. 1–6.
  * [24] V. Gokhale, J. Jin, A. Dundar, B. Martini, and E. Culurciello, “A 240 g-ops/s mobile coprocessor for deep neural networks,” in _IEEE Conf. on CVPRW, 2014_. IEEE, 2014, pp. 696–701.
  * [25] R. Tavcar, J. Dedic, D. Bokal, and A. Zemva, “Transforming the lstm training algorithm for efficient fpga-based adaptive control of nonlinear dynamic systems,” _Informacije Midem-Journal of Microelectronics Electronic Components and Materials_ , vol. 43, no. 2, pp. 131–138, 2013.
  * [26] S. Li, C. Wu, H. Li, B. Li, Y. Wang, and Q. Qiu, “Fpga acceleration of recurrent neural network based language model,” in _IEEE 23rd Inter. Symp.on FCCM_ , May 2015, pp. 111–118.
  * [27] _Zedboard Product Brief_ , Avnet, Nov. 2014. [Online]. Available: http://zedboard.org/sites/default/files/product_briefs/PB-AES-Z7EV-7Z020_G-v12.pdf