# An Investigation on Inherent Robustness of Posit Data Representation

Ihsen Alouani1, Anouar BEN KHALIFA2, Farhad Merchant3, Rainer Leupers3
<EMAIL_ADDRESS><EMAIL_ADDRESS>{farhad.merchant,
<EMAIL_ADDRESS>1IEMN Lab CNRS UMR 8520, INSA Hauts-De-France
2National Engineering School of Sousse, University of Sousse, Tunisia
3Institute for Communication Technologies and Embedded Systems, RWTH Aachen
University, Germany

###### Abstract

As the dimensions and operating voltages of computer electronics shrink to
cope with consumers’ demand for higher performance and lower power
consumption, circuit sensitivity to soft errors increases dramatically.
Recently, a new data-type is proposed in the literature called _posit_ data
type. Posit arithmetic has absolute advantages such as higher numerical
accuracy, speed, and simpler hardware design than IEEE 754-2008 technical
standard-compliant arithmetic. In this paper, we propose a comparative
robustness study between 32-bit posit and 32-bit IEEE 754-2008 compliant
representations. At first, we propose a theoretical analysis for IEEE 754
compliant numbers and posit numbers for single bit flip and double bit flips.
Then, we conduct exhaustive fault injection experiments that show a
considerable inherent resilience in posit format compared to classical IEEE
754 compliant representation. To show a relevant use-case of fault-tolerant
applications, we perform experiments on a set of machine-learning
applications. In more than $95\%$ of the exhaustive fault injection
exploration, posit representation is less impacted by faults than the IEEE 754
compliant floating-point representation. Moreover, in $100\%$ of the tested
machine-learning applications, the accuracy of posit-implemented systems is
higher than the classical floating-point-based ones.

###### Index Terms:

Computer Arithmetic, Posit Arithmetic, Machine Learning, Reliability

##  I Introduction

As sub-micron technology dimensions sharply decrease to a few nanometer ranges
in commercialized integrated circuits, the sensitivity of electronic circuits
increases drastically [1]. Hence, embedded microprocessors are becoming more
vulnerable to soft errors, and designing dependable systems is a challenging
task for chip designers. In fact, these systems have to operate reliably even
in the presence of faults, to sustain the present growth rate of device count
and clock frequency with continuously growing reliability issues. Moreover,
the sensitivity of chips is also intensified by voltage scaling [2].
Undesirable and accidental faults become more frequent in new generation
computing systems where the systems are running heavy-duty numerical
computations. A single-event upset (SEU) or multi-bit upset (MBU) could bring
a catastrophic outcome in mission-critical applications such as space and
missile-navigation applications. Hence, having reliable arithmetic that is
resilient to errors is a primary requirement for mission-critical computing
systems.

_Posit_ is a new data type that is capable of storing more information-per-bit
compared to its IEEE 754 compliant counterparts [3]. For example, a 32-bit
posit number can have a similar dynamic range and better accuracy at the same
time compared to the 32-bit IEEE 754 compliant number. In general, m-bit posit
has a higher dynamic range and better numerical accuracy properties compared
to n-bit IEEE 754 compliant number where $m=n$. It is shown in the literature
that for computing systems, $n$-bit IEEE 754 compliant numbers can be replaced
by $m$-bit posit numbers where $m<n$ since the posit number system exhibits a
trade-off between accuracy and dynamic range [4] [5]. These trade-offs allow
the selection of the desired posit format that is suitable for computing
systems without compromising accuracy and performance [6] [7]. Further details
of the number system and the formats are discussed in Section II-A.
Reliability aspects of posit arithmetic are yet to be explored by the research
community.

To the best of our knowledge, this is the first comparative study on the
inherent fault tolerance of posit arithmetic vis-à-vis its IEEE counterpart.
We carry out an extensive investigation of reliability through an exhaustive
fault injection scheme. The major contributions of the paper are as follows:

  * •

We propose a theoretical analysis and an exhaustive reliability exploration of
posit arithmetic vis-à-vis its IEEE-754 compliant counterparts.

  * •

We conduct exhaustive reliability exploration as well as machine-learning (ML)
benchmarks under fault injection.

  * •

We show promising results in posit arithmetic that may encourage its
utilization in safety-critical applications, as well as approximate computing.

For the reproducibility, we make our framework open source [8]. The rest of
the paper is organized as follows: In Section II we present a background on
posit arithmetic and soft errors followed by related work in Section III.
Section IV describes the analysis and the proposed methodology for error
resilience using posit arithmetic. In section V, experimental setup and
results are discussed. We summarize our work in Section VI.

##  II Background and Related Works

###  II-A IEEE 754 Compliant and Posit Number Systems

The IEEE 754-2008 compliant floating-point format binary numbers are composed
of three parts: a sign, an exponent and a fraction part (see Fig. 1). The sign
is the most significant bit indicating whether the number is positive or
negative. In a single-precision format, the following 8 bits represent the
exponent of the binary number ranging from $-126$ to 127. The remaining 23
bits represent the fractional part. The normalized format of floating-point
numbers is:

$val=(-1)^{\textit{sign}}\times 2^{exp-bias}\times(1.fraction)$ (1)

Posit arithmetic is proposed as a drop-in replacement for IEEE 754 compliant
arithmetic in 2017 [3]. The posit number format has several absolute
advantages over IEEE 754 compliant arithmetic such as higher accuracy, higher
dynamic range, simpler hardware implementation for arithmetic operations,
lower area and energy footprints [9]. Besides, it is shown in the literature
that m-bit posit adders/multipliers can safely replace $n$-bit IEEE 754
compliant adders/multipliers where $m<n$ [4]. Hence, posit representation
confirms more information-per-bit compared to its IEEE 754 counterpart
representation. Furthermore, with posit representation, there are no redundant
representations and the overflow/underflow in the computations is nonexistent
with posit arithmetic. The subnormal numbers are handled in a normal way with
posit representation unlike IEEE 754 representation and there are only two
exception cases: zero and not-a-real (NaR). For all other cases, the value
$val$ of a posit is given by

$\displaystyle val=$
$\displaystyle(-1)^{\textit{sign}}\times\textit{useed}^{k}\times
2^{\textit{exp}}\times(1+\sum_{i=1}^{\textit{fn}-1}b_{\textit{fn}-1-i}2^{-i})$
(2)

The regime indicates a scale factor of $\textit{useed}^{k}$ where
$\textit{useed}=2^{2^{\textit{es}}}$ and _es_ is the exponent size. The
numerical value of $k$ is determined by the _run length_ of 0 or 1 bits in the
string of regime bits. The use of run-length encoding of the regime
automatically allows more fraction bits for the more common values for which
magnitudes are closer to 1, and thus provides tapered accuracy in a bit-
efficient way. Further details about the posit number format and posit
arithmetic can be found in [3]. The posit format and IEEE 754-2008 compliant
number formats are depicted in Fig. 1.

Figure 1: Description of the IEEE 754 single-precision floating-point and
_posit_ formats

In our experiments, we have used IEEE 754 compliant 32-bit (single precision)
floating-point numbers and 32-bit posit numbers with $es=2$ that are commonly
used.

###  II-B Soft Errors

The sharp technology scaling in new generation integrated circuits accentuates
the sensitivity of electronic circuits. As a matter of fact, embedded systems
are becoming remarkably sensitive to soft errors. These errors result from a
voltage transient event induced by alpha particles from packaging material or
neutron particles from cosmic rays [10]. This event is created due to the
collection of charge at a p-n junction after a track of electron-hole pairs is
generated. In past technologies, this issue was considered in a limited range
of applications in which the circuits are operating under aggressive
environmental conditions like aerospace applications. Nevertheless, shrinking
transistor size and reducing supply voltages in new hardware platforms bring
soft errors to ground level mainstream applications [11] [12].

##  III Related Work

Since soft errors became a challenging threat to reliability, numerous
published work proposes error-resilient memories. Architecture level error
resilience techniques such as single error correction double error detection
(SECDED) have been proposed and widely used for memory protection [13]. The
main drawback of SECDED is its area overhead and the supplementary latency
leading to performance loss. A fault-tolerant architecture presented in [14]
combines both parity and single redundancy to enhance memories’ reliability.
The weakness of these techniques is their area, power and delay overheads due
to the additional memory cells and supporting circuits required for error
detection and correction.

Circuit-level techniques have been proposed to overcome architecture-level
overheads. These techniques enhance error resilience at circuit level either
by slowing down the response of the circuit to transient events or by
increasing its critical charge. Methods such as [15] suggest to harden the
cell using a pass transistor that is controlled by a refreshing signal.
Hardened memory cells were proposed in [16], [17] and [18] that add redundant
transistors to the 6T-SRAM to increase the cell critical charge. A Schmitt
trigger-based technique [19] proposes a hardened 13-T memory cell. However,
this technique slows down memory due to a Schmitt trigger’s hysteresis
temporal characteristics. In the context of emerging approximate computing
applications, recent works like [20] proposed a trade-off between reliability
and computing precision. To assess the reliability level at an early stage,
fault injection can be performed in simulation. All these techniques do not
take into account the actual data representation that is stored within the
protected memories, especially numerical values.

A number of researchers have approached the reliability issue in numerical
algorithms. The vast majority of them treat an algorithm as a black-box and
track the behavior of these applications when running with injected soft
errors. In [21], a study on soft error propagation in floating-point programs
is presented. In [22], the behavior of various Krylov methods is analysed. The
authors track the variance in iteration count based on the data structure that
experiences the bit flip. Authors in [23] analyzed the impact of bit flips in
a sparse matrix-vector multiply (SpMV). Exemplifying the concept of black-box
analysis of bit flips, [24] presents BIFIT for characterizing applications
based on their vulnerability to bit flips.

While these techniques study the reliability of applications based on
floating-point formats, none of them study the inherent sensitivity level of
floating-point representations. This paper proposes a comparative study of the
inherent sensitivity to errors in IEEE-754 compliant floating-point and posit
representations.

##  IV Proposed Methodology

We cover analyses for SEU and MBU considering different aspects. Since float
and posit have different data formats as shown in Fig. 1, a single or multiple
bit flip event in a 32-bit number results in a new different number for both
formats. In our analyses, we consider bit flips in fraction, and exponent for
both formats as well as regime bits for posit numbers. For our theoretical
analyses, we use numbers $f_{1},p_{1}\in\mathbb{R}$, where $f_{1}$ is
compliant to IEEE 754-2008 and $p_{1}$ is a posit number, and $1\leq
s_{1},s_{2}\leq 32$, $s_{1}\in\mathbb{N}$. $s_{1}$,$s_{2}$ are the positions
of the bit flips in $f_{1}$ and $p_{1}$, $s_{1}\neq s_{2}$. For both the
number formats, we assume that the SEU and MBU occur at the same position.

###  IV-A Fraction bits

The total number of fractional bits are $23$ in IEEE 754-2008 and $23+m$ in
posit compliant number respectively. $m$-bits are appended in the fraction
part in a posit compliant numbers to the left of the fraction bits. Let be
$b_{22}b_{21}b_{20},...b_{0}$ and $a_{22+m},a_{21+m},...a_{0}$, two binary
numbers that represent fraction parts of an IEEE 754 compliant number and a
posit compliant number respectively. A representative diagram to understand
the bit flip phenomena in the fraction part is shown in Fig. 2.

Figure 2: Fraction bits in IEEE 754-2008 compliant number and posit compliant
number

The largest error that can occur in IEEE 754-2008 compliant number due to a
bit flip in the fraction part is the bit flip of $b_{22}$ ($s_{1}=23$). A flip
from $1$ to $0$ or vice-versa would result in subtraction or addition of $0.5$
in the decimal value of the fraction part. On the other hand, in posit to have
the similar impact, $s_{1}$ has to be at $23+m$ bit position. In general, a
bit flip in the fraction part of IEEE 754-2008 compliant number is $s_{1}$
then similar impact in the fraction part of a posit compliant number can be
observed if there is a bit flip in the position $s_{1}+m$. The value of $m$
depends on the configuration of the posit number. For example, a $32$-bit
posit number can have k = $-31$ to $30$ regime bits since regime bits are
calculated based on run-length of ’0’ or ’1’ from the most significant bits
after the sign bit (refer equation 2. In practical scenarios the run-length of
’0’ or ’1’ is not expected to be very large since a large $k$ results in a
very high dynamic range for the numbers. $k=5$ and $es=2$ configuration
results in the dynamic range that is similar to the IEEE 754 compliant number.
In general, $m=exp\\_size-k-es$ where $exp\\_size$ is the exponent size in
IEEE 754 compliant number and $es$ is the posit exponent size. Since, in the
most realistic scenarios $exp\\_size>(k+es)$, the bit flip in the position
$s_{1}$ in IEEE 754 compliant number and posit number would result in smaller
error in the posit number.

In case of double bit-flip, the second bit flip position being $s_{2}$,
assuming that the second bit flip occurs in the same locations in an IEEE 754
compliant number and a posit number, the error due to the second bit flip is
higher in the IEEE 754 compliant number. The higher error is due the the
higher weight associated with the bit position in IEEE 754 compliant number
compared to the posit number.

###  IV-B Exponent and regime bits

A single bit flip in the exponent of IEEE 754 compliant numbers and posit
numbers injects a higher error impact in the IEEE 754 compliant number due to
the phenomena explained in Section IV-A is applicable to exponent bits as
well. Due to more weight associated with the position in the exponent part of
the IEEE 754 compliant number compared to the exponent part of the posit
number, the error incurred is higher in the IEEE 754 compliant number. A bit
flip in the regime part of posit incurs higher error compared to the bit flip
in the bits $3$ to $8$ of the float section due to higher weight associated
with the posit number. Similarly, second bit flip in the exponent incurs lower
error in posit compared to an IEEE 754 compliant number while the second bit
flip in regime section of posit number results in higher error. In the
subsequent section, we present toolflow to validate our claims.

###  IV-C Toolflow

To assess the impact of errors on both posit and IEEE 754-2008 compliant
representations, we proceed to an exhaustive fault injection exploration
process. We modified the posit public implementation [25] to support our fault
injection mechanism. Besides, we built an exhaustive exploration platform
shown in Fig. 3. The idea is to focus on the actual arithmetic representation
of the data instead of a coarse grain probabilistic study or a very fine grain
circuit simulation.

Figure 3: Toolflow of fault injection process in both posit (32,2)
configuration and IEEE-754 compliant single-precision floating-point formats
(open-source available in [8]

Figure 4: (a) Mean Relative Error distance comparison between posit and IEEE
754 compliant float under single event upset injection (b) Error distance
comparison between posit and IEEE-754 compliant float under double event upset
injection

Fig. 3 explains the followed methodology to assess the inherent reliability of
the two tested representations. Since we are considering reliability from a
hardware perspective, we are sticking to the actual bit-level data
representation. In this paper, we focus on IEEE 754 single precision floating
point and $(N,es)=(32,2)$ (where $N$ is width of the representation and $es$
is exponent size) posit representations for our experiments. Hence, from a raw
32-bit word, we generate the corresponding floating-point and posit numbers.
For a fair comparison, the errors are injected exactly in the same respective
bit of the two tested representations. For double bit upsets as well, we
choose the same locations for bitflips in both representations. The inherent
reliability of the two representations is assessed by quantifying the mean
relative error distance (MRED) of a corrupted value from a golden (non-
corrupted) value as shown in Equation 3.

$MRED=\frac{1}{32}*\sum_{i=0}^{31}\frac{|V_{i}-V_{i}^{*}|}{V_{i}}$ (3)

Where $V_{i}$ and $V_{i}^{*}$ are the golden and the corrupted value
respectively when a fault is injected in a bit $i$. MRED gives an insight on
the mean impact of bit flips that are injected in all words’ 32 bits
exhaustively.

##  V Experimental Setup and Results

Figure 5: Human action recognition rates using statistical features

Figure 6: Human action recognition rates using wavelet features.

The experiments are divided into two categories:

  * •

The first is a comparative application-agnostic exploration of the fault
injection impact on reliability of posit and IEE-754 compliant
representations.

  * •

The second is a comparative reliability study on a set of ML systems on two
different applications tested under fault injection.

This section details the experimental setup and discusses the results.

###  V-A Exhaustive comparative reliability exploration

This set of experiments follow the methodology presented in Section IV. The
toolflow is implemented in C using the SoftPosit platform [25] for posit and a
developed bit-wise fault injection platform for IEEE 754 compliant numbers.
The experiments are run on a 3 GHz Intel Core i7 processor running the OS X
10.9.5 operating system.

####  V-A1 Single Event Upset

The above-explained experimental setup aims at exploring the impact of bit
flips on a given numerical data representation in an exhaustive manner. The
results shown in Fig. 4 expose in a logarithmic scale the comparison between
posit and IEEE 754 compliant floating-point representations’ inherent
resiliency to bit flips. The comparison is performed based on the MRED between
the golden value (without fault injection) and the corrupted one in both posit
and IEEE 754 floats. The results shown in Fig. 4 represent a geometric
superposition where the IEEE 754 compliant floating-point graph is in most of
the cases above the posit graph. This indicates that the posit representation
is globally more error resilient than IEEE 754 compliant representation. In
fact, in more than 95% of the explored cases, a bit flip in an IEEE 754
compliant number deviates from the golden data more than the posit number.
Moreover, we registered only 31 cases of not a real (NaR) with posit, which
represents 0.7E-6% of the fault injections. On the other hand, for IEEE
compliant floating-point, more than 4% of the fault injections resulted in not
a number (NaN). These cases correspond to non-representable data in the IEEE
754 compliant floating-point graph of Fig. 4.

Figure 7: Biometric ECG authentication rates using statistical features.

Figure 8: Biometric ECG authentication rates using wavelet features.

####  V-A2 Double Event Upset

Starting from 40nm technology, more than 35% of bit upsets are MBUs.
Therefore, it is important to consider this phenomenon in reliability
assessment processes. In this section, we track the impact of double bit
upsets on the data representation for both posit and IEEE 754 compliant
floating-point representations.

Following the same fault injection exploration mechanism as shown in Fig. 3,
we evaluate the impact of two-bit flips on the MRED between posit and IEEE-
compliant floating point. We inject two bit flips at every iteration: the
first is injected exhaustively bit-wise, and the second location is randomly
selected following a normal distribution among the remaining bits. Fig. 4
shows the MRED caused by two-bit upsets in both representations. Injecting 2
bit flips results globally in higher error magnitudes. However, the results
still confirm the higher inherent error resilience of posit shown with the
single bit upset experiments.

The error resilience in posit is due to two main reasons: the variable size of
the scale factor (regime bits) and a larger number of bits in the fractional
part for the vast majority of cases. The larger number bits in the fractional
part is due to variable-sized regime bits. An SEU or MBU in the fractional
part results in a lower error compared to an SEU or MBU in the regime bits or
exponent bits in posit. Since the IEEE 754-compliant representation has more
exponent bits compared to the regime bits and the exponent bits in the posits,
the resulting error is higher in IEEE 754 compliant floating-point numbers
compared to posits. Better error resilience of posit data-type makes it the
right choice for mission-critical next-generation systems. Moreover, the
absence of redundant representations such as NaNs is a supplementary factor
that enhances posit robustness to errors.

###  V-B Machine-learning applications

Recent attacks on ML applications are based on deliberate fault-injection
techniques [26]. In this subsection, we show the results of fault injection
experiments applied in a set of ML applications. We evaluate two computer-
vision systems. The first is a biometric authentication system using
Electrocardiogram (ECG) signals based on LATIS ECG Database [27]. The second
is a human action recognition (HAR) system using kinematic accelerometer
signals trained with Berkley MHAD dataset [28]. For the features extraction
phase, two types of features were chosen:

  * •

Temporal features such as the mean, the standard deviation, the quadratic
mean, and the covariance.

  * •

Time-frequency type characteristics resulting from the Wavelet transformation.

We used the sliding window method to extract the characteristics of each
window. These characteristics are subsequently concatenated in a descriptor
vector.

For the classification phase, we evaluate a set of most widely used
classifiers in the literature. These ML techniques are: the Support Vector
Machines (SVM) with linear, Gaussian and Cubic kernels, Decision Trees,
Discriminant analysis classifiers, Naive Bayes classifiers, KNN classifiers (K
= 1), AdaBoost classifier, Random forest and Neural network classifiers.
Figures 5, 6, 7 and 8 show the recognition rate of the different techniques
and settings with and without fault injection. These figures show the impact
of single fault injection in the input of the different classifiers for both
IEEE floating point and posit data representation. In all these cases with
varying features and classifiers, the fault injection impact is significantly
lower on the posit implementation which confirms the findings in Section V-A.
In fact, while the overall accuracy drop in posit under fault injection varies
from $0\%$ and $30\%$, the accuracy drop in IEEE-compliant floating point
varies between $10\%$ and $67\%$.

##  VI Conclusion

This paper investigates the reliability of two prominent data representations,
namely IEEE 754 compliant single precision and (32,2) posit representation.
Firstly, we presented a brief theoretical analysis of both the number formats
for a single bit flip and double bit flip. An exhaustive fault injection
platform is implemented, and the exploration led to a promising conclusion for
posit arithmetic that corroborated to our theoretical analysis. To further
illustrate this finding, we conduct a benchmark of several ML techniques under
fault injection. The experiments demonstrate higher inherent robustness of
posit compared to the classical IEEE 754 representation. These findings are
useful for safety-critical systems design. They can also be exploited for
limiting imprecision in approximate computing designs. Future work will tackle
the implementation of a full posit-based processor architecture.

## References

  * [1] I. C. et al., “Impact of technology scaling on sram soft error rates,” _IEEE Trans. Nucl. Sci._ , vol. 61, no. 6, pp. 3512–3518, Dec. 2014.
  * [2] B. P. Sanches _et al._ , “J-swfit: A java software fault injection tool,” in _2011 5th Latin-American Symposium on Dependable Computing_ , April 2011, pp. 106–115.
  * [3] Gustafson _et al._ , “Beating floating point at its own game: Posit arithmetic,” _Supercomput. Front. Innov.: Int. J._ , vol. 4, no. 2, p. 71–86, Jun. 2017. [Online]. Available: https://doi.org/10.14529/jsfi170206
  * [4] R. Chaurasiya _et al._ , “Parameterized posit arithmetic hardware generator,” in _ICCD 2018_ , Oct 2018, pp. 334–341.
  * [5] S. Nambi _et al._ , “Expan(n)d: Exploring posits for efficient artificial neural network design in fpga-based systems,” arXiv 2020.
  * [6] R. Jain _et al._ , “CLARINET: A RISC-V Based Framework for Posit Arithmetic Empiricism,” arXiv.org 2020.
  * [7] V. Saxena _et al._ , “Brightening the optical flow through posit arithmetic,” in _International Symposium on Quality Electronic Design (ISQED)_ , Apr. 2021.
  * [8] Github repository for posit and IEEE 754 compliant fault injection platform. [Online]. Available: https://github.com/ihstein/posit_FP_reliability
  * [9] A. Guntoro _et al._ , “Next generation arithmetic for edge computing,” in _2020 DATE_ , 2020, pp. 1357–1365.
  * [10] J. Ziegler _et al._ , “Ibm experiments in soft fails in computer electronics,” _IBM Journal of Research and Development_ , vol. 40, no. 1, 1996.
  * [11] H. Quinn _et al._ , “Terrestrial-based radiation upsets: a cautionary tale,” in _FCCM 2005_ , April 2005, pp. 193–202.
  * [12] G. Just _et al._ , “Soft errors induced by natural radiation at ground level in floating gate flash memories,” in _IRPS 2013_ , April 2013, pp. 3D.4.1–3D.4.8.
  * [13] P. Reviriego _et al._ , “Error detection in majority logic decoding of euclidean geometry low density parity check (eg-ldpc) codes,” _IEEE Trans. VLSI Syst._ , vol. 21, no. 1, Jan 2013.
  * [14] I. Alouani _et al._ , “Parity-based mono-copy cache for low power consumption and high reliability,” _RSP 2012_ , Oct 2012.
  * [15] B. S. Gill _et al._ , “A new asymmetric sram cell to reduce soft errors and leakage power in fpga,” in _DATE 2007_ , ser. DATE’07, Apr 2007.
  * [16] X. Liu _et al._ , “A novel soft error immunity sram cell,” in _IRW 2013_ , ser. IRW ’13, Oct 2013.
  * [17] J. Guo _et al._ , “Novel low-power and highly reliable radiation hardened memory cell for 65 nm cmos technology,” _Circuits and Systems I: Regular Papers, IEEE Transactions on_ , vol. 61, no. 7, pp. 1994–2001, July 2014\.
  * [18] I. Alouani _et al._ , “As8-static random access memory (sram): asymmetric sram architecture for soft error hardening enhancement,” _IET Circuits, Devices Systems_ , vol. 11, no. 1, pp. 89–94, 2017.
  * [19] S. Lin _et al._ , “Analysis and design of nanoscale cmos storage elements for single-event hardening with multiple-node upset,” _Device and Materials Reliability, IEEE Transactions on_ , vol. 12, no. 1, pp. 68–77, March 2012.
  * [20] D. Shin _et al._ , “Approximate logic synthesis for error tolerant applications,” in _DATE 2010_ , March 2010, pp. 957–960.
  * [21] S. Li _et al._ , “Soft error propagation in floating-point programs,” in _International Performance Computing and Communications Conference_ , Dec 2010, pp. 239–246.
  * [22] V. Howle _et al._ , “The effects of soft errors on krylov methods,” _SIAM Parallel Processing_ , Feb. 2017.
  * [23] M. Shantharam _et al._ , “Characterizing the impact of soft errors on iterative methods in scientific computing,” in _ICS 2011_ , ser. ICS ’11. New York, NY, USA: ACM, 2011, pp. 152–161. [Online]. Available: http://doi.acm.org/10.1145/1995896.1995922
  * [24] D. Li _et al._ , “Classifying soft error vulnerabilities in extreme-scale scientific applications using a binary instrumentation tool,” in _SC ’12: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis_ , Nov 2012, pp. 1–11.
  * [25] C. Leong. (2018, Nov.) Softposit version 0.4.1rc. [Online]. Available: https://gitlab.com/cerlane/SoftPosit
  * [26] V. Venceslai _et al._ , “Neuroattack: Undermining spiking neural networks security through externally triggered bit-flips,” 2020.
  * [27] T. Hamdi _et al._ , “A novel feature extraction method in ecg biometrics,” in _International Image Processing, Applications and Systems Conference_ , Nov 2014, pp. 1–5.
  * [28] F. Ofli _et al._ , “Berkeley mhad: A comprehensive multimodal human action database,” in _WACV 2013_ , Jan 2013, pp. 53–60.


# FMM-X3D: FPGA-based modeling and mapping of X3D for Human Action Recognition

Petros Toupas  Dept. of Electrical and Electronic Engineering, Imperial
College London, UK  Information Technologies Institute, Centre for Research
and Technology Hellas, Greece  Christos-Savvas Bouganis  Dept. of Electrical
and Electronic Engineering, Imperial College London, UK  Dimitrios Tzovaras
Information Technologies Institute, Centre for Research and Technology Hellas,
Greece

###### Abstract

3D Convolutional Neural Networks are gaining increasing attention from
researchers and practitioners and have found applications in many domains,
such as surveillance systems, autonomous vehicles, human monitoring systems,
and video retrieval. However, their widespread adoption is hindered by their
high computational and memory requirements, especially when resource-
constrained systems are targeted. This paper addresses the problem of mapping
X3D, a state-of-the-art model in Human Action Recognition that achieves
accuracy of 95.5% in the UCF101 benchmark, onto any FPGA device. The proposed
toolflow generates an optimised stream-based hardware system, taking into
account the available resources and off-chip memory characteristics of the
FPGA device. The generated designs push further the current performance-
accuracy pareto front, and enable for the first time the targeting of such
complex model architectures for the Human Action Recognition task.

##  I Introduction

In recent years, two-dimensional CNNs have excelled at image-related tasks.
The growing focus and number of applications arising from video-related tasks,
such as video surveillance, autonomous driving, and elderly/patient
monitoring, have necessitated the development of algorithms that include and
account for the temporal domain. Three-dimensional CNNs are one of the most
frequently applied techniques for dealing with video and volumetric data. With
the addition of an extra dimension, such as time or depth, over 2D CNNs, 3D
CNNs improve their learning capacity by extracting features relevant to the
newly added dimension of the input as well.

Especially in the task of human action recognition, 3D CNNs have demonstrated
exceptional performance. The use of 3D CNNs enables the interpretation of
human motion across the frames of a video, enabling the detection of a variety
of human actions without the need for dedicated time-domain techniques (e.g.,
LSTMs). As can be seen in Figure 1, 3D CNNs dominate the pareto front in one
of the most widely used HAR benchmarks, Kinetics-400 [8], while the recent
emergence of vision transformers has also begun to drive some designs to the
pareto front; however, such networks require orders of magnitude more GFLOPs
to operate.

While 3D CNNs are capable of capturing time- or depth-related features, the
additional dimension in the input frequently results in greater workloads and
computational and memory requirements as compared to 2D CNNs. Numerous
hardware devices, including GPUs, FPGAs, and ASICs, have been used to mitigate
the 3D CNNs’ high processing requirements and provide high performing systems.
FPGAs are an attractive candidate as an acceleration platform since they offer
greater flexibility than ASICs and are more energy efficient than GPUs.
Furthermore, the rapid evolution of 3D CNN model designs necessitates the need
for a platform that can quickly adapt to any new model requirements.

Figure 1: Kinetics-400 pareto over the years

Previously, FPGA-based acceleration of 3D CNN models focused on C3D and
R(2+1)D, whose performance falls short of the pareto front in recent
benchmarks such as the Kinetics. In contrast, this work proposes an FPGA
architecture for X3D[5], a 3D CNN model that lies in the pareto front of
accuracy vs. number of parameters, as shown in Figure 1. Using an automated
optimization approach and by applying transformations to the SDF graph, the
design space of X3D is explored based on a streaming architecture and the
Synchronous Data Flow (SDF)[9] computation model, taking into account the
target FPGA platform’s requirements.

Figure 2: Model performance over different FPGA devices and resource
constraints

The target FPGA platform or resource constraints may change depending on the
application or system requirements. As can be seen in Figure 2, the proposed
hardware design may adapt to different devices as well as varied restrictions
regarding the FPGA utilisation percentage, resulting in pareto front solutions
in each circumstance. The key contributions of this paper are the following:

  * •

The extension of the SDF graph model that is employed for capturing the
performance requirements in CNN mapping to streaming architectures to
explicitly address irregular blocks with branching, which are widely used in
modern CNN models.

  * •

The automation and optimisation of the mapping of the X3D model, a state of
the art model for Human Action Recognition, to different FPGA devices, taking
into account the available FPGA resources and memory bandwidth
characteristics.

  * •

Two streaming-centric optimizations to improve the X3D network’s dataflow
characteristics. The use of prior batch statistics for GAP layers, and the
quantization of the model weights and feature maps to 16-bit fixed-point
arithmetic.

##  II Background

###  II-A fpgaConvNet

This work is inspired by the fpgaConvNet [17], a framework that automatically
maps 2D CNN models to FPGA platforms. The framework accepts a high-level
description of a 2D CNN model and the FPGA platform’s characteristics. The
structure and parameters of the network are captured in a Directed Acyclic
Graph (DAG), which is translated into an SDF graph. Synchronous Data-Flow
(SDF) [9] is a computation model for both parallel and sequential systems
developed by Lee et. al. in 1987. The dataflow model represents the system’s
behaviour as a graph, named SDFG. The SDFG nodes represent the operations,
while the arcs represent the data streams that connect them. SDF’s key idea is
that every node fires whenever data is available at its input arcs, resulting
in a data-driven execution paradigm. The SDFG is represented as a topology
matrix $\Gamma$ and is subjected to a series of transformations that vary the
folding factors and reconfiguration points, therefore broadening the design
space (a more detailed explanation of how $\Gamma$ matrix is constructed is
given in Section III-B). After that, an optimization technique is employed to
determine the optimal folding factors and reconfiguration points for
optimising the throughput, that result in the final design choice. The tool
concludes by providing a synthesizable Vivado HLS hardware design.

###  II-B Related Work

Although 3D CNNs have been around for a while, there have only been a few
papers aimed at their acceleration on FPGAs. The majority of these works focus
on relatively old 3D CNNs, such as the C3D [7] model, whose performance falls
short of state-of-the-art models, due to simple underlying CNN model. Fan et
al. introduced a series of works on 3D CNN acceleration for HAR on FPGA
systems, [4, 3, 2]. In their initial work [4], they proposed the F-C3D
hardware architecture for the acceleration of C3D [7], which is capable of
supporting multiple 3D convolutional layers and design strategies for
overcoming the challenges associated with 3D CNNs, such as increased
computation and memory requirements, while also allowing their design to be
ported to other FPGA devices. In their subsequent work [3], they proposed an
analytical model and a tool for optimising the hardware architecture based on
the device specification and accuracy requirements, as well as the use of
block floating point (BFP) arithmetic precision to minimise accuracy loss and
the need for retraining the model. Their design evaluation was conducted on
C3D model as well. In their most recent publication in this series, [2], they
proposed E3DNet, an efficient 3D CNN based on their proposed 3D-1 bottleneck
building block. Their hardware implementation of E3DNet, called F-E3D, is
capable of real-time performance at the execution time of 35.3 ms per clip111A
clip is defined as a stacked sequence of frames that are meant to be the input
of the 3D CNN, while achieving 85.1%222H. Duan et. al. [1] currently holds the
SoA results on UCF101 achieving 98.6% accuracy in terms of accuracy on the
UCF101 benchmark.

Liu et al. [10] proposed a unified hardware architecture for 2D and 3D CNN
acceleration based on their observation that the computing patterns of 2D and
3D CNNs are similar. Their aim was to convert the CNN convolutions to matrix
multiplication operations, paying close attention to memory optimizations in
order to overcome the difficulties of feature map replications. Additionally,
they employed an analytical model to configure the accelerators for optimal
resource use. They have targeted and evaluated their design on C3D model. Shen
et al. [12] followed a similar approach, developing a unified template-based
architecture based on the Winograd [18] algorithm capable of handling both 2D
and 3D CNNs. Additionally, they developed an analytical technique for
efficiently exploring the design space for mapping 2D and 3D CNNs on FPGA
accelerators. The authors have targeted the C3D model for the evaluation of
their proposed design. Sun et al. [14] used a blockwise pruning approach to
apply weight pruning to two distinct 3D CNN architectures, namely C3D and
R(2+1)[16]. Their hardware design, which is based on the Alternating Direction
Method of Multipliers (ADMM), together with the suggested pruning approach,
enables the acceleration of 3D CNNs with low accuracy loss when compared to
their unpruned form. Recently, Toupas et al. [15] introduced a toolflow that
automates the mapping and optimization of 3D CNN models on FPGA devices
achieving promising results on latency-oriented applications.

Until recently, the majority of research has been focused on the C3D [7] model
for HAR, which was introduced in 2013. The model’s architecture is rather
simplistic, consisting of only six consecutive layers, and it performs poorly
in terms of accuracy when compared to the modern SoA models in HAR ($85.2$% in
UCF101 compared to $98.6$% which is the current SoA). In terms of design
complexity, it is comparable to the LeNet or AlexNet in the three-dimensional
space. Because the methodologies described above are primarily tailored to the
design of the target model, they cannot be extended or evaluated in the more
complex architectures of today’s state-of-the-art HAR models. As a result, it
is unclear how well they would perform and how they could map the most recent
SoA models. This study focuses on X3D [5], which has over 200 layers and
deviates from the sequential model of early networks by introducing branching.

###  II-C X3D Model Family

C. Feichtenhofer proposed the X3D family[5] of efficient 3D CNN models in
2020. The model designs in the family range from X-Small (X3D-XS) to XX-Large
(X3D-XXL), depending on the computational complexity, number of layers, input
volume size, and number of parameters. Indicative, the computational
complexity of the X3D-XS is 0.6 GFLOPs, whereas the computational complexity
of the X3D-XXL is 143.5 GFLOPs. The number of model parameters ranges from
3.76 M to 20.3 M. Until 2021, the X3D series had the best performance in many
HAR benchmarks, including the Kinetics-400 (80.4%), Kinetics-600 (81.9%), and
Charades (47.1%). This paper focuses on the X3D-M, which has a computational
complexity of 6.2 GFLOPs, 3.76 M parameters, and receives as input a sequence
of 16 RGB frames with spatial dimensions of $256\times 256$, attaining an
accuracy of 76.0% in Kinetics-400 and 78.8% in Kinetics-600. X3D-M has a total
of 244 distinct layers, which include depth-wise and point-wise 3D
convolutional layers, various activation functions such as relu, sigmoid, and
swish, 3D global average pooling layers, and fully connected layers.

##  III Hardware-Level Interpretation of X3D

This section discusses the hardware-level interpretation and modelling of the
X3D model. The proposed framework takes as input the high-level description
file of the X3D model, and captures the parameters of each layer in a DAG
along with the connections between the layers. Each of the network’s supported
layer is mapped to a hardware building block equivalent that implements its
functionality. Subsequently the framework generates the SDFG for the network,
which is created by mapping the DAG nodes to their hardware equivalents and
adding them as nodes in the SDFG, as well as by adding the connections between
the DAG nodes as arcs in the SDFG. Finally, the estimated performance of a
particular configuration of the SDFG nodes of the network is calculated using
the SDF computation model. The following sections provide detailed
descriptions for each stage of the proposed framework’s workflow.

###  III-A X3D layers as DAG nodes

Three main parts comprise the description of a neural network model supplied
by high-level frameworks such as pytorch and onnx. First, the layers and their
connections that define the model’s structure and flow. Second, each layer’s
special attributes and configuration, and finally the actual values of the
learnable parameters associated with their layers (if any). A dedicated model
parser is provided, that parses the above descriptions to build a DAG
containing all of the relevant information about the target neural network.
The DAG structure is faithful to the original, retaining just the essential
information from the layers’ specific attributes and configuration.
Additionally, the parser stores the model parameters/weights for future use
during inference.

All the layers of the X3D model comprise of the $\mathbf{Sp_{i}}$, and
$\mathbf{Sp_{o}}$ parameters describing the input and output shapes. In
addition to these, each layer has layer-specific parameters, as shown in the
configurations below:

  * •

3D Convolutional Layer There are four different types of convolutional layers
that X3D model incorporates: 3D convolutions with kernel omitting  (a) the
temporal dimension $1\times K_{h}\times K_{w}$,  (b) the spatial dimensions
$K_{d}\times 1\times 1$,  (c) 3D depth-wise convolutions,  (d) 3D point-wise
convolutions.  Even though the are different types of convolutions, the
discrimination between them can be achieved by looking at the layer’s
configuration that is stored in the DAG nodes,
$\mathbf{<Sp_{i},Sp_{o},Ks,Sr,Pad,Gp>}$:

    * $\circ$

Ks is a 3-value vector $[K_{d},K_{h},K_{w}]$ specifying the depth, height and
width of the 3D convolution window (also known as kernel).

    * $\circ$

Sr is a 3-value vector specifying the strides of the convolution along each
dimension.

    * $\circ$

Pad is a 3-value vector denoting the amount of padding applied to each
dimension of the input

    * $\circ$

Gp a value that specifies the number of groups in which the input is split
along the channel axis

  * •

3D Activation Layers The activation functions used in X3D are the following:
(a) ReLu activation,  (b) Sigmoid activation,  (c) Swish activation which is
expressed as $y=x*sigmoid(x)$.  . The layer’s configuration is as follows,
$\mathbf{<Sp_{i},Sp_{o},T>}$:

    * $\circ$

T denotes the type of activation function.

  * •

3D Element-wise Layers In X3D, element-wise operations refer to the layers
that combine data from several branches via addition or multiplication. In
comparison to the other layer types, these layers combine several inputs into
a single output. The shapes of the layer’s inputs may or may not be identical,
resulting in different functionality of the layer (normal vs broadcasting).
The layer’s configuration is as follows,
$\mathbf{<Sp_{i1},...,Sp_{iN},Sp_{o},T,M>}$:

    * $\circ$

T denotes the type of element-wise operation, e.g. addition, multiplication,
division, and so on.

    * $\circ$

M denotes the mode of element-wise operation, normal or broadcasting.

  * •

3D Global Average Pooling Layer While the standard pooling operation samples
patches of the input feature map to decrease its size, this extreme case
samples the whole feature map into a single value, creating an output vector
with the same shape as the channels, with the configuration being like,
$\mathbf{<Sp_{i},Sp_{o},T>}$.

###  III-B SDFG representation with branch support

To take advantage of the SDF model’s capabilities, DAG nodes are mapped into
their associated hardware building blocks, which implement the functionality
of the respective network layer in the underlying hardware. Using SDF theory,
the SDFG may be represented as a topology matrix $\Gamma$. The nodes are
represented by the columns of this matrix, while the arcs that link the nodes
are represented by the rows. The data consumption/production rates for each
node in each arc can be inferred by looking the element at $(node,arc)$
position in the $\Gamma$ matrix. Positive values, by convention, drive data
production, whereas negative ones drive data consumption. The element
$\Gamma(n,a)=-1$, for example, indicates that node n consumes data at arc a at
a rate of one.

The $\Gamma$ matrix is decomposed into several matrices, allowing a more in-
depth examination of each one separately and more fine control overall. The
initial decomposition of the $\Gamma$ matrix yielded three distinct matrices,
which are as follows:

  1. i)

The stream matrix S. This matrix element stores the number of incoming and
outgoing parallel streams that arrive to each node’s input and output.

  2. ii)

The rate matrix R. The rate matrix elements include the normalised data
production and consumption rates of each node at each arc (defined as the
number of elements produced/consumed per cycle). The values in this matrix
range from 0 to 1.

  3. iii)

The data matrix C. The width of each individual stream from the S matrix is
stored in this matrix elements. Since all of the streams are assumed to have
the same bit width of 16, the above matrix will not be used in this study.

The $\Gamma$ matrix decomposition can be derived from the following Equation:
$\mathbf{\Gamma=S\times R}$.

code-for-first-row=

$\Gamma=\pNiceMatrix[first-
row]\rotatebox{45.0}{MemIn}&\rotatebox{45.0}{Relu}\rotatebox{45.0}{Conv1}\rotatebox{45.0}{Swish}\rotatebox{45.0}{Conv2}\rotatebox{45.0}{Add}\rotatebox{45.0}{MemOut}\\\
\frac{BW}{2}-R_{in_{RL}}00000\\\ 0R_{out_{RL}}-R_{in_{C1}}00-R_{in1_{AD}}0\\\
00R_{out_{C1}}-R_{in_{SW}}000\\\ 000R_{out_{SW}}-R_{in_{C2}}00\\\
0000R_{out_{C2}}-R_{in2_{AD}}0\\\ 00000R_{out_{AD}}\frac{BW}{2}$

The upper bi-diagonal structure of the $\Gamma$ matrix precludes modelling of
branching behaviours, i.e. of graphs with nodes having multiple outgoing arcs
as well as nodes accepting multiple incoming arcs. The suggested SDFG
modification facilitates the creation of graphs with numerous incoming or
outgoing arcs at nodes. The matrix below illustrates an example of the
proposed SDFG. $BW$ is the available memory bandwidth to the device off-chip
memory, which is given in the device specs that are provided as input to the
system along with the high level network description file. The rest of the
matrix elements indicates values that have emerged from the result of the
matrix multiplication in the $\Gamma$ matrix.

The depth of each side of a branch is computed to incorporate some extra
buffering for the streams that are combined at the merge points in order to
ensure the right flow of data across the design’s streams as well as to
equalise the rates at the merge points.

###  III-C X3D layers as hardware building blocks

The hardware building blocks are the major components utilised to construct
the SDFG, which will be used subsequently to estimate the network’s
performance. The configuration of these blocks, in conjunction with the
network’s topology, is utilised to automatically generate and construct the
design’s synthesisable Vitis HLS code. The representation of the supported
hardware building blocks comprises of the following:

  1. i)

DAG parameters. A set of parameters that originated from the layer’s settings
as a DAG node. These settings are the layers’ structural configuration that
cannot be changed.

  2. ii)

SDFG parameters. An additional set of parameters that go along with the
layer’s hardware building block. These parameters have an impact on the
layer’s performance and are the ones that the optimization algorithm searches
for during the design space exploration phase.

All of the layers share a common set of parameters describing their hardware
building blocks. These parameters are the number of streams accepted at the
layer’s input and provided at the layer’s output, namely $\mathbf{s_{in}}$ and
$\mathbf{s_{out}}$, the consumption rate of the layer, defined as number of
elements consumed per cycle $\mathbf{r_{in}}$, and the production rate of the
layer, defined as number of elements produced per cycle $\mathbf{r_{out}}$.
The detailed description of the parametrisation of all the layers is given
below:

  * •

3D Convolutional Layer,
$\mathbf{<{DAG_{params}},{s_{in},s_{out},r_{in},r_{out},p_{mac}}>}$

    * $\circ$

$p_{mac}$ is the number of multiply and accumulate (MAC) operations that take
place inside the convolution in parallel.

The $s_{in}$,$s_{out}$, and $p_{mac}$ are altered during the DSE and affect
the final performance of the layer. Meanwhile the $r_{in}$ and $r_{out}$
depend on the $p_{mac}$ which means they are implicitly altered during the DSE
as well. A mode detailed analysis of the convolution layer and its sub-modules
is provided in fpgaConvNet [17].

  * •

3D Activation Layers, 3D Global Average Pooiling Layer,
$\mathbf{<{DAG_{params}},{s_{in},s_{out},r_{in},r_{out}}>}$
These layers’ $r_{in}$ and $r_{out}$ can achieve consumption/production rates
of 1 without constraints from previous layers or the memory, due to their
element-wise functionality and the simplicity of their operations. The only
exception here is the 3D Global Average Pooiling, in which
$r_{out}=\frac{1}{D\times H\times W}$, where $D$ is the depth, $H$ is the
height, and $W$ is the width dimension of the input feature map.

  * •

3D Element-wise Layers,
$\mathbf{<{DAG_{params}},{s_{in1},s_{in2},s_{out},r_{in1},r_{in2},r_{out}}>}$
This layer’s $r_{in1}$, $r_{in2}$ and $r_{out}$ can achieve
consumption/production rates of 1 without constraints from previous layers or
the memory, due to their element-wise functionality and the simplicity of
their operations. It should be noted that in cases when the rates in either of
the inputs are restricted owing to a lower production rate of a previous layer
or due to memory constraints, the layer’s input rates are equalised to the
lower consumption rate among them.

###  III-D Streaming-Centric Optimizations

During the development of the proposed design, the squeeze and excitation
modules was found to hinder the throughput. Due to its averaging over the
whole feature map, the 3D Global Average Pooling layer delays the design
pipeline and demands increased branch stream buffering. The interdependence
between these layers’ output and the input of the ones that follow (the layer
after GAP is always a point-wise convolution that requires all input channels
to begin producing its output) and the shortage of on-chip memory due to the
buffers’ larger size limit performance. An approximation was introduced to
solve this. GAP re-uses the results of its previous execution saved on the on-
chip memory for each subsequent volume of data (i.e., every subsequent batch)
and forwards them immediately to the next layer in the pipeline. Thus, the
pipeline never stalls and buffering is greatly reduced. The GAP now calculates
and saves the current batch’s results on the on-chip memory without affecting
the pipeline’s other stages.

Because the authors of the original publication did not publish the model’s
performance in that specific benchmark, the model was first trained/fine-tuned
before being evaluated on the UCF101 benchmark. Following the model’s fine-
tuning, the evaluation was carried out utilising the widely accepted strategy
used on HAR benchmarks, which results in a total of 30 predictions that are
averaged from a uniform sampling of 10 temporal clips for each video and
spacial sampling of 3 crops in each frame. Furthermore, to be equivalent and
directly comparable to previous studies, the impact of employing fixed-point
precision has been investigated, by converting the model weights and all
intermediate feature mappings from 32-bit floating point to 16-bit fixed-point
arithmetic.

Figure 3 depicts the results of quantization of both the weights and feature
maps over different combinations of word lengths. The experiments were
conducted on the first split of UCF101 with a single clip evaluation (compared
to 30 clip evaluation as described above). The accuracy achieved for this
split with floating point feature maps and weights is 95.71%. On a 16-bit
fixed point precision for both the weights and feature maps we observe a drop
in accuracy of 0.28%. Moving to a 16-bit precision this drop increases to
0.66%, while at 14-bits the accuracy drop is equal to 1.19%. An interesting
insight emerged from this evaluation is that the feature maps are less
sensitive to reduction of the word length used to represent them compared to
the weights of the model. For all the experiments we have explored and kept
the best combination of the number of integer and fractional bits for each
word length both for weights and feature maps. Finally we have evaluated the
model on all three splits of UCF101 with a 30 clip evaluation using 16-bit
fixed point for weights ($Q6.10$) and for feature maps ($Q7.9$). The accuracy
drop was negligible dropping from 96.56% to 96.43%.

Figure 3: Quantization results on UCF101 over different word lengths. $W\\_X$
denotes the word length of the fixed point representation of the weights,
while $FM\\_X$ for the feature maps

##  IV Design Space Exploration

The hardware mapping of the SDFG assumes a final streaming architecture to be
inferred. Each design point in the design space has a specific combination of
the involved layers’ tunable parameters as they were described in section
III-C. Essentially using a set of transformations to the SDFG the
aforementioned parameters are being altered while the design space is being
explored by simulated annealing, the heuristic optimization algorithm used in
this study.

###  IV-A X3D Model Partitioning

There are primarily two techniques for designing CNN hardware architectures.
(a) Single computation engines rely on a more general and powerful processing
unit that is time-shared across the network’s layers, as well as a scheduler,
to execute operations in the right sequence.  (b) Streaming architectures,
such as the one presented, consist of a unique hardware block for each CNN
layer that is optimised and exploits the parallelism of that layer
independently of the others. In such designs, the more layers a CNN has or the
larger the network’s input gets, the greater the FPGA resource utilisation
becomes, which may limit the achieved parallelism of each layer in order to
fit all layers into a single design.

Splitting the network execution into smaller partitions can solve this kind of
problem while at the same time exploiting the reconfiguration capabilities of
the FPGA. Furthermore, by producing a unique architecture and delivering a
separate bitstream for each partition, it allows the design of more finely
tuned architectures that are better tailored to each layer’s characteristics.
This approach also drastically reduces off-chip memory access to only the
design’s input and output streams, allowing the on-chip memory to be used for
data reuse. Following this strategy, the cost of reconfiguration must be
considered whenever a new partition is loaded onto the board, although this
cost may be mitigated by increasing the amount of data each partition have to
process (i.e. by increasing the batch size).

The partitions in this work are determined based on three primary layer types
identified in the X3D model. Figure 4 depicts the structure of the suggested
layer types.

(a)

(b)

(c)

Figure 4: X3D model main partitions types: (a) Type 1 (b) Type 2 (c) Type 3

The X3D model comprise of 26 such layers, 4 of which are of layer type 1, 11
of which are of layer type 2, and 11 of which are of layer type 3. Each layer
type, as can be observed from their structure, displays branching behaviour,
with type 1 and 2 exhibiting double branching owing to the squeeze and
excitation module [6].

###  IV-B DSE Within X3D Partitions

Two factors determine the parallelism of each X3D partition layer’s tunable
parameters. The first is the number of parallel coarse operations in each
layer, which depends on the input feature map’s channels. The primary
operations of each layer can be executed in parallel by deploying several
instances of the processing block up to the number of channels because the
operations in each channel dimension are independent. Both input channels and
output filters can use coarse-level parallelism in 3D convolutional layers.
The $s_{in}$ and $s_{out}$ parameters of the hardware building block
configuration are updated and searched for optimal values throughout the DSE
to achieve this parallelism, affecting the stream matrix S, the topology
matrix $\Gamma$, and therefore the design performance.

The second parallelism factor affects the 3D convolutional layers and
determines the dot product operation’s parallelism during the kernel’s
convolution with an input volume. This parallelism specifies how many
multipliers will be parallel for multiplications and how many levels on the
adder tree for additions. A completely unrolled configuration uses N
multipliers and N-1 adders, producing 1 dot product per cycle. Limiting the
setup to a single multiplier and adder produces 1/N dot products per cycle,
where N is the shape of the flattened input and kernel. As can be evident, a
trade-off exists between throughput and resource utilisation. The DSE
optimises the $p_{mac}$ parameters of the 3D convolutional layer hardware
building block configuration to achieve this parallelism, altering the rate
matrix R, the topology matrix $\bm{\Gamma}$, and defining the design
performance.

###  IV-C Performance Modeling

To describe the performance of a given design based on the topology matrix
$\Gamma$, an additional matrix reflecting the workload of each layer must be
included. Because the topology matrix provides the throughput of each layer at
its input in consumptions/cycle and output in productions/cycle, constructing
a matrix with the total workload of each layer, i.e. the total number of
elements to be consumed and produced, will allow the generation of a new
matrix that provides the number of cycles each layer requires to consume its
workload. More specifically a workload matrix W has the same structure as the
topology matrix $\Gamma$ in terms of nodes and arcs of the SDFG. The values of
this matrix indicate the total number of elements to be consumed at the input
of each layer as well as the total number of elements to be produced at the
output of each layer. By element-wise dividing the W matrix with the $\Gamma$
matrix, the final $II$ matrix is being calculated as shown below:

$II=W/\Gamma$ (1)

The $II$ is the initiation interval matrix, and its entries represent the
total number of cycles required by each layer to consume its workload
completely. The maximum value of the $II$ matrix, denoted by $II_{max}$
determines the initiation interval of the whole SDFG. The total execution time
of a partition with batch size B is given by the following equation:

$\operatorname{t(B,\Gamma)}=\frac{1}{\textrm{clock
rate}}\cdot(D+II_{max}\cdot(B-1))$ (2)

where $D$ is the total number of cycles needed to fill the pipeline depth of
the whole design, and its calculated by adding the individual depths of each
layer and the extra depth added due to the extra buffering to deal with the
branches in the design.

In order to capture the model’s overall execution time, the execution times of
each individual partition are summed up with the addition of the total
reconfiguration time, as shown below:

$\operatorname{t_{total}(B,\Gamma)}=\sum_{n=0}^{N_{p}}t_{n}(B,\Gamma_{i})+(N_{p}-1)\cdot
t_{reconfig}$ (3)

where $N_{p}$ is the total number of the partitions of the model, and
$t_{reconfig}$ is the reconfiguration time needed before loading each
partition to the FPGA. As can be noticed from Eq. 3, the extra overhead caused
by the device reconfiguration is proportional to the number of partitions of
the final solution and is independent of the batch size. By increasing the
number of batches processed by the model, the first term dominates the
execution time and the cost of reconfiguration is amortised.

Finally the overall throughput of the proposed architecture is inferred by
dividing the total workload of the model in GOPs (Giga Operations) times the
batch size, with the total execution time:

$\operatorname{Throughput(B)}=\frac{Workload_{model}*B}{t_{total}(B,\Gamma)}$
(4)

The design space exploration on each partition is described as an optimization
problem with the following objective: $max(t(B,\Gamma)),s.t.rsc(\Gamma)\leq
rsc_{avail}$. The simulated annealing [11] algorithm, a heuristic approach
used to tackle this optimization problem, attempts to optimise the design’s
throughput while ensuring that FPGA resource use does not exceed the available
resources (as these provided in the device specifications)

##  V Evaluation

For evaluation purposes, the ZCU102 has been used as the target platform. In
all of the experiments the target frequency was set to 142 MHz. Vitis HLS and
Vivado Design Suite (v21.2) were used, while the reported resource results are
after place and route. In all the designs the arithmetic precision used was
16-bit fixed point arithmetic as describen in Section III-D. The UCF-101 [13]
HAR benchmark was used to assess the correctness of the X3D model and position
it in relation to prior studies. Along with the baseline architecture, a
second one is proposed that uses statistics from previous batches of GAP
layers.

###  V-A Model Prediction Accuracy Evaluation

Figure 5: Latency (ms) MAPE between modeling performance and co-simulation

To ensure that the performance of X3D predicted by the modelling is
representative of the actual performance provided by co-simulation results, a
series of experiments has been conducted for each supported layer type with
different configurations for the layer input and output shapes, as well as the
coarse in and coarse out factors. The metric of Mean Absolute Percentage Error
(MAPE) was used to measure the error on the predicted vs co-simulation latency
for all the experiments. As can be seen from Figure 5, the errors vary
depending on the coarse in and coarse out factors; the bigger the coarse
factors get the smaller the error becomes. On average, layer type 1 has an
MAPE of 11.92%, layer type 2 has 17.32%, and layer type 3 has 5.03%.

###  V-B Performance Evaluation

To the best of our knowledge, no previous works have targeted the acceleration
of X3D onto FPGA as of the date of authoring this study. To position within
the previous works, and because most current works focus on the C3D model
targeting different FPGA platforms, the assessment is being undertaken on a
number of different metrics: Clips per second (Clips/s). This metric is the
total number of clips processed over a second. The values on this metric on
Table I are measured and repodted for a batch size of 100 as denoted in the
Table’s footnotes. The positioning of the FFM-X3D related to the existing
works can be seen in Figure 6. The proposed architecture is positioned in the
pareto front of accuracy over clips/s compared to the existing works in the
field. Throughput (GOp/s). To be fully aligned with the results of the
existing works, GOp/s are accounted as the MAC (multiply and accumulate)
operations per second. For batch size 100, FFM-X3D achieves better throughput
compared to [4, 3, 14] while at the same time achieves the better accuracy
among all the existing works. DSP efficiency (GOp/s/DSP). This metric is the
normalized throughput over the DSPs used. In this work, the DPSs used are
averaged over all the 26 pre-defined partitions. A positioning of FFM-X3D in
relation to the existing works for a favorable batch size of 100, shows that
FFM-X3D does provide comparable DSP efficiency to most of the works.

Figure 6: Clips per second, positioning of FFM-X3D within the test of the
existing works

Given that the proposed design focuses on throughput, the related works are
also evaluated based on this metric. A batch size 100 was chosen for FFM-X3D
architecture to amortise the reconfiguration cost. These batch size values can
be used in systems that analyse multiple videos in parallel, or in evaluation
processes that use multiple clips from a single video. In comparison to M. Sun
et. al. [14], the sole effort targeting the same FPGA platform, FFM-X3D
outperforms its C3D and R(2+1)D counterparts by $1.07\times$ and $1.52\times$,
respectively, while latency is $4.54\times$ and $9.11\times$ lower. Table I
summarizes the positioning of FFM-X3D in relation to the rest of the existing
works.

TABLE I: Positioning of FFM-X3D in the space of 3D CNN HAR models

H. Fan [4] H. Fan [3] H. Fan [2] Z. Liu [10] J. Shen [12]$\ddagger$ M. Sun
[14] FFM-X3D (Ours) VC709 VUS440 C3D R(2+1)D-18 Baseline GAP-approx FPGA ZC706
ZC706 Intel SX660 VC709 VC709 VUS440 ZCU102 ZCU102 ZCU102 ZCU102 Model C3D C3D
E3D C3D C3D C3D C3D R(2+1)D-18 X3D-M X3D-M GFLOPS $\ast$ 38.61 38.61 6.1 38.61
- - 38.61 26.14 6.4 6.4 Accuracy 79.87 % 81.99 % 85.17 % 82.3 % 82.3 % 82.3 %
82.3 % 88.66 % 96.5 % 96.5 % Clips/s $\dagger$ 1.84 2.09 28.32 8.65 11.18
20.36 2.05 4.11 17.58 18.72 GOp/s $\dagger$ 70.41 80.12 172.8 330.74 427.29
778 78.44 111.71 112.41 119.83 GOp/s/DSP $\dagger$ 0.087 0.103 0.109 0.092
0.281 0.511 0.065 0.092 0.052 0.055 Clock (MHz) 172 200 150 120 150 200 150
150 142 142 Precision 16-bit fixed BFP 32-bit float 16-bit fixed 16-bit fixed
16-bit fixed 16-bit fixed 16-bit fixed 16-bit fixed 16-bit fixed DSP % 90 %
86.6 % 93.3 % 99.8 % 42 % 53 % 48 % 48 % 86 % 85 % $\ast$ GFLOPS are
calculated as MACs. $\dagger$ Favorable batch size 100. $\ddagger$ The C3D
model used is different/smaller version from the original one [7].

Table II shows a performance comparison of the final FFM-X3D design on a
ZCU102 FPGA platform versus a state of the art server graded GPU platform. A
GeForce RTX 3090 with 10496 CUDA cores is the target GPU utilizing CUDA and
CuDNN libraries to optimise its solution. The proposed solution has 4.5x
longer latency, but with a significant reduced clock rate.

TABLE II: Performance comparison versus GPU

| GPU | FFM-X3D (Ours)
---|---|---
Platform | GeForce RTX 3090 | ZCU102
Frequency | 1.7 GHz | 142 MHz
Power (W) | 257.9 | 26
Batch Size | 1 | 1
Precision | 32-bit float | 16-bit fixed
Clips/s/Watt (cps/W) | 0.254 | 0.0177

##  VI Conclusion

This paper introduces FFM-X3D, a hardware architecture for X3D, a 3D CNN model
for HAR that provides state of the art performance. The proposed methodology
follows the fpgaConvNet paradigm making use of the SDF theory to describe and
map X3D into hardware designs. The revised $\Gamma$ matrix construction allows
the SDF modelling approach to deal with CNN topologies that contain branches,
something that is being taken into account throughout the design space
exploration . Experiment results reveal that the FFM-X3D provides competitive
throughput and DSP efficiency while attaining up to $16.63$% better prediction
accuracy on the selected HAR benchmark. Future work might involve expanding
the design space with extra SDFG transformations like automated partition
generation, as well as generalising to enable the support of a broader
collection of 3D CNN models, expanding to other video or volume related tasks
beyond the HAR.

## Acknowledgment

For the purpose of open access, the authors have applied a Creative Commons
Attribution (CC BY) license to any Accepted Manuscript version arising.

## References

  * [1] Duan, H., Zhao, Y., Xiong, Y., Liu, W., and Lin, D. Omni-sourced Webly-supervised Learning for Video Recognition.
  * [2] Fan, H., Luo, C., Zeng, C., Ferianc, M., Que, Z., Liu, S., Niu, X., and Luk, W. F-E3D: FPGA-based acceleration of an efficient 3D convolutional neural network for human action recognition.  In Proceedings of the International Conference on Application-Specific Systems, Architectures and Processors (2019), vol. 2019-July, pp. 1–8.
  * [3] Fan, H., Ng, H. C., Liu, S., Que, Z., Niu, X., and Luk, W. Reconfigurable acceleration of 3D-CNNs for human action recognition with block floating-point representation.  In Proceedings - 2018 International Conference on Field-Programmable Logic and Applications, FPL 2018 (2018), no. Section III, pp. 287–294.
  * [4] Fan, H., Niu, X., Liu, Q., and Luk, W. F-C3D: FPGA-based 3-dimensional convolutional neural network.  In 2017 27th International Conference on Field Programmable Logic and Applications, FPL 2017 (2017), pp. 2–5.
  * [5] Feichtenhofer, C. X3D: Expanding Architectures for Efficient Video Recognition.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020), Feichtenhofer Christoph, Ed., pp. 200–210.
  * [6] Hu, J., Shen, L., Albanie, S., Sun, G., and Wu, E. Squeeze-and-Excitation Networks.  IEEE Transactions on Pattern Analysis and Machine Intelligence 42, 8 (2020), 2011–2023.
  * [7] Ji, S., Xu, W., Yang, M., and Yu, K. 3D Convolutional neural networks for human action recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 1 (2013), 221–231.
  * [8] Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., Suleyman, M., and Zisserman, A. The Kinetics Human Action Video Dataset.  Tech. rep., 5 2017.
  * [9] Lee, E. A., and Messerschmitt, D. G. Synchronous data flow.  In Proceedings of the IEEE (1987), vol. 75, pp. 1235–1245.
  * [10] Liu, Z., Chow, P., Xu, J., Jiang, J., Dou, Y., and Zhou, J. A uniform architecture design for accelerating 2d and 3d cnns on fpgas.  Electronics (Switzerland) 8, 1 (1 2019).
  * [11] Reeves, C. R., Ed.  Modern Heuristic Techniques for Combinatorial Problems.  John Wiley & Sons, Inc., USA, 1993.
  * [12] Shen, J., Huang, Y., Wang, Z., Qiao, Y., Wen, M., and Zhang, C. Towards a uniform template-based architecture for accelerating 2d and 3D CNNs on FPGA.  In FPGA 2018 - Proceedings of the 2018 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays (2018), vol. 2018-Febru, pp. 97–106.
  * [13] Soomro, K., Zamir, A. R., and Shah, M. UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild.  Tech. rep., 12 2012.
  * [14] Sun, M., Zhao, P., Gungor, M., Pedram, M., Leeser, M., and Lin, X. 3D CNN acceleration on FPGA using hardware-aware pruning.  In Proceedings - Design Automation Conference (2020), vol. 2020-July, IEEE.
  * [15] Toupas, P., Montgomerie-Corcoran, A., Bouganis, C.-S., and Tzovaras, D. HARFLOW3D: A Latency-Oriented 3D-CNN Accelerator Toolflow for HAR on FPGA Devices.  In Proceedings - 2023 International Symposium on Field-Programmable Custom Computing Machines, FCCM 2023.
  * [16] Tran, D., Wang, H., Torresani, L., Ray, J., Lecun, Y., and Paluri, M. A Closer Look at Spatiotemporal Convolutions for Action Recognition.  Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (2018), 6450–6459.
  * [17] Venieris, S. I., and Bouganis, C. S. FpgaConvNet: Mapping Regular and Irregular Convolutional Neural Networks on FPGAs.  IEEE Transactions on Neural Networks and Learning Systems 30, 2 (2019), 326–342.
  * [18] Winograd, S. Arithmetic Complexity of Computations.  Society for Industrial and Applied Mathematics, 1 1980.


* [57] G. Maurice, J.F. Ganghoffer, and Y. Rahali.  Second gradient homogenization of multilayered composites based on the method of oscillating functions.  Mathematics and Mechanics of Solids, 24(7):2197–2230, 2019.  Publisher: SAGE Publications Sage UK: London, England.
  * [58] C. Miehe.  A multi-field incremental variational framework for gradient-extended standard dissipative solids.  Journal of the Mechanics and Physics of Solids, 59(4):898–923, 2011\.
  * [59] C. Miehe, J. Schotte, and J. Schröder.  Computational micro-macro transitions and overall moduli in the analysis of polycrystals at large strains.  Computational Materials Science, 16(1-4):372 – 382, 1999.
  * [60] C. Miehe, J. Schröder, and J. Schotte.  Computational homogenization analysis in finite plasticity simulation of texture development in polycrystalline materials.  Computer methods in applied mechanics and engineering, 171(3-4):387–418, 1999.  ISBN: 0045-7825 Publisher: Elsevier.
  * [61] R.D. Mindlin.  Micro-structure in linear elasticity.  Archive for Rational Mechanics and Analysis, 16(1):51–78, 1964\.
  * [62] R.D. Mindlin.  Second gradient of strain and surface-tension in linear elasticity.  International Journal of Solids and Structures, 1(4):417–438, 1965\.  ISBN: 0020-7683 Publisher: Elsevier.
  * [63] R.D. Mindlin and N.N. Eshel.  On first strain-gradient theories in linear elasticity.  International Journal of Solids and Structures, 4(1):109–124, January 1968.
  * [64] R.D. Mindlin and H.F. Tiersten.  Effects of couple-stresses in linear elasticity.  Archive for Rational Mechanics and Analysis, 11(1):415–448, January 1962.
  * [65] T. Mori and K. Tanaka.  Average stress in matrix and average elastic energy of materials with misfitting inclusions.  Acta metallurgica, 21(5):571–574, 1973.  ISBN: 0001-6160 Publisher: Elsevier.
  * [66] H. Moulinec and P. Suquet.  A numerical method for computing the overall response of nonlinear composites with complex microstructure.  Computer methods in applied mechanics and engineering, 157(1-2):69–94, 1998.  ISBN: 0045-7825 Publisher: Elsevier.
  * [67] R.H.J. Peerlings, R. de Borst, W.A.M. Brekelmans, and J.H.P. De Vree.  Gradient enhanced damage for quasi-brittle materials.  International Journal for numerical methods in engineering, 39(19):3391–3403, 1996.  ISBN: 0029-5981 Publisher: Wiley Online Library.
  * [68] L. Piegl and W. Tiller.  The NURBS Book.  Springer, 2nd edition, 2010.
  * [69] O. Rokoš, M.M. Ameen, R.H.J. Peerlings, and M.G.D. Geers.  Micromorphic computational homogenization for mechanical metamaterials with patterning fluctuation fields.  Journal of the Mechanics and Physics of Solids, 123:119–137, 2019\.
  * [70] E. Sanchez-Palencia.  Homogenization method for the study of composite media.  In Asymptotic Analysis II—, pages 192–214. Springer, 1983\.
  * [71] J. Schröder.  A numerical two-scale homogenization scheme: the FE2-method. Springer, 2014.
  * [72] J. Schröder and M.-A. Keip.  Two-scale homogenization of electromechanically coupled boundary value problems.  Computational Mechanics, 50:229–244, 2012.  ISBN: 1432-0924.
  * [73] J. Schulte, M. Dittmann, S.R. Eugster, S. Hesch, T. Reinicke, F. Dell’Isola, and C. Hesch.  Isogeometric analysis of fiber reinforced composites using Kirchhoff–Love shell elements.  Computer Methods in Applied Mechanics and Engineering, 362:112845, 2020.  Publisher: Elsevier.
  * [74] S. Schuß, M. Dittmann, B. Wohlmuth, S. Klinkel, and C. Hesch.  Multi-patch isogeometric analysis for kirchhoff–love shell elements.  Computer Methods in Applied Mechanics and Engineering, 349:91–116, 2019.
  * [75] R.J.M. Smit, W.A.M. Brekelmans, and H.E.H. Meijer.  Prediction of the mechanical behavior of nonlinear heterogeneous systems by multi-level finite element modeling.  Computer methods in applied mechanics and engineering, 155(1-2):181–192, 1998.  ISBN: 0045-7825 Publisher: Elsevier.
  * [76] J. Spahn, H. Andrä, M. Kabel, and R. Müller.  A multiscale approach for modeling progressive damage of composite materials using fast Fourier transforms.  Computer Methods in Applied Mechanics and Engineering, 268:871–883, 2014.  ISBN: 0045-7825 Publisher: Elsevier.
  * [77] S. Teichtmeister, D. Kienle, F. Aldakheel, and M.-A. Keip.  Phase field modeling of fracture in anisotropic brittle solids.  International Journal of Non-Linear Mechanics, 97:1–21, 2017.  ISBN: 0020-7462 Publisher: Elsevier.
  * [78] R.A. Toupin.  Elastic materials with couple-stresses.  Archive for rational mechanics and analysis, 11(1):385–414, 1962\.
  * [79] R.A. Toupin.  Theories of elasticity with couple-stress.  Archive for Rational Mechanics and Analysis, 17(2):85–112, 1964\.  ISBN: 1432-0673 Publisher: Springer.
  * [80] Z.-P. Wang and L.H. Poh.  Optimal form and size characterization of planar isotropic petal-shaped auxetics with tunable effective properties using IGA.  Composite Structures, 201:486–502, 2018.  ISBN: 0263-8223 Publisher: Elsevier.
  * [81] J.R. Willis.  Bounds and self-consistent estimates for the overall properties of anisotropic composites.  Journal of the Mechanics and Physics of Solids, 25:185 – 202, 1977\.
  * [82] J. Yvonnet, N. Auffray, and V. Monchiet.  Computational second-order homogenization of materials with effective anisotropic strain-gradient behavior.  International Journal of Solids and Structures, 191-192:434–448, 2020.
  * [83] A. Zervos.  Finite elements for elasticity with microstructure and gradient elasticity.  International Journal for Numerical Methods in Engineering, 73(4):564–595, 2008.