# Design of Many-Core Big Little $\mu$Brains for Energy-Efficient Embedded
Neuromorphic Computing

M. L. Varshika†, Adarsha Balaji†, Federico Corradi‡, Anup Das†, Jan Stuijt‡
and Francky Catthoor†† †Electrical and Computer Engineering, Drexel
University, USA
‡Stichting IMEC Nederland, Netherlands
††IMEC, Belgium
Corresponding Email<EMAIL_ADDRESS>
###### Abstract

As spiking-based deep learning inference applications are increasing in
embedded systems, these systems tend to integrate neuromorphic accelerators
such as $\mu$Brain to improve energy efficiency. We propose a $\mu$Brain-based
scalable many-core neuromorphic hardware design to accelerate the computations
of spiking deep convolutional neural networks (SDCNNs). To increase energy
efficiency, cores are designed to be heterogeneous in terms of their neuron
and synapse capacity (big cores have higher capacity than the little ones),
and they are interconnected using a parallel segmented bus interconnect, which
leads to lower latency and energy compared to a traditional mesh-based
Network-on-Chip (NoC). We propose a system software framework called SentryOS
to map SDCNN inference applications to the proposed design. SentryOS consists
of a compiler and a run-time manager. The compiler compiles an SDCNN
application into sub-networks by exploiting the internal architecture of big
and little $\mu$Brain cores. The run-time manager schedules these sub-networks
onto cores and pipeline their execution to improve throughput. We evaluate the
proposed big little many-core neuromorphic design and the system software
framework with five commonly-used SDCNN inference applications and show that
the proposed solution reduces energy (between 37% and 98%), reduces latency
(between 9% and 25%), and increases application throughput (between 20% and
36%). We also show that SentryOS can be easily extended for other spiking
neuromorphic accelerators.

###### Index Terms:

neuromorphic computing, spiking deep convolutional neural networks, many-core,
embedded systems, $\mu$Brain

##  I Introduction

Spiking deep convolutional neural network (SDCNN)-based inference applications
are increasing in embedded systems [1]. To improve energy efficiency, such
systems tend to integrate neuromorphic accelerators such as $\mu$Brain [2],
DYNAPs [3], and Loihi [4]. We take the example of $\mu$Brain, which is a
neural architecture with three layers: 16 neurons in the first layer, 64
neurons in the second, and 256 neurons in the third (see Fig. 1). $\mu$Brain
is an asynchronous (clock-less) digital design with fully programmable
connections between the three layers. $\mu$Brain is shown to consume only
308nJ energy for digit recognition [2].

$\mu$Brain implements 336 neurons and 38K synaptic connections. One way to use
$\mu$Brain for larger SDCNN applications is to scale-up the design. However,
this leads to a substantial increase in static power and area (see Section
II). We propose a scalable solution by interconnecting many small $\mu$Brain
cores. However, instead of using the same neuron and synapse capacity for all
cores as in all previous designs, we show that a heterogeneous architecture
with different core capacities can improve energy efficiency of the proposed
many-core design.

Conventionally, mesh-based Network-on-Chip (NoC) is used to interconnect cores
in recent neuromorphic designs [5]. However, NoC interconnect has relatively
long time-multiplexed connections that need to be near-continuously powered up
and down, reaching from the ports of data producers/consumers (inside a core
or between different cores) up to the ports of communication switches [6, 7,
8]. Recently, segmented bus (SB)-based interconnect is proposed as an
alternative for neuromorphic hardware [9]. Here, a bus lane is partitioned
into segments, where interconnections between segments are bridged and
controlled by switches [10]. We propose a dynamic segmented bus architecture
with multiple segmented bus lanes to interconnect big little $\mu$Brain cores
in the proposed design (see Figure 2). An optimized controller is designed to
perform mapping of communication primitives to segments by profiling the
communication pattern between different $\mu$Brain cores for a given SDCNN
application. Based on this profiling and mapping, switches in the interconnect
are programmed once at design-time before admitting an application to the
hardware. By avoiding run-time routing decisions, the proposed design
significantly gains on energy and latency.

We propose SentryOS, a system software for mapping SDCNN applications to the
proposed segmented bus-based many-core big little $\mu$Brain design. SentryOS
consists of a compiler (SentryC) and a run-time manager (SentryRT). SentryC
compiles an SDCNN inference application into sub-networks by exploiting the
internal architecture of big and little $\mu$Brain cores. SentryRT uses a
dataflow analysis technique to schedule sub-networks to $\mu$Brain cores by
improving opportunities for pipelining and exploiting data-level parallelism.

We show in Section IV that SentryOS can be easily extended to other many-core
spiking neuromorphic designs such as DYNAPs [3] (which is similar to
$\mu$Brain with synaptic memory integrated closer to neuron circuitry in each
core, but with more neurons and synapses per core) and Loihi [4] (where
synaptic memory is off-chip to neuron circuity).

Following are our contributions to the neuromorphic field.

  * •

A many-core neuromorphic platform template design based on the digital
asynchronous (clock-less) $\mu$Brain architecture. Cores in the proposed
design are heterogeneous in terms of their neuron and synapse capacity. The
main objective here is to reduce energy (Section III).

  * •

A parallel segmented bus-based interconnect for data communication between
$\mu$Brain cores in the proposed many-core design. A controller to map inter-
core communication to segments for parallel execution. The main objective here
is to minimize energy and latency (Section III).

  * •

A system software framework (SentryOS) to map SDCNN applications to the
proposed design. SentryOS consists of a compiler, which compiles an
application into sub-networks and a run-time manager, which schedules sub-
networks to $\mu$Brain cores of the many-core hardware. The main objective
here is to improve throughput (Section IV).

We evaluate the proposed design with five commonly-used SDCNN applications and
show improvement in energy (average 67%), latency (average 18%), and
throughput (average 25%).

To the best of our knowledge, this is the first work that proposes a many-core
neuromorphic design with heterogeneous core capacity and using a segmented bus
interconnect.

##  II Background

###  II-A $\mu$Brain: A Digital Inference Hardware

$\mu$Brain is an asynchronous and fully-synthesizable digital inference
hardware designed in 40nm CMOS [2]. Figure 1a shows the internal architecture
of a $\mu$Brain design with three layers, which are referred to as l0, l1, and
l2. There are 336 neurons in the chip, which are of integrate-and-fire (IF)
type [11]. Figure 1b shows the state transitions in an IF neuron in
$\mu$Brain. Every neuron independently (without a global clock) accumulates
weighted incoming synaptic spikes and emits a spike itself when the neuron’s
accumulator overflows. Input spikes trigger the membrane voltage integration,
with immediate threshold evaluation, resulting in distributed granular
activations. Synaptic memory is tightly integrated in the design and
distributed closer to neurons, minimizing data (spike) movement. Static power
of the design is $40.3\mu W$ and the dynamic energy per spike is $26pJ$ [2].

Synaptic connections between the three layers are fully-programmable, allowing
implementation of SDCNN operations such as convolution, pooling, concatenation
and addition, as well as irregular network topologies, which are commonly
found in many emerging SDCNN models (see Figure 3).

(a) 3-layered $\mu$Brain architecture. (b) State diagram of an IF neuron.

Figure 1: $\mu$Brain architecture and neuron state diagram [2].

$\mu$Brain can be scaled up to implement larger network topologies. However,
static power and design area increases substantially for larger configurations
as reported in Figure 1a. To improve energy efficiency, we propose a scalable
many-core design where each core is a tiny-scale $\mu$Brain (see Section III).

###  II-B Other Neuromorphic Hardware Designs

Table I shows the capacity of recent neuromorphic designs. Here, we review two
representative designs – DYNAPs and Loihi, and show that the proposed SentryOS
framework can be easily extended to map SDCNN applications on them.

TABLE I: Capacity of recent neuromorphic hardware platforms [12, 13, 14, 15, 16]. | ODIN |  $\mathbf{\mu}$Brain | DYNAPs | BrainScaleS | SpiNNaker | Neurogrid | Loihi | TrueNorth
---|---|---|---|---|---|---|---|---
# Neurons/core | 256 | 336 | 256 | 512 | 36K | 65K | 130K | 1M
# Synapses/core | 64K | 38K | 16K | 128K | 2.8M | 8M | 130M | 256M
# Cores/chip | 1 | 1 | 1 | 1 | 144 | 128 | 128 | 4096
# Chips/board | 1 | 1 | 4 | 352 | 56 | 16 | 768 | 4096
# Neurons | 256 | 336 | 1K | 4M | 2.5B | 1M | 100M | 4B
# Synapses | 64K | 38K | 65K | 1B | 200B | 16B | 100B | 1T

DYNAPs [3] is a mixed-signal inference hardware with four neurosynaptic cores.
Each core can map up to 256 neurons and 16K synaptic connections. Cores are
interconnected using a hierarchical NoC with mesh routing. Weights of a fully-
trained network (i.e., the inference) are programmed to the synaptic cells of
the four cores. Once programmed, DYNAPs can perform inference on streaming
data continuously. Each DYNAPs core uses a crossbar where neurons are
organized into two layers. Synaptic memory is tightly integrated with neuron
circuits as in $\mu$Brain. SpiNeMap [17] is used to map applications on
DYNAPs.

Loihi [4] is a digital design of a many-core neuromorphic hardware consisting
of 128 cores with mesh routing. A Loihi core can map up to 130K neurons and
130M synaptic connections. Unlike $\mu$Brain and DYNAPs, synaptic memory in
Loihi is off-chip to the neuron circuitry, leading to higher data movements.
The LAVA framework [18] is used to map applications on Loihi.

###  II-C System Software for Neuromorphic Hardware

Apart from SpiNeMap and LAVA, there are also other system software frameworks
for spiking neuromorphic accelerators – neutram [19], neuroxplorer [20],
pacman [21], and dfsynthesizer [22], among others [23, 24, 25, 26]. All these
frameworks use graph partitioning technique to first partition an SDCNN into
clusters and then place these clusters onto homogeneous neuromorphic cores
connected in a mesh-based topology. They cannot be easily extended to hardware
with different core capacities and interconnected using segmented bus
interconnect.

##  III Many-Core $\mu$Brain Design

Figure 2 illustrates a high-level architecture of the proposed many-core big
little $\mu$Brain platform template design, where big (B) and little (L) cores
are interconnected using parallel bus lanes that are segmented using
segmentation switches (S). An SDCNN application (specified using Nengo [27],
PyCARL [28] or PyNN [29]) is admitted to this platform using the proposed
SentryOS framework. A bus controller is used to map inter-core data
communication to parallel segments.

Figure 2: Many-core big little $\mu$Brain with dynamic segmented bus
interconnect.

###  III-A Big Little $\mu$Brain Design

Figure 3 illustrates the structure of five commonly-used SDCNNs – LeNet
(1989), AlexNet (2012), VGGNet (2015), ResNet (2015), and DenseNet (2017)
[30]. We observe that LeNet, AlexNet, and VGGNet have a regular topology with
chain-like connections. However, emerging CNNs such as ResNet (identity
shortcut connections) and DenseNet (one-to-all subsequent layer connections)
have an irregular topology [31]. Additionally for energy-efficient
implementation on embedded devices, SDCNNs are subject to connection pruning,
where connections with near-zero synaptic weights are removed [32]. Such
pruning creates an irregular topology, even for LeNet, AlexNet, and VGGNet,
making it difficult to map them onto hardware accelerators [33, 34, 35, 36,
37].

To this end, Table II reports the minimum, maximum, and average number of L1
and L2 neighbors of neurons from these five SDCNNs.111L1 neighbors of a neuron
is the set of pre-synaptic neurons that are connected directly to this neuron.
L2 neighbors of a neuron is the set of pre-synaptic neurons that are connected
to L1 neighbors of the neuron. We observe that the number of L1 and L2
neighbors of neurons in an SDCNN varies widely. To use $\mu$Brain for these
models, the $\mu$Brain design needs to be optimally configured to accommodate
the maximum number of L1 and L2 neighbors. This is reported as the
conservative $\mu$Brain design choice in row 8 of Table II.

Figure 3: Structure of five commonly used CNNs with convolution (C), pooling (P), concatenation (concat), and addition (+) operations. TABLE II: Statistics of CNN sub-networks [30]. | L1 neighbors | L2 neighbors
---|---|---
| Min | Max | Avg | Min | Max | Avg
LeNet | 18 | 144 | 115.8 | 128 | 400 | 369.3
AlexNet | 9 | 2566 | 41.9 | 11 | 10154 | 544.2
VGGNet | 3 | 288 | 186.3 | 16 | 14772 | 2437.8
ResNet | 12 | 288 | 244.9 | 16 | 1568 | 695.9
DenseNet | 3 | 288 | 162.8 | 16 | 14772 | 1422.3
Conservative $\mu$Brain design |  L1 neurons = 4,096 |  L2 neurons = 16,384

Figure 4 compares the average energy-per-core of the conservative design
against a fully-custom design, where $\mu$Brain cores are configured based on
the number of L1 and L2 neighbors of neurons for each SDCNN application.
Energy numbers are normalized to the conservative design. We observe that the
average energy-per-core of the fully-custom design is on average 36% lower
than the conservative design. This is because the conservative design is sized
based on the maximum number of l1 and l2 neighbors of a neuron. Such worst-
case connectivity occurs only rarely in most applications and therefore, many
synaptic connections remain unutilized. This leads to a high static power
overhead. In Section V-D, we show that using only a few (e.g., 4) $\mu$Brain
configurations, we can achieve similar energy efficiency as a fully-custom
design.

Figure 4: Average energy-per-core for five SDCNN applications.

###  III-B Segmented Bus Interconnect

Figure 5 illustrates the concept of a segmented bus-based many-core design
(right) and its difference with a conventional bus (left) for interconnecting
7 $\mu$Brain cores ($\mathbf{C_{1}}$-$\mathbf{C_{7}}$). Let the core
$\mathbf{C_{x}}$ has $\mathbf{M_{x}}$ input ports and $\mathbf{N_{x}}$ output
ports. Without loss of generality, we consider a scenario where cores
$\mathbf{C_{1}}$-$\mathbf{C_{4}}$ can only connect to the inputs of
$\mathbf{C_{5}}$-$\mathbf{C_{7}}$. Fig. 5 (left) shows a single shared bus
connecting $G=N_{1}+N_{2}+N_{3}+N_{4}$ output ports with $H=M_{5}+M_{6}+M_{7}$
input ports. While using shared bus, only one connection between any pair of
input-output cluster is possible at a given time (shown with the arrow),
resulting in underutilization of the bus. A simple segmented bus allows to
overcome this problem by breaking the bus into multiple segments. As seen from
Fig. 5 (right), a single segmented bus can accommodate many simultaneous
connections.

Figure 5: Segmented bus-based many-core design (right) and its difference with
a conventional bus (left).

However, a single bus lane may not always be enough to support concurrent
connection requests. For instance, any connection request from
$\mathbf{C_{3}}$ to $\mathbf{C_{5}}$ will be blocked while $\mathbf{C_{2}}$ is
communicating to $\mathbf{C_{6}}$. This can be solved using parallel bus lanes
as illustrated in Figure 2. To achieve full connectivity at all time, $G*H$
separate busses are needed. However, such a full connection is rarely
exercised in a typical SDCNN application. A more realistic scenario is the one
where at different time intervals a different connection profile is active. We
analyze inter-core communications based on training data to identify the
minimum number of bus lanes needed in the parallel segmented bus interconnect
at any given time. This minimum value is not dependent on the number of cores.
The key idea here is to exploit the dynamism present in different SDCNN
applications to virtualize the segmented bus interconnect between different
cores. Hence, the bandwidth allocation needed at design-time can be reduced
from the physical maximum to what is maximally happening concurrently.
Consequently, the active wire length is reduced, which lowers energy. A bus
controller is designed to control segmentation switches and map inter-core
communications on parallel segments for concurrent execution. This reduces
spike latency.

##  IV Detailed Design of SentryOS

We represent an SDCNN application as a graph. Formally,

###### Definition 1

(SDCNN Graph) An SDCNN application
$\mathbf{G_{SDCNN}=(\mathbf{N},\mathbf{E})}$ is a directed graph consisting of
a finite set ${\mathbf{N}}$ of nodes, representing neurons and a finite set
${\mathbf{E}}$ of edges, representing synaptic connections.

SentryOS consists of a compiler (SentryC) and a run-time manager (SentryRT) as
illustrated in Figure 6.

Figure 6: Detailed architecture of SentryRT.

###  IV-A Compiler (SentryC) Design

SentryC partitions the graph $G_{SDCNN}$ into sub-networks
$\\{S_{0},S_{1},\cdots\\}$, where $S_{i}\bigcap S_{j}=\emptyset$. A sub-
network $S_{i}=\\{\texttt{l2}_{i}~{}|~{}\texttt{l1}_{i}~{}|\texttt{l0}_{i}\\}$
is a set of neurons that are organized into three layers – l2, l1, and l0.
$Area(S_{i})$ is the area of the $\mu$Brain core needed to implement $S_{i}$
and $Power(S_{i})$ is its static power. Algorithm 1 shows the pseudo-code of
SentryC.

The algorithm operates in four steps. First, for each output neuron, we assign
a distance value to all other neurons, where the distance is computed as the
longest path from a neuron to this output neuron. Neurons that are not
connected to this output neuron are assigned a large number (to prevent them
from grouping in the same sub-network). Neurons with the same distance are
clustered together as shown in Figure 7a. Second, we index each neuron by the
distance sequence as shown in Figure 7b and form a search path for
partitioning the graph into sub-networks: each sub-network must contain
neurons with contiguous indexes. In this manner, any arbitrary network
topology is transformed into a linear one.

Figure 7: Distance-based partitioning an SDCNN application into subnetworks.

Third, we cluster neurons with distance of up to 2 into a sub-network, i.e.,
$S_{0}=\\{N_{j}~{}|~{}dist(N_{j})\leq 2\\}$. This is to ensure that each sub-
network can fit into the three-layered architecture of $\mu$Brain. We compute
the remaining graph as $G^{\prime}=G-\\{S_{0}\\}$. Next, we recursively
partition the remaining graph $G^{\prime}$ for each node with distance 2 as
the new output neuron and compute distance of all other neurons in
$G^{\prime}$ by traversing backward.

(a) Four sub-networks of the SDCNN example of Figure 7. (b) Scheduling sub-
networks to $\mu$Brain pipelines.

Figure 8: (a) Generating sub-networks by exploiting area-power benefits and
(b) scheduling sub-networks of an SDCNN application to $\mu$Brain pipelines by
exploiting data-level parallelism.

Figure 8a shows the four sub-networks of the SDCNN application of Figure 7. In
generating a sub-network, it may be necessary to create neurons with unit
synaptic strength in order to fit onto the three-layer architecture of
$\mu$Brain. This is shown with the dotted neurons and connections inside sub-
networks 1, 2, and 3.

Finally, we merge sub-networks (lines 7-12 of Algorithm 1) by considering area
and power benefits of big and little $\mu$Brain cores. We formulate this
problem as follows. Let $S_{i}$ and $S_{j}$ be two sub-networks of an SDCNN
application. Formally, the merged sub-network is represented as
$S_{i,j}=\\{S_{i,j}.\texttt{l2}~{}|~{}S_{i,j}.\texttt{l1}~{}|~{}S_{i,j}.\texttt{l0}\\}$,
where $S_{i,j}.\texttt{l2}=\\{S_{i}.\texttt{l2}\cup S_{j}.\texttt{l2}\\}$, and
so on. We merge $S_{i}$ and $S_{j}$ iff

  * •

${Area(S_{i,j})}<{Area(S_{i})}+{Area(S_{j})}$

  * •

${Power(S_{i,j})}<{Power(S_{i})}+{Power(S_{j})}$

###  IV-B Extension of SentryC to Other Spiking Architectures

####  IV-B1 Extension to DYNAPs

DYNAPs is a crossbar-based two-layer architecture with fixed number of l0 and
l1 neurons. To use SentryC for DYNAPs, we make the following changes. First,
we construct sub-networks with neurons that are at a distance 0 (layer 0) and
1 (layer 1) only, with a constraint on the size of a crossbar. This is
accomplished by changing line 6 of Algorithm 1 to
$S_{i}=\texttt{create\\_subnet}\left(\underset{d(n_{j})\leq
1}{\bigcup}n_{j}\text{ such that }|S_{i}|\leq C_{n}\right)$, where $C_{n}$ is
the total number of neurons that a crossbar can map. Second, we place all
neurons that are at a distance of 2 or higher as candidates for sub-network
generation in the next iteration. To do so, we change line 14 to
$newNS.append(\underset{d(n_{j})=2}{\bigcup}n_{j})$. Finally, we assign all
unassigned neurons from the current iteration and the new candidates to the
list $NS$ for the next iteration of the algorithm, i.e., change line 16 to
$NS=\\{NS\setminus G^{\prime}\\}\bigcup newNS$.

####  IV-B2 Extension to Loihi

To use SentryRT for Loihi, where neuron circuitry is decoupled from synaptic
memory, we propose the same changes as DYNAPs with two constraints – one on
the number of neurons and the other on the size of synaptic memory.
Additionally, while placing a sub-network to a core, we decouple the neurons
from the synapses and place them separately onto the target core architecture.

Input: $G_{SDCNN}=(\textbf{N},\textbf{E})$

Output: $G_{DFG}=(\textbf{S},\textbf{C})$

$G^{\prime}=G_{SDCNN}$;

/* Initialize $G^{\prime}$ */

$NS=\\{\underset{\texttt{OutDegree}(n_{i})=0}{\bigcup}n_{i}\\}$;

/* Place all output neurons. */

1 while  _$G^{\prime}\neq\emptyset$_ do /* Run until all neurons of the graph
are placed into sub-networks */

2 for  _$n_{i}\in NS$_ do /* For each neurons in $NS$ */

$d(n_{j})=\texttt{longest\\_path}(n_{j},n_{i})~{}~{}\forall~{}n_{j}\in
G^{\prime}$;

/* Compute distance of every neuron in $G^{\prime}$ to $n_{i}$ */

$S_{i}=\texttt{create\\_subnet}\left(\underset{d(n_{j})\leq
2}{\bigcup}n_{j}\right)$;

/* Create subnet with all neurons that have a distance of 2 or less from the
output neuron */

3

Find $S_{j}~{}|~{}j=\texttt{argmin}\\{Cost(S_{k},S_{i})~{}\forall~{}S_{k}\in
G_{DFG}\\}$;

/* Find one subnet amongst the created ones to which merging this new subnet
results in the minimum cost. Cost can be assessed in terms of area, power, or
a combination of the two, which we have used in this paper. */

4 if  _$Area(S_{i,j}) <Area(S_{i})+Area(S_{j})$ and
$Power(S_{i,j})<Power(S_{i})+Power(S_{j})$_ then /* If the area and power
costs of merging the subnets is less than the individual costs. */

$S_{j}=S_{j}\bigcup S_{i}$;

/* Merge the subnets. */

5

6 else

$G_{DFG}.\texttt{append}(S_{i})$;

/* Insert this subnet to the output DFG */

7

8 end if

$G^{\prime}=G^{\prime}-S_{i}$;

/* Remove $S_{i}$ from $G^{\prime}$ */

$newNS.append(\underset{d(n_{j})=3}{\bigcup}n_{j})$;

/* Use the set $newNS$ to hold all neurons that are at a distance of 3.
Neurons in this set will be placed in the next iteration. */

9

10 end for

$NS=newNS$;

/* Update $NS$ to $newNS$. */

11

12 end while

Algorithm 1 Compiler (SentryRT) of SentryOS.

###  IV-C Run-time Manager (SentryRT) Design

Figure 6 shows the proposed SentryRT. It queues input images to a given batch
size and process them concurrently. Although queuing increases latency, the
throughput is higher due to batch processing. SentryRT uses a dataflow
analysis technique to schedule sub-networks onto $\mu$Brain cores to improve
throughput. To this end, SentryRT uses a training batch to profile an SDCNN
application and represent its sub-networks and their interconnections as a
dataflow graph (DFG). Formally,

###### Definition 2

(SDCNN DFG Graph) An SDCNN dataflow graph
$\mathbf{G_{DFG}=(\mathbf{S},\mathbf{C})}$ is a directed graph consisting of a
finite set ${\mathbf{S}}$ of sub-networks of the SDCNN application and a
finite set ${\mathbf{C}}$ of communication channels between the sub-networks.

Each sub-network $S_{i}\in\mathbf{S}$ is associated with an execution time
$t_{i}$, which represents its computation time on a $\mu$Brain core.

SentryRT uses an analytical approach to timing analysis of $G_{DFG}$ [38]. It
consists of a novel way of constructing a Max-Plus algebraic description of
the evolution of node execution times in a self-timed execution manner. The
Max-Plus semiring $\mathbb{R}_{\text{max}}$ is the set
$\mathbb{R}\cup\\{-\infty\\}$ defined with two basic operations $\oplus\text{
and }\otimes$, which are related to linear algebra as $a\oplus
b=\max(a,b)\text{ and }a\otimes b=a+b$. The identity element $\mymathbb{0}$
for the addition $\oplus$ is $-\infty$ in linear algebra, i.e.,
$a\oplus\mymathbb{0}=a$. The identity element $\mymathbb{1}$ for the
multiplication $\otimes$ is 0 in linear algebra, i.e.,
$a\otimes\mymathbb{1}=a$. The end execution time of each node of $G_{DFG}$ can
be expressed as $\mathbf{t_{k}}=\oplus\mathbf{{T}\otimes t_{k-1}}$, where
$\mathbf{{T}}$ captures execution times of sub-networks $\tau_{n}$ and
$\mathbf{t_{k}}=\\{t_{0}(k),t_{1}(k),\cdots\\}$ is the end execution of nodes
in the $k^{th}$ iteration. Figure 9a shows the schedule obtained by solving
the Max-Plus formulation of end execution time of sub-networks of an SDCNN
micro-benchmark shown in Figure 8b.

Figure 9: Schedule of sub-networks generated by SentryRT.

To exploit parallelism, SentryRT discards the precise execution time of sub-
networks, retaining only their sequence. Sub-networks are executed in a self-
timed fashion when data is available for its neurons to compute. Figure 8b
shows the allocation of sub-networks to 6 $\mu$Brain pipelines, where each
pipeline may consist of a few $\mu$Brain cores. The corresponding multi-
pipeline schedule is illustrated in Figure 9b. Finally, SentryRT overlaps
execution of pipelines for different images of a batch to improve throughput.
Figure 9c shows this overlapped execution for three different input images.

##  V Evaluation

$\mu$Brain design is synthesized at 40 nm technology node using TSMC library
CLN40LP. Area and power numbers for different $\mu$Brain configurations are
estimated using the compact model provided in [2]. We use an in-house cycle-
accurate neuromorphic platform simulator to simulate many-core $\mu$Brain
cores interconnected using a segmented bus interconnect. We use PyCARL [28] to
simulate five commonly-used SDCNN applications with 2-bit quantized synaptic
weights. These applications are described in Table III.

TABLE III: SDCNN applications used to evaluate the proposed design. SDCNN | Dataset | Neurons | Synapses | Avg. Spikes/Image | Accuracy
---|---|---|---|---|---
LeNet | CIFAR-10 | 80,271 | 275,110 | 724,565 | 86.3%
AlexNet | CIFAR-10 | 127,894 | 3,873,222 | 7,055,109 | 66.4%
VGGNet | CIFAR-10 | 448,484 | 22,215,209 | 12,826,673 | 81.4 %
ResNet | CIFAR-10 | 266,799 | 5,391,616 | 7,339,322 | 57.4%
DenseNet | CIFAR-10 | 365,200 | 11,198,470 | 1,250,976 | 46.3%

###  V-A Energy Efficiency

Figure 10 plots the core energy of DYNAPs, Loihi, and the proposed many-core
$\mu$Brain design for all five SDCNN applications. We scale DYNAPs and Loihi
to 40nm node (the same technology node as $\mu$Brain). We use the proposed
SentryOS for all these three spiking neuromorphic accelerators. Results are
normalized to DYNAPs. We make two key observations.

Figure 10: Core energy normalized to DYNAPs.

First, energy of Loihi is on average 5% lower than DYNAPs. Although Loihi
requires higher energy to access off-chip synaptic memory, the improvement is
due to larger capacity of a Loihi core compared to DYNAPs. For smaller
applications like LeNet, energy numbers are comparable. Second, energy using
the proposed design is on average 32% lower than DYNAPs and 29% lower than
Loihi. This improvement is because 1) a $\mu$Brain core consumes lower power
than a DYNAPs and Loihi core [2], 2) contrary to DYNAPs and Loihi, the
proposed design uses different capacity (big little) $\mu$Brain cores, which
significantly reduces unused synaptic connections and improves energy
efficiency, 3) the proposed design uses parallel segmented bus interconnect,
which is more energy efficient than a traditional mesh-based NoC, which is
used in DYNAPs and Loihi, and 4) contrary to Loihi, where synaptic memory is
separated from neurons, $\mu$Brain requires very low data movement because of
the integration of synaptic memory with neurons.

###  V-B Throughput

Figure 11 plots throughput of the evaluated SDCNN applications on the proposed
design. We compare SentryOS with a previously-proposed framework SpiNeMap.
Results are normalized to SpiNeMap. We make two observations.

Figure 11: Throughput normalized to SpiNeMap.

First, throughput using SentryOS is on average 25% higher than SpiNeMap. This
improvement is because, SentryOS first compiles an SDCNN into sub-networks by
exploiting the internal architecture of big and little $\mu$Brain cores and
then uses a dataflow analysis technique to schedule sub-networks to cores
improving opportunities for pipelining and exploiting data-level parallelism.
Second, even for irregular topologies such as ResNet and DenseNet, SentryOS
results in an average 22% higher throughput than SpiNeMap.

###  V-C $\mu$Brain Design Choice: Segmented Bus Interconnect

Figure 12 plots interconnect energy and latency of the proposed many-core
design with segmented bus (SB) compared to a mesh-based network-on-chip (NoC)
interconnect. We make the following two key observations.

(a) Normalized interconnect energy. (b) Normalized latency.

Figure 12: Comparing $\mu$Brain design with segmented bus (SB) interconnect to
mesh-based network-on-chip (NoC) interconnect.

First, energy using segmented bus interconnect is on average 67% lower than
NoC. This improvement is because 1) we analyze inter-core communication based
on training data to identify the minimum number of parallel bus lanes needed
in the segmented bus interconnect, which reduces the active wire length
compared to NoC, and 2) core-to-core communications do not need to wake and
utilize the entire bus; rather, only segments connecting the communicating
cores need to be powered up. Second, latency using segmented bus is on average
18% lower than that of NoC. This is because in the proposed design,
segmentation switches are programmed once at design-time before admitting an
application. This is done by analyzing the communication profile. Since there
is no run-time routing decisions involved, latency is lower.

###  V-D $\mu$Brain Design Choice: Heterogeneous Configurations

Figure 13 plots energy of the proposed $\mu$Brain-based many-core design with
1, 2, 4, and 8 different $\mu$Brain configurations. The proposed design with a
single configuration is the conservative design of Table II. All results are
normalized to this conservative design. We make two observations.

Figure 13: Energy for different $\mu$Brain configurations.

First, with 2, 4, and 8 configurations, energy is on average 16%, 35%, and 36%
lower than the conservative design, respectively. In comparison, energy of a
fully-custom SDCNN-specific design, where cores can be of difference sizes, is
36% lower than the conservative design (see Fig. 4). Energy reduces with more
configurations due to the reduction of unused synaptic connections in each
core. Second, increasing from 4 to 8 configurations, the reduction of energy
is less than 1%. In our proposed design template, we have used only four
configurations – 1) little, type 1 $(256\times 64\times 16)$, 2) little, type
2 $(1024\times 256\times 16)$, 3) big, type 1 $(4096\times 1024\times 16)$,
and 4) big. type 2 $(16384\times 4096\times 16)$, instead of adopting a fully-
custom SDCNN-specific design. This is to make the design generic and
applicable to many different SDCNN inference applications.

##  VI Conclusion

We introduce a many-core neuromorphic platform template consisting of
asynchronous (clock-less) big little digital $\mu$Brain cores interconnected
using a segmented bus interconnect. We propose a system software framework
SentryOS, consisting of a compiler and a run-time manager, to compile spiking
deep convolutional neural network (SDCNN) to the proposed design. Using five
commonly-used SDCNN applications, we show a significant improvement in energy,
latency, and throughput.

## References

  * [1] Y. Cao _et al._ , “Spiking deep convolutional neural networks for energy-efficient object recognition,” _IJCV_ , 2015.
  * [2] J. Stuijt _et al._ , “$\mu$Brain: An event-driven and fully synthesizable architecture for spiking neural networks,” _Frontiers in Neuroscience_ , 2021.
  * [3] S. Moradi _et al._ , “A scalable multicore architecture with heterogeneous memory structures for dynamic neuromorphic asynchronous processors (DYNAPs),” _TBCAS_ , 2017.
  * [4] M. Davies _et al._ , “Loihi: A neuromorphic manycore processor with on-chip learning,” _IEEE Micro_ , 2018.
  * [5] Y. S. Yang _et al._ , “Recent trend of neuromorphic computing hardware: Intel’s neuromorphic system perspective,” in _ISOCC_ , 2020.
  * [6] S. A. Wasif _et al._ , “Energy efficient synchronous-asynchronous circuit-switched NoC,” in _MOCAST_ , 2020.
  * [7] X. Liu _et al._ , “Neu-NoC: A high-efficient interconnection network for accelerated neuromorphic systems,” in _ASP-DAC_ , 2018.
  * [8] Y. J. Yoon _et al._ , “System-level design of networks-on-chip for heterogeneous systems-on-chip,” in _NOCS_ , 2017.
  * [9] A. Balaji _et al._ , “Exploration of segmented bus as scalable global interconnect for neuromorphic computing,” in _GLSVLSI_ , 2019.
  * [10] J. Chen _et al._ , “Segmented bus design for low-power systems,” _TVLSI_ , 1999.
  * [11] S. Fusi _et al._ , “Collective behavior of networks with linear (VLSI) integrate-and-fire neurons,” _Neural Computation_ , 1999.
  * [12] H. Benmeziane _et al._ , “Hardware-aware neural architecture search: Survey and taxonomy,” in _IJCAI_ , 2021.
  * [13] K. J. Lee _et al._ , “The development of silicon for AI: Different design approaches,” _TCAS I: Regular Papers_ , 2020.
  * [14] C. D. Schuman _et al._ , “A survey of neuromorphic computing and neural networks in hardware,” _arXiv preprint arXiv:1705.06963_ , 2017.
  * [15] F. Catthoor _et al._ , “Very large-scale neuromorphic systems for biological signal processing,” in _CMOS Circuits for Biological Sensing and Processing_ , 2018.
  * [16] A. Paul _et al._ , “Design technology co-optimization for neuromorphic computing,” in _IGSC Workshops_ , 2021.
  * [17] A. Balaji _et al._ , “Mapping spiking neural networks to neuromorphic hardware,” _TVLSI_ , 2020.
  * [18] C.-K. Lin _et al._ , “Mapping spiking neural networks onto a manycore neuromorphic architecture,” in _PLDI_ , 2018.
  * [19] Y. Ji _et al._ , “NEUTRAMS: Neural network transformation and co-design under neuromorphic hardware constraints,” in _MICRO_ , 2016.
  * [20] A. Balaji _et al._ , “NeuroXplorer 1.0: An extensible framework for architectural exploration with spiking neural networks,” in _ICONS_ , 2021\.
  * [21] F. Galluppi _et al._ , “A hierachical configuration system for a massively parallel neural hardware platform,” in _CF_ , 2012.
  * [22] S. Song _et al._ , “DFSynthesizer: Dataflow-based synthesis of spiking neural networks to neuromorphic hardware,” _TECS_ , 2021.
  * [23] T. Titirsha _et al._ , “Endurance-aware mapping of spiking neural networks to neuromorphic hardware,” _TPDS_ , 2021.
  * [24] T. Titirsha _et al._ , “On the role of system software in energy management of neuromorphic computing,” in _CF_ , 2021.
  * [25] S. Song _et al._ , “Dynamic reliability management in neuromorphic computing,” _JETC_ , 2021.
  * [26] A. Amir _et al._ , “Cognitive computing programming paradigm: a corelet language for composing networks of neurosynaptic cores,” in _IJCNN_ , 2013\.
  * [27] T. Bekolay _et al._ , “Nengo: a python tool for building large-scale functional brain models,” _Frontiers in Neuroinformatics_ , 2014.
  * [28] A. Balaji _et al._ , “PyCARL: A PyNN interface for hardware-software co-simulation of spiking neural network,” in _IJCNN_ , 2020.
  * [29] A. P. Davison _et al._ , “PyNN: a common interface for neuronal network simulators,” _Frontiers in Neuroinformatics_ , 2009.
  * [30] A. Sengupta _et al._ , “Going deeper in spiking neural networks: VGG and residual architectures,” _Frontiers in Neuroscience_ , 2019.
  * [31] S. Zheng _et al._ , “Efficient scheduling of irregular network structures on CNN accelerators,” _TCAD_ , 2020.
  * [32] T. Augustine _et al._ , “Generating piecewise-regular code from irregular structures,” in _PLDI_ , 2019.
  * [33] N. Voss _et al._ , “Convolutional neural networks on dataflow engines,” in _ICCD_ , 2017.
  * [34] J. Shan _et al._ , “Exact and heuristic allocation of multi-kernel applications to multi-FPGA platforms,” in _DAC_ , 2019.
  * [35] P. Xu _et al._ , “AutoDNNchip: An automated DNN chip predictor and builder for both FPGAs and ASICs,” in _FPGA_ , 2020.
  * [36] S. Song _et al._ , “A design flow for mapping spiking neural networks to many-core neuromorphic hardware,” in _ICCAD_ , 2021.
  * [37] S. Curzel _et al._ , “Automated generation of integrated digital and spiking neuromorphic machine learning accelerators,” in _ICCAD_ , 2021.
  * [38] L. Thiele _et al._ , “Real-time calculus for scheduling hard real-time systems,” in _ISCAS_ , 2000.


# Between-Sample Relationship in Learning Tabular Data Using Graph and
Attention Networks

Shourav B. Rabbani and Manar D. Samad
Department of Computer Science
Tennessee State University
Nashville, TN, USA
<EMAIL_ADDRESS>

###### Abstract

Traditional machine learning assumes samples in tabular data to be independent
and identically distributed (i.i.d). This assumption may miss useful
information within and between sample relationships in representation
learning. This paper relaxes the i.i.d assumption to learn tabular data
representations by incorporating between-sample relationships for the first
time using graph neural networks (GNN). We investigate our hypothesis using
several GNNs and state-of-the-art (SOTA) deep attention models to learn the
between-sample relationship on ten tabular data sets by comparing them to
traditional machine learning methods. GNN methods show the best performance on
tabular data with large feature-to-sample ratios. Our results reveal that
attention-based GNN methods outperform traditional machine learning on five
data sets and SOTA deep tabular learning methods on three data sets. Between-
sample learning via GNN and deep attention methods yield the best
classification accuracy on seven of the ten data sets, suggesting that the
i.i.d assumption may not always hold for most tabular data sets.

_K_ eywords embedding clustering, tabular data, Gaussian clusters,
autoencoder, representation learning, multivariate distribution

##  1 Introduction

In a node-edge-like graph structure, edges represent the connection between
data samples denoted by nodes. Graph data structures represent individual node
samples in terms of their connected neighboring samples in bioinformatics,
software vulnerability identification, and communication networks. A graph
neural network (GNN) uses neighborhood connectivity and node feature vectors
to learn a node representation for node classification [1]. GNN assumes that
node samples in a neighborhood are related at varying degrees, subsequently
characterizing individual samples. This between-sample relationship is not
considered for data samples generally assumed to be independent. For example,
tabular data sets are collections of samples in rows with features positioned
in columns. In tabular data, individual rows are assumed to be independent and
identically distributed (i.i.d.) for machine learning purposes [2, 3]. It is
important to note that each tabular data set is unique depending on the
application domain. A bioinformatics data set can be structured in a tabular
format concealing the relationships between samples. Furthermore, attention
learning can focus on how individual samples attend to other samples to find
between sample relationship [4].

We hypothesize that traditional machine or deep learning methods can disregard
useful relationships between samples that can improve the representation of
individual samples in certain domains. In other words, GNN and attention-based
methods can be competitive for certain tabular data sets over traditional
feature-based learning with the i.i.d assumption. This paper investigates
whether GNN and attention models can learn hidden relationships between
samples in tabular data to improve the baseline classification accuracy
obtained using feature vectors alone.

The remainder of the manuscript is organized as follows. Section II provides a
theoretical background on the graph and attention networks and a literature
review on deep learning methods for tabular data. Section III describes
methods for creating an adjacency matrix for learning tabular data on graph
networks, the baseline graph and deep learning methods, experimental steps,
and evaluation. Section IV summarizes the results by comparing the performance
of different graph and attention-based methods on tabular data sets. Section V
summarizes the findings, and the conclusions are provided in Section VI.

##  2 Background

###  2.1 Graph-based learning

Let $G=(V,E)$ be a graph where V is the set of $N$ nodes, and $E$ is the set
of edges connecting two nodes. The connectivity between nodes is represented
by edges and stored in an $N\times N$ adjacency matrix (A) for graph-based
computing. The rows and columns of an adjacency matrix refer to the source and
destination nodes of a directed graph, respectively. The edges in an
undirected graph are numbered by 0 (not connected) and 1 (connected) in a
symmetric binary or unweighted adjacency matrix. Edges can measure the
association between two nodes using a value greater than zero. A static
adjacency matrix is the main input to a GNN for node representation learning.
Additionally, individual nodes can be represented by an m-dimensional feature
vector. Therefore, Graph Neural Networks (GNNs) can classify nodes by
simultaneously learning from feature vectors and the node connectivity
information presented in the adjacency matrix.

A GNN model comprises sequential graph layers where each layer aggregates node
features from its ‘first-degree’ neighbors, including the node itself, via a
message-passing mechanism. The first-degree neighbors of node $x_{3}$ are
nodes $x_{1}$ and $x_{2}$, as shown in Figure 1. In the second graph layer,
nodes $z_{1}$, $z_{2}$, $z_{3}$, and $z_{4}$ represent the embeddings
corresponding to nodes $x_{1}$, $x_{2}$, $x_{3}$, and $x_{4}$, respectively.
Here, node $z_{3}$ (embedding of node $x_{3}$) will now access the information
about its second-degree neighbor, $x_{4}$, via its first-degree neighbor,
$z_{1}$. Therefore, a GNN model with a second layer will encode information
from up to the second-degree neighbors. The sequence of graph layers thus
encodes an m-dimension input feature vector to a d-dimensional embedding. The
final graph layer maps the d-dimensional embedding to the likelihood of $K$
dimensional target labels for a supervised classification task. The embedding
update at each graph layer ($l$) is shown in Equation 1.

Figure 1: A graph layer aggregating neighboring information via message
passing. Features of the node itself (self loop) and from neighbors (arrows
point towards nodes) are aggregated to create embedding ($z_{1}$, $z_{2}$,
$z_{3}$, $z_{4}$) in a graph layer.

$H^{l+1}=\sigma(A\times H^{l}\times W^{l}).$ (1)

Here, $A\in\Re^{N\times N}$ is the adjacency matrix , $H^{l}\in\Re^{N\times
d_{l}}$, $W^{l}\in\Re^{d_{l}\times d_{l+1}}$, $\mathcal{H}^{0}$ is the input
feature matrix where $d_{0}=m$ represents the dimensionality of input
features. Also, $\sigma$ is the element-wise non-linear activation function,
$l$ is the graph layer index where $l=0,1,2,...,L-1$), and $L$ is the number
of graph layers. The $l$-th graph layer maps $d_{l}$-dimensional embedding to
$d_{l+1}$ dimensional embedding via message passing. To initiate the message
passing between nodes, the diagonals of the adjacency matrix are set to $1$
representing self-connections. The presence of many neighbors (high-degree
nodes) or large edge weights can inflate the magnitude of the node embedding
following the aggregation, which requires neighborhood normalization to avoid
numerical instability. Equation 2 shows how the embedding of node $p$ as
$h_{p}$ is obtained by aggregating embeddings ($h_{q}$) of $q$ neighboring
nodes. The neighborhood embeddings are aggregated and normalized by the total
number of neighboring nodes ($N_{p}$). Other normalization methods include
symmetric normalization proposed in graph convolutional network (GCN) [5].

$h_{p}=\frac{\sum_{q\in\mathcal{N}_{p}}h_{q}}{\lvert\mathcal{N}_{p}\rvert}.$
(2)

Since GNN performs message passing between neighboring nodes, splitting the
graph into train and test subgraphs will prevent the test node information
from leaking into the trained model. This is known as the inductive learning
of GNN. Conversely, the entire graph can be kept intact in transductive
learning, where train and test nodes participate in message passing and
embedding learning. However, the test labels are not used during the
supervised training process. While transductive learning can yield better test
scores because of joint train and test node learning, inductive learning can
provide more unbiased and generalizable outcomes due to the strict separation
of train and test nodes.

###  2.2 Attention-based learning

Edges in graphs are static and precomputed representations of connectivity
between nodes. However, this connectivity may not represent the optimal
relationship between nodes in the context of a learning objective. An
attention mechanism can dynamically learn an attention score between a pair of
samples while optimizing the learning objective. Therefore, attention-based
learning weeds out bad neighbors and focuses on the relevant parts of data to
improve representation learning. Velikovic et al. introduce an attention
mechanism in their graph attention model (GAT) that focuses on relevant
neighboring nodes for better message passing [6]. Here, the attention
mechanism dynamically updates the edge weights between relevant neighbors in
each graph layer to emphasize their contribution to message passing. Instead
of an adjacency matrix with static edge weights, the embedding for a single
node is updated using dynamic attention scores, as shown in Equation 3.

$\vec{h_{p}^{\prime}}=\sigma\left({\sum_{q\in\mathcal{N}_{p}}\alpha_{pq}W\vec{h_{q}}}\right)$
(3)

For a given node $p$, the set of all neighboring nodes is $\mathcal{N}_{p}$,
and $h_{q}$ is the d-dimensional embedding of the neighboring node $q$. Here,
$\alpha_{pq}$ is the attention score iteratively updated to measure the
relevance between nodes $p$ and $q$. $W\in\Re^{d^{\prime}\times d}$ is the
trainable weight matrix and $\sigma$ is the non-linear activation function.
The attention-based GAT method uses unweighted or binary edge weights, whereas
the GCN method uses weighted graphs. Similarly, GAT-based autoencoder (GATE)
[7] converts weighted graphs to unweighted graphs by replacing non-zero edge
weights with 1 so that the model can learn attention scores from weak and
strong edges regardless.

###  2.3 Tabular data

Tabular data are structured in rows and columns of a matrix X $\in\Re^{n\times
d}$ with $n$ samples and $d$ dimensional heterogeneous feature vector. Each
sample ($X_{i}$) is represented in a feature vector form,
$X_{i}=\\{x_{1},x_{2},…,x_{d}\\}$, where i = $\\{1,2,…,n\\}$. Therefore,
tabular data are modeled using multivariate distributions
$P(x_{1},x_{2},…,x_{d})$ of heterogeneous variables. Unlike graph data, the
samples in tabular data are assumed independent without any association or
connection between rows. On the other hand, graph data may only have edge
connection information between nodes without any feature vectors. However,
tabular data sets often contain limited sample sizes and feature
dimensionality, which may not take the full benefits of deep learning. Recent
literature identifies heterogeneous tabular data as the last unconquered
castle for deep learning [8], where traditional machine learning often
performs better than deep learning methods [9, 8, 10]. Therefore, a robust
deep representation of tabular data remains an open problem.

###  2.4 Literature review

In machine learning, samples in tabular data are assumed to be independent and
identically distributed (i.i.d). The i.i.d assumption ignores any dependencies
between samples. The identically distributed assumption considers similar data
distribution to generalize the trained model on unseen test samples. The i.i.d
assumption can be relaxed to learn useful associations between samples
leveraging graph- and attention-based learning mechanisms. The relaxation of
i.i.d assumption can be achieved by learning embedding from neighboring
samples, which alters identical distribution assumption by aggregating
neighborhood embeddings in representation learning [2]. We summarize relevant
deep learning methods for tabular data below.

Because deep learning methods still fall short of traditional machine learning
on Tabular data [10, 11], recent studies propose new methods to improve the
deep representation of tabular data. For example, attention-based methods can
leverage information from other samples to improve embeddings for downstream
classification tasks [12, 13]. A transformer-based attention model, feature
tokenizer (FT) Transformer outperforms baseline methods on seven of eleven
tabular data sets [14]. Non-Parametric Transformers, proposed for tabular
data, outperform baseline models on eight classification data sets [15].
TabNet is an interpretable model that uses attention to select important
features from tabular data [16] and outperforms other baseline methods on four
out of five data sets and three out of six synthetic data sets.

In contrast, graph neural networks are not used beyond graph data sets.
Several graph-based learning methods, including graph attention network (GAT)
[6] and graph attention autoencoder (GATE) [7], have incorporated attention
mechanisms to demonstrate superior performance over standard graph-based
methods exclusively on graph data sets [5]. Because of the i.i.d assumption,
tabular data are rarely modeled as graph data to investigate the effectiveness
of graph-based learning. Few studies have proposed graph-based models for
tabular data, including bipartite graphs (GRAPE) [17] and Propagation to
Enhance Tabular data prediction (PET) [18], and multiplex graph (TabGNN) [19].
In the GRAPE approach, tabular data ($X_{m\times n}$) is modeled as a
bipartite graph $G=(V_{s},V_{f},E)$ where $V_{s}$ and $V_{f}$ denote sets of
sample and feature nodes and E represents all undirected edges connecting
sample and feature nodes. An edge (i, j) weight represents the j-th feature
value for the i-th sample. In contrast, the PET method uses unweighted edges
with feature nodes representing the feature values. However, these approaches
do not model the between-sample relationships. Lastly, TabGNN connects sample
nodes that share the same values for the categorical variables. Here, multiple
categorical features result in multiple types of edges. However, this method
leaves out numerical variables in modeling graph structures of tabular data.
To the best of our knowledge, graph neural networks have not been considered
in learning between sample relationships in tabular data with numerical
variables or compared against traditional machine learning or attention-based
learning methods.

##  3 Methods

This section provides methods to create the adjacency matrix for graph based
learning of tabular data. The section summarizes the baseline and evaluation
methods for comparing several deep learning alternatives with superior machine
learning methods on tabular data.

###  3.1 Modeling tabular data in graphs

A graph data set includes edge weights and adjacency matrix for graph-based
learning. Therefore, an adjacency matrix is formed using tabular data samples
to train GNN. There is no single or best approach to forming an adjacency
matrix with edge weights. The adjacency matrix for tabular data is first
computed by measuring 1) Euclidean distances and 2) cosine similarity between
each pair of sample feature vectors. The distance or similarity metric scale
is normalized column-wise using minimum and maximum values. The scaled
Euclidean distance values are transformed into similarity by taking their
absolute difference from unity.

Finally, a threshold is used on the similarity value to drop weak and spurious
connections in the graph. For the unweighted or binary graph scenario, a
similarity score above or equal to the threshold will be replaced by one;
otherwise by zero. In contrast, all similarity values are preserved in the
adjacency matrix for a weighted graph after pruning the below-threshold values
to zero. We use inductive graph learning to ensure the separation between
train and test nodes during supervised classification.

###  3.2 Baseline methods

We compare tabular data classification performances of nine baseline methods:
three traditional machine learning models, three recently proposed attention-
based deep methods for tabular data, and three graph-based neural networks.
Machine learning methods include logistic regression (LR), gradient boosting
tree (GBT), and multi-layer perceptron (MLP). The attention-based tabular data
learning methods include FT-Transformer (FTT) [14], TabNet [16], and Non-
Parametric Transform (NPT) [15]. These baseline methods are compared against
three graph neural network methods: Graph Convolution Network (GCN) [5], Graph
Attention Network (GAT) [6], and Graph Attention Autoencoder (GATE) [7].

####  3.2.1 TabNet

TabNet is motivated by ensemble decision trees to create sequential blocks
termed as decision steps [16]. Each block uses an attention mechanism to
select a subset of features for learning the embedding and passes the feature
subset to the next block. The next block selects a subset of the feature
subset and creates another embedding. All embeddings from the decision steps
are aggregated in the final embedding for downstream classification. In each
decision step, the sparsity of feature selection is calculated in the form of
entropy. The model is trained until the sum of prediction loss and sparse
entropy converges.

####  3.2.2 FT-Transformer

Feature Tokenizer + Transformer (FT-Transformer) is a deep tabular model that
uses multiheaded self-attention (MHSA) (attention between features) to refine
feature embeddings as tokenized features [14]. A linear layer maps the final
embedding to target classification labels. The model is trained in a
supervised manner until the convergence of the classification loss.

####  3.2.3 NPT

Non-parametric transformer (NPT) predicts target labels by learning to attend
between samples and features [15]. This differs from traditional models, where
the target label is a function of the feature vector alone. Instead, the NPT
model predicts missing entries masked stochastically in the training data. NPT
uses stacked MHSA layers to attend between sample and feature representations.
The model is trained until the sum of missing value reconstruction loss and
target loss converges.

####  3.2.4 GCN

Graph convolutional network (GCN) is built using a sequence of graph layers
[5] that perform the message passing mechanism, as discussed in Section 2.
However, instead of a normalized adjacency matrix, it uses a normalized
Laplacian matrix, also known as symmetric neighborhood normalization. Graph
convolution networks differ from convolutional neural networks that work on
image data. While CNN layers aggregate pixel information from neighbors
arranged in a regular grid, graph convolution layers aggregate irregular
neighboring node features.

####  3.2.5 GAT

Graph Attention Network (GAT) is based on GCN model but learns attention
scores between node pairs using features [6]. The attention scores are used
instead of edge weights to aggregate the neighborhood embeddings as individual
node embedding. The authors draw motivation from MHSA and use multi-head to
compute k parallel attention scores between connected nodes. The final layer
maps the final embedding to classification targets. The model is trained until
the supervised loss converges. The authors report superior performance of
their GAT method to the GCN method in both transductive and inductive learning
on three graph data sets.

####  3.2.6 GATE

Graph Attention Autoencoder (GATE) is an autoencoder model built using GAT
graph layers in its encoder and decoder stacks [7]. An unweighted (binary)
adjacency matrix is used to initiate the computing of attention scores. The
model is trained to minimize the node feature reconstruction loss and maximize
the similarity between neighboring node representations. The embedding from
the last encoder layer is used for downstream classification tasks. The
authors report superior performance of the GATE method compared to the GAT
method on the same three benchmark graph data sets.

###  3.3 Data sets

A summary of the ten tabular data sets used in this paper is provided in Table
1. These data sets are representative of distinct application domains. Except
for the Olive
dataset111https://search.r-project.org/CRAN/refmans/FlexDir/html/oliveoil.html,
all other data sets are sourced from the UCI machine learning repository. Only
numeric features are considered for this study. Except for the satellite and
malware data sets, others have less than one thousand samples. The data
dimensionality ranges from five to 753. Only three data sets: Parkinson’s,
breast cancer, and malware, have binary classification labels.

Data set | Samples | Features | Classes | F-S ratio
---|---|---|---|---
Wine | 178 | 13 | 3 | 0.073
Olive | 572 | 10 | 3 | 0.017
Vehicle | 846 | 18 | 4 | 0.021
Satellite | 6435 | 36 | 6 | 0.006
Parkinson | 756 | 753 | 2 | 0.996
Breast cancer | 569 | 30 | 2 | 0.053
Dermatology | 358 | 34 | 6 | 0.095
Mice | 552 | 77 | 8 | 0.140
Malware | 4464 | 241 | 2 | 0.054
Synthetic | 200 | 5 | 4 | 0.025

Table 1: Summary of tabular data sets used to evaluate graph and attention-
based learning methods in this paper. F-S ratio = feature-sample ratio.

###  3.4 Model evaluation

The classification performance is reported using the average F1 weighted score
in a 10-fold cross-validation scheme. In 10-fold cross-validation, a test fold
(10%) is set apart, and the remaining seven and two folds are used for
training and validation, respectively. A predefined random state seed for data
split and cross-validation reproduces the same train and test samples for all
experiments. We use the default hyperparameter setting for attention and
graph-based methods. However, hyperparameters are tuned for the traditional
machine learning methods. For graph-based learning, we tune the construction
of the adjacency matrix by varying the cut-off value between 0 and 1 at 0.1
intervals to determine the weak connections (See Section 3.1) and the
selection between weighted and unweighted graph.

|  Breast cancer |  Derma-tology |  Malware |  Mice |  Olive |  Parkin-son’s |  Satellite |  Synthetic |  Vehicle |  Wine
---|---|---|---|---|---|---|---|---|---|---
LR |  0.970 $(0.02)$ |  0.966 $(0.03)$ |  0.985 $(0.00)$ |  0.998 $(0.01)$ |  0.854 $(0.04)$ |  0.830 $(0.03)$ |  0.828 $(0.01)$ |  0.970 $(0.03)$ |  0.789 $(0.03)$ |  0.972 $(0.03)$
GBT |  0.949 $(0.03)$ |  0.969 $(0.02)$ |  0.991 $(0.00)$ |  0.961 $(0.03)$ |  0.868 $(0.05)$ |  0.848 $(0.07)$ |  0.909 $(0.01)$ |  0.980 $(0.02)$ |  0.734 $(0.02)$ |  0.937 $(0.07)$
MLP |  0.972 $(0.02)$ |  0.557 $(0.11)$ |  0.993 $(0.00)$ |  0.925 $(0.09)$ |  0.879 $(0.05)$ |  0.846 $(0.05)$ |  0.736 $(0.06)$ |  0.659 $(0.01)$ |  0.746 $(0.14)$ |  0.983 $(0.03)$
FTT |  0.970 $(0.02)$ |  0.960 $(0.03)^{\dagger}$ |  0.987 $(0.00)^{\dagger}$ |  1.000 $(0.00)^{\diamond\dagger}$ |  0.844 $(0.06)^{\dagger}$ |  0.838 $(0.07)$ |  0.911 $(0.01)^{\star\dagger}$ |  1.000 $(0.00)^{\star\diamond\dagger}$ |  0.811 $(0.03)^{\diamond}$ |  0.978 $(0.04)$
TabNet |  0.942 $(0.02)^{\star\dagger}$ |  0.941 $(0.03)^{\diamond\dagger}$ |  0.978 $(0.01)^{\star\diamond\dagger}$ |  0.876 $(0.04)^{\star\diamond}$ |  0.845 $(0.06)^{\dagger}$ |  0.818 $(0.05)$ |  0.879 $(0.01)^{\star\diamond\dagger}$ |  0.980 $(0.03)^{\dagger}$ |  0.698 $(0.04)^{\star\diamond}$ |  0.931 $(0.03)^{\star\dagger}$
NPT |  0.972 $(0.02)^{\diamond}$ |  0.959 $(0.02)^{\dagger}$ |  - |  - |  0.858 $(0.06)$ |  - |  0.915 $(0.01)^{\star\diamond\dagger}$ |  1.000 $(0.00)^{\star\diamond\dagger}$ |  0.793 $(0.05)^{\diamond}$ |  0.955 $(0.04)$
GATE_CB |  0.968 $(0.02)$ |  0.847 $(0.06)^{\star\diamond\dagger}$ |  0.987 $(0.01)^{\dagger}$ |  1.000 $(0.00)^{\diamond\dagger}$ |  0.816 $(0.05)^{\star\diamond\dagger}$ |  0.793 $(0.04)^{\star\diamond\dagger}$ |  0.784 $(0.01)^{\star\diamond}$ |  0.870 $(0.16)^{\diamond\dagger}$ |  0.639 $(0.06)^{\star\diamond}$ |  0.954 $(0.04)$
GATE_EB |  0.956 $(0.02)$ |  0.877 $(0.06)^{\star\diamond\dagger}$ |  0.985 $(0.01)^{\dagger}$ |  0.497 $(0.06)^{\star\diamond\dagger}$ |  0.622 $(0.14)^{\star\diamond\dagger}$ |  0.787 $(0.04)^{\star\diamond\dagger}$ |  0.783 $(0.01)^{\star\diamond}$ |  0.970 $(0.06)^{\dagger}$ |  0.740 $(0.05)^{\star}$ |  0.977 $(0.03)$
GAT_CB |  0.964 $(0.02)$ |  0.969 $(0.04)^{\dagger}$ |  0.983 $(0.01)^{\diamond\dagger}$ |  0.972 $(0.03)^{\star}$ |  0.817 $(0.04)^{\star\diamond\dagger}$ |  0.857 $(0.04)^{\star}$ |  0.642 $(0.16)^{\star\diamond}$ |  0.978 $(0.05)^{\dagger}$ |  0.594 $(0.06)^{\star\diamond\dagger}$ |  0.965 $(0.05)$
GAT_EB |  0.958 $(0.03)$ |  0.962 $(0.03)^{\dagger}$ |  0.982 $(0.00)^{\diamond\dagger}$ |  0.985 $(0.02)$ |  0.820 $(0.04)^{\star\diamond\dagger}$ |  0.844 $(0.05)$ |  0.841 $(0.01)^{\star\diamond\dagger}$ |  0.995 $(0.02)^{\dagger}$ |  0.608 $(0.05)^{\star\diamond\dagger}$ |  0.961 $(0.06)$
GCN_C |  0.956 $(0.03)$ |  0.960 $(0.03)^{\dagger}$ |  0.985 $(0.01)^{\dagger}$ |  0.887 $(0.06)^{\star\diamond}$ |  0.725 $(0.09)^{\star\diamond\dagger}$ |  0.860 $(0.04)^{\star}$ |  0.717 $(0.03)^{\star\diamond}$ |  0.959 $(0.11)^{\dagger}$ |  0.543 $(0.05)^{\star\diamond\dagger}$ |  0.931 $(0.10)$
GCN_CB |  0.959 $(0.03)$ |  0.951 $(0.03)^{\dagger}$ |  0.985 $(0.01)^{\dagger}$ |  0.881 $(0.06)^{\star\diamond}$ |  0.738 $(0.06)^{\star\diamond\dagger}$ |  0.845 $(0.03)$ |  0.684 $(0.13)^{\star\diamond}$ |  0.957 $(0.10)^{\dagger}$ |  0.538 $(0.05)^{\star\diamond\dagger}$ |  0.960 $(0.06)$
GCN_E |  0.954 $(0.03)$ |  0.953 $(0.03)^{\dagger}$ |  0.979 $(0.01)^{\star\diamond\dagger}$ |  0.904 $(0.05)^{\star\diamond}$ |  0.746 $(0.08)^{\star\diamond\dagger}$ |  0.852 $(0.03)$ |  0.782 $(0.05)^{\star\diamond\dagger}$ |  0.990 $(0.02)^{\dagger}$ |  0.589 $(0.06)^{\star\diamond\dagger}$ |  0.943 $(0.07)^{\dagger}$
GCN_EB |  0.970 $(0.02)$ |  0.936 $(0.04)^{\star\diamond\dagger}$ |  0.980 $(0.01)^{\diamond\dagger}$ |  0.897 $(0.06)^{\star\diamond}$ |  0.787 $(0.07)^{\star\diamond\dagger}$ |  0.835 $(0.05)$ |  0.808 $(0.03)^{\diamond\dagger}$ |  0.975 $(0.03)^{\dagger}$ |  0.613 $(0.05)^{\star\diamond\dagger}$ |  0.966 $(0.08)$

Table 2: Average F1 scores. For graph-based models, the adjacency matrix is
constructed with C = cosine or E= Euclidean similarity. B denotes an
unweighted adjacency matrix. Otherwise, adjacency matrices are weighted.
$\star$, $\diamond$, and $\dagger$ indicate that the average F1 score is
significantly different ($p<0.05$) compared to LR, GBT, and MLP, respectively.

Our MLP with three hidden layers (256-128-64) is trained with a learning rate
of 0.001 and a batch size of 64. For GBT, the hyperparameters are the number
of estimators (50, 80, or 110) and maximum tree depth (2, 5, 10, 15). The LR
classifier is tuned for L1 and L2 regularization and the inverse of the
regularization strength parameter (0.01, 0.05, 0.5, 0.8, 1, 5). However, we
train on the nine data folds for unsupervised learning (e.g., the GATE method)
and report the classification accuracy on the left-out test fold. All neural
network-based models are trained for 1000 epochs with patience 100. The
validation and training losses are used for early stopping in supervised and
unsupervised training, respectively. Scores from individual test folds are
used to perform a Wilcoxon-ranked test to compare the performance of the
methods statistically.

##  4 Results

The performance of each model on different tabular data sets is presented in
Table 2. The statistical significance in classification performance ($p\leq
0.05$) compared to traditional models (LR, GBT, and MLP) is also indicated in
the Table.

###  4.1 Performance of Graph Neural Networks

GATE and GAT methods with unweighted cosine similarity metrics perform better
than GBT on the breast cancer data set. The same GAT method ties with GBT as
the best classifier on the dermatology data set. GATE with an unweighted
cosine metric is on par with the LR classifier for the malware data set. The
same GATE method is statistically superior to GBT and MLP as the best-
performing method on the mice data set. However, graph-based learning is
statistically inferior to traditional machine learning methods (GBT, LR, and
MLP) on the olive data set. The GAT method is slightly superior to GBT and LR
methods on the Parkinson’s data set, although the improvement is not
statistically significant. Overall, the GCN method with weighted cosine
similarity outperforms all other methods on the Parkinson’s data set. However,
graph-based methods are statistically inferior to machine learning methods on
the satellite data set. The GAT method is on par with traditional classifier
models (GBT, LR) for the synthetic data set but statistically superior to MLP.
The GATE method with an unweighted Euclidean distance metric is better than LR
and GBT.

| Breast cancer | Dermatology | Malware | Mice | Olive | Parkinson’s | Satellite | Synthetic | Vehicle | Wine
---|---|---|---|---|---|---|---|---|---|---
LR | 3 | 3 | 5 | 3 | 4 | 6 | 6 | 7 | 3 | 4
GBT | 8 | 1 | 2 | 5 | 2 | 3 | 3 | 5 | 6 | 8
MLP | 1 | 9 | 1 | 6 | 1 | 4 | 9 | 9 | 4 | 1
FTT | 3 | 4 | 3 | 1 | 6 | 5 | 2 | 1 | 1 | 2
TabNet | 9 | 7 | 8 | 8 | 5 | 7 | 4 | 5 | 7 | 9
NPT | 1 | 6 | - | - | 3 | - | 1 | 1 | 2 | 7
GATE | 6 | 8 | 3 | 1 | 8 | 8 | 8 | 7 | 5 | 3
GAT | 7 | 1 | 7 | 4 | 7 | 2 | 5 | 3 | 9 | 6
GCN | 3 | 4 | 5 | 7 | 9 | 1 | 7 | 4 | 8 | 5

Table 3: Rank ordering of traditional machine learning (LR, GBT, MLP), deep
learning (FTT, TabNet, NPT), and graph neural networks (GATE, GAT, GCN) in the
classification performance across ten tabular data sets. In case of ties, the
next rank is skipped.

###  4.2 Performance of deep attention models

TabNet is a deep learning method specifically proposed for tabular data.
However, the GATE method outperforms the TabNet method on five of ten data
sets. Notably, The GAT method outperforms TabNet on seven of ten data sets.
The NPT method performs the best on the breast cancer, satellite, and
synthetic data sets. However, it fails to converge on the malware, mice, and
Parkinson’s data sets. These are the three data sets with the highest data
dimensionality. Yet the NPT method outperforms GAT on five of ten data sets.
Overall, the transformer-based FTT method is superior to graph-based methods,
with some exceptions. For instance, FTT is on par with graph-based learning on
the breast cancer, malware, mice, and wine data sets but inferior on the
dermatology and Parkinson’s data sets.

###  4.3 Best performing methods

Table 3 summarizes the rank ordering of different learning methods for the
tabular data sets. The LR classifier does not perform the best on any data
sets but challenges other deep and attention-based learning on several data
sets. GBT appears as one of the methods for the dermatology data set. MLP is
the best method on three of the ten data sets: malware, olive, and wine. Among
the deep tabular methods, FTT achieves the best F1 scores on three of ten data
sets: mice, synthetic, and vehicle. TabNet does not achieve the best F1 score
on any data set. NPT achieves the best F1 scores on the breast cancer (tied
with MLP), satellite, and synthetic (tied with FTT) data sets. GAT with
unweighted cosine similarity is the best on the dermatology data set. GATE
with unweighted cosine similarity is the best method for the mice data set.
Finally, the GCN with weighted cosine similarity metric performs the best on
the Parkinson’s data set.

##  5 Discussion of results

This paper investigates the effectiveness of learning tabular data
representation by leveraging between-sample relationships, which is generally
ignored due to the i.i.d assumption in machine learning. Traditional machine
learning methods are compared against attention and graph networks modeling
the between-sample relationship. The findings of the paper are summarized as
follows. First, no single method performs the best on all tabular data sets.
This confirms the knowledge that the performance of a method depends on the
data or domain problem. Second, graph neural networks, such as GAT, GATE, and
GCN methods with cosine similarity metrics, have shown promising results,
outperforming state-of-the-art deep tabular data learning methods on several
data sets. This reveals that some domain data sets may have useful
relationships between samples defying the i.i.d assumption. Third, attention
between samples (NPT method) and between-sample relationships in GNN methods
secure the best ranks on six of the ten tabular data sets. This observation
suggests that the independent sample assumption should be relaxed to improve
tabular data representation.

The deep learning methods proposed for tabular data (FTT, NPT, TabNet) are
among the best methods for the data sets with the largest sample sizes
(vehicle, satellite, and malware). Notably, unlike our experiments, these
methods are evaluated by selectively using data sets with a large sample size.
For example, the TabNet article selectively uses tabular data sets with sample
sizes ranging from 41 thousand to 11 million. Therefore, the performance of
similar methods may be limited to data sets with large sample sizes.
Furthermore, the deep tabular methods create larger dimensional embeddings
than the input dimension. However, our experiments find that NPT is very
memory expensive and unstable on data sets with high dimensional features
(mice (77), malware (241), and Parkinson’s (753)). Notably, the data
dimensionality in Tabnet, FTT, and NPT range from 11 to 54, 8 to 2000 , and 10
to 54, respectively. This explains that FTT is effective for a wide range of
data dimensionality, which our results support.

One interesting trend in the performance of GNN methods may be explained using
the number of features to the number of samples ratio (F-S ratio) for each
data set, as presented in Table 1. For the data sets with the highest ratios
(Parkinson’s: 0.996, Dermatology: 0.095, and Mice: 0.14), one of the GNN
methods ranks as the best. In contrast, GNN methods perform poorly on data
sets with the lowest ratios (Olive: 0.017, Vehicle: 0.02, Satellite: 0.006).
Therefore, GNN methods can be chosen for tabular data sets with limited
samples but large dimensionality. A general trend suggests that a binary
adjacency matrix built from cosine similarity is superior to other methods in
learning tabular data with GNN.

##  6 Conclusions

This paper conducts one of the first studies to investigate the effectiveness
of learning between-sample relationships in tabular data conventionally
considered independent. Graph and attention-based models are used to learn
between-sample relationships for downstream classification tasks, unlike
traditional machine learning methods. Our findings suggest that between-sample
relationships via graph edges or attention modeling can outperform traditional
machine-learning methods on seven of the ten data sets. This finding is
important because it shows an alternative approach to improving deep learning
of tabular data sets when traditional machine learning outperforms deep
learning methods.

## Acknowledgments

Research reported in this publication was supported in part by the National
Library of Medicine of the National Institutes of Health under Award Number
R15LM013569 and was partially sponsored by the Air Force Office of Scientific
Research under Grant Number W911NF-23-1-0170. The content is solely the
responsibility of the authors and does not necessarily represent the official
views of the National Institutes of Health and should not be interpreted as
representing the official policies, either expressed or implied, of the Army
Research Office or the U.S. Government.

## References

  * [1] Daokun Zhang, Jie Yin, Xingquan Zhu, and Chengqi Zhang.  Network representation learning: A survey.  IEEE Transactions on Big Data, 6(1):3–28, 2020.
  * [2] William L Hamilton.  Graph representation learning.  Synthesis Lectures on Artifical Intelligence and Machine Learning, 14(3):1–159, 2020.
  * [3] Volodya Vovk, Alexander Gammerman, and Craig Saunders.  Machine-learning applications of algorithmic randomness.  In Proceedings of the Sixteenth International Conference on Machine Learning, ICML ’99, page 444–453, San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc.
  * [4] Gowthami Somepalli, Micah Goldblum, Avi Schwarzschild, C. Bayan Bruss, and Tom Goldstein.  Saint: Improved neural networks for tabular data via row attention and contrastive pre-training.  6 2022.
  * [5] Thomas N. Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  5th International Conference on Learning Representations, ICLR 2017 - Conference Track Proceedings, 9 2016.
  * [6] Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio.  Graph attention networks.  International Conference on Learning Representations, 10 2018.
  * [7] Amin Salehi and Hasan Davulcu.  Graph attention auto-encoders.  volume 2020-Novem, pages 989–996. IEEE, 11 2020.
  * [8] Arlind Kadra, Marius Lindauer, Frank Hutter, and Josif Grabocka.  Well-tuned simple nets excel on tabular datasets.  Advances in neural information processing systems, 34:23928–23941, 2021.
  * [9] Sakib Abrar and Manar D Samad.  Perturbation of deep autoencoder weights for model compression and classification of tabular data.  Neural Networks, 156:160–169, 2022.
  * [10] Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci.  Deep neural networks and tabular data: A survey.  IEEE Transactions on Neural Networks and Learning Systems, 2022\.
  * [11] Sakib Abrar and Manar D Samad.  Effectiveness of deep image embedding clustering methods on tabular data.  arXiv preprint arXiv:2212.14111, 2022.
  * [12] Zhaoyang Niu, Guoqiang Zhong, and Hui Yu.  A review on the attention mechanism of deep learning.  Neurocomputing, 452:48–62, 9 2021.
  * [13] Sneha Chaudhari, Varun Mithal, Gungor Polatkan, and Rohan Ramanath.  An attentive survey of attention models.  ACM Transactions on Intelligent Systems and Technology, 12, 10 2021\.
  * [14] Yury Gorishniy, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko.  Revisiting deep learning models for tabular data.  Advances in Neural Information Processing Systems, 23:18932–18943, 2021.
  * [15] Jannik Kossen, Neil Band, Clare Lyle, Aidan N. Gomez, Tom Rainforth, and Yarin Gal.  Self-attention between datapoints: Going beyond individual input-output pairs in deep learning.  Advances in Neural Information Processing Systems, 34:28742–28756, 6 2021.
  * [16] Sercan O. Arik and Tomas Pfister.  Tabnet: Attentive interpretable tabular learning.  pages 6679–6687, 8 2021.
  * [17] Jiaxuan You, Xiaobai Ma, Daisy Yi Ding, Mykel Kochenderfer, and Jure Leskovec.  Handling missing data with graph representation learning.  In Advances in Neural Information Processing Systems, volume 2020-Decem, pages 19075–19087, 2020.
  * [18] Kounianhua Du, Weinan Zhang, Ruiwen Zhou, Yangkun Wang, Xilong Zhao, Jiarui Jin, Quan Gan, Zheng Zhang, and David Wipf.  Learning Enhanced Representations for Tabular Data via Neighborhood Propagation.  In Advances in Neural Information Processing Systems, oct 2022.
  * [19] Xiawei Guo, Yuhan Quan, Huan Zhao, Quanming Yao, Yong Li, and Weiwei Tu.  TabGNN: Multiplex Graph Neural Network for Tabular Data Prediction.  aug 2021.