# Analog Neural Computing with Super-resolution Memristor Crossbars

Alex James, and Leon Chua

###### Abstract

Memristor crossbar arrays are used in a wide range of in-memory and
neuromorphic computing applications. However, memristor devices suffer from
non-idealities that result in the variability of conductive states, making
programming them to a desired analog conductance value extremely difficult as
the device ages. In theory, memristors can be a nonlinear programmable analog
resistor with memory properties that can take infinite resistive states. In
practice, such memristors are hard to make, and in a crossbar, it is confined
to a limited set of stable conductance values. The number of conductance
levels available for a node in the crossbar is defined as the crossbar’s
resolution. This paper presents a technique to improve the resolution by
building a super-resolution memristor crossbar with nodes having multiple
memristors to generate $r$-simplicial sequence of unique conductance values.
The wider the range and number of conductance values, the higher the
crossbar’s resolution. This is particularly useful in building analog neural
network (ANN) layers, which are proven to be one of the go-to approaches for
forming a neural network layer in implementing neuromorphic computations.

###### Index Terms:

Crossbar, analog neural network, multiply and accumulate, analog neural
accelerator

##  I Introduction

In the design of neural network accelerators, multiply and accumulate (MAC)[1,
2, 3] block emulates the weighted summation of inputs in a neural network
layer. Every layer of the neural network requires the MAC operation. The
neural network accelerators [4, 5, 6, 7, 8] are used in the edge AI computing
domain to make sensing smarter and data processing energy efficient. The
offloading of neural network computing from high-performance computers to edge
AI processors improves energy efficiency, and reduces costs, and bandwidth
requirements.

The MAC block can be implemented with digital or analog circuits. In a
digital-only approach, the inputs, and weights are stored in a binary form.
Such a system is most suitable if using binary memory arrays, such as SRAM or
DRAM through in-memory computing [9, 10, 11]. In an analog approach, the MAC
operation uses weights and/or inputs in analog forms, such as when
implementing using a floating gate or ReRAM array[12]. The most popular
approach to building MAC in the analog domain is the crossbar architecture[13,
14].

In a crossbar architecture, the crossbar nodes can be accessed by row or
column lines. The rows are used to apply the input signals, while the columns
are used to read the output signals. The crossbar nodes of our interest have a
programmable conductance. When a voltage is applied to the crossbar node, the
column’s output current represents the weighted summation of inputs, or rather
the MAC operation.

A popular conductance device for building a crossbar node is the memristor[15,
16]. Several devices claim to have similar properties as memristors[17, 18,
19, 20]. The neural network applications of memristors in the crossbar require
the mapping of neural network weights to the crossbar nodes’ conductance
values. This mapping of weight is only possible on a single memristor having
infinite conductance levels. Because memristors with precise and stable
infinite conductance levels are currently not practically feasible, encoding
schemes need to be applied.

One approach is to encode the weights as a binary sequence or to discretize
the weights to fit the available memristor conductance levels [21, 22]. This
limitation of weight resolutions determines the overall design of the MAC
implementation, and thereby the neural network performance. Subsequent to the
resolution limitation, the memristor is also sensitive to process variations
and device-to-device variabilities, which leads to conductance variability
[23, 24, 25, 26, 27]. The endurance and aging of the device play an important
role in the reliability of the crossbar, and often make the circuit designers
work difficult [28].

Connecting memristors in parallel using PCM devices [29, 30] increases the
reliability and fault tolerance of synapses. Likewise, ReRAM based parallel
memristors were used in [31] to reduce the impact of stochasticity and
nonlinearity. Whereas, in this paper, we present the theory of combining the
memristors within a crossbar without modifying the widely used crossbar
structures as a fundamental technique to generalise crossbar for any precision
multiply and accumulate operation. A practical approach to improve the
conductance resolution of crossbar arrays is proposed. The main contributions
of this work include (1) a super-resolution crossbar design, (2) theory of
estimating the conductance levels and (3) implication on the practical design
of analog neural networks.

This paper is organized into the following sections: Section II gives a
background on crossbars that inspire analog computing, Section III provides
the details of the proposed design approach and supporting theory, Section IV
reports the results and related discussion. Finally, Section V provides the
summary of the paper.

##  II Background

Figure 1: The memristor crossbar with $v_{1},v_{2}$ as inputs, conductance
$g_{i,j}$ of memristors as weights and columns current $i_{1},i_{2},i_{3}$ as
outputs. Here, the node shown is formed with a transistor switch and memristor
(1T1M).

The crossbar arrangement of memristors in 1T1M configuration is shown in Fig.
1. The transistor switch enables or disables the column-wise memristor nodes.
In this example, the inputs ($v_{1},v_{2}$) are applied across the rows of the
crossbar, and outputs ($i_{1},i_{2},i_{3}$) measured at the columns of the
crossbar. Each memristor has a conductance $g_{ij}$, where $i,j$ are the
coordinates of the crossbar nodes. The output currents indicate the weighted
summation of inputs. The output currents from the $n$th neuron (column of
crossbar) can be represented as $i_{n}=\sum_{k=1}^{N}v_{k}g_{k,n}$, with $N$
number of inputs (number of rows in the crossbar), representing a dot-product
or MAC operation or vector-multiplication (VMM) [32, 33, 34, 35, 36, 37, 38,
39, 40]. In a neural network implementation, this weighted summation of inputs
represents a neuron representation without activation function. In the
crossbar, bias can be included by adding additional input line. The memristor
programming and line selection is done using a selector switch in series with
the memristor as shown in Fig. 1. The selector switch enables the memristor
node by applying a gate voltage sufficiently higher than the threshold voltage
to drive the transistor switch to ON mode. In analog neural networks, the
$g_{i,j}$ are weight variables that can take continuous values. In most
practical hardware implementations, the $g_{i,j}$ is a discrete variable
having limited set of values. As memristors have high readout speeds in
crossbar, the MAC operations can perform analog numerical computations [41].
An example is the Phase Change Memory based memristor crossbar to solve linear
system of equations using iterative refinement algorithm in a mixed-signal
architecture. Such a system showed computational complexity of
$O(\textnormal{D}^{2})$ for an D $\times$ D matrix, compared to
$O(\textnormal{D}^{3})$ in traditional numerical computation algorithms [42].

The crossbars’ reading and writing require additional circuits specific to the
neural network architecture. In many neural networks, the weights can be
positive and negative. To implement such a system, each synaptic column is
represented as two distinct column lines, thereby doubling up the number of
memristors required. The negative and positive signs are achieved by splitting
the conductance components as positive $g_{i,+}^{+}$ and negative
$g_{i,-}^{-}$ along two crossbar columns (say positive and negative column),
which reflects as
$i_{1}^{+}-i_{1}^{-}=\sum_{i=1}^{N}v_{i}(g_{i,+}^{+}-g_{i,-}^{-})$. Here, the
weights of the neuron $w_{i}\propto(g_{i,+}^{+}-g_{i,-}^{-})$, where
$g_{i,+}^{+}$, and $g_{i,-}^{-}$ are conductance of memristor nodes in
positive and negative columns, respectively, while current difference
$i_{1}^{+}-i_{1}^{-}$ is realised with a sense amplifier having a current
difference circuit.

The readout circuits comprises sense amplifiers and data converters [43].
There are two main possibilities in processing the current outputs. They can
be passed through an analog implementation of the activation function followed
by the next neural crossbar layer. Alternatively, the output currents can be
digitally encoded using a data converter, followed by activation function
computation and the next layer of input. In an analog-only approach, the
design of sense amplifier and activation function becomes challenging, as they
are power-hungry, although it can be area efficient. In digital-only
implementations, the power is on the lower end, but they become area
inefficient due to data converters and multiplexing units.

Figure 2: Unintentional flow of current in $g_{12}$, $g_{22}$ and $g_{21}$ is
representative of sneak path currents in a crossbar.

In Fig. 2 example, it can be noted that each column is assumed to be coupled
to a sense amplifier. If the column with $i_{2}$ is not coupled to a sense
amplifier, all current in such case will go to $i_{1}$. The sneak path
currents [44, 45, 46, 47, 48, 49, 50] in the crossbar can be controlled by
serial reads, making the crossbar reads slower. In serial read, one column in
the crossbar is enabled at a time and the output currents are read along the
column. In parallel read, multiple columns in the crossbar are enabled and the
output currents are read along the enabled columns in parallel simultaneously.
A parallel read is faster, but have more current errors. Smaller crossbars
will have a fewer paths in the crossbar to induce sneak path errors. Keeping
the crossbar size smaller also lowers the line capacitance and signal
integrity issues.

Increasing the crossbar size is required for building large neural networks.
One way to get around this problem is using modular crossbar arrays or
crossbar tiles [51, 52]. The current summation can be split into several sub-
currents by splitting the large crossbar into a set of smaller crossbars
followed by summing the output currents. As smaller crossbar will have smaller
leakage currents, lower sneak-path currents due to the use of multiple
crossbars to represent larger crossbars, and parasitic errors.

##  III Super-resolution Crossbar Nodes

In a memristor crossbar, any undesired change node conductance will lead to
errors in output column current. Ideal memristors can be programmed to an
infinite number of stable conductance values. However, practical memristors
can be programmed to only a limited number of stable conductance values. The
number of stable conductance values is defined as conductance levels. If the
conductance of the memristor is limited to $L$ levels, i.e., $L$ number of
conductance values per device, many optimal weights of the neural network will
become difficult to translate directly to hardware. One approach around this
is to encode the weights with a set of memristor devices in the crossbar.

Figure 3: Memristor crossbar showing the proposed super-resolution conductance
nodes. In the illustration, each crossbar node is made up of two memristors.

In the proposed approach, multiple rows are tied to a single input, as shown
in Fig. 3, enhancing the number of conductance levels per node and increase
the overall resolution of the crossbar. Suppose each memristor have only two
states, i.e. $\left[g_{l},\ g_{h}\right]$, then the weights are considered to
be binary. Now, when we connect two rows for a single input, the overall
conductance will be the summation of the two conductance, which means it can
take at most any of the three unique conductance values (see Table I). The
occurrence of conductance combinations can be expressed in a polynomial form.
If we represent, $p_{l}$ and $p_{h}$ as the symbolic occurrence of $g_{l}$ and
$g_{h}$, respectively, a node with two memristors can be expressed as
$\left(p_{l+}p_{h}\right)^{2}=p_{l}^{2}+p_{h}^{2}+2p_{l}p_{h}$, where
$p_{l}^{2}$ translates to $2g_{l}$, $p_{h}^{2}$ translates to $2g_{h}$, and
$2p_{l}p_{h}$ translates to $\left(g_{l}+g_{h}\right)$. Here, $p_{l}$ and
$p_{h}$ are indicative of the number of conductance levels per memristor, and
the power of 2 represents the number of memristors.

TABLE I: Unique conductance values obtained by connecting two binary memristors in parallel to form a crossbar node.  Memristor 1 | Memristor 2 | Overall conductance
---|---|---
$g_{l}$ | $g_{l}$ | $2g_{l}$
$g_{l}$ | $g_{h}$ | $g_{l}+g_{h}$
$g_{h}$ | $g_{l}$ | $g_{l}+g_{h}$
$g_{h}$ | $g_{h}$ | $2g_{h}$

Extending the idea to $m$ memristors, this can be represented as
$\left(p_{l+}p_{h}\right)^{m}=\sum_{k=0}^{m}\frac{m!}{k!\left(m-k\right)!}p_{l}^{k}p_{h}^{m-k}$,
leading to having binomial properties for finding the unique coefficients.
This indicates that the number of coefficients follows the simplicial
polytopic numbers sequences.

######  Theorem 1.

A combination of $m$ bi-level conductance memristors in parallel for creating
the super-resolution crossbar conductance node follows unique conductances
($L_{C}$) of 2,3,4..$m$+1 by encoding 1-simplicial number patterns. The
increase in $L_{C}$ enables the conversion of binary conductance nodes to
analog discrete conductance nodes in a crossbar.

Proof: Table II shows the number of memristors and corresponding unique
conductance per node. Let’s denote the unique conductance $(L_{C})$ set for
bi-level memristor nodes with $L=2$ as $U_{g}^{2}$. This sequence
$U_{g}^{2}=\\{2,3,..m+1\\}$ are part of simplicial sequence if it encodes the
1-simplicial number properties. Here, we can rewrite,
$U_{g}^{2}=\\{2,3,..m+1\\}$
$U_{g}^{2}=\\{1,2,..m\\}+\\{1,1,..1\\}$
$U_{g}^{2}=A+B$, for, $A=\\{1,2,..m\\}$ and $B=\\{1,1,..1\\}$. Suppose,
$t_{s(m)}$ represents the sum of $m$ numbers in the sequence $A$ and $t_{m}$
represents the sum of $m$ numbers in the sequence $U_{g}^{2}$. Here,
$t_{s(m)}=\frac{m(m+1)}{2}$, and therefore, $t_{m}=t_{s(m)}-1$ for $m\geq 0$.
For example, for $m=2$, $t_{s(2)}=\frac{2(2+1)}{2}=3$; for $m=3$,
$t_{s(3)}=\frac{3(3+1)}{2}=6$, and so forth, with $t_{m}=2$ for $m=2$,
$t_{m}=5$ for $m=3$, and so forth. The conductance with different memristor
numbers are given in Table II. The sum of the sequence $t_{s(m)}$ validates
the existence of 1-simplicial sequence or the Triangular gnomon (linear)
number sequence. We can also further observe additional relations between
$t_{m}$ and $t_{s(m)}$, for computing the value of $m$ given $t_{s(m)}$ or
$t_{m}$, and vice versa:
(1) $t_{m}-t_{m-1}=t_{s(m)}-t_{s(m-1)}=L_{C}(m),m\geq 1$,
(2) $t_{m}-2t_{m-1}+t_{m-2}=t_{s(m)}-2t_{s(m-1)}+t_{s(m-2)}=1,m\geq 2$,
(3) $\frac{t_{m}}{t_{m-1}}=\frac{t_{s(m)}-1}{t_{s(m-1)}-1}$.
Given that $t_{s(m)}$ validates $A$ for being 1-simplicial numbers and as it
is encoded in $U_{g}^{2}$; we can say that $L_{C}$ for $L=2$ encodes
1-simplicial numbers. This proves Theorem 1 on the existence of 1-simplicial
number patterns in the unique conductance set by combining bi-level
memristors. $\square$

Theorem 1 has implications in the design and use of binary crossbar array. Any
binary crossbar array can be transformed into an analog crossbar array using
the proposed super-resolution principle. In other words, the memristors with
two states can create nodes with several conductance levels by adding more
memristors in parallel. As shown in Table II, a node with $m$ memristors can
achieve $m+1$ conductance values. Because we use 10 rows of the crossbar to
represent single input, i.e., ten memristors per input node, with each having
a binary conductance state, we can obtain 11 unique conductive states, which
in itself does not show as a major advantage. However, with the increasing
number of conductive states per memristor, the scenario significantly changes.
For example, two memristors in a node with each memristor having two
conductive states results in unique conductance ($L_{C}$) that follow
2-dimensional simplicial number sequence as shown in Table III.

TABLE II: Number of unique conductance values obtained from $m$ binary memristor per crossbar node.  Number of memristors | Unique conductance
---|---
1 | 2
2 | 3
3 | 4
4 | 5
.. | ..
$m$ | $m+1$

The simplicial polytopic numbers belong to the broad class of figurate numbers
that are $r$-dimensional simplex for each dimension $r$ in the Euclidean space
$R^{r}$, with $r\geq 0$. In algebraic topology, the $r$\- simplex is a
$r$-dimensional polytope and is a convex hull of $r+1$ vertices. They can be
visualized as a generalization of the triangle to any arbitrary dimension,
connected to the coefficients of multi-nominal expressions. These topological
representations lay the foundations of the simplicial polytopic numbers.

The number of unique conductance combinations follows multi-nominal
expressions and can be represented as $r$ simplicial numbers. 2-simplicial
numbers originate out of 2-simplex having 3 vertices representative of the
triangle (the 3 1-cell, with 3 segments as facets). Figure 4 shows the
visualisation of the 2-simplicial numbers derived by counting the number of
segments (or edges) of triangular shapes with an increasing number of
vertices. Counting the number of segments in each shape gives the sequence,
$\\{1,3,6,10,15,21,28...\\}$, which forms the 2-simplicial numbers. Likewise,
3-simplicial numbers originate out of 3-simplex having 4-vertices
representative of the tetrahedron (the 4 2-cell, with 4 faces as facets). In
general, $r$-simplicial numbers originate out of $r$-simplex having $r+1$
vertices representative of the triangle (the $r+1$ $\left(r-1\right)$-cell,
with $r+1$ segments as facets).

Figure 4: Visualisation of 2-simplicial numbers as number of segments of
triangles. With each addition of a vertex and counting the number of all
connected segments gives simplicial polytopic number.

######  Theorem 2.

The  $m$ $L$-level conductance memristors super-resolution crossbar
conductance node follows an $r$-dimensional simplicial number sequence.
Increasing $m$ irrespective of $L$ results in an increase in $L_{C}$, such
that $L_{C}\geq L$.

Proof: Table III shows the set of unique conductance ($L_{C}$) following
simplicial numbers for nodes with $m$ number of memristors. The conductance
values for $L=\\{1,2,3\\}$, for $m=\\{1,2,3\\}$ is shown in Table IV, which
follows the $\\{1,2,3\\}$-simplicial number sequences. For a simplicial
sequence, the $k$th simplicial $r$-polytopic number can be calculated as:
$S_{r+1}^{(r)}(k)={r+(k-1)\choose k}=\frac{k^{(r)}}{r!}$. Replacing, $k$ with
$m$, and $r$ with $L$, we find that the values of $L_{C}$ in Table IV is same
as that obtained by applying equation for $S$, as $\\{1,2,3\\}$, $\\{1,3,6\\}$
and $\\{1,4,10\\}$. Here, we observe that $L_{C}=f(m,L)$ is a function of $m$
and $L$, with:
$f(2,1)=f(2,0)+f(1,1)$
$f(2,2)=f(2,1)+f(1,2)$
$f(2,3)=f(2,2)+f(1,3)$
…….
$f(m,L)=f(m,L-1)+f(m-1,L)$
This implies that,
$L_{C_{L+1}}^{(L)}(m)$=$L_{C_{L+1}}^{(L)}(m-1)$+$L_{C_{L+1}}^{(L-1)}(m)$, is
the recurrence relation for $m$th simplicial $L$-polytopic numbers. Applying
the equation for $S_{r+1}^{(r)}(k)$ and replacing $k$ with $m$, and $r$ with
$L$, the unique number of conductance sequence from a node with $m$ memristors
can be then written as
$\\{1,\frac{m^{(1)}}{1!},\frac{m^{(2)}}{2!},\frac{m^{(3)}}{3!},...,\frac{m^{(L)}}{L!}\\}$.
Here, $L\geq 1$ can be considered as the dimension of the regular convex
simplicial polytope number sequence, while $m\geq 1$ is the number of
memristors per node as the number of nondegenerate layered simplices. For any
given value of $m$, suppose the memristor has $L$ levels, then the unique
number of levels $L_{C}$ becomes $\frac{m^{(L)}}{L!}$, which makes $L_{C}=L$
for $L=1$, and $L_{C}>L$ for $L>1$. This implies that $L_{C}\geq L$ for any
value of $m$. $\square$

TABLE III: Crossbar node attain distinct conductance values for each of $M$ memristor having $L$ conductance state.  Number of memristors ($M$) | Number of levels ($L$) |  Unique conductance ($L_{C}$) | Sequence type
---|---|---|---
1 | $\\{1,2,3,4,5,6,7,8,9,10,11,12..\\}$ |  $\\{1,2,3,4,5,6,7,8,9,10,11,12..\\}$ | 1-simplicial numbers
2 | $\\{1,2,3,4,5,6,7,8,9,10,11,12..\\}$ |  $\\{1,3,6,10,15,21,28,36,45,55,66,78,..\\}$ | 2-simplicial numbers
3 | $\\{1,2,3,4,5,6,7,8,9,10,11,12..\\}$ |  $\\{1,4,10,20,35,56,84,120,165,220,286,364,..\\}$ | 3-simplicial numbers
4 | $\\{1,2,3,4,5,6,7,8,9,10,11,12..\\}$ |  $\\{1,5,15,35,70,126,210,330,495,715,1001,1365,..\\}$ | 4-simplicial numbers
5 | $\\{1,2,3,4,5,6,7,8,9,10,11,12..\\}$ |  $\\{1,6,21,56,126,252,462,792,1287,2002,3003,4368,..\\}$ | 5-simplicial numbers
6 | $\\{1,2,3,4,5,6,7,8,9,10,11,12..\\}$ |  $\\{1,7,28,84,210,462,924,1716,3003,5005,8008,12376,..\\}$ | 6-simplicial numbers
7 | $\\{1,2,3,4,5,6,7,8,9,10,11,12..\\}$ |  $\\{1,8,36,120,330,792,1716,3432,6435,11440,19448,31824,..\\}$ | 7-simplicial numbers
8 | $\\{1,2,3,4,5,6,7,8,9,10,11,12..\\}$ |  $\\{1,9,45,165,495,1287,3003,6435,12870,24310,43758,75582,..\\}$ | 8-simplicial numbers
TABLE IV: Example of unique conductance values obtained from the crossbar nodes with combination of 1, 2 and 3 memristors each having $L=\\{1,2,3\\}$ conductance levels.  $L$ | 1-M | 2-M | 3-M
---|---|---|---
|  Conductance value |  $L_{C}=L$ |  Conductance value |  $L_{C}$ |  Conductance value |  $L_{C}$
1 |  $g_{1}$ |  1 |  $2g_{1}$ |  1 |  $3g_{1}$ |  1
2 |  $g_{1},g_{2}$ |  2 |  $2g_{1},2g_{2},(g_{1}+g_{2})$ |  3 |  $3g_{1},3g_{2},g_{1}+2g_{2},g_{2}+2g_{1}$ |  4
3 |  $g_{1},g_{2},g_{3}$ |  3 |  $2g_{1},2g_{2},2g_{3},g_{1}+g_{2},g_{1}+g_{3},g_{2}+g_{3}$ |  6 |  $3g_{1},3g_{2},3g_{3},2g_{1}+g_{2},2g_{1}+g_{3},2g_{2}+g_{1},2g_{2}+g_{3},2g_{3}+g_{1},2g_{3}+g_{2},g_{1}+g_{2}+g_{3}$ |  10
Figure 5: Programming of the superresolution node. (a) Inference schematic,
(b) SET and RESET sequence for programming the set conductance to a desired
value, and (c) Programming schematic.  Figure 6: Example comparison of
crossbar node with single memristor (1-M) with three memristor (3-M). The sub-
figures (b,d,f,h) are for 1-M node with unique conductances $L_{C}$ of 2, 3, 4
and 5, respectively. The sub-figures (a,c,e,g) shows unique conductance for
3-M nodes that are built using corresponding 1-M nodes in (b,d,f,h).

In such a generalization, both the number of conductance levels and memristors
can be increased. Currently, there are memristors that can have more than 100
conductance levels [53]. However, most of the memristor devices have only very
few stable conductance levels that are easily programmed under the influence
of integrated circuit parasitics and aging. To build analog crossbar arrays
that can map weights at high accuracy and precision, it must have a higher
number of levels.

The proposed approach, when extended to include the impact of a higher number
of conductance levels per device, implies a higher number of possible
conductance per node. Table III shows the generalization of this scenario. It
is observed that the unique conductance per node for a given number of
memristors follows $r$-simplicial numbers. With memristors having even 3
stable levels, increasing the number of memristors per node can lead to a
large number of conductance levels per node as seen in Tables III and IV. As
the number of levels per memristor increases, fewer memristors are required to
achieve a high number of levels per node. For example, a node made up of 8
memristors, each having 8 levels, will result in 6435 unique conductance per
node.

###  III-A Illustrative Example

Table IV shows the conductance calculations for crossbar nodes made up of one
memristor (1-M), two memristors (2-M), and three memristors (3-M). Each
memristor has $L$ conductance levels. The $L_{C}$ for 1-M follows the set
{1,2,3..}, 2-M follows the set {1,3,6..} and 3-M follows the set {1,4,10..}.
Thus, the $m$-M node follows the simplicial number sequence
$\\{1,\frac{m^{(1)}}{1!},\frac{m^{(2)}}{2!},...\\}$, where $m$ is the number
of memristors per node. Figure 5 shows the programming scheme and using the
superresolution nodes in the inference stage. The programming of the
memristors in each node is achieved by applying the SET and RESET cycles of
pulses. For the programming stage, the desired conductance states
$g_{target1}$ and $g_{target2}$ for the 1T1M memristor node can be obtained by
repeated SET and RESET cycles, as shown in Fig. 5(b). The switch over between
the SET and RESET operations use the switching logic shown in Fig. 5(c). The
voltage pulses induce low potential difference during read (or inference or
verify operation), high positive potential difference during program
operations, and negative potential difference during the erase operation
across the memristor. During the inference, the crossbar does not need any
modification, and superresolution nodes can be formed by feeding the same
voltage $v_{1}$ to $g_{11}$ and $g_{21}$ 1T1R nodes to form $1\times 2$ sized
superresolution node, as shown in Fig. 5(a).

Figure 6 shows the example simulation of conductance level distribution
resulting from a 3-M node compared with a 1-M node. The 3-M nodes built with
memristors having 2, 3, 4, and 5 unique conductance levels can result in 4,
10, 20, and 35 unique conductance levels, respectively. In this example, the
memristors’ conductance levels are randomly assigned to a stable value that
the individual memristor can be programmed. The resulting 3-M node shows a
higher number of stabilized and unique conductance values. Therefore, the 3-M
based crossbar can have a higher weight resolution in a neural network than
1-M nodes. The conductance drift due to aging in 1-M nodes can result in loss
of conductance levels that cannot be recovered quickly. Even if one of the
memristor loss levels in a 3-M, the available conductance levels ($L$) in the
remaining two memristors can still result in large values of $L_{C}$ for the
superresolution nodes. The example also shows that individual memristors in a
3-M node can easily program the states to a few stable states distributed in
any fashion. The resulting conductance space will be large, allowing for more
considerable weight precision.

#### Programming sequence

An example sequence to program the conductance values of the 3-M node, using
three memristors each having four levels is shown in Table V. The node
conductance $g_{n}$ can be mapped to the weights in the neural network
application or general in-memory computing applications using a lookup
approach. Suppose a weight of 0.3 is mapped to the conductance of 40 $\mu S$,
the three memristors in the 3-M node are required to be programmed to the
combination without following the sequenced order 15, 15, and 10 $\mu S$
values. The intermediate weight values such as 0.33 are rounded off to 0.3 by
the nearest distance search and subsequently programmed to the value from the
lookup table, in Table V. In contrast, a 1-M node with four levels would have
resulted in a significant loss of weight precision as many intermediate levels
are missed out. Through this approach, the complexity of programming crossbar
nodes to obtain high-resolution weights substantially reduces, as fewer levels
are often easier to program and are more reliable to achieve. In addition, the
super-resolution nodes are formed by a parallel or series combination of
memristors, depending on the ease of programming conductance or resistance of
individual memristors, respectively. More details on programming memristor
based on the resistance or conductance are included in the Supplementary
material.

TABLE V: Programming sequence for memristor with 4 levels (10,15,29,1000)$\mu S$ for a 3-M node. 3-M node conductance($\mu S$) | 3-M node conductance($\mu S$)
---|---
$g_{1}$ | $g_{2}$ | $g_{3}$ | $g_{n}$
10 | 10 | 10 | 30
15 | 10 | 10 | 35
15 | 15 | 10 | 40
15 | 15 | 15 | 45
29 | 10 | 10 | 49
29 | 15 | 10 | 54
29 | 15 | 15 | 59
29 | 29 | 10 | 69
29 | 29 | 15 | 74
29 | 29 | 29 | 88
1000 | 10 | 10 | 1020
1000 | 15 | 10 | 1025
1000 | 15 | 15 | 1030
1000 | 29 | 10 | 1039
1000 | 29 | 15 | 1044
1000 | 29 | 29 | 1059
1000 | 1000 | 10 | 2010
1000 | 1000 | 15 | 2015
1000 | 1000 | 29 | 2029
1000 | 1000 | 1000 | 3000

#### Case of 3D crossbar

As an extension, the resistance range can be further increased using a 3D
crossbar array. In the example shown in Fig. 7, the top and bottom layer
conductances are denoted as $g_{ij}^{(1)}$, and $g_{ij}^{(2)}$, respectively.
Because each node is a parallel combination of two memristors connected in
series across two layers, current
$i_{1}=v_{1}*(g_{11}^{(1)}||g_{11}^{(2)})+v_{1}*(g_{21}^{(1)}||g_{21}^{(2)})$.
Each memristor node has a selector switch. When current $i_{1}$ is read, the
memristor nodes in columns $i_{2}$ and $i_{3}$ are kept to OFF state. Hence,
$g_{11}^{(1)}$ will be in series to $g_{11}^{(2)}$ In general, the read out
currents from any one column is given as, $i_{k}=\sum v_{k}g_{kl}$, with the
equivalent node conductance as follows:

$g_{kl}=\left[\sum\frac{1}{g_{ik}^{(1)}}\right]^{-1}+\left[\sum\frac{1}{g_{ik}^{(2)}}\right]^{-1}.$
(1)

Such 3D nodes can further extend the number of $L_{C}$ levels without
significantly increasing the area on-chip. The different configurations and
possibilities with the 3D crossbar array are a topic for future study.

Figure 7: Extension of the conductance range using 3D crossbar configuration.
Note that each memristor comes with a selector switch to program and select
the column.

##  IV Results and Discussion

The crossbars are used in a range of applications, mainly, in building neural
network accelerators. To check the impact of the number of conductance per
node on the overall performance of the crossbar, we take a crossbar array with
10$\times$10 nodes. The conductance programming is performed using an adaptive
program and verify approach [54]. The output currents are read from the
columns for the different memristors and conductance levels per node.
Throughout the paper, unless mentioned otherwise, random current read errors
are added as a Gaussian distribution with mean zero and standard deviation as
10% of ideal read currents to account for the non-idealities of the sensing
circuits connected to the crossbar. Figure 8 shows the relative output current
error (RCE) in percentage, calculated as
$\frac{100\times\left|I_{o-ideal}-I_{o-real}\right|}{I_{o-ideal}}$, where
$I_{o-ideal}$ is the output current obtained when nodes have infinite number
of conductance levels, and $I_{o-real}$ is the output current obtained with a
limited number of conductance levels.

#### Precision

The precision of the weights mapping to crossbar nodes is an important factor
that determines the accuracy of analog dot product computations. It can be
observed from Fig. 8 that as the number of memristor per node increases, the
current error decreases. The increase in $L$ along with increase in $m$
results in a large increase in $L_{C}$, as shown in Table III. As $L_{C}$
increases, i.e. for larger value of $m$ and $L$, the current error tends to
zero. The memristors in the superresolution crossbar are connected in parallel
and the current readout from the nodes will be the weighted average of the
input voltage, i.e. $i=\sum_{i=1}^{m}g_{i}v_{in}$, where $v_{in}$ input
voltage to a node with $m$ memristors each having its own conductance $g_{i}$.
Suppose there is a variation in one of the conductance values say in the $m$th
memristor i.e. $g_{m}=g_{i}\pm\delta$, with $\delta$ being the offset from the
true value, then we can write, $i=\left[\sum_{i=1}^{m-1}g_{i}v_{in}\right]\pm
g_{m}v_{in}$ or $i=\sum_{i=1}^{m}g_{i}v_{in}\pm\delta v_{in}$. Thus the
variations, i.e. $\delta$, will have less impact on the overall current $i$ as
the value of $m$ increases.

Figure 8: Relative output current error percentages in a crossbar having 10x10
nodes, with $R_{OFF}/R_{ON}=100k\Omega/1k\Omega$.

Figure 9 shows the average relative current errors (RCE), with changes in
$R_{OFF}/R_{ON}$ ratio of memristor relative to the node size. The larger the
$R_{OFF}/R_{ON}$ ratio, the larger the RCE values irrespective of the node
size. However, as the number of memristors increases in the node, the RCE
values drastically reduce irrespective of the changes in $R_{OFF}/R_{ON}$
ratio, indicating the crossbar superresolution’s usefulness.

(a) 1-M

(b) 2-M

(c) 3-M

(d) 4-M

(e) 5-M

(f) 6-M

Figure 9: Impact of the change in $R_{OFF}/R_{ON}$ ratios (i.e. 100, 80, 60,
40, 20, 10, 5) relative to the change in node size (i.e. 1-M, 2-M, 3-M, 4-M,
5-M, 6-M) and conductance levels.

#### Area and power

Each memristor has one selector transistor switch. Irrespective of the node
size, i.e., nodes with $m$ memristor $m$T-$m$M, the interface circuits
required at the output of the analog neural network crossbar remain the same.
If we assume the area occupied by interface circuits connected to each of the
crossbar column to be $A_{i}$, and the area of each 1T1M node to be $A_{c}$,
then the minimum total area for implementing one layer of a neural network
will be $kA_{i}+nmkA_{c}$, where $k$ is the number of columns in the crossbar,
$n$ is the number of nodes in a column, and $m$ is the number of memristors
per node. In analog neural network implementations [55], $A_{i}>>A_{c}$, for
example, in a 180 nm CMOS technology111The 180 nm CMOS node used is for
demonstrative purpose only. It is possible to use lower node sizes such as 22
nm or lower FinFETs [56] node sizes for reducing the area and power
requirements in practical applications. For memristors, we used Knowm MSS
(Multi-Stable Switch) memristor model and WOx device parameters [57]., the
interface circuits to each column of the crossbar consists of I-V converter
block, Opamp buffer, and ReLU block, occupying areas of 96.714 $\mu$m2,
44.712$\mu$m2, and 23.659$\mu$m2, respectively, as shown in Fig. 10222Here,
the ReLU PMOS has W/L=0.36/0.18, and ReLU NMOS has W/L=1/0.18. Switch
transistor sizes are 5/0.18 (PMOS), and 20/0.18(NMOS), and feedback resistors
$R_{1}=R_{2}=R_{3}=1k\Omega$. The single 1T1M memristor node occupies an area
of 0.14$\mu$m2. For a $10\times 10$ super-resolution crossbar, with 8T8M
nodes, the minimum areas are $kA_{i}=1650.85\mu m^{2}$, and $nmkA_{c}=11.2\mu
m^{2}$, where $n=10$, $m=8$, $k=10$. Alternatively, if we use a classical
crossbar, 1T1M nodes spread across 8 columns, the equivalent crossbar will
have a size of $10\times 80$, with minimum areas of $kA_{i}=13206.8\mu m^{2}$,
and $nmkA_{c}=11.2\mu m^{2}$, where $n=10$, $m=1$, and $k=80$. In addition, as
the lower and upper limits of resistance values in the crossbar node (i.e.,
$R_{ON}$ and $R_{OFF}$ limits) does not change, and reverse leakage currents
in the transistors are relatively lower than the forward currents in the
memristor, the net maximum current output range out of the crossbar columns do
not change. The average power dissipation per convolution filter when using a
ReLU block is 40.16 mW 333This can be further scaled down to 1.63 mW if using
a lower technology such as 45 nm.. The delay associated with this block is
1.28 ns. For the dense layer crossbar implemented with ReLu, the power comes
to 441.9 mW, with a propagation delay of 1.45 ns. The overall maximum power
consumption is mainly impacted by the non-idealities of the transistor, which
can be handled by using FinFETs. Further, even if there is an increase in
memristors per node and size of the crossbar, the readout circuits connected
to the crossbar columns do not need to be changed if the output current range
is kept the same by proportionally scaling the conductance levels.

Figure 10: Various circuit block in the CNN. (a) shows the circuit block for
convolution layer, (b) ReLU circuit, (c) mean pooling block, and (d) dense
layer block.

#### Case of scaling the crossbar

As the crossbar array size increases, the sneak path currents and leakage
currents in the switches that add to the current errors also increase.
Therefore, the large crossbar arrays are split into smaller arrays, known as
the modular crossbar array or tiling of crossbar arrays. In analog crossbars
tiling, the input space is split across several crossbars, and their column-
wise output is summed up. The area overhead for this is a summing amplifier,
which adds another 4.712$\mu m^{2}$ per column while allowing for very large
crossbar implementations.

#### Aging

Three types of aging are observed in memristors (for details, see
supplementary material). Higher the aging, the more the number of conductance
levels that are lost. In our study, we select type 3 aging, which increases
the value of $R_{ON}$ and decreases $R_{OFF}$ with repeated stress on the
device. Setting the memristor $R_{OFF}/R_{ON}$ ratio as 100, the effect of
aging is observed for different node sizes. The nodes with single memristors,
irrespective of the number of levels, tend to have high error levels. The
aging ratios shown in the legend of Fig. 11 indicate the ratio of change in
$R_{OFF}$ and $R_{ON}$ with respect to its original value. For example, the
aging ratio of 0.1 means the new resistance to be $0.9R_{OFF}$ and
$1.1R_{ON}$. The states are reprogrammed in these experiments; hence, attempts
are made to recover the number of conductance states that disappear with
aging. The nodes with more than three memristors show less than 10% relative
current errors under a high aging ratio of 0.7, irrespective of the number of
conductance levels. In all cases, the errors are higher for aging when the
number of conductance levels is low. The increase in the number of levels
reduces the relative current errors.

(a) 1-M

(b) 2-M

(c) 3-M

(d) 4-M

(e) 5-M

(f) 6-M

Figure 11: Impact of type 3 aging (i.e. with $R_{OFF}$ decreasing and $R_{ON}$
increasing) on the accuracy of the crossbar output currents, for different
node size, and with various aging ratios.

The memristors, like any other semiconductor device, get affected by non-
idealities and aging. As the devices are used continuously for extended
periods, the conductance states of the memristors gradually become harder to
program. When the device has a large number of conductance levels over a
period, the number of conductance levels decreases. Having a larger number of
levels would reduce reliability as even minor variations in the conductance
reduce the conductance levels. The crossbar node (165 unique conductance)
built with eight memristors, each having four conductance levels will be more
robust than a crossbar node (165 unique conductance) with three memristors
each having nine conductance levels.

#### Signal noise

The input signals to the crossbar are prone to noise. This noise can result
from sensors or signal integrity issues. To test the impact of signal noise on
the superresolution crossbar, we apply additive Gaussian noise with variances,
as shown in Fig. 12. The RCE values indicate that the superresolution nodes do
not impact the relative accuracy of the current outputs relative to the node
with ideal weights.

(a) 1-M

(b) 2-M

(c) 3-M

(d) 4-M

(e) 5-M

(f) 6-M

Figure 12: Impact of input signal noise on superresolution crossbar. The
performance of the crossbar is compared for additive Gaussian noise with
different variance values.

#### Effects of wire resistance, read instabilities and $R_{ON},R_{OFF}$
variability

To analyse the impact of the crossbar variability, we perform boundary
analysis considering the combined effect of wire-resistance, $R_{ON}$,
$R_{OFF}$ changes in the crossbar. At each node, the maximum variation is
assumed to be 2.5 $\pm$ 0.25 $\Omega$ for wire resistance, boundary
resistances change as $R_{ON}\pm 0.2R_{ON}$ and $R_{OFF}\pm 0.2R_{OFF}$. Table
VI shows that when there is wire resistance and changes in boundary
resistances, the impact is more evident in a single memristor node and less
prominent in super-resolution nodes. Here, N and Y indicate the cases without
and with wire resistance, respectively. In both cases, the boundary
resistances changes by a maximum 20% standard deviation from the mean $R_{ON}$
and $R_{OFF}$. Read instability [58, 59] can also occur in a crossbar, with
the current readouts varying from their mean value. Inclusion of a high read
instability, of 10%, on the crossbar readout currents, are shown in the column
labelled R in Table VI. Even in this case, the use of multiple memristors per
node show far more robustness than single memristor per node crossbar.

TABLE VI: RCE (%) values showing impact of wire resistance and read errors
with $R_{ON}\pm 0.2R_{ON}$ and $R_{OFF}\pm 0.2R_{OFF}$.

| 1-M node | 2-M node | 3-M node | 4-M node | 5-M node | 6-M node
---|---|---|---|---|---|---
$L$ | N | Y | R | N | Y | R | N | Y | R | N | Y | R | N | Y | R | N | Y | R
2 | 14.7 | 15.5 | 15.7 | 7.8 | 7.8 | 8 | 5.3 | 5.3 | 5.3 | 3.7 | 3.7 | 3.9 | 2.9 | 3 | 3.4 | 2.6 | 2.6 | 2.7
3 | 7.9 | 8.2 | 10 | 3.8 | 3.8 | 4.7 | 2.5 | 2.4 | 3 | 1.8 | 1.9 | 2.5 | 1.5 | 1.4 | 1.9 | 1.2 | 1.3 | 1.6
4 | 5.3 | 5.3 | 8.1 | 2.5 | 2.5 | 4 | 1.7 | 1.7 | 2.8 | 1.2 | 1.2 | 2 | 1 | 1 | 1.5 | 0.9 | 0.9 | 1.4
5 | 3.6 | 3.8 | 7.2 | 1.8 | 1.8 | 3.5 | 1.3 | 1.2 | 2.6 | 0.9 | 1 | 1.7 | 0.8 | 0.7 | 1.4 | 0.6 | 0.6 | 1.1
6 | 2.9 | 2.9 | 6.8 | 1.4 | 1.5 | 3.4 | 1 | 1 | 2.2 | 0.8 | 0.7 | 1.6 | 0.6 | 0.6 | 1.4 | 0.5 | 0.5 | 1.1
7 | 2.5 | 2.5 | 6.8 | 1.3 | 1.3 | 3.5 | 0.8 | 0.8 | 2.2 | 0.6 | 0.6 | 1.7 | 0.5 | 0.5 | 1.3 | 0.4 | 0.4 | 1.1
8 | 2.0 | 2.2 | 6.6 | 1 | 1.1 | 3.5 | 0.8 | 0.7 | 2.2 | 0.5 | 0.5 | 1.7 | 0.4 | 0.4 | 1.4 | 0.3 | 0.4 | 1.1
9 | 2 .0 | 2.0 | 6.8 | 0.9 | 0.9 | 3.3 | 0.6 | 0.6 | 2.1 | 0.5 | 0.4 | 1.7 | 0.4 | 0.4 | 1.2 | 0.3 | 0.3 | 1
10 | 1.7 | 1.8 | 6.4 | 0.8 | 0.8 | 3.4 | 0.6 | 0.5 | 2.1 | 0.4 | 0.4 | 1.7 | 0.3 | 0.3 | 1.2 | 0.3 | 0.3 | 1.1
11 | 1.5 | 1.6 | 6.5 | 0.7 | 0.7 | 3.2 | 0.5 | 0.5 | 2.2 | 0.4 | 0.4 | 1.6 | 0.3 | 0.3 | 1.3 | 0.2 | 0.2 | 1
12 | 1.3 | 1.4 | 6.4 | 0.7 | 0.7 | 3.2 | 0.5 | 0.4 | 2 | 0.3 | 0.3 | 1.6 | 0.3 | 0.3 | 1.3 | 0.2 | 0.2 | 1.1

#### Application in CNN

To perform the analysis and tests on the crossbar for an analog neural network
accelerator application, we select the convolutional neural network and its
analog hardware implementation outlined in [55]. The architecture consists of
two convolutional and two dense layers, with the circuits for implementing the
dropout444Dropout is a technique to regularise neural network by dropping OFF
neuronal nodes during the training, which avoids complex co-adaptations and
over-fitting on training data. and programming of memristor levels. The
circuit design and parameter extraction for CNN circuit blocks in [55] was
done in SPICE, while architecture level simulations for calculating
performance accuracy with and without variability were performed with
customised Python programs.

TABLE VII: Recognition accuracy when using a CNN accelerator with the proposed
memristor crossbar nodes. The tests are performed on CIFAR-10 dataset. The
ideal accuracy with infinite memristive states is 91.26%.

1-M node | 2-M node | 3-M node | 4-M node | 5-M node | 6-M node | 7-M node | 8-M node
---|---|---|---|---|---|---|---
$L_{C}=L$ |  ACC (%) |  $L_{C}$ |  ACC (%) |  $L_{C}$ |  ACC (%) |  $L_{C}$ |  ACC (%) |  $L_{C}$ |  ACC (%) |  $L_{C}$ |  ACC (%) |  $L_{C}$ |  ACC (%) |  $L_{C}$ |  ACC (%)
2 |  82.43 |  3 |  84.10 |  4 |  84.78 |  5 |  85.19 |  6 |  86.66 |  7 |  89.07 |  8 |  90.07 |  9 |  90.41
3 |  84.10 |  6 |  86.66 |  10 |  90.91 |  15 |  91.14 |  21 |  91.20 |  28 |  91.18 |  36 |  91.26 |  45 |  91.21
4 |  84.78 |  10 |  90.91 |  20 |  91.19 |  35 |  91.24 |  56 |  91.28 |  84 |  91.22 |  120 |  91.25 |  165 |  91.28
5 |  85.19 |  15 |  91.14 |  35 |  91.24 |  70 |  91.22 |  126 |  91.26 |  210 |  91.25 |  330 |  91.26 |  495 |  91.26
6 |  86.66 |  21 |  91.20 |  56 |  91.28 |  126 |  91.26 |  252 |  91.25 |  462 |  91.25 |  792 |  91.26 |  1287 |  91.26
7 |  89.07 |  28 |  91.18 |  84 |  91.22 |  210 |  91.25 |  462 |  91.25 |  924 |  91.26 |  1716 |  91.26 |  3003 |  91.26
8 |  90.07 |  36 |  91.26 |  120 |  91.25 |  330 |  91.26 |  792 |  91.26 |  1716 |  91.26 |  3432 |  91.26 |  6435 |  91.26
9 |  90.41 |  45 |  91.21 |  165 |  91.28 |  495 |  91.26 |  1287 |  91.26 |  3003 |  91.26 |  6435 |  91.26 |  12870 |  91.26
10 |  90.91 |  55 |  91.22 |  220 |  91.27 |  715 |  91.26 |  2002 |  91.26 |  5005 |  91.26 |  11440 |  91.26 |  24310 |  91.26
11 |  90.89 |  66 |  91.22 |  286 |  91.27 |  1001 |  91.25 |  3003 |  91.26 |  8008 |  91.26 |  19448 |  91.26 |  43758 |  91.26
12 |  91.02 |  78 |  91.23 |  364 |  91.27 |  1365 |  91.26 |  4368 |  91.26 |  12376 |  91.26 |  31824 |  91.26 |  75582 |  91.26

Table VII shows the recognition accuracy of implementing the CNN accelerator
with the proposed memristor crossbar nodes. The CIFAR-10 dataset has ten
classes, with each class having 6000 images. The CNN used for testing includes
two convolution layers, with average pooling 555Average pooling means
calculating the average of each patch of feature maps obtained from the
convolution layer, which goes as input to the subsequent layers. and dropout
ratio of 0.5, and two dense layers. The first layer uses the ReLu activation
function, while the last layer is the softmax function. It can be seen that at
the minimum, using two 8-level memristors or seven 3-level memristors gives
accuracy that matches with ideal case results of memristor having infinite
states, i.e., 91.26%. For reliability of operations, it is more appropriate to
use crossbar nodes with more memristors and conductive states. Even if some of
the states disappear due to aging, the impact on overall recognition accuracy
will be lower. Any conductance level of the memristor can be imagined as
having a mean conductance value with a standard variance. If the variance is
large, the conductance values can overlap and introduce larger errors. For
reducing this impact, it is recommended to allow larger separations between
the conductance levels, using more memristors per node with fewer conductance
levels. This increases robustness to conductance variations as can be observed
in the example of Table VIII, for an 8-M node with $L=3$ or $L_{C}=45$.

TABLE VIII: Recognition accuracy of using a CNN accelerator with proposed
memristor crossbar nodes. The tests are performed on CIFAR-10 dataset. Random
variability of 10% in the conductance values.

1-M node | 2-M node | 3-M node | 4-M node | 5-M node | 6-M node | 7-M node | 8-M node
---|---|---|---|---|---|---|---
$L_{C}=L$ |  ACC (%) |  $L_{C}$ |  ACC (%) |  $L_{C}$ |  ACC (%) |  $L_{C}$ |  ACC (%) |  $L_{C}$ |  ACC (%) |  $L_{C}$ |  ACC (%) |  $L_{C}$ |  ACC (%) |  $L_{C}$ |  ACC (%)
2 |  82.5 |  3 |  84.14 |  4 |  84.69 |  5 |  85.37 |  6 |  85.99 |  7 |  88.2 |  8 |  90.13 |  9 |  90.22
3 |  83.94 |  6 |  86.62 |  10 |  90.63 |  15 |  90.83 |  21 |  90.92 |  28 |  90.97 |  36 |  90.96 |  45 |  91.02
4 |  84.83 |  10 |  90.55 |  20 |  91 |  35 |  90.94 |  56 |  90.99 |  84 |  90.81 |  120 |  91.02 |  165 |  90.91
5 |  85.29 |  15 |  90.87 |  35 |  91.01 |  70 |  91 |  126 |  90.96 |  210 |  90.93 |  330 |  91.03 |  495 |  90.99
6 |  87.3 |  21 |  90.98 |  56 |  90.99 |  126 |  91.01 |  252 |  90.8 |  462 |  91 |  792 |  90.97 |  1287 |  91.06
7 |  89.11 |  28 |  90.83 |  84 |  90.98 |  210 |  90.9 |  462 |  91.01 |  924 |  91.05 |  1716 |  90.95 |  3003 |  90.91
8 |  90.16 |  36 |  91.01 |  120 |  90.93 |  330 |  91.02 |  792 |  90.92 |  1716 |  90.87 |  3432 |  90.98 |  6435 |  90.9
9 |  90.13 |  45 |  90.96 |  165 |  90.92 |  495 |  90.99 |  1287 |  91.01 |  3003 |  90.95 |  6435 |  91.04 |  12870 |  90.8
10 |  90.64 |  55 |  91 |  220 |  91 |  715 |  90.95 |  2002 |  90.98 |  5005 |  90.92 |  11440 |  91.01 |  24310 |  90.89
11 |  90.81 |  66 |  91.02 |  286 |  90.83 |  1001 |  90.95 |  3003 |  91 |  8008 |  90.92 |  19448 |  91.05 |  43758 |  90.99
12 |  90.7 |  78 |  90.94 |  364 |  90.95 |  1365 |  90.9 |  4368 |  90.89 |  12376 |  90.9 |  31824 |  90.98 |  75582 |  91

Table VIII shows the recognition accuracy of the CNN accelerator with the
proposed memristor crossbar nodes considering a 10% variability in the
conductance levels. The variation of conductance is added such that it is
random, and at a level follows a Gaussian distribution with maximum standard
deviation set at 10% of the mean value. The tests are repeated 30 times, and
the average accuracy is reported. It can be observed from Table VIII that the
conductance variability reduces the overall accuracy. An increase in the
number of memristors shows higher robustness. Here, the CNN is trained and
assumed to have obtained optimal weights. These weights are translated to the
conductance values of the memristor superresolution nodes. However, the
variability of the conductance values moves the weights further away from
optimal values. In addition, the changes in conductance can also move the
combination of weights back closer to optimal values increasing the accuracy.
For example, when ($L_{C}$=2, $M$=1) and ($L_{C}$=3, $M$=2), the recognition
accuracy is slightly higher under variations by 0.07% and 0.04%.

The over fitting in neural networks can be reduced by applying dropout based
regularization during training stages. Further, as the neural network size
grows, the energy consumption increases in both test and training stages. To
address this, dropouts can be applied in both test and training stages, as
shown in [60]. The superresolution nodes can use the same control circuits
typically required to program a regular crossbar. In addition, by connecting
the row-wise inputs together, superresolution nodes can be created without
physically changing the regular crossbar design. This makes the approach
universal to a wide range of analog neural network implementations, where the
performance of large arrays gets affected by the variability of the
conductance levels.

The ability of the proposed approach to transforming any multi-level crossbar
arrays to analog crossbar arrays offers improvement in precision and accuracy
of the crossbar for analog computing. Any reduction in current errors implies
higher accuracy of analog neural computations. Further, the superresolution
crossbar nodes also provide increased fault tolerance and reliability. Even if
any of the memristors fail or remain stuck at fault, the remaining memristors
can still offer a high number of conductive states without compromising the
analog neural network’s overall performance.

#### Optimising superresolution crossbar size

Figure 3 shows an example of a $2\times 1$ node within a crossbar. Figure 13
shows an extension by combining a different number of rows to create
superresolution nodes with different values of $m$. This approach can help to
optimise the number of memristors required per crossbar for a given
application. For example, if $3\times 1$ nodes are used to create $2\times 8$
superresolution crossbar nodes, then a total of 48 memristors are required,
while the combination of $3\times 1$ and $2\times 1$ nodes, as shown in Fig.
13 requires a total of 40 memristors, resulting in lesser usage of crossbar
area. To determine the value of $m$ for the superresolution node, for a fixed
$L$ value, the mapping of $m$ with $L$ and $L_{C}$ can be applied. Suppose
each memristor has $L=4$, and we want a conductance resolution of 0.1 (i.e.,
minimum 10 levels per node) and 0.01 (i.e., minimum 100 levels per node) in
the crossbar. Then looking at Table III we can determine the value of $m$ for
0.1 as 2 (with $L_{C}=10$), and for 0.01 as 7 (with $L_{C}=120$). Such
optimisation of crossbar will be useful for applications where parts of the
input signal vector within an input pattern such as attention or region of
interest in images require higher precision of weights in neural network
layers than others. Various applications that can be found for this approach
is left as an open problem.

Figure 13: Creating superresolution nodes with different values of $m$.

##  V Conclusion

In this paper, a compelling yet straightforward approach to improve the
performance and use of a memristive crossbar for analog neural network
implementations are presented. The system shows how a conventional crossbar
with memristor nodes with limited conductance resolution can be transformed
into a high-resolution conductive node. The resolution mapping is obtained by
input row mergers resulting in $r$-simplicial conductance sequences, with a
non-linear growth in conductance with an increase in memristors per crossbar
nodes. The increase in memristors per node, though it increases the area,
offers higher computation accuracy and a more straightforward mapping of
weight values.

The reduced relative current errors with an increase in memristors per
crossbar nodes show improvement in precision and accuracy of the crossbar MAC
calculations. This will, in particular, be useful in neural networks with many
neurons layers, where even minor variations in the weights will have a more
significant impact on inference accuracy. The current errors are not easy to
compensate for during the inference stages and require additional fine-tuning.
The need for complex fine-tuning circuits during inference stages can be
reduced using the proposed super-resolution approach.

## Acknowledgements

A. James’s research is supported by industry research grant Ind/001/2020. L.
Chua’s research is supported by Grant FA9550-18-1-0016.

## References

  * [1] H. Zhang, D. Chen, and S.-B. Ko, “New Flexible Multiple-Precision Multiply-Accumulate Unit for Deep Neural Network Training and Inference,” _IEEE Transactions on Computers_ , vol. 69, no. 1, pp. 26–38, jan 2020\. [Online]. Available: https://doi.org/10.1109%2Ftc.2019.2936192
  * [2] V. Camus, C. Enz, and M. Verhelst, “Survey of Precision-Scalable Multiply-Accumulate Units for Neural-Network Processing,” in _2019 IEEE International Conference on Artificial Intelligence Circuits and Systems (AICAS)_. IEEE, mar 2019. [Online]. Available: https://doi.org/10.1109%2Faicas.2019.8771610
  * [3] J. Kang and T. Kim, “PV-MAC: Multiply-and-accumulate unit structure exploiting precision variability in on-device convolutional neural networks,” _Integration_ , vol. 71, pp. 76–85, mar 2020. [Online]. Available: https://doi.org/10.1016%2Fj.vlsi.2019.11.003
  * [4] H. Tann, S. Hashemi, and S. Reda, “Lightweight Deep Neural Network Accelerators Using Approximate SW/HW Techniques,” in _Approximate Circuits_. Springer International Publishing, dec 2018, pp. 289–305. [Online]. Available: https://doi.org/10.1007%2F978-3-319-99322-5_14
  * [5] K. Siu, D. M. Stuart, M. Mahmoud, and A. Moshovos, “Memory Requirements for Convolutional Neural Network Hardware Accelerators,” in _2018 IEEE International Symposium on Workload Characterization (IISWC)_. IEEE, sep 2018. [Online]. Available: https://doi.org/10.1109%2Fiiswc.2018.8573527
  * [6] K. Seto, H. Nejatollahi, J. An, S. Kang, and N. Dutt, “Small Memory Footprint Neural Network Accelerators,” in _20th International Symposium on Quality Electronic Design (ISQED)_. IEEE, mar 2019. [Online]. Available: https://doi.org/10.1109%2Fisqed.2019.8697641
  * [7] H.-J. Yoo, “1.2 Intelligence on Silicon: From Deep-Neural-Network Accelerators to Brain Mimicking AI-SoCs,” in _2019 IEEE International Solid- State Circuits Conference - (ISSCC)_. IEEE, feb 2019. [Online]. Available: https://doi.org/10.1109%2Fisscc.2019.8662469
  * [8] W. Wen, Y. Zhang, and J. Yang, “ReNEW: Enhancing Lifetime for ReRAM Crossbar Based Neural Network Accelerators,” in _2019 IEEE 37th International Conference on Computer Design (ICCD)_. IEEE, nov 2019. [Online]. Available: https://doi.org/10.1109%2Ficcd46524.2019.00074
  * [9] S. Yin, Z. Jiang, J.-S. Seo, and M. Seok, “XNOR-SRAM: In-Memory Computing SRAM Macro for Binary/Ternary Deep Neural Networks,” _IEEE Journal of Solid-State Circuits_ , pp. 1–11, 2020. [Online]. Available: https://doi.org/10.1109%2Fjssc.2019.2963616
  * [10] Z. Jiang, S. Yin, J. sun Seo, and M. Seok, “C3SRAM: An In-Memory-Computing SRAM Macro Based on Robust Capacitive Coupling Computing Mechanism,” _IEEE Journal of Solid-State Circuits_ , pp. 1–1, 2020. [Online]. Available: https://doi.org/10.1109%2Fjssc.2020.2992886
  * [11] H. Choi, Y. Lee, J.-J. Kim, and S. Yoo, “A Novel In-DRAM Accelerator Architecture for Binary Neural Network,” in _2020 IEEE Symposium in Low-Power and High-Speed Chips (COOL CHIPS)_. IEEE, apr 2020. [Online]. Available: https://doi.org/10.1109%2Fcoolchips49199.2020.9097642
  * [12] Q. Liu, B. Gao, P. Yao, D. Wu, J. Chen, Y. Pang, W. Zhang, Y. Liao, C.-X. Xue, W.-H. Chen, J. Tang, Y. Wang, M.-F. Chang, H. Qian, and H. Wu, “33.2 A Fully Integrated Analog ReRAM Based 78.4TOPS/W Compute-In-Memory Chip with Fully Parallel MAC Computing,” in _2020 IEEE International Solid- State Circuits Conference - (ISSCC)_. IEEE, feb 2020. [Online]. Available: https://doi.org/10.1109%2Fisscc19947.2020.9062953
  * [13] J. A. Starzyk and Basawaraj, “Memristor Crossbar Architecture for Synchronous Neural Networks,” _IEEE Transactions on Circuits and Systems I: Regular Papers_ , vol. 61, no. 8, pp. 2390–2401, aug 2014. [Online]. Available: https://doi.org/10.1109%2Ftcsi.2014.2304653
  * [14] C. Yakopcic, M. Z. Alom, and T. M. Taha, “Extremely parallel memristor crossbar architecture for convolutional neural network implementation,” in _2017 International Joint Conference on Neural Networks (IJCNN)_. IEEE, may 2017. [Online]. Available: https://doi.org/10.1109%2Fijcnn.2017.7966055
  * [15] Y. Li, Z. Wang, R. Midya, Q. Xia, and J. J. Yang, “Review of memristor devices in neuromorphic computing: materials sciences and device challenges,” _Journal of Physics D: Applied Physics_ , vol. 51, no. 50, p. 503002, sep 2018. [Online]. Available: https://doi.org/10.1088%2F1361-6463%2Faade3f
  * [16] P. Sheridan and W. Lu, “Memristors and Memristive Devices for Neuromorphic Computing,” in _Memristor Networks_. Springer International Publishing, 2014, pp. 129–149. [Online]. Available: https://doi.org/10.1007%2F978-3-319-02630-5_8
  * [17] T. Prodromakis and C. Toumazou, “A review on memristive devices and applications,” in _2010 17th IEEE International Conference on Electronics, Circuits and Systems_. IEEE, 2010, pp. 934–937.
  * [18] P. Sheridan and W. Lu, “Memristors and memristive devices for neuromorphic computing,” in _Handbook of Memristor Networks_. Springer, 2019, pp. 369–389.
  * [19] G. Milano, S. Porro, I. Valov, and C. Ricciardi, “Nanowire memristors: Recent developments and perspectives for memristive devices based on metal oxide nanowires (adv. electron. mater. 9/2019),” _Advanced Electronic Materials_ , vol. 5, no. 9, p. 1970044, 2019.
  * [20] J. Zhu, T. Zhang, Y. Yang, and R. Huang, “A comprehensive review on emerging artificial neuromorphic devices,” _Applied Physics Reviews_ , vol. 7, no. 1, p. 011312, 2020.
  * [21] L. Ni, Z. Liu, H. Yu, and R. V. Joshi, “An Energy-Efficient Digital ReRAM-Crossbar-Based CNN With Bitwise Parallelism,” _IEEE Journal on Exploratory Solid-State Computational Devices and Circuits_ , vol. 3, pp. 37–46, dec 2017. [Online]. Available: https://doi.org/10.1109%2Fjxcdc.2017.2697910
  * [22] A. Sebastian, M. L. Gallo, R. Khaddam-Aljameh, and E. Eleftheriou, “Memory devices and applications for in-memory computing,” _Nature Nanotechnology_ , mar 2020. [Online]. Available: https://doi.org/10.1038%2Fs41565-020-0655-z
  * [23] E. Gale, “Non-ideal memristors for a non-ideal world,” _physica status solidi (a)_ , vol. 212, no. 2, pp. 229–238, 2015.
  * [24] Y. Jeong, M. A. Zidan, and W. D. Lu, “Parasitic effect analysis in memristor-array-based neuromorphic systems,” _IEEE Transactions on Nanotechnology_ , vol. 17, no. 1, pp. 184–193, 2017.
  * [25] O. Krestinskaya, A. Irmanova, and A. P. James, “Memristive non-idealities: Is there any practical implications for designing neural network chips?” in _2019 IEEE International Symposium on Circuits and Systems (ISCAS)_. IEEE, 2019, pp. 1–5.
  * [26] I. Vourkas, D. Stathis, G. C. Sirakoulis, and S. Hamdioui, “Alternative architectures toward reliable memristive crossbar memories,” _IEEE Transactions on Very Large Scale Integration (VLSI) Systems_ , vol. 24, no. 1, pp. 206–217, 2015.
  * [27] L. Chua, G. C. Sirakoulis, and A. Adamatzky, _Handbook of Memristor Networks_. Springer Nature, 2019.
  * [28] S. Zhang, G. L. Zhang, B. Li, H. H. Li, and U. Schlichtmann, “Aging-aware Lifetime Enhancement for Memristor-based Neuromorphic Computing,” in _2019 Design Automation & Test in Europe Conference & Exhibition (DATE)_. IEEE, mar 2019. [Online]. Available: https://doi.org/10.23919%2Fdate.2019.8714954
  * [29] J. Bill and R. Legenstein, “A compound memristive synapse model for statistical learning through stdp in spiking neural networks,” _Frontiers in neuroscience_ , vol. 8, p. 412, 2014.
  * [30] I. Boybat, M. Le Gallo, S. Nandakumar, T. Moraitis, T. Parnell, T. Tuma, B. Rajendran, Y. Leblebici, A. Sebastian, and E. Eleftheriou, “Neuromorphic computing with multi-memristive synapses,” _Nature communications_ , vol. 9, no. 1, pp. 1–12, 2018.
  * [31] I. Boybat, C. Giovinazzo, E. Shahrabi, I. Krawczuk, I. Giannopoulos, C. Piveteau, M. Le Gallo, C. Ricciardi, A. Sebastian, E. Eleftheriou _et al._ , “Multi-reram synapses for artificial neural network training,” in _2019 IEEE International Symposium on Circuits and Systems (ISCAS)_. IEEE, 2019, pp. 1–5.
  * [32] E. Miranda and J. SuÃ±Ã©, “Memristors for Neuromorphic Circuits and Artificial Intelligence Applications.” _Materials (Basel)_ , vol. 13, Feb 2020.
  * [33] M. Hu, C. Graves, C. Li, Y. Li, N. Ge, E. Montgomery, N. Davila, H. Jiang, R. Williams, J. Yang, Q. Xia, and J. Strachan, “Memristor-Based Analog Computation and Neural Network Classification with a Dot Product Engine.” _Adv Mater_ , vol. 30, Mar 2018.
  * [34] S. Kim, H. Kim, S. Hwang, M. Kim, Y. Chang, and B. Park, “Analog Synaptic Behavior of a Silicon Nitride Memristor.” _ACS Appl Mater Interfaces_ , vol. 9, pp. 40 420–40 427, Nov 2017.
  * [35] P. Sheridan, F. Cai, C. Du, W. Ma, Z. Zhang, and W. Lu, “Sparse coding with memristor networks.” _Nat Nanotechnol_ , vol. 12, pp. 784–789, Aug 2017\.
  * [36] T. S. Ngoc, P. K. Van, W. Yang, A. Jo, M. Lee, H. Mo, and K. Min, “Time-Shared Twin Memristor Crossbar Reducing the Number of Arrays by Half for Pattern Recognition.” _Nanoscale Res Lett_ , vol. 12, p. 205, Dec 2017\.
  * [37] M. Azghadi, B. Linares-Barranco, D. Abbott, and P. Leong, “A Hybrid CMOS-Memristor Neuromorphic Synapse.” _IEEE Trans Biomed Circuits Syst_ , vol. 11, pp. 434–445, Apr 2017.
  * [38] K. Kim, J. Zhang, C. Graves, J. Yang, B. Choi, C. Hwang, Z. Li, and R. Williams, “Low-Power, Self-Rectifying, and Forming-Free Memristor with an Asymmetric Programing Voltage for a High-Density Crossbar Application.” _Nano Lett_ , vol. 16, pp. 6724–6732, Nov 2016.
  * [39] M. Zidan, H. Omran, R. Naous, A. Sultan, H. Fahmy, W. Lu, and K. Salama, “Single-Readout High-Density Memristor Crossbar.” _Sci Rep_ , vol. 6, p. 18863, Jan 2016.
  * [40] S. Truong, S. Shin, S. Byeon, J. Song, H. Mo, and K. Min, “Comparative Study on Statistical-Variation Tolerance Between Complementary Crossbar and Twin Crossbar of Binary Nano-scale Memristors for Pattern Recognition.” _Nanoscale Res Lett_ , vol. 10, p. 405, Dec 2015.
  * [41] J. Chen, J. Li, Y. Li, and X. Miao, “Multiply accumulate operations in memristor crossbar arrays for analog computing,” _Journal of Semiconductors_ , vol. 42, no. 1, p. 013104, 2021.
  * [42] M. Le Gallo, A. Sebastian, R. Mathis, M. Manica, H. Giefers, T. Tuma, C. Bekas, A. Curioni, and E. Eleftheriou, “Mixed-precision in-memory computing,” _Nature Electronics_ , vol. 1, no. 4, pp. 246–253, 2018.
  * [43] C. Mohan, J. M. de la Rosa, E. Vianello, L. Perniola, C. Reita, B. Linares-Barranco, and T. Serrano-Gotarredona, “A Current Attenuator for Efficient Memristive Crossbars Read-Out,” in _2019 IEEE International Symposium on Circuits and Systems (ISCAS)_. IEEE, may 2019. [Online]. Available: https://doi.org/10.1109%2Fiscas.2019.8702604
  * [44] Y. Cassuto, S. Kvatinsky, and E. Yaakobi, “Sneak-path constraints in memristor crossbar arrays,” in _2013 IEEE International Symposium on Information Theory_. IEEE, jul 2013. [Online]. Available: https://doi.org/10.1109%2Fisit.2013.6620207
  * [45] L. Shi, G. Zheng, B. Tian, B. Dkhil, and C. Duan, “Research progress on solutions to the sneak path issue in memristor crossbar arrays,” _Nanoscale Advances_ , vol. 2, no. 5, pp. 1811–1827, 2020. [Online]. Available: https://doi.org/10.1039%2Fd0na00100g
  * [46] F. GÃ¼l, “Addressing the sneak-path problem in crossbar RRAM devices using memristor-based one Schottky diode-one resistor array,” _Results in Physics_ , vol. 12, pp. 1091–1096, mar 2019. [Online]. Available: https://doi.org/10.1016%2Fj.rinp.2018.12.092
  * [47] Y. Cassuto, S. Kvatinsky, and E. Yaakobi, “Information-Theoretic Sneak-Path Mitigation in Memristor Crossbar Arrays,” _IEEE Transactions on Information Theory_ , vol. 62, no. 9, pp. 4801–4813, sep 2016. [Online]. Available: https://doi.org/10.1109%2Ftit.2016.2594798
  * [48] R. Naous, M. A. Zidan, A. Sultan-Salem, and K. N. Salama, “Memristor based crossbar memory array sneak path estimation,” in _2014 14th International Workshop on Cellular Nanoscale Networks and their Applications (CNNA)_. IEEE, jul 2014. [Online]. Available: https://doi.org/10.1109%2Fcnna.2014.6888656
  * [49] S. Kannan, R. Karri, and O. Sinanoglu, “Sneak path testing and fault modeling for multilevel memristor-based memories,” in _2013 IEEE 31st International Conference on Computer Design (ICCD)_. IEEE, oct 2013. [Online]. Available: https://doi.org/10.1109%2Ficcd.2013.6657045
  * [50] M. Shevgoor, N. Muralimanohar, R. Balasubramonian, and Y. Jeon, “Improving memristor memory with sneak current sharing,” in _2015 33rd IEEE International Conference on Computer Design (ICCD)_. IEEE, oct 2015. [Online]. Available: https://doi.org/10.1109%2Ficcd.2015.7357164
  * [51] D. J. Mountain, M. R. McLean, and C. D. Krieger, “Memristor Crossbar Tiles in a Flexible General Purpose Neural Processor,” _IEEE Journal on Emerging and Selected Topics in Circuits and Systems_ , vol. 8, no. 1, pp. 137–145, mar 2018. [Online]. Available: https://doi.org/10.1109%2Fjetcas.2017.2767024
  * [52] D. Mikhailenko, C. Liyanagedera, A. P. James, and K. Roy, “M2CA: Modular Memristive Crossbar Arrays,” in _2018 IEEE International Symposium on Circuits and Systems (ISCAS)_. IEEE, 2018. [Online]. Available: https://doi.org/10.1109%2Fiscas.2018.8351112
  * [53] D. J. Mannion, A. Mehonic, W. H. Ng, and A. J. Kenyon, “Memristor-Based Edge Detection for Spike Encoded Pixels,” _Frontiers in Neuroscience_ , vol. 13, jan 2020. [Online]. Available: https://doi.org/10.3389%2Ffnins.2019.01386
  * [54] A. Irmanova, A. Maan, A. James, and L. Chua, “Analog self-timed programming circuits for aging memristors,” _IEEE Transactions on Circuits and Systems II: Express Briefs_ , 2020.
  * [55] O. Krestinskaya, B. Choubey, and A. James, “Memristive gan in analog,” _Scientific Reports_ , vol. 10, no. 1, pp. 1–14, 2020.
  * [56] P. Jain, U. Arslan, M. Sekhar, B. C. Lin, L. Wei, T. Sahu, J. Alzate-vinasco, A. Vangapaty, M. Meterelliyoz, N. Strutt _et al._ , “13.2 a 3.6 mb 10.1 mb/mm 2 embedded non-volatile reram macro in 22nm finfet technology with adaptive forming/set/reset schemes yielding down to 0.5 v with sensing time of 5ns at 0.7 v,” in _2019 IEEE International Solid-State Circuits Conference-(ISSCC)_. IEEE, 2019, pp. 212–214.
  * [57] “The generalized metastable switch memristor model.” [Online]. Available: https://knowm.org/the-generalized-metastable-switch-memristor-model/
  * [58] S. Wiefels, C. Bengel, N. Kopperberg, K. Zhang, R. Waser, and S. Menzel, “Hrs instability in oxide-based bipolar resistive switching cells,” _IEEE Transactions on Electron Devices_ , vol. 67, no. 10, pp. 4208–4215, 2020.
  * [59] F. M. Puglisi, N. Zagni, L. Larcher, and P. Pavan, “Random telegraph noise in resistive random access memories: Compact modeling and advanced circuit design,” _IEEE Transactions on Electron Devices_ , vol. 65, no. 7, pp. 2964–2972, 2018.
  * [60] O. Krestinskaya and A. James, “Analogue neuro-memristive convolutional dropout,” _Proc. R. Soc. A_ , 2020.