# In-Memory Implementation of Dynamic Neural Network Using Resistive Memory

Yue Zhang  Department of Electrical and Electronic Engineering, the University
of Hong Kong, Hong Kong, China  ACCESS – AI Chip Center for Emerging Smart
Systems, InnoHK Centers, Hong Kong Science Park, Hong Kong, China  Institute
of the Mind, the University of Hong Kong, Hong Kong, China  Woyu Zhang  Key
Laboratory of Microelectronic Devices & Integrated Technology, Institute of
Microelectronics, Chinese Academy of Sciences, Beijing 100049, China  Key Lab
of Fabrication technologies for Integrated Circuits, Chinese Academy of
Sciences, Beijing 100049, China  University of Chinese Academy of Sciences,
Beijing 100049, China  Shaocong Wang  Department of Electrical and Electronic
Engineering, the University of Hong Kong, Hong Kong, China  ACCESS – AI Chip
Center for Emerging Smart Systems, InnoHK Centers, Hong Kong Science Park,
Hong Kong, China  Institute of the Mind, the University of Hong Kong, Hong
Kong, China  Ning Lin  Department of Electrical and Electronic Engineering,
the University of Hong Kong, Hong Kong, China  ACCESS – AI Chip Center for
Emerging Smart Systems, InnoHK Centers, Hong Kong Science Park, Hong Kong,
China  Institute of the Mind, the University of Hong Kong, Hong Kong, China
Yifei Yu  Department of Electrical and Electronic Engineering, the University
of Hong Kong, Hong Kong, China  ACCESS – AI Chip Center for Emerging Smart
Systems, InnoHK Centers, Hong Kong Science Park, Hong Kong, China  Institute
of the Mind, the University of Hong Kong, Hong Kong, China  Yangu He
Department of Electrical and Electronic Engineering, the University of Hong
Kong, Hong Kong, China  ACCESS – AI Chip Center for Emerging Smart Systems,
InnoHK Centers, Hong Kong Science Park, Hong Kong, China  Institute of the
Mind, the University of Hong Kong, Hong Kong, China  Bo Wang  Department of
Electrical and Electronic Engineering, the University of Hong Kong, Hong Kong,
China  ACCESS – AI Chip Center for Emerging Smart Systems, InnoHK Centers,
Hong Kong Science Park, Hong Kong, China  Institute of the Mind, the
University of Hong Kong, Hong Kong, China  Hao Jiang  State Key Laboratory of
Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan
University, Shanghai 200433, China  Peng Lin  College of Computer Science and
Technology, Zhejiang University, Zhejiang, 310027, China  Xiaoxin Xu  Key
Laboratory of Microelectronic Devices & Integrated Technology, Institute of
Microelectronics, Chinese Academy of Sciences, Beijing 100049, China  Key Lab
of Fabrication technologies for Integrated Circuits, Chinese Academy of
Sciences, Beijing 100049, China  Xiaojuan Qi  Department of Electrical and
Electronic Engineering, the University of Hong Kong, Hong Kong, China
Zhongrui Wang  Department of Electrical and Electronic Engineering, the
University of Hong Kong, Hong Kong, China  ACCESS – AI Chip Center for
Emerging Smart Systems, InnoHK Centers, Hong Kong Science Park, Hong Kong,
China  Institute of the Mind, the University of Hong Kong, Hong Kong, China
e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Xumeng Zhang  State Key Laboratory of Integrated Chips and Systems, Frontier
Institute of Chip and System, Fudan University, Shanghai 200433, China
e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Dashan Shang  Key Laboratory of Microelectronic Devices & Integrated
Technology, Institute of Microelectronics, Chinese Academy of Sciences,
Beijing 100049, China  Key Lab of Fabrication technologies for Integrated
Circuits, Chinese Academy of Sciences, Beijing 100049, China  University of
Chinese Academy of Sciences, Beijing 100049, China  e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Qi Liu  Key Laboratory of
Microelectronic Devices & Integrated Technology, Institute of
Microelectronics, Chinese Academy of Sciences, Beijing 100049, China  State
Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and
System, Fudan University, Shanghai 200433, China  Kwang-Ting Cheng  ACCESS –
AI Chip Center for Emerging Smart Systems, InnoHK Centers, Hong Kong Science
Park, Hong Kong, China  Department of Electronic and Computer Engineering, the
Hong Kong University of Science and Technology, Hong Kong, China  Ming Liu
Key Laboratory of Microelectronic Devices & Integrated Technology, Institute
of Microelectronics, Chinese Academy of Sciences, Beijing 100049, China  State
Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and
System, Fudan University, Shanghai 200433, China

# Semantic memory-based dynamic neural network using memristive ternary CIM
and CAM for 2D and 3D vision

Yue Zhang  Department of Electrical and Electronic Engineering, the University
of Hong Kong, Hong Kong, China  ACCESS – AI Chip Center for Emerging Smart
Systems, InnoHK Centers, Hong Kong Science Park, Hong Kong, China  Institute
of the Mind, the University of Hong Kong, Hong Kong, China  Woyu Zhang  Key
Laboratory of Microelectronic Devices & Integrated Technology, Institute of
Microelectronics, Chinese Academy of Sciences, Beijing 100049, China  Key Lab
of Fabrication technologies for Integrated Circuits, Chinese Academy of
Sciences, Beijing 100049, China  University of Chinese Academy of Sciences,
Beijing 100049, China  Shaocong Wang  Department of Electrical and Electronic
Engineering, the University of Hong Kong, Hong Kong, China  ACCESS – AI Chip
Center for Emerging Smart Systems, InnoHK Centers, Hong Kong Science Park,
Hong Kong, China  Institute of the Mind, the University of Hong Kong, Hong
Kong, China  Ning Lin  Department of Electrical and Electronic Engineering,
the University of Hong Kong, Hong Kong, China  ACCESS – AI Chip Center for
Emerging Smart Systems, InnoHK Centers, Hong Kong Science Park, Hong Kong,
China  Institute of the Mind, the University of Hong Kong, Hong Kong, China
Yifei Yu  Department of Electrical and Electronic Engineering, the University
of Hong Kong, Hong Kong, China  ACCESS – AI Chip Center for Emerging Smart
Systems, InnoHK Centers, Hong Kong Science Park, Hong Kong, China  Institute
of the Mind, the University of Hong Kong, Hong Kong, China  Yangu He
Department of Electrical and Electronic Engineering, the University of Hong
Kong, Hong Kong, China  ACCESS – AI Chip Center for Emerging Smart Systems,
InnoHK Centers, Hong Kong Science Park, Hong Kong, China  Institute of the
Mind, the University of Hong Kong, Hong Kong, China  Bo Wang  Department of
Electrical and Electronic Engineering, the University of Hong Kong, Hong Kong,
China  ACCESS – AI Chip Center for Emerging Smart Systems, InnoHK Centers,
Hong Kong Science Park, Hong Kong, China  Institute of the Mind, the
University of Hong Kong, Hong Kong, China  Hao Jiang  State Key Laboratory of
Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan
University, Shanghai 200433, China  Peng Lin  College of Computer Science and
Technology, Zhejiang University, Zhejiang, 310027, China  Xiaoxin Xu  Key
Laboratory of Microelectronic Devices & Integrated Technology, Institute of
Microelectronics, Chinese Academy of Sciences, Beijing 100049, China  Key Lab
of Fabrication technologies for Integrated Circuits, Chinese Academy of
Sciences, Beijing 100049, China  Xiaojuan Qi  Department of Electrical and
Electronic Engineering, the University of Hong Kong, Hong Kong, China
Zhongrui Wang  Department of Electrical and Electronic Engineering, the
University of Hong Kong, Hong Kong, China  ACCESS – AI Chip Center for
Emerging Smart Systems, InnoHK Centers, Hong Kong Science Park, Hong Kong,
China  Institute of the Mind, the University of Hong Kong, Hong Kong, China
e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Xumeng Zhang  State Key Laboratory of Integrated Chips and Systems, Frontier
Institute of Chip and System, Fudan University, Shanghai 200433, China
e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Dashan Shang  Key Laboratory of Microelectronic Devices & Integrated
Technology, Institute of Microelectronics, Chinese Academy of Sciences,
Beijing 100049, China  Key Lab of Fabrication technologies for Integrated
Circuits, Chinese Academy of Sciences, Beijing 100049, China  University of
Chinese Academy of Sciences, Beijing 100049, China  e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Qi Liu  Key Laboratory of
Microelectronic Devices & Integrated Technology, Institute of
Microelectronics, Chinese Academy of Sciences, Beijing 100049, China  State
Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and
System, Fudan University, Shanghai 200433, China  Kwang-Ting Cheng  ACCESS –
AI Chip Center for Emerging Smart Systems, InnoHK Centers, Hong Kong Science
Park, Hong Kong, China  Department of Electronic and Computer Engineering, the
Hong Kong University of Science and Technology, Hong Kong, China  Ming Liu
Key Laboratory of Microelectronic Devices & Integrated Technology, Institute
of Microelectronics, Chinese Academy of Sciences, Beijing 100049, China  State
Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and
System, Fudan University, Shanghai 200433, China

# Dynamic neural network with memristive CIM and CAM for 2D and 3D vision

Yue Zhang  Department of Electrical and Electronic Engineering, the University
of Hong Kong, Hong Kong, China  ACCESS – AI Chip Center for Emerging Smart
Systems, InnoHK Centers, Hong Kong Science Park, Hong Kong, China  Institute
of the Mind, the University of Hong Kong, Hong Kong, China  Woyu Zhang  Key
Laboratory of Microelectronic Devices & Integrated Technology, Institute of
Microelectronics, Chinese Academy of Sciences, Beijing 100049, China  Key Lab
of Fabrication technologies for Integrated Circuits, Chinese Academy of
Sciences, Beijing 100049, China  University of Chinese Academy of Sciences,
Beijing 100049, China  Shaocong Wang  Department of Electrical and Electronic
Engineering, the University of Hong Kong, Hong Kong, China  ACCESS – AI Chip
Center for Emerging Smart Systems, InnoHK Centers, Hong Kong Science Park,
Hong Kong, China  Institute of the Mind, the University of Hong Kong, Hong
Kong, China  Ning Lin  Department of Electrical and Electronic Engineering,
the University of Hong Kong, Hong Kong, China  ACCESS – AI Chip Center for
Emerging Smart Systems, InnoHK Centers, Hong Kong Science Park, Hong Kong,
China  Institute of the Mind, the University of Hong Kong, Hong Kong, China
Yifei Yu  Department of Electrical and Electronic Engineering, the University
of Hong Kong, Hong Kong, China  ACCESS – AI Chip Center for Emerging Smart
Systems, InnoHK Centers, Hong Kong Science Park, Hong Kong, China  Institute
of the Mind, the University of Hong Kong, Hong Kong, China  Yangu He
Department of Electrical and Electronic Engineering, the University of Hong
Kong, Hong Kong, China  ACCESS – AI Chip Center for Emerging Smart Systems,
InnoHK Centers, Hong Kong Science Park, Hong Kong, China  Institute of the
Mind, the University of Hong Kong, Hong Kong, China  Bo Wang  Department of
Electrical and Electronic Engineering, the University of Hong Kong, Hong Kong,
China  ACCESS – AI Chip Center for Emerging Smart Systems, InnoHK Centers,
Hong Kong Science Park, Hong Kong, China  Institute of the Mind, the
University of Hong Kong, Hong Kong, China  Hao Jiang  State Key Laboratory of
Integrated Chips and Systems, Frontier Institute of Chip and System, Fudan
University, Shanghai 200433, China  Peng Lin  College of Computer Science and
Technology, Zhejiang University, Zhejiang, 310027, China  Xiaoxin Xu  Key
Laboratory of Microelectronic Devices & Integrated Technology, Institute of
Microelectronics, Chinese Academy of Sciences, Beijing 100049, China  Key Lab
of Fabrication technologies for Integrated Circuits, Chinese Academy of
Sciences, Beijing 100049, China  Xiaojuan Qi  Department of Electrical and
Electronic Engineering, the University of Hong Kong, Hong Kong, China
Zhongrui Wang  Department of Electrical and Electronic Engineering, the
University of Hong Kong, Hong Kong, China  ACCESS – AI Chip Center for
Emerging Smart Systems, InnoHK Centers, Hong Kong Science Park, Hong Kong,
China  Institute of the Mind, the University of Hong Kong, Hong Kong, China
e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Xumeng Zhang  State Key Laboratory of Integrated Chips and Systems, Frontier
Institute of Chip and System, Fudan University, Shanghai 200433, China
e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Dashan Shang  Key Laboratory of Microelectronic Devices & Integrated
Technology, Institute of Microelectronics, Chinese Academy of Sciences,
Beijing 100049, China  Key Lab of Fabrication technologies for Integrated
Circuits, Chinese Academy of Sciences, Beijing 100049, China  University of
Chinese Academy of Sciences, Beijing 100049, China  e-mail<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>Qi Liu  Key Laboratory of
Microelectronic Devices & Integrated Technology, Institute of
Microelectronics, Chinese Academy of Sciences, Beijing 100049, China  State
Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and
System, Fudan University, Shanghai 200433, China  Kwang-Ting Cheng  ACCESS –
AI Chip Center for Emerging Smart Systems, InnoHK Centers, Hong Kong Science
Park, Hong Kong, China  Department of Electronic and Computer Engineering, the
Hong Kong University of Science and Technology, Hong Kong, China  Ming Liu
Key Laboratory of Microelectronic Devices & Integrated Technology, Institute
of Microelectronics, Chinese Academy of Sciences, Beijing 100049, China  State
Key Laboratory of Integrated Chips and Systems, Frontier Institute of Chip and
System, Fudan University, Shanghai 200433, China

###### Abstract

The brain is dynamic, associative and efficient. It reconfigures by
associating the inputs with past experiences, with fused memory and
processing. In contrast, AI models are static, unable to associate inputs with
past experiences, and run on digital computers with physically separated
memory and processing. We propose a hardware-software co-design, a semantic
memory-based dynamic neural network (DNN) using memristor. The network
associates incoming data with the past experience stored as semantic vectors.
The network and the semantic memory are physically implemented on noise-robust
ternary memristor-based Computing-In-Memory (CIM) and Content-Addressable
Memory (CAM) circuits, respectively. We validate our co-designs, using a 40nm
memristor macro, on ResNet and PointNet++ for classifying images and 3D points
from the MNIST and ModelNet datasets, which not only achieves accuracy on par
with software but also a 48.1% and 15.9% reduction in computational budget.
Moreover, it delivers a 77.6% and 93.3% reduction in energy consumption.

## Introduction

The human brain’s efficiency in executing complex tasks on a remarkably low
energy budget relies on the synergy of dynamic reconfigurability, associative
memory with past experience, and collocation of memory and information
processing.

The brain’s processing of information is dynamic and adaptive. Rather than
maintaining a static computational model, the brain’s neural networks exhibit
complex and dynamic activation and connectivity patterns in response to
various stimuli and tasks. The dynamic connectivity of the brain primarily
hinges on structural plasticity [1, 2, 3]. Moreover, the brain displays
another layer of dynamic behavior; when individuals encounter objects or tasks
of varying complexity, different brain regions are activated [4]. Neural
activity patterns of the brain can vary across different brain regions and
change over time, enabling the brain to integrate sensory information, make
decisions, and adapt to new situations [5]. The dynamic nature of the brain
allows it to process tasks efficiently while consuming minimal power. In
contrast, the majority of neural networks are static with their fixed
topology, lacking such adaptability, leading to inefficient resource
allocation and limitations in handling evolving information or tasks
(Fig.1(a)).

In addition, the brain is capable of associating new information with its past
experience, such as in the realm of semantic and high-level visual processing.
Biological experiments [6] have demonstrated the brain’s aptitude for
accelerating the recognition of familiar objects via memory encoding and
retrieval processes within activated neurons, functioning as a semantic cache
that allows for rapid retrieval of past experience. Traditional computers,
however, rely on address-based information storage and search, rather than
associate observations in terms of similarity (Fig.1(b)).

Moreover, the brain employs chemical synapses not only to store information
but also to process information right where they are stored, featuring low
power consumption and high parallelism. This is contrasted with the high
energy consumption and latency challenges posed by the current von Neumann
architecture digital computers, where the separation between computing and
storage units necessitates frequent data transfers [7] (Fig.1(c)).

Here we propose a hardware-software co-design, semantic memory-based dynamic
neural network (DNN) using memristor, to mirror the 3 characteristics of the
brain computing paradigm.

Software-wise, semantic memory-based DNN aims to equip artificial neural
networks with the dynamic reconfigurability of the brain [8](Fig.1(a)). The
network associates the new information with its past experience and allocates
computations based on demand, more efficient and effective than static
networks in many applications [9, 10, 11, 12]. Also, the network’s
adaptability allows it to balance accuracy and efficiency by adjusting to
varying computational budgets in real-time [13, 14, 15, 16, 17, 18], for
scenarios where computational budgets may change or vary, such as in resource-
constrained devices or distributed computing environments [19].

Hardware-wise, we employ memristor, an emerging device that physically
resembles the synapses of the brain, to physically implement the ternary
weights of noise-robust dynamic neural network and the associative semantic
memory, using computing-in-memory (CIM) and content-addressable memory (CAM)
circuits, respectively.

Memristor-based CIM circuits use simple physical laws for matrix-vector
multiplications. As matrix weights are physically stored in a memristor array
and multiplication is performed right at where the weight is stored, they
amalgamate computing and memory elements, which not only overcomes the von
Neumann bottleneck but also leads to high parallelism [20, 21, 22, 23, 24, 25,
26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40] (Fig.1(c)).

Memristor-based CAM circuit is an associative memory inspired by the brain
(Fig.1(b)), functioning as semantic cache memories in the dynamic neural
networks [41]. Unlike traditional electronic memory that relies on addresses
for information retrieval, the brain and CAM feature for parallel content-
based searches [42, 43, 44, 45, 46, 47, 36]. Specifically, CAM measures the
distances between input vector and stored vectors within the memory,
eliminating the need for data movement like the CIM [48, 49, 50, 47, 51, 52].

In this article, we validate our co-design using a 40nm memristor macro on
representative image and 3D points classification tasks. Using ResNet [53] and
PointNet++ [54], we showcase notable reductions in computational budget,
achieving a 48.1% reduction for classifying the MNIST [55] dataset and a 15.9%
reduction for the ModelNet [56] dataset by semantic memory-based DNN.
Moreover, we observe substantial improvements in energy efficiency, resulting
in a 77.6% and 93.3% reduction in energy consumption for the classification of
respective datasets when compared to a state-of-the-art graphic processing
unit (GPU). These findings open up avenues for future research on emulating
the brain’s adaptive allocation of computational resources, associative recall
of past experience, and collocation of memory and processing.

Figure 1: Brain-inspired dynamic neural network with memristors. a, Comparison
of the computing model of the brain, static network and dynamic network. b,
Comparison of the associative memory mechanism in the brain, digital hardware
and memristor-based CAM. c, Comparison of the computing architecture of the
brain, digital hardware, and memristor-based CIM.

## Results

### Hardware–software co-design: Semantic memory-based DNN using memristor-
based CIM and CAM

Fig. 2 illustrates the semantic memory-based DNN using memristor-based CIM and
CAM. The DNN optimizes resource allocation and speeds up the forward inference
with an early-exiting mechanism according to the semantic memory [57]. The
early-exiting dynamically adjusts the number of layers per forward inference,
contingent upon the complexity of the input sample, thereby creating a more
adaptable network.

As illustrated in Fig.2(a-b), we use ternary weights (-1, 0, 1) in the network
instead of full-precision floating-point numbers (as to be discussed later,
ternary quantization outperforms full-precision weights subject to analogue
computing noise). Our work uses pre-trained backbone network without training
the exits, and can exit from any of the layers before the final layer during
inference [57] (see Supplementary Note 1 and Supplementary Table 1 for
comparison of early exit work). To compute semantic memory, we infer samples
from the training set and apply global average pooling (GAP) to the
intermediate layers, converting the feature maps into one-dimensional semantic
vectors (see Supplementary Note 2 for global average pooling). These semantic
vectors represent the semantic centers for each respective category. Semantic
centers undergo ternary quantization and are stored in the CAM. Upon the query
of a new test-set sample, the network calculates associated search vectors
according to each layer’s output feature map and compares them to the cached
semantic centers in the CAM (see Supplementary Note 3 for operation of CAM).
As such, each semantic memory is able to make a classification based on the
cosine similarity between a search vector and the stored semantic centers, at
different confidence.

For relatively simple input samples, the brain can react quickly without the
need for deep-level thinking. Similarly, as shown in Fig.2(a), the forward
propagation terminates at a shallow layer (e.g. layer l) with a clearly
recognizable cat image. This is because the semantic vector of layer l, when
compared to the semantic center vectors in the CAM, yields a sufficiently
large confidence in its prediction. The network thus classifies the input
accordingly, bypassing the remaining layers and thus enhancing efficiency. For
more complex input samples, as shown in Fig.2(b), the forward pass goes to
deeper layers to capture high-level features for accurate classification. As
depicted in the Fig.2(a-b), the computational budget increases with the number
of layers in the forward inference, and contributes to the overall
computational cost. Therefore, this early exit approach improves efficiency by
avoiding unnecessary computations and streamlining the inference process.

We physically implement the DNN and semantic memory using a memristor macro.
Fig.2(d) shows the marco integrated onto a printed circuit board along with a
Xilinx ZYNQ system-on-chip, forming a hybrid analogue-digital computing
platform (see Supplementary Figure 1 for the system design and photo). The
analogue CIM and CAM cores consist of CMOS-compatible nanoscale
TaN/$\mathrm{TaO}_{\mathrm{x}}$/Ta/TiN memristors (Fig.2(e-f)) fabricated
using the backend-of-line process on a 40 nm technology node tape-out
(Fig.2d). As shown in Fig.2(a-c), the ternary weights of the neural network
are encoded into the memristive conductances in CIM, and CAM stores the
semantic centers. The input samples are then quantized and mapped to voltages
applied to CIM, with the output currents representing output feature maps. The
feature maps are then digitized, activated, and pooled in the digital core
before being fed to the next layer. The feature maps also undergo GAP,
yielding the search vector. The cosine distances between the search vector (as
voltage input to the CAM) and the semantic centers stored in the CAM is
revealed by the match line signals(e.g. currents here). The currents are then
digitized to determine the confidence level associated with each semantic
center.

Figure 2: Hardware-software co-design: Semantic memory-based dynamic neural
network using CIM and CAM. a, The proposed architecture consists of a ternary
quantized neural network implemented on memristor-based CIM for feature
extraction, and an associative memory on memristor-based CAM. Based on the
global average pooling (GAP), the network encodes extracted feature maps to
low-dimensional ternary semantic center in memory. When a new sample is
subsequently queried, the network calculates the semantic vector on each
layer’s output feature map and matches them with the cached semantic centers
of n classes in memory. The search vector finds the semantic center stored in
CAM with maximum cosine similarity, which can be used to predict the class of
the query. Once well matched, the network skips the rest of the layers and
directly outputs the final results. In this example, an easy sample of a cat
can be well classified through early layers with few computing resources and
the subsequent layers will not participate in the inference. b, If the input
is a hard sample, and the early layers fail to provide reliable predictions,
classification for this sample requires a deeper network with more
computation. c, The feature extraction using a memristor-based CIM circuit. d,
Optical photo of the memristor chip. e, A cross-sectional transmission
electron micrograph showcases the memristor crossbar array, fabricated by the
backend-of-line process on a 40 nm technology node tape-out. f, The cross-
sectional transmission electron micrograph reveals a solitary nanoscale
memristor, operating on the formation and rupture of conducting filaments.

### Dynamic ResNet for 2D vision

First, we applied our co-design to classify the MNIST dataset [55]using the
ResNet model [53]. Fig.3(a) shows an experimental example of the forward
inference on the memristor-based hybrid analogue-digital system. The model and
the semantic memory were first ex situ trained, before being quantized and
physically mapped to memristor-based CIM and CAM (see Method). During
inference, a handwritten image (e.g. digit eight) is mapped to voltages and
the CIM physically computes the feature maps of each residual block. These
feature maps produce associated search vector of each residual block after
GAP, for being compared with semantic centers of the same block in CAM by
measuring cosine similarities. Upon the forward pass reaching the fourth
block, the calculated similarity for the eighth class surpasses the threshold,
indicating a sufficient confidence level to classify the input handwritten
digit as eight. The forward pass thus stops here and all remaining residual
blocks are skipped.

To benchmark the classification performance of semantic centers of different
residual blocks, intra-class distance and inter-class distance [58]are
measured, as shown in Fig.3(b-d). T-distributed stochastic neighbour embedding
(t-SNE) dimension reduction [59] visualizes the distribution of search vectors
(small labels, voltages applied to CAM) of 100 randomly selected test samples
and semantic centers (large labels, stored in CAM) from the second, fifth, and
nineth residual blocks. It is observed that samples possess significantly
different sample-center distances, qualifying the demand of a dynamic and
adaptive network. In addition, different residual block develops different
classification capability, necessitating unique threshold adjustments for each
layer to guarantee optimal performance. Here we optimize the thresholds for
semantic memory of each residual block with Tree-structured Parzen Estimator
(TPE) [60, 61] as to be discussed later.

Fig.3(e) shows the ablation and comparison studies. Software static full-
precision (SFP) and ternary quantized (Qun) models are of accuracy 98.0% and
96.5%, respectively, which slightly reduces to 97.5% (EE) and 96.0% (EE. Qun)
with the dynamic early-exit. Taking into account analogue memristor noise, the
simulated ternary quantized dynamic model (EE. Qun+Noise) shows an accuracy
96.1%, consistent with our experimental observation (Mem) 96.0%, accompanied
with a 48.1% reduction of computational budget (see Supplementary Note 4 for
performance comparison with LeNet). Fig.3(f) shows the experimentally acquired
confusion matrix, in which the prevalence of diagonal elements corroborates
the high classification accuracy. Fig.3(g) shows the computational budget of
inference samples at each residual block and the probabilities of samples
passing through each block (see Supplementary Note 5 for computational budget
breakdown). It is observed that most samples only need to go through the first
four residual blocks while difficult samples propagate to deep residual
blocks, which significantly reduces the overall computational budget without
compromising the performance.

Fig.3(h) shows the energy consumption breakdown of the randomly selected 100
inference samples (see Supplementary Note 6 for operating power and speed
estimation). The light (dark) grey bar is the estimated energy consumption of
a state-of-the-art GPU running the static (dynamic) ResNet model, consuming
$1.83\text{\times}{10}^{7}$ pJ ($9.19\text{\times}{10}^{6}$ pJ) (see
Supplementary Note 7 for energy consumption modelling). The dynamic model
saves about 49.8% energy due to the decreased network connectivity. For the
hybrid analogue-digital system, the blue, green and red bars show energy
consumption of memristor ($1.21\text{\times}{10}^{4}$ pJ for CIM and $77.1$ pJ
for CAM, see Supplementary Table 2 and Table 3 for energy efficiency of CAM
and CIM, respectively), analogue to digital conversion
($1.57\text{\times}{10}^{6}$ pJ for CIM and $4.55\text{\times}{10}^{4}$ pJ for
CAM), and digital peripherals ($3.73\text{\times}{10}^{5}$ pJ for activation
and pooling, while $6.63\text{\times}{10}^{4}$ pJ for sorting). The overall
energy consumption of DNN on the projected memristor-based hybrid
analogue–digital system is approximately $2.06\text{\times}{10}^{6}$ pJ, a
77.6% reduction of the energy consumption compared to software dynamic ResNet
due to in-memory computing. Furthermore, compared to static ResNet running on
GPU, our co-design achieves a 8.9-fold enhancement in inference energy
efficiency (see Supplementary Note 8 for energy breakdown in ResNet).

Figure 3: Hardware-software co-designed dynamic ResNet for MNIST dataset
classification. a, Schematic representation of CIM and CAM based dynamic
neural network on ResNet. The network extracts feature maps of each residual
block, then the feature maps are encoded into semantic vectors (svs) through
Global Average Pooling (GAP). Semantic vectors work as keys to lookup
corresponding semantic center (sc) by measuring cosine similarities (sim).
b-d, Visualized distribution of semantic vectors (smaller number) and semantic
centers (bigger number) for the second, fifth and ninth residual block. e, The
accuracy and budget drop of different model-quantization-noise combinations
for ablation study, including static full-precision (SFP), ternary
quantization (Qun), early exit (EE), early exit with ternary quantization
(EE.Qun), early exit with ternary quantization and noise (EE.Qun+noise), and
memristor-based hardware (Mem) experiment. f, Normalized confusion matrix of
memristor-based hardware experiment. g, Operations (OPS) per layer and the
probability of input passing through of each layer. h, Comparison of the
inference energy consumption across GPU and memristor-based CIM and CAM in a
hybrid analogue–digital system.

### Ternary quantization for analogue noise suppression

Noise suppression is critical for analogue computing with memristor. There are
two major noises in our co-design, the write and read noise. The write noise
originates from inevitable programming stochasticity with memristor, while the
read noise roots on temporal fluctuations of conductance due to the combined
effect of programming instability and other electronic noises [62, 63, 64].

Fig.4(a) illustrates the two noises. The conductance of five randomly selected
memristors programmed under the same condition are continuously sampled 10,000
times. The conductance of each memristor over time follows a quasi-normal
distribution, with unique mean and standard deviation. This is also shown in
Fig.4(b), the mean conductance map of a memristor array programmed under the
same condition, and Fig.4(c), the map of conductance standard deviation over
10,000 reading cycles. Fig.4(d) shows the correlation between mean conductance
and the standard deviation. The mean conductance of memristor reflects the
write noise, while the standard deviation reflects the read noise. Fig.4(e)
shows the histogram of Fig. 4(b), where the mean conductance of different
memristors follows a quasi-normal distribution and 15% write noise. The impact
of noise on CIM is shown in Fig.4(f), where final result of the computation is
plotted against the exact result. Such noise also impacts CAM in a similar
manner as examplified by Fig.4(g).

To mitigate the write and read noise influence over the memristor-based CIM
and CAM, we adopt ternary quantization of weights and semantic centers (see
Methods), which outperforms mapping full-precision models subject to the write
and read noise. We simulate write noise impact on the classification accuracy
of dynamic ResNet with and without ternary weight quantization. As shown in
Fig.4(h), the ternary quantized network exhibits considerable resilience
against increasing write noise, while the full-precision mapped network
performance quickly degrades with increasing programming noise. Furthermore,
we simulate read noise impact in conjunction with a fixed 15% write noise. As
shown in Fig.4(i), the ternary quantized network shows a 10% accuracy
improvement over direct mapping full-precision weights to memristors.

Figure 4: The intrinsic physical noise of memristor and the mitigation using
ternary quantization network a, Conductance variance and statistical histogram
of 5 randomly selected memristors with 10,000 read cycles. b-c, Map of the
mean and standard deviation of conductance for 8,930 memristors with 10,000
read cycles. d, Scatter plot of the mean and standard deviation of the
conductance values of 8,930 memristors. e, The histogram of mean conductance
in b. f, Noisy CIM results plotted against noise-free CIM results. The red
line represents the ideal case with matched experimental values, while the
blue points represent the observed results. g, Write noise map for the value
stored in CAM. h, Comparison of accuracy with write noise between the non-
quantized network and the ternary quantized network. i, Comparison of accuracy
with write and read noise between the non-quantized network and the ternary
quantized network.

### Dynamic PointNet++ for 3D vision

Despite 2D images, the advent of 3D sensing technologies has underscored the
significance of 3D point clouds in capturing and interpreting the spatial
attributes of 3D objects. The classification of 3D point clouds is
instrumental in autonomous driving, robotics, and augmented reality. Here, we
simulate our co-design using PointNet++ [54] model to classify ModelNet [56]
dataset. The PointNet++ contains multiple set abstraction layers. These layers
work in a hierarchical manner, each layer selects representative points,
transforms features of all points neighboring to the representative points,
and aggregates the neighboring point features to update features of
representative points (see Methods for the details of the PointNet++). In this
way, the PointNet++ captures both local and global features in a point cloud
at different depth of the model, widely used for 3D points classification and
segmentation. For better illustration, We randomly select ten categories from
the ModelNet dataset. Fig.5(a) uses a chair as an example, the point features
are transformed to voltages and then applied to CIM for feature
transformation. The search vectors are calculated for each set abstraction
layer and matched with the semantic centers stored in CAM. The system gains
enough confidence about the matching results upon forward passing to the fifth
set abstraction layer, triggering an early exit and bypassing the remaining
layers.

Fig.5(b-d) uses t-SNE [59] to visualize the distribution of the search vectors
of 100 randomly selected test samples and semantic centers. The second
(Fig.5(b)), the forth (Fig.5(c)) and the sixth (Fig.5(d)) set abstraction
layer again shows inference samples are of diverse difficulty and the model
develops increasing discrimination capability over the depth.

We also conduct ablation and comparison experiments in Fig.5(e). The software
static model accuracies are 89.1% and 82.2% with full-precision and ternary
quantized weights, respectively. For dynamic models, the corresponding
accuracies are 83.8% and 80.4%. The noise slightly reduces the accuracy of
dynamic model with ternary weights to 79.2%, while offering 15.9% reduction in
the computational budget (see Supplementary Note 9 for model size impact on
computational saving and accuracy trade-off). Fig.5(f) is the simulated
confusion matrix. The accuracy of categories marked as 3, 4, and 6 is
relatively lower, consistent with Fig.5(b-d) where the search vectors labeled
as 3, 4, and 6 cannot be completely segregated. Fig.5(g) shows the
computational budget of input samples at each layer and the probabilities of
samples passing through that layer, where the majority of samples exit early
by layer 5 (see Supplementary Note 5 for computational budget breakdown).

Fig.5(h) shows the estimated energy consumption breakdown in inferring samples
from 10 randomly selected classes. Software static and dynamic PointNet++ on a
state-of-the-art GPU (light grey bar) consumes $4.34\text{\times}{10}^{12}$ pJ
and $3.65\text{\times}{10}^{12}$ pJ, respectively. The reduction in power
consumption can be attributed to decreased network connections and a lower
number of operations. For memristor-based hybrid analogue-digital system, the
projected energy consumption of memristor, analogue-digital conversion, and
digital peripherals are $6.35\text{\times}{10}^{9}$ pJ,
$1.34\text{\times}{10}^{11}$ pJ, and $1.53\text{\times}{10}^{11}$ pJ
($2.67\text{\times}{10}^{4}$ pJ, $7.03\text{\times}{10}^{5}$ pJ, and
$1.97\text{\times}{10}^{7}$ pJ) for the CIM (CAM). The overall energy
consumption of our co-design is approximately $2.90\text{\times}{10}^{11}$ pJ,
which leads to a 93.3% reduction when compared to the static PointNet++ in
software. The reduction is due to both in-memory computing and DNN (see
Supplementary Note 8 for energy breakdown in PointNet++).

Figure 5: CIM and CAM based DNN for ModelNet classification using PointNet++.
a, Schematic of CIM and CAM based DNN for PointNet++. The input is a 3D point
cloud of a chair, the network extracts a 3D feature map of each set
abstraction layer, then the feature maps are encoded into semantic vectors
(svs) through Global Average Pooling (GAP). Semantic vectors work as keys to
lookup corresponding semantic center (sc) by measuring cosine
similarities(sim). b-d, Visualized distribution of semantic vectors (smaller
numbers) and semantic centers (bigger numbers) for the second, forth and sixth
layer. e, The accuracy and budget drop of different model-quantization-noise
combinations for ablation study, including static full-precision (SFP),
ternary quantization (Qun), early exit (EE), early exit with ternary
quantization (EE.Qun), early exit with ternary quantization and noise
(EE.Qun+noise). f, Normalized confusion matrix of memristor-based hardware
experiment. g, Operations (OPS) per layer and the probability of input passing
through of each layer. h, Comparison of the inference energy consumption
across GPU and memristor-based CIM and CAM in a hybrid analogue–digital
system.

### Solving optimization problem on trading off between budget and accuracy
using TPE

The threshold of each layer plays a critical role in determining whether an
input sample can exit early for computational budget reduction, or keep on
forward passing for better accuracy, which balance between accuracy and
efficiency. The objective here is to find a Pareto-optimal solution that
strikes a trade-off between computational budget and accuracy for the DNN. We
first use grid search to explore the correlation between computational budget
and accuracy. Then we design a target function that incorporates both
computational budget and accuracy, and use the method of TPE [60, 61] to
optimize the threshold of each layer. The optimal solution for the target
function yields a DNN that balances computational budget and accuracy.

Fig.6(a) shows the grid search results for the thresholds for the co-design
using ResNet, which illustrates the trade-off between computational budget and
accuracy. Based on that, we devise a target function depends on both
computational budget and accuracy.

$\max_{\mathrm{dm}}\quad\mathrm{Acc\left(dm\right)}\times\mathrm{\left[\frac{DCB}{B}\right]}^{\omega}$
(1)

Here, the target budget drop $\mathrm{B=0.50}$, Acc(dm) is the classification
accuracy of the DNN, DCB is the drop of computational budget. The weight
factor, $\omega$, is empirically selected to ensure that Pareto-optimal
solutions yield similar rewards under different accuracy-budget trade-offs. We
observe in grid search that an additional 1% increase in accuracy corresponds
to an approximate 4.35% increase in computation budget when accuracy is beyond
94%. Consequently, $\omega$ is set to 0.127. For convenience, we solve the
dual problem. Fig.6(b) shows the negative objective function which decreases
with the budget drop or increase of accuracy (see Method) , which is also
visualized in Fig.6(c) as a curved surface.

We minimize the negative objective function by identifying the best thresholds
of each layer. As TPE method does not model the interaction between
thresholds, we use that to solve the Pareto-optimal problem. The first step of
TPE is to initialize sampling through random search. We denote the accuracy
and budget drop as $\mathbf{x}$ and the calculated score as $y$, the observed
results${\\{(\mathbf{x}^{(1)},y^{(1)}),...,(\mathbf{x}^{(k)},y^{(k)})\\}}$ are
then divided into two groups: the group with good (poor) performance, marked
as green dot (yellow triangle) in Fig.6(c)), and the splitting value for
dividing the two groups is $score^{*}$. TPE defines the probability
distribution $p(\mathbf{x}|y)$ using the following two probability density
functions:

$p(\mathbf{x}|y)=\left\\{\begin{aligned} l(\mathbf{x})\quad\mathrm{if\ y\geq
y^{*}}\\\ g(\mathbf{x})\quad\mathrm{if\ y^{*}>y}\\\ \end{aligned}\right.$ (2)

Fig.6(f) shows the density function of good samples (purple 3D surface) and
the density function of bad samples (blue 3D surface) which are modeled by
Parzen estimators (see Methods for a detailed description) [60, 61]. TPE
utilizes the following expected improvement (EI) function as an acquisition
function:

$\begin{split}\mathrm{EI}_{y^{*}}(\mathbf{x})&=\int_{-\infty}^{y^{*}}(y^{*}-y)p(y|\mathbf{x})dy\\\
&=\int_{-\infty}^{y^{*}}(y^{*}-y)\frac{p(\mathbf{x}|y)p(y)}{p(\mathbf{x})}dy\\\
&=\frac{\gamma
y^{*}l(\mathbf{x})-l(\mathbf{x})\int_{-\infty}^{y^{*}}p(y)dy}{\gamma
l(\mathbf{x})+(1-\gamma)g(\mathbf{x})}\propto(\gamma+\frac{g(\mathbf{x})}{l(\mathbf{x})}(1-\gamma))^{-1}\end{split}$
(3)

where $\gamma$ is a hyperparameter. The last expression is obtained through
Bayes’ rule and is proportional to $l(\mathbf{x})/g(\mathbf{x})$ (see Methods
for a detailed derivation of the EI equation). Just like Bayesian optimization
[65, 66, 67], the objective is to find the thresholds which increase the
probability of $l(\mathbf{x})$ while decreasing the probability of
$g(\mathbf{x})$. The new point is searched with thresholds corresponding to
the maximum probability of EI function in Fig.6(f). As shown in Fig.6(d), the
score of the new point is higher than $Score^{*}$, so it is marked as a good
sample. The new sample in Fig.6(d) is then used for the next iteration to
calculate the new density functions and EI function as shown in Fig.6(g).
Similarly, the new sample will be searched with the thresholds corresponding
to the maximum probability of EI, and categorized as a good or bad sample for
the subsequent iteration. In this way, the TPE adjusts the size of the
parameter search space and finds the global optimal solution in as few
iterations as possible.

Fig.6(h) shows iteration results of TPE. In early iterations, there is
considerable fluctuation in both the computational budget and accuracy, as
well as a large variation in the objective function. However, as the number of
iterations approaches 400, the results become stable, indicating a gradual
approach towards the optimal solution of the objective function. This is also
shown in Fig.6(i-j) about the evolution of the thresholds of the fourth and
fifth layers. The thresholds of both gradually converge to their optimal
values, yielding the optimal solution to the Pareto-optimal problem and the
best trade-off between the computational budget and accuracy.

Figure 6: Solving the optimal problem on trading off between computational
budget and accuracy with TPE a, Results of accuracy and budget drop by
adjusting the threshold from a lower value to a higher value with the method
of grid search. b, Objective function. Assuming a budget drop of 0.5, and
considering accuracy of 0.35, 0.55, 0.75, and 0.95 from top to bottom. c,
Objective score function with accuracy and budget drop. The curved surface
represents the score of objective function. The orange surface represents the
metric score "Score*", samples with a score higher than Score* are categorized
as good samples, samples with a score lower than Score* are categorized as bad
samples. d, The first new sample is obtained by finding thresholds
corresponding to the maximum probability calculated by the EI function in
figure f. e, The second new sample is obtained by finding thresholds
corresponding to the maximum probability calculated by the EI function in
figure g. f, Method of tree-structured Bayesian optimization. The purple and
blue 3D Gaussian surface represent the density of good samples and bad samples
in figure c, respectively. The orange surface represents the EI function
calculated with the two density functions. g, Obtained density of good samples
and bad samples in figure d and EI function calculated with the two density
functions. h, Results of accuracy and budget drop with the iteration of TPE.
i, Results of threshold 4 with 1000 iteration times. The color of points
represents the value of score, the bluer the color, the higher the score. j,
Results of threshold 5 with 1000 iteration times. k, Results of obtained score
using the searched threshold with 1000 iteration times.

## Discussion

We demonstrate a semantic memory-based DNN using memristor-based CIM and CAM.
CIM processes data directly within memory, resulting in reduced energy
consumption and latency compared to conventional digital computers of von
Neumann architecture. Additionally, CAM functions as the semantic memory and
associates new inputs with past experience in a brain-like manner. Leveraging
both CIM and CAM, we physically implement noise-proof ternary quantized DNN
and optimize the thresholds using TPE by solving the Pareto-optimal problem,
which results in dynamic connectivities not only reducing computational budget
but also retaining network performance. We validate our co-designs using a
40nm memristor macro on ResNet and PointNet++ for classifying images and 3D
points from the MNIST and ModelNet datasets. Our approach achieves accuracy
comparable to software while reducing the computation budget by 48.1% and
15.9% compared to static neural networks. Furthermore, it offers a significant
reduction in energy consumption, with a 77.6% and 93.3% decrease compared to a
state-of-the-art GPU. Our results lay the foundation for future machine
intelligence that can potentially parallel the adaptability and efficiency of
the human brain.

## Materials and Methods

### Fabrication of memristor array

Memristors are integrated on the 40nm standard logic platform, forming a
512$\times$512 crossbar array. The memristors are constructed between the
metal 4 and metal 5 layers of the backend-of-line process, including bottom
electrodes, top electrodes, and a transition metal oxide dielectric layer. The
bottom electrodes have a patterned via with a diameter of 60nm, formed with
the method of photolithography and etching. Physical vapor deposition and
chemical mechanical polishing are employed to deposit material within the via.
A 10 nm TaN buffer layer is deposited on the bottom electrode via using
physical vapor deposition. Subsequently, 5 nm Ta is deposited and then
oxidized in an oxygen environment to form an 8 nm $\mathrm{Tao_{x}}$
dielectric layer. On the top electrode, we sequentially deposited 3 nm Ta and
40 nm TiN using physical vapor deposition. Next, the logic backend-of-line
metal is deposited using standard logic process. Memristors in the same column
share top electrode connections, and memristors in the same row share bottom
electrode connections. After a post-annealing process in vacuum at 400°C for
30 minutes,the fabrication of the memristor chip is completed.

### Hybrid analogue–digital computing platform

The hybrid analogue–digital computing platform consists of a 40 nm memristor
chip and the a Xilinx ZYNQ system-on-chip. There is an 8-channel digital-to-
analogue converter (DAC80508, TEXAS INSTRUMENTS) in the system to generate
parallel 64-channel analogue voltages which ranges from 0V to 5V, it is used
to transform input digital signals into corresponding analogue signals. The
convergence currents are converted to voltages using a transimpedance
amplifier (OPA4322-Q1, TEXAS INSTRUMENTS) and read out as digital signals by
an analogue-to-digital converter (ADS8324, TEXAS INSTRUMENTS) with a 14-bit
resolution for signal collections. To perform vector-matrix multiplication, a
4-channel analogue multiplexer (CD4051B, TEXAS INSTRUMENTS) with an 8-bit
shift register (SN74HC595, TEXAS INSTRUMENTS) applies DC voltages to the word
lines of the memristor chip. The multiplication results carried by the current
from the source line are converted into voltages and passed to the Xilinx ZYNQ
system-on-chip for further processing.

### DNN-based ResNet

Taking advantage of the high and low resistance states exhibited by
memristors, we leverage the properties of two such memristors to effectively
represent a ternary value, employing the principles of Ohm’s law and
Kirchhoff’s laws. The memristors in crossbar array are partitioned into two
matrices for programming the weights of ResNet and values stored in CAM. If
both corresponding memristors are adjusted to the high-resistance state, they
represent a weight of 0. If the memristor in the first part is in the low
resistance state while the second memristor is in the high resistance state,
they represent a weight of 1. Conversely, if the memristor in the first part
is in the high resistance state while the second memristor is in the low
resistance state, they represent a weight of -1. In the experiment, the ResNet
model consists of 11 residual blocks, with a total of approximately 88k weight
parameters, and there are approximately 2k values stored in CAM.

### DNN-based PointNet++

PointNet++ is built upon the foundation of PointNet, which consists of a
series of Multi-Layer Perceptrons (MLPs) and a global feature aggregation step
using a symmetric function. PointNet++ uses Set Abstraction (SA) layers to
capture local features in the point cloud using Farthest Point Sampling (FPS)
strategy to select representative points and apply local PointNet operations
on their neighboring points within a specific radius. In the experiment, the
DNN-based PointNet++ consists of eight SA layers with varying radius and
numbers of representative points. Additionally, PointNet++ consists of Feature
Propagation (FP) Layers, which use nearest neighbor interpolation and feature
concatenation from different layers to help the network reconstruct global
point cloud features from the hierarchical local features.

### Ternary Quantization

When training a ternary network, ternary quantization is performed during the
forward propagation, while weight adjustment using full-precision values is
carried out during the backward propagation. The method of ternary
quantization is as follows:

$\begin{split}w_{min}&=argmin\ W^{l},\\\ w_{max}&=argmax\ W^{l},\\\
l_{in}&=w_{min}+(w_{max}-w_{min})/3,\\\
h_{in}&=w_{max}-(w_{max}-w_{min})/3,\\\ \end{split}$ (4)

where $W^{l}$ represent the whole weight of block l, $l_{in}$ and $h_{in}$ are
two intervals. The output ternary quantized weight is as follows:

$\begin{split}w_{q}=\left\\{\begin{aligned} -1&,\ if\ w_{i}<l_{in},\\\ 0&,\
if\ h_{in}\geq w_{i}\geq l_{in},\\\ 1&,\ if\ w_{i}>h_{in},\\\
\end{aligned}\right.\end{split}$ (5)

### Tree Parzen Estimators

Bayes’ rule is a fundamental concept in probability theory and statistics. It
describes the probability of an event based on prior knowledge or information.
The formula for Bayes’ theorem is expressed as:

$\mathrm{P(A|B)=\frac{P(B|A)P(A)}{P(B)}}$ (6)

Where P(A|B) represents the conditional probability of event A given event B.
P(B|A) represents the conditional probability of event B given event A. P(A)
is the probability of event A occurring. P(B) is the probability of event B
occurring. The advantage of using Bayesian inference is that it leverages
prior experience to make inferences about the next sample. By incorporating
prior information into the analysis, Bayesian methods can accelerate the
process of finding the optimal solution. TPE uses EI as the acquisition
function. However, since p(y|x) cannot be obtained directly, we employ Bayes’
rule of 6 to perform the transformation in 3. The details for deriving the
final expression in 3 are as the following:

$p(\mathbf{x})=\int p(\mathbf{x}|y)p(y)dy=\gamma
l(\mathbf{x})+(1-\gamma)g(\mathbf{x})$ (7)

$\int_{-\infty}^{y^{*}}(y^{*}-y)p(\mathbf{x}|y)p(y)dy=l(\mathbf{x})\int_{-\infty}^{y^{*}}(y^{*}-y)p(y)dy-l(\mathbf{x})\int_{-\infty}^{y^{*}}p(y)dy$
(8)

Where $\gamma$ represents the quantile of TPE, used to divide $l(\mathbf{x})$
and $g(\mathbf{x})$, ranging between 0 and 1. For example, if $\gamma$ is set
to 0.2 and there are 10 observed samples, the top 2 best performed samples
will be used to construct the distribution of good samples, while the
remaining 8 samples will be used to construct the distribution of bad samples.

The the Parzen window [68] is a technique for kernel density estimation. It
estimates the probability density of an estimated value based on the current
observed data and the type of prior distribution. The probability density
estimation function for Parzen window is as follows:

$p(\mathbf{x})=\frac{k}{n\cdot
V}=\frac{1}{nh^{2}}\sum_{i=1}^{n}\phi(\frac{\mathbf{x_{i}}-\mathbf{x}}{h})$
(9)

The window function determines weight assigned to each observed data point
based on its distance from the point $\mathbf{x_{i}}$, which means that the
weights are higher for data points closer to $\mathbf{x_{i}}$ and decrease as
the distance increases. When using a Gaussian function as the window function,
the density function is as:

$\displaystyle
p(\mathbf{x})=\frac{1}{n}\sum_{i=1}^{n}\frac{1}{\sqrt{2\pi}\sigma}exp(-\frac{(\mathbf{x_{i}}-\mathbf{x})}{2\sigma^{2}})$
(10)

where $\int p(\mathbf{x})d\mathbf{x}=1$.

## Acknowledgements

### Funding

This research is supported by the National Key R&D Program of China (Grant No.
2020AAA0109005), the National Natural Science Foundation of China (Grant Nos.
62122004, 62374181, 62488201), the Strategic Priority Research Program of the
Chinese Academy of Sciences (Grant No. XDB44000000), Beijing Natural Science
Foundation (Grant No. Z210006), Hong Kong Research Grant Council (Grant Nos.
27206321, 17205922, 17212923). This research is also partially supported by
ACCESS – AI Chip Center for Emerging Smart Systems, sponsored by Innovation
and Technology Fund (ITF), Hong Kong SAR.

### Author contributions

Conceptualization: YZ, ZW, DS. Methodology: YZ, ZW, DS, WZ. Investigation: YZ,
DS, ZW. Visualization: YZ, DS, ZW. Funding acquisition: HJ, PL, XX, DS, ZW,
XZ, QL, KTC, ML. Project administration: DS, ZW. Supervision: ZW, DS, XQ.
Writing – original draft: YZ. Writing – review and editing: ZW, DS, YZ, SW,
NL, YY, YH, BW, HJ, PL, XX, XQ, WZ, XZ, QL, KTC, ML.

### Competing Interests

The authors declare no competing interests.

### Data and materials availability

All data needed to evaluate the conclusions in the paper are present in the
paper and/or the Supplementary Materials.

## References

  * [1] Hubel, D. H. & Wiesel, T. N.  Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex.  _The Journal of physiology_ 160, 106 (1962).
  * [2] Murata, A., Gallese, V., Luppino, G., Kaseda, M. & Sakata, H.  Selectivity for the shape, size, and orientation of objects for grasping in neurons of monkey parietal area aip.  _Journal of neurophysiology_ 83, 2580–2601 (2000).
  * [3] Diano, M. _et al._ Dynamic changes in amygdala psychophysiological connectivity reveal distinct neural networks for facial expressions of basic emotions.  _Scientific reports_ 7, 45260 (2017).
  * [4] Cichy, R. M., Pantazis, D. & Oliva, A.  Similarity-based fusion of meg and fmri reveals spatio-temporal dynamics in human cortex during visual object recognition.  _Cerebral Cortex_ 26, 3563–3579 (2016).
  * [5] Hermans, E. J., Henckens, M. J., Joëls, M. & Fernández, G.  Dynamic adaptation of large-scale brain networks in response to acute stressors.  _Trends in neurosciences_ 37, 304–314 (2014).
  * [6] Scarborough, D. L., Gerard, L. & Cortese, C.  Accessing lexical memory: The transfer of word repetition effects across task and modality.  _Memory & Cognition_ 7, 3–12 (1979).
  * [7] Wong, H.-S. P. & Salahuddin, S.  Memory leads the way to better computing.  _Nature nanotechnology_ 10, 191–194 (2015).
  * [8] Han, Y. _et al._ Dynamic neural networks: A survey.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 44, 7436–7456 (2021).
  * [9] Hu, X., Huang, Z., Huang, A., Xu, J. & Zhou, S.  A dynamic multi-scale voxel flow network for video prediction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 6121–6131 (2023).
  * [10] Li, C., Jones, E. & Furber, S.  Unleashing the potential of spiking neural networks by dynamic confidence.  _arXiv e-prints_ arXiv–2303 (2023).
  * [11] Ye, M., Meyer, G. P., Chai, Y. & Liu, Q.  Efficient transformer-based 3d object detection with dynamic token halting.  _arXiv preprint arXiv:2303.05078_ (2023).
  * [12] Yang, L., He, Z., Cao, Y. & Fan, D.  A progressive subnetwork searching framework for dynamic inference.  _IEEE Transactions on Neural Networks and Learning Systems_ (2022).
  * [13] Panda, P., Sengupta, A. & Roy, K.  Energy-efficient and improved image recognition with conditional deep learning.  _ACM Journal on Emerging Technologies in Computing Systems (JETC)_ 13, 1–21 (2017).
  * [14] Teerapittayanon, S., McDanel, B. & Kung, H.-T.  Branchynet: Fast inference via early exiting from deep neural networks.  In _2016 23rd international conference on pattern recognition (ICPR)_ , 2464–2469 (IEEE, 2016).
  * [15] Ju, W., Bao, W., Yuan, D., Ge, L. & Zhou, B. B.  Learning early exit for deep neural network inference on mobile devices through multi-armed bandits.  In _2021 IEEE/ACM 21st International Symposium on Cluster, Cloud and Internet Computing (CCGrid)_ , 11–20 (IEEE, 2021).
  * [16] Park, J.-H., Kim, Y., Kim, J., Choi, J.-Y. & Lee, S.  Dynamic structure pruning for compressing cnns.  _arXiv preprint arXiv:2303.09736_ (2023).
  * [17] Huang, G. _et al._ Multi-scale dense networks for resource efficient image classification.  _arXiv preprint arXiv:1703.09844_ (2017).
  * [18] Shazeer, N. _et al._ Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.  _arXiv preprint arXiv:1701.06538_ (2017).
  * [19] Tan, M. _et al._ Mnasnet: Platform-aware neural architecture search for mobile.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , 2820–2828 (2019).
  * [20] Alibart, F., Zamanidoost, E. & Strukov, D. B.  Pattern classification by memristive crossbar circuits using ex situ and in situ training.  _Nature communications_ 4, 2072 (2013).
  * [21] Prezioso, M. _et al._ Training and operation of an integrated neuromorphic network based on metal-oxide memristors.  _Nature_ 521, 61–64 (2015).
  * [22] Yu, S. _et al._ Binary neural network with 16 mb rram macro chip for classification and online training.  In _2016 IEEE International Electron Devices Meeting (IEDM)_ , 16–2 (IEEE, 2016).
  * [23] Yao, P. _et al._ Face classification using electronic synapses.  _Nature communications_ 8, 15199 (2017).
  * [24] Sheridan, P. M. _et al._ Sparse coding with memristor networks.  _Nature nanotechnology_ 12, 784–789 (2017).
  * [25] Du, C. _et al._ Reservoir computing using dynamic memristors for temporal information processing.  _Nature communications_ 8, 2204 (2017).
  * [26] Bayat, F. M. _et al._ Implementation of multilayer perceptron network with highly uniform passive memristive crossbar circuits.  _Nature communications_ 9, 2331 (2018).
  * [27] Hu, M. _et al._ Memristor-based analog computation and neural network classification with a dot product engine.  _Advanced materials (Deerfield Beach, Fla.)_ 30 (2018).
  * [28] Moon, J. _et al._ Temporal data classification and forecasting using a memristor-based reservoir computing system.  _Nature Electronics_ 2, 480–487 (2019).
  * [29] Cai, F. _et al._ A fully integrated reprogrammable memristor–cmos system for efficient multiply–accumulate operations.  _Nature Electronics_ 2, 290–299 (2019).
  * [30] Duan, Q. _et al._ Spiking neurons with spatiotemporal dynamics and gain modulation for monolithically integrated memristive neural networks.  _Nature communications_ 11, 3399 (2020).
  * [31] Joshi, V. _et al._ Accurate deep neural network inference using computational phase-change memory.  _Nature communications_ 11, 2473 (2020).
  * [32] Yao, P. _et al._ Fully hardware-implemented memristor convolutional neural network.  _Nature_ 577, 641–646 (2020).
  * [33] Xue, C.-X. _et al._ A cmos-integrated compute-in-memory macro based on resistive random-access memory for ai edge devices.  _Nature Electronics_ 4, 81–90 (2021).
  * [34] Karunaratne, G. _et al._ In-memory hyperdimensional computing.  _Nature Electronics_ 3, 327–337 (2020).
  * [35] Liu, Z. _et al._ Neural signal analysis with memristor arrays towards high-efficiency brain–machine interfaces.  _Nature communications_ 11, 4234 (2020).
  * [36] Karunaratne, G. _et al._ Robust high-dimensional memory-augmented neural networks.  _Nature communications_ 12, 2468 (2021).
  * [37] Zhong, Y. _et al._ Dynamic memristor-based reservoir computing for high-efficiency temporal signal processing.  _Nature communications_ 12, 408 (2021).
  * [38] Milano, G. _et al._ In materia reservoir computing with a fully memristive architecture based on self-organizing nanowire networks.  _Nature Materials_ 21, 195–202 (2022).
  * [39] Dalgaty, T. _et al._ In situ learning using intrinsic memristor variability via markov chain monte carlo sampling.  _Nature Electronics_ 4, 151–161 (2021).
  * [40] Wang, S. _et al._ Echo state graph neural networks with analogue random resistive memory arrays.  _Nature Machine Intelligence_ 5, 104–113 (2023).
  * [41] Yang, R. _et al._ Ternary content-addressable memory with mos2 transistors for massively parallel data search.  _Nature Electronics_ 2, 108–114 (2019).
  * [42] Li, C. _et al._ Analog content-addressable memories with memristors.  _Nature communications_ 11, 1638 (2020).
  * [43] Pedretti, G. _et al._ Differentiable content addressable memory with memristors.  _Advanced electronic materials_ 8, 2101198 (2022).
  * [44] Laguna, A. F. _et al._ Hardware-software co-design of an in-memory transformer network accelerator.  _Frontiers in Electronics_ 3, 847069 (2022).
  * [45] Mao, R. _et al._ Experimentally validated memristive memory augmented neural network with efficient hashing and similarity search.  _Nature communications_ 13, 6284 (2022).
  * [46] Pedretti, G. _et al._ Tree-based machine learning performed in-memory with memristive analog cam.  _Nature communications_ 12, 5806 (2021).
  * [47] Ni, K. _et al._ Ferroelectric ternary content-addressable memory for one-shot learning.  _Nature Electronics_ 2, 521–529 (2019).
  * [48] Laguna, A. F., Kazemi, A., Niemier, M. & Hu, X. S.  In-memory computing based accelerator for transformer networks for long sequences.  In _2021 Design, Automation & Test in Europe Conference & Exhibition (DATE)_, 1839–1844 (IEEE, 2021).
  * [49] Pedretti, G., Serebryakov, S., Strachan, J. P. & Graves, C. E.  A general tree-based machine learning accelerator with memristive analog cam.  In _2022 IEEE International Symposium on Circuits and Systems (ISCAS)_ , 220–224, DOI: 10.1109/ISCAS48785.2022.9937772 (2022).
  * [50] Youn, S., Kim, S., Kim, T.-H., Park, J. & Kim, H.  Memristor crossbar circuit for ternary content-addressable memory with fine-tuning operation.  _Advanced Intelligent Systems_ 5, 2200325 (2023).
  * [51] Chang, M.-F. _et al._ A 3t1r nonvolatile tcam using mlc reram for frequent-off instant-on filters in iot and big-data processing.  _IEEE Journal of Solid-State Circuits_ 52, 1664–1679 (2017).
  * [52] Pagiamtzis, K. & Sheikholeslami, A.  Content-addressable memory (cam) circuits and architectures: A tutorial and survey.  _IEEE journal of solid-state circuits_ 41, 712–727 (2006).
  * [53] He, K., Zhang, X., Ren, S. & Sun, J.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 770–778 (2016).
  * [54] Qi, C. R., Yi, L., Su, H. & Guibas, L. J.  Pointnet++: Deep hierarchical feature learning on point sets in a metric space.  _Advances in neural information processing systems_ 30 (2017).
  * [55] LeCun, Y., Bottou, L., Bengio, Y. & Haffner, P.  Gradient-based learning applied to document recognition.  _Proceedings of the IEEE_ 86, 2278–2324 (1998).
  * [56] Wu, Z. _et al._ 3d shapenets: A deep representation for volumetric shapes.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 1912–1920 (2015).
  * [57] Li, Y. _et al._ Boosting mobile cnn inference through semantic memory.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , 2362–2371 (2021).
  * [58] Schroff, F., Kalenichenko, D. & Philbin, J.  Facenet: A unified embedding for face recognition and clustering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 815–823 (2015).
  * [59] Van der Maaten, L. & Hinton, G.  Visualizing data using t-sne.  _Journal of machine learning research_ 9 (2008).
  * [60] Bergstra, J., Bardenet, R., Bengio, Y. & Kégl, B.  Algorithms for hyper-parameter optimization.  _Advances in neural information processing systems_ 24 (2011).
  * [61] Bergstra, J., Yamins, D. & Cox, D.  Making a science of model search: Hyperparameter optimization in hundreds of dimensions for vision architectures.  In _International conference on machine learning_ , 115–123 (PMLR, 2013).
  * [62] Yeo, I., Chu, M., Gi, S.-G., Hwang, H. & Lee, B.-G.  Stuck-at-fault tolerant schemes for memristor crossbar array-based neural networks.  _IEEE Transactions on Electron Devices_ 66, 2937–2945 (2019).
  * [63] Niu, G. _et al._ Geometric conductive filament confinement by nanotips for resistive switching of hfo2-rram devices with high performance.  _Scientific reports_ 6, 25757 (2016).
  * [64] Liu, B. _et al._ Vortex: Variation-aware training for memristor x-bar.  In _Proceedings of the 52nd Annual Design Automation Conference_ , 1–6 (2015).
  * [65] Močkus, J.  On bayesian methods for seeking the extremum.  In _Optimization Techniques IFIP Technical Conference: Novosibirsk, July 1–7, 1974_ , 400–404 (Springer, 1975).
  * [66] Mockus, J.  The application of bayesian methods for seeking the extremum.  _Towards global optimization_ 2, 117 (1998).
  * [67] Jones, D. R., Schonlau, M. & Welch, W. J.  Efficient global optimization of expensive black-box functions.  _Journal of Global optimization_ 13, 455–492 (1998).
  * [68] Parzen, E.  On estimation of a probability density function and mode.  _The annals of mathematical statistics_ 33, 1065–1076 (1962).