# Monolithic Silicon Photonic Architecture for Training Deep Neural Networks
with Direct Feedback Alignment

Matthew J. Filipovich<EMAIL_ADDRESS>Zhimu Guo  Department of
Physics, Engineering Physics and Astronomy, Queen’s University, Kingston, ON
K7L 3N6, Canada  Mohammed Al-Qadasi  Department of Electrical and Computer
Engineering, University of British Columbia, Vancouver, BC V6T 1Z4, Canada
Bicky A. Marquez  Hugh D. Morison  Department of Physics, Engineering Physics
and Astronomy, Queen’s University, Kingston, ON K7L 3N6, Canada  Volker J.
Sorger  Department of Electrical and Computer Engineering, George Washington
University, Washington, DC 20052, USA  Paul R. Prucnal  Department of
Electrical Engineering, Princeton University, Princeton, NJ 08542, USA  Sudip
Shekhar  Department of Electrical and Computer Engineering, University of
British Columbia, Vancouver, BC V6T 1Z4, Canada  Bhavin J. Shastri  Department
of Physics, Engineering Physics and Astronomy, Queen’s University, Kingston,
ON K7L 3N6, Canada  Vector Institute, Toronto, ON M5G 1M1, Canada

###### Abstract

The field of artificial intelligence (AI) has witnessed tremendous growth in
recent years, however some of the most pressing challenges for the continued
development of AI systems are the fundamental bandwidth, energy efficiency,
and speed limitations faced by electronic computer architectures. There has
been growing interest in using photonic processors for performing neural
network inference operations, however these networks are currently trained
using standard digital electronics. Here, we propose on-chip training of
neural networks enabled by a CMOS-compatible silicon photonic architecture to
harness the potential for massively parallel, efficient, and fast data
operations. Our scheme employs the direct feedback alignment training
algorithm, which trains neural networks using error feedback rather than error
backpropagation, and can operate at speeds of trillions of multiply-accumulate
(MAC) operations per second while consuming less than one picojoule per MAC
operation. The photonic architecture exploits parallelized matrix-vector
multiplications using arrays of microring resonators for processing multi-
channel analog signals along single waveguide buses to calculate the gradient
vector of each neural network layer in situ, which is the most computationally
expensive operation performed during the backward pass. We also experimentally
demonstrate training a deep neural network with the MNIST dataset using on-
chip MAC operation results. Our novel approach for efficient, ultra-fast
neural network training showcases photonics as a promising platform for
executing AI applications.

## Introduction

Propelled by recent advances in deep learning,1 the fields of artificial
intelligence (AI) and neuromorphic computing (neuro-biological computer
architectures) have seen a renaissance over the past decade.2 Today, the
software implementations of AI algorithms are executed on traditional
computers based on the von Neumann architecture.3 However, this architecture
faces inherent data-transfer speed limitations due to the separation of memory
and processor unit, known as the von Neumann bottleneck. Neuromorphic
computing seeks to eliminate this constraint by exploiting the underlying
elementary physics of hardware systems to create an isomorphism with AI
algorithms.4 With the ultimate limit of Moore’s law, which claims that the
number of transistors on a microchip doubles every two years, approaching
saturation and the exponentially increasing needs from advanced machine
learning algorithms,5; 6 neuromorphic computing has attracted renewed interest
as an alternative to the standard electronic digital computer architectures.7

Silicon photonics has shown to be a promising platform for developing
neuromorphic systems due to its compatibility with the mature silicon
integrated circuit industry and the availability of high-quality silicon-on-
insulator (SOI) wafers.8; 9 Monolithic silicon photonics enables the
integration of active photonic and electronic components onto a single
photonic integrated chip (PIC), including modulators, detectors, amplifiers,
complementary metal-oxide-semiconductor (CMOS) control circuits, and optical
multiplexers.10 Compared to their electronic counterparts for neuromorphic
applications, photonic systems offer low latency, high bandwidth density and
baud rates, low-cost communication, and inherent parallelism using optical
multiplexing.11; 12

Figure 1: Scheme for the DFA algorithm. (a) The input vector $\mathbf{x}$
propagates through the neural network with $l$ layers during the forward pass
(inference), where $\mathbf{h}^{(k)}$ is the output from the hidden layer $k$
and $\mathbf{\hat{y}}$ is the output vector. The error vector at the output
layer $\mathbf{e}$, defined as the gradient of the cost function $J$, is
calculated and sent directly to each hidden layer. During the backward pass,
this error is propagated through fixed random feedback connections for each
hidden layer to calculate the gradient vector $\boldsymbol{\delta}^{(k)}$
which is used for the weight updates. (b) Block diagram of the operations
performed during the forward pass at the layer $k$. (c) Block diagram of the
gradient vector calculations performed during the backward pass at the layer
$k$.

One area of machine learning that would particularly benefit from the low
power consumption and high information processing bandwidth enabled by
photonics is the training of large neural networks. Several photonic
architectures have been proposed for executing in-memory computation of neural
network inference.13; 14; 15; 16 However, for the neural network to perform a
useful task, the optimal network parameters (weights and biases) must first be
determined using deep learning training algorithms. These algorithms have high
computation and memory costs that pose challenges to the current hardware
platforms executing them,17 and the substantial energy consumption required to
train large neural networks using standard von Neumann architectures presents
a significant financial and environmental cost.18 The recently proposed direct
feedback alignment (DFA) supervised learning algorithm has gathered interest
as a bio-plausible alternative to the popular backpropagation training
algorithm. 19 The DFA algorithm, shown in Fig. 1 and further described in the
Methods section, is a supervised learning algorithm that propagates the error
through fixed random feedback connections directly from the output layer to
the hidden layers during the backward pass.20 Unlike backpropagation, the DFA
algorithm does not require the network layers to be updated sequentially
during the backward pass, enabling the algorithm to be a suitable candidate
for efficient parallelisation using photonics. The training algorithm has been
used to train neural networks using the MNIST, Cifar-10, and Cifar-100
datasets, and yields comparable performance to backpropagation.20 As well, the
DFA algorithm has been shown to obtain performances comparable to fine-tuned
backpropagation in applications requiring state-of-the-art deep learning
networks, including natural language processing and neural view synthesis.21 A
recent theory suggests that training shallow networks with the DFA algorithm
occurs in two steps: the first step is an alignment phase where the weights
are modified to align the approximate gradient with the true gradient of the
loss function, which is followed by a memorisation phase where the model
focuses on fitting the data.22

Here, we propose a silicon photonic architecture that uses an electro-optic
circuit for calculating the gradient vector of each neural network layer in
situ, which is the most computationally expensive operation performed during
the backward pass. The proposed architecture exploits the speed and energy
advantages of photonics to determine the gradient vector of each neural
network layer in a single operational cycle. The scope of this paper is
limited to the implementation of the DFA algorithm’s backward pass; however,
inference can also be performed using a similar photonic architecture.13
Training a hardware-based neural network in situ could also inherently account
for the manufacturing imperfections and variations of the physical neurons.
During the backward pass, the error from the network’s inference step is
encoded on multi-channel optical inputs. The electro-optic circuit then
calculates the gradient vector for each hidden layer, which is used to update
the network parameters stored in memory using an external digital control
system. We also demonstrate training a feed-forward neural network on the
MNIST dataset using experimental on-chip results to validate our proposed
architecture.

## Results

### Multiply-accumulate operations in photonics

The proposed architecture uses microring resonators (MRRs), which are closed-
loop waveguides that use codirectional evanescent coupling between the ring
and adjacent bus waveguides, as tunable filters to weight photonic signals
through amplitude modulation.23 By exploiting the plasma dispersion effect,
the refractive index of each MRR is adjusted by varying the concentration of
carriers through external biasing. This change in refractive index is used to
tune the MRR’s resonance peaks and modulate the transmission of optical
signals for a specified wavelength.24 In our proposed architecture, the MRRs
which perform multiply-accumulate (MAC) operations are arranged in add-drop
geometries coupled to two waveguide buses, known as the through and drop
ports, as shown in Fig. 2a. Tuning the MRR’s resonance to control the amount
of light transmission through the two ports effectively weights the input
optical signal, a technique which can be exploited to perform the dot product
between two vectors.

The dot product operation, which consists of a series of MAC operations
$\mathbf{x}\cdot\mathbf{w}=\Sigma x_{i}w_{i}$, is executed in photonics using
an array of MRRs and a balanced photodetector.25 Using wavelength-division
multiplexing (WDM) techniques to process multi-channel analog signals along
the same waveguide bus, where each optical signal with wavelength
$\lambda_{i}$ is amplitude encoded with a value $x_{i}$, each MRR in the array
is tuned with a specified weighting value $w_{i}\in[-1,1]$ for the
corresponding optical input wavelength $\lambda_{i}$. The through and drop
port transmission spectrums, $T_{p}$ and $T_{d}$, of each MRR as a function of
round-trip phase shift are Lorentzian-shaped and centered at the resonance
phase with the incoming light. When the coupling losses between the MRR and
bus waveguides are negligible, the relationship between the through and drop
port transmissions for each MRR is $T_{p}=1-T_{d}$. These two ports are
connected into a balanced photodetector which gives an effective transfer
function in the electrical domain that is proportional to
$|E_{0}|^{2}(T_{d}-T_{p})$, where $E_{0}$ is the amplitude of the input
optical signal. Thus, the assigned weighting value $w_{i}$ of each MRR is
given by $T_{d}-T_{p}$. Using this scheme, external biasing of the MRR tunes
the resonance peak to correspond with the desired weighting for a selected
optical wavelength, as shown in Fig. 2b. Finally, the dot product between two
vectors of size $N$ can be executed using an array of $N$ MRRs that are
coupled to the same through and drop ports, as shown in Fig. 2c.

The continuous and multi-channel control of MRR arrays based on feedback
control is an ongoing area of research, and a record-high accuracy of 8.5 bits
for a single MRR using thermal tuning has recently been observed with
negligible inter-channel crosstalk. 26 Due to precision limitations in PIC
manufacturing leading to device variance, the relationship between the applied
MRR bias and the change in weighting value for a specified optical wavelength
must be determined experimentally. 27 The current tuning approach relies on
feedforward control to calibrate the MRRs, as well as feedback control for
sensing the state of the system and correcting for any changes due to dynamic
variability, such as environment fluctuations.28

Figure 2: MAC operation and dot product with photonics. (a) MRR in add-drop
configuration with $8\text{\,}\mathrm{\SIUnitSymbolMicro m}$ radius coupled to
through and drop ports. (b) Theoretical transmission profile of an add-drop
MRR with a self-coupling coefficient of 0.95 and negligible attenuation. The
weighting value $w$ is given by $T_{d}-T_{p}$. (c) Array of four MRRs coupled
to the through and drop ports which are connected to a balanced photodetector.
The dot product operation is executed in the analog domain by injecting multi-
channel analog signals with wavelength $\lambda_{i}$ and amplitude $x_{i}$.
Each MRR in the array is tuned to weight a corresponding wavelength by
$w_{i}$, which yields the dot product $\mathbf{x}\cdot\mathbf{w}$ in the
electrical domain.

### Photonic deep learning architecture

Figure 3: Photonic DFA architecture. (a) Equation defining the gradient vector
$\boldsymbol{\delta}^{(k)}$ for the hidden layer $k$. (b) Silicon photonic
architecture for calculating the gradient vector $\boldsymbol{\delta}^{(k)}$
for the hidden layer $k$. The output layer gradient vector $\mathbf{e}$ is
amplitude encoded onto $N$ different wavelengths using an array of MRRs, and
the matrix-vector product of the $M\times N$ matrix $\mathbf{B}^{(k)}$ and
vector $\mathbf{e}$ is performed in the photonic weight bank. The Hadamard
product between the vectors $\mathbf{B}^{(k)}\mathbf{e}$ and
$g^{\prime}(\mathbf{a}^{(k)})$ is executed using TIAs in the electrical
domain, yielding the desired vector $\boldsymbol{\delta}^{(k)}$ which is
converted to the digital domain using ADCs. At each time step, the control
signals that encode the MRRs and TIAs are fetched from memory.

In this section, we introduce the proposed photonic architecture for executing
the DFA training algorithm. The architecture exploits MAC operations performed
in photonics, as discussed in the previous section, to execute matrix-vector
multiplications in the analog domain during the backward pass, which are the
most computationally expensive operation performed during training. For each
training example during the backward pass, the architecture calculates the
gradient vector $\boldsymbol{\delta}^{(k)}$ of the hidden layer $k$ in a
single operational cycle. The gradient vector (see Fig. 3a) is given by

$\boldsymbol{\delta}^{(k)}=\mathbf{B}^{(k)}\mathbf{e}\odot
g^{\prime}\left(\mathbf{a}^{(k)}\right),$ (1)

where $\mathbf{B}^{(k)}$ is a fixed random weight matrix with appropriate
dimensions for the hidden layer $k$, $\mathbf{e}$ is the error from the
gradient of the cost function, $\odot$ is the Hadamard product (element-wise
multiplication operator), and $g^{\prime}$ is the derivative of the activation
function with respect to $\mathbf{a}^{(k)}$, which is the sum of the weighted
input signals in the hidden layer $k$. The network’s weights and biases are
then updated using the calculated gradient vector $\boldsymbol{\delta}^{(k)}$
for each hidden layer $k$.

A schematic of the proposed silicon photonic DFA architecture, which can be
implemented on a PIC, is shown in Fig. 3b. The DFA architecture requires a
digital control system to tune the active electro-optic components on-chip,
which include the MRRs that modulate the incoming laser light with the error
vector $\mathbf{e}$ and the transimpedance amplifiers (TIAs) with tunable gain
to convert photocurrent to voltage and scale it to implement the Hadamard
product. To execute the DFA algorithm, a training example is first propagated
through a neural network of variable size. This inferencing operation can be
performed using an analog photonic circuit13 or with a separate digital
processor. The error vector $\mathbf{e}$ from the derivative of the cost
function is then calculated using a dedicated CMOS processor that can be
integrated on the chip. Finally, the electro-optic circuit calculates the DFA
gradient vector $\boldsymbol{\delta}^{(k)}$ for each hidden layer $k$, which
is used to update the network parameters.

To calculate the gradient vector $\boldsymbol{\delta}^{(k)}$ using the
electro-optic circuit, WDM is used to combine $N$ laser signals with different
wavelengths onto a single waveguide. The element values in the error vector
$\mathbf{e}$ are amplitude encoded onto the $N$ optical signals using an array
of $N$ all-pass MRRs, where each MRR is tuned for a corresponding optical
wavelength. The intensities of the input optical signals are identical to
allow an encoding scheme that linearly maps the amplitude modulation value to
the through port transmission. The modulated optical signals representing the
error vector $\mathbf{e}$ are then coupled into arrays of parallel MRRs,
herein referred to as the photonic weight bank, which perform the required
matrix-vector product with the matrix $\mathbf{B}^{(k)}$.15 The photonic
weight bank consists of $M$ rows of MRR arrays with $N$ MRRs per row, where
the incoming signals are coupled evenly into each row using 1$\times M$
symmetrical optical splitters.29 As discussed in the previous section for
performing MAC operations in photonics, each MRR in the photonic weight bank
is arranged in an add-drop configuration coupled to two separate waveguide
buses connected to balanced photodetectors. To perform the matrix-vector
product, each column of MRRs in the photonic weight bank is tuned for the
corresponding optical wavelength $\lambda_{n}$, and thus the element
$B^{(k)}_{m,n}$ is mapped to the MRR in the $m$th row and $n$th column. Using
the add-drop configuration, the incoming optical signals, which correspond to
the error vector $\mathbf{e}$, are weighted by modifying the optical
transmissions, $T_{p}$ and $T_{d}$, for the through and drop ports of the
MRRs. Summing the two transmission ports in the electrical domain allows the
MRRs to be encoded with a weighting $w\in[-1,1]$, assuming there is negligible
loss in the system.30 If the size of the photonic weight bank is larger than
the dimensions of the matrix $\mathbf{B}$, the redundant MRRs can be tuned
with a weighting of zero. A negative value in the error vector can be encoded
in the architecture by inverting the sign of the inscribed weighting values of
the corresponding column of MRRs in the photonic weight bank.

Finally, the Hadamard product between the two vectors
$\mathbf{B}^{(k)}\mathbf{e}$ and $g^{\prime}(\mathbf{a}^{(k)})$ is performed
using a set of TIAs, where each balanced photodetector output is connected to
a TIA. The vector $g^{\prime}(\mathbf{a}^{(k)})$ is calculated by the
dedicated monolithic CMOS processor (the vector $\mathbf{a}^{(k)}$ is known
from the inference step) and encoded onto the voltage signals, supplied by the
digital control system, that set the gain of the TIAs. The elements in the
vector $g^{\prime}(\mathbf{a}^{(k)})$ are binary (0 or 1) when the ReLU
activation function is employed. Since the vector $\mathbf{a}^{(k)}$ was
previously calculated during the forward inference process, setting the gain
does not impede the system’s maximum operating speed. The analog electrical
signals are then converted to digital values with analog-to-digital converters
(ADCs) and are used by the digital control system to update the neural
network’s weights.

The size of the photonic weight bank is physically bounded by the dimensions
of the PIC and the maximum number of supported WDM channels in a single
waveguide. The WDM channel limit is dependent on the finesse of the MRRs and
the channel spacing, and an optimized design of the MRRs with a finesse of 368
could support up to 108 distinct channels.31 However, the dimensions of the
photonic weight bank do not restrict the size of the neural network being
trained; if the size of the matrix $\mathbf{B}^{(k)}$ is larger than the
dimensions of the photonic weight bank, the product can be determined over
multiple operational cycles by calculating a subset of the output vector at
each cycle.

Figure 4: DFA on-chip training results. (a) Experimental accuracy of MAC
operations performed by an MRR. The error has a standard deviation of 0.019
and mean of 0.001. (b) Confusion matrices for feed-forward neural networks
(2x800 ReLU) trained on the MNIST dataset using a standard CPU as the ground
truth (95.11%) and experimental on-chip matrix-vector operation data (93.43%).

### DFA training experiment

In a proof-of-concept experiment to validate our proposed architecture, we
collected experimental MRR operation data to train a feed-forward neural
network with two hidden layers in a Python simulation using the MNIST dataset.
We used an MRR with a radius of $8\text{\,}\mathrm{\SIUnitSymbolMicro m}$
(shown in Fig. 2a) which was fabricated on an SOI wafer with a silicon
thickness of 220 nm and a buried oxide thickness of
$2\text{\,}\mathrm{\SIUnitSymbolMicro m}$. The MRR was thermally tuned using
in-ring N-doped photoconductive heaters,32; 28 and was configured in an add-
drop configuration which enabled the encoding of both positive and negative
weighting values. The MRR was calibrated to determine the mapping between the
applied bias and MRR weighting values, as described in the Methods section.
The continuous optical signal injected into the MRR’s input port was generated
from an external cavity laser and was coupled into and out of the MRR using TE
focusing grating couplers with a total loss of 12 dB. The input signal was
modulated by a variable attenuator with 5 bits of precision, and the MRR
tuning was controlled with 6 bits of precision using an applied bias voltage
from a source meter.

Multiplication operation data was collected by varying the encoded input
signal and MRR weighting value while measuring the optical power in the drop
and through ports, $P_{d}$ and $P_{t}$, using a power meter. Following the
photonic MAC scheme previously described, the multiplication result from the
two operands (the input and MRR weighting value) was given by $P_{d}-P_{t}$.
All possible non-redundant combinations of the input (5 bits) and weight (6
bits) values were measured three separate times during the data collection,
and the average measurement was recorded. We then calibrated the optical power
results to match the expected output range of values (-1 to 1) by performing a
linear regression on the data, and the results are shown in Fig. 4a. The error
from the experimental results, given by the expected product minus the
measured data, has a standard deviation of 0.039 and a mean of 0.002.

The full photonic DFA architecture was simulated using a Python simulation
that used the collected experimental results when performing the matrix-vector
operations during the backward pass. We trained a $784\times 800\times
800\times 10$ feed-forward neural network with ReLU activation functions on
the MNIST dataset, which was scaled between 0 and 1. We used the binary cross-
entropy loss function, a constant learning rate of 0.003, and a mini-batch
size of 64. The simulation achieved an accuracy of 93.43% using experimental
data, which showed only a 1.68% drop (caused by error in the experimental
multiplication operation data) from the results calculated entirely on a
standard CPU as shown in Fig. 4b. The accuracy could be improved by
implementing convolutional layers in the network, which have recently been
demonstrated in photonics,30; 33 and by using an optimization algorithm such
as RMSprop.34

### Energy and speed analysis

Matrix-vector multiplications are the fundamental operation performed during
neural network training, and although they are a computationally expensive
operation to perform in digital electronics, they can be executed in a single
operational cycle in photonics. The power efficiency and speed of the photonic
architecture are dependent on the size of the photonic weight bank and the
maximum operational rate $f_{s}$, which is limited by the electronic device
on-chip with the lowest throughput. 30 Defining an operation as either a
multiplication or addition of two inputs, the maximum number of operations per
second (OPS) computed by the photonic architecture scales linearly with the
size of the weight bank ($M\times N$) and is given by

$\mathrm{OPS}=2f_{s}MN.$ (2)

The lower bound on the required optical power per laser supplied to the
photonic weight bank to overcome both the capacitance of the photodetectors
with $N_{b}$ bits of fixed precision and the shot noise is given by35

$P_{\mathrm{laser}}\geq
M\frac{\hbar\omega}{\eta}\max\left(2^{2N_{b}+1},\frac{CV_{d}}{e}\right),$ (3)

where $\hbar\omega$ is the photon energy, $\eta$ is the combined quantum
efficiency of the laser, photodetector, and optical system loss through the
waveguide, $C$ and $V_{d}$ are the capacitance and driving voltages of the
photodetectors, and $e$ is the elementary charge.

During training, the tunable MRRs in the architecture are inscribed with a
different value at each operational cycle. The values in the error vector
$\mathbf{e}$, which can be stored in static random access memory (SRAM), are
fetched and converted into analog voltage signals for tuning the designated
MRRs using digital-to-analog converters (DACs). Unlike the dynamic
$\mathbf{e}$ vectors that change at each operational cycle, the inscribed
weighting values of the MRRs in the photonic weight bank cycle through a set
of known, unchanging values from the matrices $\mathbf{B}^{(k)}$ for each
hidden layer $k$. The control system for these MRRs can be made highly
efficient by storing the set of weight values in an analog memory
architecture, such as using resistive random access memory (RRAM) or optical
memories based on phase change materials,36; 37 where the energy consumption
required to access and switch between weights is negligible compared to the
energy cost of modulating the input optical signals with the error vector.
Thus, the estimated wall-plug power required by the DFA architecture with a
photonic weight bank of size $M\times N$ is given by

$P_{\mathrm{total}}=NP_{\mathrm{laser}}+NP_{\mathrm{DAC}}+N(M+1)P_{\mathrm{MRR}}+M(P_{\mathrm{TIA}}+P_{\mathrm{ADC}}).$
(4)

The first term is the minimum required optical power, as shown in Eq. (3).35
The second term accounts for the power required to convert the $\mathbf{e}$
vector values into analog control signals using DACs ($P_{\mathrm{DAC}}$), and
we assume that the energy required to fetch the data from the SRAM is
negligible compared to the other components in the architecture.38 The third
term is the heating power required to tune the MRRs in the architecture
($P_{\mathrm{MRR}}$), and the final term corresponds to the power from the
TIAs ($P_{\mathrm{TIA}}$) and ADCs ($P_{\mathrm{ADC}}$).

Using Eqs. (2) and (4), the expected optimal energy consumption per operation
$E_{\mathrm{op}}$, defined by $P_{\mathrm{total}}/\mathrm{OPS}$, as a function
of the number of MAC cells in the photonic weight bank is shown in Fig. 5. To
achieve an operational rate in the GHz range with low power consumption, our
calculations assume the use of efficient, high speed MRRs with small
footprints that are tuned using carrier depletion in an embedded reverse-
biased p-n junction.39; 40 The power and performance speed of the photonic
architecture were calculated assuming the use of the ReLU activation function
and the following energy consumptions for the active electronic components:
190 mW per DAC (8 bit, 12 GS/s),41 13 mW per ADC (6 bit, 12.5 GS/s),42 2.4
pJ/bit per TIA (20 GS/s),43 and a heating power of 5 mW per MRR.44 The maximum
operational rate $f_{s}$ was limited by the throughput of the DAC to 12 GHz.
We assumed an efficiency $\eta$ of 0.2 for the laser source, photodetector,
and optical system loss through the waveguide. The calculations assume the use
of an optical signal with a wavelength of 1550 nm and high performance
photodetector with a capacitance of 2.4 fF and a driving voltage of 1 V.33
Using a photonic weight bank of size $50\times 20$, we can achieve speeds of
24 teraoperations per second (TOPS) and an energy consumption
$E_{\mathrm{op}}$ of 0.46 pJ per operation (<1 pJ per MAC). The estimated
compute density, defined by the OPS divided by the chip area, is 6.94
TOPS/mm2. We assume a photonic MAC cell of size
$$47.4\text{\,}\mathrm{\SIUnitSymbolMicro
m}$\times$73.0\text{\,}\mathrm{\SIUnitSymbolMicro m}$$, which accounts for the
overhead waveguide and electronic routing, bonding pads for MRR control, and
sufficient spacing between MRRs to eliminate cross-talk (see Fig. 2a).

Figure 5: The expected optimal energy per operation $E_{\mathrm{op}}$ as a
function of the number of MAC cells in the photonic weight bank, where
$M,N\geq 5$. The total energy consumption is dependent on both the number of
MAC cells and the dimensions of the photonic weight bank, and only the lowest
value of $E_{\mathrm{op}}$ as a function of the total number of MAC cells
(i.e., the ideal photonic weight bank dimensions) is shown. The calculated
values assume an operating speed of 12 GHz and 6 bits of precision.

## Discussion

The field of silicon photonics has rapidly accelerated over the last several
years, however there exist a number of considerable challenges that must be
addressed before the widespread adoption of optical technologies for AI
applications. Crucial for the calibration and control of the photonic
components in our proposed architecture is the co-integration of active on-
chip silicon electronics with silicon photonics.45 Monolithic fabrication
processes, which integrate electronics and photonics on the same substrate,
are actively being investigated as the demand for silicon photonic
technologies continues to grow. Recently, GlobalFoundries announced a
dedicated homogeneous process that leverages state-of-the-art CMOS
manufacturing with the integration of key photonic devices including SOI
waveguides and germanium epitaxy photodetectors.46 An alternative near-term
implementation of the photonic architecture could use flip-chip bonding by
fabricating two dies and soldering them together, where one is optimized for
the silicon photonics components and the other for the CMOS electronics. This
flip-chip bonding approach does not require the development of new
manufacturing processes and can use existing designs for the electronic
components (microprocessors, SRAM, signal converters, etc.).

Another challenge facing silicon photonic computational architectures is their
current reliance on off-chip light sources connected through fiber packaging.
Several recent approaches have been proposed to integrate light sources
directly onto silicon waveguide layers, including strain engineering of
germanium and all-silicon emissive defects and rare-earth-element doping.47;
48 An alternative approach, which is well suited for our highly-parallelized
photonic architecture, is the use of an on-chip frequency comb source that
generates evenly spaced emission wavelengths.49 The use of an on-chip
frequency comb source has recently been experimentally demonstrated for
performing photonic in-memory computing.33 However, the optical source must be
capable of producing enough optical power for the entire system, which
includes the requirement to overcome the capacitance and shot noise of the
photodetectors, and must also be efficient.

In summary, we have designed a silicon photonic architecture that executes the
fundamental operations of the DFA algorithm in the analog domain, including
matrix-vector multiplication and the Hadamard product, to calculate the
gradient vector of each neural network layer in situ. The architecture is
cascadable for training neural networks of various sizes. Using a photonic
weight bank of size $50\times 20$, the system is expected to perform up to 24
TOPS while consuming less than 1 pJ per MAC operation. The expected
improvements in training time and energy efficiency offered by this photonic
architecture could enable the development of innovative neural network
applications that would be impossible to operate on current generation
hardware. Future work includes demonstrating a complete integrated system,
which includes a dedicated CMOS processor, that is capable of operating at
high speeds to train neural networks without requiring any data processing
off-chip.

## Methods

### Experimental photonic hardware operation

The experimental operation of our photonic hardware requires a control system
that consists of both source and power meters. The source meters provide
heating currents to the MRRs, which shift their resonances using the thermo-
optic effect. The power meters monitor the output from the drop and through
ports ($T_{d}$ and $T_{p}$). The source and power meters must be calibrated by
creating a mapping from their output analog values to their expected encoded
values in the photonic system.

Initially, the resonances of the MRR were determined through a wavelength
sweep using a tunable laser. The mapping between the applied current and MRR
weight value were then found by performing a sweep of the applied current
while providing a constant laser source and measuring the optical response.
The MRR mapping was found to be accurate to 6 bits of precision. The
multiplication data was then obtained by measuring all possible combinations
of weight values and injected input signals which were modulated off-chip
using an attenuator with 5 bits of precision. The product of these two values
was represented as the power difference between the drop and through ports of
the MRR, $T_{d}-T_{p}$. A linear regression on the power difference values was
then performed to reduce the error when mapping the experimental results to
the range of desired digital values, which was between -1 and 1.

### Direct feedback alignment algorithm

In feedforward neural networks, the nodes are clustered in different layers
where the signals can only pass from a preceding to a succeeding layer. The
behaviour of all nodes in an ANN is identical: the input signals from the
preceding layer are weighted and summed, and a non-linear activation function
is applied to the sum which is outputted to all the nodes in the succeeding
layer. The forward propagation of the input signal using trained weights is
known as inferencing. The inference of an ANN with $l$ layers can be described
mathematically for each layer $k$ (defining
$\mathbf{h}^{(0)}\equiv\mathbf{x}$) as

$\displaystyle\mathbf{a}^{(k)}$
$\displaystyle=\mathbf{W}^{(k)}\mathbf{h}^{(k-1)}+\mathbf{b}^{(k)},$ (5)
$\displaystyle\mathbf{h}^{(k)}$
$\displaystyle=g\left(\mathbf{a}^{(k)}\right),$ (6)

where the variables used are defined in Table 1.

Feedforward neural networks approximate a function $f$ by defining a mapping
${\mathbf{\hat{y}}=f(\mathbf{x};\boldsymbol{\theta})}$, and the goal of
training is to find the values of the network parameters
$\boldsymbol{\theta}=\\{\mathbf{W},\mathbf{b}\\}$ that reduce the overall
error between the actual and target outputs.50 The error to be minimized can
be expressed as a cost function
$J\left(\mathbf{x},\mathbf{{y}},\boldsymbol{\theta}\right)$, where
$\mathbf{y}$ is the target output and the choice of cost function depends on
the specific application of the neural network.

The DFA algorithm first determines the gradient of the cost function in the
output layer $l$, which is the error propagated to the hidden layers:

$\mathbf{e}=\nabla_{\mathbf{a}^{(l)}}J\left(\mathbf{x},\mathbf{y},\boldsymbol{\theta}\right)=\nabla_{\mathbf{h}^{(l)}}J\left(\mathbf{x},\mathbf{y},\boldsymbol{\theta}\right)\odot
g^{\prime}\left(\mathbf{a}^{(l)}\right),$ (7)

where $\odot$ is the Hadamard product (element-wise multiplication operator)
and $g^{\prime}$ is the derivative of the activation function with respect to
$\mathbf{a}^{(l)}$. The DFA algorithm calculates the gradients for each hidden
layer $k$ as

$\boldsymbol{\delta}^{(k)}=\mathbf{B}^{(k)}\mathbf{e}\odot
g^{\prime}\left(\mathbf{a}^{(k)}\right),$ (8)

where $\mathbf{B}^{(k)}$ is a fixed random weight matrix with appropriate
dimensions.

Using the gradients calculated in Eqs. 7 and 8 and defining
$\boldsymbol{\delta}^{(l)}\equiv\mathbf{e}$, the update functions of the
network parameters $\boldsymbol{\theta}$ for each layer $k$, ignoring the
learning rate, are

$\displaystyle\nabla_{\mathbf{b}^{(k)}}J\left(\mathbf{x},\mathbf{y},\boldsymbol{\theta}\right)$
$\displaystyle=\boldsymbol{\delta}^{(k)},$ (9)
$\displaystyle\nabla_{\mathbf{W}^{(k)}}J\left(\mathbf{x},\mathbf{y},\boldsymbol{\theta}\right)$
$\displaystyle=\mathbf{h}^{(k-1)}\boldsymbol{\delta}^{(k)}.$ (10)

The deep learning algorithm stochastic gradient descent can be implemented to
improve training performance. A minibatch of examples
${\mathbb{B}=\\{\mathbf{x}_{1},\dots,\mathbf{x}_{m}\\}}$ of size $m$ from the
training set are used to compute the update functions found in Eqs. 9 and
10.50 The update value for each parameter is then averaged from the minibatch
calculations and the network parameters are updated by following the estimated
gradient downhill:

$\boldsymbol{\theta}\leftarrow\boldsymbol{\theta}-\frac{\alpha}{m}\sum_{i=1}^{m}\nabla_{\boldsymbol{\theta}}J\left(\mathbf{x}_{i},\mathbf{y}_{i},\boldsymbol{\theta}\right),$
(11)

where $\alpha$ is the learning rate. The training procedure for updating the
network parameters is repeated until the required network accuracy is
satisfied.

Table 1: Machine learning notation used in this paper. Symbol | Description
---|---
$\mathbf{W}^{(k)}$ | Weight matrix between layers $k$ and $k-1$
$\mathbf{b}^{(k)}$ | Bias vector between layers $k$ and $k-1$
$\mathbf{a}^{(k)}$ | Sum of the weighted input signals in the layer $k$
$g\left(\mathbf{a}^{\left(k\right)}\right)$ | Activation function of the neurons in the layer $k$
$\mathbf{h}^{(k)}$ | Activation of the nodes in the layer $k$
$\mathbf{x}$ | Training input
$\mathbf{y}$ | Target output
$\mathbf{\hat{y}}$ | Output of neural network from training input $\mathbf{x}$
$J\left(\mathbf{x},\mathbf{y},\boldsymbol{\theta}\right)$ | Cost function
$\mathbf{\boldsymbol{\theta}}$ | Network parameters including weights and biases
$\mathbf{e}$ | Error from gradient of the cost function

## Data availability

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

## References

  * (1) LeCun, Y., Bengio, Y. & Hinton, G.  Deep learning.  _Nature_ 521, 436–444 (2015).
  * (2) Schuman, C. D. _et al._ A survey of neuromorphic computing and neural networks in hardware.  Preprint at https://arxiv.org/abs/1705.06963 (2017).
  * (3) Von Neumann, J.  First draft of a report on the EDVAC.  _IEEE Ann. Hist. Comput._ 15, 27–75 (1993).
  * (4) Mead, C.  Neuromorphic electronic systems.  _Proc. IEEE_ 78, 1629–1636 (1990).
  * (5) Amodei, D. & Hernandez, D.  AI and compute.  Blog at https://openai.com/blog/ai-and-compute (2018).
  * (6) Canziani, A., Paszke, A. & Culurciello, E.  An analysis of deep neural network models for practical applications.  Preprint at http://arxiv.org/abs/1605.07678 (2017).
  * (7) Shastri, B. J. _et al._ Photonics for artificial intelligence and neuromorphic computing.  _Nat. Photonics_ 15, 102–114 (2021).
  * (8) Prucnal, P. R. & Shastri, B. J.  _Neuromorphic Photonics_ (CRC Press, 2017).
  * (9) Chrostowski, L. & Hochberg, M.  _Silicon Photonics Design_ (Cambridge University Press, 2015).
  * (10) Hochberg, M. _et al._ Silicon photonics: The next fabless semiconductor industry.  _IEEE J. Solid-State Circuits_ 5, 48–58 (2013).
  * (11) de Lima, T. F. _et al._ Machine learning with neuromorphic photonics.  _J. Light. Technol._ 37, 1515–1534 (2019).
  * (12) Peng, H.-T., Nahmias, M. A., de Lima, T. F., Tait, A. N. & Shastri, B. J.  Neuromorphic photonic integrated circuits.  _IEEE J. Sel. Top. Quantum Electron._ 24, 1–15 (2018).
  * (13) Tait, A. N. _et al._ Neuromorphic photonic networks using silicon photonic weight banks.  _Sci. Rep._ 7, 7430 (2017).
  * (14) Shen, Y. _et al._ Deep learning with coherent nanophotonic circuits.  _Nat. Photonics_ 11, 441–446 (2017).
  * (15) Tait, A. N. _et al._ Silicon photonic modulator neuron.  _Phys. Rev. Appl._ 11, 064043 (2019).
  * (16) Zuo, Y. _et al._ All-optical neural network with nonlinear activation functions.  _Optica_ 6, 1132–1137 (2019).
  * (17) Esser, S. K. _et al._ Convolutional networks for fast, energy-efficient neuromorphic computing.  _Proc. National Academy of Sciences_ 113, 11441–11446 (2016).
  * (18) Strubell, E., Ganesh, A. & McCallum, A.  Energy and policy considerations for deep learning in NLP.  _Proc. Annual Meeting of the Association for Computational Linguistics_ , 3645–3650 (2019).
  * (19) Rumelhart, D. E., Hinton, G. E. & Williams, R. J.  Learning representations by back-propagating errors.  _Nature_ 323, 533–536 (1986).
  * (20) Nøkland, A.  Direct feedback alignment provides learning in deep neural networks.  _Proc. Advances in Neural Information Processing Systems_ 29, 1037–1045 (2016).
  * (21) Launay, J., Poli, I., Boniface, F. & Krzakala, F.  Direct feedback alignment scales to modern deep learning tasks and architectures.  _Proc. Advances in Neural Information Processing Systems_ 33, 9346–9360 (2020).
  * (22) Refinetti, M., D’Ascoli, S., Ohana, R. & Goldt, S.  Align, then memorise: the dynamics of learning with feedback alignment.  _Proc. International Conference on Machine Learning_ , 8925–8935 (2021).
  * (23) Bogaerts, W. _et al._ Silicon microring resonators.  _Laser Photonics Rev._ 6, 47–73 (2012).
  * (24) Reed, G. T., Mashanovich, G., Gardes, F. Y. & Thomson, D. J.  Silicon optical modulators.  _Nat. Photonics_ 4, 518–526 (2010).
  * (25) Tait, A. N. _et al._ Microring weight banks.  _IEEE J. Sel. Top. Quantum Electron._ 22, 312–325 (2016).
  * (26) Zhang, W. _et al._ Microring weight banks control beyond 8.5-bits accuracy.  Preprint at http://arxiv.org/abs/2104.01164 (2021).
  * (27) Tait, A. N. _et al._ Feedback control for microring weight banks.  _Opt. Express_ 26, 26422 (2018).
  * (28) Jayatilleka, H., Shoman, H., Chrostowski, L. & Shekhar, S.  Photoconductive heaters enable control of large-scale silicon photonic ring resonator circuits.  _Optica_ 6, 84–91 (2019).
  * (29) T., S., S., I. G. & M., R.  Design and analysis of 1xN symmetrical optical splitters for photonic integrated circuits.  _Optik_ 169, 321–331 (2018).
  * (30) Bangari, V. _et al._ Digital electronics and analog photonics for convolutional neural networks (DEAP-CNNs).  _IEEE J. Sel. Top. Quantum Electron._ 26, 1–13 (2020).
  * (31) Tait, A. N.  Silicon Photonic Neural Networks (Doctoral Dissertation, Princeton University, 2018).
  * (32) Jayatilleka, H. _et al._ Wavelength tuning and stabilization of microring-based filters using silicon in-resonator photoconductive heaters.  _Opt. Express_ 23, 25084–25097 (2015).
  * (33) Feldmann, J. _et al._ Parallel convolutional processing using an integrated photonic tensor core.  _Nature_ 589, 52–58 (2021).
  * (34) Tieleman, T. & Hinton, G.  Lecture 6.5–RmsProp: Divide the gradient by a running average of its recent magnitude.  _COURSERA: Neural networks for machine learning_ (2012).
  * (35) Nahmias, M. A. _et al._ Photonic multiply-accumulate operations for neural networks.  _IEEE J. Sel. Top. Quantum Electron._ 26, 1–18 (2020).
  * (36) Zahoor, F., Azni Zulkifli, T. Z. & Khanday, F. A.  Resistive random access memory (RRAM): an overview of materials, switching mechanism, performance, multilevel cell (mlc) storage, modeling, and applications.  _Nanoscale Res. Lett._ 15, 90 (2020).
  * (37) Miscuglio, M. _et al._ Artificial synapse with mnemonic functionality using GSST-based photonic integrated memory.  _Proc. 2020 International Applied Computational Electromagnetics Society Symposium (ACES)_ , 1–3 (2020).
  * (38) Boumchedda, R. _et al._ 1.45-fJ/bit access two-port SRAM interfacing a synchronous/asynchronous IoT platform for energy-efficient normally off applications.  _IEEE Solid-State Circuits Lett._ 1, 186–189 (2018).
  * (39) Dong, P. _et al._ Wavelength-tunable silicon microring modulator.  _Opt. Express_ 18, 10941 (2010).
  * (40) Pantouvaki, M. _et al._ 56Gb/s ring modulator on a 300mm silicon photonics platform.  _Proc. 2015 European Conference on Optical Communication (ECOC)_ , 1–3 (2015).
  * (41) Savoj, J., Abbasfar, A., Amirkhany, A., Jeeradit, M. & Garlepp, B. W.  A 12-GS/s phase-calibrated CMOS digital-to-analog converter for backplane communications.  _IEEE J. Solid-State Circuits_ 43, 1207–1216 (2008).
  * (42) Alphacore.  ADC (Analog-to-Digital converter) A6B12G.
  * (43) Lee, Y. & Chen, W.  A 20-Gb/s, 2.4 pJ/bit, fully integrated optical receiver with a baud-rate clock and data recovery.  In _2018 IEEE International Symposium on Circuits and Systems (ISCAS)_ , 1–4 (2018).
  * (44) Peng, H.-T., Nahmias, M. A., de Lima, T. F., Tait, A. N. & Shastri, B. J.  Neuromorphic photonic integrated circuits.  _IEEE J. Sel. Top. Quantum Electron._ 24, 1–15 (2018).
  * (45) Bogaerts, W. & Chrostowski, L.  Silicon photonics circuit design: Methods, tools and challenges.  _Laser Photonics Rev._ 12, 1700237 (2018).
  * (46) Giewont, K. _et al._ 300-mm monolithic silicon photonics foundry technology.  _IEEE J. Sel. Top. Quantum Electron._ 25, 1–11 (2019).
  * (47) Buckley, S. _et al._ All-silicon light-emitting diodes waveguide-integrated with superconducting single-photon detectors.  _Appl. Phys. Lett._ 111, 141101 (2017).
  * (48) Zhou, Z., Yin, B. & Michel, J.  On-chip light sources for silicon photonics.  _Light Sci. Appl._ 4, e358–e358 (2015).
  * (49) Gaeta, A. L., Lipson, M. & Kippenberg, T. J.  Photonic-chip-based frequency combs.  _Nat. Photonics_ 13, 158–169 (2019).
  * (50) Goodfellow, I., Bengio, Y. & Courville, A.  _Deep Learning_ (MIT Press, 2016).

## Acknowledgements

This research is supported by the Natural Sciences and Engineering Research
Council of Canada (NSERC) and the Vector Scholarship in Artificial
Intelligence, provided through the Vector Institute.

## Author contributions

M.F. and B.S. conceived the idea. Z.G., H.M., and B.M. designed and conducted
the experiments. M.F. created the Python simulation. M.F., M.A., and S.S.
analyzed the data. V.S., P.P., and B.S. supervised the research and
contributed to the general concept and interpretation of the results. All
authors discussed the results and contributed to the manuscript.

## Competing Interests

The authors declare no competing interests.