# Dictionary Learning with Accumulator Neurons

Gavin Parpart<EMAIL_ADDRESS>,  Yijing Watkins
<EMAIL_ADDRESS>,  Carlos González<EMAIL_ADDRESS>Pacific Northwest National Laboratory902 Battelle BlvdRichlandWAUSA99354 ,
Edward Kim<EMAIL_ADDRESS>,  Jocelyn Rego<EMAIL_ADDRESS>,  Andrew
O’Brien<EMAIL_ADDRESS>,  Steven C. Nesbit<EMAIL_ADDRESS>Drexel
University3141 Chestnut StPhiladelphiaPAUSA19104 ,  Terrence C. Stewart
<EMAIL_ADDRESS>National Research Council of CanadaUniversity
of Waterloo Collaboration CentreCanada and  Garrett T. Kenyon
<EMAIL_ADDRESS>Los Alamos National LaboratoryP.O. Box 1663, MS-B256Los
AlamosNMUSA87545

(2022; Date: May 2022)

###### Abstract.

The Locally Competitive Algorithm (LCA) uses local competition between non-
spiking leaky integrator neurons to infer sparse representations, allowing for
potentially real-time execution on massively parallel neuromorphic
architectures such as Intel’s Loihi processor. Here, we focus on the problem
of inferring sparse representations from streaming video using dictionaries of
spatiotemporal features optimized in an unsupervised manner for sparse
reconstruction. Non-spiking LCA has previously been used to achieve
unsupervised learning of spatiotemporal dictionaries composed of convolutional
kernels from raw, unlabeled video. We demonstrate how unsupervised dictionary
learning with spiking LCA (S-LCA) can be efficiently implemented using
accumulator neurons, which combine a conventional leaky-integrate-and-fire
(LIF) spike generator with an additional state variable that is used to
minimize the difference between the integrated input and the spiking output.
We demonstrate dictionary learning across a wide range of dynamical regimes,
from graded to intermittent spiking, for inferring sparse representations of
both static images drawn from the CIFAR database as well as video frames
captured from a DVS camera. On a classification task that requires
identification of the suite from a deck of cards being rapidly flipped through
as viewed by a DVS camera, we find essentially no degradation in performance
as the LCA model used to infer sparse spatiotemporal representations migrates
from graded to spiking. We conclude that accumulator neurons are likely to
provide a powerful enabling component of future neuromorphic hardware for
implementing online unsupervised learning of spatiotemporal dictionaries
optimized for sparse reconstruction of streaming video from event based DVS
cameras.

sparse coding, dynamic vision sensors, local competitive algorithms, spiking
neural networks, dictionary learning, accumulator neurons, leaky integrator
neurons

††copyright: acmcopyright††journalyear: 2022††doi:
XXXXXXX.XXXXXXX††conference: International Conference on Neuromorphic Systems;
July 27–29, 2022; Oak Ridge, TN††ccs: Theory of computation Unsupervised
learning and clustering††ccs: Theory of computation Online learning
theory††ccs: Computer systems organization Real-time systems††ccs: Computer
systems organization Sensors and actuators

##  1\. Introduction

Spiking neural networks (SNNs) are computational models that mimic discrete
binary output of biological neural networks. Compared with artificial neural
networks (ANN), SNNs incorporate leaky-integrate-and-fire (LIF) dynamics that
increases both algorithmic and computational complexity. Whereas the
individual neurons in ANNs are often implemented as simple ReLU transfer
functions that require no adjustment of free parameters, SNNs often utilize
LIF neurons whose time constants and firing thresholds must be carefully
adjusted, traditionally by hand, to achieve the desired input/output behavior.

The justification for such increased complexity is two fold. First, by using
dedicated, potentially analog, circuit elements to instantiate individual
neurons and by exploiting the low-bandwidth of event-based communication
enabled by SNNs, such networks can be implemented in extremely low-power
neuromorphic hardware (Boahen, 2017), enabling real-time remote applications
that depend on scavenged power sources such as solar recharge. Second, there
is evidence that biological neural circuits utilize spike timing to transmit
information more rapidly and to dynamically bind distributed features via
synchronous oscillations (Rullen and Thorpe, 2001) (Thorpe et al., 2001)
(Stephens et al., 2006) (Kenyon, 2010). The potential for mimicking the
dynamics of biological neural networks in fast, low-power neuromorphic
processors has motivated several efforts to develop such devices (Orchard et
al., 2021) (Davies et al., 2018) (Akopyan et al., 2015) (Painkras et al.,
2012).

Sparse coding accounts for a variety of experimentally measured linear and
non-linear response properties of V1 simple cells (Zhu and Rozell, 2013)
(Rozell et al., 2008) (Paiton et al., 2020) (Teti et al., 2020) and can be
implemented in a biologically-plausible manner in terms of local competition
using the Locally Competitive Algorithm (Rozell et al., 2008).
Algorithmically, sparse coding employs an overcomplete set of non-orthogonal
basis functions (feature vectors) to infer a sparse combination of non-zero
activation coefficients that most accurately reconstruct each input image.
Most relevant for the present study, sparse coding provides a powerful and
robust technique for the unsupervised learning of both static and
spatiotemporal dictionaries for image and video classification, respectively.
Importantly, sparse coding can be readily extended to the processing of
streaming event data from DVS cameras by neuromorphic processors (Lundquist et
al., 2016) (Lundquist et al., 2017) (Watkins et al., 2019) (Watkins et al.,
2018) (Kim et al., 2017). Given that DVS camera data is typically noisy and
sparse coding is a powerful denoising tool (Watkins et al., 2019) (Watkins et
al., 2018), we hypothesize that sparse coding may be ideal for inferring
discriminative and robust representations from noisy event training.

This paper demonstrates an efficient procedure based on accumulator neurons
(Voelker et al., 2020) to greatly facilitate the interpolation between non-
spiking and spiking neural networks (i.e. ANNs to SNNs). Accumulator neurons
combine a conventional leaky-integrate-and-fire (LIF) spike generator with an
additional state variable that is used to minimize the difference between the
integrated input and the spiking output. Unlike conventional LIF models,
accumulator neurons can produce multiple spikes on a single time step.
Qualitatively, the behavior of an accumulator neuron is determined by an
adjustable spike height. When the spike height is small in comparison to the
change in the membrane potential produced by the synaptic input on a given
time step, an accumulator neuron will fire multiple spikes so as to achieve
maximum fidelity between input and output on that time step. When the spike
height is large compared to the change in the membrane potential on a given
time step, the difference between input and output ”accumulates” until a
single spike can approximately make up the difference. Accumulator neurons,
thus, ensure that the input/output relationship remains approximately
satisfied over time and that the overall dynamics of the neural system remains
approximately unchanged even as the output of the individual neurons shift
from graded to intermittent spiking.

Starting with LCA models employing non-spiking neurons, whose implementation
has been extensively studied, we swap in accumulator neurons and increase the
spike height so as to achieve progressively more realistic spiking behavior.
Remarkably, we find that the resulting models remain able to learn spatial and
spatiotemporal dictionaries, respectively, that support high-quality
reconstructions as well as classification performance using a linear
classifier that is competitive with what is achieved by a fully supervised
deep CNN classifier. Our results suggest that accumulator neurons can be an
important enabling strategy for the online acquisition of adaptive behaviors
by intelligent autonomous systems equipped with event-driven DVS cameras that
provide sensory input to ultra-lightweight and low-power neuromorphic
processors.

##  2\. Datasets

###  2.1. CIFAR-10

The CIFAR-10 (Krizhevsky, 2009) is a standard dataset of labeled, thumbnail
sized images used for benchmarking computer vision algorithms in the field of
machine learning. The dataset is comprised of 60,000 32x32 RGB images from 10
classes. In this work, we center cropped the 16x16 images from the original
images to reduce processing time.

###  2.2. Poker-DVS

The silicon retina of Dynamic Vision Sensor (DVS) camera is a form of imaging
technology inspired by biological vision (Lichtsteiner P., 2008). A DVS camera
only measures and transmits signed event data when the absolute value of a
pixel’s intensity changes beyond a predefined threshold. The resulting video
resembles images run through an edge detection algorithm because light
intensity changes tend to mostly occur at the edges (figure 1). This
sensitivity to object boundaries allows the silicon retina camera to capture
very fast dynamic events with relatively small bandwidth.

Figure 1. Example frames for each Poker-DVS Class. Supra-threshold changes in
local pixel intensity, corresponding to positive (red) and negative (blue)
events, were summed over an 1 ms time window to produce a single frame.

The Poker-DVS dataset was obtained by fast browsing of a specially made poker
card decks in front of a DVS camera for approximately two seconds (Serrano-
Gotarredona and Linares-Barranco, 2015). It includes extremely high-speed and
noisy events across 35x35 pixels with 4 classes: clubs, diamonds, hearts and
spades. For 48 training samples and 20 validation samples in this database,
each card appeared on a DVS camera for about 20–30 ms. In this work, we
accumulated the events for each card into 1 ms frames. Example frames for each
class illustrated are in figure 1.

##  3\. Approach

###  3.1. Sparse Coding

Given an overcomplete basis, sparse coding algorithms seek to identify the
minimal set of generators that most accurately reconstruct each input image.
In neural terms, each neuron is a generator that adds its associated feature
vector to the reconstructed image with an amplitude equal to its activation.
For any particular input image, the optimal sparse representation is given by
the vector of neural activations that minimizes both image reconstruction
error and the number of neurons with non-zero activity. Formally, finding a
sparse representation involves finding a minimum of the following cost
function:

(1) $E(\ \overrightarrow{I},\phi,\
\overrightarrow{a})=\min\limits_{\\{\overrightarrow{a},\,\phi\\}}\left[\,\frac{1}{2}||\overrightarrow{I}-\phi*\overrightarrow{a}||^{2}+\lambda||\overrightarrow{a}||_{1}\right],$

In Eq. (1), $\overrightarrow{I}$ is an image unrolled into a vector, and
$\phi$ is a dictionary of feature kernels that are convolved with the sparse
representation $\overrightarrow{a}$. The factor $\lambda$ is a tradeoff
parameter; larger $\lambda$ values encourage greater sparsity (fewer non-zero
coefficients) at the cost of greater reconstruction error. Both the sparse
representation $\overrightarrow{a}$ and the dictionary of feature kernels
$\phi$ can be determined by a variety of standard optimization methods (Rozell
et al., 2008).

Our approach to compute a sparse representation for a given input image is
based on a convolutional generalization of a rectifying Locally Competitive
Algorithm (LCA) (Rozell et al., 2008). Once a sparse representation for a
given input image has been found, the basis elements associated with non-zero
activation coefficients are adapted according to a local Hebbian learning rule
that further reduces the remaining reconstruction error.

LCA finds a local minimum of the cost function defined in Eq. (1) by
introducing the dynamical variables (membrane potentials) $\mathbf{u}$ such
that the output $\mathbf{a}$ of each neuron is given by a soft-threshold
transfer function, with threshold $\lambda$, of the membrane
potential(Christopher J. Rozell and Olshausen, 2008):

(2) $\mathbf{a}=\begin{cases}u-\lambda,&\mathbf{u}>\lambda\\\
0,&otherwise\end{cases}$

The cost function defined in equation (1) is then minimized by taking the
gradient of the cost function with respect to $\mathbf{a}$ and solving the
resulting set of coupled differential equations for the membrane potentials
$\mathbf{u}$:

(3) $\mathbf{\dot{u}}\propto-\dfrac{\partial
E}{\partial\mathbf{a}}=-\mathbf{u}+\mathbf{\Phi}^{T}\\{\mathbf{I}-\mathbf{\Phi}T_{\lambda}(\mathbf{u})\\}+T_{\lambda}(\mathbf{u}).$

An update rule for feature kernels can be obtained by taking the gradient of
the cost function with respect to $\mathbf{\Phi}$:

(4) ${\Delta}\mathbf{\Phi}\propto-\dfrac{\partial
E}{\partial\mathbf{\Phi}}=\mathbf{a}\otimes\\{I-\mathbf{\Phi}\mathbf{a}\\}=\mathbf{a}\otimes\mathbf{R}$

where we introduced an intermediate residual layer $\mathbf{R}$ corresponding
to the sparse reconstruction error.

Figure 2. A LCA model that supports unsupervised dictionary learning via a
residual or sparse reconstruction error layer.

For non-spiking LCA, online unsupervised dictionary learning is achieved via a
two step process: First, a sparse representation for a given input is obtained
by integrating Eq. (3), after which Eq. (4) is evaluated to slightly reduce
the reconstruction error given the sparse representation of the current input.

As illustrated in Figure 2, the weight update (4) resembles a local Hebbian
learning rule for $\mathbf{\Phi}$ with pre- and post-synaptic activities
$\mathbf{a}$ and $\mathbf{R}$ respectively. However, the computation of
$\mathbf{\Phi}^{T}$ as well as the normalization constraint renders the
overall dictionary learning process a non-local operation.

###  3.2. Accumulator Neurons

Accumulator neurons allow for a gradual transition from non-spiking to spiking
regimes (Voelker et al., 2020). Accumulator neurons were originally developed
so that one could train a standard non-spiking ANN using backprop and then
gradually adjust the network towards a spiking domain using the trained
weights. Importantly, throughout the transition from non-spiking to spiking
output, the average output of the accumulator neurons stays constant.

The core idea is to think of spiking as a discretization process, which then
accumulates the error due to each discretization (i.e. the rounding error)
onto the next time step. For example, consider a non-spiking neuron whose
output varies between 0 and 1 (such as a sigmoid neuron). If we discretize the
output to steps of size 1, then the only outputs will be 0 or 1. If the
sigmoid neuron would output 0.5, then the discretized version could output a 0
on the first timestep, and a 1 on the next timestep, and then continue
alternating. In this way the average output continues to be 0.5, even though
it is now a spiking neuron.

The way constancy of output is achieved is to take the error in the output
(i.e. the difference between the original neuron’s output and the actual
discretized output) and add it to the target output on the next timestep.
Importantly, we can adjust the size of the discretization steps, using a
parameter $s$ ($1/\omega$ in (Voelker et al., 2020)) denoting the size of each
step in the output (i.e. if the allowed outputs are 0, 0.1, 0.2, 0.3, etc.
then $s=0.1$). If we interpret each step as a spike, then we can think of this
as a spiking neuron that may fire more than one time per timestep, and each
firing denotes an output of $s$). For this reason, we can think of $s$ as a
“spike height”.

Figure 3. Top to bottom. spike height discretization factor ($s$) = 0.01, 0.1,
0.5, 1, 5. The example shows a gradual transition from non-spiking to
intermittent spiking output using accumulator neurons with corresponding spike
heights given by $s$.

This simple mechanism has a few features worth pointing out. First, as $s$
approaches 0, the behaviour approaches that of whatever original non-spiking
neuron model we started from. Second, if $s$ is the maximum output value from
the original neuron (and if the neuron always outputs a non-negative number),
then we will have a neuron that will only spike at most once per timestep
(i.e. a standard spiking neuron). For values in-between, we can either think
of the neuron spiking multiple times per timestep, or we can think of it as a
single spike but the magnitude of that spike takes on a few different discrete
values. Third, if $s$ is much larger than the maximum output, then the neuron
will still only spike at most once per timestep, but will also have much more
temporal sparsity (for example, if $s=10$ for a sigmoid neuron, the neuron
will spike at most once every 10 timesteps). The $s$ parameter thus allows for
trading off sparsity and accuracy, while still always maintaining the same
average output from the neuron. Examples showing this gradual tradeoff as
spike height discretization factor $s$ is adjusted are shown in Figure 3.

Some neuromorphic hardware platforms such as SpiNNaker (Painkras et al., 2012)
and Loihi 2 (Orchard et al., 2021) offer direct support for discretized spike
magnitudes, so elements functionally similar to accumulator neurons may be
efficiently supported even with intermediate $s$ values where multiple spikes
occur per timestep.

##  4\. Results

We evaluated unsupervised dictionary learning for sparse reconstruction and
classification performance using both non-spiking and spiking LCA models,
employing rate-coded and accumulator neurons, respectively, on both CIFAR10
static images and video frame captured from Poker-DVS events. We implemented
both LCA and S-LCA in Nengo (Bekolay et al., 2014), an open source neural
simulation toolbox for building large-scale functional brain models, as well
as in PyTorch (Paszke et al., 2019). By gradually transitioning from a non-
spiking to a spiking LCA model, we demonstrate that dictionary learning
supports both sparse reconstruction and classification performance that
remains approximately constant despite the enormous range of dynamical regimes
considered. For large spike heights, corresponding to more spike-based output,
we applied a low pass filter to the sparse latent representations to
temporally average over individual spike events.

###  4.1. CIFAR-10

####  4.1.1. Unsupervised Dictionary Learning

(a) Initial state.

(b) Middle state.

(c) Final state.

Figure 4. Top 256 most active dictionary elements using rate-coded neurons.

We demonstrate how unsupervised dictionary learning using spiking LCA (S-LCA)
can be accomplished by starting with rate-coded non-spiking neurons and
employing accumulator neurons to transition the model to a spiking regime. LCA
allows for both unsupervised dictionary learning and inference to be performed
in a manner compatible with the constraints of recent architectures, such as
the Intel Loihi research chip (Orchard et al., 2021).

Non-spiking LCA was implemented using leaky integrator neurons with a ReLU-
like transfer function with a finite threshold. S-LCA models were implemented
using accumulator neurons with spike height of 4. Both LCA and S-LCA models
consisted of a single inference layer containing 3840 features($\simeq 5$
times overcomplete). We initialized both models’ dictionaries by sampling from
a normal distribution with mean 0 and variance 0.01, and then rescaled each
dictionary element to have unit L2 norm. We used center cropped $16\times 16$
pixels CIFAR-10 images as input, where the input spike firing rate for each
pixel is proportional to the pixel value.

The evolution of the dictionary learning using rate-coded neurons over one
epoch is illustrated in Figure 4. The evolution of the dictionary learning
using accumulator neurons with spike height set to 4 over one epoch is
illustrated in Figure 5. Despite the enormous differences in dynamics between
the two models, dictionary learning was largely unaffected.

(a) Initial state.

(b) Middle state.

(c) Final state.

Figure 5. Top 256 most active dictionary elements using accumulator neurons
with $s$ = 4.

Figure 7 shows representative neurons with different activation levels in the
LCA model over a single display period (2000 time steps or 2s). The least
active representative neuron was only above threshold at the beginning of the
display period. On the other hand, the most active representative neurons were
present with gradually increased activation over the display period. Note the
different activity scales on the y-axis. Figure 6 shows representative
activations in the S-LCA models using accumulator neurons with spike height =
4 over one display period. At spike height of 4, the least active accumulator
neuron fires less than one spike per time step on average. In contrast, the
most active accumulator neuron fires up to three spikes per time step.

(a) Low activation .

(b) Median activation.

(c) High activation.

Figure 6. Activation of representative neurons in the LCA model over one
display period.

(a) Low activation .

(b) Median activation.

(c) High activation.

Figure 7. The activation of representative neurons in the S-LCA model with $s$
= 4 over one display period.

####  4.1.2. Sparse Reconstructions

We measured the quality of the center cropped 16x16 CIFAR10 reconstructions by
using the root mean square error (RMSE) metric with image data between 0 and
1. Figure 8 shows examples of original and reconstructed images based on the
trained dictionaries from the non-spiking LCA using rate-coded neurons to the
S-LCA model using accumulator neurons with spike height of 4. Over 500 images,
the two models had similar sparsity values of 1.82% and 2.17%, respectively,
as measured by average number non-zero coefficient per image. After
normalizing both the original and reconstructed images between 0 and 1, sparse
reconstructions generated by non-spiking LCA had an RMSE of 0.0195 whereas
sparse reconstructions generated by S-LCA using accumulator neurons had an
RMSE of 0.0779, which was reduced to an RMSE of 0.0167 after applying a 100ms
low-pass filter to the sparse latent representations to temporally average
over individual spike events. Our results indicate that S-LCA using
accumulator neurons with spike height of 4 and a 100ms first-order low pass
filter yields lower values of RMSE compared to the non-spiking LCA using rate-
coded neurons. The origin of this improvement could be related to the slightly
lower sparsity of the S-LCA model.

(a) Inputs.

(b) LCA Reconstructions. RMSE = 0.0195.

(c) S-LCA Reconstructions with Spike Height of 4. RMSE = 0.0779.

(d) S-LCA Reconstructions with Spike Height of 4 and low-pass filter (100ms).
RMSE = 0.0167.

Figure 8. The original 16x16 RGB CIFAR10 images (top row), the reconstructions
by LCA (1st middle row), the reconstructions by S-LCA model using accumulator
neurons with $s$ = 4 (2nd middle row) and the reconstructions by S-LCA model
using accumulator neurons with $s$ of 4 and 100ms low-pass filter(bottom row).

###  4.2. Silicon Retina DVS Camera Events

####  4.2.1. Unsupervised Spatiotemporal Feature Learning

To evaluate the performance of S-LCA using accumulator neurons on silicon
retina DVS camera events, we trained spatiotemporal features in an
unsupervised manner on the Poker-DVS dataset. The Poker-DVS dataset (Serrano-
Gotarredona and Linares-Barranco, 2015) is made by fast browsing of a poker
deck in front of a DVS event camera. The dataset consists of 4 card types:
spades, hearts, diamonds, and clubs. For this work, we created each video
frame by accumulating over 1 ms events per card, as illustrated in figure 1.

We created video sequences of frames using a 5 frame window within the same
Poker-DVS class. By requiring both LCA and S-LCA to optimize features for
sparse reconstruction of all 5 frames, we anticipated the dictionary would
learn spatiotemporal features corresponding to the the target classes even
though learning was unsupervised. We measure the performance of both LCA and
S-LCA by the root mean squared error (RMSE) of the sparse reconstruction on
holdout test images. Additionally, we train a perceptron classifier on the
sparse latent representations inferred by the two LCA models on the training
set. We then capture the corresponding latent representations on the 5-frame
validation video sequences and evaluate classification accuracy.

For non-spiking LCA, we trained the model for 30 epochs until the dictionary
was roughly stable, showing each image for 100ms, with the input set to zero
for 100ms before each image. We tuned the LCA threshold across the range 0.4
to 1. For S-LCA implemented with accumulator neurons, we used the best
threshold value from the non-spiking LCA model, and trained the S-LCA model
for 10 epochs at various spike heights. As the spike height increases, the
maximum number of spikes per time-step decreases from 70 at $s=1$ to 1 at
$s=20$. At the maximum spike height of 20, the accumulator neurons behave
close to LIF neurons, typically firing less than 1 time per time-step. We also
implemented a moving average to smooth the sparse latent representations
inferred by S-LCA. We use the moving average for calculating updates to the
dictionary, computing sparse reconstructions for comparison with non-spiking
LCA, and for training the linear perceptron classifier. The dynamics of the
S-LCA algorithm other than dictionary updates employed no low-pass filtering
and used only the spiking output of accumulator neurons.

The evolution of the dictionary learning over 30 epochs using non-spiking LCA
is illustrated in Figure 9. The evolution of the dictionary learning using
accumulator neurons with spike height set to 10 over 30 epochs is illustrated
in Figure 10.

(a) Initial state.

(b) Middle state.

(c) Final state.

Figure 9. Top 10 most active dictionary elements using the LCA model. Each row
is a time-flattened dictionary element. The dictionary elements are randomly
initialized in its initial state whereas the final state is how the dictionary
elements converge at the last epoch

(a) Initial state.

(b) Middle state.

(c) Final state.

Figure 10. Top 10 most active dictionary elements using S-LCA of $s=10$. Each
row is a time-flattened dictionary element. The dictionary elements are
randomly initialized in its initial state whereas the final state is how the
dictionary elements converge at the last epoch

(a) Initial state.

(b) Middle state.

(c) Final state.

Figure 11. Top 10 most active dictionary elements using S-LCA of $s=10$ with
40 ms averaging applied as a low-pass filter for reconstructions. Each row is
a time-flattened dictionary element. The dictionary elements are randomly
initialized in its initial state whereas the final state is how the dictionary
elements converge at the last epoch

####  4.2.2. Spatiotemporal Sparse Reconstructions

(a) Inputs.

(b) LCA Reconstructions. RMSE = 0.3530

(c) S-LCA Reconstructions $s=10$. RMSE = 0.6405

(d) S-LCA Reconstructions $s=10$ with 40ms Moving Average. RMSE = 0.3592

Figure 12. The original Poker-DVS frames (top row), the reconstructions by LCA
(second row), the reconstructions by S-LCA using accumulator neurons with
$s=10$ (third row), and using $s=10$ with a 40 ms moving average applied
(bottom row). The x-axis on each of the rows discretizes between each input
while the y-axis shows every individual frame that forms part of the input
window

The LCA model was able to learn a dictionary effectively on the DVS frames,
with the most used elements shown in Figure 9. The most used elements
converged towards classes over time. Figure 12 shows the reconstructions for
various spike heights and RMSE is detailed in Table 1 with image data between
-1 and 1. Similar to CIFAR10, as the spike height increased, the
reconstruction accuracy decreased but returned to the performance of non-
spiking LCA with the inclusion of a low-pass filter.

$s$ | No Averaging | 40ms Moving Avg. | Sparsity of 40ms Avg.
---|---|---|---
1 | 0.3788 | 0.4012 | 90.23%
5 | 0.4810 | 0.3629 | 86.62%
10 | 0.6405 | 0.3592 | 87.40%
20 | 0.9958 | 0.3558 | 85.51%
Table 1. Validation RMSE for various spike heights with image data between -1
and 1. All runs with a v1 threshold of 0.6 and a dictionary size of $1/2$.

####  4.2.3. Classification

We focused on the Poker-DVS dataset for classification due to its smaller size
and complexity relative to CIFAR10. To evaluate the impact of the LCA
threshold and dictionary size on classification performance, we passed the
sparse latent representations inferred by non-spiking LCA to a linear
perceptron. LCA thresholds ranged from 0.4 to 1 for dictionary sizes of $1/8$
and $1/16$ times under-complete. As a baseline, we compared classification
performance obtained using a single hidden layer of sparse latent
representations passed to a linear perceptron to the classification
performance achieved by a 3-Layer CNN using identical training and validation
splits. Each layer consists of a 3D convolution with kernel size 3, 3D batch
normalization, ReLu activation, and a 3D max pooling with kernel size 2. The
results are shown in Figure 13. We observed that the accuracy fluctuated
substantially $(\pm~{}2\%)$ with different initial weights and so we averaged
results across 10 runs. Overall LCA thresholds between 0.6 to 0.8 had higher
accuracy on average. Increasing the dictionary size from $1/16$ to $1/8$ also
improved accuracy slightly, but further increasing it to $1/2$ with an LCA
threshold of 0.6 did not impact accuracy ($92.2\%\pm 0.8$).

Classification results for S-LCA based on accumulator neurons are depicted in
Figure 14. Overall accuracy dropped substantially as spike height increased,
but adding a moving average to the sparse latent representations inferred by
S-LCA dramatically mitigated this impact. At a spike height of 20, where
accumulator neurons behave much like a ”single spike” LIF neuron, the accuracy
is still comparable to non-spiking LCA.

Figure 13. Average validation accuracy for LCA with various threshold.
Averages are across 10 runs with 95% confidence intervals shown. Figure 14.
Single run validation accuracy for LCA and S-LCA based on accumulator neurons
with and without a moving average that smoothed over individual spike events.
All models used a dictionary size of $1/2$ under-complete and an LCA threshold
of 0.6.

##  5\. Conclusion

In this work, we demonstrate efficient interpolation from non-spiking to
spiking LCA models using accumulator neurons and find essentially no
degradation in both sparse reconstruction quality and classification
performance. First, we demonstrate how unsupervised dictionary learning using
a locally competitive algorithm LCA can shift from rate-coded neurons to
accumulator neurons on both CIFAR10 static images and Poker-DVS video frames
with essentially no hand tuning of parameters during the interpolation.
Second, we find that sparse reconstruction remains at the same quality level
when interpolating from non-spiking to spiking S-LCA models employing
accumulator neurons whether employing static or spatiotemporal dictionaries.
Furthermore, we observe no degradation in classification performance when
migrating from a non-spiking to a spiking LCA model. Moreover, the
classification performance of a single unsupervised hidden layer based on a
sparse latent representation passed to a linear perceptron is competitive with
a moderately deep, fully-supervised CNN classifier. Importantly, a classifier
based on LCA followed by a linear perceptron requires no backprop and allows
for unsupervised dictionary learning, reconstruction, and classification to be
performed in a manner compatible with neuromorphic architectures, such as the
Intel Loihi research chip.

Unsupervised dictionary learning via sparse coding accounts for many aspects
of cortical development. Our results suggest accumulator neurons provide a
powerful enabling strategy for endowing future neuromorphic processors with
the ability to achieve online unsupervised learning of spatial temporal
dictionaries and object recognition and sparse reconstructions of steaming DVS
events. Future investigations might explore how to process video frames
captured from shorter windows that consist of fewer DVS events, in order to
utilize the spatiotemporal information available on very short time scales.
Future investigations might additionally explore how over-complete
dictionaries might permit the learning of more discriminative spatiotemporal
features. The effect on classification performance of using higher spike
heights and replacing accumulator neurons with simpler LIF neurons, which can
generate only single-bit spikes per time step, is also a promising direction
future research.

In summary, our results suggest that accumulator neurons can be an important
enabling component for online unsupervised learning with event based DVS
cameras that provide streaming video to ultra-lightweight and low-power
neuromorphic processors.

##  6\. Acknowledgments

We gratefully acknowledge support from the Advanced Scientific Computing
Research (ASCR) program office in the Department of Energy’s (DOE) Office of
Science, award no.77902.

## References

  * (1)
  * Akopyan et al. (2015) Filipp Akopyan, Jun Sawada, Andrew Cassidy, Rodrigo Alvarez-Icaza, John Arthur, Paul Merolla, Nabil Imam, Yutaka Nakamura, Pallab Datta, Gi-Joon Nam, et al. 2015\.  Truenorth: Design and tool flow of a 65 mw 1 million neuron programmable neurosynaptic chip.  _IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems_ 34, 10 (2015), 1537–1557.
  * Bekolay et al. (2014) Trevor Bekolay, James Bergstra, Eric Hunsberger, Travis DeWolf, Terrence Stewart, Daniel Rasmussen, Xuan Choo, Aaron Voelker, and Chris Eliasmith. 2014.  Nengo: a Python tool for building large-scale functional brain models.  _Frontiers in Neuroinformatics_ 7, 48 (2014), 1–13.  https://doi.org/10.3389/fninf.2013.00048
  * Boahen (2017) K. Boahen. 2017\.  A neuromorph’s prospectus.  _Computing in Science Engineering_ 19, 2 (March 2017), 14–28.  https://doi.org/10.1109/MCSE.2017.33
  * Christopher J. Rozell and Olshausen (2008) Richard G. Baraniuk Christopher J. Rozell, Don H. Johnson and Bruno A. Olshausen. 2008\.  Sparse coding via thresholding and local competition in neural circuits.  _Neural Computation_ 20, 10 (2008), 2526–2563.
  * Davies et al. (2018) Mike Davies, Narayan Srinivasa, Tsung-Han Lin, Gautham Chinya, Yongqiang Cao, Sri Harsha Choday, Georgios Dimou, Prasad Joshi, Nabil Imam, Shweta Jain, et al. 2018\.  Loihi: A neuromorphic manycore processor with on-chip learning.  _IEEE Micro_ 38, 1 (2018), 82–99.
  * Kenyon (2010) Garrett T Kenyon. 2010\.  Extreme synergy: Spatiotemporal correlations enable rapid image reconstruction from computer-generated spike trains.  _Journal of vision_ 10, 3 (2010), 21–21.
  * Kim et al. (2017) Edward Kim, Darryl Hannan, and Garrett T. Kenyon. 2017\.  Deep Sparse Coding for Invariant Multimodal Halle Berry Neurons.  _CoRR_ abs/1711.07998 (2017).  arXiv:1711.07998 http://arxiv.org/abs/1711.07998
  * Krizhevsky (2009) Alex Krizhevsky. 2009\.  _Learning multiple layers of features from tiny images_.  Technical Report.
  * Lichtsteiner P. (2008) Delbruck T Lichtsteiner P., Posch C. 2008\.  A 128 X 128 120 dB 15 us Latency Asynchronous Temporal Contrast Vision Sensor.  _IEEE Journal of Solid State Circuits_ 43, 2 (2008), 566–575.
  * Lundquist et al. (2017) Sheng Y Lundquist, Melanie Mitchell, and Garrett T Kenyon. 2017\.  Sparse Coding on Stereo Video for Object Detection. In _workshop on Learning with Limited Labeled Data: Weak Supervision and Beyond, NIPS 2017_. NIPS.
  * Lundquist et al. (2016) Sheng Y Lundquist, Dylan M Paiton, Peter F Schultz, and Garrett T Kenyon. 2016.  Sparse encoding of binocular images for depth inference. In _Image Analysis and Interpretation (SSIAI), 2016 IEEE Southwest Symposium on_. IEEE, 121–124.
  * Orchard et al. (2021) Garrick Orchard, Edward Paxon Frady, Daniel Ben Dayan Rubin, Sophia Sanborn, Sumit Bam Shrestha, Friedrich T. Sommer, and Mike Davies. 2021.  Efficient Neuromorphic Signal Processing with Loihi 2\.  _CoRR_ abs/2111.03746 (2021).  arXiv:2111.03746 https://arxiv.org/abs/2111.03746
  * Painkras et al. (2012) E. Painkras, L. A. Plana, J. Garside, S. Temple, S. Davidson, J. Pepper, D. Clark, C. Patterson, and S. Furber. 2012.  SpiNNaker: A multi-core System-on-Chip for massively-parallel neural net simulation. In _Proceedings of the IEEE 2012 Custom Integrated Circuits Conference_. 1–4.  https://doi.org/10.1109/CICC.2012.6330636
  * Paiton et al. (2020) Dylan M Paiton, Charles G Frye, Sheng Y Lundquist, Joel D Bowen, Ryan Zarcone, and Bruno A Olshausen. 2020\.  Selectivity and robustness of sparse coding networks.  _Journal of vision_ 20, 12 (2020), 10–10.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019.  PyTorch: An Imperative Style, High-Performance Deep Learning Library.  In _Advances in Neural Information Processing Systems 32_ , H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., 8024–8035.  http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
  * Rozell et al. (2008) Christopher J Rozell, Don H Johnson, Richard G Baraniuk, and Bruno A Olshausen. 2008.  Sparse coding via thresholding and local competition in neural circuits.  _Neural computation_ 20, 10 (2008), 2526–2563.
  * Rullen and Thorpe (2001) Rufin Van Rullen and Simon J Thorpe. 2001.  Rate coding versus temporal order coding: what the retinal ganglion cells tell the visual cortex.  _Neural computation_ 13, 6 (2001), 1255–1283.
  * Serrano-Gotarredona and Linares-Barranco (2015) Teresa Serrano-Gotarredona and Bernab’e Linares-Barranco. 2015\.  Poker-DVS and MNIST-DVS. Their history, how they were made, and other details.  _Frontiers in neuroscience_ 9 (2015), 481.
  * Stephens et al. (2006) Greg J Stephens, Sergio Neuenschwander, John S George, Wolf Singer, and Garrett T Kenyon. 2006\.  See globally, spike locally: oscillations in a retinal model encode large visual features.  _Biological cybernetics_ 95, 4 (2006), 327–348.
  * Teti et al. (2020) Michael Teti, Emily Meyer, and Garrett Kenyon. 2020\.  Can Lateral Inhibition for Sparse Coding Help Explain V1 Neuronal Responses To Natural Stimuli?. In _2020 IEEE Southwest Symposium on Image Analysis and Interpretation (SSIAI)_. IEEE, 120–124.
  * Thorpe et al. (2001) Simon Thorpe, Arnaud Delorme, and Rufin Van Rullen. 2001\.  Spike-based strategies for rapid processing.  _Neural networks_ 14, 6-7 (2001), 715–725.
  * Voelker et al. (2020) Aaron R. Voelker, Daniel Rasmussen, and Chris Eliasmith. 2020\.  A Spike in Performance: Training Hybrid-Spiking Neural Networks with Quantized Activation Functions.  _CoRR_ abs/2002.03553 (2020).  arXiv:2002.03553 https://arxiv.org/abs/2002.03553
  * Watkins et al. (2018) Yijing Watkins, Austin Thresher, David Mascarenas, and Garrett T. Kenyon. 2018.  Sparse Coding Enables the Reconstruction of High-Fidelity Images and Video from Retinal Spike Trains. In _Proceedings of the International Conference on Neuromorphic Systems_ (Knoxville, TN, USA) _(ICONS ’18)_. ACM, New York, NY, USA, Article 8, 5 pages.  https://doi.org/10.1145/3229884.3229892
  * Watkins et al. (2019) Yijing Watkins, Austin Thresher, Peter F. Schultz, Andreas Wild, Andrew Sornborger, and Garrett T. Kenyon. 2019.  Unsupervised Dictionary Learning via a Spiking Locally Competitive Algorithm. In _Proceedings of the International Conference on Neuromorphic Systems_ (Knoxville, TN, USA) _(ICONS ’19)_. Association for Computing Machinery, New York, NY, USA, Article 11, 5 pages.  https://doi.org/10.1145/3354265.3354276
  * Zhu and Rozell (2013) Mengchen Zhu and Christopher J. Rozell. 2013.  Visual Nonclassical Receptive Field Effects Emerge from Sparse Coding in a Dynamical System.  _PLOS Computational Biology_ 9, 8 (08 2013), 1–15.  https://doi.org/10.1371/journal.pcbi.1003191


# Zero-Reference Low-Light Enhancement via Physical Quadruple Priors

Wenjing Wang
Peking University
Huan Yang
01.AI
Jianlong Fu
Microsoft Research Asia
Jiaying Liu
Peking University
Corresponding author. This work was supported in part by the National Natural
Science Foundation of China under Grant 62332010, and in part by the Key
Laboratory of Science, Technology and Standard in Press Industry (Key
Laboratory of Intelligent Press Media Technology).

###### Abstract

Understanding illumination and reducing the need for supervision pose a
significant challenge in low-light enhancement. Current approaches are highly
sensitive to data usage during training and illumination-specific hyper-
parameters, limiting their ability to handle unseen scenarios. In this paper,
we propose a new zero-reference low-light enhancement framework trainable
solely with normal light images. To accomplish this, we devise an
illumination-invariant prior inspired by the theory of physical light
transfer. This prior serves as the bridge between normal and low-light images.
Then, we develop a prior-to-image framework trained without low-light data.
During testing, this framework is able to restore our illumination-invariant
prior back to images, automatically achieving low-light enhancement. Within
this framework, we leverage a pretrained generative diffusion model for model
ability, introduce a bypass decoder to handle detail distortion, as well as
offer a lightweight version for practicality. Extensive experiments
demonstrate our framework’s superiority in various scenarios as well as good
interpretability, robustness, and efficiency. Code is available on our project
homepage.

##  1 Introduction

Restoring images in low-light conditions is an important and challenging task
in computer vision. The goal is to unveil concealed details in poorly lit
areas, ultimately elevating the overall image quality. Over time, a large
number of algorithms have been developed to address this challenge. However,
current methods exhibit limitations due to their dependence on supervisory
information and their adaptability to unseen domains. In the following, we
review recent advancements and then present our primary contributions.

Supervised Methods. Deep learning has significantly influenced the advancement
of low-light enhancement. In 2017, Li _et al_. [31] introduced the initial
deep-based low-light enhancement model using a straightforward auto-encoder.
Subsequently, a series of studies improved the network design by incorporating
concepts from the Retinex theory [48, 63, 49], Fourier transform [16], image
processing systems [15], semantics [50], and adopting innovative architectures
like Flow-based generative models [47], vision transformers [3] and diffusion
models [65, 57]. In addition to RGB images, a body of research is dedicated to
RAW data [4, 5, 17], videos [59], and multi-modalities [52, 51]. While these
models have achieved notable success, they rely on paired data for training,
which can be inflexible and less robust in unforeseen scenarios. Recent
studies explore how to reduce this dependency on supervision.

Figure 1: Comparison with a SOTA zero-reference method: SCI [34]. The SCI
model, trained on varied datasets like LOL [48] and MIT [2], yields diverse
enhancement results. Nevertheless, none effectively maintains a consistent
lighting effect across both dark and moderately dark images. In contrast, our
model demonstrates greater robustness across various scenarios.

Unsupervised Methods. Some work transitions from the strict requirement of
pairwise pairings to only necessitating unpaired normal-low light data for
training. EnlightenGAN [19], FlexiCurve [25], and NeRCo [54] employ
adversarial learning, where discriminators are constructed to guide low-light
enhancement models, _i.e_., the generators. CLIP-LIT [27] learns prompts from
images under varied illuminations. PairLIE [7] instead learns adaptive priors
from paired low-light instances of the same scene. However, these methods
retain specific training data requirements, limiting their ability to
generalize to unknown scenarios.

Zero-Reference Methods. “Zero-reference” [10] refers to a special unsupervised
setting where neither paired nor unpaired data is available for training. This
setting is more challenging, but offers greater flexibility in practical
applications. Traditional non-deep low-light enhancement algorithms [37, 11]
can also be classified as zero-reference. These methods primarily rely on
manually designed strategies, such as histogram equalization [37] or Retinex
decomposition [11, 26]. As for deep models, Zero-DCE [10] employs a neural
network to predict the parameters of a pre-defined curve function and applies
it to the input low-light image. A suite of non-reference loss functions is
employed to guide the enhancement process. Subsequent works improved speed and
curve forms [23, 24]. As an alternative approach, RUAS [30] introduces a
neural architecture search strategy based on the Retinex rule. It also
implements several reference-free losses. Furthermore, SCI [34] streamlines
the iterative process in RUAS into a single step.

Despite the success of zero-reference methods, they often require careful
parameter-tuning and can be sensitive to the distribution of training data.
For instance, in SCI [34], varying training data leads to distinct enhanced
appearances, causing over-exposure or under-exposure as shown in Fig. 1. The
crux of the matter is that these zero-reference models lack a genuine concept
of illumination. Learning lighting knowledge without reference and depending
on artificially set parameters presents an challenging and unsolved problem.

Our Contributions. In this paper, we propose a new zero-reference low-light
enhancement framework to address the aforementioned challenges. Our central
idea is to develop an illumination-invariant prior and employ it as an
intermediary between low-light and normal light images. We devise a unique
illumination prior, named the physical quadruple prior, originating from the
Kubelka–Munk theory of light transfer. Next, we construct a prior-to-image
mapping framework solely using typical normal-light images, easily obtainable
from the Internet or existing open-source visual datasets. During this stage,
the model learns the authentic concept of bright lighting from the image
distribution. Finally, when tested on low-light images, our physical quadruple
prior automatically extracts illumination-invariant features, and prior-to-
image mapping framework transfers these features to normal light images.
Throughout this process, low-light enhancement can be achieved without
requiring any low-light data or illumination-relevant hyper-parameters.

The challenge lies in that, as our prior discards a considerable amount of
illumination-relevant information, restoring it back to images is not a
straightforward task. To address this challenge, we capitalize on the
exceptional capabilities of a pretrained large-scale generative model, Stable
Diffusion (SD) [42], and construct the prior-to-image mapping by integrating
priors as conditions to control the SD model. Unlike typical generative tasks
that require high-quality data [43, 13, 66, 67, 35], our framework is
insensitive to data quality and is trained on one of the most readily
available datasets: COCO [28]. Since SD is originally designed for tasks other
than restoration, it faces challenges in detail preservation. Hence, we
propose an bypass decoder to address the distortion issue, which also proves
useful for other image processing tasks. Finally, considering practical
applications, our framework can be used to create a lighter zero-reference
model on specific data. By employing a CNN-transformer mixed model, we distill
the complex multi-step optimization of a large diffusion model into a single
forward pass within a lightweight network. This lightweight version maintains
comparable performance while significantly improving inference speed and
computational efficiency.

In summary, thanks to our physical quadruple prior, prior-to-image framework,
the lightweight version, our approach combines interpretability, robustness,
and efficiency. Experimental results demonstrate that our model attains
favorable subjective and objective performance across diverse datasets. Our
main contributions are concluded as:

  * •

We present a zero-reference low-light enhancement model that utilizes an
illumination-invariant prior as the intermediary between different
illumination. Our model exhibits superior performance in various under-lit
scenarios without relying on any specific low-light data.

  * •

We establish the physical quadruple prior, a novel learnable illumination-
invariant prior derived from a light transfer theory. This prior captures the
essence of imaging under diverse lighting conditions, freeing low-light
enhancement from dependence on reference samples or artificially set hyper-
parameters.

  * •

We develop an effective prior-to-image mapping system by incorporating the
prior as a condition to control a pretrained large-scale generative diffusion
model. We introduce a bypass decoder to address the distortion issue, and show
that our model can be distilled into a lightweight version for practical
application.

##  2 Physical Prior-based Image Restoration

###  2.1 Motivation

Developing illumination invariant features has a long history in the domain of
image restoration. One of the most representative invariants for low-light
enhancement is the Retinex model [40]. This model posits that an image $X$ can
be decomposed into illumination $I$ and illumination-invariant reflectance
$R$, expressed as $X$ $=$ $I\odot R$, where $\odot$ denotes element-wise
multiplication. However, solving this decomposition is challenging. Existing
approaches either lean on human-crafted policies [11] or rely on paired data
with varying brightness levels [48, 63], which lack robustness when faced with
unknown scenarios.

Figure 2: The overall methodology of our zero-reference low-light enhancement
approach. Our model is trained to reconstruct images from an illumination-
invariant prior (the physical quadruple prior) in the normal light domain.
During testing, the model extracts illumination-invariant priors from low-
light images and reconstructs them into normal light images. Figure 3:  Our
illumination-invariant prior and the training process for our prior-to-image
model framework. We start by predicting the physical quadruple prior from the
input image $I$. During the training phase, the model dynamically learns the
linear mapping $\mathcal{W}$ and the layers for predicting the scale $\sigma$.
In the process of reconstructing priors into images, a static SD encoder
extracts the latent representation $z_{0}$ from the input image $I$. Following
this, we sample noisy latent $z_{t}$ based on $z_{0}$. Finally, the physical
quadruple prior is encoded by convolutional and transformer modules, and is
then merged with a frozen SD U-net to predict both noise $\epsilon$ and
$z_{0}$.

Instead of decomposing images into illumination-invariant and illumination-
related information, we suggest to solely extract illumination-invariant
features from the input image and subsequently generate illumination-related
information to reconstruct the image. In contrast to traditional Retinex-based
deep models [48, 63], our framework ensures that our deep model can comprehend
lighting without the need for paired data.

The overall training and inference pipeline of our method is illustrated in
Fig. 2. We introduce a physical quadruple prior derived from the Kubelka-Munk
theory [9] to extract illumination-invariant features. While training on
normal light images, the model simultaneously learns how to reconstruct images
from priors and learns the illumination distribution of natural images.
Acknowledging the complexity of simultaneously addressing these two tasks, we
make use of a pre-trained diffusion generative model. During testing, low-
light images are initially mapped to the physical quadruple prior and
subsequently reconstructed. Given the illumination-invariant nature of our
prior, it extracts comparable features from low-light images as it does from
normal light images. Combined with the fact that our prior-to-image model is
specifically trained to reconstruct priors into normal light images, our model
achieves low-light enhancement without the need for low-light data. In the
following, we elaborate on the details of each component.

###  2.2 Learnable Illumination-Invariant Prior

Physical Quadruple Priors. We start from the Kubelka-Munk theory [9] of light
transfer. Given wavelength $\lambda$, the energy of the incoming spectrum at
spatial location $\mathbf{x}$ on the image plane is modeled as

$\displaystyle
E(\lambda,\mathbf{x})=e(\lambda,\mathbf{x})\left((1-i(\mathbf{x}))^{2}R_{\infty}(\lambda,\mathbf{x})+i(\mathbf{x})\right),$
(1)

where $e(\lambda,\mathbf{x})$ denotes the spectrum of the light source,
$i(\mathbf{x})$ the specular reflection, and $R_{\infty}(\lambda,\mathbf{x})$
the material reflectivity. Note that when the object is matte, _i.e_.,
$i(\mathbf{x})\approx 0$, Eq. (1) can be reduced to

$\displaystyle
E(\lambda,\mathbf{x})=e(\lambda,\mathbf{x})R_{\infty}(\lambda,\mathbf{x}),$
(2)

which is the same as the Retinex model. It means that the Retinex theory is a
special case of Eq. (1).

First of all, we denote some variables for simplicity

$\displaystyle E^{\lambda}$ $\displaystyle=\frac{\partial
E(\lambda,\mathbf{x})}{\partial\lambda},\;R_{\infty}^{\lambda}=\frac{\partial
R_{\infty}(\lambda,\mathbf{x})}{\partial\lambda},$ (3) $\displaystyle
E^{\lambda\lambda}$
$\displaystyle=\frac{\partial^{2}E(\lambda,\mathbf{x})}{\partial\lambda^{2}},\;R_{\infty}^{\lambda\lambda}=\frac{\partial^{2}R_{\infty}(\lambda,\mathbf{x})}{\partial\lambda^{2}}.$
(4)

Intuitively, $E$ represents spectral intensity, $E^{\lambda}$ signifies
spectral slope, and $E^{\lambda\lambda}$ denotes spectral curvature.

Following [8], through simplifying assumptions, we can obtain a series of
invariants from Eq. (1). The primary idea is to eliminate $i$ and $e$,
retaining solely $R_{\infty}$. As $R_{\infty}$ is about material property and
is independent of illumination, the derived variable will exhibit illumination
invariance.

  * •

Assuming equal energy illumination, _i.e_., $e(\lambda,\mathbf{x})$ is reduced
to $\lambda$-independent $\tilde{e}(\mathbf{x})$, and Eq. (1) is reduced to

$\displaystyle
E(\lambda,\mathbf{x})=\tilde{e}(\mathbf{x})\left((1-i(\mathbf{x}))^{2}R_{\infty}(\lambda,\mathbf{x})+i(\mathbf{x})\right),$
(5)

Substituting Eq. (5) into $E^{\lambda}/E^{\lambda\lambda}$ gives

$\displaystyle\frac{E^{\lambda}}{E^{\lambda\lambda}}=\frac{\tilde{e}(\mathbf{x})(1-i(\mathbf{x}))^{2}R_{\infty}^{\lambda}}{\tilde{e}(\mathbf{x})(1-i(\mathbf{x}))^{2}R_{\infty}^{\lambda\lambda}}=\frac{R_{\infty}^{\lambda}}{R_{\infty}^{\lambda\lambda}},$
(6)

where illumination properties $i$ and $e$ are eliminated. As the material
property $R_{\infty}$ is independent of illumination, it establishes the
illumination-invariance of $E^{\lambda}/E^{\lambda\lambda}$. Now we derive our
first illumination invariant,

$\displaystyle H=\arctan\left({E^{\lambda}}/{E^{\lambda\lambda}}\right).$ (7)

  * •

Further assuming that the surface is matte, _i.e_. $i(\mathbf{x})\approx 0$,
then Eq. (1) is reduced to

$\displaystyle
E(\lambda,\mathbf{x})=\tilde{e}(\mathbf{x})R_{\infty}(\lambda,\mathbf{x}),$
(8)

Similarly, we derive another illumination invariant,

$\displaystyle C$
$\displaystyle=\log\left(\frac{(E^{\lambda})^{2}+(E^{\lambda\lambda})^{2}}{E(\lambda,\mathbf{x})^{2}}\right)$
$\displaystyle=\log\left(\frac{(R_{\infty}^{\lambda})^{2}+(R_{\infty}^{\lambda\lambda})^{2}}{R_{\infty}(\lambda,\mathbf{x})^{2}}\right).$
(9)

  * •

Further assuming uniform illumination, _i.e_., $\tilde{e}(\mathbf{x})$ is
reduced to a parameter $\bar{e}$, and Eq. (1) is reduced to

$\displaystyle E(\lambda,\mathbf{x})=\bar{e}R_{\infty}(\lambda,\mathbf{x}),$
(10)

Similarly, we derive our third illumination invariant,

$\displaystyle W$ $\displaystyle=\tan\left(\bigg{|}\frac{\partial
E(\lambda,\mathbf{x})}{\partial\mathbf{x}}\frac{1}{E(\lambda,\mathbf{x})}\bigg{|}\right)$
$\displaystyle=\tan\left(\bigg{|}\frac{\partial
R_{\infty}(\lambda,\mathbf{x})}{\partial\mathbf{x}}\frac{1}{R_{\infty}(\lambda,\mathbf{x})}\bigg{|}\right).$
(11)

The Kubelka-Munk theory [9] is effective for grayscale images but falls short
in describing colors. The three aforementioned illumination invariants loss
part of the color information, so we add some additional color information. We
adopted a straightforward one: the relative relationship between pixel values
of the three RGB channels.

  * •

Assuming that illumination maintains the order of colors, we propose the order
of the RGB three channels as a fundamental illumination-invariant feature,
denoted as $O$.

Learning through Neural Networks. We follow Gaussian color models [9] and
CIConv [22] to obtain priors from RGB images. First, we estimate the observed
energy $\hat{E}$ along with its derivatives $\hat{E}^{\lambda}$ and
$\hat{E}^{\lambda\lambda}$ via linear mapping:

$\displaystyle\begin{bmatrix}\hat{E}(x,y)\\\ \hat{E}^{\lambda}(x,y)\\\
\hat{E}^{\lambda\lambda}(x,y)\\\
\end{bmatrix}\\!=\\!\mathcal{W}\begin{bmatrix}R(x,y)\\\ G(x,y)\\\
B(x,y)\end{bmatrix},$ (12)

where $x$ and $y$ denote positions in the image, and $\mathcal{W}$ is a
$3\times 3$ matrix. In [9, 22], $\mathcal{W}$ is manually defined. We instead
learn it from the distribution of natural images through our prior-to-image
framework.

The spatial derivative $\partial E/\partial\mathbf{x}$ in Eq. (• ‣ 2.2) is
computed in both the x- and y-direction, denoted as $\partial
E/\partial\mathbf{x}=(E_{x},E_{y})$, with its magnitude given by $|\partial
E/\partial\mathbf{x}|=\sqrt{E_{x}^{2}+E_{y}^{2}}$. Finally, $E$, $E_{x}$, and
$E_{y}$ are estimated by convolving $\hat{E}$ with Gaussian color smoothing
and derivative filters of scale $\sigma$. $\sigma$ is predicted from the input
image. Similarly, $E^{\lambda}$ is obtained from $\hat{E}^{\lambda}$, and
$E^{\lambda\lambda}$ is obtained from $\hat{E}^{\lambda\lambda}$. Now we can
compute $H$, $C$, and $W$ from the input image.

Our last illumination invariant, the order of RGB channels, is defined as
three channels as follows,

$\displaystyle O(x,y)=\left[O_{R}(x,y),O_{G}(x,y),O_{B}(x,y)\right],$ (13)

where $O_{R}$ represents the order of the $R$ channel in RGB, normalized to
[-1,1]. $O_{G}$ and $O_{B}$ are treated similarly.

Finally, $H$, $C$, $W$, and $O$ are concatenated in the channel dimension to
form our physical quadruple prior.

Physical Explanation. Firstly, the mathematical form indicates that $W$
represents the intensity-normalized spatial derivatives of the spectral
intensity. As for $H$, according to [9], it is associated with the hue,
_i.e_., $\arctan(\lambda_{\text{max}})$ of the material. As for $C$, within a
color circle based on spectral wavelengths, hue represents the angle while
chroma is the distance from the center. Additionally, when converting a
Cartesian coordinate $(a,b)$ to polar, the angle becomes $\arctan(b/a)$, and
the radius becomes $\sqrt{(a)^{2}+(b)^{2}}$. Connecting this with Eq. (7) and
Eq. (• ‣ 2.2), we find $C$ associates with chroma. A visualization can be
found in Fig. 3. More analysis and comparisons will be presented in Sec. 3.3.

###  2.3 Prior-to-Image via Diffusion Models

Ideally, we want to retain all illumination-invariant information while
discarding lighting-relevant information. However, achieving this
decomposition is challenging and remains a unsolved problem in image modeling.
Despite our physical quadruple prior, _i.e_., $H$, $C$, $W$, and $O$,
capturing illumination-independent information from various perspectives, some
information is still lost. Consequently, reconstructing images from the prior
is non-trivial.

Figure 4: Image restoration effect of the SD decoder and ours. (a) Input image
$I$, from which we extract latent $z_{0}$. (b) $z_{0}$ decoded by the SD
decoder. (c) The distorted version of $I$. (d) $z_{0}$ decoded by our decoder
using the encoder features from $\tilde{I}$.

Instead of focusing on improving the illumination-invariant prior, we propose
leveraging the capabilities of a large-scale generative model to directly
complete the missing information. We employ Stable Diffusion v1-5 [42] and
convert it into a conditional generative mode using the ControlNet [60]
framework. Our physical quadruple prior serves as the condition to control the
SD model.

The overall framework is illustrated in Fig. 3. During training, a frozen SD
encoder is employed to map the image $I$ into a compressed latent
representation $z_{0}$. We then sample $z_{t}$ at a random time step
$t\in\\{1,...,T\\}$ using

$\displaystyle
z_{t}=\sqrt{\bar{\alpha}_{t}}z_{0}+\sqrt{1-\bar{\alpha}_{t}}\epsilon,$ (14)

where $\\{\bar{\alpha}_{t}\\}$ is a sequence of pre-defined parameters [14].
The training objective is to predict $\epsilon$ from $z_{t}$ and our prior.
Originally, SD utilizes a U-Net to predict $\epsilon$ from $z_{t}$, and this
U-Net is now frozen. A set of encode modules are added to extract features
from our quadruple prior. These features are then incorporated into the SD
U-Net. The zero convolution strategy [60] is adopted to ensure that, at the
beginning of training, new layers do not influence the original SD. During
testing, given an input image $I$, we extract the physical quadruple prior and
use it as the condition to predict $z_{0}$ in the reverse diffusion process.
Subsequently, $z_{0}$ is projected back to the image space through a decoder.

Figure 5: The training strategy of our bypass decoder. We distort the input
image $I$ into $\tilde{I}$, and allow the decoder to reconstruct $I$ using
encoder features from the distorted $\tilde{I}$.

While ControlNet has demonstrated success in various applications, applying it
directly suffers from issues including slow convergence, detail degradation,
and dependence on text prompts. To address these challenges and make it more
suitable for our image restoration task, we implement the following
improvements.

  * •

In typical diffusion models, the training objective is to predict the Gaussian
noise term:

$\displaystyle\mathcal{L}_{\text{noise}}=||\epsilon-\hat{\epsilon}||^{2}_{2}.$
(15)

We additionally minimize the difference in terms of $z_{0}$, which can
accelerate convergence. Combining Eq. (14), we derive

$\displaystyle\mathcal{L}_{z_{0}}=||z_{0}-\hat{z}_{0}||^{2}_{2}=||z_{0}-\frac{z_{t}-\sqrt{1-\bar{\alpha}_{t}}\hat{\epsilon}}{\sqrt{\bar{\alpha}_{t}}}||^{2}_{2}.$
(16)

We simple combine these two losses as the final objective

$\displaystyle\mathcal{L}_{\text{DIFF}}=\mathcal{L}_{z_{0}}+\mathcal{L}_{\text{noise}}.$
(17)

  * •

SD employs an auto-encoder (AE) to compress the image $I$ into a latent
representation $z_{0}$, reducing computational cost. However, the auto-encoder
introduces severe detail distortion. As shown in Fig. 4(b), the face of the
mounted police is completely distorted. To mitigate this issue, we support the
decoder with features from the encoder and devise an effective fine-tuning
strategy. As shown in Fig. 5, during training, we distort the input image $I$
with random illumination jittering and noise, resulting in $\tilde{I}$. The
decoder then restores $z_{0}$, combining the features $z^{1}$, $z^{2}$,
$z^{3}$ extracted from $\tilde{I}$. This progress guides the decoder to
capture details from $\tilde{I}$ while preserving the illumination
characteristics of $I$. We introduce several convolutional layers for feature
fusion and a residual block for post-processing. These additional layers are
initialized as zero or self, ensuring they have minimal impact on the original
decoding process at the beginning of training. The new decoder is named bypass
decoder. As shown in Fig. 4(d), noticeable detail restoration is achieved
through our bypass decoder. During testing, features from the input image
assist in the latent decoding process, as illustrated in Fig. 6. Our bypass
decoder utilizes $z^{1}$, $z^{2}$, and $z^{3}$, extracted from the input
image, to reconstruct details and maintain the enhanced illumination within
$\hat{z}_{0}$.

  * •

Stable Diffusion is originally designed as a text-to-image model. However,
requiring users to provide text for low-light enhancement is inconvenient. To
address this, we set the text input to always be an empty string.

Figure 6: The inference pipeline of our overall framework. Given a low-light
image, we extract its physical quadruple prior. Then, this prior serves as the
condition for predicting the latent representation $\hat{z}_{0}$ from pure
noise $\epsilon_{T}$. Lastly, the bypass decoder utilizes features extracted
by the encoder from the low-light image to map the predicted $\hat{z}_{0}$
back into images.

Denoising. Noise poses a significant challenge in low-light enhancement.
Although our prior isn’t designed for denoising purposes, we implement a
simple strategy to suppress noise. During training, we apply random Gaussian-
Poisson compound noise to the input image $I$ while extracting the physical
quadruple prior. This approach guides the model to disregard high-frequency
details and concentrate solely on low-frequency illumination-invariant
information.

Distillation for Efficiency. Diffusion models require multi-step optimization
in inference. Even with DPM-Solver++ [32], 10 steps are still cumbersome. In
the pursuit of practicality, our framework can create a more lightweight
version. In short, we construct a lightweight U-net consists of residual
blocks and integrate transformer blocks from Restormer [58] at the bottleneck.
Transformers have proven to be effective in low-level vision [53, 39, 38, 29].
Then, we make 1.7k samples using our framework to teach the lightweight model.
The training objective is solely the L1 loss. More details are provided in the
supplementary.

Table 1: Benchmarking results for low-light enhancement. Among unsupervised methods, we highlight the top-ranking scores in red and the second in blue. Additionally, we denote the training set used by each model. “LOL+” indicates a fusion of LOL and other datasets. Datasets | Train Set |  LOL [48, 55] |  MIT-Adobe FiveK [2] | Unpaired Sets
---|---|---|---|---
Metrics |  |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$ |  LOE$\downarrow$ |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$ |  LOE$\downarrow$ |  BRISQUE$\downarrow$ |  NL$\downarrow$
Supervised |  Retinex-Net [48] | LOL | 16.19 | 0.403 | 0.534 | 0.346 | 12.30 | 0.687 | 0.258 | 0.244 | 27.10 | 3.254
KinD [63] | LOL | 20.21 | 0.814 | 0.147 | 0.245 | 14.71 | 0.756 | 0.176 | 0.174 | 26.89 | 0.700
KinD++ [64] | LOL | 16.64 | 0.662 | 0.410 | 0.288 | 15.76 | 0.650 | 0.319 | 0.176 | 26.16 | 0.431
URetinex-Net [49] | LOL | 20.93 | 0.854 | 0.104 | 0.245 | 14.10 | 0.734 | 0.182 | 0.187 | 23.80 | 1.319
Retinexformer [3] | LOL | 28.48 | 0.877 | 0.117 | 0.256 | 13.87 | 0.692 | 0.222 | 0.224 | 14.77 | 1.064
Retinexformer [3] | MIT | 13.02 | 0.426 | 0.365 | 0.280 | 24.93 | 0.907 | 0.063 | 0.162 | 24.13 | 0.684
DiffLL [18] | LOL+ | 28.54 | 0.870 | 0.102 | 0.253 | 15.81 | 0.719 | 0.244 | 0.213 | 14.96 | 0.888
Unsupervised |  ExCNet [61] | test images | 16.29 | 0.455 | 0.380 | 0.295 | 14.21 | 0.719 | 0.197 | 0.197 | 19.03 | 1.563
EnlightenGAN [19] | own data | 18.57 | 0.700 | 0.302 | 0.291 | 13.28 | 0.738 | 0.203 | 0.199 | 20.65 | 0.779
PairLIE [7] | LOL+ | 19.70 | 0.774 | 0.235 | 0.278 | 10.55 | 0.642 | 0.273 | 0.225 | 29.84 | 1.471
NeRCo [54] |  LSRW [12] | 19.67 | 0.720 | 0.266 | 0.310 | 17.33 | 0.767 | 0.208 | 0.213 | 22.81 | 0.603
CLIP-LIT [27] | own data | 14.82 | 0.524 | 0.371 | 0.320 | 17.00 | 0.781 | 0.159 | 0.194 | 23.44 | 1.962
ZeroDCE [10] | own data | 17.64 | 0.572 | 0.316 | 0.296 | 13.53 | 0.725 | 0.201 | 0.191 | 21.76 | 1.569
ZeroDCE++ [23] | own data | 17.03 | 0.445 | 0.314 | 0.391 | 12.33 | 0.408 | 0.280 | 0.417 | 19.34 | 1.150
RUAS [30] | MIT | 13.62 | 0.462 | 0.346 | 0.292 | 9.53 | 0.610 | 0.301 | 0.272 | 29.91 | 2.091
RUAS [30] | LOL | 15.47 | 0.490 | 0.305 | 0.330 | 5.15 | 0.373 | 0.669 | 0.399 | 44.70 | 3.312
RUAS [30] |  FACE [56] | 15.05 | 0.456 | 0.371 | 0.292 | 5.00 | 0.366 | 0.685 | 0.398 | 46.21 | 3.633
SCI [34] | MIT | 11.67 | 0.395 | 0.361 | 0.286 | 16.29 | 0.795 | 0.143 | 0.165 | 16.73 | 0.853
SCI [34] | LOL+ | 16.97 | 0.532 | 0.312 | 0.289 | 7.83 | 0.573 | 0.360 | 0.187 | 24.46 | 1.893
SCI [34] |  FACE [56] | 16.80 | 0.543 | 0.322 | 0.297 | 10.95 | 0.684 | 0.272 | 0.205 | 18.33 | 1.335
Ours |  COCO [28] | 20.31 | 0.808 | 0.202 | 0.281 | 18.51 | 0.785 | 0.163 | 0.188 | 14.64 | 0.423
Figure 7: Example low-light enhancement results on the MIT-Adobe FiveK (top
row) and LOL datasets (bottom row).

##  3 Experiments

###  3.1 Implementation Details

Framework Development. Our framework is trained on the COCO-2017 [28] train
and unlabeled set. We employ a minibatch size of 8, conducting training over
140k steps, approximately 5 epochs, with a learning rate set at 1e-4 and the
ADAM optimizer [20]. To accommodate this sizable model within limited GPU
memory, we implement float16 precision and DeepSpeed [41]. More implementation
details can be found in the supplementary.

Compared Methods. Our model is compared with nine unsupervised low-light image
enhancement methods. Among these, EnlightenGAN [19], PairLIE [7], NeRCo [54],
and CLIP-LIT [27] utilize unpaired low-light-related data. The remaining five
methods, ExCNet [61], ZeroDCE [10], ZeroDCE++[23], RUAS[30], and SCI [34], are
zero-reference. Additionally, we present the results of six supervised methods
to show the upper bound of our task.

Benchmark Settings. We report performance on three sets of widely-used low-
light datasets. The first two sets are LOL and MIT-Adobe FiveK [2]. For LOL,
we adopt the official test sets of LOL v1 [48] and LOL v2 [55], resulting in
115 low-/normal-light image pairs. For MIT, we follow Retinexformer [3] to
split 500 pairs for testing. Additionally, we gather low-light images from
LIME [11], NPE [46], MEF [33], DICM [21], and VV [45] that have no ground
truth. This set is simply called the “unpaired set”. On LOL and MIT, we report
PSNR, SSIM, LPIPS [62], and LOE [46]. On the unpaired set, we report BRISQUE
[36] and noise level (NL) estimated by [6].

###  3.2 Benchmarking Results

Tab. 1 demonstrates that our model surpasses the majority of unsupervised
techniques and notably reduces the performance gap compared to supervised
methods. Subjective results can be found in Fig. 7 and Fig. 1. Our model is
able to more effectively suppress noise and prevent overexposure or excessive
darkness. More showcases can be found in the supplementary. SCI [34]
demonstrates good performance on training-related datasets but significantly
degrades in unseen scenarios. In contrast, our model exhibits robustness
across both LOL and MIT datasets simultaneously. This resilience stems from
our model’s ability to learn comprehensive illumination knowledge from the
physical quadruple prior and normal light images. Consequently, our model
proves to be less sensitive to specific datasets.

In Tab. 1, it’s noticeable that supervised methods often tend to overfit to
their training sets, exhibiting limited generalization to unseen domains. For
instance, Retinexformer [3] and DiffLL [18], trained with LOL, achieve lower
performance than our model on MIT, and vice versa. This experiment underscores
our model’s superior adaptability to previously unseen scenarios,
outperforming even supervised methods.

###  3.3 Ablation Studies

Figure 8: Low-light enhancement effects for different prior designs (left),
and the visualization of our physical quadruple prior (right).

Prior Design. We first analyze the effect of each element, $H$, $C$, $W$, and
$O$, in our illumination-invariant prior. The evaluation is conducted on LOL,
following the same setting as Tab. 1. Removing any element reduces the
performance. It is because deleting any element would remove a corresponding
part of information. Showing that only the four prior combinations together
can reconstruct the original image without extracting the light-related
features.

Fig. 8 provides a visual comparison. The absence of $H$ or $C$ leads to color
bias or a washed-out white appearance. As previously discussed in Sec. 2.2,
$H$ and $C$ are associated with hue and chroma. But on the right of Fig. 8, we
can also see that $H$ and $C$ are not exactly hue and chroma. This shows that
prior basically conforms to the physical explanation we have derived, but can
go on to learn more advanced features. Recall that $W$ represents intensity-
normalized spatial derivatives of spectral intensity, capturing local
illumination changes. When $W$ is omitted, the alterations in light and shadow
are completely lost (as observed in the second row of Fig. 8). Additionally,
without $O$, blue was mistakenly enhanced to orange. Ultimately, our full
version showcases the most refined details and superior contrast.

We further explore the implications of replacing our prior with alternative
representations. We consider three representative ones: (1) Naive HS channels
in the HSV color space. (2) CIConv [22], a trainable prior similar to our $W$.
(3) The reflectance estimated by Retinex-based PairLIE [7] trained on LOL. In
Tab. 2, the HS channels sacrifice significant content information,
particularly impacting SSIM and LPIPS negatively. While CIConv exhibits
illumination invariance in high-level vision tasks, it suffers from excessive
color loss in the image, leading to performance degradation.‘ In comparison
with our prior learned from COCO, the reflectance derived by PairLIE
originates from low-light data pairs in LOL. Despite being trained on the
target domain data, it still performs worse than our prior.

Figure 9: Effects of using different prior-to-image frameworks.

Prior-to-Image Framework. We discuss the importance of using a pre-trained
generative model to build the prior-to-image mapping. We explore an
alternative diffusion-based backbone, SR3 [44], recognized for its efficacy in
super resolution. However, when used to replace our prior-to-image framework,
as illustrated in Fig. 9, SR3 exhibits noticeable color bias and noise issues.
It is because, in order to strip away light-related features, our prior
discarded some of the image information, requiring the prior-to-image model to
fill in the gaps and restore the complete details.

Figure 10: Effects of different decoders in our framework. Table 2: Ablation studies on the effect of our method designs. Datasets |  LOL [48]
---|---
Metrics |  PSNR$\uparrow$ |  SSIM$\uparrow$ |  LPIPS$\downarrow$ |  LOE$\downarrow$
|  Ours w/o $H$ | 17.60 | 0.756 | 0.262 | 0.314
|  Ours w/o $C$ | 17.60 | 0.762 | 0.262 | 0.313
|  Ours w/o $W$ | 17.77 | 0.749 | 0.291 | 0.313
Prior |  Ours w/o $O$ | 18.63 | 0.764 | 0.285 | 0.315
HS channels in HSV | 18.04 | 0.562 | 0.498 | 0.410
| CIConv | 17.02 | 0.455 | 0.551 | 0.421
|  Reflectance by PairLIE [7] | 20.16 | 0.790 | 0.287 | 0.296
AE |  SD Decoder [42] | 19.26 | 0.665 | 0.243 | 0.353
Consistency Decoder [1] | 19.35 | 0.686 | 0.235 | 0.350
Ours Final Version | 20.25 | 0.807 | 0.199 | 0.278

Auto-Encoder. We showcase the impact of our bypass decoder, comparing it with
the original decoder used in SD. Additionally, we evaluate the Consistency
Decoder from DALL-E 3 [1], a recent diffusion-based decoder enhancing SD VAEs’
decoding capabilities. As shown in Fig. 10, both the original decoder in SD
and the Consistency Decoder fail to preserve the text. In contrast, our
decoder utilizes illumination-irrelevant details from input images, producing
clear and undistorted text. Furthermore, we evaluate the impact of the
decoders in another image restoration task based on SD: colorization. Due to
space limit, please refer to the supplementary for corresponding results.

Figure 11: Left: Model size and PSNR comparison between our lightweight model
and the SOTA. Detailed scores are in the supplementary. Right: Visual
comparison of our full & lightweight versions and NeRCo [54]. Both our full &
lightweight versions similarly show improved contrast and reduced color bias.

Framework Distillation. Compared with the full model, our lightweight version
reduces the running time to 500x faster. It can process a 1024$\times$1024
image in 0.03 seconds on a Tesla M40. Additionally, the number of parameters
is reduced from 1.3G to 327.36k, even smaller than SOTAs with comparable
performance, as shown in Fig. 11. The performance of our lightweight model on
LOL/MIT datasets for PSNR, SSIM, LPIPS, and LOE is 20.45/18.15, 0.798/0.770,
0.290/0.175, and 0.273/0.174, respectively. Compared with our full model, the
lightweight one achieves comparable performance and even marginally improves
the PSNR and LOE. This difference might be due to the randomness inherent in
the generative diffusion model. During the fitting of the larger full version
model, randomness is averaged, further reducing noise and rectifying errors.

##  4 Conclusion

We introduce a new zero-reference low-light enhancement framework, developed
without low-light data. At its core lies a physical quadruple prior derived
from the light transfer theory, and an efficient prior-to-image framework
based on generative diffusion models. Experimental results show our superior
performance across diverse scenarios.

## References

  * Betker et al. [2023] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, Wesam Manassra† Prafulla Dhariwal, Casey Chu† Yunxin Jiao, and Aditya Ramesh.  Improving image generation with better captions.  2023\.
  * Bychkovsky et al. [2011] Vladimir Bychkovsky, Sylvain Paris, Eric Chan, and Frédo Durand.  Learning photographic global tonal adjustment with a database of input / output image pairs.  In _CVPR_ , 2011.
  * Cai et al. [2023] Yuanhao Cai, Hao Bian, Jing Lin, Haoqian Wang, Radu Timofte, and Yulun Zhang.  Retinexformer: One-stage retinex-based transformer for low-light image enhancement.  In _ICCV_ , 2023.
  * Chen et al. [2018] Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun.  Learning to see in the dark.  In _CVPR_ , 2018.
  * Chen et al. [2019] Chen Chen, Qifeng Chen, Minh N. Do, and Vladlen Koltun.  Seeing motion in the dark.  In _ICCV_ , 2019.
  * Chen et al. [2015] Guangyong Chen, Fengyuan Zhu, and Pheng-Ann Heng.  An efficient statistical method for image noise level estimation.  In _ICCV_ , 2015.
  * Fu et al. [2023] Zhenqi Fu, Yan Yang, Xiaotong Tu, Yue Huang, Xinghao Ding, and Kai-Kuang Ma.  Learning a simple low-light image enhancer from paired low-light instances.  In _CVPR_ , 2023.
  * Geusebroek et al. [2001] Jan-Mark Geusebroek, Rein van den Boomgaard, Arnold W. M. Smeulders, and Hugo Geerts.  Color invariance.  _IEEE TPAMI_ , 23(12):1338–1350, 2001.
  * Gevers et al. [2012] Theo Gevers, Arjan Gijsenij, Joost Van de Weijer, and Jan-Mark Geusebroek.  _Color in computer vision: Fundamentals and applications_.  John Wiley & Sons, 2012.
  * Guo et al. [2020] Chunle Guo, Chongyi Li, Jichang Guo, Chen Change Loy, Junhui Hou, Sam Kwong, and Runmin Cong.  Zero-reference deep curve estimation for low-light image enhancement.  In _CVPR_ , 2020.
  * Guo et al. [2017] Xiaojie Guo, Yu Li, and Haibin Ling.  LIME: Low-light image enhancement via illumination map estimation.  _IEEE TIP_ , 26(2):982–993, 2017.
  * Hai et al. [2023] Jiang Hai, Zhu Xuan, Ren Yang, Yutong Hao, Fengzhu Zou, Fang Lin, and Songchen Han.  R2RNet: Low-light image enhancement via real-low to real-normal network.  _Journal of Visual Communication and Image Representation_ , 90:103712, 2023.
  * He et al. [2023] Huiguo He, Tianfu Wang, Huan Yang, Jianlong Fu, Nicholas Jing Yuan, Jian Yin, Hongyang Chao, and Qi Zhang.  Learning profitable NFT image diffusions via multiple visual-policy guided reinforcement learning.  In _ACM MM_ , 2023.
  * Ho et al. [2020] Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In _NeurIPS_ , 2020.
  * Huang et al. [2022a] Haofeng Huang, Wenhan Yang, Yueyu Hu, Jiaying Liu, and Ling-Yu Duan.  Towards low light enhancement with RAW images.  _IEEE TIP_ , 31:1391–1405, 2022a.
  * Huang et al. [2022b] Jie Huang, Yajing Liu, Feng Zhao, Keyu Yan, Jinghao Zhang, Yukun Huang, Man Zhou, and Zhiwei Xiong.  Deep fourier-based exposure correction network with spatial-frequency interaction.  In _ECCV_ , 2022b.
  * Jiang and Zheng [2019] Haiyang Jiang and Yinqiang Zheng.  Learning to see moving objects in the dark.  In _ICCV_ , 2019.
  * Jiang et al. [2023] Hai Jiang, Ao Luo, Songchen Han, Haoqiang Fan, and Shuaicheng Liu.  Low-light image enhancement with wavelet-based diffusion models.  In _Siggraph Asia_ , 2023.
  * Jiang et al. [2021] Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang, Pan Zhou, and Zhangyang Wang.  EnlightenGAN: Deep light enhancement without paired supervision.  _IEEE TIP_ , 30:2340–2349, 2021.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv_ , 2014.
  * Lee et al. [2013] Chulwoo Lee, Chul Lee, and Chang-Su Kim.  Contrast enhancement based on layered difference representation of 2D histograms.  _IEEE TIP_ , 22(12):5372–5384, 2013.
  * Lengyel et al. [2021] Attila Lengyel, Sourav Garg, Michael Milford, and Jan C. van Gemert.  Zero-shot domain adaptation with a physics prior.  In _ICCV_ , 2021.
  * Li et al. [2021] Chongyi Li, Chunle Guo, and Chen Change Loy.  Learning to enhance low-light image via zero-reference deep curve estimation.  _IEEE TPAMI_ , 44(8):4225–4238, 2021.
  * Li et al. [2022] Chongyi Li, Chunle Guo, Ruicheng Feng, Shangchen Zhou, and Chen Change Loy.  Cudi: Curve distillation for efficient and controllable exposure adjustment.  _arXiv_ , 2022.
  * Li et al. [2023] Chongyi Li, Chunle Guo, Shangchen Zhou, Qiming Ai, Ruicheng Feng, and Chen Change Loy.  Flexicurve: Flexible piecewise curves estimation for photo retouching.  In _CVPRW_ , 2023.
  * Li et al. [2018] Mading Li, Jiaying Liu, Wenhan Yang, Xiaoyan Sun, and Zongming Guo.  Structure-revealing low-light image enhancement via robust retinex model.  _IEEE TIP_ , 27(6):2828–2841, 2018.
  * Liang et al. [2023] Zhexin Liang, Chongyi Li, Shangchen Zhou, Ruicheng Feng, and Chen Change Loy.  Iterative prompt learning for unsupervised backlit image enhancement.  In _ICCV_ , 2023.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO: Common objects in context.  In _ECCV_ , 2014.
  * Liu et al. [2022] Chengxu Liu, Huan Yang, Jianlong Fu, and Xueming Qian.  Learning trajectory-aware transformer for video super-resolution.  In _CVPR_ , 2022.
  * Liu et al. [2021] Risheng Liu, Long Ma, Jiaao Zhang, Xin Fan, and Zhongxuan Luo.  Retinex-inspired unrolling with cooperative prior architecture search for low-light image enhancement.  In _CVPR_ , 2021.
  * Lore et al. [2017] Kin Gwn Lore, Adedotun Akintayo, and Soumik Sarkar.  Llnet: A deep autoencoder approach to natural low-light image enhancement.  _PR_ , 61:650–662, 2017.
  * Lu et al. [2022] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu.  Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models.  _arXiv_ , 2022.
  * Ma et al. [2015] Kede Ma, Kai Zeng, and Zhou Wang.  Perceptual quality assessment for multi-exposure image fusion.  _IEEE TIP_ , 24(11):3345–3356, 2015.
  * Ma et al. [2022a] Long Ma, Tengyu Ma, Risheng Liu, Xin Fan, and Zhongxuan Luo.  Toward fast, flexible, and robust low-light image enhancement.  In _CVPR_ , 2022a.
  * Ma et al. [2022b] Yiyang Ma, Huan Yang, Bei Liu, Jianlong Fu, and Jiaying Liu.  AI illustrator: Translating raw descriptions into images by prompt-based cross-modal generation.  In _ACM MM_ , 2022b.
  * Mittal et al. [2012] Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik.  No-reference image quality assessment in the spatial domain.  _IEEE TIP_ , 21(12):4695–4708, 2012.
  * Pizer et al. [1990] S. M. Pizer, R. E. Johnston, J. P. Ericksen, B. C. Yankaskas, and K. E. Muller.  Contrast-limited adaptive histogram equalization: Speed and effectiveness.  In _VBC_ , 1990.
  * Qiu et al. [2022] Zhongwei Qiu, Huan Yang, Jianlong Fu, and Dongmei Fu.  Learning spatiotemporal frequency-transformer for compressed video super-resolution.  In _ECCV_ , 2022.
  * Qiu et al. [2023] Zhongwei Qiu, Huan Yang, Jianlong Fu, Daochang Liu, Chang Xu, and Dongmei Fu.  Learning degradation-robust spatiotemporal frequency-transformer for video super-resolution.  _IEEE TPAMI_ , 45(12):14888–14904, 2023.
  * Rahman et al. [2004] Zia-ur Rahman, Daniel J Jobson, and Glenn A Woodell.  Retinex processing for automatic image enhancement.  _Journal of Electronic Imaging_ , 2004.
  * Rasley et al. [2020] Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He.  Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters.  In _KDD_ , 2020.
  * Rombach et al. [2022] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _CVPR_ , 2022.
  * Ruan et al. [2023] Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo.  Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation.  In _CVPR_ , 2023.
  * Saharia et al. [2023] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi.  Image super-resolution via iterative refinement.  _IEEE TPAMI_ , 45(4):4713–4726, 2023.
  * Vonikakis et al. [2018] Vassilios Vonikakis, Rigas Kouskouridas, and Antonios Gasteratos.  On the evaluation of illumination compensation algorithms.  _Multimedia Tools and Applications_ , 77:9211–9231, 2018.
  * Wang et al. [2013] Shuhang Wang, Jin Zheng, Hai-Miao Hu, and Bo Li.  Naturalness preserved enhancement algorithm for non-uniform illumination images.  _IEEE TIP_ , 22(9):3538–3548, 2013.
  * Wang et al. [2022] Yufei Wang, Renjie Wan, Wenhan Yang, Haoliang Li, Lap-Pui Chau, and Alex C. Kot.  Low-light image enhancement with normalizing flow.  In _AAAI_ , 2022.
  * Wei et al. [2018] Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu.  Deep retinex decomposition for low-light enhancement.  In _BMVC_ , 2018.
  * Wu et al. [2022] Wenhui Wu, Jian Weng, Pingping Zhang, Xu Wang, Wenhan Yang, and Jianmin Jiang.  Uretinex-net: Retinex-based deep unfolding network for low-light image enhancement.  In _CVPR_ , 2022.
  * Wu et al. [2023] Yuhui Wu, Chen Pan, Guoqing Wang, Yang Yang, Jiwei Wei, Chongyi Li, and Heng Tao Shen.  Learning semantic-aware knowledge guidance for low-light image enhancement.  In _CVPR_ , 2023.
  * Xia et al. [2021] Zhihao Xia, Michael Gharbi, Federico Perazzi, Kalyan Sunkavalli, and Ayan Chakrabarti.  Deep denoising of flash and no-flash pairs for photography in low-light environments.  In _CVPR_ , 2021.
  * Xiong et al. [2021] Jinhui Xiong, Jian Wang, Wolfgang Heidrich, and Shree K. Nayar.  Seeing in extra darkness using a deep-red flash.  In _CVPR_ , 2021.
  * Yang et al. [2020a] Fuzhi Yang, Huan Yang, Jianlong Fu, Hongtao Lu, and Baining Guo.  Learning texture transformer network for image super-resolution.  In _CVPR_ , 2020a.
  * Yang et al. [2023] Shuzhou Yang, Moxuan Ding, Yanmin Wu, Zihan Li, and Jian Zhang.  Implicit neural representation for cooperative low-light image enhancement.  In _ICCV_ , 2023.
  * Yang et al. [2020b] Wenhan Yang, Shiqi Wang, Yuming Fang, Yue Wang, and Jiaying Liu.  From fidelity to perceptual quality: A semi-supervised approach for low-light image enhancement.  In _CVPR_ , 2020b.
  * Yang et al. [2020c] Wenhan Yang, Ye Yuan, Wenqi Ren, Jiaying Liu, Walter J Scheirer, Zhangyang Wang, Taiheng Zhang, Qiaoyong Zhong, Di Xie, Shiliang Pu, et al.  Advancing image understanding in poor visibility environments: A collective benchmark study.  _IEEE TIP_ , 29:5737–5752, 2020c.
  * Yi et al. [2023] Xunpeng Yi, Han Xu, Hao Zhang, Linfeng Tang, and Jiayi Ma.  Diff-retinex: Rethinking low-light image enhancement with A generative diffusion model.  In _ICCV_ , 2023.
  * Zamir et al. [2022] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang.  Restormer: Efficient transformer for high-resolution image restoration.  In _CVPR_ , 2022.
  * Zhang et al. [2021a] Fan Zhang, Yu Li, Shaodi You, and Ying Fu.  Learning temporal consistency for low light video enhancement from single images.  In _CVPR_ , 2021a.
  * Zhang and Agrawala [2023] Lvmin Zhang and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  In _ICCV_ , 2023.
  * Zhang et al. [2019a] Lin Zhang, Lijun Zhang, Xinyu Liu, Ying Shen, Shaoming Zhang, and Shengjie Zhao.  Zero-shot restoration of back-lit images using deep internal learning.  In _ACM MM_ , 2019a.
  * Zhang et al. [2018] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _CVPR_ , 2018.
  * Zhang et al. [2019b] Yonghua Zhang, Jiawan Zhang, and Xiaojie Guo.  Kindling the darkness: A practical low-light image enhancer.  In _ACM MM_ , 2019b.
  * Zhang et al. [2021b] Yonghua Zhang, Xiaojie Guo, Jiayi Ma, Wei Liu, and Jiawan Zhang.  Beyond brightening low-light images.  _IJCV_ , 129(4):1013–1037, 2021b.
  * Zhou et al. [2023] Dewei Zhou, Zongxin Yang, and Yi Yang.  Pyramid diffusion models for low-light image enhancement.  In _IJCAI_ , 2023.
  * Zhu et al. [2023a] Junchen Zhu, Huan Yang, Huiguo He, Wenjing Wang, Zixi Tuo, Wen-Huang Cheng, Lianli Gao, Jingkuan Song, and Jianlong Fu.  Moviefactory: Automatic movie creation from text using large generative models for language and images.  In _ACM MM_ , 2023a.
  * Zhu et al. [2023b] Junchen Zhu, Huan Yang, Wenjing Wang, Huiguo He, Zixi Tuo, Yongsheng Yu, Wen-Huang Cheng, Lianli Gao, Jingkuan Song, Jianlong Fu, and Jiebo Luo.  Mobilevidfactory: Automatic diffusion-based social media video generation for mobile devices from text.  In _ACM MM_ , 2023b.