# Procedural Kernel Networks

Bartlomiej Wronski

(Google Research)

###### Abstract

In the last decade Convolutional Neural Networks (CNNs) have defined the state
of the art for many low level image processing and restoration tasks such as
denoising, demosaicking, upscaling, or inpainting. However, on-device mobile
photography is still dominated by traditional image processing techniques, and
uses mostly simple machine learning techniques or limits the neural network
processing to producing low resolution masks. High computational and memory
requirements of CNNs, limited processing power and thermal constraints of
mobile devices, combined with large output image resolutions (typically 8–12
MPix) prevent their wider application. In this work, we introduce Procedural
Kernel Networks (PKNs), a family of machine learning models which generate
parameters of image filter kernels or other traditional algorithms. A
lightweight CNN processes the input image at a lower resolution, which yields
a significant speedup compared to other kernel-based machine learning methods
and allows for new applications. The architecture is learned end-to-end and is
especially well suited for a wide range of low-level image processing tasks,
where it improves the performance of many traditional algorithms. We also
describe how this framework unifies some previous work applying machine
learning for common image restoration tasks.

Figure 1: Overview of our method. In contrast to image predicting or kernel
prediction networks, our method uses small (shallow, aggressively spatially
pooling aggregated information) neural networks to predict a low resolution
image describing local kernel generating parameters. Those parameters are used
to compute final linear or non-linear kernels that are applied to the input
image to produce the final output. Our method relies on differentiable kernel
generator.

##  1 Introduction

In the last decade, CNNs demonstrated great performance in many image
processing and restoration tasks like denoising, demosaicking, and inpainting.
While these techniques are attractive, smartphone photography is largely
dominated by traditional image processing techniques based on signal
processing concepts, manually designed heuristics, and tuning by experts. Most
commercial vendors do not disclose details of their image signal processing
(ISP) algorithm details, but for the denoising task existing literature and
open source libraries describe the practical use of methods like bilateral
filter [1], or wavelet shrinkage techniques [2]. The computational and memory
requirements of CNNs for large output image resolutions (typically 8–12 MPix)
are incompatible with the limited processing power and thermal constraints of
mobile device. Even the authors of popular commercial packages targetting
desktop platforms offer the CNN based techniques only as an experimental, slow
processing option [3]. Machine learning is typically used in a shallow and
limited form, for example by learning direct filters [4]. Most successful
approaches using deep learning on mobile devices either predict auxiliary
images at a much lower resolution [5], or they generate coefficients for a low
resolution bilateral grid that approximates other image processing operations
[6].

Kernel Predicting Networks (KPNs) [7] are a type of CNN, originally developed
for denoising Monte Carlo ray traced images in movie production. KPNs allowed
for smaller network architectures, less visual artifacts, and much faster
training time due to the implicit regularization provided by kernel prediction
[8].

We expand on the ideas of kernel prediction networks and the deep bilateral
grid, and propose to connect and generalize those in a framework we call
Procedural Kernel Networks (PKNs). We use networks to predict functional
kernels with a number of parameters much smaller than the number of weights of
the equivalent discrete filter. PKNs don’t rely on fixed kernel sample
positions and can be predicted at any resolution. Taken to the extreme, kernel
parameters could be predicted globally per image.

We show how a network order of magnitude smaller than contemporary deep
learning approaches provides significant quality improvements over hand-tuned,
traditional image processing algorithms.

In this work, we propose and analyze a general, learned, procedural kernel
framework that:

  * •

Improves the quality of domain specific, highly optimized and specialized
traditional algorithms or hardware.

  * •

Decouples network processing resolution from the final image
resolution—ranging from full image resolution to a single predicted value.

  * •

Decouples the effective kernel spatial support from the parameter count.

  * •

Produces interpretable kernel parameter maps that can be analyzed or further
tuned by human experts.

  * •

Is able to produce continuous kernels, not relying on fixed pixel or point
locations for operation on sparse inputs, or continuous resampling.

##  2 Background

###  2.1 Kernel predicting networks

Convolutional Neural Networks have been used successfully for common image
processing operations like denoising, demosaicking, upsampling, or inpainting
and have been shown to reach or exceed performance of traditional algorithms
on the benchmark datasets. On the other hand, CNNs predicting pixel colors
directly are growing in size. These can require massive amounts of data, with
total training time expressed in years, and yet can still produce visual
artifacts or demonstrate low performance when exposed to test data outside of
the training sets.

Kernel Predicting Networks [7] are an area of active research that addresses
many of those shortcomings. Kernel Predicting networks were shown
theoretically and practically to train much faster [8] than classic pixel
predicting networks, they allow for smaller network sizes, and they have been
applied to processing bursts of images [9]. Kernel prediction regularizes the
output to be a convex combination of the input pixels, guaranteeing no
hallucinated colors or brightness levels. The ideas of kernel prediction has
also been applied to the convolution filters inside the network structure
itself [10, 11].

Simultaneously, a different line of work builds on ideas of per-pixel learned
filter banks [12] and explores the use of linear combination of either global,
predefined kernel bases for tasks of single image upsampling [13], motion blur
deconvolution [14], or learning per-pixel kernel selection for general image
processing [11].

###  2.2 Fast Neural Network processing on mobile devices

Kernel Predicting Networks help with training convolutional neural networks,
their generalization, and producing robust results when presented with novel
data, but they don’t address high computational cost as compared to
traditional algorithms. This poses a challenge for performance-constrained
environments, such as mobile phones, where practical applications of machine
learning and CNNs are constrained to either predicting images at a lower
resolution [5], or single learned filters chosen by handcrafted heuristics
[4].

Gharbi et al. [15] propose to address this by using a CNN to predict
coefficients for a low-resolution bilateral grid, which is then efficiently
sampled at the full resolution. This way, inference resolution can be
decoupled from the final image resolution and used to approximate different
image processing operations.

This idea of predicting smaller resolution coefficients was further expanded
to a more general use of a learned guided filter [16], and in recent years to
meta-learning and image-to-image translation tasks [17].

Figure 2: Oracle experiment. A reference image a) corrupted with AWGN b). When
denoised with a 5x5 NLM filter with a globally optimal single signal sigma
parameter c), we observe detail loss and leftover noise. Locally adapting
sigma d) allows for better detail preservation and noise removal.

###  2.3 Traditional imaging algorithms in CNN setting

While most of the literature considers CNNs as a standalone building block for
image processing operations, there is some interest in using machine learning
to aid traditional algorithms, or vice versa. One of the earliest examples
involves using a single multilayer perception (MLP) for predicting optimal
weights for a joint bilateral filter for denoising Monte Carlo images [18].
Similarly, convolutional neural networks can be used for using a bilateral
grid not only for approximating other image operators [6], but also
traditional bilateral denoising [19]. Recent work proposed to use bilateral
filtering with network-predicted features and network-predicted filter
parameters for achieving real-time performance on the Monte-Carlo rendering
denoising task [20].

One of the advantages of traditional algorithms apart from their computational
efficiency can be explainability, or relationship to physical processes. For
this reason, neural networks were successfully used in the field of medical
and scientific imaging as priors aiding traditional optimization techniques
[21]. Other traditional image and signal filtering operations used
successfully inside a neural network setting include Wiener deconvolution [22]
and guided filtering [16].

A separate line of work uses neural networks to approximate traditional
algorithms, including their tunable and parametric behavior [23]. Recent work
[24] combines the two ideas, and proposes to build a neural network
approximating a black box algorithm or whole image processing unit that can be
used in an optimization setting for finding best parameters for different
objectives.

##  3 Our method

###  3.1 Motivation: limitations of global parameters

We begin with the observation that despite significant advancement in image
denoising over the last fifteen years (using both traditional techniques [25]
and deep learning [26]), variants of the classic non-local means filter (NLM)
[27] and bilateral filtering are still widely used in photographic pipelines.
This is the case especially on mobile devices [1] due to their computational
efficiency, interpretability, robustness, and lack of requirements for careful
pipeline and data modeling. Such simpler algorithms are typically reused by
vendors for many years, and are efficient not only because of algorithmic
simplicity, but also get implemented in hardware (camera ISP), or using
dedicated hardware processors.

Bilateral filtering based denoisers are used in complex pipelines, involve
many heuristics and require careful tuning, which is a time consuming task
performed by specialized image quality experts. This motivated using neural
networks [24] and differentiable programming approaches [28] for finding
optimal algorithm or tuning parameters. Those approaches automate and
significantly simplify the tuning process to find optimal algorithm
parameters, but still involve trade-offs—a globally optimal tuning minimizing
the loss function across whole dataset compromises the performance
locally—both between example images in the dataset, as well as within a single
image.

For our motivation, consider a particular non-local means filter111Note that
in the text we will use NLM and bilateral kernel interchangeably, as a
bilateral kernel is a special case of the NLM kernel with a patch size of 1x1.
(3x3 patch size, 5x5 search window, Gaussian falloff) applied separately to
each channel of the image. We take images from Kodak dataset [29], add
Gaussian noise (sigma 0.1), fix the filter spatial sigma, and attempt to find
optimal signal sigma parameter minimizing the L2 loss (compared to the ground
truth) averaged across the whole image. In this toy experiment, we observe
(Figure 2a-c) that globally optimal parameters can lead to locally suboptimal
filtering—both too noisy with leftover artifacts, as well as overblurring
image details. This is caused by limitations of the NLM kernel and pixelwise
comparisons, that are unable to go beyond simple pixelwise or patchwise
comparisons.

###  3.2 Oracle experiment with locally optimal parameters

We hypothesize that the limited image quality is caused by finding a single
per-image algorithm parameter and that different regions of the image can
benefit from different parameters. To verify that, we seek “locally optimal”
parameters through an oracle optimization process.

We take pairs of clean and noisy images, and apply a NLM filter with
spatially-varying sigma on the noisy images. To find the optimal local signal
sigma (computed at half of the image resolution to enforce smoothness), we
minimize the L2 loss locally compared to the ground truth clean image. We rely
on the differentiability of the NLM kernel with regard to its parameters,
which lets us optimize spatially-varying signal sigma using gradient descent
and related optimization techniques.

This local optimization leads much better results, both in terms of detail
preservation, as well as less leftover noise and visual artifacts (Figure
2a-d). It significantly improves the PSNR by over 1 dB (Table 1).

While this experiment suggests that letting the NLM parameters vary spatially
can lead to significantly better results, the approach is not direclty
applicable, since when denoising real-world images we don’t have access to
their clean versions.

Noise sigma | PSNR Optimal | PSNR Network | PSNR Optimal
---|---|---|---
| sigma per image | predicted sigma | local sigma
0.025 | 35.93 dB | 36.25 dB | 36.68 dB
0.05 | 31.84 dB | 32.28 dB | 32.87 dB
0.1 | 27.90 dB | 28.49 dB | 29.03 dB
0.2 | 23.72 dB | 24.64 dB | 24.99 dB
Table 1: PSNR comparison of globally optimal and locally predicted tuning
values.

###  3.3 Network prediction of locally optimal NLM parameters

To turn this idea into practical application, we propose to use a simple, low
parameter neural network to predict locally good tuning values (Figure 3).

Figure 3:  A small network used to predict spatially varying algorithm
parameters.

Note that while some earlier work used networks for predicting optimal channel
weights for joint-bilateral filtering based on multi-channel, pixel-wise
properties [18], we aim to infer optimal parameters for the traditional
bilateral filter based on the neighborhood of the processed pixel. We train a
simple network with the architecture presented in Figure 4.

Figure 4:  Network architecture use for NLM parameter prediction. The yellow
block is the input (image contents and noise standard deviation), blue blocks
represent 3x3 convolution, and the green block outputs N kernel parameters
through a 1x1 convolution.

To train the network, used algorithm and its optimized parameters have to be
differentiable. In the case of non-differentiable algorithms, one might use
approach described in prior work for creating differentiable algorithm proxies
[24].

We note that the network itself has less than 20k parameters, and most
computations are done in lower resolution than the input image. Additionally,
it has a similar spatial support footprint as the denoiser itself, thus any
expected improvements would come purely from learning higher level and
structural information.

In this experiment, we feed the network with a single channel of the image to
be denoised together with expected per-pixel noise standard deviation. The
network outputs half-resolution signal constrained to range [0,4] using a
sigmoid function, which is used as local multiplier on top of the noise
standard deviation. This is used as a local spatial sigma for the NLM filter.

We train the network on a subset of 10k random images from Open Images v6
dataset (whole network training until convergence takes less than 10 minutes
on a single GPU cloud machine), and use Kodak image dataset as a validation
set. During training, we use random noise standard deviations sampled
uniformly from [0.05,0.1].

A small network without access to clean images cannot achieve the performance
of oracle, but the quantitative improvements on this simple experiment (Table
1) are within 0.32–0.92 dB as compared to per-image parameters. Note that
during the training, we never show examples with noise sigma below 0.05 or
above 0.1, and yet we observe generalization outside of this range. We observe
also qualitative improvements that we assess as being perceptually in between
the locally optimal oracle results, and the per-image optimal tuning parameter
(Figure 5). Given that the oracle experiment is the upper bound of the quality
of selected configuration of the NLM filter, we find those results confirming
our intuition of local improvements.

Figure 5:  Network parameter prediction. Left: Per-image tuning. Middle:
Oracle tuning. Right: Tuning predicted locally by a small neural network.

We analyze the outputs of the network to verify the inferred NLM sigma
multiplier maps are spatially smooth and detect salient image features (Figure
6).

Figure 6:  Demonstration of the produced local parameters. Left: Reference
image. Middle: Noisy image tuning. Right: Sigma multiplier parameter image /
map. Produced parameters delineate smooth areas (more denoising) from edges or
textured ones (less denoising).

The inferred parameter maps are interpretable, and can be used to analyze why
the denoiser produced more smooth or sharper images in given image and in a
given area. Such produced results can be subject to further tuning by human
experts or based on the user preferences without the need for retraining—to
adjust the trade-off between denoising and sharpness, one can simply rescale
the output parameters, or apply a more general non-linear remapping.
Additionally, the spatial smoothness of the inferred tuning maps suggests that
it’s possible to reduce inference resolution even further.

In the next section we explain how the idea presented connects prior work on
general image filtering and kernel predicting networks.

###  3.4 Linear algebra of image filtering

Given the motivation above, we formalize the results as Procedural Kernel
Networks and generalize them to other kernels (beyond NLM and bilateral
kernel).

Starting with the bilateral filter kernel:

$Y(x)=\frac{1}{\sum
W}\sum_{x_{i}\in\Omega}X(x_{i})W_{x}(x,x_{i})W_{X}(X(x),X(x_{i}))$ (1)

Bilateral filter is a weighted average of pixels around the neighborhood of a
destination pixel. $W_{X}(x,x_{i})$ is a spatial weighting function, and
$W_{X}(X(x),X(x_{i}))$ is a signal weighting function—both are often Gaussian
kernel. While the weighting functions are non-linear, the bilateral filter is
a linear combination of surrounding pixels, where at each pixel we can write
down the effective kernel:

$Y(x)=\sum_{x_{i}\in\Omega}X(x_{i})w_{i}$ (2)
$w_{i}=\frac{W_{x}(x,x_{i})W_{X}(X(x),X(x_{i}))}{\sum_{x_{i}\in\Omega}W_{x}(x,x_{i})W_{X}(X(x),X(x_{i}))}$
(3)

Prior work [30] has comprehensively explored how most traditional denoising,
deconvolution and other image filtering operations can be written in a matrix
form $W\cdot X$, where rows and columns of the matrix $W$ correspond to the
input and output pixels of the filter. Some filters like the bilateral filter
produce sparse $W$ matrix, while for Non-Local Means with an unlimited search
window the $W$ matrix is dense in general.

Using this notation, we can treat the bilateral kernel as an output of a
function generating matrix entries through spatial and signal relationships of
neighboring pixels.

Spatial relationships using a spatially constant function are also used for
much simpler kernels, like a box filter, or a Gaussian kernel:

$W_{x}=\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{\left\|x-x_{i}\right\|^{2}}{2\sigma^{2}}},W_{X}=1$
(4)

We can express not only standard convolutions, but we can also add a per-
element constant term to the input by expanding the dimensionality of the
input (mapping to homogeneous coordinates):

$X^{\prime}=\begin{bmatrix}X\\\ 1^{T}\end{bmatrix}$ (5)

And similarly extending the $W$ matrix by an additional column.

###  3.5 Procedural Kernel Networks

We propose to unify this framework with kernel predicting networks, regressing
optimal bilateral filter parameters, and predicting linear combinations of
basis functions, and call it Procedural Kernel Networks.

We define the filtering weight of a pixel $x_{i}$ for the pixel $x$ as a
function of spatial pixel relationships, signal levels, and a set of
parameters $\Omega$:

$W(x,x_{i})=F(\Omega(x),x,x_{i},X(x),X(x_{i}))$ (6)

Classic Kernel Predicting Networks [7] can be expressed as predicting directly
a locally constant weights that depend only on the processed pixel center, and
spatial coordinates of the surrounding pixels - identity function mapping:

$F=\Omega(x,x_{i})$ (7)

We contrast the two approaches in Figure 7.

Figure 7: KPN (top) as contrasted with PKN (bottom).

The Procedural Kernel Network framework allows for expressing wide variety of
other kernel generating functions and unifies some other prior work. A network
predicted linear combination of basis function [31, 13, 14, 11] is then
defined as:

$F=\sum_{n}\Omega(x,n)*K_{n}(x_{i})$ (8)

Where $K_{n}$ are different global kernels.

For the experiments proposed in section 3.3, $\sigma$ for the bilateral filter
acts as the single $\omega$ parameter of $\Omega$:

$\omega(x)=\sigma(x)$ (9)

Deep bilateral learning [15] that predicts coefficients of the bilateral grid
[6, 20] and predicted coefficients of a guided filter [16] can be both
written:

$F=\omega_{0}(x)+\omega_{1}(x)X(x)$ (10)

Where $\omega_{0}(x)$ and $\omega_{1}(x)$ are bi- or trilinearly interpolated.
Those approaches share fast inference, significantly faster training time, and
are deployed on mobile platforms. We observe that this is both because
procedural and functional kernels allow for decoupling of the processing
resolution from the processed image resolution, as well as provide strong
regularization (spatial smoothness and the output space constrained by the
kernel parameters).

In the next section we explore using different simple, differentiable kernels
for low parameter, learned processing in different applications beyond
bilateral denoising.

##  4 Different kernels and their applications

In the following sections, we look at some examples of different procedural
kernels and how parameter prediction can be used for a variety of traditional
image processing operations. We use the same network architecture in all of
our experiments, only changing the output remapping function and the number of
output channels.

###  4.1 Anisotropic denoising

The bilateral kernel is not the only kernel weight generating function we can
use in the PKN framework for denoising. Any kind of spatially varying blur
kernel could be parameterized to adaptively preserve details or remove noise.

Note that isotroptic Gaussian blur is a degenerate form of the bilateral
filter, where the signal sigma is infinity. Here, we propose using an
anisotropic Gaussian blur kernel for edge-preserving behavior. Such edge-aware
kernels are used in applications beyond image restoration, such as 3D mesh
reconstruction and fluid particle filtering [32], as well as multi-frame
super-resolution [33].

An anisotropic Gaussian can be parameterized by the covariance matrix
$\Sigma$, or by its inverse, the precision matrix $\Sigma^{-1}$:

$\Omega(x)=e^{-\frac{1}{2}(x_{i}-x)^{\mathrm{T}}\Sigma^{-1}(x_{i}-x)}$ (11)

One challenge with parameterizing with full covariance matrices is that
directly predicting the matrix entries can lead to a non-invertible matrix.
Similarly, if we tried to optimize precision matrix entries directly, a
network could predict an invalid precision matrix. To address this challenge,
we optimize $\sigma_{1}$, $\sigma_{2}$, and $\rho$ in the following covariance
matrix parameterization:

$\Sigma=\begin{bmatrix}\sigma_{1}^{2}&\rho\sigma_{1}\sigma_{2}\\\
\rho\sigma_{1}\sigma_{2}&\sigma_{2}^{2}\end{bmatrix}$ (12)

When $\rho$ is constrained to a range [-1, 1] and $\sigma$ values are
positive, we are guaranteed to get a correctly formed covariance matrix.

Figure 8: Visual results of anisotropic denoising. Top left: noisy input
image. Top right: globally optimal 5x5 bilateral filter. Bottom left: PKN
predicting isotropic Gaussian sigma. Bottom Right: PKN predicting anisotropic
Gaussian covariance.

We parameterize network outputs as two channels with sigma multipliers in the
range [0, 4], and a correlation coefficient in the range [-1, 1]. For
verification and ablation studies of our experiments, we train also a second
network that outputs only a single sigma shared by both directions (equivalent
to isotropic blur). We present qualitative results in Figure 8 and illustrate
the inferred coefficients in Figure 9.

Figure 9:  Denoising with isotropic (left) and anisotropic (right) Gaussian
kernels. Top: Denoised image result. Bottom: Predicted kernel parameters. The
anisotropic kernel predicts a precision matrix that visually aligns with edges
of different orientations in the image.

Even such a simple network is able to find image features like edges and
textures, and denoise the image while preserving salient features.

Anisotropic Gaussian kernels can be used for other applications beyond image
processing including 3D and geometry processing, but we don’t explore them in
this work. We suggest that for future work, it could be beneficial to analyze
the performance of a predicted full anisotropic bilateral kernel characterized
by a full 3x3 spatial-signal covariance matrix.

###  4.2 Polynomial deblurring kernel

Prior work suggests to use a prediction of coefficients for a linear
combination of bases for tasks like denoising or single frame upsampling [31,
13, 14, 11]. We analyze how some of those approaches and applications can be
unified under procedural kernel prediction network framework. Recent work [34]
has demonstrated how polynomial reblurring kernel can be used for efficiently
approximating deconvolution.

A spatially varying blur kernel can be used to not only prevent amplifying
noise while deconvolving the image, but also remove some of it. Inspired by
this work, we use the same polynomial reblurring kernel, but instead of fixed
global tuning of polynomial coefficients, we use a network to predict them and
adapt locally.

Given blur kernel matrix $B$, a polynomial reblurring kernel is defined as:

$W(x)=a_{x}I+b_{x}B+c_{x}B^{2}+d_{x}B^{3}+...$ (13)

We use a fourth degree polynomial with coefficients $a,b,c,d$. Those
coefficients need to sum to 1 to preserve constant signals, therefore we have
only three degrees of freedom and use a network to predict 3 coefficients
$b,c,d$, while we parameterize $a$ as:

$a_{x}=1-(b_{x}+c_{x}+d_{x})$ (14)

For parameters $b,c,d$, we apply a sigmoid remapping function in range [-4, 4]
and train for minimization of a L2 loss function.

Figure 10: Visual results of polynomial reblurring coefficient prediction.
Left: Original image crops. Middle: Noisy and blurry images. Right: Results of
deconvolution with denoising.

As our experiments show (Figure 10), it’s possible to achieve both detail
amplification and deconvolution of edges, as well as mild denoising effect.

Figure 11:  Left: Original image crop. Middle: Predicted coefficients, where
each color channel represents different polynomial coefficient. Right:
Deconvolved and denoised image.

The resulting predicted coefficients (Figure 11) are easily interpretable and
correspond to detected structures. In the areas in need of denoising and
lacking visible structures, the predicted polynomial forms a simple blur
kernel, while in the areas that can benefit from deconvolution, the polynomial
performs a kernel inversion.

###  4.3 Anisotropic upsampling

We expand the experiments with anisotropic Gaussian filters in section 4.1 to
single frame upsampling. Recent work on multi-frame super-resolution [33] has
used anisotropic Gaussian Radial Basis Function kernels for arbitrary,
continuous magnification of images. A continuous parameterization can be used
to increase images resolution by different magnification factors, including
fractional ones [35].

Similarly to [33], we focus on anisotropic Gaussian RBFs, but instead of using
a set of complicated, hand-tuned heuristics applied to a structure tensor
computation [36] described by authors in the supplemental material, we
automatically infer them through a neural networks.

We predict anisotropic Gaussian covariance matrix at the half of the original
image resolution parameterized the same way as in section 4.1. For simplicity
of evaluation, we bilinearly upsample the Gaussian kernel precision matrix to
the desired resolution and use as local kernel parameters.

We train the upsampling network with varying amounts of noise, and compare to
a commonly used Lanczos4 upsampling method, showing perceptually better
results for both mild and severe noise (Figure 12).

Figure 12: Qualitative evaluation of anisotropic kernel upsampling.

Figure 13:  Different upsampling factors: 1.3x, 1.8x, 3.0x. Readers are
encouraged to zoom in in the digital copy.

While most of the previous work on learning image upsampling filters is
focused on integer, predefined upsampling factors [4], we can easily upsample
by fractional amounts (Figure 13) to provide an option of fractional zoom to
the user. We note that the inference resolution, or the inference results
don’t change between the magnification factors, allowing to save or cache the
inferred results while the user can preview the image at different
magnification factors, potentially even changing in real time.

##  5 Discussion and limitations

In this section we will discuss the results, target uses of Procedural Kernel
Networks, as well as some limitations compared to prior work.

###  5.1 Kernel limitations

We observe that our proposed framework is limited by the oracle performance of
the used function generating kernel for the selected tasks. While some kernel
functions have enough degrees of freedom to predict even the target pixels
directly, it’s not the case for most of analyzed and proposed uses.

PKN cannot achieve results that are any better than the best possible
potential (local) parameters of the given kernel, similar to the oracle
bilateral kernel experiments (Table 1).

For some uses, this approach produces results that are inferior to the state
of the art, but can be still attractive for practical application and on-
device use.

###  5.2 Quantitative and qualitative performance

For the reason described above, we don’t analyze quantitative performance on
most of the proposed tasks. Even for the denoising application, we note that
achieved performance is significantly lower than many state of the art
results.

The goal of our work is to show a scalable framework that can be applied to
existing building blocks from the vast body of existing research predating the
machine learning revolution. We leverage the capabilities of neural networks
to boost the performance of these existing algorithms. The qualitative results
shown confirm that we can produce visually attractive results that are
significantly better than the baseline (spatially uniform kernel parameters).

###  5.3 Automatic tuning and boosting of existing algorithms

Our work can be seen as a framework for automatically tuning existing
traditional algorithms. Unlike prior work [24], we propose to find those
parameters during the inference time of the algorithm, not in an offline
optimization process. The inferred parameter maps are not just globally
optimal (best performance averaged over the whole dataset), but also close to
locally optimal.

One particularly interesting potential application of our work is that it can
be used to drive parameters of existing hardware blocks, which are typically
designed many years in advance and are more difficult to modify and improve
than software and algorithm blocks. Even if the hardware blocks lack the
dedicated local parameter inputs, some of the benefits can be achieved through
global parameter manipulation, or even manipulating the input—such as by
modifying the expected per-pixel standard deviation of the noise.

###  5.4 Computation resolution

We don’t assume any particular operating resolution of the kernel generating
function. This is one of the advantages of our formulation, as by using
smoothness of the parameter space, one can decouple the operating resolution
of the network. While we don’t explore it in our work, if at least a partial
correlation of the optimal parameters with the signal strength exists, one
could infer at even smaller resolutions, e.g. using a bilateral grid [6].

Taken to the extreme, our method can still predict single set of global
parameters that are optimal per image, which can be an improvement over
finding parameters that are optimal on average for the whole training set.

###  5.5 Loss function

In all of our presented experiments, we have used the L2 loss function.
Depending on the application, one could use a different loss function, for
example one that correlates better with the human perception [37], or other
desired properties for applications that are not tied to viewing images by
humans, like object recognition or classification.

###  5.6 Differentiability and parameterization

Throughout our work, we conveniently use continuous, differentiable kernels
and loss functions. We have not done any experiments with using differentiable
proxies for non-differentiable tasks, such as computing a loss after JPEG
compression [38], but we believe the proposed approach could apply to such
problems. On the other hand, we observe how some parameterizations of the
kernel generating functions are much better suited for learning and the
optimization process than the others.

In the case of anisotropic Gaussian kernel prediction, we noted that a direct
covariance matrix representation can easily produce an invalid and non-
invertible covariance matrices, which leads to exploding gradients and
instability during training. Reparameterizing the matrix using correlation and
standard deviation coefficients leads to stable convergence at training time.
We believe that while the loss function can be non-convex in the kernel
parameter space, it is required to be relatively smooth and Lipschitz-
continuous.

###  5.7 Use beyond images

Finally, we suggest that this work could be expanded beyond the image domain.
Experiments with fractional, continuous upsampling suggest that predicting
local functional parameters could work well with irregular or sparse inputs
and applications such as meshes, point clouds, or even graphs.

##  6 Summary

To conclude, we have presented a general framework of Procedural Kernel
Networks that unifies some of the existing work on real-time and
computationally efficient deep learning methods. We propose how to use this
framework with a wide range of different procedural kernel functions. We
propose some practical applications and suggest further work in under-explored
areas such as using small neural networks to boost the quality of traditional
algorithms or existing hardware blocks.

While our work has limitations and we don’t claim to achieve state of the art
on any of the analyzed applications, we hope that the vast design space
possible in this framework and an opportunity to base on existing research
predating the deep learning revolution could inspire some novel applications.

##  7 Acknowledgements

We would like to thank Sam Hasinoff, Cengiz Öztireli, Manfred Ernst, Dillon
Sharlet, Mauricio Delbracio, Julian Iseringhausen, and Jon Barron for valuable
discussions and suggestions that lead to this work and the manuscript.

## References

  * [1] S. W. Hasinoff, D. Sharlet, R. Geiss, A. Adams, J. T. Barron, F. Kainz, J. Chen, and M. Levoy, “Burst photography for high dynamic range and low-light imaging on mobile cameras,” ACM Transactions on Graphics (TOG), vol. 35, no. 6, pp. 1–12, 2016.
  * [2] D. Coffin, “Dcraw: Decoding raw digital photos in linux.” https://www.dechifro.org/dcraw/.
  * [3] S. Mangalick, “Adobe: Enhance details.” https://business.adobe.com/blog/the-latest/enhance-details.
  * [4] Y. Romano, J. Isidoro, and P. Milanfar, “Raisr: rapid and accurate image super resolution,” IEEE Transactions on Computational Imaging, vol. 3, no. 1, pp. 110–125, 2016.
  * [5] N. Wadhwa, R. Garg, D. E. Jacobs, B. E. Feldman, N. Kanazawa, R. Carroll, Y. Movshovitz-Attias, J. T. Barron, Y. Pritch, and M. Levoy, “Synthetic depth-of-field with a single-camera mobile phone,” ACM Transactions on Graphics (ToG), vol. 37, no. 4, pp. 1–13, 2018.
  * [6] J. Chen, S. Paris, and F. Durand, “Real-time edge-aware image processing with the bilateral grid,” ACM Transactions on Graphics (TOG), vol. 26, no. 3, pp. 103–es, 2007.
  * [7] S. Bako, T. Vogels, B. McWilliams, M. Meyer, J. Novák, A. Harvill, P. Sen, T. Derose, and F. Rousselle, “Kernel-predicting convolutional networks for denoising monte carlo renderings.,” ACM Trans. Graph., vol. 36, no. 4, pp. 97–1, 2017.
  * [8] T. Vogels, F. Rousselle, B. McWilliams, G. Röthlin, A. Harvill, D. Adler, M. Meyer, and J. Novák, “Denoising with kernel prediction and asymmetric loss functions,” ACM Transactions on Graphics (TOG), vol. 37, no. 4, pp. 1–15, 2018.
  * [9] B. Mildenhall, J. T. Barron, J. Chen, D. Sharlet, R. Ng, and R. Carroll, “Burst denoising with kernel prediction networks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2502–2510, 2018.
  * [10] X. Jia, B. De Brabandere, T. Tuytelaars, and L. Van Gool, “Dynamic filter networks,” in NIPS, 2016.
  * [11] F. Kokkinos, I. Marras, M. Maggioni, G. Slabaugh, and S. Zafeiriou, “Pixel adaptive filtering units,” arXiv preprint arXiv:1911.10581, 2019.
  * [12] P. Getreuer, I. Garcia-Dorado, J. Isidoro, S. Choi, F. Ong, and P. Milanfar, “Blade: Filter learning for general purpose computational photography,” in 2018 IEEE International Conference on Computational Photography (ICCP), pp. 1–11, IEEE, 2018.
  * [13] W. Li, K. Zhou, L. Qi, N. Jiang, J. Lu, and J. Jia, “Lapar: Linearly-assembled pixel-adaptive regression network for single image super-resolution and beyond,” Advances in Neural Information Processing Systems, vol. 33, 2020\.
  * [14] G. Carbajal, P. Vitoria, M. Delbracio, P. Musé, and J. Lezama, “Single image non-uniform blur kernel estimation via adaptive basis decomposition,” arXiv preprint arXiv:2102.01026, 2021.
  * [15] M. Gharbi, J. Chen, J. T. Barron, S. W. Hasinoff, and F. Durand, “Deep bilateral learning for real-time image enhancement,” ACM Transactions on Graphics (TOG), vol. 36, no. 4, pp. 1–12, 2017.
  * [16] H. Wu, S. Zheng, J. Zhang, and K. Huang, “Fast end-to-end trainable guided filter,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1838–1847, 2018.
  * [17] T. R. Shaham, M. Gharbi, R. Zhang, E. Shechtman, and T. Michaeli, “Spatially-adaptive pixelwise networks for fast image translation,” arXiv preprint arXiv:2012.02992, 2020.
  * [18] N. K. Kalantari, S. Bako, and P. Sen, “A machine learning approach for filtering monte carlo noise.,” ACM Trans. Graph., vol. 34, no. 4, pp. 122–1, 2015.
  * [19] X. Meng, Q. Zheng, A. Varshney, G. Singh, and M. Zwicker, “Real-time monte carlo denoising with the neural bilateral grid,” The Eurographics Association, 2020.
  * [20] M. Işık, K. Mullia, M. Fisher, J. Eisenmann, and M. Gharbi, “Interactive monte carlo denoising using affinity of neural features,” ACM Transactions on Graphics (TOG), vol. 40, no. 4, pp. 1–13, 2021.
  * [21] S. Diamond, V. Sitzmann, F. Heide, and G. Wetzstein, “Unrolled optimization with deep priors,” arXiv preprint arXiv:1705.08041, 2017.
  * [22] J. Dong, S. Roth, and B. Schiele, “Deep wiener deconvolution: Wiener meets deep learning for image deblurring,” Advances in Neural Information Processing Systems, vol. 33, 2020.
  * [23] Q. Fan, D. Chen, L. Yuan, G. Hua, N. Yu, and B. Chen, “A general decoupled learning framework for parameterized image operators,” IEEE transactions on pattern analysis and machine intelligence, 2019.
  * [24] E. Tseng, F. Yu, Y. Yang, F. Mannan, K. S. Arnaud, D. Nowrouzezahrai, J.-F. Lalonde, and F. Heide, “Hyperparameter optimization in black-box image processing using differentiable proxies.,” ACM Trans. Graph., vol. 38, no. 4, pp. 27–1, 2019.
  * [25] K. Dabov, A. Foi, V. Katkovnik, and K. Egiazarian, “Image denoising by sparse 3-d transform-domain collaborative filtering,” IEEE Transactions on image processing, vol. 16, no. 8, pp. 2080–2095, 2007.
  * [26] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang, “Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising,” IEEE transactions on image processing, vol. 26, no. 7, pp. 3142–3155, 2017.
  * [27] A. Buades, B. Coll, and J.-M. Morel, “A non-local algorithm for image denoising,” in 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05), vol. 2, pp. 60–65, IEEE, 2005.
  * [28] T.-M. Li, M. Gharbi, A. Adams, F. Durand, and J. Ragan-Kelley, “Differentiable programming for image processing and deep learning in halide,” ACM Transactions on Graphics (TOG), vol. 37, no. 4, pp. 1–13, 2018.
  * [29] Eastman Kodak Company, “True Color Kodak Images.” http://r0k.us/graphics/kodak/.
  * [30] P. Milanfar, “A tour of modern image filtering: New insights and methods, both practical and theoretical,” IEEE signal processing magazine, vol. 30, no. 1, pp. 106–128, 2012.
  * [31] Z. Xia, F. Perazzi, M. Gharbi, K. Sunkavalli, and A. Chakrabarti, “Basis prediction networks for effective burst denoising with large kernels,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11844–11853, 2020.
  * [32] J. Yu and G. Turk, “Reconstructing surfaces of particle-based fluids using anisotropic kernels,” ACM Transactions on Graphics (TOG), vol. 32, no. 1, pp. 1–12, 2013.
  * [33] B. Wronski, I. Garcia-Dorado, M. Ernst, D. Kelly, M. Krainin, C.-K. Liang, M. Levoy, and P. Milanfar, “Handheld multi-frame super-resolution,” ACM Transactions on Graphics (TOG), vol. 38, no. 4, pp. 1–18, 2019.
  * [34] M. Delbracio, I. Garcia-Dorado, S. Choi, D. Kelly, and P. Milanfar, “Polyblur: Removing mild blur by polynomial reblurring,” arXiv preprint arXiv:2012.09322, 2020.
  * [35] Y. Chen, S. Liu, and X. Wang, “Learning continuous image representation with local implicit image function,” arXiv preprint arXiv:2012.09161, 2020.
  * [36] J. Bigun, “Optimal orientation detection of linear symmetry,” 1987.
  * [37] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, “The unreasonable effectiveness of deep features as a perceptual metric,” in CVPR, 2018.
  * [38] R. Shin and D. Song, “Jpeg-resistant adversarial images,” in NIPS 2017 Workshop on Machine Learning and Computer Security, vol. 1, 2017.


* Latorre  _et al._ [2004] J. I. Latorre, E. Rico, and G. Vidal, Ground state entanglement in quantum spin chains, Quantum Info. Comput. 4, 48–92 (2004).
  * Klich [2014] I. Klich, A note on the full counting statistics of paired fermions, Journal of Statistical Mechanics: Theory and Experiment 2014, P11006 (2014).
  * González-Ballestero  _et al._ [2011] C. González-Ballestero, L. Robledo, and G. Bertsch, Numeric and symbolic evaluation of the pfaffian of general skew-symmetric matrices, Computer Physics Communications 182, 2213 (2011).
  * Gaudin [1960] M. Gaudin, Une démonstration simplifiée du théorème de Wick en mécanique statistique, Nuclear Physics 15, 89 (1960).
  * Perez-Martin and Robledo [2007] S. Perez-Martin and L. M. Robledo, Generalized wick’s theorem for multiquasiparticle overlaps as a limit of gaudin’s theorem, Phys. Rev. C 76, 064314 (2007).
  * Hall [2015] B. Hall,  _Lie Groups, Lie Algebras, and Representations: An Elementary Introduction_, 2nd ed. (Springer International Publishing, Cham, 2015).