# SAU: Smooth activation function using convolution with approximate
identities

Koushik Biswas
&Sandeep Kumar
&Shilpak Banerjee
&Ashish Kumar Pandey

###### Abstract

Well-known activation functions like ReLU or Leaky ReLU are non-differentiable
at the origin. Over the years, many smooth approximations of ReLU have been
proposed using various smoothing techniques. We propose new smooth
approximations of a non-differentiable activation function by convolving it
with approximate identities. In particular, we present smooth approximations
of Leaky ReLU and show that they outperform several well-known activation
functions in various datasets and models. We call this function Smooth
Activation Unit (SAU). Replacing ReLU by SAU, we get 5.12% improvement with
ShuffleNet V2 (2.0x) model on CIFAR100 dataset.

_Keywords_ Deep Learning $\cdot$ Neural Networks $\cdot$ Activation Function

##  1 Introduction

Deep networks form a crucial component of modern machine learning. Non-
linearity is introduced in such networks by the use of activation functions,
and the choice has a substantial impact on network performance and training
dynamics. Designing a new novel activation function is a difficult task.
Handcrafted activations like Rectified Linear Unit (ReLU) [1], Leaky ReLU [2]
or its variants are a very common choice for activation functions and exhibits
promising performance on different deep learning tasks. There are many
activations that have been proposed so far and some of them are ELU [3],
Parametric ReLU (PReLU) [4], Swish [5], Tanhsoft [6], EIS [7], PAU [8], OPAU
[9], ACON [10], ErfAct [11], Mish [12], GELU [13] etc. Nevertheless, ReLU
remains the favourite choice among the deep learning community due to its
simplicity and better performance when compared to Tanh or Sigmoid, though it
has a drawback known as dying ReLU in which the network starts to lose the
gradient direction due to the negative inputs and produces zero outcome. In
2017, Swish [5] was proposed by the Google brain team. Swish was found by
automatic search technique, and it has shown some promising performance across
different deep learning tasks.

Activation functions are usually handcrafted. PReLU [4] tries to overcome this
problem by introducing a learnable negative component to ReLU [1]. Maxout
[14], and Mixout [15] are constructed with piecewise linear components, and
theoretically, they are universal function approximators, though they increase
the number of parameters in the network. Recently, meta-ACON [10], a smooth
activation has been proposed, which is the generalization of the ReLU and
Maxout activations and can smoothly approximate Swish. Meta-ACON has shown
some good improvement on both small models and highly optimized large models.
PAU [8], and OPAU [9] are two promising candidates for trainable activations,
which have been introduced recently based on rational function approximation.

In this paper, we introduce a smooth approximation of known non-smooth
activation functions like ReLU or Leaky ReLU based on the approximation of
identity. Our experiments show that the proposed activations improve the
performance of different network architectures as compared to ReLU on
different deep learning problems.

##  2 Mathematical formalism

###  2.1 Convolution

Convolution is a binary operation, which takes two functions $f$ and $g$ as
input, and outputs a new function denoted by $f*g$. Mathematically, we define
this operation as follows

$\displaystyle(f*g)(x)=\int_{-\infty}^{\infty}f(y)g(x-y)\,dy.$ (1)

The convolution operation has several properties. Below, we will list two of
them which will be used larter in this article.

  1. P1.

$(f\ast g)(x)=(g\ast f)(x)$,

  2. P2.

If $f$ is $n$-times differentiable with compact support over $\mathbb{R}$ and
$g$ is locally integrable over $\mathbb{R}$ then $f\ast g$ is at least
$n$-times differentiable over $\mathbb{R}$.

Property P1 is an easy consequence of definition (1). Property P2 can be
easily obtained by moving the derivative operator inside the integral. Note
that this exchange of derivative and integral requires $f$ to be of compact
support. An immediate consequence of property P2 is that if one of the
functions $f$ or $g$ is smooth with compact support, then $f\ast g$ is also
smooth. This observation will be used later in the article to obtain smooth
approximations of non-differentiable activation functions.

###  2.2 Mollifier and Approximate identities

A smooth function $\phi$ over $\mathbb{R}$ is called a mollifier if it
satisfies the following three properties:

  1. 1.

It is compactly supported.

  2. 2.

$\int_{\mathbb{R}}\phi(x)\,dx=1.$

  3. 3.

${\displaystyle\lim_{\epsilon\to 0}\phi_{\epsilon}(x):=\lim_{\epsilon\to
0}\frac{1}{\epsilon}\phi(x/\epsilon)=\delta(x)}$, where
${\displaystyle\delta(x)}$ is the Dirac delta function.

We say that a mollifier $\phi$ is an approximate identity if for any locally
integrable function $f$ over $\mathbb{R}$, we have

$\displaystyle{\displaystyle\lim_{\epsilon\to
0}(f\ast\phi_{\epsilon})(x)=f(x)\;\text{pointwise for all }x.}$

###  2.3 Smooth approximations of non-differentiable functions

Let $\phi$ be an approximate identity. Choosing $\epsilon=1/n$ for
$n\in\mathbb{N}$, one can define

$\displaystyle\phi_{n}(x):=n\phi(nx).$ (2)

Using the property of approximate identity, for any locally integrable
function $f$ over $\mathbb{R}$, we have

$\displaystyle{\displaystyle\lim_{n\to\infty}(f\ast\phi_{n})(x)=f(x)\;\text{pointwise
for all }x.}$

That is, for large enough $n$, $f\ast\phi_{n}$ is a good approximation of $f$.
Moreover, since $\phi$ is smooth, $\phi_{n}$ is smooth for each
$n\in\mathbb{N}$ and therefore, using property P2, $f\ast\phi_{n}$ is a smooth
approximation of $f$ for large enough $n$.

Let $\sigma:\mathbb{R}\to\mathbb{R}$ be any activation function. Then, by
definition, $\sigma$ is a continuous and hence, a locally integrable function.
For a given approximate identity $\phi$ and $n\in\mathbb{N}$, we define a
smooth approximation of $\sigma$ as $\sigma\ast\phi_{n}$, where $\phi_{n}$ is
defined in (2).

##  3 Smooth Approximation Unit (SAU)

Consider the Gaussian function

$\displaystyle\phi(x)=\frac{1}{\sqrt{2\pi}}e^{-\frac{x^{2}}{2}}$

which is a well known approximate identity. Consider the Leaky Rectified
Linear Unit (Leaky ReLU) activation function

$\displaystyle\operatorname{LeakyReLU}[\alpha](x)=\begin{cases}x&x\geq 0\\\
\alpha x&x<0\end{cases}$

Note that $\operatorname{LeakyReLU}[\alpha]$ activation function is hyper-
parametrized by $\alpha$ and it is non-differentiable at the origin for all
values of $\alpha$ except $\alpha=1$. For $\alpha=0$ and $\alpha=0.01$,
$\operatorname{LeakyReLU}[\alpha]$ reduces to well known activation functions
ReLU and Leaky ReLU, respectively. For a given $n\in\mathbb{N}$, and
$\alpha\neq 1$, a smooth approximation of $\operatorname{LeakyReLU}[\alpha]$
is given by

$\displaystyle\text{SAU}=G(x,\alpha,n)=(\operatorname{LeakyReLU}[\alpha]*\phi_{n})(x)=\frac{1}{2n}\sqrt{\frac{2}{\pi}}e^{\frac{-n^{2}x^{2}}{2}}+\frac{(1+\alpha)}{2}x$
$\displaystyle+\frac{(1-\alpha)}{2}x\
\operatorname{erf}\left(\frac{nx}{\sqrt{2}}\right).$ (3)

where $\operatorname{erf}$ is the Gaussian error function

$\displaystyle\operatorname{erf}(x)=\dfrac{2}{\sqrt{\pi}}\int_{0}^{x}e^{-t^{2}}\,dt.$

It is noticeable that this function is not zero centred but passes by an
extremely close neighbourhood of zero. To make the function zero centered, we
have multiplied the first term of (3) by a linear component $x$. To further
investigate these two functions as a possible candidates for activation
function, we have conducted several experiments on MNIST [16], CIFAR10 [17],
and CIFAR100 [17] datasets with PreActResNet-18 [18], VGG-16 (with batch-
normalization) [19] [20], and DenseNet-121 [21] models, and we notice that
both of them performs almost similar in every cases. So, for the rest of the
paper, we will only consider the approximate identity of Leaky ReLU
($\alpha=0.15$) given in (3) as the activation function. We call this function
Smooth Approximation Unit (SAU).

We note that in GELU [13] paper, a similar idea is utilized to obtain their
activation functions. They use the product of $x$ with the cumulative
distribution function of a suitable probability distribution (see [13] for
further details).

Figure 1: Approximation of Leaky ReLU ($\alpha=0.15$) using SAU.

###  3.1 Learning activation parameters via back-propagation

Back-propagation algorithm [22] and gradient descent is used in neural
networks to update Weights and biases. Parameters in trainable activation
functions are updated using the same technique. The forward pass is
implemented in both Pytorch [23] & Tensorflow-Keras [24] API, and automatic
differentiation will update the parameters. Alternatively, CUDA [25] based
implementation (see [2]) can be used and the gradients of equation (3) for the
input $x$ and the parameter $\alpha$ can be computed as follows:

$\displaystyle\frac{\partial G}{\partial
x}=\frac{-n^{2}x}{2n}\sqrt{\frac{2}{\pi}}e^{\frac{-n^{2}x^{2}}{2}}+\frac{(1+\alpha)}{2}+\frac{(1-\alpha)}{2}\
\operatorname{erf}\left(\frac{nx}{\sqrt{2}}\right)+\frac{n(1-\alpha)}{\sqrt{2\pi}}\
x\ e^{-{\frac{n^{2}x^{2}}{2}}}$ (4) $\displaystyle\frac{\partial
G}{\partial\alpha}=\frac{x}{2}\left(1-\operatorname{erf}\left(\frac{nx}{\sqrt{2}}\right)\right).$
(5)

where

$\displaystyle\frac{d}{dx}\text{erf}(x)=$ $\displaystyle\
\frac{2}{\sqrt{\pi}}e^{-x^{2}}$

$\alpha$ and $n$ can be either hyperparameters or trainable parameters.
Note that the class of neural networks with SAU activation function is dense
in $C(K)$, where $K$ is a compact subset of $\mathbb{R}^{n}$ and $C(K)$ is the
space of all continuous functions over $K$.
The proof follows from the following propositions (see [8]) as the proposed
function is not a polynomial.

Proposition 1. (Theorem 1.1 in Kidger and Lyons, 2020 [26]) :- Let
$\rho:\mathbb{R}\rightarrow\mathbb{R}$ be any continuous function. Let
$N_{n}^{\rho}$ represent the class of neural networks with activation function
$\rho$, with $n$ neurons in the input layer, one neuron in the output layer,
and one hidden layer with an arbitrary number of neurons. Let
$K\subseteq\mathbb{R}^{n}$ be compact. Then $N_{n}^{\rho}$ is dense in $C(K)$
if and only if $\rho$ is non-polynomial.

##  4 Experiments

We have considered eight popular standard activation functions to compare
performance with SAU on different datasets and models on standard deep
learning problems like image classification, object detection, semantic
segmentation, and machine translation. It is evident from the experimental
results in nest sections that SAU outperform in most cases compared to the
standard activations. For the rest of our experiments, we have considered
$\alpha$ as a trainable parameter and $n$ as a hyper-parameter. We have
initialized $\alpha$ at $0.15$ and updated it via backpropagation according to
(5). The value of $n$ is considered $20000$. All the experiments are conducted
on an NVIDIA V100 GPU with 32GB RAM.

###  4.1 Image Classification

####  4.1.1 MNIST, Fashion MNIST and The Street View House Numbers (SVHN)
Database

In this section, we present results on MNIST [16], Fashion MNIST [27], and
SVHN [28] datasets. The MNIST and Fashion MNIST databases have a total of 60k
training and 10k testing $28\times 28$ grey-scale images with ten different
classes. SVHN consists of $32\times 32$ RGB images with a total of 73257
training images and 26032 testing images with ten different classes. We do not
apply any data augmentation method in MNIST and Fashion MNIST datasets. We
have applied standard data augmentation methods like rotation, zoom, height
shift, shearing on the SVHN dataset. We report results with VGG-16 [20] (with
batch-normalization) network in Table 1.

Activation Function |  MNIST |  Fashion MNIST |  SVHN
---|---|---|---
ReLU | $99.05\pm 0.11$ | $93.13\pm 0.23$ | $95.09\pm 0.26$
Swish | $99.16\pm 0.09$ | $93.34\pm 0.21$ | $95.29\pm 0.20$
Leaky ReLU($\alpha$ = 0.01) | $99.02\pm 0.14$ | $93.17\pm 0.28$ | $95.24\pm 0.23$
ELU | $99.01\pm 0.15$ | $93.12\pm 0.30$ | $95.15\pm 0.28$
Softplus | $98.97\pm 0.14$ | $92.98\pm 0.34$ | $94.94\pm 0.30$
GELU | 99.18$\pm$0.09 | 93.41$\pm$0.29 | 95.11 $\pm$0.24
PReLU | 99.01$\pm$0.09 | 93.12$\pm$0.27 | 95.14 $\pm$0.24
ReLU6 | 99.20$\pm$0.08 | 93.25$\pm$0.27 | 95.22 $\pm$0.20
SAU |  99.28$\pm$0.06 |  93.57$\pm$0.20 |  95.41$\pm$0.18
Table 1: Comparison between different baseline activations, SAU activation on
MNIST, Fashion MNIST, and SVHN datasets on VGG-16 network. We report results
for 10-fold mean accuracy (in %). mean$\pm$std is reported in the table.

####  4.1.2 CIFAR

The CIFAR [17] is one of the most popular databases for image classification
consists of a total of 60k $32\times 32$ RGB images and is divided into 50k
training and 10k test images. CIFAR has two different datasets- CIFAR10 and
CIFAR100 with a total of 10 and 100 classes, respectively. We report the top-1
accuracy on Table 2 and Table 3 on CIFAR10 dataset CIFAR100 datasets
respectively. We consider MobileNet V2 (MN V2) [29], Shufflenet V2 (SF V2)
[30], and EfficientNet B0 (EN-B0) [31]. For all the experiments to train a
model on these two datasets, we use a batch size of 128, stochastic gradient
descent ([32], [33]) optimizer with 0.9 momentum & $5e^{-4}$ weight decay, and
trained all networks up-to 200 epochs. We begin with 0.01 learning rate and
decay the learning rate with cosine annealing [34] learning rate scheduler. We
have applied Standard data augmentation methods like width shift, height
shift, horizontal flip, and rotation on both datasets. It is noticeable from
these two tables that replacing ReLU by SAU, there is an increment in top-1
accuracy from 1% to more than 5%.

Model ReLU SAU Top-1 accuracy (mean$\pm$ std) Top-1 accuracy (mean $\pm$ std)
EffitientNet B0 95.01 $\pm$ 0.12 96.07 $\pm$ 0.08 MobileNet V2 94.07 $\pm$
0.15 95.25 $\pm$ 0.10

Table 2: Experimental results on CIFAR10 dataset. Top-1 accuracy(in $\%$) for
mean of 8 different runs have been reported. mean$\pm$std is reported in the
table.

Model ReLU SAU Top-1 accuracy (mean$\pm$ std) Top-1 accuracy (mean $\pm$ std)
Shufflenet V2 0.5x 62.02 $\pm$ 0.20 64.39 $\pm$ 0.17 Shufflenet V2 1.0x 64.57
$\pm$ 0.25 68.18 $\pm$ 0.16 Shufflenet V2 2.0x 67.11 $\pm$ 0.23 72.23 $\pm$
0.16

Table 3: Experimental results on CIFAR100 dataset. Top-1 accuracy(in $\%$) for
mean of 8 different runs have been reported. mean$\pm$std is reported in the
table.

Figure 2: Top-1 Train and Test accuracy (higher is better) on CIFAR100 dataset
with ShuffleNet V2 (2.0x) model for SAU and ReLU.

Figure 3: Top-1 Train and Test loss (lower is better) on CIFAR100 dataset with
ShuffleNet V2 (2.0x) model for SAU and ReLU.

####  4.1.3 Tiny Imagenet

In this section, we present result on Tiny ImageNet dataset, a similar kind of
image classification database like the ImageNet Large Scale Visual Recognition
Challenge(ILSVRC). Tiny Imagenet contains $64\times 64$ RGB images with total
100,000 training images, 10,000 validation images, and 10,000 test images and
have total 200 image classes. We report a mean of 5 different runs for Top-1
accuracy in table 4 on WideResNet 28-10 (WRN 28-10) [35] model. We consider a
batch size of 32, He Normal initializer [4], 0.2 dropout rate [36], adam
optimizer [37], initial learning rate(lr rate) 0.01, and lr rate is reduced by
a factor of 10 after every 60 epochs up-to 300 epochs. We have applied
standard data augmentation methods like rotation, width shift, height shift,
shearing, zoom, horizontal flip, fill mode. It is evident from the table that
the proposed function performs better than the baseline functions, and top-1
accuracy is stable (mean$\pm$std) and got a good improvement for SAU over
ReLU.

Activation Function |  Wide ResNet 28-10 Model
---|---
ReLU | 61.61$\pm$0.47
Swish | 62.44$\pm$0.49
Leaky ReLU($\alpha$ = 0.01) | 61.47$\pm$0.44
ELU | 61.99$\pm$0.57
Softplus | 60.42$\pm$0.61
GELU | 62.64$\pm$0.62
PReLU | 61.25$\pm$0.51
ReLU6 | 61.72$\pm$0.56
SAU |  63.20$\pm$0.51
Table 4: Experimental results on Tiny ImageNet dataset. Mean of 5 different
runs for top-1 accuracy(in $\%$) have been reported. mean$\pm$std is reported
in the table.

###  4.2 Object Detection

A standard problem in computer vision is object detection, in which the
network model try to locate and identify each object present in the image.
Object detection is widely used in face detection, autonomous vehicle etc. In
this section, we present our results on challenging Pascal VOC dataset [38] on
Single Shot MultiBox Detector(SSD) 300 [39] with VGG-16(with batch-
normalization) [20] as the backbone network. No pre-trained weight is
considered for our experiments in the network. The network has been trained
with a batch size of 8, 0.001 learning rate, SGD optimizer [32, 33] with 0.9
momentum, 5$e^{-4}$ weight decay for 120000 iterations. We report the mean
average precision (mAP) in Table 5 for a mean of 5 different runs.

Activation Function |  mAP
---|---
ReLU | 77.2$\pm$0.14
Swish | 77.3$\pm$0.11
Leaky ReLU($\alpha$ = 0.01) | 77.2$\pm$0.19
ELU | 75.1$\pm$0.22
Softplus | 74.2$\pm$0.25
GELU | 77.3$\pm$0.12
PReLU | 77.2$\pm$0.20
ReLU6 | 77.1$\pm$0.15
SAU |  77.7$\pm$0.10
Table 5: Object Detection results on SSD 300 model in Pascal-VOC dataset.
mean$\pm$std is reported in the table.

###  4.3 Semantic Segmentation

Semantic segmentation is a computer vision problem that narrates the procedure
of associating each pixel of an image with a class label. We present our
experimental results in this section on the popular Cityscapes dataset [40].
The U-net model [41] is considered as the segmentation framework and is
trained up-to 250 epochs, with adam optimizer [37], learning rate 5$e^{-3}$,
batch size 32 and Xavier Uniform initializer [42]. We report the mean of 5
different runs for Pixel Accuracy and mean Intersection-Over-Union (mIOU) on
test data on table 6.

Activation Function |  Pixel Accuracy | mIOU
---|---|---
ReLU | 79.60$\pm$0.45 | 69.32$\pm$0.30
Swish | 79.71$\pm$0.49 | 69.68$\pm$0.31
Leaky ReLU($\alpha$ = 0.01) | 79.41$\pm$0.42 | 69.48$\pm$0.39
ELU | 79.27$\pm$0.54 | 68.12$\pm$0.41
Softplus | 78.69$\pm$0.49 | 68.12$\pm$0.55
GELU | 79.60$\pm$0.39 | 69.51$\pm$0.39
PReLU | 78.99$\pm$0.42 | 68.82$\pm$0.41
ReLU6 | 79.59$\pm$0.41 | 69.66$\pm$0.41
SAU |  81.11$\pm$0.40 |  71.02$\pm$0.32
Table 6: semantic segmentation results on U-NET model in CityScapes dataset.
mean$\pm$std is reported in the table.

###  4.4 Machine Translation

Machine Translation is a deep learning technique in which a model translate
text or speech from one language to another language. In this section, we
report results on WMT 2014 English$\rightarrow$German dataset. The database
contains 4.5 million training sentences. Network performance is evaluated on
the newstest2014 dataset using the BLEU score metric. An Attention-based
8-head transformer network [43] is used with Adam optimizer [37], 0.1 dropout
rate [36], and trained up to 100000 steps. We try to keep other hyper-
parameters similar as mentioned in the original paper [43]. We report mean of
5 runs on Table 7 on the test dataset(newstest2014).

Activation Function |  BLEU Score on the newstest2014 dataset
---|---
ReLU | 26.2$\pm$0.15
Swish | 26.4$\pm$0.10
Leaky ReLU($\alpha$ = 0.01) | 26.3$\pm$0.17
ELU | 25.1$\pm$0.15
Softplus | 23.6$\pm$0.16
GELU | 26.4$\pm$0.19
PReLU | 26.2$\pm$0.21
ReLU6 | 26.1$\pm$0.14
SAU |  26.7$\pm$0.12
Table 7: Machine translation results on transformer model in WMT-2014 dataset.
mean$\pm$std is reported in the table.

##  5 Baseline Table

In this section, we present a table for SAU and the other baseline functions,
which shows that SAU beat or perform equally well compared to baseline
activation functions in most cases. We present a detailed comparison based on
all the experiments in earlier sections with SAU and the baseline activation
functions in Table 8.

Baselines | ReLU |  Leaky ReLU | ELU | Softplus | Swish | PReLU | ReLU6 | GELU
---|---|---|---|---|---|---|---|---
SAU $>$ Baseline | 12 | 12 | 12 | 12 | 12 | 12 | 12 | 12
SAU $=$ Baseline | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
SAU $<$ Baseline | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0
Table 8: Baseline table for SAU. These numbers represent the total number of
models in which SAU underperforms, equal or outperforms compared to the
baseline activation functions

##  6 Conclusion

In this paper, we propose a new novel smooth activation function using
approximate identity, and we call it smooth activation unit (SAU). The
proposed function can approximate ReLU or its different variants (like Leaky
ReLU etc.). We show that on a wide range of experiments on different deep
learning problems, the proposed functions outperform the known activations
like ReLU or Leaky ReLU in most cases.

## References

  * [1] Vinod Nair and Geoffrey E. Hinton.  Rectified linear units improve restricted boltzmann machines.  In Johannes Fürnkranz and Thorsten Joachims, editors, Proceedings of the 27th International Conference on Machine Learning (ICML-10), June 21-24, 2010, Haifa, Israel, pages 807–814. Omnipress, 2010.
  * [2] Andrew L. Maas, Awni Y. Hannun, and Andrew Y. Ng.  Rectifier nonlinearities improve neural network acoustic models.  In in ICML Workshop on Deep Learning for Audio, Speech and Language Processing, 2013.
  * [3] Djork-Arné Clevert, Thomas Unterthiner, and Sepp Hochreiter.  Fast and accurate deep network learning by exponential linear units (elus), 2016.
  * [4] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, 2015.
  * [5] Prajit Ramachandran, Barret Zoph, and Quoc V. Le.  Searching for activation functions, 2017.
  * [6] Koushik Biswas, Sandeep Kumar, Shilpak Banerjee, and Ashish Kumar Pandey.  Tanhsoft - dynamic trainable activation functions for faster learning and better performance.  IEEE Access, pages 1–1, 2021.
  * [7] Koushik Biswas, Sandeep Kumar, Shilpak Banerjee, and Ashish Kumar Pandey.  Eis - efficient and trainable activation functions for better accuracy and performance.  In Igor Farkaš, Paolo Masulli, Sebastian Otte, and Stefan Wermter, editors, Artificial Neural Networks and Machine Learning – ICANN 2021, pages 260–272, Cham, 2021. Springer International Publishing.
  * [8] Alejandro Molina, Patrick Schramowski, and Kristian Kersting.  Padé activation units: End-to-end learning of flexible activation functions in deep networks, 2020.
  * [9] Koushik Biswas, Shilpak Banerjee, and Ashish Kumar Pandey.  Orthogonal-padé activation functions: Trainable activation functions for smooth and faster convergence in deep networks, 2021.
  * [10] Ningning Ma, Xiangyu Zhang, Ming Liu, and Jian Sun.  Activate or not: Learning customized activation, 2021.
  * [11] Koushik Biswas, Sandeep Kumar, Shilpak Banerjee, and Ashish Kumar Pandey.  Erfact and pserf: Non-monotonic smooth trainable activation functions, 2021.
  * [12] Diganta Misra.  Mish: A self regularized non-monotonic activation function, 2020.
  * [13] Dan Hendrycks and Kevin Gimpel.  Gaussian error linear units (gelus), 2020.
  * [14] Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, and Yoshua Bengio.  Maxout networks, 2013.
  * [15] Long-yue Li Hui-zhen Zhao, Fu-xian Liu.  Improving deep convolutional neural networks with mixed maxout units, 2017\.
  * [16] Yann LeCun, Corinna Cortes, and CJ Burges.  Mnist handwritten digit database.  ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist, 2, 2010.
  * [17] Alex Krizhevsky.  Learning multiple layers of features from tiny images.  Technical report, University of Toronto, 2009.
  * [18] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Identity mappings in deep residual networks, 2016.
  * [19] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift, 2015.
  * [20] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition, 2015\.
  * [21] Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger.  Densely connected convolutional networks, 2016.
  * [22] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel.  Backpropagation applied to handwritten zip code recognition.  Neural Computation, 1(4):541–551, 1989.
  * [23] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala.  Pytorch: An imperative style, high-performance deep learning library, 2019\.
  * [24] François Chollet et al.  Keras.  https://keras.io, 2015.
  * [25] J. Nickolls, I. Buck, M. Garland, and K. Skadron.  Scalable parallel programming.  In 2008 IEEE Hot Chips 20 Symposium (HCS), pages 40–53, 2008.
  * [26] Patrick Kidger and Terry Lyons.  Universal approximation with deep narrow networks, 2020.
  * [27] Han Xiao, Kashif Rasul, and Roland Vollgraf.  Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms.  arXiv preprint arXiv:1708.07747, 2017.
  * [28] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng.  Reading digits in natural images with unsupervised feature learning.  2011\.
  * [29] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen.  Mobilenetv2: Inverted residuals and linear bottlenecks, 2019.
  * [30] Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun.  Shufflenet v2: Practical guidelines for efficient cnn architecture design, 2018.
  * [31] Mingxing Tan and Quoc V. Le.  Efficientnet: Rethinking model scaling for convolutional neural networks, 2020.
  * [32] H. Robbins and S. Monro.  A stochastic approximation method.  Annals of Mathematical Statistics, 22:400–407, 1951.
  * [33] J. Kiefer and J. Wolfowitz.  Stochastic estimation of the maximum of a regression function.  Annals of Mathematical Statistics, 23:462–466, 1952.
  * [34] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts, 2017.
  * [35] Sergey Zagoruyko and Nikos Komodakis.  Wide residual networks, 2016.
  * [36] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: A simple way to prevent neural networks from overfitting.  J. Mach. Learn. Res., 15(1):1929–1958, January 2014.
  * [37] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [38] Mark Everingham, Luc Gool, Christopher K. Williams, John Winn, and Andrew Zisserman.  The pascal visual object classes (voc) challenge.  Int. J. Comput. Vision, 88(2):303–338, June 2010.
  * [39] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C. Berg.  Ssd: Single shot multibox detector.  Lecture Notes in Computer Science, page 21–37, 2016.
  * [40] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding, 2016.
  * [41] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation, 2015\.
  * [42] Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In Yee Whye Teh and Mike Titterington, editors, Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pages 249–256, Chia Laguna Resort, Sardinia, Italy, 13–15 May 2010. JMLR Workshop and Conference Proceedings.
  * [43] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need, 2017.


# Generative Flows as a General Purpose Solution for Inverse Problems

José A. Chávez
Universidad Católica San Pablo
Arequipa, Perú
<EMAIL_ADDRESS>
###### Abstract

Due to the success of generative flows to model data distributions, they have
been explored in inverse problems. Given a pre-trained generative flow,
previous work proposed to minimize the 2-norm of the latent variables as a
regularization term. The intuition behind it was to ensure high likelihood
latent variables that produce the closest restoration. However, high-
likelihood latent variables may generate unrealistic samples as we show in our
experiments. We therefore propose a solver to directly produce high-likelihood
reconstructions. We hypothesize that our approach could make generative flows
a general purpose solver for inverse problems. Furthermore, we propose
$1\times 1$ coupling functions to introduce permutations in a generative flow.
It has the advantage that its inverse does not require to be calculated in the
generation process. Finally, we evaluate our method for denoising, deblurring,
inpainting, and colorization. We observe a compelling improvement of our
method over prior works.

##  1 Introduction

With the success of generative models in the generation of synthetic data [15,
10], prior works [4, 2, 3] explored them for solving inverse problems in image
processing. In this approach, a generative model can be pre-trained with some
data such as medical images, faces, animals, and more [21, 12, 20]. Then, this
trained model is used to find the best restoration for an inverse problem.

In an inverse problem we aim to recover $x^{*}\in\mathbb{R}^{n}$ from the
possible-noisy linear measurements $y=Ax^{*}+\eta$, where
$A\in\mathbb{R}^{m\times n}$ is the matrix measurement and
$\eta\in\mathbb{R}^{m}$ is the noise. For instance, $y$ would be a noisy or
blurred image, and $x^{*}$ would be the restoration of this image. So given a
generative model $G:z\rightarrow x$, researchers [4] proposed the following
objective function

$\hat{z}=\operatorname*{arg\,min}_{z\in\mathbb{R}^{k}}\|AG(z)-y\|^{2}+\gamma\cdot\|z\|^{2},$
(1)

where $z\in\mathbb{R}^{k}$ and $x\in\mathbb{R}^{n}$. And the most typical
choice for $\|\cdot\|$ is the 2-norm [4, 3]. Note that the network $G$ was
trained with similar samples than $x^{*}$, thus the intuition is to find
$\hat{z}$ such that the synthetic image $\hat{x}=G(\hat{z})$ is close to
$x^{*}$.

Generative Adversarial Networks (GANs) have shown great success in image
synthesis [12, 18]; however they are not invertible. Experiments in image
manipulation with GANs [1] have shown that projected images [22] are not
identical to the original ones. Thus, even if we set the identity matrix as
$A$, and $\eta=0$ in Equation 1; we will probably never achieve the zero-error
reconstruction ($x^{*}=\hat{x}$). On the other hand, Variational Auto-Encoders
(VAEs) perform a stochastic mapping from samples to latent variables. Although
VAEs have an encoder, we can infer only approximately the corresponding latent
variable for a given sample.

$0.0$

$0.2$

$0.4$

$0.6$

$0.8$

$1.0$

$1.2$

$1.4$

$1.6$

$1.8$

$2.0$

Figure 1: Samples from our generative flow trained on the CelebA dataset. We
generate images from an isotropic Gaussian distribution
$\mathcal{N}(\mathbf{0},\sigma^{2}\mathbf{I})$. Each row represents a batch of
$16$ samples by setting a specific value for $\sigma$. Note that $\sigma=0$
represents $z=\mathbf{0}$.

Generative flows [8] have a particular advantage among generative models; they
are invertible networks. Hence, these models can perform the re-generation of
any image—there exists a latent variable for any image $x^{*}$ in an inverse
problem, and is unique. A sequence of bijective functions allows the exact
reconstruction of a sample with its corresponding latent variable. Therefore,
zero-error reconstruction is possible and easy to achieve. Authors in [2]
proposed to solve a unique inverse problem with generative flows. In this
approach, the aim is to encode the corresponding measurements $y$ from a given
image $x^{*}$. Then, once the model is trained with synthetic pairs of
$\\{x^{*},y\\}$, it may estimate the corresponding original image from some
measurements $y$. Note that this method recovers the original image from its
measurements once it is trained on a specific inverse problem. One must train
different pairs of $\\{x^{*},y\\}$ for each task.

In contrast, GlowIP [3] explored pre-trained generative flows for compressed
sensing, denoising and inpainting. In GlowIP, the Glow architecture [14] is
trained on the CelebA dataset [16], thus encoding a latent variable for each
image as $G^{-1}:x\rightarrow z$. Once the model is trained, GlowIP performs
Equation 1 to solve an inverse problem. This model does not require the
knowledge of any specific inverse problem in the training process—it is only
trained with images from CelebA.

Note that Equation 1 has a regularization term $\|z\|^{2}$. As authors suggest
in [4, 3], the intuition behind it is to encourage high likelihood latent
variables, because $G$ is typically trained with the latent distribution
$\mathcal{N}(\mathbf{0},\mathbf{I})$. However, a high log-likelihood in the
latent representation may generate a low likelihood sample $x$. Indeed, the
sample $x_{0}=G(\mathbf{0})$ is a blurry and unlikely image in a generative
flow (see Figure 1). Our hypothesis is that $z$ must produce realistic samples
instead. Recently, researchers [19] have attempted to generalize Equation 1,
they proposed a maximum a posteriori (MAP) objective for image denoising as
follows

$\displaystyle\hat{z}=\operatorname*{arg\,min}_{z\in\mathbb{R}^{k}}\frac{1}{2\sigma^{2}}\|G(z)-y\|^{2}-\beta\cdot\log
p_{G}(G(z)),$ (2)

where $\sqrt{\mathbb{E}[\|\eta\|^{2}]}=\sigma$, and $p_{G}(G(z))$ is the
density estimation by $G$. However, one must be careful when using density
estimates from a generative flow. In the training process, $G$ only sees
samples from its corresponding dataset. Nevertheless, one may obtain the
density estimates from out-of-distribution samples. In fact, a generative flow
may assign a high likelihood to constant or random images [17]. Additionally,
the estimation of $p_{G}(G(z))$ involves using the generative flow in the form
of $G$ and then $G^{-1}$. As the authors suggest in [19], the hyperparameter
$\beta$ should be configured to control this problem.

On the other hand, permutations represent a key process in generative flows to
handle the expressiveness of bijective functions. The Glow architecture [14]
requires invertible $1\times 1$ convolutions to permute the entries of hidden
feature maps. Note that this architecture requires computing the inverse of
the invertible $1\times 1$ convolutions in order to solve Equation 1 or
Equation 2. Invertible $1\times 1$ convolutions have shown impressive
performance in the generation of synthetic high-resolution images.
Nonetheless, the inverse of this layer has a significant cost in limited
resources, which could be avoided.

In this work, we aim to explore pre-trained generative flows as a general
purpose solution for inverse problems. The contributions are summarized in the
following points:

  * •

A solver to produce realistic restorations in inverse problems through a pre-
trained generative flow. Our approach demonstrates quantitive and qualitative
improvement among the state-of-the-art methods in many inverse problems.

  * •

We introduce $1\times 1$ coupling functions to permute the channels of feature
maps in a generative flow. This permutation does not require the computation
of its inverse during the generation of samples.

Code to implement our solver and reproduce results is publicly available111The
code is available at https://github.com/jarchv/solverip.git.

##  2 Preliminaries

In this section, we give an overview on the generative flow model as well as
existing methods to introduce permutations.

###  2.1 Generative flows

Generative flows can be trained by the maximum likelihood using the change of
variable formula

$\displaystyle\log(p_{X}(x))=\log(p_{Z}(z))+\log(|\det\mathbf{J}_{F}(x)|),$
(3)

where $F:x\rightarrow z$ is an invertible network that maps from a sample
$x\sim p_{X}$ to the latent variable $z\sim p_{Z}$; and
$\mathbf{J}_{F}(x)=\frac{\partial F(x)}{\partial x}$ is the Jacobian of $F$
evaluated at $x$. The network $F$ is composed of a sequence of bijections as
follows

$x\underset{F^{-1}_{1}}{\overset{F_{1}}{\mathrel{\begin{subarray}{c}\xrightarrow{\rule{19.91684pt}{0.0pt}}\\\\[-3.87495pt]
\xleftarrow{\rule{19.91684pt}{0.0pt}}\end{subarray}}}}h^{[1]}\underset{F^{-1}_{2}}{\overset{F_{2}}{\mathrel{\begin{subarray}{c}\xrightarrow{\rule{19.91684pt}{0.0pt}}\\\\[-3.87495pt]
\xleftarrow{\rule{19.91684pt}{0.0pt}}\end{subarray}}}}h^{[2]}\cdots
h^{[l-1]}\underset{F^{-1}_{l}}{\overset{F_{l}}{\mathrel{\begin{subarray}{c}\xrightarrow{\rule{19.91684pt}{0.0pt}}\\\\[-3.87495pt]
\xleftarrow{\rule{19.91684pt}{0.0pt}}\end{subarray}}}}h^{[l]}=z,$ (4)

where $F$ must be a transformation such that its Jacobian determinant is
computationally tractable [8]. The idea is to choose transformations $F_{i}$
whose Jacobian $\frac{\partial h_{i}}{\partial h_{i-1}}$ represents a
triangular matrix—the determinant of a triangular matrix is the product of its
diagonal elements. Training $F$ in Equation 3 with maximum likelihood is also
known as Gaussianization[6], where the dimensions in $z$ becomes independent
and therefore

$p_{Z}(z)=\prod_{i=1}^{k}p_{Z_{i}}(z_{i}),$ (5)

where $k$ is number of entries of $z$. Note that Equation 5 simplifies the
maximization of the right-hand side in Equation 3. Moreover, each bijection
$F_{i}$ is also called coupling layer[8], and it is a reversible triangular
transformation where its determinant is easy to compute.

###  2.2 Permutations in generative flows

A coupling layer leaves half of its input unchanged while the other is
modified. This expressiveness of coupling layers caused by the half unchanged
can be handled with permutations. In NICE [8], a swap operation exchanges the
halves of the coupling layer input, so in two consecutive coupling layers, the
entire input is modified. The Glow model [14] implements learned invertible
$1\times 1$ convolutions, which performs impressive results in the CelebA HQ
dataset [11]. However, as the authors said the cost of computing its
determinant is $\mathcal{O}(c^{3})$, where $c$ is the number of input channels
in the invertible $1\times 1$ convolution. Furthermore, its inverse has a cost
of $\mathcal{O}(c^{3})$. On the other hand, Glow resizes the feature maps with
squeeze layers [9]. Due to a squeeze layer should produce the same number of
elements for a given input, it increments the number of channels $c$ that will
be permuted by invertible $1\times 1$ convolutions. This process increases the
number of parameters of invertible $1\times 1$ convolution, and consequently
the cost of computing its Jacobian determinant and its inverse.

The Glow architecture [14] is used in GlowIP [3] for inverse problems. Note
the process of solving inverse problems in Equation 1 involves using the
inverse of the network. It means that GlowIP should compute the inverse of
each invertible $1\times 1$ convolution in the sample generation. Moreover,
training the Glow architecture to infer latent variables from images also
involves computing the determinant of this permutation layer.

##  3 Proposed approach

In this section, we first describe our inverse problem solver to generate
realist reconstructions, and then our proposed $1\times 1$ coupling functions
to introduce permutations.

###  3.1 Inverse problem solver

Once we our generative flow $F_{\phi}$ is trained (parametrized by $\phi$)
with the CelebA dataset[16], we proceed to solve inverse problems. Let assume
that we aim to recover $x^{*}\in\mathbb{R}^{n}$ from possible-noisy linear
measurements $y=Ax^{*}+\eta$, where $A\in\mathbb{R}^{m\times n}$ and
$\eta\in\mathbb{\mathbb{missing}}{R}^{m}$. Then, given a pre-trained
generative flow $F_{\phi}:x\rightarrow z$ , where $x,z\in\mathbb{R}^{n}$, we
may rewrite Equation 3 as

$\displaystyle\log(p_{\phi}(x))$
$\displaystyle=\log(p_{Z}(z))-\log(|\det\mathbf{J}_{F_{\phi}^{-1}}(z)|),$ (6)

where $\log(p_{\phi}(x))$ is the log-likelikehood. Note that we replace the
term $|\det\mathbf{J}_{F_{\phi}}(x)|$ by
$|\det\mathbf{J}_{F_{\phi}^{-1}}(z)|^{-1}$ in Equation 3. Now,
$\mathbf{J}_{F_{\phi}^{-1}}(z)$ is the Jacobian of $F_{\phi}^{-1}$ evaluated
at $z$.

Note that the maximization of $\log(p_{Z}(z))$ is equivalent to the
minimization of $\|z\|^{2}$, which is proposed in [4, 3]. However, it does not
ensure the maximization of the log-likelihood $\log(p_{\phi}(x))$—we must
maximize the Jacobian term as well. The maximization of the right-hand side in
Equation 6 directly ensures the generation of high log-likelihood samples $x$,
which is what we pretend. Then, the objective function to find the latent
variable that recovers $x^{*}$ from $y$ in an inverse problem is

$\displaystyle\hat{z}$
$\displaystyle=\operatorname*{arg\,min}_{z\in\mathbb{R}^{k}}\left\\{\|AF^{-1}(z)-y\|_{1}+\alpha\cdot\mathcal{L}(z)\right\\},$
(7) $\displaystyle\mathcal{L}(z)$
$\displaystyle=-\log(p_{Z}(z))+\log(|\det\mathbf{J}_{F_{\phi}^{-1}}(z)|),$ (8)

where $\alpha>0$ is a hyperparameter. We find empirically that our
regularization term $\mathcal{L}(z)$ performs better with the 1-norm in
Equation 7. Thus, while our solver generates realistic samples, it is also
finding the optimal reconstruction of the inverse problem.

###  3.2 $1\times 1$ Coupling functions

Figure 2 shows our generative flow architecture. A flow step represents a
sequence of an activation normalization [14] and a coupling layer. Inside a
coupling layer, we implement swap operations. It introduce permutations as
follow

$\displaystyle h_{1},h_{2}=\texttt{swap}(\texttt{split}(h)),$ (9)

where split separates the input tensor $h\in\mathbb{R}^{n}$ into two chunks
$h_{1}\in\mathbb{R}^{n/2}$ and $h_{2}\in\mathbb{R}^{n/2}$. Note that a swap
operation has constant time complexity $\mathcal{O}(1)$. Then, we set a
convolutional neural network CNN as the coupling function, which process the
first chunk as

$\displaystyle s,t=\texttt{CNN}(h_{1}),$ (10)

where $s$ and $t$ have the dimension as $h_{1}$. Inspired on Glow [14], we
perform the following affine transformation on $h_{2}$

$\displaystyle h^{\prime}_{2}$
$\displaystyle=h_{2}\odot\texttt{sigmoid}(s+2)+t,$ (11)

where sigmoid is the sigmoid function. Finally, we apply a concat operation to
obtain the output tensor $h^{\prime}$, which has the same dimension as the
input $h$

$\displaystyle h^{\prime}$
$\displaystyle=\texttt{concat}(\texttt{swap}(h_{1},h^{\prime}_{2})).$ (12)

Note that if we apply a swap operation in Equation 9, then we should apply
another one in Equation 12 to hold the order of the entries in $h$.

An additional permutation is performed in our generative flow architecture.
Our coupling function applies $1\times 1$ convolutions on $1\times 1$ feature
maps $h$. Hence, each $1\times 1$ coupling function permutes all the elements
of $1\times 1$ features maps. However, each of these coupling functions does
not require the computation of its inverse during sample generation. Finally,
in the process of training our generative flow, the Jacobian of $1\times 1$
coupling function is not required to be computed.

Figure 2: Our generative flow architecture. Each flow step is composed by an
activation normalization [14] followed by our proposed coupling layer. These
flow steps are combined with squeeze layers. We also split low-scale feature
maps in two chunks, and force one of the halves ($h_{z}$) to directly follow
$p_{Z}$. The other half continues its path until produce the $1\times 1$
latent variable $z$.

Target

Noisy

CSGM

GlowIP

MAP

BM3D

Ours

Figure 3: Image denoising on images from the CelebA dataset. The first row is
the target image $x^{*}$ and the second row is the noisy image $y$, we aim to
recover $x^{*}$ from $y$. From the 3rd-7th rows, we show results for CSGM,
GlowIP, MAP, BM3D and our proposed solver. We synthetically create noisy
images with a noise level of $\sqrt{\mathbb{E}[\|\eta\|^{2}]}=0.1$.

Target

Blurry

CSGM

GlowIP

Ours

Figure 4: Image deblurring on images from the CelebA dataset. The first row is
the target image $x^{*}$ and the second row is the blurry image $y$, we aim to
recover $x^{*}$ from $y$. From the 3rd-5th rows, we show results for CSGM,
GlowIP, and our proposed solver. We create synthetic blurry images by applying
a $3\times 3$ average pooling with stride $1$ on targets images.

##  4 Experiments

Target

Masked

CSGM

GlowIP

Ours

Figure 5: Image inpainting on images from the CelebA dataset. The first row is
the target image $x^{*}$ and the second row is the masked image $y$, we aim to
recover $x^{*}$ from $y$. From the 3rd-5th rows, we show results for CSGM,
GlowIP, and our proposed solver. We crop a region of images to create the
masked images.

This section illustrates the experimental details and presents the comparison
of our proposed solver to other state-of-the-art methods. The performance
evaluation is split into four parts: denoising, deblurring, inpainting, and
colorization. Then, we show how realistic are samples with different standard
deviations in the latent distribution. Finally, we conduct an ablation study
to compare the time speed and memory usage when we use $1\times 1$ coupling
functions and invertible $1\times 1$ convolution.

##  5 Training details and parameters settings

First, we resize to $32\times 32$ resolution all images of the CelebA dataset
[16]. Then, we fix a generative flow with the training samples. The Adam
optimizer [13] is used during the training with parameters $\alpha=0.0001$,
$\beta_{1}=0.5$ and $\beta_{2}=0.999$. Inspired on [9], we force half of the
low-scale feature maps to follow the latent distribution as we can see in
Figure 2. We perform squeeze layers [9] to reduce and increase the scale of
feature maps. Additionally, we set $L=5$ as the number of scales, and $K=2$ as
the number of flow steps per scale. The value of $K$ is duplicated for scales
$2\times 2$ and $1\times 1$—we use $14$ flow steps in total. We set the batch
size as $32$ and stop the training process after $100$ epochs. Finally, we use
$512$ hidden channels for each network CNN.

Once the model is trained, we optimize Equation 7 and initialize the latent
variable at $z_{0}\sim\mathcal{N}(\mathbf{0},\sigma^{2}\mathbf{I})$ for
$\sigma=0.1$. In this process, we set $\alpha=0.005$, $\beta_{1}=0.9$ and
$\beta_{2}=0.999$. We configure the batch size as $32$, and after $1500$
iterations we return the restored image for the respective inverse problem.
All experiments are made in an NVIDIA GeForce GTX 750 Ti.

To compare with other methods such as CSGM [4] and GlowIP [3], we use their
initialization and objective function in our pre-trained generative flow. Note
that CSGM and GlowIP perform the same objective function (Equation 1), the
unique difference is the initialization: random initialization for CSGM and
zero initialization for GlowIP. We therefore test many values for their
penalization hyperparameter $\gamma$. Each value of this hyperparameter in our
experiments is the one that achieves the best performance for each inverse
problem. Additionally, in image denoising, we also compare with BM3D [7] and
MAP [19]. We use the PSNR and SSIM metrics to measure the quality of
restorations on each inverse problem.

###  5.1 Denoising

For image denoising we use a noise level of
$\sqrt{\mathbb{E}[\|\eta\|^{2}]}=0.1$. Figure 3 shows that CSGM ($\gamma=0.1$)
smooths the noisy image but can not remove all distortions. These distortions
are slightly more visible in GlowIP ($\gamma=0.1$) than CSGM. On the other
hand, MAP [19] has similar visual results to GlowIP. In MAP, we use the same
initialization as ours, and $0.0015$ as the learning rate with $\beta=0.5$—we
find empirically the best performance for this method using these settings.
Our method outperforms BM3D in the PSNR and SSIM metrics as you can see in
Table 1. Indeed, our restorations look sharper and more realistic than those
from BM3D (see Figure 3). Nonetheless, our method eliminates some details from
the background in samples. In this case, we set $\alpha=0.05$ in Equation 7.

###  5.2 Deblurring

We create synthetic blurry images by applying an average pooling in the target
image $x^{*}$, and use it as the measurements $y$. There is no significant
difference between CSGM ($\gamma=0.01$) and GlowIP ($\gamma=0$) in image
deblurring (see Figure 4). Both produce visual artifacts that are exacerbated
if we increase their penalization term—CSGM and GlowIP mitigate visual
artifacts with $\gamma=0.01$ and $\gamma=0$ respectively. As you can see in
Table 1, our method outperforms GlowIP and CSGM in the PSNR and SSIM metrics.
Moreover, our results look more sharp and close to the target image. Our
hyperparameter $\alpha$ in this case is $0.02$.

Table 1: Average PSNR(dB) and SSIM for CSGM, GlowIP, MAP, BM3D and our method. We evaluate on inverse problems such as denoising, deblurring, inpainting and colorization. Each numerical measurement is the average over $192$ samples. Method | Denoising | Deblurring | Inpainting | Colorization
---|---|---|---|---
| PSNR | SSIM | PSNR | SSIM | PSNR | SSIM | PSNR | SSIM
CSGM [4] | $37.41$ | $0.897$ | $37.17$ | $0.883$ | $36.39$ | $0.839$ | $34.71$ | $0.897$
GlowIP [3] | $37.21$ | $0.894$ | $35.82$ | $0.867$ | $37.81$ | $0.903$ | $\mathbf{35.79}$ | $0.937$
MAP [19] | $37.35$ | $0.898$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
BM3D [7] | $37.65$ | $0.905$ | $-$ | $-$ | $-$ | $-$ | $-$ | $-$
Ours | $\mathbf{37.72}$ | $\mathbf{0.918}$ | $\mathbf{39.72}$ | $\mathbf{0.963}$ | $\mathbf{37.97}$ | $\mathbf{0.913}$ | $35.72$ | $\mathbf{0.940}$

###  5.3 Inpainting

For inpainting, we mask a square in the center of the target image and fill it
with zeros. We obtain the best performance in CSGM for $\gamma=0.01$. But even
so, CSGM produces many visual artifacts. In contrast, GlowIP ($\gamma=0$)
produces a realistic reconstruction of the black square with few visual
artifacts. As we can see in Table 1, our method slightly outperforms GlowIP,
however our visual results look similar to those from GlowIP (see Figure 5).
Apparently, the log-likelihood term does not significantly improve the
performance. We obtain our best performance for $\alpha=0.002$.

###  5.4 Colorization

Target

Gray

CSGM

GlowIP

Ours

Figure 6: Image colorization on images from the CelebA dataset. The first row
is the target image $x^{*}$ and the second row is the grayscale image $y$, we
aim to recover $x^{*}$ from $y$. From the 3rd-5th rows, we show results for
CSGM, GlowIP, and our proposed solver. We create grayscale images by averaging
the channels of the target images.

In the case of the colorization problem, we average the three channels of the
target image to produce $y$. CSGM ($\gamma=0.01$) generates over-saturated
images, which have different tones than target images as we can see in Figure
6. Instead, GlowIP ($\gamma=0$) produces washed-out images, and similar to
image deblurring, its regularization term does not improve its performance.
Our method performs the colorization problem, however our results are similar
to those from GlowIP. In this case, our solver does not achieve a significant
improvement over state-of-the-art methods. Table 1 shows our results for
$\alpha=0.02$.

###  5.5 Additional experiments

Due to our generative flow is trained with a latent distribution
$\mathcal{N}(\mathbf{0},\mathbf{I})$, we must use this distribution to produce
synthetic samples. Hence, we experiment with different standard deviations of
the latent distribution. Given a latent variables
$z\sim\mathcal{N}(\mathbf{0},\sigma^{2}\mathbf{I})$, we sample for many values
of $\sigma$ as you can see in Figure 1. Our experiments show that if we set
$z_{0}=\mathbf{0}$, the sample $F^{-1}(z_{0})=x_{0}$ is a blurry and
unrealistic image. As we increase the standard deviation in the latent
variables, we generate high likelihood samples, however we also introduce
visual artifacts. Our experiments show that latent variables with a $\sigma$
in $[0.8,1.2]$ produce the most realistic samples.

As you can see in Figure 1, high likelihood latent variables do not ensure
realistic samples. Both CSGM and GlowIP have problems in image denoising and
image deblurring. In the case of inpainting, all methods (including ours) can
not perform the reconstruction of non-frontal images. Maybe, a deeper
generative flow could generate more realistic restorations—we perform a
shallow generative flow for our experiments. An analysis of continuous
normalizing flows [5] as a pre-trained model is beyond the scope of this work.

###  5.6 Ablation study

We conduct an ablation study to verify the computational cost on the
generation process. As we mentioned before, solving inverse problems
implicates using the inverse of our network. Thus, we compare two
configurations of our architecture. The first one introduces permutations with
invertible $1\times 1$ convolutions; we follow the configuration of Glow [14]
with three different scales ($L=3$). In this configuration, each flow step is
a sequence of activation normalization, invertible $1\times 1$ convolution and
coupling layer. The second architecture employs our $1\times 1$ coupling
functions for permutations. Both configurations set $14$ as the total number
of flow steps.

Table 2 proves that our method is faster than invertible $1\times 1$
convolutions in the generation process. It is worth mentioning that, on
average, our generation time is about $6.4$ms less than the other
configuration. Note that both architectures have a similar amount of
parameters. Nevertheless, invertible $1\times 1$ convolutions require the
computation of its inverse in Equation 7. This process increases the time
needed to generate a sample.

Table 2: We compare two architectures with respect to the number of parameters and time speed measured using milliseconds. We configure the first one with invertible $1\times 1$ convolutions, and the second one with our proposed $1\times 1$ coupling functions. The numerical measurements for time speed show the mean and standard deviation over $1500$ iterations and $128$ samples. Method | Time (ms) | # of parameters
---|---|---
Inv. $1\times 1$ Conv. [14] | $10.86\pm 0.18$ |  $33.8$M
Ours | $\mathbf{4.46\pm 0.03}$ |  $\mathbf{32.7}$M

##  6 Discussion

We demonstrate that our solver improves the performance at denoising,
deblurring and inpainting in low-resolution images from the CelebA dataset
[16]. Indeed, our reconstructions are the sharpest and closest to the original
image. Furthermore, we introduce $1\times 1$ coupling functions to save
computational time in the generation process of generative flows. Our method
ensures the generation of high likelihood samples, which results in sharp and
realistic restorations in inverse problems. Our experiments suggest that
generative flows can be a general purpose solution for inverse problems. Note
that we use the same network for each inverse problem, we do not need to re-
train a new network for each task. A future direction is to evaluate our
method on high-resolution images and real-world examples.

## References

  * [1] Rameen Abdal, Peihao Zhu, Niloy J. Mitra, and Peter Wonka.  Styleflow: Attribute-conditioned exploration of stylegan-generated images using conditional continuous normalizing flows.  ACM Trans. Graph., 40(3), May 2021.
  * [2] Lynton Ardizzone, Jakob Kruse, Carsten Rother, and Ullrich Köthe.  Analyzing inverse problems with invertible neural networks.  In International Conference on Learning Representations, 2019.
  * [3] Muhammad Asim, Max Daniels, Oscar Leong, Ali Ahmed, and Paul Hand.  Invertible generative models for inverse problems: mitigating representation error and dataset bias.  In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 399–409. PMLR, 13–18 Jul 2020\.
  * [4] Ashish Bora, Ajil Jalal, Eric Price, and Alexandros G Dimakis.  Compressed sensing using generative models.  In International Conference on Machine Learning, pages 537–546. PMLR, 2017.
  * [5] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud.  Neural ordinary differential equations.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  * [6] Scott Chen and Ramesh Gopinath.  Gaussianization.  In T. Leen, T. Dietterich, and V. Tresp, editors, Advances in Neural Information Processing Systems, volume 13. MIT Press, 2000.
  * [7] Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian.  Image denoising by sparse 3-d transform-domain collaborative filtering.  IEEE Transactions on image processing, 16(8):2080–2095, 2007.
  * [8] Laurent Dinh, David Krueger, and Yoshua Bengio.  Nice: Non-linear independent components estimation.  arXiv preprint arXiv:1410.8516, 2014.
  * [9] Laurent Dinh, Jascha Sohl-Dickstein, and Samy Bengio.  Density estimation using real nvp.  In International Conference on Learning Representations 2017 (Conference Track), 2017.
  * [10] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014.
  * [11] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen.  Progressive growing of gans for improved quality, stability, and variation.  arXiv preprint arXiv:1710.10196, 2017.
  * [12] Tero Karras, Miika Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila.  Alias-free generative adversarial networks.  Advances in Neural Information Processing Systems, 34, 2021.
  * [13] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [14] Durk P Kingma and Prafulla Dhariwal.  Glow: Generative flow with invertible 1x1 convolutions.  In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
  * [15] Diederik P. Kingma and Max Welling.  Auto-Encoding Variational Bayes.  In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
  * [16] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Deep learning face attributes in the wild.  In Proceedings of the IEEE international conference on computer vision, pages 3730–3738, 2015.
  * [17] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan.  Do deep generative models know what they don’t know?  arXiv preprint arXiv:1810.09136, 2018.
  * [18] Axel Sauer, Katja Schwarz, and Andreas Geiger.  Stylegan-xl: Scaling stylegan to large diverse datasets.  arXiv preprint arXiv:2202.00273, 2022.
  * [19] Jay Whang, Qi Lei, and Alex Dimakis.  Solving inverse problems with a flow-based noise model.  In International Conference on Machine Learning, pages 11146–11157. PMLR, 2021.
  * [20] Yan Wu, Jeff Donahue, David Balduzzi, Karen Simonyan, and Timothy Lillicrap.  Logan: Latent optimisation for generative adversarial networks.  arXiv preprint arXiv:1912.00953, 2019.
  * [21] Xin Yi, Ekta Walia, and Paul Babyn.  Generative adversarial network in medical imaging: A review.  Medical Image Analysis, 58:101552, 2019.
  * [22] Jun-Yan Zhu, Philipp Krähenbühl, Eli Shechtman, and Alexei A. Efros.  Generative visual manipulation on the natural image manifold.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Computer Vision – ECCV 2016, pages 597–613, Cham, 2016. Springer International Publishing.