# Fast Bayesian Uncertainty Estimation of Batch Normalized Single Image Super-
Resolution Network

Aupendu Kar
Indian Institute of Technology Kharagpur
Kharagpur, WB, India
<EMAIL_ADDRESS>Prabir Kumar Biswas
Indian Institute of Technology Kharagpur
Kharagpur, WB, India
<EMAIL_ADDRESS>
###### Abstract

In recent years, deep convolutional neural network (CNN) has achieved
unprecedented success in image super-resolution (SR) task. But the black-box
nature of the neural network and due to its lack of transparency, it is hard
to trust the outcome. In this regards, we introduce a Bayesian approach for
uncertainty estimation in super-resolution network. We generate Monte Carlo
(MC) samples from a posterior distribution by using batch mean and variance as
a stochastic parameter in the batch-normalization layer during test time.
Those MC samples not only reconstruct the image from its low-resolution
counterpart but also provides a confidence map of reconstruction which will be
very impactful for practical use. We also introduce a faster approach for
estimating the uncertainty, and it can be useful for real-time applications.
We validate our results using standard datasets for performance analysis and
also for different domain-specific super-resolution task. We also estimate
uncertainty quality using standard statistical metrics and also provides a
qualitative evaluation of uncertainty for SR applications.

##  1 Introduction

(a) HR Image

(b) Reconstructed HR Image

(c) Uncertainty Map of Reconstruction

Figure 1: This example shows the performance of the Bayesian approach in SISR.
The first image from the left side is from ’Set5’ dataset; the middle one is a
satellite [1] image and the last one is a histopathology [21] image. The
reconstructed image is with scaling factor $\times 4$. We generate uncertainty
map and reconstruct HR image using $25$ MC samples. We observe model
uncertainty in edges and geometrically complex regions like boundaries of
nuclei in histopathology sample, and the gap between two buildings in the
satellite image. We use the model trained on natural images to reconstruct the
HR images of those different domain data without any domain adaptation.

Single image super-resolution (SISR) is an ill-posed low vision problem which
generates high resolution (HR) image from a low resolution (LR) image. There
is a possibility of making an infinite number of HR images which can be sub-
sampled into the same LR image. Due to the advancement of deep learning
techniques, the community has developed a state-of-the-art SISR network using
deep neural architectures [38]. SR has a wide range of applications from
surveillance and security [44] to medical imaging [30] and many more. Other
than improving the perceptual quality of images for human interpretation, SISR
helps to boost the performance of different automated machine learning or
computer vision task [29].

Uncertainty is a powerful tool for any prediction and reconstruction system.
The confidence of the system’s output will help to improve the decision-making
process. It is a useful tool for any computer vision problem. We use the
concept of uncertainty for image super-resolution. Deep learning based SISR
techniques learns features from a dataset and features are dependent on the
images of the dataset. But real-world pictures are completely different and
contain more complex textures than the training set. Unseen texture during
test time can create inappropriate reconstruction. Due to the black-box nature
of deep learning models, it is almost impossible to know the limitation of
models or trustability of reconstructed SR image which is further processed to
other computer vision task. But we have witnessed that some artifacts,
blurriness or distortions in an image can significantly degrade the
performance of deep learning based models [22]. Some deformation in
reconstructing LR facial image may lead to wrong output in a recognition
system or any deformed reconstruction in tumor image may lead to the incorrect
estimation of tumor size. Uncertainty in deep learning (DL) models have a
transparent and robust effect in DL-based computer vision task.

Bayesian approaches in super-resolution not only provides the reconstructed HR
image but also provides the posterior distribution of super-resolution. Recent
progress in Bayesian deep learning approaches uses Monte Carlo samples that
come from a posterior distribution via dropout [32] or batch-normalization
[13]. Dropout during test time or stochastic batch mean-variance during
testing helps to generate MC samples [8, 33]. Monte Carlo methods for deep
learning model uncertainty estimation is successfully applied to
classification, segmentation [16, 27], camera relocalization [17] problems.
Generally, the deep learning community does not use dropout in image
reconstruction applications, but batch normalization is most common in SISR,
denoising, etc. Therefore, we use batch-normalization uncertainty to analyze
SISR uncertainty.

In this article, we propose a Bayesian approach to measure the quality of
reconstructed HR images from downsampled LR images. For this purpose, we add
widely used batch-normalization layer in the super-resolution network, and it
helps to generate Monte Carlo (MC) samples. These samples are different
possible HR images from a single LR image. We use the mean of those HR images
to get the reconstruction, and the variation between those HR images gives an
idea about the uncertainty of reconstruction. We estimate the uncertainty of
the reconstruction from those MC samples. We measure the quality of
uncertainty using standard statistical metrics and observe the relation
between reconstruction quality and uncertainty. We also propose a faster
approach for generating MC samples and this faster approach can be extended to
any other computer vision applications. In our method, we got MC samples in a
single feed-forward, and due to this, it is useful in real-time applications.

Our contributions in this paper are as follows:

  1. 1.

We use the standard approach of uncertainty estimation of DL models using
batch-normalization. Our work proposes a better and faster strategy for
uncertainty estimation and overcame the hurdle of variable image size. Our
procedure generates any number of MC samples in a single shot.

  2. 2.

We have demonstrated a Bayesian uncertainty estimation approach for SISR, and
as of our knowledge, we are the first one to estimate uncertainty in deep
learning based image reconstruction.

  3. 3.

We use Monte Carlo batch-normalization (MCBN) for uncertainty estimation in
super-resolution network.

  4. 4.

We have discussed the advantages of uncertainty in SISR and its applications
from medical image to satellite image super-resolution. We also analyzed the
uncertainty map and its significance.

##  2 Related Work

###  2.1 Single Image Super-Resolution

SISR has extensive literature due to different studies in the last few
decades. Recent advancement of deep learning (DL) methods has achieved
significant improvement in that field [5, 18, 36] and different computer
vision task [42, 41, 43, 15]. SRCNN [5] first explored the convolutional
neural network for establishing a non-linear mapping between interpolated LR
images and its HR counterparts. It has achieved superior performance than
other example-based methods like nearest neighbor [6], sparse representation
[39], neighborhood embedding [34], etc. VDSR [18] proposed a deeper
architecture and showed performance improves with the increase of network
depth and converge faster using residual learning. After that different DL
based approaches [19, 23, 43, 24] have been proposed and achieved state-of-
the-art performance in the standard dataset. In our work, we used VDSR [18]
architecture for uncertainty analysis as it is the first deep architecture for
SISR.

###  2.2 Bayesian Uncertainty

Bayesian models are generally used to model uncertainty, and different
approaches have been developed to adapt NNs to Bayesian reasoning like placing
a prior distribution over parameters. Due to difficult in inferencing [7] of
Bayesian neural networks (BNNs), some approaches [8, 33] have been taken to
approximate BNNs. Bayesian deep learning approaches utilized MC samples
generated via dropout [32] or batch normalization [13] to approximate
posterior distribution. Dropout [32] can be treated as approximate Bayesian
model as multiple predictions through trained model by sampling predictions
using different dropout mask, and In case of batch-normalization [13],
stochastic parameters batch mean and batch variance are used to generate
multiple predictions. [33] shows batch-normalized neural network can be
approximated to the Bayesian model. We use a batch-normalized neural network
for SISR as it is widely used in image reconstruction applications.

##  3 Proposed Method

We propose a Bayesian approach on SISR that produces high-resolution images
along with a confidence map of reconstruction quality. In this regards, we
discuss a short background of Bayesian inference in this section. After that,
we define our network architecture and its modification for Bayesian
approximation. We also present a faster and better approach to overcome the
difficulties of estimating uncertainty in SISR applications. In the end, we
discuss metrics to measure the quality of uncertainty.

###  3.1 Bayesian Inference

We estimate a probabilistic function $F_{W}(I):I_{LR}\rightarrow I_{HR}$ from
a training set $D=\\{\tilde{I}_{LR},\tilde{I}_{HR}\\}$ where
$\tilde{I}_{LR}=\\{\tilde{I}_{LR_{1}},...,\tilde{I}_{LR_{n}}\\}$ are LR image
set and its corresponding HR image set
$\tilde{I}_{HR}=\\{\tilde{I}_{HR_{1}},...,\tilde{I}_{HR_{n}}\\}$. This
function is approximated to generate most likely high-resolution image
${I^{{}^{\prime}}}_{HR}$ from a low-resolution test image
${I^{{}^{\prime}}}_{LR}$. So the probabilistic estimation of HR test image is
described as

$p({I^{{}^{\prime}}}_{HR}|{I^{{}^{\prime}}}_{LR},D)=\int
p({I^{{}^{\prime}}}_{HR}|{I^{{}^{\prime}}}_{LR},W)p(W|D)dW$ (1)

where $W$ is weight parameters of a function $F_{W}(I)$. We use variational
inference to approximate Bayesian modeling. Most common approach is to learn
approximate distribution of weights $q_{\theta}(w)$ by minimizing the
Kullback–Leibler divergence $KL(q_{\theta}(w)\parallel p(W|D))$. This yields
approximate distribution

$q({I^{{}^{\prime}}}_{HR}|{I^{{}^{\prime}}}_{LR},D)=\int
p({I^{{}^{\prime}}}_{HR}|{I^{{}^{\prime}}}_{LR},w)q_{\theta}(w)dw$ (2)

In a batch-normalized neural network for Bayesian uncertainty estimation,
model parameters are $W_{L},\gamma_{L},\beta_{L},\mu_{L},{\sigma^{2}}_{L}$.
Here $\theta=\left\\{W_{L},\gamma_{L},\beta_{L}\right\\}$ is learnable model
weight parameters and $w=\left\\{\mu_{L},{\sigma^{2}}_{L}\right\\}$ are
stochastic parameters which are mean and variance of each layer.
$q_{\theta}(w)$ is a joint distribution of weights and stochastic parameters
$w$. $w_{i}$ is mean and variance of $i$’s sample and it need to be
Independent and identically distributed random variables.

###  3.2 Network Architecture

In this paper, we use very deep super-resolution (VDSR) network [18] as a base
architecture for an experimental purpose to analyze uncertainty. Our method is
a generalized approach, and it can be extended to any other super-resolution
network. We have used batch-normalization to measure uncertainty, but VDSR
paper has not used batch-normalization (BN). So we introduce some changes in
the main architecture. Our VDSR architecture has batch-normalization layer
after each convolutional layer except the last layer, and no bias is used as
batch-normalization normalizes the output. BN blocks are followed by
convolution and ReLU non-linearity.

###  3.3 Bayesian VDSR for Uncertainty Estimation

We use batch-normalization [13] to estimate the uncertainty of super-
resolution network. It is commonly used in deep networks to overcome the
problem known as internal covariate shift. Random batch members are selected
to estimate mini-batch statistics for training. We use this stochasticity to
approximate Bayesian inference. This allows us to make a meaningful estimate
of uncertainty, and it is termed as Monte Carlo Batch Normalization (MCBN)
[33]. Generally running batch mean and batch variance are estimated in each
batch normalization layer during training, but we use batch mean and variance
both during training and testing. We have learnable model parameters which are
optimized using gradient back-propagation during training and stochastic
parameters like batch mean and variances help to generate MC samples from the
posterior distribution of the model. We feed-forward a test image along with
different training batch for multiple times, and due to stochasticity of
batches, it creates various reconstructed HR images. We take mean of those MC
samples to get estimated reconstruction and variance for getting uncertainty
map.

###  3.4 Faster approach

The main drawback of Bayesian uncertainty estimation in the batch-normalized
neural network is that we need to process test image with different random
batches to generate MC samples and computation time increases exponentially
with the increase of the number of samples in a single batch or spatial
dimension of the batch.

Input: training image set $I$

Output: layer-wise batch mean and variance set
${\widehat{w}_{L}^{T}=\\{\mu_{L}^{T},{\sigma^{2}}_{L}^{T}}\\}$ of trained
network. Where $L$ is layer no and $T$ is maximum number of MC samples
required.

Training SR Network:

for total iterations do

forall images of $I$ do

$b$ number of $n\times n$ random crop;

train();

end forall

end for

$\widehat{w}_{L}^{T}$ estimation:

forall images of $I$ do

random crop; $\backslash\backslash$ crop size= $n\times n$ ;

concat patches$\rightarrow$ B $\backslash\backslash$ $T$ decides patch/image;

end forall

shuffle(B);

batchsize$=b$; $\backslash\backslash$ same as training;

forall batches of $B$ do

forward pass;

estimate $\widehat{w}_{L}^{T}$ ;

end forall

Algorithm 1 Layer-wise batch mean and variance estimation for single shot MC
samples generation

Another challenge is that in the case of SISR, test image size varies from
thousands to millions of pixel. We can not make higher spatial batch size
during training as it takes longer computation time. We train our model using
small batch size due to the computational constraint. So we have to break
larger images during testing for batch processing, and it can create a patchy
effect in images. Due to this, we propose a different approach to generate MC
samples in a single batch. After training, we estimate stochastic parameters
$w_{L}$ of each layer using different random training batches as shown in
Algorithm 1. We use the same batch shape during training and stochastic
parameter estimation. These parameters in each batch-normalization layer are
estimated for a batch, and like this, we create several stochastic parameters
set for various batches. These stochastic parameters will be used during
testing for generating MC samples. One stochastic parameter set generates one
MC sample. During testing, we concatenate the same test image based on the
required number of MC samples and in batch-normalization layer we normalize
each image separately using different stochastic parameters as shown in
Algorithm 2. Due to this, it produces various HR image as an MC samples which
come from a posterior distribution learned from the training dataset.

Input: test image $I_{LR}$, number of MC samples $T$, batch mean and variance
of layer $L$ and inference
${\widehat{w}_{L}^{T}=\mu_{L}^{T},{\sigma^{2}}_{L}^{T}}$

Output: image mean prediction $\hat{I}_{HR}$, predictive image uncertainty
$\sigma^{2}$

concat $I_{LR}$ for $T$ times (${I^{T}}_{LR}$);

${I^{T}}_{HR}=F_{w}({I^{T}}_{LR},\widehat{w}_{L}^{T})$;

$\hat{I}_{HR}=mean({I^{T}}_{HR})$;

$\sigma^{2}=var({I^{T}}_{HR})$;

Algorithm 2 MCBN algorithm for SISR uncertainty

###  3.5 Uncertainty Quality Metrices

We evaluate quality of uncertainty using two standard statiscal metrices,
Predictive Log Likelihood (PLL) and Continuous Ranked Probability Score
(CRPS).
Predictive Log Likelihood (PLL):  Predictive Log Likelihood is a widely
accepted metric for measuring the quality of uncertainty [4, 11, 33, 8]. For a
probabilistic model $F_{w}(x\mid D)$, PLL of an LR image $i_{LR}$ and HR image
$i_{HR}$ is defined as:

$PLL(F_{w}(x\mid D),(i_{LR},i_{HR}))=\log p(i_{HR}\mid F_{w}(x\mid D))$ (3)

$p(i_{HR}\mid F_{w}(x\mid D))$ is a predictive probability distribution
function of $i_{HR}$ for an input $i_{LR}$. There is no bound of PLL and it is
maximized for perfect predition of HR image without any variance. Main
property of this metric is that it does not make any assumptions about
distribution but it is criticized for have effect of outlier on score [28].
Continuous Ranked Probability Score (CRPS): Continuous Ranked Probability
Score [10] is generally used to estimate respective accuracy of two
probabilistic models. It generalizes mean absolute error for probabilistic
estimation. CRPS is defined as

$CRPS(F_{w}(i_{LR}\mid
D),i_{HR})=\int_{-\infty}^{\infty}(F(y)-\textbf{1}(y-i_{HR}))^{2}dy$ (4)

where $F(y)$ is predictive cumulative distribution function, and 1 is the
Heaviside step function. The value of $\textbf{1}(y-i_{HR})$ is $1$ if
$y-i_{HR}\geq 0$ and $0$ otherwise. There is no upper bound of crps. Perfect
prediction with no variance receives a CRPS of $0$.

##  4 Experimental Results & Discussions

In this section, we discuss the datasets used in experimental purpose and
training methodology. We also address the effect of the number of MC samples
on performance and compare our faster MC sample generation approach with
standard BN uncertainty estimation. In the end, we present our understanding
of model uncertainty for SISR applications.

###  4.1 Datasets

We use DIV2K [2, 35], a high-resolution, high-quality dataset for training
purpose. Total $800$ training images optimize trainable parameters of the SR
network, and $100$ validation images are to select the best parameter settings
for testing. We analyze the network performance using five standard benchmark
testing dataset namely Set5 [3], Set14 [40], BSD100 [25], Urban100 [12], and
Manga109 [26]. We have also experimented on satellite images downloaded from
[1] and histopathological images from MoNuSeg challenge dataset [21].

(a) SSIM vs MC Samples

(b) PSNR in dB vs MC Samples

(c) PLL vs MC Samples

(d) CRPS vs MC Samples

Figure 2: The plot shows estimated values on ’Set5’ dataset for a scale factor
$\times 8$ and MC samples vary from minimum $3$ samples to $600$ samples.

###  4.2 Training Details

We randomly extract $16$ patches of size $64\times 64$ from each HR and
bicubic interpolated LR image during training for a batch update. We augment
the patches by horizontal flip, vertical flip, and $90$-degree rotation and
randomly choose each augmentation with $50\%$ probability. We normalize each
input into $[0,1]$ before feeding to the network. We train each model for
$1,000$ iterations which is equivalent to $0.8$ million batch update. To
ensure better convergence, we use trained model of scale factor $\times 2$ as
initialization for other scale factors. We use Xavier initialization [9] for
initializing model of scaling factor $2$. We train our model with PyTorch
framework and update weights with Adam optimizer [20]. The learning rate is
initialized to $10^{-4}$ and reduced to half after every $400$ iterations. We
use mean-squared error to optimize model parameters. We extract $64\times 64$
patches from high variance regions of validation images to choose the best
model for testing. During testing, we clip the output between $[0,1]$ and map
into an 8-bit unsigned integer format. For a fair comparison, we remove
boundary pixels of each test image based on the scaling factor for image
quality evaluation as described in vdsr paper [18].

| Monte Carlo
---
Samples
| Bayesian VDSR
---
| Faster
---
Implementation
5 | 14.28 | 1.0
10 | 33.48 | 1.97
15 | 52.67 | 2.96
Table 1: All the values correspond to ’face’ image of size $276\times 276$
from ’Set14’ dataset.

Scaling Factor | $\times 2$ |  | $\times 4$ |  | $\times 8$
---|---|---|---|---|---
MC Samples | 5 | 15 | 25 |  | 5 | 15 | 25 |  | 5 | 15 | 25
Set5 | SSIM | 0.957530 | 0.957559 | 0.957562 |  | 0.884546 | 0.884700 | 0.884685 |  | 0.732343 | 0.732507 | 0.732317
PSNR | 37.5404 | 37.5440 | 37.5472 |  | 31.4436 | 31.4456 | 31.4443 |  | 26.0340 | 26.0409 | 26.0391
PLL | -11.175 | -11.187 | -11.184 |  | -41.361 | -41.392 | -41.434 |  | -157.733 | -157.559 | -157.656
CRPS | 0.008191 | 0.008162 | 0.008157 |  | 0.015771 | 0.015711 | 0.015693 |  | 0.032864 | 0.032843 | 0.032823
Set14 | SSIM | 0.912049 | 0.912211 | 0.912232 |  | 0.768863 | 0.768996 | 0.769027 |  | 0.615298 | 0.615382 | 0.615404
PSNR | 33.0858 | 33.1049 | 33.1096 |  | 28.0506 | 28.0637 | 28.0640 |  | 24.2733 | 24.2783 | 24.2816
PLL | -91.007 | -90.892 | -90.767 |  | -252.040 | -250.839 | -250.943 |  | -568.926 | -568.596 | -568.283
CRPS | 0.014960 | 0.014879 | 0.014857 |  | 0.026334 | 0.026215 | 0.026199 |  | 0.040444 | 0.040339 | 0.040343
BSD100 | SSIM | 0.894965 | 0.895038 | 0.895048 |  | 0.725807 | 0.725903 | 0.725940 |  | 0.580028 | 0.580055 | 0.580051
PSNR | 31.9119 | 31.9164 | 31.9175 |  | 27.2973 | 27.2997 | 27.3012 |  | 24.4825 | 24.4842 | 24.4845
PLL | -74.233 | -74.226 | -74.217 |  | -195.734 | -195.736 | -195.730 |  | -346.074 | -345.979 | -345.981
CRPS | 0.016905 | 0.016859 | 0.016851 |  | 0.028345 | 0.028274 | 0.028260 |  | 0.039483 | 0.039417 | 0.039400
Urban100 | SSIM | 0.916389 | 0.916472 | 0.916480 |  | 0.757138 | 0.757218 | 0.757180 |  | 0.576144 | 0.576194 | 0.576157
PSNR | 31.1339 | 31.1442 | 31.1457 |  | 25.3401 | 25.3423 | 25.3417 |  | 21.8184 | 21.8180 | 21.8176
PLL | -472.235 | -469.221 | -468.409 |  | -1553.83 | -1547.44 | -1545.33 |  | -3148.43 | -3140.58 | -3138.79
CRPS | 0.017461 | 0.017389 | 0.017373 |  | 0.033442 | 0.033353 | 0.033327 |  | 0.052008 | 0.051922 | 0.051906
Manga109 | SSIM | 0.973192 | 0.973270 | 0.973291 |  | 0.887527 | 0.887976 | 0.888069 |  | 0.726837 | 0.727066 | 0.727109
PSNR | 37.5643 | 37.5966 | 37.6036 |  | 29.1394 | 29.1608 | 29.1634 |  | 23.3169 | 23.3217 | 23.3223
PLL | -127.31 | -127.00 | -126.71 |  | -796.85 | -791.92 | -790.55 |  | -2640.54 | -2634.91 | -2632.31
CRPS | 0.007538 | 0.007450 | 0.007427 |  | 0.017915 | 0.017773 | 0.017740 |  | 0.036155 | 0.036030 | 0.035989

Table 2: Average SSIM, PSNR, PLL and CRPS for different scale factor and MC
samples on $5$ standard dataset.

Dataset | Scale | Bicubic | A+ | SRCNN | VDSR |  | VDSR (Our
---
Implementation)
| BN VDSR
---
(Ours)
| Bayesian VDSR
---
(Ours)
Set5 | x2 | 33.65 / 0.930 | 36.54 / 0.954 | 36.65 / 0.954 | 37.53 / 0.958 | 37.49 / 0.957 | 37.54 / 0.957 | 37.55 / 0.958
x4 | 28.42 / 0.810 | 30.30 / 0.859 | 30.49 / 0.862 | 31.35 / 0.882 | 31.32 / 0.882 | 31.45 / 0.884 | 31.45 / 0.885
x8 | 24.39 / 0.657 | 25.52 / 0.692 | 25.33 / 0.689 | 25.72 / 0.711 | 26.00 / 0.729 |  26.07 / 0.732 | 26.04 / 0.733
Set14 | x2 | 30.34 / 0.870 | 32.40 / 0.906 | 32.29 / 0.903 | 32.97 / 0.913 | 33.03 / 0.912 | 33.08 / 0.912 | 33.11 / 0.912
x4 | 26.10 / 0.704 | 27.43 / 0.752 | 27.61 / 0.754 | 28.03 / 0.770 | 28.02 / 0.767 |  28.07 / 0.768 | 28.06 / 0.769
x8 | 23.19 / 0.568 | 23.98 / 0.597 | 23.85 / 0.593 | 24.21 / 0.609 | 24.26 / 0.613 | 24.32 / 0.615 | 24.28 / 0.615
BSD100 | x2 | 29.56 / 0.844 | 31.22 / 0.887 | 31.36 / 0.888 | 31.90 / 0.896 | 31.88 / 0.895 | 31.91 / 0.895 | 31.92 / 0.895
x4 | 25.96 / 0.669 | 26.82 / 0.710 | 26.91 / 0.712 | 27.29 / 0.726 | 27.27 / 0.724 | 27.30 / 0.725 | 27.30 / 0.726
x8 | 23.67 / 0.547 | 24.20 / 0.568 | 24.13 / 0.565 | 24.37 / 0.576 | 24.46 / 0.579 |  24.49 / 0.579 | 24.48 / 0.580
Urban100 | x2 | 26.88 / 0.841 | 29.23 / 0.894 | 29.52 / 0.895 | 30.77 / 0.914 | 31.05 / 0.916 | 31.15 / 0.917 |  31.15 / 0.916
x4 | 23.15 / 0.659 | 24.34 / 0.720 | 24.53 / 0.724 | 25.18 / 0.753 | 25.27 / 0.754 |  25.35 / 0.756 | 25.34 / 0.757
x8 | 20.74 / 0.515 | 21.37 / 0.545 | 21.29 / 0.543 | 21.54 / 0.560 | 21.77 / 0.574 | 21.83 / 0.576 | 21.82 / 0.576
Manga109 | x2 | 30.84 / 0.935 | 35.33 / 0.967 | 35.72 / 0.968 | 37.16 / 0.974 | 37.36 / 0.973 | 37.46 / 0.973 | 37.60 / 0.973
x4 | 24.92 / 0.789 | 27.02 / 0.850 | 27.66 / 0.858 | 28.82 / 0.886 | 28.98 / 0.885 | 29.20 / 0.888 | 29.16 / 0.888
x8 | 21.47 / 0.649 | 22.39 / 0.680 | 22.37 / 0.682 | 22.83 / 0.707 | 23.23 / 0.723 | 23.35 / 0.728 | 23.32 / 0.727

Table 3: Comparison table of Bayesian VDSR with other benchmarked results.
Average PSNR / SSIM values for scaling factor $\times 2$, $\times 4$, $\times
8$

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(a) HR Image

(b) Scale factor $\times 2$

(c) Scale factor $\times 3$

(d) Scale factor $\times 4$

(e) Scale factor $\times 8$

Figure 3: Figure shows reconstructed image and uncertainty map with different
scaling factor. First two image is from ’Manga109’ dataset and last one is
from ’Urban100’ dataset.

###  4.3 Performance Analysis

####  4.3.1 Number of MC Samples

We get a better estimate of uncertainty and reconstruction with the increase
of the number of MC samples, but it also increases the inference time. So
proper choice of the number of MC samples required for the task from this
trade-off. The minimum number of MC samples should be capable enough to give a
better reconstruction than batch-normalization without stochastic mean-
variance and should also provide a stable uncertainty map. In the figure 2, we
observe the changes of reconstruction and uncertainty quality against changes
in the number of MC samples for ’Set5’ dataset. The plot shows that SSIM and
PSNR index increases with the increase of MC samples and later it settles to
some stable values. In the case of PLL and CRPS, it converges to some stable
value after initial unstable conditions. In our experiments, We use $5,15$,
and $25$ MC samples for testing.

####  4.3.2 Fast MC Sample Generation

We benchmark our faster approach with standard batch-normalized (BN)
uncertainty estimation. The time required for generating MC samples in
standard BN uncertainty [33] mainly depends on the size of the image in the
dataset and the number of MC samples. We overcome these two difficulties. Our
approach is much faster than conventional as shown in Table 1. We consider $5$
MC sample generation for an image using our method as a baseline and other
values in the table exhibits how many times more GPU time required for
inference. Our approach takes $14.28$ times lesser execution time to generate
$5$ samples for an image of size $276\times 276$.

####  4.3.3 Image Quality Analysis

We used structural similarity (SSIM) [37] and peak signal-to-noise ratio
(PSNR) metrics to measure the reconstruction quality. Table 2 shows the
performance of Bayesian VDSR for different Monte Carlo samples. We observe an
improvement of image quality with the increase of MC samples and it saturates
gradually. In Table 3, we also compared our bayesian VDSR and batch-normalized
(BN) VDSR with standard deep learning based approaches like SRCNN [5], VDSR
[18]. Our training dataset, training procedure and no bias approach are
different from VDSR paper. So we put our VDSR implementation in the table for
fair comparison. Bayesian VDSR gives a minor improvement over BN-VDSR. But
along with this, uncertainty map comes free, and it contributes a significant
boost in deep learning based super-resolution task.

###  4.4 Understanding Model Uncertainty

The Table 2 shows two standard uncertainty quality metrics PLL and CRPS for
measuring uncertainty quality using different MC samples. We observe the value
of PLL and CRPS are getting better (more stable) with the increase of MC
samples due to the availability of larger samples it can estimate better mean
and variance. Model uncertainty increases with the increase of scaling factor
due to the higher shift of mean from actual value and higher variance in MC
samples.

In Figure 3 shows $3$ different images from a standard testing dataset and its
reconstruction from LR image with the different scaling factor. Along with
this it also shows uncertainty in reconstruction. In the first image, if we
look on the English letters, we can see higher uncertainty in border of each
letter for scaling factor $\times 2$, and as scaling factor increases it
becomes difficult to reconstruct the characters perfectly for a DL models and
uncertainty increases as we can see for scaling factor $\times 8$, model shows
uncertainty all over the characters. In the second image set of figure 3, the
reconstructed image with scaling factor $\times 2$ does not show any
uncertainty. But we observe there is uncertainty in the boundary of Japanese
character and it increases gradually with the increase of scaling factor.
Uncertainty is maximum for scaling factor $\times 8$ as we can also see
visually that sharp boundary in that character has been deformed in the
reconstructed image. But in case of dotted texture in that image shows higher
uncertainty for scaling factor $\times 3$ and there is no uncertainty for
scaling factor $\times 2$ and $\times 8$. It is due to perfect reconstruction
for scaling factor $\times 2$ but for scaling factor $\times 8$, these texture
has completely been abolished in original low LR image due to high
downsampling, and the dotted region becomes continuous, and model upsampled
that continuous texture. The last image set of Figure 3 shows the uncertainty
in the edges of windows, and it increases with the increase of downsampling
factor. In our understanding, we observe that if some texture is present in LR
image and it is not reconstructed properly in HR image, those regions show
higher uncertainty in the reconstruction. Mainly ambiguous regions, object
boundaries, sharp regions or any deformed reconstruction generally receive
higher uncertainty. This is very much helpful to further process those images
for other computer vision task. Features coming from those uncertain regions
in other computer vision task can be assigned lower importance, and it may
improve performance.

For qualitative evaluation, we compare the average uncertainty with the
quality of reconstruction. In Figure 4, we use PSNR and perceptual loss [14]
to measure the quality of reconstruction. We use features of $relu2\\_2$,
$relu3\\_3$, and $relu4\\_3$ layer from popular VGG16 [31] model to calculate
perceptual loss between HR image and reconstructed image. Perceptual loss
increases with the increase of deformation in the reconstructed image and PSNR
decreases with the increase of pixel-wise loss. We observe a strong
relationship between uncertainty and image quality. PSNR decreases and
perceptual loss increases with the rise in uncertainty.

(a) PSNR vs Mean Uncertainty

(b) VGG Feature Loss ($relu2\\_2$) vs Mean Uncertainty

(c) VGG Feature Loss ($relu3\\_3$) vs Mean Uncertainty

(d) VGG Feature Loss ($relu4\\_3$) vs Mean Uncertainty

Figure 4: All the points in the plot are reconstructed images from BSD100
dataset for a scale factor $\times 2$ using $50$ MC samples. All the axis
values are normalized between $\left[0,1\right]$.

##  5 Conclusion

In this article, we introduced a Bayesian approach to estimate uncertainty in
batch normalized super-resolution network. Stochastic batch-normalization used
during test time to generate multiple Monte Carlo samples and those samples
are used to estimate uncertainty. We also propose a faster approach to produce
Monte Carlo samples and measure uncertainty quality using standard statistical
metrics. Our method is a generalized approach, and it can be applied to any
image reconstruction techniques. We show Bayesian uncertainty provides the
reliable measure of model uncertainty in SISR. We believe that uncertainty in
image super-resolution will improve the trustability of reconstructed output
for deployment in the high-risk task.

## References

  * [1] Open Data Program.  https://www.digitalglobe.com/ecosystem/open-data/.  Accessed: 2019-03-01.
  * [2] E. Agustsson and R. Timofte.  Ntire 2017 challenge on single image super-resolution: Dataset and study.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 126–135, 2017.
  * [3] M. Bevilacqua, A. Roumy, C. Guillemot, and M. L. Alberi-Morel.  Low-complexity single-image super-resolution based on nonnegative neighbor embedding.  2012\.
  * [4] T. Bui, D. Hernández-Lobato, J. Hernandez-Lobato, Y. Li, and R. Turner.  Deep gaussian processes for regression using approximate expectation propagation.  In International Conference on Machine Learning, pages 1472–1481, 2016.
  * [5] C. Dong, C. C. Loy, K. He, and X. Tang.  Image super-resolution using deep convolutional networks.  IEEE transactions on pattern analysis and machine intelligence, 38(2):295–307, 2016.
  * [6] W. T. Freeman, T. R. Jones, and E. C. Pasztor.  Example-based super-resolution.  IEEE Computer graphics and Applications, (2):56–65, 2002.
  * [7] Y. Gal.  Uncertainty in deep learning.  PhD thesis, PhD thesis, University of Cambridge, 2016.
  * [8] Y. Gal and Z. Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In International Conference on Machine Learning, pages 1050–1059, 2016.
  * [9] X. Glorot and Y. Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256, 2010.
  * [10] T. Gneiting and A. E. Raftery.  Strictly proper scoring rules, prediction, and estimation.  Journal of the American Statistical Association, 102(477):359–378, 2007.
  * [11] J. M. Hernández-Lobato and R. Adams.  Probabilistic backpropagation for scalable learning of bayesian neural networks.  In International Conference on Machine Learning, pages 1861–1869, 2015.
  * [12] J.-B. Huang, A. Singh, and N. Ahuja.  Single image super-resolution from transformed self-exemplars.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5197–5206, 2015.
  * [13] S. Ioffe and C. Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  In International Conference on Machine Learning, pages 448–456, 2015.
  * [14] J. Johnson, A. Alahi, and L. Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In European conference on computer vision, pages 694–711. Springer, 2016.
  * [15] A. Kar, S. Phani Krishna Karri, N. Ghosh, R. Sethuraman, and D. Sheet.  Fully convolutional model for variable bit length and lossy high density compression of mammograms.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 2591–2594, 2018.
  * [16] A. Kendall, V. Badrinarayanan, and R. Cipolla.  Bayesian segnet: Model uncertainty in deep convolutional encoder-decoder architectures for scene understanding.  Proceedings of the British Machine Vision Conference (BMVC), 2017\.
  * [17] A. Kendall and R. Cipolla.  Modelling uncertainty in deep learning for camera relocalization.  In 2016 IEEE international conference on Robotics and Automation (ICRA), pages 4762–4769. IEEE, 2016.
  * [18] J. Kim, J. Kwon Lee, and K. Mu Lee.  Accurate image super-resolution using very deep convolutional networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1646–1654, 2016.
  * [19] J. Kim, J. Kwon Lee, and K. Mu Lee.  Deeply-recursive convolutional network for image super-resolution.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1637–1645, 2016.
  * [20] D. P. Kingma and J. Ba.  Adam: A method for stochastic optimization.  ICLR, 2014.
  * [21] N. Kumar, R. Verma, S. Sharma, S. Bhargava, A. Vahadane, and A. Sethi.  A dataset and a technique for generalized nuclear segmentation for computational pathology.  IEEE transactions on medical imaging, 36(7):1550–1560, 2017.
  * [22] A. Kurakin, I. Goodfellow, and S. Bengio.  Adversarial examples in the physical world.  ICLR Workshop track, 2017.
  * [23] W.-S. Lai, J.-B. Huang, N. Ahuja, and M.-H. Yang.  Fast and accurate image super-resolution with deep laplacian pyramid networks.  IEEE transactions on pattern analysis and machine intelligence, 2018\.
  * [24] C. Ledig, L. Theis, F. Huszár, J. Caballero, A. Cunningham, A. Acosta, A. Aitken, A. Tejani, J. Totz, Z. Wang, et al.  Photo-realistic single image super-resolution using a generative adversarial network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4681–4690, 2017.
  * [25] D. Martin, C. Fowlkes, D. Tal, and J. Malik.  A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics.  In Proceedings Eighth IEEE International Conference on Computer Vision. ICCV 2001, volume 2, pages 416–423. IEEE, 2001.
  * [26] Y. Matsui, K. Ito, Y. Aramaki, A. Fujimoto, T. Ogawa, T. Yamasaki, and K. Aizawa.  Sketch-based manga retrieval using manga109 dataset.  Multimedia Tools and Applications, 76(20):21811–21838, 2017.
  * [27] A. G. Roy, S. Conjeti, N. Navab, and C. Wachinger.  Inherent brain segmentation quality control from fully convnet monte carlo sampling.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 664–672. Springer, 2018.
  * [28] R. Selten.  Axiomatic characterization of the quadratic scoring rule.  Experimental Economics, 1(1):43–61, 1998.
  * [29] A. Sharma, P. Kaur, A. Nigam, and A. Bhavsar.  Learning to decode 7t-like mr image reconstruction from 3t mr images.  In Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pages 245–253. Springer, 2018.
  * [30] W. Shi, J. Caballero, C. Ledig, X. Zhuang, W. Bai, K. Bhatia, A. M. S. M. de Marvao, T. Dawes, D. O’Regan, and D. Rueckert.  Cardiac image super-resolution with global correspondence using multi-atlas patchmatch.  In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 9–16. Springer, 2013.
  * [31] K. Simonyan and A. Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [32] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  The Journal of Machine Learning Research, 15(1):1929–1958, 2014\.
  * [33] M. Teye, H. Azizpour, and K. Smith.  Bayesian uncertainty estimation for batch normalized deep networks.  In International Conference on Machine Learning, pages 4914–4923, 2018.
  * [34] R. Timofte, V. De Smet, and L. Van Gool.  A+: Adjusted anchored neighborhood regression for fast super-resolution.  In Asian conference on computer vision, pages 111–126. Springer, 2014.
  * [35] R. Timofte, S. Gu, J. Wu, L. Van Gool, L. Zhang, M.-H. Yang, M. Haris, et al.  Ntire 2018 challenge on single image super-resolution: Methods and results.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2018.
  * [36] T. Tong, G. Li, X. Liu, and Q. Gao.  Image super-resolution using dense skip connections.  In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
  * [37] Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli, et al.  Image quality assessment: from error visibility to structural similarity.  IEEE transactions on image processing, 13(4):600–612, 2004.
  * [38] Z. Wang, J. Chen, and S. C. Hoi.  Deep learning for image super-resolution: A survey.  arXiv preprint arXiv:1902.06068, 2019.
  * [39] J. Yang, J. Wright, T. S. Huang, and Y. Ma.  Image super-resolution via sparse representation.  IEEE transactions on image processing, 19(11):2861–2873, 2010.
  * [40] R. Zeyde, M. Elad, and M. Protter.  On single image scale-up using sparse-representations.  In International conference on curves and surfaces, pages 711–730. Springer, 2010.
  * [41] H. Zhang and V. M. Patel.  Densely connected pyramid dehazing network.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [42] K. Zhang, W. Zuo, Y. Chen, D. Meng, and L. Zhang.  Beyond a gaussian denoiser: Residual learning of deep cnn for image denoising.  IEEE Transactions on Image Processing, 26(7):3142–3155, 2017.
  * [43] Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu.  Residual dense network for image super-resolution.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
  * [44] W. W. Zou and P. C. Yuen.  Very low resolution face recognition problem.  IEEE Transactions on image processing, 21(1):327–340, 2012.


# scikit-image: Image processing in Python111Distributed under Creative
Commons CC-BY 4.0, DOI 10.7717/peerj.453

Stéfan van der Walt  Corresponding author<EMAIL_ADDRESS>Stellenbosch
University, Stellenbosch, South Africa  Johannes L. Schönberger  Department of
Computer Science, University of North Carolina at Chapel Hill, Chapel Hill, NC
27599, USA  Juan Nunez-Iglesias  Victorian Life Sciences Computation
Initiative, Carlton, VIC, 3010, Australia  François Boulogne  Department of
Mechanical and Aerospace Engineering, Princeton University, Princeton, New
Jersey 08544, USA  Joshua D. Warner  Department of Biomedical Engineering,
Mayo Clinic, Rochester, Minnesota 55905, USA  Neil Yager  AICBT Ltd, Oxford,
UK  Emmanuelle Gouillart  Joint Unit CNRS / Saint-Gobain, Cavaillon, France
Tony Yu  Enthought Inc., Austin, TX, USA  the scikit-image contributors
https://github.com/scikit-image/scikit-image/graphs/contributors

###### Abstract

scikit-image is an image processing library that implements algorithms and
utilities for use in research, education and industry applications. It is
released under the liberal “Modified BSD” open source license, provides a
well-documented API in the Python programming language, and is developed by an
active, international team of collaborators. In this paper we highlight the
advantages of open source to achieve the goals of the scikit-image library,
and we showcase several real-world image processing applications that use
scikit-image.

###### keywords:

image processing, reproducible research, education, visualization

## Introduction

In our data-rich world, images represent a significant subset of all
measurements made. Examples include DNA microarrays, microscopy slides,
astronomical observations, satellite maps, robotic vision capture, synthetic
aperture radar images, and higher-dimensional images such as 3-D magnetic
resonance or computed tomography imaging. Exploring these rich data sources
requires sophisticated software tools that should be easy to use, free of
charge and restrictions, and able to address all the challenges posed by such
a diverse field of analysis.

This paper describes scikit-image, a collection of image processing algorithms
implemented in the Python programming language by an active community of
volunteers and available under the liberal BSD Open Source license. The rising
popularity of Python as a scientific programming language, together with the
increasing availability of a large eco-system of complementary tools, make it
an ideal environment in which to produce an image processing toolkit.

The project aims are:

  1. 1.

To provide high quality, well-documented and easy-to-use implementations of
common image processing algorithms.

Such algorithms are essential building blocks in many areas of scientific
research, algorithmic comparisons and data exploration. In the context of
reproducible science, it is important to be able to inspect any source code
used for algorithmic flaws or mistakes. Additionally, scientific research
often requires custom modification of standard algorithms, further emphasizing
the importance of open source.

  2. 2.

To facilitate education in image processing.

The library allows students in image processing to learn algorithms in a
hands-on fashion by adjusting parameters and modifying code. In addition, a
novice module is provided, not only for teaching programming in the “turtle
graphics” paradigm, but also to familiarize users with image concepts such as
color and dimensionality. Furthermore, the project takes part in the yearly
Google Summer of Code program 222https://developers.google.com/open-
source/soc, Accessed: 2014-03-30, where students learn about image processing
and software engineering through contributing to the project.

  3. 3.

To address industry challenges.

High quality reference implementations of trusted algorithms provide industry
with a reliable way of attacking problems, without having to expend
significant energy in re-implementing algorithms already available in
commercial packages. Companies may use the library entirely free of charge,
and have the option of contributing changes back, should they so wish.

## Getting started

One of the main goals of scikit-image is to make it easy for any user to get
started quickly–especially users already familiar with Python’s scientific
tools. To that end, the basic image is just a standard NumPy array, which
exposes pixel data directly to the user. A new user can simply the load an
image from disk (or use one of scikit-image’s sample images), process that
image with one or more image filters, and quickly display the results:

⬇

image = data.coins() # or any NumPy array!

edges = filter.sobel(image)

io.imshow(edges)

Figure 1: Illustration of several functions available in scikit-image:
adaptive threshold, local maxima, edge detection and labels. The use of NumPy
arrays as our data container also enables the use of NumPy’s built-in
histogram function.

The above demonstration loads data.coins, an example image shipped with
scikit-image. For a more complete example, we import NumPy for array
manipulation and matplotlib for plotting (van der Walt et al.,, 2011; Hunter,,
2007). At each step, we add the picture or the plot to a matplotlib figure
shown in Figure 1.

⬇

import matplotlib.pyplot as plt

# Load a small section of the image.

image = data.coins()[0:95, 70:370]

fig, axes = plt.subplots(ncols=2, nrows=3,

figsize=(8, 4))

ax0, ax1, ax2, ax3, ax4, ax5 = axes.flat

ax0.imshow(image, cmap=plt.cm.gray)

ax0.set_title(’Original’, fontsize=24)

ax0.axis(’off’)

Since the image is represented by a NumPy array, we can easily perform
operations such as building an histogram of the intensity values.

⬇

values, bins = np.histogram(image,

bins=np.arange(256))

ax1.plot(bins[:-1], values, lw=2, c=’k’)

ax1.set_xlim(xmax=256)

ax1.set_yticks([0, 400])

ax1.set_aspect(.2)

ax1.set_title(’Histogram’, fontsize=24)

To divide the foreground and background, we threshold the image to produce a
binary image. Several threshold algorithms are available. Here, we employ
filter.threshold_adaptive where the threshold value is the weighted mean for
the local neighborhood of a pixel.

⬇

from skimage.filter import threshold_adaptive

bw = threshold_adaptive(image, 95, offset=-15)

ax2.imshow(bw, cmap=plt.cm.gray)

ax2.set_title(’Adaptive threshold’, fontsize=24)

ax2.axis(’off’)

We can easily detect interesting features, such as local maxima and edges. The
function feature.peak_local_max can be used to return the coordinates of local
maxima in an image.

⬇

from skimage.feature import peak_local_max

coordinates = peak_local_max(image, min_distance=20)

ax3.imshow(image, cmap=plt.cm.gray)

ax3.autoscale(False)

ax3.plot(coordinates[:, 1],

coordinates[:, 0], c=’r.’)

ax3.set_title(’Peak local maxima’, fontsize=24)

ax3.axis(’off’)

Next, a Canny filter (filter.canny) (Canny,, 1986) detects the edge of each
coin.

⬇

from skimage import filter

edges = filter.canny(image, sigma=3,

low_threshold=10,

high_threshold=80)

ax4.imshow(edges, cmap=plt.cm.gray)

ax4.set_title(’Edges’, fontsize=24)

ax4.axis(’off’)

Then, we attribute to each coin a label (morphology.label) that can be used to
extract a sub-picture. Finally, physical information such as the position,
area, eccentricity, perimeter, and moments can be extracted using
measure.regionprops.

⬇

from skimage.measure import regionprops

import matplotlib.patches as mpatches

from skimage.morphology import label

label_image = label(edges)

ax5.imshow(image, cmap=plt.cm.gray)

ax5.set_title(’Labeled items’, fontsize=24)

ax5.axis(’off’)

for region in regionprops(label_image):

# Draw rectangle around segmented coins.

minr, minc, maxr, maxc = region.bbox

rect = mpatches.Rectangle((minc, minr),

maxc - minc,

maxr - minr,

fill=False,

edgecolor=’red’,

linewidth=2)

ax5.add_patch(rect)

plt.tight_layout()

plt.show()

scikit-image thus makes it possible to perform sophisticated image processing
tasks with only a few function calls.

## Library overview

The scikit-image project started in August of 2009 and has received
contributions from more than 100 individuals333https://www.ohloh.net/p/scikit-
image. The package can be installed on all major platforms (e.g. BSD,
GNU/Linux, OS X, Windows) from, amongst other sources, the Python Package
Index (PyPI)444http://pypi.python.org, Continuum Analytics
Anaconda555https://store.continuum.io/cshop/anaconda, Enthought
Canopy666https://www.enthought.com/products/canopy,
Python(x,y)777https://code.google.com/p/pythonxy, NeuroDebian (Halchenko and
Hanke,, 2012) and GNU/Linux distributions such as
Ubuntu888http://packages.ubuntu.com. In March 2014 alone, the package was
downloaded more than 5000 times from
PyPI999http://pypi.python.org/pypi/scikit-image, Accessed 2014-03-30.

As of version 0.10, the package contains the following sub-modules:

  * •

color: Color space conversion.

  * •

data: Test images and example data.

  * •

draw: Drawing primitives (lines, text, etc.) that operate on NumPy arrays.

  * •

exposure: Image intensity adjustment, e.g., histogram equalization, etc.

  * •

feature: Feature detection and extraction, e.g., texture analysis, corners,
etc.

  * •

filter: Sharpening, edge finding, rank filters, thresholding, etc.

  * •

graph: Graph-theoretic operations, e.g., shortest paths.

  * •

io: Wraps various libraries for reading, saving, and displaying images and
video, such as Pillow101010http://pillow.readthedocs.org/en/latest/, Accessed
2015-05-30 and FreeImage111111http://freeimage.sourceforge.net/, Accessed
2015-05-15.

  * •

measure: Measurement of image properties, e.g., similarity and contours.

  * •

morphology: Morphological operations, e.g., opening or skeletonization.

  * •

novice: Simplified interface for teaching purposes.

  * •

restoration: Restoration algorithms, e.g., deconvolution algorithms,
denoising, etc.

  * •

segmentation: Partitioning an image into multiple regions.

  * •

transform: Geometric and other transforms, e.g., rotation or the Radon
transform.

  * •

viewer: A simple graphical user interface for visualizing results and
exploring parameters.

For further details on each module, we refer readers to the API documentation
online121212http://scikit-image.org/docs/dev/api/api.html.

## Data format and pipelining

scikit-image represents images as NumPy arrays (van der Walt et al.,, 2011),
the de facto standard for storage of multi-dimensional data in scientific
Python. Each array has a dimensionality, such as 2 for a 2-D grayscale image,
3 for a 2-D multi-channel image, or 4 for a 3-D multi-channel image; a shape,
such as $(M,N,3)$ for an RGB color image with $M$ vertical and $N$ horizontal
pixels; and a numeric data type, such as float for continuous-valued pixels
and uint8 for 8-bit pixels. Our use of NumPy arrays as the fundamental data
structure maximizes compatibility with the rest of the scientific Python
ecosystem. Data can be passed as-is to other tools such as NumPy, SciPy,
matplotlib, scikit-learn (Pedregosa et al.,, 2011), Mahotas (Coelho,, 2013),
OpenCV, and more.

Images of differing data-types can complicate the construction of pipelines.
scikit-image follows an "Anything In, Anything Out" approach, whereby all
functions are expected to allow input of an arbitrary data-type but, for
efficiency, also get to choose their own output format. Data-type ranges are
clearly defined. Floating point images are expected to have values between 0
and 1 (unsigned images) or -1 and 1 (signed images), while 8-bit images are
expected to have values in {0, 1, 2, …, 255}. We provide utility functions,
such as img_as_float, to easily convert between data-types.

## Development practices

The purpose of scikit-image is to provide a high-quality library of powerful,
diverse image processing tools free of charge and restrictions. These
principles are the foundation for the development practices in the scikit-
image community.

The library is licensed under the _Modified BSD license_ , which allows
unrestricted redistribution for any purpose as long as copyright notices and
disclaimers of warranty are maintained (Regents of the University of
California,, 1999). It is compatible with GPL licenses, so users of scikit-
image can choose to make their code available under the GPL. However, unlike
the GPL, it does not require users to open-source derivative work (BSD is not
a so-called copyleft license). Thus, scikit-image can also be used in closed-
source, commercial environments.

The development team of scikit-image is an open community that collaborates on
the _GitHub_ platform for issue tracking, code review, and release
management131313https://github.com/scikit-image. _Google Groups_ is used as a
public discussion forum for user support, community development, and
announcements141414https://groups.google.com/group/scikit-image.

scikit-image complies with the PEP8 coding style standard (van Rossum et al.,,
2001) and the NumPy documentation format (Gommers,, 2010) in order to provide
a consistent, familiar user experience across the library similar to other
scientific Python packages. As mentioned earlier, the data representation used
is _n_ -dimensional NumPy arrays, which ensures broad interoperability within
the scientific Python ecosystem. The majority of the scikit-image API is
intentionally designed as a functional interface which allows one to simply
apply one function to the output of another. This modular approach also lowers
the barrier of entry for new contributors, since one only needs to master a
small part of the entire library in order to make an addition.

We ensure high code quality by a thorough review process using the pull
request interface on GitHub151515https://help.github.com/articles/using-pull-
requests, Accessed 2014-05-15.. This enables the core developers and other
interested parties to comment on specific lines of proposed code changes, and
for the proponents of the changes to update their submission accordingly. Once
all the changes have been approved, they can be merged automatically. This
process applies not just to outside contributions, but also to the core
developers.

The source code is mainly written in Python, although certain performance
critical sections are implemented in Cython, an optimising static compiler for
Python (Behnel et al.,, 2011). scikit-image aims to achieve full unit test
coverage, which is above 87% as of release 0.10 and continues to rise. A
continuous integration system161616https://travis-ci.org,
https://coveralls.io, Accessed 2014-03-30 automatically checks each commit for
unit test coverage and failures on both Python 2 and Python 3. Additionally,
the code is analyzed by flake8 (Cordasco,, 2010) to ensure compliance with the
PEP8 coding style standards (van Rossum et al.,, 2001). Finally, the
properties of each public function are documented thoroughly in an API
reference guide, embedded as Python docstrings and accessible through the
official project homepage or an interactive Python console. Short usage
examples are typically included inside the docstrings, and new features are
accompanied by longer, self-contained example scripts added to the narrative
documentation and compiled to a gallery on the project website. We use Sphinx
(Brandl,, 2007) to automatically generate both library documentation and the
website.

The development master branch is fully functional at all times and can be
obtained from GitHub. The community releases major updates as stable versions
approximately every six months. Major releases include new features, while
minor releases typically contain only bug fixes. Going forward, users will be
notified about API-breaking changes through deprecation warnings for two full
major releases before the changes are applied.

## Usage examples

### Research

Often, a disproportionately large component of research involves dealing with
various image data-types, color representations, and file format conversion.
scikit-image offers robust tools for converting between image data-types
(Microsoft,, 1995; Munshi and Leech,, 2010; Paeth,, 1990) and to do file
input/output (I/O) operations. Our purpose is to allow investigators to focus
their time on research, instead of expending effort on mundane low-level
tasks.

The package includes a number of algorithms with broad applications across
image processing research, from computer vision to medical image analysis. We
refer the reader to the current API documentation for a full listing of
current capabilities171717http://scikit-image.org/docs/dev, Accessed
2014-03-30. In this section we illustrate two real-world usage examples of
scikit-image in scientific research.

First, we consider the analysis of a large stack of images, each representing
drying droplets containing nanoparticles (see Figure 2). As the drying
proceeds, cracks propagate from the edge of the drop to its center. The aim is
to understand crack patterns by collecting statistical information about their
positions, as well as their time and order of appearance. To improve the speed
at which data is processed, each experiment, constituting an image stack, is
automatically analysed without human intervention. The contact line is
detected by a circular Hough transform (transform.hough_circle) providing the
drop radius and its center. Then, a smaller concentric circle is drawn
(draw.circle_perimeter) and used as a mask to extract intensity values from
the image. Repeating the process on each image in the stack, collected pixels
can be assembled to make a space-time diagram. As a result, a complex stack of
images is reduced to a single image summarizing the underlying dynamic
process.

Figure 2: scikit-image is used to track the propagation of cracks (black
lines) in a drying colloidal droplet. The sequence of pictures shows the
temporal evolution of the system with the drop contact line, in green,
detected by the Hough transform and the circle, in white, used to extract an
annulus of pixel intensities. The result shown illustrates the angular
position of cracks and their time of appearance.

Next, in regenerative medicine research, scikit-image is used to monitor the
regeneration of spinal cord cells in zebrafish embryos (Figure 3). This
process has important implications for the treatment of spinal cord injuries
in humans (Bhatt et al.,, 2004; Thuret et al.,, 2006).

To understand how spinal cords regenerate in these animals, injured cords are
subjected to different treatments. Neuronal precursor cells (labeled green in
Figure 3, left panel) are normally uniformly distributed across the spinal
cord. At the wound site, they have been removed. We wish to monitor the
arrival of new cells at the wound site over time. In Figure 3, we see an
embryo two days after wounding, with precursor cells beginning to move back
into the wound site (the site of minimum fluorescence). The
measure.profile_line function measures the fluorescence along the cord,
directly proportional to the number of cells. We can thus monitor the recovery
process and determine which treatments prevent or accelerate recovery.

Figure 3: The measure.profile_line function being used to track recovery in
spinal cord injuries. (a): an image of fluorescently-labeled nerve cells in an
injured zebrafish embryo. (b): the automatically determined region of
interest. The SciPy library was used to determine the region extent
(Oliphant,, 2007; Jones et al.,, 2001), and functions from the scikit-image
draw module were used to draw it. (c): the image intensity along the line of
interest, averaged over the displayed width.

### Education

scikit-image’s simple, well-documented application programming interface (API)
makes it ideal for educational use, either via self-taught exploration or
formal training sessions.

The online gallery of examples not only provides an overview of the
functionality available in the package but also introduces many of the
algorithms commonly used in image processing. This visual index also helps
beginners overcome a common entry barrier: locating the class (denoising,
segmentation, etc.) and name of operation desired, without being proficient
with image processing jargon. For many functions, the documentation includes
links to research papers or Wikipedia pages to further guide the user.

Demonstrating the broad utility of scikit-image in education, thirteen-year-
old Rishab Gargeya of the Harker School won the Synopsys Silicon Valley
Science and Technology Championship using scikit-image in his project, “A
software based approach for automated pathology diagnosis of diabetic
retinopathy in the human retina” (science-fair.org,, 2014).

We have delivered image processing tutorials using scikit-image at various
annual scientific Python conferences, such as PyData 2012, SciPy India 2012,
and EuroSciPy 2013. Course materials for some of these sessions are found in
Haenel et al., (2014) and are licensed under the permissive CC-BY license
(Creative Commons,, 2013). These typically include an introduction to the
package and provide intuitive, hands-on introductions to image processing
concepts. The well documented application programming interface (API) along
with tools that facilitate visualization contribute to the learning
experience, and make it easy to investigate the effect of different algorithms
and parameters. For example, when investigating denoising, it is easy to
observe the difference between applying a median filter (filter.rank.median)
and a Gaussian filter (filter.gaussian_filter), demonstrating that a median
filter preserves straight lines much better.

Finally, easy access to readable source code gives users an opportunity to
learn how algorithms are implemented and gives further insight into some of
the intricacies of a fast Python implementation, such as indexing tricks and
look-up tables.

### Industry

Due to the breadth and maturity of its code base, as well as the its
commercial-friendly license, scikit-image is well suited for industrial
applications.

BT Imaging (btimaging.com) designs and builds tools that use photoluminescence
(PL) imaging for photovoltaic applications. PL imaging can characterize the
quality of multicrystalline silicon wafers by illuminating defects that are
not visible under standard viewing conditions. The left panel of Figure 4
shows an optical image of a silicon wafer, and the center panel shows the same
wafer using PL imaging. In the right panel, the wafer defects and impurities
have been detected through automated image analysis. scikit-image plays a key
role in the image processing pipeline. For example, a Hough transform
(transform.hough_line) finds the wafer edges in order to segment the wafer
from the background. scikit-image is also used for feature extraction. Crystal
defects (dislocations) are detected using a band-pass filter, which is
implemented as a Difference of Gaussians (filter.gaussian_filter).

The image processing results are input to machine learning algorithms, which
assess intrinsic wafer quality. Solar cell manufacturers can use this
information to reject poor quality wafers and thereby increase the fraction of
solar cells that have high solar conversion efficiency.

Figure 4: (a): An image of an as-cut silicon wafer before it has been
processed into a solar cell. (b): A PL image of the same wafer. Wafer defects,
which have a negative impact solar cell efficiency, are visible as dark
regions. (c): Image processing results. Defects in the crystal growth
(dislocations) are colored blue, while red indicates the presence of
impurities.

scikit-image is also applied in a commercial setting for biometric security
applications. AICBT Ltd uses multispectral imaging to detect when a person
attempts to conceal their identity using a facial
mask181818http://www.aicbt.com/disguise-detection, Accessed 2014-03-30.
scikit-image performs file I/O (io.imread), histogram equalization
(exposure.equalize_hist), and aligns a visible wavelength image with a thermal
image (transform.AffineTransform). The system determines the surface
temperature of a subject’s skin and detects situations where the face is being
obscured.

## Example: image registration and stitching

This section gives a step-by-step outline of how to perform panorama stitching
using the primitives found in scikit-image. The full source code is at
https://github.com/scikit-image/scikit-image-demos.

### Data loading

The “ImageCollection” class provides an easy way of representing multiple
images on disk. For efficiency, images are not read until accessed.

⬇

ic = io.ImageCollection(’data/*’)

Figure 5(a) shows the Petra dataset, which displays the same facade from two
different angles. For this demonstration, we will estimate a projective
transformation that relates the two images. Since the outer parts of these
photographs do not comform well to such a model, we select only the central
parts. To further speed up the demonstration, images are downscaled to 25% of
their original size.

⬇

from skimage import transform

image0 = rgb2gray(ic[0][:, 500:500+1987, :])

image1 = rgb2gray(ic[1][:, 500:500+1987, :])

image0 = transform.rescale(image0, 0.25)

image1 = transform.rescale(image1, 0.25)

### Feature detection and matching

“Oriented FAST and rotated BRIEF” (ORB) features (Rublee et al.,, 2011) are
detected in both images. Each feature yields a binary descriptor; those are
used to find the putative matches shown in Figure 5(b).

⬇

orb = ORB(n_keypoints=1000, fast_threshold=0.05)

orb.detect_and_extract(image0)

keypoints1 = orb.keypoints

descriptors1 = orb.descriptors

orb.detect_and_extract(image1)

keypoints2 = orb.keypoints

descriptors2 = orb.descriptors

matches12 = match_descriptors(descriptors1,

descriptors2,

cross_check=True)

### Transform estimation

To filter the matches, we apply RANdom SAmple Consensus (RANSAC) (Fischler and
Bolles,, 1981), a common method for outlier rejection. This iterative process
estimates transformation models based on randomly chosen subsets of matches,
finally selecting the model which corresponds best with the majority of
matches. The new matches are shown in Figure 5(c).

⬇

# Select keypoints from the source (image to be

# registered) and target (reference image).

src = keypoints2[matches12[:, 1]][:, ::-1]

dst = keypoints1[matches12[:, 0]][:, ::-1]

model_robust, inliers = \

ransac((src, dst), ProjectiveTransform,

min_samples=4, residual_threshold=2)

### Warping

Next, we produce the panorama itself. The first step is to find the shape of
the output image by considering the extents of all warped images.

⬇

# Note that transformations take coordinates in

# (x, y) format, not (row, column), in order to be

# consistent with most literature.

corners = np.array([[0, 0],

[0, r],

[c, 0],

[c, r]])

# Warp the image corners to their new positions.

warped_corners = model_robust(corners)

# Find the extents of both the reference image and

# the warped target image.

all_corners = np.vstack((warped_corners, corners))

corner_min = np.min(all_corners, axis=0)

corner_max = np.max(all_corners, axis=0)

output_shape = (corner_max - corner_min)

output_shape = np.ceil(output_shape[::-1])

The images are now warped according to the estimated transformation model.
Values outside the input images are set to -1 to distinguish the “background”.

A shift is added to ensure that both images are visible in their entirety.
Note that warp takes the inverse mapping as input.

⬇

from skimage.exposure import rescale_intensity

from skimage.transform import warp

from skimage.transform import SimilarityTransform

offset = SimilarityTransform(translation=-corner_min)

image0_ = warp(image0, offset.inverse,

output_shape=output_shape, cval=-1)

image1_ = warp(image1, (model_robust + offset).inverse,

output_shape=output_shape, cval=-1)

An alpha channel is added to the warped images before merging them into a
single image:

⬇

"""Add an alpha layer to the image.

The alpha layer is set to 1 for foreground

and 0 for background.

"""

rgb = gray2rgb(image)

alpha = (image != background)

return np.dstack((rgb, alpha))

image0_alpha = add_alpha(image0_)

image1_alpha = add_alpha(image1_)

merged = (image0_alpha + image1_alpha)

alpha = merged[…, 3]

# The summed alpha layers give us an indication of

# how many images were combined to make up each

# pixel. Divide by the number of images to get

# an average.

merged /= np.maximum(alpha, 1)[…, np.newaxis]

The merged image is shown in Figure 5(d). Note that, while the columns are
well aligned, the color intensities at the boundaries are not well matched.

Figure 5: An example application of scikit-image: image registration and
warping to combine overlapping images. (a): Photographs taken in Petra, Jordan
by François Malan. License: CC-BY. (b): Putative matches computed from ORB
binary features. (c): Matches filtered using RANSAC. (d): The second input
frame (middle) is warped to align with the first input frame (left), yielding
the averaged image shown on the right. (e): The final panorama image,
registered and warped using scikit-image, blended with Enblend.

### Blending

To blend images smoothly we make use of the open source package Enblend
(Dersch,, 2010), which in turn employs multi-resolution splines and Laplacian
pyramids (Burt and Adelson, 1983b, ; Burt and Adelson, 1983a, ). The final
panorama is shown in Figure 5(e).

## Discussion

### Related work

In this section, we describe other libraries with similar goals to ours.

Within the scientific Python ecosystem, Mahotas contains many similar
functions, and is furthermore also designed to work with NumPy arrays
(Coelho,, 2013). The major philosophical difference between Mahotas and
scikit-image is that Mahotas is almost exclusively written in templated C++,
while scikit-image is written in Python and Cython. We feel that our choice
lowers the barrier of entry for new contributors. However, thanks to the
interoperability between the two provided by the NumPy array data format,
users don’t have to choose between them, and can simply use the best
components of each.

ImageJ and its batteries-included Fiji distribution are probably the most
popular open-source tools for image analysis (Schneider et al.,, 2012;
Schindelin et al.,, 2012). Although Fiji’s breadth of functionality is
unparalleled, it is centered around interactive, GUI use. For many developers,
then, scikit-image offers several advantages. Although Fiji offers a
programmable macro mode that supports many scripting languages, many of the
macro functions activate GUI elements and cannot run in headless mode. This is
problematic for data analysis in high-performance computing cluster
environments or web backends, for example. Additionally, Fiji’s inclusive
plugin policy results in an inconsistent API and variable documentation
quality. Using scikit-image to develop new functionality or to build batch
applications for distributed computing is often much simpler, thanks to its
consistent API and the wide distribution of the scientific Python stack.

In many respects, the image processing toolbox of the Matlab environment is
quite similar to scikit-image. For example, its API is mostly functional and
applies to generic multidimensional numeric arrays. However, Matlab’s
commercial licensing can be a significant nuisance to users. Additionally, the
licensing cost increases dramatically for parallel computing, with per-worker
pricing191919http://www.mathworks.com.au/products/distriben/description3.html,
Accessed 2014-05-09. Finally, the closed source nature of the toolbox prevents
users from learning from the code or modifying it for specific purposes, which
is a common necessity in scientific research. We refer readers back to the
Development Practices section for a summary of the practical and philosophical
advantages of our open-source licensing.

OpenCV is a BSD-licensed open-source library focused on computer vision, with
a separate module for image processing (Bradski,, 2000). It is developed in
C/C++ and the project’s main aim is to provide implementations for real-time
applications. This results in fast implementations with a comparatively high
barrier of entry for code study and modification. The library provides
interfaces for several high-level programming languages, including Python
through the NumPy-array data-type for images. The Python interface is
essentially a one-to-one copy of the underlying C/C++ API, and thus image
processing pipelines have to follow an imperative programming style. In
contrast, scikit-image provides a Pythonic interface with the option to follow
an imperative or functional approach. Beyond that, OpenCV’s image processing
module is traditionally limited to 2-dimensional imagery.

The choice of image processing package depends on several factors, including
speed, code quality and correctness, community support, ecosystem, feature
richness, and users’ ability to contribute. Sometimes, advantages in one
factor come at the cost of another. For example, our approach of writing code
in a high-level language may affect performance, or our strict code review
guidelines may hamper the number of features we ultimately provide. We
motivate our design decisions for scikit-image in the Development Practices
section, and leave readers to decide which library is right for them.

### Roadmap

In many open source projects, decisions about future development are made
through “rough consensus and working code” (Bradner,, 1998). In scikit-image
there are two ways to propose new ideas: through discussion on the mailing
list, or as pull requests. The latter route has the advantage of a concrete
implementation to guide the conversation, and often mailing list discussions
also result in a request for a proof of concept implementation. While
conversations are usually led by active developers, the entire community is
invited to participate. Once general agreement is reached that the proposed
idea aligns with the current project goals and is feasible, work is divided on
a volunteer basis. As such, the schedule for completion is often flexible.

The following goals have been identified for the next release of scikit-image:

  * •

Obtain full test coverage.

  * •

Overhaul the functions for image reading/writing.

  * •

Improve the project infrastructure, e.g. create an interactive gallery of
examples.

  * •

Add support for graph-based operations.

  * •

Significantly extend higher dimensional (multi-layer) support.

We also invite readers to submit their own feature requests to the mailing
list for further discussion.

## Conclusion

scikit-image provides easy access to a powerful array of image processing
functionality. Over the past few years, it has seen significant growth in both
adoption and contribution202020https://www.ohloh.net/p/scikit-image, Accessed
2014-05-15, and the team is excited to collaborate with others to see it grow
even further, and to establish it the de facto library for image processing in
Python.

## Acknowledgements

We thank Timo Friedrich and Jan Kaslin for providing the zebrafish lesion
data. We also acknowledge the efforts of the more than 100 contributors to the
scikit-image code base (listed as "the scikit-image contributors" in the
author list).

## References

  * Behnel et al., (2011) Behnel, S., Bradshaw, R., Citro, C., Dalcin, L., Seljebotn, D., and Smith, K. (2011).  Cython: The best of both worlds.  Computing in Science and Engineering, 13(2):31–39.
  * Bhatt et al., (2004) Bhatt, D., Otto, S., Depoister, B., and JR., F. (2004).  Cyclic amp-induced repair of zebrafish spinal circuits.  Science, 305:254–258.
  * Bradner, (1998) Bradner, S. (1998).  IETF Working Group Guidelines and Procedures.  RFC 2418 (Best Current Practice).  Updated by RFC 3934.
  * Bradski, (2000) Bradski, G. (2000).  The OpenCV library.  Dr. Dobb’s Journal of Software Tools, 25:120–126.
  * Brandl, (2007) Brandl, G. (2007).  Sphinx Python Documentation Generator.  http://sphinx-doc.org/ Accessed: 2014-03-30.
  * (6) Burt, P. and Adelson, E. (1983a).  The laplacian pyramid as a compact image code.  Communications, IEEE Transactions on, 31(4):532–540.
  * (7) Burt, P. and Adelson, E. (1983b).  A multiresolution spline with application to image mosaics.  ACM Transactions on Graphics, 2(4):217–236.
  * Canny, (1986) Canny, J. (1986).  A computational approach to edge detection.  IEEE Trans. Pattern Analysis and Machine Intelligence., 8:679–714.
  * Coelho, (2013) Coelho, L. (2013).  Mahotas: Open source software for scriptable computer vision.  Journal of Open Research Software, 1(1).
  * Cordasco, (2010) Cordasco, I. (2010).  Flake8.  https://pypi.python.org/pypi/flake8 Accessed: 2014-03-30.
  * Creative Commons, (2013) Creative Commons (2013).  CC-BY license.  http://creativecommons.org/licenses/by/4.0/ Accessed: 2014-03-30.
  * Dersch, (2010) Dersch, H. (2010).  Enblend 4.0 documentation.  http://enblend.sourceforge.net Accessed: 2014-03-30.
  * Fischler and Bolles, (1981) Fischler, M. and Bolles, R. (1981).  Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography.  Comm. of the ACM, 24(6):381–395.
  * Gommers, (2010) Gommers, R. (2010).  A Guide to NumPy/SciPy Documentation.  https://github.com/numpy/numpy/blob/master/doc/HOWTO_DOCUMENT.rst.txt Accessed: 2014-03-30.
  * Haenel et al., (2014) Haenel, V., Gouillart, E., Varoquaux, G., and scipy-lecture-notes contributors (2014).  Scipy lecture notes.  http://scipy-lectures.github.io/ Accessed: 2014-03-30.
  * Halchenko and Hanke, (2012) Halchenko, Y. O. and Hanke, M. (2012).  Open is not enough. let’s take the next step: An integrated, community-driven computing platform for neuroscience.  Frontiers in Neuroinformatics, 6(22).
  * Hunter, (2007) Hunter, J. D. (2007).  Matplotlib: A 2d graphics environment.  Computing In Science & Engineering, 9(3):90–95.
  * Jones et al., (2001) Jones, E., Oliphant, T. E., Peterson, P., et al. (2001).  SciPy: Open source scientific tools for Python.
  * Microsoft, (1995) Microsoft (1995).  DirectX data conversion rules.  http://msdn.microsoft.com/en-us/library/windows/desktop/dd607323 Accessed: 2014-03-30.
  * Munshi and Leech, (2010) Munshi, A. and Leech, J. (2010).  OpenGL ES common profile specification, version 2.0.25 (full specification).  https://www.khronos.org/registry/gles/specs/2.0/es_full_spec_2.0.25.pdf Accessed: 2014-03-30.
  * Oliphant, (2007) Oliphant, T. E. (2007).  Python for scientific computing.  Computing In Science & Engineering, 9(3):10–20.
  * Paeth, (1990) Paeth, A. (1990).  Proper treatment of pixels as integers. Graphics Gems.
  * Pedregosa et al., (2011) Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., and Duchesnay, E. (2011).  Scikit-learn: machine learning in python.  Journal of Machine Learning Research, 12:2825–2830.
  * Regents of the University of California, (1999) Regents of the University of California (1999).  http://www.gnu.org/licenses/license-list.html#ModifiedBSD Accessed: 2014-03-30.
  * Rublee et al., (2011) Rublee, E., Rabaud, V., Konolige, K., and G., B. (2011).  ORB: An efficient alternative to SIFT or SURF.  In Proceedings of the 2011 International Conference on Computer Vision (ICCV), pages 2564–2571.
  * Schindelin et al., (2012) Schindelin, J., Arganda-Carreras, I., Frise, E., Kaynig, V., Longair, M., Pietzsch, T., Preibisch, S., Rueden, C., Saalfeld, S., Schmid, B., Tinevez, J.-Y., White, D. J., Hartenstein, V., Eliceiri, K., Tomancak, P., and Cardona, A. (2012).  Fiji: an open-source platform for biological-image analysis.  Nature methods, 9(7):676–682.
  * Schneider et al., (2012) Schneider, C. A., Rasband, W. S., and Eliceiri, K. W. (2012).  NIH Image to ImageJ: 25 years of image analysis.  Nature methods, 9(7):671–675.
  * science-fair.org, (2014) science-fair.org (2014).  Privately sponsored project, project awards 2014, Synopsis Silicon Valley Science and Technology Championship.  http://science-fair.org/database/project_awards.php?schoolname=Privately+Sponsored+Project&school_year=2014 Accessed: 2014-03-30.
  * Thuret et al., (2006) Thuret, S., Moon, L., and Gage, F. (2006).  Therapeutic interventions after spinal cord injury.  Nature Rev Neurosci, 7:628–643.
  * van der Walt et al., (2011) van der Walt, S., Colbert, C., and G, V. (2011).  The NumPy array: a structure for efficient numerical computation.  Computing in Science and Engineering, 13(2):22–30.
  * van Rossum et al., (2001) van Rossum, G., Warsaw, B., and N, C. (2001).  PEP 8: Style guide for Python code.  http://www.python.org/dev/peps/pep-0008/ Accessed: 2014-03-30.