# Rawgment: Noise-Accounted RAW Augmentation Enables Recognition in a Wide
Variety of Environments

Masakazu Yoshimura Junji Otsuka Atsushi Irie Takeshi Ohashi
Sony Group Corporation
{masakazu.yoshimura, junji.otsuka, atsushi.irie<EMAIL_ADDRESS>
###### Abstract

Image recognition models that can work in challenging environments (e.g.,
extremely dark, blurry, or high dynamic range conditions) must be useful.
However, creating a training dataset for such environments is expensive and
hard due to the difficulties of data collection and annotation. It is
desirable if we could get a robust model without the need of hard-to-obtain
dataset. One simple approach is to apply data augmentation such as color
jitter and blur to standard RGB (sRGB) images in simple scenes. Unfortunately,
this approach struggles to yield realistic images in terms of pixel intensity
and noise distribution due to not considering the non-linearity of Image
Signal Processor (ISP) and noise characteristics of an image sensor. Instead,
we propose a noise-accounted RAW image augmentation method. In essence, color
jitter and blur augmentation are applied to a RAW image before applying non-
linear ISP, yielding realistic intensity. Furthermore, we introduce a noise
amount alignment method that calibrates the domain gap in noise property
caused by the augmentation. We show that our proposed noise-accounted RAW
augmentation method doubles the image recognition accuracy in challenging
environments only with simple training data.

###  1 Introduction

Although image recognition has been actively studied recently, the performance
in challenging environments still needs improvements [15]. In sensitive
application such as mobility sensing and head-mounted wearables, the devices
have to be robust against various kinds of difficulties such as low-light,
high dynamic range (HDR) illuminance, motion blur, and camera shake. One
possible solution is to use image enhancement and restoration methods. A lot
of DNN-based low-light image enhancement [29, 54, 20, 46, 12, 30], denoising
[53, 32, 43], and deblur [52, 48, 43] methods were proposed and improved the
pre-captured sRGB image quality. They are quite useful for improving pre-
captured image quality, but a recent work [15] showed that the accuracy gain
using them as a preprocessing of image recognition models was limited since
they already lost some information and restoring the lost information is
difficult.

unrealistic intensity
---
ISP
---
(Contrast)
---
augmentation
---
(Contrast)
---
augmentation
---
ISP
---
unrealistic noise
---
(a) Usual training pipeline
---
RAW image
---
luminance+noise
---
realistic intensity
---
realistic noise
---
(b) Noise-accounted RAW augmentation pipeline
---
RAW image
---
luminance+noise
---
Noise correction
---
Figure 1: The concept of the proposed noise-accounted RAW augmentation. The
conventional augmentation (a) is applied to the output of an ISP. It generates
images that cannot be taken with any ambient light intensities due to the
nonlinear operations in the ISP. Instead, ours (b) apply augmentation before
an ISP. It generates realistic pixel intensity distribution that can be taken
when the light intensity is different. Moreover, the noise amount is also
corrected to minimize the domain gap between real and augmented ones.

Another possible solution is to prepare a dataset of the difficult environment
[33, 3]. However, the mentioned datasets only cover one or a few difficulties
and to create datasets in various environments is too expensive. Especially,
manual annotation of challenging scenes is difficult and time-consuming. For
example, we can see almost nothing in usual sRGB images under extremely low-
light environments due to heavy noise. In addition, some regions in HDR scenes
suffer from halation or blocked-up shadows because the 8-bit dynamic range of
usual sRGB images cannot fully capture the real world, e.g. 0.000001
$[cd/m^{2}]$ under the starlight and 1.6 billion $[cd/m^{2}]$ under the direct
sunlight [37]. Heavy motion blur and camera shake also make annotation
difficult. Some works took paired short and long exposure image, and the long-
exposure clean image was used for annotation or ground truth [15, 16, 17, 19].
The limitation is that the target scene needed to be motionless if the pairs
are taken sequentially with one camera [15] or positional calibration is
needed if the pairs are taken with synchronized cameras [16, 17]. Some works
used a beam splitter to capture challenging images and their references
without calibration [45, 19]. However, it is difficult to apply to dark and
blurry scenes because one of them becomes blurry or dark. Moreover, HDR images
cannot be taken in the same way because some regions become overexposed or
underexposed in both of the cameras.

To this end, we aim to train image recognition models that work in various
environments only using a training dataset in simple environments like bright,
low dynamic range, or blur-less. In this case, image augmentation or domain
adaptation is important to overcome the domain gap between easy training data
and difficult test data. However, we believe usual augmentation on sRGB space
is ineffective because it does not take nonlinear mapping of an ISP into
consideration. Especially, tone mapping drastically changes the RAW image
values that are roughly proportional to physical brightness [44]. Contrast,
brightness, and hue augmentation on sRGB space result in unrealistic images
that cannot be taken under any different ambient light intensity as shown in
Fig. 1(a). In contrast, we propose augmentation on RAW images. In other words,
augmentation is applied before ISP to diminish the domain shift as shown in
Fig. 1(b).

Another possible source of domain gap is the noise amount and noise
distribution difference. To tackle these problems, we propose a method to
align both light intensity and noise domain. In fact, recent works showed that
adding physics-based realistic noise improves the performance of DNN-based
denoisers [44, 47, 2, 50] and dark image recognition [15, 4]. Although their
proposed sensor noise modelings were accurate, they assumed that original
bright images were noise free. In contrast, we propose to modify the noise
amount after contrast, brightness, and hue conversion considering the noise
amount of the original images. It enables a more accurate alignment of the
noise domain. Even in bright images, there might be some dark parts due to
shadow or the color of objects and their prior noise can not be ignored.
Another merit of our method is that it can take an input of a dark image which
already contains a lot of noise. In addition to noise amount alignment after
color jitter augmentation, we show the importance of noise alignment after
blur augmentation which is firstly proposed in this paper.

Our contributions are as follows:

  * •

It is the first work to emphasize the importance of augmentation before ISP
for image recognition to the best of our knowledge.

  * •

Noise amount alignment method is proposed to reduce the noise domain gap after
RAW image augmentation. In contrast to previous works, our proposed method
takes into account prior noise in the input image. It enables more accurate
alignment and use of any strength of augmentation and even already noisy
input.

  * •

We show qualitative analysis for the validity of our sensor noise modeling and
corresponding noise accounted augmentation. We prove that our proposed noise-
accounted RAW image augmentation has the edge over the previous methods.

###  2 Related Works

####  2.1 Recognition in Difficult Environment

Many works have tackled image recognition in difficult environments. For low-
light environments, several works improved the accuracy by replacing a
traditional ISP with a powerful DNN-based ISP to create clean images for
downstream image recognition models [5, 33, 25]. Even though these methods are
promising because there is no information loss, computational cost is the
problem. Another approach is direct RAW image recognition without ISP [39,
15]. Their image recognition models benefit from the richest information and
improve the accuracy under low-light. However, several works reported ISPs
especially tone mapping is helpful for machine vision [49, 13]. We argue that
direct RAW image recognition works well if the images have low dynamic range.
Another approach is domain adaptation or related methods which support low-
light recognition with bright images [38, 15, 4].

For HDR environments, some works have proposed DNN-based auto-exposure control
[42, 35] to improve downstream recognition. Also, multi-frame HDR synthesis
methods [6, 1] can be used as a preprocessing, but camera motion makes them
challenging. Luminance normalization method was also introduced to improve
recognition performance under varying illumination conditions [18].

For blurry environments, deblurring methods were actively studied [48, 51].
These DNN-based methods successfully restored clear images from heavy blurred
ones.

Differ from the above, we aim to do image recognition under all the above
difficulties using simple scene training data with our proposed augmentation
method. We did not use domain adaptation methods since these methods were
usually used in a setting where the target domain is equal to or smaller than
the source domain [10]. On the contrary, in our setting, the distribution of
target domain is much wider than that of source domain.

####  2.2 Image Conversion on RAW

Recently, several methods [2, 29, 4] converted light sRGB images into
realistic dark images by the following procedures. First, they inverted an ISP
pipeline to generate RAW-like images followed by illumination change on the
RAW data space with plausible sensor noise. Afterward, degraded sRGB was
generated by applying the forward ISP pipeline. By this operation, ISP’s non-
linear operation could be avoided and short exposure or dark environment could
be simulated. With a similar intention, we propose to apply augmentation
before ISPs to train image recognitiom model.

####  2.3 Noise Modeling and Noise Amount Alignment

In the electronic imaging sensor community, detailed noise modelings based on
electric current and circuit have been studied [41, 7, 23, 11]. They are
precise but difficult to be applied to the image-to-image conversion. Thus, in
the machine vision community, simplified pixel value-based noise modelings
were proposed based on electric noise modelings [44, 47, 2]. Although the
noise model of [47] was well designed with a high degree of freedom Tukey
lambda distribution [21], we are based on the well-established heteroscedastic
Gaussian model [2, 36, 9, 50] because it was still well fitted to real sensor
noise and we can consider prior noise in original images which will be
explained later.

Recently, adding realistic, model-based sensor noise to the ground truth clean
images was proved to be helpful to train DNN-based denoiser [44, 50, 47, 2]
and low-light object detection models [15, 4]. Although they use highly
consistent noise models, they regarded original images as noise-free. In
contrast, we propose to modify the noise amount after image conversion
considering the noise amount of the original images. It enables a more
accurate alignment of the noise domain and enables the use of any intensity of
augmentation and already noisy images as input.

###  3 Methodology

In this section, we introduce our noise model, calibration procedure, and
proposed noise-accounted RAW image augmentation.

####  3.1 Noise Model

First of all, we briefly introduce our noise model for later explanation
although it is based on the well-established heteroscedastic Gaussian model
[2, 36, 9, 50]. The number of photons $u$ hit the photodiode of each pixel is
converted to a voltage with quantum efficiency $\alpha$. It is then followed
by some processes to read out the voltage, in which noise $n_{d}$ is
inevitably mixed. Then, analog gain $g$ is multiplied to amplify the value.
Lastly, the voltage is converted to a digital value. We simplify and summarize
the noise after analog gain as $n_{r}$. Since it is common to use analog gain
which has better signal to noise ratio (SNR), we omit the digital gain term in
our noise model. To sum up, the photon-to-RAW pixel value conversion can be
formulated as,

$x=g\left(\alpha u+n_{d}\right)+n_{r}.$ (1)

We approximate $n_{d}$ and $n_{r}$ as Gaussian noise
$\mathcal{N}\left(0,\sigma_{d}^{2}\right)$
$\mathcal{N}\left(0,\sigma_{r}^{2}\right)$ and the number of photons $u$
itself obeys the Poisson distribution $\mathcal{P}\left(\bar{u}\right)$ where
$\bar{u}$ is the expected number of photons. If $\bar{u}$ is large enough, we
can approximate as
$\mathcal{P}\left(\bar{u}\right)\fallingdotseq\mathcal{N}\left(\bar{u},\bar{u}\right)$
[9]. Thus, our noise model is as follows:

$x\sim
g\left(\alpha\mathcal{N}\left(\bar{u},\bar{u}\right)+\mathcal{N}\left(0,\sigma_{d}^{2}\right)\right)+\mathcal{N}\left(0,\sigma_{r}^{2}\right).$
(2)

We show the validity of the Gaussian approximation of $n_{d}$, $n_{r}$, and
$\mathcal{P}\left(\bar{u}\right)$ in the Section 4.3. We don’t follow the
further development of the formula in [9] for our purpose.

Gaussian distribution has the following convenient natures;

$\begin{cases}X\sim\mathcal{N}\left(\mu_{X},\sigma_{X}^{2}\right)\\\
Y\sim\mathcal{N}\left(\mu_{Y},\sigma_{Y}^{2}\right)\\\
X+Y\sim\mathcal{N}\left(\mu_{X}+\mu_{Y},\sigma_{X}^{2}+\sigma_{Y}^{2}\right)\\\
cX\sim\mathcal{N}\left(c\mu_{X},c^{2}\sigma_{X}^{2}\right)\end{cases},$ (3)

if $X$ and $Y$ are independent, and that’s why we choose the simple Gaussian
approximation instead of the recently proposed more expressive noise model
[47]. These natures enable the proposed noise-accounted RAW augmentation to
account for prior noise in input images. Furthermore, they further simplify
our noise model as,

$x\sim\mathcal{N}\left(g\alpha\bar{u},g^{2}\alpha^{2}\bar{u}+g^{2}\sigma_{d}^{2}+\sigma_{r}^{2}\right).$
(4)

Because the expected number of photon $\bar{u}$ is inconvenient to use in
image-to-image conversion, we replace it with the expected pixel value
$\mu_{x}=g\alpha\bar{u}$ and our final noise model is defined as

$x\sim\mathcal{N}\left(\mu_{x},\sigma_{x}^{2}=g\alpha\mu_{x}+g^{2}\sigma_{d}^{2}+\sigma_{r}^{2}\right).$
(5)

####  3.2 Noise Model Calibration

N burst images
---
compute the mean and variance
---
along the time dimension
---
(a) burst capture with several analog gains
---
(b) estimate linear relationship of the mean and variance
---
(c) estimate the sensor specific parameters
---
solve the simultaneous equation
---
mean (expected) pixel value
---

noise variance
---

Figure 2: Our noise model calibration procedures to a target sensor.

Our sensor noise model shown in Eq. (5) has three parameters, $\alpha$,
$\sigma_{d}^{2}$, and $\sigma_{r}^{2}$, which have to be calibrated per target
sensor. We capture a series of raw images of a color checker as shown in Fig.
2(a). We then calculate the mean $\mu_{x}$ and variance $\sigma_{x}^{2}$ along
the time direction of each pixel position. We calculate them along the time
direction instead of the spatial direction as performed in [44] since lens
distortion changes the luminance of the same color patch. These operations are
performed several times by changing the analog gain and exposure time.
Eventually, we get various sets of $\left\\{\mu_{x},\sigma_{x}^{2}\right\\}$
for each analog gain. Note that we calculate mean and variance without
separating RGB channels because there is no significant difference in noise
properties.

In Eq. (5), $\mu_{x}$ and $\sigma_{x}^{2}$ have a linear relationship per
analog gain $g_{n}$,

$\sigma_{x}^{2}=a_{g_{n}}\mu_{x}+b_{g_{n}}.$ (6)

Therefore, we solve linear regression to estimate $a_{g_{n}}$ and $b_{g_{n}}$
per gain like Fig. 2(b). In addition, we use RANSAC [8] to robustly take care
of outlier $\left\\{\mu_{x},\sigma_{x}^{2}\right\\}$ pairs.

Finally, we estimate $\alpha$, $\sigma_{d}^{2}$, and $\sigma_{r}^{2}$ from the
following redundant simultaneous equations by least-squares method,

$\begin{cases}a_{g_{1}}=g_{1}\alpha\\\ \>\>\>\vdots\\\
a_{g_{n}}=g_{n}\alpha\end{cases}$ (7)

$\begin{cases}b_{g_{1}}=g_{1}^{2}\sigma_{d}^{2}+\sigma_{r}^{2}\\\
\>\>\>\vdots\\\ b_{g_{n}}=g_{n}^{2}\sigma_{d}^{2}+\sigma_{r}^{2}\end{cases}.$
(8)

As the procedure above, we can calibrate the sensor noise model without using
any special devices. We later show that our sensor model and the calibration
method represent the real sensor noise with enough preciseness.

####  3.3 Noise-Accounted RAW Augmentation

We propose augmentation before ISP instead of the usual augmentation after ISP
to generate realistic images. Furthermore, we improve the reality of the
augmented images by considering the sensor noise model. Unlike the previous
works [44, 47, 4, 15, 2, 50], ours takes the prior noise amount of input
images into account. It generates more realistic noise since even bright
images have some extent of noise. Especially, dark parts due to shadow or the
color of objects might have a non-negligible amount of noise. Moreover, it
allows any brightness of input images different from previous works.
Specifically, we introduce how to adjust noise amount after contrast,
brightness, hue, and blur augmentation.

#####  3.3.1 Color Jitter Augmentation

Contrast, brightness, and hue augmentation simulate different exposure time,
light intensity, and analog gain. Hence, we firstly assume to multiply the
exposure time, light intensity, and analog gain by $p_{e}$, $p_{i}$, and
$p_{g}$ respectively. Because $p_{e}$ and $p_{i}$ equally change the number of
photon $u$ in the case of our noise model, we rewrite them as
$p_{u}=p_{e}p_{i}$. Then, images in the above environment settings $x_{new}$
can be rewritten as,

$x_{new}\sim\mathcal{N}\left(\begin{split}(p_{g}g)\alpha(p_{u}\bar{u}),\\\
(p_{g}g)^{2}\alpha^{2}(p_{u}\bar{u})+(p_{g}g)^{2}\sigma_{d}^{2}+\sigma_{r}^{2}\end{split}\right).$
(9)

Based on Eq. (3), it can be expanded as

$\displaystyle x_{new}$ $\displaystyle\sim\mathcal{N}\left((p_{g}p_{u})\alpha
g\bar{u},\>(p_{g}p_{u})^{2}(g^{2}\alpha^{2}\bar{u}+g^{2}\sigma_{d}^{2}+\sigma_{r}^{2})\right)$
$\displaystyle\>\>\>\;+\mathcal{N}\left(\right.0,\>-(p_{g}p_{u})^{2}(g^{2}\alpha^{2}\bar{u}+g^{2}\sigma_{d}^{2}+\sigma_{r}^{2})$
(10)
$\displaystyle\>\>\>\>\;\>\>\;\>\>\>\>\>\>\;\>\>\>\>\>\>\>+(p_{g}g)^{2}\alpha^{2}(p_{u}\bar{u})+(p_{g}g)^{2}\sigma_{d}^{2}+\sigma_{r}^{2}\left.\right).$

By inserting $\mu_{x}=g\alpha\bar{u}$ and original pixel value,
$x_{pre}\sim\mathcal{N}\left(\mu_{x},g\alpha\mu_{x}+g^{2}\sigma_{d}^{2}+\sigma_{r}^{2}\right)$,
it can be expressed with a pixel value-based equation as follows;

$\displaystyle x_{new}$ $\displaystyle\sim p_{u}p_{g}x_{pre}+$
$\displaystyle\>\>\>\;\mathcal{N}(0,\>p_{u}(1-p_{u})p_{g}^{2}g\alpha\mu_{x}$
(11)
$\displaystyle\>\;\>\>\;\>\>\>\>\>\>\;\>\>+(1-p_{u}^{2})p_{g}^{2}g^{2}\sigma_{d}^{2}+(1-p_{u}^{2}p_{g}^{2})\sigma_{r}^{2}).$

Because the expected original pixel value $\mu_{x}$ in the Gaussian term is
impossible to obtain, we approximate it as $\mu_{x}=x_{pre}$. Based on this
equation, we can precisely simulate as if exposure time, light intensity, and
analog gain were $p_{e}$, $p_{i}$, and $p_{g}$ times. Then, let’s come back to
contrast, brightness, and hue augmentation. When contrast is multiplied by
$p_{c}$ and brightness is changed by $p_{b}$, it can be expressed as,

$x_{new}=p_{c}x_{pre}+p_{b}.$ (12)

This function is represented as multiplication by
$\frac{\left(p_{c}x_{pre}+p_{b}\right)}{x_{pre}}$. Therefore, noise-accounted
contrast and brightness augmentation is finally defined as,

$\begin{cases}random\>p_{c},\>p_{b}\\\
random\>p_{u},\>p_{g}\>(where\>p_{u}p_{g}=\frac{\left(p_{c}x_{pre}+p_{b}\right)}{x_{pre}},\>p_{u},p_{g}>0)\\\
Eq.(\ref{eq:11})\>(\mu_{x}\leftarrow x_{pre})\end{cases}.$ (13)

We can also convert hue by changing $p_{c}$ and $p_{b}$ per color filter
position in the RAW bayer.

#####  3.3.2 Blur Augmentation

Next, we introduce noise-accounted blur augmentation. Usual blur augmentation
makes noise smaller than real blur because the noise $n_{d}$ and $n_{r}$ are
smoothed although their noise amounts in reality are not related to how fast
you shake a camera or object movements. Only the photon number-related noise
is smoothed in real motion blur. The blurred pixel can be expressed as,

$x_{new}\sim\mathcal{N}\left(g\alpha\sum_{k}w_{k}\bar{u_{k}},\>g^{2}\alpha^{2}\sum_{k}w_{k}\bar{u_{k}}+g^{2}\sigma_{d}^{2}+\sigma_{r}^{2}\right),$
(14)

where $\sum_{k}w_{k}=1$ is the blur kernel. With similar equation manipulation
from Eq. (9) to Eq. (11) , noise-accounted blur augmentation is

$\displaystyle x_{new}$
$\displaystyle\sim\mathcal{N}\left(\sum_{k}w_{k}g\alpha\bar{u_{k}},\>\sum_{k}w_{k}^{2}(g^{2}\alpha^{2}\bar{u_{k}}+g^{2}\sigma_{d}^{2}+\sigma_{r}^{2})\right)$
$\displaystyle\>\;\>\>+\mathcal{N}(0,\>-\sum_{k}w_{k}^{2}(g^{2}\alpha^{2}\bar{u_{k}}+g^{2}\sigma_{d}^{2}+\sigma_{r}^{2})$
$\displaystyle\>\;\>\>\;\>\>\>\>\>\>\;\>\>\;\>\>\>\>+g^{2}\alpha^{2}\sum_{k}w_{k}\bar{u_{k}}+g^{2}\sigma_{d}^{2}+\sigma_{r}^{2})$
(15)
$\displaystyle=\sum_{k}w_{k}x_{pre}+\mathcal{N}(0,\>g\alpha\sum_{k}(1-w_{k})w_{k}x_{pre,k}$
$\displaystyle\>\;\>\;\>\>\;\>\>\>\>\>\>\;\>\>\;\>\>\>\>\>\>\;\>\>\;\>\>\;\>\>\>\;\>\>\;\>\>+(1-\sum_{k}w_{k}^{2})(g^{2}\sigma_{d}^{2}+\sigma_{r}^{2})).$

We account for prior noise but not for prior blur amounts because most of
images in usual datasets are not blurred very much. Furthermore, estimating
the prior blur amount is difficult.

In addition, please note that augmentation to make images clean (make it
brighter and deblurred) is inevitably difficult with these noise-accounted RAW
image augmentation. Clipping noise variance in Eq. (11) to zero forcibly
enables brightening but brightening too much causes a mismatch of noise
domain.

###  4 Evaluation

####  4.1 Dataset

Figure 3: Some examples of our introduced dataset. The upper row shows the
training dataset collected in simple environment while the lower row shows the
challenging test dataset collected in various environments from dark to
birght, HDR, and with or without handshake blur.

Although our method is applicable to any computer vision task, we chose a
human detection task as a target because of its wide usage. we prepared a RAW
image dataset for human detection task captured with a internally developed
sensor. As mentioned earlier, our objective is to train image recognition
models that work in various environments despite only using a training dataset
in simple environments. So, most of training images were taken under normal
light conditions with fixed camera position in several environments. Note that
moderately dark and HDR images are also included to some extent in the
training dataset. The analog gain was set to 6dB for outdoor, 12dB for
indoors, and 32db for moderately dark nights to generate realistic easy images
without auto-exposure. On the other hand, the test images were taken under HDR
or extremely dark environment. In addition, about 50% of them were taken with
strong camera shake. Moreover, the analog gain was chosen from 3dB, 6dB, 12dB,
and 24dB regardless of the environment. Both of the datasets were taken with
around 1 fps to increase diversity between images.

We manually annotated the human bounding boxes of both training and test data.
Because precise annotation of test data on sRGB was impossible due to the
noise and blur, we applied an offline ISP per image and then annotated the
bounding boxes. We manually set adequate ISP parameters per image and had to
change the parameters several times to grasp the entire image. To avoid
annotating large training dataset like these, it is desirable to train the
model with a simple dataset. In total, we collected 18,880 images for the
training and 2,800 images for the test. The examples are shown in Fig. 3.

####  4.2 Implementation Details

We mainly tested with TTFNet [27] whose backbone was ResNet18 [14]. The
network was trained for 48 epochs from scratch using Adam optimizer [22] and a
cosine decay learning rate scheduler with a linear warmup [28] for the first
1,000 iterations whose maximum and minimum learning rates were 1e-3 and 1e-4.
We implemented a simple software ISP consisting of only a gamma tone mapping.
We implemented two types of gamma tone mapping. One was the simplest gamma
tone mapping, $y=x^{\frac{1}{\gamma}}\>(0\leq x\leq 1)$. The $\gamma$ was set
to 5 after tuning with a rough grid search manner. The other was a gamma tone
mapping parameterized with three parameters [34]. Because the grid search for
three parameters is time-consuming, we tuned the parameters with
backpropagation together with the detector’s weights as performed in [35, 49].
We did not use other ISP functions because they were known to have less impact
on image recognition compared with tone mapping [13]. We also prepared an
elaborated black-box ISP consisting of many functions in addition to a tone
mapping function. The parameters were tuned for human perceptual quality by
experts. We only used the elaborated black-box ISP under the conventional
training pipeline. In other words, we experimented under ISP-augmentation-
detection order, due to the hardware limitation. If contrast augmentation was
used, the hue was also changed with a probability of 50%. In detail, the
contrast factor per color channel $p_{c,c}$ was randomized from the base
contrast factor $p_{c,base}$ by (-0.2$p_{c,base}$, 0.2$p_{c,base}$) after
$p_{c,base}$ was randomly decided. If blur augmentation was used, we applied a
random-sized blur kernel with a probability of 50%. Random shift and random
scale augmentation whose maximum transformations were 10% and 3% of the input
size were also applied with the probability of 80% before color jitter
augmentation. The input size to the detector was $\left(576,\,352,\,3\right)$.
We evaluated the performance of the detector with average precision
(AP@0.5:0.95) [24].

####  4.3 Calibration of the Noise Model

For each analog gain of 6dB, 12dB, and 24dB, we captured two burst seqences
with different illumination. Each sequence consisted of 100 images. $24\times
24$ Bayer pixels were sampled from each of the 24 color patches to calculate
the mean and variance. In total $2\times 24\times 24\times 24$ pairs of mean
and variance sets were obtained per analog gain to estimate noise model.
Thanks to the various color fillters, exposure values, and color patches, two
sequences was enough to ensure diversity. The lines in Fig. 4 shows the
estimated linear relationship of Eq. (6). The coefficients of determination,
$R^{2}$, for these line estimation were 0.9833, 0.9884, 0.9862 for 6, 12, 24dB
respectively. A high $R^{2}$ value indicated that the noise intensity was well
modeled against illumination intensity. Also, the $R^{2}$ of the Eq. (7) and
Eq. (8) were $1.0000$ and $0.9984$. It means the noise intensity was well
modeled against analog gain. Based on the above, our noise model and the
calibration method were well suited to the sensor in terms of noise intensity.

mean (expected) pixel value
---

noise variance
---

Figure 4: The calibration result of the sensor noise model. The dots in the
left graph are mean and variance pairs of each pixel and the lines are the
estimated linear relationship per analog gain. Since we plot a large numbers
of dots, they seem to be widely spread. In the other hand, the histogram plot
of per analog gain (right) shows a clear difference between them. (a) Shapiro-
Wilk test
---
expected pixel value
---
pixel value
---
(b) pixel value distributions of small p-value cases
---

p-value
---

the number of pixels
---

Figure 5: The result of Shapiro-Wilk [40] test per expected pixel value. We
tested whether the 100 pixel values of each position fall under Gaussian
distribution and most of the p-values were larger than 0.05 (a). The right (b)
shows the cases where p < 0.05. It indicates the small p-values came from the
sparsity not the skew.

Then, we checked the validity of the shape of the distribution. All of the
noises were assumed to follow a Gaussian distribution. Especially, it is
unclear whether the approximation
$\mathcal{P}\left(\bar{u}\right)\fallingdotseq\mathcal{N}\left(\bar{u},\bar{u}\right)$
[9] is true. Therefore, the Shapiro-Wilk test [40] was performed. If the
p-value of the test is higher than 0.05, it indicates the null hypothesis that
the data are normally distributed cannot be rejected with more than a 95%
confidence interval. Fig. 5(a) shows that most of them were higher than 0.05,
but some results for dark pixels were less than 0.05. However, the
distributions of the dark pixels were like Fig. 5(b). It was not very skewed
and the sparsity causes the small p-value. Thereby, we concluded that all the
noise sources can be regarded as Gaussian noise.

Based on the above, our sensor noise model and the calibration method
represent the sensor noise well in terms of both intensity and distribution.

####  4.4 Statistical Validation of the Noise Alignment

Before checking the effectiveness of the proposed noise-accounted RAW
augmentation to computer vision applications, the statistical validity was
evaluated one more time by utilizing the sequential images of the color
checker. The evaluation method was as follows. First, the contrast of the
sequential images was changed with or without noise consideration. Second,
mean and variance pairs along the sequential dimension were calculated. Third,
the distributions of real and converted pairs were examined. If the real and
converted pairs matched well, it means the converted images have the same
noise amount as the original real images.

We then compared contrast conversion method utilizing three different noise
alignment methods, i.e., no noise consideration, usual noise-accounted method
that disregards prior noise in the input [15, 4, 44, 47], and our proposed
noise alignment method. Fig. 6 shows the comparison result. It indicates that
prior noise consideration is unnecessary if pixels are darkened considerably.
However, a small contrast factor caused a noise domain gap even if the input
was bright like Fig. 6 (right). In contrast, our noise-accounted conversion
always converted images with realistic noise. It implies that our proposed
method is suited for various strengths of augmentation. If prior noise is not
accounted, the inputs always have to be darkened considerably and already dark
images are difficult to use.

variance
---

real data
---
w/o noise-accounted
---
w/o prior-noise-accounted
---
w/ prior-noise-accounted (ours)
---
real data
---
w/o noise-accounted
---
w/o prior-noise-accounted
---
w/ prior-noise-accounted (ours)
---
mean pixel value
---
mean pixel value
---
(a) dark-to-extreme-dark conversion
---
(b) bright-dark conversion
---
Figure 6: The statistical validation of the noise alignment. The green dots
represents real image data and the others are converted from it. The
conversions are ×0.1 and ×0.5 contrast conversion with several methods. The
(a) is from dark-to-extreme-dark conversion and the (b) is bright-to-dark
conversion.

####  4.5 Augmentation Parameters Tuning

The optimal augmentation parameters should be different between augmentation
before and after ISP. To make a fair comparison, we roughly tuned both of the
augmentation parameters. The strategy was as follows. First, we searched the
appropriate range of contrast factor $p_{c}$ and brightness perturbation
$p_{b}$ successively. To be robust to any illumination changes, we re-
parameterized $p_{b}$ as $p_{b}=\hat{p_{b}}min(x)$ and randomized the
$\hat{p_{b}}$ instead of $p_{b}$. Lastly, we searched for the appropriate max
blur distance $p_{d}$. In these experiments, we do not account for sensor
noise. We used the elaborated black-box ISP for the augmentation after ISP
setting and the simplest gamma function for the augmentation before ISP
setting.

The results were shown by Table 1. The best parameter settings were used in
the next section.

Table 1: The augmentation hyperparameter tuning for a fair comparison between
before and after ISP augmentation. We tuned the range of contrast, brightness,
and blur distance one by one, and the previous best parameters were taken
over.

| AP@0.5:0.95 [%]
---|---
| augmentation after ISP (tuned for the black-box ISP) | augmentation before ISP (tuned for the simplest ISP)
contrast | 0.2-1 | 0.1-1 | 0.2-5 | 0.1-10 | 0.05-20 | 0.1-1 | 0.02-1 | 0.01-1 | 0.005-1 | 0.01-1.1
41.9 | 41.8 | 43.2 | 45.1 | 44.5 | 36.9 | 39.8 | 40.4 | 35.8 | 38.9
brightness | 0-0 | -0.1-0.1 | -0.2-0.2 | -0.5-0.5 | -0.7-0.7 | 0-0 | -0.1-0.1 | -0.2-0.2 | -0.5-0.5 | -0.7-0.7
45.1 | 44.5 | 44.5 | 45.2 | 44.7 | 40.4 | 40.9 | 39.9 | 39.4 | 39.4
blur distance | 0-0 | 0-3 | 0-5 | 0-9 | 0-13 | 0-0 | 0-3 | 0-5 | 0-9 | 0-13
45.2 | 46.8 | 45.7 | 45.9 | 37.9 | 40.9 | 39.8 | 39.6 | 40.6 | 43.3

####  4.6 Evaluation of the Noise-Accounted RAW Augmentation

In this section, the proposed noise alignment was also applied. As Table 2
shows, noise alignment for both color jitter and blur augmentation improves
the accuracy against difficult test data. It suggests the noise model
decreased the noise domain gap well and the noise domain is an important
factor in the DNN detector. In the color jitter augmentation only setting, we
improved the accuracy from the general noise alignment method [9, 31, 15, 4,
26, 47, 36, 50, 2] by considering prior noise. In the color jitter and blur
augmentation setting, noise-accounted color jitter augmentation plus normal
blur augmentation did not improve much from no noise-accounted settings.
Instead, noise alignment in both color jitter and blur augmentation improved
the accuracy. It indicates that random noise is not effective and realistic
noise is important. If we compared the accuracy under the same simplest gamma
tone mapping setting, our proposed noise-accounted RAW image augmentation
doubled the accuracy of conventional augmentation after ISP. Furthermore, when
parameterized gamma tone mapping was used as our simple ISP, the accuracy was
even superior to the elaborated black-box ISP consisting of many functions in
addition to a tone mapping function. As the visualization results in Appendix
shows, the elaborated black-box ISP output more perceivable images. It
suggests that minimizing the domain gap caused by augmentation is more
important than the superiority of the ISP. We might improve the accuracy more
by an elaborated ISP and the proposed augmentation.

Table 2: Evaluation of the noise-accounted RAW augmentation. The color
augmentation contains default hue augmentation plus tuned contrast and
brightness augmentation. The _w/o prior_ means the prior input noise was
disregarded like many of the previous noise-accounted image conversion methods
[9, 31, 15, 4, 26, 47, 36, 50, 2]. Because we adopt the well-established
heteroscedastic Gaussian model Eq. (2), it is identical to the noise alignment
of [2, 36, 9, 50]. In this experiment, we also used the parameterized gamma
tone mapping as the simple ISP, although it can’t be used in the augmentation
after ISP settings because the gradient from the detection loss is needed to
tune.

|  | mAP@0.5:0.95 [%]
---|---|---
|  |  | black-box | simple ISP
augmentation | noise | ISP | simplest | parameterized
Color | after | - | 45.2 | 19.3 | -
before (ours) | - | - | 40.9 | 44.4
w/o prior | - | 43.5 | 47.7
ours | - | 44.6 | 48.1
Color + Blur | after | - | 46.8 | 20.4 | -
before (ours) | - | - | 43.3 | 43.8
ours$\dagger$ | - | 43.4 | 47.9
ours | - | 45.3 | 48.3

$\dagger$: The noise alignment was only applied to the color jitter
augmentation.

As mentioned earlier, there were noise dealing works in noise-related fields
like denoising. We compared ours with these methods in the detection task. One
was K-Sigma transform [44], a kind of noise domain generalization. It
normalized images as the pixel value and the standard deviation of noise have
a linear correlation. The other was noise amount notification with a
concatenation of noise variance map [2]. To obey the previous settings, direct
RAW input without an ISP was also compared. Color jitter and blur augmentation
were also applied to the methods different from the previous papers for a fair
comparison. Table 3 shows the comparison results. As to the K-Sigma transform,
simply applying "aug." before or after the K-Sigma transforms gives better
result. However, there is a theoretical problem in both cases. If “aug.” is
applied after the transform, the linear relation between pixel value and noise
amount is retained but pixel intensity becomes inconsistent. On the other
hand, applying “aug.” before the transform makes the noise amount unrealistic.
Changing the augmentation to “our aug.” makes the intensity and noise
realistic and improved the accuracy. From the experiment, we found the
proposed augmentation boosts previous noise-dealing methods if ISP was used.
However, if ISP was not used, noise accounted augmentation slightly
deteriorated the accuracy. We argue that it is because the intensity
distribution was too difficult and unrealistic clean images might help
training. However, the overall accuracy was lower than with ISP. Also, for the
detection task, only our proposed method was enough. It might be an indication
that, unlike the denoising task which should focus on noise, it is important
to make the detector to focus on pixel intensity distribution for the
detection task.

Table 3: The comparison results with other noise-dealing techniques. The “aug.” means contrast and blur augmentation without noise-accounted and “our aug.” means with noise-accounted. We used the simplest gamma function in the ISP. | AP@0.5:0.95 [%]
---|---
method | w/o ISP | w/ ISP
concat [2] | 16.5 | 21.5
aug. + concat [2] | 35.0 | 31.6
our aug. + concat [2] | 33.7 | 40.4
K-Sigma [44] | 14.3 | 27.5
K-Sigma [44] \+ aug. | 25.0 | 34.1
aug. + K-Sigma [44] | 26.6 | 42.1
our aug. + K-Sigma [44] | 26.3 | 44.0
our aug. | 32.8 | 45.3

###  5 Conclusion

We propose a noise-accounted RAW augmentation method in which augmentation was
applied before ISP to minimize the luminance domain gap and a sensor noise
model was taken into account to minimize the noise domain gap. Unlike previous
noise-accounted methods, ours takes the prior input noise into account. It
minimizes the domain gap more and enables the use of already noisy images as
training data. Thanks to the realistic augmentation, our method improved the
detection accuracy in difficult scenes compared to the conventional methods.
In the future, we would like to investigate whether the proposed augmentation
with an elaborate ISP improves the computer vision performance even further.
We are glad if this work shed light on the importance of RAW images.

### References

  * [1] Goutam Bhat, Martin Danelljan, Fisher Yu, Luc Van Gool, and Radu Timofte.  Deep reparametrization of multi-frame super-resolution and denoising.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2460–2470, 2021.
  * [2] Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, and Jonathan T Barron.  Unprocessing images for learned raw denoising.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11036–11045, 2019.
  * [3] Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun.  Learning to see in the dark.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3291–3300, 2018.
  * [4] Ziteng Cui, Guo-Jun Qi, Lin Gu, Shaodi You, Zenghui Zhang, and Tatsuya Harada.  Multitask aet with orthogonal tangent regularity for dark object detection.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2553–2562, 2021.
  * [5] Steven Diamond, Vincent Sitzmann, Frank Julca-Aguilar, Stephen Boyd, Gordon Wetzstein, and Felix Heide.  Dirty pixels: Towards end-to-end image processing and perception.  ACM Transactions on Graphics (TOG), 40(3):1–15, 2021.
  * [6] Akshay Dudhane, Syed Waqas Zamir, Salman Khan, Fahad Shahbaz Khan, and Ming-Hsuan Yang.  Burst image restoration and enhancement.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5759–5768, 2022.
  * [7] Abbas El Gamal, Boyd A Fowler, Hao Min, and Xinqiao Liu.  Modeling and estimation of fpn components in cmos image sensors.  In Solid State Sensor Arrays: Development and Applications II, volume 3301, pages 168–177. SPIE, 1998.
  * [8] Martin A Fischler and Robert C Bolles.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.  Communications of the ACM, 24(6):381–395, 1981.
  * [9] Alessandro Foi, Mejdi Trimeche, Vladimir Katkovnik, and Karen Egiazarian.  Practical poissonian-gaussian noise modeling and fitting for single-image raw-data.  IEEE Transactions on Image Processing, 17(10):1737–1754, 2008.
  * [10] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial nets.  Advances in neural information processing systems, 27, 2014.
  * [11] Ryan D Gow, David Renshaw, Keith Findlater, Lindsay Grant, Stuart J McLeod, John Hart, and Robert L Nicol.  A comprehensive tool for modeling cmos image-sensor-noise performance.  IEEE Transactions on Electron Devices, 54(6):1321–1329, 2007.
  * [12] Chunle Guo, Chongyi Li, Jichang Guo, Chen Change Loy, Junhui Hou, Sam Kwong, and Runmin Cong.  Zero-reference deep curve estimation for low-light image enhancement.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1780–1789, 2020.
  * [13] Patrick Hansen, Alexey Vilkin, Yury Krustalev, James Imber, Dumidu Talagala, David Hanwell, Matthew Mattina, and Paul N Whatmough.  Isp4ml: The role of image signal processing in efficient deep learning vision systems.  In 2020 25th International Conference on Pattern Recognition (ICPR), pages 2438–2445. IEEE, 2021.
  * [14] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [15] Yang Hong, Kaixuan Wei, Linwei Chen, and Ying Fu.  Crafting object detection in very low light.  In Proceedings of the British Machine Vision Virtual Conference, 2021.
  * [16] Andrey Ignatov, Nikolay Kobyshev, Radu Timofte, Kenneth Vanhoey, and Luc Van Gool.  Dslr-quality photos on mobile devices with deep convolutional networks.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3277–3285, 2017.
  * [17] Andrey Ignatov, Luc Van Gool, and Radu Timofte.  Replacing mobile camera isp with a single deep learning model.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 536–537, 2020.
  * [18] Tomas Jenicek and Ondrej Chum.  No fear of the dark: Image retrieval under varying illumination conditions.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9696–9704, 2019.
  * [19] Haiyang Jiang and Yinqiang Zheng.  Learning to see moving objects in the dark.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7324–7333, 2019.
  * [20] Yifan Jiang, Xinyu Gong, Ding Liu, Yu Cheng, Chen Fang, Xiaohui Shen, Jianchao Yang, Pan Zhou, and Zhangyang Wang.  Enlightengan: Deep light enhancement without paired supervision.  IEEE Transactions on Image Processing, 30:2340–2349, 2021.
  * [21] Brian L Joiner and Joan R Rosenblatt.  Some properties of the range in samples from tukey’s symmetric lambda distributions.  Journal of the American Statistical Association, 66(334):394–399, 1971.
  * [22] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [23] Mikhail Konnik and James Welsh.  High-level numerical simulations of noise in ccd and cmos photosensors: review and tutorial.  arXiv preprint arXiv:1412.4031, 2014.
  * [24] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [25] Shuai Liu, Chaoyu Feng, Xiaotao Wang, Hao Wang, Ran Zhu, Yongqiang Li, and Lei Lei.  Deep-flexisp: A three-stage framework for night photography rendering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1211–1220, 2022.
  * [26] Xinhao Liu, Masayuki Tanaka, and Masatoshi Okutomi.  Practical signal-dependent noise parameter estimation from a single noisy image.  IEEE Transactions on Image Processing, 23(10):4361–4371, 2014.
  * [27] Zili Liu, Tu Zheng, Guodong Xu, Zheng Yang, Haifeng Liu, and Deng Cai.  Training-time-friendly network for real-time object detection.  In proceedings of the AAAI conference on artificial intelligence, volume 34, pages 11685–11692, 2020.
  * [28] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  arXiv preprint arXiv:1608.03983, 2016.
  * [29] Feifan Lv, Feng Lu, Jianhua Wu, and Chongsoon Lim.  Mbllen: Low-light image/video enhancement using cnns.  In BMVC, volume 220, page 4, 2018.
  * [30] Long Ma, Tengyu Ma, Risheng Liu, Xin Fan, and Zhongxuan Luo.  Toward fast, flexible, and robust low-light image enhancement.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5637–5646, 2022.
  * [31] Markku Makitalo and Alessandro Foi.  Optimal inversion of the generalized anscombe transformation for poisson-gaussian noise.  IEEE transactions on image processing, 22(1):91–103, 2012.
  * [32] Kristina Monakhova, Stephan R Richter, Laura Waller, and Vladlen Koltun.  Dancing under the stars: video denoising in starlight.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16241–16251, 2022.
  * [33] Igor Morawski, Yu-An Chen, Yu-Sheng Lin, Shusil Dangi, Kai He, and Winston H Hsu.  Genisp: Neural isp for low-light machine cognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 630–639, 2022.
  * [34] Ali Mosleh, Avinash Sharma, Emmanuel Onzon, Fahim Mannan, Nicolas Robidoux, and Felix Heide.  Hardware-in-the-loop end-to-end optimization of camera image processing pipelines.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7529–7538, 2020.
  * [35] Emmanuel Onzon, Fahim Mannan, and Felix Heide.  Neural auto-exposure for high-dynamic range object detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7710–7720, 2021.
  * [36] Abhijith Punnappurath, Abdullah Abuolaim, Abdelrahman Abdelhamed, Alex Levinshtein, and Michael S Brown.  Day-to-night image synthesis for training nighttime neural isps.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10769–10778, 2022.
  * [37] Erik Reinhard, Wolfgang Heidrich, Paul Debevec, Sumanta Pattanaik, Greg Ward, and Karol Myszkowski.  High dynamic range imaging: acquisition, display, and image-based lighting.  Morgan Kaufmann, 2010.
  * [38] Yukihiro Sasagawa and Hajime Nagahara.  Yolo in the dark-domain adaptation method for merging multiple models.  In European Conference on Computer Vision, pages 345–359. Springer, 2020.
  * [39] Eli Schwartz, Alex Bronstein, and Raja Giryes.  Isp distillation.  arXiv preprint arXiv:2101.10203, 2021.
  * [40] Samuel S Shapiro and RS Francia.  An approximate analysis of variance test for normality.  Journal of the American statistical Association, 67(337):215–216, 1972.
  * [41] Sungho Suh, Shinya Itoh, Satoshi Aoyama, and Shoji Kawahito.  Column-parallel correlated multiple sampling circuits for cmos image sensors and their noise reduction effects.  Sensors, 10(10):9139–9154, 2010.
  * [42] Justin Tomasi, Brandon Wagstaff, Steven L Waslander, and Jonathan Kelly.  Learned camera gain and exposure control for improved visual feature detection and matching.  IEEE Robotics and Automation Letters, 6(2):2028–2035, 2021.
  * [43] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li.  Maxim: Multi-axis mlp for image processing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5769–5780, 2022.
  * [44] Yuzhi Wang, Haibin Huang, Qin Xu, Jiaming Liu, Yiqun Liu, and Jue Wang.  Practical deep raw image denoising on mobile devices.  In European Conference on Computer Vision, pages 1–16. Springer, 2020.
  * [45] Zhixiang Wang, Xiang Ji, Jia-Bin Huang, Shin’ichi Satoh, Xiao Zhou, and Yinqiang Zheng.  Neural global shutter: Learn to restore video from a rolling shutter camera with global reset feature.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17794–17803, 2022.
  * [46] Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu.  Deep retinex decomposition for low-light enhancement.  arXiv preprint arXiv:1808.04560, 2018.
  * [47] Kaixuan Wei, Ying Fu, Jiaolong Yang, and Hua Huang.  A physics-based noise formation model for extreme low-light raw denoising.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2758–2767, 2020.
  * [48] Jay Whang, Mauricio Delbracio, Hossein Talebi, Chitwan Saharia, Alexandros G Dimakis, and Peyman Milanfar.  Deblurring via stochastic refinement.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16293–16303, 2022.
  * [49] Chyuan-Tyng Wu, Leo F Isikdogan, Sushma Rao, Bhavin Nayak, Timo Gerasimow, Aleksandar Sutic, Liron Ain-kedem, and Gilad Michael.  Visionisp: Repurposing the image signal processor for computer vision applications.  In 2019 IEEE International Conference on Image Processing (ICIP), pages 4624–4628. IEEE, 2019.
  * [50] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, Ming-Hsuan Yang, and Ling Shao.  Cycleisp: Real image restoration via improved data synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2696–2705, 2020.
  * [51] Hongguang Zhang, Yuchao Dai, Hongdong Li, and Piotr Koniusz.  Deep stacked hierarchical multi-patch network for image deblurring.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5978–5986, 2019.
  * [52] Meina Zhang, Yingying Fang, Guoxi Ni, and Tieyong Zeng.  Pixel screening based intermediate correction for blind deblurring.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5892–5900, 2022.
  * [53] Yi Zhang, Dasong Li, Ka Lung Law, Xiaogang Wang, Hongwei Qin, and Hongsheng Li.  Idr: Self-supervised image denoising via iterative data refinement.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2098–2107, 2022.
  * [54] Yonghua Zhang, Jiawan Zhang, and Xiaojie Guo.  Kindling the darkness: A practical low-light image enhancer.  In Proceedings of the 27th ACM international conference on multimedia, pages 1632–1640, 2019.

##  Part I Visualization Results

In Fig. 7, the detection results are drawn on the corresponding ISP output.
The proposed method shows a significant improvement in accuracy under the
condition that the simplest gamma tone mapping is used as ISP. In addition,
the accuracy of the proposed method is the best despite the use of the simple
ISP with limited visibility against a rich black box ISP because of the
effective noise-accounted RAW augmentation.

parameterized gamma
---
before ISP (ours)
---
w/ (ours)
---
Black-box ISP
---
after ISP
---
w/o
---
simplest gamma
---
before ISP (ours)
---
w/ (ours)
---
simplest gamma
---
before ISP (ours)
---
w/o prior
---
simplest gamma
---
before ISP (ours)
---
w/o
---
simplest gamma
---
after ISP
---
w/o
---
ISP
---
augmentation order
---
noise-accounted
---
Figure 7: The visualization of the detection results. To make a fair
comparison, we set an adequate confidence threshold per model. Specifically,
we adjusted the threshold to achieve an<EMAIL_ADDRESS>value of 80%. The darker
bounding boxes represent the ground truth while the brighter ones represent
the prediction result.