Ziteng<EMAIL_ADDRESS>Kunchang<EMAIL_ADDRESS>Lin
Gulin.gu@riken.jp3,1* Shenghan<EMAIL_ADDRESS>Peng
<EMAIL_ADDRESS>Zhengkai<EMAIL_ADDRESS>Yu
<EMAIL_ADDRESS>Tatsuya Haradaharada@mi.t.u-tokyo.ac.jp1,3 The
University of Tokyo Shanghai AI Laboratory RIKEN AIP Shanghai Jiao Tong
University Tencent Youtu Lab BMVC Author Guidelines

# You Only Need 90K Parameters to Adapt Light: a Light Weight Transformer for
Image Enhancement and Exposure Correction

###### Abstract

Challenging illumination conditions (low light, under-exposure and over-
exposure) in the real world not only cast an unpleasant visual appearance but
also taint the computer vision tasks. After camera captures the raw-RGB data,
it renders standard sRGB images with image signal processor (ISP). By
decomposing ISP pipeline into local and global image components, we propose a
lightweight fast Illumination Adaptive Transformer (IAT) to restore the normal
lit sRGB image from either low-light or under/over-exposure conditions.
Specifically, IAT uses attention queries to represent and adjust the ISP-
related parameters such as colour correction, gamma correction. With only
$\sim$90k parameters and $\sim$0.004s processing speed, our IAT consistently
achieves superior performance over State-of-The-Art (SOTA) on the benchmark
low-light enhancement and exposure correction datasets. Competitive
experimental performance also demonstrates that our IAT significantly enhances
object detection and semantic segmentation tasks under various light
conditions. Our code and pre-trained model is available at this url.

##  1 Introduction

Computer vision has witnessed great success on well-taken images and videos.
However, the varying light conditions in the real world poses challenges on
both human visual appearance and downstream vision tasks (e.g., semantic
segmentation and object detection). Images taken under inadequate illumination
(Fig.1 left top) suffer from limited photon counts and undesirable in-camera
noise. On the other hand, outdoor scenes are often exposed to strong light
such as direct sunlight (Fig.1 left down), making image saturated due to the
limited range of sensors and non-linearity in the camera image pipeline. To
make it worse, both the under and over exposure may exist together, i.e.
spatial-variant illumination cast by shadow could make the contrast ratio to
be 1000:1 or higher.

Figure 1:  Nature lay hid in Night, apply our IAT, and all was light, middle
figure shows our results compared with other SOTA methods on LOL-V1 dataset
[Wei et al.(2018)Wei, Wang, Yang, and Liu].

Multiple techniques such as low-light enhancement [Land(1986), Tomasi and
Manduchi(1998), Guo et al.(2017)Guo, Li, and Ling, Lore et al.(2017)Lore,
Akintayo, and Sarkar, Lv et al.(2018)Lv, Lu, Wu, and Lim, Guo et al.(2020)Guo,
Li, Guo, Loy, Hou, Kwong, and Cong, Zhang et al.(2019)Zhang, Zhang, and Guo,
Kim et al.(2021)Kim, Choi, Kim, and Koh, Zeng et al.(2020)Zeng, Cai, Li, Cao,
and Zhang, Moran et al.(2020)Moran, Marza, McDonagh, Parisot, and Slabaugh, Tu
et al.(2022)Tu, Talebi, Zhang, Yang, Milanfar, Bovik, and Li, Sharma and
Tan(2021)], exposure correction [Nayar and Branzoi(2003), Yuan and Sun(2012),
Afifi et al.(2021)Afifi, Derpanis, Ommer, and Brown, Yang et al.(2022)Yang,
Cheng, Zhao, Zhang, and Li, Nsampi et al.(2021)Nsampi, Hu, and Wang] have been
proposed to adapt to the difficult light condition. Low-light enhancement
methods restore the details while suppressing the accompanying noises.
Exposure correction methods focus on adjusting the under/over exposure image
to reconstruct a clear image against short/long exposure time. While many
efforts elaborate on improving human oriented visual perceptual, there also
several methods enhance the high-level tasks by boosting their robustness
against low light [Cui et al.(2021)Cui, Qi, Gu, You, Zhang, and Harada, Liu et
al.(2021a)Liu, Xu, Yang, Fan, and Huang, Hong et al.(2021)Hong, Wei, Chen, and
Fu, Sasagawa and Nagahara(2020)] and over-exposure conditions [Minciullo et
al.(2021)Minciullo, Manhardt, Yoshikawa, Meier, Tombari, and Kobori]. As shown
in Fig.1, we aim for a unified lightweight framework that improves both the
visual appearance and consequent recognition tasks under challenging real-
world light condition.

While sRGB images are most common to everyday life, many existing light
adaptive methods operate on raw-RGB images which linearly proportion to the
actual scene irradiance. To directly process the sRGB images, we specifically
considers the image signal processor (ISP) in camera that renders sRGB from
raw-RGB image. We propose a novel two-branches transformer based model to
handle this issue, a pixel-wise local branch $f$ coupled with global ISP
branch $g$. In local branch $f$, we map the input image to latent feature
space and replace transformer’s attention block to depth-wise convolution for
light-weight design. In global branch $g$, we use transformer’s attention
queries to control and adjust the global ISP-related parameters (i.e. colour
transform matrix, gamma value). In addition, the learned queries could dynamic
change under different light condition at same time (i.e. over-exposure and
under exposure).

Extensive experiments are conducted on several real-world and synthetic
datasets, i.e., image enhancement dataset LOL [Wei et al.(2018)Wei, Wang,
Yang, and Liu] and photo retouching dataset MIT-Adobe FiveK [Bychkovsky et
al.(2011)Bychkovsky, Paris, Chan, and Durand], low-light detection dataset
EXDark [Loh and Chan(2019)] and low-light segmentation dataset ACDC [Sakaridis
et al.(2021)Sakaridis, Dai, and Van Gool]. Results show that our IAT achieve
state-of-the-art performance across a range of low-level and high-level tasks.
More importantly, our IAT model is of only 0.09M parameters, much smaller than
current SOTA transformer-based models [Chen et al.(2021a)Chen, Wang, Guo, Xu,
Deng, Liu, Ma, Xu, Xu, and Gao, Wang et al.(2022)Wang, Cun, Bao, Zhou, Liu,
and Li, Tu et al.(2022)Tu, Talebi, Zhang, Yang, Milanfar, Bovik, and Li].
Besides, our average inference speed is 0.004s per image, faster than the SOTA
methods taking around 1s per image.

Our contribution could be summarised as follow:

  * •

We have proposed a fast and light-weight framework, Illumination Adaptive
Transformer (IAT), to adapt to challenging light conditions in the real world,
which could both handle the low-light enhancement and exposure correction
tasks.

  * •

We have proposed a novel transformer-style structure to estimate ISP-related
parameters to fuse the target sRGB image, wherein the learnable attention
quires are utilised to attend the whole image, also we replace the layer norm
to a new light normalisation, for better handling low-level vision tasks.

  * •

Extensive experiments on several real-world datasets on 3 low-level tasks and
3 high-level tasks demonstrate the superior performance of IAT over SOTA
methods. IAT is light weight and mobile-friendly with only 0.09M model
parameters and 0.004s processing time per image. We will release the source
code upon publication.

##  2 Related Works

###  2.1 Enhancement against Challenging Light Condition

Earlier low-light image enhancement solutions mainly rely on RetiNex theory
[Land(1986)] or histogram equalization [Gonzalez and Woods(2006), Tomasi and
Manduchi(1998)]. Since LLNet [Lore et al.(2017)Lore, Akintayo, and Sarkar]
utilised a deep-autoencoder structure, CNN based methods [Lv et al.(2018)Lv,
Lu, Wu, and Lim, Zhang et al.(2019)Zhang, Zhang, and Guo, Kim et al.(2021)Kim,
Choi, Kim, and Koh, Guo et al.(2020)Guo, Li, Guo, Loy, Hou, Kwong, and Cong,
Moran et al.(2020)Moran, Marza, McDonagh, Parisot, and Slabaugh, Wang et
al.(2019)Wang, Zhang, Fu, Shen, Zheng, and Jia, Fu et al.(2022)Fu, Hong, Chen,
and You, Yang et al.(2022)Yang, Cheng, Zhao, Zhang, and Li, Xu et al.(2022)Xu,
Wang, Fu, and Jia, Sharma and Tan(2021)] have been widely used in this task
and gain SOTA results on the benchmark enhancement datasets [Bychkovsky et
al.(2011)Bychkovsky, Paris, Chan, and Durand, Wei et al.(2018)Wei, Wang, Yang,
and Liu].

Similar to low-light enhancement, traditional exposure correction algorithms
[Nayar and Branzoi(2003), Yuan and Sun(2012)] also use image histograms to
adjust image intensities. The strategy then tends to correct exposure errors
by adjusting the tone curve via a trained deep learning model [Yu et
al.(2018)Yu, Liu, Zhang, Qu, Zhao, and Zhang, Park et al.(2018)Park, Lee, Yoo,
and Kweon]. Very recently, Afifi et al. [Afifi et al.(2021)Afifi, Derpanis,
Ommer, and Brown] propose a coarse-to-fine neural network to correct photo
exposure, after that Nsampi et al. [Nsampi et al.(2021)Nsampi, Hu, and Wang]
introduce attention mechanism into this task.

Beyond low-level vision, low-light/ strong-light scenario also deteriorates
the performance of high level vision [Cui et al.(2021)Cui, Qi, Gu, You, Zhang,
and Harada, Hong et al.(2021)Hong, Wei, Chen, and Fu, Liu et al.(2021a)Liu,
Xu, Yang, Fan, and Huang, Zheng et al.(2020)Zheng, Zhang, and Lu, Minciullo et
al.(2021)Minciullo, Manhardt, Yoshikawa, Meier, Tombari, and Kobori, Sasagawa
and Nagahara(2020)]. Several methods based on data synthesis [Zheng et
al.(2020)Zheng, Zhang, and Lu], self-supervised learning [Cui et al.(2021)Cui,
Qi, Gu, You, Zhang, and Harada] and domain adaptation [Sasagawa and
Nagahara(2020)] have been proposed to support high level vision tasks under
challenging illumination conditions.

###  2.2 Vision Transformers

Transformer [Vaswani et al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones,
Gomez, Kaiser, and Polosukhin] was firstly proposed in NLP area to capture
long-range dependencies by global attention. ViT [Dosovitskiy et
al.(2021)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner,
Dehghani, Minderer, Heigold, Gelly, Uszkoreit, and Houlsby] made the first
attempt in vision task by splitting the image into tokens before sending into
transformer model. Since then, Transformer based models have gained superior
performances in many computer vision tasks, including image/video
classification [Liu et al.(2021b)Liu, Lin, Cao, Hu, Wei, Zhang, Lin, and Guo,
Li et al.(2022)Li, Wang, Zhang, Gao, Song, Liu, Li, and Qiao], object
detection [Carion et al.(2020)Carion, Massa, Synnaeve, Usunier, Kirillov, and
Zagoruyko, Zhang et al.(2022)Zhang, Qiu, Wang, Xu, Guo, Qiao, Gao, and Li],
semantic segmentation [Xie et al.(2021)Xie, Wang, Yu, Anandkumar, Alvarez, and
Luo], vision-language model [Radford et al.(2021)Radford, Kim, Hallacy,
Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, Krueger, and Sutskever,
Zhang et al.(2021)Zhang, Fang, Gao, Zhang, Li, Dai, Qiao, and Li] and so on.

In low-level vision area, transformer-based models has also made much progress
on several sub-directions, such as image super-resolution [Liang et
al.(2021)Liang, Cao, Sun, Zhang, Van Gool, and Timofte], image restoration
[Zamir et al.(2021)Zamir, Arora, Khan, Hayat, Khan, and Yang, Wang et
al.(2022)Wang, Cun, Bao, Zhou, Liu, and Li, Chen et al.(2021a)Chen, Wang, Guo,
Xu, Deng, Liu, Ma, Xu, Xu, and Gao], image colorization [Kumar et
al.(2021)Kumar, Weissenborn, and Kalchbrenner] and bad weather restoration
[Valanarasu et al.(2021)Valanarasu, Yasarla, and Patel]. Very recently, MAXIM
[Tu et al.(2022)Tu, Talebi, Zhang, Yang, Milanfar, Bovik, and Li] use MLP-
based model [Tolstikhin et al.(2021)Tolstikhin, Houlsby, Kolesnikov, Beyer,
Zhai, Unterthiner, Yung, Steiner, Keysers, Uszkoreit, Lucic, and Dosovitskiy]
in low-level vision area which also shows MLP’s potential on low-level vision
tasks. However, existing transformer $\&$ MLP models require much
computational cost (e.g. 115.63M for IPT [Chen et al.(2021a)Chen, Wang, Guo,
Xu, Deng, Liu, Ma, Xu, Xu, and Gao], 14.14M for MAXIM [Tu et al.(2022)Tu,
Talebi, Zhang, Yang, Milanfar, Bovik, and Li]), making it hard to implement on
mobile and edge devices. Extreme lightweight of our method (0.09M) is
particular important in low-level vision and computational photography.

##  3 Illumination Adaptive Transformer

###  3.1 Motivation

For a sRGB image $I_{i}$ taken from light condition $L_{i}$, the input photons
under light condition $L_{i}$ would project through the lens on capacitor
cluster, to pass by the in-camera process [Wei et al.(2020)Wei, Fu, Yang, and
Huang] and render with image signal processor (ISP) pipeline $G(\cdot)$
[Karaimer and Brown(2016), Brooks et al.(2019)Brooks, Mildenhall, Xue, Chen,
Sharlet, and Barron]. Our goal is to match input sRGB $I_{i}$ to the target
sRGB image $I_{t}$ (taken under light condition $L_{t}$). Existing deep-
learning based methods tend to build an end-to-end mapping between $I_{i}$ and
$I_{t}$ [Lore et al.(2017)Lore, Akintayo, and Sarkar, Lv et al.(2018)Lv, Lu,
Wu, and Lim, Afifi et al.(2021)Afifi, Derpanis, Ommer, and Brown] or estimate
some high-level representation to assist enhancement task (i.e. illumination
map [Wang et al.(2019)Wang, Zhang, Fu, Shen, Zheng, and Jia], colour transform
function [Kim et al.(2021)Kim, Choi, Kim, and Koh], 3D look-up table [Zeng et
al.(2020)Zeng, Cai, Li, Cao, and Zhang]). However, the actual lightness
degradation happens in raw-RGB space, and the processes in camera ISP involves
more elaborated non-linear operations such as white balance, colour space
transform, gamma correction, etc. Therefore, much of research conducts image
enhancement [Chen et al.(2018)Chen, Chen, Xu, and Koltun, Wei et al.(2020)Wei,
Fu, Yang, and Huang] directly on raw-RGB data rather than sRGB images.

To this end, Brooks et al. [Brooks et al.(2019)Brooks, Mildenhall, Xue, Chen,
Sharlet, and Barron] inverse each steps in ISP pipeline (i.e. gamma
correction, tone mapping, camera colour transformation) to transform input
sRGB image to "unprocessed" raw-RGB data. After that, Afifi and Brown [Afifi
and Brown(2020)] apply an encoder-decoder structure to edit the illumination
of sRGB image from input light $I_{i}$ to target light $I_{t}$ as following:

$I_{t}=G(F(I_{i})),$ (1)

where $F$ is an unknown reconstruction function maps $I_{i}$ to the
corresponding raw-RGB data $D=F(I_{i})$, and $G$ is camera rendering function
that transform $D$ back to target sRGB image $I_{t}$. Here [Afifi and
Brown(2020)] use the network encoder $f$ to represent $F$, before adding
several individual decoders $g_{t}$ upon encoder $f$. The function maps
$f(I_{i})$ to target $I_{t}$ illumination conditions is represented below:

$I_{t}=g_{t}(f(I_{i})),$ (2)

For the sake of lightweight network design, inspired by the DETR [Carion et
al.(2020)Carion, Massa, Synnaeve, Usunier, Kirillov, and Zagoruyko] which
controls different object proposals via transformer queries, here we use
different queries to control the ISP-related parameters in $g_{t}(\cdot)$.
This re-configures parameters to make the image $I_{i}$ adaptive to target
light condition $L_{t}$. In training stage, the queries is dynamically updated
in each iteration to match the target image $I_{t}$. Here we simplify the ISP
procedures [Brooks et al.(2019)Brooks, Mildenhall, Xue, Chen, Sharlet, and
Barron, Cui et al.(2021)Cui, Qi, Gu, You, Zhang, and Harada, Chen et
al.(2021b)Chen, Feng, Gao, Xu, and Chen] into the equation 3 below. The
simplification details could be found in the supplementary.

$g_{t}(\cdot)=(max(\sum_{c_{j}}W_{c_{i},c_{j}}(\cdot),\epsilon))^{\gamma},c_{i},c_{j}\in\\{r,g,b\\}.$
(3)

$W_{c_{i},c_{j}}$ is a $3\times 3$ joint colour transformation matrix,
considering the white balance and colour transform matrix. We adopt 9 queries
to control $W_{c_{i},c_{j}}$’s parameters. $\gamma$ denotes the gamma
correction parameter which we use a single query to control. $\epsilon$ is a
very small value to prevent numerical instability. Here we set
$\epsilon=1e^{-8}$ in our experiments.

For process $F$, we apply a pixel-wise least squares model $f$. Our $f$
consists of two individual branches to predict multiply map $M$ and add map
$A$. We then apply a least squares to process input sRGB image:
$f(I_{i})=I_{i}\odot M+A$. Here $M$ and $A$ has the same size with $I_{i}$ to
complete pixel-level multiplicative and additive adjustment. Finally, the
equation of our IAT model follows:

$I_{t}=(max(\sum_{c_{j}}W_{c_{i},c_{j}}(I_{i}\odot M+A)),0)^{\gamma}.$ (4)

The non-linear operations are decomposed into a local pixel-wise components
$f$ and a global ISP components $g$. Thus, we design two individual
transformer style branches: local adjustment branch and global ISP branch, to
estimate the local pixel-wise components and global ISP components
respectively.

Figure 2: Structure of our Illumination Adaptive Transformer (IAT), the black
line refers to the parameters generation while the yellow line refers to image
processing.

###  3.2 Model Structure

Given an input sRGB image $I_{i}\in\mathbb{R}^{H\times W\times 3}$ under light
condition $L_{i}$, where $H\times W$ denotes the size dimension and $3$
denotes the channel dimension ($\\{r,g,b\\}$). As shown in Fig.2, we propose
our Illumination Adaptive Transformer (IAT) to transfer the input RGB image
$I_{i}$ to a target RGB $I_{t}\in\mathbb{R}^{H\times W\times 3}$ under the
proper uniform light $L_{t}$.

Local Branch. In the local branch, we focus on estimating the local components
$M,A$ to correct the effect of illumination. Instead of adopting a U-Net
[Ronneberger et al.(2015)Ronneberger, Fischer, and Brox] style structure,
which downsamples the images first before upsampling, we aim to maintain the
input resolution through the local branch to preserve the informative details.
Therefore, we propose a transformer-style architecture for the local branch.
Compared to popular U-Net style structures [Lv et al.(2018)Lv, Lu, Wu, and
Lim, Afifi et al.(2021)Afifi, Derpanis, Ommer, and Brown], our structure could
deal with arbitrary resolution images without resizing them.

At first, we expand the channel dimension via a 3$\times$3 convolution and
pass them to two independent branches stacked by Pixel-wise Enhancement Module
(PEM). For the lightweight design in the local branch, we replace self-
attention with depth-wise convolution as suggested in the previous works [Han
et al.(2022)Han, Fan, Dai, Sun, Cheng, Liu, and Wang, Li et al.(2022)Li, Wang,
Zhang, Gao, Song, Liu, Li, and Qiao], depth-wise convolution could reduce
parameters and further save computation cost. As shown in Fig. 3 (a), our PEM
first encodes the position information by 3$\times$3 depth-wise convolution
before enhancing local details with PWConv-DWConv-PWConv. Finally, we adopt
two 1$\times$1 convolutions to enhance token representation individually.
Specially, we design new kind of normalisation names light normalisation, to
replace transformer’s Layer Normalisation [Ba et al.(2016)Ba, Kiros, and
Hinton]. As shown in Fig. 3 (a), light normalisation learns to scale $a$ and
bias $b$ via two learnable parameters before fusing the channels via the
learnable matrix. The matrix is initialised as an identity matrix. For better
convergence, we adopt Layer Scale [Touvron et al.(2021b)Touvron, Cord,
Sablayrolles, Synnaeve, and Jégou] which multiplies the features by a small
number $k_{1}$/$k_{2}$.

We stack 3 PEMs in each branch and then connect the output features with the
input features through element-wise addition. This skip connection [He et
al.(2016)He, Zhang, Ren, and Sun] helps maintain the original image details.
Finally, we decrease the channel dimension by a 3$\times$3 convolutions and
adopt ReLU/ Tanh function to generate the local components $M$/ $A$ in Eq. 4.

Global ISP Branch. Global ISP branch accounts for part of the ISP pipeline
[Heide and et.al(2014), Jiang et al.(2017)Jiang, Tian, Farrell, and Wandell,
Karaimer and Brown(2016), Brooks et al.(2019)Brooks, Mildenhall, Xue, Chen,
Sharlet, and Barron] (i.e. gamma correction, colour matrix transform, white
balance) when transferring the target RGB image $I_{t}$. Specifically, the
value of each pixel in the target image is determined by a global operation
defined in Eq.3.

Inspired by Detection Transformer DETR [Carion et al.(2020)Carion, Massa,
Synnaeve, Usunier, Kirillov, and Zagoruyko], we design global component
queries to decode and predict the $W,\gamma$ to generate sRGB image $I_{t}$.
This transformer structure allows capturing global interactions between
context and individual pixels. As shown in Fig. 2, we first stack two
convolutions as a lightweight encoder, which encodes the features in a high
dimension with lower resolution, on the one hand lower resolution would save
computational cost which contribute to the light-weight design, on the other
hand higher feature representation would be helpful to extract image’s global-
level features. Then the generated features are passed to the Global
Prediction Module (GPM), Fig. 3 (b) shows the detailed structure of GPM,
different from original DETR [Carion et al.(2020)Carion, Massa, Synnaeve,
Usunier, Kirillov, and Zagoruyko] model, our global component queries Q are
initialised as zeros without extra multi-head self-attention. Q is global
component learnable embedding that attends keys K and values V generated from
encoded features. The positional encoding for K and V is from a depth-wise
convolution, which is friendly with different input resolutions. After feed
forward network (FFN) [Dosovitskiy et al.(2021)Dosovitskiy, Beyer, Kolesnikov,
Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, Uszkoreit,
and Houlsby] with two linear layers, we add two extra parameters with special
initialisation to output colour matrix and gamma. This initialisation makes
sure the colour matrix is identity matrix $W$ and the gamma value $g$ is one
in the beginning, thus contributing to stable training.

Figure 3: Detailed structure of Pixel-wise Enhancement Module (PEM) and Global
Prediction Module (GPM).

##  4 Experiments

We evaluate our proposed IAT model on benchmark datasets and experimental
settings for both low-level and high-level vision tasks under different
illumination conditions. Three low-level vision tasks include: $(a)$ image
enhancement (LOL (V1 & V2-real) [Wei et al.(2018)Wei, Wang, Yang, and Liu]),
$(b)$ image enhancement (MIT-Adobe FiveK [Bychkovsky et al.(2011)Bychkovsky,
Paris, Chan, and Durand]), $(c)$ exposure correction [Afifi et al.(2021)Afifi,
Derpanis, Ommer, and Brown]. Three high-level visions tasks include: $(d)$
low-light object detection $(e)$ low-light semantic segmentation $(f)$
various-light object detection. The number of PEM number in local branch are
both set to 3, while the channel number in PEM is set to 16.

For all low-level vision experiments: $\left\\{(a),(b),(c)\right\\}$, the IAT
model are trained on a single GeForce RTX 3090 GPU with batch size 8. We use
Adam optimizer to train our IAT model while the initial learning rate and
weight decay are separately set to $2e^{-4}$ and $1e^{-4}$. A cosine learning
schedule has also been adopted to avoid over-fitting. For data augmentation,
horizontal and vertical flips have been used to acquire better results.

Figure 4: Results on enhancement dataset [Wei et al.(2018)Wei, Wang, Yang, and
Liu, Bychkovsky et al.(2011)Bychkovsky, Paris, Chan, and Durand]) and exposure
correction dataset [Afifi et al.(2021)Afifi, Derpanis, Ommer, and Brown].

###  4.1 Low-level Image Enhancement.

For $(a)$ and $(b)$ image enhancement task, we evaluate our IAT framework on
benchmark datasets: LOL (V1 & V2-real) [Wei et al.(2018)Wei, Wang, Yang, and
Liu] and MIT-Adobe FiveK [Bychkovsky et al.(2011)Bychkovsky, Paris, Chan, and
Durand].

LOL [Wei et al.(2018)Wei, Wang, Yang, and Liu] has two versions: LOL-V1
consists of 500 paired normal-light images and low-light images. 485 pairs are
used for training and the other 15 pairs are for testing. LOL-V2-real consists
of 789 paired normal-light images and low-light pairs. 689 pairs are used for
training and the other 100 pairs are for testing. The loss function between
input image $I_{i}$ and target image $I_{t}$ for LOL dataset training is a
mixed loss function [Valanarasu et al.(2021)Valanarasu, Yasarla, and Patel]
consisting of smooth L1 loss and VGG loss [Johnson et al.(2016)Johnson, Alahi,
and Fei-Fei]. In LOL-V1 training, the images are cropped into $256\times 256$
to train 200 epochs and then fine-tune on $600\times 400$ resolution for 100
epochs. In LOL-V2-real training, the image resolution is maintained at
$600\times 400$ and trained for 200 epochs. Both LOL-V1 and LOL-V2-real
testing the image resolution is maintained at $600\times 400$. We compare our
method with SOTA methods [Guo et al.(2017)Guo, Li, and Ling, Wei et
al.(2018)Wei, Wang, Yang, and Liu, Lv et al.(2018)Lv, Lu, Wu, and Lim, Zhang
et al.(2019)Zhang, Zhang, and Guo, Guo et al.(2020)Guo, Li, Guo, Loy, Hou,
Kwong, and Cong, Yang et al.(2020)Yang, Wang, Fang, Wang, and Liu, Kim et
al.(2021)Kim, Choi, Kim, and Koh, Chen et al.(2021a)Chen, Wang, Guo, Xu, Deng,
Liu, Ma, Xu, Xu, and Gao, Zeng et al.(2020)Zeng, Cai, Li, Cao, and Zhang, Wang
et al.(2022)Wang, Cun, Bao, Zhou, Liu, and Li, Tu et al.(2022)Tu, Talebi,
Zhang, Yang, Milanfar, Bovik, and Li]. For image quality analysis, we evaluate
the peak signal-to-noise ratio (PSNR) and structural similarity index measure
(SSIM). For the model efficiency analyze, we report three metrics: FLOPs,
model parameters and test time, as shown in the last column of Table.1. We
list different model’s test time on their corresponding code platform (M means
Matlab, T means TensorFlow, P means PyTorch). As shown in Table 1, IAT (local)
only uses the local network to train the model and IAT refers to the whole
framework. Our IAT gains SOTA result on both image quality and model
efficiency, especially less than 100$\times$ FLOPs and parameters usage
compare to the current SOTA methods MAXIM [Tu et al.(2022)Tu, Talebi, Zhang,
Yang, Milanfar, Bovik, and Li].

MIT-Adobe FiveK [Bychkovsky et al.(2011)Bychkovsky, Paris, Chan, and Durand]
dataset contains 5000 images, each was manually enhanced by five different
experts (A/B/C/D/E). Following the previous settings [Wang et al.(2019)Wang,
Zhang, Fu, Shen, Zheng, and Jia, Moran et al.(2020)Moran, Marza, McDonagh,
Parisot, and Slabaugh], we only use experts C’s adjusted images as ground
truth images. For MIT-Adobe FiveK [Bychkovsky et al.(2011)Bychkovsky, Paris,
Chan, and Durand] dataset training, we use a single L1 loss function to
optimize IAT model. Our method is compared with SOTA enhancement methods [Hu
et al.(2018)Hu, He, Xu, Wang, and Lin, Ronneberger et al.(2015)Ronneberger,
Fischer, and Brox, Chen et al.(2018b)Chen, Wang, Kao, and Chuang, Ignatov et
al.(2017)Ignatov, Kobyshev, Timofte, Vanhoey, and Van Gool, Wang et
al.(2019)Wang, Zhang, Fu, Shen, Zheng, and Jia, Wang et al.(2019)Wang, Zhang,
Fu, Shen, Zheng, and Jia, Moran et al.(2020)Moran, Marza, McDonagh, Parisot,
and Slabaugh, Zeng et al.(2020)Zeng, Cai, Li, Cao, and Zhang] on FiveK
dataset. The image quality results (PSNR, SSIM) and model parameters are
reported in Table. 2. Our IAT also gain satisfactory result in both quality
and efficiency. Qualitative results of LOL [Wei et al.(2018)Wei, Wang, Yang,
and Liu] and FiveK [Bychkovsky et al.(2011)Bychkovsky, Paris, Chan, and
Durand] has been shown in Fig.4. More results could be found in supplementary
material.

Table 1: Experimental results on LOL (V1 & V2) [Wei et al.(2018)Wei, Wang,
Yang, and Liu] datasets, best and second best results are marked in red and
blue respectively, noted here [Guo et al.(2017)Guo, Li, and Ling] is non-deep
learning method and [Guo et al.(2020)Guo, Li, Guo, Loy, Hou, Kwong, and Cong]
is self-supervised learning method.

Methods LOL-V1 LOL-V2-real Efficiency PSNR$\uparrow$ SSIM$\uparrow$
PSNR$\uparrow$ SSIM$\uparrow$ FLOPs(G)$\downarrow$ #Params(M)$\downarrow$ test
time(s)$\downarrow$ LIME* [Guo et al.(2017)Guo, Li, and Ling] 16.67 0.560
15.24 0.470 - - 3.241 (M) Zero-DCE* [Guo et al.(2020)Guo, Li, Guo, Loy, Hou,
Kwong, and Cong] 14.83 0.531 14.32 0.511 2.53 0.08 0.002 (P) RetiNexNet [Wei
et al.(2018)Wei, Wang, Yang, and Liu] 16.77 0.562 18.37 0.723 587.47 0.84
0.841 (T) MBLLEN [Lv et al.(2018)Lv, Lu, Wu, and Lim] 17.90 0.702 18.00 0.715
19.95 20.47 1.981 (T) DRBN [Yang et al.(2020)Yang, Wang, Fang, Wang, and Liu]
19.55 0.746 20.13 0.820 37.79 0.58 1.210 (P) 3D-LUT [Zeng et al.(2020)Zeng,
Cai, Li, Cao, and Zhang] 16.35 0.585 17.59 0.721 7.67 0.6 0.006 (P) KIND
[Zhang et al.(2019)Zhang, Zhang, and Guo] 20.86 0.790 19.74 0.761 356.72 8.16
0e38 (T) UFormer [Wang et al.(2022)Wang, Cun, Bao, Zhou, Liu, and Li] 16.36
0.771 18.82 0.771 12.00 5.29 0.248 (P) IPT [Chen et al.(2021a)Chen, Wang, Guo,
Xu, Deng, Liu, Ma, Xu, Xu, and Gao] 16.27 0.504 19.80 0.813 2087.35 115.63
1.365 (P) RCT [Kim et al.(2021)Kim, Choi, Kim, and Koh] 22.67 0.788 - - - - -
MAXIM [Tu et al.(2022)Tu, Talebi, Zhang, Yang, Milanfar, Bovik, and Li] 23.43
0.863 22.86 0.818 216.00 14.14 0.602 (P) IAT (local) 20.20 0.782 20.30 0.789
1.31 0.02 0.002 (P) IAT 23.38 0.809 23.50 0.824 1.44 0.09 0.004 (P)

Table 2: Experimental results on MIT-Adobe FiveK [Bychkovsky et
al.(2011)Bychkovsky, Paris, Chan, and Durand] dataset.

Metric White-Box [Hu et al.(2018)Hu, He, Xu, Wang, and Lin] U-Net [Ronneberger
et al.(2015)Ronneberger, Fischer, and Brox] DPE [Chen et al.(2018b)Chen, Wang,
Kao, and Chuang] DPED [Ignatov et al.(2017)Ignatov, Kobyshev, Timofte,
Vanhoey, and Van Gool] D-UPE [Wang et al.(2019)Wang, Zhang, Fu, Shen, Zheng,
and Jia] D-LPF [Moran et al.(2020)Moran, Marza, McDonagh, Parisot, and
Slabaugh] 3D LUT [Zeng et al.(2020)Zeng, Cai, Li, Cao, and Zhang] IAT
PSNR$\uparrow$ 18.57 21.57 23.80 21.76 23.04 23.63 25.21 25.32 SSIM$\uparrow$
0.701 0.843 0.880 0.871 0.893 0.875 0.922 0.920 #Params.$\downarrow$ - 1.3M
3.3M - 1.0M 0.8M 0.6M 0.09M

Table 3: Experimental results on exposure correction dataset [Afifi et
al.(2021)Afifi, Derpanis, Ommer, and Brown]. Note here HE and LIME [Guo et
al.(2017)Guo, Li, and Ling] are non-deep learning methods. PSNR, SSIM and PI
results, reported by competing works, are from [Afifi et al.(2021)Afifi,
Derpanis, Ommer, and Brown].

Method Expert A Expert B Expert C Expert D Expert E Avg PI$\downarrow$
PSNR$\uparrow$ SSIM$\uparrow$ PSNR$\uparrow$ SSIM$\uparrow$ PSNR$\uparrow$
SSIM$\uparrow$ PSNR$\uparrow$ SSIM$\uparrow$ PSNR$\uparrow$ SSIM$\uparrow$
PSNR$\uparrow$ SSIM$\uparrow$ HE* [Gonzalez and Woods(2006)] 16.14 0.685 16.28
0.671 16.52 0.696 16.63 0.668 17.30 0.688 16.58 0.682 2.405 LIME* [Guo et
al.(2017)Guo, Li, and Ling] 11.15 0.590 11.83 0.610 11.52 0.607 12.64 0.628
13.61 0.653 12.15 0.618 2.432 DPED [Ignatov et al.(2017)Ignatov, Kobyshev,
Timofte, Vanhoey, and Van Gool] (Sony) 17.42 0.675 18.64 0.701 18.02 0.683
17.55 0.660 17.78 0.663 17.88 0.676 2.806 DPE [Chen et al.(2018b)Chen, Wang,
Kao, and Chuang] (S-FiveK) 16.93 0.678 17.70 0.668 17.74 0.696 17.57 0.674
17.60 0.670 17.51 0.677 2.621 RetinexNet [Wei et al.(2018)Wei, Wang, Yang, and
Liu] 10.76 0.585 11.61 0.596 11.13 0.605 11.99 0.615 12.67 0.636 11.63 0.607
3.105 Deep-UPE [Wang et al.(2019)Wang, Zhang, Fu, Shen, Zheng, and Jia] 13.16
0.610 13.90 0.642 13.69 0.632 14.80 0.649 15.68 0.667 14.25 0.640 2.405 Zero-
DCE [Guo et al.(2020)Guo, Li, Guo, Loy, Hou, Kwong, and Cong] 11.64 0.536
12.56 0.539 12.06 0.544 12.96 0.548 13.77 0.580 12.60 0.549 2.865 MSEC [Afifi
et al.(2021)Afifi, Derpanis, Ommer, and Brown] 19.16 0.746 20.10 0.734 20.20
0.769 18.98 0.719 18.98 0.727 19.48 0.739 2.251 IAT (local) 16.61 0.750 17.52
0.822 16.95 0.780 17.02 0.773 16.43 0.789 16.91 0.783 2.401 IAT 19.90 0.817
21.65 0.867 21.23 0.850 19.86 0.844 19.34 0.840 20.34 0.844 2.249

###  4.2 Exposure Correction.

For the $(c)$ exposure correction task, we evaluate IAT on the benchmark
dataset proposed by [Afifi et al.(2021)Afifi, Derpanis, Ommer, and Brown]. The
dataset contains 24,330 sRGB images, divided into 17,675 training images, 750
validation images, and 5905 test images. Images in [Afifi et al.(2021)Afifi,
Derpanis, Ommer, and Brown] are adjusted by MIT-Adobe FiveK [Bychkovsky et
al.(2011)Bychkovsky, Paris, Chan, and Durand] dataset with 5 different
exposure values (EV), ranging from under-exposure to over-exposure condition.
Same as [Bychkovsky et al.(2011)Bychkovsky, Paris, Chan, and Durand], test set
has 5 different experts’ adjust results (A/B/C/D/E). Following the setting of
[Afifi et al.(2021)Afifi, Derpanis, Ommer, and Brown], the training images are
cropped to $512\times 512$ patches and the test image is resized to have a
maximum dimension of $512$ pixels. We compare the test images with all five
experts’ results. Here we use L1 loss function for exposure correction
training.

The evaluation result is shown in Table. 3. Our comparison methods include
both traditional image processing methods (Histogram Equalization [Gonzalez
and Woods(2006)], LIME [Guo et al.(2017)Guo, Li, and Ling]) and deep learning
methods (DPED [Ignatov et al.(2017)Ignatov, Kobyshev, Timofte, Vanhoey, and
Van Gool], DPE [Chen et al.(2018b)Chen, Wang, Kao, and Chuang], RetinexNet
[Wei et al.(2018)Wei, Wang, Yang, and Liu], Deep-UPE [Wang et al.(2019)Wang,
Zhang, Fu, Shen, Zheng, and Jia], Zero-DCE [Guo et al.(2020)Guo, Li, Guo, Loy,
Hou, Kwong, and Cong], MSEC [Afifi et al.(2021)Afifi, Derpanis, Ommer, and
Brown]). Evaluation metrics are same as [Afifi et al.(2021)Afifi, Derpanis,
Ommer, and Brown], including PSNR, SSIM and perceptual index (PI). Table. 3
shows that our IAT model has gained best result on all evaluation indexs.
Compared to the second best result MSEC [Afifi et al.(2021)Afifi, Derpanis,
Ommer, and Brown], IAT has much fewer parameters (0.09M v.s. 7M) and less
evaluation time (0.004s per image v.s. 0.5s per image). Qualitative result has
been shown in Fig.4 and more visual results are given in supplementary
material.

Table 4: Experimental results on low-light detection dataset EXDark [Loh and
Chan(2019)], low-light semantic segmentation dataset ACDC [Sakaridis et
al.(2021)Sakaridis, Dai, and Van Gool] and various light detection dataset
TYOL [Hodan et al.(2018)Hodan, Michel, Brachmann, Kehl, GlentBuch, Kraft,
Drost, Vidal, Ihrke, Zabulis, et al.].

Methods $(d)$ EXDark Detection [Loh and Chan(2019)] $(e)$ ACDC Segmentation
[Sakaridis et al.(2021)Sakaridis, Dai, and Van Gool] $(f)$ TYOL Detection
[Hodan et al.(2018)Hodan, Michel, Brachmann, Kehl, GlentBuch, Kraft, Drost,
Vidal, Ihrke, Zabulis, et al.] mAP$\uparrow$ time(s)$\downarrow$
mIOU$\uparrow$ time(s)$\downarrow$ mAP$\uparrow$ time(s)$\downarrow$ base-line
76.4 0.033 63.3 0.249 88.4 0.023 MBLLEN [Lv et al.(2018)Lv, Lu, Wu, and Lim]
76.3 0.086 63.0 0.332 95.3 0.105 DeepLPF [Moran et al.(2020)Moran, Marza,
McDonagh, Parisot, and Slabaugh] 76.3 0.138 61.9 0.807 94.5 0.223 Zero-DCE
[Guo et al.(2020)Guo, Li, Guo, Loy, Hou, Kwong, and Cong] 76.9 0.042 61.9
0.300 95.2 0.030 IAT 77.2 0.040 62.1 0.280 95.8 0.027

###  4.3 High-level Vision

For high-level vision tasks: $\left\\{(d),(e),(f)\right\\}$, we use IAT to
restore the image before feeding to the subsequent recognition algorithms
based on mmdetection and mmsegmentation [Chen et al.(2019)Chen, Wang, Pang,
Cao, Xiong, Li, Sun, Feng, Liu, Xu, Zhang, Cheng, Zhu, Cheng, Zhao, Li, Lu,
Zhu, Wu, Dai, Wang, Shi, Ouyang, Loy, and Lin, Contributors(2020)]. For a fair
comparison, we run all of the experiments in the same setting: same input
size, same data augmentation methods (expand, random crop, multi-size, random
flip…), same training epochs and same initial weights. We train the
recognition algorithm on the datasets enhanced by IAT. We compare our methods
with original datasets as well as datasets enhanced by other enhancement
methods [Lv et al.(2018)Lv, Lu, Wu, and Lim, Moran et al.(2020)Moran, Marza,
McDonagh, Parisot, and Slabaugh, Guo et al.(2020)Guo, Li, Guo, Loy, Hou,
Kwong, and Cong].

For object detection task in $(d)$ EXDark dataset [Loh and Chan(2019)] and
$(f)$ TYOL dataset [Hodan et al.(2018)Hodan, Michel, Brachmann, Kehl,
GlentBuch, Kraft, Drost, Vidal, Ihrke, Zabulis, et al.]. EXDark includes 7,363
real-world low-light images, ranging from twilight to extreme dark environment
with 12 object categories. We take 80$\%$ images of each category for training
and the other 20$\%$ for testing. TYOL includes 1680 images with 21 classes.
We take 1365 images for training and other for evaluation. For both datasets,
we perform object detection with YOLO-V3 [Redmon and Farhadi(2018)], all the
input images have been cropped and resized to 608 $\times$ 608 pixel size, we
use SGD optimizer to train YOLO-V3 with batch size 8 for 25 epochs to EXDark
and 45 epochs to TYO-L, the initial learning rate is 1$e^{-3}$ and weight
decay is 1$e^{-4}$. The detection metric mAP and per-image evaluation time is
shown in Table. 4. Our IAT model gains best results in both accuracy and speed
compared to the baseline model and other enhancement methods [Lv et
al.(2018)Lv, Lu, Wu, and Lim, Moran et al.(2020)Moran, Marza, McDonagh,
Parisot, and Slabaugh, Guo et al.(2020)Guo, Li, Guo, Loy, Hou, Kwong, and
Cong].

For semantic segmentation in $(e)$ ACDC dataset [Sakaridis et
al.(2021)Sakaridis, Dai, and Van Gool], we take 1006 night images in the ACDC
dataset and then adopt DeepLab-V3+ [Chen et al.(2018a)Chen, Zhu, Papandreou,
Schroff, and Adam] to train on the ACDC-night train set and test on ACDC-night
val set. The DeepLab-V3+ [Chen et al.(2018a)Chen, Zhu, Papandreou, Schroff,
and Adam] model is initialed by an Cityscape dataset [Cordts et
al.(2016)Cordts, Omran, Ramos, Rehfeld, Enzweiler, Benenson, Franke, Roth, and
Schiele] pre-train model, we tuned the pre-train model by SGD optimizer with
batch size 8 for 20000 iters, initial learning rate is set to 0.05, momentum
and weight decay are separately set to 0.9 and 5$e^{-4}$. We show the
segmentation metric mIOU and per-image evaluation time in Table. 4, we found
that all the enhancements methods invalid in this setting, this may because
the lightness condition in ACDC [Sakaridis et al.(2021)Sakaridis, Dai, and Van
Gool] is various and exceeds the generalisation ability of the enhancement
model. For this problem, we propose to joint training our IAT model with
following segmentation network (as well as detection network), which would
solves this problem and improve the semantic segmentation/ object detection
results in low-light conditions, detailed analyse please refer to Sec. 8 of
supplementary material 111For more experimental details and ablation analyse,
please refer to the supplementary material..

##  5 Conclusion

We propose a novel lightweight transformer framework IAT, by adapting ISP-
related parameters to adapt to challenging light conditions. Despite its
superior performance on several real-world datasets for both low-level and
high-level tasks, IAT is extremely light with a fast speed. The lightweight
and mobile-friendly IAT has the potential to become a standing tool for the
computer vision community.

However, one mian drawback of the IAT module is that, the image signal
processor (ISP) has been simplified due to the light-weight demand, we think
that more detailed ISP-related parts could be concerned and interpolate to the
IAT module. In further, we’d also like to implement IAT on 3D human relighting
task, to solve more complex lighting problems under 3D condition.

##  6 Acknowledgement

Corner symbol ’*’ in the author name means the corresponding author. This work
supported by JST Moonshot R$\&$D Grant Number JPMJMS2011 and JST, ACT-X Grant
Number JPMJAX190D, Japan. This work also supported by National Natural Science
Foundation of China (Grant No. 62206272) and Shanghai Committee of Science and
Technology (Grant No. 21DZ1100100).

## References

  * [Afifi and Brown(2020)] Mahmoud Afifi and Michael S. Brown.  Deep white-balance editing.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * [Afifi et al.(2021)Afifi, Derpanis, Ommer, and Brown] Mahmoud Afifi, Konstantinos G. Derpanis, Bjorn Ommer, and Michael S. Brown.  Learning multi-scale photo exposure correction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021.
  * [Afifi et al.(2022)Afifi, Brubaker, and Brown] Mahmoud Afifi, Marcus A. Brubaker, and Michael S. Brown.  Auto white-balance correction for mixed-illuminant scenes.  In _IEEE Winter Conference on Applications of Computer Vision (WACV)_ , 2022.
  * [Ba et al.(2016)Ba, Kiros, and Hinton] Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton.  Layer normalization.  _ArXiv_ , abs/1607.06450, 2016.
  * [Brooks et al.(2019)Brooks, Mildenhall, Xue, Chen, Sharlet, and Barron] Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, and Jonathan T Barron.  Unprocessing images for learned raw denoising.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2019.
  * [Bychkovsky et al.(2011)Bychkovsky, Paris, Chan, and Durand] Vladimir Bychkovsky, Sylvain Paris, Eric Chan, and Frédo Durand.  Learning photographic global tonal adjustment with a database of input / output image pairs.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2011.
  * [Carion et al.(2020)Carion, Massa, Synnaeve, Usunier, Kirillov, and Zagoruyko] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In _European conference on computer vision_ , 2020.
  * [Chen et al.(2018)Chen, Chen, Xu, and Koltun] C. Chen, Q. Chen, J. Xu, and V. Koltun.  Learning to see in the dark.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018.
  * [Chen et al.(2021a)Chen, Wang, Guo, Xu, Deng, Liu, Ma, Xu, Xu, and Gao] Hanting Chen, Yunhe Wang, Tianyu Guo, Chang Xu, Yiping Deng, Zhenhua Liu, Siwei Ma, Chunjing Xu, Chao Xu, and Wen Gao.  Pre-trained image processing transformer.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2021a.
  * [Chen et al.(2019)Chen, Wang, Pang, Cao, Xiong, Li, Sun, Feng, Liu, Xu, Zhang, Cheng, Zhu, Cheng, Zhao, Li, Lu, Zhu, Wu, Dai, Wang, Shi, Ouyang, Loy, and Lin] Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin.  MMDetection: Open mmlab detection toolbox and benchmark.  _arXiv preprint arXiv:1906.07155_ , 2019.
  * [Chen et al.(2018a)Chen, Zhu, Papandreou, Schroff, and Adam] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation.  In _Proceedings of the European conference on computer vision_ , 2018a.
  * [Chen et al.(2021b)Chen, Feng, Gao, Xu, and Chen] Shiqi Chen, Huajun Feng, Keming Gao, Zhihai Xu, and Yueting Chen.  Extreme-quality computational imaging via degradation framework.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021b.
  * [Chen et al.(2018b)Chen, Wang, Kao, and Chuang] Yu-Sheng Chen, Yu-Ching Wang, Man-Hsin Kao, and Yung-Yu Chuang.  Deep photo enhancer: Unpaired learning for image enhancement from photographs with gans.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018b.
  * [Contributors(2020)] MMSegmentation Contributors.  MMSegmentation: Openmmlab semantic segmentation toolbox and benchmark.  https://github.com/open-mmlab/mmsegmentation, 2020.
  * [Cordts et al.(2016)Cordts, Omran, Ramos, Rehfeld, Enzweiler, Benenson, Franke, Roth, and Schiele] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In _Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2016.
  * [Cui et al.(2021)Cui, Qi, Gu, You, Zhang, and Harada] Ziteng Cui, Guo-Jun Qi, Lin Gu, Shaodi You, Zenghui Zhang, and Tatsuya Harada.  Multitask aet with orthogonal tangent regularity for dark object detection.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021.
  * [Deng et al.(2009)Deng, Dong, Socher, Li, Li, and Fei-Fei] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _IEEE conference on computer vision and pattern recognition_ , 2009\.
  * [Dosovitskiy et al.(2021)Dosovitskiy, Beyer, Kolesnikov, Weissenborn, Zhai, Unterthiner, Dehghani, Minderer, Heigold, Gelly, Uszkoreit, and Houlsby] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_ , 2021.
  * [Fu et al.(2022)Fu, Hong, Chen, and You] Ying Fu, Yang Hong, Linwei Chen, and Shaodi You.  Le-gan: Unsupervised low-light image enhancement network using attention module and identity invariant loss.  _Knowledge-Based Systems_ , 2022.
  * [Gonzalez and Woods(2006)] Rafael C. Gonzalez and Richard E. Woods.  _Digital Image Processing (3rd Edition)_.  Prentice-Hall, Inc., USA, 2006.  ISBN 013168728X.
  * [Guo et al.(2020)Guo, Li, Guo, Loy, Hou, Kwong, and Cong] C. Guo, C. Li, J. Guo, C. C. Loy, J. Hou, S. Kwong, and R. Cong.  Zero-reference deep curve estimation for low-light image enhancement.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020.
  * [Guo et al.(2017)Guo, Li, and Ling] Xiaojie Guo, Yu Li, and Haibin Ling.  Lime: Low-light image enhancement via illumination map estimation.  _IEEE Transactions on Image Processing_ , 2017.
  * [Han et al.(2022)Han, Fan, Dai, Sun, Cheng, Liu, and Wang] Qi Han, Zejia Fan, Qi Dai, Lei Sun, Ming-Ming Cheng, Jiaying Liu, and Jingdong Wang.  On the connection between local attention and dynamic depth-wise convolution.  In _International Conference on Learning Representations_ , 2022.
  * [He et al.(2016)He, Zhang, Ren, and Sun] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition_ , 2016.
  * [Heide and et.al(2014)] Felix Heide and Steinberger et.al.  Flexisp: A flexible camera image processing framework.  _ACM Trans. Graph._ , 2014.
  * [Hodan et al.(2018)Hodan, Michel, Brachmann, Kehl, GlentBuch, Kraft, Drost, Vidal, Ihrke, Zabulis, et al.] Tomas Hodan, Frank Michel, Eric Brachmann, Wadim Kehl, Anders GlentBuch, Dirk Kraft, Bertram Drost, Joel Vidal, Stephan Ihrke, Xenophon Zabulis, et al.  Bop: Benchmark for 6d object pose estimation.  In _Proceedings of the European Conference on Computer Vision_ , 2018\.
  * [Hong et al.(2021)Hong, Wei, Chen, and Fu] Yang Hong, Kaixuan Wei, Linwei Chen, and Ying Fu.  Crafting object detection in very low light.  In _The British Machine Vision Conference_ , November 2021.
  * [Hu et al.(2018)Hu, He, Xu, Wang, and Lin] Yuanming Hu, Hao He, Chenxi Xu, Baoyuan Wang, and Stephen Lin.  Exposure: A white-box photo post-processing framework.  _ACM Trans. Graph._ , 2018.
  * [Ignatov et al.(2017)Ignatov, Kobyshev, Timofte, Vanhoey, and Van Gool] Andrey Ignatov, Nikolay Kobyshev, Radu Timofte, Kenneth Vanhoey, and Luc Van Gool.  Dslr-quality photos on mobile devices with deep convolutional networks.  In _Proceedings of the IEEE international conference on computer vision_ , 2017.
  * [Jiang et al.(2017)Jiang, Tian, Farrell, and Wandell] H. Jiang, Q. Tian, J. Farrell, and B. A. Wandell.  Learning the image processing pipeline.  _IEEE Transactions on Image Processing_ , 2017.
  * [Johnson et al.(2016)Johnson, Alahi, and Fei-Fei] Justin Johnson, Alexandre Alahi, and Li Fei-Fei.  Perceptual losses for real-time style transfer and super-resolution.  In _European Conference on Computer Vision_ , 2016.
  * [Karaimer and Brown(2016)] Hakki Can Karaimer and Michael S. Brown.  A software platform for manipulating the camera imaging pipeline.  In _European Conference on Computer Vision_ , 2016.
  * [Kim et al.(2021)Kim, Choi, Kim, and Koh] Hanul Kim, Su-Min Choi, Chang-Su Kim, and Yeong Jun Koh.  Representative color transform for image enhancement.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021.
  * [Kumar et al.(2021)Kumar, Weissenborn, and Kalchbrenner] Manoj Kumar, Dirk Weissenborn, and Nal Kalchbrenner.  Colorization transformer.  In _International Conference on Learning Representations_ , 2021.
  * [Land(1986)] Edwin H. Land.  An alternative technique for the computation of the designator in the retinex theory of color vision.  _Proceedings of the National Academy of Sciences of the United States of America_ , 1986.
  * [Li et al.(2022)Li, Wang, Zhang, Gao, Song, Liu, Li, and Qiao] Kunchang Li, Yali Wang, Junhao Zhang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao.  Uniformer: Unifying convolution and self-attention for visual recognition.  _arXiv preprint arXiv:2201.09450_ , 2022.
  * [Liang et al.(2021)Liang, Cao, Sun, Zhang, Van Gool, and Timofte] Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte.  Swinir: Image restoration using swin transformer.  In _IEEE International Conference on Computer Vision Workshops_ , 2021\.
  * [Lin et al.(2014)Lin, Maire, Belongie, Hays, Perona, Ramanan, Dollár, and Zitnick] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In _European conference on computer vision_ , 2014.
  * [Liu et al.(2021a)Liu, Xu, Yang, Fan, and Huang] Jiaying Liu, Dejia Xu, Wenhan Yang, Minhao Fan, and Haofeng Huang.  Benchmarking low-light image enhancement and beyond.  _International Journal of Computer Vision_ , 2021a.
  * [Liu et al.(2021b)Liu, Lin, Cao, Hu, Wei, Zhang, Lin, and Guo] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021b.
  * [Loh and Chan(2019)] Yuen Peng Loh and Chee Seng Chan.  Getting to know low-light images with the exclusively dark dataset.  _Computer Vision and Image Understanding_ , 2019.
  * [Lore et al.(2017)Lore, Akintayo, and Sarkar] Kin Gwn Lore, Adedotun Akintayo, and Soumik Sarkar.  Llnet: A deep autoencoder approach to natural low-light image enhancement.  _Pattern Recognition_ , 2017.
  * [Lv et al.(2018)Lv, Lu, Wu, and Lim] Feifan Lv, Feng Lu, Jianhua Wu, and Chongsoon Lim.  Mbllen: Low-light image/video enhancement using cnns.  In _British Machine Vision Conference_ , 2018.
  * [Minciullo et al.(2021)Minciullo, Manhardt, Yoshikawa, Meier, Tombari, and Kobori] Luca Minciullo, Fabian Manhardt, Kei Yoshikawa, Sven Meier, Federico Tombari, and Norimasa Kobori.  Db-gan: Boosting object recognition under strong lighting conditions.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , 2021.
  * [Moran et al.(2020)Moran, Marza, McDonagh, Parisot, and Slabaugh] Sean Moran, Pierre Marza, Steven McDonagh, Sarah Parisot, and Gregory Slabaugh.  Deeplpf: Deep local parametric filters for image enhancement.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020.
  * [Nayar and Branzoi(2003)] Nayar and Branzoi.  Adaptive dynamic range imaging: optical control of pixel exposures over space and time.  In _Proceedings Ninth IEEE International Conference on Computer Vision_ , pages 1168–1175 vol.2, 2003.  10.1109/ICCV.2003.1238624.
  * [Nsampi et al.(2021)Nsampi, Hu, and Wang] Ntumba Elie Nsampi, Zhongyun Hu, and Qing Wang.  Learning exposure correction via consistency modeling.  In _BMVC_ , 2021.
  * [Park et al.(2018)Park, Lee, Yoo, and Kweon] Jongchan Park, Joon-Young Lee, Donggeun Yoo, and In So Kweon.  Distort-and-recover: Color enhancement using deep reinforcement learning.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2018.
  * [Radford et al.(2021)Radford, Kim, Hallacy, Ramesh, Goh, Agarwal, Sastry, Askell, Mishkin, Clark, Krueger, and Sutskever] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In Marina Meila and Tong Zhang, editors, _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pages 8748–8763. PMLR, 18–24 Jul 2021.
  * [Redmon and Farhadi(2018)] Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement.  _arXiv preprint arXiv:1804.02767_ , 2018.
  * [Ronneberger et al.(2015)Ronneberger, Fischer, and Brox] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation, 2015\.
  * [Sakaridis et al.(2021)Sakaridis, Dai, and Van Gool] Christos Sakaridis, Dengxin Dai, and Luc Van Gool.  ACDC: The adverse conditions dataset with correspondences for semantic driving scene understanding.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021.
  * [Sasagawa and Nagahara(2020)] Yukihiro Sasagawa and Hajime Nagahara.  Yolo in the dark: Domain adaptation method for merging multiple models.  In _Proceedings of European Conference on Computer Vision_ , 2020\.
  * [Sharma and Tan(2021)] Aashish Sharma and Robby T. Tan.  Nighttime visibility enhancement by increasing the dynamic range and suppression of light effects.  In _2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 11972–11981, 2021.  10.1109/CVPR46437.2021.01180.
  * [Tolstikhin et al.(2021)Tolstikhin, Houlsby, Kolesnikov, Beyer, Zhai, Unterthiner, Yung, Steiner, Keysers, Uszkoreit, Lucic, and Dosovitskiy] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy.  Mlp-mixer: An all-mlp architecture for vision.  In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, _Advances in Neural Information Processing Systems_ , volume 34, pages 24261–24272. Curran Associates, Inc., 2021.  URL https://proceedings.neurips.cc/paper/2021/file/cba0a4ee5ccd02fda0fe3f9a3e7b89fe-Paper.pdf.
  * [Tomasi and Manduchi(1998)] Carlo Tomasi and Roberto Manduchi.  Bilateral filtering for gray and color images.  In _Sixth international conference on computer vision (IEEE Cat. No. 98CH36271)_ , 1998.
  * [Touvron et al.(2021a)Touvron, Bojanowski, Caron, Cord, El-Nouby, Grave, Izacard, Joulin, Synnaeve, Verbeek, et al.] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Gautier Izacard, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, et al.  Resmlp: Feedforward networks for image classification with data-efficient training.  _arXiv preprint arXiv:2105.03404_ , 2021a.
  * [Touvron et al.(2021b)Touvron, Cord, Sablayrolles, Synnaeve, and Jégou] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou.  Going deeper with image transformers.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021b.
  * [Tu et al.(2022)Tu, Talebi, Zhang, Yang, Milanfar, Bovik, and Li] Zhengzhong Tu, Hossein Talebi, Han Zhang, Feng Yang, Peyman Milanfar, Alan Bovik, and Yinxiao Li.  Maxim: Multi-axis mlp for image processing.  _CVPR_ , 2022.
  * [Valanarasu et al.(2021)Valanarasu, Yasarla, and Patel] Jeya Maria Jose Valanarasu, Rajeev Yasarla, and Vishal M Patel.  Transweather: Transformer-based restoration of images degraded by adverse weather conditions.  _arXiv preprint arXiv:2111.14813_ , 2021.
  * [Vaswani et al.(2017)Vaswani, Shazeer, Parmar, Uszkoreit, Jones, Gomez, Kaiser, and Polosukhin] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 30. Curran Associates, Inc., 2017.  URL https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
  * [Wang et al.(2019)Wang, Zhang, Fu, Shen, Zheng, and Jia] Ruixing Wang, Qing Zhang, Chi-Wing Fu, Xiaoyong Shen, Wei-Shi Zheng, and Jiaya Jia.  Underexposed photo enhancement using deep illumination estimation.  In _The IEEE Conference on Computer Vision and Pattern Recognition_ , 2019.
  * [Wang et al.(2022)Wang, Cun, Bao, Zhou, Liu, and Li] Zhendong Wang, Xiaodong Cun, Jianmin Bao, Wengang Zhou, Jianzhuang Liu, and Houqiang Li.  Uformer: A general u-shaped transformer for image restoration.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2022.
  * [Wei et al.(2018)Wei, Wang, Yang, and Liu] Chen Wei, Wenjing Wang, Wenhan Yang, and Jiaying Liu.  Deep retinex decomposition for low-light enhancement.  In _British Machine Vision Conference_ , 2018.
  * [Wei et al.(2020)Wei, Fu, Yang, and Huang] Kaixuan Wei, Ying Fu, Jiaolong Yang, and Hua Huang.  A physics-based noise formation model for extreme low-light raw denoising.  In _IEEE Conference on Computer Vision and Pattern Recognition_ , 2020\.
  * [Xie et al.(2021)Xie, Wang, Yu, Anandkumar, Alvarez, and Luo] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo.  Segformer: Simple and efficient design for semantic segmentation with transformers.  _arXiv preprint arXiv:2105.15203_ , 2021.
  * [Xing et al.(2021)Xing, Qian, and Chen] Yazhou Xing, Zian Qian, and Qifeng Chen.  Invertible image signal processing.  In _CVPR_ , 2021.
  * [Xu et al.(2022)Xu, Wang, Fu, and Jia] Xiaogang Xu, Ruixing Wang, Chi-Wing Fu, and Jiaya Jia.  Snr-aware low-light image enhancement.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 17714–17724, June 2022.
  * [Yang et al.(2022)Yang, Cheng, Zhao, Zhang, and Li] Kai-Fu Yang, Cheng Cheng, Shi-Xuan Zhao, Xian-Shi Zhang, and Yong-Jie Li.  Learning to adapt to light, 2022.  URL https://arxiv.org/abs/2202.08098.
  * [Yang et al.(2020)Yang, Wang, Fang, Wang, and Liu] Wenhan Yang, Shiqi Wang, Yuming Fang, Yue Wang, and Jiaying Liu.  From fidelity to perceptual quality: A semi-supervised approach for low-light image enhancement.  In _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2020.
  * [Yu et al.(2018)Yu, Liu, Zhang, Qu, Zhao, and Zhang] Runsheng Yu, Wenyu Liu, Yasen Zhang, Zhi Qu, Deli Zhao, and Bo Zhang.  Deepexposure: Learning to expose photos with asynchronously reinforced adversarial learning.  In _Advances in Neural Information Processing Systems_ , 2018.
  * [Yuan and Sun(2012)] Lu Yuan and Jian Sun.  Automatic exposure correction of consumer photographs.  In _European Conference on Computer Vision_ , 2012.
  * [Zamir et al.(2021)Zamir, Arora, Khan, Hayat, Khan, and Yang] Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Ming-Hsuan Yang.  Restormer: Efficient transformer for high-resolution image restoration.  _arXiv preprint arXiv:2111.09881_ , 2021.
  * [Zeng et al.(2020)Zeng, Cai, Li, Cao, and Zhang] Hui Zeng, Jianrui Cai, Lida Li, Zisheng Cao, and Lei Zhang.  Learning image-adaptive 3d lookup tables for high performance photo enhancement in real-time.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pages 1–1, 2020.  10.1109/TPAMI.2020.3026740.
  * [Zhang et al.(2021)Zhang, Fang, Gao, Zhang, Li, Dai, Qiao, and Li] Renrui Zhang, Rongyao Fang, Peng Gao, Wei Zhang, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li.  Tip-adapter: Training-free clip-adapter for better vision-language modeling.  _arXiv preprint arXiv:2111.03930_ , 2021.
  * [Zhang et al.(2022)Zhang, Qiu, Wang, Xu, Guo, Qiao, Gao, and Li] Renrui Zhang, Han Qiu, Tai Wang, Xuanzhuo Xu, Ziyu Guo, Yu Qiao, Peng Gao, and Hongsheng Li.  Monodetr: Depth-aware transformer for monocular 3d object detection.  _arXiv preprint arXiv:2203.13310_ , 2022.
  * [Zhang et al.(2019)Zhang, Zhang, and Guo] Yonghua Zhang, Jiawan Zhang, and Xiaojie Guo.  Kindling the darkness: A practical low-light image enhancer.  In _Proceedings of the 27th ACM international conference on multimedia_ , 2019.
  * [Zheng et al.(2020)Zheng, Zhang, and Lu] Yinqiang Zheng, Mingfang Zhang, and Feng Lu.  Optical flow in the dark.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6749–6757, 2020.

##  7 Analyse on Module Structure

For the global part $g$ of the IAT module, here we simplify the ISP procedures
[Brooks et al.(2019)Brooks, Mildenhall, Xue, Chen, Sharlet, and Barron, Cui et
al.(2021)Cui, Qi, Gu, You, Zhang, and Harada, Chen et al.(2021b)Chen, Feng,
Gao, Xu, and Chen] as the following equation:

$G(\cdot)=Gamma(W_{ccm}(W_{wb}(\cdot))).$ (5)

White balance (WB) function is an essential part in ISP pipeline. WB algorithm
estimates the per channel gain on the image, to maintain the object’s colour
constancy under various different light colour. WB is usually represented as a
$3\times 3$ diagonal von Kris matrix $W_{wb}$ in camera imaging pipeline
[Afifi and Brown(2020), Afifi et al.(2022)Afifi, Brubaker, and Brown, Brooks
et al.(2019)Brooks, Mildenhall, Xue, Chen, Sharlet, and Barron, Karaimer and
Brown(2016)]. After that, camera color matrix (CCM) $W_{ccm}$ converts the
white-balanced data from camera internal color space cRGB to sRGB colour space
[Karaimer and Brown(2016), Cui et al.(2021)Cui, Qi, Gu, You, Zhang, and
Harada, Brooks et al.(2019)Brooks, Mildenhall, Xue, Chen, Sharlet, and
Barron]. At last gamma correction aims to match non-linearity of humans
perception on dark regions. A standard gamma curve is usually represent as an
exponential function with the exponential parameter $\gamma$, so we build our
global branch $g_{t}(\cdot)$ following the equation:

$g_{t}(\cdot)=(max(\sum_{c_{j}}W_{c_{i},c_{j}}(\cdot),\epsilon))^{\gamma},c_{i},c_{j}\in\\{r,g,b\\},$
(6)

where the $W_{c_{i},c_{j}}$ is a joint colour transform function consist of
white balance matrix and CCM and $\gamma$ is the gamma correction’s
exponential value, $\epsilon$ is a minimum number to keep non-negative. Final
as we discussed in Sec.3.1, the input image $I_{i}$ would separately pass by
local branch $f$ and global branch $g$ to generate the prediction result
$\hat{I_{t}}=g_{t}(f(I_{i}))$.

We also evaluate to train the model with corresponding raw-RGB data as
additional supervision. Since it’s hard to directly get raw-RGB data from the
currect dataset, we then adopt the Invertible ISP [Xing et al.(2021)Xing,
Qian, and Chen] to generate corresponding raw-RGB data $I_{raw}$ from the
input image $I_{i}$, we use pre-train weights in [Xing et al.(2021)Xing, Qian,
and Chen] to generate $I_{raw}$. In the training stage, we additional add a
loss function $L_{raw}$ for raw-RGB supervision, the total loss function shown
as follow:

$\displaystyle L_{total}=$ $\displaystyle L_{rgb}+\lambda\cdot L_{raw}$ (7)
$\displaystyle=$ $\displaystyle L_{1}(g_{t}(f(I_{i}),I_{t})+\lambda\cdot
L_{1}(f(I_{i}),I_{raw}).$

$L_{total}$ is the total loss function that consist of two parts: the first
part $L_{rgb}$ is L1 loss function between predict result $g_{t}(f(I_{i}))$
with ground truth image $I_{t}$, while the second part $L_{raw}$ is the L1
loss function between middle representation $f(I_{i})$ and raw-RGB image
$I_{raw}$ for raw-RGB part supervision, and $\lambda$ is a balance parameter
where we set it to 0.1 in our experiments. We make the comparison experiments
on exposure correction dataset [Afifi et al.(2021)Afifi, Derpanis, Ommer, and
Brown], the training and experiments’ settings are follow the settings in
Sec.4.2, only difference is the training strategy with or without raw-RGB
supervision. The comparison results are shown in Table 5, we can find that
with the additional supervision of raw-RGB data, most of evaluation metrics on
exposure correction dataset [Afifi et al.(2021)Afifi, Derpanis, Ommer, and
Brown] would be improved.

Table 5: Comparison experiments of with (w) and without (w/o) raw-RGB
supervision on exposure correction dataset [Afifi et al.(2021)Afifi, Derpanis,
Ommer, and Brown].

Method Expert A Expert B Expert C Expert D Expert E PSNR$\uparrow$
SSIM$\uparrow$ PSNR$\uparrow$ SSIM$\uparrow$ PSNR$\uparrow$ SSIM$\uparrow$
PSNR$\uparrow$ SSIM$\uparrow$ PSNR$\uparrow$ SSIM$\uparrow$ w/o raw-RGB 19.90
0.817 21.65 0.867 21.23 0.850 19.86 0.844 19.34 0.840 w raw-RGB 19.98 0.822
22.03 0.885 21.16 0.843 19.94 0.852 19.48 0.841

##  8 Joint Training with High-level Framework

Figure 5: Joint training Enhancement Module with High-level Module.

For high-level vision tasks under challenging lighting conditions,shown in
Fig.5, current high-level vision frameworks [Carion et al.(2020)Carion, Massa,
Synnaeve, Usunier, Kirillov, and Zagoruyko, Redmon and Farhadi(2018), Chen et
al.(2018a)Chen, Zhu, Papandreou, Schroff, and Adam] usually well-trained on
large scale normal-light datasets (i.e. MS COCO [Lin et al.(2014)Lin, Maire,
Belongie, Hays, Perona, Ramanan, Dollár, and Zitnick], ImageNet [Deng et
al.(2009)Deng, Dong, Socher, Li, Li, and Fei-Fei]), so directly take low-
light/ strong-light data as input would cause the lightness in-consistency, on
the other hand, using image enhancement methods (Sec.4.3 in main text) to pre-
process images may cause target inconsistency (human vision v.s. machine
vision) [Cui et al.(2021)Cui, Qi, Gu, You, Zhang, and Harada], since the goal
of image restoration is image quality (i.e. PSNR, SSIM) and the goal of
detection/ segmentation is machine-vision accuracy (i.e. mAP, mIOU).

An example is shown in Fig. 5, by attaching IAT to the downstream task module,
our IAT could conduct object detection and semantic segmentation with the
downstream frameworks. During training, we aim to minimise the downstream
framework’s loss function (i.e. object detection loss $L_{obj}$ between
detection prediction $\hat{t}$ and ground truth $t$) by jointly optimising the
whole network’s parameters (see Eq. 8). Compared to the subsequent high-level
module, the time-complexity and model storage of our IAT main structure could
be ignored (i.e. IAT main structure v.s. YOLO-V3 [Redmon and Farhadi(2018)],
417KB v.s. 237MB).

$\displaystyle\min\limits_{i\in\mathbb{I},d\in\mathbb{D}}L_{obj}$
$\displaystyle(\hat{t},t)$ (8) $\displaystyle I_{t}(x)=\mathbb{I}(I_{1}(x)),$
$\displaystyle\hat{t}=\mathbb{D}(I_{t}(x))$

We make the comparison experiments on low-light detection dataset EXDark [Loh
and Chan(2019)] and low-light semantic segmentation dataset ACDC [Sakaridis et
al.(2021)Sakaridis, Dai, and Van Gool]. For object detection task we adopt the
YOLO-V3 [Redmon and Farhadi(2018)] object detector and for segmentation task
we adopt DeepLabV3+ [Chen et al.(2018a)Chen, Zhu, Papandreou, Schroff, and
Adam] segmentation framework, the training and experiments’ settings are
follow the settings in Sec.4.3.

Experimental results are shown in Table. 6, "original" means to take the
original low-light images for training and evaluation, "pre-enhancement" means
to pre-enhancement the EXDark [Loh and Chan(2019)] and ACDC [Sakaridis et
al.(2021)Sakaridis, Dai, and Van Gool] datasets with IAT model trained on
LOL-V1 dataset [Wei et al.(2018)Wei, Wang, Yang, and Liu] ("IAT (LOL)") and
MIT-Adobe FiveK dataset [Bychkovsky et al.(2011)Bychkovsky, Paris, Chan, and
Durand] ("IAT (MIT5K)"). The "joint training" means to joint train IAT with
the following high-level framework, and IAT model is separately random
initialize ("IAT (none)"), initialize with LOL pre-train weights ("IAT (LOL)")
and initialize with MIT-Adobe FiveK weights ("IAT (MIT5K)"), from Table. 6 we
could see that joint-training IAT with the high-level frameworks would further
improve high-level visual performance, on both of object detection and
semantic segmentation task.

Table 6: Comparison experiments on low-light detection dataset EXDark [Loh and
Chan(2019)] and low-light semantic segmentation dataset ACDC [Sakaridis et
al.(2021)Sakaridis, Dai, and Van Gool].

original pre-enhancement joint training IAT (LOL) IAT (MIT5K) IAT (none) IAT
(MIT5K) IAT (LOL) EXDark (mAP$\uparrow$) 76.4 77.2 76.9 77.1 77.6 77.8 ACDC
(mIOU$\uparrow$) 63.3 62.1 61.3 61.5 62.1 63.8

##  9 Ablation Studies

###  9.1 Contribution of each part.

To evaluate each part’s contribution in our IAT model, we make an ablation
study on the low-light enhancement task of LOL-V2-real [Wei et al.(2018)Wei,
Wang, Yang, and Liu] dataset, and the low-light object detection task of
EXDark [Loh and Chan(2019)] dataset. We report the PSNR and SSIM results of
the enhancement task and the mAP result of the detection task. We compare our
normalization with LayerNorm [Ba et al.(2016)Ba, Kiros, and Hinton] and
ResMLP’s normalization [Touvron et al.(2021a)Touvron, Bojanowski, Caron, Cord,
El-Nouby, Grave, Izacard, Joulin, Synnaeve, Verbeek, et al.], and then
evaluate different parts’ contributions of the global branch (predict matrix
and predict gamma value). The ablation results are shown in Table. 7.

Table 7: Experiments on LOL-V2-real [Wei et al.(2018)Wei, Wang, Yang, and Liu]
dataset (SSIM, PSNR) and EXDark [Loh and Chan(2019)] dataset (mAP), shows each
part’s contribution of IAT.

Local Layer [Touvron et al.(2021a)Touvron, Bojanowski, Caron, Cord, El-Nouby,
Grave, Izacard, Joulin, Synnaeve, Verbeek, et al.]’s Our Global Global
PSNR$\uparrow$ SSIM$\uparrow$ mAP$\uparrow$ Branch Norm Norm Norm (matrix)
(gamma) $\surd$ $\surd$ 18.80 0.762 75.8 $\surd$ $\surd$ 19.61 (+0.81) 0.776
(+0.014) 75.8 (+0.0) $\surd$ $\surd$ 20.01 (+1.21) 0.786 (+0.024) 76.3 (+0.5)
$\surd$ $\surd$ $\surd$ 21.95 (+3.15) 0.811 (+0.049) 76.5 (+0.7) $\surd$
$\surd$ $\surd$ 22.76 (+3.96) 0.805 (+0.043) 76.7 (+0.9) $\surd$ $\surd$
$\surd$ $\surd$ 23.50 (+4.70) 0.824 (+0.062) 77.1 (+1.3)

Table 8: Blocks Number.

$M$ $A$ | 2 | 3 | 4
---|---|---|---
2 | 22.10 | 22.85 | 22.34
3 | 22.24 | 23.50 | 22.67
4 | 22.42 | 23.00 | 23.48
|  |  |

Table 9: Channel Number.

#Channel:#Block | PSNR$\uparrow$ | SSIM$\uparrow$ | #Param.$\downarrow$
---|---|---|---
(K)
Long and Thin (12:4) | 22.60 | 0.807 | 86.22
Short and Thick (24:2) | 22.70 | 0.815 | 101.03
Ours (16:3) | 23.50 | 0.824 | 91.15
|  |  |

###  9.2 Blocks $\&$ Channels Ablation.

To evaluate the scalability of our IAT model, we try the different block
numbers and channel numbers in the local branch. We try different PEM numbers
to generate $M$ and $A$. The PSNR results on LOL-V2-real [Wei et al.(2018)Wei,
Wang, Yang, and Liu] dataset has been shown in Table.9. It shows that keeping
the same PEM number to generate $M$ and $A$ would be helpful to IAT’s
performance.

Keeping the same block number to generate $M$ and $A$, we then evaluate with
similar parameters to answer whether the local branch should be “short and
thick" or “long and thin". The local branch’s block number and channel number
are respectively set to $2/24$ and $4/12$ for comparison. The results of PSNR,
SSIM and model parameters are reported in Table. 9.

##  10 Additional Qualitative Results.

In this section we show more qualitative results on low-level vision tasks:
image enhancement (LOL (V1 $\&$ V2-real) [Wei et al.(2018)Wei, Wang, Yang, and
Liu], MIT-Adobe FiveK [Bychkovsky et al.(2011)Bychkovsky, Paris, Chan, and
Durand]) and exposure correction [Afifi et al.(2021)Afifi, Derpanis, Ommer,
and Brown].

###  10.1 Image Enhancement Results

Fig. 1 shows the image enhancement results on LOL-V1 dataset [Wei et
al.(2018)Wei, Wang, Yang, and Liu] compare with RCT [Kim et al.(2021)Kim,
Choi, Kim, and Koh] and MBLLEN [Lv et al.(2018)Lv, Lu, Wu, and Lim], Fig. 2
shows the image enhancement results on LOL-V2-real dataset [Wei et
al.(2018)Wei, Wang, Yang, and Liu] compare with MBLLEN [Lv et al.(2018)Lv, Lu,
Wu, and Lim] and KIND [Zhang et al.(2019)Zhang, Zhang, and Guo]. Fig. 3 shows
the image enhancement results on MIT-Adobe FiveK dataset [Bychkovsky et
al.(2011)Bychkovsky, Paris, Chan, and Durand] compare with Deep-UPE [Wang et
al.(2019)Wang, Zhang, Fu, Shen, Zheng, and Jia] and Deep-LPF [Moran et
al.(2020)Moran, Marza, McDonagh, Parisot, and Slabaugh]. We could see that IAT
can generate higher quality images which closer to reference target image
$I_{t}$. Meanwhile IAT also take much fewer parameters and less inference
time.

###  10.2 Exposure Correction Results

Fig. 4 shows the exposure correction results on [Afifi et al.(2021)Afifi,
Derpanis, Ommer, and Brown] dataset, we show both under-exposure and over-
exposure results of our IAT, and compare to five experts’ results. IAT also
generate high quality images, and have ability to handle under/over-exposure
at same time.

Figure 1: Qualitative comparison results on LOL-V1 [Wei et al.(2018)Wei, Wang,
Yang, and Liu] dataset, compare with enhancement methods MBLLEN [Lv et
al.(2018)Lv, Lu, Wu, and Lim] and RCT [Kim et al.(2021)Kim, Choi, Kim, and
Koh]. Figure 2: Qualitative comparison results on LOL-V2-real [Wei et
al.(2018)Wei, Wang, Yang, and Liu] dataset, compare with enhancement methods
MBLLEN [Lv et al.(2018)Lv, Lu, Wu, and Lim] and KIND [Zhang et al.(2019)Zhang,
Zhang, and Guo]. Figure 3: Qualitative comparison results on MIT-Adobe FiveK
[Bychkovsky et al.(2011)Bychkovsky, Paris, Chan, and Durand] dataset, compare
with enhancement methods Deep-UPE [Wang et al.(2019)Wang, Zhang, Fu, Shen,
Zheng, and Jia] and Deep-LPF [Moran et al.(2020)Moran, Marza, McDonagh,
Parisot, and Slabaugh]. Figure 4: Qualitative comparison results of both
under-exposure and over-exposure images on exposure correction dataset [Afifi
et al.(2021)Afifi, Derpanis, Ommer, and Brown], left is input image, second
row is output of our IAT, right are 5 experts’ results.