# YOLOP: You Only Look Once for Panoptic Driving Perception

Dong Wu Manwen Liao Weitian Zhang Xinggang Wang
Xiang Bai Wenqing Cheng Wenyu Liu
School of EIC, Huazhong University of Science & Technology
<EMAIL_ADDRESS>
###### Abstract

A panoptic driving perception system is an essential part of autonomous
driving. A high-precision and real-time perception system can assist the
vehicle in making the reasonable decision while driving. We present a panoptic
driving perception network (YOLOP) to perform traffic object detection,
drivable area segmentation and lane detection simultaneously. It is composed
of one encoder for feature extraction and three decoders to handle the
specific tasks. Our model performs extremely well on the challenging BDD100K
dataset, achieving state-of-the-art on all three tasks in terms of accuracy
and speed. Besides, we verify the effectiveness of our multi-task learning
model for joint training via ablative studies. To our best knowledge, this is
the first work that can process these three visual perception tasks
simultaneously in real-time on an embedded device Jetson TX2(23 FPS) and
maintain excellent accuracy. To facilitate further research, the source codes
and pre-trained models are released at https://github.com/hustvl/YOLOP

##  1 Introduction

Recently, extensive research on autonomous driving has revealed the importance
of panoptic driving perception system. It plays a significant role in
autonomous driving as it can extract visual information from the images taken
by the camera and assist the decision system to control the actions of the
vehicle. In order to restrict the maneuver of vehicles, the visual perception
system should be able to understand the scene and then provide the decision
system with information including: locations of the obstacles, judgements of
whether the road is drivable, the position of the lanes etc. Object detection
is usually involved in the panoptic driving perception system to help the
vehicles avoid obstacles and follow traffic rules. Drivable area segmentation
and lane detection are also needed as they are crucial for planning the
driving route of the vehicle.

(a) Input

(b) Output

Figure 1:  The input and output of our model. The purpose of our model is to
process traffic objects detection, drivable area segmentation and lane
detection simultaneously in one input image. In (b), the brown bounding boxes
indicate traffic objects, the green areas are the drivable areas, and the blue
lines represent the lane line.

For such a panoptic driving perception system, high-precision and real-time
are two most critical requirements, which are related to whether the
autonomous vehicle can make accurate and timely decision to ensure safety.
However, for practical autonomous driving system, especially the ADAS, the
computational resources are often marginal and limited. Therefore, it is very
challenging to take both requirements into account in real-world scenarios.

Many methods handle these tasks separately. For instance, Faster R-CNN [24]
and YOLOv4 [1] deal with object detection; ENet [19] and PSPNet [30] are
proposed to perform semantic segmentation. SCNN [18] and SAD-ENet [9] are used
for detecting lanes. Despite the excellent performance these methods achieve,
processing these tasks one after another takes longer time than tackling them
all at once. When deploying the panoptic driving perception system on embedded
devices commonly used in the self-driving car, limited computational resources
and latency should be taken into account. In addition, different tasks in
traffic scenes understanding often have much related information. As shown in
Figure 1, lanes are often the boundary of drivable area, and drivable area
usually closely surrounds the traffic objects. A multi-task network is more
suitable in this situation as (1) it can accelerate the image analysis process
by handling multiple tasks simultaneously rather than sequentially. (2) it can
share information among multiple tasks as multi-task network often shares the
same feature extraction backbone. Therefore, it is of essence to explore
multi-task approaches in autonomous driving.

In order to solve the multi-task problem for panoptic driving perception,
i.e., traffic object detection, drivable area segmentation and lane detection,
while obtaining high precision and fast speed, we design a simple and
efficient network architecture. We use a lightweight CNN [26] as the encoder
to extract features from the image. Then these feature maps are fed to three
decoders to complete their respective tasks. Our detection decoder is based on
the current best-performing single-stage detection network [1] for two main
reasons: (1) The single-stage detection network is faster than the two-stage
detection network. (2) The grid-based prediction mechanism of the single-stage
detector is more related to the other two semantic segmentation tasks, while
instance segmentation is usually combined with the region-based detector [7].
And we verify the two viewpoints in the experiments section. The feature map
output by the encoder incorporates semantic features of different levels and
scales, and our segmentation branch can use these feature maps to complete
pixel-wise semantic prediction excellently.

In addition to the end-to-end training strategy, we attempt some alternating
optimization paradigms which train our model step-by-step. On the one hand, we
can put unrelated tasks in different training steps to prevent inter-
limitation. On the other hand, the task trained first can guide other tasks.
So this kind of paradigm sometimes works well though cumbersome. However,
experiments show that it is unnecessary for our model as the one trained end
to end can perform well enough. Our panoptic driving perception system reaches
41 FPS on a single NVIDIA TITAN XP and 23 FPS on Jetson TX2; meanwhile, it
achieves state-of-the-art on the three tasks of the BDD100K dataset [28].

In summary, our main contributions are: (1) We put forward an efficient multi-
task network that can jointly handle three crucial tasks in autonomous
driving: object detection, drivable area segmentation and lane detection to
save computational costs and reduce inference time. Our work is the first to
reach real-time on embedded devices while maintaining state-of-the-art level
performance on the BDD100K dataset. (2) We design the ablative experiments to
verify the effectiveness of our multi-tasking scheme. It is proved that the
three tasks can be learned jointly without tedious alternating optimization.
(3) We design the ablative experiments to prove that the grid-based prediction
mechanism of detection task is more related to that of semantic segmentation
task, which is believed to provide reference for other relevant multi-task
learning research works.

##  2 Related Work

In this section, we review solutions to the above three tasks respectively,
and then introduce some related multi-task learning work. We only concentrate
on solutions based on deep learning.

###  2.1 Traffic Object Detection

In recent years, with the rapid development of deep learning, many prominent
object detection algorithms have emerged. Current mainstream object detection
algorithms can be divided into two-stage methods and one-stage methods.

Two-stage methods complete the detection task in two steps. First, regional
proposals are obtained, and then features in the regional proposals are used
to locate and classify the objects. The generation of regional proposals has
gone through several stages of development [5, 4, 24, 2].

The SSD-series [14] and YOLO-series algorithms are milestones among one-stage
methods. This kind of algorithm performs bounding box regression and object
classification simultaneously. YOLO [21] divides the picture into S$\times$S
grids instead of extracting regional proposals with the RPN network, which
significantly accelerates the detection speed. YOLO9000 [22] introduces the
anchor mechanism to improve the recall of detection. YOLOv3 [23] uses the
feature pyramid network structure to achieve multi-scale detection. YOLOv4 [1]
further improves the detection performance by refining the network structure,
activation function, loss function and applying abundant data augmentation.

###  2.2 Drivable Area Segmentation

Due to the rapid development of deep learning, a number of CNN-based methods
have made great success in semantic segmentation area, and they can be applied
in drivable area segmentation task to provide pixel-level results. FCN [15]
firstly introduces fully convolutional network to semantic segmentation.
Despite the skip-connection refinement, its performance is still limited by
low-resolution output. PSPNet [30] comes up with the pyramid pooling module to
extract features in various scales to enhance its performance. Besides
accuracy, speed is also a key element in evaluating this task. In order to
achieve real-time inference speed, ENet [19] reduces size of the feature maps.
Recently, multitask learning is introduced to deal with this task, EdgeNet [6]
combine edge detection with drivable area segmentation task to obtain more
accurate segmentation results without compromising the inference speed.

###  2.3 Lane Detection

In lane detection, there are lots of innovative researches based on deep
learning. [17] constructs a dual-branch network to perform semantic
segmentation and pixel embedding on images. It further clusters the dual-
branch features to achieve lane instance segmentation. SCNN [18] proposes
slice-by-slice convolution, which enables the message to pass between pixels
across rows and columns in a layer, but this convolution is very time-
consuming. Enet-SAD [9] uses self attention distillation method, which enables
low-level feature maps to learn from high-level feature maps. This method
improves the performance of the model while keeping the model lightweight.

###  2.4 Multi-task Approaches

The goal of multi-task learning is to learn better representations through
shared information among multiple tasks. Especially, a CNN-based multitask
learning method can also achieve convolutional sharing of the network
structure. Mask R-CNN [7] extends Faster R-CNN by adding a branch for
predicting object mask, which combines instance segmentation and object
detection tasks effectively, and these two tasks can promote each other’s
performance. LSNet [3] summarizes object detection, instance segmentation and
pose estimation as location-sensitive visual recognition and uses a unified
solution to handle these tasks. With a shared encoder and three independent
decoders, MultiNet [25] completes the three scene perception tasks of scene
classification, object detection and segmentation of the driving area
simultaneously. DLT-Net [20] inherits the encoder-decoder structure, and
contributively constructs context tensors between sub-task decoders to share
designate information among tasks. [29] puts forward mutually interlinked sub-
structures between lane area segmentation and lane boundary detection.
Meanwhile, it proposes a novel loss function to constrain the lane line to the
outer contour of the lane area so that they’re going to overlap geometrically.
However, this prior assumption also limits its application as it only works
well on scenarios where the lane line tightly wraps the lane area. What’s
more, the training paradigm of multitask model is also worth thinking about.
[10] states that the joint training is appropriate and beneficial only when
all those tasks are indeed related; otherwise, it is necessary to adopt
alternating optimization. So Faster R-CNN [24] adopts a pragmatic 4-step
training algorithm to learn shared features. This paradigm sometimes may be
helpful, but mostly it is tedious.

##  3 Methodology

We put forward a simple and efficient feed-forward network that can accomplish
traffic object detection, drivable area segmentation and lane detection tasks
altogether. As shown in Figure 2, our panoptic driving perception single-shot
network, termed as YOLOP, contains one shared encoder and three subsequent
decoders to solve specific tasks. There are no complex and redundant shared
blocks between different decoders, which reduces computational consumption and
allows our network to be easily trained end-to-end.

Figure 2: The architecture of YOLOP. YOLOP shares one encoder and combines
three decoders to solve different tasks. The encoder consists of a backbone
and a neck.

###  3.1 Encoder

Our network shares one encoder, which is composed of a backbone network and a
neck network.

####  3.1.1 Backbone

The backbone network is used to extract the features of the input image.
Usually, some classic image classification networks serve as the backbone. Due
to the excellent performance of YOLOv4 [1] on object detection, we choose
CSPDarknet [26] as the backbone, which solves the problem of gradient
duplication during optimization [27]. It supports feature propagation and
feature reuse which reduces the amount of parameters and calculations.
Therefore, it is conducive to ensuring the real-time performance of the
network.

####  3.1.2 Neck

The neck is used to fuse the features generated by the backbone. Our neck is
mainly composed of Spatial Pyramid Pooling (SPP) module [8] and Feature
Pyramid Network (FPN) module [11]. SPP generates and fuses features of
different scales, and FPN fuses features at different semantic levels, making
the generated features contain multiple scales and multiple semantic level
information. We adopt the method of concatenation to fuse features in our
work.

###  3.2 Decoders

The three heads in our network are specific decoders for the three tasks.

####  3.2.1 Detect Head

Similar to YOLOv4, we adopt an anchor-based multi-scale detection scheme.
Firstly, we use a structure called Path Aggregation Network (PAN), a bottom-up
feature pyramid network [13]. FPN transfers semantic features top-down, and
PAN transfers positioning features bottom-up. We combine them to obtain a
better feature fusion effect, and then directly use the multi-scale fusion
feature maps in the PAN for detection. Then, each grid of the multi-scale
feature map will be assigned three prior anchors with different aspect ratios,
and the detection head will predict the offset of position and the scaling of
the height and width, as well as the corresponding probability of each
category and the confidence of the prediction.

####  3.2.2 Drivable Area Segment Head & Lane Line Segment Head

Drivable area segment head and Lane line Segment head adopt the same network
structure. We feed the bottom layer of FPN to the segmentation branch, with
the size of $(W/8,H/8,256)$. Our segmentation branch is very simple. After
three upsampling processes, we restore the output feature map to the size of
$(W,H,2)$, which represents the probability of each pixel in the input image
for the drivable area/lane line and the background. Because of the shared SPP
in the neck network, we do not add an extra SPP module to segment branches
like others usually do [30], which brings no improvement to the performance of
our network. Additionally, we use the Nearest Interpolation method in our
upsampling layer to reduce computation cost instead of deconvolution. As a
result, not only do our segment decoders gain high precision output, but also
be very fast during inference.

###  3.3 Loss Function

Since there are three decoders in our network, our multi-task loss contains
three parts. As for the detection loss $\mathcal{L}_{det}$, it is a weighted
sum of classification loss, object loss and bounding box loss as in equation
1.

$\mathcal{L}_{det}=\alpha_{1}\mathcal{L}_{class}+\alpha_{2}\mathcal{L}_{obj}+\alpha_{3}\mathcal{L}_{box},$
(1)

where $\mathcal{L}_{class}$ and $\mathcal{L}_{obj}$ are focal loss [12], which
is utilized to reduce the loss of well-classified examples, thus forces the
network to focus on the hard ones. $\mathcal{L}_{class}$ is used for
penalizing classification and $\mathcal{L}_{obj}$ for the confidence of one
prediction. $\mathcal{L}_{box}$ is $\mathcal{L}_{CIoU}$ [31], which takes
distance, overlap rate, the similarity of scale and aspect ratio between the
predicted box and ground truth into consideration.

Both of the loss of drivable area segmentation $\mathcal{L}_{da-seg}$ and lane
line segmentation $\mathcal{L}_{ll-seg}$ contain Cross Entropy Loss with
Logits $\mathcal{L}_{ce}$, which aims to minimize the classification errors
between pixels of network outputs and the targets. It is worth mentioning that
IoU loss: $\mathcal{L}_{IoU}=1-\frac{TP}{TP+FP+FN}$ is added to
$\mathcal{L}_{ll-seg}$ as it is especially efficient for the prediction of the
sparse category of lane lines. $\mathcal{L}_{da}$ and $\mathcal{L}_{ll-seg}$
are defined as equation (2), (3) respectively.

$\mathcal{L}_{da-seg}=\mathcal{L}_{ce},$ (2) $\mathcal{L}_{ll-
seg}=\mathcal{L}_{ce}+\mathcal{L}_{IoU}.$ (3)

In conclusion, our final loss is a weighted sum of the three parts all
together as in equation (4).

$\mathcal{L}_{all}=\gamma_{1}\mathcal{L}_{det}+\gamma_{2}\mathcal{L}_{da-
seg}+\gamma_{3}\mathcal{L}_{ll-seg},$ (4)

where $\alpha_{1},\alpha_{2},\alpha_{3},\gamma_{1},\gamma_{2},\gamma_{3}$ can
be tuned to balance all parts of the total loss.

###  3.4 Training Paradigm

We attempt different paradigms to train our model. The simplest one is
training end to end, and then three tasks can be learned jointly. This
training paradigm is useful when all tasks are indeed related. In addition,
some alternating optimization algorithms also have been tried, which train our
model step by step. In each step, the model can focus on one or multiple
related tasks regardless of those unrelated. Even if not all tasks are
related, our model can still learn adequately on each task with this paradigm.
And Algorithm 1 illustrates the process of one step-by-step training method.

Algorithm 1 One step-by-step Training Method. First, we only train Encoder and
Detect head. Then we freeze the Encoder and Detect head as well as train two
Segmentation heads. Finally, the entire network is trained jointly for all
three tasks.

1:Target neural network $\mathcal{F}$ with parameter group:

2: $\Theta=\\{\theta_{enc},\theta_{det},\theta_{seg}\\}$;

3: Training set: $\mathcal{T}$;

4: Threshold for convergence: $thr$;

5: Loss function: $\mathcal{L}_{all}$

6:Well-trained network: $\mathcal{F}(\mathbf{x};\Theta)$

7:procedure Train($\mathcal{F}$, $\mathcal{T}$)

8: repeat

9: Sample a mini-batch $(\mathbf{x}_{s},\mathbf{y}_{s})$ from training set
$\mathcal{T}$.

10:
$\ell\leftarrow\mathcal{L}_{all}(\mathcal{F}(\mathbf{x}_{s};\Theta),\mathbf{y}_{s})$

11: $\Theta\leftarrow\arg\min_{\Theta}\ell$

12: until $\ell<thr$

13:end procedure

14:$\Theta\leftarrow\Theta\setminus\\{\theta_{seg}\\}$ // Freeze parameters of
two Segmentation heads.

15:Train($\mathcal{F}$, $\mathcal{T}$)

16:$\Theta\leftarrow\Theta\cup\\{\theta_{seg}\\}\setminus\\{\theta_{det},\theta_{enc}\\}$
// Freeze parameters of Encoder and Detect head and activate parameters of two
Segmentation heads.

17:Train($\mathcal{F}$, $\mathcal{T}$)

18:$\Theta\leftarrow\Theta\cup\\{\theta_{det},\theta_{enc}\\}$ // Activate all
parameters of the neural network.

19:Train($\mathcal{F}$, $\mathcal{T}$)

20:return Trained network $\mathcal{F}(\mathbf{x};\Theta)$

##  4 Experiments

###  4.1 Setting

####  4.1.1 Dataset Setting

The BDD100K dataset [28] supports the research of multi-task learning in the
field of autonomous driving. With 100k frames of pictures and annotations of
10 tasks, it is the largest driving video dataset. As the dataset has the
diversity of geography, environment, and weather, the algorithm trained on the
BDD100k dataset is robust enough to migrate to a new environment. Therefore,
we choose the BDD100k dataset to train and evaluate our network. The BDD100K
dataset has three parts, training set with 70K images, validation set with 10K
images, and test set with 20K images. Since the label of the test set is not
public, we evaluate our network on the validation set.

####  4.1.2 Implementation Details

In order to enhance the performance of our model, we empirically adopt some
practical techniques and methods of data augmentation.

With the purpose of enabling our detector to get more prior knowledge of the
objects in the traffic scene, we use the k-means clustering algorithm to
obtain prior anchors from all detection frames of the dataset. We use Adam as
the optimizer to train our model and the initial learning rate, ${\beta_{1}}$,
and ${\beta_{2}}$ are set to be 0.001, 0.937, and 0.999 respectively. Warm-up
and cosine annealing are used to adjust the learning rate during the training,
which aim at leading the model to converge faster and better [16].

We use data augmentation to increase the variability of images so as to make
our model robust in different environments. Photometric distortions and
geometric distortions are taken into consideration in our training scheme. For
photometric distortions, we adjust the hue, saturation and value of images. We
use random rotating, scaling, translating, shearing, and left-right flipping
to process images to handle geometric distortions.

####  4.1.3 Experimental Setting

We select some excellent multi-task networks and networks that focus on a
single task to compare with our network. Both MultiNet and DLT-Net handle
multiple panoptic driving perception tasks, and they have achieved great
performance in object detection and drivable area segmentation tasks on the
BDD100k dataset. Faster-RCNN is an outstanding representative of the two-stage
object detection network. YOLOv5 is the single-stage network that achieves
state-of-the-art performance on the COCO dataset. PSPNet achieves splendid
performance on semantic segmentation task with its superior ability to
aggregate global information. We retrain the above networks on the BDD100k
dataset and compare them with our network on object detection and drivable
area segmentation tasks. Since there is no suitable existing multi-task
network that processes lane detection task on the BDD100K dataset, we compare
our network with Enet [19], SCNN and Enet-SAD, three advanced lane detection
networks. Besides, the performance of the joint training paradigm is compared
with alternating training paradigms of many kinds. Moreover, we compare the
accuracy and speed of our multi-task model trained to handle multiple tasks
with the one trained to perform a specific task. Furthermore, we compare the
performance of semantic segmentation task combined with single-stage detection
task and two-stage detection task. Following [9], we resize images in BDD100k
dataset from 1280$\times$720$\times$3 to 640$\times$384$\times$3\. All control
experiments follow the same experimental settings and evaluation metrics, and
all experiments are run on NVIDIA GTX TITAN XP.

###  4.2 Result

In this section, we just simply train our model end to end and then compare it
with other representative models on all three tasks.

####  4.2.1 Traffic Object Detection Result

Visualization of the traffic objects detection is shown in Figure 3. Since the
Multinet and DLT-Net can only detect vehicles, we only consider the vehicle
detection results of five models on the BDD100K dataset. As shown in Table 1,
we use Recall and mAP50 as the evaluation metric of detection accuracy. Our
model exceeds Faster R-CNN, MultiNet, and DLT-Net in detection accuracy, and
is comparable to YOLOv5s that actually uses more tricks than ours. Moreover,
our model can infer in real time. YOLOv5s is faster than ours because it does
not have the lane line segment head and drivable area segment head.

Figure 4 shows the qualitative comparison between Faster R-CNN and YOLOP. Due
to the information share of multi-task, the prediction results of YOLOP are
more reasonable. For example, YOLOP will not misidentify the objects far from
the road as vehicle. Moreover, the examples of false negative are much less
and the bounding boxes are more accurate.

(a) Day-time result

(b) Night-time result

Figure 3: Visualization of the traffic objects detection results of YOLOP. Top
Row: Traffic objects detection results in day-time scenes. Bottom row: Traffic
objects detection results in night scenes.

(a) Results of Faster R-CNN

(b) Results of YOLOP

Figure 4: Comparison between the traffic objects detection results of Faster R-CNN and YOLOP. Top Row: Traffic objects detection results of Faster R-CNN. Bottom row: Traffic objects detection results of YOLOP. The green bounding boxes are the detected correct vehicles. The yellow dotted bounding boxes are the false negative. The red bounding boxes indicate the false positive. Network | Recall(%) | mAP50(%) | Speed(fps)
---|---|---|---
MultiNet | 81.3 | 60.2 | 8.6
DLT-Net | 89.4 | 68.4 | 9.3
Faster R-CNN | 81.2 | 64.9 | 8.8
YOLOv5s | 86.8 | 77.2 | 82
YOLOP (ours) | 89.2 | 76.5 | 41
Table 1: Traffic Object Detection Results: comparing the proposed YOLOP with
state-of-the-art detectors.

####  4.2.2 Drivable Area Segmentation Result

Visualization results of the drivable area segmentation can be seen in Figure
5. In this paper, both “area/drivable” and “area/alternative” classes in
BDD100K dataset are categorized as ”Drivable area” without distinction. Our
model only needs to distinguish the drivable area and the background in the
image. mIoU is used to evaluate the segmentation performance of different
models. The results are shown in Table 2. It can be seen that our model
outperforms MultiNet, DLT-Net and PSPNet by 19.9%, 20.2%, and 1.9%,
respectively. Furthermore, our inference speed is 4 to 5 times faster than
theirs.

The comparison between results of PSPNet and YOLOP is showed in Figure 6. Both
PSPNet and YOLOP have perfomed well in this task. But YOLOP is significantly
better at segmenting the edge areas that next to vehicles or lane lines. We
think it’s mainly because that both two other tasks provide the edge
information for this task. Meanwhile, YOLOP makes fewer stupid mistakes, such
as misjudging the opposite lane area as drivable area.

Network | mIoU(%) | Speed(fps)
---|---|---
MultiNet | 71.6 | 8.6
DLT-Net | 71.3 | 9.3
PSPNet | 89.6 | 11.1
YOLOP (ours) | 91.5 | 41
Table 2: Drivable Area Segmentation Results: Comparing the proposed YOLOP with
state-of-the-art drivable area segmentation or semantic segmentation methods.

(a) Day-time result

(b) Night-time result

Figure 5: Visualization of the drivable area segmentation results of YOLOP.
Top Row: Drivable area segmentation results in day-time scenes. Bottom row:
Drivable area segmentation results in night scenes.

(a) Results of PSPNet

(b) Results of YOLOP

Figure 6: Comparison between the drivable area segmentation results of PSPNet
and YOLOP. Top Row: Drivable area segmentation results of PSPNet. Bottom row:
Drivable area segmentation results of YOLOP. The yellow ellipses are the false
negative. The red ellipses indicate the false positive.

####  4.2.3 Lane Detection Result

The visualization results of lane detection can be seen in Figure 7. The lane
lines in BDD100K dataset are labeled with two lines, so it is very tricky to
directly use the annotation. The experimental settings follow the [9] in order
to compare expediently. First of all, we calculate the center lines based on
the two-line annotations. Then we draw the lane line of the training with
width set to 8 pixels while keeping the lane line width of the test set as 2
pixels. We use pixel accuracy and IoU of lanes as evaluation metrics. As shown
in the Table 3, the performance of our model dramatically exceeds the other
three models.

Figure 8 shows the comparison of Lane line detection results of ENet-SAD and
YOLOP. The segmentation results of YOLOP is more accurate and continuous than
ENet-SAD obviously. With the imformation shared by the other two tasks, YOLOP
will not mistake some areas where some vehicles are located or driveable as
lane lines, but Enet-SAD always does.

(a) Day-time result

(b) Night-time result

Figure 7: Visualization of the lane detection results of YOLOP. Top Row: Lane
detection results in day-time scenes. Bottom row: Lane detection results in
night scenes.

(a) Results of ENet-SAD

(b) Results of YOLOP

Figure 8: Comparison between the lane detection results of ENet-SAD and YOLOP. Top Row: Lane detection results of ENet-SAD. Bottom row: Lane detection results of YOLOP. The yellow ellipses are the false negative. The red ellipses indicate the false positive. Network | Accuracy(%) | IoU(%) | Speed(fps)
---|---|---|---
ENet | 34.12 | 14.64 | 100
SCNN | 35.79 | 15.84 | 19.8
ENet-SAD | 36.56 | 16.02 | 50.6
YOLOP (ours) | 70.50 | 26.20 | 41
Table 3: Lane Detection Results: comparing the proposed YOLOP with state-of-
the-art lane detection methods.

###  4.3 Ablation Studies

We designed the following three ablation experiments to further illustrate the
effectiveness of our scheme. All the evaluation metrics in this section are
consistent with above.

####  4.3.1 End-to-end v.s. Step-by-step

In Table 4, we compare the performance of joint training paradigm with
alternating training paradigms of many kinds 111E, D, S and W refer to
Encoder, Detect head, two Segment heads and whole network. So the Algorithm 1
can be marked as ED-S-W, and the same for others.. Obviously, our model has
performed very well enough through end-to-end training, so there is no need to
perform alternating optimization. However, it is interesting that the paradigm
training detection task firstly seems to perform better. We think it is mainly
because our model is closer to a complete detection model and the model is
harder to converge when performing detection tasks. What’s more, the paradigm
consist of three steps slightly outperforms that with two steps. Similar
alternating training can be run for more steps, but we have observed
negligible improvements.

Training method |  Recall(%) |  AP(%) |  mIoU(%) |  Accuracy(%) |  IoU(%)
---|---|---|---|---|---
ES-W |  87.0 |  75.3 |  90.4 |  66.8 |  26.2
ED-W |  87.3 |  76.0 |  91.6 |  71,2 |  26.1
ES-D-W |  87.0 |  75.1 |  91.7 |  68.6 |  27.0
ED-S-W |  87.5 |  76.1 |  91.6 |  68.0 |  26.8
End-to-end |  89.2 |  76.5 |  91.5 |  70.5 |  26.2
Table 4: Panoptic driving perception results: the end-to-end scheme v.s.
different step-by-step schemes.

####  4.3.2 Multi-task v.s. Single task

To verify the effectiveness of our multi-task learning scheme, we compare the
performance of the multi-task scheme and single task scheme. On the one hand,
we train our model to perform 3 tasks simultaneously. On the other hand, we
train our model to perform traffic object detection, drivable area
segmentation, and lane line segmentation tasks separately. Table 5 shows the
comparison of the performance of these two schemes on each specific task. It
can be seen that our model adopts the multi-task scheme to achieve performance
is close to that of focusing on a single task. More importantly, the multitask
model can save a lot of time compared to executing each task individually.

####  4.3.3 Region-based v.s. Grid-based

To verify the viewpoint that the grid-based prediction mechanism is more
related to the two semantic segmentation tasks than the region-based
prediction mechanism. We extend Faster R-CNN by adding two semantic segment
heads to perform three tasks in parallel as our model did, and we call such a
new model R-CNNP. We train both YOLOP and R-CNNP to (i) perform detection task
and two segmentation tasks separately and (ii) three tasks simultaneously. In
both two experiments above, the two segmentation tasks are trained jointly as
there is no need to consider the interaction between them. All the
experimental settings are the same, and the results are shown in Table 6. In
the R-CNNP framework, the performance of multi-task training is much worse
compared with training the detection task and semantic segmentation tasks
separately. Obviously, the combination of two kinds of tasks conflicts in
R-CNNP framework. But there is no such problem in our YOLOP framework, the
performance of multi-task training is equal to that of focusing only on
detection or semantic segmentation task. Thus we hold the opinion that this is
due to the detection head of YOLOP, like two other semantic segmentation
heads, directly perform global classification or regression tasks on the whole
feature map output by Encoder, so they are similar and related in terms of
prediction mechanism. Nevertheless, the detection head of R-CNNP needs to
select region proposals first, and then perform prediction on the feature maps
of each individual proposals, which is quite different from the global
prediction mechanism of semantic segmentation. In addition, R-CNNP is far
behind YOLOP in terms of inference speed. Therefore, our framework is a better
choice for joint training detection and segmentation tasks.

Training method | Recall(%) | AP(%) | mIoU(%) | Accuracy(%) | IoU(%) | Speed(ms/frame)
---|---|---|---|---|---|---
Det(only) | 88.2 | 76.9 | - | - | - | 15.7
Da-Seg(only) | - | - | 92.0 | - | - | 14.8
Ll-Seg(only) | - | - | - | 79.6 | 27.9 | 14.8
Multitask | 89.2 | 76.5 | 91.5 | 70.5 | 26.2 | 24.4
Table 5: Panoptic driving perception results: multi-task learning v.s. single task learning. Training method | Recall(%) | AP(%) | mIoU(%) | Accuracy(%) | IoU(%) | Speed(ms/frame)
---|---|---|---|---|---|---
R-CNNP | Det(only) | 79.0 | 67.3 | - | - | - | -
Seg(only) | - | - | 90.2 | 59.5 | 24.0 | -
Multitask | 77.2(-1.8) | 62.6(-4.7) | 86.8(-3.4) | 49.8(-9.7) | 21.5(-2.5) | 103.3
YOLOP | Det(only) | 88.2 | 76.9 | - | - | - | -
Seg(only) | - | - | 91.6 | 69.9 | 26.5 | -
Multitask | 89.2(+1.0) | 76.5(-0.4) | 91.5(-0.1) | 70.5(+0.6) | 26.2(-0.3) | 24.4
Table 6: Panoptic driving perception results: Grid-based v.s. Region-based.

##  5 Conclusion

In this paper, we put forward a brand-new, simple and efficient network, which
can simultaneously handle three driving perception tasks of object detection,
drivable area segmentation and lane detection and can be trained end-to-end.
Our model performs exceptionally well on the challenging BDD100k dataset,
achieving or greatly exceeding state-of-the-art level on all three tasks. And
it is the first to realize real-time reasoning on embedded device Jetson TX2,
which ensures that our network can be used in real-world scenarios. Moreover,
we have verified that the grid-based prediction mechanism is more related to
that of semantic segmentation task. which may be of certain reference
significance to similar multi-task learning research works.

Currently, although our multi-task network can be trained end-to-end without
compromising the performance of each other, we hope to improve the performance
of those tasks with more appropriate paradigm for multitask learning.
Furthermore, our model is limited in three tasks, more tasks related with
autonomous driving perception system such as depth estimation can be added in
our future frame work to make the whole system more complete and practical.

## References

  * [1] Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao.  Yolov4: Optimal speed and accuracy of object detection.  arXiv preprint arXiv:2004.10934, 2020.
  * [2] Jifeng Dai, Yi Li, Kaiming He, and Jian Sun.  R-fcn: Object detection via region-based fully convolutional networks.  arXiv preprint arXiv:1605.06409, 2016.
  * [3] Kaiwen Duan, Lingxi Xie, Honggang Qi, Song Bai, Qingming Huang, and Qi Tian.  Location-sensitive visual recognition with cross-iou loss.  arXiv preprint arXiv:2104.04899, 2021.
  * [4] Ross Girshick.  Fast r-cnn.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1440–1448, 2015.
  * [5] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik.  Rich feature hierarchies for accurate object detection and semantic segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 580–587, 2014.
  * [6] Hsiang-Yu Han, Yu-Chi Chen, Pei-Yung Hsiao, and Li-Chen Fu.  Using channel-wise attention for deep cnn based real-time semantic segmentation with class-aware edge information.  IEEE Transactions on Intelligent Transportation Systems, 22(2):1041–1051, 2020.
  * [7] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2961–2969, 2017.
  * [8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Spatial pyramid pooling in deep convolutional networks for visual recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(9):1904–1916, 2015.
  * [9] Yuenan Hou, Zheng Ma, Chunxiao Liu, and Chen Change Loy.  Learning lightweight lane detection cnns by self attention distillation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1013–1021, 2019.
  * [10] Zhuoliang Kang, Kristen Grauman, and Fei Sha.  Learning with whom to share in multi-task feature learning.  In ICML, 2011.
  * [11] Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.  Feature pyramid networks for object detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2117–2125, 2017.
  * [12] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2980–2988, 2017.
  * [13] Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia.  Path aggregation network for instance segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8759–8768, 2018.
  * [14] Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg.  Ssd: Single shot multibox detector.  In European Conference on Computer Vision, pages 21–37. Springer, 2016.
  * [15] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3431–3440, 2015.
  * [16] Ilya Loshchilov and Frank Hutter.  Sgdr: Stochastic gradient descent with warm restarts.  arXiv preprint arXiv:1608.03983, 2016.
  * [17] Davy Neven, Bert De Brabandere, Stamatios Georgoulis, Marc Proesmans, and Luc Van Gool.  Towards end-to-end lane detection: an instance segmentation approach.  In 2018 IEEE Intelligent Vehicles Symposium (IV), pages 286–291. IEEE, 2018.
  * [18] Xingang Pan, Jianping Shi, Ping Luo, Xiaogang Wang, and Xiaoou Tang.  Spatial as deep: Spatial cnn for traffic scene understanding.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  * [19] Adam Paszke, Abhishek Chaurasia, Sangpil Kim, and Eugenio Culurciello.  Enet: A deep neural network architecture for real-time semantic segmentation.  arXiv preprint arXiv:1606.02147, 2016.
  * [20] Yeqiang Qian, John M Dolan, and Ming Yang.  Dlt-net: Joint detection of drivable areas, lane lines, and traffic objects.  IEEE Transactions on Intelligent Transportation Systems, 21(11):4670–4679, 2019.
  * [21] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi.  You only look once: Unified, real-time object detection.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 779–788, 2016.
  * [22] Joseph Redmon and Ali Farhadi.  Yolo9000: better, faster, stronger.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7263–7271, 2017.
  * [23] Joseph Redmon and Ali Farhadi.  Yolov3: An incremental improvement.  arXiv preprint arXiv:1804.02767, 2018.
  * [24] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  arXiv preprint arXiv:1506.01497, 2015.
  * [25] Marvin Teichmann, Michael Weber, Marius Zoellner, Roberto Cipolla, and Raquel Urtasun.  Multinet: Real-time joint semantic reasoning for autonomous driving.  arXiv preprint arXiv:1612.07695, 2016.
  * [26] Chien-Yao Wang, Alexey Bochkovskiy, and Hong-Yuan Mark Liao.  Scaled-yolov4: Scaling cross stage partial network.  arXiv preprint arXiv:2011.08036, 2020.
  * [27] Chien-Yao Wang, Hong-Yuan Mark Liao, Yueh-Hua Wu, Ping-Yang Chen, Jun-Wei Hsieh, and I-Hau Yeh.  Cspnet: A new backbone that can enhance learning capability of cnn.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 390–391, 2020.
  * [28] Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, and Trevor Darrell.  Bdd100k: A diverse driving video database with scalable annotation tooling.  arXiv preprint arXiv:1805.04687, 2(5):6, 2018.
  * [29] Jie Zhang, Yi Xu, Bingbing Ni, and Zhenyu Duan.  Geometric constrained joint lane segmentation and lane boundary detection.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 486–502, 2018.
  * [30] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia.  Pyramid scene parsing network.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
  * [31] Zhaohui Zheng, Ping Wang, Wei Liu, Jinze Li, Rongguang Ye, and Dongwei Ren.  Distance-iou loss: Faster and better learning for bounding box regression.  In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12993–13000, 2020.


11institutetext: Computational Media Innovation Centre, Victoria University of
Wellington, New Zealand
11email: {josh.weir, j.zhao, andrew.chalmers<EMAIL_ADDRESS>
https://www.wgtn.ac.nz/cmic

# Deep Portrait Delighting

Joshua Weir*  0000-0002-4445-7278 Junhong Zhao  0000-0001-7031-3828 Andrew
Chalmers  0000-0001-6457-7341 Taehyun Rhee  0000-0002-6150-0637

###### Abstract

We present a deep neural network for removing undesirable shading features
from an unconstrained portrait image, recovering the underlying texture. Our
training scheme incorporates three regularization strategies: masked loss, to
emphasize high-frequency shading features; soft-shadow loss, which improves
sensitivity to subtle changes in lighting; and shading-offset estimation, to
supervise separation of shading and texture. Our method demonstrates improved
delighting quality and generalization when compared with the state-of-the-art.
We further demonstrate how our delighting method can enhance the performance
of light-sensitive computer vision tasks such as face relighting and semantic
parsing, allowing them to handle extreme lighting conditions.

Keywords: uniform lighting, texture recovery, shadow removal, portrait

††* Corresponding author

##  1 Introduction

Image delighting is a form of image manipulation that aims to remove unwanted
lighting features from images to recover the underlying texture. Recovering
this underlying texture benefits many light sensitive computer vision tasks
such as face recognition, parsing and relighting. Delighting has seen much
research within these application areas, but they have only addressed the
delighting problem implicitly in their pipeline. None of them have a dedicated
delighting solution independent from their application. For example, the
_relighting_ application area has seen rapid progress with deep neural
networks able to render convincing non-lambertian shading effects [19, 32, 44,
31, 41, 48]. However, the _delighting_ phase is often abstracted. As a
consequence, shading features present in the input image often propagate into
the output as distortions. In extreme cases, they can alter the identity and
perceived structure of portraits, affecting face recognition [12, 3].

Figure 1: Given a portrait image (top row), we perform _delighting_ (bottom
row): removing undesirable lighting characteristics and reconstructing the
image under uniform lighting.

The portrait delighting problem is inherently more difficult due to the
ambiguous combination of lighting and reflectance determining the colour of
any pixel. Theoretically, supervised image-to-image translation pipelines can
learn this problem given enough labeled image pairs, but acquiring a
representative dataset for this task is laborious and expensive. As a result,
most researchers are dependant on 3D renderings of scanned human subjects [21,
23, 36], which consequently leads to poor generalization to real-world images
due to oversimplified reflectance and geometry modelling. On the other hand,
real human image datasets available to the public suffer from either lighting
or subject under-representation. The CMU Multi-PIE dataset [14] for instance,
records only 19 lighting conditions for over 300 individuals, while the
Extended Yale-B dataset [25] provides 64 lightings for only 38 individuals.

Even with high-quality data [41, 32], many challenges still persist,
particularly for images exhibiting complex lighting features like reflections
and hard shadow boarders; these features have small pixel densities, making
common pixel-wise loss functions like $\ell_{1}$ and $\ell_{2}$ ineffective;
also, these features are highly embedded in the underlying texture we wish to
preserve, making them difficult to remove without losing high-frequency
details such as freckles and facial hair. Most recent relighting methods that
utilize a delighting phase usually incorporate standard network architectures
and loss functions which do not directly address the problems of dataset
sparsity or non-lambertian lighting.

We present a fully-supervised portrait delighting method that takes an upper-
body portrait lit under an arbitrary illumination, and outputs its
reconstruction under uniform white lighting (see Fig. 1).

We localize high-frequency lighting effects in our training data using a
guided-filter technique, and incorporate these into our training pipeline
using a masked loss function inspired by [19] to emphasize small but visually
significant regions of the image.

We estimate a shading offset image, which is the difference between the input
image and the ground-truth de-lit image. This facilitates learning of lighting
features directly, and greatly improves colour consistency. We also synthesize
soft-shaded images from our training data and utilize them in training,
applying a small regularization loss to their outputs. This alleviates our
dataset bias toward directional lightings, allowing us to remove both hard and
soft shadows. We demonstrate how our framework benefits the applications of
semantic parsing and relighting.

To summarize, our main contributions are as follows:

  * •

A novel portrait delighting method that can recover the underlying texture of
portraits illuminated under a wide range of complex lighting environments.

  * •

Three novel loss functions: _shading-offset_ loss, _soft-shadow_ loss and
_masked_ loss that improve our models robustness to unseen lighting
environments while preserving image detail.

  * •

Our delighting method can serve as a data normalization tool for improving
light-sensitive computer vision tasks such as face relighting and semantic
parsing.

##  2 Related Work

Much work has been made towards removing disrupting light features (e.g. dark
shadows, specularities) from faces primarily to enhance face recognition
systems [13, 6], and to increase image quality [50, 30, 5]. Pioneering works
in this field propose optimization methods using Morphable Face models [4, 11,
43, 2, 7]. However, relying on parametric models limits their ability to
capture non-facial or high-frequency details. Recent deep-learning methods
tackle this problem using feature-wise perceptual losses [26] or closed-loop
GANs [15] to recover facial details. A drawback of all these methods however
is that they either focus exclusively on face regions, perform only one aspect
of the delighting process (e.g. shadow removal [50], camera-flash removal
[5]), or use front-facing illumination as the ground-truth [30, 15], ignoring
the sharp reflections this causes.

GAN inversion methods [1, 10, 27, 46] enable face editing by projecting images
into the latent space of a pre-trained GAN, disentangling lighting, identity,
pose and expression attributes such that they can be independently
manipulated. These methods can effectively remove sharp shadows and specular
reflections, but their reconstructions don’t often preserve image content that
isn’t constrained by editing attributes, particularly high-frequency details
such as freckles, and non-face components like clothing.

Intrinsic decomposition methods _delight_ by disentangling face [36, 34, 37]
or full-body [21, 23] images into geometry, albedo, and lighting via separate
convolutional neural networks, where _relighting_ can be performed by re-
rendering with modified lighting. Estimation errors usually occur, such as
when specularities become embedded in the albedo, or when hard shadows are
predicted as geometry features. This becomes a bigger problem when relying on
synthetic training data with simplified reflectance models, so Sengupta et al.
[36] proposed a semi-supervised learning framework using reconstruction loss
on real images, but conflation between shading, geometry and texture is still
present when there’s a significant domain-gap between the real and synthetic
data.

Other deep learning methods perform portrait relighting directly [41, 51, 48,
19, 40] by co-opting standard encoder-decoder architectures like U-Net [35].
In this manner, both the _delighting_ and _relighting_ processes are a black
box represented by network activations, with supervision applied only to the
re-lit image. While this generally leads to more stable and expressive
relightings than intrinsic decomposition [41], the neural representation of
the delighting task lacks any meaningful form to facilitate supervision. Some
researchers attempt to supervise the delighting process by imposing feature-
space losses on the network bottleneck [51, 48]. While this helps recover
global details such as colour and identity, it often preserves local artifacts
such as those caused by facial shadows.

The most recent relighting methods utilize an explicit _delighting_ phase as a
preliminary step [31, 44, 32], which is the inspiration for our work. Pandey
et al. [32] use a least-squares GAN [28] with VGG-perceptual loss [20] in
their albedo prediction for accurate detail reconstruction, but colour
inconsistencies arise when presented with data unseen in training (e.g.
different clothing patterns). Wang et al. [44] supervise the delighting
process by predicting the source lighting and a face-parsing map for more
robust texture recovery, but their method struggles to remove high-frequency
shading artifacts caused by strong directional light.

##  3 Method

This section overviews how we synthesize training images (Sec. 3.1), and
integrate them into our delighting pipeline (Sec. 3.2 to Sec. 3.5).

###  3.1 Data Processing

We modify the Multi-PIE dataset [14] to synthesize our training data for
portrait delighting. We chose this dataset as it is the largest publicly
available dataset of real subjects. While other datasets capture more lighting
directions [38, 13], they lack diversity in terms of pose, expression, and
clothing. Fig. 2 illustrates our full data processing pipeline.

Figure 2: Our data processing pipeline

Each static image in the Multi-PIE database was captured under 19
illuminations; 18 directional flashes, and one non-flash image (room lights
only). We first remove the effects the room lights have on our flash images
(details are described in the supplementary document), resulting in 18 one-
light-at-a-time (OLAT) images [41] (see Fig. 2 (b)), where the target de-lit
image ($\mathbf{I_{dlt}}$) is the average over all our OLAT images, with added
luminance from our non-flash image. From this, we perform foreground masking
and face-parsing. (see Fig. 2 (e)).

We stress that the de-lit image $\mathbf{I_{dlt}}$ is different from an albedo
image. The difference is that the de-lit image contains subtle light
occlusions on non-convex geometric areas such as the ears, nose, and clothing.

To render input images, other learning based methods [41, 44, 32, 48] used
Image Based Lighting (IBL). But the mostly front-facing point lights used in
Multi-PIE prevent us from adequately sampling 360∘ environment maps. We
instead approximate environment illuminations using a weighted average of two
OLAT images, each tinted with different colour temperatures (see Fig. 2 (d))
similar to the VIDIT dataset [18]. This colour assumption in reasonable, as it
covers the majority of light found in the real world. High-intensity light
images were generated in the same manner by increasing the image brightness,
and colour adjusted versions of the de-lit, and room-light-only images were
also added to the dataset.

We chose a sample of 140 subjects (70 male, 70 female) under two arbitrary
poses, expressions and clothing, producing 240 unique images (220 for
training, 20 for testing). We use 1,293 lighting conditions for the training
set, and 1,180 for the test set, producing 284,460 images for training, and
25,860 for testing.

Figure 3: Overview of our network structure. We predict a shading-offset via a
separate decoder during training. All convolutions have a kernel size of 3,
and are followed by Instance normalization [42] and PReLU activation [17].
_Tanh_ activations are used to produce final outputs.

###  3.2 Basic Architecture

We train a U-Net [35] based CNN to estimate the ground-truth de-lit image
$\mathbf{I_{dlt}}$ from a foreground-masked upper-body portrait
$\mathbf{I_{src}}$ by minimizing a perceptual loss term
$\mathcal{L}_{\mathbf{perc}}$, which based on previous works [32, 29, 20] has
been shown to recover sharp details at multiple scales. This is computed as
the $\ell_{1}$ distance between the activations of the VGG-16 network [39]
pre-trained on ImageNet [9]. We also apply a small $\ell_{1}$ loss to speed up
color convergence:

$\begin{split}\mathcal{L}_{\mathbf{perc}}(\mathbf{A},\mathbf{B})=\sum_{i=1}^{5}\Big{[}\frac{1}{N_{i}}&\|VGG_{i}(\mathbf{A})-VGG_{i}(\mathbf{B})\|_{1}\Big{]}+\frac{0.2}{M}\|\mathbf{A}-\mathbf{B}\|_{1},\end{split}$
(1)

where $\mathbf{A}$ and $\mathbf{B}$ are images of the same subject with $M$
number of foreground pixels, $VGG_{i}$ and $N_{i}$ are the outputs and sizes
respectively of the final activations before the $i^{th}$ max-pooling layer in
VGG-16. We apply this function to our de-lit output
$\mathcal{D}_{1}(\mathbf{I_{src}})$ as:

$\mathcal{L}_{\mathbf{dlt}}=\mathcal{L}_{\mathbf{perc}}(\mathbf{I_{dlt}},\mathcal{D}_{1}(\mathbf{I_{src}})).$
(2)

This loss alone motivates a direct style transfer from $\mathbf{I_{src}}$ to
$\mathbf{I_{dlt}}$ without necessarily learning a physical separation of
shading and texture, which consequently leads to increased over-fitting to
certain modalities of our training data (i.e. lighting distribution, clothing
patterns) not fully represented in real-world portraits. Shortcomings of this
basic architecture are mitigated via our proposed contributions in Sec. 3.3,
3.4 and 3.5

###  3.3 Shading Offset

We add a separate decoder branch $\mathcal{D}_{2}$ to our network to learn the
difference between $\mathbf{I_{src}}$ and $\mathbf{I_{dlt}}$ in the form of a
shading-offset image: $\mathbf{I_{off}}=\mathbf{I_{src}}-\mathbf{I_{dlt}}$.
These two images differ only in their shading, so learning the difference
between them allows our encoder to learn a meaningful separation of shading
and texture. Different from some prior works that obtain their results by
subtracting the estimated offset image from the input [5, 30], we produce our
result directly via our delighting decoder $\mathcal{D}_{1}$ as this makes our
model less susceptible to small estimation errors in our shading-offset. Our
network pipeline is shown in Fig. 3.

We apply perceptual loss to our offset output
$\mathcal{D}_{2}(\mathbf{I_{src}})$ as:

$\mathcal{L}_{\mathbf{off}}=\mathcal{L}_{\mathbf{perc}}(\mathbf{I_{off}},\mathcal{D}_{2}(\mathbf{I_{src}})).$
(3)

###  3.4 Soft Shadowed Images

Since the light capture setup of our dataset is too sparse to represent non-
directional lighting characteristics such as soft-shadows, we approximate
these effects using a guided filter [16] technique to smooth the source image,
while preserving edges in the ground-truth image (see Fig. 2 (f)).

$\begin{split}\mathbf{I_{soft}}=&M_{nose}\odot\Omega(\mathbf{I_{src}},\mathbf{I_{dlt}},\epsilon)+\\\
&M_{mouth}\odot\Omega(\mathbf{I_{src}},\mathbf{I_{dlt}},\epsilon)+\\\
&M_{other}\odot\Omega(\mathbf{I_{src}},\mathbf{I_{dlt}},\kappa),\end{split}$
(4)

where $\Omega(I,R,r)$ is the guided filter function ($I$ is the input image,
$R$ is the edge reference, and $r$ is the window radius). To preserve
morphological features of the face, we use smaller sized filters for the nose
and mouth regions ($\epsilon\leq\kappa$). This is done using the masks
$M_{nose}$ and $M_{mouth}$ extracted from face parsing. The resulting image
approximates the effect of increasing the area of all lights in the input
image, softening sharp shadow boarders and specular highlights.

To regularize the strong influence of $\mathcal{L}_{\mathbf{dlt}}$ and
$\mathcal{L}_{\mathbf{off}}$, we apply a small $\ell_{1}$ loss to the outputs
of $\mathbf{I_{soft}}$ as:

$\displaystyle\mathcal{L}_{\mathbf{soft\mbox{-}dlt}}=\frac{0.6}{M}||\mathbf{I_{dlt}}-\mathcal{D}_{1}(\mathbf{I_{soft}})||_{1},$
(5)
$\displaystyle\mathcal{L}_{\mathbf{soft\mbox{-}off}}=\frac{0.6}{M}||\mathbf{I_{\mathbf{soft\mbox{-}off}}}-\mathcal{D}_{2}(\mathbf{I_{soft}})||_{1},$
(6)

where
$\mathbf{I_{\mathbf{soft\mbox{-}off}}}=\mathbf{I_{soft}}-\mathbf{I_{dlt}}$.

###  3.5 High-frequency Mask

Similar to [19], We emphasize sharp lighting discontinuities (e.g. shadow
boarders, reflections) in our loss function via a weight mask $\mathbf{W}$,
which we generate using the gradient difference between $\mathbf{I_{src}}$ and
its soft-shadowed version (see Fig. 2 (g)).

$\begin{split}&a=10*\max(\Delta\mathbf{I_{src}}-\Delta\Omega(\mathbf{I_{src}},\mathbf{I_{dlt}},15),0)\\\
&b=median(a)\\\ &\mathbf{W}=\min(b+gauss(b),1),\end{split}$ (7)

where $\Delta$ represents the sum of directional gradients along the vertical
and horizontal axes. For high-frequency shading features, the gradient of
$\mathbf{I_{src}}$ should be higher than its filtered counterpart, which we
store in $a$. Afterwards, we apply a median filter to remove noise, and add a
small Gaussian blur to increase its receptive field to neighbouring pixels.

Our method is different from Hou et al. [19], who fit a morphable model to the
face, estimate the lighting, and perform ray-casting to produce a shadow mask.
Their method is more useful for relighting than delighting, since estimation
errors could cause the shadow mask to miss the true shadow boarder. Ours is
less vulnerable to estimation errors, and can target areas outside the face.

We compliment the high-level feature activations of
$\mathcal{L}_{\mathbf{dlt}}$ with this importance mask:

$\mathcal{L}_{\mathbf{msk}}=\sum_{i=1}^{3}\frac{1}{S_{i}}\|W_{i}\odot\Big{(}VGG_{i}(\mathbf{I_{dlt}})-VGG_{i}(\mathcal{D}_{1}(\mathbf{I_{src}})\Big{)}\|_{1},$
(8)

where $W_{i}$ is the high-frequency shading mask resized to fit the dimensions
of $VGG_{i}$, and $S_{i}$ is the sum of $W_{i}$.

Our full pipeline is illustrated in Fig. 3, where the final loss is the sum:

$\mathcal{L}=\mathcal{L}_{\mathbf{dlt}}+\mathcal{L}_{\mathbf{off}}+\mathcal{L}_{\mathbf{soft\mbox{-}dlt}}+\mathcal{L}_{\mathbf{soft\mbox{-}off}}+\mathcal{L}_{\mathbf{msk}}.$
(9)

##  4 Results

Figure 4: Evaluation on our testing dataset. Notice how our method is the only
one to remove the scarf shadow in the middle row. A different crop of this
subject was used for EMR since their method was trained on only face regions.
Figure 5: Evaluations on in-the-wild images. Red boxes emphasize areas in
previous works improved by our method. Notice how our method provides the most
consistent results in terms of delighting and texture preservation.

###  4.1 Implementation and Data Setup

We implement our model in PyTorch [33], and train for 4 epochs with a learning
rate of 0.0002 using the Adam optimizer [22]. All images are resized to
$256\times 256$ resolution, with pixel values normalized to $[-1,1]$. The
average running time of our network for delighting (without shading-offset
prediction) is 25ms on a NVIDIA GTX 1080 GPU.

To prepare the training data in our experiments, we apply random flips along
the vertical axis, and random cropping with window sizes chosen uniformly from
$[280,480]$, where $480$ covers the entire image. When generating soft-shadow
images in Eq. 4, the large filter radius $\kappa$ is chosen randomly from
$[7,35]$ to increase robustness to different penumbra sizes, while the nose
and mouth radius $\epsilon$ is fixed at 7.

Prior Work. We compare our method with the albedo prediction modules of two
state-of-the-art methods: Explicit Multiple Reflectance Channel Modeling (EMR)
[44] and Total Relighting (TR) [32], which have demonstrated superior
performance over previous related works. TR apply VGG-perceptual loss (see Eq.
1) on the estimated albedo, along with a Least-Squares GAN [28] discriminator
to remove high-frequency shading. EMR use only $\ell_{1}$ loss on the albedo,
and estimate the source illumination and a face parsing map as auxiliary tasks
to improve training stability.

Retrained models. Besides the pretrained models from prior work, we
_retrained_ EMR and TR on our dataset, denoted EMR (retrained) and TR
(retrained) respectively. We based our implementations of their models on the
albedo prediction networks and loss functions described in their papers [32,
44]. For EMR (retrained), We use our segmentations in Fig. 2 (e) as the
ground-truth for face parsing. and since our pipeline doesn’t utilize
$360^{\textrm{o}}$ environment maps, we use our shading-offset decoder
$\mathcal{D}_{2}$ as a substitute for their source lighting estimator. This
allows us to use the lighting information found in our shading-offset images
to perform this task, just as we do in our method (see Fig. 3).

###  4.2 Qualitative Evaluations

In Fig. 4, we compare our results qualitatively against these two methods on
our testing dataset. From the results, we can see that our method outperforms
both EMR and EMR (retrained) in terms of shadow and specular removal (middle
and bottom rows). Our method also recovers large-scale textures, whereas TR
and TR (retrained) are prone to incorrect colour estimations as can be seen
with the magenta shirt in the top row. Although TR handles most shading
effects on the face, it still leaves shadow boarders in some extreme cases
(bottom row, (f)), while our proposed method can remove them effectively,
indicating that our method is generalized to harsh cases.

We show results on in-the-wild images in Fig. 5. Here, we see that our method
is able to delight subjects under a wide range of complex illuminations, while
also preserving important details such as clothing patterns and facial hair.
Our method handles variant conditions more gracefully than other works. For
example, TR (retrained) often removes important content details such as beards
((b) and (d)), while EMR (retrained) preserves shadow boarders and sharp
reflections ((b) and (c)), and is unstable when images contain large non-face
components (f). In (a), we demonstrate another failure case of TR (retrained)
where light is coming from behind the subject. Although this lighting
condition is missing from our dataset, our model generalizes to this case very
well by the contribution of our soft-shadow loss (see Sec. 4.4 for further
insight).

###  4.3 Quantitative Evaluations

Table 1: Results of delighting on our testing dataset. Arrows indicate whether loss is minimized ($\downarrow$) or maximized ($\uparrow$). Our method outperforms prior works on all metrics. | Metric
---|---
Method | RMSE$\downarrow$ | SSIM$\uparrow$ | li-SSIM$\uparrow$ | LPIPS$\downarrow$
EMR (retrained) | 0.047 | 0.940 | 0.949 | 0.047
TR (retrained) | 0.056 | 0.934 | 0.948 | 0.048
Proposed | 0.044 | 0.946 | 0.955 | 0.037

Quantitative evaluations on our testing dataset are shown Tab. 1, where we
compare our delighting performance with TR (retrained) and EMR (retrained)
using the following metrics: root mean squared error (RMSE), structural
similarity (SSIM) [45] luminance-invariant SSIM (li-SSIM) and learned
perceptual image patch similarity (LPIPS) [49] (version 0.1). Our li-SSIM is
like traditional SSIM with luminance parameter $\alpha=0$. This is meant to
avoid biases to large-scale colour variations, penalizing contrast and
structural errors only. From the results we can see that our method
outperforms other works across all metrics. Notably, TR gains a significant
improvement on the li-SSIM metric over SSIM, but is still behind our method.
These results reflect consistent improvements in most of our qualitative
images. More of our results can be found in the supplementary document.

###  4.4 Ablation study

Our baseline model consists of our network trained using only the basic loss
$\mathcal{L_{\mathbf{dlt}}}$ (see Eq. 2). We add each of our proposed losses
onto this model to evaluate their contributions. Quantitative results of all
our ablations using our testing dataset are shown in Tab. 2.

Shading-offset loss. We show the benefits provided by our offset decoder
$\mathcal{D}_{2}$ by training an additional model with shading-offset loss
$\mathcal{L_{\mathbf{off}}}$ (see Eq. 3) added to our baseline. Qualitative
results in Fig. 6 (c) demonstrate more stable texture recovery and light
removal when using offset loss. The likely reason is that it could make the
learned latent features more discriminative between different colour
variations caused by shading and texture. In Tab 2 (Testing Dataset block), we
see that our model with offset loss performs better than our baseline across
all metrics, especially RMSE and LPIPS, indicating improved large-scale
texture recovery.

Figure 6: Ablation study on (a, b, c) shading-offset loss, (d, e, f) soft-
shadow loss and (g, h, i) masked loss. Red boxes in (h) highlight shadows.

Soft-shadow loss. We add the soft-shadow loss
$\mathcal{L_{\mathbf{soft\mbox{-}dlt}}}+\mathcal{L_{\mathbf{soft\mbox{-}off}}}$
(see Eq. 5 & 6) to our shading-offset ablation. While our perceptual loss in
$\mathcal{L}_{\mathbf{dlt}}$ allows for stable delighting under harsh
illumination, it can lead to over-fitting due to the mostly front-facing
directional lights from our training data, and can’t cope with diffuse
illuminations very well. From the results in Fig. 6 (f), we can see that
adding our soft-shadow regularization not only improves performance on diffuse
illumination conditions (bottom row), but also enables our model to recognize
rare edge cases such as in the top row, where we observe that strong light
diffractions through the hair and around the left cheek are removed.

To better quantify the benefits of our soft-shadow regularization, we create a
new testing dataset (dubbed as Alt. Lighting) using the face relighting method
of Hou et al. [19] to obtain sufficiently variant testing samples. The de-lit
images from our original testing dataset were relit under 13 lighting
conditions from a wide range of angles not seen in our training dataset.
Quantitative results in Tab. 2 (Alt. Lighting block) demonstrates how our
soft-shadow loss improves robustness to these conditions, particularly on the
RMSE and LPIPS metrics.

Masked loss. We add our masked loss $\mathcal{L_{\mathbf{msk}}}$ (Eq. 7) to
our soft-shadow ablation. As the results in Fig. 6 (i) show, our masked loss
improves the removal of complex shadows that are intertwined with clothing
features. It is noteworthy that although the model without
$\mathcal{L_{\mathbf{msk}}}$ can handle a wide-range of harsh illumination
conditions situated around the face area, abnormal cast shadows, particularly
around the torso are often preserved. By adding extra importance to these
regions via $\mathcal{L_{\mathbf{msk}}}$, our model reduces the rate at which
they get preserved in the result. Quantitative results in Tab. 2 show that our
model without $\mathcal{L_{\mathbf{msk}}}$ outperforms our proposed method on
most metrics, although we can see the benefit of $\mathcal{L_{\mathbf{msk}}}$
in our testing cases visually. This could mean the weighted mask creates a
bias towards these relatively small regions in the loss function, at the
expense of larger image structures.

Table 2: Ablation results on our loss functions. $\mathcal{L_{\mathbf{soft}}}=\mathcal{L_{\mathbf{soft\mbox{-}dlt}}}+\mathcal{L_{\mathbf{soft\mbox{-}off}}}$. We evaluate against two datasets: Testing Dataset (left block) and Alt. Lighting (right block), where our testing dataset is the same one used in Tab. 1.  | Metric
---|---
| Testing Dataset | Alt. Lighting
Method | RMSE$\downarrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | RMSE$\downarrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$
(A) | $\mathcal{L_{\mathbf{dlt}}}$ | 0.056 | 0.943 | 0.042 | 0.040 | 0.979 | 0.052
(B) | (A)$+\mathcal{L_{\mathbf{off}}}$ | 0.046 | 0.949 | 0.035 | 0.041 | 0.979 | 0.052
(C) | (B)$+\mathcal{L_{\mathbf{soft}}}$ | 0.046 | 0.949 | 0.035 | 0.026 | 0.985 | 0.033
(D) | (C)$+\mathcal{L_{\mathbf{msk}}}$ | 0.044 | 0.946 | 0.037 | 0.026 | 0.984 | 0.037
Figure 7: Edge cases: Our results on images under-represented in our training
data. Source images of (d) & (e) are from the evaluation dataset of [50].

###  4.5 Edge Cases

While the Multi-PIE [14] dataset exhibits much diversity in terms of clothing,
expression and identity, it nonetheless has significant domain biases such as
a 95% European/Asian demographic captured under mostly front-facing
directional lightings. This can lead to failure cases when presented with dark
skin albedos, and challenging illuminations such as light coming from extreme
angles or shadows cast by foreign objects. In this section, we offer a
qualitative evaluation on these under-represented cases.

We present our evaluation on people with dark skin in Fig. 7 (a-c). From the
results, we can see that our method is capable of recovering dark skin
textures, although noticeable lightening is observed.

In Fig 7 (d-f), we demonstrate our results on (d, e) irregularly shaped
shadows, and (f) lighting from predominantly behind the subject. While no such
images were present in our training data, our model nonetheless generalizes to
these cases surprisingly well with a few minor artifacts.

##  5 Applications

Figure 8: Relighting. The results of face relighting [19] (top row) before and
(bottom row) after our delighting step. Figure 9: Face Parsing: Here we show
(a) the input image with (b) its semantic parsing output, and (c) our de-lit
image with (d) its semantic parsing output.

Face Relighting. Fig. 8 illustrates the benefits of our delighting as a
preprocessing step for other relighting methods. Through our shadow removal
and colour normalization, we achieve more natural looking results on HDRI
environments. Relighting is performed using the method of Hou et al. [19],
which we adapt for environment map renderings by taking a weighted sum of 512
unique illuminations (similar to [8]).

Face Parsing. Fig. 9 illustrates the importance of delighting when performing
semantic segmentation on portraits. Our model offers significant performance
gains to this task by enhancing the visual clarity of glasses, eyes and mouth
regions. Also noteworthy is the neck shadow (middle-row, (a)), where without
delighting, the neck region outside is classified as clothing (middle-row,
(b)). The face parser used is a BiSeNet model [47] trained on the CelebAMask-
HQ dataset [24].

##  6 Conclusion

We propose a deep neural network for delighting portrait images under a wide
range of illumination conditions. Texture recovery and generalization is
improved via estimating shading-offset images, using soft-shadow variants of
the input, and our weighted loss function. Each contribution greatly improves
delighting performance over the previous work in terms of removing shading
features and preserving image content. We tested our model as a useful
preprocessing tool for other computer vision tasks.

Failure cases can arise due to biases in the training data as outlined in Sec.
4.5, so future work must consider fairness in terms of physical traits and
lighting variations when creating and using datasets. To prevent softening of
dark textures (Fig. 7 (f)), future work can also focus on estimating global
image statistics, such as ambient light intensity, which will place a lower
bound on the darkest regions of the image caused by shading.

Acknowledgments This work was supported by the Entrepreneurial University
Programme from the Tertiary Education Commission, and MBIE Smart Idea
Programme by Ministry of Business, Innovation and Employment in New Zealand.
We thank all image providers, including Flickr users: ”Debarshi Ray”, ”5of7”
and ”photographer695”, whose photographs were cropped, and processed by our
neural network. Sources are provided in the supplementary material.

## References

  * [1] Abdal, R., Zhu, P., Mitra, N.J., Wonka, P.: Styleflow: Attribute-conditioned exploration of stylegan-generated images using conditional continuous normalizing flows. ACM Transactions on Graphics (TOG) 40(3), 1–21 (2021)
  * [2] Ahmed, A., Farag, A.: A new statistical model combining shape and spherical harmonics illumination for face reconstruction. In: International Symposium on Visual Computing. pp. 531–541. Springer (2007)
  * [3] Beveridge, J.R., Bolme, D.S., Draper, B.A., Givens, G.H., Lui, Y.M., Phillips, P.J.: Quantifying how lighting and focus affect face recognition performance. In: 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition-Workshops. pp. 74–81. IEEE (2010)
  * [4] Blanz, V., Vetter, T.: A morphable model for the synthesis of 3d faces. In: Proceedings of the 26th annual conference on Computer graphics and interactive techniques. pp. 187–194 (1999)
  * [5] Capece, N., Banterle, F., Cignoni, P., Ganovelli, F., Scopigno, R., Erra, U.: Deepflash: Turning a flash selfie into a studio portrait. Signal Processing: Image Communication 77, 28–39 (2019)
  * [6] Chen, B.C., Chen, C.S., Hsu, W.H.: Face recognition and retrieval using cross-age reference coding with cross-age celebrity dataset. IEEE Transactions on Multimedia 17(6), 804–815 (2015)
  * [7] Chen, X., Wu, H., Jin, X., Zhao, Q.: Face illumination manipulation using a single reference image by adaptive layer decomposition. IEEE Transactions on Image Processing 22(11), 4249–4259 (2013)
  * [8] Debevec, P., Hawkins, T., Tchou, C., Duiker, H.P., Sarokin, W., Sagar, M.: Acquiring the reflectance field of a human face. In: Proceedings of the 27th annual conference on Computer graphics and interactive techniques. pp. 145–156 (2000)
  * [9] Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical image database. In: 2009 IEEE conference on computer vision and pattern recognition. pp. 248–255. Ieee (2009)
  * [10] Deng, Y., Yang, J., Chen, D., Wen, F., Tong, X.: Disentangled and controllable face image generation via 3d imitative-contrastive learning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5154–5163 (2020)
  * [11] Egger, B., Smith, W.A., Tewari, A., Wuhrer, S., Zollhoefer, M., Beeler, T., Bernard, F., Bolkart, T., Kortylewski, A., Romdhani, S., et al.: 3d morphable face models—past, present, and future. ACM Transactions on Graphics (TOG) 39(5), 1–38 (2020)
  * [12] Fahmy, G., El-Sherbeeny, A., Mandala, S., Abdel-Mottaleb, M., Ammar, H.: The effect of lighting direction/condition on the performance of face recognition algorithms. In: Biometric Technology for Human Identification III. vol. 6202, p. 62020J. International Society for Optics and Photonics (2006)
  * [13] Georghiades, A.S., Belhumeur, P.N., Kriegman, D.J.: From few to many: Illumination cone models for face recognition under variable lighting and pose. IEEE transactions on pattern analysis and machine intelligence 23(6), 643–660 (2001)
  * [14] Gross, R., Matthews, I., Cohn, J., Kanade, T., Baker, S.: Multi-pie. Image and vision computing 28(5), 807–813 (2010)
  * [15] Han, X., Yang, H., Xing, G., Liu, Y.: Asymmetric joint gans for normalizing face illumination from a single image. IEEE Transactions on Multimedia 22(6), 1619–1633 (2019)
  * [16] He, K., Sun, J., Tang, X.: Guided image filtering. IEEE transactions on pattern analysis and machine intelligence 35(6), 1397–1409 (2012)
  * [17] He, K., Zhang, X., Ren, S., Sun, J.: Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In: Proceedings of the IEEE international conference on computer vision. pp. 1026–1034 (2015)
  * [18] Helou, M.E., Zhou, R., Barthas, J., Süsstrunk, S.: Vidit: Virtual image dataset for illumination transfer. arXiv preprint arXiv:2005.05460 (2020)
  * [19] Hou, A., Zhang, Z., Sarkis, M., Bi, N., Tong, Y., Liu, X.: Towards high fidelity face relighting with realistic shadows. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14719–14728 (2021)
  * [20] Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: European conference on computer vision. pp. 694–711. Springer (2016)
  * [21] Kanamori, Y., Endo, Y.: Relighting humans: occlusion-aware inverse rendering for full-body human images. ACM Transactions on Graphics (TOG) 37(6), 1–11 (2018)
  * [22] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)
  * [23] Lagunas, M., Sun, X., Yang, J., Villegas, R., Zhang, J., Shu, Z., Masia, B., Gutierrez, D.: Single-image full-body human relighting (2021)
  * [24] Lee, C.H., Liu, Z., Wu, L., Luo, P.: Maskgan: Towards diverse and interactive facial image manipulation. In: IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2020)
  * [25] Lee, K., Ho, J., Kriegman, D.: Acquiring linear subspaces for face recognition under variable lighting. IEEE Trans. Pattern Anal. Mach. Intelligence 27(5), 684–698 (2005)
  * [26] Ling, S., Lin, Y., Fu, K., You, D., Cheng, P.: A high-performance face illumination processing method via multi-stage feature maps. Sensors 20(17), 4869 (2020)
  * [27] Mallikarjun, B., Tewari, A., Dib, A., Weyrich, T., Bickel, B., Seidel, H.P., Pfister, H., Matusik, W., Chevallier, L., Elgharib, M., et al.: Photoapp: Photorealistic appearance editing of head portraits. ACM Transactions on Graphics 40(4), 1–16 (2021)
  * [28] Mao, X., Li, Q., Xie, H., Lau, R.Y., Wang, Z., Paul Smolley, S.: Least squares generative adversarial networks. In: Proceedings of the IEEE international conference on computer vision. pp. 2794–2802 (2017)
  * [29] Martin-Brualla, R., Pandey, R., Yang, S., Pidlypenskyi, P., Taylor, J., Valentin, J., Khamis, S., Davidson, P., Tkach, A., Lincoln, P., et al.: Lookingood: Enhancing performance capture with real-time neural re-rendering. arXiv preprint arXiv:1811.05029 (2018)
  * [30] Nagano, K., Luo, H., Wang, Z., Seo, J., Xing, J., Hu, L., Wei, L., Li, H.: Deep face normalization. ACM Transactions on Graphics (TOG) 38(6), 1–16 (2019)
  * [31] Nestmeyer, T., Lalonde, J.F., Matthews, I., Lehrmann, A.: Learning physics-guided face relighting under directional light. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5124–5133 (2020)
  * [32] Pandey, R., Escolano, S.O., Legendre, C., Haene, C., Bouaziz, S., Rhemann, C., Debevec, P., Fanello, S.: Total relighting: learning to relight portraits for background replacement. ACM Transactions on Graphics (TOG) 40(4), 1–21 (2021)
  * [33] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32, 8026–8037 (2019)
  * [34] Qiu, Y., Xiong, Z., Han, K., Wang, Z., Xiong, Z., Han, X.: Learning inverse rendering of faces from real-world videos (2020)
  * [35] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: International Conference on Medical image computing and computer-assisted intervention. pp. 234–241. Springer (2015)
  * [36] Sengupta, S., Kanazawa, A., Castillo, C.D., Jacobs, D.W.: Sfsnet: Learning shape, reflectance and illuminance of facesin the wild’. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6296–6305 (2018)
  * [37] Shu, Z., Yumer, E., Hadap, S., Sunkavalli, K., Shechtman, E., Samaras, D.: Neural face editing with intrinsic image disentangling. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 5541–5550 (2017)
  * [38] Sim, T., Baker, S., Bsat, M.: The cmu pose, illumination, and expression (pie) database. In: Proceedings of Fifth IEEE International Conference on Automatic Face Gesture Recognition. pp. 53–58. IEEE (2002)
  * [39] Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014)
  * [40] Song, G., Cham, T.J., Cai, J., Zheng, J.: Half-body portrait relighting with overcomplete lighting representation. In: Computer Graphics Forum. Wiley Online Library (2021)
  * [41] Sun, T., Barron, J.T., Tsai, Y.T., Xu, Z., Yu, X., Fyffe, G., Rhemann, C., Busch, J., Debevec, P.E., Ramamoorthi, R.: Single image portrait relighting. ACM Trans. Graph. 38(4), 79–1 (2019)
  * [42] Ulyanov, D., Vedaldi, A., Lempitsky, V.: Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022 (2016)
  * [43] Wang, Y., Zhang, L., Liu, Z., Hua, G., Wen, Z., Zhang, Z., Samaras, D.: Face relighting from a single image under arbitrary unknown lighting conditions. IEEE Transactions on Pattern Analysis and Machine Intelligence 31(11), 1968–1984 (2008)
  * [44] Wang, Z., Yu, X., Lu, M., Wang, Q., Qian, C., Xu, F.: Single image portrait relighting via explicit multiple reflectance channel modeling. ACM Transactions on Graphics (TOG) 39(6), 1–13 (2020)
  * [45] Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13(4), 600–612 (2004)
  * [46] Xia, W., Zhang, Y., Yang, Y., Xue, J.H., Zhou, B., Yang, M.H.: Gan inversion: A survey. arXiv preprint arXiv:2101.05278 (2021)
  * [47] Yu, C., Wang, J., Peng, C., Gao, C., Yu, G., Sang, N.: Bisenet: Bilateral segmentation network for real-time semantic segmentation (2018)
  * [48] Zhang, L., Zhang, Q., Wu, M., Yu, J., Xu, L.: Neural video portrait relighting in real-time via consistency modeling. arXiv preprint arXiv:2104.00484 (2021)
  * [49] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 586–595 (2018)
  * [50] Zhang, X., Barron, J.T., Tsai, Y.T., Pandey, R., Zhang, X., Ng, R., Jacobs, D.E.: Portrait shadow manipulation. ACM Transactions on Graphics (TOG) 39(4), 78–1 (2020)
  * [51] Zhou, H., Hadap, S., Sunkavalli, K., Jacobs, D.W.: Deep single-image portrait relighting. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 7194–7202 (2019)
  * [52] Bertalmio, M., Bertozzi, A.L., Sapiro, G.: Navier-stokes, fluid dynamics, and image and video inpainting. In: Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001. vol. 1, pp. I–I. IEEE (2001)

Supplementary Material: Deep Portrait Delighting

##  7 OLAT and Target synthesis

In sec. 3.1 of our paper, we describe how we process our data for training.
Here, we provide a deeper overview of how we remove the effect of the room
lights in the Multi-PIE [14] flash images, and synthesize the target de-lit
image.

Since the images with no camera-flashes (room-lit) are available, we can
simply remove their effect with the following equation.

$\mathbf{I_{ratio}=1-(\frac{L(I_{roomlights})}{L(I_{flash})})}$ (10)
$\mathbf{I_{no\\_ambient}=I_{ratio}\odot I_{flash}}$ (11)

Where $\odot$ is pixel-wise multiplication, $\mathbf{L(A)}$ refers to the
luminance channel of $\mathbf{A}$ in Lab color space.
$\mathbf{I_{roomlights}}$ and $\mathbf{I_{flash}}$ are the room-lit and flash
images respectively.

Another problem is that the room lights create sharp specular reflections on
the faces of some subjects. Since the brightness of these regions is near full
(images were taken with a LDR camera), removing these from our flash images
results in dark blemishes across the face (Fig. 10 (c)). We detect these
regions using the following equation:

$\mathbf{I_{specular}=min(1,\frac{{I_{roomlights}}^{2}}{I_{average}})^{4})},$
(12)

where $\mathbf{I_{average}}$ is the mean average over all flash images. We
correct these small regions using Navier-Strokes [52] inpainting to produce
$\mathbf{I_{OLAT}}$. An illustration of each step is shown in Fig. 10.

To synthesize the target de-lit image, we approximate ambient lighting as the
average over all our 18 OLAT images, producing $\mathbf{I_{dlt}}$. From here,
we modify $\mathbf{I_{dlt}}$ using the following equation:

$\mathbf{L(I_{dlt})}=\mathbf{L(I_{dlt})}+6\mathbf{L(I_{room\\_nospec})},$ (13)

where $\mathbf{I_{room\\_nospec}}$ is $\mathbf{I_{roomlights}}$ with
specularities removed (like $\mathbf{I_{OLAT}}$). Adding the luminance of the
room-lit image back into $\mathbf{I_{dlt}}$ makes the lighting appear more
uniform.

(a) $\mathbf{I_{flash}}$

(b) $\mathbf{I_{ratio}}$

(c) $\mathbf{I_{no\\_ambient}}$

(d) $\mathbf{I_{roomlights}}$

(e) $\mathbf{I_{dlt}}$

(f) $\mathbf{I_{specular}}$

(g) $\mathbf{I_{OLAT}}$

(h) $\mathbf{I_{room{\\_}nospec}}$

Figure 10: Illustration of our data synthesis stage.

##  8 More comparisons

In sec. 4.2 of our paper, we showed qualitative comparisons against TR and EMR
against our testing dataset. However, due to different ground-truth modalities
(i.e. albedo quality and targeted regions), we don’t compare quantitatively
against the pretrained models of TR and EMR in sec. 4.3. As an extension to
Fig. 4 in the main paper, we provide a more in-depth qualitative comparison.

Upper-body portraits In Fig. 11 We provide examples from our testing dataset
of images containing significantly large clothing regions. Since EMR wasn’t
trained on such examples, we do not present their results in this section, and
instead show only EMR (retrained).

Face portraits In Fig. 12, we show more comparisons of images cropped around
the face region. Here, results of all prior works mentioned in the paper are
shown.

##  9 More soft-shadow loss examples

In Fig. 13, we show more qualitative examples of our soft-shadow
regularization (see sec. 3.2 of our paper). Input images are from our Alt.
Lighting dataset (see sec. 4.4 of our paper).

##  10 Shadow Removal

The paper of Zhang et al. [50] focuses only on a subset of the delighting
task: shadow removal/softening, hence we did not compare against their method
in the main paper. We include this comparison in Fig. 14 so that viewers can
qualitatively assess our shadow removal. In their work, they design an
extensive dataset of shadowed/un-shadowed image pairs, in which shadows are
cast by irregularly shaped objects outside the image (foreign shadows). While
no such shadows were present in our training data, our model holds up well,
while also achieving the overall delighting task including reflection removal.

(a) Input

(b) Ground truth

(c) Ours

(d) TR (retrained)

(e) EMR (retrained)

(f) TR

Figure 11: We show more comparisons against other methods using upper body
images from our testing dataset.

(a) Input

(b) Ground truth

(c) Ours

(d) TR (retrained)

(e) EMR (retrained)

(f) TR

(g) EMR

Figure 12: We show more comparisons against other methods using face cropped
images from our testing dataset.

(a) Input

(b) w/o soft-shadow

(c) w/ soft-shadow

(d) Ground truth

Figure 13: Soft-shadow ablation: Input images from Alt. Lighting dataset.

(a) Input

(b) Zhang et al. [50]

(c) Ours

Figure 14: Shadow Removal Comparison 1: Comparison with the method of Zhang et
al. [50] on irregularly shaped shadows. All images in (a) and (b) are taken
from the author’s website.